2024-09-17T06:46:21.820 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-17T06:46:21.842 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-17T06:46:21.920 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-16_22:08:02-orch-reef-distro-default-smithi/7908127 branch: reef description: orch/cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v2only root} email: ceph-qa@ceph.com first_in_suite: false job_id: '7908127' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-09-16_22:08:02-orch-reef-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: reef ceph: conf: client.0: debug ms: 1 debug objecter: 20 debug rados: 20 global: ms bind msgr1: false ms bind msgr2: true ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: debug ms: 1 debug osd: 20 osd debug reject backfill probability: 0.3 osd delete sleep: 1 osd max backfills: 3 osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - CEPHADM_STRAY_DAEMON - CEPHADM_FAILED_DAEMON - but it is still running - objects unfound and apparently lost - \(POOL_APP_NOT_ENABLED\) log-only-match: - CEPHADM_ sha1: 19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} cephadm: cephadm_mode: root install: ceph: flavor: default sha1: 19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: reef sha1: 1cbdc9211df088cccbd94423c7cf3a2b85fe432b owner: scheduled_teuthology@teuthology priority: 930 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b - ceph.iscsi.iscsi.a seed: 5234 sha1: 19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 sleep_before_teardown: 0 subset: 46/64 suite: orch suite_branch: reef suite_path: /home/teuthworker/src/git.ceph.com_ceph_1cbdc9211df088cccbd94423c7cf3a2b85fe432b/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 1cbdc9211df088cccbd94423c7cf3a2b85fe432b targets: smithi100.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPbRaSjj7tNox1dEJDJ/NzWavomygeIPZ+2JJBJczkmHaM7h6C1kk36HEbCl08IxZaZ8l3vNyIP5zECNzG7HhD4= smithi135.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBASlQs642ZrHg/1+r0MTv8V1oM9NU8KLhaEwGz3YfEzQeK5SR2CwslGtXpzlwpzZpPg47R10F1OUkxbHQ3q7hdk= tasks: - pexec: all: - sudo dnf remove nvme-cli -y - sudo dnf install nvmetcli nvme-cli -y - install: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - full_sequential: - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-16_22:08:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-17T06:46:21.920 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_1cbdc9211df088cccbd94423c7cf3a2b85fe432b/qa; will attempt to use it 2024-09-17T06:46:21.921 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_1cbdc9211df088cccbd94423c7cf3a2b85fe432b/qa/tasks 2024-09-17T06:46:21.922 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-17T06:46:21.923 INFO:teuthology.task.internal:Checking packages... 2024-09-17T06:46:21.949 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '19e2a2beb6f41a3ab62650e38614eb29b59dcbb9' 2024-09-17T06:46:21.949 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-17T06:46:21.949 INFO:teuthology.packaging:ref: None 2024-09-17T06:46:21.949 INFO:teuthology.packaging:tag: None 2024-09-17T06:46:21.949 INFO:teuthology.packaging:branch: reef 2024-09-17T06:46:21.949 INFO:teuthology.packaging:sha1: 19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-17T06:46:21.949 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=reef 2024-09-17T06:46:22.239 INFO:teuthology.task.internal:Found packages for ceph version 18.2.4-855.g19e2a2be 2024-09-17T06:46:22.240 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-17T06:46:22.246 INFO:teuthology.task.internal:no buildpackages task found 2024-09-17T06:46:22.246 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-17T06:46:22.288 INFO:teuthology.task.internal:Saving configuration 2024-09-17T06:46:22.303 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-17T06:46:22.314 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-17T06:46:22.334 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi100.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-16_22:08:02-orch-reef-distro-default-smithi/7908127', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-17 06:41:31.497825', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPbRaSjj7tNox1dEJDJ/NzWavomygeIPZ+2JJBJczkmHaM7h6C1kk36HEbCl08IxZaZ8l3vNyIP5zECNzG7HhD4='} 2024-09-17T06:46:22.355 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi135.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-16_22:08:02-orch-reef-distro-default-smithi/7908127', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-17 06:41:31.496207', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBASlQs642ZrHg/1+r0MTv8V1oM9NU8KLhaEwGz3YfEzQeK5SR2CwslGtXpzlwpzZpPg47R10F1OUkxbHQ3q7hdk='} 2024-09-17T06:46:22.355 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-17T06:46:22.360 INFO:teuthology.task.internal:roles: ubuntu@smithi100.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-09-17T06:46:22.360 INFO:teuthology.task.internal:roles: ubuntu@smithi135.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2024-09-17T06:46:22.360 INFO:teuthology.run_tasks:Running task console_log... 2024-09-17T06:46:22.431 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fbf12d77ac0>, signals=[15]) 2024-09-17T06:46:22.432 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-17T06:46:22.437 INFO:teuthology.task.internal:Opening connections... 2024-09-17T06:46:22.437 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi100.front.sepia.ceph.com 2024-09-17T06:46:22.438 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi100.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T06:46:22.515 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi135.front.sepia.ceph.com 2024-09-17T06:46:22.516 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi135.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T06:46:22.595 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-17T06:46:22.601 DEBUG:teuthology.orchestra.run.smithi100:> uname -m 2024-09-17T06:46:22.620 INFO:teuthology.orchestra.run.smithi100.stdout:x86_64 2024-09-17T06:46:22.620 DEBUG:teuthology.orchestra.run.smithi100:> cat /etc/os-release 2024-09-17T06:46:22.677 INFO:teuthology.orchestra.run.smithi100.stdout:NAME="CentOS Stream" 2024-09-17T06:46:22.678 INFO:teuthology.orchestra.run.smithi100.stdout:VERSION="9" 2024-09-17T06:46:22.678 INFO:teuthology.orchestra.run.smithi100.stdout:ID="centos" 2024-09-17T06:46:22.678 INFO:teuthology.orchestra.run.smithi100.stdout:ID_LIKE="rhel fedora" 2024-09-17T06:46:22.678 INFO:teuthology.orchestra.run.smithi100.stdout:VERSION_ID="9" 2024-09-17T06:46:22.678 INFO:teuthology.orchestra.run.smithi100.stdout:PLATFORM_ID="platform:el9" 2024-09-17T06:46:22.678 INFO:teuthology.orchestra.run.smithi100.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-17T06:46:22.678 INFO:teuthology.orchestra.run.smithi100.stdout:ANSI_COLOR="0;31" 2024-09-17T06:46:22.678 INFO:teuthology.orchestra.run.smithi100.stdout:LOGO="fedora-logo-icon" 2024-09-17T06:46:22.678 INFO:teuthology.orchestra.run.smithi100.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-17T06:46:22.678 INFO:teuthology.orchestra.run.smithi100.stdout:HOME_URL="https://centos.org/" 2024-09-17T06:46:22.678 INFO:teuthology.orchestra.run.smithi100.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-17T06:46:22.678 INFO:teuthology.orchestra.run.smithi100.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-17T06:46:22.679 INFO:teuthology.orchestra.run.smithi100.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-17T06:46:22.679 INFO:teuthology.lock.ops:Updating smithi100.front.sepia.ceph.com on lock server 2024-09-17T06:46:22.705 DEBUG:teuthology.orchestra.run.smithi135:> uname -m 2024-09-17T06:46:22.723 INFO:teuthology.orchestra.run.smithi135.stdout:x86_64 2024-09-17T06:46:22.724 DEBUG:teuthology.orchestra.run.smithi135:> cat /etc/os-release 2024-09-17T06:46:22.782 INFO:teuthology.orchestra.run.smithi135.stdout:NAME="CentOS Stream" 2024-09-17T06:46:22.783 INFO:teuthology.orchestra.run.smithi135.stdout:VERSION="9" 2024-09-17T06:46:22.783 INFO:teuthology.orchestra.run.smithi135.stdout:ID="centos" 2024-09-17T06:46:22.783 INFO:teuthology.orchestra.run.smithi135.stdout:ID_LIKE="rhel fedora" 2024-09-17T06:46:22.783 INFO:teuthology.orchestra.run.smithi135.stdout:VERSION_ID="9" 2024-09-17T06:46:22.783 INFO:teuthology.orchestra.run.smithi135.stdout:PLATFORM_ID="platform:el9" 2024-09-17T06:46:22.783 INFO:teuthology.orchestra.run.smithi135.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-17T06:46:22.784 INFO:teuthology.orchestra.run.smithi135.stdout:ANSI_COLOR="0;31" 2024-09-17T06:46:22.784 INFO:teuthology.orchestra.run.smithi135.stdout:LOGO="fedora-logo-icon" 2024-09-17T06:46:22.784 INFO:teuthology.orchestra.run.smithi135.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-17T06:46:22.784 INFO:teuthology.orchestra.run.smithi135.stdout:HOME_URL="https://centos.org/" 2024-09-17T06:46:22.784 INFO:teuthology.orchestra.run.smithi135.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-17T06:46:22.784 INFO:teuthology.orchestra.run.smithi135.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-17T06:46:22.784 INFO:teuthology.orchestra.run.smithi135.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-17T06:46:22.785 INFO:teuthology.lock.ops:Updating smithi135.front.sepia.ceph.com on lock server 2024-09-17T06:46:22.804 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-17T06:46:22.813 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-17T06:46:22.838 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-17T06:46:22.839 DEBUG:teuthology.orchestra.run.smithi100:> test '!' -e /home/ubuntu/cephtest 2024-09-17T06:46:22.841 DEBUG:teuthology.orchestra.run.smithi135:> test '!' -e /home/ubuntu/cephtest 2024-09-17T06:46:22.858 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-17T06:46:22.864 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-17T06:46:22.864 DEBUG:teuthology.orchestra.run.smithi100:> test -z $(ls -A /var/lib/ceph) 2024-09-17T06:46:22.899 DEBUG:teuthology.orchestra.run.smithi135:> test -z $(ls -A /var/lib/ceph) 2024-09-17T06:46:22.995 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-17T06:46:23.037 INFO:teuthology.run_tasks:Running task kernel... 2024-09-17T06:46:23.052 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-17T06:46:23.052 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-17T06:46:23.052 DEBUG:teuthology.orchestra.run.smithi100:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-17T06:46:23.053 DEBUG:teuthology.orchestra.run.smithi135:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-17T06:46:23.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T06:46:23.070 DEBUG:teuthology.orchestra.run.smithi100:> uname -r 2024-09-17T06:46:23.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T06:46:23.071 DEBUG:teuthology.orchestra.run.smithi135:> uname -r 2024-09-17T06:46:23.128 INFO:teuthology.orchestra.run.smithi100.stdout:5.14.0-503.el9.x86_64 2024-09-17T06:46:23.128 INFO:teuthology.task.kernel:Running kernel on smithi100: 5.14.0-503.el9.x86_64 2024-09-17T06:46:23.128 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum install -y kernel 2024-09-17T06:46:23.129 INFO:teuthology.orchestra.run.smithi135.stdout:5.14.0-503.el9.x86_64 2024-09-17T06:46:23.130 INFO:teuthology.task.kernel:Running kernel on smithi135: 5.14.0-503.el9.x86_64 2024-09-17T06:46:23.130 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum install -y kernel 2024-09-17T06:46:26.640 INFO:teuthology.orchestra.run.smithi135.stdout:CentOS Stream 9 - BaseOS 4.7 MB/s | 8.3 MB 00:01 2024-09-17T06:46:30.739 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS Stream 9 - BaseOS 1.4 MB/s | 8.3 MB 00:05 2024-09-17T06:46:31.259 INFO:teuthology.orchestra.run.smithi135.stdout:CentOS Stream 9 - AppStream 5.8 MB/s | 20 MB 00:03 2024-09-17T06:46:36.459 INFO:teuthology.orchestra.run.smithi135.stdout:CentOS Stream 9 - CRB 4.6 MB/s | 6.5 MB 00:01 2024-09-17T06:46:37.706 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS Stream 9 - AppStream 3.5 MB/s | 20 MB 00:05 2024-09-17T06:46:37.918 INFO:teuthology.orchestra.run.smithi135.stdout:CentOS Stream 9 - Extras packages 76 kB/s | 19 kB 00:00 2024-09-17T06:46:39.031 INFO:teuthology.orchestra.run.smithi135.stdout:Extra Packages for Enterprise Linux 23 MB/s | 23 MB 00:00 2024-09-17T06:46:44.048 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS Stream 9 - CRB 2.5 MB/s | 6.5 MB 00:02 2024-09-17T06:46:44.414 INFO:teuthology.orchestra.run.smithi135.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-09-17T06:46:45.707 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS Stream 9 - Extras packages 42 kB/s | 19 kB 00:00 2024-09-17T06:46:46.270 INFO:teuthology.orchestra.run.smithi135.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-17T06:46:46.271 INFO:teuthology.orchestra.run.smithi135.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-17T06:46:46.364 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-09-17T06:46:46.371 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-09-17T06:46:46.371 INFO:teuthology.orchestra.run.smithi135.stdout: Package Architecture Version Repository Size 2024-09-17T06:46:46.371 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-09-17T06:46:46.371 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-09-17T06:46:46.371 INFO:teuthology.orchestra.run.smithi135.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-17T06:46:46.371 INFO:teuthology.orchestra.run.smithi135.stdout:Installing dependencies: 2024-09-17T06:46:46.371 INFO:teuthology.orchestra.run.smithi135.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-17T06:46:46.371 INFO:teuthology.orchestra.run.smithi135.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-17T06:46:46.371 INFO:teuthology.orchestra.run.smithi135.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-17T06:46:46.371 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:46:46.371 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-09-17T06:46:46.372 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-09-17T06:46:46.372 INFO:teuthology.orchestra.run.smithi135.stdout:Install 4 Packages 2024-09-17T06:46:46.372 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:46:46.372 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 79 M 2024-09-17T06:46:46.372 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 126 M 2024-09-17T06:46:46.372 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-09-17T06:46:46.474 INFO:teuthology.orchestra.run.smithi135.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 440 kB/s | 36 kB 00:00 2024-09-17T06:46:46.536 INFO:teuthology.orchestra.run.smithi100.stdout:Extra Packages for Enterprise Linux 32 MB/s | 23 MB 00:00 2024-09-17T06:46:47.016 INFO:teuthology.orchestra.run.smithi135.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 25 MB/s | 16 MB 00:00 2024-09-17T06:46:47.616 INFO:teuthology.orchestra.run.smithi135.stdout:(3/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 28 MB/s | 35 MB 00:01 2024-09-17T06:46:47.891 INFO:teuthology.orchestra.run.smithi135.stdout:(4/4): kernel-modules-core-5.14.0-508.el9.x86_6 20 MB/s | 29 MB 00:01 2024-09-17T06:46:47.891 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-09-17T06:46:47.891 INFO:teuthology.orchestra.run.smithi135.stdout:Total 52 MB/s | 79 MB 00:01 2024-09-17T06:46:48.574 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-09-17T06:46:48.943 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-09-17T06:46:48.943 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-09-17T06:46:49.910 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-09-17T06:46:49.910 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-09-17T06:46:51.891 INFO:teuthology.orchestra.run.smithi100.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-09-17T06:46:52.337 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-09-17T06:46:53.025 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-17T06:46:53.107 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T06:46:53.828 INFO:teuthology.orchestra.run.smithi100.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-17T06:46:53.828 INFO:teuthology.orchestra.run.smithi100.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-17T06:46:53.919 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-17T06:46:53.926 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T06:46:53.926 INFO:teuthology.orchestra.run.smithi100.stdout: Package Architecture Version Repository Size 2024-09-17T06:46:53.927 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T06:46:53.927 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-17T06:46:53.927 INFO:teuthology.orchestra.run.smithi100.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-17T06:46:53.927 INFO:teuthology.orchestra.run.smithi100.stdout:Installing dependencies: 2024-09-17T06:46:53.927 INFO:teuthology.orchestra.run.smithi100.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-17T06:46:53.927 INFO:teuthology.orchestra.run.smithi100.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-17T06:46:53.927 INFO:teuthology.orchestra.run.smithi100.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-17T06:46:53.927 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:46:53.927 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-17T06:46:53.927 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T06:46:53.927 INFO:teuthology.orchestra.run.smithi100.stdout:Install 4 Packages 2024-09-17T06:46:53.927 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:46:53.928 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 79 M 2024-09-17T06:46:53.928 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 126 M 2024-09-17T06:46:53.928 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-17T06:46:54.252 INFO:teuthology.orchestra.run.smithi100.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 516 kB/s | 36 kB 00:00 2024-09-17T06:46:55.856 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T06:46:55.992 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T06:46:58.060 INFO:teuthology.orchestra.run.smithi100.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 4.0 MB/s | 16 MB 00:03 2024-09-17T06:47:00.841 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T06:47:00.942 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-17T06:47:01.934 INFO:teuthology.orchestra.run.smithi100.stdout:(3/4): kernel-modules-core-5.14.0-508.el9.x86_6 3.7 MB/s | 29 MB 00:07 2024-09-17T06:47:03.501 INFO:teuthology.orchestra.run.smithi100.stdout:(4/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 3.7 MB/s | 35 MB 00:09 2024-09-17T06:47:03.502 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-17T06:47:03.502 INFO:teuthology.orchestra.run.smithi100.stdout:Total 8.2 MB/s | 79 MB 00:09 2024-09-17T06:47:04.184 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-17T06:47:04.552 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-17T06:47:04.552 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-17T06:47:05.497 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-17T06:47:05.497 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-17T06:47:05.811 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T06:47:07.921 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-17T06:47:08.553 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-17T06:47:08.636 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T06:47:11.386 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T06:47:11.531 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T06:47:16.347 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T06:47:16.440 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-17T06:47:21.300 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T06:47:44.180 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T06:47:44.220 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-17T06:47:44.512 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-17T06:47:44.513 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-17T06:47:44.513 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T06:47:44.513 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T06:47:45.411 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T06:47:45.411 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:47:45.411 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-09-17T06:47:45.411 INFO:teuthology.orchestra.run.smithi135.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-17T06:47:45.411 INFO:teuthology.orchestra.run.smithi135.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-17T06:47:45.411 INFO:teuthology.orchestra.run.smithi135.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-17T06:47:45.411 INFO:teuthology.orchestra.run.smithi135.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-17T06:47:45.411 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:47:45.411 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-09-17T06:47:45.761 DEBUG:teuthology.orchestra.run.smithi135:> rpm -q kernel | sort -rV | head -n 1 2024-09-17T06:47:45.850 INFO:teuthology.orchestra.run.smithi135.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-17T06:47:45.850 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-17T06:47:45.850 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-17T06:47:45.850 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-17T06:47:45.850 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-17T06:47:45.851 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-17T06:47:45.851 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi135.front.sepia.ceph.com, path=None, version=distro) 2024-09-17T06:47:45.851 DEBUG:teuthology.orchestra.run.smithi135:> rpm -q kernel | sort -rV | head -n 1 2024-09-17T06:47:45.910 INFO:teuthology.orchestra.run.smithi135.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-17T06:47:45.910 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-17T06:47:45.910 DEBUG:teuthology.orchestra.run.smithi135:> sudo rpm -qi grub2-tools 2024-09-17T06:47:45.942 INFO:teuthology.orchestra.run.smithi135.stdout:Name : grub2-tools 2024-09-17T06:47:45.942 INFO:teuthology.orchestra.run.smithi135.stdout:Epoch : 1 2024-09-17T06:47:45.942 INFO:teuthology.orchestra.run.smithi135.stdout:Version : 2.06 2024-09-17T06:47:45.942 INFO:teuthology.orchestra.run.smithi135.stdout:Release : 82.el9 2024-09-17T06:47:45.942 INFO:teuthology.orchestra.run.smithi135.stdout:Architecture: x86_64 2024-09-17T06:47:45.942 INFO:teuthology.orchestra.run.smithi135.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-17T06:47:45.942 INFO:teuthology.orchestra.run.smithi135.stdout:Group : Unspecified 2024-09-17T06:47:45.943 INFO:teuthology.orchestra.run.smithi135.stdout:Size : 8274670 2024-09-17T06:47:45.943 INFO:teuthology.orchestra.run.smithi135.stdout:License : GPLv3+ 2024-09-17T06:47:45.943 INFO:teuthology.orchestra.run.smithi135.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-17T06:47:45.943 INFO:teuthology.orchestra.run.smithi135.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-17T06:47:45.943 INFO:teuthology.orchestra.run.smithi135.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-17T06:47:45.943 INFO:teuthology.orchestra.run.smithi135.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-17T06:47:45.943 INFO:teuthology.orchestra.run.smithi135.stdout:Packager : builder@centos.org 2024-09-17T06:47:45.943 INFO:teuthology.orchestra.run.smithi135.stdout:Vendor : CentOS 2024-09-17T06:47:45.944 INFO:teuthology.orchestra.run.smithi135.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-17T06:47:45.944 INFO:teuthology.orchestra.run.smithi135.stdout:Summary : Support tools for GRUB. 2024-09-17T06:47:45.944 INFO:teuthology.orchestra.run.smithi135.stdout:Description : 2024-09-17T06:47:45.944 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:47:45.944 INFO:teuthology.orchestra.run.smithi135.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-17T06:47:45.944 INFO:teuthology.orchestra.run.smithi135.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-17T06:47:45.944 INFO:teuthology.orchestra.run.smithi135.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-17T06:47:45.944 INFO:teuthology.orchestra.run.smithi135.stdout:hardware devices. 2024-09-17T06:47:45.944 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:47:45.944 INFO:teuthology.orchestra.run.smithi135.stdout:This subpackage provides tools for support of all platforms. 2024-09-17T06:47:45.945 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-17T06:47:45.945 INFO:teuthology.task.kernel:Updating grub on smithi135 to boot 5.14.0-508.el9.x86_64 2024-09-17T06:47:45.945 DEBUG:teuthology.orchestra.run.smithi135:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-17T06:47:46.647 INFO:teuthology.orchestra.run.smithi135.stderr:Generating grub configuration file ... 2024-09-17T06:47:47.934 INFO:teuthology.orchestra.run.smithi135.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-17T06:47:47.962 INFO:teuthology.orchestra.run.smithi135.stderr:done 2024-09-17T06:47:47.964 DEBUG:teuthology.orchestra.run.smithi135:> mktemp 2024-09-17T06:47:47.980 INFO:teuthology.orchestra.run.smithi135.stdout:/tmp/tmp.ZFaFDGnAXY 2024-09-17T06:47:47.980 DEBUG:teuthology.orchestra.run.smithi135:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.ZFaFDGnAXY 2024-09-17T06:47:48.048 DEBUG:teuthology.orchestra.run.smithi135:> sudo chmod 0666 /tmp/tmp.ZFaFDGnAXY 2024-09-17T06:47:48.184 DEBUG:teuthology.orchestra.remote:smithi135:/tmp/tmp.ZFaFDGnAXY is 6KB 2024-09-17T06:47:48.205 DEBUG:teuthology.orchestra.run.smithi135:> rm -fr /tmp/tmp.ZFaFDGnAXY 2024-09-17T06:47:48.220 DEBUG:teuthology.orchestra.run.smithi135:> sudo /bin/ls /boot/loader/entries || true 2024-09-17T06:47:48.283 INFO:teuthology.orchestra.run.smithi135.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-17T06:47:48.283 INFO:teuthology.orchestra.run.smithi135.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-17T06:47:48.283 INFO:teuthology.orchestra.run.smithi135.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-17T06:47:48.283 INFO:teuthology.orchestra.run.smithi135.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-17T06:47:48.284 DEBUG:teuthology.orchestra.run.smithi135:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-17T06:47:48.458 DEBUG:teuthology.orchestra.run.smithi135:> sudo shutdown -r now 2024-09-17T06:47:59.965 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T06:48:00.012 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-17T06:48:00.395 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-17T06:48:00.395 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-17T06:48:00.395 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T06:48:00.395 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T06:48:01.220 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T06:48:01.220 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:48:01.220 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-17T06:48:01.220 INFO:teuthology.orchestra.run.smithi100.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-17T06:48:01.220 INFO:teuthology.orchestra.run.smithi100.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-17T06:48:01.221 INFO:teuthology.orchestra.run.smithi100.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-17T06:48:01.221 INFO:teuthology.orchestra.run.smithi100.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-17T06:48:01.221 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:48:01.221 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-17T06:48:01.553 DEBUG:teuthology.orchestra.run.smithi100:> rpm -q kernel | sort -rV | head -n 1 2024-09-17T06:48:01.638 INFO:teuthology.orchestra.run.smithi100.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-17T06:48:01.638 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-17T06:48:01.638 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-17T06:48:01.638 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-17T06:48:01.639 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-17T06:48:01.639 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-17T06:48:01.639 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi100.front.sepia.ceph.com, path=None, version=distro) 2024-09-17T06:48:01.639 DEBUG:teuthology.orchestra.run.smithi100:> rpm -q kernel | sort -rV | head -n 1 2024-09-17T06:48:01.697 INFO:teuthology.orchestra.run.smithi100.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-17T06:48:01.698 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-17T06:48:01.698 DEBUG:teuthology.orchestra.run.smithi100:> sudo rpm -qi grub2-tools 2024-09-17T06:48:01.729 INFO:teuthology.orchestra.run.smithi100.stdout:Name : grub2-tools 2024-09-17T06:48:01.730 INFO:teuthology.orchestra.run.smithi100.stdout:Epoch : 1 2024-09-17T06:48:01.730 INFO:teuthology.orchestra.run.smithi100.stdout:Version : 2.06 2024-09-17T06:48:01.730 INFO:teuthology.orchestra.run.smithi100.stdout:Release : 82.el9 2024-09-17T06:48:01.730 INFO:teuthology.orchestra.run.smithi100.stdout:Architecture: x86_64 2024-09-17T06:48:01.730 INFO:teuthology.orchestra.run.smithi100.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-17T06:48:01.730 INFO:teuthology.orchestra.run.smithi100.stdout:Group : Unspecified 2024-09-17T06:48:01.730 INFO:teuthology.orchestra.run.smithi100.stdout:Size : 8274670 2024-09-17T06:48:01.730 INFO:teuthology.orchestra.run.smithi100.stdout:License : GPLv3+ 2024-09-17T06:48:01.730 INFO:teuthology.orchestra.run.smithi100.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-17T06:48:01.730 INFO:teuthology.orchestra.run.smithi100.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-17T06:48:01.730 INFO:teuthology.orchestra.run.smithi100.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-17T06:48:01.731 INFO:teuthology.orchestra.run.smithi100.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-17T06:48:01.731 INFO:teuthology.orchestra.run.smithi100.stdout:Packager : builder@centos.org 2024-09-17T06:48:01.731 INFO:teuthology.orchestra.run.smithi100.stdout:Vendor : CentOS 2024-09-17T06:48:01.731 INFO:teuthology.orchestra.run.smithi100.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-17T06:48:01.731 INFO:teuthology.orchestra.run.smithi100.stdout:Summary : Support tools for GRUB. 2024-09-17T06:48:01.731 INFO:teuthology.orchestra.run.smithi100.stdout:Description : 2024-09-17T06:48:01.731 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:48:01.731 INFO:teuthology.orchestra.run.smithi100.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-17T06:48:01.731 INFO:teuthology.orchestra.run.smithi100.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-17T06:48:01.731 INFO:teuthology.orchestra.run.smithi100.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-17T06:48:01.731 INFO:teuthology.orchestra.run.smithi100.stdout:hardware devices. 2024-09-17T06:48:01.731 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:48:01.731 INFO:teuthology.orchestra.run.smithi100.stdout:This subpackage provides tools for support of all platforms. 2024-09-17T06:48:01.732 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-17T06:48:01.732 INFO:teuthology.task.kernel:Updating grub on smithi100 to boot 5.14.0-508.el9.x86_64 2024-09-17T06:48:01.732 DEBUG:teuthology.orchestra.run.smithi100:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-17T06:48:02.410 INFO:teuthology.orchestra.run.smithi100.stderr:Generating grub configuration file ... 2024-09-17T06:48:03.653 INFO:teuthology.orchestra.run.smithi100.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-17T06:48:03.676 INFO:teuthology.orchestra.run.smithi100.stderr:done 2024-09-17T06:48:03.678 DEBUG:teuthology.orchestra.run.smithi100:> mktemp 2024-09-17T06:48:03.693 INFO:teuthology.orchestra.run.smithi100.stdout:/tmp/tmp.kQA2HJ0lc4 2024-09-17T06:48:03.693 DEBUG:teuthology.orchestra.run.smithi100:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.kQA2HJ0lc4 2024-09-17T06:48:03.760 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod 0666 /tmp/tmp.kQA2HJ0lc4 2024-09-17T06:48:03.910 DEBUG:teuthology.orchestra.remote:smithi100:/tmp/tmp.kQA2HJ0lc4 is 6KB 2024-09-17T06:48:03.958 DEBUG:teuthology.orchestra.run.smithi100:> rm -fr /tmp/tmp.kQA2HJ0lc4 2024-09-17T06:48:03.973 DEBUG:teuthology.orchestra.run.smithi100:> sudo /bin/ls /boot/loader/entries || true 2024-09-17T06:48:04.036 INFO:teuthology.orchestra.run.smithi100.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-17T06:48:04.036 INFO:teuthology.orchestra.run.smithi100.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-17T06:48:04.036 INFO:teuthology.orchestra.run.smithi100.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-17T06:48:04.036 INFO:teuthology.orchestra.run.smithi100.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-17T06:48:04.037 DEBUG:teuthology.orchestra.run.smithi100:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-17T06:48:04.206 DEBUG:teuthology.orchestra.run.smithi100:> sudo shutdown -r now 2024-09-17T06:48:18.463 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-17T06:48:18.463 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi135.front.sepia.ceph.com' 2024-09-17T06:48:18.464 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi135.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T06:48:34.217 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-17T06:48:34.218 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi100.front.sepia.ceph.com' 2024-09-17T06:48:34.218 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi100.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T06:48:36.925 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.135 2024-09-17T06:48:45.934 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi135.front.sepia.ceph.com' 2024-09-17T06:48:45.935 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi135.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T06:48:48.993 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.135 2024-09-17T06:49:01.005 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi135.front.sepia.ceph.com' 2024-09-17T06:49:01.006 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi135.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T06:49:04.061 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.135 2024-09-17T06:49:19.074 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi135.front.sepia.ceph.com' 2024-09-17T06:49:19.075 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi135.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T06:49:22.145 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.135 2024-09-17T06:49:34.229 DEBUG:teuthology.orchestra.remote:timed out 2024-09-17T06:49:40.150 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi135.front.sepia.ceph.com' 2024-09-17T06:49:40.150 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi135.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T06:49:40.605 DEBUG:teuthology.orchestra.run.smithi135:> true 2024-09-17T06:49:40.938 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi135.front.sepia.ceph.com' 2024-09-17T06:49:40.938 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-508.el9.x86_64"... 2024-09-17T06:49:40.938 DEBUG:teuthology.orchestra.run.smithi135:> uname -r 2024-09-17T06:49:40.953 INFO:teuthology.orchestra.run.smithi135.stdout:5.14.0-508.el9.x86_64 2024-09-17T06:49:40.953 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-17T06:49:40.954 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-17T06:49:40.954 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-17T06:49:41.955 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-17T06:49:41.955 DEBUG:teuthology.orchestra.run.smithi135:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-17T06:49:42.039 INFO:teuthology.orchestra.run.smithi135.stdout:ttyS1 2024-09-17T06:49:42.059 DEBUG:teuthology.parallel:result is None 2024-09-17T06:49:43.230 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi100.front.sepia.ceph.com' 2024-09-17T06:49:43.231 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi100.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T06:49:43.620 DEBUG:teuthology.orchestra.run.smithi100:> true 2024-09-17T06:49:43.996 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi100.front.sepia.ceph.com' 2024-09-17T06:49:43.996 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-508.el9.x86_64"... 2024-09-17T06:49:43.997 DEBUG:teuthology.orchestra.run.smithi100:> uname -r 2024-09-17T06:49:44.011 INFO:teuthology.orchestra.run.smithi100.stdout:5.14.0-508.el9.x86_64 2024-09-17T06:49:44.011 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-17T06:49:44.012 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-17T06:49:44.012 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-17T06:49:45.012 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-17T06:49:45.013 DEBUG:teuthology.orchestra.run.smithi100:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-17T06:49:45.084 INFO:teuthology.orchestra.run.smithi100.stdout:ttyS1 2024-09-17T06:49:45.113 DEBUG:teuthology.parallel:result is None 2024-09-17T06:49:45.114 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-17T06:49:45.120 INFO:teuthology.task.internal:Creating test directory... 2024-09-17T06:49:45.120 DEBUG:teuthology.orchestra.run.smithi100:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-17T06:49:45.122 DEBUG:teuthology.orchestra.run.smithi135:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-17T06:49:45.141 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-17T06:49:45.147 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-17T06:49:45.174 INFO:teuthology.task.internal:Creating archive directory... 2024-09-17T06:49:45.175 DEBUG:teuthology.orchestra.run.smithi100:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-17T06:49:45.178 DEBUG:teuthology.orchestra.run.smithi135:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-17T06:49:45.286 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-17T06:49:45.292 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-17T06:49:45.292 DEBUG:teuthology.orchestra.run.smithi100:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-17T06:49:45.294 DEBUG:teuthology.orchestra.run.smithi135:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-17T06:49:45.321 INFO:teuthology.orchestra.run.smithi100.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T06:49:45.328 INFO:teuthology.orchestra.run.smithi135.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T06:49:45.331 INFO:teuthology.orchestra.run.smithi100.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T06:49:45.339 INFO:teuthology.orchestra.run.smithi135.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T06:49:45.341 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-17T06:49:45.348 INFO:teuthology.task.internal:Configuring sudo... 2024-09-17T06:49:45.348 DEBUG:teuthology.orchestra.run.smithi100:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-17T06:49:45.375 DEBUG:teuthology.orchestra.run.smithi135:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-17T06:49:45.408 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-17T06:49:45.417 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-17T06:49:45.417 DEBUG:teuthology.orchestra.run.smithi100:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-17T06:49:45.439 DEBUG:teuthology.orchestra.run.smithi135:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-17T06:49:45.467 DEBUG:teuthology.orchestra.run.smithi100:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-17T06:49:45.537 DEBUG:teuthology.orchestra.run.smithi100:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-17T06:49:45.584 DEBUG:teuthology.orchestra.run.smithi100:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-17T06:49:45.667 DEBUG:teuthology.orchestra.run.smithi100:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-17T06:49:45.705 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-17T06:49:45.706 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-17T06:49:45.770 DEBUG:teuthology.orchestra.run.smithi135:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-17T06:49:45.814 DEBUG:teuthology.orchestra.run.smithi135:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-17T06:49:45.866 DEBUG:teuthology.orchestra.run.smithi135:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-17T06:49:45.949 DEBUG:teuthology.orchestra.run.smithi135:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-17T06:49:45.990 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-09-17T06:49:45.990 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-17T06:49:46.056 DEBUG:teuthology.orchestra.run.smithi100:> sudo service rsyslog restart 2024-09-17T06:49:46.058 DEBUG:teuthology.orchestra.run.smithi135:> sudo service rsyslog restart 2024-09-17T06:49:46.121 INFO:teuthology.orchestra.run.smithi100.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-17T06:49:46.150 INFO:teuthology.orchestra.run.smithi135.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-17T06:49:46.661 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-17T06:49:46.668 INFO:teuthology.task.internal:Starting timer... 2024-09-17T06:49:46.668 INFO:teuthology.run_tasks:Running task pcp... 2024-09-17T06:49:46.695 INFO:teuthology.run_tasks:Running task selinux... 2024-09-17T06:49:46.736 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-09-17T06:49:46.736 DEBUG:teuthology.orchestra.run.smithi100:> sudo service auditd rotate 2024-09-17T06:49:46.800 INFO:teuthology.orchestra.run.smithi100.stdout:Rotating logs: 2024-09-17T06:49:46.802 DEBUG:teuthology.orchestra.run.smithi135:> sudo service auditd rotate 2024-09-17T06:49:46.840 INFO:teuthology.orchestra.run.smithi135.stdout:Rotating logs: 2024-09-17T06:49:46.842 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-17T06:49:46.842 DEBUG:teuthology.orchestra.run.smithi100:> /usr/sbin/getenforce 2024-09-17T06:49:46.860 INFO:teuthology.orchestra.run.smithi100.stdout:Permissive 2024-09-17T06:49:46.860 DEBUG:teuthology.orchestra.run.smithi135:> /usr/sbin/getenforce 2024-09-17T06:49:46.936 INFO:teuthology.orchestra.run.smithi135.stdout:Permissive 2024-09-17T06:49:46.936 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi100.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi135.front.sepia.ceph.com': 'permissive'} 2024-09-17T06:49:46.936 DEBUG:teuthology.orchestra.run.smithi100:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-17T06:49:46.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T06:49:46.961 DEBUG:teuthology.orchestra.run.smithi135:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-17T06:49:46.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T06:49:46.986 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-17T06:49:46.986 DEBUG:teuthology.orchestra.run.smithi100:> sudo /usr/sbin/setenforce permissive 2024-09-17T06:49:47.042 DEBUG:teuthology.orchestra.run.smithi135:> sudo /usr/sbin/setenforce permissive 2024-09-17T06:49:47.080 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-17T06:49:47.126 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-17T06:49:47.137 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-17T06:49:47.139 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi100.front.sepia.ceph.com,smithi135.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-17T06:53:32.670 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi100.front.sepia.ceph.com'), Remote(name='ubuntu@smithi135.front.sepia.ceph.com')] 2024-09-17T06:53:32.672 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi100.front.sepia.ceph.com' 2024-09-17T06:53:32.677 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi100.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T06:53:32.765 DEBUG:teuthology.orchestra.run.smithi100:> true 2024-09-17T06:53:32.844 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi100.front.sepia.ceph.com' 2024-09-17T06:53:32.845 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi135.front.sepia.ceph.com' 2024-09-17T06:53:32.845 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi135.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T06:53:32.920 DEBUG:teuthology.orchestra.run.smithi135:> true 2024-09-17T06:53:33.000 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi135.front.sepia.ceph.com' 2024-09-17T06:53:33.001 INFO:teuthology.run_tasks:Running task clock... 2024-09-17T06:53:33.011 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-17T06:53:33.011 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-17T06:53:33.012 DEBUG:teuthology.orchestra.run.smithi100:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-17T06:53:33.014 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-17T06:53:33.014 DEBUG:teuthology.orchestra.run.smithi135:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-17T06:53:33.041 INFO:teuthology.orchestra.run.smithi100.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-17T06:53:33.056 INFO:teuthology.orchestra.run.smithi100.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-17T06:53:33.070 INFO:teuthology.orchestra.run.smithi135.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-17T06:53:33.080 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: ntpd: command not found 2024-09-17T06:53:33.084 INFO:teuthology.orchestra.run.smithi135.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-17T06:53:33.092 INFO:teuthology.orchestra.run.smithi100.stdout:506 Cannot talk to daemon 2024-09-17T06:53:33.106 INFO:teuthology.orchestra.run.smithi100.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-17T06:53:33.109 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: ntpd: command not found 2024-09-17T06:53:33.120 INFO:teuthology.orchestra.run.smithi100.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-17T06:53:33.120 INFO:teuthology.orchestra.run.smithi135.stdout:506 Cannot talk to daemon 2024-09-17T06:53:33.134 INFO:teuthology.orchestra.run.smithi135.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-17T06:53:33.148 INFO:teuthology.orchestra.run.smithi135.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-17T06:53:33.178 INFO:teuthology.orchestra.run.smithi100.stderr:bash: line 1: ntpq: command not found 2024-09-17T06:53:33.181 INFO:teuthology.orchestra.run.smithi100.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-17T06:53:33.181 INFO:teuthology.orchestra.run.smithi100.stdout:=============================================================================== 2024-09-17T06:53:33.181 INFO:teuthology.orchestra.run.smithi100.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T06:53:33.181 INFO:teuthology.orchestra.run.smithi100.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T06:53:33.181 INFO:teuthology.orchestra.run.smithi100.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T06:53:33.182 INFO:teuthology.orchestra.run.smithi100.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T06:53:33.200 INFO:teuthology.orchestra.run.smithi135.stderr:bash: line 1: ntpq: command not found 2024-09-17T06:53:33.203 INFO:teuthology.orchestra.run.smithi135.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-17T06:53:33.203 INFO:teuthology.orchestra.run.smithi135.stdout:=============================================================================== 2024-09-17T06:53:33.203 INFO:teuthology.orchestra.run.smithi135.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T06:53:33.203 INFO:teuthology.orchestra.run.smithi135.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T06:53:33.203 INFO:teuthology.orchestra.run.smithi135.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T06:53:33.203 INFO:teuthology.orchestra.run.smithi135.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T06:53:33.204 INFO:teuthology.run_tasks:Running task pexec... 2024-09-17T06:53:33.213 INFO:teuthology.task.pexec:Executing custom commands... 2024-09-17T06:53:33.214 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi100.front.sepia.ceph.com 2024-09-17T06:53:33.214 DEBUG:teuthology.orchestra.run.smithi100:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-09-17T06:53:33.214 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi135.front.sepia.ceph.com 2024-09-17T06:53:33.215 DEBUG:teuthology.orchestra.run.smithi135:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-09-17T06:53:33.572 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-17T06:53:33.585 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T06:53:33.585 INFO:teuthology.orchestra.run.smithi100.stdout: Package Architecture Version Repository Size 2024-09-17T06:53:33.585 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T06:53:33.585 INFO:teuthology.orchestra.run.smithi100.stdout:Removing: 2024-09-17T06:53:33.586 INFO:teuthology.orchestra.run.smithi100.stdout: nvme-cli x86_64 2.9.1-6.el9 @baseos 5.3 M 2024-09-17T06:53:33.586 INFO:teuthology.orchestra.run.smithi100.stdout:Removing unused dependencies: 2024-09-17T06:53:33.586 INFO:teuthology.orchestra.run.smithi100.stdout: libnvme x86_64 1.9-3.el9 @baseos 273 k 2024-09-17T06:53:33.586 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:53:33.586 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-17T06:53:33.586 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T06:53:33.586 INFO:teuthology.orchestra.run.smithi100.stdout:Remove 2 Packages 2024-09-17T06:53:33.586 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:53:33.586 INFO:teuthology.orchestra.run.smithi100.stdout:Freed space: 5.5 M 2024-09-17T06:53:33.586 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-17T06:53:33.590 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-17T06:53:33.590 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-17T06:53:33.598 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-09-17T06:53:33.610 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-09-17T06:53:33.610 INFO:teuthology.orchestra.run.smithi135.stdout: Package Architecture Version Repository Size 2024-09-17T06:53:33.610 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-09-17T06:53:33.610 INFO:teuthology.orchestra.run.smithi135.stdout:Removing: 2024-09-17T06:53:33.610 INFO:teuthology.orchestra.run.smithi135.stdout: nvme-cli x86_64 2.9.1-6.el9 @baseos 5.3 M 2024-09-17T06:53:33.610 INFO:teuthology.orchestra.run.smithi135.stdout:Removing unused dependencies: 2024-09-17T06:53:33.610 INFO:teuthology.orchestra.run.smithi135.stdout: libnvme x86_64 1.9-3.el9 @baseos 273 k 2024-09-17T06:53:33.610 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:53:33.611 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-09-17T06:53:33.611 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-09-17T06:53:33.611 INFO:teuthology.orchestra.run.smithi135.stdout:Remove 2 Packages 2024-09-17T06:53:33.611 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:53:33.611 INFO:teuthology.orchestra.run.smithi135.stdout:Freed space: 5.5 M 2024-09-17T06:53:33.611 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-09-17T06:53:33.614 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-09-17T06:53:33.614 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-09-17T06:53:33.657 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-17T06:53:33.657 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-17T06:53:33.674 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-09-17T06:53:33.675 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-09-17T06:53:33.788 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-17T06:53:33.812 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-09-17T06:53:33.854 INFO:teuthology.orchestra.run.smithi100.stdout: Erasing : nvme-cli-2.9.1-6.el9.x86_64 1/2 2024-09-17T06:53:33.890 INFO:teuthology.orchestra.run.smithi135.stdout: Erasing : nvme-cli-2.9.1-6.el9.x86_64 1/2 2024-09-17T06:53:33.912 INFO:teuthology.orchestra.run.smithi100.stdout: Erasing : libnvme-1.9-3.el9.x86_64 2/2 2024-09-17T06:53:33.950 INFO:teuthology.orchestra.run.smithi135.stdout: Erasing : libnvme-1.9-3.el9.x86_64 2/2 2024-09-17T06:53:34.370 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: libnvme-1.9-3.el9.x86_64 2/2 2024-09-17T06:53:34.370 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/2 2024-09-17T06:53:34.451 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: libnvme-1.9-3.el9.x86_64 2/2 2024-09-17T06:53:34.452 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/2 2024-09-17T06:53:34.622 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/2 2024-09-17T06:53:34.622 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:53:34.622 INFO:teuthology.orchestra.run.smithi100.stdout:Removed: 2024-09-17T06:53:34.622 INFO:teuthology.orchestra.run.smithi100.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-17T06:53:34.622 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:53:34.622 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-17T06:53:34.700 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/2 2024-09-17T06:53:34.700 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:53:34.700 INFO:teuthology.orchestra.run.smithi135.stdout:Removed: 2024-09-17T06:53:34.700 INFO:teuthology.orchestra.run.smithi135.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-17T06:53:34.700 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:53:34.700 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-09-17T06:53:35.455 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:00:49 ago on Tue 17 Sep 2024 06:52:46 AM UTC. 2024-09-17T06:53:35.563 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:00:49 ago on Tue 17 Sep 2024 06:52:46 AM UTC. 2024-09-17T06:53:35.602 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-17T06:53:35.602 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T06:53:35.603 INFO:teuthology.orchestra.run.smithi100.stdout: Package Architecture Version Repository Size 2024-09-17T06:53:35.603 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T06:53:35.603 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-17T06:53:35.603 INFO:teuthology.orchestra.run.smithi100.stdout: nvme-cli x86_64 2.9.1-6.el9 baseos 906 k 2024-09-17T06:53:35.603 INFO:teuthology.orchestra.run.smithi100.stdout: nvmetcli noarch 0.7-3.el9 baseos 44 k 2024-09-17T06:53:35.603 INFO:teuthology.orchestra.run.smithi100.stdout:Installing dependencies: 2024-09-17T06:53:35.603 INFO:teuthology.orchestra.run.smithi100.stdout: libnvme x86_64 1.9-3.el9 baseos 106 k 2024-09-17T06:53:35.603 INFO:teuthology.orchestra.run.smithi100.stdout: python3-configshell noarch 1:1.1.30-1.el9 baseos 72 k 2024-09-17T06:53:35.603 INFO:teuthology.orchestra.run.smithi100.stdout: python3-kmod x86_64 0.9-32.el9 baseos 84 k 2024-09-17T06:53:35.603 INFO:teuthology.orchestra.run.smithi100.stdout: python3-urwid x86_64 2.1.2-4.el9 baseos 837 k 2024-09-17T06:53:35.603 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:53:35.603 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-17T06:53:35.603 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T06:53:35.604 INFO:teuthology.orchestra.run.smithi100.stdout:Install 6 Packages 2024-09-17T06:53:35.604 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:53:35.638 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 2.0 M 2024-09-17T06:53:35.638 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 9.2 M 2024-09-17T06:53:35.638 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-17T06:53:35.698 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-09-17T06:53:35.698 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-09-17T06:53:35.698 INFO:teuthology.orchestra.run.smithi135.stdout: Package Architecture Version Repository Size 2024-09-17T06:53:35.698 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-09-17T06:53:35.699 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-09-17T06:53:35.699 INFO:teuthology.orchestra.run.smithi135.stdout: nvme-cli x86_64 2.9.1-6.el9 baseos 906 k 2024-09-17T06:53:35.699 INFO:teuthology.orchestra.run.smithi135.stdout: nvmetcli noarch 0.7-3.el9 baseos 44 k 2024-09-17T06:53:35.699 INFO:teuthology.orchestra.run.smithi135.stdout:Installing dependencies: 2024-09-17T06:53:35.699 INFO:teuthology.orchestra.run.smithi135.stdout: libnvme x86_64 1.9-3.el9 baseos 106 k 2024-09-17T06:53:35.699 INFO:teuthology.orchestra.run.smithi135.stdout: python3-configshell noarch 1:1.1.30-1.el9 baseos 72 k 2024-09-17T06:53:35.699 INFO:teuthology.orchestra.run.smithi135.stdout: python3-kmod x86_64 0.9-32.el9 baseos 84 k 2024-09-17T06:53:35.699 INFO:teuthology.orchestra.run.smithi135.stdout: python3-urwid x86_64 2.1.2-4.el9 baseos 837 k 2024-09-17T06:53:35.699 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:53:35.699 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-09-17T06:53:35.699 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-09-17T06:53:35.699 INFO:teuthology.orchestra.run.smithi135.stdout:Install 6 Packages 2024-09-17T06:53:35.699 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:53:35.707 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 2.0 M 2024-09-17T06:53:35.708 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 9.2 M 2024-09-17T06:53:35.708 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-09-17T06:53:35.784 INFO:teuthology.orchestra.run.smithi100.stdout:(1/6): nvmetcli-0.7-3.el9.noarch.rpm 459 kB/s | 44 kB 00:00 2024-09-17T06:53:35.810 INFO:teuthology.orchestra.run.smithi100.stdout:(2/6): libnvme-1.9-3.el9.x86_64.rpm 868 kB/s | 106 kB 00:00 2024-09-17T06:53:35.843 INFO:teuthology.orchestra.run.smithi100.stdout:(3/6): python3-configshell-1.1.30-1.el9.noarch. 1.2 MB/s | 72 kB 00:00 2024-09-17T06:53:35.845 INFO:teuthology.orchestra.run.smithi135.stdout:(1/6): nvmetcli-0.7-3.el9.noarch.rpm 492 kB/s | 44 kB 00:00 2024-09-17T06:53:35.868 INFO:teuthology.orchestra.run.smithi100.stdout:(4/6): python3-kmod-0.9-32.el9.x86_64.rpm 1.4 MB/s | 84 kB 00:00 2024-09-17T06:53:35.871 INFO:teuthology.orchestra.run.smithi135.stdout:(2/6): libnvme-1.9-3.el9.x86_64.rpm 916 kB/s | 106 kB 00:00 2024-09-17T06:53:35.896 INFO:teuthology.orchestra.run.smithi135.stdout:(3/6): python3-configshell-1.1.30-1.el9.noarch. 1.4 MB/s | 72 kB 00:00 2024-09-17T06:53:35.902 INFO:teuthology.orchestra.run.smithi100.stdout:(5/6): nvme-cli-2.9.1-6.el9.x86_64.rpm 4.1 MB/s | 906 kB 00:00 2024-09-17T06:53:35.930 INFO:teuthology.orchestra.run.smithi135.stdout:(4/6): python3-kmod-0.9-32.el9.x86_64.rpm 1.4 MB/s | 84 kB 00:00 2024-09-17T06:53:35.972 INFO:teuthology.orchestra.run.smithi135.stdout:(5/6): nvme-cli-2.9.1-6.el9.x86_64.rpm 4.1 MB/s | 906 kB 00:00 2024-09-17T06:53:36.011 INFO:teuthology.orchestra.run.smithi100.stdout:(6/6): python3-urwid-2.1.2-4.el9.x86_64.rpm 4.9 MB/s | 837 kB 00:00 2024-09-17T06:53:36.011 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-17T06:53:36.011 INFO:teuthology.orchestra.run.smithi100.stdout:Total 5.4 MB/s | 2.0 MB 00:00 2024-09-17T06:53:36.047 INFO:teuthology.orchestra.run.smithi135.stdout:(6/6): python3-urwid-2.1.2-4.el9.x86_64.rpm 5.4 MB/s | 837 kB 00:00 2024-09-17T06:53:36.047 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-09-17T06:53:36.047 INFO:teuthology.orchestra.run.smithi135.stdout:Total 5.9 MB/s | 2.0 MB 00:00 2024-09-17T06:53:36.084 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-17T06:53:36.097 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-17T06:53:36.097 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-17T06:53:36.121 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-09-17T06:53:36.135 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-09-17T06:53:36.135 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-09-17T06:53:36.349 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-17T06:53:36.350 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-17T06:53:36.426 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-09-17T06:53:36.427 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-09-17T06:53:36.786 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-17T06:53:36.851 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-09-17T06:53:36.888 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-urwid-2.1.2-4.el9.x86_64 1/6 2024-09-17T06:53:36.923 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-configshell-1:1.1.30-1.el9.noarch 2/6 2024-09-17T06:53:36.950 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-urwid-2.1.2-4.el9.x86_64 1/6 2024-09-17T06:53:36.951 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-kmod-0.9-32.el9.x86_64 3/6 2024-09-17T06:53:36.984 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-configshell-1:1.1.30-1.el9.noarch 2/6 2024-09-17T06:53:37.012 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-kmod-0.9-32.el9.x86_64 3/6 2024-09-17T06:53:37.156 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libnvme-1.9-3.el9.x86_64 4/6 2024-09-17T06:53:37.191 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : nvme-cli-2.9.1-6.el9.x86_64 5/6 2024-09-17T06:53:37.216 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : libnvme-1.9-3.el9.x86_64 4/6 2024-09-17T06:53:37.252 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : nvme-cli-2.9.1-6.el9.x86_64 5/6 2024-09-17T06:53:37.629 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: nvme-cli-2.9.1-6.el9.x86_64 5/6 2024-09-17T06:53:37.652 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : nvmetcli-0.7-3.el9.noarch 6/6 2024-09-17T06:53:37.673 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: nvme-cli-2.9.1-6.el9.x86_64 5/6 2024-09-17T06:53:37.696 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : nvmetcli-0.7-3.el9.noarch 6/6 2024-09-17T06:53:38.613 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: nvmetcli-0.7-3.el9.noarch 6/6 2024-09-17T06:53:38.614 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/6 2024-09-17T06:53:38.614 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/6 2024-09-17T06:53:38.614 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : nvmetcli-0.7-3.el9.noarch 3/6 2024-09-17T06:53:38.614 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-configshell-1:1.1.30-1.el9.noarch 4/6 2024-09-17T06:53:38.614 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-kmod-0.9-32.el9.x86_64 5/6 2024-09-17T06:53:38.684 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: nvmetcli-0.7-3.el9.noarch 6/6 2024-09-17T06:53:38.684 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/6 2024-09-17T06:53:38.684 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/6 2024-09-17T06:53:38.684 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : nvmetcli-0.7-3.el9.noarch 3/6 2024-09-17T06:53:38.684 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-configshell-1:1.1.30-1.el9.noarch 4/6 2024-09-17T06:53:38.684 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-kmod-0.9-32.el9.x86_64 5/6 2024-09-17T06:53:38.953 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-urwid-2.1.2-4.el9.x86_64 6/6 2024-09-17T06:53:38.953 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:53:38.953 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-17T06:53:38.953 INFO:teuthology.orchestra.run.smithi100.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-17T06:53:38.953 INFO:teuthology.orchestra.run.smithi100.stdout: nvmetcli-0.7-3.el9.noarch python3-configshell-1:1.1.30-1.el9.noarch 2024-09-17T06:53:38.953 INFO:teuthology.orchestra.run.smithi100.stdout: python3-kmod-0.9-32.el9.x86_64 python3-urwid-2.1.2-4.el9.x86_64 2024-09-17T06:53:38.953 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:53:38.954 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-17T06:53:39.042 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-urwid-2.1.2-4.el9.x86_64 6/6 2024-09-17T06:53:39.042 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:53:39.042 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-09-17T06:53:39.042 INFO:teuthology.orchestra.run.smithi135.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-17T06:53:39.042 INFO:teuthology.orchestra.run.smithi135.stdout: nvmetcli-0.7-3.el9.noarch python3-configshell-1:1.1.30-1.el9.noarch 2024-09-17T06:53:39.042 INFO:teuthology.orchestra.run.smithi135.stdout: python3-kmod-0.9-32.el9.x86_64 python3-urwid-2.1.2-4.el9.x86_64 2024-09-17T06:53:39.042 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:53:39.042 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-09-17T06:53:39.265 DEBUG:teuthology.parallel:result is None 2024-09-17T06:53:39.359 DEBUG:teuthology.parallel:result is None 2024-09-17T06:53:39.359 INFO:teuthology.run_tasks:Running task install... 2024-09-17T06:53:39.369 DEBUG:teuthology.task.install:project ceph 2024-09-17T06:53:39.369 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '19e2a2beb6f41a3ab62650e38614eb29b59dcbb9'}} 2024-09-17T06:53:39.369 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '19e2a2beb6f41a3ab62650e38614eb29b59dcbb9'} 2024-09-17T06:53:39.369 INFO:teuthology.task.install:Using flavor: default 2024-09-17T06:53:39.376 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-17T06:53:39.377 INFO:teuthology.task.install:extra packages: [] 2024-09-17T06:53:39.377 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '19e2a2beb6f41a3ab62650e38614eb29b59dcbb9', 'tag': None, 'wait_for_package': False} 2024-09-17T06:53:39.377 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-17T06:53:39.380 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '19e2a2beb6f41a3ab62650e38614eb29b59dcbb9', 'tag': None, 'wait_for_package': False} 2024-09-17T06:53:39.380 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-17T06:53:39.525 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/centos/9/flavors/default/ 2024-09-17T06:53:39.525 INFO:teuthology.task.install.rpm:Package version is 18.2.4-855.g19e2a2be 2024-09-17T06:53:39.542 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/centos/9/flavors/default/ 2024-09-17T06:53:39.542 INFO:teuthology.task.install.rpm:Package version is 18.2.4-855.g19e2a2be 2024-09-17T06:53:39.639 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-17T06:53:39.640 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-17T06:53:39.640 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-17T06:53:39.652 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-17T06:53:39.653 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-09-17T06:53:39.653 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-17T06:53:39.670 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-17T06:53:39.671 DEBUG:teuthology.orchestra.run.smithi100:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-17T06:53:39.682 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-17T06:53:39.683 DEBUG:teuthology.orchestra.run.smithi135:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-17T06:53:39.750 DEBUG:teuthology.orchestra.run.smithi135:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-17T06:53:39.754 DEBUG:teuthology.orchestra.run.smithi100:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-17T06:53:39.829 DEBUG:teuthology.orchestra.run.smithi100:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-17T06:53:39.837 DEBUG:teuthology.orchestra.run.smithi135:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-17T06:53:39.868 INFO:teuthology.orchestra.run.smithi135.stdout:check_obsoletes = 1 2024-09-17T06:53:39.870 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum clean all 2024-09-17T06:53:39.893 INFO:teuthology.orchestra.run.smithi100.stdout:check_obsoletes = 1 2024-09-17T06:53:39.894 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum clean all 2024-09-17T06:53:40.107 INFO:teuthology.orchestra.run.smithi135.stdout:45 files removed 2024-09-17T06:53:40.115 INFO:teuthology.orchestra.run.smithi100.stdout:45 files removed 2024-09-17T06:53:40.135 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-radosgw 2024-09-17T06:53:40.142 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph-radosgw 2024-09-17T06:53:40.718 INFO:teuthology.orchestra.run.smithi135.stdout:ceph packages for x86_64 261 kB/s | 77 kB 00:00 2024-09-17T06:53:40.733 INFO:teuthology.orchestra.run.smithi100.stdout:ceph packages for x86_64 252 kB/s | 77 kB 00:00 2024-09-17T06:53:41.002 INFO:teuthology.orchestra.run.smithi135.stdout:ceph noarch packages 48 kB/s | 11 kB 00:00 2024-09-17T06:53:41.017 INFO:teuthology.orchestra.run.smithi100.stdout:ceph noarch packages 48 kB/s | 11 kB 00:00 2024-09-17T06:53:41.278 INFO:teuthology.orchestra.run.smithi135.stdout:ceph source packages 7.9 kB/s | 1.9 kB 00:00 2024-09-17T06:53:41.292 INFO:teuthology.orchestra.run.smithi100.stdout:ceph source packages 7.8 kB/s | 1.9 kB 00:00 2024-09-17T06:53:41.919 INFO:teuthology.orchestra.run.smithi135.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.3 MB 00:00 2024-09-17T06:53:44.225 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS Stream 9 - BaseOS 2.8 MB/s | 8.3 MB 00:02 2024-09-17T06:53:47.529 INFO:teuthology.orchestra.run.smithi135.stdout:CentOS Stream 9 - AppStream 4.5 MB/s | 20 MB 00:04 2024-09-17T06:53:48.501 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS Stream 9 - AppStream 6.3 MB/s | 20 MB 00:03 2024-09-17T06:53:52.594 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS Stream 9 - CRB 19 MB/s | 6.5 MB 00:00 2024-09-17T06:53:52.679 INFO:teuthology.orchestra.run.smithi135.stdout:CentOS Stream 9 - CRB 4.8 MB/s | 6.5 MB 00:01 2024-09-17T06:53:54.461 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS Stream 9 - Extras packages 38 kB/s | 19 kB 00:00 2024-09-17T06:53:54.463 INFO:teuthology.orchestra.run.smithi135.stdout:CentOS Stream 9 - Extras packages 59 kB/s | 19 kB 00:00 2024-09-17T06:53:55.524 INFO:teuthology.orchestra.run.smithi100.stdout:Extra Packages for Enterprise Linux 24 MB/s | 23 MB 00:00 2024-09-17T06:53:55.618 INFO:teuthology.orchestra.run.smithi135.stdout:Extra Packages for Enterprise Linux 22 MB/s | 23 MB 00:01 2024-09-17T06:54:00.980 INFO:teuthology.orchestra.run.smithi100.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-17T06:54:01.182 INFO:teuthology.orchestra.run.smithi135.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-09-17T06:54:03.238 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-17T06:54:03.240 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T06:54:03.240 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repository Size 2024-09-17T06:54:03.240 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T06:54:03.240 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-17T06:54:03.241 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-radosgw x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 7.7 M 2024-09-17T06:54:03.241 INFO:teuthology.orchestra.run.smithi100.stdout:Upgrading: 2024-09-17T06:54:03.241 INFO:teuthology.orchestra.run.smithi100.stdout: librados2 x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 3.3 M 2024-09-17T06:54:03.241 INFO:teuthology.orchestra.run.smithi100.stdout: librbd1 x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 3.0 M 2024-09-17T06:54:03.241 INFO:teuthology.orchestra.run.smithi100.stdout:Installing dependencies: 2024-09-17T06:54:03.241 INFO:teuthology.orchestra.run.smithi100.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-17T06:54:03.241 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-base x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 5.1 M 2024-09-17T06:54:03.241 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-common x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 18 M 2024-09-17T06:54:03.241 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-selinux x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 25 k 2024-09-17T06:54:03.241 INFO:teuthology.orchestra.run.smithi100.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-09-17T06:54:03.241 INFO:teuthology.orchestra.run.smithi100.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-17T06:54:03.242 INFO:teuthology.orchestra.run.smithi100.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-17T06:54:03.242 INFO:teuthology.orchestra.run.smithi100.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-17T06:54:03.242 INFO:teuthology.orchestra.run.smithi100.stdout: libcephfs2 x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 712 k 2024-09-17T06:54:03.242 INFO:teuthology.orchestra.run.smithi100.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-17T06:54:03.242 INFO:teuthology.orchestra.run.smithi100.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-17T06:54:03.242 INFO:teuthology.orchestra.run.smithi100.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-17T06:54:03.242 INFO:teuthology.orchestra.run.smithi100.stdout: libradosstriper1 x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 477 k 2024-09-17T06:54:03.242 INFO:teuthology.orchestra.run.smithi100.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-17T06:54:03.242 INFO:teuthology.orchestra.run.smithi100.stdout: librgw2 x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 4.5 M 2024-09-17T06:54:03.242 INFO:teuthology.orchestra.run.smithi100.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-17T06:54:03.242 INFO:teuthology.orchestra.run.smithi100.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-17T06:54:03.242 INFO:teuthology.orchestra.run.smithi100.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-17T06:54:03.242 INFO:teuthology.orchestra.run.smithi100.stdout: python3-ceph-argparse x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 45 k 2024-09-17T06:54:03.243 INFO:teuthology.orchestra.run.smithi100.stdout: python3-ceph-common x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 129 k 2024-09-17T06:54:03.243 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cephfs x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 162 k 2024-09-17T06:54:03.243 INFO:teuthology.orchestra.run.smithi100.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-17T06:54:03.243 INFO:teuthology.orchestra.run.smithi100.stdout: python3-rados x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 321 k 2024-09-17T06:54:03.243 INFO:teuthology.orchestra.run.smithi100.stdout: python3-rbd x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 300 k 2024-09-17T06:54:03.243 INFO:teuthology.orchestra.run.smithi100.stdout: python3-rgw x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 99 k 2024-09-17T06:54:03.243 INFO:teuthology.orchestra.run.smithi100.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-17T06:54:03.243 INFO:teuthology.orchestra.run.smithi100.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-17T06:54:03.243 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:54:03.243 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-17T06:54:03.243 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T06:54:03.244 INFO:teuthology.orchestra.run.smithi100.stdout:Install 28 Packages 2024-09-17T06:54:03.244 INFO:teuthology.orchestra.run.smithi100.stdout:Upgrade 2 Packages 2024-09-17T06:54:03.244 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:54:03.244 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 53 M 2024-09-17T06:54:03.244 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-17T06:54:03.381 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-09-17T06:54:03.383 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-09-17T06:54:03.383 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repository Size 2024-09-17T06:54:03.383 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-09-17T06:54:03.383 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-09-17T06:54:03.383 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-radosgw x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 7.7 M 2024-09-17T06:54:03.383 INFO:teuthology.orchestra.run.smithi135.stdout:Upgrading: 2024-09-17T06:54:03.383 INFO:teuthology.orchestra.run.smithi135.stdout: librados2 x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 3.3 M 2024-09-17T06:54:03.383 INFO:teuthology.orchestra.run.smithi135.stdout: librbd1 x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 3.0 M 2024-09-17T06:54:03.384 INFO:teuthology.orchestra.run.smithi135.stdout:Installing dependencies: 2024-09-17T06:54:03.384 INFO:teuthology.orchestra.run.smithi135.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-17T06:54:03.384 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-base x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 5.1 M 2024-09-17T06:54:03.384 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-common x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 18 M 2024-09-17T06:54:03.384 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-selinux x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 25 k 2024-09-17T06:54:03.384 INFO:teuthology.orchestra.run.smithi135.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-09-17T06:54:03.384 INFO:teuthology.orchestra.run.smithi135.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-17T06:54:03.384 INFO:teuthology.orchestra.run.smithi135.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-17T06:54:03.384 INFO:teuthology.orchestra.run.smithi135.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-17T06:54:03.384 INFO:teuthology.orchestra.run.smithi135.stdout: libcephfs2 x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 712 k 2024-09-17T06:54:03.384 INFO:teuthology.orchestra.run.smithi135.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-17T06:54:03.384 INFO:teuthology.orchestra.run.smithi135.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-17T06:54:03.384 INFO:teuthology.orchestra.run.smithi135.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-17T06:54:03.385 INFO:teuthology.orchestra.run.smithi135.stdout: libradosstriper1 x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 477 k 2024-09-17T06:54:03.385 INFO:teuthology.orchestra.run.smithi135.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-17T06:54:03.385 INFO:teuthology.orchestra.run.smithi135.stdout: librgw2 x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 4.5 M 2024-09-17T06:54:03.385 INFO:teuthology.orchestra.run.smithi135.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-17T06:54:03.385 INFO:teuthology.orchestra.run.smithi135.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-17T06:54:03.385 INFO:teuthology.orchestra.run.smithi135.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-17T06:54:03.385 INFO:teuthology.orchestra.run.smithi135.stdout: python3-ceph-argparse x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 45 k 2024-09-17T06:54:03.385 INFO:teuthology.orchestra.run.smithi135.stdout: python3-ceph-common x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 129 k 2024-09-17T06:54:03.385 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cephfs x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 162 k 2024-09-17T06:54:03.385 INFO:teuthology.orchestra.run.smithi135.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-17T06:54:03.385 INFO:teuthology.orchestra.run.smithi135.stdout: python3-rados x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 321 k 2024-09-17T06:54:03.385 INFO:teuthology.orchestra.run.smithi135.stdout: python3-rbd x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 300 k 2024-09-17T06:54:03.385 INFO:teuthology.orchestra.run.smithi135.stdout: python3-rgw x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 99 k 2024-09-17T06:54:03.386 INFO:teuthology.orchestra.run.smithi135.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-17T06:54:03.386 INFO:teuthology.orchestra.run.smithi135.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-17T06:54:03.386 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:54:03.386 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-09-17T06:54:03.386 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-09-17T06:54:03.386 INFO:teuthology.orchestra.run.smithi135.stdout:Install 28 Packages 2024-09-17T06:54:03.386 INFO:teuthology.orchestra.run.smithi135.stdout:Upgrade 2 Packages 2024-09-17T06:54:03.386 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:54:03.387 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 53 M 2024-09-17T06:54:03.387 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-09-17T06:54:03.872 INFO:teuthology.orchestra.run.smithi100.stdout:(1/30): ceph-base-18.2.4-855.g19e2a2be.el9.x86_ 9.3 MB/s | 5.1 MB 00:00 2024-09-17T06:54:03.998 INFO:teuthology.orchestra.run.smithi100.stdout:(2/30): ceph-radosgw-18.2.4-855.g19e2a2be.el9.x 12 MB/s | 7.7 MB 00:00 2024-09-17T06:54:04.023 INFO:teuthology.orchestra.run.smithi100.stdout:(3/30): ceph-selinux-18.2.4-855.g19e2a2be.el9.x 168 kB/s | 25 kB 00:00 2024-09-17T06:54:04.065 INFO:teuthology.orchestra.run.smithi100.stdout:(4/30): libcephfs2-18.2.4-855.g19e2a2be.el9.x86 10 MB/s | 712 kB 00:00 2024-09-17T06:54:04.090 INFO:teuthology.orchestra.run.smithi100.stdout:(5/30): libradosstriper1-18.2.4-855.g19e2a2be.e 7.0 MB/s | 477 kB 00:00 2024-09-17T06:54:04.149 INFO:teuthology.orchestra.run.smithi100.stdout:(6/30): python3-ceph-argparse-18.2.4-855.g19e2a 770 kB/s | 45 kB 00:00 2024-09-17T06:54:04.366 INFO:teuthology.orchestra.run.smithi100.stdout:(7/30): ceph-common-18.2.4-855.g19e2a2be.el9.x8 18 MB/s | 18 MB 00:01 2024-09-17T06:54:04.391 INFO:teuthology.orchestra.run.smithi100.stdout:(8/30): python3-ceph-common-18.2.4-855.g19e2a2b 534 kB/s | 129 kB 00:00 2024-09-17T06:54:04.458 INFO:teuthology.orchestra.run.smithi100.stdout:(9/30): librgw2-18.2.4-855.g19e2a2be.el9.x86_64 11 MB/s | 4.5 MB 00:00 2024-09-17T06:54:04.483 INFO:teuthology.orchestra.run.smithi100.stdout:(10/30): python3-cephfs-18.2.4-855.g19e2a2be.el 1.4 MB/s | 162 kB 00:00 2024-09-17T06:54:04.508 INFO:teuthology.orchestra.run.smithi100.stdout:(11/30): python3-rados-18.2.4-855.g19e2a2be.el9 2.7 MB/s | 321 kB 00:00 2024-09-17T06:54:04.533 INFO:teuthology.orchestra.run.smithi100.stdout:(12/30): python3-rgw-18.2.4-855.g19e2a2be.el9.x 1.9 MB/s | 99 kB 00:00 2024-09-17T06:54:04.559 INFO:teuthology.orchestra.run.smithi100.stdout:(13/30): python3-rbd-18.2.4-855.g19e2a2be.el9.x 2.9 MB/s | 300 kB 00:00 2024-09-17T06:54:04.626 INFO:teuthology.orchestra.run.smithi100.stdout:(14/30): boost-program-options-1.75.0-8.el9.x86 912 kB/s | 107 kB 00:00 2024-09-17T06:54:04.651 INFO:teuthology.orchestra.run.smithi100.stdout:(15/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 493 kB/s | 45 kB 00:00 2024-09-17T06:54:04.685 INFO:teuthology.orchestra.run.smithi100.stdout:(16/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.0 MB/s | 160 kB 00:00 2024-09-17T06:54:04.726 INFO:teuthology.orchestra.run.smithi100.stdout:(17/30): python3-prettytable-0.7.2-27.el9.noarc 1.0 MB/s | 42 kB 00:00 2024-09-17T06:54:04.752 INFO:teuthology.orchestra.run.smithi100.stdout:(18/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.8 MB/s | 292 kB 00:00 2024-09-17T06:54:04.802 INFO:teuthology.orchestra.run.smithi100.stdout:(19/30): librdkafka-1.6.1-102.el9.x86_64.rpm 3.7 MB/s | 662 kB 00:00 2024-09-17T06:54:04.850 INFO:teuthology.orchestra.run.smithi135.stdout:(1/30): ceph-radosgw-18.2.4-855.g19e2a2be.el9.x 9.2 MB/s | 7.7 MB 00:00 2024-09-17T06:54:04.853 INFO:teuthology.orchestra.run.smithi100.stdout:(20/30): fmt-8.1.1-5.el9.x86_64.rpm 881 kB/s | 111 kB 00:00 2024-09-17T06:54:04.878 INFO:teuthology.orchestra.run.smithi100.stdout:(21/30): libarrow-doc-9.0.0-11.el9.noarch.rpm 1.1 MB/s | 26 kB 00:00 2024-09-17T06:54:04.908 INFO:teuthology.orchestra.run.smithi135.stdout:(2/30): ceph-selinux-18.2.4-855.g19e2a2be.el9.x 433 kB/s | 25 kB 00:00 2024-09-17T06:54:04.911 INFO:teuthology.orchestra.run.smithi100.stdout:(22/30): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.0 MB/s | 308 kB 00:00 2024-09-17T06:54:04.937 INFO:teuthology.orchestra.run.smithi100.stdout:(23/30): liboath-2.6.7-2.el9.x86_64.rpm 830 kB/s | 49 kB 00:00 2024-09-17T06:54:04.962 INFO:teuthology.orchestra.run.smithi100.stdout:(24/30): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-17T06:54:04.975 INFO:teuthology.orchestra.run.smithi135.stdout:(3/30): libcephfs2-18.2.4-855.g19e2a2be.el9.x86 10 MB/s | 712 kB 00:00 2024-09-17T06:54:05.001 INFO:teuthology.orchestra.run.smithi100.stdout:(25/30): re2-20211101-20.el9.x86_64.rpm 5.6 MB/s | 191 kB 00:00 2024-09-17T06:54:05.029 INFO:teuthology.orchestra.run.smithi100.stdout:(26/30): parquet-libs-9.0.0-11.el9.x86_64.rpm 8.9 MB/s | 839 kB 00:00 2024-09-17T06:54:05.033 INFO:teuthology.orchestra.run.smithi135.stdout:(4/30): libradosstriper1-18.2.4-855.g19e2a2be.e 8.0 MB/s | 477 kB 00:00 2024-09-17T06:54:05.096 INFO:teuthology.orchestra.run.smithi100.stdout:(27/30): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-09-17T06:54:05.134 INFO:teuthology.orchestra.run.smithi135.stdout:(5/30): ceph-base-18.2.4-855.g19e2a2be.el9.x86_ 4.5 MB/s | 5.1 MB 00:01 2024-09-17T06:54:05.197 INFO:teuthology.orchestra.run.smithi100.stdout:(28/30): libarrow-9.0.0-11.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-09-17T06:54:05.226 INFO:teuthology.orchestra.run.smithi135.stdout:(6/30): python3-ceph-argparse-18.2.4-855.g19e2a 492 kB/s | 45 kB 00:00 2024-09-17T06:54:05.280 INFO:teuthology.orchestra.run.smithi100.stdout:(29/30): librados2-18.2.4-855.g19e2a2be.el9.x86 13 MB/s | 3.3 MB 00:00 2024-09-17T06:54:05.301 INFO:teuthology.orchestra.run.smithi135.stdout:(7/30): librgw2-18.2.4-855.g19e2a2be.el9.x86_64 17 MB/s | 4.5 MB 00:00 2024-09-17T06:54:05.326 INFO:teuthology.orchestra.run.smithi135.stdout:(8/30): python3-ceph-common-18.2.4-855.g19e2a2b 1.3 MB/s | 129 kB 00:00 2024-09-17T06:54:05.338 INFO:teuthology.orchestra.run.smithi100.stdout:(30/30): librbd1-18.2.4-855.g19e2a2be.el9.x86_6 12 MB/s | 3.0 MB 00:00 2024-09-17T06:54:05.343 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-17T06:54:05.344 INFO:teuthology.orchestra.run.smithi100.stdout:Total 25 MB/s | 53 MB 00:02 2024-09-17T06:54:05.352 INFO:teuthology.orchestra.run.smithi135.stdout:(9/30): python3-cephfs-18.2.4-855.g19e2a2be.el9 3.2 MB/s | 162 kB 00:00 2024-09-17T06:54:05.416 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-17T06:54:05.449 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-17T06:54:05.449 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-17T06:54:05.486 INFO:teuthology.orchestra.run.smithi135.stdout:(10/30): python3-rados-18.2.4-855.g19e2a2be.el9 2.0 MB/s | 321 kB 00:00 2024-09-17T06:54:05.511 INFO:teuthology.orchestra.run.smithi135.stdout:(11/30): python3-rbd-18.2.4-855.g19e2a2be.el9.x 1.8 MB/s | 300 kB 00:00 2024-09-17T06:54:05.628 INFO:teuthology.orchestra.run.smithi135.stdout:(12/30): boost-program-options-1.75.0-8.el9.x86 918 kB/s | 107 kB 00:00 2024-09-17T06:54:05.686 INFO:teuthology.orchestra.run.smithi135.stdout:(13/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.7 MB/s | 160 kB 00:00 2024-09-17T06:54:05.729 INFO:teuthology.orchestra.run.smithi135.stdout:(14/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.1 MB/s | 45 kB 00:00 2024-09-17T06:54:05.795 INFO:teuthology.orchestra.run.smithi135.stdout:(15/30): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 662 kB 00:00 2024-09-17T06:54:05.845 INFO:teuthology.orchestra.run.smithi135.stdout:(16/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.7 MB/s | 292 kB 00:00 2024-09-17T06:54:05.886 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-17T06:54:05.887 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-17T06:54:05.887 INFO:teuthology.orchestra.run.smithi135.stdout:(17/30): python3-prettytable-0.7.2-27.el9.noarc 1.0 MB/s | 42 kB 00:00 2024-09-17T06:54:05.929 INFO:teuthology.orchestra.run.smithi135.stdout:(18/30): fmt-8.1.1-5.el9.x86_64.rpm 2.6 MB/s | 111 kB 00:00 2024-09-17T06:54:05.955 INFO:teuthology.orchestra.run.smithi135.stdout:(19/30): gperftools-libs-2.9.1-3.el9.x86_64.rpm 12 MB/s | 308 kB 00:00 2024-09-17T06:54:06.113 INFO:teuthology.orchestra.run.smithi135.stdout:(20/30): libarrow-9.0.0-11.el9.x86_64.rpm 28 MB/s | 4.4 MB 00:00 2024-09-17T06:54:06.138 INFO:teuthology.orchestra.run.smithi135.stdout:(21/30): libarrow-doc-9.0.0-11.el9.noarch.rpm 1.0 MB/s | 26 kB 00:00 2024-09-17T06:54:06.164 INFO:teuthology.orchestra.run.smithi135.stdout:(22/30): liboath-2.6.7-2.el9.x86_64.rpm 1.9 MB/s | 49 kB 00:00 2024-09-17T06:54:06.189 INFO:teuthology.orchestra.run.smithi135.stdout:(23/30): libunwind-1.6.2-1.el9.x86_64.rpm 2.6 MB/s | 67 kB 00:00 2024-09-17T06:54:06.222 INFO:teuthology.orchestra.run.smithi135.stdout:(24/30): parquet-libs-9.0.0-11.el9.x86_64.rpm 25 MB/s | 839 kB 00:00 2024-09-17T06:54:06.248 INFO:teuthology.orchestra.run.smithi135.stdout:(25/30): re2-20211101-20.el9.x86_64.rpm 7.5 MB/s | 191 kB 00:00 2024-09-17T06:54:06.306 INFO:teuthology.orchestra.run.smithi135.stdout:(26/30): thrift-0.15.0-2.el9.x86_64.rpm 27 MB/s | 1.6 MB 00:00 2024-09-17T06:54:07.260 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-17T06:54:07.345 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-09-17T06:54:07.379 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-09-17T06:54:07.437 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-09-17T06:54:07.681 INFO:teuthology.orchestra.run.smithi135.stdout:(27/30): python3-rgw-18.2.4-855.g19e2a2be.el9.x 45 kB/s | 99 kB 00:02 2024-09-17T06:54:07.752 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-09-17T06:54:07.849 INFO:teuthology.orchestra.run.smithi135.stdout:(28/30): librados2-18.2.4-855.g19e2a2be.el9.x86 2.1 MB/s | 3.3 MB 00:01 2024-09-17T06:54:07.869 INFO:teuthology.orchestra.run.smithi100.stdout: Upgrading : librados2-2:18.2.4-855.g19e2a2be.el9.x86_64 5/32 2024-09-17T06:54:07.962 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: librados2-2:18.2.4-855.g19e2a2be.el9.x86_64 5/32 2024-09-17T06:54:08.032 INFO:teuthology.orchestra.run.smithi135.stdout:(29/30): librbd1-18.2.4-855.g19e2a2be.el9.x86_6 8.6 MB/s | 3.0 MB 00:00 2024-09-17T06:54:08.045 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-rados-2:18.2.4-855.g19e2a2be.el9.x86_64 6/32 2024-09-17T06:54:08.079 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libcephfs2-2:18.2.4-855.g19e2a2be.el9.x86_64 7/32 2024-09-17T06:54:08.142 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: libcephfs2-2:18.2.4-855.g19e2a2be.el9.x86_64 7/32 2024-09-17T06:54:08.179 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-09-17T06:54:08.221 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-ceph-argparse-2:18.2.4-855.g19e2a2be.el9.x 9/32 2024-09-17T06:54:08.224 INFO:teuthology.orchestra.run.smithi135.stdout:(30/30): ceph-common-18.2.4-855.g19e2a2be.el9.x 4.4 MB/s | 18 MB 00:04 2024-09-17T06:54:08.228 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-09-17T06:54:08.229 INFO:teuthology.orchestra.run.smithi135.stdout:Total 11 MB/s | 53 MB 00:04 2024-09-17T06:54:08.294 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-cephfs-2:18.2.4-855.g19e2a2be.el9.x86_64 10/32 2024-09-17T06:54:08.302 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-09-17T06:54:08.323 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libradosstriper1-2:18.2.4-855.g19e2a2be.el9.x86_64 11/32 2024-09-17T06:54:08.337 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-09-17T06:54:08.337 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-09-17T06:54:08.381 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: libradosstriper1-2:18.2.4-855.g19e2a2be.el9.x86_64 11/32 2024-09-17T06:54:08.493 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : re2-1:20211101-20.el9.x86_64 12/32 2024-09-17T06:54:08.531 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 13/32 2024-09-17T06:54:08.571 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-09-17T06:54:08.604 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 15/32 2024-09-17T06:54:08.631 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 16/32 2024-09-17T06:54:08.667 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-09-17T06:54:08.702 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-09-17T06:54:08.780 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-09-17T06:54:08.781 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-09-17T06:54:08.999 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-09-17T06:54:09.038 INFO:teuthology.orchestra.run.smithi100.stdout: Upgrading : librbd1-2:18.2.4-855.g19e2a2be.el9.x86_64 20/32 2024-09-17T06:54:09.135 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: librbd1-2:18.2.4-855.g19e2a2be.el9.x86_64 20/32 2024-09-17T06:54:09.159 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-rbd-2:18.2.4-855.g19e2a2be.el9.x86_64 21/32 2024-09-17T06:54:09.195 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-09-17T06:54:09.609 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 23/32 2024-09-17T06:54:09.640 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : librgw2-2:18.2.4-855.g19e2a2be.el9.x86_64 24/32 2024-09-17T06:54:09.711 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: librgw2-2:18.2.4-855.g19e2a2be.el9.x86_64 24/32 2024-09-17T06:54:09.788 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-rgw-2:18.2.4-855.g19e2a2be.el9.x86_64 25/32 2024-09-17T06:54:09.817 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-ceph-common-2:18.2.4-855.g19e2a2be.el9.x86 26/32 2024-09-17T06:54:10.117 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-09-17T06:54:10.186 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-09-17T06:54:10.229 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-09-17T06:54:10.278 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-09-17T06:54:10.604 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-09-17T06:54:10.728 INFO:teuthology.orchestra.run.smithi135.stdout: Upgrading : librados2-2:18.2.4-855.g19e2a2be.el9.x86_64 5/32 2024-09-17T06:54:10.820 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: librados2-2:18.2.4-855.g19e2a2be.el9.x86_64 5/32 2024-09-17T06:54:10.903 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-rados-2:18.2.4-855.g19e2a2be.el9.x86_64 6/32 2024-09-17T06:54:10.929 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : libcephfs2-2:18.2.4-855.g19e2a2be.el9.x86_64 7/32 2024-09-17T06:54:10.992 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: libcephfs2-2:18.2.4-855.g19e2a2be.el9.x86_64 7/32 2024-09-17T06:54:11.020 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-09-17T06:54:11.061 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-ceph-argparse-2:18.2.4-855.g19e2a2be.el9.x 9/32 2024-09-17T06:54:11.130 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-cephfs-2:18.2.4-855.g19e2a2be.el9.x86_64 10/32 2024-09-17T06:54:11.290 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : libradosstriper1-2:18.2.4-855.g19e2a2be.el9.x86_64 11/32 2024-09-17T06:54:11.356 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: libradosstriper1-2:18.2.4-855.g19e2a2be.el9.x86_64 11/32 2024-09-17T06:54:11.467 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : re2-1:20211101-20.el9.x86_64 12/32 2024-09-17T06:54:11.497 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 13/32 2024-09-17T06:54:11.537 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-09-17T06:54:11.579 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 15/32 2024-09-17T06:54:11.606 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 16/32 2024-09-17T06:54:11.642 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-09-17T06:54:11.675 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-09-17T06:54:11.963 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-common-2:18.2.4-855.g19e2a2be.el9.x86_64 27/32 2024-09-17T06:54:11.974 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-09-17T06:54:11.998 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-common-2:18.2.4-855.g19e2a2be.el9.x86_64 27/32 2024-09-17T06:54:12.012 INFO:teuthology.orchestra.run.smithi135.stdout: Upgrading : librbd1-2:18.2.4-855.g19e2a2be.el9.x86_64 20/32 2024-09-17T06:54:12.103 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: librbd1-2:18.2.4-855.g19e2a2be.el9.x86_64 20/32 2024-09-17T06:54:12.134 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-rbd-2:18.2.4-855.g19e2a2be.el9.x86_64 21/32 2024-09-17T06:54:12.169 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-09-17T06:54:12.535 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-common-2:18.2.4-855.g19e2a2be.el9.x86_64 27/32 2024-09-17T06:54:12.574 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 23/32 2024-09-17T06:54:12.576 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-base-2:18.2.4-855.g19e2a2be.el9.x86_64 28/32 2024-09-17T06:54:12.606 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : librgw2-2:18.2.4-855.g19e2a2be.el9.x86_64 24/32 2024-09-17T06:54:12.669 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-base-2:18.2.4-855.g19e2a2be.el9.x86_64 28/32 2024-09-17T06:54:12.669 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-17T06:54:12.670 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-17T06:54:12.670 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:54:12.678 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: librgw2-2:18.2.4-855.g19e2a2be.el9.x86_64 24/32 2024-09-17T06:54:12.702 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-selinux-2:18.2.4-855.g19e2a2be.el9.x86_64 29/32 2024-09-17T06:54:12.755 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-rgw-2:18.2.4-855.g19e2a2be.el9.x86_64 25/32 2024-09-17T06:54:12.784 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-ceph-common-2:18.2.4-855.g19e2a2be.el9.x86 26/32 2024-09-17T06:54:14.876 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-common-2:18.2.4-855.g19e2a2be.el9.x86_64 27/32 2024-09-17T06:54:14.905 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-common-2:18.2.4-855.g19e2a2be.el9.x86_64 27/32 2024-09-17T06:54:15.469 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-common-2:18.2.4-855.g19e2a2be.el9.x86_64 27/32 2024-09-17T06:54:15.509 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-base-2:18.2.4-855.g19e2a2be.el9.x86_64 28/32 2024-09-17T06:54:15.592 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-base-2:18.2.4-855.g19e2a2be.el9.x86_64 28/32 2024-09-17T06:54:15.592 INFO:teuthology.orchestra.run.smithi135.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-17T06:54:15.593 INFO:teuthology.orchestra.run.smithi135.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-17T06:54:15.593 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:54:15.626 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-selinux-2:18.2.4-855.g19e2a2be.el9.x86_64 29/32 2024-09-17T06:54:22.676 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-selinux-2:18.2.4-855.g19e2a2be.el9.x86_64 29/32 2024-09-17T06:54:22.676 INFO:teuthology.orchestra.run.smithi100.stdout:skipping the directory /sys 2024-09-17T06:54:22.676 INFO:teuthology.orchestra.run.smithi100.stdout:skipping the directory /proc 2024-09-17T06:54:22.676 INFO:teuthology.orchestra.run.smithi100.stdout:skipping the directory /mnt 2024-09-17T06:54:22.677 INFO:teuthology.orchestra.run.smithi100.stdout:skipping the directory /var/tmp 2024-09-17T06:54:22.677 INFO:teuthology.orchestra.run.smithi100.stdout:skipping the directory /home 2024-09-17T06:54:22.677 INFO:teuthology.orchestra.run.smithi100.stdout:skipping the directory /root 2024-09-17T06:54:22.677 INFO:teuthology.orchestra.run.smithi100.stdout:skipping the directory /tmp 2024-09-17T06:54:22.677 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:54:23.460 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-radosgw-2:18.2.4-855.g19e2a2be.el9.x86_64 30/32 2024-09-17T06:54:23.486 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-radosgw-2:18.2.4-855.g19e2a2be.el9.x86_64 30/32 2024-09-17T06:54:23.486 INFO:teuthology.orchestra.run.smithi100.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T06:54:23.486 INFO:teuthology.orchestra.run.smithi100.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-17T06:54:23.486 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-17T06:54:23.486 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-17T06:54:23.486 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:54:23.488 INFO:teuthology.orchestra.run.smithi100.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-09-17T06:54:23.563 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-09-17T06:54:23.563 INFO:teuthology.orchestra.run.smithi100.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-09-17T06:54:24.709 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-09-17T06:54:24.709 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-base-2:18.2.4-855.g19e2a2be.el9.x86_64 1/32 2024-09-17T06:54:24.709 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-common-2:18.2.4-855.g19e2a2be.el9.x86_64 2/32 2024-09-17T06:54:24.709 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-radosgw-2:18.2.4-855.g19e2a2be.el9.x86_64 3/32 2024-09-17T06:54:24.710 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-selinux-2:18.2.4-855.g19e2a2be.el9.x86_64 4/32 2024-09-17T06:54:24.710 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libcephfs2-2:18.2.4-855.g19e2a2be.el9.x86_64 5/32 2024-09-17T06:54:24.710 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libradosstriper1-2:18.2.4-855.g19e2a2be.el9.x86_64 6/32 2024-09-17T06:54:24.710 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : librgw2-2:18.2.4-855.g19e2a2be.el9.x86_64 7/32 2024-09-17T06:54:24.710 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-ceph-argparse-2:18.2.4-855.g19e2a2be.el9.x 8/32 2024-09-17T06:54:24.710 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-ceph-common-2:18.2.4-855.g19e2a2be.el9.x86 9/32 2024-09-17T06:54:24.710 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-cephfs-2:18.2.4-855.g19e2a2be.el9.x86_64 10/32 2024-09-17T06:54:24.710 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-rados-2:18.2.4-855.g19e2a2be.el9.x86_64 11/32 2024-09-17T06:54:24.710 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-rbd-2:18.2.4-855.g19e2a2be.el9.x86_64 12/32 2024-09-17T06:54:24.710 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-rgw-2:18.2.4-855.g19e2a2be.el9.x86_64 13/32 2024-09-17T06:54:24.710 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-09-17T06:54:24.711 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-09-17T06:54:24.711 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-09-17T06:54:24.711 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-09-17T06:54:24.711 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-09-17T06:54:24.711 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-09-17T06:54:24.711 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-09-17T06:54:24.711 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 21/32 2024-09-17T06:54:24.711 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 22/32 2024-09-17T06:54:24.711 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 23/32 2024-09-17T06:54:24.711 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-09-17T06:54:24.711 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-09-17T06:54:24.711 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 26/32 2024-09-17T06:54:24.711 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : re2-1:20211101-20.el9.x86_64 27/32 2024-09-17T06:54:24.712 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-09-17T06:54:24.712 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : librados2-2:18.2.4-855.g19e2a2be.el9.x86_64 29/32 2024-09-17T06:54:24.712 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-09-17T06:54:24.712 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : librbd1-2:18.2.4-855.g19e2a2be.el9.x86_64 31/32 2024-09-17T06:54:25.574 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-selinux-2:18.2.4-855.g19e2a2be.el9.x86_64 29/32 2024-09-17T06:54:25.575 INFO:teuthology.orchestra.run.smithi135.stdout:skipping the directory /sys 2024-09-17T06:54:25.575 INFO:teuthology.orchestra.run.smithi135.stdout:skipping the directory /proc 2024-09-17T06:54:25.575 INFO:teuthology.orchestra.run.smithi135.stdout:skipping the directory /mnt 2024-09-17T06:54:25.575 INFO:teuthology.orchestra.run.smithi135.stdout:skipping the directory /var/tmp 2024-09-17T06:54:25.575 INFO:teuthology.orchestra.run.smithi135.stdout:skipping the directory /home 2024-09-17T06:54:25.575 INFO:teuthology.orchestra.run.smithi135.stdout:skipping the directory /root 2024-09-17T06:54:25.575 INFO:teuthology.orchestra.run.smithi135.stdout:skipping the directory /tmp 2024-09-17T06:54:25.575 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:54:25.733 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-09-17T06:54:25.733 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:54:25.733 INFO:teuthology.orchestra.run.smithi100.stdout:Upgraded: 2024-09-17T06:54:25.733 INFO:teuthology.orchestra.run.smithi100.stdout: librados2-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T06:54:25.733 INFO:teuthology.orchestra.run.smithi100.stdout: librbd1-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T06:54:25.733 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-17T06:54:25.733 INFO:teuthology.orchestra.run.smithi100.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-17T06:54:25.734 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-base-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T06:54:25.734 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-common-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T06:54:25.734 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-radosgw-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T06:54:25.734 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-selinux-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T06:54:25.734 INFO:teuthology.orchestra.run.smithi100.stdout: fmt-8.1.1-5.el9.x86_64 2024-09-17T06:54:25.734 INFO:teuthology.orchestra.run.smithi100.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-17T06:54:25.734 INFO:teuthology.orchestra.run.smithi100.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-17T06:54:25.734 INFO:teuthology.orchestra.run.smithi100.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-17T06:54:25.734 INFO:teuthology.orchestra.run.smithi100.stdout: libcephfs2-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T06:54:25.734 INFO:teuthology.orchestra.run.smithi100.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-17T06:54:25.734 INFO:teuthology.orchestra.run.smithi100.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-17T06:54:25.734 INFO:teuthology.orchestra.run.smithi100.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-17T06:54:25.734 INFO:teuthology.orchestra.run.smithi100.stdout: libradosstriper1-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T06:54:25.735 INFO:teuthology.orchestra.run.smithi100.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-17T06:54:25.735 INFO:teuthology.orchestra.run.smithi100.stdout: librgw2-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T06:54:25.735 INFO:teuthology.orchestra.run.smithi100.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-17T06:54:25.735 INFO:teuthology.orchestra.run.smithi100.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-17T06:54:25.735 INFO:teuthology.orchestra.run.smithi100.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-17T06:54:25.735 INFO:teuthology.orchestra.run.smithi100.stdout: python3-ceph-argparse-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T06:54:25.735 INFO:teuthology.orchestra.run.smithi100.stdout: python3-ceph-common-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T06:54:25.735 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cephfs-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T06:54:25.735 INFO:teuthology.orchestra.run.smithi100.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-17T06:54:25.735 INFO:teuthology.orchestra.run.smithi100.stdout: python3-rados-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T06:54:25.735 INFO:teuthology.orchestra.run.smithi100.stdout: python3-rbd-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T06:54:25.735 INFO:teuthology.orchestra.run.smithi100.stdout: python3-rgw-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T06:54:25.735 INFO:teuthology.orchestra.run.smithi100.stdout: re2-1:20211101-20.el9.x86_64 2024-09-17T06:54:25.735 INFO:teuthology.orchestra.run.smithi100.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-17T06:54:25.736 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:54:25.736 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-17T06:54:26.099 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph-test 2024-09-17T06:54:26.326 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-radosgw-2:18.2.4-855.g19e2a2be.el9.x86_64 30/32 2024-09-17T06:54:26.351 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-radosgw-2:18.2.4-855.g19e2a2be.el9.x86_64 30/32 2024-09-17T06:54:26.352 INFO:teuthology.orchestra.run.smithi135.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T06:54:26.352 INFO:teuthology.orchestra.run.smithi135.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-17T06:54:26.352 INFO:teuthology.orchestra.run.smithi135.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-17T06:54:26.352 INFO:teuthology.orchestra.run.smithi135.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-17T06:54:26.352 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:54:26.353 INFO:teuthology.orchestra.run.smithi135.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-09-17T06:54:26.445 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-09-17T06:54:26.446 INFO:teuthology.orchestra.run.smithi135.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-09-17T06:54:26.675 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:00:26 ago on Tue 17 Sep 2024 06:54:00 AM UTC. 2024-09-17T06:54:26.807 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-17T06:54:26.807 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T06:54:26.807 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repository Size 2024-09-17T06:54:26.807 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T06:54:26.807 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-17T06:54:26.808 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-test x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 40 M 2024-09-17T06:54:26.808 INFO:teuthology.orchestra.run.smithi100.stdout:Installing dependencies: 2024-09-17T06:54:26.808 INFO:teuthology.orchestra.run.smithi100.stdout: libcephsqlite x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 166 k 2024-09-17T06:54:26.808 INFO:teuthology.orchestra.run.smithi100.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-17T06:54:26.808 INFO:teuthology.orchestra.run.smithi100.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-17T06:54:26.808 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:54:26.808 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-17T06:54:26.808 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T06:54:26.808 INFO:teuthology.orchestra.run.smithi100.stdout:Install 4 Packages 2024-09-17T06:54:26.808 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:54:26.809 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 40 M 2024-09-17T06:54:26.809 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 164 M 2024-09-17T06:54:26.809 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-17T06:54:27.055 INFO:teuthology.orchestra.run.smithi100.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.5 MB/s | 304 kB 00:00 2024-09-17T06:54:27.080 INFO:teuthology.orchestra.run.smithi100.stdout:(2/4): libcephsqlite-18.2.4-855.g19e2a2be.el9.x 740 kB/s | 166 kB 00:00 2024-09-17T06:54:27.105 INFO:teuthology.orchestra.run.smithi100.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-09-17T06:54:27.481 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-09-17T06:54:27.481 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-base-2:18.2.4-855.g19e2a2be.el9.x86_64 1/32 2024-09-17T06:54:27.481 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-common-2:18.2.4-855.g19e2a2be.el9.x86_64 2/32 2024-09-17T06:54:27.481 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-radosgw-2:18.2.4-855.g19e2a2be.el9.x86_64 3/32 2024-09-17T06:54:27.481 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-selinux-2:18.2.4-855.g19e2a2be.el9.x86_64 4/32 2024-09-17T06:54:27.481 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libcephfs2-2:18.2.4-855.g19e2a2be.el9.x86_64 5/32 2024-09-17T06:54:27.481 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libradosstriper1-2:18.2.4-855.g19e2a2be.el9.x86_64 6/32 2024-09-17T06:54:27.481 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : librgw2-2:18.2.4-855.g19e2a2be.el9.x86_64 7/32 2024-09-17T06:54:27.481 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-ceph-argparse-2:18.2.4-855.g19e2a2be.el9.x 8/32 2024-09-17T06:54:27.482 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-ceph-common-2:18.2.4-855.g19e2a2be.el9.x86 9/32 2024-09-17T06:54:27.482 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-cephfs-2:18.2.4-855.g19e2a2be.el9.x86_64 10/32 2024-09-17T06:54:27.482 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-rados-2:18.2.4-855.g19e2a2be.el9.x86_64 11/32 2024-09-17T06:54:27.482 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-rbd-2:18.2.4-855.g19e2a2be.el9.x86_64 12/32 2024-09-17T06:54:27.482 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-rgw-2:18.2.4-855.g19e2a2be.el9.x86_64 13/32 2024-09-17T06:54:27.482 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-09-17T06:54:27.482 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-09-17T06:54:27.482 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-09-17T06:54:27.482 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-09-17T06:54:27.482 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-09-17T06:54:27.482 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-09-17T06:54:27.482 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-09-17T06:54:27.482 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 21/32 2024-09-17T06:54:27.483 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 22/32 2024-09-17T06:54:27.483 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 23/32 2024-09-17T06:54:27.483 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-09-17T06:54:27.483 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-09-17T06:54:27.483 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 26/32 2024-09-17T06:54:27.483 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : re2-1:20211101-20.el9.x86_64 27/32 2024-09-17T06:54:27.483 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-09-17T06:54:27.483 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : librados2-2:18.2.4-855.g19e2a2be.el9.x86_64 29/32 2024-09-17T06:54:27.483 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-09-17T06:54:27.483 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : librbd1-2:18.2.4-855.g19e2a2be.el9.x86_64 31/32 2024-09-17T06:54:28.172 INFO:teuthology.orchestra.run.smithi100.stdout:(4/4): ceph-test-18.2.4-855.g19e2a2be.el9.x86_6 30 MB/s | 40 MB 00:01 2024-09-17T06:54:28.174 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-17T06:54:28.175 INFO:teuthology.orchestra.run.smithi100.stdout:Total 29 MB/s | 40 MB 00:01 2024-09-17T06:54:28.202 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-17T06:54:28.221 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-17T06:54:28.221 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-17T06:54:28.507 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-09-17T06:54:28.507 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:54:28.507 INFO:teuthology.orchestra.run.smithi135.stdout:Upgraded: 2024-09-17T06:54:28.507 INFO:teuthology.orchestra.run.smithi135.stdout: librados2-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T06:54:28.507 INFO:teuthology.orchestra.run.smithi135.stdout: librbd1-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T06:54:28.507 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-09-17T06:54:28.507 INFO:teuthology.orchestra.run.smithi135.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-17T06:54:28.507 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-base-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T06:54:28.508 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-common-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T06:54:28.508 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-radosgw-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T06:54:28.508 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-selinux-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T06:54:28.508 INFO:teuthology.orchestra.run.smithi135.stdout: fmt-8.1.1-5.el9.x86_64 2024-09-17T06:54:28.508 INFO:teuthology.orchestra.run.smithi135.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-17T06:54:28.508 INFO:teuthology.orchestra.run.smithi135.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-17T06:54:28.508 INFO:teuthology.orchestra.run.smithi135.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-17T06:54:28.508 INFO:teuthology.orchestra.run.smithi135.stdout: libcephfs2-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T06:54:28.508 INFO:teuthology.orchestra.run.smithi135.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-17T06:54:28.508 INFO:teuthology.orchestra.run.smithi135.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-17T06:54:28.508 INFO:teuthology.orchestra.run.smithi135.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-17T06:54:28.508 INFO:teuthology.orchestra.run.smithi135.stdout: libradosstriper1-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T06:54:28.508 INFO:teuthology.orchestra.run.smithi135.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-17T06:54:28.509 INFO:teuthology.orchestra.run.smithi135.stdout: librgw2-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T06:54:28.509 INFO:teuthology.orchestra.run.smithi135.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-17T06:54:28.509 INFO:teuthology.orchestra.run.smithi135.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-17T06:54:28.509 INFO:teuthology.orchestra.run.smithi135.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-17T06:54:28.509 INFO:teuthology.orchestra.run.smithi135.stdout: python3-ceph-argparse-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T06:54:28.509 INFO:teuthology.orchestra.run.smithi135.stdout: python3-ceph-common-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T06:54:28.509 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cephfs-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T06:54:28.509 INFO:teuthology.orchestra.run.smithi135.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-17T06:54:28.509 INFO:teuthology.orchestra.run.smithi135.stdout: python3-rados-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T06:54:28.509 INFO:teuthology.orchestra.run.smithi135.stdout: python3-rbd-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T06:54:28.509 INFO:teuthology.orchestra.run.smithi135.stdout: python3-rgw-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T06:54:28.509 INFO:teuthology.orchestra.run.smithi135.stdout: re2-1:20211101-20.el9.x86_64 2024-09-17T06:54:28.510 INFO:teuthology.orchestra.run.smithi135.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-17T06:54:28.510 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:54:28.510 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-09-17T06:54:28.543 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-17T06:54:28.544 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-17T06:54:28.865 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-test 2024-09-17T06:54:29.049 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-17T06:54:29.123 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-17T06:54:29.163 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-17T06:54:29.190 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libcephsqlite-2:18.2.4-855.g19e2a2be.el9.x86_64 3/4 2024-09-17T06:54:29.458 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:00:28 ago on Tue 17 Sep 2024 06:54:01 AM UTC. 2024-09-17T06:54:29.600 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-09-17T06:54:29.600 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-09-17T06:54:29.601 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repository Size 2024-09-17T06:54:29.601 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-09-17T06:54:29.601 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-09-17T06:54:29.601 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-test x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 40 M 2024-09-17T06:54:29.601 INFO:teuthology.orchestra.run.smithi135.stdout:Installing dependencies: 2024-09-17T06:54:29.601 INFO:teuthology.orchestra.run.smithi135.stdout: libcephsqlite x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 166 k 2024-09-17T06:54:29.601 INFO:teuthology.orchestra.run.smithi135.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-17T06:54:29.601 INFO:teuthology.orchestra.run.smithi135.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-17T06:54:29.601 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:54:29.601 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-09-17T06:54:29.601 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-09-17T06:54:29.601 INFO:teuthology.orchestra.run.smithi135.stdout:Install 4 Packages 2024-09-17T06:54:29.602 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:54:29.602 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 40 M 2024-09-17T06:54:29.602 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 164 M 2024-09-17T06:54:29.602 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-09-17T06:54:30.006 INFO:teuthology.orchestra.run.smithi135.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 2.1 MB/s | 304 kB 00:00 2024-09-17T06:54:30.048 INFO:teuthology.orchestra.run.smithi135.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-09-17T06:54:30.090 INFO:teuthology.orchestra.run.smithi135.stdout:(3/4): libcephsqlite-18.2.4-855.g19e2a2be.el9.x 729 kB/s | 166 kB 00:00 2024-09-17T06:54:31.215 INFO:teuthology.orchestra.run.smithi135.stdout:(4/4): ceph-test-18.2.4-855.g19e2a2be.el9.x86_6 29 MB/s | 40 MB 00:01 2024-09-17T06:54:31.217 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-09-17T06:54:31.218 INFO:teuthology.orchestra.run.smithi135.stdout:Total 25 MB/s | 40 MB 00:01 2024-09-17T06:54:31.246 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-09-17T06:54:31.266 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-09-17T06:54:31.266 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-09-17T06:54:31.598 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-09-17T06:54:31.598 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-09-17T06:54:32.097 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-09-17T06:54:32.170 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-17T06:54:32.214 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-17T06:54:32.237 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : libcephsqlite-2:18.2.4-855.g19e2a2be.el9.x86_64 3/4 2024-09-17T06:54:32.873 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: libcephsqlite-2:18.2.4-855.g19e2a2be.el9.x86_64 3/4 2024-09-17T06:54:32.955 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-test-2:18.2.4-855.g19e2a2be.el9.x86_64 4/4 2024-09-17T06:54:34.945 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-test-2:18.2.4-855.g19e2a2be.el9.x86_64 4/4 2024-09-17T06:54:34.945 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-test-2:18.2.4-855.g19e2a2be.el9.x86_64 1/4 2024-09-17T06:54:34.945 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libcephsqlite-2:18.2.4-855.g19e2a2be.el9.x86_64 2/4 2024-09-17T06:54:34.945 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-17T06:54:35.277 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-17T06:54:35.277 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:54:35.277 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-17T06:54:35.277 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-test-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T06:54:35.277 INFO:teuthology.orchestra.run.smithi100.stdout: libcephsqlite-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T06:54:35.277 INFO:teuthology.orchestra.run.smithi100.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-17T06:54:35.277 INFO:teuthology.orchestra.run.smithi100.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-17T06:54:35.277 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:54:35.278 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-17T06:54:35.616 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph 2024-09-17T06:54:35.940 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: libcephsqlite-2:18.2.4-855.g19e2a2be.el9.x86_64 3/4 2024-09-17T06:54:36.035 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-test-2:18.2.4-855.g19e2a2be.el9.x86_64 4/4 2024-09-17T06:54:36.201 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:00:36 ago on Tue 17 Sep 2024 06:54:00 AM UTC. 2024-09-17T06:54:36.366 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-17T06:54:36.368 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T06:54:36.368 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repository Size 2024-09-17T06:54:36.368 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T06:54:36.368 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-17T06:54:36.368 INFO:teuthology.orchestra.run.smithi100.stdout: ceph x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 6.5 k 2024-09-17T06:54:36.369 INFO:teuthology.orchestra.run.smithi100.stdout:Installing dependencies: 2024-09-17T06:54:36.369 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mds x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 2.1 M 2024-09-17T06:54:36.369 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 1.5 M 2024-09-17T06:54:36.369 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-modules-core noarch 2:18.2.4-855.g19e2a2be.el9 ceph-noarch 246 k 2024-09-17T06:54:36.369 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mon x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 4.7 M 2024-09-17T06:54:36.369 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-osd x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 17 M 2024-09-17T06:54:36.369 INFO:teuthology.orchestra.run.smithi100.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-17T06:54:36.369 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-17T06:54:36.370 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-17T06:54:36.370 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-17T06:54:36.370 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-17T06:54:36.370 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-17T06:54:36.370 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-17T06:54:36.370 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-17T06:54:36.370 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-17T06:54:36.370 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-17T06:54:36.370 INFO:teuthology.orchestra.run.smithi100.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-17T06:54:36.371 INFO:teuthology.orchestra.run.smithi100.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-17T06:54:36.371 INFO:teuthology.orchestra.run.smithi100.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-17T06:54:36.371 INFO:teuthology.orchestra.run.smithi100.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-17T06:54:36.371 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-17T06:54:36.371 INFO:teuthology.orchestra.run.smithi100.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-17T06:54:36.371 INFO:teuthology.orchestra.run.smithi100.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-17T06:54:36.371 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-17T06:54:36.371 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-17T06:54:36.371 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-17T06:54:36.371 INFO:teuthology.orchestra.run.smithi100.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-17T06:54:36.371 INFO:teuthology.orchestra.run.smithi100.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-17T06:54:36.371 INFO:teuthology.orchestra.run.smithi100.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-17T06:54:36.371 INFO:teuthology.orchestra.run.smithi100.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-17T06:54:36.372 INFO:teuthology.orchestra.run.smithi100.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-17T06:54:36.372 INFO:teuthology.orchestra.run.smithi100.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-17T06:54:36.372 INFO:teuthology.orchestra.run.smithi100.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-17T06:54:36.372 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:54:36.372 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-17T06:54:36.372 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T06:54:36.372 INFO:teuthology.orchestra.run.smithi100.stdout:Install 33 Packages 2024-09-17T06:54:36.372 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:54:36.373 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 29 M 2024-09-17T06:54:36.373 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 104 M 2024-09-17T06:54:36.373 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-17T06:54:36.608 INFO:teuthology.orchestra.run.smithi100.stdout:(1/33): ceph-18.2.4-855.g19e2a2be.el9.x86_64.rp 48 kB/s | 6.5 kB 00:00 2024-09-17T06:54:36.784 INFO:teuthology.orchestra.run.smithi100.stdout:(2/33): ceph-mds-18.2.4-855.g19e2a2be.el9.x86_6 6.8 MB/s | 2.1 MB 00:00 2024-09-17T06:54:36.825 INFO:teuthology.orchestra.run.smithi100.stdout:(3/33): ceph-mgr-18.2.4-855.g19e2a2be.el9.x86_6 4.1 MB/s | 1.5 MB 00:00 2024-09-17T06:54:36.909 INFO:teuthology.orchestra.run.smithi100.stdout:(4/33): ceph-mgr-modules-core-18.2.4-855.g19e2a 2.9 MB/s | 246 kB 00:00 2024-09-17T06:54:37.301 INFO:teuthology.orchestra.run.smithi100.stdout:(5/33): ceph-osd-18.2.4-855.g19e2a2be.el9.x86_6 32 MB/s | 17 MB 00:00 2024-09-17T06:54:37.368 INFO:teuthology.orchestra.run.smithi100.stdout:(6/33): ceph-mon-18.2.4-855.g19e2a2be.el9.x86_6 6.2 MB/s | 4.7 MB 00:00 2024-09-17T06:54:37.635 INFO:teuthology.orchestra.run.smithi100.stdout:(7/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 349 kB/s | 253 kB 00:00 2024-09-17T06:54:37.760 INFO:teuthology.orchestra.run.smithi100.stdout:(8/33): python3-pycparser-2.20-6.el9.noarch.rpm 1.1 MB/s | 135 kB 00:00 2024-09-17T06:54:37.827 INFO:teuthology.orchestra.run.smithi100.stdout:(9/33): python3-ply-3.11-14.el9.noarch.rpm 232 kB/s | 106 kB 00:00 2024-09-17T06:54:37.878 INFO:teuthology.orchestra.run.smithi100.stdout:(10/33): python3-requests-2.25.1-8.el9.noarch.r 1.0 MB/s | 125 kB 00:00 2024-09-17T06:54:38.003 INFO:teuthology.orchestra.run.smithi100.stdout:(11/33): python3-mako-1.1.4-6.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2024-09-17T06:54:38.045 INFO:teuthology.orchestra.run.smithi100.stdout:(12/33): python3-markupsafe-1.1.1-12.el9.x86_64 838 kB/s | 35 kB 00:00 2024-09-17T06:54:38.058 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-test-2:18.2.4-855.g19e2a2be.el9.x86_64 4/4 2024-09-17T06:54:38.058 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-test-2:18.2.4-855.g19e2a2be.el9.x86_64 1/4 2024-09-17T06:54:38.058 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libcephsqlite-2:18.2.4-855.g19e2a2be.el9.x86_64 2/4 2024-09-17T06:54:38.058 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-17T06:54:38.087 INFO:teuthology.orchestra.run.smithi100.stdout:(13/33): python3-pytz-2021.1-5.el9.noarch.rpm 1.2 MB/s | 51 kB 00:00 2024-09-17T06:54:38.112 INFO:teuthology.orchestra.run.smithi100.stdout:(14/33): python3-urllib3-1.26.5-6.el9.noarch.rp 755 kB/s | 215 kB 00:00 2024-09-17T06:54:38.137 INFO:teuthology.orchestra.run.smithi100.stdout:(15/33): python3-toml-0.10.2-6.el9.noarch.rpm 828 kB/s | 42 kB 00:00 2024-09-17T06:54:38.179 INFO:teuthology.orchestra.run.smithi100.stdout:(16/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 651 kB/s | 43 kB 00:00 2024-09-17T06:54:38.213 INFO:teuthology.orchestra.run.smithi100.stdout:(17/33): python3-cherrypy-18.6.1-2.el9.noarch.r 11 MB/s | 358 kB 00:00 2024-09-17T06:54:38.238 INFO:teuthology.orchestra.run.smithi100.stdout:(18/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.7 MB/s | 172 kB 00:00 2024-09-17T06:54:38.264 INFO:teuthology.orchestra.run.smithi100.stdout:(19/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 213 kB/s | 11 kB 00:00 2024-09-17T06:54:38.289 INFO:teuthology.orchestra.run.smithi100.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 353 kB/s | 18 kB 00:00 2024-09-17T06:54:38.314 INFO:teuthology.orchestra.run.smithi100.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 460 kB/s | 23 kB 00:00 2024-09-17T06:54:38.348 INFO:teuthology.orchestra.run.smithi100.stdout:(22/33): python3-cryptography-36.0.1-4.el9.x86_ 1.2 MB/s | 1.2 MB 00:01 2024-09-17T06:54:38.373 INFO:teuthology.orchestra.run.smithi100.stdout:(23/33): python3-jaraco-functools-3.5.0-2.el9.n 232 kB/s | 19 kB 00:00 2024-09-17T06:54:38.375 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-17T06:54:38.375 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:54:38.375 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-09-17T06:54:38.375 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-test-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T06:54:38.375 INFO:teuthology.orchestra.run.smithi135.stdout: libcephsqlite-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T06:54:38.375 INFO:teuthology.orchestra.run.smithi135.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-17T06:54:38.376 INFO:teuthology.orchestra.run.smithi135.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-17T06:54:38.376 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:54:38.376 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-09-17T06:54:38.398 INFO:teuthology.orchestra.run.smithi100.stdout:(24/33): python3-jaraco-text-3.2.0-6.el9.noarch 235 kB/s | 20 kB 00:00 2024-09-17T06:54:38.432 INFO:teuthology.orchestra.run.smithi100.stdout:(25/33): python3-logutils-0.3.5-21.el9.noarch.r 554 kB/s | 46 kB 00:00 2024-09-17T06:54:38.457 INFO:teuthology.orchestra.run.smithi100.stdout:(26/33): python3-portend-3.1.0-2.el9.noarch.rpm 710 kB/s | 16 kB 00:00 2024-09-17T06:54:38.482 INFO:teuthology.orchestra.run.smithi100.stdout:(27/33): python3-more-itertools-8.12.0-2.el9.no 723 kB/s | 79 kB 00:00 2024-09-17T06:54:38.524 INFO:teuthology.orchestra.run.smithi100.stdout:(28/33): python3-pecan-1.4.2-3.el9.noarch.rpm 2.3 MB/s | 272 kB 00:00 2024-09-17T06:54:38.549 INFO:teuthology.orchestra.run.smithi100.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 534 kB/s | 36 kB 00:00 2024-09-17T06:54:38.575 INFO:teuthology.orchestra.run.smithi100.stdout:(30/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 766 kB/s | 90 kB 00:00 2024-09-17T06:54:38.600 INFO:teuthology.orchestra.run.smithi100.stdout:(31/33): python3-zc-lockfile-2.0-10.el9.noarch. 800 kB/s | 20 kB 00:00 2024-09-17T06:54:38.625 INFO:teuthology.orchestra.run.smithi100.stdout:(32/33): python3-webob-1.8.8-2.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-09-17T06:54:38.650 INFO:teuthology.orchestra.run.smithi100.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.2 MB/s | 427 kB 00:00 2024-09-17T06:54:38.655 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-17T06:54:38.655 INFO:teuthology.orchestra.run.smithi100.stdout:Total 13 MB/s | 29 MB 00:02 2024-09-17T06:54:38.666 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph 2024-09-17T06:54:38.779 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-17T06:54:38.816 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-17T06:54:38.816 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-17T06:54:39.242 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-17T06:54:39.242 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-17T06:54:39.265 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:00:38 ago on Tue 17 Sep 2024 06:54:01 AM UTC. 2024-09-17T06:54:39.409 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-09-17T06:54:39.411 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-09-17T06:54:39.411 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repository Size 2024-09-17T06:54:39.411 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-09-17T06:54:39.411 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-09-17T06:54:39.411 INFO:teuthology.orchestra.run.smithi135.stdout: ceph x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 6.5 k 2024-09-17T06:54:39.411 INFO:teuthology.orchestra.run.smithi135.stdout:Installing dependencies: 2024-09-17T06:54:39.411 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mds x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 2.1 M 2024-09-17T06:54:39.411 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 1.5 M 2024-09-17T06:54:39.411 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-modules-core noarch 2:18.2.4-855.g19e2a2be.el9 ceph-noarch 246 k 2024-09-17T06:54:39.412 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mon x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 4.7 M 2024-09-17T06:54:39.412 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-osd x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 17 M 2024-09-17T06:54:39.412 INFO:teuthology.orchestra.run.smithi135.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-17T06:54:39.412 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-17T06:54:39.412 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-17T06:54:39.412 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-17T06:54:39.412 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-17T06:54:39.412 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-17T06:54:39.412 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-17T06:54:39.412 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-17T06:54:39.412 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-17T06:54:39.412 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-17T06:54:39.413 INFO:teuthology.orchestra.run.smithi135.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-17T06:54:39.413 INFO:teuthology.orchestra.run.smithi135.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-17T06:54:39.413 INFO:teuthology.orchestra.run.smithi135.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-17T06:54:39.413 INFO:teuthology.orchestra.run.smithi135.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-17T06:54:39.413 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-17T06:54:39.413 INFO:teuthology.orchestra.run.smithi135.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-17T06:54:39.413 INFO:teuthology.orchestra.run.smithi135.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-17T06:54:39.413 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-17T06:54:39.413 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-17T06:54:39.413 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-17T06:54:39.413 INFO:teuthology.orchestra.run.smithi135.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-17T06:54:39.413 INFO:teuthology.orchestra.run.smithi135.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-17T06:54:39.413 INFO:teuthology.orchestra.run.smithi135.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-17T06:54:39.414 INFO:teuthology.orchestra.run.smithi135.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-17T06:54:39.414 INFO:teuthology.orchestra.run.smithi135.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-17T06:54:39.414 INFO:teuthology.orchestra.run.smithi135.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-17T06:54:39.414 INFO:teuthology.orchestra.run.smithi135.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-17T06:54:39.414 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:54:39.414 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-09-17T06:54:39.414 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-09-17T06:54:39.414 INFO:teuthology.orchestra.run.smithi135.stdout:Install 33 Packages 2024-09-17T06:54:39.414 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:54:39.415 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 29 M 2024-09-17T06:54:39.415 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 104 M 2024-09-17T06:54:39.415 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-09-17T06:54:40.037 INFO:teuthology.orchestra.run.smithi135.stdout:(1/33): ceph-18.2.4-855.g19e2a2be.el9.x86_64.rp 48 kB/s | 6.5 kB 00:00 2024-09-17T06:54:40.220 INFO:teuthology.orchestra.run.smithi135.stdout:(2/33): ceph-mgr-18.2.4-855.g19e2a2be.el9.x86_6 4.6 MB/s | 1.5 MB 00:00 2024-09-17T06:54:40.271 INFO:teuthology.orchestra.run.smithi135.stdout:(3/33): ceph-mds-18.2.4-855.g19e2a2be.el9.x86_6 5.7 MB/s | 2.1 MB 00:00 2024-09-17T06:54:40.346 INFO:teuthology.orchestra.run.smithi135.stdout:(4/33): ceph-mgr-modules-core-18.2.4-855.g19e2a 3.2 MB/s | 246 kB 00:00 2024-09-17T06:54:40.491 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-17T06:54:40.553 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-09-17T06:54:40.555 INFO:teuthology.orchestra.run.smithi135.stdout:(5/33): ceph-mon-18.2.4-855.g19e2a2be.el9.x86_6 9.0 MB/s | 4.7 MB 00:00 2024-09-17T06:54:40.580 INFO:teuthology.orchestra.run.smithi135.stdout:(6/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.1 MB/s | 253 kB 00:00 2024-09-17T06:54:40.585 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-09-17T06:54:40.618 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-09-17T06:54:40.711 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-09-17T06:54:40.787 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-09-17T06:54:40.822 INFO:teuthology.orchestra.run.smithi135.stdout:(7/33): ceph-osd-18.2.4-855.g19e2a2be.el9.x86_6 27 MB/s | 17 MB 00:00 2024-09-17T06:54:40.829 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 6/33 2024-09-17T06:54:40.847 INFO:teuthology.orchestra.run.smithi135.stdout:(8/33): python3-ply-3.11-14.el9.noarch.rpm 399 kB/s | 106 kB 00:00 2024-09-17T06:54:40.864 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-09-17T06:54:40.894 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/33 2024-09-17T06:54:40.920 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/33 2024-09-17T06:54:40.939 INFO:teuthology.orchestra.run.smithi135.stdout:(9/33): python3-cryptography-36.0.1-4.el9.x86_6 3.2 MB/s | 1.2 MB 00:00 2024-09-17T06:54:40.953 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/33 2024-09-17T06:54:40.964 INFO:teuthology.orchestra.run.smithi135.stdout:(10/33): python3-requests-2.25.1-8.el9.noarch.r 1.0 MB/s | 125 kB 00:00 2024-09-17T06:54:40.994 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/33 2024-09-17T06:54:41.015 INFO:teuthology.orchestra.run.smithi135.stdout:(11/33): python3-pycparser-2.20-6.el9.noarch.rp 703 kB/s | 135 kB 00:00 2024-09-17T06:54:41.029 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/33 2024-09-17T06:54:41.040 INFO:teuthology.orchestra.run.smithi135.stdout:(12/33): python3-urllib3-1.26.5-6.el9.noarch.rp 2.1 MB/s | 215 kB 00:00 2024-09-17T06:54:41.063 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/33 2024-09-17T06:54:41.110 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/33 2024-09-17T06:54:41.191 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 15/33 2024-09-17T06:54:41.329 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 16/33 2024-09-17T06:54:41.417 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 17/33 2024-09-17T06:54:41.479 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 18/33 2024-09-17T06:54:41.518 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 19/33 2024-09-17T06:54:41.565 INFO:teuthology.orchestra.run.smithi135.stdout:(13/33): python3-markupsafe-1.1.1-12.el9.x86_64 63 kB/s | 35 kB 00:00 2024-09-17T06:54:41.567 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-ply-3.11-14.el9.noarch 20/33 2024-09-17T06:54:41.622 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 21/33 2024-09-17T06:54:41.682 INFO:teuthology.orchestra.run.smithi135.stdout:(14/33): python3-mako-1.1.4-6.el9.noarch.rpm 240 kB/s | 172 kB 00:00 2024-09-17T06:54:41.707 INFO:teuthology.orchestra.run.smithi135.stdout:(15/33): python3-pytz-2021.1-5.el9.noarch.rpm 76 kB/s | 51 kB 00:00 2024-09-17T06:54:41.749 INFO:teuthology.orchestra.run.smithi135.stdout:(16/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 660 kB/s | 43 kB 00:00 2024-09-17T06:54:41.783 INFO:teuthology.orchestra.run.smithi135.stdout:(17/33): python3-cherrypy-18.6.1-2.el9.noarch.r 11 MB/s | 358 kB 00:00 2024-09-17T06:54:41.806 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 22/33 2024-09-17T06:54:41.816 INFO:teuthology.orchestra.run.smithi135.stdout:(18/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.5 MB/s | 172 kB 00:00 2024-09-17T06:54:41.842 INFO:teuthology.orchestra.run.smithi135.stdout:(19/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 208 kB/s | 11 kB 00:00 2024-09-17T06:54:41.867 INFO:teuthology.orchestra.run.smithi135.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 713 kB/s | 18 kB 00:00 2024-09-17T06:54:41.871 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 23/33 2024-09-17T06:54:41.892 INFO:teuthology.orchestra.run.smithi135.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 463 kB/s | 23 kB 00:00 2024-09-17T06:54:41.917 INFO:teuthology.orchestra.run.smithi135.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 393 kB/s | 19 kB 00:00 2024-09-17T06:54:41.942 INFO:teuthology.orchestra.run.smithi135.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 393 kB/s | 20 kB 00:00 2024-09-17T06:54:41.952 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 24/33 2024-09-17T06:54:41.968 INFO:teuthology.orchestra.run.smithi135.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 924 kB/s | 46 kB 00:00 2024-09-17T06:54:41.993 INFO:teuthology.orchestra.run.smithi135.stdout:(25/33): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-09-17T06:54:42.018 INFO:teuthology.orchestra.run.smithi135.stdout:(26/33): python3-portend-3.1.0-2.el9.noarch.rpm 656 kB/s | 16 kB 00:00 2024-09-17T06:54:42.040 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 25/33 2024-09-17T06:54:42.044 INFO:teuthology.orchestra.run.smithi135.stdout:(27/33): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-09-17T06:54:42.070 INFO:teuthology.orchestra.run.smithi135.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-09-17T06:54:42.082 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 26/33 2024-09-17T06:54:42.094 INFO:teuthology.orchestra.run.smithi135.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 713 kB/s | 36 kB 00:00 2024-09-17T06:54:42.128 INFO:teuthology.orchestra.run.smithi135.stdout:(30/33): python3-webob-1.8.8-2.el9.noarch.rpm 3.8 MB/s | 230 kB 00:00 2024-09-17T06:54:42.153 INFO:teuthology.orchestra.run.smithi135.stdout:(31/33): python3-toml-0.10.2-6.el9.noarch.rpm 71 kB/s | 42 kB 00:00 2024-09-17T06:54:42.178 INFO:teuthology.orchestra.run.smithi135.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 399 kB/s | 20 kB 00:00 2024-09-17T06:54:42.197 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 27/33 2024-09-17T06:54:42.204 INFO:teuthology.orchestra.run.smithi135.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.8 MB/s | 427 kB 00:00 2024-09-17T06:54:42.208 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-09-17T06:54:42.209 INFO:teuthology.orchestra.run.smithi135.stdout:Total 10 MB/s | 29 MB 00:02 2024-09-17T06:54:42.333 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-09-17T06:54:42.351 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-mgr-modules-core-2:18.2.4-855.g19e2a2be.el9.n 28/33 2024-09-17T06:54:42.370 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-09-17T06:54:42.370 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-09-17T06:54:42.375 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-mgr-2:18.2.4-855.g19e2a2be.el9.x86_64 29/33 2024-09-17T06:54:42.401 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-mgr-2:18.2.4-855.g19e2a2be.el9.x86_64 29/33 2024-09-17T06:54:42.401 INFO:teuthology.orchestra.run.smithi100.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T06:54:42.401 INFO:teuthology.orchestra.run.smithi100.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-17T06:54:42.402 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-17T06:54:42.402 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-17T06:54:42.402 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:54:42.774 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-09-17T06:54:42.774 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-09-17T06:54:43.885 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-osd-2:18.2.4-855.g19e2a2be.el9.x86_64 30/33 2024-09-17T06:54:43.916 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-osd-2:18.2.4-855.g19e2a2be.el9.x86_64 30/33 2024-09-17T06:54:43.917 INFO:teuthology.orchestra.run.smithi100.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T06:54:43.917 INFO:teuthology.orchestra.run.smithi100.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-17T06:54:43.917 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-17T06:54:43.917 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-17T06:54:43.917 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:54:43.980 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-09-17T06:54:44.056 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-09-17T06:54:44.088 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-09-17T06:54:44.121 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-09-17T06:54:44.213 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-09-17T06:54:44.291 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-09-17T06:54:44.332 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 6/33 2024-09-17T06:54:44.361 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-mon-2:18.2.4-855.g19e2a2be.el9.x86_64 31/33 2024-09-17T06:54:44.375 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-09-17T06:54:44.385 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-mon-2:18.2.4-855.g19e2a2be.el9.x86_64 31/33 2024-09-17T06:54:44.385 INFO:teuthology.orchestra.run.smithi100.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T06:54:44.385 INFO:teuthology.orchestra.run.smithi100.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-17T06:54:44.385 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-17T06:54:44.385 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-17T06:54:44.385 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:54:44.406 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/33 2024-09-17T06:54:44.448 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/33 2024-09-17T06:54:44.481 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/33 2024-09-17T06:54:44.522 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/33 2024-09-17T06:54:44.557 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/33 2024-09-17T06:54:44.586 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-mds-2:18.2.4-855.g19e2a2be.el9.x86_64 32/33 2024-09-17T06:54:44.592 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/33 2024-09-17T06:54:44.610 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-mds-2:18.2.4-855.g19e2a2be.el9.x86_64 32/33 2024-09-17T06:54:44.610 INFO:teuthology.orchestra.run.smithi100.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T06:54:44.610 INFO:teuthology.orchestra.run.smithi100.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-17T06:54:44.610 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-17T06:54:44.610 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-17T06:54:44.611 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:54:44.630 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/33 2024-09-17T06:54:44.675 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-2:18.2.4-855.g19e2a2be.el9.x86_64 33/33 2024-09-17T06:54:44.703 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 15/33 2024-09-17T06:54:44.850 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 16/33 2024-09-17T06:54:44.929 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 17/33 2024-09-17T06:54:44.982 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 18/33 2024-09-17T06:54:45.021 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 19/33 2024-09-17T06:54:45.069 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-ply-3.11-14.el9.noarch 20/33 2024-09-17T06:54:45.125 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 21/33 2024-09-17T06:54:45.313 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 22/33 2024-09-17T06:54:45.369 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 23/33 2024-09-17T06:54:45.446 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 24/33 2024-09-17T06:54:45.535 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 25/33 2024-09-17T06:54:45.576 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 26/33 2024-09-17T06:54:45.691 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 27/33 2024-09-17T06:54:45.860 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-mgr-modules-core-2:18.2.4-855.g19e2a2be.el9.n 28/33 2024-09-17T06:54:45.886 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-mgr-2:18.2.4-855.g19e2a2be.el9.x86_64 29/33 2024-09-17T06:54:45.912 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-mgr-2:18.2.4-855.g19e2a2be.el9.x86_64 29/33 2024-09-17T06:54:45.912 INFO:teuthology.orchestra.run.smithi135.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T06:54:45.913 INFO:teuthology.orchestra.run.smithi135.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-17T06:54:45.913 INFO:teuthology.orchestra.run.smithi135.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-17T06:54:45.913 INFO:teuthology.orchestra.run.smithi135.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-17T06:54:45.913 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:54:46.392 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-2:18.2.4-855.g19e2a2be.el9.x86_64 33/33 2024-09-17T06:54:46.392 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-2:18.2.4-855.g19e2a2be.el9.x86_64 1/33 2024-09-17T06:54:46.393 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-mds-2:18.2.4-855.g19e2a2be.el9.x86_64 2/33 2024-09-17T06:54:46.393 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-mgr-2:18.2.4-855.g19e2a2be.el9.x86_64 3/33 2024-09-17T06:54:46.393 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-mon-2:18.2.4-855.g19e2a2be.el9.x86_64 4/33 2024-09-17T06:54:46.393 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-osd-2:18.2.4-855.g19e2a2be.el9.x86_64 5/33 2024-09-17T06:54:46.393 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-mgr-modules-core-2:18.2.4-855.g19e2a2be.el9.n 6/33 2024-09-17T06:54:46.393 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/33 2024-09-17T06:54:46.393 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/33 2024-09-17T06:54:46.393 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/33 2024-09-17T06:54:46.393 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/33 2024-09-17T06:54:46.393 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/33 2024-09-17T06:54:46.393 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/33 2024-09-17T06:54:46.393 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/33 2024-09-17T06:54:46.393 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/33 2024-09-17T06:54:46.394 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 15/33 2024-09-17T06:54:46.394 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 16/33 2024-09-17T06:54:46.394 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-09-17T06:54:46.394 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-09-17T06:54:46.394 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-09-17T06:54:46.394 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-09-17T06:54:46.394 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-09-17T06:54:46.394 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-09-17T06:54:46.394 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-09-17T06:54:46.394 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-09-17T06:54:46.394 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-09-17T06:54:46.394 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-09-17T06:54:46.394 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-09-17T06:54:46.395 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-09-17T06:54:46.395 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-09-17T06:54:46.395 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-09-17T06:54:46.395 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 31/33 2024-09-17T06:54:46.395 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-09-17T06:54:47.380 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-osd-2:18.2.4-855.g19e2a2be.el9.x86_64 30/33 2024-09-17T06:54:47.381 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-09-17T06:54:47.381 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:54:47.381 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-17T06:54:47.381 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T06:54:47.381 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mds-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T06:54:47.381 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T06:54:47.381 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-modules-core-2:18.2.4-855.g19e2a2be.el9.noarch 2024-09-17T06:54:47.381 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mon-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T06:54:47.381 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-osd-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T06:54:47.381 INFO:teuthology.orchestra.run.smithi100.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-17T06:54:47.382 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-17T06:54:47.382 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-17T06:54:47.382 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-17T06:54:47.382 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-17T06:54:47.382 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-17T06:54:47.382 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-17T06:54:47.382 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-17T06:54:47.382 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-17T06:54:47.382 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-17T06:54:47.382 INFO:teuthology.orchestra.run.smithi100.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-17T06:54:47.382 INFO:teuthology.orchestra.run.smithi100.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-17T06:54:47.382 INFO:teuthology.orchestra.run.smithi100.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-17T06:54:47.382 INFO:teuthology.orchestra.run.smithi100.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-17T06:54:47.383 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-17T06:54:47.383 INFO:teuthology.orchestra.run.smithi100.stdout: python3-ply-3.11-14.el9.noarch 2024-09-17T06:54:47.383 INFO:teuthology.orchestra.run.smithi100.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-17T06:54:47.383 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-17T06:54:47.383 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-17T06:54:47.383 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-17T06:54:47.383 INFO:teuthology.orchestra.run.smithi100.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-17T06:54:47.383 INFO:teuthology.orchestra.run.smithi100.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-17T06:54:47.383 INFO:teuthology.orchestra.run.smithi100.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-17T06:54:47.383 INFO:teuthology.orchestra.run.smithi100.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-17T06:54:47.383 INFO:teuthology.orchestra.run.smithi100.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-17T06:54:47.383 INFO:teuthology.orchestra.run.smithi100.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-17T06:54:47.383 INFO:teuthology.orchestra.run.smithi100.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-17T06:54:47.384 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:54:47.384 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-17T06:54:47.409 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-osd-2:18.2.4-855.g19e2a2be.el9.x86_64 30/33 2024-09-17T06:54:47.409 INFO:teuthology.orchestra.run.smithi135.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T06:54:47.409 INFO:teuthology.orchestra.run.smithi135.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-17T06:54:47.409 INFO:teuthology.orchestra.run.smithi135.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-17T06:54:47.409 INFO:teuthology.orchestra.run.smithi135.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-17T06:54:47.409 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:54:47.699 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph-base 2024-09-17T06:54:47.872 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-mon-2:18.2.4-855.g19e2a2be.el9.x86_64 31/33 2024-09-17T06:54:47.896 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-mon-2:18.2.4-855.g19e2a2be.el9.x86_64 31/33 2024-09-17T06:54:47.897 INFO:teuthology.orchestra.run.smithi135.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T06:54:47.897 INFO:teuthology.orchestra.run.smithi135.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-17T06:54:47.897 INFO:teuthology.orchestra.run.smithi135.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-17T06:54:47.897 INFO:teuthology.orchestra.run.smithi135.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-17T06:54:47.897 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:54:48.098 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-mds-2:18.2.4-855.g19e2a2be.el9.x86_64 32/33 2024-09-17T06:54:48.121 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-mds-2:18.2.4-855.g19e2a2be.el9.x86_64 32/33 2024-09-17T06:54:48.122 INFO:teuthology.orchestra.run.smithi135.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T06:54:48.122 INFO:teuthology.orchestra.run.smithi135.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-17T06:54:48.122 INFO:teuthology.orchestra.run.smithi135.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-17T06:54:48.122 INFO:teuthology.orchestra.run.smithi135.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-17T06:54:48.122 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:54:48.186 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-2:18.2.4-855.g19e2a2be.el9.x86_64 33/33 2024-09-17T06:54:48.288 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:00:48 ago on Tue 17 Sep 2024 06:54:00 AM UTC. 2024-09-17T06:54:48.376 INFO:teuthology.orchestra.run.smithi100.stdout:Package ceph-base-2:18.2.4-855.g19e2a2be.el9.x86_64 is already installed. 2024-09-17T06:54:48.417 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-17T06:54:48.418 INFO:teuthology.orchestra.run.smithi100.stdout:Nothing to do. 2024-09-17T06:54:48.418 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-17T06:54:48.487 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install cephadm 2024-09-17T06:54:49.064 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:00:49 ago on Tue 17 Sep 2024 06:54:00 AM UTC. 2024-09-17T06:54:49.195 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-17T06:54:49.196 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T06:54:49.196 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repository Size 2024-09-17T06:54:49.196 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T06:54:49.196 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-17T06:54:49.196 INFO:teuthology.orchestra.run.smithi100.stdout: cephadm noarch 2:18.2.4-855.g19e2a2be.el9 ceph-noarch 226 k 2024-09-17T06:54:49.196 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:54:49.196 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-17T06:54:49.196 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T06:54:49.196 INFO:teuthology.orchestra.run.smithi100.stdout:Install 1 Package 2024-09-17T06:54:49.196 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:54:49.196 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 226 k 2024-09-17T06:54:49.196 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 217 k 2024-09-17T06:54:49.197 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-17T06:54:49.435 INFO:teuthology.orchestra.run.smithi100.stdout:cephadm-18.2.4-855.g19e2a2be.el9.noarch.rpm 948 kB/s | 226 kB 00:00 2024-09-17T06:54:49.436 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-17T06:54:49.436 INFO:teuthology.orchestra.run.smithi100.stdout:Total 941 kB/s | 226 kB 00:00 2024-09-17T06:54:49.436 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-17T06:54:49.441 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-17T06:54:49.441 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-17T06:54:49.444 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-17T06:54:49.444 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-17T06:54:49.517 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-17T06:54:49.732 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-2:18.2.4-855.g19e2a2be.el9.x86_64 33/33 2024-09-17T06:54:49.733 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-2:18.2.4-855.g19e2a2be.el9.x86_64 1/33 2024-09-17T06:54:49.733 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-mds-2:18.2.4-855.g19e2a2be.el9.x86_64 2/33 2024-09-17T06:54:49.733 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-mgr-2:18.2.4-855.g19e2a2be.el9.x86_64 3/33 2024-09-17T06:54:49.733 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-mon-2:18.2.4-855.g19e2a2be.el9.x86_64 4/33 2024-09-17T06:54:49.733 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-osd-2:18.2.4-855.g19e2a2be.el9.x86_64 5/33 2024-09-17T06:54:49.733 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-mgr-modules-core-2:18.2.4-855.g19e2a2be.el9.n 6/33 2024-09-17T06:54:49.733 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/33 2024-09-17T06:54:49.733 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/33 2024-09-17T06:54:49.733 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/33 2024-09-17T06:54:49.733 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/33 2024-09-17T06:54:49.733 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/33 2024-09-17T06:54:49.733 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/33 2024-09-17T06:54:49.734 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/33 2024-09-17T06:54:49.734 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/33 2024-09-17T06:54:49.734 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 15/33 2024-09-17T06:54:49.734 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 16/33 2024-09-17T06:54:49.734 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-09-17T06:54:49.734 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-09-17T06:54:49.734 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-09-17T06:54:49.734 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-09-17T06:54:49.734 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-09-17T06:54:49.734 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-09-17T06:54:49.734 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-09-17T06:54:49.734 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-09-17T06:54:49.734 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-09-17T06:54:49.735 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-09-17T06:54:49.735 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-09-17T06:54:49.736 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-09-17T06:54:49.736 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-09-17T06:54:49.736 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-09-17T06:54:49.736 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 31/33 2024-09-17T06:54:49.736 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-09-17T06:54:49.991 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: cephadm-2:18.2.4-855.g19e2a2be.el9.noarch 1/1 2024-09-17T06:54:50.080 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : cephadm-2:18.2.4-855.g19e2a2be.el9.noarch 1/1 2024-09-17T06:54:50.466 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: cephadm-2:18.2.4-855.g19e2a2be.el9.noarch 1/1 2024-09-17T06:54:50.582 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-09-17T06:54:50.582 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:54:50.582 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-09-17T06:54:50.582 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T06:54:50.582 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mds-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T06:54:50.583 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T06:54:50.583 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-modules-core-2:18.2.4-855.g19e2a2be.el9.noarch 2024-09-17T06:54:50.583 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mon-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T06:54:50.583 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-osd-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T06:54:50.583 INFO:teuthology.orchestra.run.smithi135.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-17T06:54:50.583 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-17T06:54:50.583 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-17T06:54:50.583 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-17T06:54:50.583 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-17T06:54:50.583 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-17T06:54:50.583 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-17T06:54:50.583 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-17T06:54:50.583 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-17T06:54:50.584 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-17T06:54:50.584 INFO:teuthology.orchestra.run.smithi135.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-17T06:54:50.584 INFO:teuthology.orchestra.run.smithi135.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-17T06:54:50.584 INFO:teuthology.orchestra.run.smithi135.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-17T06:54:50.584 INFO:teuthology.orchestra.run.smithi135.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-17T06:54:50.584 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-17T06:54:50.584 INFO:teuthology.orchestra.run.smithi135.stdout: python3-ply-3.11-14.el9.noarch 2024-09-17T06:54:50.584 INFO:teuthology.orchestra.run.smithi135.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-17T06:54:50.584 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-17T06:54:50.584 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-17T06:54:50.584 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-17T06:54:50.584 INFO:teuthology.orchestra.run.smithi135.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-17T06:54:50.584 INFO:teuthology.orchestra.run.smithi135.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-17T06:54:50.585 INFO:teuthology.orchestra.run.smithi135.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-17T06:54:50.585 INFO:teuthology.orchestra.run.smithi135.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-17T06:54:50.585 INFO:teuthology.orchestra.run.smithi135.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-17T06:54:50.585 INFO:teuthology.orchestra.run.smithi135.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-17T06:54:50.585 INFO:teuthology.orchestra.run.smithi135.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-17T06:54:50.585 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:54:50.585 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-09-17T06:54:50.695 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : cephadm-2:18.2.4-855.g19e2a2be.el9.noarch 1/1 2024-09-17T06:54:50.695 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:54:50.695 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-17T06:54:50.695 INFO:teuthology.orchestra.run.smithi100.stdout: cephadm-2:18.2.4-855.g19e2a2be.el9.noarch 2024-09-17T06:54:50.695 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:54:50.695 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-17T06:54:50.910 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph-immutable-object-cache 2024-09-17T06:54:50.996 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-base 2024-09-17T06:54:51.507 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:00:51 ago on Tue 17 Sep 2024 06:54:00 AM UTC. 2024-09-17T06:54:51.583 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:00:50 ago on Tue 17 Sep 2024 06:54:01 AM UTC. 2024-09-17T06:54:51.643 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-17T06:54:51.643 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T06:54:51.643 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repo Size 2024-09-17T06:54:51.643 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T06:54:51.643 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-17T06:54:51.644 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-immutable-object-cache x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 143 k 2024-09-17T06:54:51.644 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:54:51.644 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-17T06:54:51.644 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T06:54:51.644 INFO:teuthology.orchestra.run.smithi100.stdout:Install 1 Package 2024-09-17T06:54:51.644 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:54:51.645 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 143 k 2024-09-17T06:54:51.645 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 395 k 2024-09-17T06:54:51.645 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-17T06:54:51.672 INFO:teuthology.orchestra.run.smithi135.stdout:Package ceph-base-2:18.2.4-855.g19e2a2be.el9.x86_64 is already installed. 2024-09-17T06:54:51.714 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-09-17T06:54:51.715 INFO:teuthology.orchestra.run.smithi135.stdout:Nothing to do. 2024-09-17T06:54:51.715 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-09-17T06:54:51.786 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install cephadm 2024-09-17T06:54:51.850 INFO:teuthology.orchestra.run.smithi100.stdout:ceph-immutable-object-cache-18.2.4-855.g19e2a2b 695 kB/s | 143 kB 00:00 2024-09-17T06:54:51.851 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-17T06:54:51.851 INFO:teuthology.orchestra.run.smithi100.stdout:Total 690 kB/s | 143 kB 00:00 2024-09-17T06:54:51.852 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-17T06:54:51.859 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-17T06:54:51.859 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-17T06:54:51.918 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-17T06:54:51.919 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-17T06:54:52.071 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-17T06:54:52.134 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-immutable-object-cache-2:18.2.4-855.g19e2a2be.e 1/1 2024-09-17T06:54:52.159 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.4-855.g19e2a2be.e 1/1 2024-09-17T06:54:52.159 INFO:teuthology.orchestra.run.smithi100.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T06:54:52.159 INFO:teuthology.orchestra.run.smithi100.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-17T06:54:52.159 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:54:52.370 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:00:51 ago on Tue 17 Sep 2024 06:54:01 AM UTC. 2024-09-17T06:54:52.503 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-09-17T06:54:52.504 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-09-17T06:54:52.504 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repository Size 2024-09-17T06:54:52.504 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-09-17T06:54:52.505 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-09-17T06:54:52.505 INFO:teuthology.orchestra.run.smithi135.stdout: cephadm noarch 2:18.2.4-855.g19e2a2be.el9 ceph-noarch 226 k 2024-09-17T06:54:52.505 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:54:52.505 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-09-17T06:54:52.505 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-09-17T06:54:52.505 INFO:teuthology.orchestra.run.smithi135.stdout:Install 1 Package 2024-09-17T06:54:52.505 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:54:52.505 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 226 k 2024-09-17T06:54:52.505 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 217 k 2024-09-17T06:54:52.505 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-09-17T06:54:52.739 INFO:teuthology.orchestra.run.smithi135.stdout:cephadm-18.2.4-855.g19e2a2be.el9.noarch.rpm 971 kB/s | 226 kB 00:00 2024-09-17T06:54:52.739 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-09-17T06:54:52.739 INFO:teuthology.orchestra.run.smithi135.stdout:Total 964 kB/s | 226 kB 00:00 2024-09-17T06:54:52.739 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-09-17T06:54:52.745 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-09-17T06:54:52.745 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-09-17T06:54:52.747 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-09-17T06:54:52.748 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-09-17T06:54:52.819 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-09-17T06:54:53.057 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-immutable-object-cache-2:18.2.4-855.g19e2a2be.e 1/1 2024-09-17T06:54:53.057 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:54:53.057 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-17T06:54:53.057 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-immutable-object-cache-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T06:54:53.057 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:54:53.058 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-17T06:54:53.259 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: cephadm-2:18.2.4-855.g19e2a2be.el9.noarch 1/1 2024-09-17T06:54:53.278 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph-mgr 2024-09-17T06:54:53.350 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : cephadm-2:18.2.4-855.g19e2a2be.el9.noarch 1/1 2024-09-17T06:54:53.684 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: cephadm-2:18.2.4-855.g19e2a2be.el9.noarch 1/1 2024-09-17T06:54:53.873 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:00:53 ago on Tue 17 Sep 2024 06:54:00 AM UTC. 2024-09-17T06:54:53.899 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : cephadm-2:18.2.4-855.g19e2a2be.el9.noarch 1/1 2024-09-17T06:54:53.899 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:54:53.899 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-09-17T06:54:53.899 INFO:teuthology.orchestra.run.smithi135.stdout: cephadm-2:18.2.4-855.g19e2a2be.el9.noarch 2024-09-17T06:54:53.899 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:54:53.899 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-09-17T06:54:53.969 INFO:teuthology.orchestra.run.smithi100.stdout:Package ceph-mgr-2:18.2.4-855.g19e2a2be.el9.x86_64 is already installed. 2024-09-17T06:54:54.013 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-17T06:54:54.015 INFO:teuthology.orchestra.run.smithi100.stdout:Nothing to do. 2024-09-17T06:54:54.015 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-17T06:54:54.086 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-immutable-object-cache 2024-09-17T06:54:54.087 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph-mgr-dashboard 2024-09-17T06:54:54.666 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:00:54 ago on Tue 17 Sep 2024 06:54:00 AM UTC. 2024-09-17T06:54:54.687 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:00:53 ago on Tue 17 Sep 2024 06:54:01 AM UTC. 2024-09-17T06:54:54.796 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-17T06:54:54.798 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T06:54:54.798 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repository Size 2024-09-17T06:54:54.798 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T06:54:54.798 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-17T06:54:54.798 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-dashboard noarch 2:18.2.4-855.g19e2a2be.el9 ceph-noarch 3.5 M 2024-09-17T06:54:54.799 INFO:teuthology.orchestra.run.smithi100.stdout:Installing dependencies: 2024-09-17T06:54:54.799 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-grafana-dashboards noarch 2:18.2.4-855.g19e2a2be.el9 ceph-noarch 24 k 2024-09-17T06:54:54.799 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-prometheus-alerts noarch 2:18.2.4-855.g19e2a2be.el9 ceph-noarch 16 k 2024-09-17T06:54:54.799 INFO:teuthology.orchestra.run.smithi100.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-17T06:54:54.799 INFO:teuthology.orchestra.run.smithi100.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-17T06:54:54.800 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:54:54.800 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-17T06:54:54.800 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T06:54:54.800 INFO:teuthology.orchestra.run.smithi100.stdout:Install 5 Packages 2024-09-17T06:54:54.800 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:54:54.800 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 3.8 M 2024-09-17T06:54:54.801 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 87 M 2024-09-17T06:54:54.801 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-17T06:54:54.825 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-09-17T06:54:54.825 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-09-17T06:54:54.825 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repo Size 2024-09-17T06:54:54.825 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-09-17T06:54:54.825 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-09-17T06:54:54.825 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-immutable-object-cache x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 143 k 2024-09-17T06:54:54.825 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:54:54.825 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-09-17T06:54:54.826 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-09-17T06:54:54.826 INFO:teuthology.orchestra.run.smithi135.stdout:Install 1 Package 2024-09-17T06:54:54.826 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:54:54.826 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 143 k 2024-09-17T06:54:54.826 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 395 k 2024-09-17T06:54:54.826 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-09-17T06:54:55.044 INFO:teuthology.orchestra.run.smithi135.stdout:ceph-immutable-object-cache-18.2.4-855.g19e2a2b 653 kB/s | 143 kB 00:00 2024-09-17T06:54:55.045 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-09-17T06:54:55.045 INFO:teuthology.orchestra.run.smithi135.stdout:Total 648 kB/s | 143 kB 00:00 2024-09-17T06:54:55.046 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-09-17T06:54:55.052 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-09-17T06:54:55.052 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-09-17T06:54:55.110 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-09-17T06:54:55.110 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-09-17T06:54:55.129 INFO:teuthology.orchestra.run.smithi100.stdout:(1/5): ceph-grafana-dashboards-18.2.4-855.g19e2 155 kB/s | 24 kB 00:00 2024-09-17T06:54:55.154 INFO:teuthology.orchestra.run.smithi100.stdout:(2/5): ceph-prometheus-alerts-18.2.4-855.g19e2a 90 kB/s | 16 kB 00:00 2024-09-17T06:54:55.205 INFO:teuthology.orchestra.run.smithi100.stdout:(3/5): python3-repoze-lru-0.7-16.el9.noarch.rpm 408 kB/s | 31 kB 00:00 2024-09-17T06:54:55.238 INFO:teuthology.orchestra.run.smithi100.stdout:(4/5): python3-routes-2.5.1-5.el9.noarch.rpm 2.2 MB/s | 188 kB 00:00 2024-09-17T06:54:55.262 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-09-17T06:54:55.328 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-immutable-object-cache-2:18.2.4-855.g19e2a2be.e 1/1 2024-09-17T06:54:55.352 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.4-855.g19e2a2be.e 1/1 2024-09-17T06:54:55.353 INFO:teuthology.orchestra.run.smithi135.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T06:54:55.353 INFO:teuthology.orchestra.run.smithi135.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-17T06:54:55.353 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:54:55.388 INFO:teuthology.orchestra.run.smithi100.stdout:(5/5): ceph-mgr-dashboard-18.2.4-855.g19e2a2be. 8.5 MB/s | 3.5 MB 00:00 2024-09-17T06:54:55.391 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-17T06:54:55.392 INFO:teuthology.orchestra.run.smithi100.stdout:Total 6.4 MB/s | 3.8 MB 00:00 2024-09-17T06:54:55.392 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-17T06:54:55.399 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-17T06:54:55.399 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-17T06:54:55.482 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-17T06:54:55.482 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-17T06:54:55.756 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-17T06:54:55.835 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/5 2024-09-17T06:54:55.868 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/5 2024-09-17T06:54:55.889 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-prometheus-alerts-2:18.2.4-855.g19e2a2be.el9.no 3/5 2024-09-17T06:54:56.310 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-immutable-object-cache-2:18.2.4-855.g19e2a2be.e 1/1 2024-09-17T06:54:56.310 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:54:56.310 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-09-17T06:54:56.310 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-immutable-object-cache-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T06:54:56.310 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:54:56.311 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-09-17T06:54:56.505 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-mgr 2024-09-17T06:54:56.935 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-grafana-dashboards-2:18.2.4-855.g19e2a2be.el9.n 4/5 2024-09-17T06:54:56.984 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-mgr-dashboard-2:18.2.4-855.g19e2a2be.el9.noarch 5/5 2024-09-17T06:54:57.116 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:00:56 ago on Tue 17 Sep 2024 06:54:01 AM UTC. 2024-09-17T06:54:57.208 INFO:teuthology.orchestra.run.smithi135.stdout:Package ceph-mgr-2:18.2.4-855.g19e2a2be.el9.x86_64 is already installed. 2024-09-17T06:54:57.250 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-09-17T06:54:57.251 INFO:teuthology.orchestra.run.smithi135.stdout:Nothing to do. 2024-09-17T06:54:57.251 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-09-17T06:54:57.324 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-mgr-dashboard 2024-09-17T06:54:57.921 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:00:56 ago on Tue 17 Sep 2024 06:54:01 AM UTC. 2024-09-17T06:54:58.055 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-09-17T06:54:58.056 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-09-17T06:54:58.056 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repository Size 2024-09-17T06:54:58.056 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-09-17T06:54:58.056 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-09-17T06:54:58.056 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-dashboard noarch 2:18.2.4-855.g19e2a2be.el9 ceph-noarch 3.5 M 2024-09-17T06:54:58.056 INFO:teuthology.orchestra.run.smithi135.stdout:Installing dependencies: 2024-09-17T06:54:58.056 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-grafana-dashboards noarch 2:18.2.4-855.g19e2a2be.el9 ceph-noarch 24 k 2024-09-17T06:54:58.056 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-prometheus-alerts noarch 2:18.2.4-855.g19e2a2be.el9 ceph-noarch 16 k 2024-09-17T06:54:58.056 INFO:teuthology.orchestra.run.smithi135.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-17T06:54:58.057 INFO:teuthology.orchestra.run.smithi135.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-17T06:54:58.057 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:54:58.057 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-09-17T06:54:58.057 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-09-17T06:54:58.057 INFO:teuthology.orchestra.run.smithi135.stdout:Install 5 Packages 2024-09-17T06:54:58.057 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:54:58.057 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 3.8 M 2024-09-17T06:54:58.058 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 87 M 2024-09-17T06:54:58.058 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-09-17T06:54:58.165 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.4-855.g19e2a2be.el9.noarch 5/5 2024-09-17T06:54:58.165 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-grafana-dashboards-2:18.2.4-855.g19e2a2be.el9.n 1/5 2024-09-17T06:54:58.165 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-mgr-dashboard-2:18.2.4-855.g19e2a2be.el9.noarch 2/5 2024-09-17T06:54:58.166 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-prometheus-alerts-2:18.2.4-855.g19e2a2be.el9.no 3/5 2024-09-17T06:54:58.166 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 4/5 2024-09-17T06:54:58.448 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 5/5 2024-09-17T06:54:58.448 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:54:58.448 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-17T06:54:58.448 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-grafana-dashboards-2:18.2.4-855.g19e2a2be.el9.noarch 2024-09-17T06:54:58.448 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-dashboard-2:18.2.4-855.g19e2a2be.el9.noarch 2024-09-17T06:54:58.448 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-prometheus-alerts-2:18.2.4-855.g19e2a2be.el9.noarch 2024-09-17T06:54:58.448 INFO:teuthology.orchestra.run.smithi100.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-17T06:54:58.448 INFO:teuthology.orchestra.run.smithi100.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-17T06:54:58.449 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:54:58.449 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-17T06:54:58.539 INFO:teuthology.orchestra.run.smithi135.stdout:(1/5): ceph-grafana-dashboards-18.2.4-855.g19e2 173 kB/s | 24 kB 00:00 2024-09-17T06:54:58.564 INFO:teuthology.orchestra.run.smithi135.stdout:(2/5): ceph-prometheus-alerts-18.2.4-855.g19e2a 99 kB/s | 16 kB 00:00 2024-09-17T06:54:58.606 INFO:teuthology.orchestra.run.smithi135.stdout:(3/5): python3-repoze-lru-0.7-16.el9.noarch.rpm 461 kB/s | 31 kB 00:00 2024-09-17T06:54:58.631 INFO:teuthology.orchestra.run.smithi135.stdout:(4/5): python3-routes-2.5.1-5.el9.noarch.rpm 2.8 MB/s | 188 kB 00:00 2024-09-17T06:54:58.667 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-17T06:54:58.798 INFO:teuthology.orchestra.run.smithi135.stdout:(5/5): ceph-mgr-dashboard-18.2.4-855.g19e2a2be. 8.8 MB/s | 3.5 MB 00:00 2024-09-17T06:54:58.802 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-09-17T06:54:58.802 INFO:teuthology.orchestra.run.smithi135.stdout:Total 5.1 MB/s | 3.8 MB 00:00 2024-09-17T06:54:58.802 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-09-17T06:54:58.809 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-09-17T06:54:58.809 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-09-17T06:54:58.894 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-09-17T06:54:58.894 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-09-17T06:54:59.151 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-09-17T06:54:59.229 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/5 2024-09-17T06:54:59.240 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:00:59 ago on Tue 17 Sep 2024 06:54:00 AM UTC. 2024-09-17T06:54:59.261 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/5 2024-09-17T06:54:59.290 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-prometheus-alerts-2:18.2.4-855.g19e2a2be.el9.no 3/5 2024-09-17T06:54:59.371 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-17T06:54:59.372 INFO:teuthology.orchestra.run.smithi100.stdout:====================================================================================== 2024-09-17T06:54:59.372 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repository Size 2024-09-17T06:54:59.373 INFO:teuthology.orchestra.run.smithi100.stdout:====================================================================================== 2024-09-17T06:54:59.373 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-17T06:54:59.373 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.4-855.g19e2a2be.el9 ceph-noarch 7.4 M 2024-09-17T06:54:59.373 INFO:teuthology.orchestra.run.smithi100.stdout:Installing dependencies: 2024-09-17T06:54:59.373 INFO:teuthology.orchestra.run.smithi100.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-17T06:54:59.373 INFO:teuthology.orchestra.run.smithi100.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-17T06:54:59.373 INFO:teuthology.orchestra.run.smithi100.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-17T06:54:59.373 INFO:teuthology.orchestra.run.smithi100.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-17T06:54:59.373 INFO:teuthology.orchestra.run.smithi100.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-17T06:54:59.373 INFO:teuthology.orchestra.run.smithi100.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-17T06:54:59.373 INFO:teuthology.orchestra.run.smithi100.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-17T06:54:59.373 INFO:teuthology.orchestra.run.smithi100.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-17T06:54:59.374 INFO:teuthology.orchestra.run.smithi100.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-17T06:54:59.374 INFO:teuthology.orchestra.run.smithi100.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-17T06:54:59.374 INFO:teuthology.orchestra.run.smithi100.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-17T06:54:59.374 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:54:59.374 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-17T06:54:59.374 INFO:teuthology.orchestra.run.smithi100.stdout:====================================================================================== 2024-09-17T06:54:59.374 INFO:teuthology.orchestra.run.smithi100.stdout:Install 12 Packages 2024-09-17T06:54:59.374 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:54:59.375 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 38 M 2024-09-17T06:54:59.375 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 212 M 2024-09-17T06:54:59.375 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-17T06:54:59.604 INFO:teuthology.orchestra.run.smithi100.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.3 MB/s | 191 kB 00:00 2024-09-17T06:54:59.654 INFO:teuthology.orchestra.run.smithi100.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 4.1 MB/s | 801 kB 00:00 2024-09-17T06:54:59.946 INFO:teuthology.orchestra.run.smithi100.stdout:(3/12): ceph-mgr-diskprediction-local-18.2.4-85 15 MB/s | 7.4 MB 00:00 2024-09-17T06:54:59.972 INFO:teuthology.orchestra.run.smithi100.stdout:(4/12): flexiblas-3.0.4-8.el9.x86_64.rpm 89 kB/s | 33 kB 00:00 2024-09-17T06:55:00.172 INFO:teuthology.orchestra.run.smithi100.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 192 kB/s | 38 kB 00:00 2024-09-17T06:55:00.222 INFO:teuthology.orchestra.run.smithi100.stdout:(6/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 66 kB/s | 18 kB 00:00 2024-09-17T06:55:00.344 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-grafana-dashboards-2:18.2.4-855.g19e2a2be.el9.n 4/5 2024-09-17T06:55:00.377 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-mgr-dashboard-2:18.2.4-855.g19e2a2be.el9.noarch 5/5 2024-09-17T06:55:00.689 INFO:teuthology.orchestra.run.smithi100.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 524 kB/s | 245 kB 00:00 2024-09-17T06:55:00.864 INFO:teuthology.orchestra.run.smithi100.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.5 MB/s | 3.0 MB 00:01 2024-09-17T06:55:00.981 INFO:teuthology.orchestra.run.smithi100.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.1 MB/s | 250 kB 00:00 2024-09-17T06:55:01.265 INFO:teuthology.orchestra.run.smithi100.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 4.5 MB/s | 4.9 MB 00:01 2024-09-17T06:55:01.484 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.4-855.g19e2a2be.el9.noarch 5/5 2024-09-17T06:55:01.484 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-grafana-dashboards-2:18.2.4-855.g19e2a2be.el9.n 1/5 2024-09-17T06:55:01.485 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-mgr-dashboard-2:18.2.4-855.g19e2a2be.el9.noarch 2/5 2024-09-17T06:55:01.485 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-prometheus-alerts-2:18.2.4-855.g19e2a2be.el9.no 3/5 2024-09-17T06:55:01.485 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 4/5 2024-09-17T06:55:01.746 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 5/5 2024-09-17T06:55:01.746 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:55:01.746 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-09-17T06:55:01.746 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-grafana-dashboards-2:18.2.4-855.g19e2a2be.el9.noarch 2024-09-17T06:55:01.746 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-dashboard-2:18.2.4-855.g19e2a2be.el9.noarch 2024-09-17T06:55:01.747 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-prometheus-alerts-2:18.2.4-855.g19e2a2be.el9.noarch 2024-09-17T06:55:01.747 INFO:teuthology.orchestra.run.smithi135.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-17T06:55:01.747 INFO:teuthology.orchestra.run.smithi135.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-17T06:55:01.747 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:55:01.747 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-09-17T06:55:01.965 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-17T06:55:02.557 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:01:01 ago on Tue 17 Sep 2024 06:54:01 AM UTC. 2024-09-17T06:55:02.691 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-09-17T06:55:02.692 INFO:teuthology.orchestra.run.smithi135.stdout:====================================================================================== 2024-09-17T06:55:02.692 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repository Size 2024-09-17T06:55:02.692 INFO:teuthology.orchestra.run.smithi135.stdout:====================================================================================== 2024-09-17T06:55:02.692 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-09-17T06:55:02.692 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.4-855.g19e2a2be.el9 ceph-noarch 7.4 M 2024-09-17T06:55:02.692 INFO:teuthology.orchestra.run.smithi135.stdout:Installing dependencies: 2024-09-17T06:55:02.692 INFO:teuthology.orchestra.run.smithi135.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-17T06:55:02.693 INFO:teuthology.orchestra.run.smithi135.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-17T06:55:02.693 INFO:teuthology.orchestra.run.smithi135.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-17T06:55:02.693 INFO:teuthology.orchestra.run.smithi135.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-17T06:55:02.693 INFO:teuthology.orchestra.run.smithi135.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-17T06:55:02.693 INFO:teuthology.orchestra.run.smithi135.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-17T06:55:02.693 INFO:teuthology.orchestra.run.smithi135.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-17T06:55:02.693 INFO:teuthology.orchestra.run.smithi135.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-17T06:55:02.693 INFO:teuthology.orchestra.run.smithi135.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-17T06:55:02.693 INFO:teuthology.orchestra.run.smithi135.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-17T06:55:02.693 INFO:teuthology.orchestra.run.smithi135.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-17T06:55:02.693 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:55:02.693 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-09-17T06:55:02.694 INFO:teuthology.orchestra.run.smithi135.stdout:====================================================================================== 2024-09-17T06:55:02.694 INFO:teuthology.orchestra.run.smithi135.stdout:Install 12 Packages 2024-09-17T06:55:02.694 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:55:02.694 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 38 M 2024-09-17T06:55:02.694 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 212 M 2024-09-17T06:55:02.694 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-09-17T06:55:02.823 INFO:teuthology.orchestra.run.smithi100.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 2.4 MB/s | 5.1 MB 00:02 2024-09-17T06:55:03.205 INFO:teuthology.orchestra.run.smithi135.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 908 kB/s | 191 kB 00:00 2024-09-17T06:55:03.297 INFO:teuthology.orchestra.run.smithi135.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 2.6 MB/s | 801 kB 00:00 2024-09-17T06:55:03.456 INFO:teuthology.orchestra.run.smithi135.stdout:(3/12): ceph-mgr-diskprediction-local-18.2.4-85 16 MB/s | 7.4 MB 00:00 2024-09-17T06:55:03.864 INFO:teuthology.orchestra.run.smithi135.stdout:(4/12): flexiblas-3.0.4-8.el9.x86_64.rpm 50 kB/s | 33 kB 00:00 2024-09-17T06:55:03.989 INFO:teuthology.orchestra.run.smithi135.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 34 kB/s | 18 kB 00:00 2024-09-17T06:55:04.423 INFO:teuthology.orchestra.run.smithi135.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.7 MB/s | 3.0 MB 00:01 2024-09-17T06:55:04.448 INFO:teuthology.orchestra.run.smithi135.stdout:(7/12): openblas-0.3.26-2.el9.x86_64.rpm 66 kB/s | 38 kB 00:00 2024-09-17T06:55:05.090 INFO:teuthology.orchestra.run.smithi135.stdout:(8/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 4.5 MB/s | 4.9 MB 00:01 2024-09-17T06:55:05.240 INFO:teuthology.orchestra.run.smithi135.stdout:(9/12): python3-devel-3.9.19-8.el9.x86_64.rpm 300 kB/s | 245 kB 00:00 2024-09-17T06:55:05.574 INFO:teuthology.orchestra.run.smithi135.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 4.5 MB/s | 5.1 MB 00:01 2024-09-17T06:55:05.882 INFO:teuthology.orchestra.run.smithi135.stdout:(11/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 317 kB/s | 250 kB 00:00 2024-09-17T06:55:06.708 INFO:teuthology.orchestra.run.smithi135.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 11 MB/s | 16 MB 00:01 2024-09-17T06:55:06.709 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-09-17T06:55:06.709 INFO:teuthology.orchestra.run.smithi135.stdout:Total 9.5 MB/s | 38 MB 00:04 2024-09-17T06:55:07.070 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-09-17T06:55:07.097 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-09-17T06:55:07.097 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-09-17T06:55:07.614 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-09-17T06:55:07.614 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-09-17T06:55:08.380 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-09-17T06:55:08.462 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-17T06:55:08.515 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-17T06:55:08.614 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-17T06:55:08.656 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-17T06:55:08.840 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-17T06:55:08.872 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-17T06:55:08.958 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-17T06:55:09.600 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-17T06:55:09.727 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-17T06:55:10.975 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-17T06:55:12.081 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-17T06:55:12.113 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.4-855.g19e2a2 12/12 2024-09-17T06:55:14.657 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.4-855.g19e2a2 12/12 2024-09-17T06:55:14.657 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.4-855.g19e2a2 1/12 2024-09-17T06:55:14.657 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-17T06:55:14.658 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-17T06:55:14.658 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-17T06:55:14.658 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-17T06:55:14.658 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-17T06:55:14.658 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-17T06:55:14.659 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-17T06:55:14.659 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-17T06:55:14.659 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-17T06:55:14.659 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-17T06:55:15.047 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-17T06:55:15.047 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:55:15.047 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-09-17T06:55:15.048 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-diskprediction-local-2:18.2.4-855.g19e2a2be.el9.noarch 2024-09-17T06:55:15.048 INFO:teuthology.orchestra.run.smithi135.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-17T06:55:15.048 INFO:teuthology.orchestra.run.smithi135.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-17T06:55:15.048 INFO:teuthology.orchestra.run.smithi135.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-17T06:55:15.048 INFO:teuthology.orchestra.run.smithi135.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-17T06:55:15.048 INFO:teuthology.orchestra.run.smithi135.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-17T06:55:15.048 INFO:teuthology.orchestra.run.smithi135.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-17T06:55:15.048 INFO:teuthology.orchestra.run.smithi135.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-17T06:55:15.048 INFO:teuthology.orchestra.run.smithi135.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-17T06:55:15.048 INFO:teuthology.orchestra.run.smithi135.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-17T06:55:15.048 INFO:teuthology.orchestra.run.smithi135.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-17T06:55:15.049 INFO:teuthology.orchestra.run.smithi135.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-17T06:55:15.049 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:55:15.049 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-09-17T06:55:15.416 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-mgr-rook 2024-09-17T06:55:16.017 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:01:15 ago on Tue 17 Sep 2024 06:54:01 AM UTC. 2024-09-17T06:55:16.150 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-09-17T06:55:16.152 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-09-17T06:55:16.152 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repository Size 2024-09-17T06:55:16.152 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-09-17T06:55:16.152 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-09-17T06:55:16.152 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-rook noarch 2:18.2.4-855.g19e2a2be.el9 ceph-noarch 49 k 2024-09-17T06:55:16.152 INFO:teuthology.orchestra.run.smithi135.stdout:Installing dependencies: 2024-09-17T06:55:16.152 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-17T06:55:16.153 INFO:teuthology.orchestra.run.smithi135.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-17T06:55:16.153 INFO:teuthology.orchestra.run.smithi135.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-17T06:55:16.153 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-17T06:55:16.153 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-17T06:55:16.153 INFO:teuthology.orchestra.run.smithi135.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-17T06:55:16.153 INFO:teuthology.orchestra.run.smithi135.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-17T06:55:16.153 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-17T06:55:16.153 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-17T06:55:16.153 INFO:teuthology.orchestra.run.smithi135.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-17T06:55:16.153 INFO:teuthology.orchestra.run.smithi135.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-17T06:55:16.153 INFO:teuthology.orchestra.run.smithi135.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-17T06:55:16.153 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:55:16.153 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-09-17T06:55:16.154 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-09-17T06:55:16.154 INFO:teuthology.orchestra.run.smithi135.stdout:Install 13 Packages 2024-09-17T06:55:16.154 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:55:16.154 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 2.2 M 2024-09-17T06:55:16.154 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 26 M 2024-09-17T06:55:16.155 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-09-17T06:55:16.787 INFO:teuthology.orchestra.run.smithi135.stdout:(1/13): ceph-mgr-rook-18.2.4-855.g19e2a2be.el9. 298 kB/s | 49 kB 00:00 2024-09-17T06:55:17.138 INFO:teuthology.orchestra.run.smithi135.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 37 kB/s | 19 kB 00:00 2024-09-17T06:55:17.180 INFO:teuthology.orchestra.run.smithi135.stdout:(3/13): python3-jsonpatch-1.21-16.el9.noarch.rp 46 kB/s | 26 kB 00:00 2024-09-17T06:55:17.613 INFO:teuthology.orchestra.run.smithi135.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 269 kB/s | 222 kB 00:00 2024-09-17T06:55:17.855 INFO:teuthology.orchestra.run.smithi135.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 221 kB/s | 159 kB 00:00 2024-09-17T06:55:17.897 INFO:teuthology.orchestra.run.smithi135.stdout:(6/13): python3-cachetools-4.2.4-1.el9.noarch.r 783 kB/s | 32 kB 00:00 2024-09-17T06:55:17.914 INFO:teuthology.orchestra.run.smithi135.stdout:(7/13): python3-certifi-2023.05.07-4.el9.noarch 851 kB/s | 14 kB 00:00 2024-09-17T06:55:17.939 INFO:teuthology.orchestra.run.smithi135.stdout:(8/13): python3-google-auth-2.34.0-1.el9.noarch 8.8 MB/s | 220 kB 00:00 2024-09-17T06:55:17.965 INFO:teuthology.orchestra.run.smithi135.stdout:(9/13): python3-pyasn1-modules-0.4.8-6.el9.noar 355 kB/s | 279 kB 00:00 2024-09-17T06:55:18.015 INFO:teuthology.orchestra.run.smithi135.stdout:(10/13): python3-kubernetes-26.1.0-2.el9.noarch 14 MB/s | 1.0 MB 00:00 2024-09-17T06:55:18.040 INFO:teuthology.orchestra.run.smithi135.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 3.8 MB/s | 90 kB 00:00 2024-09-17T06:55:18.065 INFO:teuthology.orchestra.run.smithi135.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 593 kB/s | 59 kB 00:00 2024-09-17T06:55:18.357 INFO:teuthology.orchestra.run.smithi135.stdout:(13/13): python3-requests-oauthlib-1.3.0-12.el9 72 kB/s | 54 kB 00:00 2024-09-17T06:55:18.360 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-09-17T06:55:18.360 INFO:teuthology.orchestra.run.smithi135.stdout:Total 1.0 MB/s | 2.2 MB 00:02 2024-09-17T06:55:18.427 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-09-17T06:55:18.443 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-09-17T06:55:18.443 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-09-17T06:55:18.537 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-09-17T06:55:18.537 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-09-17T06:55:19.039 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-09-17T06:55:19.203 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-17T06:55:19.266 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-17T06:55:19.323 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-17T06:55:19.359 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-17T06:55:19.400 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-17T06:55:19.511 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-17T06:55:19.621 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-17T06:55:19.677 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-17T06:55:20.396 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-17T06:55:20.441 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-17T06:55:20.475 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-17T06:55:20.523 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-17T06:55:20.545 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-mgr-rook-2:18.2.4-855.g19e2a2be.el9.noarch 13/13 2024-09-17T06:55:21.401 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.4-855.g19e2a2be.el9.noarch 13/13 2024-09-17T06:55:21.401 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-mgr-rook-2:18.2.4-855.g19e2a2be.el9.noarch 1/13 2024-09-17T06:55:21.401 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-17T06:55:21.401 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-17T06:55:21.402 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-17T06:55:21.402 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-17T06:55:21.402 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-17T06:55:21.402 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-17T06:55:21.402 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-17T06:55:21.402 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-17T06:55:21.402 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-17T06:55:21.402 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-17T06:55:21.402 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-17T06:55:21.738 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-17T06:55:21.738 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:55:21.738 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-09-17T06:55:21.738 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-rook-2:18.2.4-855.g19e2a2be.el9.noarch 2024-09-17T06:55:21.738 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-17T06:55:21.738 INFO:teuthology.orchestra.run.smithi135.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-17T06:55:21.738 INFO:teuthology.orchestra.run.smithi135.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-17T06:55:21.738 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-17T06:55:21.739 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-17T06:55:21.739 INFO:teuthology.orchestra.run.smithi135.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-17T06:55:21.739 INFO:teuthology.orchestra.run.smithi135.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-17T06:55:21.739 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-17T06:55:21.739 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-17T06:55:21.739 INFO:teuthology.orchestra.run.smithi135.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-17T06:55:21.739 INFO:teuthology.orchestra.run.smithi135.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-17T06:55:21.739 INFO:teuthology.orchestra.run.smithi135.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-17T06:55:21.739 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:55:21.739 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-09-17T06:55:21.954 INFO:teuthology.orchestra.run.smithi100.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 793 kB/s | 16 MB 00:20 2024-09-17T06:55:21.955 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-17T06:55:21.955 INFO:teuthology.orchestra.run.smithi100.stdout:Total 1.7 MB/s | 38 MB 00:22 2024-09-17T06:55:21.991 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-mgr-cephadm 2024-09-17T06:55:22.316 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-17T06:55:22.342 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-17T06:55:22.342 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-17T06:55:22.575 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:01:21 ago on Tue 17 Sep 2024 06:54:01 AM UTC. 2024-09-17T06:55:22.707 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-09-17T06:55:22.707 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-09-17T06:55:22.707 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repository Size 2024-09-17T06:55:22.708 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-09-17T06:55:22.708 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-09-17T06:55:22.708 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-cephadm noarch 2:18.2.4-855.g19e2a2be.el9 ceph-noarch 139 k 2024-09-17T06:55:22.708 INFO:teuthology.orchestra.run.smithi135.stdout:Installing dependencies: 2024-09-17T06:55:22.708 INFO:teuthology.orchestra.run.smithi135.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-17T06:55:22.708 INFO:teuthology.orchestra.run.smithi135.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-17T06:55:22.708 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-17T06:55:22.708 INFO:teuthology.orchestra.run.smithi135.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-17T06:55:22.708 INFO:teuthology.orchestra.run.smithi135.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-17T06:55:22.708 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:55:22.708 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-09-17T06:55:22.709 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-09-17T06:55:22.709 INFO:teuthology.orchestra.run.smithi135.stdout:Install 6 Packages 2024-09-17T06:55:22.709 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:55:22.709 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 7.0 M 2024-09-17T06:55:22.709 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 33 M 2024-09-17T06:55:22.709 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-09-17T06:55:22.857 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-17T06:55:22.857 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-17T06:55:22.893 INFO:teuthology.orchestra.run.smithi135.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.8 MB/s | 248 kB 00:00 2024-09-17T06:55:22.943 INFO:teuthology.orchestra.run.smithi135.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 11 MB/s | 548 kB 00:00 2024-09-17T06:55:22.969 INFO:teuthology.orchestra.run.smithi135.stdout:(3/6): ceph-mgr-cephadm-18.2.4-855.g19e2a2be.el 663 kB/s | 139 kB 00:00 2024-09-17T06:55:22.994 INFO:teuthology.orchestra.run.smithi135.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.1 MB/s | 58 kB 00:00 2024-09-17T06:55:23.036 INFO:teuthology.orchestra.run.smithi135.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.1 MB/s | 74 kB 00:00 2024-09-17T06:55:23.119 INFO:teuthology.orchestra.run.smithi135.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 17 MB/s | 6.0 MB 00:00 2024-09-17T06:55:23.122 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-09-17T06:55:23.122 INFO:teuthology.orchestra.run.smithi135.stdout:Total 17 MB/s | 7.0 MB 00:00 2024-09-17T06:55:23.192 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-09-17T06:55:23.203 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-09-17T06:55:23.204 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-09-17T06:55:23.305 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-09-17T06:55:23.305 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-09-17T06:55:23.612 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-09-17T06:55:23.641 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-17T06:55:23.719 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-17T06:55:23.760 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-17T06:55:23.776 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-17T06:55:23.829 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-17T06:55:23.862 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-17T06:55:23.903 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-17T06:55:24.089 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-17T06:55:24.118 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-17T06:55:24.206 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-17T06:55:24.359 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-17T06:55:24.457 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-17T06:55:24.519 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-17T06:55:24.550 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-mgr-cephadm-2:18.2.4-855.g19e2a2be.el9.noarch 6/6 2024-09-17T06:55:24.861 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-17T06:55:24.983 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-17T06:55:25.301 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.4-855.g19e2a2be.el9.noarch 6/6 2024-09-17T06:55:25.301 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-mgr-cephadm-2:18.2.4-855.g19e2a2be.el9.noarch 1/6 2024-09-17T06:55:25.302 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-17T06:55:25.302 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-17T06:55:25.302 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-17T06:55:25.302 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-17T06:55:25.612 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-17T06:55:25.612 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:55:25.612 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-09-17T06:55:25.612 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-cephadm-2:18.2.4-855.g19e2a2be.el9.noarch 2024-09-17T06:55:25.613 INFO:teuthology.orchestra.run.smithi135.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-17T06:55:25.613 INFO:teuthology.orchestra.run.smithi135.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-17T06:55:25.613 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-17T06:55:25.613 INFO:teuthology.orchestra.run.smithi135.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-17T06:55:25.613 INFO:teuthology.orchestra.run.smithi135.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-17T06:55:25.613 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:55:25.613 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-09-17T06:55:25.870 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-fuse 2024-09-17T06:55:26.236 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-17T06:55:26.462 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:01:25 ago on Tue 17 Sep 2024 06:54:01 AM UTC. 2024-09-17T06:55:26.595 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-09-17T06:55:26.596 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-09-17T06:55:26.596 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repository Size 2024-09-17T06:55:26.596 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-09-17T06:55:26.596 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-09-17T06:55:26.596 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-fuse x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 852 k 2024-09-17T06:55:26.596 INFO:teuthology.orchestra.run.smithi135.stdout:Installing dependencies: 2024-09-17T06:55:26.596 INFO:teuthology.orchestra.run.smithi135.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-17T06:55:26.596 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:55:26.596 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-09-17T06:55:26.597 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-09-17T06:55:26.597 INFO:teuthology.orchestra.run.smithi135.stdout:Install 2 Packages 2024-09-17T06:55:26.597 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:55:26.597 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 932 k 2024-09-17T06:55:26.597 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 2.7 M 2024-09-17T06:55:26.597 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-09-17T06:55:26.897 INFO:teuthology.orchestra.run.smithi135.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 450 kB/s | 80 kB 00:00 2024-09-17T06:55:26.996 INFO:teuthology.orchestra.run.smithi135.stdout:(2/2): ceph-fuse-18.2.4-855.g19e2a2be.el9.x86_6 3.0 MB/s | 852 kB 00:00 2024-09-17T06:55:26.996 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-09-17T06:55:26.997 INFO:teuthology.orchestra.run.smithi135.stdout:Total 2.3 MB/s | 932 kB 00:00 2024-09-17T06:55:27.008 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-09-17T06:55:27.031 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-09-17T06:55:27.031 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-09-17T06:55:27.111 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-09-17T06:55:27.111 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-09-17T06:55:27.275 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-17T06:55:27.303 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-09-17T06:55:27.310 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.4-855.g19e2a2 12/12 2024-09-17T06:55:27.427 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-17T06:55:27.519 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-fuse-2:18.2.4-855.g19e2a2be.el9.x86_64 2/2 2024-09-17T06:55:28.189 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-fuse-2:18.2.4-855.g19e2a2be.el9.x86_64 2/2 2024-09-17T06:55:28.189 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-fuse-2:18.2.4-855.g19e2a2be.el9.x86_64 1/2 2024-09-17T06:55:28.439 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-17T06:55:28.439 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:55:28.439 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-09-17T06:55:28.439 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-fuse-2:18.2.4-855.g19e2a2be.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-17T06:55:28.440 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:55:28.440 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-09-17T06:55:28.733 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-volume 2024-09-17T06:55:29.328 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:01:28 ago on Tue 17 Sep 2024 06:54:01 AM UTC. 2024-09-17T06:55:29.470 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-09-17T06:55:29.471 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-09-17T06:55:29.471 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repository Size 2024-09-17T06:55:29.471 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-09-17T06:55:29.471 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-09-17T06:55:29.471 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-volume noarch 2:18.2.4-855.g19e2a2be.el9 ceph-noarch 264 k 2024-09-17T06:55:29.471 INFO:teuthology.orchestra.run.smithi135.stdout:Installing dependencies: 2024-09-17T06:55:29.471 INFO:teuthology.orchestra.run.smithi135.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-17T06:55:29.471 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:55:29.471 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-09-17T06:55:29.471 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-09-17T06:55:29.472 INFO:teuthology.orchestra.run.smithi135.stdout:Install 2 Packages 2024-09-17T06:55:29.472 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:55:29.472 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 342 k 2024-09-17T06:55:29.472 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 1.5 M 2024-09-17T06:55:29.472 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-09-17T06:55:29.804 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.4-855.g19e2a2 12/12 2024-09-17T06:55:29.804 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.4-855.g19e2a2 1/12 2024-09-17T06:55:29.804 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-17T06:55:29.804 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-17T06:55:29.804 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-17T06:55:29.804 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-17T06:55:29.804 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-17T06:55:29.804 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-17T06:55:29.804 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-17T06:55:29.804 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-17T06:55:29.805 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-17T06:55:29.805 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-17T06:55:29.825 INFO:teuthology.orchestra.run.smithi135.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 813 kB/s | 77 kB 00:00 2024-09-17T06:55:30.009 INFO:teuthology.orchestra.run.smithi135.stdout:(2/2): ceph-volume-18.2.4-855.g19e2a2be.el9.noa 950 kB/s | 264 kB 00:00 2024-09-17T06:55:30.009 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-09-17T06:55:30.010 INFO:teuthology.orchestra.run.smithi135.stdout:Total 635 kB/s | 342 kB 00:00 2024-09-17T06:55:30.022 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-09-17T06:55:30.031 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-09-17T06:55:30.031 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-09-17T06:55:30.066 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-09-17T06:55:30.066 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-09-17T06:55:30.212 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-09-17T06:55:30.252 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-17T06:55:30.253 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:55:30.253 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-17T06:55:30.253 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-diskprediction-local-2:18.2.4-855.g19e2a2be.el9.noarch 2024-09-17T06:55:30.253 INFO:teuthology.orchestra.run.smithi100.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-17T06:55:30.253 INFO:teuthology.orchestra.run.smithi100.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-17T06:55:30.253 INFO:teuthology.orchestra.run.smithi100.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-17T06:55:30.253 INFO:teuthology.orchestra.run.smithi100.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-17T06:55:30.253 INFO:teuthology.orchestra.run.smithi100.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-17T06:55:30.253 INFO:teuthology.orchestra.run.smithi100.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-17T06:55:30.253 INFO:teuthology.orchestra.run.smithi100.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-17T06:55:30.253 INFO:teuthology.orchestra.run.smithi100.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-17T06:55:30.254 INFO:teuthology.orchestra.run.smithi100.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-17T06:55:30.254 INFO:teuthology.orchestra.run.smithi100.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-17T06:55:30.254 INFO:teuthology.orchestra.run.smithi100.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-17T06:55:30.254 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:55:30.254 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-17T06:55:30.385 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-09-17T06:55:30.420 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-volume-2:18.2.4-855.g19e2a2be.el9.noarch 2/2 2024-09-17T06:55:30.436 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-volume-2:18.2.4-855.g19e2a2be.el9.noarch 2/2 2024-09-17T06:55:30.436 INFO:teuthology.orchestra.run.smithi135.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T06:55:30.436 INFO:teuthology.orchestra.run.smithi135.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-17T06:55:30.436 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:55:30.655 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph-mgr-rook 2024-09-17T06:55:31.154 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-volume-2:18.2.4-855.g19e2a2be.el9.noarch 1/2 2024-09-17T06:55:31.252 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:31 ago on Tue 17 Sep 2024 06:54:00 AM UTC. 2024-09-17T06:55:31.386 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-17T06:55:31.387 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T06:55:31.387 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repository Size 2024-09-17T06:55:31.387 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T06:55:31.387 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-17T06:55:31.387 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-rook noarch 2:18.2.4-855.g19e2a2be.el9 ceph-noarch 49 k 2024-09-17T06:55:31.388 INFO:teuthology.orchestra.run.smithi100.stdout:Installing dependencies: 2024-09-17T06:55:31.388 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-17T06:55:31.388 INFO:teuthology.orchestra.run.smithi100.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-17T06:55:31.388 INFO:teuthology.orchestra.run.smithi100.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-17T06:55:31.388 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-17T06:55:31.388 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-17T06:55:31.389 INFO:teuthology.orchestra.run.smithi100.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-17T06:55:31.389 INFO:teuthology.orchestra.run.smithi100.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-17T06:55:31.389 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-17T06:55:31.389 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-17T06:55:31.389 INFO:teuthology.orchestra.run.smithi100.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-17T06:55:31.390 INFO:teuthology.orchestra.run.smithi100.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-17T06:55:31.390 INFO:teuthology.orchestra.run.smithi100.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-17T06:55:31.390 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:55:31.390 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-17T06:55:31.390 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T06:55:31.390 INFO:teuthology.orchestra.run.smithi100.stdout:Install 13 Packages 2024-09-17T06:55:31.390 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:55:31.391 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 2.2 M 2024-09-17T06:55:31.391 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 26 M 2024-09-17T06:55:31.392 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-17T06:55:31.462 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-09-17T06:55:31.462 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:55:31.463 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-09-17T06:55:31.463 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-volume-2:18.2.4-855.g19e2a2be.el9.noarch 2024-09-17T06:55:31.463 INFO:teuthology.orchestra.run.smithi135.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-17T06:55:31.463 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:55:31.463 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-09-17T06:55:31.645 INFO:teuthology.orchestra.run.smithi100.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 258 kB/s | 19 kB 00:00 2024-09-17T06:55:31.669 INFO:teuthology.orchestra.run.smithi100.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 258 kB/s | 26 kB 00:00 2024-09-17T06:55:31.745 INFO:teuthology.orchestra.run.smithi100.stdout:(3/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.1 MB/s | 159 kB 00:00 2024-09-17T06:55:31.770 INFO:teuthology.orchestra.run.smithi100.stdout:(4/13): ceph-mgr-rook-18.2.4-855.g19e2a2be.el9. 246 kB/s | 49 kB 00:00 2024-09-17T06:55:31.776 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install librados-devel 2024-09-17T06:55:31.795 INFO:teuthology.orchestra.run.smithi100.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.4 MB/s | 222 kB 00:00 2024-09-17T06:55:31.821 INFO:teuthology.orchestra.run.smithi100.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.6 MB/s | 279 kB 00:00 2024-09-17T06:55:31.863 INFO:teuthology.orchestra.run.smithi100.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 580 kB/s | 54 kB 00:00 2024-09-17T06:55:31.888 INFO:teuthology.orchestra.run.smithi100.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 349 kB/s | 32 kB 00:00 2024-09-17T06:55:31.922 INFO:teuthology.orchestra.run.smithi100.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 150 kB/s | 14 kB 00:00 2024-09-17T06:55:31.955 INFO:teuthology.orchestra.run.smithi100.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.8 MB/s | 59 kB 00:00 2024-09-17T06:55:31.989 INFO:teuthology.orchestra.run.smithi100.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 2.6 MB/s | 90 kB 00:00 2024-09-17T06:55:32.014 INFO:teuthology.orchestra.run.smithi100.stdout:(12/13): python3-google-auth-2.34.0-1.el9.noarc 1.4 MB/s | 220 kB 00:00 2024-09-17T06:55:32.047 INFO:teuthology.orchestra.run.smithi100.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.7 MB/s | 1.0 MB 00:00 2024-09-17T06:55:32.050 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-17T06:55:32.050 INFO:teuthology.orchestra.run.smithi100.stdout:Total 3.3 MB/s | 2.2 MB 00:00 2024-09-17T06:55:32.113 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-17T06:55:32.128 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-17T06:55:32.129 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-17T06:55:32.215 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-17T06:55:32.215 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-17T06:55:32.371 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:01:31 ago on Tue 17 Sep 2024 06:54:01 AM UTC. 2024-09-17T06:55:32.503 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-09-17T06:55:32.504 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-09-17T06:55:32.504 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repo Size 2024-09-17T06:55:32.505 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-09-17T06:55:32.505 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-09-17T06:55:32.505 INFO:teuthology.orchestra.run.smithi135.stdout: librados-devel x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 126 k 2024-09-17T06:55:32.505 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:55:32.505 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-09-17T06:55:32.505 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-09-17T06:55:32.505 INFO:teuthology.orchestra.run.smithi135.stdout:Install 1 Package 2024-09-17T06:55:32.505 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:55:32.505 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 126 k 2024-09-17T06:55:32.505 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 456 k 2024-09-17T06:55:32.505 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-09-17T06:55:32.717 INFO:teuthology.orchestra.run.smithi135.stdout:librados-devel-18.2.4-855.g19e2a2be.el9.x86_64. 604 kB/s | 126 kB 00:00 2024-09-17T06:55:32.717 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-09-17T06:55:32.717 INFO:teuthology.orchestra.run.smithi135.stdout:Total 600 kB/s | 126 kB 00:00 2024-09-17T06:55:32.717 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-09-17T06:55:32.721 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-09-17T06:55:32.721 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-09-17T06:55:32.731 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-17T06:55:32.784 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-09-17T06:55:32.784 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-09-17T06:55:32.893 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-17T06:55:32.940 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-09-17T06:55:32.957 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-17T06:55:33.013 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-17T06:55:33.045 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : librados-devel-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-17T06:55:33.049 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-17T06:55:33.091 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-17T06:55:33.194 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-17T06:55:33.303 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-17T06:55:33.359 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-17T06:55:33.461 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: librados-devel-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-17T06:55:33.737 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : librados-devel-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-17T06:55:33.737 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:55:33.737 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-09-17T06:55:33.737 INFO:teuthology.orchestra.run.smithi135.stdout: librados-devel-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T06:55:33.737 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:55:33.737 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-09-17T06:55:33.993 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install libcephfs2 2024-09-17T06:55:34.073 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-17T06:55:34.124 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-17T06:55:34.157 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-17T06:55:34.206 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-17T06:55:34.227 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-mgr-rook-2:18.2.4-855.g19e2a2be.el9.noarch 13/13 2024-09-17T06:55:34.586 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:01:33 ago on Tue 17 Sep 2024 06:54:01 AM UTC. 2024-09-17T06:55:34.676 INFO:teuthology.orchestra.run.smithi135.stdout:Package libcephfs2-2:18.2.4-855.g19e2a2be.el9.x86_64 is already installed. 2024-09-17T06:55:34.720 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-09-17T06:55:34.721 INFO:teuthology.orchestra.run.smithi135.stdout:Nothing to do. 2024-09-17T06:55:34.721 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-09-17T06:55:34.800 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install libcephfs-devel 2024-09-17T06:55:35.025 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.4-855.g19e2a2be.el9.noarch 13/13 2024-09-17T06:55:35.026 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-mgr-rook-2:18.2.4-855.g19e2a2be.el9.noarch 1/13 2024-09-17T06:55:35.026 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-17T06:55:35.026 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-17T06:55:35.026 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-17T06:55:35.026 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-17T06:55:35.026 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-17T06:55:35.026 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-17T06:55:35.026 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-17T06:55:35.026 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-17T06:55:35.026 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-17T06:55:35.026 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-17T06:55:35.026 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-17T06:55:35.358 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-17T06:55:35.359 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:55:35.359 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-17T06:55:35.359 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-rook-2:18.2.4-855.g19e2a2be.el9.noarch 2024-09-17T06:55:35.359 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-17T06:55:35.359 INFO:teuthology.orchestra.run.smithi100.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-17T06:55:35.359 INFO:teuthology.orchestra.run.smithi100.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-17T06:55:35.359 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-17T06:55:35.359 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-17T06:55:35.360 INFO:teuthology.orchestra.run.smithi100.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-17T06:55:35.360 INFO:teuthology.orchestra.run.smithi100.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-17T06:55:35.360 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-17T06:55:35.360 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-17T06:55:35.360 INFO:teuthology.orchestra.run.smithi100.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-17T06:55:35.360 INFO:teuthology.orchestra.run.smithi100.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-17T06:55:35.360 INFO:teuthology.orchestra.run.smithi100.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-17T06:55:35.360 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:55:35.360 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-17T06:55:35.386 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:01:34 ago on Tue 17 Sep 2024 06:54:01 AM UTC. 2024-09-17T06:55:35.517 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-09-17T06:55:35.518 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-09-17T06:55:35.519 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repo Size 2024-09-17T06:55:35.519 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-09-17T06:55:35.519 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-09-17T06:55:35.519 INFO:teuthology.orchestra.run.smithi135.stdout: libcephfs-devel x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 31 k 2024-09-17T06:55:35.519 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:55:35.519 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-09-17T06:55:35.519 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-09-17T06:55:35.519 INFO:teuthology.orchestra.run.smithi135.stdout:Install 1 Package 2024-09-17T06:55:35.519 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:55:35.519 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 31 k 2024-09-17T06:55:35.519 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 139 k 2024-09-17T06:55:35.519 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-09-17T06:55:35.603 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph-mgr-cephadm 2024-09-17T06:55:35.669 INFO:teuthology.orchestra.run.smithi135.stdout:libcephfs-devel-18.2.4-855.g19e2a2be.el9.x86_64 208 kB/s | 31 kB 00:00 2024-09-17T06:55:35.669 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-09-17T06:55:35.670 INFO:teuthology.orchestra.run.smithi135.stdout:Total 206 kB/s | 31 kB 00:00 2024-09-17T06:55:35.670 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-09-17T06:55:35.673 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-09-17T06:55:35.673 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-09-17T06:55:35.684 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-09-17T06:55:35.685 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-09-17T06:55:35.772 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-09-17T06:55:35.873 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : libcephfs-devel-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-17T06:55:36.180 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:36 ago on Tue 17 Sep 2024 06:54:00 AM UTC. 2024-09-17T06:55:36.244 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: libcephfs-devel-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-17T06:55:36.313 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-17T06:55:36.313 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T06:55:36.313 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repository Size 2024-09-17T06:55:36.313 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T06:55:36.314 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-17T06:55:36.314 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-cephadm noarch 2:18.2.4-855.g19e2a2be.el9 ceph-noarch 139 k 2024-09-17T06:55:36.314 INFO:teuthology.orchestra.run.smithi100.stdout:Installing dependencies: 2024-09-17T06:55:36.314 INFO:teuthology.orchestra.run.smithi100.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-17T06:55:36.314 INFO:teuthology.orchestra.run.smithi100.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-17T06:55:36.314 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-17T06:55:36.314 INFO:teuthology.orchestra.run.smithi100.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-17T06:55:36.314 INFO:teuthology.orchestra.run.smithi100.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-17T06:55:36.314 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:55:36.314 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-17T06:55:36.314 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T06:55:36.315 INFO:teuthology.orchestra.run.smithi100.stdout:Install 6 Packages 2024-09-17T06:55:36.315 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:55:36.315 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 7.0 M 2024-09-17T06:55:36.315 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 33 M 2024-09-17T06:55:36.315 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-17T06:55:36.474 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libcephfs-devel-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-17T06:55:36.474 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:55:36.474 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-09-17T06:55:36.474 INFO:teuthology.orchestra.run.smithi135.stdout: libcephfs-devel-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T06:55:36.475 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:55:36.475 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-09-17T06:55:36.559 INFO:teuthology.orchestra.run.smithi100.stdout:(1/6): ceph-mgr-cephadm-18.2.4-855.g19e2a2be.el 719 kB/s | 139 kB 00:00 2024-09-17T06:55:36.584 INFO:teuthology.orchestra.run.smithi100.stdout:(2/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.1 MB/s | 248 kB 00:00 2024-09-17T06:55:36.643 INFO:teuthology.orchestra.run.smithi100.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 6.4 MB/s | 548 kB 00:00 2024-09-17T06:55:36.650 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install librados2 2024-09-17T06:55:36.668 INFO:teuthology.orchestra.run.smithi100.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 691 kB/s | 58 kB 00:00 2024-09-17T06:55:36.693 INFO:teuthology.orchestra.run.smithi100.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.5 MB/s | 74 kB 00:00 2024-09-17T06:55:37.240 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:01:36 ago on Tue 17 Sep 2024 06:54:01 AM UTC. 2024-09-17T06:55:37.302 INFO:teuthology.orchestra.run.smithi100.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 6.4 MB/s | 6.0 MB 00:00 2024-09-17T06:55:37.303 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-17T06:55:37.304 INFO:teuthology.orchestra.run.smithi100.stdout:Total 7.1 MB/s | 7.0 MB 00:00 2024-09-17T06:55:37.330 INFO:teuthology.orchestra.run.smithi135.stdout:Package librados2-2:18.2.4-855.g19e2a2be.el9.x86_64 is already installed. 2024-09-17T06:55:37.373 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-09-17T06:55:37.373 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-17T06:55:37.374 INFO:teuthology.orchestra.run.smithi135.stdout:Nothing to do. 2024-09-17T06:55:37.374 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-09-17T06:55:37.384 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-17T06:55:37.384 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-17T06:55:37.443 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install librbd1 2024-09-17T06:55:37.484 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-17T06:55:37.485 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-17T06:55:37.797 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-17T06:55:37.960 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-17T06:55:38.003 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-17T06:55:38.029 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:01:37 ago on Tue 17 Sep 2024 06:54:01 AM UTC. 2024-09-17T06:55:38.120 INFO:teuthology.orchestra.run.smithi135.stdout:Package librbd1-2:18.2.4-855.g19e2a2be.el9.x86_64 is already installed. 2024-09-17T06:55:38.162 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-09-17T06:55:38.163 INFO:teuthology.orchestra.run.smithi135.stdout:Nothing to do. 2024-09-17T06:55:38.163 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-09-17T06:55:38.237 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install python3-rados 2024-09-17T06:55:38.528 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-17T06:55:38.615 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-17T06:55:38.677 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-17T06:55:38.707 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-mgr-cephadm-2:18.2.4-855.g19e2a2be.el9.noarch 6/6 2024-09-17T06:55:38.824 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:01:37 ago on Tue 17 Sep 2024 06:54:01 AM UTC. 2024-09-17T06:55:38.917 INFO:teuthology.orchestra.run.smithi135.stdout:Package python3-rados-2:18.2.4-855.g19e2a2be.el9.x86_64 is already installed. 2024-09-17T06:55:38.961 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-09-17T06:55:38.962 INFO:teuthology.orchestra.run.smithi135.stdout:Nothing to do. 2024-09-17T06:55:38.962 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-09-17T06:55:39.036 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install python3-rgw 2024-09-17T06:55:39.500 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.4-855.g19e2a2be.el9.noarch 6/6 2024-09-17T06:55:39.501 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-mgr-cephadm-2:18.2.4-855.g19e2a2be.el9.noarch 1/6 2024-09-17T06:55:39.501 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-17T06:55:39.501 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-17T06:55:39.501 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-17T06:55:39.501 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-17T06:55:39.623 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:01:38 ago on Tue 17 Sep 2024 06:54:01 AM UTC. 2024-09-17T06:55:39.715 INFO:teuthology.orchestra.run.smithi135.stdout:Package python3-rgw-2:18.2.4-855.g19e2a2be.el9.x86_64 is already installed. 2024-09-17T06:55:39.759 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-09-17T06:55:39.760 INFO:teuthology.orchestra.run.smithi135.stdout:Nothing to do. 2024-09-17T06:55:39.760 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-09-17T06:55:39.775 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-17T06:55:39.775 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:55:39.776 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-17T06:55:39.776 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-cephadm-2:18.2.4-855.g19e2a2be.el9.noarch 2024-09-17T06:55:39.776 INFO:teuthology.orchestra.run.smithi100.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-17T06:55:39.776 INFO:teuthology.orchestra.run.smithi100.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-17T06:55:39.776 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-17T06:55:39.776 INFO:teuthology.orchestra.run.smithi100.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-17T06:55:39.776 INFO:teuthology.orchestra.run.smithi100.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-17T06:55:39.776 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:55:39.776 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-17T06:55:39.836 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install python3-cephfs 2024-09-17T06:55:40.051 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph-fuse 2024-09-17T06:55:40.416 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:01:39 ago on Tue 17 Sep 2024 06:54:01 AM UTC. 2024-09-17T06:55:40.506 INFO:teuthology.orchestra.run.smithi135.stdout:Package python3-cephfs-2:18.2.4-855.g19e2a2be.el9.x86_64 is already installed. 2024-09-17T06:55:40.549 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-09-17T06:55:40.550 INFO:teuthology.orchestra.run.smithi135.stdout:Nothing to do. 2024-09-17T06:55:40.550 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-09-17T06:55:40.628 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:40 ago on Tue 17 Sep 2024 06:54:00 AM UTC. 2024-09-17T06:55:40.636 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install python3-rbd 2024-09-17T06:55:40.760 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-17T06:55:40.760 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T06:55:40.760 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repository Size 2024-09-17T06:55:40.760 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T06:55:40.761 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-17T06:55:40.761 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-fuse x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 852 k 2024-09-17T06:55:40.761 INFO:teuthology.orchestra.run.smithi100.stdout:Installing dependencies: 2024-09-17T06:55:40.761 INFO:teuthology.orchestra.run.smithi100.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-17T06:55:40.761 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:55:40.761 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-17T06:55:40.761 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T06:55:40.761 INFO:teuthology.orchestra.run.smithi100.stdout:Install 2 Packages 2024-09-17T06:55:40.761 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:55:40.762 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 932 k 2024-09-17T06:55:40.762 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 2.7 M 2024-09-17T06:55:40.762 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-17T06:55:41.070 INFO:teuthology.orchestra.run.smithi100.stdout:(1/2): ceph-fuse-18.2.4-855.g19e2a2be.el9.x86_6 2.9 MB/s | 852 kB 00:00 2024-09-17T06:55:41.226 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:01:40 ago on Tue 17 Sep 2024 06:54:01 AM UTC. 2024-09-17T06:55:41.318 INFO:teuthology.orchestra.run.smithi135.stdout:Package python3-rbd-2:18.2.4-855.g19e2a2be.el9.x86_64 is already installed. 2024-09-17T06:55:41.362 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-09-17T06:55:41.363 INFO:teuthology.orchestra.run.smithi135.stdout:Nothing to do. 2024-09-17T06:55:41.363 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-09-17T06:55:41.444 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install rbd-fuse 2024-09-17T06:55:41.462 INFO:teuthology.orchestra.run.smithi100.stdout:(2/2): fuse-2.9.9-16.el9.x86_64.rpm 117 kB/s | 80 kB 00:00 2024-09-17T06:55:41.463 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-17T06:55:41.463 INFO:teuthology.orchestra.run.smithi100.stdout:Total 1.3 MB/s | 932 kB 00:00 2024-09-17T06:55:41.476 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-17T06:55:41.500 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-17T06:55:41.500 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-17T06:55:41.585 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-17T06:55:41.585 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-17T06:55:41.773 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-17T06:55:41.918 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-17T06:55:41.986 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-fuse-2:18.2.4-855.g19e2a2be.el9.x86_64 2/2 2024-09-17T06:55:42.033 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:01:41 ago on Tue 17 Sep 2024 06:54:01 AM UTC. 2024-09-17T06:55:42.166 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-09-17T06:55:42.166 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-09-17T06:55:42.166 INFO:teuthology.orchestra.run.smithi135.stdout: Package Architecture Version Repository Size 2024-09-17T06:55:42.167 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-09-17T06:55:42.167 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-09-17T06:55:42.167 INFO:teuthology.orchestra.run.smithi135.stdout: rbd-fuse x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 87 k 2024-09-17T06:55:42.167 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:55:42.167 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-09-17T06:55:42.167 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-09-17T06:55:42.167 INFO:teuthology.orchestra.run.smithi135.stdout:Install 1 Package 2024-09-17T06:55:42.167 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:55:42.167 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 87 k 2024-09-17T06:55:42.167 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 227 k 2024-09-17T06:55:42.167 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-09-17T06:55:42.380 INFO:teuthology.orchestra.run.smithi135.stdout:rbd-fuse-18.2.4-855.g19e2a2be.el9.x86_64.rpm 408 kB/s | 87 kB 00:00 2024-09-17T06:55:42.380 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-09-17T06:55:42.381 INFO:teuthology.orchestra.run.smithi135.stdout:Total 405 kB/s | 87 kB 00:00 2024-09-17T06:55:42.381 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-09-17T06:55:42.387 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-09-17T06:55:42.388 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-09-17T06:55:42.451 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-09-17T06:55:42.451 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-09-17T06:55:42.598 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-09-17T06:55:42.645 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-fuse-2:18.2.4-855.g19e2a2be.el9.x86_64 2/2 2024-09-17T06:55:42.645 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-fuse-2:18.2.4-855.g19e2a2be.el9.x86_64 1/2 2024-09-17T06:55:42.685 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : rbd-fuse-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-17T06:55:42.908 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-17T06:55:42.908 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:55:42.908 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-17T06:55:42.908 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-fuse-2:18.2.4-855.g19e2a2be.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-17T06:55:42.908 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:55:42.908 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-17T06:55:43.141 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: rbd-fuse-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-17T06:55:43.148 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph-volume 2024-09-17T06:55:43.450 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : rbd-fuse-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-17T06:55:43.450 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:55:43.450 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-09-17T06:55:43.450 INFO:teuthology.orchestra.run.smithi135.stdout: rbd-fuse-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T06:55:43.450 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:55:43.450 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-09-17T06:55:43.649 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install rbd-mirror 2024-09-17T06:55:43.754 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:43 ago on Tue 17 Sep 2024 06:54:00 AM UTC. 2024-09-17T06:55:43.895 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-17T06:55:43.896 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T06:55:43.896 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repository Size 2024-09-17T06:55:43.896 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T06:55:43.896 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-17T06:55:43.896 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-volume noarch 2:18.2.4-855.g19e2a2be.el9 ceph-noarch 264 k 2024-09-17T06:55:43.896 INFO:teuthology.orchestra.run.smithi100.stdout:Installing dependencies: 2024-09-17T06:55:43.896 INFO:teuthology.orchestra.run.smithi100.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-17T06:55:43.896 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:55:43.897 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-17T06:55:43.897 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T06:55:43.897 INFO:teuthology.orchestra.run.smithi100.stdout:Install 2 Packages 2024-09-17T06:55:43.897 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:55:43.897 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 342 k 2024-09-17T06:55:43.897 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 1.5 M 2024-09-17T06:55:43.897 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-17T06:55:44.125 INFO:teuthology.orchestra.run.smithi100.stdout:(1/2): ceph-volume-18.2.4-855.g19e2a2be.el9.noa 1.3 MB/s | 264 kB 00:00 2024-09-17T06:55:44.251 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:01:43 ago on Tue 17 Sep 2024 06:54:01 AM UTC. 2024-09-17T06:55:44.390 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-09-17T06:55:44.391 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-09-17T06:55:44.391 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repo Size 2024-09-17T06:55:44.391 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-09-17T06:55:44.391 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-09-17T06:55:44.391 INFO:teuthology.orchestra.run.smithi135.stdout: rbd-mirror x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 3.0 M 2024-09-17T06:55:44.391 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:55:44.391 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-09-17T06:55:44.391 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-09-17T06:55:44.391 INFO:teuthology.orchestra.run.smithi135.stdout:Install 1 Package 2024-09-17T06:55:44.392 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:55:44.392 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 3.0 M 2024-09-17T06:55:44.392 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 12 M 2024-09-17T06:55:44.392 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-09-17T06:55:44.609 INFO:teuthology.orchestra.run.smithi100.stdout:(2/2): python3-packaging-20.9-5.el9.noarch.rpm 113 kB/s | 77 kB 00:00 2024-09-17T06:55:44.610 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-17T06:55:44.610 INFO:teuthology.orchestra.run.smithi100.stdout:Total 478 kB/s | 342 kB 00:00 2024-09-17T06:55:44.624 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-17T06:55:44.635 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-17T06:55:44.635 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-17T06:55:44.671 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-17T06:55:44.672 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-17T06:55:44.724 INFO:teuthology.orchestra.run.smithi135.stdout:rbd-mirror-18.2.4-855.g19e2a2be.el9.x86_64.rpm 9.0 MB/s | 3.0 MB 00:00 2024-09-17T06:55:44.725 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-09-17T06:55:44.725 INFO:teuthology.orchestra.run.smithi135.stdout:Total 9.0 MB/s | 3.0 MB 00:00 2024-09-17T06:55:44.725 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-09-17T06:55:44.734 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-09-17T06:55:44.734 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-09-17T06:55:44.815 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-09-17T06:55:44.815 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-09-17T06:55:44.819 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-17T06:55:45.001 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-09-17T06:55:45.044 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-volume-2:18.2.4-855.g19e2a2be.el9.noarch 2/2 2024-09-17T06:55:45.061 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-volume-2:18.2.4-855.g19e2a2be.el9.noarch 2/2 2024-09-17T06:55:45.061 INFO:teuthology.orchestra.run.smithi100.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T06:55:45.061 INFO:teuthology.orchestra.run.smithi100.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-17T06:55:45.061 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:55:45.231 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-09-17T06:55:45.291 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : rbd-mirror-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-17T06:55:45.317 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: rbd-mirror-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-17T06:55:45.317 INFO:teuthology.orchestra.run.smithi135.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T06:55:45.317 INFO:teuthology.orchestra.run.smithi135.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-17T06:55:45.317 INFO:teuthology.orchestra.run.smithi135.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-17T06:55:45.317 INFO:teuthology.orchestra.run.smithi135.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-17T06:55:45.317 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:55:45.770 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-volume-2:18.2.4-855.g19e2a2be.el9.noarch 1/2 2024-09-17T06:55:46.074 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-09-17T06:55:46.074 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:55:46.075 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-17T06:55:46.075 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-volume-2:18.2.4-855.g19e2a2be.el9.noarch 2024-09-17T06:55:46.075 INFO:teuthology.orchestra.run.smithi100.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-17T06:55:46.075 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:55:46.075 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-17T06:55:46.392 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install librados-devel 2024-09-17T06:55:46.393 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : rbd-mirror-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-17T06:55:46.393 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:55:46.393 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-09-17T06:55:46.393 INFO:teuthology.orchestra.run.smithi135.stdout: rbd-mirror-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T06:55:46.393 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:55:46.394 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-09-17T06:55:46.637 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install rbd-nbd 2024-09-17T06:55:46.988 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:46 ago on Tue 17 Sep 2024 06:54:00 AM UTC. 2024-09-17T06:55:47.120 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-17T06:55:47.120 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T06:55:47.120 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repo Size 2024-09-17T06:55:47.120 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T06:55:47.121 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-17T06:55:47.121 INFO:teuthology.orchestra.run.smithi100.stdout: librados-devel x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 126 k 2024-09-17T06:55:47.121 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:55:47.121 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-17T06:55:47.121 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T06:55:47.121 INFO:teuthology.orchestra.run.smithi100.stdout:Install 1 Package 2024-09-17T06:55:47.121 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:55:47.121 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 126 k 2024-09-17T06:55:47.121 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 456 k 2024-09-17T06:55:47.121 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-17T06:55:47.249 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:01:46 ago on Tue 17 Sep 2024 06:54:01 AM UTC. 2024-09-17T06:55:47.331 INFO:teuthology.orchestra.run.smithi100.stdout:librados-devel-18.2.4-855.g19e2a2be.el9.x86_64. 603 kB/s | 126 kB 00:00 2024-09-17T06:55:47.331 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-17T06:55:47.332 INFO:teuthology.orchestra.run.smithi100.stdout:Total 599 kB/s | 126 kB 00:00 2024-09-17T06:55:47.332 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-17T06:55:47.337 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-17T06:55:47.337 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-17T06:55:47.389 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-09-17T06:55:47.391 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-09-17T06:55:47.391 INFO:teuthology.orchestra.run.smithi135.stdout: Package Architecture Version Repository Size 2024-09-17T06:55:47.391 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-09-17T06:55:47.391 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-09-17T06:55:47.391 INFO:teuthology.orchestra.run.smithi135.stdout: rbd-nbd x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 172 k 2024-09-17T06:55:47.391 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:55:47.392 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-09-17T06:55:47.392 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-09-17T06:55:47.392 INFO:teuthology.orchestra.run.smithi135.stdout:Install 1 Package 2024-09-17T06:55:47.392 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:55:47.392 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 172 k 2024-09-17T06:55:47.392 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 494 k 2024-09-17T06:55:47.392 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-09-17T06:55:47.400 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-17T06:55:47.401 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-17T06:55:47.555 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-17T06:55:47.578 INFO:teuthology.orchestra.run.smithi135.stdout:rbd-nbd-18.2.4-855.g19e2a2be.el9.x86_64.rpm 921 kB/s | 172 kB 00:00 2024-09-17T06:55:47.579 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-09-17T06:55:47.579 INFO:teuthology.orchestra.run.smithi135.stdout:Total 913 kB/s | 172 kB 00:00 2024-09-17T06:55:47.579 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-09-17T06:55:47.587 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-09-17T06:55:47.587 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-09-17T06:55:47.657 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-09-17T06:55:47.657 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-09-17T06:55:47.670 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : librados-devel-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-17T06:55:47.817 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-09-17T06:55:47.927 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : rbd-nbd-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-17T06:55:48.130 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: librados-devel-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-17T06:55:48.337 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: rbd-nbd-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-17T06:55:48.400 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : librados-devel-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-17T06:55:48.400 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:55:48.400 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-17T06:55:48.400 INFO:teuthology.orchestra.run.smithi100.stdout: librados-devel-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T06:55:48.400 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:55:48.400 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-17T06:55:48.582 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : rbd-nbd-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-17T06:55:48.583 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:55:48.583 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-09-17T06:55:48.583 INFO:teuthology.orchestra.run.smithi135.stdout: rbd-nbd-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T06:55:48.583 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:55:48.583 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-09-17T06:55:48.650 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install libcephfs2 2024-09-17T06:55:48.833 DEBUG:teuthology.parallel:result is None 2024-09-17T06:55:49.245 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:49 ago on Tue 17 Sep 2024 06:54:00 AM UTC. 2024-09-17T06:55:49.339 INFO:teuthology.orchestra.run.smithi100.stdout:Package libcephfs2-2:18.2.4-855.g19e2a2be.el9.x86_64 is already installed. 2024-09-17T06:55:49.385 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-17T06:55:49.386 INFO:teuthology.orchestra.run.smithi100.stdout:Nothing to do. 2024-09-17T06:55:49.386 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-17T06:55:49.460 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install libcephfs-devel 2024-09-17T06:55:50.039 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:50 ago on Tue 17 Sep 2024 06:54:00 AM UTC. 2024-09-17T06:55:50.171 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-17T06:55:50.172 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T06:55:50.172 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repo Size 2024-09-17T06:55:50.172 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T06:55:50.172 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-17T06:55:50.172 INFO:teuthology.orchestra.run.smithi100.stdout: libcephfs-devel x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 31 k 2024-09-17T06:55:50.172 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:55:50.172 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-17T06:55:50.172 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T06:55:50.172 INFO:teuthology.orchestra.run.smithi100.stdout:Install 1 Package 2024-09-17T06:55:50.172 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:55:50.173 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 31 k 2024-09-17T06:55:50.173 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 139 k 2024-09-17T06:55:50.173 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-17T06:55:50.318 INFO:teuthology.orchestra.run.smithi100.stdout:libcephfs-devel-18.2.4-855.g19e2a2be.el9.x86_64 214 kB/s | 31 kB 00:00 2024-09-17T06:55:50.319 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-17T06:55:50.319 INFO:teuthology.orchestra.run.smithi100.stdout:Total 212 kB/s | 31 kB 00:00 2024-09-17T06:55:50.319 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-17T06:55:50.322 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-17T06:55:50.322 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-17T06:55:50.343 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-17T06:55:50.344 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-17T06:55:50.435 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-17T06:55:50.549 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libcephfs-devel-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-17T06:55:50.933 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: libcephfs-devel-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-17T06:55:51.171 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libcephfs-devel-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-17T06:55:51.171 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:55:51.172 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-17T06:55:51.172 INFO:teuthology.orchestra.run.smithi100.stdout: libcephfs-devel-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T06:55:51.172 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:55:51.172 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-17T06:55:51.349 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install librados2 2024-09-17T06:55:51.928 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:51 ago on Tue 17 Sep 2024 06:54:00 AM UTC. 2024-09-17T06:55:52.017 INFO:teuthology.orchestra.run.smithi100.stdout:Package librados2-2:18.2.4-855.g19e2a2be.el9.x86_64 is already installed. 2024-09-17T06:55:52.060 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-17T06:55:52.061 INFO:teuthology.orchestra.run.smithi100.stdout:Nothing to do. 2024-09-17T06:55:52.061 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-17T06:55:52.132 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install librbd1 2024-09-17T06:55:52.706 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:52 ago on Tue 17 Sep 2024 06:54:00 AM UTC. 2024-09-17T06:55:52.794 INFO:teuthology.orchestra.run.smithi100.stdout:Package librbd1-2:18.2.4-855.g19e2a2be.el9.x86_64 is already installed. 2024-09-17T06:55:52.836 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-17T06:55:52.837 INFO:teuthology.orchestra.run.smithi100.stdout:Nothing to do. 2024-09-17T06:55:52.837 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-17T06:55:52.905 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install python3-rados 2024-09-17T06:55:53.481 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:53 ago on Tue 17 Sep 2024 06:54:00 AM UTC. 2024-09-17T06:55:53.572 INFO:teuthology.orchestra.run.smithi100.stdout:Package python3-rados-2:18.2.4-855.g19e2a2be.el9.x86_64 is already installed. 2024-09-17T06:55:53.616 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-17T06:55:53.617 INFO:teuthology.orchestra.run.smithi100.stdout:Nothing to do. 2024-09-17T06:55:53.617 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-17T06:55:53.690 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install python3-rgw 2024-09-17T06:55:54.273 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:54 ago on Tue 17 Sep 2024 06:54:00 AM UTC. 2024-09-17T06:55:54.365 INFO:teuthology.orchestra.run.smithi100.stdout:Package python3-rgw-2:18.2.4-855.g19e2a2be.el9.x86_64 is already installed. 2024-09-17T06:55:54.409 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-17T06:55:54.409 INFO:teuthology.orchestra.run.smithi100.stdout:Nothing to do. 2024-09-17T06:55:54.409 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-17T06:55:54.487 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install python3-cephfs 2024-09-17T06:55:55.062 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:55 ago on Tue 17 Sep 2024 06:54:00 AM UTC. 2024-09-17T06:55:55.150 INFO:teuthology.orchestra.run.smithi100.stdout:Package python3-cephfs-2:18.2.4-855.g19e2a2be.el9.x86_64 is already installed. 2024-09-17T06:55:55.193 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-17T06:55:55.193 INFO:teuthology.orchestra.run.smithi100.stdout:Nothing to do. 2024-09-17T06:55:55.194 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-17T06:55:55.268 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install python3-rbd 2024-09-17T06:55:55.842 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:55 ago on Tue 17 Sep 2024 06:54:00 AM UTC. 2024-09-17T06:55:55.933 INFO:teuthology.orchestra.run.smithi100.stdout:Package python3-rbd-2:18.2.4-855.g19e2a2be.el9.x86_64 is already installed. 2024-09-17T06:55:55.977 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-17T06:55:55.978 INFO:teuthology.orchestra.run.smithi100.stdout:Nothing to do. 2024-09-17T06:55:55.978 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-17T06:55:56.061 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install rbd-fuse 2024-09-17T06:55:56.642 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:56 ago on Tue 17 Sep 2024 06:54:00 AM UTC. 2024-09-17T06:55:56.772 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-17T06:55:56.773 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T06:55:56.773 INFO:teuthology.orchestra.run.smithi100.stdout: Package Architecture Version Repository Size 2024-09-17T06:55:56.773 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T06:55:56.773 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-17T06:55:56.773 INFO:teuthology.orchestra.run.smithi100.stdout: rbd-fuse x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 87 k 2024-09-17T06:55:56.773 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:55:56.773 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-17T06:55:56.774 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T06:55:56.774 INFO:teuthology.orchestra.run.smithi100.stdout:Install 1 Package 2024-09-17T06:55:56.774 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:55:56.774 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 87 k 2024-09-17T06:55:56.774 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 227 k 2024-09-17T06:55:56.774 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-17T06:55:56.963 INFO:teuthology.orchestra.run.smithi100.stdout:rbd-fuse-18.2.4-855.g19e2a2be.el9.x86_64.rpm 458 kB/s | 87 kB 00:00 2024-09-17T06:55:56.963 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-17T06:55:56.963 INFO:teuthology.orchestra.run.smithi100.stdout:Total 455 kB/s | 87 kB 00:00 2024-09-17T06:55:56.964 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-17T06:55:56.970 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-17T06:55:56.971 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-17T06:55:57.034 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-17T06:55:57.034 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-17T06:55:57.186 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-17T06:55:57.293 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : rbd-fuse-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-17T06:55:57.659 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: rbd-fuse-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-17T06:55:57.932 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : rbd-fuse-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-17T06:55:57.933 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:55:57.933 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-17T06:55:57.933 INFO:teuthology.orchestra.run.smithi100.stdout: rbd-fuse-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T06:55:57.933 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:55:57.933 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-17T06:55:58.193 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install rbd-mirror 2024-09-17T06:55:58.793 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:58 ago on Tue 17 Sep 2024 06:54:00 AM UTC. 2024-09-17T06:55:58.936 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-17T06:55:58.938 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T06:55:58.938 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repo Size 2024-09-17T06:55:58.938 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T06:55:58.938 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-17T06:55:58.938 INFO:teuthology.orchestra.run.smithi100.stdout: rbd-mirror x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 3.0 M 2024-09-17T06:55:58.939 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:55:58.939 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-17T06:55:58.939 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T06:55:58.939 INFO:teuthology.orchestra.run.smithi100.stdout:Install 1 Package 2024-09-17T06:55:58.939 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:55:58.940 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 3.0 M 2024-09-17T06:55:58.940 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 12 M 2024-09-17T06:55:58.940 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-17T06:55:59.267 INFO:teuthology.orchestra.run.smithi100.stdout:rbd-mirror-18.2.4-855.g19e2a2be.el9.x86_64.rpm 9.1 MB/s | 3.0 MB 00:00 2024-09-17T06:55:59.267 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-17T06:55:59.268 INFO:teuthology.orchestra.run.smithi100.stdout:Total 9.0 MB/s | 3.0 MB 00:00 2024-09-17T06:55:59.268 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-17T06:55:59.277 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-17T06:55:59.277 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-17T06:55:59.357 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-17T06:55:59.357 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-17T06:55:59.776 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-17T06:55:59.843 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : rbd-mirror-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-17T06:55:59.869 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: rbd-mirror-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-17T06:55:59.869 INFO:teuthology.orchestra.run.smithi100.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T06:55:59.869 INFO:teuthology.orchestra.run.smithi100.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-17T06:55:59.870 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-17T06:55:59.870 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-17T06:55:59.870 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:56:01.041 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : rbd-mirror-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-17T06:56:01.041 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:56:01.042 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-17T06:56:01.042 INFO:teuthology.orchestra.run.smithi100.stdout: rbd-mirror-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T06:56:01.042 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:56:01.042 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-17T06:56:01.277 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install rbd-nbd 2024-09-17T06:56:01.872 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:02:01 ago on Tue 17 Sep 2024 06:54:00 AM UTC. 2024-09-17T06:56:02.005 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-17T06:56:02.006 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T06:56:02.006 INFO:teuthology.orchestra.run.smithi100.stdout: Package Architecture Version Repository Size 2024-09-17T06:56:02.006 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T06:56:02.006 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-17T06:56:02.006 INFO:teuthology.orchestra.run.smithi100.stdout: rbd-nbd x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 172 k 2024-09-17T06:56:02.006 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:56:02.006 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-17T06:56:02.006 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-17T06:56:02.006 INFO:teuthology.orchestra.run.smithi100.stdout:Install 1 Package 2024-09-17T06:56:02.006 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:56:02.006 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 172 k 2024-09-17T06:56:02.007 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 494 k 2024-09-17T06:56:02.007 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-17T06:56:02.329 INFO:teuthology.orchestra.run.smithi100.stdout:rbd-nbd-18.2.4-855.g19e2a2be.el9.x86_64.rpm 533 kB/s | 172 kB 00:00 2024-09-17T06:56:02.329 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-17T06:56:02.330 INFO:teuthology.orchestra.run.smithi100.stdout:Total 531 kB/s | 172 kB 00:00 2024-09-17T06:56:02.330 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-17T06:56:02.341 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-17T06:56:02.341 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-17T06:56:02.404 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-17T06:56:02.405 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-17T06:56:02.564 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-17T06:56:02.659 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : rbd-nbd-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-17T06:56:03.063 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: rbd-nbd-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-17T06:56:03.329 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : rbd-nbd-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-17T06:56:03.330 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:56:03.330 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-17T06:56:03.330 INFO:teuthology.orchestra.run.smithi100.stdout: rbd-nbd-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-17T06:56:03.330 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:56:03.330 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-17T06:56:03.575 DEBUG:teuthology.parallel:result is None 2024-09-17T06:56:03.575 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-17T06:56:03.727 DEBUG:teuthology.orchestra.run.smithi100:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-17T06:56:03.749 INFO:teuthology.orchestra.run.smithi100.stdout:18.2.4-855.g19e2a2be.el9 2024-09-17T06:56:03.749 INFO:teuthology.packaging:The installed version of ceph is 18.2.4-855.g19e2a2be.el9 2024-09-17T06:56:03.749 INFO:teuthology.task.install:The correct ceph version 18.2.4-855.g19e2a2be is installed. 2024-09-17T06:56:03.751 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-17T06:56:03.895 DEBUG:teuthology.orchestra.run.smithi135:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-17T06:56:03.920 INFO:teuthology.orchestra.run.smithi135.stdout:18.2.4-855.g19e2a2be.el9 2024-09-17T06:56:03.920 INFO:teuthology.packaging:The installed version of ceph is 18.2.4-855.g19e2a2be.el9 2024-09-17T06:56:03.920 INFO:teuthology.task.install:The correct ceph version 18.2.4-855.g19e2a2be is installed. 2024-09-17T06:56:03.922 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-17T06:56:03.923 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-17T06:56:03.923 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-17T06:56:03.948 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-09-17T06:56:03.949 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-17T06:56:03.989 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-17T06:56:03.989 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-17T06:56:03.989 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd of=/usr/bin/daemon-helper 2024-09-17T06:56:04.014 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-17T06:56:04.078 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-09-17T06:56:04.078 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd of=/usr/bin/daemon-helper 2024-09-17T06:56:04.105 DEBUG:teuthology.orchestra.run.smithi135:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-17T06:56:04.171 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-17T06:56:04.171 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-17T06:56:04.171 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-17T06:56:04.196 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-17T06:56:04.260 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-09-17T06:56:04.260 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-17T06:56:04.286 DEBUG:teuthology.orchestra.run.smithi135:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-17T06:56:04.353 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-17T06:56:04.354 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-17T06:56:04.354 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd of=/usr/bin/stdin-killer 2024-09-17T06:56:04.378 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-17T06:56:04.443 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-09-17T06:56:04.443 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd of=/usr/bin/stdin-killer 2024-09-17T06:56:04.468 DEBUG:teuthology.orchestra.run.smithi135:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-17T06:56:04.534 INFO:teuthology.run_tasks:Running task cephadm... 2024-09-17T06:56:04.632 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms bind msgr1': False, 'ms bind msgr2': True, 'ms type': 'async'}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'CEPHADM_STRAY_DAEMON', 'CEPHADM_FAILED_DAEMON', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)'], 'log-only-match': ['CEPHADM_'], 'sha1': '19e2a2beb6f41a3ab62650e38614eb29b59dcbb9', 'cephadm_mode': 'root'} 2024-09-17T06:56:04.632 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-17T06:56:04.632 INFO:tasks.cephadm:Cluster fsid is e848267c-74c1-11ef-bceb-c7b262605968 2024-09-17T06:56:04.633 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-09-17T06:56:04.633 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.100', 'mon.c': '[v2:172.21.15.100:3301,v1:172.21.15.100:6790]', 'mon.b': '172.21.15.135'} 2024-09-17T06:56:04.633 INFO:tasks.cephadm:First mon is mon.a on smithi100 2024-09-17T06:56:04.633 INFO:tasks.cephadm:First mgr is y 2024-09-17T06:56:04.633 INFO:tasks.cephadm:Normalizing hostnames... 2024-09-17T06:56:04.633 DEBUG:teuthology.orchestra.run.smithi100:> sudo hostname $(hostname -s) 2024-09-17T06:56:04.659 DEBUG:teuthology.orchestra.run.smithi135:> sudo hostname $(hostname -s) 2024-09-17T06:56:04.684 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-09-17T06:56:04.685 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-17T06:56:04.826 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '19e2a2beb6f41a3ab62650e38614eb29b59dcbb9', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/56094/', 'root_build_cause': 'SCMTRIGGER', 'version': '18.2.4-855-g19e2a2be', 'node_name': '172.21.2.7+braggi07', 'job_name': 'ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '18.2.4-855.g19e2a2be'}, 'url': 'https://4.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-09-16 16:29:21.739533', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'reef', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-09-17T06:56:04.964 INFO:tasks.util.chacra:got chacra host 4.chacra.ceph.com, ref reef, sha1 19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-17T06:56:04.965 INFO:tasks.cephadm:Discovered cachra url: https://4.chacra.ceph.com/binaries/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/centos/9/x86_64/flavors/default/cephadm 2024-09-17T06:56:04.966 DEBUG:teuthology.orchestra.run.smithi100:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-17T06:56:05.312 INFO:teuthology.orchestra.run.smithi100.stdout:-rw-r--r--. 1 ubuntu ubuntu 217222 Sep 17 06:56 /home/ubuntu/cephtest/cephadm 2024-09-17T06:56:05.313 DEBUG:teuthology.orchestra.run.smithi135:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-17T06:56:05.641 INFO:teuthology.orchestra.run.smithi135.stdout:-rw-r--r--. 1 ubuntu ubuntu 217222 Sep 17 06:56 /home/ubuntu/cephtest/cephadm 2024-09-17T06:56:05.642 DEBUG:teuthology.orchestra.run.smithi100:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-17T06:56:05.659 DEBUG:teuthology.orchestra.run.smithi135:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-17T06:56:05.689 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 on all hosts... 2024-09-17T06:56:05.690 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 pull 2024-09-17T06:56:05.701 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 pull 2024-09-17T06:56:07.314 INFO:teuthology.orchestra.run.smithi135.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9... 2024-09-17T06:56:07.326 INFO:teuthology.orchestra.run.smithi100.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9... 2024-09-17T06:56:56.237 INFO:teuthology.orchestra.run.smithi135.stdout:{ 2024-09-17T06:56:56.238 INFO:teuthology.orchestra.run.smithi135.stdout: "ceph_version": "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)", 2024-09-17T06:56:56.238 INFO:teuthology.orchestra.run.smithi135.stdout: "image_id": "980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89", 2024-09-17T06:56:56.238 INFO:teuthology.orchestra.run.smithi135.stdout: "repo_digests": [ 2024-09-17T06:56:56.238 INFO:teuthology.orchestra.run.smithi135.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9" 2024-09-17T06:56:56.238 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T06:56:56.238 INFO:teuthology.orchestra.run.smithi135.stdout:} 2024-09-17T06:56:56.276 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-17T06:56:56.276 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph_version": "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)", 2024-09-17T06:56:56.276 INFO:teuthology.orchestra.run.smithi100.stdout: "image_id": "980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89", 2024-09-17T06:56:56.276 INFO:teuthology.orchestra.run.smithi100.stdout: "repo_digests": [ 2024-09-17T06:56:56.276 INFO:teuthology.orchestra.run.smithi100.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9" 2024-09-17T06:56:56.276 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T06:56:56.276 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-17T06:56:56.296 DEBUG:teuthology.orchestra.run.smithi100:> sudo mkdir -p /etc/ceph 2024-09-17T06:56:56.358 DEBUG:teuthology.orchestra.run.smithi135:> sudo mkdir -p /etc/ceph 2024-09-17T06:56:56.385 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod 777 /etc/ceph 2024-09-17T06:56:56.410 DEBUG:teuthology.orchestra.run.smithi135:> sudo chmod 777 /etc/ceph 2024-09-17T06:56:56.449 INFO:tasks.cephadm:Writing seed config... 2024-09-17T06:56:56.451 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-09-17T06:56:56.451 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-09-17T06:56:56.451 INFO:tasks.cephadm: override: [client.0] debug ms = 1 2024-09-17T06:56:56.451 INFO:tasks.cephadm: override: [client.0] debug objecter = 20 2024-09-17T06:56:56.451 INFO:tasks.cephadm: override: [client.0] debug rados = 20 2024-09-17T06:56:56.451 INFO:tasks.cephadm: override: [global] ms bind msgr1 = False 2024-09-17T06:56:56.451 INFO:tasks.cephadm: override: [global] ms bind msgr2 = True 2024-09-17T06:56:56.451 INFO:tasks.cephadm: override: [global] ms type = async 2024-09-17T06:56:56.451 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-09-17T06:56:56.451 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-09-17T06:56:56.452 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-09-17T06:56:56.452 INFO:tasks.cephadm: override: [mon] mon min osdmap epochs = 50 2024-09-17T06:56:56.452 INFO:tasks.cephadm: override: [mon] mon osdmap full prune interval = 2 2024-09-17T06:56:56.452 INFO:tasks.cephadm: override: [mon] mon osdmap full prune min = 15 2024-09-17T06:56:56.452 INFO:tasks.cephadm: override: [mon] mon osdmap full prune txsize = 2 2024-09-17T06:56:56.452 INFO:tasks.cephadm: override: [mon] paxos service trim min = 10 2024-09-17T06:56:56.452 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-09-17T06:56:56.452 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-09-17T06:56:56.452 INFO:tasks.cephadm: override: [osd] osd debug reject backfill probability = 0.3 2024-09-17T06:56:56.452 INFO:tasks.cephadm: override: [osd] osd delete sleep = 1 2024-09-17T06:56:56.452 INFO:tasks.cephadm: override: [osd] osd max backfills = 3 2024-09-17T06:56:56.452 INFO:tasks.cephadm: override: [osd] osd scrub max interval = 120 2024-09-17T06:56:56.453 INFO:tasks.cephadm: override: [osd] osd scrub min interval = 60 2024-09-17T06:56:56.453 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-09-17T06:56:56.453 INFO:tasks.cephadm: override: [osd] osd snap trim sleep = 2 2024-09-17T06:56:56.453 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-17T06:56:56.453 DEBUG:teuthology.orchestra.run.smithi100:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-09-17T06:56:56.470 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = e848267c-74c1-11ef-bceb-c7b262605968 ms bind msgr1 = False ms bind msgr2 = True ms type = async [osd] osd scrub load threshold = 5.0 osd scrub max interval = 120 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 osd debug reject backfill probability = 0.3 osd delete sleep = 1 osd max backfills = 3 osd scrub min interval = 60 osd snap trim sleep = 2 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon min osdmap epochs = 50 mon osdmap full prune interval = 2 mon osdmap full prune min = 15 mon osdmap full prune txsize = 2 paxos service trim min = 10 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client.0] debug ms = 1 debug objecter = 20 debug rados = 20 2024-09-17T06:56:56.470 DEBUG:teuthology.orchestra.run.smithi100:mon.a> sudo journalctl -f -n 0 -u ceph-e848267c-74c1-11ef-bceb-c7b262605968@mon.a.service 2024-09-17T06:56:56.512 DEBUG:teuthology.orchestra.run.smithi100:mgr.y> sudo journalctl -f -n 0 -u ceph-e848267c-74c1-11ef-bceb-c7b262605968@mgr.y.service 2024-09-17T06:56:56.555 INFO:tasks.cephadm:Bootstrapping... 2024-09-17T06:56:56.555 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 -v bootstrap --fsid e848267c-74c1-11ef-bceb-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.100 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-09-17T06:56:56.707 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-17T06:56:56.707 INFO:teuthology.orchestra.run.smithi100.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9', '-v', 'bootstrap', '--fsid', 'e848267c-74c1-11ef-bceb-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.100', '--skip-admin-label'] 2024-09-17T06:56:56.731 INFO:teuthology.orchestra.run.smithi100.stdout:/bin/podman: stdout 5.2.2 2024-09-17T06:56:56.731 INFO:teuthology.orchestra.run.smithi100.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-09-17T06:56:56.731 INFO:teuthology.orchestra.run.smithi100.stdout:Verifying podman|docker is present... 2024-09-17T06:56:56.753 INFO:teuthology.orchestra.run.smithi100.stdout:/bin/podman: stdout 5.2.2 2024-09-17T06:56:56.754 INFO:teuthology.orchestra.run.smithi100.stdout:Verifying lvm2 is present... 2024-09-17T06:56:56.754 INFO:teuthology.orchestra.run.smithi100.stdout:Verifying time synchronization is in place... 2024-09-17T06:56:56.761 INFO:teuthology.orchestra.run.smithi100.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-17T06:56:56.761 INFO:teuthology.orchestra.run.smithi100.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-17T06:56:56.767 INFO:teuthology.orchestra.run.smithi100.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-17T06:56:56.767 INFO:teuthology.orchestra.run.smithi100.stdout:systemctl: stdout inactive 2024-09-17T06:56:56.774 INFO:teuthology.orchestra.run.smithi100.stdout:systemctl: stdout enabled 2024-09-17T06:56:56.780 INFO:teuthology.orchestra.run.smithi100.stdout:systemctl: stdout active 2024-09-17T06:56:56.780 INFO:teuthology.orchestra.run.smithi100.stdout:Unit chronyd.service is enabled and running 2024-09-17T06:56:56.780 INFO:teuthology.orchestra.run.smithi100.stdout:Repeating the final host check... 2024-09-17T06:56:56.801 INFO:teuthology.orchestra.run.smithi100.stdout:/bin/podman: stdout 5.2.2 2024-09-17T06:56:56.802 INFO:teuthology.orchestra.run.smithi100.stdout:podman (/bin/podman) version 5.2.2 is present 2024-09-17T06:56:56.802 INFO:teuthology.orchestra.run.smithi100.stdout:systemctl is present 2024-09-17T06:56:56.802 INFO:teuthology.orchestra.run.smithi100.stdout:lvcreate is present 2024-09-17T06:56:56.808 INFO:teuthology.orchestra.run.smithi100.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-17T06:56:56.808 INFO:teuthology.orchestra.run.smithi100.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-17T06:56:56.814 INFO:teuthology.orchestra.run.smithi100.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-17T06:56:56.814 INFO:teuthology.orchestra.run.smithi100.stdout:systemctl: stdout inactive 2024-09-17T06:56:56.820 INFO:teuthology.orchestra.run.smithi100.stdout:systemctl: stdout enabled 2024-09-17T06:56:56.827 INFO:teuthology.orchestra.run.smithi100.stdout:systemctl: stdout active 2024-09-17T06:56:56.827 INFO:teuthology.orchestra.run.smithi100.stdout:Unit chronyd.service is enabled and running 2024-09-17T06:56:56.827 INFO:teuthology.orchestra.run.smithi100.stdout:Host looks OK 2024-09-17T06:56:56.827 INFO:teuthology.orchestra.run.smithi100.stdout:Cluster fsid: e848267c-74c1-11ef-bceb-c7b262605968 2024-09-17T06:56:56.827 INFO:teuthology.orchestra.run.smithi100.stdout:Acquiring lock 140292383829584 on /run/cephadm/e848267c-74c1-11ef-bceb-c7b262605968.lock 2024-09-17T06:56:56.827 INFO:teuthology.orchestra.run.smithi100.stdout:Lock 140292383829584 acquired on /run/cephadm/e848267c-74c1-11ef-bceb-c7b262605968.lock 2024-09-17T06:56:56.827 INFO:teuthology.orchestra.run.smithi100.stdout:Verifying IP 172.21.15.100 port 3300 ... 2024-09-17T06:56:56.828 INFO:teuthology.orchestra.run.smithi100.stdout:Verifying IP 172.21.15.100 port 6789 ... 2024-09-17T06:56:56.828 INFO:teuthology.orchestra.run.smithi100.stdout:Base mon IP(s) is [172.21.15.100:3300, 172.21.15.100:6789], mon addrv is [v2:172.21.15.100:3300,v1:172.21.15.100:6789] 2024-09-17T06:56:56.830 INFO:teuthology.orchestra.run.smithi100.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.100 metric 100 2024-09-17T06:56:56.830 INFO:teuthology.orchestra.run.smithi100.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.100 metric 100 2024-09-17T06:56:56.833 INFO:teuthology.orchestra.run.smithi100.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-09-17T06:56:56.833 INFO:teuthology.orchestra.run.smithi100.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 1024 pref medium 2024-09-17T06:56:56.833 INFO:teuthology.orchestra.run.smithi100.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2024-09-17T06:56:56.835 INFO:teuthology.orchestra.run.smithi100.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-09-17T06:56:56.835 INFO:teuthology.orchestra.run.smithi100.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-09-17T06:56:56.835 INFO:teuthology.orchestra.run.smithi100.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-17T06:56:56.836 INFO:teuthology.orchestra.run.smithi100.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-09-17T06:56:56.836 INFO:teuthology.orchestra.run.smithi100.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:d733/64 scope link noprefixroute 2024-09-17T06:56:56.836 INFO:teuthology.orchestra.run.smithi100.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-17T06:56:56.836 INFO:teuthology.orchestra.run.smithi100.stdout:Mon IP `172.21.15.100` is in CIDR network `172.21.0.0/20` 2024-09-17T06:56:56.837 INFO:teuthology.orchestra.run.smithi100.stdout:Mon IP `172.21.15.100` is in CIDR network `172.21.0.0/20` 2024-09-17T06:56:56.837 INFO:teuthology.orchestra.run.smithi100.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-09-17T06:56:56.837 INFO:teuthology.orchestra.run.smithi100.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-09-17T06:56:56.837 INFO:teuthology.orchestra.run.smithi100.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9... 2024-09-17T06:56:57.303 INFO:teuthology.orchestra.run.smithi100.stdout:/bin/podman: stdout 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 2024-09-17T06:56:57.303 INFO:teuthology.orchestra.run.smithi100.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9... 2024-09-17T06:56:57.303 INFO:teuthology.orchestra.run.smithi100.stdout:/bin/podman: stderr Getting image source signatures 2024-09-17T06:56:57.304 INFO:teuthology.orchestra.run.smithi100.stdout:/bin/podman: stderr Copying blob sha256:d8a707c21dc06a64315160cb2d8e524fa91943a344731b2a930ee3a716bebbdd 2024-09-17T06:56:57.304 INFO:teuthology.orchestra.run.smithi100.stdout:/bin/podman: stderr Copying blob sha256:e8b54c863393b7a8216a71737771b73b16a8e43ec7acf927c7faa175c255e551 2024-09-17T06:56:57.304 INFO:teuthology.orchestra.run.smithi100.stdout:/bin/podman: stderr Copying config sha256:980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 2024-09-17T06:56:57.304 INFO:teuthology.orchestra.run.smithi100.stdout:/bin/podman: stderr Writing manifest to image destination 2024-09-17T06:56:58.199 INFO:teuthology.orchestra.run.smithi100.stdout:ceph: stdout ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable) 2024-09-17T06:56:58.199 INFO:teuthology.orchestra.run.smithi100.stdout:Ceph version: ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable) 2024-09-17T06:56:58.200 INFO:teuthology.orchestra.run.smithi100.stdout:Extracting ceph user uid/gid from container image... 2024-09-17T06:56:59.138 INFO:teuthology.orchestra.run.smithi100.stdout:stat: stdout 167 167 2024-09-17T06:56:59.138 INFO:teuthology.orchestra.run.smithi100.stdout:Creating initial keys... 2024-09-17T06:56:59.979 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph-authtool: stdout AQA7KOlmPJuXGRAAxgBzz82kXwVoqJKgAHSugg== 2024-09-17T06:57:00.935 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph-authtool: stdout AQA8KOlmLPKPERAAHOVb3VhrqmOJlnObwOzUOA== 2024-09-17T06:57:01.806 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph-authtool: stdout AQA9KOlmEKn8DRAABZNW5tE5EpRFX/skbQ1OCg== 2024-09-17T06:57:01.807 INFO:teuthology.orchestra.run.smithi100.stdout:Creating initial monmap... 2024-09-17T06:57:02.747 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-17T06:57:02.747 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = pacific 2024-09-17T06:57:02.747 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to e848267c-74c1-11ef-bceb-c7b262605968 2024-09-17T06:57:02.747 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-17T06:57:02.747 INFO:teuthology.orchestra.run.smithi100.stdout:monmaptool for a [v2:172.21.15.100:3300,v1:172.21.15.100:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-17T06:57:02.747 INFO:teuthology.orchestra.run.smithi100.stdout:setting min_mon_release = pacific 2024-09-17T06:57:02.747 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/monmaptool: set fsid to e848267c-74c1-11ef-bceb-c7b262605968 2024-09-17T06:57:02.747 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-17T06:57:02.747 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:57:02.747 INFO:teuthology.orchestra.run.smithi100.stdout:Creating mon... 2024-09-17T06:57:03.568 INFO:teuthology.orchestra.run.smithi100.stdout:create mon.a on 2024-09-17T06:57:03.771 INFO:teuthology.orchestra.run.smithi100.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-09-17T06:57:03.912 INFO:teuthology.orchestra.run.smithi100.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-09-17T06:57:04.108 INFO:teuthology.orchestra.run.smithi100.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-e848267c-74c1-11ef-bceb-c7b262605968.target → /etc/systemd/system/ceph-e848267c-74c1-11ef-bceb-c7b262605968.target. 2024-09-17T06:57:04.109 INFO:teuthology.orchestra.run.smithi100.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-e848267c-74c1-11ef-bceb-c7b262605968.target → /etc/systemd/system/ceph-e848267c-74c1-11ef-bceb-c7b262605968.target. 2024-09-17T06:57:04.344 INFO:teuthology.orchestra.run.smithi100.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-e848267c-74c1-11ef-bceb-c7b262605968@mon.a 2024-09-17T06:57:04.345 INFO:teuthology.orchestra.run.smithi100.stdout:systemctl: stderr Failed to reset failed state of unit ceph-e848267c-74c1-11ef-bceb-c7b262605968@mon.a.service: Unit ceph-e848267c-74c1-11ef-bceb-c7b262605968@mon.a.service not loaded. 2024-09-17T06:57:04.527 INFO:teuthology.orchestra.run.smithi100.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-e848267c-74c1-11ef-bceb-c7b262605968.target.wants/ceph-e848267c-74c1-11ef-bceb-c7b262605968@mon.a.service → /etc/systemd/system/ceph-e848267c-74c1-11ef-bceb-c7b262605968@.service. 2024-09-17T06:57:05.071 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:05 smithi100 ceph-mon[28725]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-17T06:57:05.071 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:05 smithi100 bash[28705]: f946d5818d0940e50f268be5a2706a9edfec99f50acc69dc9568d9d5eccd55ec 2024-09-17T06:57:05.085 INFO:teuthology.orchestra.run.smithi100.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-17T06:57:05.085 INFO:teuthology.orchestra.run.smithi100.stdout:systemctl: stdout disabled 2024-09-17T06:57:05.093 INFO:teuthology.orchestra.run.smithi100.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-17T06:57:05.093 INFO:teuthology.orchestra.run.smithi100.stdout:systemctl: stdout inactive 2024-09-17T06:57:05.093 INFO:teuthology.orchestra.run.smithi100.stdout:firewalld.service is not enabled 2024-09-17T06:57:05.093 INFO:teuthology.orchestra.run.smithi100.stdout:Not possible to enable service . firewalld.service is not available 2024-09-17T06:57:05.093 INFO:teuthology.orchestra.run.smithi100.stdout:Waiting for mon to start... 2024-09-17T06:57:05.093 INFO:teuthology.orchestra.run.smithi100.stdout:Waiting for mon... 2024-09-17T06:57:05.336 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:05 smithi100 systemd[1]: Started Ceph mon.a for e848267c-74c1-11ef-bceb-c7b262605968. 2024-09-17T06:57:06.219 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:06 smithi100 ceph-mon[28725]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-17T06:57:06.220 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:06 smithi100 ceph-mon[28725]: monmap e1: 1 mons at {a=[v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-17T06:57:06.220 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:06 smithi100 ceph-mon[28725]: fsmap 2024-09-17T06:57:06.220 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:06 smithi100 ceph-mon[28725]: osdmap e1: 0 total, 0 up, 0 in 2024-09-17T06:57:06.220 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:06 smithi100 ceph-mon[28725]: mgrmap e1: no daemons active 2024-09-17T06:57:06.220 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:06 smithi100 ceph-mon[28725]: from='client.? 172.21.15.100:0/4068929881' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-09-17T06:57:06.223 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout cluster: 2024-09-17T06:57:06.223 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout id: e848267c-74c1-11ef-bceb-c7b262605968 2024-09-17T06:57:06.223 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-09-17T06:57:06.223 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout 2024-09-17T06:57:06.223 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout services: 2024-09-17T06:57:06.224 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.627175s) 2024-09-17T06:57:06.224 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-09-17T06:57:06.224 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-09-17T06:57:06.224 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout 2024-09-17T06:57:06.224 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout data: 2024-09-17T06:57:06.224 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-09-17T06:57:06.224 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-09-17T06:57:06.224 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-09-17T06:57:06.224 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout pgs: 2024-09-17T06:57:06.224 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout 2024-09-17T06:57:06.224 INFO:teuthology.orchestra.run.smithi100.stdout:mon is available 2024-09-17T06:57:06.224 INFO:teuthology.orchestra.run.smithi100.stdout:Assimilating anything we can from ceph.conf... 2024-09-17T06:57:07.245 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout 2024-09-17T06:57:07.245 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout [global] 2024-09-17T06:57:07.246 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout fsid = e848267c-74c1-11ef-bceb-c7b262605968 2024-09-17T06:57:07.246 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.100:3300,v1:172.21.15.100:6789] 2024-09-17T06:57:07.246 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-17T06:57:07.246 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-17T06:57:07.246 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-17T06:57:07.246 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-17T06:57:07.246 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout 2024-09-17T06:57:07.246 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-17T06:57:07.246 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-17T06:57:07.246 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout 2024-09-17T06:57:07.246 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout [osd] 2024-09-17T06:57:07.246 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-17T06:57:07.246 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-17T06:57:07.247 INFO:teuthology.orchestra.run.smithi100.stdout:Generating new minimal ceph.conf... 2024-09-17T06:57:07.366 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:07 smithi100 ceph-mon[28725]: from='client.? 172.21.15.100:0/1236357077' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-09-17T06:57:07.367 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:07 smithi100 ceph-mon[28725]: from='client.? 172.21.15.100:0/1236357077' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-09-17T06:57:08.240 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:08 smithi100 ceph-mon[28725]: from='client.? 172.21.15.100:0/868914385' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T06:57:08.243 INFO:teuthology.orchestra.run.smithi100.stdout:Restarting the monitor... 2024-09-17T06:57:08.519 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:08 smithi100 systemd[1]: Stopping Ceph mon.a for e848267c-74c1-11ef-bceb-c7b262605968... 2024-09-17T06:57:08.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:08 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mon-a[28721]: 2024-09-17T06:57:08.517+0000 7f51795d7640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T06:57:08.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:08 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mon-a[28721]: 2024-09-17T06:57:08.517+0000 7f51795d7640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-09-17T06:57:08.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:08 smithi100 podman[28968]: 2024-09-17 06:57:08.722677038 +0000 UTC m=+0.318496810 container died f946d5818d0940e50f268be5a2706a9edfec99f50acc69dc9568d9d5eccd55ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-mon-a, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-19e2a2b, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-09-17T06:57:09.252 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 podman[28968]: 2024-09-17 06:57:09.059340751 +0000 UTC m=+0.655160533 container cleanup f946d5818d0940e50f268be5a2706a9edfec99f50acc69dc9568d9d5eccd55ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-mon-a, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-19e2a2b, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, ceph=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T06:57:09.252 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 bash[28968]: ceph-e848267c-74c1-11ef-bceb-c7b262605968-mon-a 2024-09-17T06:57:09.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 podman[28980]: 2024-09-17 06:57:09.251938121 +0000 UTC m=+0.523411926 container remove f946d5818d0940e50f268be5a2706a9edfec99f50acc69dc9568d9d5eccd55ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-mon-a, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=reef-19e2a2b) 2024-09-17T06:57:09.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 systemd[1]: ceph-e848267c-74c1-11ef-bceb-c7b262605968@mon.a.service: Deactivated successfully. 2024-09-17T06:57:09.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 systemd[1]: Stopped Ceph mon.a for e848267c-74c1-11ef-bceb-c7b262605968. 2024-09-17T06:57:09.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 systemd[1]: Starting Ceph mon.a for e848267c-74c1-11ef-bceb-c7b262605968... 2024-09-17T06:57:09.945 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 podman[29073]: 2024-09-17 06:57:09.59732513 +0000 UTC m=+0.015387426 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-17T06:57:09.945 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 podman[29073]: 2024-09-17 06:57:09.736155935 +0000 UTC m=+0.154218215 container create a3d283eb23fd0c82f28bb7d58bbf50082bad946543cec84a656a561fb69afffe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-mon-a, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=reef-19e2a2b, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731) 2024-09-17T06:57:09.945 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 podman[29073]: 2024-09-17 06:57:09.913492737 +0000 UTC m=+0.331555026 container init a3d283eb23fd0c82f28bb7d58bbf50082bad946543cec84a656a561fb69afffe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=reef-19e2a2b, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS) 2024-09-17T06:57:09.945 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 podman[29073]: 2024-09-17 06:57:09.916548424 +0000 UTC m=+0.334610705 container start a3d283eb23fd0c82f28bb7d58bbf50082bad946543cec84a656a561fb69afffe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-mon-a, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=reef-19e2a2b) 2024-09-17T06:57:09.946 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: set uid:gid to 167:167 (ceph:ceph) 2024-09-17T06:57:09.946 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable), process ceph-mon, pid 2 2024-09-17T06:57:09.946 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: pidfile_write: ignore empty --pid-file 2024-09-17T06:57:09.946 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: load: jerasure load: lrc 2024-09-17T06:57:09.946 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: RocksDB version: 7.9.2 2024-09-17T06:57:09.946 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Git sha 0 2024-09-17T06:57:09.946 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Compile date 2024-09-16 15:15:13 2024-09-17T06:57:09.946 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: DB SUMMARY 2024-09-17T06:57:09.946 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: DB Session ID: M53P8RKPRBTOCB851Y0E 2024-09-17T06:57:09.946 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: CURRENT file: CURRENT 2024-09-17T06:57:09.946 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: IDENTITY file: IDENTITY 2024-09-17T06:57:09.946 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-09-17T06:57:09.947 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-09-17T06:57:09.947 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 90838 ; 2024-09-17T06:57:09.947 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.error_if_exists: 0 2024-09-17T06:57:09.947 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.create_if_missing: 0 2024-09-17T06:57:09.947 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.paranoid_checks: 1 2024-09-17T06:57:09.947 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-17T06:57:09.947 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-17T06:57:09.947 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-17T06:57:09.947 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.env: 0x560f2bb84bc0 2024-09-17T06:57:09.947 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.fs: PosixFileSystem 2024-09-17T06:57:09.947 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.info_log: 0x560f2d200be0 2024-09-17T06:57:09.947 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.max_file_opening_threads: 16 2024-09-17T06:57:09.948 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.statistics: (nil) 2024-09-17T06:57:09.948 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.use_fsync: 0 2024-09-17T06:57:09.948 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.max_log_file_size: 0 2024-09-17T06:57:09.948 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-17T06:57:09.948 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-17T06:57:09.948 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.keep_log_file_num: 1000 2024-09-17T06:57:09.948 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.recycle_log_file_num: 0 2024-09-17T06:57:09.948 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.allow_fallocate: 1 2024-09-17T06:57:09.948 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.allow_mmap_reads: 0 2024-09-17T06:57:09.948 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.allow_mmap_writes: 0 2024-09-17T06:57:09.948 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.use_direct_reads: 0 2024-09-17T06:57:09.948 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-17T06:57:09.948 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.create_missing_column_families: 0 2024-09-17T06:57:09.948 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.db_log_dir: 2024-09-17T06:57:09.949 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.wal_dir: 2024-09-17T06:57:09.949 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-17T06:57:09.949 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-17T06:57:09.949 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-17T06:57:09.949 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-17T06:57:09.949 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-17T06:57:09.949 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-17T06:57:09.949 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.advise_random_on_open: 1 2024-09-17T06:57:09.949 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.db_write_buffer_size: 0 2024-09-17T06:57:09.949 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.write_buffer_manager: 0x560f2d20e320 2024-09-17T06:57:09.949 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-17T06:57:09.949 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-17T06:57:09.949 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-17T06:57:09.949 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.rate_limiter: (nil) 2024-09-17T06:57:09.949 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-17T06:57:09.950 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.wal_recovery_mode: 2 2024-09-17T06:57:09.951 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.enable_thread_tracking: 0 2024-09-17T06:57:09.951 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.enable_pipelined_write: 0 2024-09-17T06:57:09.951 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.unordered_write: 0 2024-09-17T06:57:09.951 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-17T06:57:09.951 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-17T06:57:09.951 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-17T06:57:09.951 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-17T06:57:09.951 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.row_cache: None 2024-09-17T06:57:09.951 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.wal_filter: None 2024-09-17T06:57:09.951 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-17T06:57:09.951 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.allow_ingest_behind: 0 2024-09-17T06:57:09.952 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.two_write_queues: 0 2024-09-17T06:57:09.952 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.manual_wal_flush: 0 2024-09-17T06:57:09.952 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.wal_compression: 0 2024-09-17T06:57:09.952 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.atomic_flush: 0 2024-09-17T06:57:09.952 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-17T06:57:09.952 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-17T06:57:09.952 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-17T06:57:09.952 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.log_readahead_size: 0 2024-09-17T06:57:09.952 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-17T06:57:09.952 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.best_efforts_recovery: 0 2024-09-17T06:57:09.952 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-17T06:57:09.952 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-17T06:57:09.952 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.allow_data_in_errors: 0 2024-09-17T06:57:09.953 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.db_host_id: __hostname__ 2024-09-17T06:57:09.953 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-17T06:57:09.953 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.max_background_jobs: 2 2024-09-17T06:57:09.953 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.max_background_compactions: -1 2024-09-17T06:57:09.953 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.max_subcompactions: 1 2024-09-17T06:57:09.953 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-17T06:57:09.953 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-17T06:57:09.953 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-17T06:57:09.953 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.max_total_wal_size: 0 2024-09-17T06:57:09.953 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-17T06:57:09.953 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-17T06:57:09.953 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-17T06:57:09.953 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-17T06:57:09.954 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.max_open_files: -1 2024-09-17T06:57:09.954 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.bytes_per_sync: 0 2024-09-17T06:57:09.954 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-17T06:57:09.954 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-17T06:57:09.954 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.compaction_readahead_size: 0 2024-09-17T06:57:09.954 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.max_background_flushes: -1 2024-09-17T06:57:09.954 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Compression algorithms supported: 2024-09-17T06:57:09.954 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: kZSTD supported: 0 2024-09-17T06:57:09.954 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: kXpressCompression supported: 0 2024-09-17T06:57:09.954 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: kBZip2Compression supported: 0 2024-09-17T06:57:09.954 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-17T06:57:09.954 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: kLZ4Compression supported: 1 2024-09-17T06:57:09.954 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: kZlibCompression supported: 1 2024-09-17T06:57:09.955 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: kLZ4HCCompression supported: 1 2024-09-17T06:57:09.955 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: kSnappyCompression supported: 1 2024-09-17T06:57:09.955 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-17T06:57:09.955 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-17T06:57:09.955 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-09-17T06:57:09.955 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-17T06:57:09.955 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-17T06:57:09.955 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.merge_operator: 2024-09-17T06:57:09.955 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.compaction_filter: None 2024-09-17T06:57:09.955 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.compaction_filter_factory: None 2024-09-17T06:57:09.955 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.sst_partitioner_factory: None 2024-09-17T06:57:09.955 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-17T06:57:09.955 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-17T06:57:09.955 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x560f2d2007e0) 2024-09-17T06:57:09.956 INFO:journalctl@ceph.mon.a.smithi100.stdout: cache_index_and_filter_blocks: 1 2024-09-17T06:57:09.956 INFO:journalctl@ceph.mon.a.smithi100.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-17T06:57:09.956 INFO:journalctl@ceph.mon.a.smithi100.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-17T06:57:09.956 INFO:journalctl@ceph.mon.a.smithi100.stdout: pin_top_level_index_and_filter: 1 2024-09-17T06:57:09.956 INFO:journalctl@ceph.mon.a.smithi100.stdout: index_type: 0 2024-09-17T06:57:09.956 INFO:journalctl@ceph.mon.a.smithi100.stdout: data_block_index_type: 0 2024-09-17T06:57:09.956 INFO:journalctl@ceph.mon.a.smithi100.stdout: index_shortening: 1 2024-09-17T06:57:09.956 INFO:journalctl@ceph.mon.a.smithi100.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-17T06:57:09.956 INFO:journalctl@ceph.mon.a.smithi100.stdout: checksum: 4 2024-09-17T06:57:09.956 INFO:journalctl@ceph.mon.a.smithi100.stdout: no_block_cache: 0 2024-09-17T06:57:09.956 INFO:journalctl@ceph.mon.a.smithi100.stdout: block_cache: 0x560f2d1f5350 2024-09-17T06:57:09.956 INFO:journalctl@ceph.mon.a.smithi100.stdout: block_cache_name: BinnedLRUCache 2024-09-17T06:57:09.956 INFO:journalctl@ceph.mon.a.smithi100.stdout: block_cache_options: 2024-09-17T06:57:09.956 INFO:journalctl@ceph.mon.a.smithi100.stdout: capacity : 536870912 2024-09-17T06:57:09.956 INFO:journalctl@ceph.mon.a.smithi100.stdout: num_shard_bits : 4 2024-09-17T06:57:09.957 INFO:journalctl@ceph.mon.a.smithi100.stdout: strict_capacity_limit : 0 2024-09-17T06:57:09.957 INFO:journalctl@ceph.mon.a.smithi100.stdout: high_pri_pool_ratio: 0.000 2024-09-17T06:57:09.957 INFO:journalctl@ceph.mon.a.smithi100.stdout: block_cache_compressed: (nil) 2024-09-17T06:57:09.957 INFO:journalctl@ceph.mon.a.smithi100.stdout: persistent_cache: (nil) 2024-09-17T06:57:09.957 INFO:journalctl@ceph.mon.a.smithi100.stdout: block_size: 4096 2024-09-17T06:57:09.957 INFO:journalctl@ceph.mon.a.smithi100.stdout: block_size_deviation: 10 2024-09-17T06:57:09.957 INFO:journalctl@ceph.mon.a.smithi100.stdout: block_restart_interval: 16 2024-09-17T06:57:09.957 INFO:journalctl@ceph.mon.a.smithi100.stdout: index_block_restart_interval: 1 2024-09-17T06:57:09.957 INFO:journalctl@ceph.mon.a.smithi100.stdout: metadata_block_size: 4096 2024-09-17T06:57:09.957 INFO:journalctl@ceph.mon.a.smithi100.stdout: partition_filters: 0 2024-09-17T06:57:09.957 INFO:journalctl@ceph.mon.a.smithi100.stdout: use_delta_encoding: 1 2024-09-17T06:57:09.957 INFO:journalctl@ceph.mon.a.smithi100.stdout: filter_policy: bloomfilter 2024-09-17T06:57:09.957 INFO:journalctl@ceph.mon.a.smithi100.stdout: whole_key_filtering: 1 2024-09-17T06:57:09.958 INFO:journalctl@ceph.mon.a.smithi100.stdout: verify_compression: 0 2024-09-17T06:57:09.958 INFO:journalctl@ceph.mon.a.smithi100.stdout: read_amp_bytes_per_bit: 0 2024-09-17T06:57:09.958 INFO:journalctl@ceph.mon.a.smithi100.stdout: format_version: 5 2024-09-17T06:57:09.958 INFO:journalctl@ceph.mon.a.smithi100.stdout: enable_index_compression: 1 2024-09-17T06:57:09.958 INFO:journalctl@ceph.mon.a.smithi100.stdout: block_align: 0 2024-09-17T06:57:09.958 INFO:journalctl@ceph.mon.a.smithi100.stdout: max_auto_readahead_size: 262144 2024-09-17T06:57:09.958 INFO:journalctl@ceph.mon.a.smithi100.stdout: prepopulate_block_cache: 0 2024-09-17T06:57:09.958 INFO:journalctl@ceph.mon.a.smithi100.stdout: initial_auto_readahead_size: 8192 2024-09-17T06:57:09.958 INFO:journalctl@ceph.mon.a.smithi100.stdout: num_file_reads_for_auto_readahead: 2 2024-09-17T06:57:09.958 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.write_buffer_size: 33554432 2024-09-17T06:57:09.958 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.max_write_buffer_number: 2 2024-09-17T06:57:09.958 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.compression: NoCompression 2024-09-17T06:57:09.958 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.bottommost_compression: Disabled 2024-09-17T06:57:09.958 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.prefix_extractor: nullptr 2024-09-17T06:57:09.958 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-17T06:57:09.959 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.num_levels: 7 2024-09-17T06:57:09.959 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-17T06:57:09.959 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-17T06:57:09.959 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-17T06:57:09.959 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-17T06:57:09.959 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-17T06:57:09.959 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-17T06:57:09.959 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-17T06:57:09.959 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-17T06:57:09.959 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-17T06:57:09.959 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-17T06:57:09.959 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-17T06:57:09.960 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-17T06:57:09.960 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-17T06:57:09.960 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.compression_opts.level: 32767 2024-09-17T06:57:09.960 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.compression_opts.strategy: 0 2024-09-17T06:57:09.960 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-17T06:57:09.960 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-17T06:57:09.960 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-17T06:57:09.960 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-17T06:57:09.960 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.compression_opts.enabled: false 2024-09-17T06:57:09.960 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-17T06:57:09.960 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-17T06:57:09.960 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-17T06:57:09.960 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-17T06:57:09.961 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.target_file_size_base: 67108864 2024-09-17T06:57:09.961 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-17T06:57:09.961 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-17T06:57:09.961 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-17T06:57:09.961 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-17T06:57:09.961 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-17T06:57:09.961 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-17T06:57:09.961 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-17T06:57:09.961 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-17T06:57:09.961 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-17T06:57:09.961 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-17T06:57:09.961 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-17T06:57:09.961 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-17T06:57:09.961 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-17T06:57:09.962 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-17T06:57:09.962 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.arena_block_size: 1048576 2024-09-17T06:57:09.962 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-17T06:57:09.962 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-17T06:57:09.962 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.disable_auto_compactions: 0 2024-09-17T06:57:09.962 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-17T06:57:09.962 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-17T06:57:09.962 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-17T06:57:09.962 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-17T06:57:09.962 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-17T06:57:09.962 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-17T06:57:09.962 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-17T06:57:09.962 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-17T06:57:09.962 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-17T06:57:09.963 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-17T06:57:09.963 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-17T06:57:09.963 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.inplace_update_support: 0 2024-09-17T06:57:09.963 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-17T06:57:09.963 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-17T06:57:09.963 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-17T06:57:09.963 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-17T06:57:09.963 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.bloom_locality: 0 2024-09-17T06:57:09.963 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.max_successive_merges: 0 2024-09-17T06:57:09.963 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-17T06:57:09.963 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.paranoid_file_checks: 0 2024-09-17T06:57:09.963 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.force_consistency_checks: 1 2024-09-17T06:57:09.963 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.report_bg_io_stats: 0 2024-09-17T06:57:09.964 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.ttl: 2592000 2024-09-17T06:57:09.964 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-17T06:57:09.964 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-17T06:57:09.964 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-17T06:57:09.964 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.enable_blob_files: false 2024-09-17T06:57:09.964 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.min_blob_size: 0 2024-09-17T06:57:09.964 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.blob_file_size: 268435456 2024-09-17T06:57:09.964 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-17T06:57:09.964 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-17T06:57:09.964 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-17T06:57:09.964 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-17T06:57:09.964 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-17T06:57:09.965 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.blob_file_starting_level: 0 2024-09-17T06:57:09.965 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-17T06:57:09.965 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-09-17T06:57:09.965 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-09-17T06:57:09.965 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 528f6229-09d8-4816-82c6-cce9fbe2c046 2024-09-17T06:57:09.965 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726556229942448, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-09-17T06:57:09.965 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-09-17T06:57:09.965 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726556229943342, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 86825, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 280, "table_properties": {"data_size": 84887, "index_size": 258, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 645, "raw_key_size": 11671, "raw_average_key_size": 48, "raw_value_size": 78455, "raw_average_value_size": 328, "num_data_blocks": 11, "num_entries": 239, "num_filter_entries": 239, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726556229, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "528f6229-09d8-4816-82c6-cce9fbe2c046", "db_session_id": "M53P8RKPRBTOCB851Y0E", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-09-17T06:57:09.965 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726556229943438, "job": 1, "event": "recovery_finished"} 2024-09-17T06:57:09.965 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-09-17T06:57:09.965 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-17T06:57:09.965 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x560f2d220e00 2024-09-17T06:57:09.965 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: DB pointer 0x560f2d314000 2024-09-17T06:57:09.965 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-17T06:57:09.966 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-17T06:57:09.966 INFO:journalctl@ceph.mon.a.smithi100.stdout: ** DB Stats ** 2024-09-17T06:57:09.966 INFO:journalctl@ceph.mon.a.smithi100.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-17T06:57:09.966 INFO:journalctl@ceph.mon.a.smithi100.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-17T06:57:09.966 INFO:journalctl@ceph.mon.a.smithi100.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-17T06:57:09.966 INFO:journalctl@ceph.mon.a.smithi100.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-17T06:57:09.966 INFO:journalctl@ceph.mon.a.smithi100.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-17T06:57:09.966 INFO:journalctl@ceph.mon.a.smithi100.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-17T06:57:09.966 INFO:journalctl@ceph.mon.a.smithi100.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-17T06:57:09.966 INFO:journalctl@ceph.mon.a.smithi100.stdout: 2024-09-17T06:57:09.966 INFO:journalctl@ceph.mon.a.smithi100.stdout: ** Compaction Stats [default] ** 2024-09-17T06:57:09.966 INFO:journalctl@ceph.mon.a.smithi100.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-17T06:57:09.966 INFO:journalctl@ceph.mon.a.smithi100.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-17T06:57:09.966 INFO:journalctl@ceph.mon.a.smithi100.stdout: L0 2/0 86.65 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 148.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-17T06:57:09.966 INFO:journalctl@ceph.mon.a.smithi100.stdout: Sum 2/0 86.65 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 148.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-17T06:57:09.967 INFO:journalctl@ceph.mon.a.smithi100.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 148.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-17T06:57:09.967 INFO:journalctl@ceph.mon.a.smithi100.stdout: 2024-09-17T06:57:09.967 INFO:journalctl@ceph.mon.a.smithi100.stdout: ** Compaction Stats [default] ** 2024-09-17T06:57:09.967 INFO:journalctl@ceph.mon.a.smithi100.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-17T06:57:09.967 INFO:journalctl@ceph.mon.a.smithi100.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-17T06:57:09.967 INFO:journalctl@ceph.mon.a.smithi100.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 148.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-17T06:57:09.967 INFO:journalctl@ceph.mon.a.smithi100.stdout: 2024-09-17T06:57:09.967 INFO:journalctl@ceph.mon.a.smithi100.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-17T06:57:09.967 INFO:journalctl@ceph.mon.a.smithi100.stdout: 2024-09-17T06:57:09.967 INFO:journalctl@ceph.mon.a.smithi100.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-17T06:57:09.967 INFO:journalctl@ceph.mon.a.smithi100.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-17T06:57:09.967 INFO:journalctl@ceph.mon.a.smithi100.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-17T06:57:09.967 INFO:journalctl@ceph.mon.a.smithi100.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-17T06:57:09.967 INFO:journalctl@ceph.mon.a.smithi100.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-17T06:57:09.967 INFO:journalctl@ceph.mon.a.smithi100.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-17T06:57:09.968 INFO:journalctl@ceph.mon.a.smithi100.stdout: Cumulative compaction: 0.00 GB write, 28.14 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-17T06:57:09.968 INFO:journalctl@ceph.mon.a.smithi100.stdout: Interval compaction: 0.00 GB write, 28.14 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-17T06:57:09.968 INFO:journalctl@ceph.mon.a.smithi100.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-17T06:57:09.968 INFO:journalctl@ceph.mon.a.smithi100.stdout: Block cache BinnedLRUCache@0x560f2d1f5350#2 capacity: 512.00 MB usage: 1.30 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-09-17T06:57:09.968 INFO:journalctl@ceph.mon.a.smithi100.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.83 KB,0.000157952%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-09-17T06:57:09.968 INFO:journalctl@ceph.mon.a.smithi100.stdout: 2024-09-17T06:57:09.968 INFO:journalctl@ceph.mon.a.smithi100.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-17T06:57:09.968 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: starting mon.a rank 0 at public addrs [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] at bind addrs [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid e848267c-74c1-11ef-bceb-c7b262605968 2024-09-17T06:57:09.968 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: mon.a@-1(???) e1 preinit fsid e848267c-74c1-11ef-bceb-c7b262605968 2024-09-17T06:57:09.968 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: mon.a@-1(???).mds e1 new map 2024-09-17T06:57:09.968 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: mon.a@-1(???).mds e1 print_map 2024-09-17T06:57:09.968 INFO:journalctl@ceph.mon.a.smithi100.stdout: e1 2024-09-17T06:57:09.968 INFO:journalctl@ceph.mon.a.smithi100.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-17T06:57:09.968 INFO:journalctl@ceph.mon.a.smithi100.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-17T06:57:09.968 INFO:journalctl@ceph.mon.a.smithi100.stdout: legacy client fscid: -1 2024-09-17T06:57:09.969 INFO:journalctl@ceph.mon.a.smithi100.stdout: 2024-09-17T06:57:09.969 INFO:journalctl@ceph.mon.a.smithi100.stdout: No filesystems configured 2024-09-17T06:57:09.969 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-17T06:57:09.969 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-17T06:57:09.969 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-17T06:57:09.969 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-17T06:57:09.972 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 ceph-mon[29092]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-09-17T06:57:09.997 INFO:teuthology.orchestra.run.smithi100.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-09-17T06:57:10.207 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 bash[29073]: a3d283eb23fd0c82f28bb7d58bbf50082bad946543cec84a656a561fb69afffe 2024-09-17T06:57:10.207 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:09 smithi100 systemd[1]: Started Ceph mon.a for e848267c-74c1-11ef-bceb-c7b262605968. 2024-09-17T06:57:10.207 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:10 smithi100 ceph-mon[29092]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-17T06:57:10.207 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:10 smithi100 ceph-mon[29092]: monmap e1: 1 mons at {a=[v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-17T06:57:10.207 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:10 smithi100 ceph-mon[29092]: fsmap 2024-09-17T06:57:10.207 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:10 smithi100 ceph-mon[29092]: osdmap e1: 0 total, 0 up, 0 in 2024-09-17T06:57:10.207 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:10 smithi100 ceph-mon[29092]: mgrmap e1: no daemons active 2024-09-17T06:57:11.064 INFO:teuthology.orchestra.run.smithi100.stdout:Wrote config to /etc/ceph/ceph.conf 2024-09-17T06:57:11.086 INFO:teuthology.orchestra.run.smithi100.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-09-17T06:57:11.086 INFO:teuthology.orchestra.run.smithi100.stdout:Creating mgr... 2024-09-17T06:57:11.086 INFO:teuthology.orchestra.run.smithi100.stdout:Verifying port 0.0.0.0:9283 ... 2024-09-17T06:57:11.087 INFO:teuthology.orchestra.run.smithi100.stdout:Verifying port 0.0.0.0:8765 ... 2024-09-17T06:57:11.357 INFO:teuthology.orchestra.run.smithi100.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-e848267c-74c1-11ef-bceb-c7b262605968@mgr.y 2024-09-17T06:57:11.357 INFO:teuthology.orchestra.run.smithi100.stdout:systemctl: stderr Failed to reset failed state of unit ceph-e848267c-74c1-11ef-bceb-c7b262605968@mgr.y.service: Unit ceph-e848267c-74c1-11ef-bceb-c7b262605968@mgr.y.service not loaded. 2024-09-17T06:57:11.511 INFO:teuthology.orchestra.run.smithi100.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-e848267c-74c1-11ef-bceb-c7b262605968.target.wants/ceph-e848267c-74c1-11ef-bceb-c7b262605968@mgr.y.service → /etc/systemd/system/ceph-e848267c-74c1-11ef-bceb-c7b262605968@.service. 2024-09-17T06:57:11.739 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:11 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2503014010' entity='client.admin' 2024-09-17T06:57:11.740 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:11 smithi100 systemd[1]: Starting Ceph mgr.y for e848267c-74c1-11ef-bceb-c7b262605968... 2024-09-17T06:57:12.032 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:11 smithi100 podman[29303]: 2024-09-17 06:57:11.738635587 +0000 UTC m=+0.107470207 container create 31e6cec7f4a95628807709cbc3f407552dd8f298223f106a056cfe8b265cd5d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, RELEASE=reef-19e2a2b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS) 2024-09-17T06:57:12.033 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:11 smithi100 podman[29303]: 2024-09-17 06:57:11.646754386 +0000 UTC m=+0.015589010 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-17T06:57:12.033 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:11 smithi100 podman[29303]: 2024-09-17 06:57:11.932540079 +0000 UTC m=+0.301374706 container init 31e6cec7f4a95628807709cbc3f407552dd8f298223f106a056cfe8b265cd5d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=reef-19e2a2b, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-09-17T06:57:12.033 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:11 smithi100 podman[29303]: 2024-09-17 06:57:11.935292974 +0000 UTC m=+0.304127600 container start 31e6cec7f4a95628807709cbc3f407552dd8f298223f106a056cfe8b265cd5d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=reef-19e2a2b, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T06:57:12.033 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:12 smithi100 bash[29303]: 31e6cec7f4a95628807709cbc3f407552dd8f298223f106a056cfe8b265cd5d8 2024-09-17T06:57:12.046 INFO:teuthology.orchestra.run.smithi100.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-17T06:57:12.046 INFO:teuthology.orchestra.run.smithi100.stdout:systemctl: stdout disabled 2024-09-17T06:57:12.053 INFO:teuthology.orchestra.run.smithi100.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-17T06:57:12.053 INFO:teuthology.orchestra.run.smithi100.stdout:systemctl: stdout inactive 2024-09-17T06:57:12.053 INFO:teuthology.orchestra.run.smithi100.stdout:firewalld.service is not enabled 2024-09-17T06:57:12.053 INFO:teuthology.orchestra.run.smithi100.stdout:Not possible to enable service . firewalld.service is not available 2024-09-17T06:57:12.062 INFO:teuthology.orchestra.run.smithi100.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-17T06:57:12.062 INFO:teuthology.orchestra.run.smithi100.stdout:systemctl: stdout disabled 2024-09-17T06:57:12.069 INFO:teuthology.orchestra.run.smithi100.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-17T06:57:12.069 INFO:teuthology.orchestra.run.smithi100.stdout:systemctl: stdout inactive 2024-09-17T06:57:12.069 INFO:teuthology.orchestra.run.smithi100.stdout:firewalld.service is not enabled 2024-09-17T06:57:12.069 INFO:teuthology.orchestra.run.smithi100.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-09-17T06:57:12.069 INFO:teuthology.orchestra.run.smithi100.stdout:Waiting for mgr to start... 2024-09-17T06:57:12.070 INFO:teuthology.orchestra.run.smithi100.stdout:Waiting for mgr... 2024-09-17T06:57:12.308 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:12 smithi100 systemd[1]: Started Ceph mgr.y for e848267c-74c1-11ef-bceb-c7b262605968. 2024-09-17T06:57:12.308 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:12 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T06:57:12.053+0000 7fdbf0304040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-17T06:57:12.308 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:12 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T06:57:12.115+0000 7fdbf0304040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-17T06:57:12.308 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:12 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T06:57:12.265+0000 7fdbf0304040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-17T06:57:12.563 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:12 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T06:57:12.335+0000 7fdbf0304040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-17T06:57:12.563 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:12 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T06:57:12.482+0000 7fdbf0304040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-17T06:57:12.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:12 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1729205496' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-17T06:57:12.877 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:12 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T06:57:12.630+0000 7fdbf0304040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-17T06:57:13.134 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout 2024-09-17T06:57:13.136 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout { 2024-09-17T06:57:13.137 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "fsid": "e848267c-74c1-11ef-bceb-c7b262605968", 2024-09-17T06:57:13.137 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "health": { 2024-09-17T06:57:13.137 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-17T06:57:13.137 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-17T06:57:13.137 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-17T06:57:13.137 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout }, 2024-09-17T06:57:13.137 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-17T06:57:13.137 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-17T06:57:13.137 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout 0 2024-09-17T06:57:13.137 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout ], 2024-09-17T06:57:13.137 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-17T06:57:13.137 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "a" 2024-09-17T06:57:13.137 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout ], 2024-09-17T06:57:13.138 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-09-17T06:57:13.138 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-17T06:57:13.138 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-17T06:57:13.138 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-09-17T06:57:13.138 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-17T06:57:13.138 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout }, 2024-09-17T06:57:13.138 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-17T06:57:13.138 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-17T06:57:13.138 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-17T06:57:13.138 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-17T06:57:13.138 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-17T06:57:13.138 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-17T06:57:13.142 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-17T06:57:13.142 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-17T06:57:13.142 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout }, 2024-09-17T06:57:13.142 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-17T06:57:13.142 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-17T06:57:13.142 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-17T06:57:13.143 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-17T06:57:13.143 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-17T06:57:13.143 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-17T06:57:13.143 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-17T06:57:13.143 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-17T06:57:13.143 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-17T06:57:13.143 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout }, 2024-09-17T06:57:13.144 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-17T06:57:13.144 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-17T06:57:13.144 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-17T06:57:13.144 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-17T06:57:13.144 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout }, 2024-09-17T06:57:13.144 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-17T06:57:13.144 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-17T06:57:13.144 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-17T06:57:13.144 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-17T06:57:13.144 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-17T06:57:13.145 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-17T06:57:13.145 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "restful" 2024-09-17T06:57:13.145 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout ], 2024-09-17T06:57:13.145 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-17T06:57:13.145 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout }, 2024-09-17T06:57:13.145 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-17T06:57:13.145 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-17T06:57:13.145 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "modified": "2024-09-17T06:57:05.010970+0000", 2024-09-17T06:57:13.145 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-17T06:57:13.145 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout }, 2024-09-17T06:57:13.145 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-17T06:57:13.145 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout } 2024-09-17T06:57:13.145 INFO:teuthology.orchestra.run.smithi100.stdout:mgr not available, waiting (1/15)... 2024-09-17T06:57:13.187 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:12 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T06:57:12.941+0000 7fdbf0304040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-17T06:57:13.187 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:13 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T06:57:13.002+0000 7fdbf0304040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-17T06:57:13.626 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:13 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T06:57:13.186+0000 7fdbf0304040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-17T06:57:14.376 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:14 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T06:57:14.044+0000 7fdbf0304040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-17T06:57:14.376 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:14 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T06:57:14.108+0000 7fdbf0304040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-17T06:57:15.069 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:14 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T06:57:14.812+0000 7fdbf0304040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-17T06:57:15.069 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:14 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T06:57:14.930+0000 7fdbf0304040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-17T06:57:15.069 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:15 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T06:57:15.068+0000 7fdbf0304040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-17T06:57:15.355 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:15 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T06:57:15.128+0000 7fdbf0304040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-17T06:57:15.355 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:15 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T06:57:15.284+0000 7fdbf0304040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-17T06:57:15.355 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:15 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T06:57:15.353+0000 7fdbf0304040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-17T06:57:15.620 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:15 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T06:57:15.415+0000 7fdbf0304040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-17T06:57:15.620 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:15 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T06:57:15.582+0000 7fdbf0304040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-17T06:57:15.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:15 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/3210232603' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-17T06:57:15.877 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:15 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T06:57:15.652+0000 7fdbf0304040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-17T06:57:15.877 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:15 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T06:57:15.720+0000 7fdbf0304040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-17T06:57:15.877 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:15 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T06:57:15.802+0000 7fdbf0304040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-17T06:57:15.877 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:15 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T06:57:15.863+0000 7fdbf0304040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-17T06:57:16.200 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:15 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T06:57:15.923+0000 7fdbf0304040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-17T06:57:16.208 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout 2024-09-17T06:57:16.208 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout { 2024-09-17T06:57:16.208 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "fsid": "e848267c-74c1-11ef-bceb-c7b262605968", 2024-09-17T06:57:16.208 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "health": { 2024-09-17T06:57:16.208 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-17T06:57:16.208 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-17T06:57:16.208 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-17T06:57:16.208 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout }, 2024-09-17T06:57:16.209 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-17T06:57:16.209 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-17T06:57:16.209 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout 0 2024-09-17T06:57:16.209 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout ], 2024-09-17T06:57:16.209 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-17T06:57:16.209 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "a" 2024-09-17T06:57:16.209 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout ], 2024-09-17T06:57:16.209 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-09-17T06:57:16.209 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-17T06:57:16.209 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-17T06:57:16.209 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-09-17T06:57:16.209 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-17T06:57:16.209 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout }, 2024-09-17T06:57:16.209 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-17T06:57:16.210 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-17T06:57:16.210 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-17T06:57:16.210 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-17T06:57:16.210 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-17T06:57:16.210 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-17T06:57:16.210 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-17T06:57:16.210 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-17T06:57:16.210 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout }, 2024-09-17T06:57:16.210 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-17T06:57:16.210 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-17T06:57:16.210 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-17T06:57:16.211 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-17T06:57:16.211 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-17T06:57:16.211 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-17T06:57:16.211 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-17T06:57:16.211 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-17T06:57:16.211 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-17T06:57:16.211 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout }, 2024-09-17T06:57:16.211 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-17T06:57:16.211 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-17T06:57:16.211 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-17T06:57:16.211 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-17T06:57:16.211 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout }, 2024-09-17T06:57:16.211 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-17T06:57:16.211 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-17T06:57:16.212 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-17T06:57:16.212 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-17T06:57:16.212 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-17T06:57:16.212 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-17T06:57:16.212 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "restful" 2024-09-17T06:57:16.212 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout ], 2024-09-17T06:57:16.212 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-17T06:57:16.212 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout }, 2024-09-17T06:57:16.212 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-17T06:57:16.212 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-17T06:57:16.212 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "modified": "2024-09-17T06:57:05.010970+0000", 2024-09-17T06:57:16.212 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-17T06:57:16.212 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout }, 2024-09-17T06:57:16.213 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-17T06:57:16.213 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout } 2024-09-17T06:57:16.213 INFO:teuthology.orchestra.run.smithi100.stdout:mgr not available, waiting (2/15)... 2024-09-17T06:57:17.127 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:16 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T06:57:16.698+0000 7fdbf0304040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-17T06:57:17.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:16 smithi100 ceph-mon[29092]: Activating manager daemon y 2024-09-17T06:57:17.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:16 smithi100 ceph-mon[29092]: mgrmap e2: y(active, starting, since 0.00301675s) 2024-09-17T06:57:17.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:16 smithi100 ceph-mon[29092]: from='mgr.14100 172.21.15.100:0/1921281427' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-17T06:57:17.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:16 smithi100 ceph-mon[29092]: from='mgr.14100 172.21.15.100:0/1921281427' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-17T06:57:17.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:16 smithi100 ceph-mon[29092]: from='mgr.14100 172.21.15.100:0/1921281427' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-17T06:57:17.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:16 smithi100 ceph-mon[29092]: from='mgr.14100 172.21.15.100:0/1921281427' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T06:57:17.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:16 smithi100 ceph-mon[29092]: from='mgr.14100 172.21.15.100:0/1921281427' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-17T06:57:17.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:16 smithi100 ceph-mon[29092]: Manager daemon y is now available 2024-09-17T06:57:17.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:16 smithi100 ceph-mon[29092]: from='mgr.14100 172.21.15.100:0/1921281427' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T06:57:17.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:16 smithi100 ceph-mon[29092]: from='mgr.14100 172.21.15.100:0/1921281427' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T06:57:17.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:16 smithi100 ceph-mon[29092]: from='mgr.14100 172.21.15.100:0/1921281427' entity='mgr.y' 2024-09-17T06:57:17.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:16 smithi100 ceph-mon[29092]: from='mgr.14100 172.21.15.100:0/1921281427' entity='mgr.y' 2024-09-17T06:57:17.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:16 smithi100 ceph-mon[29092]: from='mgr.14100 172.21.15.100:0/1921281427' entity='mgr.y' 2024-09-17T06:57:19.126 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:18 smithi100 ceph-mon[29092]: mgrmap e3: y(active, since 1.00644s) 2024-09-17T06:57:19.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:19 smithi100 ceph-mon[29092]: mgrmap e4: y(active, since 2s) 2024-09-17T06:57:22.126 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:21 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2677558993' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-17T06:57:28.044 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout 2024-09-17T06:57:28.045 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout { 2024-09-17T06:57:28.045 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "fsid": "e848267c-74c1-11ef-bceb-c7b262605968", 2024-09-17T06:57:28.045 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "health": { 2024-09-17T06:57:28.045 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-17T06:57:28.045 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-17T06:57:28.045 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-17T06:57:28.045 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout }, 2024-09-17T06:57:28.045 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-17T06:57:28.045 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-17T06:57:28.045 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout 0 2024-09-17T06:57:28.045 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout ], 2024-09-17T06:57:28.045 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-17T06:57:28.046 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "a" 2024-09-17T06:57:28.046 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout ], 2024-09-17T06:57:28.046 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "quorum_age": 11, 2024-09-17T06:57:28.046 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-17T06:57:28.046 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-17T06:57:28.046 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-09-17T06:57:28.046 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-17T06:57:28.046 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout }, 2024-09-17T06:57:28.046 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-17T06:57:28.046 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-17T06:57:28.046 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-17T06:57:28.046 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-17T06:57:28.046 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-17T06:57:28.047 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-17T06:57:28.050 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-17T06:57:28.050 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-17T06:57:28.050 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout }, 2024-09-17T06:57:28.050 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-17T06:57:28.050 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-17T06:57:28.050 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-17T06:57:28.050 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-17T06:57:28.050 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-17T06:57:28.050 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-17T06:57:28.050 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-17T06:57:28.050 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-17T06:57:28.051 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-17T06:57:28.051 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout }, 2024-09-17T06:57:28.051 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-17T06:57:28.051 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-17T06:57:28.051 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-17T06:57:28.051 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-17T06:57:28.051 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout }, 2024-09-17T06:57:28.051 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-17T06:57:28.051 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-17T06:57:28.051 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-17T06:57:28.051 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-17T06:57:28.051 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-17T06:57:28.051 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-17T06:57:28.052 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "restful" 2024-09-17T06:57:28.052 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout ], 2024-09-17T06:57:28.052 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-17T06:57:28.052 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout }, 2024-09-17T06:57:28.052 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-17T06:57:28.052 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-17T06:57:28.052 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "modified": "2024-09-17T06:57:05.010970+0000", 2024-09-17T06:57:28.052 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-17T06:57:28.052 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout }, 2024-09-17T06:57:28.052 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-17T06:57:28.052 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout } 2024-09-17T06:57:28.052 INFO:teuthology.orchestra.run.smithi100.stdout:mgr is available 2024-09-17T06:57:29.960 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:29 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/195602902' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-09-17T06:57:30.223 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout 2024-09-17T06:57:30.223 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout [global] 2024-09-17T06:57:30.223 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout fsid = e848267c-74c1-11ef-bceb-c7b262605968 2024-09-17T06:57:30.223 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-17T06:57:30.223 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-17T06:57:30.224 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-17T06:57:30.224 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-17T06:57:30.224 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout 2024-09-17T06:57:30.224 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-17T06:57:30.224 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-17T06:57:30.224 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout 2024-09-17T06:57:30.224 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout [osd] 2024-09-17T06:57:30.224 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-17T06:57:30.224 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-17T06:57:30.224 INFO:teuthology.orchestra.run.smithi100.stdout:Enabling cephadm module... 2024-09-17T06:57:31.238 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:30 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2136465617' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-09-17T06:57:31.238 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:30 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ignoring --setuser ceph since I am not root 2024-09-17T06:57:31.238 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:30 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ignoring --setgroup ceph since I am not root 2024-09-17T06:57:31.238 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:31 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T06:57:31.034+0000 7f2158553040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-17T06:57:31.238 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:31 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T06:57:31.090+0000 7f2158553040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-17T06:57:31.238 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:31 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T06:57:31.238+0000 7f2158553040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-17T06:57:31.502 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:31 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T06:57:31.305+0000 7f2158553040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-17T06:57:31.503 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:31 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T06:57:31.452+0000 7f2158553040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-17T06:57:31.754 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:31 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T06:57:31.597+0000 7f2158553040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-17T06:57:32.126 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:31 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2136465617' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-09-17T06:57:32.126 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:31 smithi100 ceph-mon[29092]: mgrmap e5: y(active, since 14s) 2024-09-17T06:57:32.127 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:31 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T06:57:31.916+0000 7f2158553040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-17T06:57:32.127 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:31 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T06:57:31.981+0000 7f2158553040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-17T06:57:32.626 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:32 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T06:57:32.171+0000 7f2158553040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-17T06:57:33.043 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:32 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/3602608794' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-17T06:57:33.048 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout { 2024-09-17T06:57:33.048 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-09-17T06:57:33.048 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-17T06:57:33.049 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-09-17T06:57:33.049 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-17T06:57:33.049 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout } 2024-09-17T06:57:33.049 INFO:teuthology.orchestra.run.smithi100.stdout:Waiting for the mgr to restart... 2024-09-17T06:57:33.049 INFO:teuthology.orchestra.run.smithi100.stdout:Waiting for mgr epoch 5... 2024-09-17T06:57:33.373 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:33 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T06:57:33.042+0000 7f2158553040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-17T06:57:33.373 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:33 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T06:57:33.112+0000 7f2158553040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-17T06:57:34.084 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:33 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T06:57:33.830+0000 7f2158553040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-17T06:57:34.085 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:33 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T06:57:33.944+0000 7f2158553040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-17T06:57:34.354 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:34 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T06:57:34.083+0000 7f2158553040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-17T06:57:34.355 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:34 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T06:57:34.143+0000 7f2158553040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-17T06:57:34.355 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:34 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T06:57:34.287+0000 7f2158553040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-17T06:57:34.355 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:34 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T06:57:34.353+0000 7f2158553040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-17T06:57:34.626 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:34 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T06:57:34.411+0000 7f2158553040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-17T06:57:34.626 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:34 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T06:57:34.563+0000 7f2158553040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-17T06:57:34.897 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:34 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T06:57:34.626+0000 7f2158553040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-17T06:57:34.897 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:34 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T06:57:34.693+0000 7f2158553040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-17T06:57:34.897 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:34 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T06:57:34.776+0000 7f2158553040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-17T06:57:34.897 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:34 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T06:57:34.835+0000 7f2158553040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-17T06:57:34.897 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:34 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T06:57:34.896+0000 7f2158553040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-17T06:57:36.048 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:35 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T06:57:35.675+0000 7f2158553040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-17T06:57:36.049 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:35 smithi100 ceph-mon[29092]: Active manager daemon y restarted 2024-09-17T06:57:36.049 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:35 smithi100 ceph-mon[29092]: Activating manager daemon y 2024-09-17T06:57:36.049 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:35 smithi100 ceph-mon[29092]: osdmap e2: 0 total, 0 up, 0 in 2024-09-17T06:57:36.049 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:35 smithi100 ceph-mon[29092]: mgrmap e6: y(active, starting, since 0.00307563s) 2024-09-17T06:57:36.049 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:35 smithi100 ceph-mon[29092]: from='mgr.14118 172.21.15.100:0/1799257286' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T06:57:36.049 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:35 smithi100 ceph-mon[29092]: from='mgr.14118 172.21.15.100:0/1799257286' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-17T06:57:36.049 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:35 smithi100 ceph-mon[29092]: from='mgr.14118 172.21.15.100:0/1799257286' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-17T06:57:36.049 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:35 smithi100 ceph-mon[29092]: from='mgr.14118 172.21.15.100:0/1799257286' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-17T06:57:36.050 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:35 smithi100 ceph-mon[29092]: from='mgr.14118 172.21.15.100:0/1799257286' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-17T06:57:36.050 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:35 smithi100 ceph-mon[29092]: Manager daemon y is now available 2024-09-17T06:57:36.050 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:35 smithi100 ceph-mon[29092]: from='mgr.14118 172.21.15.100:0/1799257286' entity='mgr.y' 2024-09-17T06:57:36.050 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:35 smithi100 ceph-mon[29092]: from='mgr.14118 172.21.15.100:0/1799257286' entity='mgr.y' 2024-09-17T06:57:36.050 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:35 smithi100 ceph-mon[29092]: from='mgr.14118 172.21.15.100:0/1799257286' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T06:57:36.050 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:35 smithi100 ceph-mon[29092]: from='mgr.14118 172.21.15.100:0/1799257286' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T06:57:36.050 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:35 smithi100 ceph-mon[29092]: from='mgr.14118 172.21.15.100:0/1799257286' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T06:57:36.050 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:35 smithi100 ceph-mon[29092]: from='mgr.14118 172.21.15.100:0/1799257286' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T06:57:36.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:36 smithi100 ceph-mon[29092]: Found migration_current of "None". Setting to last migration. 2024-09-17T06:57:36.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:36 smithi100 ceph-mon[29092]: mgrmap e7: y(active, since 1.00522s) 2024-09-17T06:57:37.242 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout { 2024-09-17T06:57:37.242 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-09-17T06:57:37.242 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-17T06:57:37.242 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout } 2024-09-17T06:57:37.242 INFO:teuthology.orchestra.run.smithi100.stdout:mgr epoch 5 is available 2024-09-17T06:57:37.242 INFO:teuthology.orchestra.run.smithi100.stdout:Setting orchestrator backend to cephadm... 2024-09-17T06:57:37.815 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:37 smithi100 ceph-mon[29092]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-17T06:57:37.816 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:37 smithi100 ceph-mon[29092]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-17T06:57:37.816 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:37 smithi100 ceph-mon[29092]: from='mgr.14118 172.21.15.100:0/1799257286' entity='mgr.y' 2024-09-17T06:57:37.816 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:37 smithi100 ceph-mon[29092]: from='mgr.14118 172.21.15.100:0/1799257286' entity='mgr.y' 2024-09-17T06:57:38.120 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:37 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T06:57:37.814+0000 7f20c0233640 -1 log_channel(cephadm) log [ERR] : [17/Sep/2024:06:57:37] ENGINE Error in HTTPServer.serve 2024-09-17T06:57:38.120 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:37 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: Traceback (most recent call last): 2024-09-17T06:57:38.120 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:37 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T06:57:38.121 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:37 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: self._connections.run(self.expiration_interval) 2024-09-17T06:57:38.121 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:37 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T06:57:38.121 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:37 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: self._run(expiration_interval) 2024-09-17T06:57:38.121 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:37 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T06:57:38.121 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:37 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T06:57:38.121 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:37 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T06:57:38.121 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:37 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T06:57:38.121 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:37 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T06:57:38.121 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:37 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: s = self.context.wrap_socket( 2024-09-17T06:57:38.121 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:37 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T06:57:38.121 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:37 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: return self.sslsocket_class._create( 2024-09-17T06:57:38.121 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:37 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T06:57:38.122 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:37 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: self.do_handshake() 2024-09-17T06:57:38.122 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:37 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T06:57:38.122 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:37 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: self._sslobj.do_handshake() 2024-09-17T06:57:38.122 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:37 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T06:57:38.122 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:37 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T06:57:39.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:38 smithi100 ceph-mon[29092]: [17/Sep/2024:06:57:37] ENGINE Bus STARTING 2024-09-17T06:57:39.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:38 smithi100 ceph-mon[29092]: [17/Sep/2024:06:57:37] ENGINE Serving on http://172.21.15.100:8765 2024-09-17T06:57:39.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:38 smithi100 ceph-mon[29092]: mgrmap e8: y(active, since 2s) 2024-09-17T06:57:39.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:38 smithi100 ceph-mon[29092]: [17/Sep/2024:06:57:37] ENGINE Serving on https://172.21.15.100:7150 2024-09-17T06:57:39.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:38 smithi100 ceph-mon[29092]: [17/Sep/2024:06:57:37] ENGINE Bus STARTED 2024-09-17T06:57:39.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:38 smithi100 ceph-mon[29092]: from='mgr.14118 172.21.15.100:0/1799257286' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T06:57:39.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:38 smithi100 ceph-mon[29092]: [17/Sep/2024:06:57:37] ENGINE Error in HTTPServer.serve 2024-09-17T06:57:39.127 INFO:journalctl@ceph.mon.a.smithi100.stdout: Traceback (most recent call last): 2024-09-17T06:57:39.127 INFO:journalctl@ceph.mon.a.smithi100.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T06:57:39.127 INFO:journalctl@ceph.mon.a.smithi100.stdout: self._connections.run(self.expiration_interval) 2024-09-17T06:57:39.128 INFO:journalctl@ceph.mon.a.smithi100.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T06:57:39.128 INFO:journalctl@ceph.mon.a.smithi100.stdout: self._run(expiration_interval) 2024-09-17T06:57:39.128 INFO:journalctl@ceph.mon.a.smithi100.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T06:57:39.128 INFO:journalctl@ceph.mon.a.smithi100.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T06:57:39.128 INFO:journalctl@ceph.mon.a.smithi100.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T06:57:39.128 INFO:journalctl@ceph.mon.a.smithi100.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T06:57:39.128 INFO:journalctl@ceph.mon.a.smithi100.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T06:57:39.128 INFO:journalctl@ceph.mon.a.smithi100.stdout: s = self.context.wrap_socket( 2024-09-17T06:57:39.128 INFO:journalctl@ceph.mon.a.smithi100.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T06:57:39.128 INFO:journalctl@ceph.mon.a.smithi100.stdout: return self.sslsocket_class._create( 2024-09-17T06:57:39.128 INFO:journalctl@ceph.mon.a.smithi100.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T06:57:39.128 INFO:journalctl@ceph.mon.a.smithi100.stdout: self.do_handshake() 2024-09-17T06:57:39.129 INFO:journalctl@ceph.mon.a.smithi100.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T06:57:39.129 INFO:journalctl@ceph.mon.a.smithi100.stdout: self._sslobj.do_handshake() 2024-09-17T06:57:39.129 INFO:journalctl@ceph.mon.a.smithi100.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T06:57:39.129 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:38 smithi100 ceph-mon[29092]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T06:57:39.129 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:38 smithi100 ceph-mon[29092]: from='mgr.14118 172.21.15.100:0/1799257286' entity='mgr.y' 2024-09-17T06:57:39.129 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:38 smithi100 ceph-mon[29092]: from='mgr.14118 172.21.15.100:0/1799257286' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T06:57:39.496 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout value unchanged 2024-09-17T06:57:39.496 INFO:teuthology.orchestra.run.smithi100.stdout:Generating ssh key... 2024-09-17T06:57:40.126 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:39 smithi100 ceph-mon[29092]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T06:57:40.127 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:40 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: Generating public/private rsa key pair. 2024-09-17T06:57:40.127 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:40 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: Your identification has been saved in /tmp/tmpsq2ui_pn/key 2024-09-17T06:57:40.127 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:40 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: Your public key has been saved in /tmp/tmpsq2ui_pn/key.pub 2024-09-17T06:57:40.127 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:40 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: The key fingerprint is: 2024-09-17T06:57:40.127 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:40 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: SHA256:C8YYD51LVZ4sDsyyc2MXZuRhMrr8olQmbFL0IzXNgY4 ceph-e848267c-74c1-11ef-bceb-c7b262605968 2024-09-17T06:57:40.127 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:40 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: The key's randomart image is: 2024-09-17T06:57:40.127 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:40 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: +---[RSA 3072]----+ 2024-09-17T06:57:40.127 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:40 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: | . o++.=.. | 2024-09-17T06:57:40.127 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:40 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: | . o.*oO + . | 2024-09-17T06:57:40.128 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:40 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: | ooB B * + | 2024-09-17T06:57:40.128 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:40 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: | oEo.% * o | 2024-09-17T06:57:40.128 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:40 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: |. + O X S | 2024-09-17T06:57:40.128 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:40 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: | o + * + . | 2024-09-17T06:57:40.128 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:40 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: | . . . . | 2024-09-17T06:57:40.128 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:40 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: | . . . | 2024-09-17T06:57:40.128 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:40 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: | . | 2024-09-17T06:57:40.128 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:40 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: +----[SHA256]-----+ 2024-09-17T06:57:41.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:41 smithi100 ceph-mon[29092]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T06:57:41.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:41 smithi100 ceph-mon[29092]: Generating ssh key... 2024-09-17T06:57:41.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:41 smithi100 ceph-mon[29092]: from='mgr.14118 172.21.15.100:0/1799257286' entity='mgr.y' 2024-09-17T06:57:41.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:41 smithi100 ceph-mon[29092]: from='mgr.14118 172.21.15.100:0/1799257286' entity='mgr.y' 2024-09-17T06:57:42.252 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCg96Gp5/L0F9L+x8CpeiqizxjnM85HmlNGkWdYoi0rfZ6rJmVl3zDFtg/akxaNBTPTgMfxXp8X2lx536MAjaNnispQ2ZFTi3p31ZLR55Qm52rkM3LdbWqeXXEmEml7QFyyOnKD839BafpV3cSaZIZnuQuIfjbChghEbyDppOYRrkttKCQxysiEAq79xtDaqmcNvxVq7RZYveokAEwqX9d2yTiiGWjR2VNNWjG8ZSXRyCuEL/JsYR7x/946Wx3+z0b2n65wNBQw8zJTYXsuYAeHE2XDbds3yk8+kI3TDdGRpo8N5FQzQ2fFkL+K9W/UY0pVvnYyg6QJBqa2QrdXLHILNbgCFFxWxSLqZ+63qOnZhxvPiyUMsIf3j3FRE8Z07M1ydpAITOly8OTv1tbLTy6jHSM4GTOwY4P0308vNRfFX6BbOkyxR6yKav47AwZuO/v6D5MsP8qCm3SjE5SkMip5jiOA4YmNqFQE8L1UyZQC+GDt7+SYAzJMJbLNotZNCgc= ceph-e848267c-74c1-11ef-bceb-c7b262605968 2024-09-17T06:57:42.252 INFO:teuthology.orchestra.run.smithi100.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-09-17T06:57:42.252 INFO:teuthology.orchestra.run.smithi100.stdout:Adding key to root@localhost authorized_keys... 2024-09-17T06:57:42.261 INFO:teuthology.orchestra.run.smithi100.stdout:Adding host smithi100... 2024-09-17T06:57:42.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:42 smithi100 ceph-mon[29092]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T06:57:43.375 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:43 smithi100 ceph-mon[29092]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi100", "addr": "172.21.15.100", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T06:57:44.149 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:44 smithi100 ceph-mon[29092]: Deploying cephadm binary to smithi100 2024-09-17T06:57:46.152 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout Added host 'smithi100' with addr '172.21.15.100' 2024-09-17T06:57:46.153 INFO:teuthology.orchestra.run.smithi100.stdout:Deploying unmanaged mon service... 2024-09-17T06:57:46.725 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:46 smithi100 ceph-mon[29092]: from='mgr.14118 172.21.15.100:0/1799257286' entity='mgr.y' 2024-09-17T06:57:46.725 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:46 smithi100 ceph-mon[29092]: Added host smithi100 2024-09-17T06:57:46.725 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:46 smithi100 ceph-mon[29092]: from='mgr.14118 172.21.15.100:0/1799257286' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T06:57:48.323 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-09-17T06:57:48.323 INFO:teuthology.orchestra.run.smithi100.stdout:Deploying unmanaged mgr service... 2024-09-17T06:57:48.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:48 smithi100 ceph-mon[29092]: from='mgr.14118 172.21.15.100:0/1799257286' entity='mgr.y' 2024-09-17T06:57:48.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:48 smithi100 ceph-mon[29092]: from='mgr.14118 172.21.15.100:0/1799257286' entity='mgr.y' 2024-09-17T06:57:48.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:48 smithi100 ceph-mon[29092]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T06:57:48.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:48 smithi100 ceph-mon[29092]: Saving service mon spec with placement count:5 2024-09-17T06:57:48.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:48 smithi100 ceph-mon[29092]: from='mgr.14118 172.21.15.100:0/1799257286' entity='mgr.y' 2024-09-17T06:57:49.949 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-09-17T06:57:50.073 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:49 smithi100 ceph-mon[29092]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T06:57:50.073 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:49 smithi100 ceph-mon[29092]: Saving service mgr spec with placement count:2 2024-09-17T06:57:50.073 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:49 smithi100 ceph-mon[29092]: from='mgr.14118 172.21.15.100:0/1799257286' entity='mgr.y' 2024-09-17T06:57:51.246 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:50 smithi100 ceph-mon[29092]: from='mgr.14118 172.21.15.100:0/1799257286' entity='mgr.y' 2024-09-17T06:57:51.246 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:50 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1661889744' entity='client.admin' 2024-09-17T06:57:51.246 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:50 smithi100 ceph-mon[29092]: from='mgr.14118 172.21.15.100:0/1799257286' entity='mgr.y' 2024-09-17T06:57:52.290 INFO:teuthology.orchestra.run.smithi100.stdout:Enabling the dashboard module... 2024-09-17T06:57:53.126 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:52 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/3539622556' entity='client.admin' 2024-09-17T06:57:54.065 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:53 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/3057176659' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-09-17T06:57:54.065 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:53 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ignoring --setuser ceph since I am not root 2024-09-17T06:57:54.065 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:53 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ignoring --setgroup ceph since I am not root 2024-09-17T06:57:54.065 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:53 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T06:57:53.853+0000 7f563df31040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-17T06:57:54.066 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:53 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T06:57:53.909+0000 7f563df31040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-17T06:57:54.066 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:54 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T06:57:54.063+0000 7f563df31040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-17T06:57:54.376 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:54 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T06:57:54.132+0000 7f563df31040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-17T06:57:54.377 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:54 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T06:57:54.280+0000 7f563df31040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-17T06:57:54.708 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:54 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T06:57:54.424+0000 7f563df31040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-17T06:57:54.998 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:54 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/3057176659' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-09-17T06:57:54.998 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:54 smithi100 ceph-mon[29092]: mgrmap e9: y(active, since 18s) 2024-09-17T06:57:54.998 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:54 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T06:57:54.745+0000 7f563df31040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-17T06:57:54.998 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:54 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T06:57:54.808+0000 7f563df31040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-17T06:57:54.998 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:54 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T06:57:54.996+0000 7f563df31040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-17T06:57:55.517 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout { 2024-09-17T06:57:55.518 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-09-17T06:57:55.518 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-17T06:57:55.518 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-09-17T06:57:55.518 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-17T06:57:55.518 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout } 2024-09-17T06:57:55.518 INFO:teuthology.orchestra.run.smithi100.stdout:Waiting for the mgr to restart... 2024-09-17T06:57:55.518 INFO:teuthology.orchestra.run.smithi100.stdout:Waiting for mgr epoch 9... 2024-09-17T06:57:55.933 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:55 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/4135412348' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-17T06:57:55.933 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:55 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T06:57:55.886+0000 7f563df31040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-17T06:57:56.376 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:55 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T06:57:55.955+0000 7f563df31040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-17T06:57:56.916 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:56 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T06:57:56.655+0000 7f563df31040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-17T06:57:56.916 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:56 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T06:57:56.770+0000 7f563df31040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-17T06:57:57.200 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:56 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T06:57:56.914+0000 7f563df31040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-17T06:57:57.200 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:56 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T06:57:56.976+0000 7f563df31040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-17T06:57:57.201 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:57 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T06:57:57.126+0000 7f563df31040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-17T06:57:57.201 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:57 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T06:57:57.198+0000 7f563df31040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-17T06:57:57.488 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:57 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T06:57:57.261+0000 7f563df31040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-17T06:57:57.488 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:57 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T06:57:57.420+0000 7f563df31040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-17T06:57:57.488 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:57 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T06:57:57.486+0000 7f563df31040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-17T06:57:57.754 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:57 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T06:57:57.552+0000 7f563df31040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-17T06:57:57.754 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:57 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T06:57:57.635+0000 7f563df31040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-17T06:57:57.754 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:57 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T06:57:57.693+0000 7f563df31040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-17T06:57:57.754 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:57 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T06:57:57.753+0000 7f563df31040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-17T06:57:58.876 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:57:58 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T06:57:58.524+0000 7f563df31040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-17T06:57:58.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:58 smithi100 ceph-mon[29092]: Active manager daemon y restarted 2024-09-17T06:57:58.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:58 smithi100 ceph-mon[29092]: Activating manager daemon y 2024-09-17T06:57:58.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:58 smithi100 ceph-mon[29092]: osdmap e3: 0 total, 0 up, 0 in 2024-09-17T06:57:58.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:58 smithi100 ceph-mon[29092]: mgrmap e10: y(active, starting, since 0.0031198s) 2024-09-17T06:57:58.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:58 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T06:57:58.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:58 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-17T06:57:58.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:58 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-17T06:57:58.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:58 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-17T06:57:58.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:58 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-17T06:57:58.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:58 smithi100 ceph-mon[29092]: Manager daemon y is now available 2024-09-17T06:57:58.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:58 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T06:57:58.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:58 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T06:57:58.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:57:58 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T06:58:00.174 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:58:00 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T06:58:00.069+0000 7f55a43bb640 -1 log_channel(cephadm) log [ERR] : [17/Sep/2024:06:58:00] ENGINE Error in HTTPServer.serve 2024-09-17T06:58:00.174 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:58:00 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: Traceback (most recent call last): 2024-09-17T06:58:00.174 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:58:00 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T06:58:00.175 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:58:00 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: self._connections.run(self.expiration_interval) 2024-09-17T06:58:00.175 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:58:00 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T06:58:00.175 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:58:00 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: self._run(expiration_interval) 2024-09-17T06:58:00.175 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:58:00 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T06:58:00.175 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:58:00 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T06:58:00.175 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:58:00 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T06:58:00.175 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:58:00 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T06:58:00.175 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:58:00 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T06:58:00.175 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:58:00 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: s = self.context.wrap_socket( 2024-09-17T06:58:00.176 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:58:00 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T06:58:00.176 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:58:00 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: return self.sslsocket_class._create( 2024-09-17T06:58:00.176 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:58:00 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T06:58:00.176 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:58:00 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: self.do_handshake() 2024-09-17T06:58:00.176 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:58:00 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T06:58:00.176 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:58:00 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: self._sslobj.do_handshake() 2024-09-17T06:58:00.176 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:58:00 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T06:58:00.176 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 06:58:00 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T06:58:00.177 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout { 2024-09-17T06:58:00.177 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-09-17T06:58:00.177 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-17T06:58:00.177 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout } 2024-09-17T06:58:00.177 INFO:teuthology.orchestra.run.smithi100.stdout:mgr epoch 9 is available 2024-09-17T06:58:00.177 INFO:teuthology.orchestra.run.smithi100.stdout:Generating a dashboard self-signed certificate... 2024-09-17T06:58:00.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:00 smithi100 ceph-mon[29092]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-17T06:58:00.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:00 smithi100 ceph-mon[29092]: mgrmap e11: y(active, since 1.00614s) 2024-09-17T06:58:00.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:00 smithi100 ceph-mon[29092]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-17T06:58:00.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:00 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:01.442 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-09-17T06:58:01.442 INFO:teuthology.orchestra.run.smithi100.stdout:Creating initial admin user... 2024-09-17T06:58:01.562 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:01 smithi100 ceph-mon[29092]: [17/Sep/2024:06:57:59] ENGINE Bus STARTING 2024-09-17T06:58:01.562 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:01 smithi100 ceph-mon[29092]: [17/Sep/2024:06:57:59] ENGINE Serving on http://172.21.15.100:8765 2024-09-17T06:58:01.562 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:01 smithi100 ceph-mon[29092]: [17/Sep/2024:06:58:00] ENGINE Serving on https://172.21.15.100:7150 2024-09-17T06:58:01.562 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:01 smithi100 ceph-mon[29092]: [17/Sep/2024:06:58:00] ENGINE Bus STARTED 2024-09-17T06:58:01.562 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:01 smithi100 ceph-mon[29092]: [17/Sep/2024:06:58:00] ENGINE Error in HTTPServer.serve 2024-09-17T06:58:01.562 INFO:journalctl@ceph.mon.a.smithi100.stdout: Traceback (most recent call last): 2024-09-17T06:58:01.562 INFO:journalctl@ceph.mon.a.smithi100.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T06:58:01.562 INFO:journalctl@ceph.mon.a.smithi100.stdout: self._connections.run(self.expiration_interval) 2024-09-17T06:58:01.563 INFO:journalctl@ceph.mon.a.smithi100.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T06:58:01.563 INFO:journalctl@ceph.mon.a.smithi100.stdout: self._run(expiration_interval) 2024-09-17T06:58:01.563 INFO:journalctl@ceph.mon.a.smithi100.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T06:58:01.563 INFO:journalctl@ceph.mon.a.smithi100.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T06:58:01.563 INFO:journalctl@ceph.mon.a.smithi100.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T06:58:01.563 INFO:journalctl@ceph.mon.a.smithi100.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T06:58:01.563 INFO:journalctl@ceph.mon.a.smithi100.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T06:58:01.563 INFO:journalctl@ceph.mon.a.smithi100.stdout: s = self.context.wrap_socket( 2024-09-17T06:58:01.563 INFO:journalctl@ceph.mon.a.smithi100.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T06:58:01.563 INFO:journalctl@ceph.mon.a.smithi100.stdout: return self.sslsocket_class._create( 2024-09-17T06:58:01.563 INFO:journalctl@ceph.mon.a.smithi100.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T06:58:01.563 INFO:journalctl@ceph.mon.a.smithi100.stdout: self.do_handshake() 2024-09-17T06:58:01.564 INFO:journalctl@ceph.mon.a.smithi100.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T06:58:01.564 INFO:journalctl@ceph.mon.a.smithi100.stdout: self._sslobj.do_handshake() 2024-09-17T06:58:01.564 INFO:journalctl@ceph.mon.a.smithi100.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T06:58:01.564 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:01 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:01.564 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:01 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:01.564 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:01 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:01.564 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:01 smithi100 ceph-mon[29092]: mgrmap e12: y(active, since 2s) 2024-09-17T06:58:02.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:02 smithi100 ceph-mon[29092]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T06:58:02.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:02 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:03.414 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$KuXijCNIxelSNeVOuJvKZOst6eftdYixk0QtHGGwI8ltAmlEqM8uO", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1726556282, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-09-17T06:58:03.414 INFO:teuthology.orchestra.run.smithi100.stdout:Fetching dashboard port number... 2024-09-17T06:58:03.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:03 smithi100 ceph-mon[29092]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T06:58:04.601 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:04 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1391302704' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-09-17T06:58:04.604 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout 8443 2024-09-17T06:58:04.615 INFO:teuthology.orchestra.run.smithi100.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-17T06:58:04.615 INFO:teuthology.orchestra.run.smithi100.stdout:systemctl: stdout disabled 2024-09-17T06:58:04.622 INFO:teuthology.orchestra.run.smithi100.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-17T06:58:04.623 INFO:teuthology.orchestra.run.smithi100.stdout:systemctl: stdout inactive 2024-09-17T06:58:04.623 INFO:teuthology.orchestra.run.smithi100.stdout:firewalld.service is not enabled 2024-09-17T06:58:04.623 INFO:teuthology.orchestra.run.smithi100.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-09-17T06:58:04.625 INFO:teuthology.orchestra.run.smithi100.stdout:Ceph Dashboard is now available at: 2024-09-17T06:58:04.625 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:58:04.625 INFO:teuthology.orchestra.run.smithi100.stdout: URL: https://smithi100.front.sepia.ceph.com:8443/ 2024-09-17T06:58:04.625 INFO:teuthology.orchestra.run.smithi100.stdout: User: admin 2024-09-17T06:58:04.625 INFO:teuthology.orchestra.run.smithi100.stdout: Password: r17v5hecka 2024-09-17T06:58:04.625 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:58:04.626 INFO:teuthology.orchestra.run.smithi100.stdout:Saving cluster configuration to /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/config directory 2024-09-17T06:58:04.626 INFO:teuthology.orchestra.run.smithi100.stdout:Enabling autotune for osd_memory_target 2024-09-17T06:58:05.733 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:05 smithi100 ceph-mon[29092]: mgrmap e13: y(active, since 6s) 2024-09-17T06:58:06.883 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-09-17T06:58:06.883 INFO:teuthology.orchestra.run.smithi100.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-09-17T06:58:06.883 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:58:06.884 INFO:teuthology.orchestra.run.smithi100.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-09-17T06:58:06.884 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:58:06.884 INFO:teuthology.orchestra.run.smithi100.stdout:Or, if you are only running a single cluster on this host: 2024-09-17T06:58:06.884 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:58:06.884 INFO:teuthology.orchestra.run.smithi100.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-09-17T06:58:06.884 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:58:06.884 INFO:teuthology.orchestra.run.smithi100.stdout:Please consider enabling telemetry to help improve Ceph: 2024-09-17T06:58:06.884 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:58:06.884 INFO:teuthology.orchestra.run.smithi100.stdout: ceph telemetry on 2024-09-17T06:58:06.884 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:58:06.884 INFO:teuthology.orchestra.run.smithi100.stdout:For more information see: 2024-09-17T06:58:06.884 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:58:06.884 INFO:teuthology.orchestra.run.smithi100.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-09-17T06:58:06.885 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:58:06.885 INFO:teuthology.orchestra.run.smithi100.stdout:Bootstrap complete. 2024-09-17T06:58:06.916 INFO:tasks.cephadm:Fetching config... 2024-09-17T06:58:06.916 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-17T06:58:06.916 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-09-17T06:58:06.932 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-09-17T06:58:06.932 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-17T06:58:06.932 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-09-17T06:58:06.987 INFO:tasks.cephadm:Fetching mon keyring... 2024-09-17T06:58:06.987 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-17T06:58:06.987 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/keyring of=/dev/stdout 2024-09-17T06:58:07.051 INFO:tasks.cephadm:Fetching pub ssh key... 2024-09-17T06:58:07.051 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-17T06:58:07.051 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-09-17T06:58:07.106 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-09-17T06:58:07.107 DEBUG:teuthology.orchestra.run.smithi100:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCg96Gp5/L0F9L+x8CpeiqizxjnM85HmlNGkWdYoi0rfZ6rJmVl3zDFtg/akxaNBTPTgMfxXp8X2lx536MAjaNnispQ2ZFTi3p31ZLR55Qm52rkM3LdbWqeXXEmEml7QFyyOnKD839BafpV3cSaZIZnuQuIfjbChghEbyDppOYRrkttKCQxysiEAq79xtDaqmcNvxVq7RZYveokAEwqX9d2yTiiGWjR2VNNWjG8ZSXRyCuEL/JsYR7x/946Wx3+z0b2n65wNBQw8zJTYXsuYAeHE2XDbds3yk8+kI3TDdGRpo8N5FQzQ2fFkL+K9W/UY0pVvnYyg6QJBqa2QrdXLHILNbgCFFxWxSLqZ+63qOnZhxvPiyUMsIf3j3FRE8Z07M1ydpAITOly8OTv1tbLTy6jHSM4GTOwY4P0308vNRfFX6BbOkyxR6yKav47AwZuO/v6D5MsP8qCm3SjE5SkMip5jiOA4YmNqFQE8L1UyZQC+GDt7+SYAzJMJbLNotZNCgc= ceph-e848267c-74c1-11ef-bceb-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-17T06:58:07.181 INFO:teuthology.orchestra.run.smithi100.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCg96Gp5/L0F9L+x8CpeiqizxjnM85HmlNGkWdYoi0rfZ6rJmVl3zDFtg/akxaNBTPTgMfxXp8X2lx536MAjaNnispQ2ZFTi3p31ZLR55Qm52rkM3LdbWqeXXEmEml7QFyyOnKD839BafpV3cSaZIZnuQuIfjbChghEbyDppOYRrkttKCQxysiEAq79xtDaqmcNvxVq7RZYveokAEwqX9d2yTiiGWjR2VNNWjG8ZSXRyCuEL/JsYR7x/946Wx3+z0b2n65wNBQw8zJTYXsuYAeHE2XDbds3yk8+kI3TDdGRpo8N5FQzQ2fFkL+K9W/UY0pVvnYyg6QJBqa2QrdXLHILNbgCFFxWxSLqZ+63qOnZhxvPiyUMsIf3j3FRE8Z07M1ydpAITOly8OTv1tbLTy6jHSM4GTOwY4P0308vNRfFX6BbOkyxR6yKav47AwZuO/v6D5MsP8qCm3SjE5SkMip5jiOA4YmNqFQE8L1UyZQC+GDt7+SYAzJMJbLNotZNCgc= ceph-e848267c-74c1-11ef-bceb-c7b262605968 2024-09-17T06:58:07.192 DEBUG:teuthology.orchestra.run.smithi135:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCg96Gp5/L0F9L+x8CpeiqizxjnM85HmlNGkWdYoi0rfZ6rJmVl3zDFtg/akxaNBTPTgMfxXp8X2lx536MAjaNnispQ2ZFTi3p31ZLR55Qm52rkM3LdbWqeXXEmEml7QFyyOnKD839BafpV3cSaZIZnuQuIfjbChghEbyDppOYRrkttKCQxysiEAq79xtDaqmcNvxVq7RZYveokAEwqX9d2yTiiGWjR2VNNWjG8ZSXRyCuEL/JsYR7x/946Wx3+z0b2n65wNBQw8zJTYXsuYAeHE2XDbds3yk8+kI3TDdGRpo8N5FQzQ2fFkL+K9W/UY0pVvnYyg6QJBqa2QrdXLHILNbgCFFxWxSLqZ+63qOnZhxvPiyUMsIf3j3FRE8Z07M1ydpAITOly8OTv1tbLTy6jHSM4GTOwY4P0308vNRfFX6BbOkyxR6yKav47AwZuO/v6D5MsP8qCm3SjE5SkMip5jiOA4YmNqFQE8L1UyZQC+GDt7+SYAzJMJbLNotZNCgc= ceph-e848267c-74c1-11ef-bceb-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-17T06:58:07.250 INFO:teuthology.orchestra.run.smithi135.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCg96Gp5/L0F9L+x8CpeiqizxjnM85HmlNGkWdYoi0rfZ6rJmVl3zDFtg/akxaNBTPTgMfxXp8X2lx536MAjaNnispQ2ZFTi3p31ZLR55Qm52rkM3LdbWqeXXEmEml7QFyyOnKD839BafpV3cSaZIZnuQuIfjbChghEbyDppOYRrkttKCQxysiEAq79xtDaqmcNvxVq7RZYveokAEwqX9d2yTiiGWjR2VNNWjG8ZSXRyCuEL/JsYR7x/946Wx3+z0b2n65wNBQw8zJTYXsuYAeHE2XDbds3yk8+kI3TDdGRpo8N5FQzQ2fFkL+K9W/UY0pVvnYyg6QJBqa2QrdXLHILNbgCFFxWxSLqZ+63qOnZhxvPiyUMsIf3j3FRE8Z07M1ydpAITOly8OTv1tbLTy6jHSM4GTOwY4P0308vNRfFX6BbOkyxR6yKav47AwZuO/v6D5MsP8qCm3SjE5SkMip5jiOA4YmNqFQE8L1UyZQC+GDt7+SYAzJMJbLNotZNCgc= ceph-e848267c-74c1-11ef-bceb-c7b262605968 2024-09-17T06:58:07.263 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-09-17T06:58:07.456 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T06:58:07.511 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:07 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/996819927' entity='client.admin' 2024-09-17T06:58:10.265 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-09-17T06:58:10.265 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-09-17T06:58:10.287 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:10 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1961711699' entity='client.admin' 2024-09-17T06:58:10.287 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:10 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:10.288 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:10 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:10.288 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:10 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi100", "name": "osd_memory_target"}]: dispatch 2024-09-17T06:58:10.288 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:10 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:10.288 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:10 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T06:58:10.288 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:10 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:10.752 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T06:58:13.667 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi135 2024-09-17T06:58:13.668 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-09-17T06:58:13.668 DEBUG:teuthology.orchestra.run.smithi135:> dd of=/etc/ceph/ceph.conf 2024-09-17T06:58:13.688 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-09-17T06:58:13.688 DEBUG:teuthology.orchestra.run.smithi135:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-17T06:58:13.746 INFO:tasks.cephadm:Adding host smithi135 to orchestrator... 2024-09-17T06:58:13.747 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph orch host add smithi135 2024-09-17T06:58:13.939 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T06:58:14.287 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:13 smithi100 ceph-mon[29092]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T06:58:14.287 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:13 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:14.287 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:13 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T06:58:14.287 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:13 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T06:58:14.287 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:13 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T06:58:14.288 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:13 smithi100 ceph-mon[29092]: Updating smithi100:/etc/ceph/ceph.conf 2024-09-17T06:58:14.288 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:13 smithi100 ceph-mon[29092]: Updating smithi100:/var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T06:58:14.288 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:13 smithi100 ceph-mon[29092]: Updating smithi100:/etc/ceph/ceph.client.admin.keyring 2024-09-17T06:58:14.288 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:13 smithi100 ceph-mon[29092]: Updating smithi100:/var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-17T06:58:14.288 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:13 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:14.288 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:13 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:14.288 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:13 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:16.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:15 smithi100 ceph-mon[29092]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi135", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T06:58:17.126 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:16 smithi100 ceph-mon[29092]: Deploying cephadm binary to smithi135 2024-09-17T06:58:18.010 INFO:teuthology.orchestra.run.smithi100.stdout:Added host 'smithi135' with addr '172.21.15.135' 2024-09-17T06:58:18.601 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph orch host ls --format=json 2024-09-17T06:58:18.790 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T06:58:19.209 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:19 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:19.209 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:19 smithi100 ceph-mon[29092]: Added host smithi135 2024-09-17T06:58:19.209 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:19 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T06:58:19.209 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:19 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:19.209 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:19 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:19.209 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:19 smithi100 ceph-mon[29092]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T06:58:20.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:20 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:20.485 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T06:58:20.485 INFO:teuthology.orchestra.run.smithi100.stdout:[{"addr": "172.21.15.100", "hostname": "smithi100", "labels": [], "status": ""}, {"addr": "172.21.15.135", "hostname": "smithi135", "labels": [], "status": ""}] 2024-09-17T06:58:21.048 INFO:tasks.cephadm:Setting crush tunables to default 2024-09-17T06:58:21.049 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd crush tunables default 2024-09-17T06:58:21.237 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T06:58:21.572 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:21 smithi100 ceph-mon[29092]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T06:58:21.572 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:21 smithi100 ceph-mon[29092]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T06:58:23.126 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:23 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:23.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:23 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:23.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:23 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:23.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:23 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:23.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:23 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi135", "name": "osd_memory_target"}]: dispatch 2024-09-17T06:58:23.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:23 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T06:58:23.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:23 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T06:58:23.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:23 smithi100 ceph-mon[29092]: Updating smithi135:/etc/ceph/ceph.conf 2024-09-17T06:58:23.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:23 smithi100 ceph-mon[29092]: Updating smithi135:/var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T06:58:23.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:23 smithi100 ceph-mon[29092]: Updating smithi135:/etc/ceph/ceph.client.admin.keyring 2024-09-17T06:58:23.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:23 smithi100 ceph-mon[29092]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T06:58:23.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:23 smithi100 ceph-mon[29092]: Updating smithi135:/var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-17T06:58:23.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:23 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:23.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:23 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:23.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:23 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:23.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:23 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1144436359' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-09-17T06:58:23.871 INFO:teuthology.orchestra.run.smithi100.stderr:adjusted tunables profile to default 2024-09-17T06:58:24.485 INFO:tasks.cephadm:Adding mon.a on smithi100 2024-09-17T06:58:24.485 INFO:tasks.cephadm:Adding mon.c on smithi100 2024-09-17T06:58:24.485 INFO:tasks.cephadm:Adding mon.b on smithi135 2024-09-17T06:58:24.485 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph orch apply mon '3;smithi100:172.21.15.100=a;smithi100:[v2:172.21.15.100:3301,v1:172.21.15.100:6790]=c;smithi135:172.21.15.135=b' 2024-09-17T06:58:24.684 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T06:58:25.126 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:24 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1144436359' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-17T06:58:25.126 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:24 smithi100 ceph-mon[29092]: osdmap e4: 0 total, 0 up, 0 in 2024-09-17T06:58:25.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:24 smithi100 ceph-mon[29092]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T06:58:26.211 INFO:teuthology.orchestra.run.smithi135.stdout:Scheduled mon update... 2024-09-17T06:58:26.950 DEBUG:teuthology.orchestra.run.smithi100:mon.c> sudo journalctl -f -n 0 -u ceph-e848267c-74c1-11ef-bceb-c7b262605968@mon.c.service 2024-09-17T06:58:26.953 DEBUG:teuthology.orchestra.run.smithi135:mon.b> sudo journalctl -f -n 0 -u ceph-e848267c-74c1-11ef-bceb-c7b262605968@mon.b.service 2024-09-17T06:58:26.955 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-17T06:58:26.956 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-17T06:58:27.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:27 smithi100 ceph-mon[29092]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi100:172.21.15.100=a;smithi100:[v2:172.21.15.100:3301,v1:172.21.15.100:6790]=c;smithi135:172.21.15.135=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T06:58:27.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:27 smithi100 ceph-mon[29092]: Saving service mon spec with placement smithi100:172.21.15.100=a;smithi100:[v2:172.21.15.100:3301,v1:172.21.15.100:6790]=c;smithi135:172.21.15.135=b;count:3 2024-09-17T06:58:27.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:27 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:27.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:27 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T06:58:27.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:27 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T06:58:27.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:27 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T06:58:27.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:27 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:27.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:27 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T06:58:27.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:27 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T06:58:27.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:27 smithi100 ceph-mon[29092]: Deploying daemon mon.b on smithi135 2024-09-17T06:58:27.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:27 smithi100 ceph-mon[29092]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T06:58:27.637 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T06:58:29.513 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:58:29.513 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"e848267c-74c1-11ef-bceb-c7b262605968","modified":"2024-09-17T06:57:02.188299Z","created":"2024-09-17T06:57:02.188299Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:3300","nonce":0},{"type":"v1","addr":"172.21.15.100:6789","nonce":0}]},"addr":"172.21.15.100:6789/0","public_addr":"172.21.15.100:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-17T06:58:29.513 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2024-09-17T06:58:29.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:29 smithi100 ceph-mon[29092]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T06:58:29.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:29 smithi100 ceph-mon[29092]: from='client.? 172.21.15.135:0/3274653737' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-17T06:58:31.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:31 smithi100 ceph-mon[29092]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T06:58:32.031 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-17T06:58:32.032 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-17T06:58:32.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:31 smithi135 systemd[1]: Started Ceph mon.b for e848267c-74c1-11ef-bceb-c7b262605968. 2024-09-17T06:58:32.238 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T06:58:37.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:36 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T06:58:37.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:36 smithi100 ceph-mon[29092]: mon.a calling monitor election 2024-09-17T06:58:37.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:36 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T06:58:37.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:36 smithi100 ceph-mon[29092]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T06:58:37.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:36 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T06:58:37.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:36 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T06:58:37.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:36 smithi100 ceph-mon[29092]: mon.b calling monitor election 2024-09-17T06:58:37.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:36 smithi100 ceph-mon[29092]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T06:58:37.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:36 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T06:58:37.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:36 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T06:58:37.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:36 smithi100 ceph-mon[29092]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T06:58:37.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:36 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T06:58:37.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:36 smithi100 ceph-mon[29092]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-17T06:58:37.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:36 smithi100 ceph-mon[29092]: monmap e2: 2 mons at {a=[v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0],b=[v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-17T06:58:37.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:36 smithi100 ceph-mon[29092]: fsmap 2024-09-17T06:58:37.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:36 smithi100 ceph-mon[29092]: osdmap e4: 0 total, 0 up, 0 in 2024-09-17T06:58:37.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:36 smithi100 ceph-mon[29092]: mgrmap e13: y(active, since 38s) 2024-09-17T06:58:37.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:36 smithi100 ceph-mon[29092]: overall HEALTH_OK 2024-09-17T06:58:37.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:36 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:37.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:36 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:37.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:36 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:37.129 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:36 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T06:58:37.129 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:36 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T06:58:37.157 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:58:37.157 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":2,"fsid":"e848267c-74c1-11ef-bceb-c7b262605968","modified":"2024-09-17T06:58:31.803587Z","created":"2024-09-17T06:57:02.188299Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:3300","nonce":0},{"type":"v1","addr":"172.21.15.100:6789","nonce":0}]},"addr":"172.21.15.100:6789/0","public_addr":"172.21.15.100:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:3300","nonce":0},{"type":"v1","addr":"172.21.15.135:6789","nonce":0}]},"addr":"172.21.15.135:6789/0","public_addr":"172.21.15.135:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-17T06:58:37.157 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:36 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T06:58:37.157 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:36 smithi135 ceph-mon[33238]: mon.a calling monitor election 2024-09-17T06:58:37.157 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:36 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T06:58:37.157 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:36 smithi135 ceph-mon[33238]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T06:58:37.157 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:36 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T06:58:37.158 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:36 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T06:58:37.158 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:36 smithi135 ceph-mon[33238]: mon.b calling monitor election 2024-09-17T06:58:37.158 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:36 smithi135 ceph-mon[33238]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T06:58:37.158 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:36 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T06:58:37.158 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:36 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T06:58:37.158 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:36 smithi135 ceph-mon[33238]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T06:58:37.158 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:36 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T06:58:37.158 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:36 smithi135 ceph-mon[33238]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-17T06:58:37.158 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:36 smithi135 ceph-mon[33238]: monmap e2: 2 mons at {a=[v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0],b=[v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-17T06:58:37.158 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:36 smithi135 ceph-mon[33238]: fsmap 2024-09-17T06:58:37.158 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:36 smithi135 ceph-mon[33238]: osdmap e4: 0 total, 0 up, 0 in 2024-09-17T06:58:37.158 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:36 smithi135 ceph-mon[33238]: mgrmap e13: y(active, since 38s) 2024-09-17T06:58:37.158 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:36 smithi135 ceph-mon[33238]: overall HEALTH_OK 2024-09-17T06:58:37.158 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:36 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:37.159 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:36 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:37.159 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:36 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:37.159 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:36 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T06:58:37.159 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:36 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T06:58:37.159 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 2 2024-09-17T06:58:38.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:37 smithi100 ceph-mon[29092]: Deploying daemon mon.c on smithi100 2024-09-17T06:58:38.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:37 smithi100 ceph-mon[29092]: from='client.? 172.21.15.135:0/78198088' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-17T06:58:38.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:37 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T06:58:38.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:37 smithi135 ceph-mon[33238]: Deploying daemon mon.c on smithi100 2024-09-17T06:58:38.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:37 smithi135 ceph-mon[33238]: from='client.? 172.21.15.135:0/78198088' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-17T06:58:38.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:37 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T06:58:38.656 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-17T06:58:38.657 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-17T06:58:38.861 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T06:58:38.982 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:38 smithi135 ceph-mon[33238]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T06:58:39.138 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:38 smithi100 ceph-mon[29092]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T06:58:45.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:45 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:45.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:45 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T06:58:45.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:45 smithi100 ceph-mon[29092]: mon.a calling monitor election 2024-09-17T06:58:45.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:45 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T06:58:45.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:45 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T06:58:45.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:45 smithi100 ceph-mon[29092]: mon.b calling monitor election 2024-09-17T06:58:45.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:45 smithi100 ceph-mon[29092]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T06:58:45.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:45 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T06:58:45.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:45 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T06:58:45.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:45 smithi100 ceph-mon[29092]: mon.c calling monitor election 2024-09-17T06:58:45.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:45 smithi100 ceph-mon[29092]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T06:58:45.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:45 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T06:58:45.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:45 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T06:58:45.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:45 smithi100 ceph-mon[29092]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T06:58:45.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:45 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T06:58:45.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:45 smithi100 ceph-mon[29092]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-17T06:58:45.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:45 smithi100 ceph-mon[29092]: monmap e3: 3 mons at {a=[v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0],b=[v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0],c=[v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-17T06:58:45.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:45 smithi100 ceph-mon[29092]: fsmap 2024-09-17T06:58:45.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:45 smithi100 ceph-mon[29092]: osdmap e4: 0 total, 0 up, 0 in 2024-09-17T06:58:45.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:45 smithi100 ceph-mon[29092]: mgrmap e13: y(active, since 46s) 2024-09-17T06:58:45.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:45 smithi100 ceph-mon[29092]: overall HEALTH_OK 2024-09-17T06:58:45.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:45 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:45.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:45 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:45.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:45 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:45.379 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:45 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:45.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:45 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T06:58:45.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:45 smithi135 ceph-mon[33238]: mon.a calling monitor election 2024-09-17T06:58:45.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:45 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T06:58:45.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:45 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T06:58:45.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:45 smithi135 ceph-mon[33238]: mon.b calling monitor election 2024-09-17T06:58:45.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:45 smithi135 ceph-mon[33238]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T06:58:45.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:45 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T06:58:45.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:45 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T06:58:45.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:45 smithi135 ceph-mon[33238]: mon.c calling monitor election 2024-09-17T06:58:45.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:45 smithi135 ceph-mon[33238]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T06:58:45.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:45 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T06:58:45.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:45 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T06:58:45.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:45 smithi135 ceph-mon[33238]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T06:58:45.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:45 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T06:58:45.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:45 smithi135 ceph-mon[33238]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-17T06:58:45.545 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:45 smithi135 ceph-mon[33238]: monmap e3: 3 mons at {a=[v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0],b=[v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0],c=[v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-17T06:58:45.545 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:45 smithi135 ceph-mon[33238]: fsmap 2024-09-17T06:58:45.545 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:45 smithi135 ceph-mon[33238]: osdmap e4: 0 total, 0 up, 0 in 2024-09-17T06:58:45.545 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:45 smithi135 ceph-mon[33238]: mgrmap e13: y(active, since 46s) 2024-09-17T06:58:45.545 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:45 smithi135 ceph-mon[33238]: overall HEALTH_OK 2024-09-17T06:58:45.545 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:45 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:45.545 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:45 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:45.545 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:45 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:45.545 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:45 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:46.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:46 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T06:58:46.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:46 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:46.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:46 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:46.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:46 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T06:58:46.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:46 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:46.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:46 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:46.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:46 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T06:58:46.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:46 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T06:58:46.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:46 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T06:58:46.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:46 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:46.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:46 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:46.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:46 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T06:58:46.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:46 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:46.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:46 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:46.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:46 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T06:58:46.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:46 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T06:58:46.431 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-09-17T06:58:46.431 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":3,"fsid":"e848267c-74c1-11ef-bceb-c7b262605968","modified":"2024-09-17T06:58:40.016639Z","created":"2024-09-17T06:57:02.188299Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:3300","nonce":0},{"type":"v1","addr":"172.21.15.100:6789","nonce":0}]},"addr":"172.21.15.100:6789/0","public_addr":"172.21.15.100:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:3300","nonce":0},{"type":"v1","addr":"172.21.15.135:6789","nonce":0}]},"addr":"172.21.15.135:6789/0","public_addr":"172.21.15.135:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:3301","nonce":0},{"type":"v1","addr":"172.21.15.100:6790","nonce":0}]},"addr":"172.21.15.100:6790/0","public_addr":"172.21.15.100:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-09-17T06:58:46.431 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 3 2024-09-17T06:58:46.432 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:46 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T06:58:46.432 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:46 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:46.432 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:46 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:46.432 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:46 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T06:58:46.432 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:46 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:46.432 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:46 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:46.432 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:46 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T06:58:46.432 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:46 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T06:58:47.012 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-09-17T06:58:47.012 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph config generate-minimal-conf 2024-09-17T06:58:47.143 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:47 smithi100 ceph-mon[29092]: Updating smithi100:/etc/ceph/ceph.conf 2024-09-17T06:58:47.143 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:47 smithi100 ceph-mon[29092]: Updating smithi135:/etc/ceph/ceph.conf 2024-09-17T06:58:47.143 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:47 smithi100 ceph-mon[29092]: Updating smithi100:/var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T06:58:47.143 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:47 smithi100 ceph-mon[29092]: Updating smithi135:/var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T06:58:47.143 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:47 smithi100 ceph-mon[29092]: from='client.? 172.21.15.135:0/287963256' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-17T06:58:47.143 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:47 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:47.144 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:47 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:47.144 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:47 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:47.144 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:47 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:47.144 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:47 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:47.144 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:47 smithi100 ceph-mon[29092]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T06:58:47.144 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:47 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:47.144 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:47 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:47.144 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:47 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:47.144 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:47 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:47.144 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:47 smithi100 ceph-mon[29092]: Reconfiguring mon.a (unknown last config time)... 2024-09-17T06:58:47.144 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:47 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T06:58:47.144 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:47 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T06:58:47.144 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:47 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T06:58:47.145 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:47 smithi100 ceph-mon[29092]: Reconfiguring daemon mon.a on smithi100 2024-09-17T06:58:47.145 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:47 smithi100 ceph-mon[39462]: Updating smithi100:/etc/ceph/ceph.conf 2024-09-17T06:58:47.145 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:47 smithi100 ceph-mon[39462]: Updating smithi135:/etc/ceph/ceph.conf 2024-09-17T06:58:47.145 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:47 smithi100 ceph-mon[39462]: Updating smithi100:/var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T06:58:47.145 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:47 smithi100 ceph-mon[39462]: Updating smithi135:/var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T06:58:47.145 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:47 smithi100 ceph-mon[39462]: from='client.? 172.21.15.135:0/287963256' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-17T06:58:47.145 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:47 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:47.145 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:47 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:47.145 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:47 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:47.145 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:47 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:47.145 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:47 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:47.145 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:47 smithi100 ceph-mon[39462]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T06:58:47.145 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:47 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:47.146 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:47 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:47.146 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:47 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:47.146 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:47 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:47.146 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:47 smithi100 ceph-mon[39462]: Reconfiguring mon.a (unknown last config time)... 2024-09-17T06:58:47.146 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:47 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T06:58:47.146 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:47 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T06:58:47.146 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:47 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T06:58:47.146 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:47 smithi100 ceph-mon[39462]: Reconfiguring daemon mon.a on smithi100 2024-09-17T06:58:47.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:47 smithi135 ceph-mon[33238]: Updating smithi100:/etc/ceph/ceph.conf 2024-09-17T06:58:47.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:47 smithi135 ceph-mon[33238]: Updating smithi135:/etc/ceph/ceph.conf 2024-09-17T06:58:47.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:47 smithi135 ceph-mon[33238]: Updating smithi100:/var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T06:58:47.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:47 smithi135 ceph-mon[33238]: Updating smithi135:/var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T06:58:47.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:47 smithi135 ceph-mon[33238]: from='client.? 172.21.15.135:0/287963256' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-17T06:58:47.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:47 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:47.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:47 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:47.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:47 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:47.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:47 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:47.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:47 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:47.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:47 smithi135 ceph-mon[33238]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T06:58:47.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:47 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:47.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:47 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:47.295 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:47 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:47.295 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:47 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:47.295 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:47 smithi135 ceph-mon[33238]: Reconfiguring mon.a (unknown last config time)... 2024-09-17T06:58:47.295 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:47 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T06:58:47.295 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:47 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T06:58:47.295 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:47 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T06:58:47.295 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:47 smithi135 ceph-mon[33238]: Reconfiguring daemon mon.a on smithi100 2024-09-17T06:58:47.966 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T06:58:49.044 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:49 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:49.044 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:49 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:49.044 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:49 smithi100 ceph-mon[39462]: Reconfiguring mon.c (monmap changed)... 2024-09-17T06:58:49.044 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:49 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T06:58:49.045 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:49 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T06:58:49.045 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:49 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T06:58:49.045 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:49 smithi100 ceph-mon[39462]: Reconfiguring daemon mon.c on smithi100 2024-09-17T06:58:49.045 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:49 smithi100 ceph-mon[39462]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T06:58:49.045 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:49 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:49.045 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:49 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:49.045 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:49 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:49.045 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:49 smithi100 ceph-mon[29092]: Reconfiguring mon.c (monmap changed)... 2024-09-17T06:58:49.045 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:49 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T06:58:49.045 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:49 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T06:58:49.045 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:49 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T06:58:49.046 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:49 smithi100 ceph-mon[29092]: Reconfiguring daemon mon.c on smithi100 2024-09-17T06:58:49.046 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:49 smithi100 ceph-mon[29092]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T06:58:49.046 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:49 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:49.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:49 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:49.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:49 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:49.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:49 smithi135 ceph-mon[33238]: Reconfiguring mon.c (monmap changed)... 2024-09-17T06:58:49.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:49 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T06:58:49.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:49 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T06:58:49.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:49 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T06:58:49.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:49 smithi135 ceph-mon[33238]: Reconfiguring daemon mon.c on smithi100 2024-09-17T06:58:49.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:49 smithi135 ceph-mon[33238]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T06:58:49.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:49 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:50.339 INFO:teuthology.orchestra.run.smithi100.stdout:# minimal ceph.conf for e848267c-74c1-11ef-bceb-c7b262605968 2024-09-17T06:58:50.339 INFO:teuthology.orchestra.run.smithi100.stdout:[global] 2024-09-17T06:58:50.339 INFO:teuthology.orchestra.run.smithi100.stdout: fsid = e848267c-74c1-11ef-bceb-c7b262605968 2024-09-17T06:58:50.339 INFO:teuthology.orchestra.run.smithi100.stdout: mon_host = [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] 2024-09-17T06:58:51.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:50 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/3717794087' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T06:58:51.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:50 smithi100 ceph-mon[39462]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T06:58:51.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:50 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:51.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:50 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:51.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:50 smithi100 ceph-mon[39462]: Reconfiguring mon.b (monmap changed)... 2024-09-17T06:58:51.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:50 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T06:58:51.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:50 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T06:58:51.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:50 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T06:58:51.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:50 smithi100 ceph-mon[39462]: Reconfiguring daemon mon.b on smithi135 2024-09-17T06:58:51.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:50 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/3717794087' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T06:58:51.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:50 smithi100 ceph-mon[29092]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T06:58:51.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:50 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:51.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:50 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:51.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:50 smithi100 ceph-mon[29092]: Reconfiguring mon.b (monmap changed)... 2024-09-17T06:58:51.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:50 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T06:58:51.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:50 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T06:58:51.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:50 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T06:58:51.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:50 smithi100 ceph-mon[29092]: Reconfiguring daemon mon.b on smithi135 2024-09-17T06:58:51.195 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:50 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/3717794087' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T06:58:51.195 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:50 smithi135 ceph-mon[33238]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T06:58:51.195 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:50 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:51.195 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:50 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:51.195 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:50 smithi135 ceph-mon[33238]: Reconfiguring mon.b (monmap changed)... 2024-09-17T06:58:51.195 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:50 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T06:58:51.195 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:50 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T06:58:51.195 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:50 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T06:58:51.195 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:50 smithi135 ceph-mon[33238]: Reconfiguring daemon mon.b on smithi135 2024-09-17T06:58:51.211 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-09-17T06:58:51.211 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-17T06:58:51.211 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd of=/etc/ceph/ceph.conf 2024-09-17T06:58:51.239 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-17T06:58:51.239 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-17T06:58:51.305 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-09-17T06:58:51.305 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd of=/etc/ceph/ceph.conf 2024-09-17T06:58:51.333 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-09-17T06:58:51.333 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-17T06:58:51.397 INFO:tasks.cephadm:Adding mgr.y on smithi100 2024-09-17T06:58:51.398 INFO:tasks.cephadm:Adding mgr.x on smithi135 2024-09-17T06:58:51.398 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph orch apply mgr '2;smithi100=y;smithi135=x' 2024-09-17T06:58:51.957 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T06:58:53.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:52 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:53.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:52 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:53.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:52 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T06:58:53.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:52 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T06:58:53.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:52 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T06:58:53.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:52 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:53.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:52 smithi100 ceph-mon[29092]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T06:58:53.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:52 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:53.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:52 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:53.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:52 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T06:58:53.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:52 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T06:58:53.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:52 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T06:58:53.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:52 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:53.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:52 smithi100 ceph-mon[39462]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T06:58:53.139 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:52 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:53.139 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:52 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:53.140 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:52 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T06:58:53.140 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:52 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T06:58:53.140 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:52 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T06:58:53.140 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:52 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:53.140 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:52 smithi135 ceph-mon[33238]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T06:58:53.658 INFO:teuthology.orchestra.run.smithi135.stdout:Scheduled mgr update... 2024-09-17T06:58:54.163 DEBUG:teuthology.orchestra.run.smithi135:mgr.x> sudo journalctl -f -n 0 -u ceph-e848267c-74c1-11ef-bceb-c7b262605968@mgr.x.service 2024-09-17T06:58:54.166 INFO:tasks.cephadm:Deploying OSDs... 2024-09-17T06:58:54.166 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-17T06:58:54.166 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/scratch_devs of=/dev/stdout 2024-09-17T06:58:54.186 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-17T06:58:54.186 DEBUG:teuthology.orchestra.run.smithi100:> stat /dev/vg_nvme/lv_1 2024-09-17T06:58:54.244 INFO:teuthology.orchestra.run.smithi100.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-17T06:58:54.244 INFO:teuthology.orchestra.run.smithi100.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T06:58:54.244 INFO:teuthology.orchestra.run.smithi100.stdout:Device: 5h/5d Inode: 970 Links: 1 2024-09-17T06:58:54.244 INFO:teuthology.orchestra.run.smithi100.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T06:58:54.244 INFO:teuthology.orchestra.run.smithi100.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T06:58:54.244 INFO:teuthology.orchestra.run.smithi100.stdout:Access: 2024-09-17 06:58:39.489217405 +0000 2024-09-17T06:58:54.244 INFO:teuthology.orchestra.run.smithi100.stdout:Modify: 2024-09-17 06:53:37.740557905 +0000 2024-09-17T06:58:54.245 INFO:teuthology.orchestra.run.smithi100.stdout:Change: 2024-09-17 06:53:37.740557905 +0000 2024-09-17T06:58:54.245 INFO:teuthology.orchestra.run.smithi100.stdout: Birth: 2024-09-17 06:53:37.740557905 +0000 2024-09-17T06:58:54.245 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-17T06:58:54.312 INFO:teuthology.orchestra.run.smithi100.stderr:1+0 records in 2024-09-17T06:58:54.312 INFO:teuthology.orchestra.run.smithi100.stderr:1+0 records out 2024-09-17T06:58:54.313 INFO:teuthology.orchestra.run.smithi100.stderr:512 bytes copied, 0.000229835 s, 2.2 MB/s 2024-09-17T06:58:54.314 DEBUG:teuthology.orchestra.run.smithi100:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-17T06:58:54.372 DEBUG:teuthology.orchestra.run.smithi100:> stat /dev/vg_nvme/lv_2 2024-09-17T06:58:54.430 INFO:teuthology.orchestra.run.smithi100.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-17T06:58:54.430 INFO:teuthology.orchestra.run.smithi100.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T06:58:54.430 INFO:teuthology.orchestra.run.smithi100.stdout:Device: 5h/5d Inode: 979 Links: 1 2024-09-17T06:58:54.430 INFO:teuthology.orchestra.run.smithi100.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T06:58:54.430 INFO:teuthology.orchestra.run.smithi100.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T06:58:54.430 INFO:teuthology.orchestra.run.smithi100.stdout:Access: 2024-09-17 06:58:39.490217379 +0000 2024-09-17T06:58:54.430 INFO:teuthology.orchestra.run.smithi100.stdout:Modify: 2024-09-17 06:53:37.740557905 +0000 2024-09-17T06:58:54.430 INFO:teuthology.orchestra.run.smithi100.stdout:Change: 2024-09-17 06:53:37.740557905 +0000 2024-09-17T06:58:54.430 INFO:teuthology.orchestra.run.smithi100.stdout: Birth: 2024-09-17 06:53:37.740557905 +0000 2024-09-17T06:58:54.430 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-17T06:58:54.495 INFO:teuthology.orchestra.run.smithi100.stderr:1+0 records in 2024-09-17T06:58:54.495 INFO:teuthology.orchestra.run.smithi100.stderr:1+0 records out 2024-09-17T06:58:54.495 INFO:teuthology.orchestra.run.smithi100.stderr:512 bytes copied, 0.000302172 s, 1.7 MB/s 2024-09-17T06:58:54.497 DEBUG:teuthology.orchestra.run.smithi100:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-17T06:58:54.555 DEBUG:teuthology.orchestra.run.smithi100:> stat /dev/vg_nvme/lv_3 2024-09-17T06:58:54.612 INFO:teuthology.orchestra.run.smithi100.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-17T06:58:54.613 INFO:teuthology.orchestra.run.smithi100.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T06:58:54.613 INFO:teuthology.orchestra.run.smithi100.stdout:Device: 5h/5d Inode: 969 Links: 1 2024-09-17T06:58:54.613 INFO:teuthology.orchestra.run.smithi100.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T06:58:54.613 INFO:teuthology.orchestra.run.smithi100.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T06:58:54.613 INFO:teuthology.orchestra.run.smithi100.stdout:Access: 2024-09-17 06:58:39.490217379 +0000 2024-09-17T06:58:54.613 INFO:teuthology.orchestra.run.smithi100.stdout:Modify: 2024-09-17 06:53:37.739557931 +0000 2024-09-17T06:58:54.613 INFO:teuthology.orchestra.run.smithi100.stdout:Change: 2024-09-17 06:53:37.739557931 +0000 2024-09-17T06:58:54.613 INFO:teuthology.orchestra.run.smithi100.stdout: Birth: 2024-09-17 06:53:37.739557931 +0000 2024-09-17T06:58:54.613 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-17T06:58:54.679 INFO:teuthology.orchestra.run.smithi100.stderr:1+0 records in 2024-09-17T06:58:54.679 INFO:teuthology.orchestra.run.smithi100.stderr:1+0 records out 2024-09-17T06:58:54.679 INFO:teuthology.orchestra.run.smithi100.stderr:512 bytes copied, 0.000258268 s, 2.0 MB/s 2024-09-17T06:58:54.680 DEBUG:teuthology.orchestra.run.smithi100:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-17T06:58:54.738 DEBUG:teuthology.orchestra.run.smithi100:> stat /dev/vg_nvme/lv_4 2024-09-17T06:58:54.794 INFO:teuthology.orchestra.run.smithi100.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-17T06:58:54.794 INFO:teuthology.orchestra.run.smithi100.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T06:58:54.794 INFO:teuthology.orchestra.run.smithi100.stdout:Device: 5h/5d Inode: 978 Links: 1 2024-09-17T06:58:54.794 INFO:teuthology.orchestra.run.smithi100.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T06:58:54.794 INFO:teuthology.orchestra.run.smithi100.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T06:58:54.794 INFO:teuthology.orchestra.run.smithi100.stdout:Access: 2024-09-17 06:58:39.490217379 +0000 2024-09-17T06:58:54.794 INFO:teuthology.orchestra.run.smithi100.stdout:Modify: 2024-09-17 06:53:37.740557905 +0000 2024-09-17T06:58:54.794 INFO:teuthology.orchestra.run.smithi100.stdout:Change: 2024-09-17 06:53:37.740557905 +0000 2024-09-17T06:58:54.794 INFO:teuthology.orchestra.run.smithi100.stdout: Birth: 2024-09-17 06:53:37.740557905 +0000 2024-09-17T06:58:54.795 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-17T06:58:54.795 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:54 smithi135 ceph-mon[33238]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi100=y;smithi135=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T06:58:54.795 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:54 smithi135 ceph-mon[33238]: Saving service mgr spec with placement smithi100=y;smithi135=x;count:2 2024-09-17T06:58:54.795 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:54 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:54.795 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:54 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T06:58:54.795 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:54 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T06:58:54.796 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:54 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T06:58:54.796 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:54 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:54.796 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:54 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T06:58:54.796 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:54 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-17T06:58:54.796 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:54 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T06:58:54.796 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:54 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T06:58:54.796 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:54 smithi135 ceph-mon[33238]: Deploying daemon mgr.x on smithi135 2024-09-17T06:58:54.858 INFO:teuthology.orchestra.run.smithi100.stderr:1+0 records in 2024-09-17T06:58:54.858 INFO:teuthology.orchestra.run.smithi100.stderr:1+0 records out 2024-09-17T06:58:54.858 INFO:teuthology.orchestra.run.smithi100.stderr:512 bytes copied, 0.00020335 s, 2.5 MB/s 2024-09-17T06:58:54.859 DEBUG:teuthology.orchestra.run.smithi100:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-17T06:58:54.916 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-09-17T06:58:54.916 DEBUG:teuthology.orchestra.run.smithi135:> dd if=/scratch_devs of=/dev/stdout 2024-09-17T06:58:54.932 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-17T06:58:54.933 DEBUG:teuthology.orchestra.run.smithi135:> stat /dev/vg_nvme/lv_1 2024-09-17T06:58:54.988 INFO:teuthology.orchestra.run.smithi135.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-17T06:58:54.989 INFO:teuthology.orchestra.run.smithi135.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T06:58:54.989 INFO:teuthology.orchestra.run.smithi135.stdout:Device: 5h/5d Inode: 979 Links: 1 2024-09-17T06:58:54.989 INFO:teuthology.orchestra.run.smithi135.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T06:58:54.989 INFO:teuthology.orchestra.run.smithi135.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T06:58:54.989 INFO:teuthology.orchestra.run.smithi135.stdout:Access: 2024-09-17 06:58:31.251776916 +0000 2024-09-17T06:58:54.989 INFO:teuthology.orchestra.run.smithi135.stdout:Modify: 2024-09-17 06:53:37.779241488 +0000 2024-09-17T06:58:54.989 INFO:teuthology.orchestra.run.smithi135.stdout:Change: 2024-09-17 06:53:37.779241488 +0000 2024-09-17T06:58:54.989 INFO:teuthology.orchestra.run.smithi135.stdout: Birth: 2024-09-17 06:53:37.779241488 +0000 2024-09-17T06:58:54.989 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-17T06:58:55.052 INFO:teuthology.orchestra.run.smithi135.stderr:1+0 records in 2024-09-17T06:58:55.053 INFO:teuthology.orchestra.run.smithi135.stderr:1+0 records out 2024-09-17T06:58:55.053 INFO:teuthology.orchestra.run.smithi135.stderr:512 bytes copied, 0.000217901 s, 2.3 MB/s 2024-09-17T06:58:55.054 DEBUG:teuthology.orchestra.run.smithi135:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-17T06:58:55.110 DEBUG:teuthology.orchestra.run.smithi135:> stat /dev/vg_nvme/lv_2 2024-09-17T06:58:55.126 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:54 smithi100 ceph-mon[29092]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi100=y;smithi135=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T06:58:55.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:54 smithi100 ceph-mon[29092]: Saving service mgr spec with placement smithi100=y;smithi135=x;count:2 2024-09-17T06:58:55.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:54 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:55.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:54 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T06:58:55.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:54 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T06:58:55.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:54 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T06:58:55.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:54 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:55.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:54 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T06:58:55.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:54 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-17T06:58:55.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:54 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T06:58:55.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:54 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T06:58:55.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:54 smithi100 ceph-mon[29092]: Deploying daemon mgr.x on smithi135 2024-09-17T06:58:55.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:54 smithi100 ceph-mon[39462]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi100=y;smithi135=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T06:58:55.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:54 smithi100 ceph-mon[39462]: Saving service mgr spec with placement smithi100=y;smithi135=x;count:2 2024-09-17T06:58:55.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:54 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:55.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:54 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T06:58:55.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:54 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T06:58:55.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:54 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T06:58:55.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:54 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:55.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:54 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T06:58:55.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:54 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-17T06:58:55.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:54 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T06:58:55.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:54 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T06:58:55.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:54 smithi100 ceph-mon[39462]: Deploying daemon mgr.x on smithi135 2024-09-17T06:58:55.165 INFO:teuthology.orchestra.run.smithi135.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-17T06:58:55.165 INFO:teuthology.orchestra.run.smithi135.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T06:58:55.165 INFO:teuthology.orchestra.run.smithi135.stdout:Device: 5h/5d Inode: 967 Links: 1 2024-09-17T06:58:55.166 INFO:teuthology.orchestra.run.smithi135.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T06:58:55.166 INFO:teuthology.orchestra.run.smithi135.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T06:58:55.166 INFO:teuthology.orchestra.run.smithi135.stdout:Access: 2024-09-17 06:58:31.252776897 +0000 2024-09-17T06:58:55.166 INFO:teuthology.orchestra.run.smithi135.stdout:Modify: 2024-09-17 06:53:37.778241507 +0000 2024-09-17T06:58:55.166 INFO:teuthology.orchestra.run.smithi135.stdout:Change: 2024-09-17 06:53:37.778241507 +0000 2024-09-17T06:58:55.166 INFO:teuthology.orchestra.run.smithi135.stdout: Birth: 2024-09-17 06:53:37.778241507 +0000 2024-09-17T06:58:55.166 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-17T06:58:55.229 INFO:teuthology.orchestra.run.smithi135.stderr:1+0 records in 2024-09-17T06:58:55.230 INFO:teuthology.orchestra.run.smithi135.stderr:1+0 records out 2024-09-17T06:58:55.230 INFO:teuthology.orchestra.run.smithi135.stderr:512 bytes copied, 0.000228554 s, 2.2 MB/s 2024-09-17T06:58:55.231 DEBUG:teuthology.orchestra.run.smithi135:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-17T06:58:55.287 DEBUG:teuthology.orchestra.run.smithi135:> stat /dev/vg_nvme/lv_3 2024-09-17T06:58:55.342 INFO:teuthology.orchestra.run.smithi135.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-17T06:58:55.343 INFO:teuthology.orchestra.run.smithi135.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T06:58:55.343 INFO:teuthology.orchestra.run.smithi135.stdout:Device: 5h/5d Inode: 984 Links: 1 2024-09-17T06:58:55.343 INFO:teuthology.orchestra.run.smithi135.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T06:58:55.343 INFO:teuthology.orchestra.run.smithi135.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T06:58:55.343 INFO:teuthology.orchestra.run.smithi135.stdout:Access: 2024-09-17 06:58:31.252776897 +0000 2024-09-17T06:58:55.343 INFO:teuthology.orchestra.run.smithi135.stdout:Modify: 2024-09-17 06:53:37.780241469 +0000 2024-09-17T06:58:55.343 INFO:teuthology.orchestra.run.smithi135.stdout:Change: 2024-09-17 06:53:37.780241469 +0000 2024-09-17T06:58:55.343 INFO:teuthology.orchestra.run.smithi135.stdout: Birth: 2024-09-17 06:53:37.780241469 +0000 2024-09-17T06:58:55.343 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-17T06:58:55.486 INFO:teuthology.orchestra.run.smithi135.stderr:1+0 records in 2024-09-17T06:58:55.486 INFO:teuthology.orchestra.run.smithi135.stderr:1+0 records out 2024-09-17T06:58:55.486 INFO:teuthology.orchestra.run.smithi135.stderr:512 bytes copied, 0.00024259 s, 2.1 MB/s 2024-09-17T06:58:55.487 DEBUG:teuthology.orchestra.run.smithi135:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-17T06:58:55.503 DEBUG:teuthology.orchestra.run.smithi135:> stat /dev/vg_nvme/lv_4 2024-09-17T06:58:55.560 INFO:teuthology.orchestra.run.smithi135.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-17T06:58:55.560 INFO:teuthology.orchestra.run.smithi135.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T06:58:55.561 INFO:teuthology.orchestra.run.smithi135.stdout:Device: 5h/5d Inode: 972 Links: 1 2024-09-17T06:58:55.561 INFO:teuthology.orchestra.run.smithi135.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T06:58:55.561 INFO:teuthology.orchestra.run.smithi135.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T06:58:55.561 INFO:teuthology.orchestra.run.smithi135.stdout:Access: 2024-09-17 06:58:55.441315738 +0000 2024-09-17T06:58:55.561 INFO:teuthology.orchestra.run.smithi135.stdout:Modify: 2024-09-17 06:53:37.779241488 +0000 2024-09-17T06:58:55.561 INFO:teuthology.orchestra.run.smithi135.stdout:Change: 2024-09-17 06:53:37.779241488 +0000 2024-09-17T06:58:55.561 INFO:teuthology.orchestra.run.smithi135.stdout: Birth: 2024-09-17 06:53:37.779241488 +0000 2024-09-17T06:58:55.561 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-17T06:58:55.662 INFO:teuthology.orchestra.run.smithi135.stderr:1+0 records in 2024-09-17T06:58:55.662 INFO:teuthology.orchestra.run.smithi135.stderr:1+0 records out 2024-09-17T06:58:55.662 INFO:teuthology.orchestra.run.smithi135.stderr:512 bytes copied, 0.000213906 s, 2.4 MB/s 2024-09-17T06:58:55.663 DEBUG:teuthology.orchestra.run.smithi135:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-17T06:58:55.679 INFO:tasks.cephadm:Deploying osd.0 on smithi100 with /dev/vg_nvme/lv_4... 2024-09-17T06:58:55.679 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-09-17T06:58:55.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:55 smithi135 ceph-mon[33238]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T06:58:55.793 INFO:journalctl@ceph.mgr.x.smithi135.stdout:Sep 17 06:58:55 smithi135 systemd[1]: Starting Ceph mgr.x for e848267c-74c1-11ef-bceb-c7b262605968... 2024-09-17T06:58:55.880 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T06:58:56.000 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:55 smithi100 ceph-mon[29092]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T06:58:56.000 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:55 smithi100 ceph-mon[39462]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T06:58:56.162 INFO:journalctl@ceph.mgr.x.smithi135.stdout:Sep 17 06:58:55 smithi135 podman[35083]: 2024-09-17 06:58:55.792434821 +0000 UTC m=+0.016305236 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9 2024-09-17T06:58:56.162 INFO:journalctl@ceph.mgr.x.smithi135.stdout:Sep 17 06:58:55 smithi135 podman[35083]: 2024-09-17 06:58:55.893328604 +0000 UTC m=+0.117199000 container create b040363098ac3e47322bc18c6c3d9dc3dd1f0452a9a0e317316268ace0a92fbd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-x, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, RELEASE=reef-19e2a2b, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2) 2024-09-17T06:58:56.163 INFO:journalctl@ceph.mgr.x.smithi135.stdout:Sep 17 06:58:56 smithi135 podman[35083]: 2024-09-17 06:58:56.095734143 +0000 UTC m=+0.319604549 container init b040363098ac3e47322bc18c6c3d9dc3dd1f0452a9a0e317316268ace0a92fbd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-x, org.label-schema.license=GPLv2, RELEASE=reef-19e2a2b, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T06:58:56.163 INFO:journalctl@ceph.mgr.x.smithi135.stdout:Sep 17 06:58:56 smithi135 podman[35083]: 2024-09-17 06:58:56.098479762 +0000 UTC m=+0.322350160 container start b040363098ac3e47322bc18c6c3d9dc3dd1f0452a9a0e317316268ace0a92fbd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-x, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=reef-19e2a2b, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-09-17T06:58:56.163 INFO:journalctl@ceph.mgr.x.smithi135.stdout:Sep 17 06:58:56 smithi135 ceph-mgr[35102]: -- 172.21.15.135:0/3720929654 <== mon.1 v2:172.21.15.135:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x555789708d00 con 0x5557896dc800 2024-09-17T06:58:56.163 INFO:journalctl@ceph.mgr.x.smithi135.stdout:Sep 17 06:58:56 smithi135 bash[35083]: b040363098ac3e47322bc18c6c3d9dc3dd1f0452a9a0e317316268ace0a92fbd 2024-09-17T06:58:56.437 INFO:journalctl@ceph.mgr.x.smithi135.stdout:Sep 17 06:58:56 smithi135 systemd[1]: Started Ceph mgr.x for e848267c-74c1-11ef-bceb-c7b262605968. 2024-09-17T06:58:56.437 INFO:journalctl@ceph.mgr.x.smithi135.stdout:Sep 17 06:58:56 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-x[35098]: 2024-09-17T06:58:56.219+0000 7f03b38c2040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-17T06:58:56.437 INFO:journalctl@ceph.mgr.x.smithi135.stdout:Sep 17 06:58:56 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-x[35098]: 2024-09-17T06:58:56.276+0000 7f03b38c2040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-17T06:58:56.437 INFO:journalctl@ceph.mgr.x.smithi135.stdout:Sep 17 06:58:56 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-x[35098]: 2024-09-17T06:58:56.436+0000 7f03b38c2040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-17T06:58:56.793 INFO:journalctl@ceph.mgr.x.smithi135.stdout:Sep 17 06:58:56 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-x[35098]: 2024-09-17T06:58:56.517+0000 7f03b38c2040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-17T06:58:56.793 INFO:journalctl@ceph.mgr.x.smithi135.stdout:Sep 17 06:58:56 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-x[35098]: 2024-09-17T06:58:56.685+0000 7f03b38c2040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-17T06:58:57.142 INFO:journalctl@ceph.mgr.x.smithi135.stdout:Sep 17 06:58:56 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-x[35098]: 2024-09-17T06:58:56.823+0000 7f03b38c2040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-17T06:58:57.142 INFO:journalctl@ceph.mgr.x.smithi135.stdout:Sep 17 06:58:57 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-x[35098]: 2024-09-17T06:58:57.142+0000 7f03b38c2040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-17T06:58:57.399 INFO:journalctl@ceph.mgr.x.smithi135.stdout:Sep 17 06:58:57 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-x[35098]: 2024-09-17T06:58:57.201+0000 7f03b38c2040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-17T06:58:57.399 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:57 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:57.399 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:57 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:57.400 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:57 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:57.400 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:57 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:57.400 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:57 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T06:58:57.400 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:57 smithi135 ceph-mon[33238]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T06:58:57.502 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:57 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:57.502 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:57 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:57.502 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:57 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:57.502 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:57 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:57.502 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:57 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T06:58:57.503 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:57 smithi100 ceph-mon[29092]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T06:58:57.503 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:57 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:57.503 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:57 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:57.503 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:57 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:57.503 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:57 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:57.503 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:57 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T06:58:57.503 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:57 smithi100 ceph-mon[39462]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T06:58:57.652 INFO:teuthology.orchestra.run.smithi100.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-09-17T06:58:57.744 INFO:teuthology.orchestra.run.smithi100.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-09-17T06:58:57.759 INFO:teuthology.orchestra.run.smithi100.stderr: stderr: 10+0 records in 2024-09-17T06:58:57.759 INFO:teuthology.orchestra.run.smithi100.stderr:10+0 records out 2024-09-17T06:58:57.759 INFO:teuthology.orchestra.run.smithi100.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0129582 s, 809 MB/s 2024-09-17T06:58:57.759 INFO:teuthology.orchestra.run.smithi100.stderr:--> Zapping successful for: 2024-09-17T06:58:57.767 INFO:journalctl@ceph.mgr.x.smithi135.stdout:Sep 17 06:58:57 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-x[35098]: 2024-09-17T06:58:57.398+0000 7f03b38c2040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-17T06:58:58.340 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:58 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:58.340 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:58 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:58.340 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:58 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T06:58:58.340 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:58 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T06:58:58.341 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:58 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:58.341 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:58 smithi100 ceph-mon[39462]: Reconfiguring mgr.y (unknown last config time)... 2024-09-17T06:58:58.341 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:58 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T06:58:58.341 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:58 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T06:58:58.341 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:58 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T06:58:58.341 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:58 smithi100 ceph-mon[39462]: Reconfiguring daemon mgr.y on smithi100 2024-09-17T06:58:58.341 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:58 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:58.341 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:58 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:58.341 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:58 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T06:58:58.341 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:58 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T06:58:58.341 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:58 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:58.342 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:58 smithi100 ceph-mon[29092]: Reconfiguring mgr.y (unknown last config time)... 2024-09-17T06:58:58.342 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:58 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T06:58:58.342 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:58 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T06:58:58.342 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:58 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T06:58:58.342 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:58 smithi100 ceph-mon[29092]: Reconfiguring daemon mgr.y on smithi100 2024-09-17T06:58:58.546 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:58 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:58.546 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:58 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:58.546 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:58 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T06:58:58.546 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:58 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T06:58:58.546 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:58 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:58.546 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:58 smithi135 ceph-mon[33238]: Reconfiguring mgr.y (unknown last config time)... 2024-09-17T06:58:58.547 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:58 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T06:58:58.547 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:58 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T06:58:58.547 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:58 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T06:58:58.547 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:58 smithi135 ceph-mon[33238]: Reconfiguring daemon mgr.y on smithi100 2024-09-17T06:58:58.547 INFO:journalctl@ceph.mgr.x.smithi135.stdout:Sep 17 06:58:58 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-x[35098]: 2024-09-17T06:58:58.309+0000 7f03b38c2040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-17T06:58:58.547 INFO:journalctl@ceph.mgr.x.smithi135.stdout:Sep 17 06:58:58 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-x[35098]: 2024-09-17T06:58:58.376+0000 7f03b38c2040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-17T06:58:59.208 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi100:vg_nvme/lv_4 2024-09-17T06:58:59.345 INFO:journalctl@ceph.mgr.x.smithi135.stdout:Sep 17 06:58:59 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-x[35098]: 2024-09-17T06:58:59.094+0000 7f03b38c2040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-17T06:58:59.345 INFO:journalctl@ceph.mgr.x.smithi135.stdout:Sep 17 06:58:59 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-x[35098]: 2024-09-17T06:58:59.206+0000 7f03b38c2040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-17T06:58:59.345 INFO:journalctl@ceph.mgr.x.smithi135.stdout:Sep 17 06:58:59 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-x[35098]: 2024-09-17T06:58:59.344+0000 7f03b38c2040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-17T06:58:59.402 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T06:58:59.616 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:59 smithi135 ceph-mon[33238]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T06:58:59.617 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:59 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:59.617 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:59 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:59.617 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:59 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:59.617 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:59 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T06:58:59.617 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:59 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T06:58:59.617 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:59 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T06:58:59.617 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:58:59 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:59.617 INFO:journalctl@ceph.mgr.x.smithi135.stdout:Sep 17 06:58:59 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-x[35098]: 2024-09-17T06:58:59.405+0000 7f03b38c2040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-17T06:58:59.617 INFO:journalctl@ceph.mgr.x.smithi135.stdout:Sep 17 06:58:59 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-x[35098]: 2024-09-17T06:58:59.548+0000 7f03b38c2040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-17T06:58:59.617 INFO:journalctl@ceph.mgr.x.smithi135.stdout:Sep 17 06:58:59 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-x[35098]: 2024-09-17T06:58:59.616+0000 7f03b38c2040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-17T06:58:59.857 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:59 smithi100 ceph-mon[29092]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T06:58:59.857 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:59 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:59.858 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:59 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:59.858 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:59 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:59.858 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:59 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T06:58:59.858 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:59 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T06:58:59.858 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:59 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T06:58:59.858 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:58:59 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:59.858 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:59 smithi100 ceph-mon[39462]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T06:58:59.858 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:59 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:59.858 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:59 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:59.858 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:59 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:59.858 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:59 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T06:58:59.858 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:59 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T06:58:59.858 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:59 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T06:58:59.859 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:58:59 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:58:59.890 INFO:journalctl@ceph.mgr.x.smithi135.stdout:Sep 17 06:58:59 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-x[35098]: 2024-09-17T06:58:59.675+0000 7f03b38c2040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-17T06:58:59.890 INFO:journalctl@ceph.mgr.x.smithi135.stdout:Sep 17 06:58:59 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-x[35098]: 2024-09-17T06:58:59.827+0000 7f03b38c2040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-17T06:58:59.891 INFO:journalctl@ceph.mgr.x.smithi135.stdout:Sep 17 06:58:59 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-x[35098]: 2024-09-17T06:58:59.890+0000 7f03b38c2040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-17T06:59:00.154 INFO:journalctl@ceph.mgr.x.smithi135.stdout:Sep 17 06:58:59 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-x[35098]: 2024-09-17T06:58:59.954+0000 7f03b38c2040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-17T06:59:00.154 INFO:journalctl@ceph.mgr.x.smithi135.stdout:Sep 17 06:59:00 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-x[35098]: 2024-09-17T06:59:00.037+0000 7f03b38c2040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-17T06:59:00.154 INFO:journalctl@ceph.mgr.x.smithi135.stdout:Sep 17 06:59:00 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-x[35098]: 2024-09-17T06:59:00.095+0000 7f03b38c2040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-17T06:59:00.154 INFO:journalctl@ceph.mgr.x.smithi135.stdout:Sep 17 06:59:00 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-x[35098]: 2024-09-17T06:59:00.153+0000 7f03b38c2040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-17T06:59:01.293 INFO:journalctl@ceph.mgr.x.smithi135.stdout:Sep 17 06:59:00 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-x[35098]: 2024-09-17T06:59:00.913+0000 7f03b38c2040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-17T06:59:01.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:00 smithi135 ceph-mon[33238]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T06:59:01.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:00 smithi135 ceph-mon[33238]: Standby manager daemon x started 2024-09-17T06:59:01.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:00 smithi135 ceph-mon[33238]: from='mgr.? 172.21.15.135:0/1592667664' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-17T06:59:01.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:00 smithi135 ceph-mon[33238]: from='mgr.? 172.21.15.135:0/1592667664' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T06:59:01.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:00 smithi135 ceph-mon[33238]: from='mgr.? 172.21.15.135:0/1592667664' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-17T06:59:01.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:00 smithi135 ceph-mon[33238]: from='mgr.? 172.21.15.135:0/1592667664' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T06:59:01.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:00 smithi100 ceph-mon[29092]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T06:59:01.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:00 smithi100 ceph-mon[29092]: Standby manager daemon x started 2024-09-17T06:59:01.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:00 smithi100 ceph-mon[29092]: from='mgr.? 172.21.15.135:0/1592667664' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-17T06:59:01.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:00 smithi100 ceph-mon[29092]: from='mgr.? 172.21.15.135:0/1592667664' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T06:59:01.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:00 smithi100 ceph-mon[29092]: from='mgr.? 172.21.15.135:0/1592667664' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-17T06:59:01.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:00 smithi100 ceph-mon[29092]: from='mgr.? 172.21.15.135:0/1592667664' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T06:59:01.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:00 smithi100 ceph-mon[39462]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T06:59:01.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:00 smithi100 ceph-mon[39462]: Standby manager daemon x started 2024-09-17T06:59:01.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:00 smithi100 ceph-mon[39462]: from='mgr.? 172.21.15.135:0/1592667664' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-17T06:59:01.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:00 smithi100 ceph-mon[39462]: from='mgr.? 172.21.15.135:0/1592667664' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T06:59:01.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:00 smithi100 ceph-mon[39462]: from='mgr.? 172.21.15.135:0/1592667664' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-17T06:59:01.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:00 smithi100 ceph-mon[39462]: from='mgr.? 172.21.15.135:0/1592667664' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T06:59:02.120 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:01 smithi100 ceph-mon[39462]: mgrmap e14: y(active, since 62s), standbys: x 2024-09-17T06:59:02.120 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:01 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-17T06:59:02.120 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:01 smithi100 ceph-mon[39462]: from='client.24119 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi100:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T06:59:02.120 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:01 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T06:59:02.120 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:01 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T06:59:02.120 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:01 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T06:59:02.121 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:01 smithi100 ceph-mon[29092]: mgrmap e14: y(active, since 62s), standbys: x 2024-09-17T06:59:02.121 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:01 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-17T06:59:02.121 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:01 smithi100 ceph-mon[29092]: from='client.24119 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi100:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T06:59:02.121 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:01 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T06:59:02.121 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:01 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T06:59:02.121 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:01 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T06:59:02.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:01 smithi135 ceph-mon[33238]: mgrmap e14: y(active, since 62s), standbys: x 2024-09-17T06:59:02.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:01 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-17T06:59:02.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:01 smithi135 ceph-mon[33238]: from='client.24119 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi100:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T06:59:02.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:01 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T06:59:02.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:01 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T06:59:02.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:01 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T06:59:03.126 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:02 smithi100 ceph-mon[29092]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T06:59:03.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:02 smithi100 ceph-mon[39462]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T06:59:03.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:02 smithi135 ceph-mon[33238]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T06:59:04.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:03 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/4004369650' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "27bee9ec-05e6-406c-8bf1-168098da0ec7"}]: dispatch 2024-09-17T06:59:04.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:03 smithi100 ceph-mon[29092]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "27bee9ec-05e6-406c-8bf1-168098da0ec7"}]: dispatch 2024-09-17T06:59:04.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:03 smithi100 ceph-mon[29092]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "27bee9ec-05e6-406c-8bf1-168098da0ec7"}]': finished 2024-09-17T06:59:04.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:03 smithi100 ceph-mon[29092]: osdmap e5: 1 total, 0 up, 1 in 2024-09-17T06:59:04.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:03 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T06:59:04.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:03 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/3585853287' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T06:59:04.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:03 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/4004369650' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "27bee9ec-05e6-406c-8bf1-168098da0ec7"}]: dispatch 2024-09-17T06:59:04.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:03 smithi100 ceph-mon[39462]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "27bee9ec-05e6-406c-8bf1-168098da0ec7"}]: dispatch 2024-09-17T06:59:04.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:03 smithi100 ceph-mon[39462]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "27bee9ec-05e6-406c-8bf1-168098da0ec7"}]': finished 2024-09-17T06:59:04.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:03 smithi100 ceph-mon[39462]: osdmap e5: 1 total, 0 up, 1 in 2024-09-17T06:59:04.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:03 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T06:59:04.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:03 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/3585853287' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T06:59:04.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:03 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/4004369650' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "27bee9ec-05e6-406c-8bf1-168098da0ec7"}]: dispatch 2024-09-17T06:59:04.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:03 smithi135 ceph-mon[33238]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "27bee9ec-05e6-406c-8bf1-168098da0ec7"}]: dispatch 2024-09-17T06:59:04.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:03 smithi135 ceph-mon[33238]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "27bee9ec-05e6-406c-8bf1-168098da0ec7"}]': finished 2024-09-17T06:59:04.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:03 smithi135 ceph-mon[33238]: osdmap e5: 1 total, 0 up, 1 in 2024-09-17T06:59:04.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:03 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T06:59:04.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:03 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/3585853287' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T06:59:05.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:04 smithi135 ceph-mon[33238]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T06:59:05.376 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:04 smithi100 ceph-mon[39462]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T06:59:05.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:04 smithi100 ceph-mon[29092]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T06:59:07.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:07 smithi100 ceph-mon[39462]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T06:59:07.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:07 smithi100 ceph-mon[29092]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T06:59:07.996 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:07 smithi135 ceph-mon[33238]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T06:59:09.822 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:09 smithi100 ceph-mon[29092]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T06:59:09.822 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:09 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-17T06:59:09.822 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:09 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T06:59:09.823 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:09 smithi100 ceph-mon[39462]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T06:59:09.823 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:09 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-17T06:59:09.823 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:09 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T06:59:10.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:09 smithi135 ceph-mon[33238]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T06:59:10.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:09 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-17T06:59:10.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:09 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T06:59:10.745 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:10 smithi100 ceph-mon[29092]: Deploying daemon osd.0 on smithi100 2024-09-17T06:59:10.745 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:10 smithi100 ceph-mon[39462]: Deploying daemon osd.0 on smithi100 2024-09-17T06:59:11.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:10 smithi135 ceph-mon[33238]: Deploying daemon osd.0 on smithi100 2024-09-17T06:59:11.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:11 smithi100 ceph-mon[29092]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T06:59:11.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:11 smithi100 ceph-mon[39462]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T06:59:12.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:11 smithi135 ceph-mon[33238]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T06:59:13.743 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:13 smithi100 ceph-mon[29092]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T06:59:13.743 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:13 smithi100 ceph-mon[39462]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T06:59:14.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:13 smithi135 ceph-mon[33238]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T06:59:14.693 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:14 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T06:59:14.693 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:14 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:59:14.693 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:14 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:59:14.693 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:14 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T06:59:14.693 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:14 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:59:14.693 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:14 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:59:15.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:14 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T06:59:15.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:14 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:59:15.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:14 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:59:15.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:15 smithi100 ceph-mon[29092]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T06:59:15.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:15 smithi100 ceph-mon[39462]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T06:59:16.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:15 smithi135 ceph-mon[33238]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T06:59:17.251 INFO:teuthology.orchestra.run.smithi100.stdout:Created osd(s) 0 on host 'smithi100' 2024-09-17T06:59:17.791 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:17 smithi100 ceph-mon[39462]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T06:59:17.791 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:17 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:59:17.791 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:17 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:59:17.792 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:17 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T06:59:17.792 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:17 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T06:59:17.792 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:17 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:59:17.792 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:17 smithi100 ceph-mon[39462]: from='osd.0 v2:172.21.15.100:6801/3341017455' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-17T06:59:17.792 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:17 smithi100 ceph-mon[39462]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-17T06:59:17.792 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:17 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T06:59:17.792 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:17 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:59:17.792 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:17 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:59:17.792 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:17 smithi100 ceph-mon[29092]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T06:59:17.792 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:17 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:59:17.792 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:17 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:59:17.793 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:17 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T06:59:17.793 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:17 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T06:59:17.793 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:17 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:59:17.793 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:17 smithi100 ceph-mon[29092]: from='osd.0 v2:172.21.15.100:6801/3341017455' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-17T06:59:17.793 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:17 smithi100 ceph-mon[29092]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-17T06:59:17.793 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:17 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T06:59:17.793 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:17 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:59:17.793 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:17 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:59:17.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:17 smithi135 ceph-mon[33238]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T06:59:17.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:17 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:59:17.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:17 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:59:17.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:17 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T06:59:17.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:17 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T06:59:17.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:17 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:59:17.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:17 smithi135 ceph-mon[33238]: from='osd.0 v2:172.21.15.100:6801/3341017455' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-17T06:59:17.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:17 smithi135 ceph-mon[33238]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-17T06:59:17.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:17 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T06:59:17.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:17 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:59:17.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:17 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:59:17.811 DEBUG:teuthology.orchestra.run.smithi100:osd.0> sudo journalctl -f -n 0 -u ceph-e848267c-74c1-11ef-bceb-c7b262605968@osd.0.service 2024-09-17T06:59:17.814 INFO:tasks.cephadm:Deploying osd.1 on smithi100 with /dev/vg_nvme/lv_3... 2024-09-17T06:59:17.814 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-09-17T06:59:18.553 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T06:59:19.196 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:18 smithi100 ceph-mon[29092]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-17T06:59:19.196 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:18 smithi100 ceph-mon[29092]: osdmap e6: 1 total, 0 up, 1 in 2024-09-17T06:59:19.196 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:18 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T06:59:19.196 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:18 smithi100 ceph-mon[29092]: from='osd.0 v2:172.21.15.100:6801/3341017455' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]: dispatch 2024-09-17T06:59:19.197 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:18 smithi100 ceph-mon[29092]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]: dispatch 2024-09-17T06:59:19.197 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:18 smithi100 ceph-mon[29092]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T06:59:19.197 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:18 smithi100 ceph-mon[39462]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-17T06:59:19.197 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:18 smithi100 ceph-mon[39462]: osdmap e6: 1 total, 0 up, 1 in 2024-09-17T06:59:19.197 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:18 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T06:59:19.197 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:18 smithi100 ceph-mon[39462]: from='osd.0 v2:172.21.15.100:6801/3341017455' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]: dispatch 2024-09-17T06:59:19.197 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:18 smithi100 ceph-mon[39462]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]: dispatch 2024-09-17T06:59:19.197 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:18 smithi100 ceph-mon[39462]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T06:59:19.198 INFO:journalctl@ceph.osd.0.smithi100.stdout:Sep 17 06:59:18 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-0[43366]: 2024-09-17T06:59:18.951+0000 7f16f8aed640 -1 osd.0 0 waiting for initial osdmap 2024-09-17T06:59:19.198 INFO:journalctl@ceph.osd.0.smithi100.stdout:Sep 17 06:59:18 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-0[43366]: 2024-09-17T06:59:18.953+0000 7f16f4916640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T06:59:19.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:18 smithi135 ceph-mon[33238]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-17T06:59:19.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:18 smithi135 ceph-mon[33238]: osdmap e6: 1 total, 0 up, 1 in 2024-09-17T06:59:19.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:18 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T06:59:19.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:18 smithi135 ceph-mon[33238]: from='osd.0 v2:172.21.15.100:6801/3341017455' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]: dispatch 2024-09-17T06:59:19.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:18 smithi135 ceph-mon[33238]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]: dispatch 2024-09-17T06:59:19.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:18 smithi135 ceph-mon[33238]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T06:59:20.067 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:19 smithi100 ceph-mon[29092]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]': finished 2024-09-17T06:59:20.068 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:19 smithi100 ceph-mon[29092]: osdmap e7: 1 total, 0 up, 1 in 2024-09-17T06:59:20.068 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:19 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T06:59:20.068 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:19 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T06:59:20.068 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:19 smithi100 ceph-mon[39462]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]': finished 2024-09-17T06:59:20.068 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:19 smithi100 ceph-mon[39462]: osdmap e7: 1 total, 0 up, 1 in 2024-09-17T06:59:20.068 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:19 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T06:59:20.068 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:19 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T06:59:20.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:19 smithi135 ceph-mon[33238]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]': finished 2024-09-17T06:59:20.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:19 smithi135 ceph-mon[33238]: osdmap e7: 1 total, 0 up, 1 in 2024-09-17T06:59:20.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:19 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T06:59:20.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:19 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T06:59:21.217 INFO:teuthology.orchestra.run.smithi100.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-09-17T06:59:21.217 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:20 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T06:59:21.217 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:20 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T06:59:21.217 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:20 smithi100 ceph-mon[29092]: osd.0 v2:172.21.15.100:6801/3341017455 boot 2024-09-17T06:59:21.217 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:20 smithi100 ceph-mon[29092]: osdmap e8: 1 total, 1 up, 1 in 2024-09-17T06:59:21.217 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:20 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T06:59:21.217 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:20 smithi100 ceph-mon[29092]: pgmap v39: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T06:59:21.218 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:20 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T06:59:21.218 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:20 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T06:59:21.218 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:20 smithi100 ceph-mon[39462]: osd.0 v2:172.21.15.100:6801/3341017455 boot 2024-09-17T06:59:21.218 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:20 smithi100 ceph-mon[39462]: osdmap e8: 1 total, 1 up, 1 in 2024-09-17T06:59:21.218 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:20 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T06:59:21.218 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:20 smithi100 ceph-mon[39462]: pgmap v39: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T06:59:21.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:20 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T06:59:21.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:20 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T06:59:21.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:20 smithi135 ceph-mon[33238]: osd.0 v2:172.21.15.100:6801/3341017455 boot 2024-09-17T06:59:21.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:20 smithi135 ceph-mon[33238]: osdmap e8: 1 total, 1 up, 1 in 2024-09-17T06:59:21.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:20 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T06:59:21.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:20 smithi135 ceph-mon[33238]: pgmap v39: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T06:59:21.304 INFO:teuthology.orchestra.run.smithi100.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-09-17T06:59:21.318 INFO:teuthology.orchestra.run.smithi100.stderr: stderr: 10+0 records in 2024-09-17T06:59:21.318 INFO:teuthology.orchestra.run.smithi100.stderr:10+0 records out 2024-09-17T06:59:21.318 INFO:teuthology.orchestra.run.smithi100.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0122901 s, 853 MB/s 2024-09-17T06:59:21.318 INFO:teuthology.orchestra.run.smithi100.stderr:--> Zapping successful for: 2024-09-17T06:59:22.025 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi100:vg_nvme/lv_3 2024-09-17T06:59:22.219 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T06:59:22.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:21 smithi135 ceph-mon[33238]: osdmap e9: 1 total, 1 up, 1 in 2024-09-17T06:59:22.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:21 smithi135 ceph-mon[33238]: Detected new or changed devices on smithi100 2024-09-17T06:59:22.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:21 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:59:22.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:21 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:59:22.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:21 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-17T06:59:22.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:21 smithi135 ceph-mon[33238]: Adjusting osd_memory_target on smithi100 to 16063M 2024-09-17T06:59:22.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:21 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:59:22.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:21 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T06:59:22.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:21 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T06:59:22.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:21 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:59:22.339 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:21 smithi100 ceph-mon[39462]: osdmap e9: 1 total, 1 up, 1 in 2024-09-17T06:59:22.339 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:21 smithi100 ceph-mon[39462]: Detected new or changed devices on smithi100 2024-09-17T06:59:22.339 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:21 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:59:22.339 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:21 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:59:22.339 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:21 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-17T06:59:22.339 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:21 smithi100 ceph-mon[39462]: Adjusting osd_memory_target on smithi100 to 16063M 2024-09-17T06:59:22.339 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:21 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:59:22.339 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:21 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T06:59:22.339 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:21 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T06:59:22.339 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:21 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:59:22.340 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:21 smithi100 ceph-mon[29092]: osdmap e9: 1 total, 1 up, 1 in 2024-09-17T06:59:22.340 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:21 smithi100 ceph-mon[29092]: Detected new or changed devices on smithi100 2024-09-17T06:59:22.340 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:21 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:59:22.340 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:21 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:59:22.340 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:21 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-17T06:59:22.340 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:21 smithi100 ceph-mon[29092]: Adjusting osd_memory_target on smithi100 to 16063M 2024-09-17T06:59:22.340 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:21 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:59:22.340 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:21 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T06:59:22.340 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:21 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T06:59:22.340 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:21 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:59:23.235 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:22 smithi100 ceph-mon[29092]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T06:59:23.235 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:22 smithi100 ceph-mon[39462]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T06:59:23.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:22 smithi135 ceph-mon[33238]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T06:59:24.275 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:23 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T06:59:24.276 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:23 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T06:59:24.276 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:23 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T06:59:24.276 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:23 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T06:59:24.276 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:23 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T06:59:24.276 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:23 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T06:59:24.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:23 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T06:59:24.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:23 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T06:59:24.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:23 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T06:59:25.245 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:24 smithi100 ceph-mon[29092]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi100:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T06:59:25.246 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:24 smithi100 ceph-mon[29092]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T06:59:25.246 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:24 smithi100 ceph-mon[39462]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi100:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T06:59:25.246 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:24 smithi100 ceph-mon[39462]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T06:59:25.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:24 smithi135 ceph-mon[33238]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi100:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T06:59:25.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:24 smithi135 ceph-mon[33238]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T06:59:26.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:26 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/3974013645' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b1a30c9c-eeaf-478e-9b62-2d63ad45873d"}]: dispatch 2024-09-17T06:59:26.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:26 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/3974013645' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b1a30c9c-eeaf-478e-9b62-2d63ad45873d"}]': finished 2024-09-17T06:59:26.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:26 smithi135 ceph-mon[33238]: osdmap e10: 2 total, 1 up, 2 in 2024-09-17T06:59:26.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:26 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T06:59:26.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:26 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/3974013645' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b1a30c9c-eeaf-478e-9b62-2d63ad45873d"}]: dispatch 2024-09-17T06:59:26.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:26 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/3974013645' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b1a30c9c-eeaf-478e-9b62-2d63ad45873d"}]': finished 2024-09-17T06:59:26.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:26 smithi100 ceph-mon[39462]: osdmap e10: 2 total, 1 up, 2 in 2024-09-17T06:59:26.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:26 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T06:59:26.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:26 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/3974013645' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b1a30c9c-eeaf-478e-9b62-2d63ad45873d"}]: dispatch 2024-09-17T06:59:26.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:26 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/3974013645' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b1a30c9c-eeaf-478e-9b62-2d63ad45873d"}]': finished 2024-09-17T06:59:26.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:26 smithi100 ceph-mon[29092]: osdmap e10: 2 total, 1 up, 2 in 2024-09-17T06:59:26.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:26 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T06:59:27.383 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:27 smithi100 ceph-mon[39462]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T06:59:27.384 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:27 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1662470875' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T06:59:27.384 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:27 smithi100 ceph-mon[29092]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T06:59:27.384 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:27 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1662470875' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T06:59:27.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:27 smithi135 ceph-mon[33238]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T06:59:27.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:27 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1662470875' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T06:59:29.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:29 smithi100 ceph-mon[29092]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T06:59:29.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:29 smithi100 ceph-mon[39462]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T06:59:30.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:29 smithi135 ceph-mon[33238]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T06:59:31.634 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:31 smithi100 ceph-mon[29092]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T06:59:31.760 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:31 smithi100 ceph-mon[39462]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T06:59:32.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:31 smithi135 ceph-mon[33238]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T06:59:32.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:32 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-17T06:59:32.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:32 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T06:59:32.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:32 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-17T06:59:32.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:32 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T06:59:33.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:32 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-17T06:59:33.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:32 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T06:59:33.809 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:33 smithi100 ceph-mon[39462]: Deploying daemon osd.1 on smithi100 2024-09-17T06:59:33.809 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:33 smithi100 ceph-mon[39462]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T06:59:33.809 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:33 smithi100 ceph-mon[29092]: Deploying daemon osd.1 on smithi100 2024-09-17T06:59:33.809 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:33 smithi100 ceph-mon[29092]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T06:59:34.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:33 smithi135 ceph-mon[33238]: Deploying daemon osd.1 on smithi100 2024-09-17T06:59:34.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:33 smithi135 ceph-mon[33238]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T06:59:35.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:35 smithi100 ceph-mon[29092]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T06:59:35.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:35 smithi100 ceph-mon[39462]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T06:59:36.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:35 smithi135 ceph-mon[33238]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T06:59:37.858 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:37 smithi100 ceph-mon[29092]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T06:59:37.859 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:37 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T06:59:37.859 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:37 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:59:37.859 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:37 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:59:37.859 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:37 smithi100 ceph-mon[39462]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T06:59:37.859 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:37 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T06:59:37.859 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:37 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:59:37.859 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:37 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:59:38.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:37 smithi135 ceph-mon[33238]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T06:59:38.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:37 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T06:59:38.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:37 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:59:38.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:37 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:59:39.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:39 smithi100 ceph-mon[29092]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T06:59:39.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:39 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:59:39.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:39 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:59:39.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:39 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T06:59:39.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:39 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T06:59:39.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:39 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:59:39.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:39 smithi100 ceph-mon[39462]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T06:59:39.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:39 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:59:39.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:39 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:59:39.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:39 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T06:59:39.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:39 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T06:59:39.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:39 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:59:40.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:39 smithi135 ceph-mon[33238]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T06:59:40.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:39 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:59:40.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:39 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:59:40.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:39 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T06:59:40.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:39 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T06:59:40.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:39 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:59:40.222 INFO:teuthology.orchestra.run.smithi100.stdout:Created osd(s) 1 on host 'smithi100' 2024-09-17T06:59:40.830 DEBUG:teuthology.orchestra.run.smithi100:osd.1> sudo journalctl -f -n 0 -u ceph-e848267c-74c1-11ef-bceb-c7b262605968@osd.1.service 2024-09-17T06:59:40.833 INFO:tasks.cephadm:Deploying osd.2 on smithi100 with /dev/vg_nvme/lv_2... 2024-09-17T06:59:40.834 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-09-17T06:59:41.077 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:40 smithi100 ceph-mon[29092]: from='osd.1 v2:172.21.15.100:6805/1634750965' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-17T06:59:41.077 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:40 smithi100 ceph-mon[29092]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-17T06:59:41.078 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:40 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T06:59:41.078 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:40 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:59:41.078 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:40 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:59:41.078 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:40 smithi100 ceph-mon[29092]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T06:59:41.078 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:40 smithi100 ceph-mon[39462]: from='osd.1 v2:172.21.15.100:6805/1634750965' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-17T06:59:41.078 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:40 smithi100 ceph-mon[39462]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-17T06:59:41.078 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:40 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T06:59:41.078 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:40 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:59:41.079 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:40 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:59:41.079 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:40 smithi100 ceph-mon[39462]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T06:59:41.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:40 smithi135 ceph-mon[33238]: from='osd.1 v2:172.21.15.100:6805/1634750965' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-17T06:59:41.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:40 smithi135 ceph-mon[33238]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-17T06:59:41.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:40 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T06:59:41.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:40 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:59:41.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:40 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:59:41.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:40 smithi135 ceph-mon[33238]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T06:59:41.604 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T06:59:42.248 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:41 smithi100 ceph-mon[29092]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-17T06:59:42.248 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:41 smithi100 ceph-mon[29092]: osdmap e11: 2 total, 1 up, 2 in 2024-09-17T06:59:42.248 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:41 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T06:59:42.248 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:41 smithi100 ceph-mon[29092]: from='osd.1 v2:172.21.15.100:6805/1634750965' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]: dispatch 2024-09-17T06:59:42.249 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:41 smithi100 ceph-mon[29092]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]: dispatch 2024-09-17T06:59:42.249 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:41 smithi100 ceph-mon[39462]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-17T06:59:42.249 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:41 smithi100 ceph-mon[39462]: osdmap e11: 2 total, 1 up, 2 in 2024-09-17T06:59:42.249 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:41 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T06:59:42.250 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:41 smithi100 ceph-mon[39462]: from='osd.1 v2:172.21.15.100:6805/1634750965' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]: dispatch 2024-09-17T06:59:42.250 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:41 smithi100 ceph-mon[39462]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]: dispatch 2024-09-17T06:59:42.250 INFO:journalctl@ceph.osd.1.smithi100.stdout:Sep 17 06:59:41 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-1[48943]: 2024-09-17T06:59:41.985+0000 7f475b398640 -1 osd.1 0 waiting for initial osdmap 2024-09-17T06:59:42.250 INFO:journalctl@ceph.osd.1.smithi100.stdout:Sep 17 06:59:41 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-1[48943]: 2024-09-17T06:59:41.987+0000 7f47571c1640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T06:59:42.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:41 smithi135 ceph-mon[33238]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-17T06:59:42.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:41 smithi135 ceph-mon[33238]: osdmap e11: 2 total, 1 up, 2 in 2024-09-17T06:59:42.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:41 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T06:59:42.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:41 smithi135 ceph-mon[33238]: from='osd.1 v2:172.21.15.100:6805/1634750965' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]: dispatch 2024-09-17T06:59:42.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:41 smithi135 ceph-mon[33238]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]: dispatch 2024-09-17T06:59:43.213 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:42 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T06:59:43.214 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:42 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T06:59:43.214 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:42 smithi100 ceph-mon[29092]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]': finished 2024-09-17T06:59:43.214 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:42 smithi100 ceph-mon[29092]: osdmap e12: 2 total, 1 up, 2 in 2024-09-17T06:59:43.214 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:42 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T06:59:43.214 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:42 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T06:59:43.214 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:42 smithi100 ceph-mon[29092]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T06:59:43.214 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:42 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T06:59:43.214 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:42 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T06:59:43.214 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:42 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T06:59:43.214 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:42 smithi100 ceph-mon[39462]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]': finished 2024-09-17T06:59:43.215 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:42 smithi100 ceph-mon[39462]: osdmap e12: 2 total, 1 up, 2 in 2024-09-17T06:59:43.215 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:42 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T06:59:43.215 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:42 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T06:59:43.215 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:42 smithi100 ceph-mon[39462]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T06:59:43.215 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:42 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T06:59:43.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:42 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T06:59:43.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:42 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T06:59:43.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:42 smithi135 ceph-mon[33238]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]': finished 2024-09-17T06:59:43.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:42 smithi135 ceph-mon[33238]: osdmap e12: 2 total, 1 up, 2 in 2024-09-17T06:59:43.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:42 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T06:59:43.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:42 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T06:59:43.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:42 smithi135 ceph-mon[33238]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T06:59:43.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:42 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T06:59:44.072 INFO:teuthology.orchestra.run.smithi100.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-09-17T06:59:44.073 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:43 smithi100 ceph-mon[29092]: osd.1 v2:172.21.15.100:6805/1634750965 boot 2024-09-17T06:59:44.073 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:43 smithi100 ceph-mon[29092]: osdmap e13: 2 total, 2 up, 2 in 2024-09-17T06:59:44.073 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:43 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T06:59:44.073 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:44 smithi100 ceph-mon[39462]: osd.1 v2:172.21.15.100:6805/1634750965 boot 2024-09-17T06:59:44.073 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:44 smithi100 ceph-mon[39462]: osdmap e13: 2 total, 2 up, 2 in 2024-09-17T06:59:44.073 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:44 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T06:59:44.157 INFO:teuthology.orchestra.run.smithi100.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-09-17T06:59:44.170 INFO:teuthology.orchestra.run.smithi100.stderr: stderr: 10+0 records in 2024-09-17T06:59:44.171 INFO:teuthology.orchestra.run.smithi100.stderr:10+0 records out 2024-09-17T06:59:44.171 INFO:teuthology.orchestra.run.smithi100.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0123586 s, 848 MB/s 2024-09-17T06:59:44.171 INFO:teuthology.orchestra.run.smithi100.stderr:--> Zapping successful for: 2024-09-17T06:59:44.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:44 smithi135 ceph-mon[33238]: osd.1 v2:172.21.15.100:6805/1634750965 boot 2024-09-17T06:59:44.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:44 smithi135 ceph-mon[33238]: osdmap e13: 2 total, 2 up, 2 in 2024-09-17T06:59:44.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:44 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T06:59:44.773 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi100:vg_nvme/lv_2 2024-09-17T06:59:45.281 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T06:59:45.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:45 smithi135 ceph-mon[33238]: osdmap e14: 2 total, 2 up, 2 in 2024-09-17T06:59:45.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:45 smithi135 ceph-mon[33238]: pgmap v57: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-17T06:59:45.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:45 smithi100 ceph-mon[29092]: osdmap e14: 2 total, 2 up, 2 in 2024-09-17T06:59:45.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:45 smithi100 ceph-mon[29092]: pgmap v57: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-17T06:59:45.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:45 smithi100 ceph-mon[39462]: osdmap e14: 2 total, 2 up, 2 in 2024-09-17T06:59:45.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:45 smithi100 ceph-mon[39462]: pgmap v57: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-17T06:59:46.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:46 smithi135 ceph-mon[33238]: Detected new or changed devices on smithi100 2024-09-17T06:59:46.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:46 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:59:46.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:46 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:59:46.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:46 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-17T06:59:46.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:46 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-17T06:59:46.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:46 smithi135 ceph-mon[33238]: Adjusting osd_memory_target on smithi100 to 8031M 2024-09-17T06:59:46.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:46 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:59:46.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:46 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T06:59:46.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:46 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T06:59:46.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:46 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:59:46.588 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:46 smithi100 ceph-mon[29092]: Detected new or changed devices on smithi100 2024-09-17T06:59:46.588 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:46 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:59:46.588 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:46 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:59:46.588 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:46 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-17T06:59:46.589 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:46 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-17T06:59:46.589 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:46 smithi100 ceph-mon[29092]: Adjusting osd_memory_target on smithi100 to 8031M 2024-09-17T06:59:46.589 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:46 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:59:46.589 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:46 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T06:59:46.589 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:46 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T06:59:46.589 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:46 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:59:46.589 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:46 smithi100 ceph-mon[39462]: Detected new or changed devices on smithi100 2024-09-17T06:59:46.589 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:46 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:59:46.589 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:46 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:59:46.589 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:46 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-17T06:59:46.589 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:46 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-17T06:59:46.590 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:46 smithi100 ceph-mon[39462]: Adjusting osd_memory_target on smithi100 to 8031M 2024-09-17T06:59:46.590 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:46 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:59:46.590 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:46 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T06:59:46.590 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:46 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T06:59:46.590 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:46 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T06:59:47.523 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:47 smithi100 ceph-mon[39462]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-17T06:59:47.524 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:47 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T06:59:47.524 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:47 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T06:59:47.524 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:47 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T06:59:47.524 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:47 smithi100 ceph-mon[29092]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-17T06:59:47.524 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:47 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T06:59:47.524 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:47 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T06:59:47.524 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:47 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T06:59:47.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:47 smithi135 ceph-mon[33238]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-17T06:59:47.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:47 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T06:59:47.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:47 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T06:59:47.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:47 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T06:59:48.516 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:48 smithi100 ceph-mon[29092]: from='client.14265 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi100:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T06:59:48.516 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:48 smithi100 ceph-mon[39462]: from='client.14265 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi100:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T06:59:48.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:48 smithi135 ceph-mon[33238]: from='client.14265 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi100:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T06:59:49.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:49 smithi135 ceph-mon[33238]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-17T06:59:49.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:49 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1101278106' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ce542e0d-9915-465e-aa11-7b3ce13f694f"}]: dispatch 2024-09-17T06:59:49.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:49 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1101278106' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ce542e0d-9915-465e-aa11-7b3ce13f694f"}]': finished 2024-09-17T06:59:49.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:49 smithi135 ceph-mon[33238]: osdmap e15: 3 total, 2 up, 3 in 2024-09-17T06:59:49.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:49 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T06:59:49.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:49 smithi100 ceph-mon[29092]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-17T06:59:49.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:49 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1101278106' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ce542e0d-9915-465e-aa11-7b3ce13f694f"}]: dispatch 2024-09-17T06:59:49.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:49 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1101278106' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ce542e0d-9915-465e-aa11-7b3ce13f694f"}]': finished 2024-09-17T06:59:49.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:49 smithi100 ceph-mon[29092]: osdmap e15: 3 total, 2 up, 3 in 2024-09-17T06:59:49.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:49 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T06:59:49.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:49 smithi100 ceph-mon[39462]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-17T06:59:49.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:49 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1101278106' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ce542e0d-9915-465e-aa11-7b3ce13f694f"}]: dispatch 2024-09-17T06:59:49.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:49 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1101278106' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ce542e0d-9915-465e-aa11-7b3ce13f694f"}]': finished 2024-09-17T06:59:49.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:49 smithi100 ceph-mon[39462]: osdmap e15: 3 total, 2 up, 3 in 2024-09-17T06:59:49.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:49 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T06:59:50.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:50 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/621640538' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T06:59:50.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:50 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/621640538' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T06:59:50.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:50 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/621640538' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T06:59:51.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:51 smithi135 ceph-mon[33238]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-17T06:59:51.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:51 smithi100 ceph-mon[29092]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-17T06:59:51.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:51 smithi100 ceph-mon[39462]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-17T06:59:53.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:53 smithi100 ceph-mon[29092]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-17T06:59:53.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:53 smithi100 ceph-mon[39462]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-17T06:59:54.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:53 smithi135 ceph-mon[33238]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-17T06:59:55.827 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:55 smithi100 ceph-mon[29092]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-17T06:59:55.827 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:55 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-17T06:59:55.827 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:55 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T06:59:55.828 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:55 smithi100 ceph-mon[39462]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-17T06:59:55.828 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:55 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-17T06:59:55.828 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:55 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T06:59:56.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:55 smithi135 ceph-mon[33238]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-17T06:59:56.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:55 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-17T06:59:56.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:55 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T06:59:56.763 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:56 smithi100 ceph-mon[29092]: Deploying daemon osd.2 on smithi100 2024-09-17T06:59:56.763 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:56 smithi100 ceph-mon[39462]: Deploying daemon osd.2 on smithi100 2024-09-17T06:59:57.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:56 smithi135 ceph-mon[33238]: Deploying daemon osd.2 on smithi100 2024-09-17T06:59:57.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:57 smithi100 ceph-mon[29092]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-17T06:59:57.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:57 smithi100 ceph-mon[39462]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-17T06:59:58.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:57 smithi135 ceph-mon[33238]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-17T06:59:59.697 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 06:59:59 smithi100 ceph-mon[29092]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-17T06:59:59.697 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 06:59:59 smithi100 ceph-mon[39462]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-17T07:00:00.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 06:59:59 smithi135 ceph-mon[33238]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-17T07:00:01.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:00 smithi135 ceph-mon[33238]: overall HEALTH_OK 2024-09-17T07:00:01.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:00 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:00:01.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:00 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:01.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:00 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:01.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:00 smithi135 ceph-mon[33238]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-17T07:00:01.373 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:00 smithi100 ceph-mon[29092]: overall HEALTH_OK 2024-09-17T07:00:01.373 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:00 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:00:01.373 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:00 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:01.373 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:00 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:01.373 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:00 smithi100 ceph-mon[29092]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-17T07:00:01.374 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:00 smithi100 ceph-mon[39462]: overall HEALTH_OK 2024-09-17T07:00:01.374 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:00 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:00:01.374 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:00 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:01.374 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:00 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:01.374 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:00 smithi100 ceph-mon[39462]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-17T07:00:03.374 INFO:teuthology.orchestra.run.smithi100.stdout:Created osd(s) 2 on host 'smithi100' 2024-09-17T07:00:03.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:03 smithi100 ceph-mon[29092]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-17T07:00:03.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:03 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:03.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:03 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:03.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:03 smithi100 ceph-mon[29092]: from='osd.2 v2:172.21.15.100:6809/714579846' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-17T07:00:03.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:03 smithi100 ceph-mon[39462]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-17T07:00:03.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:03 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:03.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:03 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:03.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:03 smithi100 ceph-mon[39462]: from='osd.2 v2:172.21.15.100:6809/714579846' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-17T07:00:04.001 DEBUG:teuthology.orchestra.run.smithi100:osd.2> sudo journalctl -f -n 0 -u ceph-e848267c-74c1-11ef-bceb-c7b262605968@osd.2.service 2024-09-17T07:00:04.004 INFO:tasks.cephadm:Deploying osd.3 on smithi100 with /dev/vg_nvme/lv_1... 2024-09-17T07:00:04.004 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-09-17T07:00:04.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:03 smithi135 ceph-mon[33238]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-17T07:00:04.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:03 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:04.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:03 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:04.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:03 smithi135 ceph-mon[33238]: from='osd.2 v2:172.21.15.100:6809/714579846' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-17T07:00:04.257 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:00:05.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:05 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:05.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:05 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:05.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:05 smithi100 ceph-mon[29092]: from='osd.2 v2:172.21.15.100:6809/714579846' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-17T07:00:05.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:05 smithi100 ceph-mon[29092]: osdmap e16: 3 total, 2 up, 3 in 2024-09-17T07:00:05.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:05 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T07:00:05.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:05 smithi100 ceph-mon[29092]: from='osd.2 v2:172.21.15.100:6809/714579846' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]: dispatch 2024-09-17T07:00:05.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:05 smithi100 ceph-mon[29092]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-17T07:00:05.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:05 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:05.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:05 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:05.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:05 smithi100 ceph-mon[39462]: from='osd.2 v2:172.21.15.100:6809/714579846' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-17T07:00:05.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:05 smithi100 ceph-mon[39462]: osdmap e16: 3 total, 2 up, 3 in 2024-09-17T07:00:05.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:05 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T07:00:05.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:05 smithi100 ceph-mon[39462]: from='osd.2 v2:172.21.15.100:6809/714579846' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]: dispatch 2024-09-17T07:00:05.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:05 smithi100 ceph-mon[39462]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-17T07:00:05.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:05 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:05.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:05 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:05.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:05 smithi135 ceph-mon[33238]: from='osd.2 v2:172.21.15.100:6809/714579846' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-17T07:00:05.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:05 smithi135 ceph-mon[33238]: osdmap e16: 3 total, 2 up, 3 in 2024-09-17T07:00:05.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:05 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T07:00:05.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:05 smithi135 ceph-mon[33238]: from='osd.2 v2:172.21.15.100:6809/714579846' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]: dispatch 2024-09-17T07:00:05.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:05 smithi135 ceph-mon[33238]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-17T07:00:06.127 INFO:journalctl@ceph.osd.2.smithi100.stdout:Sep 17 07:00:05 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-2[54495]: 2024-09-17T07:00:05.834+0000 7fafc92fb640 -1 osd.2 0 waiting for initial osdmap 2024-09-17T07:00:06.127 INFO:journalctl@ceph.osd.2.smithi100.stdout:Sep 17 07:00:05 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-2[54495]: 2024-09-17T07:00:05.839+0000 7fafc0905640 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T07:00:06.410 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:06 smithi100 ceph-mon[29092]: from='osd.2 v2:172.21.15.100:6809/714579846' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]': finished 2024-09-17T07:00:06.410 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:06 smithi100 ceph-mon[29092]: osdmap e17: 3 total, 2 up, 3 in 2024-09-17T07:00:06.410 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:06 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T07:00:06.410 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:06 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T07:00:06.410 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:06 smithi100 ceph-mon[29092]: from='osd.2 v2:172.21.15.100:6809/714579846' entity='osd.2' 2024-09-17T07:00:06.411 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:06 smithi100 ceph-mon[39462]: from='osd.2 v2:172.21.15.100:6809/714579846' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]': finished 2024-09-17T07:00:06.411 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:06 smithi100 ceph-mon[39462]: osdmap e17: 3 total, 2 up, 3 in 2024-09-17T07:00:06.411 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:06 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T07:00:06.411 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:06 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T07:00:06.411 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:06 smithi100 ceph-mon[39462]: from='osd.2 v2:172.21.15.100:6809/714579846' entity='osd.2' 2024-09-17T07:00:06.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:06 smithi135 ceph-mon[33238]: from='osd.2 v2:172.21.15.100:6809/714579846' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]': finished 2024-09-17T07:00:06.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:06 smithi135 ceph-mon[33238]: osdmap e17: 3 total, 2 up, 3 in 2024-09-17T07:00:06.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:06 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T07:00:06.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:06 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T07:00:06.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:06 smithi135 ceph-mon[33238]: from='osd.2 v2:172.21.15.100:6809/714579846' entity='osd.2' 2024-09-17T07:00:07.532 INFO:teuthology.orchestra.run.smithi100.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-09-17T07:00:07.532 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:07 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:00:07.533 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:07 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:00:07.533 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:07 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T07:00:07.533 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:07 smithi100 ceph-mon[29092]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-17T07:00:07.533 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:07 smithi100 ceph-mon[29092]: osd.2 v2:172.21.15.100:6809/714579846 boot 2024-09-17T07:00:07.533 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:07 smithi100 ceph-mon[29092]: osdmap e18: 3 total, 3 up, 3 in 2024-09-17T07:00:07.533 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:07 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T07:00:07.533 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:07 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:00:07.533 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:07 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:00:07.533 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:07 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T07:00:07.533 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:07 smithi100 ceph-mon[39462]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-17T07:00:07.533 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:07 smithi100 ceph-mon[39462]: osd.2 v2:172.21.15.100:6809/714579846 boot 2024-09-17T07:00:07.533 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:07 smithi100 ceph-mon[39462]: osdmap e18: 3 total, 3 up, 3 in 2024-09-17T07:00:07.534 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:07 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T07:00:07.632 INFO:teuthology.orchestra.run.smithi100.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-09-17T07:00:07.648 INFO:teuthology.orchestra.run.smithi100.stderr: stderr: 10+0 records in 2024-09-17T07:00:07.648 INFO:teuthology.orchestra.run.smithi100.stderr:10+0 records out 2024-09-17T07:00:07.648 INFO:teuthology.orchestra.run.smithi100.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0139083 s, 754 MB/s 2024-09-17T07:00:07.649 INFO:teuthology.orchestra.run.smithi100.stderr:--> Zapping successful for: 2024-09-17T07:00:07.695 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:07 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:00:07.695 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:07 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:00:07.695 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:07 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T07:00:07.695 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:07 smithi135 ceph-mon[33238]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-17T07:00:07.695 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:07 smithi135 ceph-mon[33238]: osd.2 v2:172.21.15.100:6809/714579846 boot 2024-09-17T07:00:07.695 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:07 smithi135 ceph-mon[33238]: osdmap e18: 3 total, 3 up, 3 in 2024-09-17T07:00:07.695 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:07 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T07:00:08.250 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi100:vg_nvme/lv_1 2024-09-17T07:00:08.888 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:00:09.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:09 smithi135 ceph-mon[33238]: osdmap e19: 3 total, 3 up, 3 in 2024-09-17T07:00:09.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:09 smithi135 ceph-mon[33238]: pgmap v74: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-17T07:00:09.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:09 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-09-17T07:00:09.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:09 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:09.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:09 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:09.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:09 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:00:09.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:09 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:00:09.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:09 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:00:09.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:09 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:09.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:09 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:00:09.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:09 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:00:09.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:09 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:09.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:09 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:00:09.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:09 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:00:09.545 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:09 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:00:09.545 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:09 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:09.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:09 smithi100 ceph-mon[29092]: osdmap e19: 3 total, 3 up, 3 in 2024-09-17T07:00:09.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:09 smithi100 ceph-mon[29092]: pgmap v74: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-17T07:00:09.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:09 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-09-17T07:00:09.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:09 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:09.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:09 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:09.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:09 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:00:09.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:09 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:00:09.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:09 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:00:09.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:09 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:09.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:09 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:00:09.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:09 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:00:09.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:09 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:09.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:09 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:00:09.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:09 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:00:09.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:09 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:00:09.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:09 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:09.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:09 smithi100 ceph-mon[39462]: osdmap e19: 3 total, 3 up, 3 in 2024-09-17T07:00:09.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:09 smithi100 ceph-mon[39462]: pgmap v74: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-17T07:00:09.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:09 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-09-17T07:00:09.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:09 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:09.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:09 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:09.879 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:09 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:00:09.879 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:09 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:00:09.879 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:09 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:00:09.879 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:09 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:09.879 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:09 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:00:09.879 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:09 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:00:09.879 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:09 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:09.879 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:09 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:00:09.879 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:09 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:00:09.879 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:09 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:00:09.879 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:09 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:10.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:10 smithi100 ceph-mon[39462]: Detected new or changed devices on smithi100 2024-09-17T07:00:10.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:10 smithi100 ceph-mon[39462]: Adjusting osd_memory_target on smithi100 to 5354M 2024-09-17T07:00:10.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:10 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-17T07:00:10.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:10 smithi100 ceph-mon[39462]: osdmap e20: 3 total, 3 up, 3 in 2024-09-17T07:00:10.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:10 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-17T07:00:10.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:10 smithi100 ceph-mon[29092]: Detected new or changed devices on smithi100 2024-09-17T07:00:10.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:10 smithi100 ceph-mon[29092]: Adjusting osd_memory_target on smithi100 to 5354M 2024-09-17T07:00:10.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:10 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-17T07:00:10.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:10 smithi100 ceph-mon[29092]: osdmap e20: 3 total, 3 up, 3 in 2024-09-17T07:00:10.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:10 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-17T07:00:10.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:10 smithi135 ceph-mon[33238]: Detected new or changed devices on smithi100 2024-09-17T07:00:10.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:10 smithi135 ceph-mon[33238]: Adjusting osd_memory_target on smithi100 to 5354M 2024-09-17T07:00:10.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:10 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-17T07:00:10.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:10 smithi135 ceph-mon[33238]: osdmap e20: 3 total, 3 up, 3 in 2024-09-17T07:00:10.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:10 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-17T07:00:11.258 INFO:journalctl@ceph.osd.0.smithi100.stdout:Sep 17 07:00:11 smithi100 sudo[58660]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-17T07:00:11.259 INFO:journalctl@ceph.osd.0.smithi100.stdout:Sep 17 07:00:11 smithi100 sudo[58660]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-17T07:00:11.259 INFO:journalctl@ceph.osd.0.smithi100.stdout:Sep 17 07:00:11 smithi100 sudo[58660]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-17T07:00:11.259 INFO:journalctl@ceph.osd.0.smithi100.stdout:Sep 17 07:00:11 smithi100 sudo[58660]: pam_unix(sudo:session): session closed for user root 2024-09-17T07:00:11.550 INFO:journalctl@ceph.osd.0.smithi100.stdout:Sep 17 07:00:11 smithi100 sudo[58682]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-17T07:00:11.550 INFO:journalctl@ceph.osd.0.smithi100.stdout:Sep 17 07:00:11 smithi100 sudo[58682]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-17T07:00:11.550 INFO:journalctl@ceph.osd.0.smithi100.stdout:Sep 17 07:00:11 smithi100 sudo[58682]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-17T07:00:11.550 INFO:journalctl@ceph.osd.0.smithi100.stdout:Sep 17 07:00:11 smithi100 sudo[58682]: pam_unix(sudo:session): session closed for user root 2024-09-17T07:00:11.551 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:11 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-17T07:00:11.551 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:11 smithi100 ceph-mon[29092]: osdmap e21: 3 total, 3 up, 3 in 2024-09-17T07:00:11.551 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:11 smithi100 ceph-mon[29092]: pgmap v77: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-17T07:00:11.551 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:11 smithi100 ceph-mon[29092]: from='client.24200 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi100:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:00:11.551 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:11 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T07:00:11.551 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:11 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T07:00:11.551 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:11 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:00:11.551 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:11 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:11.551 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:11 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-17T07:00:11.551 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:11 smithi100 ceph-mon[39462]: osdmap e21: 3 total, 3 up, 3 in 2024-09-17T07:00:11.551 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:11 smithi100 ceph-mon[39462]: pgmap v77: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-17T07:00:11.552 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:11 smithi100 ceph-mon[39462]: from='client.24200 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi100:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:00:11.552 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:11 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T07:00:11.552 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:11 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T07:00:11.552 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:11 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:00:11.552 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:11 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:11.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:11 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-17T07:00:11.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:11 smithi135 ceph-mon[33238]: osdmap e21: 3 total, 3 up, 3 in 2024-09-17T07:00:11.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:11 smithi135 ceph-mon[33238]: pgmap v77: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-17T07:00:11.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:11 smithi135 ceph-mon[33238]: from='client.24200 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi100:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:00:11.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:11 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T07:00:11.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:11 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T07:00:11.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:11 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:00:11.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:11 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:11.866 INFO:journalctl@ceph.osd.1.smithi100.stdout:Sep 17 07:00:11 smithi100 sudo[58693]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-17T07:00:11.866 INFO:journalctl@ceph.osd.1.smithi100.stdout:Sep 17 07:00:11 smithi100 sudo[58693]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-17T07:00:11.866 INFO:journalctl@ceph.osd.1.smithi100.stdout:Sep 17 07:00:11 smithi100 sudo[58693]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-17T07:00:11.866 INFO:journalctl@ceph.osd.1.smithi100.stdout:Sep 17 07:00:11 smithi100 sudo[58693]: pam_unix(sudo:session): session closed for user root 2024-09-17T07:00:12.122 INFO:journalctl@ceph.osd.1.smithi100.stdout:Sep 17 07:00:11 smithi100 sudo[58701]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-17T07:00:12.122 INFO:journalctl@ceph.osd.1.smithi100.stdout:Sep 17 07:00:11 smithi100 sudo[58701]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-17T07:00:12.122 INFO:journalctl@ceph.osd.1.smithi100.stdout:Sep 17 07:00:11 smithi100 sudo[58701]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-17T07:00:12.122 INFO:journalctl@ceph.osd.1.smithi100.stdout:Sep 17 07:00:11 smithi100 sudo[58701]: pam_unix(sudo:session): session closed for user root 2024-09-17T07:00:12.377 INFO:journalctl@ceph.osd.2.smithi100.stdout:Sep 17 07:00:12 smithi100 sudo[58723]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-17T07:00:12.377 INFO:journalctl@ceph.osd.2.smithi100.stdout:Sep 17 07:00:12 smithi100 sudo[58723]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-17T07:00:12.377 INFO:journalctl@ceph.osd.2.smithi100.stdout:Sep 17 07:00:12 smithi100 sudo[58723]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-17T07:00:12.377 INFO:journalctl@ceph.osd.2.smithi100.stdout:Sep 17 07:00:12 smithi100 sudo[58723]: pam_unix(sudo:session): session closed for user root 2024-09-17T07:00:12.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Sep 17 07:00:12 smithi100 sudo[58734]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-17T07:00:12.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Sep 17 07:00:12 smithi100 sudo[58734]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-17T07:00:12.645 INFO:journalctl@ceph.osd.2.smithi100.stdout:Sep 17 07:00:12 smithi100 sudo[58734]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-17T07:00:12.645 INFO:journalctl@ceph.osd.2.smithi100.stdout:Sep 17 07:00:12 smithi100 sudo[58734]: pam_unix(sudo:session): session closed for user root 2024-09-17T07:00:12.910 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:12 smithi100 sudo[58749]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-17T07:00:12.910 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:12 smithi100 sudo[58749]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-17T07:00:12.910 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:12 smithi100 sudo[58749]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-17T07:00:12.910 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:12 smithi100 sudo[58749]: pam_unix(sudo:session): session closed for user root 2024-09-17T07:00:13.174 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:12 smithi135 ceph-mon[33238]: osdmap e22: 3 total, 3 up, 3 in 2024-09-17T07:00:13.174 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:13 smithi135 sudo[35753]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-17T07:00:13.175 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:13 smithi135 sudo[35753]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-17T07:00:13.175 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:13 smithi135 sudo[35753]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-17T07:00:13.175 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:13 smithi135 sudo[35753]: pam_unix(sudo:session): session closed for user root 2024-09-17T07:00:13.376 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:12 smithi100 ceph-mon[39462]: osdmap e22: 3 total, 3 up, 3 in 2024-09-17T07:00:13.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:12 smithi100 sudo[58781]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-17T07:00:13.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:12 smithi100 sudo[58781]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-17T07:00:13.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:12 smithi100 sudo[58781]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-17T07:00:13.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:12 smithi100 sudo[58781]: pam_unix(sudo:session): session closed for user root 2024-09-17T07:00:13.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:12 smithi100 ceph-mon[29092]: osdmap e22: 3 total, 3 up, 3 in 2024-09-17T07:00:13.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:13 smithi135 sudo[35756]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-17T07:00:13.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:13 smithi135 sudo[35756]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-17T07:00:13.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:13 smithi135 sudo[35756]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-17T07:00:13.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:13 smithi135 sudo[35756]: pam_unix(sudo:session): session closed for user root 2024-09-17T07:00:13.946 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:13 smithi100 sudo[58791]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-17T07:00:13.946 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:13 smithi100 sudo[58791]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-17T07:00:13.946 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:13 smithi100 sudo[58791]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-17T07:00:13.947 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:13 smithi100 sudo[58791]: pam_unix(sudo:session): session closed for user root 2024-09-17T07:00:14.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:13 smithi135 ceph-mon[33238]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-17T07:00:14.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:13 smithi135 ceph-mon[33238]: pgmap v79: 1 pgs: 1 unknown; 0 B data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-17T07:00:14.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:13 smithi135 ceph-mon[33238]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-17T07:00:14.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:13 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T07:00:14.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:13 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T07:00:14.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:13 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T07:00:14.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:13 smithi135 ceph-mon[33238]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-17T07:00:14.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:13 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T07:00:14.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:13 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T07:00:14.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:13 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T07:00:14.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:13 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2791832597' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a1257575-3c90-41de-88f6-3960744351b1"}]: dispatch 2024-09-17T07:00:14.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:13 smithi135 ceph-mon[33238]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a1257575-3c90-41de-88f6-3960744351b1"}]: dispatch 2024-09-17T07:00:14.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:13 smithi135 ceph-mon[33238]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-17T07:00:14.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:13 smithi135 ceph-mon[33238]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a1257575-3c90-41de-88f6-3960744351b1"}]': finished 2024-09-17T07:00:14.295 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:13 smithi135 ceph-mon[33238]: osdmap e23: 4 total, 3 up, 4 in 2024-09-17T07:00:14.295 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:13 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T07:00:14.295 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:13 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:14.295 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:13 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T07:00:14.295 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:13 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T07:00:14.295 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:13 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T07:00:14.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:13 smithi100 ceph-mon[29092]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-17T07:00:14.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:13 smithi100 ceph-mon[29092]: pgmap v79: 1 pgs: 1 unknown; 0 B data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-17T07:00:14.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:13 smithi100 ceph-mon[29092]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-17T07:00:14.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:13 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T07:00:14.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:13 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T07:00:14.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:13 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T07:00:14.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:13 smithi100 ceph-mon[29092]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-17T07:00:14.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:13 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T07:00:14.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:13 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T07:00:14.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:13 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T07:00:14.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:13 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2791832597' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a1257575-3c90-41de-88f6-3960744351b1"}]: dispatch 2024-09-17T07:00:14.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:13 smithi100 ceph-mon[29092]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a1257575-3c90-41de-88f6-3960744351b1"}]: dispatch 2024-09-17T07:00:14.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:13 smithi100 ceph-mon[29092]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-17T07:00:14.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:13 smithi100 ceph-mon[29092]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a1257575-3c90-41de-88f6-3960744351b1"}]': finished 2024-09-17T07:00:14.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:13 smithi100 ceph-mon[29092]: osdmap e23: 4 total, 3 up, 4 in 2024-09-17T07:00:14.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:13 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T07:00:14.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:13 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:14.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:13 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T07:00:14.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:13 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T07:00:14.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:13 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T07:00:14.379 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:13 smithi100 sudo[58821]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-17T07:00:14.379 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:13 smithi100 sudo[58821]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-17T07:00:14.379 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:13 smithi100 sudo[58821]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-17T07:00:14.379 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:13 smithi100 sudo[58821]: pam_unix(sudo:session): session closed for user root 2024-09-17T07:00:14.379 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:13 smithi100 ceph-mon[39462]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-17T07:00:14.379 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:13 smithi100 ceph-mon[39462]: pgmap v79: 1 pgs: 1 unknown; 0 B data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-17T07:00:14.379 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:13 smithi100 ceph-mon[39462]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-17T07:00:14.379 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:13 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T07:00:14.379 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:13 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T07:00:14.379 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:13 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T07:00:14.379 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:13 smithi100 ceph-mon[39462]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-17T07:00:14.380 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:13 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T07:00:14.380 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:13 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T07:00:14.380 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:13 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T07:00:14.380 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:13 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2791832597' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a1257575-3c90-41de-88f6-3960744351b1"}]: dispatch 2024-09-17T07:00:14.380 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:13 smithi100 ceph-mon[39462]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a1257575-3c90-41de-88f6-3960744351b1"}]: dispatch 2024-09-17T07:00:14.380 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:13 smithi100 ceph-mon[39462]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-17T07:00:14.380 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:13 smithi100 ceph-mon[39462]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a1257575-3c90-41de-88f6-3960744351b1"}]': finished 2024-09-17T07:00:14.380 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:13 smithi100 ceph-mon[39462]: osdmap e23: 4 total, 3 up, 4 in 2024-09-17T07:00:14.380 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:13 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T07:00:14.380 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:13 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:14.380 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:13 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T07:00:14.380 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:13 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T07:00:14.380 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:13 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T07:00:15.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:14 smithi135 ceph-mon[33238]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-17T07:00:15.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:14 smithi135 ceph-mon[33238]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-17T07:00:15.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:14 smithi135 ceph-mon[33238]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-17T07:00:15.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:14 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/290292026' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T07:00:15.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:14 smithi135 ceph-mon[33238]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-17T07:00:15.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:14 smithi100 ceph-mon[29092]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-17T07:00:15.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:14 smithi100 ceph-mon[29092]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-17T07:00:15.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:14 smithi100 ceph-mon[29092]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-17T07:00:15.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:14 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/290292026' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T07:00:15.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:14 smithi100 ceph-mon[29092]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-17T07:00:15.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:14 smithi100 ceph-mon[39462]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-17T07:00:15.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:14 smithi100 ceph-mon[39462]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-17T07:00:15.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:14 smithi100 ceph-mon[39462]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-17T07:00:15.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:14 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/290292026' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T07:00:15.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:14 smithi100 ceph-mon[39462]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-17T07:00:17.841 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:17 smithi100 ceph-mon[29092]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-17T07:00:17.841 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:17 smithi100 ceph-mon[39462]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-17T07:00:18.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:17 smithi135 ceph-mon[33238]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-17T07:00:19.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:19 smithi135 ceph-mon[33238]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-17T07:00:19.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:19 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-17T07:00:19.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:19 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:00:19.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:19 smithi100 ceph-mon[29092]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-17T07:00:19.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:19 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-17T07:00:19.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:19 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:00:19.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:19 smithi100 ceph-mon[39462]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-17T07:00:19.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:19 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-17T07:00:19.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:19 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:00:20.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:20 smithi100 ceph-mon[29092]: Deploying daemon osd.3 on smithi100 2024-09-17T07:00:20.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:20 smithi100 ceph-mon[39462]: Deploying daemon osd.3 on smithi100 2024-09-17T07:00:21.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:20 smithi135 ceph-mon[33238]: Deploying daemon osd.3 on smithi100 2024-09-17T07:00:21.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:21 smithi100 ceph-mon[29092]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-17T07:00:21.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:21 smithi100 ceph-mon[39462]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-17T07:00:22.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:21 smithi135 ceph-mon[33238]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-17T07:00:23.729 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:23 smithi100 ceph-mon[29092]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-17T07:00:23.730 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:23 smithi100 ceph-mon[39462]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-17T07:00:24.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:23 smithi135 ceph-mon[33238]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-17T07:00:24.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:24 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:00:24.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:24 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:24.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:24 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:24.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:24 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:00:24.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:24 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:24.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:24 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:25.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:24 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:00:25.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:24 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:25.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:24 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:25.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:25 smithi100 ceph-mon[29092]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-17T07:00:25.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:25 smithi100 ceph-mon[39462]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-17T07:00:26.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:25 smithi135 ceph-mon[33238]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-17T07:00:26.620 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:26 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:00:26.620 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:26 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:00:26.876 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:26 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:00:26.876 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:26 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:00:27.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:26 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:00:27.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:26 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:00:27.703 INFO:teuthology.orchestra.run.smithi100.stdout:Created osd(s) 3 on host 'smithi100' 2024-09-17T07:00:27.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:27 smithi100 ceph-mon[29092]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-17T07:00:27.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:27 smithi100 ceph-mon[29092]: osdmap e24: 4 total, 3 up, 4 in 2024-09-17T07:00:27.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:27 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T07:00:27.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:27 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:27.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:27 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:27.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:27 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:00:27.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:27 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:00:27.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:27 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:27.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:27 smithi100 ceph-mon[29092]: from='osd.3 v2:172.21.15.100:6813/2315956572' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-17T07:00:27.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:27 smithi100 ceph-mon[39462]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-17T07:00:27.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:27 smithi100 ceph-mon[39462]: osdmap e24: 4 total, 3 up, 4 in 2024-09-17T07:00:27.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:27 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T07:00:27.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:27 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:27.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:27 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:27.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:27 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:00:27.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:27 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:00:27.879 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:27 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:27.879 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:27 smithi100 ceph-mon[39462]: from='osd.3 v2:172.21.15.100:6813/2315956572' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-17T07:00:28.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:27 smithi135 ceph-mon[33238]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-17T07:00:28.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:27 smithi135 ceph-mon[33238]: osdmap e24: 4 total, 3 up, 4 in 2024-09-17T07:00:28.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:27 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T07:00:28.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:27 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:28.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:27 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:28.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:27 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:00:28.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:27 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:00:28.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:27 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:28.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:27 smithi135 ceph-mon[33238]: from='osd.3 v2:172.21.15.100:6813/2315956572' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-17T07:00:28.333 DEBUG:teuthology.orchestra.run.smithi100:osd.3> sudo journalctl -f -n 0 -u ceph-e848267c-74c1-11ef-bceb-c7b262605968@osd.3.service 2024-09-17T07:00:28.335 INFO:tasks.cephadm:Deploying osd.4 on smithi135 with /dev/vg_nvme/lv_4... 2024-09-17T07:00:28.336 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-09-17T07:00:28.542 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:00:28.866 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:28 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:00:28.867 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:28 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:28.867 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:28 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:28.867 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:28 smithi100 ceph-mon[29092]: from='osd.3 v2:172.21.15.100:6813/2315956572' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-17T07:00:28.867 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:28 smithi100 ceph-mon[29092]: osdmap e25: 4 total, 3 up, 4 in 2024-09-17T07:00:28.867 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:28 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T07:00:28.867 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:28 smithi100 ceph-mon[29092]: from='osd.3 v2:172.21.15.100:6813/2315956572' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]: dispatch 2024-09-17T07:00:28.867 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:28 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:00:28.867 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:28 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:28.867 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:28 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:28.867 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:28 smithi100 ceph-mon[39462]: from='osd.3 v2:172.21.15.100:6813/2315956572' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-17T07:00:28.868 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:28 smithi100 ceph-mon[39462]: osdmap e25: 4 total, 3 up, 4 in 2024-09-17T07:00:28.868 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:28 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T07:00:28.868 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:28 smithi100 ceph-mon[39462]: from='osd.3 v2:172.21.15.100:6813/2315956572' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]: dispatch 2024-09-17T07:00:28.961 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:28 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:00:28.962 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:28 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:28.962 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:28 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:28.962 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:28 smithi135 ceph-mon[33238]: from='osd.3 v2:172.21.15.100:6813/2315956572' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-17T07:00:28.962 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:28 smithi135 ceph-mon[33238]: osdmap e25: 4 total, 3 up, 4 in 2024-09-17T07:00:28.962 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:28 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T07:00:28.962 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:28 smithi135 ceph-mon[33238]: from='osd.3 v2:172.21.15.100:6813/2315956572' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]: dispatch 2024-09-17T07:00:29.848 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:29 smithi100 ceph-mon[29092]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-17T07:00:29.848 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:29 smithi100 ceph-mon[29092]: from='osd.3 v2:172.21.15.100:6813/2315956572' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]': finished 2024-09-17T07:00:29.848 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:29 smithi100 ceph-mon[29092]: osdmap e26: 4 total, 3 up, 4 in 2024-09-17T07:00:29.848 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:29 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T07:00:29.849 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:29 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T07:00:29.849 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:29 smithi100 ceph-mon[39462]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-17T07:00:29.849 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:29 smithi100 ceph-mon[39462]: from='osd.3 v2:172.21.15.100:6813/2315956572' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]': finished 2024-09-17T07:00:29.849 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:29 smithi100 ceph-mon[39462]: osdmap e26: 4 total, 3 up, 4 in 2024-09-17T07:00:29.849 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:29 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T07:00:29.849 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:29 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T07:00:29.849 INFO:journalctl@ceph.osd.3.smithi100.stdout:Sep 17 07:00:29 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-3[60454]: 2024-09-17T07:00:29.804+0000 7f14a9d53640 -1 osd.3 0 waiting for initial osdmap 2024-09-17T07:00:29.850 INFO:journalctl@ceph.osd.3.smithi100.stdout:Sep 17 07:00:29 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-3[60454]: 2024-09-17T07:00:29.810+0000 7f14a1b74640 -1 osd.3 26 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T07:00:29.942 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:29 smithi135 ceph-mon[33238]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-17T07:00:29.942 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:29 smithi135 ceph-mon[33238]: from='osd.3 v2:172.21.15.100:6813/2315956572' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]': finished 2024-09-17T07:00:29.942 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:29 smithi135 ceph-mon[33238]: osdmap e26: 4 total, 3 up, 4 in 2024-09-17T07:00:29.942 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:29 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T07:00:29.942 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:29 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T07:00:30.125 INFO:teuthology.orchestra.run.smithi135.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-09-17T07:00:30.208 INFO:teuthology.orchestra.run.smithi135.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-09-17T07:00:30.221 INFO:teuthology.orchestra.run.smithi135.stderr: stderr: 10+0 records in 2024-09-17T07:00:30.221 INFO:teuthology.orchestra.run.smithi135.stderr:10+0 records out 2024-09-17T07:00:30.221 INFO:teuthology.orchestra.run.smithi135.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114189 s, 918 MB/s 2024-09-17T07:00:30.221 INFO:teuthology.orchestra.run.smithi135.stderr:--> Zapping successful for: 2024-09-17T07:00:30.835 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi135:vg_nvme/lv_4 2024-09-17T07:00:30.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:30 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:00:30.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:30 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:00:30.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:30 smithi100 ceph-mon[29092]: from='osd.3 v2:172.21.15.100:6813/2315956572' entity='osd.3' 2024-09-17T07:00:30.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:30 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T07:00:30.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:30 smithi100 ceph-mon[29092]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-17T07:00:30.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:30 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:00:30.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:30 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:00:30.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:30 smithi100 ceph-mon[39462]: from='osd.3 v2:172.21.15.100:6813/2315956572' entity='osd.3' 2024-09-17T07:00:30.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:30 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T07:00:30.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:30 smithi100 ceph-mon[39462]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-17T07:00:31.032 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:00:31.130 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:30 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:00:31.130 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:30 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:00:31.130 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:30 smithi135 ceph-mon[33238]: from='osd.3 v2:172.21.15.100:6813/2315956572' entity='osd.3' 2024-09-17T07:00:31.130 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:30 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T07:00:31.130 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:30 smithi135 ceph-mon[33238]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-17T07:00:32.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:31 smithi135 ceph-mon[33238]: osd.3 v2:172.21.15.100:6813/2315956572 boot 2024-09-17T07:00:32.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:31 smithi135 ceph-mon[33238]: osdmap e27: 4 total, 4 up, 4 in 2024-09-17T07:00:32.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:31 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T07:00:32.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:31 smithi135 ceph-mon[33238]: Detected new or changed devices on smithi100 2024-09-17T07:00:32.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:31 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:32.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:31 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:32.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:31 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:00:32.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:31 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:00:32.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:31 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:00:32.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:31 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:00:32.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:31 smithi135 ceph-mon[33238]: Adjusting osd_memory_target on smithi100 to 4015M 2024-09-17T07:00:32.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:31 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:32.045 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:31 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:00:32.045 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:31 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:00:32.045 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:31 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:32.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:31 smithi100 ceph-mon[29092]: osd.3 v2:172.21.15.100:6813/2315956572 boot 2024-09-17T07:00:32.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:31 smithi100 ceph-mon[29092]: osdmap e27: 4 total, 4 up, 4 in 2024-09-17T07:00:32.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:31 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T07:00:32.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:31 smithi100 ceph-mon[29092]: Detected new or changed devices on smithi100 2024-09-17T07:00:32.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:31 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:32.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:31 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:32.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:31 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:00:32.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:31 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:00:32.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:31 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:00:32.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:31 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:00:32.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:31 smithi100 ceph-mon[29092]: Adjusting osd_memory_target on smithi100 to 4015M 2024-09-17T07:00:32.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:31 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:32.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:31 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:00:32.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:31 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:00:32.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:31 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:32.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:31 smithi100 ceph-mon[39462]: osd.3 v2:172.21.15.100:6813/2315956572 boot 2024-09-17T07:00:32.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:31 smithi100 ceph-mon[39462]: osdmap e27: 4 total, 4 up, 4 in 2024-09-17T07:00:32.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:31 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T07:00:32.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:31 smithi100 ceph-mon[39462]: Detected new or changed devices on smithi100 2024-09-17T07:00:32.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:31 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:32.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:31 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:32.129 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:31 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:00:32.129 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:31 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:00:32.129 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:31 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:00:32.129 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:31 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:00:32.129 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:31 smithi100 ceph-mon[39462]: Adjusting osd_memory_target on smithi100 to 4015M 2024-09-17T07:00:32.129 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:31 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:32.129 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:31 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:00:32.129 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:31 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:00:32.129 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:31 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:33.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:33 smithi135 ceph-mon[33238]: osdmap e28: 4 total, 4 up, 4 in 2024-09-17T07:00:33.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:33 smithi135 ceph-mon[33238]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-17T07:00:33.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:33 smithi135 ceph-mon[33238]: from='client.24187 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi135:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:00:33.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:33 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T07:00:33.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:33 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T07:00:33.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:33 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:00:33.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:33 smithi100 ceph-mon[29092]: osdmap e28: 4 total, 4 up, 4 in 2024-09-17T07:00:33.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:33 smithi100 ceph-mon[29092]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-17T07:00:33.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:33 smithi100 ceph-mon[29092]: from='client.24187 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi135:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:00:33.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:33 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T07:00:33.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:33 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T07:00:33.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:33 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:00:33.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:33 smithi100 ceph-mon[39462]: osdmap e28: 4 total, 4 up, 4 in 2024-09-17T07:00:33.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:33 smithi100 ceph-mon[39462]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-17T07:00:33.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:33 smithi100 ceph-mon[39462]: from='client.24187 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi135:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:00:33.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:33 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T07:00:33.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:33 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T07:00:33.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:33 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:00:35.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:35 smithi100 ceph-mon[29092]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-17T07:00:35.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:35 smithi100 ceph-mon[29092]: from='client.? 172.21.15.135:0/3784435878' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e44e8922-2c5e-44e3-b556-d4dfbc499649"}]: dispatch 2024-09-17T07:00:35.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:35 smithi100 ceph-mon[29092]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e44e8922-2c5e-44e3-b556-d4dfbc499649"}]: dispatch 2024-09-17T07:00:35.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:35 smithi100 ceph-mon[29092]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e44e8922-2c5e-44e3-b556-d4dfbc499649"}]': finished 2024-09-17T07:00:35.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:35 smithi100 ceph-mon[29092]: osdmap e29: 5 total, 4 up, 5 in 2024-09-17T07:00:35.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:35 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T07:00:35.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:35 smithi100 ceph-mon[29092]: from='client.? 172.21.15.135:0/3488156822' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T07:00:35.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:35 smithi100 ceph-mon[39462]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-17T07:00:35.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:35 smithi100 ceph-mon[39462]: from='client.? 172.21.15.135:0/3784435878' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e44e8922-2c5e-44e3-b556-d4dfbc499649"}]: dispatch 2024-09-17T07:00:35.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:35 smithi100 ceph-mon[39462]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e44e8922-2c5e-44e3-b556-d4dfbc499649"}]: dispatch 2024-09-17T07:00:35.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:35 smithi100 ceph-mon[39462]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e44e8922-2c5e-44e3-b556-d4dfbc499649"}]': finished 2024-09-17T07:00:35.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:35 smithi100 ceph-mon[39462]: osdmap e29: 5 total, 4 up, 5 in 2024-09-17T07:00:35.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:35 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T07:00:35.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:35 smithi100 ceph-mon[39462]: from='client.? 172.21.15.135:0/3488156822' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T07:00:36.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:35 smithi135 ceph-mon[33238]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-17T07:00:36.045 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:35 smithi135 ceph-mon[33238]: from='client.? 172.21.15.135:0/3784435878' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e44e8922-2c5e-44e3-b556-d4dfbc499649"}]: dispatch 2024-09-17T07:00:36.045 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:35 smithi135 ceph-mon[33238]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e44e8922-2c5e-44e3-b556-d4dfbc499649"}]: dispatch 2024-09-17T07:00:36.045 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:35 smithi135 ceph-mon[33238]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e44e8922-2c5e-44e3-b556-d4dfbc499649"}]': finished 2024-09-17T07:00:36.045 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:35 smithi135 ceph-mon[33238]: osdmap e29: 5 total, 4 up, 5 in 2024-09-17T07:00:36.045 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:35 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T07:00:36.045 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:35 smithi135 ceph-mon[33238]: from='client.? 172.21.15.135:0/3488156822' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T07:00:37.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:37 smithi135 ceph-mon[33238]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-17T07:00:37.876 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:37 smithi100 ceph-mon[39462]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-17T07:00:37.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:37 smithi100 ceph-mon[29092]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-17T07:00:39.727 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:39 smithi135 ceph-mon[33238]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-17T07:00:39.876 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:39 smithi100 ceph-mon[39462]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-17T07:00:39.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:39 smithi100 ceph-mon[29092]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-17T07:00:41.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:40 smithi135 ceph-mon[33238]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-17T07:00:41.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:40 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-17T07:00:41.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:40 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:00:41.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:40 smithi135 ceph-mon[33238]: Deploying daemon osd.4 on smithi135 2024-09-17T07:00:41.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:40 smithi100 ceph-mon[29092]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-17T07:00:41.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:40 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-17T07:00:41.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:40 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:00:41.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:40 smithi100 ceph-mon[29092]: Deploying daemon osd.4 on smithi135 2024-09-17T07:00:41.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:40 smithi100 ceph-mon[39462]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-17T07:00:41.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:40 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-17T07:00:41.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:40 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:00:41.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:40 smithi100 ceph-mon[39462]: Deploying daemon osd.4 on smithi135 2024-09-17T07:00:43.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:43 smithi135 ceph-mon[33238]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-17T07:00:43.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:43 smithi100 ceph-mon[29092]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-17T07:00:43.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:43 smithi100 ceph-mon[39462]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-17T07:00:45.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:45 smithi135 ceph-mon[33238]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-17T07:00:45.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:45 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:00:45.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:45 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:45.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:45 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:45.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:45 smithi100 ceph-mon[29092]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-17T07:00:45.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:45 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:00:45.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:45 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:45.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:45 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:45.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:45 smithi100 ceph-mon[39462]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-17T07:00:45.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:45 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:00:45.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:45 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:45.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:45 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:47.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:47 smithi100 ceph-mon[29092]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-17T07:00:47.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:47 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:47.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:47 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:47.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:47 smithi100 ceph-mon[39462]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-17T07:00:47.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:47 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:47.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:47 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:48.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:47 smithi135 ceph-mon[33238]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-17T07:00:48.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:47 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:48.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:47 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:48.674 INFO:teuthology.orchestra.run.smithi135.stdout:Created osd(s) 4 on host 'smithi135' 2024-09-17T07:00:48.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:48 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:00:48.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:48 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:00:48.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:48 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:48.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:48 smithi100 ceph-mon[29092]: from='osd.4 v2:172.21.15.135:6800/1148820922' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-17T07:00:48.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:48 smithi100 ceph-mon[29092]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-17T07:00:48.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:48 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:00:48.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:48 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:00:48.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:48 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:48.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:48 smithi100 ceph-mon[39462]: from='osd.4 v2:172.21.15.135:6800/1148820922' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-17T07:00:48.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:48 smithi100 ceph-mon[39462]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-17T07:00:48.969 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:48 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:00:48.970 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:48 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:00:48.970 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:48 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:48.970 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:48 smithi135 ceph-mon[33238]: from='osd.4 v2:172.21.15.135:6800/1148820922' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-17T07:00:48.970 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:48 smithi135 ceph-mon[33238]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-17T07:00:49.267 DEBUG:teuthology.orchestra.run.smithi135:osd.4> sudo journalctl -f -n 0 -u ceph-e848267c-74c1-11ef-bceb-c7b262605968@osd.4.service 2024-09-17T07:00:49.269 INFO:tasks.cephadm:Deploying osd.5 on smithi135 with /dev/vg_nvme/lv_3... 2024-09-17T07:00:49.270 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-09-17T07:00:49.807 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:49 smithi135 ceph-mon[33238]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-17T07:00:49.807 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:49 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:00:49.807 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:49 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:49.807 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:49 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:49.807 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:49 smithi135 ceph-mon[33238]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-17T07:00:49.807 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:49 smithi135 ceph-mon[33238]: osdmap e30: 5 total, 4 up, 5 in 2024-09-17T07:00:49.807 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:49 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T07:00:49.808 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:49 smithi135 ceph-mon[33238]: from='osd.4 v2:172.21.15.135:6800/1148820922' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-09-17T07:00:49.808 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:49 smithi135 ceph-mon[33238]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-09-17T07:00:49.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:49 smithi100 ceph-mon[29092]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-17T07:00:49.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:49 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:00:49.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:49 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:49.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:49 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:49.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:49 smithi100 ceph-mon[29092]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-17T07:00:49.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:49 smithi100 ceph-mon[29092]: osdmap e30: 5 total, 4 up, 5 in 2024-09-17T07:00:49.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:49 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T07:00:49.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:49 smithi100 ceph-mon[29092]: from='osd.4 v2:172.21.15.135:6800/1148820922' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-09-17T07:00:49.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:49 smithi100 ceph-mon[29092]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-09-17T07:00:49.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:49 smithi100 ceph-mon[39462]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-17T07:00:49.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:49 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:00:49.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:49 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:49.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:49 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:49.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:49 smithi100 ceph-mon[39462]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-17T07:00:49.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:49 smithi100 ceph-mon[39462]: osdmap e30: 5 total, 4 up, 5 in 2024-09-17T07:00:49.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:49 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T07:00:49.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:49 smithi100 ceph-mon[39462]: from='osd.4 v2:172.21.15.135:6800/1148820922' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-09-17T07:00:49.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:49 smithi100 ceph-mon[39462]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-09-17T07:00:50.346 INFO:journalctl@ceph.osd.4.smithi135.stdout:Sep 17 07:00:50 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-4[38050]: 2024-09-17T07:00:50.165+0000 7fe674810640 -1 osd.4 0 waiting for initial osdmap 2024-09-17T07:00:50.346 INFO:journalctl@ceph.osd.4.smithi135.stdout:Sep 17 07:00:50 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-4[38050]: 2024-09-17T07:00:50.169+0000 7fe66c631640 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T07:00:50.837 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:00:50.958 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:50 smithi135 ceph-mon[33238]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]': finished 2024-09-17T07:00:50.958 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:50 smithi135 ceph-mon[33238]: osdmap e31: 5 total, 4 up, 5 in 2024-09-17T07:00:50.959 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:50 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T07:00:50.959 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:50 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T07:00:50.959 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:50 smithi135 ceph-mon[33238]: osdmap e32: 5 total, 4 up, 5 in 2024-09-17T07:00:50.959 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:50 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T07:00:50.959 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:50 smithi135 ceph-mon[33238]: from='osd.4 ' entity='osd.4' 2024-09-17T07:00:51.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:50 smithi100 ceph-mon[29092]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]': finished 2024-09-17T07:00:51.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:50 smithi100 ceph-mon[29092]: osdmap e31: 5 total, 4 up, 5 in 2024-09-17T07:00:51.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:50 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T07:00:51.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:50 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T07:00:51.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:50 smithi100 ceph-mon[29092]: osdmap e32: 5 total, 4 up, 5 in 2024-09-17T07:00:51.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:50 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T07:00:51.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:50 smithi100 ceph-mon[29092]: from='osd.4 ' entity='osd.4' 2024-09-17T07:00:51.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:50 smithi100 ceph-mon[39462]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]': finished 2024-09-17T07:00:51.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:50 smithi100 ceph-mon[39462]: osdmap e31: 5 total, 4 up, 5 in 2024-09-17T07:00:51.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:50 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T07:00:51.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:50 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T07:00:51.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:50 smithi100 ceph-mon[39462]: osdmap e32: 5 total, 4 up, 5 in 2024-09-17T07:00:51.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:50 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T07:00:51.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:50 smithi100 ceph-mon[39462]: from='osd.4 ' entity='osd.4' 2024-09-17T07:00:52.019 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:51 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:00:52.019 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:51 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:00:52.019 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:51 smithi135 ceph-mon[33238]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-17T07:00:52.019 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:51 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T07:00:52.019 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:51 smithi135 ceph-mon[33238]: osd.4 v2:172.21.15.135:6800/1148820922 boot 2024-09-17T07:00:52.019 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:51 smithi135 ceph-mon[33238]: osdmap e33: 5 total, 5 up, 5 in 2024-09-17T07:00:52.019 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:51 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T07:00:52.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:51 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:00:52.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:51 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:00:52.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:51 smithi100 ceph-mon[29092]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-17T07:00:52.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:51 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T07:00:52.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:51 smithi100 ceph-mon[29092]: osd.4 v2:172.21.15.135:6800/1148820922 boot 2024-09-17T07:00:52.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:51 smithi100 ceph-mon[29092]: osdmap e33: 5 total, 5 up, 5 in 2024-09-17T07:00:52.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:51 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T07:00:52.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:51 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:00:52.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:51 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:00:52.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:51 smithi100 ceph-mon[39462]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-17T07:00:52.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:51 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T07:00:52.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:51 smithi100 ceph-mon[39462]: osd.4 v2:172.21.15.135:6800/1148820922 boot 2024-09-17T07:00:52.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:51 smithi100 ceph-mon[39462]: osdmap e33: 5 total, 5 up, 5 in 2024-09-17T07:00:52.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:51 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T07:00:53.060 INFO:teuthology.orchestra.run.smithi135.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-09-17T07:00:53.143 INFO:teuthology.orchestra.run.smithi135.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-09-17T07:00:53.159 INFO:teuthology.orchestra.run.smithi135.stderr: stderr: 10+0 records in 2024-09-17T07:00:53.159 INFO:teuthology.orchestra.run.smithi135.stderr:10+0 records out 2024-09-17T07:00:53.159 INFO:teuthology.orchestra.run.smithi135.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.013914 s, 754 MB/s 2024-09-17T07:00:53.159 INFO:teuthology.orchestra.run.smithi135.stderr:--> Zapping successful for: 2024-09-17T07:00:53.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:53 smithi135 ceph-mon[33238]: osdmap e34: 5 total, 5 up, 5 in 2024-09-17T07:00:53.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:53 smithi135 ceph-mon[33238]: Detected new or changed devices on smithi135 2024-09-17T07:00:53.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:53 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:53.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:53 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:53.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:53 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:00:53.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:53 smithi135 ceph-mon[33238]: Adjusting osd_memory_target on smithi135 to 5815M 2024-09-17T07:00:53.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:53 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:53.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:53 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:00:53.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:53 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:00:53.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:53 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:53.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:53 smithi135 ceph-mon[33238]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-17T07:00:53.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:53 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:00:53.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:53 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:00:53.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:53 smithi100 ceph-mon[29092]: osdmap e34: 5 total, 5 up, 5 in 2024-09-17T07:00:53.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:53 smithi100 ceph-mon[29092]: Detected new or changed devices on smithi135 2024-09-17T07:00:53.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:53 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:53.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:53 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:53.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:53 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:00:53.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:53 smithi100 ceph-mon[29092]: Adjusting osd_memory_target on smithi135 to 5815M 2024-09-17T07:00:53.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:53 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:53.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:53 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:00:53.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:53 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:00:53.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:53 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:53.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:53 smithi100 ceph-mon[29092]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-17T07:00:53.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:53 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:00:53.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:53 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:00:53.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:53 smithi100 ceph-mon[39462]: osdmap e34: 5 total, 5 up, 5 in 2024-09-17T07:00:53.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:53 smithi100 ceph-mon[39462]: Detected new or changed devices on smithi135 2024-09-17T07:00:53.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:53 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:53.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:53 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:53.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:53 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:00:53.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:53 smithi100 ceph-mon[39462]: Adjusting osd_memory_target on smithi135 to 5815M 2024-09-17T07:00:53.379 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:53 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:53.379 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:53 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:00:53.379 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:53 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:00:53.379 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:53 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:00:53.379 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:53 smithi100 ceph-mon[39462]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-17T07:00:53.379 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:53 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:00:53.379 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:53 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:00:53.719 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi135:vg_nvme/lv_3 2024-09-17T07:00:53.911 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:00:54.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:54 smithi135 ceph-mon[33238]: osdmap e35: 5 total, 5 up, 5 in 2024-09-17T07:00:54.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:54 smithi100 ceph-mon[29092]: osdmap e35: 5 total, 5 up, 5 in 2024-09-17T07:00:54.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:54 smithi100 ceph-mon[39462]: osdmap e35: 5 total, 5 up, 5 in 2024-09-17T07:00:55.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:55 smithi135 ceph-mon[33238]: osdmap e36: 5 total, 5 up, 5 in 2024-09-17T07:00:55.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:55 smithi135 ceph-mon[33238]: pgmap v114: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-09-17T07:00:55.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:55 smithi100 ceph-mon[29092]: osdmap e36: 5 total, 5 up, 5 in 2024-09-17T07:00:55.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:55 smithi100 ceph-mon[29092]: pgmap v114: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-09-17T07:00:55.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:55 smithi100 ceph-mon[39462]: osdmap e36: 5 total, 5 up, 5 in 2024-09-17T07:00:55.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:55 smithi100 ceph-mon[39462]: pgmap v114: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-09-17T07:00:56.229 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:56 smithi135 ceph-mon[33238]: osdmap e37: 5 total, 5 up, 5 in 2024-09-17T07:00:56.230 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:56 smithi135 ceph-mon[33238]: from='client.24214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi135:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:00:56.230 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:56 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T07:00:56.230 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:56 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T07:00:56.230 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:56 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:00:56.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:56 smithi100 ceph-mon[29092]: osdmap e37: 5 total, 5 up, 5 in 2024-09-17T07:00:56.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:56 smithi100 ceph-mon[29092]: from='client.24214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi135:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:00:56.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:56 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T07:00:56.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:56 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T07:00:56.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:56 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:00:56.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:56 smithi100 ceph-mon[39462]: osdmap e37: 5 total, 5 up, 5 in 2024-09-17T07:00:56.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:56 smithi100 ceph-mon[39462]: from='client.24214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi135:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:00:56.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:56 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T07:00:56.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:56 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T07:00:56.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:56 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:00:57.136 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:57 smithi135 ceph-mon[33238]: pgmap v116: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 127 KiB/s, 0 objects/s recovering 2024-09-17T07:00:57.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:57 smithi100 ceph-mon[29092]: pgmap v116: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 127 KiB/s, 0 objects/s recovering 2024-09-17T07:00:57.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:57 smithi100 ceph-mon[39462]: pgmap v116: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 127 KiB/s, 0 objects/s recovering 2024-09-17T07:00:58.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:58 smithi135 ceph-mon[33238]: from='client.? 172.21.15.135:0/3628356194' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91e4a362-fcdb-41b9-b54e-716591ba6d79"}]: dispatch 2024-09-17T07:00:58.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:58 smithi135 ceph-mon[33238]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91e4a362-fcdb-41b9-b54e-716591ba6d79"}]: dispatch 2024-09-17T07:00:58.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:58 smithi135 ceph-mon[33238]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "91e4a362-fcdb-41b9-b54e-716591ba6d79"}]': finished 2024-09-17T07:00:58.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:58 smithi135 ceph-mon[33238]: osdmap e38: 6 total, 5 up, 6 in 2024-09-17T07:00:58.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:58 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T07:00:58.294 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:58 smithi100 ceph-mon[29092]: from='client.? 172.21.15.135:0/3628356194' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91e4a362-fcdb-41b9-b54e-716591ba6d79"}]: dispatch 2024-09-17T07:00:58.294 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:58 smithi100 ceph-mon[29092]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91e4a362-fcdb-41b9-b54e-716591ba6d79"}]: dispatch 2024-09-17T07:00:58.294 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:58 smithi100 ceph-mon[29092]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "91e4a362-fcdb-41b9-b54e-716591ba6d79"}]': finished 2024-09-17T07:00:58.294 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:58 smithi100 ceph-mon[29092]: osdmap e38: 6 total, 5 up, 6 in 2024-09-17T07:00:58.294 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:58 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T07:00:58.294 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:58 smithi100 ceph-mon[39462]: from='client.? 172.21.15.135:0/3628356194' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91e4a362-fcdb-41b9-b54e-716591ba6d79"}]: dispatch 2024-09-17T07:00:58.294 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:58 smithi100 ceph-mon[39462]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91e4a362-fcdb-41b9-b54e-716591ba6d79"}]: dispatch 2024-09-17T07:00:58.294 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:58 smithi100 ceph-mon[39462]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "91e4a362-fcdb-41b9-b54e-716591ba6d79"}]': finished 2024-09-17T07:00:58.294 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:58 smithi100 ceph-mon[39462]: osdmap e38: 6 total, 5 up, 6 in 2024-09-17T07:00:58.295 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:58 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T07:00:58.544 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:00:58 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T07:00:58.539+0000 7f55d7df5640 -1 calc_pg_upmaps abort due to max <= 0 2024-09-17T07:00:59.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:59 smithi100 ceph-mon[29092]: from='client.? 172.21.15.135:0/4180602677' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T07:00:59.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:00:59 smithi100 ceph-mon[29092]: pgmap v118: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 104 KiB/s, 0 objects/s recovering 2024-09-17T07:00:59.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:59 smithi100 ceph-mon[39462]: from='client.? 172.21.15.135:0/4180602677' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T07:00:59.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:00:59 smithi100 ceph-mon[39462]: pgmap v118: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 104 KiB/s, 0 objects/s recovering 2024-09-17T07:00:59.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:59 smithi135 ceph-mon[33238]: from='client.? 172.21.15.135:0/4180602677' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T07:00:59.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:00:59 smithi135 ceph-mon[33238]: pgmap v118: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 104 KiB/s, 0 objects/s recovering 2024-09-17T07:01:01.269 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:00 smithi135 ceph-mon[33238]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-17T07:01:01.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:00 smithi100 ceph-mon[29092]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-17T07:01:01.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:00 smithi100 ceph-mon[39462]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-17T07:01:03.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:03 smithi100 ceph-mon[29092]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-17T07:01:03.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:03 smithi100 ceph-mon[39462]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-17T07:01:04.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:03 smithi135 ceph-mon[33238]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-17T07:01:04.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:04 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-17T07:01:04.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:04 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:01:04.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:04 smithi135 ceph-mon[33238]: Deploying daemon osd.5 on smithi135 2024-09-17T07:01:04.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:04 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-17T07:01:04.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:04 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:01:04.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:04 smithi100 ceph-mon[29092]: Deploying daemon osd.5 on smithi135 2024-09-17T07:01:04.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:04 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-17T07:01:04.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:04 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:01:04.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:04 smithi100 ceph-mon[39462]: Deploying daemon osd.5 on smithi135 2024-09-17T07:01:05.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:05 smithi100 ceph-mon[29092]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-17T07:01:05.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:05 smithi100 ceph-mon[39462]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-17T07:01:06.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:05 smithi135 ceph-mon[33238]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-17T07:01:06.923 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:06 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:01:06.923 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:06 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:01:07.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:06 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:01:07.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:06 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:01:07.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:06 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:01:07.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:06 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:01:07.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:07 smithi135 ceph-mon[33238]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-17T07:01:07.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:07 smithi135 ceph-mon[33238]: osdmap e39: 6 total, 5 up, 6 in 2024-09-17T07:01:07.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:07 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T07:01:08.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:07 smithi100 ceph-mon[29092]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-17T07:01:08.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:07 smithi100 ceph-mon[29092]: osdmap e39: 6 total, 5 up, 6 in 2024-09-17T07:01:08.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:07 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T07:01:08.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:07 smithi100 ceph-mon[39462]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-17T07:01:08.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:07 smithi100 ceph-mon[39462]: osdmap e39: 6 total, 5 up, 6 in 2024-09-17T07:01:08.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:07 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T07:01:09.766 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:09 smithi135 ceph-mon[33238]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-17T07:01:09.766 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:09 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:01:09.767 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:09 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:09.767 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:09 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:09.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:09 smithi100 ceph-mon[29092]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-17T07:01:09.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:09 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:01:09.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:09 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:09.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:09 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:09.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:09 smithi100 ceph-mon[39462]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-17T07:01:09.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:09 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:01:09.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:09 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:09.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:09 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:11.437 INFO:teuthology.orchestra.run.smithi135.stdout:Created osd(s) 5 on host 'smithi135' 2024-09-17T07:01:11.766 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:11 smithi135 ceph-mon[33238]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-17T07:01:11.766 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:11 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:11.766 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:11 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:11.986 DEBUG:teuthology.orchestra.run.smithi135:osd.5> sudo journalctl -f -n 0 -u ceph-e848267c-74c1-11ef-bceb-c7b262605968@osd.5.service 2024-09-17T07:01:11.988 INFO:tasks.cephadm:Deploying osd.6 on smithi135 with /dev/vg_nvme/lv_2... 2024-09-17T07:01:11.988 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-09-17T07:01:12.126 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:11 smithi100 ceph-mon[29092]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-17T07:01:12.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:11 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:12.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:11 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:12.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:11 smithi100 ceph-mon[39462]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-17T07:01:12.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:11 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:12.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:11 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:12.361 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:01:12.770 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:12 smithi135 ceph-mon[33238]: from='osd.5 v2:172.21.15.135:6804/2256520893' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-17T07:01:12.771 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:12 smithi135 ceph-mon[33238]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-17T07:01:12.771 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:12 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:12.771 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:12 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:13.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:12 smithi100 ceph-mon[29092]: from='osd.5 v2:172.21.15.135:6804/2256520893' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-17T07:01:13.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:12 smithi100 ceph-mon[29092]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-17T07:01:13.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:12 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:13.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:12 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:13.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:12 smithi100 ceph-mon[39462]: from='osd.5 v2:172.21.15.135:6804/2256520893' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-17T07:01:13.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:12 smithi100 ceph-mon[39462]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-17T07:01:13.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:12 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:13.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:12 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:13.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:13 smithi135 ceph-mon[33238]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-17T07:01:13.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:13 smithi135 ceph-mon[33238]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-17T07:01:13.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:13 smithi135 ceph-mon[33238]: osdmap e40: 6 total, 5 up, 6 in 2024-09-17T07:01:13.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:13 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T07:01:13.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:13 smithi135 ceph-mon[33238]: from='osd.5 v2:172.21.15.135:6804/2256520893' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-09-17T07:01:13.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:13 smithi135 ceph-mon[33238]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-09-17T07:01:14.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:13 smithi100 ceph-mon[29092]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-17T07:01:14.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:13 smithi100 ceph-mon[29092]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-17T07:01:14.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:13 smithi100 ceph-mon[29092]: osdmap e40: 6 total, 5 up, 6 in 2024-09-17T07:01:14.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:13 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T07:01:14.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:13 smithi100 ceph-mon[29092]: from='osd.5 v2:172.21.15.135:6804/2256520893' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-09-17T07:01:14.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:13 smithi100 ceph-mon[29092]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-09-17T07:01:14.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:13 smithi100 ceph-mon[39462]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-17T07:01:14.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:13 smithi100 ceph-mon[39462]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-17T07:01:14.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:13 smithi100 ceph-mon[39462]: osdmap e40: 6 total, 5 up, 6 in 2024-09-17T07:01:14.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:13 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T07:01:14.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:13 smithi100 ceph-mon[39462]: from='osd.5 v2:172.21.15.135:6804/2256520893' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-09-17T07:01:14.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:13 smithi100 ceph-mon[39462]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-09-17T07:01:14.172 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:01:14 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-5[43919]: 2024-09-17T07:01:14.079+0000 7f174517c640 -1 osd.5 0 waiting for initial osdmap 2024-09-17T07:01:14.172 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:01:14 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-5[43919]: 2024-09-17T07:01:14.085+0000 7f173cf9d640 -1 osd.5 41 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T07:01:14.641 INFO:teuthology.orchestra.run.smithi135.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-09-17T07:01:14.722 INFO:teuthology.orchestra.run.smithi135.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-09-17T07:01:14.735 INFO:teuthology.orchestra.run.smithi135.stderr: stderr: 10+0 records in 2024-09-17T07:01:14.736 INFO:teuthology.orchestra.run.smithi135.stderr:10+0 records out 2024-09-17T07:01:14.736 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:14 smithi135 ceph-mon[33238]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]': finished 2024-09-17T07:01:14.736 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:14 smithi135 ceph-mon[33238]: osdmap e41: 6 total, 5 up, 6 in 2024-09-17T07:01:14.736 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:14 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T07:01:14.736 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:14 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T07:01:14.736 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:14 smithi135 ceph-mon[33238]: from='osd.5 ' entity='osd.5' 2024-09-17T07:01:14.736 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:14 smithi135 ceph-mon[33238]: osd.5 v2:172.21.15.135:6804/2256520893 boot 2024-09-17T07:01:14.737 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:14 smithi135 ceph-mon[33238]: osdmap e42: 6 total, 6 up, 6 in 2024-09-17T07:01:14.737 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:14 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T07:01:14.737 INFO:teuthology.orchestra.run.smithi135.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0124067 s, 845 MB/s 2024-09-17T07:01:14.737 INFO:teuthology.orchestra.run.smithi135.stderr:--> Zapping successful for: 2024-09-17T07:01:15.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:14 smithi100 ceph-mon[29092]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]': finished 2024-09-17T07:01:15.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:14 smithi100 ceph-mon[29092]: osdmap e41: 6 total, 5 up, 6 in 2024-09-17T07:01:15.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:14 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T07:01:15.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:14 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T07:01:15.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:14 smithi100 ceph-mon[29092]: from='osd.5 ' entity='osd.5' 2024-09-17T07:01:15.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:14 smithi100 ceph-mon[29092]: osd.5 v2:172.21.15.135:6804/2256520893 boot 2024-09-17T07:01:15.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:14 smithi100 ceph-mon[29092]: osdmap e42: 6 total, 6 up, 6 in 2024-09-17T07:01:15.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:14 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T07:01:15.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:14 smithi100 ceph-mon[39462]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]': finished 2024-09-17T07:01:15.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:14 smithi100 ceph-mon[39462]: osdmap e41: 6 total, 5 up, 6 in 2024-09-17T07:01:15.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:14 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T07:01:15.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:14 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T07:01:15.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:14 smithi100 ceph-mon[39462]: from='osd.5 ' entity='osd.5' 2024-09-17T07:01:15.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:14 smithi100 ceph-mon[39462]: osd.5 v2:172.21.15.135:6804/2256520893 boot 2024-09-17T07:01:15.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:14 smithi100 ceph-mon[39462]: osdmap e42: 6 total, 6 up, 6 in 2024-09-17T07:01:15.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:14 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T07:01:15.447 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi135:vg_nvme/lv_2 2024-09-17T07:01:15.685 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:15 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:01:15.686 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:15 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:01:15.686 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:15 smithi135 ceph-mon[33238]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-17T07:01:15.686 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:15 smithi135 ceph-mon[33238]: osdmap e43: 6 total, 6 up, 6 in 2024-09-17T07:01:15.949 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:01:16.126 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:15 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:01:16.126 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:15 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:01:16.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:15 smithi100 ceph-mon[29092]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-17T07:01:16.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:15 smithi100 ceph-mon[29092]: osdmap e43: 6 total, 6 up, 6 in 2024-09-17T07:01:16.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:15 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:01:16.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:15 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:01:16.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:15 smithi100 ceph-mon[39462]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-17T07:01:16.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:15 smithi100 ceph-mon[39462]: osdmap e43: 6 total, 6 up, 6 in 2024-09-17T07:01:17.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:17 smithi135 ceph-mon[33238]: osdmap e44: 6 total, 6 up, 6 in 2024-09-17T07:01:17.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:17 smithi135 ceph-mon[33238]: pgmap v133: 1 pgs: 1 remapped; 577 KiB data, 434 MiB used, 536 GiB / 536 GiB avail 2024-09-17T07:01:17.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:17 smithi100 ceph-mon[29092]: osdmap e44: 6 total, 6 up, 6 in 2024-09-17T07:01:17.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:17 smithi100 ceph-mon[29092]: pgmap v133: 1 pgs: 1 remapped; 577 KiB data, 434 MiB used, 536 GiB / 536 GiB avail 2024-09-17T07:01:17.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:17 smithi100 ceph-mon[39462]: osdmap e44: 6 total, 6 up, 6 in 2024-09-17T07:01:17.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:17 smithi100 ceph-mon[39462]: pgmap v133: 1 pgs: 1 remapped; 577 KiB data, 434 MiB used, 536 GiB / 536 GiB avail 2024-09-17T07:01:18.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:18 smithi135 ceph-mon[33238]: osdmap e45: 6 total, 6 up, 6 in 2024-09-17T07:01:18.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:18 smithi135 ceph-mon[33238]: Detected new or changed devices on smithi135 2024-09-17T07:01:18.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:18 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:18.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:18 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:18.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:18 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:01:18.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:18 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:01:18.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:18 smithi135 ceph-mon[33238]: Adjusting osd_memory_target on smithi135 to 2907M 2024-09-17T07:01:18.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:18 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:18.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:18 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:01:18.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:18 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:01:18.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:18 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:18.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:18 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:01:18.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:18 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:01:18.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:18 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:01:18.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:18 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:18.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:18 smithi100 ceph-mon[29092]: osdmap e45: 6 total, 6 up, 6 in 2024-09-17T07:01:18.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:18 smithi100 ceph-mon[29092]: Detected new or changed devices on smithi135 2024-09-17T07:01:18.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:18 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:18.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:18 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:18.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:18 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:01:18.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:18 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:01:18.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:18 smithi100 ceph-mon[29092]: Adjusting osd_memory_target on smithi135 to 2907M 2024-09-17T07:01:18.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:18 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:18.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:18 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:01:18.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:18 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:01:18.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:18 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:18.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:18 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:01:18.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:18 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:01:18.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:18 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:01:18.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:18 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:18.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:18 smithi100 ceph-mon[39462]: osdmap e45: 6 total, 6 up, 6 in 2024-09-17T07:01:18.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:18 smithi100 ceph-mon[39462]: Detected new or changed devices on smithi135 2024-09-17T07:01:18.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:18 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:18.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:18 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:18.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:18 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:01:18.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:18 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:01:18.379 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:18 smithi100 ceph-mon[39462]: Adjusting osd_memory_target on smithi135 to 2907M 2024-09-17T07:01:18.379 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:18 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:18.379 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:18 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:01:18.379 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:18 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:01:18.379 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:18 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:18.379 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:18 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:01:18.379 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:18 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:01:18.379 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:18 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:01:18.379 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:18 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:19.187 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:19 smithi135 ceph-mon[33238]: osdmap e46: 6 total, 6 up, 6 in 2024-09-17T07:01:19.187 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:19 smithi135 ceph-mon[33238]: from='client.24241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi135:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:01:19.187 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:19 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T07:01:19.187 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:19 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T07:01:19.187 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:19 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:01:19.187 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:19 smithi135 ceph-mon[33238]: pgmap v136: 1 pgs: 1 remapped; 577 KiB data, 434 MiB used, 536 GiB / 536 GiB avail 2024-09-17T07:01:19.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:19 smithi100 ceph-mon[39462]: osdmap e46: 6 total, 6 up, 6 in 2024-09-17T07:01:19.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:19 smithi100 ceph-mon[39462]: from='client.24241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi135:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:01:19.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:19 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T07:01:19.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:19 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T07:01:19.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:19 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:01:19.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:19 smithi100 ceph-mon[39462]: pgmap v136: 1 pgs: 1 remapped; 577 KiB data, 434 MiB used, 536 GiB / 536 GiB avail 2024-09-17T07:01:19.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:19 smithi100 ceph-mon[29092]: osdmap e46: 6 total, 6 up, 6 in 2024-09-17T07:01:19.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:19 smithi100 ceph-mon[29092]: from='client.24241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi135:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:01:19.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:19 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T07:01:19.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:19 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T07:01:19.379 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:19 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:01:19.379 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:19 smithi100 ceph-mon[29092]: pgmap v136: 1 pgs: 1 remapped; 577 KiB data, 434 MiB used, 536 GiB / 536 GiB avail 2024-09-17T07:01:21.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:20 smithi135 ceph-mon[33238]: from='client.? 172.21.15.135:0/1163069865' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "de799f64-ebd2-4a46-91f6-c2f276bee21f"}]: dispatch 2024-09-17T07:01:21.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:20 smithi135 ceph-mon[33238]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "de799f64-ebd2-4a46-91f6-c2f276bee21f"}]: dispatch 2024-09-17T07:01:21.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:20 smithi135 ceph-mon[33238]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "de799f64-ebd2-4a46-91f6-c2f276bee21f"}]': finished 2024-09-17T07:01:21.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:20 smithi135 ceph-mon[33238]: osdmap e47: 7 total, 6 up, 7 in 2024-09-17T07:01:21.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:20 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T07:01:21.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:20 smithi135 ceph-mon[33238]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 126 KiB/s, 0 objects/s recovering 2024-09-17T07:01:21.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:20 smithi135 ceph-mon[33238]: from='client.? 172.21.15.135:0/3096132520' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T07:01:21.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:20 smithi100 ceph-mon[29092]: from='client.? 172.21.15.135:0/1163069865' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "de799f64-ebd2-4a46-91f6-c2f276bee21f"}]: dispatch 2024-09-17T07:01:21.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:20 smithi100 ceph-mon[29092]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "de799f64-ebd2-4a46-91f6-c2f276bee21f"}]: dispatch 2024-09-17T07:01:21.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:20 smithi100 ceph-mon[29092]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "de799f64-ebd2-4a46-91f6-c2f276bee21f"}]': finished 2024-09-17T07:01:21.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:20 smithi100 ceph-mon[29092]: osdmap e47: 7 total, 6 up, 7 in 2024-09-17T07:01:21.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:20 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T07:01:21.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:20 smithi100 ceph-mon[29092]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 126 KiB/s, 0 objects/s recovering 2024-09-17T07:01:21.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:20 smithi100 ceph-mon[29092]: from='client.? 172.21.15.135:0/3096132520' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T07:01:21.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:20 smithi100 ceph-mon[39462]: from='client.? 172.21.15.135:0/1163069865' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "de799f64-ebd2-4a46-91f6-c2f276bee21f"}]: dispatch 2024-09-17T07:01:21.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:20 smithi100 ceph-mon[39462]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "de799f64-ebd2-4a46-91f6-c2f276bee21f"}]: dispatch 2024-09-17T07:01:21.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:20 smithi100 ceph-mon[39462]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "de799f64-ebd2-4a46-91f6-c2f276bee21f"}]': finished 2024-09-17T07:01:21.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:20 smithi100 ceph-mon[39462]: osdmap e47: 7 total, 6 up, 7 in 2024-09-17T07:01:21.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:20 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T07:01:21.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:20 smithi100 ceph-mon[39462]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 126 KiB/s, 0 objects/s recovering 2024-09-17T07:01:21.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:20 smithi100 ceph-mon[39462]: from='client.? 172.21.15.135:0/3096132520' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T07:01:23.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:23 smithi135 ceph-mon[33238]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-17T07:01:23.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:23 smithi100 ceph-mon[29092]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-17T07:01:23.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:23 smithi100 ceph-mon[39462]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-17T07:01:24.739 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:24 smithi135 ceph-mon[33238]: 1.0 scrub starts 2024-09-17T07:01:24.739 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:24 smithi135 ceph-mon[33238]: 1.0 scrub ok 2024-09-17T07:01:24.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:24 smithi100 ceph-mon[29092]: 1.0 scrub starts 2024-09-17T07:01:24.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:24 smithi100 ceph-mon[29092]: 1.0 scrub ok 2024-09-17T07:01:24.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:24 smithi100 ceph-mon[39462]: 1.0 scrub starts 2024-09-17T07:01:24.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:24 smithi100 ceph-mon[39462]: 1.0 scrub ok 2024-09-17T07:01:25.832 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:25 smithi135 ceph-mon[33238]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-09-17T07:01:25.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:25 smithi100 ceph-mon[29092]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-09-17T07:01:25.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:25 smithi100 ceph-mon[39462]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-09-17T07:01:26.779 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:26 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-17T07:01:26.780 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:26 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:01:27.126 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:26 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-17T07:01:27.126 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:26 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:01:27.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:26 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-17T07:01:27.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:26 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:01:27.726 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:27 smithi135 ceph-mon[33238]: Deploying daemon osd.6 on smithi135 2024-09-17T07:01:27.726 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:27 smithi135 ceph-mon[33238]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-09-17T07:01:28.028 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:27 smithi100 ceph-mon[29092]: Deploying daemon osd.6 on smithi135 2024-09-17T07:01:28.028 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:27 smithi100 ceph-mon[29092]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-09-17T07:01:28.028 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:27 smithi100 ceph-mon[39462]: Deploying daemon osd.6 on smithi135 2024-09-17T07:01:28.029 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:27 smithi100 ceph-mon[39462]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-09-17T07:01:29.877 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:29 smithi135 ceph-mon[33238]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-17T07:01:30.126 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:29 smithi100 ceph-mon[29092]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-17T07:01:30.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:29 smithi100 ceph-mon[39462]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-17T07:01:31.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:31 smithi135 ceph-mon[33238]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-17T07:01:31.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:31 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:01:31.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:31 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:31.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:31 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:32.126 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:31 smithi100 ceph-mon[29092]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-17T07:01:32.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:31 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:01:32.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:31 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:32.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:31 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:32.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:31 smithi100 ceph-mon[39462]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-17T07:01:32.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:31 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:01:32.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:31 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:32.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:31 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:33.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:33 smithi135 ceph-mon[33238]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-17T07:01:33.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:33 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:33.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:33 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:33.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:33 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:01:33.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:33 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:01:33.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:33 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:34.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:33 smithi100 ceph-mon[29092]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-17T07:01:34.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:33 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:34.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:33 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:34.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:33 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:01:34.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:33 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:01:34.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:33 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:34.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:33 smithi100 ceph-mon[39462]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-17T07:01:34.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:33 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:34.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:33 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:34.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:33 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:01:34.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:33 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:01:34.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:33 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:34.169 INFO:teuthology.orchestra.run.smithi135.stdout:Created osd(s) 6 on host 'smithi135' 2024-09-17T07:01:34.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:34 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:01:34.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:34 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:34.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:34 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:34.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:34 smithi135 ceph-mon[33238]: from='osd.6 v2:172.21.15.135:6808/2750184152' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-17T07:01:34.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:34 smithi135 ceph-mon[33238]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-17T07:01:34.857 DEBUG:teuthology.orchestra.run.smithi135:osd.6> sudo journalctl -f -n 0 -u ceph-e848267c-74c1-11ef-bceb-c7b262605968@osd.6.service 2024-09-17T07:01:34.859 INFO:tasks.cephadm:Deploying osd.7 on smithi135 with /dev/vg_nvme/lv_1... 2024-09-17T07:01:34.859 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-09-17T07:01:35.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:34 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:01:35.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:34 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:35.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:34 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:35.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:34 smithi100 ceph-mon[29092]: from='osd.6 v2:172.21.15.135:6808/2750184152' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-17T07:01:35.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:34 smithi100 ceph-mon[29092]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-17T07:01:35.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:34 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:01:35.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:34 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:35.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:34 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:35.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:34 smithi100 ceph-mon[39462]: from='osd.6 v2:172.21.15.135:6808/2750184152' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-17T07:01:35.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:34 smithi100 ceph-mon[39462]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-17T07:01:35.536 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:01:35.899 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:35 smithi135 ceph-mon[33238]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-17T07:01:35.899 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:35 smithi135 ceph-mon[33238]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-17T07:01:35.899 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:35 smithi135 ceph-mon[33238]: osdmap e48: 7 total, 6 up, 7 in 2024-09-17T07:01:35.899 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:35 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T07:01:35.899 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:35 smithi135 ceph-mon[33238]: from='osd.6 v2:172.21.15.135:6808/2750184152' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-09-17T07:01:35.899 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:35 smithi135 ceph-mon[33238]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-09-17T07:01:36.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:35 smithi100 ceph-mon[29092]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-17T07:01:36.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:35 smithi100 ceph-mon[29092]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-17T07:01:36.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:35 smithi100 ceph-mon[29092]: osdmap e48: 7 total, 6 up, 7 in 2024-09-17T07:01:36.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:35 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T07:01:36.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:35 smithi100 ceph-mon[29092]: from='osd.6 v2:172.21.15.135:6808/2750184152' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-09-17T07:01:36.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:35 smithi100 ceph-mon[29092]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-09-17T07:01:36.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:35 smithi100 ceph-mon[39462]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-17T07:01:36.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:35 smithi100 ceph-mon[39462]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-17T07:01:36.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:35 smithi100 ceph-mon[39462]: osdmap e48: 7 total, 6 up, 7 in 2024-09-17T07:01:36.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:35 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T07:01:36.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:35 smithi100 ceph-mon[39462]: from='osd.6 v2:172.21.15.135:6808/2750184152' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-09-17T07:01:36.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:35 smithi100 ceph-mon[39462]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-09-17T07:01:36.789 INFO:journalctl@ceph.osd.6.smithi135.stdout:Sep 17 07:01:36 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-6[49830]: 2024-09-17T07:01:36.625+0000 7fef8a2db640 -1 osd.6 0 waiting for initial osdmap 2024-09-17T07:01:36.789 INFO:journalctl@ceph.osd.6.smithi135.stdout:Sep 17 07:01:36 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-6[49830]: 2024-09-17T07:01:36.631+0000 7fef820fc640 -1 osd.6 49 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T07:01:37.291 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:37 smithi135 ceph-mon[33238]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]': finished 2024-09-17T07:01:37.291 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:37 smithi135 ceph-mon[33238]: osdmap e49: 7 total, 6 up, 7 in 2024-09-17T07:01:37.292 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:37 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T07:01:37.292 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:37 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T07:01:37.292 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:37 smithi135 ceph-mon[33238]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-17T07:01:37.292 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:37 smithi135 ceph-mon[33238]: from='osd.6 ' entity='osd.6' 2024-09-17T07:01:37.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:37 smithi100 ceph-mon[29092]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]': finished 2024-09-17T07:01:37.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:37 smithi100 ceph-mon[29092]: osdmap e49: 7 total, 6 up, 7 in 2024-09-17T07:01:37.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:37 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T07:01:37.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:37 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T07:01:37.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:37 smithi100 ceph-mon[29092]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-17T07:01:37.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:37 smithi100 ceph-mon[29092]: from='osd.6 ' entity='osd.6' 2024-09-17T07:01:37.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:37 smithi100 ceph-mon[39462]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]': finished 2024-09-17T07:01:37.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:37 smithi100 ceph-mon[39462]: osdmap e49: 7 total, 6 up, 7 in 2024-09-17T07:01:37.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:37 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T07:01:37.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:37 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T07:01:37.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:37 smithi100 ceph-mon[39462]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-17T07:01:37.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:37 smithi100 ceph-mon[39462]: from='osd.6 ' entity='osd.6' 2024-09-17T07:01:37.672 INFO:teuthology.orchestra.run.smithi135.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-09-17T07:01:37.735 INFO:teuthology.orchestra.run.smithi135.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-09-17T07:01:37.758 INFO:teuthology.orchestra.run.smithi135.stderr: stderr: 10+0 records in 2024-09-17T07:01:37.758 INFO:teuthology.orchestra.run.smithi135.stderr:10+0 records out 2024-09-17T07:01:37.758 INFO:teuthology.orchestra.run.smithi135.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0208006 s, 504 MB/s 2024-09-17T07:01:37.758 INFO:teuthology.orchestra.run.smithi135.stderr:--> Zapping successful for: 2024-09-17T07:01:38.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:38 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:01:38.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:38 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:01:38.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:38 smithi135 ceph-mon[33238]: osd.6 v2:172.21.15.135:6808/2750184152 boot 2024-09-17T07:01:38.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:38 smithi135 ceph-mon[33238]: osdmap e50: 7 total, 7 up, 7 in 2024-09-17T07:01:38.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:38 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T07:01:38.399 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi135:vg_nvme/lv_1 2024-09-17T07:01:38.594 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:01:38.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:38 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:01:38.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:38 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:01:38.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:38 smithi100 ceph-mon[29092]: osd.6 v2:172.21.15.135:6808/2750184152 boot 2024-09-17T07:01:38.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:38 smithi100 ceph-mon[29092]: osdmap e50: 7 total, 7 up, 7 in 2024-09-17T07:01:38.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:38 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T07:01:38.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:38 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:01:38.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:38 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:01:38.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:38 smithi100 ceph-mon[39462]: osd.6 v2:172.21.15.135:6808/2750184152 boot 2024-09-17T07:01:38.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:38 smithi100 ceph-mon[39462]: osdmap e50: 7 total, 7 up, 7 in 2024-09-17T07:01:38.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:38 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T07:01:39.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:39 smithi135 ceph-mon[33238]: osdmap e51: 7 total, 7 up, 7 in 2024-09-17T07:01:39.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:39 smithi135 ceph-mon[33238]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-17T07:01:39.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:39 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:39.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:39 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:39.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:39 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:01:39.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:39 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:01:39.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:39 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:01:39.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:39 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:39.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:39 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:01:39.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:39 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:01:39.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:39 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:39.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:39 smithi100 ceph-mon[29092]: osdmap e51: 7 total, 7 up, 7 in 2024-09-17T07:01:39.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:39 smithi100 ceph-mon[29092]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-17T07:01:39.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:39 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:39.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:39 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:39.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:39 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:01:39.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:39 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:01:39.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:39 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:01:39.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:39 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:39.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:39 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:01:39.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:39 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:01:39.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:39 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:39.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:39 smithi100 ceph-mon[39462]: osdmap e51: 7 total, 7 up, 7 in 2024-09-17T07:01:39.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:39 smithi100 ceph-mon[39462]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-17T07:01:39.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:39 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:39.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:39 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:39.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:39 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:01:39.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:39 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:01:39.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:39 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:01:39.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:39 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:39.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:39 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:01:39.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:39 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:01:39.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:39 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:40.339 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:40 smithi135 ceph-mon[33238]: Detected new or changed devices on smithi135 2024-09-17T07:01:40.339 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:40 smithi135 ceph-mon[33238]: Adjusting osd_memory_target on smithi135 to 1938M 2024-09-17T07:01:40.339 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:40 smithi135 ceph-mon[33238]: osdmap e52: 7 total, 7 up, 7 in 2024-09-17T07:01:40.339 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:40 smithi135 ceph-mon[33238]: osdmap e53: 7 total, 7 up, 7 in 2024-09-17T07:01:40.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:40 smithi100 ceph-mon[29092]: Detected new or changed devices on smithi135 2024-09-17T07:01:40.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:40 smithi100 ceph-mon[29092]: Adjusting osd_memory_target on smithi135 to 1938M 2024-09-17T07:01:40.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:40 smithi100 ceph-mon[29092]: osdmap e52: 7 total, 7 up, 7 in 2024-09-17T07:01:40.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:40 smithi100 ceph-mon[29092]: osdmap e53: 7 total, 7 up, 7 in 2024-09-17T07:01:40.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:40 smithi100 ceph-mon[39462]: Detected new or changed devices on smithi135 2024-09-17T07:01:40.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:40 smithi100 ceph-mon[39462]: Adjusting osd_memory_target on smithi135 to 1938M 2024-09-17T07:01:40.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:40 smithi100 ceph-mon[39462]: osdmap e52: 7 total, 7 up, 7 in 2024-09-17T07:01:40.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:40 smithi100 ceph-mon[39462]: osdmap e53: 7 total, 7 up, 7 in 2024-09-17T07:01:41.348 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:41 smithi135 ceph-mon[33238]: pgmap v154: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-09-17T07:01:41.348 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:41 smithi135 ceph-mon[33238]: from='client.24268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi135:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:01:41.349 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:41 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T07:01:41.349 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:41 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T07:01:41.349 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:41 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:01:41.349 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:41 smithi135 ceph-mon[33238]: osdmap e54: 7 total, 7 up, 7 in 2024-09-17T07:01:41.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:41 smithi100 ceph-mon[29092]: pgmap v154: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-09-17T07:01:41.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:41 smithi100 ceph-mon[29092]: from='client.24268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi135:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:01:41.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:41 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T07:01:41.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:41 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T07:01:41.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:41 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:01:41.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:41 smithi100 ceph-mon[29092]: osdmap e54: 7 total, 7 up, 7 in 2024-09-17T07:01:41.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:41 smithi100 ceph-mon[39462]: pgmap v154: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-09-17T07:01:41.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:41 smithi100 ceph-mon[39462]: from='client.24268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi135:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:01:41.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:41 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T07:01:41.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:41 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T07:01:41.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:41 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:01:41.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:41 smithi100 ceph-mon[39462]: osdmap e54: 7 total, 7 up, 7 in 2024-09-17T07:01:43.796 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:43 smithi135 ceph-mon[33238]: pgmap v156: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 132 KiB/s, 0 objects/s recovering 2024-09-17T07:01:43.796 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:43 smithi135 ceph-mon[33238]: from='client.? 172.21.15.135:0/2873854913' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "830932c7-5de1-44ff-a840-e53a6e859bda"}]: dispatch 2024-09-17T07:01:43.796 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:43 smithi135 ceph-mon[33238]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "830932c7-5de1-44ff-a840-e53a6e859bda"}]: dispatch 2024-09-17T07:01:43.796 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:43 smithi135 ceph-mon[33238]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "830932c7-5de1-44ff-a840-e53a6e859bda"}]': finished 2024-09-17T07:01:43.796 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:43 smithi135 ceph-mon[33238]: osdmap e55: 8 total, 7 up, 8 in 2024-09-17T07:01:43.797 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:43 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T07:01:43.797 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:43 smithi135 ceph-mon[33238]: from='client.? 172.21.15.135:0/1766224036' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T07:01:43.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:43 smithi100 ceph-mon[29092]: pgmap v156: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 132 KiB/s, 0 objects/s recovering 2024-09-17T07:01:43.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:43 smithi100 ceph-mon[29092]: from='client.? 172.21.15.135:0/2873854913' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "830932c7-5de1-44ff-a840-e53a6e859bda"}]: dispatch 2024-09-17T07:01:43.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:43 smithi100 ceph-mon[29092]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "830932c7-5de1-44ff-a840-e53a6e859bda"}]: dispatch 2024-09-17T07:01:43.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:43 smithi100 ceph-mon[29092]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "830932c7-5de1-44ff-a840-e53a6e859bda"}]': finished 2024-09-17T07:01:43.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:43 smithi100 ceph-mon[29092]: osdmap e55: 8 total, 7 up, 8 in 2024-09-17T07:01:43.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:43 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T07:01:43.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:43 smithi100 ceph-mon[29092]: from='client.? 172.21.15.135:0/1766224036' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T07:01:43.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:43 smithi100 ceph-mon[39462]: pgmap v156: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 132 KiB/s, 0 objects/s recovering 2024-09-17T07:01:43.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:43 smithi100 ceph-mon[39462]: from='client.? 172.21.15.135:0/2873854913' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "830932c7-5de1-44ff-a840-e53a6e859bda"}]: dispatch 2024-09-17T07:01:43.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:43 smithi100 ceph-mon[39462]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "830932c7-5de1-44ff-a840-e53a6e859bda"}]: dispatch 2024-09-17T07:01:43.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:43 smithi100 ceph-mon[39462]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "830932c7-5de1-44ff-a840-e53a6e859bda"}]': finished 2024-09-17T07:01:43.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:43 smithi100 ceph-mon[39462]: osdmap e55: 8 total, 7 up, 8 in 2024-09-17T07:01:43.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:43 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T07:01:43.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:43 smithi100 ceph-mon[39462]: from='client.? 172.21.15.135:0/1766224036' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T07:01:45.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:45 smithi100 ceph-mon[29092]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-09-17T07:01:45.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:45 smithi100 ceph-mon[39462]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-09-17T07:01:46.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:45 smithi135 ceph-mon[33238]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-09-17T07:01:47.655 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:47 smithi135 ceph-mon[33238]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-17T07:01:47.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:47 smithi100 ceph-mon[29092]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-17T07:01:47.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:47 smithi100 ceph-mon[39462]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-17T07:01:49.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:49 smithi135 ceph-mon[33238]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-17T07:01:49.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:49 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-17T07:01:49.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:49 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:01:49.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:49 smithi135 ceph-mon[33238]: Deploying daemon osd.7 on smithi135 2024-09-17T07:01:50.126 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:49 smithi100 ceph-mon[29092]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-17T07:01:50.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:49 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-17T07:01:50.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:49 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:01:50.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:49 smithi100 ceph-mon[29092]: Deploying daemon osd.7 on smithi135 2024-09-17T07:01:50.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:49 smithi100 ceph-mon[39462]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-17T07:01:50.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:49 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-17T07:01:50.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:49 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:01:50.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:49 smithi100 ceph-mon[39462]: Deploying daemon osd.7 on smithi135 2024-09-17T07:01:51.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:50 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:01:51.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:50 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:01:51.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:50 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:01:51.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:50 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:01:51.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:50 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:01:51.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:50 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:01:51.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:51 smithi135 ceph-mon[33238]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-17T07:01:51.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:51 smithi135 ceph-mon[33238]: osdmap e56: 8 total, 7 up, 8 in 2024-09-17T07:01:51.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:51 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T07:01:52.126 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:51 smithi100 ceph-mon[29092]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-17T07:01:52.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:51 smithi100 ceph-mon[29092]: osdmap e56: 8 total, 7 up, 8 in 2024-09-17T07:01:52.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:51 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T07:01:52.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:51 smithi100 ceph-mon[39462]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-17T07:01:52.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:51 smithi100 ceph-mon[39462]: osdmap e56: 8 total, 7 up, 8 in 2024-09-17T07:01:52.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:51 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T07:01:52.974 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:52 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:01:52.974 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:52 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:01:53.126 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:52 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:01:53.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:52 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:01:53.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:52 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:01:53.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:52 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:01:54.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:53 smithi135 ceph-mon[33238]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-17T07:01:54.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:53 smithi135 ceph-mon[33238]: osdmap e57: 8 total, 7 up, 8 in 2024-09-17T07:01:54.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:53 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T07:01:54.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:53 smithi100 ceph-mon[29092]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-17T07:01:54.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:53 smithi100 ceph-mon[29092]: osdmap e57: 8 total, 7 up, 8 in 2024-09-17T07:01:54.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:53 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T07:01:54.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:53 smithi100 ceph-mon[39462]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-17T07:01:54.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:53 smithi100 ceph-mon[39462]: osdmap e57: 8 total, 7 up, 8 in 2024-09-17T07:01:54.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:53 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T07:01:54.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:54 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:01:54.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:54 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:54.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:54 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:55.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:54 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:01:55.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:54 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:55.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:54 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:55.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:54 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:01:55.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:54 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:55.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:54 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:55.782 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:55 smithi135 ceph-mon[33238]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-17T07:01:56.126 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:55 smithi100 ceph-mon[29092]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-17T07:01:56.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:55 smithi100 ceph-mon[39462]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-17T07:01:57.275 INFO:teuthology.orchestra.run.smithi135.stdout:Created osd(s) 7 on host 'smithi135' 2024-09-17T07:01:57.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:57 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:57.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:57 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:57.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:57 smithi135 ceph-mon[33238]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-17T07:01:57.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:57 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:01:57.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:57 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:01:57.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:57 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:57.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:57 smithi135 ceph-mon[33238]: from='osd.7 v2:172.21.15.135:6812/160304773' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-17T07:01:57.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:57 smithi135 ceph-mon[33238]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-17T07:01:57.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:57 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:01:57.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:57 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:57.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:57 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:57.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:57 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:57.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:57 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:57.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:57 smithi100 ceph-mon[29092]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-17T07:01:57.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:57 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:01:57.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:57 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:01:57.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:57 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:57.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:57 smithi100 ceph-mon[29092]: from='osd.7 v2:172.21.15.135:6812/160304773' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-17T07:01:57.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:57 smithi100 ceph-mon[29092]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-17T07:01:57.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:57 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:01:57.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:57 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:57.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:57 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:57.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:57 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:57.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:57 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:57.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:57 smithi100 ceph-mon[39462]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-17T07:01:57.879 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:57 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:01:57.879 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:57 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:01:57.879 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:57 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:57.879 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:57 smithi100 ceph-mon[39462]: from='osd.7 v2:172.21.15.135:6812/160304773' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-17T07:01:57.879 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:57 smithi100 ceph-mon[39462]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-17T07:01:57.879 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:57 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:01:57.879 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:57 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:57.879 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:57 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:01:57.900 DEBUG:teuthology.orchestra.run.smithi135:osd.7> sudo journalctl -f -n 0 -u ceph-e848267c-74c1-11ef-bceb-c7b262605968@osd.7.service 2024-09-17T07:01:57.903 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-09-17T07:01:57.903 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd stat -f json 2024-09-17T07:01:58.103 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:01:59.222 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:58 smithi100 ceph-mon[29092]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-17T07:01:59.222 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:58 smithi100 ceph-mon[29092]: osdmap e58: 8 total, 7 up, 8 in 2024-09-17T07:01:59.222 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:58 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T07:01:59.222 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:58 smithi100 ceph-mon[29092]: from='osd.7 v2:172.21.15.135:6812/160304773' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-09-17T07:01:59.223 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:58 smithi100 ceph-mon[29092]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-09-17T07:01:59.223 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:58 smithi100 ceph-mon[29092]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-17T07:01:59.223 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:58 smithi100 ceph-mon[39462]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-17T07:01:59.223 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:58 smithi100 ceph-mon[39462]: osdmap e58: 8 total, 7 up, 8 in 2024-09-17T07:01:59.223 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:58 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T07:01:59.223 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:58 smithi100 ceph-mon[39462]: from='osd.7 v2:172.21.15.135:6812/160304773' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-09-17T07:01:59.223 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:58 smithi100 ceph-mon[39462]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-09-17T07:01:59.223 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:58 smithi100 ceph-mon[39462]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-17T07:01:59.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:58 smithi135 ceph-mon[33238]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-17T07:01:59.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:58 smithi135 ceph-mon[33238]: osdmap e58: 8 total, 7 up, 8 in 2024-09-17T07:01:59.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:58 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T07:01:59.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:58 smithi135 ceph-mon[33238]: from='osd.7 v2:172.21.15.135:6812/160304773' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-09-17T07:01:59.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:58 smithi135 ceph-mon[33238]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-09-17T07:01:59.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:58 smithi135 ceph-mon[33238]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-17T07:01:59.794 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:01:59 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:01:59.343+0000 7fab0dfed640 -1 osd.7 0 waiting for initial osdmap 2024-09-17T07:01:59.794 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:01:59 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:01:59.350+0000 7fab05e0e640 -1 osd.7 59 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T07:01:59.994 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T07:01:59.995 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:59 smithi100 ceph-mon[29092]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]': finished 2024-09-17T07:01:59.995 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:59 smithi100 ceph-mon[29092]: osdmap e59: 8 total, 7 up, 8 in 2024-09-17T07:01:59.995 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:59 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T07:01:59.995 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:59 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T07:01:59.995 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:59 smithi100 ceph-mon[29092]: from='osd.7 ' entity='osd.7' 2024-09-17T07:01:59.995 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:01:59 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T07:01:59.995 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:59 smithi100 ceph-mon[39462]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]': finished 2024-09-17T07:01:59.995 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:59 smithi100 ceph-mon[39462]: osdmap e59: 8 total, 7 up, 8 in 2024-09-17T07:01:59.995 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:59 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T07:01:59.995 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:59 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T07:01:59.996 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:59 smithi100 ceph-mon[39462]: from='osd.7 ' entity='osd.7' 2024-09-17T07:01:59.996 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:01:59 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T07:02:00.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:59 smithi135 ceph-mon[33238]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]': finished 2024-09-17T07:02:00.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:59 smithi135 ceph-mon[33238]: osdmap e59: 8 total, 7 up, 8 in 2024-09-17T07:02:00.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:59 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T07:02:00.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:59 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T07:02:00.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:59 smithi135 ceph-mon[33238]: from='osd.7 ' entity='osd.7' 2024-09-17T07:02:00.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:01:59 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T07:02:00.695 INFO:teuthology.orchestra.run.smithi100.stdout:{"epoch":60,"num_osds":8,"num_up_osds":8,"osd_up_since":1726556519,"num_in_osds":8,"osd_in_since":1726556502,"num_remapped_pgs":1} 2024-09-17T07:02:00.696 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-17T07:02:00.889 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:02:01.184 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:00 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:02:01.185 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:00 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:02:01.185 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:00 smithi100 ceph-mon[29092]: osd.7 v2:172.21.15.135:6812/160304773 boot 2024-09-17T07:02:01.185 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:00 smithi100 ceph-mon[29092]: osdmap e60: 8 total, 8 up, 8 in 2024-09-17T07:02:01.185 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:00 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T07:02:01.185 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:00 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/22018732' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-17T07:02:01.185 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:00 smithi100 ceph-mon[29092]: osdmap e61: 8 total, 8 up, 8 in 2024-09-17T07:02:01.185 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:00 smithi100 ceph-mon[29092]: pgmap v172: 1 pgs: 1 remapped+peering; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-17T07:02:01.185 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:00 smithi100 ceph-mon[29092]: Detected new or changed devices on smithi135 2024-09-17T07:02:01.185 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:00 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:01.185 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:00 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:01.185 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:00 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:02:01.185 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:00 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:02:01.186 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:00 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:02:01.186 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:00 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:02:01.186 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:00 smithi100 ceph-mon[29092]: Adjusting osd_memory_target on smithi135 to 1453M 2024-09-17T07:02:01.186 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:00 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:01.186 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:00 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:02:01.186 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:00 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:02:01.186 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:00 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:01.186 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:00 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:02:01.186 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:00 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:02:01.186 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:00 smithi100 ceph-mon[39462]: osd.7 v2:172.21.15.135:6812/160304773 boot 2024-09-17T07:02:01.186 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:00 smithi100 ceph-mon[39462]: osdmap e60: 8 total, 8 up, 8 in 2024-09-17T07:02:01.187 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:00 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T07:02:01.187 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:00 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/22018732' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-17T07:02:01.187 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:00 smithi100 ceph-mon[39462]: osdmap e61: 8 total, 8 up, 8 in 2024-09-17T07:02:01.187 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:00 smithi100 ceph-mon[39462]: pgmap v172: 1 pgs: 1 remapped+peering; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-17T07:02:01.187 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:00 smithi100 ceph-mon[39462]: Detected new or changed devices on smithi135 2024-09-17T07:02:01.187 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:00 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:01.187 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:00 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:01.187 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:00 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:02:01.187 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:00 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:02:01.187 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:00 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:02:01.187 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:00 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:02:01.187 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:00 smithi100 ceph-mon[39462]: Adjusting osd_memory_target on smithi135 to 1453M 2024-09-17T07:02:01.187 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:00 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:01.188 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:00 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:02:01.188 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:00 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:02:01.188 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:00 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:01.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:00 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:02:01.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:00 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:02:01.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:00 smithi135 ceph-mon[33238]: osd.7 v2:172.21.15.135:6812/160304773 boot 2024-09-17T07:02:01.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:00 smithi135 ceph-mon[33238]: osdmap e60: 8 total, 8 up, 8 in 2024-09-17T07:02:01.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:00 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T07:02:01.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:00 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/22018732' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-17T07:02:01.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:00 smithi135 ceph-mon[33238]: osdmap e61: 8 total, 8 up, 8 in 2024-09-17T07:02:01.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:00 smithi135 ceph-mon[33238]: pgmap v172: 1 pgs: 1 remapped+peering; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-17T07:02:01.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:00 smithi135 ceph-mon[33238]: Detected new or changed devices on smithi135 2024-09-17T07:02:01.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:00 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:01.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:00 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:01.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:00 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:02:01.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:00 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:02:01.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:00 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:02:01.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:00 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:02:01.295 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:00 smithi135 ceph-mon[33238]: Adjusting osd_memory_target on smithi135 to 1453M 2024-09-17T07:02:01.295 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:00 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:01.295 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:00 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:02:01.295 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:00 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:02:01.295 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:00 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:02.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:02 smithi135 ceph-mon[33238]: osdmap e62: 8 total, 8 up, 8 in 2024-09-17T07:02:02.313 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:02 smithi100 ceph-mon[29092]: osdmap e62: 8 total, 8 up, 8 in 2024-09-17T07:02:02.313 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:02 smithi100 ceph-mon[39462]: osdmap e62: 8 total, 8 up, 8 in 2024-09-17T07:02:02.714 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T07:02:02.714 INFO:teuthology.orchestra.run.smithi100.stdout:{"epoch":63,"fsid":"e848267c-74c1-11ef-bceb-c7b262605968","created":"2024-09-17T06:57:05.010532+0000","modified":"2024-09-17T07:02:02.004844+0000","last_up_change":"2024-09-17T07:01:59.917118+0000","last_in_change":"2024-09-17T07:01:42.942454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T07:00:08.816136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"27bee9ec-05e6-406c-8bf1-168098da0ec7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6801","nonce":3341017455}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":3341017455}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":3341017455}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6803","nonce":3341017455}]},"public_addr":"172.21.15.100:6801/3341017455","cluster_addr":"172.21.15.100:6802/3341017455","heartbeat_back_addr":"172.21.15.100:6804/3341017455","heartbeat_front_addr":"172.21.15.100:6803/3341017455","state":["exists","up"]},{"osd":1,"uuid":"b1a30c9c-eeaf-478e-9b62-2d63ad45873d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6805","nonce":1634750965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":1634750965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":1634750965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6807","nonce":1634750965}]},"public_addr":"172.21.15.100:6805/1634750965","cluster_addr":"172.21.15.100:6806/1634750965","heartbeat_back_addr":"172.21.15.100:6808/1634750965","heartbeat_front_addr":"172.21.15.100:6807/1634750965","state":["exists","up"]},{"osd":2,"uuid":"ce542e0d-9915-465e-aa11-7b3ce13f694f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":714579846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":714579846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":714579846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":714579846}]},"public_addr":"172.21.15.100:6809/714579846","cluster_addr":"172.21.15.100:6810/714579846","heartbeat_back_addr":"172.21.15.100:6812/714579846","heartbeat_front_addr":"172.21.15.100:6811/714579846","state":["exists","up"]},{"osd":3,"uuid":"a1257575-3c90-41de-88f6-3960744351b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6813","nonce":2315956572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":2315956572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":2315956572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":2315956572}]},"public_addr":"172.21.15.100:6813/2315956572","cluster_addr":"172.21.15.100:6814/2315956572","heartbeat_back_addr":"172.21.15.100:6816/2315956572","heartbeat_front_addr":"172.21.15.100:6815/2315956572","state":["exists","up"]},{"osd":4,"uuid":"e44e8922-2c5e-44e3-b556-d4dfbc499649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":1148820922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6801","nonce":1148820922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6803","nonce":1148820922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":1148820922}]},"public_addr":"172.21.15.135:6800/1148820922","cluster_addr":"172.21.15.135:6801/1148820922","heartbeat_back_addr":"172.21.15.135:6803/1148820922","heartbeat_front_addr":"172.21.15.135:6802/1148820922","state":["exists","up"]},{"osd":5,"uuid":"91e4a362-fcdb-41b9-b54e-716591ba6d79","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2256520893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6805","nonce":2256520893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6807","nonce":2256520893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2256520893}]},"public_addr":"172.21.15.135:6804/2256520893","cluster_addr":"172.21.15.135:6805/2256520893","heartbeat_back_addr":"172.21.15.135:6807/2256520893","heartbeat_front_addr":"172.21.15.135:6806/2256520893","state":["exists","up"]},{"osd":6,"uuid":"de799f64-ebd2-4a46-91f6-c2f276bee21f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":2750184152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6809","nonce":2750184152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6811","nonce":2750184152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":2750184152}]},"public_addr":"172.21.15.135:6808/2750184152","cluster_addr":"172.21.15.135:6809/2750184152","heartbeat_back_addr":"172.21.15.135:6811/2750184152","heartbeat_front_addr":"172.21.15.135:6810/2750184152","state":["exists","up"]},{"osd":7,"uuid":"830932c7-5de1-44ff-a840-e53a6e859bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":160304773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6813","nonce":160304773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6815","nonce":160304773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":160304773}]},"public_addr":"172.21.15.135:6812/160304773","cluster_addr":"172.21.15.135:6813/160304773","heartbeat_back_addr":"172.21.15.135:6815/160304773","heartbeat_front_addr":"172.21.15.135:6814/160304773","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:01:51.772595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:00:52.872374+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:01:06.311523+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:01:49.961971+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:00:49.588234+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:01:12.834256+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:01:35.329321+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:01:57.956813+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.100:0/1657291976":"2024-09-18T06:57:58.526309+0000","172.21.15.100:0/1365215411":"2024-09-18T06:57:58.526309+0000","172.21.15.100:6800/3992296558":"2024-09-18T06:57:58.526309+0000","172.21.15.100:0/3452172206":"2024-09-18T06:57:35.678309+0000","172.21.15.100:0/1626706829":"2024-09-18T06:57:35.678309+0000","172.21.15.100:0/1255197255":"2024-09-18T06:57:58.526309+0000","172.21.15.100:6800/3498574214":"2024-09-18T06:57:35.678309+0000","172.21.15.100:0/3796664909":"2024-09-18T06:57:35.678309+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T07:02:03.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:03 smithi135 ceph-mon[33238]: osdmap e63: 8 total, 8 up, 8 in 2024-09-17T07:02:03.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:03 smithi135 ceph-mon[33238]: pgmap v175: 1 pgs: 1 remapped+peering; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:02:03.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:03 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2898120224' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-17T07:02:03.376 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:03 smithi100 ceph-mon[39462]: osdmap e63: 8 total, 8 up, 8 in 2024-09-17T07:02:03.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:03 smithi100 ceph-mon[39462]: pgmap v175: 1 pgs: 1 remapped+peering; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:02:03.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:03 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2898120224' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-17T07:02:03.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:03 smithi100 ceph-mon[29092]: osdmap e63: 8 total, 8 up, 8 in 2024-09-17T07:02:03.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:03 smithi100 ceph-mon[29092]: pgmap v175: 1 pgs: 1 remapped+peering; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:02:03.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:03 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2898120224' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-17T07:02:03.441 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-17T07:00:08.816136+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-17T07:02:03.441 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-09-17T07:02:03.640 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:02:04.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:04 smithi135 ceph-mon[33238]: osdmap e64: 8 total, 8 up, 8 in 2024-09-17T07:02:04.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:04 smithi100 ceph-mon[29092]: osdmap e64: 8 total, 8 up, 8 in 2024-09-17T07:02:04.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:04 smithi100 ceph-mon[39462]: osdmap e64: 8 total, 8 up, 8 in 2024-09-17T07:02:05.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:05 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:02:05.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:05 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:02:05.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:05 smithi135 ceph-mon[33238]: pgmap v177: 1 pgs: 1 remapped+peering; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:02:05.343 INFO:teuthology.orchestra.run.smithi100.stdout:pg_num: 1 2024-09-17T07:02:05.343 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:05 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:02:05.343 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:05 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:02:05.344 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:05 smithi100 ceph-mon[29092]: pgmap v177: 1 pgs: 1 remapped+peering; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:02:05.344 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:05 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:02:05.344 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:05 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:02:05.344 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:05 smithi100 ceph-mon[39462]: pgmap v177: 1 pgs: 1 remapped+peering; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:02:06.087 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi100 2024-09-17T07:02:06.088 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph orch apply rgw foo.a --placement '1;smithi100=foo.a' 2024-09-17T07:02:06.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:06 smithi135 ceph-mon[33238]: osdmap e65: 8 total, 8 up, 8 in 2024-09-17T07:02:06.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:06 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/577847611' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-17T07:02:06.298 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:02:06.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:06 smithi100 ceph-mon[29092]: osdmap e65: 8 total, 8 up, 8 in 2024-09-17T07:02:06.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:06 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/577847611' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-17T07:02:06.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:06 smithi100 ceph-mon[39462]: osdmap e65: 8 total, 8 up, 8 in 2024-09-17T07:02:06.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:06 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/577847611' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-17T07:02:07.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:07 smithi135 ceph-mon[33238]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-09-17T07:02:07.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:07 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:02:07.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:07 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:02:07.376 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:07 smithi100 ceph-mon[39462]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-09-17T07:02:07.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:07 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:02:07.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:07 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:02:07.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:07 smithi100 ceph-mon[29092]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-09-17T07:02:07.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:07 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:02:07.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:07 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:02:08.044 INFO:teuthology.orchestra.run.smithi135.stdout:Scheduled rgw.foo.a update... 2024-09-17T07:02:08.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:08 smithi135 ceph-mon[33238]: osdmap e66: 8 total, 8 up, 8 in 2024-09-17T07:02:08.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:08 smithi100 ceph-mon[29092]: osdmap e66: 8 total, 8 up, 8 in 2024-09-17T07:02:08.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:08 smithi100 ceph-mon[39462]: osdmap e66: 8 total, 8 up, 8 in 2024-09-17T07:02:08.642 DEBUG:teuthology.orchestra.run.smithi100:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-e848267c-74c1-11ef-bceb-c7b262605968@rgw.foo.a.service 2024-09-17T07:02:08.645 INFO:tasks.cephadm:Adding ceph.iscsi.iscsi.a on smithi135 2024-09-17T07:02:08.645 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd pool create datapool 3 3 replicated 2024-09-17T07:02:08.848 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:02:09.266 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:09 smithi135 ceph-mon[33238]: from='client.24304 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi100=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:02:09.267 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:09 smithi135 ceph-mon[33238]: Saving service rgw.foo.a spec with placement smithi100=foo.a;count:1 2024-09-17T07:02:09.267 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:09 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:09.267 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:09 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:02:09.267 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:09 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:02:09.267 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:09 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:02:09.267 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:09 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:09.267 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:09 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T07:02:09.267 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:09 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-17T07:02:09.267 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:09 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:09.267 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:09 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:02:09.267 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:09 smithi135 ceph-mon[33238]: Deploying daemon rgw.foo.a on smithi100 2024-09-17T07:02:09.267 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:09 smithi135 ceph-mon[33238]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-17T07:02:09.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:09 smithi100 ceph-mon[39462]: from='client.24304 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi100=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:02:09.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:09 smithi100 ceph-mon[39462]: Saving service rgw.foo.a spec with placement smithi100=foo.a;count:1 2024-09-17T07:02:09.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:09 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:09.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:09 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:02:09.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:09 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:02:09.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:09 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:02:09.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:09 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:09.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:09 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T07:02:09.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:09 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-17T07:02:09.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:09 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:09.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:09 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:02:09.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:09 smithi100 ceph-mon[39462]: Deploying daemon rgw.foo.a on smithi100 2024-09-17T07:02:09.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:09 smithi100 ceph-mon[39462]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-17T07:02:09.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:09 smithi100 ceph-mon[29092]: from='client.24304 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi100=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:02:09.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:09 smithi100 ceph-mon[29092]: Saving service rgw.foo.a spec with placement smithi100=foo.a;count:1 2024-09-17T07:02:09.379 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:09 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:09.379 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:09 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:02:09.379 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:09 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:02:09.379 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:09 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:02:09.379 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:09 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:09.379 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:09 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T07:02:09.379 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:09 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-17T07:02:09.379 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:09 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:09.379 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:09 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:02:09.379 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:09 smithi100 ceph-mon[29092]: Deploying daemon rgw.foo.a on smithi100 2024-09-17T07:02:09.379 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:09 smithi100 ceph-mon[29092]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-17T07:02:10.074 INFO:journalctl@ceph.rgw.foo.a.smithi100.stdout:Sep 17 07:02:10 smithi100 systemd[1]: Starting Ceph rgw.foo.a for e848267c-74c1-11ef-bceb-c7b262605968... 2024-09-17T07:02:10.367 INFO:journalctl@ceph.rgw.foo.a.smithi100.stdout:Sep 17 07:02:10 smithi100 podman[65517]: 2024-09-17 07:02:10.136769211 +0000 UTC m=+0.015693133 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9 2024-09-17T07:02:10.368 INFO:journalctl@ceph.rgw.foo.a.smithi100.stdout:Sep 17 07:02:10 smithi100 podman[65517]: 2024-09-17 07:02:10.257788396 +0000 UTC m=+0.136712349 container create 8565754fa086f74ddf53618372b08689e63730ec7115ccb8c4ccb1b146b21de1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-rgw-foo-a, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, RELEASE=reef-19e2a2b, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, ceph=True) 2024-09-17T07:02:10.876 INFO:journalctl@ceph.rgw.foo.a.smithi100.stdout:Sep 17 07:02:10 smithi100 podman[65517]: 2024-09-17 07:02:10.485515041 +0000 UTC m=+0.364438956 container init 8565754fa086f74ddf53618372b08689e63730ec7115ccb8c4ccb1b146b21de1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-rgw-foo-a, ceph=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-19e2a2b, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux ) 2024-09-17T07:02:10.877 INFO:journalctl@ceph.rgw.foo.a.smithi100.stdout:Sep 17 07:02:10 smithi100 podman[65517]: 2024-09-17 07:02:10.488274461 +0000 UTC m=+0.367198370 container start 8565754fa086f74ddf53618372b08689e63730ec7115ccb8c4ccb1b146b21de1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-rgw-foo-a, GIT_CLEAN=True, RELEASE=reef-19e2a2b, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=) 2024-09-17T07:02:10.877 INFO:journalctl@ceph.rgw.foo.a.smithi100.stdout:Sep 17 07:02:10 smithi100 bash[65517]: 8565754fa086f74ddf53618372b08689e63730ec7115ccb8c4ccb1b146b21de1 2024-09-17T07:02:10.877 INFO:journalctl@ceph.rgw.foo.a.smithi100.stdout:Sep 17 07:02:10 smithi100 systemd[1]: Started Ceph rgw.foo.a for e848267c-74c1-11ef-bceb-c7b262605968. 2024-09-17T07:02:11.000 INFO:teuthology.orchestra.run.smithi135.stderr:pool 'datapool' created 2024-09-17T07:02:11.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:11 smithi135 ceph-mon[33238]: from='client.? 172.21.15.135:0/1141647998' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-09-17T07:02:11.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:11 smithi100 ceph-mon[29092]: from='client.? 172.21.15.135:0/1141647998' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-09-17T07:02:11.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:11 smithi100 ceph-mon[39462]: from='client.? 172.21.15.135:0/1141647998' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-09-17T07:02:11.624 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- rbd pool init datapool 2024-09-17T07:02:11.821 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:02:12.247 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:12 smithi135 ceph-mon[33238]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-09-17T07:02:12.247 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:12 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-09-17T07:02:12.247 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:12 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-09-17T07:02:12.247 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:12 smithi135 ceph-mon[33238]: osdmap e67: 8 total, 8 up, 8 in 2024-09-17T07:02:12.248 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:12 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1969590951' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-17T07:02:12.248 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:12 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:12.248 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:12 smithi135 ceph-mon[33238]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-17T07:02:12.248 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:12 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:12.248 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:12 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:12.248 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:12 smithi135 ceph-mon[33238]: Saving service rgw.foo.a spec with placement smithi100=foo.a;count:1 2024-09-17T07:02:12.248 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:12 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:12.248 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:12 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:12.248 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:12 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:02:12.248 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:12 smithi135 ceph-mon[33238]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-17T07:02:12.248 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:12 smithi135 ceph-mon[33238]: osdmap e68: 8 total, 8 up, 8 in 2024-09-17T07:02:12.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:12 smithi100 ceph-mon[29092]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-09-17T07:02:12.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:12 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-09-17T07:02:12.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:12 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-09-17T07:02:12.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:12 smithi100 ceph-mon[29092]: osdmap e67: 8 total, 8 up, 8 in 2024-09-17T07:02:12.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:12 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1969590951' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-17T07:02:12.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:12 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:12.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:12 smithi100 ceph-mon[29092]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-17T07:02:12.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:12 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:12.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:12 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:12.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:12 smithi100 ceph-mon[29092]: Saving service rgw.foo.a spec with placement smithi100=foo.a;count:1 2024-09-17T07:02:12.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:12 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:12.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:12 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:12.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:12 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:02:12.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:12 smithi100 ceph-mon[29092]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-17T07:02:12.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:12 smithi100 ceph-mon[29092]: osdmap e68: 8 total, 8 up, 8 in 2024-09-17T07:02:12.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:12 smithi100 ceph-mon[39462]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-09-17T07:02:12.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:12 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-09-17T07:02:12.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:12 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-09-17T07:02:12.379 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:12 smithi100 ceph-mon[39462]: osdmap e67: 8 total, 8 up, 8 in 2024-09-17T07:02:12.379 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:12 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1969590951' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-17T07:02:12.379 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:12 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:12.379 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:12 smithi100 ceph-mon[39462]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-17T07:02:12.379 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:12 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:12.379 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:12 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:12.379 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:12 smithi100 ceph-mon[39462]: Saving service rgw.foo.a spec with placement smithi100=foo.a;count:1 2024-09-17T07:02:12.379 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:12 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:12.379 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:12 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:12.379 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:12 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:02:12.379 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:12 smithi100 ceph-mon[39462]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-17T07:02:12.379 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:12 smithi100 ceph-mon[39462]: osdmap e68: 8 total, 8 up, 8 in 2024-09-17T07:02:13.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:13 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:13.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:13 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:13.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:13 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:02:13.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:13 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:02:13.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:13 smithi135 ceph-mon[33238]: Checking dashboard <-> RGW credentials 2024-09-17T07:02:13.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:13 smithi135 ceph-mon[33238]: pgmap v185: 36 pgs: 5 creating+peering, 26 unknown, 5 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 170 B/s wr, 0 op/s 2024-09-17T07:02:13.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:13 smithi135 ceph-mon[33238]: osdmap e69: 8 total, 8 up, 8 in 2024-09-17T07:02:13.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:13 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1969590951' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-17T07:02:13.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:13 smithi135 ceph-mon[33238]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-17T07:02:13.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:13 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2889069731' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-17T07:02:13.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:13 smithi135 ceph-mon[33238]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-17T07:02:13.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:13 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:13.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:13 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:13.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:13 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:02:13.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:13 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:02:13.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:13 smithi100 ceph-mon[29092]: Checking dashboard <-> RGW credentials 2024-09-17T07:02:13.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:13 smithi100 ceph-mon[29092]: pgmap v185: 36 pgs: 5 creating+peering, 26 unknown, 5 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 170 B/s wr, 0 op/s 2024-09-17T07:02:13.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:13 smithi100 ceph-mon[29092]: osdmap e69: 8 total, 8 up, 8 in 2024-09-17T07:02:13.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:13 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1969590951' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-17T07:02:13.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:13 smithi100 ceph-mon[29092]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-17T07:02:13.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:13 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2889069731' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-17T07:02:13.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:13 smithi100 ceph-mon[29092]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-17T07:02:13.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:13 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:13.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:13 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:13.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:13 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:02:13.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:13 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:02:13.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:13 smithi100 ceph-mon[39462]: Checking dashboard <-> RGW credentials 2024-09-17T07:02:13.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:13 smithi100 ceph-mon[39462]: pgmap v185: 36 pgs: 5 creating+peering, 26 unknown, 5 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 170 B/s wr, 0 op/s 2024-09-17T07:02:13.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:13 smithi100 ceph-mon[39462]: osdmap e69: 8 total, 8 up, 8 in 2024-09-17T07:02:13.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:13 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1969590951' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-17T07:02:13.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:13 smithi100 ceph-mon[39462]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-17T07:02:13.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:13 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2889069731' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-17T07:02:13.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:13 smithi100 ceph-mon[39462]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-17T07:02:14.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:14 smithi135 ceph-mon[33238]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-17T07:02:14.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:14 smithi135 ceph-mon[33238]: from='client.? 172.21.15.135:0/105367750' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-09-17T07:02:14.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:14 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-09-17T07:02:14.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:14 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:14.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:14 smithi135 ceph-mon[33238]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-17T07:02:14.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:14 smithi135 ceph-mon[33238]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-17T07:02:14.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:14 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-09-17T07:02:14.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:14 smithi135 ceph-mon[33238]: osdmap e70: 8 total, 8 up, 8 in 2024-09-17T07:02:14.876 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:14 smithi100 ceph-mon[39462]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-17T07:02:14.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:14 smithi100 ceph-mon[39462]: from='client.? 172.21.15.135:0/105367750' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-09-17T07:02:14.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:14 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-09-17T07:02:14.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:14 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:14.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:14 smithi100 ceph-mon[39462]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-17T07:02:14.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:14 smithi100 ceph-mon[39462]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-17T07:02:14.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:14 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-09-17T07:02:14.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:14 smithi100 ceph-mon[39462]: osdmap e70: 8 total, 8 up, 8 in 2024-09-17T07:02:14.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:14 smithi100 ceph-mon[29092]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-17T07:02:14.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:14 smithi100 ceph-mon[29092]: from='client.? 172.21.15.135:0/105367750' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-09-17T07:02:14.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:14 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-09-17T07:02:14.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:14 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:14.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:14 smithi100 ceph-mon[29092]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-17T07:02:14.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:14 smithi100 ceph-mon[29092]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-17T07:02:14.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:14 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-09-17T07:02:14.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:14 smithi100 ceph-mon[29092]: osdmap e70: 8 total, 8 up, 8 in 2024-09-17T07:02:15.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:15 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:02:15.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:15 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:02:15.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:15 smithi135 ceph-mon[33238]: pgmap v188: 68 pgs: 13 creating+peering, 38 unknown, 17 active+clean; 578 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 511 B/s wr, 1 op/s 2024-09-17T07:02:15.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:15 smithi135 ceph-mon[33238]: osdmap e71: 8 total, 8 up, 8 in 2024-09-17T07:02:15.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:15 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2889069731' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-17T07:02:15.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:15 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1969590951' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-17T07:02:15.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:15 smithi135 ceph-mon[33238]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-17T07:02:15.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:15 smithi135 ceph-mon[33238]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-17T07:02:15.876 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:15 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:02:15.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:15 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:02:15.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:15 smithi100 ceph-mon[39462]: pgmap v188: 68 pgs: 13 creating+peering, 38 unknown, 17 active+clean; 578 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 511 B/s wr, 1 op/s 2024-09-17T07:02:15.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:15 smithi100 ceph-mon[39462]: osdmap e71: 8 total, 8 up, 8 in 2024-09-17T07:02:15.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:15 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2889069731' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-17T07:02:15.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:15 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1969590951' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-17T07:02:15.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:15 smithi100 ceph-mon[39462]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-17T07:02:15.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:15 smithi100 ceph-mon[39462]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-17T07:02:15.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:15 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:02:15.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:15 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:02:15.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:15 smithi100 ceph-mon[29092]: pgmap v188: 68 pgs: 13 creating+peering, 38 unknown, 17 active+clean; 578 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 511 B/s wr, 1 op/s 2024-09-17T07:02:15.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:15 smithi100 ceph-mon[29092]: osdmap e71: 8 total, 8 up, 8 in 2024-09-17T07:02:15.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:15 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2889069731' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-17T07:02:15.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:15 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1969590951' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-17T07:02:15.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:15 smithi100 ceph-mon[29092]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-17T07:02:15.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:15 smithi100 ceph-mon[29092]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-17T07:02:16.574 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph orch apply iscsi datapool admin admin --trusted_ip_list 172.21.15.135 --placement '1;smithi135=iscsi.a' 2024-09-17T07:02:16.597 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:16 smithi135 ceph-mon[33238]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-17T07:02:16.597 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:16 smithi135 ceph-mon[33238]: Cluster is now healthy 2024-09-17T07:02:16.597 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:16 smithi135 ceph-mon[33238]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-17T07:02:16.597 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:16 smithi135 ceph-mon[33238]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-17T07:02:16.597 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:16 smithi135 ceph-mon[33238]: osdmap e72: 8 total, 8 up, 8 in 2024-09-17T07:02:16.768 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:02:16.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:16 smithi100 ceph-mon[29092]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-17T07:02:16.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:16 smithi100 ceph-mon[29092]: Cluster is now healthy 2024-09-17T07:02:16.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:16 smithi100 ceph-mon[29092]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-17T07:02:16.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:16 smithi100 ceph-mon[29092]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-17T07:02:16.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:16 smithi100 ceph-mon[29092]: osdmap e72: 8 total, 8 up, 8 in 2024-09-17T07:02:16.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:16 smithi100 ceph-mon[39462]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-17T07:02:16.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:16 smithi100 ceph-mon[39462]: Cluster is now healthy 2024-09-17T07:02:16.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:16 smithi100 ceph-mon[39462]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-17T07:02:16.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:16 smithi100 ceph-mon[39462]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-17T07:02:16.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:16 smithi100 ceph-mon[39462]: osdmap e72: 8 total, 8 up, 8 in 2024-09-17T07:02:17.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:17 smithi100 ceph-mon[29092]: pgmap v191: 100 pgs: 13 creating+peering, 34 unknown, 53 active+clean; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 767 B/s wr, 1 op/s 2024-09-17T07:02:17.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:17 smithi100 ceph-mon[29092]: osdmap e73: 8 total, 8 up, 8 in 2024-09-17T07:02:17.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:17 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1436748426' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-17T07:02:17.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:17 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1334829920' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-17T07:02:17.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:17 smithi100 ceph-mon[29092]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-17T07:02:17.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:17 smithi100 ceph-mon[29092]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-17T07:02:17.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:17 smithi100 ceph-mon[39462]: pgmap v191: 100 pgs: 13 creating+peering, 34 unknown, 53 active+clean; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 767 B/s wr, 1 op/s 2024-09-17T07:02:17.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:17 smithi100 ceph-mon[39462]: osdmap e73: 8 total, 8 up, 8 in 2024-09-17T07:02:17.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:17 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1436748426' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-17T07:02:17.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:17 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1334829920' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-17T07:02:17.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:17 smithi100 ceph-mon[39462]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-17T07:02:17.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:17 smithi100 ceph-mon[39462]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-17T07:02:17.936 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:17 smithi135 ceph-mon[33238]: pgmap v191: 100 pgs: 13 creating+peering, 34 unknown, 53 active+clean; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 767 B/s wr, 1 op/s 2024-09-17T07:02:17.936 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:17 smithi135 ceph-mon[33238]: osdmap e73: 8 total, 8 up, 8 in 2024-09-17T07:02:17.936 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:17 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1436748426' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-17T07:02:17.936 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:17 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1334829920' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-17T07:02:17.936 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:17 smithi135 ceph-mon[33238]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-17T07:02:17.936 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:17 smithi135 ceph-mon[33238]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-17T07:02:18.467 INFO:teuthology.orchestra.run.smithi135.stdout:Scheduled iscsi.datapool update... 2024-09-17T07:02:19.065 INFO:tasks.cephadm:Distributing iscsi-gateway.cfg... 2024-09-17T07:02:19.065 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-17T07:02:19.065 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-09-17T07:02:19.106 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-09-17T07:02:19.106 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-09-17T07:02:19.137 DEBUG:teuthology.orchestra.run.smithi135:iscsi.iscsi.a> sudo journalctl -f -n 0 -u ceph-e848267c-74c1-11ef-bceb-c7b262605968@iscsi.iscsi.a.service 2024-09-17T07:02:19.180 INFO:tasks.cephadm:Adding prometheus.a on smithi135 2024-09-17T07:02:19.180 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph orch apply prometheus '1;smithi135=a' 2024-09-17T07:02:19.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:19 smithi135 ceph-mon[33238]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-17T07:02:19.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:19 smithi135 ceph-mon[33238]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-17T07:02:19.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:19 smithi135 ceph-mon[33238]: osdmap e74: 8 total, 8 up, 8 in 2024-09-17T07:02:19.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:19 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1334829920' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-17T07:02:19.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:19 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1436748426' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-17T07:02:19.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:19 smithi135 ceph-mon[33238]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-17T07:02:19.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:19 smithi135 ceph-mon[33238]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-17T07:02:19.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:19 smithi135 ceph-mon[33238]: from='client.24367 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.135", "placement": "1;smithi135=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:02:19.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:19 smithi135 ceph-mon[33238]: Saving service iscsi.datapool spec with placement smithi135=iscsi.a;count:1 2024-09-17T07:02:19.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:19 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:19.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:19 smithi135 ceph-mon[33238]: pgmap v194: 132 pgs: 11 creating+peering, 42 unknown, 79 active+clean; 578 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-09-17T07:02:19.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:19 smithi100 ceph-mon[29092]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-17T07:02:19.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:19 smithi100 ceph-mon[29092]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-17T07:02:19.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:19 smithi100 ceph-mon[29092]: osdmap e74: 8 total, 8 up, 8 in 2024-09-17T07:02:19.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:19 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1334829920' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-17T07:02:19.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:19 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1436748426' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-17T07:02:19.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:19 smithi100 ceph-mon[29092]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-17T07:02:19.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:19 smithi100 ceph-mon[29092]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-17T07:02:19.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:19 smithi100 ceph-mon[29092]: from='client.24367 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.135", "placement": "1;smithi135=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:02:19.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:19 smithi100 ceph-mon[29092]: Saving service iscsi.datapool spec with placement smithi135=iscsi.a;count:1 2024-09-17T07:02:19.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:19 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:19.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:19 smithi100 ceph-mon[29092]: pgmap v194: 132 pgs: 11 creating+peering, 42 unknown, 79 active+clean; 578 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-09-17T07:02:19.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:19 smithi100 ceph-mon[39462]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-17T07:02:19.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:19 smithi100 ceph-mon[39462]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-17T07:02:19.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:19 smithi100 ceph-mon[39462]: osdmap e74: 8 total, 8 up, 8 in 2024-09-17T07:02:19.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:19 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1334829920' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-17T07:02:19.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:19 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1436748426' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-17T07:02:19.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:19 smithi100 ceph-mon[39462]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-17T07:02:19.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:19 smithi100 ceph-mon[39462]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-17T07:02:19.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:19 smithi100 ceph-mon[39462]: from='client.24367 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.135", "placement": "1;smithi135=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:02:19.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:19 smithi100 ceph-mon[39462]: Saving service iscsi.datapool spec with placement smithi135=iscsi.a;count:1 2024-09-17T07:02:19.379 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:19 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:19.379 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:19 smithi100 ceph-mon[39462]: pgmap v194: 132 pgs: 11 creating+peering, 42 unknown, 79 active+clean; 578 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-09-17T07:02:19.466 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:02:20.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:20 smithi135 ceph-mon[33238]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-17T07:02:20.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:20 smithi135 ceph-mon[33238]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-17T07:02:20.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:20 smithi135 ceph-mon[33238]: osdmap e75: 8 total, 8 up, 8 in 2024-09-17T07:02:20.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:20 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:20.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:20 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:20.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:20 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:20.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:20 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T07:02:20.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:20 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-17T07:02:20.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:20 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:02:20.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:20 smithi135 ceph-mon[33238]: Deploying daemon iscsi.iscsi.a on smithi135 2024-09-17T07:02:20.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:20 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:02:20.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:20 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:02:20.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:20 smithi100 ceph-mon[29092]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-17T07:02:20.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:20 smithi100 ceph-mon[29092]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-17T07:02:20.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:20 smithi100 ceph-mon[29092]: osdmap e75: 8 total, 8 up, 8 in 2024-09-17T07:02:20.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:20 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:20.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:20 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:20.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:20 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:20.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:20 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T07:02:20.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:20 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-17T07:02:20.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:20 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:02:20.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:20 smithi100 ceph-mon[29092]: Deploying daemon iscsi.iscsi.a on smithi135 2024-09-17T07:02:20.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:20 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:02:20.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:20 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:02:20.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:20 smithi100 ceph-mon[39462]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-17T07:02:20.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:20 smithi100 ceph-mon[39462]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-17T07:02:20.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:20 smithi100 ceph-mon[39462]: osdmap e75: 8 total, 8 up, 8 in 2024-09-17T07:02:20.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:20 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:20.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:20 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:20.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:20 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:20.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:20 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T07:02:20.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:20 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-17T07:02:20.379 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:20 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:02:20.379 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:20 smithi100 ceph-mon[39462]: Deploying daemon iscsi.iscsi.a on smithi135 2024-09-17T07:02:20.379 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:20 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:02:20.379 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:20 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:02:21.497 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:21 smithi135 ceph-mon[33238]: osdmap e76: 8 total, 8 up, 8 in 2024-09-17T07:02:21.497 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:21 smithi135 ceph-mon[33238]: pgmap v197: 132 pgs: 6 creating+peering, 18 unknown, 108 active+clean; 579 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 3.0 KiB/s wr, 64 op/s 2024-09-17T07:02:21.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:21 smithi100 ceph-mon[29092]: osdmap e76: 8 total, 8 up, 8 in 2024-09-17T07:02:21.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:21 smithi100 ceph-mon[29092]: pgmap v197: 132 pgs: 6 creating+peering, 18 unknown, 108 active+clean; 579 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 3.0 KiB/s wr, 64 op/s 2024-09-17T07:02:21.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:21 smithi100 ceph-mon[39462]: osdmap e76: 8 total, 8 up, 8 in 2024-09-17T07:02:21.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:21 smithi100 ceph-mon[39462]: pgmap v197: 132 pgs: 6 creating+peering, 18 unknown, 108 active+clean; 579 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 3.0 KiB/s wr, 64 op/s 2024-09-17T07:02:21.711 INFO:teuthology.orchestra.run.smithi135.stdout:Scheduled prometheus update... 2024-09-17T07:02:22.543 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:02:22 smithi135 systemd[1]: Starting Ceph iscsi.iscsi.a for e848267c-74c1-11ef-bceb-c7b262605968... 2024-09-17T07:02:22.589 DEBUG:teuthology.orchestra.run.smithi135:prometheus.a> sudo journalctl -f -n 0 -u ceph-e848267c-74c1-11ef-bceb-c7b262605968@prometheus.a.service 2024-09-17T07:02:22.591 INFO:tasks.cephadm:Adding node-exporter.a on smithi100 2024-09-17T07:02:22.591 INFO:tasks.cephadm:Adding node-exporter.b on smithi135 2024-09-17T07:02:22.591 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph orch apply node-exporter '2;smithi100=a;smithi135=b' 2024-09-17T07:02:22.854 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:22 smithi135 ceph-mon[33238]: from='client.24391 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi135=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:02:22.854 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:22 smithi135 ceph-mon[33238]: Saving service prometheus spec with placement smithi135=a;count:1 2024-09-17T07:02:22.854 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:22 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:22.854 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:02:22 smithi135 podman[60001]: 2024-09-17 07:02:22.661826333 +0000 UTC m=+0.018685340 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9 2024-09-17T07:02:22.855 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:02:22 smithi135 podman[60001]: 2024-09-17 07:02:22.772100397 +0000 UTC m=+0.128959396 container create 35b9ba9902b7e6034955514f52ce826fbee28d14f6f870ebf3c2e046f69c9e78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a-tcmu, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-19e2a2b, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True) 2024-09-17T07:02:23.126 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:22 smithi100 ceph-mon[29092]: from='client.24391 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi135=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:02:23.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:22 smithi100 ceph-mon[29092]: Saving service prometheus spec with placement smithi135=a;count:1 2024-09-17T07:02:23.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:22 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:23.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:22 smithi100 ceph-mon[39462]: from='client.24391 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi135=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:02:23.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:22 smithi100 ceph-mon[39462]: Saving service prometheus spec with placement smithi135=a;count:1 2024-09-17T07:02:23.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:22 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:23.152 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:02:23.267 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:02:22 smithi135 podman[60001]: 2024-09-17 07:02:22.974323847 +0000 UTC m=+0.331182856 container init 35b9ba9902b7e6034955514f52ce826fbee28d14f6f870ebf3c2e046f69c9e78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a-tcmu, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-19e2a2b, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-09-17T07:02:23.267 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:02:22 smithi135 podman[60001]: 2024-09-17 07:02:22.978080379 +0000 UTC m=+0.334939378 container start 35b9ba9902b7e6034955514f52ce826fbee28d14f6f870ebf3c2e046f69c9e78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a-tcmu, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=reef-19e2a2b, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T07:02:23.267 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:02:22 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a-tcmu[60066]: log file path now is '/var/log/tcmu-runner.log' 2024-09-17T07:02:23.267 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:02:23 smithi135 bash[60001]: 35b9ba9902b7e6034955514f52ce826fbee28d14f6f870ebf3c2e046f69c9e78 2024-09-17T07:02:23.543 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:02:23 smithi135 podman[60133]: 2024-09-17 07:02:23.169640755 +0000 UTC m=+0.020217048 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9 2024-09-17T07:02:23.543 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:02:23 smithi135 podman[60133]: 2024-09-17 07:02:23.283546369 +0000 UTC m=+0.134122655 container create b2b2f40ee17fcbf0e0b047246e857af4c9f4c0f36e6aa577cc384023fb0c84cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a, maintainer=Guillaume Abrioux , ceph=True, RELEASE=reef-19e2a2b, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2) 2024-09-17T07:02:23.835 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:23 smithi135 ceph-mon[33238]: pgmap v198: 132 pgs: 132 active+clean; 581 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 4.7 KiB/s wr, 143 op/s 2024-09-17T07:02:23.835 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:02:23 smithi135 podman[60133]: 2024-09-17 07:02:23.679902814 +0000 UTC m=+0.530479106 container init b2b2f40ee17fcbf0e0b047246e857af4c9f4c0f36e6aa577cc384023fb0c84cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=reef-19e2a2b, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, ceph=True, io.buildah.version=1.37.2) 2024-09-17T07:02:23.835 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:02:23 smithi135 podman[60133]: 2024-09-17 07:02:23.683505101 +0000 UTC m=+0.534081385 container start b2b2f40ee17fcbf0e0b047246e857af4c9f4c0f36e6aa577cc384023fb0c84cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240731, RELEASE=reef-19e2a2b, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, GIT_CLEAN=True) 2024-09-17T07:02:23.835 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:02:23 smithi135 bash[60133]: b2b2f40ee17fcbf0e0b047246e857af4c9f4c0f36e6aa577cc384023fb0c84cb 2024-09-17T07:02:23.835 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:02:23 smithi135 systemd[1]: Started Ceph iscsi.iscsi.a for e848267c-74c1-11ef-bceb-c7b262605968. 2024-09-17T07:02:24.126 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:23 smithi100 ceph-mon[29092]: pgmap v198: 132 pgs: 132 active+clean; 581 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 4.7 KiB/s wr, 143 op/s 2024-09-17T07:02:24.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:23 smithi100 ceph-mon[39462]: pgmap v198: 132 pgs: 132 active+clean; 581 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 4.7 KiB/s wr, 143 op/s 2024-09-17T07:02:24.284 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:02:23 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug Started the configuration object watcher 2024-09-17T07:02:24.284 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:02:23 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug Checking for config object changes every 1s 2024-09-17T07:02:24.285 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:02:23 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug Processing osd blocklist entries for this node 2024-09-17T07:02:24.543 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:02:24 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug Reading the configuration object to update local LIO configuration 2024-09-17T07:02:24.544 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:02:24 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug Configuration does not have an entry for this host(smithi135.front.sepia.ceph.com) - nothing to define to LIO 2024-09-17T07:02:24.544 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:02:24 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: * Serving Flask app 'rbd-target-api' (lazy loading) 2024-09-17T07:02:24.544 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:02:24 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: * Environment: production 2024-09-17T07:02:24.544 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:02:24 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: WARNING: This is a development server. Do not use it in a production deployment. 2024-09-17T07:02:24.544 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:02:24 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: Use a production WSGI server instead. 2024-09-17T07:02:24.544 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:02:24 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: * Debug mode: off 2024-09-17T07:02:24.544 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:02:24 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug * Running on all addresses. 2024-09-17T07:02:24.544 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:02:24 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: WARNING: This is a development server. Do not use it in a production deployment. 2024-09-17T07:02:24.544 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:02:24 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: * Running on all addresses. 2024-09-17T07:02:24.544 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:02:24 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: WARNING: This is a development server. Do not use it in a production deployment. 2024-09-17T07:02:24.544 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:02:24 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug * Running on http://[fe80::ec4:7aff:fe88:8083]:5000/ (Press CTRL+C to quit) 2024-09-17T07:02:24.545 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:02:24 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: * Running on http://[fe80::ec4:7aff:fe88:8083]:5000/ (Press CTRL+C to quit) 2024-09-17T07:02:24.995 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:24 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:24.995 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:24 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:24.996 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:24 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:24.996 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:24 smithi135 ceph-mon[33238]: Checking pool "datapool" exists for service iscsi.datapool 2024-09-17T07:02:24.996 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:24 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:24.996 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:24 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:02:24.996 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:24 smithi135 ceph-mon[33238]: from='client.? 172.21.15.135:0/155857039' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-17T07:02:24.997 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:24 smithi135 ceph-mon[33238]: pgmap v199: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 5.6 KiB/s wr, 172 op/s 2024-09-17T07:02:25.126 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:24 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:25.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:24 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:25.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:24 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:25.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:24 smithi100 ceph-mon[29092]: Checking pool "datapool" exists for service iscsi.datapool 2024-09-17T07:02:25.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:24 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:25.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:24 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:02:25.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:24 smithi100 ceph-mon[29092]: from='client.? 172.21.15.135:0/155857039' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-17T07:02:25.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:24 smithi100 ceph-mon[29092]: pgmap v199: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 5.6 KiB/s wr, 172 op/s 2024-09-17T07:02:25.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:24 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:25.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:24 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:25.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:24 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:25.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:24 smithi100 ceph-mon[39462]: Checking pool "datapool" exists for service iscsi.datapool 2024-09-17T07:02:25.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:24 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:25.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:24 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:02:25.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:24 smithi100 ceph-mon[39462]: from='client.? 172.21.15.135:0/155857039' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-17T07:02:25.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:24 smithi100 ceph-mon[39462]: pgmap v199: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 5.6 KiB/s wr, 172 op/s 2024-09-17T07:02:25.371 INFO:teuthology.orchestra.run.smithi135.stdout:Scheduled node-exporter update... 2024-09-17T07:02:26.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:25 smithi100 ceph-mon[29092]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-17T07:02:26.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:25 smithi100 ceph-mon[29092]: osdmap e77: 8 total, 8 up, 8 in 2024-09-17T07:02:26.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:25 smithi100 ceph-mon[29092]: from='client.24427 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi100=a;smithi135=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:02:26.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:25 smithi100 ceph-mon[29092]: Saving service node-exporter spec with placement smithi100=a;smithi135=b;count:2 2024-09-17T07:02:26.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:25 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:26.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:25 smithi100 ceph-mon[39462]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-17T07:02:26.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:25 smithi100 ceph-mon[39462]: osdmap e77: 8 total, 8 up, 8 in 2024-09-17T07:02:26.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:25 smithi100 ceph-mon[39462]: from='client.24427 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi100=a;smithi135=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:02:26.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:25 smithi100 ceph-mon[39462]: Saving service node-exporter spec with placement smithi100=a;smithi135=b;count:2 2024-09-17T07:02:26.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:25 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:26.178 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:25 smithi135 ceph-mon[33238]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-17T07:02:26.178 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:25 smithi135 ceph-mon[33238]: osdmap e77: 8 total, 8 up, 8 in 2024-09-17T07:02:26.178 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:25 smithi135 ceph-mon[33238]: from='client.24427 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi100=a;smithi135=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:02:26.178 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:25 smithi135 ceph-mon[33238]: Saving service node-exporter spec with placement smithi100=a;smithi135=b;count:2 2024-09-17T07:02:26.178 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:25 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:26.199 DEBUG:teuthology.orchestra.run.smithi100:node-exporter.a> sudo journalctl -f -n 0 -u ceph-e848267c-74c1-11ef-bceb-c7b262605968@node-exporter.a.service 2024-09-17T07:02:26.202 DEBUG:teuthology.orchestra.run.smithi135:node-exporter.b> sudo journalctl -f -n 0 -u ceph-e848267c-74c1-11ef-bceb-c7b262605968@node-exporter.b.service 2024-09-17T07:02:26.204 INFO:tasks.cephadm:Adding alertmanager.a on smithi100 2024-09-17T07:02:26.205 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph orch apply alertmanager '1;smithi100=a' 2024-09-17T07:02:27.083 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:02:27.126 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:26 smithi100 ceph-mon[29092]: pgmap v201: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 4.6 KiB/s wr, 147 op/s 2024-09-17T07:02:27.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:26 smithi100 ceph-mon[39462]: pgmap v201: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 4.6 KiB/s wr, 147 op/s 2024-09-17T07:02:27.204 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:26 smithi135 ceph-mon[33238]: pgmap v201: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 4.6 KiB/s wr, 147 op/s 2024-09-17T07:02:28.257 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:27 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:28.257 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:27 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:28.257 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:27 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:02:28.257 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:27 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:02:28.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:27 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:28.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:27 smithi135 ceph-mon[33238]: Deploying daemon prometheus.a on smithi135 2024-09-17T07:02:28.285 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:27 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:28.285 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:27 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:28.285 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:27 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:02:28.285 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:27 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:02:28.285 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:27 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:28.285 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:27 smithi100 ceph-mon[29092]: Deploying daemon prometheus.a on smithi135 2024-09-17T07:02:28.285 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:27 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:28.285 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:27 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:28.285 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:27 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:02:28.286 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:27 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:02:28.286 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:27 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:28.286 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:27 smithi100 ceph-mon[39462]: Deploying daemon prometheus.a on smithi135 2024-09-17T07:02:28.808 INFO:teuthology.orchestra.run.smithi135.stdout:Scheduled alertmanager update... 2024-09-17T07:02:29.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:29 smithi135 ceph-mon[33238]: pgmap v202: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s rd, 3.2 KiB/s wr, 105 op/s 2024-09-17T07:02:29.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:29 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:29.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:29 smithi135 ceph-mon[33238]: from='client.24433 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi100=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:02:29.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:29 smithi135 ceph-mon[33238]: Saving service alertmanager spec with placement smithi100=a;count:1 2024-09-17T07:02:29.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:29 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:29.863 DEBUG:teuthology.orchestra.run.smithi100:alertmanager.a> sudo journalctl -f -n 0 -u ceph-e848267c-74c1-11ef-bceb-c7b262605968@alertmanager.a.service 2024-09-17T07:02:29.866 INFO:tasks.cephadm:Adding grafana.a on smithi135 2024-09-17T07:02:29.866 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph orch apply grafana '1;smithi135=a' 2024-09-17T07:02:29.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:29 smithi100 ceph-mon[29092]: pgmap v202: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s rd, 3.2 KiB/s wr, 105 op/s 2024-09-17T07:02:29.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:29 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:29.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:29 smithi100 ceph-mon[29092]: from='client.24433 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi100=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:02:29.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:29 smithi100 ceph-mon[29092]: Saving service alertmanager spec with placement smithi100=a;count:1 2024-09-17T07:02:29.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:29 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:29.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:29 smithi100 ceph-mon[39462]: pgmap v202: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s rd, 3.2 KiB/s wr, 105 op/s 2024-09-17T07:02:29.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:29 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:29.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:29 smithi100 ceph-mon[39462]: from='client.24433 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi100=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:02:29.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:29 smithi100 ceph-mon[39462]: Saving service alertmanager spec with placement smithi100=a;count:1 2024-09-17T07:02:29.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:29 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:30.100 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:02:31.279 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:31 smithi135 ceph-mon[33238]: pgmap v203: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 2.7 KiB/s wr, 88 op/s 2024-09-17T07:02:31.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:31 smithi100 ceph-mon[29092]: pgmap v203: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 2.7 KiB/s wr, 88 op/s 2024-09-17T07:02:31.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:31 smithi100 ceph-mon[39462]: pgmap v203: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 2.7 KiB/s wr, 88 op/s 2024-09-17T07:02:32.498 INFO:teuthology.orchestra.run.smithi135.stdout:Scheduled grafana update... 2024-09-17T07:02:33.438 DEBUG:teuthology.orchestra.run.smithi135:grafana.a> sudo journalctl -f -n 0 -u ceph-e848267c-74c1-11ef-bceb-c7b262605968@grafana.a.service 2024-09-17T07:02:33.441 INFO:tasks.cephadm:Setting up client nodes... 2024-09-17T07:02:33.441 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-17T07:02:33.649 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:02:33.769 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:33 smithi100 ceph-mon[39462]: from='client.24439 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi135=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:02:33.769 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:33 smithi100 ceph-mon[39462]: Saving service grafana spec with placement smithi135=a;count:1 2024-09-17T07:02:33.769 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:33 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:33.770 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:33 smithi100 ceph-mon[39462]: pgmap v204: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 1.1 KiB/s wr, 33 op/s 2024-09-17T07:02:33.770 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:33 smithi100 ceph-mon[29092]: from='client.24439 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi135=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:02:33.770 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:33 smithi100 ceph-mon[29092]: Saving service grafana spec with placement smithi135=a;count:1 2024-09-17T07:02:33.770 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:33 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:33.770 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:33 smithi100 ceph-mon[29092]: pgmap v204: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 1.1 KiB/s wr, 33 op/s 2024-09-17T07:02:33.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:33 smithi135 ceph-mon[33238]: from='client.24439 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi135=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:02:33.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:33 smithi135 ceph-mon[33238]: Saving service grafana spec with placement smithi135=a;count:1 2024-09-17T07:02:33.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:33 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:33.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:33 smithi135 ceph-mon[33238]: pgmap v204: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 1.1 KiB/s wr, 33 op/s 2024-09-17T07:02:34.293 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:02:34 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:02:35.685 INFO:teuthology.orchestra.run.smithi100.stdout:[client.0] 2024-09-17T07:02:35.685 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQCLKelmCK1uKBAAZkmnf5/hGV+T1hVzsB3NWw== 2024-09-17T07:02:36.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:35 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:02:36.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:35 smithi135 ceph-mon[33238]: pgmap v205: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-09-17T07:02:36.107 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:35 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:02:36.108 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:35 smithi100 ceph-mon[39462]: pgmap v205: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-09-17T07:02:36.108 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:35 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:02:36.108 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:35 smithi100 ceph-mon[29092]: pgmap v205: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-09-17T07:02:36.426 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-17T07:02:36.426 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-09-17T07:02:36.426 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-17T07:02:36.464 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-17T07:02:36.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:36 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/912228491' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-17T07:02:36.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:36 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/912228491' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-17T07:02:36.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:36 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/912228491' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-17T07:02:36.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:36 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/912228491' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-17T07:02:36.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:36 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/912228491' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-17T07:02:36.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:36 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/912228491' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-17T07:02:37.966 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:37 smithi135 ceph-mon[33238]: pgmap v206: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 796 B/s rd, 0 op/s 2024-09-17T07:02:38.126 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:37 smithi100 ceph-mon[29092]: pgmap v206: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 796 B/s rd, 0 op/s 2024-09-17T07:02:38.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:37 smithi100 ceph-mon[39462]: pgmap v206: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 796 B/s rd, 0 op/s 2024-09-17T07:02:39.082 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:02:40.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:39 smithi135 ceph-mon[33238]: pgmap v207: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-17T07:02:40.126 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:39 smithi100 ceph-mon[29092]: pgmap v207: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-17T07:02:40.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:39 smithi100 ceph-mon[39462]: pgmap v207: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-17T07:02:40.965 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:02:40 smithi135 podman[61672]: 2024-09-17 07:02:40.940862943 +0000 UTC m=+0.514722866 container start e80f3f0bbd6a02365e092b9574c7e956f16cd804d0bd5d1470521e4ee6e234aa (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-17T07:02:41.180 INFO:teuthology.orchestra.run.smithi135.stdout:[client.1] 2024-09-17T07:02:41.180 INFO:teuthology.orchestra.run.smithi135.stdout: key = AQCQKelmuGmLNBAAVZYnvzMJpMcc04Z3kHnENg== 2024-09-17T07:02:41.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:41 smithi135 ceph-mon[33238]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:02:41.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:41 smithi135 ceph-mon[33238]: from='client.? 172.21.15.135:0/56398087' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-17T07:02:41.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:41 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-17T07:02:41.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:41 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-17T07:02:41.294 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:02:40 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a[61712]: ts=2024-09-17T07:02:40.964Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-09-17T07:02:41.294 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:02:40 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a[61712]: ts=2024-09-17T07:02:40.964Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-09-17T07:02:41.294 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:02:40 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a[61712]: ts=2024-09-17T07:02:40.964Z caller=main.go:570 level=info host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi135 (none))" 2024-09-17T07:02:41.294 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:02:40 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a[61712]: ts=2024-09-17T07:02:40.964Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-17T07:02:41.294 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:02:40 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a[61712]: ts=2024-09-17T07:02:40.964Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-17T07:02:41.294 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:02:40 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a[61712]: ts=2024-09-17T07:02:40.966Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-09-17T07:02:41.294 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:02:40 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a[61712]: ts=2024-09-17T07:02:40.966Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-09-17T07:02:41.295 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:02:40 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a[61712]: ts=2024-09-17T07:02:40.967Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-09-17T07:02:41.295 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:02:40 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a[61712]: ts=2024-09-17T07:02:40.967Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-17T07:02:41.295 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:02:40 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a[61712]: ts=2024-09-17T07:02:40.969Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-17T07:02:41.295 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:02:40 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a[61712]: ts=2024-09-17T07:02:40.969Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.72µs 2024-09-17T07:02:41.295 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:02:40 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a[61712]: ts=2024-09-17T07:02:40.969Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-17T07:02:41.295 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:02:40 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a[61712]: ts=2024-09-17T07:02:40.969Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-09-17T07:02:41.295 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:02:40 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a[61712]: ts=2024-09-17T07:02:40.969Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=18.229µs wal_replay_duration=180.192µs wbl_replay_duration=108ns total_replay_duration=217.271µs 2024-09-17T07:02:41.295 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:02:40 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a[61712]: ts=2024-09-17T07:02:40.970Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-09-17T07:02:41.295 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:02:40 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a[61712]: ts=2024-09-17T07:02:40.970Z caller=main.go:1029 level=info msg="TSDB started" 2024-09-17T07:02:41.295 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:02:40 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a[61712]: ts=2024-09-17T07:02:40.970Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-17T07:02:41.295 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:02:40 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a[61712]: ts=2024-09-17T07:02:40.990Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=19.814232ms db_storage=828ns remote_storage=981ns web_handler=312ns query_engine=585ns scrape=450.412µs scrape_sd=101.438µs notify=705ns notify_sd=1.699µs rules=18.95988ms tracing=6.662µs 2024-09-17T07:02:41.296 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:02:40 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a[61712]: ts=2024-09-17T07:02:40.990Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-09-17T07:02:41.296 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:02:40 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a[61712]: ts=2024-09-17T07:02:40.990Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-09-17T07:02:41.296 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:02:41 smithi135 bash[61672]: e80f3f0bbd6a02365e092b9574c7e956f16cd804d0bd5d1470521e4ee6e234aa 2024-09-17T07:02:41.296 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:02:41 smithi135 systemd[1]: Started Ceph prometheus.a for e848267c-74c1-11ef-bceb-c7b262605968. 2024-09-17T07:02:41.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:41 smithi100 ceph-mon[29092]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:02:41.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:41 smithi100 ceph-mon[29092]: from='client.? 172.21.15.135:0/56398087' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-17T07:02:41.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:41 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-17T07:02:41.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:41 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-17T07:02:41.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:41 smithi100 ceph-mon[39462]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:02:41.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:41 smithi100 ceph-mon[39462]: from='client.? 172.21.15.135:0/56398087' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-17T07:02:41.379 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:41 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-17T07:02:41.379 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:41 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-17T07:02:41.697 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-09-17T07:02:41.697 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-09-17T07:02:41.697 DEBUG:teuthology.orchestra.run.smithi135:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-09-17T07:02:41.735 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-17T07:02:41.735 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-09-17T07:02:41.735 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph mgr dump --format=json 2024-09-17T07:02:41.941 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:02:42.333 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:42 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:42.333 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:42 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:42.333 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:42 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:42.333 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:42 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-17T07:02:42.333 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:42 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:42.333 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:42 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:42.333 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:42 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:42.334 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:42 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-17T07:02:42.334 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:02:42 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ignoring --setuser ceph since I am not root 2024-09-17T07:02:42.334 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:02:42 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ignoring --setgroup ceph since I am not root 2024-09-17T07:02:42.334 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:02:42 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T07:02:42.239+0000 7f8bf3fbe040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-17T07:02:42.334 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:02:42 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T07:02:42.296+0000 7f8bf3fbe040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-17T07:02:42.439 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:42 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:42.439 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:42 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:42.440 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:42 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' 2024-09-17T07:02:42.440 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:42 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-17T07:02:42.440 INFO:journalctl@ceph.mgr.x.smithi135.stdout:Sep 17 07:02:42 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-x[35098]: ignoring --setuser ceph since I am not root 2024-09-17T07:02:42.440 INFO:journalctl@ceph.mgr.x.smithi135.stdout:Sep 17 07:02:42 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-x[35098]: ignoring --setgroup ceph since I am not root 2024-09-17T07:02:42.440 INFO:journalctl@ceph.mgr.x.smithi135.stdout:Sep 17 07:02:42 smithi135 ceph-mgr[35102]: -- 172.21.15.135:0/3815785654 <== mon.1 v2:172.21.15.135:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x555803b2ed00 con 0x555803b02800 2024-09-17T07:02:42.440 INFO:journalctl@ceph.mgr.x.smithi135.stdout:Sep 17 07:02:42 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-x[35098]: 2024-09-17T07:02:42.234+0000 7f4146d5f040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-17T07:02:42.440 INFO:journalctl@ceph.mgr.x.smithi135.stdout:Sep 17 07:02:42 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-x[35098]: 2024-09-17T07:02:42.290+0000 7f4146d5f040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-17T07:02:42.440 INFO:journalctl@ceph.mgr.x.smithi135.stdout:Sep 17 07:02:42 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-x[35098]: 2024-09-17T07:02:42.438+0000 7f4146d5f040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-17T07:02:42.626 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:02:42 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T07:02:42.448+0000 7f8bf3fbe040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-17T07:02:42.626 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:02:42 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T07:02:42.514+0000 7f8bf3fbe040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-17T07:02:42.786 INFO:journalctl@ceph.mgr.x.smithi135.stdout:Sep 17 07:02:42 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-x[35098]: 2024-09-17T07:02:42.503+0000 7f4146d5f040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-17T07:02:42.786 INFO:journalctl@ceph.mgr.x.smithi135.stdout:Sep 17 07:02:42 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-x[35098]: 2024-09-17T07:02:42.646+0000 7f4146d5f040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-17T07:02:42.787 INFO:journalctl@ceph.mgr.x.smithi135.stdout:Sep 17 07:02:42 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-x[35098]: 2024-09-17T07:02:42.785+0000 7f4146d5f040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-17T07:02:43.094 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:02:42 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T07:02:42.660+0000 7f8bf3fbe040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-17T07:02:43.094 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:02:42 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T07:02:42.801+0000 7f8bf3fbe040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-17T07:02:43.364 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:43 smithi100 ceph-mon[29092]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-17T07:02:43.364 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:43 smithi100 ceph-mon[29092]: mgrmap e17: y(active, since 4m), standbys: x 2024-09-17T07:02:43.365 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:43 smithi100 ceph-mon[39462]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-17T07:02:43.365 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:43 smithi100 ceph-mon[39462]: mgrmap e17: y(active, since 4m), standbys: x 2024-09-17T07:02:43.365 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:02:43 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T07:02:43.112+0000 7f8bf3fbe040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-17T07:02:43.365 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:02:43 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T07:02:43.176+0000 7f8bf3fbe040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-17T07:02:43.365 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:02:43 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T07:02:43.363+0000 7f8bf3fbe040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-17T07:02:43.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:43 smithi135 ceph-mon[33238]: from='mgr.14150 172.21.15.100:0/2975509461' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-17T07:02:43.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:43 smithi135 ceph-mon[33238]: mgrmap e17: y(active, since 4m), standbys: x 2024-09-17T07:02:43.543 INFO:journalctl@ceph.mgr.x.smithi135.stdout:Sep 17 07:02:43 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-x[35098]: 2024-09-17T07:02:43.089+0000 7f4146d5f040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-17T07:02:43.544 INFO:journalctl@ceph.mgr.x.smithi135.stdout:Sep 17 07:02:43 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-x[35098]: 2024-09-17T07:02:43.149+0000 7f4146d5f040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-17T07:02:43.544 INFO:journalctl@ceph.mgr.x.smithi135.stdout:Sep 17 07:02:43 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-x[35098]: 2024-09-17T07:02:43.330+0000 7f4146d5f040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-17T07:02:43.925 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T07:02:44.244 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:44 smithi100 ceph-mon[29092]: 1.0 scrub starts 2024-09-17T07:02:44.244 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:44 smithi100 ceph-mon[29092]: 1.0 scrub ok 2024-09-17T07:02:44.245 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:44 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1364031342' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-17T07:02:44.249 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:44 smithi100 ceph-mon[39462]: 1.0 scrub starts 2024-09-17T07:02:44.249 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:44 smithi100 ceph-mon[39462]: 1.0 scrub ok 2024-09-17T07:02:44.250 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:44 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1364031342' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-17T07:02:44.536 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:02:44 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T07:02:44.268+0000 7f8bf3fbe040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-17T07:02:44.537 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:02:44 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T07:02:44.334+0000 7f8bf3fbe040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-17T07:02:44.543 INFO:journalctl@ceph.mgr.x.smithi135.stdout:Sep 17 07:02:44 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-x[35098]: 2024-09-17T07:02:44.201+0000 7f4146d5f040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-17T07:02:44.543 INFO:journalctl@ceph.mgr.x.smithi135.stdout:Sep 17 07:02:44 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-x[35098]: 2024-09-17T07:02:44.267+0000 7f4146d5f040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-17T07:02:44.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:44 smithi135 ceph-mon[33238]: 1.0 scrub starts 2024-09-17T07:02:44.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:44 smithi135 ceph-mon[33238]: 1.0 scrub ok 2024-09-17T07:02:44.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:44 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1364031342' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-17T07:02:44.559 INFO:teuthology.orchestra.run.smithi100.stdout:{"epoch":17,"flags":0,"active_gid":14150,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":3472658181}]},"active_addr":"172.21.15.100:6800/3472658181","active_change":"2024-09-17T06:57:58.526411+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":24112,"name":"x","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.100:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"cephadm","addrvec":[{"type":"v2","addr":"172.21.15.100:0","nonce":380524491}]},{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.100:0","nonce":1676314112}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.100:0","nonce":2225586217}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.100:0","nonce":3362169179}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.100:0","nonce":2573779025}]}]} 2024-09-17T07:02:44.565 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-09-17T07:02:44.565 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-17T07:02:44.565 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-17T07:02:44.769 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:02:45.196 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:02:45 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T07:02:45.071+0000 7f8bf3fbe040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-17T07:02:45.196 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:02:45 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T07:02:45.195+0000 7f8bf3fbe040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-17T07:02:45.224 INFO:journalctl@ceph.mgr.x.smithi135.stdout:Sep 17 07:02:44 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-x[35098]: 2024-09-17T07:02:44.968+0000 7f4146d5f040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-17T07:02:45.224 INFO:journalctl@ceph.mgr.x.smithi135.stdout:Sep 17 07:02:45 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-x[35098]: 2024-09-17T07:02:45.081+0000 7f4146d5f040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-17T07:02:45.496 INFO:journalctl@ceph.mgr.x.smithi135.stdout:Sep 17 07:02:45 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-x[35098]: 2024-09-17T07:02:45.222+0000 7f4146d5f040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-17T07:02:45.496 INFO:journalctl@ceph.mgr.x.smithi135.stdout:Sep 17 07:02:45 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-x[35098]: 2024-09-17T07:02:45.282+0000 7f4146d5f040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-17T07:02:45.496 INFO:journalctl@ceph.mgr.x.smithi135.stdout:Sep 17 07:02:45 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-x[35098]: 2024-09-17T07:02:45.427+0000 7f4146d5f040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-17T07:02:45.497 INFO:journalctl@ceph.mgr.x.smithi135.stdout:Sep 17 07:02:45 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-x[35098]: 2024-09-17T07:02:45.494+0000 7f4146d5f040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-17T07:02:45.547 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:02:45 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T07:02:45.336+0000 7f8bf3fbe040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-17T07:02:45.547 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:02:45 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T07:02:45.395+0000 7f8bf3fbe040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-17T07:02:45.547 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:02:45 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T07:02:45.546+0000 7f8bf3fbe040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-17T07:02:45.774 INFO:journalctl@ceph.mgr.x.smithi135.stdout:Sep 17 07:02:45 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-x[35098]: 2024-09-17T07:02:45.554+0000 7f4146d5f040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-17T07:02:45.774 INFO:journalctl@ceph.mgr.x.smithi135.stdout:Sep 17 07:02:45 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-x[35098]: 2024-09-17T07:02:45.709+0000 7f4146d5f040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-17T07:02:45.774 INFO:journalctl@ceph.mgr.x.smithi135.stdout:Sep 17 07:02:45 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-x[35098]: 2024-09-17T07:02:45.773+0000 7f4146d5f040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-17T07:02:45.832 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:02:45 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T07:02:45.615+0000 7f8bf3fbe040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-17T07:02:45.832 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:02:45 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T07:02:45.674+0000 7f8bf3fbe040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-17T07:02:46.042 INFO:journalctl@ceph.mgr.x.smithi135.stdout:Sep 17 07:02:45 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-x[35098]: 2024-09-17T07:02:45.839+0000 7f4146d5f040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-17T07:02:46.042 INFO:journalctl@ceph.mgr.x.smithi135.stdout:Sep 17 07:02:45 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-x[35098]: 2024-09-17T07:02:45.921+0000 7f4146d5f040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-17T07:02:46.042 INFO:journalctl@ceph.mgr.x.smithi135.stdout:Sep 17 07:02:45 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-x[35098]: 2024-09-17T07:02:45.979+0000 7f4146d5f040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-17T07:02:46.042 INFO:journalctl@ceph.mgr.x.smithi135.stdout:Sep 17 07:02:46 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-x[35098]: 2024-09-17T07:02:46.040+0000 7f4146d5f040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-17T07:02:46.112 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:02:45 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T07:02:45.831+0000 7f8bf3fbe040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-17T07:02:46.112 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:02:45 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T07:02:45.897+0000 7f8bf3fbe040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-17T07:02:46.112 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:02:45 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T07:02:45.963+0000 7f8bf3fbe040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-17T07:02:46.112 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:02:46 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T07:02:46.052+0000 7f8bf3fbe040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-17T07:02:46.112 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:02:46 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T07:02:46.110+0000 7f8bf3fbe040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-17T07:02:46.377 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:02:46 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T07:02:46.170+0000 7f8bf3fbe040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-17T07:02:46.679 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T07:02:46.679 INFO:teuthology.orchestra.run.smithi100.stdout:{"epoch":77,"fsid":"e848267c-74c1-11ef-bceb-c7b262605968","created":"2024-09-17T06:57:05.010532+0000","modified":"2024-09-17T07:02:25.002672+0000","last_up_change":"2024-09-17T07:01:59.917118+0000","last_in_change":"2024-09-17T07:01:42.942454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T07:00:08.816136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-09-17T07:02:10.759826+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"datapool","create_time":"2024-09-17T07:02:10.991289+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":72,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-09-17T07:02:12.291462+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-09-17T07:02:14.253214+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-09-17T07:02:16.310620+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"27bee9ec-05e6-406c-8bf1-168098da0ec7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6801","nonce":3341017455}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":3341017455}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":3341017455}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6803","nonce":3341017455}]},"public_addr":"172.21.15.100:6801/3341017455","cluster_addr":"172.21.15.100:6802/3341017455","heartbeat_back_addr":"172.21.15.100:6804/3341017455","heartbeat_front_addr":"172.21.15.100:6803/3341017455","state":["exists","up"]},{"osd":1,"uuid":"b1a30c9c-eeaf-478e-9b62-2d63ad45873d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6805","nonce":1634750965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":1634750965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":1634750965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6807","nonce":1634750965}]},"public_addr":"172.21.15.100:6805/1634750965","cluster_addr":"172.21.15.100:6806/1634750965","heartbeat_back_addr":"172.21.15.100:6808/1634750965","heartbeat_front_addr":"172.21.15.100:6807/1634750965","state":["exists","up"]},{"osd":2,"uuid":"ce542e0d-9915-465e-aa11-7b3ce13f694f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":714579846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":714579846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":714579846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":714579846}]},"public_addr":"172.21.15.100:6809/714579846","cluster_addr":"172.21.15.100:6810/714579846","heartbeat_back_addr":"172.21.15.100:6812/714579846","heartbeat_front_addr":"172.21.15.100:6811/714579846","state":["exists","up"]},{"osd":3,"uuid":"a1257575-3c90-41de-88f6-3960744351b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6813","nonce":2315956572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":2315956572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":2315956572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":2315956572}]},"public_addr":"172.21.15.100:6813/2315956572","cluster_addr":"172.21.15.100:6814/2315956572","heartbeat_back_addr":"172.21.15.100:6816/2315956572","heartbeat_front_addr":"172.21.15.100:6815/2315956572","state":["exists","up"]},{"osd":4,"uuid":"e44e8922-2c5e-44e3-b556-d4dfbc499649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":1148820922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6801","nonce":1148820922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6803","nonce":1148820922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":1148820922}]},"public_addr":"172.21.15.135:6800/1148820922","cluster_addr":"172.21.15.135:6801/1148820922","heartbeat_back_addr":"172.21.15.135:6803/1148820922","heartbeat_front_addr":"172.21.15.135:6802/1148820922","state":["exists","up"]},{"osd":5,"uuid":"91e4a362-fcdb-41b9-b54e-716591ba6d79","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2256520893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6805","nonce":2256520893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6807","nonce":2256520893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2256520893}]},"public_addr":"172.21.15.135:6804/2256520893","cluster_addr":"172.21.15.135:6805/2256520893","heartbeat_back_addr":"172.21.15.135:6807/2256520893","heartbeat_front_addr":"172.21.15.135:6806/2256520893","state":["exists","up"]},{"osd":6,"uuid":"de799f64-ebd2-4a46-91f6-c2f276bee21f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":2750184152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6809","nonce":2750184152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6811","nonce":2750184152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":2750184152}]},"public_addr":"172.21.15.135:6808/2750184152","cluster_addr":"172.21.15.135:6809/2750184152","heartbeat_back_addr":"172.21.15.135:6811/2750184152","heartbeat_front_addr":"172.21.15.135:6810/2750184152","state":["exists","up"]},{"osd":7,"uuid":"830932c7-5de1-44ff-a840-e53a6e859bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":160304773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6813","nonce":160304773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6815","nonce":160304773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":160304773}]},"public_addr":"172.21.15.135:6812/160304773","cluster_addr":"172.21.15.135:6813/160304773","heartbeat_back_addr":"172.21.15.135:6815/160304773","heartbeat_front_addr":"172.21.15.135:6814/160304773","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:01:51.772595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:02:06.826651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:02:19.221402+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:01:49.961971+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:02:04.479239+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:02:13.999532+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:01:35.329321+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:01:57.956813+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.100:0/1657291976":"2024-09-18T06:57:58.526309+0000","172.21.15.100:0/1365215411":"2024-09-18T06:57:58.526309+0000","172.21.15.100:6800/3992296558":"2024-09-18T06:57:58.526309+0000","172.21.15.100:0/3452172206":"2024-09-18T06:57:35.678309+0000","172.21.15.100:0/1626706829":"2024-09-18T06:57:35.678309+0000","172.21.15.100:0/1255197255":"2024-09-18T06:57:58.526309+0000","172.21.15.100:6800/3498574214":"2024-09-18T06:57:35.678309+0000","172.21.15.100:0/3796664909":"2024-09-18T06:57:35.678309+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":2,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T07:02:46.980 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:46 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/3186829859' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-17T07:02:46.980 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:46 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/3186829859' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-17T07:02:47.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:46 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/3186829859' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-17T07:02:47.043 INFO:journalctl@ceph.mgr.x.smithi135.stdout:Sep 17 07:02:46 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-x[35098]: 2024-09-17T07:02:46.804+0000 7f4146d5f040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-17T07:02:47.043 INFO:journalctl@ceph.mgr.x.smithi135.stdout:Sep 17 07:02:46 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-x[35098]: [17/Sep/2024:07:02:46] ENGINE Bus STARTING 2024-09-17T07:02:47.043 INFO:journalctl@ceph.mgr.x.smithi135.stdout:Sep 17 07:02:46 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-x[35098]: CherryPy Checker: 2024-09-17T07:02:47.044 INFO:journalctl@ceph.mgr.x.smithi135.stdout:Sep 17 07:02:46 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-x[35098]: The Application mounted at '' has an empty config. 2024-09-17T07:02:47.044 INFO:journalctl@ceph.mgr.x.smithi135.stdout:Sep 17 07:02:46 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-x[35098]: 2024-09-17T07:02:47.044 INFO:journalctl@ceph.mgr.x.smithi135.stdout:Sep 17 07:02:46 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-x[35098]: [17/Sep/2024:07:02:46] ENGINE Serving on http://:::9283 2024-09-17T07:02:47.044 INFO:journalctl@ceph.mgr.x.smithi135.stdout:Sep 17 07:02:46 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-x[35098]: [17/Sep/2024:07:02:46] ENGINE Bus STARTED 2024-09-17T07:02:47.325 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:02:46 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T07:02:46.979+0000 7f8bf3fbe040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-17T07:02:47.325 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:02:47 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: [17/Sep/2024:07:02:47] ENGINE Bus STARTING 2024-09-17T07:02:47.325 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:02:47 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: CherryPy Checker: 2024-09-17T07:02:47.325 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:02:47 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: The Application mounted at '' has an empty config. 2024-09-17T07:02:47.325 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:02:47 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T07:02:47.325 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:02:47 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: [17/Sep/2024:07:02:47] ENGINE Serving on http://:::9283 2024-09-17T07:02:47.325 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:02:47 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: [17/Sep/2024:07:02:47] ENGINE Bus STARTED 2024-09-17T07:02:47.509 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-17T07:02:47.510 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-17T07:02:47.732 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:02:48.001 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:47 smithi135 ceph-mon[33238]: Standby manager daemon x restarted 2024-09-17T07:02:48.001 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:47 smithi135 ceph-mon[33238]: Standby manager daemon x started 2024-09-17T07:02:48.002 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:47 smithi135 ceph-mon[33238]: from='mgr.? 172.21.15.135:0/1522988406' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-17T07:02:48.002 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:47 smithi135 ceph-mon[33238]: from='mgr.? 172.21.15.135:0/1522988406' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T07:02:48.002 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:47 smithi135 ceph-mon[33238]: from='mgr.? 172.21.15.135:0/1522988406' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-17T07:02:48.002 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:47 smithi135 ceph-mon[33238]: from='mgr.? 172.21.15.135:0/1522988406' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T07:02:48.002 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:47 smithi135 ceph-mon[33238]: Active manager daemon y restarted 2024-09-17T07:02:48.002 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:47 smithi135 ceph-mon[33238]: Activating manager daemon y 2024-09-17T07:02:48.002 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:47 smithi135 ceph-mon[33238]: osdmap e78: 8 total, 8 up, 8 in 2024-09-17T07:02:48.002 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:47 smithi135 ceph-mon[33238]: mgrmap e18: y(active, starting, since 0.0101713s), standbys: x 2024-09-17T07:02:48.002 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:47 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T07:02:48.002 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:47 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T07:02:48.002 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:47 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T07:02:48.002 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:47 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-17T07:02:48.003 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:47 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-17T07:02:48.003 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:47 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T07:02:48.003 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:47 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T07:02:48.003 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:47 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T07:02:48.003 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:47 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T07:02:48.003 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:47 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T07:02:48.003 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:47 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T07:02:48.003 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:47 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T07:02:48.003 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:47 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T07:02:48.003 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:47 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-17T07:02:48.003 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:47 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-17T07:02:48.003 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:47 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-17T07:02:48.004 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:47 smithi135 ceph-mon[33238]: Manager daemon y is now available 2024-09-17T07:02:48.004 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:47 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:02:48.004 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:47 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:02:48.004 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:47 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:02:48.004 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:47 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T07:02:48.004 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:47 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T07:02:48.080 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:47 smithi100 ceph-mon[39462]: Standby manager daemon x restarted 2024-09-17T07:02:48.081 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:47 smithi100 ceph-mon[39462]: Standby manager daemon x started 2024-09-17T07:02:48.081 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:47 smithi100 ceph-mon[39462]: from='mgr.? 172.21.15.135:0/1522988406' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-17T07:02:48.081 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:47 smithi100 ceph-mon[39462]: from='mgr.? 172.21.15.135:0/1522988406' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T07:02:48.081 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:47 smithi100 ceph-mon[39462]: from='mgr.? 172.21.15.135:0/1522988406' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-17T07:02:48.081 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:47 smithi100 ceph-mon[39462]: from='mgr.? 172.21.15.135:0/1522988406' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T07:02:48.081 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:47 smithi100 ceph-mon[39462]: Active manager daemon y restarted 2024-09-17T07:02:48.081 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:47 smithi100 ceph-mon[39462]: Activating manager daemon y 2024-09-17T07:02:48.081 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:47 smithi100 ceph-mon[39462]: osdmap e78: 8 total, 8 up, 8 in 2024-09-17T07:02:48.081 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:47 smithi100 ceph-mon[39462]: mgrmap e18: y(active, starting, since 0.0101713s), standbys: x 2024-09-17T07:02:48.081 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:47 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T07:02:48.081 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:47 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T07:02:48.081 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:47 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T07:02:48.081 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:47 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-17T07:02:48.082 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:47 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-17T07:02:48.082 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:47 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T07:02:48.082 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:47 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T07:02:48.082 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:47 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T07:02:48.082 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:47 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T07:02:48.082 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:47 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T07:02:48.082 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:47 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T07:02:48.082 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:47 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T07:02:48.082 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:47 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T07:02:48.082 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:47 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-17T07:02:48.082 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:47 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-17T07:02:48.082 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:47 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-17T07:02:48.082 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:47 smithi100 ceph-mon[39462]: Manager daemon y is now available 2024-09-17T07:02:48.083 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:47 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:02:48.083 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:47 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:02:48.083 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:47 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:02:48.083 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:47 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T07:02:48.083 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:47 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T07:02:48.083 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:47 smithi100 ceph-mon[29092]: Standby manager daemon x restarted 2024-09-17T07:02:48.083 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:47 smithi100 ceph-mon[29092]: Standby manager daemon x started 2024-09-17T07:02:48.083 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:47 smithi100 ceph-mon[29092]: from='mgr.? 172.21.15.135:0/1522988406' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-17T07:02:48.083 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:47 smithi100 ceph-mon[29092]: from='mgr.? 172.21.15.135:0/1522988406' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T07:02:48.084 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:47 smithi100 ceph-mon[29092]: from='mgr.? 172.21.15.135:0/1522988406' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-17T07:02:48.084 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:47 smithi100 ceph-mon[29092]: from='mgr.? 172.21.15.135:0/1522988406' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T07:02:48.084 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:47 smithi100 ceph-mon[29092]: Active manager daemon y restarted 2024-09-17T07:02:48.084 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:47 smithi100 ceph-mon[29092]: Activating manager daemon y 2024-09-17T07:02:48.084 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:47 smithi100 ceph-mon[29092]: osdmap e78: 8 total, 8 up, 8 in 2024-09-17T07:02:48.084 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:47 smithi100 ceph-mon[29092]: mgrmap e18: y(active, starting, since 0.0101713s), standbys: x 2024-09-17T07:02:48.084 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:47 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T07:02:48.084 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:47 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T07:02:48.084 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:47 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T07:02:48.084 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:47 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-17T07:02:48.084 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:47 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-17T07:02:48.084 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:47 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T07:02:48.085 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:47 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T07:02:48.085 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:47 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T07:02:48.085 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:47 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T07:02:48.085 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:47 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T07:02:48.085 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:47 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T07:02:48.085 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:47 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T07:02:48.085 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:47 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T07:02:48.085 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:47 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-17T07:02:48.085 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:47 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-17T07:02:48.085 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:47 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-17T07:02:48.085 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:47 smithi100 ceph-mon[29092]: Manager daemon y is now available 2024-09-17T07:02:48.085 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:47 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:02:48.085 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:47 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:02:48.086 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:47 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:02:48.086 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:47 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T07:02:48.086 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:47 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T07:02:48.264 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:02:48 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:02:48.377 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:02:48 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T07:02:48.322+0000 7f8b4c4ad640 -1 log_channel(cephadm) log [ERR] : [17/Sep/2024:07:02:48] ENGINE Error in HTTPServer.serve 2024-09-17T07:02:48.377 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:02:48 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: Traceback (most recent call last): 2024-09-17T07:02:48.377 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:02:48 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T07:02:48.377 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:02:48 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: self._connections.run(self.expiration_interval) 2024-09-17T07:02:48.377 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:02:48 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T07:02:48.378 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:02:48 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: self._run(expiration_interval) 2024-09-17T07:02:48.378 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:02:48 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T07:02:48.378 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:02:48 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T07:02:48.378 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:02:48 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T07:02:48.378 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:02:48 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T07:02:48.378 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:02:48 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T07:02:48.378 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:02:48 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: s = self.context.wrap_socket( 2024-09-17T07:02:48.378 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:02:48 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T07:02:48.378 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:02:48 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: return self.sslsocket_class._create( 2024-09-17T07:02:48.378 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:02:48 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T07:02:48.379 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:02:48 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: self.do_handshake() 2024-09-17T07:02:48.379 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:02:48 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T07:02:48.379 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:02:48 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: self._sslobj.do_handshake() 2024-09-17T07:02:48.379 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:02:48 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T07:02:48.379 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:02:48 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T07:02:49.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:48 smithi100 ceph-mon[29092]: mgrmap e19: y(active, since 1.0126s), standbys: x 2024-09-17T07:02:49.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:48 smithi100 ceph-mon[29092]: [17/Sep/2024:07:02:48] ENGINE Bus STARTING 2024-09-17T07:02:49.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:49 smithi100 ceph-mon[39462]: mgrmap e19: y(active, since 1.0126s), standbys: x 2024-09-17T07:02:49.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:49 smithi100 ceph-mon[39462]: [17/Sep/2024:07:02:48] ENGINE Bus STARTING 2024-09-17T07:02:49.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:48 smithi135 ceph-mon[33238]: mgrmap e19: y(active, since 1.0126s), standbys: x 2024-09-17T07:02:49.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:48 smithi135 ceph-mon[33238]: [17/Sep/2024:07:02:48] ENGINE Bus STARTING 2024-09-17T07:02:50.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:49 smithi135 ceph-mon[33238]: [17/Sep/2024:07:02:48] ENGINE Serving on http://172.21.15.100:8765 2024-09-17T07:02:50.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:49 smithi135 ceph-mon[33238]: [17/Sep/2024:07:02:48] ENGINE Serving on https://172.21.15.100:7150 2024-09-17T07:02:50.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:49 smithi135 ceph-mon[33238]: [17/Sep/2024:07:02:48] ENGINE Bus STARTED 2024-09-17T07:02:50.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:49 smithi135 ceph-mon[33238]: [17/Sep/2024:07:02:48] ENGINE Error in HTTPServer.serve 2024-09-17T07:02:50.044 INFO:journalctl@ceph.mon.b.smithi135.stdout: Traceback (most recent call last): 2024-09-17T07:02:50.044 INFO:journalctl@ceph.mon.b.smithi135.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T07:02:50.044 INFO:journalctl@ceph.mon.b.smithi135.stdout: self._connections.run(self.expiration_interval) 2024-09-17T07:02:50.044 INFO:journalctl@ceph.mon.b.smithi135.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T07:02:50.044 INFO:journalctl@ceph.mon.b.smithi135.stdout: self._run(expiration_interval) 2024-09-17T07:02:50.044 INFO:journalctl@ceph.mon.b.smithi135.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T07:02:50.044 INFO:journalctl@ceph.mon.b.smithi135.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T07:02:50.044 INFO:journalctl@ceph.mon.b.smithi135.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T07:02:50.044 INFO:journalctl@ceph.mon.b.smithi135.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T07:02:50.044 INFO:journalctl@ceph.mon.b.smithi135.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T07:02:50.045 INFO:journalctl@ceph.mon.b.smithi135.stdout: s = self.context.wrap_socket( 2024-09-17T07:02:50.045 INFO:journalctl@ceph.mon.b.smithi135.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T07:02:50.045 INFO:journalctl@ceph.mon.b.smithi135.stdout: return self.sslsocket_class._create( 2024-09-17T07:02:50.045 INFO:journalctl@ceph.mon.b.smithi135.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T07:02:50.045 INFO:journalctl@ceph.mon.b.smithi135.stdout: self.do_handshake() 2024-09-17T07:02:50.045 INFO:journalctl@ceph.mon.b.smithi135.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T07:02:50.045 INFO:journalctl@ceph.mon.b.smithi135.stdout: self._sslobj.do_handshake() 2024-09-17T07:02:50.045 INFO:journalctl@ceph.mon.b.smithi135.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T07:02:50.045 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:49 smithi135 ceph-mon[33238]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:02:50.045 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:49 smithi135 ceph-mon[33238]: mgrmap e20: y(active, since 2s), standbys: x 2024-09-17T07:02:50.292 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T07:02:50.293 INFO:teuthology.orchestra.run.smithi100.stdout:{"epoch":78,"fsid":"e848267c-74c1-11ef-bceb-c7b262605968","created":"2024-09-17T06:57:05.010532+0000","modified":"2024-09-17T07:02:46.981516+0000","last_up_change":"2024-09-17T07:01:59.917118+0000","last_in_change":"2024-09-17T07:01:42.942454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T07:00:08.816136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-09-17T07:02:10.759826+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"datapool","create_time":"2024-09-17T07:02:10.991289+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":72,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-09-17T07:02:12.291462+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-09-17T07:02:14.253214+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-09-17T07:02:16.310620+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"27bee9ec-05e6-406c-8bf1-168098da0ec7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6801","nonce":3341017455}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":3341017455}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":3341017455}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6803","nonce":3341017455}]},"public_addr":"172.21.15.100:6801/3341017455","cluster_addr":"172.21.15.100:6802/3341017455","heartbeat_back_addr":"172.21.15.100:6804/3341017455","heartbeat_front_addr":"172.21.15.100:6803/3341017455","state":["exists","up"]},{"osd":1,"uuid":"b1a30c9c-eeaf-478e-9b62-2d63ad45873d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6805","nonce":1634750965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":1634750965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":1634750965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6807","nonce":1634750965}]},"public_addr":"172.21.15.100:6805/1634750965","cluster_addr":"172.21.15.100:6806/1634750965","heartbeat_back_addr":"172.21.15.100:6808/1634750965","heartbeat_front_addr":"172.21.15.100:6807/1634750965","state":["exists","up"]},{"osd":2,"uuid":"ce542e0d-9915-465e-aa11-7b3ce13f694f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":714579846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":714579846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":714579846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":714579846}]},"public_addr":"172.21.15.100:6809/714579846","cluster_addr":"172.21.15.100:6810/714579846","heartbeat_back_addr":"172.21.15.100:6812/714579846","heartbeat_front_addr":"172.21.15.100:6811/714579846","state":["exists","up"]},{"osd":3,"uuid":"a1257575-3c90-41de-88f6-3960744351b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6813","nonce":2315956572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":2315956572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":2315956572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":2315956572}]},"public_addr":"172.21.15.100:6813/2315956572","cluster_addr":"172.21.15.100:6814/2315956572","heartbeat_back_addr":"172.21.15.100:6816/2315956572","heartbeat_front_addr":"172.21.15.100:6815/2315956572","state":["exists","up"]},{"osd":4,"uuid":"e44e8922-2c5e-44e3-b556-d4dfbc499649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":1148820922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6801","nonce":1148820922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6803","nonce":1148820922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":1148820922}]},"public_addr":"172.21.15.135:6800/1148820922","cluster_addr":"172.21.15.135:6801/1148820922","heartbeat_back_addr":"172.21.15.135:6803/1148820922","heartbeat_front_addr":"172.21.15.135:6802/1148820922","state":["exists","up"]},{"osd":5,"uuid":"91e4a362-fcdb-41b9-b54e-716591ba6d79","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2256520893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6805","nonce":2256520893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6807","nonce":2256520893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2256520893}]},"public_addr":"172.21.15.135:6804/2256520893","cluster_addr":"172.21.15.135:6805/2256520893","heartbeat_back_addr":"172.21.15.135:6807/2256520893","heartbeat_front_addr":"172.21.15.135:6806/2256520893","state":["exists","up"]},{"osd":6,"uuid":"de799f64-ebd2-4a46-91f6-c2f276bee21f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":2750184152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6809","nonce":2750184152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6811","nonce":2750184152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":2750184152}]},"public_addr":"172.21.15.135:6808/2750184152","cluster_addr":"172.21.15.135:6809/2750184152","heartbeat_back_addr":"172.21.15.135:6811/2750184152","heartbeat_front_addr":"172.21.15.135:6810/2750184152","state":["exists","up"]},{"osd":7,"uuid":"830932c7-5de1-44ff-a840-e53a6e859bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":160304773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6813","nonce":160304773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6815","nonce":160304773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":160304773}]},"public_addr":"172.21.15.135:6812/160304773","cluster_addr":"172.21.15.135:6813/160304773","heartbeat_back_addr":"172.21.15.135:6815/160304773","heartbeat_front_addr":"172.21.15.135:6814/160304773","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:01:51.772595+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:02:06.826651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:02:19.221402+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:01:49.961971+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:02:04.479239+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:02:13.999532+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:01:35.329321+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:01:57.956813+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.100:0/3796664909":"2024-09-18T06:57:35.678309+0000","172.21.15.100:0/2225586217":"2024-09-18T07:02:46.981486+0000","172.21.15.100:0/1626706829":"2024-09-18T06:57:35.678309+0000","172.21.15.100:0/3452172206":"2024-09-18T06:57:35.678309+0000","172.21.15.100:6800/3992296558":"2024-09-18T06:57:58.526309+0000","172.21.15.100:6800/3498574214":"2024-09-18T06:57:35.678309+0000","172.21.15.100:0/1365215411":"2024-09-18T06:57:58.526309+0000","172.21.15.100:0/3362169179":"2024-09-18T07:02:46.981486+0000","172.21.15.100:0/2573779025":"2024-09-18T07:02:46.981486+0000","172.21.15.100:0/380524491":"2024-09-18T07:02:46.981486+0000","172.21.15.100:0/1657291976":"2024-09-18T06:57:58.526309+0000","172.21.15.100:0/1676314112":"2024-09-18T07:02:46.981486+0000","172.21.15.100:0/1255197255":"2024-09-18T06:57:58.526309+0000","172.21.15.100:6800/3472658181":"2024-09-18T07:02:46.981486+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T07:02:50.293 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:49 smithi100 ceph-mon[29092]: [17/Sep/2024:07:02:48] ENGINE Serving on http://172.21.15.100:8765 2024-09-17T07:02:50.293 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:49 smithi100 ceph-mon[29092]: [17/Sep/2024:07:02:48] ENGINE Serving on https://172.21.15.100:7150 2024-09-17T07:02:50.293 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:49 smithi100 ceph-mon[29092]: [17/Sep/2024:07:02:48] ENGINE Bus STARTED 2024-09-17T07:02:50.293 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:49 smithi100 ceph-mon[29092]: [17/Sep/2024:07:02:48] ENGINE Error in HTTPServer.serve 2024-09-17T07:02:50.293 INFO:journalctl@ceph.mon.a.smithi100.stdout: Traceback (most recent call last): 2024-09-17T07:02:50.294 INFO:journalctl@ceph.mon.a.smithi100.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T07:02:50.294 INFO:journalctl@ceph.mon.a.smithi100.stdout: self._connections.run(self.expiration_interval) 2024-09-17T07:02:50.294 INFO:journalctl@ceph.mon.a.smithi100.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T07:02:50.294 INFO:journalctl@ceph.mon.a.smithi100.stdout: self._run(expiration_interval) 2024-09-17T07:02:50.294 INFO:journalctl@ceph.mon.a.smithi100.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T07:02:50.294 INFO:journalctl@ceph.mon.a.smithi100.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T07:02:50.294 INFO:journalctl@ceph.mon.a.smithi100.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T07:02:50.294 INFO:journalctl@ceph.mon.a.smithi100.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T07:02:50.294 INFO:journalctl@ceph.mon.a.smithi100.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T07:02:50.294 INFO:journalctl@ceph.mon.a.smithi100.stdout: s = self.context.wrap_socket( 2024-09-17T07:02:50.294 INFO:journalctl@ceph.mon.a.smithi100.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T07:02:50.294 INFO:journalctl@ceph.mon.a.smithi100.stdout: return self.sslsocket_class._create( 2024-09-17T07:02:50.295 INFO:journalctl@ceph.mon.a.smithi100.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T07:02:50.295 INFO:journalctl@ceph.mon.a.smithi100.stdout: self.do_handshake() 2024-09-17T07:02:50.295 INFO:journalctl@ceph.mon.a.smithi100.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T07:02:50.295 INFO:journalctl@ceph.mon.a.smithi100.stdout: self._sslobj.do_handshake() 2024-09-17T07:02:50.295 INFO:journalctl@ceph.mon.a.smithi100.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T07:02:50.295 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:49 smithi100 ceph-mon[29092]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:02:50.295 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:49 smithi100 ceph-mon[29092]: mgrmap e20: y(active, since 2s), standbys: x 2024-09-17T07:02:50.295 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:50 smithi100 ceph-mon[39462]: [17/Sep/2024:07:02:48] ENGINE Serving on http://172.21.15.100:8765 2024-09-17T07:02:50.295 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:50 smithi100 ceph-mon[39462]: [17/Sep/2024:07:02:48] ENGINE Serving on https://172.21.15.100:7150 2024-09-17T07:02:50.295 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:50 smithi100 ceph-mon[39462]: [17/Sep/2024:07:02:48] ENGINE Bus STARTED 2024-09-17T07:02:50.296 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:50 smithi100 ceph-mon[39462]: [17/Sep/2024:07:02:48] ENGINE Error in HTTPServer.serve 2024-09-17T07:02:50.296 INFO:journalctl@ceph.mon.c.smithi100.stdout: Traceback (most recent call last): 2024-09-17T07:02:50.296 INFO:journalctl@ceph.mon.c.smithi100.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T07:02:50.296 INFO:journalctl@ceph.mon.c.smithi100.stdout: self._connections.run(self.expiration_interval) 2024-09-17T07:02:50.296 INFO:journalctl@ceph.mon.c.smithi100.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T07:02:50.296 INFO:journalctl@ceph.mon.c.smithi100.stdout: self._run(expiration_interval) 2024-09-17T07:02:50.296 INFO:journalctl@ceph.mon.c.smithi100.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T07:02:50.296 INFO:journalctl@ceph.mon.c.smithi100.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T07:02:50.296 INFO:journalctl@ceph.mon.c.smithi100.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T07:02:50.296 INFO:journalctl@ceph.mon.c.smithi100.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T07:02:50.296 INFO:journalctl@ceph.mon.c.smithi100.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T07:02:50.297 INFO:journalctl@ceph.mon.c.smithi100.stdout: s = self.context.wrap_socket( 2024-09-17T07:02:50.297 INFO:journalctl@ceph.mon.c.smithi100.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T07:02:50.297 INFO:journalctl@ceph.mon.c.smithi100.stdout: return self.sslsocket_class._create( 2024-09-17T07:02:50.297 INFO:journalctl@ceph.mon.c.smithi100.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T07:02:50.297 INFO:journalctl@ceph.mon.c.smithi100.stdout: self.do_handshake() 2024-09-17T07:02:50.297 INFO:journalctl@ceph.mon.c.smithi100.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T07:02:50.297 INFO:journalctl@ceph.mon.c.smithi100.stdout: self._sslobj.do_handshake() 2024-09-17T07:02:50.297 INFO:journalctl@ceph.mon.c.smithi100.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T07:02:50.297 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:50 smithi100 ceph-mon[39462]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:02:50.297 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:50 smithi100 ceph-mon[39462]: mgrmap e20: y(active, since 2s), standbys: x 2024-09-17T07:02:51.010 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-09-17T07:02:51.011 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-09-17T07:02:51.011 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-09-17T07:02:51.012 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-09-17T07:02:51.012 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-09-17T07:02:51.013 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-09-17T07:02:51.013 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-09-17T07:02:51.014 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-09-17T07:02:51.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:51 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:02:51.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:51 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:02:51.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:51 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:02:51.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:51 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:02:51.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:51 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:02:51.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:51 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:02:51.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:51 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/788237429' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-17T07:02:51.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:51 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:02:51.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:51 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:02:51.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:51 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:02:51.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:51 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:02:51.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:51 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:02:51.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:51 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:02:51.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:51 smithi100 ceph-mon[29092]: Adjusting osd_memory_target on smithi135 to 941.8M 2024-09-17T07:02:51.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:51 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:02:51.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:51 smithi100 ceph-mon[29092]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:02:51.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:51 smithi100 ceph-mon[29092]: osdmap e79: 8 total, 8 up, 8 in 2024-09-17T07:02:51.379 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:51 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:02:51.379 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:51 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:02:51.379 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:51 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:02:51.379 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:51 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:02:51.379 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:51 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:02:51.379 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:51 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:02:51.379 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:51 smithi100 ceph-mon[29092]: Adjusting osd_memory_target on smithi100 to 3759M 2024-09-17T07:02:51.379 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:51 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:02:51.379 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:51 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:02:51.379 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:51 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:02:51.379 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:51 smithi100 ceph-mon[29092]: Updating smithi100:/etc/ceph/ceph.conf 2024-09-17T07:02:51.379 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:51 smithi100 ceph-mon[29092]: Updating smithi135:/etc/ceph/ceph.conf 2024-09-17T07:02:51.380 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:51 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:02:51.380 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:51 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:02:51.380 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:51 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:02:51.380 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:51 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:02:51.380 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:51 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:02:51.380 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:51 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:02:51.380 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:51 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/788237429' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-17T07:02:51.380 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:51 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:02:51.380 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:51 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:02:51.380 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:51 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:02:51.380 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:51 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:02:51.381 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:51 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:02:51.381 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:51 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:02:51.381 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:51 smithi100 ceph-mon[39462]: Adjusting osd_memory_target on smithi135 to 941.8M 2024-09-17T07:02:51.381 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:51 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:02:51.381 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:51 smithi100 ceph-mon[39462]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:02:51.381 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:51 smithi100 ceph-mon[39462]: osdmap e79: 8 total, 8 up, 8 in 2024-09-17T07:02:51.381 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:51 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:02:51.381 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:51 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:02:51.381 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:51 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:02:51.381 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:51 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:02:51.381 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:51 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:02:51.381 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:51 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:02:51.382 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:51 smithi100 ceph-mon[39462]: Adjusting osd_memory_target on smithi100 to 3759M 2024-09-17T07:02:51.382 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:51 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:02:51.382 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:51 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:02:51.382 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:51 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:02:51.382 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:51 smithi100 ceph-mon[39462]: Updating smithi100:/etc/ceph/ceph.conf 2024-09-17T07:02:51.382 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:51 smithi100 ceph-mon[39462]: Updating smithi135:/etc/ceph/ceph.conf 2024-09-17T07:02:51.401 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:02:51.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:51 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:02:51.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:51 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:02:51.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:51 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:02:51.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:51 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:02:51.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:51 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:02:51.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:51 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:02:51.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:51 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/788237429' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-17T07:02:51.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:51 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:02:51.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:51 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:02:51.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:51 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:02:51.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:51 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:02:51.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:51 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:02:51.545 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:51 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:02:51.545 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:51 smithi135 ceph-mon[33238]: Adjusting osd_memory_target on smithi135 to 941.8M 2024-09-17T07:02:51.545 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:51 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:02:51.545 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:51 smithi135 ceph-mon[33238]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:02:51.545 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:51 smithi135 ceph-mon[33238]: osdmap e79: 8 total, 8 up, 8 in 2024-09-17T07:02:51.545 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:51 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:02:51.545 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:51 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:02:51.545 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:51 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:02:51.545 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:51 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:02:51.545 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:51 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:02:51.545 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:51 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:02:51.545 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:51 smithi135 ceph-mon[33238]: Adjusting osd_memory_target on smithi100 to 3759M 2024-09-17T07:02:51.546 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:51 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:02:51.546 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:51 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:02:51.546 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:51 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:02:51.546 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:51 smithi135 ceph-mon[33238]: Updating smithi100:/etc/ceph/ceph.conf 2024-09-17T07:02:51.546 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:51 smithi135 ceph-mon[33238]: Updating smithi135:/etc/ceph/ceph.conf 2024-09-17T07:02:51.791 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:02:51.793 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:02:51.794 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:02:52.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:52 smithi100 ceph-mon[39462]: Updating smithi135:/var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T07:02:52.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:52 smithi100 ceph-mon[39462]: Updating smithi100:/var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T07:02:52.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:52 smithi100 ceph-mon[39462]: Updating smithi135:/etc/ceph/ceph.client.admin.keyring 2024-09-17T07:02:52.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:52 smithi100 ceph-mon[39462]: mgrmap e21: y(active, since 4s), standbys: x 2024-09-17T07:02:52.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:52 smithi100 ceph-mon[39462]: Updating smithi100:/etc/ceph/ceph.client.admin.keyring 2024-09-17T07:02:52.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:52 smithi100 ceph-mon[39462]: Updating smithi135:/var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-17T07:02:52.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:52 smithi100 ceph-mon[39462]: Updating smithi100:/var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-17T07:02:52.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:52 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:02:52.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:52 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:02:52.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:52 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:02:52.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:52 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:02:52.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:52 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:02:52.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:52 smithi100 ceph-mon[29092]: Updating smithi135:/var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T07:02:52.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:52 smithi100 ceph-mon[29092]: Updating smithi100:/var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T07:02:52.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:52 smithi100 ceph-mon[29092]: Updating smithi135:/etc/ceph/ceph.client.admin.keyring 2024-09-17T07:02:52.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:52 smithi100 ceph-mon[29092]: mgrmap e21: y(active, since 4s), standbys: x 2024-09-17T07:02:52.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:52 smithi100 ceph-mon[29092]: Updating smithi100:/etc/ceph/ceph.client.admin.keyring 2024-09-17T07:02:52.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:52 smithi100 ceph-mon[29092]: Updating smithi135:/var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-17T07:02:52.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:52 smithi100 ceph-mon[29092]: Updating smithi100:/var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-17T07:02:52.879 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:52 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:02:52.879 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:52 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:02:52.879 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:52 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:02:52.879 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:52 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:02:52.879 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:52 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:02:52.939 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:02:52.939 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:02:53.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:52 smithi135 ceph-mon[33238]: Updating smithi135:/var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T07:02:53.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:52 smithi135 ceph-mon[33238]: Updating smithi100:/var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T07:02:53.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:52 smithi135 ceph-mon[33238]: Updating smithi135:/etc/ceph/ceph.client.admin.keyring 2024-09-17T07:02:53.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:52 smithi135 ceph-mon[33238]: mgrmap e21: y(active, since 4s), standbys: x 2024-09-17T07:02:53.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:52 smithi135 ceph-mon[33238]: Updating smithi100:/etc/ceph/ceph.client.admin.keyring 2024-09-17T07:02:53.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:52 smithi135 ceph-mon[33238]: Updating smithi135:/var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-17T07:02:53.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:52 smithi135 ceph-mon[33238]: Updating smithi100:/var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-17T07:02:53.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:52 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:02:53.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:52 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:02:53.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:52 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:02:53.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:52 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:02:53.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:52 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:02:54.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:53 smithi135 ceph-mon[33238]: Deploying daemon alertmanager.a on smithi100 2024-09-17T07:02:54.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:53 smithi135 ceph-mon[33238]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:02:54.069 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:53 smithi100 ceph-mon[29092]: Deploying daemon alertmanager.a on smithi100 2024-09-17T07:02:54.069 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:53 smithi100 ceph-mon[29092]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:02:54.070 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:53 smithi100 ceph-mon[39462]: Deploying daemon alertmanager.a on smithi100 2024-09-17T07:02:54.070 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:53 smithi100 ceph-mon[39462]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:02:55.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:54 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:02:55.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:54 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:02:55.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:54 smithi135 ceph-mon[33238]: osdmap e80: 8 total, 8 up, 8 in 2024-09-17T07:02:55.096 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:54 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:02:55.096 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:54 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:02:55.096 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:54 smithi100 ceph-mon[29092]: osdmap e80: 8 total, 8 up, 8 in 2024-09-17T07:02:55.096 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:54 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:02:55.097 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:54 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:02:55.097 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:54 smithi100 ceph-mon[39462]: osdmap e80: 8 total, 8 up, 8 in 2024-09-17T07:02:56.016 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:02:55 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:02:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:02:56.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:56 smithi135 ceph-mon[33238]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-09-17T07:02:56.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:56 smithi135 ceph-mon[33238]: osdmap e81: 8 total, 8 up, 8 in 2024-09-17T07:02:56.319 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:02:56.376 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:56 smithi100 ceph-mon[39462]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-09-17T07:02:56.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:56 smithi100 ceph-mon[39462]: osdmap e81: 8 total, 8 up, 8 in 2024-09-17T07:02:56.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:56 smithi100 ceph-mon[29092]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-09-17T07:02:56.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:56 smithi100 ceph-mon[29092]: osdmap e81: 8 total, 8 up, 8 in 2024-09-17T07:02:58.126 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:58 smithi100 ceph-mon[29092]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 20 op/s 2024-09-17T07:02:58.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:58 smithi100 ceph-mon[39462]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 20 op/s 2024-09-17T07:02:58.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:58 smithi135 ceph-mon[33238]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 20 op/s 2024-09-17T07:02:58.293 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:02:58 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:02:59.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:02:59 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:02:59.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:02:59 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:02:59.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:02:59 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:03:00.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:00 smithi135 ceph-mon[33238]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 15 op/s 2024-09-17T07:03:00.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:00 smithi135 ceph-mon[33238]: osdmap e82: 8 total, 8 up, 8 in 2024-09-17T07:03:00.309 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:00 smithi100 ceph-mon[29092]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 15 op/s 2024-09-17T07:03:00.309 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:00 smithi100 ceph-mon[29092]: osdmap e82: 8 total, 8 up, 8 in 2024-09-17T07:03:00.309 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:00 smithi100 ceph-mon[39462]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 15 op/s 2024-09-17T07:03:00.309 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:00 smithi100 ceph-mon[39462]: osdmap e82: 8 total, 8 up, 8 in 2024-09-17T07:03:00.440 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:03:01.292 INFO:teuthology.orchestra.run.smithi100.stdout:77309411365 2024-09-17T07:03:01.292 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-09-17T07:03:02.126 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:02 smithi100 ceph-mon[29092]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 706 B/s rd, 0 op/s 2024-09-17T07:03:02.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:02 smithi100 ceph-mon[39462]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 706 B/s rd, 0 op/s 2024-09-17T07:03:02.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:02 smithi135 ceph-mon[33238]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 706 B/s rd, 0 op/s 2024-09-17T07:03:03.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:03 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:03:03.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:03 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:03:03.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:03 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:03:04.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:04 smithi135 ceph-mon[33238]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-17T07:03:04.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:04 smithi100 ceph-mon[29092]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-17T07:03:04.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:04 smithi100 ceph-mon[39462]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-17T07:03:06.063 INFO:teuthology.orchestra.run.smithi100.stdout:214748364818 2024-09-17T07:03:06.064 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-09-17T07:03:06.157 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:06 smithi100 ceph-mon[29092]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-17T07:03:06.157 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:06 smithi100 ceph-mon[29092]: osdmap e83: 8 total, 8 up, 8 in 2024-09-17T07:03:06.157 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:06 smithi100 ceph-mon[39462]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-17T07:03:06.158 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:06 smithi100 ceph-mon[39462]: osdmap e83: 8 total, 8 up, 8 in 2024-09-17T07:03:06.158 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:03:05 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:03:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:03:06.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:06 smithi135 ceph-mon[33238]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-17T07:03:06.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:06 smithi135 ceph-mon[33238]: osdmap e83: 8 total, 8 up, 8 in 2024-09-17T07:03:07.563 INFO:teuthology.orchestra.run.smithi100.stdout:34359738413 2024-09-17T07:03:07.564 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-09-17T07:03:08.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:08 smithi135 ceph-mon[33238]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-17T07:03:08.293 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:03:08 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:03:08.353 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:08 smithi100 ceph-mon[29092]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-17T07:03:08.353 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:08 smithi100 ceph-mon[39462]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-17T07:03:09.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:09 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:03:09.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:09 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:03:09.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:09 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:03:09.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:09 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:03:09.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:09 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:03:09.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:09 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:03:09.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:09 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:03:09.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:09 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:03:09.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:09 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:03:10.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:10 smithi135 ceph-mon[33238]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T07:03:10.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:10 smithi135 ceph-mon[33238]: osdmap e84: 8 total, 8 up, 8 in 2024-09-17T07:03:10.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:10 smithi135 ceph-mon[33238]: osdmap e85: 8 total, 8 up, 8 in 2024-09-17T07:03:10.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:10 smithi100 ceph-mon[29092]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T07:03:10.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:10 smithi100 ceph-mon[29092]: osdmap e84: 8 total, 8 up, 8 in 2024-09-17T07:03:10.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:10 smithi100 ceph-mon[29092]: osdmap e85: 8 total, 8 up, 8 in 2024-09-17T07:03:10.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:10 smithi100 ceph-mon[39462]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T07:03:10.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:10 smithi100 ceph-mon[39462]: osdmap e84: 8 total, 8 up, 8 in 2024-09-17T07:03:10.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:10 smithi100 ceph-mon[39462]: osdmap e85: 8 total, 8 up, 8 in 2024-09-17T07:03:11.405 INFO:teuthology.orchestra.run.smithi100.stdout:141733920799 2024-09-17T07:03:11.405 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-09-17T07:03:12.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:12 smithi135 ceph-mon[33238]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:03:12.357 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:12 smithi100 ceph-mon[29092]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:03:12.357 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:12 smithi100 ceph-mon[39462]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:03:12.633 INFO:teuthology.orchestra.run.smithi100.stdout:180388626457 2024-09-17T07:03:12.634 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-09-17T07:03:13.168 INFO:teuthology.orchestra.run.smithi100.stdout:55834574891 2024-09-17T07:03:13.169 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-09-17T07:03:13.553 INFO:teuthology.orchestra.run.smithi100.stdout:257698037776 2024-09-17T07:03:13.553 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-09-17T07:03:14.002 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:03:14.006 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:03:14.053 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:03:14.066 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:03:14.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:14 smithi135 ceph-mon[33238]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 641 B/s rd, 0 op/s 2024-09-17T07:03:14.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:14 smithi100 ceph-mon[29092]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 641 B/s rd, 0 op/s 2024-09-17T07:03:14.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:14 smithi100 ceph-mon[39462]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 641 B/s rd, 0 op/s 2024-09-17T07:03:15.034 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:03:15.111 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:15 smithi100 ceph-mon[29092]: 2.b scrub starts 2024-09-17T07:03:15.111 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:15 smithi100 ceph-mon[29092]: 2.b scrub ok 2024-09-17T07:03:15.111 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:15 smithi100 ceph-mon[29092]: 4.14 scrub starts 2024-09-17T07:03:15.111 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:15 smithi100 ceph-mon[29092]: 4.14 scrub ok 2024-09-17T07:03:15.111 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:15 smithi100 ceph-mon[29092]: 2.3 scrub starts 2024-09-17T07:03:15.111 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:15 smithi100 ceph-mon[29092]: 2.3 scrub ok 2024-09-17T07:03:15.112 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:15 smithi100 ceph-mon[39462]: 2.b scrub starts 2024-09-17T07:03:15.112 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:15 smithi100 ceph-mon[39462]: 2.b scrub ok 2024-09-17T07:03:15.112 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:15 smithi100 ceph-mon[39462]: 4.14 scrub starts 2024-09-17T07:03:15.112 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:15 smithi100 ceph-mon[39462]: 4.14 scrub ok 2024-09-17T07:03:15.112 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:15 smithi100 ceph-mon[39462]: 2.3 scrub starts 2024-09-17T07:03:15.112 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:15 smithi100 ceph-mon[39462]: 2.3 scrub ok 2024-09-17T07:03:15.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:15 smithi135 ceph-mon[33238]: 2.b scrub starts 2024-09-17T07:03:15.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:15 smithi135 ceph-mon[33238]: 2.b scrub ok 2024-09-17T07:03:15.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:15 smithi135 ceph-mon[33238]: 4.14 scrub starts 2024-09-17T07:03:15.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:15 smithi135 ceph-mon[33238]: 4.14 scrub ok 2024-09-17T07:03:15.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:15 smithi135 ceph-mon[33238]: 2.3 scrub starts 2024-09-17T07:03:15.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:15 smithi135 ceph-mon[33238]: 2.3 scrub ok 2024-09-17T07:03:16.043 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:03:15 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:03:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:03:16.310 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:16 smithi100 ceph-mon[29092]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:03:16.310 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:16 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:03:16.310 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:16 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:03:16.310 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:16 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:03:16.310 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:16 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:03:16.310 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:16 smithi100 ceph-mon[39462]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:03:16.311 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:16 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:03:16.311 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:16 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:03:16.311 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:16 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:03:16.311 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:16 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:03:16.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:16 smithi135 ceph-mon[33238]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:03:16.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:16 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:03:16.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:16 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:03:16.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:16 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:03:16.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:16 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:03:17.111 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:17 smithi100 ceph-mon[29092]: osdmap e86: 8 total, 8 up, 8 in 2024-09-17T07:03:17.111 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:17 smithi100 ceph-mon[29092]: 4.f scrub starts 2024-09-17T07:03:17.112 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:17 smithi100 ceph-mon[29092]: 4.f scrub ok 2024-09-17T07:03:17.112 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:17 smithi100 ceph-mon[39462]: osdmap e86: 8 total, 8 up, 8 in 2024-09-17T07:03:17.112 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:17 smithi100 ceph-mon[39462]: 4.f scrub starts 2024-09-17T07:03:17.112 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:17 smithi100 ceph-mon[39462]: 4.f scrub ok 2024-09-17T07:03:17.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:17 smithi135 ceph-mon[33238]: osdmap e86: 8 total, 8 up, 8 in 2024-09-17T07:03:17.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:17 smithi135 ceph-mon[33238]: 4.f scrub starts 2024-09-17T07:03:17.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:17 smithi135 ceph-mon[33238]: 4.f scrub ok 2024-09-17T07:03:18.293 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:03:18 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:03:18.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:18 smithi135 ceph-mon[33238]: 2.7 scrub starts 2024-09-17T07:03:18.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:18 smithi135 ceph-mon[33238]: 2.7 scrub ok 2024-09-17T07:03:18.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:18 smithi135 ceph-mon[33238]: 4.1c scrub starts 2024-09-17T07:03:18.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:18 smithi135 ceph-mon[33238]: 4.1c scrub ok 2024-09-17T07:03:18.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:18 smithi135 ceph-mon[33238]: 2.13 scrub starts 2024-09-17T07:03:18.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:18 smithi135 ceph-mon[33238]: 2.13 scrub ok 2024-09-17T07:03:18.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:18 smithi135 ceph-mon[33238]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 641 B/s rd, 0 op/s 2024-09-17T07:03:18.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:18 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:03:18.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:18 smithi135 ceph-mon[33238]: 4.11 scrub starts 2024-09-17T07:03:18.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:18 smithi135 ceph-mon[33238]: 4.11 scrub ok 2024-09-17T07:03:18.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:18 smithi135 ceph-mon[33238]: 4.15 scrub starts 2024-09-17T07:03:18.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:18 smithi135 ceph-mon[33238]: 4.15 scrub ok 2024-09-17T07:03:18.377 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:Sep 17 07:03:18 smithi100 podman[71018]: 2024-09-17 07:03:18.107547048 +0000 UTC m=+3.089761714 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-09-17T07:03:18.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:18 smithi100 ceph-mon[29092]: 2.7 scrub starts 2024-09-17T07:03:18.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:18 smithi100 ceph-mon[29092]: 2.7 scrub ok 2024-09-17T07:03:18.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:18 smithi100 ceph-mon[29092]: 4.1c scrub starts 2024-09-17T07:03:18.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:18 smithi100 ceph-mon[29092]: 4.1c scrub ok 2024-09-17T07:03:18.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:18 smithi100 ceph-mon[29092]: 2.13 scrub starts 2024-09-17T07:03:18.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:18 smithi100 ceph-mon[29092]: 2.13 scrub ok 2024-09-17T07:03:18.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:18 smithi100 ceph-mon[29092]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 641 B/s rd, 0 op/s 2024-09-17T07:03:18.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:18 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:03:18.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:18 smithi100 ceph-mon[29092]: 4.11 scrub starts 2024-09-17T07:03:18.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:18 smithi100 ceph-mon[29092]: 4.11 scrub ok 2024-09-17T07:03:18.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:18 smithi100 ceph-mon[29092]: 4.15 scrub starts 2024-09-17T07:03:18.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:18 smithi100 ceph-mon[29092]: 4.15 scrub ok 2024-09-17T07:03:18.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:18 smithi100 ceph-mon[39462]: 2.7 scrub starts 2024-09-17T07:03:18.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:18 smithi100 ceph-mon[39462]: 2.7 scrub ok 2024-09-17T07:03:18.379 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:18 smithi100 ceph-mon[39462]: 4.1c scrub starts 2024-09-17T07:03:18.379 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:18 smithi100 ceph-mon[39462]: 4.1c scrub ok 2024-09-17T07:03:18.379 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:18 smithi100 ceph-mon[39462]: 2.13 scrub starts 2024-09-17T07:03:18.379 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:18 smithi100 ceph-mon[39462]: 2.13 scrub ok 2024-09-17T07:03:18.379 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:18 smithi100 ceph-mon[39462]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 641 B/s rd, 0 op/s 2024-09-17T07:03:18.379 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:18 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:03:18.379 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:18 smithi100 ceph-mon[39462]: 4.11 scrub starts 2024-09-17T07:03:18.379 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:18 smithi100 ceph-mon[39462]: 4.11 scrub ok 2024-09-17T07:03:18.379 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:18 smithi100 ceph-mon[39462]: 4.15 scrub starts 2024-09-17T07:03:18.379 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:18 smithi100 ceph-mon[39462]: 4.15 scrub ok 2024-09-17T07:03:18.580 INFO:teuthology.orchestra.run.smithi100.stdout:115964117026 2024-09-17T07:03:18.580 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-09-17T07:03:19.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:19 smithi135 ceph-mon[33238]: 4.18 scrub starts 2024-09-17T07:03:19.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:19 smithi135 ceph-mon[33238]: 4.18 scrub ok 2024-09-17T07:03:19.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:19 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:03:19.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:19 smithi135 ceph-mon[33238]: 4.1d deep-scrub starts 2024-09-17T07:03:19.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:19 smithi135 ceph-mon[33238]: 4.1d deep-scrub ok 2024-09-17T07:03:19.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:19 smithi135 ceph-mon[33238]: 5.f scrub starts 2024-09-17T07:03:19.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:19 smithi135 ceph-mon[33238]: 5.f scrub ok 2024-09-17T07:03:19.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:19 smithi100 ceph-mon[29092]: 4.18 scrub starts 2024-09-17T07:03:19.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:19 smithi100 ceph-mon[29092]: 4.18 scrub ok 2024-09-17T07:03:19.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:19 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:03:19.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:19 smithi100 ceph-mon[29092]: 4.1d deep-scrub starts 2024-09-17T07:03:19.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:19 smithi100 ceph-mon[29092]: 4.1d deep-scrub ok 2024-09-17T07:03:19.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:19 smithi100 ceph-mon[29092]: 5.f scrub starts 2024-09-17T07:03:19.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:19 smithi100 ceph-mon[29092]: 5.f scrub ok 2024-09-17T07:03:19.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:19 smithi100 ceph-mon[39462]: 4.18 scrub starts 2024-09-17T07:03:19.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:19 smithi100 ceph-mon[39462]: 4.18 scrub ok 2024-09-17T07:03:19.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:19 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:03:19.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:19 smithi100 ceph-mon[39462]: 4.1d deep-scrub starts 2024-09-17T07:03:19.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:19 smithi100 ceph-mon[39462]: 4.1d deep-scrub ok 2024-09-17T07:03:19.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:19 smithi100 ceph-mon[39462]: 5.f scrub starts 2024-09-17T07:03:19.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:19 smithi100 ceph-mon[39462]: 5.f scrub ok 2024-09-17T07:03:20.327 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:20 smithi100 ceph-mon[29092]: 4.13 scrub starts 2024-09-17T07:03:20.327 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:20 smithi100 ceph-mon[29092]: 4.13 scrub ok 2024-09-17T07:03:20.328 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:20 smithi100 ceph-mon[29092]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T07:03:20.328 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:20 smithi100 ceph-mon[29092]: 6.3 scrub starts 2024-09-17T07:03:20.328 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:20 smithi100 ceph-mon[29092]: 6.3 scrub ok 2024-09-17T07:03:20.328 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:20 smithi100 ceph-mon[29092]: 2.18 scrub starts 2024-09-17T07:03:20.328 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:20 smithi100 ceph-mon[29092]: 2.18 scrub ok 2024-09-17T07:03:20.328 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:20 smithi100 ceph-mon[29092]: osdmap e87: 8 total, 8 up, 8 in 2024-09-17T07:03:20.328 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:20 smithi100 ceph-mon[39462]: 4.13 scrub starts 2024-09-17T07:03:20.328 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:20 smithi100 ceph-mon[39462]: 4.13 scrub ok 2024-09-17T07:03:20.328 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:20 smithi100 ceph-mon[39462]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T07:03:20.329 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:20 smithi100 ceph-mon[39462]: 6.3 scrub starts 2024-09-17T07:03:20.329 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:20 smithi100 ceph-mon[39462]: 6.3 scrub ok 2024-09-17T07:03:20.329 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:20 smithi100 ceph-mon[39462]: 2.18 scrub starts 2024-09-17T07:03:20.329 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:20 smithi100 ceph-mon[39462]: 2.18 scrub ok 2024-09-17T07:03:20.329 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:20 smithi100 ceph-mon[39462]: osdmap e87: 8 total, 8 up, 8 in 2024-09-17T07:03:20.462 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:03:20.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:20 smithi135 ceph-mon[33238]: 4.13 scrub starts 2024-09-17T07:03:20.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:20 smithi135 ceph-mon[33238]: 4.13 scrub ok 2024-09-17T07:03:20.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:20 smithi135 ceph-mon[33238]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T07:03:20.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:20 smithi135 ceph-mon[33238]: 6.3 scrub starts 2024-09-17T07:03:20.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:20 smithi135 ceph-mon[33238]: 6.3 scrub ok 2024-09-17T07:03:20.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:20 smithi135 ceph-mon[33238]: 2.18 scrub starts 2024-09-17T07:03:20.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:20 smithi135 ceph-mon[33238]: 2.18 scrub ok 2024-09-17T07:03:20.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:20 smithi135 ceph-mon[33238]: osdmap e87: 8 total, 8 up, 8 in 2024-09-17T07:03:20.626 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:Sep 17 07:03:20 smithi100 podman[71018]: 2024-09-17 07:03:20.571394953 +0000 UTC m=+5.553609619 volume create c914f2c1b5d318a8a4498f8c4a94e98c4b1983be48c75042ec9d823ee4de57e3 2024-09-17T07:03:20.948 INFO:teuthology.orchestra.run.smithi100.stdout:257698037779 2024-09-17T07:03:20.948 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:Sep 17 07:03:20 smithi100 podman[71018]: 2024-09-17 07:03:20.906317363 +0000 UTC m=+5.888532027 container create 5557c6374eb7a1231bc38450b6663e0251acfdeb55fe8758e86971deb99bb341 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-17T07:03:21.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:21 smithi100 ceph-mon[29092]: 6.14 scrub starts 2024-09-17T07:03:21.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:21 smithi100 ceph-mon[29092]: 6.14 scrub ok 2024-09-17T07:03:21.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:21 smithi100 ceph-mon[29092]: 4.12 scrub starts 2024-09-17T07:03:21.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:21 smithi100 ceph-mon[29092]: 4.12 scrub ok 2024-09-17T07:03:21.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:21 smithi100 ceph-mon[29092]: 2.19 scrub starts 2024-09-17T07:03:21.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:21 smithi100 ceph-mon[29092]: 2.19 scrub ok 2024-09-17T07:03:21.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:21 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/4283146701' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-17T07:03:21.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:21 smithi100 ceph-mon[39462]: 6.14 scrub starts 2024-09-17T07:03:21.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:21 smithi100 ceph-mon[39462]: 6.14 scrub ok 2024-09-17T07:03:21.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:21 smithi100 ceph-mon[39462]: 4.12 scrub starts 2024-09-17T07:03:21.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:21 smithi100 ceph-mon[39462]: 4.12 scrub ok 2024-09-17T07:03:21.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:21 smithi100 ceph-mon[39462]: 2.19 scrub starts 2024-09-17T07:03:21.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:21 smithi100 ceph-mon[39462]: 2.19 scrub ok 2024-09-17T07:03:21.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:21 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/4283146701' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-17T07:03:21.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:21 smithi135 ceph-mon[33238]: 6.14 scrub starts 2024-09-17T07:03:21.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:21 smithi135 ceph-mon[33238]: 6.14 scrub ok 2024-09-17T07:03:21.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:21 smithi135 ceph-mon[33238]: 4.12 scrub starts 2024-09-17T07:03:21.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:21 smithi135 ceph-mon[33238]: 4.12 scrub ok 2024-09-17T07:03:21.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:21 smithi135 ceph-mon[33238]: 2.19 scrub starts 2024-09-17T07:03:21.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:21 smithi135 ceph-mon[33238]: 2.19 scrub ok 2024-09-17T07:03:21.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:21 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/4283146701' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-17T07:03:21.838 INFO:tasks.cephadm.ceph_manager.ceph:need seq 257698037776 got 257698037779 for osd.7 2024-09-17T07:03:21.839 DEBUG:teuthology.parallel:result is None 2024-09-17T07:03:22.337 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:22 smithi100 ceph-mon[29092]: 4.c scrub starts 2024-09-17T07:03:22.337 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:22 smithi100 ceph-mon[29092]: 4.c scrub ok 2024-09-17T07:03:22.337 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:22 smithi100 ceph-mon[29092]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:03:22.337 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:22 smithi100 ceph-mon[29092]: 5.17 scrub starts 2024-09-17T07:03:22.337 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:22 smithi100 ceph-mon[29092]: 5.17 scrub ok 2024-09-17T07:03:22.337 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:22 smithi100 ceph-mon[39462]: 4.c scrub starts 2024-09-17T07:03:22.337 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:22 smithi100 ceph-mon[39462]: 4.c scrub ok 2024-09-17T07:03:22.338 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:22 smithi100 ceph-mon[39462]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:03:22.338 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:22 smithi100 ceph-mon[39462]: 5.17 scrub starts 2024-09-17T07:03:22.338 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:22 smithi100 ceph-mon[39462]: 5.17 scrub ok 2024-09-17T07:03:22.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:22 smithi135 ceph-mon[33238]: 4.c scrub starts 2024-09-17T07:03:22.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:22 smithi135 ceph-mon[33238]: 4.c scrub ok 2024-09-17T07:03:22.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:22 smithi135 ceph-mon[33238]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:03:22.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:22 smithi135 ceph-mon[33238]: 5.17 scrub starts 2024-09-17T07:03:22.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:22 smithi135 ceph-mon[33238]: 5.17 scrub ok 2024-09-17T07:03:22.627 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:Sep 17 07:03:22 smithi100 podman[71018]: 2024-09-17 07:03:22.46993833 +0000 UTC m=+7.452152995 container init 5557c6374eb7a1231bc38450b6663e0251acfdeb55fe8758e86971deb99bb341 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-17T07:03:22.627 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:Sep 17 07:03:22 smithi100 podman[71018]: 2024-09-17 07:03:22.473339956 +0000 UTC m=+7.455554622 container start 5557c6374eb7a1231bc38450b6663e0251acfdeb55fe8758e86971deb99bb341 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-17T07:03:22.627 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:Sep 17 07:03:22 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-alertmanager-a[71357]: ts=2024-09-17T07:03:22.485Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-09-17T07:03:22.627 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:Sep 17 07:03:22 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-alertmanager-a[71357]: ts=2024-09-17T07:03:22.485Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-09-17T07:03:22.627 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:Sep 17 07:03:22 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-alertmanager-a[71357]: ts=2024-09-17T07:03:22.487Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.100 port=9094 2024-09-17T07:03:22.627 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:Sep 17 07:03:22 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-alertmanager-a[71357]: ts=2024-09-17T07:03:22.488Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-09-17T07:03:22.627 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:Sep 17 07:03:22 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-alertmanager-a[71357]: ts=2024-09-17T07:03:22.511Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-17T07:03:22.628 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:Sep 17 07:03:22 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-alertmanager-a[71357]: ts=2024-09-17T07:03:22.511Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-17T07:03:22.628 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:Sep 17 07:03:22 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-alertmanager-a[71357]: ts=2024-09-17T07:03:22.514Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-09-17T07:03:22.628 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:Sep 17 07:03:22 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-alertmanager-a[71357]: ts=2024-09-17T07:03:22.514Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-09-17T07:03:22.628 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:Sep 17 07:03:22 smithi100 bash[71018]: 5557c6374eb7a1231bc38450b6663e0251acfdeb55fe8758e86971deb99bb341 2024-09-17T07:03:22.628 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:Sep 17 07:03:22 smithi100 systemd[1]: Started Ceph alertmanager.a for e848267c-74c1-11ef-bceb-c7b262605968. 2024-09-17T07:03:22.965 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:03:22 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: [17/Sep/2024:07:03:22] ENGINE Bus STOPPING 2024-09-17T07:03:22.966 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:03:22 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: [17/Sep/2024:07:03:22] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-17T07:03:22.966 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:03:22 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: [17/Sep/2024:07:03:22] ENGINE Bus STOPPED 2024-09-17T07:03:22.966 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:03:22 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: [17/Sep/2024:07:03:22] ENGINE Bus STARTING 2024-09-17T07:03:23.222 INFO:teuthology.orchestra.run.smithi100.stdout:141733920801 2024-09-17T07:03:23.223 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:03:23 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: [17/Sep/2024:07:03:23] ENGINE Serving on http://:::9283 2024-09-17T07:03:23.223 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:03:23 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: [17/Sep/2024:07:03:23] ENGINE Bus STARTED 2024-09-17T07:03:23.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:23 smithi100 ceph-mon[39462]: 5.1a scrub starts 2024-09-17T07:03:23.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:23 smithi100 ceph-mon[39462]: 5.1a scrub ok 2024-09-17T07:03:23.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:23 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:23.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:23 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:23.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:23 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:23.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:23 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:23.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:23 smithi100 ceph-mon[39462]: Regenerating cephadm self-signed grafana TLS certificates 2024-09-17T07:03:23.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:23 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:23.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:23 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:23.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:23 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-17T07:03:23.879 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:23 smithi100 ceph-mon[39462]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-17T07:03:23.879 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:23 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:23.879 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:23 smithi100 ceph-mon[39462]: Deploying daemon grafana.a on smithi135 2024-09-17T07:03:23.879 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:23 smithi100 ceph-mon[39462]: 6.a scrub starts 2024-09-17T07:03:23.879 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:23 smithi100 ceph-mon[39462]: 6.a scrub ok 2024-09-17T07:03:23.880 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:23 smithi100 ceph-mon[39462]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-17T07:03:23.880 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:23 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2335644951' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-17T07:03:23.880 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:23 smithi100 ceph-mon[29092]: 5.1a scrub starts 2024-09-17T07:03:23.880 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:23 smithi100 ceph-mon[29092]: 5.1a scrub ok 2024-09-17T07:03:23.881 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:23 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:23.881 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:23 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:23.881 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:23 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:23.881 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:23 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:23.881 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:23 smithi100 ceph-mon[29092]: Regenerating cephadm self-signed grafana TLS certificates 2024-09-17T07:03:23.881 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:23 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:23.882 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:23 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:23.882 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:23 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-17T07:03:23.882 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:23 smithi100 ceph-mon[29092]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-17T07:03:23.882 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:23 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:23.882 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:23 smithi100 ceph-mon[29092]: Deploying daemon grafana.a on smithi135 2024-09-17T07:03:23.882 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:23 smithi100 ceph-mon[29092]: 6.a scrub starts 2024-09-17T07:03:23.883 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:23 smithi100 ceph-mon[29092]: 6.a scrub ok 2024-09-17T07:03:23.883 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:23 smithi100 ceph-mon[29092]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-17T07:03:23.883 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:23 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2335644951' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-17T07:03:24.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:23 smithi135 ceph-mon[33238]: 5.1a scrub starts 2024-09-17T07:03:24.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:23 smithi135 ceph-mon[33238]: 5.1a scrub ok 2024-09-17T07:03:24.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:23 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:24.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:23 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:24.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:23 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:24.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:23 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:24.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:23 smithi135 ceph-mon[33238]: Regenerating cephadm self-signed grafana TLS certificates 2024-09-17T07:03:24.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:23 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:24.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:23 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:24.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:23 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-17T07:03:24.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:23 smithi135 ceph-mon[33238]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-17T07:03:24.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:23 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:24.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:23 smithi135 ceph-mon[33238]: Deploying daemon grafana.a on smithi135 2024-09-17T07:03:24.045 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:23 smithi135 ceph-mon[33238]: 6.a scrub starts 2024-09-17T07:03:24.045 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:23 smithi135 ceph-mon[33238]: 6.a scrub ok 2024-09-17T07:03:24.045 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:23 smithi135 ceph-mon[33238]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-17T07:03:24.045 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:23 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2335644951' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-17T07:03:24.519 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:Sep 17 07:03:24 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-alertmanager-a[71357]: ts=2024-09-17T07:03:24.489Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.00090718s 2024-09-17T07:03:24.520 INFO:teuthology.orchestra.run.smithi100.stdout:77309411370 2024-09-17T07:03:24.850 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:24 smithi100 ceph-mon[29092]: 6.10 scrub starts 2024-09-17T07:03:24.850 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:24 smithi100 ceph-mon[29092]: 6.10 scrub ok 2024-09-17T07:03:24.850 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:24 smithi100 ceph-mon[29092]: 4.10 scrub starts 2024-09-17T07:03:24.850 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:24 smithi100 ceph-mon[29092]: 4.10 scrub ok 2024-09-17T07:03:24.850 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:24 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2891603221' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-17T07:03:24.850 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:24 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:03:24.851 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:24 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:03:24.851 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:24 smithi100 ceph-mon[39462]: 6.10 scrub starts 2024-09-17T07:03:24.851 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:24 smithi100 ceph-mon[39462]: 6.10 scrub ok 2024-09-17T07:03:24.851 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:24 smithi100 ceph-mon[39462]: 4.10 scrub starts 2024-09-17T07:03:24.851 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:24 smithi100 ceph-mon[39462]: 4.10 scrub ok 2024-09-17T07:03:24.851 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:24 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2891603221' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-17T07:03:24.851 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:24 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:03:24.851 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:24 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:03:25.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:24 smithi135 ceph-mon[33238]: 6.10 scrub starts 2024-09-17T07:03:25.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:24 smithi135 ceph-mon[33238]: 6.10 scrub ok 2024-09-17T07:03:25.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:24 smithi135 ceph-mon[33238]: 4.10 scrub starts 2024-09-17T07:03:25.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:24 smithi135 ceph-mon[33238]: 4.10 scrub ok 2024-09-17T07:03:25.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:24 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2891603221' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-17T07:03:25.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:24 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:03:25.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:24 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:03:25.188 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411365 got 77309411370 for osd.2 2024-09-17T07:03:25.188 DEBUG:teuthology.parallel:result is None 2024-09-17T07:03:25.192 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:03:26.015 INFO:teuthology.orchestra.run.smithi100.stdout:214748364823 2024-09-17T07:03:26.016 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:03:25 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:03:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:03:26.016 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:25 smithi100 ceph-mon[29092]: 2.6 scrub starts 2024-09-17T07:03:26.016 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:25 smithi100 ceph-mon[29092]: 2.6 scrub ok 2024-09-17T07:03:26.016 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:25 smithi100 ceph-mon[29092]: 4.1f scrub starts 2024-09-17T07:03:26.016 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:25 smithi100 ceph-mon[29092]: 4.1f scrub ok 2024-09-17T07:03:26.016 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:25 smithi100 ceph-mon[29092]: osdmap e88: 8 total, 8 up, 8 in 2024-09-17T07:03:26.016 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:25 smithi100 ceph-mon[29092]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:03:26.017 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:25 smithi100 ceph-mon[29092]: osdmap e89: 8 total, 8 up, 8 in 2024-09-17T07:03:26.017 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:25 smithi100 ceph-mon[29092]: 6.0 scrub starts 2024-09-17T07:03:26.017 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:25 smithi100 ceph-mon[29092]: 6.0 scrub ok 2024-09-17T07:03:26.017 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:25 smithi100 ceph-mon[29092]: 3.2 scrub starts 2024-09-17T07:03:26.017 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:25 smithi100 ceph-mon[29092]: 3.2 scrub ok 2024-09-17T07:03:26.017 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:25 smithi100 ceph-mon[39462]: 2.6 scrub starts 2024-09-17T07:03:26.017 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:25 smithi100 ceph-mon[39462]: 2.6 scrub ok 2024-09-17T07:03:26.017 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:25 smithi100 ceph-mon[39462]: 4.1f scrub starts 2024-09-17T07:03:26.017 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:25 smithi100 ceph-mon[39462]: 4.1f scrub ok 2024-09-17T07:03:26.017 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:25 smithi100 ceph-mon[39462]: osdmap e88: 8 total, 8 up, 8 in 2024-09-17T07:03:26.017 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:25 smithi100 ceph-mon[39462]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:03:26.018 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:25 smithi100 ceph-mon[39462]: osdmap e89: 8 total, 8 up, 8 in 2024-09-17T07:03:26.018 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:25 smithi100 ceph-mon[39462]: 6.0 scrub starts 2024-09-17T07:03:26.018 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:25 smithi100 ceph-mon[39462]: 6.0 scrub ok 2024-09-17T07:03:26.018 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:25 smithi100 ceph-mon[39462]: 3.2 scrub starts 2024-09-17T07:03:26.018 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:25 smithi100 ceph-mon[39462]: 3.2 scrub ok 2024-09-17T07:03:26.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:25 smithi135 ceph-mon[33238]: 2.6 scrub starts 2024-09-17T07:03:26.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:25 smithi135 ceph-mon[33238]: 2.6 scrub ok 2024-09-17T07:03:26.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:25 smithi135 ceph-mon[33238]: 4.1f scrub starts 2024-09-17T07:03:26.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:25 smithi135 ceph-mon[33238]: 4.1f scrub ok 2024-09-17T07:03:26.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:25 smithi135 ceph-mon[33238]: osdmap e88: 8 total, 8 up, 8 in 2024-09-17T07:03:26.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:25 smithi135 ceph-mon[33238]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:03:26.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:25 smithi135 ceph-mon[33238]: osdmap e89: 8 total, 8 up, 8 in 2024-09-17T07:03:26.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:25 smithi135 ceph-mon[33238]: 6.0 scrub starts 2024-09-17T07:03:26.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:25 smithi135 ceph-mon[33238]: 6.0 scrub ok 2024-09-17T07:03:26.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:25 smithi135 ceph-mon[33238]: 3.2 scrub starts 2024-09-17T07:03:26.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:25 smithi135 ceph-mon[33238]: 3.2 scrub ok 2024-09-17T07:03:26.621 INFO:tasks.cephadm.ceph_manager.ceph:need seq 214748364818 got 214748364823 for osd.6 2024-09-17T07:03:26.622 DEBUG:teuthology.parallel:result is None 2024-09-17T07:03:26.848 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:26 smithi100 ceph-mon[29092]: 2.4 deep-scrub starts 2024-09-17T07:03:26.848 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:26 smithi100 ceph-mon[29092]: 2.4 deep-scrub ok 2024-09-17T07:03:26.849 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:26 smithi100 ceph-mon[29092]: 2.c scrub starts 2024-09-17T07:03:26.849 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:26 smithi100 ceph-mon[29092]: 2.c scrub ok 2024-09-17T07:03:26.849 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:26 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/4050614852' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-17T07:03:26.849 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:26 smithi100 ceph-mon[29092]: 4.16 scrub starts 2024-09-17T07:03:26.849 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:26 smithi100 ceph-mon[29092]: 4.16 scrub ok 2024-09-17T07:03:26.849 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:26 smithi100 ceph-mon[29092]: 4.0 scrub starts 2024-09-17T07:03:26.849 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:26 smithi100 ceph-mon[29092]: 4.0 scrub ok 2024-09-17T07:03:26.849 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:26 smithi100 ceph-mon[39462]: 2.4 deep-scrub starts 2024-09-17T07:03:26.849 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:26 smithi100 ceph-mon[39462]: 2.4 deep-scrub ok 2024-09-17T07:03:26.849 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:26 smithi100 ceph-mon[39462]: 2.c scrub starts 2024-09-17T07:03:26.849 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:26 smithi100 ceph-mon[39462]: 2.c scrub ok 2024-09-17T07:03:26.850 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:26 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/4050614852' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-17T07:03:26.850 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:26 smithi100 ceph-mon[39462]: 4.16 scrub starts 2024-09-17T07:03:26.850 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:26 smithi100 ceph-mon[39462]: 4.16 scrub ok 2024-09-17T07:03:26.850 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:26 smithi100 ceph-mon[39462]: 4.0 scrub starts 2024-09-17T07:03:26.850 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:26 smithi100 ceph-mon[39462]: 4.0 scrub ok 2024-09-17T07:03:26.947 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:03:27.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:26 smithi135 ceph-mon[33238]: 2.4 deep-scrub starts 2024-09-17T07:03:27.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:26 smithi135 ceph-mon[33238]: 2.4 deep-scrub ok 2024-09-17T07:03:27.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:26 smithi135 ceph-mon[33238]: 2.c scrub starts 2024-09-17T07:03:27.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:26 smithi135 ceph-mon[33238]: 2.c scrub ok 2024-09-17T07:03:27.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:26 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/4050614852' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-17T07:03:27.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:26 smithi135 ceph-mon[33238]: 4.16 scrub starts 2024-09-17T07:03:27.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:26 smithi135 ceph-mon[33238]: 4.16 scrub ok 2024-09-17T07:03:27.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:26 smithi135 ceph-mon[33238]: 4.0 scrub starts 2024-09-17T07:03:27.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:26 smithi135 ceph-mon[33238]: 4.0 scrub ok 2024-09-17T07:03:27.388 INFO:teuthology.orchestra.run.smithi100.stdout:34359738418 2024-09-17T07:03:27.568 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920799 got 141733920801 for osd.4 2024-09-17T07:03:27.569 DEBUG:teuthology.parallel:result is None 2024-09-17T07:03:28.041 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:27 smithi135 ceph-mon[33238]: 2.12 scrub starts 2024-09-17T07:03:28.042 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:27 smithi135 ceph-mon[33238]: 2.12 scrub ok 2024-09-17T07:03:28.042 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:27 smithi135 ceph-mon[33238]: 4.2 scrub starts 2024-09-17T07:03:28.042 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:27 smithi135 ceph-mon[33238]: 4.2 scrub ok 2024-09-17T07:03:28.042 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:27 smithi135 ceph-mon[33238]: 2.e deep-scrub starts 2024-09-17T07:03:28.042 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:27 smithi135 ceph-mon[33238]: 2.e deep-scrub ok 2024-09-17T07:03:28.042 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:27 smithi135 ceph-mon[33238]: 5.1c scrub starts 2024-09-17T07:03:28.042 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:27 smithi135 ceph-mon[33238]: 5.1c scrub ok 2024-09-17T07:03:28.042 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:27 smithi135 ceph-mon[33238]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 732 B/s rd, 0 op/s 2024-09-17T07:03:28.042 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:27 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:28.042 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:27 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/785868571' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-17T07:03:28.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:27 smithi100 ceph-mon[29092]: 2.12 scrub starts 2024-09-17T07:03:28.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:27 smithi100 ceph-mon[29092]: 2.12 scrub ok 2024-09-17T07:03:28.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:27 smithi100 ceph-mon[29092]: 4.2 scrub starts 2024-09-17T07:03:28.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:27 smithi100 ceph-mon[29092]: 4.2 scrub ok 2024-09-17T07:03:28.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:27 smithi100 ceph-mon[29092]: 2.e deep-scrub starts 2024-09-17T07:03:28.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:27 smithi100 ceph-mon[29092]: 2.e deep-scrub ok 2024-09-17T07:03:28.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:27 smithi100 ceph-mon[29092]: 5.1c scrub starts 2024-09-17T07:03:28.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:27 smithi100 ceph-mon[29092]: 5.1c scrub ok 2024-09-17T07:03:28.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:27 smithi100 ceph-mon[29092]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 732 B/s rd, 0 op/s 2024-09-17T07:03:28.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:27 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:28.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:27 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/785868571' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-17T07:03:28.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:27 smithi100 ceph-mon[39462]: 2.12 scrub starts 2024-09-17T07:03:28.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:27 smithi100 ceph-mon[39462]: 2.12 scrub ok 2024-09-17T07:03:28.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:27 smithi100 ceph-mon[39462]: 4.2 scrub starts 2024-09-17T07:03:28.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:27 smithi100 ceph-mon[39462]: 4.2 scrub ok 2024-09-17T07:03:28.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:27 smithi100 ceph-mon[39462]: 2.e deep-scrub starts 2024-09-17T07:03:28.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:27 smithi100 ceph-mon[39462]: 2.e deep-scrub ok 2024-09-17T07:03:28.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:27 smithi100 ceph-mon[39462]: 5.1c scrub starts 2024-09-17T07:03:28.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:27 smithi100 ceph-mon[39462]: 5.1c scrub ok 2024-09-17T07:03:28.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:27 smithi100 ceph-mon[39462]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 732 B/s rd, 0 op/s 2024-09-17T07:03:28.129 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:27 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:28.129 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:27 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/785868571' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-17T07:03:28.293 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:03:28 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:03:28.298 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738413 got 34359738418 for osd.0 2024-09-17T07:03:28.298 DEBUG:teuthology.parallel:result is None 2024-09-17T07:03:28.925 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:28 smithi100 ceph-mon[39462]: 5.b scrub starts 2024-09-17T07:03:28.925 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:28 smithi100 ceph-mon[39462]: 5.b scrub ok 2024-09-17T07:03:28.925 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:28 smithi100 ceph-mon[39462]: 4.6 scrub starts 2024-09-17T07:03:28.925 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:28 smithi100 ceph-mon[39462]: 4.6 scrub ok 2024-09-17T07:03:28.925 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:28 smithi100 ceph-mon[39462]: 2.f scrub starts 2024-09-17T07:03:28.925 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:28 smithi100 ceph-mon[39462]: 2.f scrub ok 2024-09-17T07:03:28.925 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:28 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:03:28.926 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:28 smithi100 ceph-mon[29092]: 5.b scrub starts 2024-09-17T07:03:28.926 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:28 smithi100 ceph-mon[29092]: 5.b scrub ok 2024-09-17T07:03:28.926 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:28 smithi100 ceph-mon[29092]: 4.6 scrub starts 2024-09-17T07:03:28.926 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:28 smithi100 ceph-mon[29092]: 4.6 scrub ok 2024-09-17T07:03:28.926 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:28 smithi100 ceph-mon[29092]: 2.f scrub starts 2024-09-17T07:03:28.926 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:28 smithi100 ceph-mon[29092]: 2.f scrub ok 2024-09-17T07:03:28.926 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:28 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:03:29.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:28 smithi135 ceph-mon[33238]: 5.b scrub starts 2024-09-17T07:03:29.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:28 smithi135 ceph-mon[33238]: 5.b scrub ok 2024-09-17T07:03:29.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:28 smithi135 ceph-mon[33238]: 4.6 scrub starts 2024-09-17T07:03:29.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:28 smithi135 ceph-mon[33238]: 4.6 scrub ok 2024-09-17T07:03:29.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:28 smithi135 ceph-mon[33238]: 2.f scrub starts 2024-09-17T07:03:29.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:28 smithi135 ceph-mon[33238]: 2.f scrub ok 2024-09-17T07:03:29.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:28 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:03:29.874 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:29 smithi100 ceph-mon[29092]: 2.9 scrub starts 2024-09-17T07:03:29.874 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:29 smithi100 ceph-mon[29092]: 2.9 scrub ok 2024-09-17T07:03:29.874 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:29 smithi100 ceph-mon[29092]: 5.5 scrub starts 2024-09-17T07:03:29.874 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:29 smithi100 ceph-mon[29092]: 5.5 scrub ok 2024-09-17T07:03:29.874 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:29 smithi100 ceph-mon[29092]: 4.1b scrub starts 2024-09-17T07:03:29.874 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:29 smithi100 ceph-mon[29092]: 4.1b scrub ok 2024-09-17T07:03:29.874 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:29 smithi100 ceph-mon[29092]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:03:29.875 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:29 smithi100 ceph-mon[39462]: 2.9 scrub starts 2024-09-17T07:03:29.875 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:29 smithi100 ceph-mon[39462]: 2.9 scrub ok 2024-09-17T07:03:29.875 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:29 smithi100 ceph-mon[39462]: 5.5 scrub starts 2024-09-17T07:03:29.875 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:29 smithi100 ceph-mon[39462]: 5.5 scrub ok 2024-09-17T07:03:29.875 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:29 smithi100 ceph-mon[39462]: 4.1b scrub starts 2024-09-17T07:03:29.875 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:29 smithi100 ceph-mon[39462]: 4.1b scrub ok 2024-09-17T07:03:29.875 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:29 smithi100 ceph-mon[39462]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:03:30.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:29 smithi135 ceph-mon[33238]: 2.9 scrub starts 2024-09-17T07:03:30.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:29 smithi135 ceph-mon[33238]: 2.9 scrub ok 2024-09-17T07:03:30.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:29 smithi135 ceph-mon[33238]: 5.5 scrub starts 2024-09-17T07:03:30.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:29 smithi135 ceph-mon[33238]: 5.5 scrub ok 2024-09-17T07:03:30.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:29 smithi135 ceph-mon[33238]: 4.1b scrub starts 2024-09-17T07:03:30.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:29 smithi135 ceph-mon[33238]: 4.1b scrub ok 2024-09-17T07:03:30.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:29 smithi135 ceph-mon[33238]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:03:30.140 INFO:teuthology.orchestra.run.smithi100.stdout:180388626461 2024-09-17T07:03:30.886 INFO:tasks.cephadm.ceph_manager.ceph:need seq 180388626457 got 180388626461 for osd.5 2024-09-17T07:03:30.886 DEBUG:teuthology.parallel:result is None 2024-09-17T07:03:31.013 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:30 smithi100 ceph-mon[29092]: 6.13 scrub starts 2024-09-17T07:03:31.013 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:30 smithi100 ceph-mon[29092]: 6.13 scrub ok 2024-09-17T07:03:31.013 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:30 smithi100 ceph-mon[29092]: 5.3 deep-scrub starts 2024-09-17T07:03:31.013 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:30 smithi100 ceph-mon[29092]: 5.3 deep-scrub ok 2024-09-17T07:03:31.014 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:30 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2178923757' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-17T07:03:31.014 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:30 smithi100 ceph-mon[29092]: 4.19 scrub starts 2024-09-17T07:03:31.014 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:30 smithi100 ceph-mon[29092]: 4.19 scrub ok 2024-09-17T07:03:31.014 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:30 smithi100 ceph-mon[39462]: 6.13 scrub starts 2024-09-17T07:03:31.014 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:30 smithi100 ceph-mon[39462]: 6.13 scrub ok 2024-09-17T07:03:31.014 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:30 smithi100 ceph-mon[39462]: 5.3 deep-scrub starts 2024-09-17T07:03:31.014 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:30 smithi100 ceph-mon[39462]: 5.3 deep-scrub ok 2024-09-17T07:03:31.014 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:30 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2178923757' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-17T07:03:31.015 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:30 smithi100 ceph-mon[39462]: 4.19 scrub starts 2024-09-17T07:03:31.015 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:30 smithi100 ceph-mon[39462]: 4.19 scrub ok 2024-09-17T07:03:31.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:30 smithi135 ceph-mon[33238]: 6.13 scrub starts 2024-09-17T07:03:31.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:30 smithi135 ceph-mon[33238]: 6.13 scrub ok 2024-09-17T07:03:31.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:30 smithi135 ceph-mon[33238]: 5.3 deep-scrub starts 2024-09-17T07:03:31.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:30 smithi135 ceph-mon[33238]: 5.3 deep-scrub ok 2024-09-17T07:03:31.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:30 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2178923757' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-17T07:03:31.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:30 smithi135 ceph-mon[33238]: 4.19 scrub starts 2024-09-17T07:03:31.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:30 smithi135 ceph-mon[33238]: 4.19 scrub ok 2024-09-17T07:03:31.217 INFO:teuthology.orchestra.run.smithi100.stdout:115964117029 2024-09-17T07:03:31.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:31 smithi100 ceph-mon[29092]: 6.1 scrub starts 2024-09-17T07:03:31.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:31 smithi100 ceph-mon[29092]: 5.4 scrub starts 2024-09-17T07:03:31.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:31 smithi100 ceph-mon[29092]: 6.1 scrub ok 2024-09-17T07:03:31.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:31 smithi100 ceph-mon[29092]: 5.4 scrub ok 2024-09-17T07:03:31.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:31 smithi100 ceph-mon[29092]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:03:31.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:31 smithi100 ceph-mon[29092]: 6.17 scrub starts 2024-09-17T07:03:31.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:31 smithi100 ceph-mon[29092]: 6.17 scrub ok 2024-09-17T07:03:31.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:31 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1175996274' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-17T07:03:31.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:31 smithi100 ceph-mon[39462]: 6.1 scrub starts 2024-09-17T07:03:31.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:31 smithi100 ceph-mon[39462]: 5.4 scrub starts 2024-09-17T07:03:31.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:31 smithi100 ceph-mon[39462]: 6.1 scrub ok 2024-09-17T07:03:31.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:31 smithi100 ceph-mon[39462]: 5.4 scrub ok 2024-09-17T07:03:31.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:31 smithi100 ceph-mon[39462]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:03:31.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:31 smithi100 ceph-mon[39462]: 6.17 scrub starts 2024-09-17T07:03:31.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:31 smithi100 ceph-mon[39462]: 6.17 scrub ok 2024-09-17T07:03:31.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:31 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1175996274' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-17T07:03:32.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:31 smithi135 ceph-mon[33238]: 6.1 scrub starts 2024-09-17T07:03:32.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:31 smithi135 ceph-mon[33238]: 5.4 scrub starts 2024-09-17T07:03:32.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:31 smithi135 ceph-mon[33238]: 6.1 scrub ok 2024-09-17T07:03:32.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:31 smithi135 ceph-mon[33238]: 5.4 scrub ok 2024-09-17T07:03:32.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:31 smithi135 ceph-mon[33238]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:03:32.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:31 smithi135 ceph-mon[33238]: 6.17 scrub starts 2024-09-17T07:03:32.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:31 smithi135 ceph-mon[33238]: 6.17 scrub ok 2024-09-17T07:03:32.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:31 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1175996274' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-17T07:03:32.626 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:Sep 17 07:03:32 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-alertmanager-a[71357]: ts=2024-09-17T07:03:32.491Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.003198031s 2024-09-17T07:03:32.779 INFO:tasks.cephadm.ceph_manager.ceph:need seq 115964117026 got 115964117029 for osd.3 2024-09-17T07:03:32.779 DEBUG:teuthology.parallel:result is None 2024-09-17T07:03:33.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:32 smithi135 ceph-mon[33238]: 2.8 scrub starts 2024-09-17T07:03:33.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:32 smithi135 ceph-mon[33238]: 2.8 scrub ok 2024-09-17T07:03:33.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:32 smithi135 ceph-mon[33238]: 2.10 deep-scrub starts 2024-09-17T07:03:33.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:32 smithi135 ceph-mon[33238]: 2.10 deep-scrub ok 2024-09-17T07:03:33.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:32 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:03:33.044 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:32 smithi135 systemd[1]: Started Ceph grafana.a for e848267c-74c1-11ef-bceb-c7b262605968. 2024-09-17T07:03:33.070 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:32 smithi100 ceph-mon[29092]: 2.8 scrub starts 2024-09-17T07:03:33.070 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:32 smithi100 ceph-mon[29092]: 2.8 scrub ok 2024-09-17T07:03:33.070 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:32 smithi100 ceph-mon[29092]: 2.10 deep-scrub starts 2024-09-17T07:03:33.070 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:32 smithi100 ceph-mon[29092]: 2.10 deep-scrub ok 2024-09-17T07:03:33.070 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:32 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:03:33.070 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:32 smithi100 ceph-mon[39462]: 2.8 scrub starts 2024-09-17T07:03:33.071 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:32 smithi100 ceph-mon[39462]: 2.8 scrub ok 2024-09-17T07:03:33.071 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:32 smithi100 ceph-mon[39462]: 2.10 deep-scrub starts 2024-09-17T07:03:33.071 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:32 smithi100 ceph-mon[39462]: 2.10 deep-scrub ok 2024-09-17T07:03:33.071 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:32 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:03:33.295 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=settings t=2024-09-17T07:03:33.044660401Z level=info msg="Starting Grafana" version=9.4.7 commit=4add91f03d branch=HEAD compiled=2023-03-16T23:56:52Z 2024-09-17T07:03:33.295 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=settings t=2024-09-17T07:03:33.044839024Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-09-17T07:03:33.295 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=settings t=2024-09-17T07:03:33.044857777Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-09-17T07:03:33.295 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=settings t=2024-09-17T07:03:33.044864616Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-09-17T07:03:33.296 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=settings t=2024-09-17T07:03:33.044870785Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-09-17T07:03:33.296 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=settings t=2024-09-17T07:03:33.044876666Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-09-17T07:03:33.296 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=settings t=2024-09-17T07:03:33.044883159Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-09-17T07:03:33.296 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=settings t=2024-09-17T07:03:33.044888994Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-09-17T07:03:33.296 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=settings t=2024-09-17T07:03:33.044894963Z level=info msg="Path Home" path=/usr/share/grafana 2024-09-17T07:03:33.296 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=settings t=2024-09-17T07:03:33.04490061Z level=info msg="Path Data" path=/var/lib/grafana 2024-09-17T07:03:33.296 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=settings t=2024-09-17T07:03:33.044906243Z level=info msg="Path Logs" path=/var/log/grafana 2024-09-17T07:03:33.296 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=settings t=2024-09-17T07:03:33.044911685Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-09-17T07:03:33.296 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=settings t=2024-09-17T07:03:33.044917994Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-09-17T07:03:33.296 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=settings t=2024-09-17T07:03:33.044923729Z level=info msg="App mode production" 2024-09-17T07:03:33.296 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=sqlstore t=2024-09-17T07:03:33.044963646Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-09-17T07:03:33.297 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=sqlstore t=2024-09-17T07:03:33.044979652Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-09-17T07:03:33.297 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.045802582Z level=info msg="Starting DB migrations" 2024-09-17T07:03:33.297 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.045878175Z level=info msg="Executing migration" id="create migration_log table" 2024-09-17T07:03:33.297 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.046370136Z level=info msg="Executing migration" id="create user table" 2024-09-17T07:03:33.297 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.046821151Z level=info msg="Executing migration" id="add unique index user.login" 2024-09-17T07:03:33.297 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.047585697Z level=info msg="Executing migration" id="add unique index user.email" 2024-09-17T07:03:33.297 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.048428794Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-09-17T07:03:33.297 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.04926463Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-09-17T07:03:33.297 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.050142984Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-09-17T07:03:33.297 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.05101221Z level=info msg="Executing migration" id="create user table v2" 2024-09-17T07:03:33.297 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.051757194Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-09-17T07:03:33.297 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.052607818Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-09-17T07:03:33.297 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.053507455Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-09-17T07:03:33.298 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.054344465Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-09-17T07:03:33.298 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.055256798Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-09-17T07:03:33.298 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.055970419Z level=info msg="Executing migration" id="Update user table charset" 2024-09-17T07:03:33.298 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.05678248Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-09-17T07:03:33.298 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.057223037Z level=info msg="Executing migration" id="Add missing user data" 2024-09-17T07:03:33.298 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.05768746Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-09-17T07:03:33.298 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.058557852Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-09-17T07:03:33.298 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.059527941Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-09-17T07:03:33.298 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.060156187Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-09-17T07:03:33.298 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.061271754Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-09-17T07:03:33.300 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.062175458Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-09-17T07:03:33.300 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.063153131Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-09-17T07:03:33.300 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.063843199Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-09-17T07:03:33.300 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.064869657Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-09-17T07:03:33.300 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.065498016Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-09-17T07:03:33.300 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.066358054Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-09-17T07:03:33.300 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.06721607Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-09-17T07:03:33.300 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.068211505Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-09-17T07:03:33.300 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.070911432Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-09-17T07:03:33.301 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.073616277Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-09-17T07:03:33.301 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.076542545Z level=info msg="Executing migration" id="create temp_user v2" 2024-09-17T07:03:33.301 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.079651648Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-09-17T07:03:33.301 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.082237485Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-09-17T07:03:33.301 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.08468778Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-09-17T07:03:33.301 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.087592581Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-09-17T07:03:33.301 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.090680931Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-09-17T07:03:33.301 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.093434538Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-09-17T07:03:33.301 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.096068751Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-09-17T07:03:33.301 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.098760682Z level=info msg="Executing migration" id="create star table" 2024-09-17T07:03:33.301 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.099259303Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-09-17T07:03:33.301 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.102227712Z level=info msg="Executing migration" id="create org table v1" 2024-09-17T07:03:33.301 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.104909277Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-09-17T07:03:33.302 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.107441139Z level=info msg="Executing migration" id="create org_user table v1" 2024-09-17T07:03:33.302 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.110251987Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-09-17T07:03:33.302 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.113139111Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-09-17T07:03:33.302 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.115631585Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-09-17T07:03:33.302 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.116016431Z level=info msg="Executing migration" id="Update org table charset" 2024-09-17T07:03:33.302 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.118651166Z level=info msg="Executing migration" id="Update org_user table charset" 2024-09-17T07:03:33.302 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.118960241Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-09-17T07:03:33.302 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.121149007Z level=info msg="Executing migration" id="create dashboard table" 2024-09-17T07:03:33.302 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.121545568Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-09-17T07:03:33.302 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.123744577Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-09-17T07:03:33.302 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.124168258Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-09-17T07:03:33.302 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.124553618Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-09-17T07:03:33.302 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.12736768Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-09-17T07:03:33.303 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.130192885Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-09-17T07:03:33.303 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.131207411Z level=info msg="Executing migration" id="create dashboard v2" 2024-09-17T07:03:33.303 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.134254129Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-09-17T07:03:33.303 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.137129259Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-09-17T07:03:33.303 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.139708509Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-09-17T07:03:33.303 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.140164383Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-09-17T07:03:33.303 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.14056497Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-09-17T07:03:33.303 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.143372734Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-09-17T07:03:33.303 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.143984143Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-09-17T07:03:33.303 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.144471781Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-09-17T07:03:33.303 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.14500357Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-09-17T07:03:33.303 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.147485241Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-09-17T07:03:33.304 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.147991499Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-09-17T07:03:33.304 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.150258867Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-09-17T07:03:33.304 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.1507569Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-09-17T07:03:33.304 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.151070735Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-09-17T07:03:33.304 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.151344047Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-09-17T07:03:33.304 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.153039105Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-09-17T07:03:33.304 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.155497286Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-09-17T07:03:33.304 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.158171082Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-09-17T07:03:33.304 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.160744418Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-09-17T07:03:33.304 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.161172423Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-09-17T07:03:33.304 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.163707977Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-09-17T07:03:33.304 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.16661647Z level=info msg="Executing migration" id="Update dashboard title length" 2024-09-17T07:03:33.304 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.169105645Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-09-17T07:03:33.305 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.169574892Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-09-17T07:03:33.305 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.172219557Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-09-17T07:03:33.305 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.174764704Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-09-17T07:03:33.305 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.177527601Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-09-17T07:03:33.305 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.178056407Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-09-17T07:03:33.305 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.180824933Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-09-17T07:03:33.305 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.183302735Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-09-17T07:03:33.305 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.184147186Z level=info msg="Executing migration" id="Add check_sum column" 2024-09-17T07:03:33.305 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.184696621Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-09-17T07:03:33.305 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.185499831Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-09-17T07:03:33.305 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.186456416Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-09-17T07:03:33.305 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.187188546Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-09-17T07:03:33.305 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.1876196Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-09-17T07:03:33.305 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.188478723Z level=info msg="Executing migration" id="create data_source table" 2024-09-17T07:03:33.306 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.189248412Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-09-17T07:03:33.306 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.18970687Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-09-17T07:03:33.306 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.190153953Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-09-17T07:03:33.306 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.190926457Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-09-17T07:03:33.306 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.191740373Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-09-17T07:03:33.306 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.192775768Z level=info msg="Executing migration" id="create data_source table v2" 2024-09-17T07:03:33.306 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.193204841Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-09-17T07:03:33.306 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.194053526Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-09-17T07:03:33.306 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.195828298Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-09-17T07:03:33.306 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.196203744Z level=info msg="Executing migration" id="Add column with_credentials" 2024-09-17T07:03:33.306 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.196752272Z level=info msg="Executing migration" id="Add secure json data column" 2024-09-17T07:03:33.306 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.197637558Z level=info msg="Executing migration" id="Update data_source table charset" 2024-09-17T07:03:33.307 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.198593976Z level=info msg="Executing migration" id="Update initial version to 1" 2024-09-17T07:03:33.307 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.199270146Z level=info msg="Executing migration" id="Add read_only data column" 2024-09-17T07:03:33.307 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.199866814Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-09-17T07:03:33.307 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.200664693Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-09-17T07:03:33.307 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.201629027Z level=info msg="Executing migration" id="Add uid column" 2024-09-17T07:03:33.307 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.202413855Z level=info msg="Executing migration" id="Update uid value" 2024-09-17T07:03:33.307 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.202764168Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-09-17T07:03:33.307 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.203567719Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-09-17T07:03:33.307 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.204077836Z level=info msg="Executing migration" id="create api_key table" 2024-09-17T07:03:33.307 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.204513978Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-09-17T07:03:33.307 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.205487438Z level=info msg="Executing migration" id="add index api_key.key" 2024-09-17T07:03:33.307 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.206094781Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-09-17T07:03:33.308 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.207068281Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-09-17T07:03:33.308 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.207793076Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-09-17T07:03:33.308 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.208764917Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-09-17T07:03:33.308 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.209448226Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-09-17T07:03:33.308 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.210486306Z level=info msg="Executing migration" id="create api_key table v2" 2024-09-17T07:03:33.308 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.21323408Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-09-17T07:03:33.308 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.215894911Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-09-17T07:03:33.308 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.218736376Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-09-17T07:03:33.308 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.22145589Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-09-17T07:03:33.308 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.224157262Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-09-17T07:03:33.308 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.224558756Z level=info msg="Executing migration" id="Update api_key table charset" 2024-09-17T07:03:33.308 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.227293297Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-09-17T07:03:33.308 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.230197709Z level=info msg="Executing migration" id="Add service account foreign key" 2024-09-17T07:03:33.309 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.232755012Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-09-17T07:03:33.309 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.23533477Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-09-17T07:03:33.309 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.238269333Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-09-17T07:03:33.309 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.240959612Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-09-17T07:03:33.309 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.243617224Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-09-17T07:03:33.309 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.246337274Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-09-17T07:03:33.309 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.249122918Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-09-17T07:03:33.309 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.251688002Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-09-17T07:03:33.309 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.25474295Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-09-17T07:03:33.309 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.257551916Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-09-17T07:03:33.309 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.260151704Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-09-17T07:03:33.309 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.262495797Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-09-17T07:03:33.309 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.265173711Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-09-17T07:03:33.310 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.267740948Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-09-17T07:03:33.310 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.270269877Z level=info msg="Executing migration" id="create quota table v1" 2024-09-17T07:03:33.310 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.273164819Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-09-17T07:03:33.310 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.275601309Z level=info msg="Executing migration" id="Update quota table charset" 2024-09-17T07:03:33.310 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.275926564Z level=info msg="Executing migration" id="create plugin_setting table" 2024-09-17T07:03:33.310 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.278258823Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-09-17T07:03:33.310 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.280908559Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-09-17T07:03:33.310 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.283601453Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-09-17T07:03:33.310 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.283893119Z level=info msg="Executing migration" id="create session table" 2024-09-17T07:03:33.310 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.285977165Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-09-17T07:03:33.310 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.288524147Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-09-17T07:03:33.311 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.288976991Z level=info msg="Executing migration" id="create playlist table v2" 2024-09-17T07:03:33.311 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.289347416Z level=info msg="Executing migration" id="create playlist item table v2" 2024-09-17T07:03:33.311 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.291887611Z level=info msg="Executing migration" id="Update playlist table charset" 2024-09-17T07:03:33.311 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.294417078Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-09-17T07:03:33.311 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.294768945Z level=info msg="Executing migration" id="drop preferences table v2" 2024-09-17T07:03:33.311 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.295045875Z level=info msg="Executing migration" id="drop preferences table v3" 2024-09-17T07:03:33.311 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.295311519Z level=info msg="Executing migration" id="create preferences table v3" 2024-09-17T07:03:33.311 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.295715963Z level=info msg="Executing migration" id="Update preferences table charset" 2024-09-17T07:03:33.311 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.298298522Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-09-17T07:03:33.311 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.298933007Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-09-17T07:03:33.490 INFO:teuthology.orchestra.run.smithi100.stdout:55834574896 2024-09-17T07:03:33.546 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.301601783Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-09-17T07:03:33.546 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.302214774Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-09-17T07:03:33.546 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.304828744Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-09-17T07:03:33.546 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.307472786Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-09-17T07:03:33.546 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.307951118Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-09-17T07:03:33.546 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.308827283Z level=info msg="Executing migration" id="create alert table v1" 2024-09-17T07:03:33.546 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.309567627Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-09-17T07:03:33.546 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.310414062Z level=info msg="Executing migration" id="add index alert state" 2024-09-17T07:03:33.546 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.31120483Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-09-17T07:03:33.547 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.311676558Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-09-17T07:03:33.547 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.312084462Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-09-17T07:03:33.547 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.312535059Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-09-17T07:03:33.547 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.312979874Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-09-17T07:03:33.547 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.314357345Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-09-17T07:03:33.547 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.314765648Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-09-17T07:03:33.547 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.31517074Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-09-17T07:03:33.547 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.315456141Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-09-17T07:03:33.547 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.315804175Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-09-17T07:03:33.547 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.316152601Z level=info msg="Executing migration" id="Add column is_default" 2024-09-17T07:03:33.547 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.316768472Z level=info msg="Executing migration" id="Add column frequency" 2024-09-17T07:03:33.547 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.317371224Z level=info msg="Executing migration" id="Add column send_reminder" 2024-09-17T07:03:33.548 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.318022432Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-09-17T07:03:33.548 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.318638964Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-09-17T07:03:33.548 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.319045613Z level=info msg="Executing migration" id="Update alert table charset" 2024-09-17T07:03:33.548 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.32177313Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-09-17T07:03:33.548 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.322080383Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-09-17T07:03:33.548 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.324516197Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-09-17T07:03:33.548 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.327147662Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-09-17T07:03:33.548 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.329753145Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-09-17T07:03:33.548 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.332592067Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-09-17T07:03:33.548 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.33496864Z level=info msg="Executing migration" id="Add for to alert table" 2024-09-17T07:03:33.548 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.337548544Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-09-17T07:03:33.548 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.340269712Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-09-17T07:03:33.548 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.343044524Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-09-17T07:03:33.549 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.345502726Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-09-17T07:03:33.549 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.348257317Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-09-17T07:03:33.549 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.350831504Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-09-17T07:03:33.549 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.353503521Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-09-17T07:03:33.550 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.354170001Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-09-17T07:03:33.550 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.354978912Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-09-17T07:03:33.550 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.355807642Z level=info msg="Executing migration" id="create annotation table v5" 2024-09-17T07:03:33.550 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.356745364Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-09-17T07:03:33.550 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.357452751Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-09-17T07:03:33.550 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.358244661Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-09-17T07:03:33.550 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.359105163Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-09-17T07:03:33.551 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.360059877Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-09-17T07:03:33.551 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.360786892Z level=info msg="Executing migration" id="Update annotation table charset" 2024-09-17T07:03:33.551 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.361645466Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-09-17T07:03:33.551 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.362407155Z level=info msg="Executing migration" id="Drop category_id index" 2024-09-17T07:03:33.551 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.363369152Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-09-17T07:03:33.551 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.364219596Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-09-17T07:03:33.551 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.365155827Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-09-17T07:03:33.551 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.365804774Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-09-17T07:03:33.551 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.366678915Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-09-17T07:03:33.551 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.368470627Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-09-17T07:03:33.551 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.369264533Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-09-17T07:03:33.551 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.370117067Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-09-17T07:03:33.551 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.371092085Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-09-17T07:03:33.552 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.371883825Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-09-17T07:03:33.552 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.372758793Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-09-17T07:03:33.552 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.373589109Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-09-17T07:03:33.552 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.374473663Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-09-17T07:03:33.552 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.375339206Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-09-17T07:03:33.552 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.376224991Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-09-17T07:03:33.552 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.377222998Z level=info msg="Executing migration" id="Add epoch_end column" 2024-09-17T07:03:33.552 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.378072764Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-09-17T07:03:33.552 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.379061236Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-09-17T07:03:33.552 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.379831896Z level=info msg="Executing migration" id="Move region to single row" 2024-09-17T07:03:33.552 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.380739017Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-09-17T07:03:33.552 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.381544287Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-09-17T07:03:33.553 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.382334589Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-09-17T07:03:33.553 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.383174124Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-09-17T07:03:33.553 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.384095909Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-09-17T07:03:33.553 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.386784432Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-09-17T07:03:33.553 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.389569897Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-09-17T07:03:33.553 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.392500353Z level=info msg="Executing migration" id="create test_data table" 2024-09-17T07:03:33.553 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.395417585Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-09-17T07:03:33.553 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.39820105Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-09-17T07:03:33.553 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.400660784Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-09-17T07:03:33.553 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.403675623Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-09-17T07:03:33.553 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.406507805Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-09-17T07:03:33.553 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.409571704Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-09-17T07:03:33.554 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.412062448Z level=info msg="Executing migration" id="create team table" 2024-09-17T07:03:33.554 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.414746529Z level=info msg="Executing migration" id="add index team.org_id" 2024-09-17T07:03:33.554 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.417581537Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-09-17T07:03:33.554 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.420645824Z level=info msg="Executing migration" id="create team member table" 2024-09-17T07:03:33.554 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.42339872Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-09-17T07:03:33.554 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.425879581Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-09-17T07:03:33.554 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.428664775Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-09-17T07:03:33.554 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.431619596Z level=info msg="Executing migration" id="Add column email to team table" 2024-09-17T07:03:33.554 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.43447788Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-09-17T07:03:33.554 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.437033694Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-09-17T07:03:33.554 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.439527291Z level=info msg="Executing migration" id="create dashboard acl table" 2024-09-17T07:03:33.554 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.442416875Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-09-17T07:03:33.554 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.445081766Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-09-17T07:03:33.554 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.447637884Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-09-17T07:03:33.555 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.450279649Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-09-17T07:03:33.555 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.452828494Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-09-17T07:03:33.555 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.455528151Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-09-17T07:03:33.555 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.45845194Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-09-17T07:03:33.555 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.458910303Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-09-17T07:03:33.555 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.461596278Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-09-17T07:03:33.555 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.462045934Z level=info msg="Executing migration" id="create tag table" 2024-09-17T07:03:33.555 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.464193898Z level=info msg="Executing migration" id="add index tag.key_value" 2024-09-17T07:03:33.555 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.466960763Z level=info msg="Executing migration" id="create login attempt table" 2024-09-17T07:03:33.555 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.46746049Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-09-17T07:03:33.555 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.470273745Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-09-17T07:03:33.555 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.472954577Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-09-17T07:03:33.556 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.474839515Z level=info msg="Executing migration" id="create login_attempt v2" 2024-09-17T07:03:33.556 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.475252466Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-09-17T07:03:33.556 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.476074301Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-09-17T07:03:33.556 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.477013967Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-09-17T07:03:33.556 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.477739776Z level=info msg="Executing migration" id="create user auth table" 2024-09-17T07:03:33.556 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.478594912Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-09-17T07:03:33.556 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.479370035Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-09-17T07:03:33.556 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.48018762Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-09-17T07:03:33.556 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.481034005Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-09-17T07:03:33.556 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.481804604Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-09-17T07:03:33.556 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.482590134Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-09-17T07:03:33.556 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.48350117Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-09-17T07:03:33.556 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.48432306Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-09-17T07:03:33.557 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.485090474Z level=info msg="Executing migration" id="create server_lock table" 2024-09-17T07:03:33.557 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.485900643Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-09-17T07:03:33.557 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.486718393Z level=info msg="Executing migration" id="create user auth token table" 2024-09-17T07:03:33.557 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.487585506Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-09-17T07:03:33.557 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.488002045Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-09-17T07:03:33.557 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.48878261Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-09-17T07:03:33.557 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.489629004Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-09-17T07:03:33.557 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.490505298Z level=info msg="Executing migration" id="create cache_data table" 2024-09-17T07:03:33.557 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.490907055Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-09-17T07:03:33.557 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.49171755Z level=info msg="Executing migration" id="create short_url table v1" 2024-09-17T07:03:33.557 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.492595909Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-09-17T07:03:33.557 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.493604449Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-09-17T07:03:33.558 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.493950916Z level=info msg="Executing migration" id="delete alert_definition table" 2024-09-17T07:03:33.558 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.494727553Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-09-17T07:03:33.558 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.495198184Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-09-17T07:03:33.558 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.495666183Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-09-17T07:03:33.558 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.496631697Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-09-17T07:03:33.558 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.4971236Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-09-17T07:03:33.558 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.497910643Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-09-17T07:03:33.558 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.498777512Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-09-17T07:03:33.558 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.499714364Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-09-17T07:03:33.558 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.500223446Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-09-17T07:03:33.558 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.502903051Z level=info msg="Executing migration" id="drop alert_definition table" 2024-09-17T07:03:33.558 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.505755606Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-09-17T07:03:33.558 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.508614583Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-09-17T07:03:33.558 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.509063468Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-09-17T07:03:33.559 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.5119795Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-09-17T07:03:33.559 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.514381161Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-09-17T07:03:33.559 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.517263803Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-09-17T07:03:33.559 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.517740137Z level=info msg="Executing migration" id="create alert_instance table" 2024-09-17T07:03:33.559 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.520617092Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-09-17T07:03:33.559 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.523206136Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-09-17T07:03:33.559 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.526246702Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-09-17T07:03:33.559 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.527164248Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-09-17T07:03:33.559 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.527632702Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-09-17T07:03:33.559 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.528058239Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-09-17T07:03:33.559 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.531452847Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-09-17T07:03:33.559 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.53506434Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-09-17T07:03:33.559 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.53549536Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-09-17T07:03:33.560 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.535904056Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-09-17T07:03:33.560 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.536711592Z level=info msg="Executing migration" id="create alert_rule table" 2024-09-17T07:03:33.560 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.537114534Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-09-17T07:03:33.560 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.537530278Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-09-17T07:03:33.560 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.53794474Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-09-17T07:03:33.560 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.538347879Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-09-17T07:03:33.560 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.538639253Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-09-17T07:03:33.560 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.539428351Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-09-17T07:03:33.560 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.540206765Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-09-17T07:03:33.560 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.54102511Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-09-17T07:03:33.560 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.5414352Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-09-17T07:03:33.560 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.541848074Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-09-17T07:03:33.560 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.542643651Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-09-17T07:03:33.561 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.543429834Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-09-17T07:03:33.561 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.543854025Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-09-17T07:03:33.561 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.544667048Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-09-17T07:03:33.561 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.545466881Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-09-17T07:03:33.561 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.545798948Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-09-17T07:03:33.561 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.546262918Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-09-17T07:03:33.561 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.546823163Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-09-17T07:03:33.561 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.547295025Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-09-17T07:03:33.561 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.547632517Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-09-17T07:03:33.561 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.548501055Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-09-17T07:03:33.561 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.549386568Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-09-17T07:03:33.803 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-mon[33238]: 4.7 scrub starts 2024-09-17T07:03:33.803 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-mon[33238]: 4.7 scrub ok 2024-09-17T07:03:33.803 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-mon[33238]: 2.5 scrub starts 2024-09-17T07:03:33.803 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-mon[33238]: 2.5 scrub ok 2024-09-17T07:03:33.803 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-mon[33238]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:03:33.803 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:33.803 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:33.803 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:33.803 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:33.804 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-mon[33238]: Deploying daemon node-exporter.a on smithi100 2024-09-17T07:03:33.804 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-mon[33238]: 5.0 scrub starts 2024-09-17T07:03:33.804 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-mon[33238]: 5.0 scrub ok 2024-09-17T07:03:33.804 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1941356515' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-17T07:03:33.804 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.550243856Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-09-17T07:03:33.805 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.551099502Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-09-17T07:03:33.805 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.551951797Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-09-17T07:03:33.805 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.552276578Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-09-17T07:03:33.805 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.552730581Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-09-17T07:03:33.805 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.553571137Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-09-17T07:03:33.805 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.553891505Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-09-17T07:03:33.805 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.554781843Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-09-17T07:03:33.805 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.555623217Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-09-17T07:03:33.805 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.556502725Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-09-17T07:03:33.805 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.557328714Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-09-17T07:03:33.805 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.55779201Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-09-17T07:03:33.806 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.558661338Z level=info msg="Executing migration" id="create provenance_type table" 2024-09-17T07:03:33.806 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.559549061Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-09-17T07:03:33.806 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.560322305Z level=info msg="Executing migration" id="create alert_image table" 2024-09-17T07:03:33.806 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.560749892Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-09-17T07:03:33.806 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.561644946Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-09-17T07:03:33.806 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.562429809Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-09-17T07:03:33.806 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.562880563Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-09-17T07:03:33.806 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.563287642Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-09-17T07:03:33.806 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.563372891Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-09-17T07:03:33.807 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.563689444Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-09-17T07:03:33.807 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.564182111Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-09-17T07:03:33.807 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.566817297Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-09-17T07:03:33.807 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.566977392Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-09-17T07:03:33.807 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.567121808Z level=info msg="alerts found to migrate" alerts=0 2024-09-17T07:03:33.807 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.569515906Z level=info msg="Executing migration" id="create library_element table v1" 2024-09-17T07:03:33.807 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.569991485Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-09-17T07:03:33.807 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.572985526Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-09-17T07:03:33.807 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.575516299Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-09-17T07:03:33.807 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.578454899Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-09-17T07:03:33.807 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.581066081Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-09-17T07:03:33.807 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.583549148Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-09-17T07:03:33.807 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.586514802Z level=info msg="Executing migration" id="create data_keys table" 2024-09-17T07:03:33.808 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.589474497Z level=info msg="Executing migration" id="create secrets table" 2024-09-17T07:03:33.808 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.590040989Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-09-17T07:03:33.809 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.594253193Z level=info msg="Executing migration" id="add name column into data_keys" 2024-09-17T07:03:33.809 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.597213573Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-09-17T07:03:33.810 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.59978489Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-09-17T07:03:33.810 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.604031797Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-09-17T07:03:33.810 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.60820857Z level=info msg="Executing migration" id="create kv_store table v1" 2024-09-17T07:03:33.810 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.610888677Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-09-17T07:03:33.810 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.613427956Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-09-17T07:03:33.810 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.613955145Z level=info msg="Executing migration" id="create permission table" 2024-09-17T07:03:33.810 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.616566235Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-09-17T07:03:33.810 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.618999215Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-09-17T07:03:33.810 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.621720933Z level=info msg="Executing migration" id="create role table" 2024-09-17T07:03:33.810 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.622168803Z level=info msg="Executing migration" id="add column display_name" 2024-09-17T07:03:33.810 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.624788891Z level=info msg="Executing migration" id="add column group_name" 2024-09-17T07:03:33.810 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.627277896Z level=info msg="Executing migration" id="add index role.org_id" 2024-09-17T07:03:33.810 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.630159163Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-09-17T07:03:33.811 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.630648477Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-09-17T07:03:33.811 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.633387267Z level=info msg="Executing migration" id="create team role table" 2024-09-17T07:03:33.811 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.635749532Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-09-17T07:03:33.811 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.638270845Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-09-17T07:03:33.811 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.638859941Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-09-17T07:03:33.811 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.641684718Z level=info msg="Executing migration" id="create user role table" 2024-09-17T07:03:33.811 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.644294244Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-09-17T07:03:33.811 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.646782845Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-09-17T07:03:33.811 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.647284056Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-09-17T07:03:33.811 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.64808929Z level=info msg="Executing migration" id="create builtin role table" 2024-09-17T07:03:33.811 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.648884597Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-09-17T07:03:33.811 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.649703098Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-09-17T07:03:33.812 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.650160112Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-09-17T07:03:33.812 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.651130937Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-09-17T07:03:33.812 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.651680043Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-09-17T07:03:33.812 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.652151607Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-09-17T07:03:33.812 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.652569814Z level=info msg="Executing migration" id="add unique index role.uid" 2024-09-17T07:03:33.812 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.652993643Z level=info msg="Executing migration" id="create seed assignment table" 2024-09-17T07:03:33.812 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.653783137Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-09-17T07:03:33.812 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.654282149Z level=info msg="Executing migration" id="add column hidden to role table" 2024-09-17T07:03:33.812 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.655248083Z level=info msg="Executing migration" id="create query_history table v1" 2024-09-17T07:03:33.812 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.655650113Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-09-17T07:03:33.812 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.656074646Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-09-17T07:03:33.812 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.656331655Z level=info msg="Executing migration" id="teams permissions migration" 2024-09-17T07:03:33.813 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.656913269Z level=info msg="Executing migration" id="dashboard permissions" 2024-09-17T07:03:33.813 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.657320782Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-09-17T07:03:33.813 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.657684893Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-09-17T07:03:33.813 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.657989285Z level=info msg="Executing migration" id="alerting notification permissions" 2024-09-17T07:03:33.813 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.658303914Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-09-17T07:03:33.813 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.658691989Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-09-17T07:03:33.813 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.659118317Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-09-17T07:03:33.813 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.660105777Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-09-17T07:03:33.813 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.660418619Z level=info msg="Executing migration" id="create correlation table v1" 2024-09-17T07:03:33.813 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.660840157Z level=info msg="Executing migration" id="add index correlations.uid" 2024-09-17T07:03:33.813 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.661250745Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-09-17T07:03:33.814 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.661679698Z level=info msg="Executing migration" id="add correlation config column" 2024-09-17T07:03:33.814 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.662667741Z level=info msg="Executing migration" id="create entity_events table" 2024-09-17T07:03:33.814 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.663077953Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-09-17T07:03:33.814 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.663480238Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-09-17T07:03:33.814 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.663561321Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-09-17T07:03:33.814 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.663864511Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-17T07:03:33.814 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.663972609Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-17T07:03:33.814 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.66427865Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-09-17T07:03:33.814 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.664740205Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-09-17T07:03:33.814 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.665168596Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-09-17T07:03:33.814 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.66565773Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-17T07:03:33.814 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.666119156Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-09-17T07:03:33.815 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.666566141Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-09-17T07:03:33.815 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.667007438Z level=info msg="Executing migration" id="Drop public config table" 2024-09-17T07:03:33.815 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.667424632Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-09-17T07:03:33.815 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.66787264Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-09-17T07:03:33.815 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.668298536Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-09-17T07:03:33.815 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.668787197Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-09-17T07:03:33.815 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.669532728Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-09-17T07:03:33.815 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.672535915Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-09-17T07:03:33.815 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.673555603Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-09-17T07:03:33.815 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.674570194Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-09-17T07:03:33.815 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.674903847Z level=info msg="Executing migration" id="add share column" 2024-09-17T07:03:33.815 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.675965559Z level=info msg="Executing migration" id="create default alerting folders" 2024-09-17T07:03:33.815 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.676419445Z level=info msg="Executing migration" id="create file table" 2024-09-17T07:03:33.816 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.678865242Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-09-17T07:03:33.816 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.681777298Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-09-17T07:03:33.816 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.684249682Z level=info msg="Executing migration" id="create file_meta table" 2024-09-17T07:03:33.816 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.687210899Z level=info msg="Executing migration" id="file table idx: path key" 2024-09-17T07:03:33.816 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.687707118Z level=info msg="Executing migration" id="set path collation in file table" 2024-09-17T07:03:33.816 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.69058778Z level=info msg="Executing migration" id="managed permissions migration" 2024-09-17T07:03:33.816 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.693459147Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-09-17T07:03:33.816 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.695849943Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-09-17T07:03:33.816 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.696294606Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-09-17T07:03:33.816 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.699192311Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-09-17T07:03:33.816 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.702025266Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-09-17T07:03:33.816 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.704658677Z level=info msg="Executing migration" id="update group index for alert rules" 2024-09-17T07:03:33.816 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.705039889Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-09-17T07:03:33.817 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.707611344Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-09-17T07:03:33.817 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.707999279Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-09-17T07:03:33.817 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.710325044Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-09-17T07:03:33.817 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.712870275Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-09-17T07:03:33.817 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.713406347Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-09-17T07:03:33.817 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.723012557Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-09-17T07:03:33.817 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.725549177Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-09-17T07:03:33.817 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.728000172Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-09-17T07:03:33.817 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.73147579Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-09-17T07:03:33.817 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.734006664Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-09-17T07:03:33.817 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.736764516Z level=info msg="Executing migration" id="create folder table" 2024-09-17T07:03:33.817 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.739404744Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-09-17T07:03:33.818 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.739967487Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-09-17T07:03:33.818 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.742686199Z level=info msg="Executing migration" id="Update folder title length" 2024-09-17T07:03:33.818 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.745647372Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-09-17T07:03:33.818 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=migrator t=2024-09-17T07:03:33.746328989Z level=info msg="migrations completed" performed=485 skipped=0 duration=700.474729ms 2024-09-17T07:03:33.818 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=sqlstore t=2024-09-17T07:03:33.746704901Z level=info msg="Created default organization" 2024-09-17T07:03:33.818 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=plugin.loader t=2024-09-17T07:03:33.803117726Z level=info msg="Plugin registered" pluginID=input 2024-09-17T07:03:33.955 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:33 smithi100 ceph-mon[29092]: 4.7 scrub starts 2024-09-17T07:03:33.955 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:33 smithi100 ceph-mon[29092]: 4.7 scrub ok 2024-09-17T07:03:33.955 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:33 smithi100 ceph-mon[29092]: 2.5 scrub starts 2024-09-17T07:03:33.955 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:33 smithi100 ceph-mon[29092]: 2.5 scrub ok 2024-09-17T07:03:33.955 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:33 smithi100 ceph-mon[29092]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:03:33.955 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:33 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:33.955 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:33 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:33.955 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:33 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:33.955 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:33 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:33.955 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:33 smithi100 ceph-mon[29092]: Deploying daemon node-exporter.a on smithi100 2024-09-17T07:03:33.956 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:33 smithi100 ceph-mon[29092]: 5.0 scrub starts 2024-09-17T07:03:33.956 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:33 smithi100 ceph-mon[29092]: 5.0 scrub ok 2024-09-17T07:03:33.956 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:33 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1941356515' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-17T07:03:33.956 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:33 smithi100 ceph-mon[39462]: 4.7 scrub starts 2024-09-17T07:03:33.956 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:33 smithi100 ceph-mon[39462]: 4.7 scrub ok 2024-09-17T07:03:33.956 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:33 smithi100 ceph-mon[39462]: 2.5 scrub starts 2024-09-17T07:03:33.956 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:33 smithi100 ceph-mon[39462]: 2.5 scrub ok 2024-09-17T07:03:33.956 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:33 smithi100 ceph-mon[39462]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:03:33.956 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:33 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:33.956 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:33 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:33.957 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:33 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:33.957 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:33 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:33.957 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:33 smithi100 ceph-mon[39462]: Deploying daemon node-exporter.a on smithi100 2024-09-17T07:03:33.957 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:33 smithi100 ceph-mon[39462]: 5.0 scrub starts 2024-09-17T07:03:33.957 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:33 smithi100 ceph-mon[39462]: 5.0 scrub ok 2024-09-17T07:03:33.957 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:33 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1941356515' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-17T07:03:33.957 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:Sep 17 07:03:33 smithi100 systemd[1]: Starting Ceph node-exporter.a for e848267c-74c1-11ef-bceb-c7b262605968... 2024-09-17T07:03:34.281 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574891 got 55834574896 for osd.1 2024-09-17T07:03:34.281 DEBUG:teuthology.parallel:result is None 2024-09-17T07:03:34.282 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-17T07:03:34.282 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T07:03:34.293 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=plugin.loader t=2024-09-17T07:03:33.830850685Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-09-17T07:03:34.293 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=plugin.loader t=2024-09-17T07:03:33.830892058Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-09-17T07:03:34.293 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=secrets t=2024-09-17T07:03:33.830927962Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-09-17T07:03:34.293 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=query_data t=2024-09-17T07:03:33.832561324Z level=info msg="Query Service initialization" 2024-09-17T07:03:34.293 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=live.push_http t=2024-09-17T07:03:33.835069027Z level=info msg="Live Push Gateway initialization" 2024-09-17T07:03:34.493 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:03:34.614 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:Sep 17 07:03:34 smithi100 bash[72166]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-09-17T07:03:34.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:34 smithi100 ceph-mon[29092]: 5.12 scrub starts 2024-09-17T07:03:34.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:34 smithi100 ceph-mon[29092]: 5.12 scrub ok 2024-09-17T07:03:34.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:34 smithi100 ceph-mon[29092]: 2.17 scrub starts 2024-09-17T07:03:34.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:34 smithi100 ceph-mon[29092]: 2.17 scrub ok 2024-09-17T07:03:34.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:34 smithi100 ceph-mon[29092]: 2.1 scrub starts 2024-09-17T07:03:34.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:34 smithi100 ceph-mon[29092]: 2.1 scrub ok 2024-09-17T07:03:34.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:34 smithi100 ceph-mon[39462]: 5.12 scrub starts 2024-09-17T07:03:34.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:34 smithi100 ceph-mon[39462]: 5.12 scrub ok 2024-09-17T07:03:34.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:34 smithi100 ceph-mon[39462]: 2.17 scrub starts 2024-09-17T07:03:34.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:34 smithi100 ceph-mon[39462]: 2.17 scrub ok 2024-09-17T07:03:34.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:34 smithi100 ceph-mon[39462]: 2.1 scrub starts 2024-09-17T07:03:34.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:34 smithi100 ceph-mon[39462]: 2.1 scrub ok 2024-09-17T07:03:35.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:34 smithi135 ceph-mon[33238]: 5.12 scrub starts 2024-09-17T07:03:35.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:34 smithi135 ceph-mon[33238]: 5.12 scrub ok 2024-09-17T07:03:35.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:34 smithi135 ceph-mon[33238]: 2.17 scrub starts 2024-09-17T07:03:35.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:34 smithi135 ceph-mon[33238]: 2.17 scrub ok 2024-09-17T07:03:35.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:34 smithi135 ceph-mon[33238]: 2.1 scrub starts 2024-09-17T07:03:35.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:34 smithi135 ceph-mon[33238]: 2.1 scrub ok 2024-09-17T07:03:35.044 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:34 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=infra.usagestats.collector t=2024-09-17T07:03:34.601055243Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-09-17T07:03:35.044 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:34 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=provisioning.datasources t=2024-09-17T07:03:34.655050032Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-09-17T07:03:35.044 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:34 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=provisioning.datasources t=2024-09-17T07:03:34.686933129Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-09-17T07:03:35.044 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:34 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=provisioning.alerting t=2024-09-17T07:03:34.708905247Z level=info msg="starting to provision alerting" 2024-09-17T07:03:35.044 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:34 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=provisioning.alerting t=2024-09-17T07:03:34.708937741Z level=info msg="finished to provision alerting" 2024-09-17T07:03:35.045 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:34 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=ngalert.state.manager t=2024-09-17T07:03:34.709040304Z level=info msg="Warming state cache for startup" 2024-09-17T07:03:35.045 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:34 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=grafanaStorageLogger t=2024-09-17T07:03:34.709154924Z level=info msg="storage starting" 2024-09-17T07:03:35.045 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:34 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=http.server t=2024-09-17T07:03:34.710926335Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-09-17T07:03:35.045 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:34 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=sqlstore.transactions t=2024-09-17T07:03:34.719994007Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-17T07:03:35.045 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:34 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=ngalert.state.manager t=2024-09-17T07:03:34.720151511Z level=info msg="State cache has been initialized" states=0 duration=11.109767ms 2024-09-17T07:03:35.045 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:34 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=ticker t=2024-09-17T07:03:34.720224475Z level=info msg=starting first_tick=2024-09-17T07:03:40Z 2024-09-17T07:03:35.045 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:03:34 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=ngalert.multiorg.alertmanager t=2024-09-17T07:03:34.72021888Z level=info msg="starting MultiOrg Alertmanager" 2024-09-17T07:03:35.377 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:Sep 17 07:03:34 smithi100 bash[72166]: Getting image source signatures 2024-09-17T07:03:35.377 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:Sep 17 07:03:34 smithi100 bash[72166]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-09-17T07:03:35.377 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:Sep 17 07:03:34 smithi100 bash[72166]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-09-17T07:03:35.377 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:Sep 17 07:03:34 smithi100 bash[72166]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-09-17T07:03:36.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:36 smithi135 ceph-mon[33238]: 6.19 scrub starts 2024-09-17T07:03:36.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:36 smithi135 ceph-mon[33238]: 6.19 scrub ok 2024-09-17T07:03:36.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:36 smithi135 ceph-mon[33238]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T07:03:36.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:36 smithi135 ceph-mon[33238]: 6.c scrub starts 2024-09-17T07:03:36.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:36 smithi135 ceph-mon[33238]: 6.c scrub ok 2024-09-17T07:03:36.322 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:36 smithi100 ceph-mon[29092]: 6.19 scrub starts 2024-09-17T07:03:36.323 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:36 smithi100 ceph-mon[29092]: 6.19 scrub ok 2024-09-17T07:03:36.323 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:36 smithi100 ceph-mon[29092]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T07:03:36.323 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:36 smithi100 ceph-mon[29092]: 6.c scrub starts 2024-09-17T07:03:36.323 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:36 smithi100 ceph-mon[29092]: 6.c scrub ok 2024-09-17T07:03:36.323 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:03:35 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:03:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:03:36.323 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:36 smithi100 ceph-mon[39462]: 6.19 scrub starts 2024-09-17T07:03:36.324 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:36 smithi100 ceph-mon[39462]: 6.19 scrub ok 2024-09-17T07:03:36.324 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:36 smithi100 ceph-mon[39462]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T07:03:36.324 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:36 smithi100 ceph-mon[39462]: 6.c scrub starts 2024-09-17T07:03:36.324 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:36 smithi100 ceph-mon[39462]: 6.c scrub ok 2024-09-17T07:03:37.004 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:Sep 17 07:03:36 smithi100 bash[72166]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-09-17T07:03:37.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:37 smithi135 ceph-mon[33238]: 4.17 scrub starts 2024-09-17T07:03:37.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:37 smithi135 ceph-mon[33238]: 4.17 scrub ok 2024-09-17T07:03:37.316 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:37 smithi100 ceph-mon[29092]: 4.17 scrub starts 2024-09-17T07:03:37.317 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:37 smithi100 ceph-mon[29092]: 4.17 scrub ok 2024-09-17T07:03:37.317 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:Sep 17 07:03:37 smithi100 bash[72166]: Writing manifest to image destination 2024-09-17T07:03:37.317 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:37 smithi100 ceph-mon[39462]: 4.17 scrub starts 2024-09-17T07:03:37.317 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:37 smithi100 ceph-mon[39462]: 4.17 scrub ok 2024-09-17T07:03:37.430 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T07:03:37.432 INFO:teuthology.orchestra.run.smithi100.stderr:dumped all 2024-09-17T07:03:37.627 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:Sep 17 07:03:37 smithi100 podman[72166]: 2024-09-17 07:03:37.444826876 +0000 UTC m=+3.072946004 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-09-17T07:03:37.889 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:Sep 17 07:03:37 smithi100 podman[72166]: 2024-09-17 07:03:37.77814556 +0000 UTC m=+3.406264685 container create 94e0a7f831e23f91cd36bbad11105e50aecec8487ccb23a9610e7573ca695e67 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-17T07:03:38.376 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:38 smithi100 ceph-mon[39462]: 5.8 scrub starts 2024-09-17T07:03:38.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:38 smithi100 ceph-mon[39462]: 5.8 scrub ok 2024-09-17T07:03:38.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:38 smithi100 ceph-mon[39462]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 854 B/s rd, 0 op/s 2024-09-17T07:03:38.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:38 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:38.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:38 smithi100 ceph-mon[39462]: 5.10 scrub starts 2024-09-17T07:03:38.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:38 smithi100 ceph-mon[39462]: 5.10 scrub ok 2024-09-17T07:03:38.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:38 smithi100 ceph-mon[39462]: 5.19 scrub starts 2024-09-17T07:03:38.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:38 smithi100 ceph-mon[39462]: 5.19 scrub ok 2024-09-17T07:03:38.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:38 smithi100 ceph-mon[29092]: 5.8 scrub starts 2024-09-17T07:03:38.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:38 smithi100 ceph-mon[29092]: 5.8 scrub ok 2024-09-17T07:03:38.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:38 smithi100 ceph-mon[29092]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 854 B/s rd, 0 op/s 2024-09-17T07:03:38.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:38 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:38.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:38 smithi100 ceph-mon[29092]: 5.10 scrub starts 2024-09-17T07:03:38.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:38 smithi100 ceph-mon[29092]: 5.10 scrub ok 2024-09-17T07:03:38.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:38 smithi100 ceph-mon[29092]: 5.19 scrub starts 2024-09-17T07:03:38.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:38 smithi100 ceph-mon[29092]: 5.19 scrub ok 2024-09-17T07:03:38.379 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:Sep 17 07:03:38 smithi100 podman[72166]: 2024-09-17 07:03:38.00513288 +0000 UTC m=+3.633252037 container init 94e0a7f831e23f91cd36bbad11105e50aecec8487ccb23a9610e7573ca695e67 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-17T07:03:38.379 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:Sep 17 07:03:38 smithi100 podman[72166]: 2024-09-17 07:03:38.008113234 +0000 UTC m=+3.636232354 container start 94e0a7f831e23f91cd36bbad11105e50aecec8487ccb23a9610e7573ca695e67 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-17T07:03:38.379 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:Sep 17 07:03:38 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-a[72373]: ts=2024-09-17T07:03:38.012Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-09-17T07:03:38.379 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:Sep 17 07:03:38 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-a[72373]: ts=2024-09-17T07:03:38.012Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-09-17T07:03:38.379 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:Sep 17 07:03:38 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-a[72373]: ts=2024-09-17T07:03:38.012Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-17T07:03:38.379 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:Sep 17 07:03:38 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-a[72373]: ts=2024-09-17T07:03:38.012Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-17T07:03:38.379 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:Sep 17 07:03:38 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-a[72373]: ts=2024-09-17T07:03:38.013Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-17T07:03:38.379 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:Sep 17 07:03:38 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-a[72373]: ts=2024-09-17T07:03:38.013Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-17T07:03:38.379 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:Sep 17 07:03:38 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-a[72373]: ts=2024-09-17T07:03:38.013Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-17T07:03:38.379 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:Sep 17 07:03:38 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-a[72373]: ts=2024-09-17T07:03:38.013Z caller=node_exporter.go:117 level=info collector=arp 2024-09-17T07:03:38.379 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:Sep 17 07:03:38 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-a[72373]: ts=2024-09-17T07:03:38.013Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-17T07:03:38.380 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:Sep 17 07:03:38 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-a[72373]: ts=2024-09-17T07:03:38.013Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-17T07:03:38.380 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:Sep 17 07:03:38 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-a[72373]: ts=2024-09-17T07:03:38.013Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-17T07:03:38.380 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:Sep 17 07:03:38 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-a[72373]: ts=2024-09-17T07:03:38.013Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-17T07:03:38.380 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:Sep 17 07:03:38 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-a[72373]: ts=2024-09-17T07:03:38.013Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-17T07:03:38.380 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:Sep 17 07:03:38 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-a[72373]: ts=2024-09-17T07:03:38.013Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-17T07:03:38.380 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:Sep 17 07:03:38 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-a[72373]: ts=2024-09-17T07:03:38.013Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-17T07:03:38.380 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:Sep 17 07:03:38 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-a[72373]: ts=2024-09-17T07:03:38.013Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-17T07:03:38.380 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:Sep 17 07:03:38 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-a[72373]: ts=2024-09-17T07:03:38.013Z caller=node_exporter.go:117 level=info collector=edac 2024-09-17T07:03:38.380 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:Sep 17 07:03:38 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-a[72373]: ts=2024-09-17T07:03:38.013Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-17T07:03:38.380 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:Sep 17 07:03:38 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-a[72373]: ts=2024-09-17T07:03:38.013Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-17T07:03:38.380 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:Sep 17 07:03:38 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-a[72373]: ts=2024-09-17T07:03:38.013Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-17T07:03:38.380 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:Sep 17 07:03:38 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-a[72373]: ts=2024-09-17T07:03:38.013Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-17T07:03:38.381 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:Sep 17 07:03:38 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-a[72373]: ts=2024-09-17T07:03:38.013Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-17T07:03:38.381 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:Sep 17 07:03:38 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-a[72373]: ts=2024-09-17T07:03:38.013Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-17T07:03:38.381 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:Sep 17 07:03:38 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-a[72373]: ts=2024-09-17T07:03:38.013Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-17T07:03:38.381 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:Sep 17 07:03:38 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-a[72373]: ts=2024-09-17T07:03:38.013Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-17T07:03:38.381 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:Sep 17 07:03:38 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-a[72373]: ts=2024-09-17T07:03:38.013Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-17T07:03:38.381 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:Sep 17 07:03:38 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-a[72373]: ts=2024-09-17T07:03:38.013Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-17T07:03:38.381 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:Sep 17 07:03:38 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-a[72373]: ts=2024-09-17T07:03:38.013Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-17T07:03:38.381 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:Sep 17 07:03:38 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-a[72373]: ts=2024-09-17T07:03:38.013Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-17T07:03:38.381 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:Sep 17 07:03:38 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-a[72373]: ts=2024-09-17T07:03:38.013Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-17T07:03:38.381 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:Sep 17 07:03:38 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-a[72373]: ts=2024-09-17T07:03:38.013Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-17T07:03:38.381 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:Sep 17 07:03:38 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-a[72373]: ts=2024-09-17T07:03:38.013Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-17T07:03:38.381 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:Sep 17 07:03:38 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-a[72373]: ts=2024-09-17T07:03:38.013Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-17T07:03:38.381 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:Sep 17 07:03:38 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-a[72373]: ts=2024-09-17T07:03:38.013Z caller=node_exporter.go:117 level=info collector=os 2024-09-17T07:03:38.382 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:Sep 17 07:03:38 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-a[72373]: ts=2024-09-17T07:03:38.013Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-17T07:03:38.382 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:Sep 17 07:03:38 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-a[72373]: ts=2024-09-17T07:03:38.013Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-17T07:03:38.382 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:Sep 17 07:03:38 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-a[72373]: ts=2024-09-17T07:03:38.013Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-17T07:03:38.382 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:Sep 17 07:03:38 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-a[72373]: ts=2024-09-17T07:03:38.013Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-17T07:03:38.382 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:Sep 17 07:03:38 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-a[72373]: ts=2024-09-17T07:03:38.013Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-17T07:03:38.382 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:Sep 17 07:03:38 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-a[72373]: ts=2024-09-17T07:03:38.013Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-17T07:03:38.382 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:Sep 17 07:03:38 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-a[72373]: ts=2024-09-17T07:03:38.013Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-17T07:03:38.382 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:Sep 17 07:03:38 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-a[72373]: ts=2024-09-17T07:03:38.013Z caller=node_exporter.go:117 level=info collector=stat 2024-09-17T07:03:38.382 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:Sep 17 07:03:38 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-a[72373]: ts=2024-09-17T07:03:38.013Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-17T07:03:38.382 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:Sep 17 07:03:38 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-a[72373]: ts=2024-09-17T07:03:38.013Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-17T07:03:38.382 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:Sep 17 07:03:38 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-a[72373]: ts=2024-09-17T07:03:38.013Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-17T07:03:38.382 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:Sep 17 07:03:38 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-a[72373]: ts=2024-09-17T07:03:38.013Z caller=node_exporter.go:117 level=info collector=time 2024-09-17T07:03:38.382 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:Sep 17 07:03:38 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-a[72373]: ts=2024-09-17T07:03:38.013Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-17T07:03:38.383 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:Sep 17 07:03:38 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-a[72373]: ts=2024-09-17T07:03:38.013Z caller=node_exporter.go:117 level=info collector=uname 2024-09-17T07:03:38.383 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:Sep 17 07:03:38 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-a[72373]: ts=2024-09-17T07:03:38.013Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-17T07:03:38.383 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:Sep 17 07:03:38 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-a[72373]: ts=2024-09-17T07:03:38.013Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-17T07:03:38.384 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:Sep 17 07:03:38 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-a[72373]: ts=2024-09-17T07:03:38.013Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-17T07:03:38.384 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:Sep 17 07:03:38 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-a[72373]: ts=2024-09-17T07:03:38.013Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-09-17T07:03:38.384 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:Sep 17 07:03:38 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-a[72373]: ts=2024-09-17T07:03:38.013Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-17T07:03:38.384 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:Sep 17 07:03:38 smithi100 bash[72166]: 94e0a7f831e23f91cd36bbad11105e50aecec8487ccb23a9610e7573ca695e67 2024-09-17T07:03:38.384 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:Sep 17 07:03:38 smithi100 systemd[1]: Started Ceph node-exporter.a for e848267c-74c1-11ef-bceb-c7b262605968. 2024-09-17T07:03:38.506 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:03:38 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:03:38.506 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:38 smithi135 ceph-mon[33238]: 5.8 scrub starts 2024-09-17T07:03:38.506 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:38 smithi135 ceph-mon[33238]: 5.8 scrub ok 2024-09-17T07:03:38.506 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:38 smithi135 ceph-mon[33238]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 854 B/s rd, 0 op/s 2024-09-17T07:03:38.506 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:38 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:38.506 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:38 smithi135 ceph-mon[33238]: 5.10 scrub starts 2024-09-17T07:03:38.506 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:38 smithi135 ceph-mon[33238]: 5.10 scrub ok 2024-09-17T07:03:38.506 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:38 smithi135 ceph-mon[33238]: 5.19 scrub starts 2024-09-17T07:03:38.506 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:38 smithi135 ceph-mon[33238]: 5.19 scrub ok 2024-09-17T07:03:38.624 INFO:teuthology.orchestra.run.smithi100.stdout:{"pg_ready":true,"pg_map":{"version":39,"stamp":"2024-09-17T07:03:37.006645+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596246,"num_objects":196,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":996,"num_read_kb":842,"num_write":609,"num_write_kb":2459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":604,"ondisk_log_size":604,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":248600,"kb_used_data":7896,"kb_used_omap":12,"kb_used_meta":240627,"kb_avail":749712616,"statfs":{"total":767960285184,"available":767705718784,"internally_reserved":0,"allocated":8085504,"data_stored":4538583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":246402616},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.985036"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.021243+0000","last_change":"2024-09-17T07:02:18.366053+0000","last_active":"2024-09-17T07:03:20.021243+0000","last_peered":"2024-09-17T07:03:20.021243+0000","last_clean":"2024-09-17T07:03:20.021243+0000","last_became_active":"2024-09-17T07:02:18.365773+0000","last_became_peered":"2024-09-17T07:02:18.365773+0000","last_unstale":"2024-09-17T07:03:20.021243+0000","last_undegraded":"2024-09-17T07:03:20.021243+0000","last_fullsized":"2024-09-17T07:03:20.021243+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:02:17.002904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:39.725507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.746536+0000","last_change":"2024-09-17T07:02:12.007016+0000","last_active":"2024-09-17T07:03:24.746536+0000","last_peered":"2024-09-17T07:03:24.746536+0000","last_clean":"2024-09-17T07:03:24.746536+0000","last_became_active":"2024-09-17T07:02:12.006754+0000","last_became_peered":"2024-09-17T07:02:12.006754+0000","last_unstale":"2024-09-17T07:03:24.746536+0000","last_undegraded":"2024-09-17T07:03:24.746536+0000","last_fullsized":"2024-09-17T07:03:24.746536+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:02:10.991363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:33.036975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"75'15","reported_seq":64,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-17T07:03:30.436458+0000","last_change":"2024-09-17T07:03:30.436540+0000","last_active":"2024-09-17T07:03:30.436458+0000","last_peered":"2024-09-17T07:03:30.436458+0000","last_clean":"2024-09-17T07:03:30.436458+0000","last_became_active":"2024-09-17T07:02:14.012962+0000","last_became_peered":"2024-09-17T07:02:14.012962+0000","last_unstale":"2024-09-17T07:03:30.436458+0000","last_undegraded":"2024-09-17T07:03:30.436458+0000","last_fullsized":"2024-09-17T07:03:30.436458+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-09-17T07:03:30.436444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:30.436444+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:35.913395+0000","scrub_duration":0.0023734419999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.743778+0000","last_change":"2024-09-17T07:02:16.015244+0000","last_active":"2024-09-17T07:03:24.743778+0000","last_peered":"2024-09-17T07:03:24.743778+0000","last_clean":"2024-09-17T07:03:24.743778+0000","last_became_active":"2024-09-17T07:02:16.014928+0000","last_became_peered":"2024-09-17T07:02:16.014928+0000","last_unstale":"2024-09-17T07:03:24.743778+0000","last_undegraded":"2024-09-17T07:03:24.743778+0000","last_fullsized":"2024-09-17T07:03:24.743778+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:02:14.999435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:38.559521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.020720+0000","last_change":"2024-09-17T07:02:12.010335+0000","last_active":"2024-09-17T07:03:20.020720+0000","last_peered":"2024-09-17T07:03:20.020720+0000","last_clean":"2024-09-17T07:03:20.020720+0000","last_became_active":"2024-09-17T07:02:12.010069+0000","last_became_peered":"2024-09-17T07:02:12.010069+0000","last_unstale":"2024-09-17T07:03:20.020720+0000","last_undegraded":"2024-09-17T07:03:20.020720+0000","last_fullsized":"2024-09-17T07:03:20.020720+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:02:10.991363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:29.849457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"75'9","reported_seq":56,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.743781+0000","last_change":"2024-09-17T07:03:17.864856+0000","last_active":"2024-09-17T07:03:24.743781+0000","last_peered":"2024-09-17T07:03:24.743781+0000","last_clean":"2024-09-17T07:03:24.743781+0000","last_became_active":"2024-09-17T07:02:14.016244+0000","last_became_peered":"2024-09-17T07:02:14.016244+0000","last_unstale":"2024-09-17T07:03:24.743781+0000","last_undegraded":"2024-09-17T07:03:24.743781+0000","last_fullsized":"2024-09-17T07:03:24.743781+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:03:17.864726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:17.864726+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:29.697946+0000","scrub_duration":0.0021756380000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.020053+0000","last_change":"2024-09-17T07:02:16.014746+0000","last_active":"2024-09-17T07:03:20.020053+0000","last_peered":"2024-09-17T07:03:20.020053+0000","last_clean":"2024-09-17T07:03:20.020053+0000","last_became_active":"2024-09-17T07:02:16.014380+0000","last_became_peered":"2024-09-17T07:02:16.014380+0000","last_unstale":"2024-09-17T07:03:20.020053+0000","last_undegraded":"2024-09-17T07:03:20.020053+0000","last_fullsized":"2024-09-17T07:03:20.020053+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:02:14.999435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.743964+0000","last_change":"2024-09-17T07:02:18.021079+0000","last_active":"2024-09-17T07:03:24.743964+0000","last_peered":"2024-09-17T07:03:24.743964+0000","last_clean":"2024-09-17T07:03:24.743964+0000","last_became_active":"2024-09-17T07:02:18.020790+0000","last_became_peered":"2024-09-17T07:02:18.020790+0000","last_unstale":"2024-09-17T07:03:24.743964+0000","last_undegraded":"2024-09-17T07:03:24.743964+0000","last_fullsized":"2024-09-17T07:03:24.743964+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:02:17.002904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.956433+0000","last_change":"2024-09-17T07:02:12.011431+0000","last_active":"2024-09-17T07:03:24.956433+0000","last_peered":"2024-09-17T07:03:24.956433+0000","last_clean":"2024-09-17T07:03:24.956433+0000","last_became_active":"2024-09-17T07:02:12.011129+0000","last_became_peered":"2024-09-17T07:02:12.011129+0000","last_unstale":"2024-09-17T07:03:24.956433+0000","last_undegraded":"2024-09-17T07:03:24.956433+0000","last_fullsized":"2024-09-17T07:03:24.956433+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:02:10.991363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:32.031791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"75'5","reported_seq":50,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-17T07:03:29.002334+0000","last_change":"2024-09-17T07:03:29.002445+0000","last_active":"2024-09-17T07:03:29.002334+0000","last_peered":"2024-09-17T07:03:29.002334+0000","last_clean":"2024-09-17T07:03:29.002334+0000","last_became_active":"2024-09-17T07:02:14.016640+0000","last_became_peered":"2024-09-17T07:02:14.016640+0000","last_unstale":"2024-09-17T07:03:29.002334+0000","last_undegraded":"2024-09-17T07:03:29.002334+0000","last_fullsized":"2024-09-17T07:03:29.002334+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-09-17T07:03:29.002312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:29.002312+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:44.000292+0000","scrub_duration":0.002172748,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":38,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.955709+0000","last_change":"2024-09-17T07:03:22.015629+0000","last_active":"2024-09-17T07:03:24.955709+0000","last_peered":"2024-09-17T07:03:24.955709+0000","last_clean":"2024-09-17T07:03:24.955709+0000","last_became_active":"2024-09-17T07:02:16.015401+0000","last_became_peered":"2024-09-17T07:02:16.015401+0000","last_unstale":"2024-09-17T07:03:24.955709+0000","last_undegraded":"2024-09-17T07:03:24.955709+0000","last_fullsized":"2024-09-17T07:03:24.955709+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:22.015535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:22.015535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:49.909183+0000","scrub_duration":0.001507752,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.536340+0000","last_change":"2024-09-17T07:02:18.024034+0000","last_active":"2024-09-17T07:03:20.536340+0000","last_peered":"2024-09-17T07:03:20.536340+0000","last_clean":"2024-09-17T07:03:20.536340+0000","last_became_active":"2024-09-17T07:02:18.023680+0000","last_became_peered":"2024-09-17T07:02:18.023680+0000","last_unstale":"2024-09-17T07:03:20.536340+0000","last_undegraded":"2024-09-17T07:03:20.536340+0000","last_fullsized":"2024-09-17T07:03:20.536340+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:02:17.002904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.956209+0000","last_change":"2024-09-17T07:02:12.012509+0000","last_active":"2024-09-17T07:03:24.956209+0000","last_peered":"2024-09-17T07:03:24.956209+0000","last_clean":"2024-09-17T07:03:24.956209+0000","last_became_active":"2024-09-17T07:02:12.012234+0000","last_became_peered":"2024-09-17T07:02:12.012234+0000","last_unstale":"2024-09-17T07:03:24.956209+0000","last_undegraded":"2024-09-17T07:03:24.956209+0000","last_fullsized":"2024-09-17T07:03:24.956209+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:02:10.991363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:34.001627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"75'9","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.744149+0000","last_change":"2024-09-17T07:02:14.013597+0000","last_active":"2024-09-17T07:03:24.744149+0000","last_peered":"2024-09-17T07:03:24.744149+0000","last_clean":"2024-09-17T07:03:24.744149+0000","last_became_active":"2024-09-17T07:02:14.013337+0000","last_became_peered":"2024-09-17T07:02:14.013337+0000","last_unstale":"2024-09-17T07:03:24.744149+0000","last_undegraded":"2024-09-17T07:03:24.744149+0000","last_fullsized":"2024-09-17T07:03:24.744149+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:02:12.994221+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:37.293441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.536482+0000","last_change":"2024-09-17T07:02:16.024037+0000","last_active":"2024-09-17T07:03:20.536482+0000","last_peered":"2024-09-17T07:03:20.536482+0000","last_clean":"2024-09-17T07:03:20.536482+0000","last_became_active":"2024-09-17T07:02:16.023774+0000","last_became_peered":"2024-09-17T07:02:16.023774+0000","last_unstale":"2024-09-17T07:03:20.536482+0000","last_undegraded":"2024-09-17T07:03:20.536482+0000","last_fullsized":"2024-09-17T07:03:20.536482+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:02:14.999435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:44.675786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.746025+0000","last_change":"2024-09-17T07:02:18.029470+0000","last_active":"2024-09-17T07:03:24.746025+0000","last_peered":"2024-09-17T07:03:24.746025+0000","last_clean":"2024-09-17T07:03:24.746025+0000","last_became_active":"2024-09-17T07:02:18.029178+0000","last_became_peered":"2024-09-17T07:02:18.029178+0000","last_unstale":"2024-09-17T07:03:24.746025+0000","last_undegraded":"2024-09-17T07:03:24.746025+0000","last_fullsized":"2024-09-17T07:03:24.746025+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:02:17.002904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:45.877645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"75'17","reported_seq":57,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.744692+0000","last_change":"2024-09-17T07:02:14.016560+0000","last_active":"2024-09-17T07:03:24.744692+0000","last_peered":"2024-09-17T07:03:24.744692+0000","last_clean":"2024-09-17T07:03:24.744692+0000","last_became_active":"2024-09-17T07:02:14.016270+0000","last_became_peered":"2024-09-17T07:02:14.016270+0000","last_unstale":"2024-09-17T07:03:24.744692+0000","last_undegraded":"2024-09-17T07:03:24.744692+0000","last_fullsized":"2024-09-17T07:03:24.744692+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:02:12.994221+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:34.020252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":42,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.956163+0000","last_change":"2024-09-17T07:03:14.092519+0000","last_active":"2024-09-17T07:03:24.956163+0000","last_peered":"2024-09-17T07:03:24.956163+0000","last_clean":"2024-09-17T07:03:24.956163+0000","last_became_active":"2024-09-17T07:02:12.011258+0000","last_became_peered":"2024-09-17T07:02:12.011258+0000","last_unstale":"2024-09-17T07:03:24.956163+0000","last_undegraded":"2024-09-17T07:03:24.956163+0000","last_fullsized":"2024-09-17T07:03:24.956163+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:14.092369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:14.092369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:19.152787+0000","scrub_duration":0.0015447460000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.020658+0000","last_change":"2024-09-17T07:02:16.018444+0000","last_active":"2024-09-17T07:03:20.020658+0000","last_peered":"2024-09-17T07:03:20.020658+0000","last_clean":"2024-09-17T07:03:20.020658+0000","last_became_active":"2024-09-17T07:02:16.018147+0000","last_became_peered":"2024-09-17T07:02:16.018147+0000","last_unstale":"2024-09-17T07:03:20.020658+0000","last_undegraded":"2024-09-17T07:03:20.020658+0000","last_fullsized":"2024-09-17T07:03:20.020658+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:02:14.999435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.745631+0000","last_change":"2024-09-17T07:02:18.022326+0000","last_active":"2024-09-17T07:03:24.745631+0000","last_peered":"2024-09-17T07:03:24.745631+0000","last_clean":"2024-09-17T07:03:24.745631+0000","last_became_active":"2024-09-17T07:02:18.022040+0000","last_became_peered":"2024-09-17T07:02:18.022040+0000","last_unstale":"2024-09-17T07:03:24.745631+0000","last_undegraded":"2024-09-17T07:03:24.745631+0000","last_fullsized":"2024-09-17T07:03:24.745631+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:02:17.002904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:32.036001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"75'10","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.744525+0000","last_change":"2024-09-17T07:03:20.933478+0000","last_active":"2024-09-17T07:03:24.744525+0000","last_peered":"2024-09-17T07:03:24.744525+0000","last_clean":"2024-09-17T07:03:24.744525+0000","last_became_active":"2024-09-17T07:02:14.007702+0000","last_became_peered":"2024-09-17T07:02:14.007702+0000","last_unstale":"2024-09-17T07:03:24.744525+0000","last_undegraded":"2024-09-17T07:03:24.744525+0000","last_fullsized":"2024-09-17T07:03:24.744525+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-09-17T07:03:20.933353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:20.933353+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:28.865120+0000","scrub_duration":0.0021221080000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.021191+0000","last_change":"2024-09-17T07:02:12.007831+0000","last_active":"2024-09-17T07:03:20.021191+0000","last_peered":"2024-09-17T07:03:20.021191+0000","last_clean":"2024-09-17T07:03:20.021191+0000","last_became_active":"2024-09-17T07:02:12.007553+0000","last_became_peered":"2024-09-17T07:02:12.007553+0000","last_unstale":"2024-09-17T07:03:20.021191+0000","last_undegraded":"2024-09-17T07:03:20.021191+0000","last_fullsized":"2024-09-17T07:03:20.021191+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:02:10.991363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:30.657047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"75'8","reported_seq":51,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-17T07:03:25.533647+0000","last_change":"2024-09-17T07:02:16.015345+0000","last_active":"2024-09-17T07:03:25.533647+0000","last_peered":"2024-09-17T07:03:25.533647+0000","last_clean":"2024-09-17T07:03:25.533647+0000","last_became_active":"2024-09-17T07:02:16.015001+0000","last_became_peered":"2024-09-17T07:02:16.015001+0000","last_unstale":"2024-09-17T07:03:25.533647+0000","last_undegraded":"2024-09-17T07:03:25.533647+0000","last_fullsized":"2024-09-17T07:03:25.533647+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:02:14.999435+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:35.484606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.744747+0000","last_change":"2024-09-17T07:02:18.017718+0000","last_active":"2024-09-17T07:03:24.744747+0000","last_peered":"2024-09-17T07:03:24.744747+0000","last_clean":"2024-09-17T07:03:24.744747+0000","last_became_active":"2024-09-17T07:02:18.017460+0000","last_became_peered":"2024-09-17T07:02:18.017460+0000","last_unstale":"2024-09-17T07:03:24.744747+0000","last_undegraded":"2024-09-17T07:03:24.744747+0000","last_fullsized":"2024-09-17T07:03:24.744747+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:02:17.002904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:29.635650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"75'15","reported_seq":64,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.022157+0000","last_change":"2024-09-17T07:03:16.067607+0000","last_active":"2024-09-17T07:03:20.022157+0000","last_peered":"2024-09-17T07:03:20.022157+0000","last_clean":"2024-09-17T07:03:20.022157+0000","last_became_active":"2024-09-17T07:02:14.016397+0000","last_became_peered":"2024-09-17T07:02:14.016397+0000","last_unstale":"2024-09-17T07:03:20.022157+0000","last_undegraded":"2024-09-17T07:03:20.022157+0000","last_fullsized":"2024-09-17T07:03:20.022157+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-09-17T07:03:16.067479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:16.067479+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:28.548727+0000","scrub_duration":0.0025498640000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":41,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-17T07:03:28.072710+0000","last_change":"2024-09-17T07:03:28.072773+0000","last_active":"2024-09-17T07:03:28.072710+0000","last_peered":"2024-09-17T07:03:28.072710+0000","last_clean":"2024-09-17T07:03:28.072710+0000","last_became_active":"2024-09-17T07:02:12.007913+0000","last_became_peered":"2024-09-17T07:02:12.007913+0000","last_unstale":"2024-09-17T07:03:28.072710+0000","last_undegraded":"2024-09-17T07:03:28.072710+0000","last_fullsized":"2024-09-17T07:03:28.072710+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:28.072697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:28.072697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:43.368112+0000","scrub_duration":0.0012624730000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"75'8","reported_seq":51,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-17T07:03:25.533626+0000","last_change":"2024-09-17T07:02:16.022051+0000","last_active":"2024-09-17T07:03:25.533626+0000","last_peered":"2024-09-17T07:03:25.533626+0000","last_clean":"2024-09-17T07:03:25.533626+0000","last_became_active":"2024-09-17T07:02:16.021762+0000","last_became_peered":"2024-09-17T07:02:16.021762+0000","last_unstale":"2024-09-17T07:03:25.533626+0000","last_undegraded":"2024-09-17T07:03:25.533626+0000","last_fullsized":"2024-09-17T07:03:25.533626+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:02:14.999435+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:36.148452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.536729+0000","last_change":"2024-09-17T07:02:18.025574+0000","last_active":"2024-09-17T07:03:20.536729+0000","last_peered":"2024-09-17T07:03:20.536729+0000","last_clean":"2024-09-17T07:03:20.536729+0000","last_became_active":"2024-09-17T07:02:18.025332+0000","last_became_peered":"2024-09-17T07:02:18.025332+0000","last_unstale":"2024-09-17T07:03:20.536729+0000","last_undegraded":"2024-09-17T07:03:20.536729+0000","last_fullsized":"2024-09-17T07:03:20.536729+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:02:17.002904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:36.070328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"75'11","reported_seq":48,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.744909+0000","last_change":"2024-09-17T07:02:14.010584+0000","last_active":"2024-09-17T07:03:24.744909+0000","last_peered":"2024-09-17T07:03:24.744909+0000","last_clean":"2024-09-17T07:03:24.744909+0000","last_became_active":"2024-09-17T07:02:14.010227+0000","last_became_peered":"2024-09-17T07:02:14.010227+0000","last_unstale":"2024-09-17T07:03:24.744909+0000","last_undegraded":"2024-09-17T07:03:24.744909+0000","last_fullsized":"2024-09-17T07:03:24.744909+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:02:12.994221+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:33.857964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":42,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-17T07:03:31.076149+0000","last_change":"2024-09-17T07:03:31.076235+0000","last_active":"2024-09-17T07:03:31.076149+0000","last_peered":"2024-09-17T07:03:31.076149+0000","last_clean":"2024-09-17T07:03:31.076149+0000","last_became_active":"2024-09-17T07:02:12.013182+0000","last_became_peered":"2024-09-17T07:02:12.013182+0000","last_unstale":"2024-09-17T07:03:31.076149+0000","last_undegraded":"2024-09-17T07:03:31.076149+0000","last_fullsized":"2024-09-17T07:03:31.076149+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:31.076134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:31.076134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:37.803854+0000","scrub_duration":0.0012890040000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":37,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.536129+0000","last_change":"2024-09-17T07:03:18.816900+0000","last_active":"2024-09-17T07:03:20.536129+0000","last_peered":"2024-09-17T07:03:20.536129+0000","last_clean":"2024-09-17T07:03:20.536129+0000","last_became_active":"2024-09-17T07:02:16.020224+0000","last_became_peered":"2024-09-17T07:02:16.020224+0000","last_unstale":"2024-09-17T07:03:20.536129+0000","last_undegraded":"2024-09-17T07:03:20.536129+0000","last_fullsized":"2024-09-17T07:03:20.536129+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:18.816795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:18.816795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:46.254399+0000","scrub_duration":0.001573212,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.021649+0000","last_change":"2024-09-17T07:02:18.365401+0000","last_active":"2024-09-17T07:03:20.021649+0000","last_peered":"2024-09-17T07:03:20.021649+0000","last_clean":"2024-09-17T07:03:20.021649+0000","last_became_active":"2024-09-17T07:02:18.365000+0000","last_became_peered":"2024-09-17T07:02:18.365000+0000","last_unstale":"2024-09-17T07:03:20.021649+0000","last_undegraded":"2024-09-17T07:03:20.021649+0000","last_fullsized":"2024-09-17T07:03:20.021649+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:02:17.002904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"75'14","reported_seq":50,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.743996+0000","last_change":"2024-09-17T07:02:14.008065+0000","last_active":"2024-09-17T07:03:24.743996+0000","last_peered":"2024-09-17T07:03:24.743996+0000","last_clean":"2024-09-17T07:03:24.743996+0000","last_became_active":"2024-09-17T07:02:14.007810+0000","last_became_peered":"2024-09-17T07:02:14.007810+0000","last_unstale":"2024-09-17T07:03:24.743996+0000","last_undegraded":"2024-09-17T07:03:24.743996+0000","last_fullsized":"2024-09-17T07:03:24.743996+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:02:12.994221+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:38.353465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.115548+0000","last_change":"2024-09-17T07:03:16.455893+0000","last_active":"2024-09-17T07:03:20.115548+0000","last_peered":"2024-09-17T07:03:20.115548+0000","last_clean":"2024-09-17T07:03:20.115548+0000","last_became_active":"2024-09-17T07:02:12.011766+0000","last_became_peered":"2024-09-17T07:02:12.011766+0000","last_unstale":"2024-09-17T07:03:20.115548+0000","last_undegraded":"2024-09-17T07:03:20.115548+0000","last_fullsized":"2024-09-17T07:03:20.115548+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:16.455684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:16.455684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:22.392690+0000","scrub_duration":0.0020498690000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"75'8","reported_seq":50,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-17T07:03:25.533722+0000","last_change":"2024-09-17T07:02:16.021429+0000","last_active":"2024-09-17T07:03:25.533722+0000","last_peered":"2024-09-17T07:03:25.533722+0000","last_clean":"2024-09-17T07:03:25.533722+0000","last_became_active":"2024-09-17T07:02:16.020628+0000","last_became_peered":"2024-09-17T07:02:16.020628+0000","last_unstale":"2024-09-17T07:03:25.533722+0000","last_undegraded":"2024-09-17T07:03:25.533722+0000","last_fullsized":"2024-09-17T07:03:25.533722+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:02:14.999435+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":36,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.955981+0000","last_change":"2024-09-17T07:03:19.030574+0000","last_active":"2024-09-17T07:03:24.955981+0000","last_peered":"2024-09-17T07:03:24.955981+0000","last_clean":"2024-09-17T07:03:24.955981+0000","last_became_active":"2024-09-17T07:02:18.364981+0000","last_became_peered":"2024-09-17T07:02:18.364981+0000","last_unstale":"2024-09-17T07:03:24.955981+0000","last_undegraded":"2024-09-17T07:03:24.955981+0000","last_fullsized":"2024-09-17T07:03:24.955981+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:19.030470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:03:19.030470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:48.307795+0000","scrub_duration":0.00152618,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"75'18","reported_seq":71,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-17T07:03:26.413521+0000","last_change":"2024-09-17T07:03:26.413610+0000","last_active":"2024-09-17T07:03:26.413521+0000","last_peered":"2024-09-17T07:03:26.413521+0000","last_clean":"2024-09-17T07:03:26.413521+0000","last_became_active":"2024-09-17T07:02:14.028667+0000","last_became_peered":"2024-09-17T07:02:14.028667+0000","last_unstale":"2024-09-17T07:03:26.413521+0000","last_undegraded":"2024-09-17T07:03:26.413521+0000","last_fullsized":"2024-09-17T07:03:26.413521+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'18","last_scrub_stamp":"2024-09-17T07:03:26.413506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:26.413506+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:41.741155+0000","scrub_duration":0.0022855470000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.134708+0000","last_change":"2024-09-17T07:03:24.134708+0000","last_active":"2024-09-17T07:03:24.134708+0000","last_peered":"2024-09-17T07:03:24.134708+0000","last_clean":"2024-09-17T07:03:24.134708+0000","last_became_active":"2024-09-17T07:02:12.005687+0000","last_became_peered":"2024-09-17T07:02:12.005687+0000","last_unstale":"2024-09-17T07:03:24.134708+0000","last_undegraded":"2024-09-17T07:03:24.134708+0000","last_fullsized":"2024-09-17T07:03:24.134708+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:24.134627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:24.134627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:29.080338+0000","scrub_duration":0.001111148,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.743837+0000","last_change":"2024-09-17T07:02:16.020013+0000","last_active":"2024-09-17T07:03:24.743837+0000","last_peered":"2024-09-17T07:03:24.743837+0000","last_clean":"2024-09-17T07:03:24.743837+0000","last_became_active":"2024-09-17T07:02:16.019747+0000","last_became_peered":"2024-09-17T07:02:16.019747+0000","last_unstale":"2024-09-17T07:03:24.743837+0000","last_undegraded":"2024-09-17T07:03:24.743837+0000","last_fullsized":"2024-09-17T07:03:24.743837+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:02:14.999435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:30.253095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.744121+0000","last_change":"2024-09-17T07:02:18.024506+0000","last_active":"2024-09-17T07:03:24.744121+0000","last_peered":"2024-09-17T07:03:24.744121+0000","last_clean":"2024-09-17T07:03:24.744121+0000","last_became_active":"2024-09-17T07:02:18.024043+0000","last_became_peered":"2024-09-17T07:02:18.024043+0000","last_unstale":"2024-09-17T07:03:24.744121+0000","last_undegraded":"2024-09-17T07:03:24.744121+0000","last_fullsized":"2024-09-17T07:03:24.744121+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:02:17.002904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:29.154987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"75'19","reported_seq":65,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.746593+0000","last_change":"2024-09-17T07:02:14.027680+0000","last_active":"2024-09-17T07:03:24.746593+0000","last_peered":"2024-09-17T07:03:24.746593+0000","last_clean":"2024-09-17T07:03:24.746593+0000","last_became_active":"2024-09-17T07:02:14.027389+0000","last_became_peered":"2024-09-17T07:02:14.027389+0000","last_unstale":"2024-09-17T07:03:24.746593+0000","last_undegraded":"2024-09-17T07:03:24.746593+0000","last_fullsized":"2024-09-17T07:03:24.746593+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:02:12.994221+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:31.589637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":42,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-17T07:03:32.109780+0000","last_change":"2024-09-17T07:03:32.109842+0000","last_active":"2024-09-17T07:03:32.109780+0000","last_peered":"2024-09-17T07:03:32.109780+0000","last_clean":"2024-09-17T07:03:32.109780+0000","last_became_active":"2024-09-17T07:02:12.011917+0000","last_became_peered":"2024-09-17T07:02:12.011917+0000","last_unstale":"2024-09-17T07:03:32.109780+0000","last_undegraded":"2024-09-17T07:03:32.109780+0000","last_fullsized":"2024-09-17T07:03:32.109780+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:32.109765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:32.109765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:52.813415+0000","scrub_duration":0.001335314,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.114920+0000","last_change":"2024-09-17T07:02:16.016938+0000","last_active":"2024-09-17T07:03:20.114920+0000","last_peered":"2024-09-17T07:03:20.114920+0000","last_clean":"2024-09-17T07:03:20.114920+0000","last_became_active":"2024-09-17T07:02:16.016199+0000","last_became_peered":"2024-09-17T07:02:16.016199+0000","last_unstale":"2024-09-17T07:03:20.114920+0000","last_undegraded":"2024-09-17T07:03:20.114920+0000","last_fullsized":"2024-09-17T07:03:20.114920+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:02:14.999435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:36.124749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":35,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-17T07:03:30.068821+0000","last_change":"2024-09-17T07:03:30.068895+0000","last_active":"2024-09-17T07:03:30.068821+0000","last_peered":"2024-09-17T07:03:30.068821+0000","last_clean":"2024-09-17T07:03:30.068821+0000","last_became_active":"2024-09-17T07:02:18.364835+0000","last_became_peered":"2024-09-17T07:02:18.364835+0000","last_unstale":"2024-09-17T07:03:30.068821+0000","last_undegraded":"2024-09-17T07:03:30.068821+0000","last_fullsized":"2024-09-17T07:03:30.068821+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:30.068808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:03:30.068808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:51.270374+0000","scrub_duration":0.0012893679999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"75'10","reported_seq":54,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-17T07:03:26.127454+0000","last_change":"2024-09-17T07:03:26.127545+0000","last_active":"2024-09-17T07:03:26.127454+0000","last_peered":"2024-09-17T07:03:26.127454+0000","last_clean":"2024-09-17T07:03:26.127454+0000","last_became_active":"2024-09-17T07:02:14.010637+0000","last_became_peered":"2024-09-17T07:02:14.010637+0000","last_unstale":"2024-09-17T07:03:26.127454+0000","last_undegraded":"2024-09-17T07:03:26.127454+0000","last_fullsized":"2024-09-17T07:03:26.127454+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-09-17T07:03:26.127439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:26.127439+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:35.535770+0000","scrub_duration":0.001898479,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:25.148144+0000","last_change":"2024-09-17T07:03:25.148144+0000","last_active":"2024-09-17T07:03:25.148144+0000","last_peered":"2024-09-17T07:03:25.148144+0000","last_clean":"2024-09-17T07:03:25.148144+0000","last_became_active":"2024-09-17T07:02:12.010803+0000","last_became_peered":"2024-09-17T07:02:12.010803+0000","last_unstale":"2024-09-17T07:03:25.148144+0000","last_undegraded":"2024-09-17T07:03:25.148144+0000","last_fullsized":"2024-09-17T07:03:25.148144+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:25.147878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:25.147878+0000","last_clean_scrub_stamp":"2024-09-17T07:03:25.147878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:33.071698+0000","scrub_duration":0.001353723,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"89'9","reported_seq":63,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-17T07:03:29.961568+0000","last_change":"2024-09-17T07:03:29.961568+0000","last_active":"2024-09-17T07:03:29.961568+0000","last_peered":"2024-09-17T07:03:29.961568+0000","last_clean":"2024-09-17T07:03:29.961568+0000","last_became_active":"2024-09-17T07:02:16.021360+0000","last_became_peered":"2024-09-17T07:02:16.021360+0000","last_unstale":"2024-09-17T07:03:29.961568+0000","last_undegraded":"2024-09-17T07:03:29.961568+0000","last_fullsized":"2024-09-17T07:03:29.961568+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'9","last_scrub_stamp":"2024-09-17T07:03:29.961439+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-09-17T07:03:29.961439+0000","last_clean_scrub_stamp":"2024-09-17T07:03:29.961439+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:40.753813+0000","scrub_duration":0.0027696880000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"75'1","reported_seq":37,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-17T07:03:25.048964+0000","last_change":"2024-09-17T07:03:25.049036+0000","last_active":"2024-09-17T07:03:25.048964+0000","last_peered":"2024-09-17T07:03:25.048964+0000","last_clean":"2024-09-17T07:03:25.048964+0000","last_became_active":"2024-09-17T07:02:18.029772+0000","last_became_peered":"2024-09-17T07:02:18.029772+0000","last_unstale":"2024-09-17T07:03:25.048964+0000","last_undegraded":"2024-09-17T07:03:25.048964+0000","last_fullsized":"2024-09-17T07:03:25.048964+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-17T07:03:25.048949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:03:25.048949+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:53.755876+0000","scrub_duration":0.003886123,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"75'28","reported_seq":92,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-17T07:03:25.533950+0000","last_change":"2024-09-17T07:02:14.017598+0000","last_active":"2024-09-17T07:03:25.533950+0000","last_peered":"2024-09-17T07:03:25.533950+0000","last_clean":"2024-09-17T07:03:25.533950+0000","last_became_active":"2024-09-17T07:02:14.017377+0000","last_became_peered":"2024-09-17T07:02:14.017377+0000","last_unstale":"2024-09-17T07:03:25.533950+0000","last_undegraded":"2024-09-17T07:03:25.533950+0000","last_fullsized":"2024-09-17T07:03:25.533950+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:02:12.994221+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:31.123508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.536680+0000","last_change":"2024-09-17T07:02:12.009135+0000","last_active":"2024-09-17T07:03:20.536680+0000","last_peered":"2024-09-17T07:03:20.536680+0000","last_clean":"2024-09-17T07:03:20.536680+0000","last_became_active":"2024-09-17T07:02:12.008802+0000","last_became_peered":"2024-09-17T07:02:12.008802+0000","last_unstale":"2024-09-17T07:03:20.536680+0000","last_undegraded":"2024-09-17T07:03:20.536680+0000","last_fullsized":"2024-09-17T07:03:20.536680+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:02:10.991363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:27.825147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-17T07:03:28.981163+0000","last_change":"2024-09-17T07:03:28.981244+0000","last_active":"2024-09-17T07:03:28.981163+0000","last_peered":"2024-09-17T07:03:28.981163+0000","last_clean":"2024-09-17T07:03:28.981163+0000","last_became_active":"2024-09-17T07:02:16.016606+0000","last_became_peered":"2024-09-17T07:02:16.016606+0000","last_unstale":"2024-09-17T07:03:28.981163+0000","last_undegraded":"2024-09-17T07:03:28.981163+0000","last_fullsized":"2024-09-17T07:03:28.981163+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:28.981147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:28.981147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:37.078325+0000","scrub_duration":0.0012741460000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.020508+0000","last_change":"2024-09-17T07:02:18.026015+0000","last_active":"2024-09-17T07:03:20.020508+0000","last_peered":"2024-09-17T07:03:20.020508+0000","last_clean":"2024-09-17T07:03:20.020508+0000","last_became_active":"2024-09-17T07:02:18.024602+0000","last_became_peered":"2024-09-17T07:02:18.024602+0000","last_unstale":"2024-09-17T07:03:20.020508+0000","last_undegraded":"2024-09-17T07:03:20.020508+0000","last_fullsized":"2024-09-17T07:03:20.020508+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:02:17.002904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:39.729726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"75'13","reported_seq":66,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-17T07:03:32.079596+0000","last_change":"2024-09-17T07:03:32.079678+0000","last_active":"2024-09-17T07:03:32.079596+0000","last_peered":"2024-09-17T07:03:32.079596+0000","last_clean":"2024-09-17T07:03:32.079596+0000","last_became_active":"2024-09-17T07:02:14.019838+0000","last_became_peered":"2024-09-17T07:02:14.019838+0000","last_unstale":"2024-09-17T07:03:32.079596+0000","last_undegraded":"2024-09-17T07:03:32.079596+0000","last_fullsized":"2024-09-17T07:03:32.079596+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'13","last_scrub_stamp":"2024-09-17T07:03:32.079579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:32.079579+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:42.502680+0000","scrub_duration":0.001975929,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"76'5","reported_seq":112,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-17T07:03:31.102791+0000","last_change":"2024-09-17T07:02:16.008848+0000","last_active":"2024-09-17T07:03:31.102791+0000","last_peered":"2024-09-17T07:03:31.102791+0000","last_clean":"2024-09-17T07:03:31.102791+0000","last_became_active":"2024-09-17T07:02:12.006883+0000","last_became_peered":"2024-09-17T07:02:12.006883+0000","last_unstale":"2024-09-17T07:03:31.102791+0000","last_undegraded":"2024-09-17T07:03:31.102791+0000","last_fullsized":"2024-09-17T07:03:31.102791+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:02:10.991363+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033739799999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":74,"num_read_kb":69,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.745197+0000","last_change":"2024-09-17T07:02:12.011001+0000","last_active":"2024-09-17T07:03:24.745197+0000","last_peered":"2024-09-17T07:03:24.745197+0000","last_clean":"2024-09-17T07:03:24.745197+0000","last_became_active":"2024-09-17T07:02:12.010753+0000","last_became_peered":"2024-09-17T07:02:12.010753+0000","last_unstale":"2024-09-17T07:03:24.745197+0000","last_undegraded":"2024-09-17T07:03:24.745197+0000","last_fullsized":"2024-09-17T07:03:24.745197+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:02:10.991363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:29.046141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.745596+0000","last_change":"2024-09-17T07:02:16.014775+0000","last_active":"2024-09-17T07:03:24.745596+0000","last_peered":"2024-09-17T07:03:24.745596+0000","last_clean":"2024-09-17T07:03:24.745596+0000","last_became_active":"2024-09-17T07:02:16.014492+0000","last_became_peered":"2024-09-17T07:02:16.014492+0000","last_unstale":"2024-09-17T07:03:24.745596+0000","last_undegraded":"2024-09-17T07:03:24.745596+0000","last_fullsized":"2024-09-17T07:03:24.745596+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:02:14.999435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:41.880289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.956165+0000","last_change":"2024-09-17T07:02:18.365627+0000","last_active":"2024-09-17T07:03:24.956165+0000","last_peered":"2024-09-17T07:03:24.956165+0000","last_clean":"2024-09-17T07:03:24.956165+0000","last_became_active":"2024-09-17T07:02:18.365369+0000","last_became_peered":"2024-09-17T07:02:18.365369+0000","last_unstale":"2024-09-17T07:03:24.956165+0000","last_undegraded":"2024-09-17T07:03:24.956165+0000","last_fullsized":"2024-09-17T07:03:24.956165+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:02:17.002904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:28.613944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"75'12","reported_seq":58,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-17T07:03:27.987849+0000","last_change":"2024-09-17T07:03:27.987937+0000","last_active":"2024-09-17T07:03:27.987849+0000","last_peered":"2024-09-17T07:03:27.987849+0000","last_clean":"2024-09-17T07:03:27.987849+0000","last_became_active":"2024-09-17T07:02:14.016206+0000","last_became_peered":"2024-09-17T07:02:14.016206+0000","last_unstale":"2024-09-17T07:03:27.987849+0000","last_undegraded":"2024-09-17T07:03:27.987849+0000","last_fullsized":"2024-09-17T07:03:27.987849+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-09-17T07:03:27.987835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:27.987835+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:50.273053+0000","scrub_duration":0.0020148219999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"70'1","reported_seq":37,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.746267+0000","last_change":"2024-09-17T07:02:16.017796+0000","last_active":"2024-09-17T07:03:24.746267+0000","last_peered":"2024-09-17T07:03:24.746267+0000","last_clean":"2024-09-17T07:03:24.746267+0000","last_became_active":"2024-09-17T07:02:12.009476+0000","last_became_peered":"2024-09-17T07:02:12.009476+0000","last_unstale":"2024-09-17T07:03:24.746267+0000","last_undegraded":"2024-09-17T07:03:24.746267+0000","last_fullsized":"2024-09-17T07:03:24.746267+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:02:10.991363+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:37.550940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032298629999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.955625+0000","last_change":"2024-09-17T07:02:12.014798+0000","last_active":"2024-09-17T07:03:24.955625+0000","last_peered":"2024-09-17T07:03:24.955625+0000","last_clean":"2024-09-17T07:03:24.955625+0000","last_became_active":"2024-09-17T07:02:12.014623+0000","last_became_peered":"2024-09-17T07:02:12.014623+0000","last_unstale":"2024-09-17T07:03:24.955625+0000","last_undegraded":"2024-09-17T07:03:24.955625+0000","last_fullsized":"2024-09-17T07:03:24.955625+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:02:10.991363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:40.031155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.536154+0000","last_change":"2024-09-17T07:02:16.021825+0000","last_active":"2024-09-17T07:03:20.536154+0000","last_peered":"2024-09-17T07:03:20.536154+0000","last_clean":"2024-09-17T07:03:20.536154+0000","last_became_active":"2024-09-17T07:02:16.021561+0000","last_became_peered":"2024-09-17T07:02:16.021561+0000","last_unstale":"2024-09-17T07:03:20.536154+0000","last_undegraded":"2024-09-17T07:03:20.536154+0000","last_fullsized":"2024-09-17T07:03:20.536154+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:02:14.999435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.020801+0000","last_change":"2024-09-17T07:02:18.022359+0000","last_active":"2024-09-17T07:03:20.020801+0000","last_peered":"2024-09-17T07:03:20.020801+0000","last_clean":"2024-09-17T07:03:20.020801+0000","last_became_active":"2024-09-17T07:02:18.022082+0000","last_became_peered":"2024-09-17T07:02:18.022082+0000","last_unstale":"2024-09-17T07:03:20.020801+0000","last_undegraded":"2024-09-17T07:03:20.020801+0000","last_fullsized":"2024-09-17T07:03:20.020801+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:02:17.002904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:36.430496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"75'14","reported_seq":50,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.115370+0000","last_change":"2024-09-17T07:02:14.029208+0000","last_active":"2024-09-17T07:03:20.115370+0000","last_peered":"2024-09-17T07:03:20.115370+0000","last_clean":"2024-09-17T07:03:20.115370+0000","last_became_active":"2024-09-17T07:02:14.028857+0000","last_became_peered":"2024-09-17T07:02:14.028857+0000","last_unstale":"2024-09-17T07:03:20.115370+0000","last_undegraded":"2024-09-17T07:03:20.115370+0000","last_fullsized":"2024-09-17T07:03:20.115370+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:02:12.994221+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:29.390413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"72'2","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-17T07:03:25.432684+0000","last_change":"2024-09-17T07:03:25.432755+0000","last_active":"2024-09-17T07:03:25.432684+0000","last_peered":"2024-09-17T07:03:25.432684+0000","last_clean":"2024-09-17T07:03:25.432684+0000","last_became_active":"2024-09-17T07:02:12.005127+0000","last_became_peered":"2024-09-17T07:02:12.005127+0000","last_unstale":"2024-09-17T07:03:25.432684+0000","last_undegraded":"2024-09-17T07:03:25.432684+0000","last_fullsized":"2024-09-17T07:03:25.432684+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'2","last_scrub_stamp":"2024-09-17T07:03:25.432670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:25.432670+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:25.994239+0000","scrub_duration":0.001640454,"objects_trimmed":0,"snaptrim_duration":0.0010980219999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.536731+0000","last_change":"2024-09-17T07:03:14.771416+0000","last_active":"2024-09-17T07:03:20.536731+0000","last_peered":"2024-09-17T07:03:20.536731+0000","last_clean":"2024-09-17T07:03:20.536731+0000","last_became_active":"2024-09-17T07:02:12.004606+0000","last_became_peered":"2024-09-17T07:02:12.004606+0000","last_unstale":"2024-09-17T07:03:20.536731+0000","last_undegraded":"2024-09-17T07:03:20.536731+0000","last_fullsized":"2024-09-17T07:03:20.536731+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:14.771307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:14.771307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:19.133578+0000","scrub_duration":0.0020495679999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"78'128","reported_seq":290,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.955697+0000","last_change":"2024-09-17T07:02:43.225880+0000","last_active":"2024-09-17T07:03:24.955697+0000","last_peered":"2024-09-17T07:03:24.955697+0000","last_clean":"2024-09-17T07:03:24.955697+0000","last_became_active":"2024-09-17T07:02:03.020675+0000","last_became_peered":"2024-09-17T07:02:03.020675+0000","last_unstale":"2024-09-17T07:03:24.955697+0000","last_undegraded":"2024-09-17T07:03:24.955697+0000","last_fullsized":"2024-09-17T07:03:24.955697+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'117","last_scrub_stamp":"2024-09-17T07:02:43.225790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:00:09.390381+0000","last_clean_scrub_stamp":"2024-09-17T07:02:43.225790+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:12.817117+0000","scrub_duration":0.001686302,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-17T07:03:30.069621+0000","last_change":"2024-09-17T07:03:30.069684+0000","last_active":"2024-09-17T07:03:30.069621+0000","last_peered":"2024-09-17T07:03:30.069621+0000","last_clean":"2024-09-17T07:03:30.069621+0000","last_became_active":"2024-09-17T07:02:16.016031+0000","last_became_peered":"2024-09-17T07:02:16.016031+0000","last_unstale":"2024-09-17T07:03:30.069621+0000","last_undegraded":"2024-09-17T07:03:30.069621+0000","last_fullsized":"2024-09-17T07:03:30.069621+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:30.069608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:30.069608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:36.831454+0000","scrub_duration":0.0012155379999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.537052+0000","last_change":"2024-09-17T07:02:18.023684+0000","last_active":"2024-09-17T07:03:20.537052+0000","last_peered":"2024-09-17T07:03:20.537052+0000","last_clean":"2024-09-17T07:03:20.537052+0000","last_became_active":"2024-09-17T07:02:18.023321+0000","last_became_peered":"2024-09-17T07:02:18.023321+0000","last_unstale":"2024-09-17T07:03:20.537052+0000","last_undegraded":"2024-09-17T07:03:20.537052+0000","last_fullsized":"2024-09-17T07:03:20.537052+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:02:17.002904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:39.925496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"75'19","reported_seq":59,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.115170+0000","last_change":"2024-09-17T07:02:14.027873+0000","last_active":"2024-09-17T07:03:20.115170+0000","last_peered":"2024-09-17T07:03:20.115170+0000","last_clean":"2024-09-17T07:03:20.115170+0000","last_became_active":"2024-09-17T07:02:14.027313+0000","last_became_peered":"2024-09-17T07:02:14.027313+0000","last_unstale":"2024-09-17T07:03:20.115170+0000","last_undegraded":"2024-09-17T07:03:20.115170+0000","last_fullsized":"2024-09-17T07:03:20.115170+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:02:12.994221+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:28.470911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":42,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-17T07:03:25.672373+0000","last_change":"2024-09-17T07:03:25.672437+0000","last_active":"2024-09-17T07:03:25.672373+0000","last_peered":"2024-09-17T07:03:25.672373+0000","last_clean":"2024-09-17T07:03:25.672373+0000","last_became_active":"2024-09-17T07:02:12.012037+0000","last_became_peered":"2024-09-17T07:02:12.012037+0000","last_unstale":"2024-09-17T07:03:25.672373+0000","last_undegraded":"2024-09-17T07:03:25.672373+0000","last_fullsized":"2024-09-17T07:03:25.672373+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:25.672360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:25.672360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:54.809122+0000","scrub_duration":0.0012704459999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-17T07:03:27.660734+0000","last_change":"2024-09-17T07:03:27.660802+0000","last_active":"2024-09-17T07:03:27.660734+0000","last_peered":"2024-09-17T07:03:27.660734+0000","last_clean":"2024-09-17T07:03:27.660734+0000","last_became_active":"2024-09-17T07:02:16.014631+0000","last_became_peered":"2024-09-17T07:02:16.014631+0000","last_unstale":"2024-09-17T07:03:27.660734+0000","last_undegraded":"2024-09-17T07:03:27.660734+0000","last_fullsized":"2024-09-17T07:03:27.660734+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:27.660721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:27.660721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:50.654559+0000","scrub_duration":0.0012746820000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.955661+0000","last_change":"2024-09-17T07:02:18.023600+0000","last_active":"2024-09-17T07:03:24.955661+0000","last_peered":"2024-09-17T07:03:24.955661+0000","last_clean":"2024-09-17T07:03:24.955661+0000","last_became_active":"2024-09-17T07:02:18.023299+0000","last_became_peered":"2024-09-17T07:02:18.023299+0000","last_unstale":"2024-09-17T07:03:24.955661+0000","last_undegraded":"2024-09-17T07:03:24.955661+0000","last_fullsized":"2024-09-17T07:03:24.955661+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:02:17.002904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:45.370120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"75'9","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.746394+0000","last_change":"2024-09-17T07:02:14.015257+0000","last_active":"2024-09-17T07:03:24.746394+0000","last_peered":"2024-09-17T07:03:24.746394+0000","last_clean":"2024-09-17T07:03:24.746394+0000","last_became_active":"2024-09-17T07:02:14.014939+0000","last_became_peered":"2024-09-17T07:02:14.014939+0000","last_unstale":"2024-09-17T07:03:24.746394+0000","last_undegraded":"2024-09-17T07:03:24.746394+0000","last_fullsized":"2024-09-17T07:03:24.746394+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:02:12.994221+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:34.702987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.021841+0000","last_change":"2024-09-17T07:02:12.000308+0000","last_active":"2024-09-17T07:03:20.021841+0000","last_peered":"2024-09-17T07:03:20.021841+0000","last_clean":"2024-09-17T07:03:20.021841+0000","last_became_active":"2024-09-17T07:02:12.000005+0000","last_became_peered":"2024-09-17T07:02:12.000005+0000","last_unstale":"2024-09-17T07:03:20.021841+0000","last_undegraded":"2024-09-17T07:03:20.021841+0000","last_fullsized":"2024-09-17T07:03:20.021841+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:02:10.991363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.745245+0000","last_change":"2024-09-17T07:02:16.017596+0000","last_active":"2024-09-17T07:03:24.745245+0000","last_peered":"2024-09-17T07:03:24.745245+0000","last_clean":"2024-09-17T07:03:24.745245+0000","last_became_active":"2024-09-17T07:02:16.017271+0000","last_became_peered":"2024-09-17T07:02:16.017271+0000","last_unstale":"2024-09-17T07:03:24.745245+0000","last_undegraded":"2024-09-17T07:03:24.745245+0000","last_fullsized":"2024-09-17T07:03:24.745245+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:02:14.999435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:32.095723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.746161+0000","last_change":"2024-09-17T07:02:18.027276+0000","last_active":"2024-09-17T07:03:24.746161+0000","last_peered":"2024-09-17T07:03:24.746161+0000","last_clean":"2024-09-17T07:03:24.746161+0000","last_became_active":"2024-09-17T07:02:18.026994+0000","last_became_peered":"2024-09-17T07:02:18.026994+0000","last_unstale":"2024-09-17T07:03:24.746161+0000","last_undegraded":"2024-09-17T07:03:24.746161+0000","last_fullsized":"2024-09-17T07:03:24.746161+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:02:17.002904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:43.693662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"75'15","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.536176+0000","last_change":"2024-09-17T07:02:14.028644+0000","last_active":"2024-09-17T07:03:20.536176+0000","last_peered":"2024-09-17T07:03:20.536176+0000","last_clean":"2024-09-17T07:03:20.536176+0000","last_became_active":"2024-09-17T07:02:14.028392+0000","last_became_peered":"2024-09-17T07:02:14.028392+0000","last_unstale":"2024-09-17T07:03:20.536176+0000","last_undegraded":"2024-09-17T07:03:20.536176+0000","last_fullsized":"2024-09-17T07:03:20.536176+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:02:12.994221+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:36.637466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'2","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-17T07:03:26.694387+0000","last_change":"2024-09-17T07:03:26.694471+0000","last_active":"2024-09-17T07:03:26.694387+0000","last_peered":"2024-09-17T07:03:26.694387+0000","last_clean":"2024-09-17T07:03:26.694387+0000","last_became_active":"2024-09-17T07:02:12.005538+0000","last_became_peered":"2024-09-17T07:02:12.005538+0000","last_unstale":"2024-09-17T07:03:26.694387+0000","last_undegraded":"2024-09-17T07:03:26.694387+0000","last_fullsized":"2024-09-17T07:03:26.694387+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-09-17T07:03:26.694366+0000","last_deep_scrub":"89'2","last_deep_scrub_stamp":"2024-09-17T07:03:26.694366+0000","last_clean_scrub_stamp":"2024-09-17T07:03:26.694366+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:38.099234+0000","scrub_duration":0.0026359439999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":420,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"75'8","reported_seq":54,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-17T07:03:25.533637+0000","last_change":"2024-09-17T07:02:16.015713+0000","last_active":"2024-09-17T07:03:25.533637+0000","last_peered":"2024-09-17T07:03:25.533637+0000","last_clean":"2024-09-17T07:03:25.533637+0000","last_became_active":"2024-09-17T07:02:16.015418+0000","last_became_peered":"2024-09-17T07:02:16.015418+0000","last_unstale":"2024-09-17T07:03:25.533637+0000","last_undegraded":"2024-09-17T07:03:25.533637+0000","last_fullsized":"2024-09-17T07:03:25.533637+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:02:14.999435+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:34.409640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":35,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:22.823018+0000","last_change":"2024-09-17T07:03:22.823018+0000","last_active":"2024-09-17T07:03:22.823018+0000","last_peered":"2024-09-17T07:03:22.823018+0000","last_clean":"2024-09-17T07:03:22.823018+0000","last_became_active":"2024-09-17T07:02:18.363253+0000","last_became_peered":"2024-09-17T07:02:18.363253+0000","last_unstale":"2024-09-17T07:03:22.823018+0000","last_undegraded":"2024-09-17T07:03:22.823018+0000","last_fullsized":"2024-09-17T07:03:22.823018+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:22.822922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:03:22.822922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:29.158808+0000","scrub_duration":0.0012979910000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"75'12","reported_seq":52,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.743919+0000","last_change":"2024-09-17T07:02:14.010593+0000","last_active":"2024-09-17T07:03:24.743919+0000","last_peered":"2024-09-17T07:03:24.743919+0000","last_clean":"2024-09-17T07:03:24.743919+0000","last_became_active":"2024-09-17T07:02:14.010219+0000","last_became_peered":"2024-09-17T07:02:14.010219+0000","last_unstale":"2024-09-17T07:03:24.743919+0000","last_undegraded":"2024-09-17T07:03:24.743919+0000","last_fullsized":"2024-09-17T07:03:24.743919+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:02:12.994221+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:25.821031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"68'2","reported_seq":49,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-17T07:03:28.021202+0000","last_change":"2024-09-17T07:03:28.021306+0000","last_active":"2024-09-17T07:03:28.021202+0000","last_peered":"2024-09-17T07:03:28.021202+0000","last_clean":"2024-09-17T07:03:28.021202+0000","last_became_active":"2024-09-17T07:02:12.006716+0000","last_became_peered":"2024-09-17T07:02:12.006716+0000","last_unstale":"2024-09-17T07:03:28.021202+0000","last_undegraded":"2024-09-17T07:03:28.021202+0000","last_fullsized":"2024-09-17T07:03:28.021202+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-09-17T07:03:28.021180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:28.021180+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:55.299746+0000","scrub_duration":0.0016672220000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.745108+0000","last_change":"2024-09-17T07:02:16.012559+0000","last_active":"2024-09-17T07:03:24.745108+0000","last_peered":"2024-09-17T07:03:24.745108+0000","last_clean":"2024-09-17T07:03:24.745108+0000","last_became_active":"2024-09-17T07:02:16.012261+0000","last_became_peered":"2024-09-17T07:02:16.012261+0000","last_unstale":"2024-09-17T07:03:24.745108+0000","last_undegraded":"2024-09-17T07:03:24.745108+0000","last_fullsized":"2024-09-17T07:03:24.745108+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:02:14.999435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:31.705121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.021388+0000","last_change":"2024-09-17T07:02:18.025590+0000","last_active":"2024-09-17T07:03:20.021388+0000","last_peered":"2024-09-17T07:03:20.021388+0000","last_clean":"2024-09-17T07:03:20.021388+0000","last_became_active":"2024-09-17T07:02:18.024983+0000","last_became_peered":"2024-09-17T07:02:18.024983+0000","last_unstale":"2024-09-17T07:03:20.021388+0000","last_undegraded":"2024-09-17T07:03:20.021388+0000","last_fullsized":"2024-09-17T07:03:20.021388+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:02:17.002904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:40.784492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"75'9","reported_seq":56,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-17T07:03:26.005217+0000","last_change":"2024-09-17T07:03:26.005290+0000","last_active":"2024-09-17T07:03:26.005217+0000","last_peered":"2024-09-17T07:03:26.005217+0000","last_clean":"2024-09-17T07:03:26.005217+0000","last_became_active":"2024-09-17T07:02:14.029388+0000","last_became_peered":"2024-09-17T07:02:14.029388+0000","last_unstale":"2024-09-17T07:03:26.005217+0000","last_undegraded":"2024-09-17T07:03:26.005217+0000","last_fullsized":"2024-09-17T07:03:26.005217+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:03:26.005201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:26.005201+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:55.043365+0000","scrub_duration":0.0018022279999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":42,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-17T07:03:31.753486+0000","last_change":"2024-09-17T07:03:31.753577+0000","last_active":"2024-09-17T07:03:31.753486+0000","last_peered":"2024-09-17T07:03:31.753486+0000","last_clean":"2024-09-17T07:03:31.753486+0000","last_became_active":"2024-09-17T07:02:12.011327+0000","last_became_peered":"2024-09-17T07:02:12.011327+0000","last_unstale":"2024-09-17T07:03:31.753486+0000","last_undegraded":"2024-09-17T07:03:31.753486+0000","last_fullsized":"2024-09-17T07:03:31.753486+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:31.753470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:31.753470+0000","last_clean_scrub_stamp":"2024-09-17T07:03:31.753470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:51.025434+0000","scrub_duration":0.0012687670000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":37,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:21.314310+0000","last_change":"2024-09-17T07:03:21.314310+0000","last_active":"2024-09-17T07:03:21.314310+0000","last_peered":"2024-09-17T07:03:21.314310+0000","last_clean":"2024-09-17T07:03:21.314310+0000","last_became_active":"2024-09-17T07:02:16.022584+0000","last_became_peered":"2024-09-17T07:02:16.022584+0000","last_unstale":"2024-09-17T07:03:21.314310+0000","last_undegraded":"2024-09-17T07:03:21.314310+0000","last_fullsized":"2024-09-17T07:03:21.314310+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:21.314226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:21.314226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:29.256168+0000","scrub_duration":0.0016627219999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":36,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.745467+0000","last_change":"2024-09-17T07:03:19.603974+0000","last_active":"2024-09-17T07:03:24.745467+0000","last_peered":"2024-09-17T07:03:24.745467+0000","last_clean":"2024-09-17T07:03:24.745467+0000","last_became_active":"2024-09-17T07:02:18.017825+0000","last_became_peered":"2024-09-17T07:02:18.017825+0000","last_unstale":"2024-09-17T07:03:24.745467+0000","last_undegraded":"2024-09-17T07:03:24.745467+0000","last_fullsized":"2024-09-17T07:03:24.745467+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:19.603899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:03:19.603899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:31.400546+0000","scrub_duration":0.001425798,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.115085+0000","last_change":"2024-09-17T07:02:12.008326+0000","last_active":"2024-09-17T07:03:20.115085+0000","last_peered":"2024-09-17T07:03:20.115085+0000","last_clean":"2024-09-17T07:03:20.115085+0000","last_became_active":"2024-09-17T07:02:12.008065+0000","last_became_peered":"2024-09-17T07:02:12.008065+0000","last_unstale":"2024-09-17T07:03:20.115085+0000","last_undegraded":"2024-09-17T07:03:20.115085+0000","last_fullsized":"2024-09-17T07:03:20.115085+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:02:10.991363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:40.680888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"75'6","reported_seq":37,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.021104+0000","last_change":"2024-09-17T07:02:14.029039+0000","last_active":"2024-09-17T07:03:20.021104+0000","last_peered":"2024-09-17T07:03:20.021104+0000","last_clean":"2024-09-17T07:03:20.021104+0000","last_became_active":"2024-09-17T07:02:14.028655+0000","last_became_peered":"2024-09-17T07:02:14.028655+0000","last_unstale":"2024-09-17T07:03:20.021104+0000","last_undegraded":"2024-09-17T07:03:20.021104+0000","last_fullsized":"2024-09-17T07:03:20.021104+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:02:12.994221+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.536952+0000","last_change":"2024-09-17T07:02:16.016899+0000","last_active":"2024-09-17T07:03:20.536952+0000","last_peered":"2024-09-17T07:03:20.536952+0000","last_clean":"2024-09-17T07:03:20.536952+0000","last_became_active":"2024-09-17T07:02:16.016634+0000","last_became_peered":"2024-09-17T07:02:16.016634+0000","last_unstale":"2024-09-17T07:03:20.536952+0000","last_undegraded":"2024-09-17T07:03:20.536952+0000","last_fullsized":"2024-09-17T07:03:20.536952+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:02:14.999435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:26.661198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.956095+0000","last_change":"2024-09-17T07:02:18.365703+0000","last_active":"2024-09-17T07:03:24.956095+0000","last_peered":"2024-09-17T07:03:24.956095+0000","last_clean":"2024-09-17T07:03:24.956095+0000","last_became_active":"2024-09-17T07:02:18.365419+0000","last_became_peered":"2024-09-17T07:02:18.365419+0000","last_unstale":"2024-09-17T07:03:24.956095+0000","last_undegraded":"2024-09-17T07:03:24.956095+0000","last_fullsized":"2024-09-17T07:03:24.956095+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:02:17.002904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:28.424716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"75'10","reported_seq":54,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.021872+0000","last_change":"2024-09-17T07:03:14.178469+0000","last_active":"2024-09-17T07:03:20.021872+0000","last_peered":"2024-09-17T07:03:20.021872+0000","last_clean":"2024-09-17T07:03:20.021872+0000","last_became_active":"2024-09-17T07:02:14.027499+0000","last_became_peered":"2024-09-17T07:02:14.027499+0000","last_unstale":"2024-09-17T07:03:20.021872+0000","last_undegraded":"2024-09-17T07:03:20.021872+0000","last_fullsized":"2024-09-17T07:03:20.021872+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-09-17T07:03:14.178375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:14.178375+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:31.400568+0000","scrub_duration":0.0021187430000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":41,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-17T07:03:25.849723+0000","last_change":"2024-09-17T07:03:25.849841+0000","last_active":"2024-09-17T07:03:25.849723+0000","last_peered":"2024-09-17T07:03:25.849723+0000","last_clean":"2024-09-17T07:03:25.849723+0000","last_became_active":"2024-09-17T07:02:12.007280+0000","last_became_peered":"2024-09-17T07:02:12.007280+0000","last_unstale":"2024-09-17T07:03:25.849723+0000","last_undegraded":"2024-09-17T07:03:25.849723+0000","last_fullsized":"2024-09-17T07:03:25.849723+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:25.849698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:25.849698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:55.197183+0000","scrub_duration":0.0014293929999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"75'8","reported_seq":50,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-17T07:03:25.533875+0000","last_change":"2024-09-17T07:02:16.022883+0000","last_active":"2024-09-17T07:03:25.533875+0000","last_peered":"2024-09-17T07:03:25.533875+0000","last_clean":"2024-09-17T07:03:25.533875+0000","last_became_active":"2024-09-17T07:02:16.022583+0000","last_became_peered":"2024-09-17T07:02:16.022583+0000","last_unstale":"2024-09-17T07:03:25.533875+0000","last_undegraded":"2024-09-17T07:03:25.533875+0000","last_fullsized":"2024-09-17T07:03:25.533875+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:02:14.999435+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:33.977787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.746352+0000","last_change":"2024-09-17T07:02:18.029691+0000","last_active":"2024-09-17T07:03:24.746352+0000","last_peered":"2024-09-17T07:03:24.746352+0000","last_clean":"2024-09-17T07:03:24.746352+0000","last_became_active":"2024-09-17T07:02:18.028796+0000","last_became_peered":"2024-09-17T07:02:18.028796+0000","last_unstale":"2024-09-17T07:03:24.746352+0000","last_undegraded":"2024-09-17T07:03:24.746352+0000","last_fullsized":"2024-09-17T07:03:24.746352+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:02:17.002904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:40.761948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"75'9","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.536820+0000","last_change":"2024-09-17T07:03:17.798712+0000","last_active":"2024-09-17T07:03:20.536820+0000","last_peered":"2024-09-17T07:03:20.536820+0000","last_clean":"2024-09-17T07:03:20.536820+0000","last_became_active":"2024-09-17T07:02:14.028485+0000","last_became_peered":"2024-09-17T07:02:14.028485+0000","last_unstale":"2024-09-17T07:03:20.536820+0000","last_undegraded":"2024-09-17T07:03:20.536820+0000","last_fullsized":"2024-09-17T07:03:20.536820+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:03:17.798582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:17.798582+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:27.282186+0000","scrub_duration":0.0023213259999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":42,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.746635+0000","last_change":"2024-09-17T07:03:16.948419+0000","last_active":"2024-09-17T07:03:24.746635+0000","last_peered":"2024-09-17T07:03:24.746635+0000","last_clean":"2024-09-17T07:03:24.746635+0000","last_became_active":"2024-09-17T07:02:12.009935+0000","last_became_peered":"2024-09-17T07:02:12.009935+0000","last_unstale":"2024-09-17T07:03:24.746635+0000","last_undegraded":"2024-09-17T07:03:24.746635+0000","last_fullsized":"2024-09-17T07:03:24.746635+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:16.948330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:16.948330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:46.609922+0000","scrub_duration":0.0017835489999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"75'8","reported_seq":50,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-17T07:03:25.533642+0000","last_change":"2024-09-17T07:02:16.016395+0000","last_active":"2024-09-17T07:03:25.533642+0000","last_peered":"2024-09-17T07:03:25.533642+0000","last_clean":"2024-09-17T07:03:25.533642+0000","last_became_active":"2024-09-17T07:02:16.016069+0000","last_became_peered":"2024-09-17T07:02:16.016069+0000","last_unstale":"2024-09-17T07:03:25.533642+0000","last_undegraded":"2024-09-17T07:03:25.533642+0000","last_fullsized":"2024-09-17T07:03:25.533642+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:02:14.999435+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":36,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-17T07:03:31.045183+0000","last_change":"2024-09-17T07:03:31.045257+0000","last_active":"2024-09-17T07:03:31.045183+0000","last_peered":"2024-09-17T07:03:31.045183+0000","last_clean":"2024-09-17T07:03:31.045183+0000","last_became_active":"2024-09-17T07:02:18.018716+0000","last_became_peered":"2024-09-17T07:02:18.018716+0000","last_unstale":"2024-09-17T07:03:31.045183+0000","last_undegraded":"2024-09-17T07:03:31.045183+0000","last_fullsized":"2024-09-17T07:03:31.045183+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:31.045167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:03:31.045167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:36.218836+0000","scrub_duration":0.0012599709999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"75'9","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.116712+0000","last_change":"2024-09-17T07:03:20.116712+0000","last_active":"2024-09-17T07:03:20.116712+0000","last_peered":"2024-09-17T07:03:20.116712+0000","last_clean":"2024-09-17T07:03:20.116712+0000","last_became_active":"2024-09-17T07:02:14.010289+0000","last_became_peered":"2024-09-17T07:02:14.010289+0000","last_unstale":"2024-09-17T07:03:20.116712+0000","last_undegraded":"2024-09-17T07:03:20.116712+0000","last_fullsized":"2024-09-17T07:03:20.116712+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:03:20.116602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:20.116602+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:35.267610+0000","scrub_duration":0.0032469130000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.115284+0000","last_change":"2024-09-17T07:02:12.007881+0000","last_active":"2024-09-17T07:03:20.115284+0000","last_peered":"2024-09-17T07:03:20.115284+0000","last_clean":"2024-09-17T07:03:20.115284+0000","last_became_active":"2024-09-17T07:02:12.007433+0000","last_became_peered":"2024-09-17T07:02:12.007433+0000","last_unstale":"2024-09-17T07:03:20.115284+0000","last_undegraded":"2024-09-17T07:03:20.115284+0000","last_fullsized":"2024-09-17T07:03:20.115284+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:02:10.991363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:36.966808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.021413+0000","last_change":"2024-09-17T07:02:16.016504+0000","last_active":"2024-09-17T07:03:20.021413+0000","last_peered":"2024-09-17T07:03:20.021413+0000","last_clean":"2024-09-17T07:03:20.021413+0000","last_became_active":"2024-09-17T07:02:16.016216+0000","last_became_peered":"2024-09-17T07:02:16.016216+0000","last_unstale":"2024-09-17T07:03:20.021413+0000","last_undegraded":"2024-09-17T07:03:20.021413+0000","last_fullsized":"2024-09-17T07:03:20.021413+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:02:14.999435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:37.372655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":36,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.745931+0000","last_change":"2024-09-17T07:03:24.014436+0000","last_active":"2024-09-17T07:03:24.745931+0000","last_peered":"2024-09-17T07:03:24.745931+0000","last_clean":"2024-09-17T07:03:24.745931+0000","last_became_active":"2024-09-17T07:02:18.027622+0000","last_became_peered":"2024-09-17T07:02:18.027622+0000","last_unstale":"2024-09-17T07:03:24.745931+0000","last_undegraded":"2024-09-17T07:03:24.745931+0000","last_fullsized":"2024-09-17T07:03:24.745931+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:24.014336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:03:24.014336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:50.998219+0000","scrub_duration":0.001425125,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"75'11","reported_seq":59,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.744332+0000","last_change":"2024-09-17T07:03:18.899916+0000","last_active":"2024-09-17T07:03:24.744332+0000","last_peered":"2024-09-17T07:03:24.744332+0000","last_clean":"2024-09-17T07:03:24.744332+0000","last_became_active":"2024-09-17T07:02:14.016255+0000","last_became_peered":"2024-09-17T07:02:14.016255+0000","last_unstale":"2024-09-17T07:03:24.744332+0000","last_undegraded":"2024-09-17T07:03:24.744332+0000","last_fullsized":"2024-09-17T07:03:24.744332+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-09-17T07:03:18.899814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:18.899814+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:46.161729+0000","scrub_duration":0.0020491419999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"68'1","reported_seq":35,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.021771+0000","last_change":"2024-09-17T07:02:12.011847+0000","last_active":"2024-09-17T07:03:20.021771+0000","last_peered":"2024-09-17T07:03:20.021771+0000","last_clean":"2024-09-17T07:03:20.021771+0000","last_became_active":"2024-09-17T07:02:12.011461+0000","last_became_peered":"2024-09-17T07:02:12.011461+0000","last_unstale":"2024-09-17T07:03:20.021771+0000","last_undegraded":"2024-09-17T07:03:20.021771+0000","last_fullsized":"2024-09-17T07:03:20.021771+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:02:10.991363+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:37.333028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.020792+0000","last_change":"2024-09-17T07:02:16.019051+0000","last_active":"2024-09-17T07:03:20.020792+0000","last_peered":"2024-09-17T07:03:20.020792+0000","last_clean":"2024-09-17T07:03:20.020792+0000","last_became_active":"2024-09-17T07:02:16.018754+0000","last_became_peered":"2024-09-17T07:02:16.018754+0000","last_unstale":"2024-09-17T07:03:20.020792+0000","last_undegraded":"2024-09-17T07:03:20.020792+0000","last_fullsized":"2024-09-17T07:03:20.020792+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:02:14.999435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.020820+0000","last_change":"2024-09-17T07:02:18.027727+0000","last_active":"2024-09-17T07:03:20.020820+0000","last_peered":"2024-09-17T07:03:20.020820+0000","last_clean":"2024-09-17T07:03:20.020820+0000","last_became_active":"2024-09-17T07:02:18.027427+0000","last_became_peered":"2024-09-17T07:02:18.027427+0000","last_unstale":"2024-09-17T07:03:20.020820+0000","last_undegraded":"2024-09-17T07:03:20.020820+0000","last_fullsized":"2024-09-17T07:03:20.020820+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:02:17.002904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:41.608211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"75'4","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.431130+0000","last_change":"2024-09-17T07:03:24.431130+0000","last_active":"2024-09-17T07:03:24.431130+0000","last_peered":"2024-09-17T07:03:24.431130+0000","last_clean":"2024-09-17T07:03:24.431130+0000","last_became_active":"2024-09-17T07:02:14.014779+0000","last_became_peered":"2024-09-17T07:02:14.014779+0000","last_unstale":"2024-09-17T07:03:24.431130+0000","last_undegraded":"2024-09-17T07:03:24.431130+0000","last_fullsized":"2024-09-17T07:03:24.431130+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'4","last_scrub_stamp":"2024-09-17T07:03:24.431043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:24.431043+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:37.473735+0000","scrub_duration":0.0015094500000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.537004+0000","last_change":"2024-09-17T07:02:12.009186+0000","last_active":"2024-09-17T07:03:20.537004+0000","last_peered":"2024-09-17T07:03:20.537004+0000","last_clean":"2024-09-17T07:03:20.537004+0000","last_became_active":"2024-09-17T07:02:12.008898+0000","last_became_peered":"2024-09-17T07:02:12.008898+0000","last_unstale":"2024-09-17T07:03:20.537004+0000","last_undegraded":"2024-09-17T07:03:20.537004+0000","last_fullsized":"2024-09-17T07:03:20.537004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:02:10.991363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:30.703927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.114909+0000","last_change":"2024-09-17T07:02:16.015248+0000","last_active":"2024-09-17T07:03:20.114909+0000","last_peered":"2024-09-17T07:03:20.114909+0000","last_clean":"2024-09-17T07:03:20.114909+0000","last_became_active":"2024-09-17T07:02:16.014982+0000","last_became_peered":"2024-09-17T07:02:16.014982+0000","last_unstale":"2024-09-17T07:03:20.114909+0000","last_undegraded":"2024-09-17T07:03:20.114909+0000","last_fullsized":"2024-09-17T07:03:20.114909+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:02:14.999435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:34.693466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.955631+0000","last_change":"2024-09-17T07:02:18.021319+0000","last_active":"2024-09-17T07:03:24.955631+0000","last_peered":"2024-09-17T07:03:24.955631+0000","last_clean":"2024-09-17T07:03:24.955631+0000","last_became_active":"2024-09-17T07:02:18.021014+0000","last_became_peered":"2024-09-17T07:02:18.021014+0000","last_unstale":"2024-09-17T07:03:24.955631+0000","last_undegraded":"2024-09-17T07:03:24.955631+0000","last_fullsized":"2024-09-17T07:03:24.955631+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:02:17.002904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:40.046947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"75'11","reported_seq":58,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.021003+0000","last_change":"2024-09-17T07:03:17.270057+0000","last_active":"2024-09-17T07:03:20.021003+0000","last_peered":"2024-09-17T07:03:20.021003+0000","last_clean":"2024-09-17T07:03:20.021003+0000","last_became_active":"2024-09-17T07:02:14.027316+0000","last_became_peered":"2024-09-17T07:02:14.027316+0000","last_unstale":"2024-09-17T07:03:20.021003+0000","last_undegraded":"2024-09-17T07:03:20.021003+0000","last_fullsized":"2024-09-17T07:03:20.021003+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-09-17T07:03:17.269941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:17.269941+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:32.856982+0000","scrub_duration":0.002326505,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.115512+0000","last_change":"2024-09-17T07:02:12.010482+0000","last_active":"2024-09-17T07:03:20.115512+0000","last_peered":"2024-09-17T07:03:20.115512+0000","last_clean":"2024-09-17T07:03:20.115512+0000","last_became_active":"2024-09-17T07:02:12.010195+0000","last_became_peered":"2024-09-17T07:02:12.010195+0000","last_unstale":"2024-09-17T07:03:20.115512+0000","last_undegraded":"2024-09-17T07:03:20.115512+0000","last_fullsized":"2024-09-17T07:03:20.115512+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:02:10.991363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:27.744542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.955436+0000","last_change":"2024-09-17T07:02:16.016468+0000","last_active":"2024-09-17T07:03:24.955436+0000","last_peered":"2024-09-17T07:03:24.955436+0000","last_clean":"2024-09-17T07:03:24.955436+0000","last_became_active":"2024-09-17T07:02:16.016183+0000","last_became_peered":"2024-09-17T07:02:16.016183+0000","last_unstale":"2024-09-17T07:03:24.955436+0000","last_undegraded":"2024-09-17T07:03:24.955436+0000","last_fullsized":"2024-09-17T07:03:24.955436+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:02:14.999435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:27.036527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":35,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-17T07:03:29.467266+0000","last_change":"2024-09-17T07:03:29.467351+0000","last_active":"2024-09-17T07:03:29.467266+0000","last_peered":"2024-09-17T07:03:29.467266+0000","last_clean":"2024-09-17T07:03:29.467266+0000","last_became_active":"2024-09-17T07:02:18.365656+0000","last_became_peered":"2024-09-17T07:02:18.365656+0000","last_unstale":"2024-09-17T07:03:29.467266+0000","last_undegraded":"2024-09-17T07:03:29.467266+0000","last_fullsized":"2024-09-17T07:03:29.467266+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:29.467252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:03:29.467252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:58.741564+0000","scrub_duration":0.001842804,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"75'1","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.956029+0000","last_change":"2024-09-17T07:02:18.022137+0000","last_active":"2024-09-17T07:03:24.956029+0000","last_peered":"2024-09-17T07:03:24.956029+0000","last_clean":"2024-09-17T07:03:24.956029+0000","last_became_active":"2024-09-17T07:02:18.021855+0000","last_became_peered":"2024-09-17T07:02:18.021855+0000","last_unstale":"2024-09-17T07:03:24.956029+0000","last_undegraded":"2024-09-17T07:03:24.956029+0000","last_fullsized":"2024-09-17T07:03:24.956029+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:02:17.002904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:39.620232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.536263+0000","last_change":"2024-09-17T07:03:19.775506+0000","last_active":"2024-09-17T07:03:20.536263+0000","last_peered":"2024-09-17T07:03:20.536263+0000","last_clean":"2024-09-17T07:03:20.536263+0000","last_became_active":"2024-09-17T07:02:12.005329+0000","last_became_peered":"2024-09-17T07:02:12.005329+0000","last_unstale":"2024-09-17T07:03:20.536263+0000","last_undegraded":"2024-09-17T07:03:20.536263+0000","last_fullsized":"2024-09-17T07:03:20.536263+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:19.775390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:19.775390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:48.303892+0000","scrub_duration":0.0011915630000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"75'10","reported_seq":44,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.746440+0000","last_change":"2024-09-17T07:02:14.029020+0000","last_active":"2024-09-17T07:03:24.746440+0000","last_peered":"2024-09-17T07:03:24.746440+0000","last_clean":"2024-09-17T07:03:24.746440+0000","last_became_active":"2024-09-17T07:02:14.028731+0000","last_became_peered":"2024-09-17T07:02:14.028731+0000","last_unstale":"2024-09-17T07:03:24.746440+0000","last_undegraded":"2024-09-17T07:03:24.746440+0000","last_fullsized":"2024-09-17T07:03:24.746440+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:02:12.994221+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:42.804791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"75'8","reported_seq":53,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-17T07:03:25.533620+0000","last_change":"2024-09-17T07:02:16.020978+0000","last_active":"2024-09-17T07:03:25.533620+0000","last_peered":"2024-09-17T07:03:25.533620+0000","last_clean":"2024-09-17T07:03:25.533620+0000","last_became_active":"2024-09-17T07:02:16.020560+0000","last_became_peered":"2024-09-17T07:02:16.020560+0000","last_unstale":"2024-09-17T07:03:25.533620+0000","last_undegraded":"2024-09-17T07:03:25.533620+0000","last_fullsized":"2024-09-17T07:03:25.533620+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:02:14.999435+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:42.145816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.021299+0000","last_change":"2024-09-17T07:02:18.021959+0000","last_active":"2024-09-17T07:03:20.021299+0000","last_peered":"2024-09-17T07:03:20.021299+0000","last_clean":"2024-09-17T07:03:20.021299+0000","last_became_active":"2024-09-17T07:02:18.021603+0000","last_became_peered":"2024-09-17T07:02:18.021603+0000","last_unstale":"2024-09-17T07:03:20.021299+0000","last_undegraded":"2024-09-17T07:03:20.021299+0000","last_fullsized":"2024-09-17T07:03:20.021299+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:02:17.002904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:33.551380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.305248+0000","last_change":"2024-09-17T07:03:20.305248+0000","last_active":"2024-09-17T07:03:20.305248+0000","last_peered":"2024-09-17T07:03:20.305248+0000","last_clean":"2024-09-17T07:03:20.305248+0000","last_became_active":"2024-09-17T07:02:12.009837+0000","last_became_peered":"2024-09-17T07:02:12.009837+0000","last_unstale":"2024-09-17T07:03:20.305248+0000","last_undegraded":"2024-09-17T07:03:20.305248+0000","last_fullsized":"2024-09-17T07:03:20.305248+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:20.305165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:20.305165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:24.412968+0000","scrub_duration":0.0024485969999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"75'11","reported_seq":58,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.473820+0000","last_change":"2024-09-17T07:03:24.473820+0000","last_active":"2024-09-17T07:03:24.473820+0000","last_peered":"2024-09-17T07:03:24.473820+0000","last_clean":"2024-09-17T07:03:24.473820+0000","last_became_active":"2024-09-17T07:02:14.011380+0000","last_became_peered":"2024-09-17T07:02:14.011380+0000","last_unstale":"2024-09-17T07:03:24.473820+0000","last_undegraded":"2024-09-17T07:03:24.473820+0000","last_fullsized":"2024-09-17T07:03:24.473820+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-09-17T07:03:24.473712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:24.473712+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:53.121209+0000","scrub_duration":0.0021033010000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.746403+0000","last_change":"2024-09-17T07:02:16.019989+0000","last_active":"2024-09-17T07:03:24.746403+0000","last_peered":"2024-09-17T07:03:24.746403+0000","last_clean":"2024-09-17T07:03:24.746403+0000","last_became_active":"2024-09-17T07:02:16.019701+0000","last_became_peered":"2024-09-17T07:02:16.019701+0000","last_unstale":"2024-09-17T07:03:24.746403+0000","last_undegraded":"2024-09-17T07:03:24.746403+0000","last_fullsized":"2024-09-17T07:03:24.746403+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:02:14.999435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:44.035860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.744767+0000","last_change":"2024-09-17T07:02:18.364013+0000","last_active":"2024-09-17T07:03:24.744767+0000","last_peered":"2024-09-17T07:03:24.744767+0000","last_clean":"2024-09-17T07:03:24.744767+0000","last_became_active":"2024-09-17T07:02:18.363752+0000","last_became_peered":"2024-09-17T07:02:18.363752+0000","last_unstale":"2024-09-17T07:03:24.744767+0000","last_undegraded":"2024-09-17T07:03:24.744767+0000","last_fullsized":"2024-09-17T07:03:24.744767+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:02:17.002904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:33.097191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.115297+0000","last_change":"2024-09-17T07:02:12.005383+0000","last_active":"2024-09-17T07:03:20.115297+0000","last_peered":"2024-09-17T07:03:20.115297+0000","last_clean":"2024-09-17T07:03:20.115297+0000","last_became_active":"2024-09-17T07:02:12.005149+0000","last_became_peered":"2024-09-17T07:02:12.005149+0000","last_unstale":"2024-09-17T07:03:20.115297+0000","last_undegraded":"2024-09-17T07:03:20.115297+0000","last_fullsized":"2024-09-17T07:03:20.115297+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:02:10.991363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:31.958573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"75'15","reported_seq":65,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.745594+0000","last_change":"2024-09-17T07:03:16.605256+0000","last_active":"2024-09-17T07:03:24.745594+0000","last_peered":"2024-09-17T07:03:24.745594+0000","last_clean":"2024-09-17T07:03:24.745594+0000","last_became_active":"2024-09-17T07:02:14.016273+0000","last_became_peered":"2024-09-17T07:02:14.016273+0000","last_unstale":"2024-09-17T07:03:24.745594+0000","last_undegraded":"2024-09-17T07:03:24.745594+0000","last_fullsized":"2024-09-17T07:03:24.745594+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-09-17T07:03:16.605163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:16.605163+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:40.470486+0000","scrub_duration":0.002000763,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.021640+0000","last_change":"2024-09-17T07:02:16.017339+0000","last_active":"2024-09-17T07:03:20.021640+0000","last_peered":"2024-09-17T07:03:20.021640+0000","last_clean":"2024-09-17T07:03:20.021640+0000","last_became_active":"2024-09-17T07:02:16.015903+0000","last_became_peered":"2024-09-17T07:02:16.015903+0000","last_unstale":"2024-09-17T07:03:20.021640+0000","last_undegraded":"2024-09-17T07:03:20.021640+0000","last_fullsized":"2024-09-17T07:03:20.021640+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:02:14.999435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.020900+0000","last_change":"2024-09-17T07:02:18.366797+0000","last_active":"2024-09-17T07:03:20.020900+0000","last_peered":"2024-09-17T07:03:20.020900+0000","last_clean":"2024-09-17T07:03:20.020900+0000","last_became_active":"2024-09-17T07:02:18.366466+0000","last_became_peered":"2024-09-17T07:02:18.366466+0000","last_unstale":"2024-09-17T07:03:20.020900+0000","last_undegraded":"2024-09-17T07:03:20.020900+0000","last_fullsized":"2024-09-17T07:03:20.020900+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:02:17.002904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:46.082346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.021521+0000","last_change":"2024-09-17T07:02:12.005764+0000","last_active":"2024-09-17T07:03:20.021521+0000","last_peered":"2024-09-17T07:03:20.021521+0000","last_clean":"2024-09-17T07:03:20.021521+0000","last_became_active":"2024-09-17T07:02:12.005493+0000","last_became_peered":"2024-09-17T07:02:12.005493+0000","last_unstale":"2024-09-17T07:03:20.021521+0000","last_undegraded":"2024-09-17T07:03:20.021521+0000","last_fullsized":"2024-09-17T07:03:20.021521+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:02:10.991363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:35.124183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"86'17","reported_seq":67,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.021814+0000","last_change":"2024-09-17T07:03:18.241178+0000","last_active":"2024-09-17T07:03:20.021814+0000","last_peered":"2024-09-17T07:03:20.021814+0000","last_clean":"2024-09-17T07:03:20.021814+0000","last_became_active":"2024-09-17T07:02:14.017737+0000","last_became_peered":"2024-09-17T07:02:14.017737+0000","last_unstale":"2024-09-17T07:03:20.021814+0000","last_undegraded":"2024-09-17T07:03:20.021814+0000","last_fullsized":"2024-09-17T07:03:20.021814+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'17","last_scrub_stamp":"2024-09-17T07:03:18.241067+0000","last_deep_scrub":"86'17","last_deep_scrub_stamp":"2024-09-17T07:03:18.241067+0000","last_clean_scrub_stamp":"2024-09-17T07:03:18.241067+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:35.855446+0000","scrub_duration":0.0047694219999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-17T07:03:26.978275+0000","last_change":"2024-09-17T07:03:26.978352+0000","last_active":"2024-09-17T07:03:26.978275+0000","last_peered":"2024-09-17T07:03:26.978275+0000","last_clean":"2024-09-17T07:03:26.978275+0000","last_became_active":"2024-09-17T07:02:16.017362+0000","last_became_peered":"2024-09-17T07:02:16.017362+0000","last_unstale":"2024-09-17T07:03:26.978275+0000","last_undegraded":"2024-09-17T07:03:26.978275+0000","last_fullsized":"2024-09-17T07:03:26.978275+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:26.978262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:26.978262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:38.363709+0000","scrub_duration":0.0016151990000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":396,"ondisk_log_size":396,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":74,"num_read_kb":69,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1386,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":12,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":4158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037782,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32456,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93712696,"statfs":{"total":95995035648,"available":95961800704,"internally_reserved":0,"allocated":1376256,"data_stored":936056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:03:02 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.51700000000000002}]},{"osd":1,"last update":"Tue Sep 17 07:03:02 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.435}]},{"osd":2,"last update":"Tue Sep 17 07:03:02 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.622,"5min":0.622,"15min":0.622},"last":0.38},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.39200000000000002}]},{"osd":3,"last update":"Tue Sep 17 07:03:02 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.42199999999999999}]},{"osd":4,"last update":"Tue Sep 17 07:03:02 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"last":0.46400000000000002}]},{"osd":5,"last update":"Tue Sep 17 07:03:02 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.58199999999999996}]},{"osd":6,"last update":"Tue Sep 17 07:03:02 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.501}]}]},{"osd":6,"up_from":50,"seq":214748364825,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32440,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93712712,"statfs":{"total":95995035648,"available":95961817088,"internally_reserved":0,"allocated":1359872,"data_stored":935371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:02:41 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.55300000000000005}]},{"osd":1,"last update":"Tue Sep 17 07:02:41 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.54200000000000004}]},{"osd":2,"last update":"Tue Sep 17 07:02:41 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.51500000000000001}]},{"osd":3,"last update":"Tue Sep 17 07:02:41 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.497},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.48499999999999999}]},{"osd":4,"last update":"Tue Sep 17 07:02:41 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.52300000000000002}]},{"osd":5,"last update":"Tue Sep 17 07:02:41 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"last":0.439}]},{"osd":7,"last update":"Tue Sep 17 07:03:03 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.37},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.46000000000000002}]}]},{"osd":5,"up_from":42,"seq":180388626462,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31864,"kb_used_data":752,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713288,"statfs":{"total":95995035648,"available":95962406912,"internally_reserved":0,"allocated":770048,"data_stored":345813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:03:15 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.50600000000000001,"15min":0.51500000000000001},"min":{"1min":0.34100000000000003,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.56699999999999995,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.497,"15min":0.505},"min":{"1min":0.31900000000000001,"5min":0.313,"15min":0.313},"max":{"1min":0.65500000000000003,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.26700000000000002}]},{"osd":1,"last update":"Tue Sep 17 07:03:15 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.53200000000000003,"15min":0.54200000000000004},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.60799999999999998,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.443,"5min":1.0449999999999999,"15min":1.145},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.56200000000000006,"5min":14.054,"15min":14.054},"last":0.29599999999999999}]},{"osd":2,"last update":"Tue Sep 17 07:03:15 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.54800000000000004,"15min":0.56000000000000005},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.66500000000000004,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.505,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.38600000000000001,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.63600000000000001,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.45900000000000002}]},{"osd":3,"last update":"Tue Sep 17 07:03:15 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.54000000000000004,"15min":0.54900000000000004},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.63800000000000001,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.496,"5min":0.53900000000000003,"15min":0.54600000000000004},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.66400000000000003,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.36599999999999999}]},{"osd":4,"last update":"Tue Sep 17 07:03:15 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.40000000000000002,"15min":0.39900000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.432,"15min":0.434},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.60499999999999998,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.44500000000000001}]},{"osd":6,"last update":"Tue Sep 17 07:02:37 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.40699999999999997}]},{"osd":7,"last update":"Tue Sep 17 07:03:01 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.436}]}]},{"osd":4,"up_from":33,"seq":141733920804,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31880,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713272,"statfs":{"total":95995035648,"available":95962390528,"internally_reserved":0,"allocated":786432,"data_stored":345400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:02:53 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.56299999999999994,"15min":0.57499999999999996},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.71499999999999997,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.60199999999999998,"15min":0.61499999999999999},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.69799999999999995,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.52900000000000003}]},{"osd":1,"last update":"Tue Sep 17 07:02:53 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.56000000000000005,"15min":0.57199999999999995},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.68400000000000005,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.495,"5min":0.57999999999999996,"15min":0.59399999999999997},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.67900000000000005,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.47699999999999998}]},{"osd":2,"last update":"Tue Sep 17 07:02:53 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.56899999999999995,"15min":0.58299999999999996},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.69999999999999996,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.57799999999999996,"15min":0.58799999999999997},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.77100000000000002,"5min":0.877,"15min":0.877},"last":0.51000000000000001}]},{"osd":3,"last update":"Tue Sep 17 07:02:53 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.59299999999999997,"15min":0.60799999999999998},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.74299999999999999,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.55900000000000005,"15min":0.56399999999999995},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.66800000000000004,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.497}]},{"osd":5,"last update":"Tue Sep 17 07:03:19 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.39600000000000002,"15min":0.38600000000000001},"min":{"1min":0.316,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.41699999999999998,"15min":0.40699999999999997},"min":{"1min":0.34799999999999998,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.40400000000000003}]},{"osd":6,"last update":"Tue Sep 17 07:02:37 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.372}]},{"osd":7,"last update":"Tue Sep 17 07:03:00 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.443},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.41799999999999998}]}]},{"osd":3,"up_from":27,"seq":115964117030,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31904,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713248,"statfs":{"total":95995035648,"available":95962365952,"internally_reserved":0,"allocated":811008,"data_stored":346184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:02:39 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.36499999999999999,"15min":0.36399999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.38100000000000001,"15min":0.379},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.56599999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"last":0.61399999999999999}]},{"osd":1,"last update":"Tue Sep 17 07:02:39 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.40400000000000003,"15min":0.40699999999999997},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.58299999999999996,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.442,"5min":0.45000000000000001,"15min":0.45100000000000001},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.63700000000000001,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.53800000000000003}]},{"osd":2,"last update":"Tue Sep 17 07:02:39 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.42099999999999999,"15min":0.42299999999999999},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.41899999999999998,"15min":0.41999999999999998},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.33000000000000002}]},{"osd":4,"last update":"Tue Sep 17 07:02:56 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.56000000000000005,"15min":0.56399999999999995},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.754,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.53100000000000003,"15min":0.53500000000000003},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.66100000000000003}]},{"osd":5,"last update":"Tue Sep 17 07:03:18 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.59799999999999998,"15min":0.60599999999999998},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.58299999999999996,"15min":0.58499999999999996},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.79100000000000004,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.56299999999999994}]},{"osd":6,"last update":"Tue Sep 17 07:02:39 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.66800000000000004}]},{"osd":7,"last update":"Tue Sep 17 07:03:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.629}]}]},{"osd":2,"up_from":18,"seq":77309411372,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31868,"kb_used_data":756,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713284,"statfs":{"total":95995035648,"available":95962402816,"internally_reserved":0,"allocated":774144,"data_stored":345823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:03:15 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.35499999999999998,"15min":0.34699999999999998},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.35899999999999999,"15min":0.35099999999999998},"min":{"1min":0.24399999999999999,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.50600000000000001,"5min":0.625,"15min":0.625},"last":0.28100000000000003}]},{"osd":1,"last update":"Tue Sep 17 07:03:15 2024","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.35499999999999998,"15min":0.35099999999999998},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.48999999999999999,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.38500000000000001,"15min":0.374},"min":{"1min":0.28599999999999998,"5min":0.251,"15min":0.251},"max":{"1min":0.55700000000000005,"5min":0.58399999999999996,"15min":0.58399999999999996},"last":0.44800000000000001}]},{"osd":3,"last update":"Tue Sep 17 07:02:35 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.44,"15min":0.438},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.433,"5min":0.42999999999999999,"15min":0.42999999999999999},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.38600000000000001}]},{"osd":4,"last update":"Tue Sep 17 07:02:52 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56499999999999995,"15min":0.56599999999999995},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58599999999999997,"15min":0.58699999999999997},"min":{"1min":0.42999999999999999,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.69699999999999995}]},{"osd":5,"last update":"Tue Sep 17 07:03:15 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.101,"5min":1.101,"15min":1.101},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.57799999999999996,"15min":0.58999999999999997},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.78200000000000003,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.68100000000000005}]},{"osd":6,"last update":"Tue Sep 17 07:02:39 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.61299999999999999}]},{"osd":7,"last update":"Tue Sep 17 07:03:03 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":3.9289999999999998,"5min":3.9289999999999998,"15min":3.9289999999999998},"last":0.67000000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574897,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27820,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717332,"statfs":{"total":95995035648,"available":95966547968,"internally_reserved":0,"allocated":823296,"data_stored":347435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:02:43 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.34699999999999998,"15min":0.33500000000000002},"min":{"1min":0.19400000000000001,"5min":0.193,"15min":0.193},"max":{"1min":0.47399999999999998,"5min":0.56000000000000005,"15min":0.56000000000000005},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.373,"15min":0.36299999999999999},"min":{"1min":0.23400000000000001,"5min":0.216,"15min":0.216},"max":{"1min":0.51500000000000001,"5min":0.622,"15min":0.622},"last":0.38800000000000001}]},{"osd":2,"last update":"Tue Sep 17 07:03:12 2024","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.39600000000000002,"15min":0.40100000000000002},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.53100000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.40600000000000003,"15min":0.41199999999999998},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.60199999999999998,"5min":0.625,"15min":0.625},"last":0.44600000000000001}]},{"osd":3,"last update":"Tue Sep 17 07:02:34 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.45600000000000002,"15min":0.46300000000000002},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.58899999999999997,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.436,"15min":0.44500000000000001},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.505,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.372}]},{"osd":4,"last update":"Tue Sep 17 07:02:54 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.57699999999999996,"15min":0.57399999999999995},"min":{"1min":0.41199999999999998,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.56999999999999995,"15min":0.56799999999999995},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.71799999999999997,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.53800000000000003}]},{"osd":5,"last update":"Tue Sep 17 07:03:19 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.621,"15min":0.58699999999999997},"min":{"1min":0.38700000000000001,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":6.0060000000000002,"5min":6.0060000000000002,"15min":6.0060000000000002},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.56299999999999994,"15min":0.56499999999999995},"min":{"1min":0.38200000000000001,"5min":0.375,"15min":0.375},"max":{"1min":0.752,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.61599999999999999}]},{"osd":6,"last update":"Tue Sep 17 07:02:40 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.63700000000000001}]},{"osd":7,"last update":"Tue Sep 17 07:03:01 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.47999999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738420,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28368,"kb_used_data":1352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716784,"statfs":{"total":95995035648,"available":95965986816,"internally_reserved":0,"allocated":1384448,"data_stored":936501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 17 07:02:47 2024","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.34499999999999997,"15min":0.32800000000000001},"min":{"1min":0.221,"5min":0.217,"15min":0.217},"max":{"1min":0.53000000000000003,"5min":0.58499999999999996,"15min":0.58499999999999996},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.35699999999999998,"15min":0.34000000000000002},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.56799999999999995,"5min":0.57599999999999996,"15min":0.57599999999999996},"last":0.33100000000000002}]},{"osd":2,"last update":"Tue Sep 17 07:03:15 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.38400000000000001,"15min":0.377},"min":{"1min":0.251,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.61099999999999999,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.378,"5min":0.37,"15min":0.36299999999999999},"min":{"1min":0.26600000000000001,"5min":0.246,"15min":0.246},"max":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"last":0.40400000000000003}]},{"osd":3,"last update":"Tue Sep 17 07:02:39 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.434,"15min":0.44},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.59299999999999997,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.42899999999999999,"15min":0.435},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.58099999999999996,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.39400000000000002}]},{"osd":4,"last update":"Tue Sep 17 07:02:56 2024","interfaces":[{"interface":"back","average":{"1min":1.1319999999999999,"5min":0.67500000000000004,"15min":0.59899999999999998},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":12.573,"5min":12.573,"15min":12.573},"last":0.629},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.64800000000000002}]},{"osd":5,"last update":"Tue Sep 17 07:03:19 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.55500000000000005,"15min":0.55000000000000004},"min":{"1min":0.376,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.54800000000000004,"15min":0.54200000000000004},"min":{"1min":0.39700000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.69799999999999995}]},{"osd":6,"last update":"Tue Sep 17 07:02:39 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.59099999999999997}]},{"osd":7,"last update":"Tue Sep 17 07:03:01 2024","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.2010000000000001,"15min":1.2010000000000001},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":12.548,"5min":12.548,"15min":12.548},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.68600000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T07:03:38.631 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T07:03:38.830 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:03:39.148 INFO:journalctl@ceph.node-exporter.b.smithi135.stdout:Sep 17 07:03:39 smithi135 bash[63838]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-09-17T07:03:39.451 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:39 smithi135 ceph-mon[33238]: from='client.24608 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:03:39.451 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:39 smithi135 ceph-mon[33238]: 4.a scrub starts 2024-09-17T07:03:39.451 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:39 smithi135 ceph-mon[33238]: 4.a scrub ok 2024-09-17T07:03:39.451 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:39 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:03:39.451 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:39 smithi135 ceph-mon[33238]: 6.1a scrub starts 2024-09-17T07:03:39.451 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:39 smithi135 ceph-mon[33238]: 6.1a scrub ok 2024-09-17T07:03:39.451 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:39 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:39.452 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:39 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:39.452 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:39 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:39.452 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:39 smithi135 ceph-mon[33238]: 5.7 scrub starts 2024-09-17T07:03:39.452 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:39 smithi135 ceph-mon[33238]: 5.7 scrub ok 2024-09-17T07:03:39.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:39 smithi100 ceph-mon[29092]: from='client.24608 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:03:39.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:39 smithi100 ceph-mon[29092]: 4.a scrub starts 2024-09-17T07:03:39.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:39 smithi100 ceph-mon[29092]: 4.a scrub ok 2024-09-17T07:03:39.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:39 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:03:39.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:39 smithi100 ceph-mon[29092]: 6.1a scrub starts 2024-09-17T07:03:39.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:39 smithi100 ceph-mon[29092]: 6.1a scrub ok 2024-09-17T07:03:39.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:39 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:39.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:39 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:39.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:39 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:39.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:39 smithi100 ceph-mon[29092]: 5.7 scrub starts 2024-09-17T07:03:39.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:39 smithi100 ceph-mon[29092]: 5.7 scrub ok 2024-09-17T07:03:39.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:39 smithi100 ceph-mon[39462]: from='client.24608 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:03:39.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:39 smithi100 ceph-mon[39462]: 4.a scrub starts 2024-09-17T07:03:39.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:39 smithi100 ceph-mon[39462]: 4.a scrub ok 2024-09-17T07:03:39.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:39 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:03:39.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:39 smithi100 ceph-mon[39462]: 6.1a scrub starts 2024-09-17T07:03:39.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:39 smithi100 ceph-mon[39462]: 6.1a scrub ok 2024-09-17T07:03:39.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:39 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:39.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:39 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:39.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:39 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:39.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:39 smithi100 ceph-mon[39462]: 5.7 scrub starts 2024-09-17T07:03:39.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:39 smithi100 ceph-mon[39462]: 5.7 scrub ok 2024-09-17T07:03:39.793 INFO:journalctl@ceph.node-exporter.b.smithi135.stdout:Sep 17 07:03:39 smithi135 bash[63838]: Getting image source signatures 2024-09-17T07:03:39.793 INFO:journalctl@ceph.node-exporter.b.smithi135.stdout:Sep 17 07:03:39 smithi135 bash[63838]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-09-17T07:03:39.794 INFO:journalctl@ceph.node-exporter.b.smithi135.stdout:Sep 17 07:03:39 smithi135 bash[63838]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-09-17T07:03:39.794 INFO:journalctl@ceph.node-exporter.b.smithi135.stdout:Sep 17 07:03:39 smithi135 bash[63838]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-09-17T07:03:40.532 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:40 smithi100 ceph-mon[29092]: Deploying daemon node-exporter.b on smithi135 2024-09-17T07:03:40.533 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:40 smithi100 ceph-mon[29092]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:03:40.533 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:40 smithi100 ceph-mon[29092]: 4.9 scrub starts 2024-09-17T07:03:40.533 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:40 smithi100 ceph-mon[29092]: 4.9 scrub ok 2024-09-17T07:03:40.533 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:40 smithi100 ceph-mon[39462]: Deploying daemon node-exporter.b on smithi135 2024-09-17T07:03:40.533 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:40 smithi100 ceph-mon[39462]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:03:40.533 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:40 smithi100 ceph-mon[39462]: 4.9 scrub starts 2024-09-17T07:03:40.533 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:40 smithi100 ceph-mon[39462]: 4.9 scrub ok 2024-09-17T07:03:40.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:40 smithi135 ceph-mon[33238]: Deploying daemon node-exporter.b on smithi135 2024-09-17T07:03:40.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:40 smithi135 ceph-mon[33238]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:03:40.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:40 smithi135 ceph-mon[33238]: 4.9 scrub starts 2024-09-17T07:03:40.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:40 smithi135 ceph-mon[33238]: 4.9 scrub ok 2024-09-17T07:03:40.923 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T07:03:40.925 INFO:teuthology.orchestra.run.smithi100.stderr:dumped all 2024-09-17T07:03:41.170 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:41 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:03:41.170 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:41 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:03:41.170 INFO:journalctl@ceph.node-exporter.b.smithi135.stdout:Sep 17 07:03:40 smithi135 bash[63838]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-09-17T07:03:41.170 INFO:journalctl@ceph.node-exporter.b.smithi135.stdout:Sep 17 07:03:41 smithi135 bash[63838]: Writing manifest to image destination 2024-09-17T07:03:41.491 INFO:journalctl@ceph.node-exporter.b.smithi135.stdout:Sep 17 07:03:41 smithi135 podman[63838]: 2024-09-17 07:03:41.269893939 +0000 UTC m=+2.295972127 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-09-17T07:03:41.491 INFO:journalctl@ceph.node-exporter.b.smithi135.stdout:Sep 17 07:03:41 smithi135 podman[63838]: 2024-09-17 07:03:41.385127073 +0000 UTC m=+2.411205261 container create d10476c373fe3263de1f47ca8f18ac116bf0efe6e6e5f0464d666ce71533eb33 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-17T07:03:41.492 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:41 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:03:41.492 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:41 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:03:41.492 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:41 smithi135 ceph-mon[33238]: 5.16 scrub starts 2024-09-17T07:03:41.492 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:41 smithi135 ceph-mon[33238]: 5.16 scrub ok 2024-09-17T07:03:41.492 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:41 smithi135 ceph-mon[33238]: 2.d scrub starts 2024-09-17T07:03:41.492 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:41 smithi135 ceph-mon[33238]: 2.d scrub ok 2024-09-17T07:03:41.492 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:41 smithi135 ceph-mon[33238]: osdmap e90: 8 total, 8 up, 8 in 2024-09-17T07:03:41.492 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:41 smithi135 ceph-mon[33238]: 2.2 scrub starts 2024-09-17T07:03:41.492 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:41 smithi135 ceph-mon[33238]: 2.2 scrub ok 2024-09-17T07:03:41.492 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:41 smithi135 ceph-mon[33238]: from='client.24614 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:03:41.492 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:41 smithi135 ceph-mon[33238]: 4.3 scrub starts 2024-09-17T07:03:41.492 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:41 smithi135 ceph-mon[33238]: 4.3 scrub ok 2024-09-17T07:03:41.493 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:41 smithi135 ceph-mon[33238]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T07:03:41.558 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:41 smithi100 ceph-mon[29092]: 5.16 scrub starts 2024-09-17T07:03:41.559 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:41 smithi100 ceph-mon[29092]: 5.16 scrub ok 2024-09-17T07:03:41.559 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:41 smithi100 ceph-mon[29092]: 2.d scrub starts 2024-09-17T07:03:41.559 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:41 smithi100 ceph-mon[29092]: 2.d scrub ok 2024-09-17T07:03:41.559 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:41 smithi100 ceph-mon[29092]: osdmap e90: 8 total, 8 up, 8 in 2024-09-17T07:03:41.559 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:41 smithi100 ceph-mon[29092]: 2.2 scrub starts 2024-09-17T07:03:41.559 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:41 smithi100 ceph-mon[29092]: 2.2 scrub ok 2024-09-17T07:03:41.559 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:41 smithi100 ceph-mon[29092]: from='client.24614 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:03:41.559 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:41 smithi100 ceph-mon[29092]: 4.3 scrub starts 2024-09-17T07:03:41.559 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:41 smithi100 ceph-mon[29092]: 4.3 scrub ok 2024-09-17T07:03:41.559 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:41 smithi100 ceph-mon[29092]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T07:03:41.560 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:41 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:03:41.560 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:41 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:03:41.560 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:41 smithi100 ceph-mon[39462]: 5.16 scrub starts 2024-09-17T07:03:41.560 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:41 smithi100 ceph-mon[39462]: 5.16 scrub ok 2024-09-17T07:03:41.560 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:41 smithi100 ceph-mon[39462]: 2.d scrub starts 2024-09-17T07:03:41.560 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:41 smithi100 ceph-mon[39462]: 2.d scrub ok 2024-09-17T07:03:41.560 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:41 smithi100 ceph-mon[39462]: osdmap e90: 8 total, 8 up, 8 in 2024-09-17T07:03:41.560 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:41 smithi100 ceph-mon[39462]: 2.2 scrub starts 2024-09-17T07:03:41.560 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:41 smithi100 ceph-mon[39462]: 2.2 scrub ok 2024-09-17T07:03:41.560 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:41 smithi100 ceph-mon[39462]: from='client.24614 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:03:41.560 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:41 smithi100 ceph-mon[39462]: 4.3 scrub starts 2024-09-17T07:03:41.561 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:41 smithi100 ceph-mon[39462]: 4.3 scrub ok 2024-09-17T07:03:41.561 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:41 smithi100 ceph-mon[39462]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T07:03:41.579 INFO:teuthology.orchestra.run.smithi100.stdout:{"pg_ready":true,"pg_map":{"version":41,"stamp":"2024-09-17T07:03:40.166537+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596246,"num_objects":196,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":1001,"num_read_kb":847,"num_write":609,"num_write_kb":2459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":604,"ondisk_log_size":604,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":248600,"kb_used_data":7896,"kb_used_omap":12,"kb_used_meta":240627,"kb_avail":749712616,"statfs":{"total":767960285184,"available":767705718784,"internally_reserved":0,"allocated":8085504,"data_stored":4538583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":246402616},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.161652"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.021243+0000","last_change":"2024-09-17T07:02:18.366053+0000","last_active":"2024-09-17T07:03:20.021243+0000","last_peered":"2024-09-17T07:03:20.021243+0000","last_clean":"2024-09-17T07:03:20.021243+0000","last_became_active":"2024-09-17T07:02:18.365773+0000","last_became_peered":"2024-09-17T07:02:18.365773+0000","last_unstale":"2024-09-17T07:03:20.021243+0000","last_undegraded":"2024-09-17T07:03:20.021243+0000","last_fullsized":"2024-09-17T07:03:20.021243+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:02:17.002904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:39.725507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.746536+0000","last_change":"2024-09-17T07:02:12.007016+0000","last_active":"2024-09-17T07:03:24.746536+0000","last_peered":"2024-09-17T07:03:24.746536+0000","last_clean":"2024-09-17T07:03:24.746536+0000","last_became_active":"2024-09-17T07:02:12.006754+0000","last_became_peered":"2024-09-17T07:02:12.006754+0000","last_unstale":"2024-09-17T07:03:24.746536+0000","last_undegraded":"2024-09-17T07:03:24.746536+0000","last_fullsized":"2024-09-17T07:03:24.746536+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:02:10.991363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:33.036975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"75'15","reported_seq":64,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-17T07:03:30.436458+0000","last_change":"2024-09-17T07:03:30.436540+0000","last_active":"2024-09-17T07:03:30.436458+0000","last_peered":"2024-09-17T07:03:30.436458+0000","last_clean":"2024-09-17T07:03:30.436458+0000","last_became_active":"2024-09-17T07:02:14.012962+0000","last_became_peered":"2024-09-17T07:02:14.012962+0000","last_unstale":"2024-09-17T07:03:30.436458+0000","last_undegraded":"2024-09-17T07:03:30.436458+0000","last_fullsized":"2024-09-17T07:03:30.436458+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-09-17T07:03:30.436444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:30.436444+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:35.913395+0000","scrub_duration":0.0023734419999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.743778+0000","last_change":"2024-09-17T07:02:16.015244+0000","last_active":"2024-09-17T07:03:24.743778+0000","last_peered":"2024-09-17T07:03:24.743778+0000","last_clean":"2024-09-17T07:03:24.743778+0000","last_became_active":"2024-09-17T07:02:16.014928+0000","last_became_peered":"2024-09-17T07:02:16.014928+0000","last_unstale":"2024-09-17T07:03:24.743778+0000","last_undegraded":"2024-09-17T07:03:24.743778+0000","last_fullsized":"2024-09-17T07:03:24.743778+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:02:14.999435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:38.559521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.020720+0000","last_change":"2024-09-17T07:02:12.010335+0000","last_active":"2024-09-17T07:03:20.020720+0000","last_peered":"2024-09-17T07:03:20.020720+0000","last_clean":"2024-09-17T07:03:20.020720+0000","last_became_active":"2024-09-17T07:02:12.010069+0000","last_became_peered":"2024-09-17T07:02:12.010069+0000","last_unstale":"2024-09-17T07:03:20.020720+0000","last_undegraded":"2024-09-17T07:03:20.020720+0000","last_fullsized":"2024-09-17T07:03:20.020720+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:02:10.991363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:29.849457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"75'9","reported_seq":56,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.743781+0000","last_change":"2024-09-17T07:03:17.864856+0000","last_active":"2024-09-17T07:03:24.743781+0000","last_peered":"2024-09-17T07:03:24.743781+0000","last_clean":"2024-09-17T07:03:24.743781+0000","last_became_active":"2024-09-17T07:02:14.016244+0000","last_became_peered":"2024-09-17T07:02:14.016244+0000","last_unstale":"2024-09-17T07:03:24.743781+0000","last_undegraded":"2024-09-17T07:03:24.743781+0000","last_fullsized":"2024-09-17T07:03:24.743781+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:03:17.864726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:17.864726+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:29.697946+0000","scrub_duration":0.0021756380000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":37,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-17T07:03:37.112968+0000","last_change":"2024-09-17T07:03:37.113047+0000","last_active":"2024-09-17T07:03:37.112968+0000","last_peered":"2024-09-17T07:03:37.112968+0000","last_clean":"2024-09-17T07:03:37.112968+0000","last_became_active":"2024-09-17T07:02:16.014380+0000","last_became_peered":"2024-09-17T07:02:16.014380+0000","last_unstale":"2024-09-17T07:03:37.112968+0000","last_undegraded":"2024-09-17T07:03:37.112968+0000","last_fullsized":"2024-09-17T07:03:37.112968+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:37.112954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:37.112954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:45.057903+0000","scrub_duration":0.001466824,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.743964+0000","last_change":"2024-09-17T07:02:18.021079+0000","last_active":"2024-09-17T07:03:24.743964+0000","last_peered":"2024-09-17T07:03:24.743964+0000","last_clean":"2024-09-17T07:03:24.743964+0000","last_became_active":"2024-09-17T07:02:18.020790+0000","last_became_peered":"2024-09-17T07:02:18.020790+0000","last_unstale":"2024-09-17T07:03:24.743964+0000","last_undegraded":"2024-09-17T07:03:24.743964+0000","last_fullsized":"2024-09-17T07:03:24.743964+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:02:17.002904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.956433+0000","last_change":"2024-09-17T07:02:12.011431+0000","last_active":"2024-09-17T07:03:24.956433+0000","last_peered":"2024-09-17T07:03:24.956433+0000","last_clean":"2024-09-17T07:03:24.956433+0000","last_became_active":"2024-09-17T07:02:12.011129+0000","last_became_peered":"2024-09-17T07:02:12.011129+0000","last_unstale":"2024-09-17T07:03:24.956433+0000","last_undegraded":"2024-09-17T07:03:24.956433+0000","last_fullsized":"2024-09-17T07:03:24.956433+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:02:10.991363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:32.031791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"75'5","reported_seq":50,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-17T07:03:29.002334+0000","last_change":"2024-09-17T07:03:29.002445+0000","last_active":"2024-09-17T07:03:29.002334+0000","last_peered":"2024-09-17T07:03:29.002334+0000","last_clean":"2024-09-17T07:03:29.002334+0000","last_became_active":"2024-09-17T07:02:14.016640+0000","last_became_peered":"2024-09-17T07:02:14.016640+0000","last_unstale":"2024-09-17T07:03:29.002334+0000","last_undegraded":"2024-09-17T07:03:29.002334+0000","last_fullsized":"2024-09-17T07:03:29.002334+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-09-17T07:03:29.002312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:29.002312+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:44.000292+0000","scrub_duration":0.002172748,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":38,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.955709+0000","last_change":"2024-09-17T07:03:22.015629+0000","last_active":"2024-09-17T07:03:24.955709+0000","last_peered":"2024-09-17T07:03:24.955709+0000","last_clean":"2024-09-17T07:03:24.955709+0000","last_became_active":"2024-09-17T07:02:16.015401+0000","last_became_peered":"2024-09-17T07:02:16.015401+0000","last_unstale":"2024-09-17T07:03:24.955709+0000","last_undegraded":"2024-09-17T07:03:24.955709+0000","last_fullsized":"2024-09-17T07:03:24.955709+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:22.015535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:22.015535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:49.909183+0000","scrub_duration":0.001507752,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":35,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-17T07:03:34.881963+0000","last_change":"2024-09-17T07:03:34.882046+0000","last_active":"2024-09-17T07:03:34.881963+0000","last_peered":"2024-09-17T07:03:34.881963+0000","last_clean":"2024-09-17T07:03:34.881963+0000","last_became_active":"2024-09-17T07:02:18.023680+0000","last_became_peered":"2024-09-17T07:02:18.023680+0000","last_unstale":"2024-09-17T07:03:34.881963+0000","last_undegraded":"2024-09-17T07:03:34.881963+0000","last_fullsized":"2024-09-17T07:03:34.881963+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:34.881951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:03:34.881951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:36.646147+0000","scrub_duration":0.0014520379999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.956209+0000","last_change":"2024-09-17T07:02:12.012509+0000","last_active":"2024-09-17T07:03:24.956209+0000","last_peered":"2024-09-17T07:03:24.956209+0000","last_clean":"2024-09-17T07:03:24.956209+0000","last_became_active":"2024-09-17T07:02:12.012234+0000","last_became_peered":"2024-09-17T07:02:12.012234+0000","last_unstale":"2024-09-17T07:03:24.956209+0000","last_undegraded":"2024-09-17T07:03:24.956209+0000","last_fullsized":"2024-09-17T07:03:24.956209+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:02:10.991363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:34.001627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"75'9","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.744149+0000","last_change":"2024-09-17T07:02:14.013597+0000","last_active":"2024-09-17T07:03:24.744149+0000","last_peered":"2024-09-17T07:03:24.744149+0000","last_clean":"2024-09-17T07:03:24.744149+0000","last_became_active":"2024-09-17T07:02:14.013337+0000","last_became_peered":"2024-09-17T07:02:14.013337+0000","last_unstale":"2024-09-17T07:03:24.744149+0000","last_undegraded":"2024-09-17T07:03:24.744149+0000","last_fullsized":"2024-09-17T07:03:24.744149+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:02:12.994221+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:37.293441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.536482+0000","last_change":"2024-09-17T07:02:16.024037+0000","last_active":"2024-09-17T07:03:20.536482+0000","last_peered":"2024-09-17T07:03:20.536482+0000","last_clean":"2024-09-17T07:03:20.536482+0000","last_became_active":"2024-09-17T07:02:16.023774+0000","last_became_peered":"2024-09-17T07:02:16.023774+0000","last_unstale":"2024-09-17T07:03:20.536482+0000","last_undegraded":"2024-09-17T07:03:20.536482+0000","last_fullsized":"2024-09-17T07:03:20.536482+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:02:14.999435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:44.675786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.746025+0000","last_change":"2024-09-17T07:02:18.029470+0000","last_active":"2024-09-17T07:03:24.746025+0000","last_peered":"2024-09-17T07:03:24.746025+0000","last_clean":"2024-09-17T07:03:24.746025+0000","last_became_active":"2024-09-17T07:02:18.029178+0000","last_became_peered":"2024-09-17T07:02:18.029178+0000","last_unstale":"2024-09-17T07:03:24.746025+0000","last_undegraded":"2024-09-17T07:03:24.746025+0000","last_fullsized":"2024-09-17T07:03:24.746025+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:02:17.002904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:45.877645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"75'17","reported_seq":57,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.744692+0000","last_change":"2024-09-17T07:02:14.016560+0000","last_active":"2024-09-17T07:03:24.744692+0000","last_peered":"2024-09-17T07:03:24.744692+0000","last_clean":"2024-09-17T07:03:24.744692+0000","last_became_active":"2024-09-17T07:02:14.016270+0000","last_became_peered":"2024-09-17T07:02:14.016270+0000","last_unstale":"2024-09-17T07:03:24.744692+0000","last_undegraded":"2024-09-17T07:03:24.744692+0000","last_fullsized":"2024-09-17T07:03:24.744692+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:02:12.994221+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:34.020252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":42,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.956163+0000","last_change":"2024-09-17T07:03:14.092519+0000","last_active":"2024-09-17T07:03:24.956163+0000","last_peered":"2024-09-17T07:03:24.956163+0000","last_clean":"2024-09-17T07:03:24.956163+0000","last_became_active":"2024-09-17T07:02:12.011258+0000","last_became_peered":"2024-09-17T07:02:12.011258+0000","last_unstale":"2024-09-17T07:03:24.956163+0000","last_undegraded":"2024-09-17T07:03:24.956163+0000","last_fullsized":"2024-09-17T07:03:24.956163+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:14.092369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:14.092369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:19.152787+0000","scrub_duration":0.0015447460000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.020658+0000","last_change":"2024-09-17T07:02:16.018444+0000","last_active":"2024-09-17T07:03:20.020658+0000","last_peered":"2024-09-17T07:03:20.020658+0000","last_clean":"2024-09-17T07:03:20.020658+0000","last_became_active":"2024-09-17T07:02:16.018147+0000","last_became_peered":"2024-09-17T07:02:16.018147+0000","last_unstale":"2024-09-17T07:03:20.020658+0000","last_undegraded":"2024-09-17T07:03:20.020658+0000","last_fullsized":"2024-09-17T07:03:20.020658+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:02:14.999435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.745631+0000","last_change":"2024-09-17T07:02:18.022326+0000","last_active":"2024-09-17T07:03:24.745631+0000","last_peered":"2024-09-17T07:03:24.745631+0000","last_clean":"2024-09-17T07:03:24.745631+0000","last_became_active":"2024-09-17T07:02:18.022040+0000","last_became_peered":"2024-09-17T07:02:18.022040+0000","last_unstale":"2024-09-17T07:03:24.745631+0000","last_undegraded":"2024-09-17T07:03:24.745631+0000","last_fullsized":"2024-09-17T07:03:24.745631+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:02:17.002904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:32.036001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"75'10","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.744525+0000","last_change":"2024-09-17T07:03:20.933478+0000","last_active":"2024-09-17T07:03:24.744525+0000","last_peered":"2024-09-17T07:03:24.744525+0000","last_clean":"2024-09-17T07:03:24.744525+0000","last_became_active":"2024-09-17T07:02:14.007702+0000","last_became_peered":"2024-09-17T07:02:14.007702+0000","last_unstale":"2024-09-17T07:03:24.744525+0000","last_undegraded":"2024-09-17T07:03:24.744525+0000","last_fullsized":"2024-09-17T07:03:24.744525+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-09-17T07:03:20.933353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:20.933353+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:28.865120+0000","scrub_duration":0.0021221080000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.021191+0000","last_change":"2024-09-17T07:02:12.007831+0000","last_active":"2024-09-17T07:03:20.021191+0000","last_peered":"2024-09-17T07:03:20.021191+0000","last_clean":"2024-09-17T07:03:20.021191+0000","last_became_active":"2024-09-17T07:02:12.007553+0000","last_became_peered":"2024-09-17T07:02:12.007553+0000","last_unstale":"2024-09-17T07:03:20.021191+0000","last_undegraded":"2024-09-17T07:03:20.021191+0000","last_fullsized":"2024-09-17T07:03:20.021191+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:02:10.991363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:30.657047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"75'8","reported_seq":52,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-17T07:03:30.534614+0000","last_change":"2024-09-17T07:02:16.015345+0000","last_active":"2024-09-17T07:03:30.534614+0000","last_peered":"2024-09-17T07:03:30.534614+0000","last_clean":"2024-09-17T07:03:30.534614+0000","last_became_active":"2024-09-17T07:02:16.015001+0000","last_became_peered":"2024-09-17T07:02:16.015001+0000","last_unstale":"2024-09-17T07:03:30.534614+0000","last_undegraded":"2024-09-17T07:03:30.534614+0000","last_fullsized":"2024-09-17T07:03:30.534614+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:02:14.999435+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.744747+0000","last_change":"2024-09-17T07:02:18.017718+0000","last_active":"2024-09-17T07:03:24.744747+0000","last_peered":"2024-09-17T07:03:24.744747+0000","last_clean":"2024-09-17T07:03:24.744747+0000","last_became_active":"2024-09-17T07:02:18.017460+0000","last_became_peered":"2024-09-17T07:02:18.017460+0000","last_unstale":"2024-09-17T07:03:24.744747+0000","last_undegraded":"2024-09-17T07:03:24.744747+0000","last_fullsized":"2024-09-17T07:03:24.744747+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:02:17.002904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:29.635650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"75'15","reported_seq":64,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.022157+0000","last_change":"2024-09-17T07:03:16.067607+0000","last_active":"2024-09-17T07:03:20.022157+0000","last_peered":"2024-09-17T07:03:20.022157+0000","last_clean":"2024-09-17T07:03:20.022157+0000","last_became_active":"2024-09-17T07:02:14.016397+0000","last_became_peered":"2024-09-17T07:02:14.016397+0000","last_unstale":"2024-09-17T07:03:20.022157+0000","last_undegraded":"2024-09-17T07:03:20.022157+0000","last_fullsized":"2024-09-17T07:03:20.022157+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-09-17T07:03:16.067479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:16.067479+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:28.548727+0000","scrub_duration":0.0025498640000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":41,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-17T07:03:28.072710+0000","last_change":"2024-09-17T07:03:28.072773+0000","last_active":"2024-09-17T07:03:28.072710+0000","last_peered":"2024-09-17T07:03:28.072710+0000","last_clean":"2024-09-17T07:03:28.072710+0000","last_became_active":"2024-09-17T07:02:12.007913+0000","last_became_peered":"2024-09-17T07:02:12.007913+0000","last_unstale":"2024-09-17T07:03:28.072710+0000","last_undegraded":"2024-09-17T07:03:28.072710+0000","last_fullsized":"2024-09-17T07:03:28.072710+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:28.072697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:28.072697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:43.368112+0000","scrub_duration":0.0012624730000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"75'8","reported_seq":52,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-17T07:03:30.534713+0000","last_change":"2024-09-17T07:02:16.022051+0000","last_active":"2024-09-17T07:03:30.534713+0000","last_peered":"2024-09-17T07:03:30.534713+0000","last_clean":"2024-09-17T07:03:30.534713+0000","last_became_active":"2024-09-17T07:02:16.021762+0000","last_became_peered":"2024-09-17T07:02:16.021762+0000","last_unstale":"2024-09-17T07:03:30.534713+0000","last_undegraded":"2024-09-17T07:03:30.534713+0000","last_fullsized":"2024-09-17T07:03:30.534713+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:02:14.999435+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:36.148452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.536729+0000","last_change":"2024-09-17T07:02:18.025574+0000","last_active":"2024-09-17T07:03:20.536729+0000","last_peered":"2024-09-17T07:03:20.536729+0000","last_clean":"2024-09-17T07:03:20.536729+0000","last_became_active":"2024-09-17T07:02:18.025332+0000","last_became_peered":"2024-09-17T07:02:18.025332+0000","last_unstale":"2024-09-17T07:03:20.536729+0000","last_undegraded":"2024-09-17T07:03:20.536729+0000","last_fullsized":"2024-09-17T07:03:20.536729+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:02:17.002904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:36.070328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"75'11","reported_seq":48,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.744909+0000","last_change":"2024-09-17T07:02:14.010584+0000","last_active":"2024-09-17T07:03:24.744909+0000","last_peered":"2024-09-17T07:03:24.744909+0000","last_clean":"2024-09-17T07:03:24.744909+0000","last_became_active":"2024-09-17T07:02:14.010227+0000","last_became_peered":"2024-09-17T07:02:14.010227+0000","last_unstale":"2024-09-17T07:03:24.744909+0000","last_undegraded":"2024-09-17T07:03:24.744909+0000","last_fullsized":"2024-09-17T07:03:24.744909+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:02:12.994221+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:33.857964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":42,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-17T07:03:31.076149+0000","last_change":"2024-09-17T07:03:31.076235+0000","last_active":"2024-09-17T07:03:31.076149+0000","last_peered":"2024-09-17T07:03:31.076149+0000","last_clean":"2024-09-17T07:03:31.076149+0000","last_became_active":"2024-09-17T07:02:12.013182+0000","last_became_peered":"2024-09-17T07:02:12.013182+0000","last_unstale":"2024-09-17T07:03:31.076149+0000","last_undegraded":"2024-09-17T07:03:31.076149+0000","last_fullsized":"2024-09-17T07:03:31.076149+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:31.076134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:31.076134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:37.803854+0000","scrub_duration":0.0012890040000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":37,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.536129+0000","last_change":"2024-09-17T07:03:18.816900+0000","last_active":"2024-09-17T07:03:20.536129+0000","last_peered":"2024-09-17T07:03:20.536129+0000","last_clean":"2024-09-17T07:03:20.536129+0000","last_became_active":"2024-09-17T07:02:16.020224+0000","last_became_peered":"2024-09-17T07:02:16.020224+0000","last_unstale":"2024-09-17T07:03:20.536129+0000","last_undegraded":"2024-09-17T07:03:20.536129+0000","last_fullsized":"2024-09-17T07:03:20.536129+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:18.816795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:18.816795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:46.254399+0000","scrub_duration":0.001573212,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":35,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-17T07:03:35.447092+0000","last_change":"2024-09-17T07:03:35.447165+0000","last_active":"2024-09-17T07:03:35.447092+0000","last_peered":"2024-09-17T07:03:35.447092+0000","last_clean":"2024-09-17T07:03:35.447092+0000","last_became_active":"2024-09-17T07:02:18.365000+0000","last_became_peered":"2024-09-17T07:02:18.365000+0000","last_unstale":"2024-09-17T07:03:35.447092+0000","last_undegraded":"2024-09-17T07:03:35.447092+0000","last_fullsized":"2024-09-17T07:03:35.447092+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:35.447078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:03:35.447078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:38.486661+0000","scrub_duration":0.0020715600000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"75'14","reported_seq":50,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.743996+0000","last_change":"2024-09-17T07:02:14.008065+0000","last_active":"2024-09-17T07:03:24.743996+0000","last_peered":"2024-09-17T07:03:24.743996+0000","last_clean":"2024-09-17T07:03:24.743996+0000","last_became_active":"2024-09-17T07:02:14.007810+0000","last_became_peered":"2024-09-17T07:02:14.007810+0000","last_unstale":"2024-09-17T07:03:24.743996+0000","last_undegraded":"2024-09-17T07:03:24.743996+0000","last_fullsized":"2024-09-17T07:03:24.743996+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:02:12.994221+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:38.353465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.115548+0000","last_change":"2024-09-17T07:03:16.455893+0000","last_active":"2024-09-17T07:03:20.115548+0000","last_peered":"2024-09-17T07:03:20.115548+0000","last_clean":"2024-09-17T07:03:20.115548+0000","last_became_active":"2024-09-17T07:02:12.011766+0000","last_became_peered":"2024-09-17T07:02:12.011766+0000","last_unstale":"2024-09-17T07:03:20.115548+0000","last_undegraded":"2024-09-17T07:03:20.115548+0000","last_fullsized":"2024-09-17T07:03:20.115548+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:16.455684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:16.455684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:22.392690+0000","scrub_duration":0.0020498690000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"75'8","reported_seq":62,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-17T07:03:33.444197+0000","last_change":"2024-09-17T07:03:33.444197+0000","last_active":"2024-09-17T07:03:33.444197+0000","last_peered":"2024-09-17T07:03:33.444197+0000","last_clean":"2024-09-17T07:03:33.444197+0000","last_became_active":"2024-09-17T07:02:16.020628+0000","last_became_peered":"2024-09-17T07:02:16.020628+0000","last_unstale":"2024-09-17T07:03:33.444197+0000","last_undegraded":"2024-09-17T07:03:33.444197+0000","last_fullsized":"2024-09-17T07:03:33.444197+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:03:33.444104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:33.444104+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:36.767124+0000","scrub_duration":0.0014124389999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":36,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.955981+0000","last_change":"2024-09-17T07:03:19.030574+0000","last_active":"2024-09-17T07:03:24.955981+0000","last_peered":"2024-09-17T07:03:24.955981+0000","last_clean":"2024-09-17T07:03:24.955981+0000","last_became_active":"2024-09-17T07:02:18.364981+0000","last_became_peered":"2024-09-17T07:02:18.364981+0000","last_unstale":"2024-09-17T07:03:24.955981+0000","last_undegraded":"2024-09-17T07:03:24.955981+0000","last_fullsized":"2024-09-17T07:03:24.955981+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:19.030470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:03:19.030470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:48.307795+0000","scrub_duration":0.00152618,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"75'18","reported_seq":71,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-17T07:03:26.413521+0000","last_change":"2024-09-17T07:03:26.413610+0000","last_active":"2024-09-17T07:03:26.413521+0000","last_peered":"2024-09-17T07:03:26.413521+0000","last_clean":"2024-09-17T07:03:26.413521+0000","last_became_active":"2024-09-17T07:02:14.028667+0000","last_became_peered":"2024-09-17T07:02:14.028667+0000","last_unstale":"2024-09-17T07:03:26.413521+0000","last_undegraded":"2024-09-17T07:03:26.413521+0000","last_fullsized":"2024-09-17T07:03:26.413521+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'18","last_scrub_stamp":"2024-09-17T07:03:26.413506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:26.413506+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:41.741155+0000","scrub_duration":0.0022855470000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.134708+0000","last_change":"2024-09-17T07:03:24.134708+0000","last_active":"2024-09-17T07:03:24.134708+0000","last_peered":"2024-09-17T07:03:24.134708+0000","last_clean":"2024-09-17T07:03:24.134708+0000","last_became_active":"2024-09-17T07:02:12.005687+0000","last_became_peered":"2024-09-17T07:02:12.005687+0000","last_unstale":"2024-09-17T07:03:24.134708+0000","last_undegraded":"2024-09-17T07:03:24.134708+0000","last_fullsized":"2024-09-17T07:03:24.134708+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:24.134627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:24.134627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:29.080338+0000","scrub_duration":0.001111148,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.743837+0000","last_change":"2024-09-17T07:02:16.020013+0000","last_active":"2024-09-17T07:03:24.743837+0000","last_peered":"2024-09-17T07:03:24.743837+0000","last_clean":"2024-09-17T07:03:24.743837+0000","last_became_active":"2024-09-17T07:02:16.019747+0000","last_became_peered":"2024-09-17T07:02:16.019747+0000","last_unstale":"2024-09-17T07:03:24.743837+0000","last_undegraded":"2024-09-17T07:03:24.743837+0000","last_fullsized":"2024-09-17T07:03:24.743837+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:02:14.999435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:30.253095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.744121+0000","last_change":"2024-09-17T07:02:18.024506+0000","last_active":"2024-09-17T07:03:24.744121+0000","last_peered":"2024-09-17T07:03:24.744121+0000","last_clean":"2024-09-17T07:03:24.744121+0000","last_became_active":"2024-09-17T07:02:18.024043+0000","last_became_peered":"2024-09-17T07:02:18.024043+0000","last_unstale":"2024-09-17T07:03:24.744121+0000","last_undegraded":"2024-09-17T07:03:24.744121+0000","last_fullsized":"2024-09-17T07:03:24.744121+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:02:17.002904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:29.154987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"75'19","reported_seq":65,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.746593+0000","last_change":"2024-09-17T07:02:14.027680+0000","last_active":"2024-09-17T07:03:24.746593+0000","last_peered":"2024-09-17T07:03:24.746593+0000","last_clean":"2024-09-17T07:03:24.746593+0000","last_became_active":"2024-09-17T07:02:14.027389+0000","last_became_peered":"2024-09-17T07:02:14.027389+0000","last_unstale":"2024-09-17T07:03:24.746593+0000","last_undegraded":"2024-09-17T07:03:24.746593+0000","last_fullsized":"2024-09-17T07:03:24.746593+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:02:12.994221+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:31.589637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":42,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-17T07:03:32.109780+0000","last_change":"2024-09-17T07:03:32.109842+0000","last_active":"2024-09-17T07:03:32.109780+0000","last_peered":"2024-09-17T07:03:32.109780+0000","last_clean":"2024-09-17T07:03:32.109780+0000","last_became_active":"2024-09-17T07:02:12.011917+0000","last_became_peered":"2024-09-17T07:02:12.011917+0000","last_unstale":"2024-09-17T07:03:32.109780+0000","last_undegraded":"2024-09-17T07:03:32.109780+0000","last_fullsized":"2024-09-17T07:03:32.109780+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:32.109765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:32.109765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:52.813415+0000","scrub_duration":0.001335314,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.114920+0000","last_change":"2024-09-17T07:02:16.016938+0000","last_active":"2024-09-17T07:03:20.114920+0000","last_peered":"2024-09-17T07:03:20.114920+0000","last_clean":"2024-09-17T07:03:20.114920+0000","last_became_active":"2024-09-17T07:02:16.016199+0000","last_became_peered":"2024-09-17T07:02:16.016199+0000","last_unstale":"2024-09-17T07:03:20.114920+0000","last_undegraded":"2024-09-17T07:03:20.114920+0000","last_fullsized":"2024-09-17T07:03:20.114920+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:02:14.999435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:36.124749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":35,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-17T07:03:30.068821+0000","last_change":"2024-09-17T07:03:30.068895+0000","last_active":"2024-09-17T07:03:30.068821+0000","last_peered":"2024-09-17T07:03:30.068821+0000","last_clean":"2024-09-17T07:03:30.068821+0000","last_became_active":"2024-09-17T07:02:18.364835+0000","last_became_peered":"2024-09-17T07:02:18.364835+0000","last_unstale":"2024-09-17T07:03:30.068821+0000","last_undegraded":"2024-09-17T07:03:30.068821+0000","last_fullsized":"2024-09-17T07:03:30.068821+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:30.068808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:03:30.068808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:51.270374+0000","scrub_duration":0.0012893679999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"75'10","reported_seq":54,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-17T07:03:26.127454+0000","last_change":"2024-09-17T07:03:26.127545+0000","last_active":"2024-09-17T07:03:26.127454+0000","last_peered":"2024-09-17T07:03:26.127454+0000","last_clean":"2024-09-17T07:03:26.127454+0000","last_became_active":"2024-09-17T07:02:14.010637+0000","last_became_peered":"2024-09-17T07:02:14.010637+0000","last_unstale":"2024-09-17T07:03:26.127454+0000","last_undegraded":"2024-09-17T07:03:26.127454+0000","last_fullsized":"2024-09-17T07:03:26.127454+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-09-17T07:03:26.127439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:26.127439+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:35.535770+0000","scrub_duration":0.001898479,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:25.148144+0000","last_change":"2024-09-17T07:03:25.148144+0000","last_active":"2024-09-17T07:03:25.148144+0000","last_peered":"2024-09-17T07:03:25.148144+0000","last_clean":"2024-09-17T07:03:25.148144+0000","last_became_active":"2024-09-17T07:02:12.010803+0000","last_became_peered":"2024-09-17T07:02:12.010803+0000","last_unstale":"2024-09-17T07:03:25.148144+0000","last_undegraded":"2024-09-17T07:03:25.148144+0000","last_fullsized":"2024-09-17T07:03:25.148144+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:25.147878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:25.147878+0000","last_clean_scrub_stamp":"2024-09-17T07:03:25.147878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:33.071698+0000","scrub_duration":0.001353723,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"89'9","reported_seq":64,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-17T07:03:30.534612+0000","last_change":"2024-09-17T07:03:29.961568+0000","last_active":"2024-09-17T07:03:30.534612+0000","last_peered":"2024-09-17T07:03:30.534612+0000","last_clean":"2024-09-17T07:03:30.534612+0000","last_became_active":"2024-09-17T07:02:16.021360+0000","last_became_peered":"2024-09-17T07:02:16.021360+0000","last_unstale":"2024-09-17T07:03:30.534612+0000","last_undegraded":"2024-09-17T07:03:30.534612+0000","last_fullsized":"2024-09-17T07:03:30.534612+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'9","last_scrub_stamp":"2024-09-17T07:03:29.961439+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-09-17T07:03:29.961439+0000","last_clean_scrub_stamp":"2024-09-17T07:03:29.961439+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:40.753813+0000","scrub_duration":0.0027696880000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"75'1","reported_seq":37,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-17T07:03:25.048964+0000","last_change":"2024-09-17T07:03:25.049036+0000","last_active":"2024-09-17T07:03:25.048964+0000","last_peered":"2024-09-17T07:03:25.048964+0000","last_clean":"2024-09-17T07:03:25.048964+0000","last_became_active":"2024-09-17T07:02:18.029772+0000","last_became_peered":"2024-09-17T07:02:18.029772+0000","last_unstale":"2024-09-17T07:03:25.048964+0000","last_undegraded":"2024-09-17T07:03:25.048964+0000","last_fullsized":"2024-09-17T07:03:25.048964+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-17T07:03:25.048949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:03:25.048949+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:53.755876+0000","scrub_duration":0.003886123,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"75'28","reported_seq":93,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-17T07:03:30.534634+0000","last_change":"2024-09-17T07:02:14.017598+0000","last_active":"2024-09-17T07:03:30.534634+0000","last_peered":"2024-09-17T07:03:30.534634+0000","last_clean":"2024-09-17T07:03:30.534634+0000","last_became_active":"2024-09-17T07:02:14.017377+0000","last_became_peered":"2024-09-17T07:02:14.017377+0000","last_unstale":"2024-09-17T07:03:30.534634+0000","last_undegraded":"2024-09-17T07:03:30.534634+0000","last_fullsized":"2024-09-17T07:03:30.534634+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:02:12.994221+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.536680+0000","last_change":"2024-09-17T07:02:12.009135+0000","last_active":"2024-09-17T07:03:20.536680+0000","last_peered":"2024-09-17T07:03:20.536680+0000","last_clean":"2024-09-17T07:03:20.536680+0000","last_became_active":"2024-09-17T07:02:12.008802+0000","last_became_peered":"2024-09-17T07:02:12.008802+0000","last_unstale":"2024-09-17T07:03:20.536680+0000","last_undegraded":"2024-09-17T07:03:20.536680+0000","last_fullsized":"2024-09-17T07:03:20.536680+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:02:10.991363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:27.825147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-17T07:03:28.981163+0000","last_change":"2024-09-17T07:03:28.981244+0000","last_active":"2024-09-17T07:03:28.981163+0000","last_peered":"2024-09-17T07:03:28.981163+0000","last_clean":"2024-09-17T07:03:28.981163+0000","last_became_active":"2024-09-17T07:02:16.016606+0000","last_became_peered":"2024-09-17T07:02:16.016606+0000","last_unstale":"2024-09-17T07:03:28.981163+0000","last_undegraded":"2024-09-17T07:03:28.981163+0000","last_fullsized":"2024-09-17T07:03:28.981163+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:28.981147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:28.981147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:37.078325+0000","scrub_duration":0.0012741460000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.020508+0000","last_change":"2024-09-17T07:02:18.026015+0000","last_active":"2024-09-17T07:03:20.020508+0000","last_peered":"2024-09-17T07:03:20.020508+0000","last_clean":"2024-09-17T07:03:20.020508+0000","last_became_active":"2024-09-17T07:02:18.024602+0000","last_became_peered":"2024-09-17T07:02:18.024602+0000","last_unstale":"2024-09-17T07:03:20.020508+0000","last_undegraded":"2024-09-17T07:03:20.020508+0000","last_fullsized":"2024-09-17T07:03:20.020508+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:02:17.002904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:39.729726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"75'13","reported_seq":66,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-17T07:03:32.079596+0000","last_change":"2024-09-17T07:03:32.079678+0000","last_active":"2024-09-17T07:03:32.079596+0000","last_peered":"2024-09-17T07:03:32.079596+0000","last_clean":"2024-09-17T07:03:32.079596+0000","last_became_active":"2024-09-17T07:02:14.019838+0000","last_became_peered":"2024-09-17T07:02:14.019838+0000","last_unstale":"2024-09-17T07:03:32.079596+0000","last_undegraded":"2024-09-17T07:03:32.079596+0000","last_fullsized":"2024-09-17T07:03:32.079596+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'13","last_scrub_stamp":"2024-09-17T07:03:32.079579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:32.079579+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:42.502680+0000","scrub_duration":0.001975929,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"76'5","reported_seq":117,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-17T07:03:36.108784+0000","last_change":"2024-09-17T07:02:16.008848+0000","last_active":"2024-09-17T07:03:36.108784+0000","last_peered":"2024-09-17T07:03:36.108784+0000","last_clean":"2024-09-17T07:03:36.108784+0000","last_became_active":"2024-09-17T07:02:12.006883+0000","last_became_peered":"2024-09-17T07:02:12.006883+0000","last_unstale":"2024-09-17T07:03:36.108784+0000","last_undegraded":"2024-09-17T07:03:36.108784+0000","last_fullsized":"2024-09-17T07:03:36.108784+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:02:10.991363+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033739799999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":79,"num_read_kb":74,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":42,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-17T07:03:33.795319+0000","last_change":"2024-09-17T07:03:33.795388+0000","last_active":"2024-09-17T07:03:33.795319+0000","last_peered":"2024-09-17T07:03:33.795319+0000","last_clean":"2024-09-17T07:03:33.795319+0000","last_became_active":"2024-09-17T07:02:12.010753+0000","last_became_peered":"2024-09-17T07:02:12.010753+0000","last_unstale":"2024-09-17T07:03:33.795319+0000","last_undegraded":"2024-09-17T07:03:33.795319+0000","last_fullsized":"2024-09-17T07:03:33.795319+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:33.795307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:33.795307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:00.370993+0000","scrub_duration":0.0012164070000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.745596+0000","last_change":"2024-09-17T07:02:16.014775+0000","last_active":"2024-09-17T07:03:24.745596+0000","last_peered":"2024-09-17T07:03:24.745596+0000","last_clean":"2024-09-17T07:03:24.745596+0000","last_became_active":"2024-09-17T07:02:16.014492+0000","last_became_peered":"2024-09-17T07:02:16.014492+0000","last_unstale":"2024-09-17T07:03:24.745596+0000","last_undegraded":"2024-09-17T07:03:24.745596+0000","last_fullsized":"2024-09-17T07:03:24.745596+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:02:14.999435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:41.880289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.956165+0000","last_change":"2024-09-17T07:02:18.365627+0000","last_active":"2024-09-17T07:03:24.956165+0000","last_peered":"2024-09-17T07:03:24.956165+0000","last_clean":"2024-09-17T07:03:24.956165+0000","last_became_active":"2024-09-17T07:02:18.365369+0000","last_became_peered":"2024-09-17T07:02:18.365369+0000","last_unstale":"2024-09-17T07:03:24.956165+0000","last_undegraded":"2024-09-17T07:03:24.956165+0000","last_fullsized":"2024-09-17T07:03:24.956165+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:02:17.002904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:28.613944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"75'12","reported_seq":58,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-17T07:03:27.987849+0000","last_change":"2024-09-17T07:03:27.987937+0000","last_active":"2024-09-17T07:03:27.987849+0000","last_peered":"2024-09-17T07:03:27.987849+0000","last_clean":"2024-09-17T07:03:27.987849+0000","last_became_active":"2024-09-17T07:02:14.016206+0000","last_became_peered":"2024-09-17T07:02:14.016206+0000","last_unstale":"2024-09-17T07:03:27.987849+0000","last_undegraded":"2024-09-17T07:03:27.987849+0000","last_fullsized":"2024-09-17T07:03:27.987849+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-09-17T07:03:27.987835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:27.987835+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:50.273053+0000","scrub_duration":0.0020148219999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"70'1","reported_seq":37,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.746267+0000","last_change":"2024-09-17T07:02:16.017796+0000","last_active":"2024-09-17T07:03:24.746267+0000","last_peered":"2024-09-17T07:03:24.746267+0000","last_clean":"2024-09-17T07:03:24.746267+0000","last_became_active":"2024-09-17T07:02:12.009476+0000","last_became_peered":"2024-09-17T07:02:12.009476+0000","last_unstale":"2024-09-17T07:03:24.746267+0000","last_undegraded":"2024-09-17T07:03:24.746267+0000","last_fullsized":"2024-09-17T07:03:24.746267+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:02:10.991363+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:37.550940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032298629999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.955625+0000","last_change":"2024-09-17T07:02:12.014798+0000","last_active":"2024-09-17T07:03:24.955625+0000","last_peered":"2024-09-17T07:03:24.955625+0000","last_clean":"2024-09-17T07:03:24.955625+0000","last_became_active":"2024-09-17T07:02:12.014623+0000","last_became_peered":"2024-09-17T07:02:12.014623+0000","last_unstale":"2024-09-17T07:03:24.955625+0000","last_undegraded":"2024-09-17T07:03:24.955625+0000","last_fullsized":"2024-09-17T07:03:24.955625+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:02:10.991363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:40.031155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.536154+0000","last_change":"2024-09-17T07:02:16.021825+0000","last_active":"2024-09-17T07:03:20.536154+0000","last_peered":"2024-09-17T07:03:20.536154+0000","last_clean":"2024-09-17T07:03:20.536154+0000","last_became_active":"2024-09-17T07:02:16.021561+0000","last_became_peered":"2024-09-17T07:02:16.021561+0000","last_unstale":"2024-09-17T07:03:20.536154+0000","last_undegraded":"2024-09-17T07:03:20.536154+0000","last_fullsized":"2024-09-17T07:03:20.536154+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:02:14.999435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.020801+0000","last_change":"2024-09-17T07:02:18.022359+0000","last_active":"2024-09-17T07:03:20.020801+0000","last_peered":"2024-09-17T07:03:20.020801+0000","last_clean":"2024-09-17T07:03:20.020801+0000","last_became_active":"2024-09-17T07:02:18.022082+0000","last_became_peered":"2024-09-17T07:02:18.022082+0000","last_unstale":"2024-09-17T07:03:20.020801+0000","last_undegraded":"2024-09-17T07:03:20.020801+0000","last_fullsized":"2024-09-17T07:03:20.020801+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:02:17.002904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:36.430496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"75'14","reported_seq":50,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.115370+0000","last_change":"2024-09-17T07:02:14.029208+0000","last_active":"2024-09-17T07:03:20.115370+0000","last_peered":"2024-09-17T07:03:20.115370+0000","last_clean":"2024-09-17T07:03:20.115370+0000","last_became_active":"2024-09-17T07:02:14.028857+0000","last_became_peered":"2024-09-17T07:02:14.028857+0000","last_unstale":"2024-09-17T07:03:20.115370+0000","last_undegraded":"2024-09-17T07:03:20.115370+0000","last_fullsized":"2024-09-17T07:03:20.115370+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:02:12.994221+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:29.390413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"72'2","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-17T07:03:25.432684+0000","last_change":"2024-09-17T07:03:25.432755+0000","last_active":"2024-09-17T07:03:25.432684+0000","last_peered":"2024-09-17T07:03:25.432684+0000","last_clean":"2024-09-17T07:03:25.432684+0000","last_became_active":"2024-09-17T07:02:12.005127+0000","last_became_peered":"2024-09-17T07:02:12.005127+0000","last_unstale":"2024-09-17T07:03:25.432684+0000","last_undegraded":"2024-09-17T07:03:25.432684+0000","last_fullsized":"2024-09-17T07:03:25.432684+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'2","last_scrub_stamp":"2024-09-17T07:03:25.432670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:25.432670+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:25.994239+0000","scrub_duration":0.001640454,"objects_trimmed":0,"snaptrim_duration":0.0010980219999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.536731+0000","last_change":"2024-09-17T07:03:14.771416+0000","last_active":"2024-09-17T07:03:20.536731+0000","last_peered":"2024-09-17T07:03:20.536731+0000","last_clean":"2024-09-17T07:03:20.536731+0000","last_became_active":"2024-09-17T07:02:12.004606+0000","last_became_peered":"2024-09-17T07:02:12.004606+0000","last_unstale":"2024-09-17T07:03:20.536731+0000","last_undegraded":"2024-09-17T07:03:20.536731+0000","last_fullsized":"2024-09-17T07:03:20.536731+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:14.771307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:14.771307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:19.133578+0000","scrub_duration":0.0020495679999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"78'128","reported_seq":290,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.955697+0000","last_change":"2024-09-17T07:02:43.225880+0000","last_active":"2024-09-17T07:03:24.955697+0000","last_peered":"2024-09-17T07:03:24.955697+0000","last_clean":"2024-09-17T07:03:24.955697+0000","last_became_active":"2024-09-17T07:02:03.020675+0000","last_became_peered":"2024-09-17T07:02:03.020675+0000","last_unstale":"2024-09-17T07:03:24.955697+0000","last_undegraded":"2024-09-17T07:03:24.955697+0000","last_fullsized":"2024-09-17T07:03:24.955697+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'117","last_scrub_stamp":"2024-09-17T07:02:43.225790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:00:09.390381+0000","last_clean_scrub_stamp":"2024-09-17T07:02:43.225790+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:12.817117+0000","scrub_duration":0.001686302,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-17T07:03:30.069621+0000","last_change":"2024-09-17T07:03:30.069684+0000","last_active":"2024-09-17T07:03:30.069621+0000","last_peered":"2024-09-17T07:03:30.069621+0000","last_clean":"2024-09-17T07:03:30.069621+0000","last_became_active":"2024-09-17T07:02:16.016031+0000","last_became_peered":"2024-09-17T07:02:16.016031+0000","last_unstale":"2024-09-17T07:03:30.069621+0000","last_undegraded":"2024-09-17T07:03:30.069621+0000","last_fullsized":"2024-09-17T07:03:30.069621+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:30.069608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:30.069608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:36.831454+0000","scrub_duration":0.0012155379999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.537052+0000","last_change":"2024-09-17T07:02:18.023684+0000","last_active":"2024-09-17T07:03:20.537052+0000","last_peered":"2024-09-17T07:03:20.537052+0000","last_clean":"2024-09-17T07:03:20.537052+0000","last_became_active":"2024-09-17T07:02:18.023321+0000","last_became_peered":"2024-09-17T07:02:18.023321+0000","last_unstale":"2024-09-17T07:03:20.537052+0000","last_undegraded":"2024-09-17T07:03:20.537052+0000","last_fullsized":"2024-09-17T07:03:20.537052+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:02:17.002904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:39.925496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"75'19","reported_seq":70,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-17T07:03:37.517778+0000","last_change":"2024-09-17T07:03:37.517877+0000","last_active":"2024-09-17T07:03:37.517778+0000","last_peered":"2024-09-17T07:03:37.517778+0000","last_clean":"2024-09-17T07:03:37.517778+0000","last_became_active":"2024-09-17T07:02:14.027313+0000","last_became_peered":"2024-09-17T07:02:14.027313+0000","last_unstale":"2024-09-17T07:03:37.517778+0000","last_undegraded":"2024-09-17T07:03:37.517778+0000","last_fullsized":"2024-09-17T07:03:37.517778+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-09-17T07:03:37.517760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:37.517760+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:03.675532+0000","scrub_duration":0.0028316579999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":42,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-17T07:03:25.672373+0000","last_change":"2024-09-17T07:03:25.672437+0000","last_active":"2024-09-17T07:03:25.672373+0000","last_peered":"2024-09-17T07:03:25.672373+0000","last_clean":"2024-09-17T07:03:25.672373+0000","last_became_active":"2024-09-17T07:02:12.012037+0000","last_became_peered":"2024-09-17T07:02:12.012037+0000","last_unstale":"2024-09-17T07:03:25.672373+0000","last_undegraded":"2024-09-17T07:03:25.672373+0000","last_fullsized":"2024-09-17T07:03:25.672373+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:25.672360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:25.672360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:54.809122+0000","scrub_duration":0.0012704459999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-17T07:03:27.660734+0000","last_change":"2024-09-17T07:03:27.660802+0000","last_active":"2024-09-17T07:03:27.660734+0000","last_peered":"2024-09-17T07:03:27.660734+0000","last_clean":"2024-09-17T07:03:27.660734+0000","last_became_active":"2024-09-17T07:02:16.014631+0000","last_became_peered":"2024-09-17T07:02:16.014631+0000","last_unstale":"2024-09-17T07:03:27.660734+0000","last_undegraded":"2024-09-17T07:03:27.660734+0000","last_fullsized":"2024-09-17T07:03:27.660734+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:27.660721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:27.660721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:50.654559+0000","scrub_duration":0.0012746820000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.955661+0000","last_change":"2024-09-17T07:02:18.023600+0000","last_active":"2024-09-17T07:03:24.955661+0000","last_peered":"2024-09-17T07:03:24.955661+0000","last_clean":"2024-09-17T07:03:24.955661+0000","last_became_active":"2024-09-17T07:02:18.023299+0000","last_became_peered":"2024-09-17T07:02:18.023299+0000","last_unstale":"2024-09-17T07:03:24.955661+0000","last_undegraded":"2024-09-17T07:03:24.955661+0000","last_fullsized":"2024-09-17T07:03:24.955661+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:02:17.002904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:45.370120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"75'9","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.746394+0000","last_change":"2024-09-17T07:02:14.015257+0000","last_active":"2024-09-17T07:03:24.746394+0000","last_peered":"2024-09-17T07:03:24.746394+0000","last_clean":"2024-09-17T07:03:24.746394+0000","last_became_active":"2024-09-17T07:02:14.014939+0000","last_became_peered":"2024-09-17T07:02:14.014939+0000","last_unstale":"2024-09-17T07:03:24.746394+0000","last_undegraded":"2024-09-17T07:03:24.746394+0000","last_fullsized":"2024-09-17T07:03:24.746394+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:02:12.994221+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:34.702987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.021841+0000","last_change":"2024-09-17T07:02:12.000308+0000","last_active":"2024-09-17T07:03:20.021841+0000","last_peered":"2024-09-17T07:03:20.021841+0000","last_clean":"2024-09-17T07:03:20.021841+0000","last_became_active":"2024-09-17T07:02:12.000005+0000","last_became_peered":"2024-09-17T07:02:12.000005+0000","last_unstale":"2024-09-17T07:03:20.021841+0000","last_undegraded":"2024-09-17T07:03:20.021841+0000","last_fullsized":"2024-09-17T07:03:20.021841+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:02:10.991363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.745245+0000","last_change":"2024-09-17T07:02:16.017596+0000","last_active":"2024-09-17T07:03:24.745245+0000","last_peered":"2024-09-17T07:03:24.745245+0000","last_clean":"2024-09-17T07:03:24.745245+0000","last_became_active":"2024-09-17T07:02:16.017271+0000","last_became_peered":"2024-09-17T07:02:16.017271+0000","last_unstale":"2024-09-17T07:03:24.745245+0000","last_undegraded":"2024-09-17T07:03:24.745245+0000","last_fullsized":"2024-09-17T07:03:24.745245+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:02:14.999435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:32.095723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.746161+0000","last_change":"2024-09-17T07:02:18.027276+0000","last_active":"2024-09-17T07:03:24.746161+0000","last_peered":"2024-09-17T07:03:24.746161+0000","last_clean":"2024-09-17T07:03:24.746161+0000","last_became_active":"2024-09-17T07:02:18.026994+0000","last_became_peered":"2024-09-17T07:02:18.026994+0000","last_unstale":"2024-09-17T07:03:24.746161+0000","last_undegraded":"2024-09-17T07:03:24.746161+0000","last_fullsized":"2024-09-17T07:03:24.746161+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:02:17.002904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:43.693662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"75'15","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.536176+0000","last_change":"2024-09-17T07:02:14.028644+0000","last_active":"2024-09-17T07:03:20.536176+0000","last_peered":"2024-09-17T07:03:20.536176+0000","last_clean":"2024-09-17T07:03:20.536176+0000","last_became_active":"2024-09-17T07:02:14.028392+0000","last_became_peered":"2024-09-17T07:02:14.028392+0000","last_unstale":"2024-09-17T07:03:20.536176+0000","last_undegraded":"2024-09-17T07:03:20.536176+0000","last_fullsized":"2024-09-17T07:03:20.536176+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:02:12.994221+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:36.637466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'2","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-17T07:03:26.694387+0000","last_change":"2024-09-17T07:03:26.694471+0000","last_active":"2024-09-17T07:03:26.694387+0000","last_peered":"2024-09-17T07:03:26.694387+0000","last_clean":"2024-09-17T07:03:26.694387+0000","last_became_active":"2024-09-17T07:02:12.005538+0000","last_became_peered":"2024-09-17T07:02:12.005538+0000","last_unstale":"2024-09-17T07:03:26.694387+0000","last_undegraded":"2024-09-17T07:03:26.694387+0000","last_fullsized":"2024-09-17T07:03:26.694387+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-09-17T07:03:26.694366+0000","last_deep_scrub":"89'2","last_deep_scrub_stamp":"2024-09-17T07:03:26.694366+0000","last_clean_scrub_stamp":"2024-09-17T07:03:26.694366+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:38.099234+0000","scrub_duration":0.0026359439999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":420,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"75'8","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-17T07:03:30.534707+0000","last_change":"2024-09-17T07:02:16.015713+0000","last_active":"2024-09-17T07:03:30.534707+0000","last_peered":"2024-09-17T07:03:30.534707+0000","last_clean":"2024-09-17T07:03:30.534707+0000","last_became_active":"2024-09-17T07:02:16.015418+0000","last_became_peered":"2024-09-17T07:02:16.015418+0000","last_unstale":"2024-09-17T07:03:30.534707+0000","last_undegraded":"2024-09-17T07:03:30.534707+0000","last_fullsized":"2024-09-17T07:03:30.534707+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:02:14.999435+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":35,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:22.823018+0000","last_change":"2024-09-17T07:03:22.823018+0000","last_active":"2024-09-17T07:03:22.823018+0000","last_peered":"2024-09-17T07:03:22.823018+0000","last_clean":"2024-09-17T07:03:22.823018+0000","last_became_active":"2024-09-17T07:02:18.363253+0000","last_became_peered":"2024-09-17T07:02:18.363253+0000","last_unstale":"2024-09-17T07:03:22.823018+0000","last_undegraded":"2024-09-17T07:03:22.823018+0000","last_fullsized":"2024-09-17T07:03:22.823018+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:22.822922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:03:22.822922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:29.158808+0000","scrub_duration":0.0012979910000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"75'12","reported_seq":52,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.743919+0000","last_change":"2024-09-17T07:02:14.010593+0000","last_active":"2024-09-17T07:03:24.743919+0000","last_peered":"2024-09-17T07:03:24.743919+0000","last_clean":"2024-09-17T07:03:24.743919+0000","last_became_active":"2024-09-17T07:02:14.010219+0000","last_became_peered":"2024-09-17T07:02:14.010219+0000","last_unstale":"2024-09-17T07:03:24.743919+0000","last_undegraded":"2024-09-17T07:03:24.743919+0000","last_fullsized":"2024-09-17T07:03:24.743919+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:02:12.994221+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:25.821031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"68'2","reported_seq":49,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-17T07:03:28.021202+0000","last_change":"2024-09-17T07:03:28.021306+0000","last_active":"2024-09-17T07:03:28.021202+0000","last_peered":"2024-09-17T07:03:28.021202+0000","last_clean":"2024-09-17T07:03:28.021202+0000","last_became_active":"2024-09-17T07:02:12.006716+0000","last_became_peered":"2024-09-17T07:02:12.006716+0000","last_unstale":"2024-09-17T07:03:28.021202+0000","last_undegraded":"2024-09-17T07:03:28.021202+0000","last_fullsized":"2024-09-17T07:03:28.021202+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-09-17T07:03:28.021180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:28.021180+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:55.299746+0000","scrub_duration":0.0016672220000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-17T07:03:36.832766+0000","last_change":"2024-09-17T07:03:36.832832+0000","last_active":"2024-09-17T07:03:36.832766+0000","last_peered":"2024-09-17T07:03:36.832766+0000","last_clean":"2024-09-17T07:03:36.832766+0000","last_became_active":"2024-09-17T07:02:16.012261+0000","last_became_peered":"2024-09-17T07:02:16.012261+0000","last_unstale":"2024-09-17T07:03:36.832766+0000","last_undegraded":"2024-09-17T07:03:36.832766+0000","last_fullsized":"2024-09-17T07:03:36.832766+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:36.832752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:36.832752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:44.624052+0000","scrub_duration":0.001189278,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.021388+0000","last_change":"2024-09-17T07:02:18.025590+0000","last_active":"2024-09-17T07:03:20.021388+0000","last_peered":"2024-09-17T07:03:20.021388+0000","last_clean":"2024-09-17T07:03:20.021388+0000","last_became_active":"2024-09-17T07:02:18.024983+0000","last_became_peered":"2024-09-17T07:02:18.024983+0000","last_unstale":"2024-09-17T07:03:20.021388+0000","last_undegraded":"2024-09-17T07:03:20.021388+0000","last_fullsized":"2024-09-17T07:03:20.021388+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:02:17.002904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:40.784492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"75'9","reported_seq":56,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-17T07:03:26.005217+0000","last_change":"2024-09-17T07:03:26.005290+0000","last_active":"2024-09-17T07:03:26.005217+0000","last_peered":"2024-09-17T07:03:26.005217+0000","last_clean":"2024-09-17T07:03:26.005217+0000","last_became_active":"2024-09-17T07:02:14.029388+0000","last_became_peered":"2024-09-17T07:02:14.029388+0000","last_unstale":"2024-09-17T07:03:26.005217+0000","last_undegraded":"2024-09-17T07:03:26.005217+0000","last_fullsized":"2024-09-17T07:03:26.005217+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:03:26.005201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:26.005201+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:55.043365+0000","scrub_duration":0.0018022279999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":42,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-17T07:03:31.753486+0000","last_change":"2024-09-17T07:03:31.753577+0000","last_active":"2024-09-17T07:03:31.753486+0000","last_peered":"2024-09-17T07:03:31.753486+0000","last_clean":"2024-09-17T07:03:31.753486+0000","last_became_active":"2024-09-17T07:02:12.011327+0000","last_became_peered":"2024-09-17T07:02:12.011327+0000","last_unstale":"2024-09-17T07:03:31.753486+0000","last_undegraded":"2024-09-17T07:03:31.753486+0000","last_fullsized":"2024-09-17T07:03:31.753486+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:31.753470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:31.753470+0000","last_clean_scrub_stamp":"2024-09-17T07:03:31.753470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:51.025434+0000","scrub_duration":0.0012687670000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":37,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:21.314310+0000","last_change":"2024-09-17T07:03:21.314310+0000","last_active":"2024-09-17T07:03:21.314310+0000","last_peered":"2024-09-17T07:03:21.314310+0000","last_clean":"2024-09-17T07:03:21.314310+0000","last_became_active":"2024-09-17T07:02:16.022584+0000","last_became_peered":"2024-09-17T07:02:16.022584+0000","last_unstale":"2024-09-17T07:03:21.314310+0000","last_undegraded":"2024-09-17T07:03:21.314310+0000","last_fullsized":"2024-09-17T07:03:21.314310+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:21.314226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:21.314226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:29.256168+0000","scrub_duration":0.0016627219999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":36,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.745467+0000","last_change":"2024-09-17T07:03:19.603974+0000","last_active":"2024-09-17T07:03:24.745467+0000","last_peered":"2024-09-17T07:03:24.745467+0000","last_clean":"2024-09-17T07:03:24.745467+0000","last_became_active":"2024-09-17T07:02:18.017825+0000","last_became_peered":"2024-09-17T07:02:18.017825+0000","last_unstale":"2024-09-17T07:03:24.745467+0000","last_undegraded":"2024-09-17T07:03:24.745467+0000","last_fullsized":"2024-09-17T07:03:24.745467+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:19.603899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:03:19.603899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:31.400546+0000","scrub_duration":0.001425798,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.115085+0000","last_change":"2024-09-17T07:02:12.008326+0000","last_active":"2024-09-17T07:03:20.115085+0000","last_peered":"2024-09-17T07:03:20.115085+0000","last_clean":"2024-09-17T07:03:20.115085+0000","last_became_active":"2024-09-17T07:02:12.008065+0000","last_became_peered":"2024-09-17T07:02:12.008065+0000","last_unstale":"2024-09-17T07:03:20.115085+0000","last_undegraded":"2024-09-17T07:03:20.115085+0000","last_fullsized":"2024-09-17T07:03:20.115085+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:02:10.991363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:40.680888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"75'6","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-17T07:03:36.430605+0000","last_change":"2024-09-17T07:03:36.430683+0000","last_active":"2024-09-17T07:03:36.430605+0000","last_peered":"2024-09-17T07:03:36.430605+0000","last_clean":"2024-09-17T07:03:36.430605+0000","last_became_active":"2024-09-17T07:02:14.028655+0000","last_became_peered":"2024-09-17T07:02:14.028655+0000","last_unstale":"2024-09-17T07:03:36.430605+0000","last_undegraded":"2024-09-17T07:03:36.430605+0000","last_fullsized":"2024-09-17T07:03:36.430605+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'6","last_scrub_stamp":"2024-09-17T07:03:36.430591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:36.430591+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:44.221735+0000","scrub_duration":0.0020997730000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.536952+0000","last_change":"2024-09-17T07:02:16.016899+0000","last_active":"2024-09-17T07:03:20.536952+0000","last_peered":"2024-09-17T07:03:20.536952+0000","last_clean":"2024-09-17T07:03:20.536952+0000","last_became_active":"2024-09-17T07:02:16.016634+0000","last_became_peered":"2024-09-17T07:02:16.016634+0000","last_unstale":"2024-09-17T07:03:20.536952+0000","last_undegraded":"2024-09-17T07:03:20.536952+0000","last_fullsized":"2024-09-17T07:03:20.536952+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:02:14.999435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:26.661198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.956095+0000","last_change":"2024-09-17T07:02:18.365703+0000","last_active":"2024-09-17T07:03:24.956095+0000","last_peered":"2024-09-17T07:03:24.956095+0000","last_clean":"2024-09-17T07:03:24.956095+0000","last_became_active":"2024-09-17T07:02:18.365419+0000","last_became_peered":"2024-09-17T07:02:18.365419+0000","last_unstale":"2024-09-17T07:03:24.956095+0000","last_undegraded":"2024-09-17T07:03:24.956095+0000","last_fullsized":"2024-09-17T07:03:24.956095+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:02:17.002904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:28.424716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"75'10","reported_seq":54,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.021872+0000","last_change":"2024-09-17T07:03:14.178469+0000","last_active":"2024-09-17T07:03:20.021872+0000","last_peered":"2024-09-17T07:03:20.021872+0000","last_clean":"2024-09-17T07:03:20.021872+0000","last_became_active":"2024-09-17T07:02:14.027499+0000","last_became_peered":"2024-09-17T07:02:14.027499+0000","last_unstale":"2024-09-17T07:03:20.021872+0000","last_undegraded":"2024-09-17T07:03:20.021872+0000","last_fullsized":"2024-09-17T07:03:20.021872+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-09-17T07:03:14.178375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:14.178375+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:31.400568+0000","scrub_duration":0.0021187430000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":41,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-17T07:03:25.849723+0000","last_change":"2024-09-17T07:03:25.849841+0000","last_active":"2024-09-17T07:03:25.849723+0000","last_peered":"2024-09-17T07:03:25.849723+0000","last_clean":"2024-09-17T07:03:25.849723+0000","last_became_active":"2024-09-17T07:02:12.007280+0000","last_became_peered":"2024-09-17T07:02:12.007280+0000","last_unstale":"2024-09-17T07:03:25.849723+0000","last_undegraded":"2024-09-17T07:03:25.849723+0000","last_fullsized":"2024-09-17T07:03:25.849723+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:25.849698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:25.849698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:55.197183+0000","scrub_duration":0.0014293929999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"75'8","reported_seq":51,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-17T07:03:30.534720+0000","last_change":"2024-09-17T07:02:16.022883+0000","last_active":"2024-09-17T07:03:30.534720+0000","last_peered":"2024-09-17T07:03:30.534720+0000","last_clean":"2024-09-17T07:03:30.534720+0000","last_became_active":"2024-09-17T07:02:16.022583+0000","last_became_peered":"2024-09-17T07:02:16.022583+0000","last_unstale":"2024-09-17T07:03:30.534720+0000","last_undegraded":"2024-09-17T07:03:30.534720+0000","last_fullsized":"2024-09-17T07:03:30.534720+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:02:14.999435+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.746352+0000","last_change":"2024-09-17T07:02:18.029691+0000","last_active":"2024-09-17T07:03:24.746352+0000","last_peered":"2024-09-17T07:03:24.746352+0000","last_clean":"2024-09-17T07:03:24.746352+0000","last_became_active":"2024-09-17T07:02:18.028796+0000","last_became_peered":"2024-09-17T07:02:18.028796+0000","last_unstale":"2024-09-17T07:03:24.746352+0000","last_undegraded":"2024-09-17T07:03:24.746352+0000","last_fullsized":"2024-09-17T07:03:24.746352+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:02:17.002904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:40.761948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"75'9","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.536820+0000","last_change":"2024-09-17T07:03:17.798712+0000","last_active":"2024-09-17T07:03:20.536820+0000","last_peered":"2024-09-17T07:03:20.536820+0000","last_clean":"2024-09-17T07:03:20.536820+0000","last_became_active":"2024-09-17T07:02:14.028485+0000","last_became_peered":"2024-09-17T07:02:14.028485+0000","last_unstale":"2024-09-17T07:03:20.536820+0000","last_undegraded":"2024-09-17T07:03:20.536820+0000","last_fullsized":"2024-09-17T07:03:20.536820+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:03:17.798582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:17.798582+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:27.282186+0000","scrub_duration":0.0023213259999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":42,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.746635+0000","last_change":"2024-09-17T07:03:16.948419+0000","last_active":"2024-09-17T07:03:24.746635+0000","last_peered":"2024-09-17T07:03:24.746635+0000","last_clean":"2024-09-17T07:03:24.746635+0000","last_became_active":"2024-09-17T07:02:12.009935+0000","last_became_peered":"2024-09-17T07:02:12.009935+0000","last_unstale":"2024-09-17T07:03:24.746635+0000","last_undegraded":"2024-09-17T07:03:24.746635+0000","last_fullsized":"2024-09-17T07:03:24.746635+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:16.948330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:16.948330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:46.609922+0000","scrub_duration":0.0017835489999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"75'8","reported_seq":51,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-17T07:03:30.534967+0000","last_change":"2024-09-17T07:02:16.016395+0000","last_active":"2024-09-17T07:03:30.534967+0000","last_peered":"2024-09-17T07:03:30.534967+0000","last_clean":"2024-09-17T07:03:30.534967+0000","last_became_active":"2024-09-17T07:02:16.016069+0000","last_became_peered":"2024-09-17T07:02:16.016069+0000","last_unstale":"2024-09-17T07:03:30.534967+0000","last_undegraded":"2024-09-17T07:03:30.534967+0000","last_fullsized":"2024-09-17T07:03:30.534967+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:02:14.999435+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":36,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-17T07:03:31.045183+0000","last_change":"2024-09-17T07:03:31.045257+0000","last_active":"2024-09-17T07:03:31.045183+0000","last_peered":"2024-09-17T07:03:31.045183+0000","last_clean":"2024-09-17T07:03:31.045183+0000","last_became_active":"2024-09-17T07:02:18.018716+0000","last_became_peered":"2024-09-17T07:02:18.018716+0000","last_unstale":"2024-09-17T07:03:31.045183+0000","last_undegraded":"2024-09-17T07:03:31.045183+0000","last_fullsized":"2024-09-17T07:03:31.045183+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:31.045167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:03:31.045167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:36.218836+0000","scrub_duration":0.0012599709999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"75'9","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.116712+0000","last_change":"2024-09-17T07:03:20.116712+0000","last_active":"2024-09-17T07:03:20.116712+0000","last_peered":"2024-09-17T07:03:20.116712+0000","last_clean":"2024-09-17T07:03:20.116712+0000","last_became_active":"2024-09-17T07:02:14.010289+0000","last_became_peered":"2024-09-17T07:02:14.010289+0000","last_unstale":"2024-09-17T07:03:20.116712+0000","last_undegraded":"2024-09-17T07:03:20.116712+0000","last_fullsized":"2024-09-17T07:03:20.116712+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:03:20.116602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:20.116602+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:35.267610+0000","scrub_duration":0.0032469130000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.115284+0000","last_change":"2024-09-17T07:02:12.007881+0000","last_active":"2024-09-17T07:03:20.115284+0000","last_peered":"2024-09-17T07:03:20.115284+0000","last_clean":"2024-09-17T07:03:20.115284+0000","last_became_active":"2024-09-17T07:02:12.007433+0000","last_became_peered":"2024-09-17T07:02:12.007433+0000","last_unstale":"2024-09-17T07:03:20.115284+0000","last_undegraded":"2024-09-17T07:03:20.115284+0000","last_fullsized":"2024-09-17T07:03:20.115284+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:02:10.991363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:36.966808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.021413+0000","last_change":"2024-09-17T07:02:16.016504+0000","last_active":"2024-09-17T07:03:20.021413+0000","last_peered":"2024-09-17T07:03:20.021413+0000","last_clean":"2024-09-17T07:03:20.021413+0000","last_became_active":"2024-09-17T07:02:16.016216+0000","last_became_peered":"2024-09-17T07:02:16.016216+0000","last_unstale":"2024-09-17T07:03:20.021413+0000","last_undegraded":"2024-09-17T07:03:20.021413+0000","last_fullsized":"2024-09-17T07:03:20.021413+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:02:14.999435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:37.372655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":36,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.745931+0000","last_change":"2024-09-17T07:03:24.014436+0000","last_active":"2024-09-17T07:03:24.745931+0000","last_peered":"2024-09-17T07:03:24.745931+0000","last_clean":"2024-09-17T07:03:24.745931+0000","last_became_active":"2024-09-17T07:02:18.027622+0000","last_became_peered":"2024-09-17T07:02:18.027622+0000","last_unstale":"2024-09-17T07:03:24.745931+0000","last_undegraded":"2024-09-17T07:03:24.745931+0000","last_fullsized":"2024-09-17T07:03:24.745931+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:24.014336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:03:24.014336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:50.998219+0000","scrub_duration":0.001425125,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"75'11","reported_seq":59,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.744332+0000","last_change":"2024-09-17T07:03:18.899916+0000","last_active":"2024-09-17T07:03:24.744332+0000","last_peered":"2024-09-17T07:03:24.744332+0000","last_clean":"2024-09-17T07:03:24.744332+0000","last_became_active":"2024-09-17T07:02:14.016255+0000","last_became_peered":"2024-09-17T07:02:14.016255+0000","last_unstale":"2024-09-17T07:03:24.744332+0000","last_undegraded":"2024-09-17T07:03:24.744332+0000","last_fullsized":"2024-09-17T07:03:24.744332+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-09-17T07:03:18.899814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:18.899814+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:46.161729+0000","scrub_duration":0.0020491419999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"68'1","reported_seq":35,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.021771+0000","last_change":"2024-09-17T07:02:12.011847+0000","last_active":"2024-09-17T07:03:20.021771+0000","last_peered":"2024-09-17T07:03:20.021771+0000","last_clean":"2024-09-17T07:03:20.021771+0000","last_became_active":"2024-09-17T07:02:12.011461+0000","last_became_peered":"2024-09-17T07:02:12.011461+0000","last_unstale":"2024-09-17T07:03:20.021771+0000","last_undegraded":"2024-09-17T07:03:20.021771+0000","last_fullsized":"2024-09-17T07:03:20.021771+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:02:10.991363+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:37.333028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":37,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-17T07:03:33.125929+0000","last_change":"2024-09-17T07:03:33.126060+0000","last_active":"2024-09-17T07:03:33.125929+0000","last_peered":"2024-09-17T07:03:33.125929+0000","last_clean":"2024-09-17T07:03:33.125929+0000","last_became_active":"2024-09-17T07:02:16.018754+0000","last_became_peered":"2024-09-17T07:02:16.018754+0000","last_unstale":"2024-09-17T07:03:33.125929+0000","last_undegraded":"2024-09-17T07:03:33.125929+0000","last_fullsized":"2024-09-17T07:03:33.125929+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:33.125915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:33.125915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:56.098551+0000","scrub_duration":0.033614225999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.020820+0000","last_change":"2024-09-17T07:02:18.027727+0000","last_active":"2024-09-17T07:03:20.020820+0000","last_peered":"2024-09-17T07:03:20.020820+0000","last_clean":"2024-09-17T07:03:20.020820+0000","last_became_active":"2024-09-17T07:02:18.027427+0000","last_became_peered":"2024-09-17T07:02:18.027427+0000","last_unstale":"2024-09-17T07:03:20.020820+0000","last_undegraded":"2024-09-17T07:03:20.020820+0000","last_fullsized":"2024-09-17T07:03:20.020820+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:02:17.002904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:41.608211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"75'4","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.431130+0000","last_change":"2024-09-17T07:03:24.431130+0000","last_active":"2024-09-17T07:03:24.431130+0000","last_peered":"2024-09-17T07:03:24.431130+0000","last_clean":"2024-09-17T07:03:24.431130+0000","last_became_active":"2024-09-17T07:02:14.014779+0000","last_became_peered":"2024-09-17T07:02:14.014779+0000","last_unstale":"2024-09-17T07:03:24.431130+0000","last_undegraded":"2024-09-17T07:03:24.431130+0000","last_fullsized":"2024-09-17T07:03:24.431130+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'4","last_scrub_stamp":"2024-09-17T07:03:24.431043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:24.431043+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:37.473735+0000","scrub_duration":0.0015094500000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.537004+0000","last_change":"2024-09-17T07:02:12.009186+0000","last_active":"2024-09-17T07:03:20.537004+0000","last_peered":"2024-09-17T07:03:20.537004+0000","last_clean":"2024-09-17T07:03:20.537004+0000","last_became_active":"2024-09-17T07:02:12.008898+0000","last_became_peered":"2024-09-17T07:02:12.008898+0000","last_unstale":"2024-09-17T07:03:20.537004+0000","last_undegraded":"2024-09-17T07:03:20.537004+0000","last_fullsized":"2024-09-17T07:03:20.537004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:02:10.991363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:30.703927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.114909+0000","last_change":"2024-09-17T07:02:16.015248+0000","last_active":"2024-09-17T07:03:20.114909+0000","last_peered":"2024-09-17T07:03:20.114909+0000","last_clean":"2024-09-17T07:03:20.114909+0000","last_became_active":"2024-09-17T07:02:16.014982+0000","last_became_peered":"2024-09-17T07:02:16.014982+0000","last_unstale":"2024-09-17T07:03:20.114909+0000","last_undegraded":"2024-09-17T07:03:20.114909+0000","last_fullsized":"2024-09-17T07:03:20.114909+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:02:14.999435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:34.693466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.955631+0000","last_change":"2024-09-17T07:02:18.021319+0000","last_active":"2024-09-17T07:03:24.955631+0000","last_peered":"2024-09-17T07:03:24.955631+0000","last_clean":"2024-09-17T07:03:24.955631+0000","last_became_active":"2024-09-17T07:02:18.021014+0000","last_became_peered":"2024-09-17T07:02:18.021014+0000","last_unstale":"2024-09-17T07:03:24.955631+0000","last_undegraded":"2024-09-17T07:03:24.955631+0000","last_fullsized":"2024-09-17T07:03:24.955631+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:02:17.002904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:40.046947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"75'11","reported_seq":58,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.021003+0000","last_change":"2024-09-17T07:03:17.270057+0000","last_active":"2024-09-17T07:03:20.021003+0000","last_peered":"2024-09-17T07:03:20.021003+0000","last_clean":"2024-09-17T07:03:20.021003+0000","last_became_active":"2024-09-17T07:02:14.027316+0000","last_became_peered":"2024-09-17T07:02:14.027316+0000","last_unstale":"2024-09-17T07:03:20.021003+0000","last_undegraded":"2024-09-17T07:03:20.021003+0000","last_fullsized":"2024-09-17T07:03:20.021003+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-09-17T07:03:17.269941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:17.269941+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:32.856982+0000","scrub_duration":0.002326505,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":41,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-17T07:03:33.533126+0000","last_change":"2024-09-17T07:03:33.533200+0000","last_active":"2024-09-17T07:03:33.533126+0000","last_peered":"2024-09-17T07:03:33.533126+0000","last_clean":"2024-09-17T07:03:33.533126+0000","last_became_active":"2024-09-17T07:02:12.010195+0000","last_became_peered":"2024-09-17T07:02:12.010195+0000","last_unstale":"2024-09-17T07:03:33.533126+0000","last_undegraded":"2024-09-17T07:03:33.533126+0000","last_fullsized":"2024-09-17T07:03:33.533126+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:33.533112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:33.533112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:53.826577+0000","scrub_duration":0.0014453420000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-17T07:03:37.063425+0000","last_change":"2024-09-17T07:03:37.063510+0000","last_active":"2024-09-17T07:03:37.063425+0000","last_peered":"2024-09-17T07:03:37.063425+0000","last_clean":"2024-09-17T07:03:37.063425+0000","last_became_active":"2024-09-17T07:02:16.016183+0000","last_became_peered":"2024-09-17T07:02:16.016183+0000","last_unstale":"2024-09-17T07:03:37.063425+0000","last_undegraded":"2024-09-17T07:03:37.063425+0000","last_fullsized":"2024-09-17T07:03:37.063425+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:37.063409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:37.063409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:55.794307+0000","scrub_duration":0.0015220329999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":35,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-17T07:03:29.467266+0000","last_change":"2024-09-17T07:03:29.467351+0000","last_active":"2024-09-17T07:03:29.467266+0000","last_peered":"2024-09-17T07:03:29.467266+0000","last_clean":"2024-09-17T07:03:29.467266+0000","last_became_active":"2024-09-17T07:02:18.365656+0000","last_became_peered":"2024-09-17T07:02:18.365656+0000","last_unstale":"2024-09-17T07:03:29.467266+0000","last_undegraded":"2024-09-17T07:03:29.467266+0000","last_fullsized":"2024-09-17T07:03:29.467266+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:29.467252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:03:29.467252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:58.741564+0000","scrub_duration":0.001842804,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"75'1","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.956029+0000","last_change":"2024-09-17T07:02:18.022137+0000","last_active":"2024-09-17T07:03:24.956029+0000","last_peered":"2024-09-17T07:03:24.956029+0000","last_clean":"2024-09-17T07:03:24.956029+0000","last_became_active":"2024-09-17T07:02:18.021855+0000","last_became_peered":"2024-09-17T07:02:18.021855+0000","last_unstale":"2024-09-17T07:03:24.956029+0000","last_undegraded":"2024-09-17T07:03:24.956029+0000","last_fullsized":"2024-09-17T07:03:24.956029+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:02:17.002904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:39.620232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.536263+0000","last_change":"2024-09-17T07:03:19.775506+0000","last_active":"2024-09-17T07:03:20.536263+0000","last_peered":"2024-09-17T07:03:20.536263+0000","last_clean":"2024-09-17T07:03:20.536263+0000","last_became_active":"2024-09-17T07:02:12.005329+0000","last_became_peered":"2024-09-17T07:02:12.005329+0000","last_unstale":"2024-09-17T07:03:20.536263+0000","last_undegraded":"2024-09-17T07:03:20.536263+0000","last_fullsized":"2024-09-17T07:03:20.536263+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:19.775390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:19.775390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:48.303892+0000","scrub_duration":0.0011915630000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"75'10","reported_seq":44,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.746440+0000","last_change":"2024-09-17T07:02:14.029020+0000","last_active":"2024-09-17T07:03:24.746440+0000","last_peered":"2024-09-17T07:03:24.746440+0000","last_clean":"2024-09-17T07:03:24.746440+0000","last_became_active":"2024-09-17T07:02:14.028731+0000","last_became_peered":"2024-09-17T07:02:14.028731+0000","last_unstale":"2024-09-17T07:03:24.746440+0000","last_undegraded":"2024-09-17T07:03:24.746440+0000","last_fullsized":"2024-09-17T07:03:24.746440+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:02:12.994221+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:42.804791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"75'8","reported_seq":54,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-17T07:03:30.534767+0000","last_change":"2024-09-17T07:02:16.020978+0000","last_active":"2024-09-17T07:03:30.534767+0000","last_peered":"2024-09-17T07:03:30.534767+0000","last_clean":"2024-09-17T07:03:30.534767+0000","last_became_active":"2024-09-17T07:02:16.020560+0000","last_became_peered":"2024-09-17T07:02:16.020560+0000","last_unstale":"2024-09-17T07:03:30.534767+0000","last_undegraded":"2024-09-17T07:03:30.534767+0000","last_fullsized":"2024-09-17T07:03:30.534767+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:02:14.999435+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:42.145816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.021299+0000","last_change":"2024-09-17T07:02:18.021959+0000","last_active":"2024-09-17T07:03:20.021299+0000","last_peered":"2024-09-17T07:03:20.021299+0000","last_clean":"2024-09-17T07:03:20.021299+0000","last_became_active":"2024-09-17T07:02:18.021603+0000","last_became_peered":"2024-09-17T07:02:18.021603+0000","last_unstale":"2024-09-17T07:03:20.021299+0000","last_undegraded":"2024-09-17T07:03:20.021299+0000","last_fullsized":"2024-09-17T07:03:20.021299+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:02:17.002904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:33.551380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.305248+0000","last_change":"2024-09-17T07:03:20.305248+0000","last_active":"2024-09-17T07:03:20.305248+0000","last_peered":"2024-09-17T07:03:20.305248+0000","last_clean":"2024-09-17T07:03:20.305248+0000","last_became_active":"2024-09-17T07:02:12.009837+0000","last_became_peered":"2024-09-17T07:02:12.009837+0000","last_unstale":"2024-09-17T07:03:20.305248+0000","last_undegraded":"2024-09-17T07:03:20.305248+0000","last_fullsized":"2024-09-17T07:03:20.305248+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:20.305165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:20.305165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:24.412968+0000","scrub_duration":0.0024485969999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"75'11","reported_seq":58,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.473820+0000","last_change":"2024-09-17T07:03:24.473820+0000","last_active":"2024-09-17T07:03:24.473820+0000","last_peered":"2024-09-17T07:03:24.473820+0000","last_clean":"2024-09-17T07:03:24.473820+0000","last_became_active":"2024-09-17T07:02:14.011380+0000","last_became_peered":"2024-09-17T07:02:14.011380+0000","last_unstale":"2024-09-17T07:03:24.473820+0000","last_undegraded":"2024-09-17T07:03:24.473820+0000","last_fullsized":"2024-09-17T07:03:24.473820+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-09-17T07:03:24.473712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:24.473712+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:53.121209+0000","scrub_duration":0.0021033010000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.746403+0000","last_change":"2024-09-17T07:02:16.019989+0000","last_active":"2024-09-17T07:03:24.746403+0000","last_peered":"2024-09-17T07:03:24.746403+0000","last_clean":"2024-09-17T07:03:24.746403+0000","last_became_active":"2024-09-17T07:02:16.019701+0000","last_became_peered":"2024-09-17T07:02:16.019701+0000","last_unstale":"2024-09-17T07:03:24.746403+0000","last_undegraded":"2024-09-17T07:03:24.746403+0000","last_fullsized":"2024-09-17T07:03:24.746403+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:02:14.999435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:44.035860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.744767+0000","last_change":"2024-09-17T07:02:18.364013+0000","last_active":"2024-09-17T07:03:24.744767+0000","last_peered":"2024-09-17T07:03:24.744767+0000","last_clean":"2024-09-17T07:03:24.744767+0000","last_became_active":"2024-09-17T07:02:18.363752+0000","last_became_peered":"2024-09-17T07:02:18.363752+0000","last_unstale":"2024-09-17T07:03:24.744767+0000","last_undegraded":"2024-09-17T07:03:24.744767+0000","last_fullsized":"2024-09-17T07:03:24.744767+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:02:17.002904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:33.097191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.115297+0000","last_change":"2024-09-17T07:02:12.005383+0000","last_active":"2024-09-17T07:03:20.115297+0000","last_peered":"2024-09-17T07:03:20.115297+0000","last_clean":"2024-09-17T07:03:20.115297+0000","last_became_active":"2024-09-17T07:02:12.005149+0000","last_became_peered":"2024-09-17T07:02:12.005149+0000","last_unstale":"2024-09-17T07:03:20.115297+0000","last_undegraded":"2024-09-17T07:03:20.115297+0000","last_fullsized":"2024-09-17T07:03:20.115297+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:02:10.991363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:31.958573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"75'15","reported_seq":65,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T07:03:24.745594+0000","last_change":"2024-09-17T07:03:16.605256+0000","last_active":"2024-09-17T07:03:24.745594+0000","last_peered":"2024-09-17T07:03:24.745594+0000","last_clean":"2024-09-17T07:03:24.745594+0000","last_became_active":"2024-09-17T07:02:14.016273+0000","last_became_peered":"2024-09-17T07:02:14.016273+0000","last_unstale":"2024-09-17T07:03:24.745594+0000","last_undegraded":"2024-09-17T07:03:24.745594+0000","last_fullsized":"2024-09-17T07:03:24.745594+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-09-17T07:03:16.605163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:16.605163+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:40.470486+0000","scrub_duration":0.002000763,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.021640+0000","last_change":"2024-09-17T07:02:16.017339+0000","last_active":"2024-09-17T07:03:20.021640+0000","last_peered":"2024-09-17T07:03:20.021640+0000","last_clean":"2024-09-17T07:03:20.021640+0000","last_became_active":"2024-09-17T07:02:16.015903+0000","last_became_peered":"2024-09-17T07:02:16.015903+0000","last_unstale":"2024-09-17T07:03:20.021640+0000","last_undegraded":"2024-09-17T07:03:20.021640+0000","last_fullsized":"2024-09-17T07:03:20.021640+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:02:14.999435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.020900+0000","last_change":"2024-09-17T07:02:18.366797+0000","last_active":"2024-09-17T07:03:20.020900+0000","last_peered":"2024-09-17T07:03:20.020900+0000","last_clean":"2024-09-17T07:03:20.020900+0000","last_became_active":"2024-09-17T07:02:18.366466+0000","last_became_peered":"2024-09-17T07:02:18.366466+0000","last_unstale":"2024-09-17T07:03:20.020900+0000","last_undegraded":"2024-09-17T07:03:20.020900+0000","last_fullsized":"2024-09-17T07:03:20.020900+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:02:17.002904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:46.082346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.021521+0000","last_change":"2024-09-17T07:02:12.005764+0000","last_active":"2024-09-17T07:03:20.021521+0000","last_peered":"2024-09-17T07:03:20.021521+0000","last_clean":"2024-09-17T07:03:20.021521+0000","last_became_active":"2024-09-17T07:02:12.005493+0000","last_became_peered":"2024-09-17T07:02:12.005493+0000","last_unstale":"2024-09-17T07:03:20.021521+0000","last_undegraded":"2024-09-17T07:03:20.021521+0000","last_fullsized":"2024-09-17T07:03:20.021521+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:02:10.991363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:35.124183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"86'17","reported_seq":67,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-17T07:03:20.021814+0000","last_change":"2024-09-17T07:03:18.241178+0000","last_active":"2024-09-17T07:03:20.021814+0000","last_peered":"2024-09-17T07:03:20.021814+0000","last_clean":"2024-09-17T07:03:20.021814+0000","last_became_active":"2024-09-17T07:02:14.017737+0000","last_became_peered":"2024-09-17T07:02:14.017737+0000","last_unstale":"2024-09-17T07:03:20.021814+0000","last_undegraded":"2024-09-17T07:03:20.021814+0000","last_fullsized":"2024-09-17T07:03:20.021814+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'17","last_scrub_stamp":"2024-09-17T07:03:18.241067+0000","last_deep_scrub":"86'17","last_deep_scrub_stamp":"2024-09-17T07:03:18.241067+0000","last_clean_scrub_stamp":"2024-09-17T07:03:18.241067+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:35.855446+0000","scrub_duration":0.0047694219999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-17T07:03:26.978275+0000","last_change":"2024-09-17T07:03:26.978352+0000","last_active":"2024-09-17T07:03:26.978275+0000","last_peered":"2024-09-17T07:03:26.978275+0000","last_clean":"2024-09-17T07:03:26.978275+0000","last_became_active":"2024-09-17T07:02:16.017362+0000","last_became_peered":"2024-09-17T07:02:16.017362+0000","last_unstale":"2024-09-17T07:03:26.978275+0000","last_undegraded":"2024-09-17T07:03:26.978275+0000","last_fullsized":"2024-09-17T07:03:26.978275+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:26.978262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:26.978262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:38.363709+0000","scrub_duration":0.0016151990000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":396,"ondisk_log_size":396,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":79,"num_read_kb":74,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1386,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":12,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":4158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037783,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32456,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93712696,"statfs":{"total":95995035648,"available":95961800704,"internally_reserved":0,"allocated":1376256,"data_stored":936056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:03:02 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.373},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.47499999999999998}]},{"osd":1,"last update":"Tue Sep 17 07:03:02 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.435},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.41099999999999998}]},{"osd":2,"last update":"Tue Sep 17 07:03:02 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.622,"5min":0.622,"15min":0.622},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.40200000000000002}]},{"osd":3,"last update":"Tue Sep 17 07:03:02 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.501},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.34699999999999998}]},{"osd":4,"last update":"Tue Sep 17 07:03:02 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"last":0.38700000000000001}]},{"osd":5,"last update":"Tue Sep 17 07:03:02 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.44800000000000001}]},{"osd":6,"last update":"Tue Sep 17 07:03:02 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.41999999999999998}]}]},{"osd":6,"up_from":50,"seq":214748364826,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32440,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93712712,"statfs":{"total":95995035648,"available":95961817088,"internally_reserved":0,"allocated":1359872,"data_stored":935371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:02:41 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.42199999999999999}]},{"osd":1,"last update":"Tue Sep 17 07:02:41 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.45700000000000002}]},{"osd":2,"last update":"Tue Sep 17 07:02:41 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.55500000000000005}]},{"osd":3,"last update":"Tue Sep 17 07:02:41 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.40600000000000003}]},{"osd":4,"last update":"Tue Sep 17 07:02:41 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.49399999999999999}]},{"osd":5,"last update":"Tue Sep 17 07:02:41 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"last":0.438}]},{"osd":7,"last update":"Tue Sep 17 07:03:03 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.51000000000000001}]}]},{"osd":5,"up_from":42,"seq":180388626463,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31864,"kb_used_data":752,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713288,"statfs":{"total":95995035648,"available":95962406912,"internally_reserved":0,"allocated":770048,"data_stored":345813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:03:15 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.50600000000000001,"15min":0.51500000000000001},"min":{"1min":0.34100000000000003,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.56699999999999995,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.497,"15min":0.505},"min":{"1min":0.31900000000000001,"5min":0.313,"15min":0.313},"max":{"1min":0.65500000000000003,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.51800000000000002}]},{"osd":1,"last update":"Tue Sep 17 07:03:15 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.53200000000000003,"15min":0.54200000000000004},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.60799999999999998,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.443,"5min":1.0449999999999999,"15min":1.145},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.56200000000000006,"5min":14.054,"15min":14.054},"last":0.47399999999999998}]},{"osd":2,"last update":"Tue Sep 17 07:03:15 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.54800000000000004,"15min":0.56000000000000005},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.66500000000000004,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.505,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.38600000000000001,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.63600000000000001,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.44400000000000001}]},{"osd":3,"last update":"Tue Sep 17 07:03:15 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.54000000000000004,"15min":0.54900000000000004},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.63800000000000001,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.496,"5min":0.53900000000000003,"15min":0.54600000000000004},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.66400000000000003,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.48399999999999999}]},{"osd":4,"last update":"Tue Sep 17 07:03:15 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.40000000000000002,"15min":0.39900000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"last":0.37},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.432,"15min":0.434},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.60499999999999998,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.41799999999999998}]},{"osd":6,"last update":"Tue Sep 17 07:02:37 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.49399999999999999}]},{"osd":7,"last update":"Tue Sep 17 07:03:01 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.53200000000000003}]}]},{"osd":4,"up_from":33,"seq":141733920805,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31880,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713272,"statfs":{"total":95995035648,"available":95962390528,"internally_reserved":0,"allocated":786432,"data_stored":345400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:02:53 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.56299999999999994,"15min":0.57499999999999996},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.71499999999999997,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.60199999999999998,"15min":0.61499999999999999},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.69799999999999995,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.51500000000000001}]},{"osd":1,"last update":"Tue Sep 17 07:02:53 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.56000000000000005,"15min":0.57199999999999995},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.68400000000000005,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.495,"5min":0.57999999999999996,"15min":0.59399999999999997},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.67900000000000005,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.45100000000000001}]},{"osd":2,"last update":"Tue Sep 17 07:02:53 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.56899999999999995,"15min":0.58299999999999996},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.69999999999999996,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.57799999999999996,"15min":0.58799999999999997},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.77100000000000002,"5min":0.877,"15min":0.877},"last":0.47699999999999998}]},{"osd":3,"last update":"Tue Sep 17 07:02:53 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.59299999999999997,"15min":0.60799999999999998},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.74299999999999999,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.55900000000000005,"15min":0.56399999999999995},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.66800000000000004,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.55100000000000005}]},{"osd":5,"last update":"Tue Sep 17 07:03:19 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.39600000000000002,"15min":0.38600000000000001},"min":{"1min":0.316,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.41699999999999998,"15min":0.40699999999999997},"min":{"1min":0.34799999999999998,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.61299999999999999}]},{"osd":6,"last update":"Tue Sep 17 07:02:37 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.53200000000000003}]},{"osd":7,"last update":"Tue Sep 17 07:03:00 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.628}]}]},{"osd":3,"up_from":27,"seq":115964117031,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31904,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713248,"statfs":{"total":95995035648,"available":95962365952,"internally_reserved":0,"allocated":811008,"data_stored":346184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:02:39 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.36499999999999999,"15min":0.36399999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.38100000000000001,"15min":0.379},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.56599999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"last":0.35999999999999999}]},{"osd":1,"last update":"Tue Sep 17 07:02:39 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.40400000000000003,"15min":0.40699999999999997},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.58299999999999996,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.442,"5min":0.45000000000000001,"15min":0.45100000000000001},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.63700000000000001,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.34499999999999997}]},{"osd":2,"last update":"Tue Sep 17 07:02:39 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.42099999999999999,"15min":0.42299999999999999},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.307},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.41899999999999998,"15min":0.41999999999999998},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.42099999999999999}]},{"osd":4,"last update":"Tue Sep 17 07:02:56 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.56000000000000005,"15min":0.56399999999999995},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.754,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.53100000000000003,"15min":0.53500000000000003},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.70099999999999996}]},{"osd":5,"last update":"Tue Sep 17 07:03:18 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.59799999999999998,"15min":0.60599999999999998},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.503},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.58299999999999996,"15min":0.58499999999999996},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.79100000000000004,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.60299999999999998}]},{"osd":6,"last update":"Tue Sep 17 07:02:39 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.47699999999999998}]},{"osd":7,"last update":"Tue Sep 17 07:03:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.73699999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411373,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31868,"kb_used_data":756,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713284,"statfs":{"total":95995035648,"available":95962402816,"internally_reserved":0,"allocated":774144,"data_stored":345823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:03:15 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.35499999999999998,"15min":0.34699999999999998},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.253},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.35899999999999999,"15min":0.35099999999999998},"min":{"1min":0.24399999999999999,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.50600000000000001,"5min":0.625,"15min":0.625},"last":0.30199999999999999}]},{"osd":1,"last update":"Tue Sep 17 07:03:15 2024","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.35499999999999998,"15min":0.35099999999999998},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.48999999999999999,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.38500000000000001,"15min":0.374},"min":{"1min":0.28599999999999998,"5min":0.251,"15min":0.251},"max":{"1min":0.55700000000000005,"5min":0.58399999999999996,"15min":0.58399999999999996},"last":0.28799999999999998}]},{"osd":3,"last update":"Tue Sep 17 07:02:35 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.44,"15min":0.438},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.433,"5min":0.42999999999999999,"15min":0.42999999999999999},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.35999999999999999}]},{"osd":4,"last update":"Tue Sep 17 07:02:52 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56499999999999995,"15min":0.56599999999999995},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58599999999999997,"15min":0.58699999999999997},"min":{"1min":0.42999999999999999,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.39900000000000002}]},{"osd":5,"last update":"Tue Sep 17 07:03:15 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.101,"5min":1.101,"15min":1.101},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.57799999999999996,"15min":0.58999999999999997},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.78200000000000003,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.40699999999999997}]},{"osd":6,"last update":"Tue Sep 17 07:02:39 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.52800000000000002}]},{"osd":7,"last update":"Tue Sep 17 07:03:03 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":3.9289999999999998,"5min":3.9289999999999998,"15min":3.9289999999999998},"last":0.51700000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574898,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27820,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717332,"statfs":{"total":95995035648,"available":95966547968,"internally_reserved":0,"allocated":823296,"data_stored":347435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:02:43 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.34699999999999998,"15min":0.33500000000000002},"min":{"1min":0.19400000000000001,"5min":0.193,"15min":0.193},"max":{"1min":0.47399999999999998,"5min":0.56000000000000005,"15min":0.56000000000000005},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.373,"15min":0.36299999999999999},"min":{"1min":0.23400000000000001,"5min":0.216,"15min":0.216},"max":{"1min":0.51500000000000001,"5min":0.622,"15min":0.622},"last":0.35899999999999999}]},{"osd":2,"last update":"Tue Sep 17 07:03:12 2024","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.39600000000000002,"15min":0.40100000000000002},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.53100000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.40600000000000003,"15min":0.41199999999999998},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.60199999999999998,"5min":0.625,"15min":0.625},"last":0.29799999999999999}]},{"osd":3,"last update":"Tue Sep 17 07:03:37 2024","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.441,"15min":0.45800000000000002},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.41599999999999998,"15min":0.438},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.54900000000000004,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.26500000000000001}]},{"osd":4,"last update":"Tue Sep 17 07:02:54 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.57699999999999996,"15min":0.57399999999999995},"min":{"1min":0.41199999999999998,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.56999999999999995,"15min":0.56799999999999995},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.71799999999999997,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.627}]},{"osd":5,"last update":"Tue Sep 17 07:03:19 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.621,"15min":0.58699999999999997},"min":{"1min":0.38700000000000001,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":6.0060000000000002,"5min":6.0060000000000002,"15min":6.0060000000000002},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.56299999999999994,"15min":0.56499999999999995},"min":{"1min":0.38200000000000001,"5min":0.375,"15min":0.375},"max":{"1min":0.752,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.49399999999999999}]},{"osd":6,"last update":"Tue Sep 17 07:02:40 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.58799999999999997}]},{"osd":7,"last update":"Tue Sep 17 07:03:01 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.72099999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738421,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28368,"kb_used_data":1352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716784,"statfs":{"total":95995035648,"available":95965986816,"internally_reserved":0,"allocated":1384448,"data_stored":936501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 17 07:02:47 2024","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.34499999999999997,"15min":0.32800000000000001},"min":{"1min":0.221,"5min":0.217,"15min":0.217},"max":{"1min":0.53000000000000003,"5min":0.58499999999999996,"15min":0.58499999999999996},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.35699999999999998,"15min":0.34000000000000002},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.56799999999999995,"5min":0.57599999999999996,"15min":0.57599999999999996},"last":0.26400000000000001}]},{"osd":2,"last update":"Tue Sep 17 07:03:15 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.38400000000000001,"15min":0.377},"min":{"1min":0.251,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.61099999999999999,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.31},{"interface":"front","average":{"1min":0.378,"5min":0.37,"15min":0.36299999999999999},"min":{"1min":0.26600000000000001,"5min":0.246,"15min":0.246},"max":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"last":0.32100000000000001}]},{"osd":3,"last update":"Tue Sep 17 07:02:39 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.434,"15min":0.44},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.59299999999999997,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.42899999999999999,"15min":0.435},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.58099999999999996,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.33300000000000002}]},{"osd":4,"last update":"Tue Sep 17 07:02:56 2024","interfaces":[{"interface":"back","average":{"1min":1.1319999999999999,"5min":0.67500000000000004,"15min":0.59899999999999998},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":12.573,"5min":12.573,"15min":12.573},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.442}]},{"osd":5,"last update":"Tue Sep 17 07:03:19 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.55500000000000005,"15min":0.55000000000000004},"min":{"1min":0.376,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.378},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.54800000000000004,"15min":0.54200000000000004},"min":{"1min":0.39700000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.46800000000000003}]},{"osd":6,"last update":"Tue Sep 17 07:02:39 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.40500000000000003}]},{"osd":7,"last update":"Tue Sep 17 07:03:01 2024","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.2010000000000001,"15min":1.2010000000000001},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":12.548,"5min":12.548,"15min":12.548},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.41699999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T07:03:41.586 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-17T07:03:41.586 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-17T07:03:41.587 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-09-17T07:03:41.587 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph health --format=json 2024-09-17T07:03:41.785 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:03:42.044 INFO:journalctl@ceph.node-exporter.b.smithi135.stdout:Sep 17 07:03:41 smithi135 podman[63838]: 2024-09-17 07:03:41.595574064 +0000 UTC m=+2.621652253 container init d10476c373fe3263de1f47ca8f18ac116bf0efe6e6e5f0464d666ce71533eb33 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-17T07:03:42.044 INFO:journalctl@ceph.node-exporter.b.smithi135.stdout:Sep 17 07:03:41 smithi135 podman[63838]: 2024-09-17 07:03:41.598437522 +0000 UTC m=+2.624515711 container start d10476c373fe3263de1f47ca8f18ac116bf0efe6e6e5f0464d666ce71533eb33 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-17T07:03:42.044 INFO:journalctl@ceph.node-exporter.b.smithi135.stdout:Sep 17 07:03:41 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-b[63915]: ts=2024-09-17T07:03:41.602Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-09-17T07:03:42.044 INFO:journalctl@ceph.node-exporter.b.smithi135.stdout:Sep 17 07:03:41 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-b[63915]: ts=2024-09-17T07:03:41.602Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-09-17T07:03:42.044 INFO:journalctl@ceph.node-exporter.b.smithi135.stdout:Sep 17 07:03:41 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-b[63915]: ts=2024-09-17T07:03:41.603Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-17T07:03:42.044 INFO:journalctl@ceph.node-exporter.b.smithi135.stdout:Sep 17 07:03:41 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-b[63915]: ts=2024-09-17T07:03:41.603Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-17T07:03:42.045 INFO:journalctl@ceph.node-exporter.b.smithi135.stdout:Sep 17 07:03:41 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-b[63915]: ts=2024-09-17T07:03:41.603Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-17T07:03:42.045 INFO:journalctl@ceph.node-exporter.b.smithi135.stdout:Sep 17 07:03:41 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-b[63915]: ts=2024-09-17T07:03:41.603Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-17T07:03:42.045 INFO:journalctl@ceph.node-exporter.b.smithi135.stdout:Sep 17 07:03:41 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-b[63915]: ts=2024-09-17T07:03:41.603Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-17T07:03:42.045 INFO:journalctl@ceph.node-exporter.b.smithi135.stdout:Sep 17 07:03:41 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-b[63915]: ts=2024-09-17T07:03:41.603Z caller=node_exporter.go:117 level=info collector=arp 2024-09-17T07:03:42.045 INFO:journalctl@ceph.node-exporter.b.smithi135.stdout:Sep 17 07:03:41 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-b[63915]: ts=2024-09-17T07:03:41.603Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-17T07:03:42.045 INFO:journalctl@ceph.node-exporter.b.smithi135.stdout:Sep 17 07:03:41 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-b[63915]: ts=2024-09-17T07:03:41.603Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-17T07:03:42.045 INFO:journalctl@ceph.node-exporter.b.smithi135.stdout:Sep 17 07:03:41 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-b[63915]: ts=2024-09-17T07:03:41.603Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-17T07:03:42.045 INFO:journalctl@ceph.node-exporter.b.smithi135.stdout:Sep 17 07:03:41 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-b[63915]: ts=2024-09-17T07:03:41.603Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-17T07:03:42.045 INFO:journalctl@ceph.node-exporter.b.smithi135.stdout:Sep 17 07:03:41 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-b[63915]: ts=2024-09-17T07:03:41.603Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-17T07:03:42.045 INFO:journalctl@ceph.node-exporter.b.smithi135.stdout:Sep 17 07:03:41 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-b[63915]: ts=2024-09-17T07:03:41.603Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-17T07:03:42.046 INFO:journalctl@ceph.node-exporter.b.smithi135.stdout:Sep 17 07:03:41 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-b[63915]: ts=2024-09-17T07:03:41.603Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-17T07:03:42.046 INFO:journalctl@ceph.node-exporter.b.smithi135.stdout:Sep 17 07:03:41 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-b[63915]: ts=2024-09-17T07:03:41.603Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-17T07:03:42.046 INFO:journalctl@ceph.node-exporter.b.smithi135.stdout:Sep 17 07:03:41 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-b[63915]: ts=2024-09-17T07:03:41.603Z caller=node_exporter.go:117 level=info collector=edac 2024-09-17T07:03:42.046 INFO:journalctl@ceph.node-exporter.b.smithi135.stdout:Sep 17 07:03:41 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-b[63915]: ts=2024-09-17T07:03:41.603Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-17T07:03:42.046 INFO:journalctl@ceph.node-exporter.b.smithi135.stdout:Sep 17 07:03:41 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-b[63915]: ts=2024-09-17T07:03:41.603Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-17T07:03:42.046 INFO:journalctl@ceph.node-exporter.b.smithi135.stdout:Sep 17 07:03:41 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-b[63915]: ts=2024-09-17T07:03:41.603Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-17T07:03:42.046 INFO:journalctl@ceph.node-exporter.b.smithi135.stdout:Sep 17 07:03:41 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-b[63915]: ts=2024-09-17T07:03:41.603Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-17T07:03:42.046 INFO:journalctl@ceph.node-exporter.b.smithi135.stdout:Sep 17 07:03:41 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-b[63915]: ts=2024-09-17T07:03:41.603Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-17T07:03:42.047 INFO:journalctl@ceph.node-exporter.b.smithi135.stdout:Sep 17 07:03:41 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-b[63915]: ts=2024-09-17T07:03:41.603Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-17T07:03:42.047 INFO:journalctl@ceph.node-exporter.b.smithi135.stdout:Sep 17 07:03:41 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-b[63915]: ts=2024-09-17T07:03:41.603Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-17T07:03:42.047 INFO:journalctl@ceph.node-exporter.b.smithi135.stdout:Sep 17 07:03:41 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-b[63915]: ts=2024-09-17T07:03:41.603Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-17T07:03:42.047 INFO:journalctl@ceph.node-exporter.b.smithi135.stdout:Sep 17 07:03:41 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-b[63915]: ts=2024-09-17T07:03:41.603Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-17T07:03:42.047 INFO:journalctl@ceph.node-exporter.b.smithi135.stdout:Sep 17 07:03:41 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-b[63915]: ts=2024-09-17T07:03:41.603Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-17T07:03:42.047 INFO:journalctl@ceph.node-exporter.b.smithi135.stdout:Sep 17 07:03:41 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-b[63915]: ts=2024-09-17T07:03:41.603Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-17T07:03:42.047 INFO:journalctl@ceph.node-exporter.b.smithi135.stdout:Sep 17 07:03:41 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-b[63915]: ts=2024-09-17T07:03:41.603Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-17T07:03:42.047 INFO:journalctl@ceph.node-exporter.b.smithi135.stdout:Sep 17 07:03:41 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-b[63915]: ts=2024-09-17T07:03:41.603Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-17T07:03:42.047 INFO:journalctl@ceph.node-exporter.b.smithi135.stdout:Sep 17 07:03:41 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-b[63915]: ts=2024-09-17T07:03:41.603Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-17T07:03:42.047 INFO:journalctl@ceph.node-exporter.b.smithi135.stdout:Sep 17 07:03:41 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-b[63915]: ts=2024-09-17T07:03:41.603Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-17T07:03:42.047 INFO:journalctl@ceph.node-exporter.b.smithi135.stdout:Sep 17 07:03:41 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-b[63915]: ts=2024-09-17T07:03:41.603Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-17T07:03:42.048 INFO:journalctl@ceph.node-exporter.b.smithi135.stdout:Sep 17 07:03:41 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-b[63915]: ts=2024-09-17T07:03:41.603Z caller=node_exporter.go:117 level=info collector=os 2024-09-17T07:03:42.048 INFO:journalctl@ceph.node-exporter.b.smithi135.stdout:Sep 17 07:03:41 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-b[63915]: ts=2024-09-17T07:03:41.603Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-17T07:03:42.048 INFO:journalctl@ceph.node-exporter.b.smithi135.stdout:Sep 17 07:03:41 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-b[63915]: ts=2024-09-17T07:03:41.603Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-17T07:03:42.048 INFO:journalctl@ceph.node-exporter.b.smithi135.stdout:Sep 17 07:03:41 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-b[63915]: ts=2024-09-17T07:03:41.603Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-17T07:03:42.048 INFO:journalctl@ceph.node-exporter.b.smithi135.stdout:Sep 17 07:03:41 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-b[63915]: ts=2024-09-17T07:03:41.603Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-17T07:03:42.048 INFO:journalctl@ceph.node-exporter.b.smithi135.stdout:Sep 17 07:03:41 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-b[63915]: ts=2024-09-17T07:03:41.603Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-17T07:03:42.048 INFO:journalctl@ceph.node-exporter.b.smithi135.stdout:Sep 17 07:03:41 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-b[63915]: ts=2024-09-17T07:03:41.603Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-17T07:03:42.048 INFO:journalctl@ceph.node-exporter.b.smithi135.stdout:Sep 17 07:03:41 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-b[63915]: ts=2024-09-17T07:03:41.603Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-17T07:03:42.048 INFO:journalctl@ceph.node-exporter.b.smithi135.stdout:Sep 17 07:03:41 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-b[63915]: ts=2024-09-17T07:03:41.603Z caller=node_exporter.go:117 level=info collector=stat 2024-09-17T07:03:42.049 INFO:journalctl@ceph.node-exporter.b.smithi135.stdout:Sep 17 07:03:41 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-b[63915]: ts=2024-09-17T07:03:41.603Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-17T07:03:42.049 INFO:journalctl@ceph.node-exporter.b.smithi135.stdout:Sep 17 07:03:41 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-b[63915]: ts=2024-09-17T07:03:41.603Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-17T07:03:42.049 INFO:journalctl@ceph.node-exporter.b.smithi135.stdout:Sep 17 07:03:41 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-b[63915]: ts=2024-09-17T07:03:41.603Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-17T07:03:42.049 INFO:journalctl@ceph.node-exporter.b.smithi135.stdout:Sep 17 07:03:41 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-b[63915]: ts=2024-09-17T07:03:41.603Z caller=node_exporter.go:117 level=info collector=time 2024-09-17T07:03:42.049 INFO:journalctl@ceph.node-exporter.b.smithi135.stdout:Sep 17 07:03:41 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-b[63915]: ts=2024-09-17T07:03:41.603Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-17T07:03:42.049 INFO:journalctl@ceph.node-exporter.b.smithi135.stdout:Sep 17 07:03:41 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-b[63915]: ts=2024-09-17T07:03:41.603Z caller=node_exporter.go:117 level=info collector=uname 2024-09-17T07:03:42.049 INFO:journalctl@ceph.node-exporter.b.smithi135.stdout:Sep 17 07:03:41 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-b[63915]: ts=2024-09-17T07:03:41.603Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-17T07:03:42.049 INFO:journalctl@ceph.node-exporter.b.smithi135.stdout:Sep 17 07:03:41 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-b[63915]: ts=2024-09-17T07:03:41.603Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-17T07:03:42.049 INFO:journalctl@ceph.node-exporter.b.smithi135.stdout:Sep 17 07:03:41 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-b[63915]: ts=2024-09-17T07:03:41.603Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-17T07:03:42.049 INFO:journalctl@ceph.node-exporter.b.smithi135.stdout:Sep 17 07:03:41 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-b[63915]: ts=2024-09-17T07:03:41.604Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-09-17T07:03:42.049 INFO:journalctl@ceph.node-exporter.b.smithi135.stdout:Sep 17 07:03:41 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-b[63915]: ts=2024-09-17T07:03:41.604Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-17T07:03:42.049 INFO:journalctl@ceph.node-exporter.b.smithi135.stdout:Sep 17 07:03:41 smithi135 bash[63838]: d10476c373fe3263de1f47ca8f18ac116bf0efe6e6e5f0464d666ce71533eb33 2024-09-17T07:03:42.050 INFO:journalctl@ceph.node-exporter.b.smithi135.stdout:Sep 17 07:03:41 smithi135 systemd[1]: Started Ceph node-exporter.b for e848267c-74c1-11ef-bceb-c7b262605968. 2024-09-17T07:03:43.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:42 smithi135 ceph-mon[33238]: 6.15 scrub starts 2024-09-17T07:03:43.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:42 smithi135 ceph-mon[33238]: 6.15 scrub ok 2024-09-17T07:03:43.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:42 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:43.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:42 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:43.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:42 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:43.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:42 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:43.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:42 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:43.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:42 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:03:43.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:42 smithi135 ceph-mon[33238]: 2.1f scrub starts 2024-09-17T07:03:43.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:42 smithi135 ceph-mon[33238]: 2.1f scrub ok 2024-09-17T07:03:43.045 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:42 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:43.101 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:42 smithi100 ceph-mon[29092]: 6.15 scrub starts 2024-09-17T07:03:43.101 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:42 smithi100 ceph-mon[29092]: 6.15 scrub ok 2024-09-17T07:03:43.101 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:42 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:43.101 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:42 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:43.101 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:42 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:43.102 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:42 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:43.102 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:42 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:43.102 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:42 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:03:43.102 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:42 smithi100 ceph-mon[29092]: 2.1f scrub starts 2024-09-17T07:03:43.102 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:42 smithi100 ceph-mon[29092]: 2.1f scrub ok 2024-09-17T07:03:43.102 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:42 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:43.102 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:42 smithi100 ceph-mon[39462]: 6.15 scrub starts 2024-09-17T07:03:43.102 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:42 smithi100 ceph-mon[39462]: 6.15 scrub ok 2024-09-17T07:03:43.102 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:42 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:43.102 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:42 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:43.102 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:42 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:43.103 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:42 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:43.103 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:42 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:43.103 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:42 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:03:43.103 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:42 smithi100 ceph-mon[39462]: 2.1f scrub starts 2024-09-17T07:03:43.103 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:42 smithi100 ceph-mon[39462]: 2.1f scrub ok 2024-09-17T07:03:43.103 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:42 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:43.637 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T07:03:43.637 INFO:teuthology.orchestra.run.smithi100.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-17T07:03:44.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:43 smithi135 ceph-mon[33238]: 6.5 scrub starts 2024-09-17T07:03:44.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:43 smithi135 ceph-mon[33238]: 6.5 scrub ok 2024-09-17T07:03:44.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:43 smithi135 ceph-mon[33238]: 5.14 scrub starts 2024-09-17T07:03:44.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:43 smithi135 ceph-mon[33238]: 5.14 scrub ok 2024-09-17T07:03:44.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:43 smithi135 ceph-mon[33238]: 6.f deep-scrub starts 2024-09-17T07:03:44.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:43 smithi135 ceph-mon[33238]: 6.f deep-scrub ok 2024-09-17T07:03:44.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:43 smithi135 ceph-mon[33238]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T07:03:44.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:43 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1196571675' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-17T07:03:44.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:43 smithi100 ceph-mon[29092]: 6.5 scrub starts 2024-09-17T07:03:44.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:43 smithi100 ceph-mon[29092]: 6.5 scrub ok 2024-09-17T07:03:44.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:43 smithi100 ceph-mon[29092]: 5.14 scrub starts 2024-09-17T07:03:44.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:43 smithi100 ceph-mon[29092]: 5.14 scrub ok 2024-09-17T07:03:44.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:43 smithi100 ceph-mon[29092]: 6.f deep-scrub starts 2024-09-17T07:03:44.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:43 smithi100 ceph-mon[29092]: 6.f deep-scrub ok 2024-09-17T07:03:44.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:43 smithi100 ceph-mon[29092]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T07:03:44.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:43 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1196571675' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-17T07:03:44.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:43 smithi100 ceph-mon[39462]: 6.5 scrub starts 2024-09-17T07:03:44.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:43 smithi100 ceph-mon[39462]: 6.5 scrub ok 2024-09-17T07:03:44.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:43 smithi100 ceph-mon[39462]: 5.14 scrub starts 2024-09-17T07:03:44.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:43 smithi100 ceph-mon[39462]: 5.14 scrub ok 2024-09-17T07:03:44.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:43 smithi100 ceph-mon[39462]: 6.f deep-scrub starts 2024-09-17T07:03:44.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:43 smithi100 ceph-mon[39462]: 6.f deep-scrub ok 2024-09-17T07:03:44.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:43 smithi100 ceph-mon[39462]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T07:03:44.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:43 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1196571675' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-17T07:03:44.254 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-09-17T07:03:44.254 INFO:tasks.cephadm:Setup complete, yielding 2024-09-17T07:03:44.254 INFO:teuthology.run_tasks:Running task thrashosds... 2024-09-17T07:03:44.268 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-17T07:03:44.268 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'reef'}, 'ceph': {'conf': {'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms bind msgr1': False, 'ms bind msgr2': True, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'CEPHADM_STRAY_DAEMON', 'CEPHADM_FAILED_DAEMON', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)'], 'log-only-match': ['CEPHADM_'], 'sha1': '19e2a2beb6f41a3ab62650e38614eb29b59dcbb9'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'cephadm': {'cephadm_mode': 'root'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '19e2a2beb6f41a3ab62650e38614eb29b59dcbb9'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'reef', 'sha1': '1cbdc9211df088cccbd94423c7cf3a2b85fe432b'}} 2024-09-17T07:03:44.268 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-17T07:03:44.269 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-09-17T07:03:44.648 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:03:45.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:44 smithi135 ceph-mon[33238]: 2.1d deep-scrub starts 2024-09-17T07:03:45.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:44 smithi135 ceph-mon[33238]: 2.1d deep-scrub ok 2024-09-17T07:03:45.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:44 smithi135 ceph-mon[33238]: 2.1e scrub starts 2024-09-17T07:03:45.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:44 smithi135 ceph-mon[33238]: 2.1e scrub ok 2024-09-17T07:03:45.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:44 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:03:45.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:44 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:03:45.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:44 smithi135 ceph-mon[33238]: osdmap e91: 8 total, 8 up, 8 in 2024-09-17T07:03:45.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:44 smithi135 ceph-mon[33238]: 5.a scrub starts 2024-09-17T07:03:45.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:44 smithi135 ceph-mon[33238]: 5.a scrub ok 2024-09-17T07:03:45.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:44 smithi135 ceph-mon[33238]: 6.2 scrub starts 2024-09-17T07:03:45.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:44 smithi135 ceph-mon[33238]: 6.2 scrub ok 2024-09-17T07:03:45.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:44 smithi135 ceph-mon[33238]: 2.1b scrub starts 2024-09-17T07:03:45.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:44 smithi135 ceph-mon[33238]: 2.1b scrub ok 2024-09-17T07:03:45.090 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:44 smithi100 ceph-mon[29092]: 2.1d deep-scrub starts 2024-09-17T07:03:45.090 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:44 smithi100 ceph-mon[29092]: 2.1d deep-scrub ok 2024-09-17T07:03:45.090 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:44 smithi100 ceph-mon[29092]: 2.1e scrub starts 2024-09-17T07:03:45.090 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:44 smithi100 ceph-mon[29092]: 2.1e scrub ok 2024-09-17T07:03:45.091 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:44 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:03:45.091 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:44 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:03:45.091 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:44 smithi100 ceph-mon[29092]: osdmap e91: 8 total, 8 up, 8 in 2024-09-17T07:03:45.091 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:44 smithi100 ceph-mon[29092]: 5.a scrub starts 2024-09-17T07:03:45.091 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:44 smithi100 ceph-mon[29092]: 5.a scrub ok 2024-09-17T07:03:45.091 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:44 smithi100 ceph-mon[29092]: 6.2 scrub starts 2024-09-17T07:03:45.091 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:44 smithi100 ceph-mon[29092]: 6.2 scrub ok 2024-09-17T07:03:45.091 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:44 smithi100 ceph-mon[29092]: 2.1b scrub starts 2024-09-17T07:03:45.091 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:44 smithi100 ceph-mon[29092]: 2.1b scrub ok 2024-09-17T07:03:45.091 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:44 smithi100 ceph-mon[39462]: 2.1d deep-scrub starts 2024-09-17T07:03:45.091 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:44 smithi100 ceph-mon[39462]: 2.1d deep-scrub ok 2024-09-17T07:03:45.092 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:44 smithi100 ceph-mon[39462]: 2.1e scrub starts 2024-09-17T07:03:45.092 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:44 smithi100 ceph-mon[39462]: 2.1e scrub ok 2024-09-17T07:03:45.092 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:44 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:03:45.092 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:44 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:03:45.092 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:44 smithi100 ceph-mon[39462]: osdmap e91: 8 total, 8 up, 8 in 2024-09-17T07:03:45.092 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:44 smithi100 ceph-mon[39462]: 5.a scrub starts 2024-09-17T07:03:45.092 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:44 smithi100 ceph-mon[39462]: 5.a scrub ok 2024-09-17T07:03:45.092 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:44 smithi100 ceph-mon[39462]: 6.2 scrub starts 2024-09-17T07:03:45.092 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:44 smithi100 ceph-mon[39462]: 6.2 scrub ok 2024-09-17T07:03:45.092 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:44 smithi100 ceph-mon[39462]: 2.1b scrub starts 2024-09-17T07:03:45.092 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:44 smithi100 ceph-mon[39462]: 2.1b scrub ok 2024-09-17T07:03:45.938 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:03:45 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:03:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:03:45.939 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:45 smithi100 ceph-mon[29092]: 5.c scrub starts 2024-09-17T07:03:45.939 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:45 smithi100 ceph-mon[29092]: 5.c scrub ok 2024-09-17T07:03:45.939 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:45 smithi100 ceph-mon[29092]: 4.5 scrub starts 2024-09-17T07:03:45.939 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:45 smithi100 ceph-mon[29092]: 4.5 scrub ok 2024-09-17T07:03:45.939 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:45 smithi100 ceph-mon[29092]: 2.16 scrub starts 2024-09-17T07:03:45.939 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:45 smithi100 ceph-mon[29092]: 2.16 scrub ok 2024-09-17T07:03:45.939 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:45 smithi100 ceph-mon[29092]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:03:45.940 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:45 smithi100 ceph-mon[39462]: 5.c scrub starts 2024-09-17T07:03:45.940 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:45 smithi100 ceph-mon[39462]: 5.c scrub ok 2024-09-17T07:03:45.940 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:45 smithi100 ceph-mon[39462]: 4.5 scrub starts 2024-09-17T07:03:45.940 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:45 smithi100 ceph-mon[39462]: 4.5 scrub ok 2024-09-17T07:03:45.940 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:45 smithi100 ceph-mon[39462]: 2.16 scrub starts 2024-09-17T07:03:45.940 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:45 smithi100 ceph-mon[39462]: 2.16 scrub ok 2024-09-17T07:03:45.940 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:45 smithi100 ceph-mon[39462]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:03:46.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:45 smithi135 ceph-mon[33238]: 5.c scrub starts 2024-09-17T07:03:46.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:45 smithi135 ceph-mon[33238]: 5.c scrub ok 2024-09-17T07:03:46.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:45 smithi135 ceph-mon[33238]: 4.5 scrub starts 2024-09-17T07:03:46.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:45 smithi135 ceph-mon[33238]: 4.5 scrub ok 2024-09-17T07:03:46.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:45 smithi135 ceph-mon[33238]: 2.16 scrub starts 2024-09-17T07:03:46.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:45 smithi135 ceph-mon[33238]: 2.16 scrub ok 2024-09-17T07:03:46.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:45 smithi135 ceph-mon[33238]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:03:46.993 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:46 smithi100 ceph-mon[39462]: 2.1c deep-scrub starts 2024-09-17T07:03:46.993 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:46 smithi100 ceph-mon[39462]: 2.1c deep-scrub ok 2024-09-17T07:03:46.993 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:46 smithi100 ceph-mon[39462]: 5.d scrub starts 2024-09-17T07:03:46.993 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:46 smithi100 ceph-mon[39462]: 5.d scrub ok 2024-09-17T07:03:46.993 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:46 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:46.993 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:46 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:46.993 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:46 smithi100 ceph-mon[39462]: 6.e scrub starts 2024-09-17T07:03:46.993 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:46 smithi100 ceph-mon[39462]: 6.e scrub ok 2024-09-17T07:03:46.993 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:46 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' 2024-09-17T07:03:46.994 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:46 smithi100 ceph-mon[29092]: 2.1c deep-scrub starts 2024-09-17T07:03:46.994 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:46 smithi100 ceph-mon[29092]: 2.1c deep-scrub ok 2024-09-17T07:03:46.994 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:46 smithi100 ceph-mon[29092]: 5.d scrub starts 2024-09-17T07:03:46.994 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:46 smithi100 ceph-mon[29092]: 5.d scrub ok 2024-09-17T07:03:46.994 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:46 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:46.994 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:46 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:46.994 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:46 smithi100 ceph-mon[29092]: 6.e scrub starts 2024-09-17T07:03:46.994 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:46 smithi100 ceph-mon[29092]: 6.e scrub ok 2024-09-17T07:03:46.994 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:46 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' 2024-09-17T07:03:46.995 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:03:46 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: [17/Sep/2024:07:03:46] ENGINE Bus STOPPING 2024-09-17T07:03:46.995 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:03:46 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: [17/Sep/2024:07:03:46] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-17T07:03:46.995 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:03:46 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: [17/Sep/2024:07:03:46] ENGINE Bus STOPPED 2024-09-17T07:03:46.995 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:03:46 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: [17/Sep/2024:07:03:46] ENGINE Bus STARTING 2024-09-17T07:03:46.995 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:03:46 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: [17/Sep/2024:07:03:46] ENGINE Serving on http://:::9283 2024-09-17T07:03:46.995 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:03:46 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: [17/Sep/2024:07:03:46] ENGINE Bus STARTED 2024-09-17T07:03:47.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:46 smithi135 ceph-mon[33238]: 2.1c deep-scrub starts 2024-09-17T07:03:47.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:46 smithi135 ceph-mon[33238]: 2.1c deep-scrub ok 2024-09-17T07:03:47.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:46 smithi135 ceph-mon[33238]: 5.d scrub starts 2024-09-17T07:03:47.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:46 smithi135 ceph-mon[33238]: 5.d scrub ok 2024-09-17T07:03:47.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:46 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:47.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:46 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:47.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:46 smithi135 ceph-mon[33238]: 6.e scrub starts 2024-09-17T07:03:47.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:46 smithi135 ceph-mon[33238]: 6.e scrub ok 2024-09-17T07:03:47.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:46 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' 2024-09-17T07:03:47.394 INFO:tasks.thrashosds:Beginning thrashosds... 2024-09-17T07:03:47.394 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-17T07:03:47.394 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T07:03:48.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:47 smithi135 ceph-mon[33238]: 5.9 scrub starts 2024-09-17T07:03:48.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:47 smithi135 ceph-mon[33238]: 5.9 scrub ok 2024-09-17T07:03:48.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:47 smithi135 ceph-mon[33238]: 2.1a scrub starts 2024-09-17T07:03:48.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:47 smithi135 ceph-mon[33238]: 2.1a scrub ok 2024-09-17T07:03:48.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:47 smithi135 ceph-mon[33238]: 5.15 scrub starts 2024-09-17T07:03:48.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:47 smithi135 ceph-mon[33238]: 5.15 scrub ok 2024-09-17T07:03:48.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:47 smithi135 ceph-mon[33238]: 4.b scrub starts 2024-09-17T07:03:48.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:47 smithi135 ceph-mon[33238]: 4.b scrub ok 2024-09-17T07:03:48.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:47 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [7, 2]}]: dispatch 2024-09-17T07:03:48.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:47 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-09-17T07:03:48.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:47 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]: dispatch 2024-09-17T07:03:48.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:47 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-09-17T07:03:48.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:47 smithi135 ceph-mon[33238]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-17T07:03:48.045 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:47 smithi135 ceph-mon[33238]: 5.1d scrub starts 2024-09-17T07:03:48.045 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:47 smithi135 ceph-mon[33238]: 5.1d scrub ok 2024-09-17T07:03:48.045 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:47 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:03:48.061 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:03:48.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:47 smithi100 ceph-mon[29092]: 5.9 scrub starts 2024-09-17T07:03:48.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:47 smithi100 ceph-mon[29092]: 5.9 scrub ok 2024-09-17T07:03:48.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:47 smithi100 ceph-mon[29092]: 2.1a scrub starts 2024-09-17T07:03:48.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:47 smithi100 ceph-mon[29092]: 2.1a scrub ok 2024-09-17T07:03:48.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:47 smithi100 ceph-mon[29092]: 5.15 scrub starts 2024-09-17T07:03:48.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:47 smithi100 ceph-mon[29092]: 5.15 scrub ok 2024-09-17T07:03:48.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:47 smithi100 ceph-mon[29092]: 4.b scrub starts 2024-09-17T07:03:48.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:47 smithi100 ceph-mon[29092]: 4.b scrub ok 2024-09-17T07:03:48.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:47 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [7, 2]}]: dispatch 2024-09-17T07:03:48.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:47 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-09-17T07:03:48.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:47 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]: dispatch 2024-09-17T07:03:48.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:47 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-09-17T07:03:48.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:47 smithi100 ceph-mon[29092]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-17T07:03:48.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:47 smithi100 ceph-mon[29092]: 5.1d scrub starts 2024-09-17T07:03:48.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:47 smithi100 ceph-mon[29092]: 5.1d scrub ok 2024-09-17T07:03:48.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:47 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:03:48.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:47 smithi100 ceph-mon[39462]: 5.9 scrub starts 2024-09-17T07:03:48.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:47 smithi100 ceph-mon[39462]: 5.9 scrub ok 2024-09-17T07:03:48.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:47 smithi100 ceph-mon[39462]: 2.1a scrub starts 2024-09-17T07:03:48.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:47 smithi100 ceph-mon[39462]: 2.1a scrub ok 2024-09-17T07:03:48.129 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:47 smithi100 ceph-mon[39462]: 5.15 scrub starts 2024-09-17T07:03:48.129 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:47 smithi100 ceph-mon[39462]: 5.15 scrub ok 2024-09-17T07:03:48.129 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:47 smithi100 ceph-mon[39462]: 4.b scrub starts 2024-09-17T07:03:48.129 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:47 smithi100 ceph-mon[39462]: 4.b scrub ok 2024-09-17T07:03:48.129 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:47 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [7, 2]}]: dispatch 2024-09-17T07:03:48.129 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:47 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-09-17T07:03:48.129 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:47 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]: dispatch 2024-09-17T07:03:48.129 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:47 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-09-17T07:03:48.129 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:47 smithi100 ceph-mon[39462]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-17T07:03:48.129 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:47 smithi100 ceph-mon[39462]: 5.1d scrub starts 2024-09-17T07:03:48.129 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:47 smithi100 ceph-mon[39462]: 5.1d scrub ok 2024-09-17T07:03:48.129 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:47 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:03:48.543 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:03:48 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:03:49.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:48 smithi135 ceph-mon[33238]: 6.1c scrub starts 2024-09-17T07:03:49.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:48 smithi135 ceph-mon[33238]: 6.1c scrub ok 2024-09-17T07:03:49.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:48 smithi135 ceph-mon[33238]: 5.11 deep-scrub starts 2024-09-17T07:03:49.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:48 smithi135 ceph-mon[33238]: 5.11 deep-scrub ok 2024-09-17T07:03:49.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:48 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [7, 2]}]': finished 2024-09-17T07:03:49.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:48 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-09-17T07:03:49.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:48 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]': finished 2024-09-17T07:03:49.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:48 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-09-17T07:03:49.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:48 smithi135 ceph-mon[33238]: osdmap e92: 8 total, 8 up, 8 in 2024-09-17T07:03:49.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:48 smithi135 ceph-mon[33238]: 6.d scrub starts 2024-09-17T07:03:49.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:48 smithi135 ceph-mon[33238]: 6.d scrub ok 2024-09-17T07:03:49.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:48 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:03:49.049 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:48 smithi100 ceph-mon[29092]: 6.1c scrub starts 2024-09-17T07:03:49.050 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:48 smithi100 ceph-mon[29092]: 6.1c scrub ok 2024-09-17T07:03:49.050 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:48 smithi100 ceph-mon[29092]: 5.11 deep-scrub starts 2024-09-17T07:03:49.050 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:48 smithi100 ceph-mon[29092]: 5.11 deep-scrub ok 2024-09-17T07:03:49.050 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:48 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [7, 2]}]': finished 2024-09-17T07:03:49.050 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:48 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-09-17T07:03:49.050 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:48 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]': finished 2024-09-17T07:03:49.050 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:48 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-09-17T07:03:49.050 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:48 smithi100 ceph-mon[29092]: osdmap e92: 8 total, 8 up, 8 in 2024-09-17T07:03:49.050 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:48 smithi100 ceph-mon[29092]: 6.d scrub starts 2024-09-17T07:03:49.050 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:48 smithi100 ceph-mon[29092]: 6.d scrub ok 2024-09-17T07:03:49.050 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:48 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:03:49.051 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:48 smithi100 ceph-mon[39462]: 6.1c scrub starts 2024-09-17T07:03:49.051 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:48 smithi100 ceph-mon[39462]: 6.1c scrub ok 2024-09-17T07:03:49.051 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:48 smithi100 ceph-mon[39462]: 5.11 deep-scrub starts 2024-09-17T07:03:49.051 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:48 smithi100 ceph-mon[39462]: 5.11 deep-scrub ok 2024-09-17T07:03:49.051 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:48 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [7, 2]}]': finished 2024-09-17T07:03:49.051 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:48 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-09-17T07:03:49.051 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:48 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]': finished 2024-09-17T07:03:49.051 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:48 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-09-17T07:03:49.051 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:48 smithi100 ceph-mon[39462]: osdmap e92: 8 total, 8 up, 8 in 2024-09-17T07:03:49.051 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:48 smithi100 ceph-mon[39462]: 6.d scrub starts 2024-09-17T07:03:49.051 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:48 smithi100 ceph-mon[39462]: 6.d scrub ok 2024-09-17T07:03:49.051 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:48 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:03:50.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:49 smithi135 ceph-mon[33238]: osdmap e93: 8 total, 8 up, 8 in 2024-09-17T07:03:50.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:49 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:50.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:49 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:50.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:49 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:03:50.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:49 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:03:50.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:49 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:50.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:49 smithi135 ceph-mon[33238]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-17T07:03:50.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:49 smithi135 ceph-mon[33238]: Reconfiguring daemon alertmanager.a on smithi100 2024-09-17T07:03:50.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:49 smithi135 ceph-mon[33238]: pgmap v49: 132 pgs: 4 peering, 128 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:03:50.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:49 smithi100 ceph-mon[29092]: osdmap e93: 8 total, 8 up, 8 in 2024-09-17T07:03:50.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:49 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:50.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:49 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:50.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:49 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:03:50.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:49 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:03:50.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:49 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:50.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:49 smithi100 ceph-mon[29092]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-17T07:03:50.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:49 smithi100 ceph-mon[29092]: Reconfiguring daemon alertmanager.a on smithi100 2024-09-17T07:03:50.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:49 smithi100 ceph-mon[29092]: pgmap v49: 132 pgs: 4 peering, 128 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:03:50.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:49 smithi100 ceph-mon[39462]: osdmap e93: 8 total, 8 up, 8 in 2024-09-17T07:03:50.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:49 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:50.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:49 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:50.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:49 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:03:50.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:49 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:03:50.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:49 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:50.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:49 smithi100 ceph-mon[39462]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-17T07:03:50.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:49 smithi100 ceph-mon[39462]: Reconfiguring daemon alertmanager.a on smithi100 2024-09-17T07:03:50.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:49 smithi100 ceph-mon[39462]: pgmap v49: 132 pgs: 4 peering, 128 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:03:51.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:50 smithi100 ceph-mon[39462]: 5.2 scrub starts 2024-09-17T07:03:51.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:50 smithi100 ceph-mon[39462]: 5.2 scrub ok 2024-09-17T07:03:51.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:50 smithi100 ceph-mon[39462]: 4.8 scrub starts 2024-09-17T07:03:51.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:50 smithi100 ceph-mon[39462]: 4.8 scrub ok 2024-09-17T07:03:51.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:50 smithi100 ceph-mon[29092]: 5.2 scrub starts 2024-09-17T07:03:51.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:50 smithi100 ceph-mon[29092]: 5.2 scrub ok 2024-09-17T07:03:51.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:50 smithi100 ceph-mon[29092]: 4.8 scrub starts 2024-09-17T07:03:51.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:50 smithi100 ceph-mon[29092]: 4.8 scrub ok 2024-09-17T07:03:51.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:50 smithi135 ceph-mon[33238]: 5.2 scrub starts 2024-09-17T07:03:51.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:50 smithi135 ceph-mon[33238]: 5.2 scrub ok 2024-09-17T07:03:51.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:50 smithi135 ceph-mon[33238]: 4.8 scrub starts 2024-09-17T07:03:51.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:50 smithi135 ceph-mon[33238]: 4.8 scrub ok 2024-09-17T07:03:52.126 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:51 smithi100 ceph-mon[29092]: 3.0 scrub starts 2024-09-17T07:03:52.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:51 smithi100 ceph-mon[29092]: 3.0 scrub ok 2024-09-17T07:03:52.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:51 smithi100 ceph-mon[29092]: pgmap v50: 132 pgs: 4 peering, 128 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 703 B/s rd, 0 op/s 2024-09-17T07:03:52.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:51 smithi100 ceph-mon[39462]: 3.0 scrub starts 2024-09-17T07:03:52.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:51 smithi100 ceph-mon[39462]: 3.0 scrub ok 2024-09-17T07:03:52.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:51 smithi100 ceph-mon[39462]: pgmap v50: 132 pgs: 4 peering, 128 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 703 B/s rd, 0 op/s 2024-09-17T07:03:52.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:51 smithi135 ceph-mon[33238]: 3.0 scrub starts 2024-09-17T07:03:52.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:51 smithi135 ceph-mon[33238]: 3.0 scrub ok 2024-09-17T07:03:52.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:51 smithi135 ceph-mon[33238]: pgmap v50: 132 pgs: 4 peering, 128 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 703 B/s rd, 0 op/s 2024-09-17T07:03:52.821 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T07:03:52.823 INFO:teuthology.orchestra.run.smithi100.stderr:dumped all 2024-09-17T07:03:53.126 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:52 smithi100 ceph-mon[29092]: 5.13 scrub starts 2024-09-17T07:03:53.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:52 smithi100 ceph-mon[29092]: 5.13 scrub ok 2024-09-17T07:03:53.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:52 smithi100 ceph-mon[29092]: 2.14 scrub starts 2024-09-17T07:03:53.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:52 smithi100 ceph-mon[29092]: 2.14 scrub ok 2024-09-17T07:03:53.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:52 smithi100 ceph-mon[29092]: 2.a scrub starts 2024-09-17T07:03:53.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:52 smithi100 ceph-mon[29092]: 2.a scrub ok 2024-09-17T07:03:53.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:52 smithi100 ceph-mon[39462]: 5.13 scrub starts 2024-09-17T07:03:53.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:52 smithi100 ceph-mon[39462]: 5.13 scrub ok 2024-09-17T07:03:53.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:52 smithi100 ceph-mon[39462]: 2.14 scrub starts 2024-09-17T07:03:53.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:52 smithi100 ceph-mon[39462]: 2.14 scrub ok 2024-09-17T07:03:53.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:52 smithi100 ceph-mon[39462]: 2.a scrub starts 2024-09-17T07:03:53.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:52 smithi100 ceph-mon[39462]: 2.a scrub ok 2024-09-17T07:03:53.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:52 smithi135 ceph-mon[33238]: 5.13 scrub starts 2024-09-17T07:03:53.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:52 smithi135 ceph-mon[33238]: 5.13 scrub ok 2024-09-17T07:03:53.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:52 smithi135 ceph-mon[33238]: 2.14 scrub starts 2024-09-17T07:03:53.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:52 smithi135 ceph-mon[33238]: 2.14 scrub ok 2024-09-17T07:03:53.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:52 smithi135 ceph-mon[33238]: 2.a scrub starts 2024-09-17T07:03:53.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:52 smithi135 ceph-mon[33238]: 2.a scrub ok 2024-09-17T07:03:53.590 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:Sep 17 07:03:53 smithi100 systemd[1]: Stopping Ceph alertmanager.a for e848267c-74c1-11ef-bceb-c7b262605968... 2024-09-17T07:03:53.871 INFO:teuthology.orchestra.run.smithi100.stdout:{"pg_ready":true,"pg_map":{"version":50,"stamp":"2024-09-17T07:03:51.009751+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596246,"num_objects":196,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":1011,"num_read_kb":857,"num_write":609,"num_write_kb":2459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":604,"ondisk_log_size":604,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":248864,"kb_used_data":7272,"kb_used_omap":12,"kb_used_meta":240627,"kb_avail":749712352,"statfs":{"total":767960285184,"available":767705448448,"internally_reserved":0,"allocated":7446528,"data_stored":4321139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":246402616},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.276681"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:43.734739+0000","last_change":"2024-09-17T07:02:18.366053+0000","last_active":"2024-09-17T07:03:43.734739+0000","last_peered":"2024-09-17T07:03:43.734739+0000","last_clean":"2024-09-17T07:03:43.734739+0000","last_became_active":"2024-09-17T07:02:18.365773+0000","last_became_peered":"2024-09-17T07:02:18.365773+0000","last_unstale":"2024-09-17T07:03:43.734739+0000","last_undegraded":"2024-09-17T07:03:43.734739+0000","last_fullsized":"2024-09-17T07:03:43.734739+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:02:17.002904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:43.736930+0000","last_change":"2024-09-17T07:03:41.931873+0000","last_active":"2024-09-17T07:03:43.736930+0000","last_peered":"2024-09-17T07:03:43.736930+0000","last_clean":"2024-09-17T07:03:43.736930+0000","last_became_active":"2024-09-17T07:02:12.006754+0000","last_became_peered":"2024-09-17T07:02:12.006754+0000","last_unstale":"2024-09-17T07:03:43.736930+0000","last_undegraded":"2024-09-17T07:03:43.736930+0000","last_fullsized":"2024-09-17T07:03:43.736930+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:41.931780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:41.931780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:50.138892+0000","scrub_duration":0.0017747419999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"75'15","reported_seq":67,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:43.734544+0000","last_change":"2024-09-17T07:03:30.436540+0000","last_active":"2024-09-17T07:03:43.734544+0000","last_peered":"2024-09-17T07:03:43.734544+0000","last_clean":"2024-09-17T07:03:43.734544+0000","last_became_active":"2024-09-17T07:02:14.012962+0000","last_became_peered":"2024-09-17T07:02:14.012962+0000","last_unstale":"2024-09-17T07:03:43.734544+0000","last_undegraded":"2024-09-17T07:03:43.734544+0000","last_fullsized":"2024-09-17T07:03:43.734544+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-09-17T07:03:30.436444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:30.436444+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:35.913395+0000","scrub_duration":0.0023734419999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:43.734738+0000","last_change":"2024-09-17T07:02:16.015244+0000","last_active":"2024-09-17T07:03:43.734738+0000","last_peered":"2024-09-17T07:03:43.734738+0000","last_clean":"2024-09-17T07:03:43.734738+0000","last_became_active":"2024-09-17T07:02:16.014928+0000","last_became_peered":"2024-09-17T07:02:16.014928+0000","last_unstale":"2024-09-17T07:03:43.734738+0000","last_undegraded":"2024-09-17T07:03:43.734738+0000","last_fullsized":"2024-09-17T07:03:43.734738+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:02:14.999435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":44,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:43.735433+0000","last_change":"2024-09-17T07:03:43.459940+0000","last_active":"2024-09-17T07:03:43.735433+0000","last_peered":"2024-09-17T07:03:43.735433+0000","last_clean":"2024-09-17T07:03:43.735433+0000","last_became_active":"2024-09-17T07:02:12.010069+0000","last_became_peered":"2024-09-17T07:02:12.010069+0000","last_unstale":"2024-09-17T07:03:43.735433+0000","last_undegraded":"2024-09-17T07:03:43.735433+0000","last_fullsized":"2024-09-17T07:03:43.735433+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:43.459821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:43.459821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:44.534548+0000","scrub_duration":0.0015265960000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"75'9","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:43.734757+0000","last_change":"2024-09-17T07:03:17.864856+0000","last_active":"2024-09-17T07:03:43.734757+0000","last_peered":"2024-09-17T07:03:43.734757+0000","last_clean":"2024-09-17T07:03:43.734757+0000","last_became_active":"2024-09-17T07:02:14.016244+0000","last_became_peered":"2024-09-17T07:02:14.016244+0000","last_unstale":"2024-09-17T07:03:43.734757+0000","last_undegraded":"2024-09-17T07:03:43.734757+0000","last_fullsized":"2024-09-17T07:03:43.734757+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:03:17.864726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:17.864726+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:29.697946+0000","scrub_duration":0.0021756380000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:44.044525+0000","last_change":"2024-09-17T07:03:37.113047+0000","last_active":"2024-09-17T07:03:44.044525+0000","last_peered":"2024-09-17T07:03:44.044525+0000","last_clean":"2024-09-17T07:03:44.044525+0000","last_became_active":"2024-09-17T07:02:16.014380+0000","last_became_peered":"2024-09-17T07:02:16.014380+0000","last_unstale":"2024-09-17T07:03:44.044525+0000","last_undegraded":"2024-09-17T07:03:44.044525+0000","last_fullsized":"2024-09-17T07:03:44.044525+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:37.112954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:37.112954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:45.057903+0000","scrub_duration":0.001466824,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:43.734836+0000","last_change":"2024-09-17T07:03:38.109357+0000","last_active":"2024-09-17T07:03:43.734836+0000","last_peered":"2024-09-17T07:03:43.734836+0000","last_clean":"2024-09-17T07:03:43.734836+0000","last_became_active":"2024-09-17T07:02:18.020790+0000","last_became_peered":"2024-09-17T07:02:18.020790+0000","last_unstale":"2024-09-17T07:03:43.734836+0000","last_undegraded":"2024-09-17T07:03:43.734836+0000","last_fullsized":"2024-09-17T07:03:43.734836+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:38.109265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:03:38.109265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:57.231258+0000","scrub_duration":0.001349659,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:44.062009+0000","last_change":"2024-09-17T07:03:43.060196+0000","last_active":"2024-09-17T07:03:44.062009+0000","last_peered":"2024-09-17T07:03:44.062009+0000","last_clean":"2024-09-17T07:03:44.062009+0000","last_became_active":"2024-09-17T07:02:12.011129+0000","last_became_peered":"2024-09-17T07:02:12.011129+0000","last_unstale":"2024-09-17T07:03:44.062009+0000","last_undegraded":"2024-09-17T07:03:44.062009+0000","last_fullsized":"2024-09-17T07:03:44.062009+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:43.060102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:43.060102+0000","last_clean_scrub_stamp":"2024-09-17T07:03:43.060102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:09.780017+0000","scrub_duration":0.0012238170000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"75'5","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:43.734866+0000","last_change":"2024-09-17T07:03:29.002445+0000","last_active":"2024-09-17T07:03:43.734866+0000","last_peered":"2024-09-17T07:03:43.734866+0000","last_clean":"2024-09-17T07:03:43.734866+0000","last_became_active":"2024-09-17T07:02:14.016640+0000","last_became_peered":"2024-09-17T07:02:14.016640+0000","last_unstale":"2024-09-17T07:03:43.734866+0000","last_undegraded":"2024-09-17T07:03:43.734866+0000","last_fullsized":"2024-09-17T07:03:43.734866+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-09-17T07:03:29.002312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:29.002312+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:44.000292+0000","scrub_duration":0.002172748,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":41,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:44.061172+0000","last_change":"2024-09-17T07:03:22.015629+0000","last_active":"2024-09-17T07:03:44.061172+0000","last_peered":"2024-09-17T07:03:44.061172+0000","last_clean":"2024-09-17T07:03:44.061172+0000","last_became_active":"2024-09-17T07:02:16.015401+0000","last_became_peered":"2024-09-17T07:02:16.015401+0000","last_unstale":"2024-09-17T07:03:44.061172+0000","last_undegraded":"2024-09-17T07:03:44.061172+0000","last_fullsized":"2024-09-17T07:03:44.061172+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:22.015535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:22.015535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:49.909183+0000","scrub_duration":0.001507752,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:44.448939+0000","last_change":"2024-09-17T07:03:34.882046+0000","last_active":"2024-09-17T07:03:44.448939+0000","last_peered":"2024-09-17T07:03:44.448939+0000","last_clean":"2024-09-17T07:03:44.448939+0000","last_became_active":"2024-09-17T07:02:18.023680+0000","last_became_peered":"2024-09-17T07:02:18.023680+0000","last_unstale":"2024-09-17T07:03:44.448939+0000","last_undegraded":"2024-09-17T07:03:44.448939+0000","last_fullsized":"2024-09-17T07:03:44.448939+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:34.881951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:03:34.881951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:36.646147+0000","scrub_duration":0.0014520379999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:45.029350+0000","last_change":"2024-09-17T07:03:45.029350+0000","last_active":"2024-09-17T07:03:45.029350+0000","last_peered":"2024-09-17T07:03:45.029350+0000","last_clean":"2024-09-17T07:03:45.029350+0000","last_became_active":"2024-09-17T07:02:12.012234+0000","last_became_peered":"2024-09-17T07:02:12.012234+0000","last_unstale":"2024-09-17T07:03:45.029350+0000","last_undegraded":"2024-09-17T07:03:45.029350+0000","last_fullsized":"2024-09-17T07:03:45.029350+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:45.029247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:45.029247+0000","last_clean_scrub_stamp":"2024-09-17T07:03:45.029247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:07.678537+0000","scrub_duration":0.001377554,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"75'9","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:43.735451+0000","last_change":"2024-09-17T07:02:14.013597+0000","last_active":"2024-09-17T07:03:43.735451+0000","last_peered":"2024-09-17T07:03:43.735451+0000","last_clean":"2024-09-17T07:03:43.735451+0000","last_became_active":"2024-09-17T07:02:14.013337+0000","last_became_peered":"2024-09-17T07:02:14.013337+0000","last_unstale":"2024-09-17T07:03:43.735451+0000","last_undegraded":"2024-09-17T07:03:43.735451+0000","last_fullsized":"2024-09-17T07:03:43.735451+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:02:12.994221+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:44.448641+0000","last_change":"2024-09-17T07:02:16.024037+0000","last_active":"2024-09-17T07:03:44.448641+0000","last_peered":"2024-09-17T07:03:44.448641+0000","last_clean":"2024-09-17T07:03:44.448641+0000","last_became_active":"2024-09-17T07:02:16.023774+0000","last_became_peered":"2024-09-17T07:02:16.023774+0000","last_unstale":"2024-09-17T07:03:44.448641+0000","last_undegraded":"2024-09-17T07:03:44.448641+0000","last_fullsized":"2024-09-17T07:03:44.448641+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:02:14.999435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:43.735804+0000","last_change":"2024-09-17T07:02:18.029470+0000","last_active":"2024-09-17T07:03:43.735804+0000","last_peered":"2024-09-17T07:03:43.735804+0000","last_clean":"2024-09-17T07:03:43.735804+0000","last_became_active":"2024-09-17T07:02:18.029178+0000","last_became_peered":"2024-09-17T07:02:18.029178+0000","last_unstale":"2024-09-17T07:03:43.735804+0000","last_undegraded":"2024-09-17T07:03:43.735804+0000","last_fullsized":"2024-09-17T07:03:43.735804+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:02:17.002904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"75'17","reported_seq":60,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:43.735256+0000","last_change":"2024-09-17T07:02:14.016560+0000","last_active":"2024-09-17T07:03:43.735256+0000","last_peered":"2024-09-17T07:03:43.735256+0000","last_clean":"2024-09-17T07:03:43.735256+0000","last_became_active":"2024-09-17T07:02:14.016270+0000","last_became_peered":"2024-09-17T07:02:14.016270+0000","last_unstale":"2024-09-17T07:03:43.735256+0000","last_undegraded":"2024-09-17T07:03:43.735256+0000","last_fullsized":"2024-09-17T07:03:43.735256+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:02:12.994221+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:44.061788+0000","last_change":"2024-09-17T07:03:14.092519+0000","last_active":"2024-09-17T07:03:44.061788+0000","last_peered":"2024-09-17T07:03:44.061788+0000","last_clean":"2024-09-17T07:03:44.061788+0000","last_became_active":"2024-09-17T07:02:12.011258+0000","last_became_peered":"2024-09-17T07:02:12.011258+0000","last_unstale":"2024-09-17T07:03:44.061788+0000","last_undegraded":"2024-09-17T07:03:44.061788+0000","last_fullsized":"2024-09-17T07:03:44.061788+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:14.092369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:14.092369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:19.152787+0000","scrub_duration":0.0015447460000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:44.044750+0000","last_change":"2024-09-17T07:03:44.044750+0000","last_active":"2024-09-17T07:03:44.044750+0000","last_peered":"2024-09-17T07:03:44.044750+0000","last_clean":"2024-09-17T07:03:44.044750+0000","last_became_active":"2024-09-17T07:02:16.018147+0000","last_became_peered":"2024-09-17T07:02:16.018147+0000","last_unstale":"2024-09-17T07:03:44.044750+0000","last_undegraded":"2024-09-17T07:03:44.044750+0000","last_fullsized":"2024-09-17T07:03:44.044750+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:44.044635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:44.044635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:46.891263+0000","scrub_duration":0.0014202049999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:43.943095+0000","last_change":"2024-09-17T07:03:42.984226+0000","last_active":"2024-09-17T07:03:43.943095+0000","last_peered":"2024-09-17T07:03:43.943095+0000","last_clean":"2024-09-17T07:03:43.943095+0000","last_became_active":"2024-09-17T07:02:18.022040+0000","last_became_peered":"2024-09-17T07:02:18.022040+0000","last_unstale":"2024-09-17T07:03:43.943095+0000","last_undegraded":"2024-09-17T07:03:43.943095+0000","last_fullsized":"2024-09-17T07:03:43.943095+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:42.984088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:42.984088+0000","last_clean_scrub_stamp":"2024-09-17T07:03:42.984088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:59.596779+0000","scrub_duration":0.0014441129999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"75'10","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:43.735007+0000","last_change":"2024-09-17T07:03:20.933478+0000","last_active":"2024-09-17T07:03:43.735007+0000","last_peered":"2024-09-17T07:03:43.735007+0000","last_clean":"2024-09-17T07:03:43.735007+0000","last_became_active":"2024-09-17T07:02:14.007702+0000","last_became_peered":"2024-09-17T07:02:14.007702+0000","last_unstale":"2024-09-17T07:03:43.735007+0000","last_undegraded":"2024-09-17T07:03:43.735007+0000","last_fullsized":"2024-09-17T07:03:43.735007+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-09-17T07:03:20.933353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:20.933353+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:28.865120+0000","scrub_duration":0.0021221080000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:44.045110+0000","last_change":"2024-09-17T07:02:12.007831+0000","last_active":"2024-09-17T07:03:44.045110+0000","last_peered":"2024-09-17T07:03:44.045110+0000","last_clean":"2024-09-17T07:03:44.045110+0000","last_became_active":"2024-09-17T07:02:12.007553+0000","last_became_peered":"2024-09-17T07:02:12.007553+0000","last_unstale":"2024-09-17T07:03:44.045110+0000","last_undegraded":"2024-09-17T07:03:44.045110+0000","last_fullsized":"2024-09-17T07:03:44.045110+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:02:10.991363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"75'8","reported_seq":68,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:45.909722+0000","last_change":"2024-09-17T07:03:45.909722+0000","last_active":"2024-09-17T07:03:45.909722+0000","last_peered":"2024-09-17T07:03:45.909722+0000","last_clean":"2024-09-17T07:03:45.909722+0000","last_became_active":"2024-09-17T07:02:16.015001+0000","last_became_peered":"2024-09-17T07:02:16.015001+0000","last_unstale":"2024-09-17T07:03:45.909722+0000","last_undegraded":"2024-09-17T07:03:45.909722+0000","last_fullsized":"2024-09-17T07:03:45.909722+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:03:45.909617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:45.909617+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:48.865649+0000","scrub_duration":0.0021269700000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:46.241227+0000","last_change":"2024-09-17T07:03:46.241227+0000","last_active":"2024-09-17T07:03:46.241227+0000","last_peered":"2024-09-17T07:03:46.241227+0000","last_clean":"2024-09-17T07:03:46.241227+0000","last_became_active":"2024-09-17T07:02:18.017460+0000","last_became_peered":"2024-09-17T07:02:18.017460+0000","last_unstale":"2024-09-17T07:03:46.241227+0000","last_undegraded":"2024-09-17T07:03:46.241227+0000","last_fullsized":"2024-09-17T07:03:46.241227+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:46.241143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:03:46.241143+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:15.293653+0000","scrub_duration":0.0013561739999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"75'15","reported_seq":67,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:44.045485+0000","last_change":"2024-09-17T07:03:16.067607+0000","last_active":"2024-09-17T07:03:44.045485+0000","last_peered":"2024-09-17T07:03:44.045485+0000","last_clean":"2024-09-17T07:03:44.045485+0000","last_became_active":"2024-09-17T07:02:14.016397+0000","last_became_peered":"2024-09-17T07:02:14.016397+0000","last_unstale":"2024-09-17T07:03:44.045485+0000","last_undegraded":"2024-09-17T07:03:44.045485+0000","last_fullsized":"2024-09-17T07:03:44.045485+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-09-17T07:03:16.067479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:16.067479+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:28.548727+0000","scrub_duration":0.0025498640000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":44,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:44.044624+0000","last_change":"2024-09-17T07:03:28.072773+0000","last_active":"2024-09-17T07:03:44.044624+0000","last_peered":"2024-09-17T07:03:44.044624+0000","last_clean":"2024-09-17T07:03:44.044624+0000","last_became_active":"2024-09-17T07:02:12.007913+0000","last_became_peered":"2024-09-17T07:02:12.007913+0000","last_unstale":"2024-09-17T07:03:44.044624+0000","last_undegraded":"2024-09-17T07:03:44.044624+0000","last_fullsized":"2024-09-17T07:03:44.044624+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:28.072697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:28.072697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:43.368112+0000","scrub_duration":0.0012624730000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"75'8","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:45.536995+0000","last_change":"2024-09-17T07:02:16.022051+0000","last_active":"2024-09-17T07:03:45.536995+0000","last_peered":"2024-09-17T07:03:45.536995+0000","last_clean":"2024-09-17T07:03:45.536995+0000","last_became_active":"2024-09-17T07:02:16.021762+0000","last_became_peered":"2024-09-17T07:02:16.021762+0000","last_unstale":"2024-09-17T07:03:45.536995+0000","last_undegraded":"2024-09-17T07:03:45.536995+0000","last_fullsized":"2024-09-17T07:03:45.536995+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:02:14.999435+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":38,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.759489+0000","last_change":"2024-09-17T07:03:47.759562+0000","last_active":"2024-09-17T07:03:47.759489+0000","last_peered":"2024-09-17T07:03:47.759489+0000","last_clean":"2024-09-17T07:03:47.759489+0000","last_became_active":"2024-09-17T07:02:18.025332+0000","last_became_peered":"2024-09-17T07:02:18.025332+0000","last_unstale":"2024-09-17T07:03:47.759489+0000","last_undegraded":"2024-09-17T07:03:47.759489+0000","last_fullsized":"2024-09-17T07:03:47.759489+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:47.759476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:03:47.759476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:16.113819+0000","scrub_duration":0.0035797160000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"75'11","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:43.735250+0000","last_change":"2024-09-17T07:02:14.010584+0000","last_active":"2024-09-17T07:03:43.735250+0000","last_peered":"2024-09-17T07:03:43.735250+0000","last_clean":"2024-09-17T07:03:43.735250+0000","last_became_active":"2024-09-17T07:02:14.010227+0000","last_became_peered":"2024-09-17T07:02:14.010227+0000","last_unstale":"2024-09-17T07:03:43.735250+0000","last_undegraded":"2024-09-17T07:03:43.735250+0000","last_fullsized":"2024-09-17T07:03:43.735250+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:02:12.994221+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:44.061577+0000","last_change":"2024-09-17T07:03:31.076235+0000","last_active":"2024-09-17T07:03:44.061577+0000","last_peered":"2024-09-17T07:03:44.061577+0000","last_clean":"2024-09-17T07:03:44.061577+0000","last_became_active":"2024-09-17T07:02:12.013182+0000","last_became_peered":"2024-09-17T07:02:12.013182+0000","last_unstale":"2024-09-17T07:03:44.061577+0000","last_undegraded":"2024-09-17T07:03:44.061577+0000","last_fullsized":"2024-09-17T07:03:44.061577+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:31.076134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:31.076134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:37.803854+0000","scrub_duration":0.0012890040000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:44.448761+0000","last_change":"2024-09-17T07:03:18.816900+0000","last_active":"2024-09-17T07:03:44.448761+0000","last_peered":"2024-09-17T07:03:44.448761+0000","last_clean":"2024-09-17T07:03:44.448761+0000","last_became_active":"2024-09-17T07:02:16.020224+0000","last_became_peered":"2024-09-17T07:02:16.020224+0000","last_unstale":"2024-09-17T07:03:44.448761+0000","last_undegraded":"2024-09-17T07:03:44.448761+0000","last_fullsized":"2024-09-17T07:03:44.448761+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:18.816795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:18.816795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:46.254399+0000","scrub_duration":0.001573212,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:43.734830+0000","last_change":"2024-09-17T07:03:35.447165+0000","last_active":"2024-09-17T07:03:43.734830+0000","last_peered":"2024-09-17T07:03:43.734830+0000","last_clean":"2024-09-17T07:03:43.734830+0000","last_became_active":"2024-09-17T07:02:18.365000+0000","last_became_peered":"2024-09-17T07:02:18.365000+0000","last_unstale":"2024-09-17T07:03:43.734830+0000","last_undegraded":"2024-09-17T07:03:43.734830+0000","last_fullsized":"2024-09-17T07:03:43.734830+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:35.447078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:03:35.447078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:38.486661+0000","scrub_duration":0.0020715600000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"75'14","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:43.734916+0000","last_change":"2024-09-17T07:02:14.008065+0000","last_active":"2024-09-17T07:03:43.734916+0000","last_peered":"2024-09-17T07:03:43.734916+0000","last_clean":"2024-09-17T07:03:43.734916+0000","last_became_active":"2024-09-17T07:02:14.007810+0000","last_became_peered":"2024-09-17T07:02:14.007810+0000","last_unstale":"2024-09-17T07:03:43.734916+0000","last_undegraded":"2024-09-17T07:03:43.734916+0000","last_fullsized":"2024-09-17T07:03:43.734916+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:02:12.994221+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":44,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:44.445875+0000","last_change":"2024-09-17T07:03:16.455893+0000","last_active":"2024-09-17T07:03:44.445875+0000","last_peered":"2024-09-17T07:03:44.445875+0000","last_clean":"2024-09-17T07:03:44.445875+0000","last_became_active":"2024-09-17T07:02:12.011766+0000","last_became_peered":"2024-09-17T07:02:12.011766+0000","last_unstale":"2024-09-17T07:03:44.445875+0000","last_undegraded":"2024-09-17T07:03:44.445875+0000","last_fullsized":"2024-09-17T07:03:44.445875+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:16.455684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:16.455684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:22.392690+0000","scrub_duration":0.0020498690000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"75'8","reported_seq":67,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:45.536884+0000","last_change":"2024-09-17T07:03:33.444197+0000","last_active":"2024-09-17T07:03:45.536884+0000","last_peered":"2024-09-17T07:03:45.536884+0000","last_clean":"2024-09-17T07:03:45.536884+0000","last_became_active":"2024-09-17T07:02:16.020628+0000","last_became_peered":"2024-09-17T07:02:16.020628+0000","last_unstale":"2024-09-17T07:03:45.536884+0000","last_undegraded":"2024-09-17T07:03:45.536884+0000","last_fullsized":"2024-09-17T07:03:45.536884+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:03:33.444104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:33.444104+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:36.767124+0000","scrub_duration":0.0014124389999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:44.061566+0000","last_change":"2024-09-17T07:03:19.030574+0000","last_active":"2024-09-17T07:03:44.061566+0000","last_peered":"2024-09-17T07:03:44.061566+0000","last_clean":"2024-09-17T07:03:44.061566+0000","last_became_active":"2024-09-17T07:02:18.364981+0000","last_became_peered":"2024-09-17T07:02:18.364981+0000","last_unstale":"2024-09-17T07:03:44.061566+0000","last_undegraded":"2024-09-17T07:03:44.061566+0000","last_fullsized":"2024-09-17T07:03:44.061566+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:19.030470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:03:19.030470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:48.307795+0000","scrub_duration":0.00152618,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"75'18","reported_seq":74,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:43.734268+0000","last_change":"2024-09-17T07:03:26.413610+0000","last_active":"2024-09-17T07:03:43.734268+0000","last_peered":"2024-09-17T07:03:43.734268+0000","last_clean":"2024-09-17T07:03:43.734268+0000","last_became_active":"2024-09-17T07:02:14.028667+0000","last_became_peered":"2024-09-17T07:02:14.028667+0000","last_unstale":"2024-09-17T07:03:43.734268+0000","last_undegraded":"2024-09-17T07:03:43.734268+0000","last_fullsized":"2024-09-17T07:03:43.734268+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'18","last_scrub_stamp":"2024-09-17T07:03:26.413506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:26.413506+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:41.741155+0000","scrub_duration":0.0022855470000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":44,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:44.045416+0000","last_change":"2024-09-17T07:03:24.134708+0000","last_active":"2024-09-17T07:03:44.045416+0000","last_peered":"2024-09-17T07:03:44.045416+0000","last_clean":"2024-09-17T07:03:44.045416+0000","last_became_active":"2024-09-17T07:02:12.005687+0000","last_became_peered":"2024-09-17T07:02:12.005687+0000","last_unstale":"2024-09-17T07:03:44.045416+0000","last_undegraded":"2024-09-17T07:03:44.045416+0000","last_fullsized":"2024-09-17T07:03:44.045416+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:24.134627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:24.134627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:29.080338+0000","scrub_duration":0.001111148,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:43.734832+0000","last_change":"2024-09-17T07:02:16.020013+0000","last_active":"2024-09-17T07:03:43.734832+0000","last_peered":"2024-09-17T07:03:43.734832+0000","last_clean":"2024-09-17T07:03:43.734832+0000","last_became_active":"2024-09-17T07:02:16.019747+0000","last_became_peered":"2024-09-17T07:02:16.019747+0000","last_unstale":"2024-09-17T07:03:43.734832+0000","last_undegraded":"2024-09-17T07:03:43.734832+0000","last_fullsized":"2024-09-17T07:03:43.734832+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:02:14.999435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:44.196107+0000","last_change":"2024-09-17T07:03:44.196107+0000","last_active":"2024-09-17T07:03:44.196107+0000","last_peered":"2024-09-17T07:03:44.196107+0000","last_clean":"2024-09-17T07:03:44.196107+0000","last_became_active":"2024-09-17T07:02:18.024043+0000","last_became_peered":"2024-09-17T07:02:18.024043+0000","last_unstale":"2024-09-17T07:03:44.196107+0000","last_undegraded":"2024-09-17T07:03:44.196107+0000","last_fullsized":"2024-09-17T07:03:44.196107+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:44.196035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:03:44.196035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:12.177838+0000","scrub_duration":0.0015579439999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"75'19","reported_seq":79,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:43.736808+0000","last_change":"2024-09-17T07:03:40.949557+0000","last_active":"2024-09-17T07:03:43.736808+0000","last_peered":"2024-09-17T07:03:43.736808+0000","last_clean":"2024-09-17T07:03:43.736808+0000","last_became_active":"2024-09-17T07:02:14.027389+0000","last_became_peered":"2024-09-17T07:02:14.027389+0000","last_unstale":"2024-09-17T07:03:43.736808+0000","last_undegraded":"2024-09-17T07:03:43.736808+0000","last_fullsized":"2024-09-17T07:03:43.736808+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-09-17T07:03:40.949458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:40.949458+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:03.948689+0000","scrub_duration":0.0025476489999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:44.062017+0000","last_change":"2024-09-17T07:03:32.109842+0000","last_active":"2024-09-17T07:03:44.062017+0000","last_peered":"2024-09-17T07:03:44.062017+0000","last_clean":"2024-09-17T07:03:44.062017+0000","last_became_active":"2024-09-17T07:02:12.011917+0000","last_became_peered":"2024-09-17T07:02:12.011917+0000","last_unstale":"2024-09-17T07:03:44.062017+0000","last_undegraded":"2024-09-17T07:03:44.062017+0000","last_fullsized":"2024-09-17T07:03:44.062017+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:32.109765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:32.109765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:52.813415+0000","scrub_duration":0.001335314,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:44.445071+0000","last_change":"2024-09-17T07:02:16.016938+0000","last_active":"2024-09-17T07:03:44.445071+0000","last_peered":"2024-09-17T07:03:44.445071+0000","last_clean":"2024-09-17T07:03:44.445071+0000","last_became_active":"2024-09-17T07:02:16.016199+0000","last_became_peered":"2024-09-17T07:02:16.016199+0000","last_unstale":"2024-09-17T07:03:44.445071+0000","last_undegraded":"2024-09-17T07:03:44.445071+0000","last_fullsized":"2024-09-17T07:03:44.445071+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:02:14.999435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:44.044497+0000","last_change":"2024-09-17T07:03:30.068895+0000","last_active":"2024-09-17T07:03:44.044497+0000","last_peered":"2024-09-17T07:03:44.044497+0000","last_clean":"2024-09-17T07:03:44.044497+0000","last_became_active":"2024-09-17T07:02:18.364835+0000","last_became_peered":"2024-09-17T07:02:18.364835+0000","last_unstale":"2024-09-17T07:03:44.044497+0000","last_undegraded":"2024-09-17T07:03:44.044497+0000","last_fullsized":"2024-09-17T07:03:44.044497+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:30.068808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:03:30.068808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:51.270374+0000","scrub_duration":0.0012893679999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"75'10","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:44.045075+0000","last_change":"2024-09-17T07:03:26.127545+0000","last_active":"2024-09-17T07:03:44.045075+0000","last_peered":"2024-09-17T07:03:44.045075+0000","last_clean":"2024-09-17T07:03:44.045075+0000","last_became_active":"2024-09-17T07:02:14.010637+0000","last_became_peered":"2024-09-17T07:02:14.010637+0000","last_unstale":"2024-09-17T07:03:44.045075+0000","last_undegraded":"2024-09-17T07:03:44.045075+0000","last_fullsized":"2024-09-17T07:03:44.045075+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-09-17T07:03:26.127439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:26.127439+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:35.535770+0000","scrub_duration":0.001898479,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":44,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:44.045293+0000","last_change":"2024-09-17T07:03:25.148144+0000","last_active":"2024-09-17T07:03:44.045293+0000","last_peered":"2024-09-17T07:03:44.045293+0000","last_clean":"2024-09-17T07:03:44.045293+0000","last_became_active":"2024-09-17T07:02:12.010803+0000","last_became_peered":"2024-09-17T07:02:12.010803+0000","last_unstale":"2024-09-17T07:03:44.045293+0000","last_undegraded":"2024-09-17T07:03:44.045293+0000","last_fullsized":"2024-09-17T07:03:44.045293+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:25.147878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:25.147878+0000","last_clean_scrub_stamp":"2024-09-17T07:03:25.147878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:33.071698+0000","scrub_duration":0.001353723,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"89'9","reported_seq":69,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:45.536847+0000","last_change":"2024-09-17T07:03:29.961568+0000","last_active":"2024-09-17T07:03:45.536847+0000","last_peered":"2024-09-17T07:03:45.536847+0000","last_clean":"2024-09-17T07:03:45.536847+0000","last_became_active":"2024-09-17T07:02:16.021360+0000","last_became_peered":"2024-09-17T07:02:16.021360+0000","last_unstale":"2024-09-17T07:03:45.536847+0000","last_undegraded":"2024-09-17T07:03:45.536847+0000","last_fullsized":"2024-09-17T07:03:45.536847+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'9","last_scrub_stamp":"2024-09-17T07:03:29.961439+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-09-17T07:03:29.961439+0000","last_clean_scrub_stamp":"2024-09-17T07:03:29.961439+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:40.753813+0000","scrub_duration":0.0027696880000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"75'1","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:43.735766+0000","last_change":"2024-09-17T07:03:25.049036+0000","last_active":"2024-09-17T07:03:43.735766+0000","last_peered":"2024-09-17T07:03:43.735766+0000","last_clean":"2024-09-17T07:03:43.735766+0000","last_became_active":"2024-09-17T07:02:18.029772+0000","last_became_peered":"2024-09-17T07:02:18.029772+0000","last_unstale":"2024-09-17T07:03:43.735766+0000","last_undegraded":"2024-09-17T07:03:43.735766+0000","last_fullsized":"2024-09-17T07:03:43.735766+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-17T07:03:25.048949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:03:25.048949+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:53.755876+0000","scrub_duration":0.003886123,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"75'28","reported_seq":98,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:45.536838+0000","last_change":"2024-09-17T07:02:14.017598+0000","last_active":"2024-09-17T07:03:45.536838+0000","last_peered":"2024-09-17T07:03:45.536838+0000","last_clean":"2024-09-17T07:03:45.536838+0000","last_became_active":"2024-09-17T07:02:14.017377+0000","last_became_peered":"2024-09-17T07:02:14.017377+0000","last_unstale":"2024-09-17T07:03:45.536838+0000","last_undegraded":"2024-09-17T07:03:45.536838+0000","last_fullsized":"2024-09-17T07:03:45.536838+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:02:12.994221+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":44,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:44.449134+0000","last_change":"2024-09-17T07:03:40.811355+0000","last_active":"2024-09-17T07:03:44.449134+0000","last_peered":"2024-09-17T07:03:44.449134+0000","last_clean":"2024-09-17T07:03:44.449134+0000","last_became_active":"2024-09-17T07:02:12.008802+0000","last_became_peered":"2024-09-17T07:02:12.008802+0000","last_unstale":"2024-09-17T07:03:44.449134+0000","last_undegraded":"2024-09-17T07:03:44.449134+0000","last_fullsized":"2024-09-17T07:03:44.449134+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:40.811273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:40.811273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:52.601572+0000","scrub_duration":0.0012956230000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":41,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:43.736194+0000","last_change":"2024-09-17T07:03:28.981244+0000","last_active":"2024-09-17T07:03:43.736194+0000","last_peered":"2024-09-17T07:03:43.736194+0000","last_clean":"2024-09-17T07:03:43.736194+0000","last_became_active":"2024-09-17T07:02:16.016606+0000","last_became_peered":"2024-09-17T07:02:16.016606+0000","last_unstale":"2024-09-17T07:03:43.736194+0000","last_undegraded":"2024-09-17T07:03:43.736194+0000","last_fullsized":"2024-09-17T07:03:43.736194+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:28.981147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:28.981147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:37.078325+0000","scrub_duration":0.0012741460000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:43.735302+0000","last_change":"2024-09-17T07:02:18.026015+0000","last_active":"2024-09-17T07:03:43.735302+0000","last_peered":"2024-09-17T07:03:43.735302+0000","last_clean":"2024-09-17T07:03:43.735302+0000","last_became_active":"2024-09-17T07:02:18.024602+0000","last_became_peered":"2024-09-17T07:02:18.024602+0000","last_unstale":"2024-09-17T07:03:43.735302+0000","last_undegraded":"2024-09-17T07:03:43.735302+0000","last_fullsized":"2024-09-17T07:03:43.735302+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:02:17.002904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"75'13","reported_seq":69,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:44.045507+0000","last_change":"2024-09-17T07:03:32.079678+0000","last_active":"2024-09-17T07:03:44.045507+0000","last_peered":"2024-09-17T07:03:44.045507+0000","last_clean":"2024-09-17T07:03:44.045507+0000","last_became_active":"2024-09-17T07:02:14.019838+0000","last_became_peered":"2024-09-17T07:02:14.019838+0000","last_unstale":"2024-09-17T07:03:44.045507+0000","last_undegraded":"2024-09-17T07:03:44.045507+0000","last_fullsized":"2024-09-17T07:03:44.045507+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'13","last_scrub_stamp":"2024-09-17T07:03:32.079579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:32.079579+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:42.502680+0000","scrub_duration":0.001975929,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"76'5","reported_seq":130,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.126315+0000","last_change":"2024-09-17T07:02:16.008848+0000","last_active":"2024-09-17T07:03:47.126315+0000","last_peered":"2024-09-17T07:03:47.126315+0000","last_clean":"2024-09-17T07:03:47.126315+0000","last_became_active":"2024-09-17T07:02:12.006883+0000","last_became_peered":"2024-09-17T07:02:12.006883+0000","last_unstale":"2024-09-17T07:03:47.126315+0000","last_undegraded":"2024-09-17T07:03:47.126315+0000","last_fullsized":"2024-09-17T07:03:47.126315+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:02:10.991363+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033739799999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":89,"num_read_kb":84,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:43.942698+0000","last_change":"2024-09-17T07:03:33.795388+0000","last_active":"2024-09-17T07:03:43.942698+0000","last_peered":"2024-09-17T07:03:43.942698+0000","last_clean":"2024-09-17T07:03:43.942698+0000","last_became_active":"2024-09-17T07:02:12.010753+0000","last_became_peered":"2024-09-17T07:02:12.010753+0000","last_unstale":"2024-09-17T07:03:43.942698+0000","last_undegraded":"2024-09-17T07:03:43.942698+0000","last_fullsized":"2024-09-17T07:03:43.942698+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:33.795307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:33.795307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:00.370993+0000","scrub_duration":0.0012164070000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:43.943073+0000","last_change":"2024-09-17T07:02:16.014775+0000","last_active":"2024-09-17T07:03:43.943073+0000","last_peered":"2024-09-17T07:03:43.943073+0000","last_clean":"2024-09-17T07:03:43.943073+0000","last_became_active":"2024-09-17T07:02:16.014492+0000","last_became_peered":"2024-09-17T07:02:16.014492+0000","last_unstale":"2024-09-17T07:03:43.943073+0000","last_undegraded":"2024-09-17T07:03:43.943073+0000","last_fullsized":"2024-09-17T07:03:43.943073+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:02:14.999435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:44.061866+0000","last_change":"2024-09-17T07:03:42.068886+0000","last_active":"2024-09-17T07:03:44.061866+0000","last_peered":"2024-09-17T07:03:44.061866+0000","last_clean":"2024-09-17T07:03:44.061866+0000","last_became_active":"2024-09-17T07:02:18.365369+0000","last_became_peered":"2024-09-17T07:02:18.365369+0000","last_unstale":"2024-09-17T07:03:44.061866+0000","last_undegraded":"2024-09-17T07:03:44.061866+0000","last_fullsized":"2024-09-17T07:03:44.061866+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:42.068803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:03:42.068803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:47.482056+0000","scrub_duration":0.0015781599999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"75'12","reported_seq":62,"reported_epoch":92,"state":"peering","last_fresh":"2024-09-17T07:03:47.753729+0000","last_change":"2024-09-17T07:03:47.753914+0000","last_active":"2024-09-17T07:03:47.753729+0000","last_peered":"2024-09-17T07:03:43.736635+0000","last_clean":"2024-09-17T07:03:43.736635+0000","last_became_active":"2024-09-17T07:02:14.016206+0000","last_became_peered":"2024-09-17T07:02:14.016206+0000","last_unstale":"2024-09-17T07:03:47.753729+0000","last_undegraded":"2024-09-17T07:03:47.753729+0000","last_fullsized":"2024-09-17T07:03:47.753729+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-09-17T07:03:27.987835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:27.987835+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:36.887140+0000","scrub_duration":0.0020148219999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2,5],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"70'1","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:43.736147+0000","last_change":"2024-09-17T07:02:16.017796+0000","last_active":"2024-09-17T07:03:43.736147+0000","last_peered":"2024-09-17T07:03:43.736147+0000","last_clean":"2024-09-17T07:03:43.736147+0000","last_became_active":"2024-09-17T07:02:12.009476+0000","last_became_peered":"2024-09-17T07:02:12.009476+0000","last_unstale":"2024-09-17T07:03:43.736147+0000","last_undegraded":"2024-09-17T07:03:43.736147+0000","last_fullsized":"2024-09-17T07:03:43.736147+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:02:10.991363+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032298629999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:44.061630+0000","last_change":"2024-09-17T07:02:12.014798+0000","last_active":"2024-09-17T07:03:44.061630+0000","last_peered":"2024-09-17T07:03:44.061630+0000","last_clean":"2024-09-17T07:03:44.061630+0000","last_became_active":"2024-09-17T07:02:12.014623+0000","last_became_peered":"2024-09-17T07:02:12.014623+0000","last_unstale":"2024-09-17T07:03:44.061630+0000","last_undegraded":"2024-09-17T07:03:44.061630+0000","last_fullsized":"2024-09-17T07:03:44.061630+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:02:10.991363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:44.448670+0000","last_change":"2024-09-17T07:03:38.818472+0000","last_active":"2024-09-17T07:03:44.448670+0000","last_peered":"2024-09-17T07:03:44.448670+0000","last_clean":"2024-09-17T07:03:44.448670+0000","last_became_active":"2024-09-17T07:02:16.021561+0000","last_became_peered":"2024-09-17T07:02:16.021561+0000","last_unstale":"2024-09-17T07:03:44.448670+0000","last_undegraded":"2024-09-17T07:03:44.448670+0000","last_fullsized":"2024-09-17T07:03:44.448670+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:38.818329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:38.818329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:05.115055+0000","scrub_duration":0.0016477,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:44.045144+0000","last_change":"2024-09-17T07:02:18.022359+0000","last_active":"2024-09-17T07:03:44.045144+0000","last_peered":"2024-09-17T07:03:44.045144+0000","last_clean":"2024-09-17T07:03:44.045144+0000","last_became_active":"2024-09-17T07:02:18.022082+0000","last_became_peered":"2024-09-17T07:02:18.022082+0000","last_unstale":"2024-09-17T07:03:44.045144+0000","last_undegraded":"2024-09-17T07:03:44.045144+0000","last_fullsized":"2024-09-17T07:03:44.045144+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:02:17.002904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"75'14","reported_seq":64,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:44.522720+0000","last_change":"2024-09-17T07:03:44.522720+0000","last_active":"2024-09-17T07:03:44.522720+0000","last_peered":"2024-09-17T07:03:44.522720+0000","last_clean":"2024-09-17T07:03:44.522720+0000","last_became_active":"2024-09-17T07:02:14.028857+0000","last_became_peered":"2024-09-17T07:02:14.028857+0000","last_unstale":"2024-09-17T07:03:44.522720+0000","last_undegraded":"2024-09-17T07:03:44.522720+0000","last_fullsized":"2024-09-17T07:03:44.522720+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'14","last_scrub_stamp":"2024-09-17T07:03:44.522631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:44.522631+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:13.112891+0000","scrub_duration":0.0019970970000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"72'2","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:43.734653+0000","last_change":"2024-09-17T07:03:25.432755+0000","last_active":"2024-09-17T07:03:43.734653+0000","last_peered":"2024-09-17T07:03:43.734653+0000","last_clean":"2024-09-17T07:03:43.734653+0000","last_became_active":"2024-09-17T07:02:12.005127+0000","last_became_peered":"2024-09-17T07:02:12.005127+0000","last_unstale":"2024-09-17T07:03:43.734653+0000","last_undegraded":"2024-09-17T07:03:43.734653+0000","last_fullsized":"2024-09-17T07:03:43.734653+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'2","last_scrub_stamp":"2024-09-17T07:03:25.432670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:25.432670+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:25.994239+0000","scrub_duration":0.001640454,"objects_trimmed":0,"snaptrim_duration":0.0010980219999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":44,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:44.449076+0000","last_change":"2024-09-17T07:03:14.771416+0000","last_active":"2024-09-17T07:03:44.449076+0000","last_peered":"2024-09-17T07:03:44.449076+0000","last_clean":"2024-09-17T07:03:44.449076+0000","last_became_active":"2024-09-17T07:02:12.004606+0000","last_became_peered":"2024-09-17T07:02:12.004606+0000","last_unstale":"2024-09-17T07:03:44.449076+0000","last_undegraded":"2024-09-17T07:03:44.449076+0000","last_fullsized":"2024-09-17T07:03:44.449076+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:14.771307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:14.771307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:19.133578+0000","scrub_duration":0.0020495679999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"78'128","reported_seq":293,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:44.061739+0000","last_change":"2024-09-17T07:02:43.225880+0000","last_active":"2024-09-17T07:03:44.061739+0000","last_peered":"2024-09-17T07:03:44.061739+0000","last_clean":"2024-09-17T07:03:44.061739+0000","last_became_active":"2024-09-17T07:02:03.020675+0000","last_became_peered":"2024-09-17T07:02:03.020675+0000","last_unstale":"2024-09-17T07:03:44.061739+0000","last_undegraded":"2024-09-17T07:03:44.061739+0000","last_fullsized":"2024-09-17T07:03:44.061739+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'117","last_scrub_stamp":"2024-09-17T07:02:43.225790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:00:09.390381+0000","last_clean_scrub_stamp":"2024-09-17T07:02:43.225790+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:12.817117+0000","scrub_duration":0.001686302,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":41,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:44.061291+0000","last_change":"2024-09-17T07:03:30.069684+0000","last_active":"2024-09-17T07:03:44.061291+0000","last_peered":"2024-09-17T07:03:44.061291+0000","last_clean":"2024-09-17T07:03:44.061291+0000","last_became_active":"2024-09-17T07:02:16.016031+0000","last_became_peered":"2024-09-17T07:02:16.016031+0000","last_unstale":"2024-09-17T07:03:44.061291+0000","last_undegraded":"2024-09-17T07:03:44.061291+0000","last_fullsized":"2024-09-17T07:03:44.061291+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:30.069608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:30.069608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:36.831454+0000","scrub_duration":0.0012155379999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:44.449426+0000","last_change":"2024-09-17T07:02:18.023684+0000","last_active":"2024-09-17T07:03:44.449426+0000","last_peered":"2024-09-17T07:03:44.449426+0000","last_clean":"2024-09-17T07:03:44.449426+0000","last_became_active":"2024-09-17T07:02:18.023321+0000","last_became_peered":"2024-09-17T07:02:18.023321+0000","last_unstale":"2024-09-17T07:03:44.449426+0000","last_undegraded":"2024-09-17T07:03:44.449426+0000","last_fullsized":"2024-09-17T07:03:44.449426+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:02:17.002904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"75'19","reported_seq":73,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:44.445473+0000","last_change":"2024-09-17T07:03:37.517877+0000","last_active":"2024-09-17T07:03:44.445473+0000","last_peered":"2024-09-17T07:03:44.445473+0000","last_clean":"2024-09-17T07:03:44.445473+0000","last_became_active":"2024-09-17T07:02:14.027313+0000","last_became_peered":"2024-09-17T07:02:14.027313+0000","last_unstale":"2024-09-17T07:03:44.445473+0000","last_undegraded":"2024-09-17T07:03:44.445473+0000","last_fullsized":"2024-09-17T07:03:44.445473+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-09-17T07:03:37.517760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:37.517760+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:03.675532+0000","scrub_duration":0.0028316579999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:43.943042+0000","last_change":"2024-09-17T07:03:25.672437+0000","last_active":"2024-09-17T07:03:43.943042+0000","last_peered":"2024-09-17T07:03:43.943042+0000","last_clean":"2024-09-17T07:03:43.943042+0000","last_became_active":"2024-09-17T07:02:12.012037+0000","last_became_peered":"2024-09-17T07:02:12.012037+0000","last_unstale":"2024-09-17T07:03:43.943042+0000","last_undegraded":"2024-09-17T07:03:43.943042+0000","last_fullsized":"2024-09-17T07:03:43.943042+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:25.672360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:25.672360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:54.809122+0000","scrub_duration":0.0012704459999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":41,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:43.942788+0000","last_change":"2024-09-17T07:03:27.660802+0000","last_active":"2024-09-17T07:03:43.942788+0000","last_peered":"2024-09-17T07:03:43.942788+0000","last_clean":"2024-09-17T07:03:43.942788+0000","last_became_active":"2024-09-17T07:02:16.014631+0000","last_became_peered":"2024-09-17T07:02:16.014631+0000","last_unstale":"2024-09-17T07:03:43.942788+0000","last_undegraded":"2024-09-17T07:03:43.942788+0000","last_fullsized":"2024-09-17T07:03:43.942788+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:27.660721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:27.660721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:50.654559+0000","scrub_duration":0.0012746820000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:44.061648+0000","last_change":"2024-09-17T07:02:18.023600+0000","last_active":"2024-09-17T07:03:44.061648+0000","last_peered":"2024-09-17T07:03:44.061648+0000","last_clean":"2024-09-17T07:03:44.061648+0000","last_became_active":"2024-09-17T07:02:18.023299+0000","last_became_peered":"2024-09-17T07:02:18.023299+0000","last_unstale":"2024-09-17T07:03:44.061648+0000","last_undegraded":"2024-09-17T07:03:44.061648+0000","last_fullsized":"2024-09-17T07:03:44.061648+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:02:17.002904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"75'9","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:46.892737+0000","last_change":"2024-09-17T07:03:46.892737+0000","last_active":"2024-09-17T07:03:46.892737+0000","last_peered":"2024-09-17T07:03:46.892737+0000","last_clean":"2024-09-17T07:03:46.892737+0000","last_became_active":"2024-09-17T07:02:14.014939+0000","last_became_peered":"2024-09-17T07:02:14.014939+0000","last_unstale":"2024-09-17T07:03:46.892737+0000","last_undegraded":"2024-09-17T07:03:46.892737+0000","last_fullsized":"2024-09-17T07:03:46.892737+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:03:46.892648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:46.892648+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:54.689233+0000","scrub_duration":0.002021162,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":44,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:44.045422+0000","last_change":"2024-09-17T07:03:40.074926+0000","last_active":"2024-09-17T07:03:44.045422+0000","last_peered":"2024-09-17T07:03:44.045422+0000","last_clean":"2024-09-17T07:03:44.045422+0000","last_became_active":"2024-09-17T07:02:12.000005+0000","last_became_peered":"2024-09-17T07:02:12.000005+0000","last_unstale":"2024-09-17T07:03:44.045422+0000","last_undegraded":"2024-09-17T07:03:44.045422+0000","last_fullsized":"2024-09-17T07:03:44.045422+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:40.074839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:40.074839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:50.612094+0000","scrub_duration":0.0013162040000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":41,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:43.943297+0000","last_change":"2024-09-17T07:03:43.943297+0000","last_active":"2024-09-17T07:03:43.943297+0000","last_peered":"2024-09-17T07:03:43.943297+0000","last_clean":"2024-09-17T07:03:43.943297+0000","last_became_active":"2024-09-17T07:02:16.017271+0000","last_became_peered":"2024-09-17T07:02:16.017271+0000","last_unstale":"2024-09-17T07:03:43.943297+0000","last_undegraded":"2024-09-17T07:03:43.943297+0000","last_fullsized":"2024-09-17T07:03:43.943297+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:43.943168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:43.943168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:01.611368+0000","scrub_duration":0.0016514609999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:43.736054+0000","last_change":"2024-09-17T07:02:18.027276+0000","last_active":"2024-09-17T07:03:43.736054+0000","last_peered":"2024-09-17T07:03:43.736054+0000","last_clean":"2024-09-17T07:03:43.736054+0000","last_became_active":"2024-09-17T07:02:18.026994+0000","last_became_peered":"2024-09-17T07:02:18.026994+0000","last_unstale":"2024-09-17T07:03:43.736054+0000","last_undegraded":"2024-09-17T07:03:43.736054+0000","last_fullsized":"2024-09-17T07:03:43.736054+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:02:17.002904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"75'15","reported_seq":56,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:44.448916+0000","last_change":"2024-09-17T07:02:14.028644+0000","last_active":"2024-09-17T07:03:44.448916+0000","last_peered":"2024-09-17T07:03:44.448916+0000","last_clean":"2024-09-17T07:03:44.448916+0000","last_became_active":"2024-09-17T07:02:14.028392+0000","last_became_peered":"2024-09-17T07:02:14.028392+0000","last_unstale":"2024-09-17T07:03:44.448916+0000","last_undegraded":"2024-09-17T07:03:44.448916+0000","last_fullsized":"2024-09-17T07:03:44.448916+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:02:12.994221+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'2","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:43.943191+0000","last_change":"2024-09-17T07:03:26.694471+0000","last_active":"2024-09-17T07:03:43.943191+0000","last_peered":"2024-09-17T07:03:43.943191+0000","last_clean":"2024-09-17T07:03:43.943191+0000","last_became_active":"2024-09-17T07:02:12.005538+0000","last_became_peered":"2024-09-17T07:02:12.005538+0000","last_unstale":"2024-09-17T07:03:43.943191+0000","last_undegraded":"2024-09-17T07:03:43.943191+0000","last_fullsized":"2024-09-17T07:03:43.943191+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-09-17T07:03:26.694366+0000","last_deep_scrub":"89'2","last_deep_scrub_stamp":"2024-09-17T07:03:26.694366+0000","last_clean_scrub_stamp":"2024-09-17T07:03:26.694366+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:38.099234+0000","scrub_duration":0.0026359439999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":420,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"75'8","reported_seq":71,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:45.995460+0000","last_change":"2024-09-17T07:03:45.995460+0000","last_active":"2024-09-17T07:03:45.995460+0000","last_peered":"2024-09-17T07:03:45.995460+0000","last_clean":"2024-09-17T07:03:45.995460+0000","last_became_active":"2024-09-17T07:02:16.015418+0000","last_became_peered":"2024-09-17T07:02:16.015418+0000","last_unstale":"2024-09-17T07:03:45.995460+0000","last_undegraded":"2024-09-17T07:03:45.995460+0000","last_fullsized":"2024-09-17T07:03:45.995460+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:03:45.995358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:45.995358+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:52.038789+0000","scrub_duration":0.001465163,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:44.448913+0000","last_change":"2024-09-17T07:03:22.823018+0000","last_active":"2024-09-17T07:03:44.448913+0000","last_peered":"2024-09-17T07:03:44.448913+0000","last_clean":"2024-09-17T07:03:44.448913+0000","last_became_active":"2024-09-17T07:02:18.363253+0000","last_became_peered":"2024-09-17T07:02:18.363253+0000","last_unstale":"2024-09-17T07:03:44.448913+0000","last_undegraded":"2024-09-17T07:03:44.448913+0000","last_fullsized":"2024-09-17T07:03:44.448913+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:22.822922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:03:22.822922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:29.158808+0000","scrub_duration":0.0012979910000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"75'12","reported_seq":66,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:43.734884+0000","last_change":"2024-09-17T07:03:39.074060+0000","last_active":"2024-09-17T07:03:43.734884+0000","last_peered":"2024-09-17T07:03:43.734884+0000","last_clean":"2024-09-17T07:03:43.734884+0000","last_became_active":"2024-09-17T07:02:14.010219+0000","last_became_peered":"2024-09-17T07:02:14.010219+0000","last_unstale":"2024-09-17T07:03:43.734884+0000","last_undegraded":"2024-09-17T07:03:43.734884+0000","last_fullsized":"2024-09-17T07:03:43.734884+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-09-17T07:03:39.073980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:39.073980+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:07.174292+0000","scrub_duration":0.002114061,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"68'2","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:43.735474+0000","last_change":"2024-09-17T07:03:28.021306+0000","last_active":"2024-09-17T07:03:43.735474+0000","last_peered":"2024-09-17T07:03:43.735474+0000","last_clean":"2024-09-17T07:03:43.735474+0000","last_became_active":"2024-09-17T07:02:12.006716+0000","last_became_peered":"2024-09-17T07:02:12.006716+0000","last_unstale":"2024-09-17T07:03:43.735474+0000","last_undegraded":"2024-09-17T07:03:43.735474+0000","last_fullsized":"2024-09-17T07:03:43.735474+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-09-17T07:03:28.021180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:28.021180+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:55.299746+0000","scrub_duration":0.0016672220000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":41,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:43.942638+0000","last_change":"2024-09-17T07:03:36.832832+0000","last_active":"2024-09-17T07:03:43.942638+0000","last_peered":"2024-09-17T07:03:43.942638+0000","last_clean":"2024-09-17T07:03:43.942638+0000","last_became_active":"2024-09-17T07:02:16.012261+0000","last_became_peered":"2024-09-17T07:02:16.012261+0000","last_unstale":"2024-09-17T07:03:43.942638+0000","last_undegraded":"2024-09-17T07:03:43.942638+0000","last_fullsized":"2024-09-17T07:03:43.942638+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:36.832752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:36.832752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:44.624052+0000","scrub_duration":0.001189278,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:43.734842+0000","last_change":"2024-09-17T07:02:18.025590+0000","last_active":"2024-09-17T07:03:43.734842+0000","last_peered":"2024-09-17T07:03:43.734842+0000","last_clean":"2024-09-17T07:03:43.734842+0000","last_became_active":"2024-09-17T07:02:18.024983+0000","last_became_peered":"2024-09-17T07:02:18.024983+0000","last_unstale":"2024-09-17T07:03:43.734842+0000","last_undegraded":"2024-09-17T07:03:43.734842+0000","last_fullsized":"2024-09-17T07:03:43.734842+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:02:17.002904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"75'9","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:43.736568+0000","last_change":"2024-09-17T07:03:26.005290+0000","last_active":"2024-09-17T07:03:43.736568+0000","last_peered":"2024-09-17T07:03:43.736568+0000","last_clean":"2024-09-17T07:03:43.736568+0000","last_became_active":"2024-09-17T07:02:14.029388+0000","last_became_peered":"2024-09-17T07:02:14.029388+0000","last_unstale":"2024-09-17T07:03:43.736568+0000","last_undegraded":"2024-09-17T07:03:43.736568+0000","last_fullsized":"2024-09-17T07:03:43.736568+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:03:26.005201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:26.005201+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:55.043365+0000","scrub_duration":0.0018022279999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:43.942640+0000","last_change":"2024-09-17T07:03:31.753577+0000","last_active":"2024-09-17T07:03:43.942640+0000","last_peered":"2024-09-17T07:03:43.942640+0000","last_clean":"2024-09-17T07:03:43.942640+0000","last_became_active":"2024-09-17T07:02:12.011327+0000","last_became_peered":"2024-09-17T07:02:12.011327+0000","last_unstale":"2024-09-17T07:03:43.942640+0000","last_undegraded":"2024-09-17T07:03:43.942640+0000","last_fullsized":"2024-09-17T07:03:43.942640+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:31.753470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:31.753470+0000","last_clean_scrub_stamp":"2024-09-17T07:03:31.753470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:51.025434+0000","scrub_duration":0.0012687670000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:43.735275+0000","last_change":"2024-09-17T07:03:21.314310+0000","last_active":"2024-09-17T07:03:43.735275+0000","last_peered":"2024-09-17T07:03:43.735275+0000","last_clean":"2024-09-17T07:03:43.735275+0000","last_became_active":"2024-09-17T07:02:16.022584+0000","last_became_peered":"2024-09-17T07:02:16.022584+0000","last_unstale":"2024-09-17T07:03:43.735275+0000","last_undegraded":"2024-09-17T07:03:43.735275+0000","last_fullsized":"2024-09-17T07:03:43.735275+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:21.314226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:21.314226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:29.256168+0000","scrub_duration":0.0016627219999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:43.942765+0000","last_change":"2024-09-17T07:03:19.603974+0000","last_active":"2024-09-17T07:03:43.942765+0000","last_peered":"2024-09-17T07:03:43.942765+0000","last_clean":"2024-09-17T07:03:43.942765+0000","last_became_active":"2024-09-17T07:02:18.017825+0000","last_became_peered":"2024-09-17T07:02:18.017825+0000","last_unstale":"2024-09-17T07:03:43.942765+0000","last_undegraded":"2024-09-17T07:03:43.942765+0000","last_fullsized":"2024-09-17T07:03:43.942765+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:19.603899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:03:19.603899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:31.400546+0000","scrub_duration":0.001425798,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:44.445376+0000","last_change":"2024-09-17T07:02:12.008326+0000","last_active":"2024-09-17T07:03:44.445376+0000","last_peered":"2024-09-17T07:03:44.445376+0000","last_clean":"2024-09-17T07:03:44.445376+0000","last_became_active":"2024-09-17T07:02:12.008065+0000","last_became_peered":"2024-09-17T07:02:12.008065+0000","last_unstale":"2024-09-17T07:03:44.445376+0000","last_undegraded":"2024-09-17T07:03:44.445376+0000","last_fullsized":"2024-09-17T07:03:44.445376+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:02:10.991363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"75'6","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:43.735361+0000","last_change":"2024-09-17T07:03:36.430683+0000","last_active":"2024-09-17T07:03:43.735361+0000","last_peered":"2024-09-17T07:03:43.735361+0000","last_clean":"2024-09-17T07:03:43.735361+0000","last_became_active":"2024-09-17T07:02:14.028655+0000","last_became_peered":"2024-09-17T07:02:14.028655+0000","last_unstale":"2024-09-17T07:03:43.735361+0000","last_undegraded":"2024-09-17T07:03:43.735361+0000","last_fullsized":"2024-09-17T07:03:43.735361+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'6","last_scrub_stamp":"2024-09-17T07:03:36.430591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:36.430591+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:44.221735+0000","scrub_duration":0.0020997730000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:44.448652+0000","last_change":"2024-09-17T07:03:39.844283+0000","last_active":"2024-09-17T07:03:44.448652+0000","last_peered":"2024-09-17T07:03:44.448652+0000","last_clean":"2024-09-17T07:03:44.448652+0000","last_became_active":"2024-09-17T07:02:16.016634+0000","last_became_peered":"2024-09-17T07:02:16.016634+0000","last_unstale":"2024-09-17T07:03:44.448652+0000","last_undegraded":"2024-09-17T07:03:44.448652+0000","last_fullsized":"2024-09-17T07:03:44.448652+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:39.844196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:39.844196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:07.747427+0000","scrub_duration":0.001646652,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:44.061836+0000","last_change":"2024-09-17T07:03:41.093127+0000","last_active":"2024-09-17T07:03:44.061836+0000","last_peered":"2024-09-17T07:03:44.061836+0000","last_clean":"2024-09-17T07:03:44.061836+0000","last_became_active":"2024-09-17T07:02:18.365419+0000","last_became_peered":"2024-09-17T07:02:18.365419+0000","last_unstale":"2024-09-17T07:03:44.061836+0000","last_undegraded":"2024-09-17T07:03:44.061836+0000","last_fullsized":"2024-09-17T07:03:44.061836+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:41.093049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:03:41.093049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:58.545295+0000","scrub_duration":0.001187794,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"75'10","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:43.735044+0000","last_change":"2024-09-17T07:03:14.178469+0000","last_active":"2024-09-17T07:03:43.735044+0000","last_peered":"2024-09-17T07:03:43.735044+0000","last_clean":"2024-09-17T07:03:43.735044+0000","last_became_active":"2024-09-17T07:02:14.027499+0000","last_became_peered":"2024-09-17T07:02:14.027499+0000","last_unstale":"2024-09-17T07:03:43.735044+0000","last_undegraded":"2024-09-17T07:03:43.735044+0000","last_fullsized":"2024-09-17T07:03:43.735044+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-09-17T07:03:14.178375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:14.178375+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:31.400568+0000","scrub_duration":0.0021187430000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":44,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:44.449095+0000","last_change":"2024-09-17T07:03:25.849841+0000","last_active":"2024-09-17T07:03:44.449095+0000","last_peered":"2024-09-17T07:03:44.449095+0000","last_clean":"2024-09-17T07:03:44.449095+0000","last_became_active":"2024-09-17T07:02:12.007280+0000","last_became_peered":"2024-09-17T07:02:12.007280+0000","last_unstale":"2024-09-17T07:03:44.449095+0000","last_undegraded":"2024-09-17T07:03:44.449095+0000","last_fullsized":"2024-09-17T07:03:44.449095+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:25.849698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:25.849698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:55.197183+0000","scrub_duration":0.0014293929999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"75'8","reported_seq":67,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:46.807330+0000","last_change":"2024-09-17T07:03:46.807330+0000","last_active":"2024-09-17T07:03:46.807330+0000","last_peered":"2024-09-17T07:03:46.807330+0000","last_clean":"2024-09-17T07:03:46.807330+0000","last_became_active":"2024-09-17T07:02:16.022583+0000","last_became_peered":"2024-09-17T07:02:16.022583+0000","last_unstale":"2024-09-17T07:03:46.807330+0000","last_undegraded":"2024-09-17T07:03:46.807330+0000","last_fullsized":"2024-09-17T07:03:46.807330+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:03:46.807209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:46.807209+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:03.451386+0000","scrub_duration":0.0021619619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:43.736297+0000","last_change":"2024-09-17T07:02:18.029691+0000","last_active":"2024-09-17T07:03:43.736297+0000","last_peered":"2024-09-17T07:03:43.736297+0000","last_clean":"2024-09-17T07:03:43.736297+0000","last_became_active":"2024-09-17T07:02:18.028796+0000","last_became_peered":"2024-09-17T07:02:18.028796+0000","last_unstale":"2024-09-17T07:03:43.736297+0000","last_undegraded":"2024-09-17T07:03:43.736297+0000","last_fullsized":"2024-09-17T07:03:43.736297+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:02:17.002904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"75'9","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:44.449295+0000","last_change":"2024-09-17T07:03:17.798712+0000","last_active":"2024-09-17T07:03:44.449295+0000","last_peered":"2024-09-17T07:03:44.449295+0000","last_clean":"2024-09-17T07:03:44.449295+0000","last_became_active":"2024-09-17T07:02:14.028485+0000","last_became_peered":"2024-09-17T07:02:14.028485+0000","last_unstale":"2024-09-17T07:03:44.449295+0000","last_undegraded":"2024-09-17T07:03:44.449295+0000","last_fullsized":"2024-09-17T07:03:44.449295+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:03:17.798582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:17.798582+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:27.282186+0000","scrub_duration":0.0023213259999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:43.736858+0000","last_change":"2024-09-17T07:03:16.948419+0000","last_active":"2024-09-17T07:03:43.736858+0000","last_peered":"2024-09-17T07:03:43.736858+0000","last_clean":"2024-09-17T07:03:43.736858+0000","last_became_active":"2024-09-17T07:02:12.009935+0000","last_became_peered":"2024-09-17T07:02:12.009935+0000","last_unstale":"2024-09-17T07:03:43.736858+0000","last_undegraded":"2024-09-17T07:03:43.736858+0000","last_fullsized":"2024-09-17T07:03:43.736858+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:16.948330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:16.948330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:46.609922+0000","scrub_duration":0.0017835489999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"75'8","reported_seq":67,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:45.536953+0000","last_change":"2024-09-17T07:03:42.469019+0000","last_active":"2024-09-17T07:03:45.536953+0000","last_peered":"2024-09-17T07:03:45.536953+0000","last_clean":"2024-09-17T07:03:45.536953+0000","last_became_active":"2024-09-17T07:02:16.016069+0000","last_became_peered":"2024-09-17T07:02:16.016069+0000","last_unstale":"2024-09-17T07:03:45.536953+0000","last_undegraded":"2024-09-17T07:03:45.536953+0000","last_fullsized":"2024-09-17T07:03:45.536953+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:03:42.468934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:42.468934+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:07.697686+0000","scrub_duration":0.0016937079999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:43.735300+0000","last_change":"2024-09-17T07:03:31.045257+0000","last_active":"2024-09-17T07:03:43.735300+0000","last_peered":"2024-09-17T07:03:43.735300+0000","last_clean":"2024-09-17T07:03:43.735300+0000","last_became_active":"2024-09-17T07:02:18.018716+0000","last_became_peered":"2024-09-17T07:02:18.018716+0000","last_unstale":"2024-09-17T07:03:43.735300+0000","last_undegraded":"2024-09-17T07:03:43.735300+0000","last_fullsized":"2024-09-17T07:03:43.735300+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:31.045167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:03:31.045167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:36.218836+0000","scrub_duration":0.0012599709999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"75'9","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:44.044982+0000","last_change":"2024-09-17T07:03:20.116712+0000","last_active":"2024-09-17T07:03:44.044982+0000","last_peered":"2024-09-17T07:03:44.044982+0000","last_clean":"2024-09-17T07:03:44.044982+0000","last_became_active":"2024-09-17T07:02:14.010289+0000","last_became_peered":"2024-09-17T07:02:14.010289+0000","last_unstale":"2024-09-17T07:03:44.044982+0000","last_undegraded":"2024-09-17T07:03:44.044982+0000","last_fullsized":"2024-09-17T07:03:44.044982+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:03:20.116602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:20.116602+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:35.267610+0000","scrub_duration":0.0032469130000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:44.445714+0000","last_change":"2024-09-17T07:02:12.007881+0000","last_active":"2024-09-17T07:03:44.445714+0000","last_peered":"2024-09-17T07:03:44.445714+0000","last_clean":"2024-09-17T07:03:44.445714+0000","last_became_active":"2024-09-17T07:02:12.007433+0000","last_became_peered":"2024-09-17T07:02:12.007433+0000","last_unstale":"2024-09-17T07:03:44.445714+0000","last_undegraded":"2024-09-17T07:03:44.445714+0000","last_fullsized":"2024-09-17T07:03:44.445714+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:02:10.991363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:43.734922+0000","last_change":"2024-09-17T07:02:16.016504+0000","last_active":"2024-09-17T07:03:43.734922+0000","last_peered":"2024-09-17T07:03:43.734922+0000","last_clean":"2024-09-17T07:03:43.734922+0000","last_became_active":"2024-09-17T07:02:16.016216+0000","last_became_peered":"2024-09-17T07:02:16.016216+0000","last_unstale":"2024-09-17T07:03:43.734922+0000","last_undegraded":"2024-09-17T07:03:43.734922+0000","last_fullsized":"2024-09-17T07:03:43.734922+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:02:14.999435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:43.735687+0000","last_change":"2024-09-17T07:03:24.014436+0000","last_active":"2024-09-17T07:03:43.735687+0000","last_peered":"2024-09-17T07:03:43.735687+0000","last_clean":"2024-09-17T07:03:43.735687+0000","last_became_active":"2024-09-17T07:02:18.027622+0000","last_became_peered":"2024-09-17T07:02:18.027622+0000","last_unstale":"2024-09-17T07:03:43.735687+0000","last_undegraded":"2024-09-17T07:03:43.735687+0000","last_fullsized":"2024-09-17T07:03:43.735687+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:24.014336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:03:24.014336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:50.998219+0000","scrub_duration":0.001425125,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"75'11","reported_seq":62,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:43.734946+0000","last_change":"2024-09-17T07:03:18.899916+0000","last_active":"2024-09-17T07:03:43.734946+0000","last_peered":"2024-09-17T07:03:43.734946+0000","last_clean":"2024-09-17T07:03:43.734946+0000","last_became_active":"2024-09-17T07:02:14.016255+0000","last_became_peered":"2024-09-17T07:02:14.016255+0000","last_unstale":"2024-09-17T07:03:43.734946+0000","last_undegraded":"2024-09-17T07:03:43.734946+0000","last_fullsized":"2024-09-17T07:03:43.734946+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-09-17T07:03:18.899814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:18.899814+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:46.161729+0000","scrub_duration":0.0020491419999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"68'1","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:44.045390+0000","last_change":"2024-09-17T07:02:12.011847+0000","last_active":"2024-09-17T07:03:44.045390+0000","last_peered":"2024-09-17T07:03:44.045390+0000","last_clean":"2024-09-17T07:03:44.045390+0000","last_became_active":"2024-09-17T07:02:12.011461+0000","last_became_peered":"2024-09-17T07:02:12.011461+0000","last_unstale":"2024-09-17T07:03:44.045390+0000","last_undegraded":"2024-09-17T07:03:44.045390+0000","last_fullsized":"2024-09-17T07:03:44.045390+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:02:10.991363+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:44.044066+0000","last_change":"2024-09-17T07:03:33.126060+0000","last_active":"2024-09-17T07:03:44.044066+0000","last_peered":"2024-09-17T07:03:44.044066+0000","last_clean":"2024-09-17T07:03:44.044066+0000","last_became_active":"2024-09-17T07:02:16.018754+0000","last_became_peered":"2024-09-17T07:02:16.018754+0000","last_unstale":"2024-09-17T07:03:44.044066+0000","last_undegraded":"2024-09-17T07:03:44.044066+0000","last_fullsized":"2024-09-17T07:03:44.044066+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:33.125915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:33.125915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:56.098551+0000","scrub_duration":0.033614225999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:43.734278+0000","last_change":"2024-09-17T07:02:18.027727+0000","last_active":"2024-09-17T07:03:43.734278+0000","last_peered":"2024-09-17T07:03:43.734278+0000","last_clean":"2024-09-17T07:03:43.734278+0000","last_became_active":"2024-09-17T07:02:18.027427+0000","last_became_peered":"2024-09-17T07:02:18.027427+0000","last_unstale":"2024-09-17T07:03:43.734278+0000","last_undegraded":"2024-09-17T07:03:43.734278+0000","last_fullsized":"2024-09-17T07:03:43.734278+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:02:17.002904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"75'4","reported_seq":49,"reported_epoch":92,"state":"peering","last_fresh":"2024-09-17T07:03:47.757229+0000","last_change":"2024-09-17T07:03:47.757381+0000","last_active":"2024-09-17T07:03:47.757229+0000","last_peered":"2024-09-17T07:03:43.734184+0000","last_clean":"2024-09-17T07:03:43.734184+0000","last_became_active":"2024-09-17T07:02:14.014779+0000","last_became_peered":"2024-09-17T07:02:14.014779+0000","last_unstale":"2024-09-17T07:03:47.757229+0000","last_undegraded":"2024-09-17T07:03:47.757229+0000","last_fullsized":"2024-09-17T07:03:47.757229+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'4","last_scrub_stamp":"2024-09-17T07:03:24.431043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:24.431043+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:41.002655+0000","scrub_duration":0.0015094500000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2,6],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":44,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:44.809127+0000","last_change":"2024-09-17T07:03:44.809127+0000","last_active":"2024-09-17T07:03:44.809127+0000","last_peered":"2024-09-17T07:03:44.809127+0000","last_clean":"2024-09-17T07:03:44.809127+0000","last_became_active":"2024-09-17T07:02:12.008898+0000","last_became_peered":"2024-09-17T07:02:12.008898+0000","last_unstale":"2024-09-17T07:03:44.809127+0000","last_undegraded":"2024-09-17T07:03:44.809127+0000","last_fullsized":"2024-09-17T07:03:44.809127+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:44.809025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:44.809025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:03.894700+0000","scrub_duration":0.001586154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.445076+0000","last_change":"2024-09-17T07:03:47.445076+0000","last_active":"2024-09-17T07:03:47.445076+0000","last_peered":"2024-09-17T07:03:47.445076+0000","last_clean":"2024-09-17T07:03:47.445076+0000","last_became_active":"2024-09-17T07:02:16.014982+0000","last_became_peered":"2024-09-17T07:02:16.014982+0000","last_unstale":"2024-09-17T07:03:47.445076+0000","last_undegraded":"2024-09-17T07:03:47.445076+0000","last_fullsized":"2024-09-17T07:03:47.445076+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:47.444972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:47.444972+0000","last_clean_scrub_stamp":"2024-09-17T07:03:47.444972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:56.111360+0000","scrub_duration":0.0015734119999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:44.061466+0000","last_change":"2024-09-17T07:02:18.021319+0000","last_active":"2024-09-17T07:03:44.061466+0000","last_peered":"2024-09-17T07:03:44.061466+0000","last_clean":"2024-09-17T07:03:44.061466+0000","last_became_active":"2024-09-17T07:02:18.021014+0000","last_became_peered":"2024-09-17T07:02:18.021014+0000","last_unstale":"2024-09-17T07:03:44.061466+0000","last_undegraded":"2024-09-17T07:03:44.061466+0000","last_fullsized":"2024-09-17T07:03:44.061466+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:02:17.002904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"75'11","reported_seq":61,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:43.734420+0000","last_change":"2024-09-17T07:03:17.270057+0000","last_active":"2024-09-17T07:03:43.734420+0000","last_peered":"2024-09-17T07:03:43.734420+0000","last_clean":"2024-09-17T07:03:43.734420+0000","last_became_active":"2024-09-17T07:02:14.027316+0000","last_became_peered":"2024-09-17T07:02:14.027316+0000","last_unstale":"2024-09-17T07:03:43.734420+0000","last_undegraded":"2024-09-17T07:03:43.734420+0000","last_fullsized":"2024-09-17T07:03:43.734420+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-09-17T07:03:17.269941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:17.269941+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:32.856982+0000","scrub_duration":0.002326505,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":44,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:44.445839+0000","last_change":"2024-09-17T07:03:33.533200+0000","last_active":"2024-09-17T07:03:44.445839+0000","last_peered":"2024-09-17T07:03:44.445839+0000","last_clean":"2024-09-17T07:03:44.445839+0000","last_became_active":"2024-09-17T07:02:12.010195+0000","last_became_peered":"2024-09-17T07:02:12.010195+0000","last_unstale":"2024-09-17T07:03:44.445839+0000","last_undegraded":"2024-09-17T07:03:44.445839+0000","last_fullsized":"2024-09-17T07:03:44.445839+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:33.533112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:33.533112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:53.826577+0000","scrub_duration":0.0014453420000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":41,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:44.061135+0000","last_change":"2024-09-17T07:03:37.063510+0000","last_active":"2024-09-17T07:03:44.061135+0000","last_peered":"2024-09-17T07:03:44.061135+0000","last_clean":"2024-09-17T07:03:44.061135+0000","last_became_active":"2024-09-17T07:02:16.016183+0000","last_became_peered":"2024-09-17T07:02:16.016183+0000","last_unstale":"2024-09-17T07:03:44.061135+0000","last_undegraded":"2024-09-17T07:03:44.061135+0000","last_fullsized":"2024-09-17T07:03:44.061135+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:37.063409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:37.063409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:55.794307+0000","scrub_duration":0.0015220329999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:43.734835+0000","last_change":"2024-09-17T07:03:29.467351+0000","last_active":"2024-09-17T07:03:43.734835+0000","last_peered":"2024-09-17T07:03:43.734835+0000","last_clean":"2024-09-17T07:03:43.734835+0000","last_became_active":"2024-09-17T07:02:18.365656+0000","last_became_peered":"2024-09-17T07:02:18.365656+0000","last_unstale":"2024-09-17T07:03:43.734835+0000","last_undegraded":"2024-09-17T07:03:43.734835+0000","last_fullsized":"2024-09-17T07:03:43.734835+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:29.467252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:03:29.467252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:58.741564+0000","scrub_duration":0.001842804,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"75'1","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:46.957523+0000","last_change":"2024-09-17T07:03:46.957523+0000","last_active":"2024-09-17T07:03:46.957523+0000","last_peered":"2024-09-17T07:03:46.957523+0000","last_clean":"2024-09-17T07:03:46.957523+0000","last_became_active":"2024-09-17T07:02:18.021855+0000","last_became_peered":"2024-09-17T07:02:18.021855+0000","last_unstale":"2024-09-17T07:03:46.957523+0000","last_undegraded":"2024-09-17T07:03:46.957523+0000","last_fullsized":"2024-09-17T07:03:46.957523+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-17T07:03:46.957414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:03:46.957414+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:53.461463+0000","scrub_duration":0.00186878,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":44,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:44.448980+0000","last_change":"2024-09-17T07:03:19.775506+0000","last_active":"2024-09-17T07:03:44.448980+0000","last_peered":"2024-09-17T07:03:44.448980+0000","last_clean":"2024-09-17T07:03:44.448980+0000","last_became_active":"2024-09-17T07:02:12.005329+0000","last_became_peered":"2024-09-17T07:02:12.005329+0000","last_unstale":"2024-09-17T07:03:44.448980+0000","last_undegraded":"2024-09-17T07:03:44.448980+0000","last_fullsized":"2024-09-17T07:03:44.448980+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:19.775390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:19.775390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:48.303892+0000","scrub_duration":0.0011915630000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"75'10","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:43.736489+0000","last_change":"2024-09-17T07:02:14.029020+0000","last_active":"2024-09-17T07:03:43.736489+0000","last_peered":"2024-09-17T07:03:43.736489+0000","last_clean":"2024-09-17T07:03:43.736489+0000","last_became_active":"2024-09-17T07:02:14.028731+0000","last_became_peered":"2024-09-17T07:02:14.028731+0000","last_unstale":"2024-09-17T07:03:43.736489+0000","last_undegraded":"2024-09-17T07:03:43.736489+0000","last_fullsized":"2024-09-17T07:03:43.736489+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:02:12.994221+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"75'8","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:45.536996+0000","last_change":"2024-09-17T07:02:16.020978+0000","last_active":"2024-09-17T07:03:45.536996+0000","last_peered":"2024-09-17T07:03:45.536996+0000","last_clean":"2024-09-17T07:03:45.536996+0000","last_became_active":"2024-09-17T07:02:16.020560+0000","last_became_peered":"2024-09-17T07:02:16.020560+0000","last_unstale":"2024-09-17T07:03:45.536996+0000","last_undegraded":"2024-09-17T07:03:45.536996+0000","last_fullsized":"2024-09-17T07:03:45.536996+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:02:14.999435+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:44.045207+0000","last_change":"2024-09-17T07:02:18.021959+0000","last_active":"2024-09-17T07:03:44.045207+0000","last_peered":"2024-09-17T07:03:44.045207+0000","last_clean":"2024-09-17T07:03:44.045207+0000","last_became_active":"2024-09-17T07:02:18.021603+0000","last_became_peered":"2024-09-17T07:02:18.021603+0000","last_unstale":"2024-09-17T07:03:44.045207+0000","last_undegraded":"2024-09-17T07:03:44.045207+0000","last_fullsized":"2024-09-17T07:03:44.045207+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:02:17.002904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":44,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:43.734704+0000","last_change":"2024-09-17T07:03:20.305248+0000","last_active":"2024-09-17T07:03:43.734704+0000","last_peered":"2024-09-17T07:03:43.734704+0000","last_clean":"2024-09-17T07:03:43.734704+0000","last_became_active":"2024-09-17T07:02:12.009837+0000","last_became_peered":"2024-09-17T07:02:12.009837+0000","last_unstale":"2024-09-17T07:03:43.734704+0000","last_undegraded":"2024-09-17T07:03:43.734704+0000","last_fullsized":"2024-09-17T07:03:43.734704+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:20.305165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:20.305165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:24.412968+0000","scrub_duration":0.0024485969999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"75'11","reported_seq":62,"reported_epoch":92,"state":"peering","last_fresh":"2024-09-17T07:03:47.755626+0000","last_change":"2024-09-17T07:03:47.755783+0000","last_active":"2024-09-17T07:03:47.755626+0000","last_peered":"2024-09-17T07:03:44.445830+0000","last_clean":"2024-09-17T07:03:44.445830+0000","last_became_active":"2024-09-17T07:02:14.011380+0000","last_became_peered":"2024-09-17T07:02:14.011380+0000","last_unstale":"2024-09-17T07:03:47.755626+0000","last_undegraded":"2024-09-17T07:03:47.755626+0000","last_fullsized":"2024-09-17T07:03:47.755626+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-09-17T07:03:24.473712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:24.473712+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:36.435926+0000","scrub_duration":0.0021033010000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2,5],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:43.736421+0000","last_change":"2024-09-17T07:02:16.019989+0000","last_active":"2024-09-17T07:03:43.736421+0000","last_peered":"2024-09-17T07:03:43.736421+0000","last_clean":"2024-09-17T07:03:43.736421+0000","last_became_active":"2024-09-17T07:02:16.019701+0000","last_became_peered":"2024-09-17T07:02:16.019701+0000","last_unstale":"2024-09-17T07:03:43.736421+0000","last_undegraded":"2024-09-17T07:03:43.736421+0000","last_fullsized":"2024-09-17T07:03:43.736421+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:02:14.999435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:43.735169+0000","last_change":"2024-09-17T07:02:18.364013+0000","last_active":"2024-09-17T07:03:43.735169+0000","last_peered":"2024-09-17T07:03:43.735169+0000","last_clean":"2024-09-17T07:03:43.735169+0000","last_became_active":"2024-09-17T07:02:18.363752+0000","last_became_peered":"2024-09-17T07:02:18.363752+0000","last_unstale":"2024-09-17T07:03:43.735169+0000","last_undegraded":"2024-09-17T07:03:43.735169+0000","last_fullsized":"2024-09-17T07:03:43.735169+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:02:17.002904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":45,"reported_epoch":92,"state":"peering","last_fresh":"2024-09-17T07:03:47.754963+0000","last_change":"2024-09-17T07:03:47.755118+0000","last_active":"2024-09-17T07:03:47.754963+0000","last_peered":"2024-09-17T07:03:46.447991+0000","last_clean":"2024-09-17T07:03:46.447991+0000","last_became_active":"2024-09-17T07:02:12.005149+0000","last_became_peered":"2024-09-17T07:02:12.005149+0000","last_unstale":"2024-09-17T07:03:47.754963+0000","last_undegraded":"2024-09-17T07:03:47.754963+0000","last_fullsized":"2024-09-17T07:03:47.754963+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:46.447902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:46.447902+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:05.624367+0000","scrub_duration":0.0011905419999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,4,7],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"75'15","reported_seq":68,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:43.942914+0000","last_change":"2024-09-17T07:03:16.605256+0000","last_active":"2024-09-17T07:03:43.942914+0000","last_peered":"2024-09-17T07:03:43.942914+0000","last_clean":"2024-09-17T07:03:43.942914+0000","last_became_active":"2024-09-17T07:02:14.016273+0000","last_became_peered":"2024-09-17T07:02:14.016273+0000","last_unstale":"2024-09-17T07:03:43.942914+0000","last_undegraded":"2024-09-17T07:03:43.942914+0000","last_fullsized":"2024-09-17T07:03:43.942914+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-09-17T07:03:16.605163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:16.605163+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:40.470486+0000","scrub_duration":0.002000763,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.013173+0000","last_change":"2024-09-17T07:03:47.013173+0000","last_active":"2024-09-17T07:03:47.013173+0000","last_peered":"2024-09-17T07:03:47.013173+0000","last_clean":"2024-09-17T07:03:47.013173+0000","last_became_active":"2024-09-17T07:02:16.015903+0000","last_became_peered":"2024-09-17T07:02:16.015903+0000","last_unstale":"2024-09-17T07:03:47.013173+0000","last_undegraded":"2024-09-17T07:03:47.013173+0000","last_fullsized":"2024-09-17T07:03:47.013173+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:47.013069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:47.013069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:05.642647+0000","scrub_duration":0.0012432350000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:43.735237+0000","last_change":"2024-09-17T07:02:18.366797+0000","last_active":"2024-09-17T07:03:43.735237+0000","last_peered":"2024-09-17T07:03:43.735237+0000","last_clean":"2024-09-17T07:03:43.735237+0000","last_became_active":"2024-09-17T07:02:18.366466+0000","last_became_peered":"2024-09-17T07:02:18.366466+0000","last_unstale":"2024-09-17T07:03:43.735237+0000","last_undegraded":"2024-09-17T07:03:43.735237+0000","last_fullsized":"2024-09-17T07:03:43.735237+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:02:17.002904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":44,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:44.446996+0000","last_change":"2024-09-17T07:03:44.446996+0000","last_active":"2024-09-17T07:03:44.446996+0000","last_peered":"2024-09-17T07:03:44.446996+0000","last_clean":"2024-09-17T07:03:44.446996+0000","last_became_active":"2024-09-17T07:02:12.005493+0000","last_became_peered":"2024-09-17T07:02:12.005493+0000","last_unstale":"2024-09-17T07:03:44.446996+0000","last_undegraded":"2024-09-17T07:03:44.446996+0000","last_fullsized":"2024-09-17T07:03:44.446996+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:44.446914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:44.446914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:06.273158+0000","scrub_duration":0.0029497619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"86'17","reported_seq":70,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:43.735027+0000","last_change":"2024-09-17T07:03:18.241178+0000","last_active":"2024-09-17T07:03:43.735027+0000","last_peered":"2024-09-17T07:03:43.735027+0000","last_clean":"2024-09-17T07:03:43.735027+0000","last_became_active":"2024-09-17T07:02:14.017737+0000","last_became_peered":"2024-09-17T07:02:14.017737+0000","last_unstale":"2024-09-17T07:03:43.735027+0000","last_undegraded":"2024-09-17T07:03:43.735027+0000","last_fullsized":"2024-09-17T07:03:43.735027+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'17","last_scrub_stamp":"2024-09-17T07:03:18.241067+0000","last_deep_scrub":"86'17","last_deep_scrub_stamp":"2024-09-17T07:03:18.241067+0000","last_clean_scrub_stamp":"2024-09-17T07:03:18.241067+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:35.855446+0000","scrub_duration":0.0047694219999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":41,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T07:03:43.734930+0000","last_change":"2024-09-17T07:03:26.978352+0000","last_active":"2024-09-17T07:03:43.734930+0000","last_peered":"2024-09-17T07:03:43.734930+0000","last_clean":"2024-09-17T07:03:43.734930+0000","last_became_active":"2024-09-17T07:02:16.017362+0000","last_became_peered":"2024-09-17T07:02:16.017362+0000","last_unstale":"2024-09-17T07:03:43.734930+0000","last_undegraded":"2024-09-17T07:03:43.734930+0000","last_fullsized":"2024-09-17T07:03:43.734930+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:26.978262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:26.978262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:38.363709+0000","scrub_duration":0.0016151990000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":396,"ondisk_log_size":396,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":89,"num_read_kb":84,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1386,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":12,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":4158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037785,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32496,"kb_used_data":1164,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93712656,"statfs":{"total":95995035648,"available":95961759744,"internally_reserved":0,"allocated":1191936,"data_stored":868851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:03:02 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.442},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.60799999999999998}]},{"osd":1,"last update":"Tue Sep 17 07:03:02 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.63900000000000001}]},{"osd":2,"last update":"Tue Sep 17 07:03:02 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.622,"5min":0.622,"15min":0.622},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.47199999999999998}]},{"osd":3,"last update":"Tue Sep 17 07:03:02 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.504},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.54700000000000004}]},{"osd":4,"last update":"Tue Sep 17 07:03:02 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"last":0.56899999999999995}]},{"osd":5,"last update":"Tue Sep 17 07:03:02 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.56000000000000005}]},{"osd":6,"last update":"Tue Sep 17 07:03:02 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.622}]}]},{"osd":6,"up_from":50,"seq":214748364828,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32464,"kb_used_data":1352,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93712688,"statfs":{"total":95995035648,"available":95961792512,"internally_reserved":0,"allocated":1384448,"data_stored":948215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:03:43 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.49299999999999999,"15min":0.497},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.66500000000000004,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.48899999999999999,"15min":0.48999999999999999},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.63100000000000001,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.53000000000000003}]},{"osd":1,"last update":"Tue Sep 17 07:03:43 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.48599999999999999,"15min":0.48799999999999999},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.61699999999999999,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.52300000000000002,"15min":0.53000000000000003},"min":{"1min":0.34999999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.55300000000000005,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.57599999999999996}]},{"osd":2,"last update":"Tue Sep 17 07:03:43 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.51500000000000001,"15min":0.51900000000000002},"min":{"1min":0.36299999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.68500000000000005,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51800000000000002,"15min":0.51900000000000002},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.64400000000000002,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.55800000000000005}]},{"osd":3,"last update":"Tue Sep 17 07:03:43 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.47699999999999998,"15min":0.47899999999999998},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.60499999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.53600000000000003,"15min":0.53800000000000003},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.67100000000000004,"5min":0.88,"15min":0.88},"last":0.59799999999999998}]},{"osd":4,"last update":"Tue Sep 17 07:03:43 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.41999999999999998,"15min":0.41999999999999998},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.52800000000000002,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.441},{"interface":"front","average":{"1min":0.443,"5min":0.41399999999999998,"15min":0.40899999999999997},"min":{"1min":0.32600000000000001,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.59599999999999997,"5min":0.628,"15min":0.628},"last":0.48099999999999998}]},{"osd":5,"last update":"Tue Sep 17 07:03:43 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.42999999999999999,"15min":0.42399999999999999},"min":{"1min":0.32800000000000001,"5min":0.31,"15min":0.31},"max":{"1min":0.60199999999999998,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.432,"15min":0.42799999999999999},"min":{"1min":0.313,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.42399999999999999}]},{"osd":7,"last update":"Tue Sep 17 07:03:03 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.48999999999999999}]}]},{"osd":5,"up_from":42,"seq":180388626465,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31908,"kb_used_data":572,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713244,"statfs":{"total":95995035648,"available":95962361856,"internally_reserved":0,"allocated":585728,"data_stored":278608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:03:15 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.50600000000000001,"15min":0.51500000000000001},"min":{"1min":0.34100000000000003,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.56699999999999995,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.505},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.497,"15min":0.505},"min":{"1min":0.31900000000000001,"5min":0.313,"15min":0.313},"max":{"1min":0.65500000000000003,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.58199999999999996}]},{"osd":1,"last update":"Tue Sep 17 07:03:15 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.53200000000000003,"15min":0.54200000000000004},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.60799999999999998,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.443,"5min":1.0449999999999999,"15min":1.145},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.56200000000000006,"5min":14.054,"15min":14.054},"last":0.51600000000000001}]},{"osd":2,"last update":"Tue Sep 17 07:03:15 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.54800000000000004,"15min":0.56000000000000005},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.66500000000000004,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.505,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.38600000000000001,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.63600000000000001,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.56899999999999995}]},{"osd":3,"last update":"Tue Sep 17 07:03:15 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.54000000000000004,"15min":0.54900000000000004},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.63800000000000001,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.496,"5min":0.53900000000000003,"15min":0.54600000000000004},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.66400000000000003,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.54100000000000004}]},{"osd":4,"last update":"Tue Sep 17 07:03:15 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.40000000000000002,"15min":0.39900000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.432,"15min":0.434},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.60499999999999998,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.38700000000000001}]},{"osd":6,"last update":"Tue Sep 17 07:03:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.42299999999999999,"15min":0.42499999999999999},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.373},{"interface":"front","average":{"1min":0.439,"5min":0.44400000000000001,"15min":0.44500000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.63800000000000001,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.42699999999999999}]},{"osd":7,"last update":"Tue Sep 17 07:03:01 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.40200000000000002}]}]},{"osd":4,"up_from":33,"seq":141733920807,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31904,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713248,"statfs":{"total":95995035648,"available":95962365952,"internally_reserved":0,"allocated":811008,"data_stored":358244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:02:53 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.56299999999999994,"15min":0.57499999999999996},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.71499999999999997,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.60199999999999998,"15min":0.61499999999999999},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.69799999999999995,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.40200000000000002}]},{"osd":1,"last update":"Tue Sep 17 07:02:53 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.56000000000000005,"15min":0.57199999999999995},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.68400000000000005,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.495,"5min":0.57999999999999996,"15min":0.59399999999999997},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.67900000000000005,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.46500000000000002}]},{"osd":2,"last update":"Tue Sep 17 07:02:53 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.56899999999999995,"15min":0.58299999999999996},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.69999999999999996,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.432},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.57799999999999996,"15min":0.58799999999999997},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.77100000000000002,"5min":0.877,"15min":0.877},"last":0.36699999999999999}]},{"osd":3,"last update":"Tue Sep 17 07:02:53 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.59299999999999997,"15min":0.60799999999999998},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.74299999999999999,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.55900000000000005,"15min":0.56399999999999995},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.66800000000000004,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.38600000000000001}]},{"osd":5,"last update":"Tue Sep 17 07:03:19 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.39600000000000002,"15min":0.38600000000000001},"min":{"1min":0.316,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.248},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.41699999999999998,"15min":0.40699999999999997},"min":{"1min":0.34799999999999998,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.27900000000000003}]},{"osd":6,"last update":"Tue Sep 17 07:03:38 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.39900000000000002,"15min":0.39500000000000002},"min":{"1min":0.313,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.308},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.41399999999999998,"15min":0.40699999999999997},"min":{"1min":0.314,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.26300000000000001}]},{"osd":7,"last update":"Tue Sep 17 07:03:00 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.33600000000000002}]}]},{"osd":3,"up_from":27,"seq":115964117033,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31932,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713220,"statfs":{"total":95995035648,"available":95962337280,"internally_reserved":0,"allocated":835584,"data_stored":359028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:03:39 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.36099999999999999,"15min":0.36299999999999999},"min":{"1min":0.247,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.48399999999999999,"5min":0.55100000000000005,"15min":0.55100000000000005},"last":0.23699999999999999},{"interface":"front","average":{"1min":0.371,"5min":0.379,"15min":0.378},"min":{"1min":0.26000000000000001,"5min":0.189,"15min":0.189},"max":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.222}]},{"osd":1,"last update":"Tue Sep 17 07:03:39 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.39700000000000002,"15min":0.40500000000000003},"min":{"1min":0.24199999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.52100000000000002,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.441,"15min":0.44800000000000001},"min":{"1min":0.29899999999999999,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.58499999999999996,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.30499999999999999}]},{"osd":2,"last update":"Tue Sep 17 07:03:39 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.41299999999999998,"15min":0.42099999999999999},"min":{"1min":0.30599999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.59499999999999997,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.41499999999999998,"15min":0.41899999999999998},"min":{"1min":0.29499999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.36199999999999999}]},{"osd":4,"last update":"Tue Sep 17 07:02:56 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.56000000000000005,"15min":0.56399999999999995},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.754,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.53100000000000003,"15min":0.53500000000000003},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.33400000000000002}]},{"osd":5,"last update":"Tue Sep 17 07:03:18 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.59799999999999998,"15min":0.60599999999999998},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.58299999999999996,"15min":0.58499999999999996},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.79100000000000004,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.48099999999999998}]},{"osd":6,"last update":"Tue Sep 17 07:03:39 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.56999999999999995,"15min":0.56599999999999995},"min":{"1min":0.40400000000000003,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.76700000000000002,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58899999999999997,"15min":0.58999999999999997},"min":{"1min":0.42699999999999999,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.78600000000000003,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.44400000000000001}]},{"osd":7,"last update":"Tue Sep 17 07:03:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.42099999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411375,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31908,"kb_used_data":576,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713244,"statfs":{"total":95995035648,"available":95962361856,"internally_reserved":0,"allocated":589824,"data_stored":278618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:03:15 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.35499999999999998,"15min":0.34699999999999998},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.35899999999999999,"15min":0.35099999999999998},"min":{"1min":0.24399999999999999,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.50600000000000001,"5min":0.625,"15min":0.625},"last":0.47499999999999998}]},{"osd":1,"last update":"Tue Sep 17 07:03:15 2024","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.35499999999999998,"15min":0.35099999999999998},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.48999999999999999,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.38500000000000001,"15min":0.374},"min":{"1min":0.28599999999999998,"5min":0.251,"15min":0.251},"max":{"1min":0.55700000000000005,"5min":0.58399999999999996,"15min":0.58399999999999996},"last":0.35799999999999998}]},{"osd":3,"last update":"Tue Sep 17 07:03:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.435,"15min":0.437},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.57399999999999995,"5min":0.60499999999999998,"15min":0.60499999999999998},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.42599999999999999,"15min":0.42799999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.58999999999999997,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.434}]},{"osd":4,"last update":"Tue Sep 17 07:02:52 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56499999999999995,"15min":0.56599999999999995},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58599999999999997,"15min":0.58699999999999997},"min":{"1min":0.42999999999999999,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.48499999999999999}]},{"osd":5,"last update":"Tue Sep 17 07:03:15 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.101,"5min":1.101,"15min":1.101},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.57799999999999996,"15min":0.58999999999999997},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.78200000000000003,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.45400000000000001}]},{"osd":6,"last update":"Tue Sep 17 07:03:44 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.59499999999999997,"15min":0.60099999999999998},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.70799999999999996}]},{"osd":7,"last update":"Tue Sep 17 07:03:03 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":3.9289999999999998,"5min":3.9289999999999998,"15min":3.9289999999999998},"last":0.50900000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574900,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27844,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717308,"statfs":{"total":95995035648,"available":95966523392,"internally_reserved":0,"allocated":847872,"data_stored":360279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:03:46 2024","interfaces":[{"interface":"back","average":{"1min":0.315,"5min":0.34399999999999997,"15min":0.33400000000000002},"min":{"1min":0.21099999999999999,"5min":0.193,"15min":0.193},"max":{"1min":0.46100000000000002,"5min":0.56000000000000005,"15min":0.56000000000000005},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.33800000000000002,"5min":0.36899999999999999,"15min":0.36199999999999999},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.47399999999999998,"5min":0.622,"15min":0.622},"last":0.28799999999999998}]},{"osd":2,"last update":"Tue Sep 17 07:03:12 2024","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.39600000000000002,"15min":0.40100000000000002},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.53100000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"last":0.312},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.40600000000000003,"15min":0.41199999999999998},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.60199999999999998,"5min":0.625,"15min":0.625},"last":0.35499999999999998}]},{"osd":3,"last update":"Tue Sep 17 07:03:37 2024","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.441,"15min":0.45800000000000002},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.41599999999999998,"15min":0.438},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.54900000000000004,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.379}]},{"osd":4,"last update":"Tue Sep 17 07:02:54 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.57699999999999996,"15min":0.57399999999999995},"min":{"1min":0.41199999999999998,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.56999999999999995,"15min":0.56799999999999995},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.71799999999999997,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.57899999999999996}]},{"osd":5,"last update":"Tue Sep 17 07:03:19 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.621,"15min":0.58699999999999997},"min":{"1min":0.38700000000000001,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":6.0060000000000002,"5min":6.0060000000000002,"15min":6.0060000000000002},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.56299999999999994,"15min":0.56499999999999995},"min":{"1min":0.38200000000000001,"5min":0.375,"15min":0.375},"max":{"1min":0.752,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.52400000000000002}]},{"osd":6,"last update":"Tue Sep 17 07:03:41 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.58699999999999997,"15min":0.59299999999999997},"min":{"1min":0.38500000000000001,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.77200000000000002,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.49099999999999999}]},{"osd":7,"last update":"Tue Sep 17 07:03:01 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.502},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.59499999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738423,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28408,"kb_used_data":1172,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716744,"statfs":{"total":95995035648,"available":95965945856,"internally_reserved":0,"allocated":1200128,"data_stored":869296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 17 07:03:47 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.35099999999999998,"15min":0.33000000000000002},"min":{"1min":0.25600000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.52600000000000002,"5min":0.58499999999999996,"15min":0.58499999999999996},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.371,"15min":0.34499999999999997},"min":{"1min":0.26400000000000001,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.39200000000000002}]},{"osd":2,"last update":"Tue Sep 17 07:03:15 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.38400000000000001,"15min":0.377},"min":{"1min":0.251,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.61099999999999999,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.378,"5min":0.37,"15min":0.36299999999999999},"min":{"1min":0.26600000000000001,"5min":0.246,"15min":0.246},"max":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"last":0.27100000000000002}]},{"osd":3,"last update":"Tue Sep 17 07:03:43 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.42399999999999999,"15min":0.436},"min":{"1min":0.30199999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.55900000000000005,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.42199999999999999,"15min":0.433},"min":{"1min":0.313,"5min":0.248,"15min":0.248},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.372}]},{"osd":4,"last update":"Tue Sep 17 07:02:56 2024","interfaces":[{"interface":"back","average":{"1min":1.1319999999999999,"5min":0.67500000000000004,"15min":0.59899999999999998},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":12.573,"5min":12.573,"15min":12.573},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.67700000000000005}]},{"osd":5,"last update":"Tue Sep 17 07:03:19 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.55500000000000005,"15min":0.55000000000000004},"min":{"1min":0.376,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.54800000000000004,"15min":0.54200000000000004},"min":{"1min":0.39700000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.71199999999999997}]},{"osd":6,"last update":"Tue Sep 17 07:03:43 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.55000000000000004,"15min":0.54700000000000004},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.55400000000000005,"15min":0.55000000000000004},"min":{"1min":0.40500000000000003,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.68700000000000006}]},{"osd":7,"last update":"Tue Sep 17 07:03:01 2024","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.2010000000000001,"15min":1.2010000000000001},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":12.548,"5min":12.548,"15min":12.548},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.63400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T07:03:53.882 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T07:03:54.113 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:53 smithi100 ceph-mon[29092]: 2.0 scrub starts 2024-09-17T07:03:54.114 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:53 smithi100 ceph-mon[29092]: 2.0 scrub ok 2024-09-17T07:03:54.114 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:53 smithi100 ceph-mon[29092]: from='client.14730 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:03:54.114 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:53 smithi100 ceph-mon[29092]: 5.6 scrub starts 2024-09-17T07:03:54.114 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:53 smithi100 ceph-mon[29092]: 5.6 scrub ok 2024-09-17T07:03:54.114 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:53 smithi100 ceph-mon[29092]: 4.4 scrub starts 2024-09-17T07:03:54.114 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:53 smithi100 ceph-mon[29092]: 4.4 scrub ok 2024-09-17T07:03:54.114 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:53 smithi100 ceph-mon[29092]: 3.1 scrub starts 2024-09-17T07:03:54.114 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:53 smithi100 ceph-mon[29092]: 3.1 scrub ok 2024-09-17T07:03:54.114 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:53 smithi100 ceph-mon[29092]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 13 B/s, 1 objects/s recovering 2024-09-17T07:03:54.115 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:53 smithi100 ceph-mon[39462]: 2.0 scrub starts 2024-09-17T07:03:54.115 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:53 smithi100 ceph-mon[39462]: 2.0 scrub ok 2024-09-17T07:03:54.115 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:53 smithi100 ceph-mon[39462]: from='client.14730 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:03:54.115 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:53 smithi100 ceph-mon[39462]: 5.6 scrub starts 2024-09-17T07:03:54.115 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:53 smithi100 ceph-mon[39462]: 5.6 scrub ok 2024-09-17T07:03:54.115 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:53 smithi100 ceph-mon[39462]: 4.4 scrub starts 2024-09-17T07:03:54.115 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:53 smithi100 ceph-mon[39462]: 4.4 scrub ok 2024-09-17T07:03:54.115 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:53 smithi100 ceph-mon[39462]: 3.1 scrub starts 2024-09-17T07:03:54.115 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:53 smithi100 ceph-mon[39462]: 3.1 scrub ok 2024-09-17T07:03:54.115 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:53 smithi100 ceph-mon[39462]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 13 B/s, 1 objects/s recovering 2024-09-17T07:03:54.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:53 smithi135 ceph-mon[33238]: 2.0 scrub starts 2024-09-17T07:03:54.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:53 smithi135 ceph-mon[33238]: 2.0 scrub ok 2024-09-17T07:03:54.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:53 smithi135 ceph-mon[33238]: from='client.14730 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:03:54.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:53 smithi135 ceph-mon[33238]: 5.6 scrub starts 2024-09-17T07:03:54.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:53 smithi135 ceph-mon[33238]: 5.6 scrub ok 2024-09-17T07:03:54.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:53 smithi135 ceph-mon[33238]: 4.4 scrub starts 2024-09-17T07:03:54.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:53 smithi135 ceph-mon[33238]: 4.4 scrub ok 2024-09-17T07:03:54.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:53 smithi135 ceph-mon[33238]: 3.1 scrub starts 2024-09-17T07:03:54.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:53 smithi135 ceph-mon[33238]: 3.1 scrub ok 2024-09-17T07:03:54.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:53 smithi135 ceph-mon[33238]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 13 B/s, 1 objects/s recovering 2024-09-17T07:03:54.376 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:Sep 17 07:03:54 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-alertmanager-a[71357]: ts=2024-09-17T07:03:54.112Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-09-17T07:03:54.377 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:Sep 17 07:03:54 smithi100 podman[73735]: 2024-09-17 07:03:54.143641467 +0000 UTC m=+0.161254846 container died 5557c6374eb7a1231bc38450b6663e0251acfdeb55fe8758e86971deb99bb341 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-17T07:03:54.664 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:Sep 17 07:03:54 smithi100 podman[73735]: 2024-09-17 07:03:54.479479203 +0000 UTC m=+0.497092587 container cleanup 5557c6374eb7a1231bc38450b6663e0251acfdeb55fe8758e86971deb99bb341 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-17T07:03:54.664 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:Sep 17 07:03:54 smithi100 bash[73735]: ceph-e848267c-74c1-11ef-bceb-c7b262605968-alertmanager-a 2024-09-17T07:03:54.859 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:03:54.940 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:Sep 17 07:03:54 smithi100 podman[73760]: 2024-09-17 07:03:54.664158705 +0000 UTC m=+0.519843802 container remove 5557c6374eb7a1231bc38450b6663e0251acfdeb55fe8758e86971deb99bb341 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-17T07:03:54.940 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:Sep 17 07:03:54 smithi100 podman[73760]: 2024-09-17 07:03:54.756284589 +0000 UTC m=+0.611969657 volume remove c914f2c1b5d318a8a4498f8c4a94e98c4b1983be48c75042ec9d823ee4de57e3 2024-09-17T07:03:55.218 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:54 smithi100 ceph-mon[29092]: 6.12 scrub starts 2024-09-17T07:03:55.218 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:54 smithi100 ceph-mon[29092]: 6.12 scrub ok 2024-09-17T07:03:55.218 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:54 smithi100 ceph-mon[29092]: 2.11 scrub starts 2024-09-17T07:03:55.218 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:54 smithi100 ceph-mon[29092]: 2.11 scrub ok 2024-09-17T07:03:55.218 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:54 smithi100 ceph-mon[29092]: 6.16 scrub starts 2024-09-17T07:03:55.218 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:54 smithi100 ceph-mon[29092]: 6.16 scrub ok 2024-09-17T07:03:55.219 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:54 smithi100 ceph-mon[39462]: 6.12 scrub starts 2024-09-17T07:03:55.219 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:54 smithi100 ceph-mon[39462]: 6.12 scrub ok 2024-09-17T07:03:55.219 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:54 smithi100 ceph-mon[39462]: 2.11 scrub starts 2024-09-17T07:03:55.219 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:54 smithi100 ceph-mon[39462]: 2.11 scrub ok 2024-09-17T07:03:55.219 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:54 smithi100 ceph-mon[39462]: 6.16 scrub starts 2024-09-17T07:03:55.219 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:54 smithi100 ceph-mon[39462]: 6.16 scrub ok 2024-09-17T07:03:55.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:54 smithi135 ceph-mon[33238]: 6.12 scrub starts 2024-09-17T07:03:55.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:54 smithi135 ceph-mon[33238]: 6.12 scrub ok 2024-09-17T07:03:55.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:54 smithi135 ceph-mon[33238]: 2.11 scrub starts 2024-09-17T07:03:55.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:54 smithi135 ceph-mon[33238]: 2.11 scrub ok 2024-09-17T07:03:55.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:54 smithi135 ceph-mon[33238]: 6.16 scrub starts 2024-09-17T07:03:55.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:54 smithi135 ceph-mon[33238]: 6.16 scrub ok 2024-09-17T07:03:55.538 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:Sep 17 07:03:55 smithi100 podman[73818]: 2024-09-17 07:03:55.318730525 +0000 UTC m=+0.448480094 container died cee108d8f97c3375615d3b242b2fa0d62de0dcbd4e66798a734d08a5a9ad70c0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=happy_lamport, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-19e2a2b, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-09-17T07:03:55.872 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:Sep 17 07:03:55 smithi100 systemd[1]: ceph-e848267c-74c1-11ef-bceb-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-09-17T07:03:55.872 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:Sep 17 07:03:55 smithi100 systemd[1]: Stopped Ceph alertmanager.a for e848267c-74c1-11ef-bceb-c7b262605968. 2024-09-17T07:03:55.872 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:Sep 17 07:03:55 smithi100 systemd[1]: Starting Ceph alertmanager.a for e848267c-74c1-11ef-bceb-c7b262605968... 2024-09-17T07:03:56.127 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:03:55 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:03:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:03:56.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:55 smithi100 ceph-mon[29092]: 5.1f scrub starts 2024-09-17T07:03:56.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:55 smithi100 ceph-mon[29092]: 5.1f scrub ok 2024-09-17T07:03:56.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:55 smithi100 ceph-mon[29092]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 13 B/s, 1 objects/s recovering 2024-09-17T07:03:56.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:55 smithi100 ceph-mon[39462]: 5.1f scrub starts 2024-09-17T07:03:56.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:55 smithi100 ceph-mon[39462]: 5.1f scrub ok 2024-09-17T07:03:56.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:55 smithi100 ceph-mon[39462]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 13 B/s, 1 objects/s recovering 2024-09-17T07:03:56.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:55 smithi135 ceph-mon[33238]: 5.1f scrub starts 2024-09-17T07:03:56.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:55 smithi135 ceph-mon[33238]: 5.1f scrub ok 2024-09-17T07:03:56.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:55 smithi135 ceph-mon[33238]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 13 B/s, 1 objects/s recovering 2024-09-17T07:03:56.648 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T07:03:56.650 INFO:teuthology.orchestra.run.smithi100.stderr:dumped all 2024-09-17T07:03:56.948 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:Sep 17 07:03:56 smithi100 podman[73962]: 2024-09-17 07:03:56.539338929 +0000 UTC m=+0.016532268 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-09-17T07:03:56.948 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:Sep 17 07:03:56 smithi100 podman[73962]: 2024-09-17 07:03:56.637166788 +0000 UTC m=+0.114360111 volume create 276ff59be6608b8e27fad840aa45280643c54e718fa409b13bc95b1fd20ce126 2024-09-17T07:03:56.948 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:Sep 17 07:03:56 smithi100 podman[73962]: 2024-09-17 07:03:56.739272828 +0000 UTC m=+0.216466195 container create 9269bad3ef8f51d11ff3571d2ae32d698bcd73f06f6f34b94670224e1810c4f0 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-17T07:03:57.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:56 smithi135 ceph-mon[33238]: 4.1e scrub starts 2024-09-17T07:03:57.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:56 smithi135 ceph-mon[33238]: 4.1e scrub ok 2024-09-17T07:03:57.312 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:56 smithi100 ceph-mon[29092]: 4.1e scrub starts 2024-09-17T07:03:57.312 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:56 smithi100 ceph-mon[29092]: 4.1e scrub ok 2024-09-17T07:03:57.312 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:56 smithi100 ceph-mon[39462]: 4.1e scrub starts 2024-09-17T07:03:57.312 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:56 smithi100 ceph-mon[39462]: 4.1e scrub ok 2024-09-17T07:03:57.701 INFO:teuthology.orchestra.run.smithi100.stdout:{"pg_ready":true,"pg_map":{"version":52,"stamp":"2024-09-17T07:03:55.010520+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596246,"num_objects":196,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":1016,"num_read_kb":862,"num_write":609,"num_write_kb":2459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1771214,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":604,"ondisk_log_size":604,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":247272,"kb_used_data":6552,"kb_used_omap":12,"kb_used_meta":240627,"kb_avail":749713944,"statfs":{"total":767960285184,"available":767707078656,"internally_reserved":0,"allocated":6709248,"data_stored":4053039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":246402616},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001366"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.757737+0000","last_change":"2024-09-17T07:02:18.366053+0000","last_active":"2024-09-17T07:03:47.757737+0000","last_peered":"2024-09-17T07:03:47.757737+0000","last_clean":"2024-09-17T07:03:47.757737+0000","last_became_active":"2024-09-17T07:02:18.365773+0000","last_became_peered":"2024-09-17T07:02:18.365773+0000","last_unstale":"2024-09-17T07:03:47.757737+0000","last_undegraded":"2024-09-17T07:03:47.757737+0000","last_fullsized":"2024-09-17T07:03:47.757737+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:02:17.002904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":46,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.752744+0000","last_change":"2024-09-17T07:03:41.931873+0000","last_active":"2024-09-17T07:03:47.752744+0000","last_peered":"2024-09-17T07:03:47.752744+0000","last_clean":"2024-09-17T07:03:47.752744+0000","last_became_active":"2024-09-17T07:02:12.006754+0000","last_became_peered":"2024-09-17T07:02:12.006754+0000","last_unstale":"2024-09-17T07:03:47.752744+0000","last_undegraded":"2024-09-17T07:03:47.752744+0000","last_fullsized":"2024-09-17T07:03:47.752744+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:41.931780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:41.931780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:50.138892+0000","scrub_duration":0.0017747419999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"75'15","reported_seq":68,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.757235+0000","last_change":"2024-09-17T07:03:30.436540+0000","last_active":"2024-09-17T07:03:47.757235+0000","last_peered":"2024-09-17T07:03:47.757235+0000","last_clean":"2024-09-17T07:03:47.757235+0000","last_became_active":"2024-09-17T07:02:14.012962+0000","last_became_peered":"2024-09-17T07:02:14.012962+0000","last_unstale":"2024-09-17T07:03:47.757235+0000","last_undegraded":"2024-09-17T07:03:47.757235+0000","last_fullsized":"2024-09-17T07:03:47.757235+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-09-17T07:03:30.436444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:30.436444+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:35.913395+0000","scrub_duration":0.0023734419999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.757870+0000","last_change":"2024-09-17T07:02:16.015244+0000","last_active":"2024-09-17T07:03:47.757870+0000","last_peered":"2024-09-17T07:03:47.757870+0000","last_clean":"2024-09-17T07:03:47.757870+0000","last_became_active":"2024-09-17T07:02:16.014928+0000","last_became_peered":"2024-09-17T07:02:16.014928+0000","last_unstale":"2024-09-17T07:03:47.757870+0000","last_undegraded":"2024-09-17T07:03:47.757870+0000","last_fullsized":"2024-09-17T07:03:47.757870+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:02:14.999435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":45,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.757641+0000","last_change":"2024-09-17T07:03:43.459940+0000","last_active":"2024-09-17T07:03:47.757641+0000","last_peered":"2024-09-17T07:03:47.757641+0000","last_clean":"2024-09-17T07:03:47.757641+0000","last_became_active":"2024-09-17T07:02:12.010069+0000","last_became_peered":"2024-09-17T07:02:12.010069+0000","last_unstale":"2024-09-17T07:03:47.757641+0000","last_undegraded":"2024-09-17T07:03:47.757641+0000","last_fullsized":"2024-09-17T07:03:47.757641+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:43.459821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:43.459821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:44.534548+0000","scrub_duration":0.0015265960000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"75'9","reported_seq":60,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.757900+0000","last_change":"2024-09-17T07:03:17.864856+0000","last_active":"2024-09-17T07:03:47.757900+0000","last_peered":"2024-09-17T07:03:47.757900+0000","last_clean":"2024-09-17T07:03:47.757900+0000","last_became_active":"2024-09-17T07:02:14.016244+0000","last_became_peered":"2024-09-17T07:02:14.016244+0000","last_unstale":"2024-09-17T07:03:47.757900+0000","last_undegraded":"2024-09-17T07:03:47.757900+0000","last_fullsized":"2024-09-17T07:03:47.757900+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:03:17.864726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:17.864726+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:29.697946+0000","scrub_duration":0.0021756380000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.756462+0000","last_change":"2024-09-17T07:03:37.113047+0000","last_active":"2024-09-17T07:03:47.756462+0000","last_peered":"2024-09-17T07:03:47.756462+0000","last_clean":"2024-09-17T07:03:47.756462+0000","last_became_active":"2024-09-17T07:02:16.014380+0000","last_became_peered":"2024-09-17T07:02:16.014380+0000","last_unstale":"2024-09-17T07:03:47.756462+0000","last_undegraded":"2024-09-17T07:03:47.756462+0000","last_fullsized":"2024-09-17T07:03:47.756462+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:37.112954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:37.112954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:45.057903+0000","scrub_duration":0.001466824,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.758370+0000","last_change":"2024-09-17T07:03:38.109357+0000","last_active":"2024-09-17T07:03:47.758370+0000","last_peered":"2024-09-17T07:03:47.758370+0000","last_clean":"2024-09-17T07:03:47.758370+0000","last_became_active":"2024-09-17T07:02:18.020790+0000","last_became_peered":"2024-09-17T07:02:18.020790+0000","last_unstale":"2024-09-17T07:03:47.758370+0000","last_undegraded":"2024-09-17T07:03:47.758370+0000","last_fullsized":"2024-09-17T07:03:47.758370+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:38.109265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:03:38.109265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:57.231258+0000","scrub_duration":0.001349659,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":46,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.752701+0000","last_change":"2024-09-17T07:03:43.060196+0000","last_active":"2024-09-17T07:03:47.752701+0000","last_peered":"2024-09-17T07:03:47.752701+0000","last_clean":"2024-09-17T07:03:47.752701+0000","last_became_active":"2024-09-17T07:02:12.011129+0000","last_became_peered":"2024-09-17T07:02:12.011129+0000","last_unstale":"2024-09-17T07:03:47.752701+0000","last_undegraded":"2024-09-17T07:03:47.752701+0000","last_fullsized":"2024-09-17T07:03:47.752701+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:43.060102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:43.060102+0000","last_clean_scrub_stamp":"2024-09-17T07:03:43.060102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:09.780017+0000","scrub_duration":0.0012238170000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"75'5","reported_seq":54,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.758799+0000","last_change":"2024-09-17T07:03:29.002445+0000","last_active":"2024-09-17T07:03:47.758799+0000","last_peered":"2024-09-17T07:03:47.758799+0000","last_clean":"2024-09-17T07:03:47.758799+0000","last_became_active":"2024-09-17T07:02:14.016640+0000","last_became_peered":"2024-09-17T07:02:14.016640+0000","last_unstale":"2024-09-17T07:03:47.758799+0000","last_undegraded":"2024-09-17T07:03:47.758799+0000","last_fullsized":"2024-09-17T07:03:47.758799+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-09-17T07:03:29.002312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:29.002312+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:44.000292+0000","scrub_duration":0.002172748,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.751930+0000","last_change":"2024-09-17T07:03:22.015629+0000","last_active":"2024-09-17T07:03:47.751930+0000","last_peered":"2024-09-17T07:03:47.751930+0000","last_clean":"2024-09-17T07:03:47.751930+0000","last_became_active":"2024-09-17T07:02:16.015401+0000","last_became_peered":"2024-09-17T07:02:16.015401+0000","last_unstale":"2024-09-17T07:03:47.751930+0000","last_undegraded":"2024-09-17T07:03:47.751930+0000","last_fullsized":"2024-09-17T07:03:47.751930+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:22.015535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:22.015535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:49.909183+0000","scrub_duration":0.001507752,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.756583+0000","last_change":"2024-09-17T07:03:34.882046+0000","last_active":"2024-09-17T07:03:47.756583+0000","last_peered":"2024-09-17T07:03:47.756583+0000","last_clean":"2024-09-17T07:03:47.756583+0000","last_became_active":"2024-09-17T07:02:18.023680+0000","last_became_peered":"2024-09-17T07:02:18.023680+0000","last_unstale":"2024-09-17T07:03:47.756583+0000","last_undegraded":"2024-09-17T07:03:47.756583+0000","last_fullsized":"2024-09-17T07:03:47.756583+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:34.881951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:03:34.881951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:36.646147+0000","scrub_duration":0.0014520379999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":46,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.752465+0000","last_change":"2024-09-17T07:03:45.029350+0000","last_active":"2024-09-17T07:03:47.752465+0000","last_peered":"2024-09-17T07:03:47.752465+0000","last_clean":"2024-09-17T07:03:47.752465+0000","last_became_active":"2024-09-17T07:02:12.012234+0000","last_became_peered":"2024-09-17T07:02:12.012234+0000","last_unstale":"2024-09-17T07:03:47.752465+0000","last_undegraded":"2024-09-17T07:03:47.752465+0000","last_fullsized":"2024-09-17T07:03:47.752465+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:45.029247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:45.029247+0000","last_clean_scrub_stamp":"2024-09-17T07:03:45.029247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:07.678537+0000","scrub_duration":0.001377554,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"75'9","reported_seq":49,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.758582+0000","last_change":"2024-09-17T07:02:14.013597+0000","last_active":"2024-09-17T07:03:47.758582+0000","last_peered":"2024-09-17T07:03:47.758582+0000","last_clean":"2024-09-17T07:03:47.758582+0000","last_became_active":"2024-09-17T07:02:14.013337+0000","last_became_peered":"2024-09-17T07:02:14.013337+0000","last_unstale":"2024-09-17T07:03:47.758582+0000","last_undegraded":"2024-09-17T07:03:47.758582+0000","last_fullsized":"2024-09-17T07:03:47.758582+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:02:12.994221+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.756813+0000","last_change":"2024-09-17T07:02:16.024037+0000","last_active":"2024-09-17T07:03:47.756813+0000","last_peered":"2024-09-17T07:03:47.756813+0000","last_clean":"2024-09-17T07:03:47.756813+0000","last_became_active":"2024-09-17T07:02:16.023774+0000","last_became_peered":"2024-09-17T07:02:16.023774+0000","last_unstale":"2024-09-17T07:03:47.756813+0000","last_undegraded":"2024-09-17T07:03:47.756813+0000","last_fullsized":"2024-09-17T07:03:47.756813+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:02:14.999435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.752525+0000","last_change":"2024-09-17T07:02:18.029470+0000","last_active":"2024-09-17T07:03:47.752525+0000","last_peered":"2024-09-17T07:03:47.752525+0000","last_clean":"2024-09-17T07:03:47.752525+0000","last_became_active":"2024-09-17T07:02:18.029178+0000","last_became_peered":"2024-09-17T07:02:18.029178+0000","last_unstale":"2024-09-17T07:03:47.752525+0000","last_undegraded":"2024-09-17T07:03:47.752525+0000","last_fullsized":"2024-09-17T07:03:47.752525+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:02:17.002904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"75'17","reported_seq":61,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.758890+0000","last_change":"2024-09-17T07:02:14.016560+0000","last_active":"2024-09-17T07:03:47.758890+0000","last_peered":"2024-09-17T07:03:47.758890+0000","last_clean":"2024-09-17T07:03:47.758890+0000","last_became_active":"2024-09-17T07:02:14.016270+0000","last_became_peered":"2024-09-17T07:02:14.016270+0000","last_unstale":"2024-09-17T07:03:47.758890+0000","last_undegraded":"2024-09-17T07:03:47.758890+0000","last_fullsized":"2024-09-17T07:03:47.758890+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:02:12.994221+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":46,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.752469+0000","last_change":"2024-09-17T07:03:14.092519+0000","last_active":"2024-09-17T07:03:47.752469+0000","last_peered":"2024-09-17T07:03:47.752469+0000","last_clean":"2024-09-17T07:03:47.752469+0000","last_became_active":"2024-09-17T07:02:12.011258+0000","last_became_peered":"2024-09-17T07:02:12.011258+0000","last_unstale":"2024-09-17T07:03:47.752469+0000","last_undegraded":"2024-09-17T07:03:47.752469+0000","last_fullsized":"2024-09-17T07:03:47.752469+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:14.092369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:14.092369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:19.152787+0000","scrub_duration":0.0015447460000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.757706+0000","last_change":"2024-09-17T07:03:44.044750+0000","last_active":"2024-09-17T07:03:47.757706+0000","last_peered":"2024-09-17T07:03:47.757706+0000","last_clean":"2024-09-17T07:03:47.757706+0000","last_became_active":"2024-09-17T07:02:16.018147+0000","last_became_peered":"2024-09-17T07:02:16.018147+0000","last_unstale":"2024-09-17T07:03:47.757706+0000","last_undegraded":"2024-09-17T07:03:47.757706+0000","last_fullsized":"2024-09-17T07:03:47.757706+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:44.044635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:44.044635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:46.891263+0000","scrub_duration":0.0014202049999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.753570+0000","last_change":"2024-09-17T07:03:42.984226+0000","last_active":"2024-09-17T07:03:47.753570+0000","last_peered":"2024-09-17T07:03:47.753570+0000","last_clean":"2024-09-17T07:03:47.753570+0000","last_became_active":"2024-09-17T07:02:18.022040+0000","last_became_peered":"2024-09-17T07:02:18.022040+0000","last_unstale":"2024-09-17T07:03:47.753570+0000","last_undegraded":"2024-09-17T07:03:47.753570+0000","last_fullsized":"2024-09-17T07:03:47.753570+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:42.984088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:42.984088+0000","last_clean_scrub_stamp":"2024-09-17T07:03:42.984088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:59.596779+0000","scrub_duration":0.0014441129999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"75'10","reported_seq":59,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.758482+0000","last_change":"2024-09-17T07:03:20.933478+0000","last_active":"2024-09-17T07:03:47.758482+0000","last_peered":"2024-09-17T07:03:47.758482+0000","last_clean":"2024-09-17T07:03:47.758482+0000","last_became_active":"2024-09-17T07:02:14.007702+0000","last_became_peered":"2024-09-17T07:02:14.007702+0000","last_unstale":"2024-09-17T07:03:47.758482+0000","last_undegraded":"2024-09-17T07:03:47.758482+0000","last_fullsized":"2024-09-17T07:03:47.758482+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-09-17T07:03:20.933353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:20.933353+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:28.865120+0000","scrub_duration":0.0021221080000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-17T07:03:51.926925+0000","last_change":"2024-09-17T07:03:51.927006+0000","last_active":"2024-09-17T07:03:51.926925+0000","last_peered":"2024-09-17T07:03:51.926925+0000","last_clean":"2024-09-17T07:03:51.926925+0000","last_became_active":"2024-09-17T07:02:12.007553+0000","last_became_peered":"2024-09-17T07:02:12.007553+0000","last_unstale":"2024-09-17T07:03:51.926925+0000","last_undegraded":"2024-09-17T07:03:51.926925+0000","last_fullsized":"2024-09-17T07:03:51.926925+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:51.926911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:51.926911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:52.506175+0000","scrub_duration":0.0014014909999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"75'8","reported_seq":70,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-17T07:03:48.753479+0000","last_change":"2024-09-17T07:03:45.909722+0000","last_active":"2024-09-17T07:03:48.753479+0000","last_peered":"2024-09-17T07:03:48.753479+0000","last_clean":"2024-09-17T07:03:48.753479+0000","last_became_active":"2024-09-17T07:02:16.015001+0000","last_became_peered":"2024-09-17T07:02:16.015001+0000","last_unstale":"2024-09-17T07:03:48.753479+0000","last_undegraded":"2024-09-17T07:03:48.753479+0000","last_fullsized":"2024-09-17T07:03:48.753479+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:03:45.909617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:45.909617+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:48.865649+0000","scrub_duration":0.0021269700000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.759001+0000","last_change":"2024-09-17T07:03:46.241227+0000","last_active":"2024-09-17T07:03:47.759001+0000","last_peered":"2024-09-17T07:03:47.759001+0000","last_clean":"2024-09-17T07:03:47.759001+0000","last_became_active":"2024-09-17T07:02:18.017460+0000","last_became_peered":"2024-09-17T07:02:18.017460+0000","last_unstale":"2024-09-17T07:03:47.759001+0000","last_undegraded":"2024-09-17T07:03:47.759001+0000","last_fullsized":"2024-09-17T07:03:47.759001+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:46.241143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:03:46.241143+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:15.293653+0000","scrub_duration":0.0013561739999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"75'15","reported_seq":68,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.758648+0000","last_change":"2024-09-17T07:03:16.067607+0000","last_active":"2024-09-17T07:03:47.758648+0000","last_peered":"2024-09-17T07:03:47.758648+0000","last_clean":"2024-09-17T07:03:47.758648+0000","last_became_active":"2024-09-17T07:02:14.016397+0000","last_became_peered":"2024-09-17T07:02:14.016397+0000","last_unstale":"2024-09-17T07:03:47.758648+0000","last_undegraded":"2024-09-17T07:03:47.758648+0000","last_fullsized":"2024-09-17T07:03:47.758648+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-09-17T07:03:16.067479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:16.067479+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:28.548727+0000","scrub_duration":0.0025498640000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":45,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.756649+0000","last_change":"2024-09-17T07:03:28.072773+0000","last_active":"2024-09-17T07:03:47.756649+0000","last_peered":"2024-09-17T07:03:47.756649+0000","last_clean":"2024-09-17T07:03:47.756649+0000","last_became_active":"2024-09-17T07:02:12.007913+0000","last_became_peered":"2024-09-17T07:02:12.007913+0000","last_unstale":"2024-09-17T07:03:47.756649+0000","last_undegraded":"2024-09-17T07:03:47.756649+0000","last_fullsized":"2024-09-17T07:03:47.756649+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:28.072697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:28.072697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:43.368112+0000","scrub_duration":0.0012624730000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"75'8","reported_seq":59,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-17T07:03:48.755575+0000","last_change":"2024-09-17T07:02:16.022051+0000","last_active":"2024-09-17T07:03:48.755575+0000","last_peered":"2024-09-17T07:03:48.755575+0000","last_clean":"2024-09-17T07:03:48.755575+0000","last_became_active":"2024-09-17T07:02:16.021762+0000","last_became_peered":"2024-09-17T07:02:16.021762+0000","last_unstale":"2024-09-17T07:03:48.755575+0000","last_undegraded":"2024-09-17T07:03:48.755575+0000","last_fullsized":"2024-09-17T07:03:48.755575+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:02:14.999435+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.759562+0000","last_change":"2024-09-17T07:03:47.759562+0000","last_active":"2024-09-17T07:03:47.759562+0000","last_peered":"2024-09-17T07:03:47.759562+0000","last_clean":"2024-09-17T07:03:47.759562+0000","last_became_active":"2024-09-17T07:02:18.025332+0000","last_became_peered":"2024-09-17T07:02:18.025332+0000","last_unstale":"2024-09-17T07:03:47.759562+0000","last_undegraded":"2024-09-17T07:03:47.759562+0000","last_fullsized":"2024-09-17T07:03:47.759562+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:47.759476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:03:47.759476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:16.113819+0000","scrub_duration":0.0035797160000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"75'11","reported_seq":52,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.758943+0000","last_change":"2024-09-17T07:02:14.010584+0000","last_active":"2024-09-17T07:03:47.758943+0000","last_peered":"2024-09-17T07:03:47.758943+0000","last_clean":"2024-09-17T07:03:47.758943+0000","last_became_active":"2024-09-17T07:02:14.010227+0000","last_became_peered":"2024-09-17T07:02:14.010227+0000","last_unstale":"2024-09-17T07:03:47.758943+0000","last_undegraded":"2024-09-17T07:03:47.758943+0000","last_fullsized":"2024-09-17T07:03:47.758943+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:02:12.994221+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":46,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.751979+0000","last_change":"2024-09-17T07:03:31.076235+0000","last_active":"2024-09-17T07:03:47.751979+0000","last_peered":"2024-09-17T07:03:47.751979+0000","last_clean":"2024-09-17T07:03:47.751979+0000","last_became_active":"2024-09-17T07:02:12.013182+0000","last_became_peered":"2024-09-17T07:02:12.013182+0000","last_unstale":"2024-09-17T07:03:47.751979+0000","last_undegraded":"2024-09-17T07:03:47.751979+0000","last_fullsized":"2024-09-17T07:03:47.751979+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:31.076134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:31.076134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:37.803854+0000","scrub_duration":0.0012890040000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.757058+0000","last_change":"2024-09-17T07:03:18.816900+0000","last_active":"2024-09-17T07:03:47.757058+0000","last_peered":"2024-09-17T07:03:47.757058+0000","last_clean":"2024-09-17T07:03:47.757058+0000","last_became_active":"2024-09-17T07:02:16.020224+0000","last_became_peered":"2024-09-17T07:02:16.020224+0000","last_unstale":"2024-09-17T07:03:47.757058+0000","last_undegraded":"2024-09-17T07:03:47.757058+0000","last_fullsized":"2024-09-17T07:03:47.757058+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:18.816795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:18.816795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:46.254399+0000","scrub_duration":0.001573212,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.757578+0000","last_change":"2024-09-17T07:03:35.447165+0000","last_active":"2024-09-17T07:03:47.757578+0000","last_peered":"2024-09-17T07:03:47.757578+0000","last_clean":"2024-09-17T07:03:47.757578+0000","last_became_active":"2024-09-17T07:02:18.365000+0000","last_became_peered":"2024-09-17T07:02:18.365000+0000","last_unstale":"2024-09-17T07:03:47.757578+0000","last_undegraded":"2024-09-17T07:03:47.757578+0000","last_fullsized":"2024-09-17T07:03:47.757578+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:35.447078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:03:35.447078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:38.486661+0000","scrub_duration":0.0020715600000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"75'14","reported_seq":54,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.758222+0000","last_change":"2024-09-17T07:02:14.008065+0000","last_active":"2024-09-17T07:03:47.758222+0000","last_peered":"2024-09-17T07:03:47.758222+0000","last_clean":"2024-09-17T07:03:47.758222+0000","last_became_active":"2024-09-17T07:02:14.007810+0000","last_became_peered":"2024-09-17T07:02:14.007810+0000","last_unstale":"2024-09-17T07:03:47.758222+0000","last_undegraded":"2024-09-17T07:03:47.758222+0000","last_fullsized":"2024-09-17T07:03:47.758222+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:02:12.994221+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":45,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.755255+0000","last_change":"2024-09-17T07:03:16.455893+0000","last_active":"2024-09-17T07:03:47.755255+0000","last_peered":"2024-09-17T07:03:47.755255+0000","last_clean":"2024-09-17T07:03:47.755255+0000","last_became_active":"2024-09-17T07:02:12.011766+0000","last_became_peered":"2024-09-17T07:02:12.011766+0000","last_unstale":"2024-09-17T07:03:47.755255+0000","last_undegraded":"2024-09-17T07:03:47.755255+0000","last_fullsized":"2024-09-17T07:03:47.755255+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:16.455684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:16.455684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:22.392690+0000","scrub_duration":0.0020498690000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"75'8","reported_seq":69,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-17T07:03:48.755841+0000","last_change":"2024-09-17T07:03:33.444197+0000","last_active":"2024-09-17T07:03:48.755841+0000","last_peered":"2024-09-17T07:03:48.755841+0000","last_clean":"2024-09-17T07:03:48.755841+0000","last_became_active":"2024-09-17T07:02:16.020628+0000","last_became_peered":"2024-09-17T07:02:16.020628+0000","last_unstale":"2024-09-17T07:03:48.755841+0000","last_undegraded":"2024-09-17T07:03:48.755841+0000","last_fullsized":"2024-09-17T07:03:48.755841+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:03:33.444104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:33.444104+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:36.767124+0000","scrub_duration":0.0014124389999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.752257+0000","last_change":"2024-09-17T07:03:19.030574+0000","last_active":"2024-09-17T07:03:47.752257+0000","last_peered":"2024-09-17T07:03:47.752257+0000","last_clean":"2024-09-17T07:03:47.752257+0000","last_became_active":"2024-09-17T07:02:18.364981+0000","last_became_peered":"2024-09-17T07:02:18.364981+0000","last_unstale":"2024-09-17T07:03:47.752257+0000","last_undegraded":"2024-09-17T07:03:47.752257+0000","last_fullsized":"2024-09-17T07:03:47.752257+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:19.030470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:03:19.030470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:48.307795+0000","scrub_duration":0.00152618,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"75'18","reported_seq":75,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.758450+0000","last_change":"2024-09-17T07:03:26.413610+0000","last_active":"2024-09-17T07:03:47.758450+0000","last_peered":"2024-09-17T07:03:47.758450+0000","last_clean":"2024-09-17T07:03:47.758450+0000","last_became_active":"2024-09-17T07:02:14.028667+0000","last_became_peered":"2024-09-17T07:02:14.028667+0000","last_unstale":"2024-09-17T07:03:47.758450+0000","last_undegraded":"2024-09-17T07:03:47.758450+0000","last_fullsized":"2024-09-17T07:03:47.758450+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'18","last_scrub_stamp":"2024-09-17T07:03:26.413506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:26.413506+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:41.741155+0000","scrub_duration":0.0022855470000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":45,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.758379+0000","last_change":"2024-09-17T07:03:24.134708+0000","last_active":"2024-09-17T07:03:47.758379+0000","last_peered":"2024-09-17T07:03:47.758379+0000","last_clean":"2024-09-17T07:03:47.758379+0000","last_became_active":"2024-09-17T07:02:12.005687+0000","last_became_peered":"2024-09-17T07:02:12.005687+0000","last_unstale":"2024-09-17T07:03:47.758379+0000","last_undegraded":"2024-09-17T07:03:47.758379+0000","last_fullsized":"2024-09-17T07:03:47.758379+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:24.134627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:24.134627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:29.080338+0000","scrub_duration":0.001111148,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.758135+0000","last_change":"2024-09-17T07:02:16.020013+0000","last_active":"2024-09-17T07:03:47.758135+0000","last_peered":"2024-09-17T07:03:47.758135+0000","last_clean":"2024-09-17T07:03:47.758135+0000","last_became_active":"2024-09-17T07:02:16.019747+0000","last_became_peered":"2024-09-17T07:02:16.019747+0000","last_unstale":"2024-09-17T07:03:47.758135+0000","last_undegraded":"2024-09-17T07:03:47.758135+0000","last_fullsized":"2024-09-17T07:03:47.758135+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:02:14.999435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.758659+0000","last_change":"2024-09-17T07:03:44.196107+0000","last_active":"2024-09-17T07:03:47.758659+0000","last_peered":"2024-09-17T07:03:47.758659+0000","last_clean":"2024-09-17T07:03:47.758659+0000","last_became_active":"2024-09-17T07:02:18.024043+0000","last_became_peered":"2024-09-17T07:02:18.024043+0000","last_unstale":"2024-09-17T07:03:47.758659+0000","last_undegraded":"2024-09-17T07:03:47.758659+0000","last_fullsized":"2024-09-17T07:03:47.758659+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:44.196035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:03:44.196035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:12.177838+0000","scrub_duration":0.0015579439999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"75'19","reported_seq":80,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.752822+0000","last_change":"2024-09-17T07:03:40.949557+0000","last_active":"2024-09-17T07:03:47.752822+0000","last_peered":"2024-09-17T07:03:47.752822+0000","last_clean":"2024-09-17T07:03:47.752822+0000","last_became_active":"2024-09-17T07:02:14.027389+0000","last_became_peered":"2024-09-17T07:02:14.027389+0000","last_unstale":"2024-09-17T07:03:47.752822+0000","last_undegraded":"2024-09-17T07:03:47.752822+0000","last_fullsized":"2024-09-17T07:03:47.752822+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-09-17T07:03:40.949458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:40.949458+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:03.948689+0000","scrub_duration":0.0025476489999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":46,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.752750+0000","last_change":"2024-09-17T07:03:32.109842+0000","last_active":"2024-09-17T07:03:47.752750+0000","last_peered":"2024-09-17T07:03:47.752750+0000","last_clean":"2024-09-17T07:03:47.752750+0000","last_became_active":"2024-09-17T07:02:12.011917+0000","last_became_peered":"2024-09-17T07:02:12.011917+0000","last_unstale":"2024-09-17T07:03:47.752750+0000","last_undegraded":"2024-09-17T07:03:47.752750+0000","last_fullsized":"2024-09-17T07:03:47.752750+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:32.109765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:32.109765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:52.813415+0000","scrub_duration":0.001335314,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-17T07:03:49.489704+0000","last_change":"2024-09-17T07:03:49.489790+0000","last_active":"2024-09-17T07:03:49.489704+0000","last_peered":"2024-09-17T07:03:49.489704+0000","last_clean":"2024-09-17T07:03:49.489704+0000","last_became_active":"2024-09-17T07:02:16.016199+0000","last_became_peered":"2024-09-17T07:02:16.016199+0000","last_unstale":"2024-09-17T07:03:49.489704+0000","last_undegraded":"2024-09-17T07:03:49.489704+0000","last_fullsized":"2024-09-17T07:03:49.489704+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:49.489690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:49.489690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:17.412359+0000","scrub_duration":0.0016174729999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.756445+0000","last_change":"2024-09-17T07:03:30.068895+0000","last_active":"2024-09-17T07:03:47.756445+0000","last_peered":"2024-09-17T07:03:47.756445+0000","last_clean":"2024-09-17T07:03:47.756445+0000","last_became_active":"2024-09-17T07:02:18.364835+0000","last_became_peered":"2024-09-17T07:02:18.364835+0000","last_unstale":"2024-09-17T07:03:47.756445+0000","last_undegraded":"2024-09-17T07:03:47.756445+0000","last_fullsized":"2024-09-17T07:03:47.756445+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:30.068808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:03:30.068808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:51.270374+0000","scrub_duration":0.0012893679999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"75'10","reported_seq":58,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.756872+0000","last_change":"2024-09-17T07:03:26.127545+0000","last_active":"2024-09-17T07:03:47.756872+0000","last_peered":"2024-09-17T07:03:47.756872+0000","last_clean":"2024-09-17T07:03:47.756872+0000","last_became_active":"2024-09-17T07:02:14.010637+0000","last_became_peered":"2024-09-17T07:02:14.010637+0000","last_unstale":"2024-09-17T07:03:47.756872+0000","last_undegraded":"2024-09-17T07:03:47.756872+0000","last_fullsized":"2024-09-17T07:03:47.756872+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-09-17T07:03:26.127439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:26.127439+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:35.535770+0000","scrub_duration":0.001898479,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":45,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.758317+0000","last_change":"2024-09-17T07:03:25.148144+0000","last_active":"2024-09-17T07:03:47.758317+0000","last_peered":"2024-09-17T07:03:47.758317+0000","last_clean":"2024-09-17T07:03:47.758317+0000","last_became_active":"2024-09-17T07:02:12.010803+0000","last_became_peered":"2024-09-17T07:02:12.010803+0000","last_unstale":"2024-09-17T07:03:47.758317+0000","last_undegraded":"2024-09-17T07:03:47.758317+0000","last_fullsized":"2024-09-17T07:03:47.758317+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:25.147878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:25.147878+0000","last_clean_scrub_stamp":"2024-09-17T07:03:25.147878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:33.071698+0000","scrub_duration":0.001353723,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"89'9","reported_seq":71,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-17T07:03:48.756786+0000","last_change":"2024-09-17T07:03:29.961568+0000","last_active":"2024-09-17T07:03:48.756786+0000","last_peered":"2024-09-17T07:03:48.756786+0000","last_clean":"2024-09-17T07:03:48.756786+0000","last_became_active":"2024-09-17T07:02:16.021360+0000","last_became_peered":"2024-09-17T07:02:16.021360+0000","last_unstale":"2024-09-17T07:03:48.756786+0000","last_undegraded":"2024-09-17T07:03:48.756786+0000","last_fullsized":"2024-09-17T07:03:48.756786+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'9","last_scrub_stamp":"2024-09-17T07:03:29.961439+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-09-17T07:03:29.961439+0000","last_clean_scrub_stamp":"2024-09-17T07:03:29.961439+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:40.753813+0000","scrub_duration":0.0027696880000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"75'1","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.752400+0000","last_change":"2024-09-17T07:03:25.049036+0000","last_active":"2024-09-17T07:03:47.752400+0000","last_peered":"2024-09-17T07:03:47.752400+0000","last_clean":"2024-09-17T07:03:47.752400+0000","last_became_active":"2024-09-17T07:02:18.029772+0000","last_became_peered":"2024-09-17T07:02:18.029772+0000","last_unstale":"2024-09-17T07:03:47.752400+0000","last_undegraded":"2024-09-17T07:03:47.752400+0000","last_fullsized":"2024-09-17T07:03:47.752400+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-17T07:03:25.048949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:03:25.048949+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:53.755876+0000","scrub_duration":0.003886123,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"75'28","reported_seq":111,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-17T07:03:52.937902+0000","last_change":"2024-09-17T07:03:52.938018+0000","last_active":"2024-09-17T07:03:52.937902+0000","last_peered":"2024-09-17T07:03:52.937902+0000","last_clean":"2024-09-17T07:03:52.937902+0000","last_became_active":"2024-09-17T07:02:14.017377+0000","last_became_peered":"2024-09-17T07:02:14.017377+0000","last_unstale":"2024-09-17T07:03:52.937902+0000","last_undegraded":"2024-09-17T07:03:52.937902+0000","last_fullsized":"2024-09-17T07:03:52.937902+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'28","last_scrub_stamp":"2024-09-17T07:03:52.937879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:52.937879+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:09.513804+0000","scrub_duration":0.0023024629999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":45,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.757097+0000","last_change":"2024-09-17T07:03:40.811355+0000","last_active":"2024-09-17T07:03:47.757097+0000","last_peered":"2024-09-17T07:03:47.757097+0000","last_clean":"2024-09-17T07:03:47.757097+0000","last_became_active":"2024-09-17T07:02:12.008802+0000","last_became_peered":"2024-09-17T07:02:12.008802+0000","last_unstale":"2024-09-17T07:03:47.757097+0000","last_undegraded":"2024-09-17T07:03:47.757097+0000","last_fullsized":"2024-09-17T07:03:47.757097+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:40.811273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:40.811273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:52.601572+0000","scrub_duration":0.0012956230000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.753315+0000","last_change":"2024-09-17T07:03:28.981244+0000","last_active":"2024-09-17T07:03:47.753315+0000","last_peered":"2024-09-17T07:03:47.753315+0000","last_clean":"2024-09-17T07:03:47.753315+0000","last_became_active":"2024-09-17T07:02:16.016606+0000","last_became_peered":"2024-09-17T07:02:16.016606+0000","last_unstale":"2024-09-17T07:03:47.753315+0000","last_undegraded":"2024-09-17T07:03:47.753315+0000","last_fullsized":"2024-09-17T07:03:47.753315+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:28.981147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:28.981147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:37.078325+0000","scrub_duration":0.0012741460000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.757371+0000","last_change":"2024-09-17T07:02:18.026015+0000","last_active":"2024-09-17T07:03:47.757371+0000","last_peered":"2024-09-17T07:03:47.757371+0000","last_clean":"2024-09-17T07:03:47.757371+0000","last_became_active":"2024-09-17T07:02:18.024602+0000","last_became_peered":"2024-09-17T07:02:18.024602+0000","last_unstale":"2024-09-17T07:03:47.757371+0000","last_undegraded":"2024-09-17T07:03:47.757371+0000","last_fullsized":"2024-09-17T07:03:47.757371+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:02:17.002904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"75'13","reported_seq":70,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.758650+0000","last_change":"2024-09-17T07:03:32.079678+0000","last_active":"2024-09-17T07:03:47.758650+0000","last_peered":"2024-09-17T07:03:47.758650+0000","last_clean":"2024-09-17T07:03:47.758650+0000","last_became_active":"2024-09-17T07:02:14.019838+0000","last_became_peered":"2024-09-17T07:02:14.019838+0000","last_unstale":"2024-09-17T07:03:47.758650+0000","last_undegraded":"2024-09-17T07:03:47.758650+0000","last_fullsized":"2024-09-17T07:03:47.758650+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'13","last_scrub_stamp":"2024-09-17T07:03:32.079579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:32.079579+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:42.502680+0000","scrub_duration":0.001975929,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"76'5","reported_seq":147,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-17T07:03:51.133206+0000","last_change":"2024-09-17T07:03:50.944706+0000","last_active":"2024-09-17T07:03:51.133206+0000","last_peered":"2024-09-17T07:03:51.133206+0000","last_clean":"2024-09-17T07:03:51.133206+0000","last_became_active":"2024-09-17T07:02:12.006883+0000","last_became_peered":"2024-09-17T07:02:12.006883+0000","last_unstale":"2024-09-17T07:03:51.133206+0000","last_undegraded":"2024-09-17T07:03:51.133206+0000","last_fullsized":"2024-09-17T07:03:51.133206+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'5","last_scrub_stamp":"2024-09-17T07:03:50.944626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:50.944626+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:16.063852+0000","scrub_duration":0.001943529,"objects_trimmed":0,"snaptrim_duration":0.00033739799999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":94,"num_read_kb":89,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":46,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.752059+0000","last_change":"2024-09-17T07:03:33.795388+0000","last_active":"2024-09-17T07:03:47.752059+0000","last_peered":"2024-09-17T07:03:47.752059+0000","last_clean":"2024-09-17T07:03:47.752059+0000","last_became_active":"2024-09-17T07:02:12.010753+0000","last_became_peered":"2024-09-17T07:02:12.010753+0000","last_unstale":"2024-09-17T07:03:47.752059+0000","last_undegraded":"2024-09-17T07:03:47.752059+0000","last_fullsized":"2024-09-17T07:03:47.752059+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:33.795307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:33.795307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:00.370993+0000","scrub_duration":0.0012164070000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-17T07:03:52.922725+0000","last_change":"2024-09-17T07:03:52.922792+0000","last_active":"2024-09-17T07:03:52.922725+0000","last_peered":"2024-09-17T07:03:52.922725+0000","last_clean":"2024-09-17T07:03:52.922725+0000","last_became_active":"2024-09-17T07:02:16.014492+0000","last_became_peered":"2024-09-17T07:02:16.014492+0000","last_unstale":"2024-09-17T07:03:52.922725+0000","last_undegraded":"2024-09-17T07:03:52.922725+0000","last_fullsized":"2024-09-17T07:03:52.922725+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:52.922712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:52.922712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:11.791082+0000","scrub_duration":0.0016789929999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.752457+0000","last_change":"2024-09-17T07:03:42.068886+0000","last_active":"2024-09-17T07:03:47.752457+0000","last_peered":"2024-09-17T07:03:47.752457+0000","last_clean":"2024-09-17T07:03:47.752457+0000","last_became_active":"2024-09-17T07:02:18.365369+0000","last_became_peered":"2024-09-17T07:02:18.365369+0000","last_unstale":"2024-09-17T07:03:47.752457+0000","last_undegraded":"2024-09-17T07:03:47.752457+0000","last_fullsized":"2024-09-17T07:03:47.752457+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:42.068803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:03:42.068803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:47.482056+0000","scrub_duration":0.0015781599999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"75'12","reported_seq":85,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-17T07:03:48.767808+0000","last_change":"2024-09-17T07:03:48.767808+0000","last_active":"2024-09-17T07:03:48.767808+0000","last_peered":"2024-09-17T07:03:48.767808+0000","last_clean":"2024-09-17T07:03:48.767808+0000","last_became_active":"2024-09-17T07:03:48.760405+0000","last_became_peered":"2024-09-17T07:03:48.760405+0000","last_unstale":"2024-09-17T07:03:48.767808+0000","last_undegraded":"2024-09-17T07:03:48.767808+0000","last_fullsized":"2024-09-17T07:03:48.767808+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-09-17T07:03:27.987835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:27.987835+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:43.269560+0000","scrub_duration":0.0020148219999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"70'1","reported_seq":52,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-17T07:03:52.963264+0000","last_change":"2024-09-17T07:03:52.963332+0000","last_active":"2024-09-17T07:03:52.963264+0000","last_peered":"2024-09-17T07:03:52.963264+0000","last_clean":"2024-09-17T07:03:52.963264+0000","last_became_active":"2024-09-17T07:02:12.009476+0000","last_became_peered":"2024-09-17T07:02:12.009476+0000","last_unstale":"2024-09-17T07:03:52.963264+0000","last_undegraded":"2024-09-17T07:03:52.963264+0000","last_fullsized":"2024-09-17T07:03:52.963264+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-17T07:03:52.963249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:52.963249+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:07.662760+0000","scrub_duration":0.0015040780000000001,"objects_trimmed":0,"snaptrim_duration":0.0032298629999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-17T07:03:51.994773+0000","last_change":"2024-09-17T07:03:51.994858+0000","last_active":"2024-09-17T07:03:51.994773+0000","last_peered":"2024-09-17T07:03:51.994773+0000","last_clean":"2024-09-17T07:03:51.994773+0000","last_became_active":"2024-09-17T07:02:12.014623+0000","last_became_peered":"2024-09-17T07:02:12.014623+0000","last_unstale":"2024-09-17T07:03:51.994773+0000","last_undegraded":"2024-09-17T07:03:51.994773+0000","last_fullsized":"2024-09-17T07:03:51.994773+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:51.994756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:51.994756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:12.536626+0000","scrub_duration":0.001933269,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.757121+0000","last_change":"2024-09-17T07:03:38.818472+0000","last_active":"2024-09-17T07:03:47.757121+0000","last_peered":"2024-09-17T07:03:47.757121+0000","last_clean":"2024-09-17T07:03:47.757121+0000","last_became_active":"2024-09-17T07:02:16.021561+0000","last_became_peered":"2024-09-17T07:02:16.021561+0000","last_unstale":"2024-09-17T07:03:47.757121+0000","last_undegraded":"2024-09-17T07:03:47.757121+0000","last_fullsized":"2024-09-17T07:03:47.757121+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:38.818329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:38.818329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:05.115055+0000","scrub_duration":0.0016477,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.757653+0000","last_change":"2024-09-17T07:02:18.022359+0000","last_active":"2024-09-17T07:03:47.757653+0000","last_peered":"2024-09-17T07:03:47.757653+0000","last_clean":"2024-09-17T07:03:47.757653+0000","last_became_active":"2024-09-17T07:02:18.022082+0000","last_became_peered":"2024-09-17T07:02:18.022082+0000","last_unstale":"2024-09-17T07:03:47.757653+0000","last_undegraded":"2024-09-17T07:03:47.757653+0000","last_fullsized":"2024-09-17T07:03:47.757653+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:02:17.002904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"75'14","reported_seq":65,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.755036+0000","last_change":"2024-09-17T07:03:44.522720+0000","last_active":"2024-09-17T07:03:47.755036+0000","last_peered":"2024-09-17T07:03:47.755036+0000","last_clean":"2024-09-17T07:03:47.755036+0000","last_became_active":"2024-09-17T07:02:14.028857+0000","last_became_peered":"2024-09-17T07:02:14.028857+0000","last_unstale":"2024-09-17T07:03:47.755036+0000","last_undegraded":"2024-09-17T07:03:47.755036+0000","last_fullsized":"2024-09-17T07:03:47.755036+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'14","last_scrub_stamp":"2024-09-17T07:03:44.522631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:44.522631+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:13.112891+0000","scrub_duration":0.0019970970000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"72'2","reported_seq":52,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.757875+0000","last_change":"2024-09-17T07:03:25.432755+0000","last_active":"2024-09-17T07:03:47.757875+0000","last_peered":"2024-09-17T07:03:47.757875+0000","last_clean":"2024-09-17T07:03:47.757875+0000","last_became_active":"2024-09-17T07:02:12.005127+0000","last_became_peered":"2024-09-17T07:02:12.005127+0000","last_unstale":"2024-09-17T07:03:47.757875+0000","last_undegraded":"2024-09-17T07:03:47.757875+0000","last_fullsized":"2024-09-17T07:03:47.757875+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'2","last_scrub_stamp":"2024-09-17T07:03:25.432670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:25.432670+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:25.994239+0000","scrub_duration":0.001640454,"objects_trimmed":0,"snaptrim_duration":0.0010980219999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":45,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.757079+0000","last_change":"2024-09-17T07:03:14.771416+0000","last_active":"2024-09-17T07:03:47.757079+0000","last_peered":"2024-09-17T07:03:47.757079+0000","last_clean":"2024-09-17T07:03:47.757079+0000","last_became_active":"2024-09-17T07:02:12.004606+0000","last_became_peered":"2024-09-17T07:02:12.004606+0000","last_unstale":"2024-09-17T07:03:47.757079+0000","last_undegraded":"2024-09-17T07:03:47.757079+0000","last_fullsized":"2024-09-17T07:03:47.757079+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:14.771307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:14.771307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:19.133578+0000","scrub_duration":0.0020495679999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"78'128","reported_seq":294,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.752121+0000","last_change":"2024-09-17T07:02:43.225880+0000","last_active":"2024-09-17T07:03:47.752121+0000","last_peered":"2024-09-17T07:03:47.752121+0000","last_clean":"2024-09-17T07:03:47.752121+0000","last_became_active":"2024-09-17T07:02:03.020675+0000","last_became_peered":"2024-09-17T07:02:03.020675+0000","last_unstale":"2024-09-17T07:03:47.752121+0000","last_undegraded":"2024-09-17T07:03:47.752121+0000","last_fullsized":"2024-09-17T07:03:47.752121+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'117","last_scrub_stamp":"2024-09-17T07:02:43.225790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:00:09.390381+0000","last_clean_scrub_stamp":"2024-09-17T07:02:43.225790+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:12.817117+0000","scrub_duration":0.001686302,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.752328+0000","last_change":"2024-09-17T07:03:30.069684+0000","last_active":"2024-09-17T07:03:47.752328+0000","last_peered":"2024-09-17T07:03:47.752328+0000","last_clean":"2024-09-17T07:03:47.752328+0000","last_became_active":"2024-09-17T07:02:16.016031+0000","last_became_peered":"2024-09-17T07:02:16.016031+0000","last_unstale":"2024-09-17T07:03:47.752328+0000","last_undegraded":"2024-09-17T07:03:47.752328+0000","last_fullsized":"2024-09-17T07:03:47.752328+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:30.069608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:30.069608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:36.831454+0000","scrub_duration":0.0012155379999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.757283+0000","last_change":"2024-09-17T07:02:18.023684+0000","last_active":"2024-09-17T07:03:47.757283+0000","last_peered":"2024-09-17T07:03:47.757283+0000","last_clean":"2024-09-17T07:03:47.757283+0000","last_became_active":"2024-09-17T07:02:18.023321+0000","last_became_peered":"2024-09-17T07:02:18.023321+0000","last_unstale":"2024-09-17T07:03:47.757283+0000","last_undegraded":"2024-09-17T07:03:47.757283+0000","last_fullsized":"2024-09-17T07:03:47.757283+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:02:17.002904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"75'19","reported_seq":74,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.754839+0000","last_change":"2024-09-17T07:03:37.517877+0000","last_active":"2024-09-17T07:03:47.754839+0000","last_peered":"2024-09-17T07:03:47.754839+0000","last_clean":"2024-09-17T07:03:47.754839+0000","last_became_active":"2024-09-17T07:02:14.027313+0000","last_became_peered":"2024-09-17T07:02:14.027313+0000","last_unstale":"2024-09-17T07:03:47.754839+0000","last_undegraded":"2024-09-17T07:03:47.754839+0000","last_fullsized":"2024-09-17T07:03:47.754839+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-09-17T07:03:37.517760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:37.517760+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:03.675532+0000","scrub_duration":0.0028316579999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":46,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.753360+0000","last_change":"2024-09-17T07:03:25.672437+0000","last_active":"2024-09-17T07:03:47.753360+0000","last_peered":"2024-09-17T07:03:47.753360+0000","last_clean":"2024-09-17T07:03:47.753360+0000","last_became_active":"2024-09-17T07:02:12.012037+0000","last_became_peered":"2024-09-17T07:02:12.012037+0000","last_unstale":"2024-09-17T07:03:47.753360+0000","last_undegraded":"2024-09-17T07:03:47.753360+0000","last_fullsized":"2024-09-17T07:03:47.753360+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:25.672360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:25.672360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:54.809122+0000","scrub_duration":0.0012704459999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.752229+0000","last_change":"2024-09-17T07:03:27.660802+0000","last_active":"2024-09-17T07:03:47.752229+0000","last_peered":"2024-09-17T07:03:47.752229+0000","last_clean":"2024-09-17T07:03:47.752229+0000","last_became_active":"2024-09-17T07:02:16.014631+0000","last_became_peered":"2024-09-17T07:02:16.014631+0000","last_unstale":"2024-09-17T07:03:47.752229+0000","last_undegraded":"2024-09-17T07:03:47.752229+0000","last_fullsized":"2024-09-17T07:03:47.752229+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:27.660721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:27.660721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:50.654559+0000","scrub_duration":0.0012746820000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.752157+0000","last_change":"2024-09-17T07:02:18.023600+0000","last_active":"2024-09-17T07:03:47.752157+0000","last_peered":"2024-09-17T07:03:47.752157+0000","last_clean":"2024-09-17T07:03:47.752157+0000","last_became_active":"2024-09-17T07:02:18.023299+0000","last_became_peered":"2024-09-17T07:02:18.023299+0000","last_unstale":"2024-09-17T07:03:47.752157+0000","last_undegraded":"2024-09-17T07:03:47.752157+0000","last_fullsized":"2024-09-17T07:03:47.752157+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:02:17.002904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"75'9","reported_seq":60,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.752744+0000","last_change":"2024-09-17T07:03:46.892737+0000","last_active":"2024-09-17T07:03:47.752744+0000","last_peered":"2024-09-17T07:03:47.752744+0000","last_clean":"2024-09-17T07:03:47.752744+0000","last_became_active":"2024-09-17T07:02:14.014939+0000","last_became_peered":"2024-09-17T07:02:14.014939+0000","last_unstale":"2024-09-17T07:03:47.752744+0000","last_undegraded":"2024-09-17T07:03:47.752744+0000","last_fullsized":"2024-09-17T07:03:47.752744+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:03:46.892648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:46.892648+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:54.689233+0000","scrub_duration":0.002021162,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":45,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.758913+0000","last_change":"2024-09-17T07:03:40.074926+0000","last_active":"2024-09-17T07:03:47.758913+0000","last_peered":"2024-09-17T07:03:47.758913+0000","last_clean":"2024-09-17T07:03:47.758913+0000","last_became_active":"2024-09-17T07:02:12.000005+0000","last_became_peered":"2024-09-17T07:02:12.000005+0000","last_unstale":"2024-09-17T07:03:47.758913+0000","last_undegraded":"2024-09-17T07:03:47.758913+0000","last_fullsized":"2024-09-17T07:03:47.758913+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:40.074839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:40.074839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:50.612094+0000","scrub_duration":0.0013162040000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.752253+0000","last_change":"2024-09-17T07:03:43.943297+0000","last_active":"2024-09-17T07:03:47.752253+0000","last_peered":"2024-09-17T07:03:47.752253+0000","last_clean":"2024-09-17T07:03:47.752253+0000","last_became_active":"2024-09-17T07:02:16.017271+0000","last_became_peered":"2024-09-17T07:02:16.017271+0000","last_unstale":"2024-09-17T07:03:47.752253+0000","last_undegraded":"2024-09-17T07:03:47.752253+0000","last_fullsized":"2024-09-17T07:03:47.752253+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:43.943168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:43.943168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:01.611368+0000","scrub_duration":0.0016514609999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.752838+0000","last_change":"2024-09-17T07:02:18.027276+0000","last_active":"2024-09-17T07:03:47.752838+0000","last_peered":"2024-09-17T07:03:47.752838+0000","last_clean":"2024-09-17T07:03:47.752838+0000","last_became_active":"2024-09-17T07:02:18.026994+0000","last_became_peered":"2024-09-17T07:02:18.026994+0000","last_unstale":"2024-09-17T07:03:47.752838+0000","last_undegraded":"2024-09-17T07:03:47.752838+0000","last_fullsized":"2024-09-17T07:03:47.752838+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:02:17.002904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"75'15","reported_seq":68,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-17T07:03:50.737925+0000","last_change":"2024-09-17T07:03:50.738059+0000","last_active":"2024-09-17T07:03:50.737925+0000","last_peered":"2024-09-17T07:03:50.737925+0000","last_clean":"2024-09-17T07:03:50.737925+0000","last_became_active":"2024-09-17T07:02:14.028392+0000","last_became_peered":"2024-09-17T07:02:14.028392+0000","last_unstale":"2024-09-17T07:03:50.737925+0000","last_undegraded":"2024-09-17T07:03:50.737925+0000","last_fullsized":"2024-09-17T07:03:50.737925+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-09-17T07:03:50.737908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:50.737908+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:54.852092+0000","scrub_duration":0.0024040049999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'2","reported_seq":51,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.753233+0000","last_change":"2024-09-17T07:03:26.694471+0000","last_active":"2024-09-17T07:03:47.753233+0000","last_peered":"2024-09-17T07:03:47.753233+0000","last_clean":"2024-09-17T07:03:47.753233+0000","last_became_active":"2024-09-17T07:02:12.005538+0000","last_became_peered":"2024-09-17T07:02:12.005538+0000","last_unstale":"2024-09-17T07:03:47.753233+0000","last_undegraded":"2024-09-17T07:03:47.753233+0000","last_fullsized":"2024-09-17T07:03:47.753233+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-09-17T07:03:26.694366+0000","last_deep_scrub":"89'2","last_deep_scrub_stamp":"2024-09-17T07:03:26.694366+0000","last_clean_scrub_stamp":"2024-09-17T07:03:26.694366+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:38.099234+0000","scrub_duration":0.0026359439999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":420,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"75'8","reported_seq":73,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-17T07:03:48.912289+0000","last_change":"2024-09-17T07:03:45.995460+0000","last_active":"2024-09-17T07:03:48.912289+0000","last_peered":"2024-09-17T07:03:48.912289+0000","last_clean":"2024-09-17T07:03:48.912289+0000","last_became_active":"2024-09-17T07:02:16.015418+0000","last_became_peered":"2024-09-17T07:02:16.015418+0000","last_unstale":"2024-09-17T07:03:48.912289+0000","last_undegraded":"2024-09-17T07:03:48.912289+0000","last_fullsized":"2024-09-17T07:03:48.912289+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:03:45.995358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:45.995358+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:52.038789+0000","scrub_duration":0.001465163,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.756654+0000","last_change":"2024-09-17T07:03:22.823018+0000","last_active":"2024-09-17T07:03:47.756654+0000","last_peered":"2024-09-17T07:03:47.756654+0000","last_clean":"2024-09-17T07:03:47.756654+0000","last_became_active":"2024-09-17T07:02:18.363253+0000","last_became_peered":"2024-09-17T07:02:18.363253+0000","last_unstale":"2024-09-17T07:03:47.756654+0000","last_undegraded":"2024-09-17T07:03:47.756654+0000","last_fullsized":"2024-09-17T07:03:47.756654+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:22.822922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:03:22.822922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:29.158808+0000","scrub_duration":0.0012979910000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"75'12","reported_seq":67,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.758187+0000","last_change":"2024-09-17T07:03:39.074060+0000","last_active":"2024-09-17T07:03:47.758187+0000","last_peered":"2024-09-17T07:03:47.758187+0000","last_clean":"2024-09-17T07:03:47.758187+0000","last_became_active":"2024-09-17T07:02:14.010219+0000","last_became_peered":"2024-09-17T07:02:14.010219+0000","last_unstale":"2024-09-17T07:03:47.758187+0000","last_undegraded":"2024-09-17T07:03:47.758187+0000","last_fullsized":"2024-09-17T07:03:47.758187+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-09-17T07:03:39.073980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:39.073980+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:07.174292+0000","scrub_duration":0.002114061,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"68'2","reported_seq":53,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.759120+0000","last_change":"2024-09-17T07:03:28.021306+0000","last_active":"2024-09-17T07:03:47.759120+0000","last_peered":"2024-09-17T07:03:47.759120+0000","last_clean":"2024-09-17T07:03:47.759120+0000","last_became_active":"2024-09-17T07:02:12.006716+0000","last_became_peered":"2024-09-17T07:02:12.006716+0000","last_unstale":"2024-09-17T07:03:47.759120+0000","last_undegraded":"2024-09-17T07:03:47.759120+0000","last_fullsized":"2024-09-17T07:03:47.759120+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-09-17T07:03:28.021180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:28.021180+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:55.299746+0000","scrub_duration":0.0016672220000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.752000+0000","last_change":"2024-09-17T07:03:36.832832+0000","last_active":"2024-09-17T07:03:47.752000+0000","last_peered":"2024-09-17T07:03:47.752000+0000","last_clean":"2024-09-17T07:03:47.752000+0000","last_became_active":"2024-09-17T07:02:16.012261+0000","last_became_peered":"2024-09-17T07:02:16.012261+0000","last_unstale":"2024-09-17T07:03:47.752000+0000","last_undegraded":"2024-09-17T07:03:47.752000+0000","last_fullsized":"2024-09-17T07:03:47.752000+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:36.832752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:36.832752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:44.624052+0000","scrub_duration":0.001189278,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.757867+0000","last_change":"2024-09-17T07:02:18.025590+0000","last_active":"2024-09-17T07:03:47.757867+0000","last_peered":"2024-09-17T07:03:47.757867+0000","last_clean":"2024-09-17T07:03:47.757867+0000","last_became_active":"2024-09-17T07:02:18.024983+0000","last_became_peered":"2024-09-17T07:02:18.024983+0000","last_unstale":"2024-09-17T07:03:47.757867+0000","last_undegraded":"2024-09-17T07:03:47.757867+0000","last_fullsized":"2024-09-17T07:03:47.757867+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:02:17.002904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"75'9","reported_seq":60,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.752893+0000","last_change":"2024-09-17T07:03:26.005290+0000","last_active":"2024-09-17T07:03:47.752893+0000","last_peered":"2024-09-17T07:03:47.752893+0000","last_clean":"2024-09-17T07:03:47.752893+0000","last_became_active":"2024-09-17T07:02:14.029388+0000","last_became_peered":"2024-09-17T07:02:14.029388+0000","last_unstale":"2024-09-17T07:03:47.752893+0000","last_undegraded":"2024-09-17T07:03:47.752893+0000","last_fullsized":"2024-09-17T07:03:47.752893+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:03:26.005201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:26.005201+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:55.043365+0000","scrub_duration":0.0018022279999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":46,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.752079+0000","last_change":"2024-09-17T07:03:31.753577+0000","last_active":"2024-09-17T07:03:47.752079+0000","last_peered":"2024-09-17T07:03:47.752079+0000","last_clean":"2024-09-17T07:03:47.752079+0000","last_became_active":"2024-09-17T07:02:12.011327+0000","last_became_peered":"2024-09-17T07:02:12.011327+0000","last_unstale":"2024-09-17T07:03:47.752079+0000","last_undegraded":"2024-09-17T07:03:47.752079+0000","last_fullsized":"2024-09-17T07:03:47.752079+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:31.753470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:31.753470+0000","last_clean_scrub_stamp":"2024-09-17T07:03:31.753470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:51.025434+0000","scrub_duration":0.0012687670000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.758083+0000","last_change":"2024-09-17T07:03:21.314310+0000","last_active":"2024-09-17T07:03:47.758083+0000","last_peered":"2024-09-17T07:03:47.758083+0000","last_clean":"2024-09-17T07:03:47.758083+0000","last_became_active":"2024-09-17T07:02:16.022584+0000","last_became_peered":"2024-09-17T07:02:16.022584+0000","last_unstale":"2024-09-17T07:03:47.758083+0000","last_undegraded":"2024-09-17T07:03:47.758083+0000","last_fullsized":"2024-09-17T07:03:47.758083+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:21.314226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:21.314226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:29.256168+0000","scrub_duration":0.0016627219999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.752959+0000","last_change":"2024-09-17T07:03:19.603974+0000","last_active":"2024-09-17T07:03:47.752959+0000","last_peered":"2024-09-17T07:03:47.752959+0000","last_clean":"2024-09-17T07:03:47.752959+0000","last_became_active":"2024-09-17T07:02:18.017825+0000","last_became_peered":"2024-09-17T07:02:18.017825+0000","last_unstale":"2024-09-17T07:03:47.752959+0000","last_undegraded":"2024-09-17T07:03:47.752959+0000","last_fullsized":"2024-09-17T07:03:47.752959+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:19.603899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:03:19.603899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:31.400546+0000","scrub_duration":0.001425798,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.754677+0000","last_change":"2024-09-17T07:02:12.008326+0000","last_active":"2024-09-17T07:03:47.754677+0000","last_peered":"2024-09-17T07:03:47.754677+0000","last_clean":"2024-09-17T07:03:47.754677+0000","last_became_active":"2024-09-17T07:02:12.008065+0000","last_became_peered":"2024-09-17T07:02:12.008065+0000","last_unstale":"2024-09-17T07:03:47.754677+0000","last_undegraded":"2024-09-17T07:03:47.754677+0000","last_fullsized":"2024-09-17T07:03:47.754677+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:02:10.991363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"75'6","reported_seq":52,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.758217+0000","last_change":"2024-09-17T07:03:36.430683+0000","last_active":"2024-09-17T07:03:47.758217+0000","last_peered":"2024-09-17T07:03:47.758217+0000","last_clean":"2024-09-17T07:03:47.758217+0000","last_became_active":"2024-09-17T07:02:14.028655+0000","last_became_peered":"2024-09-17T07:02:14.028655+0000","last_unstale":"2024-09-17T07:03:47.758217+0000","last_undegraded":"2024-09-17T07:03:47.758217+0000","last_fullsized":"2024-09-17T07:03:47.758217+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'6","last_scrub_stamp":"2024-09-17T07:03:36.430591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:36.430591+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:44.221735+0000","scrub_duration":0.0020997730000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.757177+0000","last_change":"2024-09-17T07:03:39.844283+0000","last_active":"2024-09-17T07:03:47.757177+0000","last_peered":"2024-09-17T07:03:47.757177+0000","last_clean":"2024-09-17T07:03:47.757177+0000","last_became_active":"2024-09-17T07:02:16.016634+0000","last_became_peered":"2024-09-17T07:02:16.016634+0000","last_unstale":"2024-09-17T07:03:47.757177+0000","last_undegraded":"2024-09-17T07:03:47.757177+0000","last_fullsized":"2024-09-17T07:03:47.757177+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:39.844196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:39.844196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:07.747427+0000","scrub_duration":0.001646652,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.752562+0000","last_change":"2024-09-17T07:03:41.093127+0000","last_active":"2024-09-17T07:03:47.752562+0000","last_peered":"2024-09-17T07:03:47.752562+0000","last_clean":"2024-09-17T07:03:47.752562+0000","last_became_active":"2024-09-17T07:02:18.365419+0000","last_became_peered":"2024-09-17T07:02:18.365419+0000","last_unstale":"2024-09-17T07:03:47.752562+0000","last_undegraded":"2024-09-17T07:03:47.752562+0000","last_fullsized":"2024-09-17T07:03:47.752562+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:41.093049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:03:41.093049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:58.545295+0000","scrub_duration":0.001187794,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"75'10","reported_seq":58,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.757875+0000","last_change":"2024-09-17T07:03:14.178469+0000","last_active":"2024-09-17T07:03:47.757875+0000","last_peered":"2024-09-17T07:03:47.757875+0000","last_clean":"2024-09-17T07:03:47.757875+0000","last_became_active":"2024-09-17T07:02:14.027499+0000","last_became_peered":"2024-09-17T07:02:14.027499+0000","last_unstale":"2024-09-17T07:03:47.757875+0000","last_undegraded":"2024-09-17T07:03:47.757875+0000","last_fullsized":"2024-09-17T07:03:47.757875+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-09-17T07:03:14.178375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:14.178375+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:31.400568+0000","scrub_duration":0.0021187430000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":45,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.757005+0000","last_change":"2024-09-17T07:03:25.849841+0000","last_active":"2024-09-17T07:03:47.757005+0000","last_peered":"2024-09-17T07:03:47.757005+0000","last_clean":"2024-09-17T07:03:47.757005+0000","last_became_active":"2024-09-17T07:02:12.007280+0000","last_became_peered":"2024-09-17T07:02:12.007280+0000","last_unstale":"2024-09-17T07:03:47.757005+0000","last_undegraded":"2024-09-17T07:03:47.757005+0000","last_fullsized":"2024-09-17T07:03:47.757005+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:25.849698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:25.849698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:55.197183+0000","scrub_duration":0.0014293929999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"75'8","reported_seq":69,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-17T07:03:48.755465+0000","last_change":"2024-09-17T07:03:46.807330+0000","last_active":"2024-09-17T07:03:48.755465+0000","last_peered":"2024-09-17T07:03:48.755465+0000","last_clean":"2024-09-17T07:03:48.755465+0000","last_became_active":"2024-09-17T07:02:16.022583+0000","last_became_peered":"2024-09-17T07:02:16.022583+0000","last_unstale":"2024-09-17T07:03:48.755465+0000","last_undegraded":"2024-09-17T07:03:48.755465+0000","last_fullsized":"2024-09-17T07:03:48.755465+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:03:46.807209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:46.807209+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:03.451386+0000","scrub_duration":0.0021619619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.752609+0000","last_change":"2024-09-17T07:02:18.029691+0000","last_active":"2024-09-17T07:03:47.752609+0000","last_peered":"2024-09-17T07:03:47.752609+0000","last_clean":"2024-09-17T07:03:47.752609+0000","last_became_active":"2024-09-17T07:02:18.028796+0000","last_became_peered":"2024-09-17T07:02:18.028796+0000","last_unstale":"2024-09-17T07:03:47.752609+0000","last_undegraded":"2024-09-17T07:03:47.752609+0000","last_fullsized":"2024-09-17T07:03:47.752609+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:02:17.002904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"75'9","reported_seq":59,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.757582+0000","last_change":"2024-09-17T07:03:17.798712+0000","last_active":"2024-09-17T07:03:47.757582+0000","last_peered":"2024-09-17T07:03:47.757582+0000","last_clean":"2024-09-17T07:03:47.757582+0000","last_became_active":"2024-09-17T07:02:14.028485+0000","last_became_peered":"2024-09-17T07:02:14.028485+0000","last_unstale":"2024-09-17T07:03:47.757582+0000","last_undegraded":"2024-09-17T07:03:47.757582+0000","last_fullsized":"2024-09-17T07:03:47.757582+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:03:17.798582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:17.798582+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:27.282186+0000","scrub_duration":0.0023213259999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":46,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.752938+0000","last_change":"2024-09-17T07:03:16.948419+0000","last_active":"2024-09-17T07:03:47.752938+0000","last_peered":"2024-09-17T07:03:47.752938+0000","last_clean":"2024-09-17T07:03:47.752938+0000","last_became_active":"2024-09-17T07:02:12.009935+0000","last_became_peered":"2024-09-17T07:02:12.009935+0000","last_unstale":"2024-09-17T07:03:47.752938+0000","last_undegraded":"2024-09-17T07:03:47.752938+0000","last_fullsized":"2024-09-17T07:03:47.752938+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:16.948330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:16.948330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:46.609922+0000","scrub_duration":0.0017835489999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"75'8","reported_seq":69,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-17T07:03:48.757448+0000","last_change":"2024-09-17T07:03:42.469019+0000","last_active":"2024-09-17T07:03:48.757448+0000","last_peered":"2024-09-17T07:03:48.757448+0000","last_clean":"2024-09-17T07:03:48.757448+0000","last_became_active":"2024-09-17T07:02:16.016069+0000","last_became_peered":"2024-09-17T07:02:16.016069+0000","last_unstale":"2024-09-17T07:03:48.757448+0000","last_undegraded":"2024-09-17T07:03:48.757448+0000","last_fullsized":"2024-09-17T07:03:48.757448+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:03:42.468934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:42.468934+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:07.697686+0000","scrub_duration":0.0016937079999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.758988+0000","last_change":"2024-09-17T07:03:31.045257+0000","last_active":"2024-09-17T07:03:47.758988+0000","last_peered":"2024-09-17T07:03:47.758988+0000","last_clean":"2024-09-17T07:03:47.758988+0000","last_became_active":"2024-09-17T07:02:18.018716+0000","last_became_peered":"2024-09-17T07:02:18.018716+0000","last_unstale":"2024-09-17T07:03:47.758988+0000","last_undegraded":"2024-09-17T07:03:47.758988+0000","last_fullsized":"2024-09-17T07:03:47.758988+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:31.045167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:03:31.045167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:36.218836+0000","scrub_duration":0.0012599709999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"75'9","reported_seq":59,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.756795+0000","last_change":"2024-09-17T07:03:20.116712+0000","last_active":"2024-09-17T07:03:47.756795+0000","last_peered":"2024-09-17T07:03:47.756795+0000","last_clean":"2024-09-17T07:03:47.756795+0000","last_became_active":"2024-09-17T07:02:14.010289+0000","last_became_peered":"2024-09-17T07:02:14.010289+0000","last_unstale":"2024-09-17T07:03:47.756795+0000","last_undegraded":"2024-09-17T07:03:47.756795+0000","last_fullsized":"2024-09-17T07:03:47.756795+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:03:20.116602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:20.116602+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:35.267610+0000","scrub_duration":0.0032469130000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-17T07:03:51.518528+0000","last_change":"2024-09-17T07:03:51.518619+0000","last_active":"2024-09-17T07:03:51.518528+0000","last_peered":"2024-09-17T07:03:51.518528+0000","last_clean":"2024-09-17T07:03:51.518528+0000","last_became_active":"2024-09-17T07:02:12.007433+0000","last_became_peered":"2024-09-17T07:02:12.007433+0000","last_unstale":"2024-09-17T07:03:51.518528+0000","last_undegraded":"2024-09-17T07:03:51.518528+0000","last_fullsized":"2024-09-17T07:03:51.518528+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:51.518511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:51.518511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:09.559204+0000","scrub_duration":0.00177655,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-17T07:03:51.487623+0000","last_change":"2024-09-17T07:03:51.487699+0000","last_active":"2024-09-17T07:03:51.487623+0000","last_peered":"2024-09-17T07:03:51.487623+0000","last_clean":"2024-09-17T07:03:51.487623+0000","last_became_active":"2024-09-17T07:02:16.016216+0000","last_became_peered":"2024-09-17T07:02:16.016216+0000","last_unstale":"2024-09-17T07:03:51.487623+0000","last_undegraded":"2024-09-17T07:03:51.487623+0000","last_fullsized":"2024-09-17T07:03:51.487623+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:51.487610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:51.487610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:56.904290+0000","scrub_duration":0.0012946850000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.752318+0000","last_change":"2024-09-17T07:03:24.014436+0000","last_active":"2024-09-17T07:03:47.752318+0000","last_peered":"2024-09-17T07:03:47.752318+0000","last_clean":"2024-09-17T07:03:47.752318+0000","last_became_active":"2024-09-17T07:02:18.027622+0000","last_became_peered":"2024-09-17T07:02:18.027622+0000","last_unstale":"2024-09-17T07:03:47.752318+0000","last_undegraded":"2024-09-17T07:03:47.752318+0000","last_fullsized":"2024-09-17T07:03:47.752318+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:24.014336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:03:24.014336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:50.998219+0000","scrub_duration":0.001425125,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"75'11","reported_seq":63,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.758737+0000","last_change":"2024-09-17T07:03:18.899916+0000","last_active":"2024-09-17T07:03:47.758737+0000","last_peered":"2024-09-17T07:03:47.758737+0000","last_clean":"2024-09-17T07:03:47.758737+0000","last_became_active":"2024-09-17T07:02:14.016255+0000","last_became_peered":"2024-09-17T07:02:14.016255+0000","last_unstale":"2024-09-17T07:03:47.758737+0000","last_undegraded":"2024-09-17T07:03:47.758737+0000","last_fullsized":"2024-09-17T07:03:47.758737+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-09-17T07:03:18.899814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:18.899814+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:46.161729+0000","scrub_duration":0.0020491419999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"68'1","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.758812+0000","last_change":"2024-09-17T07:02:12.011847+0000","last_active":"2024-09-17T07:03:47.758812+0000","last_peered":"2024-09-17T07:03:47.758812+0000","last_clean":"2024-09-17T07:03:47.758812+0000","last_became_active":"2024-09-17T07:02:12.011461+0000","last_became_peered":"2024-09-17T07:02:12.011461+0000","last_unstale":"2024-09-17T07:03:47.758812+0000","last_undegraded":"2024-09-17T07:03:47.758812+0000","last_fullsized":"2024-09-17T07:03:47.758812+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:02:10.991363+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.756658+0000","last_change":"2024-09-17T07:03:33.126060+0000","last_active":"2024-09-17T07:03:47.756658+0000","last_peered":"2024-09-17T07:03:47.756658+0000","last_clean":"2024-09-17T07:03:47.756658+0000","last_became_active":"2024-09-17T07:02:16.018754+0000","last_became_peered":"2024-09-17T07:02:16.018754+0000","last_unstale":"2024-09-17T07:03:47.756658+0000","last_undegraded":"2024-09-17T07:03:47.756658+0000","last_fullsized":"2024-09-17T07:03:47.756658+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:33.125915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:33.125915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:56.098551+0000","scrub_duration":0.033614225999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.756850+0000","last_change":"2024-09-17T07:02:18.027727+0000","last_active":"2024-09-17T07:03:47.756850+0000","last_peered":"2024-09-17T07:03:47.756850+0000","last_clean":"2024-09-17T07:03:47.756850+0000","last_became_active":"2024-09-17T07:02:18.027427+0000","last_became_peered":"2024-09-17T07:02:18.027427+0000","last_unstale":"2024-09-17T07:03:47.756850+0000","last_undegraded":"2024-09-17T07:03:47.756850+0000","last_fullsized":"2024-09-17T07:03:47.756850+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:02:17.002904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"75'4","reported_seq":64,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-17T07:03:48.763164+0000","last_change":"2024-09-17T07:03:48.763164+0000","last_active":"2024-09-17T07:03:48.763164+0000","last_peered":"2024-09-17T07:03:48.763164+0000","last_clean":"2024-09-17T07:03:48.763164+0000","last_became_active":"2024-09-17T07:03:48.758220+0000","last_became_peered":"2024-09-17T07:03:48.758220+0000","last_unstale":"2024-09-17T07:03:48.763164+0000","last_undegraded":"2024-09-17T07:03:48.763164+0000","last_fullsized":"2024-09-17T07:03:48.763164+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'4","last_scrub_stamp":"2024-09-17T07:03:24.431043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:24.431043+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:38.933831+0000","scrub_duration":0.0015094500000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":45,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.757181+0000","last_change":"2024-09-17T07:03:44.809127+0000","last_active":"2024-09-17T07:03:47.757181+0000","last_peered":"2024-09-17T07:03:47.757181+0000","last_clean":"2024-09-17T07:03:47.757181+0000","last_became_active":"2024-09-17T07:02:12.008898+0000","last_became_peered":"2024-09-17T07:02:12.008898+0000","last_unstale":"2024-09-17T07:03:47.757181+0000","last_undegraded":"2024-09-17T07:03:47.757181+0000","last_fullsized":"2024-09-17T07:03:47.757181+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:44.809025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:44.809025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:03.894700+0000","scrub_duration":0.001586154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.754404+0000","last_change":"2024-09-17T07:03:47.445076+0000","last_active":"2024-09-17T07:03:47.754404+0000","last_peered":"2024-09-17T07:03:47.754404+0000","last_clean":"2024-09-17T07:03:47.754404+0000","last_became_active":"2024-09-17T07:02:16.014982+0000","last_became_peered":"2024-09-17T07:02:16.014982+0000","last_unstale":"2024-09-17T07:03:47.754404+0000","last_undegraded":"2024-09-17T07:03:47.754404+0000","last_fullsized":"2024-09-17T07:03:47.754404+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:47.444972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:47.444972+0000","last_clean_scrub_stamp":"2024-09-17T07:03:47.444972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:56.111360+0000","scrub_duration":0.0015734119999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.751937+0000","last_change":"2024-09-17T07:02:18.021319+0000","last_active":"2024-09-17T07:03:47.751937+0000","last_peered":"2024-09-17T07:03:47.751937+0000","last_clean":"2024-09-17T07:03:47.751937+0000","last_became_active":"2024-09-17T07:02:18.021014+0000","last_became_peered":"2024-09-17T07:02:18.021014+0000","last_unstale":"2024-09-17T07:03:47.751937+0000","last_undegraded":"2024-09-17T07:03:47.751937+0000","last_fullsized":"2024-09-17T07:03:47.751937+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:02:17.002904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"75'11","reported_seq":62,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.757066+0000","last_change":"2024-09-17T07:03:17.270057+0000","last_active":"2024-09-17T07:03:47.757066+0000","last_peered":"2024-09-17T07:03:47.757066+0000","last_clean":"2024-09-17T07:03:47.757066+0000","last_became_active":"2024-09-17T07:02:14.027316+0000","last_became_peered":"2024-09-17T07:02:14.027316+0000","last_unstale":"2024-09-17T07:03:47.757066+0000","last_undegraded":"2024-09-17T07:03:47.757066+0000","last_fullsized":"2024-09-17T07:03:47.757066+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-09-17T07:03:17.269941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:17.269941+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:32.856982+0000","scrub_duration":0.002326505,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":45,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.755178+0000","last_change":"2024-09-17T07:03:33.533200+0000","last_active":"2024-09-17T07:03:47.755178+0000","last_peered":"2024-09-17T07:03:47.755178+0000","last_clean":"2024-09-17T07:03:47.755178+0000","last_became_active":"2024-09-17T07:02:12.010195+0000","last_became_peered":"2024-09-17T07:02:12.010195+0000","last_unstale":"2024-09-17T07:03:47.755178+0000","last_undegraded":"2024-09-17T07:03:47.755178+0000","last_fullsized":"2024-09-17T07:03:47.755178+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:33.533112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:33.533112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:53.826577+0000","scrub_duration":0.0014453420000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.751774+0000","last_change":"2024-09-17T07:03:37.063510+0000","last_active":"2024-09-17T07:03:47.751774+0000","last_peered":"2024-09-17T07:03:47.751774+0000","last_clean":"2024-09-17T07:03:47.751774+0000","last_became_active":"2024-09-17T07:02:16.016183+0000","last_became_peered":"2024-09-17T07:02:16.016183+0000","last_unstale":"2024-09-17T07:03:47.751774+0000","last_undegraded":"2024-09-17T07:03:47.751774+0000","last_fullsized":"2024-09-17T07:03:47.751774+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:37.063409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:37.063409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:55.794307+0000","scrub_duration":0.0015220329999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.757817+0000","last_change":"2024-09-17T07:03:29.467351+0000","last_active":"2024-09-17T07:03:47.757817+0000","last_peered":"2024-09-17T07:03:47.757817+0000","last_clean":"2024-09-17T07:03:47.757817+0000","last_became_active":"2024-09-17T07:02:18.365656+0000","last_became_peered":"2024-09-17T07:02:18.365656+0000","last_unstale":"2024-09-17T07:03:47.757817+0000","last_undegraded":"2024-09-17T07:03:47.757817+0000","last_fullsized":"2024-09-17T07:03:47.757817+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:29.467252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:03:29.467252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:58.741564+0000","scrub_duration":0.001842804,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"75'1","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.752252+0000","last_change":"2024-09-17T07:03:46.957523+0000","last_active":"2024-09-17T07:03:47.752252+0000","last_peered":"2024-09-17T07:03:47.752252+0000","last_clean":"2024-09-17T07:03:47.752252+0000","last_became_active":"2024-09-17T07:02:18.021855+0000","last_became_peered":"2024-09-17T07:02:18.021855+0000","last_unstale":"2024-09-17T07:03:47.752252+0000","last_undegraded":"2024-09-17T07:03:47.752252+0000","last_fullsized":"2024-09-17T07:03:47.752252+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-17T07:03:46.957414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:03:46.957414+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:53.461463+0000","scrub_duration":0.00186878,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":45,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.756483+0000","last_change":"2024-09-17T07:03:19.775506+0000","last_active":"2024-09-17T07:03:47.756483+0000","last_peered":"2024-09-17T07:03:47.756483+0000","last_clean":"2024-09-17T07:03:47.756483+0000","last_became_active":"2024-09-17T07:02:12.005329+0000","last_became_peered":"2024-09-17T07:02:12.005329+0000","last_unstale":"2024-09-17T07:03:47.756483+0000","last_undegraded":"2024-09-17T07:03:47.756483+0000","last_fullsized":"2024-09-17T07:03:47.756483+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:19.775390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:19.775390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:48.303892+0000","scrub_duration":0.0011915630000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"75'10","reported_seq":48,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.752810+0000","last_change":"2024-09-17T07:02:14.029020+0000","last_active":"2024-09-17T07:03:47.752810+0000","last_peered":"2024-09-17T07:03:47.752810+0000","last_clean":"2024-09-17T07:03:47.752810+0000","last_became_active":"2024-09-17T07:02:14.028731+0000","last_became_peered":"2024-09-17T07:02:14.028731+0000","last_unstale":"2024-09-17T07:03:47.752810+0000","last_undegraded":"2024-09-17T07:03:47.752810+0000","last_fullsized":"2024-09-17T07:03:47.752810+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:02:12.994221+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"75'8","reported_seq":61,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-17T07:03:48.757551+0000","last_change":"2024-09-17T07:02:16.020978+0000","last_active":"2024-09-17T07:03:48.757551+0000","last_peered":"2024-09-17T07:03:48.757551+0000","last_clean":"2024-09-17T07:03:48.757551+0000","last_became_active":"2024-09-17T07:02:16.020560+0000","last_became_peered":"2024-09-17T07:02:16.020560+0000","last_unstale":"2024-09-17T07:03:48.757551+0000","last_undegraded":"2024-09-17T07:03:48.757551+0000","last_fullsized":"2024-09-17T07:03:48.757551+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:02:14.999435+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.758405+0000","last_change":"2024-09-17T07:02:18.021959+0000","last_active":"2024-09-17T07:03:47.758405+0000","last_peered":"2024-09-17T07:03:47.758405+0000","last_clean":"2024-09-17T07:03:47.758405+0000","last_became_active":"2024-09-17T07:02:18.021603+0000","last_became_peered":"2024-09-17T07:02:18.021603+0000","last_unstale":"2024-09-17T07:03:47.758405+0000","last_undegraded":"2024-09-17T07:03:47.758405+0000","last_fullsized":"2024-09-17T07:03:47.758405+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:02:17.002904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":45,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.757423+0000","last_change":"2024-09-17T07:03:20.305248+0000","last_active":"2024-09-17T07:03:47.757423+0000","last_peered":"2024-09-17T07:03:47.757423+0000","last_clean":"2024-09-17T07:03:47.757423+0000","last_became_active":"2024-09-17T07:02:12.009837+0000","last_became_peered":"2024-09-17T07:02:12.009837+0000","last_unstale":"2024-09-17T07:03:47.757423+0000","last_undegraded":"2024-09-17T07:03:47.757423+0000","last_fullsized":"2024-09-17T07:03:47.757423+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:20.305165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:20.305165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:24.412968+0000","scrub_duration":0.0024485969999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"75'11","reported_seq":83,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-17T07:03:48.766828+0000","last_change":"2024-09-17T07:03:48.766828+0000","last_active":"2024-09-17T07:03:48.766828+0000","last_peered":"2024-09-17T07:03:48.766828+0000","last_clean":"2024-09-17T07:03:48.766828+0000","last_became_active":"2024-09-17T07:03:48.760390+0000","last_became_peered":"2024-09-17T07:03:48.760390+0000","last_unstale":"2024-09-17T07:03:48.766828+0000","last_undegraded":"2024-09-17T07:03:48.766828+0000","last_fullsized":"2024-09-17T07:03:48.766828+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-09-17T07:03:24.473712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:24.473712+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:32.437523+0000","scrub_duration":0.0021033010000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.752735+0000","last_change":"2024-09-17T07:02:16.019989+0000","last_active":"2024-09-17T07:03:47.752735+0000","last_peered":"2024-09-17T07:03:47.752735+0000","last_clean":"2024-09-17T07:03:47.752735+0000","last_became_active":"2024-09-17T07:02:16.019701+0000","last_became_peered":"2024-09-17T07:02:16.019701+0000","last_unstale":"2024-09-17T07:03:47.752735+0000","last_undegraded":"2024-09-17T07:03:47.752735+0000","last_fullsized":"2024-09-17T07:03:47.752735+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:02:14.999435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.758376+0000","last_change":"2024-09-17T07:02:18.364013+0000","last_active":"2024-09-17T07:03:47.758376+0000","last_peered":"2024-09-17T07:03:47.758376+0000","last_clean":"2024-09-17T07:03:47.758376+0000","last_became_active":"2024-09-17T07:02:18.363752+0000","last_became_peered":"2024-09-17T07:02:18.363752+0000","last_unstale":"2024-09-17T07:03:47.758376+0000","last_undegraded":"2024-09-17T07:03:47.758376+0000","last_fullsized":"2024-09-17T07:03:47.758376+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:02:17.002904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":54,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-17T07:03:48.759710+0000","last_change":"2024-09-17T07:03:48.759710+0000","last_active":"2024-09-17T07:03:48.759710+0000","last_peered":"2024-09-17T07:03:48.759710+0000","last_clean":"2024-09-17T07:03:48.759710+0000","last_became_active":"2024-09-17T07:03:48.759384+0000","last_became_peered":"2024-09-17T07:03:48.759384+0000","last_unstale":"2024-09-17T07:03:48.759710+0000","last_undegraded":"2024-09-17T07:03:48.759710+0000","last_fullsized":"2024-09-17T07:03:48.759710+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:46.447902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:46.447902+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:59.151253+0000","scrub_duration":0.0011905419999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"75'15","reported_seq":69,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.753196+0000","last_change":"2024-09-17T07:03:16.605256+0000","last_active":"2024-09-17T07:03:47.753196+0000","last_peered":"2024-09-17T07:03:47.753196+0000","last_clean":"2024-09-17T07:03:47.753196+0000","last_became_active":"2024-09-17T07:02:14.016273+0000","last_became_peered":"2024-09-17T07:02:14.016273+0000","last_unstale":"2024-09-17T07:03:47.753196+0000","last_undegraded":"2024-09-17T07:03:47.753196+0000","last_fullsized":"2024-09-17T07:03:47.753196+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-09-17T07:03:16.605163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:16.605163+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:40.470486+0000","scrub_duration":0.002000763,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.758642+0000","last_change":"2024-09-17T07:03:47.013173+0000","last_active":"2024-09-17T07:03:47.758642+0000","last_peered":"2024-09-17T07:03:47.758642+0000","last_clean":"2024-09-17T07:03:47.758642+0000","last_became_active":"2024-09-17T07:02:16.015903+0000","last_became_peered":"2024-09-17T07:02:16.015903+0000","last_unstale":"2024-09-17T07:03:47.758642+0000","last_undegraded":"2024-09-17T07:03:47.758642+0000","last_fullsized":"2024-09-17T07:03:47.758642+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:47.013069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:47.013069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:05.642647+0000","scrub_duration":0.0012432350000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.758099+0000","last_change":"2024-09-17T07:02:18.366797+0000","last_active":"2024-09-17T07:03:47.758099+0000","last_peered":"2024-09-17T07:03:47.758099+0000","last_clean":"2024-09-17T07:03:47.758099+0000","last_became_active":"2024-09-17T07:02:18.366466+0000","last_became_peered":"2024-09-17T07:02:18.366466+0000","last_unstale":"2024-09-17T07:03:47.758099+0000","last_undegraded":"2024-09-17T07:03:47.758099+0000","last_fullsized":"2024-09-17T07:03:47.758099+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:02:17.002904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":45,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.757993+0000","last_change":"2024-09-17T07:03:44.446996+0000","last_active":"2024-09-17T07:03:47.757993+0000","last_peered":"2024-09-17T07:03:47.757993+0000","last_clean":"2024-09-17T07:03:47.757993+0000","last_became_active":"2024-09-17T07:02:12.005493+0000","last_became_peered":"2024-09-17T07:02:12.005493+0000","last_unstale":"2024-09-17T07:03:47.757993+0000","last_undegraded":"2024-09-17T07:03:47.757993+0000","last_fullsized":"2024-09-17T07:03:47.757993+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:44.446914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:44.446914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:06.273158+0000","scrub_duration":0.0029497619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"86'17","reported_seq":71,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.757050+0000","last_change":"2024-09-17T07:03:18.241178+0000","last_active":"2024-09-17T07:03:47.757050+0000","last_peered":"2024-09-17T07:03:47.757050+0000","last_clean":"2024-09-17T07:03:47.757050+0000","last_became_active":"2024-09-17T07:02:14.017737+0000","last_became_peered":"2024-09-17T07:02:14.017737+0000","last_unstale":"2024-09-17T07:03:47.757050+0000","last_undegraded":"2024-09-17T07:03:47.757050+0000","last_fullsized":"2024-09-17T07:03:47.757050+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'17","last_scrub_stamp":"2024-09-17T07:03:18.241067+0000","last_deep_scrub":"86'17","last_deep_scrub_stamp":"2024-09-17T07:03:18.241067+0000","last_clean_scrub_stamp":"2024-09-17T07:03:18.241067+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:35.855446+0000","scrub_duration":0.0047694219999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T07:03:47.758361+0000","last_change":"2024-09-17T07:03:26.978352+0000","last_active":"2024-09-17T07:03:47.758361+0000","last_peered":"2024-09-17T07:03:47.758361+0000","last_clean":"2024-09-17T07:03:47.758361+0000","last_became_active":"2024-09-17T07:02:16.017362+0000","last_became_peered":"2024-09-17T07:02:16.017362+0000","last_unstale":"2024-09-17T07:03:47.758361+0000","last_undegraded":"2024-09-17T07:03:47.758361+0000","last_fullsized":"2024-09-17T07:03:47.758361+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:26.978262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:26.978262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:38.363709+0000","scrub_duration":0.0016151990000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":396,"ondisk_log_size":396,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":94,"num_read_kb":89,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1386,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":12,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":4158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037786,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32292,"kb_used_data":1176,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93712860,"statfs":{"total":95995035648,"available":95961968640,"internally_reserved":0,"allocated":1204224,"data_stored":875363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:03:02 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.38900000000000001}]},{"osd":1,"last update":"Tue Sep 17 07:03:02 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.439},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.34399999999999997}]},{"osd":2,"last update":"Tue Sep 17 07:03:02 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.622,"5min":0.622,"15min":0.622},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.37}]},{"osd":3,"last update":"Tue Sep 17 07:03:02 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.40999999999999998}]},{"osd":4,"last update":"Tue Sep 17 07:03:02 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"last":0.51800000000000002}]},{"osd":5,"last update":"Tue Sep 17 07:03:02 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.495},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.55900000000000005}]},{"osd":6,"last update":"Tue Sep 17 07:03:02 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.54600000000000004}]}]},{"osd":6,"up_from":50,"seq":214748364829,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32276,"kb_used_data":1160,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93712876,"statfs":{"total":95995035648,"available":95961985024,"internally_reserved":0,"allocated":1187840,"data_stored":874678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:03:43 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.49299999999999999,"15min":0.497},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.66500000000000004,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.48899999999999999,"15min":0.48999999999999999},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.63100000000000001,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.59499999999999997}]},{"osd":1,"last update":"Tue Sep 17 07:03:43 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.48599999999999999,"15min":0.48799999999999999},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.61699999999999999,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.52300000000000002,"15min":0.53000000000000003},"min":{"1min":0.34999999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.55300000000000005,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.624}]},{"osd":2,"last update":"Tue Sep 17 07:03:43 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.51500000000000001,"15min":0.51900000000000002},"min":{"1min":0.36299999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.68500000000000005,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51800000000000002,"15min":0.51900000000000002},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.64400000000000002,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.54000000000000004}]},{"osd":3,"last update":"Tue Sep 17 07:03:43 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.47699999999999998,"15min":0.47899999999999998},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.60499999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.53600000000000003,"15min":0.53800000000000003},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.67100000000000004,"5min":0.88,"15min":0.88},"last":0.68700000000000006}]},{"osd":4,"last update":"Tue Sep 17 07:03:43 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.41999999999999998,"15min":0.41999999999999998},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.52800000000000002,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.443,"5min":0.41399999999999998,"15min":0.40899999999999997},"min":{"1min":0.32600000000000001,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.59599999999999997,"5min":0.628,"15min":0.628},"last":0.44500000000000001}]},{"osd":5,"last update":"Tue Sep 17 07:03:43 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.42999999999999999,"15min":0.42399999999999999},"min":{"1min":0.32800000000000001,"5min":0.31,"15min":0.31},"max":{"1min":0.60199999999999998,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.432,"15min":0.42799999999999999},"min":{"1min":0.313,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.38900000000000001}]},{"osd":7,"last update":"Tue Sep 17 07:03:03 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.51300000000000001}]}]},{"osd":5,"up_from":42,"seq":180388626466,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31696,"kb_used_data":584,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713456,"statfs":{"total":95995035648,"available":95962578944,"internally_reserved":0,"allocated":598016,"data_stored":285120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:03:15 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.50600000000000001,"15min":0.51500000000000001},"min":{"1min":0.34100000000000003,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.56699999999999995,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.497,"15min":0.505},"min":{"1min":0.31900000000000001,"5min":0.313,"15min":0.313},"max":{"1min":0.65500000000000003,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.51100000000000001}]},{"osd":1,"last update":"Tue Sep 17 07:03:15 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.53200000000000003,"15min":0.54200000000000004},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.60799999999999998,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.443,"5min":1.0449999999999999,"15min":1.145},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.56200000000000006,"5min":14.054,"15min":14.054},"last":0.55000000000000004}]},{"osd":2,"last update":"Tue Sep 17 07:03:15 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.54800000000000004,"15min":0.56000000000000005},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.66500000000000004,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.495},{"interface":"front","average":{"1min":0.505,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.38600000000000001,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.63600000000000001,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.623}]},{"osd":3,"last update":"Tue Sep 17 07:03:15 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.54000000000000004,"15min":0.54900000000000004},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.63800000000000001,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.496,"5min":0.53900000000000003,"15min":0.54600000000000004},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.66400000000000003,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.60099999999999998}]},{"osd":4,"last update":"Tue Sep 17 07:03:15 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.40000000000000002,"15min":0.39900000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"last":0.38},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.432,"15min":0.434},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.60499999999999998,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.34699999999999998}]},{"osd":6,"last update":"Tue Sep 17 07:03:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.42299999999999999,"15min":0.42499999999999999},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.439,"5min":0.44400000000000001,"15min":0.44500000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.63800000000000001,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.309}]},{"osd":7,"last update":"Tue Sep 17 07:03:01 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.32700000000000001}]}]},{"osd":4,"up_from":33,"seq":141733920808,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31716,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713436,"statfs":{"total":95995035648,"available":95962558464,"internally_reserved":0,"allocated":614400,"data_stored":284707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:02:53 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.56299999999999994,"15min":0.57499999999999996},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.71499999999999997,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.60199999999999998,"15min":0.61499999999999999},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.69799999999999995,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.53000000000000003}]},{"osd":1,"last update":"Tue Sep 17 07:02:53 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.56000000000000005,"15min":0.57199999999999995},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.68400000000000005,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.502},{"interface":"front","average":{"1min":0.495,"5min":0.57999999999999996,"15min":0.59399999999999997},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.67900000000000005,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.61599999999999999}]},{"osd":2,"last update":"Tue Sep 17 07:02:53 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.56899999999999995,"15min":0.58299999999999996},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.69999999999999996,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.57799999999999996,"15min":0.58799999999999997},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.77100000000000002,"5min":0.877,"15min":0.877},"last":0.48999999999999999}]},{"osd":3,"last update":"Tue Sep 17 07:02:53 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.59299999999999997,"15min":0.60799999999999998},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.74299999999999999,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.55900000000000005,"15min":0.56399999999999995},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.66800000000000004,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.55600000000000005}]},{"osd":5,"last update":"Tue Sep 17 07:03:19 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.39600000000000002,"15min":0.38600000000000001},"min":{"1min":0.316,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.41699999999999998,"15min":0.40699999999999997},"min":{"1min":0.34799999999999998,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.40400000000000003}]},{"osd":6,"last update":"Tue Sep 17 07:03:38 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.39900000000000002,"15min":0.39500000000000002},"min":{"1min":0.313,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.41399999999999998,"15min":0.40699999999999997},"min":{"1min":0.314,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.46400000000000002}]},{"osd":7,"last update":"Tue Sep 17 07:03:00 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.443}]}]},{"osd":3,"up_from":27,"seq":115964117034,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31736,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713416,"statfs":{"total":95995035648,"available":95962537984,"internally_reserved":0,"allocated":638976,"data_stored":285491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:03:39 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.36099999999999999,"15min":0.36299999999999999},"min":{"1min":0.247,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.48399999999999999,"5min":0.55100000000000005,"15min":0.55100000000000005},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.371,"5min":0.379,"15min":0.378},"min":{"1min":0.26000000000000001,"5min":0.189,"15min":0.189},"max":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.29699999999999999}]},{"osd":1,"last update":"Tue Sep 17 07:03:39 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.39700000000000002,"15min":0.40500000000000003},"min":{"1min":0.24199999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.52100000000000002,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.441,"15min":0.44800000000000001},"min":{"1min":0.29899999999999999,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.58499999999999996,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.32900000000000001}]},{"osd":2,"last update":"Tue Sep 17 07:03:39 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.41299999999999998,"15min":0.42099999999999999},"min":{"1min":0.30599999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.59499999999999997,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.41499999999999998,"15min":0.41899999999999998},"min":{"1min":0.29499999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.35199999999999998}]},{"osd":4,"last update":"Tue Sep 17 07:02:56 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.56000000000000005,"15min":0.56399999999999995},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.754,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.53100000000000003,"15min":0.53500000000000003},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.43099999999999999}]},{"osd":5,"last update":"Tue Sep 17 07:03:18 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.59799999999999998,"15min":0.60599999999999998},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.58299999999999996,"15min":0.58499999999999996},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.79100000000000004,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.58899999999999997}]},{"osd":6,"last update":"Tue Sep 17 07:03:39 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.56999999999999995,"15min":0.56599999999999995},"min":{"1min":0.40400000000000003,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.76700000000000002,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58899999999999997,"15min":0.58999999999999997},"min":{"1min":0.42699999999999999,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.78600000000000003,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.60599999999999998}]},{"osd":7,"last update":"Tue Sep 17 07:03:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.44800000000000001}]}]},{"osd":2,"up_from":18,"seq":77309411376,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31708,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713444,"statfs":{"total":95995035648,"available":95962566656,"internally_reserved":0,"allocated":606208,"data_stored":285240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:03:15 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.35499999999999998,"15min":0.34699999999999998},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.35899999999999999,"15min":0.35099999999999998},"min":{"1min":0.24399999999999999,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.50600000000000001,"5min":0.625,"15min":0.625},"last":0.33200000000000002}]},{"osd":1,"last update":"Tue Sep 17 07:03:15 2024","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.35499999999999998,"15min":0.35099999999999998},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.48999999999999999,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.23699999999999999},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.38500000000000001,"15min":0.374},"min":{"1min":0.28599999999999998,"5min":0.251,"15min":0.251},"max":{"1min":0.55700000000000005,"5min":0.58399999999999996,"15min":0.58399999999999996},"last":0.34999999999999998}]},{"osd":3,"last update":"Tue Sep 17 07:03:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.435,"15min":0.437},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.57399999999999995,"5min":0.60499999999999998,"15min":0.60499999999999998},"last":0.317},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.42599999999999999,"15min":0.42799999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.58999999999999997,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.27900000000000003}]},{"osd":4,"last update":"Tue Sep 17 07:02:52 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56499999999999995,"15min":0.56599999999999995},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58599999999999997,"15min":0.58699999999999997},"min":{"1min":0.42999999999999999,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.52000000000000002}]},{"osd":5,"last update":"Tue Sep 17 07:03:15 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.101,"5min":1.101,"15min":1.101},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.57799999999999996,"15min":0.58999999999999997},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.78200000000000003,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.504}]},{"osd":6,"last update":"Tue Sep 17 07:03:44 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.59499999999999997,"15min":0.60099999999999998},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.61499999999999999}]},{"osd":7,"last update":"Tue Sep 17 07:03:03 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":3.9289999999999998,"5min":3.9289999999999998,"15min":3.9289999999999998},"last":0.54400000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574901,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27648,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717504,"statfs":{"total":95995035648,"available":95966724096,"internally_reserved":0,"allocated":647168,"data_stored":286632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:03:46 2024","interfaces":[{"interface":"back","average":{"1min":0.315,"5min":0.34399999999999997,"15min":0.33400000000000002},"min":{"1min":0.21099999999999999,"5min":0.193,"15min":0.193},"max":{"1min":0.46100000000000002,"5min":0.56000000000000005,"15min":0.56000000000000005},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.33800000000000002,"5min":0.36899999999999999,"15min":0.36199999999999999},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.47399999999999998,"5min":0.622,"15min":0.622},"last":0.41999999999999998}]},{"osd":2,"last update":"Tue Sep 17 07:03:12 2024","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.39600000000000002,"15min":0.40100000000000002},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.53100000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.40600000000000003,"15min":0.41199999999999998},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.60199999999999998,"5min":0.625,"15min":0.625},"last":0.377}]},{"osd":3,"last update":"Tue Sep 17 07:03:37 2024","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.441,"15min":0.45800000000000002},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.41599999999999998,"15min":0.438},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.54900000000000004,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.39100000000000001}]},{"osd":4,"last update":"Tue Sep 17 07:02:54 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.57699999999999996,"15min":0.57399999999999995},"min":{"1min":0.41199999999999998,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.56999999999999995,"15min":0.56799999999999995},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.71799999999999997,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.60399999999999998}]},{"osd":5,"last update":"Tue Sep 17 07:03:19 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.621,"15min":0.58699999999999997},"min":{"1min":0.38700000000000001,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":6.0060000000000002,"5min":6.0060000000000002,"15min":6.0060000000000002},"last":0.5},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.56299999999999994,"15min":0.56499999999999995},"min":{"1min":0.38200000000000001,"5min":0.375,"15min":0.375},"max":{"1min":0.752,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.57399999999999995}]},{"osd":6,"last update":"Tue Sep 17 07:03:41 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.63},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.58699999999999997,"15min":0.59299999999999997},"min":{"1min":0.38500000000000001,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.77200000000000002,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.48099999999999998}]},{"osd":7,"last update":"Tue Sep 17 07:03:01 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.46899999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738424,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28200,"kb_used_data":1184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716952,"statfs":{"total":95995035648,"available":95966158848,"internally_reserved":0,"allocated":1212416,"data_stored":875808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 17 07:03:47 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.35099999999999998,"15min":0.33000000000000002},"min":{"1min":0.25600000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.52600000000000002,"5min":0.58499999999999996,"15min":0.58499999999999996},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.371,"15min":0.34499999999999997},"min":{"1min":0.26400000000000001,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.39200000000000002}]},{"osd":2,"last update":"Tue Sep 17 07:03:15 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.38400000000000001,"15min":0.377},"min":{"1min":0.251,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.61099999999999999,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.378,"5min":0.37,"15min":0.36299999999999999},"min":{"1min":0.26600000000000001,"5min":0.246,"15min":0.246},"max":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"last":0.27100000000000002}]},{"osd":3,"last update":"Tue Sep 17 07:03:43 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.42399999999999999,"15min":0.436},"min":{"1min":0.30199999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.55900000000000005,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.42199999999999999,"15min":0.433},"min":{"1min":0.313,"5min":0.248,"15min":0.248},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.372}]},{"osd":4,"last update":"Tue Sep 17 07:02:56 2024","interfaces":[{"interface":"back","average":{"1min":1.1319999999999999,"5min":0.67500000000000004,"15min":0.59899999999999998},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":12.573,"5min":12.573,"15min":12.573},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.67700000000000005}]},{"osd":5,"last update":"Tue Sep 17 07:03:19 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.55500000000000005,"15min":0.55000000000000004},"min":{"1min":0.376,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.54800000000000004,"15min":0.54200000000000004},"min":{"1min":0.39700000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.71199999999999997}]},{"osd":6,"last update":"Tue Sep 17 07:03:43 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.55000000000000004,"15min":0.54700000000000004},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.55400000000000005,"15min":0.55000000000000004},"min":{"1min":0.40500000000000003,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.68700000000000006}]},{"osd":7,"last update":"Tue Sep 17 07:03:01 2024","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.2010000000000001,"15min":1.2010000000000001},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":12.548,"5min":12.548,"15min":12.548},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.63400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T07:03:57.707 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-17T07:03:57.708 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd dump 2024-09-17T07:03:57.954 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:Sep 17 07:03:57 smithi100 podman[73962]: 2024-09-17 07:03:57.813544323 +0000 UTC m=+1.290737662 container init 9269bad3ef8f51d11ff3571d2ae32d698bcd73f06f6f34b94670224e1810c4f0 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-17T07:03:57.955 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:Sep 17 07:03:57 smithi100 podman[73962]: 2024-09-17 07:03:57.816893941 +0000 UTC m=+1.294087272 container start 9269bad3ef8f51d11ff3571d2ae32d698bcd73f06f6f34b94670224e1810c4f0 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-17T07:03:57.955 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:Sep 17 07:03:57 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-alertmanager-a[73990]: ts=2024-09-17T07:03:57.829Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-09-17T07:03:57.955 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:Sep 17 07:03:57 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-alertmanager-a[73990]: ts=2024-09-17T07:03:57.829Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-09-17T07:03:57.955 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:Sep 17 07:03:57 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-alertmanager-a[73990]: ts=2024-09-17T07:03:57.830Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.100 port=9094 2024-09-17T07:03:57.955 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:Sep 17 07:03:57 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-alertmanager-a[73990]: ts=2024-09-17T07:03:57.831Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-09-17T07:03:57.955 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:Sep 17 07:03:57 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-alertmanager-a[73990]: ts=2024-09-17T07:03:57.858Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-17T07:03:57.955 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:Sep 17 07:03:57 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-alertmanager-a[73990]: ts=2024-09-17T07:03:57.859Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-17T07:03:57.955 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:Sep 17 07:03:57 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-alertmanager-a[73990]: ts=2024-09-17T07:03:57.861Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-09-17T07:03:57.955 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:Sep 17 07:03:57 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-alertmanager-a[73990]: ts=2024-09-17T07:03:57.861Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-09-17T07:03:57.955 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:Sep 17 07:03:57 smithi100 bash[73962]: 9269bad3ef8f51d11ff3571d2ae32d698bcd73f06f6f34b94670224e1810c4f0 2024-09-17T07:03:57.956 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:Sep 17 07:03:57 smithi100 systemd[1]: Started Ceph alertmanager.a for e848267c-74c1-11ef-bceb-c7b262605968. 2024-09-17T07:03:58.002 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:03:58.284 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:57 smithi100 ceph-mon[29092]: from='client.14736 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:03:58.284 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:57 smithi100 ceph-mon[29092]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 11 B/s, 1 objects/s recovering 2024-09-17T07:03:58.284 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:57 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:58.284 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:57 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:58.285 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:57 smithi100 ceph-mon[39462]: from='client.14736 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:03:58.285 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:57 smithi100 ceph-mon[39462]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 11 B/s, 1 objects/s recovering 2024-09-17T07:03:58.285 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:57 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:58.285 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:57 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:58.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:57 smithi135 ceph-mon[33238]: from='client.14736 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:03:58.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:57 smithi135 ceph-mon[33238]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 11 B/s, 1 objects/s recovering 2024-09-17T07:03:58.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:57 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:58.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:57 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:03:58.294 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:03:58 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:03:59.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:58 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:03:59.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:58 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:03:59.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:58 smithi135 ceph-mon[33238]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-17T07:03:59.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:58 smithi135 ceph-mon[33238]: osdmap e94: 8 total, 8 up, 8 in 2024-09-17T07:03:59.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:58 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:03:59.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:58 smithi135 ceph-mon[33238]: 6.9 deep-scrub starts 2024-09-17T07:03:59.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:58 smithi135 ceph-mon[33238]: 6.9 deep-scrub ok 2024-09-17T07:03:59.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:58 smithi135 ceph-mon[33238]: 6.7 deep-scrub starts 2024-09-17T07:03:59.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:58 smithi135 ceph-mon[33238]: 6.7 deep-scrub ok 2024-09-17T07:03:59.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:58 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:03:59.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:58 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:03:59.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:58 smithi100 ceph-mon[39462]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-17T07:03:59.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:58 smithi100 ceph-mon[39462]: osdmap e94: 8 total, 8 up, 8 in 2024-09-17T07:03:59.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:58 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:03:59.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:58 smithi100 ceph-mon[39462]: 6.9 deep-scrub starts 2024-09-17T07:03:59.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:58 smithi100 ceph-mon[39462]: 6.9 deep-scrub ok 2024-09-17T07:03:59.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:58 smithi100 ceph-mon[39462]: 6.7 deep-scrub starts 2024-09-17T07:03:59.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:58 smithi100 ceph-mon[39462]: 6.7 deep-scrub ok 2024-09-17T07:03:59.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:58 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:03:59.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:58 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:03:59.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:58 smithi100 ceph-mon[29092]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-17T07:03:59.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:58 smithi100 ceph-mon[29092]: osdmap e94: 8 total, 8 up, 8 in 2024-09-17T07:03:59.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:58 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:03:59.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:58 smithi100 ceph-mon[29092]: 6.9 deep-scrub starts 2024-09-17T07:03:59.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:58 smithi100 ceph-mon[29092]: 6.9 deep-scrub ok 2024-09-17T07:03:59.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:58 smithi100 ceph-mon[29092]: 6.7 deep-scrub starts 2024-09-17T07:03:59.129 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:58 smithi100 ceph-mon[29092]: 6.7 deep-scrub ok 2024-09-17T07:03:59.871 INFO:teuthology.orchestra.run.smithi100.stdout:epoch 94 2024-09-17T07:03:59.871 INFO:teuthology.orchestra.run.smithi100.stdout:fsid e848267c-74c1-11ef-bceb-c7b262605968 2024-09-17T07:03:59.871 INFO:teuthology.orchestra.run.smithi100.stdout:created 2024-09-17T06:57:05.010532+0000 2024-09-17T07:03:59.871 INFO:teuthology.orchestra.run.smithi100.stdout:modified 2024-09-17T07:03:57.948666+0000 2024-09-17T07:03:59.871 INFO:teuthology.orchestra.run.smithi100.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-09-17T07:03:59.871 INFO:teuthology.orchestra.run.smithi100.stdout:crush_version 18 2024-09-17T07:03:59.872 INFO:teuthology.orchestra.run.smithi100.stdout:full_ratio 0.95 2024-09-17T07:03:59.872 INFO:teuthology.orchestra.run.smithi100.stdout:backfillfull_ratio 0.9 2024-09-17T07:03:59.872 INFO:teuthology.orchestra.run.smithi100.stdout:nearfull_ratio 0.85 2024-09-17T07:03:59.872 INFO:teuthology.orchestra.run.smithi100.stdout:require_min_compat_client luminous 2024-09-17T07:03:59.872 INFO:teuthology.orchestra.run.smithi100.stdout:min_compat_client luminous 2024-09-17T07:03:59.872 INFO:teuthology.orchestra.run.smithi100.stdout:require_osd_release reef 2024-09-17T07:03:59.872 INFO:teuthology.orchestra.run.smithi100.stdout:stretch_mode_enabled false 2024-09-17T07:03:59.872 INFO:teuthology.orchestra.run.smithi100.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 7.89 2024-09-17T07:03:59.872 INFO:teuthology.orchestra.run.smithi100.stdout:pool 2 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 69 flags hashpspool stripe_width 0 application rgw read_balance_score 1.50 2024-09-17T07:03:59.872 INFO:teuthology.orchestra.run.smithi100.stdout:pool 3 'datapool' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 3 pgp_num 3 autoscale_mode off last_change 72 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 2.65 2024-09-17T07:03:59.872 INFO:teuthology.orchestra.run.smithi100.stdout:pool 4 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 71 flags hashpspool stripe_width 0 application rgw read_balance_score 2.25 2024-09-17T07:03:59.872 INFO:teuthology.orchestra.run.smithi100.stdout:pool 5 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 73 flags hashpspool stripe_width 0 application rgw read_balance_score 1.25 2024-09-17T07:03:59.873 INFO:teuthology.orchestra.run.smithi100.stdout:pool 6 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 75 flags hashpspool stripe_width 0 pg_autoscale_bias 4 application rgw read_balance_score 1.75 2024-09-17T07:03:59.873 INFO:teuthology.orchestra.run.smithi100.stdout:max_osd 8 2024-09-17T07:03:59.873 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0 up in weight 1 up_from 8 up_thru 92 down_at 0 last_clean_interval [0,0) v2:172.21.15.100:6801/3341017455 v2:172.21.15.100:6802/3341017455 exists,up 27bee9ec-05e6-406c-8bf1-168098da0ec7 2024-09-17T07:03:59.873 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1 up in weight 1 up_from 13 up_thru 73 down_at 0 last_clean_interval [0,0) v2:172.21.15.100:6805/1634750965 v2:172.21.15.100:6806/1634750965 exists,up b1a30c9c-eeaf-478e-9b62-2d63ad45873d 2024-09-17T07:03:59.873 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2 up in weight 1 up_from 18 up_thru 73 down_at 0 last_clean_interval [0,0) v2:172.21.15.100:6809/714579846 v2:172.21.15.100:6810/714579846 exists,up ce542e0d-9915-465e-aa11-7b3ce13f694f 2024-09-17T07:03:59.873 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3 up in weight 1 up_from 27 up_thru 92 down_at 0 last_clean_interval [0,0) v2:172.21.15.100:6813/2315956572 v2:172.21.15.100:6814/2315956572 exists,up a1257575-3c90-41de-88f6-3960744351b1 2024-09-17T07:03:59.873 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4 up in weight 1 up_from 33 up_thru 73 down_at 0 last_clean_interval [0,0) v2:172.21.15.135:6800/1148820922 v2:172.21.15.135:6801/1148820922 exists,up e44e8922-2c5e-44e3-b556-d4dfbc499649 2024-09-17T07:03:59.873 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5 up in weight 1 up_from 42 up_thru 73 down_at 0 last_clean_interval [0,0) v2:172.21.15.135:6804/2256520893 v2:172.21.15.135:6805/2256520893 exists,up 91e4a362-fcdb-41b9-b54e-716591ba6d79 2024-09-17T07:03:59.873 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6 up in weight 1 up_from 50 up_thru 92 down_at 0 last_clean_interval [0,0) v2:172.21.15.135:6808/2750184152 v2:172.21.15.135:6809/2750184152 exists,up de799f64-ebd2-4a46-91f6-c2f276bee21f 2024-09-17T07:03:59.873 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7 up in weight 1 up_from 60 up_thru 73 down_at 0 last_clean_interval [0,0) v2:172.21.15.135:6812/160304773 v2:172.21.15.135:6813/160304773 exists,up 830932c7-5de1-44ff-a840-e53a6e859bda 2024-09-17T07:03:59.873 INFO:teuthology.orchestra.run.smithi100.stdout:pg_upmap_items 2.1a [7,2] 2024-09-17T07:03:59.873 INFO:teuthology.orchestra.run.smithi100.stdout:pg_upmap_items 4.6 [1,5] 2024-09-17T07:03:59.873 INFO:teuthology.orchestra.run.smithi100.stdout:pg_upmap_items 4.10 [1,2] 2024-09-17T07:03:59.874 INFO:teuthology.orchestra.run.smithi100.stdout:pg_upmap_items 4.1f [1,2] 2024-09-17T07:03:59.874 INFO:teuthology.orchestra.run.smithi100.stdout:blocklist 172.21.15.100:0/3796664909 expires 2024-09-18T06:57:35.678309+0000 2024-09-17T07:03:59.874 INFO:teuthology.orchestra.run.smithi100.stdout:blocklist 172.21.15.100:0/2225586217 expires 2024-09-18T07:02:46.981486+0000 2024-09-17T07:03:59.874 INFO:teuthology.orchestra.run.smithi100.stdout:blocklist 172.21.15.100:0/1626706829 expires 2024-09-18T06:57:35.678309+0000 2024-09-17T07:03:59.874 INFO:teuthology.orchestra.run.smithi100.stdout:blocklist 172.21.15.100:0/3452172206 expires 2024-09-18T06:57:35.678309+0000 2024-09-17T07:03:59.874 INFO:teuthology.orchestra.run.smithi100.stdout:blocklist 172.21.15.100:6800/3992296558 expires 2024-09-18T06:57:58.526309+0000 2024-09-17T07:03:59.874 INFO:teuthology.orchestra.run.smithi100.stdout:blocklist 172.21.15.100:6800/3498574214 expires 2024-09-18T06:57:35.678309+0000 2024-09-17T07:03:59.874 INFO:teuthology.orchestra.run.smithi100.stdout:blocklist 172.21.15.100:0/1365215411 expires 2024-09-18T06:57:58.526309+0000 2024-09-17T07:03:59.874 INFO:teuthology.orchestra.run.smithi100.stdout:blocklist 172.21.15.100:0/3362169179 expires 2024-09-18T07:02:46.981486+0000 2024-09-17T07:03:59.874 INFO:teuthology.orchestra.run.smithi100.stdout:blocklist 172.21.15.100:0/2573779025 expires 2024-09-18T07:02:46.981486+0000 2024-09-17T07:03:59.874 INFO:teuthology.orchestra.run.smithi100.stdout:blocklist 172.21.15.100:0/380524491 expires 2024-09-18T07:02:46.981486+0000 2024-09-17T07:03:59.874 INFO:teuthology.orchestra.run.smithi100.stdout:blocklist 172.21.15.100:0/1657291976 expires 2024-09-18T06:57:58.526309+0000 2024-09-17T07:03:59.875 INFO:teuthology.orchestra.run.smithi100.stdout:blocklist 172.21.15.100:0/1676314112 expires 2024-09-18T07:02:46.981486+0000 2024-09-17T07:03:59.875 INFO:teuthology.orchestra.run.smithi100.stdout:blocklist 172.21.15.100:0/1255197255 expires 2024-09-18T06:57:58.526309+0000 2024-09-17T07:03:59.875 INFO:teuthology.orchestra.run.smithi100.stdout:blocklist 172.21.15.100:6800/3472658181 expires 2024-09-18T07:02:46.981486+0000 2024-09-17T07:04:00.126 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:Sep 17 07:03:59 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-alertmanager-a[73990]: ts=2024-09-17T07:03:59.831Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000130492s 2024-09-17T07:04:00.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:59 smithi100 ceph-mon[39462]: 6.8 scrub starts 2024-09-17T07:04:00.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:59 smithi100 ceph-mon[39462]: 6.8 scrub ok 2024-09-17T07:04:00.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:59 smithi100 ceph-mon[39462]: Reconfiguring daemon prometheus.a on smithi135 2024-09-17T07:04:00.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:59 smithi100 ceph-mon[39462]: 6.1d scrub starts 2024-09-17T07:04:00.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:59 smithi100 ceph-mon[39462]: 6.1d scrub ok 2024-09-17T07:04:00.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:59 smithi100 ceph-mon[39462]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 10 B/s, 1 objects/s recovering 2024-09-17T07:04:00.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:59 smithi100 ceph-mon[39462]: 5.1b scrub starts 2024-09-17T07:04:00.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:59 smithi100 ceph-mon[39462]: 5.1b scrub ok 2024-09-17T07:04:00.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:03:59 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/3517167443' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-09-17T07:04:00.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:59 smithi100 ceph-mon[29092]: 6.8 scrub starts 2024-09-17T07:04:00.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:59 smithi100 ceph-mon[29092]: 6.8 scrub ok 2024-09-17T07:04:00.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:59 smithi100 ceph-mon[29092]: Reconfiguring daemon prometheus.a on smithi135 2024-09-17T07:04:00.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:59 smithi100 ceph-mon[29092]: 6.1d scrub starts 2024-09-17T07:04:00.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:59 smithi100 ceph-mon[29092]: 6.1d scrub ok 2024-09-17T07:04:00.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:59 smithi100 ceph-mon[29092]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 10 B/s, 1 objects/s recovering 2024-09-17T07:04:00.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:59 smithi100 ceph-mon[29092]: 5.1b scrub starts 2024-09-17T07:04:00.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:59 smithi100 ceph-mon[29092]: 5.1b scrub ok 2024-09-17T07:04:00.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:03:59 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/3517167443' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-09-17T07:04:00.163 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:59 smithi135 ceph-mon[33238]: 6.8 scrub starts 2024-09-17T07:04:00.164 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:59 smithi135 ceph-mon[33238]: 6.8 scrub ok 2024-09-17T07:04:00.164 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:59 smithi135 ceph-mon[33238]: Reconfiguring daemon prometheus.a on smithi135 2024-09-17T07:04:00.164 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:59 smithi135 ceph-mon[33238]: 6.1d scrub starts 2024-09-17T07:04:00.164 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:59 smithi135 ceph-mon[33238]: 6.1d scrub ok 2024-09-17T07:04:00.164 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:59 smithi135 ceph-mon[33238]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 10 B/s, 1 objects/s recovering 2024-09-17T07:04:00.164 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:59 smithi135 ceph-mon[33238]: 5.1b scrub starts 2024-09-17T07:04:00.164 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:59 smithi135 ceph-mon[33238]: 5.1b scrub ok 2024-09-17T07:04:00.164 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:03:59 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/3517167443' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-09-17T07:04:00.569 INFO:tasks.cephadm.ceph_manager.ceph:['osd.0 up in weight 1 up_from 8 up_thru 92 down_at 0 last_clean_interval [0,0) v2:172.21.15.100:6801/3341017455 v2:172.21.15.100:6802/3341017455 exists,up 27bee9ec-05e6-406c-8bf1-168098da0ec7', 'osd.1 up in weight 1 up_from 13 up_thru 73 down_at 0 last_clean_interval [0,0) v2:172.21.15.100:6805/1634750965 v2:172.21.15.100:6806/1634750965 exists,up b1a30c9c-eeaf-478e-9b62-2d63ad45873d', 'osd.2 up in weight 1 up_from 18 up_thru 73 down_at 0 last_clean_interval [0,0) v2:172.21.15.100:6809/714579846 v2:172.21.15.100:6810/714579846 exists,up ce542e0d-9915-465e-aa11-7b3ce13f694f', 'osd.3 up in weight 1 up_from 27 up_thru 92 down_at 0 last_clean_interval [0,0) v2:172.21.15.100:6813/2315956572 v2:172.21.15.100:6814/2315956572 exists,up a1257575-3c90-41de-88f6-3960744351b1', 'osd.4 up in weight 1 up_from 33 up_thru 73 down_at 0 last_clean_interval [0,0) v2:172.21.15.135:6800/1148820922 v2:172.21.15.135:6801/1148820922 exists,up e44e8922-2c5e-44e3-b556-d4dfbc499649', 'osd.5 up in weight 1 up_from 42 up_thru 73 down_at 0 last_clean_interval [0,0) v2:172.21.15.135:6804/2256520893 v2:172.21.15.135:6805/2256520893 exists,up 91e4a362-fcdb-41b9-b54e-716591ba6d79', 'osd.6 up in weight 1 up_from 50 up_thru 92 down_at 0 last_clean_interval [0,0) v2:172.21.15.135:6808/2750184152 v2:172.21.15.135:6809/2750184152 exists,up de799f64-ebd2-4a46-91f6-c2f276bee21f', 'osd.7 up in weight 1 up_from 60 up_thru 73 down_at 0 last_clean_interval [0,0) v2:172.21.15.135:6812/160304773 v2:172.21.15.135:6813/160304773 exists,up 830932c7-5de1-44ff-a840-e53a6e859bda'] 2024-09-17T07:04:00.570 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon mon.a config show 2024-09-17T07:04:00.768 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:04:01.235 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:04:00 smithi135 systemd[1]: Stopping Ceph prometheus.a for e848267c-74c1-11ef-bceb-c7b262605968... 2024-09-17T07:04:01.519 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:04:01 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a[61712]: ts=2024-09-17T07:04:01.234Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-17T07:04:01.519 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:04:01 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a[61712]: ts=2024-09-17T07:04:01.234Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-09-17T07:04:01.519 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:04:01 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a[61712]: ts=2024-09-17T07:04:01.234Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-09-17T07:04:01.519 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:04:01 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a[61712]: ts=2024-09-17T07:04:01.234Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-17T07:04:01.519 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:04:01 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a[61712]: ts=2024-09-17T07:04:01.234Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-09-17T07:04:01.519 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:04:01 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a[61712]: ts=2024-09-17T07:04:01.234Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-09-17T07:04:01.520 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:04:01 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a[61712]: ts=2024-09-17T07:04:01.235Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-09-17T07:04:01.520 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:04:01 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a[61712]: ts=2024-09-17T07:04:01.235Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-09-17T07:04:01.520 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:04:01 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a[61712]: ts=2024-09-17T07:04:01.235Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-09-17T07:04:01.520 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:04:01 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a[61712]: ts=2024-09-17T07:04:01.235Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-09-17T07:04:01.520 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:04:01 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a[61712]: ts=2024-09-17T07:04:01.235Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-09-17T07:04:01.520 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:04:01 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a[61712]: ts=2024-09-17T07:04:01.235Z caller=main.go:1147 level=info msg="See you next time!" 2024-09-17T07:04:01.520 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:04:01 smithi135 podman[64720]: 2024-09-17 07:04:01.245237131 +0000 UTC m=+0.109091386 container died e80f3f0bbd6a02365e092b9574c7e956f16cd804d0bd5d1470521e4ee6e234aa (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-17T07:04:01.793 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:04:01 smithi135 podman[64720]: 2024-09-17 07:04:01.617699346 +0000 UTC m=+0.481553607 container cleanup e80f3f0bbd6a02365e092b9574c7e956f16cd804d0bd5d1470521e4ee6e234aa (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-17T07:04:01.793 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:04:01 smithi135 bash[64720]: ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a 2024-09-17T07:04:02.253 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:02 smithi135 ceph-mon[33238]: pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 10 B/s, 1 objects/s recovering 2024-09-17T07:04:02.253 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:02 smithi135 ceph-mon[33238]: 5.1 scrub starts 2024-09-17T07:04:02.253 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:02 smithi135 ceph-mon[33238]: 5.1 scrub ok 2024-09-17T07:04:02.253 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:04:01 smithi135 podman[64733]: 2024-09-17 07:04:01.89429181 +0000 UTC m=+0.650941918 container remove e80f3f0bbd6a02365e092b9574c7e956f16cd804d0bd5d1470521e4ee6e234aa (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-17T07:04:02.253 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:04:02 smithi135 systemd[1]: ceph-e848267c-74c1-11ef-bceb-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-09-17T07:04:02.253 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:04:02 smithi135 systemd[1]: Stopped Ceph prometheus.a for e848267c-74c1-11ef-bceb-c7b262605968. 2024-09-17T07:04:02.253 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:04:02 smithi135 systemd[1]: ceph-e848267c-74c1-11ef-bceb-c7b262605968@prometheus.a.service: Consumed 1.090s CPU time. 2024-09-17T07:04:02.253 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:04:02 smithi135 systemd[1]: Starting Ceph prometheus.a for e848267c-74c1-11ef-bceb-c7b262605968... 2024-09-17T07:04:02.293 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:02 smithi100 ceph-mon[29092]: pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 10 B/s, 1 objects/s recovering 2024-09-17T07:04:02.293 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:02 smithi100 ceph-mon[29092]: 5.1 scrub starts 2024-09-17T07:04:02.293 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:02 smithi100 ceph-mon[29092]: 5.1 scrub ok 2024-09-17T07:04:02.293 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:02 smithi100 ceph-mon[39462]: pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 10 B/s, 1 objects/s recovering 2024-09-17T07:04:02.294 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:02 smithi100 ceph-mon[39462]: 5.1 scrub starts 2024-09-17T07:04:02.294 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:02 smithi100 ceph-mon[39462]: 5.1 scrub ok 2024-09-17T07:04:02.504 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-17T07:04:02.504 INFO:teuthology.orchestra.run.smithi100.stdout: "name": "mon.a", 2024-09-17T07:04:02.504 INFO:teuthology.orchestra.run.smithi100.stdout: "cluster": "ceph", 2024-09-17T07:04:02.504 INFO:teuthology.orchestra.run.smithi100.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-09-17T07:04:02.504 INFO:teuthology.orchestra.run.smithi100.stdout: "admin_socket_mode": "", 2024-09-17T07:04:02.504 INFO:teuthology.orchestra.run.smithi100.stdout: "auth_allow_insecure_global_id_reclaim": "false", 2024-09-17T07:04:02.505 INFO:teuthology.orchestra.run.smithi100.stdout: "auth_client_required": "cephx, none", 2024-09-17T07:04:02.505 INFO:teuthology.orchestra.run.smithi100.stdout: "auth_cluster_required": "cephx", 2024-09-17T07:04:02.505 INFO:teuthology.orchestra.run.smithi100.stdout: "auth_debug": "true", 2024-09-17T07:04:02.505 INFO:teuthology.orchestra.run.smithi100.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-09-17T07:04:02.505 INFO:teuthology.orchestra.run.smithi100.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-09-17T07:04:02.505 INFO:teuthology.orchestra.run.smithi100.stdout: "auth_service_required": "cephx", 2024-09-17T07:04:02.505 INFO:teuthology.orchestra.run.smithi100.stdout: "auth_service_ticket_ttl": "240.000000", 2024-09-17T07:04:02.505 INFO:teuthology.orchestra.run.smithi100.stdout: "auth_supported": "", 2024-09-17T07:04:02.505 INFO:teuthology.orchestra.run.smithi100.stdout: "bdev_aio": "true", 2024-09-17T07:04:02.505 INFO:teuthology.orchestra.run.smithi100.stdout: "bdev_aio_max_queue_depth": "1024", 2024-09-17T07:04:02.505 INFO:teuthology.orchestra.run.smithi100.stdout: "bdev_aio_poll_ms": "250", 2024-09-17T07:04:02.505 INFO:teuthology.orchestra.run.smithi100.stdout: "bdev_aio_reap_max": "16", 2024-09-17T07:04:02.506 INFO:teuthology.orchestra.run.smithi100.stdout: "bdev_async_discard": "false", 2024-09-17T07:04:02.506 INFO:teuthology.orchestra.run.smithi100.stdout: "bdev_block_size": "4096", 2024-09-17T07:04:02.506 INFO:teuthology.orchestra.run.smithi100.stdout: "bdev_debug_aio": "false", 2024-09-17T07:04:02.506 INFO:teuthology.orchestra.run.smithi100.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-09-17T07:04:02.506 INFO:teuthology.orchestra.run.smithi100.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-09-17T07:04:02.506 INFO:teuthology.orchestra.run.smithi100.stdout: "bdev_debug_inflight_ios": "false", 2024-09-17T07:04:02.506 INFO:teuthology.orchestra.run.smithi100.stdout: "bdev_enable_discard": "false", 2024-09-17T07:04:02.506 INFO:teuthology.orchestra.run.smithi100.stdout: "bdev_flock_retry": "3", 2024-09-17T07:04:02.506 INFO:teuthology.orchestra.run.smithi100.stdout: "bdev_flock_retry_interval": "0.100000", 2024-09-17T07:04:02.506 INFO:teuthology.orchestra.run.smithi100.stdout: "bdev_inject_crash": "0", 2024-09-17T07:04:02.506 INFO:teuthology.orchestra.run.smithi100.stdout: "bdev_inject_crash_flush_delay": "2", 2024-09-17T07:04:02.506 INFO:teuthology.orchestra.run.smithi100.stdout: "bdev_ioring": "false", 2024-09-17T07:04:02.506 INFO:teuthology.orchestra.run.smithi100.stdout: "bdev_ioring_hipri": "false", 2024-09-17T07:04:02.507 INFO:teuthology.orchestra.run.smithi100.stdout: "bdev_ioring_sqthread_poll": "false", 2024-09-17T07:04:02.507 INFO:teuthology.orchestra.run.smithi100.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-09-17T07:04:02.507 INFO:teuthology.orchestra.run.smithi100.stdout: "bdev_read_buffer_alignment": "4096", 2024-09-17T07:04:02.507 INFO:teuthology.orchestra.run.smithi100.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-09-17T07:04:02.507 INFO:teuthology.orchestra.run.smithi100.stdout: "bdev_type": "", 2024-09-17T07:04:02.507 INFO:teuthology.orchestra.run.smithi100.stdout: "bluefs_alloc_size": "1048576", 2024-09-17T07:04:02.507 INFO:teuthology.orchestra.run.smithi100.stdout: "bluefs_allocator": "hybrid", 2024-09-17T07:04:02.507 INFO:teuthology.orchestra.run.smithi100.stdout: "bluefs_buffered_io": "true", 2024-09-17T07:04:02.507 INFO:teuthology.orchestra.run.smithi100.stdout: "bluefs_check_for_zeros": "false", 2024-09-17T07:04:02.507 INFO:teuthology.orchestra.run.smithi100.stdout: "bluefs_check_volume_selector_often": "false", 2024-09-17T07:04:02.507 INFO:teuthology.orchestra.run.smithi100.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-09-17T07:04:02.507 INFO:teuthology.orchestra.run.smithi100.stdout: "bluefs_compact_log_sync": "false", 2024-09-17T07:04:02.508 INFO:teuthology.orchestra.run.smithi100.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-09-17T07:04:02.508 INFO:teuthology.orchestra.run.smithi100.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-09-17T07:04:02.508 INFO:teuthology.orchestra.run.smithi100.stdout: "bluefs_log_compact_min_size": "16777216", 2024-09-17T07:04:02.508 INFO:teuthology.orchestra.run.smithi100.stdout: "bluefs_log_replay_check_allocations": "true", 2024-09-17T07:04:02.508 INFO:teuthology.orchestra.run.smithi100.stdout: "bluefs_max_log_runway": "4194304", 2024-09-17T07:04:02.508 INFO:teuthology.orchestra.run.smithi100.stdout: "bluefs_max_prefetch": "1048576", 2024-09-17T07:04:02.508 INFO:teuthology.orchestra.run.smithi100.stdout: "bluefs_min_flush_size": "524288", 2024-09-17T07:04:02.508 INFO:teuthology.orchestra.run.smithi100.stdout: "bluefs_min_log_runway": "1048576", 2024-09-17T07:04:02.508 INFO:teuthology.orchestra.run.smithi100.stdout: "bluefs_replay_recovery": "false", 2024-09-17T07:04:02.508 INFO:teuthology.orchestra.run.smithi100.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-09-17T07:04:02.508 INFO:teuthology.orchestra.run.smithi100.stdout: "bluefs_shared_alloc_size": "65536", 2024-09-17T07:04:02.509 INFO:teuthology.orchestra.run.smithi100.stdout: "bluefs_sync_write": "false", 2024-09-17T07:04:02.509 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-09-17T07:04:02.509 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-09-17T07:04:02.509 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-09-17T07:04:02.509 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_allocation_from_file": "true", 2024-09-17T07:04:02.509 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_allocator": "hybrid", 2024-09-17T07:04:02.509 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-09-17T07:04:02.509 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-09-17T07:04:02.509 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-09-17T07:04:02.509 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-09-17T07:04:02.509 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-09-17T07:04:02.509 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-09-17T07:04:02.509 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_blobid_prealloc": "10240", 2024-09-17T07:04:02.510 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_block_create": "true", 2024-09-17T07:04:02.510 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_block_db_create": "false", 2024-09-17T07:04:02.510 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_block_db_path": "", 2024-09-17T07:04:02.510 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_block_db_size": "0", 2024-09-17T07:04:02.510 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_block_path": "", 2024-09-17T07:04:02.510 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_block_preallocate_file": "false", 2024-09-17T07:04:02.510 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_block_size": "107374182400", 2024-09-17T07:04:02.510 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_block_wal_create": "false", 2024-09-17T07:04:02.510 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_block_wal_path": "", 2024-09-17T07:04:02.510 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_block_wal_size": "100663296", 2024-09-17T07:04:02.510 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_bluefs": "true", 2024-09-17T07:04:02.510 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-09-17T07:04:02.510 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_bluefs_env_mirror": "false", 2024-09-17T07:04:02.511 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-09-17T07:04:02.511 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-09-17T07:04:02.511 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-09-17T07:04:02.511 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-09-17T07:04:02.511 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-09-17T07:04:02.511 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-09-17T07:04:02.511 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_cache_autotune": "true", 2024-09-17T07:04:02.511 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-09-17T07:04:02.511 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-09-17T07:04:02.511 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-09-17T07:04:02.511 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-09-17T07:04:02.511 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_cache_size": "0", 2024-09-17T07:04:02.512 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-09-17T07:04:02.512 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-09-17T07:04:02.512 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-09-17T07:04:02.512 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-09-17T07:04:02.512 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_cache_type": "2q", 2024-09-17T07:04:02.512 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-09-17T07:04:02.512 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_clone_cow": "true", 2024-09-17T07:04:02.512 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_compression_algorithm": "snappy", 2024-09-17T07:04:02.512 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_compression_max_blob_size": "0", 2024-09-17T07:04:02.512 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-09-17T07:04:02.512 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-09-17T07:04:02.512 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_compression_min_blob_size": "0", 2024-09-17T07:04:02.512 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-09-17T07:04:02.513 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-09-17T07:04:02.513 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_compression_mode": "none", 2024-09-17T07:04:02.513 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-09-17T07:04:02.513 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_csum_type": "crc32c", 2024-09-17T07:04:02.513 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_debug_enforce_settings": "default", 2024-09-17T07:04:02.513 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_debug_freelist": "false", 2024-09-17T07:04:02.513 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_debug_fsck_abort": "false", 2024-09-17T07:04:02.513 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-09-17T07:04:02.513 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_debug_inject_bug21040": "false", 2024-09-17T07:04:02.514 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-09-17T07:04:02.514 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_debug_inject_read_err": "false", 2024-09-17T07:04:02.514 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_debug_legacy_omap": "false", 2024-09-17T07:04:02.514 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-09-17T07:04:02.514 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-09-17T07:04:02.514 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-09-17T07:04:02.514 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-09-17T07:04:02.514 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_debug_prefill": "0.000000", 2024-09-17T07:04:02.514 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-09-17T07:04:02.514 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-09-17T07:04:02.514 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-09-17T07:04:02.514 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_debug_small_allocations": "0", 2024-09-17T07:04:02.514 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-09-17T07:04:02.515 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_default_buffered_read": "true", 2024-09-17T07:04:02.515 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_default_buffered_write": "false", 2024-09-17T07:04:02.515 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_deferred_batch_ops": "0", 2024-09-17T07:04:02.515 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-09-17T07:04:02.515 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-09-17T07:04:02.515 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-09-17T07:04:02.515 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-09-17T07:04:02.515 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-09-17T07:04:02.515 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-09-17T07:04:02.515 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-09-17T07:04:02.515 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_fail_eio": "false", 2024-09-17T07:04:02.515 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-09-17T07:04:02.515 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-09-17T07:04:02.515 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-09-17T07:04:02.516 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-09-17T07:04:02.516 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_fsck_on_mkfs": "true", 2024-09-17T07:04:02.516 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-09-17T07:04:02.516 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_fsck_on_mount": "false", 2024-09-17T07:04:02.516 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-09-17T07:04:02.516 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_fsck_on_umount": "false", 2024-09-17T07:04:02.516 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-09-17T07:04:02.516 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-09-17T07:04:02.516 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-09-17T07:04:02.516 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-09-17T07:04:02.516 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-09-17T07:04:02.516 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-09-17T07:04:02.516 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-09-17T07:04:02.517 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-09-17T07:04:02.517 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_ignore_data_csum": "false", 2024-09-17T07:04:02.517 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-09-17T07:04:02.517 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_kvbackend": "rocksdb", 2024-09-17T07:04:02.517 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-09-17T07:04:02.517 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-09-17T07:04:02.517 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_log_op_age": "5.000000", 2024-09-17T07:04:02.517 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_max_alloc_size": "0", 2024-09-17T07:04:02.517 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_max_blob_size": "0", 2024-09-17T07:04:02.517 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-09-17T07:04:02.517 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-09-17T07:04:02.517 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_max_defer_interval": "3.000000", 2024-09-17T07:04:02.518 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_max_deferred_txc": "32", 2024-09-17T07:04:02.518 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_min_alloc_size": "0", 2024-09-17T07:04:02.518 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-09-17T07:04:02.518 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-09-17T07:04:02.518 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_nid_prealloc": "1024", 2024-09-17T07:04:02.518 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_prefer_deferred_size": "0", 2024-09-17T07:04:02.518 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-09-17T07:04:02.518 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-09-17T07:04:02.518 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_qfsck_on_mount": "true", 2024-09-17T07:04:02.518 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_retry_disk_reads": "3", 2024-09-17T07:04:02.518 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_rocksdb_cf": "true", 2024-09-17T07:04:02.518 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-09-17T07:04:02.518 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-09-17T07:04:02.519 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_rocksdb_options_annex": "", 2024-09-17T07:04:02.519 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_spdk_coremask": "0x1", 2024-09-17T07:04:02.519 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_spdk_io_sleep": "5", 2024-09-17T07:04:02.519 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_spdk_max_io_completion": "0", 2024-09-17T07:04:02.519 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_spdk_mem": "512", 2024-09-17T07:04:02.519 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_sync_submit_transaction": "false", 2024-09-17T07:04:02.519 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_throttle_bytes": "67108864", 2024-09-17T07:04:02.519 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_throttle_cost_per_io": "0", 2024-09-17T07:04:02.519 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-09-17T07:04:02.519 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-09-17T07:04:02.519 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-09-17T07:04:02.519 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-09-17T07:04:02.520 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_tracing": "false", 2024-09-17T07:04:02.520 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-09-17T07:04:02.520 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-09-17T07:04:02.520 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_volume_selection_reserved": "0", 2024-09-17T07:04:02.520 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-09-17T07:04:02.520 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-09-17T07:04:02.520 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-09-17T07:04:02.520 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-09-17T07:04:02.520 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-09-17T07:04:02.520 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-09-17T07:04:02.520 INFO:teuthology.orchestra.run.smithi100.stdout: "bluestore_zero_block_detection": "false", 2024-09-17T07:04:02.520 INFO:teuthology.orchestra.run.smithi100.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-09-17T07:04:02.520 INFO:teuthology.orchestra.run.smithi100.stdout: "cephfs_mirror_action_update_interval": "2", 2024-09-17T07:04:02.521 INFO:teuthology.orchestra.run.smithi100.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-09-17T07:04:02.521 INFO:teuthology.orchestra.run.smithi100.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-09-17T07:04:02.521 INFO:teuthology.orchestra.run.smithi100.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-09-17T07:04:02.521 INFO:teuthology.orchestra.run.smithi100.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-09-17T07:04:02.521 INFO:teuthology.orchestra.run.smithi100.stdout: "cephfs_mirror_mount_timeout": "10", 2024-09-17T07:04:02.521 INFO:teuthology.orchestra.run.smithi100.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-09-17T07:04:02.521 INFO:teuthology.orchestra.run.smithi100.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-09-17T07:04:02.521 INFO:teuthology.orchestra.run.smithi100.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-09-17T07:04:02.521 INFO:teuthology.orchestra.run.smithi100.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-09-17T07:04:02.521 INFO:teuthology.orchestra.run.smithi100.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-09-17T07:04:02.521 INFO:teuthology.orchestra.run.smithi100.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-09-17T07:04:02.521 INFO:teuthology.orchestra.run.smithi100.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-09-17T07:04:02.522 INFO:teuthology.orchestra.run.smithi100.stdout: "cephx_cluster_require_signatures": "false", 2024-09-17T07:04:02.522 INFO:teuthology.orchestra.run.smithi100.stdout: "cephx_cluster_require_version": "2", 2024-09-17T07:04:02.522 INFO:teuthology.orchestra.run.smithi100.stdout: "cephx_require_signatures": "false", 2024-09-17T07:04:02.522 INFO:teuthology.orchestra.run.smithi100.stdout: "cephx_require_version": "2", 2024-09-17T07:04:02.522 INFO:teuthology.orchestra.run.smithi100.stdout: "cephx_service_require_signatures": "false", 2024-09-17T07:04:02.522 INFO:teuthology.orchestra.run.smithi100.stdout: "cephx_service_require_version": "2", 2024-09-17T07:04:02.522 INFO:teuthology.orchestra.run.smithi100.stdout: "cephx_sign_messages": "true", 2024-09-17T07:04:02.522 INFO:teuthology.orchestra.run.smithi100.stdout: "chdir": "", 2024-09-17T07:04:02.522 INFO:teuthology.orchestra.run.smithi100.stdout: "client_acl_type": "", 2024-09-17T07:04:02.522 INFO:teuthology.orchestra.run.smithi100.stdout: "client_asio_thread_count": "2", 2024-09-17T07:04:02.522 INFO:teuthology.orchestra.run.smithi100.stdout: "client_cache_mid": "0.750000", 2024-09-17T07:04:02.522 INFO:teuthology.orchestra.run.smithi100.stdout: "client_cache_size": "16384", 2024-09-17T07:04:02.522 INFO:teuthology.orchestra.run.smithi100.stdout: "client_caps_release_delay": "5", 2024-09-17T07:04:02.522 INFO:teuthology.orchestra.run.smithi100.stdout: "client_check_pool_perm": "true", 2024-09-17T07:04:02.522 INFO:teuthology.orchestra.run.smithi100.stdout: "client_collect_and_send_global_metrics": "false", 2024-09-17T07:04:02.523 INFO:teuthology.orchestra.run.smithi100.stdout: "client_debug_force_sync_read": "false", 2024-09-17T07:04:02.523 INFO:teuthology.orchestra.run.smithi100.stdout: "client_debug_getattr_caps": "false", 2024-09-17T07:04:02.523 INFO:teuthology.orchestra.run.smithi100.stdout: "client_debug_inject_features": "", 2024-09-17T07:04:02.523 INFO:teuthology.orchestra.run.smithi100.stdout: "client_debug_inject_tick_delay": "0", 2024-09-17T07:04:02.523 INFO:teuthology.orchestra.run.smithi100.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-09-17T07:04:02.523 INFO:teuthology.orchestra.run.smithi100.stdout: "client_die_on_failed_remount": "false", 2024-09-17T07:04:02.523 INFO:teuthology.orchestra.run.smithi100.stdout: "client_dirsize_rbytes": "true", 2024-09-17T07:04:02.523 INFO:teuthology.orchestra.run.smithi100.stdout: "client_force_lazyio": "false", 2024-09-17T07:04:02.523 INFO:teuthology.orchestra.run.smithi100.stdout: "client_fs": "", 2024-09-17T07:04:02.523 INFO:teuthology.orchestra.run.smithi100.stdout: "client_inject_fixed_oldest_tid": "false", 2024-09-17T07:04:02.523 INFO:teuthology.orchestra.run.smithi100.stdout: "client_inject_release_failure": "false", 2024-09-17T07:04:02.523 INFO:teuthology.orchestra.run.smithi100.stdout: "client_max_inline_size": "4096", 2024-09-17T07:04:02.523 INFO:teuthology.orchestra.run.smithi100.stdout: "client_max_retries_on_remount_failure": "5", 2024-09-17T07:04:02.524 INFO:teuthology.orchestra.run.smithi100.stdout: "client_mds_namespace": "", 2024-09-17T07:04:02.524 INFO:teuthology.orchestra.run.smithi100.stdout: "client_metadata": "", 2024-09-17T07:04:02.524 INFO:teuthology.orchestra.run.smithi100.stdout: "client_mount_gid": "-1", 2024-09-17T07:04:02.524 INFO:teuthology.orchestra.run.smithi100.stdout: "client_mount_timeout": "300", 2024-09-17T07:04:02.524 INFO:teuthology.orchestra.run.smithi100.stdout: "client_mount_uid": "-1", 2024-09-17T07:04:02.524 INFO:teuthology.orchestra.run.smithi100.stdout: "client_mountpoint": "/", 2024-09-17T07:04:02.524 INFO:teuthology.orchestra.run.smithi100.stdout: "client_notify_timeout": "10", 2024-09-17T07:04:02.524 INFO:teuthology.orchestra.run.smithi100.stdout: "client_oc": "true", 2024-09-17T07:04:02.524 INFO:teuthology.orchestra.run.smithi100.stdout: "client_oc_max_dirty": "104857600", 2024-09-17T07:04:02.524 INFO:teuthology.orchestra.run.smithi100.stdout: "client_oc_max_dirty_age": "5.000000", 2024-09-17T07:04:02.524 INFO:teuthology.orchestra.run.smithi100.stdout: "client_oc_max_objects": "1000", 2024-09-17T07:04:02.524 INFO:teuthology.orchestra.run.smithi100.stdout: "client_oc_size": "209715200", 2024-09-17T07:04:02.525 INFO:teuthology.orchestra.run.smithi100.stdout: "client_oc_target_dirty": "8388608", 2024-09-17T07:04:02.525 INFO:teuthology.orchestra.run.smithi100.stdout: "client_permissions": "true", 2024-09-17T07:04:02.525 INFO:teuthology.orchestra.run.smithi100.stdout: "client_quota": "true", 2024-09-17T07:04:02.525 INFO:teuthology.orchestra.run.smithi100.stdout: "client_quota_df": "true", 2024-09-17T07:04:02.525 INFO:teuthology.orchestra.run.smithi100.stdout: "client_readahead_max_bytes": "0", 2024-09-17T07:04:02.525 INFO:teuthology.orchestra.run.smithi100.stdout: "client_readahead_max_periods": "4", 2024-09-17T07:04:02.525 INFO:teuthology.orchestra.run.smithi100.stdout: "client_readahead_min": "131072", 2024-09-17T07:04:02.525 INFO:teuthology.orchestra.run.smithi100.stdout: "client_reconnect_stale": "false", 2024-09-17T07:04:02.525 INFO:teuthology.orchestra.run.smithi100.stdout: "client_shutdown_timeout": "30", 2024-09-17T07:04:02.525 INFO:teuthology.orchestra.run.smithi100.stdout: "client_snapdir": ".snap", 2024-09-17T07:04:02.525 INFO:teuthology.orchestra.run.smithi100.stdout: "client_tick_interval": "1", 2024-09-17T07:04:02.525 INFO:teuthology.orchestra.run.smithi100.stdout: "client_trace": "", 2024-09-17T07:04:02.525 INFO:teuthology.orchestra.run.smithi100.stdout: "client_try_dentry_invalidate": "false", 2024-09-17T07:04:02.526 INFO:teuthology.orchestra.run.smithi100.stdout: "client_use_faked_inos": "false", 2024-09-17T07:04:02.526 INFO:teuthology.orchestra.run.smithi100.stdout: "client_use_random_mds": "false", 2024-09-17T07:04:02.526 INFO:teuthology.orchestra.run.smithi100.stdout: "clog_to_graylog": "false", 2024-09-17T07:04:02.526 INFO:teuthology.orchestra.run.smithi100.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-09-17T07:04:02.526 INFO:teuthology.orchestra.run.smithi100.stdout: "clog_to_graylog_port": "12201", 2024-09-17T07:04:02.526 INFO:teuthology.orchestra.run.smithi100.stdout: "clog_to_monitors": "default=true", 2024-09-17T07:04:02.526 INFO:teuthology.orchestra.run.smithi100.stdout: "clog_to_syslog": "false", 2024-09-17T07:04:02.526 INFO:teuthology.orchestra.run.smithi100.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-09-17T07:04:02.526 INFO:teuthology.orchestra.run.smithi100.stdout: "clog_to_syslog_level": "info", 2024-09-17T07:04:02.526 INFO:teuthology.orchestra.run.smithi100.stdout: "cluster_addr": "-", 2024-09-17T07:04:02.526 INFO:teuthology.orchestra.run.smithi100.stdout: "cluster_network": "", 2024-09-17T07:04:02.526 INFO:teuthology.orchestra.run.smithi100.stdout: "cluster_network_interface": "", 2024-09-17T07:04:02.527 INFO:teuthology.orchestra.run.smithi100.stdout: "compressor_zlib_isal": "false", 2024-09-17T07:04:02.527 INFO:teuthology.orchestra.run.smithi100.stdout: "compressor_zlib_level": "5", 2024-09-17T07:04:02.527 INFO:teuthology.orchestra.run.smithi100.stdout: "compressor_zlib_winsize": "-15", 2024-09-17T07:04:02.527 INFO:teuthology.orchestra.run.smithi100.stdout: "compressor_zstd_level": "1", 2024-09-17T07:04:02.527 INFO:teuthology.orchestra.run.smithi100.stdout: "container_image": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9", 2024-09-17T07:04:02.527 INFO:teuthology.orchestra.run.smithi100.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-09-17T07:04:02.527 INFO:teuthology.orchestra.run.smithi100.stdout: "crimson_alien_op_num_threads": "6", 2024-09-17T07:04:02.527 INFO:teuthology.orchestra.run.smithi100.stdout: "crimson_alien_thread_cpu_cores": "", 2024-09-17T07:04:02.527 INFO:teuthology.orchestra.run.smithi100.stdout: "crimson_osd_obc_lru_size": "10", 2024-09-17T07:04:02.527 INFO:teuthology.orchestra.run.smithi100.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-09-17T07:04:02.527 INFO:teuthology.orchestra.run.smithi100.stdout: "crimson_seastar_smp": "1", 2024-09-17T07:04:02.527 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_location": "", 2024-09-17T07:04:02.527 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_location_hook": "", 2024-09-17T07:04:02.528 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_location_hook_timeout": "10", 2024-09-17T07:04:02.528 INFO:teuthology.orchestra.run.smithi100.stdout: "daemonize": "false", 2024-09-17T07:04:02.528 INFO:teuthology.orchestra.run.smithi100.stdout: "daos_pool": "tank", 2024-09-17T07:04:02.528 INFO:teuthology.orchestra.run.smithi100.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-09-17T07:04:02.528 INFO:teuthology.orchestra.run.smithi100.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-09-17T07:04:02.528 INFO:teuthology.orchestra.run.smithi100.stdout: "dbstore_db_name_prefix": "dbstore", 2024-09-17T07:04:02.528 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_alienstore": "0/5", 2024-09-17T07:04:02.528 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_allow_any_pool_priority": "false", 2024-09-17T07:04:02.528 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_asok": "1/5", 2024-09-17T07:04:02.528 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_asok_assert_abort": "false", 2024-09-17T07:04:02.528 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_asserts_on_shutdown": "true", 2024-09-17T07:04:02.528 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_auth": "1/5", 2024-09-17T07:04:02.528 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_bdev": "1/3", 2024-09-17T07:04:02.528 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_bluefs": "1/5", 2024-09-17T07:04:02.529 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_bluestore": "1/5", 2024-09-17T07:04:02.529 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_buffer": "0/1", 2024-09-17T07:04:02.529 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_ceph_exporter": "1/5", 2024-09-17T07:04:02.529 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_cephfs_mirror": "0/5", 2024-09-17T07:04:02.529 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_cephsqlite": "0/5", 2024-09-17T07:04:02.529 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_client": "0/5", 2024-09-17T07:04:02.529 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_compressor": "1/5", 2024-09-17T07:04:02.529 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_context": "0/1", 2024-09-17T07:04:02.529 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_crush": "1/1", 2024-09-17T07:04:02.529 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_crypto": "1/5", 2024-09-17T07:04:02.529 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_cyanstore": "0/5", 2024-09-17T07:04:02.529 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_deliberately_leak_memory": "false", 2024-09-17T07:04:02.529 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_disable_randomized_ping": "false", 2024-09-17T07:04:02.530 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_dpdk": "1/5", 2024-09-17T07:04:02.530 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_eventtrace": "1/5", 2024-09-17T07:04:02.530 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_filer": "0/1", 2024-09-17T07:04:02.530 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_filestore": "1/3", 2024-09-17T07:04:02.530 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_finisher": "1/1", 2024-09-17T07:04:02.530 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_fuse": "1/5", 2024-09-17T07:04:02.530 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_heartbeat_testing_span": "0", 2024-09-17T07:04:02.530 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_heartbeatmap": "1/5", 2024-09-17T07:04:02.530 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_immutable_obj_cache": "0/5", 2024-09-17T07:04:02.530 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_javaclient": "1/5", 2024-09-17T07:04:02.530 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_journal": "1/3", 2024-09-17T07:04:02.530 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_journaler": "0/5", 2024-09-17T07:04:02.530 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_kstore": "1/5", 2024-09-17T07:04:02.530 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_leveldb": "4/5", 2024-09-17T07:04:02.531 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_lockdep": "0/1", 2024-09-17T07:04:02.531 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_mclock": "1/5", 2024-09-17T07:04:02.531 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_mds": "1/5", 2024-09-17T07:04:02.531 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_mds_balancer": "1/5", 2024-09-17T07:04:02.531 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_mds_locker": "1/5", 2024-09-17T07:04:02.531 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_mds_log": "1/5", 2024-09-17T07:04:02.531 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_mds_log_expire": "1/5", 2024-09-17T07:04:02.531 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_mds_migrator": "1/5", 2024-09-17T07:04:02.531 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_memstore": "1/5", 2024-09-17T07:04:02.531 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_mgr": "2/5", 2024-09-17T07:04:02.531 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_mgrc": "1/5", 2024-09-17T07:04:02.531 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_mon": "20/20", 2024-09-17T07:04:02.531 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_monc": "0/10", 2024-09-17T07:04:02.531 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_ms": "1/1", 2024-09-17T07:04:02.532 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_none": "0/5", 2024-09-17T07:04:02.532 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_objclass": "0/5", 2024-09-17T07:04:02.532 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_objectcacher": "0/5", 2024-09-17T07:04:02.532 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_objecter": "0/1", 2024-09-17T07:04:02.532 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_optracker": "0/5", 2024-09-17T07:04:02.532 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_osd": "1/5", 2024-09-17T07:04:02.532 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_paxos": "20/20", 2024-09-17T07:04:02.532 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_perfcounter": "1/5", 2024-09-17T07:04:02.532 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_prioritycache": "1/5", 2024-09-17T07:04:02.532 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_rados": "0/5", 2024-09-17T07:04:02.532 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_rbd": "0/5", 2024-09-17T07:04:02.532 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_rbd_mirror": "0/5", 2024-09-17T07:04:02.532 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_rbd_pwl": "0/5", 2024-09-17T07:04:02.532 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_rbd_replay": "0/5", 2024-09-17T07:04:02.533 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_refs": "0/0", 2024-09-17T07:04:02.533 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_reserver": "1/1", 2024-09-17T07:04:02.533 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_rgw": "1/5", 2024-09-17T07:04:02.533 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_rgw_access": "1/5", 2024-09-17T07:04:02.533 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_rgw_datacache": "1/5", 2024-09-17T07:04:02.533 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_rgw_dbstore": "1/5", 2024-09-17T07:04:02.533 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_rgw_flight": "1/5", 2024-09-17T07:04:02.533 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_rgw_sync": "1/5", 2024-09-17T07:04:02.533 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_rocksdb": "4/5", 2024-09-17T07:04:02.533 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_seastore": "0/5", 2024-09-17T07:04:02.533 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_seastore_backref": "0/5", 2024-09-17T07:04:02.533 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_seastore_cache": "0/5", 2024-09-17T07:04:02.533 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_seastore_cleaner": "0/5", 2024-09-17T07:04:02.533 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_seastore_device": "0/5", 2024-09-17T07:04:02.534 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_seastore_epm": "0/5", 2024-09-17T07:04:02.534 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-09-17T07:04:02.534 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_seastore_journal": "0/5", 2024-09-17T07:04:02.534 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_seastore_lba": "0/5", 2024-09-17T07:04:02.534 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_seastore_odata": "0/5", 2024-09-17T07:04:02.534 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_seastore_omap": "0/5", 2024-09-17T07:04:02.535 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_seastore_onode": "0/5", 2024-09-17T07:04:02.535 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_seastore_t": "0/5", 2024-09-17T07:04:02.535 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_seastore_tm": "0/5", 2024-09-17T07:04:02.535 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_striper": "0/1", 2024-09-17T07:04:02.535 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_test": "0/5", 2024-09-17T07:04:02.535 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_throttle": "1/1", 2024-09-17T07:04:02.535 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_timer": "0/1", 2024-09-17T07:04:02.535 INFO:teuthology.orchestra.run.smithi100.stdout: "debug_tp": "0/5", 2024-09-17T07:04:02.535 INFO:teuthology.orchestra.run.smithi100.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-09-17T07:04:02.536 INFO:teuthology.orchestra.run.smithi100.stdout: "device_failure_prediction_mode": "none", 2024-09-17T07:04:02.536 INFO:teuthology.orchestra.run.smithi100.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-09-17T07:04:02.536 INFO:teuthology.orchestra.run.smithi100.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-09-17T07:04:02.536 INFO:teuthology.orchestra.run.smithi100.stdout: "err_to_graylog": "false", 2024-09-17T07:04:02.536 INFO:teuthology.orchestra.run.smithi100.stdout: "err_to_journald": "false", 2024-09-17T07:04:02.536 INFO:teuthology.orchestra.run.smithi100.stdout: "err_to_stderr": "true", 2024-09-17T07:04:02.536 INFO:teuthology.orchestra.run.smithi100.stdout: "err_to_syslog": "false", 2024-09-17T07:04:02.536 INFO:teuthology.orchestra.run.smithi100.stdout: "event_tracing": "false", 2024-09-17T07:04:02.536 INFO:teuthology.orchestra.run.smithi100.stdout: "exporter_addr": "0.0.0.0", 2024-09-17T07:04:02.536 INFO:teuthology.orchestra.run.smithi100.stdout: "exporter_http_port": "9926", 2024-09-17T07:04:02.537 INFO:teuthology.orchestra.run.smithi100.stdout: "exporter_prio_limit": "5", 2024-09-17T07:04:02.537 INFO:teuthology.orchestra.run.smithi100.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-09-17T07:04:02.537 INFO:teuthology.orchestra.run.smithi100.stdout: "exporter_sort_metrics": "true", 2024-09-17T07:04:02.537 INFO:teuthology.orchestra.run.smithi100.stdout: "exporter_stats_period": "5", 2024-09-17T07:04:02.537 INFO:teuthology.orchestra.run.smithi100.stdout: "fake_statfs_for_testing": "0", 2024-09-17T07:04:02.537 INFO:teuthology.orchestra.run.smithi100.stdout: "fatal_signal_handlers": "true", 2024-09-17T07:04:02.537 INFO:teuthology.orchestra.run.smithi100.stdout: "filer_max_purge_ops": "10", 2024-09-17T07:04:02.537 INFO:teuthology.orchestra.run.smithi100.stdout: "filer_max_truncate_ops": "128", 2024-09-17T07:04:02.537 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_apply_finisher_threads": "1", 2024-09-17T07:04:02.537 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_blackhole": "false", 2024-09-17T07:04:02.538 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_btrfs_clone_range": "true", 2024-09-17T07:04:02.538 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_btrfs_snap": "true", 2024-09-17T07:04:02.538 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_caller_concurrency": "10", 2024-09-17T07:04:02.538 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_collect_device_partition_information": "true", 2024-09-17T07:04:02.538 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_commit_timeout": "600.000000", 2024-09-17T07:04:02.538 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_debug_inject_read_err": "false", 2024-09-17T07:04:02.538 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_debug_omap_check": "false", 2024-09-17T07:04:02.538 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_debug_random_read_err": "0.000000", 2024-09-17T07:04:02.538 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_debug_verify_split": "false", 2024-09-17T07:04:02.538 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_dump_file": "", 2024-09-17T07:04:02.538 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-09-17T07:04:02.539 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-09-17T07:04:02.539 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_fadvise": "true", 2024-09-17T07:04:02.539 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_fail_eio": "true", 2024-09-17T07:04:02.539 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_fd_cache_shards": "16", 2024-09-17T07:04:02.539 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_fd_cache_size": "128", 2024-09-17T07:04:02.539 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_fiemap": "false", 2024-09-17T07:04:02.539 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_fiemap_threshold": "4096", 2024-09-17T07:04:02.539 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-09-17T07:04:02.539 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_index_retry_probability": "0.000000", 2024-09-17T07:04:02.539 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_inject_stall": "0", 2024-09-17T07:04:02.540 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_journal_parallel": "false", 2024-09-17T07:04:02.540 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_journal_trailing": "false", 2024-09-17T07:04:02.540 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_journal_writeahead": "false", 2024-09-17T07:04:02.540 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_kill_at": "0", 2024-09-17T07:04:02.540 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-09-17T07:04:02.540 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_max_inline_xattr_size": "0", 2024-09-17T07:04:02.540 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-09-17T07:04:02.540 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-09-17T07:04:02.540 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-09-17T07:04:02.540 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_max_inline_xattrs": "0", 2024-09-17T07:04:02.541 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-09-17T07:04:02.541 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_max_inline_xattrs_other": "2", 2024-09-17T07:04:02.541 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-09-17T07:04:02.541 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_max_sync_interval": "5.000000", 2024-09-17T07:04:02.541 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_max_xattr_value_size": "0", 2024-09-17T07:04:02.541 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-09-17T07:04:02.541 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-09-17T07:04:02.541 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-09-17T07:04:02.541 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_merge_threshold": "-10", 2024-09-17T07:04:02.542 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_min_sync_interval": "0.010000", 2024-09-17T07:04:02.542 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_odsync_write": "false", 2024-09-17T07:04:02.542 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_omap_backend": "rocksdb", 2024-09-17T07:04:02.542 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_omap_backend_path": "", 2024-09-17T07:04:02.542 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_omap_header_cache_size": "1024", 2024-09-17T07:04:02.542 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_ondisk_finisher_threads": "1", 2024-09-17T07:04:02.542 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-09-17T07:04:02.542 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_op_thread_timeout": "60", 2024-09-17T07:04:02.542 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_op_threads": "2", 2024-09-17T07:04:02.542 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_punch_hole": "false", 2024-09-17T07:04:02.542 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-09-17T07:04:02.543 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-09-17T07:04:02.543 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-09-17T07:04:02.543 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-09-17T07:04:02.543 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-09-17T07:04:02.543 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_queue_max_bytes": "104857600", 2024-09-17T07:04:02.543 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-09-17T07:04:02.543 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-09-17T07:04:02.543 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-09-17T07:04:02.543 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_queue_max_ops": "50", 2024-09-17T07:04:02.544 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-09-17T07:04:02.544 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_seek_data_hole": "false", 2024-09-17T07:04:02.544 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_sloppy_crc": "false", 2024-09-17T07:04:02.544 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-09-17T07:04:02.544 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_splice": "false", 2024-09-17T07:04:02.544 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_split_multiple": "2", 2024-09-17T07:04:02.544 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_split_rand_factor": "20", 2024-09-17T07:04:02.544 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_update_to": "1000", 2024-09-17T07:04:02.544 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-09-17T07:04:02.544 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-09-17T07:04:02.544 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-09-17T07:04:02.545 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-09-17T07:04:02.545 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-09-17T07:04:02.545 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-09-17T07:04:02.545 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_wbthrottle_enable": "true", 2024-09-17T07:04:02.545 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-09-17T07:04:02.545 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-09-17T07:04:02.545 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-09-17T07:04:02.545 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-09-17T07:04:02.545 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-09-17T07:04:02.545 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-09-17T07:04:02.545 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_xfs_extsize": "false", 2024-09-17T07:04:02.546 INFO:teuthology.orchestra.run.smithi100.stdout: "filestore_zfs_snap": "false", 2024-09-17T07:04:02.546 INFO:teuthology.orchestra.run.smithi100.stdout: "fio_dir": "/tmp/fio", 2024-09-17T07:04:02.546 INFO:teuthology.orchestra.run.smithi100.stdout: "fsid": "e848267c-74c1-11ef-bceb-c7b262605968", 2024-09-17T07:04:02.546 INFO:teuthology.orchestra.run.smithi100.stdout: "fuse_allow_other": "true", 2024-09-17T07:04:02.546 INFO:teuthology.orchestra.run.smithi100.stdout: "fuse_atomic_o_trunc": "true", 2024-09-17T07:04:02.546 INFO:teuthology.orchestra.run.smithi100.stdout: "fuse_big_writes": "true", 2024-09-17T07:04:02.546 INFO:teuthology.orchestra.run.smithi100.stdout: "fuse_debug": "false", 2024-09-17T07:04:02.546 INFO:teuthology.orchestra.run.smithi100.stdout: "fuse_default_permissions": "false", 2024-09-17T07:04:02.546 INFO:teuthology.orchestra.run.smithi100.stdout: "fuse_disable_pagecache": "false", 2024-09-17T07:04:02.546 INFO:teuthology.orchestra.run.smithi100.stdout: "fuse_max_write": "0", 2024-09-17T07:04:02.546 INFO:teuthology.orchestra.run.smithi100.stdout: "fuse_multithreaded": "true", 2024-09-17T07:04:02.547 INFO:teuthology.orchestra.run.smithi100.stdout: "fuse_require_active_mds": "true", 2024-09-17T07:04:02.547 INFO:teuthology.orchestra.run.smithi100.stdout: "fuse_set_user_groups": "true", 2024-09-17T07:04:02.547 INFO:teuthology.orchestra.run.smithi100.stdout: "fuse_splice_move": "true", 2024-09-17T07:04:02.547 INFO:teuthology.orchestra.run.smithi100.stdout: "fuse_splice_read": "true", 2024-09-17T07:04:02.547 INFO:teuthology.orchestra.run.smithi100.stdout: "fuse_splice_write": "true", 2024-09-17T07:04:02.547 INFO:teuthology.orchestra.run.smithi100.stdout: "fuse_syncfs_on_mksnap": "true", 2024-09-17T07:04:02.547 INFO:teuthology.orchestra.run.smithi100.stdout: "fuse_use_invalidate_cb": "true", 2024-09-17T07:04:02.547 INFO:teuthology.orchestra.run.smithi100.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-09-17T07:04:02.547 INFO:teuthology.orchestra.run.smithi100.stdout: "gss_target_name": "ceph", 2024-09-17T07:04:02.547 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_file": "", 2024-09-17T07:04:02.547 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_inject_failure": "0", 2024-09-17T07:04:02.548 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_interval": "5", 2024-09-17T07:04:02.548 INFO:teuthology.orchestra.run.smithi100.stdout: "host": "smithi100", 2024-09-17T07:04:02.548 INFO:teuthology.orchestra.run.smithi100.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-09-17T07:04:02.548 INFO:teuthology.orchestra.run.smithi100.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-09-17T07:04:02.548 INFO:teuthology.orchestra.run.smithi100.stdout: "immutable_object_cache_max_size": "1073741824", 2024-09-17T07:04:02.548 INFO:teuthology.orchestra.run.smithi100.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-09-17T07:04:02.548 INFO:teuthology.orchestra.run.smithi100.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-09-17T07:04:02.548 INFO:teuthology.orchestra.run.smithi100.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-09-17T07:04:02.548 INFO:teuthology.orchestra.run.smithi100.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-09-17T07:04:02.548 INFO:teuthology.orchestra.run.smithi100.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-09-17T07:04:02.548 INFO:teuthology.orchestra.run.smithi100.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-09-17T07:04:02.549 INFO:teuthology.orchestra.run.smithi100.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-09-17T07:04:02.549 INFO:teuthology.orchestra.run.smithi100.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-09-17T07:04:02.549 INFO:teuthology.orchestra.run.smithi100.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-09-17T07:04:02.549 INFO:teuthology.orchestra.run.smithi100.stdout: "immutable_object_cache_watermark": "0.900000", 2024-09-17T07:04:02.549 INFO:teuthology.orchestra.run.smithi100.stdout: "inject_early_sigterm": "false", 2024-09-17T07:04:02.549 INFO:teuthology.orchestra.run.smithi100.stdout: "jaeger_agent_port": "6799", 2024-09-17T07:04:02.549 INFO:teuthology.orchestra.run.smithi100.stdout: "jaeger_tracing_enable": "false", 2024-09-17T07:04:02.549 INFO:teuthology.orchestra.run.smithi100.stdout: "journal_aio": "true", 2024-09-17T07:04:02.549 INFO:teuthology.orchestra.run.smithi100.stdout: "journal_align_min_size": "65536", 2024-09-17T07:04:02.549 INFO:teuthology.orchestra.run.smithi100.stdout: "journal_block_align": "true", 2024-09-17T07:04:02.550 INFO:teuthology.orchestra.run.smithi100.stdout: "journal_block_size": "4096", 2024-09-17T07:04:02.550 INFO:teuthology.orchestra.run.smithi100.stdout: "journal_dio": "true", 2024-09-17T07:04:02.550 INFO:teuthology.orchestra.run.smithi100.stdout: "journal_discard": "false", 2024-09-17T07:04:02.550 INFO:teuthology.orchestra.run.smithi100.stdout: "journal_force_aio": "false", 2024-09-17T07:04:02.550 INFO:teuthology.orchestra.run.smithi100.stdout: "journal_ignore_corruption": "false", 2024-09-17T07:04:02.550 INFO:teuthology.orchestra.run.smithi100.stdout: "journal_max_write_bytes": "10485760", 2024-09-17T07:04:02.550 INFO:teuthology.orchestra.run.smithi100.stdout: "journal_max_write_entries": "100", 2024-09-17T07:04:02.550 INFO:teuthology.orchestra.run.smithi100.stdout: "journal_replay_from": "0", 2024-09-17T07:04:02.550 INFO:teuthology.orchestra.run.smithi100.stdout: "journal_throttle_high_multiple": "0.000000", 2024-09-17T07:04:02.550 INFO:teuthology.orchestra.run.smithi100.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-09-17T07:04:02.550 INFO:teuthology.orchestra.run.smithi100.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-09-17T07:04:02.550 INFO:teuthology.orchestra.run.smithi100.stdout: "journal_throttle_max_multiple": "0.000000", 2024-09-17T07:04:02.551 INFO:teuthology.orchestra.run.smithi100.stdout: "journal_write_header_frequency": "0", 2024-09-17T07:04:02.551 INFO:teuthology.orchestra.run.smithi100.stdout: "journal_zero_on_create": "false", 2024-09-17T07:04:02.551 INFO:teuthology.orchestra.run.smithi100.stdout: "journaler_prefetch_periods": "10", 2024-09-17T07:04:02.551 INFO:teuthology.orchestra.run.smithi100.stdout: "journaler_prezero_periods": "5", 2024-09-17T07:04:02.551 INFO:teuthology.orchestra.run.smithi100.stdout: "journaler_write_head_interval": "15", 2024-09-17T07:04:02.551 INFO:teuthology.orchestra.run.smithi100.stdout: "key": "", 2024-09-17T07:04:02.552 INFO:teuthology.orchestra.run.smithi100.stdout: "keyfile": "", 2024-09-17T07:04:02.552 INFO:teuthology.orchestra.run.smithi100.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-09-17T07:04:02.552 INFO:teuthology.orchestra.run.smithi100.stdout: "kstore_backend": "rocksdb", 2024-09-17T07:04:02.552 INFO:teuthology.orchestra.run.smithi100.stdout: "kstore_default_stripe_size": "65536", 2024-09-17T07:04:02.553 INFO:teuthology.orchestra.run.smithi100.stdout: "kstore_fsck_on_mount": "false", 2024-09-17T07:04:02.553 INFO:teuthology.orchestra.run.smithi100.stdout: "kstore_fsck_on_mount_deep": "true", 2024-09-17T07:04:02.553 INFO:teuthology.orchestra.run.smithi100.stdout: "kstore_max_bytes": "67108864", 2024-09-17T07:04:02.553 INFO:teuthology.orchestra.run.smithi100.stdout: "kstore_max_ops": "512", 2024-09-17T07:04:02.553 INFO:teuthology.orchestra.run.smithi100.stdout: "kstore_nid_prealloc": "1024", 2024-09-17T07:04:02.553 INFO:teuthology.orchestra.run.smithi100.stdout: "kstore_onode_map_size": "1024", 2024-09-17T07:04:02.553 INFO:teuthology.orchestra.run.smithi100.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-09-17T07:04:02.553 INFO:teuthology.orchestra.run.smithi100.stdout: "kstore_sync_submit_transaction": "false", 2024-09-17T07:04:02.553 INFO:teuthology.orchestra.run.smithi100.stdout: "kstore_sync_transaction": "false", 2024-09-17T07:04:02.553 INFO:teuthology.orchestra.run.smithi100.stdout: "leveldb_block_size": "65536", 2024-09-17T07:04:02.553 INFO:teuthology.orchestra.run.smithi100.stdout: "leveldb_bloom_size": "0", 2024-09-17T07:04:02.554 INFO:teuthology.orchestra.run.smithi100.stdout: "leveldb_cache_size": "536870912", 2024-09-17T07:04:02.554 INFO:teuthology.orchestra.run.smithi100.stdout: "leveldb_compact_on_mount": "false", 2024-09-17T07:04:02.554 INFO:teuthology.orchestra.run.smithi100.stdout: "leveldb_compression": "false", 2024-09-17T07:04:02.554 INFO:teuthology.orchestra.run.smithi100.stdout: "leveldb_log": "", 2024-09-17T07:04:02.554 INFO:teuthology.orchestra.run.smithi100.stdout: "leveldb_log_to_ceph_log": "true", 2024-09-17T07:04:02.554 INFO:teuthology.orchestra.run.smithi100.stdout: "leveldb_max_open_files": "0", 2024-09-17T07:04:02.554 INFO:teuthology.orchestra.run.smithi100.stdout: "leveldb_paranoid": "false", 2024-09-17T07:04:02.554 INFO:teuthology.orchestra.run.smithi100.stdout: "leveldb_write_buffer_size": "33554432", 2024-09-17T07:04:02.554 INFO:teuthology.orchestra.run.smithi100.stdout: "librados_thread_count": "2", 2024-09-17T07:04:02.554 INFO:teuthology.orchestra.run.smithi100.stdout: "lockdep": "false", 2024-09-17T07:04:02.555 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:04:02 smithi135 podman[64826]: 2024-09-17 07:04:02.156329129 +0000 UTC m=+0.016991840 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-09-17T07:04:02.555 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:04:02 smithi135 podman[64826]: 2024-09-17 07:04:02.270123384 +0000 UTC m=+0.130786121 container create 8bdc6108c6f73e24402d13d2d88f4a1e57eb1aee376eacf204aa1133b78eb22a (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-17T07:04:02.557 INFO:teuthology.orchestra.run.smithi100.stdout: "lockdep_force_backtrace": "false", 2024-09-17T07:04:02.557 INFO:teuthology.orchestra.run.smithi100.stdout: "log_coarse_timestamps": "true", 2024-09-17T07:04:02.557 INFO:teuthology.orchestra.run.smithi100.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-09-17T07:04:02.557 INFO:teuthology.orchestra.run.smithi100.stdout: "log_flush_on_exit": "false", 2024-09-17T07:04:02.557 INFO:teuthology.orchestra.run.smithi100.stdout: "log_graylog_host": "127.0.0.1", 2024-09-17T07:04:02.557 INFO:teuthology.orchestra.run.smithi100.stdout: "log_graylog_port": "12201", 2024-09-17T07:04:02.557 INFO:teuthology.orchestra.run.smithi100.stdout: "log_max_new": "1000", 2024-09-17T07:04:02.557 INFO:teuthology.orchestra.run.smithi100.stdout: "log_max_recent": "10000", 2024-09-17T07:04:02.557 INFO:teuthology.orchestra.run.smithi100.stdout: "log_stderr_prefix": "", 2024-09-17T07:04:02.558 INFO:teuthology.orchestra.run.smithi100.stdout: "log_stop_at_utilization": "0.970000", 2024-09-17T07:04:02.558 INFO:teuthology.orchestra.run.smithi100.stdout: "log_to_file": "true", 2024-09-17T07:04:02.558 INFO:teuthology.orchestra.run.smithi100.stdout: "log_to_graylog": "false", 2024-09-17T07:04:02.558 INFO:teuthology.orchestra.run.smithi100.stdout: "log_to_journald": "false", 2024-09-17T07:04:02.558 INFO:teuthology.orchestra.run.smithi100.stdout: "log_to_stderr": "false", 2024-09-17T07:04:02.558 INFO:teuthology.orchestra.run.smithi100.stdout: "log_to_syslog": "false", 2024-09-17T07:04:02.558 INFO:teuthology.orchestra.run.smithi100.stdout: "max_rotating_auth_attempts": "10", 2024-09-17T07:04:02.558 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-09-17T07:04:02.558 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_action_on_write_error": "1", 2024-09-17T07:04:02.558 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_alternate_name_max": "8192", 2024-09-17T07:04:02.559 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_asio_thread_count": "2", 2024-09-17T07:04:02.559 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_bal_export_pin": "true", 2024-09-17T07:04:02.559 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_bal_fragment_dirs": "true", 2024-09-17T07:04:02.559 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-09-17T07:04:02.559 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_bal_fragment_interval": "5", 2024-09-17T07:04:02.559 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_bal_fragment_size_max": "100000", 2024-09-17T07:04:02.559 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_bal_idle_threshold": "0.000000", 2024-09-17T07:04:02.559 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_bal_interval": "10", 2024-09-17T07:04:02.559 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_bal_max": "-1", 2024-09-17T07:04:02.559 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_bal_max_until": "-1", 2024-09-17T07:04:02.559 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_bal_merge_size": "50", 2024-09-17T07:04:02.560 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_bal_midchunk": "0.300000", 2024-09-17T07:04:02.560 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_bal_min_rebalance": "0.100000", 2024-09-17T07:04:02.560 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_bal_min_start": "0.200000", 2024-09-17T07:04:02.560 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_bal_minchunk": "0.001000", 2024-09-17T07:04:02.560 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_bal_mode": "0", 2024-09-17T07:04:02.560 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_bal_need_max": "1.200000", 2024-09-17T07:04:02.560 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_bal_need_min": "0.800000", 2024-09-17T07:04:02.560 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-09-17T07:04:02.560 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_bal_sample_interval": "3.000000", 2024-09-17T07:04:02.560 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_bal_split_bits": "3", 2024-09-17T07:04:02.561 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_bal_split_rd": "25000.000000", 2024-09-17T07:04:02.561 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_bal_split_size": "10000", 2024-09-17T07:04:02.561 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_bal_split_wr": "10000.000000", 2024-09-17T07:04:02.561 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_bal_target_decay": "10.000000", 2024-09-17T07:04:02.561 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-09-17T07:04:02.561 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_beacon_grace": "15.000000", 2024-09-17T07:04:02.561 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_beacon_interval": "4.000000", 2024-09-17T07:04:02.561 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_beacon_mon_down_grace": "60", 2024-09-17T07:04:02.561 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_cache_memory_limit": "4294967296", 2024-09-17T07:04:02.561 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_cache_mid": "0.700000", 2024-09-17T07:04:02.561 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_cache_release_free_interval": "10", 2024-09-17T07:04:02.562 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_cache_reservation": "0.050000", 2024-09-17T07:04:02.562 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-09-17T07:04:02.562 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_cache_trim_interval": "1", 2024-09-17T07:04:02.562 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_cache_trim_threshold": "262144", 2024-09-17T07:04:02.562 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-09-17T07:04:02.562 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-09-17T07:04:02.562 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_client_delegate_inos_pct": "50", 2024-09-17T07:04:02.562 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_client_prealloc_inos": "1000", 2024-09-17T07:04:02.562 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-09-17T07:04:02.562 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_connect_bootstrapping": "false", 2024-09-17T07:04:02.563 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_damage_table_max_entries": "10000", 2024-09-17T07:04:02.563 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-09-17T07:04:02.563 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_debug_auth_pins": "false", 2024-09-17T07:04:02.563 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_debug_frag": "false", 2024-09-17T07:04:02.563 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_debug_scatterstat": "false", 2024-09-17T07:04:02.563 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_debug_subtrees": "false", 2024-09-17T07:04:02.563 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_decay_halflife": "5.000000", 2024-09-17T07:04:02.563 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_default_dir_hash": "2", 2024-09-17T07:04:02.563 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_defer_session_stale": "true", 2024-09-17T07:04:02.563 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_deny_all_reconnect": "false", 2024-09-17T07:04:02.563 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_dir_keys_per_op": "16384", 2024-09-17T07:04:02.564 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_dir_max_commit_size": "10", 2024-09-17T07:04:02.564 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_dir_max_entries": "0", 2024-09-17T07:04:02.564 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_dir_prefetch": "true", 2024-09-17T07:04:02.564 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_dirstat_min_interval": "1.000000", 2024-09-17T07:04:02.564 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_dump_cache_after_rejoin": "false", 2024-09-17T07:04:02.564 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_dump_cache_on_map": "false", 2024-09-17T07:04:02.564 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_dump_cache_threshold_file": "0", 2024-09-17T07:04:02.564 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-09-17T07:04:02.564 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_early_reply": "true", 2024-09-17T07:04:02.564 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_enable_op_tracker": "true", 2024-09-17T07:04:02.565 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_enforce_unique_name": "true", 2024-09-17T07:04:02.565 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_export_ephemeral_distributed": "true", 2024-09-17T07:04:02.565 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-09-17T07:04:02.565 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_export_ephemeral_random": "true", 2024-09-17T07:04:02.565 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-09-17T07:04:02.565 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-09-17T07:04:02.565 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_forward_all_requests_to_auth": "false", 2024-09-17T07:04:02.565 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-09-17T07:04:02.566 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-09-17T07:04:02.566 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-09-17T07:04:02.566 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-09-17T07:04:02.566 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_health_cache_threshold": "1.500000", 2024-09-17T07:04:02.566 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_health_summarize_threshold": "10", 2024-09-17T07:04:02.566 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_heartbeat_grace": "15.000000", 2024-09-17T07:04:02.566 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_heartbeat_reset_grace": "1000", 2024-09-17T07:04:02.566 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_inject_health_dummy": "false", 2024-09-17T07:04:02.566 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-09-17T07:04:02.566 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_inject_migrator_session_race": "false", 2024-09-17T07:04:02.567 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-09-17T07:04:02.567 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-09-17T07:04:02.567 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-09-17T07:04:02.567 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_join_fs": "", 2024-09-17T07:04:02.567 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_journal_format": "1", 2024-09-17T07:04:02.567 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_kill_after_journal_logs_flushed": "false", 2024-09-17T07:04:02.567 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_kill_create_at": "0", 2024-09-17T07:04:02.567 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_kill_export_at": "0", 2024-09-17T07:04:02.567 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_kill_import_at": "0", 2024-09-17T07:04:02.567 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_kill_journal_at": "0", 2024-09-17T07:04:02.568 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_kill_journal_expire_at": "0", 2024-09-17T07:04:02.568 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_kill_journal_replay_at": "0", 2024-09-17T07:04:02.568 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_kill_link_at": "0", 2024-09-17T07:04:02.568 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_kill_mdstable_at": "0", 2024-09-17T07:04:02.568 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_kill_openc_at": "0", 2024-09-17T07:04:02.568 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_kill_rename_at": "0", 2024-09-17T07:04:02.568 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_log_events_per_segment": "1024", 2024-09-17T07:04:02.568 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_log_max_events": "-1", 2024-09-17T07:04:02.568 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_log_max_segments": "128", 2024-09-17T07:04:02.568 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_log_pause": "false", 2024-09-17T07:04:02.568 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_log_segment_size": "0", 2024-09-17T07:04:02.569 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_log_skip_corrupt_events": "false", 2024-09-17T07:04:02.569 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_log_warn_factor": "2.000000", 2024-09-17T07:04:02.569 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_max_caps_per_client": "1048576", 2024-09-17T07:04:02.569 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_max_completed_flushes": "100000", 2024-09-17T07:04:02.569 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_max_completed_requests": "100000", 2024-09-17T07:04:02.569 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_max_export_size": "20971520", 2024-09-17T07:04:02.569 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_max_file_recover": "32", 2024-09-17T07:04:02.569 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_max_purge_files": "64", 2024-09-17T07:04:02.569 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_max_purge_ops": "8192", 2024-09-17T07:04:02.569 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-09-17T07:04:02.570 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-09-17T07:04:02.570 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_max_snaps_per_dir": "100", 2024-09-17T07:04:02.570 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_metrics_update_interval": "2", 2024-09-17T07:04:02.570 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_min_caps_per_client": "100", 2024-09-17T07:04:02.570 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_min_caps_working_set": "10000", 2024-09-17T07:04:02.570 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-09-17T07:04:02.570 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_numa_node": "-1", 2024-09-17T07:04:02.570 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-09-17T07:04:02.570 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_op_complaint_time": "30.000000", 2024-09-17T07:04:02.570 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_op_history_duration": "600", 2024-09-17T07:04:02.570 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_op_history_size": "20", 2024-09-17T07:04:02.571 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_op_history_slow_op_size": "20", 2024-09-17T07:04:02.571 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_op_history_slow_op_threshold": "10", 2024-09-17T07:04:02.571 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_op_log_threshold": "5", 2024-09-17T07:04:02.571 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_ping_grace": "15", 2024-09-17T07:04:02.571 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_ping_interval": "5", 2024-09-17T07:04:02.571 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-09-17T07:04:02.571 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-09-17T07:04:02.571 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_recall_max_caps": "30000", 2024-09-17T07:04:02.571 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-09-17T07:04:02.571 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_recall_max_decay_threshold": "131072", 2024-09-17T07:04:02.572 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-09-17T07:04:02.572 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_recall_warning_threshold": "262144", 2024-09-17T07:04:02.572 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_reconnect_timeout": "45.000000", 2024-09-17T07:04:02.572 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_replay_interval": "1.000000", 2024-09-17T07:04:02.572 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-09-17T07:04:02.572 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-09-17T07:04:02.572 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_root_ino_gid": "0", 2024-09-17T07:04:02.572 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_root_ino_uid": "0", 2024-09-17T07:04:02.572 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-09-17T07:04:02.572 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_session_blocklist_on_evict": "true", 2024-09-17T07:04:02.572 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_session_blocklist_on_timeout": "true", 2024-09-17T07:04:02.573 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-09-17T07:04:02.573 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-09-17T07:04:02.573 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-09-17T07:04:02.573 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-09-17T07:04:02.573 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-09-17T07:04:02.573 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_session_metadata_threshold": "16777216", 2024-09-17T07:04:02.573 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-09-17T07:04:02.573 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_shutdown_check": "0", 2024-09-17T07:04:02.573 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_skip_ino": "0", 2024-09-17T07:04:02.573 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_sleep_rank_change": "0.000000", 2024-09-17T07:04:02.573 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_snap_max_uid": "4294967294", 2024-09-17T07:04:02.574 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_snap_min_uid": "0", 2024-09-17T07:04:02.574 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_snap_rstat": "false", 2024-09-17T07:04:02.574 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_standby_replay_damaged": "false", 2024-09-17T07:04:02.574 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_symlink_recovery": "true", 2024-09-17T07:04:02.574 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_task_status_update_interval": "2.000000", 2024-09-17T07:04:02.574 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_thrash_exports": "0", 2024-09-17T07:04:02.574 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_thrash_fragments": "0", 2024-09-17T07:04:02.574 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_tick_interval": "5.000000", 2024-09-17T07:04:02.574 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_valgrind_exit": "false", 2024-09-17T07:04:02.574 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_verify_backtrace": "1", 2024-09-17T07:04:02.574 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_verify_scatter": "false", 2024-09-17T07:04:02.575 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_wipe_ino_prealloc": "false", 2024-09-17T07:04:02.575 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_wipe_sessions": "false", 2024-09-17T07:04:02.575 INFO:teuthology.orchestra.run.smithi100.stdout: "mempool_debug": "false", 2024-09-17T07:04:02.575 INFO:teuthology.orchestra.run.smithi100.stdout: "memstore_debug_omit_block_device_write": "false", 2024-09-17T07:04:02.575 INFO:teuthology.orchestra.run.smithi100.stdout: "memstore_device_bytes": "1073741824", 2024-09-17T07:04:02.575 INFO:teuthology.orchestra.run.smithi100.stdout: "memstore_page_set": "false", 2024-09-17T07:04:02.575 INFO:teuthology.orchestra.run.smithi100.stdout: "memstore_page_size": "65536", 2024-09-17T07:04:02.575 INFO:teuthology.orchestra.run.smithi100.stdout: "mgr_client_bytes": "134217728", 2024-09-17T07:04:02.575 INFO:teuthology.orchestra.run.smithi100.stdout: "mgr_client_messages": "512", 2024-09-17T07:04:02.576 INFO:teuthology.orchestra.run.smithi100.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-09-17T07:04:02.576 INFO:teuthology.orchestra.run.smithi100.stdout: "mgr_connect_retry_interval": "1.000000", 2024-09-17T07:04:02.576 INFO:teuthology.orchestra.run.smithi100.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-09-17T07:04:02.576 INFO:teuthology.orchestra.run.smithi100.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-09-17T07:04:02.576 INFO:teuthology.orchestra.run.smithi100.stdout: "mgr_disabled_modules": "", 2024-09-17T07:04:02.576 INFO:teuthology.orchestra.run.smithi100.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-09-17T07:04:02.576 INFO:teuthology.orchestra.run.smithi100.stdout: "mgr_max_pg_num_change": "128", 2024-09-17T07:04:02.576 INFO:teuthology.orchestra.run.smithi100.stdout: "mgr_mds_bytes": "134217728", 2024-09-17T07:04:02.576 INFO:teuthology.orchestra.run.smithi100.stdout: "mgr_mds_messages": "128", 2024-09-17T07:04:02.576 INFO:teuthology.orchestra.run.smithi100.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-09-17T07:04:02.576 INFO:teuthology.orchestra.run.smithi100.stdout: "mgr_mon_bytes": "134217728", 2024-09-17T07:04:02.577 INFO:teuthology.orchestra.run.smithi100.stdout: "mgr_mon_messages": "128", 2024-09-17T07:04:02.577 INFO:teuthology.orchestra.run.smithi100.stdout: "mgr_osd_bytes": "536870912", 2024-09-17T07:04:02.577 INFO:teuthology.orchestra.run.smithi100.stdout: "mgr_osd_messages": "8192", 2024-09-17T07:04:02.577 INFO:teuthology.orchestra.run.smithi100.stdout: "mgr_pool": "true", 2024-09-17T07:04:02.577 INFO:teuthology.orchestra.run.smithi100.stdout: "mgr_service_beacon_grace": "60.000000", 2024-09-17T07:04:02.577 INFO:teuthology.orchestra.run.smithi100.stdout: "mgr_standby_modules": "true", 2024-09-17T07:04:02.577 INFO:teuthology.orchestra.run.smithi100.stdout: "mgr_stats_period": "5", 2024-09-17T07:04:02.577 INFO:teuthology.orchestra.run.smithi100.stdout: "mgr_stats_threshold": "5", 2024-09-17T07:04:02.577 INFO:teuthology.orchestra.run.smithi100.stdout: "mgr_tick_period": "2", 2024-09-17T07:04:02.577 INFO:teuthology.orchestra.run.smithi100.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-09-17T07:04:02.578 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_accept_timeout_factor": "2.000000", 2024-09-17T07:04:02.578 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_allow_pool_delete": "true", 2024-09-17T07:04:02.578 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_allow_pool_size_one": "false", 2024-09-17T07:04:02.578 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_auth_validate_all_caps": "true", 2024-09-17T07:04:02.578 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-09-17T07:04:02.578 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-09-17T07:04:02.578 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_client_bytes": "104857600", 2024-09-17T07:04:02.578 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_client_directed_command_retry": "2", 2024-09-17T07:04:02.578 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_client_hunt_interval": "3.000000", 2024-09-17T07:04:02.578 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-09-17T07:04:02.578 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-09-17T07:04:02.578 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-09-17T07:04:02.579 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_client_hunt_parallel": "3", 2024-09-17T07:04:02.579 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_client_log_interval": "1.000000", 2024-09-17T07:04:02.579 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-09-17T07:04:02.579 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_client_ping_interval": "10.000000", 2024-09-17T07:04:02.579 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_client_ping_timeout": "30.000000", 2024-09-17T07:04:02.579 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_clock_drift_allowed": "1.000000", 2024-09-17T07:04:02.579 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-09-17T07:04:02.579 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-09-17T07:04:02.579 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_cluster_log_file_level": "debug", 2024-09-17T07:04:02.579 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_cluster_log_to_file": "false", 2024-09-17T07:04:02.580 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_cluster_log_to_graylog": "false", 2024-09-17T07:04:02.580 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-09-17T07:04:02.580 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-09-17T07:04:02.580 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_cluster_log_to_journald": "true", 2024-09-17T07:04:02.580 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_cluster_log_to_stderr": "false", 2024-09-17T07:04:02.580 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-09-17T07:04:02.580 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-09-17T07:04:02.580 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_cluster_log_to_syslog_level": "info", 2024-09-17T07:04:02.580 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_compact_on_bootstrap": "false", 2024-09-17T07:04:02.580 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_compact_on_start": "false", 2024-09-17T07:04:02.580 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_compact_on_trim": "true", 2024-09-17T07:04:02.580 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_con_tracker_persist_interval": "10", 2024-09-17T07:04:02.581 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_con_tracker_score_halflife": "43200", 2024-09-17T07:04:02.581 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_config_key_max_entry_size": "65536", 2024-09-17T07:04:02.581 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_cpu_threads": "4", 2024-09-17T07:04:02.581 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_crush_min_required_version": "hammer", 2024-09-17T07:04:02.581 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_daemon_bytes": "419430400", 2024-09-17T07:04:02.581 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-09-17T07:04:02.581 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_data_avail_crit": "5", 2024-09-17T07:04:02.581 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_data_avail_warn": "5", 2024-09-17T07:04:02.581 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_data_size_warn": "16106127360", 2024-09-17T07:04:02.581 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_debug_block_osdmap_trim": "false", 2024-09-17T07:04:02.582 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-09-17T07:04:02.582 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_debug_dump_json": "false", 2024-09-17T07:04:02.582 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-09-17T07:04:02.582 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_debug_dump_transactions": "false", 2024-09-17T07:04:02.582 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_debug_extra_checks": "false", 2024-09-17T07:04:02.582 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-09-17T07:04:02.582 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-09-17T07:04:02.582 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_debug_no_require_quincy": "false", 2024-09-17T07:04:02.582 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_debug_no_require_reef": "false", 2024-09-17T07:04:02.582 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-09-17T07:04:02.582 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_delta_reset_interval": "10.000000", 2024-09-17T07:04:02.583 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_dns_srv_name": "ceph-mon", 2024-09-17T07:04:02.583 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_down_mkfs_grace": "60", 2024-09-17T07:04:02.583 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_election_default_strategy": "1", 2024-09-17T07:04:02.583 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_election_timeout": "5.000000", 2024-09-17T07:04:02.583 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-09-17T07:04:02.583 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_elector_ping_divisor": "2", 2024-09-17T07:04:02.583 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_elector_ping_timeout": "2.000000", 2024-09-17T07:04:02.583 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_enable_op_tracker": "true", 2024-09-17T07:04:02.583 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_fake_pool_delete": "false", 2024-09-17T07:04:02.583 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_force_quorum_join": "false", 2024-09-17T07:04:02.584 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_globalid_prealloc": "10000", 2024-09-17T07:04:02.584 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_health_detail_to_clog": "true", 2024-09-17T07:04:02.584 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_health_log_update_period": "5", 2024-09-17T07:04:02.584 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_health_max_detail": "50", 2024-09-17T07:04:02.584 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_health_to_clog": "true", 2024-09-17T07:04:02.584 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_health_to_clog_interval": "600", 2024-09-17T07:04:02.584 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-09-17T07:04:02.584 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_host": "[v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0]", 2024-09-17T07:04:02.584 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_host_override": "", 2024-09-17T07:04:02.584 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_initial_members": "", 2024-09-17T07:04:02.584 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-09-17T07:04:02.585 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-09-17T07:04:02.585 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-09-17T07:04:02.585 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-09-17T07:04:02.585 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_keyvaluedb": "rocksdb", 2024-09-17T07:04:02.585 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_lease": "5.000000", 2024-09-17T07:04:02.585 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-09-17T07:04:02.585 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-09-17T07:04:02.585 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_log_full_interval": "50", 2024-09-17T07:04:02.585 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_log_max": "10000", 2024-09-17T07:04:02.585 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_log_max_summary": "50", 2024-09-17T07:04:02.585 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_max_log_entries_per_event": "4096", 2024-09-17T07:04:02.586 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_max_log_epochs": "500", 2024-09-17T07:04:02.586 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_max_mdsmap_epochs": "500", 2024-09-17T07:04:02.586 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_max_mgrmap_epochs": "500", 2024-09-17T07:04:02.586 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_max_osd": "10000", 2024-09-17T07:04:02.586 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_max_pg_per_osd": "10000", 2024-09-17T07:04:02.586 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_max_pool_pg_num": "65536", 2024-09-17T07:04:02.586 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-09-17T07:04:02.586 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-09-17T07:04:02.586 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_mds_force_trim_to": "0", 2024-09-17T07:04:02.586 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_mds_skip_sanity": "false", 2024-09-17T07:04:02.586 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_memory_autotune": "true", 2024-09-17T07:04:02.587 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_memory_target": "2147483648", 2024-09-17T07:04:02.587 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_mgr_beacon_grace": "30", 2024-09-17T07:04:02.587 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-09-17T07:04:02.587 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_mgr_digest_period": "5", 2024-09-17T07:04:02.587 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_mgr_inactive_grace": "60", 2024-09-17T07:04:02.587 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_mgr_mkfs_grace": "240", 2024-09-17T07:04:02.587 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-09-17T07:04:02.587 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_min_osdmap_epochs": "50", 2024-09-17T07:04:02.587 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_op_complaint_time": "30", 2024-09-17T07:04:02.587 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_op_history_duration": "600", 2024-09-17T07:04:02.587 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_op_history_size": "20", 2024-09-17T07:04:02.588 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_op_history_slow_op_size": "20", 2024-09-17T07:04:02.588 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_op_history_slow_op_threshold": "10", 2024-09-17T07:04:02.588 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_op_log_threshold": "5", 2024-09-17T07:04:02.588 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-09-17T07:04:02.588 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-09-17T07:04:02.588 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-09-17T07:04:02.588 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_osd_auto_mark_in": "false", 2024-09-17T07:04:02.588 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_osd_auto_mark_new_in": "true", 2024-09-17T07:04:02.588 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-09-17T07:04:02.588 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-09-17T07:04:02.588 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_osd_cache_size": "500", 2024-09-17T07:04:02.589 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_osd_cache_size_min": "134217728", 2024-09-17T07:04:02.589 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_osd_crush_smoke_test": "true", 2024-09-17T07:04:02.589 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_osd_destroyed_out_interval": "600", 2024-09-17T07:04:02.589 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_osd_down_out_interval": "600", 2024-09-17T07:04:02.589 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-09-17T07:04:02.589 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-09-17T07:04:02.589 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_osd_force_trim_to": "0", 2024-09-17T07:04:02.589 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_osd_full_ratio": "0.950000", 2024-09-17T07:04:02.589 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-09-17T07:04:02.589 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_osd_laggy_halflife": "3600", 2024-09-17T07:04:02.590 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_osd_laggy_max_interval": "300", 2024-09-17T07:04:02.590 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_osd_laggy_weight": "0.300000", 2024-09-17T07:04:02.590 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-09-17T07:04:02.590 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_osd_max_creating_pgs": "1024", 2024-09-17T07:04:02.590 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_osd_max_initial_pgs": "1024", 2024-09-17T07:04:02.590 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_osd_min_down_reporters": "2", 2024-09-17T07:04:02.590 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-09-17T07:04:02.590 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-09-17T07:04:02.590 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-09-17T07:04:02.590 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_osd_prime_pg_temp": "true", 2024-09-17T07:04:02.590 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-09-17T07:04:02.591 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-09-17T07:04:02.591 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_osd_report_timeout": "900", 2024-09-17T07:04:02.591 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-09-17T07:04:02.591 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-09-17T07:04:02.591 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_osd_warn_num_repaired": "10", 2024-09-17T07:04:02.591 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_osd_warn_op_age": "32.000000", 2024-09-17T07:04:02.591 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-09-17T07:04:02.591 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_osdmap_full_prune_interval": "2", 2024-09-17T07:04:02.591 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_osdmap_full_prune_min": "15", 2024-09-17T07:04:02.591 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-09-17T07:04:02.591 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-09-17T07:04:02.592 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_pg_stuck_threshold": "60", 2024-09-17T07:04:02.592 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-09-17T07:04:02.592 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_pg_warn_min_objects": "10000", 2024-09-17T07:04:02.592 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_pg_warn_min_per_osd": "0", 2024-09-17T07:04:02.592 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-09-17T07:04:02.592 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_pool_quota_crit_threshold": "0", 2024-09-17T07:04:02.592 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_pool_quota_warn_threshold": "0", 2024-09-17T07:04:02.592 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_probe_timeout": "2.000000", 2024-09-17T07:04:02.592 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_reweight_max_change": "0.050000", 2024-09-17T07:04:02.592 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_reweight_max_osds": "4", 2024-09-17T07:04:02.593 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-09-17T07:04:02.593 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-09-17T07:04:02.593 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-09-17T07:04:02.593 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-09-17T07:04:02.593 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-09-17T07:04:02.593 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_scrub_interval": "86400", 2024-09-17T07:04:02.593 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_scrub_max_keys": "100", 2024-09-17T07:04:02.593 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_scrub_timeout": "300", 2024-09-17T07:04:02.593 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_session_timeout": "300", 2024-09-17T07:04:02.593 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_smart_report_timeout": "5", 2024-09-17T07:04:02.593 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_stat_smooth_intervals": "6", 2024-09-17T07:04:02.594 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-09-17T07:04:02.594 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_stretch_pool_min_size": "2", 2024-09-17T07:04:02.594 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_stretch_pool_size": "4", 2024-09-17T07:04:02.594 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-09-17T07:04:02.594 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_subscribe_interval": "86400.000000", 2024-09-17T07:04:02.594 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_sync_debug": "false", 2024-09-17T07:04:02.594 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_sync_max_payload_keys": "2000", 2024-09-17T07:04:02.594 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_sync_max_payload_size": "1048576", 2024-09-17T07:04:02.594 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_sync_provider_kill_at": "0", 2024-09-17T07:04:02.594 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_sync_requester_kill_at": "0", 2024-09-17T07:04:02.594 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_sync_timeout": "60.000000", 2024-09-17T07:04:02.595 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_target_pg_per_osd": "100", 2024-09-17T07:04:02.595 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_tick_interval": "5", 2024-09-17T07:04:02.595 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_timecheck_interval": "300.000000", 2024-09-17T07:04:02.595 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-09-17T07:04:02.595 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_use_min_delay_socket": "false", 2024-09-17T07:04:02.595 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_warn_older_version_delay": "604800", 2024-09-17T07:04:02.595 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-09-17T07:04:02.595 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-09-17T07:04:02.595 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-09-17T07:04:02.595 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_warn_on_filestore_osds": "true", 2024-09-17T07:04:02.595 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-09-17T07:04:02.596 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-09-17T07:04:02.596 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-09-17T07:04:02.596 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_warn_on_misplaced": "false", 2024-09-17T07:04:02.596 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-09-17T07:04:02.596 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_warn_on_older_version": "true", 2024-09-17T07:04:02.596 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-09-17T07:04:02.596 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_warn_on_pool_no_app": "true", 2024-09-17T07:04:02.596 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_warn_on_pool_no_redundancy": "true", 2024-09-17T07:04:02.596 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-09-17T07:04:02.596 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-09-17T07:04:02.596 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-09-17T07:04:02.597 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_warn_on_too_few_osds": "false", 2024-09-17T07:04:02.597 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-09-17T07:04:02.597 INFO:teuthology.orchestra.run.smithi100.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-09-17T07:04:02.597 INFO:teuthology.orchestra.run.smithi100.stdout: "monmap": "", 2024-09-17T07:04:02.597 INFO:teuthology.orchestra.run.smithi100.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-17T07:04:02.597 INFO:teuthology.orchestra.run.smithi100.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-09-17T07:04:02.597 INFO:teuthology.orchestra.run.smithi100.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-09-17T07:04:02.597 INFO:teuthology.orchestra.run.smithi100.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-17T07:04:02.597 INFO:teuthology.orchestra.run.smithi100.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-09-17T07:04:02.597 INFO:teuthology.orchestra.run.smithi100.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-09-17T07:04:02.598 INFO:teuthology.orchestra.run.smithi100.stdout: "motr_tracing_enabled": "false", 2024-09-17T07:04:02.598 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_async_op_threads": "3", 2024-09-17T07:04:02.598 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_async_rdma_buffer_size": "131072", 2024-09-17T07:04:02.598 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_async_rdma_cm": "false", 2024-09-17T07:04:02.598 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_async_rdma_device_name": "", 2024-09-17T07:04:02.598 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_async_rdma_dscp": "96", 2024-09-17T07:04:02.598 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-09-17T07:04:02.598 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_async_rdma_gid_idx": "0", 2024-09-17T07:04:02.598 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_async_rdma_local_gid": "", 2024-09-17T07:04:02.598 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_async_rdma_polling_us": "1000", 2024-09-17T07:04:02.598 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_async_rdma_port_num": "1", 2024-09-17T07:04:02.599 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-09-17T07:04:02.599 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-09-17T07:04:02.599 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_async_rdma_roce_ver": "1", 2024-09-17T07:04:02.599 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_async_rdma_send_buffers": "1024", 2024-09-17T07:04:02.599 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_async_rdma_sl": "3", 2024-09-17T07:04:02.599 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_async_rdma_support_srq": "true", 2024-09-17T07:04:02.599 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_async_rdma_type": "ib", 2024-09-17T07:04:02.599 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_async_reap_threshold": "5", 2024-09-17T07:04:02.599 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_bind_before_connect": "false", 2024-09-17T07:04:02.599 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_bind_ipv4": "true", 2024-09-17T07:04:02.599 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_bind_ipv6": "false", 2024-09-17T07:04:02.600 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_bind_msgr1": "false", 2024-09-17T07:04:02.600 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_bind_msgr2": "true", 2024-09-17T07:04:02.600 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_bind_port_max": "7568", 2024-09-17T07:04:02.600 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_bind_port_min": "6800", 2024-09-17T07:04:02.600 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_bind_prefer_ipv4": "false", 2024-09-17T07:04:02.600 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_bind_retry_count": "3", 2024-09-17T07:04:02.600 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_bind_retry_delay": "5", 2024-09-17T07:04:02.600 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_blackhole_client": "false", 2024-09-17T07:04:02.600 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_blackhole_mds": "false", 2024-09-17T07:04:02.600 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_blackhole_mgr": "false", 2024-09-17T07:04:02.601 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_blackhole_mon": "false", 2024-09-17T07:04:02.601 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_blackhole_osd": "false", 2024-09-17T07:04:02.601 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_client_mode": "crc secure", 2024-09-17T07:04:02.601 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_cluster_mode": "crc secure", 2024-09-17T07:04:02.601 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_cluster_type": "", 2024-09-17T07:04:02.601 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_compress_secure": "false", 2024-09-17T07:04:02.601 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_connection_idle_timeout": "900", 2024-09-17T07:04:02.601 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_connection_ready_timeout": "10", 2024-09-17T07:04:02.601 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_crc_data": "true", 2024-09-17T07:04:02.602 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_crc_header": "true", 2024-09-17T07:04:02.602 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_die_on_bad_msg": "false", 2024-09-17T07:04:02.602 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_die_on_bug": "true", 2024-09-17T07:04:02.602 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_die_on_old_message": "true", 2024-09-17T07:04:02.602 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_die_on_skipped_message": "false", 2024-09-17T07:04:02.602 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_die_on_unhandled_msg": "false", 2024-09-17T07:04:02.602 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-09-17T07:04:02.602 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_dpdk_coremask": "0xF", 2024-09-17T07:04:02.602 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-09-17T07:04:02.602 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_dpdk_devs_allowlist": "", 2024-09-17T07:04:02.603 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_dpdk_enable_tso": "true", 2024-09-17T07:04:02.603 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-09-17T07:04:02.603 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-09-17T07:04:02.603 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_dpdk_hugepages": "", 2024-09-17T07:04:02.603 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_dpdk_hw_flow_control": "true", 2024-09-17T07:04:02.603 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-09-17T07:04:02.603 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_dpdk_lro": "true", 2024-09-17T07:04:02.603 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_dpdk_memory_channel": "4", 2024-09-17T07:04:02.603 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-09-17T07:04:02.603 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_dpdk_pmd": "", 2024-09-17T07:04:02.603 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_dpdk_port_id": "0", 2024-09-17T07:04:02.603 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-09-17T07:04:02.604 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_dump_corrupt_message_level": "1", 2024-09-17T07:04:02.604 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_dump_on_send": "false", 2024-09-17T07:04:02.604 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_initial_backoff": "0.200000", 2024-09-17T07:04:02.604 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_inject_delay_max": "1.000000", 2024-09-17T07:04:02.604 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_inject_delay_probability": "0.000000", 2024-09-17T07:04:02.604 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_inject_delay_type": "", 2024-09-17T07:04:02.604 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_inject_internal_delays": "0.000000", 2024-09-17T07:04:02.604 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_inject_network_congestion": "0", 2024-09-17T07:04:02.604 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_inject_socket_failures": "0", 2024-09-17T07:04:02.604 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_learn_addr_from_peer": "true", 2024-09-17T07:04:02.605 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_max_accept_failures": "4", 2024-09-17T07:04:02.605 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_max_backoff": "15.000000", 2024-09-17T07:04:02.605 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_mon_client_mode": "secure crc", 2024-09-17T07:04:02.605 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_mon_cluster_mode": "secure crc", 2024-09-17T07:04:02.605 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_mon_service_mode": "secure crc", 2024-09-17T07:04:02.605 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_osd_compress_min_size": "1024", 2024-09-17T07:04:02.605 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_osd_compress_mode": "none", 2024-09-17T07:04:02.605 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_osd_compression_algorithm": "snappy", 2024-09-17T07:04:02.605 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-09-17T07:04:02.605 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_pq_min_cost": "65536", 2024-09-17T07:04:02.605 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_public_type": "", 2024-09-17T07:04:02.606 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_service_mode": "crc secure", 2024-09-17T07:04:02.606 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_tcp_listen_backlog": "512", 2024-09-17T07:04:02.606 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_tcp_nodelay": "true", 2024-09-17T07:04:02.606 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-09-17T07:04:02.606 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_tcp_rcvbuf": "0", 2024-09-17T07:04:02.606 INFO:teuthology.orchestra.run.smithi100.stdout: "ms_type": "async+posix", 2024-09-17T07:04:02.606 INFO:teuthology.orchestra.run.smithi100.stdout: "no_config_file": "false", 2024-09-17T07:04:02.606 INFO:teuthology.orchestra.run.smithi100.stdout: "objecter_completion_locks_per_session": "32", 2024-09-17T07:04:02.606 INFO:teuthology.orchestra.run.smithi100.stdout: "objecter_debug_inject_relock_delay": "false", 2024-09-17T07:04:02.606 INFO:teuthology.orchestra.run.smithi100.stdout: "objecter_inflight_op_bytes": "104857600", 2024-09-17T07:04:02.606 INFO:teuthology.orchestra.run.smithi100.stdout: "objecter_inflight_ops": "1024", 2024-09-17T07:04:02.607 INFO:teuthology.orchestra.run.smithi100.stdout: "objecter_inject_no_watch_ping": "false", 2024-09-17T07:04:02.607 INFO:teuthology.orchestra.run.smithi100.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-09-17T07:04:02.607 INFO:teuthology.orchestra.run.smithi100.stdout: "objecter_tick_interval": "5.000000", 2024-09-17T07:04:02.607 INFO:teuthology.orchestra.run.smithi100.stdout: "objecter_timeout": "10.000000", 2024-09-17T07:04:02.607 INFO:teuthology.orchestra.run.smithi100.stdout: "objectstore_blackhole": "false", 2024-09-17T07:04:02.607 INFO:teuthology.orchestra.run.smithi100.stdout: "openssl_engine_opts": "", 2024-09-17T07:04:02.607 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_agent_delay_time": "5.000000", 2024-09-17T07:04:02.607 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_agent_hist_halflife": "1000", 2024-09-17T07:04:02.607 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_agent_max_low_ops": "2", 2024-09-17T07:04:02.607 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_agent_max_ops": "4", 2024-09-17T07:04:02.607 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-09-17T07:04:02.608 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_agent_quantize_effort": "0.100000", 2024-09-17T07:04:02.608 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_agent_slop": "0.020000", 2024-09-17T07:04:02.608 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-09-17T07:04:02.608 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_allow_recovery_below_min_size": "true", 2024-09-17T07:04:02.608 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_asio_thread_count": "2", 2024-09-17T07:04:02.608 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_async_recovery_min_cost": "100", 2024-09-17T07:04:02.608 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_auto_mark_unfound_lost": "false", 2024-09-17T07:04:02.608 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_backfill_retry_interval": "30.000000", 2024-09-17T07:04:02.608 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_backfill_scan_max": "512", 2024-09-17T07:04:02.608 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_backfill_scan_min": "64", 2024-09-17T07:04:02.608 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_backoff_on_degraded": "false", 2024-09-17T07:04:02.609 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_backoff_on_peering": "false", 2024-09-17T07:04:02.609 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_backoff_on_unfound": "true", 2024-09-17T07:04:02.609 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_beacon_report_interval": "300", 2024-09-17T07:04:02.609 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_bench_duration": "30", 2024-09-17T07:04:02.609 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-09-17T07:04:02.609 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_bench_max_block_size": "67108864", 2024-09-17T07:04:02.609 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_bench_small_size_max_iops": "100", 2024-09-17T07:04:02.609 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_blkin_trace_all": "false", 2024-09-17T07:04:02.609 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_blocked_scrub_grace_period": "120", 2024-09-17T07:04:02.609 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-09-17T07:04:02.609 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-09-17T07:04:02.609 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-09-17T07:04:02.610 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_check_for_log_corruption": "false", 2024-09-17T07:04:02.610 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-09-17T07:04:02.610 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-17T07:04:02.610 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-09-17T07:04:02.610 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-17T07:04:02.610 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_class_update_on_start": "true", 2024-09-17T07:04:02.610 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_client_message_cap": "256", 2024-09-17T07:04:02.610 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_client_message_size_cap": "524288000", 2024-09-17T07:04:02.610 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_client_op_priority": "63", 2024-09-17T07:04:02.610 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_client_watch_timeout": "30", 2024-09-17T07:04:02.610 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_command_max_records": "256", 2024-09-17T07:04:02.611 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_compact_on_start": "false", 2024-09-17T07:04:02.611 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-09-17T07:04:02.611 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_crush_chooseleaf_type": "1", 2024-09-17T07:04:02.611 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_crush_initial_weight": "-1.000000", 2024-09-17T07:04:02.611 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_crush_update_on_start": "true", 2024-09-17T07:04:02.611 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_crush_update_weight_set": "true", 2024-09-17T07:04:02.611 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-09-17T07:04:02.611 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-09-17T07:04:02.611 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-09-17T07:04:02.611 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_debug_drop_ping_duration": "0", 2024-09-17T07:04:02.611 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-09-17T07:04:02.612 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_debug_feed_pullee": "-1", 2024-09-17T07:04:02.612 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-09-17T07:04:02.612 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-09-17T07:04:02.612 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-09-17T07:04:02.612 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_debug_misdirected_ops": "false", 2024-09-17T07:04:02.612 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_debug_no_acting_change": "false", 2024-09-17T07:04:02.612 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_debug_no_purge_strays": "false", 2024-09-17T07:04:02.612 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_debug_op_order": "false", 2024-09-17T07:04:02.612 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_debug_pg_log_writeout": "false", 2024-09-17T07:04:02.612 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_debug_pretend_recovery_active": "false", 2024-09-17T07:04:02.612 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-09-17T07:04:02.613 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-09-17T07:04:02.613 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_debug_shutdown": "false", 2024-09-17T07:04:02.613 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-09-17T07:04:02.613 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-09-17T07:04:02.614 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_debug_verify_cached_snaps": "false", 2024-09-17T07:04:02.614 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_debug_verify_missing_on_start": "false", 2024-09-17T07:04:02.614 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_debug_verify_snaps": "false", 2024-09-17T07:04:02.614 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-09-17T07:04:02.614 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-09-17T07:04:02.614 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_deep_scrub_keys": "1024", 2024-09-17T07:04:02.614 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-09-17T07:04:02.614 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-09-17T07:04:02.614 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-09-17T07:04:02.614 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_deep_scrub_stride": "524288", 2024-09-17T07:04:02.615 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-09-17T07:04:02.615 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_default_data_pool_replay_window": "45", 2024-09-17T07:04:02.615 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_default_notify_timeout": "30", 2024-09-17T07:04:02.615 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_delete_sleep": "0.000000", 2024-09-17T07:04:02.615 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-09-17T07:04:02.615 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-09-17T07:04:02.615 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-09-17T07:04:02.615 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_discard_disconnected_ops": "true", 2024-09-17T07:04:02.615 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_enable_op_tracker": "true", 2024-09-17T07:04:02.615 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-09-17T07:04:02.615 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_extblkdev_plugins": "vdo", 2024-09-17T07:04:02.615 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-09-17T07:04:02.615 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-09-17T07:04:02.616 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_fast_info": "true", 2024-09-17T07:04:02.616 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_fast_shutdown": "true", 2024-09-17T07:04:02.616 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-09-17T07:04:02.616 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_fast_shutdown_timeout": "15", 2024-09-17T07:04:02.616 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-09-17T07:04:02.616 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-09-17T07:04:02.616 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-09-17T07:04:02.616 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_function_tracing": "false", 2024-09-17T07:04:02.616 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_heartbeat_grace": "20", 2024-09-17T07:04:02.616 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_heartbeat_interval": "6", 2024-09-17T07:04:02.616 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-09-17T07:04:02.616 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_heartbeat_min_peers": "10", 2024-09-17T07:04:02.617 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_heartbeat_min_size": "2000", 2024-09-17T07:04:02.617 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_heartbeat_stale": "600", 2024-09-17T07:04:02.617 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-09-17T07:04:02.617 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_hit_set_max_size": "100000", 2024-09-17T07:04:02.617 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_hit_set_min_size": "1000", 2024-09-17T07:04:02.617 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-09-17T07:04:02.617 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-09-17T07:04:02.617 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-09-17T07:04:02.617 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-09-17T07:04:02.617 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-09-17T07:04:02.617 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_journal_flush_on_shutdown": "true", 2024-09-17T07:04:02.617 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_journal_size": "5120", 2024-09-17T07:04:02.618 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_kill_backfill_at": "0", 2024-09-17T07:04:02.618 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_loop_before_reset_tphandle": "64", 2024-09-17T07:04:02.618 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_map_cache_size": "50", 2024-09-17T07:04:02.618 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_map_dedup": "true", 2024-09-17T07:04:02.618 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_map_message_max": "40", 2024-09-17T07:04:02.618 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_map_message_max_bytes": "10485760", 2024-09-17T07:04:02.618 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_map_share_max_epochs": "40", 2024-09-17T07:04:02.618 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_max_attr_name_len": "100", 2024-09-17T07:04:02.618 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_max_attr_size": "0", 2024-09-17T07:04:02.618 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_max_backfills": "1", 2024-09-17T07:04:02.618 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_max_markdown_count": "5", 2024-09-17T07:04:02.618 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_max_markdown_period": "600", 2024-09-17T07:04:02.619 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_max_object_name_len": "2048", 2024-09-17T07:04:02.619 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_max_object_namespace_len": "256", 2024-09-17T07:04:02.619 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_max_object_size": "134217728", 2024-09-17T07:04:02.619 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-09-17T07:04:02.619 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_max_omap_entries_per_request": "1024", 2024-09-17T07:04:02.619 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_max_pg_blocked_by": "16", 2024-09-17T07:04:02.619 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_max_pg_log_entries": "10000", 2024-09-17T07:04:02.619 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-09-17T07:04:02.619 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_max_pgls": "1024", 2024-09-17T07:04:02.619 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_max_push_cost": "8388608", 2024-09-17T07:04:02.619 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_max_push_objects": "10", 2024-09-17T07:04:02.619 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_max_scrubs": "3", 2024-09-17T07:04:02.620 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-09-17T07:04:02.620 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_max_trimming_pgs": "2", 2024-09-17T07:04:02.620 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_max_write_op_reply_len": "64", 2024-09-17T07:04:02.620 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_max_write_size": "90", 2024-09-17T07:04:02.620 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-09-17T07:04:02.620 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-09-17T07:04:02.620 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-09-17T07:04:02.620 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-09-17T07:04:02.620 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-09-17T07:04:02.620 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-09-17T07:04:02.620 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-09-17T07:04:02.620 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_mclock_override_recovery_settings": "false", 2024-09-17T07:04:02.620 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_mclock_profile": "balanced", 2024-09-17T07:04:02.621 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-09-17T07:04:02.621 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-09-17T07:04:02.621 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-09-17T07:04:02.621 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-09-17T07:04:02.621 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-09-17T07:04:02.621 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-09-17T07:04:02.621 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-09-17T07:04:02.621 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-09-17T07:04:02.621 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-09-17T07:04:02.621 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-09-17T07:04:02.621 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_mclock_skip_benchmark": "false", 2024-09-17T07:04:02.621 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_memory_base": "805306368", 2024-09-17T07:04:02.622 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_memory_cache_min": "134217728", 2024-09-17T07:04:02.622 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-09-17T07:04:02.622 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-09-17T07:04:02.622 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_memory_target": "4294967296", 2024-09-17T07:04:02.622 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_memory_target_autotune": "false", 2024-09-17T07:04:02.622 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-09-17T07:04:02.622 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_min_pg_log_entries": "250", 2024-09-17T07:04:02.622 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_min_recovery_priority": "0", 2024-09-17T07:04:02.622 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_mon_heartbeat_interval": "30", 2024-09-17T07:04:02.622 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-09-17T07:04:02.622 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_mon_report_interval": "5", 2024-09-17T07:04:02.622 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_mon_report_max_in_flight": "2", 2024-09-17T07:04:02.623 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-09-17T07:04:02.623 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_num_cache_shards": "32", 2024-09-17T07:04:02.623 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_num_op_tracker_shard": "32", 2024-09-17T07:04:02.623 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_numa_auto_affinity": "true", 2024-09-17T07:04:02.623 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_numa_node": "-1", 2024-09-17T07:04:02.623 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_numa_prefer_iface": "true", 2024-09-17T07:04:02.623 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-09-17T07:04:02.623 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_objecter_finishers": "1", 2024-09-17T07:04:02.623 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_objectstore": "bluestore", 2024-09-17T07:04:02.623 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_objectstore_fuse": "false", 2024-09-17T07:04:02.623 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_objectstore_tracing": "false", 2024-09-17T07:04:02.623 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_op_complaint_time": "30.000000", 2024-09-17T07:04:02.623 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_op_history_duration": "600", 2024-09-17T07:04:02.624 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_op_history_size": "20", 2024-09-17T07:04:02.624 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_op_history_slow_op_size": "20", 2024-09-17T07:04:02.624 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-09-17T07:04:02.624 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_op_log_threshold": "5", 2024-09-17T07:04:02.624 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_op_num_shards": "0", 2024-09-17T07:04:02.624 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_op_num_shards_hdd": "5", 2024-09-17T07:04:02.624 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_op_num_shards_ssd": "8", 2024-09-17T07:04:02.624 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_op_num_threads_per_shard": "0", 2024-09-17T07:04:02.624 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-09-17T07:04:02.624 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-09-17T07:04:02.624 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-09-17T07:04:02.624 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_op_pq_min_cost": "65536", 2024-09-17T07:04:02.625 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_op_queue": "mclock_scheduler", 2024-09-17T07:04:02.625 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_op_queue_cut_off": "high", 2024-09-17T07:04:02.625 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_op_thread_suicide_timeout": "150", 2024-09-17T07:04:02.625 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_op_thread_timeout": "15", 2024-09-17T07:04:02.625 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_open_classes_on_start": "true", 2024-09-17T07:04:02.625 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_os_flags": "0", 2024-09-17T07:04:02.625 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_peering_op_priority": "255", 2024-09-17T07:04:02.625 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_pg_delete_cost": "1048576", 2024-09-17T07:04:02.625 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_pg_delete_priority": "5", 2024-09-17T07:04:02.625 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-09-17T07:04:02.625 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-09-17T07:04:02.626 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_pg_log_dups_tracked": "3000", 2024-09-17T07:04:02.626 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_pg_log_trim_max": "10000", 2024-09-17T07:04:02.626 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_pg_log_trim_min": "100", 2024-09-17T07:04:02.626 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-09-17T07:04:02.626 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_pg_object_context_cache_count": "64", 2024-09-17T07:04:02.626 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_pg_stat_report_interval_max": "500", 2024-09-17T07:04:02.626 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-09-17T07:04:02.626 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-09-17T07:04:02.626 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-09-17T07:04:02.626 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-09-17T07:04:02.626 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-09-17T07:04:02.626 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-09-17T07:04:02.626 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_pool_default_crimson": "false", 2024-09-17T07:04:02.627 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_pool_default_crush_rule": "-1", 2024-09-17T07:04:02.627 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_pool_default_ec_fast_read": "false", 2024-09-17T07:04:02.627 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-09-17T07:04:02.627 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_pool_default_flag_bulk": "false", 2024-09-17T07:04:02.627 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-09-17T07:04:02.627 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_pool_default_flag_nodelete": "false", 2024-09-17T07:04:02.627 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-09-17T07:04:02.627 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-09-17T07:04:02.627 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_pool_default_flags": "0", 2024-09-17T07:04:02.627 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-09-17T07:04:02.627 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_pool_default_min_size": "0", 2024-09-17T07:04:02.627 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-09-17T07:04:02.627 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_pool_default_pg_num": "32", 2024-09-17T07:04:02.628 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_pool_default_pgp_num": "0", 2024-09-17T07:04:02.628 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-09-17T07:04:02.628 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_pool_default_size": "3", 2024-09-17T07:04:02.628 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_pool_default_type": "replicated", 2024-09-17T07:04:02.628 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-09-17T07:04:02.628 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_pool_use_gmt_hitset": "true", 2024-09-17T07:04:02.628 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_push_per_object_cost": "1000", 2024-09-17T07:04:02.628 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_read_ec_check_for_errors": "false", 2024-09-17T07:04:02.628 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_recover_clone_overlap": "true", 2024-09-17T07:04:02.628 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_recover_clone_overlap_limit": "10", 2024-09-17T07:04:02.628 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_recovery_cost": "20971520", 2024-09-17T07:04:02.628 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_recovery_delay_start": "0.000000", 2024-09-17T07:04:02.628 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_recovery_max_active": "0", 2024-09-17T07:04:02.629 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_recovery_max_active_hdd": "3", 2024-09-17T07:04:02.629 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_recovery_max_active_ssd": "10", 2024-09-17T07:04:02.629 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_recovery_max_chunk": "8388608", 2024-09-17T07:04:02.629 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-09-17T07:04:02.629 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_recovery_max_single_start": "1", 2024-09-17T07:04:02.629 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_recovery_op_priority": "3", 2024-09-17T07:04:02.629 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_recovery_op_warn_multiple": "16", 2024-09-17T07:04:02.629 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_recovery_priority": "5", 2024-09-17T07:04:02.629 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_recovery_retry_interval": "30.000000", 2024-09-17T07:04:02.629 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_recovery_sleep": "0.000000", 2024-09-17T07:04:02.629 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-09-17T07:04:02.629 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-09-17T07:04:02.630 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-09-17T07:04:02.630 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_repair_during_recovery": "false", 2024-09-17T07:04:02.630 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_requested_scrub_priority": "120", 2024-09-17T07:04:02.630 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-09-17T07:04:02.630 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_rollback_to_cluster_snap": "", 2024-09-17T07:04:02.630 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_scrub_auto_repair": "false", 2024-09-17T07:04:02.630 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-09-17T07:04:02.630 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-09-17T07:04:02.630 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_scrub_begin_hour": "0", 2024-09-17T07:04:02.630 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_scrub_begin_week_day": "0", 2024-09-17T07:04:02.630 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_scrub_chunk_max": "25", 2024-09-17T07:04:02.630 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_scrub_chunk_min": "5", 2024-09-17T07:04:02.630 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_scrub_cost": "52428800", 2024-09-17T07:04:02.631 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_scrub_during_recovery": "false", 2024-09-17T07:04:02.631 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_scrub_end_hour": "0", 2024-09-17T07:04:02.631 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_scrub_end_week_day": "0", 2024-09-17T07:04:02.631 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_scrub_event_cost": "4096", 2024-09-17T07:04:02.631 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-09-17T07:04:02.631 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-09-17T07:04:02.631 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_scrub_invalid_stats": "true", 2024-09-17T07:04:02.631 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_scrub_load_threshold": "0.500000", 2024-09-17T07:04:02.631 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_scrub_max_interval": "604800.000000", 2024-09-17T07:04:02.631 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_scrub_max_preemptions": "5", 2024-09-17T07:04:02.631 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_scrub_min_interval": "86400.000000", 2024-09-17T07:04:02.631 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_scrub_priority": "5", 2024-09-17T07:04:02.632 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_scrub_reservation_timeout": "5000", 2024-09-17T07:04:02.632 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_scrub_sleep": "0.000000", 2024-09-17T07:04:02.632 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_scrub_slow_reservation_response": "2200", 2024-09-17T07:04:02.632 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-09-17T07:04:02.632 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-09-17T07:04:02.632 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_shutdown_pgref_assert": "false", 2024-09-17T07:04:02.632 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_skip_data_digest": "false", 2024-09-17T07:04:02.632 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_smart_report_timeout": "5", 2024-09-17T07:04:02.632 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_snap_trim_cost": "1048576", 2024-09-17T07:04:02.632 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_snap_trim_priority": "5", 2024-09-17T07:04:02.632 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_snap_trim_sleep": "0.000000", 2024-09-17T07:04:02.633 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-09-17T07:04:02.633 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-09-17T07:04:02.633 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-09-17T07:04:02.633 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-09-17T07:04:02.633 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_stats_update_period_scrubbing": "15", 2024-09-17T07:04:02.633 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-09-17T07:04:02.633 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_target_transaction_size": "30", 2024-09-17T07:04:02.633 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-09-17T07:04:02.633 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-09-17T07:04:02.633 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-09-17T07:04:02.633 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-09-17T07:04:02.633 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-09-17T07:04:02.634 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-09-17T07:04:02.634 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-09-17T07:04:02.634 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_tier_default_cache_mode": "writeback", 2024-09-17T07:04:02.634 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-09-17T07:04:02.634 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-09-17T07:04:02.634 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_tracing": "false", 2024-09-17T07:04:02.634 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_use_stale_snap": "false", 2024-09-17T07:04:02.634 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-09-17T07:04:02.634 INFO:teuthology.orchestra.run.smithi100.stdout: "osdc_blkin_trace_all": "false", 2024-09-17T07:04:02.634 INFO:teuthology.orchestra.run.smithi100.stdout: "paxos_kill_at": "0", 2024-09-17T07:04:02.634 INFO:teuthology.orchestra.run.smithi100.stdout: "paxos_max_join_drift": "10", 2024-09-17T07:04:02.634 INFO:teuthology.orchestra.run.smithi100.stdout: "paxos_min": "500", 2024-09-17T07:04:02.634 INFO:teuthology.orchestra.run.smithi100.stdout: "paxos_min_wait": "0.050000", 2024-09-17T07:04:02.634 INFO:teuthology.orchestra.run.smithi100.stdout: "paxos_propose_interval": "1.000000", 2024-09-17T07:04:02.635 INFO:teuthology.orchestra.run.smithi100.stdout: "paxos_service_trim_max": "500", 2024-09-17T07:04:02.635 INFO:teuthology.orchestra.run.smithi100.stdout: "paxos_service_trim_max_multiplier": "20", 2024-09-17T07:04:02.635 INFO:teuthology.orchestra.run.smithi100.stdout: "paxos_service_trim_min": "10", 2024-09-17T07:04:02.635 INFO:teuthology.orchestra.run.smithi100.stdout: "paxos_stash_full_interval": "25", 2024-09-17T07:04:02.635 INFO:teuthology.orchestra.run.smithi100.stdout: "paxos_trim_max": "500", 2024-09-17T07:04:02.635 INFO:teuthology.orchestra.run.smithi100.stdout: "paxos_trim_min": "250", 2024-09-17T07:04:02.635 INFO:teuthology.orchestra.run.smithi100.stdout: "perf": "true", 2024-09-17T07:04:02.635 INFO:teuthology.orchestra.run.smithi100.stdout: "pid_file": "", 2024-09-17T07:04:02.635 INFO:teuthology.orchestra.run.smithi100.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-09-17T07:04:02.635 INFO:teuthology.orchestra.run.smithi100.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-09-17T07:04:02.635 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "-", 2024-09-17T07:04:02.635 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrv": "", 2024-09-17T07:04:02.636 INFO:teuthology.orchestra.run.smithi100.stdout: "public_bind_addr": "-", 2024-09-17T07:04:02.636 INFO:teuthology.orchestra.run.smithi100.stdout: "public_network": "", 2024-09-17T07:04:02.636 INFO:teuthology.orchestra.run.smithi100.stdout: "public_network_interface": "", 2024-09-17T07:04:02.636 INFO:teuthology.orchestra.run.smithi100.stdout: "qat_compressor_enabled": "false", 2024-09-17T07:04:02.636 INFO:teuthology.orchestra.run.smithi100.stdout: "qat_compressor_session_max_number": "256", 2024-09-17T07:04:02.636 INFO:teuthology.orchestra.run.smithi100.stdout: "rados_mon_op_timeout": "0", 2024-09-17T07:04:02.636 INFO:teuthology.orchestra.run.smithi100.stdout: "rados_osd_op_timeout": "0", 2024-09-17T07:04:02.636 INFO:teuthology.orchestra.run.smithi100.stdout: "rados_tracing": "false", 2024-09-17T07:04:02.636 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_atime_update_interval": "60", 2024-09-17T07:04:02.636 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-09-17T07:04:02.636 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_balance_parent_reads": "false", 2024-09-17T07:04:02.636 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_balance_snap_reads": "false", 2024-09-17T07:04:02.636 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_blkin_trace_all": "false", 2024-09-17T07:04:02.637 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_blocklist_expire_seconds": "0", 2024-09-17T07:04:02.637 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_blocklist_on_break_lock": "true", 2024-09-17T07:04:02.637 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_cache": "true", 2024-09-17T07:04:02.637 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_cache_block_writes_upfront": "false", 2024-09-17T07:04:02.637 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_cache_max_dirty": "25165824", 2024-09-17T07:04:02.637 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-09-17T07:04:02.637 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_cache_max_dirty_object": "0", 2024-09-17T07:04:02.637 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_cache_policy": "writearound", 2024-09-17T07:04:02.637 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_cache_size": "33554432", 2024-09-17T07:04:02.637 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_cache_target_dirty": "16777216", 2024-09-17T07:04:02.637 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-09-17T07:04:02.637 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_clone_copy_on_read": "false", 2024-09-17T07:04:02.637 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_compression_hint": "none", 2024-09-17T07:04:02.638 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_concurrent_management_ops": "10", 2024-09-17T07:04:02.638 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-09-17T07:04:02.638 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_default_clone_format": "auto", 2024-09-17T07:04:02.638 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_default_data_pool": "", 2024-09-17T07:04:02.638 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_default_features": "61", 2024-09-17T07:04:02.638 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_default_format": "2", 2024-09-17T07:04:02.638 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_default_map_options": "", 2024-09-17T07:04:02.638 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_default_order": "22", 2024-09-17T07:04:02.638 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_default_pool": "rbd", 2024-09-17T07:04:02.638 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-09-17T07:04:02.638 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_default_stripe_count": "0", 2024-09-17T07:04:02.638 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_default_stripe_unit": "0", 2024-09-17T07:04:02.638 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_disable_zero_copy_writes": "true", 2024-09-17T07:04:02.639 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_discard_granularity_bytes": "65536", 2024-09-17T07:04:02.639 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-09-17T07:04:02.639 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_enable_alloc_hint": "true", 2024-09-17T07:04:02.639 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-09-17T07:04:02.639 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_io_scheduler": "simple", 2024-09-17T07:04:02.639 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-09-17T07:04:02.639 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_journal_commit_age": "5.000000", 2024-09-17T07:04:02.639 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-09-17T07:04:02.639 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-09-17T07:04:02.639 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-09-17T07:04:02.639 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-09-17T07:04:02.639 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_journal_object_flush_interval": "0", 2024-09-17T07:04:02.640 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-09-17T07:04:02.640 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-09-17T07:04:02.640 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_journal_order": "24", 2024-09-17T07:04:02.640 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_journal_pool": "", 2024-09-17T07:04:02.640 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_journal_splay_width": "4", 2024-09-17T07:04:02.640 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_localize_parent_reads": "false", 2024-09-17T07:04:02.640 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_localize_snap_reads": "false", 2024-09-17T07:04:02.640 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-09-17T07:04:02.640 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-09-17T07:04:02.640 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-09-17T07:04:02.640 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-09-17T07:04:02.640 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-09-17T07:04:02.641 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-09-17T07:04:02.641 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_mirror_image_policy_type": "simple", 2024-09-17T07:04:02.641 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-09-17T07:04:02.641 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-09-17T07:04:02.641 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-09-17T07:04:02.641 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-09-17T07:04:02.641 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-09-17T07:04:02.641 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-09-17T07:04:02.641 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-09-17T07:04:02.641 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_mirror_memory_autotune": "true", 2024-09-17T07:04:02.641 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_mirror_memory_base": "805306368", 2024-09-17T07:04:02.641 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-09-17T07:04:02.641 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-09-17T07:04:02.641 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-09-17T07:04:02.642 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-09-17T07:04:02.642 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_mirror_memory_target": "4294967296", 2024-09-17T07:04:02.642 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-09-17T07:04:02.642 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-09-17T07:04:02.642 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-09-17T07:04:02.642 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_mirroring_delete_delay": "0", 2024-09-17T07:04:02.642 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-09-17T07:04:02.642 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_mirroring_replay_delay": "0", 2024-09-17T07:04:02.642 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-09-17T07:04:02.642 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-09-17T07:04:02.642 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_move_to_trash_on_remove": "false", 2024-09-17T07:04:02.642 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-09-17T07:04:02.642 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_mtime_update_interval": "60", 2024-09-17T07:04:02.643 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_non_blocking_aio": "true", 2024-09-17T07:04:02.643 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_op_thread_timeout": "60", 2024-09-17T07:04:02.643 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_op_threads": "1", 2024-09-17T07:04:02.643 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_parent_cache_enabled": "false", 2024-09-17T07:04:02.643 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_persistent_cache_mode": "disabled", 2024-09-17T07:04:02.643 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_persistent_cache_path": "/tmp", 2024-09-17T07:04:02.643 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_persistent_cache_size": "1073741824", 2024-09-17T07:04:02.643 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_plugins": "", 2024-09-17T07:04:02.643 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_qos_bps_burst": "0", 2024-09-17T07:04:02.643 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-09-17T07:04:02.643 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_qos_bps_limit": "0", 2024-09-17T07:04:02.643 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_qos_exclude_ops": "0", 2024-09-17T07:04:02.644 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_qos_iops_burst": "0", 2024-09-17T07:04:02.644 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-09-17T07:04:02.644 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_qos_iops_limit": "0", 2024-09-17T07:04:02.644 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_qos_read_bps_burst": "0", 2024-09-17T07:04:02.644 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-09-17T07:04:02.644 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_qos_read_bps_limit": "0", 2024-09-17T07:04:02.644 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_qos_read_iops_burst": "0", 2024-09-17T07:04:02.644 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-09-17T07:04:02.644 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_qos_read_iops_limit": "0", 2024-09-17T07:04:02.644 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_qos_schedule_tick_min": "50", 2024-09-17T07:04:02.644 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_qos_write_bps_burst": "0", 2024-09-17T07:04:02.644 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-09-17T07:04:02.645 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_qos_write_bps_limit": "0", 2024-09-17T07:04:02.645 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_qos_write_iops_burst": "0", 2024-09-17T07:04:02.645 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-09-17T07:04:02.645 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_qos_write_iops_limit": "0", 2024-09-17T07:04:02.645 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_quiesce_notification_attempts": "10", 2024-09-17T07:04:02.645 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_read_from_replica_policy": "default", 2024-09-17T07:04:02.645 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-09-17T07:04:02.645 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_readahead_max_bytes": "524288", 2024-09-17T07:04:02.645 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_readahead_trigger_requests": "10", 2024-09-17T07:04:02.645 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_request_timed_out_seconds": "30", 2024-09-17T07:04:02.645 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_skip_partial_discard": "true", 2024-09-17T07:04:02.645 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-09-17T07:04:02.645 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_tracing": "false", 2024-09-17T07:04:02.646 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_validate_names": "true", 2024-09-17T07:04:02.646 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd_validate_pool": "true", 2024-09-17T07:04:02.646 INFO:teuthology.orchestra.run.smithi100.stdout: "restapi_base_url": "", 2024-09-17T07:04:02.646 INFO:teuthology.orchestra.run.smithi100.stdout: "restapi_log_level": "", 2024-09-17T07:04:02.646 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_acl_grants_max_num": "100", 2024-09-17T07:04:02.646 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_admin_entry": "admin", 2024-09-17T07:04:02.646 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-09-17T07:04:02.646 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_backend_store": "rados", 2024-09-17T07:04:02.646 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_barbican_url": "", 2024-09-17T07:04:02.646 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_beast_enable_async": "true", 2024-09-17T07:04:02.646 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-09-17T07:04:02.646 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-09-17T07:04:02.647 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_bucket_index_max_aio": "128", 2024-09-17T07:04:02.647 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-09-17T07:04:02.647 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-09-17T07:04:02.647 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_bucket_quota_ttl": "600", 2024-09-17T07:04:02.647 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-09-17T07:04:02.647 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_cache_enabled": "true", 2024-09-17T07:04:02.647 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_cache_expiry_interval": "900", 2024-09-17T07:04:02.647 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_cache_lru_size": "10000", 2024-09-17T07:04:02.647 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_config_store": "rados", 2024-09-17T07:04:02.647 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_content_length_compat": "false", 2024-09-17T07:04:02.647 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_copy_obj_progress": "true", 2024-09-17T07:04:02.647 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-09-17T07:04:02.647 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_cors_rules_max_num": "100", 2024-09-17T07:04:02.648 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_cross_domain_policy": "", 2024-09-17T07:04:02.648 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_crypt_default_encryption_key": "", 2024-09-17T07:04:02.648 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_crypt_kmip_addr": "", 2024-09-17T07:04:02.648 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_crypt_kmip_ca_path": "", 2024-09-17T07:04:02.648 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_crypt_kmip_client_cert": "", 2024-09-17T07:04:02.648 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_crypt_kmip_client_key": "", 2024-09-17T07:04:02.648 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-09-17T07:04:02.648 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_crypt_kmip_password": "", 2024-09-17T07:04:02.648 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-09-17T07:04:02.648 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_crypt_kmip_username": "", 2024-09-17T07:04:02.648 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_crypt_require_ssl": "true", 2024-09-17T07:04:02.648 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-09-17T07:04:02.649 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-09-17T07:04:02.649 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-09-17T07:04:02.649 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-09-17T07:04:02.649 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-09-17T07:04:02.649 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-09-17T07:04:02.649 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-09-17T07:04:02.649 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-09-17T07:04:02.649 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-09-17T07:04:02.649 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-09-17T07:04:02.649 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-09-17T07:04:02.649 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-09-17T07:04:02.649 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-09-17T07:04:02.649 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-09-17T07:04:02.650 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_crypt_suppress_logs": "true", 2024-09-17T07:04:02.650 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_crypt_vault_addr": "", 2024-09-17T07:04:02.650 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_crypt_vault_auth": "token", 2024-09-17T07:04:02.650 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_crypt_vault_namespace": "", 2024-09-17T07:04:02.650 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_crypt_vault_prefix": "", 2024-09-17T07:04:02.650 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-09-17T07:04:02.650 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-09-17T07:04:02.650 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-09-17T07:04:02.650 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-09-17T07:04:02.650 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_crypt_vault_token_file": "", 2024-09-17T07:04:02.650 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-09-17T07:04:02.650 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_curl_buffersize": "524288", 2024-09-17T07:04:02.650 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_curl_low_speed_limit": "1024", 2024-09-17T07:04:02.651 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_curl_low_speed_time": "300", 2024-09-17T07:04:02.651 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_curl_tcp_keepalive": "0", 2024-09-17T07:04:02.651 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-09-17T07:04:02.651 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-09-17T07:04:02.651 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-09-17T07:04:02.651 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-09-17T07:04:02.651 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-09-17T07:04:02.651 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_d3n_l1_fadvise": "4", 2024-09-17T07:04:02.651 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-09-17T07:04:02.651 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-09-17T07:04:02.651 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-09-17T07:04:02.651 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-09-17T07:04:02.652 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_data_log_changes_size": "1000", 2024-09-17T07:04:02.652 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_data_log_num_shards": "128", 2024-09-17T07:04:02.652 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-09-17T07:04:02.652 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_data_log_window": "30", 2024-09-17T07:04:02.652 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_data_notify_interval_msec": "0", 2024-09-17T07:04:02.652 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_data_sync_poll_interval": "20", 2024-09-17T07:04:02.652 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_data_sync_spawn_window": "20", 2024-09-17T07:04:02.652 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-09-17T07:04:02.652 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-09-17T07:04:02.652 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_default_data_log_backing": "fifo", 2024-09-17T07:04:02.652 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-09-17T07:04:02.652 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_default_region_info_oid": "default.region", 2024-09-17T07:04:02.653 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-09-17T07:04:02.653 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-09-17T07:04:02.653 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_defer_to_bucket_acls": "", 2024-09-17T07:04:02.653 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-09-17T07:04:02.653 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-09-17T07:04:02.653 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-09-17T07:04:02.653 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-09-17T07:04:02.653 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-09-17T07:04:02.653 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-09-17T07:04:02.653 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-09-17T07:04:02.653 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-09-17T07:04:02.653 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_dmclock_data_res": "500.000000", 2024-09-17T07:04:02.654 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-09-17T07:04:02.654 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-09-17T07:04:02.654 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-09-17T07:04:02.654 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-09-17T07:04:02.654 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_dns_name": "", 2024-09-17T07:04:02.654 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_dns_s3website_name": "", 2024-09-17T07:04:02.654 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_dynamic_resharding": "true", 2024-09-17T07:04:02.654 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-09-17T07:04:02.654 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_enable_gc_threads": "true", 2024-09-17T07:04:02.654 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_enable_lc_threads": "true", 2024-09-17T07:04:02.654 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_enable_ops_log": "false", 2024-09-17T07:04:02.654 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_enable_quota_threads": "true", 2024-09-17T07:04:02.654 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_enable_static_website": "false", 2024-09-17T07:04:02.655 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_enable_usage_log": "false", 2024-09-17T07:04:02.655 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_enforce_swift_acls": "true", 2024-09-17T07:04:02.655 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_exit_timeout_secs": "120", 2024-09-17T07:04:02.655 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_expose_bucket": "false", 2024-09-17T07:04:02.655 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_extended_http_attrs": "", 2024-09-17T07:04:02.655 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_filter": "none", 2024-09-17T07:04:02.655 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-09-17T07:04:02.655 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_frontends": "beast port=7480", 2024-09-17T07:04:02.655 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_gc_max_concurrent_io": "10", 2024-09-17T07:04:02.655 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_gc_max_deferred": "50", 2024-09-17T07:04:02.655 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-09-17T07:04:02.655 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_gc_max_objs": "32", 2024-09-17T07:04:02.656 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_gc_max_queue_size": "134213632", 2024-09-17T07:04:02.656 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_gc_max_trim_chunk": "16", 2024-09-17T07:04:02.656 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_gc_obj_min_wait": "7200", 2024-09-17T07:04:02.656 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_gc_processor_max_time": "3600", 2024-09-17T07:04:02.656 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_gc_processor_period": "3600", 2024-09-17T07:04:02.656 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-09-17T07:04:02.656 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_get_obj_window_size": "16777216", 2024-09-17T07:04:02.656 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_healthcheck_disabling_path": "", 2024-09-17T07:04:02.656 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_ignore_get_invalid_range": "false", 2024-09-17T07:04:02.656 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_init_timeout": "300", 2024-09-17T07:04:02.656 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-09-17T07:04:02.656 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-09-17T07:04:02.657 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-09-17T07:04:02.657 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-09-17T07:04:02.657 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_keystone_admin_domain": "", 2024-09-17T07:04:02.657 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_keystone_admin_password": "", 2024-09-17T07:04:02.657 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_keystone_admin_password_path": "", 2024-09-17T07:04:02.657 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_keystone_admin_project": "", 2024-09-17T07:04:02.657 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_keystone_admin_tenant": "", 2024-09-17T07:04:02.657 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_keystone_admin_token": "", 2024-09-17T07:04:02.657 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_keystone_admin_token_path": "", 2024-09-17T07:04:02.657 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_keystone_admin_user": "", 2024-09-17T07:04:02.657 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_keystone_api_version": "2", 2024-09-17T07:04:02.657 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_keystone_barbican_domain": "", 2024-09-17T07:04:02.657 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_keystone_barbican_password": "", 2024-09-17T07:04:02.658 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_keystone_barbican_project": "", 2024-09-17T07:04:02.658 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_keystone_barbican_tenant": "", 2024-09-17T07:04:02.658 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_keystone_barbican_user": "", 2024-09-17T07:04:02.658 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-09-17T07:04:02.658 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_keystone_implicit_tenants": "false", 2024-09-17T07:04:02.658 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-09-17T07:04:02.658 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_keystone_service_token_enabled": "false", 2024-09-17T07:04:02.658 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_keystone_token_cache_size": "10000", 2024-09-17T07:04:02.658 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_keystone_url": "", 2024-09-17T07:04:02.658 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_keystone_verify_ssl": "true", 2024-09-17T07:04:02.658 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_lc_debug_interval": "-1", 2024-09-17T07:04:02.658 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_lc_lock_max_time": "90", 2024-09-17T07:04:02.658 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_lc_max_objs": "32", 2024-09-17T07:04:02.659 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_lc_max_rules": "1000", 2024-09-17T07:04:02.659 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_lc_max_worker": "3", 2024-09-17T07:04:02.659 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_lc_max_wp_worker": "3", 2024-09-17T07:04:02.659 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_lc_thread_delay": "0", 2024-09-17T07:04:02.659 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-09-17T07:04:02.659 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_ldap_dnattr": "uid", 2024-09-17T07:04:02.659 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-09-17T07:04:02.659 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_ldap_searchfilter": "", 2024-09-17T07:04:02.659 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-09-17T07:04:02.659 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_ldap_uri": "ldaps://", 2024-09-17T07:04:02.659 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-09-17T07:04:02.659 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-09-17T07:04:02.659 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-09-17T07:04:02.660 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_log_http_headers": "", 2024-09-17T07:04:02.660 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_log_nonexistent_bucket": "false", 2024-09-17T07:04:02.660 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-09-17T07:04:02.660 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_log_object_name_utc": "false", 2024-09-17T07:04:02.660 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_luarocks_location": "/tmp/luarocks", 2024-09-17T07:04:02.660 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_max_attr_name_len": "0", 2024-09-17T07:04:02.660 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_max_attr_size": "0", 2024-09-17T07:04:02.660 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_max_attrs_num_in_req": "0", 2024-09-17T07:04:02.660 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_max_chunk_size": "4194304", 2024-09-17T07:04:02.660 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_max_concurrent_requests": "1024", 2024-09-17T07:04:02.660 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-09-17T07:04:02.660 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_max_dynamic_shards": "1999", 2024-09-17T07:04:02.661 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_max_listing_results": "1000", 2024-09-17T07:04:02.661 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_max_notify_retries": "10", 2024-09-17T07:04:02.661 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_max_objs_per_shard": "100000", 2024-09-17T07:04:02.661 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_max_put_param_size": "1048576", 2024-09-17T07:04:02.661 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_max_put_size": "5368709120", 2024-09-17T07:04:02.661 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_max_slo_entries": "1000", 2024-09-17T07:04:02.661 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_md_log_max_shards": "64", 2024-09-17T07:04:02.661 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_md_notify_interval_msec": "200", 2024-09-17T07:04:02.661 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_meta_sync_poll_interval": "20", 2024-09-17T07:04:02.661 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_meta_sync_spawn_window": "20", 2024-09-17T07:04:02.661 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-09-17T07:04:02.661 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_mp_lock_max_time": "600", 2024-09-17T07:04:02.662 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-09-17T07:04:02.662 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_multipart_min_part_size": "5242880", 2024-09-17T07:04:02.662 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-09-17T07:04:02.662 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-09-17T07:04:02.662 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_nfs_fhcache_size": "2017", 2024-09-17T07:04:02.662 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-09-17T07:04:02.662 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-09-17T07:04:02.662 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_nfs_lru_lanes": "5", 2024-09-17T07:04:02.662 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_nfs_max_gc": "300", 2024-09-17T07:04:02.662 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-09-17T07:04:02.662 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_nfs_run_gc_threads": "false", 2024-09-17T07:04:02.662 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_nfs_run_lc_threads": "false", 2024-09-17T07:04:02.662 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_nfs_run_quota_threads": "false", 2024-09-17T07:04:02.663 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_nfs_run_sync_thread": "false", 2024-09-17T07:04:02.663 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-09-17T07:04:02.663 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-09-17T07:04:02.663 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_num_async_rados_threads": "32", 2024-09-17T07:04:02.663 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_num_control_oids": "8", 2024-09-17T07:04:02.663 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_numa_node": "-1", 2024-09-17T07:04:02.663 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_obj_stripe_size": "4194304", 2024-09-17T07:04:02.663 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-09-17T07:04:02.663 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_objexp_chunk_size": "100", 2024-09-17T07:04:02.663 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_objexp_gc_interval": "600", 2024-09-17T07:04:02.663 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_objexp_hints_num_shards": "127", 2024-09-17T07:04:02.663 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-09-17T07:04:02.663 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-09-17T07:04:02.664 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_op_thread_timeout": "600", 2024-09-17T07:04:02.664 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_op_tracing": "false", 2024-09-17T07:04:02.664 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_opa_token": "", 2024-09-17T07:04:02.664 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_opa_url": "", 2024-09-17T07:04:02.664 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_opa_verify_ssl": "true", 2024-09-17T07:04:02.664 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-09-17T07:04:02.664 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-09-17T07:04:02.664 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_ops_log_rados": "false", 2024-09-17T07:04:02.664 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_ops_log_socket_path": "", 2024-09-17T07:04:02.664 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-09-17T07:04:02.664 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-09-17T07:04:02.664 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-09-17T07:04:02.665 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_period_push_interval": "2.000000", 2024-09-17T07:04:02.665 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_period_push_interval_max": "30.000000", 2024-09-17T07:04:02.665 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_period_root_pool": ".rgw.root", 2024-09-17T07:04:02.665 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-09-17T07:04:02.665 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_print_continue": "true", 2024-09-17T07:04:02.665 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_print_prohibited_content_length": "false", 2024-09-17T07:04:02.665 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-09-17T07:04:02.665 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-09-17T07:04:02.665 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-09-17T07:04:02.665 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-09-17T07:04:02.665 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_rados_tracing": "false", 2024-09-17T07:04:02.665 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_realm": "", 2024-09-17T07:04:02.666 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_realm_id": "", 2024-09-17T07:04:02.666 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-09-17T07:04:02.666 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_region": "", 2024-09-17T07:04:02.666 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_region_root_pool": ".rgw.root", 2024-09-17T07:04:02.666 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_relaxed_region_enforcement": "false", 2024-09-17T07:04:02.666 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-09-17T07:04:02.666 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-09-17T07:04:02.666 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_request_uri": "", 2024-09-17T07:04:02.666 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_reshard_batch_size": "64", 2024-09-17T07:04:02.666 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-09-17T07:04:02.666 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_reshard_max_aio": "128", 2024-09-17T07:04:02.666 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_reshard_num_logs": "16", 2024-09-17T07:04:02.667 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_reshard_thread_interval": "600", 2024-09-17T07:04:02.667 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_resolve_cname": "false", 2024-09-17T07:04:02.667 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_rest_getusage_op_compat": "false", 2024-09-17T07:04:02.667 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_run_sync_thread": "true", 2024-09-17T07:04:02.667 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-09-17T07:04:02.667 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_s3_auth_use_keystone": "false", 2024-09-17T07:04:02.667 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_s3_auth_use_ldap": "false", 2024-09-17T07:04:02.667 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_s3_auth_use_rados": "true", 2024-09-17T07:04:02.667 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_s3_auth_use_sts": "false", 2024-09-17T07:04:02.667 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-09-17T07:04:02.667 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_s3_success_create_obj_status": "0", 2024-09-17T07:04:02.667 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-09-17T07:04:02.667 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_scheduler_type": "throttler", 2024-09-17T07:04:02.668 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_script_uri": "", 2024-09-17T07:04:02.668 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_service_provider_name": "", 2024-09-17T07:04:02.668 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-09-17T07:04:02.668 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_sts_client_id": "", 2024-09-17T07:04:02.668 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_sts_client_secret": "", 2024-09-17T07:04:02.668 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_sts_entry": "sts", 2024-09-17T07:04:02.668 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_sts_key": "sts", 2024-09-17T07:04:02.668 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_sts_max_session_duration": "43200", 2024-09-17T07:04:02.668 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_sts_min_session_duration": "900", 2024-09-17T07:04:02.668 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_sts_token_introspection_url": "", 2024-09-17T07:04:02.668 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_swift_account_in_url": "false", 2024-09-17T07:04:02.668 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_swift_auth_entry": "auth", 2024-09-17T07:04:02.669 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_swift_auth_url": "", 2024-09-17T07:04:02.669 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_swift_custom_header": "", 2024-09-17T07:04:02.669 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_swift_enforce_content_length": "false", 2024-09-17T07:04:02.669 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_swift_need_stats": "true", 2024-09-17T07:04:02.669 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_swift_tenant_name": "", 2024-09-17T07:04:02.669 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_swift_token_expiration": "86400", 2024-09-17T07:04:02.669 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_swift_url": "", 2024-09-17T07:04:02.669 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_swift_url_prefix": "swift", 2024-09-17T07:04:02.669 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_swift_versioning_enabled": "false", 2024-09-17T07:04:02.669 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-09-17T07:04:02.669 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-09-17T07:04:02.669 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_sync_lease_period": "120", 2024-09-17T07:04:02.669 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-09-17T07:04:02.670 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_sync_log_trim_interval": "1200", 2024-09-17T07:04:02.670 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-09-17T07:04:02.670 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-09-17T07:04:02.670 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-09-17T07:04:02.670 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_sync_obj_etag_verify": "false", 2024-09-17T07:04:02.670 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_sync_trace_history_size": "4096", 2024-09-17T07:04:02.670 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-09-17T07:04:02.670 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-09-17T07:04:02.670 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_thread_pool_size": "512", 2024-09-17T07:04:02.670 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_torrent_comment": "", 2024-09-17T07:04:02.670 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_torrent_createby": "", 2024-09-17T07:04:02.670 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_torrent_encoding": "", 2024-09-17T07:04:02.671 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_torrent_flag": "false", 2024-09-17T07:04:02.671 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_torrent_origin": "", 2024-09-17T07:04:02.671 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_torrent_sha_unit": "524288", 2024-09-17T07:04:02.671 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_torrent_tracker": "", 2024-09-17T07:04:02.671 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_trust_forwarded_https": "false", 2024-09-17T07:04:02.671 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-09-17T07:04:02.671 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_usage_log_tick_interval": "30", 2024-09-17T07:04:02.671 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_usage_max_shards": "32", 2024-09-17T07:04:02.671 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_usage_max_user_shards": "1", 2024-09-17T07:04:02.671 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_use_opa_authz": "false", 2024-09-17T07:04:02.671 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-09-17T07:04:02.671 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_user_default_quota_max_size": "-1", 2024-09-17T07:04:02.671 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_user_max_buckets": "1000", 2024-09-17T07:04:02.672 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_user_policies_max_num": "100", 2024-09-17T07:04:02.672 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-09-17T07:04:02.672 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-09-17T07:04:02.672 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_user_quota_sync_interval": "86400", 2024-09-17T07:04:02.672 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-09-17T07:04:02.672 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_user_unique_email": "true", 2024-09-17T07:04:02.672 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_verify_ssl": "true", 2024-09-17T07:04:02.672 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_website_routing_rules_max_num": "50", 2024-09-17T07:04:02.672 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_zone": "", 2024-09-17T07:04:02.672 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_zone_id": "", 2024-09-17T07:04:02.672 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-09-17T07:04:02.672 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_zonegroup": "", 2024-09-17T07:04:02.672 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_zonegroup_id": "", 2024-09-17T07:04:02.673 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-09-17T07:04:02.673 INFO:teuthology.orchestra.run.smithi100.stdout: "rgwlc_auto_session_clear": "true", 2024-09-17T07:04:02.673 INFO:teuthology.orchestra.run.smithi100.stdout: "rgwlc_skip_bucket_step": "false", 2024-09-17T07:04:02.673 INFO:teuthology.orchestra.run.smithi100.stdout: "rocksdb_block_size": "4096", 2024-09-17T07:04:02.673 INFO:teuthology.orchestra.run.smithi100.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-09-17T07:04:02.673 INFO:teuthology.orchestra.run.smithi100.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-09-17T07:04:02.673 INFO:teuthology.orchestra.run.smithi100.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-09-17T07:04:02.673 INFO:teuthology.orchestra.run.smithi100.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-09-17T07:04:02.673 INFO:teuthology.orchestra.run.smithi100.stdout: "rocksdb_cache_shard_bits": "4", 2024-09-17T07:04:02.673 INFO:teuthology.orchestra.run.smithi100.stdout: "rocksdb_cache_size": "536870912", 2024-09-17T07:04:02.673 INFO:teuthology.orchestra.run.smithi100.stdout: "rocksdb_cache_type": "binned_lru", 2024-09-17T07:04:02.673 INFO:teuthology.orchestra.run.smithi100.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-09-17T07:04:02.673 INFO:teuthology.orchestra.run.smithi100.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-09-17T07:04:02.674 INFO:teuthology.orchestra.run.smithi100.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-09-17T07:04:02.674 INFO:teuthology.orchestra.run.smithi100.stdout: "rocksdb_collect_compaction_stats": "false", 2024-09-17T07:04:02.674 INFO:teuthology.orchestra.run.smithi100.stdout: "rocksdb_collect_extended_stats": "false", 2024-09-17T07:04:02.674 INFO:teuthology.orchestra.run.smithi100.stdout: "rocksdb_collect_memory_stats": "false", 2024-09-17T07:04:02.674 INFO:teuthology.orchestra.run.smithi100.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-09-17T07:04:02.674 INFO:teuthology.orchestra.run.smithi100.stdout: "rocksdb_index_type": "binary_search", 2024-09-17T07:04:02.674 INFO:teuthology.orchestra.run.smithi100.stdout: "rocksdb_log_to_ceph_log": "true", 2024-09-17T07:04:02.674 INFO:teuthology.orchestra.run.smithi100.stdout: "rocksdb_metadata_block_size": "4096", 2024-09-17T07:04:02.674 INFO:teuthology.orchestra.run.smithi100.stdout: "rocksdb_partition_filters": "false", 2024-09-17T07:04:02.674 INFO:teuthology.orchestra.run.smithi100.stdout: "rocksdb_perf": "false", 2024-09-17T07:04:02.674 INFO:teuthology.orchestra.run.smithi100.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-09-17T07:04:02.674 INFO:teuthology.orchestra.run.smithi100.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-09-17T07:04:02.674 INFO:teuthology.orchestra.run.smithi100.stdout: "rotating_keys_renewal_timeout": "10", 2024-09-17T07:04:02.675 INFO:teuthology.orchestra.run.smithi100.stdout: "run_dir": "/var/run/ceph", 2024-09-17T07:04:02.675 INFO:teuthology.orchestra.run.smithi100.stdout: "seastore_block_create": "true", 2024-09-17T07:04:02.675 INFO:teuthology.orchestra.run.smithi100.stdout: "seastore_cache_lru_size": "67108864", 2024-09-17T07:04:02.675 INFO:teuthology.orchestra.run.smithi100.stdout: "seastore_cbjournal_size": "5368709120", 2024-09-17T07:04:02.675 INFO:teuthology.orchestra.run.smithi100.stdout: "seastore_default_max_object_size": "16777216", 2024-09-17T07:04:02.675 INFO:teuthology.orchestra.run.smithi100.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-09-17T07:04:02.675 INFO:teuthology.orchestra.run.smithi100.stdout: "seastore_device_size": "53687091200", 2024-09-17T07:04:02.675 INFO:teuthology.orchestra.run.smithi100.stdout: "seastore_journal_batch_capacity": "16", 2024-09-17T07:04:02.675 INFO:teuthology.orchestra.run.smithi100.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-09-17T07:04:02.675 INFO:teuthology.orchestra.run.smithi100.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-09-17T07:04:02.675 INFO:teuthology.orchestra.run.smithi100.stdout: "seastore_journal_iodepth_limit": "5", 2024-09-17T07:04:02.675 INFO:teuthology.orchestra.run.smithi100.stdout: "seastore_main_device_type": "SSD", 2024-09-17T07:04:02.675 INFO:teuthology.orchestra.run.smithi100.stdout: "seastore_max_concurrent_transactions": "8", 2024-09-17T07:04:02.676 INFO:teuthology.orchestra.run.smithi100.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-09-17T07:04:02.676 INFO:teuthology.orchestra.run.smithi100.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-09-17T07:04:02.676 INFO:teuthology.orchestra.run.smithi100.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-09-17T07:04:02.676 INFO:teuthology.orchestra.run.smithi100.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-09-17T07:04:02.676 INFO:teuthology.orchestra.run.smithi100.stdout: "seastore_segment_size": "67108864", 2024-09-17T07:04:02.676 INFO:teuthology.orchestra.run.smithi100.stdout: "set_keepcaps": "false", 2024-09-17T07:04:02.676 INFO:teuthology.orchestra.run.smithi100.stdout: "setgroup": "ceph", 2024-09-17T07:04:02.676 INFO:teuthology.orchestra.run.smithi100.stdout: "setuser": "ceph", 2024-09-17T07:04:02.676 INFO:teuthology.orchestra.run.smithi100.stdout: "setuser_match_path": "", 2024-09-17T07:04:02.676 INFO:teuthology.orchestra.run.smithi100.stdout: "target_max_misplaced_ratio": "0.050000", 2024-09-17T07:04:02.676 INFO:teuthology.orchestra.run.smithi100.stdout: "thp": "false", 2024-09-17T07:04:02.676 INFO:teuthology.orchestra.run.smithi100.stdout: "threadpool_default_timeout": "60", 2024-09-17T07:04:02.677 INFO:teuthology.orchestra.run.smithi100.stdout: "threadpool_empty_queue_max_wait": "2", 2024-09-17T07:04:02.677 INFO:teuthology.orchestra.run.smithi100.stdout: "throttler_perf_counter": "true" 2024-09-17T07:04:02.677 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-17T07:04:03.044 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:04:02 smithi135 podman[64826]: 2024-09-17 07:04:02.580815585 +0000 UTC m=+0.441478276 container init 8bdc6108c6f73e24402d13d2d88f4a1e57eb1aee376eacf204aa1133b78eb22a (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-17T07:04:03.044 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:04:02 smithi135 podman[64826]: 2024-09-17 07:04:02.583817481 +0000 UTC m=+0.444480183 container start 8bdc6108c6f73e24402d13d2d88f4a1e57eb1aee376eacf204aa1133b78eb22a (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-17T07:04:03.044 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:04:02 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a[64841]: ts=2024-09-17T07:04:02.607Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-09-17T07:04:03.044 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:04:02 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a[64841]: ts=2024-09-17T07:04:02.607Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-09-17T07:04:03.044 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:04:02 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a[64841]: ts=2024-09-17T07:04:02.607Z caller=main.go:570 level=info host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi135 (none))" 2024-09-17T07:04:03.044 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:04:02 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a[64841]: ts=2024-09-17T07:04:02.607Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-17T07:04:03.044 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:04:02 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a[64841]: ts=2024-09-17T07:04:02.607Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-17T07:04:03.044 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:04:02 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a[64841]: ts=2024-09-17T07:04:02.608Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-09-17T07:04:03.044 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:04:02 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a[64841]: ts=2024-09-17T07:04:02.609Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-09-17T07:04:03.045 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:04:02 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a[64841]: ts=2024-09-17T07:04:02.611Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-09-17T07:04:03.045 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:04:02 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a[64841]: ts=2024-09-17T07:04:02.611Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-17T07:04:03.045 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:04:02 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a[64841]: ts=2024-09-17T07:04:02.613Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-17T07:04:03.045 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:04:02 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a[64841]: ts=2024-09-17T07:04:02.613Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.561µs 2024-09-17T07:04:03.045 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:04:02 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a[64841]: ts=2024-09-17T07:04:02.613Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-17T07:04:03.045 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:04:02 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a[64841]: ts=2024-09-17T07:04:02.613Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-09-17T07:04:03.045 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:04:02 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a[64841]: ts=2024-09-17T07:04:02.613Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-09-17T07:04:03.045 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:04:02 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a[64841]: ts=2024-09-17T07:04:02.613Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=18.38µs wal_replay_duration=525.454µs wbl_replay_duration=155ns total_replay_duration=562.113µs 2024-09-17T07:04:03.045 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:04:02 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a[64841]: ts=2024-09-17T07:04:02.614Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-09-17T07:04:03.045 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:04:02 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a[64841]: ts=2024-09-17T07:04:02.614Z caller=main.go:1029 level=info msg="TSDB started" 2024-09-17T07:04:03.045 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:04:02 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a[64841]: ts=2024-09-17T07:04:02.614Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-17T07:04:03.046 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:04:02 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a[64841]: ts=2024-09-17T07:04:02.634Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=20.47622ms db_storage=844ns remote_storage=1.393µs web_handler=641ns query_engine=511ns scrape=435.062µs scrape_sd=84.452µs notify=13.469µs notify_sd=27.382µs rules=19.620351ms tracing=6.397µs 2024-09-17T07:04:03.046 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:04:02 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a[64841]: ts=2024-09-17T07:04:02.634Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-09-17T07:04:03.046 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:04:02 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a[64841]: ts=2024-09-17T07:04:02.634Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-09-17T07:04:03.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:03 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:04:03.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:03 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:04:03.398 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-09-17T07:04:03.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:03 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:04:03.606 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:04:03.831 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:04:03 smithi135 bash[64826]: 8bdc6108c6f73e24402d13d2d88f4a1e57eb1aee376eacf204aa1133b78eb22a 2024-09-17T07:04:03.831 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:04:03 smithi135 systemd[1]: Started Ceph prometheus.a for e848267c-74c1-11ef-bceb-c7b262605968. 2024-09-17T07:04:03.972 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:04:03 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: [17/Sep/2024:07:04:03] ENGINE Bus STOPPING 2024-09-17T07:04:03.972 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:04:03 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: [17/Sep/2024:07:04:03] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-17T07:04:03.972 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:04:03 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: [17/Sep/2024:07:04:03] ENGINE Bus STOPPED 2024-09-17T07:04:03.973 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:04:03 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: [17/Sep/2024:07:04:03] ENGINE Bus STARTING 2024-09-17T07:04:03.973 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:04:03 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: [17/Sep/2024:07:04:03] ENGINE Serving on http://:::9283 2024-09-17T07:04:03.973 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:04:03 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: [17/Sep/2024:07:04:03] ENGINE Bus STARTED 2024-09-17T07:04:03.973 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:04:03 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: [17/Sep/2024:07:04:03] ENGINE Bus STOPPING 2024-09-17T07:04:04.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:04 smithi135 ceph-mon[33238]: 5.1e scrub starts 2024-09-17T07:04:04.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:04 smithi135 ceph-mon[33238]: 5.1e scrub ok 2024-09-17T07:04:04.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:04 smithi135 ceph-mon[33238]: pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T07:04:04.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:04 smithi135 ceph-mon[33238]: 6.1e scrub starts 2024-09-17T07:04:04.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:04 smithi135 ceph-mon[33238]: 6.1e scrub ok 2024-09-17T07:04:04.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:04 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:04:04.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:04 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:04:04.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:04 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-17T07:04:04.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:04 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi100.front.sepia.ceph.com:9093"}]: dispatch 2024-09-17T07:04:04.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:04 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:04:04.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:04 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T07:04:04.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:04 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi135.front.sepia.ceph.com:9095"}]: dispatch 2024-09-17T07:04:04.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:04 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:04:04.295 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:04 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-17T07:04:04.295 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:04 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi135.front.sepia.ceph.com:3000"}]: dispatch 2024-09-17T07:04:04.295 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:04 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:04:04.295 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:04 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:04:04.295 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:04 smithi135 ceph-mon[33238]: 6.4 scrub starts 2024-09-17T07:04:04.295 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:04 smithi135 ceph-mon[33238]: 6.4 scrub ok 2024-09-17T07:04:04.347 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:04:04 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: [17/Sep/2024:07:04:04] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-17T07:04:04.348 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:04:04 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: [17/Sep/2024:07:04:04] ENGINE Bus STOPPED 2024-09-17T07:04:04.348 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:04 smithi100 ceph-mon[39462]: 5.1e scrub starts 2024-09-17T07:04:04.348 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:04 smithi100 ceph-mon[39462]: 5.1e scrub ok 2024-09-17T07:04:04.348 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:04 smithi100 ceph-mon[39462]: pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T07:04:04.348 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:04 smithi100 ceph-mon[39462]: 6.1e scrub starts 2024-09-17T07:04:04.348 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:04 smithi100 ceph-mon[39462]: 6.1e scrub ok 2024-09-17T07:04:04.348 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:04 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:04:04.348 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:04 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:04:04.348 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:04 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-17T07:04:04.349 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:04 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi100.front.sepia.ceph.com:9093"}]: dispatch 2024-09-17T07:04:04.349 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:04 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:04:04.349 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:04 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T07:04:04.349 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:04 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi135.front.sepia.ceph.com:9095"}]: dispatch 2024-09-17T07:04:04.349 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:04 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:04:04.349 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:04 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-17T07:04:04.349 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:04 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi135.front.sepia.ceph.com:3000"}]: dispatch 2024-09-17T07:04:04.349 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:04 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:04:04.349 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:04 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:04:04.349 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:04 smithi100 ceph-mon[39462]: 6.4 scrub starts 2024-09-17T07:04:04.349 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:04 smithi100 ceph-mon[39462]: 6.4 scrub ok 2024-09-17T07:04:04.350 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:04 smithi100 ceph-mon[29092]: 5.1e scrub starts 2024-09-17T07:04:04.350 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:04 smithi100 ceph-mon[29092]: 5.1e scrub ok 2024-09-17T07:04:04.350 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:04 smithi100 ceph-mon[29092]: pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T07:04:04.350 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:04 smithi100 ceph-mon[29092]: 6.1e scrub starts 2024-09-17T07:04:04.350 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:04 smithi100 ceph-mon[29092]: 6.1e scrub ok 2024-09-17T07:04:04.350 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:04 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:04:04.350 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:04 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:04:04.350 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:04 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-17T07:04:04.350 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:04 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi100.front.sepia.ceph.com:9093"}]: dispatch 2024-09-17T07:04:04.350 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:04 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:04:04.350 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:04 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T07:04:04.350 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:04 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi135.front.sepia.ceph.com:9095"}]: dispatch 2024-09-17T07:04:04.350 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:04 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:04:04.351 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:04 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-17T07:04:04.351 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:04 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi135.front.sepia.ceph.com:3000"}]: dispatch 2024-09-17T07:04:04.351 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:04 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:04:04.351 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:04 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:04:04.351 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:04 smithi100 ceph-mon[29092]: 6.4 scrub starts 2024-09-17T07:04:04.351 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:04 smithi100 ceph-mon[29092]: 6.4 scrub ok 2024-09-17T07:04:04.627 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:04:04 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: [17/Sep/2024:07:04:04] ENGINE Bus STARTING 2024-09-17T07:04:04.627 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:04:04 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: [17/Sep/2024:07:04:04] ENGINE Serving on http://:::9283 2024-09-17T07:04:04.627 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:04:04 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: [17/Sep/2024:07:04:04] ENGINE Bus STARTED 2024-09-17T07:04:04.627 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:04:04 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: [17/Sep/2024:07:04:04] ENGINE Bus STOPPING 2024-09-17T07:04:05.006 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:04:04 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: [17/Sep/2024:07:04:04] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-17T07:04:05.006 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:04:04 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: [17/Sep/2024:07:04:04] ENGINE Bus STOPPED 2024-09-17T07:04:05.006 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:04:04 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: [17/Sep/2024:07:04:04] ENGINE Bus STARTING 2024-09-17T07:04:05.247 INFO:teuthology.orchestra.run.smithi100.stdout:mon.a: {} 2024-09-17T07:04:05.247 INFO:teuthology.orchestra.run.smithi100.stdout:mon.b: {} 2024-09-17T07:04:05.248 INFO:teuthology.orchestra.run.smithi100.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-17T07:04:05.248 INFO:teuthology.orchestra.run.smithi100.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-17T07:04:05.256 INFO:teuthology.orchestra.run.smithi100.stdout:mon.c: {} 2024-09-17T07:04:05.256 INFO:teuthology.orchestra.run.smithi100.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-17T07:04:05.256 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:04:05 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: [17/Sep/2024:07:04:05] ENGINE Serving on http://:::9283 2024-09-17T07:04:05.256 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:04:05 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: [17/Sep/2024:07:04:05] ENGINE Bus STARTED 2024-09-17T07:04:05.257 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:05 smithi100 ceph-mon[29092]: 6.1b scrub starts 2024-09-17T07:04:05.257 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:05 smithi100 ceph-mon[29092]: 6.1b scrub ok 2024-09-17T07:04:05.257 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:05 smithi100 ceph-mon[29092]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-17T07:04:05.257 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:05 smithi100 ceph-mon[29092]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi100.front.sepia.ceph.com:9093"}]: dispatch 2024-09-17T07:04:05.257 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:05 smithi100 ceph-mon[29092]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T07:04:05.257 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:05 smithi100 ceph-mon[29092]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi135.front.sepia.ceph.com:9095"}]: dispatch 2024-09-17T07:04:05.257 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:05 smithi100 ceph-mon[29092]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-17T07:04:05.257 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:05 smithi100 ceph-mon[29092]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi135.front.sepia.ceph.com:3000"}]: dispatch 2024-09-17T07:04:05.258 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:05 smithi100 ceph-mon[39462]: 6.1b scrub starts 2024-09-17T07:04:05.258 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:05 smithi100 ceph-mon[39462]: 6.1b scrub ok 2024-09-17T07:04:05.258 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:05 smithi100 ceph-mon[39462]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-17T07:04:05.258 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:05 smithi100 ceph-mon[39462]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi100.front.sepia.ceph.com:9093"}]: dispatch 2024-09-17T07:04:05.258 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:05 smithi100 ceph-mon[39462]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T07:04:05.258 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:05 smithi100 ceph-mon[39462]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi135.front.sepia.ceph.com:9095"}]: dispatch 2024-09-17T07:04:05.258 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:05 smithi100 ceph-mon[39462]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-17T07:04:05.258 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:05 smithi100 ceph-mon[39462]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi135.front.sepia.ceph.com:3000"}]: dispatch 2024-09-17T07:04:05.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:05 smithi135 ceph-mon[33238]: 6.1b scrub starts 2024-09-17T07:04:05.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:05 smithi135 ceph-mon[33238]: 6.1b scrub ok 2024-09-17T07:04:05.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:05 smithi135 ceph-mon[33238]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-17T07:04:05.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:05 smithi135 ceph-mon[33238]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi100.front.sepia.ceph.com:9093"}]: dispatch 2024-09-17T07:04:05.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:05 smithi135 ceph-mon[33238]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T07:04:05.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:05 smithi135 ceph-mon[33238]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi135.front.sepia.ceph.com:9095"}]: dispatch 2024-09-17T07:04:05.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:05 smithi135 ceph-mon[33238]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-17T07:04:05.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:05 smithi135 ceph-mon[33238]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi135.front.sepia.ceph.com:3000"}]: dispatch 2024-09-17T07:04:05.856 INFO:teuthology.run_tasks:Running task full_sequential... 2024-09-17T07:04:05.867 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-17T07:04:05.901 INFO:tasks.radosbench:Beginning radosbench... 2024-09-17T07:04:05.901 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_0 2024-09-17T07:04:05.901 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd pool create unique_pool_0 16 2024-09-17T07:04:05.902 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-09-17T07:04:05.902 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-17T07:04:05.902 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T07:04:05.902 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-17T07:04:05.902 INFO:tasks.thrashosds.thrasher:Setting osd 2 primary_affinity to 0.167875 2024-09-17T07:04:05.902 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 2 0.16787546240443685 2024-09-17T07:04:05.903 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-09-17T07:04:05.903 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:04:05.903 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-09-17T07:04:05.904 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:04:05.905 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-09-17T07:04:05.905 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-17T07:04:05.905 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-09-17T07:04:05.905 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-17T07:04:05.936 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:05.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:05.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:04:06.038 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:04:06.061 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:06.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:06.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:04:06.164 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:04:06.185 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:06.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:06.187 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:04:06.287 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:04:06.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:06 smithi135 ceph-mon[33238]: 6.6 scrub starts 2024-09-17T07:04:06.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:06 smithi135 ceph-mon[33238]: 6.6 scrub ok 2024-09-17T07:04:06.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:06 smithi135 ceph-mon[33238]: pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T07:04:06.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:06 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/570819718' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-17T07:04:06.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:06 smithi135 ceph-mon[33238]: 4.e scrub starts 2024-09-17T07:04:06.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:06 smithi135 ceph-mon[33238]: 4.e scrub ok 2024-09-17T07:04:06.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:06 smithi135 ceph-mon[33238]: 2.15 deep-scrub starts 2024-09-17T07:04:06.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:06 smithi135 ceph-mon[33238]: 2.15 deep-scrub ok 2024-09-17T07:04:06.309 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:06.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:06.310 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:04:06.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:06 smithi100 ceph-mon[39462]: 6.6 scrub starts 2024-09-17T07:04:06.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:06 smithi100 ceph-mon[39462]: 6.6 scrub ok 2024-09-17T07:04:06.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:06 smithi100 ceph-mon[39462]: pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T07:04:06.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:06 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/570819718' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-17T07:04:06.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:06 smithi100 ceph-mon[39462]: 4.e scrub starts 2024-09-17T07:04:06.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:06 smithi100 ceph-mon[39462]: 4.e scrub ok 2024-09-17T07:04:06.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:06 smithi100 ceph-mon[39462]: 2.15 deep-scrub starts 2024-09-17T07:04:06.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:06 smithi100 ceph-mon[39462]: 2.15 deep-scrub ok 2024-09-17T07:04:06.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:06 smithi100 ceph-mon[29092]: 6.6 scrub starts 2024-09-17T07:04:06.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:06 smithi100 ceph-mon[29092]: 6.6 scrub ok 2024-09-17T07:04:06.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:06 smithi100 ceph-mon[29092]: pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T07:04:06.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:06 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/570819718' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-17T07:04:06.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:06 smithi100 ceph-mon[29092]: 4.e scrub starts 2024-09-17T07:04:06.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:06 smithi100 ceph-mon[29092]: 4.e scrub ok 2024-09-17T07:04:06.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:06 smithi100 ceph-mon[29092]: 2.15 deep-scrub starts 2024-09-17T07:04:06.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:06 smithi100 ceph-mon[29092]: 2.15 deep-scrub ok 2024-09-17T07:04:06.411 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:04:06.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:06.434 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:06.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:04:06.535 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:04:06.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:06.557 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:06.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:04:06.601 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:04:06.610 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:04:06.659 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:04:06.684 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:06.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:06.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:04:06.786 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:04:06.807 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:06.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:06.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:04:06.910 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:04:06.931 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:06.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:06.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:04:07.033 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:04:07.054 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:07.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:07.056 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:04:07.156 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:04:07.179 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:07.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:07.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:04:07.280 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:04:07.302 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:07.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:07.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:04:07.404 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:04:07.425 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:07.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:07.426 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:04:07.526 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:04:07.548 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:07.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:07.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:04:07.650 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:04:07.671 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:07.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:07.672 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:04:07.773 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:04:07.794 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:07.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:07.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:04:07.896 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:04:07.918 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:07.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:07.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:04:08.020 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:04:08.043 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:08.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:08.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:04:08.087 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:Sep 17 07:04:07 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-alertmanager-a[73990]: ts=2024-09-17T07:04:07.833Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.002706942s 2024-09-17T07:04:08.145 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:04:08.167 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:08.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:08.168 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:04:08.269 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:04:08.290 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:08.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:08.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:04:08.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:08 smithi100 ceph-mon[29092]: 6.b scrub starts 2024-09-17T07:04:08.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:08 smithi100 ceph-mon[29092]: 6.b scrub ok 2024-09-17T07:04:08.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:08 smithi100 ceph-mon[29092]: pgmap v59: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T07:04:08.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:08 smithi100 ceph-mon[29092]: 6.18 scrub starts 2024-09-17T07:04:08.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:08 smithi100 ceph-mon[29092]: 6.18 scrub ok 2024-09-17T07:04:08.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:08 smithi100 ceph-mon[39462]: 6.b scrub starts 2024-09-17T07:04:08.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:08 smithi100 ceph-mon[39462]: 6.b scrub ok 2024-09-17T07:04:08.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:08 smithi100 ceph-mon[39462]: pgmap v59: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T07:04:08.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:08 smithi100 ceph-mon[39462]: 6.18 scrub starts 2024-09-17T07:04:08.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:08 smithi100 ceph-mon[39462]: 6.18 scrub ok 2024-09-17T07:04:08.390 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:08 smithi135 ceph-mon[33238]: 6.b scrub starts 2024-09-17T07:04:08.390 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:08 smithi135 ceph-mon[33238]: 6.b scrub ok 2024-09-17T07:04:08.390 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:08 smithi135 ceph-mon[33238]: pgmap v59: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T07:04:08.390 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:08 smithi135 ceph-mon[33238]: 6.18 scrub starts 2024-09-17T07:04:08.390 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:08 smithi135 ceph-mon[33238]: 6.18 scrub ok 2024-09-17T07:04:08.390 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:04:08 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:04:08.393 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:04:08.414 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:08.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:08.415 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:04:08.516 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:04:08.539 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:08.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:08.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:04:08.641 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:04:08.662 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:08.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:08.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:04:08.764 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:04:08.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:08.787 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:08.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:04:08.888 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:04:08.909 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:08.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:08.910 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:04:09.010 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:04:09.031 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:09.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:09.032 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:04:09.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:09 smithi100 ceph-mon[39462]: 6.11 scrub starts 2024-09-17T07:04:09.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:09 smithi100 ceph-mon[39462]: 6.11 scrub ok 2024-09-17T07:04:09.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:09 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:04:09.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:09 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:04:09.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:09 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:04:09.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:09 smithi100 ceph-mon[39462]: osdmap e95: 8 total, 8 up, 8 in 2024-09-17T07:04:09.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:09 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:04:09.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:09 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:04:09.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:09 smithi100 ceph-mon[29092]: 6.11 scrub starts 2024-09-17T07:04:09.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:09 smithi100 ceph-mon[29092]: 6.11 scrub ok 2024-09-17T07:04:09.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:09 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:04:09.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:09 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:04:09.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:09 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:04:09.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:09 smithi100 ceph-mon[29092]: osdmap e95: 8 total, 8 up, 8 in 2024-09-17T07:04:09.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:09 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:04:09.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:09 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:04:09.133 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:04:09.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:09.157 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:09.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:04:09.258 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:04:09.280 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:09.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:09.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:04:09.382 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:04:09.403 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:09.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:09.404 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:04:09.505 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:04:09.526 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:09.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:09.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:04:09.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:09 smithi135 ceph-mon[33238]: 6.11 scrub starts 2024-09-17T07:04:09.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:09 smithi135 ceph-mon[33238]: 6.11 scrub ok 2024-09-17T07:04:09.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:09 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:04:09.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:09 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:04:09.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:09 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:04:09.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:09 smithi135 ceph-mon[33238]: osdmap e95: 8 total, 8 up, 8 in 2024-09-17T07:04:09.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:09 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:04:09.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:09 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:04:09.622 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:04:09.622 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:04:09.626 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:04:09.628 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:04:09.643 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1: {} 2024-09-17T07:04:09.644 INFO:teuthology.orchestra.run.smithi100.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:04:09.649 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:09.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:09.650 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:04:09.663 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:04:09.663 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:04:09.687 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:04:09.687 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:04:09.704 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:04:09.704 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:04:09.720 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:04:09.720 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:04:09.737 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:04:09.737 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:04:09.751 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:04:09.752 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:04:09.753 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:04:09.773 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:09.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:09.774 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:04:09.875 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:04:09.898 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:09.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:09.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:04:10.000 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:04:10.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:10.023 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:10.024 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:04:10.125 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:04:10.146 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:10.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:10.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:04:10.248 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:04:10.269 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:10.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:10.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:04:10.371 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:04:10.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:10 smithi100 ceph-mon[29092]: pgmap v61: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T07:04:10.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:10 smithi100 ceph-mon[29092]: 4.d deep-scrub starts 2024-09-17T07:04:10.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:10 smithi100 ceph-mon[29092]: 4.d deep-scrub ok 2024-09-17T07:04:10.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:10 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/3434481064' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:04:10.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:10 smithi100 ceph-mon[39462]: pgmap v61: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T07:04:10.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:10 smithi100 ceph-mon[39462]: 4.d deep-scrub starts 2024-09-17T07:04:10.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:10 smithi100 ceph-mon[39462]: 4.d deep-scrub ok 2024-09-17T07:04:10.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:10 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/3434481064' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:04:10.393 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:10.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:10.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:04:10.495 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:04:10.517 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:10.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:10.518 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:04:10.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:10 smithi135 ceph-mon[33238]: pgmap v61: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T07:04:10.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:10 smithi135 ceph-mon[33238]: 4.d deep-scrub starts 2024-09-17T07:04:10.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:10 smithi135 ceph-mon[33238]: 4.d deep-scrub ok 2024-09-17T07:04:10.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:10 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/3434481064' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:04:10.619 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:04:10.640 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:10.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:10.641 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:04:10.742 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:04:10.764 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:10.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:10.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:04:10.795 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:04:10.865 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:04:10.886 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:10.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:10.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:04:10.988 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:04:11.008 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:11.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:11.010 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:04:11.108 INFO:teuthology.orchestra.run.smithi100.stderr:noscrub is set 2024-09-17T07:04:11.111 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:04:11.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:11.134 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:11.134 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:04:11.235 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:04:11.257 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:11.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:11.258 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:04:11.359 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:04:11.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:11 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/3715283423' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:04:11.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:11 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:04:11.376 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:11 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/3715283423' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:04:11.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:11 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:04:11.381 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:11.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:11.382 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:04:11.483 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:04:11.504 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:11.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:11.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:04:11.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:11 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/3715283423' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:04:11.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:11 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:04:11.606 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:04:11.627 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:11.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:11.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:04:11.729 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:04:11.751 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:11.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:11.752 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:04:11.852 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:04:11.874 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:11.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:11.875 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:04:11.976 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:04:12.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:12.005 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:12.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:04:12.106 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:04:12.132 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:12.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:12.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:04:12.234 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:04:12.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:12.260 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:12.260 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:04:12.349 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:12 smithi100 ceph-mon[29092]: 6.1f scrub starts 2024-09-17T07:04:12.349 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:12 smithi100 ceph-mon[29092]: 6.1f scrub ok 2024-09-17T07:04:12.350 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:12 smithi100 ceph-mon[29092]: pgmap v62: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T07:04:12.350 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:12 smithi100 ceph-mon[29092]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:04:12.350 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:12 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T07:04:12.350 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:12 smithi100 ceph-mon[29092]: osdmap e96: 8 total, 8 up, 8 in 2024-09-17T07:04:12.350 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:12 smithi100 ceph-mon[39462]: 6.1f scrub starts 2024-09-17T07:04:12.350 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:12 smithi100 ceph-mon[39462]: 6.1f scrub ok 2024-09-17T07:04:12.350 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:12 smithi100 ceph-mon[39462]: pgmap v62: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T07:04:12.350 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:12 smithi100 ceph-mon[39462]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:04:12.351 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:12 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T07:04:12.351 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:12 smithi100 ceph-mon[39462]: osdmap e96: 8 total, 8 up, 8 in 2024-09-17T07:04:12.361 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:04:12.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:12.387 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:12.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:04:12.488 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:04:12.510 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:12.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:12.511 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:04:12.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:12 smithi135 ceph-mon[33238]: 6.1f scrub starts 2024-09-17T07:04:12.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:12 smithi135 ceph-mon[33238]: 6.1f scrub ok 2024-09-17T07:04:12.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:12 smithi135 ceph-mon[33238]: pgmap v62: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T07:04:12.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:12 smithi135 ceph-mon[33238]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:04:12.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:12 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T07:04:12.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:12 smithi135 ceph-mon[33238]: osdmap e96: 8 total, 8 up, 8 in 2024-09-17T07:04:12.581 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:04:12.611 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:04:12.646 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:12.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:12.648 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:04:12.748 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:04:12.770 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:12.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:12.771 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:04:12.872 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:04:12.899 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:12.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:12.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:04:13.001 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:04:13.025 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:13.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:13.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:04:13.126 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:04:13.151 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:13.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:13.152 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:04:13.252 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:04:13.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:13.278 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:13.279 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:04:13.380 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:04:13.404 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:13.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:13.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:04:13.470 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:04:13.507 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:04:13.529 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:13.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:13.530 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:04:13.631 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:04:13.652 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:13.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:13.654 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:04:13.755 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:04:13.781 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:13.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:13.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:04:13.882 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:04:13.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:13.942 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:13.942 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:04:14.043 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:04:14.066 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:14.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:14.067 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:04:14.103 INFO:teuthology.orchestra.run.smithi100.stderr:set osd.2 primary-affinity to 0.167875 (8110012) 2024-09-17T07:04:14.168 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:04:14.189 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:14.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:14.190 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:04:14.291 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:04:14.314 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:14.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:14.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:04:14.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:14 smithi100 ceph-mon[29092]: pgmap v64: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:04:14.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:14 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1909032365' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.16787546240443685}]: dispatch 2024-09-17T07:04:14.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:14 smithi100 ceph-mon[39462]: pgmap v64: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:04:14.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:14 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1909032365' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.16787546240443685}]: dispatch 2024-09-17T07:04:14.416 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:04:14.439 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:14.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:14.441 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:04:14.541 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:04:14.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:14 smithi135 ceph-mon[33238]: pgmap v64: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:04:14.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:14 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1909032365' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.16787546240443685}]: dispatch 2024-09-17T07:04:14.564 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:14.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:14.565 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:04:14.665 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:04:14.690 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:14.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:14.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:04:14.792 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:04:14.815 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:14.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:14.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:04:14.917 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:04:14.938 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:14.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:14.939 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:04:15.040 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:04:15.058 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-17T07:04:15.065 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:15.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:15.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:04:15.166 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:04:15.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:15.193 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:15.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:04:15.252 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:15 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1909032365' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.16787546240443685}]': finished 2024-09-17T07:04:15.252 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:15 smithi100 ceph-mon[29092]: osdmap e97: 8 total, 8 up, 8 in 2024-09-17T07:04:15.252 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:15 smithi100 ceph-mon[29092]: osdmap e98: 8 total, 8 up, 8 in 2024-09-17T07:04:15.252 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:15 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1909032365' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.16787546240443685}]': finished 2024-09-17T07:04:15.252 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:15 smithi100 ceph-mon[39462]: osdmap e97: 8 total, 8 up, 8 in 2024-09-17T07:04:15.253 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:15 smithi100 ceph-mon[39462]: osdmap e98: 8 total, 8 up, 8 in 2024-09-17T07:04:15.293 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:04:15.316 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:15.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:15.317 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:04:15.417 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:04:15.439 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:15.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:15.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:04:15.541 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:04:15.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:15 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1909032365' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.16787546240443685}]': finished 2024-09-17T07:04:15.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:15 smithi135 ceph-mon[33238]: osdmap e97: 8 total, 8 up, 8 in 2024-09-17T07:04:15.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:15 smithi135 ceph-mon[33238]: osdmap e98: 8 total, 8 up, 8 in 2024-09-17T07:04:15.563 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:15.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:15.564 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:04:15.604 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:04:15.665 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:04:15.686 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:15.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:15.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:04:15.788 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:04:15.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:15.812 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:15.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:04:15.913 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:04:15.937 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:15.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:15.938 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:04:16.039 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:04:16.041 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:04:15 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:04:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:04:16.062 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:16.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:16.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:04:16.163 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:04:16.186 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:16.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:16.187 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:04:16.288 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:04:16.308 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:16.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:16.310 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:04:16.360 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:16 smithi100 ceph-mon[29092]: pgmap v66: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 739 B/s rd, 0 op/s 2024-09-17T07:04:16.360 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:16 smithi100 ceph-mon[39462]: pgmap v66: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 739 B/s rd, 0 op/s 2024-09-17T07:04:16.410 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:04:16.435 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:16.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:16.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:04:16.537 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:04:16.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:16 smithi135 ceph-mon[33238]: pgmap v66: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 739 B/s rd, 0 op/s 2024-09-17T07:04:16.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:16.559 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:16.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:04:16.660 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:04:16.666 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:04:16.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:16.685 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:16.685 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:04:16.786 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:04:16.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:16.808 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:16.808 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:04:16.909 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:04:16.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:16.931 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:16.931 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:04:17.032 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:04:17.061 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:17.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:17.062 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:04:17.163 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:04:17.185 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:17.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:17.186 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:04:17.286 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:04:17.311 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:17.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:17.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:04:17.413 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:04:17.434 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:17.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:17.435 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:04:17.536 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:04:17.557 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:17.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:17.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:04:17.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:17 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:04:17.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:17 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:04:17.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:17 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:04:17.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:17 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:04:17.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:17 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:04:17.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:17 smithi100 ceph-mon[29092]: pgmap v68: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:04:17.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:17 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:04:17.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:17 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:04:17.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:17 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:04:17.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:17 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:04:17.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:17 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:04:17.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:17 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:04:17.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:17 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:04:17.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:17 smithi100 ceph-mon[39462]: pgmap v68: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:04:17.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:17 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:04:17.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:17 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:04:17.659 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:04:17.685 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:17.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:17.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:04:17.787 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:04:17.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:17 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:04:17.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:17 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:04:17.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:17 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:04:17.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:17 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:04:17.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:17 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:04:17.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:17 smithi135 ceph-mon[33238]: pgmap v68: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:04:17.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:17 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:04:17.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:17 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:04:17.808 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:17.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:17.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:04:17.910 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:04:17.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:17.932 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:17.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:04:18.033 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:04:18.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:18.061 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:18.062 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:04:18.162 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:04:18.188 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:18.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:18.189 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:04:18.290 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:04:18.315 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:18.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:18.317 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:04:18.418 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:04:18.439 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:18.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:18.440 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:04:18.440 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:04:18 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:04:18.541 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:04:18.562 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:18.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:18.563 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:04:18.664 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:04:18.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:18.686 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:18.687 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:04:18.787 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:04:18.809 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:18.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:18.811 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:04:18.911 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:04:18.933 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:18.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:18.934 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:04:18.994 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:18 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:04:18.994 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:18 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:04:19.035 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:04:19.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:18 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:04:19.055 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:19.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:19.056 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:04:19.157 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:04:19.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:19.182 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:19.182 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:04:19.283 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:04:19.304 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:19.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:19.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:04:19.406 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:04:19.429 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:19.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:19.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:04:19.531 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:04:19.557 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:19.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:19.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:04:19.646 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-17T07:04:19.646 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:04:19.658 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:04:19.680 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:19.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:19.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:04:19.782 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:04:19.798 INFO:teuthology.orchestra.run.smithi100.stderr:pool 'unique_pool_0' created 2024-09-17T07:04:19.809 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:19.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:19.810 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:04:19.912 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:04:19.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:19.934 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:19.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:04:20.035 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:04:20.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:19 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/67339130' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-09-17T07:04:20.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:19 smithi135 ceph-mon[33238]: pgmap v69: 132 pgs: 132 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T07:04:20.057 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:20.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:20.059 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:04:20.126 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:19 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/67339130' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-09-17T07:04:20.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:19 smithi100 ceph-mon[29092]: pgmap v69: 132 pgs: 132 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T07:04:20.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:19 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/67339130' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-09-17T07:04:20.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:19 smithi100 ceph-mon[39462]: pgmap v69: 132 pgs: 132 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T07:04:20.159 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:04:20.181 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:20.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:20.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:04:20.276 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-17T07:04:20.276 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T07:04:20.283 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:04:20.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:20.305 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:20.306 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:04:20.406 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:04:20.428 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:20.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:20.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:04:20.530 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:04:20.552 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:20.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:20.553 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:04:20.600 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-09-17T07:04:20.654 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:04:20.674 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:20.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:20.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:04:20.776 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:04:20.802 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:20.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:20.803 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:04:20.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:20 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/67339130' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-17T07:04:20.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:20 smithi100 ceph-mon[29092]: osdmap e99: 8 total, 8 up, 8 in 2024-09-17T07:04:20.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:20 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/67339130' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-17T07:04:20.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:20 smithi100 ceph-mon[39462]: osdmap e99: 8 total, 8 up, 8 in 2024-09-17T07:04:20.903 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:04:20.925 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:20.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:20.926 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:04:21.027 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:04:21.048 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:21.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:21.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:04:21.150 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:04:21.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:21.174 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:21.174 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:04:21.275 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:04:21.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:20 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/67339130' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-17T07:04:21.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:20 smithi135 ceph-mon[33238]: osdmap e99: 8 total, 8 up, 8 in 2024-09-17T07:04:21.296 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:21.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:21.297 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:04:21.398 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:04:21.420 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:21.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:21.421 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:04:21.521 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:04:21.544 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:21.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:21.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:04:21.649 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:04:21.672 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:21.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:21.673 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:04:21.774 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:04:21.795 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:21.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:21.796 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:04:21.811 INFO:teuthology.orchestra.run.smithi100.stderr:nodeep-scrub is set 2024-09-17T07:04:21.898 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:04:21.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:21.921 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:21.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:04:22.022 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:04:22.044 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:22.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:22.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:04:22.126 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:21 smithi100 ceph-mon[29092]: osdmap e100: 8 total, 8 up, 8 in 2024-09-17T07:04:22.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:21 smithi100 ceph-mon[29092]: pgmap v72: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-17T07:04:22.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:21 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2054430713' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:04:22.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:21 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:04:22.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:21 smithi100 ceph-mon[39462]: osdmap e100: 8 total, 8 up, 8 in 2024-09-17T07:04:22.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:21 smithi100 ceph-mon[39462]: pgmap v72: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-17T07:04:22.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:21 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2054430713' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:04:22.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:21 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:04:22.146 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:04:22.171 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:22.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:22.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:04:22.272 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:04:22.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:21 smithi135 ceph-mon[33238]: osdmap e100: 8 total, 8 up, 8 in 2024-09-17T07:04:22.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:21 smithi135 ceph-mon[33238]: pgmap v72: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-17T07:04:22.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:21 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2054430713' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:04:22.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:21 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:04:22.297 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:22.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:22.298 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:04:22.398 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:04:22.422 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:22.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:22.423 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:04:22.524 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:04:22.545 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:22.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:22.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:04:22.647 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:04:22.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:22.670 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:22.670 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:04:22.771 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:04:22.796 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:22.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:22.797 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:04:22.805 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-17T07:04:22.897 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:04:22.918 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:22.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:22.919 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:04:23.020 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:04:23.041 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:23.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:23.042 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:04:23.083 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:22 smithi100 ceph-mon[39462]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-17T07:04:23.083 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:22 smithi100 ceph-mon[39462]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:04:23.083 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:22 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T07:04:23.083 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:22 smithi100 ceph-mon[39462]: osdmap e101: 8 total, 8 up, 8 in 2024-09-17T07:04:23.084 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:22 smithi100 ceph-mon[29092]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-17T07:04:23.084 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:22 smithi100 ceph-mon[29092]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:04:23.084 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:22 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T07:04:23.084 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:22 smithi100 ceph-mon[29092]: osdmap e101: 8 total, 8 up, 8 in 2024-09-17T07:04:23.144 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:04:23.166 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:23.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:23.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:04:23.268 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:04:23.282 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:04:23.283 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:04:23.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:22 smithi135 ceph-mon[33238]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-17T07:04:23.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:22 smithi135 ceph-mon[33238]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:04:23.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:22 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T07:04:23.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:22 smithi135 ceph-mon[33238]: osdmap e101: 8 total, 8 up, 8 in 2024-09-17T07:04:23.295 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:23.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:23.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:04:23.298 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1: {} 2024-09-17T07:04:23.299 INFO:teuthology.orchestra.run.smithi100.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:04:23.314 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:04:23.314 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:04:23.329 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:04:23.330 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:04:23.346 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:04:23.347 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:04:23.364 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:04:23.366 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:04:23.366 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:04:23.385 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:04:23.385 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:04:23.396 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:04:23.401 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:04:23.401 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:04:23.418 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:23.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:23.419 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:04:23.520 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:04:23.547 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:23.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:23.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:04:23.649 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:04:23.673 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:23.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:23.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:04:23.775 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:04:23.797 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:23.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:23.800 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:04:23.901 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:04:23.927 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:23.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:23.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:04:24.029 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:04:24.050 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:24.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:24.051 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:04:24.126 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:23 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:04:24.126 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:23 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:04:24.126 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:23 smithi100 ceph-mon[39462]: osdmap e102: 8 total, 8 up, 8 in 2024-09-17T07:04:24.126 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:23 smithi100 ceph-mon[39462]: pgmap v75: 148 pgs: 148 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:04:24.126 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:23 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/599191454' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:04:24.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:23 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:04:24.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:23 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:04:24.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:23 smithi100 ceph-mon[29092]: osdmap e102: 8 total, 8 up, 8 in 2024-09-17T07:04:24.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:23 smithi100 ceph-mon[29092]: pgmap v75: 148 pgs: 148 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:04:24.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:23 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/599191454' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:04:24.152 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:04:24.173 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:24.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:24.175 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:04:24.275 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:04:24.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:23 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:04:24.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:23 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:04:24.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:23 smithi135 ceph-mon[33238]: osdmap e102: 8 total, 8 up, 8 in 2024-09-17T07:04:24.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:23 smithi135 ceph-mon[33238]: pgmap v75: 148 pgs: 148 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:04:24.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:23 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/599191454' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:04:24.301 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:24.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:24.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:04:24.403 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:04:24.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:24.426 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:24.426 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:04:24.527 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:04:24.529 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:04:24.549 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:24.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:24.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:04:24.650 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:04:24.672 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:24.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:24.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:04:24.774 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:04:24.799 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:24.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:24.800 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:04:24.901 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:04:24.923 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:24.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:24.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:04:25.025 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:04:25.048 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:25.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:25.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:04:25.150 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:04:25.175 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:25.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:25.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:04:25.277 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:04:25.298 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:25.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:25.299 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:04:25.400 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:04:25.425 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:25.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:25.426 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:04:25.527 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:04:25.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:25.553 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:25.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:04:25.654 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:04:25.680 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:25.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:25.681 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:04:25.749 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T07:04:25.781 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:04:25.813 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:25.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:25.814 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:04:25.915 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:04:25.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:25.942 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:25.942 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:04:26.043 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:04:26.064 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:26.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:26.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:04:26.166 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:04:26.192 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:26.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:26.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:04:26.194 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:26 smithi100 ceph-mon[29092]: pgmap v76: 148 pgs: 148 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 980 B/s rd, 0 op/s 2024-09-17T07:04:26.195 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:26 smithi100 ceph-mon[39462]: pgmap v76: 148 pgs: 148 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 980 B/s rd, 0 op/s 2024-09-17T07:04:26.195 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:04:25 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:04:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:04:26.295 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:04:26.316 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:26.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:26.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:04:26.418 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:04:26.441 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:26.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:26.442 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:04:26.463 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:04:26.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:26 smithi135 ceph-mon[33238]: pgmap v76: 148 pgs: 148 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 980 B/s rd, 0 op/s 2024-09-17T07:04:26.543 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:04:26.565 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:26.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:26.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:04:26.666 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:04:26.688 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:26.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:26.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:04:26.790 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:04:26.815 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:26.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:26.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:04:26.917 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:04:26.942 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:26.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:26.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:04:27.044 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:04:27.069 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:27.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:27.070 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:04:27.171 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:04:27.193 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:27.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:27.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:04:27.279 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-17T07:04:27.279 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-17T07:04:27.279 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-17T07:04:27.279 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-17T07:04:27.279 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-17T07:04:27.287 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:04:27.295 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:04:27.319 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:27.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:27.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:04:27.421 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:04:27.442 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:27.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:27.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:04:27.544 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:04:27.570 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:27.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:27.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:04:27.663 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:04:27.671 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:04:27.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:27.698 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:27.698 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:04:27.799 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:04:27.820 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:27.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:27.821 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:04:27.922 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:04:27.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:27.945 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:27.945 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:04:28.046 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:04:28.070 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:28.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:28.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:04:28.081 INFO:teuthology.orchestra.run.smithi100.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-09-17T07:04:28.084 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-17T07:04:28.172 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:04:28.198 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:28.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:28.199 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:04:28.300 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:04:28.322 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:28.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:28.323 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:04:28.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:28 smithi100 ceph-mon[29092]: pgmap v77: 148 pgs: 148 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 822 B/s rd, 0 op/s 2024-09-17T07:04:28.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:28 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/3594624549' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-17T07:04:28.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:28 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-17T07:04:28.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:28 smithi100 ceph-mon[39462]: pgmap v77: 148 pgs: 148 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 822 B/s rd, 0 op/s 2024-09-17T07:04:28.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:28 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/3594624549' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-17T07:04:28.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:28 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-17T07:04:28.424 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:04:28.449 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:28.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:28.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:04:28.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:28 smithi135 ceph-mon[33238]: pgmap v77: 148 pgs: 148 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 822 B/s rd, 0 op/s 2024-09-17T07:04:28.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:28 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/3594624549' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-17T07:04:28.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:28 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-17T07:04:28.544 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:04:28 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:04:28.551 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:04:28.574 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:28.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:28.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:04:28.676 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:04:28.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:28.698 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:28.698 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:04:28.799 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:04:28.821 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:28.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:28.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:04:28.923 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:04:28.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:28.947 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:28.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:04:29.048 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:04:29.070 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:29.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:29.072 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:04:29.172 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:04:29.196 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:29.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:29.197 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:04:29.297 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:04:29.319 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:29.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:29.320 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:04:29.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:29 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-17T07:04:29.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:29 smithi100 ceph-mon[29092]: osdmap e103: 8 total, 8 up, 8 in 2024-09-17T07:04:29.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:29 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:04:29.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:29 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-17T07:04:29.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:29 smithi100 ceph-mon[39462]: osdmap e103: 8 total, 8 up, 8 in 2024-09-17T07:04:29.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:29 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:04:29.421 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:04:29.446 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:29.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:29.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:04:29.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:29 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-17T07:04:29.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:29 smithi135 ceph-mon[33238]: osdmap e103: 8 total, 8 up, 8 in 2024-09-17T07:04:29.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:29 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:04:29.548 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:04:29.574 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:29.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:29.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:04:29.676 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:04:29.699 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:29.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:29.700 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:04:29.801 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:04:29.827 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:29.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:29.828 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:04:29.928 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:04:29.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:29.954 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:29.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:04:30.055 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:04:30.079 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:30.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:30.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:04:30.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:30 smithi100 ceph-mon[29092]: pgmap v79: 148 pgs: 148 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-17T07:04:30.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:30 smithi100 ceph-mon[29092]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-17T07:04:30.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:30 smithi100 ceph-mon[39462]: pgmap v79: 148 pgs: 148 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-17T07:04:30.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:30 smithi100 ceph-mon[39462]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-17T07:04:30.182 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:04:30.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:30.205 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:30.206 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:04:30.306 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:04:30.332 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:30.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:30.334 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:04:30.435 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:04:30.460 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:30.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:30.462 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:04:30.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:30 smithi135 ceph-mon[33238]: pgmap v79: 148 pgs: 148 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-17T07:04:30.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:30 smithi135 ceph-mon[33238]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-17T07:04:30.563 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:04:30.579 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:04:30.588 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:30.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:30.589 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:04:30.689 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:04:30.715 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:30.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:30.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:04:30.817 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:04:30.843 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:30.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:30.845 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:04:30.946 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:04:30.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:30.971 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:30.972 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:04:31.072 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:04:31.097 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:31.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:31.098 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:04:31.199 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:04:31.223 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:31.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:31.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:04:31.325 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:04:31.346 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:31.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:31.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:04:31.448 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:04:31.470 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:31.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:31.471 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:04:31.559 DEBUG:teuthology.orchestra.run.smithi100:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_0 bench 90 write' 2024-09-17T07:04:31.561 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2024-09-17T07:04:31.561 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T07:04:31.563 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-17T07:04:31.563 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-17T07:04:31.572 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:04:31.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:31.628 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:31.628 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:04:31.634 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:hints = 1 2024-09-17T07:04:31.634 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-17T07:04:31.634 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Object prefix: benchmark_data_smithi100_78556 2024-09-17T07:04:31.634 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-17T07:04:31.635 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 0 0 0 0 0 0 - 0 2024-09-17T07:04:31.701 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:04:31.701 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:04:31.729 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:04:31.736 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1: {} 2024-09-17T07:04:31.737 INFO:teuthology.orchestra.run.smithi100.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:04:31.770 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:31.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:31.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:04:31.772 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:04:31.772 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:04:31.811 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:04:31.811 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:04:31.851 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:04:31.851 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:04:31.873 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:04:31.889 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:04:31.890 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:04:31.901 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:31.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:31.902 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:04:31.925 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:04:31.925 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:04:31.960 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:04:31.960 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:04:32.003 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:04:32.041 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:32.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:32.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:04:32.143 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:04:32.188 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:32.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:32.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:04:32.290 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:04:32.320 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:32.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:32.321 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:04:32.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:32 smithi100 ceph-mon[29092]: pgmap v80: 148 pgs: 148 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:04:32.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:32 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2437272568' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:04:32.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:32 smithi100 ceph-mon[39462]: pgmap v80: 148 pgs: 148 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:04:32.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:32 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2437272568' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:04:32.422 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:04:32.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:32.461 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:32.461 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:04:32.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:32 smithi135 ceph-mon[33238]: pgmap v80: 148 pgs: 148 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:04:32.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:32 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2437272568' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:04:32.563 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:04:32.600 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:32.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:32.602 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:04:32.634 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 1 16 3285 3269 204.301 204.312 0.00554066 0.00487786 2024-09-17T07:04:32.703 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:04:32.741 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:32.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:32.742 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:04:32.843 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:04:32.879 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:32.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:32.881 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:04:32.982 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:04:33.023 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:33.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:33.024 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:04:33.119 INFO:teuthology.orchestra.run.smithi100.stderr:noscrub is unset 2024-09-17T07:04:33.125 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:04:33.165 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:33.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:33.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:04:33.267 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:04:33.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:33.306 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:33.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:04:33.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:33 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:04:33.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:33 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/3610542391' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:04:33.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:33 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:04:33.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:33 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/3610542391' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:04:33.407 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:04:33.438 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:33.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:33.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:04:33.540 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:04:33.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:33 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:04:33.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:33 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/3610542391' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:04:33.574 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:33.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:33.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:04:33.634 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 2 16 6798 6782 211.924 219.562 0.0069027 0.00470779 2024-09-17T07:04:33.675 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:04:33.716 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:33.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:33.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:04:33.818 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:04:33.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:33.859 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:33.859 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:04:33.960 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:04:33.991 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:33.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:33.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:04:34.093 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:04:34.121 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:34.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:34.122 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:04:34.223 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:04:34.248 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:34.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:34.249 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:04:34.350 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:04:34.379 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:34 smithi100 ceph-mon[29092]: pgmap v81: 148 pgs: 148 active+clean; 292 MiB data, 798 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 29 MiB/s wr, 466 op/s 2024-09-17T07:04:34.379 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:34 smithi100 ceph-mon[29092]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:04:34.379 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:34 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/3610542391' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T07:04:34.379 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:34 smithi100 ceph-mon[29092]: osdmap e104: 8 total, 8 up, 8 in 2024-09-17T07:04:34.379 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:34 smithi100 ceph-mon[29092]: 6.14 scrub starts 2024-09-17T07:04:34.379 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:34 smithi100 ceph-mon[29092]: 6.14 scrub ok 2024-09-17T07:04:34.380 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:34 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:04:34.380 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:34 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:04:34.380 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:34 smithi100 ceph-mon[29092]: 5.e scrub starts 2024-09-17T07:04:34.380 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:34 smithi100 ceph-mon[29092]: 5.e scrub ok 2024-09-17T07:04:34.380 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:34 smithi100 ceph-mon[39462]: pgmap v81: 148 pgs: 148 active+clean; 292 MiB data, 798 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 29 MiB/s wr, 466 op/s 2024-09-17T07:04:34.381 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:34 smithi100 ceph-mon[39462]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:04:34.381 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:34 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/3610542391' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T07:04:34.381 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:34 smithi100 ceph-mon[39462]: osdmap e104: 8 total, 8 up, 8 in 2024-09-17T07:04:34.381 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:34 smithi100 ceph-mon[39462]: 6.14 scrub starts 2024-09-17T07:04:34.381 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:34 smithi100 ceph-mon[39462]: 6.14 scrub ok 2024-09-17T07:04:34.381 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:34 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:04:34.381 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:34 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:04:34.381 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:34 smithi100 ceph-mon[39462]: 5.e scrub starts 2024-09-17T07:04:34.381 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:34 smithi100 ceph-mon[39462]: 5.e scrub ok 2024-09-17T07:04:34.388 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:34.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:34.389 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:04:34.483 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:04:34.490 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:04:34.530 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:34.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:34.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:04:34.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:34 smithi135 ceph-mon[33238]: pgmap v81: 148 pgs: 148 active+clean; 292 MiB data, 798 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 29 MiB/s wr, 466 op/s 2024-09-17T07:04:34.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:34 smithi135 ceph-mon[33238]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:04:34.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:34 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/3610542391' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T07:04:34.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:34 smithi135 ceph-mon[33238]: osdmap e104: 8 total, 8 up, 8 in 2024-09-17T07:04:34.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:34 smithi135 ceph-mon[33238]: 6.14 scrub starts 2024-09-17T07:04:34.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:34 smithi135 ceph-mon[33238]: 6.14 scrub ok 2024-09-17T07:04:34.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:34 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:04:34.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:34 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:04:34.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:34 smithi135 ceph-mon[33238]: 5.e scrub starts 2024-09-17T07:04:34.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:34 smithi135 ceph-mon[33238]: 5.e scrub ok 2024-09-17T07:04:34.632 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:04:34.635 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 3 15 9952 9937 207.006 197.188 0.00642902 0.00482276 2024-09-17T07:04:34.672 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:34.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:34.673 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:04:34.774 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:04:34.805 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:34.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:34.806 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:04:34.906 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:04:34.934 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:34.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:34.935 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:04:34.992 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:04:35.036 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:04:35.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:35.065 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:35.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:04:35.166 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:04:35.213 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:35.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:35.214 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:04:35.315 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:04:35.344 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:35.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:35.345 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:04:35.444 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:35 smithi100 ceph-mon[29092]: osdmap e105: 8 total, 8 up, 8 in 2024-09-17T07:04:35.444 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:35 smithi100 ceph-mon[29092]: 2.3 scrub starts 2024-09-17T07:04:35.444 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:35 smithi100 ceph-mon[29092]: 2.3 scrub ok 2024-09-17T07:04:35.445 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:35 smithi100 ceph-mon[39462]: osdmap e105: 8 total, 8 up, 8 in 2024-09-17T07:04:35.445 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:35 smithi100 ceph-mon[39462]: 2.3 scrub starts 2024-09-17T07:04:35.445 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:35 smithi100 ceph-mon[39462]: 2.3 scrub ok 2024-09-17T07:04:35.446 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:04:35.473 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:35.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:35.474 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:04:35.512 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-17T07:04:35.512 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:04:35.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:35 smithi135 ceph-mon[33238]: osdmap e105: 8 total, 8 up, 8 in 2024-09-17T07:04:35.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:35 smithi135 ceph-mon[33238]: 2.3 scrub starts 2024-09-17T07:04:35.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:35 smithi135 ceph-mon[33238]: 2.3 scrub ok 2024-09-17T07:04:35.575 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:04:35.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:35.613 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:35.613 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:04:35.635 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 4 16 12836 12820 200.294 180.188 0.00527619 0.00498493 2024-09-17T07:04:35.714 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:04:35.753 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:35.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:35.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:04:35.855 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:04:35.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:35.896 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:35.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:04:35.997 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:04:36.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:36.031 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:36.031 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:04:36.126 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:04:35 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:04:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:04:36.132 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:04:36.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:36.172 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:36.172 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:04:36.273 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:04:36.303 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:36.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:36.304 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:04:36.405 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:04:36.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:36.439 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:36.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:04:36.485 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:36 smithi100 ceph-mon[29092]: 2.19 scrub starts 2024-09-17T07:04:36.485 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:36 smithi100 ceph-mon[29092]: 2.19 scrub ok 2024-09-17T07:04:36.485 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:36 smithi100 ceph-mon[29092]: pgmap v84: 148 pgs: 148 active+clean; 292 MiB data, 798 MiB used, 714 GiB / 715 GiB avail; 737 B/s rd, 42 MiB/s wr, 671 op/s 2024-09-17T07:04:36.485 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:36 smithi100 ceph-mon[29092]: 4.1a scrub starts 2024-09-17T07:04:36.485 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:36 smithi100 ceph-mon[29092]: 4.1a scrub ok 2024-09-17T07:04:36.485 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:36 smithi100 ceph-mon[39462]: 2.19 scrub starts 2024-09-17T07:04:36.485 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:36 smithi100 ceph-mon[39462]: 2.19 scrub ok 2024-09-17T07:04:36.485 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:36 smithi100 ceph-mon[39462]: pgmap v84: 148 pgs: 148 active+clean; 292 MiB data, 798 MiB used, 714 GiB / 715 GiB avail; 737 B/s rd, 42 MiB/s wr, 671 op/s 2024-09-17T07:04:36.486 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:36 smithi100 ceph-mon[39462]: 4.1a scrub starts 2024-09-17T07:04:36.486 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:36 smithi100 ceph-mon[39462]: 4.1a scrub ok 2024-09-17T07:04:36.540 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:04:36.543 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-17T07:04:36.545 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:36 smithi135 ceph-mon[33238]: 2.19 scrub starts 2024-09-17T07:04:36.545 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:36 smithi135 ceph-mon[33238]: 2.19 scrub ok 2024-09-17T07:04:36.545 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:36 smithi135 ceph-mon[33238]: pgmap v84: 148 pgs: 148 active+clean; 292 MiB data, 798 MiB used, 714 GiB / 715 GiB avail; 737 B/s rd, 42 MiB/s wr, 671 op/s 2024-09-17T07:04:36.545 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:36 smithi135 ceph-mon[33238]: 4.1a scrub starts 2024-09-17T07:04:36.545 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:36 smithi135 ceph-mon[33238]: 4.1a scrub ok 2024-09-17T07:04:36.569 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:36.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:36.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:04:36.635 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 5 16 15750 15734 196.657 182.125 0.0045261 0.00507892 2024-09-17T07:04:36.670 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:04:36.701 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:36.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:36.702 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:04:36.803 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:04:36.847 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:36.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:36.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:04:36.918 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-17T07:04:36.949 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:04:37.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:37.010 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:37.011 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:04:37.111 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:04:37.157 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:37.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:37.158 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:04:37.238 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:04:37.259 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:04:37.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:37.322 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:37.322 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:04:37.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:37 smithi100 ceph-mon[29092]: 3.2 scrub starts 2024-09-17T07:04:37.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:37 smithi100 ceph-mon[29092]: 3.2 scrub ok 2024-09-17T07:04:37.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:37 smithi100 ceph-mon[29092]: 4.15 scrub starts 2024-09-17T07:04:37.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:37 smithi100 ceph-mon[29092]: 4.15 scrub ok 2024-09-17T07:04:37.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:37 smithi100 ceph-mon[39462]: 3.2 scrub starts 2024-09-17T07:04:37.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:37 smithi100 ceph-mon[39462]: 3.2 scrub ok 2024-09-17T07:04:37.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:37 smithi100 ceph-mon[39462]: 4.15 scrub starts 2024-09-17T07:04:37.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:37 smithi100 ceph-mon[39462]: 4.15 scrub ok 2024-09-17T07:04:37.423 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:04:37.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:37.456 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:37.457 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:04:37.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:37 smithi135 ceph-mon[33238]: 3.2 scrub starts 2024-09-17T07:04:37.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:37 smithi135 ceph-mon[33238]: 3.2 scrub ok 2024-09-17T07:04:37.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:37 smithi135 ceph-mon[33238]: 4.15 scrub starts 2024-09-17T07:04:37.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:37 smithi135 ceph-mon[33238]: 4.15 scrub ok 2024-09-17T07:04:37.557 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:04:37.586 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:37.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:37.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:04:37.635 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 6 16 18379 18363 191.264 164.312 0.00348743 0.00522248 2024-09-17T07:04:37.688 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:04:37.708 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:37.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:37.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:04:37.811 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:04:37.849 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:37.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:37.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:04:37.950 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:04:37.986 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:37.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:37.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:04:38.088 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:04:38.117 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:38.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:38.118 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:04:38.219 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:04:38.259 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:38.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:38.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:04:38.361 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:04:38.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:38.397 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:38.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:04:38.444 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:04:38 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:04:38.445 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:38 smithi135 ceph-mon[33238]: 5.17 scrub starts 2024-09-17T07:04:38.445 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:38 smithi135 ceph-mon[33238]: 5.17 scrub ok 2024-09-17T07:04:38.445 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:38 smithi135 ceph-mon[33238]: pgmap v85: 148 pgs: 148 active+clean; 292 MiB data, 798 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 36 MiB/s wr, 582 op/s 2024-09-17T07:04:38.445 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:38 smithi135 ceph-mon[33238]: 1.0 scrub starts 2024-09-17T07:04:38.445 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:38 smithi135 ceph-mon[33238]: 1.0 scrub ok 2024-09-17T07:04:38.498 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:04:38.514 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:38 smithi100 ceph-mon[39462]: 5.17 scrub starts 2024-09-17T07:04:38.514 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:38 smithi100 ceph-mon[39462]: 5.17 scrub ok 2024-09-17T07:04:38.514 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:38 smithi100 ceph-mon[39462]: pgmap v85: 148 pgs: 148 active+clean; 292 MiB data, 798 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 36 MiB/s wr, 582 op/s 2024-09-17T07:04:38.514 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:38 smithi100 ceph-mon[39462]: 1.0 scrub starts 2024-09-17T07:04:38.514 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:38 smithi100 ceph-mon[39462]: 1.0 scrub ok 2024-09-17T07:04:38.514 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:38 smithi100 ceph-mon[29092]: 5.17 scrub starts 2024-09-17T07:04:38.514 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:38 smithi100 ceph-mon[29092]: 5.17 scrub ok 2024-09-17T07:04:38.515 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:38 smithi100 ceph-mon[29092]: pgmap v85: 148 pgs: 148 active+clean; 292 MiB data, 798 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 36 MiB/s wr, 582 op/s 2024-09-17T07:04:38.515 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:38 smithi100 ceph-mon[29092]: 1.0 scrub starts 2024-09-17T07:04:38.515 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:38 smithi100 ceph-mon[29092]: 1.0 scrub ok 2024-09-17T07:04:38.526 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:38.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:38.527 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:04:38.628 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:04:38.635 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 7 16 20930 20914 186.715 159.438 0.0119247 0.00534789 2024-09-17T07:04:38.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:38.659 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:38.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:04:38.760 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:04:38.791 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:38.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:38.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:04:38.807 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:04:38.892 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:04:38.933 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:38.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:38.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:04:39.036 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:04:39.066 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:39.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:39.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:04:39.168 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:04:39.197 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:39 smithi100 ceph-mon[29092]: 2.7 scrub starts 2024-09-17T07:04:39.198 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:39 smithi100 ceph-mon[29092]: 2.7 scrub ok 2024-09-17T07:04:39.198 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:39 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:04:39.198 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:39 smithi100 ceph-mon[29092]: 4.1c scrub starts 2024-09-17T07:04:39.198 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:39 smithi100 ceph-mon[29092]: 4.1c scrub ok 2024-09-17T07:04:39.198 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:39 smithi100 ceph-mon[39462]: 2.7 scrub starts 2024-09-17T07:04:39.199 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:39 smithi100 ceph-mon[39462]: 2.7 scrub ok 2024-09-17T07:04:39.199 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:39 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:04:39.199 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:39 smithi100 ceph-mon[39462]: 4.1c scrub starts 2024-09-17T07:04:39.199 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:39 smithi100 ceph-mon[39462]: 4.1c scrub ok 2024-09-17T07:04:39.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:39.201 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:39.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:04:39.302 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:04:39.324 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:39.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:39.325 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:04:39.426 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:04:39.455 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:39.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:39.457 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:04:39.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:39 smithi135 ceph-mon[33238]: 2.7 scrub starts 2024-09-17T07:04:39.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:39 smithi135 ceph-mon[33238]: 2.7 scrub ok 2024-09-17T07:04:39.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:39 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:04:39.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:39 smithi135 ceph-mon[33238]: 4.1c scrub starts 2024-09-17T07:04:39.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:39 smithi135 ceph-mon[33238]: 4.1c scrub ok 2024-09-17T07:04:39.558 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:04:39.589 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:39.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:39.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:04:39.635 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 8 15 23334 23319 182.163 150.312 0.0121232 0.00548364 2024-09-17T07:04:39.692 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:04:39.723 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:39.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:39.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:04:39.825 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:04:39.853 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:39.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:39.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:04:39.955 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:04:40.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:40.002 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:40.002 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:04:40.103 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:04:40.132 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:40.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:40.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:04:40.232 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T07:04:40.236 INFO:teuthology.orchestra.run.smithi100.stderr:dumped all 2024-09-17T07:04:40.236 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:04:40.262 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:40.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:40.263 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:04:40.364 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:04:40.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:40.413 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:40.413 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:04:40.514 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:04:40.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:40 smithi135 ceph-mon[33238]: 4.1f scrub starts 2024-09-17T07:04:40.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:40 smithi135 ceph-mon[33238]: 4.1f scrub ok 2024-09-17T07:04:40.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:40 smithi135 ceph-mon[33238]: 4.1 scrub starts 2024-09-17T07:04:40.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:40 smithi135 ceph-mon[33238]: 4.1 scrub ok 2024-09-17T07:04:40.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:40 smithi135 ceph-mon[33238]: pgmap v86: 148 pgs: 148 active+clean; 1.2 GiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 151 MiB/s wr, 2.41k op/s 2024-09-17T07:04:40.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:40 smithi135 ceph-mon[33238]: 5.5 scrub starts 2024-09-17T07:04:40.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:40 smithi135 ceph-mon[33238]: 5.5 scrub ok 2024-09-17T07:04:40.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:40 smithi135 ceph-mon[33238]: 6.a scrub starts 2024-09-17T07:04:40.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:40 smithi135 ceph-mon[33238]: 6.a scrub ok 2024-09-17T07:04:40.550 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:40.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:40.552 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:04:40.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:40 smithi100 ceph-mon[29092]: 4.1f scrub starts 2024-09-17T07:04:40.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:40 smithi100 ceph-mon[29092]: 4.1f scrub ok 2024-09-17T07:04:40.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:40 smithi100 ceph-mon[29092]: 4.1 scrub starts 2024-09-17T07:04:40.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:40 smithi100 ceph-mon[29092]: 4.1 scrub ok 2024-09-17T07:04:40.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:40 smithi100 ceph-mon[29092]: pgmap v86: 148 pgs: 148 active+clean; 1.2 GiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 151 MiB/s wr, 2.41k op/s 2024-09-17T07:04:40.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:40 smithi100 ceph-mon[29092]: 5.5 scrub starts 2024-09-17T07:04:40.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:40 smithi100 ceph-mon[29092]: 5.5 scrub ok 2024-09-17T07:04:40.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:40 smithi100 ceph-mon[29092]: 6.a scrub starts 2024-09-17T07:04:40.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:40 smithi100 ceph-mon[29092]: 6.a scrub ok 2024-09-17T07:04:40.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:40 smithi100 ceph-mon[39462]: 4.1f scrub starts 2024-09-17T07:04:40.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:40 smithi100 ceph-mon[39462]: 4.1f scrub ok 2024-09-17T07:04:40.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:40 smithi100 ceph-mon[39462]: 4.1 scrub starts 2024-09-17T07:04:40.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:40 smithi100 ceph-mon[39462]: 4.1 scrub ok 2024-09-17T07:04:40.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:40 smithi100 ceph-mon[39462]: pgmap v86: 148 pgs: 148 active+clean; 1.2 GiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 151 MiB/s wr, 2.41k op/s 2024-09-17T07:04:40.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:40 smithi100 ceph-mon[39462]: 5.5 scrub starts 2024-09-17T07:04:40.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:40 smithi100 ceph-mon[39462]: 5.5 scrub ok 2024-09-17T07:04:40.630 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:40 smithi100 ceph-mon[39462]: 6.a scrub starts 2024-09-17T07:04:40.630 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:40 smithi100 ceph-mon[39462]: 6.a scrub ok 2024-09-17T07:04:40.635 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 9 16 25617 25601 177.766 142.625 0.00363611 0.00562112 2024-09-17T07:04:40.654 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:04:40.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:40.696 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:40.696 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:04:40.796 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:04:40.839 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:40.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:40.840 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:04:40.941 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:04:40.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:40.982 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:40.982 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:04:41.083 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:04:41.120 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:41.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:41.121 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:04:41.222 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:04:41.255 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:41.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:41.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:04:41.357 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:04:41.361 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:04:41.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:41 smithi100 ceph-mon[29092]: 5.18 scrub starts 2024-09-17T07:04:41.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:41 smithi100 ceph-mon[29092]: 5.18 scrub ok 2024-09-17T07:04:41.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:41 smithi100 ceph-mon[29092]: 6.19 scrub starts 2024-09-17T07:04:41.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:41 smithi100 ceph-mon[29092]: 6.19 scrub ok 2024-09-17T07:04:41.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:41 smithi100 ceph-mon[29092]: 4.f scrub starts 2024-09-17T07:04:41.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:41 smithi100 ceph-mon[29092]: 4.f scrub ok 2024-09-17T07:04:41.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:41 smithi100 ceph-mon[39462]: 5.18 scrub starts 2024-09-17T07:04:41.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:41 smithi100 ceph-mon[39462]: 5.18 scrub ok 2024-09-17T07:04:41.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:41 smithi100 ceph-mon[39462]: 6.19 scrub starts 2024-09-17T07:04:41.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:41 smithi100 ceph-mon[39462]: 6.19 scrub ok 2024-09-17T07:04:41.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:41 smithi100 ceph-mon[39462]: 4.f scrub starts 2024-09-17T07:04:41.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:41 smithi100 ceph-mon[39462]: 4.f scrub ok 2024-09-17T07:04:41.384 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:41.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:41.385 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:04:41.486 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:04:41.517 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:41.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:41.518 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:04:41.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:41 smithi135 ceph-mon[33238]: 5.18 scrub starts 2024-09-17T07:04:41.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:41 smithi135 ceph-mon[33238]: 5.18 scrub ok 2024-09-17T07:04:41.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:41 smithi135 ceph-mon[33238]: 6.19 scrub starts 2024-09-17T07:04:41.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:41 smithi135 ceph-mon[33238]: 6.19 scrub ok 2024-09-17T07:04:41.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:41 smithi135 ceph-mon[33238]: 4.f scrub starts 2024-09-17T07:04:41.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:41 smithi135 ceph-mon[33238]: 4.f scrub ok 2024-09-17T07:04:41.618 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:04:41.635 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 10 16 27974 27958 174.719 147.312 0.00391305 0.00571727 2024-09-17T07:04:41.648 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:41.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:41.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:04:41.750 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:04:41.778 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:41.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:41.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:04:41.880 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:04:41.920 INFO:teuthology.orchestra.run.smithi100.stdout:{"pg_ready":true,"pg_map":{"version":86,"stamp":"2024-09-17T07:04:39.020944+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1264130326,"num_objects":19476,"num_object_clones":0,"num_object_copies":58428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19476,"num_whiteouts":0,"num_read":1060,"num_read_kb":906,"num_write":19889,"num_write_kb":1236379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1771214,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19894,"ondisk_log_size":19894,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":3760472,"kb_used_data":3478744,"kb_used_omap":12,"kb_used_meta":281587,"kb_avail":746200744,"statfs":{"total":767960285184,"available":764109561856,"internally_reserved":0,"allocated":3562233856,"data_stored":3559053983,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":288345656},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1263534080,"num_objects":19280,"num_object_clones":0,"num_object_copies":57840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19280,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":19280,"num_write_kb":1233920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001905"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":51,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.117286+0000","last_change":"2024-09-17T07:04:03.551432+0000","last_active":"2024-09-17T07:04:33.117286+0000","last_peered":"2024-09-17T07:04:33.117286+0000","last_clean":"2024-09-17T07:04:33.117286+0000","last_became_active":"2024-09-17T07:02:18.365773+0000","last_became_peered":"2024-09-17T07:02:18.365773+0000","last_unstale":"2024-09-17T07:04:33.117286+0000","last_undegraded":"2024-09-17T07:04:33.117286+0000","last_fullsized":"2024-09-17T07:04:33.117286+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:03.551314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:04:03.551314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:12.891955+0000","scrub_duration":0.001762752,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":59,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.112789+0000","last_change":"2024-09-17T07:03:41.931873+0000","last_active":"2024-09-17T07:04:33.112789+0000","last_peered":"2024-09-17T07:04:33.112789+0000","last_clean":"2024-09-17T07:04:33.112789+0000","last_became_active":"2024-09-17T07:02:12.006754+0000","last_became_peered":"2024-09-17T07:02:12.006754+0000","last_unstale":"2024-09-17T07:04:33.112789+0000","last_undegraded":"2024-09-17T07:04:33.112789+0000","last_fullsized":"2024-09-17T07:04:33.112789+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:41.931780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:41.931780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:50.138892+0000","scrub_duration":0.0017747419999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"75'15","reported_seq":80,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.118648+0000","last_change":"2024-09-17T07:03:30.436540+0000","last_active":"2024-09-17T07:04:33.118648+0000","last_peered":"2024-09-17T07:04:33.118648+0000","last_clean":"2024-09-17T07:04:33.118648+0000","last_became_active":"2024-09-17T07:02:14.012962+0000","last_became_peered":"2024-09-17T07:02:14.012962+0000","last_unstale":"2024-09-17T07:04:33.118648+0000","last_undegraded":"2024-09-17T07:04:33.118648+0000","last_fullsized":"2024-09-17T07:04:33.118648+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-09-17T07:03:30.436444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:30.436444+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:35.913395+0000","scrub_duration":0.0023734419999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":43,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.113689+0000","last_change":"2024-09-17T07:02:16.015244+0000","last_active":"2024-09-17T07:04:33.113689+0000","last_peered":"2024-09-17T07:04:33.113689+0000","last_clean":"2024-09-17T07:04:33.113689+0000","last_became_active":"2024-09-17T07:02:16.014928+0000","last_became_peered":"2024-09-17T07:02:16.014928+0000","last_unstale":"2024-09-17T07:04:33.113689+0000","last_undegraded":"2024-09-17T07:04:33.113689+0000","last_fullsized":"2024-09-17T07:04:33.113689+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:02:14.999435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":58,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.117979+0000","last_change":"2024-09-17T07:03:43.459940+0000","last_active":"2024-09-17T07:04:33.117979+0000","last_peered":"2024-09-17T07:04:33.117979+0000","last_clean":"2024-09-17T07:04:33.117979+0000","last_became_active":"2024-09-17T07:02:12.010069+0000","last_became_peered":"2024-09-17T07:02:12.010069+0000","last_unstale":"2024-09-17T07:04:33.117979+0000","last_undegraded":"2024-09-17T07:04:33.117979+0000","last_fullsized":"2024-09-17T07:04:33.117979+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:43.459821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:43.459821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:44.534548+0000","scrub_duration":0.0015265960000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"75'9","reported_seq":72,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.113839+0000","last_change":"2024-09-17T07:03:17.864856+0000","last_active":"2024-09-17T07:04:33.113839+0000","last_peered":"2024-09-17T07:04:33.113839+0000","last_clean":"2024-09-17T07:04:33.113839+0000","last_became_active":"2024-09-17T07:02:14.016244+0000","last_became_peered":"2024-09-17T07:02:14.016244+0000","last_unstale":"2024-09-17T07:04:33.113839+0000","last_undegraded":"2024-09-17T07:04:33.113839+0000","last_fullsized":"2024-09-17T07:04:33.113839+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:03:17.864726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:17.864726+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021756380000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":53,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.119142+0000","last_change":"2024-09-17T07:03:37.113047+0000","last_active":"2024-09-17T07:04:33.119142+0000","last_peered":"2024-09-17T07:04:33.119142+0000","last_clean":"2024-09-17T07:04:33.119142+0000","last_became_active":"2024-09-17T07:02:16.014380+0000","last_became_peered":"2024-09-17T07:02:16.014380+0000","last_unstale":"2024-09-17T07:04:33.119142+0000","last_undegraded":"2024-09-17T07:04:33.119142+0000","last_fullsized":"2024-09-17T07:04:33.119142+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:37.112954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:37.112954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:45.057903+0000","scrub_duration":0.001466824,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":52,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.113956+0000","last_change":"2024-09-17T07:03:38.109357+0000","last_active":"2024-09-17T07:04:33.113956+0000","last_peered":"2024-09-17T07:04:33.113956+0000","last_clean":"2024-09-17T07:04:33.113956+0000","last_became_active":"2024-09-17T07:02:18.020790+0000","last_became_peered":"2024-09-17T07:02:18.020790+0000","last_unstale":"2024-09-17T07:04:33.113956+0000","last_undegraded":"2024-09-17T07:04:33.113956+0000","last_fullsized":"2024-09-17T07:04:33.113956+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:38.109265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:03:38.109265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:57.231258+0000","scrub_duration":0.001349659,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":58,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.112746+0000","last_change":"2024-09-17T07:03:43.060196+0000","last_active":"2024-09-17T07:04:33.112746+0000","last_peered":"2024-09-17T07:04:33.112746+0000","last_clean":"2024-09-17T07:04:33.112746+0000","last_became_active":"2024-09-17T07:02:12.011129+0000","last_became_peered":"2024-09-17T07:02:12.011129+0000","last_unstale":"2024-09-17T07:04:33.112746+0000","last_undegraded":"2024-09-17T07:04:33.112746+0000","last_fullsized":"2024-09-17T07:04:33.112746+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:43.060102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:43.060102+0000","last_clean_scrub_stamp":"2024-09-17T07:03:43.060102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:09.780017+0000","scrub_duration":0.0012238170000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"75'5","reported_seq":66,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.113950+0000","last_change":"2024-09-17T07:03:29.002445+0000","last_active":"2024-09-17T07:04:33.113950+0000","last_peered":"2024-09-17T07:04:33.113950+0000","last_clean":"2024-09-17T07:04:33.113950+0000","last_became_active":"2024-09-17T07:02:14.016640+0000","last_became_peered":"2024-09-17T07:02:14.016640+0000","last_unstale":"2024-09-17T07:04:33.113950+0000","last_undegraded":"2024-09-17T07:04:33.113950+0000","last_fullsized":"2024-09-17T07:04:33.113950+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-09-17T07:03:29.002312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:29.002312+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:44.000292+0000","scrub_duration":0.002172748,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":54,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.111691+0000","last_change":"2024-09-17T07:03:22.015629+0000","last_active":"2024-09-17T07:04:33.111691+0000","last_peered":"2024-09-17T07:04:33.111691+0000","last_clean":"2024-09-17T07:04:33.111691+0000","last_became_active":"2024-09-17T07:02:16.015401+0000","last_became_peered":"2024-09-17T07:02:16.015401+0000","last_unstale":"2024-09-17T07:04:33.111691+0000","last_undegraded":"2024-09-17T07:04:33.111691+0000","last_fullsized":"2024-09-17T07:04:33.111691+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:22.015535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:22.015535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:49.909183+0000","scrub_duration":0.001507752,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":51,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.118485+0000","last_change":"2024-09-17T07:03:34.882046+0000","last_active":"2024-09-17T07:04:33.118485+0000","last_peered":"2024-09-17T07:04:33.118485+0000","last_clean":"2024-09-17T07:04:33.118485+0000","last_became_active":"2024-09-17T07:02:18.023680+0000","last_became_peered":"2024-09-17T07:02:18.023680+0000","last_unstale":"2024-09-17T07:04:33.118485+0000","last_undegraded":"2024-09-17T07:04:33.118485+0000","last_fullsized":"2024-09-17T07:04:33.118485+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:34.881951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:03:34.881951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:36.646147+0000","scrub_duration":0.0014520379999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":58,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.112633+0000","last_change":"2024-09-17T07:03:45.029350+0000","last_active":"2024-09-17T07:04:33.112633+0000","last_peered":"2024-09-17T07:04:33.112633+0000","last_clean":"2024-09-17T07:04:33.112633+0000","last_became_active":"2024-09-17T07:02:12.012234+0000","last_became_peered":"2024-09-17T07:02:12.012234+0000","last_unstale":"2024-09-17T07:04:33.112633+0000","last_undegraded":"2024-09-17T07:04:33.112633+0000","last_fullsized":"2024-09-17T07:04:33.112633+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:45.029247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:45.029247+0000","last_clean_scrub_stamp":"2024-09-17T07:03:45.029247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:07.678537+0000","scrub_duration":0.001377554,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"75'9","reported_seq":72,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T07:04:35.582548+0000","last_change":"2024-09-17T07:04:35.582659+0000","last_active":"2024-09-17T07:04:35.582548+0000","last_peered":"2024-09-17T07:04:35.582548+0000","last_clean":"2024-09-17T07:04:35.582548+0000","last_became_active":"2024-09-17T07:02:14.013337+0000","last_became_peered":"2024-09-17T07:02:14.013337+0000","last_unstale":"2024-09-17T07:04:35.582548+0000","last_undegraded":"2024-09-17T07:04:35.582548+0000","last_fullsized":"2024-09-17T07:04:35.582548+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:04:35.582524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:04:35.582524+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:06:04.967853+0000","scrub_duration":0.0042430649999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":53,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.118749+0000","last_change":"2024-09-17T07:03:59.693315+0000","last_active":"2024-09-17T07:04:33.118749+0000","last_peered":"2024-09-17T07:04:33.118749+0000","last_clean":"2024-09-17T07:04:33.118749+0000","last_became_active":"2024-09-17T07:02:16.023774+0000","last_became_peered":"2024-09-17T07:02:16.023774+0000","last_unstale":"2024-09-17T07:04:33.118749+0000","last_undegraded":"2024-09-17T07:04:33.118749+0000","last_fullsized":"2024-09-17T07:04:33.118749+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:59.693199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:59.693199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:27.090136+0000","scrub_duration":0.001514306,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":52,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.111378+0000","last_change":"2024-09-17T07:04:07.066830+0000","last_active":"2024-09-17T07:04:33.111378+0000","last_peered":"2024-09-17T07:04:33.111378+0000","last_clean":"2024-09-17T07:04:33.111378+0000","last_became_active":"2024-09-17T07:02:18.029178+0000","last_became_peered":"2024-09-17T07:02:18.029178+0000","last_unstale":"2024-09-17T07:04:33.111378+0000","last_undegraded":"2024-09-17T07:04:33.111378+0000","last_fullsized":"2024-09-17T07:04:33.111378+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:07.066751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:04:07.066751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:28.890050+0000","scrub_duration":0.0011842319999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"105'1223","reported_seq":1237,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T07:04:38.005852+0000","last_change":"2024-09-17T07:04:20.804306+0000","last_active":"2024-09-17T07:04:38.005852+0000","last_peered":"2024-09-17T07:04:38.005852+0000","last_clean":"2024-09-17T07:04:38.005852+0000","last_became_active":"2024-09-17T07:04:20.804060+0000","last_became_peered":"2024-09-17T07:04:20.804060+0000","last_unstale":"2024-09-17T07:04:38.005852+0000","last_undegraded":"2024-09-17T07:04:38.005852+0000","last_fullsized":"2024-09-17T07:04:38.005852+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_clean_scrub_stamp":"2024-09-17T07:04:19.785747+0000","objects_scrubbed":0,"log_size":1223,"log_dups_size":0,"ondisk_log_size":1223,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:44.538549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80150528,"num_objects":1223,"num_object_clones":0,"num_object_copies":3669,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1223,"num_write_kb":78272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"95'25","reported_seq":93,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.114438+0000","last_change":"2024-09-17T07:04:09.450155+0000","last_active":"2024-09-17T07:04:33.114438+0000","last_peered":"2024-09-17T07:04:33.114438+0000","last_clean":"2024-09-17T07:04:33.114438+0000","last_became_active":"2024-09-17T07:02:14.016270+0000","last_became_peered":"2024-09-17T07:02:14.016270+0000","last_unstale":"2024-09-17T07:04:33.114438+0000","last_undegraded":"2024-09-17T07:04:33.114438+0000","last_fullsized":"2024-09-17T07:04:33.114438+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'25","last_scrub_stamp":"2024-09-17T07:04:09.450039+0000","last_deep_scrub":"95'25","last_deep_scrub_stamp":"2024-09-17T07:04:09.450039+0000","last_clean_scrub_stamp":"2024-09-17T07:04:09.450039+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:13.708197+0000","scrub_duration":0.0050303120000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":59,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.112388+0000","last_change":"2024-09-17T07:03:14.092519+0000","last_active":"2024-09-17T07:04:33.112388+0000","last_peered":"2024-09-17T07:04:33.112388+0000","last_clean":"2024-09-17T07:04:33.112388+0000","last_became_active":"2024-09-17T07:02:12.011258+0000","last_became_peered":"2024-09-17T07:02:12.011258+0000","last_unstale":"2024-09-17T07:04:33.112388+0000","last_undegraded":"2024-09-17T07:04:33.112388+0000","last_fullsized":"2024-09-17T07:04:33.112388+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:14.092369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:14.092369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015447460000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":54,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.119147+0000","last_change":"2024-09-17T07:03:44.044750+0000","last_active":"2024-09-17T07:04:33.119147+0000","last_peered":"2024-09-17T07:04:33.119147+0000","last_clean":"2024-09-17T07:04:33.119147+0000","last_became_active":"2024-09-17T07:02:16.018147+0000","last_became_peered":"2024-09-17T07:02:16.018147+0000","last_unstale":"2024-09-17T07:04:33.119147+0000","last_undegraded":"2024-09-17T07:04:33.119147+0000","last_fullsized":"2024-09-17T07:04:33.119147+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:44.044635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:44.044635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:46.891263+0000","scrub_duration":0.0014202049999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":20,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.117719+0000","last_change":"2024-09-17T07:04:15.113476+0000","last_active":"2024-09-17T07:04:33.117719+0000","last_peered":"2024-09-17T07:04:33.117719+0000","last_clean":"2024-09-17T07:04:33.117719+0000","last_became_active":"2024-09-17T07:04:15.113196+0000","last_became_peered":"2024-09-17T07:04:15.113196+0000","last_unstale":"2024-09-17T07:04:33.117719+0000","last_undegraded":"2024-09-17T07:04:33.117719+0000","last_fullsized":"2024-09-17T07:04:33.117719+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:42.984088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:42.984088+0000","last_clean_scrub_stamp":"2024-09-17T07:03:42.984088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:49.461270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"105'1164","reported_seq":1179,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T07:04:38.007654+0000","last_change":"2024-09-17T07:04:20.800521+0000","last_active":"2024-09-17T07:04:38.007654+0000","last_peered":"2024-09-17T07:04:38.007654+0000","last_clean":"2024-09-17T07:04:38.007654+0000","last_became_active":"2024-09-17T07:04:20.800260+0000","last_became_peered":"2024-09-17T07:04:20.800260+0000","last_unstale":"2024-09-17T07:04:38.007654+0000","last_undegraded":"2024-09-17T07:04:38.007654+0000","last_fullsized":"2024-09-17T07:04:38.007654+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_clean_scrub_stamp":"2024-09-17T07:04:19.785747+0000","objects_scrubbed":0,"log_size":1164,"log_dups_size":0,"ondisk_log_size":1164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:39.086959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76283904,"num_objects":1164,"num_object_clones":0,"num_object_copies":3492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1164,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1164,"num_write_kb":74496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"75'10","reported_seq":72,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.114060+0000","last_change":"2024-09-17T07:03:20.933478+0000","last_active":"2024-09-17T07:04:33.114060+0000","last_peered":"2024-09-17T07:04:33.114060+0000","last_clean":"2024-09-17T07:04:33.114060+0000","last_became_active":"2024-09-17T07:02:14.007702+0000","last_became_peered":"2024-09-17T07:02:14.007702+0000","last_unstale":"2024-09-17T07:04:33.114060+0000","last_undegraded":"2024-09-17T07:04:33.114060+0000","last_fullsized":"2024-09-17T07:04:33.114060+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-09-17T07:03:20.933353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:20.933353+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021221080000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":58,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.120748+0000","last_change":"2024-09-17T07:03:51.927006+0000","last_active":"2024-09-17T07:04:33.120748+0000","last_peered":"2024-09-17T07:04:33.120748+0000","last_clean":"2024-09-17T07:04:33.120748+0000","last_became_active":"2024-09-17T07:02:12.007553+0000","last_became_peered":"2024-09-17T07:02:12.007553+0000","last_unstale":"2024-09-17T07:04:33.120748+0000","last_undegraded":"2024-09-17T07:04:33.120748+0000","last_fullsized":"2024-09-17T07:04:33.120748+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:51.926911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:51.926911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:52.506175+0000","scrub_duration":0.0014014909999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"98'9","reported_seq":44,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T07:04:34.123255+0000","last_change":"2024-09-17T07:04:15.111194+0000","last_active":"2024-09-17T07:04:34.123255+0000","last_peered":"2024-09-17T07:04:34.123255+0000","last_clean":"2024-09-17T07:04:34.123255+0000","last_became_active":"2024-09-17T07:04:15.110797+0000","last_became_peered":"2024-09-17T07:04:15.110797+0000","last_unstale":"2024-09-17T07:04:34.123255+0000","last_undegraded":"2024-09-17T07:04:34.123255+0000","last_fullsized":"2024-09-17T07:04:34.123255+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:03:45.909617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:45.909617+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:56.957546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":53,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T07:04:34.117114+0000","last_change":"2024-09-17T07:03:46.241227+0000","last_active":"2024-09-17T07:04:34.117114+0000","last_peered":"2024-09-17T07:04:34.117114+0000","last_clean":"2024-09-17T07:04:34.117114+0000","last_became_active":"2024-09-17T07:02:18.017460+0000","last_became_peered":"2024-09-17T07:02:18.017460+0000","last_unstale":"2024-09-17T07:04:34.117114+0000","last_undegraded":"2024-09-17T07:04:34.117114+0000","last_fullsized":"2024-09-17T07:04:34.117114+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:46.241143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:03:46.241143+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:15.293653+0000","scrub_duration":0.0013561739999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"105'1250","reported_seq":1265,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T07:04:37.992154+0000","last_change":"2024-09-17T07:04:20.804581+0000","last_active":"2024-09-17T07:04:37.992154+0000","last_peered":"2024-09-17T07:04:37.992154+0000","last_clean":"2024-09-17T07:04:37.992154+0000","last_became_active":"2024-09-17T07:04:20.804188+0000","last_became_peered":"2024-09-17T07:04:20.804188+0000","last_unstale":"2024-09-17T07:04:37.992154+0000","last_undegraded":"2024-09-17T07:04:37.992154+0000","last_fullsized":"2024-09-17T07:04:37.992154+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_clean_scrub_stamp":"2024-09-17T07:04:19.785747+0000","objects_scrubbed":0,"log_size":1250,"log_dups_size":0,"ondisk_log_size":1250,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:28.967824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81920000,"num_objects":1250,"num_object_clones":0,"num_object_copies":3750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1250,"num_write_kb":80000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"75'15","reported_seq":81,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.119618+0000","last_change":"2024-09-17T07:03:16.067607+0000","last_active":"2024-09-17T07:04:33.119618+0000","last_peered":"2024-09-17T07:04:33.119618+0000","last_clean":"2024-09-17T07:04:33.119618+0000","last_became_active":"2024-09-17T07:02:14.016397+0000","last_became_peered":"2024-09-17T07:02:14.016397+0000","last_unstale":"2024-09-17T07:04:33.119618+0000","last_undegraded":"2024-09-17T07:04:33.119618+0000","last_fullsized":"2024-09-17T07:04:33.119618+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-09-17T07:03:16.067479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:16.067479+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0025498640000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":58,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.119381+0000","last_change":"2024-09-17T07:03:28.072773+0000","last_active":"2024-09-17T07:04:33.119381+0000","last_peered":"2024-09-17T07:04:33.119381+0000","last_clean":"2024-09-17T07:04:33.119381+0000","last_became_active":"2024-09-17T07:02:12.007913+0000","last_became_peered":"2024-09-17T07:02:12.007913+0000","last_unstale":"2024-09-17T07:04:33.119381+0000","last_undegraded":"2024-09-17T07:04:33.119381+0000","last_fullsized":"2024-09-17T07:04:33.119381+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:28.072697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:28.072697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:43.368112+0000","scrub_duration":0.0012624730000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"75'8","reported_seq":92,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T07:04:34.117191+0000","last_change":"2024-09-17T07:04:33.544461+0000","last_active":"2024-09-17T07:04:34.117191+0000","last_peered":"2024-09-17T07:04:34.117191+0000","last_clean":"2024-09-17T07:04:34.117191+0000","last_became_active":"2024-09-17T07:02:16.021762+0000","last_became_peered":"2024-09-17T07:02:16.021762+0000","last_unstale":"2024-09-17T07:04:34.117191+0000","last_undegraded":"2024-09-17T07:04:34.117191+0000","last_fullsized":"2024-09-17T07:04:34.117191+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:04:33.544306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:04:33.544306+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:50.360596+0000","scrub_duration":0.0038501080000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":52,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.118923+0000","last_change":"2024-09-17T07:03:47.759562+0000","last_active":"2024-09-17T07:04:33.118923+0000","last_peered":"2024-09-17T07:04:33.118923+0000","last_clean":"2024-09-17T07:04:33.118923+0000","last_became_active":"2024-09-17T07:02:18.025332+0000","last_became_peered":"2024-09-17T07:02:18.025332+0000","last_unstale":"2024-09-17T07:04:33.118923+0000","last_undegraded":"2024-09-17T07:04:33.118923+0000","last_fullsized":"2024-09-17T07:04:33.118923+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:47.759476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:03:47.759476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:16.113819+0000","scrub_duration":0.0035797160000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"105'1199","reported_seq":1214,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T07:04:38.007805+0000","last_change":"2024-09-17T07:04:20.805603+0000","last_active":"2024-09-17T07:04:38.007805+0000","last_peered":"2024-09-17T07:04:38.007805+0000","last_clean":"2024-09-17T07:04:38.007805+0000","last_became_active":"2024-09-17T07:04:20.805282+0000","last_became_peered":"2024-09-17T07:04:20.805282+0000","last_unstale":"2024-09-17T07:04:38.007805+0000","last_undegraded":"2024-09-17T07:04:38.007805+0000","last_fullsized":"2024-09-17T07:04:38.007805+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_clean_scrub_stamp":"2024-09-17T07:04:19.785747+0000","objects_scrubbed":0,"log_size":1199,"log_dups_size":0,"ondisk_log_size":1199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:32.457152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78577664,"num_objects":1199,"num_object_clones":0,"num_object_copies":3597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1199,"num_write_kb":76736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"75'11","reported_seq":76,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.114933+0000","last_change":"2024-09-17T07:04:05.375658+0000","last_active":"2024-09-17T07:04:33.114933+0000","last_peered":"2024-09-17T07:04:33.114933+0000","last_clean":"2024-09-17T07:04:33.114933+0000","last_became_active":"2024-09-17T07:02:14.010227+0000","last_became_peered":"2024-09-17T07:02:14.010227+0000","last_unstale":"2024-09-17T07:04:33.114933+0000","last_undegraded":"2024-09-17T07:04:33.114933+0000","last_fullsized":"2024-09-17T07:04:33.114933+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-09-17T07:04:05.375563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:04:05.375563+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:05.996553+0000","scrub_duration":0.0022593959999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":59,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.111728+0000","last_change":"2024-09-17T07:03:31.076235+0000","last_active":"2024-09-17T07:04:33.111728+0000","last_peered":"2024-09-17T07:04:33.111728+0000","last_clean":"2024-09-17T07:04:33.111728+0000","last_became_active":"2024-09-17T07:02:12.013182+0000","last_became_peered":"2024-09-17T07:02:12.013182+0000","last_unstale":"2024-09-17T07:04:33.111728+0000","last_undegraded":"2024-09-17T07:04:33.111728+0000","last_fullsized":"2024-09-17T07:04:33.111728+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:31.076134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:31.076134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:37.803854+0000","scrub_duration":0.0012890040000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":54,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T07:04:34.126766+0000","last_change":"2024-09-17T07:03:18.816900+0000","last_active":"2024-09-17T07:04:34.126766+0000","last_peered":"2024-09-17T07:04:34.126766+0000","last_clean":"2024-09-17T07:04:34.126766+0000","last_became_active":"2024-09-17T07:02:16.020224+0000","last_became_peered":"2024-09-17T07:02:16.020224+0000","last_unstale":"2024-09-17T07:04:34.126766+0000","last_undegraded":"2024-09-17T07:04:34.126766+0000","last_fullsized":"2024-09-17T07:04:34.126766+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:18.816795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:18.816795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:46.254399+0000","scrub_duration":0.001573212,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":52,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T07:04:34.119571+0000","last_change":"2024-09-17T07:03:35.447165+0000","last_active":"2024-09-17T07:04:34.119571+0000","last_peered":"2024-09-17T07:04:34.119571+0000","last_clean":"2024-09-17T07:04:34.119571+0000","last_became_active":"2024-09-17T07:02:18.365000+0000","last_became_peered":"2024-09-17T07:02:18.365000+0000","last_unstale":"2024-09-17T07:04:34.119571+0000","last_undegraded":"2024-09-17T07:04:34.119571+0000","last_fullsized":"2024-09-17T07:04:34.119571+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:35.447078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:03:35.447078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:38.486661+0000","scrub_duration":0.0020715600000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"105'1170","reported_seq":1185,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T07:04:38.001853+0000","last_change":"2024-09-17T07:04:20.802241+0000","last_active":"2024-09-17T07:04:38.001853+0000","last_peered":"2024-09-17T07:04:38.001853+0000","last_clean":"2024-09-17T07:04:38.001853+0000","last_became_active":"2024-09-17T07:04:20.802003+0000","last_became_peered":"2024-09-17T07:04:20.802003+0000","last_unstale":"2024-09-17T07:04:38.001853+0000","last_undegraded":"2024-09-17T07:04:38.001853+0000","last_fullsized":"2024-09-17T07:04:38.001853+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_clean_scrub_stamp":"2024-09-17T07:04:19.785747+0000","objects_scrubbed":0,"log_size":1170,"log_dups_size":0,"ondisk_log_size":1170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:32.972300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76677120,"num_objects":1170,"num_object_clones":0,"num_object_copies":3510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1170,"num_write_kb":74880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"75'14","reported_seq":67,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.114283+0000","last_change":"2024-09-17T07:02:14.008065+0000","last_active":"2024-09-17T07:04:33.114283+0000","last_peered":"2024-09-17T07:04:33.114283+0000","last_clean":"2024-09-17T07:04:33.114283+0000","last_became_active":"2024-09-17T07:02:14.007810+0000","last_became_peered":"2024-09-17T07:02:14.007810+0000","last_unstale":"2024-09-17T07:04:33.114283+0000","last_undegraded":"2024-09-17T07:04:33.114283+0000","last_fullsized":"2024-09-17T07:04:33.114283+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:02:12.994221+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":69,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T07:04:37.435115+0000","last_change":"2024-09-17T07:04:37.435222+0000","last_active":"2024-09-17T07:04:37.435115+0000","last_peered":"2024-09-17T07:04:37.435115+0000","last_clean":"2024-09-17T07:04:37.435115+0000","last_became_active":"2024-09-17T07:02:12.011766+0000","last_became_peered":"2024-09-17T07:02:12.011766+0000","last_unstale":"2024-09-17T07:04:37.435115+0000","last_undegraded":"2024-09-17T07:04:37.435115+0000","last_fullsized":"2024-09-17T07:04:37.435115+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:37.435057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:04:37.435057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:52.133844+0000","scrub_duration":0.0032553450000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"75'8","reported_seq":91,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T07:04:34.119429+0000","last_change":"2024-09-17T07:03:33.444197+0000","last_active":"2024-09-17T07:04:34.119429+0000","last_peered":"2024-09-17T07:04:34.119429+0000","last_clean":"2024-09-17T07:04:34.119429+0000","last_became_active":"2024-09-17T07:02:16.020628+0000","last_became_peered":"2024-09-17T07:02:16.020628+0000","last_unstale":"2024-09-17T07:04:34.119429+0000","last_undegraded":"2024-09-17T07:04:34.119429+0000","last_fullsized":"2024-09-17T07:04:34.119429+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:03:33.444104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:33.444104+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:36.767124+0000","scrub_duration":0.0014124389999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":53,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.111872+0000","last_change":"2024-09-17T07:03:19.030574+0000","last_active":"2024-09-17T07:04:33.111872+0000","last_peered":"2024-09-17T07:04:33.111872+0000","last_clean":"2024-09-17T07:04:33.111872+0000","last_became_active":"2024-09-17T07:02:18.364981+0000","last_became_peered":"2024-09-17T07:02:18.364981+0000","last_unstale":"2024-09-17T07:04:33.111872+0000","last_undegraded":"2024-09-17T07:04:33.111872+0000","last_fullsized":"2024-09-17T07:04:33.111872+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:19.030470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:03:19.030470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:48.307795+0000","scrub_duration":0.00152618,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"105'1255","reported_seq":1270,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T07:04:37.998512+0000","last_change":"2024-09-17T07:04:20.803097+0000","last_active":"2024-09-17T07:04:37.998512+0000","last_peered":"2024-09-17T07:04:37.998512+0000","last_clean":"2024-09-17T07:04:37.998512+0000","last_became_active":"2024-09-17T07:04:20.802786+0000","last_became_peered":"2024-09-17T07:04:20.802786+0000","last_unstale":"2024-09-17T07:04:37.998512+0000","last_undegraded":"2024-09-17T07:04:37.998512+0000","last_fullsized":"2024-09-17T07:04:37.998512+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_clean_scrub_stamp":"2024-09-17T07:04:19.785747+0000","objects_scrubbed":0,"log_size":1255,"log_dups_size":0,"ondisk_log_size":1255,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:37.604799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":82247680,"num_objects":1255,"num_object_clones":0,"num_object_copies":3765,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1255,"num_write_kb":80320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"75'18","reported_seq":88,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.117196+0000","last_change":"2024-09-17T07:03:26.413610+0000","last_active":"2024-09-17T07:04:33.117196+0000","last_peered":"2024-09-17T07:04:33.117196+0000","last_clean":"2024-09-17T07:04:33.117196+0000","last_became_active":"2024-09-17T07:02:14.028667+0000","last_became_peered":"2024-09-17T07:02:14.028667+0000","last_unstale":"2024-09-17T07:04:33.117196+0000","last_undegraded":"2024-09-17T07:04:33.117196+0000","last_fullsized":"2024-09-17T07:04:33.117196+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'18","last_scrub_stamp":"2024-09-17T07:03:26.413506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:26.413506+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:41.741155+0000","scrub_duration":0.0022855470000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":58,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.119392+0000","last_change":"2024-09-17T07:03:24.134708+0000","last_active":"2024-09-17T07:04:33.119392+0000","last_peered":"2024-09-17T07:04:33.119392+0000","last_clean":"2024-09-17T07:04:33.119392+0000","last_became_active":"2024-09-17T07:02:12.005687+0000","last_became_peered":"2024-09-17T07:02:12.005687+0000","last_unstale":"2024-09-17T07:04:33.119392+0000","last_undegraded":"2024-09-17T07:04:33.119392+0000","last_fullsized":"2024-09-17T07:04:33.119392+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:24.134627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:24.134627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001111148,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":55,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.113755+0000","last_change":"2024-09-17T07:04:01.335116+0000","last_active":"2024-09-17T07:04:33.113755+0000","last_peered":"2024-09-17T07:04:33.113755+0000","last_clean":"2024-09-17T07:04:33.113755+0000","last_became_active":"2024-09-17T07:02:16.019747+0000","last_became_peered":"2024-09-17T07:02:16.019747+0000","last_unstale":"2024-09-17T07:04:33.113755+0000","last_undegraded":"2024-09-17T07:04:33.113755+0000","last_fullsized":"2024-09-17T07:04:33.113755+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:01.335039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:04:01.335039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:26.509599+0000","scrub_duration":0.001538607,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":53,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.114203+0000","last_change":"2024-09-17T07:03:44.196107+0000","last_active":"2024-09-17T07:04:33.114203+0000","last_peered":"2024-09-17T07:04:33.114203+0000","last_clean":"2024-09-17T07:04:33.114203+0000","last_became_active":"2024-09-17T07:02:18.024043+0000","last_became_peered":"2024-09-17T07:02:18.024043+0000","last_unstale":"2024-09-17T07:04:33.114203+0000","last_undegraded":"2024-09-17T07:04:33.114203+0000","last_fullsized":"2024-09-17T07:04:33.114203+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:44.196035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:03:44.196035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:12.177838+0000","scrub_duration":0.0015579439999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"105'1198","reported_seq":1213,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T07:04:37.997179+0000","last_change":"2024-09-17T07:04:20.806048+0000","last_active":"2024-09-17T07:04:37.997179+0000","last_peered":"2024-09-17T07:04:37.997179+0000","last_clean":"2024-09-17T07:04:37.997179+0000","last_became_active":"2024-09-17T07:04:20.805765+0000","last_became_peered":"2024-09-17T07:04:20.805765+0000","last_unstale":"2024-09-17T07:04:37.997179+0000","last_undegraded":"2024-09-17T07:04:37.997179+0000","last_fullsized":"2024-09-17T07:04:37.997179+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_clean_scrub_stamp":"2024-09-17T07:04:19.785747+0000","objects_scrubbed":0,"log_size":1198,"log_dups_size":0,"ondisk_log_size":1198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:31.184704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78512128,"num_objects":1198,"num_object_clones":0,"num_object_copies":3594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1198,"num_write_kb":76672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"75'19","reported_seq":93,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.111927+0000","last_change":"2024-09-17T07:03:40.949557+0000","last_active":"2024-09-17T07:04:33.111927+0000","last_peered":"2024-09-17T07:04:33.111927+0000","last_clean":"2024-09-17T07:04:33.111927+0000","last_became_active":"2024-09-17T07:02:14.027389+0000","last_became_peered":"2024-09-17T07:02:14.027389+0000","last_unstale":"2024-09-17T07:04:33.111927+0000","last_undegraded":"2024-09-17T07:04:33.111927+0000","last_fullsized":"2024-09-17T07:04:33.111927+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-09-17T07:03:40.949458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:40.949458+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:03.948689+0000","scrub_duration":0.0025476489999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":59,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.112773+0000","last_change":"2024-09-17T07:03:32.109842+0000","last_active":"2024-09-17T07:04:33.112773+0000","last_peered":"2024-09-17T07:04:33.112773+0000","last_clean":"2024-09-17T07:04:33.112773+0000","last_became_active":"2024-09-17T07:02:12.011917+0000","last_became_peered":"2024-09-17T07:02:12.011917+0000","last_unstale":"2024-09-17T07:04:33.112773+0000","last_undegraded":"2024-09-17T07:04:33.112773+0000","last_fullsized":"2024-09-17T07:04:33.112773+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:32.109765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:32.109765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:52.813415+0000","scrub_duration":0.001335314,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":54,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.115120+0000","last_change":"2024-09-17T07:03:49.489790+0000","last_active":"2024-09-17T07:04:33.115120+0000","last_peered":"2024-09-17T07:04:33.115120+0000","last_clean":"2024-09-17T07:04:33.115120+0000","last_became_active":"2024-09-17T07:02:16.016199+0000","last_became_peered":"2024-09-17T07:02:16.016199+0000","last_unstale":"2024-09-17T07:04:33.115120+0000","last_undegraded":"2024-09-17T07:04:33.115120+0000","last_fullsized":"2024-09-17T07:04:33.115120+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:49.489690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:49.489690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:17.412359+0000","scrub_duration":0.0016174729999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":52,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.119052+0000","last_change":"2024-09-17T07:03:30.068895+0000","last_active":"2024-09-17T07:04:33.119052+0000","last_peered":"2024-09-17T07:04:33.119052+0000","last_clean":"2024-09-17T07:04:33.119052+0000","last_became_active":"2024-09-17T07:02:18.364835+0000","last_became_peered":"2024-09-17T07:02:18.364835+0000","last_unstale":"2024-09-17T07:04:33.119052+0000","last_undegraded":"2024-09-17T07:04:33.119052+0000","last_fullsized":"2024-09-17T07:04:33.119052+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:30.068808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:03:30.068808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:51.270374+0000","scrub_duration":0.0012893679999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"105'1253","reported_seq":1268,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T07:04:37.998481+0000","last_change":"2024-09-17T07:04:20.806338+0000","last_active":"2024-09-17T07:04:37.998481+0000","last_peered":"2024-09-17T07:04:37.998481+0000","last_clean":"2024-09-17T07:04:37.998481+0000","last_became_active":"2024-09-17T07:04:20.805997+0000","last_became_peered":"2024-09-17T07:04:20.805997+0000","last_unstale":"2024-09-17T07:04:37.998481+0000","last_undegraded":"2024-09-17T07:04:37.998481+0000","last_fullsized":"2024-09-17T07:04:37.998481+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_clean_scrub_stamp":"2024-09-17T07:04:19.785747+0000","objects_scrubbed":0,"log_size":1253,"log_dups_size":0,"ondisk_log_size":1253,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:20.696985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":82116608,"num_objects":1253,"num_object_clones":0,"num_object_copies":3759,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1253,"num_write_kb":80192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"75'10","reported_seq":71,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.120621+0000","last_change":"2024-09-17T07:03:26.127545+0000","last_active":"2024-09-17T07:04:33.120621+0000","last_peered":"2024-09-17T07:04:33.120621+0000","last_clean":"2024-09-17T07:04:33.120621+0000","last_became_active":"2024-09-17T07:02:14.010637+0000","last_became_peered":"2024-09-17T07:02:14.010637+0000","last_unstale":"2024-09-17T07:04:33.120621+0000","last_undegraded":"2024-09-17T07:04:33.120621+0000","last_fullsized":"2024-09-17T07:04:33.120621+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-09-17T07:03:26.127439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:26.127439+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:35.535770+0000","scrub_duration":0.001898479,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":58,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.121226+0000","last_change":"2024-09-17T07:03:25.148144+0000","last_active":"2024-09-17T07:04:33.121226+0000","last_peered":"2024-09-17T07:04:33.121226+0000","last_clean":"2024-09-17T07:04:33.121226+0000","last_became_active":"2024-09-17T07:02:12.010803+0000","last_became_peered":"2024-09-17T07:02:12.010803+0000","last_unstale":"2024-09-17T07:04:33.121226+0000","last_undegraded":"2024-09-17T07:04:33.121226+0000","last_fullsized":"2024-09-17T07:04:33.121226+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:25.147878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:25.147878+0000","last_clean_scrub_stamp":"2024-09-17T07:03:25.147878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001353723,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"89'9","reported_seq":93,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T07:04:34.122031+0000","last_change":"2024-09-17T07:03:29.961568+0000","last_active":"2024-09-17T07:04:34.122031+0000","last_peered":"2024-09-17T07:04:34.122031+0000","last_clean":"2024-09-17T07:04:34.122031+0000","last_became_active":"2024-09-17T07:02:16.021360+0000","last_became_peered":"2024-09-17T07:02:16.021360+0000","last_unstale":"2024-09-17T07:04:34.122031+0000","last_undegraded":"2024-09-17T07:04:34.122031+0000","last_fullsized":"2024-09-17T07:04:34.122031+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'9","last_scrub_stamp":"2024-09-17T07:03:29.961439+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-09-17T07:03:29.961439+0000","last_clean_scrub_stamp":"2024-09-17T07:03:29.961439+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:40.753813+0000","scrub_duration":0.0027696880000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"75'1","reported_seq":54,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.111230+0000","last_change":"2024-09-17T07:03:25.049036+0000","last_active":"2024-09-17T07:04:33.111230+0000","last_peered":"2024-09-17T07:04:33.111230+0000","last_clean":"2024-09-17T07:04:33.111230+0000","last_became_active":"2024-09-17T07:02:18.029772+0000","last_became_peered":"2024-09-17T07:02:18.029772+0000","last_unstale":"2024-09-17T07:04:33.111230+0000","last_undegraded":"2024-09-17T07:04:33.111230+0000","last_fullsized":"2024-09-17T07:04:33.111230+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-17T07:03:25.048949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:03:25.048949+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:53.755876+0000","scrub_duration":0.003886123,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"105'1226","reported_seq":1241,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T07:04:37.967039+0000","last_change":"2024-09-17T07:04:20.805637+0000","last_active":"2024-09-17T07:04:37.967039+0000","last_peered":"2024-09-17T07:04:37.967039+0000","last_clean":"2024-09-17T07:04:37.967039+0000","last_became_active":"2024-09-17T07:04:20.805331+0000","last_became_peered":"2024-09-17T07:04:20.805331+0000","last_unstale":"2024-09-17T07:04:37.967039+0000","last_undegraded":"2024-09-17T07:04:37.967039+0000","last_fullsized":"2024-09-17T07:04:37.967039+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_clean_scrub_stamp":"2024-09-17T07:04:19.785747+0000","objects_scrubbed":0,"log_size":1226,"log_dups_size":0,"ondisk_log_size":1226,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:30.877586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80347136,"num_objects":1226,"num_object_clones":0,"num_object_copies":3678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1226,"num_write_kb":78464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"75'28","reported_seq":133,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T07:04:34.126882+0000","last_change":"2024-09-17T07:03:52.938018+0000","last_active":"2024-09-17T07:04:34.126882+0000","last_peered":"2024-09-17T07:04:34.126882+0000","last_clean":"2024-09-17T07:04:34.126882+0000","last_became_active":"2024-09-17T07:02:14.017377+0000","last_became_peered":"2024-09-17T07:02:14.017377+0000","last_unstale":"2024-09-17T07:04:34.126882+0000","last_undegraded":"2024-09-17T07:04:34.126882+0000","last_fullsized":"2024-09-17T07:04:34.126882+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'28","last_scrub_stamp":"2024-09-17T07:03:52.937879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:52.937879+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:09.513804+0000","scrub_duration":0.0023024629999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":58,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.118787+0000","last_change":"2024-09-17T07:03:40.811355+0000","last_active":"2024-09-17T07:04:33.118787+0000","last_peered":"2024-09-17T07:04:33.118787+0000","last_clean":"2024-09-17T07:04:33.118787+0000","last_became_active":"2024-09-17T07:02:12.008802+0000","last_became_peered":"2024-09-17T07:02:12.008802+0000","last_unstale":"2024-09-17T07:04:33.118787+0000","last_undegraded":"2024-09-17T07:04:33.118787+0000","last_fullsized":"2024-09-17T07:04:33.118787+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:40.811273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:40.811273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:52.601572+0000","scrub_duration":0.0012956230000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":55,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.113273+0000","last_change":"2024-09-17T07:03:28.981244+0000","last_active":"2024-09-17T07:04:33.113273+0000","last_peered":"2024-09-17T07:04:33.113273+0000","last_clean":"2024-09-17T07:04:33.113273+0000","last_became_active":"2024-09-17T07:02:16.016606+0000","last_became_peered":"2024-09-17T07:02:16.016606+0000","last_unstale":"2024-09-17T07:04:33.113273+0000","last_undegraded":"2024-09-17T07:04:33.113273+0000","last_fullsized":"2024-09-17T07:04:33.113273+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:28.981147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:28.981147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:37.078325+0000","scrub_duration":0.0012741460000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":52,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.117712+0000","last_change":"2024-09-17T07:04:04.564851+0000","last_active":"2024-09-17T07:04:33.117712+0000","last_peered":"2024-09-17T07:04:33.117712+0000","last_clean":"2024-09-17T07:04:33.117712+0000","last_became_active":"2024-09-17T07:02:18.024602+0000","last_became_peered":"2024-09-17T07:02:18.024602+0000","last_unstale":"2024-09-17T07:04:33.117712+0000","last_undegraded":"2024-09-17T07:04:33.117712+0000","last_fullsized":"2024-09-17T07:04:33.117712+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:04.564765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:04:04.564765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:25.881100+0000","scrub_duration":0.001317206,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"105'1230","reported_seq":1245,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T07:04:38.007614+0000","last_change":"2024-09-17T07:04:20.802551+0000","last_active":"2024-09-17T07:04:38.007614+0000","last_peered":"2024-09-17T07:04:38.007614+0000","last_clean":"2024-09-17T07:04:38.007614+0000","last_became_active":"2024-09-17T07:04:20.802120+0000","last_became_peered":"2024-09-17T07:04:20.802120+0000","last_unstale":"2024-09-17T07:04:38.007614+0000","last_undegraded":"2024-09-17T07:04:38.007614+0000","last_fullsized":"2024-09-17T07:04:38.007614+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_clean_scrub_stamp":"2024-09-17T07:04:19.785747+0000","objects_scrubbed":0,"log_size":1230,"log_dups_size":0,"ondisk_log_size":1230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:30.842764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80609280,"num_objects":1230,"num_object_clones":0,"num_object_copies":3690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1230,"num_write_kb":78720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"75'13","reported_seq":83,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.120831+0000","last_change":"2024-09-17T07:03:32.079678+0000","last_active":"2024-09-17T07:04:33.120831+0000","last_peered":"2024-09-17T07:04:33.120831+0000","last_clean":"2024-09-17T07:04:33.120831+0000","last_became_active":"2024-09-17T07:02:14.019838+0000","last_became_peered":"2024-09-17T07:02:14.019838+0000","last_unstale":"2024-09-17T07:04:33.120831+0000","last_undegraded":"2024-09-17T07:04:33.120831+0000","last_fullsized":"2024-09-17T07:04:33.120831+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'13","last_scrub_stamp":"2024-09-17T07:03:32.079579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:32.079579+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:42.502680+0000","scrub_duration":0.001975929,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"76'5","reported_seq":205,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T07:04:36.201866+0000","last_change":"2024-09-17T07:03:50.944706+0000","last_active":"2024-09-17T07:04:36.201866+0000","last_peered":"2024-09-17T07:04:36.201866+0000","last_clean":"2024-09-17T07:04:36.201866+0000","last_became_active":"2024-09-17T07:02:12.006883+0000","last_became_peered":"2024-09-17T07:02:12.006883+0000","last_unstale":"2024-09-17T07:04:36.201866+0000","last_undegraded":"2024-09-17T07:04:36.201866+0000","last_fullsized":"2024-09-17T07:04:36.201866+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'5","last_scrub_stamp":"2024-09-17T07:03:50.944626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:50.944626+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:16.063852+0000","scrub_duration":0.001943529,"objects_trimmed":0,"snaptrim_duration":0.00033739799999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":139,"num_read_kb":134,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.118721+0000","last_change":"2024-09-17T07:04:15.110234+0000","last_active":"2024-09-17T07:04:33.118721+0000","last_peered":"2024-09-17T07:04:33.118721+0000","last_clean":"2024-09-17T07:04:33.118721+0000","last_became_active":"2024-09-17T07:04:15.109950+0000","last_became_peered":"2024-09-17T07:04:15.109950+0000","last_unstale":"2024-09-17T07:04:33.118721+0000","last_undegraded":"2024-09-17T07:04:33.118721+0000","last_fullsized":"2024-09-17T07:04:33.118721+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:33.795307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:33.795307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:34.331687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":55,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.113758+0000","last_change":"2024-09-17T07:03:52.922792+0000","last_active":"2024-09-17T07:04:33.113758+0000","last_peered":"2024-09-17T07:04:33.113758+0000","last_clean":"2024-09-17T07:04:33.113758+0000","last_became_active":"2024-09-17T07:02:16.014492+0000","last_became_peered":"2024-09-17T07:02:16.014492+0000","last_unstale":"2024-09-17T07:04:33.113758+0000","last_undegraded":"2024-09-17T07:04:33.113758+0000","last_fullsized":"2024-09-17T07:04:33.113758+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:52.922712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:52.922712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:11.791082+0000","scrub_duration":0.0016789929999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":53,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.112366+0000","last_change":"2024-09-17T07:03:42.068886+0000","last_active":"2024-09-17T07:04:33.112366+0000","last_peered":"2024-09-17T07:04:33.112366+0000","last_clean":"2024-09-17T07:04:33.112366+0000","last_became_active":"2024-09-17T07:02:18.365369+0000","last_became_peered":"2024-09-17T07:02:18.365369+0000","last_unstale":"2024-09-17T07:04:33.112366+0000","last_undegraded":"2024-09-17T07:04:33.112366+0000","last_fullsized":"2024-09-17T07:04:33.112366+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:42.068803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:03:42.068803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:47.482056+0000","scrub_duration":0.0015781599999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"105'1192","reported_seq":1207,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T07:04:38.001182+0000","last_change":"2024-09-17T07:04:20.805219+0000","last_active":"2024-09-17T07:04:38.001182+0000","last_peered":"2024-09-17T07:04:38.001182+0000","last_clean":"2024-09-17T07:04:38.001182+0000","last_became_active":"2024-09-17T07:04:20.804778+0000","last_became_peered":"2024-09-17T07:04:20.804778+0000","last_unstale":"2024-09-17T07:04:38.001182+0000","last_undegraded":"2024-09-17T07:04:38.001182+0000","last_fullsized":"2024-09-17T07:04:38.001182+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_clean_scrub_stamp":"2024-09-17T07:04:19.785747+0000","objects_scrubbed":0,"log_size":1192,"log_dups_size":0,"ondisk_log_size":1192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:48.764834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78118912,"num_objects":1192,"num_object_clones":0,"num_object_copies":3576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1192,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1192,"num_write_kb":76288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"75'12","reported_seq":98,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.113003+0000","last_change":"2024-09-17T07:03:48.767808+0000","last_active":"2024-09-17T07:04:33.113003+0000","last_peered":"2024-09-17T07:04:33.113003+0000","last_clean":"2024-09-17T07:04:33.113003+0000","last_became_active":"2024-09-17T07:03:48.760405+0000","last_became_peered":"2024-09-17T07:03:48.760405+0000","last_unstale":"2024-09-17T07:04:33.113003+0000","last_undegraded":"2024-09-17T07:04:33.113003+0000","last_fullsized":"2024-09-17T07:04:33.113003+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-09-17T07:03:27.987835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:27.987835+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:43.269560+0000","scrub_duration":0.0020148219999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"70'1","reported_seq":65,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.111691+0000","last_change":"2024-09-17T07:03:52.963332+0000","last_active":"2024-09-17T07:04:33.111691+0000","last_peered":"2024-09-17T07:04:33.111691+0000","last_clean":"2024-09-17T07:04:33.111691+0000","last_became_active":"2024-09-17T07:02:12.009476+0000","last_became_peered":"2024-09-17T07:02:12.009476+0000","last_unstale":"2024-09-17T07:04:33.111691+0000","last_undegraded":"2024-09-17T07:04:33.111691+0000","last_fullsized":"2024-09-17T07:04:33.111691+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-17T07:03:52.963249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:52.963249+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:07.662760+0000","scrub_duration":0.0015040780000000001,"objects_trimmed":0,"snaptrim_duration":0.0032298629999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":59,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.111889+0000","last_change":"2024-09-17T07:03:51.994858+0000","last_active":"2024-09-17T07:04:33.111889+0000","last_peered":"2024-09-17T07:04:33.111889+0000","last_clean":"2024-09-17T07:04:33.111889+0000","last_became_active":"2024-09-17T07:02:12.014623+0000","last_became_peered":"2024-09-17T07:02:12.014623+0000","last_unstale":"2024-09-17T07:04:33.111889+0000","last_undegraded":"2024-09-17T07:04:33.111889+0000","last_fullsized":"2024-09-17T07:04:33.111889+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:51.994756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:51.994756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:12.536626+0000","scrub_duration":0.001933269,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":54,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.119232+0000","last_change":"2024-09-17T07:03:38.818472+0000","last_active":"2024-09-17T07:04:33.119232+0000","last_peered":"2024-09-17T07:04:33.119232+0000","last_clean":"2024-09-17T07:04:33.119232+0000","last_became_active":"2024-09-17T07:02:16.021561+0000","last_became_peered":"2024-09-17T07:02:16.021561+0000","last_unstale":"2024-09-17T07:04:33.119232+0000","last_undegraded":"2024-09-17T07:04:33.119232+0000","last_fullsized":"2024-09-17T07:04:33.119232+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:38.818329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:38.818329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:05.115055+0000","scrub_duration":0.0016477,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":52,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.119056+0000","last_change":"2024-09-17T07:04:03.980001+0000","last_active":"2024-09-17T07:04:33.119056+0000","last_peered":"2024-09-17T07:04:33.119056+0000","last_clean":"2024-09-17T07:04:33.119056+0000","last_became_active":"2024-09-17T07:02:18.022082+0000","last_became_peered":"2024-09-17T07:02:18.022082+0000","last_unstale":"2024-09-17T07:04:33.119056+0000","last_undegraded":"2024-09-17T07:04:33.119056+0000","last_fullsized":"2024-09-17T07:04:33.119056+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:03.979895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:04:03.979895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:16.572977+0000","scrub_duration":0.001298542,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"105'1188","reported_seq":1203,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T07:04:37.993874+0000","last_change":"2024-09-17T07:04:20.801089+0000","last_active":"2024-09-17T07:04:37.993874+0000","last_peered":"2024-09-17T07:04:37.993874+0000","last_clean":"2024-09-17T07:04:37.993874+0000","last_became_active":"2024-09-17T07:04:20.800841+0000","last_became_peered":"2024-09-17T07:04:20.800841+0000","last_unstale":"2024-09-17T07:04:37.993874+0000","last_undegraded":"2024-09-17T07:04:37.993874+0000","last_fullsized":"2024-09-17T07:04:37.993874+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_clean_scrub_stamp":"2024-09-17T07:04:19.785747+0000","objects_scrubbed":0,"log_size":1188,"log_dups_size":0,"ondisk_log_size":1188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:29.079045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77856768,"num_objects":1188,"num_object_clones":0,"num_object_copies":3564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1188,"num_write_kb":76032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"75'14","reported_seq":78,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.116642+0000","last_change":"2024-09-17T07:03:44.522720+0000","last_active":"2024-09-17T07:04:33.116642+0000","last_peered":"2024-09-17T07:04:33.116642+0000","last_clean":"2024-09-17T07:04:33.116642+0000","last_became_active":"2024-09-17T07:02:14.028857+0000","last_became_peered":"2024-09-17T07:02:14.028857+0000","last_unstale":"2024-09-17T07:04:33.116642+0000","last_undegraded":"2024-09-17T07:04:33.116642+0000","last_fullsized":"2024-09-17T07:04:33.116642+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'14","last_scrub_stamp":"2024-09-17T07:03:44.522631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:44.522631+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:13.112891+0000","scrub_duration":0.0019970970000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"72'2","reported_seq":76,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T07:04:35.690365+0000","last_change":"2024-09-17T07:04:35.691710+0000","last_active":"2024-09-17T07:04:35.690365+0000","last_peered":"2024-09-17T07:04:35.690365+0000","last_clean":"2024-09-17T07:04:35.690365+0000","last_became_active":"2024-09-17T07:02:12.005127+0000","last_became_peered":"2024-09-17T07:02:12.005127+0000","last_unstale":"2024-09-17T07:04:35.690365+0000","last_undegraded":"2024-09-17T07:04:35.690365+0000","last_fullsized":"2024-09-17T07:04:35.690365+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'2","last_scrub_stamp":"2024-09-17T07:04:35.690341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:04:35.690341+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:39.220149+0000","scrub_duration":0.0079935560000000006,"objects_trimmed":0,"snaptrim_duration":0.0010980219999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":69,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T07:04:34.598485+0000","last_change":"2024-09-17T07:04:34.598581+0000","last_active":"2024-09-17T07:04:34.598485+0000","last_peered":"2024-09-17T07:04:34.598485+0000","last_clean":"2024-09-17T07:04:34.598485+0000","last_became_active":"2024-09-17T07:02:12.004606+0000","last_became_peered":"2024-09-17T07:02:12.004606+0000","last_unstale":"2024-09-17T07:04:34.598485+0000","last_undegraded":"2024-09-17T07:04:34.598485+0000","last_fullsized":"2024-09-17T07:04:34.598485+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:34.598457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:04:34.598457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:43.764579+0000","scrub_duration":0.0026566570000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"78'128","reported_seq":318,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T07:04:37.751973+0000","last_change":"2024-09-17T07:04:37.752152+0000","last_active":"2024-09-17T07:04:37.751973+0000","last_peered":"2024-09-17T07:04:37.751973+0000","last_clean":"2024-09-17T07:04:37.751973+0000","last_became_active":"2024-09-17T07:02:03.020675+0000","last_became_peered":"2024-09-17T07:02:03.020675+0000","last_unstale":"2024-09-17T07:04:37.751973+0000","last_undegraded":"2024-09-17T07:04:37.751973+0000","last_fullsized":"2024-09-17T07:04:37.751973+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'128","last_scrub_stamp":"2024-09-17T07:04:37.751949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:00:09.390381+0000","last_clean_scrub_stamp":"2024-09-17T07:04:37.751949+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:06:01.096229+0000","scrub_duration":0.0040036619999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":55,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.112166+0000","last_change":"2024-09-17T07:03:30.069684+0000","last_active":"2024-09-17T07:04:33.112166+0000","last_peered":"2024-09-17T07:04:33.112166+0000","last_clean":"2024-09-17T07:04:33.112166+0000","last_became_active":"2024-09-17T07:02:16.016031+0000","last_became_peered":"2024-09-17T07:02:16.016031+0000","last_unstale":"2024-09-17T07:04:33.112166+0000","last_undegraded":"2024-09-17T07:04:33.112166+0000","last_fullsized":"2024-09-17T07:04:33.112166+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:30.069608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:30.069608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:36.831454+0000","scrub_duration":0.0012155379999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":52,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.119290+0000","last_change":"2024-09-17T07:03:58.670660+0000","last_active":"2024-09-17T07:04:33.119290+0000","last_peered":"2024-09-17T07:04:33.119290+0000","last_clean":"2024-09-17T07:04:33.119290+0000","last_became_active":"2024-09-17T07:02:18.023321+0000","last_became_peered":"2024-09-17T07:02:18.023321+0000","last_unstale":"2024-09-17T07:04:33.119290+0000","last_undegraded":"2024-09-17T07:04:33.119290+0000","last_fullsized":"2024-09-17T07:04:33.119290+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:58.670571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:58.670571+0000","last_clean_scrub_stamp":"2024-09-17T07:03:58.670571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:12.628083+0000","scrub_duration":0.0013978409999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.9","version":"105'1147","reported_seq":1161,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T07:04:38.017653+0000","last_change":"2024-09-17T07:04:20.803800+0000","last_active":"2024-09-17T07:04:38.017653+0000","last_peered":"2024-09-17T07:04:38.017653+0000","last_clean":"2024-09-17T07:04:38.017653+0000","last_became_active":"2024-09-17T07:04:20.803381+0000","last_became_peered":"2024-09-17T07:04:20.803381+0000","last_unstale":"2024-09-17T07:04:38.017653+0000","last_undegraded":"2024-09-17T07:04:38.017653+0000","last_fullsized":"2024-09-17T07:04:38.017653+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_clean_scrub_stamp":"2024-09-17T07:04:19.785747+0000","objects_scrubbed":0,"log_size":1147,"log_dups_size":0,"ondisk_log_size":1147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:35.596031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75169792,"num_objects":1147,"num_object_clones":0,"num_object_copies":3441,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1147,"num_write_kb":73408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"75'19","reported_seq":87,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.115304+0000","last_change":"2024-09-17T07:03:37.517877+0000","last_active":"2024-09-17T07:04:33.115304+0000","last_peered":"2024-09-17T07:04:33.115304+0000","last_clean":"2024-09-17T07:04:33.115304+0000","last_became_active":"2024-09-17T07:02:14.027313+0000","last_became_peered":"2024-09-17T07:02:14.027313+0000","last_unstale":"2024-09-17T07:04:33.115304+0000","last_undegraded":"2024-09-17T07:04:33.115304+0000","last_fullsized":"2024-09-17T07:04:33.115304+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-09-17T07:03:37.517760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:37.517760+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:03.675532+0000","scrub_duration":0.0028316579999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":59,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.114282+0000","last_change":"2024-09-17T07:03:25.672437+0000","last_active":"2024-09-17T07:04:33.114282+0000","last_peered":"2024-09-17T07:04:33.114282+0000","last_clean":"2024-09-17T07:04:33.114282+0000","last_became_active":"2024-09-17T07:02:12.012037+0000","last_became_peered":"2024-09-17T07:02:12.012037+0000","last_unstale":"2024-09-17T07:04:33.114282+0000","last_undegraded":"2024-09-17T07:04:33.114282+0000","last_fullsized":"2024-09-17T07:04:33.114282+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:25.672360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:25.672360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:54.809122+0000","scrub_duration":0.0012704459999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":55,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.113553+0000","last_change":"2024-09-17T07:03:27.660802+0000","last_active":"2024-09-17T07:04:33.113553+0000","last_peered":"2024-09-17T07:04:33.113553+0000","last_clean":"2024-09-17T07:04:33.113553+0000","last_became_active":"2024-09-17T07:02:16.014631+0000","last_became_peered":"2024-09-17T07:02:16.014631+0000","last_unstale":"2024-09-17T07:04:33.113553+0000","last_undegraded":"2024-09-17T07:04:33.113553+0000","last_fullsized":"2024-09-17T07:04:33.113553+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:27.660721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:27.660721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:50.654559+0000","scrub_duration":0.0012746820000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":53,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.112044+0000","last_change":"2024-09-17T07:03:58.029849+0000","last_active":"2024-09-17T07:04:33.112044+0000","last_peered":"2024-09-17T07:04:33.112044+0000","last_clean":"2024-09-17T07:04:33.112044+0000","last_became_active":"2024-09-17T07:02:18.023299+0000","last_became_peered":"2024-09-17T07:02:18.023299+0000","last_unstale":"2024-09-17T07:04:33.112044+0000","last_undegraded":"2024-09-17T07:04:33.112044+0000","last_fullsized":"2024-09-17T07:04:33.112044+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:58.029561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:03:58.029561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:10.859086+0000","scrub_duration":0.0020006690000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.8","version":"105'1202","reported_seq":1217,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T07:04:38.001756+0000","last_change":"2024-09-17T07:04:20.803041+0000","last_active":"2024-09-17T07:04:38.001756+0000","last_peered":"2024-09-17T07:04:38.001756+0000","last_clean":"2024-09-17T07:04:38.001756+0000","last_became_active":"2024-09-17T07:04:20.802560+0000","last_became_peered":"2024-09-17T07:04:20.802560+0000","last_unstale":"2024-09-17T07:04:38.001756+0000","last_undegraded":"2024-09-17T07:04:38.001756+0000","last_fullsized":"2024-09-17T07:04:38.001756+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_clean_scrub_stamp":"2024-09-17T07:04:19.785747+0000","objects_scrubbed":0,"log_size":1202,"log_dups_size":0,"ondisk_log_size":1202,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:27.315772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78774272,"num_objects":1202,"num_object_clones":0,"num_object_copies":3606,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1202,"num_write_kb":76928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"75'9","reported_seq":73,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.111852+0000","last_change":"2024-09-17T07:03:46.892737+0000","last_active":"2024-09-17T07:04:33.111852+0000","last_peered":"2024-09-17T07:04:33.111852+0000","last_clean":"2024-09-17T07:04:33.111852+0000","last_became_active":"2024-09-17T07:02:14.014939+0000","last_became_peered":"2024-09-17T07:02:14.014939+0000","last_unstale":"2024-09-17T07:04:33.111852+0000","last_undegraded":"2024-09-17T07:04:33.111852+0000","last_fullsized":"2024-09-17T07:04:33.111852+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:03:46.892648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:46.892648+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:54.689233+0000","scrub_duration":0.002021162,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":58,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.120024+0000","last_change":"2024-09-17T07:03:40.074926+0000","last_active":"2024-09-17T07:04:33.120024+0000","last_peered":"2024-09-17T07:04:33.120024+0000","last_clean":"2024-09-17T07:04:33.120024+0000","last_became_active":"2024-09-17T07:02:12.000005+0000","last_became_peered":"2024-09-17T07:02:12.000005+0000","last_unstale":"2024-09-17T07:04:33.120024+0000","last_undegraded":"2024-09-17T07:04:33.120024+0000","last_fullsized":"2024-09-17T07:04:33.120024+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:40.074839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:40.074839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:50.612094+0000","scrub_duration":0.0013162040000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":19,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.114024+0000","last_change":"2024-09-17T07:04:15.108167+0000","last_active":"2024-09-17T07:04:33.114024+0000","last_peered":"2024-09-17T07:04:33.114024+0000","last_clean":"2024-09-17T07:04:33.114024+0000","last_became_active":"2024-09-17T07:04:15.107886+0000","last_became_peered":"2024-09-17T07:04:15.107886+0000","last_unstale":"2024-09-17T07:04:33.114024+0000","last_undegraded":"2024-09-17T07:04:33.114024+0000","last_fullsized":"2024-09-17T07:04:33.114024+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:43.943168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:43.943168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:44.347240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":53,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.111618+0000","last_change":"2024-09-17T07:03:58.099938+0000","last_active":"2024-09-17T07:04:33.111618+0000","last_peered":"2024-09-17T07:04:33.111618+0000","last_clean":"2024-09-17T07:04:33.111618+0000","last_became_active":"2024-09-17T07:02:18.026994+0000","last_became_peered":"2024-09-17T07:02:18.026994+0000","last_unstale":"2024-09-17T07:04:33.111618+0000","last_undegraded":"2024-09-17T07:04:33.111618+0000","last_fullsized":"2024-09-17T07:04:33.111618+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:58.099849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:58.099849+0000","last_clean_scrub_stamp":"2024-09-17T07:03:58.099849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:04.059986+0000","scrub_duration":0.0021287960000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.b","version":"105'1201","reported_seq":1216,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T07:04:38.010516+0000","last_change":"2024-09-17T07:04:20.803445+0000","last_active":"2024-09-17T07:04:38.010516+0000","last_peered":"2024-09-17T07:04:38.010516+0000","last_clean":"2024-09-17T07:04:38.010516+0000","last_became_active":"2024-09-17T07:04:20.803245+0000","last_became_peered":"2024-09-17T07:04:20.803245+0000","last_unstale":"2024-09-17T07:04:38.010516+0000","last_undegraded":"2024-09-17T07:04:38.010516+0000","last_fullsized":"2024-09-17T07:04:38.010516+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_clean_scrub_stamp":"2024-09-17T07:04:19.785747+0000","objects_scrubbed":0,"log_size":1201,"log_dups_size":0,"ondisk_log_size":1201,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:48.872852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78708736,"num_objects":1201,"num_object_clones":0,"num_object_copies":3603,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1201,"num_write_kb":76864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"75'15","reported_seq":81,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.118464+0000","last_change":"2024-09-17T07:03:50.738059+0000","last_active":"2024-09-17T07:04:33.118464+0000","last_peered":"2024-09-17T07:04:33.118464+0000","last_clean":"2024-09-17T07:04:33.118464+0000","last_became_active":"2024-09-17T07:02:14.028392+0000","last_became_peered":"2024-09-17T07:02:14.028392+0000","last_unstale":"2024-09-17T07:04:33.118464+0000","last_undegraded":"2024-09-17T07:04:33.118464+0000","last_fullsized":"2024-09-17T07:04:33.118464+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-09-17T07:03:50.737908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:50.737908+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:54.852092+0000","scrub_duration":0.0024040049999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'2","reported_seq":59,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.117910+0000","last_change":"2024-09-17T07:04:15.110390+0000","last_active":"2024-09-17T07:04:33.117910+0000","last_peered":"2024-09-17T07:04:33.117910+0000","last_clean":"2024-09-17T07:04:33.117910+0000","last_became_active":"2024-09-17T07:04:15.110085+0000","last_became_peered":"2024-09-17T07:04:15.110085+0000","last_unstale":"2024-09-17T07:04:33.117910+0000","last_undegraded":"2024-09-17T07:04:33.117910+0000","last_fullsized":"2024-09-17T07:04:33.117910+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-09-17T07:03:26.694366+0000","last_deep_scrub":"89'2","last_deep_scrub_stamp":"2024-09-17T07:03:26.694366+0000","last_clean_scrub_stamp":"2024-09-17T07:03:26.694366+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:41.441224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":420,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"75'8","reported_seq":95,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T07:04:34.122535+0000","last_change":"2024-09-17T07:03:45.995460+0000","last_active":"2024-09-17T07:04:34.122535+0000","last_peered":"2024-09-17T07:04:34.122535+0000","last_clean":"2024-09-17T07:04:34.122535+0000","last_became_active":"2024-09-17T07:02:16.015418+0000","last_became_peered":"2024-09-17T07:02:16.015418+0000","last_unstale":"2024-09-17T07:04:34.122535+0000","last_undegraded":"2024-09-17T07:04:34.122535+0000","last_fullsized":"2024-09-17T07:04:34.122535+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:03:45.995358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:45.995358+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:52.038789+0000","scrub_duration":0.001465163,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":52,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.118504+0000","last_change":"2024-09-17T07:03:22.823018+0000","last_active":"2024-09-17T07:04:33.118504+0000","last_peered":"2024-09-17T07:04:33.118504+0000","last_clean":"2024-09-17T07:04:33.118504+0000","last_became_active":"2024-09-17T07:02:18.363253+0000","last_became_peered":"2024-09-17T07:02:18.363253+0000","last_unstale":"2024-09-17T07:04:33.118504+0000","last_undegraded":"2024-09-17T07:04:33.118504+0000","last_fullsized":"2024-09-17T07:04:33.118504+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:22.822922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:03:22.822922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0012979910000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.a","version":"105'1182","reported_seq":1197,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T07:04:38.007265+0000","last_change":"2024-09-17T07:04:20.804570+0000","last_active":"2024-09-17T07:04:38.007265+0000","last_peered":"2024-09-17T07:04:38.007265+0000","last_clean":"2024-09-17T07:04:38.007265+0000","last_became_active":"2024-09-17T07:04:20.804306+0000","last_became_peered":"2024-09-17T07:04:20.804306+0000","last_unstale":"2024-09-17T07:04:38.007265+0000","last_undegraded":"2024-09-17T07:04:38.007265+0000","last_fullsized":"2024-09-17T07:04:38.007265+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_clean_scrub_stamp":"2024-09-17T07:04:19.785747+0000","objects_scrubbed":0,"log_size":1182,"log_dups_size":0,"ondisk_log_size":1182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:30.649460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77463552,"num_objects":1182,"num_object_clones":0,"num_object_copies":3546,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1182,"num_write_kb":75648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"75'12","reported_seq":80,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.113859+0000","last_change":"2024-09-17T07:03:39.074060+0000","last_active":"2024-09-17T07:04:33.113859+0000","last_peered":"2024-09-17T07:04:33.113859+0000","last_clean":"2024-09-17T07:04:33.113859+0000","last_became_active":"2024-09-17T07:02:14.010219+0000","last_became_peered":"2024-09-17T07:02:14.010219+0000","last_unstale":"2024-09-17T07:04:33.113859+0000","last_undegraded":"2024-09-17T07:04:33.113859+0000","last_fullsized":"2024-09-17T07:04:33.113859+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-09-17T07:03:39.073980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:39.073980+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:07.174292+0000","scrub_duration":0.002114061,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"68'2","reported_seq":66,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.114607+0000","last_change":"2024-09-17T07:03:28.021306+0000","last_active":"2024-09-17T07:04:33.114607+0000","last_peered":"2024-09-17T07:04:33.114607+0000","last_clean":"2024-09-17T07:04:33.114607+0000","last_became_active":"2024-09-17T07:02:12.006716+0000","last_became_peered":"2024-09-17T07:02:12.006716+0000","last_unstale":"2024-09-17T07:04:33.114607+0000","last_undegraded":"2024-09-17T07:04:33.114607+0000","last_fullsized":"2024-09-17T07:04:33.114607+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-09-17T07:03:28.021180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:28.021180+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:55.299746+0000","scrub_duration":0.0016672220000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":19,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.111307+0000","last_change":"2024-09-17T07:04:15.111485+0000","last_active":"2024-09-17T07:04:33.111307+0000","last_peered":"2024-09-17T07:04:33.111307+0000","last_clean":"2024-09-17T07:04:33.111307+0000","last_became_active":"2024-09-17T07:04:15.111069+0000","last_became_peered":"2024-09-17T07:04:15.111069+0000","last_unstale":"2024-09-17T07:04:33.111307+0000","last_undegraded":"2024-09-17T07:04:33.111307+0000","last_fullsized":"2024-09-17T07:04:33.111307+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:36.832752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:36.832752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:48.239101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":52,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.117639+0000","last_change":"2024-09-17T07:04:06.571270+0000","last_active":"2024-09-17T07:04:33.117639+0000","last_peered":"2024-09-17T07:04:33.117639+0000","last_clean":"2024-09-17T07:04:33.117639+0000","last_became_active":"2024-09-17T07:02:18.024983+0000","last_became_peered":"2024-09-17T07:02:18.024983+0000","last_unstale":"2024-09-17T07:04:33.117639+0000","last_undegraded":"2024-09-17T07:04:33.117639+0000","last_fullsized":"2024-09-17T07:04:33.117639+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:06.571184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:04:06.571184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:18.403932+0000","scrub_duration":0.001613319,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"75'9","reported_seq":73,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T07:04:34.124998+0000","last_change":"2024-09-17T07:03:26.005290+0000","last_active":"2024-09-17T07:04:34.124998+0000","last_peered":"2024-09-17T07:04:34.124998+0000","last_clean":"2024-09-17T07:04:34.124998+0000","last_became_active":"2024-09-17T07:02:14.029388+0000","last_became_peered":"2024-09-17T07:02:14.029388+0000","last_unstale":"2024-09-17T07:04:34.124998+0000","last_undegraded":"2024-09-17T07:04:34.124998+0000","last_fullsized":"2024-09-17T07:04:34.124998+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:03:26.005201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:26.005201+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:55.043365+0000","scrub_duration":0.0018022279999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":59,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.112289+0000","last_change":"2024-09-17T07:03:31.753577+0000","last_active":"2024-09-17T07:04:33.112289+0000","last_peered":"2024-09-17T07:04:33.112289+0000","last_clean":"2024-09-17T07:04:33.112289+0000","last_became_active":"2024-09-17T07:02:12.011327+0000","last_became_peered":"2024-09-17T07:02:12.011327+0000","last_unstale":"2024-09-17T07:04:33.112289+0000","last_undegraded":"2024-09-17T07:04:33.112289+0000","last_fullsized":"2024-09-17T07:04:33.112289+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:31.753470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:31.753470+0000","last_clean_scrub_stamp":"2024-09-17T07:03:31.753470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:51.025434+0000","scrub_duration":0.0012687670000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":64,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T07:04:36.651944+0000","last_change":"2024-09-17T07:04:36.652033+0000","last_active":"2024-09-17T07:04:36.651944+0000","last_peered":"2024-09-17T07:04:36.651944+0000","last_clean":"2024-09-17T07:04:36.651944+0000","last_became_active":"2024-09-17T07:02:16.022584+0000","last_became_peered":"2024-09-17T07:02:16.022584+0000","last_unstale":"2024-09-17T07:04:36.651944+0000","last_undegraded":"2024-09-17T07:04:36.651944+0000","last_fullsized":"2024-09-17T07:04:36.651944+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:36.651925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:04:36.651925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:55.917980+0000","scrub_duration":0.0061789210000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":64,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T07:04:34.129397+0000","last_change":"2024-09-17T07:04:33.158102+0000","last_active":"2024-09-17T07:04:34.129397+0000","last_peered":"2024-09-17T07:04:34.129397+0000","last_clean":"2024-09-17T07:04:34.129397+0000","last_became_active":"2024-09-17T07:02:18.017825+0000","last_became_peered":"2024-09-17T07:02:18.017825+0000","last_unstale":"2024-09-17T07:04:34.129397+0000","last_undegraded":"2024-09-17T07:04:34.129397+0000","last_fullsized":"2024-09-17T07:04:34.129397+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:33.157964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:04:33.157964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:58.257716+0000","scrub_duration":0.0042917379999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":58,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.115188+0000","last_change":"2024-09-17T07:03:53.537565+0000","last_active":"2024-09-17T07:04:33.115188+0000","last_peered":"2024-09-17T07:04:33.115188+0000","last_clean":"2024-09-17T07:04:33.115188+0000","last_became_active":"2024-09-17T07:02:12.008065+0000","last_became_peered":"2024-09-17T07:02:12.008065+0000","last_unstale":"2024-09-17T07:04:33.115188+0000","last_undegraded":"2024-09-17T07:04:33.115188+0000","last_fullsized":"2024-09-17T07:04:33.115188+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:53.537463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:53.537463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:03.946844+0000","scrub_duration":0.0016610469999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"75'6","reported_seq":64,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.118007+0000","last_change":"2024-09-17T07:03:36.430683+0000","last_active":"2024-09-17T07:04:33.118007+0000","last_peered":"2024-09-17T07:04:33.118007+0000","last_clean":"2024-09-17T07:04:33.118007+0000","last_became_active":"2024-09-17T07:02:14.028655+0000","last_became_peered":"2024-09-17T07:02:14.028655+0000","last_unstale":"2024-09-17T07:04:33.118007+0000","last_undegraded":"2024-09-17T07:04:33.118007+0000","last_fullsized":"2024-09-17T07:04:33.118007+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'6","last_scrub_stamp":"2024-09-17T07:03:36.430591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:36.430591+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:44.221735+0000","scrub_duration":0.0020997730000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":53,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.119069+0000","last_change":"2024-09-17T07:03:39.844283+0000","last_active":"2024-09-17T07:04:33.119069+0000","last_peered":"2024-09-17T07:04:33.119069+0000","last_clean":"2024-09-17T07:04:33.119069+0000","last_became_active":"2024-09-17T07:02:16.016634+0000","last_became_peered":"2024-09-17T07:02:16.016634+0000","last_unstale":"2024-09-17T07:04:33.119069+0000","last_undegraded":"2024-09-17T07:04:33.119069+0000","last_fullsized":"2024-09-17T07:04:33.119069+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:39.844196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:39.844196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:07.747427+0000","scrub_duration":0.001646652,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":52,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.112006+0000","last_change":"2024-09-17T07:03:41.093127+0000","last_active":"2024-09-17T07:04:33.112006+0000","last_peered":"2024-09-17T07:04:33.112006+0000","last_clean":"2024-09-17T07:04:33.112006+0000","last_became_active":"2024-09-17T07:02:18.365419+0000","last_became_peered":"2024-09-17T07:02:18.365419+0000","last_unstale":"2024-09-17T07:04:33.112006+0000","last_undegraded":"2024-09-17T07:04:33.112006+0000","last_fullsized":"2024-09-17T07:04:33.112006+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:41.093049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:03:41.093049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:58.545295+0000","scrub_duration":0.001187794,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"75'10","reported_seq":71,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T07:04:34.119838+0000","last_change":"2024-09-17T07:03:14.178469+0000","last_active":"2024-09-17T07:04:34.119838+0000","last_peered":"2024-09-17T07:04:34.119838+0000","last_clean":"2024-09-17T07:04:34.119838+0000","last_became_active":"2024-09-17T07:02:14.027499+0000","last_became_peered":"2024-09-17T07:02:14.027499+0000","last_unstale":"2024-09-17T07:04:34.119838+0000","last_undegraded":"2024-09-17T07:04:34.119838+0000","last_fullsized":"2024-09-17T07:04:34.119838+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-09-17T07:03:14.178375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:14.178375+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021187430000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":58,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.118704+0000","last_change":"2024-09-17T07:03:25.849841+0000","last_active":"2024-09-17T07:04:33.118704+0000","last_peered":"2024-09-17T07:04:33.118704+0000","last_clean":"2024-09-17T07:04:33.118704+0000","last_became_active":"2024-09-17T07:02:12.007280+0000","last_became_peered":"2024-09-17T07:02:12.007280+0000","last_unstale":"2024-09-17T07:04:33.118704+0000","last_undegraded":"2024-09-17T07:04:33.118704+0000","last_fullsized":"2024-09-17T07:04:33.118704+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:25.849698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:25.849698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:55.197183+0000","scrub_duration":0.0014293929999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"75'8","reported_seq":90,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T07:04:34.126162+0000","last_change":"2024-09-17T07:03:46.807330+0000","last_active":"2024-09-17T07:04:34.126162+0000","last_peered":"2024-09-17T07:04:34.126162+0000","last_clean":"2024-09-17T07:04:34.126162+0000","last_became_active":"2024-09-17T07:02:16.022583+0000","last_became_peered":"2024-09-17T07:02:16.022583+0000","last_unstale":"2024-09-17T07:04:34.126162+0000","last_undegraded":"2024-09-17T07:04:34.126162+0000","last_fullsized":"2024-09-17T07:04:34.126162+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:03:46.807209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:46.807209+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:03.451386+0000","scrub_duration":0.0021619619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":52,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.113461+0000","last_change":"2024-09-17T07:03:53.976711+0000","last_active":"2024-09-17T07:04:33.113461+0000","last_peered":"2024-09-17T07:04:33.113461+0000","last_clean":"2024-09-17T07:04:33.113461+0000","last_became_active":"2024-09-17T07:02:18.028796+0000","last_became_peered":"2024-09-17T07:02:18.028796+0000","last_unstale":"2024-09-17T07:04:33.113461+0000","last_undegraded":"2024-09-17T07:04:33.113461+0000","last_fullsized":"2024-09-17T07:04:33.113461+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:53.976586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:03:53.976586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:20.926942+0000","scrub_duration":0.0021355279999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"75'9","reported_seq":83,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T07:04:36.651763+0000","last_change":"2024-09-17T07:04:36.651763+0000","last_active":"2024-09-17T07:04:36.651763+0000","last_peered":"2024-09-17T07:04:36.651763+0000","last_clean":"2024-09-17T07:04:36.651763+0000","last_became_active":"2024-09-17T07:02:14.028485+0000","last_became_peered":"2024-09-17T07:02:14.028485+0000","last_unstale":"2024-09-17T07:04:36.651763+0000","last_undegraded":"2024-09-17T07:04:36.651763+0000","last_fullsized":"2024-09-17T07:04:36.651763+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:04:36.651616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:04:36.651616+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:38.837798+0000","scrub_duration":0.003618132,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":59,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T07:04:34.122307+0000","last_change":"2024-09-17T07:03:16.948419+0000","last_active":"2024-09-17T07:04:34.122307+0000","last_peered":"2024-09-17T07:04:34.122307+0000","last_clean":"2024-09-17T07:04:34.122307+0000","last_became_active":"2024-09-17T07:02:12.009935+0000","last_became_peered":"2024-09-17T07:02:12.009935+0000","last_unstale":"2024-09-17T07:04:34.122307+0000","last_undegraded":"2024-09-17T07:04:34.122307+0000","last_fullsized":"2024-09-17T07:04:34.122307+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:16.948330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:16.948330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:46.609922+0000","scrub_duration":0.0017835489999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"75'8","reported_seq":91,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T07:04:35.546024+0000","last_change":"2024-09-17T07:03:42.469019+0000","last_active":"2024-09-17T07:04:35.546024+0000","last_peered":"2024-09-17T07:04:35.546024+0000","last_clean":"2024-09-17T07:04:35.546024+0000","last_became_active":"2024-09-17T07:02:16.016069+0000","last_became_peered":"2024-09-17T07:02:16.016069+0000","last_unstale":"2024-09-17T07:04:35.546024+0000","last_undegraded":"2024-09-17T07:04:35.546024+0000","last_fullsized":"2024-09-17T07:04:35.546024+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:03:42.468934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:42.468934+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:07.697686+0000","scrub_duration":0.0016937079999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":52,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.114435+0000","last_change":"2024-09-17T07:03:31.045257+0000","last_active":"2024-09-17T07:04:33.114435+0000","last_peered":"2024-09-17T07:04:33.114435+0000","last_clean":"2024-09-17T07:04:33.114435+0000","last_became_active":"2024-09-17T07:02:18.018716+0000","last_became_peered":"2024-09-17T07:02:18.018716+0000","last_unstale":"2024-09-17T07:04:33.114435+0000","last_undegraded":"2024-09-17T07:04:33.114435+0000","last_fullsized":"2024-09-17T07:04:33.114435+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:31.045167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:03:31.045167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:36.218836+0000","scrub_duration":0.0012599709999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"75'9","reported_seq":72,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.120543+0000","last_change":"2024-09-17T07:03:20.116712+0000","last_active":"2024-09-17T07:04:33.120543+0000","last_peered":"2024-09-17T07:04:33.120543+0000","last_clean":"2024-09-17T07:04:33.120543+0000","last_became_active":"2024-09-17T07:02:14.010289+0000","last_became_peered":"2024-09-17T07:02:14.010289+0000","last_unstale":"2024-09-17T07:04:33.120543+0000","last_undegraded":"2024-09-17T07:04:33.120543+0000","last_fullsized":"2024-09-17T07:04:33.120543+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:03:20.116602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:20.116602+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:35.267610+0000","scrub_duration":0.0032469130000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":58,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T07:04:34.121355+0000","last_change":"2024-09-17T07:03:51.518619+0000","last_active":"2024-09-17T07:04:34.121355+0000","last_peered":"2024-09-17T07:04:34.121355+0000","last_clean":"2024-09-17T07:04:34.121355+0000","last_became_active":"2024-09-17T07:02:12.007433+0000","last_became_peered":"2024-09-17T07:02:12.007433+0000","last_unstale":"2024-09-17T07:04:34.121355+0000","last_undegraded":"2024-09-17T07:04:34.121355+0000","last_fullsized":"2024-09-17T07:04:34.121355+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:51.518511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:51.518511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:09.559204+0000","scrub_duration":0.00177655,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":54,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T07:04:34.120462+0000","last_change":"2024-09-17T07:03:51.487699+0000","last_active":"2024-09-17T07:04:34.120462+0000","last_peered":"2024-09-17T07:04:34.120462+0000","last_clean":"2024-09-17T07:04:34.120462+0000","last_became_active":"2024-09-17T07:02:16.016216+0000","last_became_peered":"2024-09-17T07:02:16.016216+0000","last_unstale":"2024-09-17T07:04:34.120462+0000","last_undegraded":"2024-09-17T07:04:34.120462+0000","last_fullsized":"2024-09-17T07:04:34.120462+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:51.487610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:51.487610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:56.904290+0000","scrub_duration":0.0012946850000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":53,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T07:04:34.121465+0000","last_change":"2024-09-17T07:03:24.014436+0000","last_active":"2024-09-17T07:04:34.121465+0000","last_peered":"2024-09-17T07:04:34.121465+0000","last_clean":"2024-09-17T07:04:34.121465+0000","last_became_active":"2024-09-17T07:02:18.027622+0000","last_became_peered":"2024-09-17T07:02:18.027622+0000","last_unstale":"2024-09-17T07:04:34.121465+0000","last_undegraded":"2024-09-17T07:04:34.121465+0000","last_fullsized":"2024-09-17T07:04:34.121465+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:24.014336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:03:24.014336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:50.998219+0000","scrub_duration":0.001425125,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"75'11","reported_seq":76,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T07:04:34.116934+0000","last_change":"2024-09-17T07:03:18.899916+0000","last_active":"2024-09-17T07:04:34.116934+0000","last_peered":"2024-09-17T07:04:34.116934+0000","last_clean":"2024-09-17T07:04:34.116934+0000","last_became_active":"2024-09-17T07:02:14.016255+0000","last_became_peered":"2024-09-17T07:02:14.016255+0000","last_unstale":"2024-09-17T07:04:34.116934+0000","last_undegraded":"2024-09-17T07:04:34.116934+0000","last_fullsized":"2024-09-17T07:04:34.116934+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-09-17T07:03:18.899814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:18.899814+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:46.161729+0000","scrub_duration":0.0020491419999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"94'2","reported_seq":64,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T07:04:34.127062+0000","last_change":"2024-09-17T07:04:06.027032+0000","last_active":"2024-09-17T07:04:34.127062+0000","last_peered":"2024-09-17T07:04:34.127062+0000","last_clean":"2024-09-17T07:04:34.127062+0000","last_became_active":"2024-09-17T07:02:12.011461+0000","last_became_peered":"2024-09-17T07:02:12.011461+0000","last_unstale":"2024-09-17T07:04:34.127062+0000","last_undegraded":"2024-09-17T07:04:34.127062+0000","last_fullsized":"2024-09-17T07:04:34.127062+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-09-17T07:04:06.026890+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-09-17T07:04:06.026890+0000","last_clean_scrub_stamp":"2024-09-17T07:04:06.026890+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:23.430992+0000","scrub_duration":0.0031797539999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":54,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T07:04:34.124246+0000","last_change":"2024-09-17T07:03:33.126060+0000","last_active":"2024-09-17T07:04:34.124246+0000","last_peered":"2024-09-17T07:04:34.124246+0000","last_clean":"2024-09-17T07:04:34.124246+0000","last_became_active":"2024-09-17T07:02:16.018754+0000","last_became_peered":"2024-09-17T07:02:16.018754+0000","last_unstale":"2024-09-17T07:04:34.124246+0000","last_undegraded":"2024-09-17T07:04:34.124246+0000","last_fullsized":"2024-09-17T07:04:34.124246+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:33.125915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:33.125915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:56.098551+0000","scrub_duration":0.033614225999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":51,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.118339+0000","last_change":"2024-09-17T07:04:07.566836+0000","last_active":"2024-09-17T07:04:33.118339+0000","last_peered":"2024-09-17T07:04:33.118339+0000","last_clean":"2024-09-17T07:04:33.118339+0000","last_became_active":"2024-09-17T07:02:18.027427+0000","last_became_peered":"2024-09-17T07:02:18.027427+0000","last_unstale":"2024-09-17T07:04:33.118339+0000","last_undegraded":"2024-09-17T07:04:33.118339+0000","last_fullsized":"2024-09-17T07:04:33.118339+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:07.566667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:04:07.566667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:26.574551+0000","scrub_duration":0.001538364,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"75'4","reported_seq":77,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.117153+0000","last_change":"2024-09-17T07:03:48.763164+0000","last_active":"2024-09-17T07:04:33.117153+0000","last_peered":"2024-09-17T07:04:33.117153+0000","last_clean":"2024-09-17T07:04:33.117153+0000","last_became_active":"2024-09-17T07:03:48.758220+0000","last_became_peered":"2024-09-17T07:03:48.758220+0000","last_unstale":"2024-09-17T07:04:33.117153+0000","last_undegraded":"2024-09-17T07:04:33.117153+0000","last_fullsized":"2024-09-17T07:04:33.117153+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'4","last_scrub_stamp":"2024-09-17T07:03:24.431043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:24.431043+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:38.933831+0000","scrub_duration":0.0015094500000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":58,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T07:04:34.127167+0000","last_change":"2024-09-17T07:03:44.809127+0000","last_active":"2024-09-17T07:04:34.127167+0000","last_peered":"2024-09-17T07:04:34.127167+0000","last_clean":"2024-09-17T07:04:34.127167+0000","last_became_active":"2024-09-17T07:02:12.008898+0000","last_became_peered":"2024-09-17T07:02:12.008898+0000","last_unstale":"2024-09-17T07:04:34.127167+0000","last_undegraded":"2024-09-17T07:04:34.127167+0000","last_fullsized":"2024-09-17T07:04:34.127167+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:44.809025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:44.809025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:03.894700+0000","scrub_duration":0.001586154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":54,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T07:04:34.120844+0000","last_change":"2024-09-17T07:03:47.445076+0000","last_active":"2024-09-17T07:04:34.120844+0000","last_peered":"2024-09-17T07:04:34.120844+0000","last_clean":"2024-09-17T07:04:34.120844+0000","last_became_active":"2024-09-17T07:02:16.014982+0000","last_became_peered":"2024-09-17T07:02:16.014982+0000","last_unstale":"2024-09-17T07:04:34.120844+0000","last_undegraded":"2024-09-17T07:04:34.120844+0000","last_fullsized":"2024-09-17T07:04:34.120844+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:47.444972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:47.444972+0000","last_clean_scrub_stamp":"2024-09-17T07:03:47.444972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:56.111360+0000","scrub_duration":0.0015734119999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":52,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.111657+0000","last_change":"2024-09-17T07:03:53.030072+0000","last_active":"2024-09-17T07:04:33.111657+0000","last_peered":"2024-09-17T07:04:33.111657+0000","last_clean":"2024-09-17T07:04:33.111657+0000","last_became_active":"2024-09-17T07:02:18.021014+0000","last_became_peered":"2024-09-17T07:02:18.021014+0000","last_unstale":"2024-09-17T07:04:33.111657+0000","last_undegraded":"2024-09-17T07:04:33.111657+0000","last_fullsized":"2024-09-17T07:04:33.111657+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:53.029956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:03:53.029956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:58.219531+0000","scrub_duration":0.0013687969999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"75'11","reported_seq":75,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T07:04:34.119672+0000","last_change":"2024-09-17T07:03:17.270057+0000","last_active":"2024-09-17T07:04:34.119672+0000","last_peered":"2024-09-17T07:04:34.119672+0000","last_clean":"2024-09-17T07:04:34.119672+0000","last_became_active":"2024-09-17T07:02:14.027316+0000","last_became_peered":"2024-09-17T07:02:14.027316+0000","last_unstale":"2024-09-17T07:04:34.119672+0000","last_undegraded":"2024-09-17T07:04:34.119672+0000","last_fullsized":"2024-09-17T07:04:34.119672+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-09-17T07:03:17.269941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:17.269941+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002326505,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":58,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T07:04:34.121807+0000","last_change":"2024-09-17T07:03:33.533200+0000","last_active":"2024-09-17T07:04:34.121807+0000","last_peered":"2024-09-17T07:04:34.121807+0000","last_clean":"2024-09-17T07:04:34.121807+0000","last_became_active":"2024-09-17T07:02:12.010195+0000","last_became_peered":"2024-09-17T07:02:12.010195+0000","last_unstale":"2024-09-17T07:04:34.121807+0000","last_undegraded":"2024-09-17T07:04:34.121807+0000","last_fullsized":"2024-09-17T07:04:34.121807+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:33.533112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:33.533112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:53.826577+0000","scrub_duration":0.0014453420000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":55,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.111516+0000","last_change":"2024-09-17T07:03:37.063510+0000","last_active":"2024-09-17T07:04:33.111516+0000","last_peered":"2024-09-17T07:04:33.111516+0000","last_clean":"2024-09-17T07:04:33.111516+0000","last_became_active":"2024-09-17T07:02:16.016183+0000","last_became_peered":"2024-09-17T07:02:16.016183+0000","last_unstale":"2024-09-17T07:04:33.111516+0000","last_undegraded":"2024-09-17T07:04:33.111516+0000","last_fullsized":"2024-09-17T07:04:33.111516+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:37.063409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:37.063409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:55.794307+0000","scrub_duration":0.0015220329999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":51,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.117457+0000","last_change":"2024-09-17T07:03:29.467351+0000","last_active":"2024-09-17T07:04:33.117457+0000","last_peered":"2024-09-17T07:04:33.117457+0000","last_clean":"2024-09-17T07:04:33.117457+0000","last_became_active":"2024-09-17T07:02:18.365656+0000","last_became_peered":"2024-09-17T07:02:18.365656+0000","last_unstale":"2024-09-17T07:04:33.117457+0000","last_undegraded":"2024-09-17T07:04:33.117457+0000","last_fullsized":"2024-09-17T07:04:33.117457+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:29.467252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:03:29.467252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:58.741564+0000","scrub_duration":0.001842804,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"75'1","reported_seq":53,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.112071+0000","last_change":"2024-09-17T07:03:46.957523+0000","last_active":"2024-09-17T07:04:33.112071+0000","last_peered":"2024-09-17T07:04:33.112071+0000","last_clean":"2024-09-17T07:04:33.112071+0000","last_became_active":"2024-09-17T07:02:18.021855+0000","last_became_peered":"2024-09-17T07:02:18.021855+0000","last_unstale":"2024-09-17T07:04:33.112071+0000","last_undegraded":"2024-09-17T07:04:33.112071+0000","last_fullsized":"2024-09-17T07:04:33.112071+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-17T07:03:46.957414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:03:46.957414+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:53.461463+0000","scrub_duration":0.00186878,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":57,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.118462+0000","last_change":"2024-09-17T07:03:19.775506+0000","last_active":"2024-09-17T07:04:33.118462+0000","last_peered":"2024-09-17T07:04:33.118462+0000","last_clean":"2024-09-17T07:04:33.118462+0000","last_became_active":"2024-09-17T07:02:12.005329+0000","last_became_peered":"2024-09-17T07:02:12.005329+0000","last_unstale":"2024-09-17T07:04:33.118462+0000","last_undegraded":"2024-09-17T07:04:33.118462+0000","last_fullsized":"2024-09-17T07:04:33.118462+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:19.775390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:19.775390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:48.303892+0000","scrub_duration":0.0011915630000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"75'10","reported_seq":72,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.112786+0000","last_change":"2024-09-17T07:03:56.069095+0000","last_active":"2024-09-17T07:04:33.112786+0000","last_peered":"2024-09-17T07:04:33.112786+0000","last_clean":"2024-09-17T07:04:33.112786+0000","last_became_active":"2024-09-17T07:02:14.028731+0000","last_became_peered":"2024-09-17T07:02:14.028731+0000","last_unstale":"2024-09-17T07:04:33.112786+0000","last_undegraded":"2024-09-17T07:04:33.112786+0000","last_fullsized":"2024-09-17T07:04:33.112786+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-09-17T07:03:56.068998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:56.068998+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:10.396846+0000","scrub_duration":0.0020277960000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"75'8","reported_seq":94,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T07:04:34.121677+0000","last_change":"2024-09-17T07:03:54.562849+0000","last_active":"2024-09-17T07:04:34.121677+0000","last_peered":"2024-09-17T07:04:34.121677+0000","last_clean":"2024-09-17T07:04:34.121677+0000","last_became_active":"2024-09-17T07:02:16.020560+0000","last_became_peered":"2024-09-17T07:02:16.020560+0000","last_unstale":"2024-09-17T07:04:34.121677+0000","last_undegraded":"2024-09-17T07:04:34.121677+0000","last_fullsized":"2024-09-17T07:04:34.121677+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:03:54.562729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:54.562729+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:04.111678+0000","scrub_duration":0.0018220739999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":51,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.119152+0000","last_change":"2024-09-17T07:03:58.948312+0000","last_active":"2024-09-17T07:04:33.119152+0000","last_peered":"2024-09-17T07:04:33.119152+0000","last_clean":"2024-09-17T07:04:33.119152+0000","last_became_active":"2024-09-17T07:02:18.021603+0000","last_became_peered":"2024-09-17T07:02:18.021603+0000","last_unstale":"2024-09-17T07:04:33.119152+0000","last_undegraded":"2024-09-17T07:04:33.119152+0000","last_fullsized":"2024-09-17T07:04:33.119152+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:58.948227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:03:58.948227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:19.197617+0000","scrub_duration":0.0018946010000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":69,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T07:04:34.679969+0000","last_change":"2024-09-17T07:04:34.679969+0000","last_active":"2024-09-17T07:04:34.679969+0000","last_peered":"2024-09-17T07:04:34.679969+0000","last_clean":"2024-09-17T07:04:34.679969+0000","last_became_active":"2024-09-17T07:02:12.009837+0000","last_became_peered":"2024-09-17T07:02:12.009837+0000","last_unstale":"2024-09-17T07:04:34.679969+0000","last_undegraded":"2024-09-17T07:04:34.679969+0000","last_fullsized":"2024-09-17T07:04:34.679969+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:34.679855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:04:34.679855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:48.664586+0000","scrub_duration":0.0053726590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"75'11","reported_seq":96,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.116038+0000","last_change":"2024-09-17T07:03:48.766828+0000","last_active":"2024-09-17T07:04:33.116038+0000","last_peered":"2024-09-17T07:04:33.116038+0000","last_clean":"2024-09-17T07:04:33.116038+0000","last_became_active":"2024-09-17T07:03:48.760390+0000","last_became_peered":"2024-09-17T07:03:48.760390+0000","last_unstale":"2024-09-17T07:04:33.116038+0000","last_undegraded":"2024-09-17T07:04:33.116038+0000","last_fullsized":"2024-09-17T07:04:33.116038+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-09-17T07:03:24.473712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:24.473712+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021033010000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":55,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.112715+0000","last_change":"2024-09-17T07:04:02.220014+0000","last_active":"2024-09-17T07:04:33.112715+0000","last_peered":"2024-09-17T07:04:33.112715+0000","last_clean":"2024-09-17T07:04:33.112715+0000","last_became_active":"2024-09-17T07:02:16.019701+0000","last_became_peered":"2024-09-17T07:02:16.019701+0000","last_unstale":"2024-09-17T07:04:33.112715+0000","last_undegraded":"2024-09-17T07:04:33.112715+0000","last_fullsized":"2024-09-17T07:04:33.112715+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:02.219927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:04:02.219927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:26.212811+0000","scrub_duration":0.0015145639999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":53,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.114779+0000","last_change":"2024-09-17T07:04:03.393754+0000","last_active":"2024-09-17T07:04:33.114779+0000","last_peered":"2024-09-17T07:04:33.114779+0000","last_clean":"2024-09-17T07:04:33.114779+0000","last_became_active":"2024-09-17T07:02:18.363752+0000","last_became_peered":"2024-09-17T07:02:18.363752+0000","last_unstale":"2024-09-17T07:04:33.114779+0000","last_undegraded":"2024-09-17T07:04:33.114779+0000","last_fullsized":"2024-09-17T07:04:33.114779+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:03.393645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:04:03.393645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:22.066050+0000","scrub_duration":0.001487555,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":66,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.115392+0000","last_change":"2024-09-17T07:03:48.759710+0000","last_active":"2024-09-17T07:04:33.115392+0000","last_peered":"2024-09-17T07:04:33.115392+0000","last_clean":"2024-09-17T07:04:33.115392+0000","last_became_active":"2024-09-17T07:03:48.759384+0000","last_became_peered":"2024-09-17T07:03:48.759384+0000","last_unstale":"2024-09-17T07:04:33.115392+0000","last_undegraded":"2024-09-17T07:04:33.115392+0000","last_fullsized":"2024-09-17T07:04:33.115392+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:46.447902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:46.447902+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:59.151253+0000","scrub_duration":0.0011905419999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"75'15","reported_seq":57,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.121065+0000","last_change":"2024-09-17T07:04:15.114695+0000","last_active":"2024-09-17T07:04:33.121065+0000","last_peered":"2024-09-17T07:04:33.121065+0000","last_clean":"2024-09-17T07:04:33.121065+0000","last_became_active":"2024-09-17T07:04:15.114286+0000","last_became_peered":"2024-09-17T07:04:15.114286+0000","last_unstale":"2024-09-17T07:04:33.121065+0000","last_undegraded":"2024-09-17T07:04:33.121065+0000","last_fullsized":"2024-09-17T07:04:33.121065+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-09-17T07:03:16.605163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:16.605163+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":53,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.119418+0000","last_change":"2024-09-17T07:03:47.013173+0000","last_active":"2024-09-17T07:04:33.119418+0000","last_peered":"2024-09-17T07:04:33.119418+0000","last_clean":"2024-09-17T07:04:33.119418+0000","last_became_active":"2024-09-17T07:02:16.015903+0000","last_became_peered":"2024-09-17T07:02:16.015903+0000","last_unstale":"2024-09-17T07:04:33.119418+0000","last_undegraded":"2024-09-17T07:04:33.119418+0000","last_fullsized":"2024-09-17T07:04:33.119418+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:47.013069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:47.013069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:05.642647+0000","scrub_duration":0.0012432350000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":52,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.117792+0000","last_change":"2024-09-17T07:04:10.606687+0000","last_active":"2024-09-17T07:04:33.117792+0000","last_peered":"2024-09-17T07:04:33.117792+0000","last_clean":"2024-09-17T07:04:33.117792+0000","last_became_active":"2024-09-17T07:02:18.366466+0000","last_became_peered":"2024-09-17T07:02:18.366466+0000","last_unstale":"2024-09-17T07:04:33.117792+0000","last_undegraded":"2024-09-17T07:04:33.117792+0000","last_fullsized":"2024-09-17T07:04:33.117792+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:10.606552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:04:10.606552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:31.857822+0000","scrub_duration":0.00165456,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":57,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.117985+0000","last_change":"2024-09-17T07:03:44.446996+0000","last_active":"2024-09-17T07:04:33.117985+0000","last_peered":"2024-09-17T07:04:33.117985+0000","last_clean":"2024-09-17T07:04:33.117985+0000","last_became_active":"2024-09-17T07:02:12.005493+0000","last_became_peered":"2024-09-17T07:02:12.005493+0000","last_unstale":"2024-09-17T07:04:33.117985+0000","last_undegraded":"2024-09-17T07:04:33.117985+0000","last_fullsized":"2024-09-17T07:04:33.117985+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:44.446914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:44.446914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:06.273158+0000","scrub_duration":0.0029497619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"86'17","reported_seq":83,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.117471+0000","last_change":"2024-09-17T07:03:18.241178+0000","last_active":"2024-09-17T07:04:33.117471+0000","last_peered":"2024-09-17T07:04:33.117471+0000","last_clean":"2024-09-17T07:04:33.117471+0000","last_became_active":"2024-09-17T07:02:14.017737+0000","last_became_peered":"2024-09-17T07:02:14.017737+0000","last_unstale":"2024-09-17T07:04:33.117471+0000","last_undegraded":"2024-09-17T07:04:33.117471+0000","last_fullsized":"2024-09-17T07:04:33.117471+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'17","last_scrub_stamp":"2024-09-17T07:03:18.241067+0000","last_deep_scrub":"86'17","last_deep_scrub_stamp":"2024-09-17T07:03:18.241067+0000","last_clean_scrub_stamp":"2024-09-17T07:03:18.241067+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:35.855446+0000","scrub_duration":0.0047694219999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":54,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-17T07:04:33.113973+0000","last_change":"2024-09-17T07:03:26.978352+0000","last_active":"2024-09-17T07:04:33.113973+0000","last_peered":"2024-09-17T07:04:33.113973+0000","last_clean":"2024-09-17T07:04:33.113973+0000","last_became_active":"2024-09-17T07:02:16.017362+0000","last_became_peered":"2024-09-17T07:02:16.017362+0000","last_unstale":"2024-09-17T07:04:33.113973+0000","last_undegraded":"2024-09-17T07:04:33.113973+0000","last_fullsized":"2024-09-17T07:04:33.113973+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:26.978262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:26.978262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:38.363709+0000","scrub_duration":0.0016151990000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":1263534080,"num_objects":19280,"num_object_clones":0,"num_object_copies":57840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19280,"num_write_kb":1233920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3791585280,"data_stored":3791585280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19280,"ondisk_log_size":19280,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66,"ondisk_log_size":66,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":697,"num_read_kb":454,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":404,"ondisk_log_size":404,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":139,"num_read_kb":134,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1386,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":12,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":4158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037795,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":406896,"kb_used_data":371688,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":93338256,"statfs":{"total":95995035648,"available":95578374144,"internally_reserved":0,"allocated":380608512,"data_stored":380214193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":36043211},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:04:03 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.52400000000000002,"15min":0.52800000000000002},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.624,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.54300000000000004,"15min":0.52200000000000002},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":3.1419999999999999,"5min":3.1419999999999999,"15min":3.1419999999999999},"last":0.82399999999999995}]},{"osd":1,"last update":"Tue Sep 17 07:04:03 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.50700000000000001,"15min":0.50800000000000001},"min":{"1min":0.36199999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.91800000000000004},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.5}]},{"osd":2,"last update":"Tue Sep 17 07:04:03 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.48699999999999999,"15min":0.48099999999999998},"min":{"1min":0.38,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":1.196},{"interface":"front","average":{"1min":0.5,"5min":0.53300000000000003,"15min":0.53900000000000003},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.71899999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.75}]},{"osd":3,"last update":"Tue Sep 17 07:04:03 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.373,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.54500000000000004,"15min":0.54800000000000004},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.47199999999999998}]},{"osd":4,"last update":"Tue Sep 17 07:04:03 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.32600000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.436,"15min":0.43099999999999999},"min":{"1min":0.36599999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.61199999999999999}]},{"osd":5,"last update":"Tue Sep 17 07:04:03 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.436,"15min":0.434},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.68899999999999995,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.505,"5min":0.46899999999999997,"15min":0.46300000000000002},"min":{"1min":0.34300000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.72599999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.55100000000000005}]},{"osd":6,"last update":"Tue Sep 17 07:04:03 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.499,"15min":0.497},"min":{"1min":0.36199999999999999,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.49299999999999999,"15min":0.48899999999999999},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.34999999999999998}]}]},{"osd":6,"up_from":50,"seq":214748364838,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":399712,"kb_used_data":364504,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":93345440,"statfs":{"total":95995035648,"available":95585730560,"internally_reserved":0,"allocated":373252096,"data_stored":372873476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:03:43 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.49299999999999999,"15min":0.497},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.66500000000000004,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":1.0529999999999999},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.48899999999999999,"15min":0.48999999999999999},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.63100000000000001,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":1.1679999999999999}]},{"osd":1,"last update":"Tue Sep 17 07:03:43 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.48599999999999999,"15min":0.48799999999999999},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.61699999999999999,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.52300000000000002,"15min":0.53000000000000003},"min":{"1min":0.34999999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.55300000000000005,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.61299999999999999}]},{"osd":2,"last update":"Tue Sep 17 07:03:43 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.51500000000000001,"15min":0.51900000000000002},"min":{"1min":0.36299999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.68500000000000005,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51800000000000002,"15min":0.51900000000000002},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.64400000000000002,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.53500000000000003}]},{"osd":3,"last update":"Tue Sep 17 07:03:43 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.47699999999999998,"15min":0.47899999999999998},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.60499999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.53600000000000003,"15min":0.53800000000000003},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.67100000000000004,"5min":0.88,"15min":0.88},"last":1.0840000000000001}]},{"osd":4,"last update":"Tue Sep 17 07:03:43 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.41999999999999998,"15min":0.41999999999999998},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.52800000000000002,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.443,"5min":0.41399999999999998,"15min":0.40899999999999997},"min":{"1min":0.32600000000000001,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.59599999999999997,"5min":0.628,"15min":0.628},"last":0.34100000000000003}]},{"osd":5,"last update":"Tue Sep 17 07:03:43 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.42999999999999999,"15min":0.42399999999999999},"min":{"1min":0.32800000000000001,"5min":0.31,"15min":0.31},"max":{"1min":0.60199999999999998,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.432,"15min":0.42799999999999999},"min":{"1min":0.313,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.374}]},{"osd":7,"last update":"Tue Sep 17 07:04:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.49299999999999999,"15min":0.49399999999999999},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.63800000000000001,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.501,"5min":0.52100000000000002,"15min":0.52400000000000002},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.622,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.81399999999999995}]}]},{"osd":5,"up_from":42,"seq":180388626475,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":323360,"kb_used_data":292184,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93421792,"statfs":{"total":95995035648,"available":95663915008,"internally_reserved":0,"allocated":299196416,"data_stored":298818062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:04:20 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.499,"15min":0.51300000000000001},"min":{"1min":0.39800000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.68500000000000005,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.48699999999999999,"15min":0.502},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.77500000000000002,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.58499999999999996}]},{"osd":1,"last update":"Tue Sep 17 07:04:20 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.52200000000000002,"15min":0.53800000000000003},"min":{"1min":0.35999999999999999,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.79200000000000004,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.436,"5min":0.89300000000000002,"15min":1.095},"min":{"1min":0.29599999999999999,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.63,"5min":14.054,"15min":14.054},"last":1.2929999999999999}]},{"osd":2,"last update":"Tue Sep 17 07:04:20 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.53300000000000003,"15min":0.55500000000000005},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.67600000000000005,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":1.639},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.503,"15min":0.50700000000000001},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.623,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":1.698}]},{"osd":3,"last update":"Tue Sep 17 07:04:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.52300000000000002,"15min":0.54300000000000004},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.60199999999999998,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.52600000000000002,"15min":0.54200000000000004},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.75600000000000001,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.30499999999999999}]},{"osd":4,"last update":"Tue Sep 17 07:04:20 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.39300000000000002,"15min":0.39700000000000002},"min":{"1min":0.26300000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.51000000000000001,"5min":0.60099999999999998,"15min":0.60099999999999998},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.42799999999999999,"15min":0.432},"min":{"1min":0.28399999999999997,"5min":0.248,"15min":0.248},"max":{"1min":0.66700000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.32200000000000001}]},{"osd":6,"last update":"Tue Sep 17 07:03:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.42299999999999999,"15min":0.42499999999999999},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.375},{"interface":"front","average":{"1min":0.439,"5min":0.44400000000000001,"15min":0.44500000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.63800000000000001,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.27800000000000002}]},{"osd":7,"last update":"Tue Sep 17 07:04:04 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.48399999999999999,"15min":0.48199999999999998},"min":{"1min":0.35299999999999998,"5min":0.308,"15min":0.308},"max":{"1min":0.67200000000000004,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.46700000000000003,"15min":0.46500000000000002},"min":{"1min":0.32700000000000001,"5min":0.314,"15min":0.314},"max":{"1min":0.69599999999999995,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.41299999999999998}]}]},{"osd":4,"up_from":33,"seq":141733920817,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":407664,"kb_used_data":372456,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":93337488,"statfs":{"total":95995035648,"available":95577587712,"internally_reserved":0,"allocated":381394944,"data_stored":380999793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:03:54 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.54500000000000004,"15min":0.56899999999999995},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.61199999999999999,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.623},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.58599999999999997,"15min":0.60999999999999999},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.78500000000000003,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.26100000000000001}]},{"osd":1,"last update":"Tue Sep 17 07:03:54 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.54300000000000004,"15min":0.56699999999999995},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.67000000000000004,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.308},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.56599999999999995,"15min":0.58999999999999997},"min":{"1min":0.34999999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.69799999999999995,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.59299999999999997}]},{"osd":2,"last update":"Tue Sep 17 07:03:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.55700000000000005,"15min":0.57899999999999996},"min":{"1min":0.40699999999999997,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.67600000000000005,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.56499999999999995,"15min":0.58399999999999996},"min":{"1min":0.36699999999999999,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.76500000000000001,"5min":0.877,"15min":0.877},"last":1.2809999999999999}]},{"osd":3,"last update":"Tue Sep 17 07:03:54 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.57599999999999996,"15min":0.60199999999999998},"min":{"1min":0.34100000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.73299999999999998,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":1.373},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.55200000000000005,"15min":0.56200000000000006},"min":{"1min":0.38600000000000001,"5min":0.371,"15min":0.371},"max":{"1min":0.71799999999999997,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.505}]},{"osd":5,"last update":"Tue Sep 17 07:04:20 2024","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.39800000000000002,"15min":0.38600000000000001},"min":{"1min":0.248,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.63,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.42999999999999999,"15min":0.41199999999999998},"min":{"1min":0.27900000000000003,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.68100000000000005,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.29099999999999998}]},{"osd":6,"last update":"Tue Sep 17 07:03:38 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.39900000000000002,"15min":0.39500000000000002},"min":{"1min":0.313,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.41399999999999998,"15min":0.40699999999999997},"min":{"1min":0.314,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.44}]},{"osd":7,"last update":"Tue Sep 17 07:04:02 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.5,"15min":0.5},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.65100000000000002,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.47699999999999998,"15min":0.47899999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.628,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.33900000000000002}]}]},{"osd":3,"up_from":27,"seq":115964117043,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":708872,"kb_used_data":669568,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":93036280,"statfs":{"total":95995035648,"available":95269150720,"internally_reserved":0,"allocated":685637632,"data_stored":685218689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":40237512},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:03:39 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.36099999999999999,"15min":0.36299999999999999},"min":{"1min":0.247,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.48399999999999999,"5min":0.55100000000000005,"15min":0.55100000000000005},"last":2.6139999999999999},{"interface":"front","average":{"1min":0.371,"5min":0.379,"15min":0.378},"min":{"1min":0.26000000000000001,"5min":0.189,"15min":0.189},"max":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":2.351}]},{"osd":1,"last update":"Tue Sep 17 07:03:39 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.39700000000000002,"15min":0.40500000000000003},"min":{"1min":0.24199999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.52100000000000002,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.435},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.441,"15min":0.44800000000000001},"min":{"1min":0.29899999999999999,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.58499999999999996,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":1.6339999999999999}]},{"osd":2,"last update":"Tue Sep 17 07:03:39 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.41299999999999998,"15min":0.42099999999999999},"min":{"1min":0.30599999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.59499999999999997,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":1.569},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.41499999999999998,"15min":0.41899999999999998},"min":{"1min":0.29499999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":2.2559999999999998}]},{"osd":4,"last update":"Tue Sep 17 07:03:56 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55800000000000005,"15min":0.56299999999999994},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":2.2890000000000001},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.53300000000000003,"15min":0.53600000000000003},"min":{"1min":0.33400000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.68000000000000005}]},{"osd":5,"last update":"Tue Sep 17 07:04:22 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.58299999999999996,"15min":0.60099999999999998},"min":{"1min":0.34499999999999997,"5min":0.315,"15min":0.315},"max":{"1min":0.79700000000000004,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":1.5369999999999999},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.57199999999999995,"15min":0.58199999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.67300000000000004,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":2.3319999999999999}]},{"osd":6,"last update":"Tue Sep 17 07:03:39 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.56999999999999995,"15min":0.56599999999999995},"min":{"1min":0.40400000000000003,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.76700000000000002,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.97699999999999998},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58899999999999997,"15min":0.58999999999999997},"min":{"1min":0.42699999999999999,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.78600000000000003,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":1.6020000000000001}]},{"osd":7,"last update":"Tue Sep 17 07:04:02 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.58399999999999996,"15min":0.58699999999999997},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.77200000000000002,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":2.3119999999999998},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.55900000000000005,"15min":0.55800000000000005},"min":{"1min":0.38500000000000001,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.64700000000000002}]}]},{"osd":2,"up_from":18,"seq":77309411385,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":439464,"kb_used_data":404256,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":93305688,"statfs":{"total":95995035648,"available":95545024512,"internally_reserved":0,"allocated":413958144,"data_stored":413571718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:04:18 2024","interfaces":[{"interface":"back","average":{"1min":0.29299999999999998,"5min":0.34499999999999997,"15min":0.34300000000000003},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":0.42199999999999999,"5min":0.629,"15min":0.629},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.35999999999999999,"15min":0.35099999999999998},"min":{"1min":0.26100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.57399999999999995,"5min":0.625,"15min":0.625},"last":0.78200000000000003}]},{"osd":1,"last update":"Tue Sep 17 07:04:18 2024","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.35199999999999998,"15min":0.34999999999999998},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.52400000000000002,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.38200000000000001,"15min":0.373},"min":{"1min":0.25600000000000001,"5min":0.251,"15min":0.251},"max":{"1min":0.47599999999999998,"5min":0.58399999999999996,"15min":0.58399999999999996},"last":0.39700000000000002}]},{"osd":3,"last update":"Tue Sep 17 07:03:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.435,"15min":0.437},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.57399999999999995,"5min":0.60499999999999998,"15min":0.60499999999999998},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.42599999999999999,"15min":0.42799999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.58999999999999997,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.92700000000000005}]},{"osd":4,"last update":"Tue Sep 17 07:03:54 2024","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.59899999999999998,"15min":0.57699999999999996},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":3.9100000000000001,"5min":3.9100000000000001,"15min":3.9100000000000001},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.57699999999999996,"15min":0.58399999999999996},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.84499999999999997,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.42999999999999999}]},{"osd":5,"last update":"Tue Sep 17 07:04:18 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.58399999999999996,"15min":0.59199999999999997},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.77300000000000002,"5min":1.101,"15min":1.101},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.56499999999999995,"15min":0.58499999999999996},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.69799999999999995,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.48799999999999999}]},{"osd":6,"last update":"Tue Sep 17 07:03:44 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.59499999999999997,"15min":0.60099999999999998},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.81799999999999995}]},{"osd":7,"last update":"Tue Sep 17 07:04:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.56000000000000005,"15min":0.56699999999999995},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.82299999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.72799999999999998,"15min":0.76400000000000001},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.70699999999999996,"5min":3.9289999999999998,"15min":3.9289999999999998},"last":0.51500000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574910,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":484624,"kb_used_data":449416,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":93260528,"statfs":{"total":95995035648,"available":95498780672,"internally_reserved":0,"allocated":460201984,"data_stored":459775990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:03:46 2024","interfaces":[{"interface":"back","average":{"1min":0.315,"5min":0.34399999999999997,"15min":0.33400000000000002},"min":{"1min":0.21099999999999999,"5min":0.193,"15min":0.193},"max":{"1min":0.46100000000000002,"5min":0.56000000000000005,"15min":0.56000000000000005},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.33800000000000002,"5min":0.36899999999999999,"15min":0.36199999999999999},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.47399999999999998,"5min":0.622,"15min":0.622},"last":1.9990000000000001}]},{"osd":2,"last update":"Tue Sep 17 07:04:18 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.38100000000000001,"15min":0.39600000000000002},"min":{"1min":0.27600000000000002,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.501,"5min":0.53800000000000003,"15min":0.53800000000000003},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.371,"5min":0.39800000000000002,"15min":0.40899999999999997},"min":{"1min":0.29099999999999998,"5min":0.254,"15min":0.254},"max":{"1min":0.59699999999999998,"5min":0.625,"15min":0.625},"last":1.9750000000000001}]},{"osd":3,"last update":"Tue Sep 17 07:03:37 2024","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.441,"15min":0.45800000000000002},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"last":1.3029999999999999},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.41599999999999998,"15min":0.438},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.54900000000000004,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":1.0429999999999999}]},{"osd":4,"last update":"Tue Sep 17 07:03:57 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57699999999999996,"15min":0.57399999999999995},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":1.923},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57199999999999995,"15min":0.56899999999999995},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":1.0760000000000001}]},{"osd":5,"last update":"Tue Sep 17 07:04:20 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.61899999999999999,"15min":0.58699999999999997},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.91800000000000004,"5min":6.0060000000000002,"15min":6.0060000000000002},"last":1.8839999999999999},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.55400000000000005,"15min":0.56200000000000006},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.75700000000000001,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":1.1419999999999999}]},{"osd":6,"last update":"Tue Sep 17 07:03:41 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":1.167},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.58699999999999997,"15min":0.59299999999999997},"min":{"1min":0.38500000000000001,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.77200000000000002,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":1.95}]},{"osd":7,"last update":"Tue Sep 17 07:04:03 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.55800000000000005,"15min":0.56200000000000006},"min":{"1min":0.39300000000000002,"5min":0.374,"15min":0.374},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":1.103},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.56399999999999995,"15min":0.56799999999999995},"min":{"1min":0.39900000000000002,"5min":0.378,"15min":0.378},"max":{"1min":0.72099999999999997,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.622}]}]},{"osd":0,"up_from":8,"seq":34359738433,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":589880,"kb_used_data":554672,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":93155272,"statfs":{"total":95995035648,"available":95390998528,"internally_reserved":0,"allocated":567984128,"data_stored":567582062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 17 07:03:47 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.35099999999999998,"15min":0.33000000000000002},"min":{"1min":0.25600000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.52600000000000002,"5min":0.58499999999999996,"15min":0.58499999999999996},"last":0.22},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.371,"15min":0.34499999999999997},"min":{"1min":0.26400000000000001,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.45000000000000001}]},{"osd":2,"last update":"Tue Sep 17 07:04:17 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.38200000000000001,"15min":0.376},"min":{"1min":0.26400000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.626,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.36799999999999999,"15min":0.36199999999999999},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.19900000000000001}]},{"osd":3,"last update":"Tue Sep 17 07:03:43 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.42399999999999999,"15min":0.436},"min":{"1min":0.30199999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.55900000000000005,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":1.345},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.42199999999999999,"15min":0.433},"min":{"1min":0.313,"5min":0.248,"15min":0.248},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.47999999999999998}]},{"osd":4,"last update":"Tue Sep 17 07:03:57 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.67800000000000005,"15min":0.59999999999999998},"min":{"1min":0.42199999999999999,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.81599999999999995,"5min":12.573,"15min":12.573},"last":0.441},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.55200000000000005,"15min":0.54800000000000004},"min":{"1min":0.39600000000000002,"5min":0.37,"15min":0.37},"max":{"1min":0.77000000000000002,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.51400000000000001}]},{"osd":5,"last update":"Tue Sep 17 07:04:23 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.376,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.54300000000000004,"15min":0.54000000000000004},"min":{"1min":0.38500000000000001,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.71199999999999997,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.53900000000000003}]},{"osd":6,"last update":"Tue Sep 17 07:03:43 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.55000000000000004,"15min":0.54700000000000004},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.55400000000000005,"15min":0.55000000000000004},"min":{"1min":0.40500000000000003,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.50600000000000001}]},{"osd":7,"last update":"Tue Sep 17 07:04:03 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":1.087,"15min":1.163},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":2.1539999999999999,"5min":12.548,"15min":12.548},"last":1.7410000000000001},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.55800000000000005,"15min":0.56000000000000005},"min":{"1min":0.40699999999999997,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.68899999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":627769344,"data_stored":627769344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":468189184,"data_stored":468189184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":473366528,"data_stored":473366528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":713752576,"data_stored":713752576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":398786560,"data_stored":398786560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":314966016,"data_stored":314966016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":401735680,"data_stored":401735680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":393019392,"data_stored":393019392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T07:04:41.937 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-17T07:04:41.937 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd pool set unique_pool_0 pg_num 26 2024-09-17T07:04:41.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:41.939 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:41.939 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:04:42.039 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:04:42.083 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:42.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:42.085 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:04:42.185 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:04:42.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:42.234 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:42.235 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:04:42.335 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:04:42.367 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:42.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:42.369 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:04:42.470 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:04:42.505 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:42.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:42.506 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:04:42.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:42 smithi135 ceph-mon[33238]: from='client.24725 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:04:42.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:42 smithi135 ceph-mon[33238]: 4.14 scrub starts 2024-09-17T07:04:42.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:42 smithi135 ceph-mon[33238]: 4.14 scrub ok 2024-09-17T07:04:42.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:42 smithi135 ceph-mon[33238]: pgmap v87: 148 pgs: 148 active+clean; 1.2 GiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 114 MiB/s wr, 1.83k op/s 2024-09-17T07:04:42.607 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:04:42.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:42 smithi100 ceph-mon[29092]: from='client.24725 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:04:42.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:42 smithi100 ceph-mon[29092]: 4.14 scrub starts 2024-09-17T07:04:42.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:42 smithi100 ceph-mon[29092]: 4.14 scrub ok 2024-09-17T07:04:42.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:42 smithi100 ceph-mon[29092]: pgmap v87: 148 pgs: 148 active+clean; 1.2 GiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 114 MiB/s wr, 1.83k op/s 2024-09-17T07:04:42.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:42 smithi100 ceph-mon[39462]: from='client.24725 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:04:42.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:42 smithi100 ceph-mon[39462]: 4.14 scrub starts 2024-09-17T07:04:42.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:42 smithi100 ceph-mon[39462]: 4.14 scrub ok 2024-09-17T07:04:42.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:42 smithi100 ceph-mon[39462]: pgmap v87: 148 pgs: 148 active+clean; 1.2 GiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 114 MiB/s wr, 1.83k op/s 2024-09-17T07:04:42.635 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 11 16 30173 30157 171.329 137.438 0.00508657 0.00583092 2024-09-17T07:04:42.640 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:42.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:42.641 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:04:42.742 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:04:42.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:42.770 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:42.770 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:04:42.871 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:04:42.900 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:42.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:42.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:04:43.002 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:04:43.032 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:43.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:43.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:04:43.134 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:04:43.159 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:43.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:43.160 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:04:43.261 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:04:43.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:43.302 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:43.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:04:43.325 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:43 smithi100 ceph-mon[29092]: 2.b scrub starts 2024-09-17T07:04:43.325 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:43 smithi100 ceph-mon[29092]: 2.b scrub ok 2024-09-17T07:04:43.325 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:43 smithi100 ceph-mon[39462]: 2.b scrub starts 2024-09-17T07:04:43.325 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:43 smithi100 ceph-mon[39462]: 2.b scrub ok 2024-09-17T07:04:43.403 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:04:43.444 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:43.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:43.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:04:43.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:43 smithi135 ceph-mon[33238]: 2.b scrub starts 2024-09-17T07:04:43.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:43 smithi135 ceph-mon[33238]: 2.b scrub ok 2024-09-17T07:04:43.546 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:04:43.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:43.586 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:43.586 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:04:43.636 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 12 16 32083 32067 166.997 119.375 0.00351548 0.00598286 2024-09-17T07:04:43.687 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:04:43.725 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:43.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:43.726 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:04:43.727 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:04:43.728 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1: {} 2024-09-17T07:04:43.728 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:04:43.728 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:04:43.728 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:04:43.728 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:04:43.728 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:04:43.728 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:04:43.728 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:04:43.728 INFO:teuthology.orchestra.run.smithi100.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:04:43.728 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:04:43.728 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:04:43.729 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:04:43.729 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:04:43.729 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:04:43.729 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:04:43.827 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:04:43.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:43.859 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:43.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:04:43.960 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:04:43.987 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:43.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:43.988 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:04:44.089 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:04:44.123 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:44.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:44.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:04:44.224 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:04:44.254 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:44.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:44.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:04:44.356 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:04:44.384 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:44.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:44.385 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:04:44.486 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:04:44.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:44.523 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:44.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:04:44.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:44 smithi135 ceph-mon[33238]: pgmap v88: 148 pgs: 148 active+clean; 1.7 GiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 148 MiB/s wr, 2.36k op/s 2024-09-17T07:04:44.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:44 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1918587593' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:04:44.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:44 smithi135 ceph-mon[33238]: 5.4 scrub starts 2024-09-17T07:04:44.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:44 smithi135 ceph-mon[33238]: 5.4 scrub ok 2024-09-17T07:04:44.624 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:04:44.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:44 smithi100 ceph-mon[29092]: pgmap v88: 148 pgs: 148 active+clean; 1.7 GiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 148 MiB/s wr, 2.36k op/s 2024-09-17T07:04:44.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:44 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1918587593' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:04:44.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:44 smithi100 ceph-mon[29092]: 5.4 scrub starts 2024-09-17T07:04:44.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:44 smithi100 ceph-mon[29092]: 5.4 scrub ok 2024-09-17T07:04:44.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:44 smithi100 ceph-mon[39462]: pgmap v88: 148 pgs: 148 active+clean; 1.7 GiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 148 MiB/s wr, 2.36k op/s 2024-09-17T07:04:44.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:44 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1918587593' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:04:44.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:44 smithi100 ceph-mon[39462]: 5.4 scrub starts 2024-09-17T07:04:44.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:44 smithi100 ceph-mon[39462]: 5.4 scrub ok 2024-09-17T07:04:44.635 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 13 16 33846 33830 162.627 110.188 0.00179388 0.00614045 2024-09-17T07:04:44.647 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:44.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:44.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:04:44.749 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:04:44.779 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:44.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:44.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:04:44.881 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:04:44.918 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:44.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:44.919 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:04:45.020 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:04:45.043 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:45.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:45.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:04:45.145 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:04:45.183 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:45.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:45.184 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:04:45.284 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:04:45.305 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:45.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:45.306 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:04:45.407 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:04:45.432 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:45.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:45.433 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:04:45.534 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:04:45.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:45 smithi135 ceph-mon[33238]: 4.c scrub starts 2024-09-17T07:04:45.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:45 smithi135 ceph-mon[33238]: 4.c scrub ok 2024-09-17T07:04:45.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:45 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:04:45.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:45 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:04:45.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:45 smithi135 ceph-mon[33238]: 2.8 scrub starts 2024-09-17T07:04:45.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:45 smithi135 ceph-mon[33238]: 2.8 scrub ok 2024-09-17T07:04:45.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:45 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:04:45.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:45 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:04:45.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:45 smithi135 ceph-mon[33238]: pgmap v89: 148 pgs: 148 active+clean; 1.9 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 938 B/s rd, 150 MiB/s wr, 2.41k op/s 2024-09-17T07:04:45.572 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:45.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:45.573 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:04:45.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:45 smithi100 ceph-mon[29092]: 4.c scrub starts 2024-09-17T07:04:45.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:45 smithi100 ceph-mon[29092]: 4.c scrub ok 2024-09-17T07:04:45.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:45 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:04:45.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:45 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:04:45.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:45 smithi100 ceph-mon[29092]: 2.8 scrub starts 2024-09-17T07:04:45.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:45 smithi100 ceph-mon[29092]: 2.8 scrub ok 2024-09-17T07:04:45.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:45 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:04:45.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:45 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:04:45.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:45 smithi100 ceph-mon[29092]: pgmap v89: 148 pgs: 148 active+clean; 1.9 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 938 B/s rd, 150 MiB/s wr, 2.41k op/s 2024-09-17T07:04:45.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:45 smithi100 ceph-mon[39462]: 4.c scrub starts 2024-09-17T07:04:45.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:45 smithi100 ceph-mon[39462]: 4.c scrub ok 2024-09-17T07:04:45.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:45 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:04:45.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:45 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:04:45.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:45 smithi100 ceph-mon[39462]: 2.8 scrub starts 2024-09-17T07:04:45.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:45 smithi100 ceph-mon[39462]: 2.8 scrub ok 2024-09-17T07:04:45.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:45 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:04:45.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:45 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:04:45.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:45 smithi100 ceph-mon[39462]: pgmap v89: 148 pgs: 148 active+clean; 1.9 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 938 B/s rd, 150 MiB/s wr, 2.41k op/s 2024-09-17T07:04:45.636 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 14 16 35287 35271 157.443 90.0625 0.00278365 0.00634542 2024-09-17T07:04:45.674 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:04:45.704 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:45.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:45.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:04:45.806 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:04:45.831 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:45.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:45.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:04:45.933 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:04:45.965 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:45.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:45.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:04:46.067 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:04:46.088 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:46.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:46.089 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:04:46.190 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:04:46.208 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:04:45 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:04:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:04:46.219 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:46.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:46.220 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:04:46.321 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:04:46.367 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:46.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:46.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:04:46.468 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:04:46.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:46.509 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:46.509 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:04:46.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:46 smithi135 ceph-mon[33238]: osdmap e106: 8 total, 8 up, 8 in 2024-09-17T07:04:46.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:46 smithi135 ceph-mon[33238]: 2.6 scrub starts 2024-09-17T07:04:46.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:46 smithi135 ceph-mon[33238]: 2.6 scrub ok 2024-09-17T07:04:46.610 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:04:46.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:46 smithi100 ceph-mon[39462]: osdmap e106: 8 total, 8 up, 8 in 2024-09-17T07:04:46.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:46 smithi100 ceph-mon[39462]: 2.6 scrub starts 2024-09-17T07:04:46.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:46 smithi100 ceph-mon[39462]: 2.6 scrub ok 2024-09-17T07:04:46.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:46 smithi100 ceph-mon[29092]: osdmap e106: 8 total, 8 up, 8 in 2024-09-17T07:04:46.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:46 smithi100 ceph-mon[29092]: 2.6 scrub starts 2024-09-17T07:04:46.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:46 smithi100 ceph-mon[29092]: 2.6 scrub ok 2024-09-17T07:04:46.630 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:46.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:46.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:04:46.636 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 15 16 36718 36702 152.909 89.4375 0.00643997 0.00651719 2024-09-17T07:04:46.732 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:04:46.769 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:46.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:46.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:04:46.871 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:04:46.892 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:46.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:46.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:04:46.994 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:04:47.023 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:47.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:47.025 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:04:47.126 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:04:47.155 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:47.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:47.156 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:04:47.257 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:04:47.294 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:47.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:47.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:04:47.396 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:04:47.431 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:47.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:47.432 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:04:47.532 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:04:47.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:47 smithi135 ceph-mon[33238]: 5.f scrub starts 2024-09-17T07:04:47.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:47 smithi135 ceph-mon[33238]: 5.f scrub ok 2024-09-17T07:04:47.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:47 smithi135 ceph-mon[33238]: pgmap v91: 148 pgs: 148 active+clean; 1.9 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 164 MiB/s wr, 2.62k op/s 2024-09-17T07:04:47.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:47 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:04:47.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:47.566 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:47.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:04:47.616 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:47 smithi100 ceph-mon[29092]: 5.f scrub starts 2024-09-17T07:04:47.616 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:47 smithi100 ceph-mon[29092]: 5.f scrub ok 2024-09-17T07:04:47.617 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:47 smithi100 ceph-mon[29092]: pgmap v91: 148 pgs: 148 active+clean; 1.9 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 164 MiB/s wr, 2.62k op/s 2024-09-17T07:04:47.617 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:47 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:04:47.617 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:47 smithi100 ceph-mon[39462]: 5.f scrub starts 2024-09-17T07:04:47.617 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:47 smithi100 ceph-mon[39462]: 5.f scrub ok 2024-09-17T07:04:47.617 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:47 smithi100 ceph-mon[39462]: pgmap v91: 148 pgs: 148 active+clean; 1.9 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 164 MiB/s wr, 2.62k op/s 2024-09-17T07:04:47.617 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:47 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:04:47.636 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 16 16 38495 38479 150.293 111.062 0.00412859 0.0066486 2024-09-17T07:04:47.666 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:04:47.706 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:47.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:47.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:04:47.808 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:04:47.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:47.839 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:47.839 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:04:47.940 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:04:47.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:47.963 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:47.964 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:04:48.064 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:04:48.101 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:48.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:48.102 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:04:48.203 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:04:48.230 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:48.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:48.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:04:48.332 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:04:48.363 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:48.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:48.364 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:04:48.465 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:04:48.501 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:48.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:48.502 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:04:48.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:48 smithi135 ceph-mon[33238]: 4.18 scrub starts 2024-09-17T07:04:48.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:48 smithi135 ceph-mon[33238]: 4.18 scrub ok 2024-09-17T07:04:48.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:48 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:04:48.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:48 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:04:48.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:48 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:04:48.544 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:04:48 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:04:48.603 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:04:48.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:48 smithi100 ceph-mon[39462]: 4.18 scrub starts 2024-09-17T07:04:48.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:48 smithi100 ceph-mon[39462]: 4.18 scrub ok 2024-09-17T07:04:48.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:48 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:04:48.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:48 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:04:48.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:48 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:04:48.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:48 smithi100 ceph-mon[29092]: 4.18 scrub starts 2024-09-17T07:04:48.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:48 smithi100 ceph-mon[29092]: 4.18 scrub ok 2024-09-17T07:04:48.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:48 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:04:48.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:48 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:04:48.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:48 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:04:48.636 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 17 16 40111 40095 147.393 101 0.00352838 0.00677869 2024-09-17T07:04:48.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:48.640 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:48.640 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:04:48.741 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:04:48.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:48.769 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:48.770 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:04:48.871 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:04:48.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:48.909 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:48.910 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:04:49.010 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:04:49.040 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:49.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:49.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:04:49.142 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:04:49.180 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:49.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:49.181 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:04:49.282 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:04:49.307 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:49 smithi100 ceph-mon[29092]: osdmap e107: 8 total, 8 up, 8 in 2024-09-17T07:04:49.308 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:49 smithi100 ceph-mon[29092]: 5.3 scrub starts 2024-09-17T07:04:49.308 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:49 smithi100 ceph-mon[29092]: 5.3 scrub ok 2024-09-17T07:04:49.308 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:49 smithi100 ceph-mon[29092]: pgmap v93: 148 pgs: 148 active+clean; 2.4 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 153 MiB/s wr, 2.46k op/s 2024-09-17T07:04:49.308 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:49 smithi100 ceph-mon[39462]: osdmap e107: 8 total, 8 up, 8 in 2024-09-17T07:04:49.308 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:49 smithi100 ceph-mon[39462]: 5.3 scrub starts 2024-09-17T07:04:49.308 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:49 smithi100 ceph-mon[39462]: 5.3 scrub ok 2024-09-17T07:04:49.308 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:49 smithi100 ceph-mon[39462]: pgmap v93: 148 pgs: 148 active+clean; 2.4 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 153 MiB/s wr, 2.46k op/s 2024-09-17T07:04:49.310 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:49.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:49.311 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:04:49.411 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:04:49.440 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:49.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:49.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:04:49.542 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:04:49.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:49 smithi135 ceph-mon[33238]: osdmap e107: 8 total, 8 up, 8 in 2024-09-17T07:04:49.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:49 smithi135 ceph-mon[33238]: 5.3 scrub starts 2024-09-17T07:04:49.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:49 smithi135 ceph-mon[33238]: 5.3 scrub ok 2024-09-17T07:04:49.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:49 smithi135 ceph-mon[33238]: pgmap v93: 148 pgs: 148 active+clean; 2.4 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 153 MiB/s wr, 2.46k op/s 2024-09-17T07:04:49.584 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:49.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:49.585 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:04:49.636 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 18 16 41921 41905 145.489 113.125 0.0101941 0.00686793 2024-09-17T07:04:49.686 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:04:49.717 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:49.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:49.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:04:49.819 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:04:49.846 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:49.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:49.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:04:49.947 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:04:49.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:49.980 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:49.980 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:04:50.052 INFO:teuthology.orchestra.run.smithi100.stderr:nodeep-scrub is unset 2024-09-17T07:04:50.081 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:04:50.106 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:50.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:50.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:04:50.208 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:04:50.239 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:50.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:50.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:04:50.341 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:04:50.371 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:50.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:50.372 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:04:50.473 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:04:50.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:50.514 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:50.515 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:04:50.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:50 smithi135 ceph-mon[33238]: 2.18 scrub starts 2024-09-17T07:04:50.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:50 smithi135 ceph-mon[33238]: 2.18 scrub ok 2024-09-17T07:04:50.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:50 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/3495445590' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:04:50.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:50 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:04:50.545 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:50 smithi135 ceph-mon[33238]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T07:04:50.545 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:50 smithi135 ceph-mon[33238]: Cluster is now healthy 2024-09-17T07:04:50.545 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:50 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T07:04:50.545 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:50 smithi135 ceph-mon[33238]: osdmap e108: 8 total, 8 up, 8 in 2024-09-17T07:04:50.615 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:04:50.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:50 smithi100 ceph-mon[29092]: 2.18 scrub starts 2024-09-17T07:04:50.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:50 smithi100 ceph-mon[29092]: 2.18 scrub ok 2024-09-17T07:04:50.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:50 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/3495445590' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:04:50.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:50 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:04:50.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:50 smithi100 ceph-mon[29092]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T07:04:50.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:50 smithi100 ceph-mon[29092]: Cluster is now healthy 2024-09-17T07:04:50.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:50 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T07:04:50.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:50 smithi100 ceph-mon[29092]: osdmap e108: 8 total, 8 up, 8 in 2024-09-17T07:04:50.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:50 smithi100 ceph-mon[39462]: 2.18 scrub starts 2024-09-17T07:04:50.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:50 smithi100 ceph-mon[39462]: 2.18 scrub ok 2024-09-17T07:04:50.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:50 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/3495445590' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:04:50.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:50 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:04:50.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:50 smithi100 ceph-mon[39462]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T07:04:50.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:50 smithi100 ceph-mon[39462]: Cluster is now healthy 2024-09-17T07:04:50.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:50 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T07:04:50.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:50 smithi100 ceph-mon[39462]: osdmap e108: 8 total, 8 up, 8 in 2024-09-17T07:04:50.637 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 19 16 43775 43759 143.93 115.875 0.00476021 0.00694099 2024-09-17T07:04:50.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:50.655 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:50.656 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:04:50.757 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:04:50.791 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:50.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:50.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:04:50.893 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:04:50.929 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:50.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:50.930 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:04:51.031 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:04:51.056 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:51.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:51.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:04:51.157 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:04:51.191 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:51.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:51.192 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:04:51.293 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:04:51.325 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:51.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:51.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:04:51.427 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:04:51.456 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:51.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:51.458 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:04:51.501 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:04:51.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:51 smithi135 ceph-mon[33238]: 4.6 scrub starts 2024-09-17T07:04:51.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:51 smithi135 ceph-mon[33238]: 4.6 scrub ok 2024-09-17T07:04:51.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:51 smithi135 ceph-mon[33238]: pgmap v95: 148 pgs: 148 active+clean; 2.4 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 84 MiB/s wr, 1.34k op/s 2024-09-17T07:04:51.559 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:04:51.584 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:51.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:51.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:04:51.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:51 smithi100 ceph-mon[29092]: 4.6 scrub starts 2024-09-17T07:04:51.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:51 smithi100 ceph-mon[29092]: 4.6 scrub ok 2024-09-17T07:04:51.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:51 smithi100 ceph-mon[29092]: pgmap v95: 148 pgs: 148 active+clean; 2.4 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 84 MiB/s wr, 1.34k op/s 2024-09-17T07:04:51.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:51 smithi100 ceph-mon[39462]: 4.6 scrub starts 2024-09-17T07:04:51.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:51 smithi100 ceph-mon[39462]: 4.6 scrub ok 2024-09-17T07:04:51.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:51 smithi100 ceph-mon[39462]: pgmap v95: 148 pgs: 148 active+clean; 2.4 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 84 MiB/s wr, 1.34k op/s 2024-09-17T07:04:51.636 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:2024-09-17T07:04:51.637140+0000 min lat: 0.00128777 max lat: 0.16769 avg lat: 0.00701225 2024-09-17T07:04:51.636 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-17T07:04:51.636 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 20 16 45621 45605 142.5 115.375 0.00519524 0.00701225 2024-09-17T07:04:51.686 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:04:51.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:51.717 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:51.718 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:04:51.818 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:04:51.848 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:51.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:51.849 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:04:51.950 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:04:51.988 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:51.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:51.989 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:04:52.090 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:04:52.125 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:52.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:52.126 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:04:52.227 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:04:52.267 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:52.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:52.268 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:04:52.311 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:04:52.369 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:04:52.406 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-17T07:04:52.406 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-17T07:04:52.407 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:04:52.407 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.14805.0:46941 7.b 7:d1441a80:::benchmark_data_smithi100_78556_object46940:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-17T07:04:52.407 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:04:52.401002+0000", 2024-09-17T07:04:52.407 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.0026877770000000001, 2024-09-17T07:04:52.407 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0027208549999999999, 2024-09-17T07:04:52.407 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:04:52.407 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "waiting for sub ops", 2024-09-17T07:04:52.407 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:04:52.407 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.14805", 2024-09-17T07:04:52.407 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:04:52.407 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 46941 2024-09-17T07:04:52.407 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:04:52.408 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:04:52.408 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:04:52.408 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:04:52.408 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:52.401002+0000", 2024-09-17T07:04:52.408 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:04:52.408 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:04:52.408 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:04:52.408 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:04:52.408 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:52.401002+0000", 2024-09-17T07:04:52.408 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:04:52.408 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:04:52.408 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:04:52.409 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:04:52.409 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:52.401000+0000", 2024-09-17T07:04:52.409 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999981 2024-09-17T07:04:52.409 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:04:52.409 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:04:52.409 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:04:52.409 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:52.401028+0000", 2024-09-17T07:04:52.409 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7727000000000001e-05 2024-09-17T07:04:52.409 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:04:52.409 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:04:52.409 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:04:52.409 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:52.401045+0000", 2024-09-17T07:04:52.409 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7762e-05 2024-09-17T07:04:52.410 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:04:52.410 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:04:52.410 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:04:52.410 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:52.401054+0000", 2024-09-17T07:04:52.410 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.7450000000000001e-06 2024-09-17T07:04:52.410 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:04:52.410 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:04:52.410 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:04:52.410 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:52.401133+0000", 2024-09-17T07:04:52.410 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9210999999999998e-05 2024-09-17T07:04:52.410 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:04:52.411 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:04:52.411 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:04:52.411 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:52.401263+0000", 2024-09-17T07:04:52.411 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012935999999999999 2024-09-17T07:04:52.411 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:04:52.411 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:04:52.411 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 5,6", 2024-09-17T07:04:52.411 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:52.401390+0000", 2024-09-17T07:04:52.411 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012709599999999999 2024-09-17T07:04:52.411 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:04:52.411 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:04:52.411 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:04:52.412 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:04:52.412 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:04:52.412 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.14805.0:46945 7.b 7:deea5eaa:::benchmark_data_smithi100_78556_object46944:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-17T07:04:52.412 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:04:52.402394+0000", 2024-09-17T07:04:52.412 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.001295619, 2024-09-17T07:04:52.412 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0014124750000000001, 2024-09-17T07:04:52.412 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:04:52.412 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-17T07:04:52.412 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:04:52.412 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.14805", 2024-09-17T07:04:52.412 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:04:52.412 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 46945 2024-09-17T07:04:52.413 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:04:52.413 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:04:52.413 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:04:52.413 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:04:52.413 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:52.402394+0000", 2024-09-17T07:04:52.413 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:04:52.413 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:04:52.413 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:04:52.413 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:04:52.413 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:52.402394+0000", 2024-09-17T07:04:52.413 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:04:52.413 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:04:52.413 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:04:52.413 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:04:52.414 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:52.402392+0000", 2024-09-17T07:04:52.414 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999986 2024-09-17T07:04:52.414 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:04:52.414 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:04:52.414 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:04:52.414 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:52.402417+0000", 2024-09-17T07:04:52.414 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4652e-05 2024-09-17T07:04:52.414 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:04:52.414 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:04:52.414 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:04:52.414 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:52.402433+0000", 2024-09-17T07:04:52.414 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5529e-05 2024-09-17T07:04:52.415 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:04:52.415 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:04:52.415 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:04:52.415 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:52.402441+0000", 2024-09-17T07:04:52.415 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8930000000000005e-06 2024-09-17T07:04:52.415 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:04:52.415 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:04:52.415 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:04:52.415 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:52.403708+0000", 2024-09-17T07:04:52.415 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0012674310000000001 2024-09-17T07:04:52.415 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:04:52.415 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:04:52.415 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:04:52.415 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:04:52.416 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:04:52.416 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.14805.0:46939 7.9 e108/99 7:9b32a744:::benchmark_data_smithi100_78556_object46938:head v 108'2842, mlcod=108'2840)", 2024-09-17T07:04:52.416 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:04:52.402813+0000", 2024-09-17T07:04:52.416 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.00087610300000000004, 2024-09-17T07:04:52.416 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001053048, 2024-09-17T07:04:52.416 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:04:52.416 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-17T07:04:52.416 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:04:52.416 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:04:52.416 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:04:52.416 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:52.402813+0000", 2024-09-17T07:04:52.416 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:04:52.416 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:04:52.417 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:04:52.417 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:04:52.417 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:52.402813+0000", 2024-09-17T07:04:52.417 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:04:52.417 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:04:52.417 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:04:52.417 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:04:52.417 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:52.402813+0000", 2024-09-17T07:04:52.417 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-17T07:04:52.417 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:04:52.417 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:04:52.417 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:04:52.417 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:52.402835+0000", 2024-09-17T07:04:52.418 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2586e-05 2024-09-17T07:04:52.418 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:04:52.418 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:04:52.418 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:04:52.418 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:52.402847+0000", 2024-09-17T07:04:52.418 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2174999999999999e-05 2024-09-17T07:04:52.418 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:04:52.418 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:04:52.418 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:04:52.418 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:52.402854+0000", 2024-09-17T07:04:52.418 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8549999999999996e-06 2024-09-17T07:04:52.418 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:04:52.418 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:04:52.418 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:04:52.419 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:52.402917+0000", 2024-09-17T07:04:52.419 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3093000000000001e-05 2024-09-17T07:04:52.419 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:04:52.419 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:04:52.419 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:04:52.419 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:52.402937+0000", 2024-09-17T07:04:52.419 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.9854e-05 2024-09-17T07:04:52.419 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:04:52.419 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:04:52.420 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:04:52.420 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:04:52.420 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:04:52.420 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop_reply(client.14805.0:46941 7.b e108/99)", 2024-09-17T07:04:52.420 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:04:52.403147+0000", 2024-09-17T07:04:52.420 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.00054213000000000004, 2024-09-17T07:04:52.420 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00076881499999999995, 2024-09-17T07:04:52.420 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:04:52.420 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "queued for pg", 2024-09-17T07:04:52.420 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:04:52.420 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:04:52.420 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:04:52.420 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:52.403147+0000", 2024-09-17T07:04:52.421 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:04:52.421 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:04:52.421 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:04:52.421 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:04:52.421 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:52.403147+0000", 2024-09-17T07:04:52.421 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:04:52.421 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:04:52.421 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:04:52.421 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:04:52.421 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:52.403147+0000", 2024-09-17T07:04:52.421 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-17T07:04:52.421 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:04:52.421 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:04:52.422 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:04:52.422 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:52.403152+0000", 2024-09-17T07:04:52.422 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6169999999999998e-06 2024-09-17T07:04:52.422 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:04:52.422 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:04:52.422 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:04:52.422 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:52.403164+0000", 2024-09-17T07:04:52.422 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1633e-05 2024-09-17T07:04:52.422 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:04:52.422 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:04:52.422 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:04:52.423 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:52.403170+0000", 2024-09-17T07:04:52.423 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9839999999999998e-06 2024-09-17T07:04:52.423 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:04:52.423 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:04:52.423 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:04:52.423 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:04:52.423 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-17T07:04:52.423 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 4 2024-09-17T07:04:52.423 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-17T07:04:52.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:52.425 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:52.425 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:04:52.526 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:04:52.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:52 smithi135 ceph-mon[33238]: 5.b scrub starts 2024-09-17T07:04:52.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:52 smithi135 ceph-mon[33238]: 5.b scrub ok 2024-09-17T07:04:52.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:52 smithi135 ceph-mon[33238]: 2.13 scrub starts 2024-09-17T07:04:52.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:52 smithi135 ceph-mon[33238]: 2.13 scrub ok 2024-09-17T07:04:52.553 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:52.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:52.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:04:52.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:52 smithi100 ceph-mon[29092]: 5.b scrub starts 2024-09-17T07:04:52.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:52 smithi100 ceph-mon[29092]: 5.b scrub ok 2024-09-17T07:04:52.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:52 smithi100 ceph-mon[29092]: 2.13 scrub starts 2024-09-17T07:04:52.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:52 smithi100 ceph-mon[29092]: 2.13 scrub ok 2024-09-17T07:04:52.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:52 smithi100 ceph-mon[39462]: 5.b scrub starts 2024-09-17T07:04:52.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:52 smithi100 ceph-mon[39462]: 5.b scrub ok 2024-09-17T07:04:52.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:52 smithi100 ceph-mon[39462]: 2.13 scrub starts 2024-09-17T07:04:52.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:52 smithi100 ceph-mon[39462]: 2.13 scrub ok 2024-09-17T07:04:52.637 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 21 16 47321 47305 140.77 106.25 0.00957188 0.00709911 2024-09-17T07:04:52.655 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:04:52.700 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:52.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:52.702 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:04:52.802 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:04:52.839 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:52.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:52.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:04:52.941 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:04:52.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:52.980 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:52.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:04:52.993 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-17T07:04:53.082 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:04:53.124 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:53.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:53.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:04:53.226 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:04:53.254 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:53.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:53.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:04:53.356 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:04:53.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:53 smithi100 ceph-mon[29092]: pgmap v96: 148 pgs: 148 active+clean; 2.7 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 654 B/s rd, 100 MiB/s wr, 1.60k op/s 2024-09-17T07:04:53.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:53 smithi100 ceph-mon[39462]: pgmap v96: 148 pgs: 148 active+clean; 2.7 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 654 B/s rd, 100 MiB/s wr, 1.60k op/s 2024-09-17T07:04:53.386 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:53.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:53.387 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:04:53.488 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:04:53.532 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:53.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:53.533 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:04:53.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:53 smithi135 ceph-mon[33238]: pgmap v96: 148 pgs: 148 active+clean; 2.7 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 654 B/s rd, 100 MiB/s wr, 1.60k op/s 2024-09-17T07:04:53.634 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:04:53.637 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 22 16 49011 48995 139.172 105.625 0.00826909 0.00717937 2024-09-17T07:04:53.660 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:53.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:53.661 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:04:53.762 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:04:53.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:53.808 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:53.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:04:53.909 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:04:53.935 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:53.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:53.936 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:04:54.037 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:04:54.074 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:54.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:54.075 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:04:54.120 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-17T07:04:54.148 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:04:54.176 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:04:54.229 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:54.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:54.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:04:54.330 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:04:54.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:54 smithi100 ceph-mon[29092]: 4.11 scrub starts 2024-09-17T07:04:54.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:54 smithi100 ceph-mon[29092]: 4.11 scrub ok 2024-09-17T07:04:54.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:54 smithi100 ceph-mon[29092]: 2.10 scrub starts 2024-09-17T07:04:54.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:54 smithi100 ceph-mon[29092]: 2.10 scrub ok 2024-09-17T07:04:54.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:54 smithi100 ceph-mon[29092]: 6.5 scrub starts 2024-09-17T07:04:54.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:54 smithi100 ceph-mon[29092]: 6.5 scrub ok 2024-09-17T07:04:54.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:54 smithi100 ceph-mon[39462]: 4.11 scrub starts 2024-09-17T07:04:54.379 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:54 smithi100 ceph-mon[39462]: 4.11 scrub ok 2024-09-17T07:04:54.379 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:54 smithi100 ceph-mon[39462]: 2.10 scrub starts 2024-09-17T07:04:54.379 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:54 smithi100 ceph-mon[39462]: 2.10 scrub ok 2024-09-17T07:04:54.379 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:54 smithi100 ceph-mon[39462]: 6.5 scrub starts 2024-09-17T07:04:54.379 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:54 smithi100 ceph-mon[39462]: 6.5 scrub ok 2024-09-17T07:04:54.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:54.380 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:54.380 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:04:54.481 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:04:54.511 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:54.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:54.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:04:54.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:54 smithi135 ceph-mon[33238]: 4.11 scrub starts 2024-09-17T07:04:54.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:54 smithi135 ceph-mon[33238]: 4.11 scrub ok 2024-09-17T07:04:54.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:54 smithi135 ceph-mon[33238]: 2.10 scrub starts 2024-09-17T07:04:54.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:54 smithi135 ceph-mon[33238]: 2.10 scrub ok 2024-09-17T07:04:54.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:54 smithi135 ceph-mon[33238]: 6.5 scrub starts 2024-09-17T07:04:54.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:54 smithi135 ceph-mon[33238]: 6.5 scrub ok 2024-09-17T07:04:54.613 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:04:54.638 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 23 16 50613 50597 137.474 100.125 0.013016 0.00726864 2024-09-17T07:04:54.641 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:54.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:54.642 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:04:54.743 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:04:54.764 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:54.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:54.766 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:04:54.867 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:04:54.908 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:54.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:54.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:04:55.010 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:04:55.037 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:55.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:55.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:04:55.139 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:04:55.166 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:55.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:55.168 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:04:55.268 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:04:55.311 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:55.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:55.312 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:04:55.413 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:04:55.430 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:04:55.441 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:55.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:55.442 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:04:55.502 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:04:55.543 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:04:55.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:55 smithi135 ceph-mon[33238]: 2.1 scrub starts 2024-09-17T07:04:55.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:55 smithi135 ceph-mon[33238]: 2.1 scrub ok 2024-09-17T07:04:55.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:55 smithi135 ceph-mon[33238]: 6.3 scrub starts 2024-09-17T07:04:55.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:55 smithi135 ceph-mon[33238]: 6.3 scrub ok 2024-09-17T07:04:55.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:55 smithi135 ceph-mon[33238]: 2.4 scrub starts 2024-09-17T07:04:55.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:55 smithi135 ceph-mon[33238]: 2.4 scrub ok 2024-09-17T07:04:55.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:55 smithi135 ceph-mon[33238]: pgmap v97: 148 pgs: 148 active+clean; 2.9 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 133 MiB/s wr, 2.13k op/s 2024-09-17T07:04:55.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:55 smithi135 ceph-mon[33238]: osdmap e109: 8 total, 8 up, 8 in 2024-09-17T07:04:55.568 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:55 smithi100 ceph-mon[29092]: 2.1 scrub starts 2024-09-17T07:04:55.568 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:55 smithi100 ceph-mon[29092]: 2.1 scrub ok 2024-09-17T07:04:55.568 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:55 smithi100 ceph-mon[29092]: 6.3 scrub starts 2024-09-17T07:04:55.569 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:55 smithi100 ceph-mon[29092]: 6.3 scrub ok 2024-09-17T07:04:55.569 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:55 smithi100 ceph-mon[29092]: 2.4 scrub starts 2024-09-17T07:04:55.569 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:55 smithi100 ceph-mon[29092]: 2.4 scrub ok 2024-09-17T07:04:55.569 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:55 smithi100 ceph-mon[29092]: pgmap v97: 148 pgs: 148 active+clean; 2.9 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 133 MiB/s wr, 2.13k op/s 2024-09-17T07:04:55.569 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:55 smithi100 ceph-mon[29092]: osdmap e109: 8 total, 8 up, 8 in 2024-09-17T07:04:55.569 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:55 smithi100 ceph-mon[39462]: 2.1 scrub starts 2024-09-17T07:04:55.569 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:55 smithi100 ceph-mon[39462]: 2.1 scrub ok 2024-09-17T07:04:55.569 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:55 smithi100 ceph-mon[39462]: 6.3 scrub starts 2024-09-17T07:04:55.569 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:55 smithi100 ceph-mon[39462]: 6.3 scrub ok 2024-09-17T07:04:55.569 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:55 smithi100 ceph-mon[39462]: 2.4 scrub starts 2024-09-17T07:04:55.570 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:55 smithi100 ceph-mon[39462]: 2.4 scrub ok 2024-09-17T07:04:55.570 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:55 smithi100 ceph-mon[39462]: pgmap v97: 148 pgs: 148 active+clean; 2.9 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 133 MiB/s wr, 2.13k op/s 2024-09-17T07:04:55.570 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:55 smithi100 ceph-mon[39462]: osdmap e109: 8 total, 8 up, 8 in 2024-09-17T07:04:55.589 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:55.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:55.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:04:55.637 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 24 16 52125 52109 135.682 94.5 0.0120574 0.00736538 2024-09-17T07:04:55.691 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:04:55.719 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:55.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:55.720 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:04:55.821 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:04:55.857 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:55.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:55.858 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:04:55.959 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:04:55.999 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:55.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:56.000 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:04:56.101 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:04:56.102 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:04:55 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:04:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:04:56.131 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:56.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:56.132 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:04:56.232 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:04:56.267 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:56.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:56.268 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:04:56.315 INFO:teuthology.orchestra.run.smithi100.stderr:set pool 7 pg_num to 26 2024-09-17T07:04:56.369 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:04:56.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:56 smithi100 ceph-mon[29092]: 2.c scrub starts 2024-09-17T07:04:56.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:56 smithi100 ceph-mon[29092]: 2.c scrub ok 2024-09-17T07:04:56.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:56 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1310153632' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]: dispatch 2024-09-17T07:04:56.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:56 smithi100 ceph-mon[29092]: 5.8 scrub starts 2024-09-17T07:04:56.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:56 smithi100 ceph-mon[29092]: 5.8 scrub ok 2024-09-17T07:04:56.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:56 smithi100 ceph-mon[29092]: 5.1a scrub starts 2024-09-17T07:04:56.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:56 smithi100 ceph-mon[29092]: 5.1a scrub ok 2024-09-17T07:04:56.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:56 smithi100 ceph-mon[39462]: 2.c scrub starts 2024-09-17T07:04:56.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:56 smithi100 ceph-mon[39462]: 2.c scrub ok 2024-09-17T07:04:56.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:56 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1310153632' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]: dispatch 2024-09-17T07:04:56.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:56 smithi100 ceph-mon[39462]: 5.8 scrub starts 2024-09-17T07:04:56.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:56 smithi100 ceph-mon[39462]: 5.8 scrub ok 2024-09-17T07:04:56.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:56 smithi100 ceph-mon[39462]: 5.1a scrub starts 2024-09-17T07:04:56.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:56 smithi100 ceph-mon[39462]: 5.1a scrub ok 2024-09-17T07:04:56.405 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:56.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:56.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:04:56.507 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:04:56.547 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:56.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:56.549 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:04:56.638 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 25 16 53744 53728 134.302 101.188 0.00971695 0.00744083 2024-09-17T07:04:56.649 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:04:56.680 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:56.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:56.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:04:56.782 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:04:56.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:56 smithi135 ceph-mon[33238]: 2.c scrub starts 2024-09-17T07:04:56.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:56 smithi135 ceph-mon[33238]: 2.c scrub ok 2024-09-17T07:04:56.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:56 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1310153632' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]: dispatch 2024-09-17T07:04:56.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:56 smithi135 ceph-mon[33238]: 5.8 scrub starts 2024-09-17T07:04:56.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:56 smithi135 ceph-mon[33238]: 5.8 scrub ok 2024-09-17T07:04:56.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:56 smithi135 ceph-mon[33238]: 5.1a scrub starts 2024-09-17T07:04:56.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:56 smithi135 ceph-mon[33238]: 5.1a scrub ok 2024-09-17T07:04:56.816 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:56.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:56.817 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:04:56.917 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:04:56.953 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:56.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:56.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:04:57.055 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:04:57.085 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:57.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:57.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:04:57.187 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:04:57.228 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:57.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:57.229 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:04:57.330 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:04:57.377 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:57.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:57.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:04:57.479 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:04:57.503 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:57.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:57.504 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:04:57.565 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:57 smithi100 ceph-mon[29092]: 2.17 scrub starts 2024-09-17T07:04:57.566 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:57 smithi100 ceph-mon[29092]: 2.17 scrub ok 2024-09-17T07:04:57.566 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:57 smithi100 ceph-mon[29092]: 4.12 scrub starts 2024-09-17T07:04:57.566 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:57 smithi100 ceph-mon[29092]: 4.12 scrub ok 2024-09-17T07:04:57.566 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:57 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1310153632' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]': finished 2024-09-17T07:04:57.566 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:57 smithi100 ceph-mon[29092]: osdmap e110: 8 total, 8 up, 8 in 2024-09-17T07:04:57.566 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:57 smithi100 ceph-mon[29092]: 2.1f scrub starts 2024-09-17T07:04:57.566 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:57 smithi100 ceph-mon[29092]: 2.1f scrub ok 2024-09-17T07:04:57.566 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:57 smithi100 ceph-mon[29092]: 2.2 scrub starts 2024-09-17T07:04:57.566 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:57 smithi100 ceph-mon[29092]: 2.2 scrub ok 2024-09-17T07:04:57.566 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:57 smithi100 ceph-mon[29092]: pgmap v100: 148 pgs: 148 active+clean; 2.9 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 733 B/s rd, 81 MiB/s wr, 1.30k op/s 2024-09-17T07:04:57.566 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:57 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-09-17T07:04:57.567 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:57 smithi100 ceph-mon[39462]: 2.17 scrub starts 2024-09-17T07:04:57.567 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:57 smithi100 ceph-mon[39462]: 2.17 scrub ok 2024-09-17T07:04:57.567 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:57 smithi100 ceph-mon[39462]: 4.12 scrub starts 2024-09-17T07:04:57.567 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:57 smithi100 ceph-mon[39462]: 4.12 scrub ok 2024-09-17T07:04:57.567 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:57 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1310153632' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]': finished 2024-09-17T07:04:57.567 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:57 smithi100 ceph-mon[39462]: osdmap e110: 8 total, 8 up, 8 in 2024-09-17T07:04:57.567 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:57 smithi100 ceph-mon[39462]: 2.1f scrub starts 2024-09-17T07:04:57.567 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:57 smithi100 ceph-mon[39462]: 2.1f scrub ok 2024-09-17T07:04:57.567 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:57 smithi100 ceph-mon[39462]: 2.2 scrub starts 2024-09-17T07:04:57.567 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:57 smithi100 ceph-mon[39462]: 2.2 scrub ok 2024-09-17T07:04:57.567 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:57 smithi100 ceph-mon[39462]: pgmap v100: 148 pgs: 148 active+clean; 2.9 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 733 B/s rd, 81 MiB/s wr, 1.30k op/s 2024-09-17T07:04:57.568 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:57 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-09-17T07:04:57.605 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:04:57.627 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:57.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:57.628 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:04:57.638 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 26 16 54910 54894 131.939 72.875 0.0133794 0.00748773 2024-09-17T07:04:57.729 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:04:57.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:57.751 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:57.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:04:57.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:57 smithi135 ceph-mon[33238]: 2.17 scrub starts 2024-09-17T07:04:57.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:57 smithi135 ceph-mon[33238]: 2.17 scrub ok 2024-09-17T07:04:57.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:57 smithi135 ceph-mon[33238]: 4.12 scrub starts 2024-09-17T07:04:57.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:57 smithi135 ceph-mon[33238]: 4.12 scrub ok 2024-09-17T07:04:57.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:57 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1310153632' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]': finished 2024-09-17T07:04:57.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:57 smithi135 ceph-mon[33238]: osdmap e110: 8 total, 8 up, 8 in 2024-09-17T07:04:57.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:57 smithi135 ceph-mon[33238]: 2.1f scrub starts 2024-09-17T07:04:57.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:57 smithi135 ceph-mon[33238]: 2.1f scrub ok 2024-09-17T07:04:57.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:57 smithi135 ceph-mon[33238]: 2.2 scrub starts 2024-09-17T07:04:57.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:57 smithi135 ceph-mon[33238]: 2.2 scrub ok 2024-09-17T07:04:57.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:57 smithi135 ceph-mon[33238]: pgmap v100: 148 pgs: 148 active+clean; 2.9 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 733 B/s rd, 81 MiB/s wr, 1.30k op/s 2024-09-17T07:04:57.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:57 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-09-17T07:04:57.852 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:04:57.874 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:57.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:57.875 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:04:57.976 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:04:57.997 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:57.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:57.998 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:04:58.099 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:04:58.121 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:58.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:58.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:04:58.223 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:04:58.245 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:58.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:58.246 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:04:58.347 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:04:58.377 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:58.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:58.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:04:58.479 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:04:58.509 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:58.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:58.510 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:04:58.543 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:04:58 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:04:58.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:58 smithi135 ceph-mon[33238]: 4.2 deep-scrub starts 2024-09-17T07:04:58.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:58 smithi135 ceph-mon[33238]: 4.2 deep-scrub ok 2024-09-17T07:04:58.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:58 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-17T07:04:58.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:58 smithi135 ceph-mon[33238]: osdmap e111: 8 total, 8 up, 8 in 2024-09-17T07:04:58.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:58 smithi135 ceph-mon[33238]: 4.8 deep-scrub starts 2024-09-17T07:04:58.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:58 smithi135 ceph-mon[33238]: 4.8 deep-scrub ok 2024-09-17T07:04:58.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:58 smithi135 ceph-mon[33238]: 5.11 scrub starts 2024-09-17T07:04:58.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:58 smithi135 ceph-mon[33238]: 5.11 scrub ok 2024-09-17T07:04:58.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:58 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:04:58.608 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:58 smithi100 ceph-mon[29092]: 4.2 deep-scrub starts 2024-09-17T07:04:58.608 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:58 smithi100 ceph-mon[29092]: 4.2 deep-scrub ok 2024-09-17T07:04:58.608 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:58 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-17T07:04:58.608 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:58 smithi100 ceph-mon[29092]: osdmap e111: 8 total, 8 up, 8 in 2024-09-17T07:04:58.608 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:58 smithi100 ceph-mon[29092]: 4.8 deep-scrub starts 2024-09-17T07:04:58.608 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:58 smithi100 ceph-mon[29092]: 4.8 deep-scrub ok 2024-09-17T07:04:58.608 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:58 smithi100 ceph-mon[29092]: 5.11 scrub starts 2024-09-17T07:04:58.608 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:58 smithi100 ceph-mon[29092]: 5.11 scrub ok 2024-09-17T07:04:58.609 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:58 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:04:58.609 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:58 smithi100 ceph-mon[39462]: 4.2 deep-scrub starts 2024-09-17T07:04:58.609 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:58 smithi100 ceph-mon[39462]: 4.2 deep-scrub ok 2024-09-17T07:04:58.609 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:58 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-17T07:04:58.609 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:58 smithi100 ceph-mon[39462]: osdmap e111: 8 total, 8 up, 8 in 2024-09-17T07:04:58.609 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:58 smithi100 ceph-mon[39462]: 4.8 deep-scrub starts 2024-09-17T07:04:58.609 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:58 smithi100 ceph-mon[39462]: 4.8 deep-scrub ok 2024-09-17T07:04:58.609 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:58 smithi100 ceph-mon[39462]: 5.11 scrub starts 2024-09-17T07:04:58.609 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:58 smithi100 ceph-mon[39462]: 5.11 scrub ok 2024-09-17T07:04:58.609 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:58 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:04:58.612 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:04:58.638 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 27 16 55498 55482 128.412 36.75 0.00433404 0.00778273 2024-09-17T07:04:58.652 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:58.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:58.653 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:04:58.754 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:04:58.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:58.792 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:58.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:04:58.893 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:04:58.940 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:58.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:58.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:04:59.044 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:04:59.072 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:59.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:59.073 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:04:59.174 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:04:59.214 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:59.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:59.217 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:04:59.318 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:04:59.348 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:04:59.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:59.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:04:59.452 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:04:59.475 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:59 smithi100 ceph-mon[39462]: 4.7 scrub starts 2024-09-17T07:04:59.475 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:59 smithi100 ceph-mon[39462]: 4.7 scrub ok 2024-09-17T07:04:59.475 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:59 smithi100 ceph-mon[39462]: osdmap e112: 8 total, 8 up, 8 in 2024-09-17T07:04:59.475 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:59 smithi100 ceph-mon[39462]: 2.12 scrub starts 2024-09-17T07:04:59.475 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:59 smithi100 ceph-mon[39462]: 2.12 scrub ok 2024-09-17T07:04:59.475 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:59 smithi100 ceph-mon[39462]: 7.7 scrub starts 2024-09-17T07:04:59.475 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:59 smithi100 ceph-mon[39462]: pgmap v103: 158 pgs: 10 peering, 10 unknown, 138 active+clean; 3.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 108 MiB/s wr, 1.73k op/s 2024-09-17T07:04:59.476 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:59 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-09-17T07:04:59.476 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:04:59 smithi100 ceph-mon[39462]: 7.7 scrub ok 2024-09-17T07:04:59.476 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:59 smithi100 ceph-mon[29092]: 4.7 scrub starts 2024-09-17T07:04:59.476 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:59 smithi100 ceph-mon[29092]: 4.7 scrub ok 2024-09-17T07:04:59.476 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:59 smithi100 ceph-mon[29092]: osdmap e112: 8 total, 8 up, 8 in 2024-09-17T07:04:59.476 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:59 smithi100 ceph-mon[29092]: 2.12 scrub starts 2024-09-17T07:04:59.476 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:59 smithi100 ceph-mon[29092]: 2.12 scrub ok 2024-09-17T07:04:59.476 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:59 smithi100 ceph-mon[29092]: 7.7 scrub starts 2024-09-17T07:04:59.476 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:59 smithi100 ceph-mon[29092]: pgmap v103: 158 pgs: 10 peering, 10 unknown, 138 active+clean; 3.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 108 MiB/s wr, 1.73k op/s 2024-09-17T07:04:59.477 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:59 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-09-17T07:04:59.477 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:04:59 smithi100 ceph-mon[29092]: 7.7 scrub ok 2024-09-17T07:04:59.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:59.478 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:59.478 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:04:59.578 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:04:59.600 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:59.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:59.602 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:04:59.638 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 28 16 56695 56679 126.497 74.8125 0.00213208 0.00782236 2024-09-17T07:04:59.703 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:04:59.724 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:59.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:59.725 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:04:59.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:59 smithi135 ceph-mon[33238]: 4.7 scrub starts 2024-09-17T07:04:59.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:59 smithi135 ceph-mon[33238]: 4.7 scrub ok 2024-09-17T07:04:59.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:59 smithi135 ceph-mon[33238]: osdmap e112: 8 total, 8 up, 8 in 2024-09-17T07:04:59.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:59 smithi135 ceph-mon[33238]: 2.12 scrub starts 2024-09-17T07:04:59.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:59 smithi135 ceph-mon[33238]: 2.12 scrub ok 2024-09-17T07:04:59.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:59 smithi135 ceph-mon[33238]: 7.7 scrub starts 2024-09-17T07:04:59.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:59 smithi135 ceph-mon[33238]: pgmap v103: 158 pgs: 10 peering, 10 unknown, 138 active+clean; 3.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 108 MiB/s wr, 1.73k op/s 2024-09-17T07:04:59.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:59 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-09-17T07:04:59.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:04:59 smithi135 ceph-mon[33238]: 7.7 scrub ok 2024-09-17T07:04:59.799 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:04:59.799 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1: {} 2024-09-17T07:04:59.799 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:04:59.799 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:04:59.799 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:04:59.800 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:04:59.800 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:04:59.800 INFO:teuthology.orchestra.run.smithi100.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:04:59.800 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:04:59.800 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:04:59.800 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:04:59.800 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:04:59.800 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:04:59.801 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:04:59.801 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:04:59.801 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:04:59.826 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:04:59.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:59.849 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:04:59.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:04:59.950 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:04:59.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:04:59.971 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:00.118 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:05:00.219 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:05:00.240 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:00.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:00.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:05:00.342 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:05:00.363 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:00.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:00.364 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:05:00.376 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:00 smithi100 ceph-mon[39462]: 6.17 scrub starts 2024-09-17T07:05:00.376 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:00 smithi100 ceph-mon[39462]: 6.17 scrub ok 2024-09-17T07:05:00.376 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:00 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-17T07:05:00.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:00 smithi100 ceph-mon[39462]: osdmap e113: 8 total, 8 up, 8 in 2024-09-17T07:05:00.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:00 smithi100 ceph-mon[39462]: 7.0 scrub starts 2024-09-17T07:05:00.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:00 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/3820941112' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:05:00.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:00 smithi100 ceph-mon[39462]: 7.0 scrub ok 2024-09-17T07:05:00.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:00 smithi100 ceph-mon[39462]: 2.1a scrub starts 2024-09-17T07:05:00.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:00 smithi100 ceph-mon[39462]: 2.1a scrub ok 2024-09-17T07:05:00.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:00 smithi100 ceph-mon[39462]: osdmap e114: 8 total, 8 up, 8 in 2024-09-17T07:05:00.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:00 smithi100 ceph-mon[29092]: 6.17 scrub starts 2024-09-17T07:05:00.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:00 smithi100 ceph-mon[29092]: 6.17 scrub ok 2024-09-17T07:05:00.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:00 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-17T07:05:00.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:00 smithi100 ceph-mon[29092]: osdmap e113: 8 total, 8 up, 8 in 2024-09-17T07:05:00.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:00 smithi100 ceph-mon[29092]: 7.0 scrub starts 2024-09-17T07:05:00.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:00 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/3820941112' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:05:00.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:00 smithi100 ceph-mon[29092]: 7.0 scrub ok 2024-09-17T07:05:00.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:00 smithi100 ceph-mon[29092]: 2.1a scrub starts 2024-09-17T07:05:00.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:00 smithi100 ceph-mon[29092]: 2.1a scrub ok 2024-09-17T07:05:00.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:00 smithi100 ceph-mon[29092]: osdmap e114: 8 total, 8 up, 8 in 2024-09-17T07:05:00.465 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:05:00.494 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:00.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:00.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:05:00.596 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:05:00.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:00.618 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:00.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:05:00.638 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 29 16 56695 56679 122.135 0 - 0.00782236 2024-09-17T07:05:00.719 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:05:00.739 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:00.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:00.740 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:05:00.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:00 smithi135 ceph-mon[33238]: 6.17 scrub starts 2024-09-17T07:05:00.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:00 smithi135 ceph-mon[33238]: 6.17 scrub ok 2024-09-17T07:05:00.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:00 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-17T07:05:00.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:00 smithi135 ceph-mon[33238]: osdmap e113: 8 total, 8 up, 8 in 2024-09-17T07:05:00.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:00 smithi135 ceph-mon[33238]: 7.0 scrub starts 2024-09-17T07:05:00.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:00 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/3820941112' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:05:00.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:00 smithi135 ceph-mon[33238]: 7.0 scrub ok 2024-09-17T07:05:00.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:00 smithi135 ceph-mon[33238]: 2.1a scrub starts 2024-09-17T07:05:00.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:00 smithi135 ceph-mon[33238]: 2.1a scrub ok 2024-09-17T07:05:00.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:00 smithi135 ceph-mon[33238]: osdmap e114: 8 total, 8 up, 8 in 2024-09-17T07:05:00.841 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:05:00.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:00.866 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:00.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:05:00.967 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:05:00.988 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:00.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:00.989 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:05:01.089 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:05:01.137 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:01.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:01.138 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:05:01.182 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-17T07:05:01.182 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:05:01.240 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:05:01.264 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:01.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:01.265 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:05:01.366 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:05:01.403 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:01.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:01.404 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:05:01.505 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:05:01.547 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:01.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:01.548 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:05:01.639 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 30 16 56940 56924 118.575 7.65625 0.0195107 0.00842781 2024-09-17T07:05:01.649 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:05:01.677 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:01.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:01.678 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:05:01.779 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:05:01.807 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:01.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:01.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:05:01.909 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:05:01.954 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:01.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:01.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:05:02.056 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:05:02.085 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:02.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:02.086 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:05:02.187 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:05:02.216 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:02.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:02.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:05:02.318 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:05:02.354 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:02.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:02.356 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:05:02.364 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:02 smithi100 ceph-mon[29092]: 5.1c scrub starts 2024-09-17T07:05:02.365 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:02 smithi100 ceph-mon[29092]: 5.1c scrub ok 2024-09-17T07:05:02.365 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:02 smithi100 ceph-mon[29092]: pgmap v106: 158 pgs: 10 peering, 10 unknown, 138 active+clean; 3.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 108 MiB/s wr, 1.73k op/s 2024-09-17T07:05:02.365 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:02 smithi100 ceph-mon[29092]: osdmap e115: 8 total, 8 up, 8 in 2024-09-17T07:05:02.365 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:02 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1503116630' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:05:02.365 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:02 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:05:02.365 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:02 smithi100 ceph-mon[39462]: 5.1c scrub starts 2024-09-17T07:05:02.365 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:02 smithi100 ceph-mon[39462]: 5.1c scrub ok 2024-09-17T07:05:02.365 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:02 smithi100 ceph-mon[39462]: pgmap v106: 158 pgs: 10 peering, 10 unknown, 138 active+clean; 3.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 108 MiB/s wr, 1.73k op/s 2024-09-17T07:05:02.365 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:02 smithi100 ceph-mon[39462]: osdmap e115: 8 total, 8 up, 8 in 2024-09-17T07:05:02.365 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:02 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1503116630' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:05:02.365 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:02 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:05:02.368 INFO:teuthology.orchestra.run.smithi100.stderr:noscrub is set 2024-09-17T07:05:02.432 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:05:02.457 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:05:02.479 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:02.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:02.480 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:05:02.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:02 smithi135 ceph-mon[33238]: 5.1c scrub starts 2024-09-17T07:05:02.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:02 smithi135 ceph-mon[33238]: 5.1c scrub ok 2024-09-17T07:05:02.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:02 smithi135 ceph-mon[33238]: pgmap v106: 158 pgs: 10 peering, 10 unknown, 138 active+clean; 3.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 108 MiB/s wr, 1.73k op/s 2024-09-17T07:05:02.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:02 smithi135 ceph-mon[33238]: osdmap e115: 8 total, 8 up, 8 in 2024-09-17T07:05:02.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:02 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1503116630' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:05:02.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:02 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:05:02.581 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:05:02.586 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-17T07:05:02.586 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd reweight 0 1 2024-09-17T07:05:02.606 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:02.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:02.608 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:05:02.641 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 31 16 57949 57933 116.774 63.0625 0.0145668 0.00855814 2024-09-17T07:05:02.709 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:05:02.778 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:02.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:02.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:05:02.880 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:05:02.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:02.937 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:02.937 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:05:03.038 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:05:03.065 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:03.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:03.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:05:03.167 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:05:03.207 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:03.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:03.208 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:05:03.309 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:05:03.350 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:03.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:03.351 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:05:03.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:03 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:05:03.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:03 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:05:03.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:03 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:05:03.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:03 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:05:03.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:03 smithi100 ceph-mon[29092]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:05:03.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:03 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T07:05:03.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:03 smithi100 ceph-mon[29092]: osdmap e116: 8 total, 8 up, 8 in 2024-09-17T07:05:03.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:03 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:05:03.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:03 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:05:03.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:03 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:05:03.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:03 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:05:03.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:03 smithi100 ceph-mon[39462]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:05:03.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:03 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T07:05:03.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:03 smithi100 ceph-mon[39462]: osdmap e116: 8 total, 8 up, 8 in 2024-09-17T07:05:03.452 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:05:03.482 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:03.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:03.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:05:03.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:03 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:05:03.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:03 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:05:03.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:03 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:05:03.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:03 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:05:03.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:03 smithi135 ceph-mon[33238]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:05:03.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:03 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T07:05:03.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:03 smithi135 ceph-mon[33238]: osdmap e116: 8 total, 8 up, 8 in 2024-09-17T07:05:03.584 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:05:03.613 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:03.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:03.614 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:05:03.641 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 32 16 59015 58999 115.207 66.625 0.00911379 0.00867437 2024-09-17T07:05:03.716 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:05:03.744 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:03.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:03.747 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:05:03.847 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:05:03.878 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:03.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:03.879 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:05:03.980 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:05:04.026 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:04.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:04.027 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:05:04.128 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:05:04.166 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:04.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:04.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:05:04.268 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:05:04.296 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:04.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:04.298 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:05:04.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:04 smithi100 ceph-mon[29092]: pgmap v109: 158 pgs: 1 active+recovering+undersized+remapped, 5 peering, 9 unknown, 143 active+clean; 3.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 1519/169155 objects misplaced (0.898%); 3.5 MiB/s, 56 objects/s recovering 2024-09-17T07:05:04.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:04 smithi100 ceph-mon[39462]: pgmap v109: 158 pgs: 1 active+recovering+undersized+remapped, 5 peering, 9 unknown, 143 active+clean; 3.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 1519/169155 objects misplaced (0.898%); 3.5 MiB/s, 56 objects/s recovering 2024-09-17T07:05:04.398 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:05:04.419 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:04.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:04.420 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:05:04.520 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:05:04.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:04 smithi135 ceph-mon[33238]: pgmap v109: 158 pgs: 1 active+recovering+undersized+remapped, 5 peering, 9 unknown, 143 active+clean; 3.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 1519/169155 objects misplaced (0.898%); 3.5 MiB/s, 56 objects/s recovering 2024-09-17T07:05:04.549 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:04.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:04.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:05:04.641 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 33 16 59893 59877 113.379 54.875 0.013409 0.00881385 2024-09-17T07:05:04.651 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:05:04.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:04.680 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:04.681 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:05:04.781 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:05:04.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:04.818 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:04.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:05:04.919 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:05:04.944 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:04.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:04.946 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:05:05.046 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:05:05.074 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:05.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:05.075 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:05:05.176 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:05:05.203 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:05.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:05.204 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:05:05.305 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:05:05.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:05.345 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:05.346 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:05:05.447 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:05:05.477 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:05.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:05.478 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:05:05.579 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:05:05.609 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:05.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:05.610 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:05:05.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:05 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:05:05.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:05 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:05:05.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:05 smithi100 ceph-mon[39462]: osdmap e117: 8 total, 8 up, 8 in 2024-09-17T07:05:05.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:05 smithi100 ceph-mon[39462]: pgmap v111: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 148 active+clean; 3.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 14196/175596 objects degraded (8.084%); 18379/175596 objects misplaced (10.467%); 8.3 MiB/s, 133 objects/s recovering 2024-09-17T07:05:05.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:05 smithi100 ceph-mon[39462]: osdmap e118: 8 total, 8 up, 8 in 2024-09-17T07:05:05.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:05 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:05:05.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:05 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:05:05.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:05 smithi100 ceph-mon[29092]: osdmap e117: 8 total, 8 up, 8 in 2024-09-17T07:05:05.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:05 smithi100 ceph-mon[29092]: pgmap v111: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 148 active+clean; 3.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 14196/175596 objects degraded (8.084%); 18379/175596 objects misplaced (10.467%); 8.3 MiB/s, 133 objects/s recovering 2024-09-17T07:05:05.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:05 smithi100 ceph-mon[29092]: osdmap e118: 8 total, 8 up, 8 in 2024-09-17T07:05:05.641 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 34 16 60787 60771 111.688 55.875 0.0146926 0.00894765 2024-09-17T07:05:05.712 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:05:05.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:05.740 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:05.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:05:05.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:05 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:05:05.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:05 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:05:05.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:05 smithi135 ceph-mon[33238]: osdmap e117: 8 total, 8 up, 8 in 2024-09-17T07:05:05.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:05 smithi135 ceph-mon[33238]: pgmap v111: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 148 active+clean; 3.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 14196/175596 objects degraded (8.084%); 18379/175596 objects misplaced (10.467%); 8.3 MiB/s, 133 objects/s recovering 2024-09-17T07:05:05.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:05 smithi135 ceph-mon[33238]: osdmap e118: 8 total, 8 up, 8 in 2024-09-17T07:05:05.841 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:05:05.868 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:05.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:05.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:05:05.970 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:05:06.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:06.000 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:06.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:05:06.101 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:05:06.143 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:06.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:06.144 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:05:06.244 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:05:06.272 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:06.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:06.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:05:06.374 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:05:06.376 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:05:05 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:05:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:05:06.401 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:06.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:06.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:05:06.503 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:05:06.531 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:06.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:06.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:05:06.633 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:05:06.641 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 35 16 61845 61829 110.386 66.125 0.00744442 0.00904258 2024-09-17T07:05:06.656 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:06.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:06.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:05:06.758 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:05:06.783 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:06.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:06.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:05:06.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:06 smithi135 ceph-mon[33238]: Health check failed: Degraded data redundancy: 14196/175596 objects degraded (8.084%), 4 pgs degraded (PG_DEGRADED) 2024-09-17T07:05:06.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:06 smithi100 ceph-mon[29092]: Health check failed: Degraded data redundancy: 14196/175596 objects degraded (8.084%), 4 pgs degraded (PG_DEGRADED) 2024-09-17T07:05:06.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:06 smithi100 ceph-mon[39462]: Health check failed: Degraded data redundancy: 14196/175596 objects degraded (8.084%), 4 pgs degraded (PG_DEGRADED) 2024-09-17T07:05:06.885 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:05:06.935 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:06.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:06.937 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:05:07.037 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:05:07.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:07.084 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:07.084 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:05:07.185 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:05:07.227 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:07.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:07.228 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:05:07.329 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:05:07.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:07.370 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:07.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:05:07.471 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:05:07.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:07.513 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:07.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:05:07.614 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:05:07.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:07.637 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:07.638 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:05:07.641 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 36 16 62932 62916 109.207 67.9375 0.0140806 0.00914554 2024-09-17T07:05:07.738 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:05:07.778 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:07.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:07.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:05:07.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:07 smithi135 ceph-mon[33238]: pgmap v113: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 148 active+clean; 3.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 855 B/s rd, 0 op/s; 14196/175596 objects degraded (8.084%); 18379/175596 objects misplaced (10.467%); 6.9 MiB/s, 110 objects/s recovering 2024-09-17T07:05:07.876 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:07 smithi100 ceph-mon[39462]: pgmap v113: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 148 active+clean; 3.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 855 B/s rd, 0 op/s; 14196/175596 objects degraded (8.084%); 18379/175596 objects misplaced (10.467%); 6.9 MiB/s, 110 objects/s recovering 2024-09-17T07:05:07.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:07 smithi100 ceph-mon[29092]: pgmap v113: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 148 active+clean; 3.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 855 B/s rd, 0 op/s; 14196/175596 objects degraded (8.084%); 18379/175596 objects misplaced (10.467%); 6.9 MiB/s, 110 objects/s recovering 2024-09-17T07:05:07.881 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:05:07.911 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:07.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:07.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:05:08.013 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:05:08.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:08.044 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:08.045 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:05:08.145 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:05:08.181 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:08.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:08.182 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:05:08.283 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:05:08.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:08.323 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:08.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:05:08.395 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:05:08 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:05:08.424 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:05:08.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:08.453 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:08.453 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:05:08.554 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:05:08.586 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:08.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:08.587 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:05:08.642 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 37 16 64054 64038 108.151 70.125 0.0232327 0.00924075 2024-09-17T07:05:08.687 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:05:08.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:08.735 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:08.735 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:05:08.780 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:08 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:05:08.781 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:08 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:05:08.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:08 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:05:08.831 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-17T07:05:08.836 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:05:08.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:08.864 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:08.864 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:05:08.896 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:05:08.964 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:05:08.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:08.990 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:08.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:05:09.092 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:05:09.136 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:09.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:09.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:05:09.238 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:05:09.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:09.285 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:09.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:05:09.386 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:05:09.431 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:09.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:09.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:05:09.533 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:05:09.573 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:09.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:09.574 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:05:09.642 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 38 16 65035 65019 106.918 61.3125 0.00579066 0.00934793 2024-09-17T07:05:09.675 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:05:09.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:09.720 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:09.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:05:09.721 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:09 smithi100 ceph-mon[29092]: pgmap v114: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 148 active+clean; 3.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.5 KiB/s rd, 47 MiB/s wr, 746 op/s; 14196/190740 objects degraded (7.443%); 16181/190740 objects misplaced (8.483%); 23 MiB/s, 371 objects/s recovering 2024-09-17T07:05:09.721 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:09 smithi100 ceph-mon[39462]: pgmap v114: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 148 active+clean; 3.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.5 KiB/s rd, 47 MiB/s wr, 746 op/s; 14196/190740 objects degraded (7.443%); 16181/190740 objects misplaced (8.483%); 23 MiB/s, 371 objects/s recovering 2024-09-17T07:05:09.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:09 smithi135 ceph-mon[33238]: pgmap v114: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 148 active+clean; 3.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.5 KiB/s rd, 47 MiB/s wr, 746 op/s; 14196/190740 objects degraded (7.443%); 16181/190740 objects misplaced (8.483%); 23 MiB/s, 371 objects/s recovering 2024-09-17T07:05:09.821 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:05:09.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:09.862 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:09.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:05:09.964 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:05:10.018 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:05:10.027 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:10.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:10.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:05:10.129 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:05:10.192 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:10.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:10.193 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:05:10.294 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:05:10.345 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:10.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:10.346 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:05:10.447 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:05:10.475 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:10.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:10.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:05:10.578 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:05:10.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:10.616 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:10.617 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:05:10.642 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 39 16 66034 66018 105.778 62.4375 0.00976967 0.00943964 2024-09-17T07:05:10.718 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:05:10.745 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:10.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:10.746 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:05:10.847 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:05:10.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:10.890 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:10.890 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:05:10.991 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:05:11.020 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:11.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:11.021 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:05:11.122 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:05:11.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:11.162 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:11.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:05:11.263 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:05:11.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:11.293 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:11.293 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:05:11.394 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:05:11.421 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:11.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:11.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:05:11.523 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:05:11.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:11.559 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:11.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:05:11.642 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:2024-09-17T07:05:11.642652+0000 min lat: 0.00128777 max lat: 2.09639 avg lat: 0.00954425 2024-09-17T07:05:11.642 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-17T07:05:11.642 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 40 16 67041 67025 104.707 62.9375 0.00927822 0.00954425 2024-09-17T07:05:11.660 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:05:11.688 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:11.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:11.689 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:05:11.790 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:05:11.811 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:11.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:11.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:05:11.913 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:05:11.942 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:11.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:11.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:05:11.963 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-17T07:05:12.044 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:05:12.071 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:12.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:12.072 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:05:12.173 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:05:12.200 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:12.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:12.201 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:05:12.302 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:05:12.334 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:12.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:12.335 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:05:12.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:12 smithi100 ceph-mon[29092]: pgmap v115: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 148 active+clean; 3.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 56 MiB/s wr, 899 op/s; 14196/190740 objects degraded (7.443%); 16181/190740 objects misplaced (8.483%); 18 MiB/s, 281 objects/s recovering 2024-09-17T07:05:12.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:12 smithi100 ceph-mon[39462]: pgmap v115: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 148 active+clean; 3.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 56 MiB/s wr, 899 op/s; 14196/190740 objects degraded (7.443%); 16181/190740 objects misplaced (8.483%); 18 MiB/s, 281 objects/s recovering 2024-09-17T07:05:12.435 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:05:12.477 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:12.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:12.478 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:05:12.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:12 smithi135 ceph-mon[33238]: pgmap v115: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 148 active+clean; 3.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 56 MiB/s wr, 899 op/s; 14196/190740 objects degraded (7.443%); 16181/190740 objects misplaced (8.483%); 18 MiB/s, 281 objects/s recovering 2024-09-17T07:05:12.578 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:05:12.614 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:12.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:12.615 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:05:12.642 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 41 16 68041 68025 103.677 62.5 0.0086765 0.00963143 2024-09-17T07:05:12.715 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:05:12.744 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:12.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:12.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:05:12.846 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:05:12.871 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:12.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:12.872 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:05:12.973 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:05:13.002 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:13.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:13.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:05:13.104 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:05:13.134 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:13.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:13.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:05:13.237 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:05:13.281 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:13.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:13.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:05:13.384 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:05:13.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:13.422 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:13.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:05:13.523 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:05:13.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:13 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 14196/195471 objects degraded (7.262%), 4 pgs degraded (PG_DEGRADED) 2024-09-17T07:05:13.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:13.569 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:13.570 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:05:13.592 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:05:13.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:13 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 14196/195471 objects degraded (7.262%), 4 pgs degraded (PG_DEGRADED) 2024-09-17T07:05:13.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:13 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 14196/195471 objects degraded (7.262%), 4 pgs degraded (PG_DEGRADED) 2024-09-17T07:05:13.642 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 42 16 68995 68979 102.628 59.625 0.0210571 0.00973831 2024-09-17T07:05:13.671 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:05:13.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:13.723 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:13.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:05:13.824 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:05:13.875 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:13.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:13.876 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:05:13.977 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:05:14.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:14.018 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:14.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:05:14.119 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:05:14.164 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:14.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:14.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:05:14.266 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:05:14.303 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:14.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:14.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:05:14.405 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:05:14.440 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:14.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:14.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:05:14.542 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:05:14.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:14 smithi135 ceph-mon[33238]: pgmap v116: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 148 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 590 B/s rd, 48 MiB/s wr, 764 op/s; 14196/195471 objects degraded (7.262%); 15400/195471 objects misplaced (7.878%); 19 MiB/s, 299 objects/s recovering 2024-09-17T07:05:14.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:14.581 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:14.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:05:14.590 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:14 smithi100 ceph-mon[29092]: pgmap v116: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 148 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 590 B/s rd, 48 MiB/s wr, 764 op/s; 14196/195471 objects degraded (7.262%); 15400/195471 objects misplaced (7.878%); 19 MiB/s, 299 objects/s recovering 2024-09-17T07:05:14.591 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:14 smithi100 ceph-mon[39462]: pgmap v116: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 148 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 590 B/s rd, 48 MiB/s wr, 764 op/s; 14196/195471 objects degraded (7.262%); 15400/195471 objects misplaced (7.878%); 19 MiB/s, 299 objects/s recovering 2024-09-17T07:05:14.642 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 43 16 69970 69954 101.659 60.9375 0.0035772 0.00978202 2024-09-17T07:05:14.682 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:05:14.706 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:14.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:14.708 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:05:14.809 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:05:14.832 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:14.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:14.834 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:05:14.935 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:05:14.964 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:14.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:14.965 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:05:15.066 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:05:15.097 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:15.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:15.098 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:05:15.204 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:05:15.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:15.233 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:15.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:05:15.334 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:05:15.363 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:15.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:15.365 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:05:15.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:15 smithi100 ceph-mon[29092]: osdmap e119: 8 total, 8 up, 8 in 2024-09-17T07:05:15.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:15 smithi100 ceph-mon[29092]: osdmap e120: 8 total, 8 up, 8 in 2024-09-17T07:05:15.376 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:15 smithi100 ceph-mon[39462]: osdmap e119: 8 total, 8 up, 8 in 2024-09-17T07:05:15.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:15 smithi100 ceph-mon[39462]: osdmap e120: 8 total, 8 up, 8 in 2024-09-17T07:05:15.465 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:05:15.495 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:15.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:15.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:05:15.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:15 smithi135 ceph-mon[33238]: osdmap e119: 8 total, 8 up, 8 in 2024-09-17T07:05:15.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:15 smithi135 ceph-mon[33238]: osdmap e120: 8 total, 8 up, 8 in 2024-09-17T07:05:15.598 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:05:15.643 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 44 16 70796 70780 100.52 51.625 0.0180309 0.00994197 2024-09-17T07:05:15.644 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:15.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:15.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:05:15.746 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:05:15.793 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:15.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:15.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:05:15.895 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:05:15.923 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:15.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:15.924 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:05:16.024 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:05:16.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:16.054 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:16.054 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:05:16.154 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:05:16.180 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:05:15 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:05:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:05:16.202 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:16.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:16.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:05:16.304 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:05:16.344 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:16.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:16.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:05:16.446 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:05:16.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:16.492 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:16.492 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:05:16.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:16 smithi135 ceph-mon[33238]: pgmap v118: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 148 active+clean; 4.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.0 KiB/s rd, 63 MiB/s wr, 1.01k op/s; 14196/205767 objects degraded (6.899%); 13626/205767 objects misplaced (6.622%); 25 MiB/s, 397 objects/s recovering 2024-09-17T07:05:16.557 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:16 smithi100 ceph-mon[29092]: pgmap v118: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 148 active+clean; 4.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.0 KiB/s rd, 63 MiB/s wr, 1.01k op/s; 14196/205767 objects degraded (6.899%); 13626/205767 objects misplaced (6.622%); 25 MiB/s, 397 objects/s recovering 2024-09-17T07:05:16.557 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:16 smithi100 ceph-mon[39462]: pgmap v118: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 148 active+clean; 4.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.0 KiB/s rd, 63 MiB/s wr, 1.01k op/s; 14196/205767 objects degraded (6.899%); 13626/205767 objects misplaced (6.622%); 25 MiB/s, 397 objects/s recovering 2024-09-17T07:05:16.593 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:05:16.641 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:16.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:16.642 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:05:16.643 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 45 16 71927 71911 99.8561 70.6875 0.0224122 0.0100091 2024-09-17T07:05:16.743 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:05:16.769 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:16.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:16.770 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:05:16.872 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:05:16.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:16.921 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:16.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:05:17.022 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:05:17.047 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:17.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:17.048 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:05:17.149 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:05:17.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:17.190 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:17.190 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:05:17.277 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:05:17.292 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:05:17.309 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-17T07:05:17.309 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:05:17.319 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:17.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:17.320 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:05:17.412 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:17 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:05:17.412 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:17 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:05:17.421 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:05:17.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:17.468 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:17.469 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:05:17.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:17 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:05:17.569 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:05:17.598 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:17.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:17.599 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:05:17.643 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 46 16 72747 72731 98.7995 51.25 0.0112983 0.0101163 2024-09-17T07:05:17.699 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:05:17.725 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:17.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:17.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:05:17.827 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:05:17.870 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:17.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:17.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:05:17.972 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:05:18.009 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:18.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:18.010 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:05:18.111 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:05:18.155 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:18.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:18.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:05:18.257 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:05:18.284 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:18.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:18.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:05:18.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:18 smithi100 ceph-mon[29092]: pgmap v120: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 148 active+clean; 4.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 39 MiB/s wr, 626 op/s; 14196/205767 objects degraded (6.899%); 13626/205767 objects misplaced (6.622%); 17 MiB/s, 269 objects/s recovering 2024-09-17T07:05:18.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:18 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2844900547' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:05:18.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:18 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:05:18.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:18 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:05:18.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:18 smithi100 ceph-mon[39462]: pgmap v120: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 148 active+clean; 4.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 39 MiB/s wr, 626 op/s; 14196/205767 objects degraded (6.899%); 13626/205767 objects misplaced (6.622%); 17 MiB/s, 269 objects/s recovering 2024-09-17T07:05:18.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:18 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2844900547' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:05:18.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:18 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:05:18.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:18 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:05:18.386 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:05:18.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:18.431 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:18.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:05:18.532 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:05:18.543 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:05:18 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:05:18.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:18 smithi135 ceph-mon[33238]: pgmap v120: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 148 active+clean; 4.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 39 MiB/s wr, 626 op/s; 14196/205767 objects degraded (6.899%); 13626/205767 objects misplaced (6.622%); 17 MiB/s, 269 objects/s recovering 2024-09-17T07:05:18.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:18 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2844900547' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:05:18.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:18 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:05:18.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:18 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:05:18.576 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:18.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:18.577 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:05:18.645 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 47 16 73693 73677 97.9536 59.125 0.0253502 0.0102029 2024-09-17T07:05:18.646 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-17T07:05:18.678 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:05:18.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:18.738 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:18.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:05:18.839 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:05:18.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:18.896 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:18.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:05:18.997 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:05:19.025 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:19.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:19.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:05:19.127 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:05:19.154 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:19.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:19.155 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:05:19.256 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:05:19.297 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:19.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:19.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:05:19.350 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:05:19.350 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1: {} 2024-09-17T07:05:19.350 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:05:19.350 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:05:19.350 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:05:19.350 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:05:19.350 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:05:19.350 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:05:19.351 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:05:19.351 INFO:teuthology.orchestra.run.smithi100.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:05:19.351 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:05:19.351 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:05:19.351 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:05:19.351 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:05:19.351 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:05:19.351 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:05:19.399 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:05:19.428 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:19.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:19.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:05:19.530 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:05:19.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:19 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:05:19.565 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:19 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:05:19.565 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:19 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:05:19.572 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:19.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:19.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:05:19.645 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 48 16 74699 74683 97.2228 62.875 0.0146529 0.0102798 2024-09-17T07:05:19.674 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:05:19.721 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:19.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:19.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:05:19.823 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:05:19.878 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:19.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:19.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:05:19.980 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:05:20.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:20.023 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:20.023 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:05:20.124 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:05:20.155 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:20.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:20.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:05:20.258 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:05:20.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:20 smithi135 ceph-mon[33238]: pgmap v121: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 149 active+clean; 4.5 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 77 MiB/s wr, 1.23k op/s; 14196/220128 objects degraded (6.449%); 11087/220128 objects misplaced (5.037%); 32 MiB/s, 514 objects/s recovering 2024-09-17T07:05:20.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:20 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 14196/220128 objects degraded (6.449%), 4 pgs degraded (PG_DEGRADED) 2024-09-17T07:05:20.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:20 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2077269062' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-17T07:05:20.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:20 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-17T07:05:20.320 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:20.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:20.321 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:05:20.423 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:05:20.452 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:20.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:20.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:05:20.486 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:20 smithi100 ceph-mon[29092]: pgmap v121: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 149 active+clean; 4.5 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 77 MiB/s wr, 1.23k op/s; 14196/220128 objects degraded (6.449%); 11087/220128 objects misplaced (5.037%); 32 MiB/s, 514 objects/s recovering 2024-09-17T07:05:20.486 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:20 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 14196/220128 objects degraded (6.449%), 4 pgs degraded (PG_DEGRADED) 2024-09-17T07:05:20.486 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:20 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2077269062' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-17T07:05:20.486 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:20 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-17T07:05:20.487 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:20 smithi100 ceph-mon[39462]: pgmap v121: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 149 active+clean; 4.5 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 77 MiB/s wr, 1.23k op/s; 14196/220128 objects degraded (6.449%); 11087/220128 objects misplaced (5.037%); 32 MiB/s, 514 objects/s recovering 2024-09-17T07:05:20.487 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:20 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 14196/220128 objects degraded (6.449%), 4 pgs degraded (PG_DEGRADED) 2024-09-17T07:05:20.487 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:20 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2077269062' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-17T07:05:20.487 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:20 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-17T07:05:20.556 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:05:20.598 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:20.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:20.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:05:20.645 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 49 16 75752 75736 96.5817 65.8125 0.0163728 0.0103485 2024-09-17T07:05:20.700 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:05:20.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:20.734 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:20.734 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:05:20.835 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:05:20.870 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:20.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:20.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:05:20.972 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:05:21.005 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:21.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:21.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:05:21.107 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:05:21.150 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:21.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:21.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:05:21.252 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:05:21.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:21.280 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:21.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:05:21.381 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:05:21.426 INFO:teuthology.orchestra.run.smithi100.stderr:reweighted osd.0 to 1 (10000) 2024-09-17T07:05:21.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:21.438 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:21.438 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:05:21.539 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:05:21.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:21 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-17T07:05:21.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:21 smithi135 ceph-mon[33238]: osdmap e121: 8 total, 8 up, 8 in 2024-09-17T07:05:21.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:21 smithi135 ceph-mon[33238]: pgmap v123: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 149 active+clean; 4.5 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 745 B/s rd, 44 MiB/s wr, 698 op/s; 14196/220128 objects degraded (6.449%); 11087/220128 objects misplaced (5.037%); 18 MiB/s, 286 objects/s recovering 2024-09-17T07:05:21.581 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:21.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:21.582 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:05:21.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:21 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-17T07:05:21.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:21 smithi100 ceph-mon[29092]: osdmap e121: 8 total, 8 up, 8 in 2024-09-17T07:05:21.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:21 smithi100 ceph-mon[29092]: pgmap v123: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 149 active+clean; 4.5 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 745 B/s rd, 44 MiB/s wr, 698 op/s; 14196/220128 objects degraded (6.449%); 11087/220128 objects misplaced (5.037%); 18 MiB/s, 286 objects/s recovering 2024-09-17T07:05:21.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:21 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-17T07:05:21.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:21 smithi100 ceph-mon[39462]: osdmap e121: 8 total, 8 up, 8 in 2024-09-17T07:05:21.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:21 smithi100 ceph-mon[39462]: pgmap v123: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 149 active+clean; 4.5 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 745 B/s rd, 44 MiB/s wr, 698 op/s; 14196/220128 objects degraded (6.449%); 11087/220128 objects misplaced (5.037%); 18 MiB/s, 286 objects/s recovering 2024-09-17T07:05:21.644 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 50 16 76916 76900 96.105 72.75 0.0257334 0.0103997 2024-09-17T07:05:21.683 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:05:21.711 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:21.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:21.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:05:21.813 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:05:21.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:21.863 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:21.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:05:21.964 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:05:22.008 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:22.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:22.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:05:22.110 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:05:22.138 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:22.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:22.139 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:05:22.158 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:05:22.226 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd reweight 1 1 2024-09-17T07:05:22.239 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:05:22.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:22.328 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:22.329 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:05:22.429 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:05:22.489 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:22.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:22.490 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:05:22.591 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:05:22.617 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:22.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:22.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:05:22.645 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 51 16 77865 77849 95.3834 59.3125 0.0133683 0.0104585 2024-09-17T07:05:22.719 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:05:22.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:22.759 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:22.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:05:22.829 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:05:22.860 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:05:22.886 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:05:22.897 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:05:22.933 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:22.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:22.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:05:23.041 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:05:23.081 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:23.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:23.082 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:05:23.183 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:05:23.205 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:23.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:23.206 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:05:23.257 INFO:teuthology.orchestra.run.smithi100.stderr:nodeep-scrub is set 2024-09-17T07:05:23.307 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:05:23.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:23.337 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:23.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:05:23.438 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:05:23.463 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:23.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:23.465 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:05:23.542 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:23 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:05:23.542 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:23 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:05:23.542 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:23 smithi100 ceph-mon[29092]: osdmap e122: 8 total, 8 up, 8 in 2024-09-17T07:05:23.542 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:23 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:05:23.543 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:23 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:05:23.543 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:23 smithi100 ceph-mon[29092]: pgmap v125: 158 pgs: 1 creating+peering, 4 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 149 active+clean; 4.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 640 B/s rd, 50 MiB/s wr, 803 op/s; 14196/225009 objects degraded (6.309%); 10641/225009 objects misplaced (4.729%); 19 MiB/s, 301 objects/s recovering 2024-09-17T07:05:23.543 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:23 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1169926379' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:05:23.543 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:23 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:05:23.543 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:23 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:05:23.543 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:23 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:05:23.543 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:23 smithi100 ceph-mon[39462]: osdmap e122: 8 total, 8 up, 8 in 2024-09-17T07:05:23.543 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:23 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:05:23.543 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:23 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:05:23.543 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:23 smithi100 ceph-mon[39462]: pgmap v125: 158 pgs: 1 creating+peering, 4 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 149 active+clean; 4.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 640 B/s rd, 50 MiB/s wr, 803 op/s; 14196/225009 objects degraded (6.309%); 10641/225009 objects misplaced (4.729%); 19 MiB/s, 301 objects/s recovering 2024-09-17T07:05:23.543 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:23 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1169926379' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:05:23.544 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:23 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:05:23.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:23 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:05:23.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:23 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:05:23.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:23 smithi135 ceph-mon[33238]: osdmap e122: 8 total, 8 up, 8 in 2024-09-17T07:05:23.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:23 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:05:23.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:23 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:05:23.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:23 smithi135 ceph-mon[33238]: pgmap v125: 158 pgs: 1 creating+peering, 4 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 149 active+clean; 4.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 640 B/s rd, 50 MiB/s wr, 803 op/s; 14196/225009 objects degraded (6.309%); 10641/225009 objects misplaced (4.729%); 19 MiB/s, 301 objects/s recovering 2024-09-17T07:05:23.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:23 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1169926379' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:05:23.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:23 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:05:23.566 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:05:23.590 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:23.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:23.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:05:23.645 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 52 16 78350 78334 94.1321 30.3125 0.00732161 0.0105872 2024-09-17T07:05:23.691 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:05:23.730 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:23.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:23.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:05:23.831 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:05:23.866 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:23.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:23.867 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:05:23.967 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:05:24.003 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:24.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:24.004 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:05:24.105 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:05:24.136 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:24.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:24.137 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:05:24.237 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:05:24.260 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:24.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:24.261 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:05:24.362 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:05:24.390 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:24.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:24.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:05:24.492 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:05:24.519 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:24.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:24.520 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:05:24.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:24 smithi135 ceph-mon[33238]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:05:24.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:24 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T07:05:24.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:24 smithi135 ceph-mon[33238]: osdmap e123: 8 total, 8 up, 8 in 2024-09-17T07:05:24.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:24 smithi135 ceph-mon[33238]: osdmap e124: 8 total, 8 up, 8 in 2024-09-17T07:05:24.622 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:05:24.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:24 smithi100 ceph-mon[29092]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:05:24.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:24 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T07:05:24.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:24 smithi100 ceph-mon[29092]: osdmap e123: 8 total, 8 up, 8 in 2024-09-17T07:05:24.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:24 smithi100 ceph-mon[29092]: osdmap e124: 8 total, 8 up, 8 in 2024-09-17T07:05:24.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:24 smithi100 ceph-mon[39462]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:05:24.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:24 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T07:05:24.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:24 smithi100 ceph-mon[39462]: osdmap e123: 8 total, 8 up, 8 in 2024-09-17T07:05:24.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:24 smithi100 ceph-mon[39462]: osdmap e124: 8 total, 8 up, 8 in 2024-09-17T07:05:24.649 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 53 16 78898 78882 92.9965 34.25 0.0172185 0.0107479 2024-09-17T07:05:24.665 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:24.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:24.666 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:05:24.767 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:05:24.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:24.801 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:24.801 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:05:24.902 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:05:24.932 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:24.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:24.933 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:05:25.034 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:05:25.081 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:25.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:25.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:05:25.183 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:05:25.205 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:25.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:25.206 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:05:25.306 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:05:25.335 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:25.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:25.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:05:25.437 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:05:25.463 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:25.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:25.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:05:25.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:25 smithi135 ceph-mon[33238]: pgmap v128: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 1 creating+peering, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 149 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.0 KiB/s rd, 63 MiB/s wr, 1.01k op/s; 7156/234714 objects degraded (3.049%); 15494/234714 objects misplaced (6.601%); 26 MiB/s, 414 objects/s recovering 2024-09-17T07:05:25.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:25 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 14196/225009 objects degraded (6.309%), 4 pgs degraded (PG_DEGRADED) 2024-09-17T07:05:25.565 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:05:25.592 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:25.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:25.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:05:25.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:25 smithi100 ceph-mon[29092]: pgmap v128: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 1 creating+peering, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 149 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.0 KiB/s rd, 63 MiB/s wr, 1.01k op/s; 7156/234714 objects degraded (3.049%); 15494/234714 objects misplaced (6.601%); 26 MiB/s, 414 objects/s recovering 2024-09-17T07:05:25.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:25 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 14196/225009 objects degraded (6.309%), 4 pgs degraded (PG_DEGRADED) 2024-09-17T07:05:25.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:25 smithi100 ceph-mon[39462]: pgmap v128: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 1 creating+peering, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 149 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.0 KiB/s rd, 63 MiB/s wr, 1.01k op/s; 7156/234714 objects degraded (3.049%); 15494/234714 objects misplaced (6.601%); 26 MiB/s, 414 objects/s recovering 2024-09-17T07:05:25.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:25 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 14196/225009 objects degraded (6.309%), 4 pgs degraded (PG_DEGRADED) 2024-09-17T07:05:25.648 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 54 16 79797 79781 92.3148 56.1875 0.0274524 0.010827 2024-09-17T07:05:25.694 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:05:25.733 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:25.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:25.734 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:05:25.835 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:05:25.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:25.876 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:25.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:05:25.977 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:05:26.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:26.020 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:26.021 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:05:26.122 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:05:26.150 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:26.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:26.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:05:26.251 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:05:26.278 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:26.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:26.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:05:26.377 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:05:25 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:05:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:05:26.380 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:05:26.403 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:26.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:26.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:05:26.505 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:05:26.549 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:26.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:26.550 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:05:26.649 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 55 15 80742 80727 91.7114 59.125 0.0257878 0.0108987 2024-09-17T07:05:26.651 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:05:26.677 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:26.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:26.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:05:26.778 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:05:26.824 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:26.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:26.825 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:05:26.925 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:05:26.953 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:26.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:26.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:05:27.055 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:05:27.096 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:27.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:27.097 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:05:27.197 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:05:27.237 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:27.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:27.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:05:27.339 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:05:27.385 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:27.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:27.386 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:05:27.486 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:05:27.523 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:27.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:27.524 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:05:27.625 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:05:27.646 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:27.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:27.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:05:27.649 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 56 16 81806 81790 91.26 66.4375 0.00865821 0.0109479 2024-09-17T07:05:27.748 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:05:27.788 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:27.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:27.789 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:05:27.890 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:05:27.919 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:27.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:27.920 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:05:28.021 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:05:28.064 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:28.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:28.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:05:28.166 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:05:28.187 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:28.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:28.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:05:28.288 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:05:28.315 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:28.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:28.316 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:05:28.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:28 smithi100 ceph-mon[29092]: pgmap v129: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 1 creating+peering, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 149 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 51 MiB/s wr, 810 op/s; 7156/234714 objects degraded (3.049%); 15494/234714 objects misplaced (6.601%); 21 MiB/s, 332 objects/s recovering 2024-09-17T07:05:28.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:28 smithi100 ceph-mon[39462]: pgmap v129: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 1 creating+peering, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 149 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 51 MiB/s wr, 810 op/s; 7156/234714 objects degraded (3.049%); 15494/234714 objects misplaced (6.601%); 21 MiB/s, 332 objects/s recovering 2024-09-17T07:05:28.417 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:05:28.437 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:28.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:28.438 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:05:28.539 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:05:28.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:28 smithi135 ceph-mon[33238]: pgmap v129: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 1 creating+peering, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 149 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 51 MiB/s wr, 810 op/s; 7156/234714 objects degraded (3.049%); 15494/234714 objects misplaced (6.601%); 21 MiB/s, 332 objects/s recovering 2024-09-17T07:05:28.543 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:05:28 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:05:28.587 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:28.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:28.588 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:05:28.649 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 57 16 82706 82690 90.6458 56.25 0.0098999 0.0110266 2024-09-17T07:05:28.689 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:05:28.725 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:28.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:28.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:05:28.828 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:05:28.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:28.852 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:28.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:05:28.954 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:05:28.979 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:28.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:28.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:05:29.081 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:05:29.106 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:29.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:29.107 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:05:29.208 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:05:29.234 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:29.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:29.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:05:29.336 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:05:29.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:29.360 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:29.360 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:05:29.371 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:29 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:05:29.371 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:29 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:05:29.461 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:05:29.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:29.511 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:29.511 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:05:29.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:29 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:05:29.611 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:05:29.632 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:29.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:29.633 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:05:29.649 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 58 16 83536 83520 89.9774 51.875 0.00286357 0.0111057 2024-09-17T07:05:29.734 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:05:29.764 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:29.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:29.765 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:05:29.866 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:05:29.899 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:29.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:29.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:05:30.001 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:05:30.045 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:30.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:30.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:05:30.147 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:05:30.174 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:30.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:30.176 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:05:30.276 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:05:30.297 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:30.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:30.299 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:05:30.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:30 smithi100 ceph-mon[29092]: pgmap v130: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 5 active+recovering+undersized+remapped, 150 active+clean; 5.0 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.5 KiB/s rd, 84 MiB/s wr, 1.34k op/s; 7156/247401 objects degraded (2.892%); 12800/247401 objects misplaced (5.174%); 32 MiB/s, 509 objects/s recovering 2024-09-17T07:05:30.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:30 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 7156/247401 objects degraded (2.892%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T07:05:30.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:30 smithi100 ceph-mon[39462]: pgmap v130: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 5 active+recovering+undersized+remapped, 150 active+clean; 5.0 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.5 KiB/s rd, 84 MiB/s wr, 1.34k op/s; 7156/247401 objects degraded (2.892%); 12800/247401 objects misplaced (5.174%); 32 MiB/s, 509 objects/s recovering 2024-09-17T07:05:30.381 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:30 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 7156/247401 objects degraded (2.892%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T07:05:30.399 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:05:30.425 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:30.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:30.427 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:05:30.528 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:05:30.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:30 smithi135 ceph-mon[33238]: pgmap v130: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 5 active+recovering+undersized+remapped, 150 active+clean; 5.0 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.5 KiB/s rd, 84 MiB/s wr, 1.34k op/s; 7156/247401 objects degraded (2.892%); 12800/247401 objects misplaced (5.174%); 32 MiB/s, 509 objects/s recovering 2024-09-17T07:05:30.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:30 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 7156/247401 objects degraded (2.892%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T07:05:30.553 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:30.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:30.554 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:05:30.649 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 59 16 84375 84359 89.3411 52.4375 0.00318228 0.0111413 2024-09-17T07:05:30.655 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:05:30.676 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:30.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:30.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:05:30.779 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:05:30.810 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:30.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:30.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:05:30.912 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:05:30.935 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:30.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:30.936 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:05:31.037 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:05:31.058 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:31.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:31.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:05:31.160 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:05:31.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:31.193 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:31.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:05:31.294 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:05:31.335 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:31.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:31.336 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:05:31.437 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:05:31.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:31.459 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:31.459 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:05:31.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:31 smithi135 ceph-mon[33238]: osdmap e125: 8 total, 8 up, 8 in 2024-09-17T07:05:31.560 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:05:31.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:31.599 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:31.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:05:31.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:31 smithi100 ceph-mon[29092]: osdmap e125: 8 total, 8 up, 8 in 2024-09-17T07:05:31.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:31 smithi100 ceph-mon[39462]: osdmap e125: 8 total, 8 up, 8 in 2024-09-17T07:05:31.649 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:2024-09-17T07:05:31.649817+0000 min lat: 0.00128777 max lat: 2.09639 avg lat: 0.0112976 2024-09-17T07:05:31.649 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-17T07:05:31.649 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 60 15 84795 84780 88.2907 26.3125 0.00274007 0.0112976 2024-09-17T07:05:31.700 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:05:31.721 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:31.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:31.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:05:31.823 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:05:31.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:31.844 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:31.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:05:31.945 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:05:31.972 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:31.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:31.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:05:32.074 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:05:32.100 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:32.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:32.101 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:05:32.202 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:05:32.223 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:32.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:32.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:05:32.325 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:05:32.364 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:32.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:32.365 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:05:32.466 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:05:32.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:32.508 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:32.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:05:32.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:32 smithi135 ceph-mon[33238]: pgmap v132: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 5 active+recovering+undersized+remapped, 150 active+clean; 5.0 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 657 B/s rd, 34 MiB/s wr, 543 op/s; 7156/247401 objects degraded (2.892%); 12800/247401 objects misplaced (5.174%); 12 MiB/s, 187 objects/s recovering 2024-09-17T07:05:32.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:32 smithi135 ceph-mon[33238]: osdmap e126: 8 total, 8 up, 8 in 2024-09-17T07:05:32.609 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:05:32.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:32 smithi100 ceph-mon[29092]: pgmap v132: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 5 active+recovering+undersized+remapped, 150 active+clean; 5.0 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 657 B/s rd, 34 MiB/s wr, 543 op/s; 7156/247401 objects degraded (2.892%); 12800/247401 objects misplaced (5.174%); 12 MiB/s, 187 objects/s recovering 2024-09-17T07:05:32.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:32 smithi100 ceph-mon[29092]: osdmap e126: 8 total, 8 up, 8 in 2024-09-17T07:05:32.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:32 smithi100 ceph-mon[39462]: pgmap v132: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 5 active+recovering+undersized+remapped, 150 active+clean; 5.0 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 657 B/s rd, 34 MiB/s wr, 543 op/s; 7156/247401 objects degraded (2.892%); 12800/247401 objects misplaced (5.174%); 12 MiB/s, 187 objects/s recovering 2024-09-17T07:05:32.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:32 smithi100 ceph-mon[39462]: osdmap e126: 8 total, 8 up, 8 in 2024-09-17T07:05:32.650 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 61 16 85441 85425 87.5042 40.3125 0.0244771 0.0114225 2024-09-17T07:05:32.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:32.651 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:32.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:05:32.752 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:05:32.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:32.783 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:32.784 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:05:32.885 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:05:32.922 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:32.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:32.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:05:33.024 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:05:33.050 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:33.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:33.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:05:33.152 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:05:33.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:33.176 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:33.176 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:05:33.277 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:05:33.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:33.315 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:33.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:05:33.416 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:05:33.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:33.449 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:33.449 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:05:33.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:33 smithi135 ceph-mon[33238]: osdmap e127: 8 total, 8 up, 8 in 2024-09-17T07:05:33.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:33 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:05:33.550 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:05:33.571 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:33.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:33.572 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:05:33.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:33 smithi100 ceph-mon[39462]: osdmap e127: 8 total, 8 up, 8 in 2024-09-17T07:05:33.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:33 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:05:33.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:33 smithi100 ceph-mon[29092]: osdmap e127: 8 total, 8 up, 8 in 2024-09-17T07:05:33.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:33 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:05:33.649 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 62 16 86300 86284 86.9588 53.6875 0.0057331 0.0114861 2024-09-17T07:05:33.674 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:05:33.706 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:33.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:33.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:05:33.808 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:05:33.839 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:33.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:33.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:05:33.941 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:05:33.968 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:33.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:33.969 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:05:34.070 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:05:34.099 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:34.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:34.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:05:34.200 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:05:34.221 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:34.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:34.222 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:05:34.323 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:05:34.364 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:34.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:34.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:05:34.466 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:05:34.508 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:34.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:34.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:05:34.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:34 smithi135 ceph-mon[33238]: pgmap v135: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 151 active+clean; 5.1 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 58 MiB/s wr, 931 op/s; 7156/251463 objects degraded (2.846%); 12800/251463 objects misplaced (5.090%); 15 MiB/s, 243 objects/s recovering 2024-09-17T07:05:34.610 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:05:34.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:34 smithi100 ceph-mon[29092]: pgmap v135: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 151 active+clean; 5.1 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 58 MiB/s wr, 931 op/s; 7156/251463 objects degraded (2.846%); 12800/251463 objects misplaced (5.090%); 15 MiB/s, 243 objects/s recovering 2024-09-17T07:05:34.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:34 smithi100 ceph-mon[39462]: pgmap v135: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 151 active+clean; 5.1 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 58 MiB/s wr, 931 op/s; 7156/251463 objects degraded (2.846%); 12800/251463 objects misplaced (5.090%); 15 MiB/s, 243 objects/s recovering 2024-09-17T07:05:34.642 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:34.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:34.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:05:34.649 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 63 16 87335 87319 86.6053 64.6875 0.0171292 0.0115414 2024-09-17T07:05:34.744 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:05:34.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:34.775 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:34.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:05:34.876 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:05:34.915 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:34.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:34.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:05:35.017 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:05:35.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:35.045 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:35.045 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:05:35.146 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:05:35.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:35.187 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:35.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:05:35.288 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:05:35.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:35.315 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:35.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:05:35.416 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:05:35.450 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:35.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:35.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:05:35.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:35 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 7156/251463 objects degraded (2.846%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T07:05:35.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:35 smithi135 ceph-mon[33238]: osdmap e128: 8 total, 8 up, 8 in 2024-09-17T07:05:35.552 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:05:35.579 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:35.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:35.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:05:35.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:35 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 7156/251463 objects degraded (2.846%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T07:05:35.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:35 smithi100 ceph-mon[29092]: osdmap e128: 8 total, 8 up, 8 in 2024-09-17T07:05:35.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:35 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 7156/251463 objects degraded (2.846%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T07:05:35.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:35 smithi100 ceph-mon[39462]: osdmap e128: 8 total, 8 up, 8 in 2024-09-17T07:05:35.649 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 64 16 88282 88266 86.1769 59.1875 0.0155544 0.0115985 2024-09-17T07:05:35.681 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:05:35.707 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:35.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:35.709 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:05:35.809 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:05:35.832 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:35.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:35.833 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:05:35.934 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:05:35.971 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:35.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:35.972 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:05:36.073 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:05:36.105 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:36.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:36.106 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:05:36.166 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:05:35 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:05:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:05:36.207 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:05:36.237 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:36.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:36.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:05:36.339 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:05:36.368 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:36.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:36.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:05:36.470 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:05:36.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:36.513 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:36.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:05:36.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:36 smithi135 ceph-mon[33238]: pgmap v136: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 153 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 38 MiB/s wr, 612 op/s; 3532/258411 objects degraded (1.367%); 14263/258411 objects misplaced (5.520%); 11 MiB/s, 177 objects/s recovering 2024-09-17T07:05:36.613 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:05:36.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:36 smithi100 ceph-mon[29092]: pgmap v136: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 153 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 38 MiB/s wr, 612 op/s; 3532/258411 objects degraded (1.367%); 14263/258411 objects misplaced (5.520%); 11 MiB/s, 177 objects/s recovering 2024-09-17T07:05:36.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:36 smithi100 ceph-mon[39462]: pgmap v136: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 153 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 38 MiB/s wr, 612 op/s; 3532/258411 objects degraded (1.367%); 14263/258411 objects misplaced (5.520%); 11 MiB/s, 177 objects/s recovering 2024-09-17T07:05:36.650 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 65 16 89153 89137 85.6886 54.4375 0.00341037 0.0116645 2024-09-17T07:05:36.653 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:36.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:36.654 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:05:36.754 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:05:36.785 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:36.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:36.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:05:36.887 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:05:36.914 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:36.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:36.915 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:05:37.016 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:05:37.045 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:37.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:37.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:05:37.147 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:05:37.188 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:37.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:37.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:05:37.290 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:05:37.332 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:37.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:37.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:05:37.434 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:05:37.456 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:37.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:37.457 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:05:37.558 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:05:37.598 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:37.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:37.599 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:05:37.650 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 66 15 90241 90226 85.4215 68.0625 0.00656454 0.0117003 2024-09-17T07:05:37.700 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:05:37.743 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:37.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:37.744 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:05:37.844 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:05:37.865 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:37.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:37.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:05:37.967 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:05:38.007 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:38.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:38.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:05:38.109 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:05:38.151 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:38.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:38.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:05:38.253 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:05:38.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:38.275 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:38.276 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:05:38.377 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:05:38.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:38.409 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:38.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:05:38.510 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:05:38.541 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:38.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:38.542 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:05:38.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:38 smithi135 ceph-mon[33238]: pgmap v138: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 153 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 38 MiB/s wr, 612 op/s; 3532/258411 objects degraded (1.367%); 14263/258411 objects misplaced (5.520%); 11 MiB/s, 177 objects/s recovering 2024-09-17T07:05:38.543 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:05:38 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:05:38.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:38 smithi100 ceph-mon[29092]: pgmap v138: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 153 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 38 MiB/s wr, 612 op/s; 3532/258411 objects degraded (1.367%); 14263/258411 objects misplaced (5.520%); 11 MiB/s, 177 objects/s recovering 2024-09-17T07:05:38.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:38 smithi100 ceph-mon[39462]: pgmap v138: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 153 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 38 MiB/s wr, 612 op/s; 3532/258411 objects degraded (1.367%); 14263/258411 objects misplaced (5.520%); 11 MiB/s, 177 objects/s recovering 2024-09-17T07:05:38.643 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:05:38.650 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 67 16 90995 90979 84.849 47.0625 0.00956085 0.0117668 2024-09-17T07:05:38.677 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:38.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:38.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:05:38.779 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:05:38.805 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:38.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:38.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:05:38.907 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:05:38.931 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:38.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:38.932 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:05:39.033 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:05:39.056 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:39.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:39.058 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:05:39.158 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:05:39.185 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:39.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:39.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:05:39.286 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:05:39.319 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:39.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:39.320 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:05:39.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:39 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:05:39.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:39 smithi100 ceph-mon[29092]: osdmap e129: 8 total, 8 up, 8 in 2024-09-17T07:05:39.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:39 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:05:39.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:39 smithi100 ceph-mon[39462]: osdmap e129: 8 total, 8 up, 8 in 2024-09-17T07:05:39.421 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:05:39.442 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:39.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:39.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:05:39.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:39 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:05:39.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:39 smithi135 ceph-mon[33238]: osdmap e129: 8 total, 8 up, 8 in 2024-09-17T07:05:39.543 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:05:39.573 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:39.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:39.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:05:39.650 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 68 16 91445 91429 84.0149 28.125 0.0107202 0.0118968 2024-09-17T07:05:39.674 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:05:39.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:39.716 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:39.716 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:05:39.817 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:05:39.838 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:39.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:39.838 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:05:39.939 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:05:39.985 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:39.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:39.986 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:05:40.087 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:05:40.123 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:40.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:40.124 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:05:40.224 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:05:40.256 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:40.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:40.257 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:05:40.358 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:05:40.383 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:40.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:40.384 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:05:40.485 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:05:40.506 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:40.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:40.507 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:05:40.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:40 smithi135 ceph-mon[33238]: pgmap v140: 158 pgs: 5 active+recovering+undersized+remapped, 153 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.4 KiB/s rd, 75 MiB/s wr, 1.20k op/s; 16452/272217 objects misplaced (6.044%); 14 MiB/s, 229 objects/s recovering 2024-09-17T07:05:40.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:40 smithi135 ceph-mon[33238]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 3532/258411 objects degraded (1.367%), 1 pg degraded) 2024-09-17T07:05:40.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:40 smithi135 ceph-mon[33238]: osdmap e130: 8 total, 8 up, 8 in 2024-09-17T07:05:40.607 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:05:40.629 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:40 smithi100 ceph-mon[29092]: pgmap v140: 158 pgs: 5 active+recovering+undersized+remapped, 153 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.4 KiB/s rd, 75 MiB/s wr, 1.20k op/s; 16452/272217 objects misplaced (6.044%); 14 MiB/s, 229 objects/s recovering 2024-09-17T07:05:40.629 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:40 smithi100 ceph-mon[29092]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 3532/258411 objects degraded (1.367%), 1 pg degraded) 2024-09-17T07:05:40.629 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:40 smithi100 ceph-mon[29092]: osdmap e130: 8 total, 8 up, 8 in 2024-09-17T07:05:40.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:40 smithi100 ceph-mon[39462]: pgmap v140: 158 pgs: 5 active+recovering+undersized+remapped, 153 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.4 KiB/s rd, 75 MiB/s wr, 1.20k op/s; 16452/272217 objects misplaced (6.044%); 14 MiB/s, 229 objects/s recovering 2024-09-17T07:05:40.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:40 smithi100 ceph-mon[39462]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 3532/258411 objects degraded (1.367%), 1 pg degraded) 2024-09-17T07:05:40.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:40 smithi100 ceph-mon[39462]: osdmap e130: 8 total, 8 up, 8 in 2024-09-17T07:05:40.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:40.640 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:40.640 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:05:40.650 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 69 16 92339 92323 83.607 55.875 0.00795825 0.0119557 2024-09-17T07:05:40.741 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:05:40.771 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:40.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:40.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:05:40.873 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:05:40.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:40.907 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:40.908 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:05:41.008 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:05:41.042 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:41.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:41.043 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:05:41.144 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:05:41.172 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:41.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:41.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:05:41.274 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:05:41.305 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:41.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:41.306 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:05:41.407 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:05:41.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:41.452 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:41.452 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:05:41.554 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:05:41.582 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:41.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:41.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:05:41.650 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 70 16 93275 93259 83.2483 58.5 0.00291223 0.0120045 2024-09-17T07:05:41.683 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:05:41.712 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:41.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:41.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:05:41.814 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:05:41.840 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:41.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:41.841 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:05:41.942 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:05:41.970 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:41.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:41.971 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:05:42.072 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:05:42.093 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:42.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:42.094 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:05:42.194 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:05:42.239 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:42.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:42.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:05:42.341 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:05:42.374 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:42.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:42.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:05:42.476 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:05:42.504 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:42.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:42.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:05:42.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:42 smithi135 ceph-mon[33238]: pgmap v142: 158 pgs: 5 active+recovering+undersized+remapped, 153 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 48 MiB/s wr, 767 op/s; 16452/272217 objects misplaced (6.044%); 5.5 MiB/s, 87 objects/s recovering 2024-09-17T07:05:42.606 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:05:42.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:42 smithi100 ceph-mon[29092]: pgmap v142: 158 pgs: 5 active+recovering+undersized+remapped, 153 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 48 MiB/s wr, 767 op/s; 16452/272217 objects misplaced (6.044%); 5.5 MiB/s, 87 objects/s recovering 2024-09-17T07:05:42.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:42 smithi100 ceph-mon[39462]: pgmap v142: 158 pgs: 5 active+recovering+undersized+remapped, 153 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 48 MiB/s wr, 767 op/s; 16452/272217 objects misplaced (6.044%); 5.5 MiB/s, 87 objects/s recovering 2024-09-17T07:05:42.634 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:42.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:42.635 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:05:42.650 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 71 16 94215 94199 82.9033 58.75 0.00559606 0.0120569 2024-09-17T07:05:42.736 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:05:42.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:42.757 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:42.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:05:42.858 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:05:42.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:42.896 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:42.896 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:05:42.997 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:05:43.027 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:43.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:43.028 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:05:43.129 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:05:43.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:43.151 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:43.151 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:05:43.252 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:05:43.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:43.292 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:43.293 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:05:43.393 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:05:43.414 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:43.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:43.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:05:43.516 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:05:43.555 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:43.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:43.556 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:05:43.650 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 72 16 94800 94784 82.2596 36.5625 0.00556954 0.0121514 2024-09-17T07:05:43.657 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:05:43.686 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:43.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:43.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:05:43.788 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:05:43.815 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:43.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:43.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:05:43.917 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:05:43.938 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:43.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:43.939 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:05:44.040 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:05:44.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:44.078 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:44.078 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:05:44.179 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:05:44.204 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:44.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:44.205 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:05:44.306 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:05:44.334 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:44.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:44.335 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:05:44.436 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:05:44.463 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:44.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:44.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:05:44.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:44 smithi135 ceph-mon[33238]: pgmap v143: 158 pgs: 4 active+recovering+undersized+remapped, 154 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 641 B/s rd, 49 MiB/s wr, 789 op/s; 15920/277299 objects misplaced (5.741%); 6.2 MiB/s, 98 objects/s recovering 2024-09-17T07:05:44.565 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:05:44.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:44.590 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:44.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:05:44.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:44 smithi100 ceph-mon[29092]: pgmap v143: 158 pgs: 4 active+recovering+undersized+remapped, 154 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 641 B/s rd, 49 MiB/s wr, 789 op/s; 15920/277299 objects misplaced (5.741%); 6.2 MiB/s, 98 objects/s recovering 2024-09-17T07:05:44.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:44 smithi100 ceph-mon[39462]: pgmap v143: 158 pgs: 4 active+recovering+undersized+remapped, 154 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 641 B/s rd, 49 MiB/s wr, 789 op/s; 15920/277299 objects misplaced (5.741%); 6.2 MiB/s, 98 objects/s recovering 2024-09-17T07:05:44.650 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 73 16 95560 95544 81.7835 47.5 0.00359059 0.012209 2024-09-17T07:05:44.691 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:05:44.720 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:44.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:44.721 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:05:44.822 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:05:44.849 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:44.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:44.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:05:44.952 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:05:44.986 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:44.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:44.987 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:05:45.087 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:05:45.118 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:45.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:45.119 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:05:45.219 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:05:45.246 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:45.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:45.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:05:45.348 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:05:45.370 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:45.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:45.371 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:05:45.472 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:05:45.515 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:45.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:45.516 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:05:45.617 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:05:45.640 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:45.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:45.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:05:45.650 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 74 16 96394 96378 81.3826 52.125 0.00756388 0.0122781 2024-09-17T07:05:45.743 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:05:45.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:45.776 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:45.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:05:45.877 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:05:45.899 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:45.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:45.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:05:46.001 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:05:46.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:46.042 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:46.042 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:05:46.143 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:05:46.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:46.169 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:46.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:05:46.270 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:05:46.299 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:46.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:46.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:05:46.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:46 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:05:46.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:46 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:05:46.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:46 smithi100 ceph-mon[29092]: pgmap v144: 158 pgs: 4 active+recovering+undersized+remapped, 154 active+clean; 5.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 67 MiB/s wr, 1.08k op/s; 14171/284283 objects misplaced (4.985%); 11 MiB/s, 182 objects/s recovering 2024-09-17T07:05:46.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:46 smithi100 ceph-mon[29092]: osdmap e131: 8 total, 8 up, 8 in 2024-09-17T07:05:46.377 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:05:45 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:05:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:05:46.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:46 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:05:46.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:46 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:05:46.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:46 smithi100 ceph-mon[39462]: pgmap v144: 158 pgs: 4 active+recovering+undersized+remapped, 154 active+clean; 5.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 67 MiB/s wr, 1.08k op/s; 14171/284283 objects misplaced (4.985%); 11 MiB/s, 182 objects/s recovering 2024-09-17T07:05:46.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:46 smithi100 ceph-mon[39462]: osdmap e131: 8 total, 8 up, 8 in 2024-09-17T07:05:46.402 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:05:46.433 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:46.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:46.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:05:46.535 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:05:46.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:46 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:05:46.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:46 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:05:46.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:46 smithi135 ceph-mon[33238]: pgmap v144: 158 pgs: 4 active+recovering+undersized+remapped, 154 active+clean; 5.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 67 MiB/s wr, 1.08k op/s; 14171/284283 objects misplaced (4.985%); 11 MiB/s, 182 objects/s recovering 2024-09-17T07:05:46.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:46 smithi135 ceph-mon[33238]: osdmap e131: 8 total, 8 up, 8 in 2024-09-17T07:05:46.573 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:46.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:46.574 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:05:46.650 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 75 16 97254 97238 81.0142 53.75 0.0122799 0.0123388 2024-09-17T07:05:46.675 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:05:46.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:46.707 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:46.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:05:46.807 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:05:46.847 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:46.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:46.848 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:05:46.949 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:05:46.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:46.988 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:46.988 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:05:47.089 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:05:47.119 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:47.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:47.120 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:05:47.221 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:05:47.244 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:47.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:47.246 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:05:47.346 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:05:47.384 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:47.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:47.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:05:47.486 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:05:47.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:47.513 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:47.514 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:05:47.614 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:05:47.650 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 76 16 98168 98152 80.6998 57.125 0.0124447 0.0123856 2024-09-17T07:05:47.653 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:47.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:47.654 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:05:47.755 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:05:47.776 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:47.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:47.777 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:05:47.878 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:05:47.906 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:47.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:47.907 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:05:48.008 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:05:48.042 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:48.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:48.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:05:48.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:48 smithi100 ceph-mon[29092]: pgmap v146: 158 pgs: 4 active+recovering+undersized+remapped, 154 active+clean; 5.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 31 MiB/s wr, 503 op/s; 14171/284283 objects misplaced (4.985%); 7.3 MiB/s, 116 objects/s recovering 2024-09-17T07:05:48.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:48 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:05:48.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:48 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:05:48.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:48 smithi100 ceph-mon[39462]: pgmap v146: 158 pgs: 4 active+recovering+undersized+remapped, 154 active+clean; 5.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 31 MiB/s wr, 503 op/s; 14171/284283 objects misplaced (4.985%); 7.3 MiB/s, 116 objects/s recovering 2024-09-17T07:05:48.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:48 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:05:48.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:48 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:05:48.144 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:05:48.176 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:48.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:48.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:05:48.278 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:05:48.307 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:48.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:48.308 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:05:48.409 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:05:48.457 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:48.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:48.458 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:05:48.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:48 smithi135 ceph-mon[33238]: pgmap v146: 158 pgs: 4 active+recovering+undersized+remapped, 154 active+clean; 5.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 31 MiB/s wr, 503 op/s; 14171/284283 objects misplaced (4.985%); 7.3 MiB/s, 116 objects/s recovering 2024-09-17T07:05:48.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:48 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:05:48.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:48 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:05:48.544 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:05:48 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:05:48.559 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:05:48.601 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:48.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:48.602 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:05:48.651 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 77 16 99041 99025 80.3604 54.5625 0.00718591 0.0124382 2024-09-17T07:05:48.703 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:05:48.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:48.724 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:48.725 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:05:48.825 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:05:48.848 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:48.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:48.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:05:48.950 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:05:48.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:49.000 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:49.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:05:49.101 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:05:49.128 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:49.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:49.129 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:05:49.230 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:05:49.263 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:49.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:49.264 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:05:49.365 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:05:49.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:49 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:05:49.376 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:49 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:05:49.392 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:49.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:49.393 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:05:49.494 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:05:49.532 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:49.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:49.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:05:49.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:49 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:05:49.633 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:05:49.651 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 78 16 99876 99860 79.9992 52.1875 0.00565398 0.012495 2024-09-17T07:05:49.661 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:49.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:49.662 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:05:49.763 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:05:49.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:49.806 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:49.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:05:49.907 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:05:49.938 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:49.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:49.939 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:05:50.040 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:05:50.069 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:50.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:50.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:05:50.171 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:05:50.217 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:50.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:50.218 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:05:50.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:50 smithi135 ceph-mon[33238]: pgmap v147: 158 pgs: 4 active+recovering+undersized+remapped, 154 active+clean; 6.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.0 KiB/s rd, 51 MiB/s wr, 824 op/s; 12417/296568 objects misplaced (4.187%); 10 MiB/s, 163 objects/s recovering 2024-09-17T07:05:50.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:50 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:05:50.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:50 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:05:50.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:50 smithi135 ceph-mon[33238]: osdmap e132: 8 total, 8 up, 8 in 2024-09-17T07:05:50.319 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:05:50.348 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:50.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:50.349 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:05:50.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:50 smithi100 ceph-mon[29092]: pgmap v147: 158 pgs: 4 active+recovering+undersized+remapped, 154 active+clean; 6.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.0 KiB/s rd, 51 MiB/s wr, 824 op/s; 12417/296568 objects misplaced (4.187%); 10 MiB/s, 163 objects/s recovering 2024-09-17T07:05:50.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:50 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:05:50.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:50 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:05:50.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:50 smithi100 ceph-mon[29092]: osdmap e132: 8 total, 8 up, 8 in 2024-09-17T07:05:50.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:50 smithi100 ceph-mon[39462]: pgmap v147: 158 pgs: 4 active+recovering+undersized+remapped, 154 active+clean; 6.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.0 KiB/s rd, 51 MiB/s wr, 824 op/s; 12417/296568 objects misplaced (4.187%); 10 MiB/s, 163 objects/s recovering 2024-09-17T07:05:50.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:50 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:05:50.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:50 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:05:50.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:50 smithi100 ceph-mon[39462]: osdmap e132: 8 total, 8 up, 8 in 2024-09-17T07:05:50.451 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:05:50.478 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:50.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:50.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:05:50.580 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:05:50.621 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:50.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:50.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:05:50.651 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 79 16 100904 100888 79.7997 64.25 0.0108824 0.0125262 2024-09-17T07:05:50.723 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:05:50.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:50.745 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:50.745 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:05:50.846 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:05:50.886 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:50.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:50.887 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:05:50.988 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:05:51.026 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:51.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:51.027 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:05:51.128 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:05:51.166 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:51.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:51.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:05:51.268 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:05:51.306 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:51.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:51.307 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:05:51.408 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:05:51.437 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:51.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:51.438 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:05:51.539 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:05:51.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:51.560 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:51.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:05:51.651 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:2024-09-17T07:05:51.651644+0000 min lat: 0.00128777 max lat: 2.09639 avg lat: 0.0125771 2024-09-17T07:05:51.651 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-17T07:05:51.651 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 80 16 101768 101752 79.4772 54 0.0029628 0.0125771 2024-09-17T07:05:51.662 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:05:51.700 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:51.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:51.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:05:51.802 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:05:51.829 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:51.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:51.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:05:51.931 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:05:51.960 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:51.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:51.961 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:05:52.062 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:05:52.099 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:52.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:52.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:05:52.201 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:05:52.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:52.242 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:52.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:05:52.343 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:05:52.369 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:52.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:52.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:05:52.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:52 smithi100 ceph-mon[29092]: pgmap v149: 158 pgs: 4 active+recovering+undersized+remapped, 154 active+clean; 6.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 50 MiB/s wr, 803 op/s; 12417/296568 objects misplaced (4.187%); 10 MiB/s, 167 objects/s recovering 2024-09-17T07:05:52.376 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:52 smithi100 ceph-mon[39462]: pgmap v149: 158 pgs: 4 active+recovering+undersized+remapped, 154 active+clean; 6.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 50 MiB/s wr, 803 op/s; 12417/296568 objects misplaced (4.187%); 10 MiB/s, 167 objects/s recovering 2024-09-17T07:05:52.471 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:05:52.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:52.512 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:52.512 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:05:52.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:52 smithi135 ceph-mon[33238]: pgmap v149: 158 pgs: 4 active+recovering+undersized+remapped, 154 active+clean; 6.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 50 MiB/s wr, 803 op/s; 12417/296568 objects misplaced (4.187%); 10 MiB/s, 167 objects/s recovering 2024-09-17T07:05:52.614 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:05:52.643 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:52.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:52.644 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:05:52.651 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 81 16 102737 102721 79.2437 60.5625 0.0126789 0.0126133 2024-09-17T07:05:52.744 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:05:52.775 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:52.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:52.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:05:52.877 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:05:52.898 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:52.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:52.899 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:05:53.000 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:05:53.023 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:53.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:53.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:05:53.125 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:05:53.166 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:53.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:53.167 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:05:53.268 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:05:53.305 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:53.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:53.306 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:05:53.407 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:05:53.433 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:53.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:53.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:05:53.535 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:05:53.572 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:53.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:53.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:05:53.651 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 82 16 103619 103603 78.9495 55.125 0.00355453 0.0126609 2024-09-17T07:05:53.674 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:05:53.715 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:53.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:53.716 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:05:53.817 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:05:53.856 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:53.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:53.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:05:53.958 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:05:53.983 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:53.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:53.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:05:54.085 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:05:54.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:54.125 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:54.125 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:05:54.226 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:05:54.264 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:54.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:54.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:05:54.366 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:05:54.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:54 smithi100 ceph-mon[29092]: pgmap v150: 158 pgs: 4 active+recovering+undersized+remapped, 154 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 46 MiB/s wr, 742 op/s; 12417/302106 objects misplaced (4.110%); 5.2 MiB/s, 83 objects/s recovering 2024-09-17T07:05:54.376 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:54 smithi100 ceph-mon[39462]: pgmap v150: 158 pgs: 4 active+recovering+undersized+remapped, 154 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 46 MiB/s wr, 742 op/s; 12417/302106 objects misplaced (4.110%); 5.2 MiB/s, 83 objects/s recovering 2024-09-17T07:05:54.394 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:54.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:54.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:05:54.496 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:05:54.537 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:54.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:54.538 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:05:54.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:54 smithi135 ceph-mon[33238]: pgmap v150: 158 pgs: 4 active+recovering+undersized+remapped, 154 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 46 MiB/s wr, 742 op/s; 12417/302106 objects misplaced (4.110%); 5.2 MiB/s, 83 objects/s recovering 2024-09-17T07:05:54.638 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:05:54.651 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 83 16 104626 104610 78.7564 62.9375 0.0152202 0.0126922 2024-09-17T07:05:54.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:54.707 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:54.708 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:05:54.808 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:05:54.852 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:54.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:54.853 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:05:54.954 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:05:54.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:54.993 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:54.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:05:55.094 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:05:55.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:55.141 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:55.141 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:05:55.242 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:05:55.285 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:55.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:55.286 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:05:55.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:55 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:05:55.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:55 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:05:55.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:55 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:05:55.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:55 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:05:55.387 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:05:55.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:55.414 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:55.414 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:05:55.515 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:05:55.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:55 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:05:55.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:55 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:05:55.557 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:55.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:55.560 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:05:55.652 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 84 16 105577 105561 78.5259 59.4375 0.0127941 0.0127291 2024-09-17T07:05:55.661 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:05:55.688 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:55.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:55.689 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:05:55.790 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:05:55.818 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:55.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:55.819 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:05:55.920 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:05:55.968 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:55.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:55.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:05:56.048 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:05:55 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:05:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:05:56.073 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:05:56.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:56.102 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:56.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:05:56.203 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:05:56.229 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:56.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:56.230 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:05:56.317 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:56 smithi100 ceph-mon[29092]: pgmap v151: 158 pgs: 4 active+recovering+undersized+remapped, 154 active+clean; 6.3 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.0 KiB/s rd, 55 MiB/s wr, 888 op/s; 10546/310452 objects misplaced (3.397%); 8.9 MiB/s, 141 objects/s recovering 2024-09-17T07:05:56.317 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:56 smithi100 ceph-mon[29092]: osdmap e133: 8 total, 8 up, 8 in 2024-09-17T07:05:56.317 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:56 smithi100 ceph-mon[39462]: pgmap v151: 158 pgs: 4 active+recovering+undersized+remapped, 154 active+clean; 6.3 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.0 KiB/s rd, 55 MiB/s wr, 888 op/s; 10546/310452 objects misplaced (3.397%); 8.9 MiB/s, 141 objects/s recovering 2024-09-17T07:05:56.317 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:56 smithi100 ceph-mon[39462]: osdmap e133: 8 total, 8 up, 8 in 2024-09-17T07:05:56.331 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:05:56.360 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:56.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:56.361 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:05:56.462 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:05:56.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:56.510 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:56.511 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:05:56.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:56 smithi135 ceph-mon[33238]: pgmap v151: 158 pgs: 4 active+recovering+undersized+remapped, 154 active+clean; 6.3 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.0 KiB/s rd, 55 MiB/s wr, 888 op/s; 10546/310452 objects misplaced (3.397%); 8.9 MiB/s, 141 objects/s recovering 2024-09-17T07:05:56.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:56 smithi135 ceph-mon[33238]: osdmap e133: 8 total, 8 up, 8 in 2024-09-17T07:05:56.612 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:05:56.637 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:56.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:56.638 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:05:56.652 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 85 15 106635 106620 78.3807 66.1875 0.00775597 0.0127535 2024-09-17T07:05:56.739 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:05:56.765 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:56.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:56.766 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:05:56.867 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:05:56.894 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:56.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:56.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:05:56.962 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-17T07:05:56.962 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-17T07:05:56.962 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:05:56.962 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.14805.0:106860 7.e 7:7ffc5b06:::benchmark_data_smithi100_78556_object106859:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e133)", 2024-09-17T07:05:56.962 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:05:56.927035+0000", 2024-09-17T07:05:56.962 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.033595344999999999, 2024-09-17T07:05:56.962 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033633631999999997, 2024-09-17T07:05:56.963 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:05:56.963 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "waiting for sub ops", 2024-09-17T07:05:56.963 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:05:56.963 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.14805", 2024-09-17T07:05:56.963 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:05:56.963 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 106860 2024-09-17T07:05:56.963 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:05:56.963 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:05:56.963 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:05:56.963 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:05:56.963 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:56.927035+0000", 2024-09-17T07:05:56.963 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:05:56.964 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:05:56.964 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:05:56.964 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:05:56.964 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:56.927035+0000", 2024-09-17T07:05:56.964 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:05:56.964 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:05:56.964 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:05:56.964 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:05:56.964 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:56.927033+0000", 2024-09-17T07:05:56.964 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999981 2024-09-17T07:05:56.964 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:05:56.965 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:05:56.965 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:05:56.965 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:56.927057+0000", 2024-09-17T07:05:56.965 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4431999999999999e-05 2024-09-17T07:05:56.965 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:05:56.965 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:05:56.965 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:05:56.965 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:56.927070+0000", 2024-09-17T07:05:56.965 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2371000000000001e-05 2024-09-17T07:05:56.965 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:05:56.965 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:05:56.965 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:05:56.965 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:56.927077+0000", 2024-09-17T07:05:56.966 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6179999999999997e-06 2024-09-17T07:05:56.966 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:05:56.966 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:05:56.966 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:05:56.966 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:56.927235+0000", 2024-09-17T07:05:56.966 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00015810999999999999 2024-09-17T07:05:56.966 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:05:56.966 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:05:56.966 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:05:56.966 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:56.927357+0000", 2024-09-17T07:05:56.966 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012156599999999999 2024-09-17T07:05:56.966 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:05:56.967 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:05:56.967 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 0,5", 2024-09-17T07:05:56.967 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:56.927515+0000", 2024-09-17T07:05:56.967 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00015837100000000001 2024-09-17T07:05:56.967 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:05:56.967 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:05:56.967 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:05:56.967 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:56.931944+0000", 2024-09-17T07:05:56.967 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0044291290000000004 2024-09-17T07:05:56.968 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:05:56.968 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:05:56.968 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:05:56.968 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:56.937164+0000", 2024-09-17T07:05:56.968 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0052192009999999997 2024-09-17T07:05:56.968 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:05:56.968 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:05:56.968 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:05:56.968 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:05:56.968 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:05:56.968 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.14805.0:106890 7.13 7:c8bcb406:::benchmark_data_smithi100_78556_object106889:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e133)", 2024-09-17T07:05:56.968 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:05:56.959255+0000", 2024-09-17T07:05:56.968 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.0013750259999999999, 2024-09-17T07:05:56.969 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0015012269999999999, 2024-09-17T07:05:56.969 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:05:56.969 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "waiting for sub ops", 2024-09-17T07:05:56.969 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:05:56.969 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.14805", 2024-09-17T07:05:56.969 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:05:56.969 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 106890 2024-09-17T07:05:56.969 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:05:56.969 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:05:56.969 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:05:56.969 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:05:56.969 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:56.959255+0000", 2024-09-17T07:05:56.969 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:05:56.970 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:05:56.970 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:05:56.970 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:05:56.970 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:56.959255+0000", 2024-09-17T07:05:56.970 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:05:56.970 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:05:56.970 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:05:56.970 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:05:56.970 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:56.959253+0000", 2024-09-17T07:05:56.970 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999981 2024-09-17T07:05:56.970 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:05:56.970 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:05:56.970 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:05:56.971 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:56.959279+0000", 2024-09-17T07:05:56.971 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.6166999999999999e-05 2024-09-17T07:05:56.971 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:05:56.971 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:05:56.971 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:05:56.971 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:56.959297+0000", 2024-09-17T07:05:56.971 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7822999999999999e-05 2024-09-17T07:05:56.971 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:05:56.971 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:05:56.971 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:05:56.971 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:56.959305+0000", 2024-09-17T07:05:56.971 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1829999999999994e-06 2024-09-17T07:05:56.971 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:05:56.972 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:05:56.972 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:05:56.972 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:56.959382+0000", 2024-09-17T07:05:56.972 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6633999999999998e-05 2024-09-17T07:05:56.972 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:05:56.972 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:05:56.972 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:05:56.972 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:56.959544+0000", 2024-09-17T07:05:56.972 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016242200000000001 2024-09-17T07:05:56.972 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:05:56.972 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:05:56.972 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 0,7", 2024-09-17T07:05:56.973 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:56.959703+0000", 2024-09-17T07:05:56.973 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00015872600000000001 2024-09-17T07:05:56.973 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:05:56.973 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:05:56.973 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:05:56.973 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:05:56.973 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-17T07:05:56.973 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 2 2024-09-17T07:05:56.973 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-17T07:05:56.996 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:05:57.022 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:57.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:57.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:05:57.023 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:05:57.024 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1: {} 2024-09-17T07:05:57.024 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:05:57.024 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:05:57.024 INFO:teuthology.orchestra.run.smithi100.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:05:57.024 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:05:57.055 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:05:57.056 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:05:57.086 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:05:57.086 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:05:57.115 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:05:57.115 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:05:57.123 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:05:57.150 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:05:57.151 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:05:57.169 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:57.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:57.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:05:57.186 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:57 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/932066017' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:05:57.187 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:57 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/932066017' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:05:57.187 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:05:57.187 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:05:57.273 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:05:57.301 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:57.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:57.302 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:05:57.319 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T07:05:57.403 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:05:57.440 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:57.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:57.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:05:57.542 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:05:57.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:57 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/932066017' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:05:57.577 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:57.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:57.578 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:05:57.652 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 86 16 107360 107344 77.9955 45.25 0.00618509 0.0127806 2024-09-17T07:05:57.679 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:05:57.707 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:57.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:57.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:05:57.808 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:05:57.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:57.831 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:57.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:05:57.932 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:05:57.954 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:57.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:57.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:05:57.962 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-17T07:05:58.055 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:05:58.077 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:58.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:58.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:05:58.179 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:05:58.229 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:58.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:58.230 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:05:58.309 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:58 smithi100 ceph-mon[29092]: pgmap v153: 158 pgs: 4 active+recovering+undersized+remapped, 154 active+clean; 6.3 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 36 MiB/s wr, 578 op/s; 10546/310452 objects misplaced (3.397%); 5.6 MiB/s, 90 objects/s recovering 2024-09-17T07:05:58.309 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:58 smithi100 ceph-mon[29092]: osdmap e134: 8 total, 8 up, 8 in 2024-09-17T07:05:58.309 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:58 smithi100 ceph-mon[39462]: pgmap v153: 158 pgs: 4 active+recovering+undersized+remapped, 154 active+clean; 6.3 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 36 MiB/s wr, 578 op/s; 10546/310452 objects misplaced (3.397%); 5.6 MiB/s, 90 objects/s recovering 2024-09-17T07:05:58.310 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:58 smithi100 ceph-mon[39462]: osdmap e134: 8 total, 8 up, 8 in 2024-09-17T07:05:58.331 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:05:58.356 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:58.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:58.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:05:58.458 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:05:58.482 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:58.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:58.483 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:05:58.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:58 smithi135 ceph-mon[33238]: pgmap v153: 158 pgs: 4 active+recovering+undersized+remapped, 154 active+clean; 6.3 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 36 MiB/s wr, 578 op/s; 10546/310452 objects misplaced (3.397%); 5.6 MiB/s, 90 objects/s recovering 2024-09-17T07:05:58.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:58 smithi135 ceph-mon[33238]: osdmap e134: 8 total, 8 up, 8 in 2024-09-17T07:05:58.544 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:05:58 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:05:58.584 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:05:58.625 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:58.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:58.626 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:05:58.652 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 87 16 107820 107804 77.4295 28.75 0.00843606 0.0129101 2024-09-17T07:05:58.727 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:05:58.756 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:58.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:58.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:05:58.858 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:05:58.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:58.902 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:58.902 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:05:59.003 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:05:59.032 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:59.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:59.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:05:59.134 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:05:59.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:59.166 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:59.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:05:59.267 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:05:59.282 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:59 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:05:59.282 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:59 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:05:59.283 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:59 smithi100 ceph-mon[29092]: osdmap e135: 8 total, 8 up, 8 in 2024-09-17T07:05:59.283 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:05:59 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:05:59.283 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:59 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:05:59.283 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:59 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:05:59.283 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:59 smithi100 ceph-mon[39462]: osdmap e135: 8 total, 8 up, 8 in 2024-09-17T07:05:59.283 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:05:59 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:05:59.294 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:59.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:59.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:05:59.395 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:05:59.453 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:59.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:59.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:05:59.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:59 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:05:59.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:59 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:05:59.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:59 smithi135 ceph-mon[33238]: osdmap e135: 8 total, 8 up, 8 in 2024-09-17T07:05:59.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:05:59 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:05:59.555 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:05:59.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:59.579 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:05:59.579 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:05:59.652 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 88 16 108911 108895 77.3244 68.1875 0.0151539 0.0129276 2024-09-17T07:05:59.680 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:05:59.707 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:59.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:59.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:05:59.809 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:05:59.837 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:59.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:59.838 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:05:59.939 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:05:59.966 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:05:59.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:05:59.967 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:06:00.068 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:06:00.117 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:00.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:00.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:06:00.219 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:06:00.256 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:00.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:00.257 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:06:00.357 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:06:00.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:00 smithi100 ceph-mon[29092]: pgmap v156: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 6.6 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.7 KiB/s rd, 74 MiB/s wr, 1.18k op/s; 7911/323361 objects misplaced (2.446%); 18 MiB/s, 282 objects/s recovering 2024-09-17T07:06:00.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:00 smithi100 ceph-mon[29092]: osdmap e136: 8 total, 8 up, 8 in 2024-09-17T07:06:00.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:00 smithi100 ceph-mon[39462]: pgmap v156: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 6.6 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.7 KiB/s rd, 74 MiB/s wr, 1.18k op/s; 7911/323361 objects misplaced (2.446%); 18 MiB/s, 282 objects/s recovering 2024-09-17T07:06:00.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:00 smithi100 ceph-mon[39462]: osdmap e136: 8 total, 8 up, 8 in 2024-09-17T07:06:00.395 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:00.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:00.396 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:06:00.497 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:06:00.537 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:00.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:00.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:06:00.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:00 smithi135 ceph-mon[33238]: pgmap v156: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 6.6 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.7 KiB/s rd, 74 MiB/s wr, 1.18k op/s; 7911/323361 objects misplaced (2.446%); 18 MiB/s, 282 objects/s recovering 2024-09-17T07:06:00.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:00 smithi135 ceph-mon[33238]: osdmap e136: 8 total, 8 up, 8 in 2024-09-17T07:06:00.638 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:06:00.652 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 89 16 109898 109882 77.1485 61.6875 0.0186724 0.0129569 2024-09-17T07:06:00.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:00.681 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:00.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:06:00.782 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:06:00.828 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:00.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:00.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:06:00.931 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:06:00.958 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:00.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:00.959 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:06:01.060 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:06:01.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:01.090 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:01.090 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:06:01.167 INFO:teuthology.orchestra.run.smithi100.stderr:reweighted osd.1 to 1 (10000) 2024-09-17T07:06:01.190 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:06:01.214 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:01.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:01.215 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:06:01.316 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:06:01.324 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:06:01.338 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:01.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:01.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:06:01.439 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:06:01.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:01.470 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:01.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:06:01.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:01 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/62475408' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-17T07:06:01.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:01 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-17T07:06:01.570 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:06:01.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:01.592 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:01.592 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:06:01.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:01 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/62475408' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-17T07:06:01.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:01 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-17T07:06:01.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:01 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/62475408' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-17T07:06:01.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:01 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-17T07:06:01.653 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Total time run: 90.0092 2024-09-17T07:06:01.653 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Total writes made: 110800 2024-09-17T07:06:01.653 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Write size: 65536 2024-09-17T07:06:01.653 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Object size: 65536 2024-09-17T07:06:01.653 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Bandwidth (MB/sec): 76.9366 2024-09-17T07:06:01.653 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Stddev Bandwidth: 42.9964 2024-09-17T07:06:01.653 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Max bandwidth (MB/sec): 219.562 2024-09-17T07:06:01.654 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Min bandwidth (MB/sec): 0 2024-09-17T07:06:01.654 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Average IOPS: 1230 2024-09-17T07:06:01.654 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Stddev IOPS: 687.952 2024-09-17T07:06:01.654 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Max IOPS: 3513 2024-09-17T07:06:01.654 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Min IOPS: 0 2024-09-17T07:06:01.654 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Average Latency(s): 0.0129934 2024-09-17T07:06:01.654 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Stddev Latency(s): 0.0371538 2024-09-17T07:06:01.654 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Max latency(s): 2.09639 2024-09-17T07:06:01.654 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Min latency(s): 0.00128777 2024-09-17T07:06:01.658 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Cleaning up (deleting benchmark objects) 2024-09-17T07:06:01.693 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:06:01.721 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:01.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:01.722 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:06:01.823 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:06:01.846 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:01.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:01.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:06:01.948 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:06:01.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:01.987 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:01.987 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:06:02.088 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:06:02.115 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:02.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:02.116 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:06:02.217 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:06:02.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:02.241 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:02.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:06:02.341 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:06:02.369 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:02.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:02.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:06:02.471 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:06:02.493 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:02.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:02.494 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:06:02.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:02 smithi135 ceph-mon[33238]: pgmap v158: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 6.6 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 861 B/s rd, 45 MiB/s wr, 724 op/s; 7911/323361 objects misplaced (2.446%); 10 MiB/s, 164 objects/s recovering 2024-09-17T07:06:02.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:02 smithi135 ceph-mon[33238]: Health check failed: Degraded data redundancy: 2 pgs undersized (PG_DEGRADED) 2024-09-17T07:06:02.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:02 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-17T07:06:02.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:02 smithi135 ceph-mon[33238]: osdmap e137: 8 total, 8 up, 8 in 2024-09-17T07:06:02.595 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:06:02.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:02 smithi100 ceph-mon[29092]: pgmap v158: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 6.6 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 861 B/s rd, 45 MiB/s wr, 724 op/s; 7911/323361 objects misplaced (2.446%); 10 MiB/s, 164 objects/s recovering 2024-09-17T07:06:02.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:02 smithi100 ceph-mon[29092]: Health check failed: Degraded data redundancy: 2 pgs undersized (PG_DEGRADED) 2024-09-17T07:06:02.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:02 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-17T07:06:02.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:02 smithi100 ceph-mon[29092]: osdmap e137: 8 total, 8 up, 8 in 2024-09-17T07:06:02.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:02 smithi100 ceph-mon[39462]: pgmap v158: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 6.6 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 861 B/s rd, 45 MiB/s wr, 724 op/s; 7911/323361 objects misplaced (2.446%); 10 MiB/s, 164 objects/s recovering 2024-09-17T07:06:02.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:02 smithi100 ceph-mon[39462]: Health check failed: Degraded data redundancy: 2 pgs undersized (PG_DEGRADED) 2024-09-17T07:06:02.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:02 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-17T07:06:02.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:02 smithi100 ceph-mon[39462]: osdmap e137: 8 total, 8 up, 8 in 2024-09-17T07:06:02.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:02.633 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:02.633 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:06:02.734 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:06:02.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:02.762 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:02.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:06:02.863 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:06:02.893 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:02.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:02.894 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:06:02.995 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:06:03.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:03.037 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:03.038 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:06:03.140 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:06:03.171 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:03.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:03.172 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:06:03.274 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:06:03.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:03.317 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:03.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:06:03.418 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:06:03.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:03.475 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:03.476 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:06:03.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:03 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:06:03.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:03 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:06:03.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:03 smithi135 ceph-mon[33238]: pgmap v160: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.0 KiB/s rd, 60 MiB/s wr, 1.05k op/s; 7911/325629 objects misplaced (2.429%); 10 MiB/s, 164 objects/s recovering 2024-09-17T07:06:03.576 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:06:03.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:03.607 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:03.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:06:03.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:03 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:06:03.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:03 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:06:03.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:03 smithi100 ceph-mon[29092]: pgmap v160: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.0 KiB/s rd, 60 MiB/s wr, 1.05k op/s; 7911/325629 objects misplaced (2.429%); 10 MiB/s, 164 objects/s recovering 2024-09-17T07:06:03.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:03 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:06:03.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:03 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:06:03.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:03 smithi100 ceph-mon[39462]: pgmap v160: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.0 KiB/s rd, 60 MiB/s wr, 1.05k op/s; 7911/325629 objects misplaced (2.429%); 10 MiB/s, 164 objects/s recovering 2024-09-17T07:06:03.708 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:06:03.734 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:03.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:03.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:06:03.836 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:06:03.872 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:03.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:03.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:06:03.974 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:06:04.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:04.017 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:04.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:06:04.118 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:06:04.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:04.157 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:04.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:06:04.178 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd reweight 2 1 2024-09-17T07:06:04.252 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:06:04.258 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:06:04.312 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:06:04.333 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:04.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:04.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:06:04.369 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:06:04.436 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:06:04.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:04.481 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:04.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:06:04.582 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:06:04.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:04.607 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:04.608 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:06:04.708 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:06:04.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:04.761 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:04.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:06:04.861 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:06:04.892 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:04.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:04.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:06:04.994 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:06:05.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:05.030 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:05.031 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:06:05.131 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:06:05.177 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:05.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:05.178 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:06:05.279 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:06:05.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:05.322 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:05.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:06:05.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:05 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:06:05.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:05 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:06:05.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:05 smithi100 ceph-mon[29092]: osdmap e138: 8 total, 8 up, 8 in 2024-09-17T07:06:05.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:05 smithi100 ceph-mon[29092]: pgmap v162: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 34 MiB/s wr, 837 op/s; 6063/327621 objects misplaced (1.851%); 6.4 MiB/s, 102 objects/s recovering 2024-09-17T07:06:05.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:05 smithi100 ceph-mon[29092]: osdmap e139: 8 total, 8 up, 8 in 2024-09-17T07:06:05.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:05 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:06:05.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:05 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:06:05.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:05 smithi100 ceph-mon[39462]: osdmap e138: 8 total, 8 up, 8 in 2024-09-17T07:06:05.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:05 smithi100 ceph-mon[39462]: pgmap v162: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 34 MiB/s wr, 837 op/s; 6063/327621 objects misplaced (1.851%); 6.4 MiB/s, 102 objects/s recovering 2024-09-17T07:06:05.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:05 smithi100 ceph-mon[39462]: osdmap e139: 8 total, 8 up, 8 in 2024-09-17T07:06:05.423 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:06:05.449 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:05.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:05.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:06:05.551 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:06:05.579 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:05.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:05.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:06:05.681 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:06:05.703 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:05.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:05.704 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:06:05.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:05 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:06:05.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:05 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:06:05.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:05 smithi135 ceph-mon[33238]: osdmap e138: 8 total, 8 up, 8 in 2024-09-17T07:06:05.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:05 smithi135 ceph-mon[33238]: pgmap v162: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 34 MiB/s wr, 837 op/s; 6063/327621 objects misplaced (1.851%); 6.4 MiB/s, 102 objects/s recovering 2024-09-17T07:06:05.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:05 smithi135 ceph-mon[33238]: osdmap e139: 8 total, 8 up, 8 in 2024-09-17T07:06:05.805 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:06:05.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:05.832 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:05.832 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:06:05.933 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:06:05.955 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:05.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:05.956 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:06:06.056 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:06:06.088 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:06.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:06.089 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:06:06.190 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:06:06.217 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:06.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:06.218 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:06:06.319 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:06:06.357 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:06.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:06.359 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:06:06.378 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:06:05 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:06:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:06:06.459 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:06:06.486 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:06.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:06.487 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:06:06.588 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:06:06.615 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:06.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:06.616 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:06:06.716 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:06:06.753 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:06.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:06.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:06:06.855 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:06:06.893 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:06.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:06.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:06:06.995 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:06:07.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:07.032 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:07.032 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:06:07.133 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:06:07.161 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:07.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:07.162 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:06:07.263 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:06:07.295 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:07.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:07.296 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:06:07.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:07 smithi100 ceph-mon[29092]: osdmap e140: 8 total, 8 up, 8 in 2024-09-17T07:06:07.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:07 smithi100 ceph-mon[39462]: osdmap e140: 8 total, 8 up, 8 in 2024-09-17T07:06:07.397 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:06:07.425 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:07.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:07.426 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:06:07.527 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:06:07.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:07 smithi135 ceph-mon[33238]: osdmap e140: 8 total, 8 up, 8 in 2024-09-17T07:06:07.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:07.569 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:07.569 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:06:07.670 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:06:07.708 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:07.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:07.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:06:07.809 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:06:07.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:07.849 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:07.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:06:07.951 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:06:07.977 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:07.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:07.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:06:08.079 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:06:08.107 INFO:teuthology.orchestra.run.smithi100.stderr:noscrub is unset 2024-09-17T07:06:08.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:08.154 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:08.155 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:06:08.255 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:06:08.301 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:08.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:08.302 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:06:08.376 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:08 smithi100 ceph-mon[39462]: pgmap v165: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.0 KiB/s rd, 34 MiB/s wr, 852 op/s; 6063/327621 objects misplaced (1.851%); 6.5 MiB/s, 104 objects/s recovering 2024-09-17T07:06:08.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:08 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/3190428803' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:06:08.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:08 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:06:08.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:08 smithi100 ceph-mon[29092]: pgmap v165: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.0 KiB/s rd, 34 MiB/s wr, 852 op/s; 6063/327621 objects misplaced (1.851%); 6.5 MiB/s, 104 objects/s recovering 2024-09-17T07:06:08.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:08 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/3190428803' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:06:08.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:08 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:06:08.403 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:06:08.424 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:06:08.428 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:08.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:08.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:06:08.444 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:08 smithi135 ceph-mon[33238]: pgmap v165: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.0 KiB/s rd, 34 MiB/s wr, 852 op/s; 6063/327621 objects misplaced (1.851%); 6.5 MiB/s, 104 objects/s recovering 2024-09-17T07:06:08.444 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:08 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/3190428803' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:06:08.444 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:08 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:06:08.445 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:06:08 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:06:08.530 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:06:08.564 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:08.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:08.565 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:06:08.665 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:06:08.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:08.725 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:08.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:06:08.826 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:06:08.856 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:08.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:08.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:06:08.958 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:06:08.994 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:08.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:08.995 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:06:09.096 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:06:09.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:09.134 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:09.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:06:09.181 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:09 smithi135 ceph-mon[33238]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:06:09.181 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:09 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T07:06:09.181 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:09 smithi135 ceph-mon[33238]: osdmap e141: 8 total, 8 up, 8 in 2024-09-17T07:06:09.182 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:09 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:06:09.235 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:06:09.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:09.273 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:09.274 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:06:09.374 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:06:09.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:09 smithi100 ceph-mon[29092]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:06:09.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:09 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T07:06:09.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:09 smithi100 ceph-mon[29092]: osdmap e141: 8 total, 8 up, 8 in 2024-09-17T07:06:09.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:09 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:06:09.379 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:09 smithi100 ceph-mon[39462]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:06:09.379 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:09 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T07:06:09.379 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:09 smithi100 ceph-mon[39462]: osdmap e141: 8 total, 8 up, 8 in 2024-09-17T07:06:09.379 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:09 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:06:09.419 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:09.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:09.420 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:06:09.521 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:06:09.550 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:09.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:09.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:06:09.652 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:06:09.689 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:09.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:09.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:06:09.792 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:06:09.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:09.830 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:09.830 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:06:09.931 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:06:09.964 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:09.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:09.965 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:06:10.066 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:06:10.105 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:10.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:10.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:06:10.207 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:06:10.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:10.239 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:10.239 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:06:10.340 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:06:10.373 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:10.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:10.374 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:06:10.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:10 smithi100 ceph-mon[29092]: 4.1b scrub starts 2024-09-17T07:06:10.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:10 smithi100 ceph-mon[29092]: pgmap v167: 158 pgs: 2 active+recovering+undersized+remapped, 156 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.14k op/s; 4272/311397 objects misplaced (1.372%); 7.8 MiB/s, 125 objects/s recovering 2024-09-17T07:06:10.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:10 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 2 pgs undersized (PG_DEGRADED) 2024-09-17T07:06:10.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:10 smithi100 ceph-mon[29092]: osdmap e142: 8 total, 8 up, 8 in 2024-09-17T07:06:10.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:10 smithi100 ceph-mon[39462]: 4.1b scrub starts 2024-09-17T07:06:10.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:10 smithi100 ceph-mon[39462]: pgmap v167: 158 pgs: 2 active+recovering+undersized+remapped, 156 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.14k op/s; 4272/311397 objects misplaced (1.372%); 7.8 MiB/s, 125 objects/s recovering 2024-09-17T07:06:10.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:10 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 2 pgs undersized (PG_DEGRADED) 2024-09-17T07:06:10.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:10 smithi100 ceph-mon[39462]: osdmap e142: 8 total, 8 up, 8 in 2024-09-17T07:06:10.475 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:06:10.512 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:10.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:10.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:06:10.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:10 smithi135 ceph-mon[33238]: 4.1b scrub starts 2024-09-17T07:06:10.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:10 smithi135 ceph-mon[33238]: pgmap v167: 158 pgs: 2 active+recovering+undersized+remapped, 156 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.14k op/s; 4272/311397 objects misplaced (1.372%); 7.8 MiB/s, 125 objects/s recovering 2024-09-17T07:06:10.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:10 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 2 pgs undersized (PG_DEGRADED) 2024-09-17T07:06:10.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:10 smithi135 ceph-mon[33238]: osdmap e142: 8 total, 8 up, 8 in 2024-09-17T07:06:10.615 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:06:10.653 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:10.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:10.654 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:06:10.754 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:06:10.783 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:10.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:10.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:06:10.886 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:06:10.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:10.934 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:10.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:06:11.035 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:06:11.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:11.081 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:11.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:06:11.182 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:06:11.222 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:11.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:11.223 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:06:11.324 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:06:11.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:11.356 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:11.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:06:11.457 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:06:11.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:11.496 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:11.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:06:11.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:11 smithi135 ceph-mon[33238]: 5.a scrub starts 2024-09-17T07:06:11.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:11 smithi135 ceph-mon[33238]: 4.a scrub starts 2024-09-17T07:06:11.598 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:06:11.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:11.620 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:11.620 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:06:11.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:11 smithi100 ceph-mon[29092]: 5.a scrub starts 2024-09-17T07:06:11.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:11 smithi100 ceph-mon[29092]: 4.a scrub starts 2024-09-17T07:06:11.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:11 smithi100 ceph-mon[39462]: 5.a scrub starts 2024-09-17T07:06:11.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:11 smithi100 ceph-mon[39462]: 4.a scrub starts 2024-09-17T07:06:11.721 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:06:11.758 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:11.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:11.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:06:11.859 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:06:11.897 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:11.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:11.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:06:11.999 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:06:12.034 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:12.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:12.035 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:06:12.136 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:06:12.173 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:12.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:12.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:06:12.275 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:06:12.302 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:12.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:12.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:06:12.361 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:06:12.362 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:06:12.395 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1: {} 2024-09-17T07:06:12.395 INFO:teuthology.orchestra.run.smithi100.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:06:12.395 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:12 smithi100 ceph-mon[29092]: pgmap v169: 158 pgs: 2 active+recovering+undersized+remapped, 156 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 855 B/s rd, 0 B/s wr, 904 op/s; 4272/311397 objects misplaced (1.372%); 6.2 MiB/s, 99 objects/s recovering 2024-09-17T07:06:12.395 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:12 smithi100 ceph-mon[29092]: 2.11 scrub starts 2024-09-17T07:06:12.395 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:12 smithi100 ceph-mon[29092]: 2.11 scrub ok 2024-09-17T07:06:12.396 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:12 smithi100 ceph-mon[39462]: pgmap v169: 158 pgs: 2 active+recovering+undersized+remapped, 156 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 855 B/s rd, 0 B/s wr, 904 op/s; 4272/311397 objects misplaced (1.372%); 6.2 MiB/s, 99 objects/s recovering 2024-09-17T07:06:12.396 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:12 smithi100 ceph-mon[39462]: 2.11 scrub starts 2024-09-17T07:06:12.396 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:12 smithi100 ceph-mon[39462]: 2.11 scrub ok 2024-09-17T07:06:12.404 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:06:12.427 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:06:12.428 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:06:12.448 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:12.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:12.449 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:06:12.460 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:06:12.460 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:06:12.494 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:06:12.495 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:06:12.527 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:06:12.528 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:06:12.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:12 smithi135 ceph-mon[33238]: pgmap v169: 158 pgs: 2 active+recovering+undersized+remapped, 156 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 855 B/s rd, 0 B/s wr, 904 op/s; 4272/311397 objects misplaced (1.372%); 6.2 MiB/s, 99 objects/s recovering 2024-09-17T07:06:12.545 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:12 smithi135 ceph-mon[33238]: 2.11 scrub starts 2024-09-17T07:06:12.545 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:12 smithi135 ceph-mon[33238]: 2.11 scrub ok 2024-09-17T07:06:12.549 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:06:12.559 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:06:12.559 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:06:12.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:12.585 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:12.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:06:12.596 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:06:12.596 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:06:12.686 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:06:12.762 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:12.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:12.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:06:12.863 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:06:12.892 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:12.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:12.893 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:06:12.993 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:06:13.041 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:13.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:13.042 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:06:13.143 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:06:13.169 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:13.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:13.170 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:06:13.271 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:06:13.297 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:13 smithi100 ceph-mon[29092]: 4.13 scrub starts 2024-09-17T07:06:13.298 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:13 smithi100 ceph-mon[29092]: 4.13 scrub ok 2024-09-17T07:06:13.298 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:13 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2728435263' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:06:13.298 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:13 smithi100 ceph-mon[39462]: 4.13 scrub starts 2024-09-17T07:06:13.298 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:13 smithi100 ceph-mon[39462]: 4.13 scrub ok 2024-09-17T07:06:13.298 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:13 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2728435263' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:06:13.309 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:13.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:13.310 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:06:13.411 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:06:13.445 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:13.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:13.446 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:06:13.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:13 smithi135 ceph-mon[33238]: 4.13 scrub starts 2024-09-17T07:06:13.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:13 smithi135 ceph-mon[33238]: 4.13 scrub ok 2024-09-17T07:06:13.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:13 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2728435263' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:06:13.547 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:06:13.574 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:13.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:13.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:06:13.676 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:06:13.707 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:13.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:13.708 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:06:13.809 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:06:13.852 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:13.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:13.853 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:06:13.954 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:06:13.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:13.990 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:13.991 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:06:14.091 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:06:14.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:14.136 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:14.136 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:06:14.237 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:06:14.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:14.280 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:14.280 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:06:14.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:14 smithi100 ceph-mon[39462]: 2.9 scrub starts 2024-09-17T07:06:14.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:14 smithi100 ceph-mon[39462]: pgmap v170: 158 pgs: 1 active+clean+scrubbing, 2 active+recovering+undersized+remapped, 155 active+clean; 6.2 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 732 B/s rd, 0 B/s wr, 1.15k op/s; 4272/303558 objects misplaced (1.407%); 5.3 MiB/s, 85 objects/s recovering 2024-09-17T07:06:14.380 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:14 smithi100 ceph-mon[39462]: 5.1f scrub starts 2024-09-17T07:06:14.380 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:14 smithi100 ceph-mon[39462]: 5.19 scrub starts 2024-09-17T07:06:14.380 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:14 smithi100 ceph-mon[29092]: 2.9 scrub starts 2024-09-17T07:06:14.380 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:14 smithi100 ceph-mon[29092]: pgmap v170: 158 pgs: 1 active+clean+scrubbing, 2 active+recovering+undersized+remapped, 155 active+clean; 6.2 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 732 B/s rd, 0 B/s wr, 1.15k op/s; 4272/303558 objects misplaced (1.407%); 5.3 MiB/s, 85 objects/s recovering 2024-09-17T07:06:14.380 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:14 smithi100 ceph-mon[29092]: 5.1f scrub starts 2024-09-17T07:06:14.380 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:14 smithi100 ceph-mon[29092]: 5.19 scrub starts 2024-09-17T07:06:14.381 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:06:14.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:14.402 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:14.403 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:06:14.503 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:06:14.534 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:14.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:14.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:06:14.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:14 smithi135 ceph-mon[33238]: 2.9 scrub starts 2024-09-17T07:06:14.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:14 smithi135 ceph-mon[33238]: pgmap v170: 158 pgs: 1 active+clean+scrubbing, 2 active+recovering+undersized+remapped, 155 active+clean; 6.2 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 732 B/s rd, 0 B/s wr, 1.15k op/s; 4272/303558 objects misplaced (1.407%); 5.3 MiB/s, 85 objects/s recovering 2024-09-17T07:06:14.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:14 smithi135 ceph-mon[33238]: 5.1f scrub starts 2024-09-17T07:06:14.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:14 smithi135 ceph-mon[33238]: 5.19 scrub starts 2024-09-17T07:06:14.636 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:06:14.674 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:14.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:14.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:06:14.682 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-17T07:06:14.776 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:06:14.798 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:14.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:14.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:06:14.900 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:06:14.943 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:14.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:14.944 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:06:15.045 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:06:15.074 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:15.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:15.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:06:15.176 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:06:15.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:15.206 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:15.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:06:15.307 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:06:15.334 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:15.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:15.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:06:15.339 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:06:15.435 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:06:15.474 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:15.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:15.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:06:15.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:15 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:06:15.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:15 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:06:15.576 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:06:15.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:15.605 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:15.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:06:15.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:15 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:06:15.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:15 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:06:15.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:15 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:06:15.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:15 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:06:15.706 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:06:15.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:15.766 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:15.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:06:15.867 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:06:15.895 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:15.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:15.897 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:06:15.997 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:06:16.026 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:16.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:16.027 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:06:16.126 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:06:15 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:06:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:06:16.128 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:06:16.154 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:16.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:16.155 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:06:16.256 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:06:16.280 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:16.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:16.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:06:16.298 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-17T07:06:16.298 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-17T07:06:16.299 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-17T07:06:16.299 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-17T07:06:16.299 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-17T07:06:16.382 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:06:16.398 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:16 smithi100 ceph-mon[29092]: 2.f scrub starts 2024-09-17T07:06:16.399 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:16 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:06:16.399 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:16 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:06:16.399 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:16 smithi100 ceph-mon[29092]: pgmap v171: 158 pgs: 2 active+clean+scrubbing, 2 active+recovering+undersized+remapped, 154 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.42k op/s; 3008/293460 objects misplaced (1.025%); 8.0 MiB/s, 127 objects/s recovering 2024-09-17T07:06:16.399 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:16 smithi100 ceph-mon[29092]: 2.14 scrub starts 2024-09-17T07:06:16.399 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:16 smithi100 ceph-mon[29092]: osdmap e143: 8 total, 8 up, 8 in 2024-09-17T07:06:16.399 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:16 smithi100 ceph-mon[39462]: 2.f scrub starts 2024-09-17T07:06:16.399 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:16 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:06:16.399 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:16 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:06:16.400 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:16 smithi100 ceph-mon[39462]: pgmap v171: 158 pgs: 2 active+clean+scrubbing, 2 active+recovering+undersized+remapped, 154 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.42k op/s; 3008/293460 objects misplaced (1.025%); 8.0 MiB/s, 127 objects/s recovering 2024-09-17T07:06:16.400 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:16 smithi100 ceph-mon[39462]: 2.14 scrub starts 2024-09-17T07:06:16.400 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:16 smithi100 ceph-mon[39462]: osdmap e143: 8 total, 8 up, 8 in 2024-09-17T07:06:16.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:16.422 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:16.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:06:16.523 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:06:16.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:16 smithi135 ceph-mon[33238]: 2.f scrub starts 2024-09-17T07:06:16.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:16 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:06:16.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:16 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:06:16.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:16 smithi135 ceph-mon[33238]: pgmap v171: 158 pgs: 2 active+clean+scrubbing, 2 active+recovering+undersized+remapped, 154 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.42k op/s; 3008/293460 objects misplaced (1.025%); 8.0 MiB/s, 127 objects/s recovering 2024-09-17T07:06:16.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:16 smithi135 ceph-mon[33238]: 2.14 scrub starts 2024-09-17T07:06:16.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:16 smithi135 ceph-mon[33238]: osdmap e143: 8 total, 8 up, 8 in 2024-09-17T07:06:16.552 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:16.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:16.553 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:06:16.654 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:06:16.692 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:16.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:16.693 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:06:16.793 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:06:16.819 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:16.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:16.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:06:16.921 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:06:16.950 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:16.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:16.951 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:06:17.052 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:06:17.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:17.100 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:17.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:06:17.201 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:06:17.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:17.256 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:17.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:06:17.357 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:06:17.385 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:17.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:17.386 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:06:17.487 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:06:17.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:17.516 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:17.516 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:06:17.617 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:06:17.656 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:17.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:17.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:06:17.758 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:06:17.784 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:17.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:17.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:06:17.789 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:06:17.886 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:06:17.915 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:17.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:17.916 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:06:18.016 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:06:18.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:18.046 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:18.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:06:18.147 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:06:18.176 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:18.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:18.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:06:18.193 INFO:teuthology.orchestra.run.smithi100.stderr:reweighted osd.2 to 1 (10000) 2024-09-17T07:06:18.278 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:06:18.306 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:18.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:18.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:06:18.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:18 smithi100 ceph-mon[29092]: 6.1a scrub starts 2024-09-17T07:06:18.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:18 smithi100 ceph-mon[29092]: pgmap v173: 158 pgs: 2 active+clean+scrubbing, 2 active+recovering+undersized+remapped, 154 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 747 op/s; 3008/293460 objects misplaced (1.025%); 3.3 MiB/s, 52 objects/s recovering 2024-09-17T07:06:18.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:18 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:06:18.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:18 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1526149339' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-17T07:06:18.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:18 smithi100 ceph-mon[39462]: 6.1a scrub starts 2024-09-17T07:06:18.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:18 smithi100 ceph-mon[39462]: pgmap v173: 158 pgs: 2 active+clean+scrubbing, 2 active+recovering+undersized+remapped, 154 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 747 op/s; 3008/293460 objects misplaced (1.025%); 3.3 MiB/s, 52 objects/s recovering 2024-09-17T07:06:18.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:18 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:06:18.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:18 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1526149339' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-17T07:06:18.408 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:06:18.447 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:18.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:18.448 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:06:18.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:18 smithi135 ceph-mon[33238]: 6.1a scrub starts 2024-09-17T07:06:18.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:18 smithi135 ceph-mon[33238]: pgmap v173: 158 pgs: 2 active+clean+scrubbing, 2 active+recovering+undersized+remapped, 154 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 747 op/s; 3008/293460 objects misplaced (1.025%); 3.3 MiB/s, 52 objects/s recovering 2024-09-17T07:06:18.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:18 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:06:18.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:18 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1526149339' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-17T07:06:18.544 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:06:18 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:06:18.547 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-17T07:06:18.549 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:06:18.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:18.587 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:18.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:06:18.688 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:06:18.715 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:18.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:18.717 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:06:18.817 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:06:18.865 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:18.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:18.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:06:18.967 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:06:19.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:19.023 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:19.023 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:06:19.124 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:06:19.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:19.153 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:19.154 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:06:19.254 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:06:19.287 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:19.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:19.288 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:06:19.372 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:19 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1526149339' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-17T07:06:19.372 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:19 smithi100 ceph-mon[39462]: osdmap e144: 8 total, 8 up, 8 in 2024-09-17T07:06:19.372 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:19 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:06:19.373 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:19 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1526149339' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-17T07:06:19.373 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:19 smithi100 ceph-mon[29092]: osdmap e144: 8 total, 8 up, 8 in 2024-09-17T07:06:19.373 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:19 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:06:19.389 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:06:19.419 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:19.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:19.421 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:06:19.521 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:06:19.536 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:19 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1526149339' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-17T07:06:19.536 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:19 smithi135 ceph-mon[33238]: osdmap e144: 8 total, 8 up, 8 in 2024-09-17T07:06:19.536 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:19 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:06:19.559 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:19.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:19.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:06:19.661 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:06:19.694 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd reweight 3 1 2024-09-17T07:06:19.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:19.702 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:19.702 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:06:19.803 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:06:19.886 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:19.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:19.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:06:19.988 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:06:20.010 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:20.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:20.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:06:20.112 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:06:20.150 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:20.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:20.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:06:20.251 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:06:20.283 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:20.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:20.284 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:06:20.385 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:06:20.430 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:20.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:20.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:06:20.532 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:06:20.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:20 smithi135 ceph-mon[33238]: pgmap v175: 158 pgs: 1 active+clean+scrubbing, 2 active+recovering+undersized+remapped, 155 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.50k op/s; 1524/275457 objects misplaced (0.553%); 7.2 MiB/s, 114 objects/s recovering 2024-09-17T07:06:20.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:20 smithi135 ceph-mon[33238]: 4.5 scrub starts 2024-09-17T07:06:20.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:20.565 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:20.566 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:06:20.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:20 smithi100 ceph-mon[29092]: pgmap v175: 158 pgs: 1 active+clean+scrubbing, 2 active+recovering+undersized+remapped, 155 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.50k op/s; 1524/275457 objects misplaced (0.553%); 7.2 MiB/s, 114 objects/s recovering 2024-09-17T07:06:20.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:20 smithi100 ceph-mon[29092]: 4.5 scrub starts 2024-09-17T07:06:20.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:20 smithi100 ceph-mon[39462]: pgmap v175: 158 pgs: 1 active+clean+scrubbing, 2 active+recovering+undersized+remapped, 155 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.50k op/s; 1524/275457 objects misplaced (0.553%); 7.2 MiB/s, 114 objects/s recovering 2024-09-17T07:06:20.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:20 smithi100 ceph-mon[39462]: 4.5 scrub starts 2024-09-17T07:06:20.667 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:06:20.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:20.704 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:20.704 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:06:20.805 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:06:20.835 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:20.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:20.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:06:20.937 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:06:20.964 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:20.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:20.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:06:21.066 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:06:21.095 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:21.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:21.096 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:06:21.197 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:06:21.223 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:21.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:21.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:06:21.325 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:06:21.366 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:21.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:21.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:06:21.468 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:06:21.499 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:21.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:21.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:06:21.601 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:06:21.631 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:21.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:21.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:06:21.733 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:06:21.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:21.755 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:21.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:06:21.791 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:06:21.803 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:06:21.822 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:06:21.856 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:06:21.877 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:21.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:21.878 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:06:21.979 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:06:22.005 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:22.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:22.006 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:06:22.107 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:06:22.128 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:22.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:22.129 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:06:22.230 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:06:22.262 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:22.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:22.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:06:22.283 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:22 smithi100 ceph-mon[29092]: pgmap v176: 158 pgs: 1 active+clean+scrubbing, 2 active+recovering+undersized+remapped, 155 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.17k op/s; 1524/275457 objects misplaced (0.553%); 7.2 MiB/s, 114 objects/s recovering 2024-09-17T07:06:22.283 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:22 smithi100 ceph-mon[29092]: 5.2 scrub starts 2024-09-17T07:06:22.283 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:22 smithi100 ceph-mon[29092]: osdmap e145: 8 total, 8 up, 8 in 2024-09-17T07:06:22.283 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:22 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:06:22.283 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:22 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:06:22.283 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:22 smithi100 ceph-mon[29092]: 5.c scrub starts 2024-09-17T07:06:22.283 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:22 smithi100 ceph-mon[39462]: pgmap v176: 158 pgs: 1 active+clean+scrubbing, 2 active+recovering+undersized+remapped, 155 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.17k op/s; 1524/275457 objects misplaced (0.553%); 7.2 MiB/s, 114 objects/s recovering 2024-09-17T07:06:22.284 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:22 smithi100 ceph-mon[39462]: 5.2 scrub starts 2024-09-17T07:06:22.284 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:22 smithi100 ceph-mon[39462]: osdmap e145: 8 total, 8 up, 8 in 2024-09-17T07:06:22.284 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:22 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:06:22.284 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:22 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:06:22.284 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:22 smithi100 ceph-mon[39462]: 5.c scrub starts 2024-09-17T07:06:22.364 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:06:22.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:22.502 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:22.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:06:22.509 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:06:22.509 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:06:22.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:22 smithi135 ceph-mon[33238]: pgmap v176: 158 pgs: 1 active+clean+scrubbing, 2 active+recovering+undersized+remapped, 155 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.17k op/s; 1524/275457 objects misplaced (0.553%); 7.2 MiB/s, 114 objects/s recovering 2024-09-17T07:06:22.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:22 smithi135 ceph-mon[33238]: 5.2 scrub starts 2024-09-17T07:06:22.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:22 smithi135 ceph-mon[33238]: osdmap e145: 8 total, 8 up, 8 in 2024-09-17T07:06:22.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:22 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:06:22.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:22 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:06:22.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:22 smithi135 ceph-mon[33238]: 5.c scrub starts 2024-09-17T07:06:22.553 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1: {} 2024-09-17T07:06:22.553 INFO:teuthology.orchestra.run.smithi100.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:06:22.586 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:06:22.587 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:06:22.604 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:06:22.618 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:06:22.618 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:06:22.633 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:22.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:22.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:06:22.655 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:06:22.655 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:06:22.683 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:06:22.683 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:06:22.716 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:06:22.716 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:06:22.735 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:06:22.751 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:06:22.752 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:06:22.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:22.769 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:22.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:06:22.870 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:06:22.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:22.911 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:22.911 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:06:23.012 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:06:23.041 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:23.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:23.042 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:06:23.143 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:06:23.164 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:23.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:23.165 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:06:23.265 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:06:23.298 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:23.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:23.299 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:06:23.400 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:06:23.439 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:23.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:23.440 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:06:23.541 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:06:23.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:23 smithi135 ceph-mon[33238]: 7.c scrub starts 2024-09-17T07:06:23.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:23 smithi135 ceph-mon[33238]: osdmap e146: 8 total, 8 up, 8 in 2024-09-17T07:06:23.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:23 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1917196113' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:06:23.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:23 smithi135 ceph-mon[33238]: pgmap v179: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.42k op/s; 1134/267966 objects misplaced (0.423%); 6.5 MiB/s, 104 objects/s recovering 2024-09-17T07:06:23.573 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:23 smithi100 ceph-mon[39462]: 7.c scrub starts 2024-09-17T07:06:23.573 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:23 smithi100 ceph-mon[39462]: osdmap e146: 8 total, 8 up, 8 in 2024-09-17T07:06:23.574 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:23 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1917196113' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:06:23.574 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:23 smithi100 ceph-mon[39462]: pgmap v179: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.42k op/s; 1134/267966 objects misplaced (0.423%); 6.5 MiB/s, 104 objects/s recovering 2024-09-17T07:06:23.574 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:23 smithi100 ceph-mon[29092]: 7.c scrub starts 2024-09-17T07:06:23.574 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:23 smithi100 ceph-mon[29092]: osdmap e146: 8 total, 8 up, 8 in 2024-09-17T07:06:23.574 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:23 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1917196113' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:06:23.574 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:23 smithi100 ceph-mon[29092]: pgmap v179: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.42k op/s; 1134/267966 objects misplaced (0.423%); 6.5 MiB/s, 104 objects/s recovering 2024-09-17T07:06:23.582 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:23.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:23.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:06:23.691 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:06:23.727 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:23.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:23.728 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:06:23.828 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:06:23.864 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:23.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:23.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:06:23.966 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:06:23.997 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:23.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:23.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:06:24.099 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:06:24.141 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:24.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:24.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:06:24.243 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:06:24.280 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:24.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:24.281 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:06:24.381 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:06:24.414 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:24.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:24.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:06:24.481 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:24 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-09-17T07:06:24.482 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:24 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:06:24.482 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:24 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:06:24.482 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:24 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:06:24.482 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:24 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-09-17T07:06:24.482 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:24 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:06:24.482 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:24 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:06:24.482 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:24 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:06:24.516 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:06:24.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:24 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-09-17T07:06:24.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:24 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:06:24.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:24 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:06:24.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:24 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:06:24.545 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:24.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:24.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:06:24.647 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:06:24.673 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:24.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:24.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:06:24.775 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:06:24.796 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:24.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:24.797 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:06:24.898 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:06:24.920 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:24.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:24.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:06:25.021 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:06:25.042 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:25.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:25.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:06:25.144 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:06:25.173 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:25.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:25.174 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:06:25.275 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:06:25.307 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:25.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:25.308 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:06:25.409 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:06:25.438 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:25 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:06:25.439 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:25 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:06:25.439 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:25 smithi100 ceph-mon[29092]: 7.1 scrub starts 2024-09-17T07:06:25.439 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:25 smithi100 ceph-mon[29092]: osdmap e147: 8 total, 8 up, 8 in 2024-09-17T07:06:25.439 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:25 smithi100 ceph-mon[29092]: 7.1 scrub ok 2024-09-17T07:06:25.439 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:25 smithi100 ceph-mon[29092]: pgmap v181: 158 pgs: 1 active+undersized+remapped, 157 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 908 op/s; 5.3 MiB/s, 84 objects/s recovering 2024-09-17T07:06:25.439 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:25 smithi100 ceph-mon[29092]: osdmap e148: 8 total, 8 up, 8 in 2024-09-17T07:06:25.439 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:25 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:06:25.439 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:25 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:06:25.440 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:25 smithi100 ceph-mon[39462]: 7.1 scrub starts 2024-09-17T07:06:25.440 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:25 smithi100 ceph-mon[39462]: osdmap e147: 8 total, 8 up, 8 in 2024-09-17T07:06:25.440 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:25 smithi100 ceph-mon[39462]: 7.1 scrub ok 2024-09-17T07:06:25.440 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:25 smithi100 ceph-mon[39462]: pgmap v181: 158 pgs: 1 active+undersized+remapped, 157 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 908 op/s; 5.3 MiB/s, 84 objects/s recovering 2024-09-17T07:06:25.440 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:25 smithi100 ceph-mon[39462]: osdmap e148: 8 total, 8 up, 8 in 2024-09-17T07:06:25.459 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:25.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:25.462 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:06:25.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:25 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:06:25.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:25 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:06:25.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:25 smithi135 ceph-mon[33238]: 7.1 scrub starts 2024-09-17T07:06:25.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:25 smithi135 ceph-mon[33238]: osdmap e147: 8 total, 8 up, 8 in 2024-09-17T07:06:25.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:25 smithi135 ceph-mon[33238]: 7.1 scrub ok 2024-09-17T07:06:25.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:25 smithi135 ceph-mon[33238]: pgmap v181: 158 pgs: 1 active+undersized+remapped, 157 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 908 op/s; 5.3 MiB/s, 84 objects/s recovering 2024-09-17T07:06:25.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:25 smithi135 ceph-mon[33238]: osdmap e148: 8 total, 8 up, 8 in 2024-09-17T07:06:25.563 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:06:25.599 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:25.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:25.601 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:06:25.701 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:06:25.733 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:25.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:25.734 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:06:25.835 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:06:25.872 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:25.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:25.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:06:25.973 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:06:26.018 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:26.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:26.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:06:26.120 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:06:26.141 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:06:25 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:06:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:06:26.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:26.148 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:26.148 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:06:26.249 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:06:26.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:26.277 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:26.278 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:06:26.379 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:06:26.409 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:26.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:26.410 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:06:26.511 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:06:26.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:26.543 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:26.544 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:06:26.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:26 smithi135 ceph-mon[33238]: 5.15 scrub starts 2024-09-17T07:06:26.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:26 smithi135 ceph-mon[33238]: 5.15 scrub ok 2024-09-17T07:06:26.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:26 smithi135 ceph-mon[33238]: 6.10 scrub starts 2024-09-17T07:06:26.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:26 smithi135 ceph-mon[33238]: 6.10 scrub ok 2024-09-17T07:06:26.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:26 smithi135 ceph-mon[33238]: 7.d scrub starts 2024-09-17T07:06:26.545 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:26 smithi135 ceph-mon[33238]: 5.6 scrub starts 2024-09-17T07:06:26.545 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:26 smithi135 ceph-mon[33238]: 5.6 scrub ok 2024-09-17T07:06:26.545 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:26 smithi135 ceph-mon[33238]: 5.9 scrub starts 2024-09-17T07:06:26.545 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:26 smithi135 ceph-mon[33238]: 5.9 scrub ok 2024-09-17T07:06:26.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:26 smithi100 ceph-mon[29092]: 5.15 scrub starts 2024-09-17T07:06:26.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:26 smithi100 ceph-mon[29092]: 5.15 scrub ok 2024-09-17T07:06:26.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:26 smithi100 ceph-mon[29092]: 6.10 scrub starts 2024-09-17T07:06:26.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:26 smithi100 ceph-mon[29092]: 6.10 scrub ok 2024-09-17T07:06:26.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:26 smithi100 ceph-mon[29092]: 7.d scrub starts 2024-09-17T07:06:26.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:26 smithi100 ceph-mon[29092]: 5.6 scrub starts 2024-09-17T07:06:26.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:26 smithi100 ceph-mon[29092]: 5.6 scrub ok 2024-09-17T07:06:26.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:26 smithi100 ceph-mon[29092]: 5.9 scrub starts 2024-09-17T07:06:26.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:26 smithi100 ceph-mon[29092]: 5.9 scrub ok 2024-09-17T07:06:26.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:26 smithi100 ceph-mon[39462]: 5.15 scrub starts 2024-09-17T07:06:26.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:26 smithi100 ceph-mon[39462]: 5.15 scrub ok 2024-09-17T07:06:26.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:26 smithi100 ceph-mon[39462]: 6.10 scrub starts 2024-09-17T07:06:26.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:26 smithi100 ceph-mon[39462]: 6.10 scrub ok 2024-09-17T07:06:26.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:26 smithi100 ceph-mon[39462]: 7.d scrub starts 2024-09-17T07:06:26.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:26 smithi100 ceph-mon[39462]: 5.6 scrub starts 2024-09-17T07:06:26.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:26 smithi100 ceph-mon[39462]: 5.6 scrub ok 2024-09-17T07:06:26.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:26 smithi100 ceph-mon[39462]: 5.9 scrub starts 2024-09-17T07:06:26.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:26 smithi100 ceph-mon[39462]: 5.9 scrub ok 2024-09-17T07:06:26.645 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:06:26.674 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:26.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:26.675 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:06:26.776 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:06:26.813 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:26.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:26.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:06:26.878 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-17T07:06:26.878 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:06:26.915 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:06:26.943 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:26.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:26.944 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:06:27.045 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:06:27.074 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:27.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:27.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:06:27.166 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:06:27.175 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:06:27.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:27.248 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:27.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:06:27.349 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:06:27.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:27.394 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:27.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:06:27.495 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:06:27.519 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:27.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:27.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:06:27.533 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:27 smithi100 ceph-mon[29092]: 6.0 scrub starts 2024-09-17T07:06:27.533 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:27 smithi100 ceph-mon[29092]: 6.0 scrub ok 2024-09-17T07:06:27.533 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:27 smithi100 ceph-mon[29092]: 2.16 scrub starts 2024-09-17T07:06:27.533 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:27 smithi100 ceph-mon[29092]: 2.16 scrub ok 2024-09-17T07:06:27.533 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:27 smithi100 ceph-mon[29092]: pgmap v183: 158 pgs: 1 active+undersized+remapped, 157 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 876 B/s rd, 0 B/s wr, 933 op/s; 5.4 MiB/s, 86 objects/s recovering 2024-09-17T07:06:27.534 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:27 smithi100 ceph-mon[39462]: 6.0 scrub starts 2024-09-17T07:06:27.534 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:27 smithi100 ceph-mon[39462]: 6.0 scrub ok 2024-09-17T07:06:27.534 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:27 smithi100 ceph-mon[39462]: 2.16 scrub starts 2024-09-17T07:06:27.535 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:27 smithi100 ceph-mon[39462]: 2.16 scrub ok 2024-09-17T07:06:27.535 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:27 smithi100 ceph-mon[39462]: pgmap v183: 158 pgs: 1 active+undersized+remapped, 157 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 876 B/s rd, 0 B/s wr, 933 op/s; 5.4 MiB/s, 86 objects/s recovering 2024-09-17T07:06:27.548 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:27 smithi135 ceph-mon[33238]: 6.0 scrub starts 2024-09-17T07:06:27.549 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:27 smithi135 ceph-mon[33238]: 6.0 scrub ok 2024-09-17T07:06:27.552 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:27 smithi135 ceph-mon[33238]: 2.16 scrub starts 2024-09-17T07:06:27.552 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:27 smithi135 ceph-mon[33238]: 2.16 scrub ok 2024-09-17T07:06:27.552 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:27 smithi135 ceph-mon[33238]: pgmap v183: 158 pgs: 1 active+undersized+remapped, 157 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 876 B/s rd, 0 B/s wr, 933 op/s; 5.4 MiB/s, 86 objects/s recovering 2024-09-17T07:06:27.621 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:06:27.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:27.673 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:27.673 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:06:27.774 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:06:27.796 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:27.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:27.797 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:06:27.898 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:06:27.940 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:27.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:27.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:06:28.042 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:06:28.074 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:28.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:28.075 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:06:28.176 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:06:28.205 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:28.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:28.207 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:06:28.307 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:06:28.357 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:28.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:28.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:06:28.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:28 smithi100 ceph-mon[29092]: 7.d scrub ok 2024-09-17T07:06:28.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:28 smithi100 ceph-mon[29092]: 4.b scrub starts 2024-09-17T07:06:28.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:28 smithi100 ceph-mon[29092]: 4.b scrub ok 2024-09-17T07:06:28.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:28 smithi100 ceph-mon[29092]: 7.e scrub starts 2024-09-17T07:06:28.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:28 smithi100 ceph-mon[39462]: 7.d scrub ok 2024-09-17T07:06:28.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:28 smithi100 ceph-mon[39462]: 4.b scrub starts 2024-09-17T07:06:28.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:28 smithi100 ceph-mon[39462]: 4.b scrub ok 2024-09-17T07:06:28.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:28 smithi100 ceph-mon[39462]: 7.e scrub starts 2024-09-17T07:06:28.459 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:06:28.500 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:28.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:28.501 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:06:28.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:28 smithi135 ceph-mon[33238]: 7.d scrub ok 2024-09-17T07:06:28.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:28 smithi135 ceph-mon[33238]: 4.b scrub starts 2024-09-17T07:06:28.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:28 smithi135 ceph-mon[33238]: 4.b scrub ok 2024-09-17T07:06:28.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:28 smithi135 ceph-mon[33238]: 7.e scrub starts 2024-09-17T07:06:28.543 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:06:28 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:06:28.602 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:06:28.626 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:28.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:28.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:06:28.728 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:06:28.767 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:28.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:28.768 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:06:28.869 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:06:28.896 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:28.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:28.897 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:06:28.998 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:06:29.028 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:29.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:29.029 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:06:29.071 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-17T07:06:29.130 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:06:29.156 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:29.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:29.157 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:06:29.257 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:06:29.282 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:29 smithi100 ceph-mon[29092]: 4.1d scrub starts 2024-09-17T07:06:29.282 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:29 smithi100 ceph-mon[29092]: 4.1d scrub ok 2024-09-17T07:06:29.283 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:29 smithi100 ceph-mon[29092]: 5.b scrub starts 2024-09-17T07:06:29.283 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:29 smithi100 ceph-mon[29092]: 5.b scrub ok 2024-09-17T07:06:29.283 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:29 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:06:29.283 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:29 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/3161815826' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:06:29.283 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:29 smithi100 ceph-mon[29092]: 5.7 scrub starts 2024-09-17T07:06:29.283 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:29 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/659276105' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-17T07:06:29.283 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:29 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-17T07:06:29.283 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:29 smithi100 ceph-mon[29092]: pgmap v184: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 985 op/s; 4.6 MiB/s, 74 objects/s recovering 2024-09-17T07:06:29.283 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:29 smithi100 ceph-mon[39462]: 4.1d scrub starts 2024-09-17T07:06:29.283 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:29 smithi100 ceph-mon[39462]: 4.1d scrub ok 2024-09-17T07:06:29.284 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:29 smithi100 ceph-mon[39462]: 5.b scrub starts 2024-09-17T07:06:29.284 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:29 smithi100 ceph-mon[39462]: 5.b scrub ok 2024-09-17T07:06:29.284 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:29 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:06:29.284 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:29 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/3161815826' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:06:29.284 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:29 smithi100 ceph-mon[39462]: 5.7 scrub starts 2024-09-17T07:06:29.284 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:29 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/659276105' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-17T07:06:29.284 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:29 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-17T07:06:29.284 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:29 smithi100 ceph-mon[39462]: pgmap v184: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 985 op/s; 4.6 MiB/s, 74 objects/s recovering 2024-09-17T07:06:29.305 INFO:teuthology.orchestra.run.smithi100.stderr:nodeep-scrub is unset 2024-09-17T07:06:29.305 INFO:teuthology.orchestra.run.smithi100.stderr:reweighted osd.3 to 1 (10000) 2024-09-17T07:06:29.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:29.313 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:29.314 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:06:29.414 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:06:29.447 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:29.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:29.448 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:06:29.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:29 smithi135 ceph-mon[33238]: 4.1d scrub starts 2024-09-17T07:06:29.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:29 smithi135 ceph-mon[33238]: 4.1d scrub ok 2024-09-17T07:06:29.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:29 smithi135 ceph-mon[33238]: 5.b scrub starts 2024-09-17T07:06:29.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:29 smithi135 ceph-mon[33238]: 5.b scrub ok 2024-09-17T07:06:29.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:29 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:06:29.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:29 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/3161815826' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:06:29.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:29 smithi135 ceph-mon[33238]: 5.7 scrub starts 2024-09-17T07:06:29.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:29 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/659276105' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-17T07:06:29.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:29 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-17T07:06:29.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:29 smithi135 ceph-mon[33238]: pgmap v184: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 985 op/s; 4.6 MiB/s, 74 objects/s recovering 2024-09-17T07:06:29.549 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:06:29.575 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:29.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:29.576 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:06:29.677 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:06:29.709 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:29.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:29.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:06:29.811 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:06:29.853 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:29.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:29.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:06:29.954 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:06:29.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:29.995 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:29.995 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:06:30.096 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:06:30.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:30.137 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:30.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:06:30.232 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd reweight 4 1 2024-09-17T07:06:30.239 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:06:30.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:30.319 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:30.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:06:30.420 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:06:30.446 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:30.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:30.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:06:30.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:30 smithi135 ceph-mon[33238]: 5.7 scrub ok 2024-09-17T07:06:30.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:30 smithi135 ceph-mon[33238]: 4.19 scrub starts 2024-09-17T07:06:30.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:30 smithi135 ceph-mon[33238]: 4.19 scrub ok 2024-09-17T07:06:30.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:30 smithi135 ceph-mon[33238]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-09-17T07:06:30.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:30 smithi135 ceph-mon[33238]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T07:06:30.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:30 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/3161815826' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T07:06:30.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:30 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-17T07:06:30.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:30 smithi135 ceph-mon[33238]: osdmap e149: 8 total, 8 up, 8 in 2024-09-17T07:06:30.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:30 smithi135 ceph-mon[33238]: 7.e scrub ok 2024-09-17T07:06:30.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:30 smithi135 ceph-mon[33238]: 5.16 scrub starts 2024-09-17T07:06:30.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:30 smithi135 ceph-mon[33238]: 5.16 scrub ok 2024-09-17T07:06:30.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:30 smithi135 ceph-mon[33238]: 2.d deep-scrub starts 2024-09-17T07:06:30.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:30 smithi135 ceph-mon[33238]: 2.d deep-scrub ok 2024-09-17T07:06:30.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:30 smithi135 ceph-mon[33238]: osdmap e150: 8 total, 8 up, 8 in 2024-09-17T07:06:30.548 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:06:30.575 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:30.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:30.576 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:06:30.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:30 smithi100 ceph-mon[29092]: 5.7 scrub ok 2024-09-17T07:06:30.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:30 smithi100 ceph-mon[29092]: 4.19 scrub starts 2024-09-17T07:06:30.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:30 smithi100 ceph-mon[29092]: 4.19 scrub ok 2024-09-17T07:06:30.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:30 smithi100 ceph-mon[29092]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-09-17T07:06:30.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:30 smithi100 ceph-mon[29092]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T07:06:30.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:30 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/3161815826' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T07:06:30.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:30 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-17T07:06:30.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:30 smithi100 ceph-mon[29092]: osdmap e149: 8 total, 8 up, 8 in 2024-09-17T07:06:30.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:30 smithi100 ceph-mon[29092]: 7.e scrub ok 2024-09-17T07:06:30.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:30 smithi100 ceph-mon[29092]: 5.16 scrub starts 2024-09-17T07:06:30.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:30 smithi100 ceph-mon[29092]: 5.16 scrub ok 2024-09-17T07:06:30.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:30 smithi100 ceph-mon[29092]: 2.d deep-scrub starts 2024-09-17T07:06:30.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:30 smithi100 ceph-mon[29092]: 2.d deep-scrub ok 2024-09-17T07:06:30.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:30 smithi100 ceph-mon[29092]: osdmap e150: 8 total, 8 up, 8 in 2024-09-17T07:06:30.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:30 smithi100 ceph-mon[39462]: 5.7 scrub ok 2024-09-17T07:06:30.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:30 smithi100 ceph-mon[39462]: 4.19 scrub starts 2024-09-17T07:06:30.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:30 smithi100 ceph-mon[39462]: 4.19 scrub ok 2024-09-17T07:06:30.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:30 smithi100 ceph-mon[39462]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-09-17T07:06:30.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:30 smithi100 ceph-mon[39462]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T07:06:30.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:30 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/3161815826' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T07:06:30.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:30 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-17T07:06:30.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:30 smithi100 ceph-mon[39462]: osdmap e149: 8 total, 8 up, 8 in 2024-09-17T07:06:30.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:30 smithi100 ceph-mon[39462]: 7.e scrub ok 2024-09-17T07:06:30.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:30 smithi100 ceph-mon[39462]: 5.16 scrub starts 2024-09-17T07:06:30.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:30 smithi100 ceph-mon[39462]: 5.16 scrub ok 2024-09-17T07:06:30.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:30 smithi100 ceph-mon[39462]: 2.d deep-scrub starts 2024-09-17T07:06:30.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:30 smithi100 ceph-mon[39462]: 2.d deep-scrub ok 2024-09-17T07:06:30.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:30 smithi100 ceph-mon[39462]: osdmap e150: 8 total, 8 up, 8 in 2024-09-17T07:06:30.677 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:06:30.702 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:30.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:30.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:06:30.727 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:06:30.754 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:06:30.803 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:06:30.874 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:30.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:30.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:06:30.976 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:06:31.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:31.019 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:31.020 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:06:31.114 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:06:31.121 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:06:31.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:31.149 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:31.150 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:06:31.250 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:06:31.279 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:31.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:31.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:06:31.381 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:06:31.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:31.419 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:31.419 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:06:31.520 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:06:31.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:31 smithi135 ceph-mon[33238]: 4.e scrub starts 2024-09-17T07:06:31.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:31 smithi135 ceph-mon[33238]: 4.e scrub ok 2024-09-17T07:06:31.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:31 smithi135 ceph-mon[33238]: 5.0 scrub starts 2024-09-17T07:06:31.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:31 smithi135 ceph-mon[33238]: 5.0 scrub ok 2024-09-17T07:06:31.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:31 smithi135 ceph-mon[33238]: 4.16 scrub starts 2024-09-17T07:06:31.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:31 smithi135 ceph-mon[33238]: 4.16 scrub ok 2024-09-17T07:06:31.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:31 smithi135 ceph-mon[33238]: 6.7 scrub starts 2024-09-17T07:06:31.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:31 smithi135 ceph-mon[33238]: 6.7 scrub ok 2024-09-17T07:06:31.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:31 smithi135 ceph-mon[33238]: 6.1 scrub starts 2024-09-17T07:06:31.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:31 smithi135 ceph-mon[33238]: 6.1 scrub ok 2024-09-17T07:06:31.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:31 smithi135 ceph-mon[33238]: pgmap v187: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 213 op/s 2024-09-17T07:06:31.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:31.559 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:31.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:06:31.599 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:31 smithi100 ceph-mon[29092]: 4.e scrub starts 2024-09-17T07:06:31.599 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:31 smithi100 ceph-mon[29092]: 4.e scrub ok 2024-09-17T07:06:31.599 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:31 smithi100 ceph-mon[29092]: 5.0 scrub starts 2024-09-17T07:06:31.599 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:31 smithi100 ceph-mon[29092]: 5.0 scrub ok 2024-09-17T07:06:31.599 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:31 smithi100 ceph-mon[29092]: 4.16 scrub starts 2024-09-17T07:06:31.599 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:31 smithi100 ceph-mon[29092]: 4.16 scrub ok 2024-09-17T07:06:31.599 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:31 smithi100 ceph-mon[29092]: 6.7 scrub starts 2024-09-17T07:06:31.599 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:31 smithi100 ceph-mon[29092]: 6.7 scrub ok 2024-09-17T07:06:31.600 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:31 smithi100 ceph-mon[29092]: 6.1 scrub starts 2024-09-17T07:06:31.600 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:31 smithi100 ceph-mon[29092]: 6.1 scrub ok 2024-09-17T07:06:31.600 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:31 smithi100 ceph-mon[29092]: pgmap v187: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 213 op/s 2024-09-17T07:06:31.600 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:31 smithi100 ceph-mon[39462]: 4.e scrub starts 2024-09-17T07:06:31.600 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:31 smithi100 ceph-mon[39462]: 4.e scrub ok 2024-09-17T07:06:31.600 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:31 smithi100 ceph-mon[39462]: 5.0 scrub starts 2024-09-17T07:06:31.600 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:31 smithi100 ceph-mon[39462]: 5.0 scrub ok 2024-09-17T07:06:31.600 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:31 smithi100 ceph-mon[39462]: 4.16 scrub starts 2024-09-17T07:06:31.600 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:31 smithi100 ceph-mon[39462]: 4.16 scrub ok 2024-09-17T07:06:31.600 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:31 smithi100 ceph-mon[39462]: 6.7 scrub starts 2024-09-17T07:06:31.601 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:31 smithi100 ceph-mon[39462]: 6.7 scrub ok 2024-09-17T07:06:31.601 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:31 smithi100 ceph-mon[39462]: 6.1 scrub starts 2024-09-17T07:06:31.601 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:31 smithi100 ceph-mon[39462]: 6.1 scrub ok 2024-09-17T07:06:31.601 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:31 smithi100 ceph-mon[39462]: pgmap v187: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 213 op/s 2024-09-17T07:06:31.660 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:06:31.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:31.701 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:31.702 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:06:31.802 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:06:31.838 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:31.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:31.839 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:06:31.940 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:06:31.961 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:31.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:31.962 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:06:32.063 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:06:32.091 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:32.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:32.092 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:06:32.193 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:06:32.224 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:32.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:32.225 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:06:32.326 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:06:32.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:32.365 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:32.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:06:32.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:32 smithi100 ceph-mon[29092]: 6.c scrub starts 2024-09-17T07:06:32.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:32 smithi100 ceph-mon[29092]: 6.c scrub ok 2024-09-17T07:06:32.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:32 smithi100 ceph-mon[29092]: 7.12 scrub starts 2024-09-17T07:06:32.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:32 smithi100 ceph-mon[29092]: 2.5 scrub starts 2024-09-17T07:06:32.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:32 smithi100 ceph-mon[29092]: 2.5 scrub ok 2024-09-17T07:06:32.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:32 smithi100 ceph-mon[39462]: 6.c scrub starts 2024-09-17T07:06:32.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:32 smithi100 ceph-mon[39462]: 6.c scrub ok 2024-09-17T07:06:32.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:32 smithi100 ceph-mon[39462]: 7.12 scrub starts 2024-09-17T07:06:32.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:32 smithi100 ceph-mon[39462]: 2.5 scrub starts 2024-09-17T07:06:32.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:32 smithi100 ceph-mon[39462]: 2.5 scrub ok 2024-09-17T07:06:32.466 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:06:32.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:32.496 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:32.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:06:32.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:32 smithi135 ceph-mon[33238]: 6.c scrub starts 2024-09-17T07:06:32.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:32 smithi135 ceph-mon[33238]: 6.c scrub ok 2024-09-17T07:06:32.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:32 smithi135 ceph-mon[33238]: 7.12 scrub starts 2024-09-17T07:06:32.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:32 smithi135 ceph-mon[33238]: 2.5 scrub starts 2024-09-17T07:06:32.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:32 smithi135 ceph-mon[33238]: 2.5 scrub ok 2024-09-17T07:06:32.597 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:06:32.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:32.629 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:32.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:06:32.694 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-17T07:06:32.730 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:06:32.782 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:32.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:32.784 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:06:32.884 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:06:32.912 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:32.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:32.913 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:06:33.013 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:06:33.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:33.045 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:33.045 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:06:33.146 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:06:33.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:33.208 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:33.208 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:06:33.309 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:06:33.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:33.346 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:33.347 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:06:33.447 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:06:33.468 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:33.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:33.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:06:33.570 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:06:33.603 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:33.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:33.604 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:06:33.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:33 smithi100 ceph-mon[29092]: 7.12 scrub ok 2024-09-17T07:06:33.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:33 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:06:33.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:33 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:06:33.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:33 smithi100 ceph-mon[29092]: 6.1c scrub starts 2024-09-17T07:06:33.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:33 smithi100 ceph-mon[29092]: 6.1c scrub ok 2024-09-17T07:06:33.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:33 smithi100 ceph-mon[29092]: pgmap v188: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 640 B/s rd, 0 B/s wr, 364 op/s 2024-09-17T07:06:33.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:33 smithi100 ceph-mon[39462]: 7.12 scrub ok 2024-09-17T07:06:33.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:33 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:06:33.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:33 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:06:33.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:33 smithi100 ceph-mon[39462]: 6.1c scrub starts 2024-09-17T07:06:33.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:33 smithi100 ceph-mon[39462]: 6.1c scrub ok 2024-09-17T07:06:33.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:33 smithi100 ceph-mon[39462]: pgmap v188: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 640 B/s rd, 0 B/s wr, 364 op/s 2024-09-17T07:06:33.705 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:06:33.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:33.743 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:33.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:06:33.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:33 smithi135 ceph-mon[33238]: 7.12 scrub ok 2024-09-17T07:06:33.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:33 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:06:33.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:33 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:06:33.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:33 smithi135 ceph-mon[33238]: 6.1c scrub starts 2024-09-17T07:06:33.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:33 smithi135 ceph-mon[33238]: 6.1c scrub ok 2024-09-17T07:06:33.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:33 smithi135 ceph-mon[33238]: pgmap v188: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 640 B/s rd, 0 B/s wr, 364 op/s 2024-09-17T07:06:33.844 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:06:33.874 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:33.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:33.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:06:33.976 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:06:34.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:34.015 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:34.015 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:06:34.116 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:06:34.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:34.158 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:34.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:06:34.259 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:06:34.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:34.300 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:34.301 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:06:34.402 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:06:34.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:34.439 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:34.439 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:06:34.540 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:06:34.574 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:34.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:34.575 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:06:34.629 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:34 smithi100 ceph-mon[29092]: 6.d scrub starts 2024-09-17T07:06:34.629 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:34 smithi100 ceph-mon[29092]: 6.d scrub ok 2024-09-17T07:06:34.629 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:34 smithi100 ceph-mon[29092]: 5.10 scrub starts 2024-09-17T07:06:34.629 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:34 smithi100 ceph-mon[29092]: 5.10 scrub ok 2024-09-17T07:06:34.629 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:34 smithi100 ceph-mon[29092]: 2.a scrub starts 2024-09-17T07:06:34.629 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:34 smithi100 ceph-mon[29092]: 2.a scrub ok 2024-09-17T07:06:34.630 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:34 smithi100 ceph-mon[39462]: 6.d scrub starts 2024-09-17T07:06:34.630 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:34 smithi100 ceph-mon[39462]: 6.d scrub ok 2024-09-17T07:06:34.630 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:34 smithi100 ceph-mon[39462]: 5.10 scrub starts 2024-09-17T07:06:34.630 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:34 smithi100 ceph-mon[39462]: 5.10 scrub ok 2024-09-17T07:06:34.630 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:34 smithi100 ceph-mon[39462]: 2.a scrub starts 2024-09-17T07:06:34.630 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:34 smithi100 ceph-mon[39462]: 2.a scrub ok 2024-09-17T07:06:34.675 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:06:34.712 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:34.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:34.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:06:34.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:34 smithi135 ceph-mon[33238]: 6.d scrub starts 2024-09-17T07:06:34.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:34 smithi135 ceph-mon[33238]: 6.d scrub ok 2024-09-17T07:06:34.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:34 smithi135 ceph-mon[33238]: 5.10 scrub starts 2024-09-17T07:06:34.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:34 smithi135 ceph-mon[33238]: 5.10 scrub ok 2024-09-17T07:06:34.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:34 smithi135 ceph-mon[33238]: 2.a scrub starts 2024-09-17T07:06:34.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:34 smithi135 ceph-mon[33238]: 2.a scrub ok 2024-09-17T07:06:34.813 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:06:34.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:34.837 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:34.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:06:34.938 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:06:34.973 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:34.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:34.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:06:35.074 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:06:35.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:35.106 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:35.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:06:35.207 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:06:35.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:35.241 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:35.241 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:06:35.342 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:06:35.378 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:35.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:35.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:06:35.480 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:06:35.515 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:35.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:35.517 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:06:35.610 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:35 smithi100 ceph-mon[29092]: 4.1f scrub starts 2024-09-17T07:06:35.610 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:35 smithi100 ceph-mon[29092]: 4.1f scrub ok 2024-09-17T07:06:35.611 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:35 smithi100 ceph-mon[29092]: 5.1b scrub starts 2024-09-17T07:06:35.611 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:35 smithi100 ceph-mon[29092]: 5.1b scrub ok 2024-09-17T07:06:35.611 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:35 smithi100 ceph-mon[29092]: 5.d scrub starts 2024-09-17T07:06:35.611 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:35 smithi100 ceph-mon[29092]: 5.d scrub ok 2024-09-17T07:06:35.611 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:35 smithi100 ceph-mon[29092]: pgmap v189: 158 pgs: 158 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 406 op/s 2024-09-17T07:06:35.611 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:35 smithi100 ceph-mon[39462]: 4.1f scrub starts 2024-09-17T07:06:35.611 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:35 smithi100 ceph-mon[39462]: 4.1f scrub ok 2024-09-17T07:06:35.611 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:35 smithi100 ceph-mon[39462]: 5.1b scrub starts 2024-09-17T07:06:35.611 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:35 smithi100 ceph-mon[39462]: 5.1b scrub ok 2024-09-17T07:06:35.611 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:35 smithi100 ceph-mon[39462]: 5.d scrub starts 2024-09-17T07:06:35.612 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:35 smithi100 ceph-mon[39462]: 5.d scrub ok 2024-09-17T07:06:35.612 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:35 smithi100 ceph-mon[39462]: pgmap v189: 158 pgs: 158 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 406 op/s 2024-09-17T07:06:35.617 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:06:35.639 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:35.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:35.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:06:35.741 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:06:35.780 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:35.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:35.781 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:06:35.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:35 smithi135 ceph-mon[33238]: 4.1f scrub starts 2024-09-17T07:06:35.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:35 smithi135 ceph-mon[33238]: 4.1f scrub ok 2024-09-17T07:06:35.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:35 smithi135 ceph-mon[33238]: 5.1b scrub starts 2024-09-17T07:06:35.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:35 smithi135 ceph-mon[33238]: 5.1b scrub ok 2024-09-17T07:06:35.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:35 smithi135 ceph-mon[33238]: 5.d scrub starts 2024-09-17T07:06:35.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:35 smithi135 ceph-mon[33238]: 5.d scrub ok 2024-09-17T07:06:35.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:35 smithi135 ceph-mon[33238]: pgmap v189: 158 pgs: 158 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 406 op/s 2024-09-17T07:06:35.882 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:06:35.915 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:35.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:35.916 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:06:36.017 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:06:36.045 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:36.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:36.046 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:06:36.147 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:06:36.185 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:36.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:36.186 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:06:36.287 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:06:36.308 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:36.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:36.309 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:06:36.330 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:06:35 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:06:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:06:36.410 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:06:36.440 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:36.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:36.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:06:36.542 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:06:36.576 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:36.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:36.578 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:06:36.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:36 smithi100 ceph-mon[29092]: 4.15 scrub starts 2024-09-17T07:06:36.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:36 smithi100 ceph-mon[29092]: 4.15 scrub ok 2024-09-17T07:06:36.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:36 smithi100 ceph-mon[39462]: 4.15 scrub starts 2024-09-17T07:06:36.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:36 smithi100 ceph-mon[39462]: 4.15 scrub ok 2024-09-17T07:06:36.679 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:06:36.700 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:36.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:36.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:06:36.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:36 smithi135 ceph-mon[33238]: 4.15 scrub starts 2024-09-17T07:06:36.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:36 smithi135 ceph-mon[33238]: 4.15 scrub ok 2024-09-17T07:06:36.803 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:06:36.838 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:36.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:36.839 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:06:36.940 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:06:36.962 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:36.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:36.963 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:06:37.063 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:06:37.098 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:37.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:37.100 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:06:37.201 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:06:37.222 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:37.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:37.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:06:37.323 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:06:37.359 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:37.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:37.360 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:06:37.460 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:06:37.488 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:37.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:37.489 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:06:37.590 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:06:37.620 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:37.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:37.621 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:06:37.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:37 smithi100 ceph-mon[29092]: 6.14 scrub starts 2024-09-17T07:06:37.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:37 smithi100 ceph-mon[29092]: 6.14 scrub ok 2024-09-17T07:06:37.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:37 smithi100 ceph-mon[29092]: 4.10 scrub starts 2024-09-17T07:06:37.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:37 smithi100 ceph-mon[29092]: 4.10 scrub ok 2024-09-17T07:06:37.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:37 smithi100 ceph-mon[29092]: pgmap v190: 158 pgs: 158 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 246 op/s 2024-09-17T07:06:37.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:37 smithi100 ceph-mon[39462]: 6.14 scrub starts 2024-09-17T07:06:37.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:37 smithi100 ceph-mon[39462]: 6.14 scrub ok 2024-09-17T07:06:37.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:37 smithi100 ceph-mon[39462]: 4.10 scrub starts 2024-09-17T07:06:37.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:37 smithi100 ceph-mon[39462]: 4.10 scrub ok 2024-09-17T07:06:37.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:37 smithi100 ceph-mon[39462]: pgmap v190: 158 pgs: 158 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 246 op/s 2024-09-17T07:06:37.722 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:06:37.762 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:37.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:37.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:06:37.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:37 smithi135 ceph-mon[33238]: 6.14 scrub starts 2024-09-17T07:06:37.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:37 smithi135 ceph-mon[33238]: 6.14 scrub ok 2024-09-17T07:06:37.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:37 smithi135 ceph-mon[33238]: 4.10 scrub starts 2024-09-17T07:06:37.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:37 smithi135 ceph-mon[33238]: 4.10 scrub ok 2024-09-17T07:06:37.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:37 smithi135 ceph-mon[33238]: pgmap v190: 158 pgs: 158 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 246 op/s 2024-09-17T07:06:37.864 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:06:37.895 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:37.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:37.897 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:06:37.997 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:06:38.026 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:38.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:38.027 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:06:38.128 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:06:38.152 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:38.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:38.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:06:38.254 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:06:38.276 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:38.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:38.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:06:38.378 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:06:38.406 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:38.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:38.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:06:38.508 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:06:38.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:38 smithi135 ceph-mon[33238]: 2.3 scrub starts 2024-09-17T07:06:38.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:38 smithi135 ceph-mon[33238]: 2.3 scrub ok 2024-09-17T07:06:38.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:38 smithi135 ceph-mon[33238]: 7.8 deep-scrub starts 2024-09-17T07:06:38.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:38 smithi135 ceph-mon[33238]: 2.7 scrub starts 2024-09-17T07:06:38.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:38 smithi135 ceph-mon[33238]: 2.7 scrub ok 2024-09-17T07:06:38.544 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:06:38 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:06:38.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:38.554 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:38.554 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:06:38.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:38 smithi100 ceph-mon[29092]: 2.3 scrub starts 2024-09-17T07:06:38.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:38 smithi100 ceph-mon[29092]: 2.3 scrub ok 2024-09-17T07:06:38.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:38 smithi100 ceph-mon[29092]: 7.8 deep-scrub starts 2024-09-17T07:06:38.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:38 smithi100 ceph-mon[29092]: 2.7 scrub starts 2024-09-17T07:06:38.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:38 smithi100 ceph-mon[29092]: 2.7 scrub ok 2024-09-17T07:06:38.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:38 smithi100 ceph-mon[39462]: 2.3 scrub starts 2024-09-17T07:06:38.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:38 smithi100 ceph-mon[39462]: 2.3 scrub ok 2024-09-17T07:06:38.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:38 smithi100 ceph-mon[39462]: 7.8 deep-scrub starts 2024-09-17T07:06:38.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:38 smithi100 ceph-mon[39462]: 2.7 scrub starts 2024-09-17T07:06:38.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:38 smithi100 ceph-mon[39462]: 2.7 scrub ok 2024-09-17T07:06:38.655 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:06:38.693 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:38.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:38.694 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:06:38.804 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:06:38.833 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:38.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:38.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:06:38.935 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:06:38.970 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:38.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:38.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:06:39.072 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:06:39.109 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:39.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:39.110 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:06:39.211 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:06:39.249 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:39.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:39.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:06:39.351 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:06:39.387 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:39.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:39.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:06:39.489 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:06:39.522 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:39.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:39.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:06:39.624 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:06:39.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:39 smithi100 ceph-mon[29092]: 7.9 scrub starts 2024-09-17T07:06:39.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:39 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:06:39.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:39 smithi100 ceph-mon[29092]: pgmap v191: 158 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 156 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 791 op/s 2024-09-17T07:06:39.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:39 smithi100 ceph-mon[39462]: 7.9 scrub starts 2024-09-17T07:06:39.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:39 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:06:39.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:39 smithi100 ceph-mon[39462]: pgmap v191: 158 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 156 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 791 op/s 2024-09-17T07:06:39.650 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:39.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:39.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:06:39.752 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:06:39.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:39.790 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:39.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:06:39.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:39 smithi135 ceph-mon[33238]: 7.9 scrub starts 2024-09-17T07:06:39.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:39 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:06:39.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:39 smithi135 ceph-mon[33238]: pgmap v191: 158 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 156 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 791 op/s 2024-09-17T07:06:39.892 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:06:39.923 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:39.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:39.924 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:06:40.025 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:06:40.065 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:40.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:40.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:06:40.167 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:06:40.196 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:40.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:40.197 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:06:40.298 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:06:40.336 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:40.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:40.337 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:06:40.438 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:06:40.484 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:40.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:40.486 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:06:40.586 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:06:40.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:40 smithi100 ceph-mon[29092]: 4.9 deep-scrub starts 2024-09-17T07:06:40.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:40 smithi100 ceph-mon[29092]: 4.9 deep-scrub ok 2024-09-17T07:06:40.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:40 smithi100 ceph-mon[29092]: 7.9 scrub ok 2024-09-17T07:06:40.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:40 smithi100 ceph-mon[29092]: 2.c deep-scrub starts 2024-09-17T07:06:40.629 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:40 smithi100 ceph-mon[29092]: 2.c deep-scrub ok 2024-09-17T07:06:40.629 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:40 smithi100 ceph-mon[29092]: 2.e scrub starts 2024-09-17T07:06:40.629 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:40 smithi100 ceph-mon[29092]: 2.e scrub ok 2024-09-17T07:06:40.629 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:40 smithi100 ceph-mon[29092]: 7.5 deep-scrub starts 2024-09-17T07:06:40.629 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:40 smithi100 ceph-mon[29092]: 5.12 scrub starts 2024-09-17T07:06:40.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:40 smithi100 ceph-mon[39462]: 4.9 deep-scrub starts 2024-09-17T07:06:40.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:40 smithi100 ceph-mon[39462]: 4.9 deep-scrub ok 2024-09-17T07:06:40.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:40 smithi100 ceph-mon[39462]: 7.9 scrub ok 2024-09-17T07:06:40.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:40 smithi100 ceph-mon[39462]: 2.c deep-scrub starts 2024-09-17T07:06:40.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:40 smithi100 ceph-mon[39462]: 2.c deep-scrub ok 2024-09-17T07:06:40.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:40 smithi100 ceph-mon[39462]: 2.e scrub starts 2024-09-17T07:06:40.630 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:40 smithi100 ceph-mon[39462]: 2.e scrub ok 2024-09-17T07:06:40.630 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:40 smithi100 ceph-mon[39462]: 7.5 deep-scrub starts 2024-09-17T07:06:40.630 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:40 smithi100 ceph-mon[39462]: 5.12 scrub starts 2024-09-17T07:06:40.635 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:40.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:40.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:06:40.736 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:06:40.768 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:40.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:40.769 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:06:40.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:40 smithi135 ceph-mon[33238]: 4.9 deep-scrub starts 2024-09-17T07:06:40.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:40 smithi135 ceph-mon[33238]: 4.9 deep-scrub ok 2024-09-17T07:06:40.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:40 smithi135 ceph-mon[33238]: 7.9 scrub ok 2024-09-17T07:06:40.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:40 smithi135 ceph-mon[33238]: 2.c deep-scrub starts 2024-09-17T07:06:40.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:40 smithi135 ceph-mon[33238]: 2.c deep-scrub ok 2024-09-17T07:06:40.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:40 smithi135 ceph-mon[33238]: 2.e scrub starts 2024-09-17T07:06:40.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:40 smithi135 ceph-mon[33238]: 2.e scrub ok 2024-09-17T07:06:40.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:40 smithi135 ceph-mon[33238]: 7.5 deep-scrub starts 2024-09-17T07:06:40.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:40 smithi135 ceph-mon[33238]: 5.12 scrub starts 2024-09-17T07:06:40.869 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:06:40.895 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:40.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:40.896 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:06:40.997 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:06:41.028 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:41.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:41.030 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:06:41.130 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:06:41.171 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:41.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:41.172 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:06:41.272 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:06:41.302 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:41.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:41.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:06:41.404 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:06:41.431 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:41.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:41.432 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:06:41.533 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:06:41.558 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:41.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:41.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:06:41.596 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:41 smithi100 ceph-mon[29092]: 5.12 scrub ok 2024-09-17T07:06:41.597 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:41 smithi100 ceph-mon[29092]: 7.19 deep-scrub starts 2024-09-17T07:06:41.597 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:41 smithi100 ceph-mon[29092]: 2.10 scrub starts 2024-09-17T07:06:41.597 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:41 smithi100 ceph-mon[29092]: 2.10 scrub ok 2024-09-17T07:06:41.597 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:41 smithi100 ceph-mon[29092]: 4.0 scrub starts 2024-09-17T07:06:41.597 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:41 smithi100 ceph-mon[29092]: 4.0 scrub ok 2024-09-17T07:06:41.597 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:41 smithi100 ceph-mon[29092]: 6.12 scrub starts 2024-09-17T07:06:41.597 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:41 smithi100 ceph-mon[29092]: 6.12 scrub ok 2024-09-17T07:06:41.597 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:41 smithi100 ceph-mon[29092]: pgmap v192: 158 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 156 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 931 B/s rd, 0 B/s wr, 703 op/s 2024-09-17T07:06:41.597 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:41 smithi100 ceph-mon[29092]: 2.1a scrub starts 2024-09-17T07:06:41.597 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:41 smithi100 ceph-mon[29092]: 2.1a scrub ok 2024-09-17T07:06:41.598 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:41 smithi100 ceph-mon[39462]: 5.12 scrub ok 2024-09-17T07:06:41.598 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:41 smithi100 ceph-mon[39462]: 7.19 deep-scrub starts 2024-09-17T07:06:41.598 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:41 smithi100 ceph-mon[39462]: 2.10 scrub starts 2024-09-17T07:06:41.598 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:41 smithi100 ceph-mon[39462]: 2.10 scrub ok 2024-09-17T07:06:41.598 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:41 smithi100 ceph-mon[39462]: 4.0 scrub starts 2024-09-17T07:06:41.598 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:41 smithi100 ceph-mon[39462]: 4.0 scrub ok 2024-09-17T07:06:41.598 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:41 smithi100 ceph-mon[39462]: 6.12 scrub starts 2024-09-17T07:06:41.598 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:41 smithi100 ceph-mon[39462]: 6.12 scrub ok 2024-09-17T07:06:41.598 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:41 smithi100 ceph-mon[39462]: pgmap v192: 158 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 156 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 931 B/s rd, 0 B/s wr, 703 op/s 2024-09-17T07:06:41.598 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:41 smithi100 ceph-mon[39462]: 2.1a scrub starts 2024-09-17T07:06:41.599 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:41 smithi100 ceph-mon[39462]: 2.1a scrub ok 2024-09-17T07:06:41.660 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:06:41.688 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:41.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:41.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:06:41.763 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-17T07:06:41.764 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:06:41.789 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:06:41.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:41 smithi135 ceph-mon[33238]: 5.12 scrub ok 2024-09-17T07:06:41.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:41 smithi135 ceph-mon[33238]: 7.19 deep-scrub starts 2024-09-17T07:06:41.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:41 smithi135 ceph-mon[33238]: 2.10 scrub starts 2024-09-17T07:06:41.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:41 smithi135 ceph-mon[33238]: 2.10 scrub ok 2024-09-17T07:06:41.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:41 smithi135 ceph-mon[33238]: 4.0 scrub starts 2024-09-17T07:06:41.798 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:41 smithi135 ceph-mon[33238]: 4.0 scrub ok 2024-09-17T07:06:41.798 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:41 smithi135 ceph-mon[33238]: 6.12 scrub starts 2024-09-17T07:06:41.798 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:41 smithi135 ceph-mon[33238]: 6.12 scrub ok 2024-09-17T07:06:41.798 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:41 smithi135 ceph-mon[33238]: pgmap v192: 158 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 156 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 931 B/s rd, 0 B/s wr, 703 op/s 2024-09-17T07:06:41.798 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:41 smithi135 ceph-mon[33238]: 2.1a scrub starts 2024-09-17T07:06:41.798 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:41 smithi135 ceph-mon[33238]: 2.1a scrub ok 2024-09-17T07:06:41.844 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:41.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:41.845 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:06:41.946 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:06:41.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:41.986 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:41.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:06:42.009 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:06:42.009 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:06:42.046 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1: {} 2024-09-17T07:06:42.046 INFO:teuthology.orchestra.run.smithi100.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:06:42.074 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:06:42.074 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:06:42.087 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:06:42.101 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:06:42.101 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:06:42.116 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:42.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:42.117 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:06:42.129 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:06:42.129 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:06:42.164 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:06:42.165 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:06:42.200 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:06:42.200 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:06:42.218 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:06:42.228 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:06:42.229 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:06:42.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:42.256 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:42.256 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:06:42.357 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:06:42.383 INFO:teuthology.orchestra.run.smithi100.stderr:reweighted osd.4 to 1 (10000) 2024-09-17T07:06:42.392 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:42.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:42.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:06:42.494 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:06:42.502 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:42 smithi100 ceph-mon[29092]: 6.2 scrub starts 2024-09-17T07:06:42.503 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:42 smithi100 ceph-mon[29092]: 6.2 scrub ok 2024-09-17T07:06:42.503 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:42 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2682420830' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-17T07:06:42.503 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:42 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1346186885' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:06:42.503 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:42 smithi100 ceph-mon[39462]: 6.2 scrub starts 2024-09-17T07:06:42.503 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:42 smithi100 ceph-mon[39462]: 6.2 scrub ok 2024-09-17T07:06:42.503 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:42 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2682420830' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-17T07:06:42.503 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:42 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1346186885' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:06:42.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:42.535 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:42.535 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:06:42.636 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:06:42.650 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-17T07:06:42.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:42.666 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:42.667 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:06:42.767 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:06:42.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:42 smithi135 ceph-mon[33238]: 6.2 scrub starts 2024-09-17T07:06:42.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:42 smithi135 ceph-mon[33238]: 6.2 scrub ok 2024-09-17T07:06:42.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:42 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2682420830' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-17T07:06:42.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:42 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1346186885' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:06:42.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:42.832 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:42.833 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:06:42.933 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:06:42.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:42.967 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:42.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:06:43.068 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:06:43.105 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:43.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:43.106 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:06:43.207 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:06:43.239 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:43.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:43.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:06:43.320 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd reweight 5 1 2024-09-17T07:06:43.342 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:06:43.372 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:43.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:43.373 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:06:43.474 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:06:43.503 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:06:43.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:43.533 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:43.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:06:43.608 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:43 smithi100 ceph-mon[29092]: 4.d scrub starts 2024-09-17T07:06:43.608 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:43 smithi100 ceph-mon[29092]: 4.d scrub ok 2024-09-17T07:06:43.608 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:43 smithi100 ceph-mon[29092]: 5.1d scrub starts 2024-09-17T07:06:43.608 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:43 smithi100 ceph-mon[29092]: 5.1d scrub ok 2024-09-17T07:06:43.608 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:43 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2682420830' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-17T07:06:43.608 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:43 smithi100 ceph-mon[29092]: osdmap e151: 8 total, 8 up, 8 in 2024-09-17T07:06:43.608 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:43 smithi100 ceph-mon[29092]: pgmap v194: 158 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 156 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 659 op/s 2024-09-17T07:06:43.609 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:43 smithi100 ceph-mon[39462]: 4.d scrub starts 2024-09-17T07:06:43.609 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:43 smithi100 ceph-mon[39462]: 4.d scrub ok 2024-09-17T07:06:43.609 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:43 smithi100 ceph-mon[39462]: 5.1d scrub starts 2024-09-17T07:06:43.609 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:43 smithi100 ceph-mon[39462]: 5.1d scrub ok 2024-09-17T07:06:43.609 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:43 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2682420830' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-17T07:06:43.609 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:43 smithi100 ceph-mon[39462]: osdmap e151: 8 total, 8 up, 8 in 2024-09-17T07:06:43.609 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:43 smithi100 ceph-mon[39462]: pgmap v194: 158 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 156 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 659 op/s 2024-09-17T07:06:43.634 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:06:43.674 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:43.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:43.675 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:06:43.775 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:06:43.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:43 smithi135 ceph-mon[33238]: 4.d scrub starts 2024-09-17T07:06:43.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:43 smithi135 ceph-mon[33238]: 4.d scrub ok 2024-09-17T07:06:43.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:43 smithi135 ceph-mon[33238]: 5.1d scrub starts 2024-09-17T07:06:43.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:43 smithi135 ceph-mon[33238]: 5.1d scrub ok 2024-09-17T07:06:43.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:43 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2682420830' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-17T07:06:43.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:43 smithi135 ceph-mon[33238]: osdmap e151: 8 total, 8 up, 8 in 2024-09-17T07:06:43.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:43 smithi135 ceph-mon[33238]: pgmap v194: 158 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 156 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 659 op/s 2024-09-17T07:06:43.808 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:43.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:43.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:06:43.910 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:06:43.939 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:43.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:43.940 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:06:44.040 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:06:44.076 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:44.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:44.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:06:44.178 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:06:44.207 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:44.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:44.208 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:06:44.309 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:06:44.340 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:44.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:44.341 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:06:44.442 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:06:44.477 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:44.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:44.478 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:06:44.579 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:06:44.608 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:44 smithi100 ceph-mon[29092]: 6.e scrub starts 2024-09-17T07:06:44.608 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:44 smithi100 ceph-mon[29092]: 6.e scrub ok 2024-09-17T07:06:44.608 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:44 smithi100 ceph-mon[29092]: 2.2 scrub starts 2024-09-17T07:06:44.608 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:44 smithi100 ceph-mon[29092]: 2.2 scrub ok 2024-09-17T07:06:44.608 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:44 smithi100 ceph-mon[29092]: 4.17 scrub starts 2024-09-17T07:06:44.608 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:44 smithi100 ceph-mon[29092]: 4.17 scrub ok 2024-09-17T07:06:44.609 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:44 smithi100 ceph-mon[29092]: 2.17 scrub starts 2024-09-17T07:06:44.609 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:44 smithi100 ceph-mon[29092]: 2.17 scrub ok 2024-09-17T07:06:44.609 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:44 smithi100 ceph-mon[39462]: 6.e scrub starts 2024-09-17T07:06:44.609 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:44 smithi100 ceph-mon[39462]: 6.e scrub ok 2024-09-17T07:06:44.609 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:44 smithi100 ceph-mon[39462]: 2.2 scrub starts 2024-09-17T07:06:44.609 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:44 smithi100 ceph-mon[39462]: 2.2 scrub ok 2024-09-17T07:06:44.610 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:44 smithi100 ceph-mon[39462]: 4.17 scrub starts 2024-09-17T07:06:44.610 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:44 smithi100 ceph-mon[39462]: 4.17 scrub ok 2024-09-17T07:06:44.610 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:44 smithi100 ceph-mon[39462]: 2.17 scrub starts 2024-09-17T07:06:44.610 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:44 smithi100 ceph-mon[39462]: 2.17 scrub ok 2024-09-17T07:06:44.614 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:44.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:44.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:06:44.717 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:06:44.743 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:44.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:44.744 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:06:44.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:44 smithi135 ceph-mon[33238]: 6.e scrub starts 2024-09-17T07:06:44.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:44 smithi135 ceph-mon[33238]: 6.e scrub ok 2024-09-17T07:06:44.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:44 smithi135 ceph-mon[33238]: 2.2 scrub starts 2024-09-17T07:06:44.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:44 smithi135 ceph-mon[33238]: 2.2 scrub ok 2024-09-17T07:06:44.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:44 smithi135 ceph-mon[33238]: 4.17 scrub starts 2024-09-17T07:06:44.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:44 smithi135 ceph-mon[33238]: 4.17 scrub ok 2024-09-17T07:06:44.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:44 smithi135 ceph-mon[33238]: 2.17 scrub starts 2024-09-17T07:06:44.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:44 smithi135 ceph-mon[33238]: 2.17 scrub ok 2024-09-17T07:06:44.845 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:06:44.883 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:44.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:44.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:06:44.985 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:06:45.022 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:45.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:45.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:06:45.123 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:06:45.159 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:45.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:45.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:06:45.213 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:06:45.261 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:06:45.292 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:45.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:45.293 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:06:45.394 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:06:45.426 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:45.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:45.427 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:06:45.529 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:06:45.572 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:45.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:45.573 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:06:45.674 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:06:45.702 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:45.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:45.703 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:06:45.804 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:06:45.833 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:45.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:45.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:06:45.934 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:06:45.966 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:06:45.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:45.970 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:45.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:06:46.071 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:06:46.094 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:46.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:46.095 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:06:46.126 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:46 smithi100 ceph-mon[29092]: pgmap v195: 158 pgs: 2 active+clean+scrubbing+deep, 156 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 600 op/s 2024-09-17T07:06:46.126 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:46 smithi100 ceph-mon[29092]: 2.12 scrub starts 2024-09-17T07:06:46.126 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:46 smithi100 ceph-mon[29092]: 2.12 scrub ok 2024-09-17T07:06:46.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:46 smithi100 ceph-mon[39462]: pgmap v195: 158 pgs: 2 active+clean+scrubbing+deep, 156 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 600 op/s 2024-09-17T07:06:46.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:46 smithi100 ceph-mon[39462]: 2.12 scrub starts 2024-09-17T07:06:46.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:46 smithi100 ceph-mon[39462]: 2.12 scrub ok 2024-09-17T07:06:46.127 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:06:45 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:06:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:06:46.197 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:06:46.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:46.260 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:46.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:06:46.361 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:06:46.386 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:46.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:46.387 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:06:46.488 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:06:46.536 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:46.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:46.537 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:06:46.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:46 smithi135 ceph-mon[33238]: pgmap v195: 158 pgs: 2 active+clean+scrubbing+deep, 156 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 600 op/s 2024-09-17T07:06:46.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:46 smithi135 ceph-mon[33238]: 2.12 scrub starts 2024-09-17T07:06:46.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:46 smithi135 ceph-mon[33238]: 2.12 scrub ok 2024-09-17T07:06:46.638 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:06:46.676 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:46.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:46.677 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:06:46.778 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:06:46.805 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:46.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:46.806 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:06:46.907 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:06:46.926 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:06:46.946 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:46.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:46.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:06:47.048 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:06:47.078 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:47.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:47.079 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:06:47.088 INFO:teuthology.orchestra.run.smithi100.stderr:noscrub is set 2024-09-17T07:06:47.180 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:06:47.215 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:47.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:47.216 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:06:47.317 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:06:47.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:47.353 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:47.353 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:06:47.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:47 smithi100 ceph-mon[29092]: 4.4 scrub starts 2024-09-17T07:06:47.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:47 smithi100 ceph-mon[29092]: 4.4 scrub ok 2024-09-17T07:06:47.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:47 smithi100 ceph-mon[29092]: 7.6 scrub starts 2024-09-17T07:06:47.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:47 smithi100 ceph-mon[29092]: 7.7 scrub starts 2024-09-17T07:06:47.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:47 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2082738664' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:06:47.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:47 smithi100 ceph-mon[29092]: 7.8 deep-scrub ok 2024-09-17T07:06:47.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:47 smithi100 ceph-mon[39462]: 4.4 scrub starts 2024-09-17T07:06:47.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:47 smithi100 ceph-mon[39462]: 4.4 scrub ok 2024-09-17T07:06:47.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:47 smithi100 ceph-mon[39462]: 7.6 scrub starts 2024-09-17T07:06:47.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:47 smithi100 ceph-mon[39462]: 7.7 scrub starts 2024-09-17T07:06:47.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:47 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2082738664' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:06:47.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:47 smithi100 ceph-mon[39462]: 7.8 deep-scrub ok 2024-09-17T07:06:47.455 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:06:47.489 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:47.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:47.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:06:47.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:47 smithi135 ceph-mon[33238]: 4.4 scrub starts 2024-09-17T07:06:47.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:47 smithi135 ceph-mon[33238]: 4.4 scrub ok 2024-09-17T07:06:47.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:47 smithi135 ceph-mon[33238]: 7.6 scrub starts 2024-09-17T07:06:47.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:47 smithi135 ceph-mon[33238]: 7.7 scrub starts 2024-09-17T07:06:47.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:47 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2082738664' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:06:47.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:47 smithi135 ceph-mon[33238]: 7.8 deep-scrub ok 2024-09-17T07:06:47.591 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:06:47.636 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:47.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:47.637 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:06:47.694 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-17T07:06:47.694 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-17T07:06:47.694 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-17T07:06:47.694 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-17T07:06:47.695 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-17T07:06:47.738 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:06:47.765 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:47.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:47.766 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:06:47.867 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:06:47.903 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:47.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:47.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:06:48.004 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:06:48.036 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:48.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:48.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:06:48.137 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:06:48.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:48.173 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:48.173 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:06:48.274 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:06:48.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:48.305 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:48.305 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:06:48.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:48 smithi100 ceph-mon[29092]: 3.0 scrub starts 2024-09-17T07:06:48.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:48 smithi100 ceph-mon[29092]: 3.0 scrub ok 2024-09-17T07:06:48.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:48 smithi100 ceph-mon[29092]: 7.7 scrub ok 2024-09-17T07:06:48.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:48 smithi100 ceph-mon[29092]: 7.6 scrub ok 2024-09-17T07:06:48.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:48 smithi100 ceph-mon[29092]: pgmap v196: 158 pgs: 2 active+clean+scrubbing+deep, 156 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 600 op/s 2024-09-17T07:06:48.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:48 smithi100 ceph-mon[29092]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:06:48.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:48 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2082738664' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T07:06:48.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:48 smithi100 ceph-mon[29092]: osdmap e152: 8 total, 8 up, 8 in 2024-09-17T07:06:48.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:48 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:06:48.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:48 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:06:48.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:48 smithi100 ceph-mon[39462]: 3.0 scrub starts 2024-09-17T07:06:48.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:48 smithi100 ceph-mon[39462]: 3.0 scrub ok 2024-09-17T07:06:48.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:48 smithi100 ceph-mon[39462]: 7.7 scrub ok 2024-09-17T07:06:48.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:48 smithi100 ceph-mon[39462]: 7.6 scrub ok 2024-09-17T07:06:48.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:48 smithi100 ceph-mon[39462]: pgmap v196: 158 pgs: 2 active+clean+scrubbing+deep, 156 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 600 op/s 2024-09-17T07:06:48.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:48 smithi100 ceph-mon[39462]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:06:48.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:48 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2082738664' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T07:06:48.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:48 smithi100 ceph-mon[39462]: osdmap e152: 8 total, 8 up, 8 in 2024-09-17T07:06:48.379 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:48 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:06:48.379 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:48 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:06:48.406 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:06:48.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:48.446 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:48.447 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:06:48.543 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:06:48 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:06:48.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:48 smithi135 ceph-mon[33238]: 3.0 scrub starts 2024-09-17T07:06:48.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:48 smithi135 ceph-mon[33238]: 3.0 scrub ok 2024-09-17T07:06:48.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:48 smithi135 ceph-mon[33238]: 7.7 scrub ok 2024-09-17T07:06:48.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:48 smithi135 ceph-mon[33238]: 7.6 scrub ok 2024-09-17T07:06:48.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:48 smithi135 ceph-mon[33238]: pgmap v196: 158 pgs: 2 active+clean+scrubbing+deep, 156 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 600 op/s 2024-09-17T07:06:48.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:48 smithi135 ceph-mon[33238]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:06:48.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:48 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2082738664' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T07:06:48.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:48 smithi135 ceph-mon[33238]: osdmap e152: 8 total, 8 up, 8 in 2024-09-17T07:06:48.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:48 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:06:48.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:48 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:06:48.548 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:06:48.553 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:06:48.575 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:48.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:48.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:06:48.677 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:06:48.707 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:48.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:48.708 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:06:48.809 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:06:48.836 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:48.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:48.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:06:48.938 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:06:48.964 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:48.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:48.966 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:06:49.067 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:06:49.098 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:49.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:49.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:06:49.200 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:06:49.232 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:49.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:49.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:06:49.334 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:06:49.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:49.368 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:49.368 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:06:49.469 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:06:49.502 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:49.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:49.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:06:49.604 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:06:49.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:49.644 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:49.644 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:06:49.745 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:06:49.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:49.805 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:49.805 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:06:49.906 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:06:49.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:49.950 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:49.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:06:50.051 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:06:50.086 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:50.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:50.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:06:50.188 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:06:50.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:50.218 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:50.218 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:06:50.257 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:50 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:06:50.258 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:50 smithi100 ceph-mon[29092]: pgmap v198: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:06:50.258 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:50 smithi100 ceph-mon[29092]: 7.5 deep-scrub ok 2024-09-17T07:06:50.258 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:50 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:06:50.258 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:50 smithi100 ceph-mon[39462]: pgmap v198: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:06:50.258 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:50 smithi100 ceph-mon[39462]: 7.5 deep-scrub ok 2024-09-17T07:06:50.259 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-17T07:06:50.319 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:06:50.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:50.366 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:50.366 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:06:50.467 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:06:50.493 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:50.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:50.494 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:06:50.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:50 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:06:50.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:50 smithi135 ceph-mon[33238]: pgmap v198: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:06:50.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:50 smithi135 ceph-mon[33238]: 7.5 deep-scrub ok 2024-09-17T07:06:50.554 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-17T07:06:50.595 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:06:50.624 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:50.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:50.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:06:50.726 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:06:50.766 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:50.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:50.767 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:06:50.868 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:06:50.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:50.907 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:50.907 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:06:51.008 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:06:51.033 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:51.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:51.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:06:51.135 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:06:51.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:51.183 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:51.184 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:06:51.284 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:06:51.309 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:51.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:51.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:06:51.411 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:06:51.438 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:51.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:51.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:06:51.540 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:06:51.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:51.586 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:51.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:06:51.687 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:06:51.717 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:51.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:51.719 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:06:51.820 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:06:51.863 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:51.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:51.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:06:51.965 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:06:52.012 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:06:52.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:52.016 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:52.016 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:06:52.050 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:06:52.117 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:06:52.138 INFO:teuthology.orchestra.run.smithi100.stderr:reweighted osd.5 to 1 (10000) 2024-09-17T07:06:52.177 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:52.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:52.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:06:52.279 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:06:52.316 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:52.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:52.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:06:52.317 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:52 smithi100 ceph-mon[29092]: pgmap v199: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T07:06:52.317 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:52 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2945562591' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-17T07:06:52.317 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:52 smithi100 ceph-mon[39462]: pgmap v199: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T07:06:52.317 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:52 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2945562591' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-17T07:06:52.419 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:06:52.446 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:52.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:52.447 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:06:52.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:52 smithi135 ceph-mon[33238]: pgmap v199: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T07:06:52.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:52 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2945562591' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-17T07:06:52.548 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:06:52.575 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:52.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:52.576 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:06:52.676 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:06:52.705 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:52.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:52.706 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:06:52.806 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:06:52.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:52.846 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:52.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:06:52.947 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:06:52.983 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:52.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:52.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:06:53.085 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:06:53.113 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:53.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:53.114 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:06:53.215 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:06:53.241 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:53.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:53.242 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:06:53.257 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:53 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2945562591' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-17T07:06:53.257 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:53 smithi100 ceph-mon[29092]: osdmap e153: 8 total, 8 up, 8 in 2024-09-17T07:06:53.257 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:53 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2945562591' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-17T07:06:53.257 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:53 smithi100 ceph-mon[39462]: osdmap e153: 8 total, 8 up, 8 in 2024-09-17T07:06:53.343 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:06:53.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:53.374 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:53.374 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:06:53.475 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:06:53.510 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:53.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:53.512 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:06:53.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:53 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2945562591' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-17T07:06:53.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:53 smithi135 ceph-mon[33238]: osdmap e153: 8 total, 8 up, 8 in 2024-09-17T07:06:53.613 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:06:53.643 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:53.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:53.644 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:06:53.745 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:06:53.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:53.781 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:53.781 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:06:53.882 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:06:53.918 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:53.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:53.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:06:54.020 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:06:54.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:54.045 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:54.045 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:06:54.147 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:06:54.177 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:54.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:54.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:06:54.280 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:06:54.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:54.317 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:54.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:06:54.419 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:06:54.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:54.445 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:54.445 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:06:54.472 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:54 smithi100 ceph-mon[29092]: pgmap v201: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 3.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 638 B/s rd, 0 op/s 2024-09-17T07:06:54.472 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:54 smithi100 ceph-mon[39462]: pgmap v201: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 3.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 638 B/s rd, 0 op/s 2024-09-17T07:06:54.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:54 smithi135 ceph-mon[33238]: pgmap v201: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 3.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 638 B/s rd, 0 op/s 2024-09-17T07:06:54.546 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:06:54.586 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:54.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:54.587 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:06:54.688 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:06:54.720 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:54.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:54.721 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:06:54.822 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:06:54.851 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:54.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:54.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:06:54.953 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:06:54.984 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:54.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:54.985 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:06:55.039 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:06:55.040 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1: {} 2024-09-17T07:06:55.040 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:06:55.040 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:06:55.040 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:06:55.040 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:06:55.040 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:06:55.040 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:06:55.040 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:06:55.040 INFO:teuthology.orchestra.run.smithi100.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:06:55.041 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:06:55.041 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:06:55.041 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:06:55.041 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:06:55.041 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:06:55.041 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:06:55.086 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:06:55.122 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:55.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:55.123 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:06:55.223 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:06:55.269 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:55.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:55.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:06:55.371 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:06:55.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:55 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/3260421948' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:06:55.376 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:55 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/3260421948' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:06:55.398 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:55.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:55.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:06:55.500 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:06:55.539 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:55.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:55.540 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:06:55.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:55 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/3260421948' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:06:55.641 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:06:55.677 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:55.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:55.678 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:06:55.779 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:06:55.816 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:55.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:55.818 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:06:55.920 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:06:55.947 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:55.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:55.948 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:06:56.049 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:06:56.075 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:56.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:56.076 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:06:56.177 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:06:56.179 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:06:55 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:06:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:06:56.204 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:56.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:56.205 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:06:56.306 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:06:56.337 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:56.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:56.338 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:06:56.439 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:06:56.471 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:56.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:56.472 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:06:56.511 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:56 smithi100 ceph-mon[29092]: pgmap v202: 158 pgs: 158 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:06:56.511 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:56 smithi100 ceph-mon[39462]: pgmap v202: 158 pgs: 158 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:06:56.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:56 smithi135 ceph-mon[33238]: pgmap v202: 158 pgs: 158 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:06:56.544 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd reweight 6 1 2024-09-17T07:06:56.573 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:06:56.599 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:56.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:56.600 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:06:56.701 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:06:56.726 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:56.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:56.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:06:56.828 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:06:56.859 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:56.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:56.860 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:06:56.961 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:06:57.001 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:57.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:57.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:06:57.103 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:06:57.135 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:57.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:57.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:06:57.237 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:06:57.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:57.287 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:57.287 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:06:57.388 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:06:57.428 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:57.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:57.429 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:06:57.530 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:06:57.571 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:57.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:57.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:06:57.673 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:06:57.718 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:57.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:57.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:06:57.820 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:06:57.851 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:57.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:57.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:06:57.941 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:06:57.953 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:06:57.979 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:57.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:57.980 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:06:58.053 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:06:58.082 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:06:58.102 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-17T07:06:58.102 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:06:58.106 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:58.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:58.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:06:58.208 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:06:58.256 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:58.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:58.258 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:06:58.359 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:06:58.409 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:58.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:58.410 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:06:58.496 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:58 smithi100 ceph-mon[29092]: pgmap v203: 158 pgs: 158 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T07:06:58.497 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:58 smithi100 ceph-mon[39462]: pgmap v203: 158 pgs: 158 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T07:06:58.511 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:06:58.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:58.539 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:58.540 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:06:58.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:58 smithi135 ceph-mon[33238]: pgmap v203: 158 pgs: 158 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T07:06:58.543 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:06:58 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:06:58.641 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:06:58.677 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:58.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:58.678 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:06:58.779 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:06:58.807 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:58.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:58.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:06:58.909 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:06:58.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:58.971 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:58.971 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:06:59.072 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:06:59.102 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:59.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:59.104 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:06:59.204 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:06:59.245 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:59.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:59.246 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:06:59.258 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:06:59.347 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:06:59.372 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:59.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:59.373 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:06:59.474 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:06:59.510 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:59.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:59.511 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:06:59.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:59 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:06:59.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:06:59 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:06:59.612 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:06:59.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:59 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:06:59.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:06:59 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:06:59.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:59 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:06:59.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:06:59 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:06:59.640 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:06:59.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:59.641 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:06:59.724 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-17T07:06:59.743 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:06:59.778 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:59.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:59.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:06:59.881 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:06:59.916 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:06:59.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:06:59.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:07:00.014 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:07:00.019 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:07:00.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:00.055 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:00.055 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:07:00.156 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:07:00.190 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:00.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:00.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:07:00.292 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:07:00.322 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:00.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:00.323 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:07:00.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:00 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:07:00.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:00 smithi100 ceph-mon[29092]: pgmap v204: 158 pgs: 158 active+clean; 2.8 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1021 B/s rd, 0 B/s wr, 761 op/s 2024-09-17T07:07:00.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:00 smithi100 ceph-mon[29092]: osdmap e154: 8 total, 8 up, 8 in 2024-09-17T07:07:00.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:00 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:07:00.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:00 smithi100 ceph-mon[39462]: pgmap v204: 158 pgs: 158 active+clean; 2.8 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1021 B/s rd, 0 B/s wr, 761 op/s 2024-09-17T07:07:00.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:00 smithi100 ceph-mon[39462]: osdmap e154: 8 total, 8 up, 8 in 2024-09-17T07:07:00.424 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:07:00.460 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:00.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:00.461 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:07:00.469 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:00 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:07:00.469 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:00 smithi135 ceph-mon[33238]: pgmap v204: 158 pgs: 158 active+clean; 2.8 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1021 B/s rd, 0 B/s wr, 761 op/s 2024-09-17T07:07:00.469 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:00 smithi135 ceph-mon[33238]: osdmap e154: 8 total, 8 up, 8 in 2024-09-17T07:07:00.562 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:07:00.583 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:00.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:00.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:07:00.685 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:07:00.725 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:00.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:00.726 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:07:00.827 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:07:00.850 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:00.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:00.851 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:07:00.951 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:07:00.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:00.992 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:00.992 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:07:01.093 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:07:01.123 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:01.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:01.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:07:01.225 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:07:01.255 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:01.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:01.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:07:01.358 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:07:01.391 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:01.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:01.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:07:01.493 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:07:01.525 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:01.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:01.526 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:07:01.627 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:07:01.658 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:01.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:01.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:07:01.760 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:07:01.794 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:01.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:01.795 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:07:01.896 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:07:01.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:01.918 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:01.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:07:02.019 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:07:02.057 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:02.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:02.058 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:07:02.159 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:07:02.194 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:02.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:02.195 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:07:02.296 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:07:02.318 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:02.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:02.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:07:02.380 INFO:teuthology.orchestra.run.smithi135.stdout:ERROR: (22) Invalid argument 2024-09-17T07:07:02.380 INFO:teuthology.orchestra.run.smithi135.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:07:02.380 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:02 smithi135 ceph-mon[33238]: pgmap v206: 158 pgs: 158 active+clean; 2.8 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 851 op/s 2024-09-17T07:07:02.420 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:07:02.458 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:02.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:02.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:07:02.560 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:07:02.592 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:02.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:02.593 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:07:02.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:02 smithi100 ceph-mon[29092]: pgmap v206: 158 pgs: 158 active+clean; 2.8 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 851 op/s 2024-09-17T07:07:02.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:02 smithi100 ceph-mon[39462]: pgmap v206: 158 pgs: 158 active+clean; 2.8 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 851 op/s 2024-09-17T07:07:02.695 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:07:02.730 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:02.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:02.731 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:07:02.832 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:07:02.866 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:02.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:02.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:07:02.968 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:07:02.989 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:02.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:02.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:07:03.092 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:07:03.100 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-17T07:07:03.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:03.126 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:03.126 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:07:03.227 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:07:03.260 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:03.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:03.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:07:03.361 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:07:03.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:03.385 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:03.385 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:07:03.395 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:07:03.486 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:07:03.517 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:03 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:07:03.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:03.522 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:03.523 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:07:03.619 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:03 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:07:03.619 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:03 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:07:03.623 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:07:03.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:03.662 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:03.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:07:03.763 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:07:03.799 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:03.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:03.800 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:07:03.901 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:07:03.936 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:03.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:03.937 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:07:04.038 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:07:04.080 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:04.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:04.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:07:04.182 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:07:04.215 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:04.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:04.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:07:04.318 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:07:04.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:04.356 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:04.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:07:04.458 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:07:04.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:04.502 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:04.503 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:07:04.545 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:04 smithi135 ceph-mon[33238]: pgmap v207: 158 pgs: 158 active+clean; 2.7 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 751 op/s 2024-09-17T07:07:04.603 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:07:04.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:04 smithi100 ceph-mon[29092]: pgmap v207: 158 pgs: 158 active+clean; 2.7 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 751 op/s 2024-09-17T07:07:04.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:04 smithi100 ceph-mon[39462]: pgmap v207: 158 pgs: 158 active+clean; 2.7 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 751 op/s 2024-09-17T07:07:04.635 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:04.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:04.636 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:07:04.737 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:07:04.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:04.776 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:04.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:07:04.877 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:07:04.914 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:04.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:04.916 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:07:05.016 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:07:05.047 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:05.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:05.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:07:05.149 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:07:05.176 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:05.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:05.177 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:07:05.278 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:07:05.321 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:05.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:05.322 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:07:05.423 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:07:05.454 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:05.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:05.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:07:05.556 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:07:05.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:05.591 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:05.591 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:07:05.631 INFO:teuthology.orchestra.run.smithi135.stdout:ERROR: (22) Invalid argument 2024-09-17T07:07:05.631 INFO:teuthology.orchestra.run.smithi135.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:07:05.692 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:07:05.721 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:05.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:05.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:07:05.823 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:07:05.863 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:05.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:05.864 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:07:05.965 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:07:06.005 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:06.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:06.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:07:06.105 INFO:teuthology.orchestra.run.smithi100.stderr:nodeep-scrub is set 2024-09-17T07:07:06.107 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:07:06.144 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:06.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:06.145 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:07:06.246 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:07:06.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:06.279 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:06.279 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:07:06.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:06 smithi135 ceph-mon[33238]: pgmap v208: 158 pgs: 158 active+clean; 2.4 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.20k op/s 2024-09-17T07:07:06.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:06 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2461444157' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:07:06.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:06 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:07:06.376 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:07:05 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:07:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:07:06.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:06 smithi100 ceph-mon[29092]: pgmap v208: 158 pgs: 158 active+clean; 2.4 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.20k op/s 2024-09-17T07:07:06.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:06 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2461444157' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:07:06.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:06 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:07:06.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:06 smithi100 ceph-mon[39462]: pgmap v208: 158 pgs: 158 active+clean; 2.4 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.20k op/s 2024-09-17T07:07:06.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:06 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2461444157' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:07:06.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:06 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:07:06.380 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:07:06.421 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:06.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:06.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:07:06.466 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-17T07:07:06.523 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:07:06.582 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:06.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:06.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:07:06.684 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:07:06.722 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:06.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:06.723 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:07:06.824 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:07:06.828 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:07:06.868 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:06.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:06.869 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:07:06.972 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:07:07.004 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:07.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:07.006 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:07:07.107 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:07:07.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:07.156 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:07.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:07:07.247 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:07 smithi135 ceph-mon[33238]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:07:07.248 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:07 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T07:07:07.248 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:07 smithi135 ceph-mon[33238]: osdmap e155: 8 total, 8 up, 8 in 2024-09-17T07:07:07.257 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:07:07.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:07.294 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:07.295 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:07:07.371 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:07 smithi100 ceph-mon[29092]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:07:07.372 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:07 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T07:07:07.372 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:07 smithi100 ceph-mon[29092]: osdmap e155: 8 total, 8 up, 8 in 2024-09-17T07:07:07.372 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:07 smithi100 ceph-mon[39462]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:07:07.372 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:07 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T07:07:07.372 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:07 smithi100 ceph-mon[39462]: osdmap e155: 8 total, 8 up, 8 in 2024-09-17T07:07:07.396 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:07:07.431 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:07.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:07.432 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:07:07.533 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:07:07.563 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:07.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:07.564 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:07:07.665 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:07:07.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:07.694 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:07.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:07:07.795 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:07:07.831 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:07.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:07.832 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:07:07.933 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:07:07.964 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:07.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:07.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:07:08.065 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:07:08.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:08.104 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:08.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:07:08.205 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:07:08.231 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:08.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:08.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:07:08.263 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:08 smithi135 ceph-mon[33238]: pgmap v210: 158 pgs: 158 active+clean; 2.4 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 699 op/s 2024-09-17T07:07:08.263 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:08 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:07:08.263 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:08 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:07:08.267 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:08 smithi100 ceph-mon[29092]: pgmap v210: 158 pgs: 158 active+clean; 2.4 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 699 op/s 2024-09-17T07:07:08.267 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:08 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:07:08.267 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:08 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:07:08.267 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:08 smithi100 ceph-mon[39462]: pgmap v210: 158 pgs: 158 active+clean; 2.4 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 699 op/s 2024-09-17T07:07:08.267 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:08 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:07:08.267 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:08 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:07:08.319 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:07:08.320 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:07:08.333 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:07:08.360 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1: {} 2024-09-17T07:07:08.360 INFO:teuthology.orchestra.run.smithi100.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:07:08.361 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:08.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:08.362 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:07:08.397 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:07:08.397 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:07:08.427 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:07:08.428 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:07:08.461 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:07:08.461 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:07:08.463 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:07:08.492 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:07:08.493 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:07:08.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:08.504 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:08.504 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:07:08.523 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:07:08.523 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:07:08.536 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:07:08 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:07:08.553 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:07:08.554 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:07:08.605 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:07:08.636 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:08.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:08.637 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:07:08.686 INFO:teuthology.orchestra.run.smithi135.stdout:{ 2024-09-17T07:07:08.687 INFO:teuthology.orchestra.run.smithi135.stdout: "size": 20, 2024-09-17T07:07:08.687 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 600, 2024-09-17T07:07:08.687 INFO:teuthology.orchestra.run.smithi135.stdout: "ops": [ 2024-09-17T07:07:08.687 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.687 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:34813 7.1 7:8c3b067c:::benchmark_data_smithi100_78556_object34812:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-17T07:07:08.687 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:04:45.259742+0000", 2024-09-17T07:07:08.687 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 143.42171648999999, 2024-09-17T07:07:08.687 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.159191205, 2024-09-17T07:07:08.687 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:07:08.687 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:07:08.687 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:07:08.687 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:07:08.688 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:07:08.688 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 34813 2024-09-17T07:07:08.688 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.688 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:07:08.688 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.688 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:07:08.688 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:45.259742+0000", 2024-09-17T07:07:08.688 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:08.688 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.688 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.688 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:07:08.688 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:45.259742+0000", 2024-09-17T07:07:08.689 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:08.689 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.689 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.689 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:07:08.689 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:45.259740+0000", 2024-09-17T07:07:08.689 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:07:08.689 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.689 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.689 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:07:08.689 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:45.259793+0000", 2024-09-17T07:07:08.689 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.2598e-05 2024-09-17T07:07:08.689 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.690 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.690 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:07:08.690 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:45.259805+0000", 2024-09-17T07:07:08.690 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.1752e-05 2024-09-17T07:07:08.690 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.690 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.690 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:07:08.690 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:45.259812+0000", 2024-09-17T07:07:08.690 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.3409999999999997e-06 2024-09-17T07:07:08.690 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.690 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.690 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:07:08.691 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:45.259850+0000", 2024-09-17T07:07:08.691 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.8399000000000003e-05 2024-09-17T07:07:08.691 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.691 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.691 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:07:08.691 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:45.259940+0000", 2024-09-17T07:07:08.691 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.9587999999999997e-05 2024-09-17T07:07:08.691 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.691 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.691 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 3,6", 2024-09-17T07:07:08.691 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:45.260044+0000", 2024-09-17T07:07:08.691 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00010395099999999999 2024-09-17T07:07:08.692 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.692 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.692 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:07:08.692 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:45.261191+0000", 2024-09-17T07:07:08.692 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.001146747 2024-09-17T07:07:08.692 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.692 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.692 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:07:08.692 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:45.261409+0000", 2024-09-17T07:07:08.692 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00021798400000000001 2024-09-17T07:07:08.692 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.692 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.693 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:07:08.693 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:45.418847+0000", 2024-09-17T07:07:08.693 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.15743791400000001 2024-09-17T07:07:08.693 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.693 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.693 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:07:08.693 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:45.418899+0000", 2024-09-17T07:07:08.693 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.2685000000000001e-05 2024-09-17T07:07:08.693 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.693 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.693 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:07:08.693 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:45.418933+0000", 2024-09-17T07:07:08.694 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.3550000000000002e-05 2024-09-17T07:07:08.694 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:08.694 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:07:08.694 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:08.694 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.694 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.694 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:54907 7.1 7:84086a4c:::benchmark_data_smithi100_78556_object54906:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-09-17T07:07:08.694 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:04:57.352937+0000", 2024-09-17T07:07:08.694 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 131.32852102199999, 2024-09-17T07:07:08.694 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.008768664, 2024-09-17T07:07:08.694 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:07:08.694 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:07:08.695 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:07:08.695 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:07:08.695 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:07:08.695 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 54907 2024-09-17T07:07:08.695 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.695 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:07:08.695 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.695 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:07:08.695 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:57.352937+0000", 2024-09-17T07:07:08.695 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:08.695 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.695 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.696 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:07:08.696 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:57.352937+0000", 2024-09-17T07:07:08.696 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:08.696 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.696 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.696 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:07:08.696 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:57.352933+0000", 2024-09-17T07:07:08.696 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999957 2024-09-17T07:07:08.696 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.696 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.696 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:07:08.696 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:57.353001+0000", 2024-09-17T07:07:08.696 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.7626000000000006e-05 2024-09-17T07:07:08.697 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.697 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.697 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:07:08.697 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:57.353029+0000", 2024-09-17T07:07:08.697 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.8904e-05 2024-09-17T07:07:08.697 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.697 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.697 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:07:08.697 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:57.353043+0000", 2024-09-17T07:07:08.697 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.308e-05 2024-09-17T07:07:08.697 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.697 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.697 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:07:08.698 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:57.437796+0000", 2024-09-17T07:07:08.698 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.084753001999999994 2024-09-17T07:07:08.698 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.698 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.698 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:07:08.698 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:57.437809+0000", 2024-09-17T07:07:08.698 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.3312999999999999e-05 2024-09-17T07:07:08.698 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.698 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.698 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:07:08.698 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:58.344080+0000", 2024-09-17T07:07:08.698 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.90627139400000001 2024-09-17T07:07:08.698 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.699 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.699 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:07:08.699 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:58.344251+0000", 2024-09-17T07:07:08.699 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000170977 2024-09-17T07:07:08.699 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.699 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.699 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 3,6", 2024-09-17T07:07:08.699 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:58.344425+0000", 2024-09-17T07:07:08.699 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000173933 2024-09-17T07:07:08.699 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.699 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.699 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:07:08.699 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:58.345775+0000", 2024-09-17T07:07:08.700 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0013498449999999999 2024-09-17T07:07:08.700 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.700 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.700 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:07:08.700 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:58.346094+0000", 2024-09-17T07:07:08.700 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00031945299999999998 2024-09-17T07:07:08.700 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.700 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.700 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:07:08.700 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:58.361572+0000", 2024-09-17T07:07:08.700 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.015477538000000001 2024-09-17T07:07:08.700 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.700 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.701 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:07:08.701 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:58.361669+0000", 2024-09-17T07:07:08.701 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.6887999999999998e-05 2024-09-17T07:07:08.701 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.701 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.701 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:07:08.701 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:58.361706+0000", 2024-09-17T07:07:08.701 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.6869999999999998e-05 2024-09-17T07:07:08.701 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:08.701 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:07:08.701 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:08.701 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.702 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.702 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:78080 7.16 7:6df83576:::benchmark_data_smithi100_78556_object78079:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-17T07:07:08.702 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:05:23.290266+0000", 2024-09-17T07:07:08.702 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 105.391192586, 2024-09-17T07:07:08.702 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.03028199, 2024-09-17T07:07:08.702 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:07:08.702 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:07:08.702 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:07:08.702 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:07:08.702 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:07:08.702 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 78080 2024-09-17T07:07:08.702 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.702 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:07:08.703 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.703 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:07:08.703 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.290266+0000", 2024-09-17T07:07:08.703 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:08.703 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.703 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.703 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:07:08.703 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.290266+0000", 2024-09-17T07:07:08.703 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:08.703 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.703 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.703 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:07:08.703 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.290263+0000", 2024-09-17T07:07:08.704 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999976 2024-09-17T07:07:08.704 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.704 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.704 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:07:08.704 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.290329+0000", 2024-09-17T07:07:08.704 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.5493000000000005e-05 2024-09-17T07:07:08.704 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.704 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.704 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:07:08.704 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.290351+0000", 2024-09-17T07:07:08.704 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.2245000000000001e-05 2024-09-17T07:07:08.704 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.705 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.705 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:07:08.705 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.290362+0000", 2024-09-17T07:07:08.705 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0838e-05 2024-09-17T07:07:08.705 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.705 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.705 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:07:08.705 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.290436+0000", 2024-09-17T07:07:08.705 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.4530999999999995e-05 2024-09-17T07:07:08.705 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.705 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.705 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:07:08.706 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.290449+0000", 2024-09-17T07:07:08.706 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.2791e-05 2024-09-17T07:07:08.706 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.706 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.706 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:07:08.706 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.305250+0000", 2024-09-17T07:07:08.706 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.014800766 2024-09-17T07:07:08.706 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.706 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.706 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:07:08.706 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.305369+0000", 2024-09-17T07:07:08.707 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000118658 2024-09-17T07:07:08.707 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.707 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.707 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,7", 2024-09-17T07:07:08.707 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.305526+0000", 2024-09-17T07:07:08.707 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00015784300000000001 2024-09-17T07:07:08.707 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.707 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.707 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:07:08.707 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.309043+0000", 2024-09-17T07:07:08.707 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0035167050000000002 2024-09-17T07:07:08.707 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.707 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.708 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:07:08.708 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.309240+0000", 2024-09-17T07:07:08.708 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00019642100000000001 2024-09-17T07:07:08.708 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.708 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.708 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:07:08.708 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.320476+0000", 2024-09-17T07:07:08.708 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.01123681 2024-09-17T07:07:08.708 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.708 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.708 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:07:08.708 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.320510+0000", 2024-09-17T07:07:08.709 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.3553000000000002e-05 2024-09-17T07:07:08.709 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.709 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.709 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:07:08.709 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.320547+0000", 2024-09-17T07:07:08.709 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.7527999999999997e-05 2024-09-17T07:07:08.709 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:08.709 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:07:08.709 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:08.709 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.709 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.709 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:78111 7.16 7:69dc329b:::benchmark_data_smithi100_78556_object78110:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-17T07:07:08.709 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:05:23.330298+0000", 2024-09-17T07:07:08.710 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 105.351160429, 2024-09-17T07:07:08.710 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.99037067999999995, 2024-09-17T07:07:08.710 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:07:08.710 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:07:08.710 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:07:08.710 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:07:08.710 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:07:08.710 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 78111 2024-09-17T07:07:08.710 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.710 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:07:08.710 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.710 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:07:08.710 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.330298+0000", 2024-09-17T07:07:08.711 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:08.711 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.711 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.711 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:07:08.711 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.330298+0000", 2024-09-17T07:07:08.711 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:08.711 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.711 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.711 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:07:08.711 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.330295+0000", 2024-09-17T07:07:08.711 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999976 2024-09-17T07:07:08.711 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.712 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.712 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:07:08.712 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.330335+0000", 2024-09-17T07:07:08.712 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.9816e-05 2024-09-17T07:07:08.712 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.712 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.712 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:07:08.712 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.330356+0000", 2024-09-17T07:07:08.712 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.0679e-05 2024-09-17T07:07:08.712 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.712 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.712 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:07:08.712 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.330366+0000", 2024-09-17T07:07:08.713 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0421000000000001e-05 2024-09-17T07:07:08.713 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.713 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.713 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:07:08.713 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.330453+0000", 2024-09-17T07:07:08.713 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.6465000000000006e-05 2024-09-17T07:07:08.713 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.713 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.713 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:07:08.713 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.330468+0000", 2024-09-17T07:07:08.713 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.4861999999999999e-05 2024-09-17T07:07:08.713 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.713 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.714 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:07:08.714 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.306866+0000", 2024-09-17T07:07:08.714 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.97639884300000002 2024-09-17T07:07:08.714 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.714 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.714 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:07:08.714 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.307049+0000", 2024-09-17T07:07:08.714 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00018273700000000001 2024-09-17T07:07:08.714 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.714 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.714 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,7", 2024-09-17T07:07:08.714 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.307202+0000", 2024-09-17T07:07:08.714 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00015266300000000001 2024-09-17T07:07:08.714 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.715 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.715 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:07:08.715 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.309154+0000", 2024-09-17T07:07:08.715 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0019523609999999999 2024-09-17T07:07:08.715 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.715 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.715 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:07:08.715 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.309279+0000", 2024-09-17T07:07:08.715 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00012443400000000001 2024-09-17T07:07:08.715 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.715 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.715 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:07:08.715 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.320587+0000", 2024-09-17T07:07:08.716 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.011307874000000001 2024-09-17T07:07:08.716 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.716 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.716 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:07:08.716 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.320632+0000", 2024-09-17T07:07:08.716 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.5847000000000003e-05 2024-09-17T07:07:08.716 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.716 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.716 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:07:08.716 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.320668+0000", 2024-09-17T07:07:08.716 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.5936999999999998e-05 2024-09-17T07:07:08.716 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:08.716 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:07:08.717 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:08.717 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.717 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.718 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:78143 7.16 7:6a829b84:::benchmark_data_smithi100_78556_object78142:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-17T07:07:08.718 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:05:23.360285+0000", 2024-09-17T07:07:08.718 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 105.321173437, 2024-09-17T07:07:08.718 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.96129615000000002, 2024-09-17T07:07:08.718 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:07:08.718 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:07:08.718 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:07:08.718 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:07:08.718 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:07:08.718 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 78143 2024-09-17T07:07:08.718 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.719 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:07:08.719 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.719 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:07:08.719 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.360285+0000", 2024-09-17T07:07:08.719 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:08.719 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.719 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.719 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:07:08.719 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.360285+0000", 2024-09-17T07:07:08.719 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:08.719 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.719 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.719 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:07:08.720 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.360282+0000", 2024-09-17T07:07:08.720 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999976 2024-09-17T07:07:08.720 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.720 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.720 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:07:08.720 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.360333+0000", 2024-09-17T07:07:08.720 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.1029000000000002e-05 2024-09-17T07:07:08.720 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.720 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.720 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:07:08.720 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.360354+0000", 2024-09-17T07:07:08.720 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.0392999999999999e-05 2024-09-17T07:07:08.720 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.721 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.721 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:07:08.721 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.360364+0000", 2024-09-17T07:07:08.721 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0336999999999999e-05 2024-09-17T07:07:08.721 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.721 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.721 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:07:08.721 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.360436+0000", 2024-09-17T07:07:08.721 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.1639999999999998e-05 2024-09-17T07:07:08.721 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.721 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.721 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:07:08.721 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.360448+0000", 2024-09-17T07:07:08.722 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.1704e-05 2024-09-17T07:07:08.722 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.722 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.722 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:07:08.722 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.308033+0000", 2024-09-17T07:07:08.722 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.94758547000000004 2024-09-17T07:07:08.722 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.722 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.722 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:07:08.722 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.308115+0000", 2024-09-17T07:07:08.722 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.1798000000000005e-05 2024-09-17T07:07:08.722 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.722 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.723 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,7", 2024-09-17T07:07:08.723 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.308243+0000", 2024-09-17T07:07:08.723 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00012773299999999999 2024-09-17T07:07:08.723 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.723 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.723 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:07:08.723 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.311329+0000", 2024-09-17T07:07:08.723 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0030862849999999998 2024-09-17T07:07:08.723 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.723 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.723 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:07:08.723 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.311371+0000", 2024-09-17T07:07:08.723 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.2502000000000001e-05 2024-09-17T07:07:08.724 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.724 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.724 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:07:08.724 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.321532+0000", 2024-09-17T07:07:08.724 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.010161129 2024-09-17T07:07:08.724 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.724 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.724 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:07:08.724 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.321562+0000", 2024-09-17T07:07:08.724 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.9882999999999998e-05 2024-09-17T07:07:08.724 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.724 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.725 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:07:08.725 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.321581+0000", 2024-09-17T07:07:08.725 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.8488e-05 2024-09-17T07:07:08.725 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:08.725 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:07:08.725 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:08.725 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.725 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.725 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:78146 7.16 7:6cf9446e:::benchmark_data_smithi100_78556_object78145:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-17T07:07:08.725 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:05:23.363433+0000", 2024-09-17T07:07:08.725 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 105.31802492, 2024-09-17T07:07:08.725 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.95826878400000004, 2024-09-17T07:07:08.725 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:07:08.726 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:07:08.726 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:07:08.726 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:07:08.726 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:07:08.726 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 78146 2024-09-17T07:07:08.726 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.726 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:07:08.726 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.726 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:07:08.726 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.363433+0000", 2024-09-17T07:07:08.726 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:08.726 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.726 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.727 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:07:08.727 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.363433+0000", 2024-09-17T07:07:08.727 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:08.727 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.727 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.727 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:07:08.727 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.363431+0000", 2024-09-17T07:07:08.727 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999981 2024-09-17T07:07:08.727 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.727 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.727 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:07:08.727 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.363469+0000", 2024-09-17T07:07:08.727 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.8065000000000002e-05 2024-09-17T07:07:08.728 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.728 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.728 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:07:08.728 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.363486+0000", 2024-09-17T07:07:08.728 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.6946e-05 2024-09-17T07:07:08.728 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.728 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.728 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:07:08.728 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.363493+0000", 2024-09-17T07:07:08.728 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.1550000000000001e-06 2024-09-17T07:07:08.728 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.728 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.729 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:07:08.729 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.363553+0000", 2024-09-17T07:07:08.729 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.0066999999999999e-05 2024-09-17T07:07:08.729 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.729 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.729 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:07:08.729 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.363565+0000", 2024-09-17T07:07:08.729 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.1960999999999999e-05 2024-09-17T07:07:08.729 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.729 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.729 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:07:08.729 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.309302+0000", 2024-09-17T07:07:08.729 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.94573720299999997 2024-09-17T07:07:08.730 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.730 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.730 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:07:08.730 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.309414+0000", 2024-09-17T07:07:08.730 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000111467 2024-09-17T07:07:08.730 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.730 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.730 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,7", 2024-09-17T07:07:08.730 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.309535+0000", 2024-09-17T07:07:08.730 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00012128799999999999 2024-09-17T07:07:08.730 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.730 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.730 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:07:08.731 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.311362+0000", 2024-09-17T07:07:08.731 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0018264939999999999 2024-09-17T07:07:08.731 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.731 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.731 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:07:08.731 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.311376+0000", 2024-09-17T07:07:08.731 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.4374e-05 2024-09-17T07:07:08.731 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.731 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.731 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:07:08.731 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.321625+0000", 2024-09-17T07:07:08.731 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.010249121999999999 2024-09-17T07:07:08.731 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.732 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.732 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:07:08.732 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.321668+0000", 2024-09-17T07:07:08.732 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.3028000000000002e-05 2024-09-17T07:07:08.732 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.732 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.732 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:07:08.732 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.321702+0000", 2024-09-17T07:07:08.732 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.3689000000000002e-05 2024-09-17T07:07:08.732 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:08.732 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:07:08.733 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:08.733 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.733 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.733 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:78148 7.16 7:6f8b784c:::benchmark_data_smithi100_78556_object78147:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-17T07:07:08.733 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:05:23.364154+0000", 2024-09-17T07:07:08.733 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 105.3173041, 2024-09-17T07:07:08.733 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.95766116599999995, 2024-09-17T07:07:08.733 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:07:08.733 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:07:08.733 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:07:08.733 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:07:08.733 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:07:08.733 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 78148 2024-09-17T07:07:08.734 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.734 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:07:08.734 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.734 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:07:08.734 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.364154+0000", 2024-09-17T07:07:08.734 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:08.734 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.734 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.734 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:07:08.734 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.364154+0000", 2024-09-17T07:07:08.734 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:08.734 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.735 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.735 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:07:08.735 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.364153+0000", 2024-09-17T07:07:08.735 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999995 2024-09-17T07:07:08.735 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.735 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.735 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:07:08.735 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.364258+0000", 2024-09-17T07:07:08.735 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00010417 2024-09-17T07:07:08.735 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.735 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.736 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:07:08.736 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.364267+0000", 2024-09-17T07:07:08.736 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.0129999999999999e-06 2024-09-17T07:07:08.736 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.736 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.736 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:07:08.736 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.364271+0000", 2024-09-17T07:07:08.736 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.5850000000000001e-06 2024-09-17T07:07:08.736 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.736 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.736 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:07:08.736 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.364314+0000", 2024-09-17T07:07:08.736 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.337e-05 2024-09-17T07:07:08.737 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.737 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.737 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:07:08.737 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.364324+0000", 2024-09-17T07:07:08.737 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.2790000000000004e-06 2024-09-17T07:07:08.737 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.737 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.737 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:07:08.737 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.310480+0000", 2024-09-17T07:07:08.737 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.94615585199999996 2024-09-17T07:07:08.737 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.737 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.738 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:07:08.738 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.310606+0000", 2024-09-17T07:07:08.738 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00012678200000000001 2024-09-17T07:07:08.738 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.738 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.738 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,7", 2024-09-17T07:07:08.738 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.310733+0000", 2024-09-17T07:07:08.738 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00012678200000000001 2024-09-17T07:07:08.738 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.738 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.738 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:07:08.738 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.313206+0000", 2024-09-17T07:07:08.739 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.002472442 2024-09-17T07:07:08.739 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.739 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.739 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:07:08.739 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.313211+0000", 2024-09-17T07:07:08.739 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.9620000000000001e-06 2024-09-17T07:07:08.739 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.739 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.739 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:07:08.739 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.321752+0000", 2024-09-17T07:07:08.739 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0085410239999999995 2024-09-17T07:07:08.739 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.739 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.740 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:07:08.740 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.321782+0000", 2024-09-17T07:07:08.740 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.0269000000000001e-05 2024-09-17T07:07:08.740 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.740 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.740 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:07:08.740 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.321815+0000", 2024-09-17T07:07:08.740 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.3275000000000002e-05 2024-09-17T07:07:08.740 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:08.740 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:07:08.740 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:08.741 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.741 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.741 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:78206 7.16 7:6d56ab44:::benchmark_data_smithi100_78556_object78205:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-17T07:07:08.741 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:05:23.442944+0000", 2024-09-17T07:07:08.741 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 105.238514576, 2024-09-17T07:07:08.741 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.87899096200000004, 2024-09-17T07:07:08.741 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:07:08.741 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:07:08.741 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:07:08.741 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:07:08.741 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:07:08.741 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 78206 2024-09-17T07:07:08.742 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.742 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:07:08.742 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.742 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:07:08.742 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.442944+0000", 2024-09-17T07:07:08.742 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:08.742 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.742 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.742 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:07:08.742 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.442944+0000", 2024-09-17T07:07:08.742 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:08.742 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.742 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.743 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:07:08.743 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.442942+0000", 2024-09-17T07:07:08.743 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:07:08.743 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.743 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.743 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:07:08.743 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.442976+0000", 2024-09-17T07:07:08.743 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.4471999999999998e-05 2024-09-17T07:07:08.743 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.743 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.743 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:07:08.743 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.442986+0000", 2024-09-17T07:07:08.743 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.1600000000000004e-06 2024-09-17T07:07:08.744 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.744 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.744 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:07:08.744 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.442992+0000", 2024-09-17T07:07:08.744 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.3040000000000001e-06 2024-09-17T07:07:08.744 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.744 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.744 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:07:08.744 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.443041+0000", 2024-09-17T07:07:08.744 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.8633e-05 2024-09-17T07:07:08.744 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.744 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.744 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:07:08.745 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.443049+0000", 2024-09-17T07:07:08.745 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.6789999999999994e-06 2024-09-17T07:07:08.745 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.745 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.745 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:07:08.745 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.311410+0000", 2024-09-17T07:07:08.745 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.86836050300000001 2024-09-17T07:07:08.745 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.745 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.745 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:07:08.745 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.311516+0000", 2024-09-17T07:07:08.745 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00010664500000000001 2024-09-17T07:07:08.745 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.746 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.746 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,7", 2024-09-17T07:07:08.746 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.311636+0000", 2024-09-17T07:07:08.746 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000119796 2024-09-17T07:07:08.746 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.746 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.746 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:07:08.746 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.313229+0000", 2024-09-17T07:07:08.746 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0015927389999999999 2024-09-17T07:07:08.746 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.746 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.746 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:07:08.746 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.313950+0000", 2024-09-17T07:07:08.747 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00072075800000000005 2024-09-17T07:07:08.747 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.747 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.747 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:07:08.747 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.321861+0000", 2024-09-17T07:07:08.747 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0079110919999999998 2024-09-17T07:07:08.747 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.747 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.747 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:07:08.747 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.321903+0000", 2024-09-17T07:07:08.747 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.1992e-05 2024-09-17T07:07:08.747 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.748 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.748 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:07:08.748 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.321934+0000", 2024-09-17T07:07:08.748 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.1678999999999999e-05 2024-09-17T07:07:08.748 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:08.748 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:07:08.748 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:08.748 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.748 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.748 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:78211 7.16 7:6dd6ed80:::benchmark_data_smithi100_78556_object78210:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-17T07:07:08.748 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:05:23.448482+0000", 2024-09-17T07:07:08.748 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 105.232976295, 2024-09-17T07:07:08.749 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.873569598, 2024-09-17T07:07:08.749 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:07:08.749 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:07:08.749 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:07:08.749 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:07:08.749 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:07:08.749 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 78211 2024-09-17T07:07:08.749 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.749 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:07:08.749 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.749 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:07:08.749 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.448482+0000", 2024-09-17T07:07:08.749 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:08.750 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.750 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.750 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:07:08.750 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.448482+0000", 2024-09-17T07:07:08.750 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:08.750 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.750 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.750 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:07:08.750 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.448480+0000", 2024-09-17T07:07:08.750 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999981 2024-09-17T07:07:08.750 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.750 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.750 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:07:08.751 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.448529+0000", 2024-09-17T07:07:08.751 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.8891999999999999e-05 2024-09-17T07:07:08.751 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.751 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.751 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:07:08.751 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.448548+0000", 2024-09-17T07:07:08.751 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.9093000000000002e-05 2024-09-17T07:07:08.751 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.751 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.751 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:07:08.751 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.448558+0000", 2024-09-17T07:07:08.751 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0064000000000001e-05 2024-09-17T07:07:08.751 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.752 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.752 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:07:08.752 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.448621+0000", 2024-09-17T07:07:08.752 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.3278e-05 2024-09-17T07:07:08.752 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.752 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.752 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:07:08.752 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.448632+0000", 2024-09-17T07:07:08.752 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0342e-05 2024-09-17T07:07:08.752 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.752 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.752 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:07:08.753 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.312467+0000", 2024-09-17T07:07:08.753 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.86383483800000005 2024-09-17T07:07:08.753 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.753 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.753 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:07:08.753 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.312562+0000", 2024-09-17T07:07:08.753 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.5843999999999999e-05 2024-09-17T07:07:08.753 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.753 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.753 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,7", 2024-09-17T07:07:08.753 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.312644+0000", 2024-09-17T07:07:08.753 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.1299999999999997e-05 2024-09-17T07:07:08.754 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.754 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.754 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:07:08.754 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.314813+0000", 2024-09-17T07:07:08.754 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0021691929999999998 2024-09-17T07:07:08.754 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.754 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.754 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:07:08.754 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.314822+0000", 2024-09-17T07:07:08.754 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.5730000000000008e-06 2024-09-17T07:07:08.754 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.754 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.754 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:07:08.755 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.321970+0000", 2024-09-17T07:07:08.755 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.007148906 2024-09-17T07:07:08.755 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.755 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.755 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:07:08.755 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.322004+0000", 2024-09-17T07:07:08.755 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.3183999999999999e-05 2024-09-17T07:07:08.755 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.755 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.755 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:07:08.755 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.322051+0000", 2024-09-17T07:07:08.755 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.7790000000000002e-05 2024-09-17T07:07:08.755 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:08.756 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:07:08.756 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:08.756 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.756 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.756 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:78218 7.16 7:6e8f285f:::benchmark_data_smithi100_78556_object78217:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-17T07:07:08.756 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:05:23.458027+0000", 2024-09-17T07:07:08.756 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 105.223431447, 2024-09-17T07:07:08.756 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.86415862700000001, 2024-09-17T07:07:08.756 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:07:08.756 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:07:08.756 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:07:08.756 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:07:08.757 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:07:08.757 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 78218 2024-09-17T07:07:08.757 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.757 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:07:08.757 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.757 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:07:08.757 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.458027+0000", 2024-09-17T07:07:08.757 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:08.757 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.757 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.757 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:07:08.757 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.458027+0000", 2024-09-17T07:07:08.757 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:08.757 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.758 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.758 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:07:08.758 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.458025+0000", 2024-09-17T07:07:08.758 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999981 2024-09-17T07:07:08.758 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.758 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.758 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:07:08.758 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.458064+0000", 2024-09-17T07:07:08.758 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.9233000000000002e-05 2024-09-17T07:07:08.758 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.758 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.758 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:07:08.759 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.458087+0000", 2024-09-17T07:07:08.759 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.3467000000000001e-05 2024-09-17T07:07:08.759 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.759 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.759 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:07:08.759 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.458098+0000", 2024-09-17T07:07:08.759 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0664e-05 2024-09-17T07:07:08.759 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.759 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.759 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:07:08.759 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.458156+0000", 2024-09-17T07:07:08.759 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.7648000000000001e-05 2024-09-17T07:07:08.759 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.760 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.760 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:07:08.760 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.458165+0000", 2024-09-17T07:07:08.760 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.7739999999999999e-06 2024-09-17T07:07:08.760 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.760 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.760 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:07:08.760 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.313256+0000", 2024-09-17T07:07:08.760 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.85509072200000003 2024-09-17T07:07:08.760 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.760 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.760 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:07:08.760 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.313330+0000", 2024-09-17T07:07:08.761 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.3697999999999998e-05 2024-09-17T07:07:08.761 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.761 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.761 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,7", 2024-09-17T07:07:08.761 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.313425+0000", 2024-09-17T07:07:08.761 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.4932999999999994e-05 2024-09-17T07:07:08.761 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.761 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.761 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:07:08.761 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.314866+0000", 2024-09-17T07:07:08.761 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.001441775 2024-09-17T07:07:08.761 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.762 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.762 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:07:08.762 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.315863+0000", 2024-09-17T07:07:08.762 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00099606499999999997 2024-09-17T07:07:08.762 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.762 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.762 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:07:08.762 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.322103+0000", 2024-09-17T07:07:08.762 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0062408999999999997 2024-09-17T07:07:08.762 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.762 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.762 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:07:08.762 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.322151+0000", 2024-09-17T07:07:08.763 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.7744999999999998e-05 2024-09-17T07:07:08.763 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.763 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.763 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:07:08.763 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.322185+0000", 2024-09-17T07:07:08.763 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.4107999999999999e-05 2024-09-17T07:07:08.763 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:08.763 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:07:08.763 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:08.763 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.763 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.763 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:78222 7.16 7:6be7bc2c:::benchmark_data_smithi100_78556_object78221:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-17T07:07:08.764 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:05:23.462890+0000", 2024-09-17T07:07:08.764 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 105.218567596, 2024-09-17T07:07:08.764 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.85944416499999998, 2024-09-17T07:07:08.764 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:07:08.764 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:07:08.764 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:07:08.764 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:07:08.764 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:07:08.764 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 78222 2024-09-17T07:07:08.764 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.764 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:07:08.764 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.764 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:07:08.765 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.462890+0000", 2024-09-17T07:07:08.765 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:08.765 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.765 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.765 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:07:08.765 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.462890+0000", 2024-09-17T07:07:08.765 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:08.765 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.765 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.765 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:07:08.765 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.462888+0000", 2024-09-17T07:07:08.765 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999981 2024-09-17T07:07:08.765 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.766 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.766 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:07:08.766 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.462926+0000", 2024-09-17T07:07:08.766 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.79e-05 2024-09-17T07:07:08.766 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.766 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.766 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:07:08.766 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.462943+0000", 2024-09-17T07:07:08.766 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.6422999999999998e-05 2024-09-17T07:07:08.766 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.766 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.766 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:07:08.766 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.462950+0000", 2024-09-17T07:07:08.767 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.4359999999999996e-06 2024-09-17T07:07:08.767 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.767 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.767 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:07:08.767 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.462998+0000", 2024-09-17T07:07:08.767 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.7834000000000003e-05 2024-09-17T07:07:08.767 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.767 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.767 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:07:08.767 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.463009+0000", 2024-09-17T07:07:08.767 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0757999999999999e-05 2024-09-17T07:07:08.767 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.768 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.768 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:07:08.768 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.313991+0000", 2024-09-17T07:07:08.768 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.850981667 2024-09-17T07:07:08.768 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.768 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.768 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:07:08.768 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.314083+0000", 2024-09-17T07:07:08.768 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.2588000000000002e-05 2024-09-17T07:07:08.768 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.768 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.768 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,7", 2024-09-17T07:07:08.768 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.314161+0000", 2024-09-17T07:07:08.769 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.8181000000000002e-05 2024-09-17T07:07:08.769 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.769 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.769 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:07:08.769 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.316972+0000", 2024-09-17T07:07:08.769 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0028111859999999998 2024-09-17T07:07:08.769 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.769 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.769 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:07:08.769 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.316981+0000", 2024-09-17T07:07:08.769 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.7080000000000003e-06 2024-09-17T07:07:08.769 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.770 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.770 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:07:08.770 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.322254+0000", 2024-09-17T07:07:08.770 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0052724169999999997 2024-09-17T07:07:08.770 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.770 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.770 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:07:08.770 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.322300+0000", 2024-09-17T07:07:08.770 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.6822e-05 2024-09-17T07:07:08.770 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.770 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.770 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:07:08.770 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.322335+0000", 2024-09-17T07:07:08.771 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.4248000000000001e-05 2024-09-17T07:07:08.771 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:08.771 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:07:08.771 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:08.771 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.771 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.771 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:78271 7.16 7:6ab95b49:::benchmark_data_smithi100_78556_object78270:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-17T07:07:08.771 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:05:23.519157+0000", 2024-09-17T07:07:08.771 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 105.162300651, 2024-09-17T07:07:08.771 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.80363748199999996, 2024-09-17T07:07:08.771 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:07:08.771 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:07:08.771 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:07:08.772 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:07:08.772 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:07:08.772 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 78271 2024-09-17T07:07:08.772 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.772 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:07:08.772 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.772 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:07:08.772 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.519157+0000", 2024-09-17T07:07:08.772 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:08.772 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.772 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.772 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:07:08.772 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.519157+0000", 2024-09-17T07:07:08.773 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:08.773 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.773 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.773 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:07:08.773 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.519156+0000", 2024-09-17T07:07:08.773 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:07:08.773 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.773 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.773 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:07:08.773 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.519202+0000", 2024-09-17T07:07:08.773 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.5852000000000001e-05 2024-09-17T07:07:08.773 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.773 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.774 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:07:08.774 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.519219+0000", 2024-09-17T07:07:08.774 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.6844e-05 2024-09-17T07:07:08.774 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.774 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.774 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:07:08.774 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.519228+0000", 2024-09-17T07:07:08.774 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.3589999999999997e-06 2024-09-17T07:07:08.774 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.774 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.774 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:07:08.774 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.519279+0000", 2024-09-17T07:07:08.774 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.1073999999999999e-05 2024-09-17T07:07:08.774 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.775 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.775 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:07:08.775 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.519288+0000", 2024-09-17T07:07:08.775 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.2310000000000002e-06 2024-09-17T07:07:08.775 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.775 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.775 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:07:08.775 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.314895+0000", 2024-09-17T07:07:08.775 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.79560634500000005 2024-09-17T07:07:08.775 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.775 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.775 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:07:08.775 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.314975+0000", 2024-09-17T07:07:08.776 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.0292999999999998e-05 2024-09-17T07:07:08.776 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.776 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.776 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,7", 2024-09-17T07:07:08.776 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.315074+0000", 2024-09-17T07:07:08.776 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.8858999999999997e-05 2024-09-17T07:07:08.776 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.776 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.776 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:07:08.776 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.317005+0000", 2024-09-17T07:07:08.776 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.001931568 2024-09-17T07:07:08.776 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.776 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.777 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:07:08.777 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:07:08.778 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.317969+0000", 2024-09-17T07:07:08.779 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00096359500000000003 2024-09-17T07:07:08.779 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.779 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.779 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:07:08.779 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.322724+0000", 2024-09-17T07:07:08.779 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0047548649999999996 2024-09-17T07:07:08.779 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.779 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.779 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:07:08.779 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.322764+0000", 2024-09-17T07:07:08.779 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.9795000000000001e-05 2024-09-17T07:07:08.779 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.780 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.780 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:07:08.780 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.322795+0000", 2024-09-17T07:07:08.780 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.1383000000000001e-05 2024-09-17T07:07:08.780 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:08.780 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:07:08.780 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:08.780 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.780 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.780 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:78274 7.16 7:694e9368:::benchmark_data_smithi100_78556_object78273:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-17T07:07:08.780 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:05:23.522946+0000", 2024-09-17T07:07:08.780 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 105.15851170400001, 2024-09-17T07:07:08.781 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.80410938499999995, 2024-09-17T07:07:08.781 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:07:08.781 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:07:08.781 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:07:08.781 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:07:08.781 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:07:08.781 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 78274 2024-09-17T07:07:08.781 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.781 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:07:08.781 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.781 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:07:08.781 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.522946+0000", 2024-09-17T07:07:08.781 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:08.782 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.782 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.782 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:07:08.782 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.522946+0000", 2024-09-17T07:07:08.782 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:08.782 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.782 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.782 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:07:08.782 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.522945+0000", 2024-09-17T07:07:08.782 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.999999 2024-09-17T07:07:08.782 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.782 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.782 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:07:08.783 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.522993+0000", 2024-09-17T07:07:08.783 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.7135999999999997e-05 2024-09-17T07:07:08.783 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.783 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.783 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:07:08.783 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.523003+0000", 2024-09-17T07:07:08.783 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0322e-05 2024-09-17T07:07:08.783 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.783 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.783 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:07:08.783 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.523008+0000", 2024-09-17T07:07:08.783 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.3539999999999999e-06 2024-09-17T07:07:08.783 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.783 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.784 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:07:08.784 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.523062+0000", 2024-09-17T07:07:08.784 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.3881999999999997e-05 2024-09-17T07:07:08.784 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.784 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.784 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:07:08.784 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.523072+0000", 2024-09-17T07:07:08.784 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0366999999999999e-05 2024-09-17T07:07:08.784 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.784 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.784 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:07:08.784 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.315912+0000", 2024-09-17T07:07:08.785 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.79283921700000004 2024-09-17T07:07:08.785 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.785 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.785 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:07:08.785 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.316014+0000", 2024-09-17T07:07:08.785 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000102768 2024-09-17T07:07:08.785 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.785 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.785 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,7", 2024-09-17T07:07:08.785 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.316139+0000", 2024-09-17T07:07:08.785 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00012484 2024-09-17T07:07:08.785 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.785 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.786 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:07:08.786 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.318832+0000", 2024-09-17T07:07:08.786 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.002693226 2024-09-17T07:07:08.786 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.786 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.786 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:07:08.786 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.318837+0000", 2024-09-17T07:07:08.786 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.9779999999999996e-06 2024-09-17T07:07:08.786 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.786 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.786 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:07:08.786 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.326962+0000", 2024-09-17T07:07:08.786 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0081247709999999994 2024-09-17T07:07:08.787 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.787 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.787 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:07:08.787 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.327013+0000", 2024-09-17T07:07:08.787 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.0389999999999997e-05 2024-09-17T07:07:08.787 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.787 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.787 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:07:08.787 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.327056+0000", 2024-09-17T07:07:08.787 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.3133000000000002e-05 2024-09-17T07:07:08.787 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:08.787 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:07:08.788 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:08.788 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.788 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.788 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:78404 7.16 7:685f7f52:::benchmark_data_smithi100_78556_object78403:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-17T07:07:08.788 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:05:23.719796+0000", 2024-09-17T07:07:08.788 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 104.96166214100001, 2024-09-17T07:07:08.788 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.60740184600000002, 2024-09-17T07:07:08.788 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:07:08.788 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:07:08.788 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:07:08.788 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:07:08.788 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:07:08.788 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 78404 2024-09-17T07:07:08.789 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.789 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:07:08.789 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.789 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:07:08.789 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.719796+0000", 2024-09-17T07:07:08.789 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:08.789 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.789 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.789 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:07:08.789 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.719796+0000", 2024-09-17T07:07:08.789 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:08.789 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.789 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.790 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:07:08.790 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.719789+0000", 2024-09-17T07:07:08.790 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999933 2024-09-17T07:07:08.790 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.790 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.790 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:07:08.790 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.719844+0000", 2024-09-17T07:07:08.790 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.4614999999999998e-05 2024-09-17T07:07:08.790 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.790 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.790 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:07:08.790 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.719863+0000", 2024-09-17T07:07:08.790 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.9357000000000001e-05 2024-09-17T07:07:08.791 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.791 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.791 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:07:08.791 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.719874+0000", 2024-09-17T07:07:08.791 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0343000000000001e-05 2024-09-17T07:07:08.791 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.791 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.791 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:07:08.791 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.719939+0000", 2024-09-17T07:07:08.791 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.5346999999999996e-05 2024-09-17T07:07:08.791 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.791 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.792 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:07:08.792 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.719950+0000", 2024-09-17T07:07:08.792 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.046e-05 2024-09-17T07:07:08.792 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.792 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.792 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:07:08.792 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.317039+0000", 2024-09-17T07:07:08.792 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.59708894300000004 2024-09-17T07:07:08.792 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.792 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.792 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:07:08.792 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.317132+0000", 2024-09-17T07:07:08.792 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.3887000000000004e-05 2024-09-17T07:07:08.793 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.793 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.793 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,7", 2024-09-17T07:07:08.793 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.317213+0000", 2024-09-17T07:07:08.793 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.064e-05 2024-09-17T07:07:08.793 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.793 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.793 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:07:08.793 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.318881+0000", 2024-09-17T07:07:08.793 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.001667485 2024-09-17T07:07:08.793 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.793 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.794 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:07:08.794 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.319693+0000", 2024-09-17T07:07:08.794 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00081289199999999995 2024-09-17T07:07:08.794 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.794 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.794 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:07:08.794 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.327113+0000", 2024-09-17T07:07:08.794 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0074199510000000002 2024-09-17T07:07:08.794 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.794 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.794 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:07:08.794 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.327161+0000", 2024-09-17T07:07:08.794 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.7834000000000003e-05 2024-09-17T07:07:08.795 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.795 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.795 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:07:08.795 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.327198+0000", 2024-09-17T07:07:08.795 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.6551000000000003e-05 2024-09-17T07:07:08.795 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:08.795 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:07:08.795 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:08.795 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.795 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.795 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:78533 7.16 7:6b92e60b:::benchmark_data_smithi100_78556_object78532:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-17T07:07:08.795 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:05:24.007650+0000", 2024-09-17T07:07:08.795 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 104.67380799599999, 2024-09-17T07:07:08.796 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.31967541199999999, 2024-09-17T07:07:08.796 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:07:08.796 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:07:08.796 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:07:08.796 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:07:08.796 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:07:08.796 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 78533 2024-09-17T07:07:08.796 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.796 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:07:08.796 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.796 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:07:08.796 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.007650+0000", 2024-09-17T07:07:08.797 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:08.797 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.797 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.797 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:07:08.797 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.007650+0000", 2024-09-17T07:07:08.797 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:08.797 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.797 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.797 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:07:08.797 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.007648+0000", 2024-09-17T07:07:08.797 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999981 2024-09-17T07:07:08.797 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.797 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.798 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:07:08.798 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.007688+0000", 2024-09-17T07:07:08.798 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.9419000000000003e-05 2024-09-17T07:07:08.798 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.798 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.798 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:07:08.798 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.007706+0000", 2024-09-17T07:07:08.798 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.8457e-05 2024-09-17T07:07:08.798 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.798 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.798 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:07:08.798 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.007715+0000", 2024-09-17T07:07:08.798 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.3149999999999992e-06 2024-09-17T07:07:08.799 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.799 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.799 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:07:08.799 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.007765+0000", 2024-09-17T07:07:08.799 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.0175e-05 2024-09-17T07:07:08.799 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.799 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.799 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:07:08.799 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.007775+0000", 2024-09-17T07:07:08.799 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.8579999999999995e-06 2024-09-17T07:07:08.799 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.799 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.799 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:07:08.800 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.318007+0000", 2024-09-17T07:07:08.800 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.31023252600000001 2024-09-17T07:07:08.800 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.800 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.800 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:07:08.800 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.318119+0000", 2024-09-17T07:07:08.800 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00011141600000000001 2024-09-17T07:07:08.800 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.800 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.800 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,7", 2024-09-17T07:07:08.800 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.318255+0000", 2024-09-17T07:07:08.800 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000135988 2024-09-17T07:07:08.801 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.801 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.801 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:07:08.801 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.320427+0000", 2024-09-17T07:07:08.801 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0021722289999999999 2024-09-17T07:07:08.801 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.801 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.801 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:07:08.801 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.320431+0000", 2024-09-17T07:07:08.801 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.6580000000000001e-06 2024-09-17T07:07:08.801 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.801 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.801 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:07:08.802 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.327248+0000", 2024-09-17T07:07:08.802 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0068164740000000003 2024-09-17T07:07:08.802 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.802 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.802 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:07:08.802 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.327292+0000", 2024-09-17T07:07:08.802 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.4320999999999998e-05 2024-09-17T07:07:08.802 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.802 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.802 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:07:08.802 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.327326+0000", 2024-09-17T07:07:08.802 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.3294999999999999e-05 2024-09-17T07:07:08.802 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:08.803 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:07:08.803 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:08.803 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.803 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.803 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:78548 7.16 7:68e5675b:::benchmark_data_smithi100_78556_object78547:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-17T07:07:08.803 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:05:24.042142+0000", 2024-09-17T07:07:08.803 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 104.639316045, 2024-09-17T07:07:08.803 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.29248944399999999, 2024-09-17T07:07:08.803 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:07:08.803 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:07:08.803 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:07:08.803 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:07:08.804 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:07:08.804 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 78548 2024-09-17T07:07:08.804 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.804 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:07:08.804 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.804 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:07:08.804 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.042142+0000", 2024-09-17T07:07:08.804 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:08.804 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.804 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.804 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:07:08.804 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.042142+0000", 2024-09-17T07:07:08.804 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:08.805 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.805 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.805 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:07:08.805 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.042140+0000", 2024-09-17T07:07:08.805 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999981 2024-09-17T07:07:08.805 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.805 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.805 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:07:08.805 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.042193+0000", 2024-09-17T07:07:08.805 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.2577999999999997e-05 2024-09-17T07:07:08.805 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.805 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.806 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:07:08.806 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.042205+0000", 2024-09-17T07:07:08.806 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.1741e-05 2024-09-17T07:07:08.806 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.806 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.806 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:07:08.806 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.042213+0000", 2024-09-17T07:07:08.806 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.4880000000000007e-06 2024-09-17T07:07:08.806 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.806 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.806 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:07:08.806 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.042258+0000", 2024-09-17T07:07:08.806 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.5006999999999998e-05 2024-09-17T07:07:08.807 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.807 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.807 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:07:08.807 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.042266+0000", 2024-09-17T07:07:08.807 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.9869999999999999e-06 2024-09-17T07:07:08.807 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.807 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.807 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:07:08.807 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.318911+0000", 2024-09-17T07:07:08.807 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.27664440499999998 2024-09-17T07:07:08.807 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.807 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.807 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:07:08.808 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.318990+0000", 2024-09-17T07:07:08.808 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.9735999999999997e-05 2024-09-17T07:07:08.808 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.808 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.808 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,7", 2024-09-17T07:07:08.808 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.319107+0000", 2024-09-17T07:07:08.808 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00011623199999999999 2024-09-17T07:07:08.808 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.808 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.808 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:07:08.808 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.320450+0000", 2024-09-17T07:07:08.808 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.001343892 2024-09-17T07:07:08.809 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.809 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.809 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:07:08.809 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.320555+0000", 2024-09-17T07:07:08.809 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000104683 2024-09-17T07:07:08.809 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.809 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.809 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:07:08.809 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.334541+0000", 2024-09-17T07:07:08.809 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.013985854000000001 2024-09-17T07:07:08.809 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.809 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.809 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:07:08.810 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.334594+0000", 2024-09-17T07:07:08.810 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.3112999999999998e-05 2024-09-17T07:07:08.810 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.810 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.810 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:07:08.810 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.334631+0000", 2024-09-17T07:07:08.810 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.7410999999999998e-05 2024-09-17T07:07:08.810 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:08.810 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:07:08.810 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:08.810 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.810 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.810 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:78550 7.16 7:6b06646f:::benchmark_data_smithi100_78556_object78549:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-17T07:07:08.811 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:05:24.044109+0000", 2024-09-17T07:07:08.811 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 104.63734894, 2024-09-17T07:07:08.811 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.29064050899999999, 2024-09-17T07:07:08.811 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:07:08.811 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:07:08.811 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:07:08.811 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:07:08.811 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:07:08.811 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 78550 2024-09-17T07:07:08.811 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.811 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:07:08.811 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.811 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:07:08.812 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.044109+0000", 2024-09-17T07:07:08.812 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:08.812 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.812 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.812 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:07:08.812 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.044109+0000", 2024-09-17T07:07:08.812 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:08.812 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.812 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.812 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:07:08.812 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.044108+0000", 2024-09-17T07:07:08.813 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.999999 2024-09-17T07:07:08.813 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.813 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.813 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:07:08.813 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.044164+0000", 2024-09-17T07:07:08.813 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.5798999999999999e-05 2024-09-17T07:07:08.813 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.813 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.813 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:07:08.813 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.044174+0000", 2024-09-17T07:07:08.813 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.5869999999999999e-06 2024-09-17T07:07:08.813 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.814 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.814 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:07:08.814 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.044180+0000", 2024-09-17T07:07:08.814 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.7590000000000003e-06 2024-09-17T07:07:08.814 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.814 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.814 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:07:08.814 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.044222+0000", 2024-09-17T07:07:08.814 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.2264000000000001e-05 2024-09-17T07:07:08.814 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.814 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.814 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:07:08.814 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.044229+0000", 2024-09-17T07:07:08.815 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.2640000000000002e-06 2024-09-17T07:07:08.815 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.815 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.815 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:07:08.815 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.319717+0000", 2024-09-17T07:07:08.815 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.27548832200000001 2024-09-17T07:07:08.815 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.815 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.815 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:07:08.815 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.319816+0000", 2024-09-17T07:07:08.815 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.8980000000000004e-05 2024-09-17T07:07:08.815 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.815 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.816 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,7", 2024-09-17T07:07:08.816 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.319921+0000", 2024-09-17T07:07:08.816 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00010434599999999999 2024-09-17T07:07:08.816 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.816 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.816 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:07:08.816 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.320684+0000", 2024-09-17T07:07:08.816 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00076286400000000003 2024-09-17T07:07:08.816 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.816 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.816 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:07:08.816 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.320747+0000", 2024-09-17T07:07:08.817 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.3571999999999994e-05 2024-09-17T07:07:08.817 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.817 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.817 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:07:08.817 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.334673+0000", 2024-09-17T07:07:08.817 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.013925803 2024-09-17T07:07:08.817 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.817 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.817 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:07:08.817 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.334712+0000", 2024-09-17T07:07:08.817 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.8816000000000002e-05 2024-09-17T07:07:08.817 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.818 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.818 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:07:08.818 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.334750+0000", 2024-09-17T07:07:08.818 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.7857e-05 2024-09-17T07:07:08.818 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:08.818 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:07:08.818 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:08.818 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.818 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.818 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:86278 7.16 7:6882a8ee:::benchmark_data_smithi100_78556_object86277:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-17T07:07:08.818 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:05:33.468911+0000", 2024-09-17T07:07:08.818 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 95.212547345000004, 2024-09-17T07:07:08.818 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.17154867400000001, 2024-09-17T07:07:08.819 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:07:08.819 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:07:08.819 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:07:08.819 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:07:08.819 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:07:08.819 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 86278 2024-09-17T07:07:08.819 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.819 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:07:08.819 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.819 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:07:08.819 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:33.468911+0000", 2024-09-17T07:07:08.819 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:08.819 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.820 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.820 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:07:08.820 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:33.468911+0000", 2024-09-17T07:07:08.820 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:08.820 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.820 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.820 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:07:08.820 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:33.468910+0000", 2024-09-17T07:07:08.820 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:07:08.820 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.820 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.820 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:07:08.820 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:33.468966+0000", 2024-09-17T07:07:08.821 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.6433e-05 2024-09-17T07:07:08.821 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.821 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.821 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:07:08.821 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:33.468979+0000", 2024-09-17T07:07:08.821 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.2751999999999999e-05 2024-09-17T07:07:08.821 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.821 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.821 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:07:08.821 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:33.468984+0000", 2024-09-17T07:07:08.821 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.4469999999999997e-06 2024-09-17T07:07:08.821 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.821 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.822 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:07:08.822 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:33.469026+0000", 2024-09-17T07:07:08.822 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.2111999999999999e-05 2024-09-17T07:07:08.824 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.824 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.824 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:07:08.824 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:33.469120+0000", 2024-09-17T07:07:08.824 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.3695000000000003e-05 2024-09-17T07:07:08.825 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.825 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.825 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,7", 2024-09-17T07:07:08.825 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:33.469212+0000", 2024-09-17T07:07:08.825 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.1741000000000001e-05 2024-09-17T07:07:08.825 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.825 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.825 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:07:08.825 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:33.470760+0000", 2024-09-17T07:07:08.825 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0015485500000000001 2024-09-17T07:07:08.825 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.825 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.826 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:07:08.826 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:33.471000+0000", 2024-09-17T07:07:08.826 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000239695 2024-09-17T07:07:08.826 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.826 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.826 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:07:08.826 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:33.640371+0000", 2024-09-17T07:07:08.826 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.16937102900000001 2024-09-17T07:07:08.826 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.826 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.826 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:07:08.826 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:33.640420+0000", 2024-09-17T07:07:08.826 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.9483e-05 2024-09-17T07:07:08.827 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.827 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.827 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:07:08.827 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:33.640459+0000", 2024-09-17T07:07:08.827 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.8952000000000002e-05 2024-09-17T07:07:08.827 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:08.827 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:07:08.827 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:08.827 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.827 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.827 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:94250 7.16 7:6fffdba9:::benchmark_data_smithi100_78556_object94249:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-17T07:07:08.827 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:05:42.678740+0000", 2024-09-17T07:07:08.828 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 86.002718583999993, 2024-09-17T07:07:08.828 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.17737941099999999, 2024-09-17T07:07:08.828 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:07:08.828 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:07:08.828 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:07:08.828 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:07:08.828 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:07:08.828 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 94250 2024-09-17T07:07:08.828 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.828 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:07:08.828 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.828 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:07:08.829 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:42.678740+0000", 2024-09-17T07:07:08.829 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:08.829 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.829 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.829 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:07:08.829 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:42.678740+0000", 2024-09-17T07:07:08.829 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:08.829 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.829 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.829 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:07:08.829 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:42.678738+0000", 2024-09-17T07:07:08.829 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:07:08.829 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.830 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.830 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:07:08.830 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:42.678795+0000", 2024-09-17T07:07:08.830 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.7176999999999998e-05 2024-09-17T07:07:08.830 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.830 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.830 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:07:08.830 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:42.678807+0000", 2024-09-17T07:07:08.830 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.1353e-05 2024-09-17T07:07:08.830 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.830 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.830 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:07:08.831 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:42.678813+0000", 2024-09-17T07:07:08.831 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.7669999999999998e-06 2024-09-17T07:07:08.831 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.831 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.831 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:07:08.831 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:42.678856+0000", 2024-09-17T07:07:08.831 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.2917999999999998e-05 2024-09-17T07:07:08.831 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.831 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.831 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:07:08.831 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:42.678960+0000", 2024-09-17T07:07:08.831 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000103284 2024-09-17T07:07:08.831 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.832 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.832 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,7", 2024-09-17T07:07:08.832 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:42.679039+0000", 2024-09-17T07:07:08.832 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.9661999999999997e-05 2024-09-17T07:07:08.832 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.832 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.832 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:07:08.832 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:42.680959+0000", 2024-09-17T07:07:08.832 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0019197540000000001 2024-09-17T07:07:08.832 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.832 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.832 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:07:08.833 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:42.681655+0000", 2024-09-17T07:07:08.833 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00069615900000000001 2024-09-17T07:07:08.833 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.833 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.833 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:07:08.833 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:42.856025+0000", 2024-09-17T07:07:08.833 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.174369301 2024-09-17T07:07:08.833 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.833 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.833 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:07:08.833 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:42.856077+0000", 2024-09-17T07:07:08.833 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.2482999999999999e-05 2024-09-17T07:07:08.833 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.834 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.834 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:07:08.834 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:42.856119+0000", 2024-09-17T07:07:08.834 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.1913999999999999e-05 2024-09-17T07:07:08.834 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:08.834 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:07:08.834 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:08.834 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.834 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.834 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:96508 7.1 7:8183794b:::benchmark_data_smithi100_78556_object96507:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-17T07:07:08.834 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:05:45.753141+0000", 2024-09-17T07:07:08.834 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 82.928317251999999, 2024-09-17T07:07:08.834 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.158070038, 2024-09-17T07:07:08.835 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:07:08.835 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:07:08.835 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:07:08.835 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:07:08.835 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:07:08.835 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 96508 2024-09-17T07:07:08.835 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.835 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:07:08.835 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.835 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:07:08.835 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:45.753141+0000", 2024-09-17T07:07:08.835 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:08.835 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.836 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.836 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:07:08.836 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:45.753141+0000", 2024-09-17T07:07:08.836 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:08.836 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.836 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.836 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:07:08.836 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:45.753139+0000", 2024-09-17T07:07:08.836 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:07:08.836 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.836 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.836 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:07:08.836 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:45.753216+0000", 2024-09-17T07:07:08.837 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.6481000000000002e-05 2024-09-17T07:07:08.837 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.837 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.837 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:07:08.837 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:45.753229+0000", 2024-09-17T07:07:08.837 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.3414999999999999e-05 2024-09-17T07:07:08.837 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.837 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.837 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:07:08.837 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:45.753238+0000", 2024-09-17T07:07:08.837 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.6430000000000001e-06 2024-09-17T07:07:08.837 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.837 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.838 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:07:08.838 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:45.753287+0000", 2024-09-17T07:07:08.838 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.8760000000000001e-05 2024-09-17T07:07:08.838 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.838 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.838 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:07:08.838 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:45.753427+0000", 2024-09-17T07:07:08.838 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000140154 2024-09-17T07:07:08.838 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.838 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.838 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 3,6", 2024-09-17T07:07:08.838 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:45.753577+0000", 2024-09-17T07:07:08.838 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00015046100000000001 2024-09-17T07:07:08.839 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.839 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.839 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:07:08.839 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:45.755571+0000", 2024-09-17T07:07:08.839 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0019936569999999998 2024-09-17T07:07:08.839 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.839 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.839 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:07:08.839 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:45.755972+0000", 2024-09-17T07:07:08.839 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000401087 2024-09-17T07:07:08.839 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.839 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.839 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:07:08.840 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:45.911149+0000", 2024-09-17T07:07:08.840 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.155176488 2024-09-17T07:07:08.840 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.840 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.840 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:07:08.840 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:45.911183+0000", 2024-09-17T07:07:08.840 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.4266e-05 2024-09-17T07:07:08.840 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:08.840 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:08.840 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:07:08.840 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:45.911211+0000", 2024-09-17T07:07:08.840 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.8048999999999999e-05 2024-09-17T07:07:08.841 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:08.841 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:07:08.841 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:08.841 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:08.841 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:07:08.841 INFO:teuthology.orchestra.run.smithi135.stdout:} 2024-09-17T07:07:08.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:08.843 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:08.843 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:07:08.944 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:07:08.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:08.977 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:08.978 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:07:09.078 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:07:09.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:09.119 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:09.120 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:07:09.221 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:07:09.259 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:09.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:09.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:07:09.344 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:09 smithi100 ceph-mon[29092]: osdmap e156: 8 total, 8 up, 8 in 2024-09-17T07:07:09.344 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:09 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1270856574' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:07:09.344 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:09 smithi100 ceph-mon[39462]: osdmap e156: 8 total, 8 up, 8 in 2024-09-17T07:07:09.345 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:09 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1270856574' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:07:09.361 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:07:09.378 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:09 smithi135 ceph-mon[33238]: osdmap e156: 8 total, 8 up, 8 in 2024-09-17T07:07:09.378 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:09 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1270856574' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:07:09.398 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:09.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:09.399 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:07:09.500 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:07:09.527 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:09.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:09.528 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:07:09.628 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:07:09.660 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:09.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:09.662 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:07:09.733 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T07:07:09.762 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:07:09.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:09.815 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:09.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:07:09.917 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:07:09.941 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:09.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:09.942 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:07:10.003 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-17T07:07:10.043 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:07:10.086 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:10.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:10.087 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:07:10.098 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:07:10.138 INFO:teuthology.orchestra.run.smithi100.stderr:reweighted osd.6 to 1 (10000) 2024-09-17T07:07:10.188 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:07:10.231 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:10.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:10.232 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:07:10.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:10 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:07:10.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:10 smithi135 ceph-mon[33238]: pgmap v212: 158 pgs: 158 active+clean; 2.0 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.51k op/s 2024-09-17T07:07:10.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:10 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:07:10.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:10 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:07:10.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:10 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2833220361' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-17T07:07:10.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:10 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-17T07:07:10.330 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:07:10.333 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:07:10.376 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:10 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:07:10.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:10 smithi100 ceph-mon[39462]: pgmap v212: 158 pgs: 158 active+clean; 2.0 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.51k op/s 2024-09-17T07:07:10.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:10 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:07:10.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:10 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:07:10.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:10 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2833220361' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-17T07:07:10.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:10 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-17T07:07:10.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:10 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:07:10.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:10 smithi100 ceph-mon[29092]: pgmap v212: 158 pgs: 158 active+clean; 2.0 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.51k op/s 2024-09-17T07:07:10.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:10 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:07:10.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:10 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:07:10.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:10 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2833220361' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-17T07:07:10.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:10 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-17T07:07:10.378 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:10.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:10.380 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:07:10.481 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:07:10.510 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:10.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:10.511 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:07:10.612 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:07:10.640 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:10.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:10.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:07:10.742 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:07:10.779 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:10.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:10.780 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:07:10.880 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:07:10.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:10.902 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:10.902 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:07:11.003 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:07:11.032 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:11.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:11.033 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:07:11.130 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd reweight 7 1 2024-09-17T07:07:11.134 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:07:11.163 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:11.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:11.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:07:11.265 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:07:11.291 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:11.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:11.293 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:07:11.369 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:11 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-17T07:07:11.369 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:11 smithi100 ceph-mon[39462]: osdmap e157: 8 total, 8 up, 8 in 2024-09-17T07:07:11.369 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:11 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:07:11.369 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:11 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:07:11.369 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:11 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-17T07:07:11.369 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:11 smithi100 ceph-mon[29092]: osdmap e157: 8 total, 8 up, 8 in 2024-09-17T07:07:11.370 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:11 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:07:11.370 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:11 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:07:11.382 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:07:11.393 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:07:11.478 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:11.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:11.480 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:07:11.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:11 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-17T07:07:11.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:11 smithi135 ceph-mon[33238]: osdmap e157: 8 total, 8 up, 8 in 2024-09-17T07:07:11.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:11 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:07:11.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:11 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:07:11.581 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:07:11.635 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:11.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:11.636 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:07:11.737 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:07:11.767 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:11.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:11.769 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:07:11.870 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:07:11.891 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:11.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:11.892 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:07:11.993 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:07:12.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:12.047 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:12.047 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:07:12.148 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:07:12.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:12.170 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:12.170 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:07:12.271 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:07:12.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:12.311 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:12.311 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:07:12.393 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:12 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:07:12.393 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:12 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:07:12.393 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:12 smithi100 ceph-mon[29092]: pgmap v214: 158 pgs: 158 active+clean; 2.0 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.08k op/s 2024-09-17T07:07:12.393 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:12 smithi100 ceph-mon[29092]: osdmap e158: 8 total, 8 up, 8 in 2024-09-17T07:07:12.393 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:12 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:07:12.393 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:12 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:07:12.393 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:12 smithi100 ceph-mon[39462]: pgmap v214: 158 pgs: 158 active+clean; 2.0 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.08k op/s 2024-09-17T07:07:12.393 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:12 smithi100 ceph-mon[39462]: osdmap e158: 8 total, 8 up, 8 in 2024-09-17T07:07:12.412 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:07:12.442 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:12.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:12.443 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:07:12.503 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:07:12.520 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:07:12.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:12 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:07:12.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:12 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:07:12.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:12 smithi135 ceph-mon[33238]: pgmap v214: 158 pgs: 158 active+clean; 2.0 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.08k op/s 2024-09-17T07:07:12.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:12 smithi135 ceph-mon[33238]: osdmap e158: 8 total, 8 up, 8 in 2024-09-17T07:07:12.544 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:07:12.564 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:12.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:12.565 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:07:12.666 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:07:12.710 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:12.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:12.711 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:07:12.812 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:07:12.836 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:12.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:12.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:07:12.938 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:07:12.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:12.973 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:12.973 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:07:13.074 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:07:13.101 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:13.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:13.102 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:07:13.203 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:07:13.244 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:13.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:13.245 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:07:13.346 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:07:13.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:13.380 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:13.380 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:07:13.481 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:07:13.512 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:13.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:13.514 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:07:13.614 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:07:13.636 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:13.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:13.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:07:13.738 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:07:13.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:13.761 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:13.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:07:13.862 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:07:13.894 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:13.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:13.895 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:07:13.996 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:07:14.026 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:14.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:14.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:07:14.129 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:07:14.167 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:14.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:14.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:07:14.173 INFO:teuthology.orchestra.run.smithi100.stderr:noscrub is unset 2024-09-17T07:07:14.269 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:07:14.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:14.310 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:14.310 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:07:14.402 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:14 smithi100 ceph-mon[29092]: pgmap v216: 158 pgs: 158 active+clean; 1.9 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.36k op/s 2024-09-17T07:07:14.402 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:14 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/835875359' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:07:14.402 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:14 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:07:14.402 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:14 smithi100 ceph-mon[39462]: pgmap v216: 158 pgs: 158 active+clean; 1.9 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.36k op/s 2024-09-17T07:07:14.402 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:14 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/835875359' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:07:14.403 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:14 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:07:14.410 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:14 smithi135 ceph-mon[33238]: pgmap v216: 158 pgs: 158 active+clean; 1.9 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.36k op/s 2024-09-17T07:07:14.410 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:14 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/835875359' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:07:14.410 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:14 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:07:14.410 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:07:14.463 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:14.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:14.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:07:14.564 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:07:14.593 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:14.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:14.594 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:07:14.695 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:07:14.724 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:14.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:14.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:07:14.826 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:07:14.863 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:14.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:14.864 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:07:14.965 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:07:14.999 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:15.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:15.000 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:07:15.101 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:07:15.135 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:15.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:15.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:07:15.237 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:07:15.277 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:15.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:15.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:07:15.340 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:15 smithi100 ceph-mon[29092]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:07:15.340 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:15 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T07:07:15.340 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:15 smithi100 ceph-mon[29092]: osdmap e159: 8 total, 8 up, 8 in 2024-09-17T07:07:15.340 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:15 smithi100 ceph-mon[29092]: 5.f scrub starts 2024-09-17T07:07:15.341 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:15 smithi100 ceph-mon[29092]: 5.f scrub ok 2024-09-17T07:07:15.341 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:15 smithi100 ceph-mon[39462]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:07:15.341 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:15 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T07:07:15.341 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:15 smithi100 ceph-mon[39462]: osdmap e159: 8 total, 8 up, 8 in 2024-09-17T07:07:15.341 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:15 smithi100 ceph-mon[39462]: 5.f scrub starts 2024-09-17T07:07:15.341 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:15 smithi100 ceph-mon[39462]: 5.f scrub ok 2024-09-17T07:07:15.380 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:07:15.412 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:15.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:15.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:07:15.434 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:15 smithi135 ceph-mon[33238]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:07:15.434 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:15 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T07:07:15.434 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:15 smithi135 ceph-mon[33238]: osdmap e159: 8 total, 8 up, 8 in 2024-09-17T07:07:15.434 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:15 smithi135 ceph-mon[33238]: 5.f scrub starts 2024-09-17T07:07:15.434 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:15 smithi135 ceph-mon[33238]: 5.f scrub ok 2024-09-17T07:07:15.514 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:07:15.551 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:15.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:15.552 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:07:15.653 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:07:15.692 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:15.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:15.693 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:07:15.794 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:07:15.835 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:15.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:15.836 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:07:15.862 INFO:teuthology.orchestra.run.smithi135.stdout:{ 2024-09-17T07:07:15.862 INFO:teuthology.orchestra.run.smithi135.stdout: "ops": [ 2024-09-17T07:07:15.862 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:15.862 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:196802 7.0 7:05bd896b:::benchmark_data_smithi100_78556_object85999:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e159)", 2024-09-17T07:07:15.862 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:07:15.851134+0000", 2024-09-17T07:07:15.862 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 0.0095141359999999994, 2024-09-17T07:07:15.862 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0095564110000000008, 2024-09-17T07:07:15.862 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:07:15.862 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "waiting for sub ops", 2024-09-17T07:07:15.863 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:07:15.863 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:07:15.863 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:07:15.863 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 196802 2024-09-17T07:07:15.863 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:15.863 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:07:15.863 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:15.863 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:07:15.863 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:07:15.851134+0000", 2024-09-17T07:07:15.863 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:15.863 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:15.863 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:15.864 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:07:15.864 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:07:15.851134+0000", 2024-09-17T07:07:15.864 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:15.864 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:15.864 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:15.864 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:07:15.864 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:07:15.851132+0000", 2024-09-17T07:07:15.864 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999981 2024-09-17T07:07:15.864 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:15.864 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:15.864 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:07:15.864 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:07:15.851138+0000", 2024-09-17T07:07:15.865 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.9000000000000003e-06 2024-09-17T07:07:15.865 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:15.865 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:15.865 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:07:15.865 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:07:15.851150+0000", 2024-09-17T07:07:15.865 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.2092999999999999e-05 2024-09-17T07:07:15.865 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:15.865 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:15.865 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:07:15.865 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:07:15.851159+0000", 2024-09-17T07:07:15.865 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.9050000000000003e-06 2024-09-17T07:07:15.865 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:15.866 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:15.866 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:07:15.866 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:07:15.851205+0000", 2024-09-17T07:07:15.866 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.5821000000000001e-05 2024-09-17T07:07:15.866 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:15.866 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:15.866 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:07:15.866 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:07:15.851362+0000", 2024-09-17T07:07:15.866 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000157646 2024-09-17T07:07:15.866 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:15.867 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:15.867 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 1,3", 2024-09-17T07:07:15.867 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:07:15.851794+0000", 2024-09-17T07:07:15.867 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00043141099999999998 2024-09-17T07:07:15.867 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:15.867 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:15.867 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:07:15.867 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:07:15.854470+0000", 2024-09-17T07:07:15.867 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0026765349999999999 2024-09-17T07:07:15.867 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:15.867 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:07:15.867 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:15.868 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:15.868 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:15.868 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_repop(client.14805.0:196809 7.a e159/99 7:58b36e4a:::benchmark_data_smithi100_78556_object86006:head v 159'12172, mlcod=159'12169)", 2024-09-17T07:07:15.868 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:07:15.857042+0000", 2024-09-17T07:07:15.868 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 0.003605203, 2024-09-17T07:07:15.868 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0037308810000000001, 2024-09-17T07:07:15.868 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:07:15.868 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "started", 2024-09-17T07:07:15.868 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:07:15.868 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:15.868 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:07:15.868 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:07:15.857042+0000", 2024-09-17T07:07:15.869 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:15.869 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:15.869 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:15.869 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:07:15.869 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:07:15.857042+0000", 2024-09-17T07:07:15.869 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:15.869 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:15.869 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:15.869 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:07:15.869 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:07:15.857042+0000", 2024-09-17T07:07:15.869 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.999999 2024-09-17T07:07:15.869 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:15.869 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:15.870 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:07:15.870 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:07:15.857048+0000", 2024-09-17T07:07:15.870 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.8170000000000001e-06 2024-09-17T07:07:15.870 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:15.870 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:15.870 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:07:15.870 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:07:15.857063+0000", 2024-09-17T07:07:15.870 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.4158000000000001e-05 2024-09-17T07:07:15.870 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:15.870 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:15.870 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:07:15.870 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:07:15.857070+0000", 2024-09-17T07:07:15.870 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.7170000000000008e-06 2024-09-17T07:07:15.871 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:15.871 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:15.871 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:07:15.871 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:07:15.857112+0000", 2024-09-17T07:07:15.871 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.1588000000000002e-05 2024-09-17T07:07:15.871 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:15.871 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:15.871 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:07:15.871 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:07:15.857131+0000", 2024-09-17T07:07:15.871 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.9514999999999999e-05 2024-09-17T07:07:15.871 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:15.871 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:07:15.872 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:15.872 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:15.872 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:15.872 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_repop(client.14805.0:196810 7.b e159/99 7:de26ca52:::benchmark_data_smithi100_78556_object86007:head v 159'12388, mlcod=159'12385)", 2024-09-17T07:07:15.872 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:07:15.858385+0000", 2024-09-17T07:07:15.872 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 0.0022631130000000002, 2024-09-17T07:07:15.872 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0024430369999999999, 2024-09-17T07:07:15.872 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:07:15.872 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "started", 2024-09-17T07:07:15.872 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:07:15.872 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:15.872 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:07:15.872 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:07:15.858385+0000", 2024-09-17T07:07:15.873 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:15.873 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:15.873 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:15.873 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:07:15.873 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:07:15.858385+0000", 2024-09-17T07:07:15.873 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:15.873 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:15.873 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:15.873 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:07:15.873 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:07:15.858384+0000", 2024-09-17T07:07:15.873 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.999999 2024-09-17T07:07:15.873 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:15.873 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:15.874 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:07:15.874 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:07:15.858390+0000", 2024-09-17T07:07:15.874 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.2149999999999998e-06 2024-09-17T07:07:15.874 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:15.874 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:15.874 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:07:15.874 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:07:15.858407+0000", 2024-09-17T07:07:15.874 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.6702999999999999e-05 2024-09-17T07:07:15.874 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:15.874 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:15.874 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:07:15.874 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:07:15.858414+0000", 2024-09-17T07:07:15.874 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.9569999999999996e-06 2024-09-17T07:07:15.875 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:15.875 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:15.875 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:07:15.875 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:07:15.858467+0000", 2024-09-17T07:07:15.875 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.3797e-05 2024-09-17T07:07:15.875 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:15.875 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:15.875 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:07:15.875 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:07:15.858486+0000", 2024-09-17T07:07:15.875 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.8762000000000001e-05 2024-09-17T07:07:15.875 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:15.875 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:07:15.875 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:15.875 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:15.876 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:15.876 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_repop(client.14805.0:196816 7.a e159/99 7:550f3445:::benchmark_data_smithi100_78556_object86013:head v 159'12173, mlcod=159'12169)", 2024-09-17T07:07:15.876 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:07:15.859445+0000", 2024-09-17T07:07:15.876 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 0.001202219, 2024-09-17T07:07:15.876 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0014357420000000001, 2024-09-17T07:07:15.876 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:07:15.876 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "started", 2024-09-17T07:07:15.876 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:07:15.876 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:15.876 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:07:15.876 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:07:15.859445+0000", 2024-09-17T07:07:15.876 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:15.877 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:15.877 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:15.877 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:07:15.877 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:07:15.859445+0000", 2024-09-17T07:07:15.877 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:15.877 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:15.877 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:15.877 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:07:15.877 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:07:15.859444+0000", 2024-09-17T07:07:15.877 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.999999 2024-09-17T07:07:15.877 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:15.877 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:15.877 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:07:15.878 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:07:15.859454+0000", 2024-09-17T07:07:15.878 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.8980000000000008e-06 2024-09-17T07:07:15.878 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:15.878 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:15.878 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:07:15.878 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:07:15.859466+0000", 2024-09-17T07:07:15.878 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.1745e-05 2024-09-17T07:07:15.878 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:15.878 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:15.878 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:07:15.878 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:07:15.859472+0000", 2024-09-17T07:07:15.878 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.3210000000000001e-06 2024-09-17T07:07:15.878 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:15.879 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:15.879 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:07:15.879 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:07:15.859518+0000", 2024-09-17T07:07:15.879 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.6097999999999998e-05 2024-09-17T07:07:15.879 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:15.879 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:15.879 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:07:15.879 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:07:15.859536+0000", 2024-09-17T07:07:15.879 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.7288000000000001e-05 2024-09-17T07:07:15.879 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:15.879 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:07:15.879 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:15.879 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:15.880 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:15.880 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_repop(client.14805.0:196815 7.b e159/99)", 2024-09-17T07:07:15.880 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:07:15.860671+0000", 2024-09-17T07:07:15.880 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 4294967295.9999766, 2024-09-17T07:07:15.880 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00026275400000000003, 2024-09-17T07:07:15.880 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:07:15.880 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "queued for pg", 2024-09-17T07:07:15.880 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:07:15.880 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:15.880 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:07:15.880 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:07:15.860671+0000", 2024-09-17T07:07:15.880 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:15.881 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:15.881 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:15.881 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:07:15.881 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:07:15.860671+0000", 2024-09-17T07:07:15.881 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:15.881 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:15.881 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:15.881 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:07:15.881 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:07:15.860670+0000", 2024-09-17T07:07:15.881 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:07:15.881 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:15.881 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:15.881 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:07:15.882 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:07:15.860677+0000", 2024-09-17T07:07:15.882 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.3930000000000002e-06 2024-09-17T07:07:15.882 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:15.882 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:15.882 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:07:15.882 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:07:15.860690+0000", 2024-09-17T07:07:15.882 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.2958e-05 2024-09-17T07:07:15.882 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:15.882 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:15.882 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:07:15.882 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:07:15.860696+0000", 2024-09-17T07:07:15.882 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.3640000000000004e-06 2024-09-17T07:07:15.883 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:15.883 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:07:15.883 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:15.883 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:15.883 INFO:teuthology.orchestra.run.smithi135.stdout: ], 2024-09-17T07:07:15.883 INFO:teuthology.orchestra.run.smithi135.stdout: "num_ops": 5 2024-09-17T07:07:15.883 INFO:teuthology.orchestra.run.smithi135.stdout:} 2024-09-17T07:07:15.937 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:07:15.973 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:15.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:15.974 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:07:16.075 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:07:16.103 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:16.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:16.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:07:16.126 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:07:15 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:07:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:07:16.205 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:07:16.237 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:16.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:16.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:07:16.339 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:07:16.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:16.373 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:16.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:07:16.449 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:16 smithi135 ceph-mon[33238]: pgmap v218: 158 pgs: 158 active+clean; 1.7 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 933 op/s 2024-09-17T07:07:16.449 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:16 smithi135 ceph-mon[33238]: 2.1e scrub starts 2024-09-17T07:07:16.449 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:16 smithi135 ceph-mon[33238]: 2.1e scrub ok 2024-09-17T07:07:16.474 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:07:16.511 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:16.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:16.512 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:07:16.537 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:16 smithi100 ceph-mon[29092]: pgmap v218: 158 pgs: 158 active+clean; 1.7 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 933 op/s 2024-09-17T07:07:16.537 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:16 smithi100 ceph-mon[29092]: 2.1e scrub starts 2024-09-17T07:07:16.537 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:16 smithi100 ceph-mon[29092]: 2.1e scrub ok 2024-09-17T07:07:16.537 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:16 smithi100 ceph-mon[39462]: pgmap v218: 158 pgs: 158 active+clean; 1.7 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 933 op/s 2024-09-17T07:07:16.537 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:16 smithi100 ceph-mon[39462]: 2.1e scrub starts 2024-09-17T07:07:16.537 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:16 smithi100 ceph-mon[39462]: 2.1e scrub ok 2024-09-17T07:07:16.613 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:07:16.641 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:16.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:16.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:07:16.743 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:07:16.772 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:16.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:16.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:07:16.874 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:07:16.902 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:16.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:16.903 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:07:17.004 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:07:17.034 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:17.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:17.035 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:07:17.039 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-17T07:07:17.135 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:07:17.182 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:17.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:17.184 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:07:17.184 INFO:teuthology.orchestra.run.smithi100.stderr:reweighted osd.7 to 1 (10000) 2024-09-17T07:07:17.267 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-17T07:07:17.284 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:07:17.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:17.329 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:17.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:07:17.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:17 smithi100 ceph-mon[29092]: 4.1b scrub starts 2024-09-17T07:07:17.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:17 smithi100 ceph-mon[29092]: 4.1b scrub ok 2024-09-17T07:07:17.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:17 smithi100 ceph-mon[29092]: 7.4 scrub starts 2024-09-17T07:07:17.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:17 smithi100 ceph-mon[29092]: 7.4 scrub ok 2024-09-17T07:07:17.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:17 smithi100 ceph-mon[29092]: 6.f scrub starts 2024-09-17T07:07:17.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:17 smithi100 ceph-mon[29092]: 6.f scrub ok 2024-09-17T07:07:17.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:17 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1742294913' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-17T07:07:17.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:17 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-17T07:07:17.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:17 smithi100 ceph-mon[39462]: 4.1b scrub starts 2024-09-17T07:07:17.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:17 smithi100 ceph-mon[39462]: 4.1b scrub ok 2024-09-17T07:07:17.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:17 smithi100 ceph-mon[39462]: 7.4 scrub starts 2024-09-17T07:07:17.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:17 smithi100 ceph-mon[39462]: 7.4 scrub ok 2024-09-17T07:07:17.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:17 smithi100 ceph-mon[39462]: 6.f scrub starts 2024-09-17T07:07:17.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:17 smithi100 ceph-mon[39462]: 6.f scrub ok 2024-09-17T07:07:17.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:17 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1742294913' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-17T07:07:17.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:17 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-17T07:07:17.430 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:07:17.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:17.477 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:17.477 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:07:17.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:17 smithi135 ceph-mon[33238]: 4.1b scrub starts 2024-09-17T07:07:17.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:17 smithi135 ceph-mon[33238]: 4.1b scrub ok 2024-09-17T07:07:17.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:17 smithi135 ceph-mon[33238]: 7.4 scrub starts 2024-09-17T07:07:17.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:17 smithi135 ceph-mon[33238]: 7.4 scrub ok 2024-09-17T07:07:17.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:17 smithi135 ceph-mon[33238]: 6.f scrub starts 2024-09-17T07:07:17.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:17 smithi135 ceph-mon[33238]: 6.f scrub ok 2024-09-17T07:07:17.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:17 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1742294913' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-17T07:07:17.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:17 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-17T07:07:17.578 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:07:17.581 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:07:17.621 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:17.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:17.622 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:07:17.723 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:07:17.753 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:17.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:17.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:07:17.855 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:07:17.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:17.890 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:17.890 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:07:17.991 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:07:18.028 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:18.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:18.029 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:07:18.093 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:07:18.094 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:07:18.129 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1: {} 2024-09-17T07:07:18.130 INFO:teuthology.orchestra.run.smithi100.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:07:18.130 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:07:18.157 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:18.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:18.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:07:18.159 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:07:18.160 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:07:18.195 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:07:18.196 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:07:18.233 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:07:18.233 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:07:18.260 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:07:18.268 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:07:18.269 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:07:18.284 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:18 smithi135 ceph-mon[33238]: pgmap v219: 158 pgs: 158 active+clean; 1.7 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 734 B/s rd, 0 B/s wr, 803 op/s 2024-09-17T07:07:18.284 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:18 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-17T07:07:18.284 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:18 smithi135 ceph-mon[33238]: osdmap e160: 8 total, 8 up, 8 in 2024-09-17T07:07:18.284 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:18 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:07:18.284 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:18 smithi135 ceph-mon[33238]: 5.13 scrub starts 2024-09-17T07:07:18.284 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:18 smithi135 ceph-mon[33238]: 5.13 scrub ok 2024-09-17T07:07:18.284 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:18 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2976252093' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:07:18.296 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:18.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:18.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:07:18.298 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:07:18.299 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:07:18.331 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:07:18.331 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:07:18.398 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:07:18.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:18.437 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:18.437 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:07:18.508 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:18 smithi100 ceph-mon[29092]: pgmap v219: 158 pgs: 158 active+clean; 1.7 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 734 B/s rd, 0 B/s wr, 803 op/s 2024-09-17T07:07:18.508 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:18 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-17T07:07:18.508 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:18 smithi100 ceph-mon[29092]: osdmap e160: 8 total, 8 up, 8 in 2024-09-17T07:07:18.508 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:18 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:07:18.509 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:18 smithi100 ceph-mon[29092]: 5.13 scrub starts 2024-09-17T07:07:18.509 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:18 smithi100 ceph-mon[29092]: 5.13 scrub ok 2024-09-17T07:07:18.509 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:18 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2976252093' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:07:18.509 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:18 smithi100 ceph-mon[39462]: pgmap v219: 158 pgs: 158 active+clean; 1.7 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 734 B/s rd, 0 B/s wr, 803 op/s 2024-09-17T07:07:18.509 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:18 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-17T07:07:18.509 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:18 smithi100 ceph-mon[39462]: osdmap e160: 8 total, 8 up, 8 in 2024-09-17T07:07:18.509 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:18 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:07:18.509 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:18 smithi100 ceph-mon[39462]: 5.13 scrub starts 2024-09-17T07:07:18.510 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:18 smithi100 ceph-mon[39462]: 5.13 scrub ok 2024-09-17T07:07:18.510 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:18 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2976252093' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:07:18.538 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:07:18.543 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:07:18 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:07:18.546 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-17T07:07:18.546 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T07:07:18.568 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:18.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:18.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:07:18.670 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:07:18.698 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:18.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:18.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:07:18.742 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:07:18.800 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:07:18.832 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:18.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:18.833 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:07:18.934 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:07:18.972 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:18.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:18.974 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:07:19.075 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:07:19.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:19.115 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:19.115 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:07:19.216 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:07:19.253 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:19.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:19.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:07:19.343 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:19 smithi135 ceph-mon[33238]: 4.3 scrub starts 2024-09-17T07:07:19.343 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:19 smithi135 ceph-mon[33238]: 4.3 scrub ok 2024-09-17T07:07:19.343 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:19 smithi135 ceph-mon[33238]: 6.15 scrub starts 2024-09-17T07:07:19.343 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:19 smithi135 ceph-mon[33238]: 6.15 scrub ok 2024-09-17T07:07:19.344 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:19 smithi135 ceph-mon[33238]: 2.9 scrub starts 2024-09-17T07:07:19.344 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:19 smithi135 ceph-mon[33238]: 2.9 scrub ok 2024-09-17T07:07:19.355 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:07:19.392 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:19.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:19.394 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:07:19.449 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:19 smithi100 ceph-mon[29092]: 4.3 scrub starts 2024-09-17T07:07:19.449 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:19 smithi100 ceph-mon[29092]: 4.3 scrub ok 2024-09-17T07:07:19.449 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:19 smithi100 ceph-mon[29092]: 6.15 scrub starts 2024-09-17T07:07:19.450 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:19 smithi100 ceph-mon[29092]: 6.15 scrub ok 2024-09-17T07:07:19.450 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:19 smithi100 ceph-mon[29092]: 2.9 scrub starts 2024-09-17T07:07:19.450 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:19 smithi100 ceph-mon[29092]: 2.9 scrub ok 2024-09-17T07:07:19.450 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:19 smithi100 ceph-mon[39462]: 4.3 scrub starts 2024-09-17T07:07:19.450 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:19 smithi100 ceph-mon[39462]: 4.3 scrub ok 2024-09-17T07:07:19.450 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:19 smithi100 ceph-mon[39462]: 6.15 scrub starts 2024-09-17T07:07:19.450 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:19 smithi100 ceph-mon[39462]: 6.15 scrub ok 2024-09-17T07:07:19.450 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:19 smithi100 ceph-mon[39462]: 2.9 scrub starts 2024-09-17T07:07:19.450 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:19 smithi100 ceph-mon[39462]: 2.9 scrub ok 2024-09-17T07:07:19.495 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:07:19.513 INFO:teuthology.orchestra.run.smithi135.stdout:ERROR: (22) Invalid argument 2024-09-17T07:07:19.513 INFO:teuthology.orchestra.run.smithi135.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:07:19.538 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:19.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:19.539 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:07:19.640 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:07:19.675 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:19.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:19.676 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:07:19.777 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:07:19.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:19.817 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:19.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:07:19.918 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:07:19.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:19.950 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:19.950 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:07:20.051 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:07:20.078 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:20.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:20.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:07:20.180 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:07:20.224 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:20.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:20.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:07:20.238 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-17T07:07:20.325 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:07:20.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:20 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:07:20.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:20 smithi100 ceph-mon[29092]: 4.a scrub starts 2024-09-17T07:07:20.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:20 smithi100 ceph-mon[29092]: 4.a scrub ok 2024-09-17T07:07:20.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:20 smithi100 ceph-mon[29092]: pgmap v221: 158 pgs: 158 active+clean; 1.3 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 695 op/s 2024-09-17T07:07:20.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:20 smithi100 ceph-mon[29092]: 6.a scrub starts 2024-09-17T07:07:20.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:20 smithi100 ceph-mon[29092]: 6.a scrub ok 2024-09-17T07:07:20.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:20 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:07:20.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:20 smithi100 ceph-mon[39462]: 4.a scrub starts 2024-09-17T07:07:20.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:20 smithi100 ceph-mon[39462]: 4.a scrub ok 2024-09-17T07:07:20.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:20 smithi100 ceph-mon[39462]: pgmap v221: 158 pgs: 158 active+clean; 1.3 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 695 op/s 2024-09-17T07:07:20.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:20 smithi100 ceph-mon[39462]: 6.a scrub starts 2024-09-17T07:07:20.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:20 smithi100 ceph-mon[39462]: 6.a scrub ok 2024-09-17T07:07:20.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:20.380 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:20.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:07:20.402 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:07:20.480 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:07:20.518 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:20.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:20.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:07:20.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:20 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:07:20.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:20 smithi135 ceph-mon[33238]: 4.a scrub starts 2024-09-17T07:07:20.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:20 smithi135 ceph-mon[33238]: 4.a scrub ok 2024-09-17T07:07:20.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:20 smithi135 ceph-mon[33238]: pgmap v221: 158 pgs: 158 active+clean; 1.3 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 695 op/s 2024-09-17T07:07:20.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:20 smithi135 ceph-mon[33238]: 6.a scrub starts 2024-09-17T07:07:20.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:20 smithi135 ceph-mon[33238]: 6.a scrub ok 2024-09-17T07:07:20.557 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:07:20.620 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:07:20.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:20.649 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:20.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:07:20.750 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:07:20.790 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:20.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:20.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:07:20.892 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:07:20.921 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:20.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:20.923 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:07:21.023 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:07:21.070 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:21.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:21.071 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:07:21.172 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:07:21.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:21.203 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:21.203 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:07:21.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:21 smithi135 ceph-mon[33238]: 5.1f scrub starts 2024-09-17T07:07:21.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:21 smithi135 ceph-mon[33238]: 5.1f scrub ok 2024-09-17T07:07:21.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:21 smithi135 ceph-mon[33238]: 5.a scrub starts 2024-09-17T07:07:21.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:21 smithi135 ceph-mon[33238]: 5.a scrub ok 2024-09-17T07:07:21.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:21 smithi135 ceph-mon[33238]: 6.9 scrub starts 2024-09-17T07:07:21.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:21 smithi135 ceph-mon[33238]: 6.9 scrub ok 2024-09-17T07:07:21.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:21 smithi135 ceph-mon[33238]: 2.f scrub starts 2024-09-17T07:07:21.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:21 smithi135 ceph-mon[33238]: 2.f scrub ok 2024-09-17T07:07:21.304 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:07:21.340 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:21.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:21.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:07:21.394 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:21 smithi100 ceph-mon[29092]: 5.1f scrub starts 2024-09-17T07:07:21.394 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:21 smithi100 ceph-mon[29092]: 5.1f scrub ok 2024-09-17T07:07:21.394 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:21 smithi100 ceph-mon[29092]: 5.a scrub starts 2024-09-17T07:07:21.394 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:21 smithi100 ceph-mon[29092]: 5.a scrub ok 2024-09-17T07:07:21.395 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:21 smithi100 ceph-mon[29092]: 6.9 scrub starts 2024-09-17T07:07:21.395 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:21 smithi100 ceph-mon[29092]: 6.9 scrub ok 2024-09-17T07:07:21.395 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:21 smithi100 ceph-mon[29092]: 2.f scrub starts 2024-09-17T07:07:21.395 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:21 smithi100 ceph-mon[29092]: 2.f scrub ok 2024-09-17T07:07:21.395 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:21 smithi100 ceph-mon[39462]: 5.1f scrub starts 2024-09-17T07:07:21.395 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:21 smithi100 ceph-mon[39462]: 5.1f scrub ok 2024-09-17T07:07:21.395 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:21 smithi100 ceph-mon[39462]: 5.a scrub starts 2024-09-17T07:07:21.395 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:21 smithi100 ceph-mon[39462]: 5.a scrub ok 2024-09-17T07:07:21.395 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:21 smithi100 ceph-mon[39462]: 6.9 scrub starts 2024-09-17T07:07:21.395 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:21 smithi100 ceph-mon[39462]: 6.9 scrub ok 2024-09-17T07:07:21.396 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:21 smithi100 ceph-mon[39462]: 2.f scrub starts 2024-09-17T07:07:21.396 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:21 smithi100 ceph-mon[39462]: 2.f scrub ok 2024-09-17T07:07:21.442 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:07:21.479 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:21.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:21.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:07:21.484 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:07:21.582 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:07:21.609 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:21.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:21.610 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:07:21.711 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:07:21.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:21.762 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:21.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:07:21.863 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:07:21.892 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:21.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:21.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:07:21.994 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:07:22.032 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:22.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:22.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:07:22.134 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:07:22.171 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:22.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:22.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:07:22.273 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:07:22.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:22.300 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:22.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:07:22.401 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:07:22.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:22.441 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:22.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:07:22.489 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:22 smithi135 ceph-mon[33238]: 2.14 scrub starts 2024-09-17T07:07:22.489 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:22 smithi135 ceph-mon[33238]: 2.14 scrub ok 2024-09-17T07:07:22.489 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:22 smithi135 ceph-mon[33238]: 2.1c scrub starts 2024-09-17T07:07:22.489 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:22 smithi135 ceph-mon[33238]: 2.1c scrub ok 2024-09-17T07:07:22.489 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:22 smithi135 ceph-mon[33238]: 3.1 scrub starts 2024-09-17T07:07:22.489 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:22 smithi135 ceph-mon[33238]: 3.1 scrub ok 2024-09-17T07:07:22.489 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:22 smithi135 ceph-mon[33238]: pgmap v222: 158 pgs: 158 active+clean; 1.3 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 477 op/s 2024-09-17T07:07:22.542 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:07:22.587 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:22.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:22.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:07:22.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:22 smithi100 ceph-mon[29092]: 2.14 scrub starts 2024-09-17T07:07:22.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:22 smithi100 ceph-mon[29092]: 2.14 scrub ok 2024-09-17T07:07:22.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:22 smithi100 ceph-mon[29092]: 2.1c scrub starts 2024-09-17T07:07:22.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:22 smithi100 ceph-mon[29092]: 2.1c scrub ok 2024-09-17T07:07:22.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:22 smithi100 ceph-mon[29092]: 3.1 scrub starts 2024-09-17T07:07:22.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:22 smithi100 ceph-mon[29092]: 3.1 scrub ok 2024-09-17T07:07:22.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:22 smithi100 ceph-mon[29092]: pgmap v222: 158 pgs: 158 active+clean; 1.3 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 477 op/s 2024-09-17T07:07:22.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:22 smithi100 ceph-mon[39462]: 2.14 scrub starts 2024-09-17T07:07:22.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:22 smithi100 ceph-mon[39462]: 2.14 scrub ok 2024-09-17T07:07:22.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:22 smithi100 ceph-mon[39462]: 2.1c scrub starts 2024-09-17T07:07:22.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:22 smithi100 ceph-mon[39462]: 2.1c scrub ok 2024-09-17T07:07:22.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:22 smithi100 ceph-mon[39462]: 3.1 scrub starts 2024-09-17T07:07:22.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:22 smithi100 ceph-mon[39462]: 3.1 scrub ok 2024-09-17T07:07:22.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:22 smithi100 ceph-mon[39462]: pgmap v222: 158 pgs: 158 active+clean; 1.3 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 477 op/s 2024-09-17T07:07:22.635 INFO:teuthology.orchestra.run.smithi135.stdout:ERROR: (22) Invalid argument 2024-09-17T07:07:22.635 INFO:teuthology.orchestra.run.smithi135.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:07:22.688 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:07:22.714 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:22.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:22.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:07:22.816 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:07:22.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:22.859 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:22.859 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:07:22.960 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:07:22.989 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:22.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:22.990 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:07:23.091 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:07:23.119 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:23.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:23.120 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:07:23.221 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:07:23.241 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:07:23.250 INFO:teuthology.orchestra.run.smithi100.stderr:nodeep-scrub is unset 2024-09-17T07:07:23.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:23.276 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:23.276 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:07:23.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:23 smithi135 ceph-mon[33238]: 6.1a scrub starts 2024-09-17T07:07:23.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:23 smithi135 ceph-mon[33238]: 6.1a scrub ok 2024-09-17T07:07:23.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:23 smithi135 ceph-mon[33238]: 4.1e scrub starts 2024-09-17T07:07:23.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:23 smithi135 ceph-mon[33238]: 4.1e scrub ok 2024-09-17T07:07:23.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:23 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2850412481' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:07:23.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:23 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:07:23.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:23 smithi135 ceph-mon[33238]: 2.18 scrub starts 2024-09-17T07:07:23.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:23 smithi135 ceph-mon[33238]: 2.18 scrub ok 2024-09-17T07:07:23.377 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:07:23.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:23.420 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:23.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:07:23.443 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-17T07:07:23.521 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:07:23.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:23.561 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:23.561 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:07:23.617 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:23 smithi100 ceph-mon[29092]: 6.1a scrub starts 2024-09-17T07:07:23.617 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:23 smithi100 ceph-mon[29092]: 6.1a scrub ok 2024-09-17T07:07:23.617 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:23 smithi100 ceph-mon[29092]: 4.1e scrub starts 2024-09-17T07:07:23.617 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:23 smithi100 ceph-mon[29092]: 4.1e scrub ok 2024-09-17T07:07:23.617 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:23 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2850412481' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:07:23.617 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:23 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:07:23.617 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:23 smithi100 ceph-mon[29092]: 2.18 scrub starts 2024-09-17T07:07:23.618 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:23 smithi100 ceph-mon[29092]: 2.18 scrub ok 2024-09-17T07:07:23.618 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T07:07:23.619 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:23 smithi100 ceph-mon[39462]: 6.1a scrub starts 2024-09-17T07:07:23.619 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:23 smithi100 ceph-mon[39462]: 6.1a scrub ok 2024-09-17T07:07:23.619 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:23 smithi100 ceph-mon[39462]: 4.1e scrub starts 2024-09-17T07:07:23.619 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:23 smithi100 ceph-mon[39462]: 4.1e scrub ok 2024-09-17T07:07:23.619 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:23 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2850412481' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:07:23.619 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:23 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:07:23.619 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:23 smithi100 ceph-mon[39462]: 2.18 scrub starts 2024-09-17T07:07:23.619 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:23 smithi100 ceph-mon[39462]: 2.18 scrub ok 2024-09-17T07:07:23.628 INFO:teuthology.orchestra.run.smithi100.stderr:dumped all 2024-09-17T07:07:23.662 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:07:23.702 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:23.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:23.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:07:23.727 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:07:23.803 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:07:23.844 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:23.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:23.845 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:07:23.946 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:07:23.982 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:23.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:23.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:07:24.084 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:07:24.115 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:24.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:24.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:07:24.217 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:07:24.247 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:24.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:24.248 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:07:24.271 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:24 smithi100 ceph-mon[39462]: 2.1d scrub starts 2024-09-17T07:07:24.272 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:24 smithi100 ceph-mon[39462]: 2.1d scrub ok 2024-09-17T07:07:24.272 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:24 smithi100 ceph-mon[39462]: pgmap v223: 158 pgs: 158 active+clean; 1.3 GiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 572 B/s rd, 0 B/s wr, 60 op/s 2024-09-17T07:07:24.272 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:24 smithi100 ceph-mon[39462]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T07:07:24.272 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:24 smithi100 ceph-mon[39462]: Cluster is now healthy 2024-09-17T07:07:24.272 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:24 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T07:07:24.272 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:24 smithi100 ceph-mon[39462]: osdmap e161: 8 total, 8 up, 8 in 2024-09-17T07:07:24.272 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:24 smithi100 ceph-mon[39462]: 5.19 scrub starts 2024-09-17T07:07:24.272 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:24 smithi100 ceph-mon[39462]: 5.19 scrub ok 2024-09-17T07:07:24.272 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:24 smithi100 ceph-mon[39462]: 7.2 scrub starts 2024-09-17T07:07:24.272 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:24 smithi100 ceph-mon[39462]: 6.19 scrub starts 2024-09-17T07:07:24.272 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:24 smithi100 ceph-mon[39462]: 6.19 scrub ok 2024-09-17T07:07:24.273 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:24 smithi100 ceph-mon[39462]: 7.2 scrub ok 2024-09-17T07:07:24.273 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:24 smithi100 ceph-mon[29092]: 2.1d scrub starts 2024-09-17T07:07:24.273 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:24 smithi100 ceph-mon[29092]: 2.1d scrub ok 2024-09-17T07:07:24.273 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:24 smithi100 ceph-mon[29092]: pgmap v223: 158 pgs: 158 active+clean; 1.3 GiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 572 B/s rd, 0 B/s wr, 60 op/s 2024-09-17T07:07:24.273 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:24 smithi100 ceph-mon[29092]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T07:07:24.273 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:24 smithi100 ceph-mon[29092]: Cluster is now healthy 2024-09-17T07:07:24.273 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:24 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T07:07:24.273 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:24 smithi100 ceph-mon[29092]: osdmap e161: 8 total, 8 up, 8 in 2024-09-17T07:07:24.273 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:24 smithi100 ceph-mon[29092]: 5.19 scrub starts 2024-09-17T07:07:24.273 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:24 smithi100 ceph-mon[29092]: 5.19 scrub ok 2024-09-17T07:07:24.273 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:24 smithi100 ceph-mon[29092]: 7.2 scrub starts 2024-09-17T07:07:24.274 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:24 smithi100 ceph-mon[29092]: 6.19 scrub starts 2024-09-17T07:07:24.274 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:24 smithi100 ceph-mon[29092]: 6.19 scrub ok 2024-09-17T07:07:24.274 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:24 smithi100 ceph-mon[29092]: 7.2 scrub ok 2024-09-17T07:07:24.348 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:07:24.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:24.419 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:24.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:07:24.520 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:07:24.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:24 smithi135 ceph-mon[33238]: 2.1d scrub starts 2024-09-17T07:07:24.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:24 smithi135 ceph-mon[33238]: 2.1d scrub ok 2024-09-17T07:07:24.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:24 smithi135 ceph-mon[33238]: pgmap v223: 158 pgs: 158 active+clean; 1.3 GiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 572 B/s rd, 0 B/s wr, 60 op/s 2024-09-17T07:07:24.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:24 smithi135 ceph-mon[33238]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T07:07:24.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:24 smithi135 ceph-mon[33238]: Cluster is now healthy 2024-09-17T07:07:24.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:24 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T07:07:24.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:24 smithi135 ceph-mon[33238]: osdmap e161: 8 total, 8 up, 8 in 2024-09-17T07:07:24.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:24 smithi135 ceph-mon[33238]: 5.19 scrub starts 2024-09-17T07:07:24.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:24 smithi135 ceph-mon[33238]: 5.19 scrub ok 2024-09-17T07:07:24.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:24 smithi135 ceph-mon[33238]: 7.2 scrub starts 2024-09-17T07:07:24.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:24 smithi135 ceph-mon[33238]: 6.19 scrub starts 2024-09-17T07:07:24.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:24 smithi135 ceph-mon[33238]: 6.19 scrub ok 2024-09-17T07:07:24.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:24 smithi135 ceph-mon[33238]: 7.2 scrub ok 2024-09-17T07:07:24.558 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:24.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:24.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:07:24.660 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:07:24.684 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:24.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:24.685 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:07:24.786 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:07:24.826 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:24.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:24.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:07:24.928 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:07:24.962 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:24.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:24.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:07:25.064 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:07:25.102 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:25.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:25.103 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:07:25.204 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:07:25.237 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:25.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:25.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:07:25.338 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:07:25.374 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:25.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:25.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:07:25.476 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:07:25.515 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:25.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:25.516 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:07:25.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:25 smithi135 ceph-mon[33238]: from='client.14946 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:07:25.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:25 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:07:25.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:25 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:07:25.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:25 smithi135 ceph-mon[33238]: osdmap e162: 8 total, 8 up, 8 in 2024-09-17T07:07:25.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:25 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:07:25.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:25 smithi135 ceph-mon[33238]: 5.c scrub starts 2024-09-17T07:07:25.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:25 smithi135 ceph-mon[33238]: 5.c scrub ok 2024-09-17T07:07:25.617 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:07:25.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:25 smithi100 ceph-mon[29092]: from='client.14946 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:07:25.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:25 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:07:25.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:25 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:07:25.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:25 smithi100 ceph-mon[29092]: osdmap e162: 8 total, 8 up, 8 in 2024-09-17T07:07:25.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:25 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:07:25.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:25 smithi100 ceph-mon[29092]: 5.c scrub starts 2024-09-17T07:07:25.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:25 smithi100 ceph-mon[29092]: 5.c scrub ok 2024-09-17T07:07:25.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:25 smithi100 ceph-mon[39462]: from='client.14946 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:07:25.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:25 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:07:25.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:25 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:07:25.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:25 smithi100 ceph-mon[39462]: osdmap e162: 8 total, 8 up, 8 in 2024-09-17T07:07:25.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:25 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:07:25.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:25 smithi100 ceph-mon[39462]: 5.c scrub starts 2024-09-17T07:07:25.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:25 smithi100 ceph-mon[39462]: 5.c scrub ok 2024-09-17T07:07:25.638 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:25.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:25.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:07:25.739 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:07:25.775 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:25.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:25.777 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:07:25.877 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:07:25.917 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:25.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:25.919 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:07:26.020 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-17T07:07:26.020 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:07:26.066 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:26.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:26.068 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:07:26.169 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:07:26.196 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:26.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:26.197 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:07:26.241 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:07:25 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:07:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:07:26.298 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:07:26.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:26.339 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:26.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:07:26.434 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:26 smithi135 ceph-mon[33238]: pgmap v226: 158 pgs: 158 active+clean; 964 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 793 op/s 2024-09-17T07:07:26.435 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:26 smithi135 ceph-mon[33238]: 6.4 scrub starts 2024-09-17T07:07:26.435 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:26 smithi135 ceph-mon[33238]: 6.4 scrub ok 2024-09-17T07:07:26.435 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:26 smithi135 ceph-mon[33238]: 7.3 deep-scrub starts 2024-09-17T07:07:26.440 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:07:26.445 INFO:teuthology.orchestra.run.smithi135.stdout:ERROR: (22) Invalid argument 2024-09-17T07:07:26.446 INFO:teuthology.orchestra.run.smithi135.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:07:26.472 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:26.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:26.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:07:26.574 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:07:26.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:26.604 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:26.604 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:07:26.630 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:26 smithi100 ceph-mon[29092]: pgmap v226: 158 pgs: 158 active+clean; 964 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 793 op/s 2024-09-17T07:07:26.630 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:26 smithi100 ceph-mon[29092]: 6.4 scrub starts 2024-09-17T07:07:26.630 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:26 smithi100 ceph-mon[29092]: 6.4 scrub ok 2024-09-17T07:07:26.630 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:26 smithi100 ceph-mon[29092]: 7.3 deep-scrub starts 2024-09-17T07:07:26.630 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:26 smithi100 ceph-mon[39462]: pgmap v226: 158 pgs: 158 active+clean; 964 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 793 op/s 2024-09-17T07:07:26.630 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:26 smithi100 ceph-mon[39462]: 6.4 scrub starts 2024-09-17T07:07:26.631 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:26 smithi100 ceph-mon[39462]: 6.4 scrub ok 2024-09-17T07:07:26.631 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:26 smithi100 ceph-mon[39462]: 7.3 deep-scrub starts 2024-09-17T07:07:26.705 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:07:26.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:26.728 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:26.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:07:26.829 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:07:26.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:26.863 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:26.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:07:26.964 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:07:26.999 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:26.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:27.000 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:07:27.101 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:07:27.125 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:27.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:27.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:07:27.140 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-17T07:07:27.227 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:07:27.263 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:27.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:27.264 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:07:27.365 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:07:27.398 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:27.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:27.399 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:07:27.464 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:07:27.500 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:07:27.523 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:27 smithi100 ceph-mon[29092]: 6.16 scrub starts 2024-09-17T07:07:27.523 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:27 smithi100 ceph-mon[29092]: 6.16 scrub ok 2024-09-17T07:07:27.523 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:27 smithi100 ceph-mon[29092]: 6.13 scrub starts 2024-09-17T07:07:27.523 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:27 smithi100 ceph-mon[29092]: 6.13 scrub ok 2024-09-17T07:07:27.524 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:27 smithi100 ceph-mon[39462]: 6.16 scrub starts 2024-09-17T07:07:27.524 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:27 smithi100 ceph-mon[39462]: 6.16 scrub ok 2024-09-17T07:07:27.524 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:27 smithi100 ceph-mon[39462]: 6.13 scrub starts 2024-09-17T07:07:27.524 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:27 smithi100 ceph-mon[39462]: 6.13 scrub ok 2024-09-17T07:07:27.526 INFO:teuthology.orchestra.run.smithi100.stdout:{"pg_ready":true,"pg_map":{"version":224,"stamp":"2024-09-17T07:07:23.237944+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1208064302,"num_objects":18623,"num_object_clones":0,"num_object_copies":55869,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18623,"num_whiteouts":0,"num_read":1226,"num_read_kb":1072,"num_write":165481,"num_write_kb":5309244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15842,"num_bytes_recovered":1038681806,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134966,"ondisk_log_size":134966,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":478,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":5113336,"kb_used_data":3851184,"kb_used_omap":12,"kb_used_meta":1261939,"kb_avail":744847880,"statfs":{"total":767960285184,"available":762724229120,"internally_reserved":0,"allocated":3943612416,"data_stored":3940463018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":1292226104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,3,3,2,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-622297088,"num_objects":-9495,"num_object_clones":0,"num_object_copies":-28485,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-9495,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":3265,"num_write_kb":-224992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.136267"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":107,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.177897+0000","last_change":"2024-09-17T07:04:03.551432+0000","last_active":"2024-09-17T07:07:14.177897+0000","last_peered":"2024-09-17T07:07:14.177897+0000","last_clean":"2024-09-17T07:07:14.177897+0000","last_became_active":"2024-09-17T07:02:18.365773+0000","last_became_peered":"2024-09-17T07:02:18.365773+0000","last_unstale":"2024-09-17T07:07:14.177897+0000","last_undegraded":"2024-09-17T07:07:14.177897+0000","last_fullsized":"2024-09-17T07:07:14.177897+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:03.551314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:04:03.551314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001762752,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":126,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.175957+0000","last_change":"2024-09-17T07:04:56.479564+0000","last_active":"2024-09-17T07:07:14.175957+0000","last_peered":"2024-09-17T07:07:14.175957+0000","last_clean":"2024-09-17T07:07:14.175957+0000","last_became_active":"2024-09-17T07:02:12.006754+0000","last_became_peered":"2024-09-17T07:02:12.006754+0000","last_unstale":"2024-09-17T07:07:14.175957+0000","last_undegraded":"2024-09-17T07:07:14.175957+0000","last_fullsized":"2024-09-17T07:07:14.175957+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:56.479415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:04:56.479415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0038250369999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"75'15","reported_seq":148,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.178318+0000","last_change":"2024-09-17T07:06:29.008173+0000","last_active":"2024-09-17T07:07:14.178318+0000","last_peered":"2024-09-17T07:07:14.178318+0000","last_clean":"2024-09-17T07:07:14.178318+0000","last_became_active":"2024-09-17T07:02:14.012962+0000","last_became_peered":"2024-09-17T07:02:14.012962+0000","last_unstale":"2024-09-17T07:07:14.178318+0000","last_undegraded":"2024-09-17T07:07:14.178318+0000","last_fullsized":"2024-09-17T07:07:14.178318+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-09-17T07:06:29.008025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:06:29.008025+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:47.191492+0000","scrub_duration":0.011622218,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":111,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.175319+0000","last_change":"2024-09-17T07:04:39.726884+0000","last_active":"2024-09-17T07:07:14.175319+0000","last_peered":"2024-09-17T07:07:14.175319+0000","last_clean":"2024-09-17T07:07:14.175319+0000","last_became_active":"2024-09-17T07:02:16.014928+0000","last_became_peered":"2024-09-17T07:02:16.014928+0000","last_unstale":"2024-09-17T07:07:14.175319+0000","last_undegraded":"2024-09-17T07:07:14.175319+0000","last_fullsized":"2024-09-17T07:07:14.175319+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:39.726764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:04:39.726764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013379520000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":125,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:15.767618+0000","last_change":"2024-09-17T07:07:15.767618+0000","last_active":"2024-09-17T07:07:15.767618+0000","last_peered":"2024-09-17T07:07:15.767618+0000","last_clean":"2024-09-17T07:07:15.767618+0000","last_became_active":"2024-09-17T07:02:12.010069+0000","last_became_peered":"2024-09-17T07:02:12.010069+0000","last_unstale":"2024-09-17T07:07:15.767618+0000","last_undegraded":"2024-09-17T07:07:15.767618+0000","last_fullsized":"2024-09-17T07:07:15.767618+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:15.767495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:07:15.767495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:27.964043+0000","scrub_duration":0.013487149,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"75'9","reported_seq":140,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.175428+0000","last_change":"2024-09-17T07:04:46.742364+0000","last_active":"2024-09-17T07:07:14.175428+0000","last_peered":"2024-09-17T07:07:14.175428+0000","last_clean":"2024-09-17T07:07:14.175428+0000","last_became_active":"2024-09-17T07:02:14.016244+0000","last_became_peered":"2024-09-17T07:02:14.016244+0000","last_unstale":"2024-09-17T07:07:14.175428+0000","last_undegraded":"2024-09-17T07:07:14.175428+0000","last_fullsized":"2024-09-17T07:07:14.175428+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:04:46.742196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:04:46.742196+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0030184949999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":111,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.181497+0000","last_change":"2024-09-17T07:06:15.173705+0000","last_active":"2024-09-17T07:07:14.181497+0000","last_peered":"2024-09-17T07:07:14.181497+0000","last_clean":"2024-09-17T07:07:14.181497+0000","last_became_active":"2024-09-17T07:02:16.014380+0000","last_became_peered":"2024-09-17T07:02:16.014380+0000","last_unstale":"2024-09-17T07:07:14.181497+0000","last_undegraded":"2024-09-17T07:07:14.181497+0000","last_fullsized":"2024-09-17T07:07:14.181497+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:37.112954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:37.112954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001466824,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":121,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-17T07:07:21.791420+0000","last_change":"2024-09-17T07:07:21.791530+0000","last_active":"2024-09-17T07:07:21.791420+0000","last_peered":"2024-09-17T07:07:21.791420+0000","last_clean":"2024-09-17T07:07:21.791420+0000","last_became_active":"2024-09-17T07:02:18.020790+0000","last_became_peered":"2024-09-17T07:02:18.020790+0000","last_unstale":"2024-09-17T07:07:21.791420+0000","last_undegraded":"2024-09-17T07:07:21.791420+0000","last_fullsized":"2024-09-17T07:07:21.791420+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:21.791399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:07:21.791399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:25.370399+0000","scrub_duration":0.0069986889999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"160'8067","reported_seq":13250,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-17T07:07:18.917012+0000","last_change":"2024-09-17T07:05:31.148987+0000","last_active":"2024-09-17T07:07:18.917012+0000","last_peered":"2024-09-17T07:07:18.917012+0000","last_clean":"2024-09-17T07:07:18.917012+0000","last_became_active":"2024-09-17T07:05:31.144189+0000","last_became_peered":"2024-09-17T07:05:31.144189+0000","last_unstale":"2024-09-17T07:07:18.917012+0000","last_undegraded":"2024-09-17T07:07:18.917012+0000","last_fullsized":"2024-09-17T07:07:18.917012+0000","mapping_epoch":125,"log_start":"108'3007","ondisk_log_start":"108'3007","created":111,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_clean_scrub_stamp":"2024-09-17T07:04:19.785747+0000","objects_scrubbed":0,"log_size":5060,"log_dups_size":0,"ondisk_log_size":5060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48496640,"num_objects":740,"num_object_clones":0,"num_object_copies":2220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6356,"num_write_kb":227072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1695,"num_bytes_recovered":111083520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":126,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.013492+0000","last_change":"2024-09-17T07:07:23.013609+0000","last_active":"2024-09-17T07:07:23.013492+0000","last_peered":"2024-09-17T07:07:23.013492+0000","last_clean":"2024-09-17T07:07:23.013492+0000","last_became_active":"2024-09-17T07:02:12.011129+0000","last_became_peered":"2024-09-17T07:02:12.011129+0000","last_unstale":"2024-09-17T07:07:23.013492+0000","last_undegraded":"2024-09-17T07:07:23.013492+0000","last_fullsized":"2024-09-17T07:07:23.013492+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:23.013470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:43.060102+0000","last_clean_scrub_stamp":"2024-09-17T07:07:23.013470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:28.079025+0000","scrub_duration":0.006146281,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"75'5","reported_seq":135,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-17T07:07:17.182667+0000","last_change":"2024-09-17T07:07:15.848170+0000","last_active":"2024-09-17T07:07:17.182667+0000","last_peered":"2024-09-17T07:07:17.182667+0000","last_clean":"2024-09-17T07:07:17.182667+0000","last_became_active":"2024-09-17T07:02:14.016640+0000","last_became_peered":"2024-09-17T07:02:14.016640+0000","last_unstale":"2024-09-17T07:07:17.182667+0000","last_undegraded":"2024-09-17T07:07:17.182667+0000","last_fullsized":"2024-09-17T07:07:17.182667+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-09-17T07:07:15.848071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:07:15.848071+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:31.939834+0000","scrub_duration":0.010596959,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":122,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.164543+0000","last_change":"2024-09-17T07:04:55.853224+0000","last_active":"2024-09-17T07:07:14.164543+0000","last_peered":"2024-09-17T07:07:14.164543+0000","last_clean":"2024-09-17T07:07:14.164543+0000","last_became_active":"2024-09-17T07:02:16.015401+0000","last_became_peered":"2024-09-17T07:02:16.015401+0000","last_unstale":"2024-09-17T07:07:14.164543+0000","last_undegraded":"2024-09-17T07:07:14.164543+0000","last_fullsized":"2024-09-17T07:07:14.164543+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:55.853108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:04:55.853108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017379610000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":119,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.172718+0000","last_change":"2024-09-17T07:04:40.586357+0000","last_active":"2024-09-17T07:07:14.172718+0000","last_peered":"2024-09-17T07:07:14.172718+0000","last_clean":"2024-09-17T07:07:14.172718+0000","last_became_active":"2024-09-17T07:02:18.023680+0000","last_became_peered":"2024-09-17T07:02:18.023680+0000","last_unstale":"2024-09-17T07:07:14.172718+0000","last_undegraded":"2024-09-17T07:07:14.172718+0000","last_fullsized":"2024-09-17T07:07:14.172718+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:40.586234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:04:40.586234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032101370000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"160'7878","reported_seq":14664,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.070499+0000","last_change":"2024-09-17T07:06:40.353247+0000","last_active":"2024-09-17T07:07:23.070499+0000","last_peered":"2024-09-17T07:07:23.070499+0000","last_clean":"2024-09-17T07:07:23.070499+0000","last_became_active":"2024-09-17T07:06:06.086253+0000","last_became_peered":"2024-09-17T07:06:06.086253+0000","last_unstale":"2024-09-17T07:07:23.070499+0000","last_undegraded":"2024-09-17T07:07:23.070499+0000","last_fullsized":"2024-09-17T07:07:23.070499+0000","mapping_epoch":139,"log_start":"105'1045","ondisk_log_start":"105'1045","created":111,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_clean_scrub_stamp":"2024-09-17T07:04:19.785747+0000","objects_scrubbed":0,"log_size":6833,"log_dups_size":0,"ondisk_log_size":6833,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35127296,"num_objects":536,"num_object_clones":0,"num_object_copies":1608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6222,"num_write_kb":216256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1671,"num_bytes_recovered":109445120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":126,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-17T07:07:21.000654+0000","last_change":"2024-09-17T07:07:21.000757+0000","last_active":"2024-09-17T07:07:21.000654+0000","last_peered":"2024-09-17T07:07:21.000654+0000","last_clean":"2024-09-17T07:07:21.000654+0000","last_became_active":"2024-09-17T07:02:12.012234+0000","last_became_peered":"2024-09-17T07:02:12.012234+0000","last_unstale":"2024-09-17T07:07:21.000654+0000","last_undegraded":"2024-09-17T07:07:21.000654+0000","last_fullsized":"2024-09-17T07:07:21.000654+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:21.000635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:45.029247+0000","last_clean_scrub_stamp":"2024-09-17T07:07:21.000635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:23.254719+0000","scrub_duration":0.0068373009999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"75'9","reported_seq":129,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.196984+0000","last_change":"2024-09-17T07:04:35.582659+0000","last_active":"2024-09-17T07:07:14.196984+0000","last_peered":"2024-09-17T07:07:14.196984+0000","last_clean":"2024-09-17T07:07:14.196984+0000","last_became_active":"2024-09-17T07:02:14.013337+0000","last_became_peered":"2024-09-17T07:02:14.013337+0000","last_unstale":"2024-09-17T07:07:14.196984+0000","last_undegraded":"2024-09-17T07:07:14.196984+0000","last_fullsized":"2024-09-17T07:07:14.196984+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:04:35.582524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:04:35.582524+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0042430649999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":121,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-17T07:07:17.188136+0000","last_change":"2024-09-17T07:06:34.739225+0000","last_active":"2024-09-17T07:07:17.188136+0000","last_peered":"2024-09-17T07:07:17.188136+0000","last_clean":"2024-09-17T07:07:17.188136+0000","last_became_active":"2024-09-17T07:02:16.023774+0000","last_became_peered":"2024-09-17T07:02:16.023774+0000","last_unstale":"2024-09-17T07:07:17.188136+0000","last_undegraded":"2024-09-17T07:07:17.188136+0000","last_fullsized":"2024-09-17T07:07:17.188136+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:34.739103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:06:34.739103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:47.725471+0000","scrub_duration":0.0024801699999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":109,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.172631+0000","last_change":"2024-09-17T07:04:07.066830+0000","last_active":"2024-09-17T07:07:14.172631+0000","last_peered":"2024-09-17T07:07:14.172631+0000","last_clean":"2024-09-17T07:07:14.172631+0000","last_became_active":"2024-09-17T07:02:18.029178+0000","last_became_peered":"2024-09-17T07:02:18.029178+0000","last_unstale":"2024-09-17T07:07:14.172631+0000","last_undegraded":"2024-09-17T07:07:14.172631+0000","last_fullsized":"2024-09-17T07:07:14.172631+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:07.066751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:04:07.066751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0011842319999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"160'12548","reported_seq":12734,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-17T07:07:18.204680+0000","last_change":"2024-09-17T07:06:29.404750+0000","last_active":"2024-09-17T07:07:18.204680+0000","last_peered":"2024-09-17T07:07:18.204680+0000","last_clean":"2024-09-17T07:07:18.204680+0000","last_became_active":"2024-09-17T07:04:20.804060+0000","last_became_peered":"2024-09-17T07:04:20.804060+0000","last_unstale":"2024-09-17T07:07:18.204680+0000","last_undegraded":"2024-09-17T07:07:18.204680+0000","last_fullsized":"2024-09-17T07:07:18.204680+0000","mapping_epoch":99,"log_start":"135'6800","ondisk_log_start":"135'6800","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"149'8952","last_scrub_stamp":"2024-09-17T07:06:29.404590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_clean_scrub_stamp":"2024-09-17T07:06:29.404590+0000","objects_scrubbed":5038,"log_size":5748,"log_dups_size":0,"ondisk_log_size":5748,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:54.135340+0000","scrub_duration":2.2230415670000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89784320,"num_objects":1370,"num_object_clones":0,"num_object_copies":4110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1370,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12548,"num_write_kb":445376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"95'25","reported_seq":160,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.198262+0000","last_change":"2024-09-17T07:06:41.707070+0000","last_active":"2024-09-17T07:07:14.198262+0000","last_peered":"2024-09-17T07:07:14.198262+0000","last_clean":"2024-09-17T07:07:14.198262+0000","last_became_active":"2024-09-17T07:02:14.016270+0000","last_became_peered":"2024-09-17T07:02:14.016270+0000","last_unstale":"2024-09-17T07:07:14.198262+0000","last_undegraded":"2024-09-17T07:07:14.198262+0000","last_fullsized":"2024-09-17T07:07:14.198262+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'25","last_scrub_stamp":"2024-09-17T07:06:41.706948+0000","last_deep_scrub":"95'25","last_deep_scrub_stamp":"2024-09-17T07:04:09.450039+0000","last_clean_scrub_stamp":"2024-09-17T07:06:41.706948+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:03.400798+0000","scrub_duration":0.016404887999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":126,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.192807+0000","last_change":"2024-09-17T07:04:42.771521+0000","last_active":"2024-09-17T07:07:14.192807+0000","last_peered":"2024-09-17T07:07:14.192807+0000","last_clean":"2024-09-17T07:07:14.192807+0000","last_became_active":"2024-09-17T07:02:12.011258+0000","last_became_peered":"2024-09-17T07:02:12.011258+0000","last_unstale":"2024-09-17T07:07:14.192807+0000","last_undegraded":"2024-09-17T07:07:14.192807+0000","last_fullsized":"2024-09-17T07:07:14.192807+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:42.771404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:04:42.771404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017072459999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":111,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.181869+0000","last_change":"2024-09-17T07:06:22.250673+0000","last_active":"2024-09-17T07:07:14.181869+0000","last_peered":"2024-09-17T07:07:14.181869+0000","last_clean":"2024-09-17T07:07:14.181869+0000","last_became_active":"2024-09-17T07:02:16.018147+0000","last_became_peered":"2024-09-17T07:02:16.018147+0000","last_unstale":"2024-09-17T07:07:14.181869+0000","last_undegraded":"2024-09-17T07:07:14.181869+0000","last_fullsized":"2024-09-17T07:07:14.181869+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:44.044635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:44.044635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014202049999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":87,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:16.737235+0000","last_change":"2024-09-17T07:07:16.737235+0000","last_active":"2024-09-17T07:07:16.737235+0000","last_peered":"2024-09-17T07:07:16.737235+0000","last_clean":"2024-09-17T07:07:16.737235+0000","last_became_active":"2024-09-17T07:04:15.113196+0000","last_became_peered":"2024-09-17T07:04:15.113196+0000","last_unstale":"2024-09-17T07:07:16.737235+0000","last_undegraded":"2024-09-17T07:07:16.737235+0000","last_fullsized":"2024-09-17T07:07:16.737235+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:16.737086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:42.984088+0000","last_clean_scrub_stamp":"2024-09-17T07:07:16.737086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:19.824197+0000","scrub_duration":0.0083716439999999993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"160'12563","reported_seq":12634,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-17T07:07:18.910690+0000","last_change":"2024-09-17T07:04:20.800521+0000","last_active":"2024-09-17T07:07:18.910690+0000","last_peered":"2024-09-17T07:07:18.910690+0000","last_clean":"2024-09-17T07:07:18.910690+0000","last_became_active":"2024-09-17T07:04:20.800260+0000","last_became_peered":"2024-09-17T07:04:20.800260+0000","last_unstale":"2024-09-17T07:07:18.910690+0000","last_undegraded":"2024-09-17T07:07:18.910690+0000","last_fullsized":"2024-09-17T07:07:18.910690+0000","mapping_epoch":99,"log_start":"142'7700","ondisk_log_start":"142'7700","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_clean_scrub_stamp":"2024-09-17T07:04:19.785747+0000","objects_scrubbed":0,"log_size":4863,"log_dups_size":0,"ondisk_log_size":4863,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83296256,"num_objects":1271,"num_object_clones":0,"num_object_copies":3813,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1271,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12563,"num_write_kb":442688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"75'10","reported_seq":139,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.199182+0000","last_change":"2024-09-17T07:04:43.722476+0000","last_active":"2024-09-17T07:07:14.199182+0000","last_peered":"2024-09-17T07:07:14.199182+0000","last_clean":"2024-09-17T07:07:14.199182+0000","last_became_active":"2024-09-17T07:02:14.007702+0000","last_became_peered":"2024-09-17T07:02:14.007702+0000","last_unstale":"2024-09-17T07:07:14.199182+0000","last_undegraded":"2024-09-17T07:07:14.199182+0000","last_fullsized":"2024-09-17T07:07:14.199182+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-09-17T07:04:43.722288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:04:43.722288+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002985845,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":125,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.182216+0000","last_change":"2024-09-17T07:06:34.075343+0000","last_active":"2024-09-17T07:07:14.182216+0000","last_peered":"2024-09-17T07:07:14.182216+0000","last_clean":"2024-09-17T07:07:14.182216+0000","last_became_active":"2024-09-17T07:02:12.007553+0000","last_became_peered":"2024-09-17T07:02:12.007553+0000","last_unstale":"2024-09-17T07:07:14.182216+0000","last_undegraded":"2024-09-17T07:07:14.182216+0000","last_fullsized":"2024-09-17T07:07:14.182216+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:34.075214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:06:34.075214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:51.954252+0000","scrub_duration":0.031637767999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"98'9","reported_seq":144,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-17T07:07:17.187606+0000","last_change":"2024-09-17T07:06:34.881845+0000","last_active":"2024-09-17T07:07:17.187606+0000","last_peered":"2024-09-17T07:07:17.187606+0000","last_clean":"2024-09-17T07:07:17.187606+0000","last_became_active":"2024-09-17T07:04:15.110797+0000","last_became_peered":"2024-09-17T07:04:15.110797+0000","last_unstale":"2024-09-17T07:07:17.187606+0000","last_undegraded":"2024-09-17T07:07:17.187606+0000","last_fullsized":"2024-09-17T07:07:17.187606+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-09-17T07:06:34.881734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:06:34.881734+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:54.291547+0000","scrub_duration":0.027541755000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":120,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.198569+0000","last_change":"2024-09-17T07:06:42.703538+0000","last_active":"2024-09-17T07:07:14.198569+0000","last_peered":"2024-09-17T07:07:14.198569+0000","last_clean":"2024-09-17T07:07:14.198569+0000","last_became_active":"2024-09-17T07:02:18.017460+0000","last_became_peered":"2024-09-17T07:02:18.017460+0000","last_unstale":"2024-09-17T07:07:14.198569+0000","last_undegraded":"2024-09-17T07:07:14.198569+0000","last_fullsized":"2024-09-17T07:07:14.198569+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:42.703326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:06:42.703326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:48.219915+0000","scrub_duration":0.004198528,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"160'12973","reported_seq":13045,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.092171+0000","last_change":"2024-09-17T07:06:22.165610+0000","last_active":"2024-09-17T07:07:23.092171+0000","last_peered":"2024-09-17T07:07:23.092171+0000","last_clean":"2024-09-17T07:07:23.092171+0000","last_became_active":"2024-09-17T07:04:20.804188+0000","last_became_peered":"2024-09-17T07:04:20.804188+0000","last_unstale":"2024-09-17T07:07:23.092171+0000","last_undegraded":"2024-09-17T07:07:23.092171+0000","last_fullsized":"2024-09-17T07:07:23.092171+0000","mapping_epoch":99,"log_start":"140'7300","ondisk_log_start":"140'7300","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_clean_scrub_stamp":"2024-09-17T07:04:19.785747+0000","objects_scrubbed":0,"log_size":5673,"log_dups_size":0,"ondisk_log_size":5673,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63897600,"num_objects":975,"num_object_clones":0,"num_object_copies":2925,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12973,"num_write_kb":446336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"75'15","reported_seq":148,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.181820+0000","last_change":"2024-09-17T07:04:40.954912+0000","last_active":"2024-09-17T07:07:14.181820+0000","last_peered":"2024-09-17T07:07:14.181820+0000","last_clean":"2024-09-17T07:07:14.181820+0000","last_became_active":"2024-09-17T07:02:14.016397+0000","last_became_peered":"2024-09-17T07:02:14.016397+0000","last_unstale":"2024-09-17T07:07:14.181820+0000","last_undegraded":"2024-09-17T07:07:14.181820+0000","last_fullsized":"2024-09-17T07:07:14.181820+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-09-17T07:04:40.954768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:04:40.954768+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0050135199999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":126,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-17T07:07:18.550770+0000","last_change":"2024-09-17T07:07:18.550881+0000","last_active":"2024-09-17T07:07:18.550770+0000","last_peered":"2024-09-17T07:07:18.550770+0000","last_clean":"2024-09-17T07:07:18.550770+0000","last_became_active":"2024-09-17T07:02:12.007913+0000","last_became_peered":"2024-09-17T07:02:12.007913+0000","last_unstale":"2024-09-17T07:07:18.550770+0000","last_undegraded":"2024-09-17T07:07:18.550770+0000","last_fullsized":"2024-09-17T07:07:18.550770+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:18.550750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:07:18.550750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:47.637615+0000","scrub_duration":0.012153809,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"75'8","reported_seq":181,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-17T07:07:17.183700+0000","last_change":"2024-09-17T07:04:33.544461+0000","last_active":"2024-09-17T07:07:17.183700+0000","last_peered":"2024-09-17T07:07:17.183700+0000","last_clean":"2024-09-17T07:07:17.183700+0000","last_became_active":"2024-09-17T07:02:16.021762+0000","last_became_peered":"2024-09-17T07:02:16.021762+0000","last_unstale":"2024-09-17T07:07:17.183700+0000","last_undegraded":"2024-09-17T07:07:17.183700+0000","last_fullsized":"2024-09-17T07:07:17.183700+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:04:33.544306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:04:33.544306+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0038501080000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":119,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.192100+0000","last_change":"2024-09-17T07:06:33.833965+0000","last_active":"2024-09-17T07:07:14.192100+0000","last_peered":"2024-09-17T07:07:14.192100+0000","last_clean":"2024-09-17T07:07:14.192100+0000","last_became_active":"2024-09-17T07:02:18.025332+0000","last_became_peered":"2024-09-17T07:02:18.025332+0000","last_unstale":"2024-09-17T07:07:14.192100+0000","last_undegraded":"2024-09-17T07:07:14.192100+0000","last_fullsized":"2024-09-17T07:07:14.192100+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:33.833851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:06:33.833851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:37.216443+0000","scrub_duration":0.064545922000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"160'12879","reported_seq":13068,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.074957+0000","last_change":"2024-09-17T07:06:26.698678+0000","last_active":"2024-09-17T07:07:23.074957+0000","last_peered":"2024-09-17T07:07:23.074957+0000","last_clean":"2024-09-17T07:07:23.074957+0000","last_became_active":"2024-09-17T07:04:20.805282+0000","last_became_peered":"2024-09-17T07:04:20.805282+0000","last_unstale":"2024-09-17T07:07:23.074957+0000","last_undegraded":"2024-09-17T07:07:23.074957+0000","last_fullsized":"2024-09-17T07:07:23.074957+0000","mapping_epoch":99,"log_start":"140'7200","ondisk_log_start":"140'7200","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"148'8698","last_scrub_stamp":"2024-09-17T07:06:26.698194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_clean_scrub_stamp":"2024-09-17T07:06:26.698194+0000","objects_scrubbed":5238,"log_size":5679,"log_dups_size":0,"ondisk_log_size":5679,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:48.672416+0000","scrub_duration":1.569268136,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65011736,"num_objects":993,"num_object_clones":0,"num_object_copies":2979,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":993,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12879,"num_write_kb":443841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"75'11","reported_seq":143,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.198783+0000","last_change":"2024-09-17T07:06:29.766307+0000","last_active":"2024-09-17T07:07:14.198783+0000","last_peered":"2024-09-17T07:07:14.198783+0000","last_clean":"2024-09-17T07:07:14.198783+0000","last_became_active":"2024-09-17T07:02:14.010227+0000","last_became_peered":"2024-09-17T07:02:14.010227+0000","last_unstale":"2024-09-17T07:07:14.198783+0000","last_undegraded":"2024-09-17T07:07:14.198783+0000","last_fullsized":"2024-09-17T07:07:14.198783+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-09-17T07:06:29.766173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:06:29.766173+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:45.457800+0000","scrub_duration":0.0047797760000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":126,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.192915+0000","last_change":"2024-09-17T07:04:44.722990+0000","last_active":"2024-09-17T07:07:14.192915+0000","last_peered":"2024-09-17T07:07:14.192915+0000","last_clean":"2024-09-17T07:07:14.192915+0000","last_became_active":"2024-09-17T07:02:12.013182+0000","last_became_peered":"2024-09-17T07:02:12.013182+0000","last_unstale":"2024-09-17T07:07:14.192915+0000","last_undegraded":"2024-09-17T07:07:14.192915+0000","last_fullsized":"2024-09-17T07:07:14.192915+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:44.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:04:44.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026545789999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":132,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.680573+0000","last_change":"2024-09-17T07:07:14.680573+0000","last_active":"2024-09-17T07:07:14.680573+0000","last_peered":"2024-09-17T07:07:14.680573+0000","last_clean":"2024-09-17T07:07:14.680573+0000","last_became_active":"2024-09-17T07:02:16.020224+0000","last_became_peered":"2024-09-17T07:02:16.020224+0000","last_unstale":"2024-09-17T07:07:14.680573+0000","last_undegraded":"2024-09-17T07:07:14.680573+0000","last_fullsized":"2024-09-17T07:07:14.680573+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:14.680446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:07:14.680446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:32.551262+0000","scrub_duration":0.0023342229999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":119,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.178871+0000","last_change":"2024-09-17T07:06:30.971537+0000","last_active":"2024-09-17T07:07:14.178871+0000","last_peered":"2024-09-17T07:07:14.178871+0000","last_clean":"2024-09-17T07:07:14.178871+0000","last_became_active":"2024-09-17T07:02:18.365000+0000","last_became_peered":"2024-09-17T07:02:18.365000+0000","last_unstale":"2024-09-17T07:07:14.178871+0000","last_undegraded":"2024-09-17T07:07:14.178871+0000","last_fullsized":"2024-09-17T07:07:14.178871+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:30.971383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:06:30.971383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:32.519530+0000","scrub_duration":0.0078901470000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"160'7855","reported_seq":7935,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-17T07:07:18.571637+0000","last_change":"2024-09-17T07:04:58.371024+0000","last_active":"2024-09-17T07:07:18.571637+0000","last_peered":"2024-09-17T07:07:18.571637+0000","last_clean":"2024-09-17T07:07:18.571637+0000","last_became_active":"2024-09-17T07:04:58.369102+0000","last_became_peered":"2024-09-17T07:04:58.369102+0000","last_unstale":"2024-09-17T07:07:18.571637+0000","last_undegraded":"2024-09-17T07:07:18.571637+0000","last_fullsized":"2024-09-17T07:07:18.571637+0000","mapping_epoch":99,"log_start":"110'3363","ondisk_log_start":"110'3363","created":99,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_clean_scrub_stamp":"2024-09-17T07:04:19.785747+0000","objects_scrubbed":0,"log_size":4492,"log_dups_size":0,"ondisk_log_size":4492,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47185920,"num_objects":720,"num_object_clones":0,"num_object_copies":2160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6164,"num_write_kb":220288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"75'14","reported_seq":134,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.175456+0000","last_change":"2024-09-17T07:04:38.693158+0000","last_active":"2024-09-17T07:07:14.175456+0000","last_peered":"2024-09-17T07:07:14.175456+0000","last_clean":"2024-09-17T07:07:14.175456+0000","last_became_active":"2024-09-17T07:02:14.007810+0000","last_became_peered":"2024-09-17T07:02:14.007810+0000","last_unstale":"2024-09-17T07:07:14.175456+0000","last_undegraded":"2024-09-17T07:07:14.175456+0000","last_fullsized":"2024-09-17T07:07:14.175456+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'14","last_scrub_stamp":"2024-09-17T07:04:38.692905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:04:38.692905+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003339,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":136,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.204767+0000","last_change":"2024-09-17T07:06:38.303168+0000","last_active":"2024-09-17T07:07:14.204767+0000","last_peered":"2024-09-17T07:07:14.204767+0000","last_clean":"2024-09-17T07:07:14.204767+0000","last_became_active":"2024-09-17T07:02:12.011766+0000","last_became_peered":"2024-09-17T07:02:12.011766+0000","last_unstale":"2024-09-17T07:07:14.204767+0000","last_undegraded":"2024-09-17T07:07:14.204767+0000","last_fullsized":"2024-09-17T07:07:14.204767+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:38.303033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:06:38.303033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:02.394179+0000","scrub_duration":0.003556221,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"75'8","reported_seq":190,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:15.576703+0000","last_change":"2024-09-17T07:06:29.982260+0000","last_active":"2024-09-17T07:07:15.576703+0000","last_peered":"2024-09-17T07:07:15.576703+0000","last_clean":"2024-09-17T07:07:15.576703+0000","last_became_active":"2024-09-17T07:02:16.020628+0000","last_became_peered":"2024-09-17T07:02:16.020628+0000","last_unstale":"2024-09-17T07:07:15.576703+0000","last_undegraded":"2024-09-17T07:07:15.576703+0000","last_fullsized":"2024-09-17T07:07:15.576703+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:06:29.982141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:06:29.982141+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:53.993523+0000","scrub_duration":0.0052922769999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":120,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.192468+0000","last_change":"2024-09-17T07:04:54.840916+0000","last_active":"2024-09-17T07:07:14.192468+0000","last_peered":"2024-09-17T07:07:14.192468+0000","last_clean":"2024-09-17T07:07:14.192468+0000","last_became_active":"2024-09-17T07:02:18.364981+0000","last_became_peered":"2024-09-17T07:02:18.364981+0000","last_unstale":"2024-09-17T07:07:14.192468+0000","last_undegraded":"2024-09-17T07:07:14.192468+0000","last_fullsized":"2024-09-17T07:07:14.192468+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:54.840843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:04:54.840843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002308619,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"160'8025","reported_seq":8104,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-17T07:07:18.584331+0000","last_change":"2024-09-17T07:04:58.373641+0000","last_active":"2024-09-17T07:07:18.584331+0000","last_peered":"2024-09-17T07:07:18.584331+0000","last_clean":"2024-09-17T07:07:18.584331+0000","last_became_active":"2024-09-17T07:04:58.371716+0000","last_became_peered":"2024-09-17T07:04:58.371716+0000","last_unstale":"2024-09-17T07:07:18.584331+0000","last_undegraded":"2024-09-17T07:07:18.584331+0000","last_fullsized":"2024-09-17T07:07:18.584331+0000","mapping_epoch":99,"log_start":"110'3462","ondisk_log_start":"110'3462","created":99,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_clean_scrub_stamp":"2024-09-17T07:04:19.785747+0000","objects_scrubbed":0,"log_size":4563,"log_dups_size":0,"ondisk_log_size":4563,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46399488,"num_objects":708,"num_object_clones":0,"num_object_copies":2124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":708,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6252,"num_write_kb":222720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"75'18","reported_seq":155,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.179517+0000","last_change":"2024-09-17T07:06:40.794767+0000","last_active":"2024-09-17T07:07:14.179517+0000","last_peered":"2024-09-17T07:07:14.179517+0000","last_clean":"2024-09-17T07:07:14.179517+0000","last_became_active":"2024-09-17T07:02:14.028667+0000","last_became_peered":"2024-09-17T07:02:14.028667+0000","last_unstale":"2024-09-17T07:07:14.179517+0000","last_undegraded":"2024-09-17T07:07:14.179517+0000","last_fullsized":"2024-09-17T07:07:14.179517+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'18","last_scrub_stamp":"2024-09-17T07:06:40.794315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:06:40.794315+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:47.535902+0000","scrub_duration":0.017914407,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":125,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.181587+0000","last_change":"2024-09-17T07:04:45.900865+0000","last_active":"2024-09-17T07:07:14.181587+0000","last_peered":"2024-09-17T07:07:14.181587+0000","last_clean":"2024-09-17T07:07:14.181587+0000","last_became_active":"2024-09-17T07:02:12.005687+0000","last_became_peered":"2024-09-17T07:02:12.005687+0000","last_unstale":"2024-09-17T07:07:14.181587+0000","last_undegraded":"2024-09-17T07:07:14.181587+0000","last_fullsized":"2024-09-17T07:07:14.181587+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:45.900779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:04:45.900779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001545968,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":111,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.175365+0000","last_change":"2024-09-17T07:04:01.335116+0000","last_active":"2024-09-17T07:07:14.175365+0000","last_peered":"2024-09-17T07:07:14.175365+0000","last_clean":"2024-09-17T07:07:14.175365+0000","last_became_active":"2024-09-17T07:02:16.019747+0000","last_became_peered":"2024-09-17T07:02:16.019747+0000","last_unstale":"2024-09-17T07:07:14.175365+0000","last_undegraded":"2024-09-17T07:07:14.175365+0000","last_fullsized":"2024-09-17T07:07:14.175365+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:01.335039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:04:01.335039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001538607,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":120,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.196902+0000","last_change":"2024-09-17T07:06:40.684796+0000","last_active":"2024-09-17T07:07:14.196902+0000","last_peered":"2024-09-17T07:07:14.196902+0000","last_clean":"2024-09-17T07:07:14.196902+0000","last_became_active":"2024-09-17T07:02:18.024043+0000","last_became_peered":"2024-09-17T07:02:18.024043+0000","last_unstale":"2024-09-17T07:07:14.196902+0000","last_undegraded":"2024-09-17T07:07:14.196902+0000","last_fullsized":"2024-09-17T07:07:14.196902+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:40.684680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:06:40.684680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:41.222854+0000","scrub_duration":0.011253449,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"160'8177","reported_seq":8303,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-17T07:07:22.985274+0000","last_change":"2024-09-17T07:04:59.586389+0000","last_active":"2024-09-17T07:07:22.985274+0000","last_peered":"2024-09-17T07:07:22.985274+0000","last_clean":"2024-09-17T07:07:22.985274+0000","last_became_active":"2024-09-17T07:04:58.352364+0000","last_became_peered":"2024-09-17T07:04:58.352364+0000","last_unstale":"2024-09-17T07:07:22.985274+0000","last_undegraded":"2024-09-17T07:07:22.985274+0000","last_fullsized":"2024-09-17T07:07:22.985274+0000","mapping_epoch":99,"log_start":"105'2091","ondisk_log_start":"105'2091","created":99,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'3413","last_scrub_stamp":"2024-09-17T07:04:59.586257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_clean_scrub_stamp":"2024-09-17T07:04:59.586257+0000","objects_scrubbed":1771,"log_size":6086,"log_dups_size":0,"ondisk_log_size":6086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.169904692,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30605312,"num_objects":467,"num_object_clones":0,"num_object_copies":1401,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":467,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4764,"num_write_kb":110720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"75'19","reported_seq":160,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-17T07:07:17.989461+0000","last_change":"2024-09-17T07:07:17.989796+0000","last_active":"2024-09-17T07:07:17.989461+0000","last_peered":"2024-09-17T07:07:17.989461+0000","last_clean":"2024-09-17T07:07:17.989461+0000","last_became_active":"2024-09-17T07:02:14.027389+0000","last_became_peered":"2024-09-17T07:02:14.027389+0000","last_unstale":"2024-09-17T07:07:17.989461+0000","last_undegraded":"2024-09-17T07:07:17.989461+0000","last_fullsized":"2024-09-17T07:07:17.989461+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-09-17T07:07:17.989435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:07:17.989435+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:38.663377+0000","scrub_duration":0.0043356940000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":126,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.192972+0000","last_change":"2024-09-17T07:06:31.799761+0000","last_active":"2024-09-17T07:07:14.192972+0000","last_peered":"2024-09-17T07:07:14.192972+0000","last_clean":"2024-09-17T07:07:14.192972+0000","last_became_active":"2024-09-17T07:02:12.011917+0000","last_became_peered":"2024-09-17T07:02:12.011917+0000","last_unstale":"2024-09-17T07:07:14.192972+0000","last_undegraded":"2024-09-17T07:07:14.192972+0000","last_fullsized":"2024-09-17T07:07:14.192972+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:31.799640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:06:31.799640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:45.018149+0000","scrub_duration":0.00320703,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":111,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.202973+0000","last_change":"2024-09-17T07:06:21.229365+0000","last_active":"2024-09-17T07:07:14.202973+0000","last_peered":"2024-09-17T07:07:14.202973+0000","last_clean":"2024-09-17T07:07:14.202973+0000","last_became_active":"2024-09-17T07:02:16.016199+0000","last_became_peered":"2024-09-17T07:02:16.016199+0000","last_unstale":"2024-09-17T07:07:14.202973+0000","last_undegraded":"2024-09-17T07:07:14.202973+0000","last_fullsized":"2024-09-17T07:07:14.202973+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:49.489690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:49.489690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0016174729999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":119,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.181423+0000","last_change":"2024-09-17T07:06:30.992080+0000","last_active":"2024-09-17T07:07:14.181423+0000","last_peered":"2024-09-17T07:07:14.181423+0000","last_clean":"2024-09-17T07:07:14.181423+0000","last_became_active":"2024-09-17T07:02:18.364835+0000","last_became_peered":"2024-09-17T07:02:18.364835+0000","last_unstale":"2024-09-17T07:07:14.181423+0000","last_undegraded":"2024-09-17T07:07:14.181423+0000","last_fullsized":"2024-09-17T07:07:14.181423+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:30.991972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:06:30.991972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:53.454623+0000","scrub_duration":0.0045593279999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"160'8253","reported_seq":8398,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-17T07:07:22.967905+0000","last_change":"2024-09-17T07:06:24.330407+0000","last_active":"2024-09-17T07:07:22.967905+0000","last_peered":"2024-09-17T07:07:22.967905+0000","last_clean":"2024-09-17T07:07:22.967905+0000","last_became_active":"2024-09-17T07:04:58.342518+0000","last_became_peered":"2024-09-17T07:04:58.342518+0000","last_unstale":"2024-09-17T07:07:22.967905+0000","last_undegraded":"2024-09-17T07:07:22.967905+0000","last_fullsized":"2024-09-17T07:07:22.967905+0000","mapping_epoch":99,"log_start":"107'2684","ondisk_log_start":"107'2684","created":99,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'6092","last_scrub_stamp":"2024-09-17T07:06:24.330096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_clean_scrub_stamp":"2024-09-17T07:06:24.330096+0000","objects_scrubbed":2699,"log_size":5569,"log_dups_size":0,"ondisk_log_size":5569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:39.056718+0000","scrub_duration":0.54396754599999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34340864,"num_objects":524,"num_object_clones":0,"num_object_copies":1572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2175,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"110'15","reported_seq":143,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.182091+0000","last_change":"2024-09-17T07:04:57.092268+0000","last_active":"2024-09-17T07:07:14.182091+0000","last_peered":"2024-09-17T07:07:14.182091+0000","last_clean":"2024-09-17T07:07:14.182091+0000","last_became_active":"2024-09-17T07:02:14.010637+0000","last_became_peered":"2024-09-17T07:02:14.010637+0000","last_unstale":"2024-09-17T07:07:14.182091+0000","last_undegraded":"2024-09-17T07:07:14.182091+0000","last_fullsized":"2024-09-17T07:07:14.182091+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'15","last_scrub_stamp":"2024-09-17T07:04:57.091400+0000","last_deep_scrub":"110'15","last_deep_scrub_stamp":"2024-09-17T07:04:57.091400+0000","last_clean_scrub_stamp":"2024-09-17T07:04:57.091400+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012740888000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":125,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.182164+0000","last_change":"2024-09-17T07:04:55.007489+0000","last_active":"2024-09-17T07:07:14.182164+0000","last_peered":"2024-09-17T07:07:14.182164+0000","last_clean":"2024-09-17T07:07:14.182164+0000","last_became_active":"2024-09-17T07:02:12.010803+0000","last_became_peered":"2024-09-17T07:02:12.010803+0000","last_unstale":"2024-09-17T07:07:14.182164+0000","last_undegraded":"2024-09-17T07:07:14.182164+0000","last_fullsized":"2024-09-17T07:07:14.182164+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:55.007317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:25.147878+0000","last_clean_scrub_stamp":"2024-09-17T07:04:55.007317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0085334680000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"89'9","reported_seq":192,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:15.577433+0000","last_change":"2024-09-17T07:04:48.452677+0000","last_active":"2024-09-17T07:07:15.577433+0000","last_peered":"2024-09-17T07:07:15.577433+0000","last_clean":"2024-09-17T07:07:15.577433+0000","last_became_active":"2024-09-17T07:02:16.021360+0000","last_became_peered":"2024-09-17T07:02:16.021360+0000","last_unstale":"2024-09-17T07:07:15.577433+0000","last_undegraded":"2024-09-17T07:07:15.577433+0000","last_fullsized":"2024-09-17T07:07:15.577433+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'9","last_scrub_stamp":"2024-09-17T07:04:48.452558+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-09-17T07:03:29.961439+0000","last_clean_scrub_stamp":"2024-09-17T07:04:48.452558+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0057192279999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"75'1","reported_seq":121,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.172474+0000","last_change":"2024-09-17T07:06:25.992349+0000","last_active":"2024-09-17T07:07:14.172474+0000","last_peered":"2024-09-17T07:07:14.172474+0000","last_clean":"2024-09-17T07:07:14.172474+0000","last_became_active":"2024-09-17T07:02:18.029772+0000","last_became_peered":"2024-09-17T07:02:18.029772+0000","last_unstale":"2024-09-17T07:07:14.172474+0000","last_undegraded":"2024-09-17T07:07:14.172474+0000","last_fullsized":"2024-09-17T07:07:14.172474+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-17T07:06:25.992208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:06:25.992208+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:41.167851+0000","scrub_duration":0.010034507,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"160'8208","reported_seq":8381,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.085112+0000","last_change":"2024-09-17T07:06:46.756075+0000","last_active":"2024-09-17T07:07:23.085112+0000","last_peered":"2024-09-17T07:07:23.085112+0000","last_clean":"2024-09-17T07:07:23.085112+0000","last_became_active":"2024-09-17T07:04:58.356122+0000","last_became_peered":"2024-09-17T07:04:58.356122+0000","last_unstale":"2024-09-17T07:07:23.085112+0000","last_undegraded":"2024-09-17T07:07:23.085112+0000","last_fullsized":"2024-09-17T07:07:23.085112+0000","mapping_epoch":99,"log_start":"105'1612","ondisk_log_start":"105'1612","created":99,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'6802","last_scrub_stamp":"2024-09-17T07:06:46.755822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_clean_scrub_stamp":"2024-09-17T07:06:46.755822+0000","objects_scrubbed":1892,"log_size":6596,"log_dups_size":0,"ondisk_log_size":6596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:58.085658+0000","scrub_duration":0.46200559400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31195136,"num_objects":476,"num_object_clones":0,"num_object_copies":1428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":476,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4717,"num_write_kb":110656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"75'28","reported_seq":233,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-17T07:07:17.191410+0000","last_change":"2024-09-17T07:06:45.186860+0000","last_active":"2024-09-17T07:07:17.191410+0000","last_peered":"2024-09-17T07:07:17.191410+0000","last_clean":"2024-09-17T07:07:17.191410+0000","last_became_active":"2024-09-17T07:02:14.017377+0000","last_became_peered":"2024-09-17T07:02:14.017377+0000","last_unstale":"2024-09-17T07:07:17.191410+0000","last_undegraded":"2024-09-17T07:07:17.191410+0000","last_fullsized":"2024-09-17T07:07:17.191410+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'28","last_scrub_stamp":"2024-09-17T07:06:45.186408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:06:45.186408+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:03.935330+0000","scrub_duration":0.010603473,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":136,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.191938+0000","last_change":"2024-09-17T07:06:43.763211+0000","last_active":"2024-09-17T07:07:14.191938+0000","last_peered":"2024-09-17T07:07:14.191938+0000","last_clean":"2024-09-17T07:07:14.191938+0000","last_became_active":"2024-09-17T07:02:12.008802+0000","last_became_peered":"2024-09-17T07:02:12.008802+0000","last_unstale":"2024-09-17T07:07:14.191938+0000","last_undegraded":"2024-09-17T07:07:14.191938+0000","last_fullsized":"2024-09-17T07:07:14.191938+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:43.763044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:06:43.763044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:04.613175+0000","scrub_duration":0.003861303,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":122,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.173225+0000","last_change":"2024-09-17T07:04:39.385133+0000","last_active":"2024-09-17T07:07:14.173225+0000","last_peered":"2024-09-17T07:07:14.173225+0000","last_clean":"2024-09-17T07:07:14.173225+0000","last_became_active":"2024-09-17T07:02:16.016606+0000","last_became_peered":"2024-09-17T07:02:16.016606+0000","last_unstale":"2024-09-17T07:07:14.173225+0000","last_undegraded":"2024-09-17T07:07:14.173225+0000","last_fullsized":"2024-09-17T07:07:14.173225+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:39.384962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:04:39.384962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0055010839999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":108,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.178668+0000","last_change":"2024-09-17T07:04:04.564851+0000","last_active":"2024-09-17T07:07:14.178668+0000","last_peered":"2024-09-17T07:07:14.178668+0000","last_clean":"2024-09-17T07:07:14.178668+0000","last_became_active":"2024-09-17T07:02:18.024602+0000","last_became_peered":"2024-09-17T07:02:18.024602+0000","last_unstale":"2024-09-17T07:07:14.178668+0000","last_undegraded":"2024-09-17T07:07:14.178668+0000","last_fullsized":"2024-09-17T07:07:14.178668+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:04.564765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:04:04.564765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001317206,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"160'8089","reported_seq":8197,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-17T07:07:18.851137+0000","last_change":"2024-09-17T07:07:16.248588+0000","last_active":"2024-09-17T07:07:18.851137+0000","last_peered":"2024-09-17T07:07:18.851137+0000","last_clean":"2024-09-17T07:07:18.851137+0000","last_became_active":"2024-09-17T07:04:58.345101+0000","last_became_peered":"2024-09-17T07:04:58.345101+0000","last_unstale":"2024-09-17T07:07:18.851137+0000","last_undegraded":"2024-09-17T07:07:18.851137+0000","last_fullsized":"2024-09-17T07:07:18.851137+0000","mapping_epoch":99,"log_start":"108'3059","ondisk_log_start":"108'3059","created":99,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"159'7976","last_scrub_stamp":"2024-09-17T07:07:16.248093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_clean_scrub_stamp":"2024-09-17T07:07:16.248093+0000","objects_scrubbed":800,"log_size":5030,"log_dups_size":0,"ondisk_log_size":5030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:45.018010+0000","scrub_duration":0.257620882,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44761088,"num_objects":683,"num_object_clones":0,"num_object_copies":2049,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"75'13","reported_seq":150,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.181900+0000","last_change":"2024-09-17T07:04:58.081870+0000","last_active":"2024-09-17T07:07:14.181900+0000","last_peered":"2024-09-17T07:07:14.181900+0000","last_clean":"2024-09-17T07:07:14.181900+0000","last_became_active":"2024-09-17T07:02:14.019838+0000","last_became_peered":"2024-09-17T07:02:14.019838+0000","last_unstale":"2024-09-17T07:07:14.181900+0000","last_undegraded":"2024-09-17T07:07:14.181900+0000","last_fullsized":"2024-09-17T07:07:14.181900+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'13","last_scrub_stamp":"2024-09-17T07:04:58.081771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:04:58.081771+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001806263,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"76'5","reported_seq":437,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-17T07:07:21.547460+0000","last_change":"2024-09-17T07:06:46.230250+0000","last_active":"2024-09-17T07:07:21.547460+0000","last_peered":"2024-09-17T07:07:21.547460+0000","last_clean":"2024-09-17T07:07:21.547460+0000","last_became_active":"2024-09-17T07:02:12.006883+0000","last_became_peered":"2024-09-17T07:02:12.006883+0000","last_unstale":"2024-09-17T07:07:21.547460+0000","last_undegraded":"2024-09-17T07:07:21.547460+0000","last_fullsized":"2024-09-17T07:07:21.547460+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'5","last_scrub_stamp":"2024-09-17T07:06:46.229804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:06:46.229804+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:06.428152+0000","scrub_duration":0.064119354000000003,"objects_trimmed":0,"snaptrim_duration":0.00033739799999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":304,"num_read_kb":299,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.179059+0000","last_change":"2024-09-17T07:04:53.698772+0000","last_active":"2024-09-17T07:07:14.179059+0000","last_peered":"2024-09-17T07:07:14.179059+0000","last_clean":"2024-09-17T07:07:14.179059+0000","last_became_active":"2024-09-17T07:04:15.109950+0000","last_became_peered":"2024-09-17T07:04:15.109950+0000","last_unstale":"2024-09-17T07:07:14.179059+0000","last_undegraded":"2024-09-17T07:07:14.179059+0000","last_fullsized":"2024-09-17T07:07:14.179059+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:53.698301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:04:53.698301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045574200000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":122,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.171418+0000","last_change":"2024-09-17T07:06:25.185133+0000","last_active":"2024-09-17T07:07:14.171418+0000","last_peered":"2024-09-17T07:07:14.171418+0000","last_clean":"2024-09-17T07:07:14.171418+0000","last_became_active":"2024-09-17T07:02:16.014492+0000","last_became_peered":"2024-09-17T07:02:16.014492+0000","last_unstale":"2024-09-17T07:07:14.171418+0000","last_undegraded":"2024-09-17T07:07:14.171418+0000","last_fullsized":"2024-09-17T07:07:14.171418+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:25.184907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:06:25.184907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:37.815314+0000","scrub_duration":0.0051503060000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":120,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.192529+0000","last_change":"2024-09-17T07:04:53.836533+0000","last_active":"2024-09-17T07:07:14.192529+0000","last_peered":"2024-09-17T07:07:14.192529+0000","last_clean":"2024-09-17T07:07:14.192529+0000","last_became_active":"2024-09-17T07:02:18.365369+0000","last_became_peered":"2024-09-17T07:02:18.365369+0000","last_unstale":"2024-09-17T07:07:14.192529+0000","last_undegraded":"2024-09-17T07:07:14.192529+0000","last_fullsized":"2024-09-17T07:07:14.192529+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:53.836372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:04:53.836372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0024095980000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"160'10197","reported_seq":10450,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.112983+0000","last_change":"2024-09-17T07:06:49.883933+0000","last_active":"2024-09-17T07:07:23.112983+0000","last_peered":"2024-09-17T07:07:23.112983+0000","last_clean":"2024-09-17T07:07:23.112983+0000","last_became_active":"2024-09-17T07:04:58.346352+0000","last_became_peered":"2024-09-17T07:04:58.346352+0000","last_unstale":"2024-09-17T07:07:23.112983+0000","last_undegraded":"2024-09-17T07:07:23.112983+0000","last_fullsized":"2024-09-17T07:07:23.112983+0000","mapping_epoch":99,"log_start":"143'5769","ondisk_log_start":"143'5769","created":99,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'8912","last_scrub_stamp":"2024-09-17T07:06:49.883785+0000","last_deep_scrub":"152'8912","last_deep_scrub_stamp":"2024-09-17T07:06:49.883785+0000","last_clean_scrub_stamp":"2024-09-17T07:06:49.883785+0000","objects_scrubbed":1930,"log_size":4428,"log_dups_size":0,"ondisk_log_size":4428,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":11,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:09.746728+0000","scrub_duration":10.040198303,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31784960,"num_objects":485,"num_object_clones":0,"num_object_copies":1455,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1445,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"75'12","reported_seq":165,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.176484+0000","last_change":"2024-09-17T07:04:50.489946+0000","last_active":"2024-09-17T07:07:14.176484+0000","last_peered":"2024-09-17T07:07:14.176484+0000","last_clean":"2024-09-17T07:07:14.176484+0000","last_became_active":"2024-09-17T07:03:48.760405+0000","last_became_peered":"2024-09-17T07:03:48.760405+0000","last_unstale":"2024-09-17T07:07:14.176484+0000","last_undegraded":"2024-09-17T07:07:14.176484+0000","last_fullsized":"2024-09-17T07:07:14.176484+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-09-17T07:04:50.489723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:04:50.489723+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.007032679,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"70'1","reported_seq":121,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.173889+0000","last_change":"2024-09-17T07:03:52.963332+0000","last_active":"2024-09-17T07:07:14.173889+0000","last_peered":"2024-09-17T07:07:14.173889+0000","last_clean":"2024-09-17T07:07:14.173889+0000","last_became_active":"2024-09-17T07:02:12.009476+0000","last_became_peered":"2024-09-17T07:02:12.009476+0000","last_unstale":"2024-09-17T07:07:14.173889+0000","last_undegraded":"2024-09-17T07:07:14.173889+0000","last_fullsized":"2024-09-17T07:07:14.173889+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-17T07:03:52.963249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:52.963249+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015040780000000001,"objects_trimmed":0,"snaptrim_duration":0.0032298629999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":115,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.193060+0000","last_change":"2024-09-17T07:03:51.994858+0000","last_active":"2024-09-17T07:07:14.193060+0000","last_peered":"2024-09-17T07:07:14.193060+0000","last_clean":"2024-09-17T07:07:14.193060+0000","last_became_active":"2024-09-17T07:02:12.014623+0000","last_became_peered":"2024-09-17T07:02:12.014623+0000","last_unstale":"2024-09-17T07:07:14.193060+0000","last_undegraded":"2024-09-17T07:07:14.193060+0000","last_fullsized":"2024-09-17T07:07:14.193060+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:51.994756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:51.994756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001933269,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":121,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.192623+0000","last_change":"2024-09-17T07:06:28.863501+0000","last_active":"2024-09-17T07:07:14.192623+0000","last_peered":"2024-09-17T07:07:14.192623+0000","last_clean":"2024-09-17T07:07:14.192623+0000","last_became_active":"2024-09-17T07:02:16.021561+0000","last_became_peered":"2024-09-17T07:02:16.021561+0000","last_unstale":"2024-09-17T07:07:14.192623+0000","last_undegraded":"2024-09-17T07:07:14.192623+0000","last_fullsized":"2024-09-17T07:07:14.192623+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:28.863357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:06:28.863357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:40.252322+0000","scrub_duration":0.014593309,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":108,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.181801+0000","last_change":"2024-09-17T07:04:03.980001+0000","last_active":"2024-09-17T07:07:14.181801+0000","last_peered":"2024-09-17T07:07:14.181801+0000","last_clean":"2024-09-17T07:07:14.181801+0000","last_became_active":"2024-09-17T07:02:18.022082+0000","last_became_peered":"2024-09-17T07:02:18.022082+0000","last_unstale":"2024-09-17T07:07:14.181801+0000","last_undegraded":"2024-09-17T07:07:14.181801+0000","last_fullsized":"2024-09-17T07:07:14.181801+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:03.979895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:04:03.979895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001298542,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"160'7781","reported_seq":7911,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-17T07:07:18.827142+0000","last_change":"2024-09-17T07:06:46.844354+0000","last_active":"2024-09-17T07:07:18.827142+0000","last_peered":"2024-09-17T07:07:18.827142+0000","last_clean":"2024-09-17T07:07:18.827142+0000","last_became_active":"2024-09-17T07:04:58.361207+0000","last_became_peered":"2024-09-17T07:04:58.361207+0000","last_unstale":"2024-09-17T07:07:18.827142+0000","last_undegraded":"2024-09-17T07:07:18.827142+0000","last_fullsized":"2024-09-17T07:07:18.827142+0000","mapping_epoch":99,"log_start":"107'2510","ondisk_log_start":"107'2510","created":99,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'6605","last_scrub_stamp":"2024-09-17T07:06:46.844191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_clean_scrub_stamp":"2024-09-17T07:06:46.844191+0000","objects_scrubbed":1865,"log_size":5271,"log_dups_size":0,"ondisk_log_size":5271,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:01.893755+0000","scrub_duration":0.59670863900000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44498944,"num_objects":679,"num_object_clones":0,"num_object_copies":2037,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1186,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"75'14","reported_seq":135,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.204335+0000","last_change":"2024-09-17T07:06:21.228996+0000","last_active":"2024-09-17T07:07:14.204335+0000","last_peered":"2024-09-17T07:07:14.204335+0000","last_clean":"2024-09-17T07:07:14.204335+0000","last_became_active":"2024-09-17T07:02:14.028857+0000","last_became_peered":"2024-09-17T07:02:14.028857+0000","last_unstale":"2024-09-17T07:07:14.204335+0000","last_undegraded":"2024-09-17T07:07:14.204335+0000","last_fullsized":"2024-09-17T07:07:14.204335+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'14","last_scrub_stamp":"2024-09-17T07:03:44.522631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:44.522631+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019970970000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"72'2","reported_seq":132,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.179149+0000","last_change":"2024-09-17T07:04:35.691710+0000","last_active":"2024-09-17T07:07:14.179149+0000","last_peered":"2024-09-17T07:07:14.179149+0000","last_clean":"2024-09-17T07:07:14.179149+0000","last_became_active":"2024-09-17T07:02:12.005127+0000","last_became_peered":"2024-09-17T07:02:12.005127+0000","last_unstale":"2024-09-17T07:07:14.179149+0000","last_undegraded":"2024-09-17T07:07:14.179149+0000","last_fullsized":"2024-09-17T07:07:14.179149+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'2","last_scrub_stamp":"2024-09-17T07:04:35.690341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:04:35.690341+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0079935560000000006,"objects_trimmed":0,"snaptrim_duration":0.0010980219999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":136,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.191935+0000","last_change":"2024-09-17T07:06:37.732110+0000","last_active":"2024-09-17T07:07:14.191935+0000","last_peered":"2024-09-17T07:07:14.191935+0000","last_clean":"2024-09-17T07:07:14.191935+0000","last_became_active":"2024-09-17T07:02:12.004606+0000","last_became_peered":"2024-09-17T07:02:12.004606+0000","last_unstale":"2024-09-17T07:07:14.191935+0000","last_undegraded":"2024-09-17T07:07:14.191935+0000","last_fullsized":"2024-09-17T07:07:14.191935+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:37.731844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:06:37.731844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:04.459809+0000","scrub_duration":0.0070683819999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"78'128","reported_seq":374,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.193217+0000","last_change":"2024-09-17T07:04:37.752152+0000","last_active":"2024-09-17T07:07:14.193217+0000","last_peered":"2024-09-17T07:07:14.193217+0000","last_clean":"2024-09-17T07:07:14.193217+0000","last_became_active":"2024-09-17T07:02:03.020675+0000","last_became_peered":"2024-09-17T07:02:03.020675+0000","last_unstale":"2024-09-17T07:07:14.193217+0000","last_undegraded":"2024-09-17T07:07:14.193217+0000","last_fullsized":"2024-09-17T07:07:14.193217+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'128","last_scrub_stamp":"2024-09-17T07:04:37.751949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:00:09.390381+0000","last_clean_scrub_stamp":"2024-09-17T07:04:37.751949+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040036619999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":122,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.192144+0000","last_change":"2024-09-17T07:04:43.722620+0000","last_active":"2024-09-17T07:07:14.192144+0000","last_peered":"2024-09-17T07:07:14.192144+0000","last_clean":"2024-09-17T07:07:14.192144+0000","last_became_active":"2024-09-17T07:02:16.016031+0000","last_became_peered":"2024-09-17T07:02:16.016031+0000","last_unstale":"2024-09-17T07:07:14.192144+0000","last_undegraded":"2024-09-17T07:07:14.192144+0000","last_fullsized":"2024-09-17T07:07:14.192144+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:43.722510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:04:43.722510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0024960820000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":119,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.192814+0000","last_change":"2024-09-17T07:06:30.808658+0000","last_active":"2024-09-17T07:07:14.192814+0000","last_peered":"2024-09-17T07:07:14.192814+0000","last_clean":"2024-09-17T07:07:14.192814+0000","last_became_active":"2024-09-17T07:02:18.023321+0000","last_became_peered":"2024-09-17T07:02:18.023321+0000","last_unstale":"2024-09-17T07:07:14.192814+0000","last_undegraded":"2024-09-17T07:07:14.192814+0000","last_fullsized":"2024-09-17T07:07:14.192814+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:30.808562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:58.670571+0000","last_clean_scrub_stamp":"2024-09-17T07:06:30.808562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:55.726519+0000","scrub_duration":0.005261338,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.9","version":"160'8082","reported_seq":8219,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-17T07:07:18.940232+0000","last_change":"2024-09-17T07:06:38.907019+0000","last_active":"2024-09-17T07:07:18.940232+0000","last_peered":"2024-09-17T07:07:18.940232+0000","last_clean":"2024-09-17T07:07:18.940232+0000","last_became_active":"2024-09-17T07:04:58.345129+0000","last_became_peered":"2024-09-17T07:04:58.345129+0000","last_unstale":"2024-09-17T07:07:18.940232+0000","last_undegraded":"2024-09-17T07:07:18.940232+0000","last_fullsized":"2024-09-17T07:07:18.940232+0000","mapping_epoch":99,"log_start":"109'3146","ondisk_log_start":"109'3146","created":99,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'6535","last_scrub_stamp":"2024-09-17T07:06:38.906726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_clean_scrub_stamp":"2024-09-17T07:06:38.906726+0000","objects_scrubbed":2255,"log_size":4936,"log_dups_size":0,"ondisk_log_size":4936,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:51.910446+0000","scrub_duration":0.73214807299999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45875200,"num_objects":700,"num_object_clones":0,"num_object_copies":2100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":700,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1555,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"75'19","reported_seq":155,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-17T07:07:18.383839+0000","last_change":"2024-09-17T07:07:18.384022+0000","last_active":"2024-09-17T07:07:18.383839+0000","last_peered":"2024-09-17T07:07:18.383839+0000","last_clean":"2024-09-17T07:07:18.383839+0000","last_became_active":"2024-09-17T07:02:14.027313+0000","last_became_peered":"2024-09-17T07:02:14.027313+0000","last_unstale":"2024-09-17T07:07:18.383839+0000","last_undegraded":"2024-09-17T07:07:18.383839+0000","last_fullsized":"2024-09-17T07:07:18.383839+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-09-17T07:07:18.383813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:07:18.383813+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:42.977846+0000","scrub_duration":0.0040351470000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":137,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.169141+0000","last_change":"2024-09-17T07:06:39.322648+0000","last_active":"2024-09-17T07:07:14.169141+0000","last_peered":"2024-09-17T07:07:14.169141+0000","last_clean":"2024-09-17T07:07:14.169141+0000","last_became_active":"2024-09-17T07:02:12.012037+0000","last_became_peered":"2024-09-17T07:02:12.012037+0000","last_unstale":"2024-09-17T07:07:14.169141+0000","last_undegraded":"2024-09-17T07:07:14.169141+0000","last_fullsized":"2024-09-17T07:07:14.169141+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:39.322535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:06:39.322535+0000","last_clean_scrub_stamp":"2024-09-17T07:06:39.322535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:46.785051+0000","scrub_duration":0.002360993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":133,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.171146+0000","last_change":"2024-09-17T07:06:28.171453+0000","last_active":"2024-09-17T07:07:14.171146+0000","last_peered":"2024-09-17T07:07:14.171146+0000","last_clean":"2024-09-17T07:07:14.171146+0000","last_became_active":"2024-09-17T07:02:16.014631+0000","last_became_peered":"2024-09-17T07:02:16.014631+0000","last_unstale":"2024-09-17T07:07:14.171146+0000","last_undegraded":"2024-09-17T07:07:14.171146+0000","last_fullsized":"2024-09-17T07:07:14.171146+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:28.170477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:06:28.170477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:50.964072+0000","scrub_duration":0.0059381709999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":109,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.193137+0000","last_change":"2024-09-17T07:03:58.029849+0000","last_active":"2024-09-17T07:07:14.193137+0000","last_peered":"2024-09-17T07:07:14.193137+0000","last_clean":"2024-09-17T07:07:14.193137+0000","last_became_active":"2024-09-17T07:02:18.023299+0000","last_became_peered":"2024-09-17T07:02:18.023299+0000","last_unstale":"2024-09-17T07:07:14.193137+0000","last_undegraded":"2024-09-17T07:07:14.193137+0000","last_fullsized":"2024-09-17T07:07:14.193137+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:58.029561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:03:58.029561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0020006690000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.8","version":"160'10181","reported_seq":10434,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.215501+0000","last_change":"2024-09-17T07:06:46.778697+0000","last_active":"2024-09-17T07:07:23.215501+0000","last_peered":"2024-09-17T07:07:23.215501+0000","last_clean":"2024-09-17T07:07:23.215501+0000","last_became_active":"2024-09-17T07:04:58.343731+0000","last_became_peered":"2024-09-17T07:04:58.343731+0000","last_unstale":"2024-09-17T07:07:23.215501+0000","last_undegraded":"2024-09-17T07:07:23.215501+0000","last_fullsized":"2024-09-17T07:07:23.215501+0000","mapping_epoch":99,"log_start":"144'5762","ondisk_log_start":"144'5762","created":99,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'8793","last_scrub_stamp":"2024-09-17T07:06:46.778519+0000","last_deep_scrub":"151'8793","last_deep_scrub_stamp":"2024-09-17T07:06:46.778519+0000","last_clean_scrub_stamp":"2024-09-17T07:06:46.778519+0000","objects_scrubbed":1992,"log_size":4419,"log_dups_size":0,"ondisk_log_size":4419,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":9,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:06.324620+0000","scrub_duration":8.9076282960000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30539776,"num_objects":466,"num_object_clones":0,"num_object_copies":1398,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1526,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"75'9","reported_seq":140,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.172339+0000","last_change":"2024-09-17T07:06:26.996048+0000","last_active":"2024-09-17T07:07:14.172339+0000","last_peered":"2024-09-17T07:07:14.172339+0000","last_clean":"2024-09-17T07:07:14.172339+0000","last_became_active":"2024-09-17T07:02:14.014939+0000","last_became_peered":"2024-09-17T07:02:14.014939+0000","last_unstale":"2024-09-17T07:07:14.172339+0000","last_undegraded":"2024-09-17T07:07:14.172339+0000","last_fullsized":"2024-09-17T07:07:14.172339+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:06:26.995923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:06:26.995923+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:49.741971+0000","scrub_duration":0.01417006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":125,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.182713+0000","last_change":"2024-09-17T07:06:30.038396+0000","last_active":"2024-09-17T07:07:14.182713+0000","last_peered":"2024-09-17T07:07:14.182713+0000","last_clean":"2024-09-17T07:07:14.182713+0000","last_became_active":"2024-09-17T07:02:12.000005+0000","last_became_peered":"2024-09-17T07:02:12.000005+0000","last_unstale":"2024-09-17T07:07:14.182713+0000","last_undegraded":"2024-09-17T07:07:14.182713+0000","last_fullsized":"2024-09-17T07:07:14.182713+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:30.038104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:06:30.038104+0000","last_clean_scrub_stamp":"2024-09-17T07:06:30.038104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:38.139496+0000","scrub_duration":0.021438763,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":87,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-17T07:07:19.789153+0000","last_change":"2024-09-17T07:07:19.789263+0000","last_active":"2024-09-17T07:07:19.789153+0000","last_peered":"2024-09-17T07:07:19.789153+0000","last_clean":"2024-09-17T07:07:19.789153+0000","last_became_active":"2024-09-17T07:04:15.107886+0000","last_became_peered":"2024-09-17T07:04:15.107886+0000","last_unstale":"2024-09-17T07:07:19.789153+0000","last_undegraded":"2024-09-17T07:07:19.789153+0000","last_fullsized":"2024-09-17T07:07:19.789153+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:19.789134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:07:19.789134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:20.148885+0000","scrub_duration":0.010505654,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":109,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.173817+0000","last_change":"2024-09-17T07:03:58.099938+0000","last_active":"2024-09-17T07:07:14.173817+0000","last_peered":"2024-09-17T07:07:14.173817+0000","last_clean":"2024-09-17T07:07:14.173817+0000","last_became_active":"2024-09-17T07:02:18.026994+0000","last_became_peered":"2024-09-17T07:02:18.026994+0000","last_unstale":"2024-09-17T07:07:14.173817+0000","last_undegraded":"2024-09-17T07:07:14.173817+0000","last_fullsized":"2024-09-17T07:07:14.173817+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:58.099849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:58.099849+0000","last_clean_scrub_stamp":"2024-09-17T07:03:58.099849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021287960000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.b","version":"160'12962","reported_seq":13033,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.130109+0000","last_change":"2024-09-17T07:04:20.803445+0000","last_active":"2024-09-17T07:07:23.130109+0000","last_peered":"2024-09-17T07:07:23.130109+0000","last_clean":"2024-09-17T07:07:23.130109+0000","last_became_active":"2024-09-17T07:04:20.803245+0000","last_became_peered":"2024-09-17T07:04:20.803245+0000","last_unstale":"2024-09-17T07:07:23.130109+0000","last_undegraded":"2024-09-17T07:07:23.130109+0000","last_fullsized":"2024-09-17T07:07:23.130109+0000","mapping_epoch":99,"log_start":"143'8100","ondisk_log_start":"143'8100","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_clean_scrub_stamp":"2024-09-17T07:04:19.785747+0000","objects_scrubbed":0,"log_size":4862,"log_dups_size":0,"ondisk_log_size":4862,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62521344,"num_objects":954,"num_object_clones":0,"num_object_copies":2862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12962,"num_write_kb":445312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"111'22","reported_seq":155,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.191808+0000","last_change":"2024-09-17T07:04:57.500608+0000","last_active":"2024-09-17T07:07:14.191808+0000","last_peered":"2024-09-17T07:07:14.191808+0000","last_clean":"2024-09-17T07:07:14.191808+0000","last_became_active":"2024-09-17T07:02:14.028392+0000","last_became_peered":"2024-09-17T07:02:14.028392+0000","last_unstale":"2024-09-17T07:07:14.191808+0000","last_undegraded":"2024-09-17T07:07:14.191808+0000","last_fullsized":"2024-09-17T07:07:14.191808+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'22","last_scrub_stamp":"2024-09-17T07:04:57.500450+0000","last_deep_scrub":"111'22","last_deep_scrub_stamp":"2024-09-17T07:04:57.500450+0000","last_clean_scrub_stamp":"2024-09-17T07:04:57.500450+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0053988259999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'2","reported_seq":126,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.178911+0000","last_change":"2024-09-17T07:06:39.828818+0000","last_active":"2024-09-17T07:07:14.178911+0000","last_peered":"2024-09-17T07:07:14.178911+0000","last_clean":"2024-09-17T07:07:14.178911+0000","last_became_active":"2024-09-17T07:04:15.110085+0000","last_became_peered":"2024-09-17T07:04:15.110085+0000","last_unstale":"2024-09-17T07:07:14.178911+0000","last_undegraded":"2024-09-17T07:07:14.178911+0000","last_fullsized":"2024-09-17T07:07:14.178911+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-09-17T07:06:39.828638+0000","last_deep_scrub":"89'2","last_deep_scrub_stamp":"2024-09-17T07:03:26.694366+0000","last_clean_scrub_stamp":"2024-09-17T07:06:39.828638+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:09.464498+0000","scrub_duration":0.0068983610000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":420,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"75'8","reported_seq":195,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-17T07:07:17.186740+0000","last_change":"2024-09-17T07:06:25.761229+0000","last_active":"2024-09-17T07:07:17.186740+0000","last_peered":"2024-09-17T07:07:17.186740+0000","last_clean":"2024-09-17T07:07:17.186740+0000","last_became_active":"2024-09-17T07:02:16.015418+0000","last_became_peered":"2024-09-17T07:02:16.015418+0000","last_unstale":"2024-09-17T07:07:17.186740+0000","last_undegraded":"2024-09-17T07:07:17.186740+0000","last_fullsized":"2024-09-17T07:07:17.186740+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:06:25.760922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:06:25.760922+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:34.826717+0000","scrub_duration":0.0073758620000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":130,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-17T07:07:19.786465+0000","last_change":"2024-09-17T07:07:19.786753+0000","last_active":"2024-09-17T07:07:19.786465+0000","last_peered":"2024-09-17T07:07:19.786465+0000","last_clean":"2024-09-17T07:07:19.786465+0000","last_became_active":"2024-09-17T07:02:18.363253+0000","last_became_peered":"2024-09-17T07:02:18.363253+0000","last_unstale":"2024-09-17T07:07:19.786465+0000","last_undegraded":"2024-09-17T07:07:19.786465+0000","last_fullsized":"2024-09-17T07:07:19.786465+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:19.786442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:07:19.786442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:41.487701+0000","scrub_duration":0.011747911999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.a","version":"160'12371","reported_seq":12442,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-17T07:07:18.566910+0000","last_change":"2024-09-17T07:04:20.804570+0000","last_active":"2024-09-17T07:07:18.566910+0000","last_peered":"2024-09-17T07:07:18.566910+0000","last_clean":"2024-09-17T07:07:18.566910+0000","last_became_active":"2024-09-17T07:04:20.804306+0000","last_became_peered":"2024-09-17T07:04:20.804306+0000","last_unstale":"2024-09-17T07:07:18.566910+0000","last_undegraded":"2024-09-17T07:07:18.566910+0000","last_fullsized":"2024-09-17T07:07:18.566910+0000","mapping_epoch":99,"log_start":"143'7800","ondisk_log_start":"143'7800","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_clean_scrub_stamp":"2024-09-17T07:04:19.785747+0000","objects_scrubbed":0,"log_size":4571,"log_dups_size":0,"ondisk_log_size":4571,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88932352,"num_objects":1357,"num_object_clones":0,"num_object_copies":4071,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12371,"num_write_kb":439296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"150'17","reported_seq":152,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.175431+0000","last_change":"2024-09-17T07:06:38.775932+0000","last_active":"2024-09-17T07:07:14.175431+0000","last_peered":"2024-09-17T07:07:14.175431+0000","last_clean":"2024-09-17T07:07:14.175431+0000","last_became_active":"2024-09-17T07:02:14.010219+0000","last_became_peered":"2024-09-17T07:02:14.010219+0000","last_unstale":"2024-09-17T07:07:14.175431+0000","last_undegraded":"2024-09-17T07:07:14.175431+0000","last_fullsized":"2024-09-17T07:07:14.175431+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'17","last_scrub_stamp":"2024-09-17T07:06:38.775788+0000","last_deep_scrub":"150'17","last_deep_scrub_stamp":"2024-09-17T07:06:38.775788+0000","last_clean_scrub_stamp":"2024-09-17T07:06:38.775788+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:54.992016+0000","scrub_duration":0.011503368999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"68'2","reported_seq":134,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-17T07:07:20.764310+0000","last_change":"2024-09-17T07:07:20.764435+0000","last_active":"2024-09-17T07:07:20.764310+0000","last_peered":"2024-09-17T07:07:20.764310+0000","last_clean":"2024-09-17T07:07:20.764310+0000","last_became_active":"2024-09-17T07:02:12.006716+0000","last_became_peered":"2024-09-17T07:02:12.006716+0000","last_unstale":"2024-09-17T07:07:20.764310+0000","last_undegraded":"2024-09-17T07:07:20.764310+0000","last_fullsized":"2024-09-17T07:07:20.764310+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-09-17T07:07:20.764283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:07:20.764283+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:50.410328+0000","scrub_duration":0.0060160040000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":86,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.172555+0000","last_change":"2024-09-17T07:04:55.444916+0000","last_active":"2024-09-17T07:07:14.172555+0000","last_peered":"2024-09-17T07:07:14.172555+0000","last_clean":"2024-09-17T07:07:14.172555+0000","last_became_active":"2024-09-17T07:04:15.111069+0000","last_became_peered":"2024-09-17T07:04:15.111069+0000","last_unstale":"2024-09-17T07:07:14.172555+0000","last_undegraded":"2024-09-17T07:07:14.172555+0000","last_fullsized":"2024-09-17T07:07:14.172555+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:55.444802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:04:55.444802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0079380379999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":108,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.178069+0000","last_change":"2024-09-17T07:04:06.571270+0000","last_active":"2024-09-17T07:07:14.178069+0000","last_peered":"2024-09-17T07:07:14.178069+0000","last_clean":"2024-09-17T07:07:14.178069+0000","last_became_active":"2024-09-17T07:02:18.024983+0000","last_became_peered":"2024-09-17T07:02:18.024983+0000","last_unstale":"2024-09-17T07:07:14.178069+0000","last_undegraded":"2024-09-17T07:07:14.178069+0000","last_fullsized":"2024-09-17T07:07:14.178069+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:06.571184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:04:06.571184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001613319,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.15","version":"160'8057","reported_seq":11643,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.174372+0000","last_change":"2024-09-17T07:05:15.062360+0000","last_active":"2024-09-17T07:07:23.174372+0000","last_peered":"2024-09-17T07:07:23.174372+0000","last_clean":"2024-09-17T07:07:23.174372+0000","last_became_active":"2024-09-17T07:05:15.059992+0000","last_became_peered":"2024-09-17T07:05:15.059992+0000","last_unstale":"2024-09-17T07:07:23.174372+0000","last_undegraded":"2024-09-17T07:07:23.174372+0000","last_fullsized":"2024-09-17T07:07:23.174372+0000","mapping_epoch":119,"log_start":"118'3645","ondisk_log_start":"118'3645","created":111,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_clean_scrub_stamp":"2024-09-17T07:04:19.785747+0000","objects_scrubbed":0,"log_size":4412,"log_dups_size":0,"ondisk_log_size":4412,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32538624,"num_objects":497,"num_object_clones":0,"num_object_copies":1491,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6323,"num_write_kb":218208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1745,"num_bytes_recovered":114360320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"75'9","reported_seq":140,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.176416+0000","last_change":"2024-09-17T07:06:30.011545+0000","last_active":"2024-09-17T07:07:14.176416+0000","last_peered":"2024-09-17T07:07:14.176416+0000","last_clean":"2024-09-17T07:07:14.176416+0000","last_became_active":"2024-09-17T07:02:14.029388+0000","last_became_peered":"2024-09-17T07:02:14.029388+0000","last_unstale":"2024-09-17T07:07:14.176416+0000","last_undegraded":"2024-09-17T07:07:14.176416+0000","last_fullsized":"2024-09-17T07:07:14.176416+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:06:30.011337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:06:30.011337+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:41.133399+0000","scrub_duration":0.013862154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":137,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.169087+0000","last_change":"2024-09-17T07:06:40.406566+0000","last_active":"2024-09-17T07:07:14.169087+0000","last_peered":"2024-09-17T07:07:14.169087+0000","last_clean":"2024-09-17T07:07:14.169087+0000","last_became_active":"2024-09-17T07:02:12.011327+0000","last_became_peered":"2024-09-17T07:02:12.011327+0000","last_unstale":"2024-09-17T07:07:14.169087+0000","last_undegraded":"2024-09-17T07:07:14.169087+0000","last_fullsized":"2024-09-17T07:07:14.169087+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:40.406459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:31.753470+0000","last_clean_scrub_stamp":"2024-09-17T07:06:40.406459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:54.638620+0000","scrub_duration":0.03274349,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":121,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.178174+0000","last_change":"2024-09-17T07:04:36.652033+0000","last_active":"2024-09-17T07:07:14.178174+0000","last_peered":"2024-09-17T07:07:14.178174+0000","last_clean":"2024-09-17T07:07:14.178174+0000","last_became_active":"2024-09-17T07:02:16.022584+0000","last_became_peered":"2024-09-17T07:02:16.022584+0000","last_unstale":"2024-09-17T07:07:14.178174+0000","last_undegraded":"2024-09-17T07:07:14.178174+0000","last_fullsized":"2024-09-17T07:07:14.178174+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:36.651925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:04:36.651925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0061789210000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":131,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.168731+0000","last_change":"2024-09-17T07:06:36.398341+0000","last_active":"2024-09-17T07:07:14.168731+0000","last_peered":"2024-09-17T07:07:14.168731+0000","last_clean":"2024-09-17T07:07:14.168731+0000","last_became_active":"2024-09-17T07:02:18.017825+0000","last_became_peered":"2024-09-17T07:02:18.017825+0000","last_unstale":"2024-09-17T07:07:14.168731+0000","last_undegraded":"2024-09-17T07:07:14.168731+0000","last_fullsized":"2024-09-17T07:07:14.168731+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:36.398177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:06:36.398177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:57.728751+0000","scrub_duration":0.099287720999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.14","version":"160'7962","reported_seq":15126,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-17T07:07:18.591862+0000","last_change":"2024-09-17T07:06:25.086462+0000","last_active":"2024-09-17T07:07:18.591862+0000","last_peered":"2024-09-17T07:07:18.591862+0000","last_clean":"2024-09-17T07:07:18.591862+0000","last_became_active":"2024-09-17T07:06:25.076747+0000","last_became_peered":"2024-09-17T07:06:25.076747+0000","last_unstale":"2024-09-17T07:07:18.591862+0000","last_undegraded":"2024-09-17T07:07:18.591862+0000","last_fullsized":"2024-09-17T07:07:18.591862+0000","mapping_epoch":147,"log_start":"110'3434","ondisk_log_start":"110'3434","created":111,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_clean_scrub_stamp":"2024-09-17T07:04:19.785747+0000","objects_scrubbed":0,"log_size":4528,"log_dups_size":0,"ondisk_log_size":4528,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44072960,"num_objects":673,"num_object_clones":0,"num_object_copies":2019,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6239,"num_write_kb":221152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1767,"num_bytes_recovered":115802112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":125,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.203260+0000","last_change":"2024-09-17T07:06:11.251464+0000","last_active":"2024-09-17T07:07:14.203260+0000","last_peered":"2024-09-17T07:07:14.203260+0000","last_clean":"2024-09-17T07:07:14.203260+0000","last_became_active":"2024-09-17T07:02:12.008065+0000","last_became_peered":"2024-09-17T07:02:12.008065+0000","last_unstale":"2024-09-17T07:07:14.203260+0000","last_undegraded":"2024-09-17T07:07:14.203260+0000","last_fullsized":"2024-09-17T07:07:14.203260+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:11.251341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:06:11.251341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:28.019325+0000","scrub_duration":0.0029670590000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"75'6","reported_seq":132,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.178251+0000","last_change":"2024-09-17T07:06:43.839442+0000","last_active":"2024-09-17T07:07:14.178251+0000","last_peered":"2024-09-17T07:07:14.178251+0000","last_clean":"2024-09-17T07:07:14.178251+0000","last_became_active":"2024-09-17T07:02:14.028655+0000","last_became_peered":"2024-09-17T07:02:14.028655+0000","last_unstale":"2024-09-17T07:07:14.178251+0000","last_undegraded":"2024-09-17T07:07:14.178251+0000","last_fullsized":"2024-09-17T07:07:14.178251+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'6","last_scrub_stamp":"2024-09-17T07:06:43.836545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:06:43.836545+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:48.040328+0000","scrub_duration":0.0059941980000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":121,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.193081+0000","last_change":"2024-09-17T07:06:29.854234+0000","last_active":"2024-09-17T07:07:14.193081+0000","last_peered":"2024-09-17T07:07:14.193081+0000","last_clean":"2024-09-17T07:07:14.193081+0000","last_became_active":"2024-09-17T07:02:16.016634+0000","last_became_peered":"2024-09-17T07:02:16.016634+0000","last_unstale":"2024-09-17T07:07:14.193081+0000","last_undegraded":"2024-09-17T07:07:14.193081+0000","last_fullsized":"2024-09-17T07:07:14.193081+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:29.854126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:06:29.854126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:37.183540+0000","scrub_duration":0.0052853860000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":120,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-17T07:07:18.042118+0000","last_change":"2024-09-17T07:07:18.042204+0000","last_active":"2024-09-17T07:07:18.042118+0000","last_peered":"2024-09-17T07:07:18.042118+0000","last_clean":"2024-09-17T07:07:18.042118+0000","last_became_active":"2024-09-17T07:02:18.365419+0000","last_became_peered":"2024-09-17T07:02:18.365419+0000","last_unstale":"2024-09-17T07:07:18.042118+0000","last_undegraded":"2024-09-17T07:07:18.042118+0000","last_fullsized":"2024-09-17T07:07:18.042118+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:18.042105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:07:18.042105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:18.135607+0000","scrub_duration":0.0020576729999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.17","version":"160'8147","reported_seq":15491,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.158575+0000","last_change":"2024-09-17T07:06:22.239760+0000","last_active":"2024-09-17T07:07:23.158575+0000","last_peered":"2024-09-17T07:07:23.158575+0000","last_clean":"2024-09-17T07:07:23.158575+0000","last_became_active":"2024-09-17T07:06:22.231579+0000","last_became_peered":"2024-09-17T07:06:22.231579+0000","last_unstale":"2024-09-17T07:07:23.158575+0000","last_undegraded":"2024-09-17T07:07:23.158575+0000","last_fullsized":"2024-09-17T07:07:23.158575+0000","mapping_epoch":145,"log_start":"118'3699","ondisk_log_start":"118'3699","created":111,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_clean_scrub_stamp":"2024-09-17T07:04:19.785747+0000","objects_scrubbed":0,"log_size":4448,"log_dups_size":0,"ondisk_log_size":4448,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28573696,"num_objects":436,"num_object_clones":0,"num_object_copies":1308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6420,"num_write_kb":219392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1812,"num_bytes_recovered":118751232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"75'10","reported_seq":138,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.179077+0000","last_change":"2024-09-17T07:04:40.721975+0000","last_active":"2024-09-17T07:07:14.179077+0000","last_peered":"2024-09-17T07:07:14.179077+0000","last_clean":"2024-09-17T07:07:14.179077+0000","last_became_active":"2024-09-17T07:02:14.027499+0000","last_became_peered":"2024-09-17T07:02:14.027499+0000","last_unstale":"2024-09-17T07:07:14.179077+0000","last_undegraded":"2024-09-17T07:07:14.179077+0000","last_fullsized":"2024-09-17T07:07:14.179077+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-09-17T07:04:40.721843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:04:40.721843+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0057465399999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":136,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.191859+0000","last_change":"2024-09-17T07:06:45.799922+0000","last_active":"2024-09-17T07:07:14.191859+0000","last_peered":"2024-09-17T07:07:14.191859+0000","last_clean":"2024-09-17T07:07:14.191859+0000","last_became_active":"2024-09-17T07:02:12.007280+0000","last_became_peered":"2024-09-17T07:02:12.007280+0000","last_unstale":"2024-09-17T07:07:14.191859+0000","last_undegraded":"2024-09-17T07:07:14.191859+0000","last_fullsized":"2024-09-17T07:07:14.191859+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:45.799808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:06:45.799808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:47.601528+0000","scrub_duration":0.0021198189999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"75'8","reported_seq":191,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-17T07:07:17.188687+0000","last_change":"2024-09-17T07:06:24.892916+0000","last_active":"2024-09-17T07:07:17.188687+0000","last_peered":"2024-09-17T07:07:17.188687+0000","last_clean":"2024-09-17T07:07:17.188687+0000","last_became_active":"2024-09-17T07:02:16.022583+0000","last_became_peered":"2024-09-17T07:02:16.022583+0000","last_unstale":"2024-09-17T07:07:17.188687+0000","last_undegraded":"2024-09-17T07:07:17.188687+0000","last_fullsized":"2024-09-17T07:07:17.188687+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:06:24.892804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:06:24.892804+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:29.372425+0000","scrub_duration":0.0018794949999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":109,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.176060+0000","last_change":"2024-09-17T07:03:53.976711+0000","last_active":"2024-09-17T07:07:14.176060+0000","last_peered":"2024-09-17T07:07:14.176060+0000","last_clean":"2024-09-17T07:07:14.176060+0000","last_became_active":"2024-09-17T07:02:18.028796+0000","last_became_peered":"2024-09-17T07:02:18.028796+0000","last_unstale":"2024-09-17T07:07:14.176060+0000","last_undegraded":"2024-09-17T07:07:14.176060+0000","last_fullsized":"2024-09-17T07:07:14.176060+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:53.976586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:03:53.976586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021355279999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"160'7923","reported_seq":11563,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-17T07:07:22.981867+0000","last_change":"2024-09-17T07:05:24.306755+0000","last_active":"2024-09-17T07:07:22.981867+0000","last_peered":"2024-09-17T07:07:22.981867+0000","last_clean":"2024-09-17T07:07:22.981867+0000","last_became_active":"2024-09-17T07:05:24.303086+0000","last_became_peered":"2024-09-17T07:05:24.303086+0000","last_unstale":"2024-09-17T07:07:22.981867+0000","last_undegraded":"2024-09-17T07:07:22.981867+0000","last_fullsized":"2024-09-17T07:07:22.981867+0000","mapping_epoch":123,"log_start":"106'2168","ondisk_log_start":"106'2168","created":111,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_clean_scrub_stamp":"2024-09-17T07:04:19.785747+0000","objects_scrubbed":0,"log_size":5755,"log_dups_size":0,"ondisk_log_size":5755,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28540928,"num_objects":436,"num_object_clones":0,"num_object_copies":1308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6250,"num_write_kb":213920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1771,"num_bytes_recovered":116064256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"75'9","reported_seq":150,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.192995+0000","last_change":"2024-09-17T07:06:35.727847+0000","last_active":"2024-09-17T07:07:14.192995+0000","last_peered":"2024-09-17T07:07:14.192995+0000","last_clean":"2024-09-17T07:07:14.192995+0000","last_became_active":"2024-09-17T07:02:14.028485+0000","last_became_peered":"2024-09-17T07:02:14.028485+0000","last_unstale":"2024-09-17T07:07:14.192995+0000","last_undegraded":"2024-09-17T07:07:14.192995+0000","last_fullsized":"2024-09-17T07:07:14.192995+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:06:35.727700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:06:35.727700+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:52.353085+0000","scrub_duration":0.0045062289999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":126,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.172600+0000","last_change":"2024-09-17T07:04:51.449374+0000","last_active":"2024-09-17T07:07:14.172600+0000","last_peered":"2024-09-17T07:07:14.172600+0000","last_clean":"2024-09-17T07:07:14.172600+0000","last_became_active":"2024-09-17T07:02:12.009935+0000","last_became_peered":"2024-09-17T07:02:12.009935+0000","last_unstale":"2024-09-17T07:07:14.172600+0000","last_undegraded":"2024-09-17T07:07:14.172600+0000","last_fullsized":"2024-09-17T07:07:14.172600+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:51.449197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:04:51.449197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0069844269999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"75'8","reported_seq":179,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:15.577656+0000","last_change":"2024-09-17T07:03:42.469019+0000","last_active":"2024-09-17T07:07:15.577656+0000","last_peered":"2024-09-17T07:07:15.577656+0000","last_clean":"2024-09-17T07:07:15.577656+0000","last_became_active":"2024-09-17T07:02:16.016069+0000","last_became_peered":"2024-09-17T07:02:16.016069+0000","last_unstale":"2024-09-17T07:07:15.577656+0000","last_undegraded":"2024-09-17T07:07:15.577656+0000","last_fullsized":"2024-09-17T07:07:15.577656+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:03:42.468934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:42.468934+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0016937079999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":120,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.198349+0000","last_change":"2024-09-17T07:04:58.773306+0000","last_active":"2024-09-17T07:07:14.198349+0000","last_peered":"2024-09-17T07:07:14.198349+0000","last_clean":"2024-09-17T07:07:14.198349+0000","last_became_active":"2024-09-17T07:02:18.018716+0000","last_became_peered":"2024-09-17T07:02:18.018716+0000","last_unstale":"2024-09-17T07:07:14.198349+0000","last_undegraded":"2024-09-17T07:07:14.198349+0000","last_fullsized":"2024-09-17T07:07:14.198349+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:58.773188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:04:58.773188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0024048339999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.11","version":"160'8148","reported_seq":11877,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.202698+0000","last_change":"2024-09-17T07:05:30.096012+0000","last_active":"2024-09-17T07:07:23.202698+0000","last_peered":"2024-09-17T07:07:23.202698+0000","last_clean":"2024-09-17T07:07:23.202698+0000","last_became_active":"2024-09-17T07:05:23.265789+0000","last_became_peered":"2024-09-17T07:05:23.265789+0000","last_unstale":"2024-09-17T07:07:23.202698+0000","last_undegraded":"2024-09-17T07:07:23.202698+0000","last_fullsized":"2024-09-17T07:07:23.202698+0000","mapping_epoch":122,"log_start":"118'3699","ondisk_log_start":"118'3699","created":111,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_clean_scrub_stamp":"2024-09-17T07:04:19.785747+0000","objects_scrubbed":0,"log_size":4449,"log_dups_size":0,"ondisk_log_size":4449,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31195136,"num_objects":476,"num_object_clones":0,"num_object_copies":1428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":476,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6404,"num_write_kb":220160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1815,"num_bytes_recovered":118947840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"75'9","reported_seq":139,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.181996+0000","last_change":"2024-09-17T07:04:56.049769+0000","last_active":"2024-09-17T07:07:14.181996+0000","last_peered":"2024-09-17T07:07:14.181996+0000","last_clean":"2024-09-17T07:07:14.181996+0000","last_became_active":"2024-09-17T07:02:14.010289+0000","last_became_peered":"2024-09-17T07:02:14.010289+0000","last_unstale":"2024-09-17T07:07:14.181996+0000","last_undegraded":"2024-09-17T07:07:14.181996+0000","last_fullsized":"2024-09-17T07:07:14.181996+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:04:56.049602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:04:56.049602+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0066971340000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":126,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-17T07:07:20.458331+0000","last_change":"2024-09-17T07:07:20.458487+0000","last_active":"2024-09-17T07:07:20.458331+0000","last_peered":"2024-09-17T07:07:20.458331+0000","last_clean":"2024-09-17T07:07:20.458331+0000","last_became_active":"2024-09-17T07:02:12.007433+0000","last_became_peered":"2024-09-17T07:02:12.007433+0000","last_unstale":"2024-09-17T07:07:20.458331+0000","last_undegraded":"2024-09-17T07:07:20.458331+0000","last_fullsized":"2024-09-17T07:07:20.458331+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:20.458316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:07:20.458316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:42.134938+0000","scrub_duration":0.0043263889999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":121,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-17T07:07:17.713051+0000","last_change":"2024-09-17T07:07:17.713153+0000","last_active":"2024-09-17T07:07:17.713051+0000","last_peered":"2024-09-17T07:07:17.713051+0000","last_clean":"2024-09-17T07:07:17.713051+0000","last_became_active":"2024-09-17T07:02:16.016216+0000","last_became_peered":"2024-09-17T07:02:16.016216+0000","last_unstale":"2024-09-17T07:07:17.713051+0000","last_undegraded":"2024-09-17T07:07:17.713051+0000","last_fullsized":"2024-09-17T07:07:17.713051+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:17.713027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:07:17.713027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:19.920380+0000","scrub_duration":0.017242598000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":120,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.172345+0000","last_change":"2024-09-17T07:06:24.966953+0000","last_active":"2024-09-17T07:07:14.172345+0000","last_peered":"2024-09-17T07:07:14.172345+0000","last_clean":"2024-09-17T07:07:14.172345+0000","last_became_active":"2024-09-17T07:02:18.027622+0000","last_became_peered":"2024-09-17T07:02:18.027622+0000","last_unstale":"2024-09-17T07:07:14.172345+0000","last_undegraded":"2024-09-17T07:07:14.172345+0000","last_fullsized":"2024-09-17T07:07:14.172345+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:24.966867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:06:24.966867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:26.322962+0000","scrub_duration":0.001663292,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"160'7736","reported_seq":11223,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-17T07:07:18.551047+0000","last_change":"2024-09-17T07:05:32.282685+0000","last_active":"2024-09-17T07:07:18.551047+0000","last_peered":"2024-09-17T07:07:18.551047+0000","last_clean":"2024-09-17T07:07:18.551047+0000","last_became_active":"2024-09-17T07:05:32.280398+0000","last_became_peered":"2024-09-17T07:05:32.280398+0000","last_unstale":"2024-09-17T07:07:18.551047+0000","last_undegraded":"2024-09-17T07:07:18.551047+0000","last_fullsized":"2024-09-17T07:07:18.551047+0000","mapping_epoch":126,"log_start":"109'3163","ondisk_log_start":"109'3163","created":111,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_clean_scrub_stamp":"2024-09-17T07:04:19.785747+0000","objects_scrubbed":0,"log_size":4573,"log_dups_size":0,"ondisk_log_size":4573,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43581440,"num_objects":665,"num_object_clones":0,"num_object_copies":1995,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6055,"num_write_kb":215040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1695,"num_bytes_recovered":111083520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"75'11","reported_seq":143,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.198483+0000","last_change":"2024-09-17T07:06:11.787256+0000","last_active":"2024-09-17T07:07:14.198483+0000","last_peered":"2024-09-17T07:07:14.198483+0000","last_clean":"2024-09-17T07:07:14.198483+0000","last_became_active":"2024-09-17T07:02:14.016255+0000","last_became_peered":"2024-09-17T07:02:14.016255+0000","last_unstale":"2024-09-17T07:07:14.198483+0000","last_undegraded":"2024-09-17T07:07:14.198483+0000","last_fullsized":"2024-09-17T07:07:14.198483+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-09-17T07:06:11.787156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:06:11.787156+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:26.657923+0000","scrub_duration":0.0035465940000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"94'2","reported_seq":120,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.182628+0000","last_change":"2024-09-17T07:04:06.027032+0000","last_active":"2024-09-17T07:07:14.182628+0000","last_peered":"2024-09-17T07:07:14.182628+0000","last_clean":"2024-09-17T07:07:14.182628+0000","last_became_active":"2024-09-17T07:02:12.011461+0000","last_became_peered":"2024-09-17T07:02:12.011461+0000","last_unstale":"2024-09-17T07:07:14.182628+0000","last_undegraded":"2024-09-17T07:07:14.182628+0000","last_fullsized":"2024-09-17T07:07:14.182628+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-09-17T07:04:06.026890+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-09-17T07:04:06.026890+0000","last_clean_scrub_stamp":"2024-09-17T07:04:06.026890+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0031797539999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":121,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.181800+0000","last_change":"2024-09-17T07:06:40.147092+0000","last_active":"2024-09-17T07:07:14.181800+0000","last_peered":"2024-09-17T07:07:14.181800+0000","last_clean":"2024-09-17T07:07:14.181800+0000","last_became_active":"2024-09-17T07:02:16.018754+0000","last_became_peered":"2024-09-17T07:02:16.018754+0000","last_unstale":"2024-09-17T07:07:14.181800+0000","last_undegraded":"2024-09-17T07:07:14.181800+0000","last_fullsized":"2024-09-17T07:07:14.181800+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:40.146846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:06:40.146846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:05.696921+0000","scrub_duration":0.00753165,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":108,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.177905+0000","last_change":"2024-09-17T07:04:07.566836+0000","last_active":"2024-09-17T07:07:14.177905+0000","last_peered":"2024-09-17T07:07:14.177905+0000","last_clean":"2024-09-17T07:07:14.177905+0000","last_became_active":"2024-09-17T07:02:18.027427+0000","last_became_peered":"2024-09-17T07:02:18.027427+0000","last_unstale":"2024-09-17T07:07:14.177905+0000","last_undegraded":"2024-09-17T07:07:14.177905+0000","last_fullsized":"2024-09-17T07:07:14.177905+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:07.566667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:04:07.566667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001538364,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"160'8270","reported_seq":13922,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-17T07:07:18.884823+0000","last_change":"2024-09-17T07:05:58.153071+0000","last_active":"2024-09-17T07:07:18.884823+0000","last_peered":"2024-09-17T07:07:18.884823+0000","last_clean":"2024-09-17T07:07:18.884823+0000","last_became_active":"2024-09-17T07:05:58.145854+0000","last_became_peered":"2024-09-17T07:05:58.145854+0000","last_unstale":"2024-09-17T07:07:18.884823+0000","last_undegraded":"2024-09-17T07:07:18.884823+0000","last_fullsized":"2024-09-17T07:07:18.884823+0000","mapping_epoch":134,"log_start":"109'3325","ondisk_log_start":"109'3325","created":111,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_clean_scrub_stamp":"2024-09-17T07:04:19.785747+0000","objects_scrubbed":0,"log_size":4945,"log_dups_size":0,"ondisk_log_size":4945,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41877504,"num_objects":639,"num_object_clones":0,"num_object_copies":1917,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":639,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6497,"num_write_kb":228352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1852,"num_bytes_recovered":121372672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"75'4","reported_seq":144,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.179442+0000","last_change":"2024-09-17T07:06:36.857728+0000","last_active":"2024-09-17T07:07:14.179442+0000","last_peered":"2024-09-17T07:07:14.179442+0000","last_clean":"2024-09-17T07:07:14.179442+0000","last_became_active":"2024-09-17T07:03:48.758220+0000","last_became_peered":"2024-09-17T07:03:48.758220+0000","last_unstale":"2024-09-17T07:07:14.179442+0000","last_undegraded":"2024-09-17T07:07:14.179442+0000","last_fullsized":"2024-09-17T07:07:14.179442+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'4","last_scrub_stamp":"2024-09-17T07:06:36.857471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:06:36.857471+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:56.904497+0000","scrub_duration":0.0095623189999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":125,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.193251+0000","last_change":"2024-09-17T07:06:26.844807+0000","last_active":"2024-09-17T07:07:14.193251+0000","last_peered":"2024-09-17T07:07:14.193251+0000","last_clean":"2024-09-17T07:07:14.193251+0000","last_became_active":"2024-09-17T07:02:12.008898+0000","last_became_peered":"2024-09-17T07:02:12.008898+0000","last_unstale":"2024-09-17T07:07:14.193251+0000","last_undegraded":"2024-09-17T07:07:14.193251+0000","last_fullsized":"2024-09-17T07:07:14.193251+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:26.844664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:06:26.844664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:32.712860+0000","scrub_duration":0.0012489000000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":121,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.202981+0000","last_change":"2024-09-17T07:04:57.667081+0000","last_active":"2024-09-17T07:07:14.202981+0000","last_peered":"2024-09-17T07:07:14.202981+0000","last_clean":"2024-09-17T07:07:14.202981+0000","last_became_active":"2024-09-17T07:02:16.014982+0000","last_became_peered":"2024-09-17T07:02:16.014982+0000","last_unstale":"2024-09-17T07:07:14.202981+0000","last_undegraded":"2024-09-17T07:07:14.202981+0000","last_fullsized":"2024-09-17T07:07:14.202981+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:57.667006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:47.444972+0000","last_clean_scrub_stamp":"2024-09-17T07:04:57.667006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0011999210000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":120,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.192888+0000","last_change":"2024-09-17T07:06:40.865239+0000","last_active":"2024-09-17T07:07:14.192888+0000","last_peered":"2024-09-17T07:07:14.192888+0000","last_clean":"2024-09-17T07:07:14.192888+0000","last_became_active":"2024-09-17T07:02:18.021014+0000","last_became_peered":"2024-09-17T07:02:18.021014+0000","last_unstale":"2024-09-17T07:07:14.192888+0000","last_undegraded":"2024-09-17T07:07:14.192888+0000","last_fullsized":"2024-09-17T07:07:14.192888+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:40.865049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:06:40.865049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:48.155323+0000","scrub_duration":0.0029834100000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.12","version":"160'8144","reported_seq":13582,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.094069+0000","last_change":"2024-09-17T07:06:31.715791+0000","last_active":"2024-09-17T07:07:23.094069+0000","last_peered":"2024-09-17T07:07:23.094069+0000","last_clean":"2024-09-17T07:07:23.094069+0000","last_became_active":"2024-09-17T07:05:39.199163+0000","last_became_peered":"2024-09-17T07:05:39.199163+0000","last_unstale":"2024-09-17T07:07:23.094069+0000","last_undegraded":"2024-09-17T07:07:23.094069+0000","last_fullsized":"2024-09-17T07:07:23.094069+0000","mapping_epoch":129,"log_start":"105'1598","ondisk_log_start":"105'1598","created":111,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":5,"last_scrub":"150'6246","last_scrub_stamp":"2024-09-17T07:06:31.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_clean_scrub_stamp":"2024-09-17T07:06:31.715683+0000","objects_scrubbed":2410,"log_size":6546,"log_dups_size":0,"ondisk_log_size":6546,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:52.906727+0000","scrub_duration":0.55965373699999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32833536,"num_objects":501,"num_object_clones":0,"num_object_copies":1503,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1909,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"75'11","reported_seq":142,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.178118+0000","last_change":"2024-09-17T07:04:52.733713+0000","last_active":"2024-09-17T07:07:14.178118+0000","last_peered":"2024-09-17T07:07:14.178118+0000","last_clean":"2024-09-17T07:07:14.178118+0000","last_became_active":"2024-09-17T07:02:14.027316+0000","last_became_peered":"2024-09-17T07:02:14.027316+0000","last_unstale":"2024-09-17T07:07:14.178118+0000","last_undegraded":"2024-09-17T07:07:14.178118+0000","last_fullsized":"2024-09-17T07:07:14.178118+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-09-17T07:04:52.733516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:04:52.733516+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.013209306000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":136,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.204708+0000","last_change":"2024-09-17T07:06:44.298186+0000","last_active":"2024-09-17T07:07:14.204708+0000","last_peered":"2024-09-17T07:07:14.204708+0000","last_clean":"2024-09-17T07:07:14.204708+0000","last_became_active":"2024-09-17T07:02:12.010195+0000","last_became_peered":"2024-09-17T07:02:12.010195+0000","last_unstale":"2024-09-17T07:07:14.204708+0000","last_undegraded":"2024-09-17T07:07:14.204708+0000","last_fullsized":"2024-09-17T07:07:14.204708+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:44.298065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:06:44.298065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:00.552155+0000","scrub_duration":0.0032727260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":122,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.194446+0000","last_change":"2024-09-17T07:06:33.856836+0000","last_active":"2024-09-17T07:07:14.194446+0000","last_peered":"2024-09-17T07:07:14.194446+0000","last_clean":"2024-09-17T07:07:14.194446+0000","last_became_active":"2024-09-17T07:02:16.016183+0000","last_became_peered":"2024-09-17T07:02:16.016183+0000","last_unstale":"2024-09-17T07:07:14.194446+0000","last_undegraded":"2024-09-17T07:07:14.194446+0000","last_fullsized":"2024-09-17T07:07:14.194446+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:33.856556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:06:33.856556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:00.460166+0000","scrub_duration":0.0017432000000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":108,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.177975+0000","last_change":"2024-09-17T07:03:29.467351+0000","last_active":"2024-09-17T07:07:14.177975+0000","last_peered":"2024-09-17T07:07:14.177975+0000","last_clean":"2024-09-17T07:07:14.177975+0000","last_became_active":"2024-09-17T07:02:18.365656+0000","last_became_peered":"2024-09-17T07:02:18.365656+0000","last_unstale":"2024-09-17T07:07:14.177975+0000","last_undegraded":"2024-09-17T07:07:14.177975+0000","last_fullsized":"2024-09-17T07:07:14.177975+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:29.467252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:03:29.467252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001842804,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"75'1","reported_seq":121,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-17T07:07:17.187195+0000","last_change":"2024-09-17T07:06:32.836489+0000","last_active":"2024-09-17T07:07:17.187195+0000","last_peered":"2024-09-17T07:07:17.187195+0000","last_clean":"2024-09-17T07:07:17.187195+0000","last_became_active":"2024-09-17T07:02:18.021855+0000","last_became_peered":"2024-09-17T07:02:18.021855+0000","last_unstale":"2024-09-17T07:07:17.187195+0000","last_undegraded":"2024-09-17T07:07:17.187195+0000","last_fullsized":"2024-09-17T07:07:17.187195+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-17T07:06:32.836363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:06:32.836363+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:34.947914+0000","scrub_duration":0.0056683599999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":136,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-17T07:07:22.781765+0000","last_change":"2024-09-17T07:07:22.782091+0000","last_active":"2024-09-17T07:07:22.781765+0000","last_peered":"2024-09-17T07:07:22.781765+0000","last_clean":"2024-09-17T07:07:22.781765+0000","last_became_active":"2024-09-17T07:02:12.005329+0000","last_became_peered":"2024-09-17T07:02:12.005329+0000","last_unstale":"2024-09-17T07:07:22.781765+0000","last_undegraded":"2024-09-17T07:07:22.781765+0000","last_fullsized":"2024-09-17T07:07:22.781765+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:22.781739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:07:22.781739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:44.511386+0000","scrub_duration":0.0037808680000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"75'10","reported_seq":128,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.176349+0000","last_change":"2024-09-17T07:03:56.069095+0000","last_active":"2024-09-17T07:07:14.176349+0000","last_peered":"2024-09-17T07:07:14.176349+0000","last_clean":"2024-09-17T07:07:14.176349+0000","last_became_active":"2024-09-17T07:02:14.028731+0000","last_became_peered":"2024-09-17T07:02:14.028731+0000","last_unstale":"2024-09-17T07:07:14.176349+0000","last_undegraded":"2024-09-17T07:07:14.176349+0000","last_fullsized":"2024-09-17T07:07:14.176349+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-09-17T07:03:56.068998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:03:56.068998+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0020277960000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"75'8","reported_seq":195,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-17T07:07:19.425224+0000","last_change":"2024-09-17T07:07:19.425224+0000","last_active":"2024-09-17T07:07:19.425224+0000","last_peered":"2024-09-17T07:07:19.425224+0000","last_clean":"2024-09-17T07:07:19.425224+0000","last_became_active":"2024-09-17T07:02:16.020560+0000","last_became_peered":"2024-09-17T07:02:16.020560+0000","last_unstale":"2024-09-17T07:07:19.425224+0000","last_undegraded":"2024-09-17T07:07:19.425224+0000","last_fullsized":"2024-09-17T07:07:19.425224+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:07:19.423853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:07:19.423853+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:23.974900+0000","scrub_duration":0.0037511480000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":108,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-17T07:07:17.189570+0000","last_change":"2024-09-17T07:03:58.948312+0000","last_active":"2024-09-17T07:07:17.189570+0000","last_peered":"2024-09-17T07:07:17.189570+0000","last_clean":"2024-09-17T07:07:17.189570+0000","last_became_active":"2024-09-17T07:02:18.021603+0000","last_became_peered":"2024-09-17T07:02:18.021603+0000","last_unstale":"2024-09-17T07:07:17.189570+0000","last_undegraded":"2024-09-17T07:07:17.189570+0000","last_fullsized":"2024-09-17T07:07:17.189570+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:58.948227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:03:58.948227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0018946010000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":125,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.179190+0000","last_change":"2024-09-17T07:04:34.679969+0000","last_active":"2024-09-17T07:07:14.179190+0000","last_peered":"2024-09-17T07:07:14.179190+0000","last_clean":"2024-09-17T07:07:14.179190+0000","last_became_active":"2024-09-17T07:02:12.009837+0000","last_became_peered":"2024-09-17T07:02:12.009837+0000","last_unstale":"2024-09-17T07:07:14.179190+0000","last_undegraded":"2024-09-17T07:07:14.179190+0000","last_fullsized":"2024-09-17T07:07:14.179190+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:34.679855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:04:34.679855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0053726590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"75'11","reported_seq":174,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.204628+0000","last_change":"2024-09-17T07:06:34.254104+0000","last_active":"2024-09-17T07:07:14.204628+0000","last_peered":"2024-09-17T07:07:14.204628+0000","last_clean":"2024-09-17T07:07:14.204628+0000","last_became_active":"2024-09-17T07:03:48.760390+0000","last_became_peered":"2024-09-17T07:03:48.760390+0000","last_unstale":"2024-09-17T07:07:14.204628+0000","last_undegraded":"2024-09-17T07:07:14.204628+0000","last_fullsized":"2024-09-17T07:07:14.204628+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-09-17T07:06:34.253977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:06:34.253977+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:36.450626+0000","scrub_duration":0.002455334,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":111,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.176284+0000","last_change":"2024-09-17T07:04:02.220014+0000","last_active":"2024-09-17T07:07:14.176284+0000","last_peered":"2024-09-17T07:07:14.176284+0000","last_clean":"2024-09-17T07:07:14.176284+0000","last_became_active":"2024-09-17T07:02:16.019701+0000","last_became_peered":"2024-09-17T07:02:16.019701+0000","last_unstale":"2024-09-17T07:07:14.176284+0000","last_undegraded":"2024-09-17T07:07:14.176284+0000","last_fullsized":"2024-09-17T07:07:14.176284+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:02.219927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:04:02.219927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015145639999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":109,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.198641+0000","last_change":"2024-09-17T07:04:03.393754+0000","last_active":"2024-09-17T07:07:14.198641+0000","last_peered":"2024-09-17T07:07:14.198641+0000","last_clean":"2024-09-17T07:07:14.198641+0000","last_became_active":"2024-09-17T07:02:18.363752+0000","last_became_peered":"2024-09-17T07:02:18.363752+0000","last_unstale":"2024-09-17T07:07:14.198641+0000","last_undegraded":"2024-09-17T07:07:14.198641+0000","last_fullsized":"2024-09-17T07:07:14.198641+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:03.393645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:04:03.393645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001487555,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":145,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.203323+0000","last_change":"2024-09-17T07:06:41.312034+0000","last_active":"2024-09-17T07:07:14.203323+0000","last_peered":"2024-09-17T07:07:14.203323+0000","last_clean":"2024-09-17T07:07:14.203323+0000","last_became_active":"2024-09-17T07:03:48.759384+0000","last_became_peered":"2024-09-17T07:03:48.759384+0000","last_unstale":"2024-09-17T07:07:14.203323+0000","last_undegraded":"2024-09-17T07:07:14.203323+0000","last_fullsized":"2024-09-17T07:07:14.203323+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:41.311912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:06:41.311912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:55.289346+0000","scrub_duration":0.016335471000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"75'15","reported_seq":124,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.182020+0000","last_change":"2024-09-17T07:04:38.880694+0000","last_active":"2024-09-17T07:07:14.182020+0000","last_peered":"2024-09-17T07:07:14.182020+0000","last_clean":"2024-09-17T07:07:14.182020+0000","last_became_active":"2024-09-17T07:04:15.114286+0000","last_became_peered":"2024-09-17T07:04:15.114286+0000","last_unstale":"2024-09-17T07:07:14.182020+0000","last_undegraded":"2024-09-17T07:07:14.182020+0000","last_fullsized":"2024-09-17T07:07:14.182020+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-09-17T07:04:38.880277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:04:38.880277+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0059818789999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":121,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-17T07:07:17.189794+0000","last_change":"2024-09-17T07:06:42.201934+0000","last_active":"2024-09-17T07:07:17.189794+0000","last_peered":"2024-09-17T07:07:17.189794+0000","last_clean":"2024-09-17T07:07:17.189794+0000","last_became_active":"2024-09-17T07:02:16.015903+0000","last_became_peered":"2024-09-17T07:02:16.015903+0000","last_unstale":"2024-09-17T07:07:17.189794+0000","last_undegraded":"2024-09-17T07:07:17.189794+0000","last_fullsized":"2024-09-17T07:07:17.189794+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:42.200604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:06:42.200604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:54.763122+0000","scrub_duration":0.051542886000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":108,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.178161+0000","last_change":"2024-09-17T07:04:10.606687+0000","last_active":"2024-09-17T07:07:14.178161+0000","last_peered":"2024-09-17T07:07:14.178161+0000","last_clean":"2024-09-17T07:07:14.178161+0000","last_became_active":"2024-09-17T07:02:18.366466+0000","last_became_peered":"2024-09-17T07:02:18.366466+0000","last_unstale":"2024-09-17T07:07:14.178161+0000","last_undegraded":"2024-09-17T07:07:14.178161+0000","last_fullsized":"2024-09-17T07:07:14.178161+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:10.606552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:04:10.606552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00165456,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":114,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-17T07:07:14.178799+0000","last_change":"2024-09-17T07:03:44.446996+0000","last_active":"2024-09-17T07:07:14.178799+0000","last_peered":"2024-09-17T07:07:14.178799+0000","last_clean":"2024-09-17T07:07:14.178799+0000","last_became_active":"2024-09-17T07:02:12.005493+0000","last_became_peered":"2024-09-17T07:02:12.005493+0000","last_unstale":"2024-09-17T07:07:14.178799+0000","last_undegraded":"2024-09-17T07:07:14.178799+0000","last_fullsized":"2024-09-17T07:07:14.178799+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:44.446914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:44.446914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0029497619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"86'17","reported_seq":151,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-17T07:07:17.182088+0000","last_change":"2024-09-17T07:06:27.962393+0000","last_active":"2024-09-17T07:07:17.182088+0000","last_peered":"2024-09-17T07:07:17.182088+0000","last_clean":"2024-09-17T07:07:17.182088+0000","last_became_active":"2024-09-17T07:02:14.017737+0000","last_became_peered":"2024-09-17T07:02:14.017737+0000","last_unstale":"2024-09-17T07:07:17.182088+0000","last_undegraded":"2024-09-17T07:07:17.182088+0000","last_fullsized":"2024-09-17T07:07:17.182088+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'17","last_scrub_stamp":"2024-09-17T07:06:27.962276+0000","last_deep_scrub":"86'17","last_deep_scrub_stamp":"2024-09-17T07:03:18.241067+0000","last_clean_scrub_stamp":"2024-09-17T07:06:27.962276+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:48.911685+0000","scrub_duration":0.0050727970000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":122,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-17T07:07:17.182968+0000","last_change":"2024-09-17T07:05:00.800939+0000","last_active":"2024-09-17T07:07:17.182968+0000","last_peered":"2024-09-17T07:07:17.182968+0000","last_clean":"2024-09-17T07:07:17.182968+0000","last_became_active":"2024-09-17T07:02:16.017362+0000","last_became_peered":"2024-09-17T07:02:16.017362+0000","last_unstale":"2024-09-17T07:07:17.182968+0000","last_undegraded":"2024-09-17T07:07:17.182968+0000","last_fullsized":"2024-09-17T07:07:17.182968+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:05:00.800823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:05:00.800823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014180900000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":26,"stat_sum":{"num_bytes":1207468056,"num_objects":18427,"num_object_clones":0,"num_object_copies":55281,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18427,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":164872,"num_write_kb":5306785,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15823,"num_bytes_recovered":1036910592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3803918336,"data_stored":3803906120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134335,"ondisk_log_size":134335,"up":78,"acting":78,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66,"ondisk_log_size":66,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":697,"num_read_kb":454,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":421,"ondisk_log_size":421,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":304,"num_read_kb":299,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1386,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":12,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":4158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037828,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":632620,"kb_used_data":439076,"kb_used_omap":1,"kb_used_meta":193534,"kb_avail":93112532,"statfs":{"total":95995035648,"available":95347232768,"internally_reserved":0,"allocated":449613824,"data_stored":449222459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":198179275},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:07:13 2024","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":1.897,"15min":0.98499999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":3.613,"5min":52.478999999999999,"15min":52.478999999999999},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.98399999999999999,"5min":1.8979999999999999,"15min":0.97399999999999998},"min":{"1min":0.25900000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":2.9449999999999998,"5min":52.871000000000002,"15min":52.871000000000002},"last":0.51100000000000001}]},{"osd":1,"last update":"Tue Sep 17 07:07:13 2024","interfaces":[{"interface":"back","average":{"1min":1.2869999999999999,"5min":0.71799999999999997,"15min":0.57799999999999996},"min":{"1min":0.25600000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4.5140000000000002,"5min":4.5140000000000002,"15min":4.5140000000000002},"last":0.41299999999999998},{"interface":"front","average":{"1min":1.2270000000000001,"5min":0.74399999999999999,"15min":0.59699999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":3.8399999999999999,"5min":4.5039999999999996,"15min":4.5039999999999996},"last":0.67400000000000004}]},{"osd":2,"last update":"Tue Sep 17 07:07:13 2024","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":1.2190000000000001,"15min":0.72499999999999998},"min":{"1min":0.38700000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":3.3759999999999999,"5min":52.055999999999997,"15min":52.055999999999997},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.93000000000000005,"5min":1.256,"15min":0.78000000000000003},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":4.4749999999999996,"5min":52.064999999999998,"15min":52.064999999999998},"last":1.327}]},{"osd":3,"last update":"Tue Sep 17 07:07:13 2024","interfaces":[{"interface":"back","average":{"1min":1.0720000000000001,"5min":1.5720000000000001,"15min":0.874},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":3.9550000000000001,"5min":51.689,"15min":51.689},"last":0.57399999999999995},{"interface":"front","average":{"1min":1.54,"5min":1.8340000000000001,"15min":0.97799999999999998},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":3.5659999999999998,"5min":52.145000000000003,"15min":52.145000000000003},"last":0.748}]},{"osd":4,"last update":"Tue Sep 17 07:07:13 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.436,"15min":0.44500000000000001},"min":{"1min":0.25800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":1.5980000000000001,"5min":1.5980000000000001,"15min":1.5980000000000001},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.48999999999999999,"15min":0.44900000000000001},"min":{"1min":0.26800000000000002,"5min":0.248,"15min":0.248},"max":{"1min":3.4089999999999998,"5min":3.4089999999999998,"15min":3.4089999999999998},"last":0.46000000000000002}]},{"osd":5,"last update":"Tue Sep 17 07:07:13 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.48799999999999999,"15min":0.45100000000000001},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":3.3620000000000001,"5min":3.3620000000000001,"15min":3.3620000000000001},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.47799999999999998,"15min":0.46600000000000003},"min":{"1min":0.311,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.6459999999999999,"5min":1.6459999999999999,"15min":1.6459999999999999},"last":0.53100000000000003}]},{"osd":6,"last update":"Tue Sep 17 07:07:13 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.53200000000000003,"15min":0.50800000000000001},"min":{"1min":0.34499999999999997,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":1.675,"5min":1.675,"15min":1.675},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.91500000000000004,"5min":0.56699999999999995,"15min":0.51300000000000001},"min":{"1min":0.28899999999999998,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":3.2669999999999999,"5min":3.2669999999999999,"15min":3.2669999999999999},"last":0.81599999999999995}]}]},{"osd":6,"up_from":50,"seq":214748364871,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":550176,"kb_used_data":398488,"kb_used_omap":1,"kb_used_meta":151678,"kb_avail":93194976,"statfs":{"total":95995035648,"available":95431655424,"internally_reserved":0,"allocated":408051712,"data_stored":407671966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":155318727},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:06:50 2024","interfaces":[{"interface":"back","average":{"1min":7.2679999999999998,"5min":2.5960000000000001,"15min":1.198},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":75.323999999999998,"5min":75.323999999999998,"15min":75.323999999999998},"last":0.20899999999999999},{"interface":"front","average":{"1min":7.1449999999999996,"5min":2.5489999999999999,"15min":1.177},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":75.168999999999997,"5min":75.168999999999997,"15min":75.168999999999997},"last":2.8010000000000002}]},{"osd":1,"last update":"Tue Sep 17 07:06:50 2024","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":0.58999999999999997,"15min":0.52200000000000002},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":4.4790000000000001,"5min":4.4790000000000001,"15min":4.4790000000000001},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.89900000000000002,"5min":0.68999999999999995,"15min":0.58599999999999997},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":2.73,"5min":5.9530000000000003,"15min":5.9530000000000003},"last":0.29799999999999999}]},{"osd":2,"last update":"Tue Sep 17 07:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":0.97099999999999997,"15min":0.67100000000000004},"min":{"1min":0.28100000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":4.9850000000000003,"5min":33.079000000000001,"15min":33.079000000000001},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.97699999999999998,"15min":0.67200000000000004},"min":{"1min":0.32700000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":2.8759999999999999,"5min":33.100000000000001,"15min":33.100000000000001},"last":0.38}]},{"osd":3,"last update":"Tue Sep 17 07:06:50 2024","interfaces":[{"interface":"back","average":{"1min":3.544,"5min":1.6419999999999999,"15min":0.86699999999999999},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":52.944000000000003,"5min":52.944000000000003,"15min":52.944000000000003},"last":2.641},{"interface":"front","average":{"1min":3.3719999999999999,"5min":1.5700000000000001,"15min":0.88300000000000001},"min":{"1min":0.36199999999999999,"5min":0.308,"15min":0.308},"max":{"1min":52.747,"5min":52.747,"15min":52.747},"last":0.33900000000000002}]},{"osd":4,"last update":"Tue Sep 17 07:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.41999999999999998,"15min":0.41999999999999998},"min":{"1min":0.25900000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.311},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.48199999999999998,"15min":0.432},"min":{"1min":0.27200000000000002,"5min":0.25,"15min":0.25},"max":{"1min":2.9119999999999999,"5min":2.9119999999999999,"15min":2.9119999999999999},"last":0.503}]},{"osd":5,"last update":"Tue Sep 17 07:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.45500000000000002,"15min":0.433},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":1.6419999999999999,"5min":1.6419999999999999,"15min":1.6419999999999999},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.497,"15min":0.45000000000000001},"min":{"1min":0.29899999999999999,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":2.9390000000000001,"5min":2.9390000000000001,"15min":2.9390000000000001},"last":0.56100000000000005}]},{"osd":7,"last update":"Tue Sep 17 07:07:11 2024","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.55500000000000005,"15min":0.51500000000000001},"min":{"1min":0.307,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":2.859,"5min":2.859,"15min":2.859},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.57399999999999995,"15min":0.54200000000000004},"min":{"1min":0.38100000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.9339999999999999,"5min":1.9339999999999999,"15min":1.9339999999999999},"last":0.33400000000000002}]}]},{"osd":5,"up_from":42,"seq":180388626508,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":396508,"kb_used_data":285012,"kb_used_omap":1,"kb_used_meta":111486,"kb_avail":93348644,"statfs":{"total":95995035648,"available":95589011456,"internally_reserved":0,"allocated":291852288,"data_stored":291476880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":114162118},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:06:30 2024","interfaces":[{"interface":"back","average":{"1min":8.6699999999999999,"5min":2.1640000000000001,"15min":1.0680000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":49.265000000000001,"5min":49.265000000000001,"15min":49.265000000000001},"last":4.1870000000000003},{"interface":"front","average":{"1min":8.5050000000000008,"5min":2.294,"15min":1.1040000000000001},"min":{"1min":0.27300000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":47.350000000000001,"5min":47.350000000000001,"15min":47.350000000000001},"last":0.34000000000000002}]},{"osd":1,"last update":"Tue Sep 17 07:06:30 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.25600000000000001,"5min":0.25,"15min":0.25},"max":{"1min":1.7909999999999999,"5min":2.98,"15min":2.98},"last":1.53},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.82099999999999995,"15min":1.071},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":2.633,"5min":17.620999999999999,"15min":17.620999999999999},"last":0.316}]},{"osd":2,"last update":"Tue Sep 17 07:06:30 2024","interfaces":[{"interface":"back","average":{"1min":1.5149999999999999,"5min":0.81599999999999995,"15min":0.65000000000000002},"min":{"1min":0.32200000000000001,"5min":0.252,"15min":0.252},"max":{"1min":7.7779999999999996,"5min":7.7779999999999996,"15min":7.7779999999999996},"last":0.48899999999999999},{"interface":"front","average":{"1min":1.272,"5min":0.78600000000000003,"15min":0.60099999999999998},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":8.4450000000000003,"5min":8.4450000000000003,"15min":8.4450000000000003},"last":2.254}]},{"osd":3,"last update":"Tue Sep 17 07:06:30 2024","interfaces":[{"interface":"back","average":{"1min":1.5429999999999999,"5min":0.82199999999999995,"15min":0.64300000000000002},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":7.7190000000000003,"5min":7.7190000000000003,"15min":7.7190000000000003},"last":0.25800000000000001},{"interface":"front","average":{"1min":1.335,"5min":0.73999999999999999,"15min":0.61299999999999999},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":7.4660000000000002,"5min":7.4660000000000002,"15min":7.4660000000000002},"last":5.7210000000000001}]},{"osd":4,"last update":"Tue Sep 17 07:06:30 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.39200000000000002,"15min":0.39600000000000002},"min":{"1min":0.26600000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.53200000000000003,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.441,"15min":0.437},"min":{"1min":0.28799999999999998,"5min":0.248,"15min":0.248},"max":{"1min":1.5920000000000001,"5min":2.2160000000000002,"15min":2.2160000000000002},"last":0.28499999999999998}]},{"osd":6,"last update":"Tue Sep 17 07:06:42 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.42999999999999999,"15min":0.42699999999999999},"min":{"1min":0.27500000000000002,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.48699999999999999,"15min":0.45900000000000002},"min":{"1min":0.31900000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":2.5590000000000002,"5min":2.5590000000000002,"15min":2.5590000000000002},"last":0.439}]},{"osd":7,"last update":"Tue Sep 17 07:07:06 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.51000000000000001,"15min":0.49099999999999999},"min":{"1min":0.307,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":2.5910000000000002,"5min":2.5910000000000002,"15min":2.5910000000000002},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.47899999999999998,"15min":0.46899999999999997},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.30399999999999999}]}]},{"osd":4,"up_from":33,"seq":141733920850,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":300648,"kb_used_data":197344,"kb_used_omap":1,"kb_used_meta":103294,"kb_avail":93444504,"statfs":{"total":95995035648,"available":95687172096,"internally_reserved":0,"allocated":202080256,"data_stored":201691874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":105773510},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:06:56 2024","interfaces":[{"interface":"back","average":{"1min":3.1600000000000001,"5min":2.6579999999999999,"15min":1.274},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":48.825000000000003,"5min":74.551000000000002,"15min":74.551000000000002},"last":0.59099999999999997},{"interface":"front","average":{"1min":3.758,"5min":2.839,"15min":1.361},"min":{"1min":0.27500000000000002,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":50.911999999999999,"5min":74.778999999999996,"15min":74.778999999999996},"last":0.54000000000000004}]},{"osd":1,"last update":"Tue Sep 17 07:06:56 2024","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":0.628,"15min":0.59499999999999997},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4.1539999999999999,"5min":4.1539999999999999,"15min":4.1539999999999999},"last":0.33800000000000002},{"interface":"front","average":{"1min":1.222,"5min":0.66900000000000004,"15min":0.624},"min":{"1min":0.25600000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":3.5409999999999999,"5min":3.5409999999999999,"15min":3.5409999999999999},"last":0.28699999999999998}]},{"osd":2,"last update":"Tue Sep 17 07:06:56 2024","interfaces":[{"interface":"back","average":{"1min":3.7629999999999999,"5min":1.2490000000000001,"15min":0.81000000000000005},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":49.651000000000003,"5min":49.651000000000003,"15min":49.651000000000003},"last":0.35899999999999999},{"interface":"front","average":{"1min":3.9950000000000001,"5min":1.3080000000000001,"15min":0.83199999999999996},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":48.813000000000002,"5min":48.813000000000002,"15min":48.813000000000002},"last":2.6669999999999998}]},{"osd":3,"last update":"Tue Sep 17 07:06:56 2024","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.194,"15min":0.80800000000000005},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":3.4590000000000001,"5min":39.042999999999999,"15min":39.042999999999999},"last":2.4199999999999999},{"interface":"front","average":{"1min":1.0009999999999999,"5min":1.2490000000000001,"15min":0.79400000000000004},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":4.5679999999999996,"5min":38.920000000000002,"15min":38.920000000000002},"last":0.29999999999999999}]},{"osd":5,"last update":"Tue Sep 17 07:06:23 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.442,"15min":0.40100000000000002},"min":{"1min":0.25900000000000001,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":2.4780000000000002,"5min":2.4780000000000002,"15min":2.4780000000000002},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.45900000000000002,"15min":0.42099999999999999},"min":{"1min":0.27900000000000003,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":2.5419999999999998,"5min":2.5419999999999998,"15min":2.5419999999999998},"last":0.51100000000000001}]},{"osd":6,"last update":"Tue Sep 17 07:06:45 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.48799999999999999,"15min":0.42499999999999999},"min":{"1min":0.26400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":3.7770000000000001,"5min":3.7770000000000001,"15min":3.7770000000000001},"last":5.2149999999999999},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.48299999999999998,"15min":0.42999999999999999},"min":{"1min":0.30599999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":3.081,"5min":3.081,"15min":3.081},"last":0.32300000000000001}]},{"osd":7,"last update":"Tue Sep 17 07:07:07 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.54000000000000004,"15min":0.51400000000000001},"min":{"1min":0.28899999999999998,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":2.637,"5min":2.637,"15min":2.637},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.53200000000000003,"15min":0.497},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":3.7400000000000002,"5min":3.7400000000000002,"15min":3.7400000000000002},"last":0.44700000000000001}]}]},{"osd":3,"up_from":27,"seq":115964117075,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":829444,"kb_used_data":676088,"kb_used_omap":1,"kb_used_meta":153342,"kb_avail":92915708,"statfs":{"total":95995035648,"available":95145684992,"internally_reserved":0,"allocated":692314112,"data_stored":691901938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":157022664},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:06:43 2024","interfaces":[{"interface":"back","average":{"1min":1.3080000000000001,"5min":1.2,"15min":0.64200000000000002},"min":{"1min":0.20499999999999999,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":4.1879999999999997,"5min":17.364000000000001,"15min":17.364000000000001},"last":3.8210000000000002},{"interface":"front","average":{"1min":1.681,"5min":1.1339999999999999,"15min":0.63},"min":{"1min":0.38800000000000001,"5min":0.189,"15min":0.189},"max":{"1min":4.3259999999999996,"5min":18.521000000000001,"15min":18.521000000000001},"last":0.69799999999999995}]},{"osd":1,"last update":"Tue Sep 17 07:06:43 2024","interfaces":[{"interface":"back","average":{"1min":1.2210000000000001,"5min":0.79100000000000004,"15min":0.53600000000000003},"min":{"1min":0.32300000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":2.988,"5min":5.4619999999999997,"15min":5.4619999999999997},"last":0.59099999999999997},{"interface":"front","average":{"1min":1.454,"5min":1.0149999999999999,"15min":0.63900000000000001},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":4.0549999999999997,"5min":8.0670000000000002,"15min":8.0670000000000002},"last":2.661}]},{"osd":2,"last update":"Tue Sep 17 07:06:43 2024","interfaces":[{"interface":"back","average":{"1min":1.694,"5min":1.204,"15min":0.68400000000000005},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":4.4359999999999999,"5min":9.7319999999999993,"15min":9.7319999999999993},"last":3.7789999999999999},{"interface":"front","average":{"1min":1.8700000000000001,"5min":1.026,"15min":0.622},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":7.8470000000000004,"5min":8.3350000000000009,"15min":8.3350000000000009},"last":0.89000000000000001}]},{"osd":4,"last update":"Tue Sep 17 07:06:58 2024","interfaces":[{"interface":"back","average":{"1min":1.54,"5min":0.96799999999999997,"15min":0.69999999999999996},"min":{"1min":0.38400000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":5.4539999999999997,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.63700000000000001},{"interface":"front","average":{"1min":1.167,"5min":0.81100000000000005,"15min":0.629},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":5.7850000000000001,"5min":5.7850000000000001,"15min":5.7850000000000001},"last":0.56399999999999995}]},{"osd":5,"last update":"Tue Sep 17 07:06:23 2024","interfaces":[{"interface":"back","average":{"1min":1.145,"5min":0.93300000000000005,"15min":0.71699999999999997},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":6.508,"5min":7.9669999999999996,"15min":7.9669999999999996},"last":3.746},{"interface":"front","average":{"1min":1.0660000000000001,"5min":0.82299999999999995,"15min":0.66500000000000004},"min":{"1min":0.40500000000000003,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":4.4160000000000004,"5min":4.4160000000000004,"15min":4.4160000000000004},"last":0.66000000000000003}]},{"osd":6,"last update":"Tue Sep 17 07:06:43 2024","interfaces":[{"interface":"back","average":{"1min":1.601,"5min":0.97599999999999998,"15min":0.70199999999999996},"min":{"1min":0.31900000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":5.9199999999999999,"5min":5.9199999999999999,"15min":5.9199999999999999},"last":0.60999999999999999},{"interface":"front","average":{"1min":1.2749999999999999,"5min":1.034,"15min":0.73799999999999999},"min":{"1min":0.376,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":4.3550000000000004,"5min":7.9329999999999998,"15min":7.9329999999999998},"last":3.8010000000000002}]},{"osd":7,"last update":"Tue Sep 17 07:07:04 2024","interfaces":[{"interface":"back","average":{"1min":1.7869999999999999,"5min":1.085,"15min":0.754},"min":{"1min":0.40699999999999997,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":4.5119999999999996,"5min":6.1440000000000001,"15min":6.1440000000000001},"last":0.67800000000000005},{"interface":"front","average":{"1min":1.6559999999999999,"5min":1.0669999999999999,"15min":0.72699999999999998},"min":{"1min":0.39300000000000002,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":5.8170000000000002,"5min":5.8170000000000002,"15min":5.8170000000000002},"last":2.5630000000000002}]}]},{"osd":2,"up_from":18,"seq":77309411417,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":871968,"kb_used_data":656152,"kb_used_omap":1,"kb_used_meta":215678,"kb_avail":92873184,"statfs":{"total":95995035648,"available":95102140416,"internally_reserved":0,"allocated":671899648,"data_stored":671519991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":220854726},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:06:22 2024","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":0.63100000000000001,"15min":0.439},"min":{"1min":0.21099999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":3.5150000000000001,"5min":3.9169999999999998,"15min":3.9169999999999998},"last":7.2709999999999999},{"interface":"front","average":{"1min":2.9460000000000002,"5min":1.1080000000000001,"15min":0.59999999999999998},"min":{"1min":0.47799999999999998,"5min":0.23300000000000001,"15min":0.23100000000000001},"max":{"1min":8.6869999999999994,"5min":8.6869999999999994,"15min":8.6869999999999994},"last":1.0349999999999999}]},{"osd":1,"last update":"Tue Sep 17 07:06:22 2024","interfaces":[{"interface":"back","average":{"1min":1.4359999999999999,"5min":0.70999999999999996,"15min":0.46999999999999997},"min":{"1min":0.35899999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":3.6110000000000002,"5min":5.0030000000000001,"15min":5.0030000000000001},"last":0.53300000000000003},{"interface":"front","average":{"1min":1.008,"5min":0.61799999999999999,"15min":0.45200000000000001},"min":{"1min":0.252,"5min":0.252,"15min":0.251},"max":{"1min":3.4180000000000001,"5min":3.4180000000000001,"15min":3.4180000000000001},"last":1.2609999999999999}]},{"osd":3,"last update":"Tue Sep 17 07:06:47 2024","interfaces":[{"interface":"back","average":{"1min":1.319,"5min":0.746,"15min":0.54000000000000004},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":3.3889999999999998,"5min":3.3889999999999998,"15min":3.3889999999999998},"last":1.212},{"interface":"front","average":{"1min":2.294,"5min":1.129,"15min":0.66300000000000003},"min":{"1min":0.312,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":5.6180000000000003,"5min":5.6180000000000003,"15min":5.6180000000000003},"last":0.28999999999999998}]},{"osd":4,"last update":"Tue Sep 17 07:07:00 2024","interfaces":[{"interface":"back","average":{"1min":1.0609999999999999,"5min":0.877,"15min":0.67000000000000004},"min":{"1min":0.32000000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":2.653,"5min":5.2359999999999998,"15min":5.2359999999999998},"last":0.31900000000000001},{"interface":"front","average":{"1min":1.254,"5min":0.75900000000000001,"15min":0.64500000000000002},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":3.4689999999999999,"5min":3.4689999999999999,"15min":3.4689999999999999},"last":1.1299999999999999}]},{"osd":5,"last update":"Tue Sep 17 07:06:22 2024","interfaces":[{"interface":"back","average":{"1min":1.486,"5min":0.84899999999999998,"15min":0.68000000000000005},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":3.266,"5min":4.484,"15min":4.484},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.746,"15min":0.64600000000000002},"min":{"1min":0.35799999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":3.5529999999999999,"5min":5.3600000000000003,"15min":5.3600000000000003},"last":0.34699999999999998}]},{"osd":6,"last update":"Tue Sep 17 07:06:47 2024","interfaces":[{"interface":"back","average":{"1min":1.5589999999999999,"5min":0.84899999999999998,"15min":0.68500000000000005},"min":{"1min":0.30399999999999999,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":9.2509999999999994,"5min":9.2509999999999994,"15min":9.2509999999999994},"last":1.2889999999999999},{"interface":"front","average":{"1min":1.4850000000000001,"5min":0.96199999999999997,"15min":0.71799999999999997},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":3.6269999999999998,"5min":4.6580000000000004,"15min":4.6580000000000004},"last":0.41399999999999998}]},{"osd":7,"last update":"Tue Sep 17 07:07:17 2024","interfaces":[{"interface":"back","average":{"1min":1.1759999999999999,"5min":0.90500000000000003,"15min":0.68200000000000005},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":2.7090000000000001,"5min":5.3250000000000002,"15min":5.3250000000000002},"last":0.51000000000000001},{"interface":"front","average":{"1min":1.9950000000000001,"5min":1.0129999999999999,"15min":0.85899999999999999},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":9.2919999999999998,"5min":9.2919999999999998,"15min":9.2919999999999998},"last":1.3180000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574943,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":452748,"kb_used_data":327300,"kb_used_omap":1,"kb_used_meta":125438,"kb_avail":93292404,"statfs":{"total":95995035648,"available":95531421696,"internally_reserved":0,"allocated":335155200,"data_stored":334731903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":128448966},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:06:51 2024","interfaces":[{"interface":"back","average":{"1min":1.3149999999999999,"5min":1.099,"15min":0.58999999999999997},"min":{"1min":0.26000000000000001,"5min":0.155,"15min":0.155},"max":{"1min":3.9929999999999999,"5min":13.837,"15min":13.837},"last":0.58199999999999996},{"interface":"front","average":{"1min":2.0600000000000001,"5min":1.1599999999999999,"15min":0.63},"min":{"1min":0.186,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":10.342000000000001,"5min":10.342000000000001,"15min":10.342000000000001},"last":1.794}]},{"osd":2,"last update":"Tue Sep 17 07:07:21 2024","interfaces":[{"interface":"back","average":{"1min":1.8,"5min":1.2150000000000001,"15min":0.67400000000000004},"min":{"1min":0.23799999999999999,"5min":0.192,"15min":0.192},"max":{"1min":5.9509999999999996,"5min":11.055999999999999,"15min":11.055999999999999},"last":0.61499999999999999},{"interface":"front","average":{"1min":1.4470000000000001,"5min":1.302,"15min":0.70999999999999996},"min":{"1min":0.24199999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":5.4269999999999996,"5min":9.3160000000000007,"15min":9.3160000000000007},"last":1.1659999999999999}]},{"osd":3,"last update":"Tue Sep 17 07:06:43 2024","interfaces":[{"interface":"back","average":{"1min":1.579,"5min":1.121,"15min":0.68500000000000005},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":7.2270000000000003,"5min":7.766,"15min":7.766},"last":1.1459999999999999},{"interface":"front","average":{"1min":1.6799999999999999,"5min":1.0860000000000001,"15min":0.66200000000000003},"min":{"1min":0.32900000000000001,"5min":0.222,"15min":0.222},"max":{"1min":4.548,"5min":19.602,"15min":19.602},"last":2.3279999999999998}]},{"osd":4,"last update":"Tue Sep 17 07:07:03 2024","interfaces":[{"interface":"back","average":{"1min":1.149,"5min":1.02,"15min":0.72199999999999998},"min":{"1min":0.29899999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":5.5030000000000001,"5min":7.8840000000000003,"15min":7.8840000000000003},"last":0.79100000000000004},{"interface":"front","average":{"1min":1.363,"5min":0.95199999999999996,"15min":0.69599999999999995},"min":{"1min":0.34200000000000003,"5min":0.307,"15min":0.307},"max":{"1min":7.1749999999999998,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.32000000000000001}]},{"osd":5,"last update":"Tue Sep 17 07:07:21 2024","interfaces":[{"interface":"back","average":{"1min":1.3129999999999999,"5min":1.087,"15min":0.74299999999999999},"min":{"1min":0.31,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":5.5259999999999998,"5min":6.0060000000000002,"15min":6.0060000000000002},"last":1.0509999999999999},{"interface":"front","average":{"1min":1.623,"5min":1.0600000000000001,"15min":0.72999999999999998},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":7.2080000000000002,"5min":8.8889999999999993,"15min":8.8889999999999993},"last":0.28899999999999998}]},{"osd":6,"last update":"Tue Sep 17 07:06:47 2024","interfaces":[{"interface":"back","average":{"1min":1.177,"5min":0.871,"15min":0.65300000000000002},"min":{"1min":0.36699999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":3.1920000000000002,"5min":8.843,"15min":8.843},"last":2.351},{"interface":"front","average":{"1min":1.111,"5min":0.93100000000000005,"15min":0.70799999999999996},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":3.746,"5min":5.7290000000000001,"15min":5.7290000000000001},"last":1.123}]},{"osd":7,"last update":"Tue Sep 17 07:07:07 2024","interfaces":[{"interface":"back","average":{"1min":1.5089999999999999,"5min":1.004,"15min":0.70999999999999996},"min":{"1min":0.38800000000000001,"5min":0.37,"15min":0.37},"max":{"1min":7.2320000000000002,"5min":8.9309999999999992,"15min":8.9309999999999992},"last":2.2930000000000001},{"interface":"front","average":{"1min":1.3120000000000001,"5min":1.0209999999999999,"15min":0.72099999999999997},"min":{"1min":0.372,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":3.5819999999999999,"5min":11.007999999999999,"15min":11.007999999999999},"last":0.64000000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738465,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1079224,"kb_used_data":871724,"kb_used_omap":1,"kb_used_meta":207486,"kb_avail":92665928,"statfs":{"total":95995035648,"available":94889910272,"internally_reserved":0,"allocated":892645376,"data_stored":892246007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":212466118},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,2,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 17 07:06:50 2024","interfaces":[{"interface":"back","average":{"1min":2.2610000000000001,"5min":1.0149999999999999,"15min":0.55700000000000005},"min":{"1min":0.22800000000000001,"5min":0.192,"15min":0.192},"max":{"1min":8.0259999999999998,"5min":8.0259999999999998,"15min":8.0259999999999998},"last":0.52900000000000003},{"interface":"front","average":{"1min":1.8220000000000001,"5min":0.96199999999999997,"15min":0.54800000000000004},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":7.8849999999999998,"5min":7.8849999999999998,"15min":7.8849999999999998},"last":4.3070000000000004}]},{"osd":2,"last update":"Tue Sep 17 07:06:23 2024","interfaces":[{"interface":"back","average":{"1min":3.3599999999999999,"5min":1.4339999999999999,"15min":0.72699999999999998},"min":{"1min":0.313,"5min":0.248,"15min":0.22500000000000001},"max":{"1min":13.621,"5min":16.155000000000001,"15min":16.155000000000001},"last":0.69299999999999995},{"interface":"front","average":{"1min":1.9119999999999999,"5min":1.048,"15min":0.58899999999999997},"min":{"1min":0.24399999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":5.4379999999999997,"5min":14.802,"15min":14.802},"last":1.653}]},{"osd":3,"last update":"Tue Sep 17 07:06:50 2024","interfaces":[{"interface":"back","average":{"1min":3.415,"5min":1.3340000000000001,"15min":0.73999999999999999},"min":{"1min":0.41599999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":18.629000000000001,"5min":18.629000000000001,"15min":18.629000000000001},"last":1.764},{"interface":"front","average":{"1min":3.972,"5min":1.627,"15min":0.83499999999999996},"min":{"1min":0.45600000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":22.358000000000001,"5min":22.358000000000001,"15min":22.358000000000001},"last":0.55200000000000005}]},{"osd":4,"last update":"Tue Sep 17 07:07:02 2024","interfaces":[{"interface":"back","average":{"1min":3.0979999999999999,"5min":1.474,"15min":0.86499999999999999},"min":{"1min":0.45200000000000001,"5min":0.316,"15min":0.316},"max":{"1min":18.568000000000001,"5min":18.568000000000001,"15min":18.568000000000001},"last":0.57399999999999995},{"interface":"front","average":{"1min":2.4380000000000002,"5min":1.2090000000000001,"15min":0.76700000000000002},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":18.765000000000001,"5min":18.765000000000001,"15min":18.765000000000001},"last":0.67100000000000004}]},{"osd":5,"last update":"Tue Sep 17 07:06:28 2024","interfaces":[{"interface":"back","average":{"1min":2.1070000000000002,"5min":0.97199999999999998,"15min":0.68899999999999995},"min":{"1min":0.52500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":5.0940000000000003,"5min":5.0940000000000003,"15min":5.0940000000000003},"last":0.59699999999999998},{"interface":"front","average":{"1min":1.782,"5min":0.94099999999999995,"15min":0.67300000000000004},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":4.1529999999999996,"5min":4.1529999999999996,"15min":4.1529999999999996},"last":0.71299999999999997}]},{"osd":6,"last update":"Tue Sep 17 07:06:50 2024","interfaces":[{"interface":"back","average":{"1min":2.738,"5min":1.1850000000000001,"15min":0.75900000000000001},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":18.602,"5min":18.602,"15min":18.602},"last":0.45600000000000002},{"interface":"front","average":{"1min":3.198,"5min":1.3660000000000001,"15min":0.82099999999999995},"min":{"1min":0.48399999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":18.692,"5min":18.692,"15min":18.692},"last":0.61699999999999999}]},{"osd":7,"last update":"Tue Sep 17 07:07:13 2024","interfaces":[{"interface":"back","average":{"1min":4.6580000000000004,"5min":1.879,"15min":1.427},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":46.183,"5min":46.183,"15min":46.183},"last":0.73599999999999999},{"interface":"front","average":{"1min":4.8620000000000001,"5min":1.6819999999999999,"15min":0.93400000000000005},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":45.914999999999999,"5min":45.914999999999999,"15min":45.914999999999999},"last":0.5}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":848367616,"data_stored":848363544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319229952,"data_stored":319225880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":668139520,"data_stored":668139520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":658898944,"data_stored":658898944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":198246400,"data_stored":198246400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":284426240,"data_stored":284426240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":386600960,"data_stored":386596888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":440008704,"data_stored":440008704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T07:07:27.535 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T07:07:27.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:27 smithi135 ceph-mon[33238]: 6.16 scrub starts 2024-09-17T07:07:27.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:27 smithi135 ceph-mon[33238]: 6.16 scrub ok 2024-09-17T07:07:27.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:27 smithi135 ceph-mon[33238]: 6.13 scrub starts 2024-09-17T07:07:27.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:27 smithi135 ceph-mon[33238]: 6.13 scrub ok 2024-09-17T07:07:27.546 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:27.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:27.547 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:07:27.649 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:07:27.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:27.722 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:27.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:07:27.824 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:07:27.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:27.869 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:27.870 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:07:27.970 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:07:28.003 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:28.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:28.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:07:28.105 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:07:28.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:28.138 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:28.138 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:07:28.239 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:07:28.272 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:28.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:28.274 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:07:28.374 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:07:28.403 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:28.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:28.404 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:07:28.465 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:28 smithi100 ceph-mon[39462]: 4.5 scrub starts 2024-09-17T07:07:28.465 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:28 smithi100 ceph-mon[39462]: 4.5 scrub ok 2024-09-17T07:07:28.465 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:28 smithi100 ceph-mon[39462]: 6.8 scrub starts 2024-09-17T07:07:28.465 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:28 smithi100 ceph-mon[39462]: 6.8 scrub ok 2024-09-17T07:07:28.465 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:28 smithi100 ceph-mon[39462]: pgmap v227: 158 pgs: 158 active+clean; 964 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 794 op/s 2024-09-17T07:07:28.465 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:28 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:07:28.465 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:28 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:07:28.466 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:28 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:07:28.466 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:28 smithi100 ceph-mon[29092]: 4.5 scrub starts 2024-09-17T07:07:28.466 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:28 smithi100 ceph-mon[29092]: 4.5 scrub ok 2024-09-17T07:07:28.466 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:28 smithi100 ceph-mon[29092]: 6.8 scrub starts 2024-09-17T07:07:28.466 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:28 smithi100 ceph-mon[29092]: 6.8 scrub ok 2024-09-17T07:07:28.466 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:28 smithi100 ceph-mon[29092]: pgmap v227: 158 pgs: 158 active+clean; 964 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 794 op/s 2024-09-17T07:07:28.466 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:28 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:07:28.466 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:28 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:07:28.466 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:28 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:07:28.505 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:07:28.530 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:28.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:28.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:07:28.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:28 smithi135 ceph-mon[33238]: 4.5 scrub starts 2024-09-17T07:07:28.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:28 smithi135 ceph-mon[33238]: 4.5 scrub ok 2024-09-17T07:07:28.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:28 smithi135 ceph-mon[33238]: 6.8 scrub starts 2024-09-17T07:07:28.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:28 smithi135 ceph-mon[33238]: 6.8 scrub ok 2024-09-17T07:07:28.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:28 smithi135 ceph-mon[33238]: pgmap v227: 158 pgs: 158 active+clean; 964 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 794 op/s 2024-09-17T07:07:28.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:28 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:07:28.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:28 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:07:28.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:28 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:07:28.545 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:07:28 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:07:28.574 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:07:28.575 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:07:28.631 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:07:28.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:28.673 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:28.673 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:07:28.774 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:07:28.813 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:28.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:28.814 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:07:28.915 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:07:28.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:28.954 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:28.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:07:29.055 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:07:29.087 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:29.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:29.089 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:07:29.189 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:07:29.231 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:29.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:29.232 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:07:29.333 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:07:29.362 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:07:29.362 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:07:29.366 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:29.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:29.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:07:29.402 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1: {} 2024-09-17T07:07:29.402 INFO:teuthology.orchestra.run.smithi100.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:07:29.435 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:07:29.435 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:07:29.441 INFO:teuthology.orchestra.run.smithi135.stdout:ERROR: (22) Invalid argument 2024-09-17T07:07:29.441 INFO:teuthology.orchestra.run.smithi135.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:07:29.467 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:07:29.476 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:07:29.476 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:07:29.508 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:29.509 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:07:29.509 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:07:29.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:29.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:07:29.541 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:07:29.541 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:07:29.542 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:29 smithi100 ceph-mon[29092]: 7.3 deep-scrub ok 2024-09-17T07:07:29.542 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:29 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:07:29.542 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:29 smithi100 ceph-mon[29092]: pgmap v228: 158 pgs: 158 active+clean; 625 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:07:29.542 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:29 smithi100 ceph-mon[39462]: 7.3 deep-scrub ok 2024-09-17T07:07:29.542 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:29 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:07:29.542 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:29 smithi100 ceph-mon[39462]: pgmap v228: 158 pgs: 158 active+clean; 625 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:07:29.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:29 smithi135 ceph-mon[33238]: 7.3 deep-scrub ok 2024-09-17T07:07:29.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:29 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:07:29.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:29 smithi135 ceph-mon[33238]: pgmap v228: 158 pgs: 158 active+clean; 625 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:07:29.572 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:07:29.573 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:07:29.604 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:07:29.604 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:07:29.611 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:07:29.637 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:29.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:29.638 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:07:29.739 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:07:29.780 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:29.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:29.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:07:29.882 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:07:29.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:29.918 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:29.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:07:30.019 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:07:30.061 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:30.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:30.062 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:07:30.163 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:07:30.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:30.201 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:30.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:07:30.302 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:07:30.330 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:30.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:30.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:07:30.432 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:07:30.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:30.476 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:30.477 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:07:30.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:30 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1506390595' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:07:30.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:30 smithi135 ceph-mon[33238]: 2.1b scrub starts 2024-09-17T07:07:30.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:30 smithi135 ceph-mon[33238]: 2.1b scrub ok 2024-09-17T07:07:30.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:30 smithi135 ceph-mon[33238]: 4.8 scrub starts 2024-09-17T07:07:30.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:30 smithi135 ceph-mon[33238]: 4.8 scrub ok 2024-09-17T07:07:30.558 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:30 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1506390595' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:07:30.558 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:30 smithi100 ceph-mon[39462]: 2.1b scrub starts 2024-09-17T07:07:30.559 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:30 smithi100 ceph-mon[39462]: 2.1b scrub ok 2024-09-17T07:07:30.559 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:30 smithi100 ceph-mon[39462]: 4.8 scrub starts 2024-09-17T07:07:30.559 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:30 smithi100 ceph-mon[39462]: 4.8 scrub ok 2024-09-17T07:07:30.559 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:30 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1506390595' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:07:30.559 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:30 smithi100 ceph-mon[29092]: 2.1b scrub starts 2024-09-17T07:07:30.559 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:30 smithi100 ceph-mon[29092]: 2.1b scrub ok 2024-09-17T07:07:30.559 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:30 smithi100 ceph-mon[29092]: 4.8 scrub starts 2024-09-17T07:07:30.559 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:30 smithi100 ceph-mon[29092]: 4.8 scrub ok 2024-09-17T07:07:30.578 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:07:30.606 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:30.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:30.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:07:30.708 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:07:30.744 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:30.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:30.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:07:30.846 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:07:30.891 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:30.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:30.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:07:30.957 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-17T07:07:30.992 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:07:31.033 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:31.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:31.034 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:07:31.135 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:07:31.171 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:31.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:31.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:07:31.257 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:07:31.272 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:07:31.311 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:31.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:31.312 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:07:31.376 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:31 smithi100 ceph-mon[39462]: 6.1d scrub starts 2024-09-17T07:07:31.376 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:31 smithi100 ceph-mon[39462]: 6.1d scrub ok 2024-09-17T07:07:31.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:31 smithi100 ceph-mon[39462]: pgmap v229: 158 pgs: 158 active+clean; 625 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:07:31.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:31 smithi100 ceph-mon[29092]: 6.1d scrub starts 2024-09-17T07:07:31.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:31 smithi100 ceph-mon[29092]: 6.1d scrub ok 2024-09-17T07:07:31.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:31 smithi100 ceph-mon[29092]: pgmap v229: 158 pgs: 158 active+clean; 625 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:07:31.413 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:07:31.450 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:31.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:31.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:07:31.552 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:07:31.557 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:31 smithi135 ceph-mon[33238]: 6.1d scrub starts 2024-09-17T07:07:31.558 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:31 smithi135 ceph-mon[33238]: 6.1d scrub ok 2024-09-17T07:07:31.558 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:31 smithi135 ceph-mon[33238]: pgmap v229: 158 pgs: 158 active+clean; 625 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:07:31.588 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:31.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:31.589 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:07:31.690 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:07:31.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:31.743 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:31.744 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:07:31.845 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:07:31.883 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:31.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:31.884 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:07:31.985 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:07:32.026 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:32.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:32.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:07:32.128 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:07:32.164 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:32.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:32.165 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:07:32.265 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:07:32.298 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:32.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:32.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:07:32.400 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:07:32.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:32.432 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:32.433 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:07:32.448 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:07:32.533 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:07:32.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:32.570 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:32.570 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:07:32.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:32 smithi100 ceph-mon[39462]: 6.1e scrub starts 2024-09-17T07:07:32.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:32 smithi100 ceph-mon[39462]: 6.1e scrub ok 2024-09-17T07:07:32.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:32 smithi100 ceph-mon[39462]: 5.1e scrub starts 2024-09-17T07:07:32.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:32 smithi100 ceph-mon[39462]: 5.1e scrub ok 2024-09-17T07:07:32.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:32 smithi100 ceph-mon[39462]: 2.15 deep-scrub starts 2024-09-17T07:07:32.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:32 smithi100 ceph-mon[39462]: 2.15 deep-scrub ok 2024-09-17T07:07:32.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:32 smithi100 ceph-mon[39462]: 7.0 scrub starts 2024-09-17T07:07:32.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:32 smithi100 ceph-mon[29092]: 6.1e scrub starts 2024-09-17T07:07:32.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:32 smithi100 ceph-mon[29092]: 6.1e scrub ok 2024-09-17T07:07:32.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:32 smithi100 ceph-mon[29092]: 5.1e scrub starts 2024-09-17T07:07:32.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:32 smithi100 ceph-mon[29092]: 5.1e scrub ok 2024-09-17T07:07:32.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:32 smithi100 ceph-mon[29092]: 2.15 deep-scrub starts 2024-09-17T07:07:32.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:32 smithi100 ceph-mon[29092]: 2.15 deep-scrub ok 2024-09-17T07:07:32.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:32 smithi100 ceph-mon[29092]: 7.0 scrub starts 2024-09-17T07:07:32.670 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:07:32.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:32.723 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:32.723 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:07:32.769 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:32 smithi135 ceph-mon[33238]: 6.1e scrub starts 2024-09-17T07:07:32.769 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:32 smithi135 ceph-mon[33238]: 6.1e scrub ok 2024-09-17T07:07:32.769 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:32 smithi135 ceph-mon[33238]: 5.1e scrub starts 2024-09-17T07:07:32.769 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:32 smithi135 ceph-mon[33238]: 5.1e scrub ok 2024-09-17T07:07:32.770 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:32 smithi135 ceph-mon[33238]: 2.15 deep-scrub starts 2024-09-17T07:07:32.770 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:32 smithi135 ceph-mon[33238]: 2.15 deep-scrub ok 2024-09-17T07:07:32.770 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:32 smithi135 ceph-mon[33238]: 7.0 scrub starts 2024-09-17T07:07:32.824 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:07:32.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:32.861 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:32.861 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:07:32.962 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:07:32.996 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:32.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:32.997 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:07:33.098 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:07:33.132 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:33.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:33.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:07:33.233 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:07:33.263 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:33.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:33.264 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:07:33.364 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:07:33.376 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:33 smithi135 ceph-mon[33238]: 5.2 scrub starts 2024-09-17T07:07:33.377 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:33 smithi135 ceph-mon[33238]: 5.2 scrub ok 2024-09-17T07:07:33.377 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:33 smithi135 ceph-mon[33238]: 7.0 scrub ok 2024-09-17T07:07:33.377 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:33 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:07:33.377 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:33 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:07:33.377 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:33 smithi135 ceph-mon[33238]: osdmap e163: 8 total, 8 up, 8 in 2024-09-17T07:07:33.377 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:33 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:07:33.377 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:33 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:07:33.377 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:33 smithi135 ceph-mon[33238]: 5.14 scrub starts 2024-09-17T07:07:33.377 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:33 smithi135 ceph-mon[33238]: 5.14 scrub ok 2024-09-17T07:07:33.377 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:33 smithi135 ceph-mon[33238]: pgmap v231: 158 pgs: 158 active+clean; 589 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 577 B/s rd, 0 op/s 2024-09-17T07:07:33.404 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:33.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:33.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:07:33.415 INFO:teuthology.orchestra.run.smithi135.stdout:{ 2024-09-17T07:07:33.415 INFO:teuthology.orchestra.run.smithi135.stdout: "size": 20, 2024-09-17T07:07:33.415 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 600, 2024-09-17T07:07:33.415 INFO:teuthology.orchestra.run.smithi135.stdout: "ops": [ 2024-09-17T07:07:33.415 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.415 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:54899 7.7 7:e2e5199d:::benchmark_data_smithi100_78556_object54898:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e111)", 2024-09-17T07:07:33.415 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:04:57.339110+0000", 2024-09-17T07:07:33.415 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 156.072732124, 2024-09-17T07:07:33.415 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0186058010000001, 2024-09-17T07:07:33.415 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:07:33.416 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "reached pg", 2024-09-17T07:07:33.416 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:07:33.416 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:07:33.416 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:07:33.416 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 54899 2024-09-17T07:07:33.416 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.416 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:07:33.416 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.416 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:07:33.416 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:57.339110+0000", 2024-09-17T07:07:33.416 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:33.416 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.417 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.417 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:07:33.417 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:57.339110+0000", 2024-09-17T07:07:33.417 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:33.417 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.417 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.417 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:07:33.417 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:57.339108+0000", 2024-09-17T07:07:33.417 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999981 2024-09-17T07:07:33.417 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.417 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.417 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:07:33.418 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:57.339145+0000", 2024-09-17T07:07:33.418 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.6754000000000001e-05 2024-09-17T07:07:33.418 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.418 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.418 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:07:33.418 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:57.339156+0000", 2024-09-17T07:07:33.418 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.115e-05 2024-09-17T07:07:33.418 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.418 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.418 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:07:33.418 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:57.339164+0000", 2024-09-17T07:07:33.418 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.909e-06 2024-09-17T07:07:33.419 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.419 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.419 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:07:33.419 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:57.424983+0000", 2024-09-17T07:07:33.419 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.085818679999999994 2024-09-17T07:07:33.419 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.419 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.419 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:07:33.419 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:57.424994+0000", 2024-09-17T07:07:33.419 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.1432e-05 2024-09-17T07:07:33.419 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.419 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.419 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:07:33.420 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:58.357650+0000", 2024-09-17T07:07:33.420 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.93265618299999997 2024-09-17T07:07:33.420 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.420 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.420 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:07:33.420 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:58.357716+0000", 2024-09-17T07:07:33.420 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.5579000000000004e-05 2024-09-17T07:07:33.420 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:33.420 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:07:33.420 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:33.420 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.420 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.421 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:54910 7.17 7:e939e698:::benchmark_data_smithi100_78556_object54909:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-09-17T07:07:33.421 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:04:57.368094+0000", 2024-09-17T07:07:33.421 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 156.04374808200001, 2024-09-17T07:07:33.421 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0100409610000001, 2024-09-17T07:07:33.421 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:07:33.421 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:07:33.421 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:07:33.421 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:07:33.421 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:07:33.421 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 54910 2024-09-17T07:07:33.421 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.421 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:07:33.421 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.422 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:07:33.422 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:57.368094+0000", 2024-09-17T07:07:33.422 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:33.422 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.422 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.422 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:07:33.422 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:57.368094+0000", 2024-09-17T07:07:33.422 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:33.422 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.422 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.422 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:07:33.422 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:57.368091+0000", 2024-09-17T07:07:33.422 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999967 2024-09-17T07:07:33.422 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.423 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.423 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:07:33.423 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:57.368164+0000", 2024-09-17T07:07:33.423 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.3077999999999994e-05 2024-09-17T07:07:33.423 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.423 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.423 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:07:33.423 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:57.368211+0000", 2024-09-17T07:07:33.423 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.6356999999999997e-05 2024-09-17T07:07:33.423 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.423 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.423 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:07:33.423 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:57.368229+0000", 2024-09-17T07:07:33.423 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.8425999999999999e-05 2024-09-17T07:07:33.424 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.424 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.424 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:07:33.424 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:57.425154+0000", 2024-09-17T07:07:33.424 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.056925396000000003 2024-09-17T07:07:33.424 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.424 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.424 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:07:33.424 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:57.425162+0000", 2024-09-17T07:07:33.424 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.2599999999999999e-06 2024-09-17T07:07:33.424 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.424 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.425 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:07:33.425 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:58.368459+0000", 2024-09-17T07:07:33.425 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.94329758900000005 2024-09-17T07:07:33.425 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.425 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.425 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:07:33.425 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:58.368589+0000", 2024-09-17T07:07:33.425 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00013005100000000001 2024-09-17T07:07:33.425 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.425 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.425 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,3", 2024-09-17T07:07:33.425 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:58.368741+0000", 2024-09-17T07:07:33.425 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00015186299999999999 2024-09-17T07:07:33.426 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.426 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.426 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:07:33.426 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:58.369926+0000", 2024-09-17T07:07:33.426 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0011852900000000001 2024-09-17T07:07:33.426 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.426 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.426 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:07:33.426 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:58.374167+0000", 2024-09-17T07:07:33.426 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0042405339999999998 2024-09-17T07:07:33.426 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.426 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.427 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:07:33.427 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:58.378070+0000", 2024-09-17T07:07:33.427 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0039034460000000001 2024-09-17T07:07:33.427 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.427 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.427 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:07:33.427 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:58.378103+0000", 2024-09-17T07:07:33.427 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.2332999999999997e-05 2024-09-17T07:07:33.427 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.427 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.427 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:07:33.427 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:58.378135+0000", 2024-09-17T07:07:33.427 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.2515000000000003e-05 2024-09-17T07:07:33.428 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:33.428 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:07:33.428 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:33.428 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.428 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.428 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:90793 7.12 7:4bb4ecc8:::benchmark_data_smithi100_78556_object90792:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-17T07:07:33.428 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:05:38.427843+0000", 2024-09-17T07:07:33.428 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 114.983999193, 2024-09-17T07:07:33.428 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.78499230399999997, 2024-09-17T07:07:33.428 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:07:33.428 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:07:33.428 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:07:33.428 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:07:33.429 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:07:33.429 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 90793 2024-09-17T07:07:33.429 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.429 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:07:33.429 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.429 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:07:33.429 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.427843+0000", 2024-09-17T07:07:33.429 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:33.429 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.429 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.429 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:07:33.429 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.427843+0000", 2024-09-17T07:07:33.430 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:33.430 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.430 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.430 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:07:33.431 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.427842+0000", 2024-09-17T07:07:33.431 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.999999 2024-09-17T07:07:33.431 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.431 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.431 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:07:33.431 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.427922+0000", 2024-09-17T07:07:33.431 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.9587999999999998e-05 2024-09-17T07:07:33.431 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.431 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.431 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:07:33.431 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.427932+0000", 2024-09-17T07:07:33.431 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0176999999999999e-05 2024-09-17T07:07:33.431 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.432 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.432 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:07:33.432 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.427939+0000", 2024-09-17T07:07:33.432 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.7229999999999998e-06 2024-09-17T07:07:33.432 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.432 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.432 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:07:33.432 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.428022+0000", 2024-09-17T07:07:33.432 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.2738999999999995e-05 2024-09-17T07:07:33.432 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.432 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.432 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:07:33.432 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.428034+0000", 2024-09-17T07:07:33.433 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.2415999999999999e-05 2024-09-17T07:07:33.433 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.433 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.433 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:07:33.433 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.201180+0000", 2024-09-17T07:07:33.433 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.773146369 2024-09-17T07:07:33.433 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.433 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.433 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:07:33.433 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.201327+0000", 2024-09-17T07:07:33.433 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00014637799999999999 2024-09-17T07:07:33.433 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.434 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.434 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,2", 2024-09-17T07:07:33.434 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.201497+0000", 2024-09-17T07:07:33.434 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000170345 2024-09-17T07:07:33.434 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.434 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.434 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:07:33.434 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.204531+0000", 2024-09-17T07:07:33.434 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0030341790000000001 2024-09-17T07:07:33.434 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.434 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.434 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:07:33.434 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.207437+0000", 2024-09-17T07:07:33.435 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0029059569999999998 2024-09-17T07:07:33.435 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.435 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.435 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:07:33.435 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.212762+0000", 2024-09-17T07:07:33.435 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0053249609999999996 2024-09-17T07:07:33.435 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.435 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.435 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:07:33.435 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.212795+0000", 2024-09-17T07:07:33.435 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.2459999999999998e-05 2024-09-17T07:07:33.435 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.435 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.436 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:07:33.436 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.212835+0000", 2024-09-17T07:07:33.436 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.0763000000000003e-05 2024-09-17T07:07:33.436 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:33.436 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:07:33.436 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:33.436 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.436 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.436 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:90813 7.12 7:49534b43:::benchmark_data_smithi100_78556_object90812:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-17T07:07:33.436 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:05:38.447256+0000", 2024-09-17T07:07:33.436 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 114.964586594, 2024-09-17T07:07:33.436 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.76565891100000005, 2024-09-17T07:07:33.436 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:07:33.437 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:07:33.437 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:07:33.437 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:07:33.437 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:07:33.437 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 90813 2024-09-17T07:07:33.437 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.437 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:07:33.437 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.437 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:07:33.437 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.447256+0000", 2024-09-17T07:07:33.437 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:33.437 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.438 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.438 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:07:33.438 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.447256+0000", 2024-09-17T07:07:33.438 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:33.438 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.438 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.438 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:07:33.438 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.447253+0000", 2024-09-17T07:07:33.438 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999976 2024-09-17T07:07:33.438 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.438 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.438 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:07:33.438 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.447306+0000", 2024-09-17T07:07:33.439 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.2253000000000003e-05 2024-09-17T07:07:33.439 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.439 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.439 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:07:33.439 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.447318+0000", 2024-09-17T07:07:33.439 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.2143999999999999e-05 2024-09-17T07:07:33.439 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.439 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.439 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:07:33.439 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.447323+0000", 2024-09-17T07:07:33.439 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.3430000000000003e-06 2024-09-17T07:07:33.439 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.439 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.440 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:07:33.440 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.447374+0000", 2024-09-17T07:07:33.440 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.0676000000000001e-05 2024-09-17T07:07:33.440 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.440 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.440 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:07:33.440 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.447385+0000", 2024-09-17T07:07:33.440 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.1395e-05 2024-09-17T07:07:33.440 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.440 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.440 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:07:33.440 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.202849+0000", 2024-09-17T07:07:33.441 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.75546397600000004 2024-09-17T07:07:33.441 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.441 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.441 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:07:33.441 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.202930+0000", 2024-09-17T07:07:33.441 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.1080000000000003e-05 2024-09-17T07:07:33.441 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.441 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.441 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,2", 2024-09-17T07:07:33.441 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.203018+0000", 2024-09-17T07:07:33.441 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.7695999999999995e-05 2024-09-17T07:07:33.441 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.442 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.442 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:07:33.442 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.206503+0000", 2024-09-17T07:07:33.442 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.003484816 2024-09-17T07:07:33.442 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.442 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.442 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:07:33.442 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.207478+0000", 2024-09-17T07:07:33.442 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00097474300000000003 2024-09-17T07:07:33.442 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.442 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.442 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:07:33.443 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.212868+0000", 2024-09-17T07:07:33.443 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0053907130000000001 2024-09-17T07:07:33.443 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.443 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.443 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:07:33.443 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.212895+0000", 2024-09-17T07:07:33.443 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.6602e-05 2024-09-17T07:07:33.443 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.443 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.443 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:07:33.443 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.212915+0000", 2024-09-17T07:07:33.443 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.9724999999999998e-05 2024-09-17T07:07:33.443 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:33.444 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:07:33.444 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:33.444 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.444 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.444 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:90860 7.12 7:48288261:::benchmark_data_smithi100_78556_object90859:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-17T07:07:33.444 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:05:38.496803+0000", 2024-09-17T07:07:33.444 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 114.915039194, 2024-09-17T07:07:33.444 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.72173665899999995, 2024-09-17T07:07:33.444 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:07:33.444 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:07:33.444 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:07:33.444 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:07:33.444 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:07:33.445 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 90860 2024-09-17T07:07:33.445 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.445 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:07:33.445 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.445 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:07:33.445 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.496803+0000", 2024-09-17T07:07:33.445 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:33.445 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.445 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.445 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:07:33.445 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.496803+0000", 2024-09-17T07:07:33.445 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:33.445 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.446 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.446 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:07:33.446 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.496802+0000", 2024-09-17T07:07:33.446 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:07:33.446 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.446 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.446 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:07:33.446 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.496851+0000", 2024-09-17T07:07:33.446 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.9397999999999997e-05 2024-09-17T07:07:33.446 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.446 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.446 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:07:33.446 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.496863+0000", 2024-09-17T07:07:33.446 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.1633e-05 2024-09-17T07:07:33.447 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.447 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.447 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:07:33.447 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.496869+0000", 2024-09-17T07:07:33.447 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.8159999999999999e-06 2024-09-17T07:07:33.447 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.447 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.447 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:07:33.447 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.496924+0000", 2024-09-17T07:07:33.447 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.5458999999999999e-05 2024-09-17T07:07:33.447 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.447 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.447 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:07:33.448 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.496935+0000", 2024-09-17T07:07:33.448 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0658e-05 2024-09-17T07:07:33.448 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.448 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.448 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:07:33.448 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.203685+0000", 2024-09-17T07:07:33.448 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.706750604 2024-09-17T07:07:33.448 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.448 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.448 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:07:33.448 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.203780+0000", 2024-09-17T07:07:33.448 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.4462999999999993e-05 2024-09-17T07:07:33.448 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.449 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.449 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,2", 2024-09-17T07:07:33.449 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.203863+0000", 2024-09-17T07:07:33.449 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.3236000000000001e-05 2024-09-17T07:07:33.449 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.449 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.449 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:07:33.449 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.207445+0000", 2024-09-17T07:07:33.449 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0035815589999999998 2024-09-17T07:07:33.449 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.449 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.449 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:07:33.451 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.209736+0000", 2024-09-17T07:07:33.451 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0022916249999999998 2024-09-17T07:07:33.451 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.451 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.451 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:07:33.451 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.218451+0000", 2024-09-17T07:07:33.451 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0087142590000000002 2024-09-17T07:07:33.451 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.451 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.451 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:07:33.451 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.218499+0000", 2024-09-17T07:07:33.451 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.8624000000000001e-05 2024-09-17T07:07:33.452 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.452 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.452 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:07:33.452 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.218540+0000", 2024-09-17T07:07:33.452 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.0633999999999998e-05 2024-09-17T07:07:33.452 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:33.452 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:07:33.452 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:33.452 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.452 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.452 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:90870 7.12 7:4b253461:::benchmark_data_smithi100_78556_object90869:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-17T07:07:33.452 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:05:38.502763+0000", 2024-09-17T07:07:33.452 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 114.90907883200001, 2024-09-17T07:07:33.453 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.723378619, 2024-09-17T07:07:33.453 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:07:33.453 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:07:33.453 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:07:33.453 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:07:33.453 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:07:33.453 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 90870 2024-09-17T07:07:33.453 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.453 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:07:33.453 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.453 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:07:33.453 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.502763+0000", 2024-09-17T07:07:33.453 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:33.454 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.454 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.454 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:07:33.454 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.502763+0000", 2024-09-17T07:07:33.454 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:33.454 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.454 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.454 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:07:33.454 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.502763+0000", 2024-09-17T07:07:33.454 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.999999 2024-09-17T07:07:33.454 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.454 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.454 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:07:33.455 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.502862+0000", 2024-09-17T07:07:33.455 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.9003000000000001e-05 2024-09-17T07:07:33.455 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.455 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.455 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:07:33.455 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.502873+0000", 2024-09-17T07:07:33.455 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0919e-05 2024-09-17T07:07:33.455 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.455 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.455 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:07:33.455 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.502881+0000", 2024-09-17T07:07:33.455 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.3080000000000008e-06 2024-09-17T07:07:33.455 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.456 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.456 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:07:33.456 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.502962+0000", 2024-09-17T07:07:33.456 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.0871000000000005e-05 2024-09-17T07:07:33.456 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.456 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.456 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:07:33.456 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.502971+0000", 2024-09-17T07:07:33.456 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.4930000000000004e-06 2024-09-17T07:07:33.456 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.456 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.456 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:07:33.456 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.204570+0000", 2024-09-17T07:07:33.456 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.70159855100000001 2024-09-17T07:07:33.457 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.457 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.457 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:07:33.457 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.204687+0000", 2024-09-17T07:07:33.457 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00011735000000000001 2024-09-17T07:07:33.457 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.457 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.457 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,2", 2024-09-17T07:07:33.457 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.204770+0000", 2024-09-17T07:07:33.457 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.2836000000000004e-05 2024-09-17T07:07:33.457 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.457 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.458 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:07:33.458 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.207451+0000", 2024-09-17T07:07:33.458 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0026814059999999999 2024-09-17T07:07:33.458 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.458 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.458 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:07:33.458 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.209766+0000", 2024-09-17T07:07:33.458 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00231452 2024-09-17T07:07:33.458 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.458 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.458 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:07:33.458 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.226055+0000", 2024-09-17T07:07:33.458 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.016288830000000001 2024-09-17T07:07:33.458 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.459 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.459 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:07:33.459 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.226104+0000", 2024-09-17T07:07:33.459 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.8875999999999998e-05 2024-09-17T07:07:33.459 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.459 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.459 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:07:33.459 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.226142+0000", 2024-09-17T07:07:33.459 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.8529000000000003e-05 2024-09-17T07:07:33.459 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:33.459 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:07:33.459 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:33.459 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.460 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.460 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:90872 7.12 7:48b9e8b9:::benchmark_data_smithi100_78556_object90871:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-17T07:07:33.460 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:05:38.503420+0000", 2024-09-17T07:07:33.460 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 114.908422599, 2024-09-17T07:07:33.460 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.72282881499999996, 2024-09-17T07:07:33.460 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:07:33.460 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:07:33.460 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:07:33.460 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:07:33.460 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:07:33.460 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 90872 2024-09-17T07:07:33.460 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.460 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:07:33.461 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.461 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:07:33.461 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.503420+0000", 2024-09-17T07:07:33.461 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:33.461 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.461 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.461 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:07:33.461 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.503420+0000", 2024-09-17T07:07:33.461 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:33.461 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.461 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.461 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:07:33.461 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.503419+0000", 2024-09-17T07:07:33.462 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.999999 2024-09-17T07:07:33.462 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.462 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.462 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:07:33.462 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.503493+0000", 2024-09-17T07:07:33.462 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.3919999999999997e-05 2024-09-17T07:07:33.462 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.462 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.462 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:07:33.462 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.503505+0000", 2024-09-17T07:07:33.462 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.2248e-05 2024-09-17T07:07:33.462 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.463 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.463 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:07:33.463 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.503512+0000", 2024-09-17T07:07:33.463 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.9199999999999998e-06 2024-09-17T07:07:33.463 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.463 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.463 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:07:33.463 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.503564+0000", 2024-09-17T07:07:33.463 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.2142000000000003e-05 2024-09-17T07:07:33.463 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.463 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.463 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:07:33.463 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.503572+0000", 2024-09-17T07:07:33.463 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.3720000000000005e-06 2024-09-17T07:07:33.464 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.464 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.464 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:07:33.464 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.205473+0000", 2024-09-17T07:07:33.464 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.70190045599999995 2024-09-17T07:07:33.464 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.464 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.464 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:07:33.464 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.205577+0000", 2024-09-17T07:07:33.464 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00010372100000000001 2024-09-17T07:07:33.464 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.464 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.464 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,2", 2024-09-17T07:07:33.464 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.205712+0000", 2024-09-17T07:07:33.465 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00013573400000000001 2024-09-17T07:07:33.465 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.465 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.465 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:07:33.465 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.208223+0000", 2024-09-17T07:07:33.465 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0025110140000000002 2024-09-17T07:07:33.465 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.465 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.465 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:07:33.465 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.209806+0000", 2024-09-17T07:07:33.465 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0015828330000000001 2024-09-17T07:07:33.465 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.466 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.466 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:07:33.466 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.226184+0000", 2024-09-17T07:07:33.466 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.016377454 2024-09-17T07:07:33.466 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.466 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.466 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:07:33.466 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.226217+0000", 2024-09-17T07:07:33.466 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.3169e-05 2024-09-17T07:07:33.466 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.466 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.466 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:07:33.466 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.226249+0000", 2024-09-17T07:07:33.467 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.1807000000000002e-05 2024-09-17T07:07:33.467 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:33.467 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:07:33.467 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:33.467 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.467 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.467 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:90873 7.12 7:4f7608af:::benchmark_data_smithi100_78556_object90872:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-17T07:07:33.467 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:05:38.505355+0000", 2024-09-17T07:07:33.467 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 114.906487117, 2024-09-17T07:07:33.467 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.72099324600000003, 2024-09-17T07:07:33.467 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:07:33.467 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:07:33.467 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:07:33.468 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:07:33.468 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:07:33.468 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 90873 2024-09-17T07:07:33.468 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.468 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:07:33.468 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.468 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:07:33.468 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.505355+0000", 2024-09-17T07:07:33.468 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:33.468 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.468 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.468 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:07:33.468 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.505355+0000", 2024-09-17T07:07:33.469 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:33.469 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.469 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.469 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:07:33.469 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.505353+0000", 2024-09-17T07:07:33.469 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999981 2024-09-17T07:07:33.469 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.469 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.469 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:07:33.469 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.505426+0000", 2024-09-17T07:07:33.469 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.3030999999999999e-05 2024-09-17T07:07:33.469 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.470 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.470 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:07:33.470 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.505441+0000", 2024-09-17T07:07:33.470 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.4871999999999999e-05 2024-09-17T07:07:33.470 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.470 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.470 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:07:33.470 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.505456+0000", 2024-09-17T07:07:33.470 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.487e-05 2024-09-17T07:07:33.470 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.470 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.470 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:07:33.470 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.505512+0000", 2024-09-17T07:07:33.471 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.5949999999999998e-05 2024-09-17T07:07:33.471 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.471 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.471 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:07:33.471 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.505524+0000", 2024-09-17T07:07:33.471 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.1613e-05 2024-09-17T07:07:33.471 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.471 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.471 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:07:33.471 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.206533+0000", 2024-09-17T07:07:33.471 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.70100958300000005 2024-09-17T07:07:33.471 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.471 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.471 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:07:33.472 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.206627+0000", 2024-09-17T07:07:33.472 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.4192999999999998e-05 2024-09-17T07:07:33.472 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.472 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.472 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,2", 2024-09-17T07:07:33.472 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.206714+0000", 2024-09-17T07:07:33.472 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.6081999999999993e-05 2024-09-17T07:07:33.472 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.473 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.473 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:07:33.473 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.209706+0000", 2024-09-17T07:07:33.473 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0029928770000000001 2024-09-17T07:07:33.473 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.473 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.473 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:07:33.473 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.212641+0000", 2024-09-17T07:07:33.473 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0029341910000000001 2024-09-17T07:07:33.473 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.473 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.473 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:07:33.474 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.226285+0000", 2024-09-17T07:07:33.474 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.013644 2024-09-17T07:07:33.474 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.474 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.474 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:07:33.474 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.226319+0000", 2024-09-17T07:07:33.474 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.4233000000000002e-05 2024-09-17T07:07:33.474 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.474 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.474 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:07:33.474 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.226348+0000", 2024-09-17T07:07:33.474 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.9641999999999998e-05 2024-09-17T07:07:33.474 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:33.475 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:07:33.475 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:33.475 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.475 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.475 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:90894 7.12 7:499604d5:::benchmark_data_smithi100_78556_object90893:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-17T07:07:33.475 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:05:38.528712+0000", 2024-09-17T07:07:33.475 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 114.883130772, 2024-09-17T07:07:33.475 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.69785077299999998, 2024-09-17T07:07:33.476 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:07:33.476 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:07:33.476 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:07:33.476 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:07:33.476 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:07:33.476 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 90894 2024-09-17T07:07:33.476 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.476 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:07:33.477 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.477 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:07:33.477 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.528712+0000", 2024-09-17T07:07:33.477 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:33.477 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.477 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.477 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:07:33.477 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.528712+0000", 2024-09-17T07:07:33.477 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:33.477 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.477 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.477 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:07:33.478 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.528710+0000", 2024-09-17T07:07:33.478 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999981 2024-09-17T07:07:33.478 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.478 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.478 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:07:33.478 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.528748+0000", 2024-09-17T07:07:33.478 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.8711999999999998e-05 2024-09-17T07:07:33.478 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.478 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.478 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:07:33.478 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.528759+0000", 2024-09-17T07:07:33.478 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0056e-05 2024-09-17T07:07:33.478 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.479 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.479 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:07:33.479 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.528765+0000", 2024-09-17T07:07:33.479 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.6359999999999997e-06 2024-09-17T07:07:33.479 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.479 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.479 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:07:33.479 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.528814+0000", 2024-09-17T07:07:33.479 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.8872999999999998e-05 2024-09-17T07:07:33.479 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.479 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.479 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:07:33.479 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.528824+0000", 2024-09-17T07:07:33.480 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0145e-05 2024-09-17T07:07:33.480 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.480 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.480 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:07:33.480 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.207500+0000", 2024-09-17T07:07:33.480 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.67867577700000004 2024-09-17T07:07:33.480 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.480 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.480 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:07:33.480 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.207567+0000", 2024-09-17T07:07:33.480 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.6982000000000003e-05 2024-09-17T07:07:33.480 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.480 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.481 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,2", 2024-09-17T07:07:33.481 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.207644+0000", 2024-09-17T07:07:33.481 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.6905999999999999e-05 2024-09-17T07:07:33.481 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.481 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.481 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:07:33.481 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.209772+0000", 2024-09-17T07:07:33.481 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0021282419999999998 2024-09-17T07:07:33.481 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.481 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.481 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:07:33.481 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.212700+0000", 2024-09-17T07:07:33.481 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0029276889999999998 2024-09-17T07:07:33.482 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.482 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.482 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:07:33.482 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.226515+0000", 2024-09-17T07:07:33.482 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.013815403 2024-09-17T07:07:33.482 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.482 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.482 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:07:33.482 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.226539+0000", 2024-09-17T07:07:33.482 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.4128999999999999e-05 2024-09-17T07:07:33.482 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.482 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.482 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:07:33.483 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.226562+0000", 2024-09-17T07:07:33.483 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.3037999999999999e-05 2024-09-17T07:07:33.483 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:33.483 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:07:33.483 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:33.483 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.483 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.483 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:90928 7.12 7:4920d0db:::benchmark_data_smithi100_78556_object90927:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-17T07:07:33.483 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:05:38.563430+0000", 2024-09-17T07:07:33.483 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 114.84841205799999, 2024-09-17T07:07:33.483 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.66321884600000003, 2024-09-17T07:07:33.483 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:07:33.483 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:07:33.484 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:07:33.484 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:07:33.484 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:07:33.484 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 90928 2024-09-17T07:07:33.484 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.484 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:07:33.484 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.484 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:07:33.484 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.563430+0000", 2024-09-17T07:07:33.484 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:33.484 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.484 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.484 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:07:33.485 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.563430+0000", 2024-09-17T07:07:33.485 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:33.485 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.485 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.485 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:07:33.485 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.563428+0000", 2024-09-17T07:07:33.485 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999981 2024-09-17T07:07:33.485 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.485 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.485 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:07:33.485 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.563483+0000", 2024-09-17T07:07:33.485 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.4409e-05 2024-09-17T07:07:33.485 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.486 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.486 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:07:33.486 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.563493+0000", 2024-09-17T07:07:33.486 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.059e-05 2024-09-17T07:07:33.486 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.486 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.486 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:07:33.486 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.563500+0000", 2024-09-17T07:07:33.486 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.1129999999999997e-06 2024-09-17T07:07:33.486 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.486 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.486 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:07:33.486 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.563563+0000", 2024-09-17T07:07:33.487 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.3647999999999998e-05 2024-09-17T07:07:33.487 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.487 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.487 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:07:33.487 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.563574+0000", 2024-09-17T07:07:33.487 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.1260000000000001e-05 2024-09-17T07:07:33.487 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.487 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.487 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:07:33.487 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.208250+0000", 2024-09-17T07:07:33.487 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.64467564499999996 2024-09-17T07:07:33.487 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.487 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.487 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:07:33.488 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.208335+0000", 2024-09-17T07:07:33.488 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.4809999999999996e-05 2024-09-17T07:07:33.488 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.488 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.488 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,2", 2024-09-17T07:07:33.488 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.208426+0000", 2024-09-17T07:07:33.488 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.0944000000000002e-05 2024-09-17T07:07:33.488 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.488 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.488 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:07:33.488 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.209778+0000", 2024-09-17T07:07:33.488 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0013524609999999999 2024-09-17T07:07:33.488 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.489 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.489 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:07:33.489 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.212731+0000", 2024-09-17T07:07:33.489 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0029530670000000002 2024-09-17T07:07:33.489 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.489 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.489 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:07:33.489 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.226586+0000", 2024-09-17T07:07:33.489 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.013855012 2024-09-17T07:07:33.489 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.489 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.489 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:07:33.489 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.226613+0000", 2024-09-17T07:07:33.489 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.7013e-05 2024-09-17T07:07:33.490 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.490 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.490 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:07:33.490 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.226649+0000", 2024-09-17T07:07:33.490 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.57e-05 2024-09-17T07:07:33.490 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:33.490 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:07:33.490 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:33.490 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.490 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.490 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:90963 7.12 7:4e9f616d:::benchmark_data_smithi100_78556_object90962:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-17T07:07:33.490 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:05:38.599732+0000", 2024-09-17T07:07:33.490 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 114.81211078, 2024-09-17T07:07:33.491 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.636136068, 2024-09-17T07:07:33.491 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:07:33.491 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:07:33.491 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:07:33.491 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:07:33.491 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:07:33.491 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 90963 2024-09-17T07:07:33.491 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.491 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:07:33.491 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.491 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:07:33.491 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.599732+0000", 2024-09-17T07:07:33.491 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:33.492 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.492 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.492 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:07:33.492 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.599732+0000", 2024-09-17T07:07:33.492 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:33.492 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.492 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.492 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:07:33.492 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.599730+0000", 2024-09-17T07:07:33.492 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:07:33.492 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.492 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.492 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:07:33.493 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.599776+0000", 2024-09-17T07:07:33.493 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.5624000000000002e-05 2024-09-17T07:07:33.493 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.493 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.493 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:07:33.493 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.599792+0000", 2024-09-17T07:07:33.493 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.6042999999999999e-05 2024-09-17T07:07:33.493 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.493 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.493 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:07:33.493 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.599800+0000", 2024-09-17T07:07:33.493 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.1769999999999998e-06 2024-09-17T07:07:33.493 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.494 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.494 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:07:33.494 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.599856+0000", 2024-09-17T07:07:33.494 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.6582999999999997e-05 2024-09-17T07:07:33.494 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.494 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.494 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:07:33.494 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.599866+0000", 2024-09-17T07:07:33.494 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.3659999999999998e-06 2024-09-17T07:07:33.494 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.494 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.494 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:07:33.494 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.209005+0000", 2024-09-17T07:07:33.495 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.60913872499999999 2024-09-17T07:07:33.495 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.495 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.495 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:07:33.495 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.209072+0000", 2024-09-17T07:07:33.495 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.7325999999999999e-05 2024-09-17T07:07:33.495 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.495 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.495 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,2", 2024-09-17T07:07:33.495 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.209145+0000", 2024-09-17T07:07:33.495 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.2743000000000005e-05 2024-09-17T07:07:33.495 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.495 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.495 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:07:33.496 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.211831+0000", 2024-09-17T07:07:33.496 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0026865859999999999 2024-09-17T07:07:33.496 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.496 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.496 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:07:33.496 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.212951+0000", 2024-09-17T07:07:33.496 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.001119718 2024-09-17T07:07:33.496 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.496 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.496 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:07:33.496 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.235636+0000", 2024-09-17T07:07:33.496 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.022684708000000001 2024-09-17T07:07:33.496 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.497 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.497 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:07:33.497 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.235822+0000", 2024-09-17T07:07:33.497 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00018626999999999999 2024-09-17T07:07:33.497 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.497 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.497 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:07:33.497 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.235868+0000", 2024-09-17T07:07:33.497 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.5720000000000003e-05 2024-09-17T07:07:33.497 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:33.497 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:07:33.497 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:33.497 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.497 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.498 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:90966 7.12 7:4d208069:::benchmark_data_smithi100_78556_object90965:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-17T07:07:33.498 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:05:38.605986+0000", 2024-09-17T07:07:33.498 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 114.80585615699999, 2024-09-17T07:07:33.498 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.63002641199999998, 2024-09-17T07:07:33.498 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:07:33.498 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:07:33.498 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:07:33.498 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:07:33.498 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:07:33.498 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 90966 2024-09-17T07:07:33.498 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.498 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:07:33.498 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.499 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:07:33.499 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.605986+0000", 2024-09-17T07:07:33.499 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:33.499 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.499 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.499 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:07:33.499 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.605986+0000", 2024-09-17T07:07:33.499 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:33.499 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.499 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.499 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:07:33.499 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.605985+0000", 2024-09-17T07:07:33.499 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:07:33.500 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.500 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.500 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:07:33.500 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.606057+0000", 2024-09-17T07:07:33.500 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.2602999999999996e-05 2024-09-17T07:07:33.500 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.500 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.500 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:07:33.500 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.606071+0000", 2024-09-17T07:07:33.500 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.4341999999999999e-05 2024-09-17T07:07:33.500 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.500 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.500 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:07:33.501 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.606078+0000", 2024-09-17T07:07:33.501 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.5180000000000001e-06 2024-09-17T07:07:33.501 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.501 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.501 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:07:33.501 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.606133+0000", 2024-09-17T07:07:33.501 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.5234e-05 2024-09-17T07:07:33.501 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.501 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.501 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:07:33.501 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.606145+0000", 2024-09-17T07:07:33.501 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.2072000000000001e-05 2024-09-17T07:07:33.501 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.502 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.502 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:07:33.502 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.209827+0000", 2024-09-17T07:07:33.502 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.60368213999999998 2024-09-17T07:07:33.502 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.502 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.502 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:07:33.502 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.209896+0000", 2024-09-17T07:07:33.502 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.8246999999999999e-05 2024-09-17T07:07:33.502 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.502 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.502 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,2", 2024-09-17T07:07:33.502 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.210025+0000", 2024-09-17T07:07:33.502 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00012941999999999999 2024-09-17T07:07:33.503 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.503 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.503 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:07:33.503 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.212650+0000", 2024-09-17T07:07:33.503 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00262496 2024-09-17T07:07:33.503 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.503 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.503 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:07:33.503 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.217625+0000", 2024-09-17T07:07:33.503 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0049752490000000002 2024-09-17T07:07:33.503 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.503 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.503 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:07:33.504 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.235925+0000", 2024-09-17T07:07:33.504 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.018299536000000002 2024-09-17T07:07:33.504 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.504 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.504 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:07:33.504 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.235974+0000", 2024-09-17T07:07:33.504 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.8662999999999998e-05 2024-09-17T07:07:33.504 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.504 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.504 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:07:33.504 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.236013+0000", 2024-09-17T07:07:33.504 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.9050000000000001e-05 2024-09-17T07:07:33.504 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:33.505 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:07:33.505 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:33.505 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.505 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.505 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:91033 7.12 7:4d9e7439:::benchmark_data_smithi100_78556_object91032:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-17T07:07:33.505 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:05:38.703237+0000", 2024-09-17T07:07:33.505 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 114.708604845, 2024-09-17T07:07:33.505 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.53285939400000004, 2024-09-17T07:07:33.505 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:07:33.505 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:07:33.505 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:07:33.505 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:07:33.505 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:07:33.506 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 91033 2024-09-17T07:07:33.506 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.506 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:07:33.506 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.506 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:07:33.506 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.703237+0000", 2024-09-17T07:07:33.506 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:33.506 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.506 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.506 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:07:33.506 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.703237+0000", 2024-09-17T07:07:33.506 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:33.506 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.507 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.507 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:07:33.507 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.703236+0000", 2024-09-17T07:07:33.507 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999981 2024-09-17T07:07:33.507 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.507 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.507 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:07:33.507 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.703290+0000", 2024-09-17T07:07:33.507 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.4296000000000003e-05 2024-09-17T07:07:33.507 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.507 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.507 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:07:33.507 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.703306+0000", 2024-09-17T07:07:33.508 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.5642999999999999e-05 2024-09-17T07:07:33.508 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.508 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.508 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:07:33.508 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.703312+0000", 2024-09-17T07:07:33.508 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.4300000000000003e-06 2024-09-17T07:07:33.508 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.508 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.508 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:07:33.508 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.703362+0000", 2024-09-17T07:07:33.508 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.9867000000000002e-05 2024-09-17T07:07:33.508 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.508 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.509 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:07:33.509 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.703371+0000", 2024-09-17T07:07:33.509 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.9420000000000001e-06 2024-09-17T07:07:33.509 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.509 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.509 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:07:33.509 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.210878+0000", 2024-09-17T07:07:33.509 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.50750747399999996 2024-09-17T07:07:33.509 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.509 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.509 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:07:33.509 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.210980+0000", 2024-09-17T07:07:33.509 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00010117 2024-09-17T07:07:33.510 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.510 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.510 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,2", 2024-09-17T07:07:33.510 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.211100+0000", 2024-09-17T07:07:33.510 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000120254 2024-09-17T07:07:33.510 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.510 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.510 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:07:33.510 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.212738+0000", 2024-09-17T07:07:33.510 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.001638245 2024-09-17T07:07:33.510 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.510 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.511 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:07:33.511 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.217672+0000", 2024-09-17T07:07:33.511 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0049344089999999998 2024-09-17T07:07:33.511 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.511 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.511 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:07:33.511 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.236041+0000", 2024-09-17T07:07:33.511 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.018368922999999999 2024-09-17T07:07:33.511 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.511 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.511 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:07:33.511 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.236077+0000", 2024-09-17T07:07:33.511 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.5235999999999999e-05 2024-09-17T07:07:33.512 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.512 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.512 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:07:33.512 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.236097+0000", 2024-09-17T07:07:33.512 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.0236000000000001e-05 2024-09-17T07:07:33.512 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:33.512 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:07:33.512 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:33.512 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.512 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.512 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:91055 7.12 7:4bab7a93:::benchmark_data_smithi100_78556_object91054:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-17T07:07:33.512 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:05:38.799197+0000", 2024-09-17T07:07:33.512 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 114.612645241, 2024-09-17T07:07:33.513 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.43702168200000002, 2024-09-17T07:07:33.513 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:07:33.513 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:07:33.513 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:07:33.513 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:07:33.513 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:07:33.513 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 91055 2024-09-17T07:07:33.513 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.513 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:07:33.513 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.513 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:07:33.513 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.799197+0000", 2024-09-17T07:07:33.514 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:33.514 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.514 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.514 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:07:33.514 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.799197+0000", 2024-09-17T07:07:33.514 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:33.514 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.514 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.514 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:07:33.514 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.799195+0000", 2024-09-17T07:07:33.514 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999976 2024-09-17T07:07:33.514 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.514 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.514 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:07:33.515 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.799242+0000", 2024-09-17T07:07:33.515 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.6836999999999999e-05 2024-09-17T07:07:33.515 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.515 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.515 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:07:33.515 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.799262+0000", 2024-09-17T07:07:33.515 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.0701000000000001e-05 2024-09-17T07:07:33.515 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.515 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.515 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:07:33.515 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.799272+0000", 2024-09-17T07:07:33.515 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.7280000000000007e-06 2024-09-17T07:07:33.515 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.515 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.516 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:07:33.516 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.799342+0000", 2024-09-17T07:07:33.516 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.9871999999999998e-05 2024-09-17T07:07:33.516 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.516 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.516 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:07:33.516 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.799356+0000", 2024-09-17T07:07:33.516 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.3816e-05 2024-09-17T07:07:33.516 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.516 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.516 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:07:33.516 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.211858+0000", 2024-09-17T07:07:33.516 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.41250220100000001 2024-09-17T07:07:33.517 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.517 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.517 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:07:33.517 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.211924+0000", 2024-09-17T07:07:33.517 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.5924000000000001e-05 2024-09-17T07:07:33.517 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.517 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.517 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,2", 2024-09-17T07:07:33.517 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.212011+0000", 2024-09-17T07:07:33.517 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.7026000000000003e-05 2024-09-17T07:07:33.517 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.518 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.518 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:07:33.518 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.212924+0000", 2024-09-17T07:07:33.518 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00091303 2024-09-17T07:07:33.518 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.518 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.518 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:07:33.518 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.217747+0000", 2024-09-17T07:07:33.518 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0048232259999999999 2024-09-17T07:07:33.518 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.518 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.518 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:07:33.518 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.236140+0000", 2024-09-17T07:07:33.518 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.018392737999999999 2024-09-17T07:07:33.519 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.519 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.519 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:07:33.519 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.236182+0000", 2024-09-17T07:07:33.519 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.2129999999999998e-05 2024-09-17T07:07:33.519 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.519 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.519 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:07:33.519 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:07:33.521 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.236219+0000", 2024-09-17T07:07:33.521 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.667e-05 2024-09-17T07:07:33.521 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:33.521 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:07:33.521 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:33.521 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.521 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.521 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:91058 7.12 7:4ef33ee8:::benchmark_data_smithi100_78556_object91057:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-17T07:07:33.521 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:05:38.802806+0000", 2024-09-17T07:07:33.521 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 114.60903642, 2024-09-17T07:07:33.522 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.43349131200000002, 2024-09-17T07:07:33.522 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:07:33.522 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:07:33.522 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:07:33.522 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:07:33.522 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:07:33.522 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 91058 2024-09-17T07:07:33.522 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.522 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:07:33.522 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.522 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:07:33.522 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.802806+0000", 2024-09-17T07:07:33.523 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:33.523 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.523 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.523 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:07:33.523 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.802806+0000", 2024-09-17T07:07:33.523 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:33.523 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.523 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.523 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:07:33.523 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.802804+0000", 2024-09-17T07:07:33.523 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999981 2024-09-17T07:07:33.523 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.523 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.524 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:07:33.524 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.802859+0000", 2024-09-17T07:07:33.524 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.5124000000000003e-05 2024-09-17T07:07:33.524 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.524 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.524 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:07:33.524 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.802877+0000", 2024-09-17T07:07:33.524 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.8198000000000001e-05 2024-09-17T07:07:33.524 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.524 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.524 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:07:33.524 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.802886+0000", 2024-09-17T07:07:33.525 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.9029999999999993e-06 2024-09-17T07:07:33.525 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.525 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.525 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:07:33.525 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.802943+0000", 2024-09-17T07:07:33.525 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.6873999999999998e-05 2024-09-17T07:07:33.525 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.525 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.525 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:07:33.525 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.802957+0000", 2024-09-17T07:07:33.525 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.3492000000000001e-05 2024-09-17T07:07:33.525 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.525 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.526 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:07:33.526 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.212973+0000", 2024-09-17T07:07:33.526 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.41001628800000001 2024-09-17T07:07:33.526 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.526 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.526 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:07:33.526 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.213043+0000", 2024-09-17T07:07:33.526 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.9663e-05 2024-09-17T07:07:33.526 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.526 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.526 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,2", 2024-09-17T07:07:33.526 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.213122+0000", 2024-09-17T07:07:33.526 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.9629999999999995e-05 2024-09-17T07:07:33.527 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.527 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.527 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:07:33.527 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.217682+0000", 2024-09-17T07:07:33.527 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.004559479 2024-09-17T07:07:33.527 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.527 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.527 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:07:33.527 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.217785+0000", 2024-09-17T07:07:33.527 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000103597 2024-09-17T07:07:33.527 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.527 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.527 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:07:33.528 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.236252+0000", 2024-09-17T07:07:33.528 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.018466372000000002 2024-09-17T07:07:33.528 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.528 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.528 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:07:33.528 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.236277+0000", 2024-09-17T07:07:33.528 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.4828999999999999e-05 2024-09-17T07:07:33.528 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.528 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.528 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:07:33.528 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.236297+0000", 2024-09-17T07:07:33.528 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.0676999999999999e-05 2024-09-17T07:07:33.528 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:33.529 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:07:33.529 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:33.529 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.529 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.529 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:91075 7.12 7:4be87e86:::benchmark_data_smithi100_78556_object91074:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-17T07:07:33.529 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:05:38.831097+0000", 2024-09-17T07:07:33.529 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 114.580745243, 2024-09-17T07:07:33.529 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.40532509100000003, 2024-09-17T07:07:33.529 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:07:33.529 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:07:33.529 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:07:33.529 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:07:33.529 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:07:33.529 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 91075 2024-09-17T07:07:33.530 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.530 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:07:33.530 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.530 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:07:33.530 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.831097+0000", 2024-09-17T07:07:33.530 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:33.530 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.530 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.530 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:07:33.530 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.831097+0000", 2024-09-17T07:07:33.530 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:33.530 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.530 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.531 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:07:33.531 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.831096+0000", 2024-09-17T07:07:33.531 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.999999 2024-09-17T07:07:33.531 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.531 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.531 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:07:33.531 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.831150+0000", 2024-09-17T07:07:33.531 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.4138999999999998e-05 2024-09-17T07:07:33.531 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.531 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.531 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:07:33.531 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.831161+0000", 2024-09-17T07:07:33.532 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.1005000000000001e-05 2024-09-17T07:07:33.532 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.532 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.532 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:07:33.532 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.831166+0000", 2024-09-17T07:07:33.532 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.7060000000000003e-06 2024-09-17T07:07:33.532 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.532 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.532 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:07:33.532 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.831222+0000", 2024-09-17T07:07:33.532 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.5866999999999999e-05 2024-09-17T07:07:33.532 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.532 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.532 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:07:33.533 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.831234+0000", 2024-09-17T07:07:33.533 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.2119e-05 2024-09-17T07:07:33.533 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.533 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.533 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:07:33.533 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.213810+0000", 2024-09-17T07:07:33.533 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.38257537800000002 2024-09-17T07:07:33.533 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.533 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.533 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:07:33.533 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.213967+0000", 2024-09-17T07:07:33.533 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00015702199999999999 2024-09-17T07:07:33.533 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.534 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.534 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,2", 2024-09-17T07:07:33.534 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.214099+0000", 2024-09-17T07:07:33.534 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00013254900000000001 2024-09-17T07:07:33.534 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.534 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.534 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:07:33.534 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.217687+0000", 2024-09-17T07:07:33.534 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.003588141 2024-09-17T07:07:33.534 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.534 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.534 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:07:33.534 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.221425+0000", 2024-09-17T07:07:33.535 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0037375970000000001 2024-09-17T07:07:33.535 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.535 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.535 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:07:33.535 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.236337+0000", 2024-09-17T07:07:33.535 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.01491165 2024-09-17T07:07:33.535 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.535 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.535 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:07:33.535 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.236379+0000", 2024-09-17T07:07:33.535 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.2904999999999997e-05 2024-09-17T07:07:33.535 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.535 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.536 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:07:33.536 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.236422+0000", 2024-09-17T07:07:33.536 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.2760999999999999e-05 2024-09-17T07:07:33.536 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:33.536 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:07:33.536 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:33.536 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.536 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.536 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:91109 7.12 7:4c36141b:::benchmark_data_smithi100_78556_object91108:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-17T07:07:33.536 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:05:38.908421+0000", 2024-09-17T07:07:33.536 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 114.50342114, 2024-09-17T07:07:33.536 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.32807808900000002, 2024-09-17T07:07:33.537 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:07:33.537 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:07:33.537 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:07:33.537 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:07:33.537 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:07:33.537 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 91109 2024-09-17T07:07:33.537 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.537 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:07:33.537 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.537 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:07:33.537 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.908421+0000", 2024-09-17T07:07:33.537 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:33.537 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.537 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.538 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:07:33.538 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.908421+0000", 2024-09-17T07:07:33.538 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:33.538 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.538 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.538 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:07:33.538 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.908420+0000", 2024-09-17T07:07:33.538 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.999999 2024-09-17T07:07:33.538 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.538 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.538 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:07:33.538 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.908466+0000", 2024-09-17T07:07:33.539 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.6315e-05 2024-09-17T07:07:33.539 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.539 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.539 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:07:33.539 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.908477+0000", 2024-09-17T07:07:33.539 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0127e-05 2024-09-17T07:07:33.539 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.539 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.539 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:07:33.539 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.908484+0000", 2024-09-17T07:07:33.539 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.5190000000000003e-06 2024-09-17T07:07:33.539 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.539 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.540 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:07:33.540 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.908533+0000", 2024-09-17T07:07:33.540 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.8971000000000003e-05 2024-09-17T07:07:33.540 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.540 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.540 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:07:33.540 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.908541+0000", 2024-09-17T07:07:33.540 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.7190000000000001e-06 2024-09-17T07:07:33.540 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.540 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.540 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:07:33.540 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.215081+0000", 2024-09-17T07:07:33.541 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.30653980199999997 2024-09-17T07:07:33.541 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.541 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.541 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:07:33.541 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.215194+0000", 2024-09-17T07:07:33.541 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000113596 2024-09-17T07:07:33.541 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.541 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.541 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,2", 2024-09-17T07:07:33.541 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.215318+0000", 2024-09-17T07:07:33.541 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00012414699999999999 2024-09-17T07:07:33.541 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.541 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.542 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:07:33.542 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.217693+0000", 2024-09-17T07:07:33.542 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0023749029999999998 2024-09-17T07:07:33.542 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.542 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.542 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:07:33.542 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.223267+0000", 2024-09-17T07:07:33.542 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00557384 2024-09-17T07:07:33.542 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.542 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.542 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:07:33.542 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.236456+0000", 2024-09-17T07:07:33.542 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.013188942 2024-09-17T07:07:33.542 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.543 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.543 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:07:33.543 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.236479+0000", 2024-09-17T07:07:33.543 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.2785999999999998e-05 2024-09-17T07:07:33.543 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.543 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.543 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:07:33.543 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.236499+0000", 2024-09-17T07:07:33.543 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.0537999999999999e-05 2024-09-17T07:07:33.543 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:33.543 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:07:33.543 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:33.543 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.544 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.544 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:94245 7.7 7:e1cebfa6:::benchmark_data_smithi100_78556_object94244:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-17T07:07:33.544 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:05:42.675680+0000", 2024-09-17T07:07:33.544 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 110.736162724, 2024-09-17T07:07:33.544 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.21869654899999999, 2024-09-17T07:07:33.544 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:07:33.544 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:07:33.544 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:07:33.544 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:07:33.544 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:07:33.544 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 94245 2024-09-17T07:07:33.544 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.544 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:07:33.545 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.545 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:07:33.545 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:42.675680+0000", 2024-09-17T07:07:33.545 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:33.545 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.545 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.545 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:07:33.545 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:42.675680+0000", 2024-09-17T07:07:33.545 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:33.545 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.545 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.545 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:07:33.545 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:42.675678+0000", 2024-09-17T07:07:33.546 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:07:33.546 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.546 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.546 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:07:33.546 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:42.675741+0000", 2024-09-17T07:07:33.546 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.2485999999999998e-05 2024-09-17T07:07:33.546 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.546 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.546 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:07:33.546 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:42.675753+0000", 2024-09-17T07:07:33.546 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.269e-05 2024-09-17T07:07:33.546 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.546 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.546 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:07:33.547 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:42.675761+0000", 2024-09-17T07:07:33.547 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.2690000000000002e-06 2024-09-17T07:07:33.547 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.547 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.547 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:07:33.547 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:42.675812+0000", 2024-09-17T07:07:33.547 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.1041999999999997e-05 2024-09-17T07:07:33.547 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.547 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.547 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:07:33.547 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:42.675893+0000", 2024-09-17T07:07:33.547 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.0958000000000007e-05 2024-09-17T07:07:33.547 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.547 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.548 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,3", 2024-09-17T07:07:33.548 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:42.675975+0000", 2024-09-17T07:07:33.548 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.2306999999999996e-05 2024-09-17T07:07:33.548 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.548 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.548 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:07:33.548 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:42.677435+0000", 2024-09-17T07:07:33.548 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0014597099999999999 2024-09-17T07:07:33.548 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.548 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.548 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:07:33.548 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:42.679293+0000", 2024-09-17T07:07:33.548 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.001858405 2024-09-17T07:07:33.549 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.549 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.549 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:07:33.549 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:42.894281+0000", 2024-09-17T07:07:33.549 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.214988502 2024-09-17T07:07:33.549 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.549 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.549 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:07:33.549 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:42.894335+0000", 2024-09-17T07:07:33.549 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.3786000000000003e-05 2024-09-17T07:07:33.549 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.549 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.549 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:07:33.550 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:42.894376+0000", 2024-09-17T07:07:33.550 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.0914999999999997e-05 2024-09-17T07:07:33.550 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:33.550 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:07:33.550 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:33.550 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.550 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.550 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:94259 7.7 7:e0fabcbf:::benchmark_data_smithi100_78556_object94258:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-17T07:07:33.550 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:05:42.686358+0000", 2024-09-17T07:07:33.550 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 110.725484362, 2024-09-17T07:07:33.550 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.21047728800000001, 2024-09-17T07:07:33.550 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:07:33.550 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:07:33.551 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:07:33.551 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:07:33.551 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:07:33.551 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 94259 2024-09-17T07:07:33.551 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.551 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:07:33.551 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.551 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:07:33.551 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:42.686358+0000", 2024-09-17T07:07:33.551 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:33.551 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.551 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.551 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:07:33.552 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:42.686358+0000", 2024-09-17T07:07:33.552 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:33.552 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.552 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.552 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:07:33.552 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:42.686357+0000", 2024-09-17T07:07:33.552 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:07:33.552 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.552 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.552 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:07:33.552 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:42.686416+0000", 2024-09-17T07:07:33.552 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.9768000000000001e-05 2024-09-17T07:07:33.553 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.553 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.553 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:07:33.553 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:42.686428+0000", 2024-09-17T07:07:33.553 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.1208e-05 2024-09-17T07:07:33.553 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.553 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.553 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:07:33.553 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:42.686433+0000", 2024-09-17T07:07:33.553 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.5400000000000003e-06 2024-09-17T07:07:33.553 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.553 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.554 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:07:33.554 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:42.686472+0000", 2024-09-17T07:07:33.554 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.8516999999999997e-05 2024-09-17T07:07:33.554 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.554 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.554 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:07:33.554 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:42.686565+0000", 2024-09-17T07:07:33.554 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.3184999999999996e-05 2024-09-17T07:07:33.554 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.554 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.554 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,3", 2024-09-17T07:07:33.554 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:42.686663+0000", 2024-09-17T07:07:33.554 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.7617999999999999e-05 2024-09-17T07:07:33.554 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.555 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.555 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:07:33.555 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:42.687928+0000", 2024-09-17T07:07:33.555 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.001265146 2024-09-17T07:07:33.555 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.555 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.555 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:07:33.555 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:42.837296+0000", 2024-09-17T07:07:33.555 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.149368642 2024-09-17T07:07:33.555 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.555 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.555 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:07:33.556 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:42.896752+0000", 2024-09-17T07:07:33.556 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.059455988000000001 2024-09-17T07:07:33.556 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.556 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.556 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:07:33.556 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:42.896802+0000", 2024-09-17T07:07:33.556 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.0074000000000002e-05 2024-09-17T07:07:33.556 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.556 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.556 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:07:33.556 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:42.896835+0000", 2024-09-17T07:07:33.556 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.2904000000000002e-05 2024-09-17T07:07:33.556 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:33.557 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:07:33.557 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:33.557 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.557 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.557 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:94479 7.7 7:e5cde295:::benchmark_data_smithi100_78556_object94478:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-17T07:07:33.557 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:05:43.059491+0000", 2024-09-17T07:07:33.557 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 110.352351743, 2024-09-17T07:07:33.557 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.197441328, 2024-09-17T07:07:33.557 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:07:33.557 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:07:33.557 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:07:33.557 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:07:33.557 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:07:33.557 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 94479 2024-09-17T07:07:33.558 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.558 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:07:33.558 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.558 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:07:33.558 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:43.059491+0000", 2024-09-17T07:07:33.558 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:33.558 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.558 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.558 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:07:33.558 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:43.059491+0000", 2024-09-17T07:07:33.558 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:07:33.558 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.558 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.559 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:07:33.559 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:43.059489+0000", 2024-09-17T07:07:33.559 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:07:33.559 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.559 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.559 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:07:33.559 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:43.059547+0000", 2024-09-17T07:07:33.559 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.8136e-05 2024-09-17T07:07:33.559 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.559 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.559 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:07:33.559 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:43.059562+0000", 2024-09-17T07:07:33.559 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.4932999999999999e-05 2024-09-17T07:07:33.560 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.560 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.560 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:07:33.560 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:43.059570+0000", 2024-09-17T07:07:33.560 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.0190000000000006e-06 2024-09-17T07:07:33.560 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.560 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.560 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:07:33.560 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:43.059621+0000", 2024-09-17T07:07:33.560 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.0488999999999997e-05 2024-09-17T07:07:33.560 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.560 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.560 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:07:33.561 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:43.059732+0000", 2024-09-17T07:07:33.561 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000111722 2024-09-17T07:07:33.561 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.561 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.561 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,3", 2024-09-17T07:07:33.561 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:43.059860+0000", 2024-09-17T07:07:33.561 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00012725599999999999 2024-09-17T07:07:33.561 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.561 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.561 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:07:33.561 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:43.061565+0000", 2024-09-17T07:07:33.561 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.001705834 2024-09-17T07:07:33.561 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.561 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.562 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:07:33.562 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:43.065093+0000", 2024-09-17T07:07:33.562 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0035274249999999998 2024-09-17T07:07:33.562 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.562 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.562 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:07:33.562 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:43.256840+0000", 2024-09-17T07:07:33.562 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.191746839 2024-09-17T07:07:33.562 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.562 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.562 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:07:33.562 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:43.256894+0000", 2024-09-17T07:07:33.562 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.3903999999999998e-05 2024-09-17T07:07:33.563 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:07:33.563 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:07:33.563 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:07:33.563 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:43.256932+0000", 2024-09-17T07:07:33.563 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.8373e-05 2024-09-17T07:07:33.563 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:33.563 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:07:33.563 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:33.563 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:07:33.563 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:07:33.563 INFO:teuthology.orchestra.run.smithi135.stdout:} 2024-09-17T07:07:33.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:33.565 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:33.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:07:33.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:33 smithi100 ceph-mon[29092]: 5.2 scrub starts 2024-09-17T07:07:33.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:33 smithi100 ceph-mon[29092]: 5.2 scrub ok 2024-09-17T07:07:33.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:33 smithi100 ceph-mon[29092]: 7.0 scrub ok 2024-09-17T07:07:33.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:33 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:07:33.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:33 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:07:33.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:33 smithi100 ceph-mon[29092]: osdmap e163: 8 total, 8 up, 8 in 2024-09-17T07:07:33.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:33 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:07:33.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:33 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:07:33.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:33 smithi100 ceph-mon[29092]: 5.14 scrub starts 2024-09-17T07:07:33.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:33 smithi100 ceph-mon[29092]: 5.14 scrub ok 2024-09-17T07:07:33.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:33 smithi100 ceph-mon[29092]: pgmap v231: 158 pgs: 158 active+clean; 589 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 577 B/s rd, 0 op/s 2024-09-17T07:07:33.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:33 smithi100 ceph-mon[39462]: 5.2 scrub starts 2024-09-17T07:07:33.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:33 smithi100 ceph-mon[39462]: 5.2 scrub ok 2024-09-17T07:07:33.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:33 smithi100 ceph-mon[39462]: 7.0 scrub ok 2024-09-17T07:07:33.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:33 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:07:33.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:33 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:07:33.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:33 smithi100 ceph-mon[39462]: osdmap e163: 8 total, 8 up, 8 in 2024-09-17T07:07:33.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:33 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:07:33.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:33 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:07:33.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:33 smithi100 ceph-mon[39462]: 5.14 scrub starts 2024-09-17T07:07:33.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:33 smithi100 ceph-mon[39462]: 5.14 scrub ok 2024-09-17T07:07:33.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:33 smithi100 ceph-mon[39462]: pgmap v231: 158 pgs: 158 active+clean; 589 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 577 B/s rd, 0 op/s 2024-09-17T07:07:33.666 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:07:33.702 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:33.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:33.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:07:33.804 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:07:33.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:33.843 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:33.844 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:07:33.944 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:07:33.982 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:33.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:33.983 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:07:34.084 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:07:34.123 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:34.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:34.124 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:07:34.225 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:07:34.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:34.271 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:34.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:07:34.372 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:07:34.396 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:34 smithi100 ceph-mon[29092]: 7.c scrub starts 2024-09-17T07:07:34.396 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:34 smithi100 ceph-mon[29092]: 7.c scrub ok 2024-09-17T07:07:34.396 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:34 smithi100 ceph-mon[29092]: 6.1b deep-scrub starts 2024-09-17T07:07:34.396 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:34 smithi100 ceph-mon[29092]: 6.1b deep-scrub ok 2024-09-17T07:07:34.396 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:34 smithi100 ceph-mon[29092]: 7.16 scrub starts 2024-09-17T07:07:34.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:34 smithi100 ceph-mon[29092]: 4.1c scrub starts 2024-09-17T07:07:34.397 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:34 smithi100 ceph-mon[29092]: 4.1c scrub ok 2024-09-17T07:07:34.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:34 smithi100 ceph-mon[39462]: 7.c scrub starts 2024-09-17T07:07:34.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:34 smithi100 ceph-mon[39462]: 7.c scrub ok 2024-09-17T07:07:34.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:34 smithi100 ceph-mon[39462]: 6.1b deep-scrub starts 2024-09-17T07:07:34.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:34 smithi100 ceph-mon[39462]: 6.1b deep-scrub ok 2024-09-17T07:07:34.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:34 smithi100 ceph-mon[39462]: 7.16 scrub starts 2024-09-17T07:07:34.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:34 smithi100 ceph-mon[39462]: 4.1c scrub starts 2024-09-17T07:07:34.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:34 smithi100 ceph-mon[39462]: 4.1c scrub ok 2024-09-17T07:07:34.408 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:34 smithi135 ceph-mon[33238]: 7.c scrub starts 2024-09-17T07:07:34.408 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:34 smithi135 ceph-mon[33238]: 7.c scrub ok 2024-09-17T07:07:34.408 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:34 smithi135 ceph-mon[33238]: 6.1b deep-scrub starts 2024-09-17T07:07:34.408 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:34 smithi135 ceph-mon[33238]: 6.1b deep-scrub ok 2024-09-17T07:07:34.408 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:34 smithi135 ceph-mon[33238]: 7.16 scrub starts 2024-09-17T07:07:34.408 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:34 smithi135 ceph-mon[33238]: 4.1c scrub starts 2024-09-17T07:07:34.408 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:34 smithi135 ceph-mon[33238]: 4.1c scrub ok 2024-09-17T07:07:34.413 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-17T07:07:34.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:34.416 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:34.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:07:34.517 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:07:34.555 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:34.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:34.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:07:34.659 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:07:34.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:34.701 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:34.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:07:34.767 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:07:34.802 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:07:34.833 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:34.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:34.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:07:34.910 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T07:07:34.914 INFO:teuthology.orchestra.run.smithi100.stderr:dumped all 2024-09-17T07:07:34.934 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:07:35.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:35.010 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:35.010 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:07:35.111 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:07:35.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:35.144 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:35.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:07:35.198 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:07:35.245 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:07:35.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:35.268 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:35.268 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:07:35.343 INFO:teuthology.orchestra.run.smithi100.stderr:noscrub is set 2024-09-17T07:07:35.369 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:07:35.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:35.407 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:35.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:07:35.508 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:07:35.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:35.542 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:35.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:07:35.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:35 smithi135 ceph-mon[33238]: 7.19 scrub starts 2024-09-17T07:07:35.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:35 smithi135 ceph-mon[33238]: 7.19 scrub ok 2024-09-17T07:07:35.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:35 smithi135 ceph-mon[33238]: 7.16 scrub ok 2024-09-17T07:07:35.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:35 smithi135 ceph-mon[33238]: 6.18 scrub starts 2024-09-17T07:07:35.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:35 smithi135 ceph-mon[33238]: 6.18 scrub ok 2024-09-17T07:07:35.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:35 smithi135 ceph-mon[33238]: from='client.24848 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:07:35.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:35 smithi135 ceph-mon[33238]: pgmap v232: 158 pgs: 158 active+clean; 246 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T07:07:35.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:35 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2543975949' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:07:35.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:35 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:07:35.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:35 smithi100 ceph-mon[29092]: 7.19 scrub starts 2024-09-17T07:07:35.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:35 smithi100 ceph-mon[29092]: 7.19 scrub ok 2024-09-17T07:07:35.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:35 smithi100 ceph-mon[29092]: 7.16 scrub ok 2024-09-17T07:07:35.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:35 smithi100 ceph-mon[29092]: 6.18 scrub starts 2024-09-17T07:07:35.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:35 smithi100 ceph-mon[29092]: 6.18 scrub ok 2024-09-17T07:07:35.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:35 smithi100 ceph-mon[29092]: from='client.24848 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:07:35.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:35 smithi100 ceph-mon[29092]: pgmap v232: 158 pgs: 158 active+clean; 246 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T07:07:35.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:35 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2543975949' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:07:35.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:35 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:07:35.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:35 smithi100 ceph-mon[39462]: 7.19 scrub starts 2024-09-17T07:07:35.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:35 smithi100 ceph-mon[39462]: 7.19 scrub ok 2024-09-17T07:07:35.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:35 smithi100 ceph-mon[39462]: 7.16 scrub ok 2024-09-17T07:07:35.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:35 smithi100 ceph-mon[39462]: 6.18 scrub starts 2024-09-17T07:07:35.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:35 smithi100 ceph-mon[39462]: 6.18 scrub ok 2024-09-17T07:07:35.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:35 smithi100 ceph-mon[39462]: from='client.24848 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:07:35.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:35 smithi100 ceph-mon[39462]: pgmap v232: 158 pgs: 158 active+clean; 246 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T07:07:35.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:35 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2543975949' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:07:35.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:35 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:07:35.643 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:07:35.680 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:35.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:35.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:07:35.781 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:07:35.807 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:35.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:35.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:07:35.909 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:07:35.947 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:35.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:35.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:07:36.050 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:07:36.090 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:36.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:36.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:07:36.091 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:07:35 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:07:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:07:36.130 INFO:teuthology.orchestra.run.smithi100.stdout:{"pg_ready":true,"pg_map":{"version":231,"stamp":"2024-09-17T07:07:33.108168+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":617453870,"num_objects":9611,"num_object_clones":0,"num_object_copies":28833,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9611,"num_whiteouts":0,"num_read":1231,"num_read_kb":1077,"num_write":161531,"num_write_kb":4866236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15842,"num_bytes_recovered":1038681806,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132974,"ondisk_log_size":132974,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":478,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":3313356,"kb_used_data":1994632,"kb_used_omap":12,"kb_used_meta":1318515,"kb_avail":746647860,"statfs":{"total":767960285184,"available":764567408640,"internally_reserved":0,"allocated":2042503168,"data_stored":2039681720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":1350159928},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-393084928,"num_objects":-5998,"num_object_clones":0,"num_object_copies":-17994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5998,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":-6964,"num_write_kb":-443008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.870141"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":110,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.625192+0000","last_change":"2024-09-17T07:04:03.551432+0000","last_active":"2024-09-17T07:07:23.625192+0000","last_peered":"2024-09-17T07:07:23.625192+0000","last_clean":"2024-09-17T07:07:23.625192+0000","last_became_active":"2024-09-17T07:02:18.365773+0000","last_became_peered":"2024-09-17T07:02:18.365773+0000","last_unstale":"2024-09-17T07:07:23.625192+0000","last_undegraded":"2024-09-17T07:07:23.625192+0000","last_fullsized":"2024-09-17T07:07:23.625192+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:03.551314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:04:03.551314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001762752,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":128,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.254413+0000","last_change":"2024-09-17T07:04:56.479564+0000","last_active":"2024-09-17T07:07:23.254413+0000","last_peered":"2024-09-17T07:07:23.254413+0000","last_clean":"2024-09-17T07:07:23.254413+0000","last_became_active":"2024-09-17T07:02:12.006754+0000","last_became_peered":"2024-09-17T07:02:12.006754+0000","last_unstale":"2024-09-17T07:07:23.254413+0000","last_undegraded":"2024-09-17T07:07:23.254413+0000","last_fullsized":"2024-09-17T07:07:23.254413+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:56.479415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:04:56.479415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0038250369999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"75'15","reported_seq":150,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.254716+0000","last_change":"2024-09-17T07:06:29.008173+0000","last_active":"2024-09-17T07:07:23.254716+0000","last_peered":"2024-09-17T07:07:23.254716+0000","last_clean":"2024-09-17T07:07:23.254716+0000","last_became_active":"2024-09-17T07:02:14.012962+0000","last_became_peered":"2024-09-17T07:02:14.012962+0000","last_unstale":"2024-09-17T07:07:23.254716+0000","last_undegraded":"2024-09-17T07:07:23.254716+0000","last_fullsized":"2024-09-17T07:07:23.254716+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-09-17T07:06:29.008025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:06:29.008025+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:47.191492+0000","scrub_duration":0.011622218,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":114,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-17T07:07:24.241234+0000","last_change":"2024-09-17T07:04:39.726884+0000","last_active":"2024-09-17T07:07:24.241234+0000","last_peered":"2024-09-17T07:07:24.241234+0000","last_clean":"2024-09-17T07:07:24.241234+0000","last_became_active":"2024-09-17T07:02:16.014928+0000","last_became_peered":"2024-09-17T07:02:16.014928+0000","last_unstale":"2024-09-17T07:07:24.241234+0000","last_undegraded":"2024-09-17T07:07:24.241234+0000","last_fullsized":"2024-09-17T07:07:24.241234+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:39.726764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:04:39.726764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013379520000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":127,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.246063+0000","last_change":"2024-09-17T07:07:15.767618+0000","last_active":"2024-09-17T07:07:23.246063+0000","last_peered":"2024-09-17T07:07:23.246063+0000","last_clean":"2024-09-17T07:07:23.246063+0000","last_became_active":"2024-09-17T07:02:12.010069+0000","last_became_peered":"2024-09-17T07:02:12.010069+0000","last_unstale":"2024-09-17T07:07:23.246063+0000","last_undegraded":"2024-09-17T07:07:23.246063+0000","last_fullsized":"2024-09-17T07:07:23.246063+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:15.767495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:07:15.767495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:27.964043+0000","scrub_duration":0.013487149,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"75'9","reported_seq":143,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-17T07:07:24.241314+0000","last_change":"2024-09-17T07:04:46.742364+0000","last_active":"2024-09-17T07:07:24.241314+0000","last_peered":"2024-09-17T07:07:24.241314+0000","last_clean":"2024-09-17T07:07:24.241314+0000","last_became_active":"2024-09-17T07:02:14.016244+0000","last_became_peered":"2024-09-17T07:02:14.016244+0000","last_unstale":"2024-09-17T07:07:24.241314+0000","last_undegraded":"2024-09-17T07:07:24.241314+0000","last_fullsized":"2024-09-17T07:07:24.241314+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:04:46.742196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:04:46.742196+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0030184949999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":124,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.591188+0000","last_change":"2024-09-17T07:07:23.591188+0000","last_active":"2024-09-17T07:07:23.591188+0000","last_peered":"2024-09-17T07:07:23.591188+0000","last_clean":"2024-09-17T07:07:23.591188+0000","last_became_active":"2024-09-17T07:02:16.014380+0000","last_became_peered":"2024-09-17T07:02:16.014380+0000","last_unstale":"2024-09-17T07:07:23.591188+0000","last_undegraded":"2024-09-17T07:07:23.591188+0000","last_fullsized":"2024-09-17T07:07:23.591188+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:23.590927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:07:23.590927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:50.614784+0000","scrub_duration":0.0024742240000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":124,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-17T07:07:24.241854+0000","last_change":"2024-09-17T07:07:21.791530+0000","last_active":"2024-09-17T07:07:24.241854+0000","last_peered":"2024-09-17T07:07:24.241854+0000","last_clean":"2024-09-17T07:07:24.241854+0000","last_became_active":"2024-09-17T07:02:18.020790+0000","last_became_peered":"2024-09-17T07:02:18.020790+0000","last_unstale":"2024-09-17T07:07:24.241854+0000","last_undegraded":"2024-09-17T07:07:24.241854+0000","last_fullsized":"2024-09-17T07:07:24.241854+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:21.791399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:07:21.791399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:25.370399+0000","scrub_duration":0.0069986889999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"162'8425","reported_seq":13610,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-17T07:07:28.912160+0000","last_change":"2024-09-17T07:05:31.148987+0000","last_active":"2024-09-17T07:07:28.912160+0000","last_peered":"2024-09-17T07:07:28.912160+0000","last_clean":"2024-09-17T07:07:28.912160+0000","last_became_active":"2024-09-17T07:05:31.144189+0000","last_became_peered":"2024-09-17T07:05:31.144189+0000","last_unstale":"2024-09-17T07:07:28.912160+0000","last_undegraded":"2024-09-17T07:07:28.912160+0000","last_fullsized":"2024-09-17T07:07:28.912160+0000","mapping_epoch":125,"log_start":"116'3572","ondisk_log_start":"116'3572","created":111,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_clean_scrub_stamp":"2024-09-17T07:04:19.785747+0000","objects_scrubbed":0,"log_size":4853,"log_dups_size":0,"ondisk_log_size":4853,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25034752,"num_objects":382,"num_object_clones":0,"num_object_copies":1146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":382,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6714,"num_write_kb":227072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1695,"num_bytes_recovered":111083520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":128,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.258174+0000","last_change":"2024-09-17T07:07:23.013609+0000","last_active":"2024-09-17T07:07:23.258174+0000","last_peered":"2024-09-17T07:07:23.258174+0000","last_clean":"2024-09-17T07:07:23.258174+0000","last_became_active":"2024-09-17T07:02:12.011129+0000","last_became_peered":"2024-09-17T07:02:12.011129+0000","last_unstale":"2024-09-17T07:07:23.258174+0000","last_undegraded":"2024-09-17T07:07:23.258174+0000","last_fullsized":"2024-09-17T07:07:23.258174+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:23.013470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:43.060102+0000","last_clean_scrub_stamp":"2024-09-17T07:07:23.013470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:28.079025+0000","scrub_duration":0.006146281,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"75'5","reported_seq":138,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-17T07:07:24.242435+0000","last_change":"2024-09-17T07:07:15.848170+0000","last_active":"2024-09-17T07:07:24.242435+0000","last_peered":"2024-09-17T07:07:24.242435+0000","last_clean":"2024-09-17T07:07:24.242435+0000","last_became_active":"2024-09-17T07:02:14.016640+0000","last_became_peered":"2024-09-17T07:02:14.016640+0000","last_unstale":"2024-09-17T07:07:24.242435+0000","last_undegraded":"2024-09-17T07:07:24.242435+0000","last_fullsized":"2024-09-17T07:07:24.242435+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-09-17T07:07:15.848071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:07:15.848071+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:31.939834+0000","scrub_duration":0.010596959,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":124,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.257276+0000","last_change":"2024-09-17T07:04:55.853224+0000","last_active":"2024-09-17T07:07:23.257276+0000","last_peered":"2024-09-17T07:07:23.257276+0000","last_clean":"2024-09-17T07:07:23.257276+0000","last_became_active":"2024-09-17T07:02:16.015401+0000","last_became_peered":"2024-09-17T07:02:16.015401+0000","last_unstale":"2024-09-17T07:07:23.257276+0000","last_undegraded":"2024-09-17T07:07:23.257276+0000","last_fullsized":"2024-09-17T07:07:23.257276+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:55.853108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:04:55.853108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017379610000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":133,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-17T07:07:24.247413+0000","last_change":"2024-09-17T07:07:23.739288+0000","last_active":"2024-09-17T07:07:24.247413+0000","last_peered":"2024-09-17T07:07:24.247413+0000","last_clean":"2024-09-17T07:07:24.247413+0000","last_became_active":"2024-09-17T07:02:18.023680+0000","last_became_peered":"2024-09-17T07:02:18.023680+0000","last_unstale":"2024-09-17T07:07:24.247413+0000","last_undegraded":"2024-09-17T07:07:24.247413+0000","last_fullsized":"2024-09-17T07:07:24.247413+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:23.739157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:07:23.739157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:39.559278+0000","scrub_duration":0.004894603,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"162'8037","reported_seq":14825,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-17T07:07:28.013378+0000","last_change":"2024-09-17T07:06:40.353247+0000","last_active":"2024-09-17T07:07:28.013378+0000","last_peered":"2024-09-17T07:07:28.013378+0000","last_clean":"2024-09-17T07:07:28.013378+0000","last_became_active":"2024-09-17T07:06:06.086253+0000","last_became_peered":"2024-09-17T07:06:06.086253+0000","last_unstale":"2024-09-17T07:07:28.013378+0000","last_undegraded":"2024-09-17T07:07:28.013378+0000","last_fullsized":"2024-09-17T07:07:28.013378+0000","mapping_epoch":139,"log_start":"105'1333","ondisk_log_start":"105'1333","created":111,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_clean_scrub_stamp":"2024-09-17T07:04:19.785747+0000","objects_scrubbed":0,"log_size":6704,"log_dups_size":0,"ondisk_log_size":6704,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24707072,"num_objects":377,"num_object_clones":0,"num_object_copies":1131,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6381,"num_write_kb":216256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1671,"num_bytes_recovered":109445120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":128,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.257863+0000","last_change":"2024-09-17T07:07:21.000757+0000","last_active":"2024-09-17T07:07:23.257863+0000","last_peered":"2024-09-17T07:07:23.257863+0000","last_clean":"2024-09-17T07:07:23.257863+0000","last_became_active":"2024-09-17T07:02:12.012234+0000","last_became_peered":"2024-09-17T07:02:12.012234+0000","last_unstale":"2024-09-17T07:07:23.257863+0000","last_undegraded":"2024-09-17T07:07:23.257863+0000","last_fullsized":"2024-09-17T07:07:23.257863+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:21.000635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:45.029247+0000","last_clean_scrub_stamp":"2024-09-17T07:07:21.000635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:23.254719+0000","scrub_duration":0.0068373009999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"75'9","reported_seq":132,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-17T07:07:24.242116+0000","last_change":"2024-09-17T07:04:35.582659+0000","last_active":"2024-09-17T07:07:24.242116+0000","last_peered":"2024-09-17T07:07:24.242116+0000","last_clean":"2024-09-17T07:07:24.242116+0000","last_became_active":"2024-09-17T07:02:14.013337+0000","last_became_peered":"2024-09-17T07:02:14.013337+0000","last_unstale":"2024-09-17T07:07:24.242116+0000","last_undegraded":"2024-09-17T07:07:24.242116+0000","last_fullsized":"2024-09-17T07:07:24.242116+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:04:35.582524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:04:35.582524+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0042430649999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":124,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-17T07:07:24.247978+0000","last_change":"2024-09-17T07:06:34.739225+0000","last_active":"2024-09-17T07:07:24.247978+0000","last_peered":"2024-09-17T07:07:24.247978+0000","last_clean":"2024-09-17T07:07:24.247978+0000","last_became_active":"2024-09-17T07:02:16.023774+0000","last_became_peered":"2024-09-17T07:02:16.023774+0000","last_unstale":"2024-09-17T07:07:24.247978+0000","last_undegraded":"2024-09-17T07:07:24.247978+0000","last_fullsized":"2024-09-17T07:07:24.247978+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:34.739103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:06:34.739103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:47.725471+0000","scrub_duration":0.0024801699999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":111,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.252996+0000","last_change":"2024-09-17T07:04:07.066830+0000","last_active":"2024-09-17T07:07:23.252996+0000","last_peered":"2024-09-17T07:07:23.252996+0000","last_clean":"2024-09-17T07:07:23.252996+0000","last_became_active":"2024-09-17T07:02:18.029178+0000","last_became_peered":"2024-09-17T07:02:18.029178+0000","last_unstale":"2024-09-17T07:07:23.252996+0000","last_undegraded":"2024-09-17T07:07:23.252996+0000","last_fullsized":"2024-09-17T07:07:23.252996+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:07.066751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:04:07.066751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0011842319999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"162'13300","reported_seq":13488,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-17T07:07:28.262105+0000","last_change":"2024-09-17T07:06:29.404750+0000","last_active":"2024-09-17T07:07:28.262105+0000","last_peered":"2024-09-17T07:07:28.262105+0000","last_clean":"2024-09-17T07:07:28.262105+0000","last_became_active":"2024-09-17T07:04:20.804060+0000","last_became_peered":"2024-09-17T07:04:20.804060+0000","last_unstale":"2024-09-17T07:07:28.262105+0000","last_undegraded":"2024-09-17T07:07:28.262105+0000","last_fullsized":"2024-09-17T07:07:28.262105+0000","mapping_epoch":99,"log_start":"142'7600","ondisk_log_start":"142'7600","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"149'8952","last_scrub_stamp":"2024-09-17T07:06:29.404590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_clean_scrub_stamp":"2024-09-17T07:06:29.404590+0000","objects_scrubbed":5038,"log_size":5700,"log_dups_size":0,"ondisk_log_size":5700,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:54.135340+0000","scrub_duration":2.2230415670000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40501248,"num_objects":618,"num_object_clones":0,"num_object_copies":1854,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":618,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13300,"num_write_kb":445376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"95'25","reported_seq":163,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-17T07:07:24.242176+0000","last_change":"2024-09-17T07:06:41.707070+0000","last_active":"2024-09-17T07:07:24.242176+0000","last_peered":"2024-09-17T07:07:24.242176+0000","last_clean":"2024-09-17T07:07:24.242176+0000","last_became_active":"2024-09-17T07:02:14.016270+0000","last_became_peered":"2024-09-17T07:02:14.016270+0000","last_unstale":"2024-09-17T07:07:24.242176+0000","last_undegraded":"2024-09-17T07:07:24.242176+0000","last_fullsized":"2024-09-17T07:07:24.242176+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'25","last_scrub_stamp":"2024-09-17T07:06:41.706948+0000","last_deep_scrub":"95'25","last_deep_scrub_stamp":"2024-09-17T07:04:09.450039+0000","last_clean_scrub_stamp":"2024-09-17T07:06:41.706948+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:03.400798+0000","scrub_duration":0.016404887999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":128,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.257822+0000","last_change":"2024-09-17T07:04:42.771521+0000","last_active":"2024-09-17T07:07:23.257822+0000","last_peered":"2024-09-17T07:07:23.257822+0000","last_clean":"2024-09-17T07:07:23.257822+0000","last_became_active":"2024-09-17T07:02:12.011258+0000","last_became_peered":"2024-09-17T07:02:12.011258+0000","last_unstale":"2024-09-17T07:07:23.257822+0000","last_undegraded":"2024-09-17T07:07:23.257822+0000","last_fullsized":"2024-09-17T07:07:23.257822+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:42.771404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:04:42.771404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017072459999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":124,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-17T07:07:24.620779+0000","last_change":"2024-09-17T07:07:24.620899+0000","last_active":"2024-09-17T07:07:24.620779+0000","last_peered":"2024-09-17T07:07:24.620779+0000","last_clean":"2024-09-17T07:07:24.620779+0000","last_became_active":"2024-09-17T07:02:16.018147+0000","last_became_peered":"2024-09-17T07:02:16.018147+0000","last_unstale":"2024-09-17T07:07:24.620779+0000","last_undegraded":"2024-09-17T07:07:24.620779+0000","last_fullsized":"2024-09-17T07:07:24.620779+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:24.620758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:07:24.620758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:54.505017+0000","scrub_duration":0.0050498469999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":89,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.246325+0000","last_change":"2024-09-17T07:07:16.737235+0000","last_active":"2024-09-17T07:07:23.246325+0000","last_peered":"2024-09-17T07:07:23.246325+0000","last_clean":"2024-09-17T07:07:23.246325+0000","last_became_active":"2024-09-17T07:04:15.113196+0000","last_became_peered":"2024-09-17T07:04:15.113196+0000","last_unstale":"2024-09-17T07:07:23.246325+0000","last_undegraded":"2024-09-17T07:07:23.246325+0000","last_fullsized":"2024-09-17T07:07:23.246325+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:16.737086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:42.984088+0000","last_clean_scrub_stamp":"2024-09-17T07:07:16.737086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:19.824197+0000","scrub_duration":0.0083716439999999993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"162'13266","reported_seq":13336,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-17T07:07:28.949143+0000","last_change":"2024-09-17T07:04:20.800521+0000","last_active":"2024-09-17T07:07:28.949143+0000","last_peered":"2024-09-17T07:07:28.949143+0000","last_clean":"2024-09-17T07:07:28.949143+0000","last_became_active":"2024-09-17T07:04:20.800260+0000","last_became_peered":"2024-09-17T07:04:20.800260+0000","last_unstale":"2024-09-17T07:07:28.949143+0000","last_undegraded":"2024-09-17T07:07:28.949143+0000","last_fullsized":"2024-09-17T07:07:28.949143+0000","mapping_epoch":99,"log_start":"146'8400","ondisk_log_start":"146'8400","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_clean_scrub_stamp":"2024-09-17T07:04:19.785747+0000","objects_scrubbed":0,"log_size":4866,"log_dups_size":0,"ondisk_log_size":4866,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37224448,"num_objects":568,"num_object_clones":0,"num_object_copies":1704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":568,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13266,"num_write_kb":442688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"75'10","reported_seq":142,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-17T07:07:24.242683+0000","last_change":"2024-09-17T07:04:43.722476+0000","last_active":"2024-09-17T07:07:24.242683+0000","last_peered":"2024-09-17T07:07:24.242683+0000","last_clean":"2024-09-17T07:07:24.242683+0000","last_became_active":"2024-09-17T07:02:14.007702+0000","last_became_peered":"2024-09-17T07:02:14.007702+0000","last_unstale":"2024-09-17T07:07:24.242683+0000","last_undegraded":"2024-09-17T07:07:24.242683+0000","last_fullsized":"2024-09-17T07:07:24.242683+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-09-17T07:04:43.722288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:04:43.722288+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002985845,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":127,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.269907+0000","last_change":"2024-09-17T07:06:34.075343+0000","last_active":"2024-09-17T07:07:23.269907+0000","last_peered":"2024-09-17T07:07:23.269907+0000","last_clean":"2024-09-17T07:07:23.269907+0000","last_became_active":"2024-09-17T07:02:12.007553+0000","last_became_peered":"2024-09-17T07:02:12.007553+0000","last_unstale":"2024-09-17T07:07:23.269907+0000","last_undegraded":"2024-09-17T07:07:23.269907+0000","last_fullsized":"2024-09-17T07:07:23.269907+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:34.075214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:06:34.075214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:51.954252+0000","scrub_duration":0.031637767999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"98'9","reported_seq":147,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-17T07:07:24.247487+0000","last_change":"2024-09-17T07:06:34.881845+0000","last_active":"2024-09-17T07:07:24.247487+0000","last_peered":"2024-09-17T07:07:24.247487+0000","last_clean":"2024-09-17T07:07:24.247487+0000","last_became_active":"2024-09-17T07:04:15.110797+0000","last_became_peered":"2024-09-17T07:04:15.110797+0000","last_unstale":"2024-09-17T07:07:24.247487+0000","last_undegraded":"2024-09-17T07:07:24.247487+0000","last_fullsized":"2024-09-17T07:07:24.247487+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-09-17T07:06:34.881734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:06:34.881734+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:54.291547+0000","scrub_duration":0.027541755000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":123,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-17T07:07:24.242823+0000","last_change":"2024-09-17T07:06:42.703538+0000","last_active":"2024-09-17T07:07:24.242823+0000","last_peered":"2024-09-17T07:07:24.242823+0000","last_clean":"2024-09-17T07:07:24.242823+0000","last_became_active":"2024-09-17T07:02:18.017460+0000","last_became_peered":"2024-09-17T07:02:18.017460+0000","last_unstale":"2024-09-17T07:07:24.242823+0000","last_undegraded":"2024-09-17T07:07:24.242823+0000","last_fullsized":"2024-09-17T07:07:24.242823+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:42.703326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:06:42.703326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:48.219915+0000","scrub_duration":0.004198528,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"162'13321","reported_seq":13395,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-17T07:07:28.103888+0000","last_change":"2024-09-17T07:06:22.165610+0000","last_active":"2024-09-17T07:07:28.103888+0000","last_peered":"2024-09-17T07:07:28.103888+0000","last_clean":"2024-09-17T07:07:28.103888+0000","last_became_active":"2024-09-17T07:04:20.804188+0000","last_became_peered":"2024-09-17T07:04:20.804188+0000","last_unstale":"2024-09-17T07:07:28.103888+0000","last_undegraded":"2024-09-17T07:07:28.103888+0000","last_fullsized":"2024-09-17T07:07:28.103888+0000","mapping_epoch":99,"log_start":"142'7600","ondisk_log_start":"142'7600","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_clean_scrub_stamp":"2024-09-17T07:04:19.785747+0000","objects_scrubbed":0,"log_size":5721,"log_dups_size":0,"ondisk_log_size":5721,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41091072,"num_objects":627,"num_object_clones":0,"num_object_copies":1881,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13321,"num_write_kb":446336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"75'15","reported_seq":150,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.270165+0000","last_change":"2024-09-17T07:04:40.954912+0000","last_active":"2024-09-17T07:07:23.270165+0000","last_peered":"2024-09-17T07:07:23.270165+0000","last_clean":"2024-09-17T07:07:23.270165+0000","last_became_active":"2024-09-17T07:02:14.016397+0000","last_became_peered":"2024-09-17T07:02:14.016397+0000","last_unstale":"2024-09-17T07:07:23.270165+0000","last_undegraded":"2024-09-17T07:07:23.270165+0000","last_fullsized":"2024-09-17T07:07:23.270165+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-09-17T07:04:40.954768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:04:40.954768+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0050135199999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":128,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.265278+0000","last_change":"2024-09-17T07:07:18.550881+0000","last_active":"2024-09-17T07:07:23.265278+0000","last_peered":"2024-09-17T07:07:23.265278+0000","last_clean":"2024-09-17T07:07:23.265278+0000","last_became_active":"2024-09-17T07:02:12.007913+0000","last_became_peered":"2024-09-17T07:02:12.007913+0000","last_unstale":"2024-09-17T07:07:23.265278+0000","last_undegraded":"2024-09-17T07:07:23.265278+0000","last_fullsized":"2024-09-17T07:07:23.265278+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:18.550750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:07:18.550750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:47.637615+0000","scrub_duration":0.012153809,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"75'8","reported_seq":186,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-17T07:07:30.578696+0000","last_change":"2024-09-17T07:04:33.544461+0000","last_active":"2024-09-17T07:07:30.578696+0000","last_peered":"2024-09-17T07:07:30.578696+0000","last_clean":"2024-09-17T07:07:30.578696+0000","last_became_active":"2024-09-17T07:02:16.021762+0000","last_became_peered":"2024-09-17T07:02:16.021762+0000","last_unstale":"2024-09-17T07:07:30.578696+0000","last_undegraded":"2024-09-17T07:07:30.578696+0000","last_fullsized":"2024-09-17T07:07:30.578696+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:04:33.544306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:04:33.544306+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0038501080000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":122,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-17T07:07:24.247604+0000","last_change":"2024-09-17T07:06:33.833965+0000","last_active":"2024-09-17T07:07:24.247604+0000","last_peered":"2024-09-17T07:07:24.247604+0000","last_clean":"2024-09-17T07:07:24.247604+0000","last_became_active":"2024-09-17T07:02:18.025332+0000","last_became_peered":"2024-09-17T07:02:18.025332+0000","last_unstale":"2024-09-17T07:07:24.247604+0000","last_undegraded":"2024-09-17T07:07:24.247604+0000","last_fullsized":"2024-09-17T07:07:24.247604+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:33.833851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:06:33.833851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:37.216443+0000","scrub_duration":0.064545922000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"162'13242","reported_seq":13432,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-17T07:07:28.098600+0000","last_change":"2024-09-17T07:06:26.698678+0000","last_active":"2024-09-17T07:07:28.098600+0000","last_peered":"2024-09-17T07:07:28.098600+0000","last_clean":"2024-09-17T07:07:28.098600+0000","last_became_active":"2024-09-17T07:04:20.805282+0000","last_became_peered":"2024-09-17T07:04:20.805282+0000","last_unstale":"2024-09-17T07:07:28.098600+0000","last_undegraded":"2024-09-17T07:07:28.098600+0000","last_fullsized":"2024-09-17T07:07:28.098600+0000","mapping_epoch":99,"log_start":"141'7500","ondisk_log_start":"141'7500","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"148'8698","last_scrub_stamp":"2024-09-17T07:06:26.698194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_clean_scrub_stamp":"2024-09-17T07:06:26.698194+0000","objects_scrubbed":5238,"log_size":5742,"log_dups_size":0,"ondisk_log_size":5742,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:48.672416+0000","scrub_duration":1.569268136,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41222168,"num_objects":630,"num_object_clones":0,"num_object_copies":1890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":630,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13242,"num_write_kb":443841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"75'11","reported_seq":146,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-17T07:07:24.243004+0000","last_change":"2024-09-17T07:06:29.766307+0000","last_active":"2024-09-17T07:07:24.243004+0000","last_peered":"2024-09-17T07:07:24.243004+0000","last_clean":"2024-09-17T07:07:24.243004+0000","last_became_active":"2024-09-17T07:02:14.010227+0000","last_became_peered":"2024-09-17T07:02:14.010227+0000","last_unstale":"2024-09-17T07:07:24.243004+0000","last_undegraded":"2024-09-17T07:07:24.243004+0000","last_fullsized":"2024-09-17T07:07:24.243004+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-09-17T07:06:29.766173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:06:29.766173+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:45.457800+0000","scrub_duration":0.0047797760000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":128,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.257300+0000","last_change":"2024-09-17T07:04:44.722990+0000","last_active":"2024-09-17T07:07:23.257300+0000","last_peered":"2024-09-17T07:07:23.257300+0000","last_clean":"2024-09-17T07:07:23.257300+0000","last_became_active":"2024-09-17T07:02:12.013182+0000","last_became_peered":"2024-09-17T07:02:12.013182+0000","last_unstale":"2024-09-17T07:07:23.257300+0000","last_undegraded":"2024-09-17T07:07:23.257300+0000","last_fullsized":"2024-09-17T07:07:23.257300+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:44.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:04:44.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026545789999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":135,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-17T07:07:24.248495+0000","last_change":"2024-09-17T07:07:14.680573+0000","last_active":"2024-09-17T07:07:24.248495+0000","last_peered":"2024-09-17T07:07:24.248495+0000","last_clean":"2024-09-17T07:07:24.248495+0000","last_became_active":"2024-09-17T07:02:16.020224+0000","last_became_peered":"2024-09-17T07:02:16.020224+0000","last_unstale":"2024-09-17T07:07:24.248495+0000","last_undegraded":"2024-09-17T07:07:24.248495+0000","last_fullsized":"2024-09-17T07:07:24.248495+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:14.680446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:07:14.680446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:32.551262+0000","scrub_duration":0.0023342229999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":121,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.253325+0000","last_change":"2024-09-17T07:06:30.971537+0000","last_active":"2024-09-17T07:07:23.253325+0000","last_peered":"2024-09-17T07:07:23.253325+0000","last_clean":"2024-09-17T07:07:23.253325+0000","last_became_active":"2024-09-17T07:02:18.365000+0000","last_became_peered":"2024-09-17T07:02:18.365000+0000","last_unstale":"2024-09-17T07:07:23.253325+0000","last_undegraded":"2024-09-17T07:07:23.253325+0000","last_fullsized":"2024-09-17T07:07:23.253325+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:30.971383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:06:30.971383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:32.519530+0000","scrub_duration":0.0078901470000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"162'8250","reported_seq":8352,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-17T07:07:28.572226+0000","last_change":"2024-09-17T07:07:23.846967+0000","last_active":"2024-09-17T07:07:28.572226+0000","last_peered":"2024-09-17T07:07:28.572226+0000","last_clean":"2024-09-17T07:07:28.572226+0000","last_became_active":"2024-09-17T07:04:58.369102+0000","last_became_peered":"2024-09-17T07:04:58.369102+0000","last_unstale":"2024-09-17T07:07:28.572226+0000","last_undegraded":"2024-09-17T07:07:28.572226+0000","last_fullsized":"2024-09-17T07:07:28.572226+0000","mapping_epoch":99,"log_start":"118'3763","ondisk_log_start":"118'3763","created":99,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"161'8072","last_scrub_stamp":"2024-09-17T07:07:23.846642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_clean_scrub_stamp":"2024-09-17T07:07:23.846642+0000","objects_scrubbed":499,"log_size":4487,"log_dups_size":0,"ondisk_log_size":4487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:35.452442+0000","scrub_duration":0.15615135599999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21037056,"num_objects":321,"num_object_clones":0,"num_object_copies":963,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":178,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"75'14","reported_seq":137,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-17T07:07:24.241652+0000","last_change":"2024-09-17T07:04:38.693158+0000","last_active":"2024-09-17T07:07:24.241652+0000","last_peered":"2024-09-17T07:07:24.241652+0000","last_clean":"2024-09-17T07:07:24.241652+0000","last_became_active":"2024-09-17T07:02:14.007810+0000","last_became_peered":"2024-09-17T07:02:14.007810+0000","last_unstale":"2024-09-17T07:07:24.241652+0000","last_undegraded":"2024-09-17T07:07:24.241652+0000","last_fullsized":"2024-09-17T07:07:24.241652+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'14","last_scrub_stamp":"2024-09-17T07:04:38.692905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:04:38.692905+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003339,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":138,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.253150+0000","last_change":"2024-09-17T07:06:38.303168+0000","last_active":"2024-09-17T07:07:23.253150+0000","last_peered":"2024-09-17T07:07:23.253150+0000","last_clean":"2024-09-17T07:07:23.253150+0000","last_became_active":"2024-09-17T07:02:12.011766+0000","last_became_peered":"2024-09-17T07:02:12.011766+0000","last_unstale":"2024-09-17T07:07:23.253150+0000","last_undegraded":"2024-09-17T07:07:23.253150+0000","last_fullsized":"2024-09-17T07:07:23.253150+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:38.303033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:06:38.303033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:02.394179+0000","scrub_duration":0.003556221,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"75'8","reported_seq":194,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-17T07:07:24.249910+0000","last_change":"2024-09-17T07:06:29.982260+0000","last_active":"2024-09-17T07:07:24.249910+0000","last_peered":"2024-09-17T07:07:24.249910+0000","last_clean":"2024-09-17T07:07:24.249910+0000","last_became_active":"2024-09-17T07:02:16.020628+0000","last_became_peered":"2024-09-17T07:02:16.020628+0000","last_unstale":"2024-09-17T07:07:24.249910+0000","last_undegraded":"2024-09-17T07:07:24.249910+0000","last_fullsized":"2024-09-17T07:07:24.249910+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:06:29.982141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:06:29.982141+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:53.993523+0000","scrub_duration":0.0052922769999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":122,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.257536+0000","last_change":"2024-09-17T07:04:54.840916+0000","last_active":"2024-09-17T07:07:23.257536+0000","last_peered":"2024-09-17T07:07:23.257536+0000","last_clean":"2024-09-17T07:07:23.257536+0000","last_became_active":"2024-09-17T07:02:18.364981+0000","last_became_peered":"2024-09-17T07:02:18.364981+0000","last_unstale":"2024-09-17T07:07:23.257536+0000","last_undegraded":"2024-09-17T07:07:23.257536+0000","last_fullsized":"2024-09-17T07:07:23.257536+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:54.840843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:04:54.840843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002308619,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"162'8770","reported_seq":8898,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-17T07:07:28.589182+0000","last_change":"2024-09-17T07:07:28.112894+0000","last_active":"2024-09-17T07:07:28.589182+0000","last_peered":"2024-09-17T07:07:28.589182+0000","last_clean":"2024-09-17T07:07:28.589182+0000","last_became_active":"2024-09-17T07:04:58.371716+0000","last_became_peered":"2024-09-17T07:04:58.371716+0000","last_unstale":"2024-09-17T07:07:28.589182+0000","last_undegraded":"2024-09-17T07:07:28.589182+0000","last_fullsized":"2024-09-17T07:07:28.589182+0000","mapping_epoch":99,"log_start":"124'4263","ondisk_log_start":"124'4263","created":99,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"162'8745","last_scrub_stamp":"2024-09-17T07:07:28.112756+0000","last_deep_scrub":"162'8745","last_deep_scrub_stamp":"2024-09-17T07:07:28.112756+0000","last_clean_scrub_stamp":"2024-09-17T07:07:28.112756+0000","objects_scrubbed":383,"log_size":4507,"log_dups_size":0,"ondisk_log_size":4507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:50.906365+0000","scrub_duration":2.4011101880000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20840448,"num_objects":318,"num_object_clones":0,"num_object_copies":954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":65,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"75'18","reported_seq":157,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.251584+0000","last_change":"2024-09-17T07:06:40.794767+0000","last_active":"2024-09-17T07:07:23.251584+0000","last_peered":"2024-09-17T07:07:23.251584+0000","last_clean":"2024-09-17T07:07:23.251584+0000","last_became_active":"2024-09-17T07:02:14.028667+0000","last_became_peered":"2024-09-17T07:02:14.028667+0000","last_unstale":"2024-09-17T07:07:23.251584+0000","last_undegraded":"2024-09-17T07:07:23.251584+0000","last_fullsized":"2024-09-17T07:07:23.251584+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'18","last_scrub_stamp":"2024-09-17T07:06:40.794315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:06:40.794315+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:47.535902+0000","scrub_duration":0.017914407,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":127,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.270395+0000","last_change":"2024-09-17T07:04:45.900865+0000","last_active":"2024-09-17T07:07:23.270395+0000","last_peered":"2024-09-17T07:07:23.270395+0000","last_clean":"2024-09-17T07:07:23.270395+0000","last_became_active":"2024-09-17T07:02:12.005687+0000","last_became_peered":"2024-09-17T07:02:12.005687+0000","last_unstale":"2024-09-17T07:07:23.270395+0000","last_undegraded":"2024-09-17T07:07:23.270395+0000","last_fullsized":"2024-09-17T07:07:23.270395+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:45.900779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:04:45.900779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001545968,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":114,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-17T07:07:24.241580+0000","last_change":"2024-09-17T07:04:01.335116+0000","last_active":"2024-09-17T07:07:24.241580+0000","last_peered":"2024-09-17T07:07:24.241580+0000","last_clean":"2024-09-17T07:07:24.241580+0000","last_became_active":"2024-09-17T07:02:16.019747+0000","last_became_peered":"2024-09-17T07:02:16.019747+0000","last_unstale":"2024-09-17T07:07:24.241580+0000","last_undegraded":"2024-09-17T07:07:24.241580+0000","last_fullsized":"2024-09-17T07:07:24.241580+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:01.335039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:04:01.335039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001538607,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":123,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-17T07:07:24.242045+0000","last_change":"2024-09-17T07:06:40.684796+0000","last_active":"2024-09-17T07:07:24.242045+0000","last_peered":"2024-09-17T07:07:24.242045+0000","last_clean":"2024-09-17T07:07:24.242045+0000","last_became_active":"2024-09-17T07:02:18.024043+0000","last_became_peered":"2024-09-17T07:02:18.024043+0000","last_unstale":"2024-09-17T07:07:24.242045+0000","last_undegraded":"2024-09-17T07:07:24.242045+0000","last_fullsized":"2024-09-17T07:07:24.242045+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:40.684680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:06:40.684680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:41.222854+0000","scrub_duration":0.011253449,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"163'8512","reported_seq":8654,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T07:07:33.037322+0000","last_change":"2024-09-17T07:07:31.880838+0000","last_active":"2024-09-17T07:07:33.037322+0000","last_peered":"2024-09-17T07:07:33.037322+0000","last_clean":"2024-09-17T07:07:33.037322+0000","last_became_active":"2024-09-17T07:04:58.352364+0000","last_became_peered":"2024-09-17T07:04:58.352364+0000","last_unstale":"2024-09-17T07:07:33.037322+0000","last_undegraded":"2024-09-17T07:07:33.037322+0000","last_fullsized":"2024-09-17T07:07:33.037322+0000","mapping_epoch":99,"log_start":"108'2710","ondisk_log_start":"108'2710","created":99,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"162'8477","last_scrub_stamp":"2024-09-17T07:07:31.880634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_clean_scrub_stamp":"2024-09-17T07:07:31.880634+0000","objects_scrubbed":167,"log_size":5802,"log_dups_size":0,"ondisk_log_size":5802,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:54.060330+0000","scrub_duration":0.043444901000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8650752,"num_objects":132,"num_object_clones":0,"num_object_copies":396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5099,"num_write_kb":110720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"75'19","reported_seq":162,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.252265+0000","last_change":"2024-09-17T07:07:17.989796+0000","last_active":"2024-09-17T07:07:23.252265+0000","last_peered":"2024-09-17T07:07:23.252265+0000","last_clean":"2024-09-17T07:07:23.252265+0000","last_became_active":"2024-09-17T07:02:14.027389+0000","last_became_peered":"2024-09-17T07:02:14.027389+0000","last_unstale":"2024-09-17T07:07:23.252265+0000","last_undegraded":"2024-09-17T07:07:23.252265+0000","last_fullsized":"2024-09-17T07:07:23.252265+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-09-17T07:07:17.989435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:07:17.989435+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:38.663377+0000","scrub_duration":0.0043356940000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":128,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.258241+0000","last_change":"2024-09-17T07:06:31.799761+0000","last_active":"2024-09-17T07:07:23.258241+0000","last_peered":"2024-09-17T07:07:23.258241+0000","last_clean":"2024-09-17T07:07:23.258241+0000","last_became_active":"2024-09-17T07:02:12.011917+0000","last_became_peered":"2024-09-17T07:02:12.011917+0000","last_unstale":"2024-09-17T07:07:23.258241+0000","last_undegraded":"2024-09-17T07:07:23.258241+0000","last_fullsized":"2024-09-17T07:07:23.258241+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:31.799640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:06:31.799640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:45.018149+0000","scrub_duration":0.00320703,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":113,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.252531+0000","last_change":"2024-09-17T07:06:21.229365+0000","last_active":"2024-09-17T07:07:23.252531+0000","last_peered":"2024-09-17T07:07:23.252531+0000","last_clean":"2024-09-17T07:07:23.252531+0000","last_became_active":"2024-09-17T07:02:16.016199+0000","last_became_peered":"2024-09-17T07:02:16.016199+0000","last_unstale":"2024-09-17T07:07:23.252531+0000","last_undegraded":"2024-09-17T07:07:23.252531+0000","last_fullsized":"2024-09-17T07:07:23.252531+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:49.489690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:49.489690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0016174729999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":121,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.265024+0000","last_change":"2024-09-17T07:06:30.992080+0000","last_active":"2024-09-17T07:07:23.265024+0000","last_peered":"2024-09-17T07:07:23.265024+0000","last_clean":"2024-09-17T07:07:23.265024+0000","last_became_active":"2024-09-17T07:02:18.364835+0000","last_became_peered":"2024-09-17T07:02:18.364835+0000","last_unstale":"2024-09-17T07:07:23.265024+0000","last_undegraded":"2024-09-17T07:07:23.265024+0000","last_fullsized":"2024-09-17T07:07:23.265024+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:30.991972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:06:30.991972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:53.454623+0000","scrub_duration":0.0045593279999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"163'8631","reported_seq":8779,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T07:07:33.046431+0000","last_change":"2024-09-17T07:06:24.330407+0000","last_active":"2024-09-17T07:07:33.046431+0000","last_peered":"2024-09-17T07:07:33.046431+0000","last_clean":"2024-09-17T07:07:33.046431+0000","last_became_active":"2024-09-17T07:04:58.342518+0000","last_became_peered":"2024-09-17T07:04:58.342518+0000","last_unstale":"2024-09-17T07:07:33.046431+0000","last_undegraded":"2024-09-17T07:07:33.046431+0000","last_fullsized":"2024-09-17T07:07:33.046431+0000","mapping_epoch":99,"log_start":"109'3379","ondisk_log_start":"109'3379","created":99,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'6092","last_scrub_stamp":"2024-09-17T07:06:24.330096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_clean_scrub_stamp":"2024-09-17T07:06:24.330096+0000","objects_scrubbed":2699,"log_size":5252,"log_dups_size":0,"ondisk_log_size":5252,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:39.056718+0000","scrub_duration":0.54396754599999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9568256,"num_objects":146,"num_object_clones":0,"num_object_copies":438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2553,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"110'15","reported_seq":145,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.265224+0000","last_change":"2024-09-17T07:04:57.092268+0000","last_active":"2024-09-17T07:07:23.265224+0000","last_peered":"2024-09-17T07:07:23.265224+0000","last_clean":"2024-09-17T07:07:23.265224+0000","last_became_active":"2024-09-17T07:02:14.010637+0000","last_became_peered":"2024-09-17T07:02:14.010637+0000","last_unstale":"2024-09-17T07:07:23.265224+0000","last_undegraded":"2024-09-17T07:07:23.265224+0000","last_fullsized":"2024-09-17T07:07:23.265224+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'15","last_scrub_stamp":"2024-09-17T07:04:57.091400+0000","last_deep_scrub":"110'15","last_deep_scrub_stamp":"2024-09-17T07:04:57.091400+0000","last_clean_scrub_stamp":"2024-09-17T07:04:57.091400+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012740888000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":127,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.270772+0000","last_change":"2024-09-17T07:04:55.007489+0000","last_active":"2024-09-17T07:07:23.270772+0000","last_peered":"2024-09-17T07:07:23.270772+0000","last_clean":"2024-09-17T07:07:23.270772+0000","last_became_active":"2024-09-17T07:02:12.010803+0000","last_became_peered":"2024-09-17T07:02:12.010803+0000","last_unstale":"2024-09-17T07:07:23.270772+0000","last_undegraded":"2024-09-17T07:07:23.270772+0000","last_fullsized":"2024-09-17T07:07:23.270772+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:55.007317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:25.147878+0000","last_clean_scrub_stamp":"2024-09-17T07:04:55.007317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0085334680000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"89'9","reported_seq":196,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-17T07:07:24.260673+0000","last_change":"2024-09-17T07:04:48.452677+0000","last_active":"2024-09-17T07:07:24.260673+0000","last_peered":"2024-09-17T07:07:24.260673+0000","last_clean":"2024-09-17T07:07:24.260673+0000","last_became_active":"2024-09-17T07:02:16.021360+0000","last_became_peered":"2024-09-17T07:02:16.021360+0000","last_unstale":"2024-09-17T07:07:24.260673+0000","last_undegraded":"2024-09-17T07:07:24.260673+0000","last_fullsized":"2024-09-17T07:07:24.260673+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'9","last_scrub_stamp":"2024-09-17T07:04:48.452558+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-09-17T07:03:29.961439+0000","last_clean_scrub_stamp":"2024-09-17T07:04:48.452558+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0057192279999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"75'1","reported_seq":123,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.252854+0000","last_change":"2024-09-17T07:06:25.992349+0000","last_active":"2024-09-17T07:07:23.252854+0000","last_peered":"2024-09-17T07:07:23.252854+0000","last_clean":"2024-09-17T07:07:23.252854+0000","last_became_active":"2024-09-17T07:02:18.029772+0000","last_became_peered":"2024-09-17T07:02:18.029772+0000","last_unstale":"2024-09-17T07:07:23.252854+0000","last_undegraded":"2024-09-17T07:07:23.252854+0000","last_fullsized":"2024-09-17T07:07:23.252854+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-17T07:06:25.992208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:06:25.992208+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:41.167851+0000","scrub_duration":0.010034507,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"162'8356","reported_seq":8531,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-17T07:07:28.061216+0000","last_change":"2024-09-17T07:06:46.756075+0000","last_active":"2024-09-17T07:07:28.061216+0000","last_peered":"2024-09-17T07:07:28.061216+0000","last_clean":"2024-09-17T07:07:28.061216+0000","last_became_active":"2024-09-17T07:04:58.356122+0000","last_became_peered":"2024-09-17T07:04:58.356122+0000","last_unstale":"2024-09-17T07:07:28.061216+0000","last_undegraded":"2024-09-17T07:07:28.061216+0000","last_fullsized":"2024-09-17T07:07:28.061216+0000","mapping_epoch":99,"log_start":"105'1916","ondisk_log_start":"105'1916","created":99,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'6802","last_scrub_stamp":"2024-09-17T07:06:46.755822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_clean_scrub_stamp":"2024-09-17T07:06:46.755822+0000","objects_scrubbed":1892,"log_size":6440,"log_dups_size":0,"ondisk_log_size":6440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:58.085658+0000","scrub_duration":0.46200559400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21495808,"num_objects":328,"num_object_clones":0,"num_object_copies":984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4865,"num_write_kb":110656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"75'28","reported_seq":236,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-17T07:07:24.243446+0000","last_change":"2024-09-17T07:06:45.186860+0000","last_active":"2024-09-17T07:07:24.243446+0000","last_peered":"2024-09-17T07:07:24.243446+0000","last_clean":"2024-09-17T07:07:24.243446+0000","last_became_active":"2024-09-17T07:02:14.017377+0000","last_became_peered":"2024-09-17T07:02:14.017377+0000","last_unstale":"2024-09-17T07:07:24.243446+0000","last_undegraded":"2024-09-17T07:07:24.243446+0000","last_fullsized":"2024-09-17T07:07:24.243446+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'28","last_scrub_stamp":"2024-09-17T07:06:45.186408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:06:45.186408+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:03.935330+0000","scrub_duration":0.010603473,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":139,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-17T07:07:24.248665+0000","last_change":"2024-09-17T07:06:43.763211+0000","last_active":"2024-09-17T07:07:24.248665+0000","last_peered":"2024-09-17T07:07:24.248665+0000","last_clean":"2024-09-17T07:07:24.248665+0000","last_became_active":"2024-09-17T07:02:12.008802+0000","last_became_peered":"2024-09-17T07:02:12.008802+0000","last_unstale":"2024-09-17T07:07:24.248665+0000","last_undegraded":"2024-09-17T07:07:24.248665+0000","last_fullsized":"2024-09-17T07:07:24.248665+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:43.763044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:06:43.763044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:04.613175+0000","scrub_duration":0.003861303,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":124,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.253871+0000","last_change":"2024-09-17T07:04:39.385133+0000","last_active":"2024-09-17T07:07:23.253871+0000","last_peered":"2024-09-17T07:07:23.253871+0000","last_clean":"2024-09-17T07:07:23.253871+0000","last_became_active":"2024-09-17T07:02:16.016606+0000","last_became_peered":"2024-09-17T07:02:16.016606+0000","last_unstale":"2024-09-17T07:07:23.253871+0000","last_undegraded":"2024-09-17T07:07:23.253871+0000","last_fullsized":"2024-09-17T07:07:23.253871+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:39.384962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:04:39.384962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0055010839999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":110,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.245829+0000","last_change":"2024-09-17T07:04:04.564851+0000","last_active":"2024-09-17T07:07:23.245829+0000","last_peered":"2024-09-17T07:07:23.245829+0000","last_clean":"2024-09-17T07:07:23.245829+0000","last_became_active":"2024-09-17T07:02:18.024602+0000","last_became_peered":"2024-09-17T07:02:18.024602+0000","last_unstale":"2024-09-17T07:07:23.245829+0000","last_undegraded":"2024-09-17T07:07:23.245829+0000","last_fullsized":"2024-09-17T07:07:23.245829+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:04.564765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:04:04.564765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001317206,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"162'8490","reported_seq":8600,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-17T07:07:28.897732+0000","last_change":"2024-09-17T07:07:16.248588+0000","last_active":"2024-09-17T07:07:28.897732+0000","last_peered":"2024-09-17T07:07:28.897732+0000","last_clean":"2024-09-17T07:07:28.897732+0000","last_became_active":"2024-09-17T07:04:58.345101+0000","last_became_peered":"2024-09-17T07:04:58.345101+0000","last_unstale":"2024-09-17T07:07:28.897732+0000","last_undegraded":"2024-09-17T07:07:28.897732+0000","last_fullsized":"2024-09-17T07:07:28.897732+0000","mapping_epoch":99,"log_start":"116'3583","ondisk_log_start":"116'3583","created":99,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"159'7976","last_scrub_stamp":"2024-09-17T07:07:16.248093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_clean_scrub_stamp":"2024-09-17T07:07:16.248093+0000","objects_scrubbed":800,"log_size":4907,"log_dups_size":0,"ondisk_log_size":4907,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:45.018010+0000","scrub_duration":0.257620882,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18481152,"num_objects":282,"num_object_clones":0,"num_object_copies":846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":282,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":518,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"75'13","reported_seq":152,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.270244+0000","last_change":"2024-09-17T07:04:58.081870+0000","last_active":"2024-09-17T07:07:23.270244+0000","last_peered":"2024-09-17T07:07:23.270244+0000","last_clean":"2024-09-17T07:07:23.270244+0000","last_became_active":"2024-09-17T07:02:14.019838+0000","last_became_peered":"2024-09-17T07:02:14.019838+0000","last_unstale":"2024-09-17T07:07:23.270244+0000","last_undegraded":"2024-09-17T07:07:23.270244+0000","last_fullsized":"2024-09-17T07:07:23.270244+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'13","last_scrub_stamp":"2024-09-17T07:04:58.081771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:04:58.081771+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001806263,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"76'5","reported_seq":444,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-17T07:07:26.560771+0000","last_change":"2024-09-17T07:06:46.230250+0000","last_active":"2024-09-17T07:07:26.560771+0000","last_peered":"2024-09-17T07:07:26.560771+0000","last_clean":"2024-09-17T07:07:26.560771+0000","last_became_active":"2024-09-17T07:02:12.006883+0000","last_became_peered":"2024-09-17T07:02:12.006883+0000","last_unstale":"2024-09-17T07:07:26.560771+0000","last_undegraded":"2024-09-17T07:07:26.560771+0000","last_fullsized":"2024-09-17T07:07:26.560771+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'5","last_scrub_stamp":"2024-09-17T07:06:46.229804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:06:46.229804+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:06.428152+0000","scrub_duration":0.064119354000000003,"objects_trimmed":0,"snaptrim_duration":0.00033739799999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":309,"num_read_kb":304,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":88,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.254802+0000","last_change":"2024-09-17T07:04:53.698772+0000","last_active":"2024-09-17T07:07:23.254802+0000","last_peered":"2024-09-17T07:07:23.254802+0000","last_clean":"2024-09-17T07:07:23.254802+0000","last_became_active":"2024-09-17T07:04:15.109950+0000","last_became_peered":"2024-09-17T07:04:15.109950+0000","last_unstale":"2024-09-17T07:07:23.254802+0000","last_undegraded":"2024-09-17T07:07:23.254802+0000","last_fullsized":"2024-09-17T07:07:23.254802+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:53.698301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:04:53.698301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045574200000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":124,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.270051+0000","last_change":"2024-09-17T07:06:25.185133+0000","last_active":"2024-09-17T07:07:23.270051+0000","last_peered":"2024-09-17T07:07:23.270051+0000","last_clean":"2024-09-17T07:07:23.270051+0000","last_became_active":"2024-09-17T07:02:16.014492+0000","last_became_peered":"2024-09-17T07:02:16.014492+0000","last_unstale":"2024-09-17T07:07:23.270051+0000","last_undegraded":"2024-09-17T07:07:23.270051+0000","last_fullsized":"2024-09-17T07:07:23.270051+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:25.184907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:06:25.184907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:37.815314+0000","scrub_duration":0.0051503060000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":122,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.257992+0000","last_change":"2024-09-17T07:04:53.836533+0000","last_active":"2024-09-17T07:07:23.257992+0000","last_peered":"2024-09-17T07:07:23.257992+0000","last_clean":"2024-09-17T07:07:23.257992+0000","last_became_active":"2024-09-17T07:02:18.365369+0000","last_became_peered":"2024-09-17T07:02:18.365369+0000","last_unstale":"2024-09-17T07:07:23.257992+0000","last_undegraded":"2024-09-17T07:07:23.257992+0000","last_fullsized":"2024-09-17T07:07:23.257992+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:53.836372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:04:53.836372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0024095980000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"162'10352","reported_seq":10607,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-17T07:07:28.206619+0000","last_change":"2024-09-17T07:06:49.883933+0000","last_active":"2024-09-17T07:07:28.206619+0000","last_peered":"2024-09-17T07:07:28.206619+0000","last_clean":"2024-09-17T07:07:28.206619+0000","last_became_active":"2024-09-17T07:04:58.346352+0000","last_became_peered":"2024-09-17T07:04:58.346352+0000","last_unstale":"2024-09-17T07:07:28.206619+0000","last_undegraded":"2024-09-17T07:07:28.206619+0000","last_fullsized":"2024-09-17T07:07:28.206619+0000","mapping_epoch":99,"log_start":"144'5869","ondisk_log_start":"144'5869","created":99,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'8912","last_scrub_stamp":"2024-09-17T07:06:49.883785+0000","last_deep_scrub":"152'8912","last_deep_scrub_stamp":"2024-09-17T07:06:49.883785+0000","last_clean_scrub_stamp":"2024-09-17T07:06:49.883785+0000","objects_scrubbed":1930,"log_size":4483,"log_dups_size":0,"ondisk_log_size":4483,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":11,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:09.746728+0000","scrub_duration":10.040198303,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21626880,"num_objects":330,"num_object_clones":0,"num_object_copies":990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1600,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"75'12","reported_seq":167,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.255927+0000","last_change":"2024-09-17T07:04:50.489946+0000","last_active":"2024-09-17T07:07:23.255927+0000","last_peered":"2024-09-17T07:07:23.255927+0000","last_clean":"2024-09-17T07:07:23.255927+0000","last_became_active":"2024-09-17T07:03:48.760405+0000","last_became_peered":"2024-09-17T07:03:48.760405+0000","last_unstale":"2024-09-17T07:07:23.255927+0000","last_undegraded":"2024-09-17T07:07:23.255927+0000","last_fullsized":"2024-09-17T07:07:23.255927+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-09-17T07:04:50.489723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:04:50.489723+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.007032679,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"70'1","reported_seq":134,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.252266+0000","last_change":"2024-09-17T07:07:21.050377+0000","last_active":"2024-09-17T07:07:23.252266+0000","last_peered":"2024-09-17T07:07:23.252266+0000","last_clean":"2024-09-17T07:07:23.252266+0000","last_became_active":"2024-09-17T07:02:12.009476+0000","last_became_peered":"2024-09-17T07:02:12.009476+0000","last_unstale":"2024-09-17T07:07:23.252266+0000","last_undegraded":"2024-09-17T07:07:23.252266+0000","last_fullsized":"2024-09-17T07:07:23.252266+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-17T07:07:21.050247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:07:21.050247+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:38.188477+0000","scrub_duration":0.005277505,"objects_trimmed":0,"snaptrim_duration":0.0032298629999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":117,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.257377+0000","last_change":"2024-09-17T07:03:51.994858+0000","last_active":"2024-09-17T07:07:23.257377+0000","last_peered":"2024-09-17T07:07:23.257377+0000","last_clean":"2024-09-17T07:07:23.257377+0000","last_became_active":"2024-09-17T07:02:12.014623+0000","last_became_peered":"2024-09-17T07:02:12.014623+0000","last_unstale":"2024-09-17T07:07:23.257377+0000","last_undegraded":"2024-09-17T07:07:23.257377+0000","last_fullsized":"2024-09-17T07:07:23.257377+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:51.994756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:51.994756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001933269,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":124,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-17T07:07:24.248565+0000","last_change":"2024-09-17T07:06:28.863501+0000","last_active":"2024-09-17T07:07:24.248565+0000","last_peered":"2024-09-17T07:07:24.248565+0000","last_clean":"2024-09-17T07:07:24.248565+0000","last_became_active":"2024-09-17T07:02:16.021561+0000","last_became_peered":"2024-09-17T07:02:16.021561+0000","last_unstale":"2024-09-17T07:07:24.248565+0000","last_undegraded":"2024-09-17T07:07:24.248565+0000","last_fullsized":"2024-09-17T07:07:24.248565+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:28.863357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:06:28.863357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:40.252322+0000","scrub_duration":0.014593309,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":121,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-17T07:07:25.570138+0000","last_change":"2024-09-17T07:07:25.570226+0000","last_active":"2024-09-17T07:07:25.570138+0000","last_peered":"2024-09-17T07:07:25.570138+0000","last_clean":"2024-09-17T07:07:25.570138+0000","last_became_active":"2024-09-17T07:02:18.022082+0000","last_became_peered":"2024-09-17T07:02:18.022082+0000","last_unstale":"2024-09-17T07:07:25.570138+0000","last_undegraded":"2024-09-17T07:07:25.570138+0000","last_fullsized":"2024-09-17T07:07:25.570138+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:25.570107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:07:25.570107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:37.811807+0000","scrub_duration":0.0049089310000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"162'8171","reported_seq":8303,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-17T07:07:28.895608+0000","last_change":"2024-09-17T07:06:46.844354+0000","last_active":"2024-09-17T07:07:28.895608+0000","last_peered":"2024-09-17T07:07:28.895608+0000","last_clean":"2024-09-17T07:07:28.895608+0000","last_became_active":"2024-09-17T07:04:58.361207+0000","last_became_peered":"2024-09-17T07:04:58.361207+0000","last_unstale":"2024-09-17T07:07:28.895608+0000","last_undegraded":"2024-09-17T07:07:28.895608+0000","last_fullsized":"2024-09-17T07:07:28.895608+0000","mapping_epoch":99,"log_start":"110'3305","ondisk_log_start":"110'3305","created":99,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'6605","last_scrub_stamp":"2024-09-17T07:06:46.844191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_clean_scrub_stamp":"2024-09-17T07:06:46.844191+0000","objects_scrubbed":1865,"log_size":4866,"log_dups_size":0,"ondisk_log_size":4866,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:01.893755+0000","scrub_duration":0.59670863900000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18939904,"num_objects":289,"num_object_clones":0,"num_object_copies":867,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1576,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"75'14","reported_seq":148,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-17T07:07:26.395456+0000","last_change":"2024-09-17T07:07:26.395820+0000","last_active":"2024-09-17T07:07:26.395456+0000","last_peered":"2024-09-17T07:07:26.395456+0000","last_clean":"2024-09-17T07:07:26.395456+0000","last_became_active":"2024-09-17T07:02:14.028857+0000","last_became_peered":"2024-09-17T07:02:14.028857+0000","last_unstale":"2024-09-17T07:07:26.395456+0000","last_undegraded":"2024-09-17T07:07:26.395456+0000","last_fullsized":"2024-09-17T07:07:26.395456+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'14","last_scrub_stamp":"2024-09-17T07:07:26.395429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:07:26.395429+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:27.868879+0000","scrub_duration":0.0028559779999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"72'2","reported_seq":134,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.256533+0000","last_change":"2024-09-17T07:04:35.691710+0000","last_active":"2024-09-17T07:07:23.256533+0000","last_peered":"2024-09-17T07:07:23.256533+0000","last_clean":"2024-09-17T07:07:23.256533+0000","last_became_active":"2024-09-17T07:02:12.005127+0000","last_became_peered":"2024-09-17T07:02:12.005127+0000","last_unstale":"2024-09-17T07:07:23.256533+0000","last_undegraded":"2024-09-17T07:07:23.256533+0000","last_fullsized":"2024-09-17T07:07:23.256533+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'2","last_scrub_stamp":"2024-09-17T07:04:35.690341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:04:35.690341+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0079935560000000006,"objects_trimmed":0,"snaptrim_duration":0.0010980219999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":139,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-17T07:07:24.248174+0000","last_change":"2024-09-17T07:06:37.732110+0000","last_active":"2024-09-17T07:07:24.248174+0000","last_peered":"2024-09-17T07:07:24.248174+0000","last_clean":"2024-09-17T07:07:24.248174+0000","last_became_active":"2024-09-17T07:02:12.004606+0000","last_became_peered":"2024-09-17T07:02:12.004606+0000","last_unstale":"2024-09-17T07:07:24.248174+0000","last_undegraded":"2024-09-17T07:07:24.248174+0000","last_fullsized":"2024-09-17T07:07:24.248174+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:37.731844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:06:37.731844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:04.459809+0000","scrub_duration":0.0070683819999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"78'128","reported_seq":376,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.257545+0000","last_change":"2024-09-17T07:04:37.752152+0000","last_active":"2024-09-17T07:07:23.257545+0000","last_peered":"2024-09-17T07:07:23.257545+0000","last_clean":"2024-09-17T07:07:23.257545+0000","last_became_active":"2024-09-17T07:02:03.020675+0000","last_became_peered":"2024-09-17T07:02:03.020675+0000","last_unstale":"2024-09-17T07:07:23.257545+0000","last_undegraded":"2024-09-17T07:07:23.257545+0000","last_fullsized":"2024-09-17T07:07:23.257545+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'128","last_scrub_stamp":"2024-09-17T07:04:37.751949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:00:09.390381+0000","last_clean_scrub_stamp":"2024-09-17T07:04:37.751949+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040036619999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":124,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.257710+0000","last_change":"2024-09-17T07:04:43.722620+0000","last_active":"2024-09-17T07:07:23.257710+0000","last_peered":"2024-09-17T07:07:23.257710+0000","last_clean":"2024-09-17T07:07:23.257710+0000","last_became_active":"2024-09-17T07:02:16.016031+0000","last_became_peered":"2024-09-17T07:02:16.016031+0000","last_unstale":"2024-09-17T07:07:23.257710+0000","last_undegraded":"2024-09-17T07:07:23.257710+0000","last_fullsized":"2024-09-17T07:07:23.257710+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:43.722510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:04:43.722510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0024960820000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":122,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-17T07:07:24.248739+0000","last_change":"2024-09-17T07:06:30.808658+0000","last_active":"2024-09-17T07:07:24.248739+0000","last_peered":"2024-09-17T07:07:24.248739+0000","last_clean":"2024-09-17T07:07:24.248739+0000","last_became_active":"2024-09-17T07:02:18.023321+0000","last_became_peered":"2024-09-17T07:02:18.023321+0000","last_unstale":"2024-09-17T07:07:24.248739+0000","last_undegraded":"2024-09-17T07:07:24.248739+0000","last_fullsized":"2024-09-17T07:07:24.248739+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:30.808562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:58.670571+0000","last_clean_scrub_stamp":"2024-09-17T07:06:30.808562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:55.726519+0000","scrub_duration":0.005261338,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.9","version":"162'8476","reported_seq":8615,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-17T07:07:28.897338+0000","last_change":"2024-09-17T07:06:38.907019+0000","last_active":"2024-09-17T07:07:28.897338+0000","last_peered":"2024-09-17T07:07:28.897338+0000","last_clean":"2024-09-17T07:07:28.897338+0000","last_became_active":"2024-09-17T07:04:58.345129+0000","last_became_peered":"2024-09-17T07:04:58.345129+0000","last_unstale":"2024-09-17T07:07:28.897338+0000","last_undegraded":"2024-09-17T07:07:28.897338+0000","last_fullsized":"2024-09-17T07:07:28.897338+0000","mapping_epoch":99,"log_start":"118'3548","ondisk_log_start":"118'3548","created":99,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'6535","last_scrub_stamp":"2024-09-17T07:06:38.906726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_clean_scrub_stamp":"2024-09-17T07:06:38.906726+0000","objects_scrubbed":2255,"log_size":4928,"log_dups_size":0,"ondisk_log_size":4928,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:51.910446+0000","scrub_duration":0.73214807299999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20054016,"num_objects":306,"num_object_clones":0,"num_object_copies":918,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1949,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"75'19","reported_seq":157,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.252626+0000","last_change":"2024-09-17T07:07:18.384022+0000","last_active":"2024-09-17T07:07:23.252626+0000","last_peered":"2024-09-17T07:07:23.252626+0000","last_clean":"2024-09-17T07:07:23.252626+0000","last_became_active":"2024-09-17T07:02:14.027313+0000","last_became_peered":"2024-09-17T07:02:14.027313+0000","last_unstale":"2024-09-17T07:07:23.252626+0000","last_undegraded":"2024-09-17T07:07:23.252626+0000","last_fullsized":"2024-09-17T07:07:23.252626+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-09-17T07:07:18.383813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:07:18.383813+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:42.977846+0000","scrub_duration":0.0040351470000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":139,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.268486+0000","last_change":"2024-09-17T07:06:39.322648+0000","last_active":"2024-09-17T07:07:23.268486+0000","last_peered":"2024-09-17T07:07:23.268486+0000","last_clean":"2024-09-17T07:07:23.268486+0000","last_became_active":"2024-09-17T07:02:12.012037+0000","last_became_peered":"2024-09-17T07:02:12.012037+0000","last_unstale":"2024-09-17T07:07:23.268486+0000","last_undegraded":"2024-09-17T07:07:23.268486+0000","last_fullsized":"2024-09-17T07:07:23.268486+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:39.322535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:06:39.322535+0000","last_clean_scrub_stamp":"2024-09-17T07:06:39.322535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:46.785051+0000","scrub_duration":0.002360993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":135,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.268735+0000","last_change":"2024-09-17T07:06:28.171453+0000","last_active":"2024-09-17T07:07:23.268735+0000","last_peered":"2024-09-17T07:07:23.268735+0000","last_clean":"2024-09-17T07:07:23.268735+0000","last_became_active":"2024-09-17T07:02:16.014631+0000","last_became_peered":"2024-09-17T07:02:16.014631+0000","last_unstale":"2024-09-17T07:07:23.268735+0000","last_undegraded":"2024-09-17T07:07:23.268735+0000","last_fullsized":"2024-09-17T07:07:23.268735+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:28.170477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:06:28.170477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:50.964072+0000","scrub_duration":0.0059381709999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":122,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-17T07:07:27.004707+0000","last_change":"2024-09-17T07:07:27.004939+0000","last_active":"2024-09-17T07:07:27.004707+0000","last_peered":"2024-09-17T07:07:27.004707+0000","last_clean":"2024-09-17T07:07:27.004707+0000","last_became_active":"2024-09-17T07:02:18.023299+0000","last_became_peered":"2024-09-17T07:02:18.023299+0000","last_unstale":"2024-09-17T07:07:27.004707+0000","last_undegraded":"2024-09-17T07:07:27.004707+0000","last_fullsized":"2024-09-17T07:07:27.004707+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:27.004683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:07:27.004683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:37.142184+0000","scrub_duration":0.010022120000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.8","version":"162'10354","reported_seq":10609,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-17T07:07:28.212386+0000","last_change":"2024-09-17T07:06:46.778697+0000","last_active":"2024-09-17T07:07:28.212386+0000","last_peered":"2024-09-17T07:07:28.212386+0000","last_clean":"2024-09-17T07:07:28.212386+0000","last_became_active":"2024-09-17T07:04:58.343731+0000","last_became_peered":"2024-09-17T07:04:58.343731+0000","last_unstale":"2024-09-17T07:07:28.212386+0000","last_undegraded":"2024-09-17T07:07:28.212386+0000","last_fullsized":"2024-09-17T07:07:28.212386+0000","mapping_epoch":99,"log_start":"145'5862","ondisk_log_start":"145'5862","created":99,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'8793","last_scrub_stamp":"2024-09-17T07:06:46.778519+0000","last_deep_scrub":"151'8793","last_deep_scrub_stamp":"2024-09-17T07:06:46.778519+0000","last_clean_scrub_stamp":"2024-09-17T07:06:46.778519+0000","objects_scrubbed":1992,"log_size":4492,"log_dups_size":0,"ondisk_log_size":4492,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":9,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:06.324620+0000","scrub_duration":8.9076282960000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19202048,"num_objects":293,"num_object_clones":0,"num_object_copies":879,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1699,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"75'9","reported_seq":142,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.252090+0000","last_change":"2024-09-17T07:06:26.996048+0000","last_active":"2024-09-17T07:07:23.252090+0000","last_peered":"2024-09-17T07:07:23.252090+0000","last_clean":"2024-09-17T07:07:23.252090+0000","last_became_active":"2024-09-17T07:02:14.014939+0000","last_became_peered":"2024-09-17T07:02:14.014939+0000","last_unstale":"2024-09-17T07:07:23.252090+0000","last_undegraded":"2024-09-17T07:07:23.252090+0000","last_fullsized":"2024-09-17T07:07:23.252090+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:06:26.995923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:06:26.995923+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:49.741971+0000","scrub_duration":0.01417006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":127,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.271457+0000","last_change":"2024-09-17T07:06:30.038396+0000","last_active":"2024-09-17T07:07:23.271457+0000","last_peered":"2024-09-17T07:07:23.271457+0000","last_clean":"2024-09-17T07:07:23.271457+0000","last_became_active":"2024-09-17T07:02:12.000005+0000","last_became_peered":"2024-09-17T07:02:12.000005+0000","last_unstale":"2024-09-17T07:07:23.271457+0000","last_undegraded":"2024-09-17T07:07:23.271457+0000","last_fullsized":"2024-09-17T07:07:23.271457+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:30.038104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:06:30.038104+0000","last_clean_scrub_stamp":"2024-09-17T07:06:30.038104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:38.139496+0000","scrub_duration":0.021438763,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":90,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-17T07:07:24.241855+0000","last_change":"2024-09-17T07:07:19.789263+0000","last_active":"2024-09-17T07:07:24.241855+0000","last_peered":"2024-09-17T07:07:24.241855+0000","last_clean":"2024-09-17T07:07:24.241855+0000","last_became_active":"2024-09-17T07:04:15.107886+0000","last_became_peered":"2024-09-17T07:04:15.107886+0000","last_unstale":"2024-09-17T07:07:24.241855+0000","last_undegraded":"2024-09-17T07:07:24.241855+0000","last_fullsized":"2024-09-17T07:07:24.241855+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:19.789134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:07:19.789134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:20.148885+0000","scrub_duration":0.010505654,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":122,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.252190+0000","last_change":"2024-09-17T07:07:20.025023+0000","last_active":"2024-09-17T07:07:23.252190+0000","last_peered":"2024-09-17T07:07:23.252190+0000","last_clean":"2024-09-17T07:07:23.252190+0000","last_became_active":"2024-09-17T07:02:18.026994+0000","last_became_peered":"2024-09-17T07:02:18.026994+0000","last_unstale":"2024-09-17T07:07:23.252190+0000","last_undegraded":"2024-09-17T07:07:23.252190+0000","last_fullsized":"2024-09-17T07:07:23.252190+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:20.024902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:58.099849+0000","last_clean_scrub_stamp":"2024-09-17T07:07:20.024902+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:30.210770+0000","scrub_duration":0.013970381,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.b","version":"162'13288","reported_seq":13361,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-17T07:07:28.109062+0000","last_change":"2024-09-17T07:04:20.803445+0000","last_active":"2024-09-17T07:07:28.109062+0000","last_peered":"2024-09-17T07:07:28.109062+0000","last_clean":"2024-09-17T07:07:28.109062+0000","last_became_active":"2024-09-17T07:04:20.803245+0000","last_became_peered":"2024-09-17T07:04:20.803245+0000","last_unstale":"2024-09-17T07:07:28.109062+0000","last_undegraded":"2024-09-17T07:07:28.109062+0000","last_fullsized":"2024-09-17T07:07:28.109062+0000","mapping_epoch":99,"log_start":"144'8300","ondisk_log_start":"144'8300","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_clean_scrub_stamp":"2024-09-17T07:04:19.785747+0000","objects_scrubbed":0,"log_size":4988,"log_dups_size":0,"ondisk_log_size":4988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41156608,"num_objects":628,"num_object_clones":0,"num_object_copies":1884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13288,"num_write_kb":445312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"111'22","reported_seq":169,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-17T07:07:29.816377+0000","last_change":"2024-09-17T07:07:29.816377+0000","last_active":"2024-09-17T07:07:29.816377+0000","last_peered":"2024-09-17T07:07:29.816377+0000","last_clean":"2024-09-17T07:07:29.816377+0000","last_became_active":"2024-09-17T07:02:14.028392+0000","last_became_peered":"2024-09-17T07:02:14.028392+0000","last_unstale":"2024-09-17T07:07:29.816377+0000","last_undegraded":"2024-09-17T07:07:29.816377+0000","last_fullsized":"2024-09-17T07:07:29.816377+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'22","last_scrub_stamp":"2024-09-17T07:07:29.816241+0000","last_deep_scrub":"111'22","last_deep_scrub_stamp":"2024-09-17T07:04:57.500450+0000","last_clean_scrub_stamp":"2024-09-17T07:07:29.816241+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:53.480974+0000","scrub_duration":0.0034677570000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'2","reported_seq":128,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.245998+0000","last_change":"2024-09-17T07:06:39.828818+0000","last_active":"2024-09-17T07:07:23.245998+0000","last_peered":"2024-09-17T07:07:23.245998+0000","last_clean":"2024-09-17T07:07:23.245998+0000","last_became_active":"2024-09-17T07:04:15.110085+0000","last_became_peered":"2024-09-17T07:04:15.110085+0000","last_unstale":"2024-09-17T07:07:23.245998+0000","last_undegraded":"2024-09-17T07:07:23.245998+0000","last_fullsized":"2024-09-17T07:07:23.245998+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-09-17T07:06:39.828638+0000","last_deep_scrub":"89'2","last_deep_scrub_stamp":"2024-09-17T07:03:26.694366+0000","last_clean_scrub_stamp":"2024-09-17T07:06:39.828638+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:09.464498+0000","scrub_duration":0.0068983610000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":420,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"75'8","reported_seq":198,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-17T07:07:24.244442+0000","last_change":"2024-09-17T07:06:25.761229+0000","last_active":"2024-09-17T07:07:24.244442+0000","last_peered":"2024-09-17T07:07:24.244442+0000","last_clean":"2024-09-17T07:07:24.244442+0000","last_became_active":"2024-09-17T07:02:16.015418+0000","last_became_peered":"2024-09-17T07:02:16.015418+0000","last_unstale":"2024-09-17T07:07:24.244442+0000","last_undegraded":"2024-09-17T07:07:24.244442+0000","last_fullsized":"2024-09-17T07:07:24.244442+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:06:25.760922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:06:25.760922+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:34.826717+0000","scrub_duration":0.0073758620000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":133,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-17T07:07:24.248228+0000","last_change":"2024-09-17T07:07:19.786753+0000","last_active":"2024-09-17T07:07:24.248228+0000","last_peered":"2024-09-17T07:07:24.248228+0000","last_clean":"2024-09-17T07:07:24.248228+0000","last_became_active":"2024-09-17T07:02:18.363253+0000","last_became_peered":"2024-09-17T07:02:18.363253+0000","last_unstale":"2024-09-17T07:07:24.248228+0000","last_undegraded":"2024-09-17T07:07:24.248228+0000","last_fullsized":"2024-09-17T07:07:24.248228+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:19.786442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:07:19.786442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:41.487701+0000","scrub_duration":0.011747911999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.a","version":"162'13147","reported_seq":13220,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-17T07:07:28.628999+0000","last_change":"2024-09-17T07:04:20.804570+0000","last_active":"2024-09-17T07:07:28.628999+0000","last_peered":"2024-09-17T07:07:28.628999+0000","last_clean":"2024-09-17T07:07:28.628999+0000","last_became_active":"2024-09-17T07:04:20.804306+0000","last_became_peered":"2024-09-17T07:04:20.804306+0000","last_unstale":"2024-09-17T07:07:28.628999+0000","last_undegraded":"2024-09-17T07:07:28.628999+0000","last_fullsized":"2024-09-17T07:07:28.628999+0000","mapping_epoch":99,"log_start":"148'8600","ondisk_log_start":"148'8600","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_clean_scrub_stamp":"2024-09-17T07:04:19.785747+0000","objects_scrubbed":0,"log_size":4547,"log_dups_size":0,"ondisk_log_size":4547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38076416,"num_objects":581,"num_object_clones":0,"num_object_copies":1743,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13147,"num_write_kb":439296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"150'17","reported_seq":155,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-17T07:07:24.241638+0000","last_change":"2024-09-17T07:06:38.775932+0000","last_active":"2024-09-17T07:07:24.241638+0000","last_peered":"2024-09-17T07:07:24.241638+0000","last_clean":"2024-09-17T07:07:24.241638+0000","last_became_active":"2024-09-17T07:02:14.010219+0000","last_became_peered":"2024-09-17T07:02:14.010219+0000","last_unstale":"2024-09-17T07:07:24.241638+0000","last_undegraded":"2024-09-17T07:07:24.241638+0000","last_fullsized":"2024-09-17T07:07:24.241638+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'17","last_scrub_stamp":"2024-09-17T07:06:38.775788+0000","last_deep_scrub":"150'17","last_deep_scrub_stamp":"2024-09-17T07:06:38.775788+0000","last_clean_scrub_stamp":"2024-09-17T07:06:38.775788+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:54.992016+0000","scrub_duration":0.011503368999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"68'2","reported_seq":137,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-17T07:07:24.242352+0000","last_change":"2024-09-17T07:07:20.764435+0000","last_active":"2024-09-17T07:07:24.242352+0000","last_peered":"2024-09-17T07:07:24.242352+0000","last_clean":"2024-09-17T07:07:24.242352+0000","last_became_active":"2024-09-17T07:02:12.006716+0000","last_became_peered":"2024-09-17T07:02:12.006716+0000","last_unstale":"2024-09-17T07:07:24.242352+0000","last_undegraded":"2024-09-17T07:07:24.242352+0000","last_fullsized":"2024-09-17T07:07:24.242352+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-09-17T07:07:20.764283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:07:20.764283+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:50.410328+0000","scrub_duration":0.0060160040000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":88,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.252926+0000","last_change":"2024-09-17T07:04:55.444916+0000","last_active":"2024-09-17T07:07:23.252926+0000","last_peered":"2024-09-17T07:07:23.252926+0000","last_clean":"2024-09-17T07:07:23.252926+0000","last_became_active":"2024-09-17T07:04:15.111069+0000","last_became_peered":"2024-09-17T07:04:15.111069+0000","last_unstale":"2024-09-17T07:07:23.252926+0000","last_undegraded":"2024-09-17T07:07:23.252926+0000","last_fullsized":"2024-09-17T07:07:23.252926+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:55.444802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:04:55.444802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0079380379999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":110,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.245298+0000","last_change":"2024-09-17T07:04:06.571270+0000","last_active":"2024-09-17T07:07:23.245298+0000","last_peered":"2024-09-17T07:07:23.245298+0000","last_clean":"2024-09-17T07:07:23.245298+0000","last_became_active":"2024-09-17T07:02:18.024983+0000","last_became_peered":"2024-09-17T07:02:18.024983+0000","last_unstale":"2024-09-17T07:07:23.245298+0000","last_undegraded":"2024-09-17T07:07:23.245298+0000","last_fullsized":"2024-09-17T07:07:23.245298+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:06.571184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:04:06.571184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001613319,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.15","version":"162'8211","reported_seq":11801,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-17T07:07:28.214595+0000","last_change":"2024-09-17T07:05:15.062360+0000","last_active":"2024-09-17T07:07:28.214595+0000","last_peered":"2024-09-17T07:07:28.214595+0000","last_clean":"2024-09-17T07:07:28.214595+0000","last_became_active":"2024-09-17T07:05:15.059992+0000","last_became_peered":"2024-09-17T07:05:15.059992+0000","last_unstale":"2024-09-17T07:07:28.214595+0000","last_undegraded":"2024-09-17T07:07:28.214595+0000","last_fullsized":"2024-09-17T07:07:28.214595+0000","mapping_epoch":119,"log_start":"118'3745","ondisk_log_start":"118'3745","created":111,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_clean_scrub_stamp":"2024-09-17T07:04:19.785747+0000","objects_scrubbed":0,"log_size":4466,"log_dups_size":0,"ondisk_log_size":4466,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22446080,"num_objects":343,"num_object_clones":0,"num_object_copies":1029,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6477,"num_write_kb":218208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1745,"num_bytes_recovered":114360320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"75'9","reported_seq":142,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.261030+0000","last_change":"2024-09-17T07:06:30.011545+0000","last_active":"2024-09-17T07:07:23.261030+0000","last_peered":"2024-09-17T07:07:23.261030+0000","last_clean":"2024-09-17T07:07:23.261030+0000","last_became_active":"2024-09-17T07:02:14.029388+0000","last_became_peered":"2024-09-17T07:02:14.029388+0000","last_unstale":"2024-09-17T07:07:23.261030+0000","last_undegraded":"2024-09-17T07:07:23.261030+0000","last_fullsized":"2024-09-17T07:07:23.261030+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:06:30.011337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:06:30.011337+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:41.133399+0000","scrub_duration":0.013862154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":139,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.274468+0000","last_change":"2024-09-17T07:06:40.406566+0000","last_active":"2024-09-17T07:07:23.274468+0000","last_peered":"2024-09-17T07:07:23.274468+0000","last_clean":"2024-09-17T07:07:23.274468+0000","last_became_active":"2024-09-17T07:02:12.011327+0000","last_became_peered":"2024-09-17T07:02:12.011327+0000","last_unstale":"2024-09-17T07:07:23.274468+0000","last_undegraded":"2024-09-17T07:07:23.274468+0000","last_fullsized":"2024-09-17T07:07:23.274468+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:40.406459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:31.753470+0000","last_clean_scrub_stamp":"2024-09-17T07:06:40.406459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:54.638620+0000","scrub_duration":0.03274349,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":123,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.246462+0000","last_change":"2024-09-17T07:04:36.652033+0000","last_active":"2024-09-17T07:07:23.246462+0000","last_peered":"2024-09-17T07:07:23.246462+0000","last_clean":"2024-09-17T07:07:23.246462+0000","last_became_active":"2024-09-17T07:02:16.022584+0000","last_became_peered":"2024-09-17T07:02:16.022584+0000","last_unstale":"2024-09-17T07:07:23.246462+0000","last_undegraded":"2024-09-17T07:07:23.246462+0000","last_fullsized":"2024-09-17T07:07:23.246462+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:36.651925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:04:36.651925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0061789210000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":133,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.268061+0000","last_change":"2024-09-17T07:06:36.398341+0000","last_active":"2024-09-17T07:07:23.268061+0000","last_peered":"2024-09-17T07:07:23.268061+0000","last_clean":"2024-09-17T07:07:23.268061+0000","last_became_active":"2024-09-17T07:02:18.017825+0000","last_became_peered":"2024-09-17T07:02:18.017825+0000","last_unstale":"2024-09-17T07:07:23.268061+0000","last_undegraded":"2024-09-17T07:07:23.268061+0000","last_fullsized":"2024-09-17T07:07:23.268061+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:36.398177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:06:36.398177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:57.728751+0000","scrub_duration":0.099287720999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.14","version":"162'8327","reported_seq":15493,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-17T07:07:28.589115+0000","last_change":"2024-09-17T07:06:25.086462+0000","last_active":"2024-09-17T07:07:28.589115+0000","last_peered":"2024-09-17T07:07:28.589115+0000","last_clean":"2024-09-17T07:07:28.589115+0000","last_became_active":"2024-09-17T07:06:25.076747+0000","last_became_peered":"2024-09-17T07:06:25.076747+0000","last_unstale":"2024-09-17T07:07:28.589115+0000","last_undegraded":"2024-09-17T07:07:28.589115+0000","last_fullsized":"2024-09-17T07:07:28.589115+0000","mapping_epoch":147,"log_start":"118'3834","ondisk_log_start":"118'3834","created":111,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_clean_scrub_stamp":"2024-09-17T07:04:19.785747+0000","objects_scrubbed":0,"log_size":4493,"log_dups_size":0,"ondisk_log_size":4493,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20152320,"num_objects":308,"num_object_clones":0,"num_object_copies":924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6604,"num_write_kb":221152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1767,"num_bytes_recovered":115802112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":127,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.252903+0000","last_change":"2024-09-17T07:06:11.251464+0000","last_active":"2024-09-17T07:07:23.252903+0000","last_peered":"2024-09-17T07:07:23.252903+0000","last_clean":"2024-09-17T07:07:23.252903+0000","last_became_active":"2024-09-17T07:02:12.008065+0000","last_became_peered":"2024-09-17T07:02:12.008065+0000","last_unstale":"2024-09-17T07:07:23.252903+0000","last_undegraded":"2024-09-17T07:07:23.252903+0000","last_fullsized":"2024-09-17T07:07:23.252903+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:11.251341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:06:11.251341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:28.019325+0000","scrub_duration":0.0029670590000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"75'6","reported_seq":134,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.246579+0000","last_change":"2024-09-17T07:06:43.839442+0000","last_active":"2024-09-17T07:07:23.246579+0000","last_peered":"2024-09-17T07:07:23.246579+0000","last_clean":"2024-09-17T07:07:23.246579+0000","last_became_active":"2024-09-17T07:02:14.028655+0000","last_became_peered":"2024-09-17T07:02:14.028655+0000","last_unstale":"2024-09-17T07:07:23.246579+0000","last_undegraded":"2024-09-17T07:07:23.246579+0000","last_fullsized":"2024-09-17T07:07:23.246579+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'6","last_scrub_stamp":"2024-09-17T07:06:43.836545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:06:43.836545+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:48.040328+0000","scrub_duration":0.0059941980000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":124,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-17T07:07:24.248488+0000","last_change":"2024-09-17T07:06:29.854234+0000","last_active":"2024-09-17T07:07:24.248488+0000","last_peered":"2024-09-17T07:07:24.248488+0000","last_clean":"2024-09-17T07:07:24.248488+0000","last_became_active":"2024-09-17T07:02:16.016634+0000","last_became_peered":"2024-09-17T07:02:16.016634+0000","last_unstale":"2024-09-17T07:07:24.248488+0000","last_undegraded":"2024-09-17T07:07:24.248488+0000","last_fullsized":"2024-09-17T07:07:24.248488+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:29.854126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:06:29.854126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:37.183540+0000","scrub_duration":0.0052853860000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":122,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.257950+0000","last_change":"2024-09-17T07:07:18.042204+0000","last_active":"2024-09-17T07:07:23.257950+0000","last_peered":"2024-09-17T07:07:23.257950+0000","last_clean":"2024-09-17T07:07:23.257950+0000","last_became_active":"2024-09-17T07:02:18.365419+0000","last_became_peered":"2024-09-17T07:02:18.365419+0000","last_unstale":"2024-09-17T07:07:23.257950+0000","last_undegraded":"2024-09-17T07:07:23.257950+0000","last_fullsized":"2024-09-17T07:07:23.257950+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:18.042105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:07:18.042105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:18.135607+0000","scrub_duration":0.0020576729999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.17","version":"162'8305","reported_seq":15651,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-17T07:07:28.200874+0000","last_change":"2024-09-17T07:06:22.239760+0000","last_active":"2024-09-17T07:07:28.200874+0000","last_peered":"2024-09-17T07:07:28.200874+0000","last_clean":"2024-09-17T07:07:28.200874+0000","last_became_active":"2024-09-17T07:06:22.231579+0000","last_became_peered":"2024-09-17T07:06:22.231579+0000","last_unstale":"2024-09-17T07:07:28.200874+0000","last_undegraded":"2024-09-17T07:07:28.200874+0000","last_fullsized":"2024-09-17T07:07:28.200874+0000","mapping_epoch":145,"log_start":"118'3799","ondisk_log_start":"118'3799","created":111,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_clean_scrub_stamp":"2024-09-17T07:04:19.785747+0000","objects_scrubbed":0,"log_size":4506,"log_dups_size":0,"ondisk_log_size":4506,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18219008,"num_objects":278,"num_object_clones":0,"num_object_copies":834,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6578,"num_write_kb":219392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1812,"num_bytes_recovered":118751232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"75'10","reported_seq":140,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.253621+0000","last_change":"2024-09-17T07:04:40.721975+0000","last_active":"2024-09-17T07:07:23.253621+0000","last_peered":"2024-09-17T07:07:23.253621+0000","last_clean":"2024-09-17T07:07:23.253621+0000","last_became_active":"2024-09-17T07:02:14.027499+0000","last_became_peered":"2024-09-17T07:02:14.027499+0000","last_unstale":"2024-09-17T07:07:23.253621+0000","last_undegraded":"2024-09-17T07:07:23.253621+0000","last_fullsized":"2024-09-17T07:07:23.253621+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-09-17T07:04:40.721843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:04:40.721843+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0057465399999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":139,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-17T07:07:24.248537+0000","last_change":"2024-09-17T07:06:45.799922+0000","last_active":"2024-09-17T07:07:24.248537+0000","last_peered":"2024-09-17T07:07:24.248537+0000","last_clean":"2024-09-17T07:07:24.248537+0000","last_became_active":"2024-09-17T07:02:12.007280+0000","last_became_peered":"2024-09-17T07:02:12.007280+0000","last_unstale":"2024-09-17T07:07:24.248537+0000","last_undegraded":"2024-09-17T07:07:24.248537+0000","last_fullsized":"2024-09-17T07:07:24.248537+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:45.799808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:06:45.799808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:47.601528+0000","scrub_duration":0.0021198189999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"75'8","reported_seq":196,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-17T07:07:30.578745+0000","last_change":"2024-09-17T07:06:24.892916+0000","last_active":"2024-09-17T07:07:30.578745+0000","last_peered":"2024-09-17T07:07:30.578745+0000","last_clean":"2024-09-17T07:07:30.578745+0000","last_became_active":"2024-09-17T07:02:16.022583+0000","last_became_peered":"2024-09-17T07:02:16.022583+0000","last_unstale":"2024-09-17T07:07:30.578745+0000","last_undegraded":"2024-09-17T07:07:30.578745+0000","last_fullsized":"2024-09-17T07:07:30.578745+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:06:24.892804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:06:24.892804+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0018794949999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":122,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-17T07:07:26.014418+0000","last_change":"2024-09-17T07:07:26.014516+0000","last_active":"2024-09-17T07:07:26.014418+0000","last_peered":"2024-09-17T07:07:26.014418+0000","last_clean":"2024-09-17T07:07:26.014418+0000","last_became_active":"2024-09-17T07:02:18.028796+0000","last_became_peered":"2024-09-17T07:02:18.028796+0000","last_unstale":"2024-09-17T07:07:26.014418+0000","last_undegraded":"2024-09-17T07:07:26.014418+0000","last_fullsized":"2024-09-17T07:07:26.014418+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:26.014398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:07:26.014398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:36.974117+0000","scrub_duration":0.0066951459999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"163'8267","reported_seq":11910,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T07:07:33.039218+0000","last_change":"2024-09-17T07:05:24.306755+0000","last_active":"2024-09-17T07:07:33.039218+0000","last_peered":"2024-09-17T07:07:33.039218+0000","last_clean":"2024-09-17T07:07:33.039218+0000","last_became_active":"2024-09-17T07:05:24.303086+0000","last_became_peered":"2024-09-17T07:05:24.303086+0000","last_unstale":"2024-09-17T07:07:33.039218+0000","last_undegraded":"2024-09-17T07:07:33.039218+0000","last_fullsized":"2024-09-17T07:07:33.039218+0000","mapping_epoch":123,"log_start":"108'2849","ondisk_log_start":"108'2849","created":111,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_clean_scrub_stamp":"2024-09-17T07:04:19.785747+0000","objects_scrubbed":0,"log_size":5418,"log_dups_size":0,"ondisk_log_size":5418,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5996544,"num_objects":92,"num_object_clones":0,"num_object_copies":276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6594,"num_write_kb":213920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1771,"num_bytes_recovered":116064256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"75'9","reported_seq":153,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-17T07:07:24.247812+0000","last_change":"2024-09-17T07:06:35.727847+0000","last_active":"2024-09-17T07:07:24.247812+0000","last_peered":"2024-09-17T07:07:24.247812+0000","last_clean":"2024-09-17T07:07:24.247812+0000","last_became_active":"2024-09-17T07:02:14.028485+0000","last_became_peered":"2024-09-17T07:02:14.028485+0000","last_unstale":"2024-09-17T07:07:24.247812+0000","last_undegraded":"2024-09-17T07:07:24.247812+0000","last_fullsized":"2024-09-17T07:07:24.247812+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:06:35.727700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:06:35.727700+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:52.353085+0000","scrub_duration":0.0045062289999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":128,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.252388+0000","last_change":"2024-09-17T07:04:51.449374+0000","last_active":"2024-09-17T07:07:23.252388+0000","last_peered":"2024-09-17T07:07:23.252388+0000","last_clean":"2024-09-17T07:07:23.252388+0000","last_became_active":"2024-09-17T07:02:12.009935+0000","last_became_peered":"2024-09-17T07:02:12.009935+0000","last_unstale":"2024-09-17T07:07:23.252388+0000","last_undegraded":"2024-09-17T07:07:23.252388+0000","last_fullsized":"2024-09-17T07:07:23.252388+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:51.449197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:04:51.449197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0069844269999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"75'8","reported_seq":183,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-17T07:07:24.248700+0000","last_change":"2024-09-17T07:03:42.469019+0000","last_active":"2024-09-17T07:07:24.248700+0000","last_peered":"2024-09-17T07:07:24.248700+0000","last_clean":"2024-09-17T07:07:24.248700+0000","last_became_active":"2024-09-17T07:02:16.016069+0000","last_became_peered":"2024-09-17T07:02:16.016069+0000","last_unstale":"2024-09-17T07:07:24.248700+0000","last_undegraded":"2024-09-17T07:07:24.248700+0000","last_fullsized":"2024-09-17T07:07:24.248700+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:03:42.468934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:03:42.468934+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0016937079999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":123,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-17T07:07:24.242061+0000","last_change":"2024-09-17T07:04:58.773306+0000","last_active":"2024-09-17T07:07:24.242061+0000","last_peered":"2024-09-17T07:07:24.242061+0000","last_clean":"2024-09-17T07:07:24.242061+0000","last_became_active":"2024-09-17T07:02:18.018716+0000","last_became_peered":"2024-09-17T07:02:18.018716+0000","last_unstale":"2024-09-17T07:07:24.242061+0000","last_undegraded":"2024-09-17T07:07:24.242061+0000","last_fullsized":"2024-09-17T07:07:24.242061+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:58.773188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:04:58.773188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0024048339999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.11","version":"162'8328","reported_seq":12059,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-17T07:07:28.218077+0000","last_change":"2024-09-17T07:05:30.096012+0000","last_active":"2024-09-17T07:07:28.218077+0000","last_peered":"2024-09-17T07:07:28.218077+0000","last_clean":"2024-09-17T07:07:28.218077+0000","last_became_active":"2024-09-17T07:05:23.265789+0000","last_became_peered":"2024-09-17T07:05:23.265789+0000","last_unstale":"2024-09-17T07:07:28.218077+0000","last_undegraded":"2024-09-17T07:07:28.218077+0000","last_fullsized":"2024-09-17T07:07:28.218077+0000","mapping_epoch":122,"log_start":"118'3899","ondisk_log_start":"118'3899","created":111,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_clean_scrub_stamp":"2024-09-17T07:04:19.785747+0000","objects_scrubbed":0,"log_size":4429,"log_dups_size":0,"ondisk_log_size":4429,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19398656,"num_objects":296,"num_object_clones":0,"num_object_copies":888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":296,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6584,"num_write_kb":220160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1815,"num_bytes_recovered":118947840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"75'9","reported_seq":141,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.265148+0000","last_change":"2024-09-17T07:04:56.049769+0000","last_active":"2024-09-17T07:07:23.265148+0000","last_peered":"2024-09-17T07:07:23.265148+0000","last_clean":"2024-09-17T07:07:23.265148+0000","last_became_active":"2024-09-17T07:02:14.010289+0000","last_became_peered":"2024-09-17T07:02:14.010289+0000","last_unstale":"2024-09-17T07:07:23.265148+0000","last_undegraded":"2024-09-17T07:07:23.265148+0000","last_fullsized":"2024-09-17T07:07:23.265148+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:04:56.049602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:04:56.049602+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0066971340000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":128,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.252714+0000","last_change":"2024-09-17T07:07:20.458487+0000","last_active":"2024-09-17T07:07:23.252714+0000","last_peered":"2024-09-17T07:07:23.252714+0000","last_clean":"2024-09-17T07:07:23.252714+0000","last_became_active":"2024-09-17T07:02:12.007433+0000","last_became_peered":"2024-09-17T07:02:12.007433+0000","last_unstale":"2024-09-17T07:07:23.252714+0000","last_undegraded":"2024-09-17T07:07:23.252714+0000","last_fullsized":"2024-09-17T07:07:23.252714+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:20.458316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:07:20.458316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:42.134938+0000","scrub_duration":0.0043263889999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":123,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.245366+0000","last_change":"2024-09-17T07:07:17.713153+0000","last_active":"2024-09-17T07:07:23.245366+0000","last_peered":"2024-09-17T07:07:23.245366+0000","last_clean":"2024-09-17T07:07:23.245366+0000","last_became_active":"2024-09-17T07:02:16.016216+0000","last_became_peered":"2024-09-17T07:02:16.016216+0000","last_unstale":"2024-09-17T07:07:23.245366+0000","last_undegraded":"2024-09-17T07:07:23.245366+0000","last_fullsized":"2024-09-17T07:07:23.245366+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:17.713027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:07:17.713027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:19.920380+0000","scrub_duration":0.017242598000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":122,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.252753+0000","last_change":"2024-09-17T07:06:24.966953+0000","last_active":"2024-09-17T07:07:23.252753+0000","last_peered":"2024-09-17T07:07:23.252753+0000","last_clean":"2024-09-17T07:07:23.252753+0000","last_became_active":"2024-09-17T07:02:18.027622+0000","last_became_peered":"2024-09-17T07:02:18.027622+0000","last_unstale":"2024-09-17T07:07:23.252753+0000","last_undegraded":"2024-09-17T07:07:23.252753+0000","last_fullsized":"2024-09-17T07:07:23.252753+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:24.966867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:06:24.966867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:26.322962+0000","scrub_duration":0.001663292,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"162'8081","reported_seq":11571,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-17T07:07:28.621538+0000","last_change":"2024-09-17T07:05:32.282685+0000","last_active":"2024-09-17T07:07:28.621538+0000","last_peered":"2024-09-17T07:07:28.621538+0000","last_clean":"2024-09-17T07:07:28.621538+0000","last_became_active":"2024-09-17T07:05:32.280398+0000","last_became_peered":"2024-09-17T07:05:32.280398+0000","last_unstale":"2024-09-17T07:07:28.621538+0000","last_undegraded":"2024-09-17T07:07:28.621538+0000","last_fullsized":"2024-09-17T07:07:28.621538+0000","mapping_epoch":126,"log_start":"118'3596","ondisk_log_start":"118'3596","created":111,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_clean_scrub_stamp":"2024-09-17T07:04:19.785747+0000","objects_scrubbed":0,"log_size":4485,"log_dups_size":0,"ondisk_log_size":4485,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":320,"num_object_clones":0,"num_object_copies":960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6400,"num_write_kb":215040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1695,"num_bytes_recovered":111083520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"75'11","reported_seq":146,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-17T07:07:24.242514+0000","last_change":"2024-09-17T07:06:11.787256+0000","last_active":"2024-09-17T07:07:24.242514+0000","last_peered":"2024-09-17T07:07:24.242514+0000","last_clean":"2024-09-17T07:07:24.242514+0000","last_became_active":"2024-09-17T07:02:14.016255+0000","last_became_peered":"2024-09-17T07:02:14.016255+0000","last_unstale":"2024-09-17T07:07:24.242514+0000","last_undegraded":"2024-09-17T07:07:24.242514+0000","last_fullsized":"2024-09-17T07:07:24.242514+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-09-17T07:06:11.787156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:06:11.787156+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035465940000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"94'2","reported_seq":122,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.271391+0000","last_change":"2024-09-17T07:04:06.027032+0000","last_active":"2024-09-17T07:07:23.271391+0000","last_peered":"2024-09-17T07:07:23.271391+0000","last_clean":"2024-09-17T07:07:23.271391+0000","last_became_active":"2024-09-17T07:02:12.011461+0000","last_became_peered":"2024-09-17T07:02:12.011461+0000","last_unstale":"2024-09-17T07:07:23.271391+0000","last_undegraded":"2024-09-17T07:07:23.271391+0000","last_fullsized":"2024-09-17T07:07:23.271391+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-09-17T07:04:06.026890+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-09-17T07:04:06.026890+0000","last_clean_scrub_stamp":"2024-09-17T07:04:06.026890+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0031797539999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":123,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.265008+0000","last_change":"2024-09-17T07:06:40.147092+0000","last_active":"2024-09-17T07:07:23.265008+0000","last_peered":"2024-09-17T07:07:23.265008+0000","last_clean":"2024-09-17T07:07:23.265008+0000","last_became_active":"2024-09-17T07:02:16.018754+0000","last_became_peered":"2024-09-17T07:02:16.018754+0000","last_unstale":"2024-09-17T07:07:23.265008+0000","last_undegraded":"2024-09-17T07:07:23.265008+0000","last_fullsized":"2024-09-17T07:07:23.265008+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:40.146846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:06:40.146846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:05.696921+0000","scrub_duration":0.00753165,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":110,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.253949+0000","last_change":"2024-09-17T07:04:07.566836+0000","last_active":"2024-09-17T07:07:23.253949+0000","last_peered":"2024-09-17T07:07:23.253949+0000","last_clean":"2024-09-17T07:07:23.253949+0000","last_became_active":"2024-09-17T07:02:18.027427+0000","last_became_peered":"2024-09-17T07:02:18.027427+0000","last_unstale":"2024-09-17T07:07:23.253949+0000","last_undegraded":"2024-09-17T07:07:23.253949+0000","last_fullsized":"2024-09-17T07:07:23.253949+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:07.566667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:04:07.566667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001538364,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"162'8641","reported_seq":14295,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-17T07:07:28.933815+0000","last_change":"2024-09-17T07:05:58.153071+0000","last_active":"2024-09-17T07:07:28.933815+0000","last_peered":"2024-09-17T07:07:28.933815+0000","last_clean":"2024-09-17T07:07:28.933815+0000","last_became_active":"2024-09-17T07:05:58.145854+0000","last_became_peered":"2024-09-17T07:05:58.145854+0000","last_unstale":"2024-09-17T07:07:28.933815+0000","last_undegraded":"2024-09-17T07:07:28.933815+0000","last_fullsized":"2024-09-17T07:07:28.933815+0000","mapping_epoch":134,"log_start":"118'3785","ondisk_log_start":"118'3785","created":111,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_clean_scrub_stamp":"2024-09-17T07:04:19.785747+0000","objects_scrubbed":0,"log_size":4856,"log_dups_size":0,"ondisk_log_size":4856,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17563648,"num_objects":268,"num_object_clones":0,"num_object_copies":804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":268,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6868,"num_write_kb":228352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1852,"num_bytes_recovered":121372672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"75'4","reported_seq":146,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.251504+0000","last_change":"2024-09-17T07:06:36.857728+0000","last_active":"2024-09-17T07:07:23.251504+0000","last_peered":"2024-09-17T07:07:23.251504+0000","last_clean":"2024-09-17T07:07:23.251504+0000","last_became_active":"2024-09-17T07:03:48.758220+0000","last_became_peered":"2024-09-17T07:03:48.758220+0000","last_unstale":"2024-09-17T07:07:23.251504+0000","last_undegraded":"2024-09-17T07:07:23.251504+0000","last_fullsized":"2024-09-17T07:07:23.251504+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'4","last_scrub_stamp":"2024-09-17T07:06:36.857471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:06:36.857471+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:56.904497+0000","scrub_duration":0.0095623189999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":128,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-17T07:07:24.248801+0000","last_change":"2024-09-17T07:06:26.844807+0000","last_active":"2024-09-17T07:07:24.248801+0000","last_peered":"2024-09-17T07:07:24.248801+0000","last_clean":"2024-09-17T07:07:24.248801+0000","last_became_active":"2024-09-17T07:02:12.008898+0000","last_became_peered":"2024-09-17T07:02:12.008898+0000","last_unstale":"2024-09-17T07:07:24.248801+0000","last_undegraded":"2024-09-17T07:07:24.248801+0000","last_fullsized":"2024-09-17T07:07:24.248801+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:26.844664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:06:26.844664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:32.712860+0000","scrub_duration":0.0012489000000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":123,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.252617+0000","last_change":"2024-09-17T07:04:57.667081+0000","last_active":"2024-09-17T07:07:23.252617+0000","last_peered":"2024-09-17T07:07:23.252617+0000","last_clean":"2024-09-17T07:07:23.252617+0000","last_became_active":"2024-09-17T07:02:16.014982+0000","last_became_peered":"2024-09-17T07:02:16.014982+0000","last_unstale":"2024-09-17T07:07:23.252617+0000","last_undegraded":"2024-09-17T07:07:23.252617+0000","last_fullsized":"2024-09-17T07:07:23.252617+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:57.667006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:47.444972+0000","last_clean_scrub_stamp":"2024-09-17T07:04:57.667006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0011999210000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":122,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.257269+0000","last_change":"2024-09-17T07:06:40.865239+0000","last_active":"2024-09-17T07:07:23.257269+0000","last_peered":"2024-09-17T07:07:23.257269+0000","last_clean":"2024-09-17T07:07:23.257269+0000","last_became_active":"2024-09-17T07:02:18.021014+0000","last_became_peered":"2024-09-17T07:02:18.021014+0000","last_unstale":"2024-09-17T07:07:23.257269+0000","last_undegraded":"2024-09-17T07:07:23.257269+0000","last_fullsized":"2024-09-17T07:07:23.257269+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:40.865049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:06:40.865049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:48.155323+0000","scrub_duration":0.0029834100000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.12","version":"162'8291","reported_seq":13731,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-17T07:07:28.000769+0000","last_change":"2024-09-17T07:06:31.715791+0000","last_active":"2024-09-17T07:07:28.000769+0000","last_peered":"2024-09-17T07:07:28.000769+0000","last_clean":"2024-09-17T07:07:28.000769+0000","last_became_active":"2024-09-17T07:05:39.199163+0000","last_became_peered":"2024-09-17T07:05:39.199163+0000","last_unstale":"2024-09-17T07:07:28.000769+0000","last_undegraded":"2024-09-17T07:07:28.000769+0000","last_fullsized":"2024-09-17T07:07:28.000769+0000","mapping_epoch":129,"log_start":"105'1886","ondisk_log_start":"105'1886","created":111,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":5,"last_scrub":"150'6246","last_scrub_stamp":"2024-09-17T07:06:31.715683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:04:19.785747+0000","last_clean_scrub_stamp":"2024-09-17T07:06:31.715683+0000","objects_scrubbed":2410,"log_size":6405,"log_dups_size":0,"ondisk_log_size":6405,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:52.906727+0000","scrub_duration":0.55965373699999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23199744,"num_objects":354,"num_object_clones":0,"num_object_copies":1062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":354,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2056,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"75'11","reported_seq":144,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.254106+0000","last_change":"2024-09-17T07:04:52.733713+0000","last_active":"2024-09-17T07:07:23.254106+0000","last_peered":"2024-09-17T07:07:23.254106+0000","last_clean":"2024-09-17T07:07:23.254106+0000","last_became_active":"2024-09-17T07:02:14.027316+0000","last_became_peered":"2024-09-17T07:02:14.027316+0000","last_unstale":"2024-09-17T07:07:23.254106+0000","last_undegraded":"2024-09-17T07:07:23.254106+0000","last_fullsized":"2024-09-17T07:07:23.254106+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-09-17T07:04:52.733516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:04:52.733516+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.013209306000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":138,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.253115+0000","last_change":"2024-09-17T07:06:44.298186+0000","last_active":"2024-09-17T07:07:23.253115+0000","last_peered":"2024-09-17T07:07:23.253115+0000","last_clean":"2024-09-17T07:07:23.253115+0000","last_became_active":"2024-09-17T07:02:12.010195+0000","last_became_peered":"2024-09-17T07:02:12.010195+0000","last_unstale":"2024-09-17T07:07:23.253115+0000","last_undegraded":"2024-09-17T07:07:23.253115+0000","last_fullsized":"2024-09-17T07:07:23.253115+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:44.298065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:06:44.298065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:00.552155+0000","scrub_duration":0.0032727260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":124,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.257122+0000","last_change":"2024-09-17T07:06:33.856836+0000","last_active":"2024-09-17T07:07:23.257122+0000","last_peered":"2024-09-17T07:07:23.257122+0000","last_clean":"2024-09-17T07:07:23.257122+0000","last_became_active":"2024-09-17T07:02:16.016183+0000","last_became_peered":"2024-09-17T07:02:16.016183+0000","last_unstale":"2024-09-17T07:07:23.257122+0000","last_undegraded":"2024-09-17T07:07:23.257122+0000","last_fullsized":"2024-09-17T07:07:23.257122+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:33.856556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:06:33.856556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:00.460166+0000","scrub_duration":0.0017432000000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":121,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-17T07:07:26.832533+0000","last_change":"2024-09-17T07:07:26.832785+0000","last_active":"2024-09-17T07:07:26.832533+0000","last_peered":"2024-09-17T07:07:26.832533+0000","last_clean":"2024-09-17T07:07:26.832533+0000","last_became_active":"2024-09-17T07:02:18.365656+0000","last_became_peered":"2024-09-17T07:02:18.365656+0000","last_unstale":"2024-09-17T07:07:26.832533+0000","last_undegraded":"2024-09-17T07:07:26.832533+0000","last_fullsized":"2024-09-17T07:07:26.832533+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:26.832514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:07:26.832514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:29.955192+0000","scrub_duration":0.01115028,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"75'1","reported_seq":123,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.257710+0000","last_change":"2024-09-17T07:06:32.836489+0000","last_active":"2024-09-17T07:07:23.257710+0000","last_peered":"2024-09-17T07:07:23.257710+0000","last_clean":"2024-09-17T07:07:23.257710+0000","last_became_active":"2024-09-17T07:02:18.021855+0000","last_became_peered":"2024-09-17T07:02:18.021855+0000","last_unstale":"2024-09-17T07:07:23.257710+0000","last_undegraded":"2024-09-17T07:07:23.257710+0000","last_fullsized":"2024-09-17T07:07:23.257710+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-17T07:06:32.836363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:06:32.836363+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:34.947914+0000","scrub_duration":0.0056683599999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":139,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-17T07:07:24.248094+0000","last_change":"2024-09-17T07:07:22.782091+0000","last_active":"2024-09-17T07:07:24.248094+0000","last_peered":"2024-09-17T07:07:24.248094+0000","last_clean":"2024-09-17T07:07:24.248094+0000","last_became_active":"2024-09-17T07:02:12.005329+0000","last_became_peered":"2024-09-17T07:02:12.005329+0000","last_unstale":"2024-09-17T07:07:24.248094+0000","last_undegraded":"2024-09-17T07:07:24.248094+0000","last_fullsized":"2024-09-17T07:07:24.248094+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:22.781739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:07:22.781739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:44.511386+0000","scrub_duration":0.0037808680000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"75'10","reported_seq":141,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.255833+0000","last_change":"2024-09-17T07:07:22.081513+0000","last_active":"2024-09-17T07:07:23.255833+0000","last_peered":"2024-09-17T07:07:23.255833+0000","last_clean":"2024-09-17T07:07:23.255833+0000","last_became_active":"2024-09-17T07:02:14.028731+0000","last_became_peered":"2024-09-17T07:02:14.028731+0000","last_unstale":"2024-09-17T07:07:23.255833+0000","last_undegraded":"2024-09-17T07:07:23.255833+0000","last_fullsized":"2024-09-17T07:07:23.255833+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-09-17T07:07:22.081382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:07:22.081382+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:37.716407+0000","scrub_duration":0.056222292,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"75'8","reported_seq":198,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-17T07:07:24.248421+0000","last_change":"2024-09-17T07:07:19.425224+0000","last_active":"2024-09-17T07:07:24.248421+0000","last_peered":"2024-09-17T07:07:24.248421+0000","last_clean":"2024-09-17T07:07:24.248421+0000","last_became_active":"2024-09-17T07:02:16.020560+0000","last_became_peered":"2024-09-17T07:02:16.020560+0000","last_unstale":"2024-09-17T07:07:24.248421+0000","last_undegraded":"2024-09-17T07:07:24.248421+0000","last_fullsized":"2024-09-17T07:07:24.248421+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:07:19.423853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:07:19.423853+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:23.974900+0000","scrub_duration":0.0037511480000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":110,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.270916+0000","last_change":"2024-09-17T07:03:58.948312+0000","last_active":"2024-09-17T07:07:23.270916+0000","last_peered":"2024-09-17T07:07:23.270916+0000","last_clean":"2024-09-17T07:07:23.270916+0000","last_became_active":"2024-09-17T07:02:18.021603+0000","last_became_peered":"2024-09-17T07:02:18.021603+0000","last_unstale":"2024-09-17T07:07:23.270916+0000","last_undegraded":"2024-09-17T07:07:23.270916+0000","last_fullsized":"2024-09-17T07:07:23.270916+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:58.948227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:03:58.948227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0018946010000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":127,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.254911+0000","last_change":"2024-09-17T07:04:34.679969+0000","last_active":"2024-09-17T07:07:23.254911+0000","last_peered":"2024-09-17T07:07:23.254911+0000","last_clean":"2024-09-17T07:07:23.254911+0000","last_became_active":"2024-09-17T07:02:12.009837+0000","last_became_peered":"2024-09-17T07:02:12.009837+0000","last_unstale":"2024-09-17T07:07:23.254911+0000","last_undegraded":"2024-09-17T07:07:23.254911+0000","last_fullsized":"2024-09-17T07:07:23.254911+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:34.679855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:04:34.679855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0053726590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"75'11","reported_seq":176,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.253074+0000","last_change":"2024-09-17T07:06:34.254104+0000","last_active":"2024-09-17T07:07:23.253074+0000","last_peered":"2024-09-17T07:07:23.253074+0000","last_clean":"2024-09-17T07:07:23.253074+0000","last_became_active":"2024-09-17T07:03:48.760390+0000","last_became_peered":"2024-09-17T07:03:48.760390+0000","last_unstale":"2024-09-17T07:07:23.253074+0000","last_undegraded":"2024-09-17T07:07:23.253074+0000","last_fullsized":"2024-09-17T07:07:23.253074+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-09-17T07:06:34.253977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:06:34.253977+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:36.450626+0000","scrub_duration":0.002455334,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":113,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.255754+0000","last_change":"2024-09-17T07:04:02.220014+0000","last_active":"2024-09-17T07:07:23.255754+0000","last_peered":"2024-09-17T07:07:23.255754+0000","last_clean":"2024-09-17T07:07:23.255754+0000","last_became_active":"2024-09-17T07:02:16.019701+0000","last_became_peered":"2024-09-17T07:02:16.019701+0000","last_unstale":"2024-09-17T07:07:23.255754+0000","last_undegraded":"2024-09-17T07:07:23.255754+0000","last_fullsized":"2024-09-17T07:07:23.255754+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:02.219927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:04:02.219927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015145639999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":123,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-17T07:07:30.779462+0000","last_change":"2024-09-17T07:07:30.779462+0000","last_active":"2024-09-17T07:07:30.779462+0000","last_peered":"2024-09-17T07:07:30.779462+0000","last_clean":"2024-09-17T07:07:30.779462+0000","last_became_active":"2024-09-17T07:02:18.363752+0000","last_became_peered":"2024-09-17T07:02:18.363752+0000","last_unstale":"2024-09-17T07:07:30.779462+0000","last_undegraded":"2024-09-17T07:07:30.779462+0000","last_fullsized":"2024-09-17T07:07:30.779462+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:30.779382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:07:30.779382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:38.926296+0000","scrub_duration":0.0011291059999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":147,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.252824+0000","last_change":"2024-09-17T07:06:41.312034+0000","last_active":"2024-09-17T07:07:23.252824+0000","last_peered":"2024-09-17T07:07:23.252824+0000","last_clean":"2024-09-17T07:07:23.252824+0000","last_became_active":"2024-09-17T07:03:48.759384+0000","last_became_peered":"2024-09-17T07:03:48.759384+0000","last_unstale":"2024-09-17T07:07:23.252824+0000","last_undegraded":"2024-09-17T07:07:23.252824+0000","last_fullsized":"2024-09-17T07:07:23.252824+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:41.311912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:06:41.311912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:55.289346+0000","scrub_duration":0.016335471000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"75'15","reported_seq":126,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.270610+0000","last_change":"2024-09-17T07:04:38.880694+0000","last_active":"2024-09-17T07:07:23.270610+0000","last_peered":"2024-09-17T07:07:23.270610+0000","last_clean":"2024-09-17T07:07:23.270610+0000","last_became_active":"2024-09-17T07:04:15.114286+0000","last_became_peered":"2024-09-17T07:04:15.114286+0000","last_unstale":"2024-09-17T07:07:23.270610+0000","last_undegraded":"2024-09-17T07:07:23.270610+0000","last_fullsized":"2024-09-17T07:07:23.270610+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-09-17T07:04:38.880277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:04:38.880277+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0059818789999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":123,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.271140+0000","last_change":"2024-09-17T07:06:42.201934+0000","last_active":"2024-09-17T07:07:23.271140+0000","last_peered":"2024-09-17T07:07:23.271140+0000","last_clean":"2024-09-17T07:07:23.271140+0000","last_became_active":"2024-09-17T07:02:16.015903+0000","last_became_peered":"2024-09-17T07:02:16.015903+0000","last_unstale":"2024-09-17T07:07:23.271140+0000","last_undegraded":"2024-09-17T07:07:23.271140+0000","last_fullsized":"2024-09-17T07:07:23.271140+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:42.200604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:06:42.200604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:54.763122+0000","scrub_duration":0.051542886000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":110,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.246398+0000","last_change":"2024-09-17T07:04:10.606687+0000","last_active":"2024-09-17T07:07:23.246398+0000","last_peered":"2024-09-17T07:07:23.246398+0000","last_clean":"2024-09-17T07:07:23.246398+0000","last_became_active":"2024-09-17T07:02:18.366466+0000","last_became_peered":"2024-09-17T07:02:18.366466+0000","last_unstale":"2024-09-17T07:07:23.246398+0000","last_undegraded":"2024-09-17T07:07:23.246398+0000","last_fullsized":"2024-09-17T07:07:23.246398+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:10.606552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:04:10.606552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00165456,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":116,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.245614+0000","last_change":"2024-09-17T07:03:44.446996+0000","last_active":"2024-09-17T07:07:23.245614+0000","last_peered":"2024-09-17T07:07:23.245614+0000","last_clean":"2024-09-17T07:07:23.245614+0000","last_became_active":"2024-09-17T07:02:12.005493+0000","last_became_peered":"2024-09-17T07:02:12.005493+0000","last_unstale":"2024-09-17T07:07:23.245614+0000","last_undegraded":"2024-09-17T07:07:23.245614+0000","last_fullsized":"2024-09-17T07:07:23.245614+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:44.446914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:44.446914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0029497619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"86'17","reported_seq":153,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T07:07:23.245097+0000","last_change":"2024-09-17T07:06:27.962393+0000","last_active":"2024-09-17T07:07:23.245097+0000","last_peered":"2024-09-17T07:07:23.245097+0000","last_clean":"2024-09-17T07:07:23.245097+0000","last_became_active":"2024-09-17T07:02:14.017737+0000","last_became_peered":"2024-09-17T07:02:14.017737+0000","last_unstale":"2024-09-17T07:07:23.245097+0000","last_undegraded":"2024-09-17T07:07:23.245097+0000","last_fullsized":"2024-09-17T07:07:23.245097+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'17","last_scrub_stamp":"2024-09-17T07:06:27.962276+0000","last_deep_scrub":"86'17","last_deep_scrub_stamp":"2024-09-17T07:03:18.241067+0000","last_clean_scrub_stamp":"2024-09-17T07:06:27.962276+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:48.911685+0000","scrub_duration":0.0050727970000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":125,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-17T07:07:24.242608+0000","last_change":"2024-09-17T07:05:00.800939+0000","last_active":"2024-09-17T07:07:24.242608+0000","last_peered":"2024-09-17T07:07:24.242608+0000","last_clean":"2024-09-17T07:07:24.242608+0000","last_became_active":"2024-09-17T07:02:16.017362+0000","last_became_peered":"2024-09-17T07:02:16.017362+0000","last_unstale":"2024-09-17T07:07:24.242608+0000","last_undegraded":"2024-09-17T07:07:24.242608+0000","last_fullsized":"2024-09-17T07:07:24.242608+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:05:00.800823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:05:00.800823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014180900000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":26,"stat_sum":{"num_bytes":616857624,"num_objects":9415,"num_object_clones":0,"num_object_copies":28245,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9415,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":160922,"num_write_kb":4863777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15823,"num_bytes_recovered":1036910592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1910124544,"data_stored":1910112328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132343,"ondisk_log_size":132343,"up":78,"acting":78,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66,"ondisk_log_size":66,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":697,"num_read_kb":454,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":421,"ondisk_log_size":421,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":309,"num_read_kb":304,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1386,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":12,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":4158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037829,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":491724,"kb_used_data":289988,"kb_used_omap":1,"kb_used_meta":201726,"kb_avail":93253428,"statfs":{"total":95995035648,"available":95491510272,"internally_reserved":0,"allocated":296947712,"data_stored":296598679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":206567883},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:07:13 2024","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":1.897,"15min":0.98499999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":3.613,"5min":52.478999999999999,"15min":52.478999999999999},"last":0.192},{"interface":"front","average":{"1min":0.98399999999999999,"5min":1.8979999999999999,"15min":0.97399999999999998},"min":{"1min":0.25900000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":2.9449999999999998,"5min":52.871000000000002,"15min":52.871000000000002},"last":0.44600000000000001}]},{"osd":1,"last update":"Tue Sep 17 07:07:13 2024","interfaces":[{"interface":"back","average":{"1min":1.2869999999999999,"5min":0.71799999999999997,"15min":0.57799999999999996},"min":{"1min":0.25600000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4.5140000000000002,"5min":4.5140000000000002,"15min":4.5140000000000002},"last":0.96299999999999997},{"interface":"front","average":{"1min":1.2270000000000001,"5min":0.74399999999999999,"15min":0.59699999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":3.8399999999999999,"5min":4.5039999999999996,"15min":4.5039999999999996},"last":1.9650000000000001}]},{"osd":2,"last update":"Tue Sep 17 07:07:13 2024","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":1.2190000000000001,"15min":0.72499999999999998},"min":{"1min":0.38700000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":3.3759999999999999,"5min":52.055999999999997,"15min":52.055999999999997},"last":0.245},{"interface":"front","average":{"1min":0.93000000000000005,"5min":1.256,"15min":0.78000000000000003},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":4.4749999999999996,"5min":52.064999999999998,"15min":52.064999999999998},"last":1.6339999999999999}]},{"osd":3,"last update":"Tue Sep 17 07:07:13 2024","interfaces":[{"interface":"back","average":{"1min":1.0720000000000001,"5min":1.5720000000000001,"15min":0.874},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":3.9550000000000001,"5min":51.689,"15min":51.689},"last":0.621},{"interface":"front","average":{"1min":1.54,"5min":1.8340000000000001,"15min":0.97799999999999998},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":3.5659999999999998,"5min":52.145000000000003,"15min":52.145000000000003},"last":1.171}]},{"osd":4,"last update":"Tue Sep 17 07:07:13 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.436,"15min":0.44500000000000001},"min":{"1min":0.25800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":1.5980000000000001,"5min":1.5980000000000001,"15min":1.5980000000000001},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.48999999999999999,"15min":0.44900000000000001},"min":{"1min":0.26800000000000002,"5min":0.248,"15min":0.248},"max":{"1min":3.4089999999999998,"5min":3.4089999999999998,"15min":3.4089999999999998},"last":0.32600000000000001}]},{"osd":5,"last update":"Tue Sep 17 07:07:13 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.48799999999999999,"15min":0.45100000000000001},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":3.3620000000000001,"5min":3.3620000000000001,"15min":3.3620000000000001},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.47799999999999998,"15min":0.46600000000000003},"min":{"1min":0.311,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.6459999999999999,"5min":1.6459999999999999,"15min":1.6459999999999999},"last":0.50600000000000001}]},{"osd":6,"last update":"Tue Sep 17 07:07:13 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.53200000000000003,"15min":0.50800000000000001},"min":{"1min":0.34499999999999997,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":1.675,"5min":1.675,"15min":1.675},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.91500000000000004,"5min":0.56699999999999995,"15min":0.51300000000000001},"min":{"1min":0.28899999999999998,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":3.2669999999999999,"5min":3.2669999999999999,"15min":3.2669999999999999},"last":0.34999999999999998}]}]},{"osd":6,"up_from":50,"seq":214748364872,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":414016,"kb_used_data":270520,"kb_used_omap":1,"kb_used_meta":143486,"kb_avail":93331136,"statfs":{"total":95995035648,"available":95571083264,"internally_reserved":0,"allocated":277012480,"data_stored":276675066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":146930119},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:06:50 2024","interfaces":[{"interface":"back","average":{"1min":7.2679999999999998,"5min":2.5960000000000001,"15min":1.198},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":75.323999999999998,"5min":75.323999999999998,"15min":75.323999999999998},"last":2.3999999999999999},{"interface":"front","average":{"1min":7.1449999999999996,"5min":2.5489999999999999,"15min":1.177},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":75.168999999999997,"5min":75.168999999999997,"15min":75.168999999999997},"last":1.637}]},{"osd":1,"last update":"Tue Sep 17 07:06:50 2024","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":0.58999999999999997,"15min":0.52200000000000002},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":4.4790000000000001,"5min":4.4790000000000001,"15min":4.4790000000000001},"last":0.98499999999999999},{"interface":"front","average":{"1min":0.89900000000000002,"5min":0.68999999999999995,"15min":0.58599999999999997},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":2.73,"5min":5.9530000000000003,"15min":5.9530000000000003},"last":1.304}]},{"osd":2,"last update":"Tue Sep 17 07:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":0.97099999999999997,"15min":0.67100000000000004},"min":{"1min":0.28100000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":4.9850000000000003,"5min":33.079000000000001,"15min":33.079000000000001},"last":1.329},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.97699999999999998,"15min":0.67200000000000004},"min":{"1min":0.32700000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":2.8759999999999999,"5min":33.100000000000001,"15min":33.100000000000001},"last":1.087}]},{"osd":3,"last update":"Tue Sep 17 07:06:50 2024","interfaces":[{"interface":"back","average":{"1min":3.544,"5min":1.6419999999999999,"15min":0.86699999999999999},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":52.944000000000003,"5min":52.944000000000003,"15min":52.944000000000003},"last":0.95299999999999996},{"interface":"front","average":{"1min":3.3719999999999999,"5min":1.5700000000000001,"15min":0.88300000000000001},"min":{"1min":0.36199999999999999,"5min":0.308,"15min":0.308},"max":{"1min":52.747,"5min":52.747,"15min":52.747},"last":2.3239999999999998}]},{"osd":4,"last update":"Tue Sep 17 07:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.41999999999999998,"15min":0.41999999999999998},"min":{"1min":0.25900000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":1.054},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.48199999999999998,"15min":0.432},"min":{"1min":0.27200000000000002,"5min":0.25,"15min":0.25},"max":{"1min":2.9119999999999999,"5min":2.9119999999999999,"15min":2.9119999999999999},"last":1.4019999999999999}]},{"osd":5,"last update":"Tue Sep 17 07:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.45500000000000002,"15min":0.433},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":1.6419999999999999,"5min":1.6419999999999999,"15min":1.6419999999999999},"last":1.1910000000000001},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.497,"15min":0.45000000000000001},"min":{"1min":0.29899999999999999,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":2.9390000000000001,"5min":2.9390000000000001,"15min":2.9390000000000001},"last":1.032}]},{"osd":7,"last update":"Tue Sep 17 07:07:11 2024","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.55500000000000005,"15min":0.51500000000000001},"min":{"1min":0.307,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":2.859,"5min":2.859,"15min":2.859},"last":1.006},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.57399999999999995,"15min":0.54200000000000004},"min":{"1min":0.38100000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.9339999999999999,"5min":1.9339999999999999,"15min":1.9339999999999999},"last":1.1140000000000001}]}]},{"osd":5,"up_from":42,"seq":180388626510,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":204360,"kb_used_data":78912,"kb_used_omap":1,"kb_used_meta":125438,"kb_avail":93540792,"statfs":{"total":95995035648,"available":95785771008,"internally_reserved":0,"allocated":80805888,"data_stored":80467459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":128448966},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:07:31 2024","interfaces":[{"interface":"back","average":{"1min":2.9020000000000001,"5min":2.6400000000000001,"15min":1.226},"min":{"1min":0.23499999999999999,"5min":0.221,"15min":0.221},"max":{"1min":28.727,"5min":49.265000000000001,"15min":49.265000000000001},"last":2.718},{"interface":"front","average":{"1min":2.7639999999999998,"5min":2.7450000000000001,"15min":1.2549999999999999},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":27.824999999999999,"5min":47.350000000000001,"15min":47.350000000000001},"last":0.47399999999999998}]},{"osd":1,"last update":"Tue Sep 17 07:07:31 2024","interfaces":[{"interface":"back","average":{"1min":1.766,"5min":0.79000000000000004,"15min":0.628},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":20.329000000000001,"5min":20.329000000000001,"15min":20.329000000000001},"last":0.29999999999999999},{"interface":"front","average":{"1min":1.9450000000000001,"5min":0.97099999999999997,"15min":1.121},"min":{"1min":0.27300000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":20.408999999999999,"5min":20.408999999999999,"15min":20.408999999999999},"last":2.1269999999999998}]},{"osd":2,"last update":"Tue Sep 17 07:07:31 2024","interfaces":[{"interface":"back","average":{"1min":2.0390000000000001,"5min":1.1100000000000001,"15min":0.748},"min":{"1min":0.34200000000000003,"5min":0.252,"15min":0.252},"max":{"1min":20.486000000000001,"5min":20.486000000000001,"15min":20.486000000000001},"last":0.34200000000000003},{"interface":"front","average":{"1min":2.1739999999999999,"5min":1.119,"15min":0.71199999999999997},"min":{"1min":0.26300000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":23.925999999999998,"5min":23.925999999999998,"15min":23.925999999999998},"last":0.35099999999999998}]},{"osd":3,"last update":"Tue Sep 17 07:07:31 2024","interfaces":[{"interface":"back","average":{"1min":6.8310000000000004,"5min":2.077,"15min":1.0609999999999999},"min":{"1min":0.25800000000000001,"5min":0.249,"15min":0.249},"max":{"1min":68.978999999999999,"5min":68.978999999999999,"15min":68.978999999999999},"last":0.36399999999999999},{"interface":"front","average":{"1min":6.851,"5min":2,"15min":1.0329999999999999},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":69.084000000000003,"5min":69.084000000000003,"15min":69.084000000000003},"last":1.6180000000000001}]},{"osd":4,"last update":"Tue Sep 17 07:07:31 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.436,"15min":0.41099999999999998},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":3.2989999999999999,"5min":3.2989999999999999,"15min":3.2989999999999999},"last":0.22},{"interface":"front","average":{"1min":0.439,"5min":0.442,"15min":0.437},"min":{"1min":0.26700000000000002,"5min":0.248,"15min":0.248},"max":{"1min":0.90900000000000003,"5min":2.2160000000000002,"15min":2.2160000000000002},"last":0.28199999999999997}]},{"osd":6,"last update":"Tue Sep 17 07:06:42 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.42999999999999999,"15min":0.42699999999999999},"min":{"1min":0.27500000000000002,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.23499999999999999},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.48699999999999999,"15min":0.45900000000000002},"min":{"1min":0.31900000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":2.5590000000000002,"5min":2.5590000000000002,"15min":2.5590000000000002},"last":0.39100000000000001}]},{"osd":7,"last update":"Tue Sep 17 07:07:06 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.51000000000000001,"15min":0.49099999999999999},"min":{"1min":0.307,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":2.5910000000000002,"5min":2.5910000000000002,"15min":2.5910000000000002},"last":0.318},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.47899999999999998,"15min":0.46899999999999997},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.27100000000000002}]}]},{"osd":4,"up_from":33,"seq":141733920852,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":171480,"kb_used_data":59980,"kb_used_omap":1,"kb_used_meta":111486,"kb_avail":93573672,"statfs":{"total":95995035648,"available":95819440128,"internally_reserved":0,"allocated":61419520,"data_stored":61068117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":114162118},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:06:56 2024","interfaces":[{"interface":"back","average":{"1min":3.1600000000000001,"5min":2.6579999999999999,"15min":1.274},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":48.825000000000003,"5min":74.551000000000002,"15min":74.551000000000002},"last":1.474},{"interface":"front","average":{"1min":3.758,"5min":2.839,"15min":1.361},"min":{"1min":0.27500000000000002,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":50.911999999999999,"5min":74.778999999999996,"15min":74.778999999999996},"last":0.371}]},{"osd":1,"last update":"Tue Sep 17 07:06:56 2024","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":0.628,"15min":0.59499999999999997},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4.1539999999999999,"5min":4.1539999999999999,"15min":4.1539999999999999},"last":2.6549999999999998},{"interface":"front","average":{"1min":1.222,"5min":0.66900000000000004,"15min":0.624},"min":{"1min":0.25600000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":3.5409999999999999,"5min":3.5409999999999999,"15min":3.5409999999999999},"last":1.4339999999999999}]},{"osd":2,"last update":"Tue Sep 17 07:06:56 2024","interfaces":[{"interface":"back","average":{"1min":3.7629999999999999,"5min":1.2490000000000001,"15min":0.81000000000000005},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":49.651000000000003,"5min":49.651000000000003,"15min":49.651000000000003},"last":2.2610000000000001},{"interface":"front","average":{"1min":3.9950000000000001,"5min":1.3080000000000001,"15min":0.83199999999999996},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":48.813000000000002,"5min":48.813000000000002,"15min":48.813000000000002},"last":2.4590000000000001}]},{"osd":3,"last update":"Tue Sep 17 07:06:56 2024","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.194,"15min":0.80800000000000005},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":3.4590000000000001,"5min":39.042999999999999,"15min":39.042999999999999},"last":0.34399999999999997},{"interface":"front","average":{"1min":1.0009999999999999,"5min":1.2490000000000001,"15min":0.79400000000000004},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":4.5679999999999996,"5min":38.920000000000002,"15min":38.920000000000002},"last":6.157}]},{"osd":5,"last update":"Tue Sep 17 07:07:27 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.47499999999999998,"15min":0.41199999999999998},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":1.9510000000000001,"5min":2.4780000000000002,"15min":2.4780000000000002},"last":0.314},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.48599999999999999,"15min":0.42999999999999999},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":2.0299999999999998,"5min":2.5419999999999998,"15min":2.5419999999999998},"last":0.39300000000000002}]},{"osd":6,"last update":"Tue Sep 17 07:06:45 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.48799999999999999,"15min":0.42499999999999999},"min":{"1min":0.26400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":3.7770000000000001,"5min":3.7770000000000001,"15min":3.7770000000000001},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.48299999999999998,"15min":0.42999999999999999},"min":{"1min":0.30599999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":3.081,"5min":3.081,"15min":3.081},"last":1.6080000000000001}]},{"osd":7,"last update":"Tue Sep 17 07:07:07 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.54000000000000004,"15min":0.51400000000000001},"min":{"1min":0.28899999999999998,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":2.637,"5min":2.637,"15min":2.637},"last":1.6839999999999999},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.53200000000000003,"15min":0.497},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":3.7400000000000002,"5min":3.7400000000000002,"15min":3.7400000000000002},"last":0.35699999999999998}]}]},{"osd":3,"up_from":27,"seq":115964117077,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":495008,"kb_used_data":319512,"kb_used_omap":1,"kb_used_meta":175486,"kb_avail":93250144,"statfs":{"total":95995035648,"available":95488147456,"internally_reserved":0,"allocated":327180288,"data_stored":326810446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":179698120},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:06:43 2024","interfaces":[{"interface":"back","average":{"1min":1.3080000000000001,"5min":1.2,"15min":0.64200000000000002},"min":{"1min":0.20499999999999999,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":4.1879999999999997,"5min":17.364000000000001,"15min":17.364000000000001},"last":1.3109999999999999},{"interface":"front","average":{"1min":1.681,"5min":1.1339999999999999,"15min":0.63},"min":{"1min":0.38800000000000001,"5min":0.189,"15min":0.189},"max":{"1min":4.3259999999999996,"5min":18.521000000000001,"15min":18.521000000000001},"last":0.39600000000000002}]},{"osd":1,"last update":"Tue Sep 17 07:06:43 2024","interfaces":[{"interface":"back","average":{"1min":1.2210000000000001,"5min":0.79100000000000004,"15min":0.53600000000000003},"min":{"1min":0.32300000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":2.988,"5min":5.4619999999999997,"15min":5.4619999999999997},"last":0.27200000000000002},{"interface":"front","average":{"1min":1.454,"5min":1.0149999999999999,"15min":0.63900000000000001},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":4.0549999999999997,"5min":8.0670000000000002,"15min":8.0670000000000002},"last":1.3480000000000001}]},{"osd":2,"last update":"Tue Sep 17 07:06:43 2024","interfaces":[{"interface":"back","average":{"1min":1.694,"5min":1.204,"15min":0.68400000000000005},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":4.4359999999999999,"5min":9.7319999999999993,"15min":9.7319999999999993},"last":4.827},{"interface":"front","average":{"1min":1.8700000000000001,"5min":1.026,"15min":0.622},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":7.8470000000000004,"5min":8.3350000000000009,"15min":8.3350000000000009},"last":4.5049999999999999}]},{"osd":4,"last update":"Tue Sep 17 07:06:58 2024","interfaces":[{"interface":"back","average":{"1min":1.54,"5min":0.96799999999999997,"15min":0.69999999999999996},"min":{"1min":0.38400000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":5.4539999999999997,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.42699999999999999},{"interface":"front","average":{"1min":1.167,"5min":0.81100000000000005,"15min":0.629},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":5.7850000000000001,"5min":5.7850000000000001,"15min":5.7850000000000001},"last":1.429}]},{"osd":5,"last update":"Tue Sep 17 07:07:24 2024","interfaces":[{"interface":"back","average":{"1min":1.988,"5min":1.2090000000000001,"15min":0.80900000000000005},"min":{"1min":0.309,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":5.3200000000000003,"5min":7.9669999999999996,"15min":7.9669999999999996},"last":0.53400000000000003},{"interface":"front","average":{"1min":1.3899999999999999,"5min":0.98399999999999999,"15min":0.71899999999999997},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":3.9039999999999999,"5min":4.4160000000000004,"15min":4.4160000000000004},"last":0.46800000000000003}]},{"osd":6,"last update":"Tue Sep 17 07:06:43 2024","interfaces":[{"interface":"back","average":{"1min":1.601,"5min":0.97599999999999998,"15min":0.70199999999999996},"min":{"1min":0.31900000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":5.9199999999999999,"5min":5.9199999999999999,"15min":5.9199999999999999},"last":0.56699999999999995},{"interface":"front","average":{"1min":1.2749999999999999,"5min":1.034,"15min":0.73799999999999999},"min":{"1min":0.376,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":4.3550000000000004,"5min":7.9329999999999998,"15min":7.9329999999999998},"last":0.59399999999999997}]},{"osd":7,"last update":"Tue Sep 17 07:07:04 2024","interfaces":[{"interface":"back","average":{"1min":1.7869999999999999,"5min":1.085,"15min":0.754},"min":{"1min":0.40699999999999997,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":4.5119999999999996,"5min":6.1440000000000001,"15min":6.1440000000000001},"last":0.44900000000000001},{"interface":"front","average":{"1min":1.6559999999999999,"5min":1.0669999999999999,"15min":0.72699999999999998},"min":{"1min":0.39300000000000002,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":5.8170000000000002,"5min":5.8170000000000002,"15min":5.8170000000000002},"last":1.4630000000000001}]}]},{"osd":2,"up_from":18,"seq":77309411419,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":521664,"kb_used_data":301816,"kb_used_omap":1,"kb_used_meta":219774,"kb_avail":93223488,"statfs":{"total":95995035648,"available":95460851712,"internally_reserved":0,"allocated":309059584,"data_stored":308722259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":225049030},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:07:27 2024","interfaces":[{"interface":"back","average":{"1min":2,"5min":0.95499999999999996,"15min":0.54900000000000004},"min":{"1min":0.248,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":7.2709999999999999,"5min":7.2709999999999999,"15min":7.2709999999999999},"last":2.5369999999999999},{"interface":"front","average":{"1min":1.716,"5min":1.3700000000000001,"15min":0.69199999999999995},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":5.9610000000000003,"5min":8.6869999999999994,"15min":8.6869999999999994},"last":0.56899999999999995}]},{"osd":1,"last update":"Tue Sep 17 07:07:27 2024","interfaces":[{"interface":"back","average":{"1min":1.5349999999999999,"5min":0.93600000000000005,"15min":0.54900000000000004},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":5.5010000000000003,"5min":5.5010000000000003,"15min":5.5010000000000003},"last":0.63800000000000001},{"interface":"front","average":{"1min":1.5880000000000001,"5min":0.84899999999999998,"15min":0.53300000000000003},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":6.9829999999999997,"5min":6.9829999999999997,"15min":6.9829999999999997},"last":0.50900000000000001}]},{"osd":3,"last update":"Tue Sep 17 07:06:47 2024","interfaces":[{"interface":"back","average":{"1min":1.319,"5min":0.746,"15min":0.54000000000000004},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":3.3889999999999998,"5min":3.3889999999999998,"15min":3.3889999999999998},"last":1.119},{"interface":"front","average":{"1min":2.294,"5min":1.129,"15min":0.66300000000000003},"min":{"1min":0.312,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":5.6180000000000003,"5min":5.6180000000000003,"15min":5.6180000000000003},"last":0.65700000000000003}]},{"osd":4,"last update":"Tue Sep 17 07:07:00 2024","interfaces":[{"interface":"back","average":{"1min":1.0609999999999999,"5min":0.877,"15min":0.67000000000000004},"min":{"1min":0.32000000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":2.653,"5min":5.2359999999999998,"15min":5.2359999999999998},"last":0.69299999999999995},{"interface":"front","average":{"1min":1.254,"5min":0.75900000000000001,"15min":0.64500000000000002},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":3.4689999999999999,"5min":3.4689999999999999,"15min":3.4689999999999999},"last":0.48499999999999999}]},{"osd":5,"last update":"Tue Sep 17 07:07:27 2024","interfaces":[{"interface":"back","average":{"1min":1.381,"5min":1.006,"15min":0.73199999999999998},"min":{"1min":0.26100000000000001,"5min":0.249,"15min":0.249},"max":{"1min":5.383,"5min":5.383,"15min":5.383},"last":0.59199999999999997},{"interface":"front","average":{"1min":1.121,"5min":0.85099999999999998,"15min":0.68100000000000005},"min":{"1min":0.34699999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":2.6880000000000002,"5min":5.3600000000000003,"15min":5.3600000000000003},"last":0.67000000000000004}]},{"osd":6,"last update":"Tue Sep 17 07:06:47 2024","interfaces":[{"interface":"back","average":{"1min":1.5589999999999999,"5min":0.84899999999999998,"15min":0.68500000000000005},"min":{"1min":0.30399999999999999,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":9.2509999999999994,"5min":9.2509999999999994,"15min":9.2509999999999994},"last":0.45300000000000001},{"interface":"front","average":{"1min":1.4850000000000001,"5min":0.96199999999999997,"15min":0.71799999999999997},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":3.6269999999999998,"5min":4.6580000000000004,"15min":4.6580000000000004},"last":0.61299999999999999}]},{"osd":7,"last update":"Tue Sep 17 07:07:17 2024","interfaces":[{"interface":"back","average":{"1min":1.1759999999999999,"5min":0.90500000000000003,"15min":0.68200000000000005},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":2.7090000000000001,"5min":5.3250000000000002,"15min":5.3250000000000002},"last":0.746},{"interface":"front","average":{"1min":1.9950000000000001,"5min":1.0129999999999999,"15min":0.85899999999999999},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":9.2919999999999998,"5min":9.2919999999999998,"15min":9.2919999999999998},"last":0.53700000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574944,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":343532,"kb_used_data":213924,"kb_used_omap":1,"kb_used_meta":129534,"kb_avail":93401620,"statfs":{"total":95995035648,"available":95643258880,"internally_reserved":0,"allocated":219058176,"data_stored":218677211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":132643270},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:06:51 2024","interfaces":[{"interface":"back","average":{"1min":1.3149999999999999,"5min":1.099,"15min":0.58999999999999997},"min":{"1min":0.26000000000000001,"5min":0.155,"15min":0.155},"max":{"1min":3.9929999999999999,"5min":13.837,"15min":13.837},"last":0.58199999999999996},{"interface":"front","average":{"1min":2.0600000000000001,"5min":1.1599999999999999,"15min":0.63},"min":{"1min":0.186,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":10.342000000000001,"5min":10.342000000000001,"15min":10.342000000000001},"last":1.794}]},{"osd":2,"last update":"Tue Sep 17 07:07:21 2024","interfaces":[{"interface":"back","average":{"1min":1.8,"5min":1.2150000000000001,"15min":0.67400000000000004},"min":{"1min":0.23799999999999999,"5min":0.192,"15min":0.192},"max":{"1min":5.9509999999999996,"5min":11.055999999999999,"15min":11.055999999999999},"last":0.61499999999999999},{"interface":"front","average":{"1min":1.4470000000000001,"5min":1.302,"15min":0.70999999999999996},"min":{"1min":0.24199999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":5.4269999999999996,"5min":9.3160000000000007,"15min":9.3160000000000007},"last":1.1659999999999999}]},{"osd":3,"last update":"Tue Sep 17 07:06:43 2024","interfaces":[{"interface":"back","average":{"1min":1.579,"5min":1.121,"15min":0.68500000000000005},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":7.2270000000000003,"5min":7.766,"15min":7.766},"last":1.1459999999999999},{"interface":"front","average":{"1min":1.6799999999999999,"5min":1.0860000000000001,"15min":0.66200000000000003},"min":{"1min":0.32900000000000001,"5min":0.222,"15min":0.222},"max":{"1min":4.548,"5min":19.602,"15min":19.602},"last":2.3279999999999998}]},{"osd":4,"last update":"Tue Sep 17 07:07:03 2024","interfaces":[{"interface":"back","average":{"1min":1.149,"5min":1.02,"15min":0.72199999999999998},"min":{"1min":0.29899999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":5.5030000000000001,"5min":7.8840000000000003,"15min":7.8840000000000003},"last":0.79100000000000004},{"interface":"front","average":{"1min":1.363,"5min":0.95199999999999996,"15min":0.69599999999999995},"min":{"1min":0.34200000000000003,"5min":0.307,"15min":0.307},"max":{"1min":7.1749999999999998,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.32000000000000001}]},{"osd":5,"last update":"Tue Sep 17 07:07:21 2024","interfaces":[{"interface":"back","average":{"1min":1.3129999999999999,"5min":1.087,"15min":0.74299999999999999},"min":{"1min":0.31,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":5.5259999999999998,"5min":6.0060000000000002,"15min":6.0060000000000002},"last":1.0509999999999999},{"interface":"front","average":{"1min":1.623,"5min":1.0600000000000001,"15min":0.72999999999999998},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":7.2080000000000002,"5min":8.8889999999999993,"15min":8.8889999999999993},"last":0.28899999999999998}]},{"osd":6,"last update":"Tue Sep 17 07:06:47 2024","interfaces":[{"interface":"back","average":{"1min":1.177,"5min":0.871,"15min":0.65300000000000002},"min":{"1min":0.36699999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":3.1920000000000002,"5min":8.843,"15min":8.843},"last":2.351},{"interface":"front","average":{"1min":1.111,"5min":0.93100000000000005,"15min":0.70799999999999996},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":3.746,"5min":5.7290000000000001,"15min":5.7290000000000001},"last":1.123}]},{"osd":7,"last update":"Tue Sep 17 07:07:07 2024","interfaces":[{"interface":"back","average":{"1min":1.5089999999999999,"5min":1.004,"15min":0.70999999999999996},"min":{"1min":0.38800000000000001,"5min":0.37,"15min":0.37},"max":{"1min":7.2320000000000002,"5min":8.9309999999999992,"15min":8.9309999999999992},"last":2.2930000000000001},{"interface":"front","average":{"1min":1.3120000000000001,"5min":1.0209999999999999,"15min":0.72099999999999997},"min":{"1min":0.372,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":3.5819999999999999,"5min":11.007999999999999,"15min":11.007999999999999},"last":0.64000000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738467,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":671572,"kb_used_data":459980,"kb_used_omap":1,"kb_used_meta":211582,"kb_avail":93073580,"statfs":{"total":95995035648,"available":95307345920,"internally_reserved":0,"allocated":471019520,"data_stored":470662483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":216660422},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 17 07:06:50 2024","interfaces":[{"interface":"back","average":{"1min":2.2610000000000001,"5min":1.0149999999999999,"15min":0.55700000000000005},"min":{"1min":0.22800000000000001,"5min":0.192,"15min":0.192},"max":{"1min":8.0259999999999998,"5min":8.0259999999999998,"15min":8.0259999999999998},"last":1.05},{"interface":"front","average":{"1min":1.8220000000000001,"5min":0.96199999999999997,"15min":0.54800000000000004},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":7.8849999999999998,"5min":7.8849999999999998,"15min":7.8849999999999998},"last":0.46500000000000002}]},{"osd":2,"last update":"Tue Sep 17 07:07:24 2024","interfaces":[{"interface":"back","average":{"1min":2.1619999999999999,"5min":1.7829999999999999,"15min":0.84599999999999997},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":18.643999999999998,"5min":18.643999999999998,"15min":18.643999999999998},"last":0.49099999999999999},{"interface":"front","average":{"1min":2.839,"5min":1.5369999999999999,"15min":0.754},"min":{"1min":0.27200000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":17.972999999999999,"5min":17.972999999999999,"15min":17.972999999999999},"last":2.903}]},{"osd":3,"last update":"Tue Sep 17 07:06:50 2024","interfaces":[{"interface":"back","average":{"1min":3.415,"5min":1.3340000000000001,"15min":0.73999999999999999},"min":{"1min":0.41599999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":18.629000000000001,"5min":18.629000000000001,"15min":18.629000000000001},"last":0.86799999999999999},{"interface":"front","average":{"1min":3.972,"5min":1.627,"15min":0.83499999999999996},"min":{"1min":0.45600000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":22.358000000000001,"5min":22.358000000000001,"15min":22.358000000000001},"last":1.1599999999999999}]},{"osd":4,"last update":"Tue Sep 17 07:07:02 2024","interfaces":[{"interface":"back","average":{"1min":3.0979999999999999,"5min":1.474,"15min":0.86499999999999999},"min":{"1min":0.45200000000000001,"5min":0.316,"15min":0.316},"max":{"1min":18.568000000000001,"5min":18.568000000000001,"15min":18.568000000000001},"last":1.093},{"interface":"front","average":{"1min":2.4380000000000002,"5min":1.2090000000000001,"15min":0.76700000000000002},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":18.765000000000001,"5min":18.765000000000001,"15min":18.765000000000001},"last":0.51400000000000001}]},{"osd":5,"last update":"Tue Sep 17 07:06:28 2024","interfaces":[{"interface":"back","average":{"1min":2.1070000000000002,"5min":0.97199999999999998,"15min":0.68899999999999995},"min":{"1min":0.52500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":5.0940000000000003,"5min":5.0940000000000003,"15min":5.0940000000000003},"last":1.119},{"interface":"front","average":{"1min":1.782,"5min":0.94099999999999995,"15min":0.67300000000000004},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":4.1529999999999996,"5min":4.1529999999999996,"15min":4.1529999999999996},"last":0.58399999999999996}]},{"osd":6,"last update":"Tue Sep 17 07:06:50 2024","interfaces":[{"interface":"back","average":{"1min":2.738,"5min":1.1850000000000001,"15min":0.75900000000000001},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":18.602,"5min":18.602,"15min":18.602},"last":0.36399999999999999},{"interface":"front","average":{"1min":3.198,"5min":1.3660000000000001,"15min":0.82099999999999995},"min":{"1min":0.48399999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":18.692,"5min":18.692,"15min":18.692},"last":1.1399999999999999}]},{"osd":7,"last update":"Tue Sep 17 07:07:13 2024","interfaces":[{"interface":"back","average":{"1min":4.6580000000000004,"5min":1.879,"15min":1.427},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":46.183,"5min":46.183,"15min":46.183},"last":2.9860000000000002},{"interface":"front","average":{"1min":4.8620000000000001,"5min":1.6819999999999999,"15min":0.93400000000000005},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":45.914999999999999,"5min":45.914999999999999,"15min":45.914999999999999},"last":0.42999999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":426119168,"data_stored":426115096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":206901248,"data_stored":206897176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":308084736,"data_stored":308084736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":294912000,"data_stored":294912000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":54788096,"data_stored":54788096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":75825152,"data_stored":75825152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":255594496,"data_stored":255590424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":287899648,"data_stored":287899648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T07:07:36.139 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-17T07:07:36.139 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T07:07:36.139 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-17T07:07:36.140 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-17T07:07:36.192 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:07:36.219 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:36.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:36.220 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:07:36.321 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:07:36.362 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:36.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:36.364 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:07:36.376 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:36 smithi100 ceph-mon[39462]: 5.15 deep-scrub starts 2024-09-17T07:07:36.376 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:36 smithi100 ceph-mon[39462]: 5.15 deep-scrub ok 2024-09-17T07:07:36.376 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:36 smithi100 ceph-mon[39462]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:07:36.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:36 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T07:07:36.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:36 smithi100 ceph-mon[39462]: osdmap e164: 8 total, 8 up, 8 in 2024-09-17T07:07:36.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:36 smithi100 ceph-mon[29092]: 5.15 deep-scrub starts 2024-09-17T07:07:36.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:36 smithi100 ceph-mon[29092]: 5.15 deep-scrub ok 2024-09-17T07:07:36.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:36 smithi100 ceph-mon[29092]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:07:36.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:36 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T07:07:36.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:36 smithi100 ceph-mon[29092]: osdmap e164: 8 total, 8 up, 8 in 2024-09-17T07:07:36.465 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:07:36.513 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:36.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:36.514 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:07:36.615 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:07:36.627 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Removed 110800 objects 2024-09-17T07:07:36.627 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Clean up completed and total clean up time :94.9693 2024-09-17T07:07:36.642 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:36.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:36.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:07:36.744 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:07:36.766 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:36.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:36.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:07:36.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:36 smithi135 ceph-mon[33238]: 5.15 deep-scrub starts 2024-09-17T07:07:36.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:36 smithi135 ceph-mon[33238]: 5.15 deep-scrub ok 2024-09-17T07:07:36.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:36 smithi135 ceph-mon[33238]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:07:36.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:36 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T07:07:36.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:36 smithi135 ceph-mon[33238]: osdmap e164: 8 total, 8 up, 8 in 2024-09-17T07:07:36.868 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:07:36.889 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:36.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:36.890 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:07:36.991 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:07:37.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:37.014 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:37.014 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:07:37.115 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:07:37.137 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:37.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:37.138 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:07:37.239 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:07:37.261 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:37.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:37.262 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:07:37.363 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:07:37.385 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:37.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:37.386 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:07:37.408 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:37 smithi100 ceph-mon[29092]: pgmap v234: 158 pgs: 158 active+clean; 246 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 757 op/s 2024-09-17T07:07:37.408 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:37 smithi100 ceph-mon[39462]: pgmap v234: 158 pgs: 158 active+clean; 246 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 757 op/s 2024-09-17T07:07:37.487 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:07:37.492 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:37 smithi135 ceph-mon[33238]: pgmap v234: 158 pgs: 158 active+clean; 246 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 757 op/s 2024-09-17T07:07:37.510 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:37.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:37.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:07:37.612 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:07:37.633 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:37.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:37.634 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:07:37.735 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:07:37.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:37.759 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:37.759 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:07:37.860 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:07:37.881 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:37.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:37.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:07:37.982 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:07:38.003 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:38.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:38.004 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:07:38.105 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:07:38.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:38.129 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:38.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:07:38.230 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:07:38.251 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:38.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:38.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:07:38.341 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:07:38 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:07:38.353 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:07:38.377 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:38.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:38.378 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:07:38.430 INFO:teuthology.orchestra.run.smithi135.stdout:{ 2024-09-17T07:07:38.430 INFO:teuthology.orchestra.run.smithi135.stdout: "ops": [], 2024-09-17T07:07:38.430 INFO:teuthology.orchestra.run.smithi135.stdout: "num_ops": 0 2024-09-17T07:07:38.430 INFO:teuthology.orchestra.run.smithi135.stdout:} 2024-09-17T07:07:38.479 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:07:38.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:38.502 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:38.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:07:38.603 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:07:38.626 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:38.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:38 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:07:38.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:38 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:07:38.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:38 smithi100 ceph-mon[39462]: osdmap e165: 8 total, 8 up, 8 in 2024-09-17T07:07:38.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:38 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:07:38.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:38 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:07:38.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:38 smithi100 ceph-mon[29092]: osdmap e165: 8 total, 8 up, 8 in 2024-09-17T07:07:38.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:38.628 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:07:38.728 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:07:38.750 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:38.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:38.751 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:07:38.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:38 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:07:38.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:38 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:07:38.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:38 smithi135 ceph-mon[33238]: osdmap e165: 8 total, 8 up, 8 in 2024-09-17T07:07:38.852 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:07:38.873 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:38.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:38.874 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:07:38.972 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:07:38.975 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:07:38.996 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:38.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:38.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:07:39.098 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:07:39.119 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:39.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:39.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:07:39.177 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-17T07:07:39.179 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-17T07:07:39.201 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:07:39.202 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:07:39.219 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1: {} 2024-09-17T07:07:39.219 INFO:teuthology.orchestra.run.smithi100.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:07:39.222 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:07:39.240 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:07:39.241 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:07:39.243 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:39.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:39.244 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:07:39.257 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:07:39.257 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:07:39.273 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:07:39.273 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:07:39.289 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:07:39.290 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:07:39.306 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:07:39.306 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:07:39.324 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:07:39.324 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:07:39.345 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:07:39.367 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:39.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:39.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:07:39.381 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:07:39.462 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:39 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:07:39.462 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:39 smithi100 ceph-mon[39462]: pgmap v236: 158 pgs: 158 active+clean; 12 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-17T07:07:39.462 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:39 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1731324228' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:07:39.463 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:39 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:07:39.463 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:39 smithi100 ceph-mon[29092]: pgmap v236: 158 pgs: 158 active+clean; 12 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-17T07:07:39.463 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:39 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1731324228' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:07:39.469 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:07:39.491 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:39.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:39.492 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:07:39.593 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:07:39.615 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:39.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:39.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:07:39.629 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:39 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:07:39.629 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:39 smithi135 ceph-mon[33238]: pgmap v236: 158 pgs: 158 active+clean; 12 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-17T07:07:39.629 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:39 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1731324228' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:07:39.718 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:07:39.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:39.742 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:39.742 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:07:39.843 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:07:39.885 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:39.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:39.886 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:07:39.987 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:07:40.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:40.021 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:40.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:07:40.123 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:07:40.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:40.152 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:40.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:07:40.253 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:07:40.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:40.277 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:40.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:07:40.378 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:07:40.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:40.400 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:40.400 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:07:40.501 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:07:40.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:40.524 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:40.525 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:07:40.625 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:07:40.648 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:40.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:40.649 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:07:40.707 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_0 2024-09-17T07:07:40.707 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-09-17T07:07:40.752 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:07:40.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:40.775 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:40.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:07:40.876 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:07:40.897 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:40.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:40.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:07:40.944 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:07:40.999 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:07:41.020 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:41.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:41.021 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:07:41.073 INFO:teuthology.orchestra.run.smithi135.stdout:ERROR: (22) Invalid argument 2024-09-17T07:07:41.073 INFO:teuthology.orchestra.run.smithi135.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:07:41.122 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:07:41.143 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:41.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:41.144 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:07:41.245 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:07:41.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:41.268 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:41.268 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:07:41.369 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:07:41.390 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:41.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:41.391 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:07:41.492 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:07:41.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:41.514 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:41.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:07:41.615 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:07:41.637 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:41.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:41.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:07:41.738 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:07:41.757 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-17T07:07:41.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:41.760 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:41.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:07:41.861 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:07:41.883 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:41.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:41.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:07:41.954 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:07:41.985 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:07:42.008 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:42.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:42.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:07:42.110 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:07:42.120 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:07:42.134 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:42.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:42.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:07:42.236 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:07:42.257 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:42.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:42.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:07:42.347 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:42 smithi135 ceph-mon[33238]: pgmap v237: 158 pgs: 158 active+clean; 12 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:07:42.357 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:42 smithi100 ceph-mon[29092]: pgmap v237: 158 pgs: 158 active+clean; 12 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:07:42.357 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:42 smithi100 ceph-mon[39462]: pgmap v237: 158 pgs: 158 active+clean; 12 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:07:42.359 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:07:42.381 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:42.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:42.382 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:07:42.482 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:07:42.504 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:42.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:42.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:07:42.605 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:07:42.626 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:42.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:42.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:07:42.728 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:07:42.750 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:42.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:42.751 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:07:42.851 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:07:42.873 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:42.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:42.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:07:42.918 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T07:07:42.918 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-17T07:07:42.919 INFO:teuthology.orchestra.run.smithi100.stdout: "epoch": 165, 2024-09-17T07:07:42.919 INFO:teuthology.orchestra.run.smithi100.stdout: "fsid": "e848267c-74c1-11ef-bceb-c7b262605968", 2024-09-17T07:07:42.919 INFO:teuthology.orchestra.run.smithi100.stdout: "created": "2024-09-17T06:57:05.010532+0000", 2024-09-17T07:07:42.919 INFO:teuthology.orchestra.run.smithi100.stdout: "modified": "2024-09-17T07:07:37.335513+0000", 2024-09-17T07:07:42.919 INFO:teuthology.orchestra.run.smithi100.stdout: "last_up_change": "2024-09-17T07:01:59.917118+0000", 2024-09-17T07:07:42.919 INFO:teuthology.orchestra.run.smithi100.stdout: "last_in_change": "2024-09-17T07:01:42.942454+0000", 2024-09-17T07:07:42.919 INFO:teuthology.orchestra.run.smithi100.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-17T07:07:42.919 INFO:teuthology.orchestra.run.smithi100.stdout: "flags_num": 5801984, 2024-09-17T07:07:42.919 INFO:teuthology.orchestra.run.smithi100.stdout: "flags_set": [ 2024-09-17T07:07:42.919 INFO:teuthology.orchestra.run.smithi100.stdout: "noscrub", 2024-09-17T07:07:42.919 INFO:teuthology.orchestra.run.smithi100.stdout: "pglog_hardlimit", 2024-09-17T07:07:42.919 INFO:teuthology.orchestra.run.smithi100.stdout: "purged_snapdirs", 2024-09-17T07:07:42.920 INFO:teuthology.orchestra.run.smithi100.stdout: "recovery_deletes", 2024-09-17T07:07:42.920 INFO:teuthology.orchestra.run.smithi100.stdout: "sortbitwise" 2024-09-17T07:07:42.920 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-17T07:07:42.920 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_version": 18, 2024-09-17T07:07:42.920 INFO:teuthology.orchestra.run.smithi100.stdout: "full_ratio": 0.94999998807907104, 2024-09-17T07:07:42.920 INFO:teuthology.orchestra.run.smithi100.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-17T07:07:42.920 INFO:teuthology.orchestra.run.smithi100.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-17T07:07:42.920 INFO:teuthology.orchestra.run.smithi100.stdout: "cluster_snapshot": "", 2024-09-17T07:07:42.920 INFO:teuthology.orchestra.run.smithi100.stdout: "pool_max": 7, 2024-09-17T07:07:42.920 INFO:teuthology.orchestra.run.smithi100.stdout: "max_osd": 8, 2024-09-17T07:07:42.920 INFO:teuthology.orchestra.run.smithi100.stdout: "require_min_compat_client": "luminous", 2024-09-17T07:07:42.920 INFO:teuthology.orchestra.run.smithi100.stdout: "min_compat_client": "luminous", 2024-09-17T07:07:42.921 INFO:teuthology.orchestra.run.smithi100.stdout: "require_osd_release": "reef", 2024-09-17T07:07:42.921 INFO:teuthology.orchestra.run.smithi100.stdout: "allow_crimson": false, 2024-09-17T07:07:42.921 INFO:teuthology.orchestra.run.smithi100.stdout: "pools": [ 2024-09-17T07:07:42.921 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:07:42.921 INFO:teuthology.orchestra.run.smithi100.stdout: "pool": 1, 2024-09-17T07:07:42.921 INFO:teuthology.orchestra.run.smithi100.stdout: "pool_name": ".mgr", 2024-09-17T07:07:42.921 INFO:teuthology.orchestra.run.smithi100.stdout: "create_time": "2024-09-17T07:00:08.816136+0000", 2024-09-17T07:07:42.921 INFO:teuthology.orchestra.run.smithi100.stdout: "flags": 1, 2024-09-17T07:07:42.921 INFO:teuthology.orchestra.run.smithi100.stdout: "flags_names": "hashpspool", 2024-09-17T07:07:42.921 INFO:teuthology.orchestra.run.smithi100.stdout: "type": 1, 2024-09-17T07:07:42.921 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 3, 2024-09-17T07:07:42.921 INFO:teuthology.orchestra.run.smithi100.stdout: "min_size": 2, 2024-09-17T07:07:42.921 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_rule": 0, 2024-09-17T07:07:42.922 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_count": 0, 2024-09-17T07:07:42.922 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_target": 0, 2024-09-17T07:07:42.922 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T07:07:42.922 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T07:07:42.922 INFO:teuthology.orchestra.run.smithi100.stdout: "object_hash": 2, 2024-09-17T07:07:42.922 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_autoscale_mode": "off", 2024-09-17T07:07:42.922 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num": 1, 2024-09-17T07:07:42.922 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_placement_num": 1, 2024-09-17T07:07:42.922 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_placement_num_target": 1, 2024-09-17T07:07:42.922 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num_target": 1, 2024-09-17T07:07:42.923 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num_pending": 1, 2024-09-17T07:07:42.923 INFO:teuthology.orchestra.run.smithi100.stdout: "last_pg_merge_meta": { 2024-09-17T07:07:42.923 INFO:teuthology.orchestra.run.smithi100.stdout: "source_pgid": "0.0", 2024-09-17T07:07:42.923 INFO:teuthology.orchestra.run.smithi100.stdout: "ready_epoch": 0, 2024-09-17T07:07:42.923 INFO:teuthology.orchestra.run.smithi100.stdout: "last_epoch_started": 0, 2024-09-17T07:07:42.923 INFO:teuthology.orchestra.run.smithi100.stdout: "last_epoch_clean": 0, 2024-09-17T07:07:42.923 INFO:teuthology.orchestra.run.smithi100.stdout: "source_version": "0'0", 2024-09-17T07:07:42.923 INFO:teuthology.orchestra.run.smithi100.stdout: "target_version": "0'0" 2024-09-17T07:07:42.923 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:07:42.923 INFO:teuthology.orchestra.run.smithi100.stdout: "last_change": "22", 2024-09-17T07:07:42.923 INFO:teuthology.orchestra.run.smithi100.stdout: "last_force_op_resend": "0", 2024-09-17T07:07:42.924 INFO:teuthology.orchestra.run.smithi100.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-17T07:07:42.924 INFO:teuthology.orchestra.run.smithi100.stdout: "last_force_op_resend_preluminous": "0", 2024-09-17T07:07:42.924 INFO:teuthology.orchestra.run.smithi100.stdout: "auid": 0, 2024-09-17T07:07:42.924 INFO:teuthology.orchestra.run.smithi100.stdout: "snap_mode": "selfmanaged", 2024-09-17T07:07:42.924 INFO:teuthology.orchestra.run.smithi100.stdout: "snap_seq": 0, 2024-09-17T07:07:42.924 INFO:teuthology.orchestra.run.smithi100.stdout: "snap_epoch": 0, 2024-09-17T07:07:42.924 INFO:teuthology.orchestra.run.smithi100.stdout: "pool_snaps": [], 2024-09-17T07:07:42.924 INFO:teuthology.orchestra.run.smithi100.stdout: "removed_snaps": "[]", 2024-09-17T07:07:42.924 INFO:teuthology.orchestra.run.smithi100.stdout: "quota_max_bytes": 0, 2024-09-17T07:07:42.924 INFO:teuthology.orchestra.run.smithi100.stdout: "quota_max_objects": 0, 2024-09-17T07:07:42.924 INFO:teuthology.orchestra.run.smithi100.stdout: "tiers": [], 2024-09-17T07:07:42.924 INFO:teuthology.orchestra.run.smithi100.stdout: "tier_of": -1, 2024-09-17T07:07:42.925 INFO:teuthology.orchestra.run.smithi100.stdout: "read_tier": -1, 2024-09-17T07:07:42.925 INFO:teuthology.orchestra.run.smithi100.stdout: "write_tier": -1, 2024-09-17T07:07:42.925 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_mode": "none", 2024-09-17T07:07:42.925 INFO:teuthology.orchestra.run.smithi100.stdout: "target_max_bytes": 0, 2024-09-17T07:07:42.925 INFO:teuthology.orchestra.run.smithi100.stdout: "target_max_objects": 0, 2024-09-17T07:07:42.925 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T07:07:42.925 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T07:07:42.925 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T07:07:42.925 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_min_flush_age": 0, 2024-09-17T07:07:42.925 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_min_evict_age": 0, 2024-09-17T07:07:42.925 INFO:teuthology.orchestra.run.smithi100.stdout: "erasure_code_profile": "", 2024-09-17T07:07:42.925 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_params": { 2024-09-17T07:07:42.925 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "none" 2024-09-17T07:07:42.926 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:07:42.926 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_period": 0, 2024-09-17T07:07:42.926 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_count": 0, 2024-09-17T07:07:42.926 INFO:teuthology.orchestra.run.smithi100.stdout: "use_gmt_hitset": true, 2024-09-17T07:07:42.926 INFO:teuthology.orchestra.run.smithi100.stdout: "min_read_recency_for_promote": 0, 2024-09-17T07:07:42.926 INFO:teuthology.orchestra.run.smithi100.stdout: "min_write_recency_for_promote": 0, 2024-09-17T07:07:42.926 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T07:07:42.926 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_search_last_n": 0, 2024-09-17T07:07:42.926 INFO:teuthology.orchestra.run.smithi100.stdout: "grade_table": [], 2024-09-17T07:07:42.926 INFO:teuthology.orchestra.run.smithi100.stdout: "stripe_width": 0, 2024-09-17T07:07:42.926 INFO:teuthology.orchestra.run.smithi100.stdout: "expected_num_objects": 0, 2024-09-17T07:07:42.926 INFO:teuthology.orchestra.run.smithi100.stdout: "fast_read": false, 2024-09-17T07:07:42.927 INFO:teuthology.orchestra.run.smithi100.stdout: "options": { 2024-09-17T07:07:42.927 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num_max": 32, 2024-09-17T07:07:42.927 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num_min": 1 2024-09-17T07:07:42.927 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:07:42.927 INFO:teuthology.orchestra.run.smithi100.stdout: "application_metadata": { 2024-09-17T07:07:42.927 INFO:teuthology.orchestra.run.smithi100.stdout: "mgr": {} 2024-09-17T07:07:42.927 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:07:42.927 INFO:teuthology.orchestra.run.smithi100.stdout: "read_balance": { 2024-09-17T07:07:42.927 INFO:teuthology.orchestra.run.smithi100.stdout: "score_acting": 7.8899998664855957, 2024-09-17T07:07:42.927 INFO:teuthology.orchestra.run.smithi100.stdout: "score_stable": 7.8899998664855957, 2024-09-17T07:07:42.927 INFO:teuthology.orchestra.run.smithi100.stdout: "optimal_score": 0.37999999523162842, 2024-09-17T07:07:42.927 INFO:teuthology.orchestra.run.smithi100.stdout: "raw_score_acting": 3, 2024-09-17T07:07:42.927 INFO:teuthology.orchestra.run.smithi100.stdout: "raw_score_stable": 3, 2024-09-17T07:07:42.928 INFO:teuthology.orchestra.run.smithi100.stdout: "primary_affinity_weighted": 0.89598274230957031, 2024-09-17T07:07:42.928 INFO:teuthology.orchestra.run.smithi100.stdout: "average_primary_affinity": 0.89999997615814209, 2024-09-17T07:07:42.928 INFO:teuthology.orchestra.run.smithi100.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-09-17T07:07:42.928 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:07:42.928 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:07:42.928 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:07:42.928 INFO:teuthology.orchestra.run.smithi100.stdout: "pool": 2, 2024-09-17T07:07:42.928 INFO:teuthology.orchestra.run.smithi100.stdout: "pool_name": ".rgw.root", 2024-09-17T07:07:42.928 INFO:teuthology.orchestra.run.smithi100.stdout: "create_time": "2024-09-17T07:02:10.759826+0000", 2024-09-17T07:07:42.928 INFO:teuthology.orchestra.run.smithi100.stdout: "flags": 1, 2024-09-17T07:07:42.928 INFO:teuthology.orchestra.run.smithi100.stdout: "flags_names": "hashpspool", 2024-09-17T07:07:42.928 INFO:teuthology.orchestra.run.smithi100.stdout: "type": 1, 2024-09-17T07:07:42.928 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 3, 2024-09-17T07:07:42.929 INFO:teuthology.orchestra.run.smithi100.stdout: "min_size": 2, 2024-09-17T07:07:42.929 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_rule": 0, 2024-09-17T07:07:42.929 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_count": 0, 2024-09-17T07:07:42.929 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_target": 0, 2024-09-17T07:07:42.929 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T07:07:42.929 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T07:07:42.929 INFO:teuthology.orchestra.run.smithi100.stdout: "object_hash": 2, 2024-09-17T07:07:42.929 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_autoscale_mode": "off", 2024-09-17T07:07:42.929 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num": 32, 2024-09-17T07:07:42.929 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_placement_num": 32, 2024-09-17T07:07:42.929 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_placement_num_target": 32, 2024-09-17T07:07:42.929 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num_target": 32, 2024-09-17T07:07:42.930 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num_pending": 32, 2024-09-17T07:07:42.930 INFO:teuthology.orchestra.run.smithi100.stdout: "last_pg_merge_meta": { 2024-09-17T07:07:42.930 INFO:teuthology.orchestra.run.smithi100.stdout: "source_pgid": "0.0", 2024-09-17T07:07:42.930 INFO:teuthology.orchestra.run.smithi100.stdout: "ready_epoch": 0, 2024-09-17T07:07:42.930 INFO:teuthology.orchestra.run.smithi100.stdout: "last_epoch_started": 0, 2024-09-17T07:07:42.930 INFO:teuthology.orchestra.run.smithi100.stdout: "last_epoch_clean": 0, 2024-09-17T07:07:42.930 INFO:teuthology.orchestra.run.smithi100.stdout: "source_version": "0'0", 2024-09-17T07:07:42.930 INFO:teuthology.orchestra.run.smithi100.stdout: "target_version": "0'0" 2024-09-17T07:07:42.930 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:07:42.930 INFO:teuthology.orchestra.run.smithi100.stdout: "last_change": "69", 2024-09-17T07:07:42.930 INFO:teuthology.orchestra.run.smithi100.stdout: "last_force_op_resend": "0", 2024-09-17T07:07:42.930 INFO:teuthology.orchestra.run.smithi100.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-17T07:07:42.930 INFO:teuthology.orchestra.run.smithi100.stdout: "last_force_op_resend_preluminous": "0", 2024-09-17T07:07:42.931 INFO:teuthology.orchestra.run.smithi100.stdout: "auid": 0, 2024-09-17T07:07:42.931 INFO:teuthology.orchestra.run.smithi100.stdout: "snap_mode": "selfmanaged", 2024-09-17T07:07:42.931 INFO:teuthology.orchestra.run.smithi100.stdout: "snap_seq": 0, 2024-09-17T07:07:42.931 INFO:teuthology.orchestra.run.smithi100.stdout: "snap_epoch": 0, 2024-09-17T07:07:42.931 INFO:teuthology.orchestra.run.smithi100.stdout: "pool_snaps": [], 2024-09-17T07:07:42.931 INFO:teuthology.orchestra.run.smithi100.stdout: "removed_snaps": "[]", 2024-09-17T07:07:42.931 INFO:teuthology.orchestra.run.smithi100.stdout: "quota_max_bytes": 0, 2024-09-17T07:07:42.931 INFO:teuthology.orchestra.run.smithi100.stdout: "quota_max_objects": 0, 2024-09-17T07:07:42.931 INFO:teuthology.orchestra.run.smithi100.stdout: "tiers": [], 2024-09-17T07:07:42.931 INFO:teuthology.orchestra.run.smithi100.stdout: "tier_of": -1, 2024-09-17T07:07:42.931 INFO:teuthology.orchestra.run.smithi100.stdout: "read_tier": -1, 2024-09-17T07:07:42.931 INFO:teuthology.orchestra.run.smithi100.stdout: "write_tier": -1, 2024-09-17T07:07:42.931 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_mode": "none", 2024-09-17T07:07:42.932 INFO:teuthology.orchestra.run.smithi100.stdout: "target_max_bytes": 0, 2024-09-17T07:07:42.932 INFO:teuthology.orchestra.run.smithi100.stdout: "target_max_objects": 0, 2024-09-17T07:07:42.932 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T07:07:42.932 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T07:07:42.932 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T07:07:42.932 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_min_flush_age": 0, 2024-09-17T07:07:42.932 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_min_evict_age": 0, 2024-09-17T07:07:42.932 INFO:teuthology.orchestra.run.smithi100.stdout: "erasure_code_profile": "", 2024-09-17T07:07:42.932 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_params": { 2024-09-17T07:07:42.932 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "none" 2024-09-17T07:07:42.932 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:07:42.932 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_period": 0, 2024-09-17T07:07:42.932 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_count": 0, 2024-09-17T07:07:42.933 INFO:teuthology.orchestra.run.smithi100.stdout: "use_gmt_hitset": true, 2024-09-17T07:07:42.933 INFO:teuthology.orchestra.run.smithi100.stdout: "min_read_recency_for_promote": 0, 2024-09-17T07:07:42.933 INFO:teuthology.orchestra.run.smithi100.stdout: "min_write_recency_for_promote": 0, 2024-09-17T07:07:42.933 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T07:07:42.933 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_search_last_n": 0, 2024-09-17T07:07:42.933 INFO:teuthology.orchestra.run.smithi100.stdout: "grade_table": [], 2024-09-17T07:07:42.933 INFO:teuthology.orchestra.run.smithi100.stdout: "stripe_width": 0, 2024-09-17T07:07:42.933 INFO:teuthology.orchestra.run.smithi100.stdout: "expected_num_objects": 0, 2024-09-17T07:07:42.933 INFO:teuthology.orchestra.run.smithi100.stdout: "fast_read": false, 2024-09-17T07:07:42.933 INFO:teuthology.orchestra.run.smithi100.stdout: "options": {}, 2024-09-17T07:07:42.933 INFO:teuthology.orchestra.run.smithi100.stdout: "application_metadata": { 2024-09-17T07:07:42.933 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw": {} 2024-09-17T07:07:42.933 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:07:42.934 INFO:teuthology.orchestra.run.smithi100.stdout: "read_balance": { 2024-09-17T07:07:42.934 INFO:teuthology.orchestra.run.smithi100.stdout: "score_acting": 1.5, 2024-09-17T07:07:42.934 INFO:teuthology.orchestra.run.smithi100.stdout: "score_stable": 1.5, 2024-09-17T07:07:42.934 INFO:teuthology.orchestra.run.smithi100.stdout: "optimal_score": 1, 2024-09-17T07:07:42.934 INFO:teuthology.orchestra.run.smithi100.stdout: "raw_score_acting": 1.5, 2024-09-17T07:07:42.934 INFO:teuthology.orchestra.run.smithi100.stdout: "raw_score_stable": 1.5, 2024-09-17T07:07:42.934 INFO:teuthology.orchestra.run.smithi100.stdout: "primary_affinity_weighted": 0.89598274230957031, 2024-09-17T07:07:42.934 INFO:teuthology.orchestra.run.smithi100.stdout: "average_primary_affinity": 0.89999997615814209, 2024-09-17T07:07:42.934 INFO:teuthology.orchestra.run.smithi100.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-09-17T07:07:42.934 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:07:42.934 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:07:42.934 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:07:42.934 INFO:teuthology.orchestra.run.smithi100.stdout: "pool": 3, 2024-09-17T07:07:42.935 INFO:teuthology.orchestra.run.smithi100.stdout: "pool_name": "datapool", 2024-09-17T07:07:42.935 INFO:teuthology.orchestra.run.smithi100.stdout: "create_time": "2024-09-17T07:02:10.991289+0000", 2024-09-17T07:07:42.935 INFO:teuthology.orchestra.run.smithi100.stdout: "flags": 8193, 2024-09-17T07:07:42.935 INFO:teuthology.orchestra.run.smithi100.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-17T07:07:42.935 INFO:teuthology.orchestra.run.smithi100.stdout: "type": 1, 2024-09-17T07:07:42.935 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 3, 2024-09-17T07:07:42.935 INFO:teuthology.orchestra.run.smithi100.stdout: "min_size": 2, 2024-09-17T07:07:42.935 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_rule": 0, 2024-09-17T07:07:42.935 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_count": 0, 2024-09-17T07:07:42.935 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_target": 0, 2024-09-17T07:07:42.935 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T07:07:42.935 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T07:07:42.935 INFO:teuthology.orchestra.run.smithi100.stdout: "object_hash": 2, 2024-09-17T07:07:42.936 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_autoscale_mode": "off", 2024-09-17T07:07:42.936 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num": 3, 2024-09-17T07:07:42.936 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_placement_num": 3, 2024-09-17T07:07:42.936 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_placement_num_target": 3, 2024-09-17T07:07:42.936 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num_target": 3, 2024-09-17T07:07:42.936 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num_pending": 3, 2024-09-17T07:07:42.936 INFO:teuthology.orchestra.run.smithi100.stdout: "last_pg_merge_meta": { 2024-09-17T07:07:42.936 INFO:teuthology.orchestra.run.smithi100.stdout: "source_pgid": "0.0", 2024-09-17T07:07:42.936 INFO:teuthology.orchestra.run.smithi100.stdout: "ready_epoch": 0, 2024-09-17T07:07:42.936 INFO:teuthology.orchestra.run.smithi100.stdout: "last_epoch_started": 0, 2024-09-17T07:07:42.936 INFO:teuthology.orchestra.run.smithi100.stdout: "last_epoch_clean": 0, 2024-09-17T07:07:42.936 INFO:teuthology.orchestra.run.smithi100.stdout: "source_version": "0'0", 2024-09-17T07:07:42.936 INFO:teuthology.orchestra.run.smithi100.stdout: "target_version": "0'0" 2024-09-17T07:07:42.937 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:07:42.937 INFO:teuthology.orchestra.run.smithi100.stdout: "last_change": "72", 2024-09-17T07:07:42.937 INFO:teuthology.orchestra.run.smithi100.stdout: "last_force_op_resend": "0", 2024-09-17T07:07:42.937 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:07:42.938 INFO:teuthology.orchestra.run.smithi100.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-17T07:07:42.938 INFO:teuthology.orchestra.run.smithi100.stdout: "last_force_op_resend_preluminous": "0", 2024-09-17T07:07:42.938 INFO:teuthology.orchestra.run.smithi100.stdout: "auid": 0, 2024-09-17T07:07:42.938 INFO:teuthology.orchestra.run.smithi100.stdout: "snap_mode": "selfmanaged", 2024-09-17T07:07:42.939 INFO:teuthology.orchestra.run.smithi100.stdout: "snap_seq": 3, 2024-09-17T07:07:42.939 INFO:teuthology.orchestra.run.smithi100.stdout: "snap_epoch": 72, 2024-09-17T07:07:42.939 INFO:teuthology.orchestra.run.smithi100.stdout: "pool_snaps": [], 2024-09-17T07:07:42.939 INFO:teuthology.orchestra.run.smithi100.stdout: "removed_snaps": "[]", 2024-09-17T07:07:42.939 INFO:teuthology.orchestra.run.smithi100.stdout: "quota_max_bytes": 0, 2024-09-17T07:07:42.939 INFO:teuthology.orchestra.run.smithi100.stdout: "quota_max_objects": 0, 2024-09-17T07:07:42.939 INFO:teuthology.orchestra.run.smithi100.stdout: "tiers": [], 2024-09-17T07:07:42.939 INFO:teuthology.orchestra.run.smithi100.stdout: "tier_of": -1, 2024-09-17T07:07:42.939 INFO:teuthology.orchestra.run.smithi100.stdout: "read_tier": -1, 2024-09-17T07:07:42.939 INFO:teuthology.orchestra.run.smithi100.stdout: "write_tier": -1, 2024-09-17T07:07:42.939 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_mode": "none", 2024-09-17T07:07:42.939 INFO:teuthology.orchestra.run.smithi100.stdout: "target_max_bytes": 0, 2024-09-17T07:07:42.939 INFO:teuthology.orchestra.run.smithi100.stdout: "target_max_objects": 0, 2024-09-17T07:07:42.940 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T07:07:42.940 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T07:07:42.940 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T07:07:42.940 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_min_flush_age": 0, 2024-09-17T07:07:42.940 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_min_evict_age": 0, 2024-09-17T07:07:42.940 INFO:teuthology.orchestra.run.smithi100.stdout: "erasure_code_profile": "", 2024-09-17T07:07:42.940 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_params": { 2024-09-17T07:07:42.940 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "none" 2024-09-17T07:07:42.940 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:07:42.940 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_period": 0, 2024-09-17T07:07:42.940 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_count": 0, 2024-09-17T07:07:42.940 INFO:teuthology.orchestra.run.smithi100.stdout: "use_gmt_hitset": true, 2024-09-17T07:07:42.941 INFO:teuthology.orchestra.run.smithi100.stdout: "min_read_recency_for_promote": 0, 2024-09-17T07:07:42.941 INFO:teuthology.orchestra.run.smithi100.stdout: "min_write_recency_for_promote": 0, 2024-09-17T07:07:42.941 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T07:07:42.941 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_search_last_n": 0, 2024-09-17T07:07:42.941 INFO:teuthology.orchestra.run.smithi100.stdout: "grade_table": [], 2024-09-17T07:07:42.941 INFO:teuthology.orchestra.run.smithi100.stdout: "stripe_width": 0, 2024-09-17T07:07:42.941 INFO:teuthology.orchestra.run.smithi100.stdout: "expected_num_objects": 0, 2024-09-17T07:07:42.941 INFO:teuthology.orchestra.run.smithi100.stdout: "fast_read": false, 2024-09-17T07:07:42.941 INFO:teuthology.orchestra.run.smithi100.stdout: "options": {}, 2024-09-17T07:07:42.941 INFO:teuthology.orchestra.run.smithi100.stdout: "application_metadata": { 2024-09-17T07:07:42.941 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd": {} 2024-09-17T07:07:42.941 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:07:42.941 INFO:teuthology.orchestra.run.smithi100.stdout: "read_balance": { 2024-09-17T07:07:42.942 INFO:teuthology.orchestra.run.smithi100.stdout: "score_acting": 2.6500000953674316, 2024-09-17T07:07:42.942 INFO:teuthology.orchestra.run.smithi100.stdout: "score_stable": 2.6500000953674316, 2024-09-17T07:07:42.942 INFO:teuthology.orchestra.run.smithi100.stdout: "optimal_score": 0.87999999523162842, 2024-09-17T07:07:42.942 INFO:teuthology.orchestra.run.smithi100.stdout: "raw_score_acting": 2.3299999237060547, 2024-09-17T07:07:42.942 INFO:teuthology.orchestra.run.smithi100.stdout: "raw_score_stable": 2.3299999237060547, 2024-09-17T07:07:42.942 INFO:teuthology.orchestra.run.smithi100.stdout: "primary_affinity_weighted": 0.89598274230957031, 2024-09-17T07:07:42.942 INFO:teuthology.orchestra.run.smithi100.stdout: "average_primary_affinity": 0.89999997615814209, 2024-09-17T07:07:42.942 INFO:teuthology.orchestra.run.smithi100.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-09-17T07:07:42.942 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:07:42.942 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:07:42.942 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:07:42.942 INFO:teuthology.orchestra.run.smithi100.stdout: "pool": 4, 2024-09-17T07:07:42.942 INFO:teuthology.orchestra.run.smithi100.stdout: "pool_name": "default.rgw.log", 2024-09-17T07:07:42.943 INFO:teuthology.orchestra.run.smithi100.stdout: "create_time": "2024-09-17T07:02:12.291462+0000", 2024-09-17T07:07:42.943 INFO:teuthology.orchestra.run.smithi100.stdout: "flags": 1, 2024-09-17T07:07:42.943 INFO:teuthology.orchestra.run.smithi100.stdout: "flags_names": "hashpspool", 2024-09-17T07:07:42.943 INFO:teuthology.orchestra.run.smithi100.stdout: "type": 1, 2024-09-17T07:07:42.943 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 3, 2024-09-17T07:07:42.943 INFO:teuthology.orchestra.run.smithi100.stdout: "min_size": 2, 2024-09-17T07:07:42.943 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_rule": 0, 2024-09-17T07:07:42.943 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_count": 0, 2024-09-17T07:07:42.943 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_target": 0, 2024-09-17T07:07:42.943 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T07:07:42.943 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T07:07:42.943 INFO:teuthology.orchestra.run.smithi100.stdout: "object_hash": 2, 2024-09-17T07:07:42.944 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_autoscale_mode": "off", 2024-09-17T07:07:42.944 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num": 32, 2024-09-17T07:07:42.944 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_placement_num": 32, 2024-09-17T07:07:42.944 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_placement_num_target": 32, 2024-09-17T07:07:42.944 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num_target": 32, 2024-09-17T07:07:42.944 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num_pending": 32, 2024-09-17T07:07:42.944 INFO:teuthology.orchestra.run.smithi100.stdout: "last_pg_merge_meta": { 2024-09-17T07:07:42.944 INFO:teuthology.orchestra.run.smithi100.stdout: "source_pgid": "0.0", 2024-09-17T07:07:42.944 INFO:teuthology.orchestra.run.smithi100.stdout: "ready_epoch": 0, 2024-09-17T07:07:42.944 INFO:teuthology.orchestra.run.smithi100.stdout: "last_epoch_started": 0, 2024-09-17T07:07:42.944 INFO:teuthology.orchestra.run.smithi100.stdout: "last_epoch_clean": 0, 2024-09-17T07:07:42.944 INFO:teuthology.orchestra.run.smithi100.stdout: "source_version": "0'0", 2024-09-17T07:07:42.944 INFO:teuthology.orchestra.run.smithi100.stdout: "target_version": "0'0" 2024-09-17T07:07:42.945 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:07:42.945 INFO:teuthology.orchestra.run.smithi100.stdout: "last_change": "71", 2024-09-17T07:07:42.945 INFO:teuthology.orchestra.run.smithi100.stdout: "last_force_op_resend": "0", 2024-09-17T07:07:42.945 INFO:teuthology.orchestra.run.smithi100.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-17T07:07:42.945 INFO:teuthology.orchestra.run.smithi100.stdout: "last_force_op_resend_preluminous": "0", 2024-09-17T07:07:42.945 INFO:teuthology.orchestra.run.smithi100.stdout: "auid": 0, 2024-09-17T07:07:42.945 INFO:teuthology.orchestra.run.smithi100.stdout: "snap_mode": "selfmanaged", 2024-09-17T07:07:42.945 INFO:teuthology.orchestra.run.smithi100.stdout: "snap_seq": 0, 2024-09-17T07:07:42.945 INFO:teuthology.orchestra.run.smithi100.stdout: "snap_epoch": 0, 2024-09-17T07:07:42.945 INFO:teuthology.orchestra.run.smithi100.stdout: "pool_snaps": [], 2024-09-17T07:07:42.945 INFO:teuthology.orchestra.run.smithi100.stdout: "removed_snaps": "[]", 2024-09-17T07:07:42.945 INFO:teuthology.orchestra.run.smithi100.stdout: "quota_max_bytes": 0, 2024-09-17T07:07:42.946 INFO:teuthology.orchestra.run.smithi100.stdout: "quota_max_objects": 0, 2024-09-17T07:07:42.946 INFO:teuthology.orchestra.run.smithi100.stdout: "tiers": [], 2024-09-17T07:07:42.946 INFO:teuthology.orchestra.run.smithi100.stdout: "tier_of": -1, 2024-09-17T07:07:42.946 INFO:teuthology.orchestra.run.smithi100.stdout: "read_tier": -1, 2024-09-17T07:07:42.946 INFO:teuthology.orchestra.run.smithi100.stdout: "write_tier": -1, 2024-09-17T07:07:42.946 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_mode": "none", 2024-09-17T07:07:42.946 INFO:teuthology.orchestra.run.smithi100.stdout: "target_max_bytes": 0, 2024-09-17T07:07:42.946 INFO:teuthology.orchestra.run.smithi100.stdout: "target_max_objects": 0, 2024-09-17T07:07:42.946 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T07:07:42.946 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T07:07:42.946 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T07:07:42.946 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_min_flush_age": 0, 2024-09-17T07:07:42.946 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_min_evict_age": 0, 2024-09-17T07:07:42.946 INFO:teuthology.orchestra.run.smithi100.stdout: "erasure_code_profile": "", 2024-09-17T07:07:42.947 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_params": { 2024-09-17T07:07:42.947 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "none" 2024-09-17T07:07:42.947 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:07:42.947 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_period": 0, 2024-09-17T07:07:42.947 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_count": 0, 2024-09-17T07:07:42.947 INFO:teuthology.orchestra.run.smithi100.stdout: "use_gmt_hitset": true, 2024-09-17T07:07:42.947 INFO:teuthology.orchestra.run.smithi100.stdout: "min_read_recency_for_promote": 0, 2024-09-17T07:07:42.947 INFO:teuthology.orchestra.run.smithi100.stdout: "min_write_recency_for_promote": 0, 2024-09-17T07:07:42.947 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T07:07:42.947 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_search_last_n": 0, 2024-09-17T07:07:42.947 INFO:teuthology.orchestra.run.smithi100.stdout: "grade_table": [], 2024-09-17T07:07:42.947 INFO:teuthology.orchestra.run.smithi100.stdout: "stripe_width": 0, 2024-09-17T07:07:42.947 INFO:teuthology.orchestra.run.smithi100.stdout: "expected_num_objects": 0, 2024-09-17T07:07:42.948 INFO:teuthology.orchestra.run.smithi100.stdout: "fast_read": false, 2024-09-17T07:07:42.948 INFO:teuthology.orchestra.run.smithi100.stdout: "options": {}, 2024-09-17T07:07:42.948 INFO:teuthology.orchestra.run.smithi100.stdout: "application_metadata": { 2024-09-17T07:07:42.948 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw": {} 2024-09-17T07:07:42.948 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:07:42.948 INFO:teuthology.orchestra.run.smithi100.stdout: "read_balance": { 2024-09-17T07:07:42.948 INFO:teuthology.orchestra.run.smithi100.stdout: "score_acting": 2.25, 2024-09-17T07:07:42.948 INFO:teuthology.orchestra.run.smithi100.stdout: "score_stable": 2.25, 2024-09-17T07:07:42.948 INFO:teuthology.orchestra.run.smithi100.stdout: "optimal_score": 1, 2024-09-17T07:07:42.948 INFO:teuthology.orchestra.run.smithi100.stdout: "raw_score_acting": 2.25, 2024-09-17T07:07:42.948 INFO:teuthology.orchestra.run.smithi100.stdout: "raw_score_stable": 2.25, 2024-09-17T07:07:42.948 INFO:teuthology.orchestra.run.smithi100.stdout: "primary_affinity_weighted": 0.89598274230957031, 2024-09-17T07:07:42.949 INFO:teuthology.orchestra.run.smithi100.stdout: "average_primary_affinity": 0.89999997615814209, 2024-09-17T07:07:42.949 INFO:teuthology.orchestra.run.smithi100.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-09-17T07:07:42.949 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:07:42.949 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:07:42.949 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:07:42.949 INFO:teuthology.orchestra.run.smithi100.stdout: "pool": 5, 2024-09-17T07:07:42.949 INFO:teuthology.orchestra.run.smithi100.stdout: "pool_name": "default.rgw.control", 2024-09-17T07:07:42.949 INFO:teuthology.orchestra.run.smithi100.stdout: "create_time": "2024-09-17T07:02:14.253214+0000", 2024-09-17T07:07:42.949 INFO:teuthology.orchestra.run.smithi100.stdout: "flags": 1, 2024-09-17T07:07:42.949 INFO:teuthology.orchestra.run.smithi100.stdout: "flags_names": "hashpspool", 2024-09-17T07:07:42.949 INFO:teuthology.orchestra.run.smithi100.stdout: "type": 1, 2024-09-17T07:07:42.949 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 3, 2024-09-17T07:07:42.949 INFO:teuthology.orchestra.run.smithi100.stdout: "min_size": 2, 2024-09-17T07:07:42.950 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_rule": 0, 2024-09-17T07:07:42.950 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_count": 0, 2024-09-17T07:07:42.950 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_target": 0, 2024-09-17T07:07:42.950 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T07:07:42.950 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T07:07:42.950 INFO:teuthology.orchestra.run.smithi100.stdout: "object_hash": 2, 2024-09-17T07:07:42.950 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_autoscale_mode": "off", 2024-09-17T07:07:42.950 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num": 32, 2024-09-17T07:07:42.950 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_placement_num": 32, 2024-09-17T07:07:42.950 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_placement_num_target": 32, 2024-09-17T07:07:42.950 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num_target": 32, 2024-09-17T07:07:42.950 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num_pending": 32, 2024-09-17T07:07:42.950 INFO:teuthology.orchestra.run.smithi100.stdout: "last_pg_merge_meta": { 2024-09-17T07:07:42.951 INFO:teuthology.orchestra.run.smithi100.stdout: "source_pgid": "0.0", 2024-09-17T07:07:42.951 INFO:teuthology.orchestra.run.smithi100.stdout: "ready_epoch": 0, 2024-09-17T07:07:42.951 INFO:teuthology.orchestra.run.smithi100.stdout: "last_epoch_started": 0, 2024-09-17T07:07:42.951 INFO:teuthology.orchestra.run.smithi100.stdout: "last_epoch_clean": 0, 2024-09-17T07:07:42.951 INFO:teuthology.orchestra.run.smithi100.stdout: "source_version": "0'0", 2024-09-17T07:07:42.951 INFO:teuthology.orchestra.run.smithi100.stdout: "target_version": "0'0" 2024-09-17T07:07:42.951 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:07:42.951 INFO:teuthology.orchestra.run.smithi100.stdout: "last_change": "73", 2024-09-17T07:07:42.951 INFO:teuthology.orchestra.run.smithi100.stdout: "last_force_op_resend": "0", 2024-09-17T07:07:42.951 INFO:teuthology.orchestra.run.smithi100.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-17T07:07:42.951 INFO:teuthology.orchestra.run.smithi100.stdout: "last_force_op_resend_preluminous": "0", 2024-09-17T07:07:42.951 INFO:teuthology.orchestra.run.smithi100.stdout: "auid": 0, 2024-09-17T07:07:42.951 INFO:teuthology.orchestra.run.smithi100.stdout: "snap_mode": "selfmanaged", 2024-09-17T07:07:42.951 INFO:teuthology.orchestra.run.smithi100.stdout: "snap_seq": 0, 2024-09-17T07:07:42.952 INFO:teuthology.orchestra.run.smithi100.stdout: "snap_epoch": 0, 2024-09-17T07:07:42.952 INFO:teuthology.orchestra.run.smithi100.stdout: "pool_snaps": [], 2024-09-17T07:07:42.952 INFO:teuthology.orchestra.run.smithi100.stdout: "removed_snaps": "[]", 2024-09-17T07:07:42.952 INFO:teuthology.orchestra.run.smithi100.stdout: "quota_max_bytes": 0, 2024-09-17T07:07:42.952 INFO:teuthology.orchestra.run.smithi100.stdout: "quota_max_objects": 0, 2024-09-17T07:07:42.952 INFO:teuthology.orchestra.run.smithi100.stdout: "tiers": [], 2024-09-17T07:07:42.952 INFO:teuthology.orchestra.run.smithi100.stdout: "tier_of": -1, 2024-09-17T07:07:42.952 INFO:teuthology.orchestra.run.smithi100.stdout: "read_tier": -1, 2024-09-17T07:07:42.952 INFO:teuthology.orchestra.run.smithi100.stdout: "write_tier": -1, 2024-09-17T07:07:42.952 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_mode": "none", 2024-09-17T07:07:42.952 INFO:teuthology.orchestra.run.smithi100.stdout: "target_max_bytes": 0, 2024-09-17T07:07:42.952 INFO:teuthology.orchestra.run.smithi100.stdout: "target_max_objects": 0, 2024-09-17T07:07:42.952 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T07:07:42.953 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T07:07:42.953 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T07:07:42.953 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_min_flush_age": 0, 2024-09-17T07:07:42.953 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_min_evict_age": 0, 2024-09-17T07:07:42.953 INFO:teuthology.orchestra.run.smithi100.stdout: "erasure_code_profile": "", 2024-09-17T07:07:42.953 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_params": { 2024-09-17T07:07:42.953 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "none" 2024-09-17T07:07:42.953 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:07:42.953 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_period": 0, 2024-09-17T07:07:42.953 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_count": 0, 2024-09-17T07:07:42.953 INFO:teuthology.orchestra.run.smithi100.stdout: "use_gmt_hitset": true, 2024-09-17T07:07:42.953 INFO:teuthology.orchestra.run.smithi100.stdout: "min_read_recency_for_promote": 0, 2024-09-17T07:07:42.954 INFO:teuthology.orchestra.run.smithi100.stdout: "min_write_recency_for_promote": 0, 2024-09-17T07:07:42.954 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T07:07:42.954 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_search_last_n": 0, 2024-09-17T07:07:42.954 INFO:teuthology.orchestra.run.smithi100.stdout: "grade_table": [], 2024-09-17T07:07:42.954 INFO:teuthology.orchestra.run.smithi100.stdout: "stripe_width": 0, 2024-09-17T07:07:42.954 INFO:teuthology.orchestra.run.smithi100.stdout: "expected_num_objects": 0, 2024-09-17T07:07:42.954 INFO:teuthology.orchestra.run.smithi100.stdout: "fast_read": false, 2024-09-17T07:07:42.954 INFO:teuthology.orchestra.run.smithi100.stdout: "options": {}, 2024-09-17T07:07:42.954 INFO:teuthology.orchestra.run.smithi100.stdout: "application_metadata": { 2024-09-17T07:07:42.954 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw": {} 2024-09-17T07:07:42.954 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:07:42.954 INFO:teuthology.orchestra.run.smithi100.stdout: "read_balance": { 2024-09-17T07:07:42.954 INFO:teuthology.orchestra.run.smithi100.stdout: "score_acting": 1.25, 2024-09-17T07:07:42.955 INFO:teuthology.orchestra.run.smithi100.stdout: "score_stable": 1.25, 2024-09-17T07:07:42.955 INFO:teuthology.orchestra.run.smithi100.stdout: "optimal_score": 1, 2024-09-17T07:07:42.955 INFO:teuthology.orchestra.run.smithi100.stdout: "raw_score_acting": 1.25, 2024-09-17T07:07:42.955 INFO:teuthology.orchestra.run.smithi100.stdout: "raw_score_stable": 1.25, 2024-09-17T07:07:42.955 INFO:teuthology.orchestra.run.smithi100.stdout: "primary_affinity_weighted": 0.89598274230957031, 2024-09-17T07:07:42.955 INFO:teuthology.orchestra.run.smithi100.stdout: "average_primary_affinity": 0.89999997615814209, 2024-09-17T07:07:42.955 INFO:teuthology.orchestra.run.smithi100.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-09-17T07:07:42.955 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:07:42.955 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:07:42.955 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:07:42.955 INFO:teuthology.orchestra.run.smithi100.stdout: "pool": 6, 2024-09-17T07:07:42.955 INFO:teuthology.orchestra.run.smithi100.stdout: "pool_name": "default.rgw.meta", 2024-09-17T07:07:42.955 INFO:teuthology.orchestra.run.smithi100.stdout: "create_time": "2024-09-17T07:02:16.310620+0000", 2024-09-17T07:07:42.955 INFO:teuthology.orchestra.run.smithi100.stdout: "flags": 1, 2024-09-17T07:07:42.956 INFO:teuthology.orchestra.run.smithi100.stdout: "flags_names": "hashpspool", 2024-09-17T07:07:42.956 INFO:teuthology.orchestra.run.smithi100.stdout: "type": 1, 2024-09-17T07:07:42.956 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 3, 2024-09-17T07:07:42.956 INFO:teuthology.orchestra.run.smithi100.stdout: "min_size": 2, 2024-09-17T07:07:42.956 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_rule": 0, 2024-09-17T07:07:42.956 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_count": 0, 2024-09-17T07:07:42.956 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_target": 0, 2024-09-17T07:07:42.956 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T07:07:42.956 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T07:07:42.956 INFO:teuthology.orchestra.run.smithi100.stdout: "object_hash": 2, 2024-09-17T07:07:42.956 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_autoscale_mode": "off", 2024-09-17T07:07:42.956 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num": 32, 2024-09-17T07:07:42.957 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_placement_num": 32, 2024-09-17T07:07:42.957 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_placement_num_target": 32, 2024-09-17T07:07:42.957 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num_target": 32, 2024-09-17T07:07:42.957 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num_pending": 32, 2024-09-17T07:07:42.957 INFO:teuthology.orchestra.run.smithi100.stdout: "last_pg_merge_meta": { 2024-09-17T07:07:42.957 INFO:teuthology.orchestra.run.smithi100.stdout: "source_pgid": "0.0", 2024-09-17T07:07:42.957 INFO:teuthology.orchestra.run.smithi100.stdout: "ready_epoch": 0, 2024-09-17T07:07:42.957 INFO:teuthology.orchestra.run.smithi100.stdout: "last_epoch_started": 0, 2024-09-17T07:07:42.957 INFO:teuthology.orchestra.run.smithi100.stdout: "last_epoch_clean": 0, 2024-09-17T07:07:42.957 INFO:teuthology.orchestra.run.smithi100.stdout: "source_version": "0'0", 2024-09-17T07:07:42.957 INFO:teuthology.orchestra.run.smithi100.stdout: "target_version": "0'0" 2024-09-17T07:07:42.957 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:07:42.957 INFO:teuthology.orchestra.run.smithi100.stdout: "last_change": "75", 2024-09-17T07:07:42.958 INFO:teuthology.orchestra.run.smithi100.stdout: "last_force_op_resend": "0", 2024-09-17T07:07:42.958 INFO:teuthology.orchestra.run.smithi100.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-17T07:07:42.958 INFO:teuthology.orchestra.run.smithi100.stdout: "last_force_op_resend_preluminous": "0", 2024-09-17T07:07:42.958 INFO:teuthology.orchestra.run.smithi100.stdout: "auid": 0, 2024-09-17T07:07:42.958 INFO:teuthology.orchestra.run.smithi100.stdout: "snap_mode": "selfmanaged", 2024-09-17T07:07:42.958 INFO:teuthology.orchestra.run.smithi100.stdout: "snap_seq": 0, 2024-09-17T07:07:42.958 INFO:teuthology.orchestra.run.smithi100.stdout: "snap_epoch": 0, 2024-09-17T07:07:42.958 INFO:teuthology.orchestra.run.smithi100.stdout: "pool_snaps": [], 2024-09-17T07:07:42.958 INFO:teuthology.orchestra.run.smithi100.stdout: "removed_snaps": "[]", 2024-09-17T07:07:42.958 INFO:teuthology.orchestra.run.smithi100.stdout: "quota_max_bytes": 0, 2024-09-17T07:07:42.958 INFO:teuthology.orchestra.run.smithi100.stdout: "quota_max_objects": 0, 2024-09-17T07:07:42.958 INFO:teuthology.orchestra.run.smithi100.stdout: "tiers": [], 2024-09-17T07:07:42.958 INFO:teuthology.orchestra.run.smithi100.stdout: "tier_of": -1, 2024-09-17T07:07:42.959 INFO:teuthology.orchestra.run.smithi100.stdout: "read_tier": -1, 2024-09-17T07:07:42.959 INFO:teuthology.orchestra.run.smithi100.stdout: "write_tier": -1, 2024-09-17T07:07:42.959 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_mode": "none", 2024-09-17T07:07:42.959 INFO:teuthology.orchestra.run.smithi100.stdout: "target_max_bytes": 0, 2024-09-17T07:07:42.959 INFO:teuthology.orchestra.run.smithi100.stdout: "target_max_objects": 0, 2024-09-17T07:07:42.959 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T07:07:42.959 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T07:07:42.959 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T07:07:42.959 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_min_flush_age": 0, 2024-09-17T07:07:42.959 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_min_evict_age": 0, 2024-09-17T07:07:42.959 INFO:teuthology.orchestra.run.smithi100.stdout: "erasure_code_profile": "", 2024-09-17T07:07:42.959 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_params": { 2024-09-17T07:07:42.959 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "none" 2024-09-17T07:07:42.959 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:07:42.960 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_period": 0, 2024-09-17T07:07:42.960 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_count": 0, 2024-09-17T07:07:42.960 INFO:teuthology.orchestra.run.smithi100.stdout: "use_gmt_hitset": true, 2024-09-17T07:07:42.960 INFO:teuthology.orchestra.run.smithi100.stdout: "min_read_recency_for_promote": 0, 2024-09-17T07:07:42.960 INFO:teuthology.orchestra.run.smithi100.stdout: "min_write_recency_for_promote": 0, 2024-09-17T07:07:42.960 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T07:07:42.960 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_search_last_n": 0, 2024-09-17T07:07:42.960 INFO:teuthology.orchestra.run.smithi100.stdout: "grade_table": [], 2024-09-17T07:07:42.960 INFO:teuthology.orchestra.run.smithi100.stdout: "stripe_width": 0, 2024-09-17T07:07:42.960 INFO:teuthology.orchestra.run.smithi100.stdout: "expected_num_objects": 0, 2024-09-17T07:07:42.960 INFO:teuthology.orchestra.run.smithi100.stdout: "fast_read": false, 2024-09-17T07:07:42.960 INFO:teuthology.orchestra.run.smithi100.stdout: "options": { 2024-09-17T07:07:42.960 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_autoscale_bias": 4 2024-09-17T07:07:42.961 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:07:42.961 INFO:teuthology.orchestra.run.smithi100.stdout: "application_metadata": { 2024-09-17T07:07:42.961 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw": {} 2024-09-17T07:07:42.961 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:07:42.961 INFO:teuthology.orchestra.run.smithi100.stdout: "read_balance": { 2024-09-17T07:07:42.961 INFO:teuthology.orchestra.run.smithi100.stdout: "score_acting": 2, 2024-09-17T07:07:42.961 INFO:teuthology.orchestra.run.smithi100.stdout: "score_stable": 2, 2024-09-17T07:07:42.961 INFO:teuthology.orchestra.run.smithi100.stdout: "optimal_score": 1, 2024-09-17T07:07:42.961 INFO:teuthology.orchestra.run.smithi100.stdout: "raw_score_acting": 2, 2024-09-17T07:07:42.961 INFO:teuthology.orchestra.run.smithi100.stdout: "raw_score_stable": 2, 2024-09-17T07:07:42.961 INFO:teuthology.orchestra.run.smithi100.stdout: "primary_affinity_weighted": 0.89598274230957031, 2024-09-17T07:07:42.961 INFO:teuthology.orchestra.run.smithi100.stdout: "average_primary_affinity": 0.89999997615814209, 2024-09-17T07:07:42.961 INFO:teuthology.orchestra.run.smithi100.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-09-17T07:07:42.962 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:07:42.962 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:07:42.962 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:07:42.962 INFO:teuthology.orchestra.run.smithi100.stdout: "pool": 7, 2024-09-17T07:07:42.962 INFO:teuthology.orchestra.run.smithi100.stdout: "pool_name": "unique_pool_0", 2024-09-17T07:07:42.962 INFO:teuthology.orchestra.run.smithi100.stdout: "create_time": "2024-09-17T07:04:19.222948+0000", 2024-09-17T07:07:42.962 INFO:teuthology.orchestra.run.smithi100.stdout: "flags": 1, 2024-09-17T07:07:42.962 INFO:teuthology.orchestra.run.smithi100.stdout: "flags_names": "hashpspool", 2024-09-17T07:07:42.962 INFO:teuthology.orchestra.run.smithi100.stdout: "type": 1, 2024-09-17T07:07:42.962 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 3, 2024-09-17T07:07:42.962 INFO:teuthology.orchestra.run.smithi100.stdout: "min_size": 2, 2024-09-17T07:07:42.962 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_rule": 0, 2024-09-17T07:07:42.962 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_count": 0, 2024-09-17T07:07:42.963 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_target": 0, 2024-09-17T07:07:42.963 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T07:07:42.963 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T07:07:42.963 INFO:teuthology.orchestra.run.smithi100.stdout: "object_hash": 2, 2024-09-17T07:07:42.963 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_autoscale_mode": "off", 2024-09-17T07:07:42.963 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num": 26, 2024-09-17T07:07:42.963 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_placement_num": 26, 2024-09-17T07:07:42.963 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_placement_num_target": 26, 2024-09-17T07:07:42.963 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num_target": 26, 2024-09-17T07:07:42.963 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num_pending": 26, 2024-09-17T07:07:42.963 INFO:teuthology.orchestra.run.smithi100.stdout: "last_pg_merge_meta": { 2024-09-17T07:07:42.963 INFO:teuthology.orchestra.run.smithi100.stdout: "source_pgid": "0.0", 2024-09-17T07:07:42.963 INFO:teuthology.orchestra.run.smithi100.stdout: "ready_epoch": 0, 2024-09-17T07:07:42.964 INFO:teuthology.orchestra.run.smithi100.stdout: "last_epoch_started": 0, 2024-09-17T07:07:42.964 INFO:teuthology.orchestra.run.smithi100.stdout: "last_epoch_clean": 0, 2024-09-17T07:07:42.964 INFO:teuthology.orchestra.run.smithi100.stdout: "source_version": "0'0", 2024-09-17T07:07:42.964 INFO:teuthology.orchestra.run.smithi100.stdout: "target_version": "0'0" 2024-09-17T07:07:42.964 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:07:42.964 INFO:teuthology.orchestra.run.smithi100.stdout: "last_change": "113", 2024-09-17T07:07:42.964 INFO:teuthology.orchestra.run.smithi100.stdout: "last_force_op_resend": "0", 2024-09-17T07:07:42.964 INFO:teuthology.orchestra.run.smithi100.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-17T07:07:42.964 INFO:teuthology.orchestra.run.smithi100.stdout: "last_force_op_resend_preluminous": "111", 2024-09-17T07:07:42.964 INFO:teuthology.orchestra.run.smithi100.stdout: "auid": 0, 2024-09-17T07:07:42.964 INFO:teuthology.orchestra.run.smithi100.stdout: "snap_mode": "selfmanaged", 2024-09-17T07:07:42.964 INFO:teuthology.orchestra.run.smithi100.stdout: "snap_seq": 0, 2024-09-17T07:07:42.964 INFO:teuthology.orchestra.run.smithi100.stdout: "snap_epoch": 0, 2024-09-17T07:07:42.965 INFO:teuthology.orchestra.run.smithi100.stdout: "pool_snaps": [], 2024-09-17T07:07:42.965 INFO:teuthology.orchestra.run.smithi100.stdout: "removed_snaps": "[]", 2024-09-17T07:07:42.965 INFO:teuthology.orchestra.run.smithi100.stdout: "quota_max_bytes": 0, 2024-09-17T07:07:42.965 INFO:teuthology.orchestra.run.smithi100.stdout: "quota_max_objects": 0, 2024-09-17T07:07:42.965 INFO:teuthology.orchestra.run.smithi100.stdout: "tiers": [], 2024-09-17T07:07:42.965 INFO:teuthology.orchestra.run.smithi100.stdout: "tier_of": -1, 2024-09-17T07:07:42.965 INFO:teuthology.orchestra.run.smithi100.stdout: "read_tier": -1, 2024-09-17T07:07:42.965 INFO:teuthology.orchestra.run.smithi100.stdout: "write_tier": -1, 2024-09-17T07:07:42.965 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_mode": "none", 2024-09-17T07:07:42.965 INFO:teuthology.orchestra.run.smithi100.stdout: "target_max_bytes": 0, 2024-09-17T07:07:42.965 INFO:teuthology.orchestra.run.smithi100.stdout: "target_max_objects": 0, 2024-09-17T07:07:42.965 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T07:07:42.965 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T07:07:42.965 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T07:07:42.966 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_min_flush_age": 0, 2024-09-17T07:07:42.966 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_min_evict_age": 0, 2024-09-17T07:07:42.966 INFO:teuthology.orchestra.run.smithi100.stdout: "erasure_code_profile": "", 2024-09-17T07:07:42.966 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_params": { 2024-09-17T07:07:42.966 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "none" 2024-09-17T07:07:42.966 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:07:42.966 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_period": 0, 2024-09-17T07:07:42.966 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_count": 0, 2024-09-17T07:07:42.966 INFO:teuthology.orchestra.run.smithi100.stdout: "use_gmt_hitset": true, 2024-09-17T07:07:42.966 INFO:teuthology.orchestra.run.smithi100.stdout: "min_read_recency_for_promote": 0, 2024-09-17T07:07:42.966 INFO:teuthology.orchestra.run.smithi100.stdout: "min_write_recency_for_promote": 0, 2024-09-17T07:07:42.966 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T07:07:42.966 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_search_last_n": 0, 2024-09-17T07:07:42.967 INFO:teuthology.orchestra.run.smithi100.stdout: "grade_table": [], 2024-09-17T07:07:42.967 INFO:teuthology.orchestra.run.smithi100.stdout: "stripe_width": 0, 2024-09-17T07:07:42.967 INFO:teuthology.orchestra.run.smithi100.stdout: "expected_num_objects": 0, 2024-09-17T07:07:42.967 INFO:teuthology.orchestra.run.smithi100.stdout: "fast_read": false, 2024-09-17T07:07:42.967 INFO:teuthology.orchestra.run.smithi100.stdout: "options": {}, 2024-09-17T07:07:42.967 INFO:teuthology.orchestra.run.smithi100.stdout: "application_metadata": { 2024-09-17T07:07:42.967 INFO:teuthology.orchestra.run.smithi100.stdout: "rados": {} 2024-09-17T07:07:42.967 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:07:42.967 INFO:teuthology.orchestra.run.smithi100.stdout: "read_balance": { 2024-09-17T07:07:42.967 INFO:teuthology.orchestra.run.smithi100.stdout: "score_acting": 1.8500000238418579, 2024-09-17T07:07:42.967 INFO:teuthology.orchestra.run.smithi100.stdout: "score_stable": 1.8500000238418579, 2024-09-17T07:07:42.967 INFO:teuthology.orchestra.run.smithi100.stdout: "optimal_score": 1, 2024-09-17T07:07:42.967 INFO:teuthology.orchestra.run.smithi100.stdout: "raw_score_acting": 1.8500000238418579, 2024-09-17T07:07:42.968 INFO:teuthology.orchestra.run.smithi100.stdout: "raw_score_stable": 1.8500000238418579, 2024-09-17T07:07:42.968 INFO:teuthology.orchestra.run.smithi100.stdout: "primary_affinity_weighted": 0.89598274230957031, 2024-09-17T07:07:42.968 INFO:teuthology.orchestra.run.smithi100.stdout: "average_primary_affinity": 0.89999997615814209, 2024-09-17T07:07:42.968 INFO:teuthology.orchestra.run.smithi100.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-09-17T07:07:42.968 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:07:42.968 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:07:42.968 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-17T07:07:42.968 INFO:teuthology.orchestra.run.smithi100.stdout: "osds": [ 2024-09-17T07:07:42.968 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:07:42.968 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": 0, 2024-09-17T07:07:42.968 INFO:teuthology.orchestra.run.smithi100.stdout: "uuid": "27bee9ec-05e6-406c-8bf1-168098da0ec7", 2024-09-17T07:07:42.968 INFO:teuthology.orchestra.run.smithi100.stdout: "up": 1, 2024-09-17T07:07:42.968 INFO:teuthology.orchestra.run.smithi100.stdout: "in": 1, 2024-09-17T07:07:42.968 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 1, 2024-09-17T07:07:42.969 INFO:teuthology.orchestra.run.smithi100.stdout: "primary_affinity": 1, 2024-09-17T07:07:42.969 INFO:teuthology.orchestra.run.smithi100.stdout: "last_clean_begin": 0, 2024-09-17T07:07:42.969 INFO:teuthology.orchestra.run.smithi100.stdout: "last_clean_end": 0, 2024-09-17T07:07:42.969 INFO:teuthology.orchestra.run.smithi100.stdout: "up_from": 8, 2024-09-17T07:07:42.969 INFO:teuthology.orchestra.run.smithi100.stdout: "up_thru": 146, 2024-09-17T07:07:42.969 INFO:teuthology.orchestra.run.smithi100.stdout: "down_at": 0, 2024-09-17T07:07:42.969 INFO:teuthology.orchestra.run.smithi100.stdout: "lost_at": 0, 2024-09-17T07:07:42.969 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-17T07:07:42.969 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:07:42.969 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:07:42.969 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:07:42.969 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6801", 2024-09-17T07:07:42.969 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 3341017455 2024-09-17T07:07:42.970 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:07:42.970 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:07:42.970 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:07:42.970 INFO:teuthology.orchestra.run.smithi100.stdout: "cluster_addrs": { 2024-09-17T07:07:42.970 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:07:42.970 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:07:42.970 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:07:42.970 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6802", 2024-09-17T07:07:42.970 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 3341017455 2024-09-17T07:07:42.970 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:07:42.970 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:07:42.970 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:07:42.970 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_back_addrs": { 2024-09-17T07:07:42.971 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:07:42.971 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:07:42.971 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:07:42.971 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6804", 2024-09-17T07:07:42.971 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 3341017455 2024-09-17T07:07:42.971 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:07:42.971 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:07:42.971 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:07:42.971 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_front_addrs": { 2024-09-17T07:07:42.971 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:07:42.971 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:07:42.971 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:07:42.971 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6803", 2024-09-17T07:07:42.972 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 3341017455 2024-09-17T07:07:42.972 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:07:42.972 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:07:42.972 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:07:42.972 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.100:6801/3341017455", 2024-09-17T07:07:42.972 INFO:teuthology.orchestra.run.smithi100.stdout: "cluster_addr": "172.21.15.100:6802/3341017455", 2024-09-17T07:07:42.972 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_back_addr": "172.21.15.100:6804/3341017455", 2024-09-17T07:07:42.972 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_front_addr": "172.21.15.100:6803/3341017455", 2024-09-17T07:07:42.972 INFO:teuthology.orchestra.run.smithi100.stdout: "state": [ 2024-09-17T07:07:42.972 INFO:teuthology.orchestra.run.smithi100.stdout: "exists", 2024-09-17T07:07:42.972 INFO:teuthology.orchestra.run.smithi100.stdout: "up" 2024-09-17T07:07:42.972 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:07:42.972 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:07:42.973 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:07:42.973 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": 1, 2024-09-17T07:07:42.973 INFO:teuthology.orchestra.run.smithi100.stdout: "uuid": "b1a30c9c-eeaf-478e-9b62-2d63ad45873d", 2024-09-17T07:07:42.973 INFO:teuthology.orchestra.run.smithi100.stdout: "up": 1, 2024-09-17T07:07:42.973 INFO:teuthology.orchestra.run.smithi100.stdout: "in": 1, 2024-09-17T07:07:42.973 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 1, 2024-09-17T07:07:42.973 INFO:teuthology.orchestra.run.smithi100.stdout: "primary_affinity": 1, 2024-09-17T07:07:42.973 INFO:teuthology.orchestra.run.smithi100.stdout: "last_clean_begin": 0, 2024-09-17T07:07:42.973 INFO:teuthology.orchestra.run.smithi100.stdout: "last_clean_end": 0, 2024-09-17T07:07:42.973 INFO:teuthology.orchestra.run.smithi100.stdout: "up_from": 13, 2024-09-17T07:07:42.973 INFO:teuthology.orchestra.run.smithi100.stdout: "up_thru": 99, 2024-09-17T07:07:42.973 INFO:teuthology.orchestra.run.smithi100.stdout: "down_at": 0, 2024-09-17T07:07:42.973 INFO:teuthology.orchestra.run.smithi100.stdout: "lost_at": 0, 2024-09-17T07:07:42.974 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-17T07:07:42.974 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:07:42.974 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:07:42.974 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:07:42.974 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6805", 2024-09-17T07:07:42.974 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 1634750965 2024-09-17T07:07:42.974 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:07:42.974 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:07:42.974 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:07:42.974 INFO:teuthology.orchestra.run.smithi100.stdout: "cluster_addrs": { 2024-09-17T07:07:42.974 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:07:42.974 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:07:42.974 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:07:42.975 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6806", 2024-09-17T07:07:42.975 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 1634750965 2024-09-17T07:07:42.975 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:07:42.975 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:07:42.975 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:07:42.975 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_back_addrs": { 2024-09-17T07:07:42.975 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:07:42.975 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:07:42.975 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:07:42.975 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6808", 2024-09-17T07:07:42.975 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 1634750965 2024-09-17T07:07:42.975 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:07:42.975 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:07:42.976 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:07:42.976 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_front_addrs": { 2024-09-17T07:07:42.976 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:07:42.976 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:07:42.976 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:07:42.976 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6807", 2024-09-17T07:07:42.976 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 1634750965 2024-09-17T07:07:42.976 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:07:42.976 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:07:42.976 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:07:42.976 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.100:6805/1634750965", 2024-09-17T07:07:42.976 INFO:teuthology.orchestra.run.smithi100.stdout: "cluster_addr": "172.21.15.100:6806/1634750965", 2024-09-17T07:07:42.976 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_back_addr": "172.21.15.100:6808/1634750965", 2024-09-17T07:07:42.977 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_front_addr": "172.21.15.100:6807/1634750965", 2024-09-17T07:07:42.977 INFO:teuthology.orchestra.run.smithi100.stdout: "state": [ 2024-09-17T07:07:42.977 INFO:teuthology.orchestra.run.smithi100.stdout: "exists", 2024-09-17T07:07:42.977 INFO:teuthology.orchestra.run.smithi100.stdout: "up" 2024-09-17T07:07:42.977 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:07:42.977 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:07:42.977 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:07:42.977 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": 2, 2024-09-17T07:07:42.977 INFO:teuthology.orchestra.run.smithi100.stdout: "uuid": "ce542e0d-9915-465e-aa11-7b3ce13f694f", 2024-09-17T07:07:42.977 INFO:teuthology.orchestra.run.smithi100.stdout: "up": 1, 2024-09-17T07:07:42.977 INFO:teuthology.orchestra.run.smithi100.stdout: "in": 1, 2024-09-17T07:07:42.977 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 1, 2024-09-17T07:07:42.977 INFO:teuthology.orchestra.run.smithi100.stdout: "primary_affinity": 0.1678619384765625, 2024-09-17T07:07:42.978 INFO:teuthology.orchestra.run.smithi100.stdout: "last_clean_begin": 0, 2024-09-17T07:07:42.978 INFO:teuthology.orchestra.run.smithi100.stdout: "last_clean_end": 0, 2024-09-17T07:07:42.978 INFO:teuthology.orchestra.run.smithi100.stdout: "up_from": 18, 2024-09-17T07:07:42.978 INFO:teuthology.orchestra.run.smithi100.stdout: "up_thru": 133, 2024-09-17T07:07:42.978 INFO:teuthology.orchestra.run.smithi100.stdout: "down_at": 0, 2024-09-17T07:07:42.978 INFO:teuthology.orchestra.run.smithi100.stdout: "lost_at": 0, 2024-09-17T07:07:42.978 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-17T07:07:42.978 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:07:42.978 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:07:42.978 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:07:42.978 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6809", 2024-09-17T07:07:42.978 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 714579846 2024-09-17T07:07:42.978 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:07:42.978 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:07:42.979 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:07:42.979 INFO:teuthology.orchestra.run.smithi100.stdout: "cluster_addrs": { 2024-09-17T07:07:42.979 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:07:42.979 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:07:42.979 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:07:42.979 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6810", 2024-09-17T07:07:42.979 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 714579846 2024-09-17T07:07:42.979 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:07:42.979 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:07:42.979 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:07:42.979 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_back_addrs": { 2024-09-17T07:07:42.979 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:07:42.979 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:07:42.980 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:07:42.980 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6812", 2024-09-17T07:07:42.980 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 714579846 2024-09-17T07:07:42.980 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:07:42.980 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:07:42.980 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:07:42.980 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_front_addrs": { 2024-09-17T07:07:42.980 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:07:42.980 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:07:42.980 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:07:42.980 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6811", 2024-09-17T07:07:42.980 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 714579846 2024-09-17T07:07:42.980 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:07:42.981 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:07:42.981 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:07:42.981 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.100:6809/714579846", 2024-09-17T07:07:42.981 INFO:teuthology.orchestra.run.smithi100.stdout: "cluster_addr": "172.21.15.100:6810/714579846", 2024-09-17T07:07:42.981 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_back_addr": "172.21.15.100:6812/714579846", 2024-09-17T07:07:42.981 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_front_addr": "172.21.15.100:6811/714579846", 2024-09-17T07:07:42.981 INFO:teuthology.orchestra.run.smithi100.stdout: "state": [ 2024-09-17T07:07:42.981 INFO:teuthology.orchestra.run.smithi100.stdout: "exists", 2024-09-17T07:07:42.981 INFO:teuthology.orchestra.run.smithi100.stdout: "up" 2024-09-17T07:07:42.981 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:07:42.981 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:07:42.981 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:07:42.981 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": 3, 2024-09-17T07:07:42.982 INFO:teuthology.orchestra.run.smithi100.stdout: "uuid": "a1257575-3c90-41de-88f6-3960744351b1", 2024-09-17T07:07:42.982 INFO:teuthology.orchestra.run.smithi100.stdout: "up": 1, 2024-09-17T07:07:42.982 INFO:teuthology.orchestra.run.smithi100.stdout: "in": 1, 2024-09-17T07:07:42.982 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 1, 2024-09-17T07:07:42.982 INFO:teuthology.orchestra.run.smithi100.stdout: "primary_affinity": 1, 2024-09-17T07:07:42.982 INFO:teuthology.orchestra.run.smithi100.stdout: "last_clean_begin": 0, 2024-09-17T07:07:42.982 INFO:teuthology.orchestra.run.smithi100.stdout: "last_clean_end": 0, 2024-09-17T07:07:42.982 INFO:teuthology.orchestra.run.smithi100.stdout: "up_from": 27, 2024-09-17T07:07:42.982 INFO:teuthology.orchestra.run.smithi100.stdout: "up_thru": 147, 2024-09-17T07:07:42.982 INFO:teuthology.orchestra.run.smithi100.stdout: "down_at": 0, 2024-09-17T07:07:42.982 INFO:teuthology.orchestra.run.smithi100.stdout: "lost_at": 0, 2024-09-17T07:07:42.982 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-17T07:07:42.983 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:07:42.983 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:07:42.983 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:07:42.983 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6813", 2024-09-17T07:07:42.983 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 2315956572 2024-09-17T07:07:42.983 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:07:42.983 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:07:42.983 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:07:42.983 INFO:teuthology.orchestra.run.smithi100.stdout: "cluster_addrs": { 2024-09-17T07:07:42.983 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:07:42.983 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:07:42.983 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:07:42.983 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6814", 2024-09-17T07:07:42.983 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 2315956572 2024-09-17T07:07:42.984 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:07:42.984 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:07:42.984 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:07:42.984 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_back_addrs": { 2024-09-17T07:07:42.984 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:07:42.984 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:07:42.984 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:07:42.984 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6816", 2024-09-17T07:07:42.984 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 2315956572 2024-09-17T07:07:42.984 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:07:42.984 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:07:42.984 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:07:42.985 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_front_addrs": { 2024-09-17T07:07:42.985 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:07:42.985 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:07:42.985 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:07:42.985 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6815", 2024-09-17T07:07:42.985 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 2315956572 2024-09-17T07:07:42.985 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:07:42.985 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:07:42.985 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:07:42.985 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.100:6813/2315956572", 2024-09-17T07:07:42.985 INFO:teuthology.orchestra.run.smithi100.stdout: "cluster_addr": "172.21.15.100:6814/2315956572", 2024-09-17T07:07:42.985 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_back_addr": "172.21.15.100:6816/2315956572", 2024-09-17T07:07:42.985 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_front_addr": "172.21.15.100:6815/2315956572", 2024-09-17T07:07:42.986 INFO:teuthology.orchestra.run.smithi100.stdout: "state": [ 2024-09-17T07:07:42.986 INFO:teuthology.orchestra.run.smithi100.stdout: "exists", 2024-09-17T07:07:42.986 INFO:teuthology.orchestra.run.smithi100.stdout: "up" 2024-09-17T07:07:42.986 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:07:42.986 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:07:42.986 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:07:42.986 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": 4, 2024-09-17T07:07:42.986 INFO:teuthology.orchestra.run.smithi100.stdout: "uuid": "e44e8922-2c5e-44e3-b556-d4dfbc499649", 2024-09-17T07:07:42.986 INFO:teuthology.orchestra.run.smithi100.stdout: "up": 1, 2024-09-17T07:07:42.986 INFO:teuthology.orchestra.run.smithi100.stdout: "in": 1, 2024-09-17T07:07:42.986 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 1, 2024-09-17T07:07:42.986 INFO:teuthology.orchestra.run.smithi100.stdout: "primary_affinity": 1, 2024-09-17T07:07:42.986 INFO:teuthology.orchestra.run.smithi100.stdout: "last_clean_begin": 0, 2024-09-17T07:07:42.986 INFO:teuthology.orchestra.run.smithi100.stdout: "last_clean_end": 0, 2024-09-17T07:07:42.987 INFO:teuthology.orchestra.run.smithi100.stdout: "up_from": 33, 2024-09-17T07:07:42.987 INFO:teuthology.orchestra.run.smithi100.stdout: "up_thru": 123, 2024-09-17T07:07:42.987 INFO:teuthology.orchestra.run.smithi100.stdout: "down_at": 0, 2024-09-17T07:07:42.987 INFO:teuthology.orchestra.run.smithi100.stdout: "lost_at": 0, 2024-09-17T07:07:42.987 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-17T07:07:42.987 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:07:42.987 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:07:42.987 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:07:42.987 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.135:6800", 2024-09-17T07:07:42.987 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 1148820922 2024-09-17T07:07:42.987 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:07:42.987 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:07:42.987 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:07:42.988 INFO:teuthology.orchestra.run.smithi100.stdout: "cluster_addrs": { 2024-09-17T07:07:42.988 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:07:42.988 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:07:42.988 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:07:42.988 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.135:6801", 2024-09-17T07:07:42.988 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 1148820922 2024-09-17T07:07:42.988 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:07:42.988 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:07:42.988 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:07:42.988 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_back_addrs": { 2024-09-17T07:07:42.988 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:07:42.988 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:07:42.988 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:07:42.989 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.135:6803", 2024-09-17T07:07:42.989 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 1148820922 2024-09-17T07:07:42.989 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:07:42.989 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:07:42.989 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:07:42.989 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_front_addrs": { 2024-09-17T07:07:42.989 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:07:42.989 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:07:42.989 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:07:42.989 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.135:6802", 2024-09-17T07:07:42.989 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 1148820922 2024-09-17T07:07:42.989 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:07:42.989 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:07:42.990 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:07:42.990 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.135:6800/1148820922", 2024-09-17T07:07:42.990 INFO:teuthology.orchestra.run.smithi100.stdout: "cluster_addr": "172.21.15.135:6801/1148820922", 2024-09-17T07:07:42.990 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_back_addr": "172.21.15.135:6803/1148820922", 2024-09-17T07:07:42.990 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_front_addr": "172.21.15.135:6802/1148820922", 2024-09-17T07:07:42.990 INFO:teuthology.orchestra.run.smithi100.stdout: "state": [ 2024-09-17T07:07:42.990 INFO:teuthology.orchestra.run.smithi100.stdout: "exists", 2024-09-17T07:07:42.990 INFO:teuthology.orchestra.run.smithi100.stdout: "up" 2024-09-17T07:07:42.990 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:07:42.990 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:07:42.990 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:07:42.991 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": 5, 2024-09-17T07:07:42.991 INFO:teuthology.orchestra.run.smithi100.stdout: "uuid": "91e4a362-fcdb-41b9-b54e-716591ba6d79", 2024-09-17T07:07:42.991 INFO:teuthology.orchestra.run.smithi100.stdout: "up": 1, 2024-09-17T07:07:42.991 INFO:teuthology.orchestra.run.smithi100.stdout: "in": 1, 2024-09-17T07:07:42.991 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 1, 2024-09-17T07:07:42.991 INFO:teuthology.orchestra.run.smithi100.stdout: "primary_affinity": 1, 2024-09-17T07:07:42.991 INFO:teuthology.orchestra.run.smithi100.stdout: "last_clean_begin": 0, 2024-09-17T07:07:42.991 INFO:teuthology.orchestra.run.smithi100.stdout: "last_clean_end": 0, 2024-09-17T07:07:42.991 INFO:teuthology.orchestra.run.smithi100.stdout: "up_from": 42, 2024-09-17T07:07:42.991 INFO:teuthology.orchestra.run.smithi100.stdout: "up_thru": 111, 2024-09-17T07:07:42.991 INFO:teuthology.orchestra.run.smithi100.stdout: "down_at": 0, 2024-09-17T07:07:42.991 INFO:teuthology.orchestra.run.smithi100.stdout: "lost_at": 0, 2024-09-17T07:07:42.991 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-17T07:07:42.991 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:07:42.992 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:07:42.992 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:07:42.992 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.135:6804", 2024-09-17T07:07:42.992 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 2256520893 2024-09-17T07:07:42.992 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:07:42.992 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:07:42.992 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:07:42.992 INFO:teuthology.orchestra.run.smithi100.stdout: "cluster_addrs": { 2024-09-17T07:07:42.992 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:07:42.992 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:07:42.992 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:07:42.992 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.135:6805", 2024-09-17T07:07:42.992 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 2256520893 2024-09-17T07:07:42.993 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:07:42.993 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:07:42.993 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:07:42.993 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_back_addrs": { 2024-09-17T07:07:42.993 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:07:42.993 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:07:42.993 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:07:42.993 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.135:6807", 2024-09-17T07:07:42.993 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 2256520893 2024-09-17T07:07:42.993 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:07:42.993 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:07:42.993 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:07:42.993 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_front_addrs": { 2024-09-17T07:07:42.994 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:07:42.994 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:07:42.994 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:07:42.994 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.135:6806", 2024-09-17T07:07:42.994 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 2256520893 2024-09-17T07:07:42.994 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:07:42.994 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:07:42.994 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:07:42.994 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.135:6804/2256520893", 2024-09-17T07:07:42.994 INFO:teuthology.orchestra.run.smithi100.stdout: "cluster_addr": "172.21.15.135:6805/2256520893", 2024-09-17T07:07:42.994 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_back_addr": "172.21.15.135:6807/2256520893", 2024-09-17T07:07:42.994 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_front_addr": "172.21.15.135:6806/2256520893", 2024-09-17T07:07:42.994 INFO:teuthology.orchestra.run.smithi100.stdout: "state": [ 2024-09-17T07:07:42.995 INFO:teuthology.orchestra.run.smithi100.stdout: "exists", 2024-09-17T07:07:42.995 INFO:teuthology.orchestra.run.smithi100.stdout: "up" 2024-09-17T07:07:42.995 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:07:42.995 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:07:42.995 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:07:42.995 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": 6, 2024-09-17T07:07:42.995 INFO:teuthology.orchestra.run.smithi100.stdout: "uuid": "de799f64-ebd2-4a46-91f6-c2f276bee21f", 2024-09-17T07:07:42.995 INFO:teuthology.orchestra.run.smithi100.stdout: "up": 1, 2024-09-17T07:07:42.995 INFO:teuthology.orchestra.run.smithi100.stdout: "in": 1, 2024-09-17T07:07:42.995 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 1, 2024-09-17T07:07:42.995 INFO:teuthology.orchestra.run.smithi100.stdout: "primary_affinity": 1, 2024-09-17T07:07:42.995 INFO:teuthology.orchestra.run.smithi100.stdout: "last_clean_begin": 0, 2024-09-17T07:07:42.995 INFO:teuthology.orchestra.run.smithi100.stdout: "last_clean_end": 0, 2024-09-17T07:07:42.996 INFO:teuthology.orchestra.run.smithi100.stdout: "up_from": 50, 2024-09-17T07:07:42.996 INFO:teuthology.orchestra.run.smithi100.stdout: "up_thru": 139, 2024-09-17T07:07:42.996 INFO:teuthology.orchestra.run.smithi100.stdout: "down_at": 0, 2024-09-17T07:07:42.996 INFO:teuthology.orchestra.run.smithi100.stdout: "lost_at": 0, 2024-09-17T07:07:42.996 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-17T07:07:42.996 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:07:42.996 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:07:42.996 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:07:42.996 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.135:6808", 2024-09-17T07:07:42.996 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 2750184152 2024-09-17T07:07:42.996 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:07:42.996 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:07:42.996 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:07:42.997 INFO:teuthology.orchestra.run.smithi100.stdout: "cluster_addrs": { 2024-09-17T07:07:42.997 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:07:42.997 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:07:42.997 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:07:42.997 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.135:6809", 2024-09-17T07:07:42.997 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 2750184152 2024-09-17T07:07:42.997 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:07:42.997 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:07:42.997 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:07:42.997 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_back_addrs": { 2024-09-17T07:07:42.997 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:07:42.997 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:07:42.997 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:07:42.998 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.135:6811", 2024-09-17T07:07:42.998 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 2750184152 2024-09-17T07:07:42.998 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:07:42.998 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:07:42.998 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:07:42.998 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_front_addrs": { 2024-09-17T07:07:42.998 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:07:42.998 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:07:42.998 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:07:42.998 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.135:6810", 2024-09-17T07:07:42.998 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 2750184152 2024-09-17T07:07:42.998 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:07:42.999 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:07:42.999 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:07:42.999 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.135:6808/2750184152", 2024-09-17T07:07:42.999 INFO:teuthology.orchestra.run.smithi100.stdout: "cluster_addr": "172.21.15.135:6809/2750184152", 2024-09-17T07:07:42.999 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_back_addr": "172.21.15.135:6811/2750184152", 2024-09-17T07:07:42.999 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_front_addr": "172.21.15.135:6810/2750184152", 2024-09-17T07:07:42.999 INFO:teuthology.orchestra.run.smithi100.stdout: "state": [ 2024-09-17T07:07:42.999 INFO:teuthology.orchestra.run.smithi100.stdout: "exists", 2024-09-17T07:07:42.999 INFO:teuthology.orchestra.run.smithi100.stdout: "up" 2024-09-17T07:07:42.999 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:07:42.999 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:07:42.999 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:07:42.999 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": 7, 2024-09-17T07:07:43.000 INFO:teuthology.orchestra.run.smithi100.stdout: "uuid": "830932c7-5de1-44ff-a840-e53a6e859bda", 2024-09-17T07:07:43.000 INFO:teuthology.orchestra.run.smithi100.stdout: "up": 1, 2024-09-17T07:07:43.000 INFO:teuthology.orchestra.run.smithi100.stdout: "in": 1, 2024-09-17T07:07:43.000 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 1, 2024-09-17T07:07:43.000 INFO:teuthology.orchestra.run.smithi100.stdout: "primary_affinity": 1, 2024-09-17T07:07:43.000 INFO:teuthology.orchestra.run.smithi100.stdout: "last_clean_begin": 0, 2024-09-17T07:07:43.000 INFO:teuthology.orchestra.run.smithi100.stdout: "last_clean_end": 0, 2024-09-17T07:07:43.000 INFO:teuthology.orchestra.run.smithi100.stdout: "up_from": 60, 2024-09-17T07:07:43.000 INFO:teuthology.orchestra.run.smithi100.stdout: "up_thru": 145, 2024-09-17T07:07:43.000 INFO:teuthology.orchestra.run.smithi100.stdout: "down_at": 0, 2024-09-17T07:07:43.000 INFO:teuthology.orchestra.run.smithi100.stdout: "lost_at": 0, 2024-09-17T07:07:43.000 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-17T07:07:43.000 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:07:43.001 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:07:43.001 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:07:43.001 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.135:6812", 2024-09-17T07:07:43.001 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 160304773 2024-09-17T07:07:43.001 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:07:43.001 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:07:43.001 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:07:43.001 INFO:teuthology.orchestra.run.smithi100.stdout: "cluster_addrs": { 2024-09-17T07:07:43.001 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:07:43.001 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:07:43.001 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:07:43.001 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.135:6813", 2024-09-17T07:07:43.002 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 160304773 2024-09-17T07:07:43.002 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:07:43.002 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:07:43.002 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:07:43.002 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_back_addrs": { 2024-09-17T07:07:43.002 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:07:43.002 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:07:43.002 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:07:43.002 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.135:6815", 2024-09-17T07:07:43.002 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 160304773 2024-09-17T07:07:43.002 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:07:43.003 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:07:43.003 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:07:43.003 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_front_addrs": { 2024-09-17T07:07:43.003 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:07:43.003 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:07:43.003 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:07:43.003 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.135:6814", 2024-09-17T07:07:43.003 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 160304773 2024-09-17T07:07:43.003 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:07:43.003 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:07:43.003 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:07:43.003 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.135:6812/160304773", 2024-09-17T07:07:43.003 INFO:teuthology.orchestra.run.smithi100.stdout: "cluster_addr": "172.21.15.135:6813/160304773", 2024-09-17T07:07:43.004 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_back_addr": "172.21.15.135:6815/160304773", 2024-09-17T07:07:43.004 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_front_addr": "172.21.15.135:6814/160304773", 2024-09-17T07:07:43.004 INFO:teuthology.orchestra.run.smithi100.stdout: "state": [ 2024-09-17T07:07:43.004 INFO:teuthology.orchestra.run.smithi100.stdout: "exists", 2024-09-17T07:07:43.004 INFO:teuthology.orchestra.run.smithi100.stdout: "up" 2024-09-17T07:07:43.004 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:07:43.004 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:07:43.004 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-17T07:07:43.004 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_xinfo": [ 2024-09-17T07:07:43.004 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:07:43.004 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": 0, 2024-09-17T07:07:43.004 INFO:teuthology.orchestra.run.smithi100.stdout: "down_stamp": "0.000000", 2024-09-17T07:07:43.004 INFO:teuthology.orchestra.run.smithi100.stdout: "laggy_probability": 0, 2024-09-17T07:07:43.005 INFO:teuthology.orchestra.run.smithi100.stdout: "laggy_interval": 0, 2024-09-17T07:07:43.005 INFO:teuthology.orchestra.run.smithi100.stdout: "features": 4540138322906710015, 2024-09-17T07:07:43.005 INFO:teuthology.orchestra.run.smithi100.stdout: "old_weight": 0, 2024-09-17T07:07:43.005 INFO:teuthology.orchestra.run.smithi100.stdout: "last_purged_snaps_scrub": "2024-09-17T07:07:24.189700+0000", 2024-09-17T07:07:43.005 INFO:teuthology.orchestra.run.smithi100.stdout: "dead_epoch": 0 2024-09-17T07:07:43.005 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:07:43.005 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:07:43.005 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": 1, 2024-09-17T07:07:43.005 INFO:teuthology.orchestra.run.smithi100.stdout: "down_stamp": "0.000000", 2024-09-17T07:07:43.005 INFO:teuthology.orchestra.run.smithi100.stdout: "laggy_probability": 0, 2024-09-17T07:07:43.005 INFO:teuthology.orchestra.run.smithi100.stdout: "laggy_interval": 0, 2024-09-17T07:07:43.005 INFO:teuthology.orchestra.run.smithi100.stdout: "features": 4540138322906710015, 2024-09-17T07:07:43.005 INFO:teuthology.orchestra.run.smithi100.stdout: "old_weight": 0, 2024-09-17T07:07:43.006 INFO:teuthology.orchestra.run.smithi100.stdout: "last_purged_snaps_scrub": "2024-09-17T07:06:58.606168+0000", 2024-09-17T07:07:43.006 INFO:teuthology.orchestra.run.smithi100.stdout: "dead_epoch": 0 2024-09-17T07:07:43.006 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:07:43.006 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:07:43.006 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": 2, 2024-09-17T07:07:43.006 INFO:teuthology.orchestra.run.smithi100.stdout: "down_stamp": "0.000000", 2024-09-17T07:07:43.006 INFO:teuthology.orchestra.run.smithi100.stdout: "laggy_probability": 0, 2024-09-17T07:07:43.006 INFO:teuthology.orchestra.run.smithi100.stdout: "laggy_interval": 0, 2024-09-17T07:07:43.006 INFO:teuthology.orchestra.run.smithi100.stdout: "features": 4540138322906710015, 2024-09-17T07:07:43.006 INFO:teuthology.orchestra.run.smithi100.stdout: "old_weight": 0, 2024-09-17T07:07:43.006 INFO:teuthology.orchestra.run.smithi100.stdout: "last_purged_snaps_scrub": "2024-09-17T07:07:32.041868+0000", 2024-09-17T07:07:43.006 INFO:teuthology.orchestra.run.smithi100.stdout: "dead_epoch": 0 2024-09-17T07:07:43.006 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:07:43.006 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:07:43.007 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": 3, 2024-09-17T07:07:43.007 INFO:teuthology.orchestra.run.smithi100.stdout: "down_stamp": "0.000000", 2024-09-17T07:07:43.007 INFO:teuthology.orchestra.run.smithi100.stdout: "laggy_probability": 0, 2024-09-17T07:07:43.007 INFO:teuthology.orchestra.run.smithi100.stdout: "laggy_interval": 0, 2024-09-17T07:07:43.007 INFO:teuthology.orchestra.run.smithi100.stdout: "features": 4540138322906710015, 2024-09-17T07:07:43.007 INFO:teuthology.orchestra.run.smithi100.stdout: "old_weight": 0, 2024-09-17T07:07:43.007 INFO:teuthology.orchestra.run.smithi100.stdout: "last_purged_snaps_scrub": "2024-09-17T07:07:09.366163+0000", 2024-09-17T07:07:43.007 INFO:teuthology.orchestra.run.smithi100.stdout: "dead_epoch": 0 2024-09-17T07:07:43.007 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:07:43.007 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:07:43.007 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": 4, 2024-09-17T07:07:43.007 INFO:teuthology.orchestra.run.smithi100.stdout: "down_stamp": "0.000000", 2024-09-17T07:07:43.007 INFO:teuthology.orchestra.run.smithi100.stdout: "laggy_probability": 0, 2024-09-17T07:07:43.007 INFO:teuthology.orchestra.run.smithi100.stdout: "laggy_interval": 0, 2024-09-17T07:07:43.008 INFO:teuthology.orchestra.run.smithi100.stdout: "features": 4540138322906710015, 2024-09-17T07:07:43.008 INFO:teuthology.orchestra.run.smithi100.stdout: "old_weight": 0, 2024-09-17T07:07:43.008 INFO:teuthology.orchestra.run.smithi100.stdout: "last_purged_snaps_scrub": "2024-09-17T07:07:10.689524+0000", 2024-09-17T07:07:43.008 INFO:teuthology.orchestra.run.smithi100.stdout: "dead_epoch": 0 2024-09-17T07:07:43.008 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:07:43.008 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:07:43.008 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": 5, 2024-09-17T07:07:43.008 INFO:teuthology.orchestra.run.smithi100.stdout: "down_stamp": "0.000000", 2024-09-17T07:07:43.008 INFO:teuthology.orchestra.run.smithi100.stdout: "laggy_probability": 0, 2024-09-17T07:07:43.008 INFO:teuthology.orchestra.run.smithi100.stdout: "laggy_interval": 0, 2024-09-17T07:07:43.008 INFO:teuthology.orchestra.run.smithi100.stdout: "features": 4540138322906710015, 2024-09-17T07:07:43.008 INFO:teuthology.orchestra.run.smithi100.stdout: "old_weight": 0, 2024-09-17T07:07:43.008 INFO:teuthology.orchestra.run.smithi100.stdout: "last_purged_snaps_scrub": "2024-09-17T07:07:10.504050+0000", 2024-09-17T07:07:43.009 INFO:teuthology.orchestra.run.smithi100.stdout: "dead_epoch": 0 2024-09-17T07:07:43.009 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:07:43.009 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:07:43.009 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": 6, 2024-09-17T07:07:43.009 INFO:teuthology.orchestra.run.smithi100.stdout: "down_stamp": "0.000000", 2024-09-17T07:07:43.009 INFO:teuthology.orchestra.run.smithi100.stdout: "laggy_probability": 0, 2024-09-17T07:07:43.009 INFO:teuthology.orchestra.run.smithi100.stdout: "laggy_interval": 0, 2024-09-17T07:07:43.009 INFO:teuthology.orchestra.run.smithi100.stdout: "features": 4540138322906710015, 2024-09-17T07:07:43.009 INFO:teuthology.orchestra.run.smithi100.stdout: "old_weight": 0, 2024-09-17T07:07:43.009 INFO:teuthology.orchestra.run.smithi100.stdout: "last_purged_snaps_scrub": "2024-09-17T07:07:36.868729+0000", 2024-09-17T07:07:43.009 INFO:teuthology.orchestra.run.smithi100.stdout: "dead_epoch": 0 2024-09-17T07:07:43.009 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:07:43.009 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:07:43.009 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": 7, 2024-09-17T07:07:43.010 INFO:teuthology.orchestra.run.smithi100.stdout: "down_stamp": "0.000000", 2024-09-17T07:07:43.010 INFO:teuthology.orchestra.run.smithi100.stdout: "laggy_probability": 0, 2024-09-17T07:07:43.010 INFO:teuthology.orchestra.run.smithi100.stdout: "laggy_interval": 0, 2024-09-17T07:07:43.010 INFO:teuthology.orchestra.run.smithi100.stdout: "features": 4540138322906710015, 2024-09-17T07:07:43.010 INFO:teuthology.orchestra.run.smithi100.stdout: "old_weight": 0, 2024-09-17T07:07:43.010 INFO:teuthology.orchestra.run.smithi100.stdout: "last_purged_snaps_scrub": "2024-09-17T07:07:07.725941+0000", 2024-09-17T07:07:43.010 INFO:teuthology.orchestra.run.smithi100.stdout: "dead_epoch": 0 2024-09-17T07:07:43.010 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:07:43.010 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-17T07:07:43.010 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_upmap": [], 2024-09-17T07:07:43.010 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_upmap_items": [ 2024-09-17T07:07:43.010 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:07:43.010 INFO:teuthology.orchestra.run.smithi100.stdout: "pgid": "2.1a", 2024-09-17T07:07:43.011 INFO:teuthology.orchestra.run.smithi100.stdout: "mappings": [ 2024-09-17T07:07:43.011 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:07:43.011 INFO:teuthology.orchestra.run.smithi100.stdout: "from": 7, 2024-09-17T07:07:43.011 INFO:teuthology.orchestra.run.smithi100.stdout: "to": 2 2024-09-17T07:07:43.011 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:07:43.011 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:07:43.011 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:07:43.011 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:07:43.011 INFO:teuthology.orchestra.run.smithi100.stdout: "pgid": "4.6", 2024-09-17T07:07:43.011 INFO:teuthology.orchestra.run.smithi100.stdout: "mappings": [ 2024-09-17T07:07:43.011 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:07:43.011 INFO:teuthology.orchestra.run.smithi100.stdout: "from": 1, 2024-09-17T07:07:43.011 INFO:teuthology.orchestra.run.smithi100.stdout: "to": 5 2024-09-17T07:07:43.011 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:07:43.012 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:07:43.012 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:07:43.012 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:07:43.012 INFO:teuthology.orchestra.run.smithi100.stdout: "pgid": "4.10", 2024-09-17T07:07:43.012 INFO:teuthology.orchestra.run.smithi100.stdout: "mappings": [ 2024-09-17T07:07:43.012 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:07:43.012 INFO:teuthology.orchestra.run.smithi100.stdout: "from": 1, 2024-09-17T07:07:43.012 INFO:teuthology.orchestra.run.smithi100.stdout: "to": 2 2024-09-17T07:07:43.012 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:07:43.012 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:07:43.012 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:07:43.012 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:07:43.012 INFO:teuthology.orchestra.run.smithi100.stdout: "pgid": "4.1f", 2024-09-17T07:07:43.013 INFO:teuthology.orchestra.run.smithi100.stdout: "mappings": [ 2024-09-17T07:07:43.013 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:07:43.013 INFO:teuthology.orchestra.run.smithi100.stdout: "from": 1, 2024-09-17T07:07:43.013 INFO:teuthology.orchestra.run.smithi100.stdout: "to": 2 2024-09-17T07:07:43.013 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:07:43.013 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:07:43.013 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:07:43.013 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-17T07:07:43.013 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_upmap_primaries": [], 2024-09-17T07:07:43.013 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_temp": [], 2024-09-17T07:07:43.013 INFO:teuthology.orchestra.run.smithi100.stdout: "primary_temp": [], 2024-09-17T07:07:43.013 INFO:teuthology.orchestra.run.smithi100.stdout: "blocklist": { 2024-09-17T07:07:43.013 INFO:teuthology.orchestra.run.smithi100.stdout: "172.21.15.100:0/3796664909": "2024-09-18T06:57:35.678309+0000", 2024-09-17T07:07:43.014 INFO:teuthology.orchestra.run.smithi100.stdout: "172.21.15.100:0/2225586217": "2024-09-18T07:02:46.981486+0000", 2024-09-17T07:07:43.014 INFO:teuthology.orchestra.run.smithi100.stdout: "172.21.15.100:0/1626706829": "2024-09-18T06:57:35.678309+0000", 2024-09-17T07:07:43.014 INFO:teuthology.orchestra.run.smithi100.stdout: "172.21.15.100:0/3452172206": "2024-09-18T06:57:35.678309+0000", 2024-09-17T07:07:43.014 INFO:teuthology.orchestra.run.smithi100.stdout: "172.21.15.100:6800/3992296558": "2024-09-18T06:57:58.526309+0000", 2024-09-17T07:07:43.014 INFO:teuthology.orchestra.run.smithi100.stdout: "172.21.15.100:6800/3498574214": "2024-09-18T06:57:35.678309+0000", 2024-09-17T07:07:43.014 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:07:43.015 INFO:teuthology.orchestra.run.smithi100.stdout: "172.21.15.100:0/1365215411": "2024-09-18T06:57:58.526309+0000", 2024-09-17T07:07:43.015 INFO:teuthology.orchestra.run.smithi100.stdout: "172.21.15.100:0/3362169179": "2024-09-18T07:02:46.981486+0000", 2024-09-17T07:07:43.015 INFO:teuthology.orchestra.run.smithi100.stdout: "172.21.15.100:0/2573779025": "2024-09-18T07:02:46.981486+0000", 2024-09-17T07:07:43.015 INFO:teuthology.orchestra.run.smithi100.stdout: "172.21.15.100:0/380524491": "2024-09-18T07:02:46.981486+0000", 2024-09-17T07:07:43.015 INFO:teuthology.orchestra.run.smithi100.stdout: "172.21.15.100:0/1657291976": "2024-09-18T06:57:58.526309+0000", 2024-09-17T07:07:43.015 INFO:teuthology.orchestra.run.smithi100.stdout: "172.21.15.100:0/1676314112": "2024-09-18T07:02:46.981486+0000", 2024-09-17T07:07:43.015 INFO:teuthology.orchestra.run.smithi100.stdout: "172.21.15.100:0/1255197255": "2024-09-18T06:57:58.526309+0000", 2024-09-17T07:07:43.016 INFO:teuthology.orchestra.run.smithi100.stdout: "172.21.15.100:6800/3472658181": "2024-09-18T07:02:46.981486+0000" 2024-09-17T07:07:43.016 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:07:43.016 INFO:teuthology.orchestra.run.smithi100.stdout: "range_blocklist": {}, 2024-09-17T07:07:43.016 INFO:teuthology.orchestra.run.smithi100.stdout: "erasure_code_profiles": { 2024-09-17T07:07:43.016 INFO:teuthology.orchestra.run.smithi100.stdout: "default": { 2024-09-17T07:07:43.016 INFO:teuthology.orchestra.run.smithi100.stdout: "crush-failure-domain": "osd", 2024-09-17T07:07:43.016 INFO:teuthology.orchestra.run.smithi100.stdout: "k": "2", 2024-09-17T07:07:43.016 INFO:teuthology.orchestra.run.smithi100.stdout: "m": "1", 2024-09-17T07:07:43.016 INFO:teuthology.orchestra.run.smithi100.stdout: "plugin": "jerasure", 2024-09-17T07:07:43.016 INFO:teuthology.orchestra.run.smithi100.stdout: "technique": "reed_sol_van" 2024-09-17T07:07:43.016 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:07:43.016 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:07:43.016 INFO:teuthology.orchestra.run.smithi100.stdout: "removed_snaps_queue": [], 2024-09-17T07:07:43.017 INFO:teuthology.orchestra.run.smithi100.stdout: "new_removed_snaps": [], 2024-09-17T07:07:43.017 INFO:teuthology.orchestra.run.smithi100.stdout: "new_purged_snaps": [], 2024-09-17T07:07:43.017 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_node_flags": {}, 2024-09-17T07:07:43.017 INFO:teuthology.orchestra.run.smithi100.stdout: "device_class_flags": {}, 2024-09-17T07:07:43.017 INFO:teuthology.orchestra.run.smithi100.stdout: "stretch_mode": { 2024-09-17T07:07:43.017 INFO:teuthology.orchestra.run.smithi100.stdout: "stretch_mode_enabled": false, 2024-09-17T07:07:43.017 INFO:teuthology.orchestra.run.smithi100.stdout: "stretch_bucket_count": 0, 2024-09-17T07:07:43.017 INFO:teuthology.orchestra.run.smithi100.stdout: "degraded_stretch_mode": 0, 2024-09-17T07:07:43.017 INFO:teuthology.orchestra.run.smithi100.stdout: "recovering_stretch_mode": 0, 2024-09-17T07:07:43.017 INFO:teuthology.orchestra.run.smithi100.stdout: "stretch_mode_bucket": 0 2024-09-17T07:07:43.017 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:07:43.017 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-17T07:07:43.038 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:43.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:43.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:07:43.139 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:07:43.161 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:43.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:43.162 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:07:43.263 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:07:43.284 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:43.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:43.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:07:43.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:43 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/3162122245' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-17T07:07:43.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:43 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/3162122245' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-17T07:07:43.386 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:07:43.388 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:43 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/3162122245' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-17T07:07:43.407 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:43.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:43.408 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:07:43.509 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:07:43.530 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:43.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:43.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:07:43.613 INFO:teuthology.orchestra.run.smithi135.stdout:ERROR: (22) Invalid argument 2024-09-17T07:07:43.613 INFO:teuthology.orchestra.run.smithi135.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:07:43.631 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:07:43.652 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:43.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:43.653 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:07:43.754 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:07:43.777 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:43.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:43.778 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:07:43.878 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:07:43.900 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:43.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:43.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:07:44.001 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:07:44.022 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:44.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:44.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:07:44.124 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:07:44.145 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:44.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:44.146 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:07:44.180 INFO:teuthology.orchestra.run.smithi100.stderr:nodeep-scrub is set 2024-09-17T07:07:44.230 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-17T07:07:44.247 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:07:44.294 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:44.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:44.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:07:44.396 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:07:44.417 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:44.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:44.418 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:07:44.516 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:44 smithi100 ceph-mon[29092]: pgmap v238: 158 pgs: 158 active+clean; 12 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-17T07:07:44.516 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:44 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/4187607726' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:07:44.516 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:44 smithi100 ceph-mon[39462]: pgmap v238: 158 pgs: 158 active+clean; 12 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-17T07:07:44.516 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:44 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/4187607726' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:07:44.519 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:07:44.541 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:44.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:44.542 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:07:44.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:44 smithi135 ceph-mon[33238]: pgmap v238: 158 pgs: 158 active+clean; 12 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-17T07:07:44.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:44 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/4187607726' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:07:44.643 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:07:44.664 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:44.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:44.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:07:44.766 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:07:44.786 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:44.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:44.787 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:07:44.888 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:07:44.909 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:44.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:44.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:07:45.011 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:07:45.031 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:45.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:45.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:07:45.133 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:07:45.154 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:45.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:45.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:07:45.256 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:07:45.277 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:45.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:45.278 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:07:45.379 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:07:45.402 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:45.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:45.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:07:45.425 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:45 smithi100 ceph-mon[29092]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:07:45.425 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:45 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/4187607726' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T07:07:45.425 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:45 smithi100 ceph-mon[29092]: osdmap e166: 8 total, 8 up, 8 in 2024-09-17T07:07:45.425 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:45 smithi100 ceph-mon[39462]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:07:45.426 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:45 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/4187607726' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T07:07:45.426 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:45 smithi100 ceph-mon[39462]: osdmap e166: 8 total, 8 up, 8 in 2024-09-17T07:07:45.503 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:07:45.524 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:45.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:45.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:07:45.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:45 smithi135 ceph-mon[33238]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:07:45.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:45 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/4187607726' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T07:07:45.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:45 smithi135 ceph-mon[33238]: osdmap e166: 8 total, 8 up, 8 in 2024-09-17T07:07:45.626 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:07:45.647 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:45.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:45.648 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:07:45.742 INFO:tasks.thrashosds.thrasher:j is {'epoch': 165, 'fsid': 'e848267c-74c1-11ef-bceb-c7b262605968', 'created': '2024-09-17T06:57:05.010532+0000', 'modified': '2024-09-17T07:07:37.335513+0000', 'last_up_change': '2024-09-17T07:01:59.917118+0000', 'last_in_change': '2024-09-17T07:01:42.942454+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'reef', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-17T07:00:08.816136+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.8959827423095703, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-09-17T07:02:10.759826+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.8959827423095703, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 3, 'pool_name': 'datapool', 'create_time': '2024-09-17T07:02:10.991289+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 72, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 0.8959827423095703, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-17T07:02:12.291462+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.8959827423095703, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-17T07:02:14.253214+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '73', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 0.8959827423095703, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-17T07:02:16.310620+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8959827423095703, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 7, 'pool_name': 'unique_pool_0', 'create_time': '2024-09-17T07:04:19.222948+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 26, 'pg_placement_num': 26, 'pg_placement_num_target': 26, 'pg_num_target': 26, 'pg_num_pending': 26, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '113', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '111', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_acting': 1.850000023841858, 'score_stable': 1.850000023841858, 'optimal_score': 1, 'raw_score_acting': 1.850000023841858, 'raw_score_stable': 1.850000023841858, 'primary_affinity_weighted': 0.8959827423095703, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}], 'osds': [{'osd': 0, 'uuid': '27bee9ec-05e6-406c-8bf1-168098da0ec7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 146, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6801', 'nonce': 3341017455}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6802', 'nonce': 3341017455}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6804', 'nonce': 3341017455}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6803', 'nonce': 3341017455}]}, 'public_addr': '172.21.15.100:6801/3341017455', 'cluster_addr': '172.21.15.100:6802/3341017455', 'heartbeat_back_addr': '172.21.15.100:6804/3341017455', 'heartbeat_front_addr': '172.21.15.100:6803/3341017455', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'b1a30c9c-eeaf-478e-9b62-2d63ad45873d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 99, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6805', 'nonce': 1634750965}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6806', 'nonce': 1634750965}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6808', 'nonce': 1634750965}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6807', 'nonce': 1634750965}]}, 'public_addr': '172.21.15.100:6805/1634750965', 'cluster_addr': '172.21.15.100:6806/1634750965', 'heartbeat_back_addr': '172.21.15.100:6808/1634750965', 'heartbeat_front_addr': '172.21.15.100:6807/1634750965', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'ce542e0d-9915-465e-aa11-7b3ce13f694f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.1678619384765625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 133, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6809', 'nonce': 714579846}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6810', 'nonce': 714579846}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6812', 'nonce': 714579846}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6811', 'nonce': 714579846}]}, 'public_addr': '172.21.15.100:6809/714579846', 'cluster_addr': '172.21.15.100:6810/714579846', 'heartbeat_back_addr': '172.21.15.100:6812/714579846', 'heartbeat_front_addr': '172.21.15.100:6811/714579846', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'a1257575-3c90-41de-88f6-3960744351b1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 27, 'up_thru': 147, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6813', 'nonce': 2315956572}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6814', 'nonce': 2315956572}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6816', 'nonce': 2315956572}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6815', 'nonce': 2315956572}]}, 'public_addr': '172.21.15.100:6813/2315956572', 'cluster_addr': '172.21.15.100:6814/2315956572', 'heartbeat_back_addr': '172.21.15.100:6816/2315956572', 'heartbeat_front_addr': '172.21.15.100:6815/2315956572', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'e44e8922-2c5e-44e3-b556-d4dfbc499649', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 33, 'up_thru': 123, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.135:6800', 'nonce': 1148820922}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.135:6801', 'nonce': 1148820922}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.135:6803', 'nonce': 1148820922}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.135:6802', 'nonce': 1148820922}]}, 'public_addr': '172.21.15.135:6800/1148820922', 'cluster_addr': '172.21.15.135:6801/1148820922', 'heartbeat_back_addr': '172.21.15.135:6803/1148820922', 'heartbeat_front_addr': '172.21.15.135:6802/1148820922', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '91e4a362-fcdb-41b9-b54e-716591ba6d79', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 42, 'up_thru': 111, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.135:6804', 'nonce': 2256520893}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.135:6805', 'nonce': 2256520893}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.135:6807', 'nonce': 2256520893}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.135:6806', 'nonce': 2256520893}]}, 'public_addr': '172.21.15.135:6804/2256520893', 'cluster_addr': '172.21.15.135:6805/2256520893', 'heartbeat_back_addr': '172.21.15.135:6807/2256520893', 'heartbeat_front_addr': '172.21.15.135:6806/2256520893', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'de799f64-ebd2-4a46-91f6-c2f276bee21f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 50, 'up_thru': 139, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.135:6808', 'nonce': 2750184152}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.135:6809', 'nonce': 2750184152}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.135:6811', 'nonce': 2750184152}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.135:6810', 'nonce': 2750184152}]}, 'public_addr': '172.21.15.135:6808/2750184152', 'cluster_addr': '172.21.15.135:6809/2750184152', 'heartbeat_back_addr': '172.21.15.135:6811/2750184152', 'heartbeat_front_addr': '172.21.15.135:6810/2750184152', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '830932c7-5de1-44ff-a840-e53a6e859bda', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 60, 'up_thru': 145, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.135:6812', 'nonce': 160304773}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.135:6813', 'nonce': 160304773}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.135:6815', 'nonce': 160304773}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.135:6814', 'nonce': 160304773}]}, 'public_addr': '172.21.15.135:6812/160304773', 'cluster_addr': '172.21.15.135:6813/160304773', 'heartbeat_back_addr': '172.21.15.135:6815/160304773', 'heartbeat_front_addr': '172.21.15.135:6814/160304773', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T07:07:24.189700+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T07:06:58.606168+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T07:07:32.041868+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T07:07:09.366163+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T07:07:10.689524+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T07:07:10.504050+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T07:07:36.868729+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T07:07:07.725941+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '2.1a', 'mappings': [{'from': 7, 'to': 2}]}, {'pgid': '4.6', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.10', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1f', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.100:0/3796664909': '2024-09-18T06:57:35.678309+0000', '172.21.15.100:0/2225586217': '2024-09-18T07:02:46.981486+0000', '172.21.15.100:0/1626706829': '2024-09-18T06:57:35.678309+0000', '172.21.15.100:0/3452172206': '2024-09-18T06:57:35.678309+0000', '172.21.15.100:6800/3992296558': '2024-09-18T06:57:58.526309+0000', '172.21.15.100:6800/3498574214': '2024-09-18T06:57:35.678309+0000', '172.21.15.100:0/1365215411': '2024-09-18T06:57:58.526309+0000', '172.21.15.100:0/3362169179': '2024-09-18T07:02:46.981486+0000', '172.21.15.100:0/2573779025': '2024-09-18T07:02:46.981486+0000', '172.21.15.100:0/380524491': '2024-09-18T07:02:46.981486+0000', '172.21.15.100:0/1657291976': '2024-09-18T06:57:58.526309+0000', '172.21.15.100:0/1676314112': '2024-09-18T07:02:46.981486+0000', '172.21.15.100:0/1255197255': '2024-09-18T06:57:58.526309+0000', '172.21.15.100:6800/3472658181': '2024-09-18T07:02:46.981486+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-17T07:07:45.743 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T07:07:45.748 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:07:45.770 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:45.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:45.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:07:45.872 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:07:45.893 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:45.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:45.894 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:07:45.994 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:07:46.015 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:46.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:46.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:07:46.117 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:07:46.126 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:07:45 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:07:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:07:46.138 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:46.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:46.139 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:07:46.240 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:07:46.260 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:46.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:46.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:07:46.362 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:07:46.386 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:46.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:46.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:07:46.402 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:46 smithi100 ceph-mon[29092]: pgmap v240: 158 pgs: 158 active+clean; 12 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:07:46.402 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:46 smithi100 ceph-mon[39462]: pgmap v240: 158 pgs: 158 active+clean; 12 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:07:46.454 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:07:46.457 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:07:46.467 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:07:46.487 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:07:46.510 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:46.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:46.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:07:46.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:46 smithi135 ceph-mon[33238]: pgmap v240: 158 pgs: 158 active+clean; 12 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:07:46.611 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:07:46.632 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:46.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:46.633 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:07:46.734 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:07:46.759 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:46.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:46.760 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:07:46.861 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:07:46.884 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:46.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:46.885 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:07:46.986 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:07:47.007 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:47.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:47.008 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:07:47.108 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:07:47.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:47.131 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:47.131 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:07:47.182 INFO:teuthology.orchestra.run.smithi100.stderr:pool 'unique_pool_0' removed 2024-09-17T07:07:47.232 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:07:47.254 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:47.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:47.255 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:07:47.355 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:07:47.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:47.378 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:47.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:07:47.479 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:07:47.501 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:47.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:47.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:07:47.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:47 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/60813197' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-17T07:07:47.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:47 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-17T07:07:47.550 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:47 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/60813197' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-17T07:07:47.550 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:47 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-17T07:07:47.554 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:47 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/60813197' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-17T07:07:47.554 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:47 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-17T07:07:47.605 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:07:47.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:47.640 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:47.641 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:07:47.741 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:07:47.762 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:47.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:47.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:07:47.864 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:07:47.886 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:47.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:47.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:07:47.987 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:07:48.008 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:48.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:48.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:07:48.110 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:07:48.131 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:48.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:48.132 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:07:48.161 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T07:07:48.233 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:07:48.254 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:48.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:48.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:07:48.356 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:07:48.378 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:48.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:48.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:07:48.480 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:07:48.502 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:48.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:48.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:07:48.543 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:07:48 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:07:48.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:48 smithi135 ceph-mon[33238]: pgmap v241: 158 pgs: 158 active+clean; 12 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-17T07:07:48.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:48 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-09-17T07:07:48.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:48 smithi135 ceph-mon[33238]: osdmap e167: 8 total, 8 up, 8 in 2024-09-17T07:07:48.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:48 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:07:48.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:48 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:07:48.604 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:07:48.626 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:48.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:48.627 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:07:48.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:48 smithi100 ceph-mon[29092]: pgmap v241: 158 pgs: 158 active+clean; 12 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-17T07:07:48.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:48 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-09-17T07:07:48.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:48 smithi100 ceph-mon[29092]: osdmap e167: 8 total, 8 up, 8 in 2024-09-17T07:07:48.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:48 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:07:48.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:48 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:07:48.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:48 smithi100 ceph-mon[39462]: pgmap v241: 158 pgs: 158 active+clean; 12 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-17T07:07:48.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:48 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-09-17T07:07:48.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:48 smithi100 ceph-mon[39462]: osdmap e167: 8 total, 8 up, 8 in 2024-09-17T07:07:48.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:48 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:07:48.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:48 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:07:48.713 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-17T07:07:48.714 INFO:tasks.radosbench:Beginning radosbench... 2024-09-17T07:07:48.715 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_1 2024-09-17T07:07:48.715 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd pool create unique_pool_1 16 2024-09-17T07:07:48.728 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:07:48.748 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:48.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:48.749 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:07:48.850 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:07:48.871 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:48.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:48.872 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:07:48.973 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:07:48.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:48.995 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:48.995 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:07:49.096 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:07:49.118 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:49.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:49.118 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:07:49.219 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:07:49.240 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:49.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:49.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:07:49.342 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:07:49.363 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:49.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:49.364 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:07:49.465 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:07:49.487 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:49.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:49.488 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:07:49.589 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:07:49.611 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:49.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:49.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:07:49.713 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:07:49.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:49.736 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:49.736 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:07:49.837 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:07:49.860 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:49.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:49.862 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:07:49.963 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:07:49.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:49.986 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:49.986 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:07:50.087 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:07:50.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:50.110 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:50.110 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:07:50.211 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:07:50.232 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:50.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:50.233 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:07:50.334 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:07:50.355 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:50.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:50.356 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:07:50.457 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:07:50.478 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:50.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:50.479 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:07:50.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:50 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:07:50.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:50 smithi135 ceph-mon[33238]: pgmap v243: 132 pgs: 132 active+clean; 582 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T07:07:50.548 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:50 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:07:50.548 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:50 smithi100 ceph-mon[29092]: pgmap v243: 132 pgs: 132 active+clean; 582 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T07:07:50.549 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:50 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:07:50.549 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:50 smithi100 ceph-mon[39462]: pgmap v243: 132 pgs: 132 active+clean; 582 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T07:07:50.580 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:07:50.602 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:50.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:50.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:07:50.704 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:07:50.729 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:50.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:50.730 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:07:50.831 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:07:50.852 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:50.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:50.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:07:50.876 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:07:50.876 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:07:50.892 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1: {} 2024-09-17T07:07:50.892 INFO:teuthology.orchestra.run.smithi100.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:07:50.907 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:07:50.907 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:07:50.923 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:07:50.923 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:07:50.940 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:07:50.940 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:07:50.954 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:07:50.957 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:07:50.957 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:07:50.973 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:07:50.973 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:07:50.974 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:50.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:50.975 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:07:50.991 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:07:50.991 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:07:51.077 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:07:51.086 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-17T07:07:51.086 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-17T07:07:51.086 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-17T07:07:51.086 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-17T07:07:51.098 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:51.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:51.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:07:51.200 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:07:51.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:51.226 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:51.226 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:07:51.327 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:07:51.348 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:51.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:51.351 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:07:51.452 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:07:51.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:51.476 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:51.476 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:07:51.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:51 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2463236348' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:07:51.577 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:07:51.602 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:51.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:51.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:07:51.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:51 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2463236348' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:07:51.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:51 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2463236348' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:07:51.704 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:07:51.729 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:51.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:51.730 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:07:51.831 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:07:51.853 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:51.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:51.854 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:07:51.954 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:07:51.976 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:51.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:51.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:07:52.078 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:07:52.081 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:07:52.104 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:52.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:52.105 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:07:52.206 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:07:52.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:52.228 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:52.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:07:52.330 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:07:52.351 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:52.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:52.352 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:07:52.453 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:07:52.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:52.475 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:52.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:07:52.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:52 smithi135 ceph-mon[33238]: pgmap v244: 132 pgs: 132 active+clean; 582 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T07:07:52.576 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:07:52.602 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:52.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:52.603 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:07:52.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:52 smithi100 ceph-mon[29092]: pgmap v244: 132 pgs: 132 active+clean; 582 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T07:07:52.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:52 smithi100 ceph-mon[39462]: pgmap v244: 132 pgs: 132 active+clean; 582 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T07:07:52.704 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:07:52.726 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:52.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:52.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:07:52.828 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:07:52.854 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:52.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:52.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:07:52.956 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:07:52.978 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:52.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:52.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:07:53.079 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:07:53.105 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:53.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:53.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:07:53.207 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:07:53.229 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:53.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:53.230 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:07:53.331 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:07:53.353 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:53.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:53.354 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:07:53.455 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:07:53.477 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:53.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:53.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:07:53.532 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-17T07:07:53.579 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:07:53.601 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:53.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:53.602 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:07:53.703 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:07:53.723 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:53.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:53.725 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:07:53.825 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:07:53.847 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:53.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:53.848 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:07:53.949 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:07:53.971 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:53.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:53.972 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:07:54.072 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:07:54.094 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:54.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:54.095 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:07:54.196 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:07:54.218 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:54.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:54.219 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:07:54.243 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T07:07:54.245 INFO:teuthology.orchestra.run.smithi100.stderr:dumped all 2024-09-17T07:07:54.247 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:07:54.320 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:07:54.341 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:54.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:54.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:07:54.443 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:07:54.464 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:54.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:54.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:07:54.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:54 smithi135 ceph-mon[33238]: pgmap v245: 132 pgs: 132 active+clean; 582 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 458 B/s rd, 0 op/s 2024-09-17T07:07:54.565 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:07:54.569 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:54 smithi100 ceph-mon[29092]: pgmap v245: 132 pgs: 132 active+clean; 582 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 458 B/s rd, 0 op/s 2024-09-17T07:07:54.569 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:54 smithi100 ceph-mon[39462]: pgmap v245: 132 pgs: 132 active+clean; 582 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 458 B/s rd, 0 op/s 2024-09-17T07:07:54.588 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:54.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:54.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:07:54.674 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:07:54.689 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:07:54.738 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:54.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:54.739 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:07:54.840 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:07:54.861 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:54.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:54.862 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:07:54.963 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:07:54.984 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:54.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:54.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:07:55.086 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:07:55.114 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:55.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:55.115 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:07:55.216 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:07:55.237 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:55.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:55.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:07:55.313 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:55 smithi100 ceph-mon[29092]: from='client.14991 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:07:55.313 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:55 smithi100 ceph-mon[39462]: from='client.14991 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:07:55.339 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:07:55.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:55.663 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:55.664 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:07:55.664 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:55 smithi135 ceph-mon[33238]: from='client.14991 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:07:55.764 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:07:55.777 INFO:teuthology.orchestra.run.smithi100.stdout:{"pg_ready":true,"pg_map":{"version":245,"stamp":"2024-09-17T07:07:53.113229+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596246,"num_objects":196,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":1250,"num_read_kb":1096,"num_write":609,"num_write_kb":2459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1771214,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":632,"ondisk_log_size":632,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":411,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":1401152,"kb_used_data":40660,"kb_used_omap":12,"kb_used_meta":1360371,"kb_avail":748560064,"statfs":{"total":767960285184,"available":766525505536,"internally_reserved":0,"allocated":41635840,"data_stored":38598781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":1393020472},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":73,"apply_latency_ms":73,"commit_latency_ns":73000000,"apply_latency_ns":73000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-12124160,"num_objects":-186,"num_object_clones":0,"num_object_copies":-558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-186,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":-156983,"num_write_kb":-4433601,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-12381,"num_bytes_recovered":-811401216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.935483"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":126,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:45.732075+0000","last_change":"2024-09-17T07:07:33.644835+0000","last_active":"2024-09-17T07:07:45.732075+0000","last_peered":"2024-09-17T07:07:45.732075+0000","last_clean":"2024-09-17T07:07:45.732075+0000","last_became_active":"2024-09-17T07:02:18.365773+0000","last_became_peered":"2024-09-17T07:02:18.365773+0000","last_unstale":"2024-09-17T07:07:45.732075+0000","last_undegraded":"2024-09-17T07:07:45.732075+0000","last_fullsized":"2024-09-17T07:07:45.732075+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:33.644728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:33.644728+0000","last_clean_scrub_stamp":"2024-09-17T07:07:33.644728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:46.708857+0000","scrub_duration":0.0070156680000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":133,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:46.115472+0000","last_change":"2024-09-17T07:04:56.479564+0000","last_active":"2024-09-17T07:07:46.115472+0000","last_peered":"2024-09-17T07:07:46.115472+0000","last_clean":"2024-09-17T07:07:46.115472+0000","last_became_active":"2024-09-17T07:02:12.006754+0000","last_became_peered":"2024-09-17T07:02:12.006754+0000","last_unstale":"2024-09-17T07:07:46.115472+0000","last_undegraded":"2024-09-17T07:07:46.115472+0000","last_fullsized":"2024-09-17T07:07:46.115472+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:56.479415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:04:56.479415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0038250369999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"75'15","reported_seq":155,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:45.732651+0000","last_change":"2024-09-17T07:06:29.008173+0000","last_active":"2024-09-17T07:07:45.732651+0000","last_peered":"2024-09-17T07:07:45.732651+0000","last_clean":"2024-09-17T07:07:45.732651+0000","last_became_active":"2024-09-17T07:02:14.012962+0000","last_became_peered":"2024-09-17T07:02:14.012962+0000","last_unstale":"2024-09-17T07:07:45.732651+0000","last_undegraded":"2024-09-17T07:07:45.732651+0000","last_fullsized":"2024-09-17T07:07:45.732651+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-09-17T07:06:29.008025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:06:29.008025+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011622218,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":118,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:44.180439+0000","last_change":"2024-09-17T07:04:39.726884+0000","last_active":"2024-09-17T07:07:44.180439+0000","last_peered":"2024-09-17T07:07:44.180439+0000","last_clean":"2024-09-17T07:07:44.180439+0000","last_became_active":"2024-09-17T07:02:16.014928+0000","last_became_peered":"2024-09-17T07:02:16.014928+0000","last_unstale":"2024-09-17T07:07:44.180439+0000","last_undegraded":"2024-09-17T07:07:44.180439+0000","last_fullsized":"2024-09-17T07:07:44.180439+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:39.726764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:04:39.726764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013379520000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":132,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:45.732620+0000","last_change":"2024-09-17T07:07:15.767618+0000","last_active":"2024-09-17T07:07:45.732620+0000","last_peered":"2024-09-17T07:07:45.732620+0000","last_clean":"2024-09-17T07:07:45.732620+0000","last_became_active":"2024-09-17T07:02:12.010069+0000","last_became_peered":"2024-09-17T07:02:12.010069+0000","last_unstale":"2024-09-17T07:07:45.732620+0000","last_undegraded":"2024-09-17T07:07:45.732620+0000","last_fullsized":"2024-09-17T07:07:45.732620+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:15.767495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:07:15.767495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:27.964043+0000","scrub_duration":0.013487149,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"75'9","reported_seq":147,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:44.180523+0000","last_change":"2024-09-17T07:04:46.742364+0000","last_active":"2024-09-17T07:07:44.180523+0000","last_peered":"2024-09-17T07:07:44.180523+0000","last_clean":"2024-09-17T07:07:44.180523+0000","last_became_active":"2024-09-17T07:02:14.016244+0000","last_became_peered":"2024-09-17T07:02:14.016244+0000","last_unstale":"2024-09-17T07:07:44.180523+0000","last_undegraded":"2024-09-17T07:07:44.180523+0000","last_fullsized":"2024-09-17T07:07:44.180523+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:04:46.742196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:04:46.742196+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0030184949999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":129,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:44.179745+0000","last_change":"2024-09-17T07:07:23.591188+0000","last_active":"2024-09-17T07:07:44.179745+0000","last_peered":"2024-09-17T07:07:44.179745+0000","last_clean":"2024-09-17T07:07:44.179745+0000","last_became_active":"2024-09-17T07:02:16.014380+0000","last_became_peered":"2024-09-17T07:02:16.014380+0000","last_unstale":"2024-09-17T07:07:44.179745+0000","last_undegraded":"2024-09-17T07:07:44.179745+0000","last_fullsized":"2024-09-17T07:07:44.179745+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:23.590927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:07:23.590927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:50.614784+0000","scrub_duration":0.0024742240000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":128,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:44.180470+0000","last_change":"2024-09-17T07:07:21.791530+0000","last_active":"2024-09-17T07:07:44.180470+0000","last_peered":"2024-09-17T07:07:44.180470+0000","last_clean":"2024-09-17T07:07:44.180470+0000","last_became_active":"2024-09-17T07:02:18.020790+0000","last_became_peered":"2024-09-17T07:02:18.020790+0000","last_unstale":"2024-09-17T07:07:44.180470+0000","last_undegraded":"2024-09-17T07:07:44.180470+0000","last_fullsized":"2024-09-17T07:07:44.180470+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:21.791399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:07:21.791399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:25.370399+0000","scrub_duration":0.0069986889999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":133,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:46.034317+0000","last_change":"2024-09-17T07:07:23.013609+0000","last_active":"2024-09-17T07:07:46.034317+0000","last_peered":"2024-09-17T07:07:46.034317+0000","last_clean":"2024-09-17T07:07:46.034317+0000","last_became_active":"2024-09-17T07:02:12.011129+0000","last_became_peered":"2024-09-17T07:02:12.011129+0000","last_unstale":"2024-09-17T07:07:46.034317+0000","last_undegraded":"2024-09-17T07:07:46.034317+0000","last_fullsized":"2024-09-17T07:07:46.034317+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:23.013470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:43.060102+0000","last_clean_scrub_stamp":"2024-09-17T07:07:23.013470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:28.079025+0000","scrub_duration":0.006146281,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"75'5","reported_seq":142,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:44.180762+0000","last_change":"2024-09-17T07:07:15.848170+0000","last_active":"2024-09-17T07:07:44.180762+0000","last_peered":"2024-09-17T07:07:44.180762+0000","last_clean":"2024-09-17T07:07:44.180762+0000","last_became_active":"2024-09-17T07:02:14.016640+0000","last_became_peered":"2024-09-17T07:02:14.016640+0000","last_unstale":"2024-09-17T07:07:44.180762+0000","last_undegraded":"2024-09-17T07:07:44.180762+0000","last_fullsized":"2024-09-17T07:07:44.180762+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-09-17T07:07:15.848071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:07:15.848071+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:31.939834+0000","scrub_duration":0.010596959,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":129,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:46.033715+0000","last_change":"2024-09-17T07:04:55.853224+0000","last_active":"2024-09-17T07:07:46.033715+0000","last_peered":"2024-09-17T07:07:46.033715+0000","last_clean":"2024-09-17T07:07:46.033715+0000","last_became_active":"2024-09-17T07:02:16.015401+0000","last_became_peered":"2024-09-17T07:02:16.015401+0000","last_unstale":"2024-09-17T07:07:46.033715+0000","last_undegraded":"2024-09-17T07:07:46.033715+0000","last_fullsized":"2024-09-17T07:07:46.033715+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:55.853108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:04:55.853108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017379610000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":137,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:46.114369+0000","last_change":"2024-09-17T07:07:23.739288+0000","last_active":"2024-09-17T07:07:46.114369+0000","last_peered":"2024-09-17T07:07:46.114369+0000","last_clean":"2024-09-17T07:07:46.114369+0000","last_became_active":"2024-09-17T07:02:18.023680+0000","last_became_peered":"2024-09-17T07:02:18.023680+0000","last_unstale":"2024-09-17T07:07:46.114369+0000","last_undegraded":"2024-09-17T07:07:46.114369+0000","last_fullsized":"2024-09-17T07:07:46.114369+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:23.739157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:07:23.739157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:39.559278+0000","scrub_duration":0.004894603,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":133,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:46.034262+0000","last_change":"2024-09-17T07:07:21.000757+0000","last_active":"2024-09-17T07:07:46.034262+0000","last_peered":"2024-09-17T07:07:46.034262+0000","last_clean":"2024-09-17T07:07:46.034262+0000","last_became_active":"2024-09-17T07:02:12.012234+0000","last_became_peered":"2024-09-17T07:02:12.012234+0000","last_unstale":"2024-09-17T07:07:46.034262+0000","last_undegraded":"2024-09-17T07:07:46.034262+0000","last_fullsized":"2024-09-17T07:07:46.034262+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:21.000635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:45.029247+0000","last_clean_scrub_stamp":"2024-09-17T07:07:21.000635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:23.254719+0000","scrub_duration":0.0068373009999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"75'9","reported_seq":136,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:44.180725+0000","last_change":"2024-09-17T07:04:35.582659+0000","last_active":"2024-09-17T07:07:44.180725+0000","last_peered":"2024-09-17T07:07:44.180725+0000","last_clean":"2024-09-17T07:07:44.180725+0000","last_became_active":"2024-09-17T07:02:14.013337+0000","last_became_peered":"2024-09-17T07:02:14.013337+0000","last_unstale":"2024-09-17T07:07:44.180725+0000","last_undegraded":"2024-09-17T07:07:44.180725+0000","last_fullsized":"2024-09-17T07:07:44.180725+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:04:35.582524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:04:35.582524+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0042430649999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":128,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:46.115536+0000","last_change":"2024-09-17T07:06:34.739225+0000","last_active":"2024-09-17T07:07:46.115536+0000","last_peered":"2024-09-17T07:07:46.115536+0000","last_clean":"2024-09-17T07:07:46.115536+0000","last_became_active":"2024-09-17T07:02:16.023774+0000","last_became_peered":"2024-09-17T07:02:16.023774+0000","last_unstale":"2024-09-17T07:07:46.115536+0000","last_undegraded":"2024-09-17T07:07:46.115536+0000","last_fullsized":"2024-09-17T07:07:46.115536+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:34.739103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:06:34.739103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:47.725471+0000","scrub_duration":0.0024801699999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":127,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:46.114349+0000","last_change":"2024-09-17T07:07:34.205329+0000","last_active":"2024-09-17T07:07:46.114349+0000","last_peered":"2024-09-17T07:07:46.114349+0000","last_clean":"2024-09-17T07:07:46.114349+0000","last_became_active":"2024-09-17T07:02:18.029178+0000","last_became_peered":"2024-09-17T07:02:18.029178+0000","last_unstale":"2024-09-17T07:07:46.114349+0000","last_undegraded":"2024-09-17T07:07:46.114349+0000","last_fullsized":"2024-09-17T07:07:46.114349+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:34.205205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:07:34.205205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:03.288846+0000","scrub_duration":0.0057819899999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"95'25","reported_seq":167,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:44.181111+0000","last_change":"2024-09-17T07:06:41.707070+0000","last_active":"2024-09-17T07:07:44.181111+0000","last_peered":"2024-09-17T07:07:44.181111+0000","last_clean":"2024-09-17T07:07:44.181111+0000","last_became_active":"2024-09-17T07:02:14.016270+0000","last_became_peered":"2024-09-17T07:02:14.016270+0000","last_unstale":"2024-09-17T07:07:44.181111+0000","last_undegraded":"2024-09-17T07:07:44.181111+0000","last_fullsized":"2024-09-17T07:07:44.181111+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'25","last_scrub_stamp":"2024-09-17T07:06:41.706948+0000","last_deep_scrub":"95'25","last_deep_scrub_stamp":"2024-09-17T07:04:09.450039+0000","last_clean_scrub_stamp":"2024-09-17T07:06:41.706948+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:03.400798+0000","scrub_duration":0.016404887999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":133,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:46.034130+0000","last_change":"2024-09-17T07:04:42.771521+0000","last_active":"2024-09-17T07:07:46.034130+0000","last_peered":"2024-09-17T07:07:46.034130+0000","last_clean":"2024-09-17T07:07:46.034130+0000","last_became_active":"2024-09-17T07:02:12.011258+0000","last_became_peered":"2024-09-17T07:02:12.011258+0000","last_unstale":"2024-09-17T07:07:46.034130+0000","last_undegraded":"2024-09-17T07:07:46.034130+0000","last_fullsized":"2024-09-17T07:07:46.034130+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:42.771404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:04:42.771404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017072459999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":129,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:44.179919+0000","last_change":"2024-09-17T07:07:24.620899+0000","last_active":"2024-09-17T07:07:44.179919+0000","last_peered":"2024-09-17T07:07:44.179919+0000","last_clean":"2024-09-17T07:07:44.179919+0000","last_became_active":"2024-09-17T07:02:16.018147+0000","last_became_peered":"2024-09-17T07:02:16.018147+0000","last_unstale":"2024-09-17T07:07:44.179919+0000","last_undegraded":"2024-09-17T07:07:44.179919+0000","last_fullsized":"2024-09-17T07:07:44.179919+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:24.620758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:07:24.620758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:54.505017+0000","scrub_duration":0.0050498469999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":94,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:45.732642+0000","last_change":"2024-09-17T07:07:16.737235+0000","last_active":"2024-09-17T07:07:45.732642+0000","last_peered":"2024-09-17T07:07:45.732642+0000","last_clean":"2024-09-17T07:07:45.732642+0000","last_became_active":"2024-09-17T07:04:15.113196+0000","last_became_peered":"2024-09-17T07:04:15.113196+0000","last_unstale":"2024-09-17T07:07:45.732642+0000","last_undegraded":"2024-09-17T07:07:45.732642+0000","last_fullsized":"2024-09-17T07:07:45.732642+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:16.737086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:42.984088+0000","last_clean_scrub_stamp":"2024-09-17T07:07:16.737086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:19.824197+0000","scrub_duration":0.0083716439999999993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"75'10","reported_seq":146,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:44.180937+0000","last_change":"2024-09-17T07:04:43.722476+0000","last_active":"2024-09-17T07:07:44.180937+0000","last_peered":"2024-09-17T07:07:44.180937+0000","last_clean":"2024-09-17T07:07:44.180937+0000","last_became_active":"2024-09-17T07:02:14.007702+0000","last_became_peered":"2024-09-17T07:02:14.007702+0000","last_unstale":"2024-09-17T07:07:44.180937+0000","last_undegraded":"2024-09-17T07:07:44.180937+0000","last_fullsized":"2024-09-17T07:07:44.180937+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-09-17T07:04:43.722288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:04:43.722288+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002985845,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":132,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:44.179606+0000","last_change":"2024-09-17T07:06:34.075343+0000","last_active":"2024-09-17T07:07:44.179606+0000","last_peered":"2024-09-17T07:07:44.179606+0000","last_clean":"2024-09-17T07:07:44.179606+0000","last_became_active":"2024-09-17T07:02:12.007553+0000","last_became_peered":"2024-09-17T07:02:12.007553+0000","last_unstale":"2024-09-17T07:07:44.179606+0000","last_undegraded":"2024-09-17T07:07:44.179606+0000","last_fullsized":"2024-09-17T07:07:44.179606+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:34.075214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:06:34.075214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:51.954252+0000","scrub_duration":0.031637767999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"98'9","reported_seq":156,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:46.034137+0000","last_change":"2024-09-17T07:06:34.881845+0000","last_active":"2024-09-17T07:07:46.034137+0000","last_peered":"2024-09-17T07:07:46.034137+0000","last_clean":"2024-09-17T07:07:46.034137+0000","last_became_active":"2024-09-17T07:04:15.110797+0000","last_became_peered":"2024-09-17T07:04:15.110797+0000","last_unstale":"2024-09-17T07:07:46.034137+0000","last_undegraded":"2024-09-17T07:07:46.034137+0000","last_fullsized":"2024-09-17T07:07:46.034137+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-09-17T07:06:34.881734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:06:34.881734+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:54.291547+0000","scrub_duration":0.027541755000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":127,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:44.181112+0000","last_change":"2024-09-17T07:06:42.703538+0000","last_active":"2024-09-17T07:07:44.181112+0000","last_peered":"2024-09-17T07:07:44.181112+0000","last_clean":"2024-09-17T07:07:44.181112+0000","last_became_active":"2024-09-17T07:02:18.017460+0000","last_became_peered":"2024-09-17T07:02:18.017460+0000","last_unstale":"2024-09-17T07:07:44.181112+0000","last_undegraded":"2024-09-17T07:07:44.181112+0000","last_fullsized":"2024-09-17T07:07:44.181112+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:42.703326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:06:42.703326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:48.219915+0000","scrub_duration":0.004198528,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"75'15","reported_seq":155,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:44.181002+0000","last_change":"2024-09-17T07:04:40.954912+0000","last_active":"2024-09-17T07:07:44.181002+0000","last_peered":"2024-09-17T07:07:44.181002+0000","last_clean":"2024-09-17T07:07:44.181002+0000","last_became_active":"2024-09-17T07:02:14.016397+0000","last_became_peered":"2024-09-17T07:02:14.016397+0000","last_unstale":"2024-09-17T07:07:44.181002+0000","last_undegraded":"2024-09-17T07:07:44.181002+0000","last_fullsized":"2024-09-17T07:07:44.181002+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-09-17T07:04:40.954768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:04:40.954768+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0050135199999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":133,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:44.179826+0000","last_change":"2024-09-17T07:07:18.550881+0000","last_active":"2024-09-17T07:07:44.179826+0000","last_peered":"2024-09-17T07:07:44.179826+0000","last_clean":"2024-09-17T07:07:44.179826+0000","last_became_active":"2024-09-17T07:02:12.007913+0000","last_became_peered":"2024-09-17T07:02:12.007913+0000","last_unstale":"2024-09-17T07:07:44.179826+0000","last_undegraded":"2024-09-17T07:07:44.179826+0000","last_fullsized":"2024-09-17T07:07:44.179826+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:18.550750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:07:18.550750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:47.637615+0000","scrub_duration":0.012153809,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"75'8","reported_seq":194,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-17T07:07:47.179404+0000","last_change":"2024-09-17T07:04:33.544461+0000","last_active":"2024-09-17T07:07:47.179404+0000","last_peered":"2024-09-17T07:07:47.179404+0000","last_clean":"2024-09-17T07:07:47.179404+0000","last_became_active":"2024-09-17T07:02:16.021762+0000","last_became_peered":"2024-09-17T07:02:16.021762+0000","last_unstale":"2024-09-17T07:07:47.179404+0000","last_undegraded":"2024-09-17T07:07:47.179404+0000","last_fullsized":"2024-09-17T07:07:47.179404+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:04:33.544306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:04:33.544306+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0038501080000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":126,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:46.114406+0000","last_change":"2024-09-17T07:06:33.833965+0000","last_active":"2024-09-17T07:07:46.114406+0000","last_peered":"2024-09-17T07:07:46.114406+0000","last_clean":"2024-09-17T07:07:46.114406+0000","last_became_active":"2024-09-17T07:02:18.025332+0000","last_became_peered":"2024-09-17T07:02:18.025332+0000","last_unstale":"2024-09-17T07:07:46.114406+0000","last_undegraded":"2024-09-17T07:07:46.114406+0000","last_fullsized":"2024-09-17T07:07:46.114406+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:33.833851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:06:33.833851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.064545922000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"75'11","reported_seq":150,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:44.181177+0000","last_change":"2024-09-17T07:06:29.766307+0000","last_active":"2024-09-17T07:07:44.181177+0000","last_peered":"2024-09-17T07:07:44.181177+0000","last_clean":"2024-09-17T07:07:44.181177+0000","last_became_active":"2024-09-17T07:02:14.010227+0000","last_became_peered":"2024-09-17T07:02:14.010227+0000","last_unstale":"2024-09-17T07:07:44.181177+0000","last_undegraded":"2024-09-17T07:07:44.181177+0000","last_fullsized":"2024-09-17T07:07:44.181177+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-09-17T07:06:29.766173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:06:29.766173+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0047797760000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":133,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:46.033843+0000","last_change":"2024-09-17T07:04:44.722990+0000","last_active":"2024-09-17T07:07:46.033843+0000","last_peered":"2024-09-17T07:07:46.033843+0000","last_clean":"2024-09-17T07:07:46.033843+0000","last_became_active":"2024-09-17T07:02:12.013182+0000","last_became_peered":"2024-09-17T07:02:12.013182+0000","last_unstale":"2024-09-17T07:07:46.033843+0000","last_undegraded":"2024-09-17T07:07:46.033843+0000","last_fullsized":"2024-09-17T07:07:46.033843+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:44.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:04:44.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026545789999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":139,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:46.115561+0000","last_change":"2024-09-17T07:07:14.680573+0000","last_active":"2024-09-17T07:07:46.115561+0000","last_peered":"2024-09-17T07:07:46.115561+0000","last_clean":"2024-09-17T07:07:46.115561+0000","last_became_active":"2024-09-17T07:02:16.020224+0000","last_became_peered":"2024-09-17T07:02:16.020224+0000","last_unstale":"2024-09-17T07:07:46.115561+0000","last_undegraded":"2024-09-17T07:07:46.115561+0000","last_fullsized":"2024-09-17T07:07:46.115561+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:14.680446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:07:14.680446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:32.551262+0000","scrub_duration":0.0023342229999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":126,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:45.732241+0000","last_change":"2024-09-17T07:06:30.971537+0000","last_active":"2024-09-17T07:07:45.732241+0000","last_peered":"2024-09-17T07:07:45.732241+0000","last_clean":"2024-09-17T07:07:45.732241+0000","last_became_active":"2024-09-17T07:02:18.365000+0000","last_became_peered":"2024-09-17T07:02:18.365000+0000","last_unstale":"2024-09-17T07:07:45.732241+0000","last_undegraded":"2024-09-17T07:07:45.732241+0000","last_fullsized":"2024-09-17T07:07:45.732241+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:30.971383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:06:30.971383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0078901470000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"75'14","reported_seq":141,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:44.180530+0000","last_change":"2024-09-17T07:04:38.693158+0000","last_active":"2024-09-17T07:07:44.180530+0000","last_peered":"2024-09-17T07:07:44.180530+0000","last_clean":"2024-09-17T07:07:44.180530+0000","last_became_active":"2024-09-17T07:02:14.007810+0000","last_became_peered":"2024-09-17T07:02:14.007810+0000","last_unstale":"2024-09-17T07:07:44.180530+0000","last_undegraded":"2024-09-17T07:07:44.180530+0000","last_fullsized":"2024-09-17T07:07:44.180530+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'14","last_scrub_stamp":"2024-09-17T07:04:38.692905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:04:38.692905+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003339,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":143,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:46.115503+0000","last_change":"2024-09-17T07:06:38.303168+0000","last_active":"2024-09-17T07:07:46.115503+0000","last_peered":"2024-09-17T07:07:46.115503+0000","last_clean":"2024-09-17T07:07:46.115503+0000","last_became_active":"2024-09-17T07:02:12.011766+0000","last_became_peered":"2024-09-17T07:02:12.011766+0000","last_unstale":"2024-09-17T07:07:46.115503+0000","last_undegraded":"2024-09-17T07:07:46.115503+0000","last_fullsized":"2024-09-17T07:07:46.115503+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:38.303033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:06:38.303033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:02.394179+0000","scrub_duration":0.003556221,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"75'8","reported_seq":203,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:45.731889+0000","last_change":"2024-09-17T07:06:29.982260+0000","last_active":"2024-09-17T07:07:45.731889+0000","last_peered":"2024-09-17T07:07:45.731889+0000","last_clean":"2024-09-17T07:07:45.731889+0000","last_became_active":"2024-09-17T07:02:16.020628+0000","last_became_peered":"2024-09-17T07:02:16.020628+0000","last_unstale":"2024-09-17T07:07:45.731889+0000","last_undegraded":"2024-09-17T07:07:45.731889+0000","last_fullsized":"2024-09-17T07:07:45.731889+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:06:29.982141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:06:29.982141+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:53.993523+0000","scrub_duration":0.0052922769999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":127,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:46.033885+0000","last_change":"2024-09-17T07:04:54.840916+0000","last_active":"2024-09-17T07:07:46.033885+0000","last_peered":"2024-09-17T07:07:46.033885+0000","last_clean":"2024-09-17T07:07:46.033885+0000","last_became_active":"2024-09-17T07:02:18.364981+0000","last_became_peered":"2024-09-17T07:02:18.364981+0000","last_unstale":"2024-09-17T07:07:46.033885+0000","last_undegraded":"2024-09-17T07:07:46.033885+0000","last_fullsized":"2024-09-17T07:07:46.033885+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:54.840843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:04:54.840843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002308619,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"75'18","reported_seq":162,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:45.731994+0000","last_change":"2024-09-17T07:06:40.794767+0000","last_active":"2024-09-17T07:07:45.731994+0000","last_peered":"2024-09-17T07:07:45.731994+0000","last_clean":"2024-09-17T07:07:45.731994+0000","last_became_active":"2024-09-17T07:02:14.028667+0000","last_became_peered":"2024-09-17T07:02:14.028667+0000","last_unstale":"2024-09-17T07:07:45.731994+0000","last_undegraded":"2024-09-17T07:07:45.731994+0000","last_fullsized":"2024-09-17T07:07:45.731994+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'18","last_scrub_stamp":"2024-09-17T07:06:40.794315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:06:40.794315+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.017914407,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":132,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:44.180549+0000","last_change":"2024-09-17T07:04:45.900865+0000","last_active":"2024-09-17T07:07:44.180549+0000","last_peered":"2024-09-17T07:07:44.180549+0000","last_clean":"2024-09-17T07:07:44.180549+0000","last_became_active":"2024-09-17T07:02:12.005687+0000","last_became_peered":"2024-09-17T07:02:12.005687+0000","last_unstale":"2024-09-17T07:07:44.180549+0000","last_undegraded":"2024-09-17T07:07:44.180549+0000","last_fullsized":"2024-09-17T07:07:44.180549+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:45.900779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:04:45.900779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001545968,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":118,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:44.180454+0000","last_change":"2024-09-17T07:04:01.335116+0000","last_active":"2024-09-17T07:07:44.180454+0000","last_peered":"2024-09-17T07:07:44.180454+0000","last_clean":"2024-09-17T07:07:44.180454+0000","last_became_active":"2024-09-17T07:02:16.019747+0000","last_became_peered":"2024-09-17T07:02:16.019747+0000","last_unstale":"2024-09-17T07:07:44.180454+0000","last_undegraded":"2024-09-17T07:07:44.180454+0000","last_fullsized":"2024-09-17T07:07:44.180454+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:01.335039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:04:01.335039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001538607,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":127,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:44.180682+0000","last_change":"2024-09-17T07:06:40.684796+0000","last_active":"2024-09-17T07:07:44.180682+0000","last_peered":"2024-09-17T07:07:44.180682+0000","last_clean":"2024-09-17T07:07:44.180682+0000","last_became_active":"2024-09-17T07:02:18.024043+0000","last_became_peered":"2024-09-17T07:02:18.024043+0000","last_unstale":"2024-09-17T07:07:44.180682+0000","last_undegraded":"2024-09-17T07:07:44.180682+0000","last_fullsized":"2024-09-17T07:07:44.180682+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:40.684680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:06:40.684680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011253449,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"75'19","reported_seq":167,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:46.115292+0000","last_change":"2024-09-17T07:07:17.989796+0000","last_active":"2024-09-17T07:07:46.115292+0000","last_peered":"2024-09-17T07:07:46.115292+0000","last_clean":"2024-09-17T07:07:46.115292+0000","last_became_active":"2024-09-17T07:02:14.027389+0000","last_became_peered":"2024-09-17T07:02:14.027389+0000","last_unstale":"2024-09-17T07:07:46.115292+0000","last_undegraded":"2024-09-17T07:07:46.115292+0000","last_fullsized":"2024-09-17T07:07:46.115292+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-09-17T07:07:17.989435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:07:17.989435+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:38.663377+0000","scrub_duration":0.0043356940000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":133,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:46.034341+0000","last_change":"2024-09-17T07:06:31.799761+0000","last_active":"2024-09-17T07:07:46.034341+0000","last_peered":"2024-09-17T07:07:46.034341+0000","last_clean":"2024-09-17T07:07:46.034341+0000","last_became_active":"2024-09-17T07:02:12.011917+0000","last_became_peered":"2024-09-17T07:02:12.011917+0000","last_unstale":"2024-09-17T07:07:46.034341+0000","last_undegraded":"2024-09-17T07:07:46.034341+0000","last_fullsized":"2024-09-17T07:07:46.034341+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:31.799640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:06:31.799640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00320703,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":129,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:46.114826+0000","last_change":"2024-09-17T07:07:31.453488+0000","last_active":"2024-09-17T07:07:46.114826+0000","last_peered":"2024-09-17T07:07:46.114826+0000","last_clean":"2024-09-17T07:07:46.114826+0000","last_became_active":"2024-09-17T07:02:16.016199+0000","last_became_peered":"2024-09-17T07:02:16.016199+0000","last_unstale":"2024-09-17T07:07:46.114826+0000","last_undegraded":"2024-09-17T07:07:46.114826+0000","last_fullsized":"2024-09-17T07:07:46.114826+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:31.453355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:07:31.453355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:43.002059+0000","scrub_duration":0.0055939470000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":126,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:44.179584+0000","last_change":"2024-09-17T07:06:30.992080+0000","last_active":"2024-09-17T07:07:44.179584+0000","last_peered":"2024-09-17T07:07:44.179584+0000","last_clean":"2024-09-17T07:07:44.179584+0000","last_became_active":"2024-09-17T07:02:18.364835+0000","last_became_peered":"2024-09-17T07:02:18.364835+0000","last_unstale":"2024-09-17T07:07:44.179584+0000","last_undegraded":"2024-09-17T07:07:44.179584+0000","last_fullsized":"2024-09-17T07:07:44.179584+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:30.991972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:06:30.991972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:53.454623+0000","scrub_duration":0.0045593279999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"110'15","reported_seq":150,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:44.179548+0000","last_change":"2024-09-17T07:04:57.092268+0000","last_active":"2024-09-17T07:07:44.179548+0000","last_peered":"2024-09-17T07:07:44.179548+0000","last_clean":"2024-09-17T07:07:44.179548+0000","last_became_active":"2024-09-17T07:02:14.010637+0000","last_became_peered":"2024-09-17T07:02:14.010637+0000","last_unstale":"2024-09-17T07:07:44.179548+0000","last_undegraded":"2024-09-17T07:07:44.179548+0000","last_fullsized":"2024-09-17T07:07:44.179548+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'15","last_scrub_stamp":"2024-09-17T07:04:57.091400+0000","last_deep_scrub":"110'15","last_deep_scrub_stamp":"2024-09-17T07:04:57.091400+0000","last_clean_scrub_stamp":"2024-09-17T07:04:57.091400+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012740888000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":132,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:44.180213+0000","last_change":"2024-09-17T07:04:55.007489+0000","last_active":"2024-09-17T07:07:44.180213+0000","last_peered":"2024-09-17T07:07:44.180213+0000","last_clean":"2024-09-17T07:07:44.180213+0000","last_became_active":"2024-09-17T07:02:12.010803+0000","last_became_peered":"2024-09-17T07:02:12.010803+0000","last_unstale":"2024-09-17T07:07:44.180213+0000","last_undegraded":"2024-09-17T07:07:44.180213+0000","last_fullsized":"2024-09-17T07:07:44.180213+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:55.007317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:25.147878+0000","last_clean_scrub_stamp":"2024-09-17T07:04:55.007317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0085334680000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"89'9","reported_seq":205,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:46.115006+0000","last_change":"2024-09-17T07:04:48.452677+0000","last_active":"2024-09-17T07:07:46.115006+0000","last_peered":"2024-09-17T07:07:46.115006+0000","last_clean":"2024-09-17T07:07:46.115006+0000","last_became_active":"2024-09-17T07:02:16.021360+0000","last_became_peered":"2024-09-17T07:02:16.021360+0000","last_unstale":"2024-09-17T07:07:46.115006+0000","last_undegraded":"2024-09-17T07:07:46.115006+0000","last_fullsized":"2024-09-17T07:07:46.115006+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'9","last_scrub_stamp":"2024-09-17T07:04:48.452558+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-09-17T07:03:29.961439+0000","last_clean_scrub_stamp":"2024-09-17T07:04:48.452558+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0057192279999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"75'1","reported_seq":128,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:46.114435+0000","last_change":"2024-09-17T07:06:25.992349+0000","last_active":"2024-09-17T07:07:46.114435+0000","last_peered":"2024-09-17T07:07:46.114435+0000","last_clean":"2024-09-17T07:07:46.114435+0000","last_became_active":"2024-09-17T07:02:18.029772+0000","last_became_peered":"2024-09-17T07:02:18.029772+0000","last_unstale":"2024-09-17T07:07:46.114435+0000","last_undegraded":"2024-09-17T07:07:46.114435+0000","last_fullsized":"2024-09-17T07:07:46.114435+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-17T07:06:25.992208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:06:25.992208+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010034507,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"75'28","reported_seq":245,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:45.581842+0000","last_change":"2024-09-17T07:06:45.186860+0000","last_active":"2024-09-17T07:07:45.581842+0000","last_peered":"2024-09-17T07:07:45.581842+0000","last_clean":"2024-09-17T07:07:45.581842+0000","last_became_active":"2024-09-17T07:02:14.017377+0000","last_became_peered":"2024-09-17T07:02:14.017377+0000","last_unstale":"2024-09-17T07:07:45.581842+0000","last_undegraded":"2024-09-17T07:07:45.581842+0000","last_fullsized":"2024-09-17T07:07:45.581842+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'28","last_scrub_stamp":"2024-09-17T07:06:45.186408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:06:45.186408+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:03.935330+0000","scrub_duration":0.010603473,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":143,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:46.115215+0000","last_change":"2024-09-17T07:06:43.763211+0000","last_active":"2024-09-17T07:07:46.115215+0000","last_peered":"2024-09-17T07:07:46.115215+0000","last_clean":"2024-09-17T07:07:46.115215+0000","last_became_active":"2024-09-17T07:02:12.008802+0000","last_became_peered":"2024-09-17T07:02:12.008802+0000","last_unstale":"2024-09-17T07:07:46.115215+0000","last_undegraded":"2024-09-17T07:07:46.115215+0000","last_fullsized":"2024-09-17T07:07:46.115215+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:43.763044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:06:43.763044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:04.613175+0000","scrub_duration":0.003861303,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":129,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:46.115007+0000","last_change":"2024-09-17T07:04:39.385133+0000","last_active":"2024-09-17T07:07:46.115007+0000","last_peered":"2024-09-17T07:07:46.115007+0000","last_clean":"2024-09-17T07:07:46.115007+0000","last_became_active":"2024-09-17T07:02:16.016606+0000","last_became_peered":"2024-09-17T07:02:16.016606+0000","last_unstale":"2024-09-17T07:07:46.115007+0000","last_undegraded":"2024-09-17T07:07:46.115007+0000","last_fullsized":"2024-09-17T07:07:46.115007+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:39.384962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:04:39.384962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0055010839999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":115,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:45.732480+0000","last_change":"2024-09-17T07:04:04.564851+0000","last_active":"2024-09-17T07:07:45.732480+0000","last_peered":"2024-09-17T07:07:45.732480+0000","last_clean":"2024-09-17T07:07:45.732480+0000","last_became_active":"2024-09-17T07:02:18.024602+0000","last_became_peered":"2024-09-17T07:02:18.024602+0000","last_unstale":"2024-09-17T07:07:45.732480+0000","last_undegraded":"2024-09-17T07:07:45.732480+0000","last_fullsized":"2024-09-17T07:07:45.732480+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:04.564765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:04:04.564765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001317206,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"75'13","reported_seq":157,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:44.181055+0000","last_change":"2024-09-17T07:04:58.081870+0000","last_active":"2024-09-17T07:07:44.181055+0000","last_peered":"2024-09-17T07:07:44.181055+0000","last_clean":"2024-09-17T07:07:44.181055+0000","last_became_active":"2024-09-17T07:02:14.019838+0000","last_became_peered":"2024-09-17T07:02:14.019838+0000","last_unstale":"2024-09-17T07:07:44.181055+0000","last_undegraded":"2024-09-17T07:07:44.181055+0000","last_fullsized":"2024-09-17T07:07:44.181055+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'13","last_scrub_stamp":"2024-09-17T07:04:58.081771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:04:58.081771+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001806263,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"76'5","reported_seq":469,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-17T07:07:47.184343+0000","last_change":"2024-09-17T07:06:46.230250+0000","last_active":"2024-09-17T07:07:47.184343+0000","last_peered":"2024-09-17T07:07:47.184343+0000","last_clean":"2024-09-17T07:07:47.184343+0000","last_became_active":"2024-09-17T07:02:12.006883+0000","last_became_peered":"2024-09-17T07:02:12.006883+0000","last_unstale":"2024-09-17T07:07:47.184343+0000","last_undegraded":"2024-09-17T07:07:47.184343+0000","last_fullsized":"2024-09-17T07:07:47.184343+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'5","last_scrub_stamp":"2024-09-17T07:06:46.229804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:06:46.229804+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:06.428152+0000","scrub_duration":0.064119354000000003,"objects_trimmed":0,"snaptrim_duration":0.00033739799999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":329,"num_read_kb":324,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":93,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:45.732729+0000","last_change":"2024-09-17T07:04:53.698772+0000","last_active":"2024-09-17T07:07:45.732729+0000","last_peered":"2024-09-17T07:07:45.732729+0000","last_clean":"2024-09-17T07:07:45.732729+0000","last_became_active":"2024-09-17T07:04:15.109950+0000","last_became_peered":"2024-09-17T07:04:15.109950+0000","last_unstale":"2024-09-17T07:07:45.732729+0000","last_undegraded":"2024-09-17T07:07:45.732729+0000","last_fullsized":"2024-09-17T07:07:45.732729+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:53.698301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:04:53.698301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045574200000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":129,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:44.180646+0000","last_change":"2024-09-17T07:06:25.185133+0000","last_active":"2024-09-17T07:07:44.180646+0000","last_peered":"2024-09-17T07:07:44.180646+0000","last_clean":"2024-09-17T07:07:44.180646+0000","last_became_active":"2024-09-17T07:02:16.014492+0000","last_became_peered":"2024-09-17T07:02:16.014492+0000","last_unstale":"2024-09-17T07:07:44.180646+0000","last_undegraded":"2024-09-17T07:07:44.180646+0000","last_fullsized":"2024-09-17T07:07:44.180646+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:25.184907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:06:25.184907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0051503060000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":127,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:46.034054+0000","last_change":"2024-09-17T07:04:53.836533+0000","last_active":"2024-09-17T07:07:46.034054+0000","last_peered":"2024-09-17T07:07:46.034054+0000","last_clean":"2024-09-17T07:07:46.034054+0000","last_became_active":"2024-09-17T07:02:18.365369+0000","last_became_peered":"2024-09-17T07:02:18.365369+0000","last_unstale":"2024-09-17T07:07:46.034054+0000","last_undegraded":"2024-09-17T07:07:46.034054+0000","last_fullsized":"2024-09-17T07:07:46.034054+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:53.836372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:04:53.836372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0024095980000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"75'12","reported_seq":172,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:46.115203+0000","last_change":"2024-09-17T07:04:50.489946+0000","last_active":"2024-09-17T07:07:46.115203+0000","last_peered":"2024-09-17T07:07:46.115203+0000","last_clean":"2024-09-17T07:07:46.115203+0000","last_became_active":"2024-09-17T07:03:48.760405+0000","last_became_peered":"2024-09-17T07:03:48.760405+0000","last_unstale":"2024-09-17T07:07:46.115203+0000","last_undegraded":"2024-09-17T07:07:46.115203+0000","last_fullsized":"2024-09-17T07:07:46.115203+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-09-17T07:04:50.489723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:04:50.489723+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.007032679,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"70'1","reported_seq":139,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:46.114838+0000","last_change":"2024-09-17T07:07:21.050377+0000","last_active":"2024-09-17T07:07:46.114838+0000","last_peered":"2024-09-17T07:07:46.114838+0000","last_clean":"2024-09-17T07:07:46.114838+0000","last_became_active":"2024-09-17T07:02:12.009476+0000","last_became_peered":"2024-09-17T07:02:12.009476+0000","last_unstale":"2024-09-17T07:07:46.114838+0000","last_undegraded":"2024-09-17T07:07:46.114838+0000","last_fullsized":"2024-09-17T07:07:46.114838+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-17T07:07:21.050247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:07:21.050247+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:38.188477+0000","scrub_duration":0.005277505,"objects_trimmed":0,"snaptrim_duration":0.0032298629999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":122,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:46.033886+0000","last_change":"2024-09-17T07:03:51.994858+0000","last_active":"2024-09-17T07:07:46.033886+0000","last_peered":"2024-09-17T07:07:46.033886+0000","last_clean":"2024-09-17T07:07:46.033886+0000","last_became_active":"2024-09-17T07:02:12.014623+0000","last_became_peered":"2024-09-17T07:02:12.014623+0000","last_unstale":"2024-09-17T07:07:46.033886+0000","last_undegraded":"2024-09-17T07:07:46.033886+0000","last_fullsized":"2024-09-17T07:07:46.033886+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:03:51.994756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:03:51.994756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001933269,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":128,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:46.115623+0000","last_change":"2024-09-17T07:06:28.863501+0000","last_active":"2024-09-17T07:07:46.115623+0000","last_peered":"2024-09-17T07:07:46.115623+0000","last_clean":"2024-09-17T07:07:46.115623+0000","last_became_active":"2024-09-17T07:02:16.021561+0000","last_became_peered":"2024-09-17T07:02:16.021561+0000","last_unstale":"2024-09-17T07:07:46.115623+0000","last_undegraded":"2024-09-17T07:07:46.115623+0000","last_fullsized":"2024-09-17T07:07:46.115623+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:28.863357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:06:28.863357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.014593309,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":126,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:44.179928+0000","last_change":"2024-09-17T07:07:25.570226+0000","last_active":"2024-09-17T07:07:44.179928+0000","last_peered":"2024-09-17T07:07:44.179928+0000","last_clean":"2024-09-17T07:07:44.179928+0000","last_became_active":"2024-09-17T07:02:18.022082+0000","last_became_peered":"2024-09-17T07:02:18.022082+0000","last_unstale":"2024-09-17T07:07:44.179928+0000","last_undegraded":"2024-09-17T07:07:44.179928+0000","last_fullsized":"2024-09-17T07:07:44.179928+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:25.570107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:07:25.570107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:37.811807+0000","scrub_duration":0.0049089310000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"75'14","reported_seq":153,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:46.115265+0000","last_change":"2024-09-17T07:07:26.395820+0000","last_active":"2024-09-17T07:07:46.115265+0000","last_peered":"2024-09-17T07:07:46.115265+0000","last_clean":"2024-09-17T07:07:46.115265+0000","last_became_active":"2024-09-17T07:02:14.028857+0000","last_became_peered":"2024-09-17T07:02:14.028857+0000","last_unstale":"2024-09-17T07:07:46.115265+0000","last_undegraded":"2024-09-17T07:07:46.115265+0000","last_fullsized":"2024-09-17T07:07:46.115265+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'14","last_scrub_stamp":"2024-09-17T07:07:26.395429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:07:26.395429+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:27.868879+0000","scrub_duration":0.0028559779999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"72'2","reported_seq":139,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:45.732177+0000","last_change":"2024-09-17T07:04:35.691710+0000","last_active":"2024-09-17T07:07:45.732177+0000","last_peered":"2024-09-17T07:07:45.732177+0000","last_clean":"2024-09-17T07:07:45.732177+0000","last_became_active":"2024-09-17T07:02:12.005127+0000","last_became_peered":"2024-09-17T07:02:12.005127+0000","last_unstale":"2024-09-17T07:07:45.732177+0000","last_undegraded":"2024-09-17T07:07:45.732177+0000","last_fullsized":"2024-09-17T07:07:45.732177+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'2","last_scrub_stamp":"2024-09-17T07:04:35.690341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:04:35.690341+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0079935560000000006,"objects_trimmed":0,"snaptrim_duration":0.0010980219999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":143,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:46.115708+0000","last_change":"2024-09-17T07:06:37.732110+0000","last_active":"2024-09-17T07:07:46.115708+0000","last_peered":"2024-09-17T07:07:46.115708+0000","last_clean":"2024-09-17T07:07:46.115708+0000","last_became_active":"2024-09-17T07:02:12.004606+0000","last_became_peered":"2024-09-17T07:02:12.004606+0000","last_unstale":"2024-09-17T07:07:46.115708+0000","last_undegraded":"2024-09-17T07:07:46.115708+0000","last_fullsized":"2024-09-17T07:07:46.115708+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:37.731844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:06:37.731844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:04.459809+0000","scrub_duration":0.0070683819999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"78'128","reported_seq":381,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:46.034095+0000","last_change":"2024-09-17T07:04:37.752152+0000","last_active":"2024-09-17T07:07:46.034095+0000","last_peered":"2024-09-17T07:07:46.034095+0000","last_clean":"2024-09-17T07:07:46.034095+0000","last_became_active":"2024-09-17T07:02:03.020675+0000","last_became_peered":"2024-09-17T07:02:03.020675+0000","last_unstale":"2024-09-17T07:07:46.034095+0000","last_undegraded":"2024-09-17T07:07:46.034095+0000","last_fullsized":"2024-09-17T07:07:46.034095+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'128","last_scrub_stamp":"2024-09-17T07:04:37.751949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:00:09.390381+0000","last_clean_scrub_stamp":"2024-09-17T07:04:37.751949+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040036619999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":129,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:46.034177+0000","last_change":"2024-09-17T07:04:43.722620+0000","last_active":"2024-09-17T07:07:46.034177+0000","last_peered":"2024-09-17T07:07:46.034177+0000","last_clean":"2024-09-17T07:07:46.034177+0000","last_became_active":"2024-09-17T07:02:16.016031+0000","last_became_peered":"2024-09-17T07:02:16.016031+0000","last_unstale":"2024-09-17T07:07:46.034177+0000","last_undegraded":"2024-09-17T07:07:46.034177+0000","last_fullsized":"2024-09-17T07:07:46.034177+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:43.722510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:04:43.722510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0024960820000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":126,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:46.114498+0000","last_change":"2024-09-17T07:06:30.808658+0000","last_active":"2024-09-17T07:07:46.114498+0000","last_peered":"2024-09-17T07:07:46.114498+0000","last_clean":"2024-09-17T07:07:46.114498+0000","last_became_active":"2024-09-17T07:02:18.023321+0000","last_became_peered":"2024-09-17T07:02:18.023321+0000","last_unstale":"2024-09-17T07:07:46.114498+0000","last_undegraded":"2024-09-17T07:07:46.114498+0000","last_fullsized":"2024-09-17T07:07:46.114498+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:30.808562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:58.670571+0000","last_clean_scrub_stamp":"2024-09-17T07:06:30.808562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:55.726519+0000","scrub_duration":0.005261338,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"75'19","reported_seq":162,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:46.115033+0000","last_change":"2024-09-17T07:07:18.384022+0000","last_active":"2024-09-17T07:07:46.115033+0000","last_peered":"2024-09-17T07:07:46.115033+0000","last_clean":"2024-09-17T07:07:46.115033+0000","last_became_active":"2024-09-17T07:02:14.027313+0000","last_became_peered":"2024-09-17T07:02:14.027313+0000","last_unstale":"2024-09-17T07:07:46.115033+0000","last_undegraded":"2024-09-17T07:07:46.115033+0000","last_fullsized":"2024-09-17T07:07:46.115033+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-09-17T07:07:18.383813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:07:18.383813+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:42.977846+0000","scrub_duration":0.0040351470000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":144,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:44.179785+0000","last_change":"2024-09-17T07:06:39.322648+0000","last_active":"2024-09-17T07:07:44.179785+0000","last_peered":"2024-09-17T07:07:44.179785+0000","last_clean":"2024-09-17T07:07:44.179785+0000","last_became_active":"2024-09-17T07:02:12.012037+0000","last_became_peered":"2024-09-17T07:02:12.012037+0000","last_unstale":"2024-09-17T07:07:44.179785+0000","last_undegraded":"2024-09-17T07:07:44.179785+0000","last_fullsized":"2024-09-17T07:07:44.179785+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:39.322535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:06:39.322535+0000","last_clean_scrub_stamp":"2024-09-17T07:06:39.322535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002360993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":140,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:44.179392+0000","last_change":"2024-09-17T07:06:28.171453+0000","last_active":"2024-09-17T07:07:44.179392+0000","last_peered":"2024-09-17T07:07:44.179392+0000","last_clean":"2024-09-17T07:07:44.179392+0000","last_became_active":"2024-09-17T07:02:16.014631+0000","last_became_peered":"2024-09-17T07:02:16.014631+0000","last_unstale":"2024-09-17T07:07:44.179392+0000","last_undegraded":"2024-09-17T07:07:44.179392+0000","last_fullsized":"2024-09-17T07:07:44.179392+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:28.170477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:06:28.170477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:50.964072+0000","scrub_duration":0.0059381709999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":127,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:46.033655+0000","last_change":"2024-09-17T07:07:27.004939+0000","last_active":"2024-09-17T07:07:46.033655+0000","last_peered":"2024-09-17T07:07:46.033655+0000","last_clean":"2024-09-17T07:07:46.033655+0000","last_became_active":"2024-09-17T07:02:18.023299+0000","last_became_peered":"2024-09-17T07:02:18.023299+0000","last_unstale":"2024-09-17T07:07:46.033655+0000","last_undegraded":"2024-09-17T07:07:46.033655+0000","last_fullsized":"2024-09-17T07:07:46.033655+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:27.004683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:07:27.004683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:37.142184+0000","scrub_duration":0.010022120000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"75'9","reported_seq":147,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:46.115090+0000","last_change":"2024-09-17T07:06:26.996048+0000","last_active":"2024-09-17T07:07:46.115090+0000","last_peered":"2024-09-17T07:07:46.115090+0000","last_clean":"2024-09-17T07:07:46.115090+0000","last_became_active":"2024-09-17T07:02:14.014939+0000","last_became_peered":"2024-09-17T07:02:14.014939+0000","last_unstale":"2024-09-17T07:07:46.115090+0000","last_undegraded":"2024-09-17T07:07:46.115090+0000","last_fullsized":"2024-09-17T07:07:46.115090+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:06:26.995923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:06:26.995923+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:49.741971+0000","scrub_duration":0.01417006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":132,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:44.181278+0000","last_change":"2024-09-17T07:06:30.038396+0000","last_active":"2024-09-17T07:07:44.181278+0000","last_peered":"2024-09-17T07:07:44.181278+0000","last_clean":"2024-09-17T07:07:44.181278+0000","last_became_active":"2024-09-17T07:02:12.000005+0000","last_became_peered":"2024-09-17T07:02:12.000005+0000","last_unstale":"2024-09-17T07:07:44.181278+0000","last_undegraded":"2024-09-17T07:07:44.181278+0000","last_fullsized":"2024-09-17T07:07:44.181278+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:30.038104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:06:30.038104+0000","last_clean_scrub_stamp":"2024-09-17T07:06:30.038104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.021438763,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":94,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:44.180552+0000","last_change":"2024-09-17T07:07:19.789263+0000","last_active":"2024-09-17T07:07:44.180552+0000","last_peered":"2024-09-17T07:07:44.180552+0000","last_clean":"2024-09-17T07:07:44.180552+0000","last_became_active":"2024-09-17T07:04:15.107886+0000","last_became_peered":"2024-09-17T07:04:15.107886+0000","last_unstale":"2024-09-17T07:07:44.180552+0000","last_undegraded":"2024-09-17T07:07:44.180552+0000","last_fullsized":"2024-09-17T07:07:44.180552+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:19.789134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:07:19.789134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:20.148885+0000","scrub_duration":0.010505654,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":127,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:46.114369+0000","last_change":"2024-09-17T07:07:20.025023+0000","last_active":"2024-09-17T07:07:46.114369+0000","last_peered":"2024-09-17T07:07:46.114369+0000","last_clean":"2024-09-17T07:07:46.114369+0000","last_became_active":"2024-09-17T07:02:18.026994+0000","last_became_peered":"2024-09-17T07:02:18.026994+0000","last_unstale":"2024-09-17T07:07:46.114369+0000","last_undegraded":"2024-09-17T07:07:46.114369+0000","last_fullsized":"2024-09-17T07:07:46.114369+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:20.024902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:58.099849+0000","last_clean_scrub_stamp":"2024-09-17T07:07:20.024902+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:30.210770+0000","scrub_duration":0.013970381,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"111'22","reported_seq":173,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:46.114655+0000","last_change":"2024-09-17T07:07:29.816377+0000","last_active":"2024-09-17T07:07:46.114655+0000","last_peered":"2024-09-17T07:07:46.114655+0000","last_clean":"2024-09-17T07:07:46.114655+0000","last_became_active":"2024-09-17T07:02:14.028392+0000","last_became_peered":"2024-09-17T07:02:14.028392+0000","last_unstale":"2024-09-17T07:07:46.114655+0000","last_undegraded":"2024-09-17T07:07:46.114655+0000","last_fullsized":"2024-09-17T07:07:46.114655+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'22","last_scrub_stamp":"2024-09-17T07:07:29.816241+0000","last_deep_scrub":"111'22","last_deep_scrub_stamp":"2024-09-17T07:04:57.500450+0000","last_clean_scrub_stamp":"2024-09-17T07:07:29.816241+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:53.480974+0000","scrub_duration":0.0034677570000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'2","reported_seq":133,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:45.732587+0000","last_change":"2024-09-17T07:06:39.828818+0000","last_active":"2024-09-17T07:07:45.732587+0000","last_peered":"2024-09-17T07:07:45.732587+0000","last_clean":"2024-09-17T07:07:45.732587+0000","last_became_active":"2024-09-17T07:04:15.110085+0000","last_became_peered":"2024-09-17T07:04:15.110085+0000","last_unstale":"2024-09-17T07:07:45.732587+0000","last_undegraded":"2024-09-17T07:07:45.732587+0000","last_fullsized":"2024-09-17T07:07:45.732587+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-09-17T07:06:39.828638+0000","last_deep_scrub":"89'2","last_deep_scrub_stamp":"2024-09-17T07:03:26.694366+0000","last_clean_scrub_stamp":"2024-09-17T07:06:39.828638+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:09.464498+0000","scrub_duration":0.0068983610000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":420,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"75'8","reported_seq":207,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:46.033848+0000","last_change":"2024-09-17T07:06:25.761229+0000","last_active":"2024-09-17T07:07:46.033848+0000","last_peered":"2024-09-17T07:07:46.033848+0000","last_clean":"2024-09-17T07:07:46.033848+0000","last_became_active":"2024-09-17T07:02:16.015418+0000","last_became_peered":"2024-09-17T07:02:16.015418+0000","last_unstale":"2024-09-17T07:07:46.033848+0000","last_undegraded":"2024-09-17T07:07:46.033848+0000","last_fullsized":"2024-09-17T07:07:46.033848+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:06:25.760922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:06:25.760922+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0073758620000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":137,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:46.114802+0000","last_change":"2024-09-17T07:07:19.786753+0000","last_active":"2024-09-17T07:07:46.114802+0000","last_peered":"2024-09-17T07:07:46.114802+0000","last_clean":"2024-09-17T07:07:46.114802+0000","last_became_active":"2024-09-17T07:02:18.363253+0000","last_became_peered":"2024-09-17T07:02:18.363253+0000","last_unstale":"2024-09-17T07:07:46.114802+0000","last_undegraded":"2024-09-17T07:07:46.114802+0000","last_fullsized":"2024-09-17T07:07:46.114802+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:19.786442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:07:19.786442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:41.487701+0000","scrub_duration":0.011747911999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"150'17","reported_seq":159,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:44.180536+0000","last_change":"2024-09-17T07:06:38.775932+0000","last_active":"2024-09-17T07:07:44.180536+0000","last_peered":"2024-09-17T07:07:44.180536+0000","last_clean":"2024-09-17T07:07:44.180536+0000","last_became_active":"2024-09-17T07:02:14.010219+0000","last_became_peered":"2024-09-17T07:02:14.010219+0000","last_unstale":"2024-09-17T07:07:44.180536+0000","last_undegraded":"2024-09-17T07:07:44.180536+0000","last_fullsized":"2024-09-17T07:07:44.180536+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'17","last_scrub_stamp":"2024-09-17T07:06:38.775788+0000","last_deep_scrub":"150'17","last_deep_scrub_stamp":"2024-09-17T07:06:38.775788+0000","last_clean_scrub_stamp":"2024-09-17T07:06:38.775788+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:54.992016+0000","scrub_duration":0.011503368999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"68'2","reported_seq":141,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:44.181315+0000","last_change":"2024-09-17T07:07:20.764435+0000","last_active":"2024-09-17T07:07:44.181315+0000","last_peered":"2024-09-17T07:07:44.181315+0000","last_clean":"2024-09-17T07:07:44.181315+0000","last_became_active":"2024-09-17T07:02:12.006716+0000","last_became_peered":"2024-09-17T07:02:12.006716+0000","last_unstale":"2024-09-17T07:07:44.181315+0000","last_undegraded":"2024-09-17T07:07:44.181315+0000","last_fullsized":"2024-09-17T07:07:44.181315+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-09-17T07:07:20.764283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:07:20.764283+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:50.410328+0000","scrub_duration":0.0060160040000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":93,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:46.114784+0000","last_change":"2024-09-17T07:04:55.444916+0000","last_active":"2024-09-17T07:07:46.114784+0000","last_peered":"2024-09-17T07:07:46.114784+0000","last_clean":"2024-09-17T07:07:46.114784+0000","last_became_active":"2024-09-17T07:04:15.111069+0000","last_became_peered":"2024-09-17T07:04:15.111069+0000","last_unstale":"2024-09-17T07:07:46.114784+0000","last_undegraded":"2024-09-17T07:07:46.114784+0000","last_fullsized":"2024-09-17T07:07:46.114784+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:55.444802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:04:55.444802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0079380379999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":115,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:45.732166+0000","last_change":"2024-09-17T07:04:06.571270+0000","last_active":"2024-09-17T07:07:45.732166+0000","last_peered":"2024-09-17T07:07:45.732166+0000","last_clean":"2024-09-17T07:07:45.732166+0000","last_became_active":"2024-09-17T07:02:18.024983+0000","last_became_peered":"2024-09-17T07:02:18.024983+0000","last_unstale":"2024-09-17T07:07:45.732166+0000","last_undegraded":"2024-09-17T07:07:45.732166+0000","last_fullsized":"2024-09-17T07:07:45.732166+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:06.571184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:04:06.571184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001613319,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"75'9","reported_seq":147,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:46.115227+0000","last_change":"2024-09-17T07:06:30.011545+0000","last_active":"2024-09-17T07:07:46.115227+0000","last_peered":"2024-09-17T07:07:46.115227+0000","last_clean":"2024-09-17T07:07:46.115227+0000","last_became_active":"2024-09-17T07:02:14.029388+0000","last_became_peered":"2024-09-17T07:02:14.029388+0000","last_unstale":"2024-09-17T07:07:46.115227+0000","last_undegraded":"2024-09-17T07:07:46.115227+0000","last_fullsized":"2024-09-17T07:07:46.115227+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:06:30.011337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:06:30.011337+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.013862154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":144,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:44.179087+0000","last_change":"2024-09-17T07:06:40.406566+0000","last_active":"2024-09-17T07:07:44.179087+0000","last_peered":"2024-09-17T07:07:44.179087+0000","last_clean":"2024-09-17T07:07:44.179087+0000","last_became_active":"2024-09-17T07:02:12.011327+0000","last_became_peered":"2024-09-17T07:02:12.011327+0000","last_unstale":"2024-09-17T07:07:44.179087+0000","last_undegraded":"2024-09-17T07:07:44.179087+0000","last_fullsized":"2024-09-17T07:07:44.179087+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:40.406459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:31.753470+0000","last_clean_scrub_stamp":"2024-09-17T07:06:40.406459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:54.638620+0000","scrub_duration":0.03274349,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":128,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:45.732714+0000","last_change":"2024-09-17T07:04:36.652033+0000","last_active":"2024-09-17T07:07:45.732714+0000","last_peered":"2024-09-17T07:07:45.732714+0000","last_clean":"2024-09-17T07:07:45.732714+0000","last_became_active":"2024-09-17T07:02:16.022584+0000","last_became_peered":"2024-09-17T07:02:16.022584+0000","last_unstale":"2024-09-17T07:07:45.732714+0000","last_undegraded":"2024-09-17T07:07:45.732714+0000","last_fullsized":"2024-09-17T07:07:45.732714+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:36.651925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:04:36.651925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0061789210000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":138,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:44.179174+0000","last_change":"2024-09-17T07:06:36.398341+0000","last_active":"2024-09-17T07:07:44.179174+0000","last_peered":"2024-09-17T07:07:44.179174+0000","last_clean":"2024-09-17T07:07:44.179174+0000","last_became_active":"2024-09-17T07:02:18.017825+0000","last_became_peered":"2024-09-17T07:02:18.017825+0000","last_unstale":"2024-09-17T07:07:44.179174+0000","last_undegraded":"2024-09-17T07:07:44.179174+0000","last_fullsized":"2024-09-17T07:07:44.179174+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:36.398177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:06:36.398177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:57.728751+0000","scrub_duration":0.099287720999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":132,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:46.115661+0000","last_change":"2024-09-17T07:06:11.251464+0000","last_active":"2024-09-17T07:07:46.115661+0000","last_peered":"2024-09-17T07:07:46.115661+0000","last_clean":"2024-09-17T07:07:46.115661+0000","last_became_active":"2024-09-17T07:02:12.008065+0000","last_became_peered":"2024-09-17T07:02:12.008065+0000","last_unstale":"2024-09-17T07:07:46.115661+0000","last_undegraded":"2024-09-17T07:07:46.115661+0000","last_fullsized":"2024-09-17T07:07:46.115661+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:11.251341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:06:11.251341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0029670590000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"75'6","reported_seq":139,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:45.732788+0000","last_change":"2024-09-17T07:06:43.839442+0000","last_active":"2024-09-17T07:07:45.732788+0000","last_peered":"2024-09-17T07:07:45.732788+0000","last_clean":"2024-09-17T07:07:45.732788+0000","last_became_active":"2024-09-17T07:02:14.028655+0000","last_became_peered":"2024-09-17T07:02:14.028655+0000","last_unstale":"2024-09-17T07:07:45.732788+0000","last_undegraded":"2024-09-17T07:07:45.732788+0000","last_fullsized":"2024-09-17T07:07:45.732788+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'6","last_scrub_stamp":"2024-09-17T07:06:43.836545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:06:43.836545+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:48.040328+0000","scrub_duration":0.0059941980000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":128,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:46.115870+0000","last_change":"2024-09-17T07:06:29.854234+0000","last_active":"2024-09-17T07:07:46.115870+0000","last_peered":"2024-09-17T07:07:46.115870+0000","last_clean":"2024-09-17T07:07:46.115870+0000","last_became_active":"2024-09-17T07:02:16.016634+0000","last_became_peered":"2024-09-17T07:02:16.016634+0000","last_unstale":"2024-09-17T07:07:46.115870+0000","last_undegraded":"2024-09-17T07:07:46.115870+0000","last_fullsized":"2024-09-17T07:07:46.115870+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:29.854126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:06:29.854126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0052853860000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":127,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:46.034127+0000","last_change":"2024-09-17T07:07:18.042204+0000","last_active":"2024-09-17T07:07:46.034127+0000","last_peered":"2024-09-17T07:07:46.034127+0000","last_clean":"2024-09-17T07:07:46.034127+0000","last_became_active":"2024-09-17T07:02:18.365419+0000","last_became_peered":"2024-09-17T07:02:18.365419+0000","last_unstale":"2024-09-17T07:07:46.034127+0000","last_undegraded":"2024-09-17T07:07:46.034127+0000","last_fullsized":"2024-09-17T07:07:46.034127+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:18.042105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:07:18.042105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:18.135607+0000","scrub_duration":0.0020576729999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"75'10","reported_seq":145,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:45.732722+0000","last_change":"2024-09-17T07:04:40.721975+0000","last_active":"2024-09-17T07:07:45.732722+0000","last_peered":"2024-09-17T07:07:45.732722+0000","last_clean":"2024-09-17T07:07:45.732722+0000","last_became_active":"2024-09-17T07:02:14.027499+0000","last_became_peered":"2024-09-17T07:02:14.027499+0000","last_unstale":"2024-09-17T07:07:45.732722+0000","last_undegraded":"2024-09-17T07:07:45.732722+0000","last_fullsized":"2024-09-17T07:07:45.732722+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-09-17T07:04:40.721843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:04:40.721843+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0057465399999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":143,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:46.115060+0000","last_change":"2024-09-17T07:06:45.799922+0000","last_active":"2024-09-17T07:07:46.115060+0000","last_peered":"2024-09-17T07:07:46.115060+0000","last_clean":"2024-09-17T07:07:46.115060+0000","last_became_active":"2024-09-17T07:02:12.007280+0000","last_became_peered":"2024-09-17T07:02:12.007280+0000","last_unstale":"2024-09-17T07:07:46.115060+0000","last_undegraded":"2024-09-17T07:07:46.115060+0000","last_fullsized":"2024-09-17T07:07:46.115060+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:45.799808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:06:45.799808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:47.601528+0000","scrub_duration":0.0021198189999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"163'9","reported_seq":216,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-17T07:07:47.564380+0000","last_change":"2024-09-17T07:07:34.933956+0000","last_active":"2024-09-17T07:07:47.564380+0000","last_peered":"2024-09-17T07:07:47.564380+0000","last_clean":"2024-09-17T07:07:47.564380+0000","last_became_active":"2024-09-17T07:02:16.022583+0000","last_became_peered":"2024-09-17T07:02:16.022583+0000","last_unstale":"2024-09-17T07:07:47.564380+0000","last_undegraded":"2024-09-17T07:07:47.564380+0000","last_fullsized":"2024-09-17T07:07:47.564380+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'9","last_scrub_stamp":"2024-09-17T07:07:34.933808+0000","last_deep_scrub":"163'9","last_deep_scrub_stamp":"2024-09-17T07:07:34.933808+0000","last_clean_scrub_stamp":"2024-09-17T07:07:34.933808+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:57.452885+0000","scrub_duration":0.019504579000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":127,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:46.114404+0000","last_change":"2024-09-17T07:07:26.014516+0000","last_active":"2024-09-17T07:07:46.114404+0000","last_peered":"2024-09-17T07:07:46.114404+0000","last_clean":"2024-09-17T07:07:46.114404+0000","last_became_active":"2024-09-17T07:02:18.028796+0000","last_became_peered":"2024-09-17T07:02:18.028796+0000","last_unstale":"2024-09-17T07:07:46.114404+0000","last_undegraded":"2024-09-17T07:07:46.114404+0000","last_fullsized":"2024-09-17T07:07:46.114404+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:26.014398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:07:26.014398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:36.974117+0000","scrub_duration":0.0066951459999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"75'9","reported_seq":157,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:46.115309+0000","last_change":"2024-09-17T07:06:35.727847+0000","last_active":"2024-09-17T07:07:46.115309+0000","last_peered":"2024-09-17T07:07:46.115309+0000","last_clean":"2024-09-17T07:07:46.115309+0000","last_became_active":"2024-09-17T07:02:14.028485+0000","last_became_peered":"2024-09-17T07:02:14.028485+0000","last_unstale":"2024-09-17T07:07:46.115309+0000","last_undegraded":"2024-09-17T07:07:46.115309+0000","last_fullsized":"2024-09-17T07:07:46.115309+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:06:35.727700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:06:35.727700+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:52.353085+0000","scrub_duration":0.0045062289999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":133,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:46.115342+0000","last_change":"2024-09-17T07:04:51.449374+0000","last_active":"2024-09-17T07:07:46.115342+0000","last_peered":"2024-09-17T07:07:46.115342+0000","last_clean":"2024-09-17T07:07:46.115342+0000","last_became_active":"2024-09-17T07:02:12.009935+0000","last_became_peered":"2024-09-17T07:02:12.009935+0000","last_unstale":"2024-09-17T07:07:46.115342+0000","last_undegraded":"2024-09-17T07:07:46.115342+0000","last_fullsized":"2024-09-17T07:07:46.115342+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:51.449197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:04:51.449197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0069844269999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"75'8","reported_seq":203,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:45.732447+0000","last_change":"2024-09-17T07:07:32.670103+0000","last_active":"2024-09-17T07:07:45.732447+0000","last_peered":"2024-09-17T07:07:45.732447+0000","last_clean":"2024-09-17T07:07:45.732447+0000","last_became_active":"2024-09-17T07:02:16.016069+0000","last_became_peered":"2024-09-17T07:02:16.016069+0000","last_unstale":"2024-09-17T07:07:45.732447+0000","last_undegraded":"2024-09-17T07:07:45.732447+0000","last_fullsized":"2024-09-17T07:07:45.732447+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:07:32.669921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:07:32.669921+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:44.233420+0000","scrub_duration":0.0099051529999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":127,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:44.181136+0000","last_change":"2024-09-17T07:04:58.773306+0000","last_active":"2024-09-17T07:07:44.181136+0000","last_peered":"2024-09-17T07:07:44.181136+0000","last_clean":"2024-09-17T07:07:44.181136+0000","last_became_active":"2024-09-17T07:02:18.018716+0000","last_became_peered":"2024-09-17T07:02:18.018716+0000","last_unstale":"2024-09-17T07:07:44.181136+0000","last_undegraded":"2024-09-17T07:07:44.181136+0000","last_fullsized":"2024-09-17T07:07:44.181136+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:58.773188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:04:58.773188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0024048339999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"75'9","reported_seq":146,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:44.179481+0000","last_change":"2024-09-17T07:04:56.049769+0000","last_active":"2024-09-17T07:07:44.179481+0000","last_peered":"2024-09-17T07:07:44.179481+0000","last_clean":"2024-09-17T07:07:44.179481+0000","last_became_active":"2024-09-17T07:02:14.010289+0000","last_became_peered":"2024-09-17T07:02:14.010289+0000","last_unstale":"2024-09-17T07:07:44.179481+0000","last_undegraded":"2024-09-17T07:07:44.179481+0000","last_fullsized":"2024-09-17T07:07:44.179481+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:04:56.049602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:04:56.049602+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0066971340000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":133,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:46.115356+0000","last_change":"2024-09-17T07:07:20.458487+0000","last_active":"2024-09-17T07:07:46.115356+0000","last_peered":"2024-09-17T07:07:46.115356+0000","last_clean":"2024-09-17T07:07:46.115356+0000","last_became_active":"2024-09-17T07:02:12.007433+0000","last_became_peered":"2024-09-17T07:02:12.007433+0000","last_unstale":"2024-09-17T07:07:46.115356+0000","last_undegraded":"2024-09-17T07:07:46.115356+0000","last_fullsized":"2024-09-17T07:07:46.115356+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:20.458316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:07:20.458316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:42.134938+0000","scrub_duration":0.0043263889999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":128,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:45.732208+0000","last_change":"2024-09-17T07:07:17.713153+0000","last_active":"2024-09-17T07:07:45.732208+0000","last_peered":"2024-09-17T07:07:45.732208+0000","last_clean":"2024-09-17T07:07:45.732208+0000","last_became_active":"2024-09-17T07:02:16.016216+0000","last_became_peered":"2024-09-17T07:02:16.016216+0000","last_unstale":"2024-09-17T07:07:45.732208+0000","last_undegraded":"2024-09-17T07:07:45.732208+0000","last_fullsized":"2024-09-17T07:07:45.732208+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:17.713027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:07:17.713027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:19.920380+0000","scrub_duration":0.017242598000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":127,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:46.114533+0000","last_change":"2024-09-17T07:06:24.966953+0000","last_active":"2024-09-17T07:07:46.114533+0000","last_peered":"2024-09-17T07:07:46.114533+0000","last_clean":"2024-09-17T07:07:46.114533+0000","last_became_active":"2024-09-17T07:02:18.027622+0000","last_became_peered":"2024-09-17T07:02:18.027622+0000","last_unstale":"2024-09-17T07:07:46.114533+0000","last_undegraded":"2024-09-17T07:07:46.114533+0000","last_fullsized":"2024-09-17T07:07:46.114533+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:24.966867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:06:24.966867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001663292,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"75'11","reported_seq":150,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:44.180798+0000","last_change":"2024-09-17T07:06:11.787256+0000","last_active":"2024-09-17T07:07:44.180798+0000","last_peered":"2024-09-17T07:07:44.180798+0000","last_clean":"2024-09-17T07:07:44.180798+0000","last_became_active":"2024-09-17T07:02:14.016255+0000","last_became_peered":"2024-09-17T07:02:14.016255+0000","last_unstale":"2024-09-17T07:07:44.180798+0000","last_undegraded":"2024-09-17T07:07:44.180798+0000","last_fullsized":"2024-09-17T07:07:44.180798+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-09-17T07:06:11.787156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:06:11.787156+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035465940000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"94'2","reported_seq":138,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:44.181260+0000","last_change":"2024-09-17T07:07:31.718074+0000","last_active":"2024-09-17T07:07:44.181260+0000","last_peered":"2024-09-17T07:07:44.181260+0000","last_clean":"2024-09-17T07:07:44.181260+0000","last_became_active":"2024-09-17T07:02:12.011461+0000","last_became_peered":"2024-09-17T07:02:12.011461+0000","last_unstale":"2024-09-17T07:07:44.181260+0000","last_undegraded":"2024-09-17T07:07:44.181260+0000","last_fullsized":"2024-09-17T07:07:44.181260+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-09-17T07:07:31.717888+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-09-17T07:07:31.717888+0000","last_clean_scrub_stamp":"2024-09-17T07:07:31.717888+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:39.938682+0000","scrub_duration":0.0094559669999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":128,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:44.179404+0000","last_change":"2024-09-17T07:06:40.147092+0000","last_active":"2024-09-17T07:07:44.179404+0000","last_peered":"2024-09-17T07:07:44.179404+0000","last_clean":"2024-09-17T07:07:44.179404+0000","last_became_active":"2024-09-17T07:02:16.018754+0000","last_became_peered":"2024-09-17T07:02:16.018754+0000","last_unstale":"2024-09-17T07:07:44.179404+0000","last_undegraded":"2024-09-17T07:07:44.179404+0000","last_fullsized":"2024-09-17T07:07:44.179404+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:40.146846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:06:40.146846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:05.696921+0000","scrub_duration":0.00753165,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":115,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:45.731934+0000","last_change":"2024-09-17T07:04:07.566836+0000","last_active":"2024-09-17T07:07:45.731934+0000","last_peered":"2024-09-17T07:07:45.731934+0000","last_clean":"2024-09-17T07:07:45.731934+0000","last_became_active":"2024-09-17T07:02:18.027427+0000","last_became_peered":"2024-09-17T07:02:18.027427+0000","last_unstale":"2024-09-17T07:07:45.731934+0000","last_undegraded":"2024-09-17T07:07:45.731934+0000","last_fullsized":"2024-09-17T07:07:45.731934+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:07.566667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:04:07.566667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001538364,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"75'4","reported_seq":151,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:45.731995+0000","last_change":"2024-09-17T07:06:36.857728+0000","last_active":"2024-09-17T07:07:45.731995+0000","last_peered":"2024-09-17T07:07:45.731995+0000","last_clean":"2024-09-17T07:07:45.731995+0000","last_became_active":"2024-09-17T07:03:48.758220+0000","last_became_peered":"2024-09-17T07:03:48.758220+0000","last_unstale":"2024-09-17T07:07:45.731995+0000","last_undegraded":"2024-09-17T07:07:45.731995+0000","last_fullsized":"2024-09-17T07:07:45.731995+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'4","last_scrub_stamp":"2024-09-17T07:06:36.857471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:06:36.857471+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:56.904497+0000","scrub_duration":0.0095623189999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":132,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:46.115955+0000","last_change":"2024-09-17T07:06:26.844807+0000","last_active":"2024-09-17T07:07:46.115955+0000","last_peered":"2024-09-17T07:07:46.115955+0000","last_clean":"2024-09-17T07:07:46.115955+0000","last_became_active":"2024-09-17T07:02:12.008898+0000","last_became_peered":"2024-09-17T07:02:12.008898+0000","last_unstale":"2024-09-17T07:07:46.115955+0000","last_undegraded":"2024-09-17T07:07:46.115955+0000","last_fullsized":"2024-09-17T07:07:46.115955+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:26.844664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:06:26.844664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0012489000000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":128,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:46.114960+0000","last_change":"2024-09-17T07:04:57.667081+0000","last_active":"2024-09-17T07:07:46.114960+0000","last_peered":"2024-09-17T07:07:46.114960+0000","last_clean":"2024-09-17T07:07:46.114960+0000","last_became_active":"2024-09-17T07:02:16.014982+0000","last_became_peered":"2024-09-17T07:02:16.014982+0000","last_unstale":"2024-09-17T07:07:46.114960+0000","last_undegraded":"2024-09-17T07:07:46.114960+0000","last_fullsized":"2024-09-17T07:07:46.114960+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:57.667006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:47.444972+0000","last_clean_scrub_stamp":"2024-09-17T07:04:57.667006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0011999210000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":127,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:46.033454+0000","last_change":"2024-09-17T07:06:40.865239+0000","last_active":"2024-09-17T07:07:46.033454+0000","last_peered":"2024-09-17T07:07:46.033454+0000","last_clean":"2024-09-17T07:07:46.033454+0000","last_became_active":"2024-09-17T07:02:18.021014+0000","last_became_peered":"2024-09-17T07:02:18.021014+0000","last_unstale":"2024-09-17T07:07:46.033454+0000","last_undegraded":"2024-09-17T07:07:46.033454+0000","last_fullsized":"2024-09-17T07:07:46.033454+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:40.865049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:06:40.865049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:48.155323+0000","scrub_duration":0.0029834100000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"75'11","reported_seq":149,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:45.732336+0000","last_change":"2024-09-17T07:04:52.733713+0000","last_active":"2024-09-17T07:07:45.732336+0000","last_peered":"2024-09-17T07:07:45.732336+0000","last_clean":"2024-09-17T07:07:45.732336+0000","last_became_active":"2024-09-17T07:02:14.027316+0000","last_became_peered":"2024-09-17T07:02:14.027316+0000","last_unstale":"2024-09-17T07:07:45.732336+0000","last_undegraded":"2024-09-17T07:07:45.732336+0000","last_fullsized":"2024-09-17T07:07:45.732336+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-09-17T07:04:52.733516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:04:52.733516+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.013209306000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":143,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:46.115437+0000","last_change":"2024-09-17T07:06:44.298186+0000","last_active":"2024-09-17T07:07:46.115437+0000","last_peered":"2024-09-17T07:07:46.115437+0000","last_clean":"2024-09-17T07:07:46.115437+0000","last_became_active":"2024-09-17T07:02:12.010195+0000","last_became_peered":"2024-09-17T07:02:12.010195+0000","last_unstale":"2024-09-17T07:07:46.115437+0000","last_undegraded":"2024-09-17T07:07:46.115437+0000","last_fullsized":"2024-09-17T07:07:46.115437+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:44.298065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:06:44.298065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:00.552155+0000","scrub_duration":0.0032727260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":129,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:46.033546+0000","last_change":"2024-09-17T07:06:33.856836+0000","last_active":"2024-09-17T07:07:46.033546+0000","last_peered":"2024-09-17T07:07:46.033546+0000","last_clean":"2024-09-17T07:07:46.033546+0000","last_became_active":"2024-09-17T07:02:16.016183+0000","last_became_peered":"2024-09-17T07:02:16.016183+0000","last_unstale":"2024-09-17T07:07:46.033546+0000","last_undegraded":"2024-09-17T07:07:46.033546+0000","last_fullsized":"2024-09-17T07:07:46.033546+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:33.856556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:06:33.856556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:00.460166+0000","scrub_duration":0.0017432000000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":126,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:45.732110+0000","last_change":"2024-09-17T07:07:26.832785+0000","last_active":"2024-09-17T07:07:45.732110+0000","last_peered":"2024-09-17T07:07:45.732110+0000","last_clean":"2024-09-17T07:07:45.732110+0000","last_became_active":"2024-09-17T07:02:18.365656+0000","last_became_peered":"2024-09-17T07:02:18.365656+0000","last_unstale":"2024-09-17T07:07:45.732110+0000","last_undegraded":"2024-09-17T07:07:45.732110+0000","last_fullsized":"2024-09-17T07:07:45.732110+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:26.832514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:07:26.832514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:29.955192+0000","scrub_duration":0.01115028,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"75'1","reported_seq":128,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:46.033489+0000","last_change":"2024-09-17T07:06:32.836489+0000","last_active":"2024-09-17T07:07:46.033489+0000","last_peered":"2024-09-17T07:07:46.033489+0000","last_clean":"2024-09-17T07:07:46.033489+0000","last_became_active":"2024-09-17T07:02:18.021855+0000","last_became_peered":"2024-09-17T07:02:18.021855+0000","last_unstale":"2024-09-17T07:07:46.033489+0000","last_undegraded":"2024-09-17T07:07:46.033489+0000","last_fullsized":"2024-09-17T07:07:46.033489+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-17T07:06:32.836363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:06:32.836363+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0056683599999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":143,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:46.114659+0000","last_change":"2024-09-17T07:07:22.782091+0000","last_active":"2024-09-17T07:07:46.114659+0000","last_peered":"2024-09-17T07:07:46.114659+0000","last_clean":"2024-09-17T07:07:46.114659+0000","last_became_active":"2024-09-17T07:02:12.005329+0000","last_became_peered":"2024-09-17T07:02:12.005329+0000","last_unstale":"2024-09-17T07:07:46.114659+0000","last_undegraded":"2024-09-17T07:07:46.114659+0000","last_fullsized":"2024-09-17T07:07:46.114659+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:22.781739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:07:22.781739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:44.511386+0000","scrub_duration":0.0037808680000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"75'10","reported_seq":146,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:46.115131+0000","last_change":"2024-09-17T07:07:22.081513+0000","last_active":"2024-09-17T07:07:46.115131+0000","last_peered":"2024-09-17T07:07:46.115131+0000","last_clean":"2024-09-17T07:07:46.115131+0000","last_became_active":"2024-09-17T07:02:14.028731+0000","last_became_peered":"2024-09-17T07:02:14.028731+0000","last_unstale":"2024-09-17T07:07:46.115131+0000","last_undegraded":"2024-09-17T07:07:46.115131+0000","last_fullsized":"2024-09-17T07:07:46.115131+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-09-17T07:07:22.081382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:07:22.081382+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:37.716407+0000","scrub_duration":0.056222292,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"75'8","reported_seq":207,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:46.115286+0000","last_change":"2024-09-17T07:07:19.425224+0000","last_active":"2024-09-17T07:07:46.115286+0000","last_peered":"2024-09-17T07:07:46.115286+0000","last_clean":"2024-09-17T07:07:46.115286+0000","last_became_active":"2024-09-17T07:02:16.020560+0000","last_became_peered":"2024-09-17T07:02:16.020560+0000","last_unstale":"2024-09-17T07:07:46.115286+0000","last_undegraded":"2024-09-17T07:07:46.115286+0000","last_fullsized":"2024-09-17T07:07:46.115286+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:07:19.423853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:07:19.423853+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:23.974900+0000","scrub_duration":0.0037511480000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":126,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:44.180469+0000","last_change":"2024-09-17T07:07:30.668537+0000","last_active":"2024-09-17T07:07:44.180469+0000","last_peered":"2024-09-17T07:07:44.180469+0000","last_clean":"2024-09-17T07:07:44.180469+0000","last_became_active":"2024-09-17T07:02:18.021603+0000","last_became_peered":"2024-09-17T07:02:18.021603+0000","last_unstale":"2024-09-17T07:07:44.180469+0000","last_undegraded":"2024-09-17T07:07:44.180469+0000","last_fullsized":"2024-09-17T07:07:44.180469+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:30.668428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:07:30.668428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:31.854801+0000","scrub_duration":0.0047713119999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":132,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:45.732837+0000","last_change":"2024-09-17T07:04:34.679969+0000","last_active":"2024-09-17T07:07:45.732837+0000","last_peered":"2024-09-17T07:07:45.732837+0000","last_clean":"2024-09-17T07:07:45.732837+0000","last_became_active":"2024-09-17T07:02:12.009837+0000","last_became_peered":"2024-09-17T07:02:12.009837+0000","last_unstale":"2024-09-17T07:07:45.732837+0000","last_undegraded":"2024-09-17T07:07:45.732837+0000","last_fullsized":"2024-09-17T07:07:45.732837+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:34.679855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:04:34.679855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0053726590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"75'11","reported_seq":181,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:46.115361+0000","last_change":"2024-09-17T07:06:34.254104+0000","last_active":"2024-09-17T07:07:46.115361+0000","last_peered":"2024-09-17T07:07:46.115361+0000","last_clean":"2024-09-17T07:07:46.115361+0000","last_became_active":"2024-09-17T07:03:48.760390+0000","last_became_peered":"2024-09-17T07:03:48.760390+0000","last_unstale":"2024-09-17T07:07:46.115361+0000","last_undegraded":"2024-09-17T07:07:46.115361+0000","last_fullsized":"2024-09-17T07:07:46.115361+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-09-17T07:06:34.253977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:06:34.253977+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002455334,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":129,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:46.115099+0000","last_change":"2024-09-17T07:07:31.068035+0000","last_active":"2024-09-17T07:07:46.115099+0000","last_peered":"2024-09-17T07:07:46.115099+0000","last_clean":"2024-09-17T07:07:46.115099+0000","last_became_active":"2024-09-17T07:02:16.019701+0000","last_became_peered":"2024-09-17T07:02:16.019701+0000","last_unstale":"2024-09-17T07:07:46.115099+0000","last_undegraded":"2024-09-17T07:07:46.115099+0000","last_fullsized":"2024-09-17T07:07:46.115099+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:31.067529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:07:31.067529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:32.075799+0000","scrub_duration":0.013304165,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":127,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:44.181082+0000","last_change":"2024-09-17T07:07:30.779462+0000","last_active":"2024-09-17T07:07:44.181082+0000","last_peered":"2024-09-17T07:07:44.181082+0000","last_clean":"2024-09-17T07:07:44.181082+0000","last_became_active":"2024-09-17T07:02:18.363752+0000","last_became_peered":"2024-09-17T07:02:18.363752+0000","last_unstale":"2024-09-17T07:07:44.181082+0000","last_undegraded":"2024-09-17T07:07:44.181082+0000","last_fullsized":"2024-09-17T07:07:44.181082+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:30.779382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:07:30.779382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:38.926296+0000","scrub_duration":0.0011291059999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":152,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:46.115171+0000","last_change":"2024-09-17T07:06:41.312034+0000","last_active":"2024-09-17T07:07:46.115171+0000","last_peered":"2024-09-17T07:07:46.115171+0000","last_clean":"2024-09-17T07:07:46.115171+0000","last_became_active":"2024-09-17T07:03:48.759384+0000","last_became_peered":"2024-09-17T07:03:48.759384+0000","last_unstale":"2024-09-17T07:07:46.115171+0000","last_undegraded":"2024-09-17T07:07:46.115171+0000","last_fullsized":"2024-09-17T07:07:46.115171+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:41.311912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:06:41.311912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:55.289346+0000","scrub_duration":0.016335471000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"75'15","reported_seq":142,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:44.180273+0000","last_change":"2024-09-17T07:07:33.775051+0000","last_active":"2024-09-17T07:07:44.180273+0000","last_peered":"2024-09-17T07:07:44.180273+0000","last_clean":"2024-09-17T07:07:44.180273+0000","last_became_active":"2024-09-17T07:04:15.114286+0000","last_became_peered":"2024-09-17T07:04:15.114286+0000","last_unstale":"2024-09-17T07:07:44.180273+0000","last_undegraded":"2024-09-17T07:07:44.180273+0000","last_fullsized":"2024-09-17T07:07:44.180273+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-09-17T07:07:33.774757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:07:33.774757+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:34.474071+0000","scrub_duration":0.0071874820000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":128,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:44.181101+0000","last_change":"2024-09-17T07:06:42.201934+0000","last_active":"2024-09-17T07:07:44.181101+0000","last_peered":"2024-09-17T07:07:44.181101+0000","last_clean":"2024-09-17T07:07:44.181101+0000","last_became_active":"2024-09-17T07:02:16.015903+0000","last_became_peered":"2024-09-17T07:02:16.015903+0000","last_unstale":"2024-09-17T07:07:44.181101+0000","last_undegraded":"2024-09-17T07:07:44.181101+0000","last_fullsized":"2024-09-17T07:07:44.181101+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:42.200604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:06:42.200604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:54.763122+0000","scrub_duration":0.051542886000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":115,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:45.732676+0000","last_change":"2024-09-17T07:04:10.606687+0000","last_active":"2024-09-17T07:07:45.732676+0000","last_peered":"2024-09-17T07:07:45.732676+0000","last_clean":"2024-09-17T07:07:45.732676+0000","last_became_active":"2024-09-17T07:02:18.366466+0000","last_became_peered":"2024-09-17T07:02:18.366466+0000","last_unstale":"2024-09-17T07:07:45.732676+0000","last_undegraded":"2024-09-17T07:07:45.732676+0000","last_fullsized":"2024-09-17T07:07:45.732676+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:10.606552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:04:10.606552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00165456,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":132,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:45.732346+0000","last_change":"2024-09-17T07:07:29.761118+0000","last_active":"2024-09-17T07:07:45.732346+0000","last_peered":"2024-09-17T07:07:45.732346+0000","last_clean":"2024-09-17T07:07:45.732346+0000","last_became_active":"2024-09-17T07:02:12.005493+0000","last_became_peered":"2024-09-17T07:02:12.005493+0000","last_unstale":"2024-09-17T07:07:45.732346+0000","last_undegraded":"2024-09-17T07:07:45.732346+0000","last_fullsized":"2024-09-17T07:07:45.732346+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:29.761001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:07:29.761001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:58.064357+0000","scrub_duration":0.0083413440000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"86'17","reported_seq":158,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:45.732375+0000","last_change":"2024-09-17T07:06:27.962393+0000","last_active":"2024-09-17T07:07:45.732375+0000","last_peered":"2024-09-17T07:07:45.732375+0000","last_clean":"2024-09-17T07:07:45.732375+0000","last_became_active":"2024-09-17T07:02:14.017737+0000","last_became_peered":"2024-09-17T07:02:14.017737+0000","last_unstale":"2024-09-17T07:07:45.732375+0000","last_undegraded":"2024-09-17T07:07:45.732375+0000","last_fullsized":"2024-09-17T07:07:45.732375+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'17","last_scrub_stamp":"2024-09-17T07:06:27.962276+0000","last_deep_scrub":"86'17","last_deep_scrub_stamp":"2024-09-17T07:03:18.241067+0000","last_clean_scrub_stamp":"2024-09-17T07:06:27.962276+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:48.911685+0000","scrub_duration":0.0050727970000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":129,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-17T07:07:44.180927+0000","last_change":"2024-09-17T07:05:00.800939+0000","last_active":"2024-09-17T07:07:44.180927+0000","last_peered":"2024-09-17T07:07:44.180927+0000","last_clean":"2024-09-17T07:07:44.180927+0000","last_became_active":"2024-09-17T07:02:16.017362+0000","last_became_peered":"2024-09-17T07:02:16.017362+0000","last_unstale":"2024-09-17T07:07:44.180927+0000","last_undegraded":"2024-09-17T07:07:44.180927+0000","last_fullsized":"2024-09-17T07:07:44.180927+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:05:00.800823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:05:00.800823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014180900000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67,"ondisk_log_size":67,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":697,"num_read_kb":454,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":421,"ondisk_log_size":421,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":329,"num_read_kb":324,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1386,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":12,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":4158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037833,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":207188,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":205822,"kb_avail":93537964,"statfs":{"total":95995035648,"available":95782875136,"internally_reserved":0,"allocated":1380352,"data_stored":1000316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":210762187},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:07:13 2024","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":1.897,"15min":0.98499999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":3.613,"5min":52.478999999999999,"15min":52.478999999999999},"last":0.502},{"interface":"front","average":{"1min":0.98399999999999999,"5min":1.8979999999999999,"15min":0.97399999999999998},"min":{"1min":0.25900000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":2.9449999999999998,"5min":52.871000000000002,"15min":52.871000000000002},"last":0.40000000000000002}]},{"osd":1,"last update":"Tue Sep 17 07:07:13 2024","interfaces":[{"interface":"back","average":{"1min":1.2869999999999999,"5min":0.71799999999999997,"15min":0.57799999999999996},"min":{"1min":0.25600000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4.5140000000000002,"5min":4.5140000000000002,"15min":4.5140000000000002},"last":0.47099999999999997},{"interface":"front","average":{"1min":1.2270000000000001,"5min":0.74399999999999999,"15min":0.59699999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":3.8399999999999999,"5min":4.5039999999999996,"15min":4.5039999999999996},"last":0.46200000000000002}]},{"osd":2,"last update":"Tue Sep 17 07:07:13 2024","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":1.2190000000000001,"15min":0.72499999999999998},"min":{"1min":0.38700000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":3.3759999999999999,"5min":52.055999999999997,"15min":52.055999999999997},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.93000000000000005,"5min":1.256,"15min":0.78000000000000003},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":4.4749999999999996,"5min":52.064999999999998,"15min":52.064999999999998},"last":0.42999999999999999}]},{"osd":3,"last update":"Tue Sep 17 07:07:13 2024","interfaces":[{"interface":"back","average":{"1min":1.0720000000000001,"5min":1.5720000000000001,"15min":0.874},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":3.9550000000000001,"5min":51.689,"15min":51.689},"last":0.51200000000000001},{"interface":"front","average":{"1min":1.54,"5min":1.8340000000000001,"15min":0.97799999999999998},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":3.5659999999999998,"5min":52.145000000000003,"15min":52.145000000000003},"last":0.33800000000000002}]},{"osd":4,"last update":"Tue Sep 17 07:07:13 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.436,"15min":0.44500000000000001},"min":{"1min":0.25800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":1.5980000000000001,"5min":1.5980000000000001,"15min":1.5980000000000001},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.48999999999999999,"15min":0.44900000000000001},"min":{"1min":0.26800000000000002,"5min":0.248,"15min":0.248},"max":{"1min":3.4089999999999998,"5min":3.4089999999999998,"15min":3.4089999999999998},"last":0.41099999999999998}]},{"osd":5,"last update":"Tue Sep 17 07:07:13 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.48799999999999999,"15min":0.45100000000000001},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":3.3620000000000001,"5min":3.3620000000000001,"15min":3.3620000000000001},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.47799999999999998,"15min":0.46600000000000003},"min":{"1min":0.311,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.6459999999999999,"5min":1.6459999999999999,"15min":1.6459999999999999},"last":0.443}]},{"osd":6,"last update":"Tue Sep 17 07:07:13 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.53200000000000003,"15min":0.50800000000000001},"min":{"1min":0.34499999999999997,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":1.675,"5min":1.675,"15min":1.675},"last":0.495},{"interface":"front","average":{"1min":0.91500000000000004,"5min":0.56699999999999995,"15min":0.51300000000000001},"min":{"1min":0.28899999999999998,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":3.2669999999999999,"5min":3.2669999999999999,"15min":3.2669999999999999},"last":0.48499999999999999}]}]},{"osd":6,"up_from":50,"seq":214748364876,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":158768,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":157438,"kb_avail":93586384,"statfs":{"total":95995035648,"available":95832457216,"internally_reserved":0,"allocated":1347584,"data_stored":992730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":161216967},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:06:50 2024","interfaces":[{"interface":"back","average":{"1min":7.2679999999999998,"5min":2.5960000000000001,"15min":1.198},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":75.323999999999998,"5min":75.323999999999998,"15min":75.323999999999998},"last":0.55000000000000004},{"interface":"front","average":{"1min":7.1449999999999996,"5min":2.5489999999999999,"15min":1.177},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":75.168999999999997,"5min":75.168999999999997,"15min":75.168999999999997},"last":0.40699999999999997}]},{"osd":1,"last update":"Tue Sep 17 07:06:50 2024","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":0.58999999999999997,"15min":0.52200000000000002},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":4.4790000000000001,"5min":4.4790000000000001,"15min":4.4790000000000001},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.89900000000000002,"5min":0.68999999999999995,"15min":0.58599999999999997},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":2.73,"5min":5.9530000000000003,"15min":5.9530000000000003},"last":0.53000000000000003}]},{"osd":2,"last update":"Tue Sep 17 07:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":0.97099999999999997,"15min":0.67100000000000004},"min":{"1min":0.28100000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":4.9850000000000003,"5min":33.079000000000001,"15min":33.079000000000001},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.97699999999999998,"15min":0.67200000000000004},"min":{"1min":0.32700000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":2.8759999999999999,"5min":33.100000000000001,"15min":33.100000000000001},"last":0.57399999999999995}]},{"osd":3,"last update":"Tue Sep 17 07:06:50 2024","interfaces":[{"interface":"back","average":{"1min":3.544,"5min":1.6419999999999999,"15min":0.86699999999999999},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":52.944000000000003,"5min":52.944000000000003,"15min":52.944000000000003},"last":0.45400000000000001},{"interface":"front","average":{"1min":3.3719999999999999,"5min":1.5700000000000001,"15min":0.88300000000000001},"min":{"1min":0.36199999999999999,"5min":0.308,"15min":0.308},"max":{"1min":52.747,"5min":52.747,"15min":52.747},"last":0.56499999999999995}]},{"osd":4,"last update":"Tue Sep 17 07:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.41999999999999998,"15min":0.41999999999999998},"min":{"1min":0.25900000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.48199999999999998,"15min":0.432},"min":{"1min":0.27200000000000002,"5min":0.25,"15min":0.25},"max":{"1min":2.9119999999999999,"5min":2.9119999999999999,"15min":2.9119999999999999},"last":0.31}]},{"osd":5,"last update":"Tue Sep 17 07:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.45500000000000002,"15min":0.433},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":1.6419999999999999,"5min":1.6419999999999999,"15min":1.6419999999999999},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.497,"15min":0.45000000000000001},"min":{"1min":0.29899999999999999,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":2.9390000000000001,"5min":2.9390000000000001,"15min":2.9390000000000001},"last":0.28299999999999997}]},{"osd":7,"last update":"Tue Sep 17 07:07:11 2024","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.55500000000000005,"15min":0.51500000000000001},"min":{"1min":0.307,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":2.859,"5min":2.859,"15min":2.859},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.57399999999999995,"15min":0.54200000000000004},"min":{"1min":0.38100000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.9339999999999999,"5min":1.9339999999999999,"15min":1.9339999999999999},"last":0.35099999999999998}]}]},{"osd":5,"up_from":42,"seq":180388626514,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":130308,"kb_used_data":756,"kb_used_omap":1,"kb_used_meta":129534,"kb_avail":93614844,"statfs":{"total":95995035648,"available":95861600256,"internally_reserved":0,"allocated":774144,"data_stored":410065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":132643270},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:07:31 2024","interfaces":[{"interface":"back","average":{"1min":2.9020000000000001,"5min":2.6400000000000001,"15min":1.226},"min":{"1min":0.23499999999999999,"5min":0.221,"15min":0.221},"max":{"1min":28.727,"5min":49.265000000000001,"15min":49.265000000000001},"last":0.38500000000000001},{"interface":"front","average":{"1min":2.7639999999999998,"5min":2.7450000000000001,"15min":1.2549999999999999},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":27.824999999999999,"5min":47.350000000000001,"15min":47.350000000000001},"last":0.42799999999999999}]},{"osd":1,"last update":"Tue Sep 17 07:07:31 2024","interfaces":[{"interface":"back","average":{"1min":1.766,"5min":0.79000000000000004,"15min":0.628},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":20.329000000000001,"5min":20.329000000000001,"15min":20.329000000000001},"last":0.44700000000000001},{"interface":"front","average":{"1min":1.9450000000000001,"5min":0.97099999999999997,"15min":1.121},"min":{"1min":0.27300000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":20.408999999999999,"5min":20.408999999999999,"15min":20.408999999999999},"last":0.40300000000000002}]},{"osd":2,"last update":"Tue Sep 17 07:07:31 2024","interfaces":[{"interface":"back","average":{"1min":2.0390000000000001,"5min":1.1100000000000001,"15min":0.748},"min":{"1min":0.34200000000000003,"5min":0.252,"15min":0.252},"max":{"1min":20.486000000000001,"5min":20.486000000000001,"15min":20.486000000000001},"last":0.41799999999999998},{"interface":"front","average":{"1min":2.1739999999999999,"5min":1.119,"15min":0.71199999999999997},"min":{"1min":0.26300000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":23.925999999999998,"5min":23.925999999999998,"15min":23.925999999999998},"last":0.35299999999999998}]},{"osd":3,"last update":"Tue Sep 17 07:07:31 2024","interfaces":[{"interface":"back","average":{"1min":6.8310000000000004,"5min":2.077,"15min":1.0609999999999999},"min":{"1min":0.25800000000000001,"5min":0.249,"15min":0.249},"max":{"1min":68.978999999999999,"5min":68.978999999999999,"15min":68.978999999999999},"last":0.39600000000000002},{"interface":"front","average":{"1min":6.851,"5min":2,"15min":1.0329999999999999},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":69.084000000000003,"5min":69.084000000000003,"15min":69.084000000000003},"last":0.51500000000000001}]},{"osd":4,"last update":"Tue Sep 17 07:07:31 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.436,"15min":0.41099999999999998},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":3.2989999999999999,"5min":3.2989999999999999,"15min":3.2989999999999999},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.439,"5min":0.442,"15min":0.437},"min":{"1min":0.26700000000000002,"5min":0.248,"15min":0.248},"max":{"1min":0.90900000000000003,"5min":2.2160000000000002,"15min":2.2160000000000002},"last":0.32200000000000001}]},{"osd":6,"last update":"Tue Sep 17 07:07:48 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.45900000000000002,"15min":0.437},"min":{"1min":0.23499999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":1.9199999999999999,"5min":1.9199999999999999,"15min":1.9199999999999999},"last":0.499},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.51900000000000002,"15min":0.46999999999999997},"min":{"1min":0.31900000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":2.3250000000000002,"5min":2.5590000000000002,"15min":2.5590000000000002},"last":0.36599999999999999}]},{"osd":7,"last update":"Tue Sep 17 07:07:06 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.51000000000000001,"15min":0.49099999999999999},"min":{"1min":0.307,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":2.5910000000000002,"5min":2.5910000000000002,"15min":2.5910000000000002},"last":0.439},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.47899999999999998,"15min":0.46899999999999997},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.48099999999999998}]}]},{"osd":4,"up_from":33,"seq":141733920856,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":99980,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93645172,"statfs":{"total":95995035648,"available":95892656128,"internally_reserved":0,"allocated":786432,"data_stored":409379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":101579206},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:06:56 2024","interfaces":[{"interface":"back","average":{"1min":3.1600000000000001,"5min":2.6579999999999999,"15min":1.274},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":48.825000000000003,"5min":74.551000000000002,"15min":74.551000000000002},"last":0.38100000000000001},{"interface":"front","average":{"1min":3.758,"5min":2.839,"15min":1.361},"min":{"1min":0.27500000000000002,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":50.911999999999999,"5min":74.778999999999996,"15min":74.778999999999996},"last":0.40899999999999997}]},{"osd":1,"last update":"Tue Sep 17 07:06:56 2024","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":0.628,"15min":0.59499999999999997},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4.1539999999999999,"5min":4.1539999999999999,"15min":4.1539999999999999},"last":0.42299999999999999},{"interface":"front","average":{"1min":1.222,"5min":0.66900000000000004,"15min":0.624},"min":{"1min":0.25600000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":3.5409999999999999,"5min":3.5409999999999999,"15min":3.5409999999999999},"last":0.46000000000000002}]},{"osd":2,"last update":"Tue Sep 17 07:06:56 2024","interfaces":[{"interface":"back","average":{"1min":3.7629999999999999,"5min":1.2490000000000001,"15min":0.81000000000000005},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":49.651000000000003,"5min":49.651000000000003,"15min":49.651000000000003},"last":0.27600000000000002},{"interface":"front","average":{"1min":3.9950000000000001,"5min":1.3080000000000001,"15min":0.83199999999999996},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":48.813000000000002,"5min":48.813000000000002,"15min":48.813000000000002},"last":0.49299999999999999}]},{"osd":3,"last update":"Tue Sep 17 07:06:56 2024","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.194,"15min":0.80800000000000005},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":3.4590000000000001,"5min":39.042999999999999,"15min":39.042999999999999},"last":0.54800000000000004},{"interface":"front","average":{"1min":1.0009999999999999,"5min":1.2490000000000001,"15min":0.79400000000000004},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":4.5679999999999996,"5min":38.920000000000002,"15min":38.920000000000002},"last":0.39800000000000002}]},{"osd":5,"last update":"Tue Sep 17 07:07:27 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.47499999999999998,"15min":0.41199999999999998},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":1.9510000000000001,"5min":2.4780000000000002,"15min":2.4780000000000002},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.48599999999999999,"15min":0.42999999999999999},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":2.0299999999999998,"5min":2.5419999999999998,"15min":2.5419999999999998},"last":0.34000000000000002}]},{"osd":6,"last update":"Tue Sep 17 07:07:47 2024","interfaces":[{"interface":"back","average":{"1min":3.1680000000000001,"5min":1.0429999999999999,"15min":0.60999999999999999},"min":{"1min":0.249,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":47.923999999999999,"5min":47.923999999999999,"15min":47.923999999999999},"last":0.28899999999999998},{"interface":"front","average":{"1min":3.2530000000000001,"5min":1.0529999999999999,"15min":0.62},"min":{"1min":0.28899999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":47.600999999999999,"5min":47.600999999999999,"15min":47.600999999999999},"last":0.32500000000000001}]},{"osd":7,"last update":"Tue Sep 17 07:07:07 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.54000000000000004,"15min":0.51400000000000001},"min":{"1min":0.28899999999999998,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":2.637,"5min":2.637,"15min":2.637},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.53200000000000003,"15min":0.497},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":3.7400000000000002,"5min":3.7400000000000002,"15min":3.7400000000000002},"last":0.35499999999999998}]}]},{"osd":3,"up_from":27,"seq":115964117081,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":176292,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":175486,"kb_avail":93568860,"statfs":{"total":95995035648,"available":95814512640,"internally_reserved":0,"allocated":811008,"data_stored":410163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":179698120},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:07:44 2024","interfaces":[{"interface":"back","average":{"1min":2.0739999999999998,"5min":1.54,"15min":0.75600000000000001},"min":{"1min":0.21199999999999999,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":8.9849999999999994,"5min":17.364000000000001,"15min":17.364000000000001},"last":0.25700000000000001},{"interface":"front","average":{"1min":1.7729999999999999,"5min":1.4099999999999999,"15min":0.72299999999999998},"min":{"1min":0.27400000000000002,"5min":0.20399999999999999,"15min":0.189},"max":{"1min":5.9560000000000004,"5min":18.521000000000001,"15min":18.521000000000001},"last":0.318}]},{"osd":1,"last update":"Tue Sep 17 07:07:44 2024","interfaces":[{"interface":"back","average":{"1min":1.163,"5min":0.94599999999999995,"15min":0.58599999999999997},"min":{"1min":0.20999999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":8.3300000000000001,"5min":8.3300000000000001,"15min":8.3300000000000001},"last":0.33600000000000002},{"interface":"front","average":{"1min":1.552,"5min":1.2370000000000001,"15min":0.71299999999999997},"min":{"1min":0.19800000000000001,"5min":0.182,"15min":0.182},"max":{"1min":5.258,"5min":8.0670000000000002,"15min":8.0670000000000002},"last":0.41499999999999998}]},{"osd":2,"last update":"Tue Sep 17 07:07:44 2024","interfaces":[{"interface":"back","average":{"1min":2.1720000000000002,"5min":1.5569999999999999,"15min":0.80100000000000005},"min":{"1min":0.26100000000000001,"5min":0.22,"15min":0.22},"max":{"1min":5.3419999999999996,"5min":9.7319999999999993,"15min":9.7319999999999993},"last":0.36899999999999999},{"interface":"front","average":{"1min":1.2350000000000001,"5min":1.1899999999999999,"15min":0.67700000000000005},"min":{"1min":0.25,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":4.5049999999999999,"5min":8.3350000000000009,"15min":8.3350000000000009},"last":0.35699999999999998}]},{"osd":4,"last update":"Tue Sep 17 07:06:58 2024","interfaces":[{"interface":"back","average":{"1min":1.54,"5min":0.96799999999999997,"15min":0.69999999999999996},"min":{"1min":0.38400000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":5.4539999999999997,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.51200000000000001},{"interface":"front","average":{"1min":1.167,"5min":0.81100000000000005,"15min":0.629},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":5.7850000000000001,"5min":5.7850000000000001,"15min":5.7850000000000001},"last":0.55000000000000004}]},{"osd":5,"last update":"Tue Sep 17 07:07:24 2024","interfaces":[{"interface":"back","average":{"1min":1.988,"5min":1.2090000000000001,"15min":0.80900000000000005},"min":{"1min":0.309,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":5.3200000000000003,"5min":7.9669999999999996,"15min":7.9669999999999996},"last":0.53600000000000003},{"interface":"front","average":{"1min":1.3899999999999999,"5min":0.98399999999999999,"15min":0.71899999999999997},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":3.9039999999999999,"5min":4.4160000000000004,"15min":4.4160000000000004},"last":0.46100000000000002}]},{"osd":6,"last update":"Tue Sep 17 07:07:44 2024","interfaces":[{"interface":"back","average":{"1min":1.73,"5min":1.2090000000000001,"15min":0.77900000000000003},"min":{"1min":0.36299999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":8.2919999999999998,"5min":8.2919999999999998,"15min":8.2919999999999998},"last":0.44700000000000001},{"interface":"front","average":{"1min":1.5129999999999999,"5min":1.2190000000000001,"15min":0.80000000000000004},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":5.5529999999999999,"5min":7.9329999999999998,"15min":7.9329999999999998},"last":0.46899999999999997}]},{"osd":7,"last update":"Tue Sep 17 07:07:04 2024","interfaces":[{"interface":"back","average":{"1min":1.7869999999999999,"5min":1.085,"15min":0.754},"min":{"1min":0.40699999999999997,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":4.5119999999999996,"5min":6.1440000000000001,"15min":6.1440000000000001},"last":0.48699999999999999},{"interface":"front","average":{"1min":1.6559999999999999,"5min":1.0669999999999999,"15min":0.72699999999999998},"min":{"1min":0.39300000000000002,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":5.8170000000000002,"5min":5.8170000000000002,"15min":5.8170000000000002},"last":0.52600000000000002}]}]},{"osd":2,"up_from":18,"seq":77309411423,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":238596,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":237822,"kb_avail":93506556,"statfs":{"total":95995035648,"available":95750713344,"internally_reserved":0,"allocated":778240,"data_stored":409912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":243530182},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:07:27 2024","interfaces":[{"interface":"back","average":{"1min":2,"5min":0.95499999999999996,"15min":0.54900000000000004},"min":{"1min":0.248,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":7.2709999999999999,"5min":7.2709999999999999,"15min":7.2709999999999999},"last":0.25800000000000001},{"interface":"front","average":{"1min":1.716,"5min":1.3700000000000001,"15min":0.69199999999999995},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":5.9610000000000003,"5min":8.6869999999999994,"15min":8.6869999999999994},"last":0.19900000000000001}]},{"osd":1,"last update":"Tue Sep 17 07:07:27 2024","interfaces":[{"interface":"back","average":{"1min":1.5349999999999999,"5min":0.93600000000000005,"15min":0.54900000000000004},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":5.5010000000000003,"5min":5.5010000000000003,"15min":5.5010000000000003},"last":0.23000000000000001},{"interface":"front","average":{"1min":1.5880000000000001,"5min":0.84899999999999998,"15min":0.53300000000000003},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":6.9829999999999997,"5min":6.9829999999999997,"15min":6.9829999999999997},"last":0.28499999999999998}]},{"osd":3,"last update":"Tue Sep 17 07:07:47 2024","interfaces":[{"interface":"back","average":{"1min":1.129,"5min":0.88300000000000001,"15min":0.58699999999999997},"min":{"1min":0.27100000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":5.4930000000000003,"5min":5.4930000000000003,"15min":5.4930000000000003},"last":0.30099999999999999},{"interface":"front","average":{"1min":1.254,"5min":1.294,"15min":0.71799999999999997},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":6.0679999999999996,"5min":6.0679999999999996,"15min":6.0679999999999996},"last":0.247}]},{"osd":4,"last update":"Tue Sep 17 07:07:00 2024","interfaces":[{"interface":"back","average":{"1min":1.0609999999999999,"5min":0.877,"15min":0.67000000000000004},"min":{"1min":0.32000000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":2.653,"5min":5.2359999999999998,"15min":5.2359999999999998},"last":0.32700000000000001},{"interface":"front","average":{"1min":1.254,"5min":0.75900000000000001,"15min":0.64500000000000002},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":3.4689999999999999,"5min":3.4689999999999999,"15min":3.4689999999999999},"last":0.40999999999999998}]},{"osd":5,"last update":"Tue Sep 17 07:07:27 2024","interfaces":[{"interface":"back","average":{"1min":1.381,"5min":1.006,"15min":0.73199999999999998},"min":{"1min":0.26100000000000001,"5min":0.249,"15min":0.249},"max":{"1min":5.383,"5min":5.383,"15min":5.383},"last":0.31900000000000001},{"interface":"front","average":{"1min":1.121,"5min":0.85099999999999998,"15min":0.68100000000000005},"min":{"1min":0.34699999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":2.6880000000000002,"5min":5.3600000000000003,"15min":5.3600000000000003},"last":0.37}]},{"osd":6,"last update":"Tue Sep 17 07:07:47 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.90500000000000003,"15min":0.70399999999999996},"min":{"1min":0.33600000000000002,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":2.1459999999999999,"5min":9.2509999999999994,"15min":9.2509999999999994},"last":0.33600000000000002},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.0700000000000001,"15min":0.754},"min":{"1min":0.33400000000000002,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":5.3330000000000002,"5min":5.3330000000000002,"15min":5.3330000000000002},"last":0.39600000000000002}]},{"osd":7,"last update":"Tue Sep 17 07:07:17 2024","interfaces":[{"interface":"back","average":{"1min":1.1759999999999999,"5min":0.90500000000000003,"15min":0.68200000000000005},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":2.7090000000000001,"5min":5.3250000000000002,"15min":5.3250000000000002},"last":0.39100000000000001},{"interface":"front","average":{"1min":1.9950000000000001,"5min":1.0129999999999999,"15min":0.85899999999999999},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":9.2919999999999998,"5min":9.2919999999999998,"15min":9.2919999999999998},"last":0.36199999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574948,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":126252,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":125438,"kb_avail":93618900,"statfs":{"total":95995035648,"available":95865753600,"internally_reserved":0,"allocated":819200,"data_stored":411304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":128448966},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:06:51 2024","interfaces":[{"interface":"back","average":{"1min":1.3149999999999999,"5min":1.099,"15min":0.58999999999999997},"min":{"1min":0.26000000000000001,"5min":0.155,"15min":0.155},"max":{"1min":3.9929999999999999,"5min":13.837,"15min":13.837},"last":0.21199999999999999},{"interface":"front","average":{"1min":2.0600000000000001,"5min":1.1599999999999999,"15min":0.63},"min":{"1min":0.186,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":10.342000000000001,"5min":10.342000000000001,"15min":10.342000000000001},"last":0.35299999999999998}]},{"osd":2,"last update":"Tue Sep 17 07:07:21 2024","interfaces":[{"interface":"back","average":{"1min":1.8,"5min":1.2150000000000001,"15min":0.67400000000000004},"min":{"1min":0.23799999999999999,"5min":0.192,"15min":0.192},"max":{"1min":5.9509999999999996,"5min":11.055999999999999,"15min":11.055999999999999},"last":0.27700000000000002},{"interface":"front","average":{"1min":1.4470000000000001,"5min":1.302,"15min":0.70999999999999996},"min":{"1min":0.24199999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":5.4269999999999996,"5min":9.3160000000000007,"15min":9.3160000000000007},"last":0.29099999999999998}]},{"osd":3,"last update":"Tue Sep 17 07:07:46 2024","interfaces":[{"interface":"back","average":{"1min":1.125,"5min":1.2629999999999999,"15min":0.72899999999999998},"min":{"1min":0.27800000000000002,"5min":0.219,"15min":0.219},"max":{"1min":5.4740000000000002,"5min":7.766,"15min":7.766},"last":0.32700000000000001},{"interface":"front","average":{"1min":1.7030000000000001,"5min":1.351,"15min":0.745},"min":{"1min":0.28899999999999998,"5min":0.222,"15min":0.222},"max":{"1min":7.2759999999999998,"5min":19.602,"15min":19.602},"last":0.26600000000000001}]},{"osd":4,"last update":"Tue Sep 17 07:07:03 2024","interfaces":[{"interface":"back","average":{"1min":1.149,"5min":1.02,"15min":0.72199999999999998},"min":{"1min":0.29899999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":5.5030000000000001,"5min":7.8840000000000003,"15min":7.8840000000000003},"last":0.45200000000000001},{"interface":"front","average":{"1min":1.363,"5min":0.95199999999999996,"15min":0.69599999999999995},"min":{"1min":0.34200000000000003,"5min":0.307,"15min":0.307},"max":{"1min":7.1749999999999998,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.46400000000000002}]},{"osd":5,"last update":"Tue Sep 17 07:07:21 2024","interfaces":[{"interface":"back","average":{"1min":1.3129999999999999,"5min":1.087,"15min":0.74299999999999999},"min":{"1min":0.31,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":5.5259999999999998,"5min":6.0060000000000002,"15min":6.0060000000000002},"last":0.40699999999999997},{"interface":"front","average":{"1min":1.623,"5min":1.0600000000000001,"15min":0.72999999999999998},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":7.2080000000000002,"5min":8.8889999999999993,"15min":8.8889999999999993},"last":0.34499999999999997}]},{"osd":6,"last update":"Tue Sep 17 07:07:47 2024","interfaces":[{"interface":"back","average":{"1min":1.764,"5min":1.115,"15min":0.73399999999999999},"min":{"1min":0.36899999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":7.2539999999999996,"5min":8.843,"15min":8.843},"last":0.41999999999999998},{"interface":"front","average":{"1min":1.276,"5min":1.0669999999999999,"15min":0.753},"min":{"1min":0.38300000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":7.1130000000000004,"5min":7.1130000000000004,"15min":7.1130000000000004},"last":0.504}]},{"osd":7,"last update":"Tue Sep 17 07:07:07 2024","interfaces":[{"interface":"back","average":{"1min":1.5089999999999999,"5min":1.004,"15min":0.70999999999999996},"min":{"1min":0.38800000000000001,"5min":0.37,"15min":0.37},"max":{"1min":7.2320000000000002,"5min":8.9309999999999992,"15min":8.9309999999999992},"last":0.38500000000000001},{"interface":"front","average":{"1min":1.3120000000000001,"5min":1.0209999999999999,"15min":0.72099999999999997},"min":{"1min":0.372,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":3.5819999999999999,"5min":11.007999999999999,"15min":11.007999999999999},"last":0.438}]}]},{"osd":0,"up_from":8,"seq":34359738471,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":263768,"kb_used_data":34120,"kb_used_omap":1,"kb_used_meta":229630,"kb_avail":93481384,"statfs":{"total":95995035648,"available":95724937216,"internally_reserved":0,"allocated":34938880,"data_stored":34554912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":235141574},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 17 07:06:50 2024","interfaces":[{"interface":"back","average":{"1min":2.2610000000000001,"5min":1.0149999999999999,"15min":0.55700000000000005},"min":{"1min":0.22800000000000001,"5min":0.192,"15min":0.192},"max":{"1min":8.0259999999999998,"5min":8.0259999999999998,"15min":8.0259999999999998},"last":0.45500000000000002},{"interface":"front","average":{"1min":1.8220000000000001,"5min":0.96199999999999997,"15min":0.54800000000000004},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":7.8849999999999998,"5min":7.8849999999999998,"15min":7.8849999999999998},"last":0.35399999999999998}]},{"osd":2,"last update":"Tue Sep 17 07:07:24 2024","interfaces":[{"interface":"back","average":{"1min":2.1619999999999999,"5min":1.7829999999999999,"15min":0.84599999999999997},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":18.643999999999998,"5min":18.643999999999998,"15min":18.643999999999998},"last":0.38600000000000001},{"interface":"front","average":{"1min":2.839,"5min":1.5369999999999999,"15min":0.754},"min":{"1min":0.27200000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":17.972999999999999,"5min":17.972999999999999,"15min":17.972999999999999},"last":0.41599999999999998}]},{"osd":3,"last update":"Tue Sep 17 07:06:50 2024","interfaces":[{"interface":"back","average":{"1min":3.415,"5min":1.3340000000000001,"15min":0.73999999999999999},"min":{"1min":0.41599999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":18.629000000000001,"5min":18.629000000000001,"15min":18.629000000000001},"last":0.442},{"interface":"front","average":{"1min":3.972,"5min":1.627,"15min":0.83499999999999996},"min":{"1min":0.45600000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":22.358000000000001,"5min":22.358000000000001,"15min":22.358000000000001},"last":0.42799999999999999}]},{"osd":4,"last update":"Tue Sep 17 07:07:02 2024","interfaces":[{"interface":"back","average":{"1min":3.0979999999999999,"5min":1.474,"15min":0.86499999999999999},"min":{"1min":0.45200000000000001,"5min":0.316,"15min":0.316},"max":{"1min":18.568000000000001,"5min":18.568000000000001,"15min":18.568000000000001},"last":0.626},{"interface":"front","average":{"1min":2.4380000000000002,"5min":1.2090000000000001,"15min":0.76700000000000002},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":18.765000000000001,"5min":18.765000000000001,"15min":18.765000000000001},"last":0.46300000000000002}]},{"osd":5,"last update":"Tue Sep 17 07:07:29 2024","interfaces":[{"interface":"back","average":{"1min":2.3919999999999999,"5min":1.341,"15min":0.81200000000000006},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":18.614999999999998,"5min":18.614999999999998,"15min":18.614999999999998},"last":0.66300000000000003},{"interface":"front","average":{"1min":1.9870000000000001,"5min":1.23,"15min":0.76900000000000002},"min":{"1min":0.33600000000000002,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":18.791,"5min":18.791,"15min":18.791},"last":0.49099999999999999}]},{"osd":6,"last update":"Tue Sep 17 07:06:50 2024","interfaces":[{"interface":"back","average":{"1min":2.738,"5min":1.1850000000000001,"15min":0.75900000000000001},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":18.602,"5min":18.602,"15min":18.602},"last":0.54900000000000004},{"interface":"front","average":{"1min":3.198,"5min":1.3660000000000001,"15min":0.82099999999999995},"min":{"1min":0.48399999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":18.692,"5min":18.692,"15min":18.692},"last":0.64800000000000002}]},{"osd":7,"last update":"Tue Sep 17 07:07:13 2024","interfaces":[{"interface":"back","average":{"1min":4.6580000000000004,"5min":1.879,"15min":1.427},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":46.183,"5min":46.183,"15min":46.183},"last":0.51800000000000002},{"interface":"front","average":{"1min":4.8620000000000001,"5min":1.6819999999999999,"15min":0.93400000000000005},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":45.914999999999999,"5min":45.914999999999999,"15min":45.914999999999999},"last":0.56699999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T07:07:55.784 INFO:tasks.thrashosds.thrasher:Setting 5.1f to [0, 7, 5, 1, 3, 4] 2024-09-17T07:07:55.784 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '5.1f', '0', '7', '5', '1', '3', '4'] 2024-09-17T07:07:55.784 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd pg-upmap-items 5.1f 0 7 5 1 3 4 2024-09-17T07:07:55.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:55.786 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:55.786 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:07:55.833 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:07:55.888 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:07:55.913 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:55.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:55.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:07:56.015 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:07:56.041 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:56.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:56.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:07:56.054 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:07:55 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:07:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:07:56.142 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:07:56.167 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:56.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:56.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:07:56.269 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:07:56.293 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:56.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:56.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:07:56.366 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:56 smithi100 ceph-mon[29092]: pgmap v246: 132 pgs: 132 active+clean; 582 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-09-17T07:07:56.366 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:56 smithi100 ceph-mon[39462]: pgmap v246: 132 pgs: 132 active+clean; 582 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-09-17T07:07:56.394 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:07:56.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:56.418 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:56.418 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:07:56.520 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:07:56.542 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:56.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:56.543 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:07:56.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:56 smithi135 ceph-mon[33238]: pgmap v246: 132 pgs: 132 active+clean; 582 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-09-17T07:07:56.644 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:07:56.656 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:07:56.669 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:56.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:56.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:07:56.771 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:07:56.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:56.793 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:56.794 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:07:56.894 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:07:56.920 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:56.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:56.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:07:57.022 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:07:57.044 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:57.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:57.045 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:07:57.146 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:07:57.167 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:57.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:57.168 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:07:57.271 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:07:57.292 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:57.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:57.293 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:07:57.394 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:07:57.415 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:57.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:57.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:07:57.517 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:07:57.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:57.541 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:57.542 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:07:57.642 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:07:57.663 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:57.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:57.664 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:07:57.765 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:07:57.791 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:57.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:57.792 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:07:57.893 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:07:57.916 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:57.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:57.917 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:07:58.017 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:07:58.041 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:58.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:58.042 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:07:58.143 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:07:58.166 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:58.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:58.167 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:07:58.268 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:07:58.288 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:58.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:58.289 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:07:58.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:58 smithi100 ceph-mon[29092]: pgmap v247: 132 pgs: 132 active+clean; 582 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-09-17T07:07:58.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:58 smithi100 ceph-mon[39462]: pgmap v247: 132 pgs: 132 active+clean; 582 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-09-17T07:07:58.390 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:07:58.410 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:58.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:58.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:07:58.512 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:07:58.539 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:58.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:58.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:07:58.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:58 smithi135 ceph-mon[33238]: pgmap v247: 132 pgs: 132 active+clean; 582 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-09-17T07:07:58.543 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:07:58 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:07:58.641 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:07:58.661 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:58.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:58.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:07:58.763 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:07:58.789 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:58.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:58.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:07:58.890 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:07:58.913 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:58.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:58.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:07:59.015 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:07:59.035 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:59.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:59.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:07:59.137 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:07:59.162 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:59.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:59.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:07:59.233 INFO:teuthology.orchestra.run.smithi100.stderr:pool 'unique_pool_1' created 2024-09-17T07:07:59.264 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:07:59.286 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:59.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:59.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:07:59.387 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:07:59.409 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:59.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:59.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:07:59.477 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:07:59 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2314858566' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-09-17T07:07:59.477 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:07:59 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2314858566' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-09-17T07:07:59.510 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:07:59.536 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:59.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:59.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:07:59.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:07:59 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2314858566' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-09-17T07:07:59.639 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:07:59.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:59.662 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:07:59.662 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:07:59.763 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:07:59.789 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:59.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:59.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:07:59.893 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:07:59.918 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:07:59.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:07:59.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:08:00.020 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:08:00.046 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:00.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:00.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:08:00.148 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:08:00.164 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:08:00.164 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:08:00.170 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:00.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:00.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:08:00.183 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1: {} 2024-09-17T07:08:00.184 INFO:teuthology.orchestra.run.smithi100.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:08:00.199 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:08:00.199 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:08:00.215 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:08:00.215 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:08:00.239 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:08:00.239 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:08:00.260 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:08:00.261 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:08:00.272 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:08:00.277 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:08:00.277 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:08:00.295 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:00.295 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:08:00.295 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:08:00.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:00.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:08:00.395 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:08:00.396 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:08:00.419 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:00.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:00.420 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:08:00.511 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:00 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:08:00.511 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:00 smithi100 ceph-mon[29092]: pgmap v248: 132 pgs: 132 active+clean; 582 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-17T07:08:00.511 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:00 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2314858566' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-09-17T07:08:00.511 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:00 smithi100 ceph-mon[29092]: osdmap e168: 8 total, 8 up, 8 in 2024-09-17T07:08:00.511 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:00 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2859196076' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:08:00.512 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:00 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:08:00.512 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:00 smithi100 ceph-mon[39462]: pgmap v248: 132 pgs: 132 active+clean; 582 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-17T07:08:00.512 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:00 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2314858566' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-09-17T07:08:00.512 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:00 smithi100 ceph-mon[39462]: osdmap e168: 8 total, 8 up, 8 in 2024-09-17T07:08:00.512 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:00 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2859196076' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:08:00.521 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:08:00.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:00 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:08:00.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:00 smithi135 ceph-mon[33238]: pgmap v248: 132 pgs: 132 active+clean; 582 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-17T07:08:00.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:00 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2314858566' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-09-17T07:08:00.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:00 smithi135 ceph-mon[33238]: osdmap e168: 8 total, 8 up, 8 in 2024-09-17T07:08:00.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:00 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2859196076' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:08:00.544 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:00.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:00.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:08:00.646 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:08:00.669 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:00.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:00.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:08:00.771 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:08:00.793 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:00.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:00.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:08:00.894 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:08:00.916 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:00.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:00.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:08:00.974 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd pool application enable unique_pool_1 rados --yes-i-really-mean-it || true 2024-09-17T07:08:01.018 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:08:01.040 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:01.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:01.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:08:01.141 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:08:01.163 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:01.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:01.164 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:08:01.265 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:08:01.290 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:01.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:01.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:08:01.392 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:08:01.415 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:01.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:01.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:08:01.517 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:08:01.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:01 smithi135 ceph-mon[33238]: osdmap e169: 8 total, 8 up, 8 in 2024-09-17T07:08:01.543 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:01.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:01.544 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:08:01.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:01 smithi100 ceph-mon[29092]: osdmap e169: 8 total, 8 up, 8 in 2024-09-17T07:08:01.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:01 smithi100 ceph-mon[39462]: osdmap e169: 8 total, 8 up, 8 in 2024-09-17T07:08:01.645 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:08:01.666 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:01.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:01.667 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:08:01.768 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:08:01.793 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:01.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:01.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:08:01.895 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:08:01.921 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:01.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:01.922 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:08:02.023 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:08:02.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:02.049 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:02.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:08:02.150 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:08:02.173 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:02.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:02.174 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:08:02.275 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:08:02.302 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:02.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:02.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:08:02.404 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:08:02.428 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:02.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:02.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:08:02.529 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:08:02.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:02 smithi135 ceph-mon[33238]: pgmap v251: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:08:02.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:02 smithi135 ceph-mon[33238]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-17T07:08:02.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:02 smithi135 ceph-mon[33238]: osdmap e170: 8 total, 8 up, 8 in 2024-09-17T07:08:02.551 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:02.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:02.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:08:02.594 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:02 smithi100 ceph-mon[29092]: pgmap v251: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:08:02.594 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:02 smithi100 ceph-mon[29092]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-17T07:08:02.594 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:02 smithi100 ceph-mon[29092]: osdmap e170: 8 total, 8 up, 8 in 2024-09-17T07:08:02.595 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:02 smithi100 ceph-mon[39462]: pgmap v251: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:08:02.595 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:02 smithi100 ceph-mon[39462]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-17T07:08:02.595 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:02 smithi100 ceph-mon[39462]: osdmap e170: 8 total, 8 up, 8 in 2024-09-17T07:08:02.653 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:08:02.674 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:02.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:02.675 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:08:02.776 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:08:02.797 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:02.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:02.798 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:08:02.899 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:08:02.925 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:02.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:02.926 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:08:03.026 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:08:03.048 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:03.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:03.049 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:08:03.067 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-17T07:08:03.067 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:08:03.150 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:08:03.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:03.176 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:03.176 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:08:03.261 INFO:teuthology.orchestra.run.smithi100.stderr:noscrub is unset 2024-09-17T07:08:03.277 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:08:03.298 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:03.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:03.299 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:08:03.399 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:08:03.423 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:03.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:03.424 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:08:03.525 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:08:03.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:03 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1477828923' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:08:03.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:03 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:08:03.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:03 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:08:03.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:03 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:08:03.546 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:03.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:03.547 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:08:03.606 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:03 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1477828923' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:08:03.607 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:03 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:08:03.607 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:03 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:08:03.607 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:03 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:08:03.607 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:03 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1477828923' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:08:03.607 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:03 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:08:03.607 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:03 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:08:03.607 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:03 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:08:03.648 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:08:03.669 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:03.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:03.670 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:08:03.770 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:08:03.791 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:03.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:03.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:08:03.892 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:08:03.954 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:03.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:03.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:08:04.055 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:08:04.076 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:04.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:04.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:08:04.178 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:08:04.203 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:04.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:04.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:08:04.305 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:08:04.332 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:04.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:04.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:08:04.378 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:08:04.434 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:08:04.459 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:04.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:04.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:08:04.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:04 smithi135 ceph-mon[33238]: pgmap v253: 148 pgs: 12 unknown, 136 active+clean; 582 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:08:04.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:04 smithi135 ceph-mon[33238]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:08:04.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:04 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T07:08:04.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:04 smithi135 ceph-mon[33238]: osdmap e171: 8 total, 8 up, 8 in 2024-09-17T07:08:04.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:04 smithi135 ceph-mon[33238]: 5.1 scrub starts 2024-09-17T07:08:04.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:04 smithi135 ceph-mon[33238]: 5.1 scrub ok 2024-09-17T07:08:04.561 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:08:04.582 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:04.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:04.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:08:04.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:04 smithi100 ceph-mon[39462]: pgmap v253: 148 pgs: 12 unknown, 136 active+clean; 582 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:08:04.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:04 smithi100 ceph-mon[39462]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:08:04.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:04 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T07:08:04.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:04 smithi100 ceph-mon[39462]: osdmap e171: 8 total, 8 up, 8 in 2024-09-17T07:08:04.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:04 smithi100 ceph-mon[39462]: 5.1 scrub starts 2024-09-17T07:08:04.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:04 smithi100 ceph-mon[39462]: 5.1 scrub ok 2024-09-17T07:08:04.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:04 smithi100 ceph-mon[29092]: pgmap v253: 148 pgs: 12 unknown, 136 active+clean; 582 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:08:04.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:04 smithi100 ceph-mon[29092]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:08:04.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:04 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T07:08:04.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:04 smithi100 ceph-mon[29092]: osdmap e171: 8 total, 8 up, 8 in 2024-09-17T07:08:04.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:04 smithi100 ceph-mon[29092]: 5.1 scrub starts 2024-09-17T07:08:04.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:04 smithi100 ceph-mon[29092]: 5.1 scrub ok 2024-09-17T07:08:04.684 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:08:04.705 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:04.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:04.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:08:04.807 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:08:04.830 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-17T07:08:04.834 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:04.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:04.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:08:04.935 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:08:04.957 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:04.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:04.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:08:05.059 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:08:05.085 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:05.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:05.086 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:08:05.186 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:08:05.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:05.212 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:05.212 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:08:05.311 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:08:05.313 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:08:05.340 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:05.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:05.341 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:08:05.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:05 smithi100 ceph-mon[29092]: 5.11 scrub starts 2024-09-17T07:08:05.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:05 smithi100 ceph-mon[29092]: 5.11 scrub ok 2024-09-17T07:08:05.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:05 smithi100 ceph-mon[29092]: 2.0 scrub starts 2024-09-17T07:08:05.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:05 smithi100 ceph-mon[29092]: 2.0 scrub ok 2024-09-17T07:08:05.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:05 smithi100 ceph-mon[29092]: 5.5 scrub starts 2024-09-17T07:08:05.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:05 smithi100 ceph-mon[29092]: 5.5 scrub ok 2024-09-17T07:08:05.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:05 smithi100 ceph-mon[29092]: 2.6 scrub starts 2024-09-17T07:08:05.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:05 smithi100 ceph-mon[29092]: 2.6 scrub ok 2024-09-17T07:08:05.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:05 smithi100 ceph-mon[39462]: 5.11 scrub starts 2024-09-17T07:08:05.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:05 smithi100 ceph-mon[39462]: 5.11 scrub ok 2024-09-17T07:08:05.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:05 smithi100 ceph-mon[39462]: 2.0 scrub starts 2024-09-17T07:08:05.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:05 smithi100 ceph-mon[39462]: 2.0 scrub ok 2024-09-17T07:08:05.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:05 smithi100 ceph-mon[39462]: 5.5 scrub starts 2024-09-17T07:08:05.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:05 smithi100 ceph-mon[39462]: 5.5 scrub ok 2024-09-17T07:08:05.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:05 smithi100 ceph-mon[39462]: 2.6 scrub starts 2024-09-17T07:08:05.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:05 smithi100 ceph-mon[39462]: 2.6 scrub ok 2024-09-17T07:08:05.443 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:08:05.464 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:05.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:05.465 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:08:05.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:05 smithi135 ceph-mon[33238]: 5.11 scrub starts 2024-09-17T07:08:05.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:05 smithi135 ceph-mon[33238]: 5.11 scrub ok 2024-09-17T07:08:05.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:05 smithi135 ceph-mon[33238]: 2.0 scrub starts 2024-09-17T07:08:05.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:05 smithi135 ceph-mon[33238]: 2.0 scrub ok 2024-09-17T07:08:05.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:05 smithi135 ceph-mon[33238]: 5.5 scrub starts 2024-09-17T07:08:05.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:05 smithi135 ceph-mon[33238]: 5.5 scrub ok 2024-09-17T07:08:05.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:05 smithi135 ceph-mon[33238]: 2.6 scrub starts 2024-09-17T07:08:05.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:05 smithi135 ceph-mon[33238]: 2.6 scrub ok 2024-09-17T07:08:05.566 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:08:05.591 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:05.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:05.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:08:05.693 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:08:05.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:05.715 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:05.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:08:05.816 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:08:05.841 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:05.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:05.842 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:08:05.860 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:08:05.894 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-17T07:08:05.944 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:08:05.969 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:05.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:05.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:08:06.071 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:08:06.092 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:06.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:06.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:08:06.114 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:08:05 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:08:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:08:06.194 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:08:06.219 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:06.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:06.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:08:06.321 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:08:06.348 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:06.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:06.349 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:08:06.365 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:06 smithi100 ceph-mon[29092]: 2.11 scrub starts 2024-09-17T07:08:06.365 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:06 smithi100 ceph-mon[29092]: 2.11 scrub ok 2024-09-17T07:08:06.365 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:06 smithi100 ceph-mon[29092]: 2.16 scrub starts 2024-09-17T07:08:06.365 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:06 smithi100 ceph-mon[29092]: 2.16 scrub ok 2024-09-17T07:08:06.365 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:06 smithi100 ceph-mon[29092]: 5.4 scrub starts 2024-09-17T07:08:06.365 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:06 smithi100 ceph-mon[29092]: 5.4 scrub ok 2024-09-17T07:08:06.366 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:06 smithi100 ceph-mon[29092]: pgmap v255: 148 pgs: 148 active+clean; 582 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 869 B/s rd, 0 op/s 2024-09-17T07:08:06.366 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:06 smithi100 ceph-mon[29092]: 4.7 scrub starts 2024-09-17T07:08:06.366 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:06 smithi100 ceph-mon[29092]: 4.7 scrub ok 2024-09-17T07:08:06.366 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:06 smithi100 ceph-mon[39462]: 2.11 scrub starts 2024-09-17T07:08:06.366 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:06 smithi100 ceph-mon[39462]: 2.11 scrub ok 2024-09-17T07:08:06.366 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:06 smithi100 ceph-mon[39462]: 2.16 scrub starts 2024-09-17T07:08:06.366 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:06 smithi100 ceph-mon[39462]: 2.16 scrub ok 2024-09-17T07:08:06.366 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:06 smithi100 ceph-mon[39462]: 5.4 scrub starts 2024-09-17T07:08:06.366 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:06 smithi100 ceph-mon[39462]: 5.4 scrub ok 2024-09-17T07:08:06.366 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:06 smithi100 ceph-mon[39462]: pgmap v255: 148 pgs: 148 active+clean; 582 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 869 B/s rd, 0 op/s 2024-09-17T07:08:06.367 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:06 smithi100 ceph-mon[39462]: 4.7 scrub starts 2024-09-17T07:08:06.367 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:06 smithi100 ceph-mon[39462]: 4.7 scrub ok 2024-09-17T07:08:06.449 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:08:06.473 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:06.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:06.474 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:08:06.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:06 smithi135 ceph-mon[33238]: 2.11 scrub starts 2024-09-17T07:08:06.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:06 smithi135 ceph-mon[33238]: 2.11 scrub ok 2024-09-17T07:08:06.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:06 smithi135 ceph-mon[33238]: 2.16 scrub starts 2024-09-17T07:08:06.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:06 smithi135 ceph-mon[33238]: 2.16 scrub ok 2024-09-17T07:08:06.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:06 smithi135 ceph-mon[33238]: 5.4 scrub starts 2024-09-17T07:08:06.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:06 smithi135 ceph-mon[33238]: 5.4 scrub ok 2024-09-17T07:08:06.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:06 smithi135 ceph-mon[33238]: pgmap v255: 148 pgs: 148 active+clean; 582 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 869 B/s rd, 0 op/s 2024-09-17T07:08:06.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:06 smithi135 ceph-mon[33238]: 4.7 scrub starts 2024-09-17T07:08:06.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:06 smithi135 ceph-mon[33238]: 4.7 scrub ok 2024-09-17T07:08:06.575 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:08:06.597 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:06.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:06.598 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:08:06.699 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:08:06.720 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:06.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:06.721 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:08:06.821 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:08:06.842 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:06.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:06.843 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:08:06.944 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:08:06.965 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:06.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:06.966 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:08:07.066 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:08:07.087 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:07.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:07.088 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:08:07.189 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:08:07.209 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:07.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:07.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:08:07.267 INFO:teuthology.orchestra.run.smithi100.stderr:set 5.1f pg_upmap_items mapping to [0->7,5->1,3->4] 2024-09-17T07:08:07.278 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:07 smithi100 ceph-mon[29092]: 2.8 scrub starts 2024-09-17T07:08:07.278 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:07 smithi100 ceph-mon[29092]: 2.8 scrub ok 2024-09-17T07:08:07.278 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:07 smithi100 ceph-mon[29092]: 4.1 scrub starts 2024-09-17T07:08:07.278 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:07 smithi100 ceph-mon[29092]: 4.1 scrub ok 2024-09-17T07:08:07.278 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:07 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/3700305702' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "5.1f", "id": [0, 7, 5, 1, 3, 4]}]: dispatch 2024-09-17T07:08:07.279 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:07 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "5.1f", "id": [0, 7, 5, 1, 3, 4]}]: dispatch 2024-09-17T07:08:07.279 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:07 smithi100 ceph-mon[29092]: pgmap v256: 148 pgs: 148 active+clean; 582 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 743 B/s rd, 0 op/s 2024-09-17T07:08:07.279 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:07 smithi100 ceph-mon[39462]: 2.8 scrub starts 2024-09-17T07:08:07.279 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:07 smithi100 ceph-mon[39462]: 2.8 scrub ok 2024-09-17T07:08:07.279 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:07 smithi100 ceph-mon[39462]: 4.1 scrub starts 2024-09-17T07:08:07.279 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:07 smithi100 ceph-mon[39462]: 4.1 scrub ok 2024-09-17T07:08:07.279 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:07 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/3700305702' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "5.1f", "id": [0, 7, 5, 1, 3, 4]}]: dispatch 2024-09-17T07:08:07.279 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:07 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "5.1f", "id": [0, 7, 5, 1, 3, 4]}]: dispatch 2024-09-17T07:08:07.279 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:07 smithi100 ceph-mon[39462]: pgmap v256: 148 pgs: 148 active+clean; 582 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 743 B/s rd, 0 op/s 2024-09-17T07:08:07.312 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:08:07.336 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:07.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:07.337 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:08:07.437 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:08:07.461 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:07.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:07.462 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:08:07.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:07 smithi135 ceph-mon[33238]: 2.8 scrub starts 2024-09-17T07:08:07.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:07 smithi135 ceph-mon[33238]: 2.8 scrub ok 2024-09-17T07:08:07.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:07 smithi135 ceph-mon[33238]: 4.1 scrub starts 2024-09-17T07:08:07.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:07 smithi135 ceph-mon[33238]: 4.1 scrub ok 2024-09-17T07:08:07.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:07 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/3700305702' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "5.1f", "id": [0, 7, 5, 1, 3, 4]}]: dispatch 2024-09-17T07:08:07.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:07 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "5.1f", "id": [0, 7, 5, 1, 3, 4]}]: dispatch 2024-09-17T07:08:07.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:07 smithi135 ceph-mon[33238]: pgmap v256: 148 pgs: 148 active+clean; 582 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 743 B/s rd, 0 op/s 2024-09-17T07:08:07.563 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:08:07.587 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:07.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:07.587 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:08:07.688 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:08:07.712 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:07.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:07.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:08:07.813 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:08:07.837 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:07.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:07.838 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:08:07.938 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:08:07.961 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:07.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:07.962 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:08:08.015 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:08:08.063 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:08:08.085 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:08.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:08.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:08:08.187 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:08:08.209 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:08.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:08.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:08:08.311 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:08:08.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:08.332 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:08.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:08:08.433 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:08:08.458 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:08.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:08.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:08:08.543 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:08:08 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:08:08.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:08 smithi135 ceph-mon[33238]: 5.16 scrub starts 2024-09-17T07:08:08.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:08 smithi135 ceph-mon[33238]: 5.16 scrub ok 2024-09-17T07:08:08.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:08 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "5.1f", "id": [0, 7, 5, 1, 3, 4]}]': finished 2024-09-17T07:08:08.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:08 smithi135 ceph-mon[33238]: osdmap e172: 8 total, 8 up, 8 in 2024-09-17T07:08:08.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:08 smithi135 ceph-mon[33238]: 6.b scrub starts 2024-09-17T07:08:08.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:08 smithi135 ceph-mon[33238]: 6.b scrub ok 2024-09-17T07:08:08.559 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:08:08.580 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:08.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:08.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:08:08.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:08 smithi100 ceph-mon[29092]: 5.16 scrub starts 2024-09-17T07:08:08.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:08 smithi100 ceph-mon[29092]: 5.16 scrub ok 2024-09-17T07:08:08.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:08 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "5.1f", "id": [0, 7, 5, 1, 3, 4]}]': finished 2024-09-17T07:08:08.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:08 smithi100 ceph-mon[29092]: osdmap e172: 8 total, 8 up, 8 in 2024-09-17T07:08:08.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:08 smithi100 ceph-mon[29092]: 6.b scrub starts 2024-09-17T07:08:08.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:08 smithi100 ceph-mon[29092]: 6.b scrub ok 2024-09-17T07:08:08.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:08 smithi100 ceph-mon[39462]: 5.16 scrub starts 2024-09-17T07:08:08.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:08 smithi100 ceph-mon[39462]: 5.16 scrub ok 2024-09-17T07:08:08.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:08 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "5.1f", "id": [0, 7, 5, 1, 3, 4]}]': finished 2024-09-17T07:08:08.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:08 smithi100 ceph-mon[39462]: osdmap e172: 8 total, 8 up, 8 in 2024-09-17T07:08:08.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:08 smithi100 ceph-mon[39462]: 6.b scrub starts 2024-09-17T07:08:08.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:08 smithi100 ceph-mon[39462]: 6.b scrub ok 2024-09-17T07:08:08.682 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:08:08.704 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:08.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:08.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:08:08.805 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:08:08.828 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:08.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:08.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:08:08.930 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:08:08.951 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:08.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:08.952 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:08:09.053 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:08:09.074 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:09.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:09.075 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:08:09.175 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:08:09.196 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:09.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:09.197 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:08:09.266 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:09 smithi100 ceph-mon[29092]: 4.1f scrub starts 2024-09-17T07:08:09.266 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:09 smithi100 ceph-mon[29092]: 4.1f scrub ok 2024-09-17T07:08:09.266 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:09 smithi100 ceph-mon[29092]: 4.6 scrub starts 2024-09-17T07:08:09.266 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:09 smithi100 ceph-mon[29092]: 4.6 scrub ok 2024-09-17T07:08:09.267 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:09 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:08:09.267 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:09 smithi100 ceph-mon[29092]: 6.6 scrub starts 2024-09-17T07:08:09.267 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:09 smithi100 ceph-mon[29092]: 6.6 scrub ok 2024-09-17T07:08:09.297 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:08:09.319 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:09.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:09.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:08:09.421 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:08:09.442 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:09.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:09.443 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:08:09.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:09 smithi135 ceph-mon[33238]: 4.1f scrub starts 2024-09-17T07:08:09.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:09 smithi135 ceph-mon[33238]: 4.1f scrub ok 2024-09-17T07:08:09.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:09 smithi135 ceph-mon[33238]: 4.6 scrub starts 2024-09-17T07:08:09.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:09 smithi135 ceph-mon[33238]: 4.6 scrub ok 2024-09-17T07:08:09.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:09 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:08:09.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:09 smithi135 ceph-mon[33238]: 6.6 scrub starts 2024-09-17T07:08:09.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:09 smithi135 ceph-mon[33238]: 6.6 scrub ok 2024-09-17T07:08:09.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:09 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:08:09.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:09 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:08:09.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:09 smithi135 ceph-mon[33238]: pgmap v258: 148 pgs: 148 active+clean; 582 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-17T07:08:09.544 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:08:09.565 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:09.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:09.566 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:08:09.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:09 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:08:09.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:09 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:08:09.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:09 smithi100 ceph-mon[29092]: pgmap v258: 148 pgs: 148 active+clean; 582 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-17T07:08:09.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:09 smithi100 ceph-mon[39462]: 4.1f scrub starts 2024-09-17T07:08:09.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:09 smithi100 ceph-mon[39462]: 4.1f scrub ok 2024-09-17T07:08:09.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:09 smithi100 ceph-mon[39462]: 4.6 scrub starts 2024-09-17T07:08:09.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:09 smithi100 ceph-mon[39462]: 4.6 scrub ok 2024-09-17T07:08:09.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:09 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:08:09.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:09 smithi100 ceph-mon[39462]: 6.6 scrub starts 2024-09-17T07:08:09.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:09 smithi100 ceph-mon[39462]: 6.6 scrub ok 2024-09-17T07:08:09.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:09 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:08:09.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:09 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:08:09.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:09 smithi100 ceph-mon[39462]: pgmap v258: 148 pgs: 148 active+clean; 582 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-17T07:08:09.666 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:08:09.688 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:09.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:09.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:08:09.789 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:08:09.811 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:09.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:09.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:08:09.913 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:08:09.935 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:09.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:09.936 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:08:10.037 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:08:10.058 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:10.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:10.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:08:10.160 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:08:10.182 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:10.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:10.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:08:10.284 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:08:10.305 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:10.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:10.306 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:08:10.330 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:10 smithi100 ceph-mon[39462]: osdmap e173: 8 total, 8 up, 8 in 2024-09-17T07:08:10.331 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:10 smithi100 ceph-mon[29092]: osdmap e173: 8 total, 8 up, 8 in 2024-09-17T07:08:10.407 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:08:10.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:10.430 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:10.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:08:10.531 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:08:10.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:10 smithi135 ceph-mon[33238]: osdmap e173: 8 total, 8 up, 8 in 2024-09-17T07:08:10.552 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:10.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:10.553 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:08:10.654 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:08:10.676 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:10.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:10.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:08:10.777 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:08:10.799 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:10.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:10.800 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:08:10.901 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:08:10.923 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:10.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:10.924 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:08:11.025 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:08:11.046 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:11.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:11.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:08:11.147 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:08:11.170 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:11.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:11.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:08:11.272 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:08:11.295 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:11.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:11.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:08:11.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:11 smithi100 ceph-mon[29092]: 5.e scrub starts 2024-09-17T07:08:11.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:11 smithi100 ceph-mon[29092]: 5.e scrub ok 2024-09-17T07:08:11.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:11 smithi100 ceph-mon[29092]: 5.8 scrub starts 2024-09-17T07:08:11.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:11 smithi100 ceph-mon[29092]: 5.8 scrub ok 2024-09-17T07:08:11.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:11 smithi100 ceph-mon[29092]: 6.11 scrub starts 2024-09-17T07:08:11.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:11 smithi100 ceph-mon[29092]: 6.11 scrub ok 2024-09-17T07:08:11.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:11 smithi100 ceph-mon[29092]: pgmap v260: 148 pgs: 148 active+clean; 582 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 651 B/s rd, 0 op/s 2024-09-17T07:08:11.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:11 smithi100 ceph-mon[39462]: 5.e scrub starts 2024-09-17T07:08:11.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:11 smithi100 ceph-mon[39462]: 5.e scrub ok 2024-09-17T07:08:11.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:11 smithi100 ceph-mon[39462]: 5.8 scrub starts 2024-09-17T07:08:11.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:11 smithi100 ceph-mon[39462]: 5.8 scrub ok 2024-09-17T07:08:11.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:11 smithi100 ceph-mon[39462]: 6.11 scrub starts 2024-09-17T07:08:11.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:11 smithi100 ceph-mon[39462]: 6.11 scrub ok 2024-09-17T07:08:11.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:11 smithi100 ceph-mon[39462]: pgmap v260: 148 pgs: 148 active+clean; 582 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 651 B/s rd, 0 op/s 2024-09-17T07:08:11.397 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:08:11.420 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:11.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:11.421 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:08:11.521 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:08:11.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:11 smithi135 ceph-mon[33238]: 5.e scrub starts 2024-09-17T07:08:11.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:11 smithi135 ceph-mon[33238]: 5.e scrub ok 2024-09-17T07:08:11.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:11 smithi135 ceph-mon[33238]: 5.8 scrub starts 2024-09-17T07:08:11.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:11 smithi135 ceph-mon[33238]: 5.8 scrub ok 2024-09-17T07:08:11.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:11 smithi135 ceph-mon[33238]: 6.11 scrub starts 2024-09-17T07:08:11.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:11 smithi135 ceph-mon[33238]: 6.11 scrub ok 2024-09-17T07:08:11.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:11 smithi135 ceph-mon[33238]: pgmap v260: 148 pgs: 148 active+clean; 582 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 651 B/s rd, 0 op/s 2024-09-17T07:08:11.545 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:11.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:11.547 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:08:11.648 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:08:11.672 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:11.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:11.673 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:08:11.774 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:08:11.795 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:11.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:11.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:08:11.897 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:08:11.922 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:11.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:11.923 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:08:11.993 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:08:11.993 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:08:12.006 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1: {} 2024-09-17T07:08:12.007 INFO:teuthology.orchestra.run.smithi100.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:08:12.023 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:08:12.026 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:08:12.026 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:08:12.041 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:08:12.042 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:08:12.044 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:12.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:12.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:08:12.058 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:08:12.058 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:08:12.075 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:08:12.075 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:08:12.092 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:08:12.092 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:08:12.108 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:08:12.108 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:08:12.147 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:08:12.168 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:12.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:12.169 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:08:12.270 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-17T07:08:12.270 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-17T07:08:12.270 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-17T07:08:12.270 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-17T07:08:12.270 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.270 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.14805.0:54890 7.14 7:2f55ae45:::benchmark_data_smithi100_78556_object54889:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e111)", 2024-09-17T07:08:12.270 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:04:57.343812+0000", 2024-09-17T07:08:12.270 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 194.90234826599999, 2024-09-17T07:08:12.270 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0203684420000001, 2024-09-17T07:08:12.270 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:08:12.270 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-17T07:08:12.271 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:08:12.271 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.14805", 2024-09-17T07:08:12.271 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:08:12.271 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 54890 2024-09-17T07:08:12.271 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.271 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:08:12.271 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.271 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:08:12.271 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.343812+0000", 2024-09-17T07:08:12.271 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:12.271 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.271 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.272 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:08:12.272 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.343812+0000", 2024-09-17T07:08:12.272 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:12.272 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.272 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.272 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:08:12.272 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.343811+0000", 2024-09-17T07:08:12.272 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-17T07:08:12.272 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.272 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.272 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:08:12.272 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.343835+0000", 2024-09-17T07:08:12.273 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3618e-05 2024-09-17T07:08:12.273 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.273 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.273 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:08:12.273 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.343841+0000", 2024-09-17T07:08:12.273 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9440000000000002e-06 2024-09-17T07:08:12.273 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.273 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.273 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:08:12.273 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.343845+0000", 2024-09-17T07:08:12.273 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5870000000000003e-06 2024-09-17T07:08:12.273 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.274 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.274 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:08:12.274 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.485155+0000", 2024-09-17T07:08:12.274 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.141309198 2024-09-17T07:08:12.274 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.274 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.274 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-17T07:08:12.274 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.485163+0000", 2024-09-17T07:08:12.274 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0909999999999993e-06 2024-09-17T07:08:12.274 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.274 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.274 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:08:12.275 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:58.356461+0000", 2024-09-17T07:08:12.275 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.871298605 2024-09-17T07:08:12.275 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.275 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.275 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for degraded object", 2024-09-17T07:08:12.275 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:58.359041+0000", 2024-09-17T07:08:12.275 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0025792749999999998 2024-09-17T07:08:12.275 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.275 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.275 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:08:12.275 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:58.364109+0000", 2024-09-17T07:08:12.276 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.005068197 2024-09-17T07:08:12.276 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.276 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.276 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:08:12.276 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:58.364180+0000", 2024-09-17T07:08:12.276 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1373999999999998e-05 2024-09-17T07:08:12.276 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:12.276 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:12.276 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:12.276 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.276 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.277 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.14805.0:54891 7.16 7:691d1cb1:::benchmark_data_smithi100_78556_object54890:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e111)", 2024-09-17T07:08:12.277 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:04:57.343856+0000", 2024-09-17T07:08:12.277 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 194.90230364600001, 2024-09-17T07:08:12.277 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0321880130000001, 2024-09-17T07:08:12.277 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:08:12.277 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-17T07:08:12.277 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:08:12.277 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.14805", 2024-09-17T07:08:12.277 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:08:12.277 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 54891 2024-09-17T07:08:12.277 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.278 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:08:12.278 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.278 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:08:12.278 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.343856+0000", 2024-09-17T07:08:12.278 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:12.278 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.278 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.278 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:08:12.278 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.343856+0000", 2024-09-17T07:08:12.278 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:12.278 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.278 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.278 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:08:12.279 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.343856+0000", 2024-09-17T07:08:12.279 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-17T07:08:12.279 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.279 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.279 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:08:12.279 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.343879+0000", 2024-09-17T07:08:12.280 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.302e-05 2024-09-17T07:08:12.280 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.280 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.280 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:08:12.280 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.343885+0000", 2024-09-17T07:08:12.280 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2339999999999999e-06 2024-09-17T07:08:12.280 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.280 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.281 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:08:12.281 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.343890+0000", 2024-09-17T07:08:12.281 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8359999999999999e-06 2024-09-17T07:08:12.281 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.281 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.281 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:08:12.281 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.486174+0000", 2024-09-17T07:08:12.281 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.14228400799999999 2024-09-17T07:08:12.281 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.281 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.281 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-17T07:08:12.281 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.486189+0000", 2024-09-17T07:08:12.282 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4939999999999999e-05 2024-09-17T07:08:12.282 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.282 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.282 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:08:12.282 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:58.375979+0000", 2024-09-17T07:08:12.282 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.88978975800000004 2024-09-17T07:08:12.282 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.282 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.282 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:08:12.282 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:58.376044+0000", 2024-09-17T07:08:12.282 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5617999999999994e-05 2024-09-17T07:08:12.282 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:12.282 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:12.283 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:12.283 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.283 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.283 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.14805.0:54895 7.6 7:63ba05c1:::benchmark_data_smithi100_78556_object54894:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e111)", 2024-09-17T07:08:12.283 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:04:57.343895+0000", 2024-09-17T07:08:12.283 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 194.90226498300001, 2024-09-17T07:08:12.283 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0193373509999999, 2024-09-17T07:08:12.283 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:08:12.283 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-17T07:08:12.283 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:08:12.283 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.14805", 2024-09-17T07:08:12.283 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:08:12.284 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 54895 2024-09-17T07:08:12.284 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.284 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:08:12.284 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.284 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:08:12.284 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.343895+0000", 2024-09-17T07:08:12.284 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:12.284 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.284 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.284 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:08:12.284 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.343895+0000", 2024-09-17T07:08:12.284 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:12.285 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.285 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.285 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:08:12.285 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.343895+0000", 2024-09-17T07:08:12.285 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-17T07:08:12.285 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.285 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.285 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:08:12.285 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.343917+0000", 2024-09-17T07:08:12.285 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1951e-05 2024-09-17T07:08:12.285 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.285 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.285 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:08:12.285 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.343922+0000", 2024-09-17T07:08:12.286 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4430000000000002e-06 2024-09-17T07:08:12.286 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.286 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.286 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:08:12.286 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.343926+0000", 2024-09-17T07:08:12.286 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.296e-06 2024-09-17T07:08:12.286 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.286 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.286 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:08:12.286 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.344913+0000", 2024-09-17T07:08:12.286 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00098710500000000006 2024-09-17T07:08:12.286 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.286 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.286 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op must wait for map", 2024-09-17T07:08:12.286 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.344918+0000", 2024-09-17T07:08:12.287 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0159999999999999e-06 2024-09-17T07:08:12.287 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.287 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.287 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:08:12.287 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.482949+0000", 2024-09-17T07:08:12.287 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.13803019799999999 2024-09-17T07:08:12.287 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.287 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.287 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-17T07:08:12.287 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.482961+0000", 2024-09-17T07:08:12.287 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2052000000000001e-05 2024-09-17T07:08:12.287 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.287 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.287 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:08:12.288 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:58.363163+0000", 2024-09-17T07:08:12.288 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.88020196299999998 2024-09-17T07:08:12.288 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.288 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.288 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:08:12.288 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:58.363232+0000", 2024-09-17T07:08:12.288 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9688000000000001e-05 2024-09-17T07:08:12.288 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:12.288 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:12.288 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:12.288 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.288 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.289 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.14805.0:54896 7.16 7:68fa3457:::benchmark_data_smithi100_78556_object54895:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e111)", 2024-09-17T07:08:12.289 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:04:57.343930+0000", 2024-09-17T07:08:12.289 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 194.90222969999999, 2024-09-17T07:08:12.289 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.035847894, 2024-09-17T07:08:12.289 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:08:12.289 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-17T07:08:12.289 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:08:12.289 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.14805", 2024-09-17T07:08:12.289 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:08:12.289 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 54896 2024-09-17T07:08:12.289 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.289 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:08:12.289 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.289 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:08:12.290 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.343930+0000", 2024-09-17T07:08:12.290 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:12.290 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.290 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.290 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:08:12.290 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.343930+0000", 2024-09-17T07:08:12.290 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:12.290 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.290 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.290 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:08:12.290 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.343930+0000", 2024-09-17T07:08:12.290 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-17T07:08:12.290 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.291 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.291 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:08:12.291 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.343994+0000", 2024-09-17T07:08:12.291 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4443999999999995e-05 2024-09-17T07:08:12.291 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.291 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.291 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:08:12.291 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.344000+0000", 2024-09-17T07:08:12.291 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7270000000000004e-06 2024-09-17T07:08:12.291 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.291 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.291 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:08:12.291 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.344005+0000", 2024-09-17T07:08:12.291 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.527e-06 2024-09-17T07:08:12.292 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.292 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.292 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:08:12.292 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.486209+0000", 2024-09-17T07:08:12.292 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.14220397200000001 2024-09-17T07:08:12.292 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.292 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.292 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-17T07:08:12.292 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.486214+0000", 2024-09-17T07:08:12.292 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8250000000000001e-06 2024-09-17T07:08:12.292 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.293 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.293 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:08:12.293 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:58.376324+0000", 2024-09-17T07:08:12.293 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.89010998200000002 2024-09-17T07:08:12.293 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.293 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.293 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for degraded object", 2024-09-17T07:08:12.293 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:58.377026+0000", 2024-09-17T07:08:12.293 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00070122200000000004 2024-09-17T07:08:12.293 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.293 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.293 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:08:12.293 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:58.379701+0000", 2024-09-17T07:08:12.294 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0026749579999999998 2024-09-17T07:08:12.294 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.294 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.294 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:08:12.294 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:58.379778+0000", 2024-09-17T07:08:12.294 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7606000000000002e-05 2024-09-17T07:08:12.294 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:12.294 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:12.294 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:12.294 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.294 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.294 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.14805.0:54905 7.14 7:2f70dbc4:::benchmark_data_smithi100_78556_object54904:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-09-17T07:08:12.294 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:04:57.346340+0000", 2024-09-17T07:08:12.295 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 194.89982034799999, 2024-09-17T07:08:12.295 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0173668300000001, 2024-09-17T07:08:12.295 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:08:12.295 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:08:12.295 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:08:12.295 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.14805", 2024-09-17T07:08:12.295 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:08:12.295 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 54905 2024-09-17T07:08:12.295 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.295 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:08:12.295 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.295 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:08:12.295 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.346340+0000", 2024-09-17T07:08:12.295 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:12.295 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.296 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.296 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:08:12.296 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.346340+0000", 2024-09-17T07:08:12.296 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:12.296 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.296 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.296 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:08:12.296 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.346338+0000", 2024-09-17T07:08:12.296 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999986 2024-09-17T07:08:12.296 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.296 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.297 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:08:12.297 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.346366+0000", 2024-09-17T07:08:12.297 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.811e-05 2024-09-17T07:08:12.297 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.297 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.297 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:08:12.297 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.346377+0000", 2024-09-17T07:08:12.297 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1100999999999999e-05 2024-09-17T07:08:12.297 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.297 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.298 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:08:12.298 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.346388+0000", 2024-09-17T07:08:12.298 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0282000000000001e-05 2024-09-17T07:08:12.298 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.298 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.298 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:08:12.298 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.485178+0000", 2024-09-17T07:08:12.299 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.13879050700000001 2024-09-17T07:08:12.299 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.299 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.299 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-17T07:08:12.299 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.485183+0000", 2024-09-17T07:08:12.299 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0329999999999999e-06 2024-09-17T07:08:12.299 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.299 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.299 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:08:12.299 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:58.359332+0000", 2024-09-17T07:08:12.299 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.87414892399999999 2024-09-17T07:08:12.299 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.299 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.299 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:08:12.300 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:58.359468+0000", 2024-09-17T07:08:12.300 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000136162 2024-09-17T07:08:12.300 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.300 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.300 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 1,4", 2024-09-17T07:08:12.300 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:58.359621+0000", 2024-09-17T07:08:12.300 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000153051 2024-09-17T07:08:12.300 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.300 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.300 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:08:12.300 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:58.362451+0000", 2024-09-17T07:08:12.300 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0028294259999999999 2024-09-17T07:08:12.301 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.301 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.301 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:08:12.301 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:58.363057+0000", 2024-09-17T07:08:12.301 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00060569200000000001 2024-09-17T07:08:12.301 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.301 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.301 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:08:12.301 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:58.363591+0000", 2024-09-17T07:08:12.301 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00053405499999999997 2024-09-17T07:08:12.301 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.301 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.302 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:08:12.302 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:58.363631+0000", 2024-09-17T07:08:12.302 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0664999999999998e-05 2024-09-17T07:08:12.302 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.302 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.302 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:08:12.302 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:58.363707+0000", 2024-09-17T07:08:12.302 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5242999999999998e-05 2024-09-17T07:08:12.302 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:12.302 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:12.302 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:12.302 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.302 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.303 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.14805.0:84002 7.18 7:1e146e87:::benchmark_data_smithi100_78556_object84001:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-09-17T07:08:12.303 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:05:30.141882+0000", 2024-09-17T07:08:12.303 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 162.104278225, 2024-09-17T07:08:12.303 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0090834440000001, 2024-09-17T07:08:12.303 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:08:12.303 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:08:12.303 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:08:12.303 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.14805", 2024-09-17T07:08:12.303 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:08:12.303 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 84002 2024-09-17T07:08:12.304 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.304 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:08:12.304 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.304 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:08:12.304 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.141882+0000", 2024-09-17T07:08:12.304 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:12.304 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.304 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.304 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:08:12.304 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.141882+0000", 2024-09-17T07:08:12.304 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:12.304 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.305 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.305 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:08:12.305 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.141880+0000", 2024-09-17T07:08:12.305 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999981 2024-09-17T07:08:12.305 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.305 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.305 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:08:12.305 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.141913+0000", 2024-09-17T07:08:12.305 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3009999999999997e-05 2024-09-17T07:08:12.305 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.305 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.305 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:08:12.306 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.141926+0000", 2024-09-17T07:08:12.306 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2680999999999999e-05 2024-09-17T07:08:12.306 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.306 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.306 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:08:12.306 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.141934+0000", 2024-09-17T07:08:12.306 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9850000000000006e-06 2024-09-17T07:08:12.306 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.306 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.306 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:08:12.306 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.145584+0000", 2024-09-17T07:08:12.306 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.003650665 2024-09-17T07:08:12.306 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.306 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.307 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-17T07:08:12.307 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.145594+0000", 2024-09-17T07:08:12.307 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.8570000000000007e-06 2024-09-17T07:08:12.307 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.307 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.307 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:08:12.307 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.147036+0000", 2024-09-17T07:08:12.307 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.001441904 2024-09-17T07:08:12.307 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.307 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.307 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:08:12.307 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.147199+0000", 2024-09-17T07:08:12.308 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016296799999999999 2024-09-17T07:08:12.308 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.308 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.308 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6,7", 2024-09-17T07:08:12.308 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.147348+0000", 2024-09-17T07:08:12.308 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014852800000000001 2024-09-17T07:08:12.308 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.308 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.308 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:08:12.308 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.150802+0000", 2024-09-17T07:08:12.308 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.003454188 2024-09-17T07:08:12.308 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.309 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.309 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:08:12.309 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.150845+0000", 2024-09-17T07:08:12.309 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3349000000000001e-05 2024-09-17T07:08:12.309 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.309 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.309 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:08:12.309 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.150853+0000", 2024-09-17T07:08:12.309 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3590000000000008e-06 2024-09-17T07:08:12.309 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.309 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.310 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:08:12.310 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.150924+0000", 2024-09-17T07:08:12.310 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0389999999999995e-05 2024-09-17T07:08:12.310 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.310 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.310 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:08:12.310 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.150965+0000", 2024-09-17T07:08:12.310 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1443000000000003e-05 2024-09-17T07:08:12.310 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:12.310 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:12.310 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:12.310 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.310 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.311 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.14805.0:84044 7.18 7:1f7aa994:::benchmark_data_smithi100_78556_object84043:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-09-17T07:08:12.311 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:05:30.182888+0000", 2024-09-17T07:08:12.311 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 162.063271945, 2024-09-17T07:08:12.311 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.97126389700000004, 2024-09-17T07:08:12.311 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:08:12.311 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:08:12.311 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:08:12.311 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.14805", 2024-09-17T07:08:12.311 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:08:12.311 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 84044 2024-09-17T07:08:12.311 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.311 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:08:12.311 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.311 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:08:12.312 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.182888+0000", 2024-09-17T07:08:12.312 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:12.312 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.312 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.312 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:08:12.312 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.182888+0000", 2024-09-17T07:08:12.312 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:12.312 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.312 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.312 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:08:12.312 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.182887+0000", 2024-09-17T07:08:12.312 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-17T07:08:12.313 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.313 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.313 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:08:12.313 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.182908+0000", 2024-09-17T07:08:12.313 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.086e-05 2024-09-17T07:08:12.313 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.313 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.313 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:08:12.313 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.182915+0000", 2024-09-17T07:08:12.313 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5270000000000001e-06 2024-09-17T07:08:12.313 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.313 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.314 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:08:12.314 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.182921+0000", 2024-09-17T07:08:12.314 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.305e-06 2024-09-17T07:08:12.314 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.314 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.314 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:08:12.314 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.183176+0000", 2024-09-17T07:08:12.314 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00025525999999999999 2024-09-17T07:08:12.314 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.314 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.314 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-17T07:08:12.314 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.183186+0000", 2024-09-17T07:08:12.314 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9170000000000001e-06 2024-09-17T07:08:12.314 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.314 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.315 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:08:12.315 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.149202+0000", 2024-09-17T07:08:12.315 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.96601636199999996 2024-09-17T07:08:12.315 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.315 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.315 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:08:12.315 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.149325+0000", 2024-09-17T07:08:12.315 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012303300000000001 2024-09-17T07:08:12.315 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.315 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.315 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6,7", 2024-09-17T07:08:12.316 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.149466+0000", 2024-09-17T07:08:12.316 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014086700000000001 2024-09-17T07:08:12.316 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.316 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.316 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:08:12.316 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.151097+0000", 2024-09-17T07:08:12.316 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001630793 2024-09-17T07:08:12.316 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.316 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.316 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:08:12.316 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.151156+0000", 2024-09-17T07:08:12.316 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9160000000000003e-05 2024-09-17T07:08:12.317 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.317 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.317 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:08:12.317 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.154052+0000", 2024-09-17T07:08:12.317 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0028960169999999999 2024-09-17T07:08:12.317 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.317 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.317 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:08:12.317 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.154100+0000", 2024-09-17T07:08:12.317 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8062000000000002e-05 2024-09-17T07:08:12.317 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.317 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.317 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:08:12.318 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.154152+0000", 2024-09-17T07:08:12.318 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1833000000000003e-05 2024-09-17T07:08:12.318 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:12.318 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:12.318 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:12.318 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.318 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.318 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.14805.0:84080 7.18 7:1999f9fa:::benchmark_data_smithi100_78556_object84079:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-09-17T07:08:12.318 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:05:30.213543+0000", 2024-09-17T07:08:12.318 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 162.03261680599999, 2024-09-17T07:08:12.319 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.94079719699999997, 2024-09-17T07:08:12.319 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:08:12.319 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:08:12.319 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:08:12.319 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.14805", 2024-09-17T07:08:12.319 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:08:12.320 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 84080 2024-09-17T07:08:12.320 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.320 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:08:12.320 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.320 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:08:12.320 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.213543+0000", 2024-09-17T07:08:12.320 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:12.320 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.320 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.320 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:08:12.321 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.213543+0000", 2024-09-17T07:08:12.321 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:12.321 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.321 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.321 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:08:12.321 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.213542+0000", 2024-09-17T07:08:12.321 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-17T07:08:12.321 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.321 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.321 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:08:12.321 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.213564+0000", 2024-09-17T07:08:12.321 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1260999999999999e-05 2024-09-17T07:08:12.322 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.322 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.322 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:08:12.322 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.213574+0000", 2024-09-17T07:08:12.322 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0071000000000001e-05 2024-09-17T07:08:12.322 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.322 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.322 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:08:12.322 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.213580+0000", 2024-09-17T07:08:12.322 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.421e-06 2024-09-17T07:08:12.322 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.322 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.323 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:08:12.323 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.213839+0000", 2024-09-17T07:08:12.323 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00025876000000000002 2024-09-17T07:08:12.323 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.323 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.323 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-17T07:08:12.323 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.213849+0000", 2024-09-17T07:08:12.323 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.8400000000000007e-06 2024-09-17T07:08:12.323 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.323 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.323 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:08:12.323 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.151200+0000", 2024-09-17T07:08:12.323 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.93735180600000001 2024-09-17T07:08:12.324 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.324 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.324 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:08:12.324 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.151318+0000", 2024-09-17T07:08:12.324 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000117757 2024-09-17T07:08:12.324 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.324 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.324 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6,7", 2024-09-17T07:08:12.324 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.151475+0000", 2024-09-17T07:08:12.324 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00015659499999999999 2024-09-17T07:08:12.324 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.324 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.324 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:08:12.325 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.154041+0000", 2024-09-17T07:08:12.325 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0025661619999999999 2024-09-17T07:08:12.325 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.325 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.325 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:08:12.325 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.154190+0000", 2024-09-17T07:08:12.325 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014864300000000001 2024-09-17T07:08:12.325 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.325 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.325 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:08:12.326 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.154196+0000", 2024-09-17T07:08:12.326 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5039999999999999e-06 2024-09-17T07:08:12.326 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.326 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.326 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:08:12.326 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.154234+0000", 2024-09-17T07:08:12.326 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7562999999999999e-05 2024-09-17T07:08:12.326 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.326 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.326 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:08:12.326 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.154340+0000", 2024-09-17T07:08:12.327 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000106777 2024-09-17T07:08:12.327 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:12.327 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:12.327 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:12.327 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.327 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.327 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.14805.0:84111 7.18 7:1dd58e1f:::benchmark_data_smithi100_78556_object84110:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-09-17T07:08:12.327 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:05:30.240539+0000", 2024-09-17T07:08:12.327 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 162.005620972, 2024-09-17T07:08:12.327 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.91743996999999999, 2024-09-17T07:08:12.327 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:08:12.327 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:08:12.327 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:08:12.328 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.14805", 2024-09-17T07:08:12.328 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:08:12.328 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 84111 2024-09-17T07:08:12.328 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.328 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:08:12.328 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.328 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:08:12.328 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.240539+0000", 2024-09-17T07:08:12.328 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:12.328 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.329 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.329 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:08:12.329 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.240539+0000", 2024-09-17T07:08:12.329 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:12.329 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.329 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.329 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:08:12.329 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.240537+0000", 2024-09-17T07:08:12.329 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999986 2024-09-17T07:08:12.329 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.329 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.329 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:08:12.329 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.240565+0000", 2024-09-17T07:08:12.329 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7404000000000001e-05 2024-09-17T07:08:12.330 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.330 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.330 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:08:12.330 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.299225+0000", 2024-09-17T07:08:12.330 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.058660122000000002 2024-09-17T07:08:12.330 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.330 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.330 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:08:12.330 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.299235+0000", 2024-09-17T07:08:12.330 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7370000000000001e-06 2024-09-17T07:08:12.330 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.330 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.330 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:08:12.330 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.299678+0000", 2024-09-17T07:08:12.331 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0004437 2024-09-17T07:08:12.331 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.331 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.331 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-17T07:08:12.331 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.299687+0000", 2024-09-17T07:08:12.331 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.0650000000000005e-06 2024-09-17T07:08:12.331 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.331 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.331 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:08:12.331 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.152594+0000", 2024-09-17T07:08:12.331 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.852906991 2024-09-17T07:08:12.331 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.331 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.332 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:08:12.332 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.152698+0000", 2024-09-17T07:08:12.332 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000103175 2024-09-17T07:08:12.332 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.332 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.332 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6,7", 2024-09-17T07:08:12.332 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.152844+0000", 2024-09-17T07:08:12.333 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014609599999999999 2024-09-17T07:08:12.333 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.333 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.333 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:08:12.333 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.154641+0000", 2024-09-17T07:08:12.334 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0017969069999999999 2024-09-17T07:08:12.334 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.334 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.334 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:08:12.334 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.157865+0000", 2024-09-17T07:08:12.334 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0032239489999999998 2024-09-17T07:08:12.335 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.335 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.335 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:08:12.335 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.157900+0000", 2024-09-17T07:08:12.335 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5244999999999999e-05 2024-09-17T07:08:12.336 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.336 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.336 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:08:12.336 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.157945+0000", 2024-09-17T07:08:12.336 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5407000000000001e-05 2024-09-17T07:08:12.337 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.337 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.337 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:08:12.337 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.157979+0000", 2024-09-17T07:08:12.337 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3825000000000002e-05 2024-09-17T07:08:12.338 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:12.338 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:12.338 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:12.338 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.338 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.338 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.14805.0:84132 7.18 7:1cbb02d1:::benchmark_data_smithi100_78556_object84131:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-09-17T07:08:12.338 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:05:30.316148+0000", 2024-09-17T07:08:12.339 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 161.930011753, 2024-09-17T07:08:12.339 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.84196913299999998, 2024-09-17T07:08:12.339 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:08:12.339 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:08:12.339 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:08:12.339 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.14805", 2024-09-17T07:08:12.339 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:08:12.340 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 84132 2024-09-17T07:08:12.340 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.340 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:08:12.340 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.340 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:08:12.341 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.316148+0000", 2024-09-17T07:08:12.341 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:12.341 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.341 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.341 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:08:12.341 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.316148+0000", 2024-09-17T07:08:12.342 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:12.342 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.342 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.342 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:08:12.342 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.316146+0000", 2024-09-17T07:08:12.343 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999981 2024-09-17T07:08:12.343 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.343 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.343 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:08:12.343 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.316171+0000", 2024-09-17T07:08:12.343 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4522e-05 2024-09-17T07:08:12.344 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.344 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.344 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:08:12.344 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.316182+0000", 2024-09-17T07:08:12.344 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0648e-05 2024-09-17T07:08:12.344 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.345 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.345 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:08:12.345 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.316189+0000", 2024-09-17T07:08:12.345 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.729e-06 2024-09-17T07:08:12.345 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.345 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.345 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:08:12.346 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.316243+0000", 2024-09-17T07:08:12.346 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3483000000000003e-05 2024-09-17T07:08:12.346 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.346 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.346 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-17T07:08:12.346 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.316252+0000", 2024-09-17T07:08:12.346 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.4690000000000003e-06 2024-09-17T07:08:12.347 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.347 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.347 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:08:12.347 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.154692+0000", 2024-09-17T07:08:12.347 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.83843984199999999 2024-09-17T07:08:12.348 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.348 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.348 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:08:12.348 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.154852+0000", 2024-09-17T07:08:12.348 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00015942800000000001 2024-09-17T07:08:12.348 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.349 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.349 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6,7", 2024-09-17T07:08:12.349 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.155014+0000", 2024-09-17T07:08:12.349 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016287499999999999 2024-09-17T07:08:12.349 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.349 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.350 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:08:12.350 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.157988+0000", 2024-09-17T07:08:12.350 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0029739699999999998 2024-09-17T07:08:12.350 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.350 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.350 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:08:12.350 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.158021+0000", 2024-09-17T07:08:12.350 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.2101999999999999e-05 2024-09-17T07:08:12.350 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.350 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.351 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:08:12.351 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.158049+0000", 2024-09-17T07:08:12.351 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8255999999999998e-05 2024-09-17T07:08:12.351 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.351 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.351 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:08:12.351 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.158088+0000", 2024-09-17T07:08:12.351 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9319000000000001e-05 2024-09-17T07:08:12.351 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.351 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.351 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:08:12.351 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.158117+0000", 2024-09-17T07:08:12.351 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9268000000000002e-05 2024-09-17T07:08:12.352 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:12.352 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:12.352 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:12.352 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.352 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.352 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.14805.0:84172 7.18 7:1ab41362:::benchmark_data_smithi100_78556_object84171:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-09-17T07:08:12.352 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:05:30.354283+0000", 2024-09-17T07:08:12.352 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 161.89187732299999, 2024-09-17T07:08:12.352 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.81521177700000003, 2024-09-17T07:08:12.352 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:08:12.352 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:08:12.353 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:08:12.353 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.14805", 2024-09-17T07:08:12.353 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:08:12.353 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 84172 2024-09-17T07:08:12.353 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.353 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:08:12.353 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.353 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:08:12.353 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.354283+0000", 2024-09-17T07:08:12.353 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:12.353 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.353 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.353 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:08:12.354 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.354283+0000", 2024-09-17T07:08:12.354 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:12.354 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.354 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.354 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:08:12.354 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.354282+0000", 2024-09-17T07:08:12.354 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-17T07:08:12.354 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.354 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.354 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:08:12.354 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.354301+0000", 2024-09-17T07:08:12.354 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.9596e-05 2024-09-17T07:08:12.354 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.355 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.355 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:08:12.355 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.354310+0000", 2024-09-17T07:08:12.355 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.9020000000000005e-06 2024-09-17T07:08:12.355 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.355 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.355 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:08:12.355 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.354316+0000", 2024-09-17T07:08:12.355 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0800000000000002e-06 2024-09-17T07:08:12.355 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.356 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.356 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:08:12.356 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.354362+0000", 2024-09-17T07:08:12.356 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5775000000000001e-05 2024-09-17T07:08:12.356 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.356 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.356 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-17T07:08:12.356 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.354371+0000", 2024-09-17T07:08:12.356 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8340000000000005e-06 2024-09-17T07:08:12.356 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.357 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.357 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:08:12.357 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.156387+0000", 2024-09-17T07:08:12.357 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.80201631500000004 2024-09-17T07:08:12.357 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.357 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.357 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:08:12.357 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.156505+0000", 2024-09-17T07:08:12.357 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00011775199999999999 2024-09-17T07:08:12.357 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.357 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.357 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6,7", 2024-09-17T07:08:12.357 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.156639+0000", 2024-09-17T07:08:12.358 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00013390099999999999 2024-09-17T07:08:12.358 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.358 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.358 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:08:12.358 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.158165+0000", 2024-09-17T07:08:12.358 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0015256549999999999 2024-09-17T07:08:12.358 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.358 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.358 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:08:12.358 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.158198+0000", 2024-09-17T07:08:12.358 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3921000000000003e-05 2024-09-17T07:08:12.358 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.359 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.359 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:08:12.359 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.169374+0000", 2024-09-17T07:08:12.359 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.011175345 2024-09-17T07:08:12.359 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.359 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.359 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:08:12.359 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.169437+0000", 2024-09-17T07:08:12.359 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3408e-05 2024-09-17T07:08:12.359 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.359 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.360 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:08:12.360 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.169494+0000", 2024-09-17T07:08:12.360 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7234000000000001e-05 2024-09-17T07:08:12.360 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:12.360 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:12.360 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:12.360 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.360 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.360 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.14805.0:84182 7.18 7:1f6d53df:::benchmark_data_smithi100_78556_object84181:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-09-17T07:08:12.360 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:05:30.363797+0000", 2024-09-17T07:08:12.360 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 161.882362793, 2024-09-17T07:08:12.361 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.80578439000000002, 2024-09-17T07:08:12.361 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:08:12.361 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:08:12.361 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:08:12.361 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.14805", 2024-09-17T07:08:12.361 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:08:12.361 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 84182 2024-09-17T07:08:12.361 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.361 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:08:12.361 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.361 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:08:12.362 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.363797+0000", 2024-09-17T07:08:12.362 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:12.362 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.362 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.362 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:08:12.362 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.363797+0000", 2024-09-17T07:08:12.362 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:12.362 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.362 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.362 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:08:12.362 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.363796+0000", 2024-09-17T07:08:12.362 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999986 2024-09-17T07:08:12.363 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.363 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.363 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:08:12.363 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.363819+0000", 2024-09-17T07:08:12.363 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3042000000000001e-05 2024-09-17T07:08:12.363 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.363 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.363 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:08:12.363 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.363831+0000", 2024-09-17T07:08:12.363 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2014999999999999e-05 2024-09-17T07:08:12.363 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.363 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.363 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:08:12.364 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.363839+0000", 2024-09-17T07:08:12.364 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6660000000000008e-06 2024-09-17T07:08:12.364 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.364 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.364 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:08:12.364 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.363896+0000", 2024-09-17T07:08:12.364 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7153e-05 2024-09-17T07:08:12.365 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.365 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.365 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-17T07:08:12.365 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.363907+0000", 2024-09-17T07:08:12.365 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0774000000000001e-05 2024-09-17T07:08:12.365 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.366 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.366 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:08:12.366 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.158220+0000", 2024-09-17T07:08:12.366 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.79431372200000006 2024-09-17T07:08:12.366 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.366 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.367 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:08:12.367 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.158642+0000", 2024-09-17T07:08:12.367 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000421848 2024-09-17T07:08:12.367 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.367 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.367 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6,7", 2024-09-17T07:08:12.367 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.158758+0000", 2024-09-17T07:08:12.368 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00011592200000000001 2024-09-17T07:08:12.368 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.368 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.368 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:08:12.368 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.162687+0000", 2024-09-17T07:08:12.368 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0039287560000000003 2024-09-17T07:08:12.368 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.368 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.369 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:08:12.369 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.164120+0000", 2024-09-17T07:08:12.369 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0014329919999999999 2024-09-17T07:08:12.369 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.369 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.370 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:08:12.370 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.169507+0000", 2024-09-17T07:08:12.370 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.005387101 2024-09-17T07:08:12.370 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.370 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.370 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:08:12.371 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.169550+0000", 2024-09-17T07:08:12.371 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2852999999999998e-05 2024-09-17T07:08:12.371 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.371 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.371 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:08:12.371 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.169582+0000", 2024-09-17T07:08:12.372 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1789999999999999e-05 2024-09-17T07:08:12.372 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:12.372 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:12.372 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:12.372 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.372 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.372 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.14805.0:84203 7.18 7:1a6f3cfc:::benchmark_data_smithi100_78556_object84202:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-09-17T07:08:12.372 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:05:30.391006+0000", 2024-09-17T07:08:12.372 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 161.85515387500001, 2024-09-17T07:08:12.373 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.77878669899999997, 2024-09-17T07:08:12.373 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:08:12.373 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:08:12.373 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:08:12.373 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.14805", 2024-09-17T07:08:12.373 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:08:12.373 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 84203 2024-09-17T07:08:12.373 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.373 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:08:12.373 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.373 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:08:12.373 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.391006+0000", 2024-09-17T07:08:12.374 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:12.374 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.374 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.374 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:08:12.374 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.391006+0000", 2024-09-17T07:08:12.374 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:12.374 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.374 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.374 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:08:12.374 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.391004+0000", 2024-09-17T07:08:12.375 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999981 2024-09-17T07:08:12.375 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.375 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.375 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:08:12.375 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.391027+0000", 2024-09-17T07:08:12.375 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2889999999999999e-05 2024-09-17T07:08:12.375 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.375 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.375 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:08:12.375 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.391042+0000", 2024-09-17T07:08:12.375 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4609e-05 2024-09-17T07:08:12.375 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.376 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.376 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:08:12.376 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.391049+0000", 2024-09-17T07:08:12.376 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4769999999999997e-06 2024-09-17T07:08:12.376 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.376 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.376 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:08:12.376 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.391107+0000", 2024-09-17T07:08:12.376 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7457999999999998e-05 2024-09-17T07:08:12.377 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.377 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.377 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-17T07:08:12.377 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.391131+0000", 2024-09-17T07:08:12.377 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4032e-05 2024-09-17T07:08:12.377 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.377 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.377 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:08:12.377 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.161034+0000", 2024-09-17T07:08:12.377 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.769902906 2024-09-17T07:08:12.377 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.378 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.378 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:08:12.379 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:08:12.379 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.161155+0000", 2024-09-17T07:08:12.379 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000121673 2024-09-17T07:08:12.379 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.379 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.379 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6,7", 2024-09-17T07:08:12.379 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.161292+0000", 2024-09-17T07:08:12.379 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000136277 2024-09-17T07:08:12.380 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.380 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.380 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:08:12.380 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.164234+0000", 2024-09-17T07:08:12.380 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.002942414 2024-09-17T07:08:12.380 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.380 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.380 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:08:12.380 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.166530+0000", 2024-09-17T07:08:12.380 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00229583 2024-09-17T07:08:12.380 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.380 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.381 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:08:12.381 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.169712+0000", 2024-09-17T07:08:12.381 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0031819280000000001 2024-09-17T07:08:12.381 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.381 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.381 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:08:12.381 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.169756+0000", 2024-09-17T07:08:12.381 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4131999999999997e-05 2024-09-17T07:08:12.381 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.381 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.382 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:08:12.382 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.169793+0000", 2024-09-17T07:08:12.382 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6931999999999998e-05 2024-09-17T07:08:12.382 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:12.382 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:12.382 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:12.382 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.382 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.382 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.14805.0:84230 7.18 7:1ef434f2:::benchmark_data_smithi100_78556_object84229:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-09-17T07:08:12.382 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:05:30.414950+0000", 2024-09-17T07:08:12.382 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 161.83121035100001, 2024-09-17T07:08:12.383 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.75492697600000003, 2024-09-17T07:08:12.383 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:08:12.383 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:08:12.383 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:08:12.383 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.14805", 2024-09-17T07:08:12.383 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:08:12.383 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 84230 2024-09-17T07:08:12.383 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.383 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:08:12.383 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.383 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:08:12.383 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.414950+0000", 2024-09-17T07:08:12.384 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:12.384 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.384 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.384 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:08:12.384 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.414950+0000", 2024-09-17T07:08:12.384 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:12.384 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.384 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.384 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:08:12.384 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.414949+0000", 2024-09-17T07:08:12.384 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-17T07:08:12.385 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.385 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.385 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:08:12.385 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.414972+0000", 2024-09-17T07:08:12.385 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3638999999999998e-05 2024-09-17T07:08:12.385 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.385 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.385 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:08:12.385 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.414987+0000", 2024-09-17T07:08:12.385 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5209e-05 2024-09-17T07:08:12.386 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.386 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.386 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:08:12.386 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.414995+0000", 2024-09-17T07:08:12.386 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2389999999999996e-06 2024-09-17T07:08:12.386 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.386 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.386 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:08:12.386 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.415044+0000", 2024-09-17T07:08:12.386 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9471000000000001e-05 2024-09-17T07:08:12.386 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.386 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.387 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-17T07:08:12.387 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.415198+0000", 2024-09-17T07:08:12.387 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00015380400000000001 2024-09-17T07:08:12.387 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.387 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.387 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:08:12.387 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.162713+0000", 2024-09-17T07:08:12.387 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.74751521099999996 2024-09-17T07:08:12.387 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.387 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.387 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:08:12.387 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.162813+0000", 2024-09-17T07:08:12.388 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9946999999999997e-05 2024-09-17T07:08:12.388 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.388 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.388 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6,7", 2024-09-17T07:08:12.388 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.162923+0000", 2024-09-17T07:08:12.388 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00010950699999999999 2024-09-17T07:08:12.388 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.388 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.388 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:08:12.388 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.166506+0000", 2024-09-17T07:08:12.388 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.003582928 2024-09-17T07:08:12.389 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.389 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.389 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:08:12.389 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.169699+0000", 2024-09-17T07:08:12.389 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0031935420000000002 2024-09-17T07:08:12.389 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.389 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.389 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:08:12.389 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.169803+0000", 2024-09-17T07:08:12.389 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000103566 2024-09-17T07:08:12.389 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.390 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.390 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:08:12.390 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.169844+0000", 2024-09-17T07:08:12.390 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.121e-05 2024-09-17T07:08:12.390 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.390 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.390 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:08:12.390 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.169877+0000", 2024-09-17T07:08:12.390 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.2808999999999997e-05 2024-09-17T07:08:12.390 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:12.390 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:12.390 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:12.391 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.391 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.391 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.14805.0:84239 7.18 7:1d5db2f1:::benchmark_data_smithi100_78556_object84238:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-09-17T07:08:12.391 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:05:30.428155+0000", 2024-09-17T07:08:12.391 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 161.818005301, 2024-09-17T07:08:12.391 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.74195996500000005, 2024-09-17T07:08:12.391 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:08:12.391 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:08:12.391 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:08:12.391 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.14805", 2024-09-17T07:08:12.391 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:08:12.391 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 84239 2024-09-17T07:08:12.392 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.392 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:08:12.392 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.392 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:08:12.392 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.428155+0000", 2024-09-17T07:08:12.392 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:12.392 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.392 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.392 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:08:12.392 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.428155+0000", 2024-09-17T07:08:12.392 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:12.393 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.393 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.393 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:08:12.393 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.428153+0000", 2024-09-17T07:08:12.393 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999981 2024-09-17T07:08:12.393 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.393 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.393 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:08:12.393 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.428175+0000", 2024-09-17T07:08:12.393 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2569999999999999e-05 2024-09-17T07:08:12.393 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.393 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.393 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:08:12.394 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.428186+0000", 2024-09-17T07:08:12.394 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0429e-05 2024-09-17T07:08:12.394 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.394 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.394 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:08:12.394 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.428193+0000", 2024-09-17T07:08:12.394 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8940000000000004e-06 2024-09-17T07:08:12.394 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.394 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.394 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:08:12.394 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.428243+0000", 2024-09-17T07:08:12.394 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0494000000000001e-05 2024-09-17T07:08:12.395 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.395 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.395 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-17T07:08:12.395 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.428253+0000", 2024-09-17T07:08:12.395 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7860000000000008e-06 2024-09-17T07:08:12.396 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.396 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.396 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:08:12.397 INFO:teuthology.orchestra.run.smithi100.stderr:enabled application 'rados' on pool 'unique_pool_1' 2024-09-17T07:08:12.398 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.164256+0000", 2024-09-17T07:08:12.398 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.73600358600000004 2024-09-17T07:08:12.398 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.398 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.398 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:08:12.399 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.164377+0000", 2024-09-17T07:08:12.399 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012071 2024-09-17T07:08:12.399 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.399 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.399 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6,7", 2024-09-17T07:08:12.399 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.164531+0000", 2024-09-17T07:08:12.399 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00015407600000000001 2024-09-17T07:08:12.399 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.400 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.400 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:08:12.400 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.169651+0000", 2024-09-17T07:08:12.400 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0051196640000000003 2024-09-17T07:08:12.400 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.400 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.401 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:08:12.401 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.169887+0000", 2024-09-17T07:08:12.401 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00023590500000000001 2024-09-17T07:08:12.401 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.401 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.402 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:08:12.402 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.170038+0000", 2024-09-17T07:08:12.402 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00015090500000000001 2024-09-17T07:08:12.402 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.402 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.402 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:08:12.402 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.170081+0000", 2024-09-17T07:08:12.403 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3464000000000003e-05 2024-09-17T07:08:12.403 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.403 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.403 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:08:12.403 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.170115+0000", 2024-09-17T07:08:12.403 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3553000000000002e-05 2024-09-17T07:08:12.403 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:12.403 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:12.403 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:12.403 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.403 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.404 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.14805.0:84247 7.18 7:1f13e6b2:::benchmark_data_smithi100_78556_object84246:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-09-17T07:08:12.404 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:05:30.435367+0000", 2024-09-17T07:08:12.404 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 161.81079281300001, 2024-09-17T07:08:12.404 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.74378478199999998, 2024-09-17T07:08:12.404 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:08:12.404 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:08:12.404 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:08:12.404 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.14805", 2024-09-17T07:08:12.404 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:08:12.404 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 84247 2024-09-17T07:08:12.404 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.404 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:08:12.405 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.405 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:08:12.405 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.435367+0000", 2024-09-17T07:08:12.405 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:12.405 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.405 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.405 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:08:12.405 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.435367+0000", 2024-09-17T07:08:12.405 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:12.405 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.405 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.405 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:08:12.406 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.435365+0000", 2024-09-17T07:08:12.406 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999981 2024-09-17T07:08:12.406 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.406 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.406 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:08:12.406 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.435404+0000", 2024-09-17T07:08:12.406 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8220000000000003e-05 2024-09-17T07:08:12.406 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.406 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.406 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:08:12.406 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.435416+0000", 2024-09-17T07:08:12.406 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.205e-05 2024-09-17T07:08:12.406 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.407 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.407 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:08:12.407 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.435423+0000", 2024-09-17T07:08:12.407 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1760000000000004e-06 2024-09-17T07:08:12.407 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.407 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.407 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:08:12.407 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.435482+0000", 2024-09-17T07:08:12.407 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8857000000000002e-05 2024-09-17T07:08:12.407 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.407 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.407 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-17T07:08:12.407 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.435492+0000", 2024-09-17T07:08:12.407 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7610000000000002e-06 2024-09-17T07:08:12.408 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.408 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.408 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:08:12.408 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.166549+0000", 2024-09-17T07:08:12.408 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.73105739300000006 2024-09-17T07:08:12.408 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.408 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.408 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:08:12.408 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.166642+0000", 2024-09-17T07:08:12.408 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.2769000000000006e-05 2024-09-17T07:08:12.408 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.408 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.409 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6,7", 2024-09-17T07:08:12.409 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.167044+0000", 2024-09-17T07:08:12.409 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00040220799999999998 2024-09-17T07:08:12.409 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.409 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.409 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:08:12.409 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.169894+0000", 2024-09-17T07:08:12.409 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.002849697 2024-09-17T07:08:12.409 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.409 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.409 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:08:12.409 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.170001+0000", 2024-09-17T07:08:12.409 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000107262 2024-09-17T07:08:12.410 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.410 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.410 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:08:12.410 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.179069+0000", 2024-09-17T07:08:12.410 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0090679720000000005 2024-09-17T07:08:12.410 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.410 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.410 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:08:12.410 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.179110+0000", 2024-09-17T07:08:12.410 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1483000000000002e-05 2024-09-17T07:08:12.410 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.411 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.411 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:08:12.411 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.179152+0000", 2024-09-17T07:08:12.411 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1672999999999999e-05 2024-09-17T07:08:12.411 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:12.411 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:12.411 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:12.411 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.411 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.411 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.14805.0:84302 7.18 7:1819d156:::benchmark_data_smithi100_78556_object84301:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-09-17T07:08:12.411 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:05:30.494328+0000", 2024-09-17T07:08:12.411 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 161.75183248799999, 2024-09-17T07:08:12.412 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.68497140400000001, 2024-09-17T07:08:12.412 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:08:12.412 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:08:12.412 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:08:12.412 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.14805", 2024-09-17T07:08:12.412 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:08:12.412 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 84302 2024-09-17T07:08:12.412 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.412 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:08:12.412 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.412 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:08:12.412 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.494328+0000", 2024-09-17T07:08:12.413 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:12.413 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.413 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.413 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:08:12.413 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.494328+0000", 2024-09-17T07:08:12.413 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:12.413 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.413 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.413 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:08:12.413 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.494327+0000", 2024-09-17T07:08:12.414 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-17T07:08:12.414 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.414 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.414 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:08:12.414 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.494346+0000", 2024-09-17T07:08:12.414 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.9245e-05 2024-09-17T07:08:12.414 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.414 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.414 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:08:12.414 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.494355+0000", 2024-09-17T07:08:12.414 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3249999999999997e-06 2024-09-17T07:08:12.414 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.415 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.415 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:08:12.415 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.494360+0000", 2024-09-17T07:08:12.415 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9320000000000004e-06 2024-09-17T07:08:12.415 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.415 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.415 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:08:12.415 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.494400+0000", 2024-09-17T07:08:12.415 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9918e-05 2024-09-17T07:08:12.415 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.415 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.416 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-17T07:08:12.416 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.494408+0000", 2024-09-17T07:08:12.416 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6140000000000003e-06 2024-09-17T07:08:12.416 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.416 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.416 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:08:12.416 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.170148+0000", 2024-09-17T07:08:12.416 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.67574066899999996 2024-09-17T07:08:12.416 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.416 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.417 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:08:12.417 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.170241+0000", 2024-09-17T07:08:12.417 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.2353000000000002e-05 2024-09-17T07:08:12.417 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.417 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.417 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6,7", 2024-09-17T07:08:12.417 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.170352+0000", 2024-09-17T07:08:12.417 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000111702 2024-09-17T07:08:12.417 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.417 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.418 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:08:12.418 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.179010+0000", 2024-09-17T07:08:12.418 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0086577300000000006 2024-09-17T07:08:12.418 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.418 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.418 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:08:12.418 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.179019+0000", 2024-09-17T07:08:12.418 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.7309999999999999e-06 2024-09-17T07:08:12.418 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.418 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.418 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:08:12.418 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.179228+0000", 2024-09-17T07:08:12.419 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00020955299999999999 2024-09-17T07:08:12.419 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.419 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.419 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:08:12.419 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.179268+0000", 2024-09-17T07:08:12.419 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9848999999999997e-05 2024-09-17T07:08:12.419 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.419 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.419 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:08:12.419 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.179299+0000", 2024-09-17T07:08:12.419 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0614000000000002e-05 2024-09-17T07:08:12.420 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:12.420 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:12.420 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:12.420 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.420 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.420 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.14805.0:84355 7.18 7:1becf356:::benchmark_data_smithi100_78556_object84354:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-09-17T07:08:12.420 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:05:30.572233+0000", 2024-09-17T07:08:12.420 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 161.673927069, 2024-09-17T07:08:12.420 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.60719921300000002, 2024-09-17T07:08:12.420 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:08:12.420 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:08:12.421 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:08:12.421 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.14805", 2024-09-17T07:08:12.421 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:08:12.421 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 84355 2024-09-17T07:08:12.421 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.421 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:08:12.421 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.421 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:08:12.421 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.572233+0000", 2024-09-17T07:08:12.421 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:12.422 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.422 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.422 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:08:12.422 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.572233+0000", 2024-09-17T07:08:12.422 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:12.422 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.422 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.422 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:08:12.422 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.572231+0000", 2024-09-17T07:08:12.422 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999981 2024-09-17T07:08:12.422 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.423 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.423 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:08:12.423 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.572297+0000", 2024-09-17T07:08:12.423 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5572000000000002e-05 2024-09-17T07:08:12.423 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.423 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.423 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:08:12.424 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.572474+0000", 2024-09-17T07:08:12.424 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000177102 2024-09-17T07:08:12.424 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.424 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.424 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:08:12.425 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.572486+0000", 2024-09-17T07:08:12.425 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1865000000000001e-05 2024-09-17T07:08:12.425 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.425 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.425 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:08:12.425 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.572539+0000", 2024-09-17T07:08:12.426 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3007000000000003e-05 2024-09-17T07:08:12.426 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.426 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.426 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-17T07:08:12.426 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.572550+0000", 2024-09-17T07:08:12.426 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1779e-05 2024-09-17T07:08:12.427 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.427 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.427 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:08:12.427 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.171857+0000", 2024-09-17T07:08:12.427 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.59930705299999998 2024-09-17T07:08:12.427 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.428 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.428 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:08:12.428 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.171963+0000", 2024-09-17T07:08:12.428 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00010545200000000001 2024-09-17T07:08:12.428 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.429 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.429 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6,7", 2024-09-17T07:08:12.429 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.172088+0000", 2024-09-17T07:08:12.429 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000125017 2024-09-17T07:08:12.429 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.429 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.430 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:08:12.430 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.179025+0000", 2024-09-17T07:08:12.430 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0069367259999999998 2024-09-17T07:08:12.430 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.430 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.430 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:08:12.431 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.179193+0000", 2024-09-17T07:08:12.431 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016800499999999999 2024-09-17T07:08:12.431 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.431 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.431 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:08:12.432 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.179366+0000", 2024-09-17T07:08:12.432 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00017284399999999999 2024-09-17T07:08:12.432 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.432 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.432 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:08:12.432 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.179402+0000", 2024-09-17T07:08:12.433 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6442000000000001e-05 2024-09-17T07:08:12.433 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.433 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.433 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:08:12.433 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.179432+0000", 2024-09-17T07:08:12.434 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0199999999999999e-05 2024-09-17T07:08:12.434 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:12.434 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:12.434 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:12.434 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.435 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.435 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.14805.0:84356 7.18 7:19fa6950:::benchmark_data_smithi100_78556_object84355:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-09-17T07:08:12.435 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:05:30.572501+0000", 2024-09-17T07:08:12.435 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 161.673658725, 2024-09-17T07:08:12.435 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.60703522399999998, 2024-09-17T07:08:12.435 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:08:12.436 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:08:12.436 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:08:12.436 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.14805", 2024-09-17T07:08:12.436 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:08:12.436 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 84356 2024-09-17T07:08:12.436 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.437 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:08:12.437 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.437 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:08:12.437 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.572501+0000", 2024-09-17T07:08:12.437 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:12.438 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.438 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.438 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:08:12.438 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.572501+0000", 2024-09-17T07:08:12.438 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:12.438 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.439 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.439 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:08:12.439 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.572500+0000", 2024-09-17T07:08:12.439 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-17T07:08:12.439 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.439 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.439 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:08:12.440 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.572523+0000", 2024-09-17T07:08:12.440 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2782999999999999e-05 2024-09-17T07:08:12.440 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.440 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.440 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:08:12.440 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.572533+0000", 2024-09-17T07:08:12.441 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.628e-06 2024-09-17T07:08:12.441 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.441 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.441 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:08:12.441 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.572539+0000", 2024-09-17T07:08:12.441 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0179999999999998e-06 2024-09-17T07:08:12.441 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.441 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.441 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:08:12.441 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.572584+0000", 2024-09-17T07:08:12.442 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5587999999999998e-05 2024-09-17T07:08:12.442 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.442 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.442 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-17T07:08:12.442 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.572594+0000", 2024-09-17T07:08:12.442 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.2779999999999999e-06 2024-09-17T07:08:12.442 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.442 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.442 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:08:12.442 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.173514+0000", 2024-09-17T07:08:12.442 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.60092057099999996 2024-09-17T07:08:12.442 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.442 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.443 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:08:12.443 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.173635+0000", 2024-09-17T07:08:12.443 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012019999999999999 2024-09-17T07:08:12.443 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.443 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.443 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6,7", 2024-09-17T07:08:12.443 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.173772+0000", 2024-09-17T07:08:12.443 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000137138 2024-09-17T07:08:12.443 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.443 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.443 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:08:12.443 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.179030+0000", 2024-09-17T07:08:12.443 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0052585599999999998 2024-09-17T07:08:12.444 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.444 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.444 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:08:12.444 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.179333+0000", 2024-09-17T07:08:12.444 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00030268299999999999 2024-09-17T07:08:12.444 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.444 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.444 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:08:12.444 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.179468+0000", 2024-09-17T07:08:12.444 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00013528999999999999 2024-09-17T07:08:12.444 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.445 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.445 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:08:12.445 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.179506+0000", 2024-09-17T07:08:12.445 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8195000000000003e-05 2024-09-17T07:08:12.445 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.445 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.445 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:08:12.445 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.179537+0000", 2024-09-17T07:08:12.445 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0141000000000002e-05 2024-09-17T07:08:12.445 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:12.445 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:12.445 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:12.445 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.446 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.446 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.14805.0:84444 7.18 7:1b0bae24:::benchmark_data_smithi100_78556_object84443:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-09-17T07:08:12.446 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:05:30.911988+0000", 2024-09-17T07:08:12.446 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 161.33417201899999, 2024-09-17T07:08:12.446 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.273581097, 2024-09-17T07:08:12.446 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:08:12.446 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:08:12.446 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:08:12.446 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.14805", 2024-09-17T07:08:12.446 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:08:12.446 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 84444 2024-09-17T07:08:12.446 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.447 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:08:12.447 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.447 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:08:12.447 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.911988+0000", 2024-09-17T07:08:12.447 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:12.447 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.447 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.447 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:08:12.447 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.911988+0000", 2024-09-17T07:08:12.447 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:12.447 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.447 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.447 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:08:12.447 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.911987+0000", 2024-09-17T07:08:12.448 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999986 2024-09-17T07:08:12.448 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.448 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.448 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:08:12.448 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.912012+0000", 2024-09-17T07:08:12.448 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4989000000000001e-05 2024-09-17T07:08:12.448 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.448 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.448 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:08:12.448 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.912027+0000", 2024-09-17T07:08:12.448 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5752000000000001e-05 2024-09-17T07:08:12.448 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.448 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.449 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:08:12.449 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.912036+0000", 2024-09-17T07:08:12.449 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5150000000000006e-06 2024-09-17T07:08:12.449 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.449 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.449 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:08:12.449 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.912091+0000", 2024-09-17T07:08:12.449 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5526000000000003e-05 2024-09-17T07:08:12.449 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.449 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.449 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-17T07:08:12.449 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.912104+0000", 2024-09-17T07:08:12.450 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2245999999999999e-05 2024-09-17T07:08:12.450 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.450 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.450 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:08:12.450 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.175255+0000", 2024-09-17T07:08:12.450 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.26315107599999998 2024-09-17T07:08:12.450 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.450 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.450 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:08:12.450 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.175373+0000", 2024-09-17T07:08:12.450 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000117946 2024-09-17T07:08:12.450 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.451 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.451 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6,7", 2024-09-17T07:08:12.451 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.176814+0000", 2024-09-17T07:08:12.451 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0014411199999999999 2024-09-17T07:08:12.451 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.451 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.451 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:08:12.451 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.179576+0000", 2024-09-17T07:08:12.451 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0027623560000000001 2024-09-17T07:08:12.451 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.451 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.451 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:08:12.451 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.185288+0000", 2024-09-17T07:08:12.451 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0057121389999999998 2024-09-17T07:08:12.452 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.452 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.452 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:08:12.452 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.185299+0000", 2024-09-17T07:08:12.452 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0691e-05 2024-09-17T07:08:12.452 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.452 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.452 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:08:12.452 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.185342+0000", 2024-09-17T07:08:12.452 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3127000000000002e-05 2024-09-17T07:08:12.452 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:12.452 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:12.452 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:08:12.453 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.185569+0000", 2024-09-17T07:08:12.453 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00022708400000000001 2024-09-17T07:08:12.453 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:12.453 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:12.453 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:12.453 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:12.453 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:12.453 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-17T07:08:12.473 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:12.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:12.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:08:12.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:12 smithi135 ceph-mon[33238]: 6.3 scrub starts 2024-09-17T07:08:12.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:12 smithi135 ceph-mon[33238]: 6.3 scrub ok 2024-09-17T07:08:12.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:12 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:08:12.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:12 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:08:12.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:12 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2893872118' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-17T07:08:12.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:12 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:08:12.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:12 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:08:12.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:12 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1032797582' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:08:12.575 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:08:12.601 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:12.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:12.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:08:12.625 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:12 smithi100 ceph-mon[39462]: 6.3 scrub starts 2024-09-17T07:08:12.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:12 smithi100 ceph-mon[39462]: 6.3 scrub ok 2024-09-17T07:08:12.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:12 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:08:12.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:12 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:08:12.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:12 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2893872118' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-17T07:08:12.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:12 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:08:12.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:12 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:08:12.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:12 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1032797582' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:08:12.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:12 smithi100 ceph-mon[29092]: 6.3 scrub starts 2024-09-17T07:08:12.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:12 smithi100 ceph-mon[29092]: 6.3 scrub ok 2024-09-17T07:08:12.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:12 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:08:12.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:12 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:08:12.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:12 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2893872118' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-17T07:08:12.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:12 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:08:12.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:12 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:08:12.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:12 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1032797582' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:08:12.703 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:08:12.729 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:12.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:12.730 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:08:12.830 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:08:12.853 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:12.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:12.854 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:08:12.954 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:08:12.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:12.981 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:12.981 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:08:13.007 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-17T07:08:13.008 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T07:08:13.082 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:08:13.103 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:13.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:13.104 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:08:13.205 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:08:13.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:13.226 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:13.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:08:13.252 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:08:13.327 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:08:13.349 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:13.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:13.350 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:08:13.356 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:13 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2893872118' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-17T07:08:13.356 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:13 smithi100 ceph-mon[29092]: osdmap e174: 8 total, 8 up, 8 in 2024-09-17T07:08:13.357 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:13 smithi100 ceph-mon[29092]: pgmap v262: 148 pgs: 148 active+clean; 582 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:08:13.357 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:13 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2893872118' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-17T07:08:13.357 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:13 smithi100 ceph-mon[39462]: osdmap e174: 8 total, 8 up, 8 in 2024-09-17T07:08:13.357 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:13 smithi100 ceph-mon[39462]: pgmap v262: 148 pgs: 148 active+clean; 582 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:08:13.451 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:08:13.475 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:13.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:13.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:08:13.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:13 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2893872118' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-17T07:08:13.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:13 smithi135 ceph-mon[33238]: osdmap e174: 8 total, 8 up, 8 in 2024-09-17T07:08:13.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:13 smithi135 ceph-mon[33238]: pgmap v262: 148 pgs: 148 active+clean; 582 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:08:13.577 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:08:13.601 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:13.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:13.602 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:08:13.703 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:08:13.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:13.725 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:13.725 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:08:13.787 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-17T07:08:13.826 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:08:13.851 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:13.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:13.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:08:13.952 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:08:13.974 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:13.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:13.975 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:08:14.075 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:08:14.101 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:14.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:14.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:08:14.203 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:08:14.229 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:14.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:14.230 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:08:14.331 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:08:14.355 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:14.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:14.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:08:14.456 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:08:14.481 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:14.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:14.482 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:08:14.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:14 smithi135 ceph-mon[33238]: 2.1a scrub starts 2024-09-17T07:08:14.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:14 smithi135 ceph-mon[33238]: 2.1a scrub ok 2024-09-17T07:08:14.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:14 smithi135 ceph-mon[33238]: 6.d scrub starts 2024-09-17T07:08:14.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:14 smithi135 ceph-mon[33238]: 6.d scrub ok 2024-09-17T07:08:14.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:14 smithi135 ceph-mon[33238]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-17T07:08:14.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:14 smithi135 ceph-mon[33238]: 4.f scrub starts 2024-09-17T07:08:14.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:14 smithi135 ceph-mon[33238]: 4.f scrub ok 2024-09-17T07:08:14.583 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:08:14.607 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:14.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:14.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:08:14.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:14 smithi100 ceph-mon[39462]: 2.1a scrub starts 2024-09-17T07:08:14.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:14 smithi100 ceph-mon[39462]: 2.1a scrub ok 2024-09-17T07:08:14.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:14 smithi100 ceph-mon[39462]: 6.d scrub starts 2024-09-17T07:08:14.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:14 smithi100 ceph-mon[39462]: 6.d scrub ok 2024-09-17T07:08:14.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:14 smithi100 ceph-mon[39462]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-17T07:08:14.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:14 smithi100 ceph-mon[39462]: 4.f scrub starts 2024-09-17T07:08:14.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:14 smithi100 ceph-mon[39462]: 4.f scrub ok 2024-09-17T07:08:14.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:14 smithi100 ceph-mon[29092]: 2.1a scrub starts 2024-09-17T07:08:14.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:14 smithi100 ceph-mon[29092]: 2.1a scrub ok 2024-09-17T07:08:14.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:14 smithi100 ceph-mon[29092]: 6.d scrub starts 2024-09-17T07:08:14.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:14 smithi100 ceph-mon[29092]: 6.d scrub ok 2024-09-17T07:08:14.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:14 smithi100 ceph-mon[29092]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-17T07:08:14.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:14 smithi100 ceph-mon[29092]: 4.f scrub starts 2024-09-17T07:08:14.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:14 smithi100 ceph-mon[29092]: 4.f scrub ok 2024-09-17T07:08:14.709 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:08:14.732 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:14.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:14.733 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:08:14.833 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:08:14.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:14.859 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:14.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:08:14.960 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:08:14.982 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:14.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:14.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:08:15.084 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:08:15.110 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:15.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:15.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:08:15.212 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:08:15.237 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:15.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:15.239 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:08:15.339 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:08:15.361 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:15.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:15.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:08:15.463 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:08:15.489 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:15.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:15.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:08:15.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:15 smithi135 ceph-mon[33238]: 1.0 scrub starts 2024-09-17T07:08:15.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:15 smithi135 ceph-mon[33238]: 1.0 scrub ok 2024-09-17T07:08:15.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:15 smithi135 ceph-mon[33238]: 5.3 scrub starts 2024-09-17T07:08:15.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:15 smithi135 ceph-mon[33238]: 5.3 scrub ok 2024-09-17T07:08:15.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:15 smithi135 ceph-mon[33238]: 5.1c scrub starts 2024-09-17T07:08:15.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:15 smithi135 ceph-mon[33238]: 5.1c scrub ok 2024-09-17T07:08:15.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:15 smithi135 ceph-mon[33238]: 6.1f scrub starts 2024-09-17T07:08:15.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:15 smithi135 ceph-mon[33238]: 6.1f scrub ok 2024-09-17T07:08:15.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:15 smithi135 ceph-mon[33238]: 5.6 scrub starts 2024-09-17T07:08:15.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:15 smithi135 ceph-mon[33238]: 5.6 scrub ok 2024-09-17T07:08:15.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:15 smithi135 ceph-mon[33238]: pgmap v263: 148 pgs: 148 active+clean; 582 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-17T07:08:15.547 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:15 smithi100 ceph-mon[29092]: 1.0 scrub starts 2024-09-17T07:08:15.547 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:15 smithi100 ceph-mon[29092]: 1.0 scrub ok 2024-09-17T07:08:15.547 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:15 smithi100 ceph-mon[29092]: 5.3 scrub starts 2024-09-17T07:08:15.548 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:15 smithi100 ceph-mon[29092]: 5.3 scrub ok 2024-09-17T07:08:15.548 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:15 smithi100 ceph-mon[29092]: 5.1c scrub starts 2024-09-17T07:08:15.548 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:15 smithi100 ceph-mon[29092]: 5.1c scrub ok 2024-09-17T07:08:15.548 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:15 smithi100 ceph-mon[29092]: 6.1f scrub starts 2024-09-17T07:08:15.548 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:15 smithi100 ceph-mon[29092]: 6.1f scrub ok 2024-09-17T07:08:15.548 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:15 smithi100 ceph-mon[29092]: 5.6 scrub starts 2024-09-17T07:08:15.549 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:15 smithi100 ceph-mon[29092]: 5.6 scrub ok 2024-09-17T07:08:15.549 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:15 smithi100 ceph-mon[29092]: pgmap v263: 148 pgs: 148 active+clean; 582 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-17T07:08:15.549 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:15 smithi100 ceph-mon[39462]: 1.0 scrub starts 2024-09-17T07:08:15.549 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:15 smithi100 ceph-mon[39462]: 1.0 scrub ok 2024-09-17T07:08:15.549 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:15 smithi100 ceph-mon[39462]: 5.3 scrub starts 2024-09-17T07:08:15.550 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:15 smithi100 ceph-mon[39462]: 5.3 scrub ok 2024-09-17T07:08:15.550 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:15 smithi100 ceph-mon[39462]: 5.1c scrub starts 2024-09-17T07:08:15.550 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:15 smithi100 ceph-mon[39462]: 5.1c scrub ok 2024-09-17T07:08:15.550 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:15 smithi100 ceph-mon[39462]: 6.1f scrub starts 2024-09-17T07:08:15.550 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:15 smithi100 ceph-mon[39462]: 6.1f scrub ok 2024-09-17T07:08:15.550 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:15 smithi100 ceph-mon[39462]: 5.6 scrub starts 2024-09-17T07:08:15.550 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:15 smithi100 ceph-mon[39462]: 5.6 scrub ok 2024-09-17T07:08:15.550 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:15 smithi100 ceph-mon[39462]: pgmap v263: 148 pgs: 148 active+clean; 582 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-17T07:08:15.592 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:08:15.613 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:15.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:15.614 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:08:15.715 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:08:15.737 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:15.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:15.738 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:08:15.839 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:08:15.865 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:15.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:15.866 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:08:15.967 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:08:15.982 DEBUG:teuthology.orchestra.run.smithi100:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_1 bench 90 write' 2024-09-17T07:08:15.982 INFO:tasks.thrashosds.thrasher:Shrinking pool unique_pool_1 2024-09-17T07:08:15.983 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool unique_pool_1 by 10 min 8 2024-09-17T07:08:15.983 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T07:08:15.989 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-17T07:08:15.989 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-17T07:08:15.992 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:15.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:15.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:08:16.041 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:hints = 1 2024-09-17T07:08:16.042 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-17T07:08:16.042 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Object prefix: benchmark_data_smithi100_102803 2024-09-17T07:08:16.042 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-17T07:08:16.042 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 0 0 0 0 0 0 - 0 2024-09-17T07:08:16.095 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:08:16.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:16.129 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:16.129 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:08:16.191 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:08:15 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:08:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:08:16.229 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:08:16.264 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:16.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:16.265 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:08:16.366 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:08:16.388 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:16.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:16.389 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:08:16.487 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:16 smithi100 ceph-mon[29092]: 2.17 scrub starts 2024-09-17T07:08:16.487 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:16 smithi100 ceph-mon[29092]: 2.17 scrub ok 2024-09-17T07:08:16.487 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:16 smithi100 ceph-mon[29092]: 2.13 scrub starts 2024-09-17T07:08:16.488 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:16 smithi100 ceph-mon[29092]: 2.13 scrub ok 2024-09-17T07:08:16.488 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:16 smithi100 ceph-mon[29092]: 3.2 scrub starts 2024-09-17T07:08:16.488 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:16 smithi100 ceph-mon[29092]: 3.2 scrub ok 2024-09-17T07:08:16.488 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:16 smithi100 ceph-mon[29092]: 2.c scrub starts 2024-09-17T07:08:16.488 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:16 smithi100 ceph-mon[29092]: 2.c scrub ok 2024-09-17T07:08:16.488 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:16 smithi100 ceph-mon[39462]: 2.17 scrub starts 2024-09-17T07:08:16.488 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:16 smithi100 ceph-mon[39462]: 2.17 scrub ok 2024-09-17T07:08:16.488 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:16 smithi100 ceph-mon[39462]: 2.13 scrub starts 2024-09-17T07:08:16.488 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:16 smithi100 ceph-mon[39462]: 2.13 scrub ok 2024-09-17T07:08:16.488 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:16 smithi100 ceph-mon[39462]: 3.2 scrub starts 2024-09-17T07:08:16.489 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:16 smithi100 ceph-mon[39462]: 3.2 scrub ok 2024-09-17T07:08:16.489 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:16 smithi100 ceph-mon[39462]: 2.c scrub starts 2024-09-17T07:08:16.489 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:16 smithi100 ceph-mon[39462]: 2.c scrub ok 2024-09-17T07:08:16.490 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:08:16.527 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:16.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:16.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:08:16.534 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:08:16.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:16 smithi135 ceph-mon[33238]: 2.17 scrub starts 2024-09-17T07:08:16.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:16 smithi135 ceph-mon[33238]: 2.17 scrub ok 2024-09-17T07:08:16.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:16 smithi135 ceph-mon[33238]: 2.13 scrub starts 2024-09-17T07:08:16.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:16 smithi135 ceph-mon[33238]: 2.13 scrub ok 2024-09-17T07:08:16.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:16 smithi135 ceph-mon[33238]: 3.2 scrub starts 2024-09-17T07:08:16.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:16 smithi135 ceph-mon[33238]: 3.2 scrub ok 2024-09-17T07:08:16.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:16 smithi135 ceph-mon[33238]: 2.c scrub starts 2024-09-17T07:08:16.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:16 smithi135 ceph-mon[33238]: 2.c scrub ok 2024-09-17T07:08:16.629 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:08:16.657 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:08:16.669 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:08:16.685 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:16.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:16.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:08:16.787 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:08:16.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:16.834 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:16.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:08:16.934 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:08:16.965 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:16.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:16.966 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:08:17.042 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 1 16 3108 3092 193.23 193.25 0.00543759 0.00516004 2024-09-17T07:08:17.067 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:08:17.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:17.101 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:17.101 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:08:17.202 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:08:17.234 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:17.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:17.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:08:17.316 INFO:teuthology.orchestra.run.smithi100.stderr:nodeep-scrub is unset 2024-09-17T07:08:17.336 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:08:17.373 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:17.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:17.374 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:08:17.450 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:17 smithi100 ceph-mon[29092]: 6.17 scrub starts 2024-09-17T07:08:17.450 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:17 smithi100 ceph-mon[29092]: 6.17 scrub ok 2024-09-17T07:08:17.450 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:17 smithi100 ceph-mon[29092]: 4.12 scrub starts 2024-09-17T07:08:17.451 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:17 smithi100 ceph-mon[29092]: 4.12 scrub ok 2024-09-17T07:08:17.451 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:17 smithi100 ceph-mon[29092]: 5.b scrub starts 2024-09-17T07:08:17.451 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:17 smithi100 ceph-mon[29092]: 5.b scrub ok 2024-09-17T07:08:17.451 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:17 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/469528541' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:08:17.451 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:17 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:08:17.451 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:17 smithi100 ceph-mon[29092]: pgmap v264: 148 pgs: 148 active+clean; 582 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-17T07:08:17.451 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:17 smithi100 ceph-mon[39462]: 6.17 scrub starts 2024-09-17T07:08:17.451 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:17 smithi100 ceph-mon[39462]: 6.17 scrub ok 2024-09-17T07:08:17.451 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:17 smithi100 ceph-mon[39462]: 4.12 scrub starts 2024-09-17T07:08:17.451 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:17 smithi100 ceph-mon[39462]: 4.12 scrub ok 2024-09-17T07:08:17.452 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:17 smithi100 ceph-mon[39462]: 5.b scrub starts 2024-09-17T07:08:17.452 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:17 smithi100 ceph-mon[39462]: 5.b scrub ok 2024-09-17T07:08:17.452 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:17 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/469528541' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:08:17.452 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:17 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:08:17.452 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:17 smithi100 ceph-mon[39462]: pgmap v264: 148 pgs: 148 active+clean; 582 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-17T07:08:17.475 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:08:17.516 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:17.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:17.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:08:17.618 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:08:17.654 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:17.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:17.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:08:17.756 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:08:17.789 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:17.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:17.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:08:17.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:17 smithi135 ceph-mon[33238]: 6.17 scrub starts 2024-09-17T07:08:17.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:17 smithi135 ceph-mon[33238]: 6.17 scrub ok 2024-09-17T07:08:17.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:17 smithi135 ceph-mon[33238]: 4.12 scrub starts 2024-09-17T07:08:17.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:17 smithi135 ceph-mon[33238]: 4.12 scrub ok 2024-09-17T07:08:17.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:17 smithi135 ceph-mon[33238]: 5.b scrub starts 2024-09-17T07:08:17.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:17 smithi135 ceph-mon[33238]: 5.b scrub ok 2024-09-17T07:08:17.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:17 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/469528541' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:08:17.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:17 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:08:17.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:17 smithi135 ceph-mon[33238]: pgmap v264: 148 pgs: 148 active+clean; 582 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-17T07:08:17.890 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:08:17.928 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:17.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:17.929 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:08:18.032 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:08:18.042 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 2 16 6720 6704 209.479 225.75 0.00428331 0.00476348 2024-09-17T07:08:18.064 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:18.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:18.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:08:18.166 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:08:18.195 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:18.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:18.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:08:18.297 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:08:18.336 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:18.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:18.337 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:08:18.440 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:08:18.475 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:18.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:18.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:08:18.491 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:18 smithi100 ceph-mon[29092]: 2.7 scrub starts 2024-09-17T07:08:18.491 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:18 smithi100 ceph-mon[29092]: 2.7 scrub ok 2024-09-17T07:08:18.491 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:18 smithi100 ceph-mon[29092]: 6.5 scrub starts 2024-09-17T07:08:18.492 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:18 smithi100 ceph-mon[29092]: 6.5 scrub ok 2024-09-17T07:08:18.492 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:18 smithi100 ceph-mon[29092]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T07:08:18.492 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:18 smithi100 ceph-mon[29092]: Cluster is now healthy 2024-09-17T07:08:18.492 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:18 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T07:08:18.492 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:18 smithi100 ceph-mon[29092]: osdmap e175: 8 total, 8 up, 8 in 2024-09-17T07:08:18.492 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:18 smithi100 ceph-mon[29092]: 2.1f scrub starts 2024-09-17T07:08:18.492 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:18 smithi100 ceph-mon[29092]: 2.1f scrub ok 2024-09-17T07:08:18.492 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:18 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:08:18.492 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:18 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:08:18.492 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:18 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:08:18.493 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:18 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:08:18.493 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:18 smithi100 ceph-mon[29092]: 4.1a scrub starts 2024-09-17T07:08:18.493 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:18 smithi100 ceph-mon[29092]: 4.1a scrub ok 2024-09-17T07:08:18.493 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:18 smithi100 ceph-mon[29092]: 2.19 scrub starts 2024-09-17T07:08:18.493 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:18 smithi100 ceph-mon[29092]: 2.19 scrub ok 2024-09-17T07:08:18.493 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:18 smithi100 ceph-mon[29092]: 2.4 scrub starts 2024-09-17T07:08:18.493 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:18 smithi100 ceph-mon[29092]: 2.4 scrub ok 2024-09-17T07:08:18.493 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:18 smithi100 ceph-mon[29092]: 2.10 deep-scrub starts 2024-09-17T07:08:18.493 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:18 smithi100 ceph-mon[29092]: 2.10 deep-scrub ok 2024-09-17T07:08:18.494 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:18 smithi100 ceph-mon[39462]: 2.7 scrub starts 2024-09-17T07:08:18.494 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:18 smithi100 ceph-mon[39462]: 2.7 scrub ok 2024-09-17T07:08:18.494 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:18 smithi100 ceph-mon[39462]: 6.5 scrub starts 2024-09-17T07:08:18.494 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:18 smithi100 ceph-mon[39462]: 6.5 scrub ok 2024-09-17T07:08:18.494 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:18 smithi100 ceph-mon[39462]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T07:08:18.494 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:18 smithi100 ceph-mon[39462]: Cluster is now healthy 2024-09-17T07:08:18.494 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:18 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T07:08:18.494 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:18 smithi100 ceph-mon[39462]: osdmap e175: 8 total, 8 up, 8 in 2024-09-17T07:08:18.494 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:18 smithi100 ceph-mon[39462]: 2.1f scrub starts 2024-09-17T07:08:18.494 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:18 smithi100 ceph-mon[39462]: 2.1f scrub ok 2024-09-17T07:08:18.494 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:18 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:08:18.495 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:18 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:08:18.495 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:18 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:08:18.495 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:18 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:08:18.495 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:18 smithi100 ceph-mon[39462]: 4.1a scrub starts 2024-09-17T07:08:18.495 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:18 smithi100 ceph-mon[39462]: 4.1a scrub ok 2024-09-17T07:08:18.495 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:18 smithi100 ceph-mon[39462]: 2.19 scrub starts 2024-09-17T07:08:18.495 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:18 smithi100 ceph-mon[39462]: 2.19 scrub ok 2024-09-17T07:08:18.495 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:18 smithi100 ceph-mon[39462]: 2.4 scrub starts 2024-09-17T07:08:18.495 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:18 smithi100 ceph-mon[39462]: 2.4 scrub ok 2024-09-17T07:08:18.495 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:18 smithi100 ceph-mon[39462]: 2.10 deep-scrub starts 2024-09-17T07:08:18.495 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:18 smithi100 ceph-mon[39462]: 2.10 deep-scrub ok 2024-09-17T07:08:18.577 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:08:18.608 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:18.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:18.609 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:08:18.710 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:08:18.743 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:18.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:18.744 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:08:18.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:18 smithi135 ceph-mon[33238]: 2.7 scrub starts 2024-09-17T07:08:18.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:18 smithi135 ceph-mon[33238]: 2.7 scrub ok 2024-09-17T07:08:18.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:18 smithi135 ceph-mon[33238]: 6.5 scrub starts 2024-09-17T07:08:18.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:18 smithi135 ceph-mon[33238]: 6.5 scrub ok 2024-09-17T07:08:18.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:18 smithi135 ceph-mon[33238]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T07:08:18.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:18 smithi135 ceph-mon[33238]: Cluster is now healthy 2024-09-17T07:08:18.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:18 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T07:08:18.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:18 smithi135 ceph-mon[33238]: osdmap e175: 8 total, 8 up, 8 in 2024-09-17T07:08:18.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:18 smithi135 ceph-mon[33238]: 2.1f scrub starts 2024-09-17T07:08:18.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:18 smithi135 ceph-mon[33238]: 2.1f scrub ok 2024-09-17T07:08:18.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:18 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:08:18.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:18 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:08:18.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:18 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:08:18.795 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:18 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:08:18.795 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:18 smithi135 ceph-mon[33238]: 4.1a scrub starts 2024-09-17T07:08:18.795 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:18 smithi135 ceph-mon[33238]: 4.1a scrub ok 2024-09-17T07:08:18.795 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:18 smithi135 ceph-mon[33238]: 2.19 scrub starts 2024-09-17T07:08:18.795 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:18 smithi135 ceph-mon[33238]: 2.19 scrub ok 2024-09-17T07:08:18.795 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:18 smithi135 ceph-mon[33238]: 2.4 scrub starts 2024-09-17T07:08:18.795 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:18 smithi135 ceph-mon[33238]: 2.4 scrub ok 2024-09-17T07:08:18.795 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:18 smithi135 ceph-mon[33238]: 2.10 deep-scrub starts 2024-09-17T07:08:18.795 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:18 smithi135 ceph-mon[33238]: 2.10 deep-scrub ok 2024-09-17T07:08:18.796 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:08:18 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:08:18.845 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:08:18.880 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:18.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:18.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:08:18.981 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:08:19.012 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:19.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:19.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:08:19.042 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 3 16 10060 10044 209.225 208.75 0.00526798 0.00476844 2024-09-17T07:08:19.114 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:08:19.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:19.147 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:19.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:08:19.248 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:08:19.289 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:19.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:19.290 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:08:19.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:19 smithi100 ceph-mon[29092]: 5.7 scrub starts 2024-09-17T07:08:19.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:19 smithi100 ceph-mon[29092]: 5.7 scrub ok 2024-09-17T07:08:19.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:19 smithi100 ceph-mon[29092]: 2.b scrub starts 2024-09-17T07:08:19.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:19 smithi100 ceph-mon[29092]: 2.b scrub ok 2024-09-17T07:08:19.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:19 smithi100 ceph-mon[29092]: osdmap e176: 8 total, 8 up, 8 in 2024-09-17T07:08:19.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:19 smithi100 ceph-mon[29092]: 6.10 scrub starts 2024-09-17T07:08:19.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:19 smithi100 ceph-mon[29092]: 6.10 scrub ok 2024-09-17T07:08:19.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:19 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:08:19.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:19 smithi100 ceph-mon[29092]: 4.14 scrub starts 2024-09-17T07:08:19.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:19 smithi100 ceph-mon[29092]: 4.14 scrub ok 2024-09-17T07:08:19.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:19 smithi100 ceph-mon[29092]: pgmap v267: 148 pgs: 148 active+clean; 493 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 72 MiB/s wr, 1.15k op/s 2024-09-17T07:08:19.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:19 smithi100 ceph-mon[39462]: 5.7 scrub starts 2024-09-17T07:08:19.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:19 smithi100 ceph-mon[39462]: 5.7 scrub ok 2024-09-17T07:08:19.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:19 smithi100 ceph-mon[39462]: 2.b scrub starts 2024-09-17T07:08:19.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:19 smithi100 ceph-mon[39462]: 2.b scrub ok 2024-09-17T07:08:19.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:19 smithi100 ceph-mon[39462]: osdmap e176: 8 total, 8 up, 8 in 2024-09-17T07:08:19.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:19 smithi100 ceph-mon[39462]: 6.10 scrub starts 2024-09-17T07:08:19.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:19 smithi100 ceph-mon[39462]: 6.10 scrub ok 2024-09-17T07:08:19.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:19 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:08:19.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:19 smithi100 ceph-mon[39462]: 4.14 scrub starts 2024-09-17T07:08:19.379 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:19 smithi100 ceph-mon[39462]: 4.14 scrub ok 2024-09-17T07:08:19.379 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:19 smithi100 ceph-mon[39462]: pgmap v267: 148 pgs: 148 active+clean; 493 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 72 MiB/s wr, 1.15k op/s 2024-09-17T07:08:19.391 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:08:19.426 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:19.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:19.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:08:19.528 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:08:19.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:19 smithi135 ceph-mon[33238]: 5.7 scrub starts 2024-09-17T07:08:19.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:19 smithi135 ceph-mon[33238]: 5.7 scrub ok 2024-09-17T07:08:19.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:19 smithi135 ceph-mon[33238]: 2.b scrub starts 2024-09-17T07:08:19.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:19 smithi135 ceph-mon[33238]: 2.b scrub ok 2024-09-17T07:08:19.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:19 smithi135 ceph-mon[33238]: osdmap e176: 8 total, 8 up, 8 in 2024-09-17T07:08:19.545 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:19 smithi135 ceph-mon[33238]: 6.10 scrub starts 2024-09-17T07:08:19.545 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:19 smithi135 ceph-mon[33238]: 6.10 scrub ok 2024-09-17T07:08:19.545 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:19 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:08:19.545 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:19 smithi135 ceph-mon[33238]: 4.14 scrub starts 2024-09-17T07:08:19.545 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:19 smithi135 ceph-mon[33238]: 4.14 scrub ok 2024-09-17T07:08:19.545 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:19 smithi135 ceph-mon[33238]: pgmap v267: 148 pgs: 148 active+clean; 493 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 72 MiB/s wr, 1.15k op/s 2024-09-17T07:08:19.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:19.565 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:19.566 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:08:19.668 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:08:19.701 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:19.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:19.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:08:19.803 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:08:19.809 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:08:19.861 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:19.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:19.862 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:08:19.963 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:08:20.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:20.012 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:20.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:08:20.042 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 4 16 12914 12898 201.509 178.375 0.0059143 0.00495519 2024-09-17T07:08:20.113 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:08:20.141 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:20.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:20.142 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:08:20.158 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T07:08:20.163 INFO:teuthology.orchestra.run.smithi100.stderr:dumped all 2024-09-17T07:08:20.243 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:08:20.279 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:20.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:20.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:08:20.381 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:08:20.408 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:20.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:20.410 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:08:20.510 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:08:20.536 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:20 smithi100 ceph-mon[29092]: 5.1a scrub starts 2024-09-17T07:08:20.537 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:20 smithi100 ceph-mon[29092]: 5.1a scrub ok 2024-09-17T07:08:20.537 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:20 smithi100 ceph-mon[29092]: 4.16 scrub starts 2024-09-17T07:08:20.537 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:20 smithi100 ceph-mon[29092]: 4.16 scrub ok 2024-09-17T07:08:20.537 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:20 smithi100 ceph-mon[29092]: 4.c deep-scrub starts 2024-09-17T07:08:20.537 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:20 smithi100 ceph-mon[29092]: 4.c deep-scrub ok 2024-09-17T07:08:20.538 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:20 smithi100 ceph-mon[29092]: 4.2 scrub starts 2024-09-17T07:08:20.538 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:20 smithi100 ceph-mon[29092]: 4.2 scrub ok 2024-09-17T07:08:20.538 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:20 smithi100 ceph-mon[29092]: 6.14 scrub starts 2024-09-17T07:08:20.538 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:20 smithi100 ceph-mon[29092]: 6.14 scrub ok 2024-09-17T07:08:20.538 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:20 smithi100 ceph-mon[29092]: osdmap e177: 8 total, 8 up, 8 in 2024-09-17T07:08:20.538 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:20 smithi100 ceph-mon[29092]: from='client.15048 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:08:20.538 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:20 smithi100 ceph-mon[39462]: 5.1a scrub starts 2024-09-17T07:08:20.538 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:20 smithi100 ceph-mon[39462]: 5.1a scrub ok 2024-09-17T07:08:20.538 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:20 smithi100 ceph-mon[39462]: 4.16 scrub starts 2024-09-17T07:08:20.538 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:20 smithi100 ceph-mon[39462]: 4.16 scrub ok 2024-09-17T07:08:20.538 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:20 smithi100 ceph-mon[39462]: 4.c deep-scrub starts 2024-09-17T07:08:20.539 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:20 smithi100 ceph-mon[39462]: 4.c deep-scrub ok 2024-09-17T07:08:20.539 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:20 smithi100 ceph-mon[39462]: 4.2 scrub starts 2024-09-17T07:08:20.539 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:20 smithi100 ceph-mon[39462]: 4.2 scrub ok 2024-09-17T07:08:20.539 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:20 smithi100 ceph-mon[39462]: 6.14 scrub starts 2024-09-17T07:08:20.539 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:20 smithi100 ceph-mon[39462]: 6.14 scrub ok 2024-09-17T07:08:20.539 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:20 smithi100 ceph-mon[39462]: osdmap e177: 8 total, 8 up, 8 in 2024-09-17T07:08:20.539 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:20 smithi100 ceph-mon[39462]: from='client.15048 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:08:20.542 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:20.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:20.543 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:08:20.644 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:08:20.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:20.669 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:20.669 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:08:20.770 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:08:20.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:20 smithi135 ceph-mon[33238]: 5.1a scrub starts 2024-09-17T07:08:20.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:20 smithi135 ceph-mon[33238]: 5.1a scrub ok 2024-09-17T07:08:20.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:20 smithi135 ceph-mon[33238]: 4.16 scrub starts 2024-09-17T07:08:20.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:20 smithi135 ceph-mon[33238]: 4.16 scrub ok 2024-09-17T07:08:20.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:20 smithi135 ceph-mon[33238]: 4.c deep-scrub starts 2024-09-17T07:08:20.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:20 smithi135 ceph-mon[33238]: 4.c deep-scrub ok 2024-09-17T07:08:20.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:20 smithi135 ceph-mon[33238]: 4.2 scrub starts 2024-09-17T07:08:20.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:20 smithi135 ceph-mon[33238]: 4.2 scrub ok 2024-09-17T07:08:20.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:20 smithi135 ceph-mon[33238]: 6.14 scrub starts 2024-09-17T07:08:20.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:20 smithi135 ceph-mon[33238]: 6.14 scrub ok 2024-09-17T07:08:20.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:20 smithi135 ceph-mon[33238]: osdmap e177: 8 total, 8 up, 8 in 2024-09-17T07:08:20.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:20 smithi135 ceph-mon[33238]: from='client.15048 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:08:20.806 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:20.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:20.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:08:20.908 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:08:20.942 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:20.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:20.943 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:08:21.042 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 5 16 15472 15456 193.179 159.875 0.00702721 0.00516964 2024-09-17T07:08:21.045 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:08:21.045 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-17T07:08:21.082 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:21.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:21.084 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:08:21.185 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:08:21.208 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:21.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:21.209 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:08:21.310 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:08:21.355 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:21.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:21.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:08:21.457 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:08:21.487 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:21.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:21.489 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:08:21.589 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:08:21.614 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:21 smithi100 ceph-mon[29092]: 5.9 scrub starts 2024-09-17T07:08:21.614 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:21 smithi100 ceph-mon[29092]: 5.9 scrub ok 2024-09-17T07:08:21.614 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:21 smithi100 ceph-mon[29092]: 2.d scrub starts 2024-09-17T07:08:21.614 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:21 smithi100 ceph-mon[29092]: 2.d scrub ok 2024-09-17T07:08:21.614 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:21 smithi100 ceph-mon[29092]: pgmap v269: 148 pgs: 148 active+clean; 493 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 82 MiB/s wr, 1.31k op/s 2024-09-17T07:08:21.614 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:21 smithi100 ceph-mon[39462]: 5.9 scrub starts 2024-09-17T07:08:21.615 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:21 smithi100 ceph-mon[39462]: 5.9 scrub ok 2024-09-17T07:08:21.615 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:21 smithi100 ceph-mon[39462]: 2.d scrub starts 2024-09-17T07:08:21.615 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:21 smithi100 ceph-mon[39462]: 2.d scrub ok 2024-09-17T07:08:21.615 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:21 smithi100 ceph-mon[39462]: pgmap v269: 148 pgs: 148 active+clean; 493 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 82 MiB/s wr, 1.31k op/s 2024-09-17T07:08:21.620 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:21.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:21.621 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:08:21.722 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:08:21.757 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:21.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:21.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:08:21.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:21 smithi135 ceph-mon[33238]: 5.9 scrub starts 2024-09-17T07:08:21.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:21 smithi135 ceph-mon[33238]: 5.9 scrub ok 2024-09-17T07:08:21.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:21 smithi135 ceph-mon[33238]: 2.d scrub starts 2024-09-17T07:08:21.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:21 smithi135 ceph-mon[33238]: 2.d scrub ok 2024-09-17T07:08:21.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:21 smithi135 ceph-mon[33238]: pgmap v269: 148 pgs: 148 active+clean; 493 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 82 MiB/s wr, 1.31k op/s 2024-09-17T07:08:21.859 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:08:21.895 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:21.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:21.897 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:08:21.997 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:08:22.030 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:22.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:22.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:08:22.042 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 6 16 18262 18246 190.043 174.375 0.00278885 0.00525559 2024-09-17T07:08:22.133 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:08:22.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:22.169 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:22.169 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:08:22.270 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:08:22.301 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:22.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:22.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:08:22.403 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:08:22.455 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:22.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:22.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:08:22.557 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:08:22.594 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:22.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:22.595 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:08:22.621 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:22 smithi100 ceph-mon[29092]: 5.18 scrub starts 2024-09-17T07:08:22.621 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:22 smithi100 ceph-mon[29092]: 5.18 scrub ok 2024-09-17T07:08:22.621 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:22 smithi100 ceph-mon[39462]: 5.18 scrub starts 2024-09-17T07:08:22.621 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:22 smithi100 ceph-mon[39462]: 5.18 scrub ok 2024-09-17T07:08:22.696 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:08:22.734 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:22.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:22.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:08:22.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:22 smithi135 ceph-mon[33238]: 5.18 scrub starts 2024-09-17T07:08:22.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:22 smithi135 ceph-mon[33238]: 5.18 scrub ok 2024-09-17T07:08:22.837 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:08:22.866 INFO:teuthology.orchestra.run.smithi100.stdout:{"pg_ready":true,"pg_map":{"version":268,"stamp":"2024-09-17T07:08:20.100871+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":516430102,"num_objects":8067,"num_object_clones":0,"num_object_copies":24201,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8067,"num_whiteouts":0,"num_read":1280,"num_read_kb":1126,"num_write":8480,"num_write_kb":506203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1771214,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8503,"ondisk_log_size":8503,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2571920,"kb_used_data":1189296,"kb_used_omap":12,"kb_used_meta":1382515,"kb_avail":747389296,"statfs":{"total":767960285184,"available":765326639104,"internally_reserved":0,"allocated":1217839104,"data_stored":1214450817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":1415695928},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,5],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":515833856,"num_objects":7871,"num_object_clones":0,"num_object_copies":23613,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7871,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":7871,"num_write_kb":503744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.982446"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":135,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-17T07:08:17.307469+0000","last_change":"2024-09-17T07:07:33.644835+0000","last_active":"2024-09-17T07:08:17.307469+0000","last_peered":"2024-09-17T07:08:17.307469+0000","last_clean":"2024-09-17T07:08:17.307469+0000","last_became_active":"2024-09-17T07:02:18.365773+0000","last_became_peered":"2024-09-17T07:02:18.365773+0000","last_unstale":"2024-09-17T07:08:17.307469+0000","last_undegraded":"2024-09-17T07:08:17.307469+0000","last_fullsized":"2024-09-17T07:08:17.307469+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:33.644728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:33.644728+0000","last_clean_scrub_stamp":"2024-09-17T07:07:33.644728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:46.708857+0000","scrub_duration":0.0070156680000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":153,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-17T07:08:17.332759+0000","last_change":"2024-09-17T07:08:17.332759+0000","last_active":"2024-09-17T07:08:17.332759+0000","last_peered":"2024-09-17T07:08:17.332759+0000","last_clean":"2024-09-17T07:08:17.332759+0000","last_became_active":"2024-09-17T07:02:12.006754+0000","last_became_peered":"2024-09-17T07:02:12.006754+0000","last_unstale":"2024-09-17T07:08:17.332759+0000","last_undegraded":"2024-09-17T07:08:17.332759+0000","last_fullsized":"2024-09-17T07:08:17.332759+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:17.332631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:08:17.332631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:41.165952+0000","scrub_duration":0.0023942220000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"75'15","reported_seq":164,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-17T07:08:17.308693+0000","last_change":"2024-09-17T07:06:29.008173+0000","last_active":"2024-09-17T07:08:17.308693+0000","last_peered":"2024-09-17T07:08:17.308693+0000","last_clean":"2024-09-17T07:08:17.308693+0000","last_became_active":"2024-09-17T07:02:14.012962+0000","last_became_peered":"2024-09-17T07:02:14.012962+0000","last_unstale":"2024-09-17T07:08:17.308693+0000","last_undegraded":"2024-09-17T07:08:17.308693+0000","last_fullsized":"2024-09-17T07:08:17.308693+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-09-17T07:06:29.008025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:06:29.008025+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011622218,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":126,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T07:08:12.278992+0000","last_change":"2024-09-17T07:04:39.726884+0000","last_active":"2024-09-17T07:08:12.278992+0000","last_peered":"2024-09-17T07:08:12.278992+0000","last_clean":"2024-09-17T07:08:12.278992+0000","last_became_active":"2024-09-17T07:02:16.014928+0000","last_became_peered":"2024-09-17T07:02:16.014928+0000","last_unstale":"2024-09-17T07:08:12.278992+0000","last_undegraded":"2024-09-17T07:08:12.278992+0000","last_fullsized":"2024-09-17T07:08:12.278992+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:39.726764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:04:39.726764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013379520000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":142,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-17T07:08:17.308386+0000","last_change":"2024-09-17T07:07:15.767618+0000","last_active":"2024-09-17T07:08:17.308386+0000","last_peered":"2024-09-17T07:08:17.308386+0000","last_clean":"2024-09-17T07:08:17.308386+0000","last_became_active":"2024-09-17T07:02:12.010069+0000","last_became_peered":"2024-09-17T07:02:12.010069+0000","last_unstale":"2024-09-17T07:08:17.308386+0000","last_undegraded":"2024-09-17T07:08:17.308386+0000","last_fullsized":"2024-09-17T07:08:17.308386+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:15.767495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:07:15.767495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:27.964043+0000","scrub_duration":0.013487149,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"75'9","reported_seq":155,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T07:08:12.278998+0000","last_change":"2024-09-17T07:04:46.742364+0000","last_active":"2024-09-17T07:08:12.278998+0000","last_peered":"2024-09-17T07:08:12.278998+0000","last_clean":"2024-09-17T07:08:12.278998+0000","last_became_active":"2024-09-17T07:02:14.016244+0000","last_became_peered":"2024-09-17T07:02:14.016244+0000","last_unstale":"2024-09-17T07:08:12.278998+0000","last_undegraded":"2024-09-17T07:08:12.278998+0000","last_fullsized":"2024-09-17T07:08:12.278998+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:04:46.742196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:04:46.742196+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0030184949999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":137,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T07:08:12.990835+0000","last_change":"2024-09-17T07:07:23.591188+0000","last_active":"2024-09-17T07:08:12.990835+0000","last_peered":"2024-09-17T07:08:12.990835+0000","last_clean":"2024-09-17T07:08:12.990835+0000","last_became_active":"2024-09-17T07:02:16.014380+0000","last_became_peered":"2024-09-17T07:02:16.014380+0000","last_unstale":"2024-09-17T07:08:12.990835+0000","last_undegraded":"2024-09-17T07:08:12.990835+0000","last_fullsized":"2024-09-17T07:08:12.990835+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:23.590927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:07:23.590927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:50.614784+0000","scrub_duration":0.0024742240000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":136,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T07:08:12.279216+0000","last_change":"2024-09-17T07:07:21.791530+0000","last_active":"2024-09-17T07:08:12.279216+0000","last_peered":"2024-09-17T07:08:12.279216+0000","last_clean":"2024-09-17T07:08:12.279216+0000","last_became_active":"2024-09-17T07:02:18.020790+0000","last_became_peered":"2024-09-17T07:02:18.020790+0000","last_unstale":"2024-09-17T07:08:12.279216+0000","last_undegraded":"2024-09-17T07:08:12.279216+0000","last_fullsized":"2024-09-17T07:08:12.279216+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:21.791399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:07:21.791399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:25.370399+0000","scrub_duration":0.0069986889999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":141,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T07:08:13.400004+0000","last_change":"2024-09-17T07:07:23.013609+0000","last_active":"2024-09-17T07:08:13.400004+0000","last_peered":"2024-09-17T07:08:13.400004+0000","last_clean":"2024-09-17T07:08:13.400004+0000","last_became_active":"2024-09-17T07:02:12.011129+0000","last_became_peered":"2024-09-17T07:02:12.011129+0000","last_unstale":"2024-09-17T07:08:13.400004+0000","last_undegraded":"2024-09-17T07:08:13.400004+0000","last_fullsized":"2024-09-17T07:08:13.400004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:23.013470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:43.060102+0000","last_clean_scrub_stamp":"2024-09-17T07:07:23.013470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:28.079025+0000","scrub_duration":0.006146281,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"75'5","reported_seq":150,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T07:08:12.279340+0000","last_change":"2024-09-17T07:07:15.848170+0000","last_active":"2024-09-17T07:08:12.279340+0000","last_peered":"2024-09-17T07:08:12.279340+0000","last_clean":"2024-09-17T07:08:12.279340+0000","last_became_active":"2024-09-17T07:02:14.016640+0000","last_became_peered":"2024-09-17T07:02:14.016640+0000","last_unstale":"2024-09-17T07:08:12.279340+0000","last_undegraded":"2024-09-17T07:08:12.279340+0000","last_fullsized":"2024-09-17T07:08:12.279340+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-09-17T07:07:15.848071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:07:15.848071+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:31.939834+0000","scrub_duration":0.010596959,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":137,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T07:08:13.399265+0000","last_change":"2024-09-17T07:04:55.853224+0000","last_active":"2024-09-17T07:08:13.399265+0000","last_peered":"2024-09-17T07:08:13.399265+0000","last_clean":"2024-09-17T07:08:13.399265+0000","last_became_active":"2024-09-17T07:02:16.015401+0000","last_became_peered":"2024-09-17T07:02:16.015401+0000","last_unstale":"2024-09-17T07:08:13.399265+0000","last_undegraded":"2024-09-17T07:08:13.399265+0000","last_fullsized":"2024-09-17T07:08:13.399265+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:55.853108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:04:55.853108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017379610000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":145,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T07:08:12.990026+0000","last_change":"2024-09-17T07:07:23.739288+0000","last_active":"2024-09-17T07:08:12.990026+0000","last_peered":"2024-09-17T07:08:12.990026+0000","last_clean":"2024-09-17T07:08:12.990026+0000","last_became_active":"2024-09-17T07:02:18.023680+0000","last_became_peered":"2024-09-17T07:02:18.023680+0000","last_unstale":"2024-09-17T07:08:12.990026+0000","last_undegraded":"2024-09-17T07:08:12.990026+0000","last_fullsized":"2024-09-17T07:08:12.990026+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:23.739157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:07:23.739157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:39.559278+0000","scrub_duration":0.004894603,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":141,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T07:08:13.400059+0000","last_change":"2024-09-17T07:07:21.000757+0000","last_active":"2024-09-17T07:08:13.400059+0000","last_peered":"2024-09-17T07:08:13.400059+0000","last_clean":"2024-09-17T07:08:13.400059+0000","last_became_active":"2024-09-17T07:02:12.012234+0000","last_became_peered":"2024-09-17T07:02:12.012234+0000","last_unstale":"2024-09-17T07:08:13.400059+0000","last_undegraded":"2024-09-17T07:08:13.400059+0000","last_fullsized":"2024-09-17T07:08:13.400059+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:21.000635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:45.029247+0000","last_clean_scrub_stamp":"2024-09-17T07:07:21.000635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:23.254719+0000","scrub_duration":0.0068373009999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"75'9","reported_seq":155,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-17T07:08:17.673270+0000","last_change":"2024-09-17T07:08:17.673480+0000","last_active":"2024-09-17T07:08:17.673270+0000","last_peered":"2024-09-17T07:08:17.673270+0000","last_clean":"2024-09-17T07:08:17.673270+0000","last_became_active":"2024-09-17T07:02:14.013337+0000","last_became_peered":"2024-09-17T07:02:14.013337+0000","last_unstale":"2024-09-17T07:08:17.673270+0000","last_undegraded":"2024-09-17T07:08:17.673270+0000","last_fullsized":"2024-09-17T07:08:17.673270+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:08:17.673242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:08:17.673242+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:28.557333+0000","scrub_duration":0.0059611450000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":136,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T07:08:12.990319+0000","last_change":"2024-09-17T07:06:34.739225+0000","last_active":"2024-09-17T07:08:12.990319+0000","last_peered":"2024-09-17T07:08:12.990319+0000","last_clean":"2024-09-17T07:08:12.990319+0000","last_became_active":"2024-09-17T07:02:16.023774+0000","last_became_peered":"2024-09-17T07:02:16.023774+0000","last_unstale":"2024-09-17T07:08:12.990319+0000","last_undegraded":"2024-09-17T07:08:12.990319+0000","last_fullsized":"2024-09-17T07:08:12.990319+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:34.739103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:06:34.739103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0024801699999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":136,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-17T07:08:17.310181+0000","last_change":"2024-09-17T07:07:34.205329+0000","last_active":"2024-09-17T07:08:17.310181+0000","last_peered":"2024-09-17T07:08:17.310181+0000","last_clean":"2024-09-17T07:08:17.310181+0000","last_became_active":"2024-09-17T07:02:18.029178+0000","last_became_peered":"2024-09-17T07:02:18.029178+0000","last_unstale":"2024-09-17T07:08:17.310181+0000","last_undegraded":"2024-09-17T07:08:17.310181+0000","last_fullsized":"2024-09-17T07:08:17.310181+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:34.205205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:07:34.205205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:03.288846+0000","scrub_duration":0.0057819899999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"175'488","reported_seq":503,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-17T07:08:18.155556+0000","last_change":"2024-09-17T07:08:00.759806+0000","last_active":"2024-09-17T07:08:18.155556+0000","last_peered":"2024-09-17T07:08:18.155556+0000","last_clean":"2024-09-17T07:08:18.155556+0000","last_became_active":"2024-09-17T07:08:00.759498+0000","last_became_peered":"2024-09-17T07:08:00.759498+0000","last_unstale":"2024-09-17T07:08:18.155556+0000","last_undegraded":"2024-09-17T07:08:18.155556+0000","last_fullsized":"2024-09-17T07:08:18.155556+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_clean_scrub_stamp":"2024-09-17T07:07:59.220681+0000","objects_scrubbed":0,"log_size":488,"log_dups_size":0,"ondisk_log_size":488,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:28.264255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31981568,"num_objects":488,"num_object_clones":0,"num_object_copies":1464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":488,"num_write_kb":31232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"95'25","reported_seq":175,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T07:08:12.279925+0000","last_change":"2024-09-17T07:06:41.707070+0000","last_active":"2024-09-17T07:08:12.279925+0000","last_peered":"2024-09-17T07:08:12.279925+0000","last_clean":"2024-09-17T07:08:12.279925+0000","last_became_active":"2024-09-17T07:02:14.016270+0000","last_became_peered":"2024-09-17T07:02:14.016270+0000","last_unstale":"2024-09-17T07:08:12.279925+0000","last_undegraded":"2024-09-17T07:08:12.279925+0000","last_fullsized":"2024-09-17T07:08:12.279925+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'25","last_scrub_stamp":"2024-09-17T07:06:41.706948+0000","last_deep_scrub":"95'25","last_deep_scrub_stamp":"2024-09-17T07:04:09.450039+0000","last_clean_scrub_stamp":"2024-09-17T07:06:41.706948+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.016404887999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":152,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-17T07:08:18.088259+0000","last_change":"2024-09-17T07:08:18.088347+0000","last_active":"2024-09-17T07:08:18.088259+0000","last_peered":"2024-09-17T07:08:18.088259+0000","last_clean":"2024-09-17T07:08:18.088259+0000","last_became_active":"2024-09-17T07:02:12.011258+0000","last_became_peered":"2024-09-17T07:02:12.011258+0000","last_unstale":"2024-09-17T07:08:18.088259+0000","last_undegraded":"2024-09-17T07:08:18.088259+0000","last_fullsized":"2024-09-17T07:08:18.088259+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:18.088246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:08:18.088246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:21.774873+0000","scrub_duration":0.0020459860000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":137,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T07:08:12.990873+0000","last_change":"2024-09-17T07:07:24.620899+0000","last_active":"2024-09-17T07:08:12.990873+0000","last_peered":"2024-09-17T07:08:12.990873+0000","last_clean":"2024-09-17T07:08:12.990873+0000","last_became_active":"2024-09-17T07:02:16.018147+0000","last_became_peered":"2024-09-17T07:02:16.018147+0000","last_unstale":"2024-09-17T07:08:12.990873+0000","last_undegraded":"2024-09-17T07:08:12.990873+0000","last_fullsized":"2024-09-17T07:08:12.990873+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:24.620758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:07:24.620758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:54.505017+0000","scrub_duration":0.0050498469999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":103,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-17T07:08:17.309574+0000","last_change":"2024-09-17T07:07:16.737235+0000","last_active":"2024-09-17T07:08:17.309574+0000","last_peered":"2024-09-17T07:08:17.309574+0000","last_clean":"2024-09-17T07:08:17.309574+0000","last_became_active":"2024-09-17T07:04:15.113196+0000","last_became_peered":"2024-09-17T07:04:15.113196+0000","last_unstale":"2024-09-17T07:08:17.309574+0000","last_undegraded":"2024-09-17T07:08:17.309574+0000","last_fullsized":"2024-09-17T07:08:17.309574+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:16.737086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:42.984088+0000","last_clean_scrub_stamp":"2024-09-17T07:07:16.737086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:19.824197+0000","scrub_duration":0.0083716439999999993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"176'564","reported_seq":581,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-17T07:08:18.635727+0000","last_change":"2024-09-17T07:08:00.240249+0000","last_active":"2024-09-17T07:08:18.635727+0000","last_peered":"2024-09-17T07:08:18.635727+0000","last_clean":"2024-09-17T07:08:18.635727+0000","last_became_active":"2024-09-17T07:08:00.239910+0000","last_became_peered":"2024-09-17T07:08:00.239910+0000","last_unstale":"2024-09-17T07:08:18.635727+0000","last_undegraded":"2024-09-17T07:08:18.635727+0000","last_fullsized":"2024-09-17T07:08:18.635727+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_clean_scrub_stamp":"2024-09-17T07:07:59.220681+0000","objects_scrubbed":0,"log_size":564,"log_dups_size":0,"ondisk_log_size":564,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:13.311531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36962304,"num_objects":564,"num_object_clones":0,"num_object_copies":1692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":564,"num_write_kb":36096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"75'10","reported_seq":154,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T07:08:12.279522+0000","last_change":"2024-09-17T07:04:43.722476+0000","last_active":"2024-09-17T07:08:12.279522+0000","last_peered":"2024-09-17T07:08:12.279522+0000","last_clean":"2024-09-17T07:08:12.279522+0000","last_became_active":"2024-09-17T07:02:14.007702+0000","last_became_peered":"2024-09-17T07:02:14.007702+0000","last_unstale":"2024-09-17T07:08:12.279522+0000","last_undegraded":"2024-09-17T07:08:12.279522+0000","last_fullsized":"2024-09-17T07:08:12.279522+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-09-17T07:04:43.722288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:04:43.722288+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002985845,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":140,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T07:08:12.991135+0000","last_change":"2024-09-17T07:06:34.075343+0000","last_active":"2024-09-17T07:08:12.991135+0000","last_peered":"2024-09-17T07:08:12.991135+0000","last_clean":"2024-09-17T07:08:12.991135+0000","last_became_active":"2024-09-17T07:02:12.007553+0000","last_became_peered":"2024-09-17T07:02:12.007553+0000","last_unstale":"2024-09-17T07:08:12.991135+0000","last_undegraded":"2024-09-17T07:08:12.991135+0000","last_fullsized":"2024-09-17T07:08:12.991135+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:34.075214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:06:34.075214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.031637767999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"98'9","reported_seq":170,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T07:08:15.587023+0000","last_change":"2024-09-17T07:06:34.881845+0000","last_active":"2024-09-17T07:08:15.587023+0000","last_peered":"2024-09-17T07:08:15.587023+0000","last_clean":"2024-09-17T07:08:15.587023+0000","last_became_active":"2024-09-17T07:04:15.110797+0000","last_became_peered":"2024-09-17T07:04:15.110797+0000","last_unstale":"2024-09-17T07:08:15.587023+0000","last_undegraded":"2024-09-17T07:08:15.587023+0000","last_fullsized":"2024-09-17T07:08:15.587023+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-09-17T07:06:34.881734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:06:34.881734+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.027541755000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":135,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T07:08:12.279640+0000","last_change":"2024-09-17T07:06:42.703538+0000","last_active":"2024-09-17T07:08:12.279640+0000","last_peered":"2024-09-17T07:08:12.279640+0000","last_clean":"2024-09-17T07:08:12.279640+0000","last_became_active":"2024-09-17T07:02:18.017460+0000","last_became_peered":"2024-09-17T07:02:18.017460+0000","last_unstale":"2024-09-17T07:08:12.279640+0000","last_undegraded":"2024-09-17T07:08:12.279640+0000","last_fullsized":"2024-09-17T07:08:12.279640+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:42.703326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:06:42.703326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004198528,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"175'439","reported_seq":455,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-17T07:08:18.143286+0000","last_change":"2024-09-17T07:08:00.242384+0000","last_active":"2024-09-17T07:08:18.143286+0000","last_peered":"2024-09-17T07:08:18.143286+0000","last_clean":"2024-09-17T07:08:18.143286+0000","last_became_active":"2024-09-17T07:08:00.241996+0000","last_became_peered":"2024-09-17T07:08:00.241996+0000","last_unstale":"2024-09-17T07:08:18.143286+0000","last_undegraded":"2024-09-17T07:08:18.143286+0000","last_fullsized":"2024-09-17T07:08:18.143286+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_clean_scrub_stamp":"2024-09-17T07:07:59.220681+0000","objects_scrubbed":0,"log_size":439,"log_dups_size":0,"ondisk_log_size":439,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:18.953261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28770304,"num_objects":439,"num_object_clones":0,"num_object_copies":1317,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":439,"num_write_kb":28096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"75'15","reported_seq":174,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T07:08:13.784634+0000","last_change":"2024-09-17T07:08:13.784634+0000","last_active":"2024-09-17T07:08:13.784634+0000","last_peered":"2024-09-17T07:08:13.784634+0000","last_clean":"2024-09-17T07:08:13.784634+0000","last_became_active":"2024-09-17T07:02:14.016397+0000","last_became_peered":"2024-09-17T07:02:14.016397+0000","last_unstale":"2024-09-17T07:08:13.784634+0000","last_undegraded":"2024-09-17T07:08:13.784634+0000","last_fullsized":"2024-09-17T07:08:13.784634+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-09-17T07:08:13.784543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:08:13.784543+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:15.750232+0000","scrub_duration":0.002584257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":141,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T07:08:12.990975+0000","last_change":"2024-09-17T07:07:18.550881+0000","last_active":"2024-09-17T07:08:12.990975+0000","last_peered":"2024-09-17T07:08:12.990975+0000","last_clean":"2024-09-17T07:08:12.990975+0000","last_became_active":"2024-09-17T07:02:12.007913+0000","last_became_peered":"2024-09-17T07:02:12.007913+0000","last_unstale":"2024-09-17T07:08:12.990975+0000","last_undegraded":"2024-09-17T07:08:12.990975+0000","last_fullsized":"2024-09-17T07:08:12.990975+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:18.550750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:07:18.550750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:47.637615+0000","scrub_duration":0.012153809,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"75'8","reported_seq":218,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T07:08:15.586979+0000","last_change":"2024-09-17T07:08:09.787205+0000","last_active":"2024-09-17T07:08:15.586979+0000","last_peered":"2024-09-17T07:08:15.586979+0000","last_clean":"2024-09-17T07:08:15.586979+0000","last_became_active":"2024-09-17T07:02:16.021762+0000","last_became_peered":"2024-09-17T07:02:16.021762+0000","last_unstale":"2024-09-17T07:08:15.586979+0000","last_undegraded":"2024-09-17T07:08:15.586979+0000","last_fullsized":"2024-09-17T07:08:15.586979+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:08:09.787123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:08:09.787123+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:27.940895+0000","scrub_duration":0.001487141,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":145,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T07:08:12.991029+0000","last_change":"2024-09-17T07:08:12.991029+0000","last_active":"2024-09-17T07:08:12.991029+0000","last_peered":"2024-09-17T07:08:12.991029+0000","last_clean":"2024-09-17T07:08:12.991029+0000","last_became_active":"2024-09-17T07:02:18.025332+0000","last_became_peered":"2024-09-17T07:02:18.025332+0000","last_unstale":"2024-09-17T07:08:12.991029+0000","last_undegraded":"2024-09-17T07:08:12.991029+0000","last_fullsized":"2024-09-17T07:08:12.991029+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:12.990915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:08:12.990915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:33.309556+0000","scrub_duration":0.0021966199999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.2","version":"175'420","reported_seq":435,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-17T07:08:18.060804+0000","last_change":"2024-09-17T07:08:00.237404+0000","last_active":"2024-09-17T07:08:18.060804+0000","last_peered":"2024-09-17T07:08:18.060804+0000","last_clean":"2024-09-17T07:08:18.060804+0000","last_became_active":"2024-09-17T07:08:00.237068+0000","last_became_peered":"2024-09-17T07:08:00.237068+0000","last_unstale":"2024-09-17T07:08:18.060804+0000","last_undegraded":"2024-09-17T07:08:18.060804+0000","last_fullsized":"2024-09-17T07:08:18.060804+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_clean_scrub_stamp":"2024-09-17T07:07:59.220681+0000","objects_scrubbed":0,"log_size":420,"log_dups_size":0,"ondisk_log_size":420,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:21.514789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27525120,"num_objects":420,"num_object_clones":0,"num_object_copies":1260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":420,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":420,"num_write_kb":26880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"75'11","reported_seq":158,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T07:08:12.279825+0000","last_change":"2024-09-17T07:06:29.766307+0000","last_active":"2024-09-17T07:08:12.279825+0000","last_peered":"2024-09-17T07:08:12.279825+0000","last_clean":"2024-09-17T07:08:12.279825+0000","last_became_active":"2024-09-17T07:02:14.010227+0000","last_became_peered":"2024-09-17T07:02:14.010227+0000","last_unstale":"2024-09-17T07:08:12.279825+0000","last_undegraded":"2024-09-17T07:08:12.279825+0000","last_fullsized":"2024-09-17T07:08:12.279825+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-09-17T07:06:29.766173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:06:29.766173+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0047797760000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":152,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T07:08:13.399327+0000","last_change":"2024-09-17T07:08:06.100857+0000","last_active":"2024-09-17T07:08:13.399327+0000","last_peered":"2024-09-17T07:08:13.399327+0000","last_clean":"2024-09-17T07:08:13.399327+0000","last_became_active":"2024-09-17T07:02:12.013182+0000","last_became_peered":"2024-09-17T07:02:12.013182+0000","last_unstale":"2024-09-17T07:08:13.399327+0000","last_undegraded":"2024-09-17T07:08:13.399327+0000","last_fullsized":"2024-09-17T07:08:13.399327+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:06.100741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:08:06.100741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:26.438221+0000","scrub_duration":0.0012621170000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":147,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T07:08:12.990739+0000","last_change":"2024-09-17T07:07:14.680573+0000","last_active":"2024-09-17T07:08:12.990739+0000","last_peered":"2024-09-17T07:08:12.990739+0000","last_clean":"2024-09-17T07:08:12.990739+0000","last_became_active":"2024-09-17T07:02:16.020224+0000","last_became_peered":"2024-09-17T07:02:16.020224+0000","last_unstale":"2024-09-17T07:08:12.990739+0000","last_undegraded":"2024-09-17T07:08:12.990739+0000","last_fullsized":"2024-09-17T07:08:12.990739+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:14.680446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:07:14.680446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:32.551262+0000","scrub_duration":0.0023342229999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":135,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-17T07:08:17.307701+0000","last_change":"2024-09-17T07:06:30.971537+0000","last_active":"2024-09-17T07:08:17.307701+0000","last_peered":"2024-09-17T07:08:17.307701+0000","last_clean":"2024-09-17T07:08:17.307701+0000","last_became_active":"2024-09-17T07:02:18.365000+0000","last_became_peered":"2024-09-17T07:02:18.365000+0000","last_unstale":"2024-09-17T07:08:17.307701+0000","last_undegraded":"2024-09-17T07:08:17.307701+0000","last_fullsized":"2024-09-17T07:08:17.307701+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:30.971383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:06:30.971383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0078901470000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"175'449","reported_seq":464,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-17T07:08:18.056636+0000","last_change":"2024-09-17T07:08:00.760018+0000","last_active":"2024-09-17T07:08:18.056636+0000","last_peered":"2024-09-17T07:08:18.056636+0000","last_clean":"2024-09-17T07:08:18.056636+0000","last_became_active":"2024-09-17T07:08:00.759711+0000","last_became_peered":"2024-09-17T07:08:00.759711+0000","last_unstale":"2024-09-17T07:08:18.056636+0000","last_undegraded":"2024-09-17T07:08:18.056636+0000","last_fullsized":"2024-09-17T07:08:18.056636+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_clean_scrub_stamp":"2024-09-17T07:07:59.220681+0000","objects_scrubbed":0,"log_size":449,"log_dups_size":0,"ondisk_log_size":449,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:20.125638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29425664,"num_objects":449,"num_object_clones":0,"num_object_copies":1347,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":449,"num_write_kb":28736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1","version":"75'14","reported_seq":160,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T07:08:12.279094+0000","last_change":"2024-09-17T07:08:06.763827+0000","last_active":"2024-09-17T07:08:12.279094+0000","last_peered":"2024-09-17T07:08:12.279094+0000","last_clean":"2024-09-17T07:08:12.279094+0000","last_became_active":"2024-09-17T07:02:14.007810+0000","last_became_peered":"2024-09-17T07:02:14.007810+0000","last_unstale":"2024-09-17T07:08:12.279094+0000","last_undegraded":"2024-09-17T07:08:12.279094+0000","last_fullsized":"2024-09-17T07:08:12.279094+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'14","last_scrub_stamp":"2024-09-17T07:08:06.763689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:08:06.763689+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:34.614294+0000","scrub_duration":0.0033699329999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":162,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T07:08:16.485918+0000","last_change":"2024-09-17T07:08:16.485918+0000","last_active":"2024-09-17T07:08:16.485918+0000","last_peered":"2024-09-17T07:08:16.485918+0000","last_clean":"2024-09-17T07:08:16.485918+0000","last_became_active":"2024-09-17T07:02:12.011766+0000","last_became_peered":"2024-09-17T07:02:12.011766+0000","last_unstale":"2024-09-17T07:08:16.485918+0000","last_undegraded":"2024-09-17T07:08:16.485918+0000","last_fullsized":"2024-09-17T07:08:16.485918+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:16.485782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:08:16.485782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:21.067797+0000","scrub_duration":0.0023681280000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"75'8","reported_seq":219,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-17T07:08:18.313360+0000","last_change":"2024-09-17T07:06:29.982260+0000","last_active":"2024-09-17T07:08:18.313360+0000","last_peered":"2024-09-17T07:08:18.313360+0000","last_clean":"2024-09-17T07:08:18.313360+0000","last_became_active":"2024-09-17T07:02:16.020628+0000","last_became_peered":"2024-09-17T07:02:16.020628+0000","last_unstale":"2024-09-17T07:08:18.313360+0000","last_undegraded":"2024-09-17T07:08:18.313360+0000","last_fullsized":"2024-09-17T07:08:18.313360+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:06:29.982141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:06:29.982141+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0052922769999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":146,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T07:08:13.399383+0000","last_change":"2024-09-17T07:08:11.123128+0000","last_active":"2024-09-17T07:08:13.399383+0000","last_peered":"2024-09-17T07:08:13.399383+0000","last_clean":"2024-09-17T07:08:13.399383+0000","last_became_active":"2024-09-17T07:02:18.364981+0000","last_became_peered":"2024-09-17T07:02:18.364981+0000","last_unstale":"2024-09-17T07:08:13.399383+0000","last_undegraded":"2024-09-17T07:08:13.399383+0000","last_fullsized":"2024-09-17T07:08:13.399383+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:11.123047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:08:11.123047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:27.378027+0000","scrub_duration":0.0014377210000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"176'623","reported_seq":640,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-17T07:08:18.968072+0000","last_change":"2024-09-17T07:08:00.598282+0000","last_active":"2024-09-17T07:08:18.968072+0000","last_peered":"2024-09-17T07:08:18.968072+0000","last_clean":"2024-09-17T07:08:18.968072+0000","last_became_active":"2024-09-17T07:08:00.597939+0000","last_became_peered":"2024-09-17T07:08:00.597939+0000","last_unstale":"2024-09-17T07:08:18.968072+0000","last_undegraded":"2024-09-17T07:08:18.968072+0000","last_fullsized":"2024-09-17T07:08:18.968072+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_clean_scrub_stamp":"2024-09-17T07:07:59.220681+0000","objects_scrubbed":0,"log_size":623,"log_dups_size":0,"ondisk_log_size":623,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:08.411403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40828928,"num_objects":623,"num_object_clones":0,"num_object_copies":1869,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":623,"num_write_kb":39872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"75'18","reported_seq":172,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-17T07:08:18.313563+0000","last_change":"2024-09-17T07:06:40.794767+0000","last_active":"2024-09-17T07:08:18.313563+0000","last_peered":"2024-09-17T07:08:18.313563+0000","last_clean":"2024-09-17T07:08:18.313563+0000","last_became_active":"2024-09-17T07:02:14.028667+0000","last_became_peered":"2024-09-17T07:02:14.028667+0000","last_unstale":"2024-09-17T07:08:18.313563+0000","last_undegraded":"2024-09-17T07:08:18.313563+0000","last_fullsized":"2024-09-17T07:08:18.313563+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'18","last_scrub_stamp":"2024-09-17T07:06:40.794315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:06:40.794315+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.017914407,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":151,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T07:08:12.991233+0000","last_change":"2024-09-17T07:08:04.692191+0000","last_active":"2024-09-17T07:08:12.991233+0000","last_peered":"2024-09-17T07:08:12.991233+0000","last_clean":"2024-09-17T07:08:12.991233+0000","last_became_active":"2024-09-17T07:02:12.005687+0000","last_became_peered":"2024-09-17T07:02:12.005687+0000","last_unstale":"2024-09-17T07:08:12.991233+0000","last_undegraded":"2024-09-17T07:08:12.991233+0000","last_fullsized":"2024-09-17T07:08:12.991233+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:04.692099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:08:04.692099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:09.486554+0000","scrub_duration":0.001661023,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":137,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T07:08:12.279006+0000","last_change":"2024-09-17T07:08:03.726918+0000","last_active":"2024-09-17T07:08:12.279006+0000","last_peered":"2024-09-17T07:08:12.279006+0000","last_clean":"2024-09-17T07:08:12.279006+0000","last_became_active":"2024-09-17T07:02:16.019747+0000","last_became_peered":"2024-09-17T07:02:16.019747+0000","last_unstale":"2024-09-17T07:08:12.279006+0000","last_undegraded":"2024-09-17T07:08:12.279006+0000","last_fullsized":"2024-09-17T07:08:12.279006+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:03.726764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:08:03.726764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:09.587359+0000","scrub_duration":0.002251201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":135,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T07:08:12.279464+0000","last_change":"2024-09-17T07:06:40.684796+0000","last_active":"2024-09-17T07:08:12.279464+0000","last_peered":"2024-09-17T07:08:12.279464+0000","last_clean":"2024-09-17T07:08:12.279464+0000","last_became_active":"2024-09-17T07:02:18.024043+0000","last_became_peered":"2024-09-17T07:02:18.024043+0000","last_unstale":"2024-09-17T07:08:12.279464+0000","last_undegraded":"2024-09-17T07:08:12.279464+0000","last_fullsized":"2024-09-17T07:08:12.279464+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:40.684680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:06:40.684680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011253449,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"176'518","reported_seq":532,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-17T07:08:18.639821+0000","last_change":"2024-09-17T07:08:00.759808+0000","last_active":"2024-09-17T07:08:18.639821+0000","last_peered":"2024-09-17T07:08:18.639821+0000","last_clean":"2024-09-17T07:08:18.639821+0000","last_became_active":"2024-09-17T07:08:00.759502+0000","last_became_peered":"2024-09-17T07:08:00.759502+0000","last_unstale":"2024-09-17T07:08:18.639821+0000","last_undegraded":"2024-09-17T07:08:18.639821+0000","last_fullsized":"2024-09-17T07:08:18.639821+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_clean_scrub_stamp":"2024-09-17T07:07:59.220681+0000","objects_scrubbed":0,"log_size":518,"log_dups_size":0,"ondisk_log_size":518,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:59.866829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33947648,"num_objects":518,"num_object_clones":0,"num_object_copies":1554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":518,"num_write_kb":33152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"75'19","reported_seq":176,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-17T07:08:17.311039+0000","last_change":"2024-09-17T07:07:17.989796+0000","last_active":"2024-09-17T07:08:17.311039+0000","last_peered":"2024-09-17T07:08:17.311039+0000","last_clean":"2024-09-17T07:08:17.311039+0000","last_became_active":"2024-09-17T07:02:14.027389+0000","last_became_peered":"2024-09-17T07:02:14.027389+0000","last_unstale":"2024-09-17T07:08:17.311039+0000","last_undegraded":"2024-09-17T07:08:17.311039+0000","last_fullsized":"2024-09-17T07:08:17.311039+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-09-17T07:07:17.989435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:07:17.989435+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:38.663377+0000","scrub_duration":0.0043356940000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":141,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T07:08:13.400053+0000","last_change":"2024-09-17T07:06:31.799761+0000","last_active":"2024-09-17T07:08:13.400053+0000","last_peered":"2024-09-17T07:08:13.400053+0000","last_clean":"2024-09-17T07:08:13.400053+0000","last_became_active":"2024-09-17T07:02:12.011917+0000","last_became_peered":"2024-09-17T07:02:12.011917+0000","last_unstale":"2024-09-17T07:08:13.400053+0000","last_undegraded":"2024-09-17T07:08:13.400053+0000","last_fullsized":"2024-09-17T07:08:13.400053+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:31.799640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:06:31.799640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00320703,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":137,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T07:08:12.462137+0000","last_change":"2024-09-17T07:07:31.453488+0000","last_active":"2024-09-17T07:08:12.462137+0000","last_peered":"2024-09-17T07:08:12.462137+0000","last_clean":"2024-09-17T07:08:12.462137+0000","last_became_active":"2024-09-17T07:02:16.016199+0000","last_became_peered":"2024-09-17T07:02:16.016199+0000","last_unstale":"2024-09-17T07:08:12.462137+0000","last_undegraded":"2024-09-17T07:08:12.462137+0000","last_fullsized":"2024-09-17T07:08:12.462137+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:31.453355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:07:31.453355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:43.002059+0000","scrub_duration":0.0055939470000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":134,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T07:08:12.990796+0000","last_change":"2024-09-17T07:06:30.992080+0000","last_active":"2024-09-17T07:08:12.990796+0000","last_peered":"2024-09-17T07:08:12.990796+0000","last_clean":"2024-09-17T07:08:12.990796+0000","last_became_active":"2024-09-17T07:02:18.364835+0000","last_became_peered":"2024-09-17T07:02:18.364835+0000","last_unstale":"2024-09-17T07:08:12.990796+0000","last_undegraded":"2024-09-17T07:08:12.990796+0000","last_fullsized":"2024-09-17T07:08:12.990796+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:30.991972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:06:30.991972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045593279999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.e","version":"176'613","reported_seq":630,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-17T07:08:18.966399+0000","last_change":"2024-09-17T07:08:00.759796+0000","last_active":"2024-09-17T07:08:18.966399+0000","last_peered":"2024-09-17T07:08:18.966399+0000","last_clean":"2024-09-17T07:08:18.966399+0000","last_became_active":"2024-09-17T07:08:00.759537+0000","last_became_peered":"2024-09-17T07:08:00.759537+0000","last_unstale":"2024-09-17T07:08:18.966399+0000","last_undegraded":"2024-09-17T07:08:18.966399+0000","last_fullsized":"2024-09-17T07:08:18.966399+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_clean_scrub_stamp":"2024-09-17T07:07:59.220681+0000","objects_scrubbed":0,"log_size":613,"log_dups_size":0,"ondisk_log_size":613,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:27.967133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40173568,"num_objects":613,"num_object_clones":0,"num_object_copies":1839,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":613,"num_write_kb":39232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"110'15","reported_seq":158,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T07:08:12.990989+0000","last_change":"2024-09-17T07:04:57.092268+0000","last_active":"2024-09-17T07:08:12.990989+0000","last_peered":"2024-09-17T07:08:12.990989+0000","last_clean":"2024-09-17T07:08:12.990989+0000","last_became_active":"2024-09-17T07:02:14.010637+0000","last_became_peered":"2024-09-17T07:02:14.010637+0000","last_unstale":"2024-09-17T07:08:12.990989+0000","last_undegraded":"2024-09-17T07:08:12.990989+0000","last_fullsized":"2024-09-17T07:08:12.990989+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'15","last_scrub_stamp":"2024-09-17T07:04:57.091400+0000","last_deep_scrub":"110'15","last_deep_scrub_stamp":"2024-09-17T07:04:57.091400+0000","last_clean_scrub_stamp":"2024-09-17T07:04:57.091400+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012740888000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":151,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-17T07:08:17.806026+0000","last_change":"2024-09-17T07:08:17.806138+0000","last_active":"2024-09-17T07:08:17.806026+0000","last_peered":"2024-09-17T07:08:17.806026+0000","last_clean":"2024-09-17T07:08:17.806026+0000","last_became_active":"2024-09-17T07:02:12.010803+0000","last_became_peered":"2024-09-17T07:02:12.010803+0000","last_unstale":"2024-09-17T07:08:17.806026+0000","last_undegraded":"2024-09-17T07:08:17.806026+0000","last_fullsized":"2024-09-17T07:08:17.806026+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:17.806001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:25.147878+0000","last_clean_scrub_stamp":"2024-09-17T07:08:17.806001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:29.869447+0000","scrub_duration":0.00606531,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"89'9","reported_seq":231,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-17T07:08:17.310927+0000","last_change":"2024-09-17T07:08:14.309093+0000","last_active":"2024-09-17T07:08:17.310927+0000","last_peered":"2024-09-17T07:08:17.310927+0000","last_clean":"2024-09-17T07:08:17.310927+0000","last_became_active":"2024-09-17T07:02:16.021360+0000","last_became_peered":"2024-09-17T07:02:16.021360+0000","last_unstale":"2024-09-17T07:08:17.310927+0000","last_undegraded":"2024-09-17T07:08:17.310927+0000","last_fullsized":"2024-09-17T07:08:17.310927+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'9","last_scrub_stamp":"2024-09-17T07:08:14.308970+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-09-17T07:03:29.961439+0000","last_clean_scrub_stamp":"2024-09-17T07:08:14.308970+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:21.985874+0000","scrub_duration":0.001890011,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"75'1","reported_seq":137,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-17T07:08:17.310016+0000","last_change":"2024-09-17T07:06:25.992349+0000","last_active":"2024-09-17T07:08:17.310016+0000","last_peered":"2024-09-17T07:08:17.310016+0000","last_clean":"2024-09-17T07:08:17.310016+0000","last_became_active":"2024-09-17T07:02:18.029772+0000","last_became_peered":"2024-09-17T07:02:18.029772+0000","last_unstale":"2024-09-17T07:08:17.310016+0000","last_undegraded":"2024-09-17T07:08:17.310016+0000","last_fullsized":"2024-09-17T07:08:17.310016+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-17T07:06:25.992208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:06:25.992208+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010034507,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"175'439","reported_seq":455,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-17T07:08:18.041830+0000","last_change":"2024-09-17T07:08:00.598765+0000","last_active":"2024-09-17T07:08:18.041830+0000","last_peered":"2024-09-17T07:08:18.041830+0000","last_clean":"2024-09-17T07:08:18.041830+0000","last_became_active":"2024-09-17T07:08:00.598467+0000","last_became_peered":"2024-09-17T07:08:00.598467+0000","last_unstale":"2024-09-17T07:08:18.041830+0000","last_undegraded":"2024-09-17T07:08:18.041830+0000","last_fullsized":"2024-09-17T07:08:18.041830+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_clean_scrub_stamp":"2024-09-17T07:07:59.220681+0000","objects_scrubbed":0,"log_size":439,"log_dups_size":0,"ondisk_log_size":439,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:19.831763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28770304,"num_objects":439,"num_object_clones":0,"num_object_copies":1317,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":439,"num_write_kb":28096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"75'28","reported_seq":259,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T07:08:15.586810+0000","last_change":"2024-09-17T07:06:45.186860+0000","last_active":"2024-09-17T07:08:15.586810+0000","last_peered":"2024-09-17T07:08:15.586810+0000","last_clean":"2024-09-17T07:08:15.586810+0000","last_became_active":"2024-09-17T07:02:14.017377+0000","last_became_peered":"2024-09-17T07:02:14.017377+0000","last_unstale":"2024-09-17T07:08:15.586810+0000","last_undegraded":"2024-09-17T07:08:15.586810+0000","last_fullsized":"2024-09-17T07:08:15.586810+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'28","last_scrub_stamp":"2024-09-17T07:06:45.186408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:06:45.186408+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010603473,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":151,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T07:08:12.990501+0000","last_change":"2024-09-17T07:06:43.763211+0000","last_active":"2024-09-17T07:08:12.990501+0000","last_peered":"2024-09-17T07:08:12.990501+0000","last_clean":"2024-09-17T07:08:12.990501+0000","last_became_active":"2024-09-17T07:02:12.008802+0000","last_became_peered":"2024-09-17T07:02:12.008802+0000","last_unstale":"2024-09-17T07:08:12.990501+0000","last_undegraded":"2024-09-17T07:08:12.990501+0000","last_fullsized":"2024-09-17T07:08:12.990501+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:43.763044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:06:43.763044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003861303,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":149,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-17T07:08:17.310066+0000","last_change":"2024-09-17T07:08:04.266668+0000","last_active":"2024-09-17T07:08:17.310066+0000","last_peered":"2024-09-17T07:08:17.310066+0000","last_clean":"2024-09-17T07:08:17.310066+0000","last_became_active":"2024-09-17T07:02:16.016606+0000","last_became_peered":"2024-09-17T07:02:16.016606+0000","last_unstale":"2024-09-17T07:08:17.310066+0000","last_undegraded":"2024-09-17T07:08:17.310066+0000","last_fullsized":"2024-09-17T07:08:17.310066+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:04.266574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:08:04.266574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:04.575911+0000","scrub_duration":0.0014388140000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":135,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-17T07:08:17.308130+0000","last_change":"2024-09-17T07:08:08.849222+0000","last_active":"2024-09-17T07:08:17.308130+0000","last_peered":"2024-09-17T07:08:17.308130+0000","last_clean":"2024-09-17T07:08:17.308130+0000","last_became_active":"2024-09-17T07:02:18.024602+0000","last_became_peered":"2024-09-17T07:02:18.024602+0000","last_unstale":"2024-09-17T07:08:17.308130+0000","last_undegraded":"2024-09-17T07:08:17.308130+0000","last_fullsized":"2024-09-17T07:08:17.308130+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:08.849142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:08:08.849142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:09.331821+0000","scrub_duration":0.0014278870000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.b","version":"176'556","reported_seq":572,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-17T07:08:18.637596+0000","last_change":"2024-09-17T07:08:00.243217+0000","last_active":"2024-09-17T07:08:18.637596+0000","last_peered":"2024-09-17T07:08:18.637596+0000","last_clean":"2024-09-17T07:08:18.637596+0000","last_became_active":"2024-09-17T07:08:00.242842+0000","last_became_peered":"2024-09-17T07:08:00.242842+0000","last_unstale":"2024-09-17T07:08:18.637596+0000","last_undegraded":"2024-09-17T07:08:18.637596+0000","last_fullsized":"2024-09-17T07:08:18.637596+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_clean_scrub_stamp":"2024-09-17T07:07:59.220681+0000","objects_scrubbed":0,"log_size":556,"log_dups_size":0,"ondisk_log_size":556,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:03.017656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36438016,"num_objects":556,"num_object_clones":0,"num_object_copies":1668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":556,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":556,"num_write_kb":35584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"75'13","reported_seq":176,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T07:08:12.991524+0000","last_change":"2024-09-17T07:08:05.678169+0000","last_active":"2024-09-17T07:08:12.991524+0000","last_peered":"2024-09-17T07:08:12.991524+0000","last_clean":"2024-09-17T07:08:12.991524+0000","last_became_active":"2024-09-17T07:02:14.019838+0000","last_became_peered":"2024-09-17T07:02:14.019838+0000","last_unstale":"2024-09-17T07:08:12.991524+0000","last_undegraded":"2024-09-17T07:08:12.991524+0000","last_fullsized":"2024-09-17T07:08:12.991524+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'13","last_scrub_stamp":"2024-09-17T07:08:05.678063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:08:05.678063+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:17.545005+0000","scrub_duration":0.0024636459999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"76'5","reported_seq":508,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-17T07:08:17.653953+0000","last_change":"2024-09-17T07:06:46.230250+0000","last_active":"2024-09-17T07:08:17.653953+0000","last_peered":"2024-09-17T07:08:17.653953+0000","last_clean":"2024-09-17T07:08:17.653953+0000","last_became_active":"2024-09-17T07:02:12.006883+0000","last_became_peered":"2024-09-17T07:02:12.006883+0000","last_unstale":"2024-09-17T07:08:17.653953+0000","last_undegraded":"2024-09-17T07:08:17.653953+0000","last_fullsized":"2024-09-17T07:08:17.653953+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'5","last_scrub_stamp":"2024-09-17T07:06:46.229804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:06:46.229804+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.064119354000000003,"objects_trimmed":0,"snaptrim_duration":0.00033739799999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":359,"num_read_kb":354,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":102,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-17T07:08:17.308760+0000","last_change":"2024-09-17T07:04:53.698772+0000","last_active":"2024-09-17T07:08:17.308760+0000","last_peered":"2024-09-17T07:08:17.308760+0000","last_clean":"2024-09-17T07:08:17.308760+0000","last_became_active":"2024-09-17T07:04:15.109950+0000","last_became_peered":"2024-09-17T07:04:15.109950+0000","last_unstale":"2024-09-17T07:08:17.308760+0000","last_undegraded":"2024-09-17T07:08:17.308760+0000","last_fullsized":"2024-09-17T07:08:17.308760+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:53.698301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:04:53.698301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045574200000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":148,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T07:08:14.935764+0000","last_change":"2024-09-17T07:08:14.935764+0000","last_active":"2024-09-17T07:08:14.935764+0000","last_peered":"2024-09-17T07:08:14.935764+0000","last_clean":"2024-09-17T07:08:14.935764+0000","last_became_active":"2024-09-17T07:02:16.014492+0000","last_became_peered":"2024-09-17T07:02:16.014492+0000","last_unstale":"2024-09-17T07:08:14.935764+0000","last_undegraded":"2024-09-17T07:08:14.935764+0000","last_fullsized":"2024-09-17T07:08:14.935764+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:14.935679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:08:14.935679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:37.535235+0000","scrub_duration":0.001590459,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":146,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T07:08:17.127147+0000","last_change":"2024-09-17T07:08:17.127147+0000","last_active":"2024-09-17T07:08:17.127147+0000","last_peered":"2024-09-17T07:08:17.127147+0000","last_clean":"2024-09-17T07:08:17.127147+0000","last_became_active":"2024-09-17T07:02:18.365369+0000","last_became_peered":"2024-09-17T07:02:18.365369+0000","last_unstale":"2024-09-17T07:08:17.127147+0000","last_undegraded":"2024-09-17T07:08:17.127147+0000","last_fullsized":"2024-09-17T07:08:17.127147+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:17.127034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:08:17.127034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:20.871917+0000","scrub_duration":0.002677081,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"175'404","reported_seq":420,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-17T07:08:18.158198+0000","last_change":"2024-09-17T07:08:00.760107+0000","last_active":"2024-09-17T07:08:18.158198+0000","last_peered":"2024-09-17T07:08:18.158198+0000","last_clean":"2024-09-17T07:08:18.158198+0000","last_became_active":"2024-09-17T07:08:00.759827+0000","last_became_peered":"2024-09-17T07:08:00.759827+0000","last_unstale":"2024-09-17T07:08:18.158198+0000","last_undegraded":"2024-09-17T07:08:18.158198+0000","last_fullsized":"2024-09-17T07:08:18.158198+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_clean_scrub_stamp":"2024-09-17T07:07:59.220681+0000","objects_scrubbed":0,"log_size":404,"log_dups_size":0,"ondisk_log_size":404,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:16.600109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26476544,"num_objects":404,"num_object_clones":0,"num_object_copies":1212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":404,"num_write_kb":25856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"75'12","reported_seq":192,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-17T07:08:17.311214+0000","last_change":"2024-09-17T07:08:08.295553+0000","last_active":"2024-09-17T07:08:17.311214+0000","last_peered":"2024-09-17T07:08:17.311214+0000","last_clean":"2024-09-17T07:08:17.311214+0000","last_became_active":"2024-09-17T07:03:48.760405+0000","last_became_peered":"2024-09-17T07:03:48.760405+0000","last_unstale":"2024-09-17T07:08:17.311214+0000","last_undegraded":"2024-09-17T07:08:17.311214+0000","last_fullsized":"2024-09-17T07:08:17.311214+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-09-17T07:08:08.295449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:08:08.295449+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:37.355090+0000","scrub_duration":0.0023245919999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"70'1","reported_seq":148,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-17T07:08:17.312299+0000","last_change":"2024-09-17T07:07:21.050377+0000","last_active":"2024-09-17T07:08:17.312299+0000","last_peered":"2024-09-17T07:08:17.312299+0000","last_clean":"2024-09-17T07:08:17.312299+0000","last_became_active":"2024-09-17T07:02:12.009476+0000","last_became_peered":"2024-09-17T07:02:12.009476+0000","last_unstale":"2024-09-17T07:08:17.312299+0000","last_undegraded":"2024-09-17T07:08:17.312299+0000","last_fullsized":"2024-09-17T07:08:17.312299+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-17T07:07:21.050247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:07:21.050247+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:38.188477+0000","scrub_duration":0.005277505,"objects_trimmed":0,"snaptrim_duration":0.0032298629999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":142,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-17T07:08:17.314461+0000","last_change":"2024-09-17T07:08:04.086805+0000","last_active":"2024-09-17T07:08:17.314461+0000","last_peered":"2024-09-17T07:08:17.314461+0000","last_clean":"2024-09-17T07:08:17.314461+0000","last_became_active":"2024-09-17T07:02:12.014623+0000","last_became_peered":"2024-09-17T07:02:12.014623+0000","last_unstale":"2024-09-17T07:08:17.314461+0000","last_undegraded":"2024-09-17T07:08:17.314461+0000","last_fullsized":"2024-09-17T07:08:17.314461+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:04.086709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:08:04.086709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:14.260720+0000","scrub_duration":0.0017330640000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":147,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-17T07:08:18.008626+0000","last_change":"2024-09-17T07:08:18.008734+0000","last_active":"2024-09-17T07:08:18.008626+0000","last_peered":"2024-09-17T07:08:18.008626+0000","last_clean":"2024-09-17T07:08:18.008626+0000","last_became_active":"2024-09-17T07:02:16.021561+0000","last_became_peered":"2024-09-17T07:02:16.021561+0000","last_unstale":"2024-09-17T07:08:18.008626+0000","last_undegraded":"2024-09-17T07:08:18.008626+0000","last_fullsized":"2024-09-17T07:08:18.008626+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:18.008600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:08:18.008600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:38.354418+0000","scrub_duration":0.0044198520000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":134,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T07:08:12.990859+0000","last_change":"2024-09-17T07:07:25.570226+0000","last_active":"2024-09-17T07:08:12.990859+0000","last_peered":"2024-09-17T07:08:12.990859+0000","last_clean":"2024-09-17T07:08:12.990859+0000","last_became_active":"2024-09-17T07:02:18.022082+0000","last_became_peered":"2024-09-17T07:02:18.022082+0000","last_unstale":"2024-09-17T07:08:12.990859+0000","last_undegraded":"2024-09-17T07:08:12.990859+0000","last_fullsized":"2024-09-17T07:08:12.990859+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:25.570107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:07:25.570107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:37.811807+0000","scrub_duration":0.0049089310000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"175'416","reported_seq":432,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-17T07:08:18.113896+0000","last_change":"2024-09-17T07:08:00.243111+0000","last_active":"2024-09-17T07:08:18.113896+0000","last_peered":"2024-09-17T07:08:18.113896+0000","last_clean":"2024-09-17T07:08:18.113896+0000","last_became_active":"2024-09-17T07:08:00.242803+0000","last_became_peered":"2024-09-17T07:08:00.242803+0000","last_unstale":"2024-09-17T07:08:18.113896+0000","last_undegraded":"2024-09-17T07:08:18.113896+0000","last_fullsized":"2024-09-17T07:08:18.113896+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_clean_scrub_stamp":"2024-09-17T07:07:59.220681+0000","objects_scrubbed":0,"log_size":416,"log_dups_size":0,"ondisk_log_size":416,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:17.233293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27262976,"num_objects":416,"num_object_clones":0,"num_object_copies":1248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":416,"num_write_kb":26624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"75'14","reported_seq":161,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T07:08:12.462749+0000","last_change":"2024-09-17T07:07:26.395820+0000","last_active":"2024-09-17T07:08:12.462749+0000","last_peered":"2024-09-17T07:08:12.462749+0000","last_clean":"2024-09-17T07:08:12.462749+0000","last_became_active":"2024-09-17T07:02:14.028857+0000","last_became_peered":"2024-09-17T07:02:14.028857+0000","last_unstale":"2024-09-17T07:08:12.462749+0000","last_undegraded":"2024-09-17T07:08:12.462749+0000","last_fullsized":"2024-09-17T07:08:12.462749+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'14","last_scrub_stamp":"2024-09-17T07:07:26.395429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:07:26.395429+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:27.868879+0000","scrub_duration":0.0028559779999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"72'2","reported_seq":159,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-17T07:08:17.307712+0000","last_change":"2024-09-17T07:08:15.763809+0000","last_active":"2024-09-17T07:08:17.307712+0000","last_peered":"2024-09-17T07:08:17.307712+0000","last_clean":"2024-09-17T07:08:17.307712+0000","last_became_active":"2024-09-17T07:02:12.005127+0000","last_became_peered":"2024-09-17T07:02:12.005127+0000","last_unstale":"2024-09-17T07:08:17.307712+0000","last_undegraded":"2024-09-17T07:08:17.307712+0000","last_fullsized":"2024-09-17T07:08:17.307712+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'2","last_scrub_stamp":"2024-09-17T07:08:15.763727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:08:15.763727+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:28.540387+0000","scrub_duration":0.0016958979999999999,"objects_trimmed":0,"snaptrim_duration":0.0010980219999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":151,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T07:08:12.990466+0000","last_change":"2024-09-17T07:06:37.732110+0000","last_active":"2024-09-17T07:08:12.990466+0000","last_peered":"2024-09-17T07:08:12.990466+0000","last_clean":"2024-09-17T07:08:12.990466+0000","last_became_active":"2024-09-17T07:02:12.004606+0000","last_became_peered":"2024-09-17T07:02:12.004606+0000","last_unstale":"2024-09-17T07:08:12.990466+0000","last_undegraded":"2024-09-17T07:08:12.990466+0000","last_fullsized":"2024-09-17T07:08:12.990466+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:37.731844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:06:37.731844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0070683819999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"78'128","reported_seq":401,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T07:08:14.128641+0000","last_change":"2024-09-17T07:08:14.128641+0000","last_active":"2024-09-17T07:08:14.128641+0000","last_peered":"2024-09-17T07:08:14.128641+0000","last_clean":"2024-09-17T07:08:14.128641+0000","last_became_active":"2024-09-17T07:02:03.020675+0000","last_became_peered":"2024-09-17T07:02:03.020675+0000","last_unstale":"2024-09-17T07:08:14.128641+0000","last_undegraded":"2024-09-17T07:08:14.128641+0000","last_fullsized":"2024-09-17T07:08:14.128641+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'128","last_scrub_stamp":"2024-09-17T07:08:14.128519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:00:09.390381+0000","last_clean_scrub_stamp":"2024-09-17T07:08:14.128519+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:29.403199+0000","scrub_duration":0.0021141889999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":148,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T07:08:13.400002+0000","last_change":"2024-09-17T07:08:05.064380+0000","last_active":"2024-09-17T07:08:13.400002+0000","last_peered":"2024-09-17T07:08:13.400002+0000","last_clean":"2024-09-17T07:08:13.400002+0000","last_became_active":"2024-09-17T07:02:16.016031+0000","last_became_peered":"2024-09-17T07:02:16.016031+0000","last_unstale":"2024-09-17T07:08:13.400002+0000","last_undegraded":"2024-09-17T07:08:13.400002+0000","last_fullsized":"2024-09-17T07:08:13.400002+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:05.064241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:08:05.064241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:28.339433+0000","scrub_duration":0.0015017990000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":134,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T07:08:12.990922+0000","last_change":"2024-09-17T07:06:30.808658+0000","last_active":"2024-09-17T07:08:12.990922+0000","last_peered":"2024-09-17T07:08:12.990922+0000","last_clean":"2024-09-17T07:08:12.990922+0000","last_became_active":"2024-09-17T07:02:18.023321+0000","last_became_peered":"2024-09-17T07:02:18.023321+0000","last_unstale":"2024-09-17T07:08:12.990922+0000","last_undegraded":"2024-09-17T07:08:12.990922+0000","last_fullsized":"2024-09-17T07:08:12.990922+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:30.808562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:58.670571+0000","last_clean_scrub_stamp":"2024-09-17T07:06:30.808562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.005261338,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"176'623","reported_seq":640,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-17T07:08:18.972566+0000","last_change":"2024-09-17T07:08:00.599954+0000","last_active":"2024-09-17T07:08:18.972566+0000","last_peered":"2024-09-17T07:08:18.972566+0000","last_clean":"2024-09-17T07:08:18.972566+0000","last_became_active":"2024-09-17T07:08:00.599685+0000","last_became_peered":"2024-09-17T07:08:00.599685+0000","last_unstale":"2024-09-17T07:08:18.972566+0000","last_undegraded":"2024-09-17T07:08:18.972566+0000","last_fullsized":"2024-09-17T07:08:18.972566+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_clean_scrub_stamp":"2024-09-17T07:07:59.220681+0000","objects_scrubbed":0,"log_size":623,"log_dups_size":0,"ondisk_log_size":623,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:08.732487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40828928,"num_objects":623,"num_object_clones":0,"num_object_copies":1869,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":623,"num_write_kb":39872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"75'19","reported_seq":170,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T07:08:12.462316+0000","last_change":"2024-09-17T07:07:18.384022+0000","last_active":"2024-09-17T07:08:12.462316+0000","last_peered":"2024-09-17T07:08:12.462316+0000","last_clean":"2024-09-17T07:08:12.462316+0000","last_became_active":"2024-09-17T07:02:14.027313+0000","last_became_peered":"2024-09-17T07:02:14.027313+0000","last_unstale":"2024-09-17T07:08:12.462316+0000","last_undegraded":"2024-09-17T07:08:12.462316+0000","last_fullsized":"2024-09-17T07:08:12.462316+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-09-17T07:07:18.383813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:07:18.383813+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:42.977846+0000","scrub_duration":0.0040351470000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":163,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T07:08:15.952173+0000","last_change":"2024-09-17T07:08:15.952173+0000","last_active":"2024-09-17T07:08:15.952173+0000","last_peered":"2024-09-17T07:08:15.952173+0000","last_clean":"2024-09-17T07:08:15.952173+0000","last_became_active":"2024-09-17T07:02:12.012037+0000","last_became_peered":"2024-09-17T07:02:12.012037+0000","last_unstale":"2024-09-17T07:08:15.952173+0000","last_undegraded":"2024-09-17T07:08:15.952173+0000","last_fullsized":"2024-09-17T07:08:15.952173+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:15.952078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:06:39.322535+0000","last_clean_scrub_stamp":"2024-09-17T07:08:15.952078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:25.194960+0000","scrub_duration":0.0017662470000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":159,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T07:08:17.003265+0000","last_change":"2024-09-17T07:08:17.003265+0000","last_active":"2024-09-17T07:08:17.003265+0000","last_peered":"2024-09-17T07:08:17.003265+0000","last_clean":"2024-09-17T07:08:17.003265+0000","last_became_active":"2024-09-17T07:02:16.014631+0000","last_became_peered":"2024-09-17T07:02:16.014631+0000","last_unstale":"2024-09-17T07:08:17.003265+0000","last_undegraded":"2024-09-17T07:08:17.003265+0000","last_fullsized":"2024-09-17T07:08:17.003265+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:17.003116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:08:17.003116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:32.776213+0000","scrub_duration":0.0035024880000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":135,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T07:08:13.399451+0000","last_change":"2024-09-17T07:07:27.004939+0000","last_active":"2024-09-17T07:08:13.399451+0000","last_peered":"2024-09-17T07:08:13.399451+0000","last_clean":"2024-09-17T07:08:13.399451+0000","last_became_active":"2024-09-17T07:02:18.023299+0000","last_became_peered":"2024-09-17T07:02:18.023299+0000","last_unstale":"2024-09-17T07:08:13.399451+0000","last_undegraded":"2024-09-17T07:08:13.399451+0000","last_fullsized":"2024-09-17T07:08:13.399451+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:27.004683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:07:27.004683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:37.142184+0000","scrub_duration":0.010022120000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.7","version":"175'438","reported_seq":453,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-17T07:08:18.158544+0000","last_change":"2024-09-17T07:08:00.245682+0000","last_active":"2024-09-17T07:08:18.158544+0000","last_peered":"2024-09-17T07:08:18.158544+0000","last_clean":"2024-09-17T07:08:18.158544+0000","last_became_active":"2024-09-17T07:08:00.245345+0000","last_became_peered":"2024-09-17T07:08:00.245345+0000","last_unstale":"2024-09-17T07:08:18.158544+0000","last_undegraded":"2024-09-17T07:08:18.158544+0000","last_fullsized":"2024-09-17T07:08:18.158544+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_clean_scrub_stamp":"2024-09-17T07:07:59.220681+0000","objects_scrubbed":0,"log_size":438,"log_dups_size":0,"ondisk_log_size":438,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:26.022804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28704768,"num_objects":438,"num_object_clones":0,"num_object_copies":1314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":438,"num_write_kb":28032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"75'9","reported_seq":156,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-17T07:08:17.311006+0000","last_change":"2024-09-17T07:06:26.996048+0000","last_active":"2024-09-17T07:08:17.311006+0000","last_peered":"2024-09-17T07:08:17.311006+0000","last_clean":"2024-09-17T07:08:17.311006+0000","last_became_active":"2024-09-17T07:02:14.014939+0000","last_became_peered":"2024-09-17T07:02:14.014939+0000","last_unstale":"2024-09-17T07:08:17.311006+0000","last_undegraded":"2024-09-17T07:08:17.311006+0000","last_fullsized":"2024-09-17T07:08:17.311006+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:06:26.995923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:06:26.995923+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.01417006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":140,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T07:08:12.991364+0000","last_change":"2024-09-17T07:06:30.038396+0000","last_active":"2024-09-17T07:08:12.991364+0000","last_peered":"2024-09-17T07:08:12.991364+0000","last_clean":"2024-09-17T07:08:12.991364+0000","last_became_active":"2024-09-17T07:02:12.000005+0000","last_became_peered":"2024-09-17T07:02:12.000005+0000","last_unstale":"2024-09-17T07:08:12.991364+0000","last_undegraded":"2024-09-17T07:08:12.991364+0000","last_fullsized":"2024-09-17T07:08:12.991364+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:30.038104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:06:30.038104+0000","last_clean_scrub_stamp":"2024-09-17T07:06:30.038104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.021438763,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":102,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T07:08:12.279214+0000","last_change":"2024-09-17T07:07:19.789263+0000","last_active":"2024-09-17T07:08:12.279214+0000","last_peered":"2024-09-17T07:08:12.279214+0000","last_clean":"2024-09-17T07:08:12.279214+0000","last_became_active":"2024-09-17T07:04:15.107886+0000","last_became_peered":"2024-09-17T07:04:15.107886+0000","last_unstale":"2024-09-17T07:08:12.279214+0000","last_undegraded":"2024-09-17T07:08:12.279214+0000","last_fullsized":"2024-09-17T07:08:12.279214+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:19.789134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:07:19.789134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:20.148885+0000","scrub_duration":0.010505654,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":136,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-17T07:08:17.312220+0000","last_change":"2024-09-17T07:07:20.025023+0000","last_active":"2024-09-17T07:08:17.312220+0000","last_peered":"2024-09-17T07:08:17.312220+0000","last_clean":"2024-09-17T07:08:17.312220+0000","last_became_active":"2024-09-17T07:02:18.026994+0000","last_became_peered":"2024-09-17T07:02:18.026994+0000","last_unstale":"2024-09-17T07:08:17.312220+0000","last_undegraded":"2024-09-17T07:08:17.312220+0000","last_fullsized":"2024-09-17T07:08:17.312220+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:20.024902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:58.099849+0000","last_clean_scrub_stamp":"2024-09-17T07:07:20.024902+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:30.210770+0000","scrub_duration":0.013970381,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"175'429","reported_seq":445,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-17T07:08:18.045261+0000","last_change":"2024-09-17T07:08:00.760307+0000","last_active":"2024-09-17T07:08:18.045261+0000","last_peered":"2024-09-17T07:08:18.045261+0000","last_clean":"2024-09-17T07:08:18.045261+0000","last_became_active":"2024-09-17T07:08:00.759662+0000","last_became_peered":"2024-09-17T07:08:00.759662+0000","last_unstale":"2024-09-17T07:08:18.045261+0000","last_undegraded":"2024-09-17T07:08:18.045261+0000","last_fullsized":"2024-09-17T07:08:18.045261+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_clean_scrub_stamp":"2024-09-17T07:07:59.220681+0000","objects_scrubbed":0,"log_size":429,"log_dups_size":0,"ondisk_log_size":429,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:17.112326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28114944,"num_objects":429,"num_object_clones":0,"num_object_copies":1287,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":429,"num_write_kb":27456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"111'22","reported_seq":181,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T07:08:12.989876+0000","last_change":"2024-09-17T07:07:29.816377+0000","last_active":"2024-09-17T07:08:12.989876+0000","last_peered":"2024-09-17T07:08:12.989876+0000","last_clean":"2024-09-17T07:08:12.989876+0000","last_became_active":"2024-09-17T07:02:14.028392+0000","last_became_peered":"2024-09-17T07:02:14.028392+0000","last_unstale":"2024-09-17T07:08:12.989876+0000","last_undegraded":"2024-09-17T07:08:12.989876+0000","last_fullsized":"2024-09-17T07:08:12.989876+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'22","last_scrub_stamp":"2024-09-17T07:07:29.816241+0000","last_deep_scrub":"111'22","last_deep_scrub_stamp":"2024-09-17T07:04:57.500450+0000","last_clean_scrub_stamp":"2024-09-17T07:07:29.816241+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:53.480974+0000","scrub_duration":0.0034677570000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'2","reported_seq":142,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-17T07:08:17.308317+0000","last_change":"2024-09-17T07:06:39.828818+0000","last_active":"2024-09-17T07:08:17.308317+0000","last_peered":"2024-09-17T07:08:17.308317+0000","last_clean":"2024-09-17T07:08:17.308317+0000","last_became_active":"2024-09-17T07:04:15.110085+0000","last_became_peered":"2024-09-17T07:04:15.110085+0000","last_unstale":"2024-09-17T07:08:17.308317+0000","last_undegraded":"2024-09-17T07:08:17.308317+0000","last_fullsized":"2024-09-17T07:08:17.308317+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-09-17T07:06:39.828638+0000","last_deep_scrub":"89'2","last_deep_scrub_stamp":"2024-09-17T07:03:26.694366+0000","last_clean_scrub_stamp":"2024-09-17T07:06:39.828638+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0068983610000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":420,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"75'8","reported_seq":221,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T07:08:15.587026+0000","last_change":"2024-09-17T07:06:25.761229+0000","last_active":"2024-09-17T07:08:15.587026+0000","last_peered":"2024-09-17T07:08:15.587026+0000","last_clean":"2024-09-17T07:08:15.587026+0000","last_became_active":"2024-09-17T07:02:16.015418+0000","last_became_peered":"2024-09-17T07:02:16.015418+0000","last_unstale":"2024-09-17T07:08:15.587026+0000","last_undegraded":"2024-09-17T07:08:15.587026+0000","last_fullsized":"2024-09-17T07:08:15.587026+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:06:25.760922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:06:25.760922+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0073758620000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":145,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T07:08:12.990093+0000","last_change":"2024-09-17T07:07:19.786753+0000","last_active":"2024-09-17T07:08:12.990093+0000","last_peered":"2024-09-17T07:08:12.990093+0000","last_clean":"2024-09-17T07:08:12.990093+0000","last_became_active":"2024-09-17T07:02:18.363253+0000","last_became_peered":"2024-09-17T07:02:18.363253+0000","last_unstale":"2024-09-17T07:08:12.990093+0000","last_undegraded":"2024-09-17T07:08:12.990093+0000","last_fullsized":"2024-09-17T07:08:12.990093+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:19.786442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:07:19.786442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:41.487701+0000","scrub_duration":0.011747911999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.5","version":"175'452","reported_seq":468,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-17T07:08:18.110839+0000","last_change":"2024-09-17T07:08:00.598660+0000","last_active":"2024-09-17T07:08:18.110839+0000","last_peered":"2024-09-17T07:08:18.110839+0000","last_clean":"2024-09-17T07:08:18.110839+0000","last_became_active":"2024-09-17T07:08:00.598353+0000","last_became_peered":"2024-09-17T07:08:00.598353+0000","last_unstale":"2024-09-17T07:08:18.110839+0000","last_undegraded":"2024-09-17T07:08:18.110839+0000","last_fullsized":"2024-09-17T07:08:18.110839+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_clean_scrub_stamp":"2024-09-17T07:07:59.220681+0000","objects_scrubbed":0,"log_size":452,"log_dups_size":0,"ondisk_log_size":452,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:23.902877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29622272,"num_objects":452,"num_object_clones":0,"num_object_copies":1356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":452,"num_write_kb":28928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"150'17","reported_seq":167,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T07:08:12.279073+0000","last_change":"2024-09-17T07:06:38.775932+0000","last_active":"2024-09-17T07:08:12.279073+0000","last_peered":"2024-09-17T07:08:12.279073+0000","last_clean":"2024-09-17T07:08:12.279073+0000","last_became_active":"2024-09-17T07:02:14.010219+0000","last_became_peered":"2024-09-17T07:02:14.010219+0000","last_unstale":"2024-09-17T07:08:12.279073+0000","last_undegraded":"2024-09-17T07:08:12.279073+0000","last_fullsized":"2024-09-17T07:08:12.279073+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'17","last_scrub_stamp":"2024-09-17T07:06:38.775788+0000","last_deep_scrub":"150'17","last_deep_scrub_stamp":"2024-09-17T07:06:38.775788+0000","last_clean_scrub_stamp":"2024-09-17T07:06:38.775788+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011503368999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"68'2","reported_seq":149,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T07:08:12.279977+0000","last_change":"2024-09-17T07:07:20.764435+0000","last_active":"2024-09-17T07:08:12.279977+0000","last_peered":"2024-09-17T07:08:12.279977+0000","last_clean":"2024-09-17T07:08:12.279977+0000","last_became_active":"2024-09-17T07:02:12.006716+0000","last_became_peered":"2024-09-17T07:02:12.006716+0000","last_unstale":"2024-09-17T07:08:12.279977+0000","last_undegraded":"2024-09-17T07:08:12.279977+0000","last_fullsized":"2024-09-17T07:08:12.279977+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-09-17T07:07:20.764283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:07:20.764283+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:50.410328+0000","scrub_duration":0.0060160040000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":113,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-17T07:08:17.310097+0000","last_change":"2024-09-17T07:08:10.273572+0000","last_active":"2024-09-17T07:08:17.310097+0000","last_peered":"2024-09-17T07:08:17.310097+0000","last_clean":"2024-09-17T07:08:17.310097+0000","last_became_active":"2024-09-17T07:04:15.111069+0000","last_became_peered":"2024-09-17T07:04:15.111069+0000","last_unstale":"2024-09-17T07:08:17.310097+0000","last_undegraded":"2024-09-17T07:08:17.310097+0000","last_fullsized":"2024-09-17T07:08:17.310097+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:10.273485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:08:10.273485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:11.548372+0000","scrub_duration":0.0013521,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":135,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-17T07:08:17.307616+0000","last_change":"2024-09-17T07:08:07.878593+0000","last_active":"2024-09-17T07:08:17.307616+0000","last_peered":"2024-09-17T07:08:17.307616+0000","last_clean":"2024-09-17T07:08:17.307616+0000","last_became_active":"2024-09-17T07:02:18.024983+0000","last_became_peered":"2024-09-17T07:02:18.024983+0000","last_unstale":"2024-09-17T07:08:17.307616+0000","last_undegraded":"2024-09-17T07:08:17.307616+0000","last_fullsized":"2024-09-17T07:08:17.307616+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:07.878506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:08:07.878506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:23.059846+0000","scrub_duration":0.001815145,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"75'9","reported_seq":156,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-17T07:08:17.311136+0000","last_change":"2024-09-17T07:06:30.011545+0000","last_active":"2024-09-17T07:08:17.311136+0000","last_peered":"2024-09-17T07:08:17.311136+0000","last_clean":"2024-09-17T07:08:17.311136+0000","last_became_active":"2024-09-17T07:02:14.029388+0000","last_became_peered":"2024-09-17T07:02:14.029388+0000","last_unstale":"2024-09-17T07:08:17.311136+0000","last_undegraded":"2024-09-17T07:08:17.311136+0000","last_fullsized":"2024-09-17T07:08:17.311136+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:06:30.011337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:06:30.011337+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.013862154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":163,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-17T07:08:17.976299+0000","last_change":"2024-09-17T07:08:17.976403+0000","last_active":"2024-09-17T07:08:17.976299+0000","last_peered":"2024-09-17T07:08:17.976299+0000","last_clean":"2024-09-17T07:08:17.976299+0000","last_became_active":"2024-09-17T07:02:12.011327+0000","last_became_peered":"2024-09-17T07:02:12.011327+0000","last_unstale":"2024-09-17T07:08:17.976299+0000","last_undegraded":"2024-09-17T07:08:17.976299+0000","last_fullsized":"2024-09-17T07:08:17.976299+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:17.976267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:17.976267+0000","last_clean_scrub_stamp":"2024-09-17T07:08:17.976267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:43.173493+0000","scrub_duration":0.0035138019999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":137,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-17T07:08:17.309711+0000","last_change":"2024-09-17T07:04:36.652033+0000","last_active":"2024-09-17T07:08:17.309711+0000","last_peered":"2024-09-17T07:08:17.309711+0000","last_clean":"2024-09-17T07:08:17.309711+0000","last_became_active":"2024-09-17T07:02:16.022584+0000","last_became_peered":"2024-09-17T07:02:16.022584+0000","last_unstale":"2024-09-17T07:08:17.309711+0000","last_undegraded":"2024-09-17T07:08:17.309711+0000","last_fullsized":"2024-09-17T07:08:17.309711+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:04:36.651925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:04:36.651925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0061789210000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":146,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T07:08:12.279095+0000","last_change":"2024-09-17T07:06:36.398341+0000","last_active":"2024-09-17T07:08:12.279095+0000","last_peered":"2024-09-17T07:08:12.279095+0000","last_clean":"2024-09-17T07:08:12.279095+0000","last_became_active":"2024-09-17T07:02:18.017825+0000","last_became_peered":"2024-09-17T07:02:18.017825+0000","last_unstale":"2024-09-17T07:08:12.279095+0000","last_undegraded":"2024-09-17T07:08:12.279095+0000","last_fullsized":"2024-09-17T07:08:12.279095+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:36.398177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:06:36.398177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.099287720999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":151,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T07:08:12.462793+0000","last_change":"2024-09-17T07:08:04.480597+0000","last_active":"2024-09-17T07:08:12.462793+0000","last_peered":"2024-09-17T07:08:12.462793+0000","last_clean":"2024-09-17T07:08:12.462793+0000","last_became_active":"2024-09-17T07:02:12.008065+0000","last_became_peered":"2024-09-17T07:02:12.008065+0000","last_unstale":"2024-09-17T07:08:12.462793+0000","last_undegraded":"2024-09-17T07:08:12.462793+0000","last_fullsized":"2024-09-17T07:08:12.462793+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:04.480504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:08:04.480504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:23.678622+0000","scrub_duration":0.0013838889999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"75'6","reported_seq":148,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-17T07:08:17.309841+0000","last_change":"2024-09-17T07:06:43.839442+0000","last_active":"2024-09-17T07:08:17.309841+0000","last_peered":"2024-09-17T07:08:17.309841+0000","last_clean":"2024-09-17T07:08:17.309841+0000","last_became_active":"2024-09-17T07:02:14.028655+0000","last_became_peered":"2024-09-17T07:02:14.028655+0000","last_unstale":"2024-09-17T07:08:17.309841+0000","last_undegraded":"2024-09-17T07:08:17.309841+0000","last_fullsized":"2024-09-17T07:08:17.309841+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'6","last_scrub_stamp":"2024-09-17T07:06:43.836545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:06:43.836545+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0059941980000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":147,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T07:08:12.990767+0000","last_change":"2024-09-17T07:08:06.932434+0000","last_active":"2024-09-17T07:08:12.990767+0000","last_peered":"2024-09-17T07:08:12.990767+0000","last_clean":"2024-09-17T07:08:12.990767+0000","last_became_active":"2024-09-17T07:02:16.016634+0000","last_became_peered":"2024-09-17T07:02:16.016634+0000","last_unstale":"2024-09-17T07:08:12.990767+0000","last_undegraded":"2024-09-17T07:08:12.990767+0000","last_fullsized":"2024-09-17T07:08:12.990767+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:06.932318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:08:06.932318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:33.508360+0000","scrub_duration":0.0017321260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":135,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T07:08:13.399612+0000","last_change":"2024-09-17T07:07:18.042204+0000","last_active":"2024-09-17T07:08:13.399612+0000","last_peered":"2024-09-17T07:08:13.399612+0000","last_clean":"2024-09-17T07:08:13.399612+0000","last_became_active":"2024-09-17T07:02:18.365419+0000","last_became_peered":"2024-09-17T07:02:18.365419+0000","last_unstale":"2024-09-17T07:08:13.399612+0000","last_undegraded":"2024-09-17T07:08:13.399612+0000","last_fullsized":"2024-09-17T07:08:13.399612+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:18.042105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:07:18.042105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:18.135607+0000","scrub_duration":0.0020576729999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"75'10","reported_seq":154,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-17T07:08:17.308093+0000","last_change":"2024-09-17T07:04:40.721975+0000","last_active":"2024-09-17T07:08:17.308093+0000","last_peered":"2024-09-17T07:08:17.308093+0000","last_clean":"2024-09-17T07:08:17.308093+0000","last_became_active":"2024-09-17T07:02:14.027499+0000","last_became_peered":"2024-09-17T07:02:14.027499+0000","last_unstale":"2024-09-17T07:08:17.308093+0000","last_undegraded":"2024-09-17T07:08:17.308093+0000","last_fullsized":"2024-09-17T07:08:17.308093+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-09-17T07:04:40.721843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:04:40.721843+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0057465399999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":151,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T07:08:12.990361+0000","last_change":"2024-09-17T07:06:45.799922+0000","last_active":"2024-09-17T07:08:12.990361+0000","last_peered":"2024-09-17T07:08:12.990361+0000","last_clean":"2024-09-17T07:08:12.990361+0000","last_became_active":"2024-09-17T07:02:12.007280+0000","last_became_peered":"2024-09-17T07:02:12.007280+0000","last_unstale":"2024-09-17T07:08:12.990361+0000","last_undegraded":"2024-09-17T07:08:12.990361+0000","last_fullsized":"2024-09-17T07:08:12.990361+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:45.799808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:06:45.799808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021198189999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"163'9","reported_seq":229,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T07:08:15.586995+0000","last_change":"2024-09-17T07:07:34.933956+0000","last_active":"2024-09-17T07:08:15.586995+0000","last_peered":"2024-09-17T07:08:15.586995+0000","last_clean":"2024-09-17T07:08:15.586995+0000","last_became_active":"2024-09-17T07:02:16.022583+0000","last_became_peered":"2024-09-17T07:02:16.022583+0000","last_unstale":"2024-09-17T07:08:15.586995+0000","last_undegraded":"2024-09-17T07:08:15.586995+0000","last_fullsized":"2024-09-17T07:08:15.586995+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'9","last_scrub_stamp":"2024-09-17T07:07:34.933808+0000","last_deep_scrub":"163'9","last_deep_scrub_stamp":"2024-09-17T07:07:34.933808+0000","last_clean_scrub_stamp":"2024-09-17T07:07:34.933808+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:57.452885+0000","scrub_duration":0.019504579000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":136,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-17T07:08:17.310769+0000","last_change":"2024-09-17T07:07:26.014516+0000","last_active":"2024-09-17T07:08:17.310769+0000","last_peered":"2024-09-17T07:08:17.310769+0000","last_clean":"2024-09-17T07:08:17.310769+0000","last_became_active":"2024-09-17T07:02:18.028796+0000","last_became_peered":"2024-09-17T07:02:18.028796+0000","last_unstale":"2024-09-17T07:08:17.310769+0000","last_undegraded":"2024-09-17T07:08:17.310769+0000","last_fullsized":"2024-09-17T07:08:17.310769+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:26.014398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:07:26.014398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:36.974117+0000","scrub_duration":0.0066951459999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"75'9","reported_seq":165,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T07:08:12.990787+0000","last_change":"2024-09-17T07:06:35.727847+0000","last_active":"2024-09-17T07:08:12.990787+0000","last_peered":"2024-09-17T07:08:12.990787+0000","last_clean":"2024-09-17T07:08:12.990787+0000","last_became_active":"2024-09-17T07:02:14.028485+0000","last_became_peered":"2024-09-17T07:02:14.028485+0000","last_unstale":"2024-09-17T07:08:12.990787+0000","last_undegraded":"2024-09-17T07:08:12.990787+0000","last_fullsized":"2024-09-17T07:08:12.990787+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:06:35.727700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:06:35.727700+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045062289999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":153,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-17T07:08:17.311144+0000","last_change":"2024-09-17T07:08:15.356266+0000","last_active":"2024-09-17T07:08:17.311144+0000","last_peered":"2024-09-17T07:08:17.311144+0000","last_clean":"2024-09-17T07:08:17.311144+0000","last_became_active":"2024-09-17T07:02:12.009935+0000","last_became_peered":"2024-09-17T07:02:12.009935+0000","last_unstale":"2024-09-17T07:08:17.311144+0000","last_undegraded":"2024-09-17T07:08:17.311144+0000","last_fullsized":"2024-09-17T07:08:17.311144+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:15.356187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:08:15.356187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:27.551829+0000","scrub_duration":0.0015925290000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"75'8","reported_seq":218,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-17T07:08:17.307887+0000","last_change":"2024-09-17T07:07:32.670103+0000","last_active":"2024-09-17T07:08:17.307887+0000","last_peered":"2024-09-17T07:08:17.307887+0000","last_clean":"2024-09-17T07:08:17.307887+0000","last_became_active":"2024-09-17T07:02:16.016069+0000","last_became_peered":"2024-09-17T07:02:16.016069+0000","last_unstale":"2024-09-17T07:08:17.307887+0000","last_undegraded":"2024-09-17T07:08:17.307887+0000","last_fullsized":"2024-09-17T07:08:17.307887+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:07:32.669921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:07:32.669921+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:44.233420+0000","scrub_duration":0.0099051529999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":146,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T07:08:16.703606+0000","last_change":"2024-09-17T07:08:16.703606+0000","last_active":"2024-09-17T07:08:16.703606+0000","last_peered":"2024-09-17T07:08:16.703606+0000","last_clean":"2024-09-17T07:08:16.703606+0000","last_became_active":"2024-09-17T07:02:18.018716+0000","last_became_peered":"2024-09-17T07:02:18.018716+0000","last_unstale":"2024-09-17T07:08:16.703606+0000","last_undegraded":"2024-09-17T07:08:16.703606+0000","last_fullsized":"2024-09-17T07:08:16.703606+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:16.703532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:08:16.703532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:45.505475+0000","scrub_duration":0.0019164340000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"75'9","reported_seq":165,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T07:08:16.756725+0000","last_change":"2024-09-17T07:08:16.756725+0000","last_active":"2024-09-17T07:08:16.756725+0000","last_peered":"2024-09-17T07:08:16.756725+0000","last_clean":"2024-09-17T07:08:16.756725+0000","last_became_active":"2024-09-17T07:02:14.010289+0000","last_became_peered":"2024-09-17T07:02:14.010289+0000","last_unstale":"2024-09-17T07:08:16.756725+0000","last_undegraded":"2024-09-17T07:08:16.756725+0000","last_fullsized":"2024-09-17T07:08:16.756725+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:08:16.756185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:08:16.756185+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:22.278273+0000","scrub_duration":0.004786584,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":141,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T07:08:12.462410+0000","last_change":"2024-09-17T07:07:20.458487+0000","last_active":"2024-09-17T07:08:12.462410+0000","last_peered":"2024-09-17T07:08:12.462410+0000","last_clean":"2024-09-17T07:08:12.462410+0000","last_became_active":"2024-09-17T07:02:12.007433+0000","last_became_peered":"2024-09-17T07:02:12.007433+0000","last_unstale":"2024-09-17T07:08:12.462410+0000","last_undegraded":"2024-09-17T07:08:12.462410+0000","last_fullsized":"2024-09-17T07:08:12.462410+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:20.458316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:07:20.458316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:42.134938+0000","scrub_duration":0.0043263889999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":137,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-17T07:08:17.307691+0000","last_change":"2024-09-17T07:07:17.713153+0000","last_active":"2024-09-17T07:08:17.307691+0000","last_peered":"2024-09-17T07:08:17.307691+0000","last_clean":"2024-09-17T07:08:17.307691+0000","last_became_active":"2024-09-17T07:02:16.016216+0000","last_became_peered":"2024-09-17T07:02:16.016216+0000","last_unstale":"2024-09-17T07:08:17.307691+0000","last_undegraded":"2024-09-17T07:08:17.307691+0000","last_fullsized":"2024-09-17T07:08:17.307691+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:17.713027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:07:17.713027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:19.920380+0000","scrub_duration":0.017242598000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":147,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-17T07:08:18.338887+0000","last_change":"2024-09-17T07:08:18.338994+0000","last_active":"2024-09-17T07:08:18.338887+0000","last_peered":"2024-09-17T07:08:18.338887+0000","last_clean":"2024-09-17T07:08:18.338887+0000","last_became_active":"2024-09-17T07:02:18.027622+0000","last_became_peered":"2024-09-17T07:02:18.027622+0000","last_unstale":"2024-09-17T07:08:18.338887+0000","last_undegraded":"2024-09-17T07:08:18.338887+0000","last_fullsized":"2024-09-17T07:08:18.338887+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:18.338863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:08:18.338863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:36.839965+0000","scrub_duration":0.0025905709999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"75'11","reported_seq":158,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T07:08:12.279371+0000","last_change":"2024-09-17T07:06:11.787256+0000","last_active":"2024-09-17T07:08:12.279371+0000","last_peered":"2024-09-17T07:08:12.279371+0000","last_clean":"2024-09-17T07:08:12.279371+0000","last_became_active":"2024-09-17T07:02:14.016255+0000","last_became_peered":"2024-09-17T07:02:14.016255+0000","last_unstale":"2024-09-17T07:08:12.279371+0000","last_undegraded":"2024-09-17T07:08:12.279371+0000","last_fullsized":"2024-09-17T07:08:12.279371+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-09-17T07:06:11.787156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:06:11.787156+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035465940000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"94'2","reported_seq":146,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T07:08:12.991350+0000","last_change":"2024-09-17T07:07:31.718074+0000","last_active":"2024-09-17T07:08:12.991350+0000","last_peered":"2024-09-17T07:08:12.991350+0000","last_clean":"2024-09-17T07:08:12.991350+0000","last_became_active":"2024-09-17T07:02:12.011461+0000","last_became_peered":"2024-09-17T07:02:12.011461+0000","last_unstale":"2024-09-17T07:08:12.991350+0000","last_undegraded":"2024-09-17T07:08:12.991350+0000","last_fullsized":"2024-09-17T07:08:12.991350+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-09-17T07:07:31.717888+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-09-17T07:07:31.717888+0000","last_clean_scrub_stamp":"2024-09-17T07:07:31.717888+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:39.938682+0000","scrub_duration":0.0094559669999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":136,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T07:08:12.990839+0000","last_change":"2024-09-17T07:06:40.147092+0000","last_active":"2024-09-17T07:08:12.990839+0000","last_peered":"2024-09-17T07:08:12.990839+0000","last_clean":"2024-09-17T07:08:12.990839+0000","last_became_active":"2024-09-17T07:02:16.018754+0000","last_became_peered":"2024-09-17T07:02:16.018754+0000","last_unstale":"2024-09-17T07:08:12.990839+0000","last_undegraded":"2024-09-17T07:08:12.990839+0000","last_fullsized":"2024-09-17T07:08:12.990839+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:40.146846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:06:40.146846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00753165,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":135,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-17T07:08:17.306621+0000","last_change":"2024-09-17T07:08:10.805778+0000","last_active":"2024-09-17T07:08:17.306621+0000","last_peered":"2024-09-17T07:08:17.306621+0000","last_clean":"2024-09-17T07:08:17.306621+0000","last_became_active":"2024-09-17T07:02:18.027427+0000","last_became_peered":"2024-09-17T07:02:18.027427+0000","last_unstale":"2024-09-17T07:08:17.306621+0000","last_undegraded":"2024-09-17T07:08:17.306621+0000","last_fullsized":"2024-09-17T07:08:17.306621+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:10.805704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:08:10.805704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:17.680378+0000","scrub_duration":0.00179312,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"75'4","reported_seq":160,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-17T07:08:17.306787+0000","last_change":"2024-09-17T07:06:36.857728+0000","last_active":"2024-09-17T07:08:17.306787+0000","last_peered":"2024-09-17T07:08:17.306787+0000","last_clean":"2024-09-17T07:08:17.306787+0000","last_became_active":"2024-09-17T07:03:48.758220+0000","last_became_peered":"2024-09-17T07:03:48.758220+0000","last_unstale":"2024-09-17T07:08:17.306787+0000","last_undegraded":"2024-09-17T07:08:17.306787+0000","last_fullsized":"2024-09-17T07:08:17.306787+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'4","last_scrub_stamp":"2024-09-17T07:06:36.857471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:06:36.857471+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0095623189999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":151,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T07:08:12.990851+0000","last_change":"2024-09-17T07:08:04.972061+0000","last_active":"2024-09-17T07:08:12.990851+0000","last_peered":"2024-09-17T07:08:12.990851+0000","last_clean":"2024-09-17T07:08:12.990851+0000","last_became_active":"2024-09-17T07:02:12.008898+0000","last_became_peered":"2024-09-17T07:02:12.008898+0000","last_unstale":"2024-09-17T07:08:12.990851+0000","last_undegraded":"2024-09-17T07:08:12.990851+0000","last_fullsized":"2024-09-17T07:08:12.990851+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:04.971954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:08:04.971954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:23.242122+0000","scrub_duration":0.0014792150000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":147,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T07:08:12.462560+0000","last_change":"2024-09-17T07:08:03.496692+0000","last_active":"2024-09-17T07:08:12.462560+0000","last_peered":"2024-09-17T07:08:12.462560+0000","last_clean":"2024-09-17T07:08:12.462560+0000","last_became_active":"2024-09-17T07:02:16.014982+0000","last_became_peered":"2024-09-17T07:02:16.014982+0000","last_unstale":"2024-09-17T07:08:12.462560+0000","last_undegraded":"2024-09-17T07:08:12.462560+0000","last_fullsized":"2024-09-17T07:08:12.462560+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:03.496602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:47.444972+0000","last_clean_scrub_stamp":"2024-09-17T07:08:03.496602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:21.598625+0000","scrub_duration":0.001400943,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":135,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T07:08:13.399341+0000","last_change":"2024-09-17T07:06:40.865239+0000","last_active":"2024-09-17T07:08:13.399341+0000","last_peered":"2024-09-17T07:08:13.399341+0000","last_clean":"2024-09-17T07:08:13.399341+0000","last_became_active":"2024-09-17T07:02:18.021014+0000","last_became_peered":"2024-09-17T07:02:18.021014+0000","last_unstale":"2024-09-17T07:08:13.399341+0000","last_undegraded":"2024-09-17T07:08:13.399341+0000","last_fullsized":"2024-09-17T07:08:13.399341+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:40.865049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:06:40.865049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0029834100000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"75'11","reported_seq":158,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-17T07:08:17.308573+0000","last_change":"2024-09-17T07:04:52.733713+0000","last_active":"2024-09-17T07:08:17.308573+0000","last_peered":"2024-09-17T07:08:17.308573+0000","last_clean":"2024-09-17T07:08:17.308573+0000","last_became_active":"2024-09-17T07:02:14.027316+0000","last_became_peered":"2024-09-17T07:02:14.027316+0000","last_unstale":"2024-09-17T07:08:17.308573+0000","last_undegraded":"2024-09-17T07:08:17.308573+0000","last_fullsized":"2024-09-17T07:08:17.308573+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-09-17T07:04:52.733516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:04:52.733516+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.013209306000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":162,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T07:08:14.519328+0000","last_change":"2024-09-17T07:08:14.519328+0000","last_active":"2024-09-17T07:08:14.519328+0000","last_peered":"2024-09-17T07:08:14.519328+0000","last_clean":"2024-09-17T07:08:14.519328+0000","last_became_active":"2024-09-17T07:02:12.010195+0000","last_became_peered":"2024-09-17T07:02:12.010195+0000","last_unstale":"2024-09-17T07:08:14.519328+0000","last_undegraded":"2024-09-17T07:08:14.519328+0000","last_fullsized":"2024-09-17T07:08:14.519328+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:14.519233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:08:14.519233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:35.668223+0000","scrub_duration":0.001420021,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":137,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T07:08:13.399002+0000","last_change":"2024-09-17T07:06:33.856836+0000","last_active":"2024-09-17T07:08:13.399002+0000","last_peered":"2024-09-17T07:08:13.399002+0000","last_clean":"2024-09-17T07:08:13.399002+0000","last_became_active":"2024-09-17T07:02:16.016183+0000","last_became_peered":"2024-09-17T07:02:16.016183+0000","last_unstale":"2024-09-17T07:08:13.399002+0000","last_undegraded":"2024-09-17T07:08:13.399002+0000","last_fullsized":"2024-09-17T07:08:13.399002+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:33.856556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:06:33.856556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017432000000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":135,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-17T07:08:17.307544+0000","last_change":"2024-09-17T07:07:26.832785+0000","last_active":"2024-09-17T07:08:17.307544+0000","last_peered":"2024-09-17T07:08:17.307544+0000","last_clean":"2024-09-17T07:08:17.307544+0000","last_became_active":"2024-09-17T07:02:18.365656+0000","last_became_peered":"2024-09-17T07:02:18.365656+0000","last_unstale":"2024-09-17T07:08:17.307544+0000","last_undegraded":"2024-09-17T07:08:17.307544+0000","last_fullsized":"2024-09-17T07:08:17.307544+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:26.832514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:07:26.832514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:29.955192+0000","scrub_duration":0.01115028,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"75'1","reported_seq":136,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T07:08:13.399505+0000","last_change":"2024-09-17T07:06:32.836489+0000","last_active":"2024-09-17T07:08:13.399505+0000","last_peered":"2024-09-17T07:08:13.399505+0000","last_clean":"2024-09-17T07:08:13.399505+0000","last_became_active":"2024-09-17T07:02:18.021855+0000","last_became_peered":"2024-09-17T07:02:18.021855+0000","last_unstale":"2024-09-17T07:08:13.399505+0000","last_undegraded":"2024-09-17T07:08:13.399505+0000","last_fullsized":"2024-09-17T07:08:13.399505+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-17T07:06:32.836363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:06:32.836363+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0056683599999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":151,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T07:08:12.989970+0000","last_change":"2024-09-17T07:07:22.782091+0000","last_active":"2024-09-17T07:08:12.989970+0000","last_peered":"2024-09-17T07:08:12.989970+0000","last_clean":"2024-09-17T07:08:12.989970+0000","last_became_active":"2024-09-17T07:02:12.005329+0000","last_became_peered":"2024-09-17T07:02:12.005329+0000","last_unstale":"2024-09-17T07:08:12.989970+0000","last_undegraded":"2024-09-17T07:08:12.989970+0000","last_fullsized":"2024-09-17T07:08:12.989970+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:22.781739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:07:22.781739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:44.511386+0000","scrub_duration":0.0037808680000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"75'10","reported_seq":156,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-17T07:08:18.317094+0000","last_change":"2024-09-17T07:07:22.081513+0000","last_active":"2024-09-17T07:08:18.317094+0000","last_peered":"2024-09-17T07:08:18.317094+0000","last_clean":"2024-09-17T07:08:18.317094+0000","last_became_active":"2024-09-17T07:02:14.028731+0000","last_became_peered":"2024-09-17T07:02:14.028731+0000","last_unstale":"2024-09-17T07:08:18.317094+0000","last_undegraded":"2024-09-17T07:08:18.317094+0000","last_fullsized":"2024-09-17T07:08:18.317094+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-09-17T07:07:22.081382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:07:22.081382+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:37.716407+0000","scrub_duration":0.056222292,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"75'8","reported_seq":221,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T07:08:15.586999+0000","last_change":"2024-09-17T07:07:19.425224+0000","last_active":"2024-09-17T07:08:15.586999+0000","last_peered":"2024-09-17T07:08:15.586999+0000","last_clean":"2024-09-17T07:08:15.586999+0000","last_became_active":"2024-09-17T07:02:16.020560+0000","last_became_peered":"2024-09-17T07:02:16.020560+0000","last_unstale":"2024-09-17T07:08:15.586999+0000","last_undegraded":"2024-09-17T07:08:15.586999+0000","last_fullsized":"2024-09-17T07:08:15.586999+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:07:19.423853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:07:19.423853+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:23.974900+0000","scrub_duration":0.0037511480000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":134,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T07:08:12.991008+0000","last_change":"2024-09-17T07:07:30.668537+0000","last_active":"2024-09-17T07:08:12.991008+0000","last_peered":"2024-09-17T07:08:12.991008+0000","last_clean":"2024-09-17T07:08:12.991008+0000","last_became_active":"2024-09-17T07:02:18.021603+0000","last_became_peered":"2024-09-17T07:02:18.021603+0000","last_unstale":"2024-09-17T07:08:12.991008+0000","last_undegraded":"2024-09-17T07:08:12.991008+0000","last_fullsized":"2024-09-17T07:08:12.991008+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:30.668428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:07:30.668428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:31.854801+0000","scrub_duration":0.0047713119999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":152,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-17T07:08:17.704801+0000","last_change":"2024-09-17T07:08:17.704801+0000","last_active":"2024-09-17T07:08:17.704801+0000","last_peered":"2024-09-17T07:08:17.704801+0000","last_clean":"2024-09-17T07:08:17.704801+0000","last_became_active":"2024-09-17T07:02:12.009837+0000","last_became_peered":"2024-09-17T07:02:12.009837+0000","last_unstale":"2024-09-17T07:08:17.704801+0000","last_undegraded":"2024-09-17T07:08:17.704801+0000","last_fullsized":"2024-09-17T07:08:17.704801+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:17.704687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:08:17.704687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:31.103100+0000","scrub_duration":0.0038132069999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"75'11","reported_seq":200,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T07:08:12.462898+0000","last_change":"2024-09-17T07:08:07.442547+0000","last_active":"2024-09-17T07:08:12.462898+0000","last_peered":"2024-09-17T07:08:12.462898+0000","last_clean":"2024-09-17T07:08:12.462898+0000","last_became_active":"2024-09-17T07:03:48.760390+0000","last_became_peered":"2024-09-17T07:03:48.760390+0000","last_unstale":"2024-09-17T07:08:12.462898+0000","last_undegraded":"2024-09-17T07:08:12.462898+0000","last_fullsized":"2024-09-17T07:08:12.462898+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-09-17T07:08:07.442416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:08:07.442416+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:07.965821+0000","scrub_duration":0.003039281,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":138,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-17T07:08:17.310969+0000","last_change":"2024-09-17T07:07:31.068035+0000","last_active":"2024-09-17T07:08:17.310969+0000","last_peered":"2024-09-17T07:08:17.310969+0000","last_clean":"2024-09-17T07:08:17.310969+0000","last_became_active":"2024-09-17T07:02:16.019701+0000","last_became_peered":"2024-09-17T07:02:16.019701+0000","last_unstale":"2024-09-17T07:08:17.310969+0000","last_undegraded":"2024-09-17T07:08:17.310969+0000","last_fullsized":"2024-09-17T07:08:17.310969+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:31.067529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:07:31.067529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:32.075799+0000","scrub_duration":0.013304165,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":135,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T07:08:12.279725+0000","last_change":"2024-09-17T07:07:30.779462+0000","last_active":"2024-09-17T07:08:12.279725+0000","last_peered":"2024-09-17T07:08:12.279725+0000","last_clean":"2024-09-17T07:08:12.279725+0000","last_became_active":"2024-09-17T07:02:18.363752+0000","last_became_peered":"2024-09-17T07:02:18.363752+0000","last_unstale":"2024-09-17T07:08:12.279725+0000","last_undegraded":"2024-09-17T07:08:12.279725+0000","last_fullsized":"2024-09-17T07:08:12.279725+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:30.779382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:07:30.779382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:38.926296+0000","scrub_duration":0.0011291059999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":171,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T07:08:12.462985+0000","last_change":"2024-09-17T07:08:12.462985+0000","last_active":"2024-09-17T07:08:12.462985+0000","last_peered":"2024-09-17T07:08:12.462985+0000","last_clean":"2024-09-17T07:08:12.462985+0000","last_became_active":"2024-09-17T07:03:48.759384+0000","last_became_peered":"2024-09-17T07:03:48.759384+0000","last_unstale":"2024-09-17T07:08:12.462985+0000","last_undegraded":"2024-09-17T07:08:12.462985+0000","last_fullsized":"2024-09-17T07:08:12.462985+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:12.462862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:08:12.462862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:20.330872+0000","scrub_duration":0.0022151050000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"75'15","reported_seq":150,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T07:08:12.991076+0000","last_change":"2024-09-17T07:07:33.775051+0000","last_active":"2024-09-17T07:08:12.991076+0000","last_peered":"2024-09-17T07:08:12.991076+0000","last_clean":"2024-09-17T07:08:12.991076+0000","last_became_active":"2024-09-17T07:04:15.114286+0000","last_became_peered":"2024-09-17T07:04:15.114286+0000","last_unstale":"2024-09-17T07:08:12.991076+0000","last_undegraded":"2024-09-17T07:08:12.991076+0000","last_fullsized":"2024-09-17T07:08:12.991076+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-09-17T07:07:33.774757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:07:33.774757+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:34.474071+0000","scrub_duration":0.0071874820000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":136,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T07:08:12.991192+0000","last_change":"2024-09-17T07:06:42.201934+0000","last_active":"2024-09-17T07:08:12.991192+0000","last_peered":"2024-09-17T07:08:12.991192+0000","last_clean":"2024-09-17T07:08:12.991192+0000","last_became_active":"2024-09-17T07:02:16.015903+0000","last_became_peered":"2024-09-17T07:02:16.015903+0000","last_unstale":"2024-09-17T07:08:12.991192+0000","last_undegraded":"2024-09-17T07:08:12.991192+0000","last_fullsized":"2024-09-17T07:08:12.991192+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:42.200604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:06:42.200604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.051542886000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":135,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-17T07:08:17.309644+0000","last_change":"2024-09-17T07:08:14.764859+0000","last_active":"2024-09-17T07:08:17.309644+0000","last_peered":"2024-09-17T07:08:17.309644+0000","last_clean":"2024-09-17T07:08:17.309644+0000","last_became_active":"2024-09-17T07:02:18.366466+0000","last_became_peered":"2024-09-17T07:02:18.366466+0000","last_unstale":"2024-09-17T07:08:17.309644+0000","last_undegraded":"2024-09-17T07:08:17.309644+0000","last_fullsized":"2024-09-17T07:08:17.309644+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:14.764781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:08:14.764781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:40.723176+0000","scrub_duration":0.0018717720000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":141,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-17T07:08:17.308142+0000","last_change":"2024-09-17T07:07:29.761118+0000","last_active":"2024-09-17T07:08:17.308142+0000","last_peered":"2024-09-17T07:08:17.308142+0000","last_clean":"2024-09-17T07:08:17.308142+0000","last_became_active":"2024-09-17T07:02:12.005493+0000","last_became_peered":"2024-09-17T07:02:12.005493+0000","last_unstale":"2024-09-17T07:08:17.308142+0000","last_undegraded":"2024-09-17T07:08:17.308142+0000","last_fullsized":"2024-09-17T07:08:17.308142+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:29.761001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:07:29.761001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:58.064357+0000","scrub_duration":0.0083413440000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"86'17","reported_seq":167,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-17T07:08:17.307819+0000","last_change":"2024-09-17T07:06:27.962393+0000","last_active":"2024-09-17T07:08:17.307819+0000","last_peered":"2024-09-17T07:08:17.307819+0000","last_clean":"2024-09-17T07:08:17.307819+0000","last_became_active":"2024-09-17T07:02:14.017737+0000","last_became_peered":"2024-09-17T07:02:14.017737+0000","last_unstale":"2024-09-17T07:08:17.307819+0000","last_undegraded":"2024-09-17T07:08:17.307819+0000","last_fullsized":"2024-09-17T07:08:17.307819+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'17","last_scrub_stamp":"2024-09-17T07:06:27.962276+0000","last_deep_scrub":"86'17","last_deep_scrub_stamp":"2024-09-17T07:03:18.241067+0000","last_clean_scrub_stamp":"2024-09-17T07:06:27.962276+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0050727970000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":148,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T07:08:14.706829+0000","last_change":"2024-09-17T07:08:14.706829+0000","last_active":"2024-09-17T07:08:14.706829+0000","last_peered":"2024-09-17T07:08:14.706829+0000","last_clean":"2024-09-17T07:08:14.706829+0000","last_became_active":"2024-09-17T07:02:16.017362+0000","last_became_peered":"2024-09-17T07:02:16.017362+0000","last_unstale":"2024-09-17T07:08:14.706829+0000","last_undegraded":"2024-09-17T07:08:14.706829+0000","last_fullsized":"2024-09-17T07:08:14.706829+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:14.706740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:08:14.706740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:19.562882+0000","scrub_duration":0.00164321,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":515833856,"num_objects":7871,"num_object_clones":0,"num_object_copies":23613,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7871,"num_write_kb":503744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1505427456,"data_stored":1505427456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7871,"ondisk_log_size":7871,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67,"ondisk_log_size":67,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":697,"num_read_kb":454,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":421,"ondisk_log_size":421,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":359,"num_read_kb":354,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1386,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":12,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":4158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037839,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":339312,"kb_used_data":141668,"kb_used_omap":1,"kb_used_meta":197630,"kb_avail":93405840,"statfs":{"total":95995035648,"available":95647580160,"internally_reserved":0,"allocated":145068032,"data_stored":144645948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":202373579},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:08:13 2024","interfaces":[{"interface":"back","average":{"1min":3.355,"5min":2.4620000000000002,"15min":1.1739999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":48.493000000000002,"5min":52.478999999999999,"15min":52.478999999999999},"last":0.67400000000000004},{"interface":"front","average":{"1min":3.2959999999999998,"5min":2.4540000000000002,"15min":1.159},"min":{"1min":0.312,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":48.738,"5min":52.871000000000002,"15min":52.871000000000002},"last":0.44}]},{"osd":1,"last update":"Tue Sep 17 07:08:13 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.73599999999999999,"15min":0.58399999999999996},"min":{"1min":0.255,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":1.8480000000000001,"5min":4.5140000000000002,"15min":4.5140000000000002},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.77300000000000002,"15min":0.60699999999999998},"min":{"1min":0.308,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.9650000000000001,"5min":4.5039999999999996,"15min":4.5039999999999996},"last":0.49199999999999999}]},{"osd":2,"last update":"Tue Sep 17 07:08:13 2024","interfaces":[{"interface":"back","average":{"1min":2.0150000000000001,"5min":1.5269999999999999,"15min":0.82799999999999996},"min":{"1min":0.245,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":22.376000000000001,"5min":52.055999999999997,"15min":52.055999999999997},"last":0.46400000000000002},{"interface":"front","average":{"1min":2.2109999999999999,"5min":1.5900000000000001,"15min":0.89100000000000001},"min":{"1min":0.35499999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":22.298999999999999,"5min":52.064999999999998,"15min":52.064999999999998},"last":0.56299999999999994}]},{"osd":3,"last update":"Tue Sep 17 07:08:13 2024","interfaces":[{"interface":"back","average":{"1min":2.0390000000000001,"5min":1.875,"15min":0.97499999999999998},"min":{"1min":0.34599999999999997,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":22.196000000000002,"5min":51.689,"15min":51.689},"last":0.54200000000000004},{"interface":"front","average":{"1min":2.9100000000000001,"5min":2.306,"15min":1.135},"min":{"1min":0.33800000000000002,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":22.234000000000002,"5min":52.145000000000003,"15min":52.145000000000003},"last":0.61399999999999999}]},{"osd":4,"last update":"Tue Sep 17 07:08:13 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.46200000000000002,"15min":0.45400000000000001},"min":{"1min":0.28799999999999998,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":1.3400000000000001,"5min":1.5980000000000001,"15min":1.5980000000000001},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.51500000000000001,"15min":0.45700000000000002},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.734,"5min":3.4089999999999998,"15min":3.4089999999999998},"last":0.50600000000000001}]},{"osd":5,"last update":"Tue Sep 17 07:08:13 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.51800000000000002,"15min":0.46100000000000002},"min":{"1min":0.28599999999999998,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.8169999999999999,"5min":3.3620000000000001,"15min":3.3620000000000001},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.502,"15min":0.47399999999999998},"min":{"1min":0.36099999999999999,"5min":0.28999999999999998,"15min":0.28000000000000003},"max":{"1min":1.383,"5min":1.6459999999999999,"15min":1.6459999999999999},"last":0.65000000000000002}]},{"osd":6,"last update":"Tue Sep 17 07:08:13 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.55400000000000005,"15min":0.51500000000000001},"min":{"1min":0.32100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":1.4339999999999999,"5min":1.675,"15min":1.675},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.628,"5min":0.59499999999999997,"15min":0.52300000000000002},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":1.792,"5min":3.2669999999999999,"15min":3.2669999999999999},"last":0.58299999999999996}]}]},{"osd":6,"up_from":50,"seq":214748364882,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":256096,"kb_used_data":90452,"kb_used_omap":1,"kb_used_meta":165630,"kb_avail":93489056,"statfs":{"total":95995035648,"available":95732793344,"internally_reserved":0,"allocated":92622848,"data_stored":92216455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":169605575},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:07:50 2024","interfaces":[{"interface":"back","average":{"1min":2.323,"5min":2.9609999999999999,"15min":1.3200000000000001},"min":{"1min":0.20899999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":16.696999999999999,"5min":75.323999999999998,"15min":75.323999999999998},"last":0.51800000000000002},{"interface":"front","average":{"1min":1.8899999999999999,"5min":2.8279999999999998,"15min":1.27},"min":{"1min":0.312,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":16.483000000000001,"5min":75.168999999999997,"15min":75.168999999999997},"last":0.71299999999999997}]},{"osd":1,"last update":"Tue Sep 17 07:07:50 2024","interfaces":[{"interface":"back","average":{"1min":0.90500000000000003,"5min":0.67300000000000004,"15min":0.55000000000000004},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":3.1850000000000001,"5min":4.4790000000000001,"15min":4.4790000000000001},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.74199999999999999,"15min":0.60299999999999998},"min":{"1min":0.29799999999999999,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":2.7789999999999999,"5min":5.9530000000000003,"15min":5.9530000000000003},"last":1.367}]},{"osd":2,"last update":"Tue Sep 17 07:07:50 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":1.0169999999999999,"15min":0.68700000000000006},"min":{"1min":0.30199999999999999,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":3.5800000000000001,"5min":33.079000000000001,"15min":33.079000000000001},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.79900000000000004,"5min":1.0329999999999999,"15min":0.69099999999999995},"min":{"1min":0.25700000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":2.8839999999999999,"5min":33.100000000000001,"15min":33.100000000000001},"last":0.79600000000000004}]},{"osd":3,"last update":"Tue Sep 17 07:07:50 2024","interfaces":[{"interface":"back","average":{"1min":1.7250000000000001,"5min":1.891,"15min":0.94999999999999996},"min":{"1min":0.27300000000000002,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":16.984000000000002,"5min":52.944000000000003,"15min":52.944000000000003},"last":0.47199999999999998},{"interface":"front","average":{"1min":1.907,"5min":1.843,"15min":0.97399999999999998},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":16.745999999999999,"5min":52.747,"15min":52.747},"last":0.53500000000000003}]},{"osd":4,"last update":"Tue Sep 17 07:07:50 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.42299999999999999,"15min":0.42099999999999999},"min":{"1min":0.26300000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.054,"5min":1.054,"15min":1.054},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.51400000000000001,"15min":0.442},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":2.8900000000000001,"5min":2.9119999999999999,"15min":2.9119999999999999},"last":0.255}]},{"osd":5,"last update":"Tue Sep 17 07:07:50 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.47699999999999998,"15min":0.44},"min":{"1min":0.28499999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":1.2150000000000001,"5min":1.6419999999999999,"15min":1.6419999999999999},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.53300000000000003,"15min":0.46200000000000002},"min":{"1min":0.28299999999999997,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":3.0379999999999998,"5min":3.0379999999999998,"15min":3.0379999999999998},"last":0.312}]},{"osd":7,"last update":"Tue Sep 17 07:08:13 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56799999999999995,"15min":0.51900000000000002},"min":{"1min":0.33200000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.01,"5min":2.859,"15min":2.859},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.58199999999999996,"15min":0.54400000000000004},"min":{"1min":0.33400000000000002,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.1140000000000001,"5min":1.9339999999999999,"15min":1.9339999999999999},"last":0.32000000000000001}]}]},{"osd":5,"up_from":42,"seq":180388626519,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":335580,"kb_used_data":210132,"kb_used_omap":1,"kb_used_meta":125438,"kb_avail":93409572,"statfs":{"total":95995035648,"available":95651401728,"internally_reserved":0,"allocated":215175168,"data_stored":214769041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":128448966},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:07:31 2024","interfaces":[{"interface":"back","average":{"1min":2.9020000000000001,"5min":2.6400000000000001,"15min":1.226},"min":{"1min":0.23499999999999999,"5min":0.221,"15min":0.221},"max":{"1min":28.727,"5min":49.265000000000001,"15min":49.265000000000001},"last":0.53700000000000003},{"interface":"front","average":{"1min":2.7639999999999998,"5min":2.7450000000000001,"15min":1.2549999999999999},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":27.824999999999999,"5min":47.350000000000001,"15min":47.350000000000001},"last":0.45800000000000002}]},{"osd":1,"last update":"Tue Sep 17 07:07:31 2024","interfaces":[{"interface":"back","average":{"1min":1.766,"5min":0.79000000000000004,"15min":0.628},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":20.329000000000001,"5min":20.329000000000001,"15min":20.329000000000001},"last":0.439},{"interface":"front","average":{"1min":1.9450000000000001,"5min":0.97099999999999997,"15min":1.121},"min":{"1min":0.27300000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":20.408999999999999,"5min":20.408999999999999,"15min":20.408999999999999},"last":0.42899999999999999}]},{"osd":2,"last update":"Tue Sep 17 07:07:31 2024","interfaces":[{"interface":"back","average":{"1min":2.0390000000000001,"5min":1.1100000000000001,"15min":0.748},"min":{"1min":0.34200000000000003,"5min":0.252,"15min":0.252},"max":{"1min":20.486000000000001,"5min":20.486000000000001,"15min":20.486000000000001},"last":0.56200000000000006},{"interface":"front","average":{"1min":2.1739999999999999,"5min":1.119,"15min":0.71199999999999997},"min":{"1min":0.26300000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":23.925999999999998,"5min":23.925999999999998,"15min":23.925999999999998},"last":0.57499999999999996}]},{"osd":3,"last update":"Tue Sep 17 07:07:31 2024","interfaces":[{"interface":"back","average":{"1min":6.8310000000000004,"5min":2.077,"15min":1.0609999999999999},"min":{"1min":0.25800000000000001,"5min":0.249,"15min":0.249},"max":{"1min":68.978999999999999,"5min":68.978999999999999,"15min":68.978999999999999},"last":0.61799999999999999},{"interface":"front","average":{"1min":6.851,"5min":2,"15min":1.0329999999999999},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":69.084000000000003,"5min":69.084000000000003,"15min":69.084000000000003},"last":0.40300000000000002}]},{"osd":4,"last update":"Tue Sep 17 07:07:31 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.436,"15min":0.41099999999999998},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":3.2989999999999999,"5min":3.2989999999999999,"15min":3.2989999999999999},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.439,"5min":0.442,"15min":0.437},"min":{"1min":0.26700000000000002,"5min":0.248,"15min":0.248},"max":{"1min":0.90900000000000003,"5min":2.2160000000000002,"15min":2.2160000000000002},"last":0.51100000000000001}]},{"osd":6,"last update":"Tue Sep 17 07:07:48 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.45900000000000002,"15min":0.437},"min":{"1min":0.23499999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":1.9199999999999999,"5min":1.9199999999999999,"15min":1.9199999999999999},"last":0.498},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.51900000000000002,"15min":0.46999999999999997},"min":{"1min":0.31900000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":2.3250000000000002,"5min":2.5590000000000002,"15min":2.5590000000000002},"last":0.59299999999999997}]},{"osd":7,"last update":"Tue Sep 17 07:08:06 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.55000000000000004,"15min":0.504},"min":{"1min":0.308,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":4.0999999999999996,"5min":4.0999999999999996,"15min":4.0999999999999996},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.51800000000000002,"15min":0.48199999999999998},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":3.9100000000000001,"5min":3.9100000000000001,"15min":3.9100000000000001},"last":0.60899999999999999}]}]},{"osd":4,"up_from":33,"seq":141733920861,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":233708,"kb_used_data":108256,"kb_used_omap":1,"kb_used_meta":125438,"kb_avail":93511444,"statfs":{"total":95995035648,"available":95755718656,"internally_reserved":0,"allocated":110854144,"data_stored":110435043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":128448966},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:07:58 2024","interfaces":[{"interface":"back","average":{"1min":3.4329999999999998,"5min":3.2469999999999999,"15min":1.464},"min":{"1min":0.218,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":58.097999999999999,"5min":74.551000000000002,"15min":74.551000000000002},"last":0.32400000000000001},{"interface":"front","average":{"1min":3.7610000000000001,"5min":3.4860000000000002,"15min":1.5700000000000001},"min":{"1min":0.248,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":58.383000000000003,"5min":74.778999999999996,"15min":74.778999999999996},"last":0.74299999999999999}]},{"osd":1,"last update":"Tue Sep 17 07:07:58 2024","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.67500000000000004,"15min":0.60499999999999998},"min":{"1min":0.27700000000000002,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":3.1360000000000001,"5min":4.1539999999999999,"15min":4.1539999999999999},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.69099999999999995,"15min":0.624},"min":{"1min":0.255,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":3.0379999999999998,"5min":3.5409999999999999,"15min":3.5409999999999999},"last":0.28100000000000003}]},{"osd":2,"last update":"Tue Sep 17 07:07:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":1.258,"15min":0.80600000000000005},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":2.2610000000000001,"5min":49.651000000000003,"15min":49.651000000000003},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.85699999999999998,"5min":1.3759999999999999,"15min":0.84899999999999998},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":3.2589999999999999,"5min":48.813000000000002,"15min":48.813000000000002},"last":0.441}]},{"osd":3,"last update":"Tue Sep 17 07:07:58 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":1.2050000000000001,"15min":0.80500000000000005},"min":{"1min":0.28899999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":2.4199999999999999,"5min":39.042999999999999,"15min":39.042999999999999},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.82999999999999996,"5min":1.3100000000000001,"15min":0.81200000000000006},"min":{"1min":0.29999999999999999,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":6.157,"5min":38.920000000000002,"15min":38.920000000000002},"last":0.59999999999999998}]},{"osd":5,"last update":"Tue Sep 17 07:07:27 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.47499999999999998,"15min":0.41199999999999998},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":1.9510000000000001,"5min":2.4780000000000002,"15min":2.4780000000000002},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.48599999999999999,"15min":0.42999999999999999},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":2.0299999999999998,"5min":2.5419999999999998,"15min":2.5419999999999998},"last":0.375}]},{"osd":6,"last update":"Tue Sep 17 07:07:47 2024","interfaces":[{"interface":"back","average":{"1min":3.1680000000000001,"5min":1.0429999999999999,"15min":0.60999999999999999},"min":{"1min":0.249,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":47.923999999999999,"5min":47.923999999999999,"15min":47.923999999999999},"last":0.46300000000000002},{"interface":"front","average":{"1min":3.2530000000000001,"5min":1.0529999999999999,"15min":0.62},"min":{"1min":0.28899999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":47.600999999999999,"5min":47.600999999999999,"15min":47.600999999999999},"last":0.36699999999999999}]},{"osd":7,"last update":"Tue Sep 17 07:08:09 2024","interfaces":[{"interface":"back","average":{"1min":3.0619999999999998,"5min":1.052,"15min":0.68400000000000005},"min":{"1min":0.32400000000000001,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":48.719000000000001,"5min":48.719000000000001,"15min":48.719000000000001},"last":0.52900000000000003},{"interface":"front","average":{"1min":3.069,"5min":1.0489999999999999,"15min":0.66900000000000004},"min":{"1min":0.35099999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":50.527000000000001,"5min":50.527000000000001,"15min":50.527000000000001},"last":0.42499999999999999}]}]},{"osd":3,"up_from":27,"seq":115964117087,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":315140,"kb_used_data":131448,"kb_used_omap":1,"kb_used_meta":183678,"kb_avail":93430012,"statfs":{"total":95995035648,"available":95672332288,"internally_reserved":0,"allocated":134602752,"data_stored":134159859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":188086728},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:07:44 2024","interfaces":[{"interface":"back","average":{"1min":2.0739999999999998,"5min":1.54,"15min":0.75600000000000001},"min":{"1min":0.21199999999999999,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":8.9849999999999994,"5min":17.364000000000001,"15min":17.364000000000001},"last":1.3660000000000001},{"interface":"front","average":{"1min":1.7729999999999999,"5min":1.4099999999999999,"15min":0.72299999999999998},"min":{"1min":0.27400000000000002,"5min":0.20399999999999999,"15min":0.189},"max":{"1min":5.9560000000000004,"5min":18.521000000000001,"15min":18.521000000000001},"last":0.39600000000000002}]},{"osd":1,"last update":"Tue Sep 17 07:07:44 2024","interfaces":[{"interface":"back","average":{"1min":1.163,"5min":0.94599999999999995,"15min":0.58599999999999997},"min":{"1min":0.20999999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":8.3300000000000001,"5min":8.3300000000000001,"15min":8.3300000000000001},"last":0.29899999999999999},{"interface":"front","average":{"1min":1.552,"5min":1.2370000000000001,"15min":0.71299999999999997},"min":{"1min":0.19800000000000001,"5min":0.182,"15min":0.182},"max":{"1min":5.258,"5min":8.0670000000000002,"15min":8.0670000000000002},"last":0.53500000000000003}]},{"osd":2,"last update":"Tue Sep 17 07:07:44 2024","interfaces":[{"interface":"back","average":{"1min":2.1720000000000002,"5min":1.5569999999999999,"15min":0.80100000000000005},"min":{"1min":0.26100000000000001,"5min":0.22,"15min":0.22},"max":{"1min":5.3419999999999996,"5min":9.7319999999999993,"15min":9.7319999999999993},"last":0.47199999999999998},{"interface":"front","average":{"1min":1.2350000000000001,"5min":1.1899999999999999,"15min":0.67700000000000005},"min":{"1min":0.25,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":4.5049999999999999,"5min":8.3350000000000009,"15min":8.3350000000000009},"last":0.35999999999999999}]},{"osd":4,"last update":"Tue Sep 17 07:07:59 2024","interfaces":[{"interface":"back","average":{"1min":0.94999999999999996,"5min":1.05,"15min":0.72499999999999998},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":3.4470000000000001,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":0.441},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.877,"15min":0.64800000000000002},"min":{"1min":0.33100000000000002,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":2.9129999999999998,"5min":5.7850000000000001,"15min":5.7850000000000001},"last":0.377}]},{"osd":5,"last update":"Tue Sep 17 07:07:24 2024","interfaces":[{"interface":"back","average":{"1min":1.988,"5min":1.2090000000000001,"15min":0.80900000000000005},"min":{"1min":0.309,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":5.3200000000000003,"5min":7.9669999999999996,"15min":7.9669999999999996},"last":1.258},{"interface":"front","average":{"1min":1.3899999999999999,"5min":0.98399999999999999,"15min":0.71899999999999997},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":3.9039999999999999,"5min":4.4160000000000004,"15min":4.4160000000000004},"last":3.1440000000000001}]},{"osd":6,"last update":"Tue Sep 17 07:07:44 2024","interfaces":[{"interface":"back","average":{"1min":1.73,"5min":1.2090000000000001,"15min":0.77900000000000003},"min":{"1min":0.36299999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":8.2919999999999998,"5min":8.2919999999999998,"15min":8.2919999999999998},"last":0.41299999999999998},{"interface":"front","average":{"1min":1.5129999999999999,"5min":1.2190000000000001,"15min":0.80000000000000004},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":5.5529999999999999,"5min":7.9329999999999998,"15min":7.9329999999999998},"last":0.50700000000000001}]},{"osd":7,"last update":"Tue Sep 17 07:08:06 2024","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":1.143,"15min":0.77300000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":3.4820000000000002,"5min":6.1440000000000001,"15min":6.1440000000000001},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.84799999999999998,"5min":1.125,"15min":0.747},"min":{"1min":0.39500000000000002,"5min":0.38500000000000001,"15min":0.34599999999999997},"max":{"1min":2.9329999999999998,"5min":5.8170000000000002,"15min":5.8170000000000002},"last":2.9060000000000001}]}]},{"osd":2,"up_from":18,"seq":77309411429,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":386020,"kb_used_data":156376,"kb_used_omap":1,"kb_used_meta":229630,"kb_avail":93359132,"statfs":{"total":95995035648,"available":95599751168,"internally_reserved":0,"allocated":160129024,"data_stored":159718648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":235141574},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:07:27 2024","interfaces":[{"interface":"back","average":{"1min":2,"5min":0.95499999999999996,"15min":0.54900000000000004},"min":{"1min":0.248,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":7.2709999999999999,"5min":7.2709999999999999,"15min":7.2709999999999999},"last":0.69099999999999995},{"interface":"front","average":{"1min":1.716,"5min":1.3700000000000001,"15min":0.69199999999999995},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":5.9610000000000003,"5min":8.6869999999999994,"15min":8.6869999999999994},"last":0.65700000000000003}]},{"osd":1,"last update":"Tue Sep 17 07:07:27 2024","interfaces":[{"interface":"back","average":{"1min":1.5349999999999999,"5min":0.93600000000000005,"15min":0.54900000000000004},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":5.5010000000000003,"5min":5.5010000000000003,"15min":5.5010000000000003},"last":0.24399999999999999},{"interface":"front","average":{"1min":1.5880000000000001,"5min":0.84899999999999998,"15min":0.53300000000000003},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":6.9829999999999997,"5min":6.9829999999999997,"15min":6.9829999999999997},"last":0.84699999999999998}]},{"osd":3,"last update":"Tue Sep 17 07:07:47 2024","interfaces":[{"interface":"back","average":{"1min":1.129,"5min":0.88300000000000001,"15min":0.58699999999999997},"min":{"1min":0.27100000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":5.4930000000000003,"5min":5.4930000000000003,"15min":5.4930000000000003},"last":1.2390000000000001},{"interface":"front","average":{"1min":1.254,"5min":1.294,"15min":0.71799999999999997},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":6.0679999999999996,"5min":6.0679999999999996,"15min":6.0679999999999996},"last":0.70899999999999996}]},{"osd":4,"last update":"Tue Sep 17 07:08:06 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.86799999999999999,"15min":0.66600000000000004},"min":{"1min":0.31900000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.73599999999999999,"5min":5.2359999999999998,"15min":5.2359999999999998},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.77200000000000002,"15min":0.64900000000000002},"min":{"1min":0.378,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":1.855,"5min":3.4689999999999999,"15min":3.4689999999999999},"last":0.81799999999999995}]},{"osd":5,"last update":"Tue Sep 17 07:07:27 2024","interfaces":[{"interface":"back","average":{"1min":1.381,"5min":1.006,"15min":0.73199999999999998},"min":{"1min":0.26100000000000001,"5min":0.249,"15min":0.249},"max":{"1min":5.383,"5min":5.383,"15min":5.383},"last":0.59699999999999998},{"interface":"front","average":{"1min":1.121,"5min":0.85099999999999998,"15min":0.68100000000000005},"min":{"1min":0.34699999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":2.6880000000000002,"5min":5.3600000000000003,"15min":5.3600000000000003},"last":0.41399999999999998}]},{"osd":6,"last update":"Tue Sep 17 07:07:47 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.90500000000000003,"15min":0.70399999999999996},"min":{"1min":0.33600000000000002,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":2.1459999999999999,"5min":9.2509999999999994,"15min":9.2509999999999994},"last":0.72199999999999998},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.0700000000000001,"15min":0.754},"min":{"1min":0.33400000000000002,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":5.3330000000000002,"5min":5.3330000000000002,"15min":5.3330000000000002},"last":0.63400000000000001}]},{"osd":7,"last update":"Tue Sep 17 07:07:17 2024","interfaces":[{"interface":"back","average":{"1min":1.1759999999999999,"5min":0.90500000000000003,"15min":0.68200000000000005},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":2.7090000000000001,"5min":5.3250000000000002,"15min":5.3250000000000002},"last":0.378},{"interface":"front","average":{"1min":1.9950000000000001,"5min":1.0129999999999999,"15min":0.85899999999999999},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":9.2919999999999998,"5min":9.2919999999999998,"15min":9.2919999999999998},"last":0.746}]}]},{"osd":1,"up_from":13,"seq":55834574954,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":280844,"kb_used_data":163584,"kb_used_omap":1,"kb_used_meta":117246,"kb_avail":93464308,"statfs":{"total":95995035648,"available":95707451392,"internally_reserved":0,"allocated":167510016,"data_stored":167060072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":120060358},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:07:52 2024","interfaces":[{"interface":"back","average":{"1min":1.552,"5min":1.339,"15min":0.67100000000000004},"min":{"1min":0.19400000000000001,"5min":0.155,"15min":0.155},"max":{"1min":6.6020000000000003,"5min":13.837,"15min":13.837},"last":0.78600000000000003},{"interface":"front","average":{"1min":1.0840000000000001,"5min":1.304,"15min":0.67900000000000005},"min":{"1min":0.22700000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":5.6020000000000003,"5min":10.342000000000001,"15min":10.342000000000001},"last":0.46800000000000003}]},{"osd":2,"last update":"Tue Sep 17 07:07:21 2024","interfaces":[{"interface":"back","average":{"1min":1.8,"5min":1.2150000000000001,"15min":0.67400000000000004},"min":{"1min":0.23799999999999999,"5min":0.192,"15min":0.192},"max":{"1min":5.9509999999999996,"5min":11.055999999999999,"15min":11.055999999999999},"last":0.45500000000000002},{"interface":"front","average":{"1min":1.4470000000000001,"5min":1.302,"15min":0.70999999999999996},"min":{"1min":0.24199999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":5.4269999999999996,"5min":9.3160000000000007,"15min":9.3160000000000007},"last":0.50600000000000001}]},{"osd":3,"last update":"Tue Sep 17 07:07:46 2024","interfaces":[{"interface":"back","average":{"1min":1.125,"5min":1.2629999999999999,"15min":0.72899999999999998},"min":{"1min":0.27800000000000002,"5min":0.219,"15min":0.219},"max":{"1min":5.4740000000000002,"5min":7.766,"15min":7.766},"last":0.59899999999999998},{"interface":"front","average":{"1min":1.7030000000000001,"5min":1.351,"15min":0.745},"min":{"1min":0.28899999999999998,"5min":0.222,"15min":0.222},"max":{"1min":7.2759999999999998,"5min":19.602,"15min":19.602},"last":1.9790000000000001}]},{"osd":4,"last update":"Tue Sep 17 07:08:06 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":1.0229999999999999,"15min":0.72399999999999998},"min":{"1min":0.32100000000000001,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":1.6739999999999999,"5min":7.8840000000000003,"15min":7.8840000000000003},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.88400000000000001,"5min":1.0129999999999999,"15min":0.71699999999999997},"min":{"1min":0.32000000000000001,"5min":0.307,"15min":0.307},"max":{"1min":4.133,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":1.8069999999999999}]},{"osd":5,"last update":"Tue Sep 17 07:07:21 2024","interfaces":[{"interface":"back","average":{"1min":1.3129999999999999,"5min":1.087,"15min":0.74299999999999999},"min":{"1min":0.31,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":5.5259999999999998,"5min":6.0060000000000002,"15min":6.0060000000000002},"last":0.71599999999999997},{"interface":"front","average":{"1min":1.623,"5min":1.0600000000000001,"15min":0.72999999999999998},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":7.2080000000000002,"5min":8.8889999999999993,"15min":8.8889999999999993},"last":1.8999999999999999}]},{"osd":6,"last update":"Tue Sep 17 07:07:47 2024","interfaces":[{"interface":"back","average":{"1min":1.764,"5min":1.115,"15min":0.73399999999999999},"min":{"1min":0.36899999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":7.2539999999999996,"5min":8.843,"15min":8.843},"last":1.8440000000000001},{"interface":"front","average":{"1min":1.276,"5min":1.0669999999999999,"15min":0.753},"min":{"1min":0.38300000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":7.1130000000000004,"5min":7.1130000000000004,"15min":7.1130000000000004},"last":0.68899999999999995}]},{"osd":7,"last update":"Tue Sep 17 07:08:09 2024","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":1.079,"15min":0.73499999999999999},"min":{"1min":0.38500000000000001,"5min":0.37,"15min":0.37},"max":{"1min":4.2089999999999996,"5min":8.9309999999999992,"15min":8.9309999999999992},"last":1.8720000000000001},{"interface":"front","average":{"1min":0.69099999999999995,"5min":1.0449999999999999,"15min":0.72899999999999998},"min":{"1min":0.41499999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":2.7120000000000002,"5min":11.007999999999999,"15min":11.007999999999999},"last":0.752}]}]},{"osd":0,"up_from":8,"seq":34359738477,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":425220,"kb_used_data":187380,"kb_used_omap":1,"kb_used_meta":237822,"kb_avail":93319932,"statfs":{"total":95995035648,"available":95559610368,"internally_reserved":0,"allocated":191877120,"data_stored":191445751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":243530182},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 17 07:07:54 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":1.107,"15min":0.58999999999999997},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":5.4489999999999998,"5min":8.0259999999999998,"15min":8.0259999999999998},"last":0.83599999999999997},{"interface":"front","average":{"1min":1.538,"5min":1.1970000000000001,"15min":0.628},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":6.867,"5min":7.8849999999999998,"15min":7.8849999999999998},"last":2.157}]},{"osd":2,"last update":"Tue Sep 17 07:07:24 2024","interfaces":[{"interface":"back","average":{"1min":2.1619999999999999,"5min":1.7829999999999999,"15min":0.84599999999999997},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":18.643999999999998,"5min":18.643999999999998,"15min":18.643999999999998},"last":0.35299999999999998},{"interface":"front","average":{"1min":2.839,"5min":1.5369999999999999,"15min":0.754},"min":{"1min":0.27200000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":17.972999999999999,"5min":17.972999999999999,"15min":17.972999999999999},"last":0.59799999999999998}]},{"osd":3,"last update":"Tue Sep 17 07:07:54 2024","interfaces":[{"interface":"back","average":{"1min":1.8129999999999999,"5min":1.6160000000000001,"15min":0.83099999999999996},"min":{"1min":0.315,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":11.598000000000001,"5min":18.629000000000001,"15min":18.629000000000001},"last":0.54000000000000004},{"interface":"front","average":{"1min":1.1699999999999999,"5min":1.782,"15min":0.88300000000000001},"min":{"1min":0.30399999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":5.4169999999999998,"5min":22.358000000000001,"15min":22.358000000000001},"last":0.39900000000000002}]},{"osd":4,"last update":"Tue Sep 17 07:08:06 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":1.385,"15min":0.874},"min":{"1min":0.32100000000000001,"5min":0.316,"15min":0.316},"max":{"1min":1.54,"5min":18.568000000000001,"15min":18.568000000000001},"last":0.42899999999999999},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1.3080000000000001,"15min":0.80000000000000004},"min":{"1min":0.27400000000000002,"5min":0.255,"15min":0.255},"max":{"1min":4.7709999999999999,"5min":18.765000000000001,"15min":18.765000000000001},"last":0.38500000000000001}]},{"osd":5,"last update":"Tue Sep 17 07:07:29 2024","interfaces":[{"interface":"back","average":{"1min":2.3919999999999999,"5min":1.341,"15min":0.81200000000000006},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":18.614999999999998,"5min":18.614999999999998,"15min":18.614999999999998},"last":1.052},{"interface":"front","average":{"1min":1.9870000000000001,"5min":1.23,"15min":0.76900000000000002},"min":{"1min":0.33600000000000002,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":18.791,"5min":18.791,"15min":18.791},"last":1.75}]},{"osd":6,"last update":"Tue Sep 17 07:07:54 2024","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.3129999999999999,"15min":0.80100000000000005},"min":{"1min":0.34699999999999998,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":5.2240000000000002,"5min":18.602,"15min":18.602},"last":0.871},{"interface":"front","average":{"1min":1.2010000000000001,"5min":1.496,"15min":0.86399999999999999},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":6.149,"5min":18.692,"15min":18.692},"last":0.77300000000000002}]},{"osd":7,"last update":"Tue Sep 17 07:08:14 2024","interfaces":[{"interface":"back","average":{"1min":1.113,"5min":1.8620000000000001,"15min":1.421},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":4.8490000000000002,"5min":46.183,"15min":46.183},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.98799999999999999,"5min":1.7669999999999999,"15min":0.96299999999999997},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":5.0190000000000001,"5min":45.914999999999999,"15min":45.914999999999999},"last":0.93200000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":243924992,"data_stored":243924992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":202309632,"data_stored":202309632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":185073664,"data_stored":185073664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":216399872,"data_stored":216399872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":140705792,"data_stored":140705792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":220987392,"data_stored":220987392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":141295616,"data_stored":141295616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":154730496,"data_stored":154730496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T07:08:22.874 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj 6, currently 16 2024-09-17T07:08:22.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:22.876 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:22.876 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:08:22.976 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:08:23.010 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:23.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:23.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:08:23.042 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 7 16 20947 20931 186.864 167.812 0.0015894 0.00532549 2024-09-17T07:08:23.112 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:08:23.141 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:23.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:23.142 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:08:23.243 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:08:23.268 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:23.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:23.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:08:23.370 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:08:23.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:23.406 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:23.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:08:23.507 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:08:23.542 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:23.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:23.544 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:08:23.644 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:08:23.681 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:23.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:23.682 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:08:23.783 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:08:23.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:23 smithi135 ceph-mon[33238]: 6.1c scrub starts 2024-09-17T07:08:23.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:23 smithi135 ceph-mon[33238]: 6.1c scrub ok 2024-09-17T07:08:23.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:23 smithi135 ceph-mon[33238]: 4.18 scrub starts 2024-09-17T07:08:23.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:23 smithi135 ceph-mon[33238]: 4.18 scrub ok 2024-09-17T07:08:23.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:23 smithi135 ceph-mon[33238]: 5.17 scrub starts 2024-09-17T07:08:23.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:23 smithi135 ceph-mon[33238]: 5.17 scrub ok 2024-09-17T07:08:23.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:23 smithi135 ceph-mon[33238]: pgmap v270: 148 pgs: 148 active+clean; 713 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 119 MiB/s wr, 1.90k op/s 2024-09-17T07:08:23.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:23.810 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:23.810 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:08:23.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:23 smithi100 ceph-mon[29092]: 6.1c scrub starts 2024-09-17T07:08:23.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:23 smithi100 ceph-mon[29092]: 6.1c scrub ok 2024-09-17T07:08:23.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:23 smithi100 ceph-mon[29092]: 4.18 scrub starts 2024-09-17T07:08:23.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:23 smithi100 ceph-mon[29092]: 4.18 scrub ok 2024-09-17T07:08:23.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:23 smithi100 ceph-mon[29092]: 5.17 scrub starts 2024-09-17T07:08:23.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:23 smithi100 ceph-mon[29092]: 5.17 scrub ok 2024-09-17T07:08:23.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:23 smithi100 ceph-mon[29092]: pgmap v270: 148 pgs: 148 active+clean; 713 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 119 MiB/s wr, 1.90k op/s 2024-09-17T07:08:23.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:23 smithi100 ceph-mon[39462]: 6.1c scrub starts 2024-09-17T07:08:23.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:23 smithi100 ceph-mon[39462]: 6.1c scrub ok 2024-09-17T07:08:23.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:23 smithi100 ceph-mon[39462]: 4.18 scrub starts 2024-09-17T07:08:23.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:23 smithi100 ceph-mon[39462]: 4.18 scrub ok 2024-09-17T07:08:23.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:23 smithi100 ceph-mon[39462]: 5.17 scrub starts 2024-09-17T07:08:23.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:23 smithi100 ceph-mon[39462]: 5.17 scrub ok 2024-09-17T07:08:23.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:23 smithi100 ceph-mon[39462]: pgmap v270: 148 pgs: 148 active+clean; 713 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 119 MiB/s wr, 1.90k op/s 2024-09-17T07:08:23.911 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:08:23.943 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:23.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:23.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:08:24.043 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 8 16 23202 23186 181.121 140.938 0.00945465 0.00548851 2024-09-17T07:08:24.044 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:08:24.077 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:24.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:24.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:08:24.180 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:08:24.215 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:24.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:24.216 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:08:24.317 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:08:24.350 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:24.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:24.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:08:24.452 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:08:24.484 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:24.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:24.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:08:24.586 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:08:24.610 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:24 smithi100 ceph-mon[29092]: 4.13 scrub starts 2024-09-17T07:08:24.610 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:24 smithi100 ceph-mon[29092]: 4.13 scrub ok 2024-09-17T07:08:24.610 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:24 smithi100 ceph-mon[29092]: 2.a scrub starts 2024-09-17T07:08:24.610 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:24 smithi100 ceph-mon[29092]: 2.a scrub ok 2024-09-17T07:08:24.610 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:24 smithi100 ceph-mon[29092]: 2.1 deep-scrub starts 2024-09-17T07:08:24.610 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:24 smithi100 ceph-mon[29092]: 2.1 deep-scrub ok 2024-09-17T07:08:24.610 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:24 smithi100 ceph-mon[39462]: 4.13 scrub starts 2024-09-17T07:08:24.611 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:24 smithi100 ceph-mon[39462]: 4.13 scrub ok 2024-09-17T07:08:24.611 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:24 smithi100 ceph-mon[39462]: 2.a scrub starts 2024-09-17T07:08:24.611 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:24 smithi100 ceph-mon[39462]: 2.a scrub ok 2024-09-17T07:08:24.611 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:24 smithi100 ceph-mon[39462]: 2.1 deep-scrub starts 2024-09-17T07:08:24.611 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:24 smithi100 ceph-mon[39462]: 2.1 deep-scrub ok 2024-09-17T07:08:24.621 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:24.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:24.622 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:08:24.723 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:08:24.756 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:24.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:24.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:08:24.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:24 smithi135 ceph-mon[33238]: 4.13 scrub starts 2024-09-17T07:08:24.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:24 smithi135 ceph-mon[33238]: 4.13 scrub ok 2024-09-17T07:08:24.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:24 smithi135 ceph-mon[33238]: 2.a scrub starts 2024-09-17T07:08:24.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:24 smithi135 ceph-mon[33238]: 2.a scrub ok 2024-09-17T07:08:24.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:24 smithi135 ceph-mon[33238]: 2.1 deep-scrub starts 2024-09-17T07:08:24.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:24 smithi135 ceph-mon[33238]: 2.1 deep-scrub ok 2024-09-17T07:08:24.858 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:08:24.896 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:24.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:24.897 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:08:24.998 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:08:25.040 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:25.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:25.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:08:25.043 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 9 16 25733 25717 178.562 158.188 0.00956471 0.00559307 2024-09-17T07:08:25.054 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:08:25.142 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:08:25.183 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:25.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:25.184 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:08:25.285 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:08:25.322 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:25.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:25.323 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:08:25.423 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:08:25.448 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:25.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:25.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:08:25.550 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:08:25.583 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:25.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:25.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:08:25.611 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:25 smithi100 ceph-mon[29092]: 2.5 deep-scrub starts 2024-09-17T07:08:25.611 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:25 smithi100 ceph-mon[29092]: 2.5 deep-scrub ok 2024-09-17T07:08:25.611 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:25 smithi100 ceph-mon[29092]: 5.1f scrub starts 2024-09-17T07:08:25.612 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:25 smithi100 ceph-mon[29092]: 5.1f scrub ok 2024-09-17T07:08:25.612 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:25 smithi100 ceph-mon[29092]: 6.1 scrub starts 2024-09-17T07:08:25.612 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:25 smithi100 ceph-mon[29092]: 6.1 scrub ok 2024-09-17T07:08:25.612 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:25 smithi100 ceph-mon[29092]: osdmap e178: 8 total, 8 up, 8 in 2024-09-17T07:08:25.612 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:25 smithi100 ceph-mon[29092]: pgmap v272: 148 pgs: 148 active+clean; 1.3 GiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 751 B/s rd, 159 MiB/s wr, 2.54k op/s 2024-09-17T07:08:25.612 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:25 smithi100 ceph-mon[39462]: 2.5 deep-scrub starts 2024-09-17T07:08:25.612 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:25 smithi100 ceph-mon[39462]: 2.5 deep-scrub ok 2024-09-17T07:08:25.612 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:25 smithi100 ceph-mon[39462]: 5.1f scrub starts 2024-09-17T07:08:25.613 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:25 smithi100 ceph-mon[39462]: 5.1f scrub ok 2024-09-17T07:08:25.613 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:25 smithi100 ceph-mon[39462]: 6.1 scrub starts 2024-09-17T07:08:25.613 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:25 smithi100 ceph-mon[39462]: 6.1 scrub ok 2024-09-17T07:08:25.613 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:25 smithi100 ceph-mon[39462]: osdmap e178: 8 total, 8 up, 8 in 2024-09-17T07:08:25.613 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:25 smithi100 ceph-mon[39462]: pgmap v272: 148 pgs: 148 active+clean; 1.3 GiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 751 B/s rd, 159 MiB/s wr, 2.54k op/s 2024-09-17T07:08:25.686 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:08:25.717 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:25.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:25.719 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:08:25.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:25 smithi135 ceph-mon[33238]: 2.5 deep-scrub starts 2024-09-17T07:08:25.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:25 smithi135 ceph-mon[33238]: 2.5 deep-scrub ok 2024-09-17T07:08:25.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:25 smithi135 ceph-mon[33238]: 5.1f scrub starts 2024-09-17T07:08:25.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:25 smithi135 ceph-mon[33238]: 5.1f scrub ok 2024-09-17T07:08:25.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:25 smithi135 ceph-mon[33238]: 6.1 scrub starts 2024-09-17T07:08:25.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:25 smithi135 ceph-mon[33238]: 6.1 scrub ok 2024-09-17T07:08:25.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:25 smithi135 ceph-mon[33238]: osdmap e178: 8 total, 8 up, 8 in 2024-09-17T07:08:25.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:25 smithi135 ceph-mon[33238]: pgmap v272: 148 pgs: 148 active+clean; 1.3 GiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 751 B/s rd, 159 MiB/s wr, 2.54k op/s 2024-09-17T07:08:25.821 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:08:25.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:25.859 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:25.859 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:08:25.896 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-17T07:08:25.896 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-17T07:08:25.896 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:25.896 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.24929.0:27360 8.a 8:57fedbf6:::benchmark_data_smithi100_102803_object27359:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-09-17T07:08:25.896 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:08:25.874285+0000", 2024-09-17T07:08:25.896 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.020924371000000001, 2024-09-17T07:08:25.896 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.020951911, 2024-09-17T07:08:25.896 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:08:25.897 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "waiting for sub ops", 2024-09-17T07:08:25.897 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:08:25.897 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.24929", 2024-09-17T07:08:25.897 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/564515582", 2024-09-17T07:08:25.897 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 27360 2024-09-17T07:08:25.897 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:25.897 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:08:25.897 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:25.897 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:08:25.897 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:25.874285+0000", 2024-09-17T07:08:25.897 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:25.897 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:25.897 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:25.898 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:08:25.898 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:25.874285+0000", 2024-09-17T07:08:25.898 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:25.898 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:25.898 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:25.898 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:08:25.898 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:25.874284+0000", 2024-09-17T07:08:25.898 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999986 2024-09-17T07:08:25.898 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:25.898 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:25.898 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:08:25.898 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:25.874304+0000", 2024-09-17T07:08:25.898 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0883999999999999e-05 2024-09-17T07:08:25.899 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:25.899 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:25.899 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:08:25.899 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:25.874316+0000", 2024-09-17T07:08:25.899 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1943999999999999e-05 2024-09-17T07:08:25.899 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:25.899 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:25.899 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:08:25.899 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:25.874322+0000", 2024-09-17T07:08:25.899 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5300000000000004e-06 2024-09-17T07:08:25.899 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:25.899 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:25.899 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:08:25.900 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:25.874358+0000", 2024-09-17T07:08:25.900 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6193999999999999e-05 2024-09-17T07:08:25.900 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:25.900 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:25.900 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:08:25.900 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:25.874447+0000", 2024-09-17T07:08:25.900 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.9139000000000002e-05 2024-09-17T07:08:25.900 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:25.900 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:25.900 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 2,5", 2024-09-17T07:08:25.900 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:25.874560+0000", 2024-09-17T07:08:25.900 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000112836 2024-09-17T07:08:25.900 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:25.901 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:25.901 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:08:25.901 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:25.875530+0000", 2024-09-17T07:08:25.901 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00097026800000000004 2024-09-17T07:08:25.901 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:25.901 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:25.901 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:08:25.901 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:25.875825+0000", 2024-09-17T07:08:25.901 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00029449800000000003 2024-09-17T07:08:25.901 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:25.901 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:25.901 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:25.901 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:25.902 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:25.902 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.24929.0:27330 8.a 8:5fb86fd5:::benchmark_data_smithi100_102803_object27329:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-09-17T07:08:25.902 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:08:25.856057+0000", 2024-09-17T07:08:25.902 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.039152467000000003, 2024-09-17T07:08:25.902 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039235267999999997, 2024-09-17T07:08:25.902 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:08:25.902 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "waiting for sub ops", 2024-09-17T07:08:25.902 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:08:25.902 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.24929", 2024-09-17T07:08:25.902 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/564515582", 2024-09-17T07:08:25.902 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 27330 2024-09-17T07:08:25.902 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:25.903 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:08:25.903 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:25.903 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:08:25.903 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:25.856057+0000", 2024-09-17T07:08:25.903 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:25.903 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:25.903 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:25.903 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:08:25.903 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:25.856057+0000", 2024-09-17T07:08:25.904 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:25.904 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:25.904 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:25.904 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:08:25.904 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:25.856056+0000", 2024-09-17T07:08:25.904 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-17T07:08:25.904 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:25.904 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:25.904 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:08:25.904 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:25.856077+0000", 2024-09-17T07:08:25.904 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0645999999999999e-05 2024-09-17T07:08:25.904 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:25.905 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:25.905 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:08:25.905 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:25.856088+0000", 2024-09-17T07:08:25.905 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1233000000000001e-05 2024-09-17T07:08:25.905 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:25.905 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:25.905 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:08:25.905 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:25.856094+0000", 2024-09-17T07:08:25.905 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5869999999999998e-06 2024-09-17T07:08:25.905 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:25.905 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:25.905 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:08:25.905 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:25.856140+0000", 2024-09-17T07:08:25.905 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5698999999999997e-05 2024-09-17T07:08:25.906 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:25.906 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:25.906 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:08:25.906 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:25.856262+0000", 2024-09-17T07:08:25.906 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012150300000000001 2024-09-17T07:08:25.906 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:25.906 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:25.906 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 2,5", 2024-09-17T07:08:25.906 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:25.856394+0000", 2024-09-17T07:08:25.906 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00013293199999999999 2024-09-17T07:08:25.906 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:25.906 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:25.906 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:08:25.907 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:25.858534+0000", 2024-09-17T07:08:25.907 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0021396710000000001 2024-09-17T07:08:25.907 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:25.907 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:25.907 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:08:25.907 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:25.859608+0000", 2024-09-17T07:08:25.907 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001073449 2024-09-17T07:08:25.907 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:25.907 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:25.907 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:25.907 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:25.907 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-17T07:08:25.907 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 2 2024-09-17T07:08:25.908 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-17T07:08:25.960 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:08:25.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:25.993 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:25.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:08:26.043 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 10 16 27630 27614 172.562 118.562 0.00728163 0.00578915 2024-09-17T07:08:26.094 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:08:26.133 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:26.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:26.323 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:08:26.323 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:08:25 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:08:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:08:26.424 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:08:26.458 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:26.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:26.459 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:08:26.560 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:08:26.581 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:26.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:26.582 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:08:26.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:26 smithi100 ceph-mon[29092]: 2.12 scrub starts 2024-09-17T07:08:26.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:26 smithi100 ceph-mon[29092]: 2.12 scrub ok 2024-09-17T07:08:26.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:26 smithi100 ceph-mon[29092]: 6.0 scrub starts 2024-09-17T07:08:26.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:26 smithi100 ceph-mon[29092]: 6.0 scrub ok 2024-09-17T07:08:26.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:26 smithi100 ceph-mon[29092]: 6.2 scrub starts 2024-09-17T07:08:26.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:26 smithi100 ceph-mon[29092]: 6.2 scrub ok 2024-09-17T07:08:26.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:26 smithi100 ceph-mon[29092]: 5.1d deep-scrub starts 2024-09-17T07:08:26.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:26 smithi100 ceph-mon[29092]: 5.1d deep-scrub ok 2024-09-17T07:08:26.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:26 smithi100 ceph-mon[29092]: 4.11 scrub starts 2024-09-17T07:08:26.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:26 smithi100 ceph-mon[29092]: 4.11 scrub ok 2024-09-17T07:08:26.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:26 smithi100 ceph-mon[39462]: 2.12 scrub starts 2024-09-17T07:08:26.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:26 smithi100 ceph-mon[39462]: 2.12 scrub ok 2024-09-17T07:08:26.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:26 smithi100 ceph-mon[39462]: 6.0 scrub starts 2024-09-17T07:08:26.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:26 smithi100 ceph-mon[39462]: 6.0 scrub ok 2024-09-17T07:08:26.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:26 smithi100 ceph-mon[39462]: 6.2 scrub starts 2024-09-17T07:08:26.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:26 smithi100 ceph-mon[39462]: 6.2 scrub ok 2024-09-17T07:08:26.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:26 smithi100 ceph-mon[39462]: 5.1d deep-scrub starts 2024-09-17T07:08:26.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:26 smithi100 ceph-mon[39462]: 5.1d deep-scrub ok 2024-09-17T07:08:26.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:26 smithi100 ceph-mon[39462]: 4.11 scrub starts 2024-09-17T07:08:26.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:26 smithi100 ceph-mon[39462]: 4.11 scrub ok 2024-09-17T07:08:26.682 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:08:26.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:26.718 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:26.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:08:26.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:26 smithi135 ceph-mon[33238]: 2.12 scrub starts 2024-09-17T07:08:26.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:26 smithi135 ceph-mon[33238]: 2.12 scrub ok 2024-09-17T07:08:26.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:26 smithi135 ceph-mon[33238]: 6.0 scrub starts 2024-09-17T07:08:26.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:26 smithi135 ceph-mon[33238]: 6.0 scrub ok 2024-09-17T07:08:26.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:26 smithi135 ceph-mon[33238]: 6.2 scrub starts 2024-09-17T07:08:26.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:26 smithi135 ceph-mon[33238]: 6.2 scrub ok 2024-09-17T07:08:26.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:26 smithi135 ceph-mon[33238]: 5.1d deep-scrub starts 2024-09-17T07:08:26.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:26 smithi135 ceph-mon[33238]: 5.1d deep-scrub ok 2024-09-17T07:08:26.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:26 smithi135 ceph-mon[33238]: 4.11 scrub starts 2024-09-17T07:08:26.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:26 smithi135 ceph-mon[33238]: 4.11 scrub ok 2024-09-17T07:08:26.819 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:08:26.841 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:26.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:26.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:08:26.943 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:08:26.979 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:26.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:26.980 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:08:27.043 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 11 16 29694 29678 168.601 129 0.00586613 0.00592529 2024-09-17T07:08:27.082 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:08:27.108 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:27.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:27.109 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:08:27.210 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:08:27.246 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:27.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:27.247 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:08:27.348 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:08:27.369 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:27.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:27.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:08:27.471 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:08:27.506 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:27.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:27.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:08:27.608 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:08:27.641 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:27.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:27.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:08:27.743 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:08:27.764 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:27.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:27.765 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:08:27.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:27 smithi135 ceph-mon[33238]: 4.e scrub starts 2024-09-17T07:08:27.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:27 smithi135 ceph-mon[33238]: 4.e scrub ok 2024-09-17T07:08:27.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:27 smithi135 ceph-mon[33238]: 4.4 scrub starts 2024-09-17T07:08:27.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:27 smithi135 ceph-mon[33238]: 4.4 scrub ok 2024-09-17T07:08:27.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:27 smithi135 ceph-mon[33238]: 6.12 scrub starts 2024-09-17T07:08:27.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:27 smithi135 ceph-mon[33238]: 6.12 scrub ok 2024-09-17T07:08:27.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:27 smithi135 ceph-mon[33238]: pgmap v273: 148 pgs: 148 active+clean; 1.3 GiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 108 MiB/s wr, 1.73k op/s 2024-09-17T07:08:27.869 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:08:27.875 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-17T07:08:27.875 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T07:08:27.876 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-17T07:08:27.876 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-17T07:08:27.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:27 smithi100 ceph-mon[29092]: 4.e scrub starts 2024-09-17T07:08:27.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:27 smithi100 ceph-mon[29092]: 4.e scrub ok 2024-09-17T07:08:27.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:27 smithi100 ceph-mon[29092]: 4.4 scrub starts 2024-09-17T07:08:27.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:27 smithi100 ceph-mon[29092]: 4.4 scrub ok 2024-09-17T07:08:27.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:27 smithi100 ceph-mon[29092]: 6.12 scrub starts 2024-09-17T07:08:27.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:27 smithi100 ceph-mon[29092]: 6.12 scrub ok 2024-09-17T07:08:27.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:27 smithi100 ceph-mon[29092]: pgmap v273: 148 pgs: 148 active+clean; 1.3 GiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 108 MiB/s wr, 1.73k op/s 2024-09-17T07:08:27.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:27 smithi100 ceph-mon[39462]: 4.e scrub starts 2024-09-17T07:08:27.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:27 smithi100 ceph-mon[39462]: 4.e scrub ok 2024-09-17T07:08:27.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:27 smithi100 ceph-mon[39462]: 4.4 scrub starts 2024-09-17T07:08:27.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:27 smithi100 ceph-mon[39462]: 4.4 scrub ok 2024-09-17T07:08:27.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:27 smithi100 ceph-mon[39462]: 6.12 scrub starts 2024-09-17T07:08:27.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:27 smithi100 ceph-mon[39462]: 6.12 scrub ok 2024-09-17T07:08:27.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:27 smithi100 ceph-mon[39462]: pgmap v273: 148 pgs: 148 active+clean; 1.3 GiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 108 MiB/s wr, 1.73k op/s 2024-09-17T07:08:27.902 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:27.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:27.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:08:28.004 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:08:28.026 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:28.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:28.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:08:28.043 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 12 16 31383 31367 163.347 105.562 0.00199714 0.0060933 2024-09-17T07:08:28.129 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:08:28.191 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:28.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:28.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:08:28.294 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:08:28.338 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:28.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:28.340 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:08:28.440 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:08:28.463 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:28.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:28.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:08:28.564 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:08:28.601 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:28.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:28.602 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:08:28.703 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:08:28.744 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:28.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:28.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:08:28.793 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:08:28 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:08:28.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:28 smithi135 ceph-mon[33238]: 5.12 scrub starts 2024-09-17T07:08:28.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:28 smithi135 ceph-mon[33238]: 5.12 scrub ok 2024-09-17T07:08:28.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:28 smithi135 ceph-mon[33238]: 6.c scrub starts 2024-09-17T07:08:28.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:28 smithi135 ceph-mon[33238]: 6.c scrub ok 2024-09-17T07:08:28.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:28 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:08:28.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:28 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:08:28.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:28 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:08:28.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:28 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:08:28.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:28 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:08:28.846 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:08:28.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:28 smithi100 ceph-mon[29092]: 5.12 scrub starts 2024-09-17T07:08:28.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:28 smithi100 ceph-mon[29092]: 5.12 scrub ok 2024-09-17T07:08:28.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:28 smithi100 ceph-mon[29092]: 6.c scrub starts 2024-09-17T07:08:28.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:28 smithi100 ceph-mon[29092]: 6.c scrub ok 2024-09-17T07:08:28.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:28 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:08:28.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:28 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:08:28.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:28 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:08:28.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:28 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:08:28.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:28 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:08:28.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:28 smithi100 ceph-mon[39462]: 5.12 scrub starts 2024-09-17T07:08:28.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:28 smithi100 ceph-mon[39462]: 5.12 scrub ok 2024-09-17T07:08:28.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:28 smithi100 ceph-mon[39462]: 6.c scrub starts 2024-09-17T07:08:28.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:28 smithi100 ceph-mon[39462]: 6.c scrub ok 2024-09-17T07:08:28.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:28 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:08:28.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:28 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:08:28.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:28 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:08:28.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:28 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:08:28.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:28 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:08:28.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:28.884 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:28.884 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:08:28.985 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:08:29.013 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:29.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:29.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:08:29.043 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 13 16 33079 33063 158.935 106 0.00562325 0.00628188 2024-09-17T07:08:29.115 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:08:29.154 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:29.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:29.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:08:29.256 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:08:29.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:29.289 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:29.289 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:08:29.390 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:08:29.416 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:29.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:29.417 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:08:29.518 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:08:29.552 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:29.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:29.553 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:08:29.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:29 smithi100 ceph-mon[29092]: 5.1b deep-scrub starts 2024-09-17T07:08:29.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:29 smithi100 ceph-mon[29092]: 5.1b deep-scrub ok 2024-09-17T07:08:29.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:29 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:08:29.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:29 smithi100 ceph-mon[29092]: osdmap e179: 8 total, 8 up, 8 in 2024-09-17T07:08:29.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:29 smithi100 ceph-mon[29092]: 4.5 scrub starts 2024-09-17T07:08:29.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:29 smithi100 ceph-mon[29092]: 4.5 scrub ok 2024-09-17T07:08:29.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:29 smithi100 ceph-mon[29092]: 4.19 scrub starts 2024-09-17T07:08:29.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:29 smithi100 ceph-mon[29092]: 4.19 scrub ok 2024-09-17T07:08:29.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:29 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/4286102506' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:08:29.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:29 smithi100 ceph-mon[29092]: pgmap v275: 148 pgs: 148 active+clean; 2.0 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 188 MiB/s wr, 3.01k op/s 2024-09-17T07:08:29.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:29 smithi100 ceph-mon[39462]: 5.1b deep-scrub starts 2024-09-17T07:08:29.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:29 smithi100 ceph-mon[39462]: 5.1b deep-scrub ok 2024-09-17T07:08:29.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:29 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:08:29.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:29 smithi100 ceph-mon[39462]: osdmap e179: 8 total, 8 up, 8 in 2024-09-17T07:08:29.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:29 smithi100 ceph-mon[39462]: 4.5 scrub starts 2024-09-17T07:08:29.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:29 smithi100 ceph-mon[39462]: 4.5 scrub ok 2024-09-17T07:08:29.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:29 smithi100 ceph-mon[39462]: 4.19 scrub starts 2024-09-17T07:08:29.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:29 smithi100 ceph-mon[39462]: 4.19 scrub ok 2024-09-17T07:08:29.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:29 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/4286102506' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:08:29.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:29 smithi100 ceph-mon[39462]: pgmap v275: 148 pgs: 148 active+clean; 2.0 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 188 MiB/s wr, 3.01k op/s 2024-09-17T07:08:29.654 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:08:29.683 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:29.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:29.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:08:29.785 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:08:29.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:29 smithi135 ceph-mon[33238]: 5.1b deep-scrub starts 2024-09-17T07:08:29.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:29 smithi135 ceph-mon[33238]: 5.1b deep-scrub ok 2024-09-17T07:08:29.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:29 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:08:29.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:29 smithi135 ceph-mon[33238]: osdmap e179: 8 total, 8 up, 8 in 2024-09-17T07:08:29.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:29 smithi135 ceph-mon[33238]: 4.5 scrub starts 2024-09-17T07:08:29.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:29 smithi135 ceph-mon[33238]: 4.5 scrub ok 2024-09-17T07:08:29.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:29 smithi135 ceph-mon[33238]: 4.19 scrub starts 2024-09-17T07:08:29.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:29 smithi135 ceph-mon[33238]: 4.19 scrub ok 2024-09-17T07:08:29.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:29 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/4286102506' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:08:29.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:29 smithi135 ceph-mon[33238]: pgmap v275: 148 pgs: 148 active+clean; 2.0 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 188 MiB/s wr, 3.01k op/s 2024-09-17T07:08:29.814 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:29.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:29.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:08:29.917 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:08:29.943 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:29.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:29.944 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:08:30.043 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 14 16 34714 34698 154.881 102.188 0.00859817 0.00645128 2024-09-17T07:08:30.045 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:08:30.075 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:30.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:30.076 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:08:30.129 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:08:30.129 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1: {} 2024-09-17T07:08:30.129 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:08:30.130 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:08:30.130 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:08:30.130 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:08:30.130 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:08:30.130 INFO:teuthology.orchestra.run.smithi100.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:08:30.130 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:08:30.130 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:08:30.130 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:08:30.130 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:08:30.130 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:08:30.131 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:08:30.131 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:08:30.131 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:08:30.177 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:08:30.217 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:30.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:30.219 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:08:30.319 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:08:30.352 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:30.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:30.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:08:30.454 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:08:30.492 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:30.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:30.493 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:08:30.594 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:08:30.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:30.629 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:30.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:08:30.730 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:08:30.760 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:30.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:30.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:08:30.862 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:08:30.898 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:30.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:30.899 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:08:31.000 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:08:31.041 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:31.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:31.042 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:08:31.043 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 15 15 36820 36805 153.334 131.688 0.00727881 0.00651681 2024-09-17T07:08:31.086 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-17T07:08:31.143 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:08:31.175 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:31.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:31.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:08:31.279 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:08:31.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:31.322 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:31.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:08:31.322 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:31 smithi100 ceph-mon[39462]: 3.0 scrub starts 2024-09-17T07:08:31.322 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:31 smithi100 ceph-mon[39462]: 3.0 scrub ok 2024-09-17T07:08:31.322 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:31 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:08:31.322 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:31 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:08:31.323 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:31 smithi100 ceph-mon[39462]: osdmap e180: 8 total, 8 up, 8 in 2024-09-17T07:08:31.323 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:31 smithi100 ceph-mon[39462]: 4.b scrub starts 2024-09-17T07:08:31.323 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:31 smithi100 ceph-mon[39462]: 4.b scrub ok 2024-09-17T07:08:31.323 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:31 smithi100 ceph-mon[39462]: 6.e scrub starts 2024-09-17T07:08:31.323 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:31 smithi100 ceph-mon[39462]: 6.e scrub ok 2024-09-17T07:08:31.323 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:31 smithi100 ceph-mon[29092]: 3.0 scrub starts 2024-09-17T07:08:31.323 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:31 smithi100 ceph-mon[29092]: 3.0 scrub ok 2024-09-17T07:08:31.324 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:31 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:08:31.324 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:31 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:08:31.324 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:31 smithi100 ceph-mon[29092]: osdmap e180: 8 total, 8 up, 8 in 2024-09-17T07:08:31.324 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:31 smithi100 ceph-mon[29092]: 4.b scrub starts 2024-09-17T07:08:31.324 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:31 smithi100 ceph-mon[29092]: 4.b scrub ok 2024-09-17T07:08:31.324 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:31 smithi100 ceph-mon[29092]: 6.e scrub starts 2024-09-17T07:08:31.324 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:31 smithi100 ceph-mon[29092]: 6.e scrub ok 2024-09-17T07:08:31.423 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:08:31.451 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:31.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:31.453 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:08:31.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:31 smithi135 ceph-mon[33238]: 3.0 scrub starts 2024-09-17T07:08:31.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:31 smithi135 ceph-mon[33238]: 3.0 scrub ok 2024-09-17T07:08:31.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:31 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:08:31.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:31 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:08:31.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:31 smithi135 ceph-mon[33238]: osdmap e180: 8 total, 8 up, 8 in 2024-09-17T07:08:31.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:31 smithi135 ceph-mon[33238]: 4.b scrub starts 2024-09-17T07:08:31.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:31 smithi135 ceph-mon[33238]: 4.b scrub ok 2024-09-17T07:08:31.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:31 smithi135 ceph-mon[33238]: 6.e scrub starts 2024-09-17T07:08:31.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:31 smithi135 ceph-mon[33238]: 6.e scrub ok 2024-09-17T07:08:31.554 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:08:31.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:31.600 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:31.601 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:08:31.701 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:08:31.749 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:31.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:31.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:08:31.851 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:08:31.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:31.889 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:31.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:08:31.990 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:08:32.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:32.023 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:32.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:08:32.044 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 16 16 38641 38625 150.86 113.75 0.00362892 0.00662408 2024-09-17T07:08:32.124 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:08:32.179 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:32.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:32.180 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:08:32.281 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:08:32.322 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:32.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:32.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:08:32.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:32 smithi100 ceph-mon[29092]: 4.0 scrub starts 2024-09-17T07:08:32.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:32 smithi100 ceph-mon[29092]: 4.0 scrub ok 2024-09-17T07:08:32.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:32 smithi100 ceph-mon[29092]: pgmap v277: 148 pgs: 148 active+clean; 2.0 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 850 B/s rd, 106 MiB/s wr, 1.70k op/s 2024-09-17T07:08:32.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:32 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:08:32.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:32 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:08:32.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:32 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:08:32.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:32 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:08:32.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:32 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:08:32.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:32 smithi100 ceph-mon[29092]: 5.d scrub starts 2024-09-17T07:08:32.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:32 smithi100 ceph-mon[29092]: 5.d scrub ok 2024-09-17T07:08:32.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:32 smithi100 ceph-mon[39462]: 4.0 scrub starts 2024-09-17T07:08:32.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:32 smithi100 ceph-mon[39462]: 4.0 scrub ok 2024-09-17T07:08:32.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:32 smithi100 ceph-mon[39462]: pgmap v277: 148 pgs: 148 active+clean; 2.0 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 850 B/s rd, 106 MiB/s wr, 1.70k op/s 2024-09-17T07:08:32.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:32 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:08:32.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:32 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:08:32.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:32 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:08:32.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:32 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:08:32.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:32 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:08:32.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:32 smithi100 ceph-mon[39462]: 5.d scrub starts 2024-09-17T07:08:32.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:32 smithi100 ceph-mon[39462]: 5.d scrub ok 2024-09-17T07:08:32.424 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:08:32.459 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:32.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:32.460 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:08:32.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:32 smithi135 ceph-mon[33238]: 4.0 scrub starts 2024-09-17T07:08:32.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:32 smithi135 ceph-mon[33238]: 4.0 scrub ok 2024-09-17T07:08:32.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:32 smithi135 ceph-mon[33238]: pgmap v277: 148 pgs: 148 active+clean; 2.0 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 850 B/s rd, 106 MiB/s wr, 1.70k op/s 2024-09-17T07:08:32.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:32 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:08:32.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:32 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:08:32.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:32 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:08:32.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:32 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:08:32.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:32 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:08:32.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:32 smithi135 ceph-mon[33238]: 5.d scrub starts 2024-09-17T07:08:32.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:32 smithi135 ceph-mon[33238]: 5.d scrub ok 2024-09-17T07:08:32.561 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:08:32.597 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:32.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:32.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:08:32.662 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:08:32.698 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:08:32.708 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:08:32.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:32.743 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:32.743 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:08:32.844 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:08:32.878 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:32.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:32.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:08:32.980 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:08:33.027 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:33.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:33.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:08:33.044 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 17 16 40620 40604 149.261 123.688 0.00701406 0.00669473 2024-09-17T07:08:33.129 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:08:33.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:33.159 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:33.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:08:33.261 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:08:33.296 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:33.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:33.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:08:33.340 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:33 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:08:33.340 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:33 smithi100 ceph-mon[29092]: 5.10 scrub starts 2024-09-17T07:08:33.340 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:33 smithi100 ceph-mon[29092]: 5.10 scrub ok 2024-09-17T07:08:33.340 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:33 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:08:33.340 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:33 smithi100 ceph-mon[39462]: 5.10 scrub starts 2024-09-17T07:08:33.341 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:33 smithi100 ceph-mon[39462]: 5.10 scrub ok 2024-09-17T07:08:33.398 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:08:33.434 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:33.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:33.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:08:33.536 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:08:33.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:33 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:08:33.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:33 smithi135 ceph-mon[33238]: 5.10 scrub starts 2024-09-17T07:08:33.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:33 smithi135 ceph-mon[33238]: 5.10 scrub ok 2024-09-17T07:08:33.569 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:33.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:33.570 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:08:33.671 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:08:33.702 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:33.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:33.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:08:33.804 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:08:33.831 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:33.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:33.832 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:08:33.933 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:08:33.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:33.958 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:33.958 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:08:34.044 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 18 16 42558 42542 147.697 121.125 0.0195905 0.00676519 2024-09-17T07:08:34.058 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:08:34.091 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:34.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:34.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:08:34.190 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:08:34.193 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:08:34.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:34.221 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:34.221 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:08:34.310 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:34 smithi100 ceph-mon[29092]: pgmap v278: 148 pgs: 148 active+clean; 2.2 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 108 MiB/s wr, 1.73k op/s 2024-09-17T07:08:34.310 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:34 smithi100 ceph-mon[29092]: 4.17 scrub starts 2024-09-17T07:08:34.310 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:34 smithi100 ceph-mon[29092]: 4.17 scrub ok 2024-09-17T07:08:34.310 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:34 smithi100 ceph-mon[29092]: 6.15 scrub starts 2024-09-17T07:08:34.311 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:34 smithi100 ceph-mon[29092]: 6.15 scrub ok 2024-09-17T07:08:34.311 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:34 smithi100 ceph-mon[39462]: pgmap v278: 148 pgs: 148 active+clean; 2.2 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 108 MiB/s wr, 1.73k op/s 2024-09-17T07:08:34.311 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:34 smithi100 ceph-mon[39462]: 4.17 scrub starts 2024-09-17T07:08:34.311 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:34 smithi100 ceph-mon[39462]: 4.17 scrub ok 2024-09-17T07:08:34.311 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:34 smithi100 ceph-mon[39462]: 6.15 scrub starts 2024-09-17T07:08:34.311 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:34 smithi100 ceph-mon[39462]: 6.15 scrub ok 2024-09-17T07:08:34.322 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:08:34.370 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:34.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:34.372 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:08:34.472 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:08:34.503 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:34.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:34.504 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:08:34.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:34 smithi135 ceph-mon[33238]: pgmap v278: 148 pgs: 148 active+clean; 2.2 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 108 MiB/s wr, 1.73k op/s 2024-09-17T07:08:34.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:34 smithi135 ceph-mon[33238]: 4.17 scrub starts 2024-09-17T07:08:34.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:34 smithi135 ceph-mon[33238]: 4.17 scrub ok 2024-09-17T07:08:34.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:34 smithi135 ceph-mon[33238]: 6.15 scrub starts 2024-09-17T07:08:34.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:34 smithi135 ceph-mon[33238]: 6.15 scrub ok 2024-09-17T07:08:34.604 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:08:34.637 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:34.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:34.638 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:08:34.739 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:08:34.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:34.790 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:34.790 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:08:34.891 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:08:34.934 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:34.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:34.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:08:35.036 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:08:35.044 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 19 16 44321 44305 145.723 110.188 0.00557151 0.00685725 2024-09-17T07:08:35.071 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:35.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:35.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:08:35.117 INFO:teuthology.orchestra.run.smithi100.stderr:noscrub is set 2024-09-17T07:08:35.173 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:08:35.209 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:35.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:35.210 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:08:35.255 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:35 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/374876015' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:08:35.255 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:35 smithi100 ceph-mon[29092]: 4.9 scrub starts 2024-09-17T07:08:35.255 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:35 smithi100 ceph-mon[29092]: 4.9 scrub ok 2024-09-17T07:08:35.256 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:35 smithi100 ceph-mon[29092]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:08:35.256 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:35 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/374876015' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T07:08:35.256 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:35 smithi100 ceph-mon[29092]: osdmap e181: 8 total, 8 up, 8 in 2024-09-17T07:08:35.256 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:35 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/374876015' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:08:35.256 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:35 smithi100 ceph-mon[39462]: 4.9 scrub starts 2024-09-17T07:08:35.256 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:35 smithi100 ceph-mon[39462]: 4.9 scrub ok 2024-09-17T07:08:35.256 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:35 smithi100 ceph-mon[39462]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:08:35.256 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:35 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/374876015' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T07:08:35.256 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:35 smithi100 ceph-mon[39462]: osdmap e181: 8 total, 8 up, 8 in 2024-09-17T07:08:35.311 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:08:35.343 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:35.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:35.344 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:08:35.444 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:08:35.477 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:35.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:35.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:08:35.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:35 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/374876015' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:08:35.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:35 smithi135 ceph-mon[33238]: 4.9 scrub starts 2024-09-17T07:08:35.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:35 smithi135 ceph-mon[33238]: 4.9 scrub ok 2024-09-17T07:08:35.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:35 smithi135 ceph-mon[33238]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:08:35.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:35 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/374876015' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T07:08:35.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:35 smithi135 ceph-mon[33238]: osdmap e181: 8 total, 8 up, 8 in 2024-09-17T07:08:35.578 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:08:35.608 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:35.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:35.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:08:35.710 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:08:35.742 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:35.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:35.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:08:35.844 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:08:35.873 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:35.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:35.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:08:35.975 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:08:36.017 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T07:08:36.018 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-17T07:08:36.019 INFO:teuthology.orchestra.run.smithi100.stdout: "epoch": 181, 2024-09-17T07:08:36.019 INFO:teuthology.orchestra.run.smithi100.stdout: "fsid": "e848267c-74c1-11ef-bceb-c7b262605968", 2024-09-17T07:08:36.019 INFO:teuthology.orchestra.run.smithi100.stdout: "created": "2024-09-17T06:57:05.010532+0000", 2024-09-17T07:08:36.019 INFO:teuthology.orchestra.run.smithi100.stdout: "modified": "2024-09-17T07:08:35.102587+0000", 2024-09-17T07:08:36.019 INFO:teuthology.orchestra.run.smithi100.stdout: "last_up_change": "2024-09-17T07:01:59.917118+0000", 2024-09-17T07:08:36.019 INFO:teuthology.orchestra.run.smithi100.stdout: "last_in_change": "2024-09-17T07:01:42.942454+0000", 2024-09-17T07:08:36.019 INFO:teuthology.orchestra.run.smithi100.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-17T07:08:36.019 INFO:teuthology.orchestra.run.smithi100.stdout: "flags_num": 5801984, 2024-09-17T07:08:36.019 INFO:teuthology.orchestra.run.smithi100.stdout: "flags_set": [ 2024-09-17T07:08:36.019 INFO:teuthology.orchestra.run.smithi100.stdout: "noscrub", 2024-09-17T07:08:36.019 INFO:teuthology.orchestra.run.smithi100.stdout: "pglog_hardlimit", 2024-09-17T07:08:36.019 INFO:teuthology.orchestra.run.smithi100.stdout: "purged_snapdirs", 2024-09-17T07:08:36.020 INFO:teuthology.orchestra.run.smithi100.stdout: "recovery_deletes", 2024-09-17T07:08:36.020 INFO:teuthology.orchestra.run.smithi100.stdout: "sortbitwise" 2024-09-17T07:08:36.020 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-17T07:08:36.020 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_version": 18, 2024-09-17T07:08:36.020 INFO:teuthology.orchestra.run.smithi100.stdout: "full_ratio": 0.94999998807907104, 2024-09-17T07:08:36.020 INFO:teuthology.orchestra.run.smithi100.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-17T07:08:36.020 INFO:teuthology.orchestra.run.smithi100.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-17T07:08:36.020 INFO:teuthology.orchestra.run.smithi100.stdout: "cluster_snapshot": "", 2024-09-17T07:08:36.020 INFO:teuthology.orchestra.run.smithi100.stdout: "pool_max": 8, 2024-09-17T07:08:36.020 INFO:teuthology.orchestra.run.smithi100.stdout: "max_osd": 8, 2024-09-17T07:08:36.020 INFO:teuthology.orchestra.run.smithi100.stdout: "require_min_compat_client": "luminous", 2024-09-17T07:08:36.020 INFO:teuthology.orchestra.run.smithi100.stdout: "min_compat_client": "luminous", 2024-09-17T07:08:36.021 INFO:teuthology.orchestra.run.smithi100.stdout: "require_osd_release": "reef", 2024-09-17T07:08:36.021 INFO:teuthology.orchestra.run.smithi100.stdout: "allow_crimson": false, 2024-09-17T07:08:36.021 INFO:teuthology.orchestra.run.smithi100.stdout: "pools": [ 2024-09-17T07:08:36.021 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:36.021 INFO:teuthology.orchestra.run.smithi100.stdout: "pool": 1, 2024-09-17T07:08:36.021 INFO:teuthology.orchestra.run.smithi100.stdout: "pool_name": ".mgr", 2024-09-17T07:08:36.021 INFO:teuthology.orchestra.run.smithi100.stdout: "create_time": "2024-09-17T07:00:08.816136+0000", 2024-09-17T07:08:36.021 INFO:teuthology.orchestra.run.smithi100.stdout: "flags": 1, 2024-09-17T07:08:36.021 INFO:teuthology.orchestra.run.smithi100.stdout: "flags_names": "hashpspool", 2024-09-17T07:08:36.021 INFO:teuthology.orchestra.run.smithi100.stdout: "type": 1, 2024-09-17T07:08:36.021 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 3, 2024-09-17T07:08:36.021 INFO:teuthology.orchestra.run.smithi100.stdout: "min_size": 2, 2024-09-17T07:08:36.022 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_rule": 0, 2024-09-17T07:08:36.022 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_count": 0, 2024-09-17T07:08:36.022 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_target": 0, 2024-09-17T07:08:36.022 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T07:08:36.022 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T07:08:36.022 INFO:teuthology.orchestra.run.smithi100.stdout: "object_hash": 2, 2024-09-17T07:08:36.022 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_autoscale_mode": "off", 2024-09-17T07:08:36.022 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num": 1, 2024-09-17T07:08:36.022 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_placement_num": 1, 2024-09-17T07:08:36.022 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_placement_num_target": 1, 2024-09-17T07:08:36.022 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num_target": 1, 2024-09-17T07:08:36.022 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num_pending": 1, 2024-09-17T07:08:36.022 INFO:teuthology.orchestra.run.smithi100.stdout: "last_pg_merge_meta": { 2024-09-17T07:08:36.023 INFO:teuthology.orchestra.run.smithi100.stdout: "source_pgid": "0.0", 2024-09-17T07:08:36.023 INFO:teuthology.orchestra.run.smithi100.stdout: "ready_epoch": 0, 2024-09-17T07:08:36.023 INFO:teuthology.orchestra.run.smithi100.stdout: "last_epoch_started": 0, 2024-09-17T07:08:36.023 INFO:teuthology.orchestra.run.smithi100.stdout: "last_epoch_clean": 0, 2024-09-17T07:08:36.023 INFO:teuthology.orchestra.run.smithi100.stdout: "source_version": "0'0", 2024-09-17T07:08:36.023 INFO:teuthology.orchestra.run.smithi100.stdout: "target_version": "0'0" 2024-09-17T07:08:36.023 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:36.023 INFO:teuthology.orchestra.run.smithi100.stdout: "last_change": "22", 2024-09-17T07:08:36.023 INFO:teuthology.orchestra.run.smithi100.stdout: "last_force_op_resend": "0", 2024-09-17T07:08:36.023 INFO:teuthology.orchestra.run.smithi100.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-17T07:08:36.023 INFO:teuthology.orchestra.run.smithi100.stdout: "last_force_op_resend_preluminous": "0", 2024-09-17T07:08:36.023 INFO:teuthology.orchestra.run.smithi100.stdout: "auid": 0, 2024-09-17T07:08:36.024 INFO:teuthology.orchestra.run.smithi100.stdout: "snap_mode": "selfmanaged", 2024-09-17T07:08:36.024 INFO:teuthology.orchestra.run.smithi100.stdout: "snap_seq": 0, 2024-09-17T07:08:36.024 INFO:teuthology.orchestra.run.smithi100.stdout: "snap_epoch": 0, 2024-09-17T07:08:36.024 INFO:teuthology.orchestra.run.smithi100.stdout: "pool_snaps": [], 2024-09-17T07:08:36.024 INFO:teuthology.orchestra.run.smithi100.stdout: "removed_snaps": "[]", 2024-09-17T07:08:36.024 INFO:teuthology.orchestra.run.smithi100.stdout: "quota_max_bytes": 0, 2024-09-17T07:08:36.024 INFO:teuthology.orchestra.run.smithi100.stdout: "quota_max_objects": 0, 2024-09-17T07:08:36.024 INFO:teuthology.orchestra.run.smithi100.stdout: "tiers": [], 2024-09-17T07:08:36.024 INFO:teuthology.orchestra.run.smithi100.stdout: "tier_of": -1, 2024-09-17T07:08:36.024 INFO:teuthology.orchestra.run.smithi100.stdout: "read_tier": -1, 2024-09-17T07:08:36.024 INFO:teuthology.orchestra.run.smithi100.stdout: "write_tier": -1, 2024-09-17T07:08:36.024 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_mode": "none", 2024-09-17T07:08:36.024 INFO:teuthology.orchestra.run.smithi100.stdout: "target_max_bytes": 0, 2024-09-17T07:08:36.025 INFO:teuthology.orchestra.run.smithi100.stdout: "target_max_objects": 0, 2024-09-17T07:08:36.025 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T07:08:36.025 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T07:08:36.025 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T07:08:36.025 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_min_flush_age": 0, 2024-09-17T07:08:36.025 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_min_evict_age": 0, 2024-09-17T07:08:36.025 INFO:teuthology.orchestra.run.smithi100.stdout: "erasure_code_profile": "", 2024-09-17T07:08:36.025 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_params": { 2024-09-17T07:08:36.025 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "none" 2024-09-17T07:08:36.025 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:36.025 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_period": 0, 2024-09-17T07:08:36.025 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_count": 0, 2024-09-17T07:08:36.026 INFO:teuthology.orchestra.run.smithi100.stdout: "use_gmt_hitset": true, 2024-09-17T07:08:36.026 INFO:teuthology.orchestra.run.smithi100.stdout: "min_read_recency_for_promote": 0, 2024-09-17T07:08:36.026 INFO:teuthology.orchestra.run.smithi100.stdout: "min_write_recency_for_promote": 0, 2024-09-17T07:08:36.026 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T07:08:36.026 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_search_last_n": 0, 2024-09-17T07:08:36.026 INFO:teuthology.orchestra.run.smithi100.stdout: "grade_table": [], 2024-09-17T07:08:36.026 INFO:teuthology.orchestra.run.smithi100.stdout: "stripe_width": 0, 2024-09-17T07:08:36.026 INFO:teuthology.orchestra.run.smithi100.stdout: "expected_num_objects": 0, 2024-09-17T07:08:36.026 INFO:teuthology.orchestra.run.smithi100.stdout: "fast_read": false, 2024-09-17T07:08:36.026 INFO:teuthology.orchestra.run.smithi100.stdout: "options": { 2024-09-17T07:08:36.026 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num_max": 32, 2024-09-17T07:08:36.026 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num_min": 1 2024-09-17T07:08:36.027 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:36.027 INFO:teuthology.orchestra.run.smithi100.stdout: "application_metadata": { 2024-09-17T07:08:36.027 INFO:teuthology.orchestra.run.smithi100.stdout: "mgr": {} 2024-09-17T07:08:36.027 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:36.027 INFO:teuthology.orchestra.run.smithi100.stdout: "read_balance": { 2024-09-17T07:08:36.027 INFO:teuthology.orchestra.run.smithi100.stdout: "score_acting": 7.8899998664855957, 2024-09-17T07:08:36.027 INFO:teuthology.orchestra.run.smithi100.stdout: "score_stable": 7.8899998664855957, 2024-09-17T07:08:36.027 INFO:teuthology.orchestra.run.smithi100.stdout: "optimal_score": 0.37999999523162842, 2024-09-17T07:08:36.027 INFO:teuthology.orchestra.run.smithi100.stdout: "raw_score_acting": 3, 2024-09-17T07:08:36.027 INFO:teuthology.orchestra.run.smithi100.stdout: "raw_score_stable": 3, 2024-09-17T07:08:36.027 INFO:teuthology.orchestra.run.smithi100.stdout: "primary_affinity_weighted": 0.89598274230957031, 2024-09-17T07:08:36.027 INFO:teuthology.orchestra.run.smithi100.stdout: "average_primary_affinity": 0.89999997615814209, 2024-09-17T07:08:36.027 INFO:teuthology.orchestra.run.smithi100.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-09-17T07:08:36.028 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:36.028 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:36.028 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:36.028 INFO:teuthology.orchestra.run.smithi100.stdout: "pool": 2, 2024-09-17T07:08:36.028 INFO:teuthology.orchestra.run.smithi100.stdout: "pool_name": ".rgw.root", 2024-09-17T07:08:36.028 INFO:teuthology.orchestra.run.smithi100.stdout: "create_time": "2024-09-17T07:02:10.759826+0000", 2024-09-17T07:08:36.028 INFO:teuthology.orchestra.run.smithi100.stdout: "flags": 1, 2024-09-17T07:08:36.028 INFO:teuthology.orchestra.run.smithi100.stdout: "flags_names": "hashpspool", 2024-09-17T07:08:36.028 INFO:teuthology.orchestra.run.smithi100.stdout: "type": 1, 2024-09-17T07:08:36.028 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 3, 2024-09-17T07:08:36.028 INFO:teuthology.orchestra.run.smithi100.stdout: "min_size": 2, 2024-09-17T07:08:36.028 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_rule": 0, 2024-09-17T07:08:36.029 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_count": 0, 2024-09-17T07:08:36.029 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_target": 0, 2024-09-17T07:08:36.029 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T07:08:36.029 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T07:08:36.029 INFO:teuthology.orchestra.run.smithi100.stdout: "object_hash": 2, 2024-09-17T07:08:36.029 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_autoscale_mode": "off", 2024-09-17T07:08:36.029 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num": 32, 2024-09-17T07:08:36.029 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_placement_num": 32, 2024-09-17T07:08:36.029 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_placement_num_target": 32, 2024-09-17T07:08:36.029 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num_target": 32, 2024-09-17T07:08:36.029 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num_pending": 32, 2024-09-17T07:08:36.029 INFO:teuthology.orchestra.run.smithi100.stdout: "last_pg_merge_meta": { 2024-09-17T07:08:36.030 INFO:teuthology.orchestra.run.smithi100.stdout: "source_pgid": "0.0", 2024-09-17T07:08:36.030 INFO:teuthology.orchestra.run.smithi100.stdout: "ready_epoch": 0, 2024-09-17T07:08:36.030 INFO:teuthology.orchestra.run.smithi100.stdout: "last_epoch_started": 0, 2024-09-17T07:08:36.030 INFO:teuthology.orchestra.run.smithi100.stdout: "last_epoch_clean": 0, 2024-09-17T07:08:36.030 INFO:teuthology.orchestra.run.smithi100.stdout: "source_version": "0'0", 2024-09-17T07:08:36.030 INFO:teuthology.orchestra.run.smithi100.stdout: "target_version": "0'0" 2024-09-17T07:08:36.030 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:36.030 INFO:teuthology.orchestra.run.smithi100.stdout: "last_change": "69", 2024-09-17T07:08:36.030 INFO:teuthology.orchestra.run.smithi100.stdout: "last_force_op_resend": "0", 2024-09-17T07:08:36.030 INFO:teuthology.orchestra.run.smithi100.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-17T07:08:36.030 INFO:teuthology.orchestra.run.smithi100.stdout: "last_force_op_resend_preluminous": "0", 2024-09-17T07:08:36.030 INFO:teuthology.orchestra.run.smithi100.stdout: "auid": 0, 2024-09-17T07:08:36.030 INFO:teuthology.orchestra.run.smithi100.stdout: "snap_mode": "selfmanaged", 2024-09-17T07:08:36.030 INFO:teuthology.orchestra.run.smithi100.stdout: "snap_seq": 0, 2024-09-17T07:08:36.031 INFO:teuthology.orchestra.run.smithi100.stdout: "snap_epoch": 0, 2024-09-17T07:08:36.031 INFO:teuthology.orchestra.run.smithi100.stdout: "pool_snaps": [], 2024-09-17T07:08:36.031 INFO:teuthology.orchestra.run.smithi100.stdout: "removed_snaps": "[]", 2024-09-17T07:08:36.031 INFO:teuthology.orchestra.run.smithi100.stdout: "quota_max_bytes": 0, 2024-09-17T07:08:36.031 INFO:teuthology.orchestra.run.smithi100.stdout: "quota_max_objects": 0, 2024-09-17T07:08:36.031 INFO:teuthology.orchestra.run.smithi100.stdout: "tiers": [], 2024-09-17T07:08:36.031 INFO:teuthology.orchestra.run.smithi100.stdout: "tier_of": -1, 2024-09-17T07:08:36.031 INFO:teuthology.orchestra.run.smithi100.stdout: "read_tier": -1, 2024-09-17T07:08:36.031 INFO:teuthology.orchestra.run.smithi100.stdout: "write_tier": -1, 2024-09-17T07:08:36.031 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_mode": "none", 2024-09-17T07:08:36.031 INFO:teuthology.orchestra.run.smithi100.stdout: "target_max_bytes": 0, 2024-09-17T07:08:36.031 INFO:teuthology.orchestra.run.smithi100.stdout: "target_max_objects": 0, 2024-09-17T07:08:36.031 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T07:08:36.032 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T07:08:36.032 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T07:08:36.032 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_min_flush_age": 0, 2024-09-17T07:08:36.032 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_min_evict_age": 0, 2024-09-17T07:08:36.032 INFO:teuthology.orchestra.run.smithi100.stdout: "erasure_code_profile": "", 2024-09-17T07:08:36.032 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_params": { 2024-09-17T07:08:36.032 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "none" 2024-09-17T07:08:36.032 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:36.032 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_period": 0, 2024-09-17T07:08:36.032 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_count": 0, 2024-09-17T07:08:36.032 INFO:teuthology.orchestra.run.smithi100.stdout: "use_gmt_hitset": true, 2024-09-17T07:08:36.032 INFO:teuthology.orchestra.run.smithi100.stdout: "min_read_recency_for_promote": 0, 2024-09-17T07:08:36.032 INFO:teuthology.orchestra.run.smithi100.stdout: "min_write_recency_for_promote": 0, 2024-09-17T07:08:36.033 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T07:08:36.033 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_search_last_n": 0, 2024-09-17T07:08:36.033 INFO:teuthology.orchestra.run.smithi100.stdout: "grade_table": [], 2024-09-17T07:08:36.033 INFO:teuthology.orchestra.run.smithi100.stdout: "stripe_width": 0, 2024-09-17T07:08:36.033 INFO:teuthology.orchestra.run.smithi100.stdout: "expected_num_objects": 0, 2024-09-17T07:08:36.033 INFO:teuthology.orchestra.run.smithi100.stdout: "fast_read": false, 2024-09-17T07:08:36.033 INFO:teuthology.orchestra.run.smithi100.stdout: "options": {}, 2024-09-17T07:08:36.033 INFO:teuthology.orchestra.run.smithi100.stdout: "application_metadata": { 2024-09-17T07:08:36.033 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw": {} 2024-09-17T07:08:36.033 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:36.033 INFO:teuthology.orchestra.run.smithi100.stdout: "read_balance": { 2024-09-17T07:08:36.033 INFO:teuthology.orchestra.run.smithi100.stdout: "score_acting": 1.5, 2024-09-17T07:08:36.033 INFO:teuthology.orchestra.run.smithi100.stdout: "score_stable": 1.5, 2024-09-17T07:08:36.034 INFO:teuthology.orchestra.run.smithi100.stdout: "optimal_score": 1, 2024-09-17T07:08:36.034 INFO:teuthology.orchestra.run.smithi100.stdout: "raw_score_acting": 1.5, 2024-09-17T07:08:36.034 INFO:teuthology.orchestra.run.smithi100.stdout: "raw_score_stable": 1.5, 2024-09-17T07:08:36.034 INFO:teuthology.orchestra.run.smithi100.stdout: "primary_affinity_weighted": 0.89598274230957031, 2024-09-17T07:08:36.034 INFO:teuthology.orchestra.run.smithi100.stdout: "average_primary_affinity": 0.89999997615814209, 2024-09-17T07:08:36.034 INFO:teuthology.orchestra.run.smithi100.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-09-17T07:08:36.034 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:36.034 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:36.034 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:36.034 INFO:teuthology.orchestra.run.smithi100.stdout: "pool": 3, 2024-09-17T07:08:36.034 INFO:teuthology.orchestra.run.smithi100.stdout: "pool_name": "datapool", 2024-09-17T07:08:36.034 INFO:teuthology.orchestra.run.smithi100.stdout: "create_time": "2024-09-17T07:02:10.991289+0000", 2024-09-17T07:08:36.034 INFO:teuthology.orchestra.run.smithi100.stdout: "flags": 8193, 2024-09-17T07:08:36.035 INFO:teuthology.orchestra.run.smithi100.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-17T07:08:36.035 INFO:teuthology.orchestra.run.smithi100.stdout: "type": 1, 2024-09-17T07:08:36.035 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 3, 2024-09-17T07:08:36.035 INFO:teuthology.orchestra.run.smithi100.stdout: "min_size": 2, 2024-09-17T07:08:36.035 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_rule": 0, 2024-09-17T07:08:36.035 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_count": 0, 2024-09-17T07:08:36.035 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_target": 0, 2024-09-17T07:08:36.035 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T07:08:36.035 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T07:08:36.035 INFO:teuthology.orchestra.run.smithi100.stdout: "object_hash": 2, 2024-09-17T07:08:36.035 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_autoscale_mode": "off", 2024-09-17T07:08:36.035 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num": 3, 2024-09-17T07:08:36.035 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_placement_num": 3, 2024-09-17T07:08:36.036 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_placement_num_target": 3, 2024-09-17T07:08:36.036 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num_target": 3, 2024-09-17T07:08:36.036 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num_pending": 3, 2024-09-17T07:08:36.036 INFO:teuthology.orchestra.run.smithi100.stdout: "last_pg_merge_meta": { 2024-09-17T07:08:36.036 INFO:teuthology.orchestra.run.smithi100.stdout: "source_pgid": "0.0", 2024-09-17T07:08:36.036 INFO:teuthology.orchestra.run.smithi100.stdout: "ready_epoch": 0, 2024-09-17T07:08:36.036 INFO:teuthology.orchestra.run.smithi100.stdout: "last_epoch_started": 0, 2024-09-17T07:08:36.036 INFO:teuthology.orchestra.run.smithi100.stdout: "last_epoch_clean": 0, 2024-09-17T07:08:36.036 INFO:teuthology.orchestra.run.smithi100.stdout: "source_version": "0'0", 2024-09-17T07:08:36.036 INFO:teuthology.orchestra.run.smithi100.stdout: "target_version": "0'0" 2024-09-17T07:08:36.036 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:36.036 INFO:teuthology.orchestra.run.smithi100.stdout: "last_change": "72", 2024-09-17T07:08:36.037 INFO:teuthology.orchestra.run.smithi100.stdout: "last_force_op_resend": "0", 2024-09-17T07:08:36.037 INFO:teuthology.orchestra.run.smithi100.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-17T07:08:36.037 INFO:teuthology.orchestra.run.smithi100.stdout: "last_force_op_resend_preluminous": "0", 2024-09-17T07:08:36.037 INFO:teuthology.orchestra.run.smithi100.stdout: "auid": 0, 2024-09-17T07:08:36.037 INFO:teuthology.orchestra.run.smithi100.stdout: "snap_mode": "selfmanaged", 2024-09-17T07:08:36.037 INFO:teuthology.orchestra.run.smithi100.stdout: "snap_seq": 3, 2024-09-17T07:08:36.037 INFO:teuthology.orchestra.run.smithi100.stdout: "snap_epoch": 72, 2024-09-17T07:08:36.037 INFO:teuthology.orchestra.run.smithi100.stdout: "pool_snaps": [], 2024-09-17T07:08:36.037 INFO:teuthology.orchestra.run.smithi100.stdout: "removed_snaps": "[]", 2024-09-17T07:08:36.037 INFO:teuthology.orchestra.run.smithi100.stdout: "quota_max_bytes": 0, 2024-09-17T07:08:36.037 INFO:teuthology.orchestra.run.smithi100.stdout: "quota_max_objects": 0, 2024-09-17T07:08:36.037 INFO:teuthology.orchestra.run.smithi100.stdout: "tiers": [], 2024-09-17T07:08:36.037 INFO:teuthology.orchestra.run.smithi100.stdout: "tier_of": -1, 2024-09-17T07:08:36.038 INFO:teuthology.orchestra.run.smithi100.stdout: "read_tier": -1, 2024-09-17T07:08:36.038 INFO:teuthology.orchestra.run.smithi100.stdout: "write_tier": -1, 2024-09-17T07:08:36.038 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_mode": "none", 2024-09-17T07:08:36.038 INFO:teuthology.orchestra.run.smithi100.stdout: "target_max_bytes": 0, 2024-09-17T07:08:36.038 INFO:teuthology.orchestra.run.smithi100.stdout: "target_max_objects": 0, 2024-09-17T07:08:36.038 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T07:08:36.038 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T07:08:36.038 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T07:08:36.038 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_min_flush_age": 0, 2024-09-17T07:08:36.038 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_min_evict_age": 0, 2024-09-17T07:08:36.038 INFO:teuthology.orchestra.run.smithi100.stdout: "erasure_code_profile": "", 2024-09-17T07:08:36.038 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_params": { 2024-09-17T07:08:36.038 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "none" 2024-09-17T07:08:36.039 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:36.039 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_period": 0, 2024-09-17T07:08:36.039 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_count": 0, 2024-09-17T07:08:36.039 INFO:teuthology.orchestra.run.smithi100.stdout: "use_gmt_hitset": true, 2024-09-17T07:08:36.039 INFO:teuthology.orchestra.run.smithi100.stdout: "min_read_recency_for_promote": 0, 2024-09-17T07:08:36.039 INFO:teuthology.orchestra.run.smithi100.stdout: "min_write_recency_for_promote": 0, 2024-09-17T07:08:36.039 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T07:08:36.039 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_search_last_n": 0, 2024-09-17T07:08:36.039 INFO:teuthology.orchestra.run.smithi100.stdout: "grade_table": [], 2024-09-17T07:08:36.039 INFO:teuthology.orchestra.run.smithi100.stdout: "stripe_width": 0, 2024-09-17T07:08:36.039 INFO:teuthology.orchestra.run.smithi100.stdout: "expected_num_objects": 0, 2024-09-17T07:08:36.039 INFO:teuthology.orchestra.run.smithi100.stdout: "fast_read": false, 2024-09-17T07:08:36.039 INFO:teuthology.orchestra.run.smithi100.stdout: "options": {}, 2024-09-17T07:08:36.040 INFO:teuthology.orchestra.run.smithi100.stdout: "application_metadata": { 2024-09-17T07:08:36.040 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd": {} 2024-09-17T07:08:36.040 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:36.040 INFO:teuthology.orchestra.run.smithi100.stdout: "read_balance": { 2024-09-17T07:08:36.040 INFO:teuthology.orchestra.run.smithi100.stdout: "score_acting": 2.6500000953674316, 2024-09-17T07:08:36.040 INFO:teuthology.orchestra.run.smithi100.stdout: "score_stable": 2.6500000953674316, 2024-09-17T07:08:36.040 INFO:teuthology.orchestra.run.smithi100.stdout: "optimal_score": 0.87999999523162842, 2024-09-17T07:08:36.040 INFO:teuthology.orchestra.run.smithi100.stdout: "raw_score_acting": 2.3299999237060547, 2024-09-17T07:08:36.040 INFO:teuthology.orchestra.run.smithi100.stdout: "raw_score_stable": 2.3299999237060547, 2024-09-17T07:08:36.040 INFO:teuthology.orchestra.run.smithi100.stdout: "primary_affinity_weighted": 0.89598274230957031, 2024-09-17T07:08:36.040 INFO:teuthology.orchestra.run.smithi100.stdout: "average_primary_affinity": 0.89999997615814209, 2024-09-17T07:08:36.040 INFO:teuthology.orchestra.run.smithi100.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-09-17T07:08:36.041 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:36.041 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:36.041 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:36.041 INFO:teuthology.orchestra.run.smithi100.stdout: "pool": 4, 2024-09-17T07:08:36.041 INFO:teuthology.orchestra.run.smithi100.stdout: "pool_name": "default.rgw.log", 2024-09-17T07:08:36.041 INFO:teuthology.orchestra.run.smithi100.stdout: "create_time": "2024-09-17T07:02:12.291462+0000", 2024-09-17T07:08:36.041 INFO:teuthology.orchestra.run.smithi100.stdout: "flags": 1, 2024-09-17T07:08:36.041 INFO:teuthology.orchestra.run.smithi100.stdout: "flags_names": "hashpspool", 2024-09-17T07:08:36.041 INFO:teuthology.orchestra.run.smithi100.stdout: "type": 1, 2024-09-17T07:08:36.041 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 3, 2024-09-17T07:08:36.041 INFO:teuthology.orchestra.run.smithi100.stdout: "min_size": 2, 2024-09-17T07:08:36.041 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_rule": 0, 2024-09-17T07:08:36.042 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_count": 0, 2024-09-17T07:08:36.042 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_target": 0, 2024-09-17T07:08:36.042 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T07:08:36.042 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T07:08:36.042 INFO:teuthology.orchestra.run.smithi100.stdout: "object_hash": 2, 2024-09-17T07:08:36.042 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_autoscale_mode": "off", 2024-09-17T07:08:36.042 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num": 32, 2024-09-17T07:08:36.042 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_placement_num": 32, 2024-09-17T07:08:36.042 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_placement_num_target": 32, 2024-09-17T07:08:36.042 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num_target": 32, 2024-09-17T07:08:36.042 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num_pending": 32, 2024-09-17T07:08:36.042 INFO:teuthology.orchestra.run.smithi100.stdout: "last_pg_merge_meta": { 2024-09-17T07:08:36.043 INFO:teuthology.orchestra.run.smithi100.stdout: "source_pgid": "0.0", 2024-09-17T07:08:36.043 INFO:teuthology.orchestra.run.smithi100.stdout: "ready_epoch": 0, 2024-09-17T07:08:36.043 INFO:teuthology.orchestra.run.smithi100.stdout: "last_epoch_started": 0, 2024-09-17T07:08:36.043 INFO:teuthology.orchestra.run.smithi100.stdout: "last_epoch_clean": 0, 2024-09-17T07:08:36.043 INFO:teuthology.orchestra.run.smithi100.stdout: "source_version": "0'0", 2024-09-17T07:08:36.043 INFO:teuthology.orchestra.run.smithi100.stdout: "target_version": "0'0" 2024-09-17T07:08:36.043 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:36.043 INFO:teuthology.orchestra.run.smithi100.stdout: "last_change": "71", 2024-09-17T07:08:36.043 INFO:teuthology.orchestra.run.smithi100.stdout: "last_force_op_resend": "0", 2024-09-17T07:08:36.043 INFO:teuthology.orchestra.run.smithi100.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-17T07:08:36.043 INFO:teuthology.orchestra.run.smithi100.stdout: "last_force_op_resend_preluminous": "0", 2024-09-17T07:08:36.043 INFO:teuthology.orchestra.run.smithi100.stdout: "auid": 0, 2024-09-17T07:08:36.043 INFO:teuthology.orchestra.run.smithi100.stdout: "snap_mode": "selfmanaged", 2024-09-17T07:08:36.044 INFO:teuthology.orchestra.run.smithi100.stdout: "snap_seq": 0, 2024-09-17T07:08:36.044 INFO:teuthology.orchestra.run.smithi100.stdout: "snap_epoch": 0, 2024-09-17T07:08:36.044 INFO:teuthology.orchestra.run.smithi100.stdout: "pool_snaps": [], 2024-09-17T07:08:36.044 INFO:teuthology.orchestra.run.smithi100.stdout: "removed_snaps": "[]", 2024-09-17T07:08:36.044 INFO:teuthology.orchestra.run.smithi100.stdout: "quota_max_bytes": 0, 2024-09-17T07:08:36.044 INFO:teuthology.orchestra.run.smithi100.stdout: "quota_max_objects": 0, 2024-09-17T07:08:36.044 INFO:teuthology.orchestra.run.smithi100.stdout: "tiers": [], 2024-09-17T07:08:36.044 INFO:teuthology.orchestra.run.smithi100.stdout: "tier_of": -1, 2024-09-17T07:08:36.044 INFO:teuthology.orchestra.run.smithi100.stdout: "read_tier": -1, 2024-09-17T07:08:36.044 INFO:teuthology.orchestra.run.smithi100.stdout: "write_tier": -1, 2024-09-17T07:08:36.044 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_mode": "none", 2024-09-17T07:08:36.044 INFO:teuthology.orchestra.run.smithi100.stdout: "target_max_bytes": 0, 2024-09-17T07:08:36.045 INFO:teuthology.orchestra.run.smithi100.stdout: "target_max_objects": 0, 2024-09-17T07:08:36.045 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T07:08:36.045 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T07:08:36.045 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T07:08:36.045 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_min_flush_age": 0, 2024-09-17T07:08:36.045 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_min_evict_age": 0, 2024-09-17T07:08:36.045 INFO:teuthology.orchestra.run.smithi100.stdout: "erasure_code_profile": "", 2024-09-17T07:08:36.045 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_params": { 2024-09-17T07:08:36.045 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "none" 2024-09-17T07:08:36.045 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:36.045 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_period": 0, 2024-09-17T07:08:36.045 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_count": 0, 2024-09-17T07:08:36.046 INFO:teuthology.orchestra.run.smithi100.stdout: "use_gmt_hitset": true, 2024-09-17T07:08:36.046 INFO:teuthology.orchestra.run.smithi100.stdout: "min_read_recency_for_promote": 0, 2024-09-17T07:08:36.046 INFO:teuthology.orchestra.run.smithi100.stdout: "min_write_recency_for_promote": 0, 2024-09-17T07:08:36.046 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T07:08:36.046 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_search_last_n": 0, 2024-09-17T07:08:36.046 INFO:teuthology.orchestra.run.smithi100.stdout: "grade_table": [], 2024-09-17T07:08:36.046 INFO:teuthology.orchestra.run.smithi100.stdout: "stripe_width": 0, 2024-09-17T07:08:36.046 INFO:teuthology.orchestra.run.smithi100.stdout: "expected_num_objects": 0, 2024-09-17T07:08:36.046 INFO:teuthology.orchestra.run.smithi100.stdout: "fast_read": false, 2024-09-17T07:08:36.046 INFO:teuthology.orchestra.run.smithi100.stdout: "options": {}, 2024-09-17T07:08:36.046 INFO:teuthology.orchestra.run.smithi100.stdout: "application_metadata": { 2024-09-17T07:08:36.046 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw": {} 2024-09-17T07:08:36.046 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:36.047 INFO:teuthology.orchestra.run.smithi100.stdout: "read_balance": { 2024-09-17T07:08:36.047 INFO:teuthology.orchestra.run.smithi100.stdout: "score_acting": 2.25, 2024-09-17T07:08:36.047 INFO:teuthology.orchestra.run.smithi100.stdout: "score_stable": 2.25, 2024-09-17T07:08:36.047 INFO:teuthology.orchestra.run.smithi100.stdout: "optimal_score": 1, 2024-09-17T07:08:36.047 INFO:teuthology.orchestra.run.smithi100.stdout: "raw_score_acting": 2.25, 2024-09-17T07:08:36.047 INFO:teuthology.orchestra.run.smithi100.stdout: "raw_score_stable": 2.25, 2024-09-17T07:08:36.047 INFO:teuthology.orchestra.run.smithi100.stdout: "primary_affinity_weighted": 0.89598274230957031, 2024-09-17T07:08:36.047 INFO:teuthology.orchestra.run.smithi100.stdout: "average_primary_affinity": 0.89999997615814209, 2024-09-17T07:08:36.047 INFO:teuthology.orchestra.run.smithi100.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-09-17T07:08:36.047 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:36.047 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:36.047 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:36.047 INFO:teuthology.orchestra.run.smithi100.stdout: "pool": 5, 2024-09-17T07:08:36.048 INFO:teuthology.orchestra.run.smithi100.stdout: "pool_name": "default.rgw.control", 2024-09-17T07:08:36.048 INFO:teuthology.orchestra.run.smithi100.stdout: "create_time": "2024-09-17T07:02:14.253214+0000", 2024-09-17T07:08:36.048 INFO:teuthology.orchestra.run.smithi100.stdout: "flags": 1, 2024-09-17T07:08:36.048 INFO:teuthology.orchestra.run.smithi100.stdout: "flags_names": "hashpspool", 2024-09-17T07:08:36.048 INFO:teuthology.orchestra.run.smithi100.stdout: "type": 1, 2024-09-17T07:08:36.048 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 3, 2024-09-17T07:08:36.048 INFO:teuthology.orchestra.run.smithi100.stdout: "min_size": 2, 2024-09-17T07:08:36.048 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_rule": 0, 2024-09-17T07:08:36.048 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_count": 0, 2024-09-17T07:08:36.048 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_target": 0, 2024-09-17T07:08:36.048 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T07:08:36.048 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T07:08:36.048 INFO:teuthology.orchestra.run.smithi100.stdout: "object_hash": 2, 2024-09-17T07:08:36.049 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_autoscale_mode": "off", 2024-09-17T07:08:36.049 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num": 32, 2024-09-17T07:08:36.049 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_placement_num": 32, 2024-09-17T07:08:36.049 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_placement_num_target": 32, 2024-09-17T07:08:36.049 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num_target": 32, 2024-09-17T07:08:36.049 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num_pending": 32, 2024-09-17T07:08:36.049 INFO:teuthology.orchestra.run.smithi100.stdout: "last_pg_merge_meta": { 2024-09-17T07:08:36.049 INFO:teuthology.orchestra.run.smithi100.stdout: "source_pgid": "0.0", 2024-09-17T07:08:36.049 INFO:teuthology.orchestra.run.smithi100.stdout: "ready_epoch": 0, 2024-09-17T07:08:36.049 INFO:teuthology.orchestra.run.smithi100.stdout: "last_epoch_started": 0, 2024-09-17T07:08:36.049 INFO:teuthology.orchestra.run.smithi100.stdout: "last_epoch_clean": 0, 2024-09-17T07:08:36.049 INFO:teuthology.orchestra.run.smithi100.stdout: "source_version": "0'0", 2024-09-17T07:08:36.049 INFO:teuthology.orchestra.run.smithi100.stdout: "target_version": "0'0" 2024-09-17T07:08:36.050 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:36.050 INFO:teuthology.orchestra.run.smithi100.stdout: "last_change": "73", 2024-09-17T07:08:36.050 INFO:teuthology.orchestra.run.smithi100.stdout: "last_force_op_resend": "0", 2024-09-17T07:08:36.050 INFO:teuthology.orchestra.run.smithi100.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-17T07:08:36.050 INFO:teuthology.orchestra.run.smithi100.stdout: "last_force_op_resend_preluminous": "0", 2024-09-17T07:08:36.050 INFO:teuthology.orchestra.run.smithi100.stdout: "auid": 0, 2024-09-17T07:08:36.050 INFO:teuthology.orchestra.run.smithi100.stdout: "snap_mode": "selfmanaged", 2024-09-17T07:08:36.050 INFO:teuthology.orchestra.run.smithi100.stdout: "snap_seq": 0, 2024-09-17T07:08:36.050 INFO:teuthology.orchestra.run.smithi100.stdout: "snap_epoch": 0, 2024-09-17T07:08:36.050 INFO:teuthology.orchestra.run.smithi100.stdout: "pool_snaps": [], 2024-09-17T07:08:36.050 INFO:teuthology.orchestra.run.smithi100.stdout: "removed_snaps": "[]", 2024-09-17T07:08:36.050 INFO:teuthology.orchestra.run.smithi100.stdout: "quota_max_bytes": 0, 2024-09-17T07:08:36.051 INFO:teuthology.orchestra.run.smithi100.stdout: "quota_max_objects": 0, 2024-09-17T07:08:36.051 INFO:teuthology.orchestra.run.smithi100.stdout: "tiers": [], 2024-09-17T07:08:36.051 INFO:teuthology.orchestra.run.smithi100.stdout: "tier_of": -1, 2024-09-17T07:08:36.051 INFO:teuthology.orchestra.run.smithi100.stdout: "read_tier": -1, 2024-09-17T07:08:36.051 INFO:teuthology.orchestra.run.smithi100.stdout: "write_tier": -1, 2024-09-17T07:08:36.051 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_mode": "none", 2024-09-17T07:08:36.051 INFO:teuthology.orchestra.run.smithi100.stdout: "target_max_bytes": 0, 2024-09-17T07:08:36.051 INFO:teuthology.orchestra.run.smithi100.stdout: "target_max_objects": 0, 2024-09-17T07:08:36.051 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T07:08:36.051 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T07:08:36.051 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T07:08:36.051 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_min_flush_age": 0, 2024-09-17T07:08:36.051 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_min_evict_age": 0, 2024-09-17T07:08:36.052 INFO:teuthology.orchestra.run.smithi100.stdout: "erasure_code_profile": "", 2024-09-17T07:08:36.052 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_params": { 2024-09-17T07:08:36.052 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "none" 2024-09-17T07:08:36.052 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:36.052 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_period": 0, 2024-09-17T07:08:36.052 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_count": 0, 2024-09-17T07:08:36.052 INFO:teuthology.orchestra.run.smithi100.stdout: "use_gmt_hitset": true, 2024-09-17T07:08:36.052 INFO:teuthology.orchestra.run.smithi100.stdout: "min_read_recency_for_promote": 0, 2024-09-17T07:08:36.052 INFO:teuthology.orchestra.run.smithi100.stdout: "min_write_recency_for_promote": 0, 2024-09-17T07:08:36.052 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T07:08:36.052 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_search_last_n": 0, 2024-09-17T07:08:36.052 INFO:teuthology.orchestra.run.smithi100.stdout: "grade_table": [], 2024-09-17T07:08:36.053 INFO:teuthology.orchestra.run.smithi100.stdout: "stripe_width": 0, 2024-09-17T07:08:36.053 INFO:teuthology.orchestra.run.smithi100.stdout: "expected_num_objects": 0, 2024-09-17T07:08:36.053 INFO:teuthology.orchestra.run.smithi100.stdout: "fast_read": false, 2024-09-17T07:08:36.053 INFO:teuthology.orchestra.run.smithi100.stdout: "options": {}, 2024-09-17T07:08:36.053 INFO:teuthology.orchestra.run.smithi100.stdout: "application_metadata": { 2024-09-17T07:08:36.053 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw": {} 2024-09-17T07:08:36.053 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:36.053 INFO:teuthology.orchestra.run.smithi100.stdout: "read_balance": { 2024-09-17T07:08:36.053 INFO:teuthology.orchestra.run.smithi100.stdout: "score_acting": 1.25, 2024-09-17T07:08:36.053 INFO:teuthology.orchestra.run.smithi100.stdout: "score_stable": 1.25, 2024-09-17T07:08:36.053 INFO:teuthology.orchestra.run.smithi100.stdout: "optimal_score": 1, 2024-09-17T07:08:36.053 INFO:teuthology.orchestra.run.smithi100.stdout: "raw_score_acting": 1.25, 2024-09-17T07:08:36.053 INFO:teuthology.orchestra.run.smithi100.stdout: "raw_score_stable": 1.25, 2024-09-17T07:08:36.054 INFO:teuthology.orchestra.run.smithi100.stdout: "primary_affinity_weighted": 0.89598274230957031, 2024-09-17T07:08:36.054 INFO:teuthology.orchestra.run.smithi100.stdout: "average_primary_affinity": 0.89999997615814209, 2024-09-17T07:08:36.054 INFO:teuthology.orchestra.run.smithi100.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-09-17T07:08:36.054 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:36.054 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:36.054 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:36.054 INFO:teuthology.orchestra.run.smithi100.stdout: "pool": 6, 2024-09-17T07:08:36.054 INFO:teuthology.orchestra.run.smithi100.stdout: "pool_name": "default.rgw.meta", 2024-09-17T07:08:36.054 INFO:teuthology.orchestra.run.smithi100.stdout: "create_time": "2024-09-17T07:02:16.310620+0000", 2024-09-17T07:08:36.054 INFO:teuthology.orchestra.run.smithi100.stdout: "flags": 1, 2024-09-17T07:08:36.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:36.055 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:36.056 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:08:36.056 INFO:teuthology.orchestra.run.smithi100.stdout: "flags_names": "hashpspool", 2024-09-17T07:08:36.056 INFO:teuthology.orchestra.run.smithi100.stdout: "type": 1, 2024-09-17T07:08:36.056 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 3, 2024-09-17T07:08:36.056 INFO:teuthology.orchestra.run.smithi100.stdout: "min_size": 2, 2024-09-17T07:08:36.056 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_rule": 0, 2024-09-17T07:08:36.056 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_count": 0, 2024-09-17T07:08:36.057 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_target": 0, 2024-09-17T07:08:36.057 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T07:08:36.057 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T07:08:36.057 INFO:teuthology.orchestra.run.smithi100.stdout: "object_hash": 2, 2024-09-17T07:08:36.057 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_autoscale_mode": "off", 2024-09-17T07:08:36.057 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num": 32, 2024-09-17T07:08:36.057 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_placement_num": 32, 2024-09-17T07:08:36.057 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_placement_num_target": 32, 2024-09-17T07:08:36.057 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num_target": 32, 2024-09-17T07:08:36.057 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num_pending": 32, 2024-09-17T07:08:36.057 INFO:teuthology.orchestra.run.smithi100.stdout: "last_pg_merge_meta": { 2024-09-17T07:08:36.057 INFO:teuthology.orchestra.run.smithi100.stdout: "source_pgid": "0.0", 2024-09-17T07:08:36.058 INFO:teuthology.orchestra.run.smithi100.stdout: "ready_epoch": 0, 2024-09-17T07:08:36.058 INFO:teuthology.orchestra.run.smithi100.stdout: "last_epoch_started": 0, 2024-09-17T07:08:36.058 INFO:teuthology.orchestra.run.smithi100.stdout: "last_epoch_clean": 0, 2024-09-17T07:08:36.058 INFO:teuthology.orchestra.run.smithi100.stdout: "source_version": "0'0", 2024-09-17T07:08:36.058 INFO:teuthology.orchestra.run.smithi100.stdout: "target_version": "0'0" 2024-09-17T07:08:36.058 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:36.058 INFO:teuthology.orchestra.run.smithi100.stdout: "last_change": "75", 2024-09-17T07:08:36.058 INFO:teuthology.orchestra.run.smithi100.stdout: "last_force_op_resend": "0", 2024-09-17T07:08:36.058 INFO:teuthology.orchestra.run.smithi100.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-17T07:08:36.058 INFO:teuthology.orchestra.run.smithi100.stdout: "last_force_op_resend_preluminous": "0", 2024-09-17T07:08:36.058 INFO:teuthology.orchestra.run.smithi100.stdout: "auid": 0, 2024-09-17T07:08:36.058 INFO:teuthology.orchestra.run.smithi100.stdout: "snap_mode": "selfmanaged", 2024-09-17T07:08:36.058 INFO:teuthology.orchestra.run.smithi100.stdout: "snap_seq": 0, 2024-09-17T07:08:36.059 INFO:teuthology.orchestra.run.smithi100.stdout: "snap_epoch": 0, 2024-09-17T07:08:36.059 INFO:teuthology.orchestra.run.smithi100.stdout: "pool_snaps": [], 2024-09-17T07:08:36.059 INFO:teuthology.orchestra.run.smithi100.stdout: "removed_snaps": "[]", 2024-09-17T07:08:36.059 INFO:teuthology.orchestra.run.smithi100.stdout: "quota_max_bytes": 0, 2024-09-17T07:08:36.059 INFO:teuthology.orchestra.run.smithi100.stdout: "quota_max_objects": 0, 2024-09-17T07:08:36.059 INFO:teuthology.orchestra.run.smithi100.stdout: "tiers": [], 2024-09-17T07:08:36.059 INFO:teuthology.orchestra.run.smithi100.stdout: "tier_of": -1, 2024-09-17T07:08:36.059 INFO:teuthology.orchestra.run.smithi100.stdout: "read_tier": -1, 2024-09-17T07:08:36.059 INFO:teuthology.orchestra.run.smithi100.stdout: "write_tier": -1, 2024-09-17T07:08:36.059 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_mode": "none", 2024-09-17T07:08:36.059 INFO:teuthology.orchestra.run.smithi100.stdout: "target_max_bytes": 0, 2024-09-17T07:08:36.059 INFO:teuthology.orchestra.run.smithi100.stdout: "target_max_objects": 0, 2024-09-17T07:08:36.059 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T07:08:36.060 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T07:08:36.060 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T07:08:36.060 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_min_flush_age": 0, 2024-09-17T07:08:36.060 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_min_evict_age": 0, 2024-09-17T07:08:36.060 INFO:teuthology.orchestra.run.smithi100.stdout: "erasure_code_profile": "", 2024-09-17T07:08:36.060 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_params": { 2024-09-17T07:08:36.060 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "none" 2024-09-17T07:08:36.060 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:36.060 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_period": 0, 2024-09-17T07:08:36.060 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_count": 0, 2024-09-17T07:08:36.060 INFO:teuthology.orchestra.run.smithi100.stdout: "use_gmt_hitset": true, 2024-09-17T07:08:36.060 INFO:teuthology.orchestra.run.smithi100.stdout: "min_read_recency_for_promote": 0, 2024-09-17T07:08:36.061 INFO:teuthology.orchestra.run.smithi100.stdout: "min_write_recency_for_promote": 0, 2024-09-17T07:08:36.061 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T07:08:36.061 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_search_last_n": 0, 2024-09-17T07:08:36.061 INFO:teuthology.orchestra.run.smithi100.stdout: "grade_table": [], 2024-09-17T07:08:36.061 INFO:teuthology.orchestra.run.smithi100.stdout: "stripe_width": 0, 2024-09-17T07:08:36.061 INFO:teuthology.orchestra.run.smithi100.stdout: "expected_num_objects": 0, 2024-09-17T07:08:36.061 INFO:teuthology.orchestra.run.smithi100.stdout: "fast_read": false, 2024-09-17T07:08:36.061 INFO:teuthology.orchestra.run.smithi100.stdout: "options": { 2024-09-17T07:08:36.061 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_autoscale_bias": 4 2024-09-17T07:08:36.061 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:36.061 INFO:teuthology.orchestra.run.smithi100.stdout: "application_metadata": { 2024-09-17T07:08:36.061 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw": {} 2024-09-17T07:08:36.062 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:36.062 INFO:teuthology.orchestra.run.smithi100.stdout: "read_balance": { 2024-09-17T07:08:36.062 INFO:teuthology.orchestra.run.smithi100.stdout: "score_acting": 2, 2024-09-17T07:08:36.062 INFO:teuthology.orchestra.run.smithi100.stdout: "score_stable": 2, 2024-09-17T07:08:36.062 INFO:teuthology.orchestra.run.smithi100.stdout: "optimal_score": 1, 2024-09-17T07:08:36.062 INFO:teuthology.orchestra.run.smithi100.stdout: "raw_score_acting": 2, 2024-09-17T07:08:36.062 INFO:teuthology.orchestra.run.smithi100.stdout: "raw_score_stable": 2, 2024-09-17T07:08:36.062 INFO:teuthology.orchestra.run.smithi100.stdout: "primary_affinity_weighted": 0.89598274230957031, 2024-09-17T07:08:36.062 INFO:teuthology.orchestra.run.smithi100.stdout: "average_primary_affinity": 0.89999997615814209, 2024-09-17T07:08:36.062 INFO:teuthology.orchestra.run.smithi100.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-09-17T07:08:36.062 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:36.062 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:36.063 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:36.063 INFO:teuthology.orchestra.run.smithi100.stdout: "pool": 8, 2024-09-17T07:08:36.063 INFO:teuthology.orchestra.run.smithi100.stdout: "pool_name": "unique_pool_1", 2024-09-17T07:08:36.063 INFO:teuthology.orchestra.run.smithi100.stdout: "create_time": "2024-09-17T07:07:58.536259+0000", 2024-09-17T07:08:36.063 INFO:teuthology.orchestra.run.smithi100.stdout: "flags": 1, 2024-09-17T07:08:36.063 INFO:teuthology.orchestra.run.smithi100.stdout: "flags_names": "hashpspool", 2024-09-17T07:08:36.063 INFO:teuthology.orchestra.run.smithi100.stdout: "type": 1, 2024-09-17T07:08:36.063 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 3, 2024-09-17T07:08:36.063 INFO:teuthology.orchestra.run.smithi100.stdout: "min_size": 2, 2024-09-17T07:08:36.063 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_rule": 0, 2024-09-17T07:08:36.063 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_count": 0, 2024-09-17T07:08:36.063 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_target": 0, 2024-09-17T07:08:36.064 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T07:08:36.064 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T07:08:36.064 INFO:teuthology.orchestra.run.smithi100.stdout: "object_hash": 2, 2024-09-17T07:08:36.064 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_autoscale_mode": "off", 2024-09-17T07:08:36.064 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num": 16, 2024-09-17T07:08:36.064 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_placement_num": 16, 2024-09-17T07:08:36.064 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_placement_num_target": 16, 2024-09-17T07:08:36.064 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num_target": 16, 2024-09-17T07:08:36.064 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num_pending": 16, 2024-09-17T07:08:36.064 INFO:teuthology.orchestra.run.smithi100.stdout: "last_pg_merge_meta": { 2024-09-17T07:08:36.064 INFO:teuthology.orchestra.run.smithi100.stdout: "source_pgid": "0.0", 2024-09-17T07:08:36.064 INFO:teuthology.orchestra.run.smithi100.stdout: "ready_epoch": 0, 2024-09-17T07:08:36.064 INFO:teuthology.orchestra.run.smithi100.stdout: "last_epoch_started": 0, 2024-09-17T07:08:36.065 INFO:teuthology.orchestra.run.smithi100.stdout: "last_epoch_clean": 0, 2024-09-17T07:08:36.065 INFO:teuthology.orchestra.run.smithi100.stdout: "source_version": "0'0", 2024-09-17T07:08:36.065 INFO:teuthology.orchestra.run.smithi100.stdout: "target_version": "0'0" 2024-09-17T07:08:36.065 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:36.065 INFO:teuthology.orchestra.run.smithi100.stdout: "last_change": "174", 2024-09-17T07:08:36.065 INFO:teuthology.orchestra.run.smithi100.stdout: "last_force_op_resend": "0", 2024-09-17T07:08:36.065 INFO:teuthology.orchestra.run.smithi100.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-17T07:08:36.065 INFO:teuthology.orchestra.run.smithi100.stdout: "last_force_op_resend_preluminous": "0", 2024-09-17T07:08:36.065 INFO:teuthology.orchestra.run.smithi100.stdout: "auid": 0, 2024-09-17T07:08:36.065 INFO:teuthology.orchestra.run.smithi100.stdout: "snap_mode": "selfmanaged", 2024-09-17T07:08:36.065 INFO:teuthology.orchestra.run.smithi100.stdout: "snap_seq": 0, 2024-09-17T07:08:36.065 INFO:teuthology.orchestra.run.smithi100.stdout: "snap_epoch": 0, 2024-09-17T07:08:36.066 INFO:teuthology.orchestra.run.smithi100.stdout: "pool_snaps": [], 2024-09-17T07:08:36.066 INFO:teuthology.orchestra.run.smithi100.stdout: "removed_snaps": "[]", 2024-09-17T07:08:36.066 INFO:teuthology.orchestra.run.smithi100.stdout: "quota_max_bytes": 0, 2024-09-17T07:08:36.066 INFO:teuthology.orchestra.run.smithi100.stdout: "quota_max_objects": 0, 2024-09-17T07:08:36.066 INFO:teuthology.orchestra.run.smithi100.stdout: "tiers": [], 2024-09-17T07:08:36.066 INFO:teuthology.orchestra.run.smithi100.stdout: "tier_of": -1, 2024-09-17T07:08:36.066 INFO:teuthology.orchestra.run.smithi100.stdout: "read_tier": -1, 2024-09-17T07:08:36.066 INFO:teuthology.orchestra.run.smithi100.stdout: "write_tier": -1, 2024-09-17T07:08:36.066 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_mode": "none", 2024-09-17T07:08:36.066 INFO:teuthology.orchestra.run.smithi100.stdout: "target_max_bytes": 0, 2024-09-17T07:08:36.066 INFO:teuthology.orchestra.run.smithi100.stdout: "target_max_objects": 0, 2024-09-17T07:08:36.066 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T07:08:36.066 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T07:08:36.067 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T07:08:36.067 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_min_flush_age": 0, 2024-09-17T07:08:36.067 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_min_evict_age": 0, 2024-09-17T07:08:36.067 INFO:teuthology.orchestra.run.smithi100.stdout: "erasure_code_profile": "", 2024-09-17T07:08:36.067 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_params": { 2024-09-17T07:08:36.067 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "none" 2024-09-17T07:08:36.067 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:36.067 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_period": 0, 2024-09-17T07:08:36.067 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_count": 0, 2024-09-17T07:08:36.067 INFO:teuthology.orchestra.run.smithi100.stdout: "use_gmt_hitset": true, 2024-09-17T07:08:36.067 INFO:teuthology.orchestra.run.smithi100.stdout: "min_read_recency_for_promote": 0, 2024-09-17T07:08:36.067 INFO:teuthology.orchestra.run.smithi100.stdout: "min_write_recency_for_promote": 0, 2024-09-17T07:08:36.068 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T07:08:36.068 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_search_last_n": 0, 2024-09-17T07:08:36.068 INFO:teuthology.orchestra.run.smithi100.stdout: "grade_table": [], 2024-09-17T07:08:36.068 INFO:teuthology.orchestra.run.smithi100.stdout: "stripe_width": 0, 2024-09-17T07:08:36.068 INFO:teuthology.orchestra.run.smithi100.stdout: "expected_num_objects": 0, 2024-09-17T07:08:36.068 INFO:teuthology.orchestra.run.smithi100.stdout: "fast_read": false, 2024-09-17T07:08:36.068 INFO:teuthology.orchestra.run.smithi100.stdout: "options": {}, 2024-09-17T07:08:36.068 INFO:teuthology.orchestra.run.smithi100.stdout: "application_metadata": { 2024-09-17T07:08:36.068 INFO:teuthology.orchestra.run.smithi100.stdout: "rados": {} 2024-09-17T07:08:36.068 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:36.068 INFO:teuthology.orchestra.run.smithi100.stdout: "read_balance": { 2024-09-17T07:08:36.068 INFO:teuthology.orchestra.run.smithi100.stdout: "score_acting": 2, 2024-09-17T07:08:36.069 INFO:teuthology.orchestra.run.smithi100.stdout: "score_stable": 2, 2024-09-17T07:08:36.069 INFO:teuthology.orchestra.run.smithi100.stdout: "optimal_score": 1, 2024-09-17T07:08:36.069 INFO:teuthology.orchestra.run.smithi100.stdout: "raw_score_acting": 2, 2024-09-17T07:08:36.069 INFO:teuthology.orchestra.run.smithi100.stdout: "raw_score_stable": 2, 2024-09-17T07:08:36.069 INFO:teuthology.orchestra.run.smithi100.stdout: "primary_affinity_weighted": 0.89598274230957031, 2024-09-17T07:08:36.069 INFO:teuthology.orchestra.run.smithi100.stdout: "average_primary_affinity": 0.89999997615814209, 2024-09-17T07:08:36.069 INFO:teuthology.orchestra.run.smithi100.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-09-17T07:08:36.069 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:36.069 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:36.069 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-17T07:08:36.069 INFO:teuthology.orchestra.run.smithi100.stdout: "osds": [ 2024-09-17T07:08:36.069 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:36.069 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": 0, 2024-09-17T07:08:36.070 INFO:teuthology.orchestra.run.smithi100.stdout: "uuid": "27bee9ec-05e6-406c-8bf1-168098da0ec7", 2024-09-17T07:08:36.070 INFO:teuthology.orchestra.run.smithi100.stdout: "up": 1, 2024-09-17T07:08:36.070 INFO:teuthology.orchestra.run.smithi100.stdout: "in": 1, 2024-09-17T07:08:36.070 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 1, 2024-09-17T07:08:36.070 INFO:teuthology.orchestra.run.smithi100.stdout: "primary_affinity": 1, 2024-09-17T07:08:36.070 INFO:teuthology.orchestra.run.smithi100.stdout: "last_clean_begin": 0, 2024-09-17T07:08:36.070 INFO:teuthology.orchestra.run.smithi100.stdout: "last_clean_end": 0, 2024-09-17T07:08:36.070 INFO:teuthology.orchestra.run.smithi100.stdout: "up_from": 8, 2024-09-17T07:08:36.070 INFO:teuthology.orchestra.run.smithi100.stdout: "up_thru": 168, 2024-09-17T07:08:36.070 INFO:teuthology.orchestra.run.smithi100.stdout: "down_at": 0, 2024-09-17T07:08:36.070 INFO:teuthology.orchestra.run.smithi100.stdout: "lost_at": 0, 2024-09-17T07:08:36.070 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-17T07:08:36.071 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:08:36.071 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:36.071 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:08:36.071 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6801", 2024-09-17T07:08:36.071 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 3341017455 2024-09-17T07:08:36.071 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:36.071 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:36.071 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:36.071 INFO:teuthology.orchestra.run.smithi100.stdout: "cluster_addrs": { 2024-09-17T07:08:36.071 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:08:36.071 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:36.071 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:08:36.071 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6802", 2024-09-17T07:08:36.072 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 3341017455 2024-09-17T07:08:36.072 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:36.072 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:36.072 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:36.072 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_back_addrs": { 2024-09-17T07:08:36.072 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:08:36.072 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:36.072 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:08:36.072 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6804", 2024-09-17T07:08:36.072 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 3341017455 2024-09-17T07:08:36.072 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:36.072 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:36.073 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:36.073 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_front_addrs": { 2024-09-17T07:08:36.073 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:08:36.073 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:36.073 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:08:36.073 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6803", 2024-09-17T07:08:36.073 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 3341017455 2024-09-17T07:08:36.073 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:36.073 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:36.073 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:36.073 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.100:6801/3341017455", 2024-09-17T07:08:36.073 INFO:teuthology.orchestra.run.smithi100.stdout: "cluster_addr": "172.21.15.100:6802/3341017455", 2024-09-17T07:08:36.073 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_back_addr": "172.21.15.100:6804/3341017455", 2024-09-17T07:08:36.074 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_front_addr": "172.21.15.100:6803/3341017455", 2024-09-17T07:08:36.074 INFO:teuthology.orchestra.run.smithi100.stdout: "state": [ 2024-09-17T07:08:36.074 INFO:teuthology.orchestra.run.smithi100.stdout: "exists", 2024-09-17T07:08:36.074 INFO:teuthology.orchestra.run.smithi100.stdout: "up" 2024-09-17T07:08:36.074 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:36.074 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:36.074 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:36.074 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": 1, 2024-09-17T07:08:36.074 INFO:teuthology.orchestra.run.smithi100.stdout: "uuid": "b1a30c9c-eeaf-478e-9b62-2d63ad45873d", 2024-09-17T07:08:36.074 INFO:teuthology.orchestra.run.smithi100.stdout: "up": 1, 2024-09-17T07:08:36.074 INFO:teuthology.orchestra.run.smithi100.stdout: "in": 1, 2024-09-17T07:08:36.074 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 1, 2024-09-17T07:08:36.075 INFO:teuthology.orchestra.run.smithi100.stdout: "primary_affinity": 1, 2024-09-17T07:08:36.075 INFO:teuthology.orchestra.run.smithi100.stdout: "last_clean_begin": 0, 2024-09-17T07:08:36.075 INFO:teuthology.orchestra.run.smithi100.stdout: "last_clean_end": 0, 2024-09-17T07:08:36.075 INFO:teuthology.orchestra.run.smithi100.stdout: "up_from": 13, 2024-09-17T07:08:36.075 INFO:teuthology.orchestra.run.smithi100.stdout: "up_thru": 168, 2024-09-17T07:08:36.075 INFO:teuthology.orchestra.run.smithi100.stdout: "down_at": 0, 2024-09-17T07:08:36.075 INFO:teuthology.orchestra.run.smithi100.stdout: "lost_at": 0, 2024-09-17T07:08:36.075 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-17T07:08:36.075 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:08:36.075 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:36.076 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:2024-09-17T07:08:36.044695+0000 min lat: 0.0012044 max lat: 0.142002 avg lat: 0.00694556 2024-09-17T07:08:36.076 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-17T07:08:36.076 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 20 16 46058 46042 143.864 108.562 0.00366448 0.00694556 2024-09-17T07:08:36.076 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:08:36.076 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6805", 2024-09-17T07:08:36.077 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 1634750965 2024-09-17T07:08:36.077 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:36.077 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:36.077 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:36.077 INFO:teuthology.orchestra.run.smithi100.stdout: "cluster_addrs": { 2024-09-17T07:08:36.077 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:08:36.077 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:36.077 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:08:36.077 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6806", 2024-09-17T07:08:36.077 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 1634750965 2024-09-17T07:08:36.077 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:36.077 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:36.077 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:36.078 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_back_addrs": { 2024-09-17T07:08:36.078 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:08:36.078 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:36.078 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:08:36.078 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6808", 2024-09-17T07:08:36.078 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 1634750965 2024-09-17T07:08:36.078 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:36.078 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:36.078 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:36.078 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_front_addrs": { 2024-09-17T07:08:36.078 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:08:36.078 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:36.079 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:08:36.079 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6807", 2024-09-17T07:08:36.079 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 1634750965 2024-09-17T07:08:36.079 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:36.079 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:36.079 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:36.079 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.100:6805/1634750965", 2024-09-17T07:08:36.079 INFO:teuthology.orchestra.run.smithi100.stdout: "cluster_addr": "172.21.15.100:6806/1634750965", 2024-09-17T07:08:36.079 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_back_addr": "172.21.15.100:6808/1634750965", 2024-09-17T07:08:36.079 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_front_addr": "172.21.15.100:6807/1634750965", 2024-09-17T07:08:36.079 INFO:teuthology.orchestra.run.smithi100.stdout: "state": [ 2024-09-17T07:08:36.079 INFO:teuthology.orchestra.run.smithi100.stdout: "exists", 2024-09-17T07:08:36.080 INFO:teuthology.orchestra.run.smithi100.stdout: "up" 2024-09-17T07:08:36.080 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:36.080 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:36.080 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:36.080 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": 2, 2024-09-17T07:08:36.080 INFO:teuthology.orchestra.run.smithi100.stdout: "uuid": "ce542e0d-9915-465e-aa11-7b3ce13f694f", 2024-09-17T07:08:36.080 INFO:teuthology.orchestra.run.smithi100.stdout: "up": 1, 2024-09-17T07:08:36.080 INFO:teuthology.orchestra.run.smithi100.stdout: "in": 1, 2024-09-17T07:08:36.080 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 1, 2024-09-17T07:08:36.080 INFO:teuthology.orchestra.run.smithi100.stdout: "primary_affinity": 0.1678619384765625, 2024-09-17T07:08:36.080 INFO:teuthology.orchestra.run.smithi100.stdout: "last_clean_begin": 0, 2024-09-17T07:08:36.080 INFO:teuthology.orchestra.run.smithi100.stdout: "last_clean_end": 0, 2024-09-17T07:08:36.081 INFO:teuthology.orchestra.run.smithi100.stdout: "up_from": 18, 2024-09-17T07:08:36.081 INFO:teuthology.orchestra.run.smithi100.stdout: "up_thru": 133, 2024-09-17T07:08:36.081 INFO:teuthology.orchestra.run.smithi100.stdout: "down_at": 0, 2024-09-17T07:08:36.081 INFO:teuthology.orchestra.run.smithi100.stdout: "lost_at": 0, 2024-09-17T07:08:36.081 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-17T07:08:36.081 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:08:36.081 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:36.081 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:08:36.081 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6809", 2024-09-17T07:08:36.081 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 714579846 2024-09-17T07:08:36.081 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:36.081 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:36.082 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:36.082 INFO:teuthology.orchestra.run.smithi100.stdout: "cluster_addrs": { 2024-09-17T07:08:36.082 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:08:36.082 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:36.082 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:08:36.082 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6810", 2024-09-17T07:08:36.082 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 714579846 2024-09-17T07:08:36.082 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:36.082 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:36.082 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:36.082 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_back_addrs": { 2024-09-17T07:08:36.082 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:08:36.082 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:36.083 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:08:36.083 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6812", 2024-09-17T07:08:36.083 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 714579846 2024-09-17T07:08:36.083 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:36.083 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:36.083 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:36.083 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_front_addrs": { 2024-09-17T07:08:36.083 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:08:36.083 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:36.083 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:08:36.083 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6811", 2024-09-17T07:08:36.083 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 714579846 2024-09-17T07:08:36.083 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:36.084 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:36.084 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:36.084 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.100:6809/714579846", 2024-09-17T07:08:36.084 INFO:teuthology.orchestra.run.smithi100.stdout: "cluster_addr": "172.21.15.100:6810/714579846", 2024-09-17T07:08:36.084 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_back_addr": "172.21.15.100:6812/714579846", 2024-09-17T07:08:36.084 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_front_addr": "172.21.15.100:6811/714579846", 2024-09-17T07:08:36.084 INFO:teuthology.orchestra.run.smithi100.stdout: "state": [ 2024-09-17T07:08:36.084 INFO:teuthology.orchestra.run.smithi100.stdout: "exists", 2024-09-17T07:08:36.084 INFO:teuthology.orchestra.run.smithi100.stdout: "up" 2024-09-17T07:08:36.084 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:36.084 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:36.084 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:36.085 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": 3, 2024-09-17T07:08:36.085 INFO:teuthology.orchestra.run.smithi100.stdout: "uuid": "a1257575-3c90-41de-88f6-3960744351b1", 2024-09-17T07:08:36.085 INFO:teuthology.orchestra.run.smithi100.stdout: "up": 1, 2024-09-17T07:08:36.085 INFO:teuthology.orchestra.run.smithi100.stdout: "in": 1, 2024-09-17T07:08:36.085 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 1, 2024-09-17T07:08:36.085 INFO:teuthology.orchestra.run.smithi100.stdout: "primary_affinity": 1, 2024-09-17T07:08:36.085 INFO:teuthology.orchestra.run.smithi100.stdout: "last_clean_begin": 0, 2024-09-17T07:08:36.085 INFO:teuthology.orchestra.run.smithi100.stdout: "last_clean_end": 0, 2024-09-17T07:08:36.085 INFO:teuthology.orchestra.run.smithi100.stdout: "up_from": 27, 2024-09-17T07:08:36.085 INFO:teuthology.orchestra.run.smithi100.stdout: "up_thru": 168, 2024-09-17T07:08:36.085 INFO:teuthology.orchestra.run.smithi100.stdout: "down_at": 0, 2024-09-17T07:08:36.086 INFO:teuthology.orchestra.run.smithi100.stdout: "lost_at": 0, 2024-09-17T07:08:36.086 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-17T07:08:36.086 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:08:36.086 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:36.086 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:08:36.086 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6813", 2024-09-17T07:08:36.086 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 2315956572 2024-09-17T07:08:36.086 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:36.086 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:36.086 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:36.086 INFO:teuthology.orchestra.run.smithi100.stdout: "cluster_addrs": { 2024-09-17T07:08:36.086 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:08:36.086 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:36.087 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:08:36.087 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6814", 2024-09-17T07:08:36.087 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 2315956572 2024-09-17T07:08:36.087 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:36.087 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:36.087 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:36.087 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_back_addrs": { 2024-09-17T07:08:36.087 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:08:36.087 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:36.087 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:08:36.087 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6816", 2024-09-17T07:08:36.087 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 2315956572 2024-09-17T07:08:36.087 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:36.088 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:36.088 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:36.088 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_front_addrs": { 2024-09-17T07:08:36.088 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:08:36.088 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:36.088 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:08:36.088 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6815", 2024-09-17T07:08:36.088 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 2315956572 2024-09-17T07:08:36.088 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:36.088 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:36.088 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:36.088 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.100:6813/2315956572", 2024-09-17T07:08:36.089 INFO:teuthology.orchestra.run.smithi100.stdout: "cluster_addr": "172.21.15.100:6814/2315956572", 2024-09-17T07:08:36.089 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_back_addr": "172.21.15.100:6816/2315956572", 2024-09-17T07:08:36.089 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_front_addr": "172.21.15.100:6815/2315956572", 2024-09-17T07:08:36.089 INFO:teuthology.orchestra.run.smithi100.stdout: "state": [ 2024-09-17T07:08:36.089 INFO:teuthology.orchestra.run.smithi100.stdout: "exists", 2024-09-17T07:08:36.089 INFO:teuthology.orchestra.run.smithi100.stdout: "up" 2024-09-17T07:08:36.089 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:36.089 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:36.089 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:36.089 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": 4, 2024-09-17T07:08:36.089 INFO:teuthology.orchestra.run.smithi100.stdout: "uuid": "e44e8922-2c5e-44e3-b556-d4dfbc499649", 2024-09-17T07:08:36.089 INFO:teuthology.orchestra.run.smithi100.stdout: "up": 1, 2024-09-17T07:08:36.090 INFO:teuthology.orchestra.run.smithi100.stdout: "in": 1, 2024-09-17T07:08:36.090 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 1, 2024-09-17T07:08:36.090 INFO:teuthology.orchestra.run.smithi100.stdout: "primary_affinity": 1, 2024-09-17T07:08:36.090 INFO:teuthology.orchestra.run.smithi100.stdout: "last_clean_begin": 0, 2024-09-17T07:08:36.090 INFO:teuthology.orchestra.run.smithi100.stdout: "last_clean_end": 0, 2024-09-17T07:08:36.090 INFO:teuthology.orchestra.run.smithi100.stdout: "up_from": 33, 2024-09-17T07:08:36.090 INFO:teuthology.orchestra.run.smithi100.stdout: "up_thru": 168, 2024-09-17T07:08:36.090 INFO:teuthology.orchestra.run.smithi100.stdout: "down_at": 0, 2024-09-17T07:08:36.090 INFO:teuthology.orchestra.run.smithi100.stdout: "lost_at": 0, 2024-09-17T07:08:36.090 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-17T07:08:36.090 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:08:36.090 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:36.090 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:08:36.091 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.135:6800", 2024-09-17T07:08:36.091 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 1148820922 2024-09-17T07:08:36.091 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:36.091 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:36.091 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:36.091 INFO:teuthology.orchestra.run.smithi100.stdout: "cluster_addrs": { 2024-09-17T07:08:36.091 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:08:36.091 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:36.091 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:08:36.091 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.135:6801", 2024-09-17T07:08:36.091 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 1148820922 2024-09-17T07:08:36.091 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:36.092 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:36.092 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:36.092 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_back_addrs": { 2024-09-17T07:08:36.092 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:08:36.092 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:36.092 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:08:36.092 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.135:6803", 2024-09-17T07:08:36.092 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 1148820922 2024-09-17T07:08:36.092 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:36.092 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:36.092 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:36.092 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_front_addrs": { 2024-09-17T07:08:36.092 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:08:36.093 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:36.093 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:08:36.093 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.135:6802", 2024-09-17T07:08:36.093 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 1148820922 2024-09-17T07:08:36.093 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:36.093 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:36.093 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:36.093 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.135:6800/1148820922", 2024-09-17T07:08:36.093 INFO:teuthology.orchestra.run.smithi100.stdout: "cluster_addr": "172.21.15.135:6801/1148820922", 2024-09-17T07:08:36.093 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_back_addr": "172.21.15.135:6803/1148820922", 2024-09-17T07:08:36.093 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_front_addr": "172.21.15.135:6802/1148820922", 2024-09-17T07:08:36.093 INFO:teuthology.orchestra.run.smithi100.stdout: "state": [ 2024-09-17T07:08:36.094 INFO:teuthology.orchestra.run.smithi100.stdout: "exists", 2024-09-17T07:08:36.094 INFO:teuthology.orchestra.run.smithi100.stdout: "up" 2024-09-17T07:08:36.094 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:36.094 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:36.094 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:36.094 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": 5, 2024-09-17T07:08:36.094 INFO:teuthology.orchestra.run.smithi100.stdout: "uuid": "91e4a362-fcdb-41b9-b54e-716591ba6d79", 2024-09-17T07:08:36.094 INFO:teuthology.orchestra.run.smithi100.stdout: "up": 1, 2024-09-17T07:08:36.094 INFO:teuthology.orchestra.run.smithi100.stdout: "in": 1, 2024-09-17T07:08:36.094 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 1, 2024-09-17T07:08:36.094 INFO:teuthology.orchestra.run.smithi100.stdout: "primary_affinity": 1, 2024-09-17T07:08:36.094 INFO:teuthology.orchestra.run.smithi100.stdout: "last_clean_begin": 0, 2024-09-17T07:08:36.094 INFO:teuthology.orchestra.run.smithi100.stdout: "last_clean_end": 0, 2024-09-17T07:08:36.095 INFO:teuthology.orchestra.run.smithi100.stdout: "up_from": 42, 2024-09-17T07:08:36.095 INFO:teuthology.orchestra.run.smithi100.stdout: "up_thru": 168, 2024-09-17T07:08:36.095 INFO:teuthology.orchestra.run.smithi100.stdout: "down_at": 0, 2024-09-17T07:08:36.095 INFO:teuthology.orchestra.run.smithi100.stdout: "lost_at": 0, 2024-09-17T07:08:36.095 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-17T07:08:36.095 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:08:36.095 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:36.095 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:08:36.095 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.135:6804", 2024-09-17T07:08:36.095 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 2256520893 2024-09-17T07:08:36.095 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:36.095 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:36.095 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:36.095 INFO:teuthology.orchestra.run.smithi100.stdout: "cluster_addrs": { 2024-09-17T07:08:36.096 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:08:36.096 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:36.096 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:08:36.096 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.135:6805", 2024-09-17T07:08:36.096 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 2256520893 2024-09-17T07:08:36.096 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:36.096 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:36.096 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:36.096 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_back_addrs": { 2024-09-17T07:08:36.096 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:08:36.096 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:36.096 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:08:36.096 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.135:6807", 2024-09-17T07:08:36.097 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 2256520893 2024-09-17T07:08:36.097 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:36.097 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:36.097 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:36.097 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_front_addrs": { 2024-09-17T07:08:36.097 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:08:36.097 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:36.097 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:08:36.097 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.135:6806", 2024-09-17T07:08:36.097 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 2256520893 2024-09-17T07:08:36.097 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:36.098 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:36.098 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:36.098 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.135:6804/2256520893", 2024-09-17T07:08:36.098 INFO:teuthology.orchestra.run.smithi100.stdout: "cluster_addr": "172.21.15.135:6805/2256520893", 2024-09-17T07:08:36.098 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_back_addr": "172.21.15.135:6807/2256520893", 2024-09-17T07:08:36.098 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_front_addr": "172.21.15.135:6806/2256520893", 2024-09-17T07:08:36.098 INFO:teuthology.orchestra.run.smithi100.stdout: "state": [ 2024-09-17T07:08:36.098 INFO:teuthology.orchestra.run.smithi100.stdout: "exists", 2024-09-17T07:08:36.098 INFO:teuthology.orchestra.run.smithi100.stdout: "up" 2024-09-17T07:08:36.098 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:36.098 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:36.099 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:36.099 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": 6, 2024-09-17T07:08:36.099 INFO:teuthology.orchestra.run.smithi100.stdout: "uuid": "de799f64-ebd2-4a46-91f6-c2f276bee21f", 2024-09-17T07:08:36.099 INFO:teuthology.orchestra.run.smithi100.stdout: "up": 1, 2024-09-17T07:08:36.099 INFO:teuthology.orchestra.run.smithi100.stdout: "in": 1, 2024-09-17T07:08:36.099 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 1, 2024-09-17T07:08:36.099 INFO:teuthology.orchestra.run.smithi100.stdout: "primary_affinity": 1, 2024-09-17T07:08:36.099 INFO:teuthology.orchestra.run.smithi100.stdout: "last_clean_begin": 0, 2024-09-17T07:08:36.099 INFO:teuthology.orchestra.run.smithi100.stdout: "last_clean_end": 0, 2024-09-17T07:08:36.099 INFO:teuthology.orchestra.run.smithi100.stdout: "up_from": 50, 2024-09-17T07:08:36.099 INFO:teuthology.orchestra.run.smithi100.stdout: "up_thru": 168, 2024-09-17T07:08:36.099 INFO:teuthology.orchestra.run.smithi100.stdout: "down_at": 0, 2024-09-17T07:08:36.099 INFO:teuthology.orchestra.run.smithi100.stdout: "lost_at": 0, 2024-09-17T07:08:36.100 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-17T07:08:36.100 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:08:36.100 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:36.100 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:08:36.100 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.135:6808", 2024-09-17T07:08:36.100 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 2750184152 2024-09-17T07:08:36.100 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:36.100 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:36.100 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:36.100 INFO:teuthology.orchestra.run.smithi100.stdout: "cluster_addrs": { 2024-09-17T07:08:36.100 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:08:36.100 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:36.101 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:08:36.101 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.135:6809", 2024-09-17T07:08:36.101 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 2750184152 2024-09-17T07:08:36.101 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:36.101 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:36.101 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:36.101 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_back_addrs": { 2024-09-17T07:08:36.101 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:08:36.101 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:36.101 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:08:36.101 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.135:6811", 2024-09-17T07:08:36.101 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 2750184152 2024-09-17T07:08:36.102 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:36.102 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:36.102 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:36.102 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_front_addrs": { 2024-09-17T07:08:36.102 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:08:36.102 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:36.102 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:08:36.102 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.135:6810", 2024-09-17T07:08:36.102 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 2750184152 2024-09-17T07:08:36.102 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:36.102 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:36.102 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:36.102 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.135:6808/2750184152", 2024-09-17T07:08:36.103 INFO:teuthology.orchestra.run.smithi100.stdout: "cluster_addr": "172.21.15.135:6809/2750184152", 2024-09-17T07:08:36.103 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_back_addr": "172.21.15.135:6811/2750184152", 2024-09-17T07:08:36.103 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_front_addr": "172.21.15.135:6810/2750184152", 2024-09-17T07:08:36.103 INFO:teuthology.orchestra.run.smithi100.stdout: "state": [ 2024-09-17T07:08:36.103 INFO:teuthology.orchestra.run.smithi100.stdout: "exists", 2024-09-17T07:08:36.103 INFO:teuthology.orchestra.run.smithi100.stdout: "up" 2024-09-17T07:08:36.103 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:36.103 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:36.103 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:36.103 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": 7, 2024-09-17T07:08:36.103 INFO:teuthology.orchestra.run.smithi100.stdout: "uuid": "830932c7-5de1-44ff-a840-e53a6e859bda", 2024-09-17T07:08:36.104 INFO:teuthology.orchestra.run.smithi100.stdout: "up": 1, 2024-09-17T07:08:36.104 INFO:teuthology.orchestra.run.smithi100.stdout: "in": 1, 2024-09-17T07:08:36.104 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 1, 2024-09-17T07:08:36.104 INFO:teuthology.orchestra.run.smithi100.stdout: "primary_affinity": 1, 2024-09-17T07:08:36.104 INFO:teuthology.orchestra.run.smithi100.stdout: "last_clean_begin": 0, 2024-09-17T07:08:36.104 INFO:teuthology.orchestra.run.smithi100.stdout: "last_clean_end": 0, 2024-09-17T07:08:36.104 INFO:teuthology.orchestra.run.smithi100.stdout: "up_from": 60, 2024-09-17T07:08:36.104 INFO:teuthology.orchestra.run.smithi100.stdout: "up_thru": 145, 2024-09-17T07:08:36.104 INFO:teuthology.orchestra.run.smithi100.stdout: "down_at": 0, 2024-09-17T07:08:36.104 INFO:teuthology.orchestra.run.smithi100.stdout: "lost_at": 0, 2024-09-17T07:08:36.104 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-17T07:08:36.104 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:08:36.105 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:36.105 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:08:36.105 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.135:6812", 2024-09-17T07:08:36.105 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 160304773 2024-09-17T07:08:36.105 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:36.105 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:36.105 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:36.105 INFO:teuthology.orchestra.run.smithi100.stdout: "cluster_addrs": { 2024-09-17T07:08:36.105 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:08:36.105 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:36.105 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:08:36.105 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.135:6813", 2024-09-17T07:08:36.105 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 160304773 2024-09-17T07:08:36.106 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:36.106 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:36.106 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:36.106 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_back_addrs": { 2024-09-17T07:08:36.106 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:08:36.106 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:36.106 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:08:36.106 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.135:6815", 2024-09-17T07:08:36.106 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 160304773 2024-09-17T07:08:36.106 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:36.106 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:36.106 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:36.106 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_front_addrs": { 2024-09-17T07:08:36.107 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:08:36.107 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:36.107 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:08:36.107 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.135:6814", 2024-09-17T07:08:36.107 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 160304773 2024-09-17T07:08:36.107 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:36.107 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:36.107 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:36.107 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.135:6812/160304773", 2024-09-17T07:08:36.107 INFO:teuthology.orchestra.run.smithi100.stdout: "cluster_addr": "172.21.15.135:6813/160304773", 2024-09-17T07:08:36.107 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_back_addr": "172.21.15.135:6815/160304773", 2024-09-17T07:08:36.108 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_front_addr": "172.21.15.135:6814/160304773", 2024-09-17T07:08:36.108 INFO:teuthology.orchestra.run.smithi100.stdout: "state": [ 2024-09-17T07:08:36.108 INFO:teuthology.orchestra.run.smithi100.stdout: "exists", 2024-09-17T07:08:36.108 INFO:teuthology.orchestra.run.smithi100.stdout: "up" 2024-09-17T07:08:36.108 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:36.108 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:36.108 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-17T07:08:36.108 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_xinfo": [ 2024-09-17T07:08:36.108 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:36.108 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": 0, 2024-09-17T07:08:36.108 INFO:teuthology.orchestra.run.smithi100.stdout: "down_stamp": "0.000000", 2024-09-17T07:08:36.108 INFO:teuthology.orchestra.run.smithi100.stdout: "laggy_probability": 0, 2024-09-17T07:08:36.109 INFO:teuthology.orchestra.run.smithi100.stdout: "laggy_interval": 0, 2024-09-17T07:08:36.109 INFO:teuthology.orchestra.run.smithi100.stdout: "features": 4540138322906710015, 2024-09-17T07:08:36.109 INFO:teuthology.orchestra.run.smithi100.stdout: "old_weight": 0, 2024-09-17T07:08:36.109 INFO:teuthology.orchestra.run.smithi100.stdout: "last_purged_snaps_scrub": "2024-09-17T07:08:29.877041+0000", 2024-09-17T07:08:36.109 INFO:teuthology.orchestra.run.smithi100.stdout: "dead_epoch": 0 2024-09-17T07:08:36.109 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:36.109 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:36.109 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": 1, 2024-09-17T07:08:36.109 INFO:teuthology.orchestra.run.smithi100.stdout: "down_stamp": "0.000000", 2024-09-17T07:08:36.109 INFO:teuthology.orchestra.run.smithi100.stdout: "laggy_probability": 0, 2024-09-17T07:08:36.109 INFO:teuthology.orchestra.run.smithi100.stdout: "laggy_interval": 0, 2024-09-17T07:08:36.109 INFO:teuthology.orchestra.run.smithi100.stdout: "features": 4540138322906710015, 2024-09-17T07:08:36.109 INFO:teuthology.orchestra.run.smithi100.stdout: "old_weight": 0, 2024-09-17T07:08:36.110 INFO:teuthology.orchestra.run.smithi100.stdout: "last_purged_snaps_scrub": "2024-09-17T07:08:27.892594+0000", 2024-09-17T07:08:36.110 INFO:teuthology.orchestra.run.smithi100.stdout: "dead_epoch": 0 2024-09-17T07:08:36.110 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:36.110 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:36.110 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": 2, 2024-09-17T07:08:36.110 INFO:teuthology.orchestra.run.smithi100.stdout: "down_stamp": "0.000000", 2024-09-17T07:08:36.110 INFO:teuthology.orchestra.run.smithi100.stdout: "laggy_probability": 0, 2024-09-17T07:08:36.110 INFO:teuthology.orchestra.run.smithi100.stdout: "laggy_interval": 0, 2024-09-17T07:08:36.110 INFO:teuthology.orchestra.run.smithi100.stdout: "features": 4540138322906710015, 2024-09-17T07:08:36.110 INFO:teuthology.orchestra.run.smithi100.stdout: "old_weight": 0, 2024-09-17T07:08:36.110 INFO:teuthology.orchestra.run.smithi100.stdout: "last_purged_snaps_scrub": "2024-09-17T07:07:32.041868+0000", 2024-09-17T07:08:36.110 INFO:teuthology.orchestra.run.smithi100.stdout: "dead_epoch": 0 2024-09-17T07:08:36.111 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:36.111 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:36.111 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": 3, 2024-09-17T07:08:36.111 INFO:teuthology.orchestra.run.smithi100.stdout: "down_stamp": "0.000000", 2024-09-17T07:08:36.111 INFO:teuthology.orchestra.run.smithi100.stdout: "laggy_probability": 0, 2024-09-17T07:08:36.111 INFO:teuthology.orchestra.run.smithi100.stdout: "laggy_interval": 0, 2024-09-17T07:08:36.111 INFO:teuthology.orchestra.run.smithi100.stdout: "features": 4540138322906710015, 2024-09-17T07:08:36.111 INFO:teuthology.orchestra.run.smithi100.stdout: "old_weight": 0, 2024-09-17T07:08:36.111 INFO:teuthology.orchestra.run.smithi100.stdout: "last_purged_snaps_scrub": "2024-09-17T07:08:11.272607+0000", 2024-09-17T07:08:36.111 INFO:teuthology.orchestra.run.smithi100.stdout: "dead_epoch": 0 2024-09-17T07:08:36.111 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:36.111 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:36.111 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": 4, 2024-09-17T07:08:36.112 INFO:teuthology.orchestra.run.smithi100.stdout: "down_stamp": "0.000000", 2024-09-17T07:08:36.112 INFO:teuthology.orchestra.run.smithi100.stdout: "laggy_probability": 0, 2024-09-17T07:08:36.112 INFO:teuthology.orchestra.run.smithi100.stdout: "laggy_interval": 0, 2024-09-17T07:08:36.112 INFO:teuthology.orchestra.run.smithi100.stdout: "features": 4540138322906710015, 2024-09-17T07:08:36.112 INFO:teuthology.orchestra.run.smithi100.stdout: "old_weight": 0, 2024-09-17T07:08:36.112 INFO:teuthology.orchestra.run.smithi100.stdout: "last_purged_snaps_scrub": "2024-09-17T07:08:11.650237+0000", 2024-09-17T07:08:36.112 INFO:teuthology.orchestra.run.smithi100.stdout: "dead_epoch": 0 2024-09-17T07:08:36.112 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:36.112 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:36.112 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": 5, 2024-09-17T07:08:36.112 INFO:teuthology.orchestra.run.smithi100.stdout: "down_stamp": "0.000000", 2024-09-17T07:08:36.112 INFO:teuthology.orchestra.run.smithi100.stdout: "laggy_probability": 0, 2024-09-17T07:08:36.112 INFO:teuthology.orchestra.run.smithi100.stdout: "laggy_interval": 0, 2024-09-17T07:08:36.113 INFO:teuthology.orchestra.run.smithi100.stdout: "features": 4540138322906710015, 2024-09-17T07:08:36.113 INFO:teuthology.orchestra.run.smithi100.stdout: "old_weight": 0, 2024-09-17T07:08:36.113 INFO:teuthology.orchestra.run.smithi100.stdout: "last_purged_snaps_scrub": "2024-09-17T07:08:17.396878+0000", 2024-09-17T07:08:36.113 INFO:teuthology.orchestra.run.smithi100.stdout: "dead_epoch": 0 2024-09-17T07:08:36.113 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:36.113 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:36.113 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": 6, 2024-09-17T07:08:36.113 INFO:teuthology.orchestra.run.smithi100.stdout: "down_stamp": "0.000000", 2024-09-17T07:08:36.113 INFO:teuthology.orchestra.run.smithi100.stdout: "laggy_probability": 0, 2024-09-17T07:08:36.113 INFO:teuthology.orchestra.run.smithi100.stdout: "laggy_interval": 0, 2024-09-17T07:08:36.113 INFO:teuthology.orchestra.run.smithi100.stdout: "features": 4540138322906710015, 2024-09-17T07:08:36.113 INFO:teuthology.orchestra.run.smithi100.stdout: "old_weight": 0, 2024-09-17T07:08:36.114 INFO:teuthology.orchestra.run.smithi100.stdout: "last_purged_snaps_scrub": "2024-09-17T07:07:36.868729+0000", 2024-09-17T07:08:36.114 INFO:teuthology.orchestra.run.smithi100.stdout: "dead_epoch": 0 2024-09-17T07:08:36.114 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:36.114 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:36.114 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": 7, 2024-09-17T07:08:36.114 INFO:teuthology.orchestra.run.smithi100.stdout: "down_stamp": "0.000000", 2024-09-17T07:08:36.114 INFO:teuthology.orchestra.run.smithi100.stdout: "laggy_probability": 0, 2024-09-17T07:08:36.114 INFO:teuthology.orchestra.run.smithi100.stdout: "laggy_interval": 0, 2024-09-17T07:08:36.114 INFO:teuthology.orchestra.run.smithi100.stdout: "features": 4540138322906710015, 2024-09-17T07:08:36.114 INFO:teuthology.orchestra.run.smithi100.stdout: "old_weight": 0, 2024-09-17T07:08:36.114 INFO:teuthology.orchestra.run.smithi100.stdout: "last_purged_snaps_scrub": "2024-09-17T07:08:08.868194+0000", 2024-09-17T07:08:36.114 INFO:teuthology.orchestra.run.smithi100.stdout: "dead_epoch": 0 2024-09-17T07:08:36.114 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:36.115 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-17T07:08:36.115 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_upmap": [], 2024-09-17T07:08:36.115 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_upmap_items": [ 2024-09-17T07:08:36.115 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:36.115 INFO:teuthology.orchestra.run.smithi100.stdout: "pgid": "2.1a", 2024-09-17T07:08:36.115 INFO:teuthology.orchestra.run.smithi100.stdout: "mappings": [ 2024-09-17T07:08:36.115 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:36.115 INFO:teuthology.orchestra.run.smithi100.stdout: "from": 7, 2024-09-17T07:08:36.115 INFO:teuthology.orchestra.run.smithi100.stdout: "to": 2 2024-09-17T07:08:36.115 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:36.115 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:36.115 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:36.115 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:36.116 INFO:teuthology.orchestra.run.smithi100.stdout: "pgid": "4.6", 2024-09-17T07:08:36.116 INFO:teuthology.orchestra.run.smithi100.stdout: "mappings": [ 2024-09-17T07:08:36.116 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:36.116 INFO:teuthology.orchestra.run.smithi100.stdout: "from": 1, 2024-09-17T07:08:36.116 INFO:teuthology.orchestra.run.smithi100.stdout: "to": 5 2024-09-17T07:08:36.116 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:36.116 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:36.116 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:36.116 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:36.116 INFO:teuthology.orchestra.run.smithi100.stdout: "pgid": "4.10", 2024-09-17T07:08:36.116 INFO:teuthology.orchestra.run.smithi100.stdout: "mappings": [ 2024-09-17T07:08:36.116 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:36.116 INFO:teuthology.orchestra.run.smithi100.stdout: "from": 1, 2024-09-17T07:08:36.117 INFO:teuthology.orchestra.run.smithi100.stdout: "to": 2 2024-09-17T07:08:36.117 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:36.117 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:36.117 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:36.117 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:36.117 INFO:teuthology.orchestra.run.smithi100.stdout: "pgid": "4.1f", 2024-09-17T07:08:36.117 INFO:teuthology.orchestra.run.smithi100.stdout: "mappings": [ 2024-09-17T07:08:36.117 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:36.117 INFO:teuthology.orchestra.run.smithi100.stdout: "from": 1, 2024-09-17T07:08:36.117 INFO:teuthology.orchestra.run.smithi100.stdout: "to": 2 2024-09-17T07:08:36.117 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:36.117 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:36.118 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:36.118 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-17T07:08:36.118 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_upmap_primaries": [], 2024-09-17T07:08:36.118 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_temp": [], 2024-09-17T07:08:36.118 INFO:teuthology.orchestra.run.smithi100.stdout: "primary_temp": [], 2024-09-17T07:08:36.118 INFO:teuthology.orchestra.run.smithi100.stdout: "blocklist": { 2024-09-17T07:08:36.118 INFO:teuthology.orchestra.run.smithi100.stdout: "172.21.15.100:0/3796664909": "2024-09-18T06:57:35.678309+0000", 2024-09-17T07:08:36.118 INFO:teuthology.orchestra.run.smithi100.stdout: "172.21.15.100:0/2225586217": "2024-09-18T07:02:46.981486+0000", 2024-09-17T07:08:36.118 INFO:teuthology.orchestra.run.smithi100.stdout: "172.21.15.100:0/1626706829": "2024-09-18T06:57:35.678309+0000", 2024-09-17T07:08:36.118 INFO:teuthology.orchestra.run.smithi100.stdout: "172.21.15.100:0/3452172206": "2024-09-18T06:57:35.678309+0000", 2024-09-17T07:08:36.118 INFO:teuthology.orchestra.run.smithi100.stdout: "172.21.15.100:6800/3992296558": "2024-09-18T06:57:58.526309+0000", 2024-09-17T07:08:36.118 INFO:teuthology.orchestra.run.smithi100.stdout: "172.21.15.100:6800/3498574214": "2024-09-18T06:57:35.678309+0000", 2024-09-17T07:08:36.118 INFO:teuthology.orchestra.run.smithi100.stdout: "172.21.15.100:0/1365215411": "2024-09-18T06:57:58.526309+0000", 2024-09-17T07:08:36.119 INFO:teuthology.orchestra.run.smithi100.stdout: "172.21.15.100:0/3362169179": "2024-09-18T07:02:46.981486+0000", 2024-09-17T07:08:36.119 INFO:teuthology.orchestra.run.smithi100.stdout: "172.21.15.100:0/2573779025": "2024-09-18T07:02:46.981486+0000", 2024-09-17T07:08:36.119 INFO:teuthology.orchestra.run.smithi100.stdout: "172.21.15.100:0/380524491": "2024-09-18T07:02:46.981486+0000", 2024-09-17T07:08:36.119 INFO:teuthology.orchestra.run.smithi100.stdout: "172.21.15.100:0/1657291976": "2024-09-18T06:57:58.526309+0000", 2024-09-17T07:08:36.119 INFO:teuthology.orchestra.run.smithi100.stdout: "172.21.15.100:0/1676314112": "2024-09-18T07:02:46.981486+0000", 2024-09-17T07:08:36.119 INFO:teuthology.orchestra.run.smithi100.stdout: "172.21.15.100:0/1255197255": "2024-09-18T06:57:58.526309+0000", 2024-09-17T07:08:36.119 INFO:teuthology.orchestra.run.smithi100.stdout: "172.21.15.100:6800/3472658181": "2024-09-18T07:02:46.981486+0000" 2024-09-17T07:08:36.119 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:36.119 INFO:teuthology.orchestra.run.smithi100.stdout: "range_blocklist": {}, 2024-09-17T07:08:36.119 INFO:teuthology.orchestra.run.smithi100.stdout: "erasure_code_profiles": { 2024-09-17T07:08:36.119 INFO:teuthology.orchestra.run.smithi100.stdout: "default": { 2024-09-17T07:08:36.119 INFO:teuthology.orchestra.run.smithi100.stdout: "crush-failure-domain": "osd", 2024-09-17T07:08:36.119 INFO:teuthology.orchestra.run.smithi100.stdout: "k": "2", 2024-09-17T07:08:36.120 INFO:teuthology.orchestra.run.smithi100.stdout: "m": "1", 2024-09-17T07:08:36.120 INFO:teuthology.orchestra.run.smithi100.stdout: "plugin": "jerasure", 2024-09-17T07:08:36.120 INFO:teuthology.orchestra.run.smithi100.stdout: "technique": "reed_sol_van" 2024-09-17T07:08:36.120 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:36.120 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:36.120 INFO:teuthology.orchestra.run.smithi100.stdout: "removed_snaps_queue": [], 2024-09-17T07:08:36.120 INFO:teuthology.orchestra.run.smithi100.stdout: "new_removed_snaps": [], 2024-09-17T07:08:36.120 INFO:teuthology.orchestra.run.smithi100.stdout: "new_purged_snaps": [], 2024-09-17T07:08:36.120 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_node_flags": {}, 2024-09-17T07:08:36.120 INFO:teuthology.orchestra.run.smithi100.stdout: "device_class_flags": {}, 2024-09-17T07:08:36.120 INFO:teuthology.orchestra.run.smithi100.stdout: "stretch_mode": { 2024-09-17T07:08:36.120 INFO:teuthology.orchestra.run.smithi100.stdout: "stretch_mode_enabled": false, 2024-09-17T07:08:36.120 INFO:teuthology.orchestra.run.smithi100.stdout: "stretch_bucket_count": 0, 2024-09-17T07:08:36.121 INFO:teuthology.orchestra.run.smithi100.stdout: "degraded_stretch_mode": 0, 2024-09-17T07:08:36.121 INFO:teuthology.orchestra.run.smithi100.stdout: "recovering_stretch_mode": 0, 2024-09-17T07:08:36.121 INFO:teuthology.orchestra.run.smithi100.stdout: "stretch_mode_bucket": 0 2024-09-17T07:08:36.121 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:36.121 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-17T07:08:36.141 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:08:35 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:08:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:08:36.157 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:08:36.191 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:36.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:36.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:08:36.293 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:08:36.329 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:36.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:36.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:08:36.431 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:08:36.466 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:36.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:36.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:08:36.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:36 smithi135 ceph-mon[33238]: 6.1d scrub starts 2024-09-17T07:08:36.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:36 smithi135 ceph-mon[33238]: 6.1d scrub ok 2024-09-17T07:08:36.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:36 smithi135 ceph-mon[33238]: pgmap v280: 148 pgs: 148 active+clean; 2.5 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 763 B/s rd, 123 MiB/s wr, 1.97k op/s 2024-09-17T07:08:36.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:36 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/4248707136' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-17T07:08:36.568 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:08:36.573 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:36 smithi100 ceph-mon[29092]: 6.1d scrub starts 2024-09-17T07:08:36.573 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:36 smithi100 ceph-mon[29092]: 6.1d scrub ok 2024-09-17T07:08:36.573 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:36 smithi100 ceph-mon[29092]: pgmap v280: 148 pgs: 148 active+clean; 2.5 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 763 B/s rd, 123 MiB/s wr, 1.97k op/s 2024-09-17T07:08:36.573 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:36 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/4248707136' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-17T07:08:36.573 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:36 smithi100 ceph-mon[39462]: 6.1d scrub starts 2024-09-17T07:08:36.574 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:36 smithi100 ceph-mon[39462]: 6.1d scrub ok 2024-09-17T07:08:36.574 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:36 smithi100 ceph-mon[39462]: pgmap v280: 148 pgs: 148 active+clean; 2.5 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 763 B/s rd, 123 MiB/s wr, 1.97k op/s 2024-09-17T07:08:36.574 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:36 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/4248707136' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-17T07:08:36.594 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:36.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:36.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:08:36.696 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:08:36.727 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:36.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:36.728 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:08:36.829 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:08:36.866 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:36.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:36.867 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:08:36.968 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:08:37.003 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:37.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:37.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:08:37.044 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 21 16 47941 47925 142.617 117.688 0.00983977 0.00700656 2024-09-17T07:08:37.106 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:08:37.139 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:37.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:37.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:08:37.241 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:08:37.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:37.275 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:37.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:08:37.376 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:08:37.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:37.413 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:37.413 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:08:37.514 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:08:37.552 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:37.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:37.553 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:08:37.654 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:08:37.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:37.690 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:37.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:08:37.791 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:08:37.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:37.827 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:37.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:08:37.928 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:08:37.934 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:08:37.955 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:37.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:37.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:08:38.044 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 22 15 49802 49787 141.424 116.375 0.00874814 0.00706639 2024-09-17T07:08:38.057 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:08:38.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:38.092 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:38.092 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:08:38.193 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:08:38.233 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:38.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:38.234 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:08:38.334 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:08:38.368 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:38.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:38.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:08:38.375 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:38 smithi100 ceph-mon[29092]: pgmap v281: 148 pgs: 148 active+clean; 2.5 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 73 MiB/s wr, 1.17k op/s 2024-09-17T07:08:38.375 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:38 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:08:38.375 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:38 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:08:38.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:38 smithi100 ceph-mon[39462]: pgmap v281: 148 pgs: 148 active+clean; 2.5 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 73 MiB/s wr, 1.17k op/s 2024-09-17T07:08:38.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:38 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:08:38.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:38 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:08:38.416 INFO:tasks.thrashosds.thrasher:j is {'epoch': 181, 'fsid': 'e848267c-74c1-11ef-bceb-c7b262605968', 'created': '2024-09-17T06:57:05.010532+0000', 'modified': '2024-09-17T07:08:35.102587+0000', 'last_up_change': '2024-09-17T07:01:59.917118+0000', 'last_in_change': '2024-09-17T07:01:42.942454+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 8, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'reef', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-17T07:00:08.816136+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.8959827423095703, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-09-17T07:02:10.759826+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.8959827423095703, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 3, 'pool_name': 'datapool', 'create_time': '2024-09-17T07:02:10.991289+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 72, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 0.8959827423095703, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-17T07:02:12.291462+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.8959827423095703, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-17T07:02:14.253214+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '73', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 0.8959827423095703, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-17T07:02:16.310620+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8959827423095703, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 8, 'pool_name': 'unique_pool_1', 'create_time': '2024-09-17T07:07:58.536259+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '174', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8959827423095703, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}], 'osds': [{'osd': 0, 'uuid': '27bee9ec-05e6-406c-8bf1-168098da0ec7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 168, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6801', 'nonce': 3341017455}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6802', 'nonce': 3341017455}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6804', 'nonce': 3341017455}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6803', 'nonce': 3341017455}]}, 'public_addr': '172.21.15.100:6801/3341017455', 'cluster_addr': '172.21.15.100:6802/3341017455', 'heartbeat_back_addr': '172.21.15.100:6804/3341017455', 'heartbeat_front_addr': '172.21.15.100:6803/3341017455', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'b1a30c9c-eeaf-478e-9b62-2d63ad45873d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 168, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6805', 'nonce': 1634750965}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6806', 'nonce': 1634750965}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6808', 'nonce': 1634750965}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6807', 'nonce': 1634750965}]}, 'public_addr': '172.21.15.100:6805/1634750965', 'cluster_addr': '172.21.15.100:6806/1634750965', 'heartbeat_back_addr': '172.21.15.100:6808/1634750965', 'heartbeat_front_addr': '172.21.15.100:6807/1634750965', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'ce542e0d-9915-465e-aa11-7b3ce13f694f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.1678619384765625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 133, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6809', 'nonce': 714579846}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6810', 'nonce': 714579846}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6812', 'nonce': 714579846}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6811', 'nonce': 714579846}]}, 'public_addr': '172.21.15.100:6809/714579846', 'cluster_addr': '172.21.15.100:6810/714579846', 'heartbeat_back_addr': '172.21.15.100:6812/714579846', 'heartbeat_front_addr': '172.21.15.100:6811/714579846', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'a1257575-3c90-41de-88f6-3960744351b1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 27, 'up_thru': 168, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6813', 'nonce': 2315956572}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6814', 'nonce': 2315956572}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6816', 'nonce': 2315956572}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6815', 'nonce': 2315956572}]}, 'public_addr': '172.21.15.100:6813/2315956572', 'cluster_addr': '172.21.15.100:6814/2315956572', 'heartbeat_back_addr': '172.21.15.100:6816/2315956572', 'heartbeat_front_addr': '172.21.15.100:6815/2315956572', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'e44e8922-2c5e-44e3-b556-d4dfbc499649', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 33, 'up_thru': 168, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.135:6800', 'nonce': 1148820922}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.135:6801', 'nonce': 1148820922}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.135:6803', 'nonce': 1148820922}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.135:6802', 'nonce': 1148820922}]}, 'public_addr': '172.21.15.135:6800/1148820922', 'cluster_addr': '172.21.15.135:6801/1148820922', 'heartbeat_back_addr': '172.21.15.135:6803/1148820922', 'heartbeat_front_addr': '172.21.15.135:6802/1148820922', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '91e4a362-fcdb-41b9-b54e-716591ba6d79', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 42, 'up_thru': 168, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.135:6804', 'nonce': 2256520893}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.135:6805', 'nonce': 2256520893}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.135:6807', 'nonce': 2256520893}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.135:6806', 'nonce': 2256520893}]}, 'public_addr': '172.21.15.135:6804/2256520893', 'cluster_addr': '172.21.15.135:6805/2256520893', 'heartbeat_back_addr': '172.21.15.135:6807/2256520893', 'heartbeat_front_addr': '172.21.15.135:6806/2256520893', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'de799f64-ebd2-4a46-91f6-c2f276bee21f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 50, 'up_thru': 168, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.135:6808', 'nonce': 2750184152}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.135:6809', 'nonce': 2750184152}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.135:6811', 'nonce': 2750184152}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.135:6810', 'nonce': 2750184152}]}, 'public_addr': '172.21.15.135:6808/2750184152', 'cluster_addr': '172.21.15.135:6809/2750184152', 'heartbeat_back_addr': '172.21.15.135:6811/2750184152', 'heartbeat_front_addr': '172.21.15.135:6810/2750184152', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '830932c7-5de1-44ff-a840-e53a6e859bda', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 60, 'up_thru': 145, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.135:6812', 'nonce': 160304773}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.135:6813', 'nonce': 160304773}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.135:6815', 'nonce': 160304773}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.135:6814', 'nonce': 160304773}]}, 'public_addr': '172.21.15.135:6812/160304773', 'cluster_addr': '172.21.15.135:6813/160304773', 'heartbeat_back_addr': '172.21.15.135:6815/160304773', 'heartbeat_front_addr': '172.21.15.135:6814/160304773', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T07:08:29.877041+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T07:08:27.892594+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T07:07:32.041868+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T07:08:11.272607+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T07:08:11.650237+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T07:08:17.396878+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T07:07:36.868729+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T07:08:08.868194+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '2.1a', 'mappings': [{'from': 7, 'to': 2}]}, {'pgid': '4.6', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.10', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1f', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.100:0/3796664909': '2024-09-18T06:57:35.678309+0000', '172.21.15.100:0/2225586217': '2024-09-18T07:02:46.981486+0000', '172.21.15.100:0/1626706829': '2024-09-18T06:57:35.678309+0000', '172.21.15.100:0/3452172206': '2024-09-18T06:57:35.678309+0000', '172.21.15.100:6800/3992296558': '2024-09-18T06:57:58.526309+0000', '172.21.15.100:6800/3498574214': '2024-09-18T06:57:35.678309+0000', '172.21.15.100:0/1365215411': '2024-09-18T06:57:58.526309+0000', '172.21.15.100:0/3362169179': '2024-09-18T07:02:46.981486+0000', '172.21.15.100:0/2573779025': '2024-09-18T07:02:46.981486+0000', '172.21.15.100:0/380524491': '2024-09-18T07:02:46.981486+0000', '172.21.15.100:0/1657291976': '2024-09-18T06:57:58.526309+0000', '172.21.15.100:0/1676314112': '2024-09-18T07:02:46.981486+0000', '172.21.15.100:0/1255197255': '2024-09-18T06:57:58.526309+0000', '172.21.15.100:6800/3472658181': '2024-09-18T07:02:46.981486+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-17T07:08:38.417 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T07:08:38.469 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:08:38.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:38.523 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:38.524 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:08:38.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:38 smithi135 ceph-mon[33238]: pgmap v281: 148 pgs: 148 active+clean; 2.5 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 73 MiB/s wr, 1.17k op/s 2024-09-17T07:08:38.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:38 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:08:38.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:38 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:08:38.544 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:08:38 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:08:38.625 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:08:38.660 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:38.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:38.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:08:38.764 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:08:38.796 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:38.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:38.797 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:08:38.898 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:08:38.935 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:38.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:38.936 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:08:39.036 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:08:39.044 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 23 16 51434 51418 139.707 101.938 0.00329708 0.0071487 2024-09-17T07:08:39.058 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:39.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:39.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:08:39.160 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:08:39.197 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:39.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:39.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:08:39.298 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:08:39.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:39.333 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:39.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:08:39.377 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-17T07:08:39.377 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:08:39.391 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:08:39.434 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:08:39.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:39.473 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:39.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:08:39.500 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:39 smithi100 ceph-mon[29092]: osdmap e182: 8 total, 8 up, 8 in 2024-09-17T07:08:39.500 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:39 smithi100 ceph-mon[39462]: osdmap e182: 8 total, 8 up, 8 in 2024-09-17T07:08:39.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:39 smithi135 ceph-mon[33238]: osdmap e182: 8 total, 8 up, 8 in 2024-09-17T07:08:39.573 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:08:39.606 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:39.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:39.607 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:08:39.708 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:08:39.746 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:39.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:39.747 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:08:39.848 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:08:39.870 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:39.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:39.871 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:08:39.972 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:08:39.974 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-17T07:08:40.016 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:40.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:40.018 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:08:40.045 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 24 16 52994 52978 137.946 97.5 0.0142659 0.00724359 2024-09-17T07:08:40.118 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:08:40.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:40.161 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:40.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:08:40.262 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:08:40.299 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:40.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:40.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:08:40.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:40 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:08:40.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:40 smithi100 ceph-mon[29092]: pgmap v283: 148 pgs: 148 active+clean; 3.1 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 144 MiB/s wr, 2.30k op/s 2024-09-17T07:08:40.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:40 smithi100 ceph-mon[29092]: osdmap e183: 8 total, 8 up, 8 in 2024-09-17T07:08:40.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:40 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:08:40.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:40 smithi100 ceph-mon[39462]: pgmap v283: 148 pgs: 148 active+clean; 3.1 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 144 MiB/s wr, 2.30k op/s 2024-09-17T07:08:40.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:40 smithi100 ceph-mon[39462]: osdmap e183: 8 total, 8 up, 8 in 2024-09-17T07:08:40.401 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:08:40.436 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:40.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:40.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:08:40.538 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:08:40.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:40 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:08:40.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:40 smithi135 ceph-mon[33238]: pgmap v283: 148 pgs: 148 active+clean; 3.1 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 144 MiB/s wr, 2.30k op/s 2024-09-17T07:08:40.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:40 smithi135 ceph-mon[33238]: osdmap e183: 8 total, 8 up, 8 in 2024-09-17T07:08:40.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:40.571 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:40.571 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:08:40.672 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:08:40.706 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:40.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:40.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:08:40.808 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:08:40.836 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:40.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:40.837 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:08:40.938 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:08:40.974 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:40.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:40.975 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:08:41.045 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 25 15 54523 54508 136.253 95.625 0.0164494 0.00733443 2024-09-17T07:08:41.076 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:08:41.115 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:41.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:41.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:08:41.217 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:08:41.254 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:41.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:41.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:08:41.356 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:08:41.389 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:41.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:41.391 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:08:41.491 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:08:41.527 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:41.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:41.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:08:41.629 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:08:41.669 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-17T07:08:41.684 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:41.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:41.685 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:08:41.786 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:08:41.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:41.828 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:41.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:08:41.929 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:08:41.974 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:41.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:41.975 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:08:42.045 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 26 16 56209 56193 135.062 105.312 0.00499495 0.00739914 2024-09-17T07:08:42.075 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:08:42.113 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:42.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:42.114 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:08:42.215 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:08:42.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:42.254 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:42.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:08:42.355 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:08:42.388 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:42.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:42.389 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:08:42.490 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:08:42.526 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:42.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:42.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:08:42.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:42 smithi135 ceph-mon[33238]: pgmap v285: 148 pgs: 148 active+clean; 3.1 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 850 B/s rd, 94 MiB/s wr, 1.51k op/s 2024-09-17T07:08:42.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:42 smithi100 ceph-mon[29092]: pgmap v285: 148 pgs: 148 active+clean; 3.1 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 850 B/s rd, 94 MiB/s wr, 1.51k op/s 2024-09-17T07:08:42.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:42 smithi100 ceph-mon[39462]: pgmap v285: 148 pgs: 148 active+clean; 3.1 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 850 B/s rd, 94 MiB/s wr, 1.51k op/s 2024-09-17T07:08:42.628 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:08:42.661 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:42.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:42.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:08:42.763 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:08:42.801 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:42.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:42.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:08:42.903 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:08:42.969 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:42.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:42.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:08:43.046 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 27 16 57866 57850 133.893 103.562 0.00473034 0.00746339 2024-09-17T07:08:43.072 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:08:43.103 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:43.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:43.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:08:43.205 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:08:43.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:43.234 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:43.234 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:08:43.335 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:08:43.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:43.379 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:43.379 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:08:43.480 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:08:43.511 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:43.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:43.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:08:43.613 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:08:43.647 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:43.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:43.649 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:08:43.749 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:08:43.783 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:43.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:43.784 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:08:43.885 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:08:43.921 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:43.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:43.923 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:08:44.023 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:08:44.046 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 28 16 59466 59450 132.682 100 0.00604467 0.00753149 2024-09-17T07:08:44.059 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:44.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:44.060 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:08:44.161 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:08:44.199 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:44.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:44.200 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:08:44.301 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:08:44.335 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:44.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:44.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:08:44.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:44 smithi100 ceph-mon[29092]: pgmap v286: 148 pgs: 148 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 94 MiB/s wr, 1.51k op/s 2024-09-17T07:08:44.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:44 smithi100 ceph-mon[39462]: pgmap v286: 148 pgs: 148 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 94 MiB/s wr, 1.51k op/s 2024-09-17T07:08:44.437 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:08:44.467 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:44.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:44.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:08:44.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:44 smithi135 ceph-mon[33238]: pgmap v286: 148 pgs: 148 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 94 MiB/s wr, 1.51k op/s 2024-09-17T07:08:44.569 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:08:44.606 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:44.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:44.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:08:44.708 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:08:44.715 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:08:44.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:44.763 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:44.763 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:08:44.864 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:08:44.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:44.916 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:44.916 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:08:44.995 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:08:44.998 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:08:44.999 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:08:45.017 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:08:45.038 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1: {} 2024-09-17T07:08:45.038 INFO:teuthology.orchestra.run.smithi100.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:08:45.046 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 29 16 61003 60987 131.417 96.0625 0.00587574 0.00760244 2024-09-17T07:08:45.065 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:08:45.066 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:08:45.067 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:45.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:45.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:08:45.103 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:08:45.103 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:08:45.142 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:08:45.143 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:08:45.169 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:08:45.176 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:08:45.176 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:08:45.208 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:08:45.208 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:08:45.209 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:45.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:45.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:08:45.237 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:08:45.238 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:08:45.238 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:45 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/3530204220' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:08:45.238 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:45 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/3530204220' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:08:45.311 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:08:45.353 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:45.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:45.354 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:08:45.454 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:08:45.483 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:45.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:45.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:08:45.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:45 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/3530204220' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:08:45.586 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:08:45.618 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:45.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:45.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:08:45.720 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:08:45.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:45.759 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:45.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:08:45.860 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:08:45.885 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:45.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:45.886 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:08:45.986 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:08:46.022 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:46.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:46.023 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:08:46.046 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 30 16 62279 62263 129.695 79.75 0.0118381 0.00770562 2024-09-17T07:08:46.124 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:08:46.145 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:46.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:46.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:08:46.195 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:08:45 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:08:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:08:46.247 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:08:46.280 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:46.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:46.281 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:08:46.382 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:08:46.415 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:46.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:46.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:08:46.517 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:08:46.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:46 smithi135 ceph-mon[33238]: pgmap v287: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 134 MiB/s wr, 2.14k op/s 2024-09-17T07:08:46.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:46.553 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:46.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:08:46.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:46 smithi100 ceph-mon[29092]: pgmap v287: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 134 MiB/s wr, 2.14k op/s 2024-09-17T07:08:46.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:46 smithi100 ceph-mon[39462]: pgmap v287: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 134 MiB/s wr, 2.14k op/s 2024-09-17T07:08:46.654 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:08:46.689 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:46.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:46.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:08:46.791 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:08:46.820 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:46.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:46.822 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:08:46.924 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:08:46.957 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:46.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:46.958 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:08:47.046 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 31 16 63757 63741 128.491 92.375 0.00738114 0.00777824 2024-09-17T07:08:47.059 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:08:47.096 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:47.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:47.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:08:47.198 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:08:47.232 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:47.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:47.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:08:47.334 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:08:47.369 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:47.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:47.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:08:47.471 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:08:47.510 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:47.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:47.511 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:08:47.612 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:08:47.649 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:47.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:47.650 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:08:47.751 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:08:47.785 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:47.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:47.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:08:47.887 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:08:47.925 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:47.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:47.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:08:48.027 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:08:48.048 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 32 16 65184 65168 127.254 89.1875 0.00637149 0.00785325 2024-09-17T07:08:48.062 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:48.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:48.063 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:08:48.164 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:08:48.198 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:48.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:48.199 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:08:48.300 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:08:48.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:48.337 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:48.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:08:48.438 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:08:48.469 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:48.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:48.470 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:08:48.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:48 smithi135 ceph-mon[33238]: pgmap v288: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.1 KiB/s rd, 95 MiB/s wr, 1.53k op/s 2024-09-17T07:08:48.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:48 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:08:48.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:48 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:08:48.543 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:08:48 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:08:48.551 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:48 smithi100 ceph-mon[29092]: pgmap v288: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.1 KiB/s rd, 95 MiB/s wr, 1.53k op/s 2024-09-17T07:08:48.551 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:48 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:08:48.552 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:48 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:08:48.552 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:48 smithi100 ceph-mon[39462]: pgmap v288: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.1 KiB/s rd, 95 MiB/s wr, 1.53k op/s 2024-09-17T07:08:48.552 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:48 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:08:48.552 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:48 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:08:48.571 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:08:48.603 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:48.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:48.604 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:08:48.705 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:08:48.741 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:48.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:48.742 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:08:48.843 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:08:48.881 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:48.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:48.882 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:08:48.983 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:08:49.019 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:49.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:49.020 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:08:49.048 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 33 16 66682 66666 126.235 93.625 0.00320383 0.00791664 2024-09-17T07:08:49.121 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:08:49.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:49.158 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:49.158 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:08:49.259 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:08:49.288 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:49.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:49.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:08:49.389 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:08:49.419 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:49.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:49.420 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:08:49.521 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:08:49.558 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:49.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:49.560 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:08:49.660 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:08:49.690 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:49.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:49.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:08:49.792 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:08:49.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:49.825 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:49.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:08:49.926 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:08:49.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:49.981 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:49.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:08:50.050 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 34 16 68175 68159 125.263 93.3125 0.00998945 0.00797805 2024-09-17T07:08:50.082 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:08:50.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:50.126 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:50.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:08:50.227 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:08:50.262 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:50.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:50.263 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:08:50.364 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:08:50.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:50.403 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:50.403 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:08:50.464 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:50 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:08:50.464 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:50 smithi100 ceph-mon[29092]: pgmap v289: 148 pgs: 148 active+clean; 4.0 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 96 MiB/s wr, 1.53k op/s 2024-09-17T07:08:50.465 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:50 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:08:50.465 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:50 smithi100 ceph-mon[39462]: pgmap v289: 148 pgs: 148 active+clean; 4.0 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 96 MiB/s wr, 1.53k op/s 2024-09-17T07:08:50.504 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:08:50.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:50.526 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:50.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:08:50.536 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T07:08:50.539 INFO:teuthology.orchestra.run.smithi100.stderr:dumped all 2024-09-17T07:08:50.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:50 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:08:50.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:50 smithi135 ceph-mon[33238]: pgmap v289: 148 pgs: 148 active+clean; 4.0 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 96 MiB/s wr, 1.53k op/s 2024-09-17T07:08:50.627 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:08:50.664 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:50.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:50.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:08:50.766 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:08:50.794 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:50.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:50.795 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:08:50.896 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:08:50.927 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:50.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:50.928 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:08:51.029 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:08:51.050 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 35 16 69422 69406 123.911 77.9375 0.00912317 0.00806472 2024-09-17T07:08:51.057 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:51.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:51.058 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:08:51.159 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:08:51.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:51.196 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:51.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:08:51.297 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:08:51.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:51.333 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:51.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:08:51.434 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:08:51.473 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:51.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:51.474 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:08:51.494 INFO:teuthology.orchestra.run.smithi100.stderr:nodeep-scrub is set 2024-09-17T07:08:51.494 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:51 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/370320959' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:08:51.495 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:51 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/370320959' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:08:51.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:51 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/370320959' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:08:51.575 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:08:51.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:51.609 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:51.610 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:08:51.710 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:08:51.745 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:51.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:51.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:08:51.847 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:08:51.878 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:51.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:51.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:08:51.980 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:08:52.017 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:52.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:52.018 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:08:52.050 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 36 16 70714 70698 122.712 80.75 0.00541851 0.00814402 2024-09-17T07:08:52.120 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:08:52.156 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:52.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:52.157 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:08:52.258 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:08:52.290 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:52.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:52.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:08:52.391 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:08:52.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:52.427 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:52.427 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:08:52.528 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:08:52.545 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:52 smithi135 ceph-mon[33238]: from='client.24962 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:08:52.545 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:52 smithi135 ceph-mon[33238]: pgmap v290: 148 pgs: 148 active+clean; 4.0 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 929 B/s rd, 87 MiB/s wr, 1.39k op/s 2024-09-17T07:08:52.545 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:52 smithi135 ceph-mon[33238]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:08:52.545 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:52 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/370320959' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T07:08:52.545 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:52 smithi135 ceph-mon[33238]: osdmap e184: 8 total, 8 up, 8 in 2024-09-17T07:08:52.559 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:52.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:52.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:08:52.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:52 smithi100 ceph-mon[29092]: from='client.24962 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:08:52.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:52 smithi100 ceph-mon[29092]: pgmap v290: 148 pgs: 148 active+clean; 4.0 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 929 B/s rd, 87 MiB/s wr, 1.39k op/s 2024-09-17T07:08:52.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:52 smithi100 ceph-mon[29092]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:08:52.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:52 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/370320959' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T07:08:52.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:52 smithi100 ceph-mon[29092]: osdmap e184: 8 total, 8 up, 8 in 2024-09-17T07:08:52.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:52 smithi100 ceph-mon[39462]: from='client.24962 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:08:52.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:52 smithi100 ceph-mon[39462]: pgmap v290: 148 pgs: 148 active+clean; 4.0 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 929 B/s rd, 87 MiB/s wr, 1.39k op/s 2024-09-17T07:08:52.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:52 smithi100 ceph-mon[39462]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:08:52.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:52 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/370320959' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T07:08:52.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:52 smithi100 ceph-mon[39462]: osdmap e184: 8 total, 8 up, 8 in 2024-09-17T07:08:52.660 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:08:52.690 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:52.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:52.691 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:08:52.792 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:08:52.819 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:52.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:52.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:08:52.921 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:08:52.955 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:52.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:52.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:08:53.052 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 37 16 72080 72064 121.703 85.375 0.00820717 0.00821178 2024-09-17T07:08:53.058 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:08:53.092 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:53.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:53.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:08:53.194 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:08:53.230 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:53.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:53.231 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:08:53.331 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:08:53.367 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:53.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:53.368 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:08:53.404 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:08:53.469 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:08:53.501 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:53.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:53.502 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:08:53.603 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:08:53.643 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:53.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:53.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:08:53.745 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:08:53.782 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:53.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:53.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:08:53.884 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:08:53.931 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:53.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:53.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:08:54.033 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:08:54.050 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 38 16 73455 73439 120.761 85.9375 0.0114015 0.00827572 2024-09-17T07:08:54.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:54.067 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:54.067 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:08:54.168 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:08:54.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:54.204 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:54.205 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:08:54.305 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:08:54.342 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:54.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:54.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:08:54.444 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:08:54.475 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:54.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:54.477 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:08:54.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:54 smithi135 ceph-mon[33238]: pgmap v292: 148 pgs: 148 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 88 MiB/s wr, 1.40k op/s 2024-09-17T07:08:54.552 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:54 smithi100 ceph-mon[29092]: pgmap v292: 148 pgs: 148 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 88 MiB/s wr, 1.40k op/s 2024-09-17T07:08:54.552 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:54 smithi100 ceph-mon[39462]: pgmap v292: 148 pgs: 148 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 88 MiB/s wr, 1.40k op/s 2024-09-17T07:08:54.577 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:08:54.606 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:54.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:54.607 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:08:54.707 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:08:54.741 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:54.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:54.742 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:08:54.843 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:08:54.874 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:54.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:54.875 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:08:54.902 INFO:teuthology.orchestra.run.smithi100.stdout:{"pg_ready":true,"pg_map":{"version":289,"stamp":"2024-09-17T07:08:49.130115+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4306639126,"num_objects":65901,"num_object_clones":0,"num_object_copies":197703,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65901,"num_whiteouts":0,"num_read":1310,"num_read_kb":1156,"num_write":66314,"num_write_kb":4207579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1771214,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66342,"ondisk_log_size":66342,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":14272676,"kb_used_data":12452836,"kb_used_omap":12,"kb_used_meta":1819763,"kb_avail":735688540,"statfs":{"total":767960285184,"available":753345064960,"internally_reserved":0,"allocated":12751704064,"data_stored":12747976938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":1863437880},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,1,12,7,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1004994560,"num_objects":15335,"num_object_clones":0,"num_object_copies":46005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15335,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":15335,"num_write_kb":981440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002874"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":142,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.158747+0000","last_change":"2024-09-17T07:07:33.644835+0000","last_active":"2024-09-17T07:08:38.158747+0000","last_peered":"2024-09-17T07:08:38.158747+0000","last_clean":"2024-09-17T07:08:38.158747+0000","last_became_active":"2024-09-17T07:02:18.365773+0000","last_became_peered":"2024-09-17T07:02:18.365773+0000","last_unstale":"2024-09-17T07:08:38.158747+0000","last_undegraded":"2024-09-17T07:08:38.158747+0000","last_fullsized":"2024-09-17T07:08:38.158747+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:33.644728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:33.644728+0000","last_clean_scrub_stamp":"2024-09-17T07:07:33.644728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:46.708857+0000","scrub_duration":0.0070156680000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":161,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.158504+0000","last_change":"2024-09-17T07:08:17.332759+0000","last_active":"2024-09-17T07:08:38.158504+0000","last_peered":"2024-09-17T07:08:38.158504+0000","last_clean":"2024-09-17T07:08:38.158504+0000","last_became_active":"2024-09-17T07:02:12.006754+0000","last_became_peered":"2024-09-17T07:02:12.006754+0000","last_unstale":"2024-09-17T07:08:38.158504+0000","last_undegraded":"2024-09-17T07:08:38.158504+0000","last_fullsized":"2024-09-17T07:08:38.158504+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:17.332631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:08:17.332631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:41.165952+0000","scrub_duration":0.0023942220000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"75'15","reported_seq":182,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.158383+0000","last_change":"2024-09-17T07:08:28.845257+0000","last_active":"2024-09-17T07:08:38.158383+0000","last_peered":"2024-09-17T07:08:38.158383+0000","last_clean":"2024-09-17T07:08:38.158383+0000","last_became_active":"2024-09-17T07:02:14.012962+0000","last_became_peered":"2024-09-17T07:02:14.012962+0000","last_unstale":"2024-09-17T07:08:38.158383+0000","last_undegraded":"2024-09-17T07:08:38.158383+0000","last_fullsized":"2024-09-17T07:08:38.158383+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-09-17T07:08:28.845108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:08:28.845108+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:35.980199+0000","scrub_duration":0.011483625000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":145,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.150110+0000","last_change":"2024-09-17T07:08:21.564706+0000","last_active":"2024-09-17T07:08:38.150110+0000","last_peered":"2024-09-17T07:08:38.150110+0000","last_clean":"2024-09-17T07:08:38.150110+0000","last_became_active":"2024-09-17T07:02:16.014928+0000","last_became_peered":"2024-09-17T07:02:16.014928+0000","last_unstale":"2024-09-17T07:08:38.150110+0000","last_undegraded":"2024-09-17T07:08:38.150110+0000","last_fullsized":"2024-09-17T07:08:38.150110+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:21.564567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:08:21.564567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:42.719076+0000","scrub_duration":0.0039106849999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":149,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.160389+0000","last_change":"2024-09-17T07:07:15.767618+0000","last_active":"2024-09-17T07:08:38.160389+0000","last_peered":"2024-09-17T07:08:38.160389+0000","last_clean":"2024-09-17T07:08:38.160389+0000","last_became_active":"2024-09-17T07:02:12.010069+0000","last_became_peered":"2024-09-17T07:02:12.010069+0000","last_unstale":"2024-09-17T07:08:38.160389+0000","last_undegraded":"2024-09-17T07:08:38.160389+0000","last_fullsized":"2024-09-17T07:08:38.160389+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:15.767495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:07:15.767495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.013487149,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"75'9","reported_seq":174,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.150386+0000","last_change":"2024-09-17T07:08:22.546769+0000","last_active":"2024-09-17T07:08:38.150386+0000","last_peered":"2024-09-17T07:08:38.150386+0000","last_clean":"2024-09-17T07:08:38.150386+0000","last_became_active":"2024-09-17T07:02:14.016244+0000","last_became_peered":"2024-09-17T07:02:14.016244+0000","last_unstale":"2024-09-17T07:08:38.150386+0000","last_undegraded":"2024-09-17T07:08:38.150386+0000","last_fullsized":"2024-09-17T07:08:38.150386+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:08:22.546632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:08:22.546632+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:45.063195+0000","scrub_duration":0.0041894990000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":145,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.159952+0000","last_change":"2024-09-17T07:07:23.591188+0000","last_active":"2024-09-17T07:08:38.159952+0000","last_peered":"2024-09-17T07:08:38.159952+0000","last_clean":"2024-09-17T07:08:38.159952+0000","last_became_active":"2024-09-17T07:02:16.014380+0000","last_became_peered":"2024-09-17T07:02:16.014380+0000","last_unstale":"2024-09-17T07:08:38.159952+0000","last_undegraded":"2024-09-17T07:08:38.159952+0000","last_fullsized":"2024-09-17T07:08:38.159952+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:23.590927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:07:23.590927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:50.614784+0000","scrub_duration":0.0024742240000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":144,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.151719+0000","last_change":"2024-09-17T07:07:21.791530+0000","last_active":"2024-09-17T07:08:38.151719+0000","last_peered":"2024-09-17T07:08:38.151719+0000","last_clean":"2024-09-17T07:08:38.151719+0000","last_became_active":"2024-09-17T07:02:18.020790+0000","last_became_peered":"2024-09-17T07:02:18.020790+0000","last_unstale":"2024-09-17T07:08:38.151719+0000","last_undegraded":"2024-09-17T07:08:38.151719+0000","last_fullsized":"2024-09-17T07:08:38.151719+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:21.791399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:07:21.791399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0069986889999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":149,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.159914+0000","last_change":"2024-09-17T07:07:23.013609+0000","last_active":"2024-09-17T07:08:38.159914+0000","last_peered":"2024-09-17T07:08:38.159914+0000","last_clean":"2024-09-17T07:08:38.159914+0000","last_became_active":"2024-09-17T07:02:12.011129+0000","last_became_peered":"2024-09-17T07:02:12.011129+0000","last_unstale":"2024-09-17T07:08:38.159914+0000","last_undegraded":"2024-09-17T07:08:38.159914+0000","last_fullsized":"2024-09-17T07:08:38.159914+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:23.013470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:43.060102+0000","last_clean_scrub_stamp":"2024-09-17T07:07:23.013470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.006146281,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"75'5","reported_seq":158,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.152928+0000","last_change":"2024-09-17T07:07:15.848170+0000","last_active":"2024-09-17T07:08:38.152928+0000","last_peered":"2024-09-17T07:08:38.152928+0000","last_clean":"2024-09-17T07:08:38.152928+0000","last_became_active":"2024-09-17T07:02:14.016640+0000","last_became_peered":"2024-09-17T07:02:14.016640+0000","last_unstale":"2024-09-17T07:08:38.152928+0000","last_undegraded":"2024-09-17T07:08:38.152928+0000","last_fullsized":"2024-09-17T07:08:38.152928+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-09-17T07:07:15.848071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:07:15.848071+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010596959,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":156,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.158541+0000","last_change":"2024-09-17T07:08:19.081985+0000","last_active":"2024-09-17T07:08:38.158541+0000","last_peered":"2024-09-17T07:08:38.158541+0000","last_clean":"2024-09-17T07:08:38.158541+0000","last_became_active":"2024-09-17T07:02:16.015401+0000","last_became_peered":"2024-09-17T07:02:16.015401+0000","last_unstale":"2024-09-17T07:08:38.158541+0000","last_undegraded":"2024-09-17T07:08:38.158541+0000","last_fullsized":"2024-09-17T07:08:38.158541+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:19.081861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:08:19.081861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:42.103046+0000","scrub_duration":0.002870844,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":153,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.155086+0000","last_change":"2024-09-17T07:07:23.739288+0000","last_active":"2024-09-17T07:08:38.155086+0000","last_peered":"2024-09-17T07:08:38.155086+0000","last_clean":"2024-09-17T07:08:38.155086+0000","last_became_active":"2024-09-17T07:02:18.023680+0000","last_became_peered":"2024-09-17T07:02:18.023680+0000","last_unstale":"2024-09-17T07:08:38.155086+0000","last_undegraded":"2024-09-17T07:08:38.155086+0000","last_fullsized":"2024-09-17T07:08:38.155086+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:23.739157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:07:23.739157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004894603,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":149,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.159207+0000","last_change":"2024-09-17T07:07:21.000757+0000","last_active":"2024-09-17T07:08:38.159207+0000","last_peered":"2024-09-17T07:08:38.159207+0000","last_clean":"2024-09-17T07:08:38.159207+0000","last_became_active":"2024-09-17T07:02:12.012234+0000","last_became_peered":"2024-09-17T07:02:12.012234+0000","last_unstale":"2024-09-17T07:08:38.159207+0000","last_undegraded":"2024-09-17T07:08:38.159207+0000","last_fullsized":"2024-09-17T07:08:38.159207+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:21.000635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:45.029247+0000","last_clean_scrub_stamp":"2024-09-17T07:07:21.000635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0068373009999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"75'9","reported_seq":163,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.152073+0000","last_change":"2024-09-17T07:08:17.673480+0000","last_active":"2024-09-17T07:08:38.152073+0000","last_peered":"2024-09-17T07:08:38.152073+0000","last_clean":"2024-09-17T07:08:38.152073+0000","last_became_active":"2024-09-17T07:02:14.013337+0000","last_became_peered":"2024-09-17T07:02:14.013337+0000","last_unstale":"2024-09-17T07:08:38.152073+0000","last_undegraded":"2024-09-17T07:08:38.152073+0000","last_fullsized":"2024-09-17T07:08:38.152073+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:08:17.673242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:08:17.673242+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:28.557333+0000","scrub_duration":0.0059611450000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":155,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.155455+0000","last_change":"2024-09-17T07:08:28.096341+0000","last_active":"2024-09-17T07:08:38.155455+0000","last_peered":"2024-09-17T07:08:38.155455+0000","last_clean":"2024-09-17T07:08:38.155455+0000","last_became_active":"2024-09-17T07:02:16.023774+0000","last_became_peered":"2024-09-17T07:02:16.023774+0000","last_unstale":"2024-09-17T07:08:38.155455+0000","last_undegraded":"2024-09-17T07:08:38.155455+0000","last_fullsized":"2024-09-17T07:08:38.155455+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:28.096266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:28.096266+0000","last_clean_scrub_stamp":"2024-09-17T07:08:28.096266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:30.237292+0000","scrub_duration":0.010716228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":143,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.158112+0000","last_change":"2024-09-17T07:07:34.205329+0000","last_active":"2024-09-17T07:08:38.158112+0000","last_peered":"2024-09-17T07:08:38.158112+0000","last_clean":"2024-09-17T07:08:38.158112+0000","last_became_active":"2024-09-17T07:02:18.029178+0000","last_became_peered":"2024-09-17T07:02:18.029178+0000","last_unstale":"2024-09-17T07:08:38.158112+0000","last_undegraded":"2024-09-17T07:08:38.158112+0000","last_fullsized":"2024-09-17T07:08:38.158112+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:34.205205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:07:34.205205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:03.288846+0000","scrub_duration":0.0057819899999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"183'4070","reported_seq":4094,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-09-17T07:08:48.155766+0000","last_change":"2024-09-17T07:08:00.759806+0000","last_active":"2024-09-17T07:08:48.155766+0000","last_peered":"2024-09-17T07:08:48.155766+0000","last_clean":"2024-09-17T07:08:48.155766+0000","last_became_active":"2024-09-17T07:08:00.759498+0000","last_became_peered":"2024-09-17T07:08:00.759498+0000","last_unstale":"2024-09-17T07:08:48.155766+0000","last_undegraded":"2024-09-17T07:08:48.155766+0000","last_fullsized":"2024-09-17T07:08:48.155766+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_clean_scrub_stamp":"2024-09-17T07:07:59.220681+0000","objects_scrubbed":0,"log_size":4070,"log_dups_size":0,"ondisk_log_size":4070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:28.264255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":266731520,"num_objects":4070,"num_object_clones":0,"num_object_copies":12210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4070,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4070,"num_write_kb":260480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"95'25","reported_seq":183,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.151545+0000","last_change":"2024-09-17T07:06:41.707070+0000","last_active":"2024-09-17T07:08:38.151545+0000","last_peered":"2024-09-17T07:08:38.151545+0000","last_clean":"2024-09-17T07:08:38.151545+0000","last_became_active":"2024-09-17T07:02:14.016270+0000","last_became_peered":"2024-09-17T07:02:14.016270+0000","last_unstale":"2024-09-17T07:08:38.151545+0000","last_undegraded":"2024-09-17T07:08:38.151545+0000","last_fullsized":"2024-09-17T07:08:38.151545+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'25","last_scrub_stamp":"2024-09-17T07:06:41.706948+0000","last_deep_scrub":"95'25","last_deep_scrub_stamp":"2024-09-17T07:04:09.450039+0000","last_clean_scrub_stamp":"2024-09-17T07:06:41.706948+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.016404887999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":160,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.159297+0000","last_change":"2024-09-17T07:08:18.088347+0000","last_active":"2024-09-17T07:08:38.159297+0000","last_peered":"2024-09-17T07:08:38.159297+0000","last_clean":"2024-09-17T07:08:38.159297+0000","last_became_active":"2024-09-17T07:02:12.011258+0000","last_became_peered":"2024-09-17T07:02:12.011258+0000","last_unstale":"2024-09-17T07:08:38.159297+0000","last_undegraded":"2024-09-17T07:08:38.159297+0000","last_fullsized":"2024-09-17T07:08:38.159297+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:18.088246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:08:18.088246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:21.774873+0000","scrub_duration":0.0020459860000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":145,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.159420+0000","last_change":"2024-09-17T07:07:24.620899+0000","last_active":"2024-09-17T07:08:38.159420+0000","last_peered":"2024-09-17T07:08:38.159420+0000","last_clean":"2024-09-17T07:08:38.159420+0000","last_became_active":"2024-09-17T07:02:16.018147+0000","last_became_peered":"2024-09-17T07:02:16.018147+0000","last_unstale":"2024-09-17T07:08:38.159420+0000","last_undegraded":"2024-09-17T07:08:38.159420+0000","last_fullsized":"2024-09-17T07:08:38.159420+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:24.620758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:07:24.620758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:54.505017+0000","scrub_duration":0.0050498469999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":111,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-09-17T07:08:40.127051+0000","last_change":"2024-09-17T07:07:16.737235+0000","last_active":"2024-09-17T07:08:40.127051+0000","last_peered":"2024-09-17T07:08:40.127051+0000","last_clean":"2024-09-17T07:08:40.127051+0000","last_became_active":"2024-09-17T07:04:15.113196+0000","last_became_peered":"2024-09-17T07:04:15.113196+0000","last_unstale":"2024-09-17T07:08:40.127051+0000","last_undegraded":"2024-09-17T07:08:40.127051+0000","last_fullsized":"2024-09-17T07:08:40.127051+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:16.737086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:42.984088+0000","last_clean_scrub_stamp":"2024-09-17T07:07:16.737086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0083716439999999993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"183'4147","reported_seq":4170,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-09-17T07:08:48.668778+0000","last_change":"2024-09-17T07:08:00.240249+0000","last_active":"2024-09-17T07:08:48.668778+0000","last_peered":"2024-09-17T07:08:48.668778+0000","last_clean":"2024-09-17T07:08:48.668778+0000","last_became_active":"2024-09-17T07:08:00.239910+0000","last_became_peered":"2024-09-17T07:08:00.239910+0000","last_unstale":"2024-09-17T07:08:48.668778+0000","last_undegraded":"2024-09-17T07:08:48.668778+0000","last_fullsized":"2024-09-17T07:08:48.668778+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_clean_scrub_stamp":"2024-09-17T07:07:59.220681+0000","objects_scrubbed":0,"log_size":4147,"log_dups_size":0,"ondisk_log_size":4147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:13.311531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":271777792,"num_objects":4147,"num_object_clones":0,"num_object_copies":12441,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4147,"num_write_kb":265408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"176'15","reported_seq":178,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.151846+0000","last_change":"2024-09-17T07:08:19.633903+0000","last_active":"2024-09-17T07:08:38.151846+0000","last_peered":"2024-09-17T07:08:38.151846+0000","last_clean":"2024-09-17T07:08:38.151846+0000","last_became_active":"2024-09-17T07:02:14.007702+0000","last_became_peered":"2024-09-17T07:02:14.007702+0000","last_unstale":"2024-09-17T07:08:38.151846+0000","last_undegraded":"2024-09-17T07:08:38.151846+0000","last_fullsized":"2024-09-17T07:08:38.151846+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'15","last_scrub_stamp":"2024-09-17T07:08:19.633791+0000","last_deep_scrub":"176'15","last_deep_scrub_stamp":"2024-09-17T07:08:19.633791+0000","last_clean_scrub_stamp":"2024-09-17T07:08:19.633791+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:26.572318+0000","scrub_duration":0.009818284,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":160,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-09-17T07:08:40.126734+0000","last_change":"2024-09-17T07:08:23.802520+0000","last_active":"2024-09-17T07:08:40.126734+0000","last_peered":"2024-09-17T07:08:40.126734+0000","last_clean":"2024-09-17T07:08:40.126734+0000","last_became_active":"2024-09-17T07:02:12.007553+0000","last_became_peered":"2024-09-17T07:02:12.007553+0000","last_unstale":"2024-09-17T07:08:40.126734+0000","last_undegraded":"2024-09-17T07:08:40.126734+0000","last_fullsized":"2024-09-17T07:08:40.126734+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:23.802226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:08:23.802226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:24.700852+0000","scrub_duration":0.0065215869999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"98'9","reported_seq":196,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-09-17T07:08:45.592907+0000","last_change":"2024-09-17T07:08:31.923373+0000","last_active":"2024-09-17T07:08:45.592907+0000","last_peered":"2024-09-17T07:08:45.592907+0000","last_clean":"2024-09-17T07:08:45.592907+0000","last_became_active":"2024-09-17T07:04:15.110797+0000","last_became_peered":"2024-09-17T07:04:15.110797+0000","last_unstale":"2024-09-17T07:08:45.592907+0000","last_undegraded":"2024-09-17T07:08:45.592907+0000","last_fullsized":"2024-09-17T07:08:45.592907+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-09-17T07:08:31.923228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:08:31.923228+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:49.885819+0000","scrub_duration":0.015229100000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":154,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.150531+0000","last_change":"2024-09-17T07:08:30.650769+0000","last_active":"2024-09-17T07:08:38.150531+0000","last_peered":"2024-09-17T07:08:38.150531+0000","last_clean":"2024-09-17T07:08:38.150531+0000","last_became_active":"2024-09-17T07:02:18.017460+0000","last_became_peered":"2024-09-17T07:02:18.017460+0000","last_unstale":"2024-09-17T07:08:38.150531+0000","last_undegraded":"2024-09-17T07:08:38.150531+0000","last_fullsized":"2024-09-17T07:08:38.150531+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:30.650419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:08:30.650419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:34.258125+0000","scrub_duration":0.003351316,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"183'3989","reported_seq":4013,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-09-17T07:08:48.154776+0000","last_change":"2024-09-17T07:08:00.242384+0000","last_active":"2024-09-17T07:08:48.154776+0000","last_peered":"2024-09-17T07:08:48.154776+0000","last_clean":"2024-09-17T07:08:48.154776+0000","last_became_active":"2024-09-17T07:08:00.241996+0000","last_became_peered":"2024-09-17T07:08:00.241996+0000","last_unstale":"2024-09-17T07:08:48.154776+0000","last_undegraded":"2024-09-17T07:08:48.154776+0000","last_fullsized":"2024-09-17T07:08:48.154776+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_clean_scrub_stamp":"2024-09-17T07:07:59.220681+0000","objects_scrubbed":0,"log_size":3989,"log_dups_size":0,"ondisk_log_size":3989,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:18.953261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":261423104,"num_objects":3989,"num_object_clones":0,"num_object_copies":11967,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3989,"num_write_kb":255296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"75'15","reported_seq":182,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.162497+0000","last_change":"2024-09-17T07:08:13.784634+0000","last_active":"2024-09-17T07:08:38.162497+0000","last_peered":"2024-09-17T07:08:38.162497+0000","last_clean":"2024-09-17T07:08:38.162497+0000","last_became_active":"2024-09-17T07:02:14.016397+0000","last_became_peered":"2024-09-17T07:02:14.016397+0000","last_unstale":"2024-09-17T07:08:38.162497+0000","last_undegraded":"2024-09-17T07:08:38.162497+0000","last_fullsized":"2024-09-17T07:08:38.162497+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-09-17T07:08:13.784543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:08:13.784543+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:15.750232+0000","scrub_duration":0.002584257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":150,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-09-17T07:08:40.125040+0000","last_change":"2024-09-17T07:07:18.550881+0000","last_active":"2024-09-17T07:08:40.125040+0000","last_peered":"2024-09-17T07:08:40.125040+0000","last_clean":"2024-09-17T07:08:40.125040+0000","last_became_active":"2024-09-17T07:02:12.007913+0000","last_became_peered":"2024-09-17T07:02:12.007913+0000","last_unstale":"2024-09-17T07:08:40.125040+0000","last_undegraded":"2024-09-17T07:08:40.125040+0000","last_fullsized":"2024-09-17T07:08:40.125040+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:18.550750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:07:18.550750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:47.637615+0000","scrub_duration":0.012153809,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"75'8","reported_seq":232,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-09-17T07:08:40.593041+0000","last_change":"2024-09-17T07:08:09.787205+0000","last_active":"2024-09-17T07:08:40.593041+0000","last_peered":"2024-09-17T07:08:40.593041+0000","last_clean":"2024-09-17T07:08:40.593041+0000","last_became_active":"2024-09-17T07:02:16.021762+0000","last_became_peered":"2024-09-17T07:02:16.021762+0000","last_unstale":"2024-09-17T07:08:40.593041+0000","last_undegraded":"2024-09-17T07:08:40.593041+0000","last_fullsized":"2024-09-17T07:08:40.593041+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:08:09.787123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:08:09.787123+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:27.940895+0000","scrub_duration":0.001487141,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":153,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.157818+0000","last_change":"2024-09-17T07:08:12.991029+0000","last_active":"2024-09-17T07:08:38.157818+0000","last_peered":"2024-09-17T07:08:38.157818+0000","last_clean":"2024-09-17T07:08:38.157818+0000","last_became_active":"2024-09-17T07:02:18.025332+0000","last_became_peered":"2024-09-17T07:02:18.025332+0000","last_unstale":"2024-09-17T07:08:38.157818+0000","last_undegraded":"2024-09-17T07:08:38.157818+0000","last_fullsized":"2024-09-17T07:08:38.157818+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:12.990915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:08:12.990915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:33.309556+0000","scrub_duration":0.0021966199999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.2","version":"183'4159","reported_seq":4183,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-09-17T07:08:48.061138+0000","last_change":"2024-09-17T07:08:00.237404+0000","last_active":"2024-09-17T07:08:48.061138+0000","last_peered":"2024-09-17T07:08:48.061138+0000","last_clean":"2024-09-17T07:08:48.061138+0000","last_became_active":"2024-09-17T07:08:00.237068+0000","last_became_peered":"2024-09-17T07:08:00.237068+0000","last_unstale":"2024-09-17T07:08:48.061138+0000","last_undegraded":"2024-09-17T07:08:48.061138+0000","last_fullsized":"2024-09-17T07:08:48.061138+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_clean_scrub_stamp":"2024-09-17T07:07:59.220681+0000","objects_scrubbed":0,"log_size":4159,"log_dups_size":0,"ondisk_log_size":4159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:21.514789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":272564224,"num_objects":4159,"num_object_clones":0,"num_object_copies":12477,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4159,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4159,"num_write_kb":266176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"75'11","reported_seq":177,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.151660+0000","last_change":"2024-09-17T07:08:26.626034+0000","last_active":"2024-09-17T07:08:38.151660+0000","last_peered":"2024-09-17T07:08:38.151660+0000","last_clean":"2024-09-17T07:08:38.151660+0000","last_became_active":"2024-09-17T07:02:14.010227+0000","last_became_peered":"2024-09-17T07:02:14.010227+0000","last_unstale":"2024-09-17T07:08:38.151660+0000","last_undegraded":"2024-09-17T07:08:38.151660+0000","last_fullsized":"2024-09-17T07:08:38.151660+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-09-17T07:08:26.625892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:08:26.625892+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:34.185426+0000","scrub_duration":0.038079981999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":161,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-09-17T07:08:40.126142+0000","last_change":"2024-09-17T07:08:06.100857+0000","last_active":"2024-09-17T07:08:40.126142+0000","last_peered":"2024-09-17T07:08:40.126142+0000","last_clean":"2024-09-17T07:08:40.126142+0000","last_became_active":"2024-09-17T07:02:12.013182+0000","last_became_peered":"2024-09-17T07:02:12.013182+0000","last_unstale":"2024-09-17T07:08:40.126142+0000","last_undegraded":"2024-09-17T07:08:40.126142+0000","last_fullsized":"2024-09-17T07:08:40.126142+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:06.100741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:08:06.100741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:26.438221+0000","scrub_duration":0.0012621170000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":155,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.156038+0000","last_change":"2024-09-17T07:07:14.680573+0000","last_active":"2024-09-17T07:08:38.156038+0000","last_peered":"2024-09-17T07:08:38.156038+0000","last_clean":"2024-09-17T07:08:38.156038+0000","last_became_active":"2024-09-17T07:02:16.020224+0000","last_became_peered":"2024-09-17T07:02:16.020224+0000","last_unstale":"2024-09-17T07:08:38.156038+0000","last_undegraded":"2024-09-17T07:08:38.156038+0000","last_fullsized":"2024-09-17T07:08:38.156038+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:14.680446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:07:14.680446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023342229999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":153,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.158534+0000","last_change":"2024-09-17T07:08:27.883872+0000","last_active":"2024-09-17T07:08:38.158534+0000","last_peered":"2024-09-17T07:08:38.158534+0000","last_clean":"2024-09-17T07:08:38.158534+0000","last_became_active":"2024-09-17T07:02:18.365000+0000","last_became_peered":"2024-09-17T07:02:18.365000+0000","last_unstale":"2024-09-17T07:08:38.158534+0000","last_undegraded":"2024-09-17T07:08:38.158534+0000","last_fullsized":"2024-09-17T07:08:38.158534+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:27.883528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:08:27.883528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:45.322542+0000","scrub_duration":0.0071806070000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"183'4128","reported_seq":4152,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-09-17T07:08:48.039388+0000","last_change":"2024-09-17T07:08:00.760018+0000","last_active":"2024-09-17T07:08:48.039388+0000","last_peered":"2024-09-17T07:08:48.039388+0000","last_clean":"2024-09-17T07:08:48.039388+0000","last_became_active":"2024-09-17T07:08:00.759711+0000","last_became_peered":"2024-09-17T07:08:00.759711+0000","last_unstale":"2024-09-17T07:08:48.039388+0000","last_undegraded":"2024-09-17T07:08:48.039388+0000","last_fullsized":"2024-09-17T07:08:48.039388+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_clean_scrub_stamp":"2024-09-17T07:07:59.220681+0000","objects_scrubbed":0,"log_size":4128,"log_dups_size":0,"ondisk_log_size":4128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:20.125638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":270532608,"num_objects":4128,"num_object_clones":0,"num_object_copies":12384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4128,"num_write_kb":264192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1","version":"75'14","reported_seq":169,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.150249+0000","last_change":"2024-09-17T07:08:06.763827+0000","last_active":"2024-09-17T07:08:38.150249+0000","last_peered":"2024-09-17T07:08:38.150249+0000","last_clean":"2024-09-17T07:08:38.150249+0000","last_became_active":"2024-09-17T07:02:14.007810+0000","last_became_peered":"2024-09-17T07:02:14.007810+0000","last_unstale":"2024-09-17T07:08:38.150249+0000","last_undegraded":"2024-09-17T07:08:38.150249+0000","last_fullsized":"2024-09-17T07:08:38.150249+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'14","last_scrub_stamp":"2024-09-17T07:08:06.763689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:08:06.763689+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:34.614294+0000","scrub_duration":0.0033699329999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":171,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-09-17T07:08:40.122887+0000","last_change":"2024-09-17T07:08:16.485918+0000","last_active":"2024-09-17T07:08:40.122887+0000","last_peered":"2024-09-17T07:08:40.122887+0000","last_clean":"2024-09-17T07:08:40.122887+0000","last_became_active":"2024-09-17T07:02:12.011766+0000","last_became_peered":"2024-09-17T07:02:12.011766+0000","last_unstale":"2024-09-17T07:08:40.122887+0000","last_undegraded":"2024-09-17T07:08:40.122887+0000","last_fullsized":"2024-09-17T07:08:40.122887+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:16.485782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:08:16.485782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:21.067797+0000","scrub_duration":0.0023681280000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"75'8","reported_seq":232,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-09-17T07:08:40.592552+0000","last_change":"2024-09-17T07:06:29.982260+0000","last_active":"2024-09-17T07:08:40.592552+0000","last_peered":"2024-09-17T07:08:40.592552+0000","last_clean":"2024-09-17T07:08:40.592552+0000","last_became_active":"2024-09-17T07:02:16.020628+0000","last_became_peered":"2024-09-17T07:02:16.020628+0000","last_unstale":"2024-09-17T07:08:40.592552+0000","last_undegraded":"2024-09-17T07:08:40.592552+0000","last_fullsized":"2024-09-17T07:08:40.592552+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:06:29.982141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:06:29.982141+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0052922769999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":155,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.244232+0000","last_change":"2024-09-17T07:08:11.123128+0000","last_active":"2024-09-17T07:08:38.244232+0000","last_peered":"2024-09-17T07:08:38.244232+0000","last_clean":"2024-09-17T07:08:38.244232+0000","last_became_active":"2024-09-17T07:02:18.364981+0000","last_became_peered":"2024-09-17T07:02:18.364981+0000","last_unstale":"2024-09-17T07:08:38.244232+0000","last_undegraded":"2024-09-17T07:08:38.244232+0000","last_fullsized":"2024-09-17T07:08:38.244232+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:11.123047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:08:11.123047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:27.378027+0000","scrub_duration":0.0014377210000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"183'4335","reported_seq":4358,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-09-17T07:08:49.013850+0000","last_change":"2024-09-17T07:08:00.598282+0000","last_active":"2024-09-17T07:08:49.013850+0000","last_peered":"2024-09-17T07:08:49.013850+0000","last_clean":"2024-09-17T07:08:49.013850+0000","last_became_active":"2024-09-17T07:08:00.597939+0000","last_became_peered":"2024-09-17T07:08:00.597939+0000","last_unstale":"2024-09-17T07:08:49.013850+0000","last_undegraded":"2024-09-17T07:08:49.013850+0000","last_fullsized":"2024-09-17T07:08:49.013850+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_clean_scrub_stamp":"2024-09-17T07:07:59.220681+0000","objects_scrubbed":0,"log_size":4335,"log_dups_size":0,"ondisk_log_size":4335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:08.411403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284098560,"num_objects":4335,"num_object_clones":0,"num_object_copies":13005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4335,"num_write_kb":277440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"75'18","reported_seq":190,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.161105+0000","last_change":"2024-09-17T07:08:30.831998+0000","last_active":"2024-09-17T07:08:38.161105+0000","last_peered":"2024-09-17T07:08:38.161105+0000","last_clean":"2024-09-17T07:08:38.161105+0000","last_became_active":"2024-09-17T07:02:14.028667+0000","last_became_peered":"2024-09-17T07:02:14.028667+0000","last_unstale":"2024-09-17T07:08:38.161105+0000","last_undegraded":"2024-09-17T07:08:38.161105+0000","last_fullsized":"2024-09-17T07:08:38.161105+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'18","last_scrub_stamp":"2024-09-17T07:08:30.831301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:08:30.831301+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:32.186014+0000","scrub_duration":0.0072869559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":160,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-09-17T07:08:40.123842+0000","last_change":"2024-09-17T07:08:04.692191+0000","last_active":"2024-09-17T07:08:40.123842+0000","last_peered":"2024-09-17T07:08:40.123842+0000","last_clean":"2024-09-17T07:08:40.123842+0000","last_became_active":"2024-09-17T07:02:12.005687+0000","last_became_peered":"2024-09-17T07:02:12.005687+0000","last_unstale":"2024-09-17T07:08:40.123842+0000","last_undegraded":"2024-09-17T07:08:40.123842+0000","last_fullsized":"2024-09-17T07:08:40.123842+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:04.692099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:08:04.692099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:09.486554+0000","scrub_duration":0.001661023,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":146,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.150168+0000","last_change":"2024-09-17T07:08:03.726918+0000","last_active":"2024-09-17T07:08:38.150168+0000","last_peered":"2024-09-17T07:08:38.150168+0000","last_clean":"2024-09-17T07:08:38.150168+0000","last_became_active":"2024-09-17T07:02:16.019747+0000","last_became_peered":"2024-09-17T07:02:16.019747+0000","last_unstale":"2024-09-17T07:08:38.150168+0000","last_undegraded":"2024-09-17T07:08:38.150168+0000","last_fullsized":"2024-09-17T07:08:38.150168+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:03.726764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:08:03.726764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:09.587359+0000","scrub_duration":0.002251201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":155,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.152004+0000","last_change":"2024-09-17T07:08:25.589052+0000","last_active":"2024-09-17T07:08:38.152004+0000","last_peered":"2024-09-17T07:08:38.152004+0000","last_clean":"2024-09-17T07:08:38.152004+0000","last_became_active":"2024-09-17T07:02:18.024043+0000","last_became_peered":"2024-09-17T07:02:18.024043+0000","last_unstale":"2024-09-17T07:08:38.152004+0000","last_undegraded":"2024-09-17T07:08:38.152004+0000","last_fullsized":"2024-09-17T07:08:38.152004+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:25.588903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:08:25.588903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:29.872595+0000","scrub_duration":0.0021632869999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"183'4111","reported_seq":4135,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-09-17T07:08:48.671040+0000","last_change":"2024-09-17T07:08:00.759808+0000","last_active":"2024-09-17T07:08:48.671040+0000","last_peered":"2024-09-17T07:08:48.671040+0000","last_clean":"2024-09-17T07:08:48.671040+0000","last_became_active":"2024-09-17T07:08:00.759502+0000","last_became_peered":"2024-09-17T07:08:00.759502+0000","last_unstale":"2024-09-17T07:08:48.671040+0000","last_undegraded":"2024-09-17T07:08:48.671040+0000","last_fullsized":"2024-09-17T07:08:48.671040+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_clean_scrub_stamp":"2024-09-17T07:07:59.220681+0000","objects_scrubbed":0,"log_size":4111,"log_dups_size":0,"ondisk_log_size":4111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:59.866829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269418496,"num_objects":4111,"num_object_clones":0,"num_object_copies":12333,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4111,"num_write_kb":263104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"75'19","reported_seq":184,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.158153+0000","last_change":"2024-09-17T07:07:17.989796+0000","last_active":"2024-09-17T07:08:38.158153+0000","last_peered":"2024-09-17T07:08:38.158153+0000","last_clean":"2024-09-17T07:08:38.158153+0000","last_became_active":"2024-09-17T07:02:14.027389+0000","last_became_peered":"2024-09-17T07:02:14.027389+0000","last_unstale":"2024-09-17T07:08:38.158153+0000","last_undegraded":"2024-09-17T07:08:38.158153+0000","last_fullsized":"2024-09-17T07:08:38.158153+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-09-17T07:07:17.989435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:07:17.989435+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0043356940000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":161,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.159979+0000","last_change":"2024-09-17T07:08:24.072911+0000","last_active":"2024-09-17T07:08:38.159979+0000","last_peered":"2024-09-17T07:08:38.159979+0000","last_clean":"2024-09-17T07:08:38.159979+0000","last_became_active":"2024-09-17T07:02:12.011917+0000","last_became_peered":"2024-09-17T07:02:12.011917+0000","last_unstale":"2024-09-17T07:08:38.159979+0000","last_undegraded":"2024-09-17T07:08:38.159979+0000","last_fullsized":"2024-09-17T07:08:38.159979+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:24.072799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:24.072799+0000","last_clean_scrub_stamp":"2024-09-17T07:08:24.072799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:25.317799+0000","scrub_duration":0.0047928600000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":146,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.163299+0000","last_change":"2024-09-17T07:07:31.453488+0000","last_active":"2024-09-17T07:08:38.163299+0000","last_peered":"2024-09-17T07:08:38.163299+0000","last_clean":"2024-09-17T07:08:38.163299+0000","last_became_active":"2024-09-17T07:02:16.016199+0000","last_became_peered":"2024-09-17T07:02:16.016199+0000","last_unstale":"2024-09-17T07:08:38.163299+0000","last_undegraded":"2024-09-17T07:08:38.163299+0000","last_fullsized":"2024-09-17T07:08:38.163299+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:31.453355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:07:31.453355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:43.002059+0000","scrub_duration":0.0055939470000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":154,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.159877+0000","last_change":"2024-09-17T07:08:24.840933+0000","last_active":"2024-09-17T07:08:38.159877+0000","last_peered":"2024-09-17T07:08:38.159877+0000","last_clean":"2024-09-17T07:08:38.159877+0000","last_became_active":"2024-09-17T07:02:18.364835+0000","last_became_peered":"2024-09-17T07:02:18.364835+0000","last_unstale":"2024-09-17T07:08:38.159877+0000","last_undegraded":"2024-09-17T07:08:38.159877+0000","last_fullsized":"2024-09-17T07:08:38.159877+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:24.840821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:08:24.840821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:25.514588+0000","scrub_duration":0.002523197,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.e","version":"183'4170","reported_seq":4194,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-09-17T07:08:49.011226+0000","last_change":"2024-09-17T07:08:00.759796+0000","last_active":"2024-09-17T07:08:49.011226+0000","last_peered":"2024-09-17T07:08:49.011226+0000","last_clean":"2024-09-17T07:08:49.011226+0000","last_became_active":"2024-09-17T07:08:00.759537+0000","last_became_peered":"2024-09-17T07:08:00.759537+0000","last_unstale":"2024-09-17T07:08:49.011226+0000","last_undegraded":"2024-09-17T07:08:49.011226+0000","last_fullsized":"2024-09-17T07:08:49.011226+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_clean_scrub_stamp":"2024-09-17T07:07:59.220681+0000","objects_scrubbed":0,"log_size":4170,"log_dups_size":0,"ondisk_log_size":4170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:27.967133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":273285120,"num_objects":4170,"num_object_clones":0,"num_object_copies":12510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4170,"num_write_kb":266880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"110'15","reported_seq":178,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.159809+0000","last_change":"2024-09-17T07:08:19.835062+0000","last_active":"2024-09-17T07:08:38.159809+0000","last_peered":"2024-09-17T07:08:38.159809+0000","last_clean":"2024-09-17T07:08:38.159809+0000","last_became_active":"2024-09-17T07:02:14.010637+0000","last_became_peered":"2024-09-17T07:02:14.010637+0000","last_unstale":"2024-09-17T07:08:38.159809+0000","last_undegraded":"2024-09-17T07:08:38.159809+0000","last_fullsized":"2024-09-17T07:08:38.159809+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'15","last_scrub_stamp":"2024-09-17T07:08:19.834380+0000","last_deep_scrub":"110'15","last_deep_scrub_stamp":"2024-09-17T07:04:57.091400+0000","last_clean_scrub_stamp":"2024-09-17T07:08:19.834380+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:44.803473+0000","scrub_duration":0.0055436599999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":160,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.159721+0000","last_change":"2024-09-17T07:08:17.806138+0000","last_active":"2024-09-17T07:08:38.159721+0000","last_peered":"2024-09-17T07:08:38.159721+0000","last_clean":"2024-09-17T07:08:38.159721+0000","last_became_active":"2024-09-17T07:02:12.010803+0000","last_became_peered":"2024-09-17T07:02:12.010803+0000","last_unstale":"2024-09-17T07:08:38.159721+0000","last_undegraded":"2024-09-17T07:08:38.159721+0000","last_fullsized":"2024-09-17T07:08:38.159721+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:17.806001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:25.147878+0000","last_clean_scrub_stamp":"2024-09-17T07:08:17.806001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:29.869447+0000","scrub_duration":0.00606531,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"89'9","reported_seq":245,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-09-17T07:08:40.623040+0000","last_change":"2024-09-17T07:08:14.309093+0000","last_active":"2024-09-17T07:08:40.623040+0000","last_peered":"2024-09-17T07:08:40.623040+0000","last_clean":"2024-09-17T07:08:40.623040+0000","last_became_active":"2024-09-17T07:02:16.021360+0000","last_became_peered":"2024-09-17T07:02:16.021360+0000","last_unstale":"2024-09-17T07:08:40.623040+0000","last_undegraded":"2024-09-17T07:08:40.623040+0000","last_fullsized":"2024-09-17T07:08:40.623040+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'9","last_scrub_stamp":"2024-09-17T07:08:14.308970+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-09-17T07:03:29.961439+0000","last_clean_scrub_stamp":"2024-09-17T07:08:14.308970+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:21.985874+0000","scrub_duration":0.001890011,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"75'1","reported_seq":156,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.157753+0000","last_change":"2024-09-17T07:08:25.356066+0000","last_active":"2024-09-17T07:08:38.157753+0000","last_peered":"2024-09-17T07:08:38.157753+0000","last_clean":"2024-09-17T07:08:38.157753+0000","last_became_active":"2024-09-17T07:02:18.029772+0000","last_became_peered":"2024-09-17T07:02:18.029772+0000","last_unstale":"2024-09-17T07:08:38.157753+0000","last_undegraded":"2024-09-17T07:08:38.157753+0000","last_fullsized":"2024-09-17T07:08:38.157753+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-17T07:08:25.355935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:08:25.355935+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:41.337593+0000","scrub_duration":0.0052992940000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"183'4009","reported_seq":4033,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-09-17T07:08:48.035662+0000","last_change":"2024-09-17T07:08:00.598765+0000","last_active":"2024-09-17T07:08:48.035662+0000","last_peered":"2024-09-17T07:08:48.035662+0000","last_clean":"2024-09-17T07:08:48.035662+0000","last_became_active":"2024-09-17T07:08:00.598467+0000","last_became_peered":"2024-09-17T07:08:00.598467+0000","last_unstale":"2024-09-17T07:08:48.035662+0000","last_undegraded":"2024-09-17T07:08:48.035662+0000","last_fullsized":"2024-09-17T07:08:48.035662+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_clean_scrub_stamp":"2024-09-17T07:07:59.220681+0000","objects_scrubbed":0,"log_size":4009,"log_dups_size":0,"ondisk_log_size":4009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:19.831763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262733824,"num_objects":4009,"num_object_clones":0,"num_object_copies":12027,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4009,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4009,"num_write_kb":256576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"75'28","reported_seq":285,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-09-17T07:08:40.622448+0000","last_change":"2024-09-17T07:08:26.795763+0000","last_active":"2024-09-17T07:08:40.622448+0000","last_peered":"2024-09-17T07:08:40.622448+0000","last_clean":"2024-09-17T07:08:40.622448+0000","last_became_active":"2024-09-17T07:02:14.017377+0000","last_became_peered":"2024-09-17T07:02:14.017377+0000","last_unstale":"2024-09-17T07:08:40.622448+0000","last_undegraded":"2024-09-17T07:08:40.622448+0000","last_fullsized":"2024-09-17T07:08:40.622448+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'28","last_scrub_stamp":"2024-09-17T07:08:26.795647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:08:26.795647+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:44.549954+0000","scrub_duration":0.0043656659999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":160,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.158222+0000","last_change":"2024-09-17T07:06:43.763211+0000","last_active":"2024-09-17T07:08:38.158222+0000","last_peered":"2024-09-17T07:08:38.158222+0000","last_clean":"2024-09-17T07:08:38.158222+0000","last_became_active":"2024-09-17T07:02:12.008802+0000","last_became_peered":"2024-09-17T07:02:12.008802+0000","last_unstale":"2024-09-17T07:08:38.158222+0000","last_undegraded":"2024-09-17T07:08:38.158222+0000","last_fullsized":"2024-09-17T07:08:38.158222+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:43.763044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:06:43.763044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003861303,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":157,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.993590+0000","last_change":"2024-09-17T07:08:04.266668+0000","last_active":"2024-09-17T07:08:38.993590+0000","last_peered":"2024-09-17T07:08:38.993590+0000","last_clean":"2024-09-17T07:08:38.993590+0000","last_became_active":"2024-09-17T07:02:16.016606+0000","last_became_peered":"2024-09-17T07:02:16.016606+0000","last_unstale":"2024-09-17T07:08:38.993590+0000","last_undegraded":"2024-09-17T07:08:38.993590+0000","last_fullsized":"2024-09-17T07:08:38.993590+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:04.266574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:08:04.266574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:04.575911+0000","scrub_duration":0.0014388140000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":143,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-09-17T07:08:40.125694+0000","last_change":"2024-09-17T07:08:08.849222+0000","last_active":"2024-09-17T07:08:40.125694+0000","last_peered":"2024-09-17T07:08:40.125694+0000","last_clean":"2024-09-17T07:08:40.125694+0000","last_became_active":"2024-09-17T07:02:18.024602+0000","last_became_peered":"2024-09-17T07:02:18.024602+0000","last_unstale":"2024-09-17T07:08:40.125694+0000","last_undegraded":"2024-09-17T07:08:40.125694+0000","last_fullsized":"2024-09-17T07:08:40.125694+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:08.849142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:08:08.849142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:09.331821+0000","scrub_duration":0.0014278870000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.b","version":"183'4181","reported_seq":4205,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-09-17T07:08:48.626698+0000","last_change":"2024-09-17T07:08:00.243217+0000","last_active":"2024-09-17T07:08:48.626698+0000","last_peered":"2024-09-17T07:08:48.626698+0000","last_clean":"2024-09-17T07:08:48.626698+0000","last_became_active":"2024-09-17T07:08:00.242842+0000","last_became_peered":"2024-09-17T07:08:00.242842+0000","last_unstale":"2024-09-17T07:08:48.626698+0000","last_undegraded":"2024-09-17T07:08:48.626698+0000","last_fullsized":"2024-09-17T07:08:48.626698+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_clean_scrub_stamp":"2024-09-17T07:07:59.220681+0000","objects_scrubbed":0,"log_size":4181,"log_dups_size":0,"ondisk_log_size":4181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:03.017656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":274006016,"num_objects":4181,"num_object_clones":0,"num_object_copies":12543,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4181,"num_write_kb":267584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"75'13","reported_seq":185,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-09-17T07:08:40.123430+0000","last_change":"2024-09-17T07:08:05.678169+0000","last_active":"2024-09-17T07:08:40.123430+0000","last_peered":"2024-09-17T07:08:40.123430+0000","last_clean":"2024-09-17T07:08:40.123430+0000","last_became_active":"2024-09-17T07:02:14.019838+0000","last_became_peered":"2024-09-17T07:02:14.019838+0000","last_unstale":"2024-09-17T07:08:40.123430+0000","last_undegraded":"2024-09-17T07:08:40.123430+0000","last_fullsized":"2024-09-17T07:08:40.123430+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'13","last_scrub_stamp":"2024-09-17T07:08:05.678063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:08:05.678063+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:17.545005+0000","scrub_duration":0.0024636459999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"76'5","reported_seq":557,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-09-17T07:08:46.715103+0000","last_change":"2024-09-17T07:08:29.818116+0000","last_active":"2024-09-17T07:08:46.715103+0000","last_peered":"2024-09-17T07:08:46.715103+0000","last_clean":"2024-09-17T07:08:46.715103+0000","last_became_active":"2024-09-17T07:02:12.006883+0000","last_became_peered":"2024-09-17T07:02:12.006883+0000","last_unstale":"2024-09-17T07:08:46.715103+0000","last_undegraded":"2024-09-17T07:08:46.715103+0000","last_fullsized":"2024-09-17T07:08:46.715103+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'5","last_scrub_stamp":"2024-09-17T07:08:29.817785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:08:29.817785+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:45.372335+0000","scrub_duration":0.0034070260000000001,"objects_trimmed":0,"snaptrim_duration":0.00033739799999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":389,"num_read_kb":384,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":121,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.158445+0000","last_change":"2024-09-17T07:08:23.833769+0000","last_active":"2024-09-17T07:08:38.158445+0000","last_peered":"2024-09-17T07:08:38.158445+0000","last_clean":"2024-09-17T07:08:38.158445+0000","last_became_active":"2024-09-17T07:04:15.109950+0000","last_became_peered":"2024-09-17T07:04:15.109950+0000","last_unstale":"2024-09-17T07:08:38.158445+0000","last_undegraded":"2024-09-17T07:08:38.158445+0000","last_fullsized":"2024-09-17T07:08:38.158445+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:23.833170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:23.833170+0000","last_clean_scrub_stamp":"2024-09-17T07:08:23.833170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:53.203011+0000","scrub_duration":0.0046351129999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":157,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-09-17T07:08:40.116534+0000","last_change":"2024-09-17T07:08:14.935764+0000","last_active":"2024-09-17T07:08:40.116534+0000","last_peered":"2024-09-17T07:08:40.116534+0000","last_clean":"2024-09-17T07:08:40.116534+0000","last_became_active":"2024-09-17T07:02:16.014492+0000","last_became_peered":"2024-09-17T07:02:16.014492+0000","last_unstale":"2024-09-17T07:08:40.116534+0000","last_undegraded":"2024-09-17T07:08:40.116534+0000","last_fullsized":"2024-09-17T07:08:40.116534+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:14.935679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:08:14.935679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:37.535235+0000","scrub_duration":0.001590459,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":155,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.244304+0000","last_change":"2024-09-17T07:08:17.127147+0000","last_active":"2024-09-17T07:08:38.244304+0000","last_peered":"2024-09-17T07:08:38.244304+0000","last_clean":"2024-09-17T07:08:38.244304+0000","last_became_active":"2024-09-17T07:02:18.365369+0000","last_became_peered":"2024-09-17T07:02:18.365369+0000","last_unstale":"2024-09-17T07:08:38.244304+0000","last_undegraded":"2024-09-17T07:08:38.244304+0000","last_fullsized":"2024-09-17T07:08:38.244304+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:17.127034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:08:17.127034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:20.871917+0000","scrub_duration":0.002677081,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"183'3923","reported_seq":3946,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-09-17T07:08:48.189264+0000","last_change":"2024-09-17T07:08:00.760107+0000","last_active":"2024-09-17T07:08:48.189264+0000","last_peered":"2024-09-17T07:08:48.189264+0000","last_clean":"2024-09-17T07:08:48.189264+0000","last_became_active":"2024-09-17T07:08:00.759827+0000","last_became_peered":"2024-09-17T07:08:00.759827+0000","last_unstale":"2024-09-17T07:08:48.189264+0000","last_undegraded":"2024-09-17T07:08:48.189264+0000","last_fullsized":"2024-09-17T07:08:48.189264+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_clean_scrub_stamp":"2024-09-17T07:07:59.220681+0000","objects_scrubbed":0,"log_size":3923,"log_dups_size":0,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:16.600109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":257097728,"num_objects":3923,"num_object_clones":0,"num_object_copies":11769,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":251072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"75'12","reported_seq":200,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-09-17T07:08:40.128938+0000","last_change":"2024-09-17T07:08:08.295553+0000","last_active":"2024-09-17T07:08:40.128938+0000","last_peered":"2024-09-17T07:08:40.128938+0000","last_clean":"2024-09-17T07:08:40.128938+0000","last_became_active":"2024-09-17T07:03:48.760405+0000","last_became_peered":"2024-09-17T07:03:48.760405+0000","last_unstale":"2024-09-17T07:08:40.128938+0000","last_undegraded":"2024-09-17T07:08:40.128938+0000","last_fullsized":"2024-09-17T07:08:40.128938+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-09-17T07:08:08.295449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:08:08.295449+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:37.355090+0000","scrub_duration":0.0023245919999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"70'1","reported_seq":156,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.159231+0000","last_change":"2024-09-17T07:07:21.050377+0000","last_active":"2024-09-17T07:08:38.159231+0000","last_peered":"2024-09-17T07:08:38.159231+0000","last_clean":"2024-09-17T07:08:38.159231+0000","last_became_active":"2024-09-17T07:02:12.009476+0000","last_became_peered":"2024-09-17T07:02:12.009476+0000","last_unstale":"2024-09-17T07:08:38.159231+0000","last_undegraded":"2024-09-17T07:08:38.159231+0000","last_fullsized":"2024-09-17T07:08:38.159231+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-17T07:07:21.050247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:07:21.050247+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.005277505,"objects_trimmed":0,"snaptrim_duration":0.0032298629999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":150,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.159074+0000","last_change":"2024-09-17T07:08:04.086805+0000","last_active":"2024-09-17T07:08:38.159074+0000","last_peered":"2024-09-17T07:08:38.159074+0000","last_clean":"2024-09-17T07:08:38.159074+0000","last_became_active":"2024-09-17T07:02:12.014623+0000","last_became_peered":"2024-09-17T07:02:12.014623+0000","last_unstale":"2024-09-17T07:08:38.159074+0000","last_undegraded":"2024-09-17T07:08:38.159074+0000","last_fullsized":"2024-09-17T07:08:38.159074+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:04.086709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:08:04.086709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:14.260720+0000","scrub_duration":0.0017330640000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":156,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-09-17T07:08:40.121307+0000","last_change":"2024-09-17T07:08:18.008734+0000","last_active":"2024-09-17T07:08:40.121307+0000","last_peered":"2024-09-17T07:08:40.121307+0000","last_clean":"2024-09-17T07:08:40.121307+0000","last_became_active":"2024-09-17T07:02:16.021561+0000","last_became_peered":"2024-09-17T07:02:16.021561+0000","last_unstale":"2024-09-17T07:08:40.121307+0000","last_undegraded":"2024-09-17T07:08:40.121307+0000","last_fullsized":"2024-09-17T07:08:40.121307+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:18.008600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:08:18.008600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:38.354418+0000","scrub_duration":0.0044198520000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":143,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.159338+0000","last_change":"2024-09-17T07:07:25.570226+0000","last_active":"2024-09-17T07:08:38.159338+0000","last_peered":"2024-09-17T07:08:38.159338+0000","last_clean":"2024-09-17T07:08:38.159338+0000","last_became_active":"2024-09-17T07:02:18.022082+0000","last_became_peered":"2024-09-17T07:02:18.022082+0000","last_unstale":"2024-09-17T07:08:38.159338+0000","last_undegraded":"2024-09-17T07:08:38.159338+0000","last_fullsized":"2024-09-17T07:08:38.159338+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:25.570107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:07:25.570107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0049089310000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"183'3903","reported_seq":3927,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-09-17T07:08:48.113873+0000","last_change":"2024-09-17T07:08:00.243111+0000","last_active":"2024-09-17T07:08:48.113873+0000","last_peered":"2024-09-17T07:08:48.113873+0000","last_clean":"2024-09-17T07:08:48.113873+0000","last_became_active":"2024-09-17T07:08:00.242803+0000","last_became_peered":"2024-09-17T07:08:00.242803+0000","last_unstale":"2024-09-17T07:08:48.113873+0000","last_undegraded":"2024-09-17T07:08:48.113873+0000","last_fullsized":"2024-09-17T07:08:48.113873+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_clean_scrub_stamp":"2024-09-17T07:07:59.220681+0000","objects_scrubbed":0,"log_size":3903,"log_dups_size":0,"ondisk_log_size":3903,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:17.233293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":255787008,"num_objects":3903,"num_object_clones":0,"num_object_copies":11709,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3903,"num_write_kb":249792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"75'14","reported_seq":181,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.165864+0000","last_change":"2024-09-17T07:08:28.485668+0000","last_active":"2024-09-17T07:08:38.165864+0000","last_peered":"2024-09-17T07:08:38.165864+0000","last_clean":"2024-09-17T07:08:38.165864+0000","last_became_active":"2024-09-17T07:02:14.028857+0000","last_became_peered":"2024-09-17T07:02:14.028857+0000","last_unstale":"2024-09-17T07:08:38.165864+0000","last_undegraded":"2024-09-17T07:08:38.165864+0000","last_fullsized":"2024-09-17T07:08:38.165864+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'14","last_scrub_stamp":"2024-09-17T07:08:28.485338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:08:28.485338+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:54.485230+0000","scrub_duration":0.0037222499999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"72'2","reported_seq":167,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.159200+0000","last_change":"2024-09-17T07:08:15.763809+0000","last_active":"2024-09-17T07:08:38.159200+0000","last_peered":"2024-09-17T07:08:38.159200+0000","last_clean":"2024-09-17T07:08:38.159200+0000","last_became_active":"2024-09-17T07:02:12.005127+0000","last_became_peered":"2024-09-17T07:02:12.005127+0000","last_unstale":"2024-09-17T07:08:38.159200+0000","last_undegraded":"2024-09-17T07:08:38.159200+0000","last_fullsized":"2024-09-17T07:08:38.159200+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'2","last_scrub_stamp":"2024-09-17T07:08:15.763727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:08:15.763727+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:28.540387+0000","scrub_duration":0.0016958979999999999,"objects_trimmed":0,"snaptrim_duration":0.0010980219999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":160,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.155730+0000","last_change":"2024-09-17T07:06:37.732110+0000","last_active":"2024-09-17T07:08:38.155730+0000","last_peered":"2024-09-17T07:08:38.155730+0000","last_clean":"2024-09-17T07:08:38.155730+0000","last_became_active":"2024-09-17T07:02:12.004606+0000","last_became_peered":"2024-09-17T07:02:12.004606+0000","last_unstale":"2024-09-17T07:08:38.155730+0000","last_undegraded":"2024-09-17T07:08:38.155730+0000","last_fullsized":"2024-09-17T07:08:38.155730+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:37.731844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:06:37.731844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0070683819999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"78'128","reported_seq":409,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.159173+0000","last_change":"2024-09-17T07:08:14.128641+0000","last_active":"2024-09-17T07:08:38.159173+0000","last_peered":"2024-09-17T07:08:38.159173+0000","last_clean":"2024-09-17T07:08:38.159173+0000","last_became_active":"2024-09-17T07:02:03.020675+0000","last_became_peered":"2024-09-17T07:02:03.020675+0000","last_unstale":"2024-09-17T07:08:38.159173+0000","last_undegraded":"2024-09-17T07:08:38.159173+0000","last_fullsized":"2024-09-17T07:08:38.159173+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'128","last_scrub_stamp":"2024-09-17T07:08:14.128519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:00:09.390381+0000","last_clean_scrub_stamp":"2024-09-17T07:08:14.128519+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:29.403199+0000","scrub_duration":0.0021141889999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":157,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.159048+0000","last_change":"2024-09-17T07:08:05.064380+0000","last_active":"2024-09-17T07:08:38.159048+0000","last_peered":"2024-09-17T07:08:38.159048+0000","last_clean":"2024-09-17T07:08:38.159048+0000","last_became_active":"2024-09-17T07:02:16.016031+0000","last_became_peered":"2024-09-17T07:02:16.016031+0000","last_unstale":"2024-09-17T07:08:38.159048+0000","last_undegraded":"2024-09-17T07:08:38.159048+0000","last_fullsized":"2024-09-17T07:08:38.159048+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:05.064241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:08:05.064241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:28.339433+0000","scrub_duration":0.0015017990000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":143,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-09-17T07:08:40.121449+0000","last_change":"2024-09-17T07:06:30.808658+0000","last_active":"2024-09-17T07:08:40.121449+0000","last_peered":"2024-09-17T07:08:40.121449+0000","last_clean":"2024-09-17T07:08:40.121449+0000","last_became_active":"2024-09-17T07:02:18.023321+0000","last_became_peered":"2024-09-17T07:02:18.023321+0000","last_unstale":"2024-09-17T07:08:40.121449+0000","last_undegraded":"2024-09-17T07:08:40.121449+0000","last_fullsized":"2024-09-17T07:08:40.121449+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:06:30.808562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:58.670571+0000","last_clean_scrub_stamp":"2024-09-17T07:06:30.808562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.005261338,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"183'4219","reported_seq":4243,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-09-17T07:08:48.990193+0000","last_change":"2024-09-17T07:08:00.599954+0000","last_active":"2024-09-17T07:08:48.990193+0000","last_peered":"2024-09-17T07:08:48.990193+0000","last_clean":"2024-09-17T07:08:48.990193+0000","last_became_active":"2024-09-17T07:08:00.599685+0000","last_became_peered":"2024-09-17T07:08:00.599685+0000","last_unstale":"2024-09-17T07:08:48.990193+0000","last_undegraded":"2024-09-17T07:08:48.990193+0000","last_fullsized":"2024-09-17T07:08:48.990193+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_clean_scrub_stamp":"2024-09-17T07:07:59.220681+0000","objects_scrubbed":0,"log_size":4219,"log_dups_size":0,"ondisk_log_size":4219,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:08.732487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276496384,"num_objects":4219,"num_object_clones":0,"num_object_copies":12657,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4219,"num_write_kb":270016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"75'19","reported_seq":179,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-09-17T07:08:40.121905+0000","last_change":"2024-09-17T07:07:18.384022+0000","last_active":"2024-09-17T07:08:40.121905+0000","last_peered":"2024-09-17T07:08:40.121905+0000","last_clean":"2024-09-17T07:08:40.121905+0000","last_became_active":"2024-09-17T07:02:14.027313+0000","last_became_peered":"2024-09-17T07:02:14.027313+0000","last_unstale":"2024-09-17T07:08:40.121905+0000","last_undegraded":"2024-09-17T07:08:40.121905+0000","last_fullsized":"2024-09-17T07:08:40.121905+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-09-17T07:07:18.383813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:07:18.383813+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040351470000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":171,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.155042+0000","last_change":"2024-09-17T07:08:15.952173+0000","last_active":"2024-09-17T07:08:38.155042+0000","last_peered":"2024-09-17T07:08:38.155042+0000","last_clean":"2024-09-17T07:08:38.155042+0000","last_became_active":"2024-09-17T07:02:12.012037+0000","last_became_peered":"2024-09-17T07:02:12.012037+0000","last_unstale":"2024-09-17T07:08:38.155042+0000","last_undegraded":"2024-09-17T07:08:38.155042+0000","last_fullsized":"2024-09-17T07:08:38.155042+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:15.952078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:06:39.322535+0000","last_clean_scrub_stamp":"2024-09-17T07:08:15.952078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:25.194960+0000","scrub_duration":0.0017662470000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":167,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.153450+0000","last_change":"2024-09-17T07:08:17.003265+0000","last_active":"2024-09-17T07:08:38.153450+0000","last_peered":"2024-09-17T07:08:38.153450+0000","last_clean":"2024-09-17T07:08:38.153450+0000","last_became_active":"2024-09-17T07:02:16.014631+0000","last_became_peered":"2024-09-17T07:02:16.014631+0000","last_unstale":"2024-09-17T07:08:38.153450+0000","last_undegraded":"2024-09-17T07:08:38.153450+0000","last_fullsized":"2024-09-17T07:08:38.153450+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:17.003116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:08:17.003116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:32.776213+0000","scrub_duration":0.0035024880000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":143,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.159175+0000","last_change":"2024-09-17T07:07:27.004939+0000","last_active":"2024-09-17T07:08:38.159175+0000","last_peered":"2024-09-17T07:08:38.159175+0000","last_clean":"2024-09-17T07:08:38.159175+0000","last_became_active":"2024-09-17T07:02:18.023299+0000","last_became_peered":"2024-09-17T07:02:18.023299+0000","last_unstale":"2024-09-17T07:08:38.159175+0000","last_undegraded":"2024-09-17T07:08:38.159175+0000","last_fullsized":"2024-09-17T07:08:38.159175+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:27.004683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:07:27.004683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010022120000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.7","version":"183'4058","reported_seq":4082,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-09-17T07:08:48.167092+0000","last_change":"2024-09-17T07:08:00.245682+0000","last_active":"2024-09-17T07:08:48.167092+0000","last_peered":"2024-09-17T07:08:48.167092+0000","last_clean":"2024-09-17T07:08:48.167092+0000","last_became_active":"2024-09-17T07:08:00.245345+0000","last_became_peered":"2024-09-17T07:08:00.245345+0000","last_unstale":"2024-09-17T07:08:48.167092+0000","last_undegraded":"2024-09-17T07:08:48.167092+0000","last_fullsized":"2024-09-17T07:08:48.167092+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_clean_scrub_stamp":"2024-09-17T07:07:59.220681+0000","objects_scrubbed":0,"log_size":4058,"log_dups_size":0,"ondisk_log_size":4058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:26.022804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":265945088,"num_objects":4058,"num_object_clones":0,"num_object_copies":12174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4058,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4058,"num_write_kb":259712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"75'9","reported_seq":174,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.158074+0000","last_change":"2024-09-17T07:08:30.220173+0000","last_active":"2024-09-17T07:08:38.158074+0000","last_peered":"2024-09-17T07:08:38.158074+0000","last_clean":"2024-09-17T07:08:38.158074+0000","last_became_active":"2024-09-17T07:02:14.014939+0000","last_became_peered":"2024-09-17T07:02:14.014939+0000","last_unstale":"2024-09-17T07:08:38.158074+0000","last_undegraded":"2024-09-17T07:08:38.158074+0000","last_fullsized":"2024-09-17T07:08:38.158074+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:08:30.219974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:08:30.219974+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:38.511826+0000","scrub_duration":0.0033974780000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":159,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.160908+0000","last_change":"2024-09-17T07:08:20.806586+0000","last_active":"2024-09-17T07:08:38.160908+0000","last_peered":"2024-09-17T07:08:38.160908+0000","last_clean":"2024-09-17T07:08:38.160908+0000","last_became_active":"2024-09-17T07:02:12.000005+0000","last_became_peered":"2024-09-17T07:02:12.000005+0000","last_unstale":"2024-09-17T07:08:38.160908+0000","last_undegraded":"2024-09-17T07:08:38.160908+0000","last_fullsized":"2024-09-17T07:08:38.160908+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:20.806468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:06:30.038104+0000","last_clean_scrub_stamp":"2024-09-17T07:08:20.806468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:44.770744+0000","scrub_duration":0.005853245,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":111,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.151861+0000","last_change":"2024-09-17T07:07:19.789263+0000","last_active":"2024-09-17T07:08:38.151861+0000","last_peered":"2024-09-17T07:08:38.151861+0000","last_clean":"2024-09-17T07:08:38.151861+0000","last_became_active":"2024-09-17T07:04:15.107886+0000","last_became_peered":"2024-09-17T07:04:15.107886+0000","last_unstale":"2024-09-17T07:08:38.151861+0000","last_undegraded":"2024-09-17T07:08:38.151861+0000","last_fullsized":"2024-09-17T07:08:38.151861+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:19.789134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:07:19.789134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010505654,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":143,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.159154+0000","last_change":"2024-09-17T07:07:20.025023+0000","last_active":"2024-09-17T07:08:38.159154+0000","last_peered":"2024-09-17T07:08:38.159154+0000","last_clean":"2024-09-17T07:08:38.159154+0000","last_became_active":"2024-09-17T07:02:18.026994+0000","last_became_peered":"2024-09-17T07:02:18.026994+0000","last_unstale":"2024-09-17T07:08:38.159154+0000","last_undegraded":"2024-09-17T07:08:38.159154+0000","last_fullsized":"2024-09-17T07:08:38.159154+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:20.024902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:58.099849+0000","last_clean_scrub_stamp":"2024-09-17T07:07:20.024902+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.013970381,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"183'4132","reported_seq":4156,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-09-17T07:08:48.048920+0000","last_change":"2024-09-17T07:08:00.760307+0000","last_active":"2024-09-17T07:08:48.048920+0000","last_peered":"2024-09-17T07:08:48.048920+0000","last_clean":"2024-09-17T07:08:48.048920+0000","last_became_active":"2024-09-17T07:08:00.759662+0000","last_became_peered":"2024-09-17T07:08:00.759662+0000","last_unstale":"2024-09-17T07:08:48.048920+0000","last_undegraded":"2024-09-17T07:08:48.048920+0000","last_fullsized":"2024-09-17T07:08:48.048920+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_clean_scrub_stamp":"2024-09-17T07:07:59.220681+0000","objects_scrubbed":0,"log_size":4132,"log_dups_size":0,"ondisk_log_size":4132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:17.112326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":270794752,"num_objects":4132,"num_object_clones":0,"num_object_copies":12396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4132,"num_write_kb":264448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"111'22","reported_seq":190,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-09-17T07:08:40.120961+0000","last_change":"2024-09-17T07:07:29.816377+0000","last_active":"2024-09-17T07:08:40.120961+0000","last_peered":"2024-09-17T07:08:40.120961+0000","last_clean":"2024-09-17T07:08:40.120961+0000","last_became_active":"2024-09-17T07:02:14.028392+0000","last_became_peered":"2024-09-17T07:02:14.028392+0000","last_unstale":"2024-09-17T07:08:40.120961+0000","last_undegraded":"2024-09-17T07:08:40.120961+0000","last_fullsized":"2024-09-17T07:08:40.120961+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'22","last_scrub_stamp":"2024-09-17T07:07:29.816241+0000","last_deep_scrub":"111'22","last_deep_scrub_stamp":"2024-09-17T07:04:57.500450+0000","last_clean_scrub_stamp":"2024-09-17T07:07:29.816241+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:53.480974+0000","scrub_duration":0.0034677570000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'2","reported_seq":150,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-09-17T07:08:40.125880+0000","last_change":"2024-09-17T07:06:39.828818+0000","last_active":"2024-09-17T07:08:40.125880+0000","last_peered":"2024-09-17T07:08:40.125880+0000","last_clean":"2024-09-17T07:08:40.125880+0000","last_became_active":"2024-09-17T07:04:15.110085+0000","last_became_peered":"2024-09-17T07:04:15.110085+0000","last_unstale":"2024-09-17T07:08:40.125880+0000","last_undegraded":"2024-09-17T07:08:40.125880+0000","last_fullsized":"2024-09-17T07:08:40.125880+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-09-17T07:06:39.828638+0000","last_deep_scrub":"89'2","last_deep_scrub_stamp":"2024-09-17T07:03:26.694366+0000","last_clean_scrub_stamp":"2024-09-17T07:06:39.828638+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0068983610000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":420,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"75'8","reported_seq":247,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-09-17T07:08:45.592847+0000","last_change":"2024-09-17T07:08:20.045908+0000","last_active":"2024-09-17T07:08:45.592847+0000","last_peered":"2024-09-17T07:08:45.592847+0000","last_clean":"2024-09-17T07:08:45.592847+0000","last_became_active":"2024-09-17T07:02:16.015418+0000","last_became_peered":"2024-09-17T07:02:16.015418+0000","last_unstale":"2024-09-17T07:08:45.592847+0000","last_undegraded":"2024-09-17T07:08:45.592847+0000","last_fullsized":"2024-09-17T07:08:45.592847+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:08:20.045781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:08:20.045781+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:45.677411+0000","scrub_duration":0.0023108299999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":153,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.155318+0000","last_change":"2024-09-17T07:07:19.786753+0000","last_active":"2024-09-17T07:08:38.155318+0000","last_peered":"2024-09-17T07:08:38.155318+0000","last_clean":"2024-09-17T07:08:38.155318+0000","last_became_active":"2024-09-17T07:02:18.363253+0000","last_became_peered":"2024-09-17T07:02:18.363253+0000","last_unstale":"2024-09-17T07:08:38.155318+0000","last_undegraded":"2024-09-17T07:08:38.155318+0000","last_fullsized":"2024-09-17T07:08:38.155318+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:19.786442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:07:19.786442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:41.487701+0000","scrub_duration":0.011747911999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.5","version":"183'4171","reported_seq":4195,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-09-17T07:08:48.124105+0000","last_change":"2024-09-17T07:08:00.598660+0000","last_active":"2024-09-17T07:08:48.124105+0000","last_peered":"2024-09-17T07:08:48.124105+0000","last_clean":"2024-09-17T07:08:48.124105+0000","last_became_active":"2024-09-17T07:08:00.598353+0000","last_became_peered":"2024-09-17T07:08:00.598353+0000","last_unstale":"2024-09-17T07:08:48.124105+0000","last_undegraded":"2024-09-17T07:08:48.124105+0000","last_fullsized":"2024-09-17T07:08:48.124105+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_clean_scrub_stamp":"2024-09-17T07:07:59.220681+0000","objects_scrubbed":0,"log_size":4171,"log_dups_size":0,"ondisk_log_size":4171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:23.902877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":273350656,"num_objects":4171,"num_object_clones":0,"num_object_copies":12513,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4171,"num_write_kb":266944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"150'17","reported_seq":187,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-09-17T07:08:40.116865+0000","last_change":"2024-09-17T07:08:34.714341+0000","last_active":"2024-09-17T07:08:40.116865+0000","last_peered":"2024-09-17T07:08:40.116865+0000","last_clean":"2024-09-17T07:08:40.116865+0000","last_became_active":"2024-09-17T07:02:14.010219+0000","last_became_peered":"2024-09-17T07:02:14.010219+0000","last_unstale":"2024-09-17T07:08:40.116865+0000","last_undegraded":"2024-09-17T07:08:40.116865+0000","last_fullsized":"2024-09-17T07:08:40.116865+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'17","last_scrub_stamp":"2024-09-17T07:08:34.714204+0000","last_deep_scrub":"150'17","last_deep_scrub_stamp":"2024-09-17T07:06:38.775788+0000","last_clean_scrub_stamp":"2024-09-17T07:08:34.714204+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:51.870348+0000","scrub_duration":0.0065052799999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"68'2","reported_seq":157,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.150664+0000","last_change":"2024-09-17T07:07:20.764435+0000","last_active":"2024-09-17T07:08:38.150664+0000","last_peered":"2024-09-17T07:08:38.150664+0000","last_clean":"2024-09-17T07:08:38.150664+0000","last_became_active":"2024-09-17T07:02:12.006716+0000","last_became_peered":"2024-09-17T07:02:12.006716+0000","last_unstale":"2024-09-17T07:08:38.150664+0000","last_undegraded":"2024-09-17T07:08:38.150664+0000","last_fullsized":"2024-09-17T07:08:38.150664+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-09-17T07:07:20.764283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:07:20.764283+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:50.410328+0000","scrub_duration":0.0060160040000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":121,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.993501+0000","last_change":"2024-09-17T07:08:10.273572+0000","last_active":"2024-09-17T07:08:38.993501+0000","last_peered":"2024-09-17T07:08:38.993501+0000","last_clean":"2024-09-17T07:08:38.993501+0000","last_became_active":"2024-09-17T07:04:15.111069+0000","last_became_peered":"2024-09-17T07:04:15.111069+0000","last_unstale":"2024-09-17T07:08:38.993501+0000","last_undegraded":"2024-09-17T07:08:38.993501+0000","last_fullsized":"2024-09-17T07:08:38.993501+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:10.273485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:08:10.273485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:11.548372+0000","scrub_duration":0.0013521,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":142,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.158891+0000","last_change":"2024-09-17T07:08:07.878593+0000","last_active":"2024-09-17T07:08:38.158891+0000","last_peered":"2024-09-17T07:08:38.158891+0000","last_clean":"2024-09-17T07:08:38.158891+0000","last_became_active":"2024-09-17T07:02:18.024983+0000","last_became_peered":"2024-09-17T07:02:18.024983+0000","last_unstale":"2024-09-17T07:08:38.158891+0000","last_undegraded":"2024-09-17T07:08:38.158891+0000","last_fullsized":"2024-09-17T07:08:38.158891+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:07.878506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:08:07.878506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:23.059846+0000","scrub_duration":0.001815145,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"75'9","reported_seq":174,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.158574+0000","last_change":"2024-09-17T07:08:19.317165+0000","last_active":"2024-09-17T07:08:38.158574+0000","last_peered":"2024-09-17T07:08:38.158574+0000","last_clean":"2024-09-17T07:08:38.158574+0000","last_became_active":"2024-09-17T07:02:14.029388+0000","last_became_peered":"2024-09-17T07:02:14.029388+0000","last_unstale":"2024-09-17T07:08:38.158574+0000","last_undegraded":"2024-09-17T07:08:38.158574+0000","last_fullsized":"2024-09-17T07:08:38.158574+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:08:19.316248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:08:19.316248+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:33.738658+0000","scrub_duration":0.0059518610000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":171,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.153365+0000","last_change":"2024-09-17T07:08:17.976403+0000","last_active":"2024-09-17T07:08:38.153365+0000","last_peered":"2024-09-17T07:08:38.153365+0000","last_clean":"2024-09-17T07:08:38.153365+0000","last_became_active":"2024-09-17T07:02:12.011327+0000","last_became_peered":"2024-09-17T07:02:12.011327+0000","last_unstale":"2024-09-17T07:08:38.153365+0000","last_undegraded":"2024-09-17T07:08:38.153365+0000","last_fullsized":"2024-09-17T07:08:38.153365+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:17.976267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:17.976267+0000","last_clean_scrub_stamp":"2024-09-17T07:08:17.976267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:43.173493+0000","scrub_duration":0.0035138019999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":155,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.159534+0000","last_change":"2024-09-17T07:08:22.799236+0000","last_active":"2024-09-17T07:08:38.159534+0000","last_peered":"2024-09-17T07:08:38.159534+0000","last_clean":"2024-09-17T07:08:38.159534+0000","last_became_active":"2024-09-17T07:02:16.022584+0000","last_became_peered":"2024-09-17T07:02:16.022584+0000","last_unstale":"2024-09-17T07:08:38.159534+0000","last_undegraded":"2024-09-17T07:08:38.159534+0000","last_fullsized":"2024-09-17T07:08:38.159534+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:22.799056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:08:22.799056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:48.739024+0000","scrub_duration":0.006609101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":165,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.153647+0000","last_change":"2024-09-17T07:08:19.979330+0000","last_active":"2024-09-17T07:08:38.153647+0000","last_peered":"2024-09-17T07:08:38.153647+0000","last_clean":"2024-09-17T07:08:38.153647+0000","last_became_active":"2024-09-17T07:02:18.017825+0000","last_became_peered":"2024-09-17T07:02:18.017825+0000","last_unstale":"2024-09-17T07:08:38.153647+0000","last_undegraded":"2024-09-17T07:08:38.153647+0000","last_fullsized":"2024-09-17T07:08:38.153647+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:19.979173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:08:19.979173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:21.767365+0000","scrub_duration":0.0023321790000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":159,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.163755+0000","last_change":"2024-09-17T07:08:04.480597+0000","last_active":"2024-09-17T07:08:38.163755+0000","last_peered":"2024-09-17T07:08:38.163755+0000","last_clean":"2024-09-17T07:08:38.163755+0000","last_became_active":"2024-09-17T07:02:12.008065+0000","last_became_peered":"2024-09-17T07:02:12.008065+0000","last_unstale":"2024-09-17T07:08:38.163755+0000","last_undegraded":"2024-09-17T07:08:38.163755+0000","last_fullsized":"2024-09-17T07:08:38.163755+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:04.480504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:08:04.480504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:23.678622+0000","scrub_duration":0.0013838889999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"75'6","reported_seq":166,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.159649+0000","last_change":"2024-09-17T07:08:33.821667+0000","last_active":"2024-09-17T07:08:38.159649+0000","last_peered":"2024-09-17T07:08:38.159649+0000","last_clean":"2024-09-17T07:08:38.159649+0000","last_became_active":"2024-09-17T07:02:14.028655+0000","last_became_peered":"2024-09-17T07:02:14.028655+0000","last_unstale":"2024-09-17T07:08:38.159649+0000","last_undegraded":"2024-09-17T07:08:38.159649+0000","last_fullsized":"2024-09-17T07:08:38.159649+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'6","last_scrub_stamp":"2024-09-17T07:08:33.821542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:08:33.821542+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:47.560414+0000","scrub_duration":0.0019623269999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":155,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.155246+0000","last_change":"2024-09-17T07:08:06.932434+0000","last_active":"2024-09-17T07:08:38.155246+0000","last_peered":"2024-09-17T07:08:38.155246+0000","last_clean":"2024-09-17T07:08:38.155246+0000","last_became_active":"2024-09-17T07:02:16.016634+0000","last_became_peered":"2024-09-17T07:02:16.016634+0000","last_unstale":"2024-09-17T07:08:38.155246+0000","last_undegraded":"2024-09-17T07:08:38.155246+0000","last_fullsized":"2024-09-17T07:08:38.155246+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:06.932318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:08:06.932318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:33.508360+0000","scrub_duration":0.0017321260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":154,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.159510+0000","last_change":"2024-09-17T07:08:33.926391+0000","last_active":"2024-09-17T07:08:38.159510+0000","last_peered":"2024-09-17T07:08:38.159510+0000","last_clean":"2024-09-17T07:08:38.159510+0000","last_became_active":"2024-09-17T07:02:18.365419+0000","last_became_peered":"2024-09-17T07:02:18.365419+0000","last_unstale":"2024-09-17T07:08:38.159510+0000","last_undegraded":"2024-09-17T07:08:38.159510+0000","last_fullsized":"2024-09-17T07:08:38.159510+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:33.926286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:08:33.926286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:56.396105+0000","scrub_duration":0.0014019519999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"75'10","reported_seq":172,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.158916+0000","last_change":"2024-09-17T07:08:18.723401+0000","last_active":"2024-09-17T07:08:38.158916+0000","last_peered":"2024-09-17T07:08:38.158916+0000","last_clean":"2024-09-17T07:08:38.158916+0000","last_became_active":"2024-09-17T07:02:14.027499+0000","last_became_peered":"2024-09-17T07:02:14.027499+0000","last_unstale":"2024-09-17T07:08:38.158916+0000","last_undegraded":"2024-09-17T07:08:38.158916+0000","last_fullsized":"2024-09-17T07:08:38.158916+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-09-17T07:08:18.723247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:08:18.723247+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:25.719249+0000","scrub_duration":0.0065449590000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":170,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.158205+0000","last_change":"2024-09-17T07:08:25.066079+0000","last_active":"2024-09-17T07:08:38.158205+0000","last_peered":"2024-09-17T07:08:38.158205+0000","last_clean":"2024-09-17T07:08:38.158205+0000","last_became_active":"2024-09-17T07:02:12.007280+0000","last_became_peered":"2024-09-17T07:02:12.007280+0000","last_unstale":"2024-09-17T07:08:38.158205+0000","last_undegraded":"2024-09-17T07:08:38.158205+0000","last_fullsized":"2024-09-17T07:08:38.158205+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:25.065984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:08:25.065984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:43.700404+0000","scrub_duration":0.0052651929999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"163'9","reported_seq":243,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-09-17T07:08:40.593164+0000","last_change":"2024-09-17T07:07:34.933956+0000","last_active":"2024-09-17T07:08:40.593164+0000","last_peered":"2024-09-17T07:08:40.593164+0000","last_clean":"2024-09-17T07:08:40.593164+0000","last_became_active":"2024-09-17T07:02:16.022583+0000","last_became_peered":"2024-09-17T07:02:16.022583+0000","last_unstale":"2024-09-17T07:08:40.593164+0000","last_undegraded":"2024-09-17T07:08:40.593164+0000","last_fullsized":"2024-09-17T07:08:40.593164+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'9","last_scrub_stamp":"2024-09-17T07:07:34.933808+0000","last_deep_scrub":"163'9","last_deep_scrub_stamp":"2024-09-17T07:07:34.933808+0000","last_clean_scrub_stamp":"2024-09-17T07:07:34.933808+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:57.452885+0000","scrub_duration":0.019504579000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":143,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.158068+0000","last_change":"2024-09-17T07:07:26.014516+0000","last_active":"2024-09-17T07:08:38.158068+0000","last_peered":"2024-09-17T07:08:38.158068+0000","last_clean":"2024-09-17T07:08:38.158068+0000","last_became_active":"2024-09-17T07:02:18.028796+0000","last_became_peered":"2024-09-17T07:02:18.028796+0000","last_unstale":"2024-09-17T07:08:38.158068+0000","last_undegraded":"2024-09-17T07:08:38.158068+0000","last_fullsized":"2024-09-17T07:08:38.158068+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:26.014398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:07:26.014398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0066951459999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"75'9","reported_seq":173,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.158006+0000","last_change":"2024-09-17T07:06:35.727847+0000","last_active":"2024-09-17T07:08:38.158006+0000","last_peered":"2024-09-17T07:08:38.158006+0000","last_clean":"2024-09-17T07:08:38.158006+0000","last_became_active":"2024-09-17T07:02:14.028485+0000","last_became_peered":"2024-09-17T07:02:14.028485+0000","last_unstale":"2024-09-17T07:08:38.158006+0000","last_undegraded":"2024-09-17T07:08:38.158006+0000","last_fullsized":"2024-09-17T07:08:38.158006+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:06:35.727700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:06:35.727700+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045062289999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":160,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.158263+0000","last_change":"2024-09-17T07:08:15.356266+0000","last_active":"2024-09-17T07:08:38.158263+0000","last_peered":"2024-09-17T07:08:38.158263+0000","last_clean":"2024-09-17T07:08:38.158263+0000","last_became_active":"2024-09-17T07:02:12.009935+0000","last_became_peered":"2024-09-17T07:02:12.009935+0000","last_unstale":"2024-09-17T07:08:38.158263+0000","last_undegraded":"2024-09-17T07:08:38.158263+0000","last_fullsized":"2024-09-17T07:08:38.158263+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:15.356187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:08:15.356187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:27.551829+0000","scrub_duration":0.0015925290000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"75'8","reported_seq":231,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-09-17T07:08:40.592554+0000","last_change":"2024-09-17T07:07:32.670103+0000","last_active":"2024-09-17T07:08:40.592554+0000","last_peered":"2024-09-17T07:08:40.592554+0000","last_clean":"2024-09-17T07:08:40.592554+0000","last_became_active":"2024-09-17T07:02:16.016069+0000","last_became_peered":"2024-09-17T07:02:16.016069+0000","last_unstale":"2024-09-17T07:08:40.592554+0000","last_undegraded":"2024-09-17T07:08:40.592554+0000","last_fullsized":"2024-09-17T07:08:40.592554+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:07:32.669921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:07:32.669921+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0099051529999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":154,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.150427+0000","last_change":"2024-09-17T07:08:16.703606+0000","last_active":"2024-09-17T07:08:38.150427+0000","last_peered":"2024-09-17T07:08:38.150427+0000","last_clean":"2024-09-17T07:08:38.150427+0000","last_became_active":"2024-09-17T07:02:18.018716+0000","last_became_peered":"2024-09-17T07:02:18.018716+0000","last_unstale":"2024-09-17T07:08:38.150427+0000","last_undegraded":"2024-09-17T07:08:38.150427+0000","last_fullsized":"2024-09-17T07:08:38.150427+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:16.703532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:08:16.703532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:45.505475+0000","scrub_duration":0.0019164340000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"75'9","reported_seq":173,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.159711+0000","last_change":"2024-09-17T07:08:16.756725+0000","last_active":"2024-09-17T07:08:38.159711+0000","last_peered":"2024-09-17T07:08:38.159711+0000","last_clean":"2024-09-17T07:08:38.159711+0000","last_became_active":"2024-09-17T07:02:14.010289+0000","last_became_peered":"2024-09-17T07:02:14.010289+0000","last_unstale":"2024-09-17T07:08:38.159711+0000","last_undegraded":"2024-09-17T07:08:38.159711+0000","last_fullsized":"2024-09-17T07:08:38.159711+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:08:16.756185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:08:16.756185+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:22.278273+0000","scrub_duration":0.004786584,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":149,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.163690+0000","last_change":"2024-09-17T07:07:20.458487+0000","last_active":"2024-09-17T07:08:38.163690+0000","last_peered":"2024-09-17T07:08:38.163690+0000","last_clean":"2024-09-17T07:08:38.163690+0000","last_became_active":"2024-09-17T07:02:12.007433+0000","last_became_peered":"2024-09-17T07:02:12.007433+0000","last_unstale":"2024-09-17T07:08:38.163690+0000","last_undegraded":"2024-09-17T07:08:38.163690+0000","last_fullsized":"2024-09-17T07:08:38.163690+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:20.458316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:07:20.458316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:42.134938+0000","scrub_duration":0.0043263889999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":144,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.158961+0000","last_change":"2024-09-17T07:07:17.713153+0000","last_active":"2024-09-17T07:08:38.158961+0000","last_peered":"2024-09-17T07:08:38.158961+0000","last_clean":"2024-09-17T07:08:38.158961+0000","last_became_active":"2024-09-17T07:02:16.016216+0000","last_became_peered":"2024-09-17T07:02:16.016216+0000","last_unstale":"2024-09-17T07:08:38.158961+0000","last_undegraded":"2024-09-17T07:08:38.158961+0000","last_fullsized":"2024-09-17T07:08:38.158961+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:17.713027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:07:17.713027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.017242598000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":154,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.157311+0000","last_change":"2024-09-17T07:08:18.338994+0000","last_active":"2024-09-17T07:08:38.157311+0000","last_peered":"2024-09-17T07:08:38.157311+0000","last_clean":"2024-09-17T07:08:38.157311+0000","last_became_active":"2024-09-17T07:02:18.027622+0000","last_became_peered":"2024-09-17T07:02:18.027622+0000","last_unstale":"2024-09-17T07:08:38.157311+0000","last_undegraded":"2024-09-17T07:08:38.157311+0000","last_fullsized":"2024-09-17T07:08:38.157311+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:18.338863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:08:18.338863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:36.839965+0000","scrub_duration":0.0025905709999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"75'11","reported_seq":177,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.153012+0000","last_change":"2024-09-17T07:08:23.583577+0000","last_active":"2024-09-17T07:08:38.153012+0000","last_peered":"2024-09-17T07:08:38.153012+0000","last_clean":"2024-09-17T07:08:38.153012+0000","last_became_active":"2024-09-17T07:02:14.016255+0000","last_became_peered":"2024-09-17T07:02:14.016255+0000","last_unstale":"2024-09-17T07:08:38.153012+0000","last_undegraded":"2024-09-17T07:08:38.153012+0000","last_fullsized":"2024-09-17T07:08:38.153012+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-09-17T07:08:23.583408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:08:23.583408+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:43.889948+0000","scrub_duration":0.0056203690000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"94'2","reported_seq":154,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.160844+0000","last_change":"2024-09-17T07:07:31.718074+0000","last_active":"2024-09-17T07:08:38.160844+0000","last_peered":"2024-09-17T07:08:38.160844+0000","last_clean":"2024-09-17T07:08:38.160844+0000","last_became_active":"2024-09-17T07:02:12.011461+0000","last_became_peered":"2024-09-17T07:02:12.011461+0000","last_unstale":"2024-09-17T07:08:38.160844+0000","last_undegraded":"2024-09-17T07:08:38.160844+0000","last_fullsized":"2024-09-17T07:08:38.160844+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-09-17T07:07:31.717888+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-09-17T07:07:31.717888+0000","last_clean_scrub_stamp":"2024-09-17T07:07:31.717888+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0094559669999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":155,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.159569+0000","last_change":"2024-09-17T07:08:27.841615+0000","last_active":"2024-09-17T07:08:38.159569+0000","last_peered":"2024-09-17T07:08:38.159569+0000","last_clean":"2024-09-17T07:08:38.159569+0000","last_became_active":"2024-09-17T07:02:16.018754+0000","last_became_peered":"2024-09-17T07:02:16.018754+0000","last_unstale":"2024-09-17T07:08:38.159569+0000","last_undegraded":"2024-09-17T07:08:38.159569+0000","last_fullsized":"2024-09-17T07:08:38.159569+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:27.841486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:08:27.841486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:48.023441+0000","scrub_duration":0.0037672679999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":142,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.158101+0000","last_change":"2024-09-17T07:08:10.805778+0000","last_active":"2024-09-17T07:08:38.158101+0000","last_peered":"2024-09-17T07:08:38.158101+0000","last_clean":"2024-09-17T07:08:38.158101+0000","last_became_active":"2024-09-17T07:02:18.027427+0000","last_became_peered":"2024-09-17T07:02:18.027427+0000","last_unstale":"2024-09-17T07:08:38.158101+0000","last_undegraded":"2024-09-17T07:08:38.158101+0000","last_fullsized":"2024-09-17T07:08:38.158101+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:10.805704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:08:10.805704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:17.680378+0000","scrub_duration":0.00179312,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"75'4","reported_seq":167,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.162949+0000","last_change":"2024-09-17T07:06:36.857728+0000","last_active":"2024-09-17T07:08:38.162949+0000","last_peered":"2024-09-17T07:08:38.162949+0000","last_clean":"2024-09-17T07:08:38.162949+0000","last_became_active":"2024-09-17T07:03:48.758220+0000","last_became_peered":"2024-09-17T07:03:48.758220+0000","last_unstale":"2024-09-17T07:08:38.162949+0000","last_undegraded":"2024-09-17T07:08:38.162949+0000","last_fullsized":"2024-09-17T07:08:38.162949+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'4","last_scrub_stamp":"2024-09-17T07:06:36.857471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:06:36.857471+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0095623189999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":159,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.155414+0000","last_change":"2024-09-17T07:08:04.972061+0000","last_active":"2024-09-17T07:08:38.155414+0000","last_peered":"2024-09-17T07:08:38.155414+0000","last_clean":"2024-09-17T07:08:38.155414+0000","last_became_active":"2024-09-17T07:02:12.008898+0000","last_became_peered":"2024-09-17T07:02:12.008898+0000","last_unstale":"2024-09-17T07:08:38.155414+0000","last_undegraded":"2024-09-17T07:08:38.155414+0000","last_fullsized":"2024-09-17T07:08:38.155414+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:04.971954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:08:04.971954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:23.242122+0000","scrub_duration":0.0014792150000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":155,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.163290+0000","last_change":"2024-09-17T07:08:03.496692+0000","last_active":"2024-09-17T07:08:38.163290+0000","last_peered":"2024-09-17T07:08:38.163290+0000","last_clean":"2024-09-17T07:08:38.163290+0000","last_became_active":"2024-09-17T07:02:16.014982+0000","last_became_peered":"2024-09-17T07:02:16.014982+0000","last_unstale":"2024-09-17T07:08:38.163290+0000","last_undegraded":"2024-09-17T07:08:38.163290+0000","last_fullsized":"2024-09-17T07:08:38.163290+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:03.496602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:47.444972+0000","last_clean_scrub_stamp":"2024-09-17T07:08:03.496602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:21.598625+0000","scrub_duration":0.001400943,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":154,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.158464+0000","last_change":"2024-09-17T07:08:26.996645+0000","last_active":"2024-09-17T07:08:38.158464+0000","last_peered":"2024-09-17T07:08:38.158464+0000","last_clean":"2024-09-17T07:08:38.158464+0000","last_became_active":"2024-09-17T07:02:18.021014+0000","last_became_peered":"2024-09-17T07:02:18.021014+0000","last_unstale":"2024-09-17T07:08:38.158464+0000","last_undegraded":"2024-09-17T07:08:38.158464+0000","last_fullsized":"2024-09-17T07:08:38.158464+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:26.996525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:08:26.996525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:38.090266+0000","scrub_duration":0.001722168,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"75'11","reported_seq":176,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.158269+0000","last_change":"2024-09-17T07:08:25.883563+0000","last_active":"2024-09-17T07:08:38.158269+0000","last_peered":"2024-09-17T07:08:38.158269+0000","last_clean":"2024-09-17T07:08:38.158269+0000","last_became_active":"2024-09-17T07:02:14.027316+0000","last_became_peered":"2024-09-17T07:02:14.027316+0000","last_unstale":"2024-09-17T07:08:38.158269+0000","last_undegraded":"2024-09-17T07:08:38.158269+0000","last_fullsized":"2024-09-17T07:08:38.158269+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-09-17T07:08:25.883470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:08:25.883470+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:33.152699+0000","scrub_duration":0.0023329620000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":170,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.165956+0000","last_change":"2024-09-17T07:08:14.519328+0000","last_active":"2024-09-17T07:08:38.165956+0000","last_peered":"2024-09-17T07:08:38.165956+0000","last_clean":"2024-09-17T07:08:38.165956+0000","last_became_active":"2024-09-17T07:02:12.010195+0000","last_became_peered":"2024-09-17T07:02:12.010195+0000","last_unstale":"2024-09-17T07:08:38.165956+0000","last_undegraded":"2024-09-17T07:08:38.165956+0000","last_fullsized":"2024-09-17T07:08:38.165956+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:14.519233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:08:14.519233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:35.668223+0000","scrub_duration":0.001420021,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":156,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.158270+0000","last_change":"2024-09-17T07:08:32.894541+0000","last_active":"2024-09-17T07:08:38.158270+0000","last_peered":"2024-09-17T07:08:38.158270+0000","last_clean":"2024-09-17T07:08:38.158270+0000","last_became_active":"2024-09-17T07:02:16.016183+0000","last_became_peered":"2024-09-17T07:02:16.016183+0000","last_unstale":"2024-09-17T07:08:38.158270+0000","last_undegraded":"2024-09-17T07:08:38.158270+0000","last_fullsized":"2024-09-17T07:08:38.158270+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:32.894433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:08:32.894433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:49.732924+0000","scrub_duration":0.001707264,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":142,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.158814+0000","last_change":"2024-09-17T07:07:26.832785+0000","last_active":"2024-09-17T07:08:38.158814+0000","last_peered":"2024-09-17T07:08:38.158814+0000","last_clean":"2024-09-17T07:08:38.158814+0000","last_became_active":"2024-09-17T07:02:18.365656+0000","last_became_peered":"2024-09-17T07:02:18.365656+0000","last_unstale":"2024-09-17T07:08:38.158814+0000","last_undegraded":"2024-09-17T07:08:38.158814+0000","last_fullsized":"2024-09-17T07:08:38.158814+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:26.832514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:07:26.832514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.01115028,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"75'1","reported_seq":155,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.158967+0000","last_change":"2024-09-17T07:08:22.083583+0000","last_active":"2024-09-17T07:08:38.158967+0000","last_peered":"2024-09-17T07:08:38.158967+0000","last_clean":"2024-09-17T07:08:38.158967+0000","last_became_active":"2024-09-17T07:02:18.021855+0000","last_became_peered":"2024-09-17T07:02:18.021855+0000","last_unstale":"2024-09-17T07:08:38.158967+0000","last_undegraded":"2024-09-17T07:08:38.158967+0000","last_fullsized":"2024-09-17T07:08:38.158967+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-17T07:08:22.083502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:08:22.083502+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:34.543490+0000","scrub_duration":0.0018653739999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":159,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.155877+0000","last_change":"2024-09-17T07:07:22.782091+0000","last_active":"2024-09-17T07:08:38.155877+0000","last_peered":"2024-09-17T07:08:38.155877+0000","last_clean":"2024-09-17T07:08:38.155877+0000","last_became_active":"2024-09-17T07:02:12.005329+0000","last_became_peered":"2024-09-17T07:02:12.005329+0000","last_unstale":"2024-09-17T07:08:38.155877+0000","last_undegraded":"2024-09-17T07:08:38.155877+0000","last_fullsized":"2024-09-17T07:08:38.155877+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:22.781739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:07:22.781739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:44.511386+0000","scrub_duration":0.0037808680000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"75'10","reported_seq":163,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.158492+0000","last_change":"2024-09-17T07:07:22.081513+0000","last_active":"2024-09-17T07:08:38.158492+0000","last_peered":"2024-09-17T07:08:38.158492+0000","last_clean":"2024-09-17T07:08:38.158492+0000","last_became_active":"2024-09-17T07:02:14.028731+0000","last_became_peered":"2024-09-17T07:02:14.028731+0000","last_unstale":"2024-09-17T07:08:38.158492+0000","last_undegraded":"2024-09-17T07:08:38.158492+0000","last_fullsized":"2024-09-17T07:08:38.158492+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-09-17T07:07:22.081382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:07:22.081382+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.056222292,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"75'8","reported_seq":247,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-09-17T07:08:40.593031+0000","last_change":"2024-09-17T07:08:24.422649+0000","last_active":"2024-09-17T07:08:40.593031+0000","last_peered":"2024-09-17T07:08:40.593031+0000","last_clean":"2024-09-17T07:08:40.593031+0000","last_became_active":"2024-09-17T07:02:16.020560+0000","last_became_peered":"2024-09-17T07:02:16.020560+0000","last_unstale":"2024-09-17T07:08:40.593031+0000","last_undegraded":"2024-09-17T07:08:40.593031+0000","last_fullsized":"2024-09-17T07:08:40.593031+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:08:24.422313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:08:24.422313+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:27.347535+0000","scrub_duration":0.0027600989999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":153,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.159322+0000","last_change":"2024-09-17T07:08:34.803052+0000","last_active":"2024-09-17T07:08:38.159322+0000","last_peered":"2024-09-17T07:08:38.159322+0000","last_clean":"2024-09-17T07:08:38.159322+0000","last_became_active":"2024-09-17T07:02:18.021603+0000","last_became_peered":"2024-09-17T07:02:18.021603+0000","last_unstale":"2024-09-17T07:08:38.159322+0000","last_undegraded":"2024-09-17T07:08:38.159322+0000","last_fullsized":"2024-09-17T07:08:38.159322+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:34.802938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:08:34.802938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:45.165231+0000","scrub_duration":0.0022680489999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":159,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.158547+0000","last_change":"2024-09-17T07:08:17.704801+0000","last_active":"2024-09-17T07:08:38.158547+0000","last_peered":"2024-09-17T07:08:38.158547+0000","last_clean":"2024-09-17T07:08:38.158547+0000","last_became_active":"2024-09-17T07:02:12.009837+0000","last_became_peered":"2024-09-17T07:02:12.009837+0000","last_unstale":"2024-09-17T07:08:38.158547+0000","last_undegraded":"2024-09-17T07:08:38.158547+0000","last_fullsized":"2024-09-17T07:08:38.158547+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:17.704687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:08:17.704687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:31.103100+0000","scrub_duration":0.0038132069999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"75'11","reported_seq":209,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.165910+0000","last_change":"2024-09-17T07:08:07.442547+0000","last_active":"2024-09-17T07:08:38.165910+0000","last_peered":"2024-09-17T07:08:38.165910+0000","last_clean":"2024-09-17T07:08:38.165910+0000","last_became_active":"2024-09-17T07:03:48.760390+0000","last_became_peered":"2024-09-17T07:03:48.760390+0000","last_unstale":"2024-09-17T07:08:38.165910+0000","last_undegraded":"2024-09-17T07:08:38.165910+0000","last_fullsized":"2024-09-17T07:08:38.165910+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-09-17T07:08:07.442416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:08:07.442416+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:07.965821+0000","scrub_duration":0.003039281,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":146,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.158417+0000","last_change":"2024-09-17T07:07:31.068035+0000","last_active":"2024-09-17T07:08:38.158417+0000","last_peered":"2024-09-17T07:08:38.158417+0000","last_clean":"2024-09-17T07:08:38.158417+0000","last_became_active":"2024-09-17T07:02:16.019701+0000","last_became_peered":"2024-09-17T07:02:16.019701+0000","last_unstale":"2024-09-17T07:08:38.158417+0000","last_undegraded":"2024-09-17T07:08:38.158417+0000","last_fullsized":"2024-09-17T07:08:38.158417+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:31.067529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:07:31.067529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.013304165,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":144,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.150604+0000","last_change":"2024-09-17T07:07:30.779462+0000","last_active":"2024-09-17T07:08:38.150604+0000","last_peered":"2024-09-17T07:08:38.150604+0000","last_clean":"2024-09-17T07:08:38.150604+0000","last_became_active":"2024-09-17T07:02:18.363752+0000","last_became_peered":"2024-09-17T07:02:18.363752+0000","last_unstale":"2024-09-17T07:08:38.150604+0000","last_undegraded":"2024-09-17T07:08:38.150604+0000","last_fullsized":"2024-09-17T07:08:38.150604+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:30.779382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:07:30.779382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0011291059999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":179,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.163537+0000","last_change":"2024-09-17T07:08:12.462985+0000","last_active":"2024-09-17T07:08:38.163537+0000","last_peered":"2024-09-17T07:08:38.163537+0000","last_clean":"2024-09-17T07:08:38.163537+0000","last_became_active":"2024-09-17T07:03:48.759384+0000","last_became_peered":"2024-09-17T07:03:48.759384+0000","last_unstale":"2024-09-17T07:08:38.163537+0000","last_undegraded":"2024-09-17T07:08:38.163537+0000","last_fullsized":"2024-09-17T07:08:38.163537+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:12.462862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:08:12.462862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:20.330872+0000","scrub_duration":0.0022151050000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"75'15","reported_seq":158,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.159561+0000","last_change":"2024-09-17T07:07:33.775051+0000","last_active":"2024-09-17T07:08:38.159561+0000","last_peered":"2024-09-17T07:08:38.159561+0000","last_clean":"2024-09-17T07:08:38.159561+0000","last_became_active":"2024-09-17T07:04:15.114286+0000","last_became_peered":"2024-09-17T07:04:15.114286+0000","last_unstale":"2024-09-17T07:08:38.159561+0000","last_undegraded":"2024-09-17T07:08:38.159561+0000","last_fullsized":"2024-09-17T07:08:38.159561+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-09-17T07:07:33.774757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:07:33.774757+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0071874820000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":155,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.160668+0000","last_change":"2024-09-17T07:08:25.794548+0000","last_active":"2024-09-17T07:08:38.160668+0000","last_peered":"2024-09-17T07:08:38.160668+0000","last_clean":"2024-09-17T07:08:38.160668+0000","last_became_active":"2024-09-17T07:02:16.015903+0000","last_became_peered":"2024-09-17T07:02:16.015903+0000","last_unstale":"2024-09-17T07:08:38.160668+0000","last_undegraded":"2024-09-17T07:08:38.160668+0000","last_fullsized":"2024-09-17T07:08:38.160668+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:25.794418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:25.794418+0000","last_clean_scrub_stamp":"2024-09-17T07:08:25.794418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:29.296727+0000","scrub_duration":0.002916175,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":143,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.159474+0000","last_change":"2024-09-17T07:08:14.764859+0000","last_active":"2024-09-17T07:08:38.159474+0000","last_peered":"2024-09-17T07:08:38.159474+0000","last_clean":"2024-09-17T07:08:38.159474+0000","last_became_active":"2024-09-17T07:02:18.366466+0000","last_became_peered":"2024-09-17T07:02:18.366466+0000","last_unstale":"2024-09-17T07:08:38.159474+0000","last_undegraded":"2024-09-17T07:08:38.159474+0000","last_fullsized":"2024-09-17T07:08:38.159474+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:14.764781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:08:14.764781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:40.723176+0000","scrub_duration":0.0018717720000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":148,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.159133+0000","last_change":"2024-09-17T07:07:29.761118+0000","last_active":"2024-09-17T07:08:38.159133+0000","last_peered":"2024-09-17T07:08:38.159133+0000","last_clean":"2024-09-17T07:08:38.159133+0000","last_became_active":"2024-09-17T07:02:12.005493+0000","last_became_peered":"2024-09-17T07:02:12.005493+0000","last_unstale":"2024-09-17T07:08:38.159133+0000","last_undegraded":"2024-09-17T07:08:38.159133+0000","last_fullsized":"2024-09-17T07:08:38.159133+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:29.761001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:07:29.761001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:58.064357+0000","scrub_duration":0.0083413440000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"86'17","reported_seq":174,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.160611+0000","last_change":"2024-09-17T07:06:27.962393+0000","last_active":"2024-09-17T07:08:38.160611+0000","last_peered":"2024-09-17T07:08:38.160611+0000","last_clean":"2024-09-17T07:08:38.160611+0000","last_became_active":"2024-09-17T07:02:14.017737+0000","last_became_peered":"2024-09-17T07:02:14.017737+0000","last_unstale":"2024-09-17T07:08:38.160611+0000","last_undegraded":"2024-09-17T07:08:38.160611+0000","last_fullsized":"2024-09-17T07:08:38.160611+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'17","last_scrub_stamp":"2024-09-17T07:06:27.962276+0000","last_deep_scrub":"86'17","last_deep_scrub_stamp":"2024-09-17T07:03:18.241067+0000","last_clean_scrub_stamp":"2024-09-17T07:06:27.962276+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0050727970000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":156,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-17T07:08:38.151776+0000","last_change":"2024-09-17T07:08:14.706829+0000","last_active":"2024-09-17T07:08:38.151776+0000","last_peered":"2024-09-17T07:08:38.151776+0000","last_clean":"2024-09-17T07:08:38.151776+0000","last_became_active":"2024-09-17T07:02:16.017362+0000","last_became_peered":"2024-09-17T07:02:16.017362+0000","last_unstale":"2024-09-17T07:08:38.151776+0000","last_undegraded":"2024-09-17T07:08:38.151776+0000","last_fullsized":"2024-09-17T07:08:38.151776+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:14.706740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:08:14.706740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:19.562882+0000","scrub_duration":0.00164321,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":4306042880,"num_objects":65705,"num_object_clones":0,"num_object_copies":197115,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":65705,"num_write_kb":4205120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12897746944,"data_stored":12897746944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65705,"ondisk_log_size":65705,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67,"ondisk_log_size":67,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":697,"num_read_kb":454,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":426,"ondisk_log_size":426,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":389,"num_read_kb":384,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1386,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":12,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":4158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037845,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1558796,"kb_used_data":1320964,"kb_used_omap":1,"kb_used_meta":237822,"kb_avail":92186356,"statfs":{"total":95995035648,"available":94398828544,"internally_reserved":0,"allocated":1352667136,"data_stored":1352202028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":243530187},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:08:13 2024","interfaces":[{"interface":"back","average":{"1min":3.355,"5min":2.4620000000000002,"15min":1.1739999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":48.493000000000002,"5min":52.478999999999999,"15min":52.478999999999999},"last":0.69399999999999995},{"interface":"front","average":{"1min":3.2959999999999998,"5min":2.4540000000000002,"15min":1.159},"min":{"1min":0.312,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":48.738,"5min":52.871000000000002,"15min":52.871000000000002},"last":1.748}]},{"osd":1,"last update":"Tue Sep 17 07:08:13 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.73599999999999999,"15min":0.58399999999999996},"min":{"1min":0.255,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":1.8480000000000001,"5min":4.5140000000000002,"15min":4.5140000000000002},"last":3.7250000000000001},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.77300000000000002,"15min":0.60699999999999998},"min":{"1min":0.308,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.9650000000000001,"5min":4.5039999999999996,"15min":4.5039999999999996},"last":2.7370000000000001}]},{"osd":2,"last update":"Tue Sep 17 07:08:13 2024","interfaces":[{"interface":"back","average":{"1min":2.0150000000000001,"5min":1.5269999999999999,"15min":0.82799999999999996},"min":{"1min":0.245,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":22.376000000000001,"5min":52.055999999999997,"15min":52.055999999999997},"last":2.6379999999999999},{"interface":"front","average":{"1min":2.2109999999999999,"5min":1.5900000000000001,"15min":0.89100000000000001},"min":{"1min":0.35499999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":22.298999999999999,"5min":52.064999999999998,"15min":52.064999999999998},"last":0.74399999999999999}]},{"osd":3,"last update":"Tue Sep 17 07:08:13 2024","interfaces":[{"interface":"back","average":{"1min":2.0390000000000001,"5min":1.875,"15min":0.97499999999999998},"min":{"1min":0.34599999999999997,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":22.196000000000002,"5min":51.689,"15min":51.689},"last":0.38},{"interface":"front","average":{"1min":2.9100000000000001,"5min":2.306,"15min":1.135},"min":{"1min":0.33800000000000002,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":22.234000000000002,"5min":52.145000000000003,"15min":52.145000000000003},"last":0.71899999999999997}]},{"osd":4,"last update":"Tue Sep 17 07:08:13 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.46200000000000002,"15min":0.45400000000000001},"min":{"1min":0.28799999999999998,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":1.3400000000000001,"5min":1.5980000000000001,"15min":1.5980000000000001},"last":1.7290000000000001},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.51500000000000001,"15min":0.45700000000000002},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.734,"5min":3.4089999999999998,"15min":3.4089999999999998},"last":0.35199999999999998}]},{"osd":5,"last update":"Tue Sep 17 07:08:13 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.51800000000000002,"15min":0.46100000000000002},"min":{"1min":0.28599999999999998,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.8169999999999999,"5min":3.3620000000000001,"15min":3.3620000000000001},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.502,"15min":0.47399999999999998},"min":{"1min":0.36099999999999999,"5min":0.28999999999999998,"15min":0.28000000000000003},"max":{"1min":1.383,"5min":1.6459999999999999,"15min":1.6459999999999999},"last":1.708}]},{"osd":6,"last update":"Tue Sep 17 07:08:13 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.55400000000000005,"15min":0.51500000000000001},"min":{"1min":0.32100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":1.4339999999999999,"5min":1.675,"15min":1.675},"last":1.6759999999999999},{"interface":"front","average":{"1min":0.628,"5min":0.59499999999999997,"15min":0.52300000000000002},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":1.792,"5min":3.2669999999999999,"15min":3.2669999999999999},"last":0.40000000000000002}]}]},{"osd":6,"up_from":50,"seq":214748364888,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1446908,"kb_used_data":1275508,"kb_used_omap":1,"kb_used_meta":171390,"kb_avail":92298244,"statfs":{"total":95995035648,"available":94513401856,"internally_reserved":0,"allocated":1306120192,"data_stored":1305670775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":175503815},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:07:50 2024","interfaces":[{"interface":"back","average":{"1min":2.323,"5min":2.9609999999999999,"15min":1.3200000000000001},"min":{"1min":0.20899999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":16.696999999999999,"5min":75.323999999999998,"15min":75.323999999999998},"last":0.32000000000000001},{"interface":"front","average":{"1min":1.8899999999999999,"5min":2.8279999999999998,"15min":1.27},"min":{"1min":0.312,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":16.483000000000001,"5min":75.168999999999997,"15min":75.168999999999997},"last":2.7050000000000001}]},{"osd":1,"last update":"Tue Sep 17 07:07:50 2024","interfaces":[{"interface":"back","average":{"1min":0.90500000000000003,"5min":0.67300000000000004,"15min":0.55000000000000004},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":3.1850000000000001,"5min":4.4790000000000001,"15min":4.4790000000000001},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.74199999999999999,"15min":0.60299999999999998},"min":{"1min":0.29799999999999999,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":2.7789999999999999,"5min":5.9530000000000003,"15min":5.9530000000000003},"last":0.48199999999999998}]},{"osd":2,"last update":"Tue Sep 17 07:07:50 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":1.0169999999999999,"15min":0.68700000000000006},"min":{"1min":0.30199999999999999,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":3.5800000000000001,"5min":33.079000000000001,"15min":33.079000000000001},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.79900000000000004,"5min":1.0329999999999999,"15min":0.69099999999999995},"min":{"1min":0.25700000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":2.8839999999999999,"5min":33.100000000000001,"15min":33.100000000000001},"last":0.254}]},{"osd":3,"last update":"Tue Sep 17 07:07:50 2024","interfaces":[{"interface":"back","average":{"1min":1.7250000000000001,"5min":1.891,"15min":0.94999999999999996},"min":{"1min":0.27300000000000002,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":16.984000000000002,"5min":52.944000000000003,"15min":52.944000000000003},"last":0.27300000000000002},{"interface":"front","average":{"1min":1.907,"5min":1.843,"15min":0.97399999999999998},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":16.745999999999999,"5min":52.747,"15min":52.747},"last":0.377}]},{"osd":4,"last update":"Tue Sep 17 07:07:50 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.42299999999999999,"15min":0.42099999999999999},"min":{"1min":0.26300000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.054,"5min":1.054,"15min":1.054},"last":0.224},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.51400000000000001,"15min":0.442},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":2.8900000000000001,"5min":2.9119999999999999,"15min":2.9119999999999999},"last":0.23699999999999999}]},{"osd":5,"last update":"Tue Sep 17 07:07:50 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.47699999999999998,"15min":0.44},"min":{"1min":0.28499999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":1.2150000000000001,"5min":1.6419999999999999,"15min":1.6419999999999999},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.53300000000000003,"15min":0.46200000000000002},"min":{"1min":0.28299999999999997,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":3.0379999999999998,"5min":3.0379999999999998,"15min":3.0379999999999998},"last":0.30499999999999999}]},{"osd":7,"last update":"Tue Sep 17 07:08:13 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56799999999999995,"15min":0.51900000000000002},"min":{"1min":0.33200000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.01,"5min":2.859,"15min":2.859},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.58199999999999996,"15min":0.54400000000000004},"min":{"1min":0.33400000000000002,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.1140000000000001,"5min":1.9339999999999999,"15min":1.9339999999999999},"last":0.36599999999999999}]}]},{"osd":5,"up_from":42,"seq":180388626525,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2194364,"kb_used_data":2028724,"kb_used_omap":1,"kb_used_meta":165630,"kb_avail":91550788,"statfs":{"total":95995035648,"available":93748006912,"internally_reserved":0,"allocated":2077413376,"data_stored":2076964225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":169605574},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:08:33 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":2.6459999999999999,"15min":1.224},"min":{"1min":0.28199999999999997,"5min":0.221,"15min":0.221},"max":{"1min":0.72999999999999998,"5min":49.265000000000001,"15min":49.265000000000001},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.58599999999999997,"5min":2.7730000000000001,"15min":1.26},"min":{"1min":0.32700000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":1.996,"5min":47.350000000000001,"15min":47.350000000000001},"last":0.92200000000000004}]},{"osd":1,"last update":"Tue Sep 17 07:08:33 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.81899999999999995,"15min":0.63300000000000001},"min":{"1min":0.311,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":1.8839999999999999,"5min":20.329000000000001,"15min":20.329000000000001},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.998,"15min":1.079},"min":{"1min":0.35299999999999998,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":1.946,"5min":20.408999999999999,"15min":20.408999999999999},"last":1.1140000000000001}]},{"osd":2,"last update":"Tue Sep 17 07:08:33 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":1.1419999999999999,"15min":0.752},"min":{"1min":0.34999999999999998,"5min":0.252,"15min":0.252},"max":{"1min":1.665,"5min":20.486000000000001,"15min":20.486000000000001},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.59199999999999997,"5min":1.137,"15min":0.71799999999999997},"min":{"1min":0.34399999999999997,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":1.8300000000000001,"5min":23.925999999999998,"15min":23.925999999999998},"last":0.76800000000000002}]},{"osd":3,"last update":"Tue Sep 17 07:08:33 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":2.117,"15min":1.0700000000000001},"min":{"1min":0.375,"5min":0.249,"15min":0.249},"max":{"1min":3.3780000000000001,"5min":68.978999999999999,"15min":68.978999999999999},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.71899999999999997,"5min":2.0449999999999999,"15min":1.044},"min":{"1min":0.36699999999999999,"5min":0.253,"15min":0.253},"max":{"1min":2.883,"5min":69.084000000000003,"15min":69.084000000000003},"last":0.66100000000000003}]},{"osd":4,"last update":"Tue Sep 17 07:08:33 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.46000000000000002,"15min":0.41899999999999998},"min":{"1min":0.28000000000000003,"5min":0.22,"15min":0.22},"max":{"1min":1.9199999999999999,"5min":3.2989999999999999,"15min":3.2989999999999999},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.495,"5min":0.45700000000000002,"15min":0.441},"min":{"1min":0.29199999999999998,"5min":0.26700000000000002,"15min":0.248},"max":{"1min":1.506,"5min":2.2160000000000002,"15min":2.2160000000000002},"last":0.48699999999999999}]},{"osd":6,"last update":"Tue Sep 17 07:07:48 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.45900000000000002,"15min":0.437},"min":{"1min":0.23499999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":1.9199999999999999,"5min":1.9199999999999999,"15min":1.9199999999999999},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.51900000000000002,"15min":0.46999999999999997},"min":{"1min":0.31900000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":2.3250000000000002,"5min":2.5590000000000002,"15min":2.5590000000000002},"last":0.82499999999999996}]},{"osd":7,"last update":"Tue Sep 17 07:08:06 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.55000000000000004,"15min":0.504},"min":{"1min":0.308,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":4.0999999999999996,"5min":4.0999999999999996,"15min":4.0999999999999996},"last":0.90000000000000002},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.51800000000000002,"15min":0.48199999999999998},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":3.9100000000000001,"5min":3.9100000000000001,"15min":3.9100000000000001},"last":0.75}]}]},{"osd":4,"up_from":33,"seq":141733920867,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1457416,"kb_used_data":1309824,"kb_used_omap":1,"kb_used_meta":147582,"kb_avail":92287736,"statfs":{"total":95995035648,"available":94502641664,"internally_reserved":0,"allocated":1341259776,"data_stored":1340797651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":151124422},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:07:58 2024","interfaces":[{"interface":"back","average":{"1min":3.4329999999999998,"5min":3.2469999999999999,"15min":1.464},"min":{"1min":0.218,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":58.097999999999999,"5min":74.551000000000002,"15min":74.551000000000002},"last":0.69299999999999995},{"interface":"front","average":{"1min":3.7610000000000001,"5min":3.4860000000000002,"15min":1.5700000000000001},"min":{"1min":0.248,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":58.383000000000003,"5min":74.778999999999996,"15min":74.778999999999996},"last":0.26900000000000002}]},{"osd":1,"last update":"Tue Sep 17 07:07:58 2024","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.67500000000000004,"15min":0.60499999999999998},"min":{"1min":0.27700000000000002,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":3.1360000000000001,"5min":4.1539999999999999,"15min":4.1539999999999999},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.69099999999999995,"15min":0.624},"min":{"1min":0.255,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":3.0379999999999998,"5min":3.5409999999999999,"15min":3.5409999999999999},"last":0.74199999999999999}]},{"osd":2,"last update":"Tue Sep 17 07:07:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":1.258,"15min":0.80600000000000005},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":2.2610000000000001,"5min":49.651000000000003,"15min":49.651000000000003},"last":0.82299999999999995},{"interface":"front","average":{"1min":0.85699999999999998,"5min":1.3759999999999999,"15min":0.84899999999999998},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":3.2589999999999999,"5min":48.813000000000002,"15min":48.813000000000002},"last":0.307}]},{"osd":3,"last update":"Tue Sep 17 07:07:58 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":1.2050000000000001,"15min":0.80500000000000005},"min":{"1min":0.28899999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":2.4199999999999999,"5min":39.042999999999999,"15min":39.042999999999999},"last":0.505},{"interface":"front","average":{"1min":0.82999999999999996,"5min":1.3100000000000001,"15min":0.81200000000000006},"min":{"1min":0.29999999999999999,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":6.157,"5min":38.920000000000002,"15min":38.920000000000002},"last":0.29299999999999998}]},{"osd":5,"last update":"Tue Sep 17 07:08:27 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.501,"15min":0.42599999999999999},"min":{"1min":0.24299999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":1.5649999999999999,"5min":2.4780000000000002,"15min":2.4780000000000002},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.52100000000000002,"15min":0.44700000000000001},"min":{"1min":0.26600000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":2.4980000000000002,"5min":2.5419999999999998,"15min":2.5419999999999998},"last":0.81200000000000006}]},{"osd":6,"last update":"Tue Sep 17 07:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":1.0489999999999999,"15min":0.61399999999999999},"min":{"1min":0.29399999999999998,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.66700000000000004,"5min":47.923999999999999,"15min":47.923999999999999},"last":0.373},{"interface":"front","average":{"1min":0.502,"5min":1.0620000000000001,"15min":0.626},"min":{"1min":0.27800000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.80200000000000005,"5min":47.600999999999999,"15min":47.600999999999999},"last":0.71999999999999997}]},{"osd":7,"last update":"Tue Sep 17 07:08:09 2024","interfaces":[{"interface":"back","average":{"1min":3.0619999999999998,"5min":1.052,"15min":0.68400000000000005},"min":{"1min":0.32400000000000001,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":48.719000000000001,"5min":48.719000000000001,"15min":48.719000000000001},"last":0.35999999999999999},{"interface":"front","average":{"1min":3.069,"5min":1.0489999999999999,"15min":0.66900000000000004},"min":{"1min":0.35099999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":50.527000000000001,"5min":50.527000000000001,"15min":50.527000000000001},"last":0.42399999999999999}]}]},{"osd":3,"up_from":27,"seq":115964117093,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1832352,"kb_used_data":1566616,"kb_used_omap":1,"kb_used_meta":265726,"kb_avail":91912800,"statfs":{"total":95995035648,"available":94118707200,"internally_reserved":0,"allocated":1604214784,"data_stored":1603728867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":272103880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,3,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":1.5600000000000001,"15min":0.76200000000000001},"min":{"1min":0.19700000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":1.431,"5min":17.364000000000001,"15min":17.364000000000001},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.497,"5min":1.4359999999999999,"15min":0.73099999999999998},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":2.105,"5min":18.521000000000001,"15min":18.521000000000001},"last":0.85999999999999999}]},{"osd":1,"last update":"Tue Sep 17 07:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":1.0129999999999999,"15min":0.60599999999999998},"min":{"1min":0.218,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":3.7400000000000002,"5min":8.3300000000000001,"15min":8.3300000000000001},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.59799999999999998,"5min":1.2749999999999999,"15min":0.72199999999999998},"min":{"1min":0.26900000000000002,"5min":0.182,"15min":0.182},"max":{"1min":1.6819999999999999,"5min":8.0670000000000002,"15min":8.0670000000000002},"last":0.64800000000000002}]},{"osd":2,"last update":"Tue Sep 17 07:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":1.579,"15min":0.80500000000000005},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":1.5569999999999999,"5min":9.7319999999999993,"15min":9.7319999999999993},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.45900000000000002,"5min":1.2030000000000001,"15min":0.67900000000000005},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":1.7010000000000001,"5min":8.3350000000000009,"15min":8.3350000000000009},"last":0.88700000000000001}]},{"osd":4,"last update":"Tue Sep 17 07:07:59 2024","interfaces":[{"interface":"back","average":{"1min":0.94999999999999996,"5min":1.05,"15min":0.72499999999999998},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":3.4470000000000001,"5min":5.4539999999999997,"15min":5.4539999999999997},"last":1.327},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.877,"15min":0.64800000000000002},"min":{"1min":0.33100000000000002,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":2.9129999999999998,"5min":5.7850000000000001,"15min":5.7850000000000001},"last":1.4219999999999999}]},{"osd":5,"last update":"Tue Sep 17 07:08:25 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":1.2390000000000001,"15min":0.81499999999999995},"min":{"1min":0.315,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":3.742,"5min":7.9669999999999996,"15min":7.9669999999999996},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.93200000000000005,"5min":1.056,"15min":0.74199999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":3.5899999999999999,"5min":4.4160000000000004,"15min":4.4160000000000004},"last":0.90800000000000003}]},{"osd":6,"last update":"Tue Sep 17 07:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":1.242,"15min":0.79200000000000004},"min":{"1min":0.34300000000000003,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":3.778,"5min":8.2919999999999998,"15min":8.2919999999999998},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.621,"5min":1.226,"15min":0.80200000000000005},"min":{"1min":0.37,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.637,"5min":7.9329999999999998,"15min":7.9329999999999998},"last":0.70899999999999996}]},{"osd":7,"last update":"Tue Sep 17 07:08:06 2024","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":1.143,"15min":0.77300000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":3.4820000000000002,"5min":6.1440000000000001,"15min":6.1440000000000001},"last":0.95299999999999996},{"interface":"front","average":{"1min":0.84799999999999998,"5min":1.125,"15min":0.747},"min":{"1min":0.39500000000000002,"5min":0.38500000000000001,"15min":0.34599999999999997},"max":{"1min":2.9329999999999998,"5min":5.8170000000000002,"15min":5.8170000000000002},"last":0.58799999999999997}]}]},{"osd":2,"up_from":18,"seq":77309411435,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1890624,"kb_used_data":1558456,"kb_used_omap":1,"kb_used_meta":332158,"kb_avail":91854528,"statfs":{"total":95995035648,"available":94059036672,"internally_reserved":0,"allocated":1595858944,"data_stored":1595405544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":340130246},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.995,"15min":0.56399999999999995},"min":{"1min":0.25800000000000001,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":1.51,"5min":7.2709999999999999,"15min":7.2709999999999999},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.55000000000000004,"5min":1.4099999999999999,"15min":0.70499999999999996},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":2.1579999999999999,"5min":8.6869999999999994,"15min":8.6869999999999994},"last":2.9169999999999998}]},{"osd":1,"last update":"Tue Sep 17 07:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.98499999999999999,"15min":0.56299999999999994},"min":{"1min":0.23000000000000001,"5min":0.184,"15min":0.184},"max":{"1min":2.0739999999999998,"5min":5.5010000000000003,"15min":5.5010000000000003},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.89200000000000002,"15min":0.54900000000000004},"min":{"1min":0.23999999999999999,"5min":0.224,"15min":0.224},"max":{"1min":1.804,"5min":6.9829999999999997,"15min":6.9829999999999997},"last":2.2759999999999998}]},{"osd":3,"last update":"Tue Sep 17 07:08:48 2024","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.056,"15min":0.64300000000000002},"min":{"1min":0.25600000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":4.9960000000000004,"5min":5.4930000000000003,"15min":5.4930000000000003},"last":3.5499999999999998},{"interface":"front","average":{"1min":1.3480000000000001,"5min":1.4810000000000001,"15min":0.77900000000000003},"min":{"1min":0.372,"5min":0.247,"15min":0.247},"max":{"1min":4.6189999999999998,"5min":6.0679999999999996,"15min":6.0679999999999996},"last":3.008}]},{"osd":4,"last update":"Tue Sep 17 07:08:06 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.86799999999999999,"15min":0.66600000000000004},"min":{"1min":0.31900000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.73599999999999999,"5min":5.2359999999999998,"15min":5.2359999999999998},"last":0.496},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.77200000000000002,"15min":0.64900000000000002},"min":{"1min":0.378,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":1.855,"5min":3.4689999999999999,"15min":3.4689999999999999},"last":1.073}]},{"osd":5,"last update":"Tue Sep 17 07:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":1.0169999999999999,"15min":0.73599999999999999},"min":{"1min":0.31900000000000001,"5min":0.249,"15min":0.249},"max":{"1min":2.1040000000000001,"5min":5.383,"15min":5.383},"last":2.9820000000000002},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.86499999999999999,"15min":0.68000000000000005},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":2.044,"5min":5.3600000000000003,"15min":5.3600000000000003},"last":1.1200000000000001}]},{"osd":6,"last update":"Tue Sep 17 07:08:48 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.94199999999999995,"15min":0.71399999999999997},"min":{"1min":0.39700000000000002,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.546,"5min":9.2509999999999994,"15min":9.2509999999999994},"last":1.161},{"interface":"front","average":{"1min":0.84999999999999998,"5min":1.121,"15min":0.77100000000000002},"min":{"1min":0.34799999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":2.9550000000000001,"5min":5.3330000000000002,"15min":5.3330000000000002},"last":2.9550000000000001}]},{"osd":7,"last update":"Tue Sep 17 07:08:21 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.90100000000000002,"15min":0.68100000000000005},"min":{"1min":0.378,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.76900000000000002,"5min":5.3250000000000002,"15min":5.3250000000000002},"last":1.149},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.97499999999999998,"15min":0.84599999999999997},"min":{"1min":0.36199999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.343,"5min":9.2919999999999998,"15min":9.2919999999999998},"last":1.1830000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574960,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1983336,"kb_used_data":1807840,"kb_used_omap":1,"kb_used_meta":175486,"kb_avail":91761816,"statfs":{"total":95995035648,"available":93964099584,"internally_reserved":0,"allocated":1851228160,"data_stored":1850735192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":179698118},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:07:52 2024","interfaces":[{"interface":"back","average":{"1min":1.552,"5min":1.339,"15min":0.67100000000000004},"min":{"1min":0.19400000000000001,"5min":0.155,"15min":0.155},"max":{"1min":6.6020000000000003,"5min":13.837,"15min":13.837},"last":0.82099999999999995},{"interface":"front","average":{"1min":1.0840000000000001,"5min":1.304,"15min":0.67900000000000005},"min":{"1min":0.22700000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":5.6020000000000003,"5min":10.342000000000001,"15min":10.342000000000001},"last":5.4119999999999999}]},{"osd":2,"last update":"Tue Sep 17 07:08:25 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":1.264,"15min":0.68799999999999994},"min":{"1min":0.23899999999999999,"5min":0.192,"15min":0.192},"max":{"1min":2.9700000000000002,"5min":11.055999999999999,"15min":11.055999999999999},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.46300000000000002,"5min":1.3169999999999999,"15min":0.71299999999999997},"min":{"1min":0.23999999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":1.4950000000000001,"5min":9.3160000000000007,"15min":9.3160000000000007},"last":5.242}]},{"osd":3,"last update":"Tue Sep 17 07:07:46 2024","interfaces":[{"interface":"back","average":{"1min":1.125,"5min":1.2629999999999999,"15min":0.72899999999999998},"min":{"1min":0.27800000000000002,"5min":0.219,"15min":0.219},"max":{"1min":5.4740000000000002,"5min":7.766,"15min":7.766},"last":5.2800000000000002},{"interface":"front","average":{"1min":1.7030000000000001,"5min":1.351,"15min":0.745},"min":{"1min":0.28899999999999998,"5min":0.222,"15min":0.222},"max":{"1min":7.2759999999999998,"5min":19.602,"15min":19.602},"last":0.745}]},{"osd":4,"last update":"Tue Sep 17 07:08:06 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":1.0229999999999999,"15min":0.72399999999999998},"min":{"1min":0.32100000000000001,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":1.6739999999999999,"5min":7.8840000000000003,"15min":7.8840000000000003},"last":5.3040000000000003},{"interface":"front","average":{"1min":0.88400000000000001,"5min":1.0129999999999999,"15min":0.71699999999999997},"min":{"1min":0.32000000000000001,"5min":0.307,"15min":0.307},"max":{"1min":4.133,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.45400000000000001}]},{"osd":5,"last update":"Tue Sep 17 07:08:25 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":1.0409999999999999,"15min":0.74399999999999999},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":1.4710000000000001,"5min":5.617,"15min":6.0060000000000002},"last":5.3470000000000004},{"interface":"front","average":{"1min":0.72599999999999998,"5min":1.0940000000000001,"15min":0.74099999999999999},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":1.8999999999999999,"5min":8.8889999999999993,"15min":8.8889999999999993},"last":0.41899999999999998}]},{"osd":6,"last update":"Tue Sep 17 07:07:47 2024","interfaces":[{"interface":"back","average":{"1min":1.764,"5min":1.115,"15min":0.73399999999999999},"min":{"1min":0.36899999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":7.2539999999999996,"5min":8.843,"15min":8.843},"last":0.48399999999999999},{"interface":"front","average":{"1min":1.276,"5min":1.0669999999999999,"15min":0.753},"min":{"1min":0.38300000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":7.1130000000000004,"5min":7.1130000000000004,"15min":7.1130000000000004},"last":5.3250000000000002}]},{"osd":7,"last update":"Tue Sep 17 07:08:09 2024","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":1.079,"15min":0.73499999999999999},"min":{"1min":0.38500000000000001,"5min":0.37,"15min":0.37},"max":{"1min":4.2089999999999996,"5min":8.9309999999999992,"15min":8.9309999999999992},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.69099999999999995,"5min":1.0449999999999999,"15min":0.72899999999999998},"min":{"1min":0.41499999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":2.7120000000000002,"5min":11.007999999999999,"15min":11.007999999999999},"last":0.56699999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738483,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1908880,"kb_used_data":1584904,"kb_used_omap":1,"kb_used_meta":323966,"kb_avail":91836272,"statfs":{"total":95995035648,"available":94040342528,"internally_reserved":0,"allocated":1622941696,"data_stored":1622472656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":331741638},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 17 07:07:54 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":1.107,"15min":0.58999999999999997},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":5.4489999999999998,"5min":8.0259999999999998,"15min":8.0259999999999998},"last":0.316},{"interface":"front","average":{"1min":1.538,"5min":1.1970000000000001,"15min":0.628},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":6.867,"5min":7.8849999999999998,"15min":7.8849999999999998},"last":1.216}]},{"osd":2,"last update":"Tue Sep 17 07:08:24 2024","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":1.8740000000000001,"15min":0.877},"min":{"1min":0.248,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":5.242,"5min":18.643999999999998,"15min":18.643999999999998},"last":1.4299999999999999},{"interface":"front","average":{"1min":1.123,"5min":1.6859999999999999,"15min":0.80500000000000005},"min":{"1min":0.26000000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":4.9690000000000003,"5min":17.972999999999999,"15min":17.972999999999999},"last":0.27600000000000002}]},{"osd":3,"last update":"Tue Sep 17 07:07:54 2024","interfaces":[{"interface":"back","average":{"1min":1.8129999999999999,"5min":1.6160000000000001,"15min":0.83099999999999996},"min":{"1min":0.315,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":11.598000000000001,"5min":18.629000000000001,"15min":18.629000000000001},"last":1.0229999999999999},{"interface":"front","average":{"1min":1.1699999999999999,"5min":1.782,"15min":0.88300000000000001},"min":{"1min":0.30399999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":5.4169999999999998,"5min":22.358000000000001,"15min":22.358000000000001},"last":0.35399999999999998}]},{"osd":4,"last update":"Tue Sep 17 07:08:06 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":1.385,"15min":0.874},"min":{"1min":0.32100000000000001,"5min":0.316,"15min":0.316},"max":{"1min":1.54,"5min":18.568000000000001,"15min":18.568000000000001},"last":2.1520000000000001},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1.3080000000000001,"15min":0.80000000000000004},"min":{"1min":0.27400000000000002,"5min":0.255,"15min":0.255},"max":{"1min":4.7709999999999999,"5min":18.765000000000001,"15min":18.765000000000001},"last":1.488}]},{"osd":5,"last update":"Tue Sep 17 07:08:32 2024","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":1.4330000000000001,"15min":0.84499999999999997},"min":{"1min":0.40300000000000002,"5min":0.309,"15min":0.309},"max":{"1min":7.2190000000000003,"5min":18.614999999999998,"15min":18.614999999999998},"last":0.38900000000000001},{"interface":"front","average":{"1min":1.5489999999999999,"5min":1.423,"15min":0.83599999999999997},"min":{"1min":0.32500000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":7.0839999999999996,"5min":18.791,"15min":18.791},"last":0.377}]},{"osd":6,"last update":"Tue Sep 17 07:07:54 2024","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.3129999999999999,"15min":0.80100000000000005},"min":{"1min":0.34699999999999998,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":5.2240000000000002,"5min":18.602,"15min":18.602},"last":2.2799999999999998},{"interface":"front","average":{"1min":1.2010000000000001,"5min":1.496,"15min":0.86399999999999999},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":6.149,"5min":18.692,"15min":18.692},"last":0.41699999999999998}]},{"osd":7,"last update":"Tue Sep 17 07:08:14 2024","interfaces":[{"interface":"back","average":{"1min":1.113,"5min":1.8620000000000001,"15min":1.421},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":4.8490000000000002,"5min":46.183,"15min":46.183},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.98799999999999999,"5min":1.7669999999999999,"15min":0.96299999999999997},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":5.0190000000000001,"5min":45.914999999999999,"15min":45.914999999999999},"last":0.48099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1643970560,"data_stored":1643970560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1858011136,"data_stored":1858011136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1603731456,"data_stored":1603731456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1632043008,"data_stored":1632043008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1354039296,"data_stored":1354039296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2117468160,"data_stored":2117468160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1327300608,"data_stored":1327300608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1361182720,"data_stored":1361182720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T07:08:54.909 INFO:tasks.thrashosds.thrasher:Setting 2.15 to [7, 6, 3, 5, 2, 1] 2024-09-17T07:08:54.909 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '2.15', '7', '6', '3', '5', '2', '1'] 2024-09-17T07:08:54.909 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd pg-upmap-items 2.15 7 6 3 5 2 1 2024-09-17T07:08:54.975 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:08:55.006 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:55.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:55.007 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:08:55.053 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 39 16 74841 74825 119.877 86.625 0.00861955 0.0083366 2024-09-17T07:08:55.108 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:08:55.162 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:55.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:55.163 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:08:55.264 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:08:55.291 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:55.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:55.292 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:08:55.393 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:08:55.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:55.453 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:55.453 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:08:55.554 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:08:55.580 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:55.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:55.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:08:55.664 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:08:55.682 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:08:55.727 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-17T07:08:55.727 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-17T07:08:55.728 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-17T07:08:55.728 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-17T07:08:55.728 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.728 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.14156.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-09-17T07:08:55.728 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:00:10.403439+0000", 2024-09-17T07:08:55.728 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 525.30434138400005, 2024-09-17T07:08:55.728 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.32094682499999999, 2024-09-17T07:08:55.728 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:08:55.728 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:08:55.728 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:08:55.728 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.14156", 2024-09-17T07:08:55.728 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/2225586217", 2024-09-17T07:08:55.729 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1 2024-09-17T07:08:55.729 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.729 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:08:55.729 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.729 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:08:55.729 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:00:10.403439+0000", 2024-09-17T07:08:55.729 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:55.729 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.729 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.729 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:08:55.729 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:00:10.403439+0000", 2024-09-17T07:08:55.730 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:55.730 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.730 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.730 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:08:55.730 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:00:10.403438+0000", 2024-09-17T07:08:55.730 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999986 2024-09-17T07:08:55.730 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.730 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.730 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:08:55.730 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:00:10.403445+0000", 2024-09-17T07:08:55.730 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7039999999999997e-06 2024-09-17T07:08:55.730 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.731 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.731 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:08:55.731 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:00:10.403452+0000", 2024-09-17T07:08:55.731 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8359999999999992e-06 2024-09-17T07:08:55.731 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.731 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.731 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:08:55.731 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:00:10.403459+0000", 2024-09-17T07:08:55.731 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7469999999999999e-06 2024-09-17T07:08:55.731 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.731 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.731 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:08:55.731 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:00:10.403507+0000", 2024-09-17T07:08:55.732 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8173000000000001e-05 2024-09-17T07:08:55.732 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.732 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.732 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-17T07:08:55.732 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:00:10.403518+0000", 2024-09-17T07:08:55.732 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0735e-05 2024-09-17T07:08:55.732 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.732 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.732 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:08:55.732 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:00:10.723292+0000", 2024-09-17T07:08:55.732 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.31977418600000002 2024-09-17T07:08:55.732 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.732 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.733 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:08:55.733 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:00:10.723418+0000", 2024-09-17T07:08:55.733 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000125742 2024-09-17T07:08:55.733 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.733 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.733 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 0,2", 2024-09-17T07:08:55.733 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:00:10.723581+0000", 2024-09-17T07:08:55.733 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016337900000000001 2024-09-17T07:08:55.733 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.733 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.733 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:08:55.733 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:00:10.724185+0000", 2024-09-17T07:08:55.734 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00060370900000000002 2024-09-17T07:08:55.734 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.734 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.734 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:08:55.734 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:00:10.724297+0000", 2024-09-17T07:08:55.734 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000111753 2024-09-17T07:08:55.734 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.734 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.734 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:08:55.734 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:00:10.724326+0000", 2024-09-17T07:08:55.734 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8898e-05 2024-09-17T07:08:55.734 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.734 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.735 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:08:55.735 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:00:10.724364+0000", 2024-09-17T07:08:55.735 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8238000000000002e-05 2024-09-17T07:08:55.735 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.735 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.735 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:08:55.735 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:00:10.724386+0000", 2024-09-17T07:08:55.735 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2061000000000001e-05 2024-09-17T07:08:55.735 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:55.735 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:55.735 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:55.735 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.735 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.736 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.14805.0:54889 7.2 e110/99 7:40e8e483:::benchmark_data_smithi100_78556_object54888:head v 110'3382, mlcod=110'3379)", 2024-09-17T07:08:55.736 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:04:57.326084+0000", 2024-09-17T07:08:55.736 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 238.38169706100001, 2024-09-17T07:08:55.736 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.110644224, 2024-09-17T07:08:55.736 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:08:55.736 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-17T07:08:55.736 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:08:55.736 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.736 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:08:55.736 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.326084+0000", 2024-09-17T07:08:55.736 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:55.736 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.736 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.737 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:08:55.737 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.326084+0000", 2024-09-17T07:08:55.737 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:55.737 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.737 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.737 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:08:55.737 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.326083+0000", 2024-09-17T07:08:55.737 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-17T07:08:55.737 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.737 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.737 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:08:55.737 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.326105+0000", 2024-09-17T07:08:55.738 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2331e-05 2024-09-17T07:08:55.738 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.738 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.738 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:08:55.738 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.326115+0000", 2024-09-17T07:08:55.738 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0168e-05 2024-09-17T07:08:55.738 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.738 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.738 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:08:55.738 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.326128+0000", 2024-09-17T07:08:55.738 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2853e-05 2024-09-17T07:08:55.738 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.738 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.739 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:08:55.739 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.326427+0000", 2024-09-17T07:08:55.739 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00029850599999999998 2024-09-17T07:08:55.739 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.739 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.739 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:08:55.739 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.326445+0000", 2024-09-17T07:08:55.739 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.8291000000000002e-05 2024-09-17T07:08:55.739 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.739 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.739 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:08:55.739 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.436728+0000", 2024-09-17T07:08:55.739 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.110282854 2024-09-17T07:08:55.740 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:55.740 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:55.740 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:55.740 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.740 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.740 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.14805.0:54893 7.0 e110/99 7:0b6d2812:::benchmark_data_smithi100_78556_object54892:head v 110'3362, mlcod=110'3360)", 2024-09-17T07:08:55.740 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:04:57.328476+0000", 2024-09-17T07:08:55.740 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 238.37930513000001, 2024-09-17T07:08:55.740 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.10262708800000001, 2024-09-17T07:08:55.740 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:08:55.740 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-17T07:08:55.740 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:08:55.741 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.741 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:08:55.741 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.328476+0000", 2024-09-17T07:08:55.741 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:55.741 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.741 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.741 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:08:55.741 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.328476+0000", 2024-09-17T07:08:55.741 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:55.741 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.741 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.741 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:08:55.741 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.328474+0000", 2024-09-17T07:08:55.742 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-17T07:08:55.742 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.742 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.742 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:08:55.742 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.328504+0000", 2024-09-17T07:08:55.742 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9706999999999999e-05 2024-09-17T07:08:55.742 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.742 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.742 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:08:55.742 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.328514+0000", 2024-09-17T07:08:55.742 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0016e-05 2024-09-17T07:08:55.742 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.742 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.743 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:08:55.743 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.328520+0000", 2024-09-17T07:08:55.743 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1739999999999997e-06 2024-09-17T07:08:55.743 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.743 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.743 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:08:55.743 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.329430+0000", 2024-09-17T07:08:55.743 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00091004699999999998 2024-09-17T07:08:55.743 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.743 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.743 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:08:55.743 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.329449+0000", 2024-09-17T07:08:55.744 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.9023999999999999e-05 2024-09-17T07:08:55.744 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.744 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.744 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:08:55.744 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.431103+0000", 2024-09-17T07:08:55.744 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.101653179 2024-09-17T07:08:55.744 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:55.744 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:55.744 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:55.744 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.744 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.744 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.14805.0:190849 7.b 7:d0440278:::benchmark_data_smithi100_78556_object80046:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e157)", 2024-09-17T07:08:55.745 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:07:10.518615+0000", 2024-09-17T07:08:55.745 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 105.189165997, 2024-09-17T07:08:55.745 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.099308860999999998, 2024-09-17T07:08:55.745 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:08:55.745 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:08:55.745 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:08:55.745 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.14805", 2024-09-17T07:08:55.745 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:08:55.745 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 190849 2024-09-17T07:08:55.745 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.745 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:08:55.745 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.745 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:08:55.746 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:07:10.518615+0000", 2024-09-17T07:08:55.746 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:55.746 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.746 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.746 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:08:55.746 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:07:10.518615+0000", 2024-09-17T07:08:55.746 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:55.746 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.746 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.746 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:08:55.746 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:07:10.518613+0000", 2024-09-17T07:08:55.746 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999986 2024-09-17T07:08:55.746 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.747 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.747 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:08:55.747 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:07:10.518619+0000", 2024-09-17T07:08:55.747 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0739999999999998e-06 2024-09-17T07:08:55.747 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.747 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.747 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:08:55.747 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:07:10.518631+0000", 2024-09-17T07:08:55.747 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1681e-05 2024-09-17T07:08:55.747 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.747 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.747 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:08:55.747 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:07:10.518639+0000", 2024-09-17T07:08:55.747 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8029999999999997e-06 2024-09-17T07:08:55.748 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.748 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.748 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:08:55.748 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:07:10.519126+0000", 2024-09-17T07:08:55.748 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00048766599999999999 2024-09-17T07:08:55.748 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.748 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.748 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:08:55.748 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:07:10.519229+0000", 2024-09-17T07:08:55.748 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000102026 2024-09-17T07:08:55.748 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.748 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.748 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 5,6", 2024-09-17T07:08:55.749 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:07:10.519330+0000", 2024-09-17T07:08:55.749 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00010165799999999999 2024-09-17T07:08:55.749 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.749 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.749 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:08:55.749 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:07:10.524641+0000", 2024-09-17T07:08:55.749 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0053107420000000002 2024-09-17T07:08:55.749 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.749 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.749 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:08:55.749 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:07:10.524717+0000", 2024-09-17T07:08:55.749 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6398999999999998e-05 2024-09-17T07:08:55.749 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.750 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.750 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:08:55.750 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:07:10.617831+0000", 2024-09-17T07:08:55.750 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.093113734000000004 2024-09-17T07:08:55.750 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.750 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.750 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:08:55.750 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:07:10.617882+0000", 2024-09-17T07:08:55.750 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0568000000000001e-05 2024-09-17T07:08:55.750 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.750 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.750 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:08:55.750 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:07:10.617923+0000", 2024-09-17T07:08:55.750 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1894000000000002e-05 2024-09-17T07:08:55.751 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:55.751 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:55.751 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:55.751 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.751 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.751 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.14805.0:190850 7.b 7:d823d0cb:::benchmark_data_smithi100_78556_object80047:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e157)", 2024-09-17T07:08:55.751 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:07:10.518651+0000", 2024-09-17T07:08:55.751 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 105.189130046, 2024-09-17T07:08:55.751 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.099376032000000003, 2024-09-17T07:08:55.751 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:08:55.752 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:08:55.752 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:08:55.752 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.14805", 2024-09-17T07:08:55.752 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:08:55.752 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 190850 2024-09-17T07:08:55.752 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.752 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:08:55.752 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.752 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:08:55.752 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:07:10.518651+0000", 2024-09-17T07:08:55.752 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:55.752 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.752 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.753 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:08:55.753 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:07:10.518651+0000", 2024-09-17T07:08:55.753 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:55.753 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.753 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.753 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:08:55.753 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:07:10.518650+0000", 2024-09-17T07:08:55.753 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-17T07:08:55.753 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.753 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.753 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:08:55.753 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:07:10.518653+0000", 2024-09-17T07:08:55.753 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4839999999999998e-06 2024-09-17T07:08:55.754 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.754 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.754 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:08:55.754 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:07:10.518659+0000", 2024-09-17T07:08:55.754 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9660000000000001e-06 2024-09-17T07:08:55.754 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.754 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.754 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:08:55.754 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:07:10.518663+0000", 2024-09-17T07:08:55.754 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7879999999999997e-06 2024-09-17T07:08:55.754 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.754 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.754 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:08:55.755 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:07:10.521972+0000", 2024-09-17T07:08:55.755 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0033081109999999999 2024-09-17T07:08:55.755 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.755 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.755 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:08:55.755 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:07:10.522081+0000", 2024-09-17T07:08:55.755 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000109183 2024-09-17T07:08:55.755 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.755 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.755 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 5,6", 2024-09-17T07:08:55.755 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:07:10.522195+0000", 2024-09-17T07:08:55.755 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00011383999999999999 2024-09-17T07:08:55.755 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.756 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.756 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:08:55.756 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:07:10.524709+0000", 2024-09-17T07:08:55.756 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0025143940000000001 2024-09-17T07:08:55.756 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.756 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.756 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:08:55.756 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:07:10.524797+0000", 2024-09-17T07:08:55.756 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8510000000000005e-05 2024-09-17T07:08:55.756 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.756 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.756 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:08:55.757 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:07:10.617959+0000", 2024-09-17T07:08:55.757 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.093161646000000001 2024-09-17T07:08:55.757 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.757 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.757 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:08:55.757 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:07:10.617995+0000", 2024-09-17T07:08:55.757 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5516999999999999e-05 2024-09-17T07:08:55.757 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.757 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.757 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:08:55.757 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:07:10.618027+0000", 2024-09-17T07:08:55.758 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.2019e-05 2024-09-17T07:08:55.758 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:55.758 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:55.758 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:55.758 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.758 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.758 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.24929.0:30976 8.7 8:e9519e28:::benchmark_data_smithi100_102803_object30975:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-09-17T07:08:55.758 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:08:27.692985+0000", 2024-09-17T07:08:55.758 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 28.014795761999999, 2024-09-17T07:08:55.758 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.14041921600000001, 2024-09-17T07:08:55.758 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:08:55.758 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:08:55.758 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:08:55.759 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.24929", 2024-09-17T07:08:55.759 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/564515582", 2024-09-17T07:08:55.759 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 30976 2024-09-17T07:08:55.759 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.759 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:08:55.759 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.759 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:08:55.759 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.692985+0000", 2024-09-17T07:08:55.759 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:55.759 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.759 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.759 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:08:55.759 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.692985+0000", 2024-09-17T07:08:55.760 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:55.760 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.760 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.760 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:08:55.760 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.692983+0000", 2024-09-17T07:08:55.760 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999986 2024-09-17T07:08:55.760 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.760 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.760 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:08:55.760 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.693005+0000", 2024-09-17T07:08:55.760 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1574999999999999e-05 2024-09-17T07:08:55.760 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.760 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.761 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:08:55.761 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.693016+0000", 2024-09-17T07:08:55.761 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1467999999999999e-05 2024-09-17T07:08:55.761 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.761 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.761 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:08:55.761 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.693023+0000", 2024-09-17T07:08:55.761 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8240000000000002e-06 2024-09-17T07:08:55.761 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.761 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.761 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:08:55.761 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.693480+0000", 2024-09-17T07:08:55.761 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00045714600000000002 2024-09-17T07:08:55.762 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.762 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.762 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:08:55.762 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.693594+0000", 2024-09-17T07:08:55.762 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000113527 2024-09-17T07:08:55.762 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.762 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.762 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 3,6", 2024-09-17T07:08:55.762 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.693893+0000", 2024-09-17T07:08:55.762 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000299537 2024-09-17T07:08:55.762 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.762 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.762 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:08:55.763 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.696526+0000", 2024-09-17T07:08:55.763 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0026324830000000001 2024-09-17T07:08:55.763 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.763 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.763 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:08:55.763 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.697817+0000", 2024-09-17T07:08:55.763 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00129093 2024-09-17T07:08:55.763 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.763 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.763 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:08:55.763 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.833288+0000", 2024-09-17T07:08:55.763 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.135471177 2024-09-17T07:08:55.763 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.763 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.764 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:08:55.764 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.833367+0000", 2024-09-17T07:08:55.764 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9555000000000007e-05 2024-09-17T07:08:55.764 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.764 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.764 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:08:55.764 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.833404+0000", 2024-09-17T07:08:55.764 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6607999999999999e-05 2024-09-17T07:08:55.764 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:55.764 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:55.764 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:55.764 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.764 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.765 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.24929.0:30977 8.7 8:ea4a4a82:::benchmark_data_smithi100_102803_object30976:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-09-17T07:08:55.765 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:08:27.693501+0000", 2024-09-17T07:08:55.765 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 28.014279383000002, 2024-09-17T07:08:55.765 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.14004154299999999, 2024-09-17T07:08:55.765 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:08:55.765 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:08:55.765 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:08:55.765 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.24929", 2024-09-17T07:08:55.765 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/564515582", 2024-09-17T07:08:55.765 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 30977 2024-09-17T07:08:55.765 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.765 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:08:55.765 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.766 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:08:55.766 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.693501+0000", 2024-09-17T07:08:55.766 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:55.766 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.766 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.766 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:08:55.766 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.693501+0000", 2024-09-17T07:08:55.766 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:55.766 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.766 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.766 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:08:55.766 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.693500+0000", 2024-09-17T07:08:55.766 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-17T07:08:55.767 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.767 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.767 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:08:55.767 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.693520+0000", 2024-09-17T07:08:55.767 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.9755999999999999e-05 2024-09-17T07:08:55.767 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.767 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.767 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:08:55.767 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.693529+0000", 2024-09-17T07:08:55.767 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5429999999999994e-06 2024-09-17T07:08:55.767 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.767 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.767 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:08:55.768 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.693535+0000", 2024-09-17T07:08:55.768 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9630000000000003e-06 2024-09-17T07:08:55.768 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.768 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.768 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:08:55.768 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.695098+0000", 2024-09-17T07:08:55.768 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0015631880000000001 2024-09-17T07:08:55.768 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.768 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.768 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:08:55.768 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.695214+0000", 2024-09-17T07:08:55.768 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000115798 2024-09-17T07:08:55.769 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.769 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.769 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 3,6", 2024-09-17T07:08:55.769 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.695336+0000", 2024-09-17T07:08:55.769 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012214100000000001 2024-09-17T07:08:55.769 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.769 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.769 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:08:55.769 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.697810+0000", 2024-09-17T07:08:55.769 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0024742509999999998 2024-09-17T07:08:55.769 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.769 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.769 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:08:55.770 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.697849+0000", 2024-09-17T07:08:55.770 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9385999999999999e-05 2024-09-17T07:08:55.770 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.770 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.770 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:08:55.770 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.833441+0000", 2024-09-17T07:08:55.770 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.13559180000000001 2024-09-17T07:08:55.770 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.770 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.770 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:08:55.770 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.833512+0000", 2024-09-17T07:08:55.770 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0403999999999999e-05 2024-09-17T07:08:55.770 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.771 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.771 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:08:55.771 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.833543+0000", 2024-09-17T07:08:55.771 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1214000000000003e-05 2024-09-17T07:08:55.771 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:55.771 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:55.771 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:55.771 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.771 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.771 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.24929.0:30980 8.7 8:e876fbe7:::benchmark_data_smithi100_102803_object30979:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-09-17T07:08:55.771 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:08:27.695551+0000", 2024-09-17T07:08:55.771 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 28.012229164000001, 2024-09-17T07:08:55.771 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.13810655699999999, 2024-09-17T07:08:55.771 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:08:55.772 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:08:55.772 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:08:55.772 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.24929", 2024-09-17T07:08:55.772 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/564515582", 2024-09-17T07:08:55.773 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 30980 2024-09-17T07:08:55.773 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.773 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:08:55.773 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.773 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:08:55.773 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.695551+0000", 2024-09-17T07:08:55.773 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:55.773 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.773 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.773 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:08:55.773 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.695551+0000", 2024-09-17T07:08:55.773 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:55.773 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.774 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.774 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:08:55.774 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.695551+0000", 2024-09-17T07:08:55.774 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-17T07:08:55.774 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.774 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.774 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:08:55.774 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.695573+0000", 2024-09-17T07:08:55.774 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2158e-05 2024-09-17T07:08:55.774 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.774 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.774 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:08:55.775 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.695580+0000", 2024-09-17T07:08:55.775 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8750000000000002e-06 2024-09-17T07:08:55.775 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.775 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.775 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:08:55.775 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.695585+0000", 2024-09-17T07:08:55.775 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1309999999999997e-06 2024-09-17T07:08:55.775 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.775 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.775 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:08:55.775 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.696557+0000", 2024-09-17T07:08:55.775 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000972104 2024-09-17T07:08:55.775 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.776 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.776 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:08:55.776 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.696654+0000", 2024-09-17T07:08:55.776 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7264999999999998e-05 2024-09-17T07:08:55.776 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.776 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.776 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 3,6", 2024-09-17T07:08:55.776 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.696769+0000", 2024-09-17T07:08:55.776 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000114838 2024-09-17T07:08:55.776 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.776 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.776 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:08:55.776 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.697959+0000", 2024-09-17T07:08:55.777 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00118951 2024-09-17T07:08:55.777 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.777 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.777 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:08:55.777 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.698737+0000", 2024-09-17T07:08:55.777 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00077891199999999999 2024-09-17T07:08:55.777 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.777 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.777 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:08:55.777 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.833584+0000", 2024-09-17T07:08:55.777 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.134846934 2024-09-17T07:08:55.777 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.778 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.778 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:08:55.778 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.833626+0000", 2024-09-17T07:08:55.778 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2052000000000003e-05 2024-09-17T07:08:55.778 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.778 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.778 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:08:55.778 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.833658+0000", 2024-09-17T07:08:55.778 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1616999999999999e-05 2024-09-17T07:08:55.778 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:55.778 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:55.778 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:55.778 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.778 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.779 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.24929.0:30985 8.0 e178/168 8:0d803640:::benchmark_data_smithi100_102803_object30984:head v 178'1940, mlcod=178'1936)", 2024-09-17T07:08:55.779 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:08:27.698226+0000", 2024-09-17T07:08:55.779 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 28.009554788999999, 2024-09-17T07:08:55.779 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.13454022299999999, 2024-09-17T07:08:55.779 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:08:55.779 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:08:55.779 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:08:55.779 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.779 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:08:55.779 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.698226+0000", 2024-09-17T07:08:55.779 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:55.779 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.779 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.780 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:08:55.780 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.698226+0000", 2024-09-17T07:08:55.780 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:55.780 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.780 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.780 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:08:55.780 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.698225+0000", 2024-09-17T07:08:55.780 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-17T07:08:55.780 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.780 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.780 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:08:55.780 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.698241+0000", 2024-09-17T07:08:55.781 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5786000000000001e-05 2024-09-17T07:08:55.781 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.781 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.781 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:08:55.781 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.698250+0000", 2024-09-17T07:08:55.781 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5939999999999994e-06 2024-09-17T07:08:55.781 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.781 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.781 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:08:55.781 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.698255+0000", 2024-09-17T07:08:55.781 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5709999999999997e-06 2024-09-17T07:08:55.781 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.781 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.781 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:08:55.782 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.698742+0000", 2024-09-17T07:08:55.782 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00048683500000000002 2024-09-17T07:08:55.782 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.782 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.782 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:08:55.782 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.698761+0000", 2024-09-17T07:08:55.782 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.9301999999999999e-05 2024-09-17T07:08:55.782 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.782 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.782 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:08:55.782 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.832729+0000", 2024-09-17T07:08:55.782 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.133967482 2024-09-17T07:08:55.782 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.782 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.783 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:08:55.783 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.832766+0000", 2024-09-17T07:08:55.783 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7329999999999997e-05 2024-09-17T07:08:55.783 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:55.783 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:55.783 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:55.783 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.783 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.783 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.24929.0:30989 8.a 8:550502fa:::benchmark_data_smithi100_102803_object30988:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-09-17T07:08:55.783 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:08:27.699274+0000", 2024-09-17T07:08:55.783 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 28.008506700000002, 2024-09-17T07:08:55.783 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.13832038899999999, 2024-09-17T07:08:55.783 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:08:55.784 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:08:55.784 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:08:55.784 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.24929", 2024-09-17T07:08:55.784 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/564515582", 2024-09-17T07:08:55.784 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 30989 2024-09-17T07:08:55.784 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.784 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:08:55.784 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.784 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:08:55.784 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.699274+0000", 2024-09-17T07:08:55.784 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:55.784 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.784 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.785 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:08:55.785 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.699274+0000", 2024-09-17T07:08:55.785 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:55.785 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.785 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.785 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:08:55.785 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.699273+0000", 2024-09-17T07:08:55.785 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999986 2024-09-17T07:08:55.785 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.785 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.785 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:08:55.785 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.699357+0000", 2024-09-17T07:08:55.786 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.4247999999999997e-05 2024-09-17T07:08:55.786 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.786 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.786 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:08:55.786 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.699370+0000", 2024-09-17T07:08:55.786 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.3227999999999999e-05 2024-09-17T07:08:55.786 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.786 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.786 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:08:55.786 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.699378+0000", 2024-09-17T07:08:55.786 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9829999999999996e-06 2024-09-17T07:08:55.786 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.786 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.786 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:08:55.787 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.699478+0000", 2024-09-17T07:08:55.787 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9607999999999999e-05 2024-09-17T07:08:55.787 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.787 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.787 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:08:55.787 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.699591+0000", 2024-09-17T07:08:55.787 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000113626 2024-09-17T07:08:55.787 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.787 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.787 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 2,5", 2024-09-17T07:08:55.787 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.699729+0000", 2024-09-17T07:08:55.787 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00013739500000000001 2024-09-17T07:08:55.787 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.788 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.788 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:08:55.788 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.837485+0000", 2024-09-17T07:08:55.788 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.13775602000000001 2024-09-17T07:08:55.788 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.788 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.788 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:08:55.788 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.837491+0000", 2024-09-17T07:08:55.788 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1460000000000001e-06 2024-09-17T07:08:55.788 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.788 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.788 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:08:55.789 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.837520+0000", 2024-09-17T07:08:55.789 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9403e-05 2024-09-17T07:08:55.789 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.789 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.789 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:08:55.789 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.837561+0000", 2024-09-17T07:08:55.789 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0388e-05 2024-09-17T07:08:55.789 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.789 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.789 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:08:55.789 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.837594+0000", 2024-09-17T07:08:55.789 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3689000000000002e-05 2024-09-17T07:08:55.790 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:55.790 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:55.790 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:55.790 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.790 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.790 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.24929.0:30990 8.7 8:edb2861a:::benchmark_data_smithi100_102803_object30989:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-09-17T07:08:55.790 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:08:27.699389+0000", 2024-09-17T07:08:55.790 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 28.008391774, 2024-09-17T07:08:55.790 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.13590692600000001, 2024-09-17T07:08:55.790 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:08:55.790 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:08:55.790 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:08:55.790 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.24929", 2024-09-17T07:08:55.790 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/564515582", 2024-09-17T07:08:55.791 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 30990 2024-09-17T07:08:55.791 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.791 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:08:55.791 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.791 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:08:55.791 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.699389+0000", 2024-09-17T07:08:55.791 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:55.791 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.791 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.791 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:08:55.791 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.699389+0000", 2024-09-17T07:08:55.791 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:55.791 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.792 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.792 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:08:55.792 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.699388+0000", 2024-09-17T07:08:55.792 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-17T07:08:55.792 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.792 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.792 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:08:55.792 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.699425+0000", 2024-09-17T07:08:55.792 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6525e-05 2024-09-17T07:08:55.792 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.792 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.792 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:08:55.793 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.699431+0000", 2024-09-17T07:08:55.793 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6359999999999997e-06 2024-09-17T07:08:55.793 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.793 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.793 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:08:55.793 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.699436+0000", 2024-09-17T07:08:55.793 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7639999999999996e-06 2024-09-17T07:08:55.793 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.793 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.793 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:08:55.793 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.699492+0000", 2024-09-17T07:08:55.793 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5761999999999999e-05 2024-09-17T07:08:55.793 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.794 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.794 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:08:55.794 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.699590+0000", 2024-09-17T07:08:55.794 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.8332999999999995e-05 2024-09-17T07:08:55.794 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.794 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.794 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 3,6", 2024-09-17T07:08:55.794 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.699729+0000", 2024-09-17T07:08:55.794 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00013841799999999999 2024-09-17T07:08:55.794 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.794 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.794 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:08:55.794 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.833238+0000", 2024-09-17T07:08:55.794 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.13350885400000001 2024-09-17T07:08:55.795 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.795 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.795 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:08:55.795 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.833722+0000", 2024-09-17T07:08:55.795 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00048449400000000003 2024-09-17T07:08:55.795 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.795 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.795 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:08:55.795 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.835230+0000", 2024-09-17T07:08:55.795 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001507516 2024-09-17T07:08:55.795 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.795 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.796 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:08:55.796 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.835264+0000", 2024-09-17T07:08:55.796 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.4594000000000001e-05 2024-09-17T07:08:55.796 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.796 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.796 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:08:55.796 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.835296+0000", 2024-09-17T07:08:55.796 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1649999999999997e-05 2024-09-17T07:08:55.796 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:55.796 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:55.796 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:55.796 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.796 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.797 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.24929.0:30993 8.a 8:5cd4bee0:::benchmark_data_smithi100_102803_object30992:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-09-17T07:08:55.797 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:08:27.699447+0000", 2024-09-17T07:08:55.797 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 28.008334088000002, 2024-09-17T07:08:55.797 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.14152471599999999, 2024-09-17T07:08:55.797 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:08:55.797 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:08:55.797 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:08:55.797 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.24929", 2024-09-17T07:08:55.797 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/564515582", 2024-09-17T07:08:55.797 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 30993 2024-09-17T07:08:55.797 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.797 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:08:55.797 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.798 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:08:55.798 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.699447+0000", 2024-09-17T07:08:55.798 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:55.798 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.798 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.798 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:08:55.798 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.699447+0000", 2024-09-17T07:08:55.798 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:55.798 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.798 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.798 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:08:55.798 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.699446+0000", 2024-09-17T07:08:55.798 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-17T07:08:55.799 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.799 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.799 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:08:55.799 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.699464+0000", 2024-09-17T07:08:55.799 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7725999999999999e-05 2024-09-17T07:08:55.799 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.799 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.799 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:08:55.799 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.699469+0000", 2024-09-17T07:08:55.799 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.694e-06 2024-09-17T07:08:55.799 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.799 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.799 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:08:55.800 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.699476+0000", 2024-09-17T07:08:55.800 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3110000000000002e-06 2024-09-17T07:08:55.800 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.800 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.800 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:08:55.800 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.834115+0000", 2024-09-17T07:08:55.800 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.13463902599999999 2024-09-17T07:08:55.800 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.800 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.800 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:08:55.800 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.834239+0000", 2024-09-17T07:08:55.800 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012448799999999999 2024-09-17T07:08:55.800 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.801 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.801 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 2,5", 2024-09-17T07:08:55.801 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.834355+0000", 2024-09-17T07:08:55.801 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000115816 2024-09-17T07:08:55.801 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.801 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.801 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:08:55.801 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.837495+0000", 2024-09-17T07:08:55.801 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0031403170000000001 2024-09-17T07:08:55.801 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.801 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.801 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:08:55.801 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.837628+0000", 2024-09-17T07:08:55.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:55.802 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:55.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:08:55.803 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00013224299999999999 2024-09-17T07:08:55.803 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.804 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.804 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:08:55.804 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.840883+0000", 2024-09-17T07:08:55.804 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00325567 2024-09-17T07:08:55.804 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.804 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.804 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:08:55.804 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.840939+0000", 2024-09-17T07:08:55.804 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5611e-05 2024-09-17T07:08:55.804 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.804 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.804 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:08:55.804 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.840971+0000", 2024-09-17T07:08:55.805 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.2391000000000002e-05 2024-09-17T07:08:55.805 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:55.805 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:55.805 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:55.805 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.805 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.805 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop_reply(client.24929.0:30976 8.7 e178/168 ondisk, result = 0)", 2024-09-17T07:08:55.805 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:08:27.699966+0000", 2024-09-17T07:08:55.805 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 28.007814531000001, 2024-09-17T07:08:55.805 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.13344409300000001, 2024-09-17T07:08:55.805 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:08:55.805 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-17T07:08:55.805 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:08:55.806 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.806 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:08:55.806 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.699966+0000", 2024-09-17T07:08:55.806 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:55.806 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.806 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.806 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:08:55.806 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.699966+0000", 2024-09-17T07:08:55.806 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:55.806 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.806 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.806 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:08:55.806 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.699965+0000", 2024-09-17T07:08:55.807 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-17T07:08:55.807 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.807 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.807 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:08:55.807 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.699970+0000", 2024-09-17T07:08:55.807 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.865e-06 2024-09-17T07:08:55.807 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.807 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.807 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:08:55.807 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.831386+0000", 2024-09-17T07:08:55.807 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.13141614400000001 2024-09-17T07:08:55.807 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.807 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.808 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:08:55.808 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.831397+0000", 2024-09-17T07:08:55.808 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0423e-05 2024-09-17T07:08:55.808 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.808 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.808 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:08:55.808 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.833264+0000", 2024-09-17T07:08:55.808 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0018674169999999999 2024-09-17T07:08:55.808 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.808 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.808 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:08:55.808 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.833276+0000", 2024-09-17T07:08:55.809 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2048000000000001e-05 2024-09-17T07:08:55.809 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.809 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.809 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:08:55.809 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.833410+0000", 2024-09-17T07:08:55.809 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00013389700000000001 2024-09-17T07:08:55.809 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:55.809 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:55.809 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:55.809 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.809 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.809 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop_reply(client.24929.0:30990 8.7 e178/168 ondisk, result = 0)", 2024-09-17T07:08:55.809 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:08:27.701703+0000", 2024-09-17T07:08:55.809 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 28.006077969, 2024-09-17T07:08:55.810 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.13153779800000001, 2024-09-17T07:08:55.810 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:08:55.810 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-17T07:08:55.810 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:08:55.810 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.810 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:08:55.810 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.701703+0000", 2024-09-17T07:08:55.810 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:55.810 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.810 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.810 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:08:55.810 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.701703+0000", 2024-09-17T07:08:55.811 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:55.811 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.811 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.811 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:08:55.811 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.701702+0000", 2024-09-17T07:08:55.811 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-17T07:08:55.811 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.811 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.811 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:08:55.811 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.701711+0000", 2024-09-17T07:08:55.811 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.7949999999999996e-06 2024-09-17T07:08:55.811 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.811 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.812 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:08:55.812 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.831369+0000", 2024-09-17T07:08:55.812 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.129658728 2024-09-17T07:08:55.812 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.812 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.812 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:08:55.812 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.831385+0000", 2024-09-17T07:08:55.812 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5282999999999999e-05 2024-09-17T07:08:55.812 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.812 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.812 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:08:55.812 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.833188+0000", 2024-09-17T07:08:55.812 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0018033330000000001 2024-09-17T07:08:55.813 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.813 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.813 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:08:55.813 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.833232+0000", 2024-09-17T07:08:55.813 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3758999999999999e-05 2024-09-17T07:08:55.813 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.813 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.813 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:08:55.813 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.833240+0000", 2024-09-17T07:08:55.813 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.7949999999999996e-06 2024-09-17T07:08:55.813 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:55.813 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:55.813 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:55.814 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.814 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.814 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.24929.0:31363 8.3 8:cc23617a:::benchmark_data_smithi100_102803_object31362:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-09-17T07:08:55.814 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:08:27.992328+0000", 2024-09-17T07:08:55.814 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 27.715452791000001, 2024-09-17T07:08:55.814 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.104534686, 2024-09-17T07:08:55.814 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:08:55.814 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:08:55.814 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:08:55.814 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.24929", 2024-09-17T07:08:55.814 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/564515582", 2024-09-17T07:08:55.814 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 31363 2024-09-17T07:08:55.814 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.815 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:08:55.815 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.815 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:08:55.815 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.992328+0000", 2024-09-17T07:08:55.815 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:55.815 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.815 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.815 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:08:55.815 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.992328+0000", 2024-09-17T07:08:55.815 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:55.815 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.815 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.815 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:08:55.816 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.992326+0000", 2024-09-17T07:08:55.816 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999986 2024-09-17T07:08:55.816 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.816 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.816 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:08:55.816 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.992350+0000", 2024-09-17T07:08:55.816 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3263000000000001e-05 2024-09-17T07:08:55.816 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.816 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.816 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:08:55.816 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.992360+0000", 2024-09-17T07:08:55.816 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0091999999999999e-05 2024-09-17T07:08:55.816 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.817 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.817 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:08:55.817 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.992366+0000", 2024-09-17T07:08:55.817 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6449999999999999e-06 2024-09-17T07:08:55.817 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.817 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.817 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:08:55.817 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.992756+0000", 2024-09-17T07:08:55.817 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00039000800000000001 2024-09-17T07:08:55.817 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.817 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.817 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:08:55.817 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.992862+0000", 2024-09-17T07:08:55.818 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000105498 2024-09-17T07:08:55.818 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.818 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.818 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 5,6", 2024-09-17T07:08:55.818 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.992981+0000", 2024-09-17T07:08:55.818 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000119269 2024-09-17T07:08:55.818 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.818 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.818 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:08:55.818 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.995973+0000", 2024-09-17T07:08:55.818 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0029923800000000002 2024-09-17T07:08:55.818 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.818 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.819 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:08:55.819 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.996020+0000", 2024-09-17T07:08:55.819 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6591000000000002e-05 2024-09-17T07:08:55.819 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.819 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.819 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:08:55.819 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:28.096731+0000", 2024-09-17T07:08:55.819 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.100710805 2024-09-17T07:08:55.819 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.819 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.819 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:08:55.819 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:28.096778+0000", 2024-09-17T07:08:55.820 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7119000000000001e-05 2024-09-17T07:08:55.820 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.820 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.820 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:08:55.820 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:28.096863+0000", 2024-09-17T07:08:55.820 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.4554000000000004e-05 2024-09-17T07:08:55.820 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:55.820 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:55.820 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:55.820 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.820 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.820 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.24929.0:31367 8.3 8:c9d8b26c:::benchmark_data_smithi100_102803_object31366:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-09-17T07:08:55.821 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:08:27.992599+0000", 2024-09-17T07:08:55.821 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 27.715181545, 2024-09-17T07:08:55.821 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.104340234, 2024-09-17T07:08:55.821 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:08:55.821 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:08:55.821 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:08:55.821 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.24929", 2024-09-17T07:08:55.821 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/564515582", 2024-09-17T07:08:55.821 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 31367 2024-09-17T07:08:55.821 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.821 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:08:55.821 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.821 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:08:55.822 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.992599+0000", 2024-09-17T07:08:55.822 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:55.822 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.822 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.822 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:08:55.822 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.992599+0000", 2024-09-17T07:08:55.822 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:55.822 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.822 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.822 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:08:55.822 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.992598+0000", 2024-09-17T07:08:55.822 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-17T07:08:55.822 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.823 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.823 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:08:55.823 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.992637+0000", 2024-09-17T07:08:55.823 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8787e-05 2024-09-17T07:08:55.823 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.823 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.823 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:08:55.823 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.992647+0000", 2024-09-17T07:08:55.823 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0040999999999999e-05 2024-09-17T07:08:55.823 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.823 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.823 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:08:55.823 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.992661+0000", 2024-09-17T07:08:55.823 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4239999999999999e-05 2024-09-17T07:08:55.824 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.824 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.824 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:08:55.824 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.994404+0000", 2024-09-17T07:08:55.824 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001742866 2024-09-17T07:08:55.824 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.824 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.824 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:08:55.824 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.994530+0000", 2024-09-17T07:08:55.824 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000126169 2024-09-17T07:08:55.824 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.824 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.825 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 5,6", 2024-09-17T07:08:55.825 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.994733+0000", 2024-09-17T07:08:55.825 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00020271699999999999 2024-09-17T07:08:55.825 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.825 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.825 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:08:55.825 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.996051+0000", 2024-09-17T07:08:55.825 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0013181779999999999 2024-09-17T07:08:55.825 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.825 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.825 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:08:55.825 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.996377+0000", 2024-09-17T07:08:55.825 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00032603999999999999 2024-09-17T07:08:55.826 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.826 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.826 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:08:55.826 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:28.096871+0000", 2024-09-17T07:08:55.826 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.10049406399999999 2024-09-17T07:08:55.826 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.826 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.826 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:08:55.826 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:28.096908+0000", 2024-09-17T07:08:55.826 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6125999999999999e-05 2024-09-17T07:08:55.826 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.826 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.826 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:08:55.826 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:28.096939+0000", 2024-09-17T07:08:55.827 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1769e-05 2024-09-17T07:08:55.827 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:55.827 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:55.827 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:55.827 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.827 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.827 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.24929.0:31360 8.0 e178/168 8:01433a86:::benchmark_data_smithi100_102803_object31359:head v 178'1962, mlcod=178'1959)", 2024-09-17T07:08:55.827 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:08:27.992676+0000", 2024-09-17T07:08:55.827 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 27.715104564000001, 2024-09-17T07:08:55.827 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.104044567, 2024-09-17T07:08:55.827 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:08:55.827 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:08:55.827 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:08:55.828 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.828 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:08:55.828 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.992676+0000", 2024-09-17T07:08:55.828 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:55.828 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.828 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.828 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:08:55.828 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.992676+0000", 2024-09-17T07:08:55.828 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:55.829 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.829 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.829 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:08:55.829 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.992676+0000", 2024-09-17T07:08:55.829 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-17T07:08:55.829 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.829 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.829 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:08:55.829 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.992697+0000", 2024-09-17T07:08:55.829 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1903999999999999e-05 2024-09-17T07:08:55.829 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.829 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.829 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:08:55.830 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.992705+0000", 2024-09-17T07:08:55.830 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.165e-06 2024-09-17T07:08:55.830 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.830 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.830 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:08:55.830 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.992710+0000", 2024-09-17T07:08:55.830 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2349999999999999e-06 2024-09-17T07:08:55.830 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.830 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.830 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:08:55.830 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.993700+0000", 2024-09-17T07:08:55.830 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00098975699999999996 2024-09-17T07:08:55.830 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.830 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.831 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:08:55.831 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.993720+0000", 2024-09-17T07:08:55.831 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0080000000000001e-05 2024-09-17T07:08:55.831 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.831 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.831 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:08:55.831 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.996942+0000", 2024-09-17T07:08:55.831 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0032224049999999998 2024-09-17T07:08:55.831 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.831 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.831 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:08:55.831 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:28.096721+0000", 2024-09-17T07:08:55.831 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.099778520999999995 2024-09-17T07:08:55.832 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:55.832 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:55.832 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:55.832 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.832 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.832 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.24929.0:31375 8.1 8:8507bc25:::benchmark_data_smithi100_102803_object31374:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-09-17T07:08:55.832 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:08:27.996353+0000", 2024-09-17T07:08:55.832 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 27.711427688000001, 2024-09-17T07:08:55.832 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.11230029, 2024-09-17T07:08:55.832 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:08:55.832 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:08:55.832 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:08:55.832 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.24929", 2024-09-17T07:08:55.833 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/564515582", 2024-09-17T07:08:55.833 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 31375 2024-09-17T07:08:55.833 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.833 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:08:55.833 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.833 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:08:55.833 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.996353+0000", 2024-09-17T07:08:55.833 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:55.833 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.833 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.833 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:08:55.833 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.996353+0000", 2024-09-17T07:08:55.833 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:55.834 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.834 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.834 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:08:55.834 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.996352+0000", 2024-09-17T07:08:55.834 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999986 2024-09-17T07:08:55.834 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.834 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.834 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:08:55.834 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.996379+0000", 2024-09-17T07:08:55.834 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.6959999999999999e-05 2024-09-17T07:08:55.834 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.834 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.834 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:08:55.835 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.996390+0000", 2024-09-17T07:08:55.835 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1555e-05 2024-09-17T07:08:55.835 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.835 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.835 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:08:55.835 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.996398+0000", 2024-09-17T07:08:55.835 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4939999999999997e-06 2024-09-17T07:08:55.835 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.835 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.835 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:08:55.835 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:28.095013+0000", 2024-09-17T07:08:55.835 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.098615690000000006 2024-09-17T07:08:55.835 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.836 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.836 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:08:55.836 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:28.095158+0000", 2024-09-17T07:08:55.836 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000145146 2024-09-17T07:08:55.836 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.836 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.836 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 0,2", 2024-09-17T07:08:55.836 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:28.095283+0000", 2024-09-17T07:08:55.836 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000124292 2024-09-17T07:08:55.836 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.836 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.836 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:08:55.836 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:28.096683+0000", 2024-09-17T07:08:55.837 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0013999640000000001 2024-09-17T07:08:55.837 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.837 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.837 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:08:55.837 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:28.108508+0000", 2024-09-17T07:08:55.837 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.011825097 2024-09-17T07:08:55.837 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.837 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.837 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:08:55.837 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:28.108570+0000", 2024-09-17T07:08:55.837 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2102999999999998e-05 2024-09-17T07:08:55.837 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.837 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.838 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:08:55.838 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:28.108618+0000", 2024-09-17T07:08:55.838 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8408999999999997e-05 2024-09-17T07:08:55.838 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.838 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.838 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:08:55.838 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:28.108653+0000", 2024-09-17T07:08:55.838 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.489e-05 2024-09-17T07:08:55.838 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:55.838 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:55.838 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:55.838 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.838 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.839 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.24929.0:31377 8.7 8:ebe360ec:::benchmark_data_smithi100_102803_object31376:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-09-17T07:08:55.839 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:08:27.996411+0000", 2024-09-17T07:08:55.839 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 27.71136928, 2024-09-17T07:08:55.839 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.10902583, 2024-09-17T07:08:55.839 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:08:55.839 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:08:55.839 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:08:55.839 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.24929", 2024-09-17T07:08:55.839 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/564515582", 2024-09-17T07:08:55.839 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 31377 2024-09-17T07:08:55.839 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.839 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:08:55.839 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.839 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:08:55.840 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.996411+0000", 2024-09-17T07:08:55.840 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:55.840 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.840 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.840 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:08:55.840 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.996411+0000", 2024-09-17T07:08:55.840 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:55.840 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.840 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.840 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:08:55.840 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.996411+0000", 2024-09-17T07:08:55.840 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-17T07:08:55.840 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.841 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.841 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:08:55.841 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.996434+0000", 2024-09-17T07:08:55.841 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2806000000000002e-05 2024-09-17T07:08:55.841 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.841 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.841 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:08:55.841 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.996441+0000", 2024-09-17T07:08:55.841 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1740000000000003e-06 2024-09-17T07:08:55.841 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.841 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.841 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:08:55.841 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.996447+0000", 2024-09-17T07:08:55.842 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9109999999999998e-06 2024-09-17T07:08:55.842 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.842 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.842 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:08:55.842 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.996501+0000", 2024-09-17T07:08:55.842 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4357000000000002e-05 2024-09-17T07:08:55.842 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.842 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.842 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:08:55.842 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:28.094827+0000", 2024-09-17T07:08:55.842 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.098326403000000007 2024-09-17T07:08:55.842 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.842 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.842 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 3,6", 2024-09-17T07:08:55.843 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:28.094969+0000", 2024-09-17T07:08:55.843 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014136699999999999 2024-09-17T07:08:55.843 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.843 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.843 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:08:55.843 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:28.096295+0000", 2024-09-17T07:08:55.843 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001326537 2024-09-17T07:08:55.843 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.843 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.843 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:08:55.843 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:28.097999+0000", 2024-09-17T07:08:55.843 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0017040619999999999 2024-09-17T07:08:55.843 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.844 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.844 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:08:55.844 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:28.105351+0000", 2024-09-17T07:08:55.844 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0073514940000000001 2024-09-17T07:08:55.844 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.844 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.844 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:08:55.844 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:28.105397+0000", 2024-09-17T07:08:55.844 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6038999999999997e-05 2024-09-17T07:08:55.844 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.844 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.844 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:08:55.844 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:28.105437+0000", 2024-09-17T07:08:55.845 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0284999999999998e-05 2024-09-17T07:08:55.845 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:55.845 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:55.845 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:55.845 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.845 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.845 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.24929.0:31370 8.0 e178/168 8:043f437c:::benchmark_data_smithi100_102803_object31369:head v 178'1963, mlcod=178'1960)", 2024-09-17T07:08:55.845 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:08:27.997010+0000", 2024-09-17T07:08:55.845 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 27.710770655000001, 2024-09-17T07:08:55.845 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.10251250100000001, 2024-09-17T07:08:55.845 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:08:55.845 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:08:55.845 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:08:55.846 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.846 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:08:55.846 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.997010+0000", 2024-09-17T07:08:55.846 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:55.846 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.846 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.846 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:08:55.846 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.997010+0000", 2024-09-17T07:08:55.846 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:08:55.846 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.846 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.846 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:08:55.846 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.997009+0000", 2024-09-17T07:08:55.847 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-17T07:08:55.847 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.847 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.847 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:08:55.847 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:27.997036+0000", 2024-09-17T07:08:55.847 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7189999999999999e-05 2024-09-17T07:08:55.847 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.847 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.847 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:08:55.847 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:28.097022+0000", 2024-09-17T07:08:55.847 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.099985457 2024-09-17T07:08:55.847 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.848 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.848 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:08:55.848 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:28.097031+0000", 2024-09-17T07:08:55.848 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.1630000000000002e-06 2024-09-17T07:08:55.848 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.848 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.848 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:08:55.848 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:28.097070+0000", 2024-09-17T07:08:55.848 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9399e-05 2024-09-17T07:08:55.848 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.848 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.848 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:08:55.849 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:28.097092+0000", 2024-09-17T07:08:55.849 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1359999999999999e-05 2024-09-17T07:08:55.849 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.849 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.849 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:08:55.849 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:28.099493+0000", 2024-09-17T07:08:55.849 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0024015600000000001 2024-09-17T07:08:55.849 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:08:55.849 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:08:55.849 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:08:55.849 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:08:28.099522+0000", 2024-09-17T07:08:55.849 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9124000000000001e-05 2024-09-17T07:08:55.849 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:55.850 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:55.850 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:55.850 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:08:55.850 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:08:55.850 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-17T07:08:55.903 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:08:55.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:55.938 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:55.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:08:55.956 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:08:55 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:08:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:08:56.039 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:08:56.053 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:2024-09-17T07:08:56.053766+0000 min lat: 0.0012044 max lat: 0.142002 avg lat: 0.00841466 2024-09-17T07:08:56.053 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-17T07:08:56.053 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 40 16 76053 76037 118.774 75.75 0.0110904 0.00841466 2024-09-17T07:08:56.072 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:56.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:56.073 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:08:56.174 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:08:56.205 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:56.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:56.206 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:08:56.307 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:08:56.344 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:56.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:56.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:08:56.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:56 smithi100 ceph-mon[29092]: osdmap e185: 8 total, 8 up, 8 in 2024-09-17T07:08:56.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:56 smithi100 ceph-mon[29092]: pgmap v294: 148 pgs: 148 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 110 MiB/s wr, 1.77k op/s 2024-09-17T07:08:56.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:56 smithi100 ceph-mon[39462]: osdmap e185: 8 total, 8 up, 8 in 2024-09-17T07:08:56.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:56 smithi100 ceph-mon[39462]: pgmap v294: 148 pgs: 148 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 110 MiB/s wr, 1.77k op/s 2024-09-17T07:08:56.419 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T07:08:56.448 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:08:56.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:56.482 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:56.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:08:56.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:56 smithi135 ceph-mon[33238]: osdmap e185: 8 total, 8 up, 8 in 2024-09-17T07:08:56.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:56 smithi135 ceph-mon[33238]: pgmap v294: 148 pgs: 148 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 110 MiB/s wr, 1.77k op/s 2024-09-17T07:08:56.582 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:08:56.609 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:56.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:56.611 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:08:56.711 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:08:56.749 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:56.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:56.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:08:56.851 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:08:56.912 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:56.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:56.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:08:57.013 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:08:57.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:57.051 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:57.051 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:08:57.053 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 41 16 77415 77399 117.953 85.125 0.0104531 0.00847293 2024-09-17T07:08:57.153 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:08:57.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:57.184 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:57.185 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:08:57.285 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:08:57.314 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:57.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:57.315 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:08:57.355 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-17T07:08:57.416 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:08:57.485 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:57.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:57.486 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:08:57.587 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:08:57.619 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:57.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:57.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:08:57.720 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:08:57.757 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:57.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:57.758 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:08:57.859 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:08:57.898 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:57.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:57.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:08:57.906 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:08:58.000 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:08:58.033 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:58.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:58.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:08:58.053 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 42 16 78710 78694 117.072 80.9375 0.006075 0.0085366 2024-09-17T07:08:58.135 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:08:58.166 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:58.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:58.167 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:08:58.267 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:08:58.295 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:58.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:58.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:08:58.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:08:58 smithi100 ceph-mon[29092]: pgmap v295: 148 pgs: 148 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 53 MiB/s wr, 854 op/s 2024-09-17T07:08:58.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:08:58 smithi100 ceph-mon[39462]: pgmap v295: 148 pgs: 148 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 53 MiB/s wr, 854 op/s 2024-09-17T07:08:58.397 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:08:58.435 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:58.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:58.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:08:58.537 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:08:58.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:08:58 smithi135 ceph-mon[33238]: pgmap v295: 148 pgs: 148 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 53 MiB/s wr, 854 op/s 2024-09-17T07:08:58.543 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:08:58 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:08:58.581 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:58.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:58.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:08:58.682 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:08:58.717 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:58.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:58.718 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:08:58.819 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:08:58.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:58.879 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:58.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:08:58.980 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:08:59.018 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:59.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:59.019 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:08:59.053 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 43 16 80024 80008 116.259 82.125 0.0088568 0.00859609 2024-09-17T07:08:59.120 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:08:59.181 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:59.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:59.181 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:08:59.282 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:08:59.325 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:59.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:59.326 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:08:59.427 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:08:59.454 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:59.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:59.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:08:59.556 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:08:59.569 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:08:59.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:59.609 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:59.609 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:08:59.709 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:08:59.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:59.772 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:08:59.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:08:59.868 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:08:59.873 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:08:59.902 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:08:59.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:08:59.903 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:08:59.913 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:08:59.913 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:08:59.949 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1: {} 2024-09-17T07:08:59.949 INFO:teuthology.orchestra.run.smithi100.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:08:59.985 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:08:59.985 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:09:00.003 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:09:00.026 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:09:00.026 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:09:00.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:00.031 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:00.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:09:00.053 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 44 16 81107 81091 115.155 67.6875 0.0119518 0.00867844 2024-09-17T07:09:00.062 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:09:00.062 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:09:00.093 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:09:00.094 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:09:00.128 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:09:00.128 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:09:00.132 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:09:00.155 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:00.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:00.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:09:00.169 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:09:00.169 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:09:00.257 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:09:00.282 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:00.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:00.283 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:09:00.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:00 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:09:00.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:00 smithi100 ceph-mon[29092]: pgmap v296: 148 pgs: 148 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 105 MiB/s wr, 1.69k op/s 2024-09-17T07:09:00.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:00 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/978455207' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:09:00.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:00 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:09:00.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:00 smithi100 ceph-mon[39462]: pgmap v296: 148 pgs: 148 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 105 MiB/s wr, 1.69k op/s 2024-09-17T07:09:00.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:00 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/978455207' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:09:00.384 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:09:00.418 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:00.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:00.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:09:00.520 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:09:00.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:00 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:09:00.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:00 smithi135 ceph-mon[33238]: pgmap v296: 148 pgs: 148 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 105 MiB/s wr, 1.69k op/s 2024-09-17T07:09:00.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:00 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/978455207' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:09:00.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:00.561 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:00.561 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:09:00.662 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:09:00.700 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:00.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:00.702 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:09:00.803 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:09:00.836 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:00.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:00.837 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:09:00.938 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:09:00.968 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:00.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:00.970 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:09:01.053 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 45 16 82159 82143 114.057 65.75 0.0127094 0.00876264 2024-09-17T07:09:01.070 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:09:01.109 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:01.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:01.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:09:01.211 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:09:01.247 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:01.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:01.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:09:01.349 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:09:01.388 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:01.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:01.389 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:09:01.489 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:09:01.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:01.530 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:01.530 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:09:01.631 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:09:01.665 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:01.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:01.666 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:09:01.767 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:09:01.799 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:01.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:01.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:09:01.900 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:09:01.927 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:01.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:01.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:09:02.030 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:09:02.056 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 46 15 83479 83464 113.372 82.5625 0.0160218 0.0088159 2024-09-17T07:09:02.067 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:02.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:02.068 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:09:02.169 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:09:02.195 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:02.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:02.197 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:09:02.297 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:09:02.331 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:02.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:02.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:09:02.433 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:09:02.465 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:02.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:02.466 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:09:02.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:02 smithi135 ceph-mon[33238]: pgmap v297: 148 pgs: 148 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.0 KiB/s rd, 85 MiB/s wr, 1.36k op/s 2024-09-17T07:09:02.567 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:09:02.594 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:02.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:02.595 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:09:02.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:02 smithi100 ceph-mon[29092]: pgmap v297: 148 pgs: 148 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.0 KiB/s rd, 85 MiB/s wr, 1.36k op/s 2024-09-17T07:09:02.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:02 smithi100 ceph-mon[39462]: pgmap v297: 148 pgs: 148 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.0 KiB/s rd, 85 MiB/s wr, 1.36k op/s 2024-09-17T07:09:02.697 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:09:02.730 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:02.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:02.731 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:09:02.832 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:09:02.863 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:02.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:02.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:09:02.965 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:09:02.996 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:02.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:02.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:09:03.054 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 47 16 84766 84750 112.67 80.375 0.00984491 0.00887078 2024-09-17T07:09:03.098 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:09:03.123 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:03.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:03.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:09:03.225 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:09:03.260 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:03.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:03.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:09:03.362 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:09:03.397 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:03.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:03.398 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:09:03.499 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:09:03.520 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:03.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:03.521 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:09:03.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:03 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:09:03.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:03 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:09:03.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:03 smithi135 ceph-mon[33238]: osdmap e186: 8 total, 8 up, 8 in 2024-09-17T07:09:03.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:03 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:09:03.622 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:09:03.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:03 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:09:03.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:03 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:09:03.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:03 smithi100 ceph-mon[39462]: osdmap e186: 8 total, 8 up, 8 in 2024-09-17T07:09:03.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:03 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:09:03.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:03 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:09:03.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:03 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:09:03.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:03 smithi100 ceph-mon[29092]: osdmap e186: 8 total, 8 up, 8 in 2024-09-17T07:09:03.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:03 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:09:03.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:03.656 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:03.656 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:09:03.757 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:09:03.783 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:03.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:03.784 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:09:03.885 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:09:03.906 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:03.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:03.907 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:09:04.007 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:09:04.040 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:04.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:04.042 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:09:04.054 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 48 16 85784 85768 111.648 63.625 0.0122844 0.00895132 2024-09-17T07:09:04.143 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:09:04.178 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:04.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:04.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:09:04.280 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:09:04.321 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:04.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:04.322 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:09:04.423 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:09:04.458 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:04.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:04.459 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:09:04.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:04 smithi135 ceph-mon[33238]: pgmap v299: 148 pgs: 148 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 638 B/s rd, 64 MiB/s wr, 1.03k op/s 2024-09-17T07:09:04.559 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:09:04.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:04.581 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:04.581 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:09:04.631 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:04 smithi100 ceph-mon[39462]: pgmap v299: 148 pgs: 148 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 638 B/s rd, 64 MiB/s wr, 1.03k op/s 2024-09-17T07:09:04.632 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:04 smithi100 ceph-mon[29092]: pgmap v299: 148 pgs: 148 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 638 B/s rd, 64 MiB/s wr, 1.03k op/s 2024-09-17T07:09:04.682 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:09:04.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:04.718 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:04.718 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:09:04.819 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:09:04.852 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:04.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:04.853 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:09:04.954 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:09:04.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:04.993 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:04.994 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:09:05.054 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 49 16 87051 87035 110.986 79.1875 0.00752466 0.00900486 2024-09-17T07:09:05.095 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:09:05.131 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:05.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:05.133 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:09:05.233 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:09:05.268 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:05.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:05.269 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:09:05.370 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:09:05.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:05.398 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:05.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:09:05.500 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:09:05.544 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:05.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:05.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:09:05.646 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:09:05.678 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:05.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:05.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:09:05.780 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:09:05.827 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:05.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:05.828 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:09:05.929 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:09:05.970 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:05.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:05.971 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:09:06.021 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:09:05 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:09:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:09:06.054 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 50 15 88274 88259 110.296 76.5 0.0122091 0.00906215 2024-09-17T07:09:06.072 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:09:06.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:06.112 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:06.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:09:06.213 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:09:06.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:06.253 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:06.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:09:06.354 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:09:06.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:06 smithi100 ceph-mon[29092]: osdmap e187: 8 total, 8 up, 8 in 2024-09-17T07:09:06.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:06 smithi100 ceph-mon[29092]: pgmap v301: 148 pgs: 148 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 98 MiB/s wr, 1.57k op/s 2024-09-17T07:09:06.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:06 smithi100 ceph-mon[39462]: osdmap e187: 8 total, 8 up, 8 in 2024-09-17T07:09:06.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:06 smithi100 ceph-mon[39462]: pgmap v301: 148 pgs: 148 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 98 MiB/s wr, 1.57k op/s 2024-09-17T07:09:06.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:06.397 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:06.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:09:06.498 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:09:06.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:06.542 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:06.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:09:06.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:06 smithi135 ceph-mon[33238]: osdmap e187: 8 total, 8 up, 8 in 2024-09-17T07:09:06.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:06 smithi135 ceph-mon[33238]: pgmap v301: 148 pgs: 148 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 98 MiB/s wr, 1.57k op/s 2024-09-17T07:09:06.642 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:09:06.678 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:06.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:06.679 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:09:06.779 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:09:06.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:06.814 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:06.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:09:06.916 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:09:06.943 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:06.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:06.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:09:07.045 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:09:07.054 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 51 16 89430 89414 109.549 72.1875 0.0193718 0.00912349 2024-09-17T07:09:07.090 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:07.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:07.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:09:07.093 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:09:07.191 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:09:07.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:07.219 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:07.220 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:09:07.221 INFO:teuthology.orchestra.run.smithi100.stderr:noscrub is unset 2024-09-17T07:09:07.250 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:07 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/3467419663' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:09:07.251 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:07 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:09:07.252 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:07 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/3467419663' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:09:07.252 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:07 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:09:07.321 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:09:07.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:07.356 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:07.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:09:07.457 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:09:07.497 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:07.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:07.498 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:09:07.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:07 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/3467419663' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:09:07.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:07 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:09:07.599 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:09:07.629 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:07.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:07.630 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:09:07.731 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:09:07.750 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-17T07:09:07.750 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:09:07.758 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:07.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:07.759 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:09:07.860 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:09:07.894 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:07.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:07.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:09:07.996 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:09:08.027 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:08.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:08.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:09:08.054 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 52 16 90562 90546 108.803 70.75 0.0104309 0.0091863 2024-09-17T07:09:08.129 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:09:08.166 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:08.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:08.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:09:08.242 INFO:teuthology.orchestra.run.smithi100.stderr:set 2.15 pg_upmap_items mapping to [7->6,3->5,2->1] 2024-09-17T07:09:08.268 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:09:08.302 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:08.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:08.303 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:09:08.404 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:09:08.447 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:08.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:08.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:09:08.543 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:09:08 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:09:08.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:08 smithi135 ceph-mon[33238]: pgmap v302: 148 pgs: 148 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 46 MiB/s wr, 744 op/s 2024-09-17T07:09:08.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:08 smithi135 ceph-mon[33238]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:09:08.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:08 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T07:09:08.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:08 smithi135 ceph-mon[33238]: osdmap e188: 8 total, 8 up, 8 in 2024-09-17T07:09:08.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:08 smithi135 ceph-mon[33238]: 2.14 scrub starts 2024-09-17T07:09:08.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:08 smithi135 ceph-mon[33238]: 2.14 scrub ok 2024-09-17T07:09:08.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:08 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1304865063' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.15", "id": [7, 6, 3, 5, 2, 1]}]: dispatch 2024-09-17T07:09:08.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:08 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.15", "id": [7, 6, 3, 5, 2, 1]}]: dispatch 2024-09-17T07:09:08.548 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:09:08.586 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:08.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:08.587 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:09:08.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:08 smithi100 ceph-mon[39462]: pgmap v302: 148 pgs: 148 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 46 MiB/s wr, 744 op/s 2024-09-17T07:09:08.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:08 smithi100 ceph-mon[39462]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:09:08.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:08 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T07:09:08.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:08 smithi100 ceph-mon[39462]: osdmap e188: 8 total, 8 up, 8 in 2024-09-17T07:09:08.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:08 smithi100 ceph-mon[39462]: 2.14 scrub starts 2024-09-17T07:09:08.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:08 smithi100 ceph-mon[39462]: 2.14 scrub ok 2024-09-17T07:09:08.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:08 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1304865063' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.15", "id": [7, 6, 3, 5, 2, 1]}]: dispatch 2024-09-17T07:09:08.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:08 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.15", "id": [7, 6, 3, 5, 2, 1]}]: dispatch 2024-09-17T07:09:08.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:08 smithi100 ceph-mon[29092]: pgmap v302: 148 pgs: 148 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 46 MiB/s wr, 744 op/s 2024-09-17T07:09:08.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:08 smithi100 ceph-mon[29092]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:09:08.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:08 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T07:09:08.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:08 smithi100 ceph-mon[29092]: osdmap e188: 8 total, 8 up, 8 in 2024-09-17T07:09:08.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:08 smithi100 ceph-mon[29092]: 2.14 scrub starts 2024-09-17T07:09:08.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:08 smithi100 ceph-mon[29092]: 2.14 scrub ok 2024-09-17T07:09:08.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:08 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1304865063' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.15", "id": [7, 6, 3, 5, 2, 1]}]: dispatch 2024-09-17T07:09:08.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:08 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.15", "id": [7, 6, 3, 5, 2, 1]}]: dispatch 2024-09-17T07:09:08.687 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:09:08.727 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:08.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:08.728 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:09:08.829 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:09:08.870 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:08.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:08.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:09:08.972 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:09:09.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:09.013 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:09.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:09:09.055 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 53 16 91833 91817 108.248 79.4375 0.00888244 0.00923275 2024-09-17T07:09:09.114 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:09:09.151 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:09.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:09.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:09:09.253 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:09:09.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:09.294 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:09.294 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:09:09.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:09 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "2.15", "id": [7, 6, 3, 5, 2, 1]}]': finished 2024-09-17T07:09:09.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:09 smithi100 ceph-mon[29092]: osdmap e189: 8 total, 8 up, 8 in 2024-09-17T07:09:09.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:09 smithi100 ceph-mon[29092]: 6.9 scrub starts 2024-09-17T07:09:09.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:09 smithi100 ceph-mon[29092]: 6.9 scrub ok 2024-09-17T07:09:09.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:09 smithi100 ceph-mon[29092]: 2.1c scrub starts 2024-09-17T07:09:09.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:09 smithi100 ceph-mon[29092]: 2.1c scrub ok 2024-09-17T07:09:09.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:09 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "2.15", "id": [7, 6, 3, 5, 2, 1]}]': finished 2024-09-17T07:09:09.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:09 smithi100 ceph-mon[39462]: osdmap e189: 8 total, 8 up, 8 in 2024-09-17T07:09:09.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:09 smithi100 ceph-mon[39462]: 6.9 scrub starts 2024-09-17T07:09:09.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:09 smithi100 ceph-mon[39462]: 6.9 scrub ok 2024-09-17T07:09:09.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:09 smithi100 ceph-mon[39462]: 2.1c scrub starts 2024-09-17T07:09:09.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:09 smithi100 ceph-mon[39462]: 2.1c scrub ok 2024-09-17T07:09:09.395 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:09:09.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:09.429 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:09.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:09:09.525 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-17T07:09:09.530 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:09:09.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:09 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "2.15", "id": [7, 6, 3, 5, 2, 1]}]': finished 2024-09-17T07:09:09.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:09 smithi135 ceph-mon[33238]: osdmap e189: 8 total, 8 up, 8 in 2024-09-17T07:09:09.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:09 smithi135 ceph-mon[33238]: 6.9 scrub starts 2024-09-17T07:09:09.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:09 smithi135 ceph-mon[33238]: 6.9 scrub ok 2024-09-17T07:09:09.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:09 smithi135 ceph-mon[33238]: 2.1c scrub starts 2024-09-17T07:09:09.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:09 smithi135 ceph-mon[33238]: 2.1c scrub ok 2024-09-17T07:09:09.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:09.614 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:09.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:09:09.716 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:09:09.743 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:09.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:09.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:09:09.845 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:09:09.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:09.889 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:09.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:09:09.990 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:09:10.029 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:10.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:10.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:09:10.055 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 54 16 93098 93082 107.708 79.0625 0.00805888 0.00927908 2024-09-17T07:09:10.131 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:09:10.144 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:09:10.174 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:10.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:10.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:09:10.265 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-17T07:09:10.277 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:09:10.289 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:10 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:09:10.289 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:10 smithi100 ceph-mon[29092]: pgmap v305: 148 pgs: 148 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.7 KiB/s rd, 107 MiB/s wr, 1.72k op/s 2024-09-17T07:09:10.289 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:10 smithi100 ceph-mon[29092]: 4.15 scrub starts 2024-09-17T07:09:10.289 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:10 smithi100 ceph-mon[29092]: 4.15 scrub ok 2024-09-17T07:09:10.289 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:10 smithi100 ceph-mon[29092]: 2.1d scrub starts 2024-09-17T07:09:10.289 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:10 smithi100 ceph-mon[29092]: 2.1d scrub ok 2024-09-17T07:09:10.290 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:10 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:09:10.290 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:10 smithi100 ceph-mon[39462]: pgmap v305: 148 pgs: 148 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.7 KiB/s rd, 107 MiB/s wr, 1.72k op/s 2024-09-17T07:09:10.290 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:10 smithi100 ceph-mon[39462]: 4.15 scrub starts 2024-09-17T07:09:10.290 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:10 smithi100 ceph-mon[39462]: 4.15 scrub ok 2024-09-17T07:09:10.290 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:10 smithi100 ceph-mon[39462]: 2.1d scrub starts 2024-09-17T07:09:10.290 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:10 smithi100 ceph-mon[39462]: 2.1d scrub ok 2024-09-17T07:09:10.332 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:10.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:10.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:09:10.434 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:09:10.442 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:09:10.461 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:10.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:10.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:09:10.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:10 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:09:10.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:10 smithi135 ceph-mon[33238]: pgmap v305: 148 pgs: 148 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.7 KiB/s rd, 107 MiB/s wr, 1.72k op/s 2024-09-17T07:09:10.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:10 smithi135 ceph-mon[33238]: 4.15 scrub starts 2024-09-17T07:09:10.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:10 smithi135 ceph-mon[33238]: 4.15 scrub ok 2024-09-17T07:09:10.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:10 smithi135 ceph-mon[33238]: 2.1d scrub starts 2024-09-17T07:09:10.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:10 smithi135 ceph-mon[33238]: 2.1d scrub ok 2024-09-17T07:09:10.563 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:09:10.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:10.600 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:10.600 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:09:10.701 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:09:10.741 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:10.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:10.742 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:09:10.843 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:09:10.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:10.864 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:10.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:09:10.965 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:09:11.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:11.002 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:11.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:09:11.055 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 55 16 94180 94164 106.979 67.625 0.0180374 0.00934297 2024-09-17T07:09:11.103 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:09:11.139 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:11.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:11.140 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:09:11.241 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:09:11.262 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:11.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:11.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:09:11.364 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:09:11.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:11 smithi100 ceph-mon[29092]: 5.1e scrub starts 2024-09-17T07:09:11.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:11 smithi100 ceph-mon[29092]: 5.1e scrub ok 2024-09-17T07:09:11.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:11 smithi100 ceph-mon[29092]: 6.8 scrub starts 2024-09-17T07:09:11.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:11 smithi100 ceph-mon[29092]: 6.8 scrub ok 2024-09-17T07:09:11.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:11 smithi100 ceph-mon[39462]: 5.1e scrub starts 2024-09-17T07:09:11.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:11 smithi100 ceph-mon[39462]: 5.1e scrub ok 2024-09-17T07:09:11.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:11 smithi100 ceph-mon[39462]: 6.8 scrub starts 2024-09-17T07:09:11.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:11 smithi100 ceph-mon[39462]: 6.8 scrub ok 2024-09-17T07:09:11.394 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:11.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:11.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:09:11.496 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:09:11.525 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:11.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:11.526 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:09:11.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:11 smithi135 ceph-mon[33238]: 5.1e scrub starts 2024-09-17T07:09:11.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:11 smithi135 ceph-mon[33238]: 5.1e scrub ok 2024-09-17T07:09:11.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:11 smithi135 ceph-mon[33238]: 6.8 scrub starts 2024-09-17T07:09:11.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:11 smithi135 ceph-mon[33238]: 6.8 scrub ok 2024-09-17T07:09:11.627 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:09:11.661 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:11.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:11.662 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:09:11.763 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:09:11.796 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:11.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:11.797 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:09:11.897 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:09:11.934 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:11.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:11.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:09:12.036 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:09:12.056 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 56 16 95308 95292 106.325 70.5 0.0328086 0.00939966 2024-09-17T07:09:12.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:12.073 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:12.074 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:09:12.175 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:09:12.208 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:12.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:12.209 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:09:12.310 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:09:12.334 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:12.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:12.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:09:12.437 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:09:12.470 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:12.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:12.471 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:09:12.530 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:12 smithi100 ceph-mon[29092]: pgmap v306: 148 pgs: 148 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 851 B/s rd, 62 MiB/s wr, 987 op/s 2024-09-17T07:09:12.530 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:12 smithi100 ceph-mon[29092]: 4.a scrub starts 2024-09-17T07:09:12.530 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:12 smithi100 ceph-mon[29092]: 4.a scrub ok 2024-09-17T07:09:12.530 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:12 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:09:12.530 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:12 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:09:12.531 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:12 smithi100 ceph-mon[39462]: pgmap v306: 148 pgs: 148 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 851 B/s rd, 62 MiB/s wr, 987 op/s 2024-09-17T07:09:12.531 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:12 smithi100 ceph-mon[39462]: 4.a scrub starts 2024-09-17T07:09:12.531 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:12 smithi100 ceph-mon[39462]: 4.a scrub ok 2024-09-17T07:09:12.531 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:12 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:09:12.531 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:12 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:09:12.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:12 smithi135 ceph-mon[33238]: pgmap v306: 148 pgs: 148 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 851 B/s rd, 62 MiB/s wr, 987 op/s 2024-09-17T07:09:12.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:12 smithi135 ceph-mon[33238]: 4.a scrub starts 2024-09-17T07:09:12.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:12 smithi135 ceph-mon[33238]: 4.a scrub ok 2024-09-17T07:09:12.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:12 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:09:12.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:12 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:09:12.573 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:09:12.602 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:12.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:12.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:09:12.703 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:09:12.733 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:12.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:12.734 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:09:12.835 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:09:12.878 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:12.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:12.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:09:12.980 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:09:13.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:13.020 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:13.020 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:09:13.061 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 57 16 96452 96436 105.709 71.5 0.0118891 0.00945498 2024-09-17T07:09:13.121 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:09:13.155 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:13.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:13.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:09:13.257 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:09:13.288 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:13.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:13.289 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:09:13.390 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:09:13.432 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:13.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:13.433 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:09:13.534 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:09:13.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:13 smithi135 ceph-mon[33238]: osdmap e190: 8 total, 8 up, 8 in 2024-09-17T07:09:13.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:13 smithi135 ceph-mon[33238]: 6.16 scrub starts 2024-09-17T07:09:13.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:13 smithi135 ceph-mon[33238]: 6.16 scrub ok 2024-09-17T07:09:13.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:13 smithi135 ceph-mon[33238]: 4.1d scrub starts 2024-09-17T07:09:13.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:13 smithi135 ceph-mon[33238]: 4.1d scrub ok 2024-09-17T07:09:13.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:13 smithi135 ceph-mon[33238]: pgmap v308: 148 pgs: 148 active+clean; 5.6 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 77 MiB/s wr, 1.23k op/s 2024-09-17T07:09:13.564 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:13.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:13.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:09:13.581 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:13 smithi100 ceph-mon[39462]: osdmap e190: 8 total, 8 up, 8 in 2024-09-17T07:09:13.581 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:13 smithi100 ceph-mon[39462]: 6.16 scrub starts 2024-09-17T07:09:13.581 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:13 smithi100 ceph-mon[39462]: 6.16 scrub ok 2024-09-17T07:09:13.581 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:13 smithi100 ceph-mon[39462]: 4.1d scrub starts 2024-09-17T07:09:13.581 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:13 smithi100 ceph-mon[39462]: 4.1d scrub ok 2024-09-17T07:09:13.582 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:13 smithi100 ceph-mon[39462]: pgmap v308: 148 pgs: 148 active+clean; 5.6 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 77 MiB/s wr, 1.23k op/s 2024-09-17T07:09:13.585 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:13 smithi100 ceph-mon[29092]: osdmap e190: 8 total, 8 up, 8 in 2024-09-17T07:09:13.585 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:13 smithi100 ceph-mon[29092]: 6.16 scrub starts 2024-09-17T07:09:13.585 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:13 smithi100 ceph-mon[29092]: 6.16 scrub ok 2024-09-17T07:09:13.585 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:13 smithi100 ceph-mon[29092]: 4.1d scrub starts 2024-09-17T07:09:13.585 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:13 smithi100 ceph-mon[29092]: 4.1d scrub ok 2024-09-17T07:09:13.585 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:13 smithi100 ceph-mon[29092]: pgmap v308: 148 pgs: 148 active+clean; 5.6 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 77 MiB/s wr, 1.23k op/s 2024-09-17T07:09:13.593 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:09:13.666 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:09:13.697 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:13.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:13.698 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:09:13.799 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:09:13.839 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:13.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:13.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:09:13.941 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:09:13.975 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:13.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:13.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:09:14.059 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 58 16 97711 97695 105.243 78.6875 0.0101459 0.0094968 2024-09-17T07:09:14.077 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:09:14.098 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:14.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:14.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:09:14.200 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:09:14.231 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:14.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:14.232 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:09:14.333 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:09:14.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:14.367 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:14.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:09:14.468 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:09:14.506 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:14.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:14.507 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:09:14.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:14 smithi135 ceph-mon[33238]: 4.1e scrub starts 2024-09-17T07:09:14.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:14 smithi135 ceph-mon[33238]: 4.1e scrub ok 2024-09-17T07:09:14.607 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:09:14.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:14 smithi100 ceph-mon[29092]: 4.1e scrub starts 2024-09-17T07:09:14.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:14 smithi100 ceph-mon[29092]: 4.1e scrub ok 2024-09-17T07:09:14.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:14 smithi100 ceph-mon[39462]: 4.1e scrub starts 2024-09-17T07:09:14.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:14 smithi100 ceph-mon[39462]: 4.1e scrub ok 2024-09-17T07:09:14.648 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:14.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:14.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:09:14.749 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:09:14.776 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:14.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:14.777 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:09:14.877 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:09:14.913 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:14.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:14.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:09:15.015 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:09:15.051 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:15.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:15.052 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:09:15.059 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 59 16 98869 98853 104.686 72.375 0.0177878 0.00954751 2024-09-17T07:09:15.070 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-17T07:09:15.071 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T07:09:15.071 INFO:tasks.thrashosds.thrasher:Removing osd 3, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-17T07:09:15.071 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd out 3 2024-09-17T07:09:15.153 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:09:15.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:15.193 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:15.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:09:15.294 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:09:15.343 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:15.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:15.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:09:15.370 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:15 smithi100 ceph-mon[29092]: 5.0 scrub starts 2024-09-17T07:09:15.371 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:15 smithi100 ceph-mon[29092]: 5.0 scrub ok 2024-09-17T07:09:15.371 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:15 smithi100 ceph-mon[29092]: 4.1c scrub starts 2024-09-17T07:09:15.371 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:15 smithi100 ceph-mon[29092]: 4.1c scrub ok 2024-09-17T07:09:15.371 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:15 smithi100 ceph-mon[29092]: pgmap v309: 148 pgs: 148 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.3 KiB/s rd, 93 MiB/s wr, 1.49k op/s 2024-09-17T07:09:15.371 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:15 smithi100 ceph-mon[39462]: 5.0 scrub starts 2024-09-17T07:09:15.371 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:15 smithi100 ceph-mon[39462]: 5.0 scrub ok 2024-09-17T07:09:15.371 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:15 smithi100 ceph-mon[39462]: 4.1c scrub starts 2024-09-17T07:09:15.371 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:15 smithi100 ceph-mon[39462]: 4.1c scrub ok 2024-09-17T07:09:15.371 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:15 smithi100 ceph-mon[39462]: pgmap v309: 148 pgs: 148 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.3 KiB/s rd, 93 MiB/s wr, 1.49k op/s 2024-09-17T07:09:15.445 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:09:15.478 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:15.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:15.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:09:15.580 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:09:15.613 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:15.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:15.614 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:09:15.715 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:09:15.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:15.753 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:15.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:09:15.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:15 smithi135 ceph-mon[33238]: 5.0 scrub starts 2024-09-17T07:09:15.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:15 smithi135 ceph-mon[33238]: 5.0 scrub ok 2024-09-17T07:09:15.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:15 smithi135 ceph-mon[33238]: 4.1c scrub starts 2024-09-17T07:09:15.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:15 smithi135 ceph-mon[33238]: 4.1c scrub ok 2024-09-17T07:09:15.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:15 smithi135 ceph-mon[33238]: pgmap v309: 148 pgs: 148 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.3 KiB/s rd, 93 MiB/s wr, 1.49k op/s 2024-09-17T07:09:15.855 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:09:15.918 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:15.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:15.919 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:09:16.019 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:09:16.047 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:09:15 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:09:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:09:16.050 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:16.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:16.051 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:09:16.059 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:2024-09-17T07:09:16.060289+0000 min lat: 0.0012044 max lat: 0.142002 avg lat: 0.00959176 2024-09-17T07:09:16.060 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-17T07:09:16.060 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 60 16 100081 100065 104.203 75.75 0.00862674 0.00959176 2024-09-17T07:09:16.152 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:09:16.185 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:16.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:16.186 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:09:16.286 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:09:16.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:16.322 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:16.322 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:09:16.423 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:09:16.453 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:16.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:16.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:09:16.555 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:09:16.580 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:16.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:16.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:09:16.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:16 smithi100 ceph-mon[29092]: 6.7 scrub starts 2024-09-17T07:09:16.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:16 smithi100 ceph-mon[29092]: 6.7 scrub ok 2024-09-17T07:09:16.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:16 smithi100 ceph-mon[29092]: 5.2 scrub starts 2024-09-17T07:09:16.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:16 smithi100 ceph-mon[39462]: 6.7 scrub starts 2024-09-17T07:09:16.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:16 smithi100 ceph-mon[39462]: 6.7 scrub ok 2024-09-17T07:09:16.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:16 smithi100 ceph-mon[39462]: 5.2 scrub starts 2024-09-17T07:09:16.682 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:09:16.714 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:16.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:16.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:09:16.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:16 smithi135 ceph-mon[33238]: 6.7 scrub starts 2024-09-17T07:09:16.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:16 smithi135 ceph-mon[33238]: 6.7 scrub ok 2024-09-17T07:09:16.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:16 smithi135 ceph-mon[33238]: 5.2 scrub starts 2024-09-17T07:09:16.816 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:09:16.843 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:16.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:16.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:09:16.945 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:09:16.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:16.977 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:16.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:09:17.060 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 61 16 101255 101239 103.698 73.375 0.00778099 0.00963867 2024-09-17T07:09:17.078 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:09:17.099 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:17.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:17.101 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:09:17.202 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:09:17.244 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:17.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:17.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:09:17.346 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:09:17.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:17.372 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:17.373 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:09:17.473 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:09:17.505 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:17.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:17.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:09:17.577 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:17 smithi100 ceph-mon[29092]: 5.2 scrub ok 2024-09-17T07:09:17.577 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:17 smithi100 ceph-mon[29092]: 2.3 scrub starts 2024-09-17T07:09:17.577 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:17 smithi100 ceph-mon[29092]: 2.3 scrub ok 2024-09-17T07:09:17.577 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:17 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:09:17.577 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:17 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:09:17.577 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:17 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1829444278' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:09:17.577 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:17 smithi100 ceph-mon[29092]: 2.0 scrub starts 2024-09-17T07:09:17.578 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:17 smithi100 ceph-mon[29092]: 2.0 scrub ok 2024-09-17T07:09:17.578 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:17 smithi100 ceph-mon[29092]: 6.4 scrub starts 2024-09-17T07:09:17.578 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:17 smithi100 ceph-mon[29092]: pgmap v310: 148 pgs: 148 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 67 MiB/s wr, 1.08k op/s 2024-09-17T07:09:17.578 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:17 smithi100 ceph-mon[29092]: 6.4 scrub ok 2024-09-17T07:09:17.578 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:17 smithi100 ceph-mon[39462]: 5.2 scrub ok 2024-09-17T07:09:17.578 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:17 smithi100 ceph-mon[39462]: 2.3 scrub starts 2024-09-17T07:09:17.578 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:17 smithi100 ceph-mon[39462]: 2.3 scrub ok 2024-09-17T07:09:17.578 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:17 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:09:17.579 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:17 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:09:17.579 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:17 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1829444278' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:09:17.579 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:17 smithi100 ceph-mon[39462]: 2.0 scrub starts 2024-09-17T07:09:17.579 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:17 smithi100 ceph-mon[39462]: 2.0 scrub ok 2024-09-17T07:09:17.579 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:17 smithi100 ceph-mon[39462]: 6.4 scrub starts 2024-09-17T07:09:17.579 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:17 smithi100 ceph-mon[39462]: pgmap v310: 148 pgs: 148 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 67 MiB/s wr, 1.08k op/s 2024-09-17T07:09:17.579 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:17 smithi100 ceph-mon[39462]: 6.4 scrub ok 2024-09-17T07:09:17.607 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:09:17.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:17.638 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:17.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:09:17.641 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-17T07:09:17.641 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-17T07:09:17.641 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-17T07:09:17.641 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-17T07:09:17.641 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-17T07:09:17.698 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:09:17.699 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1: {} 2024-09-17T07:09:17.699 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:09:17.699 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:09:17.699 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:09:17.699 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:09:17.699 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:09:17.699 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:09:17.699 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:09:17.700 INFO:teuthology.orchestra.run.smithi100.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:09:17.700 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:09:17.700 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:09:17.700 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:09:17.700 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:09:17.700 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:09:17.700 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:09:17.740 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:09:17.771 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:17.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:17.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:09:17.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:17 smithi135 ceph-mon[33238]: 5.2 scrub ok 2024-09-17T07:09:17.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:17 smithi135 ceph-mon[33238]: 2.3 scrub starts 2024-09-17T07:09:17.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:17 smithi135 ceph-mon[33238]: 2.3 scrub ok 2024-09-17T07:09:17.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:17 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:09:17.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:17 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:09:17.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:17 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1829444278' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:09:17.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:17 smithi135 ceph-mon[33238]: 2.0 scrub starts 2024-09-17T07:09:17.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:17 smithi135 ceph-mon[33238]: 2.0 scrub ok 2024-09-17T07:09:17.795 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:17 smithi135 ceph-mon[33238]: 6.4 scrub starts 2024-09-17T07:09:17.795 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:17 smithi135 ceph-mon[33238]: pgmap v310: 148 pgs: 148 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 67 MiB/s wr, 1.08k op/s 2024-09-17T07:09:17.795 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:17 smithi135 ceph-mon[33238]: 6.4 scrub ok 2024-09-17T07:09:17.874 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:09:17.910 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:17.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:17.911 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:09:18.012 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:09:18.050 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:18.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:18.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:09:18.060 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 62 15 102347 102332 103.127 68.3125 0.00650079 0.00969276 2024-09-17T07:09:18.151 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:09:18.181 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:18.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:18.182 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:09:18.283 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:09:18.319 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:18 smithi100 ceph-mon[29092]: osdmap e191: 8 total, 8 up, 8 in 2024-09-17T07:09:18.319 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:18 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:09:18.320 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:18 smithi100 ceph-mon[39462]: osdmap e191: 8 total, 8 up, 8 in 2024-09-17T07:09:18.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:18.322 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:18.323 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:09:18.423 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:09:18.457 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:18.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:18.458 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:09:18.559 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:09:18.591 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:18.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:18.592 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:09:18.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:18 smithi100 ceph-mon[29092]: 4.d scrub starts 2024-09-17T07:09:18.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:18 smithi100 ceph-mon[29092]: 4.d scrub ok 2024-09-17T07:09:18.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:18 smithi100 ceph-mon[29092]: 4.10 scrub starts 2024-09-17T07:09:18.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:18 smithi100 ceph-mon[29092]: 4.10 scrub ok 2024-09-17T07:09:18.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:18 smithi100 ceph-mon[29092]: 2.15 scrub starts 2024-09-17T07:09:18.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:18 smithi100 ceph-mon[29092]: 2.15 scrub ok 2024-09-17T07:09:18.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:18 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:09:18.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:18 smithi100 ceph-mon[39462]: 4.d scrub starts 2024-09-17T07:09:18.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:18 smithi100 ceph-mon[39462]: 4.d scrub ok 2024-09-17T07:09:18.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:18 smithi100 ceph-mon[39462]: 4.10 scrub starts 2024-09-17T07:09:18.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:18 smithi100 ceph-mon[39462]: 4.10 scrub ok 2024-09-17T07:09:18.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:18 smithi100 ceph-mon[39462]: 2.15 scrub starts 2024-09-17T07:09:18.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:18 smithi100 ceph-mon[39462]: 2.15 scrub ok 2024-09-17T07:09:18.693 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:09:18.726 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:18.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:18.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:09:18.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:18 smithi135 ceph-mon[33238]: osdmap e191: 8 total, 8 up, 8 in 2024-09-17T07:09:18.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:18 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:09:18.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:18 smithi135 ceph-mon[33238]: 4.d scrub starts 2024-09-17T07:09:18.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:18 smithi135 ceph-mon[33238]: 4.d scrub ok 2024-09-17T07:09:18.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:18 smithi135 ceph-mon[33238]: 4.10 scrub starts 2024-09-17T07:09:18.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:18 smithi135 ceph-mon[33238]: 4.10 scrub ok 2024-09-17T07:09:18.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:18 smithi135 ceph-mon[33238]: 2.15 scrub starts 2024-09-17T07:09:18.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:18 smithi135 ceph-mon[33238]: 2.15 scrub ok 2024-09-17T07:09:18.794 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:09:18 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:09:18.829 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:09:18.862 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:18.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:18.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:09:18.964 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:09:19.005 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:19.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:19.006 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:09:19.060 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 63 16 103623 103607 102.755 79.6875 0.00893194 0.00972694 2024-09-17T07:09:19.107 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:09:19.144 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:19.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:19.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:09:19.246 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:09:19.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:19.282 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:19.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:09:19.383 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:09:19.413 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:19.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:19.414 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:09:19.514 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:09:19.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:19.554 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:19.554 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:09:19.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:19 smithi100 ceph-mon[29092]: 4.1f scrub starts 2024-09-17T07:09:19.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:19 smithi100 ceph-mon[29092]: 4.1f scrub ok 2024-09-17T07:09:19.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:19 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:09:19.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:19 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:09:19.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:19 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:09:19.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:19 smithi100 ceph-mon[29092]: pgmap v312: 148 pgs: 148 active+clean; 6.3 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 92 MiB/s wr, 1.47k op/s 2024-09-17T07:09:19.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:19 smithi100 ceph-mon[39462]: 4.1f scrub starts 2024-09-17T07:09:19.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:19 smithi100 ceph-mon[39462]: 4.1f scrub ok 2024-09-17T07:09:19.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:19 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:09:19.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:19 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:09:19.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:19 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:09:19.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:19 smithi100 ceph-mon[39462]: pgmap v312: 148 pgs: 148 active+clean; 6.3 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 92 MiB/s wr, 1.47k op/s 2024-09-17T07:09:19.655 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:09:19.690 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:19.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:19.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:09:19.792 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:09:19.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:19 smithi135 ceph-mon[33238]: 4.1f scrub starts 2024-09-17T07:09:19.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:19 smithi135 ceph-mon[33238]: 4.1f scrub ok 2024-09-17T07:09:19.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:19 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:09:19.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:19 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:09:19.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:19 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:09:19.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:19 smithi135 ceph-mon[33238]: pgmap v312: 148 pgs: 148 active+clean; 6.3 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 92 MiB/s wr, 1.47k op/s 2024-09-17T07:09:19.799 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-17T07:09:19.821 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:19.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:19.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:09:19.831 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:09:19.923 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:09:19.959 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:19.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:19.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:09:20.060 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 64 15 104839 104824 102.338 76.0625 0.00999282 0.00976682 2024-09-17T07:09:20.060 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:09:20.111 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:20.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:20.112 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:09:20.212 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:09:20.247 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:20.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:20.248 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:09:20.349 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:09:20.381 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:20.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:20.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:09:20.482 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:09:20.519 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:20.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:20.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:09:20.540 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:09:20.597 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:20 smithi100 ceph-mon[29092]: 2.2 scrub starts 2024-09-17T07:09:20.597 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:20 smithi100 ceph-mon[29092]: 2.2 scrub ok 2024-09-17T07:09:20.597 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:20 smithi100 ceph-mon[29092]: osdmap e192: 8 total, 8 up, 8 in 2024-09-17T07:09:20.598 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:20 smithi100 ceph-mon[29092]: 2.e scrub starts 2024-09-17T07:09:20.598 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:20 smithi100 ceph-mon[29092]: 2.e scrub ok 2024-09-17T07:09:20.598 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:20 smithi100 ceph-mon[29092]: 2.9 scrub starts 2024-09-17T07:09:20.598 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:20 smithi100 ceph-mon[29092]: 2.9 scrub ok 2024-09-17T07:09:20.598 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:20 smithi100 ceph-mon[39462]: 2.2 scrub starts 2024-09-17T07:09:20.598 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:20 smithi100 ceph-mon[39462]: 2.2 scrub ok 2024-09-17T07:09:20.598 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:20 smithi100 ceph-mon[39462]: osdmap e192: 8 total, 8 up, 8 in 2024-09-17T07:09:20.598 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:20 smithi100 ceph-mon[39462]: 2.e scrub starts 2024-09-17T07:09:20.598 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:20 smithi100 ceph-mon[39462]: 2.e scrub ok 2024-09-17T07:09:20.598 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:20 smithi100 ceph-mon[39462]: 2.9 scrub starts 2024-09-17T07:09:20.598 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:20 smithi100 ceph-mon[39462]: 2.9 scrub ok 2024-09-17T07:09:20.621 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:09:20.652 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:20.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:20.653 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:09:20.754 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:09:20.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:20 smithi135 ceph-mon[33238]: 2.2 scrub starts 2024-09-17T07:09:20.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:20 smithi135 ceph-mon[33238]: 2.2 scrub ok 2024-09-17T07:09:20.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:20 smithi135 ceph-mon[33238]: osdmap e192: 8 total, 8 up, 8 in 2024-09-17T07:09:20.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:20 smithi135 ceph-mon[33238]: 2.e scrub starts 2024-09-17T07:09:20.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:20 smithi135 ceph-mon[33238]: 2.e scrub ok 2024-09-17T07:09:20.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:20 smithi135 ceph-mon[33238]: 2.9 scrub starts 2024-09-17T07:09:20.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:20 smithi135 ceph-mon[33238]: 2.9 scrub ok 2024-09-17T07:09:20.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:20.806 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:20.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:09:20.907 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:09:20.942 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:20.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:20.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:09:21.044 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:09:21.060 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 65 16 105951 105935 101.832 69.4375 0.0229949 0.00981507 2024-09-17T07:09:21.071 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:21.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:21.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:09:21.173 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:09:21.202 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:21.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:21.203 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:09:21.304 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:09:21.336 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:21.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:21.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:09:21.437 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:09:21.465 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:21 smithi100 ceph-mon[29092]: pgmap v314: 148 pgs: 148 active+clean; 6.3 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.29k op/s 2024-09-17T07:09:21.465 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:21 smithi100 ceph-mon[39462]: pgmap v314: 148 pgs: 148 active+clean; 6.3 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.29k op/s 2024-09-17T07:09:21.477 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:21.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:21.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:09:21.578 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:09:21.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:21.617 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:21.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:09:21.718 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:09:21.757 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:21.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:21.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:09:21.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:21 smithi135 ceph-mon[33238]: pgmap v314: 148 pgs: 148 active+clean; 6.3 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.29k op/s 2024-09-17T07:09:21.858 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:09:21.896 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:21.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:21.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:09:21.998 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:09:22.056 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:22.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:22.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:09:22.062 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 66 16 107116 107100 101.389 72.8125 0.0193639 0.00985882 2024-09-17T07:09:22.158 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:09:22.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:22.198 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:22.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:09:22.299 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:09:22.343 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:22.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:22.344 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:09:22.361 INFO:teuthology.orchestra.run.smithi100.stderr:nodeep-scrub is unset 2024-09-17T07:09:22.445 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:09:22.484 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:22.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:22.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:09:22.509 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:22 smithi100 ceph-mon[29092]: 8.9 scrub starts 2024-09-17T07:09:22.510 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:22 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2603482201' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:09:22.510 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:22 smithi100 ceph-mon[29092]: 6.f scrub starts 2024-09-17T07:09:22.510 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:22 smithi100 ceph-mon[29092]: 6.f scrub ok 2024-09-17T07:09:22.510 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:22 smithi100 ceph-mon[39462]: 8.9 scrub starts 2024-09-17T07:09:22.510 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:22 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2603482201' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:09:22.511 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:22 smithi100 ceph-mon[39462]: 6.f scrub starts 2024-09-17T07:09:22.511 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:22 smithi100 ceph-mon[39462]: 6.f scrub ok 2024-09-17T07:09:22.586 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:09:22.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:22.613 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:22.613 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:09:22.714 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:09:22.743 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:22.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:22.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:09:22.760 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:09:22.767 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:09:22.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:22 smithi135 ceph-mon[33238]: 8.9 scrub starts 2024-09-17T07:09:22.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:22 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2603482201' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:09:22.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:22 smithi135 ceph-mon[33238]: 6.f scrub starts 2024-09-17T07:09:22.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:22 smithi135 ceph-mon[33238]: 6.f scrub ok 2024-09-17T07:09:22.846 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:09:22.885 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:22.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:22.886 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:09:22.987 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:09:23.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:23.022 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:23.022 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:09:23.063 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 67 16 108198 108182 100.885 67.625 0.00440031 0.00990778 2024-09-17T07:09:23.123 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:09:23.162 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:23.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:23.163 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:09:23.264 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:09:23.267 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:22 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mon-a[29088]: 2024-09-17T07:09:22.893+0000 7f59b37e7640 -1 mon.a@0(leader).osd e193 definitely_dead 0 2024-09-17T07:09:23.298 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:23.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:23.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:09:23.394 INFO:teuthology.orchestra.run.smithi100.stderr:marked out osd.3. 2024-09-17T07:09:23.401 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:09:23.425 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:23.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:23.426 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:09:23.527 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:09:23.548 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:23.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:23.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:09:23.567 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:23 smithi100 ceph-mon[29092]: 2.1a scrub starts 2024-09-17T07:09:23.567 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:23 smithi100 ceph-mon[29092]: 2.1a scrub ok 2024-09-17T07:09:23.567 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:23 smithi100 ceph-mon[29092]: 3.1 scrub starts 2024-09-17T07:09:23.567 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:23 smithi100 ceph-mon[29092]: 3.1 scrub ok 2024-09-17T07:09:23.567 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:23 smithi100 ceph-mon[29092]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T07:09:23.567 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:23 smithi100 ceph-mon[29092]: Cluster is now healthy 2024-09-17T07:09:23.568 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:23 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2603482201' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T07:09:23.568 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:23 smithi100 ceph-mon[29092]: osdmap e193: 8 total, 8 up, 8 in 2024-09-17T07:09:23.568 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:23 smithi100 ceph-mon[29092]: 6.5 scrub starts 2024-09-17T07:09:23.568 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:23 smithi100 ceph-mon[29092]: 6.5 scrub ok 2024-09-17T07:09:23.568 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:23 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/227675106' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-09-17T07:09:23.568 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:23 smithi100 ceph-mon[29092]: Client client.admin marked osd.3 out, while it was still marked up 2024-09-17T07:09:23.568 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:23 smithi100 ceph-mon[29092]: 8.9 scrub ok 2024-09-17T07:09:23.568 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:23 smithi100 ceph-mon[29092]: pgmap v316: 148 pgs: 148 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 76 MiB/s wr, 1.21k op/s 2024-09-17T07:09:23.568 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:23 smithi100 ceph-mon[29092]: 2.7 deep-scrub starts 2024-09-17T07:09:23.568 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:23 smithi100 ceph-mon[29092]: 2.7 deep-scrub ok 2024-09-17T07:09:23.568 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:23 smithi100 ceph-mon[39462]: 2.1a scrub starts 2024-09-17T07:09:23.569 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:23 smithi100 ceph-mon[39462]: 2.1a scrub ok 2024-09-17T07:09:23.569 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:23 smithi100 ceph-mon[39462]: 3.1 scrub starts 2024-09-17T07:09:23.569 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:23 smithi100 ceph-mon[39462]: 3.1 scrub ok 2024-09-17T07:09:23.569 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:23 smithi100 ceph-mon[39462]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T07:09:23.569 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:23 smithi100 ceph-mon[39462]: Cluster is now healthy 2024-09-17T07:09:23.569 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:23 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2603482201' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T07:09:23.569 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:23 smithi100 ceph-mon[39462]: osdmap e193: 8 total, 8 up, 8 in 2024-09-17T07:09:23.570 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:23 smithi100 ceph-mon[39462]: 6.5 scrub starts 2024-09-17T07:09:23.570 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:23 smithi100 ceph-mon[39462]: 6.5 scrub ok 2024-09-17T07:09:23.570 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:23 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/227675106' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-09-17T07:09:23.570 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:23 smithi100 ceph-mon[39462]: Client client.admin marked osd.3 out, while it was still marked up 2024-09-17T07:09:23.570 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:23 smithi100 ceph-mon[39462]: 8.9 scrub ok 2024-09-17T07:09:23.570 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:23 smithi100 ceph-mon[39462]: pgmap v316: 148 pgs: 148 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 76 MiB/s wr, 1.21k op/s 2024-09-17T07:09:23.571 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:23 smithi100 ceph-mon[39462]: 2.7 deep-scrub starts 2024-09-17T07:09:23.571 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:23 smithi100 ceph-mon[39462]: 2.7 deep-scrub ok 2024-09-17T07:09:23.650 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:09:23.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:23.674 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:23.674 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:09:23.775 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:09:23.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:23 smithi135 ceph-mon[33238]: 2.1a scrub starts 2024-09-17T07:09:23.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:23 smithi135 ceph-mon[33238]: 2.1a scrub ok 2024-09-17T07:09:23.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:23 smithi135 ceph-mon[33238]: 3.1 scrub starts 2024-09-17T07:09:23.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:23 smithi135 ceph-mon[33238]: 3.1 scrub ok 2024-09-17T07:09:23.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:23 smithi135 ceph-mon[33238]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T07:09:23.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:23 smithi135 ceph-mon[33238]: Cluster is now healthy 2024-09-17T07:09:23.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:23 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2603482201' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T07:09:23.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:23 smithi135 ceph-mon[33238]: osdmap e193: 8 total, 8 up, 8 in 2024-09-17T07:09:23.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:23 smithi135 ceph-mon[33238]: 6.5 scrub starts 2024-09-17T07:09:23.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:23 smithi135 ceph-mon[33238]: 6.5 scrub ok 2024-09-17T07:09:23.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:23 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/227675106' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-09-17T07:09:23.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:23 smithi135 ceph-mon[33238]: Client client.admin marked osd.3 out, while it was still marked up 2024-09-17T07:09:23.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:23 smithi135 ceph-mon[33238]: 8.9 scrub ok 2024-09-17T07:09:23.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:23 smithi135 ceph-mon[33238]: pgmap v316: 148 pgs: 148 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 76 MiB/s wr, 1.21k op/s 2024-09-17T07:09:23.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:23 smithi135 ceph-mon[33238]: 2.7 deep-scrub starts 2024-09-17T07:09:23.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:23 smithi135 ceph-mon[33238]: 2.7 deep-scrub ok 2024-09-17T07:09:23.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:23.797 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:23.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:09:23.898 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:09:23.920 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:23.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:23.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:09:24.022 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:09:24.044 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:24.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:24.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:09:24.063 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 68 16 108617 108601 99.7863 26.1875 0.00333316 0.00991985 2024-09-17T07:09:24.146 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:09:24.168 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:24.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:24.169 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:09:24.269 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:09:24.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:24.293 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:24.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:09:24.394 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:09:24.430 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:24.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:24.431 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:09:24.532 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:09:24.554 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:24.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:24.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:09:24.655 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:09:24.678 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:24.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:24.679 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:09:24.780 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:09:24.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:24 smithi135 ceph-mon[33238]: 6.14 deep-scrub starts 2024-09-17T07:09:24.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:24 smithi135 ceph-mon[33238]: 6.14 deep-scrub ok 2024-09-17T07:09:24.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:24 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/227675106' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-09-17T07:09:24.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:24 smithi135 ceph-mon[33238]: osdmap e194: 8 total, 8 up, 7 in 2024-09-17T07:09:24.802 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:24.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:24.803 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:09:24.847 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:24 smithi100 ceph-mon[29092]: 6.14 deep-scrub starts 2024-09-17T07:09:24.847 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:24 smithi100 ceph-mon[29092]: 6.14 deep-scrub ok 2024-09-17T07:09:24.847 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:24 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/227675106' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-09-17T07:09:24.847 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:24 smithi100 ceph-mon[29092]: osdmap e194: 8 total, 8 up, 7 in 2024-09-17T07:09:24.848 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:24 smithi100 ceph-mon[39462]: 6.14 deep-scrub starts 2024-09-17T07:09:24.848 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:24 smithi100 ceph-mon[39462]: 6.14 deep-scrub ok 2024-09-17T07:09:24.848 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:24 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/227675106' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-09-17T07:09:24.848 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:24 smithi100 ceph-mon[39462]: osdmap e194: 8 total, 8 up, 7 in 2024-09-17T07:09:24.904 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:09:24.926 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:24.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:24.928 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:09:25.028 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:09:25.051 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:25.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:25.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:09:25.062 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 69 16 108617 108601 98.3405 0 - 0.00991985 2024-09-17T07:09:25.153 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:09:25.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:25.176 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:25.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:09:25.277 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:09:25.302 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:25.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:25.303 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:09:25.304 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-17T07:09:25.405 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:09:25.431 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:25.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:25.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:09:25.433 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:25 smithi100 ceph-mon[29092]: 5.f scrub starts 2024-09-17T07:09:25.433 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:25 smithi100 ceph-mon[29092]: 5.f scrub ok 2024-09-17T07:09:25.433 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:25 smithi100 ceph-mon[29092]: osdmap e195: 8 total, 8 up, 7 in 2024-09-17T07:09:25.433 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:25 smithi100 ceph-mon[29092]: 5.19 scrub starts 2024-09-17T07:09:25.434 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:25 smithi100 ceph-mon[29092]: 5.19 scrub ok 2024-09-17T07:09:25.434 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:25 smithi100 ceph-mon[29092]: pgmap v319: 148 pgs: 13 peering, 135 active+clean; 6.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 705 B/s rd, 49 MiB/s wr, 789 op/s 2024-09-17T07:09:25.434 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:25 smithi100 ceph-mon[29092]: osdmap e196: 8 total, 8 up, 7 in 2024-09-17T07:09:25.434 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:25 smithi100 ceph-mon[39462]: 5.f scrub starts 2024-09-17T07:09:25.434 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:25 smithi100 ceph-mon[39462]: 5.f scrub ok 2024-09-17T07:09:25.434 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:25 smithi100 ceph-mon[39462]: osdmap e195: 8 total, 8 up, 7 in 2024-09-17T07:09:25.434 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:25 smithi100 ceph-mon[39462]: 5.19 scrub starts 2024-09-17T07:09:25.434 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:25 smithi100 ceph-mon[39462]: 5.19 scrub ok 2024-09-17T07:09:25.434 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:25 smithi100 ceph-mon[39462]: pgmap v319: 148 pgs: 13 peering, 135 active+clean; 6.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 705 B/s rd, 49 MiB/s wr, 789 op/s 2024-09-17T07:09:25.435 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:25 smithi100 ceph-mon[39462]: osdmap e196: 8 total, 8 up, 7 in 2024-09-17T07:09:25.533 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:09:25.565 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:25.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:25.566 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:09:25.667 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:09:25.702 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:25.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:25.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:09:25.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:25 smithi135 ceph-mon[33238]: 5.f scrub starts 2024-09-17T07:09:25.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:25 smithi135 ceph-mon[33238]: 5.f scrub ok 2024-09-17T07:09:25.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:25 smithi135 ceph-mon[33238]: osdmap e195: 8 total, 8 up, 7 in 2024-09-17T07:09:25.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:25 smithi135 ceph-mon[33238]: 5.19 scrub starts 2024-09-17T07:09:25.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:25 smithi135 ceph-mon[33238]: 5.19 scrub ok 2024-09-17T07:09:25.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:25 smithi135 ceph-mon[33238]: pgmap v319: 148 pgs: 13 peering, 135 active+clean; 6.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 705 B/s rd, 49 MiB/s wr, 789 op/s 2024-09-17T07:09:25.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:25 smithi135 ceph-mon[33238]: osdmap e196: 8 total, 8 up, 7 in 2024-09-17T07:09:25.804 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:09:25.842 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:25.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:25.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:09:25.944 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:09:25.981 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:25.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:25.982 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:09:26.063 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 70 16 109274 109258 97.5222 20.5312 0.0081147 0.0102492 2024-09-17T07:09:26.082 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:09:26.118 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:26.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:26.119 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:09:26.126 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:09:25 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:09:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:09:26.220 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:09:26.261 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:26.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:26.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:09:26.363 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:09:26.403 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:26.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:26.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:09:26.505 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:09:26.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:26.550 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:26.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:09:26.584 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:26 smithi100 ceph-mon[29092]: 8.b starting backfill to osd.2 from (0'0,0'0] MAX to 193'6931 2024-09-17T07:09:26.584 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:26 smithi100 ceph-mon[29092]: 8.f starting backfill to osd.4 from (0'0,0'0] MAX to 193'6680 2024-09-17T07:09:26.584 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:26 smithi100 ceph-mon[29092]: 8.0 starting backfill to osd.4 from (0'0,0'0] MAX to 193'6790 2024-09-17T07:09:26.585 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:26 smithi100 ceph-mon[29092]: 8.c starting backfill to osd.5 from (0'0,0'0] MAX to 193'7020 2024-09-17T07:09:26.585 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:26 smithi100 ceph-mon[29092]: 8.7 starting backfill to osd.2 from (0'0,0'0] MAX to 193'6796 2024-09-17T07:09:26.585 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:26 smithi100 ceph-mon[29092]: 8.8 starting backfill to osd.0 from (0'0,0'0] MAX to 193'6741 2024-09-17T07:09:26.585 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:26 smithi100 ceph-mon[29092]: Health check failed: Reduced data availability: 5 pgs inactive, 6 pgs peering (PG_AVAILABILITY) 2024-09-17T07:09:26.585 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:26 smithi100 ceph-mon[39462]: 8.b starting backfill to osd.2 from (0'0,0'0] MAX to 193'6931 2024-09-17T07:09:26.585 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:26 smithi100 ceph-mon[39462]: 8.f starting backfill to osd.4 from (0'0,0'0] MAX to 193'6680 2024-09-17T07:09:26.585 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:26 smithi100 ceph-mon[39462]: 8.0 starting backfill to osd.4 from (0'0,0'0] MAX to 193'6790 2024-09-17T07:09:26.585 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:26 smithi100 ceph-mon[39462]: 8.c starting backfill to osd.5 from (0'0,0'0] MAX to 193'7020 2024-09-17T07:09:26.585 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:26 smithi100 ceph-mon[39462]: 8.7 starting backfill to osd.2 from (0'0,0'0] MAX to 193'6796 2024-09-17T07:09:26.585 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:26 smithi100 ceph-mon[39462]: 8.8 starting backfill to osd.0 from (0'0,0'0] MAX to 193'6741 2024-09-17T07:09:26.585 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:26 smithi100 ceph-mon[39462]: Health check failed: Reduced data availability: 5 pgs inactive, 6 pgs peering (PG_AVAILABILITY) 2024-09-17T07:09:26.651 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:09:26.682 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:26.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:26.683 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:09:26.784 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:09:26.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:26 smithi135 ceph-mon[33238]: 8.b starting backfill to osd.2 from (0'0,0'0] MAX to 193'6931 2024-09-17T07:09:26.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:26 smithi135 ceph-mon[33238]: 8.f starting backfill to osd.4 from (0'0,0'0] MAX to 193'6680 2024-09-17T07:09:26.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:26 smithi135 ceph-mon[33238]: 8.0 starting backfill to osd.4 from (0'0,0'0] MAX to 193'6790 2024-09-17T07:09:26.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:26 smithi135 ceph-mon[33238]: 8.c starting backfill to osd.5 from (0'0,0'0] MAX to 193'7020 2024-09-17T07:09:26.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:26 smithi135 ceph-mon[33238]: 8.7 starting backfill to osd.2 from (0'0,0'0] MAX to 193'6796 2024-09-17T07:09:26.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:26 smithi135 ceph-mon[33238]: 8.8 starting backfill to osd.0 from (0'0,0'0] MAX to 193'6741 2024-09-17T07:09:26.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:26 smithi135 ceph-mon[33238]: Health check failed: Reduced data availability: 5 pgs inactive, 6 pgs peering (PG_AVAILABILITY) 2024-09-17T07:09:26.814 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:26.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:26.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:09:26.916 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:09:26.949 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:26.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:26.950 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:09:27.051 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:09:27.063 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 71 16 110402 110386 97.1416 70.5 0.00624679 0.0102897 2024-09-17T07:09:27.082 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:27.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:27.084 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:09:27.184 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:09:27.222 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:27.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:27.223 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:09:27.323 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:09:27.348 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:09:27.349 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:09:27.354 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:27.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:27.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:09:27.392 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1: {} 2024-09-17T07:09:27.393 INFO:teuthology.orchestra.run.smithi100.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:09:27.446 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:09:27.446 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:09:27.446 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:27 smithi100 ceph-mon[29092]: 2.1b deep-scrub starts 2024-09-17T07:09:27.446 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:27 smithi100 ceph-mon[29092]: pgmap v321: 148 pgs: 13 peering, 135 active+clean; 6.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 855 B/s rd, 60 MiB/s wr, 957 op/s 2024-09-17T07:09:27.447 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:27 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1789001181' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:09:27.447 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:27 smithi100 ceph-mon[39462]: 2.1b deep-scrub starts 2024-09-17T07:09:27.447 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:27 smithi100 ceph-mon[39462]: pgmap v321: 148 pgs: 13 peering, 135 active+clean; 6.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 855 B/s rd, 60 MiB/s wr, 957 op/s 2024-09-17T07:09:27.447 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:27 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1789001181' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:09:27.456 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:09:27.491 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:09:27.491 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:09:27.503 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:27.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:27.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:09:27.531 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:09:27.532 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:09:27.555 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:09:27.568 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-17T07:09:27.568 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:09:27.573 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:09:27.573 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:09:27.606 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:09:27.611 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:09:27.611 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:09:27.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:27.655 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:27.655 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:09:27.660 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:09:27.660 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:09:27.757 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:09:27.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:27 smithi135 ceph-mon[33238]: 2.1b deep-scrub starts 2024-09-17T07:09:27.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:27 smithi135 ceph-mon[33238]: pgmap v321: 148 pgs: 13 peering, 135 active+clean; 6.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 855 B/s rd, 60 MiB/s wr, 957 op/s 2024-09-17T07:09:27.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:27 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1789001181' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:09:27.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:27.797 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:27.797 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:09:27.898 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:09:27.937 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:27.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:27.938 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:09:28.038 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:09:28.101 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 72 16 111441 111425 96.6944 64.9375 0.0131368 0.0103362 2024-09-17T07:09:28.133 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:28.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:28.134 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:09:28.235 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:09:28.272 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:28.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:28.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:09:28.374 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:09:28.415 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:28.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:28.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:09:28.516 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:09:28.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:28.554 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:28.554 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:09:28.655 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:09:28.701 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:28.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:28.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:09:28.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:28 smithi135 ceph-mon[33238]: 5.14 scrub starts 2024-09-17T07:09:28.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:28 smithi135 ceph-mon[33238]: 6.1f scrub starts 2024-09-17T07:09:28.793 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:09:28 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:09:28.803 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:09:28.840 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:28.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:28.841 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:09:28.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:28 smithi100 ceph-mon[29092]: 5.14 scrub starts 2024-09-17T07:09:28.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:28 smithi100 ceph-mon[29092]: 6.1f scrub starts 2024-09-17T07:09:28.880 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:28 smithi100 ceph-mon[39462]: 5.14 scrub starts 2024-09-17T07:09:28.880 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:28 smithi100 ceph-mon[39462]: 6.1f scrub starts 2024-09-17T07:09:28.942 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:09:28.983 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:28.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:28.984 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:09:29.057 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-17T07:09:29.063 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 73 16 112615 112599 96.3749 73.375 0.0099625 0.0103714 2024-09-17T07:09:29.085 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:09:29.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:29.121 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:29.121 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:09:29.222 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:09:29.255 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:29.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:29.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:09:29.356 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:09:29.401 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:29.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:29.403 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:09:29.503 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:09:29.548 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:29.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:29.550 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:09:29.651 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:09:29.688 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:29.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:29.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:09:29.790 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:09:29.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:29 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:09:29.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:29 smithi135 ceph-mon[33238]: 6.1b scrub starts 2024-09-17T07:09:29.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:29 smithi135 ceph-mon[33238]: pgmap v322: 148 pgs: 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 142 active+clean; 6.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 852 B/s rd, 79 MiB/s wr, 1.26k op/s; 40784/336000 objects misplaced (12.138%); 14 MiB/s, 243 objects/s recovering 2024-09-17T07:09:29.824 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:29.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:29.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:09:29.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:29 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:09:29.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:29 smithi100 ceph-mon[29092]: 6.1b scrub starts 2024-09-17T07:09:29.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:29 smithi100 ceph-mon[29092]: pgmap v322: 148 pgs: 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 142 active+clean; 6.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 852 B/s rd, 79 MiB/s wr, 1.26k op/s; 40784/336000 objects misplaced (12.138%); 14 MiB/s, 243 objects/s recovering 2024-09-17T07:09:29.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:29 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:09:29.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:29 smithi100 ceph-mon[39462]: 6.1b scrub starts 2024-09-17T07:09:29.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:29 smithi100 ceph-mon[39462]: pgmap v322: 148 pgs: 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 142 active+clean; 6.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 852 B/s rd, 79 MiB/s wr, 1.26k op/s; 40784/336000 objects misplaced (12.138%); 14 MiB/s, 243 objects/s recovering 2024-09-17T07:09:29.926 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:09:29.962 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:29.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:29.964 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:09:30.063 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 74 16 113765 113749 96.0438 71.875 0.0132365 0.0104076 2024-09-17T07:09:30.065 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:09:30.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:30.110 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:30.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:09:30.200 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7] out_osds: [3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-17T07:09:30.200 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T07:09:30.201 INFO:tasks.thrashosds.thrasher:Shrinking pool unique_pool_1 2024-09-17T07:09:30.201 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool unique_pool_1 by 10 min 8 2024-09-17T07:09:30.201 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T07:09:30.211 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:09:30.243 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:30.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:30.245 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:09:30.332 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:09:30.345 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:09:30.376 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:30.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:30.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:09:30.478 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:09:30.515 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:30 smithi100 ceph-mon[29092]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-17T07:09:30.515 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:30 smithi100 ceph-mon[29092]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs inactive, 6 pgs peering) 2024-09-17T07:09:30.515 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:30 smithi100 ceph-mon[29092]: 5.11 scrub starts 2024-09-17T07:09:30.515 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:30 smithi100 ceph-mon[39462]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-17T07:09:30.515 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:30 smithi100 ceph-mon[39462]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs inactive, 6 pgs peering) 2024-09-17T07:09:30.515 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:30 smithi100 ceph-mon[39462]: 5.11 scrub starts 2024-09-17T07:09:30.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:30.517 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:30.517 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:09:30.618 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:09:30.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:30.654 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:30.655 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:09:30.755 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:09:30.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:30 smithi135 ceph-mon[33238]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-17T07:09:30.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:30 smithi135 ceph-mon[33238]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs inactive, 6 pgs peering) 2024-09-17T07:09:30.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:30 smithi135 ceph-mon[33238]: 5.11 scrub starts 2024-09-17T07:09:30.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:30.811 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:30.811 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:09:30.912 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:09:30.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:30.959 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:30.959 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:09:31.060 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:09:31.063 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 75 16 114859 114843 95.6749 68.375 0.0163692 0.0104471 2024-09-17T07:09:31.092 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:31.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:31.094 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:09:31.194 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:09:31.233 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:31.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:31.234 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:09:31.335 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:09:31.376 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:31.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:31.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:09:31.478 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:09:31.509 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:09:31.520 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:31.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:31.521 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:09:31.622 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:09:31.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:31.688 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:31.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:09:31.790 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:09:31.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:31 smithi135 ceph-mon[33238]: 6.b deep-scrub starts 2024-09-17T07:09:31.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:31 smithi135 ceph-mon[33238]: pgmap v323: 148 pgs: 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 142 active+clean; 6.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 43 MiB/s wr, 690 op/s; 40784/336000 objects misplaced (12.138%); 11 MiB/s, 188 objects/s recovering 2024-09-17T07:09:31.839 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:31.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:31.840 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:09:31.843 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:31 smithi100 ceph-mon[29092]: 6.b deep-scrub starts 2024-09-17T07:09:31.843 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:31 smithi100 ceph-mon[29092]: pgmap v323: 148 pgs: 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 142 active+clean; 6.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 43 MiB/s wr, 690 op/s; 40784/336000 objects misplaced (12.138%); 11 MiB/s, 188 objects/s recovering 2024-09-17T07:09:31.843 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:31 smithi100 ceph-mon[39462]: 6.b deep-scrub starts 2024-09-17T07:09:31.843 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:31 smithi100 ceph-mon[39462]: pgmap v323: 148 pgs: 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 142 active+clean; 6.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 43 MiB/s wr, 690 op/s; 40784/336000 objects misplaced (12.138%); 11 MiB/s, 188 objects/s recovering 2024-09-17T07:09:31.941 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:09:32.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:32.003 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:32.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:09:32.063 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 76 16 115929 115913 95.2959 66.875 0.0081777 0.0104877 2024-09-17T07:09:32.104 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:09:32.144 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:32.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:32.145 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:09:32.246 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:09:32.296 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:32.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:32.297 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:09:32.397 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:09:32.448 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:32.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:32.449 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:09:32.452 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:09:32.550 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:09:32.589 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:32 smithi100 ceph-mon[29092]: 2.b scrub starts 2024-09-17T07:09:32.589 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:32 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:09:32.589 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:32 smithi100 ceph-mon[29092]: 2.11 scrub starts 2024-09-17T07:09:32.590 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:32 smithi100 ceph-mon[39462]: 2.b scrub starts 2024-09-17T07:09:32.590 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:32 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:09:32.590 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:32 smithi100 ceph-mon[39462]: 2.11 scrub starts 2024-09-17T07:09:32.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:32.616 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:32.616 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:09:32.717 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:09:32.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:32.759 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:32.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:09:32.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:32 smithi135 ceph-mon[33238]: 2.b scrub starts 2024-09-17T07:09:32.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:32 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:09:32.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:32 smithi135 ceph-mon[33238]: 2.11 scrub starts 2024-09-17T07:09:32.860 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:09:32.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:32.904 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:32.904 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:09:33.005 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:09:33.027 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:33.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:33.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:09:33.063 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 77 16 116814 116798 94.7767 55.3125 0.00630864 0.0105389 2024-09-17T07:09:33.129 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:09:33.166 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:33.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:33.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:09:33.268 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:09:33.289 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:33.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:33.290 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:09:33.390 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:09:33.432 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:33.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:33.434 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:09:33.534 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:09:33.575 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:33.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:33.576 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:09:33.677 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:09:33.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:33.726 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:33.726 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:09:33.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:33 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:09:33.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:33 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:09:33.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:33 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:09:33.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:33 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:09:33.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:33 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:09:33.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:33 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1816237972' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:09:33.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:33 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:09:33.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:33 smithi135 ceph-mon[33238]: pgmap v324: 148 pgs: 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 142 active+clean; 6.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 116 B/s rd, 46 MiB/s wr, 732 op/s; 40352/341976 objects misplaced (11.800%); 20 MiB/s, 322 objects/s recovering 2024-09-17T07:09:33.800 INFO:teuthology.orchestra.run.smithi100.stderr:noscrub is set 2024-09-17T07:09:33.804 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:33 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:09:33.804 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:33 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:09:33.805 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:33 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:09:33.805 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:33 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:09:33.805 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:33 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:09:33.805 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:33 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1816237972' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:09:33.805 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:33 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:09:33.805 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:33 smithi100 ceph-mon[29092]: pgmap v324: 148 pgs: 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 142 active+clean; 6.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 116 B/s rd, 46 MiB/s wr, 732 op/s; 40352/341976 objects misplaced (11.800%); 20 MiB/s, 322 objects/s recovering 2024-09-17T07:09:33.806 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:33 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:09:33.806 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:33 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:09:33.806 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:33 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:09:33.806 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:33 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:09:33.806 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:33 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:09:33.806 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:33 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1816237972' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:09:33.806 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:33 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:09:33.806 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:33 smithi100 ceph-mon[39462]: pgmap v324: 148 pgs: 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 142 active+clean; 6.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 116 B/s rd, 46 MiB/s wr, 732 op/s; 40352/341976 objects misplaced (11.800%); 20 MiB/s, 322 objects/s recovering 2024-09-17T07:09:33.828 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:09:33.859 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:33.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:33.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:09:33.961 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:09:33.996 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:33.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:33.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:09:34.066 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 78 15 117891 117876 94.4231 67.375 0.00528704 0.0105852 2024-09-17T07:09:34.098 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:09:34.150 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:34.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:34.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:09:34.252 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:09:34.288 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:34.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:34.289 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:09:34.390 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:09:34.424 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:34.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:34.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:09:34.528 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:09:34.564 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:34.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:34.565 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:09:34.666 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:09:34.701 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:34.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:34.702 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:09:34.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:34 smithi135 ceph-mon[33238]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:09:34.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:34 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T07:09:34.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:34 smithi135 ceph-mon[33238]: osdmap e197: 8 total, 8 up, 7 in 2024-09-17T07:09:34.803 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:09:34.842 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:34.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:34.843 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:09:34.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:34 smithi100 ceph-mon[29092]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:09:34.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:34 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T07:09:34.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:34 smithi100 ceph-mon[29092]: osdmap e197: 8 total, 8 up, 7 in 2024-09-17T07:09:34.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:34 smithi100 ceph-mon[39462]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:09:34.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:34 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T07:09:34.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:34 smithi100 ceph-mon[39462]: osdmap e197: 8 total, 8 up, 7 in 2024-09-17T07:09:34.945 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:09:34.985 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:34.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:34.986 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:09:35.066 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 79 16 119016 119000 94.1171 70.25 0.0108076 0.0106206 2024-09-17T07:09:35.087 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:09:35.129 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:35.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:35.130 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:09:35.231 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:09:35.266 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:35.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:35.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:09:35.369 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:09:35.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:35.405 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:35.405 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:09:35.506 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:09:35.547 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:35.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:35.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:09:35.648 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:09:35.686 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:35.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:35.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:09:35.788 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:09:35.822 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:35.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:35.824 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:09:35.924 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:09:35.960 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:35.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:35.961 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:09:36.062 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:09:36.066 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:2024-09-17T07:09:36.066312+0000 min lat: 0.0012044 max lat: 2.14048 avg lat: 0.0106509 2024-09-17T07:09:36.066 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-17T07:09:36.066 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 80 16 120179 120163 93.8492 72.6875 0.0191323 0.0106509 2024-09-17T07:09:36.091 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:36.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:36.092 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:09:36.140 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:09:35 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:09:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:09:36.141 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:36 smithi100 ceph-mon[29092]: pgmap v326: 148 pgs: 1 active+clean+scrubbing+deep, 5 active+clean+scrubbing, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 136 active+clean; 7.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 628 B/s rd, 62 MiB/s wr, 996 op/s; 40377/351915 objects misplaced (11.474%); 24 MiB/s, 387 objects/s recovering 2024-09-17T07:09:36.193 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:09:36.220 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:36.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:36.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:09:36.322 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:09:36.362 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:36.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:36.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:09:36.464 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:09:36.469 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-17T07:09:36.470 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:09:36.470 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:36 smithi100 ceph-mon[39462]: pgmap v326: 148 pgs: 1 active+clean+scrubbing+deep, 5 active+clean+scrubbing, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 136 active+clean; 7.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 628 B/s rd, 62 MiB/s wr, 996 op/s; 40377/351915 objects misplaced (11.474%); 24 MiB/s, 387 objects/s recovering 2024-09-17T07:09:36.502 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:36.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:36.503 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:09:36.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:36 smithi135 ceph-mon[33238]: pgmap v326: 148 pgs: 1 active+clean+scrubbing+deep, 5 active+clean+scrubbing, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 136 active+clean; 7.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 628 B/s rd, 62 MiB/s wr, 996 op/s; 40377/351915 objects misplaced (11.474%); 24 MiB/s, 387 objects/s recovering 2024-09-17T07:09:36.605 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:09:36.643 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:36.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:36.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:09:36.683 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:09:36.746 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:09:36.757 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-17T07:09:36.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:36.781 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:36.781 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:09:36.882 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:09:36.916 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:36.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:36.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:09:37.018 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:09:37.053 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:37.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:37.054 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:09:37.066 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 81 16 121302 121286 93.5567 70.1875 0.00980846 0.0106845 2024-09-17T07:09:37.154 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:09:37.165 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-17T07:09:37.183 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:37.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:37.184 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:09:37.285 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:09:37.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:37.339 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:37.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:09:37.440 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:09:37.478 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:37.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:37.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:09:37.580 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:09:37.626 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:37.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:37.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:09:37.728 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:09:37.760 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:37.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:37.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:09:37.862 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:09:37.901 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:37.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:37.902 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:09:38.003 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:09:38.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:38.045 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:38.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:09:38.066 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 82 16 122360 122344 93.2221 66.125 0.00971039 0.0107224 2024-09-17T07:09:38.147 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:09:38.189 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:38.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:38.191 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:09:38.291 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:09:38.328 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:38.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:38.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:09:38.430 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:09:38.466 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:38.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:38.467 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:09:38.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:38 smithi135 ceph-mon[33238]: pgmap v327: 148 pgs: 1 active+clean+scrubbing+deep, 5 active+clean+scrubbing, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 136 active+clean; 7.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 614 B/s rd, 61 MiB/s wr, 973 op/s; 40377/351915 objects misplaced (11.474%); 23 MiB/s, 378 objects/s recovering 2024-09-17T07:09:38.543 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:09:38 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:09:38.568 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:09:38.605 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:38.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:38.606 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:09:38.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:38 smithi100 ceph-mon[29092]: pgmap v327: 148 pgs: 1 active+clean+scrubbing+deep, 5 active+clean+scrubbing, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 136 active+clean; 7.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 614 B/s rd, 61 MiB/s wr, 973 op/s; 40377/351915 objects misplaced (11.474%); 23 MiB/s, 378 objects/s recovering 2024-09-17T07:09:38.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:38 smithi100 ceph-mon[39462]: pgmap v327: 148 pgs: 1 active+clean+scrubbing+deep, 5 active+clean+scrubbing, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 136 active+clean; 7.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 614 B/s rd, 61 MiB/s wr, 973 op/s; 40377/351915 objects misplaced (11.474%); 23 MiB/s, 378 objects/s recovering 2024-09-17T07:09:38.707 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:09:38.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:38.740 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:38.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:09:38.841 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:09:38.882 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:38.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:38.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:09:38.984 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:09:39.015 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:39.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:39.016 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:09:39.066 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 83 16 123367 123351 92.8573 62.9375 0.012456 0.0107649 2024-09-17T07:09:39.117 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:09:39.150 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:39.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:39.151 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:09:39.252 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:09:39.279 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:39.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:39.280 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:09:39.381 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:09:39.420 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:39.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:39.421 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:09:39.522 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:09:39.546 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:39.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:39.547 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:09:39.648 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:09:39.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:39.688 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:39.689 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:09:39.790 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:09:39.828 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:39.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:39.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:09:39.930 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:09:39.970 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:39.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:39.971 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:09:40.066 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 84 16 124404 124388 92.5234 64.8125 0.00846094 0.0108036 2024-09-17T07:09:40.072 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:09:40.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:40.097 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:40.098 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:09:40.198 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:09:40.237 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:40.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:40.239 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:09:40.339 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:09:40.377 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:40.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:40.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:09:40.479 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:09:40.504 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:40.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:40.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:09:40.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:40 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:09:40.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:40 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:09:40.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:40 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:09:40.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:40 smithi135 ceph-mon[33238]: pgmap v328: 148 pgs: 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 142 active+clean; 7.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1023 B/s rd, 68 MiB/s wr, 1.09k op/s; 39794/368640 objects misplaced (10.795%); 30 MiB/s, 473 objects/s recovering 2024-09-17T07:09:40.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:40 smithi135 ceph-mon[33238]: osdmap e198: 8 total, 8 up, 7 in 2024-09-17T07:09:40.606 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:09:40.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:40 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:09:40.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:40 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:09:40.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:40 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:09:40.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:40 smithi100 ceph-mon[29092]: pgmap v328: 148 pgs: 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 142 active+clean; 7.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1023 B/s rd, 68 MiB/s wr, 1.09k op/s; 39794/368640 objects misplaced (10.795%); 30 MiB/s, 473 objects/s recovering 2024-09-17T07:09:40.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:40 smithi100 ceph-mon[29092]: osdmap e198: 8 total, 8 up, 7 in 2024-09-17T07:09:40.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:40 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:09:40.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:40 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:09:40.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:40 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:09:40.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:40 smithi100 ceph-mon[39462]: pgmap v328: 148 pgs: 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 142 active+clean; 7.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1023 B/s rd, 68 MiB/s wr, 1.09k op/s; 39794/368640 objects misplaced (10.795%); 30 MiB/s, 473 objects/s recovering 2024-09-17T07:09:40.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:40 smithi100 ceph-mon[39462]: osdmap e198: 8 total, 8 up, 7 in 2024-09-17T07:09:40.643 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:40.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:40.644 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:09:40.745 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:09:40.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:40.784 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:40.784 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:09:40.885 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:09:40.925 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:40.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:40.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:09:41.027 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:09:41.067 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:41.067 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 85 16 125340 125324 92.1232 58.5 0.0127086 0.0108503 2024-09-17T07:09:41.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:41.068 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:09:41.169 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:09:41.193 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:41.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:41.194 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:09:41.295 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:09:41.332 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:41.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:41.333 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:09:41.434 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:09:41.473 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:41.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:41.474 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:09:41.575 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:09:41.618 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:41.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:41.619 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:09:41.720 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:09:41.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:41.753 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:41.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:09:41.854 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:09:41.887 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:41.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:41.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:09:41.989 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:09:42.024 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:42.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:42.025 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:09:42.067 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 86 15 126407 126392 91.8281 66.75 0.0260244 0.0108856 2024-09-17T07:09:42.126 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:09:42.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:42.149 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:42.149 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:09:42.250 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:09:42.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:42.281 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:42.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:09:42.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:42 smithi100 ceph-mon[29092]: pgmap v330: 148 pgs: 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 142 active+clean; 7.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 69 MiB/s wr, 1.11k op/s; 39794/368640 objects misplaced (10.795%); 26 MiB/s, 420 objects/s recovering 2024-09-17T07:09:42.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:42 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:09:42.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:42 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:09:42.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:42 smithi100 ceph-mon[39462]: pgmap v330: 148 pgs: 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 142 active+clean; 7.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 69 MiB/s wr, 1.11k op/s; 39794/368640 objects misplaced (10.795%); 26 MiB/s, 420 objects/s recovering 2024-09-17T07:09:42.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:42 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:09:42.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:42 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:09:42.382 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:09:42.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:42.412 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:42.412 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:09:42.513 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:09:42.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:42 smithi135 ceph-mon[33238]: pgmap v330: 148 pgs: 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 142 active+clean; 7.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 69 MiB/s wr, 1.11k op/s; 39794/368640 objects misplaced (10.795%); 26 MiB/s, 420 objects/s recovering 2024-09-17T07:09:42.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:42 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:09:42.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:42 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:09:42.552 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:42.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:42.553 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:09:42.654 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:09:42.689 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:42.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:42.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:09:42.791 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:09:42.832 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:42.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:42.834 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:09:42.934 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:09:42.969 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:42.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:42.970 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:09:43.067 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 87 16 127352 127336 91.4508 59 0.0137389 0.0109305 2024-09-17T07:09:43.070 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:09:43.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:43.109 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:43.109 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:09:43.210 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:09:43.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:43.248 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:43.248 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:09:43.350 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:09:43.387 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:43.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:43.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:09:43.488 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:09:43.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:43.530 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:43.530 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:09:43.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:43 smithi135 ceph-mon[33238]: osdmap e199: 8 total, 8 up, 7 in 2024-09-17T07:09:43.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:43 smithi100 ceph-mon[29092]: osdmap e199: 8 total, 8 up, 7 in 2024-09-17T07:09:43.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:43 smithi100 ceph-mon[39462]: osdmap e199: 8 total, 8 up, 7 in 2024-09-17T07:09:43.631 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:09:43.654 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:43.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:43.655 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:09:43.756 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:09:43.789 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:43.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:43.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:09:43.891 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:09:43.924 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:43.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:43.926 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:09:44.027 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:09:44.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:44.066 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:44.067 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:09:44.067 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 88 16 128370 128354 91.1346 63.625 0.0150064 0.0109682 2024-09-17T07:09:44.168 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:09:44.205 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:44.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:44.206 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:09:44.307 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:09:44.336 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:44.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:44.337 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:09:44.438 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:09:44.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:44.491 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:44.491 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:09:44.523 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:44 smithi100 ceph-mon[39462]: pgmap v332: 148 pgs: 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 142 active+clean; 7.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 639 B/s rd, 58 MiB/s wr, 924 op/s; 39200/374097 objects misplaced (10.479%); 29 MiB/s, 471 objects/s recovering 2024-09-17T07:09:44.523 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:44 smithi100 ceph-mon[29092]: pgmap v332: 148 pgs: 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 142 active+clean; 7.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 639 B/s rd, 58 MiB/s wr, 924 op/s; 39200/374097 objects misplaced (10.479%); 29 MiB/s, 471 objects/s recovering 2024-09-17T07:09:44.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:44 smithi135 ceph-mon[33238]: pgmap v332: 148 pgs: 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 142 active+clean; 7.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 639 B/s rd, 58 MiB/s wr, 924 op/s; 39200/374097 objects misplaced (10.479%); 29 MiB/s, 471 objects/s recovering 2024-09-17T07:09:44.592 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:09:44.625 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:44.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:44.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:09:44.727 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:09:44.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:44.759 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:44.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:09:44.860 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:09:44.902 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:44.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:44.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:09:45.004 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:09:45.036 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:45.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:45.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:09:45.070 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 89 16 129468 129452 90.8787 68.625 0.0122762 0.010999 2024-09-17T07:09:45.138 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:09:45.167 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:45.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:45.168 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:09:45.268 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:09:45.302 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:45.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:45.304 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:09:45.405 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:09:45.411 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T07:09:45.417 INFO:teuthology.orchestra.run.smithi100.stderr:dumped all 2024-09-17T07:09:45.434 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:45.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:45.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:09:45.536 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:09:45.568 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:45.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:45.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:09:45.670 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:09:45.673 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:09:45.674 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:09:45.706 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1: {} 2024-09-17T07:09:45.706 INFO:teuthology.orchestra.run.smithi100.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:09:45.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:45.712 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:45.712 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:09:45.741 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:09:45.741 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:09:45.774 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:09:45.774 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:09:45.809 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:09:45.809 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:09:45.813 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:09:45.846 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:09:45.846 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:09:45.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:45.856 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:45.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:09:45.879 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:09:45.880 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:09:45.915 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:09:45.915 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:09:45.957 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:09:45.989 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:45.990 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:09:45 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:09:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:09:45.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:45.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:09:46.070 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Total time run: 90.0084 2024-09-17T07:09:46.070 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Total writes made: 130501 2024-09-17T07:09:46.070 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Write size: 65536 2024-09-17T07:09:46.070 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Object size: 65536 2024-09-17T07:09:46.070 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Bandwidth (MB/sec): 90.6173 2024-09-17T07:09:46.070 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Stddev Bandwidth: 37.3419 2024-09-17T07:09:46.070 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Max bandwidth (MB/sec): 225.75 2024-09-17T07:09:46.071 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Min bandwidth (MB/sec): 0 2024-09-17T07:09:46.071 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Average IOPS: 1449 2024-09-17T07:09:46.071 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Stddev IOPS: 597.481 2024-09-17T07:09:46.071 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Max IOPS: 3612 2024-09-17T07:09:46.071 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Min IOPS: 0 2024-09-17T07:09:46.071 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Average Latency(s): 0.0110316 2024-09-17T07:09:46.071 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Stddev Latency(s): 0.0245153 2024-09-17T07:09:46.071 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Max latency(s): 2.14048 2024-09-17T07:09:46.071 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Min latency(s): 0.0012044 2024-09-17T07:09:46.075 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Cleaning up (deleting benchmark objects) 2024-09-17T07:09:46.091 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:09:46.125 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:46.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:46.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:09:46.227 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:09:46.262 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:46.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:46.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:09:46.363 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:09:46.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:46 smithi100 ceph-mon[29092]: pgmap v333: 148 pgs: 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 142 active+clean; 7.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.32k op/s; 38885/383517 objects misplaced (10.139%); 39 MiB/s, 616 objects/s recovering 2024-09-17T07:09:46.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:46 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1497667529' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:09:46.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:46 smithi100 ceph-mon[39462]: pgmap v333: 148 pgs: 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 142 active+clean; 7.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.32k op/s; 38885/383517 objects misplaced (10.139%); 39 MiB/s, 616 objects/s recovering 2024-09-17T07:09:46.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:46 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1497667529' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:09:46.396 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:46.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:46.398 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:09:46.498 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:09:46.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:46.542 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:46.542 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:09:46.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:46 smithi135 ceph-mon[33238]: pgmap v333: 148 pgs: 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 142 active+clean; 7.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.32k op/s; 38885/383517 objects misplaced (10.139%); 39 MiB/s, 616 objects/s recovering 2024-09-17T07:09:46.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:46 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1497667529' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:09:46.571 INFO:teuthology.orchestra.run.smithi100.stdout:{"pg_ready":true,"pg_map":{"version":333,"stamp":"2024-09-17T07:09:45.153396+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8365807894,"num_objects":127839,"num_object_clones":0,"num_object_copies":383517,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":38885,"num_objects_unfound":0,"num_objects_dirty":127839,"num_whiteouts":0,"num_read":1360,"num_read_kb":1206,"num_write":128252,"num_write_kb":8171611,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8742,"num_bytes_recovered":567414780,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76736,"ondisk_log_size":76736,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":23851808,"kb_used_data":21864528,"kb_used_omap":10,"kb_used_meta":1987189,"kb_avail":632364256,"statfs":{"total":671965249536,"available":647540998144,"internally_reserved":0,"allocated":22389276672,"data_stored":22386422893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11152,"internal_metadata":2034881648},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":690356224,"num_objects":10534,"num_object_clones":0,"num_object_copies":31602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1492,"num_objects_unfound":0,"num_objects_dirty":10534,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":10534,"num_write_kb":674176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4936,"num_bytes_recovered":323485696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002401"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":17,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.253566+0000","last_change":"2024-09-17T07:09:33.811862+0000","last_active":"2024-09-17T07:09:39.253566+0000","last_peered":"2024-09-17T07:09:39.253566+0000","last_clean":"2024-09-17T07:09:39.253566+0000","last_became_active":"2024-09-17T07:09:24.401344+0000","last_became_peered":"2024-09-17T07:09:24.401344+0000","last_unstale":"2024-09-17T07:09:39.253566+0000","last_undegraded":"2024-09-17T07:09:39.253566+0000","last_fullsized":"2024-09-17T07:09:39.253566+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:33.644728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:33.644728+0000","last_clean_scrub_stamp":"2024-09-17T07:07:33.644728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":177,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.251275+0000","last_change":"2024-09-17T07:08:17.332759+0000","last_active":"2024-09-17T07:09:39.251275+0000","last_peered":"2024-09-17T07:09:39.251275+0000","last_clean":"2024-09-17T07:09:39.251275+0000","last_became_active":"2024-09-17T07:02:12.006754+0000","last_became_peered":"2024-09-17T07:02:12.006754+0000","last_unstale":"2024-09-17T07:09:39.251275+0000","last_undegraded":"2024-09-17T07:09:39.251275+0000","last_fullsized":"2024-09-17T07:09:39.251275+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:17.332631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:08:17.332631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023942220000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"75'15","reported_seq":70,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.243594+0000","last_change":"2024-09-17T07:09:24.419732+0000","last_active":"2024-09-17T07:09:39.243594+0000","last_peered":"2024-09-17T07:09:39.243594+0000","last_clean":"2024-09-17T07:09:39.243594+0000","last_became_active":"2024-09-17T07:09:24.402042+0000","last_became_peered":"2024-09-17T07:09:24.402042+0000","last_unstale":"2024-09-17T07:09:39.243594+0000","last_undegraded":"2024-09-17T07:09:39.243594+0000","last_fullsized":"2024-09-17T07:09:39.243594+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-09-17T07:08:28.845108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:08:28.845108+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:53.323983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":162,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.281958+0000","last_change":"2024-09-17T07:08:21.564706+0000","last_active":"2024-09-17T07:09:39.281958+0000","last_peered":"2024-09-17T07:09:39.281958+0000","last_clean":"2024-09-17T07:09:39.281958+0000","last_became_active":"2024-09-17T07:02:16.014928+0000","last_became_peered":"2024-09-17T07:02:16.014928+0000","last_unstale":"2024-09-17T07:09:39.281958+0000","last_undegraded":"2024-09-17T07:09:39.281958+0000","last_fullsized":"2024-09-17T07:09:39.281958+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:21.564567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:08:21.564567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:42.719076+0000","scrub_duration":0.0039106849999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":17,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.251633+0000","last_change":"2024-09-17T07:09:24.408521+0000","last_active":"2024-09-17T07:09:39.251633+0000","last_peered":"2024-09-17T07:09:39.251633+0000","last_clean":"2024-09-17T07:09:39.251633+0000","last_became_active":"2024-09-17T07:09:24.402568+0000","last_became_peered":"2024-09-17T07:09:24.402568+0000","last_unstale":"2024-09-17T07:09:39.251633+0000","last_undegraded":"2024-09-17T07:09:39.251633+0000","last_fullsized":"2024-09-17T07:09:39.251633+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:15.767495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:07:15.767495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"75'9","reported_seq":191,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.282024+0000","last_change":"2024-09-17T07:08:22.546769+0000","last_active":"2024-09-17T07:09:39.282024+0000","last_peered":"2024-09-17T07:09:39.282024+0000","last_clean":"2024-09-17T07:09:39.282024+0000","last_became_active":"2024-09-17T07:02:14.016244+0000","last_became_peered":"2024-09-17T07:02:14.016244+0000","last_unstale":"2024-09-17T07:09:39.282024+0000","last_undegraded":"2024-09-17T07:09:39.282024+0000","last_fullsized":"2024-09-17T07:09:39.282024+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:08:22.546632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:08:22.546632+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:45.063195+0000","scrub_duration":0.0041894990000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":173,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.240101+0000","last_change":"2024-09-17T07:09:25.123274+0000","last_active":"2024-09-17T07:09:39.240101+0000","last_peered":"2024-09-17T07:09:39.240101+0000","last_clean":"2024-09-17T07:09:39.240101+0000","last_became_active":"2024-09-17T07:02:16.014380+0000","last_became_peered":"2024-09-17T07:02:16.014380+0000","last_unstale":"2024-09-17T07:09:39.240101+0000","last_undegraded":"2024-09-17T07:09:39.240101+0000","last_fullsized":"2024-09-17T07:09:39.240101+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:25.123158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:09:25.123158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:49.651515+0000","scrub_duration":0.00166512,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":160,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.279951+0000","last_change":"2024-09-17T07:07:21.791530+0000","last_active":"2024-09-17T07:09:39.279951+0000","last_peered":"2024-09-17T07:09:39.279951+0000","last_clean":"2024-09-17T07:09:39.279951+0000","last_became_active":"2024-09-17T07:02:18.020790+0000","last_became_peered":"2024-09-17T07:02:18.020790+0000","last_unstale":"2024-09-17T07:09:39.279951+0000","last_undegraded":"2024-09-17T07:09:39.279951+0000","last_fullsized":"2024-09-17T07:09:39.279951+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:21.791399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:07:21.791399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0069986889999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":176,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.255836+0000","last_change":"2024-09-17T07:09:09.812811+0000","last_active":"2024-09-17T07:09:39.255836+0000","last_peered":"2024-09-17T07:09:39.255836+0000","last_clean":"2024-09-17T07:09:39.255836+0000","last_became_active":"2024-09-17T07:02:12.011129+0000","last_became_peered":"2024-09-17T07:02:12.011129+0000","last_unstale":"2024-09-17T07:09:39.255836+0000","last_undegraded":"2024-09-17T07:09:39.255836+0000","last_fullsized":"2024-09-17T07:09:39.255836+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:09.812692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:43.060102+0000","last_clean_scrub_stamp":"2024-09-17T07:09:09.812692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:22.110015+0000","scrub_duration":0.0062170050000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"75'5","reported_seq":189,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.289309+0000","last_change":"2024-09-17T07:09:24.415312+0000","last_active":"2024-09-17T07:09:39.289309+0000","last_peered":"2024-09-17T07:09:39.289309+0000","last_clean":"2024-09-17T07:09:39.289309+0000","last_became_active":"2024-09-17T07:09:24.396900+0000","last_became_peered":"2024-09-17T07:09:24.396900+0000","last_unstale":"2024-09-17T07:09:39.289309+0000","last_undegraded":"2024-09-17T07:09:39.289309+0000","last_fullsized":"2024-09-17T07:09:39.289309+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-09-17T07:07:15.848071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:07:15.848071+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010596959,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":173,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-17T07:09:42.242696+0000","last_change":"2024-09-17T07:08:19.081985+0000","last_active":"2024-09-17T07:09:42.242696+0000","last_peered":"2024-09-17T07:09:42.242696+0000","last_clean":"2024-09-17T07:09:42.242696+0000","last_became_active":"2024-09-17T07:02:16.015401+0000","last_became_peered":"2024-09-17T07:02:16.015401+0000","last_unstale":"2024-09-17T07:09:42.242696+0000","last_undegraded":"2024-09-17T07:09:42.242696+0000","last_fullsized":"2024-09-17T07:09:42.242696+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:19.081861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:08:19.081861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002870844,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":179,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.242284+0000","last_change":"2024-09-17T07:09:24.404913+0000","last_active":"2024-09-17T07:09:39.242284+0000","last_peered":"2024-09-17T07:09:39.242284+0000","last_clean":"2024-09-17T07:09:39.242284+0000","last_became_active":"2024-09-17T07:09:24.402616+0000","last_became_peered":"2024-09-17T07:09:24.402616+0000","last_unstale":"2024-09-17T07:09:39.242284+0000","last_undegraded":"2024-09-17T07:09:39.242284+0000","last_fullsized":"2024-09-17T07:09:39.242284+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:23.739157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:07:23.739157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004894603,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":176,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.254913+0000","last_change":"2024-09-17T07:09:08.807223+0000","last_active":"2024-09-17T07:09:39.254913+0000","last_peered":"2024-09-17T07:09:39.254913+0000","last_clean":"2024-09-17T07:09:39.254913+0000","last_became_active":"2024-09-17T07:02:12.012234+0000","last_became_peered":"2024-09-17T07:02:12.012234+0000","last_unstale":"2024-09-17T07:09:39.254913+0000","last_undegraded":"2024-09-17T07:09:39.254913+0000","last_fullsized":"2024-09-17T07:09:39.254913+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:08.807058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:45.029247+0000","last_clean_scrub_stamp":"2024-09-17T07:09:08.807058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:14.065777+0000","scrub_duration":0.015274785000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"75'9","reported_seq":199,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-17T07:09:42.236365+0000","last_change":"2024-09-17T07:09:24.432560+0000","last_active":"2024-09-17T07:09:42.236365+0000","last_peered":"2024-09-17T07:09:42.236365+0000","last_clean":"2024-09-17T07:09:42.236365+0000","last_became_active":"2024-09-17T07:09:24.397954+0000","last_became_peered":"2024-09-17T07:09:24.397954+0000","last_unstale":"2024-09-17T07:09:42.236365+0000","last_undegraded":"2024-09-17T07:09:42.236365+0000","last_fullsized":"2024-09-17T07:09:42.236365+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:08:17.673242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:08:17.673242+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0059611450000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":171,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.242145+0000","last_change":"2024-09-17T07:08:28.096341+0000","last_active":"2024-09-17T07:09:39.242145+0000","last_peered":"2024-09-17T07:09:39.242145+0000","last_clean":"2024-09-17T07:09:39.242145+0000","last_became_active":"2024-09-17T07:02:16.023774+0000","last_became_peered":"2024-09-17T07:02:16.023774+0000","last_unstale":"2024-09-17T07:09:39.242145+0000","last_undegraded":"2024-09-17T07:09:39.242145+0000","last_fullsized":"2024-09-17T07:09:39.242145+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:28.096266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:28.096266+0000","last_clean_scrub_stamp":"2024-09-17T07:08:28.096266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010716228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":160,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.243442+0000","last_change":"2024-09-17T07:07:34.205329+0000","last_active":"2024-09-17T07:09:39.243442+0000","last_peered":"2024-09-17T07:09:39.243442+0000","last_clean":"2024-09-17T07:09:39.243442+0000","last_became_active":"2024-09-17T07:02:18.029178+0000","last_became_peered":"2024-09-17T07:02:18.029178+0000","last_unstale":"2024-09-17T07:09:39.243442+0000","last_undegraded":"2024-09-17T07:09:39.243442+0000","last_fullsized":"2024-09-17T07:09:39.243442+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:34.205205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:07:34.205205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0057819899999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"199'7924","reported_seq":7964,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-17T07:09:43.300790+0000","last_change":"2024-09-17T07:08:00.759806+0000","last_active":"2024-09-17T07:09:43.300790+0000","last_peered":"2024-09-17T07:09:43.300790+0000","last_clean":"2024-09-17T07:09:43.300790+0000","last_became_active":"2024-09-17T07:08:00.759498+0000","last_became_peered":"2024-09-17T07:08:00.759498+0000","last_unstale":"2024-09-17T07:09:43.300790+0000","last_undegraded":"2024-09-17T07:09:43.300790+0000","last_fullsized":"2024-09-17T07:09:43.300790+0000","mapping_epoch":168,"log_start":"183'3445","ondisk_log_start":"183'3445","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_clean_scrub_stamp":"2024-09-17T07:07:59.220681+0000","objects_scrubbed":0,"log_size":4479,"log_dups_size":0,"ondisk_log_size":4479,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":519307264,"num_objects":7924,"num_object_clones":0,"num_object_copies":23772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7924,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7924,"num_write_kb":507136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"95'25","reported_seq":211,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.281703+0000","last_change":"2024-09-17T07:09:17.694556+0000","last_active":"2024-09-17T07:09:39.281703+0000","last_peered":"2024-09-17T07:09:39.281703+0000","last_clean":"2024-09-17T07:09:39.281703+0000","last_became_active":"2024-09-17T07:02:14.016270+0000","last_became_peered":"2024-09-17T07:02:14.016270+0000","last_unstale":"2024-09-17T07:09:39.281703+0000","last_undegraded":"2024-09-17T07:09:39.281703+0000","last_fullsized":"2024-09-17T07:09:39.281703+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'25","last_scrub_stamp":"2024-09-17T07:09:17.694236+0000","last_deep_scrub":"95'25","last_deep_scrub_stamp":"2024-09-17T07:04:09.450039+0000","last_clean_scrub_stamp":"2024-09-17T07:09:17.694236+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:24.489956+0000","scrub_duration":0.014129604,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":178,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.254050+0000","last_change":"2024-09-17T07:09:33.812141+0000","last_active":"2024-09-17T07:09:39.254050+0000","last_peered":"2024-09-17T07:09:39.254050+0000","last_clean":"2024-09-17T07:09:39.254050+0000","last_became_active":"2024-09-17T07:02:12.011258+0000","last_became_peered":"2024-09-17T07:02:12.011258+0000","last_unstale":"2024-09-17T07:09:39.254050+0000","last_undegraded":"2024-09-17T07:09:39.254050+0000","last_fullsized":"2024-09-17T07:09:39.254050+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:18.088246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:08:18.088246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0020459860000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":162,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.253034+0000","last_change":"2024-09-17T07:07:24.620899+0000","last_active":"2024-09-17T07:09:39.253034+0000","last_peered":"2024-09-17T07:09:39.253034+0000","last_clean":"2024-09-17T07:09:39.253034+0000","last_became_active":"2024-09-17T07:02:16.018147+0000","last_became_peered":"2024-09-17T07:02:16.018147+0000","last_unstale":"2024-09-17T07:09:39.253034+0000","last_undegraded":"2024-09-17T07:09:39.253034+0000","last_fullsized":"2024-09-17T07:09:39.253034+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:24.620758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:07:24.620758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0050498469999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":19,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.279710+0000","last_change":"2024-09-17T07:09:24.397784+0000","last_active":"2024-09-17T07:09:39.279710+0000","last_peered":"2024-09-17T07:09:39.279710+0000","last_clean":"2024-09-17T07:09:39.279710+0000","last_became_active":"2024-09-17T07:09:24.396334+0000","last_became_peered":"2024-09-17T07:09:24.396334+0000","last_unstale":"2024-09-17T07:09:39.279710+0000","last_undegraded":"2024-09-17T07:09:39.279710+0000","last_fullsized":"2024-09-17T07:09:39.279710+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:21.753939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:42.984088+0000","last_clean_scrub_stamp":"2024-09-17T07:09:21.753939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:28.208006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.0","version":"199'7953","reported_seq":8004,"reported_epoch":199,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-17T07:09:43.095274+0000","last_change":"2024-09-17T07:09:25.475601+0000","last_active":"2024-09-17T07:09:43.095274+0000","last_peered":"2024-09-17T07:09:43.095274+0000","last_clean":"2024-09-17T07:09:23.364870+0000","last_became_active":"2024-09-17T07:09:25.460711+0000","last_became_peered":"2024-09-17T07:09:25.460711+0000","last_unstale":"2024-09-17T07:09:43.095274+0000","last_undegraded":"2024-09-17T07:09:43.095274+0000","last_fullsized":"2024-09-17T07:09:43.095274+0000","mapping_epoch":195,"log_start":"182'3203","ondisk_log_start":"182'3203","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_clean_scrub_stamp":"2024-09-17T07:07:59.220681+0000","objects_scrubbed":0,"log_size":4750,"log_dups_size":0,"ondisk_log_size":4750,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":521207808,"num_objects":7953,"num_object_clones":0,"num_object_copies":23859,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7953,"num_objects_unfound":0,"num_objects_dirty":7953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7953,"num_write_kb":508992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,3],"avail_no_missing":["5","1","3"],"object_location_counts":[{"shards":"1,3,5","objects":7953}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.c","version":"176'15","reported_seq":216,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.282899+0000","last_change":"2024-09-17T07:09:24.427752+0000","last_active":"2024-09-17T07:09:39.282899+0000","last_peered":"2024-09-17T07:09:39.282899+0000","last_clean":"2024-09-17T07:09:39.282899+0000","last_became_active":"2024-09-17T07:09:24.399287+0000","last_became_peered":"2024-09-17T07:09:24.399287+0000","last_unstale":"2024-09-17T07:09:39.282899+0000","last_undegraded":"2024-09-17T07:09:39.282899+0000","last_fullsized":"2024-09-17T07:09:39.282899+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'15","last_scrub_stamp":"2024-09-17T07:08:19.633791+0000","last_deep_scrub":"176'15","last_deep_scrub_stamp":"2024-09-17T07:08:19.633791+0000","last_clean_scrub_stamp":"2024-09-17T07:08:19.633791+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.009818284,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":185,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.252718+0000","last_change":"2024-09-17T07:09:24.407640+0000","last_active":"2024-09-17T07:09:39.252718+0000","last_peered":"2024-09-17T07:09:39.252718+0000","last_clean":"2024-09-17T07:09:39.252718+0000","last_became_active":"2024-09-17T07:09:24.406973+0000","last_became_peered":"2024-09-17T07:09:24.406973+0000","last_unstale":"2024-09-17T07:09:39.252718+0000","last_undegraded":"2024-09-17T07:09:39.252718+0000","last_fullsized":"2024-09-17T07:09:39.252718+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:23.802226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:08:23.802226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0065215869999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"98'9","reported_seq":223,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:40.604090+0000","last_change":"2024-09-17T07:08:31.923373+0000","last_active":"2024-09-17T07:09:40.604090+0000","last_peered":"2024-09-17T07:09:40.604090+0000","last_clean":"2024-09-17T07:09:40.604090+0000","last_became_active":"2024-09-17T07:04:15.110797+0000","last_became_peered":"2024-09-17T07:04:15.110797+0000","last_unstale":"2024-09-17T07:09:40.604090+0000","last_undegraded":"2024-09-17T07:09:40.604090+0000","last_fullsized":"2024-09-17T07:09:40.604090+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-09-17T07:08:31.923228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:08:31.923228+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:49.885819+0000","scrub_duration":0.015229100000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":171,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.279505+0000","last_change":"2024-09-17T07:08:30.650769+0000","last_active":"2024-09-17T07:09:39.279505+0000","last_peered":"2024-09-17T07:09:39.279505+0000","last_clean":"2024-09-17T07:09:39.279505+0000","last_became_active":"2024-09-17T07:02:18.017460+0000","last_became_peered":"2024-09-17T07:02:18.017460+0000","last_unstale":"2024-09-17T07:09:39.279505+0000","last_undegraded":"2024-09-17T07:09:39.279505+0000","last_fullsized":"2024-09-17T07:09:39.279505+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:30.650419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:08:30.650419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003351316,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"199'7829","reported_seq":7869,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-17T07:09:43.290091+0000","last_change":"2024-09-17T07:08:00.242384+0000","last_active":"2024-09-17T07:09:43.290091+0000","last_peered":"2024-09-17T07:09:43.290091+0000","last_clean":"2024-09-17T07:09:43.290091+0000","last_became_active":"2024-09-17T07:08:00.241996+0000","last_became_peered":"2024-09-17T07:08:00.241996+0000","last_unstale":"2024-09-17T07:09:43.290091+0000","last_undegraded":"2024-09-17T07:09:43.290091+0000","last_fullsized":"2024-09-17T07:09:43.290091+0000","mapping_epoch":168,"log_start":"183'3382","ondisk_log_start":"183'3382","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_clean_scrub_stamp":"2024-09-17T07:07:59.220681+0000","objects_scrubbed":0,"log_size":4447,"log_dups_size":0,"ondisk_log_size":4447,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":513081344,"num_objects":7829,"num_object_clones":0,"num_object_copies":23487,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7829,"num_write_kb":501056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"75'15","reported_seq":224,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.255580+0000","last_change":"2024-09-17T07:09:24.436271+0000","last_active":"2024-09-17T07:09:39.255580+0000","last_peered":"2024-09-17T07:09:39.255580+0000","last_clean":"2024-09-17T07:09:39.255580+0000","last_became_active":"2024-09-17T07:09:24.407311+0000","last_became_peered":"2024-09-17T07:09:24.407311+0000","last_unstale":"2024-09-17T07:09:39.255580+0000","last_undegraded":"2024-09-17T07:09:39.255580+0000","last_fullsized":"2024-09-17T07:09:39.255580+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-09-17T07:08:13.784543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:08:13.784543+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002584257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":186,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.240297+0000","last_change":"2024-09-17T07:09:24.405992+0000","last_active":"2024-09-17T07:09:39.240297+0000","last_peered":"2024-09-17T07:09:39.240297+0000","last_clean":"2024-09-17T07:09:39.240297+0000","last_became_active":"2024-09-17T07:09:24.405655+0000","last_became_peered":"2024-09-17T07:09:24.405655+0000","last_unstale":"2024-09-17T07:09:39.240297+0000","last_undegraded":"2024-09-17T07:09:39.240297+0000","last_fullsized":"2024-09-17T07:09:39.240297+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:20.182256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:09:20.182256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:21.411135+0000","scrub_duration":0.0034116369999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"75'8","reported_seq":260,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:40.605037+0000","last_change":"2024-09-17T07:08:09.787205+0000","last_active":"2024-09-17T07:09:40.605037+0000","last_peered":"2024-09-17T07:09:40.605037+0000","last_clean":"2024-09-17T07:09:40.605037+0000","last_became_active":"2024-09-17T07:02:16.021762+0000","last_became_peered":"2024-09-17T07:02:16.021762+0000","last_unstale":"2024-09-17T07:09:40.605037+0000","last_undegraded":"2024-09-17T07:09:40.605037+0000","last_fullsized":"2024-09-17T07:09:40.605037+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:08:09.787123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:08:09.787123+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001487141,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":170,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.242666+0000","last_change":"2024-09-17T07:08:12.991029+0000","last_active":"2024-09-17T07:09:39.242666+0000","last_peered":"2024-09-17T07:09:39.242666+0000","last_clean":"2024-09-17T07:09:39.242666+0000","last_became_active":"2024-09-17T07:02:18.025332+0000","last_became_peered":"2024-09-17T07:02:18.025332+0000","last_unstale":"2024-09-17T07:09:39.242666+0000","last_undegraded":"2024-09-17T07:09:39.242666+0000","last_fullsized":"2024-09-17T07:09:39.242666+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:12.990915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:08:12.990915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021966199999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.2","version":"199'8102","reported_seq":8142,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-17T07:09:43.075043+0000","last_change":"2024-09-17T07:08:00.237404+0000","last_active":"2024-09-17T07:09:43.075043+0000","last_peered":"2024-09-17T07:09:43.075043+0000","last_clean":"2024-09-17T07:09:43.075043+0000","last_became_active":"2024-09-17T07:08:00.237068+0000","last_became_peered":"2024-09-17T07:08:00.237068+0000","last_unstale":"2024-09-17T07:09:43.075043+0000","last_undegraded":"2024-09-17T07:09:43.075043+0000","last_fullsized":"2024-09-17T07:09:43.075043+0000","mapping_epoch":168,"log_start":"182'3286","ondisk_log_start":"182'3286","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_clean_scrub_stamp":"2024-09-17T07:07:59.220681+0000","objects_scrubbed":0,"log_size":4816,"log_dups_size":0,"ondisk_log_size":4816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":530972672,"num_objects":8102,"num_object_clones":0,"num_object_copies":24306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8102,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8102,"num_write_kb":518528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"75'11","reported_seq":194,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.279650+0000","last_change":"2024-09-17T07:08:26.626034+0000","last_active":"2024-09-17T07:09:39.279650+0000","last_peered":"2024-09-17T07:09:39.279650+0000","last_clean":"2024-09-17T07:09:39.279650+0000","last_became_active":"2024-09-17T07:02:14.010227+0000","last_became_peered":"2024-09-17T07:02:14.010227+0000","last_unstale":"2024-09-17T07:09:39.279650+0000","last_undegraded":"2024-09-17T07:09:39.279650+0000","last_fullsized":"2024-09-17T07:09:39.279650+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-09-17T07:08:26.625892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:08:26.625892+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.038079981999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":177,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.253466+0000","last_change":"2024-09-17T07:08:06.100857+0000","last_active":"2024-09-17T07:09:39.253466+0000","last_peered":"2024-09-17T07:09:39.253466+0000","last_clean":"2024-09-17T07:09:39.253466+0000","last_became_active":"2024-09-17T07:02:12.013182+0000","last_became_peered":"2024-09-17T07:02:12.013182+0000","last_unstale":"2024-09-17T07:09:39.253466+0000","last_undegraded":"2024-09-17T07:09:39.253466+0000","last_fullsized":"2024-09-17T07:09:39.253466+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:06.100741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:08:06.100741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0012621170000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":183,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.243097+0000","last_change":"2024-09-17T07:09:24.229155+0000","last_active":"2024-09-17T07:09:39.243097+0000","last_peered":"2024-09-17T07:09:39.243097+0000","last_clean":"2024-09-17T07:09:39.243097+0000","last_became_active":"2024-09-17T07:02:16.020224+0000","last_became_peered":"2024-09-17T07:02:16.020224+0000","last_unstale":"2024-09-17T07:09:39.243097+0000","last_undegraded":"2024-09-17T07:09:39.243097+0000","last_fullsized":"2024-09-17T07:09:39.243097+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:24.229069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:09:24.229069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:38.285780+0000","scrub_duration":0.001199566,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":16,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.256149+0000","last_change":"2024-09-17T07:09:24.408435+0000","last_active":"2024-09-17T07:09:39.256149+0000","last_peered":"2024-09-17T07:09:39.256149+0000","last_clean":"2024-09-17T07:09:39.256149+0000","last_became_active":"2024-09-17T07:09:24.408109+0000","last_became_peered":"2024-09-17T07:09:24.408109+0000","last_unstale":"2024-09-17T07:09:39.256149+0000","last_undegraded":"2024-09-17T07:09:39.256149+0000","last_fullsized":"2024-09-17T07:09:39.256149+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:27.883528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:08:27.883528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:52.363126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.d","version":"199'7934","reported_seq":7974,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-17T07:09:43.062182+0000","last_change":"2024-09-17T07:08:00.760018+0000","last_active":"2024-09-17T07:09:43.062182+0000","last_peered":"2024-09-17T07:09:43.062182+0000","last_clean":"2024-09-17T07:09:43.062182+0000","last_became_active":"2024-09-17T07:08:00.759711+0000","last_became_peered":"2024-09-17T07:08:00.759711+0000","last_unstale":"2024-09-17T07:09:43.062182+0000","last_undegraded":"2024-09-17T07:09:43.062182+0000","last_fullsized":"2024-09-17T07:09:43.062182+0000","mapping_epoch":168,"log_start":"182'3218","ondisk_log_start":"182'3218","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_clean_scrub_stamp":"2024-09-17T07:07:59.220681+0000","objects_scrubbed":0,"log_size":4716,"log_dups_size":0,"ondisk_log_size":4716,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":519962624,"num_objects":7934,"num_object_clones":0,"num_object_copies":23802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7934,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7934,"num_write_kb":507776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1","version":"75'14","reported_seq":185,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.279312+0000","last_change":"2024-09-17T07:08:06.763827+0000","last_active":"2024-09-17T07:09:39.279312+0000","last_peered":"2024-09-17T07:09:39.279312+0000","last_clean":"2024-09-17T07:09:39.279312+0000","last_became_active":"2024-09-17T07:02:14.007810+0000","last_became_peered":"2024-09-17T07:02:14.007810+0000","last_unstale":"2024-09-17T07:09:39.279312+0000","last_undegraded":"2024-09-17T07:09:39.279312+0000","last_fullsized":"2024-09-17T07:09:39.279312+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'14","last_scrub_stamp":"2024-09-17T07:08:06.763689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:08:06.763689+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033699329999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":198,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.256706+0000","last_change":"2024-09-17T07:09:23.181150+0000","last_active":"2024-09-17T07:09:39.256706+0000","last_peered":"2024-09-17T07:09:39.256706+0000","last_clean":"2024-09-17T07:09:39.256706+0000","last_became_active":"2024-09-17T07:02:12.011766+0000","last_became_peered":"2024-09-17T07:02:12.011766+0000","last_unstale":"2024-09-17T07:09:39.256706+0000","last_undegraded":"2024-09-17T07:09:39.256706+0000","last_fullsized":"2024-09-17T07:09:39.256706+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:23.181028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:09:23.181028+0000","last_clean_scrub_stamp":"2024-09-17T07:09:23.181028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:51.612899+0000","scrub_duration":0.0051947429999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"196'9","reported_seq":44,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:40.605616+0000","last_change":"2024-09-17T07:09:24.427192+0000","last_active":"2024-09-17T07:09:40.605616+0000","last_peered":"2024-09-17T07:09:40.605616+0000","last_clean":"2024-09-17T07:09:40.605616+0000","last_became_active":"2024-09-17T07:09:24.406056+0000","last_became_peered":"2024-09-17T07:09:24.406056+0000","last_unstale":"2024-09-17T07:09:40.605616+0000","last_undegraded":"2024-09-17T07:09:40.605616+0000","last_fullsized":"2024-09-17T07:09:40.605616+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:09:14.776794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:09:14.776794+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:27.629020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":171,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.253652+0000","last_change":"2024-09-17T07:08:11.123128+0000","last_active":"2024-09-17T07:09:39.253652+0000","last_peered":"2024-09-17T07:09:39.253652+0000","last_clean":"2024-09-17T07:09:39.253652+0000","last_became_active":"2024-09-17T07:02:18.364981+0000","last_became_peered":"2024-09-17T07:02:18.364981+0000","last_unstale":"2024-09-17T07:09:39.253652+0000","last_undegraded":"2024-09-17T07:09:39.253652+0000","last_fullsized":"2024-09-17T07:09:39.253652+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:11.123047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:08:11.123047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014377210000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"199'8247","reported_seq":12300,"reported_epoch":199,"state":"active+remapped+backfilling","last_fresh":"2024-09-17T07:09:44.097161+0000","last_change":"2024-09-17T07:09:25.483625+0000","last_active":"2024-09-17T07:09:44.097161+0000","last_peered":"2024-09-17T07:09:44.097161+0000","last_clean":"2024-09-17T07:09:23.381756+0000","last_became_active":"2024-09-17T07:09:25.464119+0000","last_became_peered":"2024-09-17T07:09:25.464119+0000","last_unstale":"2024-09-17T07:09:44.097161+0000","last_undegraded":"2024-09-17T07:09:44.097161+0000","last_fullsized":"2024-09-17T07:09:44.097161+0000","mapping_epoch":195,"log_start":"181'3220","ondisk_log_start":"181'3220","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_clean_scrub_stamp":"2024-09-17T07:07:59.220681+0000","objects_scrubbed":0,"log_size":5027,"log_dups_size":0,"ondisk_log_size":5027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540475392,"num_objects":8247,"num_object_clones":0,"num_object_copies":24741,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6067,"num_objects_unfound":0,"num_objects_dirty":8247,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8247,"num_write_kb":527808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2003,"num_bytes_recovered":131268608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,3],"avail_no_missing":["0","3","7"],"object_location_counts":[{"shards":"0,3,7","objects":8247}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"75'18","reported_seq":79,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.253175+0000","last_change":"2024-09-17T07:09:24.424752+0000","last_active":"2024-09-17T07:09:39.253175+0000","last_peered":"2024-09-17T07:09:39.253175+0000","last_clean":"2024-09-17T07:09:39.253175+0000","last_became_active":"2024-09-17T07:09:24.403240+0000","last_became_peered":"2024-09-17T07:09:24.403240+0000","last_unstale":"2024-09-17T07:09:39.253175+0000","last_undegraded":"2024-09-17T07:09:39.253175+0000","last_fullsized":"2024-09-17T07:09:39.253175+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'18","last_scrub_stamp":"2024-09-17T07:08:30.831301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:08:30.831301+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":33,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":176,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.252870+0000","last_change":"2024-09-17T07:08:04.692191+0000","last_active":"2024-09-17T07:09:39.252870+0000","last_peered":"2024-09-17T07:09:39.252870+0000","last_clean":"2024-09-17T07:09:39.252870+0000","last_became_active":"2024-09-17T07:02:12.005687+0000","last_became_peered":"2024-09-17T07:02:12.005687+0000","last_unstale":"2024-09-17T07:09:39.252870+0000","last_undegraded":"2024-09-17T07:09:39.252870+0000","last_fullsized":"2024-09-17T07:09:39.252870+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:04.692099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:08:04.692099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001661023,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":171,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.279146+0000","last_change":"2024-09-17T07:09:24.393694+0000","last_active":"2024-09-17T07:09:39.279146+0000","last_peered":"2024-09-17T07:09:39.279146+0000","last_clean":"2024-09-17T07:09:39.279146+0000","last_became_active":"2024-09-17T07:09:24.393375+0000","last_became_peered":"2024-09-17T07:09:24.393375+0000","last_unstale":"2024-09-17T07:09:39.279146+0000","last_undegraded":"2024-09-17T07:09:39.279146+0000","last_fullsized":"2024-09-17T07:09:39.279146+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:03.726764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:08:03.726764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002251201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":180,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.280308+0000","last_change":"2024-09-17T07:09:24.399627+0000","last_active":"2024-09-17T07:09:39.280308+0000","last_peered":"2024-09-17T07:09:39.280308+0000","last_clean":"2024-09-17T07:09:39.280308+0000","last_became_active":"2024-09-17T07:09:24.398659+0000","last_became_peered":"2024-09-17T07:09:24.398659+0000","last_unstale":"2024-09-17T07:09:39.280308+0000","last_undegraded":"2024-09-17T07:09:39.280308+0000","last_fullsized":"2024-09-17T07:09:39.280308+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:25.588903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:08:25.588903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:49.276950+0000","scrub_duration":0.0021632869999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"199'7912","reported_seq":7960,"reported_epoch":199,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-17T07:09:43.367438+0000","last_change":"2024-09-17T07:09:25.473026+0000","last_active":"2024-09-17T07:09:43.367438+0000","last_peered":"2024-09-17T07:09:43.367438+0000","last_clean":"2024-09-17T07:09:23.369855+0000","last_became_active":"2024-09-17T07:09:25.456528+0000","last_became_peered":"2024-09-17T07:09:25.456528+0000","last_unstale":"2024-09-17T07:09:43.367438+0000","last_undegraded":"2024-09-17T07:09:43.367438+0000","last_fullsized":"2024-09-17T07:09:43.367438+0000","mapping_epoch":195,"log_start":"183'3469","ondisk_log_start":"183'3469","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_clean_scrub_stamp":"2024-09-17T07:07:59.220681+0000","objects_scrubbed":0,"log_size":4443,"log_dups_size":0,"ondisk_log_size":4443,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":518520832,"num_objects":7912,"num_object_clones":0,"num_object_copies":23736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7912,"num_objects_unfound":0,"num_objects_dirty":7912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7912,"num_write_kb":506368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,3],"avail_no_missing":["1","2","3"],"object_location_counts":[{"shards":"1,2,3","objects":7912}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"75'19","reported_seq":200,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.251018+0000","last_change":"2024-09-17T07:07:17.989796+0000","last_active":"2024-09-17T07:09:39.251018+0000","last_peered":"2024-09-17T07:09:39.251018+0000","last_clean":"2024-09-17T07:09:39.251018+0000","last_became_active":"2024-09-17T07:02:14.027389+0000","last_became_peered":"2024-09-17T07:02:14.027389+0000","last_unstale":"2024-09-17T07:09:39.251018+0000","last_undegraded":"2024-09-17T07:09:39.251018+0000","last_fullsized":"2024-09-17T07:09:39.251018+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-09-17T07:07:17.989435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:07:17.989435+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0043356940000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":177,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.255908+0000","last_change":"2024-09-17T07:08:24.072911+0000","last_active":"2024-09-17T07:09:39.255908+0000","last_peered":"2024-09-17T07:09:39.255908+0000","last_clean":"2024-09-17T07:09:39.255908+0000","last_became_active":"2024-09-17T07:02:12.011917+0000","last_became_peered":"2024-09-17T07:02:12.011917+0000","last_unstale":"2024-09-17T07:09:39.255908+0000","last_undegraded":"2024-09-17T07:09:39.255908+0000","last_fullsized":"2024-09-17T07:09:39.255908+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:24.072799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:24.072799+0000","last_clean_scrub_stamp":"2024-09-17T07:08:24.072799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0047928600000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":173,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.254862+0000","last_change":"2024-09-17T07:09:15.355446+0000","last_active":"2024-09-17T07:09:39.254862+0000","last_peered":"2024-09-17T07:09:39.254862+0000","last_clean":"2024-09-17T07:09:39.254862+0000","last_became_active":"2024-09-17T07:02:16.016199+0000","last_became_peered":"2024-09-17T07:02:16.016199+0000","last_unstale":"2024-09-17T07:09:39.254862+0000","last_undegraded":"2024-09-17T07:09:39.254862+0000","last_fullsized":"2024-09-17T07:09:39.254862+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:15.355318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:09:15.355318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:31.109239+0000","scrub_duration":0.0028918149999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":170,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.240107+0000","last_change":"2024-09-17T07:08:24.840933+0000","last_active":"2024-09-17T07:09:39.240107+0000","last_peered":"2024-09-17T07:09:39.240107+0000","last_clean":"2024-09-17T07:09:39.240107+0000","last_became_active":"2024-09-17T07:02:18.364835+0000","last_became_peered":"2024-09-17T07:02:18.364835+0000","last_unstale":"2024-09-17T07:09:39.240107+0000","last_undegraded":"2024-09-17T07:09:39.240107+0000","last_fullsized":"2024-09-17T07:09:39.240107+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:24.840821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:08:24.840821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002523197,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.e","version":"199'8025","reported_seq":8065,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-17T07:09:44.036767+0000","last_change":"2024-09-17T07:08:00.759796+0000","last_active":"2024-09-17T07:09:44.036767+0000","last_peered":"2024-09-17T07:09:44.036767+0000","last_clean":"2024-09-17T07:09:44.036767+0000","last_became_active":"2024-09-17T07:08:00.759537+0000","last_became_peered":"2024-09-17T07:08:00.759537+0000","last_unstale":"2024-09-17T07:09:44.036767+0000","last_undegraded":"2024-09-17T07:09:44.036767+0000","last_fullsized":"2024-09-17T07:09:44.036767+0000","mapping_epoch":168,"log_start":"181'2951","ondisk_log_start":"181'2951","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_clean_scrub_stamp":"2024-09-17T07:07:59.220681+0000","objects_scrubbed":0,"log_size":5074,"log_dups_size":0,"ondisk_log_size":5074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":525926400,"num_objects":8025,"num_object_clones":0,"num_object_copies":24075,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8025,"num_write_kb":513600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"110'15","reported_seq":194,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.252657+0000","last_change":"2024-09-17T07:08:19.835062+0000","last_active":"2024-09-17T07:09:39.252657+0000","last_peered":"2024-09-17T07:09:39.252657+0000","last_clean":"2024-09-17T07:09:39.252657+0000","last_became_active":"2024-09-17T07:02:14.010637+0000","last_became_peered":"2024-09-17T07:02:14.010637+0000","last_unstale":"2024-09-17T07:09:39.252657+0000","last_undegraded":"2024-09-17T07:09:39.252657+0000","last_fullsized":"2024-09-17T07:09:39.252657+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'15","last_scrub_stamp":"2024-09-17T07:08:19.834380+0000","last_deep_scrub":"110'15","last_deep_scrub_stamp":"2024-09-17T07:04:57.091400+0000","last_clean_scrub_stamp":"2024-09-17T07:08:19.834380+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:44.803473+0000","scrub_duration":0.0055436599999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":176,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.253247+0000","last_change":"2024-09-17T07:08:17.806138+0000","last_active":"2024-09-17T07:09:39.253247+0000","last_peered":"2024-09-17T07:09:39.253247+0000","last_clean":"2024-09-17T07:09:39.253247+0000","last_became_active":"2024-09-17T07:02:12.010803+0000","last_became_peered":"2024-09-17T07:02:12.010803+0000","last_unstale":"2024-09-17T07:09:39.253247+0000","last_undegraded":"2024-09-17T07:09:39.253247+0000","last_fullsized":"2024-09-17T07:09:39.253247+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:17.806001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:25.147878+0000","last_clean_scrub_stamp":"2024-09-17T07:08:17.806001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00606531,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"89'9","reported_seq":272,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:40.603439+0000","last_change":"2024-09-17T07:08:14.309093+0000","last_active":"2024-09-17T07:09:40.603439+0000","last_peered":"2024-09-17T07:09:40.603439+0000","last_clean":"2024-09-17T07:09:40.603439+0000","last_became_active":"2024-09-17T07:02:16.021360+0000","last_became_peered":"2024-09-17T07:02:16.021360+0000","last_unstale":"2024-09-17T07:09:40.603439+0000","last_undegraded":"2024-09-17T07:09:40.603439+0000","last_fullsized":"2024-09-17T07:09:40.603439+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'9","last_scrub_stamp":"2024-09-17T07:08:14.308970+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-09-17T07:03:29.961439+0000","last_clean_scrub_stamp":"2024-09-17T07:08:14.308970+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001890011,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"75'1","reported_seq":185,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.243279+0000","last_change":"2024-09-17T07:09:24.411824+0000","last_active":"2024-09-17T07:09:39.243279+0000","last_peered":"2024-09-17T07:09:39.243279+0000","last_clean":"2024-09-17T07:09:39.243279+0000","last_became_active":"2024-09-17T07:09:24.399565+0000","last_became_peered":"2024-09-17T07:09:24.399565+0000","last_unstale":"2024-09-17T07:09:39.243279+0000","last_undegraded":"2024-09-17T07:09:39.243279+0000","last_fullsized":"2024-09-17T07:09:39.243279+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-17T07:08:25.355935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:08:25.355935+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:44.557324+0000","scrub_duration":0.0052992940000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"199'7917","reported_seq":12271,"reported_epoch":199,"state":"active+remapped+backfilling","last_fresh":"2024-09-17T07:09:43.114587+0000","last_change":"2024-09-17T07:09:25.539073+0000","last_active":"2024-09-17T07:09:43.114587+0000","last_peered":"2024-09-17T07:09:43.114587+0000","last_clean":"2024-09-17T07:09:23.376900+0000","last_became_active":"2024-09-17T07:09:25.511414+0000","last_became_peered":"2024-09-17T07:09:25.511414+0000","last_unstale":"2024-09-17T07:09:43.114587+0000","last_undegraded":"2024-09-17T07:09:43.114587+0000","last_fullsized":"2024-09-17T07:09:43.114587+0000","mapping_epoch":195,"log_start":"182'3154","ondisk_log_start":"182'3154","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_clean_scrub_stamp":"2024-09-17T07:07:59.220681+0000","objects_scrubbed":0,"log_size":4763,"log_dups_size":0,"ondisk_log_size":4763,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":518848512,"num_objects":7917,"num_object_clones":0,"num_object_copies":23751,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5608,"num_objects_unfound":0,"num_objects_dirty":7917,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7917,"num_write_kb":506688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2152,"num_bytes_recovered":141033472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,3],"avail_no_missing":["5","3","7"],"object_location_counts":[{"shards":"3,5,7","objects":7917}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"196'29","reported_seq":343,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:40.605397+0000","last_change":"2024-09-17T07:09:24.422580+0000","last_active":"2024-09-17T07:09:40.605397+0000","last_peered":"2024-09-17T07:09:40.605397+0000","last_clean":"2024-09-17T07:09:40.605397+0000","last_became_active":"2024-09-17T07:09:24.403267+0000","last_became_peered":"2024-09-17T07:09:24.403267+0000","last_unstale":"2024-09-17T07:09:40.605397+0000","last_undegraded":"2024-09-17T07:09:40.605397+0000","last_fullsized":"2024-09-17T07:09:40.605397+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'28","last_scrub_stamp":"2024-09-17T07:08:26.795647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:08:26.795647+0000","objects_scrubbed":10,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0043656659999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":187,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.244334+0000","last_change":"2024-09-17T07:09:19.253480+0000","last_active":"2024-09-17T07:09:39.244334+0000","last_peered":"2024-09-17T07:09:39.244334+0000","last_clean":"2024-09-17T07:09:39.244334+0000","last_became_active":"2024-09-17T07:02:12.008802+0000","last_became_peered":"2024-09-17T07:02:12.008802+0000","last_unstale":"2024-09-17T07:09:39.244334+0000","last_undegraded":"2024-09-17T07:09:39.244334+0000","last_fullsized":"2024-09-17T07:09:39.244334+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:19.253385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:09:19.253385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:36.878327+0000","scrub_duration":0.0017535820000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":173,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.251283+0000","last_change":"2024-09-17T07:08:04.266668+0000","last_active":"2024-09-17T07:09:39.251283+0000","last_peered":"2024-09-17T07:09:39.251283+0000","last_clean":"2024-09-17T07:09:39.251283+0000","last_became_active":"2024-09-17T07:02:16.016606+0000","last_became_peered":"2024-09-17T07:02:16.016606+0000","last_unstale":"2024-09-17T07:09:39.251283+0000","last_undegraded":"2024-09-17T07:09:39.251283+0000","last_fullsized":"2024-09-17T07:09:39.251283+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:04.266574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:08:04.266574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014388140000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":16,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.279498+0000","last_change":"2024-09-17T07:09:24.403411+0000","last_active":"2024-09-17T07:09:39.279498+0000","last_peered":"2024-09-17T07:09:39.279498+0000","last_clean":"2024-09-17T07:09:39.279498+0000","last_became_active":"2024-09-17T07:09:24.398878+0000","last_became_peered":"2024-09-17T07:09:24.398878+0000","last_unstale":"2024-09-17T07:09:39.279498+0000","last_undegraded":"2024-09-17T07:09:39.279498+0000","last_fullsized":"2024-09-17T07:09:39.279498+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:08.849142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:08:08.849142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.b","version":"199'8113","reported_seq":13363,"reported_epoch":199,"state":"active+remapped+backfilling","last_fresh":"2024-09-17T07:09:43.083804+0000","last_change":"2024-09-17T07:09:25.458949+0000","last_active":"2024-09-17T07:09:43.083804+0000","last_peered":"2024-09-17T07:09:43.083804+0000","last_clean":"2024-09-17T07:09:23.353085+0000","last_became_active":"2024-09-17T07:09:25.453311+0000","last_became_peered":"2024-09-17T07:09:25.453311+0000","last_unstale":"2024-09-17T07:09:43.083804+0000","last_undegraded":"2024-09-17T07:09:43.083804+0000","last_fullsized":"2024-09-17T07:09:43.083804+0000","mapping_epoch":195,"log_start":"182'3270","ondisk_log_start":"182'3270","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_clean_scrub_stamp":"2024-09-17T07:07:59.220681+0000","objects_scrubbed":0,"log_size":4843,"log_dups_size":0,"ondisk_log_size":4843,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":531693568,"num_objects":8113,"num_object_clones":0,"num_object_copies":24339,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5323,"num_objects_unfound":0,"num_objects_dirty":8113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8113,"num_write_kb":519232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2600,"num_bytes_recovered":170393600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,3],"avail_no_missing":["4","1","3"],"object_location_counts":[{"shards":"1,3,4","objects":8113}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.7","version":"75'13","reported_seq":201,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.255683+0000","last_change":"2024-09-17T07:08:05.678169+0000","last_active":"2024-09-17T07:09:39.255683+0000","last_peered":"2024-09-17T07:09:39.255683+0000","last_clean":"2024-09-17T07:09:39.255683+0000","last_became_active":"2024-09-17T07:02:14.019838+0000","last_became_peered":"2024-09-17T07:02:14.019838+0000","last_unstale":"2024-09-17T07:09:39.255683+0000","last_undegraded":"2024-09-17T07:09:39.255683+0000","last_fullsized":"2024-09-17T07:09:39.255683+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'13","last_scrub_stamp":"2024-09-17T07:08:05.678063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:08:05.678063+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0024636459999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"76'5","reported_seq":628,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-17T07:09:42.236136+0000","last_change":"2024-09-17T07:08:29.818116+0000","last_active":"2024-09-17T07:09:42.236136+0000","last_peered":"2024-09-17T07:09:42.236136+0000","last_clean":"2024-09-17T07:09:42.236136+0000","last_became_active":"2024-09-17T07:02:12.006883+0000","last_became_peered":"2024-09-17T07:02:12.006883+0000","last_unstale":"2024-09-17T07:09:42.236136+0000","last_undegraded":"2024-09-17T07:09:42.236136+0000","last_fullsized":"2024-09-17T07:09:42.236136+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'5","last_scrub_stamp":"2024-09-17T07:08:29.817785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:08:29.817785+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:45.372335+0000","scrub_duration":0.0034070260000000001,"objects_trimmed":0,"snaptrim_duration":0.00033739799999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":444,"num_read_kb":439,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":18,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.251680+0000","last_change":"2024-09-17T07:09:24.393139+0000","last_active":"2024-09-17T07:09:39.251680+0000","last_peered":"2024-09-17T07:09:39.251680+0000","last_clean":"2024-09-17T07:09:39.251680+0000","last_became_active":"2024-09-17T07:09:24.392850+0000","last_became_peered":"2024-09-17T07:09:24.392850+0000","last_unstale":"2024-09-17T07:09:39.251680+0000","last_undegraded":"2024-09-17T07:09:39.251680+0000","last_fullsized":"2024-09-17T07:09:39.251680+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:23.833170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:23.833170+0000","last_clean_scrub_stamp":"2024-09-17T07:08:23.833170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":173,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.250288+0000","last_change":"2024-09-17T07:08:14.935764+0000","last_active":"2024-09-17T07:09:39.250288+0000","last_peered":"2024-09-17T07:09:39.250288+0000","last_clean":"2024-09-17T07:09:39.250288+0000","last_became_active":"2024-09-17T07:02:16.014492+0000","last_became_peered":"2024-09-17T07:02:16.014492+0000","last_unstale":"2024-09-17T07:09:39.250288+0000","last_undegraded":"2024-09-17T07:09:39.250288+0000","last_fullsized":"2024-09-17T07:09:39.250288+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:14.935679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:08:14.935679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001590459,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":191,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.255506+0000","last_change":"2024-09-17T07:09:24.403678+0000","last_active":"2024-09-17T07:09:39.255506+0000","last_peered":"2024-09-17T07:09:39.255506+0000","last_clean":"2024-09-17T07:09:39.255506+0000","last_became_active":"2024-09-17T07:09:24.402704+0000","last_became_peered":"2024-09-17T07:09:24.402704+0000","last_unstale":"2024-09-17T07:09:39.255506+0000","last_undegraded":"2024-09-17T07:09:39.255506+0000","last_fullsized":"2024-09-17T07:09:39.255506+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:22.853652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:09:22.853652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:26.725996+0000","scrub_duration":0.0041575259999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"199'7800","reported_seq":7840,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-17T07:09:43.285921+0000","last_change":"2024-09-17T07:08:00.760107+0000","last_active":"2024-09-17T07:09:43.285921+0000","last_peered":"2024-09-17T07:09:43.285921+0000","last_clean":"2024-09-17T07:09:43.285921+0000","last_became_active":"2024-09-17T07:08:00.759827+0000","last_became_peered":"2024-09-17T07:08:00.759827+0000","last_unstale":"2024-09-17T07:09:43.285921+0000","last_undegraded":"2024-09-17T07:09:43.285921+0000","last_fullsized":"2024-09-17T07:09:43.285921+0000","mapping_epoch":168,"log_start":"183'3300","ondisk_log_start":"183'3300","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_clean_scrub_stamp":"2024-09-17T07:07:59.220681+0000","objects_scrubbed":0,"log_size":4500,"log_dups_size":0,"ondisk_log_size":4500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":511180800,"num_objects":7800,"num_object_clones":0,"num_object_copies":23400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7800,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7800,"num_write_kb":499200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"75'12","reported_seq":216,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.251601+0000","last_change":"2024-09-17T07:08:08.295553+0000","last_active":"2024-09-17T07:09:39.251601+0000","last_peered":"2024-09-17T07:09:39.251601+0000","last_clean":"2024-09-17T07:09:39.251601+0000","last_became_active":"2024-09-17T07:03:48.760405+0000","last_became_peered":"2024-09-17T07:03:48.760405+0000","last_unstale":"2024-09-17T07:09:39.251601+0000","last_undegraded":"2024-09-17T07:09:39.251601+0000","last_fullsized":"2024-09-17T07:09:39.251601+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-09-17T07:08:08.295449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:08:08.295449+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023245919999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"70'1","reported_seq":196,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.251075+0000","last_change":"2024-09-17T07:09:24.415023+0000","last_active":"2024-09-17T07:09:39.251075+0000","last_peered":"2024-09-17T07:09:39.251075+0000","last_clean":"2024-09-17T07:09:39.251075+0000","last_became_active":"2024-09-17T07:09:24.402499+0000","last_became_peered":"2024-09-17T07:09:24.402499+0000","last_unstale":"2024-09-17T07:09:39.251075+0000","last_undegraded":"2024-09-17T07:09:39.251075+0000","last_fullsized":"2024-09-17T07:09:39.251075+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-17T07:09:22.249833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:09:22.249833+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:45.077540+0000","scrub_duration":0.0061513059999999996,"objects_trimmed":0,"snaptrim_duration":0.0032298629999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":177,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.253346+0000","last_change":"2024-09-17T07:09:16.865242+0000","last_active":"2024-09-17T07:09:39.253346+0000","last_peered":"2024-09-17T07:09:39.253346+0000","last_clean":"2024-09-17T07:09:39.253346+0000","last_became_active":"2024-09-17T07:02:12.014623+0000","last_became_peered":"2024-09-17T07:02:12.014623+0000","last_unstale":"2024-09-17T07:09:39.253346+0000","last_undegraded":"2024-09-17T07:09:39.253346+0000","last_fullsized":"2024-09-17T07:09:39.253346+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:16.865147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:09:16.865147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:30.769978+0000","scrub_duration":0.005396222,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":172,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.243171+0000","last_change":"2024-09-17T07:08:18.008734+0000","last_active":"2024-09-17T07:09:39.243171+0000","last_peered":"2024-09-17T07:09:39.243171+0000","last_clean":"2024-09-17T07:09:39.243171+0000","last_became_active":"2024-09-17T07:02:16.021561+0000","last_became_peered":"2024-09-17T07:02:16.021561+0000","last_unstale":"2024-09-17T07:09:39.243171+0000","last_undegraded":"2024-09-17T07:09:39.243171+0000","last_fullsized":"2024-09-17T07:09:39.243171+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:18.008600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:08:18.008600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0044198520000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":179,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.240249+0000","last_change":"2024-09-17T07:09:24.396448+0000","last_active":"2024-09-17T07:09:39.240249+0000","last_peered":"2024-09-17T07:09:39.240249+0000","last_clean":"2024-09-17T07:09:39.240249+0000","last_became_active":"2024-09-17T07:09:24.396154+0000","last_became_peered":"2024-09-17T07:09:24.396154+0000","last_unstale":"2024-09-17T07:09:39.240249+0000","last_undegraded":"2024-09-17T07:09:39.240249+0000","last_fullsized":"2024-09-17T07:09:39.240249+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:17.192238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:09:17.192238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:20.769603+0000","scrub_duration":0.058290885000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"199'7726","reported_seq":7907,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-17T07:09:43.143887+0000","last_change":"2024-09-17T07:09:23.077196+0000","last_active":"2024-09-17T07:09:43.143887+0000","last_peered":"2024-09-17T07:09:43.143887+0000","last_clean":"2024-09-17T07:09:43.143887+0000","last_became_active":"2024-09-17T07:08:00.242803+0000","last_became_peered":"2024-09-17T07:08:00.242803+0000","last_unstale":"2024-09-17T07:09:43.143887+0000","last_undegraded":"2024-09-17T07:09:43.143887+0000","last_fullsized":"2024-09-17T07:09:43.143887+0000","mapping_epoch":168,"log_start":"181'2755","ondisk_log_start":"181'2755","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"193'6546","last_scrub_stamp":"2024-09-17T07:09:23.076982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_clean_scrub_stamp":"2024-09-17T07:09:23.076982+0000","objects_scrubbed":6477,"log_size":4971,"log_dups_size":0,"ondisk_log_size":4971,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:40.229937+0000","scrub_duration":1.8549729770000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":506331136,"num_objects":7726,"num_object_clones":0,"num_object_copies":23178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7726,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7726,"num_write_kb":494464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"75'14","reported_seq":197,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.256416+0000","last_change":"2024-09-17T07:08:28.485668+0000","last_active":"2024-09-17T07:09:39.256416+0000","last_peered":"2024-09-17T07:09:39.256416+0000","last_clean":"2024-09-17T07:09:39.256416+0000","last_became_active":"2024-09-17T07:02:14.028857+0000","last_became_peered":"2024-09-17T07:02:14.028857+0000","last_unstale":"2024-09-17T07:09:39.256416+0000","last_undegraded":"2024-09-17T07:09:39.256416+0000","last_fullsized":"2024-09-17T07:09:39.256416+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'14","last_scrub_stamp":"2024-09-17T07:08:28.485338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:08:28.485338+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:54.485230+0000","scrub_duration":0.0037222499999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"72'2","reported_seq":35,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.244279+0000","last_change":"2024-09-17T07:09:24.420989+0000","last_active":"2024-09-17T07:09:39.244279+0000","last_peered":"2024-09-17T07:09:39.244279+0000","last_clean":"2024-09-17T07:09:39.244279+0000","last_became_active":"2024-09-17T07:09:24.404532+0000","last_became_peered":"2024-09-17T07:09:24.404532+0000","last_unstale":"2024-09-17T07:09:39.244279+0000","last_undegraded":"2024-09-17T07:09:39.244279+0000","last_fullsized":"2024-09-17T07:09:39.244279+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'2","last_scrub_stamp":"2024-09-17T07:08:15.763727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:08:15.763727+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":187,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.242582+0000","last_change":"2024-09-17T07:09:16.180998+0000","last_active":"2024-09-17T07:09:39.242582+0000","last_peered":"2024-09-17T07:09:39.242582+0000","last_clean":"2024-09-17T07:09:39.242582+0000","last_became_active":"2024-09-17T07:02:12.004606+0000","last_became_peered":"2024-09-17T07:02:12.004606+0000","last_unstale":"2024-09-17T07:09:39.242582+0000","last_undegraded":"2024-09-17T07:09:39.242582+0000","last_fullsized":"2024-09-17T07:09:39.242582+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:16.180756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:09:16.180756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:20.772806+0000","scrub_duration":0.014479524000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"78'128","reported_seq":425,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.253510+0000","last_change":"2024-09-17T07:08:14.128641+0000","last_active":"2024-09-17T07:09:39.253510+0000","last_peered":"2024-09-17T07:09:39.253510+0000","last_clean":"2024-09-17T07:09:39.253510+0000","last_became_active":"2024-09-17T07:02:03.020675+0000","last_became_peered":"2024-09-17T07:02:03.020675+0000","last_unstale":"2024-09-17T07:09:39.253510+0000","last_undegraded":"2024-09-17T07:09:39.253510+0000","last_fullsized":"2024-09-17T07:09:39.253510+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'128","last_scrub_stamp":"2024-09-17T07:08:14.128519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:00:09.390381+0000","last_clean_scrub_stamp":"2024-09-17T07:08:14.128519+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021141889999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":173,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.254657+0000","last_change":"2024-09-17T07:08:05.064380+0000","last_active":"2024-09-17T07:09:39.254657+0000","last_peered":"2024-09-17T07:09:39.254657+0000","last_clean":"2024-09-17T07:09:39.254657+0000","last_became_active":"2024-09-17T07:02:16.016031+0000","last_became_peered":"2024-09-17T07:02:16.016031+0000","last_unstale":"2024-09-17T07:09:39.254657+0000","last_undegraded":"2024-09-17T07:09:39.254657+0000","last_fullsized":"2024-09-17T07:09:39.254657+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:05.064241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:08:05.064241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015017990000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":179,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.243305+0000","last_change":"2024-09-17T07:09:24.400356+0000","last_active":"2024-09-17T07:09:39.243305+0000","last_peered":"2024-09-17T07:09:39.243305+0000","last_clean":"2024-09-17T07:09:39.243305+0000","last_became_active":"2024-09-17T07:09:24.400041+0000","last_became_peered":"2024-09-17T07:09:24.400041+0000","last_unstale":"2024-09-17T07:09:39.243305+0000","last_undegraded":"2024-09-17T07:09:39.243305+0000","last_fullsized":"2024-09-17T07:09:39.243305+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:15.194496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:58.670571+0000","last_clean_scrub_stamp":"2024-09-17T07:09:15.194496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:22.750595+0000","scrub_duration":0.0091630700000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"199'8031","reported_seq":8070,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-17T07:09:44.098346+0000","last_change":"2024-09-17T07:08:00.599954+0000","last_active":"2024-09-17T07:09:44.098346+0000","last_peered":"2024-09-17T07:09:44.098346+0000","last_clean":"2024-09-17T07:09:44.098346+0000","last_became_active":"2024-09-17T07:08:00.599685+0000","last_became_peered":"2024-09-17T07:08:00.599685+0000","last_unstale":"2024-09-17T07:09:44.098346+0000","last_undegraded":"2024-09-17T07:09:44.098346+0000","last_fullsized":"2024-09-17T07:09:44.098346+0000","mapping_epoch":168,"log_start":"181'2941","ondisk_log_start":"181'2941","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_clean_scrub_stamp":"2024-09-17T07:07:59.220681+0000","objects_scrubbed":0,"log_size":5090,"log_dups_size":0,"ondisk_log_size":5090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":526319616,"num_objects":8031,"num_object_clones":0,"num_object_copies":24093,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8031,"num_write_kb":513984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"75'19","reported_seq":206,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.255028+0000","last_change":"2024-09-17T07:09:11.299059+0000","last_active":"2024-09-17T07:09:39.255028+0000","last_peered":"2024-09-17T07:09:39.255028+0000","last_clean":"2024-09-17T07:09:39.255028+0000","last_became_active":"2024-09-17T07:02:14.027313+0000","last_became_peered":"2024-09-17T07:02:14.027313+0000","last_unstale":"2024-09-17T07:09:39.255028+0000","last_undegraded":"2024-09-17T07:09:39.255028+0000","last_fullsized":"2024-09-17T07:09:39.255028+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-09-17T07:09:11.298950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:09:11.298950+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:23.840692+0000","scrub_duration":0.0052578449999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":188,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.249673+0000","last_change":"2024-09-17T07:08:15.952173+0000","last_active":"2024-09-17T07:09:39.249673+0000","last_peered":"2024-09-17T07:09:39.249673+0000","last_clean":"2024-09-17T07:09:39.249673+0000","last_became_active":"2024-09-17T07:02:12.012037+0000","last_became_peered":"2024-09-17T07:02:12.012037+0000","last_unstale":"2024-09-17T07:09:39.249673+0000","last_undegraded":"2024-09-17T07:09:39.249673+0000","last_fullsized":"2024-09-17T07:09:39.249673+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:15.952078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:06:39.322535+0000","last_clean_scrub_stamp":"2024-09-17T07:08:15.952078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017662470000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":184,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.250481+0000","last_change":"2024-09-17T07:08:17.003265+0000","last_active":"2024-09-17T07:09:39.250481+0000","last_peered":"2024-09-17T07:09:39.250481+0000","last_clean":"2024-09-17T07:09:39.250481+0000","last_became_active":"2024-09-17T07:02:16.014631+0000","last_became_peered":"2024-09-17T07:02:16.014631+0000","last_unstale":"2024-09-17T07:09:39.250481+0000","last_undegraded":"2024-09-17T07:09:39.250481+0000","last_fullsized":"2024-09-17T07:09:39.250481+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:17.003116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:08:17.003116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035024880000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":180,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.253424+0000","last_change":"2024-09-17T07:09:24.404437+0000","last_active":"2024-09-17T07:09:39.253424+0000","last_peered":"2024-09-17T07:09:39.253424+0000","last_clean":"2024-09-17T07:09:39.253424+0000","last_became_active":"2024-09-17T07:09:24.403904+0000","last_became_peered":"2024-09-17T07:09:24.403904+0000","last_unstale":"2024-09-17T07:09:39.253424+0000","last_undegraded":"2024-09-17T07:09:39.253424+0000","last_fullsized":"2024-09-17T07:09:39.253424+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:10.781244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:09:10.781244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:39.854595+0000","scrub_duration":0.010463343,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.7","version":"199'8066","reported_seq":11865,"reported_epoch":199,"state":"active+remapped+backfilling","last_fresh":"2024-09-17T07:09:43.367829+0000","last_change":"2024-09-17T07:09:25.506880+0000","last_active":"2024-09-17T07:09:43.367829+0000","last_peered":"2024-09-17T07:09:43.367829+0000","last_clean":"2024-09-17T07:09:23.361471+0000","last_became_active":"2024-09-17T07:09:25.487755+0000","last_became_peered":"2024-09-17T07:09:25.487755+0000","last_unstale":"2024-09-17T07:09:43.367829+0000","last_undegraded":"2024-09-17T07:09:43.367829+0000","last_fullsized":"2024-09-17T07:09:43.367829+0000","mapping_epoch":195,"log_start":"183'3585","ondisk_log_start":"183'3585","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_clean_scrub_stamp":"2024-09-17T07:07:59.220681+0000","objects_scrubbed":0,"log_size":4481,"log_dups_size":0,"ondisk_log_size":4481,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":528613376,"num_objects":8066,"num_object_clones":0,"num_object_copies":24198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6022,"num_objects_unfound":0,"num_objects_dirty":8066,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8066,"num_write_kb":516224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1876,"num_bytes_recovered":122945536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,3],"avail_no_missing":["1","3","6"],"object_location_counts":[{"shards":"1,3,6","objects":8066}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"75'9","reported_seq":191,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.250930+0000","last_change":"2024-09-17T07:08:30.220173+0000","last_active":"2024-09-17T07:09:39.250930+0000","last_peered":"2024-09-17T07:09:39.250930+0000","last_clean":"2024-09-17T07:09:39.250930+0000","last_became_active":"2024-09-17T07:02:14.014939+0000","last_became_peered":"2024-09-17T07:02:14.014939+0000","last_unstale":"2024-09-17T07:09:39.250930+0000","last_undegraded":"2024-09-17T07:09:39.250930+0000","last_fullsized":"2024-09-17T07:09:39.250930+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:08:30.219974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:08:30.219974+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033974780000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":185,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.252948+0000","last_change":"2024-09-17T07:09:24.391612+0000","last_active":"2024-09-17T07:09:39.252948+0000","last_peered":"2024-09-17T07:09:39.252948+0000","last_clean":"2024-09-17T07:09:39.252948+0000","last_became_active":"2024-09-17T07:09:24.391207+0000","last_became_peered":"2024-09-17T07:09:24.391207+0000","last_unstale":"2024-09-17T07:09:39.252948+0000","last_undegraded":"2024-09-17T07:09:39.252948+0000","last_fullsized":"2024-09-17T07:09:39.252948+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:20.806468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:06:30.038104+0000","last_clean_scrub_stamp":"2024-09-17T07:08:20.806468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.005853245,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":136,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.280015+0000","last_change":"2024-09-17T07:09:24.404480+0000","last_active":"2024-09-17T07:09:39.280015+0000","last_peered":"2024-09-17T07:09:39.280015+0000","last_clean":"2024-09-17T07:09:39.280015+0000","last_became_active":"2024-09-17T07:09:24.396642+0000","last_became_peered":"2024-09-17T07:09:24.396642+0000","last_unstale":"2024-09-17T07:09:39.280015+0000","last_undegraded":"2024-09-17T07:09:39.280015+0000","last_fullsized":"2024-09-17T07:09:39.280015+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:19.789134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:07:19.789134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010505654,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":171,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.250990+0000","last_change":"2024-09-17T07:09:08.316161+0000","last_active":"2024-09-17T07:09:39.250990+0000","last_peered":"2024-09-17T07:09:39.250990+0000","last_clean":"2024-09-17T07:09:39.250990+0000","last_became_active":"2024-09-17T07:02:18.026994+0000","last_became_peered":"2024-09-17T07:02:18.026994+0000","last_unstale":"2024-09-17T07:09:39.250990+0000","last_undegraded":"2024-09-17T07:09:39.250990+0000","last_fullsized":"2024-09-17T07:09:39.250990+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:08.315842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:58.099849+0000","last_clean_scrub_stamp":"2024-09-17T07:09:08.315842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:28.565288+0000","scrub_duration":0.0081844919999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"199'8041","reported_seq":8081,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-17T07:09:43.072511+0000","last_change":"2024-09-17T07:08:00.760307+0000","last_active":"2024-09-17T07:09:43.072511+0000","last_peered":"2024-09-17T07:09:43.072511+0000","last_clean":"2024-09-17T07:09:43.072511+0000","last_became_active":"2024-09-17T07:08:00.759662+0000","last_became_peered":"2024-09-17T07:08:00.759662+0000","last_unstale":"2024-09-17T07:09:43.072511+0000","last_undegraded":"2024-09-17T07:09:43.072511+0000","last_fullsized":"2024-09-17T07:09:43.072511+0000","mapping_epoch":168,"log_start":"182'3273","ondisk_log_start":"182'3273","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_clean_scrub_stamp":"2024-09-17T07:07:59.220681+0000","objects_scrubbed":0,"log_size":4768,"log_dups_size":0,"ondisk_log_size":4768,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":526974976,"num_objects":8041,"num_object_clones":0,"num_object_copies":24123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8041,"num_write_kb":514624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"111'22","reported_seq":206,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.242154+0000","last_change":"2024-09-17T07:07:29.816377+0000","last_active":"2024-09-17T07:09:39.242154+0000","last_peered":"2024-09-17T07:09:39.242154+0000","last_clean":"2024-09-17T07:09:39.242154+0000","last_became_active":"2024-09-17T07:02:14.028392+0000","last_became_peered":"2024-09-17T07:02:14.028392+0000","last_unstale":"2024-09-17T07:09:39.242154+0000","last_undegraded":"2024-09-17T07:09:39.242154+0000","last_fullsized":"2024-09-17T07:09:39.242154+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'22","last_scrub_stamp":"2024-09-17T07:07:29.816241+0000","last_deep_scrub":"111'22","last_deep_scrub_stamp":"2024-09-17T07:04:57.500450+0000","last_clean_scrub_stamp":"2024-09-17T07:07:29.816241+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034677570000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'2","reported_seq":60,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.255829+0000","last_change":"2024-09-17T07:09:24.428473+0000","last_active":"2024-09-17T07:09:39.255829+0000","last_peered":"2024-09-17T07:09:39.255829+0000","last_clean":"2024-09-17T07:09:39.255829+0000","last_became_active":"2024-09-17T07:09:24.402909+0000","last_became_peered":"2024-09-17T07:09:24.402909+0000","last_unstale":"2024-09-17T07:09:39.255829+0000","last_undegraded":"2024-09-17T07:09:39.255829+0000","last_fullsized":"2024-09-17T07:09:39.255829+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-09-17T07:09:19.720425+0000","last_deep_scrub":"89'2","last_deep_scrub_stamp":"2024-09-17T07:03:26.694366+0000","last_clean_scrub_stamp":"2024-09-17T07:09:19.720425+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:33.916032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":420,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":420,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"75'8","reported_seq":274,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:40.603646+0000","last_change":"2024-09-17T07:08:20.045908+0000","last_active":"2024-09-17T07:09:40.603646+0000","last_peered":"2024-09-17T07:09:40.603646+0000","last_clean":"2024-09-17T07:09:40.603646+0000","last_became_active":"2024-09-17T07:02:16.015418+0000","last_became_peered":"2024-09-17T07:02:16.015418+0000","last_unstale":"2024-09-17T07:09:40.603646+0000","last_undegraded":"2024-09-17T07:09:40.603646+0000","last_fullsized":"2024-09-17T07:09:40.603646+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:08:20.045781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:08:20.045781+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:45.677411+0000","scrub_duration":0.0023108299999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":170,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.244084+0000","last_change":"2024-09-17T07:07:19.786753+0000","last_active":"2024-09-17T07:09:39.244084+0000","last_peered":"2024-09-17T07:09:39.244084+0000","last_clean":"2024-09-17T07:09:39.244084+0000","last_became_active":"2024-09-17T07:02:18.363253+0000","last_became_peered":"2024-09-17T07:02:18.363253+0000","last_unstale":"2024-09-17T07:09:39.244084+0000","last_undegraded":"2024-09-17T07:09:39.244084+0000","last_fullsized":"2024-09-17T07:09:39.244084+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:19.786442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:07:19.786442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011747911999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.5","version":"199'8023","reported_seq":8063,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-17T07:09:43.053735+0000","last_change":"2024-09-17T07:08:00.598660+0000","last_active":"2024-09-17T07:09:43.053735+0000","last_peered":"2024-09-17T07:09:43.053735+0000","last_clean":"2024-09-17T07:09:43.053735+0000","last_became_active":"2024-09-17T07:08:00.598353+0000","last_became_peered":"2024-09-17T07:08:00.598353+0000","last_unstale":"2024-09-17T07:09:43.053735+0000","last_undegraded":"2024-09-17T07:09:43.053735+0000","last_fullsized":"2024-09-17T07:09:43.053735+0000","mapping_epoch":168,"log_start":"181'3095","ondisk_log_start":"181'3095","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_clean_scrub_stamp":"2024-09-17T07:07:59.220681+0000","objects_scrubbed":0,"log_size":4928,"log_dups_size":0,"ondisk_log_size":4928,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":525795328,"num_objects":8023,"num_object_clones":0,"num_object_copies":24069,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8023,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8023,"num_write_kb":513472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"150'17","reported_seq":224,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.279233+0000","last_change":"2024-09-17T07:09:24.423923+0000","last_active":"2024-09-17T07:09:39.279233+0000","last_peered":"2024-09-17T07:09:39.279233+0000","last_clean":"2024-09-17T07:09:39.279233+0000","last_became_active":"2024-09-17T07:09:24.399164+0000","last_became_peered":"2024-09-17T07:09:24.399164+0000","last_unstale":"2024-09-17T07:09:39.279233+0000","last_undegraded":"2024-09-17T07:09:39.279233+0000","last_fullsized":"2024-09-17T07:09:39.279233+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'17","last_scrub_stamp":"2024-09-17T07:08:34.714204+0000","last_deep_scrub":"150'17","last_deep_scrub_stamp":"2024-09-17T07:06:38.775788+0000","last_clean_scrub_stamp":"2024-09-17T07:08:34.714204+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0065052799999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"68'2","reported_seq":174,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.279949+0000","last_change":"2024-09-17T07:07:20.764435+0000","last_active":"2024-09-17T07:09:39.279949+0000","last_peered":"2024-09-17T07:09:39.279949+0000","last_clean":"2024-09-17T07:09:39.279949+0000","last_became_active":"2024-09-17T07:02:12.006716+0000","last_became_peered":"2024-09-17T07:02:12.006716+0000","last_unstale":"2024-09-17T07:09:39.279949+0000","last_undegraded":"2024-09-17T07:09:39.279949+0000","last_fullsized":"2024-09-17T07:09:39.279949+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-09-17T07:07:20.764283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:07:20.764283+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0060160040000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":137,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.243391+0000","last_change":"2024-09-17T07:08:10.273572+0000","last_active":"2024-09-17T07:09:39.243391+0000","last_peered":"2024-09-17T07:09:39.243391+0000","last_clean":"2024-09-17T07:09:39.243391+0000","last_became_active":"2024-09-17T07:04:15.111069+0000","last_became_peered":"2024-09-17T07:04:15.111069+0000","last_unstale":"2024-09-17T07:09:39.243391+0000","last_undegraded":"2024-09-17T07:09:39.243391+0000","last_fullsized":"2024-09-17T07:09:39.243391+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:10.273485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:08:10.273485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013521,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":17,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.253469+0000","last_change":"2024-09-17T07:09:33.811857+0000","last_active":"2024-09-17T07:09:39.253469+0000","last_peered":"2024-09-17T07:09:39.253469+0000","last_clean":"2024-09-17T07:09:39.253469+0000","last_became_active":"2024-09-17T07:09:24.397669+0000","last_became_peered":"2024-09-17T07:09:24.397669+0000","last_unstale":"2024-09-17T07:09:39.253469+0000","last_undegraded":"2024-09-17T07:09:39.253469+0000","last_fullsized":"2024-09-17T07:09:39.253469+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:07.878506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:08:07.878506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"75'9","reported_seq":210,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.251561+0000","last_change":"2024-09-17T07:09:24.429695+0000","last_active":"2024-09-17T07:09:39.251561+0000","last_peered":"2024-09-17T07:09:39.251561+0000","last_clean":"2024-09-17T07:09:39.251561+0000","last_became_active":"2024-09-17T07:09:24.408875+0000","last_became_peered":"2024-09-17T07:09:24.408875+0000","last_unstale":"2024-09-17T07:09:39.251561+0000","last_undegraded":"2024-09-17T07:09:39.251561+0000","last_fullsized":"2024-09-17T07:09:39.251561+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:08:19.316248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:08:19.316248+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0059518610000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":188,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.228317+0000","last_change":"2024-09-17T07:08:17.976403+0000","last_active":"2024-09-17T07:09:39.228317+0000","last_peered":"2024-09-17T07:09:39.228317+0000","last_clean":"2024-09-17T07:09:39.228317+0000","last_became_active":"2024-09-17T07:02:12.011327+0000","last_became_peered":"2024-09-17T07:02:12.011327+0000","last_unstale":"2024-09-17T07:09:39.228317+0000","last_undegraded":"2024-09-17T07:09:39.228317+0000","last_fullsized":"2024-09-17T07:09:39.228317+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:17.976267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:17.976267+0000","last_clean_scrub_stamp":"2024-09-17T07:08:17.976267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:43.173493+0000","scrub_duration":0.0035138019999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":16,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.252068+0000","last_change":"2024-09-17T07:09:24.409485+0000","last_active":"2024-09-17T07:09:39.252068+0000","last_peered":"2024-09-17T07:09:39.252068+0000","last_clean":"2024-09-17T07:09:39.252068+0000","last_became_active":"2024-09-17T07:09:24.407806+0000","last_became_peered":"2024-09-17T07:09:24.407806+0000","last_unstale":"2024-09-17T07:09:39.252068+0000","last_undegraded":"2024-09-17T07:09:39.252068+0000","last_fullsized":"2024-09-17T07:09:39.252068+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:22.799056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:08:22.799056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:43.870719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":193,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.249402+0000","last_change":"2024-09-17T07:09:23.379011+0000","last_active":"2024-09-17T07:09:39.249402+0000","last_peered":"2024-09-17T07:09:39.249402+0000","last_clean":"2024-09-17T07:09:39.249402+0000","last_became_active":"2024-09-17T07:02:18.017825+0000","last_became_peered":"2024-09-17T07:02:18.017825+0000","last_unstale":"2024-09-17T07:09:39.249402+0000","last_undegraded":"2024-09-17T07:09:39.249402+0000","last_fullsized":"2024-09-17T07:09:39.249402+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:23.378805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:09:23.378805+0000","last_clean_scrub_stamp":"2024-09-17T07:09:23.378805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:33.079700+0000","scrub_duration":0.0078152310000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":177,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.254812+0000","last_change":"2024-09-17T07:09:33.804153+0000","last_active":"2024-09-17T07:09:39.254812+0000","last_peered":"2024-09-17T07:09:39.254812+0000","last_clean":"2024-09-17T07:09:39.254812+0000","last_became_active":"2024-09-17T07:02:12.008065+0000","last_became_peered":"2024-09-17T07:02:12.008065+0000","last_unstale":"2024-09-17T07:09:39.254812+0000","last_undegraded":"2024-09-17T07:09:39.254812+0000","last_fullsized":"2024-09-17T07:09:39.254812+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:04.480504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:08:04.480504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013838889999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"75'6","reported_seq":46,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.254378+0000","last_change":"2024-09-17T07:09:24.433976+0000","last_active":"2024-09-17T07:09:39.254378+0000","last_peered":"2024-09-17T07:09:39.254378+0000","last_clean":"2024-09-17T07:09:39.254378+0000","last_became_active":"2024-09-17T07:09:24.405385+0000","last_became_peered":"2024-09-17T07:09:24.405385+0000","last_unstale":"2024-09-17T07:09:39.254378+0000","last_undegraded":"2024-09-17T07:09:39.254378+0000","last_fullsized":"2024-09-17T07:09:39.254378+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'6","last_scrub_stamp":"2024-09-17T07:08:33.821542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:08:33.821542+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:49.932589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":181,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.242821+0000","last_change":"2024-09-17T07:09:24.399278+0000","last_active":"2024-09-17T07:09:39.242821+0000","last_peered":"2024-09-17T07:09:39.242821+0000","last_clean":"2024-09-17T07:09:39.242821+0000","last_became_active":"2024-09-17T07:09:24.398922+0000","last_became_peered":"2024-09-17T07:09:24.398922+0000","last_unstale":"2024-09-17T07:09:39.242821+0000","last_undegraded":"2024-09-17T07:09:39.242821+0000","last_fullsized":"2024-09-17T07:09:39.242821+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:06.932318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:08:06.932318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017321260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":171,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.255895+0000","last_change":"2024-09-17T07:08:33.926391+0000","last_active":"2024-09-17T07:09:39.255895+0000","last_peered":"2024-09-17T07:09:39.255895+0000","last_clean":"2024-09-17T07:09:39.255895+0000","last_became_active":"2024-09-17T07:02:18.365419+0000","last_became_peered":"2024-09-17T07:02:18.365419+0000","last_unstale":"2024-09-17T07:09:39.255895+0000","last_undegraded":"2024-09-17T07:09:39.255895+0000","last_fullsized":"2024-09-17T07:09:39.255895+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:33.926286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:08:33.926286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:56.396105+0000","scrub_duration":0.0014019519999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"75'10","reported_seq":56,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.253116+0000","last_change":"2024-09-17T07:09:24.435532+0000","last_active":"2024-09-17T07:09:39.253116+0000","last_peered":"2024-09-17T07:09:39.253116+0000","last_clean":"2024-09-17T07:09:39.253116+0000","last_became_active":"2024-09-17T07:09:24.405944+0000","last_became_peered":"2024-09-17T07:09:24.405944+0000","last_unstale":"2024-09-17T07:09:39.253116+0000","last_undegraded":"2024-09-17T07:09:39.253116+0000","last_fullsized":"2024-09-17T07:09:39.253116+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-09-17T07:08:18.723247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:08:18.723247+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":196,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.244217+0000","last_change":"2024-09-17T07:09:24.403615+0000","last_active":"2024-09-17T07:09:39.244217+0000","last_peered":"2024-09-17T07:09:39.244217+0000","last_clean":"2024-09-17T07:09:39.244217+0000","last_became_active":"2024-09-17T07:09:24.402355+0000","last_became_peered":"2024-09-17T07:09:24.402355+0000","last_unstale":"2024-09-17T07:09:39.244217+0000","last_undegraded":"2024-09-17T07:09:39.244217+0000","last_fullsized":"2024-09-17T07:09:39.244217+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:25.065984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:08:25.065984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:51.463782+0000","scrub_duration":0.0052651929999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"163'9","reported_seq":271,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:40.604258+0000","last_change":"2024-09-17T07:07:34.933956+0000","last_active":"2024-09-17T07:09:40.604258+0000","last_peered":"2024-09-17T07:09:40.604258+0000","last_clean":"2024-09-17T07:09:40.604258+0000","last_became_active":"2024-09-17T07:02:16.022583+0000","last_became_peered":"2024-09-17T07:02:16.022583+0000","last_unstale":"2024-09-17T07:09:40.604258+0000","last_undegraded":"2024-09-17T07:09:40.604258+0000","last_fullsized":"2024-09-17T07:09:40.604258+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'9","last_scrub_stamp":"2024-09-17T07:07:34.933808+0000","last_deep_scrub":"163'9","last_deep_scrub_stamp":"2024-09-17T07:07:34.933808+0000","last_clean_scrub_stamp":"2024-09-17T07:07:34.933808+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.019504579000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":180,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.251276+0000","last_change":"2024-09-17T07:09:24.409388+0000","last_active":"2024-09-17T07:09:39.251276+0000","last_peered":"2024-09-17T07:09:39.251276+0000","last_clean":"2024-09-17T07:09:39.251276+0000","last_became_active":"2024-09-17T07:09:24.409051+0000","last_became_peered":"2024-09-17T07:09:24.409051+0000","last_unstale":"2024-09-17T07:09:39.251276+0000","last_undegraded":"2024-09-17T07:09:39.251276+0000","last_fullsized":"2024-09-17T07:09:39.251276+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:12.338453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:09:12.338453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:19.395637+0000","scrub_duration":0.0011356249999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"75'9","reported_seq":220,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.242886+0000","last_change":"2024-09-17T07:09:24.418573+0000","last_active":"2024-09-17T07:09:39.242886+0000","last_peered":"2024-09-17T07:09:39.242886+0000","last_clean":"2024-09-17T07:09:39.242886+0000","last_became_active":"2024-09-17T07:09:24.406333+0000","last_became_peered":"2024-09-17T07:09:24.406333+0000","last_unstale":"2024-09-17T07:09:39.242886+0000","last_undegraded":"2024-09-17T07:09:39.242886+0000","last_fullsized":"2024-09-17T07:09:39.242886+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:09:09.142335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:09:09.142335+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:09.764304+0000","scrub_duration":0.0041253560000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":186,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.251183+0000","last_change":"2024-09-17T07:09:24.402987+0000","last_active":"2024-09-17T07:09:39.251183+0000","last_peered":"2024-09-17T07:09:39.251183+0000","last_clean":"2024-09-17T07:09:39.251183+0000","last_became_active":"2024-09-17T07:09:24.402704+0000","last_became_peered":"2024-09-17T07:09:24.402704+0000","last_unstale":"2024-09-17T07:09:39.251183+0000","last_undegraded":"2024-09-17T07:09:39.251183+0000","last_fullsized":"2024-09-17T07:09:39.251183+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:15.356187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:08:15.356187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:44.929336+0000","scrub_duration":0.0015925290000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"196'9","reported_seq":45,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:40.603827+0000","last_change":"2024-09-17T07:09:28.275921+0000","last_active":"2024-09-17T07:09:40.603827+0000","last_peered":"2024-09-17T07:09:40.603827+0000","last_clean":"2024-09-17T07:09:40.603827+0000","last_became_active":"2024-09-17T07:09:24.406267+0000","last_became_peered":"2024-09-17T07:09:24.406267+0000","last_unstale":"2024-09-17T07:09:40.603827+0000","last_undegraded":"2024-09-17T07:09:40.603827+0000","last_fullsized":"2024-09-17T07:09:40.603827+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:07:32.669921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:07:32.669921+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":171,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.279766+0000","last_change":"2024-09-17T07:08:16.703606+0000","last_active":"2024-09-17T07:09:39.279766+0000","last_peered":"2024-09-17T07:09:39.279766+0000","last_clean":"2024-09-17T07:09:39.279766+0000","last_became_active":"2024-09-17T07:02:18.018716+0000","last_became_peered":"2024-09-17T07:02:18.018716+0000","last_unstale":"2024-09-17T07:09:39.279766+0000","last_undegraded":"2024-09-17T07:09:39.279766+0000","last_fullsized":"2024-09-17T07:09:39.279766+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:16.703532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:08:16.703532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:45.505475+0000","scrub_duration":0.0019164340000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"75'9","reported_seq":190,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.252590+0000","last_change":"2024-09-17T07:08:16.756725+0000","last_active":"2024-09-17T07:09:39.252590+0000","last_peered":"2024-09-17T07:09:39.252590+0000","last_clean":"2024-09-17T07:09:39.252590+0000","last_became_active":"2024-09-17T07:02:14.010289+0000","last_became_peered":"2024-09-17T07:02:14.010289+0000","last_unstale":"2024-09-17T07:09:39.252590+0000","last_undegraded":"2024-09-17T07:09:39.252590+0000","last_fullsized":"2024-09-17T07:09:39.252590+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:08:16.756185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:08:16.756185+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004786584,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":186,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.256228+0000","last_change":"2024-09-17T07:09:24.408438+0000","last_active":"2024-09-17T07:09:39.256228+0000","last_peered":"2024-09-17T07:09:39.256228+0000","last_clean":"2024-09-17T07:09:39.256228+0000","last_became_active":"2024-09-17T07:09:24.408139+0000","last_became_peered":"2024-09-17T07:09:24.408139+0000","last_unstale":"2024-09-17T07:09:39.256228+0000","last_undegraded":"2024-09-17T07:09:39.256228+0000","last_fullsized":"2024-09-17T07:09:39.256228+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:07.329897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:09:07.329897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:12.235332+0000","scrub_duration":0.010864379,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":16,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.250666+0000","last_change":"2024-09-17T07:09:24.399604+0000","last_active":"2024-09-17T07:09:39.250666+0000","last_peered":"2024-09-17T07:09:39.250666+0000","last_clean":"2024-09-17T07:09:39.250666+0000","last_became_active":"2024-09-17T07:09:24.399116+0000","last_became_peered":"2024-09-17T07:09:24.399116+0000","last_unstale":"2024-09-17T07:09:39.250666+0000","last_undegraded":"2024-09-17T07:09:39.250666+0000","last_fullsized":"2024-09-17T07:09:39.250666+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:17.713027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:07:17.713027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":171,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.243314+0000","last_change":"2024-09-17T07:08:18.338994+0000","last_active":"2024-09-17T07:09:39.243314+0000","last_peered":"2024-09-17T07:09:39.243314+0000","last_clean":"2024-09-17T07:09:39.243314+0000","last_became_active":"2024-09-17T07:02:18.027622+0000","last_became_peered":"2024-09-17T07:02:18.027622+0000","last_unstale":"2024-09-17T07:09:39.243314+0000","last_undegraded":"2024-09-17T07:09:39.243314+0000","last_fullsized":"2024-09-17T07:09:39.243314+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:18.338863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:08:18.338863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0025905709999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"75'11","reported_seq":194,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.289390+0000","last_change":"2024-09-17T07:08:23.583577+0000","last_active":"2024-09-17T07:09:39.289390+0000","last_peered":"2024-09-17T07:09:39.289390+0000","last_clean":"2024-09-17T07:09:39.289390+0000","last_became_active":"2024-09-17T07:02:14.016255+0000","last_became_peered":"2024-09-17T07:02:14.016255+0000","last_unstale":"2024-09-17T07:09:39.289390+0000","last_undegraded":"2024-09-17T07:09:39.289390+0000","last_fullsized":"2024-09-17T07:09:39.289390+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-09-17T07:08:23.583408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:08:23.583408+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:43.889948+0000","scrub_duration":0.0056203690000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"94'2","reported_seq":182,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.252876+0000","last_change":"2024-09-17T07:09:18.169710+0000","last_active":"2024-09-17T07:09:39.252876+0000","last_peered":"2024-09-17T07:09:39.252876+0000","last_clean":"2024-09-17T07:09:39.252876+0000","last_became_active":"2024-09-17T07:02:12.011461+0000","last_became_peered":"2024-09-17T07:02:12.011461+0000","last_unstale":"2024-09-17T07:09:39.252876+0000","last_undegraded":"2024-09-17T07:09:39.252876+0000","last_fullsized":"2024-09-17T07:09:39.252876+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-09-17T07:09:18.169580+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-09-17T07:07:31.717888+0000","last_clean_scrub_stamp":"2024-09-17T07:09:18.169580+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:20.742759+0000","scrub_duration":0.011743155,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":181,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.252452+0000","last_change":"2024-09-17T07:09:24.407995+0000","last_active":"2024-09-17T07:09:39.252452+0000","last_peered":"2024-09-17T07:09:39.252452+0000","last_clean":"2024-09-17T07:09:39.252452+0000","last_became_active":"2024-09-17T07:09:24.407362+0000","last_became_peered":"2024-09-17T07:09:24.407362+0000","last_unstale":"2024-09-17T07:09:39.252452+0000","last_undegraded":"2024-09-17T07:09:39.252452+0000","last_fullsized":"2024-09-17T07:09:39.252452+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:27.841486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:08:27.841486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:48.091419+0000","scrub_duration":0.0037672679999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":16,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.243487+0000","last_change":"2024-09-17T07:09:24.403561+0000","last_active":"2024-09-17T07:09:39.243487+0000","last_peered":"2024-09-17T07:09:39.243487+0000","last_clean":"2024-09-17T07:09:39.243487+0000","last_became_active":"2024-09-17T07:09:24.400791+0000","last_became_peered":"2024-09-17T07:09:24.400791+0000","last_unstale":"2024-09-17T07:09:39.243487+0000","last_undegraded":"2024-09-17T07:09:39.243487+0000","last_fullsized":"2024-09-17T07:09:39.243487+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:10.805704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:08:10.805704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"75'4","reported_seq":71,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.228056+0000","last_change":"2024-09-17T07:09:24.412549+0000","last_active":"2024-09-17T07:09:39.228056+0000","last_peered":"2024-09-17T07:09:39.228056+0000","last_clean":"2024-09-17T07:09:39.228056+0000","last_became_active":"2024-09-17T07:09:24.396368+0000","last_became_peered":"2024-09-17T07:09:24.396368+0000","last_unstale":"2024-09-17T07:09:39.228056+0000","last_undegraded":"2024-09-17T07:09:39.228056+0000","last_fullsized":"2024-09-17T07:09:39.228056+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'4","last_scrub_stamp":"2024-09-17T07:09:17.769705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:09:17.769705+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:19.586607+0000","scrub_duration":0.0015094500000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":176,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.242996+0000","last_change":"2024-09-17T07:08:04.972061+0000","last_active":"2024-09-17T07:09:39.242996+0000","last_peered":"2024-09-17T07:09:39.242996+0000","last_clean":"2024-09-17T07:09:39.242996+0000","last_became_active":"2024-09-17T07:02:12.008898+0000","last_became_peered":"2024-09-17T07:02:12.008898+0000","last_unstale":"2024-09-17T07:09:39.242996+0000","last_undegraded":"2024-09-17T07:09:39.242996+0000","last_fullsized":"2024-09-17T07:09:39.242996+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:04.971954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:08:04.971954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014792150000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":173,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.254513+0000","last_change":"2024-09-17T07:09:33.804039+0000","last_active":"2024-09-17T07:09:39.254513+0000","last_peered":"2024-09-17T07:09:39.254513+0000","last_clean":"2024-09-17T07:09:39.254513+0000","last_became_active":"2024-09-17T07:02:16.014982+0000","last_became_peered":"2024-09-17T07:02:16.014982+0000","last_unstale":"2024-09-17T07:09:39.254513+0000","last_undegraded":"2024-09-17T07:09:39.254513+0000","last_fullsized":"2024-09-17T07:09:39.254513+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:03.496602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:47.444972+0000","last_clean_scrub_stamp":"2024-09-17T07:08:03.496602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001400943,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":171,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.253444+0000","last_change":"2024-09-17T07:08:26.996645+0000","last_active":"2024-09-17T07:09:39.253444+0000","last_peered":"2024-09-17T07:09:39.253444+0000","last_clean":"2024-09-17T07:09:39.253444+0000","last_became_active":"2024-09-17T07:02:18.021014+0000","last_became_peered":"2024-09-17T07:02:18.021014+0000","last_unstale":"2024-09-17T07:09:39.253444+0000","last_undegraded":"2024-09-17T07:09:39.253444+0000","last_fullsized":"2024-09-17T07:09:39.253444+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:26.996525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:08:26.996525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001722168,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"75'11","reported_seq":60,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.253347+0000","last_change":"2024-09-17T07:09:24.428977+0000","last_active":"2024-09-17T07:09:39.253347+0000","last_peered":"2024-09-17T07:09:39.253347+0000","last_clean":"2024-09-17T07:09:39.253347+0000","last_became_active":"2024-09-17T07:09:24.404942+0000","last_became_peered":"2024-09-17T07:09:24.404942+0000","last_unstale":"2024-09-17T07:09:39.253347+0000","last_undegraded":"2024-09-17T07:09:39.253347+0000","last_fullsized":"2024-09-17T07:09:39.253347+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-09-17T07:08:25.883470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:08:25.883470+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":187,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.256640+0000","last_change":"2024-09-17T07:08:14.519328+0000","last_active":"2024-09-17T07:09:39.256640+0000","last_peered":"2024-09-17T07:09:39.256640+0000","last_clean":"2024-09-17T07:09:39.256640+0000","last_became_active":"2024-09-17T07:02:12.010195+0000","last_became_peered":"2024-09-17T07:02:12.010195+0000","last_unstale":"2024-09-17T07:09:39.256640+0000","last_undegraded":"2024-09-17T07:09:39.256640+0000","last_fullsized":"2024-09-17T07:09:39.256640+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:14.519233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:08:14.519233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001420021,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":173,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.253100+0000","last_change":"2024-09-17T07:08:32.894541+0000","last_active":"2024-09-17T07:09:39.253100+0000","last_peered":"2024-09-17T07:09:39.253100+0000","last_clean":"2024-09-17T07:09:39.253100+0000","last_became_active":"2024-09-17T07:02:16.016183+0000","last_became_peered":"2024-09-17T07:02:16.016183+0000","last_unstale":"2024-09-17T07:09:39.253100+0000","last_undegraded":"2024-09-17T07:09:39.253100+0000","last_fullsized":"2024-09-17T07:09:39.253100+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:32.894433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:08:32.894433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:49.732924+0000","scrub_duration":0.001707264,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":16,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.250719+0000","last_change":"2024-09-17T07:09:24.392552+0000","last_active":"2024-09-17T07:09:39.250719+0000","last_peered":"2024-09-17T07:09:39.250719+0000","last_clean":"2024-09-17T07:09:39.250719+0000","last_became_active":"2024-09-17T07:09:24.392216+0000","last_became_peered":"2024-09-17T07:09:24.392216+0000","last_unstale":"2024-09-17T07:09:39.250719+0000","last_undegraded":"2024-09-17T07:09:39.250719+0000","last_fullsized":"2024-09-17T07:09:39.250719+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:26.832514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:07:26.832514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1c","version":"75'1","reported_seq":171,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.254589+0000","last_change":"2024-09-17T07:08:22.083583+0000","last_active":"2024-09-17T07:09:39.254589+0000","last_peered":"2024-09-17T07:09:39.254589+0000","last_clean":"2024-09-17T07:09:39.254589+0000","last_became_active":"2024-09-17T07:02:18.021855+0000","last_became_peered":"2024-09-17T07:02:18.021855+0000","last_unstale":"2024-09-17T07:09:39.254589+0000","last_undegraded":"2024-09-17T07:09:39.254589+0000","last_fullsized":"2024-09-17T07:09:39.254589+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-17T07:08:22.083502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:08:22.083502+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0018653739999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":176,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.242277+0000","last_change":"2024-09-17T07:07:22.782091+0000","last_active":"2024-09-17T07:09:39.242277+0000","last_peered":"2024-09-17T07:09:39.242277+0000","last_clean":"2024-09-17T07:09:39.242277+0000","last_became_active":"2024-09-17T07:02:12.005329+0000","last_became_peered":"2024-09-17T07:02:12.005329+0000","last_unstale":"2024-09-17T07:09:39.242277+0000","last_undegraded":"2024-09-17T07:09:39.242277+0000","last_fullsized":"2024-09-17T07:09:39.242277+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:22.781739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:07:22.781739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037808680000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"75'10","reported_seq":211,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.251523+0000","last_change":"2024-09-17T07:09:24.430545+0000","last_active":"2024-09-17T07:09:39.251523+0000","last_peered":"2024-09-17T07:09:39.251523+0000","last_clean":"2024-09-17T07:09:39.251523+0000","last_became_active":"2024-09-17T07:09:24.404495+0000","last_became_peered":"2024-09-17T07:09:24.404495+0000","last_unstale":"2024-09-17T07:09:39.251523+0000","last_undegraded":"2024-09-17T07:09:39.251523+0000","last_fullsized":"2024-09-17T07:09:39.251523+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-09-17T07:09:13.341701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:09:13.341701+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:31.527633+0000","scrub_duration":0.024611266999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"75'8","reported_seq":274,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:40.603755+0000","last_change":"2024-09-17T07:08:24.422649+0000","last_active":"2024-09-17T07:09:40.603755+0000","last_peered":"2024-09-17T07:09:40.603755+0000","last_clean":"2024-09-17T07:09:40.603755+0000","last_became_active":"2024-09-17T07:02:16.020560+0000","last_became_peered":"2024-09-17T07:02:16.020560+0000","last_unstale":"2024-09-17T07:09:40.603755+0000","last_undegraded":"2024-09-17T07:09:40.603755+0000","last_fullsized":"2024-09-17T07:09:40.603755+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:08:24.422313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:08:24.422313+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0027600989999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":169,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.240345+0000","last_change":"2024-09-17T07:08:34.803052+0000","last_active":"2024-09-17T07:09:39.240345+0000","last_peered":"2024-09-17T07:09:39.240345+0000","last_clean":"2024-09-17T07:09:39.240345+0000","last_became_active":"2024-09-17T07:02:18.021603+0000","last_became_peered":"2024-09-17T07:02:18.021603+0000","last_unstale":"2024-09-17T07:09:39.240345+0000","last_undegraded":"2024-09-17T07:09:39.240345+0000","last_fullsized":"2024-09-17T07:09:39.240345+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:34.802938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:08:34.802938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:45.165231+0000","scrub_duration":0.0022680489999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":17,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.254749+0000","last_change":"2024-09-17T07:09:24.405301+0000","last_active":"2024-09-17T07:09:39.254749+0000","last_peered":"2024-09-17T07:09:39.254749+0000","last_clean":"2024-09-17T07:09:39.254749+0000","last_became_active":"2024-09-17T07:09:24.403360+0000","last_became_peered":"2024-09-17T07:09:24.403360+0000","last_unstale":"2024-09-17T07:09:39.254749+0000","last_undegraded":"2024-09-17T07:09:39.254749+0000","last_fullsized":"2024-09-17T07:09:39.254749+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:17.704687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:08:17.704687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:45.026008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"75'11","reported_seq":236,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.256568+0000","last_change":"2024-09-17T07:09:18.263884+0000","last_active":"2024-09-17T07:09:39.256568+0000","last_peered":"2024-09-17T07:09:39.256568+0000","last_clean":"2024-09-17T07:09:39.256568+0000","last_became_active":"2024-09-17T07:03:48.760390+0000","last_became_peered":"2024-09-17T07:03:48.760390+0000","last_unstale":"2024-09-17T07:09:39.256568+0000","last_undegraded":"2024-09-17T07:09:39.256568+0000","last_fullsized":"2024-09-17T07:09:39.256568+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-09-17T07:09:18.263404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:09:18.263404+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:39.849976+0000","scrub_duration":0.0065910480000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":173,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.251484+0000","last_change":"2024-09-17T07:09:10.293301+0000","last_active":"2024-09-17T07:09:39.251484+0000","last_peered":"2024-09-17T07:09:39.251484+0000","last_clean":"2024-09-17T07:09:39.251484+0000","last_became_active":"2024-09-17T07:02:16.019701+0000","last_became_peered":"2024-09-17T07:02:16.019701+0000","last_unstale":"2024-09-17T07:09:39.251484+0000","last_undegraded":"2024-09-17T07:09:39.251484+0000","last_fullsized":"2024-09-17T07:09:39.251484+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:10.292615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:09:10.292615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:40.263740+0000","scrub_duration":0.010112249,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":160,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.279686+0000","last_change":"2024-09-17T07:07:30.779462+0000","last_active":"2024-09-17T07:09:39.279686+0000","last_peered":"2024-09-17T07:09:39.279686+0000","last_clean":"2024-09-17T07:09:39.279686+0000","last_became_active":"2024-09-17T07:02:18.363752+0000","last_became_peered":"2024-09-17T07:02:18.363752+0000","last_unstale":"2024-09-17T07:09:39.279686+0000","last_undegraded":"2024-09-17T07:09:39.279686+0000","last_fullsized":"2024-09-17T07:09:39.279686+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:30.779382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:07:30.779382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0011291059999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":207,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-17T07:09:42.338259+0000","last_change":"2024-09-17T07:09:22.215844+0000","last_active":"2024-09-17T07:09:42.338259+0000","last_peered":"2024-09-17T07:09:42.338259+0000","last_clean":"2024-09-17T07:09:42.338259+0000","last_became_active":"2024-09-17T07:03:48.759384+0000","last_became_peered":"2024-09-17T07:03:48.759384+0000","last_unstale":"2024-09-17T07:09:42.338259+0000","last_undegraded":"2024-09-17T07:09:42.338259+0000","last_fullsized":"2024-09-17T07:09:42.338259+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:22.215729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:09:22.215729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:37.706145+0000","scrub_duration":0.0051243160000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"75'15","reported_seq":211,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.253177+0000","last_change":"2024-09-17T07:09:24.427421+0000","last_active":"2024-09-17T07:09:39.253177+0000","last_peered":"2024-09-17T07:09:39.253177+0000","last_clean":"2024-09-17T07:09:39.253177+0000","last_became_active":"2024-09-17T07:09:24.403030+0000","last_became_peered":"2024-09-17T07:09:24.403030+0000","last_unstale":"2024-09-17T07:09:39.253177+0000","last_undegraded":"2024-09-17T07:09:39.253177+0000","last_fullsized":"2024-09-17T07:09:39.253177+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-09-17T07:09:15.110940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:09:15.110940+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:21.125725+0000","scrub_duration":0.017595989999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":171,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.252656+0000","last_change":"2024-09-17T07:08:25.794548+0000","last_active":"2024-09-17T07:09:39.252656+0000","last_peered":"2024-09-17T07:09:39.252656+0000","last_clean":"2024-09-17T07:09:39.252656+0000","last_became_active":"2024-09-17T07:02:16.015903+0000","last_became_peered":"2024-09-17T07:02:16.015903+0000","last_unstale":"2024-09-17T07:09:39.252656+0000","last_undegraded":"2024-09-17T07:09:39.252656+0000","last_fullsized":"2024-09-17T07:09:39.252656+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:25.794418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:25.794418+0000","last_clean_scrub_stamp":"2024-09-17T07:08:25.794418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002916175,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":17,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.256326+0000","last_change":"2024-09-17T07:09:33.804712+0000","last_active":"2024-09-17T07:09:39.256326+0000","last_peered":"2024-09-17T07:09:39.256326+0000","last_clean":"2024-09-17T07:09:39.256326+0000","last_became_active":"2024-09-17T07:09:24.402411+0000","last_became_peered":"2024-09-17T07:09:24.402411+0000","last_unstale":"2024-09-17T07:09:39.256326+0000","last_undegraded":"2024-09-17T07:09:39.256326+0000","last_fullsized":"2024-09-17T07:09:39.256326+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:14.764781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:08:14.764781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":18,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.253967+0000","last_change":"2024-09-17T07:09:28.275884+0000","last_active":"2024-09-17T07:09:39.253967+0000","last_peered":"2024-09-17T07:09:39.253967+0000","last_clean":"2024-09-17T07:09:39.253967+0000","last_became_active":"2024-09-17T07:09:24.404795+0000","last_became_peered":"2024-09-17T07:09:24.404795+0000","last_unstale":"2024-09-17T07:09:39.253967+0000","last_undegraded":"2024-09-17T07:09:39.253967+0000","last_fullsized":"2024-09-17T07:09:39.253967+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:29.761001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:07:29.761001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"86'17","reported_seq":82,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.252741+0000","last_change":"2024-09-17T07:09:24.416689+0000","last_active":"2024-09-17T07:09:39.252741+0000","last_peered":"2024-09-17T07:09:39.252741+0000","last_clean":"2024-09-17T07:09:39.252741+0000","last_became_active":"2024-09-17T07:09:24.394476+0000","last_became_peered":"2024-09-17T07:09:24.394476+0000","last_unstale":"2024-09-17T07:09:39.252741+0000","last_undegraded":"2024-09-17T07:09:39.252741+0000","last_fullsized":"2024-09-17T07:09:39.252741+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'17","last_scrub_stamp":"2024-09-17T07:09:12.729002+0000","last_deep_scrub":"86'17","last_deep_scrub_stamp":"2024-09-17T07:03:18.241067+0000","last_clean_scrub_stamp":"2024-09-17T07:09:12.729002+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:13.772930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":181,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-17T07:09:39.282972+0000","last_change":"2024-09-17T07:09:24.393651+0000","last_active":"2024-09-17T07:09:39.282972+0000","last_peered":"2024-09-17T07:09:39.282972+0000","last_clean":"2024-09-17T07:09:39.282972+0000","last_became_active":"2024-09-17T07:09:24.393329+0000","last_became_peered":"2024-09-17T07:09:24.393329+0000","last_unstale":"2024-09-17T07:09:39.282972+0000","last_undegraded":"2024-09-17T07:09:39.282972+0000","last_fullsized":"2024-09-17T07:09:39.282972+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:14.706740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:08:14.706740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00164321,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":8365211648,"num_objects":127643,"num_object_clones":0,"num_object_copies":382929,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":38885,"num_objects_unfound":0,"num_objects_dirty":127643,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":127643,"num_write_kb":8169152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8631,"num_bytes_recovered":565641216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":25717833728,"data_stored":25717833728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76096,"ondisk_log_size":76096,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69,"ondisk_log_size":69,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":692,"num_read_kb":449,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":2008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":13004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":427,"ondisk_log_size":427,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":444,"num_read_kb":439,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1386,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":12,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":420,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":4578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037856,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2873400,"kb_used_data":2553516,"kb_used_omap":1,"kb_used_meta":319870,"kb_avail":90871752,"statfs":{"total":95995035648,"available":93052674048,"internally_reserved":0,"allocated":2614800384,"data_stored":2614397853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":327547339},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:09:14 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":2.512,"15min":1.1879999999999999},"min":{"1min":0.20599999999999999,"5min":0.192,"15min":0.192},"max":{"1min":2.3039999999999998,"5min":52.478999999999999,"15min":52.478999999999999},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.95399999999999996,"5min":2.5110000000000001,"15min":1.1890000000000001},"min":{"1min":0.26200000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":2.677,"5min":52.871000000000002,"15min":52.871000000000002},"last":0.314}]},{"osd":1,"last update":"Tue Sep 17 07:09:14 2024","interfaces":[{"interface":"back","average":{"1min":5.4059999999999997,"5min":1.7170000000000001,"15min":0.91100000000000003},"min":{"1min":0.27000000000000002,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":73.549000000000007,"5min":73.549000000000007,"15min":73.549000000000007},"last":0.56100000000000005},{"interface":"front","average":{"1min":5.375,"5min":1.7430000000000001,"15min":0.93000000000000005},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":73.116,"5min":73.116,"15min":73.116},"last":0.90000000000000002}]},{"osd":2,"last update":"Tue Sep 17 07:09:14 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":1.5660000000000001,"15min":0.84399999999999997},"min":{"1min":0.26600000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":2.6379999999999999,"5min":52.055999999999997,"15min":52.055999999999997},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.61799999999999999,"5min":1.613,"15min":0.89600000000000002},"min":{"1min":0.28299999999999997,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":2.4129999999999998,"5min":52.064999999999998,"15min":52.064999999999998},"last":0.54300000000000004}]},{"osd":3,"last update":"Tue Sep 17 07:09:14 2024","interfaces":[{"interface":"back","average":{"1min":1.8919999999999999,"5min":2.1480000000000001,"15min":1.0660000000000001},"min":{"1min":0.372,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":11.167,"5min":51.689,"15min":51.689},"last":0.94099999999999995},{"interface":"front","average":{"1min":0.75600000000000001,"5min":2.3519999999999999,"15min":1.149},"min":{"1min":0.308,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":1.97,"5min":52.145000000000003,"15min":52.145000000000003},"last":0.73599999999999999}]},{"osd":4,"last update":"Tue Sep 17 07:09:14 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.50700000000000001,"15min":0.46800000000000003},"min":{"1min":0.29699999999999999,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":1.7290000000000001,"5min":1.7290000000000001,"15min":1.7290000000000001},"last":0.247},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.50700000000000001,"15min":0.45700000000000002},"min":{"1min":0.26000000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.92100000000000004,"5min":3.4089999999999998,"15min":3.4089999999999998},"last":0.81299999999999994}]},{"osd":5,"last update":"Tue Sep 17 07:09:14 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.54800000000000004,"15min":0.47199999999999998},"min":{"1min":0.27800000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.635,"5min":3.3620000000000001,"15min":3.3620000000000001},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.55800000000000005,"15min":0.496},"min":{"1min":0.307,"5min":0.28999999999999998,"15min":0.28000000000000003},"max":{"1min":1.708,"5min":1.708,"15min":1.708},"last":0.39700000000000002}]},{"osd":6,"last update":"Tue Sep 17 07:09:14 2024","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":0.63400000000000001,"15min":0.54300000000000004},"min":{"1min":0.35099999999999998,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":1.6759999999999999,"5min":1.6759999999999999,"15min":1.6759999999999999},"last":0.432},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.59299999999999997,"15min":0.52400000000000002},"min":{"1min":0.31,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.93899999999999995,"5min":3.2669999999999999,"15min":3.2669999999999999},"last":0.96899999999999997}]}]},{"osd":6,"up_from":50,"seq":214748364899,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2785340,"kb_used_data":2515508,"kb_used_omap":1,"kb_used_meta":269822,"kb_avail":90959812,"statfs":{"total":95995035648,"available":93142847488,"internally_reserved":0,"allocated":2575880192,"data_stored":2575462861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":276298183},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":2.9910000000000001,"15min":1.3280000000000001},"min":{"1min":0.27000000000000002,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":1.8109999999999999,"5min":75.323999999999998,"15min":75.323999999999998},"last":0.48799999999999999},{"interface":"front","average":{"1min":1.0029999999999999,"5min":2.9319999999999999,"15min":1.304},"min":{"1min":0.25600000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":3.556,"5min":75.168999999999997,"15min":75.168999999999997},"last":0.28599999999999998}]},{"osd":1,"last update":"Tue Sep 17 07:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.71199999999999997,"15min":0.56200000000000006},"min":{"1min":0.308,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":2.6669999999999998,"5min":4.4790000000000001,"15min":4.4790000000000001},"last":4.2409999999999997},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.81799999999999995,"15min":0.625},"min":{"1min":0.29999999999999999,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":1.849,"5min":5.9530000000000003,"15min":5.9530000000000003},"last":2.3460000000000001}]},{"osd":2,"last update":"Tue Sep 17 07:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":1.071,"15min":0.70199999999999996},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":1.8720000000000001,"5min":33.079000000000001,"15min":33.079000000000001},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.64800000000000002,"5min":1.0600000000000001,"15min":0.69899999999999995},"min":{"1min":0.254,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":1.7589999999999999,"5min":33.100000000000001,"15min":33.100000000000001},"last":0.439}]},{"osd":3,"last update":"Tue Sep 17 07:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":1.98,"15min":0.97899999999999998},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":2.1560000000000001,"5min":52.944000000000003,"15min":52.944000000000003},"last":0.53200000000000003},{"interface":"front","average":{"1min":1.1140000000000001,"5min":1.9610000000000001,"15min":1.012},"min":{"1min":0.36399999999999999,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":3.339,"5min":52.747,"15min":52.747},"last":0.59499999999999997}]},{"osd":4,"last update":"Tue Sep 17 07:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.46100000000000002,"15min":0.433},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":2.1699999999999999,"5min":2.1699999999999999,"15min":2.1699999999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.54500000000000004,"15min":0.45500000000000002},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":2.1299999999999999,"5min":2.9119999999999999,"15min":2.9119999999999999},"last":0.42599999999999999}]},{"osd":5,"last update":"Tue Sep 17 07:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.54600000000000004,"15min":0.46600000000000003},"min":{"1min":0.28299999999999997,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":4.9329999999999998,"5min":4.9329999999999998,"15min":4.9329999999999998},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.59499999999999997,"15min":0.48399999999999999},"min":{"1min":0.30499999999999999,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.859,"5min":3.0379999999999998,"15min":3.0379999999999998},"last":0.90400000000000003}]},{"osd":7,"last update":"Tue Sep 17 07:09:15 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.626,"15min":0.53800000000000003},"min":{"1min":0.29499999999999998,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":3.9399999999999999,"5min":3.9399999999999999,"15min":3.9399999999999999},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.876,"5min":0.65700000000000003,"15min":0.56799999999999995},"min":{"1min":0.30099999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":3.1930000000000001,"5min":3.1930000000000001,"15min":3.1930000000000001},"last":0.371}]}]},{"osd":5,"up_from":42,"seq":180388626536,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4397872,"kb_used_data":4146084,"kb_used_omap":1,"kb_used_meta":251774,"kb_avail":89347280,"statfs":{"total":95995035648,"available":91491614720,"internally_reserved":0,"allocated":4245590016,"data_stored":4245195373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":257817030},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:09:36 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":2.7970000000000002,"15min":1.272},"min":{"1min":0.251,"5min":0.221,"15min":0.221},"max":{"1min":4.577,"5min":49.265000000000001,"15min":49.265000000000001},"last":4.5910000000000002},{"interface":"front","average":{"1min":1.4399999999999999,"5min":2.9689999999999999,"15min":1.3220000000000001},"min":{"1min":0.39500000000000002,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":4.4260000000000002,"5min":47.350000000000001,"15min":47.350000000000001},"last":1.5449999999999999}]},{"osd":1,"last update":"Tue Sep 17 07:09:36 2024","interfaces":[{"interface":"back","average":{"1min":1.1879999999999999,"5min":0.95699999999999996,"15min":0.67500000000000004},"min":{"1min":0.23799999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":3.5600000000000001,"5min":20.329000000000001,"15min":20.329000000000001},"last":2.056},{"interface":"front","average":{"1min":1.089,"5min":1.1279999999999999,"15min":1.0720000000000001},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":4.077,"5min":20.408999999999999,"15min":20.408999999999999},"last":5.931}]},{"osd":2,"last update":"Tue Sep 17 07:09:36 2024","interfaces":[{"interface":"back","average":{"1min":1.8240000000000001,"5min":1.4079999999999999,"15min":0.83599999999999997},"min":{"1min":0.36499999999999999,"5min":0.252,"15min":0.252},"max":{"1min":6.242,"5min":20.486000000000001,"15min":20.486000000000001},"last":0.28599999999999998},{"interface":"front","average":{"1min":1.115,"5min":1.2629999999999999,"15min":0.75800000000000001},"min":{"1min":0.28899999999999998,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":3.2599999999999998,"5min":23.925999999999998,"15min":23.925999999999998},"last":0.498}]},{"osd":3,"last update":"Tue Sep 17 07:09:36 2024","interfaces":[{"interface":"back","average":{"1min":1.5209999999999999,"5min":2.3279999999999998,"15min":1.135},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":6.5439999999999996,"5min":68.978999999999999,"15min":68.978999999999999},"last":0.318},{"interface":"front","average":{"1min":1.3280000000000001,"5min":2.2130000000000001,"15min":1.0960000000000001},"min":{"1min":0.26800000000000002,"5min":0.253,"15min":0.253},"max":{"1min":4.5490000000000004,"5min":69.084000000000003,"15min":69.084000000000003},"last":0.47399999999999998}]},{"osd":4,"last update":"Tue Sep 17 07:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.53800000000000003,"15min":0.443},"min":{"1min":0.26800000000000002,"5min":0.22,"15min":0.22},"max":{"1min":4.0300000000000002,"5min":4.0300000000000002,"15min":4.0300000000000002},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.55200000000000005,"15min":0.47099999999999997},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":2.6469999999999998,"5min":2.6469999999999998,"15min":2.6469999999999998},"last":0.34200000000000003}]},{"osd":6,"last update":"Tue Sep 17 07:08:48 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.50700000000000001,"15min":0.45200000000000001},"min":{"1min":0.30499999999999999,"5min":0.23499999999999999,"15min":0.23200000000000001},"max":{"1min":1.79,"5min":1.9199999999999999,"15min":1.9199999999999999},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.57199999999999995,"15min":0.48699999999999999},"min":{"1min":0.35599999999999998,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":2.2189999999999999,"5min":2.5590000000000002,"15min":2.5590000000000002},"last":0.46200000000000002}]},{"osd":7,"last update":"Tue Sep 17 07:09:11 2024","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.64100000000000001,"15min":0.53500000000000003},"min":{"1min":0.45600000000000002,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":2.6150000000000002,"5min":4.0999999999999996,"15min":4.0999999999999996},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.57699999999999996,"15min":0.502},"min":{"1min":0.34000000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":2.27,"5min":3.9100000000000001,"15min":3.9100000000000001},"last":0.44800000000000001}]}]},{"osd":4,"up_from":33,"seq":141733920878,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2735348,"kb_used_data":2565608,"kb_used_omap":1,"kb_used_meta":169726,"kb_avail":91009804,"statfs":{"total":95995035648,"available":93194039296,"internally_reserved":0,"allocated":2627182592,"data_stored":2626782953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":173799878},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:08:58 2024","interfaces":[{"interface":"back","average":{"1min":4.5389999999999997,"5min":4.0570000000000004,"15min":1.728},"min":{"1min":0.249,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":55.310000000000002,"5min":74.551000000000002,"15min":74.551000000000002},"last":0.29299999999999998},{"interface":"front","average":{"1min":4.8250000000000002,"5min":4.343,"15min":1.8500000000000001},"min":{"1min":0.26900000000000002,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":53.771000000000001,"5min":74.778999999999996,"15min":74.778999999999996},"last":1.905}]},{"osd":1,"last update":"Tue Sep 17 07:08:58 2024","interfaces":[{"interface":"back","average":{"1min":4.9249999999999998,"5min":1.5620000000000001,"15min":0.89400000000000002},"min":{"1min":0.23599999999999999,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":54.914999999999999,"5min":54.914999999999999,"15min":54.914999999999999},"last":0.98099999999999998},{"interface":"front","average":{"1min":4.7320000000000002,"5min":1.5309999999999999,"15min":0.90000000000000002},"min":{"1min":0.26900000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":53.725000000000001,"5min":53.725000000000001,"15min":53.725000000000001},"last":0.25800000000000001}]},{"osd":2,"last update":"Tue Sep 17 07:08:58 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":1.3169999999999999,"15min":0.82199999999999995},"min":{"1min":0.28499999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":4.9530000000000003,"5min":49.651000000000003,"15min":49.651000000000003},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.60999999999999999,"5min":1.3919999999999999,"15min":0.84999999999999998},"min":{"1min":0.307,"5min":0.247,"15min":0.247},"max":{"1min":2.2429999999999999,"5min":48.813000000000002,"15min":48.813000000000002},"last":1.9359999999999999}]},{"osd":3,"last update":"Tue Sep 17 07:08:58 2024","interfaces":[{"interface":"back","average":{"1min":1.6040000000000001,"5min":1.4199999999999999,"15min":0.87},"min":{"1min":0.30599999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":17.872,"5min":39.042999999999999,"15min":39.042999999999999},"last":2.0670000000000002},{"interface":"front","average":{"1min":1.5529999999999999,"5min":1.5129999999999999,"15min":0.877},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":17.908000000000001,"5min":38.920000000000002,"15min":38.920000000000002},"last":0.96099999999999997}]},{"osd":5,"last update":"Tue Sep 17 07:09:28 2024","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.59399999999999997,"15min":0.45700000000000002},"min":{"1min":0.27100000000000002,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":3.4390000000000001,"5min":3.4390000000000001,"15min":3.4390000000000001},"last":3.008},{"interface":"front","average":{"1min":0.93999999999999995,"5min":0.61599999999999999,"15min":0.48299999999999998},"min":{"1min":0.27500000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":3.0630000000000002,"5min":3.0630000000000002,"15min":3.0630000000000002},"last":2.282}]},{"osd":6,"last update":"Tue Sep 17 07:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":1.0489999999999999,"15min":0.61399999999999999},"min":{"1min":0.29399999999999998,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.66700000000000004,"5min":47.923999999999999,"15min":47.923999999999999},"last":2.0139999999999998},{"interface":"front","average":{"1min":0.502,"5min":1.0620000000000001,"15min":0.626},"min":{"1min":0.27800000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.80200000000000005,"5min":47.600999999999999,"15min":47.600999999999999},"last":0.34699999999999998}]},{"osd":7,"last update":"Tue Sep 17 07:09:13 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":1.085,"15min":0.69499999999999995},"min":{"1min":0.31,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":2.677,"5min":48.719000000000001,"15min":48.719000000000001},"last":2.6150000000000002},{"interface":"front","average":{"1min":0.998,"5min":1.155,"15min":0.70399999999999996},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":3.7480000000000002,"5min":50.527000000000001,"15min":50.527000000000001},"last":0.93300000000000005}]}]},{"osd":3,"up_from":0,"seq":115964117104,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":18,"seq":77309411446,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3679540,"kb_used_data":3315368,"kb_used_omap":1,"kb_used_meta":364158,"kb_avail":90065612,"statfs":{"total":95995035648,"available":92227186688,"internally_reserved":0,"allocated":3394936832,"data_stored":3394538037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":372898246},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:09:31 2024","interfaces":[{"interface":"back","average":{"1min":1.569,"5min":1.25,"15min":0.64600000000000002},"min":{"1min":0.193,"5min":0.193,"15min":0.17599999999999999},"max":{"1min":5.0259999999999998,"5min":7.2709999999999999,"15min":7.2709999999999999},"last":0.76100000000000001},{"interface":"front","average":{"1min":2.137,"5min":1.768,"15min":0.82499999999999996},"min":{"1min":0.219,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":6.8689999999999998,"5min":8.6869999999999994,"15min":8.6869999999999994},"last":2.8490000000000002}]},{"osd":1,"last update":"Tue Sep 17 07:09:31 2024","interfaces":[{"interface":"back","average":{"1min":1.7709999999999999,"5min":1.272,"15min":0.65800000000000003},"min":{"1min":0.36499999999999999,"5min":0.184,"15min":0.184},"max":{"1min":6.3499999999999996,"5min":6.3499999999999996,"15min":6.3499999999999996},"last":0.47599999999999998},{"interface":"front","average":{"1min":1.768,"5min":1.175,"15min":0.64200000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":5.2759999999999998,"5min":6.9829999999999997,"15min":6.9829999999999997},"last":1.109}]},{"osd":3,"last update":"Tue Sep 17 07:08:48 2024","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.056,"15min":0.64300000000000002},"min":{"1min":0.25600000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":4.9960000000000004,"5min":5.4930000000000003,"15min":5.4930000000000003},"last":1.897},{"interface":"front","average":{"1min":1.3480000000000001,"5min":1.4810000000000001,"15min":0.77900000000000003},"min":{"1min":0.372,"5min":0.247,"15min":0.247},"max":{"1min":4.6189999999999998,"5min":6.0679999999999996,"15min":6.0679999999999996},"last":2.984}]},{"osd":4,"last update":"Tue Sep 17 07:09:08 2024","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":0.90800000000000003,"15min":0.69099999999999995},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":2.7650000000000001,"5min":5.2359999999999998,"15min":5.2359999999999998},"last":0.45500000000000002},{"interface":"front","average":{"1min":1.4870000000000001,"5min":0.96099999999999997,"15min":0.70799999999999996},"min":{"1min":0.32500000000000001,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":6.4539999999999997,"5min":6.4539999999999997,"15min":6.4539999999999997},"last":0.81899999999999995}]},{"osd":5,"last update":"Tue Sep 17 07:09:31 2024","interfaces":[{"interface":"back","average":{"1min":1.794,"5min":1.268,"15min":0.81599999999999995},"min":{"1min":0.38700000000000001,"5min":0.249,"15min":0.249},"max":{"1min":4.9619999999999997,"5min":5.383,"15min":5.383},"last":2.9279999999999999},{"interface":"front","average":{"1min":1.6950000000000001,"5min":1.0980000000000001,"15min":0.753},"min":{"1min":0.32300000000000001,"5min":0.312,"15min":0.312},"max":{"1min":7.1790000000000003,"5min":7.1790000000000003,"15min":7.1790000000000003},"last":0.39500000000000002}]},{"osd":6,"last update":"Tue Sep 17 07:08:48 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.94199999999999995,"15min":0.71399999999999997},"min":{"1min":0.39700000000000002,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.546,"5min":9.2509999999999994,"15min":9.2509999999999994},"last":101.587},{"interface":"front","average":{"1min":0.84999999999999998,"5min":1.121,"15min":0.77100000000000002},"min":{"1min":0.34799999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":2.9550000000000001,"5min":5.3330000000000002,"15min":5.3330000000000002},"last":100.696}]},{"osd":7,"last update":"Tue Sep 17 07:09:21 2024","interfaces":[{"interface":"back","average":{"1min":1.2929999999999999,"5min":1.056,"15min":0.72899999999999998},"min":{"1min":0.33300000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":6.9450000000000003,"5min":6.9450000000000003,"15min":6.9450000000000003},"last":0.433},{"interface":"front","average":{"1min":1.3700000000000001,"5min":1.147,"15min":0.88500000000000001},"min":{"1min":0.36899999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":4.1929999999999996,"5min":9.2919999999999998,"15min":9.2919999999999998},"last":0.85199999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574971,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3807896,"kb_used_data":3552012,"kb_used_omap":1,"kb_used_meta":255870,"kb_avail":89937256,"statfs":{"total":95995035648,"available":92095750144,"internally_reserved":0,"allocated":3637260288,"data_stored":3636825911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":262011334},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:08:54 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":1.4139999999999999,"15min":0.69499999999999995},"min":{"1min":0.20999999999999999,"5min":0.155,"15min":0.155},"max":{"1min":2.4710000000000001,"5min":13.837,"15min":13.837},"last":0.39100000000000001},{"interface":"front","average":{"1min":1.016,"5min":1.4399999999999999,"15min":0.72199999999999998},"min":{"1min":0.22900000000000001,"5min":0.186,"15min":0.17000000000000001},"max":{"1min":5.4119999999999999,"5min":10.342000000000001,"15min":10.342000000000001},"last":0.42699999999999999}]},{"osd":2,"last update":"Tue Sep 17 07:09:26 2024","interfaces":[{"interface":"back","average":{"1min":1.9259999999999999,"5min":1.5840000000000001,"15min":0.78900000000000003},"min":{"1min":0.20499999999999999,"5min":0.192,"15min":0.192},"max":{"1min":6.2130000000000001,"5min":11.055999999999999,"15min":11.055999999999999},"last":0.35799999999999998},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.514,"15min":0.77500000000000002},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":5.242,"5min":9.3160000000000007,"15min":9.3160000000000007},"last":0.25800000000000001}]},{"osd":3,"last update":"Tue Sep 17 07:08:49 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":1.3260000000000001,"15min":0.745},"min":{"1min":0.223,"5min":0.219,"15min":0.219},"max":{"1min":5.2800000000000002,"5min":7.766,"15min":7.766},"last":0.64200000000000002},{"interface":"front","average":{"1min":1.079,"5min":1.4970000000000001,"15min":0.78700000000000003},"min":{"1min":0.26600000000000001,"5min":0.222,"15min":0.222},"max":{"1min":7.5629999999999997,"5min":19.602,"15min":19.602},"last":0.89000000000000001}]},{"osd":4,"last update":"Tue Sep 17 07:09:09 2024","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":1.1399999999999999,"15min":0.76300000000000001},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":5.3040000000000003,"5min":7.8840000000000003,"15min":7.8840000000000003},"last":0.373},{"interface":"front","average":{"1min":2.9009999999999998,"5min":1.478,"15min":0.872},"min":{"1min":0.42299999999999999,"5min":0.307,"15min":0.307},"max":{"1min":27.097999999999999,"5min":27.097999999999999,"15min":27.097999999999999},"last":0.58399999999999996}]},{"osd":5,"last update":"Tue Sep 17 07:09:26 2024","interfaces":[{"interface":"back","average":{"1min":1.246,"5min":1.177,"15min":0.78900000000000003},"min":{"1min":0.33400000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":5.3470000000000004,"5min":5.617,"15min":6.0060000000000002},"last":0.34000000000000002},{"interface":"front","average":{"1min":2.141,"5min":1.4179999999999999,"15min":0.84599999999999997},"min":{"1min":0.376,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":8.1920000000000002,"5min":8.8889999999999993,"15min":8.8889999999999993},"last":0.66000000000000003}]},{"osd":6,"last update":"Tue Sep 17 07:08:49 2024","interfaces":[{"interface":"back","average":{"1min":1.3720000000000001,"5min":1.2809999999999999,"15min":0.78900000000000003},"min":{"1min":0.443,"5min":0.36599999999999999,"15min":0.36499999999999999},"max":{"1min":8.1219999999999999,"5min":8.843,"15min":8.843},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.85599999999999998,"5min":1.1279999999999999,"15min":0.77100000000000002},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":5.3250000000000002,"5min":7.1130000000000004,"15min":7.1130000000000004},"last":0.40300000000000002}]},{"osd":7,"last update":"Tue Sep 17 07:09:12 2024","interfaces":[{"interface":"back","average":{"1min":2.2189999999999999,"5min":1.4159999999999999,"15min":0.84599999999999997},"min":{"1min":0.47399999999999998,"5min":0.37,"15min":0.37},"max":{"1min":8.1660000000000004,"5min":8.9309999999999992,"15min":8.9309999999999992},"last":0.76000000000000001},{"interface":"front","average":{"1min":1.325,"5min":1.2030000000000001,"15min":0.77900000000000003},"min":{"1min":0.34100000000000003,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":4.6369999999999996,"5min":11.007999999999999,"15min":11.007999999999999},"last":0.61499999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738494,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3572412,"kb_used_data":3216432,"kb_used_omap":1,"kb_used_meta":355966,"kb_avail":90172740,"statfs":{"total":95995035648,"available":92336885760,"internally_reserved":0,"allocated":3293626368,"data_stored":3293219905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":364509638},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 17 07:08:57 2024","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":1.2,"15min":0.624},"min":{"1min":0.19900000000000001,"5min":0.183,"15min":0.183},"max":{"1min":2.8879999999999999,"5min":8.0259999999999998,"15min":8.0259999999999998},"last":0.80300000000000005},{"interface":"front","average":{"1min":1.1799999999999999,"5min":1.353,"15min":0.68500000000000005},"min":{"1min":0.17799999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":2.8410000000000002,"5min":7.8849999999999998,"15min":7.8849999999999998},"last":0.96499999999999997}]},{"osd":2,"last update":"Tue Sep 17 07:09:24 2024","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":2.0459999999999998,"15min":0.93400000000000005},"min":{"1min":0.255,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":3.0939999999999999,"5min":18.643999999999998,"15min":18.643999999999998},"last":4.7089999999999996},{"interface":"front","average":{"1min":1.0009999999999999,"5min":1.8160000000000001,"15min":0.84799999999999998},"min":{"1min":0.27600000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":5.8440000000000003,"5min":17.972999999999999,"15min":17.972999999999999},"last":0.91300000000000003}]},{"osd":3,"last update":"Tue Sep 17 07:08:57 2024","interfaces":[{"interface":"back","average":{"1min":1.0189999999999999,"5min":1.7410000000000001,"15min":0.86899999999999999},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":3.7599999999999998,"5min":18.629000000000001,"15min":18.629000000000001},"last":1.1479999999999999},{"interface":"front","average":{"1min":0.71599999999999997,"5min":1.8460000000000001,"15min":0.90200000000000002},"min":{"1min":0.24299999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.9039999999999999,"5min":22.358000000000001,"15min":22.358000000000001},"last":1.1359999999999999}]},{"osd":4,"last update":"Tue Sep 17 07:09:06 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":1.4239999999999999,"15min":0.88800000000000001},"min":{"1min":0.317,"5min":0.316,"15min":0.316},"max":{"1min":2.1520000000000001,"5min":18.568000000000001,"15min":18.568000000000001},"last":0.93700000000000006},{"interface":"front","average":{"1min":0.88400000000000001,"5min":1.3700000000000001,"15min":0.82299999999999995},"min":{"1min":0.33300000000000002,"5min":0.255,"15min":0.255},"max":{"1min":1.514,"5min":18.765000000000001,"15min":18.765000000000001},"last":1.042}]},{"osd":5,"last update":"Tue Sep 17 07:09:36 2024","interfaces":[{"interface":"back","average":{"1min":1.1459999999999999,"5min":1.5600000000000001,"15min":0.88500000000000001},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":5.2229999999999999,"5min":18.614999999999998,"15min":18.614999999999998},"last":1.083},{"interface":"front","average":{"1min":1.0640000000000001,"5min":1.5329999999999999,"15min":0.871},"min":{"1min":0.30199999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":2.923,"5min":18.791,"15min":18.791},"last":1.0049999999999999}]},{"osd":6,"last update":"Tue Sep 17 07:08:57 2024","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":1.3879999999999999,"15min":0.82799999999999996},"min":{"1min":0.33100000000000002,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":2.6629999999999998,"5min":18.602,"15min":18.602},"last":1.177},{"interface":"front","average":{"1min":0.65000000000000002,"5min":1.5109999999999999,"15min":0.871},"min":{"1min":0.379,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.978,"5min":18.692,"15min":18.692},"last":1.113}]},{"osd":7,"last update":"Tue Sep 17 07:09:16 2024","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":1.9390000000000001,"15min":1.409},"min":{"1min":0.32900000000000001,"5min":0.311,"15min":0.311},"max":{"1min":2.8500000000000001,"5min":46.183,"15min":46.183},"last":1.069},{"interface":"front","average":{"1min":1.008,"5min":1.859,"15min":0.99299999999999999},"min":{"1min":0.33600000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":2.7370000000000001,"5min":45.914999999999999,"15min":45.914999999999999},"last":1.2030000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":2448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3319529472,"data_stored":3319529472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3646488576,"data_stored":3646488576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3441098752,"data_stored":3441098752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3166568448,"data_stored":3166568448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2637430784,"data_stored":2637430784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4282908672,"data_stored":4282908672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2595749888,"data_stored":2595749888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2628059136,"data_stored":2628059136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T07:09:46.633 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj 6, currently 16 2024-09-17T07:09:46.634 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:09:46.643 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:09:46.677 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:46.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:46.678 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:09:46.778 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:09:46.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:46.821 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:46.821 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:09:46.921 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:09:46.962 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:46.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:46.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:09:47.063 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:09:47.095 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:47.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:47.096 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:09:47.097 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:09:47.196 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:09:47.228 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:47.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:47.230 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:09:47.330 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:09:47.366 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:47 smithi100 ceph-mon[29092]: from='client.24862 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:09:47.366 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:47 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:09:47.366 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:47 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:09:47.366 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:47 smithi100 ceph-mon[29092]: osdmap e200: 8 total, 8 up, 7 in 2024-09-17T07:09:47.366 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:47 smithi100 ceph-mon[39462]: from='client.24862 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:09:47.366 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:47 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:09:47.366 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:47 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:09:47.367 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:47 smithi100 ceph-mon[39462]: osdmap e200: 8 total, 8 up, 7 in 2024-09-17T07:09:47.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:47.368 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:47.368 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:09:47.469 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:09:47.499 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:47.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:47.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:09:47.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:47 smithi135 ceph-mon[33238]: from='client.24862 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:09:47.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:47 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:09:47.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:47 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:09:47.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:47 smithi135 ceph-mon[33238]: osdmap e200: 8 total, 8 up, 7 in 2024-09-17T07:09:47.601 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:09:47.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:47.633 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:47.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:09:47.734 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:09:47.762 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:47.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:47.763 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:09:47.864 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:09:47.901 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:47.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:47.902 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:09:48.003 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:09:48.038 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:48.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:48.039 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:09:48.142 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:09:48.172 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:48.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:48.174 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:09:48.274 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:09:48.309 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:48.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:48.310 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:09:48.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:48 smithi100 ceph-mon[29092]: pgmap v335: 148 pgs: 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 142 active+clean; 7.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 645 B/s rd, 39 MiB/s wr, 626 op/s; 38885/383517 objects misplaced (10.139%); 20 MiB/s, 318 objects/s recovering 2024-09-17T07:09:48.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:48 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:09:48.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:48 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:09:48.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:48 smithi100 ceph-mon[39462]: pgmap v335: 148 pgs: 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 142 active+clean; 7.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 645 B/s rd, 39 MiB/s wr, 626 op/s; 38885/383517 objects misplaced (10.139%); 20 MiB/s, 318 objects/s recovering 2024-09-17T07:09:48.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:48 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:09:48.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:48 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:09:48.412 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:09:48.444 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:48 smithi135 ceph-mon[33238]: pgmap v335: 148 pgs: 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 142 active+clean; 7.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 645 B/s rd, 39 MiB/s wr, 626 op/s; 38885/383517 objects misplaced (10.139%); 20 MiB/s, 318 objects/s recovering 2024-09-17T07:09:48.444 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:48 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:09:48.444 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:48 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:09:48.450 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:48.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:48.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:09:48.551 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:09:48.592 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:48.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:48.593 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:09:48.694 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:09:48.734 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:48.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:48.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:09:48.793 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:09:48 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:09:48.836 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:09:48.868 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:48.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:48.869 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:09:48.970 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:09:49.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:49.015 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:49.016 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:09:49.117 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:09:49.118 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:09:49.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:49.146 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:49.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:09:49.247 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:09:49.274 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:49.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:49.275 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:09:49.376 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:09:49.427 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:49.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:49.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:09:49.530 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:09:49.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:49.575 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:49.575 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:09:49.676 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:09:49.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:49.717 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:49.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:09:49.818 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:09:49.860 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:49.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:49.861 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:09:49.962 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:09:50.006 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:50.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:50.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:09:50.108 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:09:50.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:50.152 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:50.152 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:09:50.253 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:09:50.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:50.293 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:50.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:09:50.394 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:09:50.424 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:50.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:50.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:09:50.526 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:09:50.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:50 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:09:50.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:50 smithi135 ceph-mon[33238]: pgmap v336: 148 pgs: 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 142 active+clean; 7.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.4 KiB/s rd, 61 MiB/s wr, 1.33k op/s; 36482/383730 objects misplaced (9.507%); 39 MiB/s, 622 objects/s recovering 2024-09-17T07:09:50.593 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:50 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:09:50.593 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:50 smithi100 ceph-mon[29092]: pgmap v336: 148 pgs: 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 142 active+clean; 7.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.4 KiB/s rd, 61 MiB/s wr, 1.33k op/s; 36482/383730 objects misplaced (9.507%); 39 MiB/s, 622 objects/s recovering 2024-09-17T07:09:50.594 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:50 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:09:50.594 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:50 smithi100 ceph-mon[39462]: pgmap v336: 148 pgs: 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 142 active+clean; 7.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.4 KiB/s rd, 61 MiB/s wr, 1.33k op/s; 36482/383730 objects misplaced (9.507%); 39 MiB/s, 622 objects/s recovering 2024-09-17T07:09:50.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:50.595 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:50.595 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:09:50.696 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:09:50.737 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:50.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:50.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:09:50.839 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:09:50.875 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:50.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:50.876 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:09:50.977 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:09:51.013 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:51.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:51.014 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:09:51.087 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:09:51.115 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:09:51.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:51.173 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:51.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:09:51.201 INFO:teuthology.orchestra.run.smithi100.stderr:nodeep-scrub is set 2024-09-17T07:09:51.274 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:09:51.284 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:51 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2890045223' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:09:51.284 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:51 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2890045223' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:09:51.311 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:51.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:51.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:09:51.413 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:09:51.458 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:51.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:51.459 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:09:51.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:51 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2890045223' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:09:51.559 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:09:51.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:51.598 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:51.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:09:51.633 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7] out_osds: [3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-17T07:09:51.634 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T07:09:51.634 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-09-17T07:09:51.634 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T07:09:51.699 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:09:51.725 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:51.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:51.726 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:09:51.827 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:09:51.878 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:51.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:51.880 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:09:51.980 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:09:52.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:52.026 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:52.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:09:52.127 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:09:52.177 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:52.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:52.178 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:09:52.279 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:09:52.313 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:52.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:52.314 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:09:52.415 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:09:52.447 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:52.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:52.448 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:09:52.487 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:52 smithi100 ceph-mon[29092]: pgmap v337: 148 pgs: 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 142 active+clean; 7.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 55 MiB/s wr, 1.19k op/s; 36482/383730 objects misplaced (9.507%); 35 MiB/s, 557 objects/s recovering 2024-09-17T07:09:52.487 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:52 smithi100 ceph-mon[29092]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:09:52.487 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:52 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2890045223' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T07:09:52.487 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:52 smithi100 ceph-mon[29092]: osdmap e201: 8 total, 8 up, 7 in 2024-09-17T07:09:52.487 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:52 smithi100 ceph-mon[39462]: pgmap v337: 148 pgs: 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 142 active+clean; 7.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 55 MiB/s wr, 1.19k op/s; 36482/383730 objects misplaced (9.507%); 35 MiB/s, 557 objects/s recovering 2024-09-17T07:09:52.488 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:52 smithi100 ceph-mon[39462]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:09:52.488 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:52 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2890045223' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T07:09:52.488 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:52 smithi100 ceph-mon[39462]: osdmap e201: 8 total, 8 up, 7 in 2024-09-17T07:09:52.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:52 smithi135 ceph-mon[33238]: pgmap v337: 148 pgs: 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 142 active+clean; 7.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 55 MiB/s wr, 1.19k op/s; 36482/383730 objects misplaced (9.507%); 35 MiB/s, 557 objects/s recovering 2024-09-17T07:09:52.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:52 smithi135 ceph-mon[33238]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:09:52.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:52 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2890045223' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T07:09:52.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:52 smithi135 ceph-mon[33238]: osdmap e201: 8 total, 8 up, 7 in 2024-09-17T07:09:52.549 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:09:52.586 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:52.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:52.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:09:52.688 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:09:52.719 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:52.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:52.720 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:09:52.802 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-17T07:09:52.802 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-17T07:09:52.802 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-17T07:09:52.802 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-17T07:09:52.802 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-17T07:09:52.821 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:09:52.866 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:52.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:52.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:09:52.968 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:09:53.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:53.013 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:53.013 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:09:53.114 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:09:53.152 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:53.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:53.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:09:53.255 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:09:53.292 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:53.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:53.293 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:09:53.394 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:09:53.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:53.432 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:53.433 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:09:53.533 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:09:53.571 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:53.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:53.572 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:09:53.673 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:09:53.710 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:53.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:53.711 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:09:53.812 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:09:53.844 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:53.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:53.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:09:53.922 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-17T07:09:53.946 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:09:53.966 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:09:53.976 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:53.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:53.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:09:54.078 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:09:54.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:54.118 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:54.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:09:54.219 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:09:54.257 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:54.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:54.259 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:09:54.359 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:09:54.398 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:54.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:54.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:09:54.501 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:09:54.526 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T07:09:54.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:54 smithi135 ceph-mon[33238]: pgmap v339: 148 pgs: 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 142 active+clean; 7.7 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 767 B/s rd, 22 MiB/s wr, 976 op/s; 34426/377250 objects misplaced (9.126%); 29 MiB/s, 467 objects/s recovering 2024-09-17T07:09:54.544 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:54 smithi100 ceph-mon[39462]: pgmap v339: 148 pgs: 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 142 active+clean; 7.7 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 767 B/s rd, 22 MiB/s wr, 976 op/s; 34426/377250 objects misplaced (9.126%); 29 MiB/s, 467 objects/s recovering 2024-09-17T07:09:54.544 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:54 smithi100 ceph-mon[29092]: pgmap v339: 148 pgs: 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 142 active+clean; 7.7 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 767 B/s rd, 22 MiB/s wr, 976 op/s; 34426/377250 objects misplaced (9.126%); 29 MiB/s, 467 objects/s recovering 2024-09-17T07:09:54.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:54.559 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:54.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:09:54.660 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:09:54.694 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:54.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:54.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:09:54.796 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:09:54.853 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:54.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:54.854 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:09:54.955 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:09:54.996 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:54.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:54.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:09:55.098 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:09:55.136 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:55.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:55.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:09:55.238 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:09:55.272 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:55.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:55.273 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:09:55.373 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:09:55.411 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:55.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:55.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:09:55.513 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:09:55.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:55 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:09:55.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:55 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:09:55.558 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:55.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:55.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:09:55.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:55 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:09:55.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:55 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:09:55.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:55 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:09:55.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:55 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:09:55.661 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:09:55.697 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:55.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:55.698 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:09:55.799 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:09:55.836 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:55.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:55.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:09:55.877 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:09:55.878 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:09:55.915 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1: {} 2024-09-17T07:09:55.916 INFO:teuthology.orchestra.run.smithi100.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:09:55.938 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:09:55.959 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:09:55 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:09:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:09:55.960 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:09:55.960 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:09:55.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:55.985 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:55.986 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:09:55.998 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:09:55.998 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:09:56.035 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:09:56.036 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:09:56.077 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:09:56.077 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:09:56.086 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:09:56.111 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:09:56.111 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:09:56.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:56.125 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:56.125 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:09:56.151 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:09:56.152 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:09:56.226 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:09:56.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:56.263 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:56.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:09:56.364 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:09:56.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:56 smithi100 ceph-mon[29092]: pgmap v340: 148 pgs: 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 142 active+clean; 7.5 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 20 MiB/s wr, 1.25k op/s; 32690/367065 objects misplaced (8.906%); 33 MiB/s, 527 objects/s recovering 2024-09-17T07:09:56.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:56 smithi100 ceph-mon[29092]: osdmap e202: 8 total, 8 up, 7 in 2024-09-17T07:09:56.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:56 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/3850786753' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:09:56.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:56 smithi100 ceph-mon[39462]: pgmap v340: 148 pgs: 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 142 active+clean; 7.5 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 20 MiB/s wr, 1.25k op/s; 32690/367065 objects misplaced (8.906%); 33 MiB/s, 527 objects/s recovering 2024-09-17T07:09:56.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:56 smithi100 ceph-mon[39462]: osdmap e202: 8 total, 8 up, 7 in 2024-09-17T07:09:56.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:56 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/3850786753' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:09:56.399 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:56.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:56.400 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:09:56.501 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:09:56.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:56.539 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:56.539 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:09:56.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:56 smithi135 ceph-mon[33238]: pgmap v340: 148 pgs: 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 142 active+clean; 7.5 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 20 MiB/s wr, 1.25k op/s; 32690/367065 objects misplaced (8.906%); 33 MiB/s, 527 objects/s recovering 2024-09-17T07:09:56.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:56 smithi135 ceph-mon[33238]: osdmap e202: 8 total, 8 up, 7 in 2024-09-17T07:09:56.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:56 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/3850786753' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:09:56.641 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:09:56.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:56.685 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:56.685 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:09:56.786 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:09:56.816 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:56.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:56.817 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:09:56.918 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:09:56.952 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:56.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:56.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:09:57.054 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:09:57.096 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:57.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:57.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:09:57.198 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:09:57.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:57.232 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:57.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:09:57.333 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:09:57.370 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:57.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:57.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:09:57.472 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:09:57.520 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:57.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:57.521 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:09:57.623 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:09:57.664 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:57.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:57.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:09:57.766 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:09:57.786 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:09:57.824 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:57.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:57.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:09:57.926 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:09:57.961 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:57.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:57.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:09:58.057 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:09:58.063 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:09:58.119 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:58.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:58.120 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:09:58.221 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:09:58.254 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:58.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:58.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:09:58.356 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:09:58.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:58 smithi100 ceph-mon[29092]: pgmap v342: 148 pgs: 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 142 active+clean; 7.5 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 694 op/s; 32690/367065 objects misplaced (8.906%); 18 MiB/s, 286 objects/s recovering 2024-09-17T07:09:58.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:58 smithi100 ceph-mon[39462]: pgmap v342: 148 pgs: 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 142 active+clean; 7.5 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 694 op/s; 32690/367065 objects misplaced (8.906%); 18 MiB/s, 286 objects/s recovering 2024-09-17T07:09:58.394 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:58.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:58.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:09:58.445 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T07:09:58.448 INFO:teuthology.orchestra.run.smithi100.stderr:dumped all 2024-09-17T07:09:58.454 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:58 smithi135 ceph-mon[33238]: pgmap v342: 148 pgs: 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 142 active+clean; 7.5 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 694 op/s; 32690/367065 objects misplaced (8.906%); 18 MiB/s, 286 objects/s recovering 2024-09-17T07:09:58.496 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:09:58.536 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:58.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:58.537 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:09:58.638 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:09:58.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:58.673 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:58.673 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:09:58.774 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:09:58.793 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:09:58 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:09:58.810 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:58.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:58.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:09:58.912 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:09:58.951 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:58.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:58.952 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:09:59.053 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:09:59.119 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:59.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:59.120 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:09:59.220 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:09:59.255 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:59.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:59.256 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:09:59.357 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:09:59.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:09:59 smithi100 ceph-mon[29092]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-17T07:09:59.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:09:59 smithi100 ceph-mon[39462]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-17T07:09:59.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:59.398 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:59.398 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:09:59.499 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:09:59.530 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:09:59.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:59.535 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:59.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:09:59.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:09:59 smithi135 ceph-mon[33238]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-17T07:09:59.636 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:09:59.675 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:59.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:59.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:09:59.778 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:09:59.835 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:09:59.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:59.836 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:09:59.937 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:09:59.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:09:59.974 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:09:59.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:10:00.075 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:10:00.118 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:00.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:00.120 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:10:00.220 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:10:00.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:00.247 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:00.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:10:00.349 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:10:00.384 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:00.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:00.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:10:00.486 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:10:00.518 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:00.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:00.519 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:10:00.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:00 smithi135 ceph-mon[33238]: from='client.24877 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:10:00.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:00 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:10:00.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:00 smithi135 ceph-mon[33238]: pgmap v343: 148 pgs: 5 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 142 active+clean; 7.1 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.42k op/s; 28335/349635 objects misplaced (8.104%); 40 MiB/s, 641 objects/s recovering 2024-09-17T07:10:00.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:00 smithi135 ceph-mon[33238]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set; Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull 2024-09-17T07:10:00.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:00 smithi135 ceph-mon[33238]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-09-17T07:10:00.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:00 smithi135 ceph-mon[33238]: [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull 2024-09-17T07:10:00.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:00 smithi135 ceph-mon[33238]: pg 8.f is active+remapped+backfill_toofull, acting [1,2,3] 2024-09-17T07:10:00.619 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:10:00.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:00 smithi100 ceph-mon[39462]: from='client.24877 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:10:00.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:00 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:10:00.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:00 smithi100 ceph-mon[39462]: pgmap v343: 148 pgs: 5 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 142 active+clean; 7.1 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.42k op/s; 28335/349635 objects misplaced (8.104%); 40 MiB/s, 641 objects/s recovering 2024-09-17T07:10:00.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:00 smithi100 ceph-mon[39462]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set; Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull 2024-09-17T07:10:00.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:00 smithi100 ceph-mon[39462]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-09-17T07:10:00.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:00 smithi100 ceph-mon[39462]: [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull 2024-09-17T07:10:00.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:00 smithi100 ceph-mon[39462]: pg 8.f is active+remapped+backfill_toofull, acting [1,2,3] 2024-09-17T07:10:00.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:00 smithi100 ceph-mon[29092]: from='client.24877 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:10:00.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:00 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:10:00.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:00 smithi100 ceph-mon[29092]: pgmap v343: 148 pgs: 5 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 142 active+clean; 7.1 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.42k op/s; 28335/349635 objects misplaced (8.104%); 40 MiB/s, 641 objects/s recovering 2024-09-17T07:10:00.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:00 smithi100 ceph-mon[29092]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set; Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull 2024-09-17T07:10:00.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:00 smithi100 ceph-mon[29092]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-09-17T07:10:00.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:00 smithi100 ceph-mon[29092]: [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull 2024-09-17T07:10:00.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:00 smithi100 ceph-mon[29092]: pg 8.f is active+remapped+backfill_toofull, acting [1,2,3] 2024-09-17T07:10:00.652 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:00.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:00.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:10:00.754 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:10:00.790 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:00.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:00.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:10:00.892 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:10:00.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:00.933 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:00.933 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:10:01.034 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:10:01.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:01.078 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:01.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:10:01.179 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:10:01.211 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:01.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:01.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:10:01.263 INFO:teuthology.orchestra.run.smithi100.stdout:{"pg_ready":true,"pg_map":{"version":342,"stamp":"2024-09-17T07:09:57.159794+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8006342958,"num_objects":122355,"num_object_clones":0,"num_object_copies":367065,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":32690,"num_objects_unfound":0,"num_objects_dirty":122355,"num_whiteouts":0,"num_read":1371,"num_read_kb":1217,"num_write":139454,"num_write_kb":8354524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13489,"num_bytes_recovered":878514172,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76638,"ondisk_log_size":76638,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":23591300,"kb_used_data":21497864,"kb_used_omap":10,"kb_used_meta":2093045,"kb_avail":632624764,"statfs":{"total":671965249536,"available":647807758336,"internally_reserved":0,"allocated":22013812736,"data_stored":22011242551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11152,"internal_metadata":2143278192},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[9,4,9,11,7,6],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-364052480,"num_objects":-5555,"num_object_clones":0,"num_object_copies":-16665,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-3792,"num_objects_unfound":0,"num_objects_dirty":-5555,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":5555,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2291,"num_bytes_recovered":150142976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.005179"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":19,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.185715+0000","last_change":"2024-09-17T07:09:33.811862+0000","last_active":"2024-09-17T07:09:46.185715+0000","last_peered":"2024-09-17T07:09:46.185715+0000","last_clean":"2024-09-17T07:09:46.185715+0000","last_became_active":"2024-09-17T07:09:24.401344+0000","last_became_peered":"2024-09-17T07:09:24.401344+0000","last_unstale":"2024-09-17T07:09:46.185715+0000","last_undegraded":"2024-09-17T07:09:46.185715+0000","last_fullsized":"2024-09-17T07:09:46.185715+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:33.644728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:33.644728+0000","last_clean_scrub_stamp":"2024-09-17T07:07:33.644728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":179,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.182150+0000","last_change":"2024-09-17T07:08:17.332759+0000","last_active":"2024-09-17T07:09:46.182150+0000","last_peered":"2024-09-17T07:09:46.182150+0000","last_clean":"2024-09-17T07:09:46.182150+0000","last_became_active":"2024-09-17T07:02:12.006754+0000","last_became_peered":"2024-09-17T07:02:12.006754+0000","last_unstale":"2024-09-17T07:09:46.182150+0000","last_undegraded":"2024-09-17T07:09:46.182150+0000","last_fullsized":"2024-09-17T07:09:46.182150+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:17.332631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:08:17.332631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023942220000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"75'15","reported_seq":72,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.181617+0000","last_change":"2024-09-17T07:09:24.419732+0000","last_active":"2024-09-17T07:09:46.181617+0000","last_peered":"2024-09-17T07:09:46.181617+0000","last_clean":"2024-09-17T07:09:46.181617+0000","last_became_active":"2024-09-17T07:09:24.402042+0000","last_became_peered":"2024-09-17T07:09:24.402042+0000","last_unstale":"2024-09-17T07:09:46.181617+0000","last_undegraded":"2024-09-17T07:09:46.181617+0000","last_fullsized":"2024-09-17T07:09:46.181617+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-09-17T07:08:28.845108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:08:28.845108+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:53.323983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":164,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.166044+0000","last_change":"2024-09-17T07:08:21.564706+0000","last_active":"2024-09-17T07:09:46.166044+0000","last_peered":"2024-09-17T07:09:46.166044+0000","last_clean":"2024-09-17T07:09:46.166044+0000","last_became_active":"2024-09-17T07:02:16.014928+0000","last_became_peered":"2024-09-17T07:02:16.014928+0000","last_unstale":"2024-09-17T07:09:46.166044+0000","last_undegraded":"2024-09-17T07:09:46.166044+0000","last_fullsized":"2024-09-17T07:09:46.166044+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:21.564567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:08:21.564567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0039106849999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":19,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.183184+0000","last_change":"2024-09-17T07:09:24.408521+0000","last_active":"2024-09-17T07:09:46.183184+0000","last_peered":"2024-09-17T07:09:46.183184+0000","last_clean":"2024-09-17T07:09:46.183184+0000","last_became_active":"2024-09-17T07:09:24.402568+0000","last_became_peered":"2024-09-17T07:09:24.402568+0000","last_unstale":"2024-09-17T07:09:46.183184+0000","last_undegraded":"2024-09-17T07:09:46.183184+0000","last_fullsized":"2024-09-17T07:09:46.183184+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:15.767495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:07:15.767495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"75'9","reported_seq":193,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.166125+0000","last_change":"2024-09-17T07:08:22.546769+0000","last_active":"2024-09-17T07:09:46.166125+0000","last_peered":"2024-09-17T07:09:46.166125+0000","last_clean":"2024-09-17T07:09:46.166125+0000","last_became_active":"2024-09-17T07:02:14.016244+0000","last_became_peered":"2024-09-17T07:02:14.016244+0000","last_unstale":"2024-09-17T07:09:46.166125+0000","last_undegraded":"2024-09-17T07:09:46.166125+0000","last_fullsized":"2024-09-17T07:09:46.166125+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:08:22.546632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:08:22.546632+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0041894990000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":175,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.170372+0000","last_change":"2024-09-17T07:09:25.123274+0000","last_active":"2024-09-17T07:09:46.170372+0000","last_peered":"2024-09-17T07:09:46.170372+0000","last_clean":"2024-09-17T07:09:46.170372+0000","last_became_active":"2024-09-17T07:02:16.014380+0000","last_became_peered":"2024-09-17T07:02:16.014380+0000","last_unstale":"2024-09-17T07:09:46.170372+0000","last_undegraded":"2024-09-17T07:09:46.170372+0000","last_fullsized":"2024-09-17T07:09:46.170372+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:25.123158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:09:25.123158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:49.651515+0000","scrub_duration":0.00166512,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":163,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:48.092971+0000","last_change":"2024-09-17T07:07:21.791530+0000","last_active":"2024-09-17T07:09:48.092971+0000","last_peered":"2024-09-17T07:09:48.092971+0000","last_clean":"2024-09-17T07:09:48.092971+0000","last_became_active":"2024-09-17T07:02:18.020790+0000","last_became_peered":"2024-09-17T07:02:18.020790+0000","last_unstale":"2024-09-17T07:09:48.092971+0000","last_undegraded":"2024-09-17T07:09:48.092971+0000","last_fullsized":"2024-09-17T07:09:48.092971+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:21.791399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:07:21.791399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0069986889999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":179,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:48.280159+0000","last_change":"2024-09-17T07:09:09.812811+0000","last_active":"2024-09-17T07:09:48.280159+0000","last_peered":"2024-09-17T07:09:48.280159+0000","last_clean":"2024-09-17T07:09:48.280159+0000","last_became_active":"2024-09-17T07:02:12.011129+0000","last_became_peered":"2024-09-17T07:02:12.011129+0000","last_unstale":"2024-09-17T07:09:48.280159+0000","last_undegraded":"2024-09-17T07:09:48.280159+0000","last_fullsized":"2024-09-17T07:09:48.280159+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:09.812692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:43.060102+0000","last_clean_scrub_stamp":"2024-09-17T07:09:09.812692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:22.110015+0000","scrub_duration":0.0062170050000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"75'5","reported_seq":192,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:48.093005+0000","last_change":"2024-09-17T07:09:24.415312+0000","last_active":"2024-09-17T07:09:48.093005+0000","last_peered":"2024-09-17T07:09:48.093005+0000","last_clean":"2024-09-17T07:09:48.093005+0000","last_became_active":"2024-09-17T07:09:24.396900+0000","last_became_peered":"2024-09-17T07:09:24.396900+0000","last_unstale":"2024-09-17T07:09:48.093005+0000","last_undegraded":"2024-09-17T07:09:48.093005+0000","last_fullsized":"2024-09-17T07:09:48.093005+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-09-17T07:07:15.848071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:07:15.848071+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010596959,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":175,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.185473+0000","last_change":"2024-09-17T07:08:19.081985+0000","last_active":"2024-09-17T07:09:46.185473+0000","last_peered":"2024-09-17T07:09:46.185473+0000","last_clean":"2024-09-17T07:09:46.185473+0000","last_became_active":"2024-09-17T07:02:16.015401+0000","last_became_peered":"2024-09-17T07:02:16.015401+0000","last_unstale":"2024-09-17T07:09:46.185473+0000","last_undegraded":"2024-09-17T07:09:46.185473+0000","last_fullsized":"2024-09-17T07:09:46.185473+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:19.081861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:08:19.081861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002870844,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":181,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.171672+0000","last_change":"2024-09-17T07:09:24.404913+0000","last_active":"2024-09-17T07:09:46.171672+0000","last_peered":"2024-09-17T07:09:46.171672+0000","last_clean":"2024-09-17T07:09:46.171672+0000","last_became_active":"2024-09-17T07:09:24.402616+0000","last_became_peered":"2024-09-17T07:09:24.402616+0000","last_unstale":"2024-09-17T07:09:46.171672+0000","last_undegraded":"2024-09-17T07:09:46.171672+0000","last_fullsized":"2024-09-17T07:09:46.171672+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:23.739157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:07:23.739157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004894603,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":179,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:48.280120+0000","last_change":"2024-09-17T07:09:08.807223+0000","last_active":"2024-09-17T07:09:48.280120+0000","last_peered":"2024-09-17T07:09:48.280120+0000","last_clean":"2024-09-17T07:09:48.280120+0000","last_became_active":"2024-09-17T07:02:12.012234+0000","last_became_peered":"2024-09-17T07:02:12.012234+0000","last_unstale":"2024-09-17T07:09:48.280120+0000","last_undegraded":"2024-09-17T07:09:48.280120+0000","last_fullsized":"2024-09-17T07:09:48.280120+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:08.807058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:45.029247+0000","last_clean_scrub_stamp":"2024-09-17T07:09:08.807058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:14.065777+0000","scrub_duration":0.015274785000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"75'9","reported_seq":201,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.170111+0000","last_change":"2024-09-17T07:09:24.432560+0000","last_active":"2024-09-17T07:09:46.170111+0000","last_peered":"2024-09-17T07:09:46.170111+0000","last_clean":"2024-09-17T07:09:46.170111+0000","last_became_active":"2024-09-17T07:09:24.397954+0000","last_became_peered":"2024-09-17T07:09:24.397954+0000","last_unstale":"2024-09-17T07:09:46.170111+0000","last_undegraded":"2024-09-17T07:09:46.170111+0000","last_fullsized":"2024-09-17T07:09:46.170111+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:08:17.673242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:08:17.673242+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0059611450000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":174,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:48.122669+0000","last_change":"2024-09-17T07:08:28.096341+0000","last_active":"2024-09-17T07:09:48.122669+0000","last_peered":"2024-09-17T07:09:48.122669+0000","last_clean":"2024-09-17T07:09:48.122669+0000","last_became_active":"2024-09-17T07:02:16.023774+0000","last_became_peered":"2024-09-17T07:02:16.023774+0000","last_unstale":"2024-09-17T07:09:48.122669+0000","last_undegraded":"2024-09-17T07:09:48.122669+0000","last_fullsized":"2024-09-17T07:09:48.122669+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:28.096266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:28.096266+0000","last_clean_scrub_stamp":"2024-09-17T07:08:28.096266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010716228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":162,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.182174+0000","last_change":"2024-09-17T07:07:34.205329+0000","last_active":"2024-09-17T07:09:46.182174+0000","last_peered":"2024-09-17T07:09:46.182174+0000","last_clean":"2024-09-17T07:09:46.182174+0000","last_became_active":"2024-09-17T07:02:18.029178+0000","last_became_peered":"2024-09-17T07:02:18.029178+0000","last_unstale":"2024-09-17T07:09:46.182174+0000","last_undegraded":"2024-09-17T07:09:46.182174+0000","last_fullsized":"2024-09-17T07:09:46.182174+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:34.205205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:07:34.205205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0057819899999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"201'8672","reported_seq":8714,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-17T07:09:53.405701+0000","last_change":"2024-09-17T07:08:00.759806+0000","last_active":"2024-09-17T07:09:53.405701+0000","last_peered":"2024-09-17T07:09:53.405701+0000","last_clean":"2024-09-17T07:09:53.405701+0000","last_became_active":"2024-09-17T07:08:00.759498+0000","last_became_peered":"2024-09-17T07:08:00.759498+0000","last_unstale":"2024-09-17T07:09:53.405701+0000","last_undegraded":"2024-09-17T07:09:53.405701+0000","last_fullsized":"2024-09-17T07:09:53.405701+0000","mapping_epoch":168,"log_start":"183'4245","ondisk_log_start":"183'4245","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_clean_scrub_stamp":"2024-09-17T07:07:59.220681+0000","objects_scrubbed":0,"log_size":4427,"log_dups_size":0,"ondisk_log_size":4427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":493879296,"num_objects":7536,"num_object_clones":0,"num_object_copies":22608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8672,"num_write_kb":518656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"95'25","reported_seq":213,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.166728+0000","last_change":"2024-09-17T07:09:17.694556+0000","last_active":"2024-09-17T07:09:46.166728+0000","last_peered":"2024-09-17T07:09:46.166728+0000","last_clean":"2024-09-17T07:09:46.166728+0000","last_became_active":"2024-09-17T07:02:14.016270+0000","last_became_peered":"2024-09-17T07:02:14.016270+0000","last_unstale":"2024-09-17T07:09:46.166728+0000","last_undegraded":"2024-09-17T07:09:46.166728+0000","last_fullsized":"2024-09-17T07:09:46.166728+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'25","last_scrub_stamp":"2024-09-17T07:09:17.694236+0000","last_deep_scrub":"95'25","last_deep_scrub_stamp":"2024-09-17T07:04:09.450039+0000","last_clean_scrub_stamp":"2024-09-17T07:09:17.694236+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:24.489956+0000","scrub_duration":0.014129604,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":180,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.186090+0000","last_change":"2024-09-17T07:09:33.812141+0000","last_active":"2024-09-17T07:09:46.186090+0000","last_peered":"2024-09-17T07:09:46.186090+0000","last_clean":"2024-09-17T07:09:46.186090+0000","last_became_active":"2024-09-17T07:02:12.011258+0000","last_became_peered":"2024-09-17T07:02:12.011258+0000","last_unstale":"2024-09-17T07:09:46.186090+0000","last_undegraded":"2024-09-17T07:09:46.186090+0000","last_fullsized":"2024-09-17T07:09:46.186090+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:18.088246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:08:18.088246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0020459860000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":164,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.170626+0000","last_change":"2024-09-17T07:07:24.620899+0000","last_active":"2024-09-17T07:09:46.170626+0000","last_peered":"2024-09-17T07:09:46.170626+0000","last_clean":"2024-09-17T07:09:46.170626+0000","last_became_active":"2024-09-17T07:02:16.018147+0000","last_became_peered":"2024-09-17T07:02:16.018147+0000","last_unstale":"2024-09-17T07:09:46.170626+0000","last_undegraded":"2024-09-17T07:09:46.170626+0000","last_fullsized":"2024-09-17T07:09:46.170626+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:24.620758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:07:24.620758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0050498469999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":21,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.167415+0000","last_change":"2024-09-17T07:09:24.397784+0000","last_active":"2024-09-17T07:09:46.167415+0000","last_peered":"2024-09-17T07:09:46.167415+0000","last_clean":"2024-09-17T07:09:46.167415+0000","last_became_active":"2024-09-17T07:09:24.396334+0000","last_became_peered":"2024-09-17T07:09:24.396334+0000","last_unstale":"2024-09-17T07:09:46.167415+0000","last_undegraded":"2024-09-17T07:09:46.167415+0000","last_fullsized":"2024-09-17T07:09:46.167415+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:21.753939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:42.984088+0000","last_clean_scrub_stamp":"2024-09-17T07:09:21.753939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:28.208006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.0","version":"201'8649","reported_seq":8702,"reported_epoch":201,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-17T07:09:53.111198+0000","last_change":"2024-09-17T07:09:25.475601+0000","last_active":"2024-09-17T07:09:53.111198+0000","last_peered":"2024-09-17T07:09:53.111198+0000","last_clean":"2024-09-17T07:09:23.364870+0000","last_became_active":"2024-09-17T07:09:25.460711+0000","last_became_peered":"2024-09-17T07:09:25.460711+0000","last_unstale":"2024-09-17T07:09:53.111198+0000","last_undegraded":"2024-09-17T07:09:53.111198+0000","last_fullsized":"2024-09-17T07:09:53.111198+0000","mapping_epoch":195,"log_start":"183'3903","ondisk_log_start":"183'3903","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_clean_scrub_stamp":"2024-09-17T07:07:59.220681+0000","objects_scrubbed":0,"log_size":4746,"log_dups_size":0,"ondisk_log_size":4746,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":499843072,"num_objects":7627,"num_object_clones":0,"num_object_copies":22881,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7627,"num_objects_unfound":0,"num_objects_dirty":7627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8649,"num_write_kb":520832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,3],"avail_no_missing":["5","1","3"],"object_location_counts":[{"shards":"1,3,5","objects":7627}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.c","version":"176'15","reported_seq":218,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.166492+0000","last_change":"2024-09-17T07:09:24.427752+0000","last_active":"2024-09-17T07:09:46.166492+0000","last_peered":"2024-09-17T07:09:46.166492+0000","last_clean":"2024-09-17T07:09:46.166492+0000","last_became_active":"2024-09-17T07:09:24.399287+0000","last_became_peered":"2024-09-17T07:09:24.399287+0000","last_unstale":"2024-09-17T07:09:46.166492+0000","last_undegraded":"2024-09-17T07:09:46.166492+0000","last_fullsized":"2024-09-17T07:09:46.166492+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'15","last_scrub_stamp":"2024-09-17T07:08:19.633791+0000","last_deep_scrub":"176'15","last_deep_scrub_stamp":"2024-09-17T07:08:19.633791+0000","last_clean_scrub_stamp":"2024-09-17T07:08:19.633791+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.009818284,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":187,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.174733+0000","last_change":"2024-09-17T07:09:24.407640+0000","last_active":"2024-09-17T07:09:46.174733+0000","last_peered":"2024-09-17T07:09:46.174733+0000","last_clean":"2024-09-17T07:09:46.174733+0000","last_became_active":"2024-09-17T07:09:24.406973+0000","last_became_peered":"2024-09-17T07:09:24.406973+0000","last_unstale":"2024-09-17T07:09:46.174733+0000","last_undegraded":"2024-09-17T07:09:46.174733+0000","last_fullsized":"2024-09-17T07:09:46.174733+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:23.802226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:08:23.802226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0065215869999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"98'9","reported_seq":227,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:50.605772+0000","last_change":"2024-09-17T07:08:31.923373+0000","last_active":"2024-09-17T07:09:50.605772+0000","last_peered":"2024-09-17T07:09:50.605772+0000","last_clean":"2024-09-17T07:09:50.605772+0000","last_became_active":"2024-09-17T07:04:15.110797+0000","last_became_peered":"2024-09-17T07:04:15.110797+0000","last_unstale":"2024-09-17T07:09:50.605772+0000","last_undegraded":"2024-09-17T07:09:50.605772+0000","last_fullsized":"2024-09-17T07:09:50.605772+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-09-17T07:08:31.923228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:08:31.923228+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.015229100000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":173,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.167421+0000","last_change":"2024-09-17T07:08:30.650769+0000","last_active":"2024-09-17T07:09:46.167421+0000","last_peered":"2024-09-17T07:09:46.167421+0000","last_clean":"2024-09-17T07:09:46.167421+0000","last_became_active":"2024-09-17T07:02:18.017460+0000","last_became_peered":"2024-09-17T07:02:18.017460+0000","last_unstale":"2024-09-17T07:09:46.167421+0000","last_undegraded":"2024-09-17T07:09:46.167421+0000","last_fullsized":"2024-09-17T07:09:46.167421+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:30.650419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:08:30.650419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003351316,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"201'8539","reported_seq":8580,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-17T07:09:53.413093+0000","last_change":"2024-09-17T07:08:00.242384+0000","last_active":"2024-09-17T07:09:53.413093+0000","last_peered":"2024-09-17T07:09:53.413093+0000","last_clean":"2024-09-17T07:09:53.413093+0000","last_became_active":"2024-09-17T07:08:00.241996+0000","last_became_peered":"2024-09-17T07:08:00.241996+0000","last_unstale":"2024-09-17T07:09:53.413093+0000","last_undegraded":"2024-09-17T07:09:53.413093+0000","last_fullsized":"2024-09-17T07:09:53.413093+0000","mapping_epoch":168,"log_start":"183'4082","ondisk_log_start":"183'4082","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_clean_scrub_stamp":"2024-09-17T07:07:59.220681+0000","objects_scrubbed":0,"log_size":4457,"log_dups_size":0,"ondisk_log_size":4457,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":491847680,"num_objects":7505,"num_object_clones":0,"num_object_copies":22515,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8539,"num_write_kb":513408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"75'15","reported_seq":226,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.171804+0000","last_change":"2024-09-17T07:09:24.436271+0000","last_active":"2024-09-17T07:09:46.171804+0000","last_peered":"2024-09-17T07:09:46.171804+0000","last_clean":"2024-09-17T07:09:46.171804+0000","last_became_active":"2024-09-17T07:09:24.407311+0000","last_became_peered":"2024-09-17T07:09:24.407311+0000","last_unstale":"2024-09-17T07:09:46.171804+0000","last_undegraded":"2024-09-17T07:09:46.171804+0000","last_fullsized":"2024-09-17T07:09:46.171804+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-09-17T07:08:13.784543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:08:13.784543+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002584257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":188,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.170311+0000","last_change":"2024-09-17T07:09:24.405992+0000","last_active":"2024-09-17T07:09:46.170311+0000","last_peered":"2024-09-17T07:09:46.170311+0000","last_clean":"2024-09-17T07:09:46.170311+0000","last_became_active":"2024-09-17T07:09:24.405655+0000","last_became_peered":"2024-09-17T07:09:24.405655+0000","last_unstale":"2024-09-17T07:09:46.170311+0000","last_undegraded":"2024-09-17T07:09:46.170311+0000","last_fullsized":"2024-09-17T07:09:46.170311+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:20.182256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:09:20.182256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:21.411135+0000","scrub_duration":0.0034116369999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"75'8","reported_seq":264,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:50.605745+0000","last_change":"2024-09-17T07:08:09.787205+0000","last_active":"2024-09-17T07:09:50.605745+0000","last_peered":"2024-09-17T07:09:50.605745+0000","last_clean":"2024-09-17T07:09:50.605745+0000","last_became_active":"2024-09-17T07:02:16.021762+0000","last_became_peered":"2024-09-17T07:02:16.021762+0000","last_unstale":"2024-09-17T07:09:50.605745+0000","last_undegraded":"2024-09-17T07:09:50.605745+0000","last_fullsized":"2024-09-17T07:09:50.605745+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:08:09.787123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:08:09.787123+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001487141,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":172,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.171745+0000","last_change":"2024-09-17T07:08:12.991029+0000","last_active":"2024-09-17T07:09:46.171745+0000","last_peered":"2024-09-17T07:09:46.171745+0000","last_clean":"2024-09-17T07:09:46.171745+0000","last_became_active":"2024-09-17T07:02:18.025332+0000","last_became_peered":"2024-09-17T07:02:18.025332+0000","last_unstale":"2024-09-17T07:09:46.171745+0000","last_undegraded":"2024-09-17T07:09:46.171745+0000","last_fullsized":"2024-09-17T07:09:46.171745+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:12.990915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:08:12.990915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021966199999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.2","version":"201'8791","reported_seq":8831,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-17T07:09:53.056413+0000","last_change":"2024-09-17T07:08:00.237404+0000","last_active":"2024-09-17T07:09:53.056413+0000","last_peered":"2024-09-17T07:09:53.056413+0000","last_clean":"2024-09-17T07:09:53.056413+0000","last_became_active":"2024-09-17T07:08:00.237068+0000","last_became_peered":"2024-09-17T07:08:00.237068+0000","last_unstale":"2024-09-17T07:09:53.056413+0000","last_undegraded":"2024-09-17T07:09:53.056413+0000","last_fullsized":"2024-09-17T07:09:53.056413+0000","mapping_epoch":168,"log_start":"183'3986","ondisk_log_start":"183'3986","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_clean_scrub_stamp":"2024-09-17T07:07:59.220681+0000","objects_scrubbed":0,"log_size":4805,"log_dups_size":0,"ondisk_log_size":4805,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":510984192,"num_objects":7797,"num_object_clones":0,"num_object_copies":23391,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7797,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8791,"num_write_kb":530816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"75'11","reported_seq":196,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.167650+0000","last_change":"2024-09-17T07:08:26.626034+0000","last_active":"2024-09-17T07:09:46.167650+0000","last_peered":"2024-09-17T07:09:46.167650+0000","last_clean":"2024-09-17T07:09:46.167650+0000","last_became_active":"2024-09-17T07:02:14.010227+0000","last_became_peered":"2024-09-17T07:02:14.010227+0000","last_unstale":"2024-09-17T07:09:46.167650+0000","last_undegraded":"2024-09-17T07:09:46.167650+0000","last_fullsized":"2024-09-17T07:09:46.167650+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-09-17T07:08:26.625892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:08:26.625892+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.038079981999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":179,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.187515+0000","last_change":"2024-09-17T07:08:06.100857+0000","last_active":"2024-09-17T07:09:46.187515+0000","last_peered":"2024-09-17T07:09:46.187515+0000","last_clean":"2024-09-17T07:09:46.187515+0000","last_became_active":"2024-09-17T07:02:12.013182+0000","last_became_peered":"2024-09-17T07:02:12.013182+0000","last_unstale":"2024-09-17T07:09:46.187515+0000","last_undegraded":"2024-09-17T07:09:46.187515+0000","last_fullsized":"2024-09-17T07:09:46.187515+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:06.100741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:08:06.100741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0012621170000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":185,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.171816+0000","last_change":"2024-09-17T07:09:24.229155+0000","last_active":"2024-09-17T07:09:46.171816+0000","last_peered":"2024-09-17T07:09:46.171816+0000","last_clean":"2024-09-17T07:09:46.171816+0000","last_became_active":"2024-09-17T07:02:16.020224+0000","last_became_peered":"2024-09-17T07:02:16.020224+0000","last_unstale":"2024-09-17T07:09:46.171816+0000","last_undegraded":"2024-09-17T07:09:46.171816+0000","last_fullsized":"2024-09-17T07:09:46.171816+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:24.229069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:09:24.229069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:38.285780+0000","scrub_duration":0.001199566,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":18,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.176113+0000","last_change":"2024-09-17T07:09:24.408435+0000","last_active":"2024-09-17T07:09:46.176113+0000","last_peered":"2024-09-17T07:09:46.176113+0000","last_clean":"2024-09-17T07:09:46.176113+0000","last_became_active":"2024-09-17T07:09:24.408109+0000","last_became_peered":"2024-09-17T07:09:24.408109+0000","last_unstale":"2024-09-17T07:09:46.176113+0000","last_undegraded":"2024-09-17T07:09:46.176113+0000","last_fullsized":"2024-09-17T07:09:46.176113+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:27.883528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:08:27.883528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:52.363126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.d","version":"201'8659","reported_seq":8702,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-17T07:09:53.120775+0000","last_change":"2024-09-17T07:08:00.760018+0000","last_active":"2024-09-17T07:09:53.120775+0000","last_peered":"2024-09-17T07:09:53.120775+0000","last_clean":"2024-09-17T07:09:53.120775+0000","last_became_active":"2024-09-17T07:08:00.759711+0000","last_became_peered":"2024-09-17T07:08:00.759711+0000","last_unstale":"2024-09-17T07:09:53.120775+0000","last_undegraded":"2024-09-17T07:09:53.120775+0000","last_fullsized":"2024-09-17T07:09:53.120775+0000","mapping_epoch":168,"log_start":"183'3918","ondisk_log_start":"183'3918","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_clean_scrub_stamp":"2024-09-17T07:07:59.220681+0000","objects_scrubbed":0,"log_size":4741,"log_dups_size":0,"ondisk_log_size":4741,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":499515416,"num_objects":7623,"num_object_clones":0,"num_object_copies":22869,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7623,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8659,"num_write_kb":520961,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1","version":"75'14","reported_seq":187,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.166277+0000","last_change":"2024-09-17T07:08:06.763827+0000","last_active":"2024-09-17T07:09:46.166277+0000","last_peered":"2024-09-17T07:09:46.166277+0000","last_clean":"2024-09-17T07:09:46.166277+0000","last_became_active":"2024-09-17T07:02:14.007810+0000","last_became_peered":"2024-09-17T07:02:14.007810+0000","last_unstale":"2024-09-17T07:09:46.166277+0000","last_undegraded":"2024-09-17T07:09:46.166277+0000","last_fullsized":"2024-09-17T07:09:46.166277+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'14","last_scrub_stamp":"2024-09-17T07:08:06.763689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:08:06.763689+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033699329999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":200,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.176529+0000","last_change":"2024-09-17T07:09:23.181150+0000","last_active":"2024-09-17T07:09:46.176529+0000","last_peered":"2024-09-17T07:09:46.176529+0000","last_clean":"2024-09-17T07:09:46.176529+0000","last_became_active":"2024-09-17T07:02:12.011766+0000","last_became_peered":"2024-09-17T07:02:12.011766+0000","last_unstale":"2024-09-17T07:09:46.176529+0000","last_undegraded":"2024-09-17T07:09:46.176529+0000","last_fullsized":"2024-09-17T07:09:46.176529+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:23.181028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:09:23.181028+0000","last_clean_scrub_stamp":"2024-09-17T07:09:23.181028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:51.612899+0000","scrub_duration":0.0051947429999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"196'9","reported_seq":48,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:50.612443+0000","last_change":"2024-09-17T07:09:24.427192+0000","last_active":"2024-09-17T07:09:50.612443+0000","last_peered":"2024-09-17T07:09:50.612443+0000","last_clean":"2024-09-17T07:09:50.612443+0000","last_became_active":"2024-09-17T07:09:24.406056+0000","last_became_peered":"2024-09-17T07:09:24.406056+0000","last_unstale":"2024-09-17T07:09:50.612443+0000","last_undegraded":"2024-09-17T07:09:50.612443+0000","last_fullsized":"2024-09-17T07:09:50.612443+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:09:14.776794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:09:14.776794+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:27.629020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":173,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.185775+0000","last_change":"2024-09-17T07:08:11.123128+0000","last_active":"2024-09-17T07:09:46.185775+0000","last_peered":"2024-09-17T07:09:46.185775+0000","last_clean":"2024-09-17T07:09:46.185775+0000","last_became_active":"2024-09-17T07:02:18.364981+0000","last_became_peered":"2024-09-17T07:02:18.364981+0000","last_unstale":"2024-09-17T07:09:46.185775+0000","last_undegraded":"2024-09-17T07:09:46.185775+0000","last_fullsized":"2024-09-17T07:09:46.185775+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:11.123047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:08:11.123047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014377210000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"201'8952","reported_seq":15319,"reported_epoch":201,"state":"active+remapped+backfilling","last_fresh":"2024-09-17T07:09:54.118023+0000","last_change":"2024-09-17T07:09:25.483625+0000","last_active":"2024-09-17T07:09:54.118023+0000","last_peered":"2024-09-17T07:09:54.118023+0000","last_clean":"2024-09-17T07:09:23.381756+0000","last_became_active":"2024-09-17T07:09:25.464119+0000","last_became_peered":"2024-09-17T07:09:25.464119+0000","last_unstale":"2024-09-17T07:09:54.118023+0000","last_undegraded":"2024-09-17T07:09:54.118023+0000","last_fullsized":"2024-09-17T07:09:54.118023+0000","mapping_epoch":195,"log_start":"183'3920","ondisk_log_start":"183'3920","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_clean_scrub_stamp":"2024-09-17T07:07:59.220681+0000","objects_scrubbed":0,"log_size":5032,"log_dups_size":0,"ondisk_log_size":5032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":510787584,"num_objects":7794,"num_object_clones":0,"num_object_copies":23382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4646,"num_objects_unfound":0,"num_objects_dirty":7794,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8952,"num_write_kb":535872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3157,"num_bytes_recovered":206897152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,3],"avail_no_missing":["0","3","7"],"object_location_counts":[{"shards":"0,3,7","objects":7794}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"75'18","reported_seq":81,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.187202+0000","last_change":"2024-09-17T07:09:24.424752+0000","last_active":"2024-09-17T07:09:46.187202+0000","last_peered":"2024-09-17T07:09:46.187202+0000","last_clean":"2024-09-17T07:09:46.187202+0000","last_became_active":"2024-09-17T07:09:24.403240+0000","last_became_peered":"2024-09-17T07:09:24.403240+0000","last_unstale":"2024-09-17T07:09:46.187202+0000","last_undegraded":"2024-09-17T07:09:46.187202+0000","last_fullsized":"2024-09-17T07:09:46.187202+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'18","last_scrub_stamp":"2024-09-17T07:08:30.831301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:08:30.831301+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":33,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":178,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.171825+0000","last_change":"2024-09-17T07:08:04.692191+0000","last_active":"2024-09-17T07:09:46.171825+0000","last_peered":"2024-09-17T07:09:46.171825+0000","last_clean":"2024-09-17T07:09:46.171825+0000","last_became_active":"2024-09-17T07:02:12.005687+0000","last_became_peered":"2024-09-17T07:02:12.005687+0000","last_unstale":"2024-09-17T07:09:46.171825+0000","last_undegraded":"2024-09-17T07:09:46.171825+0000","last_fullsized":"2024-09-17T07:09:46.171825+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:04.692099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:08:04.692099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001661023,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":173,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.166134+0000","last_change":"2024-09-17T07:09:24.393694+0000","last_active":"2024-09-17T07:09:46.166134+0000","last_peered":"2024-09-17T07:09:46.166134+0000","last_clean":"2024-09-17T07:09:46.166134+0000","last_became_active":"2024-09-17T07:09:24.393375+0000","last_became_peered":"2024-09-17T07:09:24.393375+0000","last_unstale":"2024-09-17T07:09:46.166134+0000","last_undegraded":"2024-09-17T07:09:46.166134+0000","last_fullsized":"2024-09-17T07:09:46.166134+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:03.726764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:08:03.726764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002251201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":182,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.170103+0000","last_change":"2024-09-17T07:09:24.399627+0000","last_active":"2024-09-17T07:09:46.170103+0000","last_peered":"2024-09-17T07:09:46.170103+0000","last_clean":"2024-09-17T07:09:46.170103+0000","last_became_active":"2024-09-17T07:09:24.398659+0000","last_became_peered":"2024-09-17T07:09:24.398659+0000","last_unstale":"2024-09-17T07:09:46.170103+0000","last_undegraded":"2024-09-17T07:09:46.170103+0000","last_fullsized":"2024-09-17T07:09:46.170103+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:25.588903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:08:25.588903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021632869999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"201'8612","reported_seq":8662,"reported_epoch":201,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-17T07:09:53.394736+0000","last_change":"2024-09-17T07:09:25.473026+0000","last_active":"2024-09-17T07:09:53.394736+0000","last_peered":"2024-09-17T07:09:53.394736+0000","last_clean":"2024-09-17T07:09:23.369855+0000","last_became_active":"2024-09-17T07:09:25.456528+0000","last_became_peered":"2024-09-17T07:09:25.456528+0000","last_unstale":"2024-09-17T07:09:53.394736+0000","last_undegraded":"2024-09-17T07:09:53.394736+0000","last_fullsized":"2024-09-17T07:09:53.394736+0000","mapping_epoch":195,"log_start":"183'4169","ondisk_log_start":"183'4169","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_clean_scrub_stamp":"2024-09-17T07:07:59.220681+0000","objects_scrubbed":0,"log_size":4443,"log_dups_size":0,"ondisk_log_size":4443,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":500432896,"num_objects":7636,"num_object_clones":0,"num_object_copies":22908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7636,"num_objects_unfound":0,"num_objects_dirty":7636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8612,"num_write_kb":519936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,3],"avail_no_missing":["1","2","3"],"object_location_counts":[{"shards":"1,2,3","objects":7636}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"75'19","reported_seq":202,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.180784+0000","last_change":"2024-09-17T07:07:17.989796+0000","last_active":"2024-09-17T07:09:46.180784+0000","last_peered":"2024-09-17T07:09:46.180784+0000","last_clean":"2024-09-17T07:09:46.180784+0000","last_became_active":"2024-09-17T07:02:14.027389+0000","last_became_peered":"2024-09-17T07:02:14.027389+0000","last_unstale":"2024-09-17T07:09:46.180784+0000","last_undegraded":"2024-09-17T07:09:46.180784+0000","last_fullsized":"2024-09-17T07:09:46.180784+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-09-17T07:07:17.989435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:07:17.989435+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0043356940000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":179,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.186692+0000","last_change":"2024-09-17T07:08:24.072911+0000","last_active":"2024-09-17T07:09:46.186692+0000","last_peered":"2024-09-17T07:09:46.186692+0000","last_clean":"2024-09-17T07:09:46.186692+0000","last_became_active":"2024-09-17T07:02:12.011917+0000","last_became_peered":"2024-09-17T07:02:12.011917+0000","last_unstale":"2024-09-17T07:09:46.186692+0000","last_undegraded":"2024-09-17T07:09:46.186692+0000","last_fullsized":"2024-09-17T07:09:46.186692+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:24.072799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:24.072799+0000","last_clean_scrub_stamp":"2024-09-17T07:08:24.072799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0047928600000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":175,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.175819+0000","last_change":"2024-09-17T07:09:15.355446+0000","last_active":"2024-09-17T07:09:46.175819+0000","last_peered":"2024-09-17T07:09:46.175819+0000","last_clean":"2024-09-17T07:09:46.175819+0000","last_became_active":"2024-09-17T07:02:16.016199+0000","last_became_peered":"2024-09-17T07:02:16.016199+0000","last_unstale":"2024-09-17T07:09:46.175819+0000","last_undegraded":"2024-09-17T07:09:46.175819+0000","last_fullsized":"2024-09-17T07:09:46.175819+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:15.355318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:09:15.355318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:31.109239+0000","scrub_duration":0.0028918149999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":172,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.170102+0000","last_change":"2024-09-17T07:08:24.840933+0000","last_active":"2024-09-17T07:09:46.170102+0000","last_peered":"2024-09-17T07:09:46.170102+0000","last_clean":"2024-09-17T07:09:46.170102+0000","last_became_active":"2024-09-17T07:02:18.364835+0000","last_became_peered":"2024-09-17T07:02:18.364835+0000","last_unstale":"2024-09-17T07:09:46.170102+0000","last_undegraded":"2024-09-17T07:09:46.170102+0000","last_fullsized":"2024-09-17T07:09:46.170102+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:24.840821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:08:24.840821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002523197,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.e","version":"201'8733","reported_seq":8775,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-17T07:09:54.119705+0000","last_change":"2024-09-17T07:08:00.759796+0000","last_active":"2024-09-17T07:09:54.119705+0000","last_peered":"2024-09-17T07:09:54.119705+0000","last_clean":"2024-09-17T07:09:54.119705+0000","last_became_active":"2024-09-17T07:08:00.759537+0000","last_became_peered":"2024-09-17T07:08:00.759537+0000","last_unstale":"2024-09-17T07:09:54.119705+0000","last_undegraded":"2024-09-17T07:09:54.119705+0000","last_fullsized":"2024-09-17T07:09:54.119705+0000","mapping_epoch":168,"log_start":"183'3651","ondisk_log_start":"183'3651","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_clean_scrub_stamp":"2024-09-17T07:07:59.220681+0000","objects_scrubbed":0,"log_size":5082,"log_dups_size":0,"ondisk_log_size":5082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":498008064,"num_objects":7599,"num_object_clones":0,"num_object_copies":22797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8733,"num_write_kb":522624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"110'15","reported_seq":196,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.174604+0000","last_change":"2024-09-17T07:08:19.835062+0000","last_active":"2024-09-17T07:09:46.174604+0000","last_peered":"2024-09-17T07:09:46.174604+0000","last_clean":"2024-09-17T07:09:46.174604+0000","last_became_active":"2024-09-17T07:02:14.010637+0000","last_became_peered":"2024-09-17T07:02:14.010637+0000","last_unstale":"2024-09-17T07:09:46.174604+0000","last_undegraded":"2024-09-17T07:09:46.174604+0000","last_fullsized":"2024-09-17T07:09:46.174604+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'15","last_scrub_stamp":"2024-09-17T07:08:19.834380+0000","last_deep_scrub":"110'15","last_deep_scrub_stamp":"2024-09-17T07:04:57.091400+0000","last_clean_scrub_stamp":"2024-09-17T07:08:19.834380+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0055436599999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":178,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.170950+0000","last_change":"2024-09-17T07:08:17.806138+0000","last_active":"2024-09-17T07:09:46.170950+0000","last_peered":"2024-09-17T07:09:46.170950+0000","last_clean":"2024-09-17T07:09:46.170950+0000","last_became_active":"2024-09-17T07:02:12.010803+0000","last_became_peered":"2024-09-17T07:02:12.010803+0000","last_unstale":"2024-09-17T07:09:46.170950+0000","last_undegraded":"2024-09-17T07:09:46.170950+0000","last_fullsized":"2024-09-17T07:09:46.170950+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:17.806001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:25.147878+0000","last_clean_scrub_stamp":"2024-09-17T07:08:17.806001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00606531,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"89'9","reported_seq":276,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:50.605710+0000","last_change":"2024-09-17T07:08:14.309093+0000","last_active":"2024-09-17T07:09:50.605710+0000","last_peered":"2024-09-17T07:09:50.605710+0000","last_clean":"2024-09-17T07:09:50.605710+0000","last_became_active":"2024-09-17T07:02:16.021360+0000","last_became_peered":"2024-09-17T07:02:16.021360+0000","last_unstale":"2024-09-17T07:09:50.605710+0000","last_undegraded":"2024-09-17T07:09:50.605710+0000","last_fullsized":"2024-09-17T07:09:50.605710+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'9","last_scrub_stamp":"2024-09-17T07:08:14.308970+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-09-17T07:03:29.961439+0000","last_clean_scrub_stamp":"2024-09-17T07:08:14.308970+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001890011,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"75'1","reported_seq":187,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.181957+0000","last_change":"2024-09-17T07:09:24.411824+0000","last_active":"2024-09-17T07:09:46.181957+0000","last_peered":"2024-09-17T07:09:46.181957+0000","last_clean":"2024-09-17T07:09:46.181957+0000","last_became_active":"2024-09-17T07:09:24.399565+0000","last_became_peered":"2024-09-17T07:09:24.399565+0000","last_unstale":"2024-09-17T07:09:46.181957+0000","last_undegraded":"2024-09-17T07:09:46.181957+0000","last_fullsized":"2024-09-17T07:09:46.181957+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-17T07:08:25.355935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:08:25.355935+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0052992940000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"201'8625","reported_seq":15474,"reported_epoch":201,"state":"active+remapped+backfilling","last_fresh":"2024-09-17T07:09:53.124843+0000","last_change":"2024-09-17T07:09:25.539073+0000","last_active":"2024-09-17T07:09:53.124843+0000","last_peered":"2024-09-17T07:09:53.124843+0000","last_clean":"2024-09-17T07:09:23.376900+0000","last_became_active":"2024-09-17T07:09:25.511414+0000","last_became_peered":"2024-09-17T07:09:25.511414+0000","last_unstale":"2024-09-17T07:09:53.124843+0000","last_undegraded":"2024-09-17T07:09:53.124843+0000","last_fullsized":"2024-09-17T07:09:53.124843+0000","mapping_epoch":195,"log_start":"183'3854","ondisk_log_start":"183'3854","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_clean_scrub_stamp":"2024-09-17T07:07:59.220681+0000","objects_scrubbed":0,"log_size":4771,"log_dups_size":0,"ondisk_log_size":4771,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":497352704,"num_objects":7589,"num_object_clones":0,"num_object_copies":22767,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4165,"num_objects_unfound":0,"num_objects_dirty":7589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8625,"num_write_kb":518848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3399,"num_bytes_recovered":222756864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,3],"avail_no_missing":["5","3","7"],"object_location_counts":[{"shards":"3,5,7","objects":7589}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"196'29","reported_seq":347,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:50.605719+0000","last_change":"2024-09-17T07:09:24.422580+0000","last_active":"2024-09-17T07:09:50.605719+0000","last_peered":"2024-09-17T07:09:50.605719+0000","last_clean":"2024-09-17T07:09:50.605719+0000","last_became_active":"2024-09-17T07:09:24.403267+0000","last_became_peered":"2024-09-17T07:09:24.403267+0000","last_unstale":"2024-09-17T07:09:50.605719+0000","last_undegraded":"2024-09-17T07:09:50.605719+0000","last_fullsized":"2024-09-17T07:09:50.605719+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'28","last_scrub_stamp":"2024-09-17T07:08:26.795647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:08:26.795647+0000","objects_scrubbed":10,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0043656659999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":189,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.172473+0000","last_change":"2024-09-17T07:09:19.253480+0000","last_active":"2024-09-17T07:09:46.172473+0000","last_peered":"2024-09-17T07:09:46.172473+0000","last_clean":"2024-09-17T07:09:46.172473+0000","last_became_active":"2024-09-17T07:02:12.008802+0000","last_became_peered":"2024-09-17T07:02:12.008802+0000","last_unstale":"2024-09-17T07:09:46.172473+0000","last_undegraded":"2024-09-17T07:09:46.172473+0000","last_fullsized":"2024-09-17T07:09:46.172473+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:19.253385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:09:19.253385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:36.878327+0000","scrub_duration":0.0017535820000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":175,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.181266+0000","last_change":"2024-09-17T07:08:04.266668+0000","last_active":"2024-09-17T07:09:46.181266+0000","last_peered":"2024-09-17T07:09:46.181266+0000","last_clean":"2024-09-17T07:09:46.181266+0000","last_became_active":"2024-09-17T07:02:16.016606+0000","last_became_peered":"2024-09-17T07:02:16.016606+0000","last_unstale":"2024-09-17T07:09:46.181266+0000","last_undegraded":"2024-09-17T07:09:46.181266+0000","last_fullsized":"2024-09-17T07:09:46.181266+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:04.266574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:08:04.266574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014388140000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":18,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.167773+0000","last_change":"2024-09-17T07:09:24.403411+0000","last_active":"2024-09-17T07:09:46.167773+0000","last_peered":"2024-09-17T07:09:46.167773+0000","last_clean":"2024-09-17T07:09:46.167773+0000","last_became_active":"2024-09-17T07:09:24.398878+0000","last_became_peered":"2024-09-17T07:09:24.398878+0000","last_unstale":"2024-09-17T07:09:46.167773+0000","last_undegraded":"2024-09-17T07:09:46.167773+0000","last_fullsized":"2024-09-17T07:09:46.167773+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:08.849142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:08:08.849142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.b","version":"201'8810","reported_seq":16840,"reported_epoch":201,"state":"active+remapped+backfilling","last_fresh":"2024-09-17T07:09:53.102179+0000","last_change":"2024-09-17T07:09:25.458949+0000","last_active":"2024-09-17T07:09:53.102179+0000","last_peered":"2024-09-17T07:09:53.102179+0000","last_clean":"2024-09-17T07:09:23.353085+0000","last_became_active":"2024-09-17T07:09:25.453311+0000","last_became_peered":"2024-09-17T07:09:25.453311+0000","last_unstale":"2024-09-17T07:09:53.102179+0000","last_undegraded":"2024-09-17T07:09:53.102179+0000","last_fullsized":"2024-09-17T07:09:53.102179+0000","mapping_epoch":195,"log_start":"183'3970","ondisk_log_start":"183'3970","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_clean_scrub_stamp":"2024-09-17T07:07:59.220681+0000","objects_scrubbed":0,"log_size":4840,"log_dups_size":0,"ondisk_log_size":4840,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":510525440,"num_objects":7790,"num_object_clones":0,"num_object_copies":23370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3771,"num_objects_unfound":0,"num_objects_dirty":7790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8810,"num_write_kb":531200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3989,"num_bytes_recovered":261423104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,3],"avail_no_missing":["4","1","3"],"object_location_counts":[{"shards":"1,3,4","objects":7790}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.7","version":"75'13","reported_seq":203,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.171898+0000","last_change":"2024-09-17T07:08:05.678169+0000","last_active":"2024-09-17T07:09:46.171898+0000","last_peered":"2024-09-17T07:09:46.171898+0000","last_clean":"2024-09-17T07:09:46.171898+0000","last_became_active":"2024-09-17T07:02:14.019838+0000","last_became_peered":"2024-09-17T07:02:14.019838+0000","last_unstale":"2024-09-17T07:09:46.171898+0000","last_undegraded":"2024-09-17T07:09:46.171898+0000","last_fullsized":"2024-09-17T07:09:46.171898+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'13","last_scrub_stamp":"2024-09-17T07:08:05.678063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:08:05.678063+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0024636459999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"76'5","reported_seq":640,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-17T07:09:51.932631+0000","last_change":"2024-09-17T07:08:29.818116+0000","last_active":"2024-09-17T07:09:51.932631+0000","last_peered":"2024-09-17T07:09:51.932631+0000","last_clean":"2024-09-17T07:09:51.932631+0000","last_became_active":"2024-09-17T07:02:12.006883+0000","last_became_peered":"2024-09-17T07:02:12.006883+0000","last_unstale":"2024-09-17T07:09:51.932631+0000","last_undegraded":"2024-09-17T07:09:51.932631+0000","last_fullsized":"2024-09-17T07:09:51.932631+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'5","last_scrub_stamp":"2024-09-17T07:08:29.817785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:08:29.817785+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034070260000000001,"objects_trimmed":0,"snaptrim_duration":0.00033739799999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":454,"num_read_kb":449,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.181682+0000","last_change":"2024-09-17T07:09:24.393139+0000","last_active":"2024-09-17T07:09:46.181682+0000","last_peered":"2024-09-17T07:09:46.181682+0000","last_clean":"2024-09-17T07:09:46.181682+0000","last_became_active":"2024-09-17T07:09:24.392850+0000","last_became_peered":"2024-09-17T07:09:24.392850+0000","last_unstale":"2024-09-17T07:09:46.181682+0000","last_undegraded":"2024-09-17T07:09:46.181682+0000","last_fullsized":"2024-09-17T07:09:46.181682+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:23.833170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:23.833170+0000","last_clean_scrub_stamp":"2024-09-17T07:08:23.833170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":175,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.179549+0000","last_change":"2024-09-17T07:08:14.935764+0000","last_active":"2024-09-17T07:09:46.179549+0000","last_peered":"2024-09-17T07:09:46.179549+0000","last_clean":"2024-09-17T07:09:46.179549+0000","last_became_active":"2024-09-17T07:02:16.014492+0000","last_became_peered":"2024-09-17T07:02:16.014492+0000","last_unstale":"2024-09-17T07:09:46.179549+0000","last_undegraded":"2024-09-17T07:09:46.179549+0000","last_fullsized":"2024-09-17T07:09:46.179549+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:14.935679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:08:14.935679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001590459,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":193,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.186313+0000","last_change":"2024-09-17T07:09:24.403678+0000","last_active":"2024-09-17T07:09:46.186313+0000","last_peered":"2024-09-17T07:09:46.186313+0000","last_clean":"2024-09-17T07:09:46.186313+0000","last_became_active":"2024-09-17T07:09:24.402704+0000","last_became_peered":"2024-09-17T07:09:24.402704+0000","last_unstale":"2024-09-17T07:09:46.186313+0000","last_undegraded":"2024-09-17T07:09:46.186313+0000","last_fullsized":"2024-09-17T07:09:46.186313+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:22.853652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:09:22.853652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:26.725996+0000","scrub_duration":0.0041575259999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"201'8440","reported_seq":8482,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-17T07:09:53.376388+0000","last_change":"2024-09-17T07:08:00.760107+0000","last_active":"2024-09-17T07:09:53.376388+0000","last_peered":"2024-09-17T07:09:53.376388+0000","last_clean":"2024-09-17T07:09:53.376388+0000","last_became_active":"2024-09-17T07:08:00.759827+0000","last_became_peered":"2024-09-17T07:08:00.759827+0000","last_unstale":"2024-09-17T07:09:53.376388+0000","last_undegraded":"2024-09-17T07:09:53.376388+0000","last_fullsized":"2024-09-17T07:09:53.376388+0000","mapping_epoch":168,"log_start":"183'4000","ondisk_log_start":"183'4000","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_clean_scrub_stamp":"2024-09-17T07:07:59.220681+0000","objects_scrubbed":0,"log_size":4440,"log_dups_size":0,"ondisk_log_size":4440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":490340352,"num_objects":7482,"num_object_clones":0,"num_object_copies":22446,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8440,"num_write_kb":509504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"75'12","reported_seq":218,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.183109+0000","last_change":"2024-09-17T07:08:08.295553+0000","last_active":"2024-09-17T07:09:46.183109+0000","last_peered":"2024-09-17T07:09:46.183109+0000","last_clean":"2024-09-17T07:09:46.183109+0000","last_became_active":"2024-09-17T07:03:48.760405+0000","last_became_peered":"2024-09-17T07:03:48.760405+0000","last_unstale":"2024-09-17T07:09:46.183109+0000","last_undegraded":"2024-09-17T07:09:46.183109+0000","last_fullsized":"2024-09-17T07:09:46.183109+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-09-17T07:08:08.295449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:08:08.295449+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023245919999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"70'1","reported_seq":198,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.181813+0000","last_change":"2024-09-17T07:09:24.415023+0000","last_active":"2024-09-17T07:09:46.181813+0000","last_peered":"2024-09-17T07:09:46.181813+0000","last_clean":"2024-09-17T07:09:46.181813+0000","last_became_active":"2024-09-17T07:09:24.402499+0000","last_became_peered":"2024-09-17T07:09:24.402499+0000","last_unstale":"2024-09-17T07:09:46.181813+0000","last_undegraded":"2024-09-17T07:09:46.181813+0000","last_fullsized":"2024-09-17T07:09:46.181813+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-17T07:09:22.249833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:09:22.249833+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:45.077540+0000","scrub_duration":0.0061513059999999996,"objects_trimmed":0,"snaptrim_duration":0.0032298629999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":179,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.187576+0000","last_change":"2024-09-17T07:09:16.865242+0000","last_active":"2024-09-17T07:09:46.187576+0000","last_peered":"2024-09-17T07:09:46.187576+0000","last_clean":"2024-09-17T07:09:46.187576+0000","last_became_active":"2024-09-17T07:02:12.014623+0000","last_became_peered":"2024-09-17T07:02:12.014623+0000","last_unstale":"2024-09-17T07:09:46.187576+0000","last_undegraded":"2024-09-17T07:09:46.187576+0000","last_fullsized":"2024-09-17T07:09:46.187576+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:16.865147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:09:16.865147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:30.769978+0000","scrub_duration":0.005396222,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":174,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.171884+0000","last_change":"2024-09-17T07:08:18.008734+0000","last_active":"2024-09-17T07:09:46.171884+0000","last_peered":"2024-09-17T07:09:46.171884+0000","last_clean":"2024-09-17T07:09:46.171884+0000","last_became_active":"2024-09-17T07:02:16.021561+0000","last_became_peered":"2024-09-17T07:02:16.021561+0000","last_unstale":"2024-09-17T07:09:46.171884+0000","last_undegraded":"2024-09-17T07:09:46.171884+0000","last_fullsized":"2024-09-17T07:09:46.171884+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:18.008600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:08:18.008600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0044198520000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":181,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.170286+0000","last_change":"2024-09-17T07:09:24.396448+0000","last_active":"2024-09-17T07:09:46.170286+0000","last_peered":"2024-09-17T07:09:46.170286+0000","last_clean":"2024-09-17T07:09:46.170286+0000","last_became_active":"2024-09-17T07:09:24.396154+0000","last_became_peered":"2024-09-17T07:09:24.396154+0000","last_unstale":"2024-09-17T07:09:46.170286+0000","last_undegraded":"2024-09-17T07:09:46.170286+0000","last_fullsized":"2024-09-17T07:09:46.170286+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:17.192238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:09:17.192238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:20.769603+0000","scrub_duration":0.058290885000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"201'8378","reported_seq":8561,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-17T07:09:53.160653+0000","last_change":"2024-09-17T07:09:23.077196+0000","last_active":"2024-09-17T07:09:53.160653+0000","last_peered":"2024-09-17T07:09:53.160653+0000","last_clean":"2024-09-17T07:09:53.160653+0000","last_became_active":"2024-09-17T07:08:00.242803+0000","last_became_peered":"2024-09-17T07:08:00.242803+0000","last_unstale":"2024-09-17T07:09:53.160653+0000","last_undegraded":"2024-09-17T07:09:53.160653+0000","last_fullsized":"2024-09-17T07:09:53.160653+0000","mapping_epoch":168,"log_start":"183'3455","ondisk_log_start":"183'3455","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"193'6546","last_scrub_stamp":"2024-09-17T07:09:23.076982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_clean_scrub_stamp":"2024-09-17T07:09:23.076982+0000","objects_scrubbed":6477,"log_size":4923,"log_dups_size":0,"ondisk_log_size":4923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:40.229937+0000","scrub_duration":1.8549729770000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":486932480,"num_objects":7430,"num_object_clones":0,"num_object_copies":22290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8378,"num_write_kb":505856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"75'14","reported_seq":199,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.176332+0000","last_change":"2024-09-17T07:08:28.485668+0000","last_active":"2024-09-17T07:09:46.176332+0000","last_peered":"2024-09-17T07:09:46.176332+0000","last_clean":"2024-09-17T07:09:46.176332+0000","last_became_active":"2024-09-17T07:02:14.028857+0000","last_became_peered":"2024-09-17T07:02:14.028857+0000","last_unstale":"2024-09-17T07:09:46.176332+0000","last_undegraded":"2024-09-17T07:09:46.176332+0000","last_fullsized":"2024-09-17T07:09:46.176332+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'14","last_scrub_stamp":"2024-09-17T07:08:28.485338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:08:28.485338+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:54.485230+0000","scrub_duration":0.0037222499999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"72'2","reported_seq":37,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.172300+0000","last_change":"2024-09-17T07:09:24.420989+0000","last_active":"2024-09-17T07:09:46.172300+0000","last_peered":"2024-09-17T07:09:46.172300+0000","last_clean":"2024-09-17T07:09:46.172300+0000","last_became_active":"2024-09-17T07:09:24.404532+0000","last_became_peered":"2024-09-17T07:09:24.404532+0000","last_unstale":"2024-09-17T07:09:46.172300+0000","last_undegraded":"2024-09-17T07:09:46.172300+0000","last_fullsized":"2024-09-17T07:09:46.172300+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'2","last_scrub_stamp":"2024-09-17T07:08:15.763727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:08:15.763727+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":189,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.174909+0000","last_change":"2024-09-17T07:09:16.180998+0000","last_active":"2024-09-17T07:09:46.174909+0000","last_peered":"2024-09-17T07:09:46.174909+0000","last_clean":"2024-09-17T07:09:46.174909+0000","last_became_active":"2024-09-17T07:02:12.004606+0000","last_became_peered":"2024-09-17T07:02:12.004606+0000","last_unstale":"2024-09-17T07:09:46.174909+0000","last_undegraded":"2024-09-17T07:09:46.174909+0000","last_fullsized":"2024-09-17T07:09:46.174909+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:16.180756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:09:16.180756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:20.772806+0000","scrub_duration":0.014479524000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"78'128","reported_seq":427,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.187720+0000","last_change":"2024-09-17T07:08:14.128641+0000","last_active":"2024-09-17T07:09:46.187720+0000","last_peered":"2024-09-17T07:09:46.187720+0000","last_clean":"2024-09-17T07:09:46.187720+0000","last_became_active":"2024-09-17T07:02:03.020675+0000","last_became_peered":"2024-09-17T07:02:03.020675+0000","last_unstale":"2024-09-17T07:09:46.187720+0000","last_undegraded":"2024-09-17T07:09:46.187720+0000","last_fullsized":"2024-09-17T07:09:46.187720+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'128","last_scrub_stamp":"2024-09-17T07:08:14.128519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:00:09.390381+0000","last_clean_scrub_stamp":"2024-09-17T07:08:14.128519+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021141889999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":175,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.187158+0000","last_change":"2024-09-17T07:08:05.064380+0000","last_active":"2024-09-17T07:09:46.187158+0000","last_peered":"2024-09-17T07:09:46.187158+0000","last_clean":"2024-09-17T07:09:46.187158+0000","last_became_active":"2024-09-17T07:02:16.016031+0000","last_became_peered":"2024-09-17T07:02:16.016031+0000","last_unstale":"2024-09-17T07:09:46.187158+0000","last_undegraded":"2024-09-17T07:09:46.187158+0000","last_fullsized":"2024-09-17T07:09:46.187158+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:05.064241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:08:05.064241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015017990000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":181,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.172015+0000","last_change":"2024-09-17T07:09:24.400356+0000","last_active":"2024-09-17T07:09:46.172015+0000","last_peered":"2024-09-17T07:09:46.172015+0000","last_clean":"2024-09-17T07:09:46.172015+0000","last_became_active":"2024-09-17T07:09:24.400041+0000","last_became_peered":"2024-09-17T07:09:24.400041+0000","last_unstale":"2024-09-17T07:09:46.172015+0000","last_undegraded":"2024-09-17T07:09:46.172015+0000","last_fullsized":"2024-09-17T07:09:46.172015+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:15.194496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:58.670571+0000","last_clean_scrub_stamp":"2024-09-17T07:09:15.194496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:22.750595+0000","scrub_duration":0.0091630700000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"201'8726","reported_seq":8768,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-17T07:09:54.130631+0000","last_change":"2024-09-17T07:08:00.599954+0000","last_active":"2024-09-17T07:09:54.130631+0000","last_peered":"2024-09-17T07:09:54.130631+0000","last_clean":"2024-09-17T07:09:54.130631+0000","last_became_active":"2024-09-17T07:08:00.599685+0000","last_became_peered":"2024-09-17T07:08:00.599685+0000","last_unstale":"2024-09-17T07:09:54.130631+0000","last_undegraded":"2024-09-17T07:09:54.130631+0000","last_fullsized":"2024-09-17T07:09:54.130631+0000","mapping_epoch":168,"log_start":"183'3641","ondisk_log_start":"183'3641","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_clean_scrub_stamp":"2024-09-17T07:07:59.220681+0000","objects_scrubbed":0,"log_size":5085,"log_dups_size":0,"ondisk_log_size":5085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":496238592,"num_objects":7572,"num_object_clones":0,"num_object_copies":22716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8726,"num_write_kb":521536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"75'19","reported_seq":208,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.176121+0000","last_change":"2024-09-17T07:09:11.299059+0000","last_active":"2024-09-17T07:09:46.176121+0000","last_peered":"2024-09-17T07:09:46.176121+0000","last_clean":"2024-09-17T07:09:46.176121+0000","last_became_active":"2024-09-17T07:02:14.027313+0000","last_became_peered":"2024-09-17T07:02:14.027313+0000","last_unstale":"2024-09-17T07:09:46.176121+0000","last_undegraded":"2024-09-17T07:09:46.176121+0000","last_fullsized":"2024-09-17T07:09:46.176121+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-09-17T07:09:11.298950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:09:11.298950+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:23.840692+0000","scrub_duration":0.0052578449999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":190,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.178971+0000","last_change":"2024-09-17T07:08:15.952173+0000","last_active":"2024-09-17T07:09:46.178971+0000","last_peered":"2024-09-17T07:09:46.178971+0000","last_clean":"2024-09-17T07:09:46.178971+0000","last_became_active":"2024-09-17T07:02:12.012037+0000","last_became_peered":"2024-09-17T07:02:12.012037+0000","last_unstale":"2024-09-17T07:09:46.178971+0000","last_undegraded":"2024-09-17T07:09:46.178971+0000","last_fullsized":"2024-09-17T07:09:46.178971+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:15.952078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:06:39.322535+0000","last_clean_scrub_stamp":"2024-09-17T07:08:15.952078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017662470000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":186,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.179577+0000","last_change":"2024-09-17T07:08:17.003265+0000","last_active":"2024-09-17T07:09:46.179577+0000","last_peered":"2024-09-17T07:09:46.179577+0000","last_clean":"2024-09-17T07:09:46.179577+0000","last_became_active":"2024-09-17T07:02:16.014631+0000","last_became_peered":"2024-09-17T07:02:16.014631+0000","last_unstale":"2024-09-17T07:09:46.179577+0000","last_undegraded":"2024-09-17T07:09:46.179577+0000","last_fullsized":"2024-09-17T07:09:46.179577+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:17.003116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:08:17.003116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035024880000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":182,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.187649+0000","last_change":"2024-09-17T07:09:24.404437+0000","last_active":"2024-09-17T07:09:46.187649+0000","last_peered":"2024-09-17T07:09:46.187649+0000","last_clean":"2024-09-17T07:09:46.187649+0000","last_became_active":"2024-09-17T07:09:24.403904+0000","last_became_peered":"2024-09-17T07:09:24.403904+0000","last_unstale":"2024-09-17T07:09:46.187649+0000","last_undegraded":"2024-09-17T07:09:46.187649+0000","last_fullsized":"2024-09-17T07:09:46.187649+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:10.781244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:09:10.781244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:39.854595+0000","scrub_duration":0.010463343,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.7","version":"201'8759","reported_seq":14477,"reported_epoch":201,"state":"active+remapped+backfilling","last_fresh":"2024-09-17T07:09:53.416529+0000","last_change":"2024-09-17T07:09:25.506880+0000","last_active":"2024-09-17T07:09:53.416529+0000","last_peered":"2024-09-17T07:09:53.416529+0000","last_clean":"2024-09-17T07:09:23.361471+0000","last_became_active":"2024-09-17T07:09:25.487755+0000","last_became_peered":"2024-09-17T07:09:25.487755+0000","last_unstale":"2024-09-17T07:09:53.416529+0000","last_undegraded":"2024-09-17T07:09:53.416529+0000","last_fullsized":"2024-09-17T07:09:53.416529+0000","mapping_epoch":195,"log_start":"183'4285","ondisk_log_start":"183'4285","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_clean_scrub_stamp":"2024-09-17T07:07:59.220681+0000","objects_scrubbed":0,"log_size":4474,"log_dups_size":0,"ondisk_log_size":4474,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":506658816,"num_objects":7731,"num_object_clones":0,"num_object_copies":23193,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4845,"num_objects_unfound":0,"num_objects_dirty":7731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8759,"num_write_kb":527680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2833,"num_bytes_recovered":185663488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,3],"avail_no_missing":["1","3","6"],"object_location_counts":[{"shards":"1,3,6","objects":7731}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"75'9","reported_seq":193,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.180712+0000","last_change":"2024-09-17T07:08:30.220173+0000","last_active":"2024-09-17T07:09:46.180712+0000","last_peered":"2024-09-17T07:09:46.180712+0000","last_clean":"2024-09-17T07:09:46.180712+0000","last_became_active":"2024-09-17T07:02:14.014939+0000","last_became_peered":"2024-09-17T07:02:14.014939+0000","last_unstale":"2024-09-17T07:09:46.180712+0000","last_undegraded":"2024-09-17T07:09:46.180712+0000","last_fullsized":"2024-09-17T07:09:46.180712+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:08:30.219974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:08:30.219974+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033974780000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":187,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.174014+0000","last_change":"2024-09-17T07:09:24.391612+0000","last_active":"2024-09-17T07:09:46.174014+0000","last_peered":"2024-09-17T07:09:46.174014+0000","last_clean":"2024-09-17T07:09:46.174014+0000","last_became_active":"2024-09-17T07:09:24.391207+0000","last_became_peered":"2024-09-17T07:09:24.391207+0000","last_unstale":"2024-09-17T07:09:46.174014+0000","last_undegraded":"2024-09-17T07:09:46.174014+0000","last_fullsized":"2024-09-17T07:09:46.174014+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:20.806468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:06:30.038104+0000","last_clean_scrub_stamp":"2024-09-17T07:08:20.806468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.005853245,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":138,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.169965+0000","last_change":"2024-09-17T07:09:24.404480+0000","last_active":"2024-09-17T07:09:46.169965+0000","last_peered":"2024-09-17T07:09:46.169965+0000","last_clean":"2024-09-17T07:09:46.169965+0000","last_became_active":"2024-09-17T07:09:24.396642+0000","last_became_peered":"2024-09-17T07:09:24.396642+0000","last_unstale":"2024-09-17T07:09:46.169965+0000","last_undegraded":"2024-09-17T07:09:46.169965+0000","last_fullsized":"2024-09-17T07:09:46.169965+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:19.789134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:07:19.789134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010505654,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":173,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.181746+0000","last_change":"2024-09-17T07:09:08.316161+0000","last_active":"2024-09-17T07:09:46.181746+0000","last_peered":"2024-09-17T07:09:46.181746+0000","last_clean":"2024-09-17T07:09:46.181746+0000","last_became_active":"2024-09-17T07:02:18.026994+0000","last_became_peered":"2024-09-17T07:02:18.026994+0000","last_unstale":"2024-09-17T07:09:46.181746+0000","last_undegraded":"2024-09-17T07:09:46.181746+0000","last_fullsized":"2024-09-17T07:09:46.181746+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:08.315842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:58.099849+0000","last_clean_scrub_stamp":"2024-09-17T07:09:08.315842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:28.565288+0000","scrub_duration":0.0081844919999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"201'8757","reported_seq":8799,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-17T07:09:53.120525+0000","last_change":"2024-09-17T07:08:00.760307+0000","last_active":"2024-09-17T07:09:53.120525+0000","last_peered":"2024-09-17T07:09:53.120525+0000","last_clean":"2024-09-17T07:09:53.120525+0000","last_became_active":"2024-09-17T07:08:00.759662+0000","last_became_peered":"2024-09-17T07:08:00.759662+0000","last_unstale":"2024-09-17T07:09:53.120525+0000","last_undegraded":"2024-09-17T07:09:53.120525+0000","last_fullsized":"2024-09-17T07:09:53.120525+0000","mapping_epoch":168,"log_start":"183'3973","ondisk_log_start":"183'3973","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_clean_scrub_stamp":"2024-09-17T07:07:59.220681+0000","objects_scrubbed":0,"log_size":4784,"log_dups_size":0,"ondisk_log_size":4784,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":506527744,"num_objects":7729,"num_object_clones":0,"num_object_copies":23187,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8757,"num_write_kb":527552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"111'22","reported_seq":208,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.171442+0000","last_change":"2024-09-17T07:07:29.816377+0000","last_active":"2024-09-17T07:09:46.171442+0000","last_peered":"2024-09-17T07:09:46.171442+0000","last_clean":"2024-09-17T07:09:46.171442+0000","last_became_active":"2024-09-17T07:02:14.028392+0000","last_became_peered":"2024-09-17T07:02:14.028392+0000","last_unstale":"2024-09-17T07:09:46.171442+0000","last_undegraded":"2024-09-17T07:09:46.171442+0000","last_fullsized":"2024-09-17T07:09:46.171442+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'22","last_scrub_stamp":"2024-09-17T07:07:29.816241+0000","last_deep_scrub":"111'22","last_deep_scrub_stamp":"2024-09-17T07:04:57.500450+0000","last_clean_scrub_stamp":"2024-09-17T07:07:29.816241+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034677570000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'2","reported_seq":62,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.186790+0000","last_change":"2024-09-17T07:09:24.428473+0000","last_active":"2024-09-17T07:09:46.186790+0000","last_peered":"2024-09-17T07:09:46.186790+0000","last_clean":"2024-09-17T07:09:46.186790+0000","last_became_active":"2024-09-17T07:09:24.402909+0000","last_became_peered":"2024-09-17T07:09:24.402909+0000","last_unstale":"2024-09-17T07:09:46.186790+0000","last_undegraded":"2024-09-17T07:09:46.186790+0000","last_fullsized":"2024-09-17T07:09:46.186790+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-09-17T07:09:19.720425+0000","last_deep_scrub":"89'2","last_deep_scrub_stamp":"2024-09-17T07:03:26.694366+0000","last_clean_scrub_stamp":"2024-09-17T07:09:19.720425+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:33.916032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":420,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":420,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"75'8","reported_seq":278,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:50.605789+0000","last_change":"2024-09-17T07:08:20.045908+0000","last_active":"2024-09-17T07:09:50.605789+0000","last_peered":"2024-09-17T07:09:50.605789+0000","last_clean":"2024-09-17T07:09:50.605789+0000","last_became_active":"2024-09-17T07:02:16.015418+0000","last_became_peered":"2024-09-17T07:02:16.015418+0000","last_unstale":"2024-09-17T07:09:50.605789+0000","last_undegraded":"2024-09-17T07:09:50.605789+0000","last_fullsized":"2024-09-17T07:09:50.605789+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:08:20.045781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:08:20.045781+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023108299999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":172,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.172019+0000","last_change":"2024-09-17T07:07:19.786753+0000","last_active":"2024-09-17T07:09:46.172019+0000","last_peered":"2024-09-17T07:09:46.172019+0000","last_clean":"2024-09-17T07:09:46.172019+0000","last_became_active":"2024-09-17T07:02:18.363253+0000","last_became_peered":"2024-09-17T07:02:18.363253+0000","last_unstale":"2024-09-17T07:09:46.172019+0000","last_undegraded":"2024-09-17T07:09:46.172019+0000","last_fullsized":"2024-09-17T07:09:46.172019+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:19.786442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:07:19.786442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011747911999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.5","version":"201'8743","reported_seq":8785,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-17T07:09:53.121338+0000","last_change":"2024-09-17T07:08:00.598660+0000","last_active":"2024-09-17T07:09:53.121338+0000","last_peered":"2024-09-17T07:09:53.121338+0000","last_clean":"2024-09-17T07:09:53.121338+0000","last_became_active":"2024-09-17T07:08:00.598353+0000","last_became_peered":"2024-09-17T07:08:00.598353+0000","last_unstale":"2024-09-17T07:09:53.121338+0000","last_undegraded":"2024-09-17T07:09:53.121338+0000","last_fullsized":"2024-09-17T07:09:53.121338+0000","mapping_epoch":168,"log_start":"183'3795","ondisk_log_start":"183'3795","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_clean_scrub_stamp":"2024-09-17T07:07:59.220681+0000","objects_scrubbed":0,"log_size":4948,"log_dups_size":0,"ondisk_log_size":4948,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":505872384,"num_objects":7719,"num_object_clones":0,"num_object_copies":23157,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7719,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8743,"num_write_kb":526784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"150'17","reported_seq":226,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.166206+0000","last_change":"2024-09-17T07:09:24.423923+0000","last_active":"2024-09-17T07:09:46.166206+0000","last_peered":"2024-09-17T07:09:46.166206+0000","last_clean":"2024-09-17T07:09:46.166206+0000","last_became_active":"2024-09-17T07:09:24.399164+0000","last_became_peered":"2024-09-17T07:09:24.399164+0000","last_unstale":"2024-09-17T07:09:46.166206+0000","last_undegraded":"2024-09-17T07:09:46.166206+0000","last_fullsized":"2024-09-17T07:09:46.166206+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'17","last_scrub_stamp":"2024-09-17T07:08:34.714204+0000","last_deep_scrub":"150'17","last_deep_scrub_stamp":"2024-09-17T07:06:38.775788+0000","last_clean_scrub_stamp":"2024-09-17T07:08:34.714204+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0065052799999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"68'2","reported_seq":176,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.167729+0000","last_change":"2024-09-17T07:07:20.764435+0000","last_active":"2024-09-17T07:09:46.167729+0000","last_peered":"2024-09-17T07:09:46.167729+0000","last_clean":"2024-09-17T07:09:46.167729+0000","last_became_active":"2024-09-17T07:02:12.006716+0000","last_became_peered":"2024-09-17T07:02:12.006716+0000","last_unstale":"2024-09-17T07:09:46.167729+0000","last_undegraded":"2024-09-17T07:09:46.167729+0000","last_fullsized":"2024-09-17T07:09:46.167729+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-09-17T07:07:20.764283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:07:20.764283+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0060160040000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":139,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.182085+0000","last_change":"2024-09-17T07:08:10.273572+0000","last_active":"2024-09-17T07:09:46.182085+0000","last_peered":"2024-09-17T07:09:46.182085+0000","last_clean":"2024-09-17T07:09:46.182085+0000","last_became_active":"2024-09-17T07:04:15.111069+0000","last_became_peered":"2024-09-17T07:04:15.111069+0000","last_unstale":"2024-09-17T07:09:46.182085+0000","last_undegraded":"2024-09-17T07:09:46.182085+0000","last_fullsized":"2024-09-17T07:09:46.182085+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:10.273485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:08:10.273485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013521,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":19,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.185649+0000","last_change":"2024-09-17T07:09:33.811857+0000","last_active":"2024-09-17T07:09:46.185649+0000","last_peered":"2024-09-17T07:09:46.185649+0000","last_clean":"2024-09-17T07:09:46.185649+0000","last_became_active":"2024-09-17T07:09:24.397669+0000","last_became_peered":"2024-09-17T07:09:24.397669+0000","last_unstale":"2024-09-17T07:09:46.185649+0000","last_undegraded":"2024-09-17T07:09:46.185649+0000","last_fullsized":"2024-09-17T07:09:46.185649+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:07.878506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:08:07.878506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"75'9","reported_seq":212,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.183037+0000","last_change":"2024-09-17T07:09:24.429695+0000","last_active":"2024-09-17T07:09:46.183037+0000","last_peered":"2024-09-17T07:09:46.183037+0000","last_clean":"2024-09-17T07:09:46.183037+0000","last_became_active":"2024-09-17T07:09:24.408875+0000","last_became_peered":"2024-09-17T07:09:24.408875+0000","last_unstale":"2024-09-17T07:09:46.183037+0000","last_undegraded":"2024-09-17T07:09:46.183037+0000","last_fullsized":"2024-09-17T07:09:46.183037+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:08:19.316248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:08:19.316248+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0059518610000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":190,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.175881+0000","last_change":"2024-09-17T07:08:17.976403+0000","last_active":"2024-09-17T07:09:46.175881+0000","last_peered":"2024-09-17T07:09:46.175881+0000","last_clean":"2024-09-17T07:09:46.175881+0000","last_became_active":"2024-09-17T07:02:12.011327+0000","last_became_peered":"2024-09-17T07:02:12.011327+0000","last_unstale":"2024-09-17T07:09:46.175881+0000","last_undegraded":"2024-09-17T07:09:46.175881+0000","last_fullsized":"2024-09-17T07:09:46.175881+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:17.976267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:17.976267+0000","last_clean_scrub_stamp":"2024-09-17T07:08:17.976267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035138019999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":18,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.171372+0000","last_change":"2024-09-17T07:09:24.409485+0000","last_active":"2024-09-17T07:09:46.171372+0000","last_peered":"2024-09-17T07:09:46.171372+0000","last_clean":"2024-09-17T07:09:46.171372+0000","last_became_active":"2024-09-17T07:09:24.407806+0000","last_became_peered":"2024-09-17T07:09:24.407806+0000","last_unstale":"2024-09-17T07:09:46.171372+0000","last_undegraded":"2024-09-17T07:09:46.171372+0000","last_fullsized":"2024-09-17T07:09:46.171372+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:22.799056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:08:22.799056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":195,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.176481+0000","last_change":"2024-09-17T07:09:23.379011+0000","last_active":"2024-09-17T07:09:46.176481+0000","last_peered":"2024-09-17T07:09:46.176481+0000","last_clean":"2024-09-17T07:09:46.176481+0000","last_became_active":"2024-09-17T07:02:18.017825+0000","last_became_peered":"2024-09-17T07:02:18.017825+0000","last_unstale":"2024-09-17T07:09:46.176481+0000","last_undegraded":"2024-09-17T07:09:46.176481+0000","last_fullsized":"2024-09-17T07:09:46.176481+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:23.378805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:09:23.378805+0000","last_clean_scrub_stamp":"2024-09-17T07:09:23.378805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:33.079700+0000","scrub_duration":0.0078152310000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":179,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.176047+0000","last_change":"2024-09-17T07:09:33.804153+0000","last_active":"2024-09-17T07:09:46.176047+0000","last_peered":"2024-09-17T07:09:46.176047+0000","last_clean":"2024-09-17T07:09:46.176047+0000","last_became_active":"2024-09-17T07:02:12.008065+0000","last_became_peered":"2024-09-17T07:02:12.008065+0000","last_unstale":"2024-09-17T07:09:46.176047+0000","last_undegraded":"2024-09-17T07:09:46.176047+0000","last_fullsized":"2024-09-17T07:09:46.176047+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:04.480504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:08:04.480504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013838889999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"75'6","reported_seq":48,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.185971+0000","last_change":"2024-09-17T07:09:24.433976+0000","last_active":"2024-09-17T07:09:46.185971+0000","last_peered":"2024-09-17T07:09:46.185971+0000","last_clean":"2024-09-17T07:09:46.185971+0000","last_became_active":"2024-09-17T07:09:24.405385+0000","last_became_peered":"2024-09-17T07:09:24.405385+0000","last_unstale":"2024-09-17T07:09:46.185971+0000","last_undegraded":"2024-09-17T07:09:46.185971+0000","last_fullsized":"2024-09-17T07:09:46.185971+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'6","last_scrub_stamp":"2024-09-17T07:08:33.821542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:08:33.821542+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":183,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.171837+0000","last_change":"2024-09-17T07:09:24.399278+0000","last_active":"2024-09-17T07:09:46.171837+0000","last_peered":"2024-09-17T07:09:46.171837+0000","last_clean":"2024-09-17T07:09:46.171837+0000","last_became_active":"2024-09-17T07:09:24.398922+0000","last_became_peered":"2024-09-17T07:09:24.398922+0000","last_unstale":"2024-09-17T07:09:46.171837+0000","last_undegraded":"2024-09-17T07:09:46.171837+0000","last_fullsized":"2024-09-17T07:09:46.171837+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:06.932318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:08:06.932318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017321260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":173,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.186248+0000","last_change":"2024-09-17T07:08:33.926391+0000","last_active":"2024-09-17T07:09:46.186248+0000","last_peered":"2024-09-17T07:09:46.186248+0000","last_clean":"2024-09-17T07:09:46.186248+0000","last_became_active":"2024-09-17T07:02:18.365419+0000","last_became_peered":"2024-09-17T07:02:18.365419+0000","last_unstale":"2024-09-17T07:09:46.186248+0000","last_undegraded":"2024-09-17T07:09:46.186248+0000","last_fullsized":"2024-09-17T07:09:46.186248+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:33.926286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:08:33.926286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:56.396105+0000","scrub_duration":0.0014019519999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"75'10","reported_seq":58,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.170721+0000","last_change":"2024-09-17T07:09:24.435532+0000","last_active":"2024-09-17T07:09:46.170721+0000","last_peered":"2024-09-17T07:09:46.170721+0000","last_clean":"2024-09-17T07:09:46.170721+0000","last_became_active":"2024-09-17T07:09:24.405944+0000","last_became_peered":"2024-09-17T07:09:24.405944+0000","last_unstale":"2024-09-17T07:09:46.170721+0000","last_undegraded":"2024-09-17T07:09:46.170721+0000","last_fullsized":"2024-09-17T07:09:46.170721+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-09-17T07:08:18.723247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:08:18.723247+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":198,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.172374+0000","last_change":"2024-09-17T07:09:24.403615+0000","last_active":"2024-09-17T07:09:46.172374+0000","last_peered":"2024-09-17T07:09:46.172374+0000","last_clean":"2024-09-17T07:09:46.172374+0000","last_became_active":"2024-09-17T07:09:24.402355+0000","last_became_peered":"2024-09-17T07:09:24.402355+0000","last_unstale":"2024-09-17T07:09:46.172374+0000","last_undegraded":"2024-09-17T07:09:46.172374+0000","last_fullsized":"2024-09-17T07:09:46.172374+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:25.065984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:08:25.065984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:51.463782+0000","scrub_duration":0.0052651929999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"163'9","reported_seq":275,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:50.605658+0000","last_change":"2024-09-17T07:07:34.933956+0000","last_active":"2024-09-17T07:09:50.605658+0000","last_peered":"2024-09-17T07:09:50.605658+0000","last_clean":"2024-09-17T07:09:50.605658+0000","last_became_active":"2024-09-17T07:02:16.022583+0000","last_became_peered":"2024-09-17T07:02:16.022583+0000","last_unstale":"2024-09-17T07:09:50.605658+0000","last_undegraded":"2024-09-17T07:09:50.605658+0000","last_fullsized":"2024-09-17T07:09:50.605658+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'9","last_scrub_stamp":"2024-09-17T07:07:34.933808+0000","last_deep_scrub":"163'9","last_deep_scrub_stamp":"2024-09-17T07:07:34.933808+0000","last_clean_scrub_stamp":"2024-09-17T07:07:34.933808+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.019504579000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":182,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.182706+0000","last_change":"2024-09-17T07:09:24.409388+0000","last_active":"2024-09-17T07:09:46.182706+0000","last_peered":"2024-09-17T07:09:46.182706+0000","last_clean":"2024-09-17T07:09:46.182706+0000","last_became_active":"2024-09-17T07:09:24.409051+0000","last_became_peered":"2024-09-17T07:09:24.409051+0000","last_unstale":"2024-09-17T07:09:46.182706+0000","last_undegraded":"2024-09-17T07:09:46.182706+0000","last_fullsized":"2024-09-17T07:09:46.182706+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:12.338453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:09:12.338453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:19.395637+0000","scrub_duration":0.0011356249999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"75'9","reported_seq":222,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.171952+0000","last_change":"2024-09-17T07:09:24.418573+0000","last_active":"2024-09-17T07:09:46.171952+0000","last_peered":"2024-09-17T07:09:46.171952+0000","last_clean":"2024-09-17T07:09:46.171952+0000","last_became_active":"2024-09-17T07:09:24.406333+0000","last_became_peered":"2024-09-17T07:09:24.406333+0000","last_unstale":"2024-09-17T07:09:46.171952+0000","last_undegraded":"2024-09-17T07:09:46.171952+0000","last_fullsized":"2024-09-17T07:09:46.171952+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:09:09.142335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:09:09.142335+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:09.764304+0000","scrub_duration":0.0041253560000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":188,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.180864+0000","last_change":"2024-09-17T07:09:24.402987+0000","last_active":"2024-09-17T07:09:46.180864+0000","last_peered":"2024-09-17T07:09:46.180864+0000","last_clean":"2024-09-17T07:09:46.180864+0000","last_became_active":"2024-09-17T07:09:24.402704+0000","last_became_peered":"2024-09-17T07:09:24.402704+0000","last_unstale":"2024-09-17T07:09:46.180864+0000","last_undegraded":"2024-09-17T07:09:46.180864+0000","last_fullsized":"2024-09-17T07:09:46.180864+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:15.356187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:08:15.356187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015925290000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"196'9","reported_seq":49,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:50.605967+0000","last_change":"2024-09-17T07:09:28.275921+0000","last_active":"2024-09-17T07:09:50.605967+0000","last_peered":"2024-09-17T07:09:50.605967+0000","last_clean":"2024-09-17T07:09:50.605967+0000","last_became_active":"2024-09-17T07:09:24.406267+0000","last_became_peered":"2024-09-17T07:09:24.406267+0000","last_unstale":"2024-09-17T07:09:50.605967+0000","last_undegraded":"2024-09-17T07:09:50.605967+0000","last_fullsized":"2024-09-17T07:09:50.605967+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:07:32.669921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:07:32.669921+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":173,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.167482+0000","last_change":"2024-09-17T07:08:16.703606+0000","last_active":"2024-09-17T07:09:46.167482+0000","last_peered":"2024-09-17T07:09:46.167482+0000","last_clean":"2024-09-17T07:09:46.167482+0000","last_became_active":"2024-09-17T07:02:18.018716+0000","last_became_peered":"2024-09-17T07:02:18.018716+0000","last_unstale":"2024-09-17T07:09:46.167482+0000","last_undegraded":"2024-09-17T07:09:46.167482+0000","last_fullsized":"2024-09-17T07:09:46.167482+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:16.703532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:08:16.703532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019164340000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"75'9","reported_seq":192,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.174525+0000","last_change":"2024-09-17T07:08:16.756725+0000","last_active":"2024-09-17T07:09:46.174525+0000","last_peered":"2024-09-17T07:09:46.174525+0000","last_clean":"2024-09-17T07:09:46.174525+0000","last_became_active":"2024-09-17T07:02:14.010289+0000","last_became_peered":"2024-09-17T07:02:14.010289+0000","last_unstale":"2024-09-17T07:09:46.174525+0000","last_undegraded":"2024-09-17T07:09:46.174525+0000","last_fullsized":"2024-09-17T07:09:46.174525+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:08:16.756185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:08:16.756185+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004786584,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":188,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.176240+0000","last_change":"2024-09-17T07:09:24.408438+0000","last_active":"2024-09-17T07:09:46.176240+0000","last_peered":"2024-09-17T07:09:46.176240+0000","last_clean":"2024-09-17T07:09:46.176240+0000","last_became_active":"2024-09-17T07:09:24.408139+0000","last_became_peered":"2024-09-17T07:09:24.408139+0000","last_unstale":"2024-09-17T07:09:46.176240+0000","last_undegraded":"2024-09-17T07:09:46.176240+0000","last_fullsized":"2024-09-17T07:09:46.176240+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:07.329897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:09:07.329897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:12.235332+0000","scrub_duration":0.010864379,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":18,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.180509+0000","last_change":"2024-09-17T07:09:24.399604+0000","last_active":"2024-09-17T07:09:46.180509+0000","last_peered":"2024-09-17T07:09:46.180509+0000","last_clean":"2024-09-17T07:09:46.180509+0000","last_became_active":"2024-09-17T07:09:24.399116+0000","last_became_peered":"2024-09-17T07:09:24.399116+0000","last_unstale":"2024-09-17T07:09:46.180509+0000","last_undegraded":"2024-09-17T07:09:46.180509+0000","last_fullsized":"2024-09-17T07:09:46.180509+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:17.713027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:07:17.713027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":173,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.181808+0000","last_change":"2024-09-17T07:08:18.338994+0000","last_active":"2024-09-17T07:09:46.181808+0000","last_peered":"2024-09-17T07:09:46.181808+0000","last_clean":"2024-09-17T07:09:46.181808+0000","last_became_active":"2024-09-17T07:02:18.027622+0000","last_became_peered":"2024-09-17T07:02:18.027622+0000","last_unstale":"2024-09-17T07:09:46.181808+0000","last_undegraded":"2024-09-17T07:09:46.181808+0000","last_fullsized":"2024-09-17T07:09:46.181808+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:18.338863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:08:18.338863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0025905709999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"75'11","reported_seq":196,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.166500+0000","last_change":"2024-09-17T07:08:23.583577+0000","last_active":"2024-09-17T07:09:46.166500+0000","last_peered":"2024-09-17T07:09:46.166500+0000","last_clean":"2024-09-17T07:09:46.166500+0000","last_became_active":"2024-09-17T07:02:14.016255+0000","last_became_peered":"2024-09-17T07:02:14.016255+0000","last_unstale":"2024-09-17T07:09:46.166500+0000","last_undegraded":"2024-09-17T07:09:46.166500+0000","last_fullsized":"2024-09-17T07:09:46.166500+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-09-17T07:08:23.583408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:08:23.583408+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0056203690000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"94'2","reported_seq":184,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.173950+0000","last_change":"2024-09-17T07:09:18.169710+0000","last_active":"2024-09-17T07:09:46.173950+0000","last_peered":"2024-09-17T07:09:46.173950+0000","last_clean":"2024-09-17T07:09:46.173950+0000","last_became_active":"2024-09-17T07:02:12.011461+0000","last_became_peered":"2024-09-17T07:02:12.011461+0000","last_unstale":"2024-09-17T07:09:46.173950+0000","last_undegraded":"2024-09-17T07:09:46.173950+0000","last_fullsized":"2024-09-17T07:09:46.173950+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-09-17T07:09:18.169580+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-09-17T07:07:31.717888+0000","last_clean_scrub_stamp":"2024-09-17T07:09:18.169580+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:20.742759+0000","scrub_duration":0.011743155,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":183,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.174393+0000","last_change":"2024-09-17T07:09:24.407995+0000","last_active":"2024-09-17T07:09:46.174393+0000","last_peered":"2024-09-17T07:09:46.174393+0000","last_clean":"2024-09-17T07:09:46.174393+0000","last_became_active":"2024-09-17T07:09:24.407362+0000","last_became_peered":"2024-09-17T07:09:24.407362+0000","last_unstale":"2024-09-17T07:09:46.174393+0000","last_undegraded":"2024-09-17T07:09:46.174393+0000","last_fullsized":"2024-09-17T07:09:46.174393+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:27.841486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:08:27.841486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037672679999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":18,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.181448+0000","last_change":"2024-09-17T07:09:24.403561+0000","last_active":"2024-09-17T07:09:46.181448+0000","last_peered":"2024-09-17T07:09:46.181448+0000","last_clean":"2024-09-17T07:09:46.181448+0000","last_became_active":"2024-09-17T07:09:24.400791+0000","last_became_peered":"2024-09-17T07:09:24.400791+0000","last_unstale":"2024-09-17T07:09:46.181448+0000","last_undegraded":"2024-09-17T07:09:46.181448+0000","last_fullsized":"2024-09-17T07:09:46.181448+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:10.805704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:08:10.805704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"75'4","reported_seq":73,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.175667+0000","last_change":"2024-09-17T07:09:24.412549+0000","last_active":"2024-09-17T07:09:46.175667+0000","last_peered":"2024-09-17T07:09:46.175667+0000","last_clean":"2024-09-17T07:09:46.175667+0000","last_became_active":"2024-09-17T07:09:24.396368+0000","last_became_peered":"2024-09-17T07:09:24.396368+0000","last_unstale":"2024-09-17T07:09:46.175667+0000","last_undegraded":"2024-09-17T07:09:46.175667+0000","last_fullsized":"2024-09-17T07:09:46.175667+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'4","last_scrub_stamp":"2024-09-17T07:09:17.769705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:09:17.769705+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:19.586607+0000","scrub_duration":0.0015094500000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":178,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.171907+0000","last_change":"2024-09-17T07:08:04.972061+0000","last_active":"2024-09-17T07:09:46.171907+0000","last_peered":"2024-09-17T07:09:46.171907+0000","last_clean":"2024-09-17T07:09:46.171907+0000","last_became_active":"2024-09-17T07:02:12.008898+0000","last_became_peered":"2024-09-17T07:02:12.008898+0000","last_unstale":"2024-09-17T07:09:46.171907+0000","last_undegraded":"2024-09-17T07:09:46.171907+0000","last_fullsized":"2024-09-17T07:09:46.171907+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:04.971954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:08:04.971954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014792150000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":175,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.175751+0000","last_change":"2024-09-17T07:09:33.804039+0000","last_active":"2024-09-17T07:09:46.175751+0000","last_peered":"2024-09-17T07:09:46.175751+0000","last_clean":"2024-09-17T07:09:46.175751+0000","last_became_active":"2024-09-17T07:02:16.014982+0000","last_became_peered":"2024-09-17T07:02:16.014982+0000","last_unstale":"2024-09-17T07:09:46.175751+0000","last_undegraded":"2024-09-17T07:09:46.175751+0000","last_fullsized":"2024-09-17T07:09:46.175751+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:03.496602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:47.444972+0000","last_clean_scrub_stamp":"2024-09-17T07:08:03.496602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001400943,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":173,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.185409+0000","last_change":"2024-09-17T07:08:26.996645+0000","last_active":"2024-09-17T07:09:46.185409+0000","last_peered":"2024-09-17T07:09:46.185409+0000","last_clean":"2024-09-17T07:09:46.185409+0000","last_became_active":"2024-09-17T07:02:18.021014+0000","last_became_peered":"2024-09-17T07:02:18.021014+0000","last_unstale":"2024-09-17T07:09:46.185409+0000","last_undegraded":"2024-09-17T07:09:46.185409+0000","last_fullsized":"2024-09-17T07:09:46.185409+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:26.996525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:08:26.996525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001722168,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"75'11","reported_seq":62,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.185489+0000","last_change":"2024-09-17T07:09:24.428977+0000","last_active":"2024-09-17T07:09:46.185489+0000","last_peered":"2024-09-17T07:09:46.185489+0000","last_clean":"2024-09-17T07:09:46.185489+0000","last_became_active":"2024-09-17T07:09:24.404942+0000","last_became_peered":"2024-09-17T07:09:24.404942+0000","last_unstale":"2024-09-17T07:09:46.185489+0000","last_undegraded":"2024-09-17T07:09:46.185489+0000","last_fullsized":"2024-09-17T07:09:46.185489+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-09-17T07:08:25.883470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:08:25.883470+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":189,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.176528+0000","last_change":"2024-09-17T07:08:14.519328+0000","last_active":"2024-09-17T07:09:46.176528+0000","last_peered":"2024-09-17T07:09:46.176528+0000","last_clean":"2024-09-17T07:09:46.176528+0000","last_became_active":"2024-09-17T07:02:12.010195+0000","last_became_peered":"2024-09-17T07:02:12.010195+0000","last_unstale":"2024-09-17T07:09:46.176528+0000","last_undegraded":"2024-09-17T07:09:46.176528+0000","last_fullsized":"2024-09-17T07:09:46.176528+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:14.519233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:08:14.519233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001420021,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":175,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.187065+0000","last_change":"2024-09-17T07:08:32.894541+0000","last_active":"2024-09-17T07:09:46.187065+0000","last_peered":"2024-09-17T07:09:46.187065+0000","last_clean":"2024-09-17T07:09:46.187065+0000","last_became_active":"2024-09-17T07:02:16.016183+0000","last_became_peered":"2024-09-17T07:02:16.016183+0000","last_unstale":"2024-09-17T07:09:46.187065+0000","last_undegraded":"2024-09-17T07:09:46.187065+0000","last_fullsized":"2024-09-17T07:09:46.187065+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:32.894433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:08:32.894433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001707264,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":18,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.180435+0000","last_change":"2024-09-17T07:09:24.392552+0000","last_active":"2024-09-17T07:09:46.180435+0000","last_peered":"2024-09-17T07:09:46.180435+0000","last_clean":"2024-09-17T07:09:46.180435+0000","last_became_active":"2024-09-17T07:09:24.392216+0000","last_became_peered":"2024-09-17T07:09:24.392216+0000","last_unstale":"2024-09-17T07:09:46.180435+0000","last_undegraded":"2024-09-17T07:09:46.180435+0000","last_fullsized":"2024-09-17T07:09:46.180435+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:26.832514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:07:26.832514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1c","version":"75'1","reported_seq":174,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-17T07:09:51.207071+0000","last_change":"2024-09-17T07:08:22.083583+0000","last_active":"2024-09-17T07:09:51.207071+0000","last_peered":"2024-09-17T07:09:51.207071+0000","last_clean":"2024-09-17T07:09:51.207071+0000","last_became_active":"2024-09-17T07:02:18.021855+0000","last_became_peered":"2024-09-17T07:02:18.021855+0000","last_unstale":"2024-09-17T07:09:51.207071+0000","last_undegraded":"2024-09-17T07:09:51.207071+0000","last_fullsized":"2024-09-17T07:09:51.207071+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-17T07:08:22.083502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:08:22.083502+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0018653739999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":178,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.171631+0000","last_change":"2024-09-17T07:07:22.782091+0000","last_active":"2024-09-17T07:09:46.171631+0000","last_peered":"2024-09-17T07:09:46.171631+0000","last_clean":"2024-09-17T07:09:46.171631+0000","last_became_active":"2024-09-17T07:02:12.005329+0000","last_became_peered":"2024-09-17T07:02:12.005329+0000","last_unstale":"2024-09-17T07:09:46.171631+0000","last_undegraded":"2024-09-17T07:09:46.171631+0000","last_fullsized":"2024-09-17T07:09:46.171631+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:22.781739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:07:22.781739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037808680000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"75'10","reported_seq":213,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.182952+0000","last_change":"2024-09-17T07:09:24.430545+0000","last_active":"2024-09-17T07:09:46.182952+0000","last_peered":"2024-09-17T07:09:46.182952+0000","last_clean":"2024-09-17T07:09:46.182952+0000","last_became_active":"2024-09-17T07:09:24.404495+0000","last_became_peered":"2024-09-17T07:09:24.404495+0000","last_unstale":"2024-09-17T07:09:46.182952+0000","last_undegraded":"2024-09-17T07:09:46.182952+0000","last_fullsized":"2024-09-17T07:09:46.182952+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-09-17T07:09:13.341701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:09:13.341701+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:31.527633+0000","scrub_duration":0.024611266999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"75'8","reported_seq":278,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:50.606028+0000","last_change":"2024-09-17T07:08:24.422649+0000","last_active":"2024-09-17T07:09:50.606028+0000","last_peered":"2024-09-17T07:09:50.606028+0000","last_clean":"2024-09-17T07:09:50.606028+0000","last_became_active":"2024-09-17T07:02:16.020560+0000","last_became_peered":"2024-09-17T07:02:16.020560+0000","last_unstale":"2024-09-17T07:09:50.606028+0000","last_undegraded":"2024-09-17T07:09:50.606028+0000","last_fullsized":"2024-09-17T07:09:50.606028+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:08:24.422313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:08:24.422313+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0027600989999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":172,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:48.395889+0000","last_change":"2024-09-17T07:08:34.803052+0000","last_active":"2024-09-17T07:09:48.395889+0000","last_peered":"2024-09-17T07:09:48.395889+0000","last_clean":"2024-09-17T07:09:48.395889+0000","last_became_active":"2024-09-17T07:02:18.021603+0000","last_became_peered":"2024-09-17T07:02:18.021603+0000","last_unstale":"2024-09-17T07:09:48.395889+0000","last_undegraded":"2024-09-17T07:09:48.395889+0000","last_fullsized":"2024-09-17T07:09:48.395889+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:34.802938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:08:34.802938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0022680489999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":19,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.175904+0000","last_change":"2024-09-17T07:09:24.405301+0000","last_active":"2024-09-17T07:09:46.175904+0000","last_peered":"2024-09-17T07:09:46.175904+0000","last_clean":"2024-09-17T07:09:46.175904+0000","last_became_active":"2024-09-17T07:09:24.403360+0000","last_became_peered":"2024-09-17T07:09:24.403360+0000","last_unstale":"2024-09-17T07:09:46.175904+0000","last_undegraded":"2024-09-17T07:09:46.175904+0000","last_fullsized":"2024-09-17T07:09:46.175904+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:17.704687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:08:17.704687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"75'11","reported_seq":238,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.176479+0000","last_change":"2024-09-17T07:09:18.263884+0000","last_active":"2024-09-17T07:09:46.176479+0000","last_peered":"2024-09-17T07:09:46.176479+0000","last_clean":"2024-09-17T07:09:46.176479+0000","last_became_active":"2024-09-17T07:03:48.760390+0000","last_became_peered":"2024-09-17T07:03:48.760390+0000","last_unstale":"2024-09-17T07:09:46.176479+0000","last_undegraded":"2024-09-17T07:09:46.176479+0000","last_fullsized":"2024-09-17T07:09:46.176479+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-09-17T07:09:18.263404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:09:18.263404+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:39.849976+0000","scrub_duration":0.0065910480000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":175,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.182885+0000","last_change":"2024-09-17T07:09:10.293301+0000","last_active":"2024-09-17T07:09:46.182885+0000","last_peered":"2024-09-17T07:09:46.182885+0000","last_clean":"2024-09-17T07:09:46.182885+0000","last_became_active":"2024-09-17T07:02:16.019701+0000","last_became_peered":"2024-09-17T07:02:16.019701+0000","last_unstale":"2024-09-17T07:09:46.182885+0000","last_undegraded":"2024-09-17T07:09:46.182885+0000","last_fullsized":"2024-09-17T07:09:46.182885+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:10.292615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:09:10.292615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:40.263740+0000","scrub_duration":0.010112249,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":162,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.167494+0000","last_change":"2024-09-17T07:07:30.779462+0000","last_active":"2024-09-17T07:09:46.167494+0000","last_peered":"2024-09-17T07:09:46.167494+0000","last_clean":"2024-09-17T07:09:46.167494+0000","last_became_active":"2024-09-17T07:02:18.363752+0000","last_became_peered":"2024-09-17T07:02:18.363752+0000","last_unstale":"2024-09-17T07:09:46.167494+0000","last_undegraded":"2024-09-17T07:09:46.167494+0000","last_fullsized":"2024-09-17T07:09:46.167494+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:30.779382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:07:30.779382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0011291059999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":209,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.176083+0000","last_change":"2024-09-17T07:09:22.215844+0000","last_active":"2024-09-17T07:09:46.176083+0000","last_peered":"2024-09-17T07:09:46.176083+0000","last_clean":"2024-09-17T07:09:46.176083+0000","last_became_active":"2024-09-17T07:03:48.759384+0000","last_became_peered":"2024-09-17T07:03:48.759384+0000","last_unstale":"2024-09-17T07:09:46.176083+0000","last_undegraded":"2024-09-17T07:09:46.176083+0000","last_fullsized":"2024-09-17T07:09:46.176083+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:22.215729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:09:22.215729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:37.706145+0000","scrub_duration":0.0051243160000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"75'15","reported_seq":213,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.170799+0000","last_change":"2024-09-17T07:09:24.427421+0000","last_active":"2024-09-17T07:09:46.170799+0000","last_peered":"2024-09-17T07:09:46.170799+0000","last_clean":"2024-09-17T07:09:46.170799+0000","last_became_active":"2024-09-17T07:09:24.403030+0000","last_became_peered":"2024-09-17T07:09:24.403030+0000","last_unstale":"2024-09-17T07:09:46.170799+0000","last_undegraded":"2024-09-17T07:09:46.170799+0000","last_fullsized":"2024-09-17T07:09:46.170799+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-09-17T07:09:15.110940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:09:15.110940+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:21.125725+0000","scrub_duration":0.017595989999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":174,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:48.395910+0000","last_change":"2024-09-17T07:08:25.794548+0000","last_active":"2024-09-17T07:09:48.395910+0000","last_peered":"2024-09-17T07:09:48.395910+0000","last_clean":"2024-09-17T07:09:48.395910+0000","last_became_active":"2024-09-17T07:02:16.015903+0000","last_became_peered":"2024-09-17T07:02:16.015903+0000","last_unstale":"2024-09-17T07:09:48.395910+0000","last_undegraded":"2024-09-17T07:09:48.395910+0000","last_fullsized":"2024-09-17T07:09:48.395910+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:25.794418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:25.794418+0000","last_clean_scrub_stamp":"2024-09-17T07:08:25.794418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002916175,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":19,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.176426+0000","last_change":"2024-09-17T07:09:33.804712+0000","last_active":"2024-09-17T07:09:46.176426+0000","last_peered":"2024-09-17T07:09:46.176426+0000","last_clean":"2024-09-17T07:09:46.176426+0000","last_became_active":"2024-09-17T07:09:24.402411+0000","last_became_peered":"2024-09-17T07:09:24.402411+0000","last_unstale":"2024-09-17T07:09:46.176426+0000","last_undegraded":"2024-09-17T07:09:46.176426+0000","last_fullsized":"2024-09-17T07:09:46.176426+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:14.764781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:08:14.764781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":20,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.186024+0000","last_change":"2024-09-17T07:09:28.275884+0000","last_active":"2024-09-17T07:09:46.186024+0000","last_peered":"2024-09-17T07:09:46.186024+0000","last_clean":"2024-09-17T07:09:46.186024+0000","last_became_active":"2024-09-17T07:09:24.404795+0000","last_became_peered":"2024-09-17T07:09:24.404795+0000","last_unstale":"2024-09-17T07:09:46.186024+0000","last_undegraded":"2024-09-17T07:09:46.186024+0000","last_fullsized":"2024-09-17T07:09:46.186024+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:29.761001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:07:29.761001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"86'17","reported_seq":84,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:46.173831+0000","last_change":"2024-09-17T07:09:24.416689+0000","last_active":"2024-09-17T07:09:46.173831+0000","last_peered":"2024-09-17T07:09:46.173831+0000","last_clean":"2024-09-17T07:09:46.173831+0000","last_became_active":"2024-09-17T07:09:24.394476+0000","last_became_peered":"2024-09-17T07:09:24.394476+0000","last_unstale":"2024-09-17T07:09:46.173831+0000","last_undegraded":"2024-09-17T07:09:46.173831+0000","last_fullsized":"2024-09-17T07:09:46.173831+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'17","last_scrub_stamp":"2024-09-17T07:09:12.729002+0000","last_deep_scrub":"86'17","last_deep_scrub_stamp":"2024-09-17T07:03:18.241067+0000","last_clean_scrub_stamp":"2024-09-17T07:09:12.729002+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:13.772930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":184,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T07:09:48.093022+0000","last_change":"2024-09-17T07:09:24.393651+0000","last_active":"2024-09-17T07:09:48.093022+0000","last_peered":"2024-09-17T07:09:48.093022+0000","last_clean":"2024-09-17T07:09:48.093022+0000","last_became_active":"2024-09-17T07:09:24.393329+0000","last_became_peered":"2024-09-17T07:09:24.393329+0000","last_unstale":"2024-09-17T07:09:48.093022+0000","last_undegraded":"2024-09-17T07:09:48.093022+0000","last_fullsized":"2024-09-17T07:09:48.093022+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:14.706740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:08:14.706740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00164321,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":8005746712,"num_objects":122159,"num_object_clones":0,"num_object_copies":366477,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":32690,"num_objects_unfound":0,"num_objects_dirty":122159,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":138845,"num_write_kb":8352065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13378,"num_bytes_recovered":876740608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24899563520,"data_stored":24899551304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75998,"ondisk_log_size":75998,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69,"ondisk_log_size":69,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":692,"num_read_kb":449,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":2008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":13004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":427,"ondisk_log_size":427,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":454,"num_read_kb":449,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1386,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":12,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":420,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":4578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037858,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2788504,"kb_used_data":2472720,"kb_used_omap":1,"kb_used_meta":315774,"kb_avail":90956648,"statfs":{"total":95995035648,"available":93139607552,"internally_reserved":0,"allocated":2532065280,"data_stored":2531701672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":323353035},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:09:14 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":2.512,"15min":1.1879999999999999},"min":{"1min":0.20599999999999999,"5min":0.192,"15min":0.192},"max":{"1min":2.3039999999999998,"5min":52.478999999999999,"15min":52.478999999999999},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.95399999999999996,"5min":2.5110000000000001,"15min":1.1890000000000001},"min":{"1min":0.26200000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":2.677,"5min":52.871000000000002,"15min":52.871000000000002},"last":0.73899999999999999}]},{"osd":1,"last update":"Tue Sep 17 07:09:14 2024","interfaces":[{"interface":"back","average":{"1min":5.4059999999999997,"5min":1.7170000000000001,"15min":0.91100000000000003},"min":{"1min":0.27000000000000002,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":73.549000000000007,"5min":73.549000000000007,"15min":73.549000000000007},"last":0.29699999999999999},{"interface":"front","average":{"1min":5.375,"5min":1.7430000000000001,"15min":0.93000000000000005},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":73.116,"5min":73.116,"15min":73.116},"last":0.61299999999999999}]},{"osd":2,"last update":"Tue Sep 17 07:09:14 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":1.5660000000000001,"15min":0.84399999999999997},"min":{"1min":0.26600000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":2.6379999999999999,"5min":52.055999999999997,"15min":52.055999999999997},"last":0.433},{"interface":"front","average":{"1min":0.61799999999999999,"5min":1.613,"15min":0.89600000000000002},"min":{"1min":0.28299999999999997,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":2.4129999999999998,"5min":52.064999999999998,"15min":52.064999999999998},"last":0.98599999999999999}]},{"osd":3,"last update":"Tue Sep 17 07:09:14 2024","interfaces":[{"interface":"back","average":{"1min":1.8919999999999999,"5min":2.1480000000000001,"15min":1.0660000000000001},"min":{"1min":0.372,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":11.167,"5min":51.689,"15min":51.689},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.75600000000000001,"5min":2.3519999999999999,"15min":1.149},"min":{"1min":0.308,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":1.97,"5min":52.145000000000003,"15min":52.145000000000003},"last":1.2609999999999999}]},{"osd":4,"last update":"Tue Sep 17 07:09:14 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.50700000000000001,"15min":0.46800000000000003},"min":{"1min":0.29699999999999999,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":1.7290000000000001,"5min":1.7290000000000001,"15min":1.7290000000000001},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.50700000000000001,"15min":0.45700000000000002},"min":{"1min":0.26000000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.92100000000000004,"5min":3.4089999999999998,"15min":3.4089999999999998},"last":0.28000000000000003}]},{"osd":5,"last update":"Tue Sep 17 07:09:14 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.54800000000000004,"15min":0.47199999999999998},"min":{"1min":0.27800000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.635,"5min":3.3620000000000001,"15min":3.3620000000000001},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.55800000000000005,"15min":0.496},"min":{"1min":0.307,"5min":0.28999999999999998,"15min":0.28000000000000003},"max":{"1min":1.708,"5min":1.708,"15min":1.708},"last":0.63700000000000001}]},{"osd":6,"last update":"Tue Sep 17 07:09:14 2024","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":0.63400000000000001,"15min":0.54300000000000004},"min":{"1min":0.35099999999999998,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":1.6759999999999999,"5min":1.6759999999999999,"15min":1.6759999999999999},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.59299999999999997,"15min":0.52400000000000002},"min":{"1min":0.31,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.93899999999999995,"5min":3.2669999999999999,"15min":3.2669999999999999},"last":0.40699999999999997}]}]},{"osd":6,"up_from":50,"seq":214748364901,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2722476,"kb_used_data":2456672,"kb_used_omap":1,"kb_used_meta":265726,"kb_avail":91022676,"statfs":{"total":95995035648,"available":93207220224,"internally_reserved":0,"allocated":2515632128,"data_stored":2515252637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":272103879},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":2.9910000000000001,"15min":1.3280000000000001},"min":{"1min":0.27000000000000002,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":1.8109999999999999,"5min":75.323999999999998,"15min":75.323999999999998},"last":2.508},{"interface":"front","average":{"1min":1.0029999999999999,"5min":2.9319999999999999,"15min":1.304},"min":{"1min":0.25600000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":3.556,"5min":75.168999999999997,"15min":75.168999999999997},"last":2.2639999999999998}]},{"osd":1,"last update":"Tue Sep 17 07:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.71199999999999997,"15min":0.56200000000000006},"min":{"1min":0.308,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":2.6669999999999998,"5min":4.4790000000000001,"15min":4.4790000000000001},"last":0.312},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.81799999999999995,"15min":0.625},"min":{"1min":0.29999999999999999,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":1.849,"5min":5.9530000000000003,"15min":5.9530000000000003},"last":5.4930000000000003}]},{"osd":2,"last update":"Tue Sep 17 07:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":1.071,"15min":0.70199999999999996},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":1.8720000000000001,"5min":33.079000000000001,"15min":33.079000000000001},"last":5.1159999999999997},{"interface":"front","average":{"1min":0.64800000000000002,"5min":1.0600000000000001,"15min":0.69899999999999995},"min":{"1min":0.254,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":1.7589999999999999,"5min":33.100000000000001,"15min":33.100000000000001},"last":2.448}]},{"osd":3,"last update":"Tue Sep 17 07:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":1.98,"15min":0.97899999999999998},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":2.1560000000000001,"5min":52.944000000000003,"15min":52.944000000000003},"last":0.40899999999999997},{"interface":"front","average":{"1min":1.1140000000000001,"5min":1.9610000000000001,"15min":1.012},"min":{"1min":0.36399999999999999,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":3.339,"5min":52.747,"15min":52.747},"last":2.1600000000000001}]},{"osd":4,"last update":"Tue Sep 17 07:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.46100000000000002,"15min":0.433},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":2.1699999999999999,"5min":2.1699999999999999,"15min":2.1699999999999999},"last":2.7490000000000001},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.54500000000000004,"15min":0.45500000000000002},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":2.1299999999999999,"5min":2.9119999999999999,"15min":2.9119999999999999},"last":1.1539999999999999}]},{"osd":5,"last update":"Tue Sep 17 07:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.54600000000000004,"15min":0.46600000000000003},"min":{"1min":0.28299999999999997,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":4.9329999999999998,"5min":4.9329999999999998,"15min":4.9329999999999998},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.59499999999999997,"15min":0.48399999999999999},"min":{"1min":0.30499999999999999,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.859,"5min":3.0379999999999998,"15min":3.0379999999999998},"last":1.1739999999999999}]},{"osd":7,"last update":"Tue Sep 17 07:09:15 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.626,"15min":0.53800000000000003},"min":{"1min":0.29499999999999998,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":3.9399999999999999,"5min":3.9399999999999999,"15min":3.9399999999999999},"last":1.1220000000000001},{"interface":"front","average":{"1min":0.876,"5min":0.65700000000000003,"15min":0.56799999999999995},"min":{"1min":0.30099999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":3.1930000000000001,"5min":3.1930000000000001,"15min":3.1930000000000001},"last":0.45100000000000001}]}]},{"osd":5,"up_from":42,"seq":180388626538,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4404560,"kb_used_data":4120776,"kb_used_omap":1,"kb_used_meta":283774,"kb_avail":89340592,"statfs":{"total":95995035648,"available":91484766208,"internally_reserved":0,"allocated":4219674624,"data_stored":4219318904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":290585030},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,1,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:09:36 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":2.7970000000000002,"15min":1.272},"min":{"1min":0.251,"5min":0.221,"15min":0.221},"max":{"1min":4.577,"5min":49.265000000000001,"15min":49.265000000000001},"last":0.90900000000000003},{"interface":"front","average":{"1min":1.4399999999999999,"5min":2.9689999999999999,"15min":1.3220000000000001},"min":{"1min":0.39500000000000002,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":4.4260000000000002,"5min":47.350000000000001,"15min":47.350000000000001},"last":0.32100000000000001}]},{"osd":1,"last update":"Tue Sep 17 07:09:36 2024","interfaces":[{"interface":"back","average":{"1min":1.1879999999999999,"5min":0.95699999999999996,"15min":0.67500000000000004},"min":{"1min":0.23799999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":3.5600000000000001,"5min":20.329000000000001,"15min":20.329000000000001},"last":0.51200000000000001},{"interface":"front","average":{"1min":1.089,"5min":1.1279999999999999,"15min":1.0720000000000001},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":4.077,"5min":20.408999999999999,"15min":20.408999999999999},"last":0.34799999999999998}]},{"osd":2,"last update":"Tue Sep 17 07:09:36 2024","interfaces":[{"interface":"back","average":{"1min":1.8240000000000001,"5min":1.4079999999999999,"15min":0.83599999999999997},"min":{"1min":0.36499999999999999,"5min":0.252,"15min":0.252},"max":{"1min":6.242,"5min":20.486000000000001,"15min":20.486000000000001},"last":5.9509999999999996},{"interface":"front","average":{"1min":1.115,"5min":1.2629999999999999,"15min":0.75800000000000001},"min":{"1min":0.28899999999999998,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":3.2599999999999998,"5min":23.925999999999998,"15min":23.925999999999998},"last":7.1310000000000002}]},{"osd":3,"last update":"Tue Sep 17 07:09:36 2024","interfaces":[{"interface":"back","average":{"1min":1.5209999999999999,"5min":2.3279999999999998,"15min":1.135},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":6.5439999999999996,"5min":68.978999999999999,"15min":68.978999999999999},"last":2.2389999999999999},{"interface":"front","average":{"1min":1.3280000000000001,"5min":2.2130000000000001,"15min":1.0960000000000001},"min":{"1min":0.26800000000000002,"5min":0.253,"15min":0.253},"max":{"1min":4.5490000000000004,"5min":69.084000000000003,"15min":69.084000000000003},"last":6.4500000000000002}]},{"osd":4,"last update":"Tue Sep 17 07:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.53800000000000003,"15min":0.443},"min":{"1min":0.26800000000000002,"5min":0.22,"15min":0.22},"max":{"1min":4.0300000000000002,"5min":4.0300000000000002,"15min":4.0300000000000002},"last":0.374},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.55200000000000005,"15min":0.47099999999999997},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":2.6469999999999998,"5min":2.6469999999999998,"15min":2.6469999999999998},"last":0.78800000000000003}]},{"osd":6,"last update":"Tue Sep 17 07:09:50 2024","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.59699999999999998,"15min":0.48299999999999998},"min":{"1min":0.26100000000000001,"5min":0.23499999999999999,"15min":0.23200000000000001},"max":{"1min":3.1349999999999998,"5min":3.1349999999999998,"15min":3.1349999999999998},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.94599999999999995,"5min":0.67900000000000005,"15min":0.52100000000000002},"min":{"1min":0.314,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":3.4780000000000002,"5min":3.4780000000000002,"15min":3.4780000000000002},"last":1.1459999999999999}]},{"osd":7,"last update":"Tue Sep 17 07:09:11 2024","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.64100000000000001,"15min":0.53500000000000003},"min":{"1min":0.45600000000000002,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":2.6150000000000002,"5min":4.0999999999999996,"15min":4.0999999999999996},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.57699999999999996,"15min":0.502},"min":{"1min":0.34000000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":2.27,"5min":3.9100000000000001,"15min":3.9100000000000001},"last":0.54300000000000004}]}]},{"osd":4,"up_from":33,"seq":141733920880,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2662672,"kb_used_data":2491208,"kb_used_omap":1,"kb_used_meta":171390,"kb_avail":91082480,"statfs":{"total":95995035648,"available":93268459520,"internally_reserved":0,"allocated":2550996992,"data_stored":2550640348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":175503814},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,1,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:08:58 2024","interfaces":[{"interface":"back","average":{"1min":4.5389999999999997,"5min":4.0570000000000004,"15min":1.728},"min":{"1min":0.249,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":55.310000000000002,"5min":74.551000000000002,"15min":74.551000000000002},"last":0.76600000000000001},{"interface":"front","average":{"1min":4.8250000000000002,"5min":4.343,"15min":1.8500000000000001},"min":{"1min":0.26900000000000002,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":53.771000000000001,"5min":74.778999999999996,"15min":74.778999999999996},"last":0.32500000000000001}]},{"osd":1,"last update":"Tue Sep 17 07:08:58 2024","interfaces":[{"interface":"back","average":{"1min":4.9249999999999998,"5min":1.5620000000000001,"15min":0.89400000000000002},"min":{"1min":0.23599999999999999,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":54.914999999999999,"5min":54.914999999999999,"15min":54.914999999999999},"last":0.82999999999999996},{"interface":"front","average":{"1min":4.7320000000000002,"5min":1.5309999999999999,"15min":0.90000000000000002},"min":{"1min":0.26900000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":53.725000000000001,"5min":53.725000000000001,"15min":53.725000000000001},"last":0.996}]},{"osd":2,"last update":"Tue Sep 17 07:08:58 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":1.3169999999999999,"15min":0.82199999999999995},"min":{"1min":0.28499999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":4.9530000000000003,"5min":49.651000000000003,"15min":49.651000000000003},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.60999999999999999,"5min":1.3919999999999999,"15min":0.84999999999999998},"min":{"1min":0.307,"5min":0.247,"15min":0.247},"max":{"1min":2.2429999999999999,"5min":48.813000000000002,"15min":48.813000000000002},"last":0.625}]},{"osd":3,"last update":"Tue Sep 17 07:08:58 2024","interfaces":[{"interface":"back","average":{"1min":1.6040000000000001,"5min":1.4199999999999999,"15min":0.87},"min":{"1min":0.30599999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":17.872,"5min":39.042999999999999,"15min":39.042999999999999},"last":0.35099999999999998},{"interface":"front","average":{"1min":1.5529999999999999,"5min":1.5129999999999999,"15min":0.877},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":17.908000000000001,"5min":38.920000000000002,"15min":38.920000000000002},"last":0.497}]},{"osd":5,"last update":"Tue Sep 17 07:09:28 2024","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.59399999999999997,"15min":0.45700000000000002},"min":{"1min":0.27100000000000002,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":3.4390000000000001,"5min":3.4390000000000001,"15min":3.4390000000000001},"last":0.753},{"interface":"front","average":{"1min":0.93999999999999995,"5min":0.61599999999999999,"15min":0.48299999999999998},"min":{"1min":0.27500000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":3.0630000000000002,"5min":3.0630000000000002,"15min":3.0630000000000002},"last":0.58299999999999996}]},{"osd":6,"last update":"Tue Sep 17 07:09:51 2024","interfaces":[{"interface":"back","average":{"1min":1.056,"5min":1.1679999999999999,"15min":0.65800000000000003},"min":{"1min":0.31,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":3.714,"5min":47.923999999999999,"15min":47.923999999999999},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.78000000000000003,"5min":1.117,"15min":0.65100000000000002},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.26300000000000001},"max":{"1min":4.8440000000000003,"5min":47.600999999999999,"15min":47.600999999999999},"last":0.55900000000000005}]},{"osd":7,"last update":"Tue Sep 17 07:09:13 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":1.085,"15min":0.69499999999999995},"min":{"1min":0.31,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":2.677,"5min":48.719000000000001,"15min":48.719000000000001},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.998,"5min":1.155,"15min":0.70399999999999996},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":3.7480000000000002,"5min":50.527000000000001,"15min":50.527000000000001},"last":0.33600000000000002}]}]},{"osd":3,"up_from":0,"seq":115964117106,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":18,"seq":77309411448,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3781460,"kb_used_data":3381196,"kb_used_omap":1,"kb_used_meta":400254,"kb_avail":89963692,"statfs":{"total":95995035648,"available":92122820608,"internally_reserved":0,"allocated":3462344704,"data_stored":3461984832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":409860550},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,3,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:09:31 2024","interfaces":[{"interface":"back","average":{"1min":1.569,"5min":1.25,"15min":0.64600000000000002},"min":{"1min":0.193,"5min":0.193,"15min":0.17599999999999999},"max":{"1min":5.0259999999999998,"5min":7.2709999999999999,"15min":7.2709999999999999},"last":0.34999999999999998},{"interface":"front","average":{"1min":2.137,"5min":1.768,"15min":0.82499999999999996},"min":{"1min":0.219,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":6.8689999999999998,"5min":8.6869999999999994,"15min":8.6869999999999994},"last":0.84799999999999998}]},{"osd":1,"last update":"Tue Sep 17 07:09:31 2024","interfaces":[{"interface":"back","average":{"1min":1.7709999999999999,"5min":1.272,"15min":0.65800000000000003},"min":{"1min":0.36499999999999999,"5min":0.184,"15min":0.184},"max":{"1min":6.3499999999999996,"5min":6.3499999999999996,"15min":6.3499999999999996},"last":0.46999999999999997},{"interface":"front","average":{"1min":1.768,"5min":1.175,"15min":0.64200000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":5.2759999999999998,"5min":6.9829999999999997,"15min":6.9829999999999997},"last":0.46000000000000002}]},{"osd":3,"last update":"Tue Sep 17 07:09:49 2024","interfaces":[{"interface":"back","average":{"1min":2.5059999999999998,"5min":1.448,"15min":0.78000000000000003},"min":{"1min":0.29599999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":6.4809999999999999,"5min":6.4809999999999999,"15min":6.4809999999999999},"last":0.40699999999999997},{"interface":"front","average":{"1min":1.8939999999999999,"5min":1.7130000000000001,"15min":0.877},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":4.9029999999999996,"5min":6.0679999999999996,"15min":6.0679999999999996},"last":0.33600000000000002}]},{"osd":4,"last update":"Tue Sep 17 07:09:08 2024","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":0.90800000000000003,"15min":0.69099999999999995},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":2.7650000000000001,"5min":5.2359999999999998,"15min":5.2359999999999998},"last":0.50800000000000001},{"interface":"front","average":{"1min":1.4870000000000001,"5min":0.96099999999999997,"15min":0.70799999999999996},"min":{"1min":0.32500000000000001,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":6.4539999999999997,"5min":6.4539999999999997,"15min":6.4539999999999997},"last":0.38100000000000001}]},{"osd":5,"last update":"Tue Sep 17 07:09:31 2024","interfaces":[{"interface":"back","average":{"1min":1.794,"5min":1.268,"15min":0.81599999999999995},"min":{"1min":0.38700000000000001,"5min":0.249,"15min":0.249},"max":{"1min":4.9619999999999997,"5min":5.383,"15min":5.383},"last":0.88200000000000001},{"interface":"front","average":{"1min":1.6950000000000001,"5min":1.0980000000000001,"15min":0.753},"min":{"1min":0.32300000000000001,"5min":0.312,"15min":0.312},"max":{"1min":7.1790000000000003,"5min":7.1790000000000003,"15min":7.1790000000000003},"last":0.66600000000000004}]},{"osd":6,"last update":"Tue Sep 17 07:09:49 2024","interfaces":[{"interface":"back","average":{"1min":7.3819999999999997,"5min":2.29,"15min":1.1659999999999999},"min":{"1min":0.29999999999999999,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":101.587,"5min":101.587,"15min":101.587},"last":0.52000000000000002},{"interface":"front","average":{"1min":6.9850000000000003,"5min":2.3439999999999999,"15min":1.1970000000000001},"min":{"1min":0.30599999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":100.696,"5min":100.696,"15min":100.696},"last":0.90900000000000003}]},{"osd":7,"last update":"Tue Sep 17 07:09:21 2024","interfaces":[{"interface":"back","average":{"1min":1.2929999999999999,"5min":1.056,"15min":0.72899999999999998},"min":{"1min":0.33300000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":6.9450000000000003,"5min":6.9450000000000003,"15min":6.9450000000000003},"last":0.53800000000000003},{"interface":"front","average":{"1min":1.3700000000000001,"5min":1.147,"15min":0.88500000000000001},"min":{"1min":0.36899999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":4.1929999999999996,"5min":9.2919999999999998,"15min":9.2919999999999998},"last":0.61599999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574973,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3691956,"kb_used_data":3417836,"kb_used_omap":1,"kb_used_meta":273918,"kb_avail":90053196,"statfs":{"total":95995035648,"available":92214472704,"internally_reserved":0,"allocated":3499864064,"data_stored":3499472682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":280492486},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,1,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:08:54 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":1.4139999999999999,"15min":0.69499999999999995},"min":{"1min":0.20999999999999999,"5min":0.155,"15min":0.155},"max":{"1min":2.4710000000000001,"5min":13.837,"15min":13.837},"last":4.7389999999999999},{"interface":"front","average":{"1min":1.016,"5min":1.4399999999999999,"15min":0.72199999999999998},"min":{"1min":0.22900000000000001,"5min":0.186,"15min":0.17000000000000001},"max":{"1min":5.4119999999999999,"5min":10.342000000000001,"15min":10.342000000000001},"last":1.7250000000000001}]},{"osd":2,"last update":"Tue Sep 17 07:09:26 2024","interfaces":[{"interface":"back","average":{"1min":1.9259999999999999,"5min":1.5840000000000001,"15min":0.78900000000000003},"min":{"1min":0.20499999999999999,"5min":0.192,"15min":0.192},"max":{"1min":6.2130000000000001,"5min":11.055999999999999,"15min":11.055999999999999},"last":4.7969999999999997},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.514,"15min":0.77500000000000002},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":5.242,"5min":9.3160000000000007,"15min":9.3160000000000007},"last":1.7669999999999999}]},{"osd":3,"last update":"Tue Sep 17 07:08:49 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":1.3260000000000001,"15min":0.745},"min":{"1min":0.223,"5min":0.219,"15min":0.219},"max":{"1min":5.2800000000000002,"5min":7.766,"15min":7.766},"last":1.788},{"interface":"front","average":{"1min":1.079,"5min":1.4970000000000001,"15min":0.78700000000000003},"min":{"1min":0.26600000000000001,"5min":0.222,"15min":0.222},"max":{"1min":7.5629999999999997,"5min":19.602,"15min":19.602},"last":2.4710000000000001}]},{"osd":4,"last update":"Tue Sep 17 07:09:09 2024","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":1.1399999999999999,"15min":0.76300000000000001},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":5.3040000000000003,"5min":7.8840000000000003,"15min":7.8840000000000003},"last":1.6910000000000001},{"interface":"front","average":{"1min":2.9009999999999998,"5min":1.478,"15min":0.872},"min":{"1min":0.42299999999999999,"5min":0.307,"15min":0.307},"max":{"1min":27.097999999999999,"5min":27.097999999999999,"15min":27.097999999999999},"last":0.73699999999999999}]},{"osd":5,"last update":"Tue Sep 17 07:09:26 2024","interfaces":[{"interface":"back","average":{"1min":1.246,"5min":1.177,"15min":0.78900000000000003},"min":{"1min":0.33400000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":5.3470000000000004,"5min":5.617,"15min":6.0060000000000002},"last":1.8120000000000001},{"interface":"front","average":{"1min":2.141,"5min":1.4179999999999999,"15min":0.84599999999999997},"min":{"1min":0.376,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":8.1920000000000002,"5min":8.8889999999999993,"15min":8.8889999999999993},"last":2.831}]},{"osd":6,"last update":"Tue Sep 17 07:08:49 2024","interfaces":[{"interface":"back","average":{"1min":1.3720000000000001,"5min":1.2809999999999999,"15min":0.78900000000000003},"min":{"1min":0.443,"5min":0.36599999999999999,"15min":0.36499999999999999},"max":{"1min":8.1219999999999999,"5min":8.843,"15min":8.843},"last":2.516},{"interface":"front","average":{"1min":0.85599999999999998,"5min":1.1279999999999999,"15min":0.77100000000000002},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":5.3250000000000002,"5min":7.1130000000000004,"15min":7.1130000000000004},"last":1.7470000000000001}]},{"osd":7,"last update":"Tue Sep 17 07:09:12 2024","interfaces":[{"interface":"back","average":{"1min":2.2189999999999999,"5min":1.4159999999999999,"15min":0.84599999999999997},"min":{"1min":0.47399999999999998,"5min":0.37,"15min":0.37},"max":{"1min":8.1660000000000004,"5min":8.9309999999999992,"15min":8.9309999999999992},"last":2.7090000000000001},{"interface":"front","average":{"1min":1.325,"5min":1.2030000000000001,"15min":0.77900000000000003},"min":{"1min":0.34100000000000003,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":4.6369999999999996,"5min":11.007999999999999,"15min":11.007999999999999},"last":4.3659999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738496,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3539672,"kb_used_data":3157456,"kb_used_omap":1,"kb_used_meta":382206,"kb_avail":90205480,"statfs":{"total":95995035648,"available":92370411520,"internally_reserved":0,"allocated":3233234944,"data_stored":3232871476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":391379398},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 17 07:08:57 2024","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":1.2,"15min":0.624},"min":{"1min":0.19900000000000001,"5min":0.183,"15min":0.183},"max":{"1min":2.8879999999999999,"5min":8.0259999999999998,"15min":8.0259999999999998},"last":6.6200000000000001},{"interface":"front","average":{"1min":1.1799999999999999,"5min":1.353,"15min":0.68500000000000005},"min":{"1min":0.17799999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":2.8410000000000002,"5min":7.8849999999999998,"15min":7.8849999999999998},"last":2.5870000000000002}]},{"osd":2,"last update":"Tue Sep 17 07:09:24 2024","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":2.0459999999999998,"15min":0.93400000000000005},"min":{"1min":0.255,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":3.0939999999999999,"5min":18.643999999999998,"15min":18.643999999999998},"last":2.5579999999999998},{"interface":"front","average":{"1min":1.0009999999999999,"5min":1.8160000000000001,"15min":0.84799999999999998},"min":{"1min":0.27600000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":5.8440000000000003,"5min":17.972999999999999,"15min":17.972999999999999},"last":2.7330000000000001}]},{"osd":3,"last update":"Tue Sep 17 07:08:57 2024","interfaces":[{"interface":"back","average":{"1min":1.0189999999999999,"5min":1.7410000000000001,"15min":0.86899999999999999},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":3.7599999999999998,"5min":18.629000000000001,"15min":18.629000000000001},"last":2.7639999999999998},{"interface":"front","average":{"1min":0.71599999999999997,"5min":1.8460000000000001,"15min":0.90200000000000002},"min":{"1min":0.24299999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.9039999999999999,"5min":22.358000000000001,"15min":22.358000000000001},"last":1.028}]},{"osd":4,"last update":"Tue Sep 17 07:09:06 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":1.4239999999999999,"15min":0.88800000000000001},"min":{"1min":0.317,"5min":0.316,"15min":0.316},"max":{"1min":2.1520000000000001,"5min":18.568000000000001,"15min":18.568000000000001},"last":0.442},{"interface":"front","average":{"1min":0.88400000000000001,"5min":1.3700000000000001,"15min":0.82299999999999995},"min":{"1min":0.33300000000000002,"5min":0.255,"15min":0.255},"max":{"1min":1.514,"5min":18.765000000000001,"15min":18.765000000000001},"last":0.49199999999999999}]},{"osd":5,"last update":"Tue Sep 17 07:09:36 2024","interfaces":[{"interface":"back","average":{"1min":1.1459999999999999,"5min":1.5600000000000001,"15min":0.88500000000000001},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":5.2229999999999999,"5min":18.614999999999998,"15min":18.614999999999998},"last":0.41199999999999998},{"interface":"front","average":{"1min":1.0640000000000001,"5min":1.5329999999999999,"15min":0.871},"min":{"1min":0.30199999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":2.923,"5min":18.791,"15min":18.791},"last":0.51600000000000001}]},{"osd":6,"last update":"Tue Sep 17 07:08:57 2024","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":1.3879999999999999,"15min":0.82799999999999996},"min":{"1min":0.33100000000000002,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":2.6629999999999998,"5min":18.602,"15min":18.602},"last":1.117},{"interface":"front","average":{"1min":0.65000000000000002,"5min":1.5109999999999999,"15min":0.871},"min":{"1min":0.379,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.978,"5min":18.692,"15min":18.692},"last":0.46100000000000002}]},{"osd":7,"last update":"Tue Sep 17 07:09:16 2024","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":1.9390000000000001,"15min":1.409},"min":{"1min":0.32900000000000001,"5min":0.311,"15min":0.311},"max":{"1min":2.8500000000000001,"5min":46.183,"15min":46.183},"last":0.53400000000000003},{"interface":"front","average":{"1min":1.008,"5min":1.859,"15min":0.99299999999999999},"min":{"1min":0.33600000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":2.7370000000000001,"5min":45.914999999999999,"15min":45.914999999999999},"last":1.1519999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":2448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3215785984,"data_stored":3215785984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3490316288,"data_stored":3490316288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3444117504,"data_stored":3444113432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3014918144,"data_stored":3014918144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2536505344,"data_stored":2536505344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4187885568,"data_stored":4187881496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2494038016,"data_stored":2494038016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2515996672,"data_stored":2515992600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T07:10:01.272 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd pool set .mgr pgp_num 1 2024-09-17T07:10:01.313 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:10:01.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:01.366 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:01.366 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:10:01.467 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:10:01.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:01.519 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:01.519 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:10:01.620 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:10:01.661 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:01.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:01.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:10:01.763 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:10:01.788 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:01.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:01.789 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:10:01.829 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-17T07:10:01.830 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:10:01.890 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:10:01.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:01.933 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:01.933 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:10:02.034 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:10:02.068 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:02.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:02.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:10:02.170 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:10:02.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:02.214 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:02.214 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:10:02.315 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:10:02.337 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:02.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:02.339 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:10:02.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:02 smithi100 ceph-mon[29092]: pgmap v344: 148 pgs: 5 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 142 active+clean; 7.1 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.14k op/s; 28335/349635 objects misplaced (8.104%); 32 MiB/s, 515 objects/s recovering 2024-09-17T07:10:02.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:02 smithi100 ceph-mon[39462]: pgmap v344: 148 pgs: 5 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 142 active+clean; 7.1 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.14k op/s; 28335/349635 objects misplaced (8.104%); 32 MiB/s, 515 objects/s recovering 2024-09-17T07:10:02.440 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:10:02.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:02.470 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:02.471 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:10:02.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:02 smithi135 ceph-mon[33238]: pgmap v344: 148 pgs: 5 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 142 active+clean; 7.1 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.14k op/s; 28335/349635 objects misplaced (8.104%); 32 MiB/s, 515 objects/s recovering 2024-09-17T07:10:02.571 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:10:02.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:02.610 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:02.611 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:10:02.711 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:10:02.746 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:02.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:02.748 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:10:02.848 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:10:02.889 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:02.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:02.890 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:10:02.990 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:10:03.013 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:03.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:03.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:10:03.115 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:10:03.156 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:03.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:03.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:10:03.258 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:10:03.288 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:03.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:03.289 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:10:03.390 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:10:03.425 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:03.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:03.426 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:10:03.527 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:10:03.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:03 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:10:03.565 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:03.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:03.566 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:10:03.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:03 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:10:03.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:03 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:10:03.667 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:10:03.698 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:03.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:03.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:10:03.800 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:10:03.837 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:03.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:03.838 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:10:03.939 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:10:03.972 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:03.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:03.973 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:10:04.074 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:10:04.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:04.096 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:04.096 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:10:04.197 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:10:04.233 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:04.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:04.234 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:10:04.335 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:10:04.358 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:04.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:04.359 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:10:04.460 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:10:04.500 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:04.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:04.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:10:04.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:04 smithi135 ceph-mon[33238]: pgmap v345: 148 pgs: 5 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 142 active+clean; 7.0 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.13k op/s; 25525/343344 objects misplaced (7.434%); 38 MiB/s, 609 objects/s recovering 2024-09-17T07:10:04.601 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:10:04.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:04 smithi100 ceph-mon[29092]: pgmap v345: 148 pgs: 5 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 142 active+clean; 7.0 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.13k op/s; 25525/343344 objects misplaced (7.434%); 38 MiB/s, 609 objects/s recovering 2024-09-17T07:10:04.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:04 smithi100 ceph-mon[39462]: pgmap v345: 148 pgs: 5 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 142 active+clean; 7.0 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.13k op/s; 25525/343344 objects misplaced (7.434%); 38 MiB/s, 609 objects/s recovering 2024-09-17T07:10:04.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:04.628 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:04.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:10:04.729 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:10:04.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:04.767 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:04.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:10:04.868 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:10:04.900 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:04.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:04.902 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:10:05.002 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:10:05.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:05.047 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:05.047 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:10:05.148 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:10:05.182 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:05.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:05.183 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:10:05.283 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:10:05.318 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:05.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:05.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:10:05.420 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:10:05.451 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:05.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:05.452 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:10:05.553 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:10:05.575 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:05.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:05.576 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:10:05.677 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:10:05.713 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:05.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:05.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:10:05.815 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:10:05.848 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:05.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:05.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:10:05.950 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:10:05.989 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:05.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:05.990 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:10:06.091 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:10:06.128 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:06.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:06.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:10:06.147 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:10:05 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:10:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:10:06.230 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:10:06.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:06.267 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:06.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:10:06.368 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:10:06.400 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:06.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:06.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:10:06.502 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:10:06.527 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:06 smithi100 ceph-mon[29092]: pgmap v346: 148 pgs: 5 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 142 active+clean; 6.8 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.14k op/s; 23776/332769 objects misplaced (7.145%); 39 MiB/s, 616 objects/s recovering 2024-09-17T07:10:06.529 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:06 smithi100 ceph-mon[39462]: pgmap v346: 148 pgs: 5 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 142 active+clean; 6.8 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.14k op/s; 23776/332769 objects misplaced (7.145%); 39 MiB/s, 616 objects/s recovering 2024-09-17T07:10:06.533 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:06.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:06.534 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:10:06.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:06 smithi135 ceph-mon[33238]: pgmap v346: 148 pgs: 5 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 142 active+clean; 6.8 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.14k op/s; 23776/332769 objects misplaced (7.145%); 39 MiB/s, 616 objects/s recovering 2024-09-17T07:10:06.635 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:10:06.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:06.662 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:06.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:10:06.730 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:10:06.735 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:10:06.763 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:10:06.806 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:06.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:06.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:10:06.908 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:10:06.945 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:06.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:06.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:10:07.047 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:10:07.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:07.079 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:07.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:10:07.180 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:10:07.213 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:07.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:07.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:10:07.315 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:10:07.337 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:07.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:07.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:10:07.439 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:10:07.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:07.480 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:07.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:10:07.581 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:10:07.607 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:07.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:07.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:10:07.709 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:10:07.745 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:07.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:07.746 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:10:07.847 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:10:07.886 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:07.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:07.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:10:07.988 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:10:08.010 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:08.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:08.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:10:08.112 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:10:08.145 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:08.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:08.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:10:08.247 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:10:08.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:08.288 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:08.288 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:10:08.389 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:10:08.421 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:08.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:08.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:10:08.522 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:10:08.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:08 smithi135 ceph-mon[33238]: pgmap v347: 148 pgs: 5 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 142 active+clean; 6.8 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 855 B/s rd, 0 B/s wr, 955 op/s; 23776/332769 objects misplaced (7.145%); 32 MiB/s, 514 objects/s recovering 2024-09-17T07:10:08.543 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:10:08 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:10:08.561 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:08.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:08.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:10:08.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:08 smithi100 ceph-mon[29092]: pgmap v347: 148 pgs: 5 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 142 active+clean; 6.8 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 855 B/s rd, 0 B/s wr, 955 op/s; 23776/332769 objects misplaced (7.145%); 32 MiB/s, 514 objects/s recovering 2024-09-17T07:10:08.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:08 smithi100 ceph-mon[39462]: pgmap v347: 148 pgs: 5 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 142 active+clean; 6.8 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 855 B/s rd, 0 B/s wr, 955 op/s; 23776/332769 objects misplaced (7.145%); 32 MiB/s, 514 objects/s recovering 2024-09-17T07:10:08.663 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:10:08.699 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:08.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:08.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:10:08.801 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:10:08.835 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:08.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:08.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:10:08.937 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:10:08.977 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:08.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:08.978 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:10:09.079 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:10:09.111 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:09.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:09.112 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:10:09.212 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-17T07:10:09.213 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:10:09.254 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:09.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:09.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:10:09.356 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:10:09.403 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:09.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:09.404 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:10:09.505 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:10:09.544 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:09.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:09.546 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:10:09.647 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:10:09.688 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:09.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:09.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:10:09.790 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:10:09.829 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:09.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:09.830 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:10:09.931 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:10:09.972 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:09.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:09.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:10:10.055 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:10:10.074 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:10:10.107 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:10.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:10.108 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:10:10.209 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:10:10.242 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:10.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:10.243 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:10:10.327 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:10 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:10:10.327 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:10 smithi100 ceph-mon[29092]: pgmap v348: 148 pgs: 5 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 142 active+clean; 6.4 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s; 19182/317187 objects misplaced (6.048%); 51 MiB/s, 809 objects/s recovering 2024-09-17T07:10:10.328 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:10 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:10:10.328 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:10 smithi100 ceph-mon[39462]: pgmap v348: 148 pgs: 5 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 142 active+clean; 6.4 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s; 19182/317187 objects misplaced (6.048%); 51 MiB/s, 809 objects/s recovering 2024-09-17T07:10:10.344 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:10:10.377 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:10.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:10.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:10:10.479 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:10:10.506 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:10 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:10:10.506 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:10 smithi135 ceph-mon[33238]: pgmap v348: 148 pgs: 5 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 142 active+clean; 6.4 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s; 19182/317187 objects misplaced (6.048%); 51 MiB/s, 809 objects/s recovering 2024-09-17T07:10:10.518 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:10.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:10.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:10:10.620 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:10:10.649 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:10.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:10.650 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:10:10.751 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:10:10.785 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:10.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:10.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:10:10.887 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:10:10.923 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:10.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:10.924 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:10:11.025 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:10:11.054 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:11.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:11.055 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:10:11.156 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:10:11.192 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:11.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:11.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:10:11.294 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:10:11.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:11.316 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:11.316 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:10:11.417 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:10:11.453 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:11.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:11.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:10:11.555 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:10:11.586 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:11.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:11.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:10:11.688 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:10:11.731 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:11.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:11.732 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:10:11.833 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:10:11.871 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:11.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:11.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:10:11.973 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:10:12.011 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:12.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:12.013 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:10:12.113 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:10:12.136 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:12.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:12.137 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:10:12.232 INFO:teuthology.orchestra.run.smithi100.stderr:noscrub is unset 2024-09-17T07:10:12.238 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:10:12.267 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:12.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:12.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:10:12.369 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:10:12.404 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:12.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:12.405 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:10:12.506 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:10:12.526 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:12.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:12.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:10:12.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:12 smithi135 ceph-mon[33238]: pgmap v349: 148 pgs: 5 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 142 active+clean; 6.4 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 901 op/s; 19182/317187 objects misplaced (6.048%); 36 MiB/s, 572 objects/s recovering 2024-09-17T07:10:12.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:12 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1463935268' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:10:12.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:12 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:10:12.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:12 smithi100 ceph-mon[29092]: pgmap v349: 148 pgs: 5 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 142 active+clean; 6.4 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 901 op/s; 19182/317187 objects misplaced (6.048%); 36 MiB/s, 572 objects/s recovering 2024-09-17T07:10:12.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:12 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1463935268' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:10:12.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:12 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:10:12.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:12 smithi100 ceph-mon[39462]: pgmap v349: 148 pgs: 5 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 142 active+clean; 6.4 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 901 op/s; 19182/317187 objects misplaced (6.048%); 36 MiB/s, 572 objects/s recovering 2024-09-17T07:10:12.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:12 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1463935268' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:10:12.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:12 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:10:12.628 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:10:12.661 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:12.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:12.662 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:10:12.763 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:10:12.797 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:12.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:12.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:10:12.899 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:10:12.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:12.938 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:12.938 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:10:13.039 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:10:13.076 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:13.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:13.077 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:10:13.178 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:10:13.207 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:13.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:13.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:10:13.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:13 smithi135 ceph-mon[33238]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:10:13.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:13 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T07:10:13.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:13 smithi135 ceph-mon[33238]: osdmap e203: 8 total, 8 up, 7 in 2024-09-17T07:10:13.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:13 smithi135 ceph-mon[33238]: 2.1b scrub starts 2024-09-17T07:10:13.309 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:10:13.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:13.350 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:13.350 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:10:13.373 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:13 smithi100 ceph-mon[29092]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:10:13.373 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:13 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T07:10:13.373 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:13 smithi100 ceph-mon[29092]: osdmap e203: 8 total, 8 up, 7 in 2024-09-17T07:10:13.373 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:13 smithi100 ceph-mon[29092]: 2.1b scrub starts 2024-09-17T07:10:13.374 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:13 smithi100 ceph-mon[39462]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:10:13.374 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:13 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T07:10:13.374 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:13 smithi100 ceph-mon[39462]: osdmap e203: 8 total, 8 up, 7 in 2024-09-17T07:10:13.374 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:13 smithi100 ceph-mon[39462]: 2.1b scrub starts 2024-09-17T07:10:13.452 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:10:13.496 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:13.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:13.498 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:10:13.598 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:10:13.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:13.633 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:13.634 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:10:13.735 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:10:13.768 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:13.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:13.770 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:10:13.870 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:10:13.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:13.905 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:13.905 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:10:13.925 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:10:13.926 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:10:13.959 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1: {} 2024-09-17T07:10:13.959 INFO:teuthology.orchestra.run.smithi100.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:10:13.992 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:10:13.993 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:10:14.006 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:10:14.030 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:10:14.030 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:10:14.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:14.043 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:14.043 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:10:14.062 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:10:14.062 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:10:14.099 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:10:14.099 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:10:14.122 INFO:teuthology.orchestra.run.smithi135.stdout:{ 2024-09-17T07:10:14.123 INFO:teuthology.orchestra.run.smithi135.stdout: "ops": [ 2024-09-17T07:10:14.123 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:14.123 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.24929.0:161721 8.2 8:48c9c0af:::benchmark_data_smithi100_102803_object31217:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e203)", 2024-09-17T07:10:14.123 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:10:14.115295+0000", 2024-09-17T07:10:14.123 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 0.0059999629999999997, 2024-09-17T07:10:14.123 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0060312930000000001, 2024-09-17T07:10:14.123 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:10:14.123 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "waiting for sub ops", 2024-09-17T07:10:14.124 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:10:14.124 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.24929", 2024-09-17T07:10:14.124 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/564515582", 2024-09-17T07:10:14.124 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 161721 2024-09-17T07:10:14.124 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:14.124 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:10:14.124 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:14.124 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:10:14.124 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:14.115295+0000", 2024-09-17T07:10:14.124 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:10:14.124 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:14.124 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:14.125 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:10:14.125 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:14.115295+0000", 2024-09-17T07:10:14.125 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:10:14.125 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:14.125 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:14.125 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:10:14.125 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:14.115294+0000", 2024-09-17T07:10:14.125 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:10:14.126 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:14.126 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:14.126 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:10:14.126 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:14.115300+0000", 2024-09-17T07:10:14.126 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.6109999999999999e-06 2024-09-17T07:10:14.126 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:14.126 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:14.126 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:10:14.126 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:14.115312+0000", 2024-09-17T07:10:14.126 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.1463e-05 2024-09-17T07:10:14.127 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:14.127 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:14.127 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:10:14.127 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:14.115319+0000", 2024-09-17T07:10:14.127 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.4889999999999998e-06 2024-09-17T07:10:14.127 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:14.127 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:14.127 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:10:14.127 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:14.115370+0000", 2024-09-17T07:10:14.127 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.0725e-05 2024-09-17T07:10:14.127 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:14.127 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:14.128 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:10:14.128 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:14.115505+0000", 2024-09-17T07:10:14.128 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000134728 2024-09-17T07:10:14.128 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:14.128 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:14.128 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,5", 2024-09-17T07:10:14.128 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:14.115610+0000", 2024-09-17T07:10:14.128 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000105611 2024-09-17T07:10:14.128 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:14.128 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:14.128 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:10:14.128 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:14.118832+0000", 2024-09-17T07:10:14.129 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0032214840000000002 2024-09-17T07:10:14.129 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:14.129 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:14.129 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:10:14.129 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:14.120169+0000", 2024-09-17T07:10:14.129 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0013368570000000001 2024-09-17T07:10:14.129 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:10:14.129 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:10:14.129 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:10:14.129 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:10:14.129 INFO:teuthology.orchestra.run.smithi135.stdout: ], 2024-09-17T07:10:14.130 INFO:teuthology.orchestra.run.smithi135.stdout: "num_ops": 1 2024-09-17T07:10:14.130 INFO:teuthology.orchestra.run.smithi135.stdout:} 2024-09-17T07:10:14.132 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:10:14.132 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:10:14.144 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:10:14.170 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:10:14.170 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:10:14.180 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:14.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:14.181 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:10:14.282 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:10:14.323 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:14.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:14.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:10:14.425 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:10:14.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:14.458 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:14.458 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:10:14.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:14 smithi135 ceph-mon[33238]: pgmap v351: 148 pgs: 5 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 142 active+clean; 6.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.08k op/s; 16920/310860 objects misplaced (5.443%); 41 MiB/s, 657 objects/s recovering 2024-09-17T07:10:14.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:14 smithi135 ceph-mon[33238]: 6.1f scrub starts 2024-09-17T07:10:14.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:14 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2248499035' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:10:14.559 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:10:14.582 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:14.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:14.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:10:14.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:14 smithi100 ceph-mon[29092]: pgmap v351: 148 pgs: 5 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 142 active+clean; 6.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.08k op/s; 16920/310860 objects misplaced (5.443%); 41 MiB/s, 657 objects/s recovering 2024-09-17T07:10:14.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:14 smithi100 ceph-mon[29092]: 6.1f scrub starts 2024-09-17T07:10:14.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:14 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2248499035' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:10:14.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:14 smithi100 ceph-mon[39462]: pgmap v351: 148 pgs: 5 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 142 active+clean; 6.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.08k op/s; 16920/310860 objects misplaced (5.443%); 41 MiB/s, 657 objects/s recovering 2024-09-17T07:10:14.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:14 smithi100 ceph-mon[39462]: 6.1f scrub starts 2024-09-17T07:10:14.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:14 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2248499035' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:10:14.684 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:10:14.717 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:14.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:14.718 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:10:14.819 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:10:14.846 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:14.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:14.847 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:10:14.948 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:10:14.981 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:14.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:14.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:10:15.083 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:10:15.119 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:15.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:15.120 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:10:15.221 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:10:15.243 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:15.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:15.245 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:10:15.346 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:10:15.382 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:15.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:15.384 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:10:15.485 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:10:15.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:15.517 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:15.517 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:10:15.618 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:10:15.654 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:15.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:15.655 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:10:15.756 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:10:15.792 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:15.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:15.793 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:10:15.826 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-17T07:10:15.894 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:10:15.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:15.932 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:15.932 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:10:15.959 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:10:15 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:10:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:10:16.033 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:10:16.076 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:16.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:16.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:10:16.178 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:10:16.213 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:16.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:16.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:10:16.298 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:10:16.316 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:10:16.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:16.352 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:16.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:10:16.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:16 smithi100 ceph-mon[29092]: pgmap v352: 148 pgs: 1 active+clean+scrubbing, 5 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 141 active+clean; 6.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.07k op/s; 15136/300636 objects misplaced (5.035%); 42 MiB/s, 667 objects/s recovering 2024-09-17T07:10:16.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:16 smithi100 ceph-mon[29092]: 5.11 scrub starts 2024-09-17T07:10:16.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:16 smithi100 ceph-mon[29092]: 5.14 scrub starts 2024-09-17T07:10:16.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:16 smithi100 ceph-mon[39462]: pgmap v352: 148 pgs: 1 active+clean+scrubbing, 5 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 141 active+clean; 6.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.07k op/s; 15136/300636 objects misplaced (5.035%); 42 MiB/s, 667 objects/s recovering 2024-09-17T07:10:16.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:16 smithi100 ceph-mon[39462]: 5.11 scrub starts 2024-09-17T07:10:16.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:16 smithi100 ceph-mon[39462]: 5.14 scrub starts 2024-09-17T07:10:16.431 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:16 smithi135 ceph-mon[33238]: pgmap v352: 148 pgs: 1 active+clean+scrubbing, 5 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 141 active+clean; 6.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.07k op/s; 15136/300636 objects misplaced (5.035%); 42 MiB/s, 667 objects/s recovering 2024-09-17T07:10:16.431 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:16 smithi135 ceph-mon[33238]: 5.11 scrub starts 2024-09-17T07:10:16.431 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:16 smithi135 ceph-mon[33238]: 5.14 scrub starts 2024-09-17T07:10:16.454 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:10:16.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:16.492 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:16.492 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:10:16.593 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:10:16.635 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:16.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:16.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:10:16.737 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:10:16.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:16.774 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:16.774 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:10:16.875 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:10:16.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:16.902 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:16.902 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:10:17.003 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:10:17.031 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:17.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:17.032 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:10:17.133 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:10:17.174 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:17.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:17.175 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:10:17.276 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:10:17.305 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:17.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:17.306 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:10:17.407 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:10:17.434 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:17.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:17.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:10:17.536 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:10:17.561 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:17.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:17.562 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:10:17.663 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:10:17.690 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:17.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:17.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:10:17.793 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:10:17.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:17.817 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:17.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:10:17.918 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:10:17.942 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:17.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:17.943 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:10:18.000 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:10:18.044 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:10:18.065 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:18.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:18.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:10:18.167 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:10:18.168 INFO:teuthology.orchestra.run.smithi100.stderr:set pool 1 pgp_num to 1 2024-09-17T07:10:18.195 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:18.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:18.196 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:10:18.296 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:10:18.333 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:18.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:18.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:10:18.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:18 smithi100 ceph-mon[29092]: osdmap e204: 8 total, 8 up, 7 in 2024-09-17T07:10:18.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:18 smithi100 ceph-mon[29092]: pgmap v354: 148 pgs: 1 active+clean+scrubbing, 5 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 141 active+clean; 6.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 689 op/s; 15136/300636 objects misplaced (5.035%); 24 MiB/s, 389 objects/s recovering 2024-09-17T07:10:18.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:18 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/3130894516' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-09-17T07:10:18.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:18 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:10:18.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:18 smithi100 ceph-mon[29092]: 6.1b scrub starts 2024-09-17T07:10:18.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:18 smithi100 ceph-mon[39462]: osdmap e204: 8 total, 8 up, 7 in 2024-09-17T07:10:18.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:18 smithi100 ceph-mon[39462]: pgmap v354: 148 pgs: 1 active+clean+scrubbing, 5 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 141 active+clean; 6.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 689 op/s; 15136/300636 objects misplaced (5.035%); 24 MiB/s, 389 objects/s recovering 2024-09-17T07:10:18.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:18 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/3130894516' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-09-17T07:10:18.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:18 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:10:18.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:18 smithi100 ceph-mon[39462]: 6.1b scrub starts 2024-09-17T07:10:18.435 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:10:18.480 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:18 smithi135 ceph-mon[33238]: osdmap e204: 8 total, 8 up, 7 in 2024-09-17T07:10:18.480 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:18 smithi135 ceph-mon[33238]: pgmap v354: 148 pgs: 1 active+clean+scrubbing, 5 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 141 active+clean; 6.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 689 op/s; 15136/300636 objects misplaced (5.035%); 24 MiB/s, 389 objects/s recovering 2024-09-17T07:10:18.481 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:18 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/3130894516' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-09-17T07:10:18.481 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:18 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:10:18.481 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:18 smithi135 ceph-mon[33238]: 6.1b scrub starts 2024-09-17T07:10:18.483 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:18.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:18.484 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:10:18.584 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:10:18.620 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:18.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:18.621 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:10:18.722 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:10:18.766 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:18.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:18.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:10:18.793 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:10:18 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:10:18.868 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:10:18.905 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:18.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:18.907 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:10:19.007 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:10:19.053 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:19.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:19.055 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:10:19.135 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:10:19.156 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:10:19.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:19.190 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:19.190 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:10:19.279 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:19 smithi100 ceph-mon[29092]: 2.11 scrub starts 2024-09-17T07:10:19.279 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:19 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/3130894516' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-09-17T07:10:19.279 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:19 smithi100 ceph-mon[29092]: osdmap e205: 8 total, 8 up, 7 in 2024-09-17T07:10:19.279 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:19 smithi100 ceph-mon[39462]: 2.11 scrub starts 2024-09-17T07:10:19.279 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:19 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/3130894516' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-09-17T07:10:19.280 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:19 smithi100 ceph-mon[39462]: osdmap e205: 8 total, 8 up, 7 in 2024-09-17T07:10:19.291 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:10:19.331 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:19.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:19.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:10:19.436 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:10:19.476 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:19.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:19.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:10:19.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:19 smithi135 ceph-mon[33238]: 2.11 scrub starts 2024-09-17T07:10:19.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:19 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/3130894516' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-09-17T07:10:19.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:19 smithi135 ceph-mon[33238]: osdmap e205: 8 total, 8 up, 7 in 2024-09-17T07:10:19.578 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:10:19.613 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:19.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:19.614 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:10:19.692 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-17T07:10:19.715 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:10:19.798 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:19.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:19.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:10:19.900 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:10:19.940 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:19.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:19.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:10:20.042 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:10:20.078 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:20.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:20.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:10:20.179 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:10:20.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:20.218 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:20.219 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:10:20.319 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:10:20.361 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:20.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:20.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:10:20.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:20 smithi100 ceph-mon[29092]: 8.5 scrub starts 2024-09-17T07:10:20.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:20 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:10:20.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:20 smithi100 ceph-mon[29092]: pgmap v356: 148 pgs: 1 peering, 1 active+clean+scrubbing, 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 141 active+clean; 5.9 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.28k op/s; 11188/290580 objects misplaced (3.850%); 59 MiB/s, 946 objects/s recovering 2024-09-17T07:10:20.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:20 smithi100 ceph-mon[39462]: 8.5 scrub starts 2024-09-17T07:10:20.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:20 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:10:20.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:20 smithi100 ceph-mon[39462]: pgmap v356: 148 pgs: 1 peering, 1 active+clean+scrubbing, 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 141 active+clean; 5.9 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.28k op/s; 11188/290580 objects misplaced (3.850%); 59 MiB/s, 946 objects/s recovering 2024-09-17T07:10:20.463 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:10:20.491 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:20.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:20.492 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:10:20.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:20 smithi135 ceph-mon[33238]: 8.5 scrub starts 2024-09-17T07:10:20.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:20 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:10:20.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:20 smithi135 ceph-mon[33238]: pgmap v356: 148 pgs: 1 peering, 1 active+clean+scrubbing, 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 141 active+clean; 5.9 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.28k op/s; 11188/290580 objects misplaced (3.850%); 59 MiB/s, 946 objects/s recovering 2024-09-17T07:10:20.593 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:10:20.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:20.625 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:20.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:10:20.726 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:10:20.747 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:20.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:20.748 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:10:20.849 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:10:20.880 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:20.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:20.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:10:20.982 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:10:21.017 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:21.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:21.018 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:10:21.119 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:10:21.146 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:21.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:21.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:10:21.248 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:10:21.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:21.292 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:21.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:10:21.393 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:10:21.429 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:21.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:21.430 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:10:21.531 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:10:21.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:21 smithi135 ceph-mon[33238]: 6.b scrub starts 2024-09-17T07:10:21.573 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:21.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:21.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:10:21.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:21 smithi100 ceph-mon[29092]: 6.b scrub starts 2024-09-17T07:10:21.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:21 smithi100 ceph-mon[39462]: 6.b scrub starts 2024-09-17T07:10:21.674 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:10:21.718 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:21.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:21.720 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:10:21.820 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:10:21.873 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:21.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:21.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:10:21.885 INFO:teuthology.orchestra.run.smithi135.stdout:{ 2024-09-17T07:10:21.885 INFO:teuthology.orchestra.run.smithi135.stdout: "ops": [], 2024-09-17T07:10:21.885 INFO:teuthology.orchestra.run.smithi135.stdout: "complaint_time": 30, 2024-09-17T07:10:21.885 INFO:teuthology.orchestra.run.smithi135.stdout: "num_blocked_ops": 0 2024-09-17T07:10:21.885 INFO:teuthology.orchestra.run.smithi135.stdout:} 2024-09-17T07:10:21.975 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:10:22.007 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:22.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:22.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:10:22.108 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:10:22.190 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:22.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:22.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:10:22.247 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:22 smithi100 ceph-mon[29092]: pgmap v357: 148 pgs: 1 peering, 1 active+clean+scrubbing, 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 141 active+clean; 5.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 953 op/s; 10364/287994 objects misplaced (3.599%); 42 MiB/s, 670 objects/s recovering 2024-09-17T07:10:22.247 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:22 smithi100 ceph-mon[29092]: 5.1f scrub starts 2024-09-17T07:10:22.247 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:22 smithi100 ceph-mon[39462]: pgmap v357: 148 pgs: 1 peering, 1 active+clean+scrubbing, 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 141 active+clean; 5.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 953 op/s; 10364/287994 objects misplaced (3.599%); 42 MiB/s, 670 objects/s recovering 2024-09-17T07:10:22.248 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:22 smithi100 ceph-mon[39462]: 5.1f scrub starts 2024-09-17T07:10:22.292 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:10:22.326 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:22.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:22.327 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:10:22.389 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:10:22.427 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:10:22.449 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:10:22.449 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:10:22.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:22.473 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:22.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:10:22.485 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1: {} 2024-09-17T07:10:22.485 INFO:teuthology.orchestra.run.smithi100.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:10:22.514 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:10:22.515 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:10:22.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:22 smithi135 ceph-mon[33238]: pgmap v357: 148 pgs: 1 peering, 1 active+clean+scrubbing, 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 141 active+clean; 5.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 953 op/s; 10364/287994 objects misplaced (3.599%); 42 MiB/s, 670 objects/s recovering 2024-09-17T07:10:22.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:22 smithi135 ceph-mon[33238]: 5.1f scrub starts 2024-09-17T07:10:22.547 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:10:22.547 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:10:22.575 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:10:22.582 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:10:22.582 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:10:22.613 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:22.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:22.615 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:10:22.624 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:10:22.625 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:10:22.644 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-17T07:10:22.660 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:10:22.660 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:10:22.695 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:10:22.695 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:10:22.716 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:10:22.747 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:22.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:22.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:10:22.849 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:10:22.878 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:22.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:22.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:10:22.980 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:10:23.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:23.038 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:23.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:10:23.059 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:10:23.139 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:10:23.174 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:23.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:23.175 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:10:23.276 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:10:23.297 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:23.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:23.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:10:23.399 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:10:23.433 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:23.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:23.434 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:10:23.479 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:23 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/841232687' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:10:23.536 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:10:23.560 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:23.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:23.561 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:10:23.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:23 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/841232687' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:10:23.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:23 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/841232687' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:10:23.662 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:10:23.688 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:23.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:23.689 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:10:23.790 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:10:23.812 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:23.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:23.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:10:23.913 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:10:23.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:23.938 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:23.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:10:24.039 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:10:24.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:24.064 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:24.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:10:24.066 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7] out_osds: [3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-17T07:10:24.066 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T07:10:24.067 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_1 2024-09-17T07:10:24.067 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T07:10:24.166 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:10:24.187 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:24.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:24.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:10:24.289 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:10:24.300 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:24 smithi100 ceph-mon[29092]: pgmap v358: 148 pgs: 1 active+clean+scrubbing, 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 142 active+clean; 5.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 784 op/s; 9011/281805 objects misplaced (3.198%); 43 MiB/s, 680 objects/s recovering 2024-09-17T07:10:24.300 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:24 smithi100 ceph-mon[29092]: osdmap e206: 8 total, 8 up, 7 in 2024-09-17T07:10:24.300 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:24 smithi100 ceph-mon[29092]: 2.c scrub starts 2024-09-17T07:10:24.300 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:24 smithi100 ceph-mon[29092]: 2.17 scrub starts 2024-09-17T07:10:24.300 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:24 smithi100 ceph-mon[39462]: pgmap v358: 148 pgs: 1 active+clean+scrubbing, 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 142 active+clean; 5.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 784 op/s; 9011/281805 objects misplaced (3.198%); 43 MiB/s, 680 objects/s recovering 2024-09-17T07:10:24.301 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:24 smithi100 ceph-mon[39462]: osdmap e206: 8 total, 8 up, 7 in 2024-09-17T07:10:24.301 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:24 smithi100 ceph-mon[39462]: 2.c scrub starts 2024-09-17T07:10:24.301 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:24 smithi100 ceph-mon[39462]: 2.17 scrub starts 2024-09-17T07:10:24.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:24.331 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:24.332 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:10:24.433 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:10:24.456 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:24.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:24.457 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:10:24.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:24 smithi135 ceph-mon[33238]: pgmap v358: 148 pgs: 1 active+clean+scrubbing, 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 142 active+clean; 5.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 784 op/s; 9011/281805 objects misplaced (3.198%); 43 MiB/s, 680 objects/s recovering 2024-09-17T07:10:24.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:24 smithi135 ceph-mon[33238]: osdmap e206: 8 total, 8 up, 7 in 2024-09-17T07:10:24.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:24 smithi135 ceph-mon[33238]: 2.c scrub starts 2024-09-17T07:10:24.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:24 smithi135 ceph-mon[33238]: 2.17 scrub starts 2024-09-17T07:10:24.558 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:10:24.580 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:24.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:24.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:10:24.682 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:10:24.706 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:24.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:24.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:10:24.808 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:10:24.831 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:24.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:24.833 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:10:24.853 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:10:24.934 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:10:24.958 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:24.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:24.959 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:10:25.060 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:10:25.083 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:25.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:25.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:10:25.185 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:10:25.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:25.227 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:25.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:10:25.328 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:10:25.359 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:25.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:25.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:10:25.461 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:10:25.473 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:25 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:10:25.473 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:25 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:10:25.473 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:25 smithi100 ceph-mon[29092]: osdmap e207: 8 total, 8 up, 7 in 2024-09-17T07:10:25.474 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:25 smithi100 ceph-mon[29092]: osdmap e208: 8 total, 8 up, 7 in 2024-09-17T07:10:25.474 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:25 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:10:25.474 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:25 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:10:25.474 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:25 smithi100 ceph-mon[39462]: osdmap e207: 8 total, 8 up, 7 in 2024-09-17T07:10:25.474 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:25 smithi100 ceph-mon[39462]: osdmap e208: 8 total, 8 up, 7 in 2024-09-17T07:10:25.483 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:25.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:25.484 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:10:25.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:25 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:10:25.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:25 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:10:25.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:25 smithi135 ceph-mon[33238]: osdmap e207: 8 total, 8 up, 7 in 2024-09-17T07:10:25.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:25 smithi135 ceph-mon[33238]: osdmap e208: 8 total, 8 up, 7 in 2024-09-17T07:10:25.585 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:10:25.607 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:25.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:25.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:10:25.709 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:10:25.732 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:25.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:25.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:10:25.834 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:10:25.856 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:25.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:25.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:10:25.958 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:10:25.980 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:25.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:25.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:10:26.082 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:10:26.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:26.105 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:26.105 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:10:26.126 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:10:25 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:10:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:10:26.206 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:10:26.241 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:26.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:26.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:10:26.338 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:10:26.343 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:10:26.411 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:26.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:26.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:10:26.513 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:10:26.539 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:26.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:26.540 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:10:26.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:26 smithi135 ceph-mon[33238]: pgmap v362: 148 pgs: 1 peering, 2 active+clean+scrubbing, 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 141 active+clean; 5.5 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.01k op/s; 7179/272316 objects misplaced (2.636%); 36 MiB/s, 571 objects/s recovering 2024-09-17T07:10:26.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:26 smithi135 ceph-mon[33238]: 4.3 scrub starts 2024-09-17T07:10:26.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:26 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/3192728459' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:10:26.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:26 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:10:26.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:26 smithi135 ceph-mon[33238]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T07:10:26.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:26 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T07:10:26.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:26 smithi135 ceph-mon[33238]: osdmap e209: 8 total, 8 up, 7 in 2024-09-17T07:10:26.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:26 smithi100 ceph-mon[29092]: pgmap v362: 148 pgs: 1 peering, 2 active+clean+scrubbing, 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 141 active+clean; 5.5 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.01k op/s; 7179/272316 objects misplaced (2.636%); 36 MiB/s, 571 objects/s recovering 2024-09-17T07:10:26.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:26 smithi100 ceph-mon[29092]: 4.3 scrub starts 2024-09-17T07:10:26.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:26 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/3192728459' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:10:26.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:26 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:10:26.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:26 smithi100 ceph-mon[29092]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T07:10:26.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:26 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T07:10:26.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:26 smithi100 ceph-mon[29092]: osdmap e209: 8 total, 8 up, 7 in 2024-09-17T07:10:26.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:26 smithi100 ceph-mon[39462]: pgmap v362: 148 pgs: 1 peering, 2 active+clean+scrubbing, 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 141 active+clean; 5.5 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.01k op/s; 7179/272316 objects misplaced (2.636%); 36 MiB/s, 571 objects/s recovering 2024-09-17T07:10:26.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:26 smithi100 ceph-mon[39462]: 4.3 scrub starts 2024-09-17T07:10:26.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:26 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/3192728459' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:10:26.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:26 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:10:26.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:26 smithi100 ceph-mon[39462]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T07:10:26.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:26 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T07:10:26.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:26 smithi100 ceph-mon[39462]: osdmap e209: 8 total, 8 up, 7 in 2024-09-17T07:10:26.641 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:10:26.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:26.663 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:26.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:10:26.764 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:10:26.787 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:26.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:26.788 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:10:26.889 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:10:26.910 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:26.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:26.911 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:10:26.922 INFO:teuthology.orchestra.run.smithi100.stderr:nodeep-scrub is unset 2024-09-17T07:10:27.012 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:10:27.033 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:27.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:27.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:10:27.135 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:10:27.161 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:27.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:27.162 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:10:27.263 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:10:27.272 INFO:teuthology.orchestra.run.smithi135.stdout:ERROR: (22) Invalid argument 2024-09-17T07:10:27.272 INFO:teuthology.orchestra.run.smithi135.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:10:27.305 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:27.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:27.306 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:10:27.407 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:10:27.443 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:27.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:27.444 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:10:27.518 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:27 smithi135 ceph-mon[33238]: 6.19 scrub starts 2024-09-17T07:10:27.518 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:27 smithi135 ceph-mon[33238]: 5.b scrub starts 2024-09-17T07:10:27.518 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:27 smithi135 ceph-mon[33238]: 5.b scrub ok 2024-09-17T07:10:27.518 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:27 smithi135 ceph-mon[33238]: 2.1e scrub starts 2024-09-17T07:10:27.518 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:27 smithi135 ceph-mon[33238]: 2.1e scrub ok 2024-09-17T07:10:27.518 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:27 smithi135 ceph-mon[33238]: 2.b deep-scrub starts 2024-09-17T07:10:27.518 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:27 smithi135 ceph-mon[33238]: osdmap e210: 8 total, 8 up, 7 in 2024-09-17T07:10:27.545 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:10:27.584 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:27.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:27.585 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:10:27.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:27 smithi100 ceph-mon[29092]: 6.19 scrub starts 2024-09-17T07:10:27.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:27 smithi100 ceph-mon[29092]: 5.b scrub starts 2024-09-17T07:10:27.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:27 smithi100 ceph-mon[29092]: 5.b scrub ok 2024-09-17T07:10:27.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:27 smithi100 ceph-mon[29092]: 2.1e scrub starts 2024-09-17T07:10:27.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:27 smithi100 ceph-mon[29092]: 2.1e scrub ok 2024-09-17T07:10:27.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:27 smithi100 ceph-mon[29092]: 2.b deep-scrub starts 2024-09-17T07:10:27.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:27 smithi100 ceph-mon[29092]: osdmap e210: 8 total, 8 up, 7 in 2024-09-17T07:10:27.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:27 smithi100 ceph-mon[39462]: 6.19 scrub starts 2024-09-17T07:10:27.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:27 smithi100 ceph-mon[39462]: 5.b scrub starts 2024-09-17T07:10:27.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:27 smithi100 ceph-mon[39462]: 5.b scrub ok 2024-09-17T07:10:27.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:27 smithi100 ceph-mon[39462]: 2.1e scrub starts 2024-09-17T07:10:27.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:27 smithi100 ceph-mon[39462]: 2.1e scrub ok 2024-09-17T07:10:27.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:27 smithi100 ceph-mon[39462]: 2.b deep-scrub starts 2024-09-17T07:10:27.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:27 smithi100 ceph-mon[39462]: osdmap e210: 8 total, 8 up, 7 in 2024-09-17T07:10:27.686 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:10:27.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:27.719 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:27.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:10:27.820 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:10:27.852 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:27.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:27.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:10:27.951 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-17T07:10:27.954 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:10:28.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:28.045 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:28.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:10:28.146 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:10:28.196 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:28.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:28.197 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:10:28.210 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:10:28.298 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:10:28.300 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:10:28.329 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:28.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:28.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:10:28.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:28 smithi100 ceph-mon[29092]: pgmap v365: 148 pgs: 1 peering, 2 active+clean+scrubbing, 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 141 active+clean; 5.5 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 626 op/s; 7179/272316 objects misplaced (2.636%); 23 MiB/s, 362 objects/s recovering 2024-09-17T07:10:28.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:28 smithi100 ceph-mon[39462]: pgmap v365: 148 pgs: 1 peering, 2 active+clean+scrubbing, 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 141 active+clean; 5.5 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 626 op/s; 7179/272316 objects misplaced (2.636%); 23 MiB/s, 362 objects/s recovering 2024-09-17T07:10:28.431 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:10:28.467 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:28.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:28.468 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:10:28.480 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:28 smithi135 ceph-mon[33238]: pgmap v365: 148 pgs: 1 peering, 2 active+clean+scrubbing, 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 141 active+clean; 5.5 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 626 op/s; 7179/272316 objects misplaced (2.636%); 23 MiB/s, 362 objects/s recovering 2024-09-17T07:10:28.569 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:10:28.603 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:28.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:28.604 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:10:28.705 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:10:28.731 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:10:28 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:10:28.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:28.746 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:28.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:10:28.847 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:10:28.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:28.884 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:28.885 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:10:28.985 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:10:29.021 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:29.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:29.022 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:10:29.123 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:10:29.163 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:29.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:29.165 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:10:29.266 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:10:29.300 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:29.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:29.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:10:29.373 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:29 smithi100 ceph-mon[39462]: 2.5 scrub starts 2024-09-17T07:10:29.373 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:29 smithi100 ceph-mon[29092]: 2.5 scrub starts 2024-09-17T07:10:29.380 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T07:10:29.383 INFO:teuthology.orchestra.run.smithi100.stderr:dumped all 2024-09-17T07:10:29.402 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:10:29.431 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:29.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:29.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:10:29.533 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:10:29.539 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:29 smithi135 ceph-mon[33238]: 2.5 scrub starts 2024-09-17T07:10:29.567 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:29.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:29.568 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:10:29.669 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:10:29.702 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:30.356 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-17T07:10:30.357 INFO:teuthology.orchestra.run.smithi135.stdout:ERROR: (22) Invalid argument 2024-09-17T07:10:30.358 INFO:teuthology.orchestra.run.smithi135.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:10:30.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:30.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:10:30.429 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:30 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:10:30.429 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:30 smithi100 ceph-mon[29092]: 6.a scrub starts 2024-09-17T07:10:30.429 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:30 smithi100 ceph-mon[29092]: 6.a scrub ok 2024-09-17T07:10:30.429 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:30 smithi100 ceph-mon[29092]: pgmap v366: 148 pgs: 2 active+remapped+backfilling, 146 active+clean; 5.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 413 op/s; 4809/266181 objects misplaced (1.807%); 28 MiB/s, 449 objects/s recovering 2024-09-17T07:10:30.429 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:30 smithi100 ceph-mon[29092]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-17T07:10:30.429 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:30 smithi100 ceph-mon[29092]: Cluster is now healthy 2024-09-17T07:10:30.429 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:30 smithi100 ceph-mon[29092]: 5.6 scrub starts 2024-09-17T07:10:30.429 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:30 smithi100 ceph-mon[29092]: 5.6 scrub ok 2024-09-17T07:10:30.429 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:30 smithi100 ceph-mon[29092]: 2.19 deep-scrub starts 2024-09-17T07:10:30.430 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:30 smithi100 ceph-mon[29092]: 2.19 deep-scrub ok 2024-09-17T07:10:30.430 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:30 smithi100 ceph-mon[29092]: 2.8 scrub starts 2024-09-17T07:10:30.430 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:30 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:10:30.430 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:30 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:10:30.430 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:30 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:10:30.430 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:30 smithi100 ceph-mon[39462]: 6.a scrub starts 2024-09-17T07:10:30.430 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:30 smithi100 ceph-mon[39462]: 6.a scrub ok 2024-09-17T07:10:30.430 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:30 smithi100 ceph-mon[39462]: pgmap v366: 148 pgs: 2 active+remapped+backfilling, 146 active+clean; 5.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 413 op/s; 4809/266181 objects misplaced (1.807%); 28 MiB/s, 449 objects/s recovering 2024-09-17T07:10:30.430 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:30 smithi100 ceph-mon[39462]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-17T07:10:30.430 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:30 smithi100 ceph-mon[39462]: Cluster is now healthy 2024-09-17T07:10:30.431 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:30 smithi100 ceph-mon[39462]: 5.6 scrub starts 2024-09-17T07:10:30.431 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:30 smithi100 ceph-mon[39462]: 5.6 scrub ok 2024-09-17T07:10:30.431 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:30 smithi100 ceph-mon[39462]: 2.19 deep-scrub starts 2024-09-17T07:10:30.431 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:30 smithi100 ceph-mon[39462]: 2.19 deep-scrub ok 2024-09-17T07:10:30.431 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:30 smithi100 ceph-mon[39462]: 2.8 scrub starts 2024-09-17T07:10:30.431 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:30 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:10:30.431 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:30 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:10:30.440 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:30 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:10:30.440 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:30 smithi135 ceph-mon[33238]: 6.a scrub starts 2024-09-17T07:10:30.441 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:30 smithi135 ceph-mon[33238]: 6.a scrub ok 2024-09-17T07:10:30.441 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:30 smithi135 ceph-mon[33238]: pgmap v366: 148 pgs: 2 active+remapped+backfilling, 146 active+clean; 5.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 413 op/s; 4809/266181 objects misplaced (1.807%); 28 MiB/s, 449 objects/s recovering 2024-09-17T07:10:30.441 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:30 smithi135 ceph-mon[33238]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-17T07:10:30.441 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:30 smithi135 ceph-mon[33238]: Cluster is now healthy 2024-09-17T07:10:30.441 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:30 smithi135 ceph-mon[33238]: 5.6 scrub starts 2024-09-17T07:10:30.441 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:30 smithi135 ceph-mon[33238]: 5.6 scrub ok 2024-09-17T07:10:30.441 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:30 smithi135 ceph-mon[33238]: 2.19 deep-scrub starts 2024-09-17T07:10:30.441 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:30 smithi135 ceph-mon[33238]: 2.19 deep-scrub ok 2024-09-17T07:10:30.441 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:30 smithi135 ceph-mon[33238]: 2.8 scrub starts 2024-09-17T07:10:30.441 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:30 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:10:30.441 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:30 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:10:30.459 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:10:30.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:30.501 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:30.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:10:30.602 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:10:30.643 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:30.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:30.644 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:10:30.744 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:10:30.780 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:30.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:30.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:10:30.882 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:10:30.912 INFO:teuthology.orchestra.run.smithi100.stdout:{"pg_ready":true,"pg_map":{"version":366,"stamp":"2024-09-17T07:10:29.178485+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5802498350,"num_objects":88727,"num_object_clones":0,"num_object_copies":266181,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4809,"num_objects_unfound":0,"num_objects_dirty":88727,"num_whiteouts":0,"num_read":1406,"num_read_kb":1252,"num_write":173082,"num_write_kb":8354524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35449,"num_bytes_recovered":2317684732,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76766,"ondisk_log_size":76766,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":19050400,"kb_used_data":16772428,"kb_used_omap":10,"kb_used_meta":2277621,"kb_avail":637165664,"statfs":{"total":671965249536,"available":652457639936,"internally_reserved":0,"allocated":17174966272,"data_stored":17172089118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11152,"internal_metadata":2332284016},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-134021120,"num_objects":-2045,"num_object_clones":0,"num_object_copies":-6135,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2370,"num_objects_unfound":0,"num_objects_dirty":-2045,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":2045,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2227,"num_bytes_recovered":145948672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.957378"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":29,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.156189+0000","last_change":"2024-09-17T07:10:18.216883+0000","last_active":"2024-09-17T07:10:26.156189+0000","last_peered":"2024-09-17T07:10:26.156189+0000","last_clean":"2024-09-17T07:10:26.156189+0000","last_became_active":"2024-09-17T07:09:24.401344+0000","last_became_peered":"2024-09-17T07:09:24.401344+0000","last_unstale":"2024-09-17T07:10:26.156189+0000","last_undegraded":"2024-09-17T07:10:26.156189+0000","last_fullsized":"2024-09-17T07:10:26.156189+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:33.644728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:33.644728+0000","last_clean_scrub_stamp":"2024-09-17T07:07:33.644728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":184,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-17T07:10:18.168615+0000","last_change":"2024-09-17T07:08:17.332759+0000","last_active":"2024-09-17T07:10:18.168615+0000","last_peered":"2024-09-17T07:10:18.168615+0000","last_clean":"2024-09-17T07:10:18.168615+0000","last_became_active":"2024-09-17T07:02:12.006754+0000","last_became_peered":"2024-09-17T07:02:12.006754+0000","last_unstale":"2024-09-17T07:10:18.168615+0000","last_undegraded":"2024-09-17T07:10:18.168615+0000","last_fullsized":"2024-09-17T07:10:18.168615+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:17.332631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:08:17.332631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023942220000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"75'15","reported_seq":77,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-17T07:10:18.166947+0000","last_change":"2024-09-17T07:09:24.419732+0000","last_active":"2024-09-17T07:10:18.166947+0000","last_peered":"2024-09-17T07:10:18.166947+0000","last_clean":"2024-09-17T07:10:18.166947+0000","last_became_active":"2024-09-17T07:09:24.402042+0000","last_became_peered":"2024-09-17T07:09:24.402042+0000","last_unstale":"2024-09-17T07:10:18.166947+0000","last_undegraded":"2024-09-17T07:10:18.166947+0000","last_fullsized":"2024-09-17T07:10:18.166947+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-09-17T07:08:28.845108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:08:28.845108+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":173,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.149851+0000","last_change":"2024-09-17T07:08:21.564706+0000","last_active":"2024-09-17T07:10:26.149851+0000","last_peered":"2024-09-17T07:10:26.149851+0000","last_clean":"2024-09-17T07:10:26.149851+0000","last_became_active":"2024-09-17T07:02:16.014928+0000","last_became_peered":"2024-09-17T07:02:16.014928+0000","last_unstale":"2024-09-17T07:10:26.149851+0000","last_undegraded":"2024-09-17T07:10:26.149851+0000","last_fullsized":"2024-09-17T07:10:26.149851+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:21.564567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:08:21.564567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0039106849999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":24,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-17T07:10:18.168957+0000","last_change":"2024-09-17T07:09:24.408521+0000","last_active":"2024-09-17T07:10:18.168957+0000","last_peered":"2024-09-17T07:10:18.168957+0000","last_clean":"2024-09-17T07:10:18.168957+0000","last_became_active":"2024-09-17T07:09:24.402568+0000","last_became_peered":"2024-09-17T07:09:24.402568+0000","last_unstale":"2024-09-17T07:10:18.168957+0000","last_undegraded":"2024-09-17T07:10:18.168957+0000","last_fullsized":"2024-09-17T07:10:18.168957+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:15.767495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:07:15.767495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"75'9","reported_seq":202,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.149933+0000","last_change":"2024-09-17T07:08:22.546769+0000","last_active":"2024-09-17T07:10:26.149933+0000","last_peered":"2024-09-17T07:10:26.149933+0000","last_clean":"2024-09-17T07:10:26.149933+0000","last_became_active":"2024-09-17T07:02:14.016244+0000","last_became_peered":"2024-09-17T07:02:14.016244+0000","last_unstale":"2024-09-17T07:10:26.149933+0000","last_undegraded":"2024-09-17T07:10:26.149933+0000","last_fullsized":"2024-09-17T07:10:26.149933+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:08:22.546632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:08:22.546632+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0041894990000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":184,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.153955+0000","last_change":"2024-09-17T07:09:25.123274+0000","last_active":"2024-09-17T07:10:26.153955+0000","last_peered":"2024-09-17T07:10:26.153955+0000","last_clean":"2024-09-17T07:10:26.153955+0000","last_became_active":"2024-09-17T07:02:16.014380+0000","last_became_peered":"2024-09-17T07:02:16.014380+0000","last_unstale":"2024-09-17T07:10:26.153955+0000","last_undegraded":"2024-09-17T07:10:26.153955+0000","last_fullsized":"2024-09-17T07:10:26.153955+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:25.123158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:09:25.123158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:49.651515+0000","scrub_duration":0.00166512,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":172,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.149959+0000","last_change":"2024-09-17T07:07:21.791530+0000","last_active":"2024-09-17T07:10:26.149959+0000","last_peered":"2024-09-17T07:10:26.149959+0000","last_clean":"2024-09-17T07:10:26.149959+0000","last_became_active":"2024-09-17T07:02:18.020790+0000","last_became_peered":"2024-09-17T07:02:18.020790+0000","last_unstale":"2024-09-17T07:10:26.149959+0000","last_undegraded":"2024-09-17T07:10:26.149959+0000","last_fullsized":"2024-09-17T07:10:26.149959+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:21.791399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:07:21.791399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0069986889999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":188,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.156758+0000","last_change":"2024-09-17T07:09:09.812811+0000","last_active":"2024-09-17T07:10:26.156758+0000","last_peered":"2024-09-17T07:10:26.156758+0000","last_clean":"2024-09-17T07:10:26.156758+0000","last_became_active":"2024-09-17T07:02:12.011129+0000","last_became_peered":"2024-09-17T07:02:12.011129+0000","last_unstale":"2024-09-17T07:10:26.156758+0000","last_undegraded":"2024-09-17T07:10:26.156758+0000","last_fullsized":"2024-09-17T07:10:26.156758+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:09.812692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:43.060102+0000","last_clean_scrub_stamp":"2024-09-17T07:09:09.812692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0062170050000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"75'5","reported_seq":201,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.150175+0000","last_change":"2024-09-17T07:09:24.415312+0000","last_active":"2024-09-17T07:10:26.150175+0000","last_peered":"2024-09-17T07:10:26.150175+0000","last_clean":"2024-09-17T07:10:26.150175+0000","last_became_active":"2024-09-17T07:09:24.396900+0000","last_became_peered":"2024-09-17T07:09:24.396900+0000","last_unstale":"2024-09-17T07:10:26.150175+0000","last_undegraded":"2024-09-17T07:10:26.150175+0000","last_fullsized":"2024-09-17T07:10:26.150175+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-09-17T07:07:15.848071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:07:15.848071+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010596959,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":184,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.156286+0000","last_change":"2024-09-17T07:08:19.081985+0000","last_active":"2024-09-17T07:10:26.156286+0000","last_peered":"2024-09-17T07:10:26.156286+0000","last_clean":"2024-09-17T07:10:26.156286+0000","last_became_active":"2024-09-17T07:02:16.015401+0000","last_became_peered":"2024-09-17T07:02:16.015401+0000","last_unstale":"2024-09-17T07:10:26.156286+0000","last_undegraded":"2024-09-17T07:10:26.156286+0000","last_fullsized":"2024-09-17T07:10:26.156286+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:19.081861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:08:19.081861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002870844,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":191,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.151703+0000","last_change":"2024-09-17T07:10:26.151703+0000","last_active":"2024-09-17T07:10:26.151703+0000","last_peered":"2024-09-17T07:10:26.151703+0000","last_clean":"2024-09-17T07:10:26.151703+0000","last_became_active":"2024-09-17T07:09:24.402616+0000","last_became_peered":"2024-09-17T07:09:24.402616+0000","last_unstale":"2024-09-17T07:10:26.151703+0000","last_undegraded":"2024-09-17T07:10:26.151703+0000","last_fullsized":"2024-09-17T07:10:26.151703+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:23.739157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:07:23.739157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004894603,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":188,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.157121+0000","last_change":"2024-09-17T07:09:08.807223+0000","last_active":"2024-09-17T07:10:26.157121+0000","last_peered":"2024-09-17T07:10:26.157121+0000","last_clean":"2024-09-17T07:10:26.157121+0000","last_became_active":"2024-09-17T07:02:12.012234+0000","last_became_peered":"2024-09-17T07:02:12.012234+0000","last_unstale":"2024-09-17T07:10:26.157121+0000","last_undegraded":"2024-09-17T07:10:26.157121+0000","last_fullsized":"2024-09-17T07:10:26.157121+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:08.807058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:45.029247+0000","last_clean_scrub_stamp":"2024-09-17T07:09:08.807058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.015274785000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"75'9","reported_seq":210,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.150411+0000","last_change":"2024-09-17T07:09:24.432560+0000","last_active":"2024-09-17T07:10:26.150411+0000","last_peered":"2024-09-17T07:10:26.150411+0000","last_clean":"2024-09-17T07:10:26.150411+0000","last_became_active":"2024-09-17T07:09:24.397954+0000","last_became_peered":"2024-09-17T07:09:24.397954+0000","last_unstale":"2024-09-17T07:10:26.150411+0000","last_undegraded":"2024-09-17T07:10:26.150411+0000","last_fullsized":"2024-09-17T07:10:26.150411+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:08:17.673242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:08:17.673242+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0059611450000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":183,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.151779+0000","last_change":"2024-09-17T07:08:28.096341+0000","last_active":"2024-09-17T07:10:26.151779+0000","last_peered":"2024-09-17T07:10:26.151779+0000","last_clean":"2024-09-17T07:10:26.151779+0000","last_became_active":"2024-09-17T07:02:16.023774+0000","last_became_peered":"2024-09-17T07:02:16.023774+0000","last_unstale":"2024-09-17T07:10:26.151779+0000","last_undegraded":"2024-09-17T07:10:26.151779+0000","last_fullsized":"2024-09-17T07:10:26.151779+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:28.096266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:28.096266+0000","last_clean_scrub_stamp":"2024-09-17T07:08:28.096266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010716228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":167,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-17T07:10:18.166833+0000","last_change":"2024-09-17T07:07:34.205329+0000","last_active":"2024-09-17T07:10:18.166833+0000","last_peered":"2024-09-17T07:10:18.166833+0000","last_clean":"2024-09-17T07:10:18.166833+0000","last_became_active":"2024-09-17T07:02:18.029178+0000","last_became_peered":"2024-09-17T07:02:18.029178+0000","last_unstale":"2024-09-17T07:10:18.166833+0000","last_undegraded":"2024-09-17T07:10:18.166833+0000","last_fullsized":"2024-09-17T07:10:18.166833+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:34.205205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:07:34.205205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0057819899999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"210'10767","reported_seq":10819,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-17T07:10:28.409067+0000","last_change":"2024-09-17T07:08:00.759806+0000","last_active":"2024-09-17T07:10:28.409067+0000","last_peered":"2024-09-17T07:10:28.409067+0000","last_clean":"2024-09-17T07:10:28.409067+0000","last_became_active":"2024-09-17T07:08:00.759498+0000","last_became_peered":"2024-09-17T07:08:00.759498+0000","last_unstale":"2024-09-17T07:10:28.409067+0000","last_undegraded":"2024-09-17T07:10:28.409067+0000","last_fullsized":"2024-09-17T07:10:28.409067+0000","mapping_epoch":168,"log_start":"191'6345","ondisk_log_start":"191'6345","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_clean_scrub_stamp":"2024-09-17T07:07:59.220681+0000","objects_scrubbed":0,"log_size":4422,"log_dups_size":0,"ondisk_log_size":4422,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":356581376,"num_objects":5441,"num_object_clones":0,"num_object_copies":16323,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5441,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10767,"num_write_kb":518656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"95'25","reported_seq":222,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.150489+0000","last_change":"2024-09-17T07:09:17.694556+0000","last_active":"2024-09-17T07:10:26.150489+0000","last_peered":"2024-09-17T07:10:26.150489+0000","last_clean":"2024-09-17T07:10:26.150489+0000","last_became_active":"2024-09-17T07:02:14.016270+0000","last_became_peered":"2024-09-17T07:02:14.016270+0000","last_unstale":"2024-09-17T07:10:26.150489+0000","last_undegraded":"2024-09-17T07:10:26.150489+0000","last_fullsized":"2024-09-17T07:10:26.150489+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'25","last_scrub_stamp":"2024-09-17T07:09:17.694236+0000","last_deep_scrub":"95'25","last_deep_scrub_stamp":"2024-09-17T07:04:09.450039+0000","last_clean_scrub_stamp":"2024-09-17T07:09:17.694236+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.014129604,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":190,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.698621+0000","last_change":"2024-09-17T07:10:27.149591+0000","last_active":"2024-09-17T07:10:26.698621+0000","last_peered":"2024-09-17T07:10:26.698621+0000","last_clean":"2024-09-17T07:10:26.698621+0000","last_became_active":"2024-09-17T07:02:12.011258+0000","last_became_peered":"2024-09-17T07:02:12.011258+0000","last_unstale":"2024-09-17T07:10:26.698621+0000","last_undegraded":"2024-09-17T07:10:26.698621+0000","last_fullsized":"2024-09-17T07:10:26.698621+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:18.088246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:08:18.088246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0020459860000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":173,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.156287+0000","last_change":"2024-09-17T07:07:24.620899+0000","last_active":"2024-09-17T07:10:26.156287+0000","last_peered":"2024-09-17T07:10:26.156287+0000","last_clean":"2024-09-17T07:10:26.156287+0000","last_became_active":"2024-09-17T07:02:16.018147+0000","last_became_peered":"2024-09-17T07:02:16.018147+0000","last_unstale":"2024-09-17T07:10:26.156287+0000","last_undegraded":"2024-09-17T07:10:26.156287+0000","last_fullsized":"2024-09-17T07:10:26.156287+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:24.620758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:07:24.620758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0050498469999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":30,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.150678+0000","last_change":"2024-09-17T07:09:24.397784+0000","last_active":"2024-09-17T07:10:26.150678+0000","last_peered":"2024-09-17T07:10:26.150678+0000","last_clean":"2024-09-17T07:10:26.150678+0000","last_became_active":"2024-09-17T07:09:24.396334+0000","last_became_peered":"2024-09-17T07:09:24.396334+0000","last_unstale":"2024-09-17T07:10:26.150678+0000","last_undegraded":"2024-09-17T07:10:26.150678+0000","last_fullsized":"2024-09-17T07:10:26.150678+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:21.753939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:42.984088+0000","last_clean_scrub_stamp":"2024-09-17T07:09:21.753939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:28.208006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.0","version":"210'10769","reported_seq":23566,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-17T07:10:28.122248+0000","last_change":"2024-09-17T07:10:27.184838+0000","last_active":"2024-09-17T07:10:28.122248+0000","last_peered":"2024-09-17T07:10:28.122248+0000","last_clean":"2024-09-17T07:10:28.122248+0000","last_became_active":"2024-09-17T07:10:27.154509+0000","last_became_peered":"2024-09-17T07:10:27.154509+0000","last_unstale":"2024-09-17T07:10:28.122248+0000","last_undegraded":"2024-09-17T07:10:28.122248+0000","last_fullsized":"2024-09-17T07:10:28.122248+0000","mapping_epoch":209,"log_start":"190'6003","ondisk_log_start":"190'6003","created":168,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_clean_scrub_stamp":"2024-09-17T07:07:59.220681+0000","objects_scrubbed":0,"log_size":4766,"log_dups_size":0,"ondisk_log_size":4766,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":360906752,"num_objects":5507,"num_object_clones":0,"num_object_copies":16521,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5507,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10769,"num_write_kb":520832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6360,"num_bytes_recovered":416808960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.c","version":"176'15","reported_seq":227,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.150330+0000","last_change":"2024-09-17T07:09:24.427752+0000","last_active":"2024-09-17T07:10:26.150330+0000","last_peered":"2024-09-17T07:10:26.150330+0000","last_clean":"2024-09-17T07:10:26.150330+0000","last_became_active":"2024-09-17T07:09:24.399287+0000","last_became_peered":"2024-09-17T07:09:24.399287+0000","last_unstale":"2024-09-17T07:10:26.150330+0000","last_undegraded":"2024-09-17T07:10:26.150330+0000","last_fullsized":"2024-09-17T07:10:26.150330+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'15","last_scrub_stamp":"2024-09-17T07:08:19.633791+0000","last_deep_scrub":"176'15","last_deep_scrub_stamp":"2024-09-17T07:08:19.633791+0000","last_clean_scrub_stamp":"2024-09-17T07:08:19.633791+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.009818284,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":196,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.156037+0000","last_change":"2024-09-17T07:09:24.407640+0000","last_active":"2024-09-17T07:10:26.156037+0000","last_peered":"2024-09-17T07:10:26.156037+0000","last_clean":"2024-09-17T07:10:26.156037+0000","last_became_active":"2024-09-17T07:09:24.406973+0000","last_became_peered":"2024-09-17T07:09:24.406973+0000","last_unstale":"2024-09-17T07:10:26.156037+0000","last_undegraded":"2024-09-17T07:10:26.156037+0000","last_fullsized":"2024-09-17T07:10:26.156037+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:23.802226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:08:23.802226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0065215869999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"98'9","reported_seq":243,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.156597+0000","last_change":"2024-09-17T07:08:31.923373+0000","last_active":"2024-09-17T07:10:26.156597+0000","last_peered":"2024-09-17T07:10:26.156597+0000","last_clean":"2024-09-17T07:10:26.156597+0000","last_became_active":"2024-09-17T07:04:15.110797+0000","last_became_peered":"2024-09-17T07:04:15.110797+0000","last_unstale":"2024-09-17T07:10:26.156597+0000","last_undegraded":"2024-09-17T07:10:26.156597+0000","last_fullsized":"2024-09-17T07:10:26.156597+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-09-17T07:08:31.923228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:08:31.923228+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.015229100000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":182,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.150691+0000","last_change":"2024-09-17T07:08:30.650769+0000","last_active":"2024-09-17T07:10:26.150691+0000","last_peered":"2024-09-17T07:10:26.150691+0000","last_clean":"2024-09-17T07:10:26.150691+0000","last_became_active":"2024-09-17T07:02:18.017460+0000","last_became_peered":"2024-09-17T07:02:18.017460+0000","last_unstale":"2024-09-17T07:10:26.150691+0000","last_undegraded":"2024-09-17T07:10:26.150691+0000","last_fullsized":"2024-09-17T07:10:26.150691+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:30.650419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:08:30.650419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003351316,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"210'10618","reported_seq":10670,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-17T07:10:28.451961+0000","last_change":"2024-09-17T07:08:00.242384+0000","last_active":"2024-09-17T07:10:28.451961+0000","last_peered":"2024-09-17T07:10:28.451961+0000","last_clean":"2024-09-17T07:10:28.451961+0000","last_became_active":"2024-09-17T07:08:00.241996+0000","last_became_peered":"2024-09-17T07:08:00.241996+0000","last_unstale":"2024-09-17T07:10:28.451961+0000","last_undegraded":"2024-09-17T07:10:28.451961+0000","last_fullsized":"2024-09-17T07:10:28.451961+0000","mapping_epoch":168,"log_start":"190'6182","ondisk_log_start":"190'6182","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_clean_scrub_stamp":"2024-09-17T07:07:59.220681+0000","objects_scrubbed":0,"log_size":4436,"log_dups_size":0,"ondisk_log_size":4436,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":355598336,"num_objects":5426,"num_object_clones":0,"num_object_copies":16278,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5426,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10618,"num_write_kb":513408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"75'15","reported_seq":235,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.154660+0000","last_change":"2024-09-17T07:09:24.436271+0000","last_active":"2024-09-17T07:10:26.154660+0000","last_peered":"2024-09-17T07:10:26.154660+0000","last_clean":"2024-09-17T07:10:26.154660+0000","last_became_active":"2024-09-17T07:09:24.407311+0000","last_became_peered":"2024-09-17T07:09:24.407311+0000","last_unstale":"2024-09-17T07:10:26.154660+0000","last_undegraded":"2024-09-17T07:10:26.154660+0000","last_fullsized":"2024-09-17T07:10:26.154660+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-09-17T07:08:13.784543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:08:13.784543+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002584257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":197,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.154120+0000","last_change":"2024-09-17T07:09:24.405992+0000","last_active":"2024-09-17T07:10:26.154120+0000","last_peered":"2024-09-17T07:10:26.154120+0000","last_clean":"2024-09-17T07:10:26.154120+0000","last_became_active":"2024-09-17T07:09:24.405655+0000","last_became_peered":"2024-09-17T07:09:24.405655+0000","last_unstale":"2024-09-17T07:10:26.154120+0000","last_undegraded":"2024-09-17T07:10:26.154120+0000","last_fullsized":"2024-09-17T07:10:26.154120+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:20.182256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:09:20.182256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034116369999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"75'8","reported_seq":280,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.150770+0000","last_change":"2024-09-17T07:08:09.787205+0000","last_active":"2024-09-17T07:10:26.150770+0000","last_peered":"2024-09-17T07:10:26.150770+0000","last_clean":"2024-09-17T07:10:26.150770+0000","last_became_active":"2024-09-17T07:02:16.021762+0000","last_became_peered":"2024-09-17T07:02:16.021762+0000","last_unstale":"2024-09-17T07:10:26.150770+0000","last_undegraded":"2024-09-17T07:10:26.150770+0000","last_fullsized":"2024-09-17T07:10:26.150770+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:08:09.787123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:08:09.787123+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001487141,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":181,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.151978+0000","last_change":"2024-09-17T07:08:12.991029+0000","last_active":"2024-09-17T07:10:26.151978+0000","last_peered":"2024-09-17T07:10:26.151978+0000","last_clean":"2024-09-17T07:10:26.151978+0000","last_became_active":"2024-09-17T07:02:18.025332+0000","last_became_peered":"2024-09-17T07:02:18.025332+0000","last_unstale":"2024-09-17T07:10:26.151978+0000","last_undegraded":"2024-09-17T07:10:26.151978+0000","last_fullsized":"2024-09-17T07:10:26.151978+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:12.990915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:08:12.990915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021966199999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.2","version":"210'10982","reported_seq":11034,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-17T07:10:28.085338+0000","last_change":"2024-09-17T07:08:00.237404+0000","last_active":"2024-09-17T07:10:28.085338+0000","last_peered":"2024-09-17T07:10:28.085338+0000","last_clean":"2024-09-17T07:10:28.085338+0000","last_became_active":"2024-09-17T07:08:00.237068+0000","last_became_peered":"2024-09-17T07:08:00.237068+0000","last_unstale":"2024-09-17T07:10:28.085338+0000","last_undegraded":"2024-09-17T07:10:28.085338+0000","last_fullsized":"2024-09-17T07:10:28.085338+0000","mapping_epoch":168,"log_start":"190'6186","ondisk_log_start":"190'6186","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_clean_scrub_stamp":"2024-09-17T07:07:59.220681+0000","objects_scrubbed":0,"log_size":4796,"log_dups_size":0,"ondisk_log_size":4796,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":367394816,"num_objects":5606,"num_object_clones":0,"num_object_copies":16818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10982,"num_write_kb":530816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"75'11","reported_seq":205,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.150830+0000","last_change":"2024-09-17T07:08:26.626034+0000","last_active":"2024-09-17T07:10:26.150830+0000","last_peered":"2024-09-17T07:10:26.150830+0000","last_clean":"2024-09-17T07:10:26.150830+0000","last_became_active":"2024-09-17T07:02:14.010227+0000","last_became_peered":"2024-09-17T07:02:14.010227+0000","last_unstale":"2024-09-17T07:10:26.150830+0000","last_undegraded":"2024-09-17T07:10:26.150830+0000","last_fullsized":"2024-09-17T07:10:26.150830+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-09-17T07:08:26.625892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:08:26.625892+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.038079981999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":188,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.156205+0000","last_change":"2024-09-17T07:08:06.100857+0000","last_active":"2024-09-17T07:10:26.156205+0000","last_peered":"2024-09-17T07:10:26.156205+0000","last_clean":"2024-09-17T07:10:26.156205+0000","last_became_active":"2024-09-17T07:02:12.013182+0000","last_became_peered":"2024-09-17T07:02:12.013182+0000","last_unstale":"2024-09-17T07:10:26.156205+0000","last_undegraded":"2024-09-17T07:10:26.156205+0000","last_fullsized":"2024-09-17T07:10:26.156205+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:06.100741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:08:06.100741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0012621170000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":194,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.152088+0000","last_change":"2024-09-17T07:09:24.229155+0000","last_active":"2024-09-17T07:10:26.152088+0000","last_peered":"2024-09-17T07:10:26.152088+0000","last_clean":"2024-09-17T07:10:26.152088+0000","last_became_active":"2024-09-17T07:02:16.020224+0000","last_became_peered":"2024-09-17T07:02:16.020224+0000","last_unstale":"2024-09-17T07:10:26.152088+0000","last_undegraded":"2024-09-17T07:10:26.152088+0000","last_fullsized":"2024-09-17T07:10:26.152088+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:24.229069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:09:24.229069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:38.285780+0000","scrub_duration":0.001199566,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":27,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.183486+0000","last_change":"2024-09-17T07:09:24.408435+0000","last_active":"2024-09-17T07:10:26.183486+0000","last_peered":"2024-09-17T07:10:26.183486+0000","last_clean":"2024-09-17T07:10:26.183486+0000","last_became_active":"2024-09-17T07:09:24.408109+0000","last_became_peered":"2024-09-17T07:09:24.408109+0000","last_unstale":"2024-09-17T07:10:26.183486+0000","last_undegraded":"2024-09-17T07:10:26.183486+0000","last_fullsized":"2024-09-17T07:10:26.183486+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:27.883528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:08:27.883528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.d","version":"210'10779","reported_seq":10832,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-17T07:10:28.167272+0000","last_change":"2024-09-17T07:08:00.760018+0000","last_active":"2024-09-17T07:10:28.167272+0000","last_peered":"2024-09-17T07:10:28.167272+0000","last_clean":"2024-09-17T07:10:28.167272+0000","last_became_active":"2024-09-17T07:08:00.759711+0000","last_became_peered":"2024-09-17T07:08:00.759711+0000","last_unstale":"2024-09-17T07:10:28.167272+0000","last_undegraded":"2024-09-17T07:10:28.167272+0000","last_fullsized":"2024-09-17T07:10:28.167272+0000","mapping_epoch":168,"log_start":"190'6018","ondisk_log_start":"190'6018","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_clean_scrub_stamp":"2024-09-17T07:07:59.220681+0000","objects_scrubbed":0,"log_size":4761,"log_dups_size":0,"ondisk_log_size":4761,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":360579096,"num_objects":5503,"num_object_clones":0,"num_object_copies":16509,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5503,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10779,"num_write_kb":520961,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1","version":"75'14","reported_seq":196,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.150310+0000","last_change":"2024-09-17T07:08:06.763827+0000","last_active":"2024-09-17T07:10:26.150310+0000","last_peered":"2024-09-17T07:10:26.150310+0000","last_clean":"2024-09-17T07:10:26.150310+0000","last_became_active":"2024-09-17T07:02:14.007810+0000","last_became_peered":"2024-09-17T07:02:14.007810+0000","last_unstale":"2024-09-17T07:10:26.150310+0000","last_undegraded":"2024-09-17T07:10:26.150310+0000","last_fullsized":"2024-09-17T07:10:26.150310+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'14","last_scrub_stamp":"2024-09-17T07:08:06.763689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:08:06.763689+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033699329999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":209,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.184244+0000","last_change":"2024-09-17T07:09:23.181150+0000","last_active":"2024-09-17T07:10:26.184244+0000","last_peered":"2024-09-17T07:10:26.184244+0000","last_clean":"2024-09-17T07:10:26.184244+0000","last_became_active":"2024-09-17T07:02:12.011766+0000","last_became_peered":"2024-09-17T07:02:12.011766+0000","last_unstale":"2024-09-17T07:10:26.184244+0000","last_undegraded":"2024-09-17T07:10:26.184244+0000","last_fullsized":"2024-09-17T07:10:26.184244+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:23.181028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:09:23.181028+0000","last_clean_scrub_stamp":"2024-09-17T07:09:23.181028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:51.612899+0000","scrub_duration":0.0051947429999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"196'9","reported_seq":64,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.153522+0000","last_change":"2024-09-17T07:09:24.427192+0000","last_active":"2024-09-17T07:10:26.153522+0000","last_peered":"2024-09-17T07:10:26.153522+0000","last_clean":"2024-09-17T07:10:26.153522+0000","last_became_active":"2024-09-17T07:09:24.406056+0000","last_became_peered":"2024-09-17T07:09:24.406056+0000","last_unstale":"2024-09-17T07:10:26.153522+0000","last_undegraded":"2024-09-17T07:10:26.153522+0000","last_fullsized":"2024-09-17T07:10:26.153522+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:09:14.776794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:09:14.776794+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:27.629020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":182,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.156309+0000","last_change":"2024-09-17T07:08:11.123128+0000","last_active":"2024-09-17T07:10:26.156309+0000","last_peered":"2024-09-17T07:10:26.156309+0000","last_clean":"2024-09-17T07:10:26.156309+0000","last_became_active":"2024-09-17T07:02:18.364981+0000","last_became_peered":"2024-09-17T07:02:18.364981+0000","last_unstale":"2024-09-17T07:10:26.156309+0000","last_undegraded":"2024-09-17T07:10:26.156309+0000","last_fullsized":"2024-09-17T07:10:26.156309+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:11.123047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:08:11.123047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014377210000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"206'10975","reported_seq":24412,"reported_epoch":206,"state":"active+remapped+backfilling","last_fresh":"2024-09-17T07:10:24.147710+0000","last_change":"2024-09-17T07:09:25.483625+0000","last_active":"2024-09-17T07:10:24.147710+0000","last_peered":"2024-09-17T07:10:24.147710+0000","last_clean":"2024-09-17T07:09:23.381756+0000","last_became_active":"2024-09-17T07:09:25.464119+0000","last_became_peered":"2024-09-17T07:09:25.464119+0000","last_unstale":"2024-09-17T07:10:24.147710+0000","last_undegraded":"2024-09-17T07:10:24.147710+0000","last_fullsized":"2024-09-17T07:10:24.147710+0000","mapping_epoch":195,"log_start":"189'5920","ondisk_log_start":"189'5920","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_clean_scrub_stamp":"2024-09-17T07:07:59.220681+0000","objects_scrubbed":0,"log_size":5055,"log_dups_size":0,"ondisk_log_size":5055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":378208256,"num_objects":5771,"num_object_clones":0,"num_object_copies":17313,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":342,"num_objects_unfound":0,"num_objects_dirty":5771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10975,"num_write_kb":535872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6689,"num_bytes_recovered":438370304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,3],"avail_no_missing":["0","3","7"],"object_location_counts":[{"shards":"0,3,7","objects":5771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"75'18","reported_seq":90,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.156046+0000","last_change":"2024-09-17T07:09:24.424752+0000","last_active":"2024-09-17T07:10:26.156046+0000","last_peered":"2024-09-17T07:10:26.156046+0000","last_clean":"2024-09-17T07:10:26.156046+0000","last_became_active":"2024-09-17T07:09:24.403240+0000","last_became_peered":"2024-09-17T07:09:24.403240+0000","last_unstale":"2024-09-17T07:10:26.156046+0000","last_undegraded":"2024-09-17T07:10:26.156046+0000","last_fullsized":"2024-09-17T07:10:26.156046+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'18","last_scrub_stamp":"2024-09-17T07:08:30.831301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:08:30.831301+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":33,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":187,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.156189+0000","last_change":"2024-09-17T07:08:04.692191+0000","last_active":"2024-09-17T07:10:26.156189+0000","last_peered":"2024-09-17T07:10:26.156189+0000","last_clean":"2024-09-17T07:10:26.156189+0000","last_became_active":"2024-09-17T07:02:12.005687+0000","last_became_peered":"2024-09-17T07:02:12.005687+0000","last_unstale":"2024-09-17T07:10:26.156189+0000","last_undegraded":"2024-09-17T07:10:26.156189+0000","last_fullsized":"2024-09-17T07:10:26.156189+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:04.692099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:08:04.692099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001661023,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":182,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.149844+0000","last_change":"2024-09-17T07:09:24.393694+0000","last_active":"2024-09-17T07:10:26.149844+0000","last_peered":"2024-09-17T07:10:26.149844+0000","last_clean":"2024-09-17T07:10:26.149844+0000","last_became_active":"2024-09-17T07:09:24.393375+0000","last_became_peered":"2024-09-17T07:09:24.393375+0000","last_unstale":"2024-09-17T07:10:26.149844+0000","last_undegraded":"2024-09-17T07:10:26.149844+0000","last_fullsized":"2024-09-17T07:10:26.149844+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:03.726764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:08:03.726764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002251201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":191,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.150345+0000","last_change":"2024-09-17T07:09:24.399627+0000","last_active":"2024-09-17T07:10:26.150345+0000","last_peered":"2024-09-17T07:10:26.150345+0000","last_clean":"2024-09-17T07:10:26.150345+0000","last_became_active":"2024-09-17T07:09:24.398659+0000","last_became_peered":"2024-09-17T07:09:24.398659+0000","last_unstale":"2024-09-17T07:10:26.150345+0000","last_undegraded":"2024-09-17T07:10:26.150345+0000","last_fullsized":"2024-09-17T07:10:26.150345+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:25.588903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:08:25.588903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021632869999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"210'10766","reported_seq":12901,"reported_epoch":210,"state":"active+remapped+backfilling","last_fresh":"2024-09-17T07:10:28.477352+0000","last_change":"2024-09-17T07:10:25.475475+0000","last_active":"2024-09-17T07:10:28.477352+0000","last_peered":"2024-09-17T07:10:28.477352+0000","last_clean":"2024-09-17T07:09:23.369855+0000","last_became_active":"2024-09-17T07:09:25.456528+0000","last_became_peered":"2024-09-17T07:09:25.456528+0000","last_unstale":"2024-09-17T07:10:28.477352+0000","last_undegraded":"2024-09-17T07:10:28.477352+0000","last_fullsized":"2024-09-17T07:10:28.477352+0000","mapping_epoch":195,"log_start":"191'6269","ondisk_log_start":"191'6269","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_clean_scrub_stamp":"2024-09-17T07:07:59.220681+0000","objects_scrubbed":0,"log_size":4497,"log_dups_size":0,"ondisk_log_size":4497,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":359268352,"num_objects":5482,"num_object_clones":0,"num_object_copies":16446,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4467,"num_objects_unfound":0,"num_objects_dirty":5482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10766,"num_write_kb":519936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1035,"num_bytes_recovered":67829760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,3],"avail_no_missing":["1","2","3"],"object_location_counts":[{"shards":"1,2,3","objects":5482}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"75'19","reported_seq":207,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-17T07:10:18.167642+0000","last_change":"2024-09-17T07:07:17.989796+0000","last_active":"2024-09-17T07:10:18.167642+0000","last_peered":"2024-09-17T07:10:18.167642+0000","last_clean":"2024-09-17T07:10:18.167642+0000","last_became_active":"2024-09-17T07:02:14.027389+0000","last_became_peered":"2024-09-17T07:02:14.027389+0000","last_unstale":"2024-09-17T07:10:18.167642+0000","last_undegraded":"2024-09-17T07:10:18.167642+0000","last_fullsized":"2024-09-17T07:10:18.167642+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-09-17T07:07:17.989435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:07:17.989435+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0043356940000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":188,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.156945+0000","last_change":"2024-09-17T07:08:24.072911+0000","last_active":"2024-09-17T07:10:26.156945+0000","last_peered":"2024-09-17T07:10:26.156945+0000","last_clean":"2024-09-17T07:10:26.156945+0000","last_became_active":"2024-09-17T07:02:12.011917+0000","last_became_peered":"2024-09-17T07:02:12.011917+0000","last_unstale":"2024-09-17T07:10:26.156945+0000","last_undegraded":"2024-09-17T07:10:26.156945+0000","last_fullsized":"2024-09-17T07:10:26.156945+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:24.072799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:24.072799+0000","last_clean_scrub_stamp":"2024-09-17T07:08:24.072799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0047928600000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":184,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.183355+0000","last_change":"2024-09-17T07:09:15.355446+0000","last_active":"2024-09-17T07:10:26.183355+0000","last_peered":"2024-09-17T07:10:26.183355+0000","last_clean":"2024-09-17T07:10:26.183355+0000","last_became_active":"2024-09-17T07:02:16.016199+0000","last_became_peered":"2024-09-17T07:02:16.016199+0000","last_unstale":"2024-09-17T07:10:26.183355+0000","last_undegraded":"2024-09-17T07:10:26.183355+0000","last_fullsized":"2024-09-17T07:10:26.183355+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:15.355318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:09:15.355318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:31.109239+0000","scrub_duration":0.0028918149999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":181,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.153886+0000","last_change":"2024-09-17T07:08:24.840933+0000","last_active":"2024-09-17T07:10:26.153886+0000","last_peered":"2024-09-17T07:10:26.153886+0000","last_clean":"2024-09-17T07:10:26.153886+0000","last_became_active":"2024-09-17T07:02:18.364835+0000","last_became_peered":"2024-09-17T07:02:18.364835+0000","last_unstale":"2024-09-17T07:10:26.153886+0000","last_undegraded":"2024-09-17T07:10:26.153886+0000","last_fullsized":"2024-09-17T07:10:26.153886+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:24.840821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:08:24.840821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002523197,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.e","version":"206'10740","reported_seq":10788,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-17T07:10:23.538892+0000","last_change":"2024-09-17T07:08:00.759796+0000","last_active":"2024-09-17T07:10:23.538892+0000","last_peered":"2024-09-17T07:10:23.538892+0000","last_clean":"2024-09-17T07:10:23.538892+0000","last_became_active":"2024-09-17T07:08:00.759537+0000","last_became_peered":"2024-09-17T07:08:00.759537+0000","last_unstale":"2024-09-17T07:10:23.538892+0000","last_undegraded":"2024-09-17T07:10:23.538892+0000","last_fullsized":"2024-09-17T07:10:23.538892+0000","mapping_epoch":168,"log_start":"189'5651","ondisk_log_start":"189'5651","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_clean_scrub_stamp":"2024-09-17T07:07:59.220681+0000","objects_scrubbed":0,"log_size":5089,"log_dups_size":0,"ondisk_log_size":5089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":366477312,"num_objects":5592,"num_object_clones":0,"num_object_copies":16776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10740,"num_write_kb":522624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"110'15","reported_seq":205,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.155908+0000","last_change":"2024-09-17T07:08:19.835062+0000","last_active":"2024-09-17T07:10:26.155908+0000","last_peered":"2024-09-17T07:10:26.155908+0000","last_clean":"2024-09-17T07:10:26.155908+0000","last_became_active":"2024-09-17T07:02:14.010637+0000","last_became_peered":"2024-09-17T07:02:14.010637+0000","last_unstale":"2024-09-17T07:10:26.155908+0000","last_undegraded":"2024-09-17T07:10:26.155908+0000","last_fullsized":"2024-09-17T07:10:26.155908+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'15","last_scrub_stamp":"2024-09-17T07:08:19.834380+0000","last_deep_scrub":"110'15","last_deep_scrub_stamp":"2024-09-17T07:04:57.091400+0000","last_clean_scrub_stamp":"2024-09-17T07:08:19.834380+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0055436599999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":187,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.156456+0000","last_change":"2024-09-17T07:08:17.806138+0000","last_active":"2024-09-17T07:10:26.156456+0000","last_peered":"2024-09-17T07:10:26.156456+0000","last_clean":"2024-09-17T07:10:26.156456+0000","last_became_active":"2024-09-17T07:02:12.010803+0000","last_became_peered":"2024-09-17T07:02:12.010803+0000","last_unstale":"2024-09-17T07:10:26.156456+0000","last_undegraded":"2024-09-17T07:10:26.156456+0000","last_fullsized":"2024-09-17T07:10:26.156456+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:17.806001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:25.147878+0000","last_clean_scrub_stamp":"2024-09-17T07:08:17.806001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00606531,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"89'9","reported_seq":287,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-17T07:10:20.611968+0000","last_change":"2024-09-17T07:08:14.309093+0000","last_active":"2024-09-17T07:10:20.611968+0000","last_peered":"2024-09-17T07:10:20.611968+0000","last_clean":"2024-09-17T07:10:20.611968+0000","last_became_active":"2024-09-17T07:02:16.021360+0000","last_became_peered":"2024-09-17T07:02:16.021360+0000","last_unstale":"2024-09-17T07:10:20.611968+0000","last_undegraded":"2024-09-17T07:10:20.611968+0000","last_fullsized":"2024-09-17T07:10:20.611968+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'9","last_scrub_stamp":"2024-09-17T07:08:14.308970+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-09-17T07:03:29.961439+0000","last_clean_scrub_stamp":"2024-09-17T07:08:14.308970+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001890011,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"75'1","reported_seq":193,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-17T07:10:23.224403+0000","last_change":"2024-09-17T07:09:24.411824+0000","last_active":"2024-09-17T07:10:23.224403+0000","last_peered":"2024-09-17T07:10:23.224403+0000","last_clean":"2024-09-17T07:10:23.224403+0000","last_became_active":"2024-09-17T07:09:24.399565+0000","last_became_peered":"2024-09-17T07:09:24.399565+0000","last_unstale":"2024-09-17T07:10:23.224403+0000","last_undegraded":"2024-09-17T07:10:23.224403+0000","last_fullsized":"2024-09-17T07:10:23.224403+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-17T07:08:25.355935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:08:25.355935+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0052992940000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"210'10703","reported_seq":24747,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-17T07:10:28.148766+0000","last_change":"2024-09-17T07:10:25.165785+0000","last_active":"2024-09-17T07:10:28.148766+0000","last_peered":"2024-09-17T07:10:28.148766+0000","last_clean":"2024-09-17T07:10:28.148766+0000","last_became_active":"2024-09-17T07:10:25.147242+0000","last_became_peered":"2024-09-17T07:10:25.147242+0000","last_unstale":"2024-09-17T07:10:28.148766+0000","last_undegraded":"2024-09-17T07:10:28.148766+0000","last_fullsized":"2024-09-17T07:10:28.148766+0000","mapping_epoch":207,"log_start":"190'5954","ondisk_log_start":"190'5954","created":168,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_clean_scrub_stamp":"2024-09-17T07:07:59.220681+0000","objects_scrubbed":0,"log_size":4749,"log_dups_size":0,"ondisk_log_size":4749,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":361168896,"num_objects":5511,"num_object_clones":0,"num_object_copies":16533,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10703,"num_write_kb":518848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6986,"num_bytes_recovered":457834496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"196'29","reported_seq":363,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.156381+0000","last_change":"2024-09-17T07:09:24.422580+0000","last_active":"2024-09-17T07:10:26.156381+0000","last_peered":"2024-09-17T07:10:26.156381+0000","last_clean":"2024-09-17T07:10:26.156381+0000","last_became_active":"2024-09-17T07:09:24.403267+0000","last_became_peered":"2024-09-17T07:09:24.403267+0000","last_unstale":"2024-09-17T07:10:26.156381+0000","last_undegraded":"2024-09-17T07:10:26.156381+0000","last_fullsized":"2024-09-17T07:10:26.156381+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'28","last_scrub_stamp":"2024-09-17T07:08:26.795647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:08:26.795647+0000","objects_scrubbed":10,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0043656659999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":198,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.151987+0000","last_change":"2024-09-17T07:09:19.253480+0000","last_active":"2024-09-17T07:10:26.151987+0000","last_peered":"2024-09-17T07:10:26.151987+0000","last_clean":"2024-09-17T07:10:26.151987+0000","last_became_active":"2024-09-17T07:02:12.008802+0000","last_became_peered":"2024-09-17T07:02:12.008802+0000","last_unstale":"2024-09-17T07:10:26.151987+0000","last_undegraded":"2024-09-17T07:10:26.151987+0000","last_fullsized":"2024-09-17T07:10:26.151987+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:19.253385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:09:19.253385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:36.878327+0000","scrub_duration":0.0017535820000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":180,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-17T07:10:18.167978+0000","last_change":"2024-09-17T07:08:04.266668+0000","last_active":"2024-09-17T07:10:18.167978+0000","last_peered":"2024-09-17T07:10:18.167978+0000","last_clean":"2024-09-17T07:10:18.167978+0000","last_became_active":"2024-09-17T07:02:16.016606+0000","last_became_peered":"2024-09-17T07:02:16.016606+0000","last_unstale":"2024-09-17T07:10:18.167978+0000","last_undegraded":"2024-09-17T07:10:18.167978+0000","last_fullsized":"2024-09-17T07:10:18.167978+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:04.266574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:08:04.266574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014388140000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":27,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.150614+0000","last_change":"2024-09-17T07:09:24.403411+0000","last_active":"2024-09-17T07:10:26.150614+0000","last_peered":"2024-09-17T07:10:26.150614+0000","last_clean":"2024-09-17T07:10:26.150614+0000","last_became_active":"2024-09-17T07:09:24.398878+0000","last_became_peered":"2024-09-17T07:09:24.398878+0000","last_unstale":"2024-09-17T07:10:26.150614+0000","last_undegraded":"2024-09-17T07:10:26.150614+0000","last_fullsized":"2024-09-17T07:10:26.150614+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:08.849142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:08:08.849142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.b","version":"210'10952","reported_seq":25687,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-17T07:10:28.106827+0000","last_change":"2024-09-17T07:10:18.199701+0000","last_active":"2024-09-17T07:10:28.106827+0000","last_peered":"2024-09-17T07:10:28.106827+0000","last_clean":"2024-09-17T07:10:28.106827+0000","last_became_active":"2024-09-17T07:10:18.175774+0000","last_became_peered":"2024-09-17T07:10:18.175774+0000","last_unstale":"2024-09-17T07:10:28.106827+0000","last_undegraded":"2024-09-17T07:10:28.106827+0000","last_fullsized":"2024-09-17T07:10:28.106827+0000","mapping_epoch":204,"log_start":"190'6170","ondisk_log_start":"190'6170","created":168,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_clean_scrub_stamp":"2024-09-17T07:07:59.220681+0000","objects_scrubbed":0,"log_size":4782,"log_dups_size":0,"ondisk_log_size":4782,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370147328,"num_objects":5648,"num_object_clones":0,"num_object_copies":16944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10952,"num_write_kb":531200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7331,"num_bytes_recovered":480444416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.7","version":"75'13","reported_seq":212,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.154684+0000","last_change":"2024-09-17T07:08:05.678169+0000","last_active":"2024-09-17T07:10:26.154684+0000","last_peered":"2024-09-17T07:10:26.154684+0000","last_clean":"2024-09-17T07:10:26.154684+0000","last_became_active":"2024-09-17T07:02:14.019838+0000","last_became_peered":"2024-09-17T07:02:14.019838+0000","last_unstale":"2024-09-17T07:10:26.154684+0000","last_undegraded":"2024-09-17T07:10:26.154684+0000","last_fullsized":"2024-09-17T07:10:26.154684+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'13","last_scrub_stamp":"2024-09-17T07:08:05.678063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:08:05.678063+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0024636459999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"76'5","reported_seq":685,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-17T07:10:27.150099+0000","last_change":"2024-09-17T07:08:29.818116+0000","last_active":"2024-09-17T07:10:27.150099+0000","last_peered":"2024-09-17T07:10:27.150099+0000","last_clean":"2024-09-17T07:10:27.150099+0000","last_became_active":"2024-09-17T07:02:12.006883+0000","last_became_peered":"2024-09-17T07:02:12.006883+0000","last_unstale":"2024-09-17T07:10:27.150099+0000","last_undegraded":"2024-09-17T07:10:27.150099+0000","last_fullsized":"2024-09-17T07:10:27.150099+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'5","last_scrub_stamp":"2024-09-17T07:08:29.817785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:08:29.817785+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034070260000000001,"objects_trimmed":0,"snaptrim_duration":0.00033739799999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":489,"num_read_kb":484,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":25,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-17T07:10:18.167016+0000","last_change":"2024-09-17T07:09:24.393139+0000","last_active":"2024-09-17T07:10:18.167016+0000","last_peered":"2024-09-17T07:10:18.167016+0000","last_clean":"2024-09-17T07:10:18.167016+0000","last_became_active":"2024-09-17T07:09:24.392850+0000","last_became_peered":"2024-09-17T07:09:24.392850+0000","last_unstale":"2024-09-17T07:10:18.167016+0000","last_undegraded":"2024-09-17T07:10:18.167016+0000","last_fullsized":"2024-09-17T07:10:18.167016+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:23.833170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:23.833170+0000","last_clean_scrub_stamp":"2024-09-17T07:08:23.833170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":184,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.153248+0000","last_change":"2024-09-17T07:08:14.935764+0000","last_active":"2024-09-17T07:10:26.153248+0000","last_peered":"2024-09-17T07:10:26.153248+0000","last_clean":"2024-09-17T07:10:26.153248+0000","last_became_active":"2024-09-17T07:02:16.014492+0000","last_became_peered":"2024-09-17T07:02:16.014492+0000","last_unstale":"2024-09-17T07:10:26.153248+0000","last_undegraded":"2024-09-17T07:10:26.153248+0000","last_fullsized":"2024-09-17T07:10:26.153248+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:14.935679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:08:14.935679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001590459,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":202,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.156517+0000","last_change":"2024-09-17T07:09:24.403678+0000","last_active":"2024-09-17T07:10:26.156517+0000","last_peered":"2024-09-17T07:10:26.156517+0000","last_clean":"2024-09-17T07:10:26.156517+0000","last_became_active":"2024-09-17T07:09:24.402704+0000","last_became_peered":"2024-09-17T07:09:24.402704+0000","last_unstale":"2024-09-17T07:10:26.156517+0000","last_undegraded":"2024-09-17T07:10:26.156517+0000","last_fullsized":"2024-09-17T07:10:26.156517+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:22.853652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:09:22.853652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0041575259999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"210'10503","reported_seq":10555,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-17T07:10:28.473346+0000","last_change":"2024-09-17T07:08:00.760107+0000","last_active":"2024-09-17T07:10:28.473346+0000","last_peered":"2024-09-17T07:10:28.473346+0000","last_clean":"2024-09-17T07:10:28.473346+0000","last_became_active":"2024-09-17T07:08:00.759827+0000","last_became_peered":"2024-09-17T07:08:00.759827+0000","last_unstale":"2024-09-17T07:10:28.473346+0000","last_undegraded":"2024-09-17T07:10:28.473346+0000","last_fullsized":"2024-09-17T07:10:28.473346+0000","mapping_epoch":168,"log_start":"190'6000","ondisk_log_start":"190'6000","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_clean_scrub_stamp":"2024-09-17T07:07:59.220681+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":355139584,"num_objects":5419,"num_object_clones":0,"num_object_copies":16257,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10503,"num_write_kb":509504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"75'12","reported_seq":223,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-17T07:10:18.168885+0000","last_change":"2024-09-17T07:08:08.295553+0000","last_active":"2024-09-17T07:10:18.168885+0000","last_peered":"2024-09-17T07:10:18.168885+0000","last_clean":"2024-09-17T07:10:18.168885+0000","last_became_active":"2024-09-17T07:03:48.760405+0000","last_became_peered":"2024-09-17T07:03:48.760405+0000","last_unstale":"2024-09-17T07:10:18.168885+0000","last_undegraded":"2024-09-17T07:10:18.168885+0000","last_fullsized":"2024-09-17T07:10:18.168885+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-09-17T07:08:08.295449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:08:08.295449+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023245919999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"70'1","reported_seq":203,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-17T07:10:18.167181+0000","last_change":"2024-09-17T07:09:24.415023+0000","last_active":"2024-09-17T07:10:18.167181+0000","last_peered":"2024-09-17T07:10:18.167181+0000","last_clean":"2024-09-17T07:10:18.167181+0000","last_became_active":"2024-09-17T07:09:24.402499+0000","last_became_peered":"2024-09-17T07:09:24.402499+0000","last_unstale":"2024-09-17T07:10:18.167181+0000","last_undegraded":"2024-09-17T07:10:18.167181+0000","last_fullsized":"2024-09-17T07:10:18.167181+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-17T07:09:22.249833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:09:22.249833+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:45.077540+0000","scrub_duration":0.0061513059999999996,"objects_trimmed":0,"snaptrim_duration":0.0032298629999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":189,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.156369+0000","last_change":"2024-09-17T07:09:16.865242+0000","last_active":"2024-09-17T07:10:26.156369+0000","last_peered":"2024-09-17T07:10:26.156369+0000","last_clean":"2024-09-17T07:10:26.156369+0000","last_became_active":"2024-09-17T07:02:12.014623+0000","last_became_peered":"2024-09-17T07:02:12.014623+0000","last_unstale":"2024-09-17T07:10:26.156369+0000","last_undegraded":"2024-09-17T07:10:26.156369+0000","last_fullsized":"2024-09-17T07:10:26.156369+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:16.865147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:09:16.865147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:30.769978+0000","scrub_duration":0.005396222,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":183,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.152185+0000","last_change":"2024-09-17T07:08:18.008734+0000","last_active":"2024-09-17T07:10:26.152185+0000","last_peered":"2024-09-17T07:10:26.152185+0000","last_clean":"2024-09-17T07:10:26.152185+0000","last_became_active":"2024-09-17T07:02:16.021561+0000","last_became_peered":"2024-09-17T07:02:16.021561+0000","last_unstale":"2024-09-17T07:10:26.152185+0000","last_undegraded":"2024-09-17T07:10:26.152185+0000","last_fullsized":"2024-09-17T07:10:26.152185+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:18.008600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:08:18.008600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0044198520000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":190,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.156220+0000","last_change":"2024-09-17T07:09:24.396448+0000","last_active":"2024-09-17T07:10:26.156220+0000","last_peered":"2024-09-17T07:10:26.156220+0000","last_clean":"2024-09-17T07:10:26.156220+0000","last_became_active":"2024-09-17T07:09:24.396154+0000","last_became_peered":"2024-09-17T07:09:24.396154+0000","last_unstale":"2024-09-17T07:10:26.156220+0000","last_undegraded":"2024-09-17T07:10:26.156220+0000","last_fullsized":"2024-09-17T07:10:26.156220+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:17.192238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:09:17.192238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.058290885000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"210'10439","reported_seq":10632,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-17T07:10:28.115648+0000","last_change":"2024-09-17T07:09:23.077196+0000","last_active":"2024-09-17T07:10:28.115648+0000","last_peered":"2024-09-17T07:10:28.115648+0000","last_clean":"2024-09-17T07:10:28.115648+0000","last_became_active":"2024-09-17T07:08:00.242803+0000","last_became_peered":"2024-09-17T07:08:00.242803+0000","last_unstale":"2024-09-17T07:10:28.115648+0000","last_undegraded":"2024-09-17T07:10:28.115648+0000","last_fullsized":"2024-09-17T07:10:28.115648+0000","mapping_epoch":168,"log_start":"189'5455","ondisk_log_start":"189'5455","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"193'6546","last_scrub_stamp":"2024-09-17T07:09:23.076982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_clean_scrub_stamp":"2024-09-17T07:09:23.076982+0000","objects_scrubbed":6477,"log_size":4984,"log_dups_size":0,"ondisk_log_size":4984,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:40.229937+0000","scrub_duration":1.8549729770000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":351862784,"num_objects":5369,"num_object_clones":0,"num_object_copies":16107,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10439,"num_write_kb":505856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"75'14","reported_seq":208,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.183840+0000","last_change":"2024-09-17T07:08:28.485668+0000","last_active":"2024-09-17T07:10:26.183840+0000","last_peered":"2024-09-17T07:10:26.183840+0000","last_clean":"2024-09-17T07:10:26.183840+0000","last_became_active":"2024-09-17T07:02:14.028857+0000","last_became_peered":"2024-09-17T07:02:14.028857+0000","last_unstale":"2024-09-17T07:10:26.183840+0000","last_undegraded":"2024-09-17T07:10:26.183840+0000","last_fullsized":"2024-09-17T07:10:26.183840+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'14","last_scrub_stamp":"2024-09-17T07:08:28.485338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:08:28.485338+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037222499999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"72'2","reported_seq":46,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.151834+0000","last_change":"2024-09-17T07:09:24.420989+0000","last_active":"2024-09-17T07:10:26.151834+0000","last_peered":"2024-09-17T07:10:26.151834+0000","last_clean":"2024-09-17T07:10:26.151834+0000","last_became_active":"2024-09-17T07:09:24.404532+0000","last_became_peered":"2024-09-17T07:09:24.404532+0000","last_unstale":"2024-09-17T07:10:26.151834+0000","last_undegraded":"2024-09-17T07:10:26.151834+0000","last_fullsized":"2024-09-17T07:10:26.151834+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'2","last_scrub_stamp":"2024-09-17T07:08:15.763727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:08:15.763727+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":198,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.152119+0000","last_change":"2024-09-17T07:09:16.180998+0000","last_active":"2024-09-17T07:10:26.152119+0000","last_peered":"2024-09-17T07:10:26.152119+0000","last_clean":"2024-09-17T07:10:26.152119+0000","last_became_active":"2024-09-17T07:02:12.004606+0000","last_became_peered":"2024-09-17T07:02:12.004606+0000","last_unstale":"2024-09-17T07:10:26.152119+0000","last_undegraded":"2024-09-17T07:10:26.152119+0000","last_fullsized":"2024-09-17T07:10:26.152119+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:16.180756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:09:16.180756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.014479524000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"78'128","reported_seq":437,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.156652+0000","last_change":"2024-09-17T07:08:14.128641+0000","last_active":"2024-09-17T07:10:26.156652+0000","last_peered":"2024-09-17T07:10:26.156652+0000","last_clean":"2024-09-17T07:10:26.156652+0000","last_became_active":"2024-09-17T07:02:03.020675+0000","last_became_peered":"2024-09-17T07:02:03.020675+0000","last_unstale":"2024-09-17T07:10:26.156652+0000","last_undegraded":"2024-09-17T07:10:26.156652+0000","last_fullsized":"2024-09-17T07:10:26.156652+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'128","last_scrub_stamp":"2024-09-17T07:08:14.128519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:00:09.390381+0000","last_clean_scrub_stamp":"2024-09-17T07:08:14.128519+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021141889999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":184,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.156878+0000","last_change":"2024-09-17T07:08:05.064380+0000","last_active":"2024-09-17T07:10:26.156878+0000","last_peered":"2024-09-17T07:10:26.156878+0000","last_clean":"2024-09-17T07:10:26.156878+0000","last_became_active":"2024-09-17T07:02:16.016031+0000","last_became_peered":"2024-09-17T07:02:16.016031+0000","last_unstale":"2024-09-17T07:10:26.156878+0000","last_undegraded":"2024-09-17T07:10:26.156878+0000","last_fullsized":"2024-09-17T07:10:26.156878+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:05.064241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:08:05.064241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015017990000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":190,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.152450+0000","last_change":"2024-09-17T07:09:24.400356+0000","last_active":"2024-09-17T07:10:26.152450+0000","last_peered":"2024-09-17T07:10:26.152450+0000","last_clean":"2024-09-17T07:10:26.152450+0000","last_became_active":"2024-09-17T07:09:24.400041+0000","last_became_peered":"2024-09-17T07:09:24.400041+0000","last_unstale":"2024-09-17T07:10:26.152450+0000","last_undegraded":"2024-09-17T07:10:26.152450+0000","last_fullsized":"2024-09-17T07:10:26.152450+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:15.194496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:58.670571+0000","last_clean_scrub_stamp":"2024-09-17T07:09:15.194496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0091630700000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"206'10724","reported_seq":10772,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-17T07:10:23.512099+0000","last_change":"2024-09-17T07:08:00.599954+0000","last_active":"2024-09-17T07:10:23.512099+0000","last_peered":"2024-09-17T07:10:23.512099+0000","last_clean":"2024-09-17T07:10:23.512099+0000","last_became_active":"2024-09-17T07:08:00.599685+0000","last_became_peered":"2024-09-17T07:08:00.599685+0000","last_unstale":"2024-09-17T07:10:23.512099+0000","last_undegraded":"2024-09-17T07:10:23.512099+0000","last_fullsized":"2024-09-17T07:10:23.512099+0000","mapping_epoch":168,"log_start":"188'5641","ondisk_log_start":"188'5641","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_clean_scrub_stamp":"2024-09-17T07:07:59.220681+0000","objects_scrubbed":0,"log_size":5083,"log_dups_size":0,"ondisk_log_size":5083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":365297664,"num_objects":5574,"num_object_clones":0,"num_object_copies":16722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10724,"num_write_kb":521536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"75'19","reported_seq":217,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.183548+0000","last_change":"2024-09-17T07:09:11.299059+0000","last_active":"2024-09-17T07:10:26.183548+0000","last_peered":"2024-09-17T07:10:26.183548+0000","last_clean":"2024-09-17T07:10:26.183548+0000","last_became_active":"2024-09-17T07:02:14.027313+0000","last_became_peered":"2024-09-17T07:02:14.027313+0000","last_unstale":"2024-09-17T07:10:26.183548+0000","last_undegraded":"2024-09-17T07:10:26.183548+0000","last_fullsized":"2024-09-17T07:10:26.183548+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-09-17T07:09:11.298950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:09:11.298950+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0052578449999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":200,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.154087+0000","last_change":"2024-09-17T07:10:24.223473+0000","last_active":"2024-09-17T07:10:26.154087+0000","last_peered":"2024-09-17T07:10:26.154087+0000","last_clean":"2024-09-17T07:10:26.154087+0000","last_became_active":"2024-09-17T07:02:12.012037+0000","last_became_peered":"2024-09-17T07:02:12.012037+0000","last_unstale":"2024-09-17T07:10:26.154087+0000","last_undegraded":"2024-09-17T07:10:26.154087+0000","last_fullsized":"2024-09-17T07:10:26.154087+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:15.952078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:06:39.322535+0000","last_clean_scrub_stamp":"2024-09-17T07:08:15.952078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017662470000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":206,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.283094+0000","last_change":"2024-09-17T07:10:26.283094+0000","last_active":"2024-09-17T07:10:26.283094+0000","last_peered":"2024-09-17T07:10:26.283094+0000","last_clean":"2024-09-17T07:10:26.283094+0000","last_became_active":"2024-09-17T07:02:16.014631+0000","last_became_peered":"2024-09-17T07:02:16.014631+0000","last_unstale":"2024-09-17T07:10:26.283094+0000","last_undegraded":"2024-09-17T07:10:26.283094+0000","last_fullsized":"2024-09-17T07:10:26.283094+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:10:26.282797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:10:26.282797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:11:52.164708+0000","scrub_duration":0.0032940579999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":191,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.156337+0000","last_change":"2024-09-17T07:09:24.404437+0000","last_active":"2024-09-17T07:10:26.156337+0000","last_peered":"2024-09-17T07:10:26.156337+0000","last_clean":"2024-09-17T07:10:26.156337+0000","last_became_active":"2024-09-17T07:09:24.403904+0000","last_became_peered":"2024-09-17T07:09:24.403904+0000","last_unstale":"2024-09-17T07:10:26.156337+0000","last_undegraded":"2024-09-17T07:10:26.156337+0000","last_fullsized":"2024-09-17T07:10:26.156337+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:10.781244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:09:10.781244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:39.854595+0000","scrub_duration":0.010463343,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.7","version":"210'10884","reported_seq":24832,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-17T07:10:28.464540+0000","last_change":"2024-09-17T07:10:24.242036+0000","last_active":"2024-09-17T07:10:28.464540+0000","last_peered":"2024-09-17T07:10:28.464540+0000","last_clean":"2024-09-17T07:10:28.464540+0000","last_became_active":"2024-09-17T07:10:24.227488+0000","last_became_peered":"2024-09-17T07:10:24.227488+0000","last_unstale":"2024-09-17T07:10:28.464540+0000","last_undegraded":"2024-09-17T07:10:28.464540+0000","last_fullsized":"2024-09-17T07:10:28.464540+0000","mapping_epoch":206,"log_start":"191'6385","ondisk_log_start":"191'6385","created":168,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_clean_scrub_stamp":"2024-09-17T07:07:59.220681+0000","objects_scrubbed":0,"log_size":4499,"log_dups_size":0,"ondisk_log_size":4499,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":367394816,"num_objects":5606,"num_object_clones":0,"num_object_copies":16818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10884,"num_write_kb":527680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6937,"num_bytes_recovered":454623232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"75'9","reported_seq":198,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-17T07:10:18.167568+0000","last_change":"2024-09-17T07:08:30.220173+0000","last_active":"2024-09-17T07:10:18.167568+0000","last_peered":"2024-09-17T07:10:18.167568+0000","last_clean":"2024-09-17T07:10:18.167568+0000","last_became_active":"2024-09-17T07:02:14.014939+0000","last_became_peered":"2024-09-17T07:02:14.014939+0000","last_unstale":"2024-09-17T07:10:18.167568+0000","last_undegraded":"2024-09-17T07:10:18.167568+0000","last_fullsized":"2024-09-17T07:10:18.167568+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:08:30.219974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:08:30.219974+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033974780000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":196,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.156514+0000","last_change":"2024-09-17T07:09:24.391612+0000","last_active":"2024-09-17T07:10:26.156514+0000","last_peered":"2024-09-17T07:10:26.156514+0000","last_clean":"2024-09-17T07:10:26.156514+0000","last_became_active":"2024-09-17T07:09:24.391207+0000","last_became_peered":"2024-09-17T07:09:24.391207+0000","last_unstale":"2024-09-17T07:10:26.156514+0000","last_undegraded":"2024-09-17T07:10:26.156514+0000","last_fullsized":"2024-09-17T07:10:26.156514+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:20.806468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:06:30.038104+0000","last_clean_scrub_stamp":"2024-09-17T07:08:20.806468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.005853245,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":147,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.150014+0000","last_change":"2024-09-17T07:09:24.404480+0000","last_active":"2024-09-17T07:10:26.150014+0000","last_peered":"2024-09-17T07:10:26.150014+0000","last_clean":"2024-09-17T07:10:26.150014+0000","last_became_active":"2024-09-17T07:09:24.396642+0000","last_became_peered":"2024-09-17T07:09:24.396642+0000","last_unstale":"2024-09-17T07:10:26.150014+0000","last_undegraded":"2024-09-17T07:10:26.150014+0000","last_fullsized":"2024-09-17T07:10:26.150014+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:19.789134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:07:19.789134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010505654,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":178,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-17T07:10:18.167091+0000","last_change":"2024-09-17T07:09:08.316161+0000","last_active":"2024-09-17T07:10:18.167091+0000","last_peered":"2024-09-17T07:10:18.167091+0000","last_clean":"2024-09-17T07:10:18.167091+0000","last_became_active":"2024-09-17T07:02:18.026994+0000","last_became_peered":"2024-09-17T07:02:18.026994+0000","last_unstale":"2024-09-17T07:10:18.167091+0000","last_undegraded":"2024-09-17T07:10:18.167091+0000","last_fullsized":"2024-09-17T07:10:18.167091+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:08.315842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:58.099849+0000","last_clean_scrub_stamp":"2024-09-17T07:09:08.315842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:28.565288+0000","scrub_duration":0.0081844919999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"210'10961","reported_seq":11013,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-17T07:10:28.165758+0000","last_change":"2024-09-17T07:08:00.760307+0000","last_active":"2024-09-17T07:10:28.165758+0000","last_peered":"2024-09-17T07:10:28.165758+0000","last_clean":"2024-09-17T07:10:28.165758+0000","last_became_active":"2024-09-17T07:08:00.759662+0000","last_became_peered":"2024-09-17T07:08:00.759662+0000","last_unstale":"2024-09-17T07:10:28.165758+0000","last_undegraded":"2024-09-17T07:10:28.165758+0000","last_fullsized":"2024-09-17T07:10:28.165758+0000","mapping_epoch":168,"log_start":"190'6273","ondisk_log_start":"190'6273","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_clean_scrub_stamp":"2024-09-17T07:07:59.220681+0000","objects_scrubbed":0,"log_size":4688,"log_dups_size":0,"ondisk_log_size":4688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362086400,"num_objects":5525,"num_object_clones":0,"num_object_copies":16575,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10961,"num_write_kb":527552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"111'22","reported_seq":217,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.154804+0000","last_change":"2024-09-17T07:07:29.816377+0000","last_active":"2024-09-17T07:10:26.154804+0000","last_peered":"2024-09-17T07:10:26.154804+0000","last_clean":"2024-09-17T07:10:26.154804+0000","last_became_active":"2024-09-17T07:02:14.028392+0000","last_became_peered":"2024-09-17T07:02:14.028392+0000","last_unstale":"2024-09-17T07:10:26.154804+0000","last_undegraded":"2024-09-17T07:10:26.154804+0000","last_fullsized":"2024-09-17T07:10:26.154804+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'22","last_scrub_stamp":"2024-09-17T07:07:29.816241+0000","last_deep_scrub":"111'22","last_deep_scrub_stamp":"2024-09-17T07:04:57.500450+0000","last_clean_scrub_stamp":"2024-09-17T07:07:29.816241+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034677570000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'2","reported_seq":71,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.157016+0000","last_change":"2024-09-17T07:09:24.428473+0000","last_active":"2024-09-17T07:10:26.157016+0000","last_peered":"2024-09-17T07:10:26.157016+0000","last_clean":"2024-09-17T07:10:26.157016+0000","last_became_active":"2024-09-17T07:09:24.402909+0000","last_became_peered":"2024-09-17T07:09:24.402909+0000","last_unstale":"2024-09-17T07:10:26.157016+0000","last_undegraded":"2024-09-17T07:10:26.157016+0000","last_fullsized":"2024-09-17T07:10:26.157016+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-09-17T07:09:19.720425+0000","last_deep_scrub":"89'2","last_deep_scrub_stamp":"2024-09-17T07:03:26.694366+0000","last_clean_scrub_stamp":"2024-09-17T07:09:19.720425+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:33.916032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":420,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":420,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"75'8","reported_seq":294,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.156232+0000","last_change":"2024-09-17T07:08:20.045908+0000","last_active":"2024-09-17T07:10:26.156232+0000","last_peered":"2024-09-17T07:10:26.156232+0000","last_clean":"2024-09-17T07:10:26.156232+0000","last_became_active":"2024-09-17T07:02:16.015418+0000","last_became_peered":"2024-09-17T07:02:16.015418+0000","last_unstale":"2024-09-17T07:10:26.156232+0000","last_undegraded":"2024-09-17T07:10:26.156232+0000","last_fullsized":"2024-09-17T07:10:26.156232+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:08:20.045781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:08:20.045781+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023108299999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":181,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.151619+0000","last_change":"2024-09-17T07:07:19.786753+0000","last_active":"2024-09-17T07:10:26.151619+0000","last_peered":"2024-09-17T07:10:26.151619+0000","last_clean":"2024-09-17T07:10:26.151619+0000","last_became_active":"2024-09-17T07:02:18.363253+0000","last_became_peered":"2024-09-17T07:02:18.363253+0000","last_unstale":"2024-09-17T07:10:26.151619+0000","last_undegraded":"2024-09-17T07:10:26.151619+0000","last_fullsized":"2024-09-17T07:10:26.151619+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:19.786442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:07:19.786442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011747911999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.5","version":"210'10911","reported_seq":10964,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-17T07:10:28.128405+0000","last_change":"2024-09-17T07:10:18.411537+0000","last_active":"2024-09-17T07:10:28.128405+0000","last_peered":"2024-09-17T07:10:28.128405+0000","last_clean":"2024-09-17T07:10:28.128405+0000","last_became_active":"2024-09-17T07:08:00.598353+0000","last_became_peered":"2024-09-17T07:08:00.598353+0000","last_unstale":"2024-09-17T07:10:28.128405+0000","last_undegraded":"2024-09-17T07:10:28.128405+0000","last_fullsized":"2024-09-17T07:10:28.128405+0000","mapping_epoch":168,"log_start":"189'5895","ondisk_log_start":"189'5895","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:59.220681+0000","last_clean_scrub_stamp":"2024-09-17T07:07:59.220681+0000","objects_scrubbed":0,"log_size":5016,"log_dups_size":0,"ondisk_log_size":5016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":363790336,"num_objects":5551,"num_object_clones":0,"num_object_copies":16653,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5551,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10911,"num_write_kb":526784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"150'17","reported_seq":235,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.150015+0000","last_change":"2024-09-17T07:09:24.423923+0000","last_active":"2024-09-17T07:10:26.150015+0000","last_peered":"2024-09-17T07:10:26.150015+0000","last_clean":"2024-09-17T07:10:26.150015+0000","last_became_active":"2024-09-17T07:09:24.399164+0000","last_became_peered":"2024-09-17T07:09:24.399164+0000","last_unstale":"2024-09-17T07:10:26.150015+0000","last_undegraded":"2024-09-17T07:10:26.150015+0000","last_fullsized":"2024-09-17T07:10:26.150015+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'17","last_scrub_stamp":"2024-09-17T07:08:34.714204+0000","last_deep_scrub":"150'17","last_deep_scrub_stamp":"2024-09-17T07:06:38.775788+0000","last_clean_scrub_stamp":"2024-09-17T07:08:34.714204+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0065052799999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"68'2","reported_seq":185,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.150877+0000","last_change":"2024-09-17T07:07:20.764435+0000","last_active":"2024-09-17T07:10:26.150877+0000","last_peered":"2024-09-17T07:10:26.150877+0000","last_clean":"2024-09-17T07:10:26.150877+0000","last_became_active":"2024-09-17T07:02:12.006716+0000","last_became_peered":"2024-09-17T07:02:12.006716+0000","last_unstale":"2024-09-17T07:10:26.150877+0000","last_undegraded":"2024-09-17T07:10:26.150877+0000","last_fullsized":"2024-09-17T07:10:26.150877+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-09-17T07:07:20.764283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:07:20.764283+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0060160040000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":144,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-17T07:10:18.166760+0000","last_change":"2024-09-17T07:08:10.273572+0000","last_active":"2024-09-17T07:10:18.166760+0000","last_peered":"2024-09-17T07:10:18.166760+0000","last_clean":"2024-09-17T07:10:18.166760+0000","last_became_active":"2024-09-17T07:04:15.111069+0000","last_became_peered":"2024-09-17T07:04:15.111069+0000","last_unstale":"2024-09-17T07:10:18.166760+0000","last_undegraded":"2024-09-17T07:10:18.166760+0000","last_fullsized":"2024-09-17T07:10:18.166760+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:10.273485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:08:10.273485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013521,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":29,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.156288+0000","last_change":"2024-09-17T07:10:23.221279+0000","last_active":"2024-09-17T07:10:26.156288+0000","last_peered":"2024-09-17T07:10:26.156288+0000","last_clean":"2024-09-17T07:10:26.156288+0000","last_became_active":"2024-09-17T07:09:24.397669+0000","last_became_peered":"2024-09-17T07:09:24.397669+0000","last_unstale":"2024-09-17T07:10:26.156288+0000","last_undegraded":"2024-09-17T07:10:26.156288+0000","last_fullsized":"2024-09-17T07:10:26.156288+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:07.878506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:08:07.878506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"75'9","reported_seq":217,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-17T07:10:18.169022+0000","last_change":"2024-09-17T07:09:24.429695+0000","last_active":"2024-09-17T07:10:18.169022+0000","last_peered":"2024-09-17T07:10:18.169022+0000","last_clean":"2024-09-17T07:10:18.169022+0000","last_became_active":"2024-09-17T07:09:24.408875+0000","last_became_peered":"2024-09-17T07:09:24.408875+0000","last_unstale":"2024-09-17T07:10:18.169022+0000","last_undegraded":"2024-09-17T07:10:18.169022+0000","last_fullsized":"2024-09-17T07:10:18.169022+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:08:19.316248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:08:19.316248+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0059518610000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":199,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.152755+0000","last_change":"2024-09-17T07:08:17.976403+0000","last_active":"2024-09-17T07:10:26.152755+0000","last_peered":"2024-09-17T07:10:26.152755+0000","last_clean":"2024-09-17T07:10:26.152755+0000","last_became_active":"2024-09-17T07:02:12.011327+0000","last_became_peered":"2024-09-17T07:02:12.011327+0000","last_unstale":"2024-09-17T07:10:26.152755+0000","last_undegraded":"2024-09-17T07:10:26.152755+0000","last_fullsized":"2024-09-17T07:10:26.152755+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:17.976267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:17.976267+0000","last_clean_scrub_stamp":"2024-09-17T07:08:17.976267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035138019999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":27,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.154491+0000","last_change":"2024-09-17T07:09:24.409485+0000","last_active":"2024-09-17T07:10:26.154491+0000","last_peered":"2024-09-17T07:10:26.154491+0000","last_clean":"2024-09-17T07:10:26.154491+0000","last_became_active":"2024-09-17T07:09:24.407806+0000","last_became_peered":"2024-09-17T07:09:24.407806+0000","last_unstale":"2024-09-17T07:10:26.154491+0000","last_undegraded":"2024-09-17T07:10:26.154491+0000","last_fullsized":"2024-09-17T07:10:26.154491+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:22.799056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:08:22.799056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":204,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.153608+0000","last_change":"2024-09-17T07:09:23.379011+0000","last_active":"2024-09-17T07:10:26.153608+0000","last_peered":"2024-09-17T07:10:26.153608+0000","last_clean":"2024-09-17T07:10:26.153608+0000","last_became_active":"2024-09-17T07:02:18.017825+0000","last_became_peered":"2024-09-17T07:02:18.017825+0000","last_unstale":"2024-09-17T07:10:26.153608+0000","last_undegraded":"2024-09-17T07:10:26.153608+0000","last_fullsized":"2024-09-17T07:10:26.153608+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:23.378805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:09:23.378805+0000","last_clean_scrub_stamp":"2024-09-17T07:09:23.378805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:33.079700+0000","scrub_duration":0.0078152310000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":189,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.183328+0000","last_change":"2024-09-17T07:10:18.211310+0000","last_active":"2024-09-17T07:10:26.183328+0000","last_peered":"2024-09-17T07:10:26.183328+0000","last_clean":"2024-09-17T07:10:26.183328+0000","last_became_active":"2024-09-17T07:02:12.008065+0000","last_became_peered":"2024-09-17T07:02:12.008065+0000","last_unstale":"2024-09-17T07:10:26.183328+0000","last_undegraded":"2024-09-17T07:10:26.183328+0000","last_fullsized":"2024-09-17T07:10:26.183328+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:04.480504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:08:04.480504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013838889999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"75'6","reported_seq":57,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.156556+0000","last_change":"2024-09-17T07:09:24.433976+0000","last_active":"2024-09-17T07:10:26.156556+0000","last_peered":"2024-09-17T07:10:26.156556+0000","last_clean":"2024-09-17T07:10:26.156556+0000","last_became_active":"2024-09-17T07:09:24.405385+0000","last_became_peered":"2024-09-17T07:09:24.405385+0000","last_unstale":"2024-09-17T07:10:26.156556+0000","last_undegraded":"2024-09-17T07:10:26.156556+0000","last_fullsized":"2024-09-17T07:10:26.156556+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'6","last_scrub_stamp":"2024-09-17T07:08:33.821542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:08:33.821542+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":192,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.152044+0000","last_change":"2024-09-17T07:09:24.399278+0000","last_active":"2024-09-17T07:10:26.152044+0000","last_peered":"2024-09-17T07:10:26.152044+0000","last_clean":"2024-09-17T07:10:26.152044+0000","last_became_active":"2024-09-17T07:09:24.398922+0000","last_became_peered":"2024-09-17T07:09:24.398922+0000","last_unstale":"2024-09-17T07:10:26.152044+0000","last_undegraded":"2024-09-17T07:10:26.152044+0000","last_fullsized":"2024-09-17T07:10:26.152044+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:06.932318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:08:06.932318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017321260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":182,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.156468+0000","last_change":"2024-09-17T07:08:33.926391+0000","last_active":"2024-09-17T07:10:26.156468+0000","last_peered":"2024-09-17T07:10:26.156468+0000","last_clean":"2024-09-17T07:10:26.156468+0000","last_became_active":"2024-09-17T07:02:18.365419+0000","last_became_peered":"2024-09-17T07:02:18.365419+0000","last_unstale":"2024-09-17T07:10:26.156468+0000","last_undegraded":"2024-09-17T07:10:26.156468+0000","last_fullsized":"2024-09-17T07:10:26.156468+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:33.926286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:08:33.926286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014019519999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"75'10","reported_seq":67,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.156299+0000","last_change":"2024-09-17T07:09:24.435532+0000","last_active":"2024-09-17T07:10:26.156299+0000","last_peered":"2024-09-17T07:10:26.156299+0000","last_clean":"2024-09-17T07:10:26.156299+0000","last_became_active":"2024-09-17T07:09:24.405944+0000","last_became_peered":"2024-09-17T07:09:24.405944+0000","last_unstale":"2024-09-17T07:10:26.156299+0000","last_undegraded":"2024-09-17T07:10:26.156299+0000","last_fullsized":"2024-09-17T07:10:26.156299+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-09-17T07:08:18.723247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:08:18.723247+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":207,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.151912+0000","last_change":"2024-09-17T07:09:24.403615+0000","last_active":"2024-09-17T07:10:26.151912+0000","last_peered":"2024-09-17T07:10:26.151912+0000","last_clean":"2024-09-17T07:10:26.151912+0000","last_became_active":"2024-09-17T07:09:24.402355+0000","last_became_peered":"2024-09-17T07:09:24.402355+0000","last_unstale":"2024-09-17T07:10:26.151912+0000","last_undegraded":"2024-09-17T07:10:26.151912+0000","last_fullsized":"2024-09-17T07:10:26.151912+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:25.065984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:08:25.065984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0052651929999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"163'9","reported_seq":291,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.152113+0000","last_change":"2024-09-17T07:07:34.933956+0000","last_active":"2024-09-17T07:10:26.152113+0000","last_peered":"2024-09-17T07:10:26.152113+0000","last_clean":"2024-09-17T07:10:26.152113+0000","last_became_active":"2024-09-17T07:02:16.022583+0000","last_became_peered":"2024-09-17T07:02:16.022583+0000","last_unstale":"2024-09-17T07:10:26.152113+0000","last_undegraded":"2024-09-17T07:10:26.152113+0000","last_fullsized":"2024-09-17T07:10:26.152113+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'9","last_scrub_stamp":"2024-09-17T07:07:34.933808+0000","last_deep_scrub":"163'9","last_deep_scrub_stamp":"2024-09-17T07:07:34.933808+0000","last_clean_scrub_stamp":"2024-09-17T07:07:34.933808+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.019504579000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":187,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-17T07:10:18.167215+0000","last_change":"2024-09-17T07:09:24.409388+0000","last_active":"2024-09-17T07:10:18.167215+0000","last_peered":"2024-09-17T07:10:18.167215+0000","last_clean":"2024-09-17T07:10:18.167215+0000","last_became_active":"2024-09-17T07:09:24.409051+0000","last_became_peered":"2024-09-17T07:09:24.409051+0000","last_unstale":"2024-09-17T07:10:18.167215+0000","last_undegraded":"2024-09-17T07:10:18.167215+0000","last_fullsized":"2024-09-17T07:10:18.167215+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:12.338453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:09:12.338453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0011356249999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"75'9","reported_seq":231,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.152143+0000","last_change":"2024-09-17T07:09:24.418573+0000","last_active":"2024-09-17T07:10:26.152143+0000","last_peered":"2024-09-17T07:10:26.152143+0000","last_clean":"2024-09-17T07:10:26.152143+0000","last_became_active":"2024-09-17T07:09:24.406333+0000","last_became_peered":"2024-09-17T07:09:24.406333+0000","last_unstale":"2024-09-17T07:10:26.152143+0000","last_undegraded":"2024-09-17T07:10:26.152143+0000","last_fullsized":"2024-09-17T07:10:26.152143+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:09:09.142335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:09:09.142335+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0041253560000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":193,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-17T07:10:18.167754+0000","last_change":"2024-09-17T07:09:24.402987+0000","last_active":"2024-09-17T07:10:18.167754+0000","last_peered":"2024-09-17T07:10:18.167754+0000","last_clean":"2024-09-17T07:10:18.167754+0000","last_became_active":"2024-09-17T07:09:24.402704+0000","last_became_peered":"2024-09-17T07:09:24.402704+0000","last_unstale":"2024-09-17T07:10:18.167754+0000","last_undegraded":"2024-09-17T07:10:18.167754+0000","last_fullsized":"2024-09-17T07:10:18.167754+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:15.356187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:08:15.356187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015925290000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"196'9","reported_seq":66,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.157049+0000","last_change":"2024-09-17T07:10:17.173811+0000","last_active":"2024-09-17T07:10:26.157049+0000","last_peered":"2024-09-17T07:10:26.157049+0000","last_clean":"2024-09-17T07:10:26.157049+0000","last_became_active":"2024-09-17T07:09:24.406267+0000","last_became_peered":"2024-09-17T07:09:24.406267+0000","last_unstale":"2024-09-17T07:10:26.157049+0000","last_undegraded":"2024-09-17T07:10:26.157049+0000","last_fullsized":"2024-09-17T07:10:26.157049+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:07:32.669921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:07:32.669921+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":182,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.150690+0000","last_change":"2024-09-17T07:08:16.703606+0000","last_active":"2024-09-17T07:10:26.150690+0000","last_peered":"2024-09-17T07:10:26.150690+0000","last_clean":"2024-09-17T07:10:26.150690+0000","last_became_active":"2024-09-17T07:02:18.018716+0000","last_became_peered":"2024-09-17T07:02:18.018716+0000","last_unstale":"2024-09-17T07:10:26.150690+0000","last_undegraded":"2024-09-17T07:10:26.150690+0000","last_fullsized":"2024-09-17T07:10:26.150690+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:16.703532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:08:16.703532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019164340000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"75'9","reported_seq":201,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.155817+0000","last_change":"2024-09-17T07:08:16.756725+0000","last_active":"2024-09-17T07:10:26.155817+0000","last_peered":"2024-09-17T07:10:26.155817+0000","last_clean":"2024-09-17T07:10:26.155817+0000","last_became_active":"2024-09-17T07:02:14.010289+0000","last_became_peered":"2024-09-17T07:02:14.010289+0000","last_unstale":"2024-09-17T07:10:26.155817+0000","last_undegraded":"2024-09-17T07:10:26.155817+0000","last_fullsized":"2024-09-17T07:10:26.155817+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:08:16.756185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:08:16.756185+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004786584,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":197,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.183566+0000","last_change":"2024-09-17T07:09:24.408438+0000","last_active":"2024-09-17T07:10:26.183566+0000","last_peered":"2024-09-17T07:10:26.183566+0000","last_clean":"2024-09-17T07:10:26.183566+0000","last_became_active":"2024-09-17T07:09:24.408139+0000","last_became_peered":"2024-09-17T07:09:24.408139+0000","last_unstale":"2024-09-17T07:10:26.183566+0000","last_undegraded":"2024-09-17T07:10:26.183566+0000","last_fullsized":"2024-09-17T07:10:26.183566+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:07.329897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:09:07.329897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010864379,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":23,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-17T07:10:18.167368+0000","last_change":"2024-09-17T07:09:24.399604+0000","last_active":"2024-09-17T07:10:18.167368+0000","last_peered":"2024-09-17T07:10:18.167368+0000","last_clean":"2024-09-17T07:10:18.167368+0000","last_became_active":"2024-09-17T07:09:24.399116+0000","last_became_peered":"2024-09-17T07:09:24.399116+0000","last_unstale":"2024-09-17T07:10:18.167368+0000","last_undegraded":"2024-09-17T07:10:18.167368+0000","last_fullsized":"2024-09-17T07:10:18.167368+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:17.713027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:07:17.713027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":178,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-17T07:10:18.166614+0000","last_change":"2024-09-17T07:08:18.338994+0000","last_active":"2024-09-17T07:10:18.166614+0000","last_peered":"2024-09-17T07:10:18.166614+0000","last_clean":"2024-09-17T07:10:18.166614+0000","last_became_active":"2024-09-17T07:02:18.027622+0000","last_became_peered":"2024-09-17T07:02:18.027622+0000","last_unstale":"2024-09-17T07:10:18.166614+0000","last_undegraded":"2024-09-17T07:10:18.166614+0000","last_fullsized":"2024-09-17T07:10:18.166614+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:18.338863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:08:18.338863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0025905709999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"75'11","reported_seq":205,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.150210+0000","last_change":"2024-09-17T07:08:23.583577+0000","last_active":"2024-09-17T07:10:26.150210+0000","last_peered":"2024-09-17T07:10:26.150210+0000","last_clean":"2024-09-17T07:10:26.150210+0000","last_became_active":"2024-09-17T07:02:14.016255+0000","last_became_peered":"2024-09-17T07:02:14.016255+0000","last_unstale":"2024-09-17T07:10:26.150210+0000","last_undegraded":"2024-09-17T07:10:26.150210+0000","last_fullsized":"2024-09-17T07:10:26.150210+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-09-17T07:08:23.583408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:08:23.583408+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0056203690000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"94'2","reported_seq":193,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.156482+0000","last_change":"2024-09-17T07:09:18.169710+0000","last_active":"2024-09-17T07:10:26.156482+0000","last_peered":"2024-09-17T07:10:26.156482+0000","last_clean":"2024-09-17T07:10:26.156482+0000","last_became_active":"2024-09-17T07:02:12.011461+0000","last_became_peered":"2024-09-17T07:02:12.011461+0000","last_unstale":"2024-09-17T07:10:26.156482+0000","last_undegraded":"2024-09-17T07:10:26.156482+0000","last_fullsized":"2024-09-17T07:10:26.156482+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-09-17T07:09:18.169580+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-09-17T07:07:31.717888+0000","last_clean_scrub_stamp":"2024-09-17T07:09:18.169580+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011743155,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":192,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.154921+0000","last_change":"2024-09-17T07:09:24.407995+0000","last_active":"2024-09-17T07:10:26.154921+0000","last_peered":"2024-09-17T07:10:26.154921+0000","last_clean":"2024-09-17T07:10:26.154921+0000","last_became_active":"2024-09-17T07:09:24.407362+0000","last_became_peered":"2024-09-17T07:09:24.407362+0000","last_unstale":"2024-09-17T07:10:26.154921+0000","last_undegraded":"2024-09-17T07:10:26.154921+0000","last_fullsized":"2024-09-17T07:10:26.154921+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:27.841486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:08:27.841486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037672679999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":23,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-17T07:10:18.166755+0000","last_change":"2024-09-17T07:09:24.403561+0000","last_active":"2024-09-17T07:10:18.166755+0000","last_peered":"2024-09-17T07:10:18.166755+0000","last_clean":"2024-09-17T07:10:18.166755+0000","last_became_active":"2024-09-17T07:09:24.400791+0000","last_became_peered":"2024-09-17T07:09:24.400791+0000","last_unstale":"2024-09-17T07:10:18.166755+0000","last_undegraded":"2024-09-17T07:10:18.166755+0000","last_fullsized":"2024-09-17T07:10:18.166755+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:10.805704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:08:10.805704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"75'4","reported_seq":82,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.152610+0000","last_change":"2024-09-17T07:09:24.412549+0000","last_active":"2024-09-17T07:10:26.152610+0000","last_peered":"2024-09-17T07:10:26.152610+0000","last_clean":"2024-09-17T07:10:26.152610+0000","last_became_active":"2024-09-17T07:09:24.396368+0000","last_became_peered":"2024-09-17T07:09:24.396368+0000","last_unstale":"2024-09-17T07:10:26.152610+0000","last_undegraded":"2024-09-17T07:10:26.152610+0000","last_fullsized":"2024-09-17T07:10:26.152610+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'4","last_scrub_stamp":"2024-09-17T07:09:17.769705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:09:17.769705+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015094500000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":187,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.152368+0000","last_change":"2024-09-17T07:08:04.972061+0000","last_active":"2024-09-17T07:10:26.152368+0000","last_peered":"2024-09-17T07:10:26.152368+0000","last_clean":"2024-09-17T07:10:26.152368+0000","last_became_active":"2024-09-17T07:02:12.008898+0000","last_became_peered":"2024-09-17T07:02:12.008898+0000","last_unstale":"2024-09-17T07:10:26.152368+0000","last_undegraded":"2024-09-17T07:10:26.152368+0000","last_fullsized":"2024-09-17T07:10:26.152368+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:04.971954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:08:04.971954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014792150000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":185,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.183156+0000","last_change":"2024-09-17T07:10:17.170375+0000","last_active":"2024-09-17T07:10:26.183156+0000","last_peered":"2024-09-17T07:10:26.183156+0000","last_clean":"2024-09-17T07:10:26.183156+0000","last_became_active":"2024-09-17T07:02:16.014982+0000","last_became_peered":"2024-09-17T07:02:16.014982+0000","last_unstale":"2024-09-17T07:10:26.183156+0000","last_undegraded":"2024-09-17T07:10:26.183156+0000","last_fullsized":"2024-09-17T07:10:26.183156+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:03.496602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:47.444972+0000","last_clean_scrub_stamp":"2024-09-17T07:08:03.496602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001400943,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":182,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.156173+0000","last_change":"2024-09-17T07:08:26.996645+0000","last_active":"2024-09-17T07:10:26.156173+0000","last_peered":"2024-09-17T07:10:26.156173+0000","last_clean":"2024-09-17T07:10:26.156173+0000","last_became_active":"2024-09-17T07:02:18.021014+0000","last_became_peered":"2024-09-17T07:02:18.021014+0000","last_unstale":"2024-09-17T07:10:26.156173+0000","last_undegraded":"2024-09-17T07:10:26.156173+0000","last_fullsized":"2024-09-17T07:10:26.156173+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:26.996525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:08:26.996525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001722168,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"75'11","reported_seq":71,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.156161+0000","last_change":"2024-09-17T07:09:24.428977+0000","last_active":"2024-09-17T07:10:26.156161+0000","last_peered":"2024-09-17T07:10:26.156161+0000","last_clean":"2024-09-17T07:10:26.156161+0000","last_became_active":"2024-09-17T07:09:24.404942+0000","last_became_peered":"2024-09-17T07:09:24.404942+0000","last_unstale":"2024-09-17T07:10:26.156161+0000","last_undegraded":"2024-09-17T07:10:26.156161+0000","last_fullsized":"2024-09-17T07:10:26.156161+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-09-17T07:08:25.883470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:08:25.883470+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":199,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.184164+0000","last_change":"2024-09-17T07:10:24.228045+0000","last_active":"2024-09-17T07:10:26.184164+0000","last_peered":"2024-09-17T07:10:26.184164+0000","last_clean":"2024-09-17T07:10:26.184164+0000","last_became_active":"2024-09-17T07:02:12.010195+0000","last_became_peered":"2024-09-17T07:02:12.010195+0000","last_unstale":"2024-09-17T07:10:26.184164+0000","last_undegraded":"2024-09-17T07:10:26.184164+0000","last_fullsized":"2024-09-17T07:10:26.184164+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:14.519233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:08:14.519233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001420021,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":184,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.155953+0000","last_change":"2024-09-17T07:08:32.894541+0000","last_active":"2024-09-17T07:10:26.155953+0000","last_peered":"2024-09-17T07:10:26.155953+0000","last_clean":"2024-09-17T07:10:26.155953+0000","last_became_active":"2024-09-17T07:02:16.016183+0000","last_became_peered":"2024-09-17T07:02:16.016183+0000","last_unstale":"2024-09-17T07:10:26.155953+0000","last_undegraded":"2024-09-17T07:10:26.155953+0000","last_fullsized":"2024-09-17T07:10:26.155953+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:32.894433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:08:32.894433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001707264,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":23,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-17T07:10:18.167272+0000","last_change":"2024-09-17T07:09:24.392552+0000","last_active":"2024-09-17T07:10:18.167272+0000","last_peered":"2024-09-17T07:10:18.167272+0000","last_clean":"2024-09-17T07:10:18.167272+0000","last_became_active":"2024-09-17T07:09:24.392216+0000","last_became_peered":"2024-09-17T07:09:24.392216+0000","last_unstale":"2024-09-17T07:10:18.167272+0000","last_undegraded":"2024-09-17T07:10:18.167272+0000","last_fullsized":"2024-09-17T07:10:18.167272+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:26.832514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:07:26.832514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1c","version":"75'1","reported_seq":183,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.157131+0000","last_change":"2024-09-17T07:08:22.083583+0000","last_active":"2024-09-17T07:10:26.157131+0000","last_peered":"2024-09-17T07:10:26.157131+0000","last_clean":"2024-09-17T07:10:26.157131+0000","last_became_active":"2024-09-17T07:02:18.021855+0000","last_became_peered":"2024-09-17T07:02:18.021855+0000","last_unstale":"2024-09-17T07:10:26.157131+0000","last_undegraded":"2024-09-17T07:10:26.157131+0000","last_fullsized":"2024-09-17T07:10:26.157131+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-17T07:08:22.083502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:08:22.083502+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0018653739999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":187,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.154927+0000","last_change":"2024-09-17T07:07:22.782091+0000","last_active":"2024-09-17T07:10:26.154927+0000","last_peered":"2024-09-17T07:10:26.154927+0000","last_clean":"2024-09-17T07:10:26.154927+0000","last_became_active":"2024-09-17T07:02:12.005329+0000","last_became_peered":"2024-09-17T07:02:12.005329+0000","last_unstale":"2024-09-17T07:10:26.154927+0000","last_undegraded":"2024-09-17T07:10:26.154927+0000","last_fullsized":"2024-09-17T07:10:26.154927+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:22.781739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:07:22.781739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037808680000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"75'10","reported_seq":219,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-17T07:10:19.178844+0000","last_change":"2024-09-17T07:09:24.430545+0000","last_active":"2024-09-17T07:10:19.178844+0000","last_peered":"2024-09-17T07:10:19.178844+0000","last_clean":"2024-09-17T07:10:19.178844+0000","last_became_active":"2024-09-17T07:09:24.404495+0000","last_became_peered":"2024-09-17T07:09:24.404495+0000","last_unstale":"2024-09-17T07:10:19.178844+0000","last_undegraded":"2024-09-17T07:10:19.178844+0000","last_fullsized":"2024-09-17T07:10:19.178844+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-09-17T07:09:13.341701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:09:13.341701+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:31.527633+0000","scrub_duration":0.024611266999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"75'8","reported_seq":295,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.184072+0000","last_change":"2024-09-17T07:10:23.224635+0000","last_active":"2024-09-17T07:10:26.184072+0000","last_peered":"2024-09-17T07:10:26.184072+0000","last_clean":"2024-09-17T07:10:26.184072+0000","last_became_active":"2024-09-17T07:02:16.020560+0000","last_became_peered":"2024-09-17T07:02:16.020560+0000","last_unstale":"2024-09-17T07:10:26.184072+0000","last_undegraded":"2024-09-17T07:10:26.184072+0000","last_fullsized":"2024-09-17T07:10:26.184072+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:08:24.422313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:08:24.422313+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0027600989999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":181,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.156047+0000","last_change":"2024-09-17T07:08:34.803052+0000","last_active":"2024-09-17T07:10:26.156047+0000","last_peered":"2024-09-17T07:10:26.156047+0000","last_clean":"2024-09-17T07:10:26.156047+0000","last_became_active":"2024-09-17T07:02:18.021603+0000","last_became_peered":"2024-09-17T07:02:18.021603+0000","last_unstale":"2024-09-17T07:10:26.156047+0000","last_undegraded":"2024-09-17T07:10:26.156047+0000","last_fullsized":"2024-09-17T07:10:26.156047+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:34.802938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:08:34.802938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0022680489999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":28,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.183272+0000","last_change":"2024-09-17T07:09:24.405301+0000","last_active":"2024-09-17T07:10:26.183272+0000","last_peered":"2024-09-17T07:10:26.183272+0000","last_clean":"2024-09-17T07:10:26.183272+0000","last_became_active":"2024-09-17T07:09:24.403360+0000","last_became_peered":"2024-09-17T07:09:24.403360+0000","last_unstale":"2024-09-17T07:10:26.183272+0000","last_undegraded":"2024-09-17T07:10:26.183272+0000","last_fullsized":"2024-09-17T07:10:26.183272+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:17.704687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:08:17.704687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"75'11","reported_seq":247,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.184166+0000","last_change":"2024-09-17T07:09:18.263884+0000","last_active":"2024-09-17T07:10:26.184166+0000","last_peered":"2024-09-17T07:10:26.184166+0000","last_clean":"2024-09-17T07:10:26.184166+0000","last_became_active":"2024-09-17T07:03:48.760390+0000","last_became_peered":"2024-09-17T07:03:48.760390+0000","last_unstale":"2024-09-17T07:10:26.184166+0000","last_undegraded":"2024-09-17T07:10:26.184166+0000","last_fullsized":"2024-09-17T07:10:26.184166+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-09-17T07:09:18.263404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:09:18.263404+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:39.849976+0000","scrub_duration":0.0065910480000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":181,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-17T07:10:23.223931+0000","last_change":"2024-09-17T07:09:10.293301+0000","last_active":"2024-09-17T07:10:23.223931+0000","last_peered":"2024-09-17T07:10:23.223931+0000","last_clean":"2024-09-17T07:10:23.223931+0000","last_became_active":"2024-09-17T07:02:16.019701+0000","last_became_peered":"2024-09-17T07:02:16.019701+0000","last_unstale":"2024-09-17T07:10:23.223931+0000","last_undegraded":"2024-09-17T07:10:23.223931+0000","last_fullsized":"2024-09-17T07:10:23.223931+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:10.292615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:09:10.292615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:40.263740+0000","scrub_duration":0.010112249,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":172,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.150750+0000","last_change":"2024-09-17T07:07:30.779462+0000","last_active":"2024-09-17T07:10:26.150750+0000","last_peered":"2024-09-17T07:10:26.150750+0000","last_clean":"2024-09-17T07:10:26.150750+0000","last_became_active":"2024-09-17T07:02:18.363752+0000","last_became_peered":"2024-09-17T07:02:18.363752+0000","last_unstale":"2024-09-17T07:10:26.150750+0000","last_undegraded":"2024-09-17T07:10:26.150750+0000","last_fullsized":"2024-09-17T07:10:26.150750+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:30.779382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:07:30.779382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0011291059999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":218,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.183682+0000","last_change":"2024-09-17T07:09:22.215844+0000","last_active":"2024-09-17T07:10:26.183682+0000","last_peered":"2024-09-17T07:10:26.183682+0000","last_clean":"2024-09-17T07:10:26.183682+0000","last_became_active":"2024-09-17T07:03:48.759384+0000","last_became_peered":"2024-09-17T07:03:48.759384+0000","last_unstale":"2024-09-17T07:10:26.183682+0000","last_undegraded":"2024-09-17T07:10:26.183682+0000","last_fullsized":"2024-09-17T07:10:26.183682+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:22.215729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:09:22.215729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:37.706145+0000","scrub_duration":0.0051243160000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"75'15","reported_seq":222,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.156381+0000","last_change":"2024-09-17T07:09:24.427421+0000","last_active":"2024-09-17T07:10:26.156381+0000","last_peered":"2024-09-17T07:10:26.156381+0000","last_clean":"2024-09-17T07:10:26.156381+0000","last_became_active":"2024-09-17T07:09:24.403030+0000","last_became_peered":"2024-09-17T07:09:24.403030+0000","last_unstale":"2024-09-17T07:10:26.156381+0000","last_undegraded":"2024-09-17T07:10:26.156381+0000","last_fullsized":"2024-09-17T07:10:26.156381+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-09-17T07:09:15.110940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:09:15.110940+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.017595989999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":183,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.156304+0000","last_change":"2024-09-17T07:08:25.794548+0000","last_active":"2024-09-17T07:10:26.156304+0000","last_peered":"2024-09-17T07:10:26.156304+0000","last_clean":"2024-09-17T07:10:26.156304+0000","last_became_active":"2024-09-17T07:02:16.015903+0000","last_became_peered":"2024-09-17T07:02:16.015903+0000","last_unstale":"2024-09-17T07:10:26.156304+0000","last_undegraded":"2024-09-17T07:10:26.156304+0000","last_fullsized":"2024-09-17T07:10:26.156304+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:25.794418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:25.794418+0000","last_clean_scrub_stamp":"2024-09-17T07:08:25.794418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002916175,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":29,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.183947+0000","last_change":"2024-09-17T07:10:17.174897+0000","last_active":"2024-09-17T07:10:26.183947+0000","last_peered":"2024-09-17T07:10:26.183947+0000","last_clean":"2024-09-17T07:10:26.183947+0000","last_became_active":"2024-09-17T07:09:24.402411+0000","last_became_peered":"2024-09-17T07:09:24.402411+0000","last_unstale":"2024-09-17T07:10:26.183947+0000","last_undegraded":"2024-09-17T07:10:26.183947+0000","last_fullsized":"2024-09-17T07:10:26.183947+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:14.764781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:08:14.764781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":30,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.156983+0000","last_change":"2024-09-17T07:10:17.173461+0000","last_active":"2024-09-17T07:10:26.156983+0000","last_peered":"2024-09-17T07:10:26.156983+0000","last_clean":"2024-09-17T07:10:26.156983+0000","last_became_active":"2024-09-17T07:09:24.404795+0000","last_became_peered":"2024-09-17T07:09:24.404795+0000","last_unstale":"2024-09-17T07:10:26.156983+0000","last_undegraded":"2024-09-17T07:10:26.156983+0000","last_fullsized":"2024-09-17T07:10:26.156983+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:07:29.761001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:07:29.761001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"86'17","reported_seq":93,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.156386+0000","last_change":"2024-09-17T07:09:24.416689+0000","last_active":"2024-09-17T07:10:26.156386+0000","last_peered":"2024-09-17T07:10:26.156386+0000","last_clean":"2024-09-17T07:10:26.156386+0000","last_became_active":"2024-09-17T07:09:24.394476+0000","last_became_peered":"2024-09-17T07:09:24.394476+0000","last_unstale":"2024-09-17T07:10:26.156386+0000","last_undegraded":"2024-09-17T07:10:26.156386+0000","last_fullsized":"2024-09-17T07:10:26.156386+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'17","last_scrub_stamp":"2024-09-17T07:09:12.729002+0000","last_deep_scrub":"86'17","last_deep_scrub_stamp":"2024-09-17T07:03:18.241067+0000","last_clean_scrub_stamp":"2024-09-17T07:09:12.729002+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":193,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-17T07:10:26.150238+0000","last_change":"2024-09-17T07:09:24.393651+0000","last_active":"2024-09-17T07:10:26.150238+0000","last_peered":"2024-09-17T07:10:26.150238+0000","last_clean":"2024-09-17T07:10:26.150238+0000","last_became_active":"2024-09-17T07:09:24.393329+0000","last_became_peered":"2024-09-17T07:09:24.393329+0000","last_unstale":"2024-09-17T07:10:26.150238+0000","last_undegraded":"2024-09-17T07:10:26.150238+0000","last_fullsized":"2024-09-17T07:10:26.150238+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:14.706740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:08:14.706740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00164321,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":5801902104,"num_objects":88531,"num_object_clones":0,"num_object_copies":265593,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4809,"num_objects_unfound":0,"num_objects_dirty":88531,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":172473,"num_write_kb":8352065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35338,"num_bytes_recovered":2315911168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17445957632,"data_stored":17445945416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76126,"ondisk_log_size":76126,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69,"ondisk_log_size":69,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":692,"num_read_kb":449,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":2008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":13004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":427,"ondisk_log_size":427,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":489,"num_read_kb":484,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1386,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":12,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":420,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":4578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037865,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2107084,"kb_used_data":1783100,"kb_used_omap":1,"kb_used_meta":323966,"kb_avail":91638068,"statfs":{"total":95995035648,"available":93837381632,"internally_reserved":0,"allocated":1825894400,"data_stored":1825483909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":331741643},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:10:17 2024","interfaces":[{"interface":"back","average":{"1min":2.9079999999999999,"5min":2.9980000000000002,"15min":1.347},"min":{"1min":0.224,"5min":0.192,"15min":0.192},"max":{"1min":41.783999999999999,"5min":52.478999999999999,"15min":52.478999999999999},"last":0.219},{"interface":"front","average":{"1min":3.4129999999999998,"5min":3.0710000000000002,"15min":1.3819999999999999},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":42.149999999999999,"5min":52.871000000000002,"15min":52.871000000000002},"last":0.52700000000000002}]},{"osd":1,"last update":"Tue Sep 17 07:10:17 2024","interfaces":[{"interface":"back","average":{"1min":2.9180000000000001,"5min":2.1779999999999999,"15min":1.071},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":39.018000000000001,"5min":73.549000000000007,"15min":73.549000000000007},"last":1.0940000000000001},{"interface":"front","average":{"1min":3.46,"5min":2.3039999999999998,"15min":1.1259999999999999},"min":{"1min":0.25600000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":39.045999999999999,"5min":73.116,"15min":73.116},"last":1.496}]},{"osd":2,"last update":"Tue Sep 17 07:10:17 2024","interfaces":[{"interface":"back","average":{"1min":1.141,"5min":1.6910000000000001,"15min":0.88800000000000001},"min":{"1min":0.29299999999999998,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":4.3070000000000004,"5min":52.055999999999997,"15min":52.055999999999997},"last":2.2549999999999999},{"interface":"front","average":{"1min":0.90900000000000003,"5min":1.6899999999999999,"15min":0.92000000000000004},"min":{"1min":0.28499999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":2.3620000000000001,"5min":52.064999999999998,"15min":52.064999999999998},"last":0.39200000000000002}]},{"osd":3,"last update":"Tue Sep 17 07:10:17 2024","interfaces":[{"interface":"back","average":{"1min":0.97699999999999998,"5min":2.1949999999999998,"15min":1.0960000000000001},"min":{"1min":0.29499999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":4.0289999999999999,"5min":51.689,"15min":51.689},"last":2.302},{"interface":"front","average":{"1min":1.389,"5min":2.2210000000000001,"15min":1.2050000000000001},"min":{"1min":0.34200000000000003,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":3.633,"5min":52.145000000000003,"15min":52.145000000000003},"last":0.29599999999999999}]},{"osd":4,"last update":"Tue Sep 17 07:10:17 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.57399999999999995,"15min":0.48899999999999999},"min":{"1min":0.245,"5min":0.245,"15min":0.22800000000000001},"max":{"1min":3.5310000000000001,"5min":3.5310000000000001,"15min":3.5310000000000001},"last":0.497},{"interface":"front","average":{"1min":0.871,"5min":0.60499999999999998,"15min":0.48699999999999999},"min":{"1min":0.26700000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":5.8380000000000001,"5min":5.8380000000000001,"15min":5.8380000000000001},"last":2.2200000000000002}]},{"osd":5,"last update":"Tue Sep 17 07:10:17 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.59499999999999997,"15min":0.48599999999999999},"min":{"1min":0.29299999999999998,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":3.4649999999999999,"5min":3.4649999999999999,"15min":3.4649999999999999},"last":0.253},{"interface":"front","average":{"1min":0.94799999999999995,"5min":0.65800000000000003,"15min":0.52900000000000003},"min":{"1min":0.314,"5min":0.29999999999999999,"15min":0.28000000000000003},"max":{"1min":3.8639999999999999,"5min":3.8639999999999999,"15min":3.8639999999999999},"last":1.5309999999999999}]},{"osd":6,"last update":"Tue Sep 17 07:10:17 2024","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.69299999999999995,"15min":0.56299999999999994},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.26100000000000001},"max":{"1min":3.7690000000000001,"5min":3.7690000000000001,"15min":3.7690000000000001},"last":3.3900000000000001},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.67500000000000004,"15min":0.54800000000000004},"min":{"1min":0.318,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":2.8100000000000001,"5min":3.2669999999999999,"15min":3.2669999999999999},"last":2.2799999999999998}]}]},{"osd":6,"up_from":50,"seq":214748364908,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2070936,"kb_used_data":1778956,"kb_used_omap":1,"kb_used_meta":291966,"kb_avail":91674216,"statfs":{"total":95995035648,"available":93874397184,"internally_reserved":0,"allocated":1821650944,"data_stored":1821224570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":298973639},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:09:56 2024","interfaces":[{"interface":"back","average":{"1min":3.1659999999999999,"5min":3.5059999999999998,"15min":1.506},"min":{"1min":0.45200000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":24.106999999999999,"5min":75.323999999999998,"15min":75.323999999999998},"last":0.21199999999999999},{"interface":"front","average":{"1min":2.488,"5min":3.3170000000000002,"15min":1.4370000000000001},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":26.193000000000001,"5min":75.168999999999997,"15min":75.168999999999997},"last":0.26900000000000002}]},{"osd":1,"last update":"Tue Sep 17 07:09:56 2024","interfaces":[{"interface":"back","average":{"1min":1.4119999999999999,"5min":0.89500000000000002,"15min":0.624},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":4.2409999999999997,"5min":4.4790000000000001,"15min":4.4790000000000001},"last":1.478},{"interface":"front","average":{"1min":1.4530000000000001,"5min":1.004,"15min":0.68600000000000005},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":5.4930000000000003,"5min":5.9530000000000003,"15min":5.9530000000000003},"last":2.8319999999999999}]},{"osd":2,"last update":"Tue Sep 17 07:09:56 2024","interfaces":[{"interface":"back","average":{"1min":1.208,"5min":1.214,"15min":0.748},"min":{"1min":0.28799999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":5.1159999999999997,"5min":33.079000000000001,"15min":33.079000000000001},"last":0.23999999999999999},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.1599999999999999,"15min":0.73399999999999999},"min":{"1min":0.33900000000000002,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":3.2120000000000002,"5min":33.100000000000001,"15min":33.100000000000001},"last":0.30599999999999999}]},{"osd":3,"last update":"Tue Sep 17 07:09:56 2024","interfaces":[{"interface":"back","average":{"1min":1.347,"5min":2.1200000000000001,"15min":1.0369999999999999},"min":{"1min":0.245,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":3.9409999999999998,"5min":52.944000000000003,"15min":52.944000000000003},"last":0.40200000000000002},{"interface":"front","average":{"1min":1.488,"5min":2.1379999999999999,"15min":1.0760000000000001},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":5.3200000000000003,"5min":52.747,"15min":52.747},"last":1.3220000000000001}]},{"osd":4,"last update":"Tue Sep 17 07:09:56 2024","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":0.60599999999999998,"15min":0.48299999999999998},"min":{"1min":0.23100000000000001,"5min":0.224,"15min":0.224},"max":{"1min":3.1739999999999999,"5min":3.1739999999999999,"15min":3.1739999999999999},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.92900000000000005,"5min":0.63800000000000001,"15min":0.48999999999999999},"min":{"1min":0.27600000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":4.0030000000000001,"5min":4.0030000000000001,"15min":4.0030000000000001},"last":0.372}]},{"osd":5,"last update":"Tue Sep 17 07:09:56 2024","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.60999999999999999,"15min":0.49099999999999999},"min":{"1min":0.26700000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":3.149,"5min":4.9329999999999998,"15min":4.9329999999999998},"last":0.438},{"interface":"front","average":{"1min":0.995,"5min":0.70099999999999996,"15min":0.52200000000000002},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":4.024,"5min":4.024,"15min":4.024},"last":0.33600000000000002}]},{"osd":7,"last update":"Tue Sep 17 07:10:18 2024","interfaces":[{"interface":"back","average":{"1min":1.28,"5min":0.78500000000000003,"15min":0.58999999999999997},"min":{"1min":0.31900000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":5.7110000000000003,"5min":5.7110000000000003,"15min":5.7110000000000003},"last":0.66700000000000004},{"interface":"front","average":{"1min":1.0349999999999999,"5min":0.75800000000000001,"15min":0.60199999999999998},"min":{"1min":0.371,"5min":0.30099999999999999,"15min":0.29099999999999998},"max":{"1min":5.6760000000000002,"5min":5.6760000000000002,"15min":5.6760000000000002},"last":0.63100000000000001}]}]},{"osd":5,"up_from":42,"seq":180388626545,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3472444,"kb_used_data":3174516,"kb_used_omap":1,"kb_used_meta":297726,"kb_avail":90272708,"statfs":{"total":95995035648,"available":92439252992,"internally_reserved":0,"allocated":3250704384,"data_stored":3250301781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":304871878},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:09:36 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":2.7970000000000002,"15min":1.272},"min":{"1min":0.251,"5min":0.221,"15min":0.221},"max":{"1min":4.577,"5min":49.265000000000001,"15min":49.265000000000001},"last":0.27400000000000002},{"interface":"front","average":{"1min":1.4399999999999999,"5min":2.9689999999999999,"15min":1.3220000000000001},"min":{"1min":0.39500000000000002,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":4.4260000000000002,"5min":47.350000000000001,"15min":47.350000000000001},"last":0.373}]},{"osd":1,"last update":"Tue Sep 17 07:09:36 2024","interfaces":[{"interface":"back","average":{"1min":1.1879999999999999,"5min":0.95699999999999996,"15min":0.67500000000000004},"min":{"1min":0.23799999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":3.5600000000000001,"5min":20.329000000000001,"15min":20.329000000000001},"last":0.30099999999999999},{"interface":"front","average":{"1min":1.089,"5min":1.1279999999999999,"15min":1.0720000000000001},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":4.077,"5min":20.408999999999999,"15min":20.408999999999999},"last":0.314}]},{"osd":2,"last update":"Tue Sep 17 07:09:36 2024","interfaces":[{"interface":"back","average":{"1min":1.8240000000000001,"5min":1.4079999999999999,"15min":0.83599999999999997},"min":{"1min":0.36499999999999999,"5min":0.252,"15min":0.252},"max":{"1min":6.242,"5min":20.486000000000001,"15min":20.486000000000001},"last":0.40999999999999998},{"interface":"front","average":{"1min":1.115,"5min":1.2629999999999999,"15min":0.75800000000000001},"min":{"1min":0.28899999999999998,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":3.2599999999999998,"5min":23.925999999999998,"15min":23.925999999999998},"last":0.35099999999999998}]},{"osd":3,"last update":"Tue Sep 17 07:09:36 2024","interfaces":[{"interface":"back","average":{"1min":1.5209999999999999,"5min":2.3279999999999998,"15min":1.135},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":6.5439999999999996,"5min":68.978999999999999,"15min":68.978999999999999},"last":0.44500000000000001},{"interface":"front","average":{"1min":1.3280000000000001,"5min":2.2130000000000001,"15min":1.0960000000000001},"min":{"1min":0.26800000000000002,"5min":0.253,"15min":0.253},"max":{"1min":4.5490000000000004,"5min":69.084000000000003,"15min":69.084000000000003},"last":0.42899999999999999}]},{"osd":4,"last update":"Tue Sep 17 07:09:36 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.53800000000000003,"15min":0.443},"min":{"1min":0.26800000000000002,"5min":0.22,"15min":0.22},"max":{"1min":4.0300000000000002,"5min":4.0300000000000002,"15min":4.0300000000000002},"last":142.327},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.55200000000000005,"15min":0.47099999999999997},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":2.6469999999999998,"5min":2.6469999999999998,"15min":2.6469999999999998},"last":142.13200000000001}]},{"osd":6,"last update":"Tue Sep 17 07:09:50 2024","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.59699999999999998,"15min":0.48299999999999998},"min":{"1min":0.26100000000000001,"5min":0.23499999999999999,"15min":0.23200000000000001},"max":{"1min":3.1349999999999998,"5min":3.1349999999999998,"15min":3.1349999999999998},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.94599999999999995,"5min":0.67900000000000005,"15min":0.52100000000000002},"min":{"1min":0.314,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":3.4780000000000002,"5min":3.4780000000000002,"15min":3.4780000000000002},"last":0.35899999999999999}]},{"osd":7,"last update":"Tue Sep 17 07:10:12 2024","interfaces":[{"interface":"back","average":{"1min":1.331,"5min":0.79900000000000004,"15min":0.59199999999999997},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29099999999999998},"max":{"1min":5.1239999999999997,"5min":5.1239999999999997,"15min":5.1239999999999997},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.67000000000000004,"15min":0.53700000000000003},"min":{"1min":0.29899999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":4.3010000000000002,"5min":4.3010000000000002,"15min":4.3010000000000002},"last":0.45500000000000002}]}]},{"osd":4,"up_from":33,"seq":141733920887,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2448452,"kb_used_data":2214708,"kb_used_omap":1,"kb_used_meta":233726,"kb_avail":91296700,"statfs":{"total":95995035648,"available":93487820800,"internally_reserved":0,"allocated":2267860992,"data_stored":2267457465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":239335878},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:09:58 2024","interfaces":[{"interface":"back","average":{"1min":1.1140000000000001,"5min":4.1360000000000001,"15min":1.7629999999999999},"min":{"1min":0.29299999999999998,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":4.6829999999999998,"5min":74.551000000000002,"15min":74.551000000000002},"last":0.222},{"interface":"front","average":{"1min":1.1850000000000001,"5min":4.4660000000000002,"15min":1.8879999999999999},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":2.6440000000000001,"5min":74.778999999999996,"15min":74.778999999999996},"last":1.27}]},{"osd":1,"last update":"Tue Sep 17 07:09:58 2024","interfaces":[{"interface":"back","average":{"1min":1.696,"5min":1.764,"15min":0.96899999999999997},"min":{"1min":0.23300000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4.8319999999999999,"5min":54.914999999999999,"15min":54.914999999999999},"last":0.89700000000000002},{"interface":"front","average":{"1min":0.78700000000000003,"5min":1.5720000000000001,"15min":0.91200000000000003},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.22800000000000001},"max":{"1min":3.6499999999999999,"5min":53.725000000000001,"15min":53.725000000000001},"last":0.30399999999999999}]},{"osd":2,"last update":"Tue Sep 17 07:09:58 2024","interfaces":[{"interface":"back","average":{"1min":1.5109999999999999,"5min":1.5,"15min":0.88300000000000001},"min":{"1min":0.28499999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":4.7839999999999998,"5min":49.651000000000003,"15min":49.651000000000003},"last":0.69599999999999995},{"interface":"front","average":{"1min":1.0509999999999999,"5min":1.4830000000000001,"15min":0.88100000000000001},"min":{"1min":0.33400000000000002,"5min":0.247,"15min":0.247},"max":{"1min":2.7450000000000001,"5min":48.813000000000002,"15min":48.813000000000002},"last":1.196}]},{"osd":3,"last update":"Tue Sep 17 07:09:58 2024","interfaces":[{"interface":"back","average":{"1min":1.1679999999999999,"5min":1.508,"15min":0.90700000000000003},"min":{"1min":0.30199999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":3.7690000000000001,"5min":39.042999999999999,"15min":39.042999999999999},"last":1.228},{"interface":"front","average":{"1min":0.96399999999999997,"5min":1.554,"15min":0.90400000000000003},"min":{"1min":0.38500000000000001,"5min":0.253,"15min":0.253},"max":{"1min":3.1659999999999999,"5min":38.920000000000002,"15min":38.920000000000002},"last":0.87}]},{"osd":5,"last update":"Tue Sep 17 07:09:28 2024","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.59399999999999997,"15min":0.45700000000000002},"min":{"1min":0.27100000000000002,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":3.4390000000000001,"5min":3.4390000000000001,"15min":3.4390000000000001},"last":1.5029999999999999},{"interface":"front","average":{"1min":0.93999999999999995,"5min":0.61599999999999999,"15min":0.48299999999999998},"min":{"1min":0.27500000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":3.0630000000000002,"5min":3.0630000000000002,"15min":3.0630000000000002},"last":1.915}]},{"osd":6,"last update":"Tue Sep 17 07:09:51 2024","interfaces":[{"interface":"back","average":{"1min":1.056,"5min":1.1679999999999999,"15min":0.65800000000000003},"min":{"1min":0.31,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":3.714,"5min":47.923999999999999,"15min":47.923999999999999},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.78000000000000003,"5min":1.117,"15min":0.65100000000000002},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.26300000000000001},"max":{"1min":4.8440000000000003,"5min":47.600999999999999,"15min":47.600999999999999},"last":0.72099999999999997}]},{"osd":7,"last update":"Tue Sep 17 07:10:17 2024","interfaces":[{"interface":"back","average":{"1min":1.351,"5min":1.2609999999999999,"15min":0.752},"min":{"1min":0.29299999999999998,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":3.6400000000000001,"5min":48.719000000000001,"15min":48.719000000000001},"last":1.2490000000000001},{"interface":"front","average":{"1min":0.86199999999999999,"5min":1.214,"15min":0.72899999999999998},"min":{"1min":0.312,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":3.3639999999999999,"5min":50.527000000000001,"15min":50.527000000000001},"last":0.41499999999999998}]}]},{"osd":3,"up_from":0,"seq":115964117113,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":18,"seq":77309411455,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3285960,"kb_used_data":2821688,"kb_used_omap":1,"kb_used_meta":464254,"kb_avail":90459192,"statfs":{"total":95995035648,"available":92630212608,"internally_reserved":0,"allocated":2889408512,"data_stored":2889001757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":475396550},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:09:31 2024","interfaces":[{"interface":"back","average":{"1min":1.569,"5min":1.25,"15min":0.64600000000000002},"min":{"1min":0.193,"5min":0.193,"15min":0.17599999999999999},"max":{"1min":5.0259999999999998,"5min":7.2709999999999999,"15min":7.2709999999999999},"last":0.20699999999999999},{"interface":"front","average":{"1min":2.137,"5min":1.768,"15min":0.82499999999999996},"min":{"1min":0.219,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":6.8689999999999998,"5min":8.6869999999999994,"15min":8.6869999999999994},"last":0.221}]},{"osd":1,"last update":"Tue Sep 17 07:09:31 2024","interfaces":[{"interface":"back","average":{"1min":1.7709999999999999,"5min":1.272,"15min":0.65800000000000003},"min":{"1min":0.36499999999999999,"5min":0.184,"15min":0.184},"max":{"1min":6.3499999999999996,"5min":6.3499999999999996,"15min":6.3499999999999996},"last":37.774000000000001},{"interface":"front","average":{"1min":1.768,"5min":1.175,"15min":0.64200000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":5.2759999999999998,"5min":6.9829999999999997,"15min":6.9829999999999997},"last":36.808}]},{"osd":3,"last update":"Tue Sep 17 07:09:49 2024","interfaces":[{"interface":"back","average":{"1min":2.5059999999999998,"5min":1.448,"15min":0.78000000000000003},"min":{"1min":0.29599999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":6.4809999999999999,"5min":6.4809999999999999,"15min":6.4809999999999999},"last":23.43},{"interface":"front","average":{"1min":1.8939999999999999,"5min":1.7130000000000001,"15min":0.877},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":4.9029999999999996,"5min":6.0679999999999996,"15min":6.0679999999999996},"last":23.408000000000001}]},{"osd":4,"last update":"Tue Sep 17 07:10:10 2024","interfaces":[{"interface":"back","average":{"1min":1.5629999999999999,"5min":1.0740000000000001,"15min":0.75800000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":5.944,"5min":5.944,"15min":5.944},"last":0.309},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.1040000000000001,"15min":0.76200000000000001},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":3.6459999999999999,"5min":6.4539999999999997,"15min":6.4539999999999997},"last":0.33100000000000002}]},{"osd":5,"last update":"Tue Sep 17 07:09:31 2024","interfaces":[{"interface":"back","average":{"1min":1.794,"5min":1.268,"15min":0.81599999999999995},"min":{"1min":0.38700000000000001,"5min":0.249,"15min":0.249},"max":{"1min":4.9619999999999997,"5min":5.383,"15min":5.383},"last":0.29399999999999998},{"interface":"front","average":{"1min":1.6950000000000001,"5min":1.0980000000000001,"15min":0.753},"min":{"1min":0.32300000000000001,"5min":0.312,"15min":0.312},"max":{"1min":7.1790000000000003,"5min":7.1790000000000003,"15min":7.1790000000000003},"last":0.38800000000000001}]},{"osd":6,"last update":"Tue Sep 17 07:09:49 2024","interfaces":[{"interface":"back","average":{"1min":7.3819999999999997,"5min":2.29,"15min":1.1659999999999999},"min":{"1min":0.29999999999999999,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":101.587,"5min":101.587,"15min":101.587},"last":0.37},{"interface":"front","average":{"1min":6.9850000000000003,"5min":2.3439999999999999,"15min":1.1970000000000001},"min":{"1min":0.30599999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":100.696,"5min":100.696,"15min":100.696},"last":0.317}]},{"osd":7,"last update":"Tue Sep 17 07:10:23 2024","interfaces":[{"interface":"back","average":{"1min":1.248,"5min":1.115,"15min":0.77400000000000002},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":3.375,"5min":6.9450000000000003,"15min":6.9450000000000003},"last":0.34799999999999998},{"interface":"front","average":{"1min":1.2290000000000001,"5min":1.1990000000000001,"15min":0.91500000000000004},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":3.6240000000000001,"5min":9.2919999999999998,"15min":9.2919999999999998},"last":0.39600000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574980,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2755104,"kb_used_data":2481112,"kb_used_omap":1,"kb_used_meta":273918,"kb_avail":90990048,"statfs":{"total":95995035648,"available":93173809152,"internally_reserved":0,"allocated":2540658688,"data_stored":2540220423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":280492486},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:09:54 2024","interfaces":[{"interface":"back","average":{"1min":2.7360000000000002,"5min":1.76,"15min":0.85599999999999998},"min":{"1min":0.20200000000000001,"5min":0.19400000000000001,"15min":0.155},"max":{"1min":11.656000000000001,"5min":13.837,"15min":13.837},"last":0.36199999999999999},{"interface":"front","average":{"1min":1.518,"5min":1.581,"15min":0.80000000000000004},"min":{"1min":0.29199999999999998,"5min":0.186,"15min":0.17000000000000001},"max":{"1min":5.6369999999999996,"5min":10.342000000000001,"15min":10.342000000000001},"last":0.57299999999999995}]},{"osd":2,"last update":"Tue Sep 17 07:09:26 2024","interfaces":[{"interface":"back","average":{"1min":1.9259999999999999,"5min":1.5840000000000001,"15min":0.78900000000000003},"min":{"1min":0.20499999999999999,"5min":0.192,"15min":0.192},"max":{"1min":6.2130000000000001,"5min":11.055999999999999,"15min":11.055999999999999},"last":0.41999999999999998},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.514,"15min":0.77500000000000002},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":5.242,"5min":9.3160000000000007,"15min":9.3160000000000007},"last":0.46100000000000002}]},{"osd":3,"last update":"Tue Sep 17 07:09:54 2024","interfaces":[{"interface":"back","average":{"1min":1.1879999999999999,"5min":1.4119999999999999,"15min":0.79300000000000004},"min":{"1min":0.33300000000000002,"5min":0.219,"15min":0.219},"max":{"1min":3.4159999999999999,"5min":7.766,"15min":7.766},"last":0.55200000000000005},{"interface":"front","average":{"1min":2.4220000000000002,"5min":1.8460000000000001,"15min":0.91900000000000004},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":8.6400000000000006,"5min":19.602,"15min":19.602},"last":0.44500000000000001}]},{"osd":4,"last update":"Tue Sep 17 07:10:10 2024","interfaces":[{"interface":"back","average":{"1min":1.589,"5min":1.2330000000000001,"15min":0.83099999999999996},"min":{"1min":0.34100000000000003,"5min":0.25,"15min":0.25},"max":{"1min":3.5750000000000002,"5min":5.6970000000000001,"15min":7.8840000000000003},"last":0.51700000000000002},{"interface":"front","average":{"1min":1.962,"5min":1.7350000000000001,"15min":0.96499999999999997},"min":{"1min":0.40799999999999997,"5min":0.307,"15min":0.307},"max":{"1min":6.6989999999999998,"5min":27.097999999999999,"15min":27.097999999999999},"last":0.505}]},{"osd":5,"last update":"Tue Sep 17 07:09:26 2024","interfaces":[{"interface":"back","average":{"1min":1.246,"5min":1.177,"15min":0.78900000000000003},"min":{"1min":0.33400000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":5.3470000000000004,"5min":5.617,"15min":6.0060000000000002},"last":16.84},{"interface":"front","average":{"1min":2.141,"5min":1.4179999999999999,"15min":0.84599999999999997},"min":{"1min":0.376,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":8.1920000000000002,"5min":8.8889999999999993,"15min":8.8889999999999993},"last":16.949999999999999}]},{"osd":6,"last update":"Tue Sep 17 07:09:54 2024","interfaces":[{"interface":"back","average":{"1min":2.1629999999999998,"5min":1.5940000000000001,"15min":0.89700000000000002},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":7.6390000000000002,"5min":8.843,"15min":8.843},"last":0.53200000000000003},{"interface":"front","average":{"1min":1.5049999999999999,"5min":1.306,"15min":0.83099999999999996},"min":{"1min":0.35099999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":3.6019999999999999,"5min":7.1130000000000004,"15min":7.1130000000000004},"last":0.58999999999999997}]},{"osd":7,"last update":"Tue Sep 17 07:10:13 2024","interfaces":[{"interface":"back","average":{"1min":1.863,"5min":1.6399999999999999,"15min":0.93200000000000005},"min":{"1min":0.377,"5min":0.377,"15min":0.37},"max":{"1min":6.6360000000000001,"5min":8.9309999999999992,"15min":8.9309999999999992},"last":0.56299999999999994},{"interface":"front","average":{"1min":2.1989999999999998,"5min":1.484,"15min":0.88800000000000001},"min":{"1min":0.39500000000000002,"5min":0.33900000000000002,"15min":0.32900000000000001},"max":{"1min":6.2590000000000003,"5min":11.007999999999999,"15min":11.007999999999999},"last":0.54000000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738502,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2910420,"kb_used_data":2518348,"kb_used_omap":1,"kb_used_meta":392062,"kb_avail":90834732,"statfs":{"total":95995035648,"available":93014765568,"internally_reserved":0,"allocated":2578788352,"data_stored":2578399213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":401471942},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 17 07:10:01 2024","interfaces":[{"interface":"back","average":{"1min":1.502,"5min":1.431,"15min":0.70199999999999996},"min":{"1min":0.16900000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":6.6200000000000001,"5min":8.0259999999999998,"15min":8.0259999999999998},"last":0.23499999999999999},{"interface":"front","average":{"1min":1.7609999999999999,"5min":1.605,"15min":0.78000000000000003},"min":{"1min":0.24299999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":10.808,"5min":10.808,"15min":10.808},"last":0.70599999999999996}]},{"osd":2,"last update":"Tue Sep 17 07:09:24 2024","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":2.0459999999999998,"15min":0.93400000000000005},"min":{"1min":0.255,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":3.0939999999999999,"5min":18.643999999999998,"15min":18.643999999999998},"last":0.56999999999999995},{"interface":"front","average":{"1min":1.0009999999999999,"5min":1.8160000000000001,"15min":0.84799999999999998},"min":{"1min":0.27600000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":5.8440000000000003,"5min":17.972999999999999,"15min":17.972999999999999},"last":1.042}]},{"osd":3,"last update":"Tue Sep 17 07:10:01 2024","interfaces":[{"interface":"back","average":{"1min":1.3420000000000001,"5min":1.883,"15min":0.92900000000000005},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":5.2080000000000002,"5min":18.629000000000001,"15min":18.629000000000001},"last":1.0069999999999999},{"interface":"front","average":{"1min":1.4319999999999999,"5min":2.032,"15min":0.96799999999999997},"min":{"1min":0.25800000000000001,"5min":0.24299999999999999,"15min":0.23699999999999999},"max":{"1min":3.4809999999999999,"5min":22.358000000000001,"15min":22.358000000000001},"last":0.42899999999999999}]},{"osd":4,"last update":"Tue Sep 17 07:10:09 2024","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.5069999999999999,"15min":0.92900000000000005},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":3.617,"5min":18.568000000000001,"15min":18.568000000000001},"last":1.7050000000000001},{"interface":"front","average":{"1min":1.353,"5min":1.456,"15min":0.876},"min":{"1min":0.46000000000000002,"5min":0.255,"15min":0.255},"max":{"1min":3.6459999999999999,"5min":18.765000000000001,"15min":18.765000000000001},"last":1.6779999999999999}]},{"osd":5,"last update":"Tue Sep 17 07:09:36 2024","interfaces":[{"interface":"back","average":{"1min":1.1459999999999999,"5min":1.5600000000000001,"15min":0.88500000000000001},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":5.2229999999999999,"5min":18.614999999999998,"15min":18.614999999999998},"last":0.45900000000000002},{"interface":"front","average":{"1min":1.0640000000000001,"5min":1.5329999999999999,"15min":0.871},"min":{"1min":0.30199999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":2.923,"5min":18.791,"15min":18.791},"last":0.35399999999999998}]},{"osd":6,"last update":"Tue Sep 17 07:10:01 2024","interfaces":[{"interface":"back","average":{"1min":1.2070000000000001,"5min":1.5149999999999999,"15min":0.872},"min":{"1min":0.31900000000000001,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":3.1429999999999998,"5min":18.602,"15min":18.602},"last":1.7929999999999999},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.611,"15min":0.91200000000000003},"min":{"1min":0.38100000000000001,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":3.6539999999999999,"5min":18.692,"15min":18.692},"last":1.8779999999999999}]},{"osd":7,"last update":"Tue Sep 17 07:10:16 2024","interfaces":[{"interface":"back","average":{"1min":0.96399999999999997,"5min":1.911,"15min":1.393},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":2.9540000000000002,"5min":46.183,"15min":46.183},"last":1.847},{"interface":"front","average":{"1min":1.2390000000000001,"5min":1.893,"15min":1.038},"min":{"1min":0.48399999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":3.1400000000000001,"5min":45.914999999999999,"15min":45.914999999999999},"last":1.9850000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":2448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2581004288,"data_stored":2581004288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2520776704,"data_stored":2520776704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2886668288,"data_stored":2886664216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":357695488,"data_stored":357695488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2247426048,"data_stored":2247426048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3245281280,"data_stored":3245277208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1798897664,"data_stored":1798897664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1808207872,"data_stored":1808203800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T07:10:30.921 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd pool set unique_pool_1 pgp_num 16 2024-09-17T07:10:30.923 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:30.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:30.925 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:10:30.933 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-17T07:10:31.025 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:10:31.068 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:31.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:31.069 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:10:31.170 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:10:31.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:31.222 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:31.223 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:10:31.324 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:10:31.336 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:10:31.356 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:31.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:31.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:10:31.455 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:31 smithi100 ceph-mon[29092]: from='client.24904 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:10:31.455 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:31 smithi100 ceph-mon[29092]: osdmap e211: 8 total, 8 up, 7 in 2024-09-17T07:10:31.455 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:31 smithi100 ceph-mon[29092]: 6.c scrub starts 2024-09-17T07:10:31.456 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:31 smithi100 ceph-mon[29092]: 6.c scrub ok 2024-09-17T07:10:31.456 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:31 smithi100 ceph-mon[39462]: from='client.24904 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:10:31.456 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:31 smithi100 ceph-mon[39462]: osdmap e211: 8 total, 8 up, 7 in 2024-09-17T07:10:31.456 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:31 smithi100 ceph-mon[39462]: 6.c scrub starts 2024-09-17T07:10:31.456 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:31 smithi100 ceph-mon[39462]: 6.c scrub ok 2024-09-17T07:10:31.459 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:10:31.492 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:31.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:31.493 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:10:31.515 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:10:31.516 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:10:31.538 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:10:31.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:31 smithi135 ceph-mon[33238]: from='client.24904 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:10:31.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:31 smithi135 ceph-mon[33238]: osdmap e211: 8 total, 8 up, 7 in 2024-09-17T07:10:31.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:31 smithi135 ceph-mon[33238]: 6.c scrub starts 2024-09-17T07:10:31.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:31 smithi135 ceph-mon[33238]: 6.c scrub ok 2024-09-17T07:10:31.559 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1: {} 2024-09-17T07:10:31.559 INFO:teuthology.orchestra.run.smithi100.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:10:31.572 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:10:31.590 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:10:31.591 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:10:31.594 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:10:31.624 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:10:31.625 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:10:31.640 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:31.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:31.641 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:10:31.663 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:10:31.664 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:10:31.691 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:10:31.691 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:10:31.726 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:10:31.727 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:10:31.742 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:10:31.757 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:10:31.757 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:10:31.778 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:31.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:31.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:10:31.879 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:10:31.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:31.918 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:31.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:10:32.019 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:10:32.048 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:32.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:32.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:10:32.151 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:10:32.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:32.191 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:32.192 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:10:32.293 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:10:32.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:32.329 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:32.329 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:10:32.430 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:10:32.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:32.468 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:32.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:10:32.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:32 smithi135 ceph-mon[33238]: pgmap v368: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 5.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 452 op/s; 4467/264183 objects misplaced (1.691%); 27 MiB/s, 427 objects/s recovering 2024-09-17T07:10:32.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:32 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1727787127' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:10:32.569 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:10:32.610 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:32.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:32.611 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:10:32.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:32 smithi100 ceph-mon[29092]: pgmap v368: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 5.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 452 op/s; 4467/264183 objects misplaced (1.691%); 27 MiB/s, 427 objects/s recovering 2024-09-17T07:10:32.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:32 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1727787127' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:10:32.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:32 smithi100 ceph-mon[39462]: pgmap v368: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 5.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 452 op/s; 4467/264183 objects misplaced (1.691%); 27 MiB/s, 427 objects/s recovering 2024-09-17T07:10:32.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:32 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1727787127' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:10:32.712 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:10:32.745 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:32.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:32.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:10:32.847 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:10:32.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:32.886 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:32.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:10:32.987 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:10:33.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:33.031 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:33.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:10:33.132 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:10:33.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:33.174 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:33.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:10:33.275 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:10:33.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:33.346 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:33.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:10:33.447 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:10:33.483 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:33.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:33.484 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:10:33.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:33 smithi135 ceph-mon[33238]: 2.18 scrub starts 2024-09-17T07:10:33.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:33 smithi135 ceph-mon[33238]: 4.5 scrub starts 2024-09-17T07:10:33.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:33 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:10:33.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:33 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:10:33.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:33 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:10:33.584 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:10:33.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:33 smithi100 ceph-mon[29092]: 2.18 scrub starts 2024-09-17T07:10:33.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:33 smithi100 ceph-mon[29092]: 4.5 scrub starts 2024-09-17T07:10:33.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:33 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:10:33.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:33 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:10:33.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:33 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:10:33.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:33 smithi100 ceph-mon[39462]: 2.18 scrub starts 2024-09-17T07:10:33.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:33 smithi100 ceph-mon[39462]: 4.5 scrub starts 2024-09-17T07:10:33.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:33 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:10:33.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:33 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:10:33.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:33 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:10:33.630 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:33.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:33.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:10:33.731 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:10:33.773 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:33.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:33.774 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:10:33.875 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:10:33.919 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:33.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:33.920 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:10:34.021 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:10:34.039 INFO:teuthology.orchestra.run.smithi135.stdout:{ 2024-09-17T07:10:34.039 INFO:teuthology.orchestra.run.smithi135.stdout: "ops": [], 2024-09-17T07:10:34.039 INFO:teuthology.orchestra.run.smithi135.stdout: "complaint_time": 30, 2024-09-17T07:10:34.039 INFO:teuthology.orchestra.run.smithi135.stdout: "num_blocked_ops": 0 2024-09-17T07:10:34.040 INFO:teuthology.orchestra.run.smithi135.stdout:} 2024-09-17T07:10:34.058 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:34.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:34.059 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:10:34.160 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:10:34.198 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:34.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:34.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:10:34.300 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:10:34.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:34.336 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:34.336 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:10:34.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:34 smithi100 ceph-mon[29092]: pgmap v369: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 5.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 728 B/s rd, 0 B/s wr, 506 op/s; 4467/261654 objects misplaced (1.707%); 23 MiB/s, 365 objects/s recovering 2024-09-17T07:10:34.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:34 smithi100 ceph-mon[29092]: 2.10 scrub starts 2024-09-17T07:10:34.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:34 smithi100 ceph-mon[29092]: 5.13 scrub starts 2024-09-17T07:10:34.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:34 smithi100 ceph-mon[29092]: 6.3 deep-scrub starts 2024-09-17T07:10:34.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:34 smithi100 ceph-mon[29092]: 6.3 deep-scrub ok 2024-09-17T07:10:34.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:34 smithi100 ceph-mon[39462]: pgmap v369: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 5.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 728 B/s rd, 0 B/s wr, 506 op/s; 4467/261654 objects misplaced (1.707%); 23 MiB/s, 365 objects/s recovering 2024-09-17T07:10:34.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:34 smithi100 ceph-mon[39462]: 2.10 scrub starts 2024-09-17T07:10:34.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:34 smithi100 ceph-mon[39462]: 5.13 scrub starts 2024-09-17T07:10:34.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:34 smithi100 ceph-mon[39462]: 6.3 deep-scrub starts 2024-09-17T07:10:34.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:34 smithi100 ceph-mon[39462]: 6.3 deep-scrub ok 2024-09-17T07:10:34.437 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:10:34.475 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:34.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:34.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:10:34.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:34 smithi135 ceph-mon[33238]: pgmap v369: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 5.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 728 B/s rd, 0 B/s wr, 506 op/s; 4467/261654 objects misplaced (1.707%); 23 MiB/s, 365 objects/s recovering 2024-09-17T07:10:34.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:34 smithi135 ceph-mon[33238]: 2.10 scrub starts 2024-09-17T07:10:34.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:34 smithi135 ceph-mon[33238]: 5.13 scrub starts 2024-09-17T07:10:34.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:34 smithi135 ceph-mon[33238]: 6.3 deep-scrub starts 2024-09-17T07:10:34.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:34 smithi135 ceph-mon[33238]: 6.3 deep-scrub ok 2024-09-17T07:10:34.577 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:10:34.611 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:34.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:34.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:10:34.713 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:10:34.756 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-17T07:10:34.757 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:34.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:34.758 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:10:34.859 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:10:34.907 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:34.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:34.909 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:10:35.010 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:10:35.055 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:35.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:35.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:10:35.116 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:10:35.157 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:10:35.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:35.204 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:35.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:10:35.286 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:35 smithi100 ceph-mon[29092]: 6.13 scrub starts 2024-09-17T07:10:35.286 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:35 smithi100 ceph-mon[29092]: 6.13 scrub ok 2024-09-17T07:10:35.286 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:35 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:10:35.286 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:35 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:10:35.286 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:35 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:10:35.286 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:35 smithi100 ceph-mon[39462]: 6.13 scrub starts 2024-09-17T07:10:35.287 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:35 smithi100 ceph-mon[39462]: 6.13 scrub ok 2024-09-17T07:10:35.287 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:35 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:10:35.287 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:35 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:10:35.287 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:35 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:10:35.304 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:10:35.332 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:35.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:35.333 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:10:35.434 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:10:35.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:35.471 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:35.471 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:10:35.545 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:35 smithi135 ceph-mon[33238]: 6.13 scrub starts 2024-09-17T07:10:35.545 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:35 smithi135 ceph-mon[33238]: 6.13 scrub ok 2024-09-17T07:10:35.545 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:35 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:10:35.545 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:35 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:10:35.545 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:35 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:10:35.572 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:10:35.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:35.602 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:35.602 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:10:35.703 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:10:35.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:35.737 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:35.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:10:35.790 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:10:35.838 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:10:35.876 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:35.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:35.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:10:35.977 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:10:35.992 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:10:35 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:10:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:10:36.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:36.033 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:36.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:10:36.134 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:10:36.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:36.170 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:36.170 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:10:36.271 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:10:36.305 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:36.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:36.306 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:10:36.406 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:10:36.461 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:36.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:36.462 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:10:36.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:36 smithi135 ceph-mon[33238]: 4.8 scrub starts 2024-09-17T07:10:36.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:36 smithi135 ceph-mon[33238]: 4.8 scrub ok 2024-09-17T07:10:36.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:36 smithi135 ceph-mon[33238]: pgmap v370: 148 pgs: 1 active+remapped+backfilling, 4 active+clean+scrubbing, 143 active+clean; 5.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.16k op/s; 3566/244488 objects misplaced (1.459%); 24 MiB/s, 389 objects/s recovering 2024-09-17T07:10:36.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:36 smithi135 ceph-mon[33238]: 4.10 scrub starts 2024-09-17T07:10:36.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:36 smithi135 ceph-mon[33238]: 6.18 scrub starts 2024-09-17T07:10:36.563 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:10:36.592 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:36 smithi100 ceph-mon[29092]: 4.8 scrub starts 2024-09-17T07:10:36.592 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:36 smithi100 ceph-mon[29092]: 4.8 scrub ok 2024-09-17T07:10:36.592 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:36 smithi100 ceph-mon[29092]: pgmap v370: 148 pgs: 1 active+remapped+backfilling, 4 active+clean+scrubbing, 143 active+clean; 5.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.16k op/s; 3566/244488 objects misplaced (1.459%); 24 MiB/s, 389 objects/s recovering 2024-09-17T07:10:36.592 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:36 smithi100 ceph-mon[29092]: 4.10 scrub starts 2024-09-17T07:10:36.592 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:36 smithi100 ceph-mon[29092]: 6.18 scrub starts 2024-09-17T07:10:36.592 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:36 smithi100 ceph-mon[39462]: 4.8 scrub starts 2024-09-17T07:10:36.592 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:36 smithi100 ceph-mon[39462]: 4.8 scrub ok 2024-09-17T07:10:36.592 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:36 smithi100 ceph-mon[39462]: pgmap v370: 148 pgs: 1 active+remapped+backfilling, 4 active+clean+scrubbing, 143 active+clean; 5.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.16k op/s; 3566/244488 objects misplaced (1.459%); 24 MiB/s, 389 objects/s recovering 2024-09-17T07:10:36.592 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:36 smithi100 ceph-mon[39462]: 4.10 scrub starts 2024-09-17T07:10:36.593 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:36 smithi100 ceph-mon[39462]: 6.18 scrub starts 2024-09-17T07:10:36.604 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:36.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:36.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:10:36.706 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:10:36.737 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:36.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:36.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:10:36.839 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:10:36.879 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:36.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:36.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:10:36.981 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:10:37.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:37.016 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:37.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:10:37.047 INFO:teuthology.orchestra.run.smithi135.stdout:{ 2024-09-17T07:10:37.047 INFO:teuthology.orchestra.run.smithi135.stdout: "size": 20, 2024-09-17T07:10:37.047 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 600, 2024-09-17T07:10:37.047 INFO:teuthology.orchestra.run.smithi135.stdout: "ops": [ 2024-09-17T07:10:37.047 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.047 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:34820 7.0 7:0ea320a1:::benchmark_data_smithi100_78556_object34819:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-17T07:10:37.047 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:04:45.261816+0000", 2024-09-17T07:10:37.048 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 351.779382274, 2024-09-17T07:10:37.048 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.16082640100000001, 2024-09-17T07:10:37.048 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:10:37.048 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:10:37.048 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:10:37.048 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:10:37.048 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:10:37.048 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 34820 2024-09-17T07:10:37.048 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.048 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:10:37.048 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.048 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:10:37.048 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:45.261816+0000", 2024-09-17T07:10:37.049 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:10:37.049 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.049 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.049 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:10:37.049 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:45.261816+0000", 2024-09-17T07:10:37.049 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:10:37.049 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.049 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.049 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:10:37.049 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:45.261815+0000", 2024-09-17T07:10:37.049 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:10:37.049 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.049 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.050 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:10:37.050 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:45.261889+0000", 2024-09-17T07:10:37.050 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.3473999999999995e-05 2024-09-17T07:10:37.050 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.050 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.050 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:10:37.050 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:45.261900+0000", 2024-09-17T07:10:37.050 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.1623999999999999e-05 2024-09-17T07:10:37.050 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.050 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.050 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:10:37.050 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:45.261908+0000", 2024-09-17T07:10:37.050 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.7370000000000006e-06 2024-09-17T07:10:37.051 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.051 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.051 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:10:37.051 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:45.261946+0000", 2024-09-17T07:10:37.051 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.8408000000000002e-05 2024-09-17T07:10:37.051 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.051 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.051 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:10:37.051 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:45.262065+0000", 2024-09-17T07:10:37.051 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000118646 2024-09-17T07:10:37.051 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.051 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.051 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 1,3", 2024-09-17T07:10:37.051 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:45.262200+0000", 2024-09-17T07:10:37.052 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00013496100000000001 2024-09-17T07:10:37.052 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.052 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.052 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:10:37.052 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:45.263611+0000", 2024-09-17T07:10:37.052 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00141091 2024-09-17T07:10:37.052 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.052 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.052 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:10:37.052 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:45.264780+0000", 2024-09-17T07:10:37.052 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.001169607 2024-09-17T07:10:37.052 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.052 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.053 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:10:37.053 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:45.422549+0000", 2024-09-17T07:10:37.053 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.157769033 2024-09-17T07:10:37.053 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.053 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.053 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:10:37.053 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:45.422603+0000", 2024-09-17T07:10:37.053 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.3279000000000003e-05 2024-09-17T07:10:37.053 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.053 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.053 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:10:37.053 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:45.422643+0000", 2024-09-17T07:10:37.053 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.9949e-05 2024-09-17T07:10:37.054 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:10:37.054 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:10:37.054 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:10:37.054 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.054 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.054 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:35412 7.0 7:08452bb0:::benchmark_data_smithi100_78556_object35411:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-17T07:10:37.054 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:04:45.692219+0000", 2024-09-17T07:10:37.054 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 351.348979853, 2024-09-17T07:10:37.054 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.12724049600000001, 2024-09-17T07:10:37.054 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:10:37.054 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:10:37.054 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:10:37.054 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:10:37.055 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:10:37.055 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 35412 2024-09-17T07:10:37.055 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.055 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:10:37.055 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.055 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:10:37.055 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:45.692219+0000", 2024-09-17T07:10:37.055 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:10:37.055 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.055 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.055 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:10:37.055 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:45.692219+0000", 2024-09-17T07:10:37.055 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:10:37.056 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.056 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.056 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:10:37.056 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:45.692217+0000", 2024-09-17T07:10:37.056 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:10:37.056 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.056 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.056 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:10:37.056 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:45.692263+0000", 2024-09-17T07:10:37.056 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.5572999999999999e-05 2024-09-17T07:10:37.056 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.056 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.056 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:10:37.056 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:45.692276+0000", 2024-09-17T07:10:37.057 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.3132999999999999e-05 2024-09-17T07:10:37.057 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.057 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.057 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:10:37.057 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:45.692284+0000", 2024-09-17T07:10:37.057 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.4670000000000004e-06 2024-09-17T07:10:37.057 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.057 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.057 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:10:37.057 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:45.692333+0000", 2024-09-17T07:10:37.057 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.8675999999999999e-05 2024-09-17T07:10:37.057 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.057 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.058 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:10:37.058 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:45.692465+0000", 2024-09-17T07:10:37.058 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00013242700000000001 2024-09-17T07:10:37.058 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.058 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.058 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 1,3", 2024-09-17T07:10:37.058 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:45.692605+0000", 2024-09-17T07:10:37.058 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000139393 2024-09-17T07:10:37.058 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.058 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.058 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:10:37.058 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:45.693737+0000", 2024-09-17T07:10:37.058 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0011323399999999999 2024-09-17T07:10:37.058 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.059 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.059 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:10:37.059 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:45.694476+0000", 2024-09-17T07:10:37.059 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00073894399999999995 2024-09-17T07:10:37.059 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.059 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.059 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:10:37.059 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:45.819349+0000", 2024-09-17T07:10:37.059 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.12487330200000001 2024-09-17T07:10:37.059 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.059 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.059 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:10:37.059 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:45.819411+0000", 2024-09-17T07:10:37.060 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.1389000000000005e-05 2024-09-17T07:10:37.060 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.060 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.060 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:10:37.060 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:45.819459+0000", 2024-09-17T07:10:37.060 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.8392e-05 2024-09-17T07:10:37.060 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:10:37.060 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:10:37.060 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:10:37.060 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.060 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.061 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:54893 7.10 7:0b6d2812:::benchmark_data_smithi100_78556_object54892:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e111)", 2024-09-17T07:10:37.061 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:04:57.338642+0000", 2024-09-17T07:10:37.061 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 339.70255703300001, 2024-09-17T07:10:37.061 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0185187929999999, 2024-09-17T07:10:37.061 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:10:37.061 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "reached pg", 2024-09-17T07:10:37.061 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:10:37.061 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:10:37.061 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:10:37.061 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 54893 2024-09-17T07:10:37.061 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.061 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:10:37.061 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.062 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:10:37.062 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:57.338642+0000", 2024-09-17T07:10:37.062 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:10:37.062 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.062 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.062 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:10:37.062 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:57.338642+0000", 2024-09-17T07:10:37.062 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:10:37.062 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.062 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.062 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:10:37.062 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:57.338640+0000", 2024-09-17T07:10:37.062 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:10:37.062 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.063 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.063 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:10:37.063 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:57.338677+0000", 2024-09-17T07:10:37.063 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.6844000000000002e-05 2024-09-17T07:10:37.063 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.063 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.063 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:10:37.063 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:57.338686+0000", 2024-09-17T07:10:37.063 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.2979999999999997e-06 2024-09-17T07:10:37.063 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.063 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.063 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:10:37.063 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:57.338693+0000", 2024-09-17T07:10:37.064 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.7700000000000004e-06 2024-09-17T07:10:37.064 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.064 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.064 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:10:37.064 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:57.411884+0000", 2024-09-17T07:10:37.064 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.073191292000000005 2024-09-17T07:10:37.064 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.064 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.064 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:10:37.064 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:57.411897+0000", 2024-09-17T07:10:37.064 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.257e-05 2024-09-17T07:10:37.064 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.064 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.065 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:10:37.065 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:58.357062+0000", 2024-09-17T07:10:37.065 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.94516572499999996 2024-09-17T07:10:37.065 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.065 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.065 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:10:37.065 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:58.357160+0000", 2024-09-17T07:10:37.065 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.7918999999999995e-05 2024-09-17T07:10:37.065 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:10:37.065 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:10:37.065 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:10:37.065 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.065 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.066 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:88337 7.0 7:06989d0c:::benchmark_data_smithi100_78556_object88336:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-09-17T07:10:37.066 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:05:35.697330+0000", 2024-09-17T07:10:37.066 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 301.34386809, 2024-09-17T07:10:37.066 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.14474398999999999, 2024-09-17T07:10:37.066 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:10:37.066 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:10:37.066 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:10:37.066 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:10:37.066 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:10:37.066 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 88337 2024-09-17T07:10:37.066 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.066 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:10:37.066 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.067 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:10:37.067 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:35.697330+0000", 2024-09-17T07:10:37.067 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:10:37.067 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.067 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.067 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:10:37.067 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:35.697330+0000", 2024-09-17T07:10:37.067 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:10:37.067 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.067 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.067 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:10:37.067 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:35.697329+0000", 2024-09-17T07:10:37.067 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999981 2024-09-17T07:10:37.068 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.068 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.068 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:10:37.068 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:35.697388+0000", 2024-09-17T07:10:37.068 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.9249000000000001e-05 2024-09-17T07:10:37.068 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.068 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.068 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:10:37.068 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:35.697419+0000", 2024-09-17T07:10:37.068 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.0796000000000001e-05 2024-09-17T07:10:37.068 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.068 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.068 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:10:37.068 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:35.697427+0000", 2024-09-17T07:10:37.069 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.3529999999999995e-06 2024-09-17T07:10:37.069 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.069 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.069 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:10:37.069 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:35.697495+0000", 2024-09-17T07:10:37.069 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.7799000000000006e-05 2024-09-17T07:10:37.069 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.069 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.069 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:10:37.069 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:35.697635+0000", 2024-09-17T07:10:37.069 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00013994699999999999 2024-09-17T07:10:37.069 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.069 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.070 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 1,3", 2024-09-17T07:10:37.070 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:35.697938+0000", 2024-09-17T07:10:37.070 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00030266400000000001 2024-09-17T07:10:37.070 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.070 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.070 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:10:37.070 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:35.700831+0000", 2024-09-17T07:10:37.070 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0028937619999999998 2024-09-17T07:10:37.070 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.070 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.070 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:10:37.070 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:35.700980+0000", 2024-09-17T07:10:37.070 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00014870899999999999 2024-09-17T07:10:37.070 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.071 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.071 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:10:37.071 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:35.841985+0000", 2024-09-17T07:10:37.071 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.141005303 2024-09-17T07:10:37.071 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.071 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.071 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:10:37.071 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:35.842037+0000", 2024-09-17T07:10:37.071 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.1598000000000003e-05 2024-09-17T07:10:37.071 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.071 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.071 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:10:37.071 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:35.842074+0000", 2024-09-17T07:10:37.072 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.7546999999999998e-05 2024-09-17T07:10:37.072 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:10:37.072 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:10:37.072 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:10:37.072 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.072 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.072 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:88342 7.0 7:06273ca1:::benchmark_data_smithi100_78556_object88341:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-09-17T07:10:37.072 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:05:35.701941+0000", 2024-09-17T07:10:37.072 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 301.33925775500001, 2024-09-17T07:10:37.072 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.14250195399999999, 2024-09-17T07:10:37.072 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:10:37.072 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:10:37.072 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:10:37.072 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:10:37.073 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:10:37.073 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 88342 2024-09-17T07:10:37.073 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.073 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:10:37.073 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.073 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:10:37.073 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:35.701941+0000", 2024-09-17T07:10:37.073 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:10:37.073 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.073 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.073 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:10:37.073 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:35.701941+0000", 2024-09-17T07:10:37.073 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:10:37.074 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.074 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.074 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:10:37.074 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:35.701939+0000", 2024-09-17T07:10:37.074 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999981 2024-09-17T07:10:37.074 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.074 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.074 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:10:37.074 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:35.701983+0000", 2024-09-17T07:10:37.074 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.4378000000000001e-05 2024-09-17T07:10:37.074 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.074 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.074 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:10:37.074 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:35.701999+0000", 2024-09-17T07:10:37.075 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.5353999999999999e-05 2024-09-17T07:10:37.075 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.075 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.075 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:10:37.075 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:35.702007+0000", 2024-09-17T07:10:37.075 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.7409999999999998e-06 2024-09-17T07:10:37.075 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.075 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.075 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:10:37.075 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:35.702174+0000", 2024-09-17T07:10:37.075 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00016615099999999999 2024-09-17T07:10:37.075 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.075 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.076 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:10:37.076 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:35.702315+0000", 2024-09-17T07:10:37.076 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000141674 2024-09-17T07:10:37.076 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.076 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.076 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 1,3", 2024-09-17T07:10:37.076 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:35.702460+0000", 2024-09-17T07:10:37.076 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00014431099999999999 2024-09-17T07:10:37.076 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.076 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.076 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:10:37.076 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:35.704244+0000", 2024-09-17T07:10:37.076 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0017842019999999999 2024-09-17T07:10:37.077 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.077 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.077 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:10:37.077 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:35.705733+0000", 2024-09-17T07:10:37.077 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0014894489999999999 2024-09-17T07:10:37.077 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.077 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.077 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:10:37.077 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:35.844335+0000", 2024-09-17T07:10:37.078 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.13860206799999999 2024-09-17T07:10:37.078 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.078 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.078 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:10:37.078 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:35.844409+0000", 2024-09-17T07:10:37.078 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.3363999999999998e-05 2024-09-17T07:10:37.078 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.078 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.078 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:10:37.078 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:35.844443+0000", 2024-09-17T07:10:37.078 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.4159999999999998e-05 2024-09-17T07:10:37.078 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:10:37.078 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:10:37.078 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:10:37.079 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.079 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.079 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:91337 7.0 7:03d2673e:::benchmark_data_smithi100_78556_object91336:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-17T07:10:37.079 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:05:39.395319+0000", 2024-09-17T07:10:37.079 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 297.64587997799998, 2024-09-17T07:10:37.079 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.147113518, 2024-09-17T07:10:37.079 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:10:37.079 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:10:37.079 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:10:37.079 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:10:37.079 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:10:37.079 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 91337 2024-09-17T07:10:37.079 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.080 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:10:37.080 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.080 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:10:37.080 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.395319+0000", 2024-09-17T07:10:37.080 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:10:37.080 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.080 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.080 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:10:37.080 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.395319+0000", 2024-09-17T07:10:37.080 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:10:37.080 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.080 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.080 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:10:37.081 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.395317+0000", 2024-09-17T07:10:37.081 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:10:37.081 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.081 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.081 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:10:37.081 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.395366+0000", 2024-09-17T07:10:37.081 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.8291999999999998e-05 2024-09-17T07:10:37.081 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.081 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.081 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:10:37.081 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.395376+0000", 2024-09-17T07:10:37.081 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0369e-05 2024-09-17T07:10:37.081 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.082 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.082 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:10:37.082 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.395381+0000", 2024-09-17T07:10:37.082 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.4539999999999998e-06 2024-09-17T07:10:37.082 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.082 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.082 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:10:37.083 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.395433+0000", 2024-09-17T07:10:37.083 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.1853e-05 2024-09-17T07:10:37.083 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.083 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.083 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:10:37.083 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.395518+0000", 2024-09-17T07:10:37.083 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.4369000000000005e-05 2024-09-17T07:10:37.083 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.083 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.083 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 1,3", 2024-09-17T07:10:37.083 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.395609+0000", 2024-09-17T07:10:37.083 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.1749999999999994e-05 2024-09-17T07:10:37.083 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.084 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.084 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:10:37.084 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.397143+0000", 2024-09-17T07:10:37.084 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0015340460000000001 2024-09-17T07:10:37.084 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.084 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.084 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:10:37.084 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.397825+0000", 2024-09-17T07:10:37.084 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00068158800000000001 2024-09-17T07:10:37.084 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.085 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.085 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:10:37.085 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.542333+0000", 2024-09-17T07:10:37.085 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.14450797700000001 2024-09-17T07:10:37.085 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.085 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.085 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:10:37.085 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.542386+0000", 2024-09-17T07:10:37.085 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.3467000000000002e-05 2024-09-17T07:10:37.085 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.085 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.085 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:10:37.085 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.542432+0000", 2024-09-17T07:10:37.086 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.57e-05 2024-09-17T07:10:37.086 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:10:37.086 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:10:37.086 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:10:37.086 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.086 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.086 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:99098 7.0 7:07fddb68:::benchmark_data_smithi100_78556_object99097:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-17T07:10:37.086 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:05:48.703621+0000", 2024-09-17T07:10:37.086 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 288.33757796200001, 2024-09-17T07:10:37.086 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.13504548299999999, 2024-09-17T07:10:37.086 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:10:37.086 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:10:37.086 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:10:37.087 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:10:37.087 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:10:37.087 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 99098 2024-09-17T07:10:37.087 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.087 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:10:37.087 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.087 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:10:37.087 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:48.703621+0000", 2024-09-17T07:10:37.087 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:10:37.087 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.087 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.087 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:10:37.087 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:48.703621+0000", 2024-09-17T07:10:37.087 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:10:37.088 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.088 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.088 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:10:37.088 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:48.703619+0000", 2024-09-17T07:10:37.088 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:10:37.088 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.088 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.088 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:10:37.088 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:48.703664+0000", 2024-09-17T07:10:37.088 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.4263e-05 2024-09-17T07:10:37.088 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.088 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.088 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:10:37.089 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:48.703675+0000", 2024-09-17T07:10:37.089 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.1134e-05 2024-09-17T07:10:37.089 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.089 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.089 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:10:37.089 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:48.703682+0000", 2024-09-17T07:10:37.089 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.379e-06 2024-09-17T07:10:37.089 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.089 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.089 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:10:37.089 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:48.703726+0000", 2024-09-17T07:10:37.089 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.4094000000000002e-05 2024-09-17T07:10:37.089 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.089 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.090 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:10:37.090 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:48.703836+0000", 2024-09-17T07:10:37.090 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00011032299999999999 2024-09-17T07:10:37.090 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.090 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.090 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 1,3", 2024-09-17T07:10:37.090 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:48.703944+0000", 2024-09-17T07:10:37.090 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000107051 2024-09-17T07:10:37.090 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.090 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.090 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:10:37.090 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:48.706186+0000", 2024-09-17T07:10:37.090 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0022428449999999998 2024-09-17T07:10:37.090 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.091 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.091 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:10:37.091 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:48.706440+0000", 2024-09-17T07:10:37.092 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00025352799999999999 2024-09-17T07:10:37.092 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.092 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.092 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:10:37.092 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:48.838601+0000", 2024-09-17T07:10:37.092 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.132161589 2024-09-17T07:10:37.092 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.092 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.092 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:10:37.092 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:48.838638+0000", 2024-09-17T07:10:37.092 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.6297000000000002e-05 2024-09-17T07:10:37.092 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.092 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.093 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:10:37.093 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:48.838666+0000", 2024-09-17T07:10:37.093 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.8311999999999999e-05 2024-09-17T07:10:37.093 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:10:37.093 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:10:37.093 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:10:37.093 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.093 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.093 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:99099 7.0 7:0793d0c4:::benchmark_data_smithi100_78556_object99098:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-17T07:10:37.093 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:05:48.704021+0000", 2024-09-17T07:10:37.093 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 288.33717783999998, 2024-09-17T07:10:37.093 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.13475561599999999, 2024-09-17T07:10:37.093 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:10:37.094 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:10:37.094 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:10:37.094 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:10:37.094 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:10:37.094 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 99099 2024-09-17T07:10:37.094 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.094 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:10:37.094 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.094 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:10:37.094 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:48.704021+0000", 2024-09-17T07:10:37.094 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:10:37.094 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.094 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.095 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:10:37.095 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:48.704021+0000", 2024-09-17T07:10:37.095 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:10:37.095 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.095 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.095 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:10:37.095 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:48.704020+0000", 2024-09-17T07:10:37.095 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999995 2024-09-17T07:10:37.095 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.095 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.095 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:10:37.095 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:48.704081+0000", 2024-09-17T07:10:37.095 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.1290000000000004e-05 2024-09-17T07:10:37.095 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.096 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.096 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:10:37.096 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:48.704090+0000", 2024-09-17T07:10:37.096 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.4290000000000001e-06 2024-09-17T07:10:37.096 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.096 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.096 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:10:37.096 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:48.704094+0000", 2024-09-17T07:10:37.096 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.4530000000000004e-06 2024-09-17T07:10:37.096 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.096 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.096 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:10:37.096 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:48.705185+0000", 2024-09-17T07:10:37.097 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0010908560000000001 2024-09-17T07:10:37.097 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.097 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.097 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:10:37.097 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:48.705274+0000", 2024-09-17T07:10:37.097 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.8845999999999996e-05 2024-09-17T07:10:37.097 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.097 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.097 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 1,3", 2024-09-17T07:10:37.097 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:48.705369+0000", 2024-09-17T07:10:37.097 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.5169999999999999e-05 2024-09-17T07:10:37.097 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.097 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.098 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:10:37.098 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:48.706281+0000", 2024-09-17T07:10:37.098 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00091163599999999996 2024-09-17T07:10:37.098 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.098 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.098 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:10:37.098 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:48.706542+0000", 2024-09-17T07:10:37.098 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00026105200000000001 2024-09-17T07:10:37.098 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.098 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.098 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:10:37.098 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:48.838732+0000", 2024-09-17T07:10:37.098 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.13219030000000001 2024-09-17T07:10:37.099 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.099 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.099 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:10:37.099 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:48.838758+0000", 2024-09-17T07:10:37.099 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.5488000000000001e-05 2024-09-17T07:10:37.099 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.099 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.099 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:10:37.099 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:48.838776+0000", 2024-09-17T07:10:37.099 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.8678e-05 2024-09-17T07:10:37.099 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:10:37.099 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:10:37.099 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:10:37.099 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.100 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.100 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.24929.0:101252 8.8 8:1b887a11:::benchmark_data_smithi100_102803_object101251:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e190)", 2024-09-17T07:10:37.100 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:09:17.058938+0000", 2024-09-17T07:10:37.100 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 79.982260922999998, 2024-09-17T07:10:37.100 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.13463831700000001, 2024-09-17T07:10:37.100 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:10:37.100 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:10:37.100 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:10:37.100 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.24929", 2024-09-17T07:10:37.100 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/564515582", 2024-09-17T07:10:37.100 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 101252 2024-09-17T07:10:37.100 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.100 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:10:37.101 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.101 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:10:37.101 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.058938+0000", 2024-09-17T07:10:37.101 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:10:37.101 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.101 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.101 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:10:37.101 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.058938+0000", 2024-09-17T07:10:37.101 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:10:37.101 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.101 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.101 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:10:37.101 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.058937+0000", 2024-09-17T07:10:37.102 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.999999 2024-09-17T07:10:37.102 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.102 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.102 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:10:37.102 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.058991+0000", 2024-09-17T07:10:37.102 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.3964e-05 2024-09-17T07:10:37.102 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.102 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.102 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:10:37.102 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.059000+0000", 2024-09-17T07:10:37.102 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.38e-06 2024-09-17T07:10:37.102 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.102 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.102 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:10:37.103 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.059007+0000", 2024-09-17T07:10:37.103 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.7229999999999998e-06 2024-09-17T07:10:37.103 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.103 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.103 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:10:37.103 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.059036+0000", 2024-09-17T07:10:37.103 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.9660000000000001e-05 2024-09-17T07:10:37.103 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.103 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.103 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:10:37.103 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.059130+0000", 2024-09-17T07:10:37.103 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.3706e-05 2024-09-17T07:10:37.103 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.103 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.104 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 3,7", 2024-09-17T07:10:37.104 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.059241+0000", 2024-09-17T07:10:37.104 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00011086699999999999 2024-09-17T07:10:37.104 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.104 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.104 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:10:37.104 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.064685+0000", 2024-09-17T07:10:37.104 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0054439969999999999 2024-09-17T07:10:37.104 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.104 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.104 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:10:37.104 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.064715+0000", 2024-09-17T07:10:37.104 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.9730000000000002e-05 2024-09-17T07:10:37.105 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.105 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.105 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:10:37.105 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.193448+0000", 2024-09-17T07:10:37.105 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.12873359000000001 2024-09-17T07:10:37.105 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.105 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.105 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:10:37.105 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.193530+0000", 2024-09-17T07:10:37.105 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.2287999999999995e-05 2024-09-17T07:10:37.105 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.105 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.105 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:10:37.106 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.193576+0000", 2024-09-17T07:10:37.106 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.5510999999999998e-05 2024-09-17T07:10:37.106 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:10:37.106 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:10:37.106 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:10:37.106 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.106 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.106 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.24929.0:101255 8.8 8:1e616ec4:::benchmark_data_smithi100_102803_object101254:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e190)", 2024-09-17T07:10:37.106 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:09:17.059907+0000", 2024-09-17T07:10:37.106 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 79.981291915, 2024-09-17T07:10:37.106 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.13385095299999999, 2024-09-17T07:10:37.106 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:10:37.106 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:10:37.107 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:10:37.107 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.24929", 2024-09-17T07:10:37.107 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/564515582", 2024-09-17T07:10:37.107 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 101255 2024-09-17T07:10:37.107 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.107 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:10:37.107 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.107 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:10:37.107 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.059907+0000", 2024-09-17T07:10:37.107 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:10:37.107 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.107 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.107 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:10:37.107 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.059907+0000", 2024-09-17T07:10:37.108 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:10:37.108 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.108 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.108 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:10:37.108 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.059906+0000", 2024-09-17T07:10:37.108 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.999999 2024-09-17T07:10:37.108 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.108 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.108 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:10:37.108 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.059965+0000", 2024-09-17T07:10:37.108 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.9457999999999999e-05 2024-09-17T07:10:37.108 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.108 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.109 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:10:37.109 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.059974+0000", 2024-09-17T07:10:37.109 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.7949999999999996e-06 2024-09-17T07:10:37.109 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.109 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.109 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:10:37.109 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.059979+0000", 2024-09-17T07:10:37.109 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.118e-06 2024-09-17T07:10:37.109 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.109 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.109 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:10:37.109 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.061382+0000", 2024-09-17T07:10:37.109 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.001403004 2024-09-17T07:10:37.110 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.110 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.110 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:10:37.110 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.061529+0000", 2024-09-17T07:10:37.110 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00014668600000000001 2024-09-17T07:10:37.110 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.110 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.110 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 3,7", 2024-09-17T07:10:37.110 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.061667+0000", 2024-09-17T07:10:37.110 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00013760700000000001 2024-09-17T07:10:37.110 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.110 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.110 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:10:37.111 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.064947+0000", 2024-09-17T07:10:37.111 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.003280687 2024-09-17T07:10:37.111 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.111 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.111 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:10:37.111 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.064956+0000", 2024-09-17T07:10:37.111 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.1910000000000006e-06 2024-09-17T07:10:37.111 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.111 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.111 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:10:37.111 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.193635+0000", 2024-09-17T07:10:37.111 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.12867822100000001 2024-09-17T07:10:37.111 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.112 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.112 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:10:37.112 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.193723+0000", 2024-09-17T07:10:37.112 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.8017999999999997e-05 2024-09-17T07:10:37.112 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.112 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.112 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:10:37.112 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.193758+0000", 2024-09-17T07:10:37.112 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.4958e-05 2024-09-17T07:10:37.112 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:10:37.112 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:10:37.112 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:10:37.112 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.112 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.113 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.24929.0:101257 8.4 8:2c725b9b:::benchmark_data_smithi100_102803_object101256:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e190)", 2024-09-17T07:10:37.113 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:09:17.061734+0000", 2024-09-17T07:10:37.113 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 79.979464987, 2024-09-17T07:10:37.113 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.128919799, 2024-09-17T07:10:37.113 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:10:37.113 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:10:37.113 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:10:37.113 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.24929", 2024-09-17T07:10:37.113 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/564515582", 2024-09-17T07:10:37.113 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 101257 2024-09-17T07:10:37.113 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.113 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:10:37.114 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.114 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:10:37.114 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.061734+0000", 2024-09-17T07:10:37.114 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:10:37.114 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.114 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.114 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:10:37.114 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.061734+0000", 2024-09-17T07:10:37.114 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:10:37.114 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.115 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.115 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:10:37.115 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.061733+0000", 2024-09-17T07:10:37.115 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999995 2024-09-17T07:10:37.115 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.115 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.115 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:10:37.115 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.061800+0000", 2024-09-17T07:10:37.115 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.7211000000000004e-05 2024-09-17T07:10:37.115 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.115 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.115 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:10:37.115 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.061810+0000", 2024-09-17T07:10:37.115 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.4129999999999995e-06 2024-09-17T07:10:37.116 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.116 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.116 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:10:37.116 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.061816+0000", 2024-09-17T07:10:37.116 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.9580000000000001e-06 2024-09-17T07:10:37.116 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.116 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.116 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:10:37.116 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.061859+0000", 2024-09-17T07:10:37.116 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.2036999999999997e-05 2024-09-17T07:10:37.116 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.116 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.116 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:10:37.117 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.061970+0000", 2024-09-17T07:10:37.117 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000111036 2024-09-17T07:10:37.117 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.117 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.117 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 2,4", 2024-09-17T07:10:37.117 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.062095+0000", 2024-09-17T07:10:37.117 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00012541200000000001 2024-09-17T07:10:37.117 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.117 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.117 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:10:37.117 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.065733+0000", 2024-09-17T07:10:37.117 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0036382060000000002 2024-09-17T07:10:37.117 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.118 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.118 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:10:37.118 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.065820+0000", 2024-09-17T07:10:37.118 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.6998999999999998e-05 2024-09-17T07:10:37.118 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.118 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.118 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:10:37.118 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.190550+0000", 2024-09-17T07:10:37.118 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.124730016 2024-09-17T07:10:37.118 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.118 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.118 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:10:37.118 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.190608+0000", 2024-09-17T07:10:37.119 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.7575999999999999e-05 2024-09-17T07:10:37.119 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.119 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.119 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:10:37.119 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.190653+0000", 2024-09-17T07:10:37.119 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.5639000000000001e-05 2024-09-17T07:10:37.119 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:10:37.119 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:10:37.119 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:10:37.119 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.119 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.119 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_repop(client.24929.0:101256 8.3 e190/168 8:cc13ab33:::benchmark_data_smithi100_102803_object101255:head v 190'6228, mlcod=190'6226)", 2024-09-17T07:10:37.119 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:09:17.063368+0000", 2024-09-17T07:10:37.119 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 79.977830808999997, 2024-09-17T07:10:37.120 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.126860263, 2024-09-17T07:10:37.120 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:10:37.120 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:10:37.120 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:10:37.120 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.120 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:10:37.120 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.063368+0000", 2024-09-17T07:10:37.120 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:10:37.120 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.120 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.120 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:10:37.120 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.063368+0000", 2024-09-17T07:10:37.121 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:10:37.121 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.121 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.121 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:10:37.121 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.063367+0000", 2024-09-17T07:10:37.121 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.999999 2024-09-17T07:10:37.121 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.121 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.121 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:10:37.121 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.063468+0000", 2024-09-17T07:10:37.121 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000101043 2024-09-17T07:10:37.121 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.121 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.122 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:10:37.122 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.063478+0000", 2024-09-17T07:10:37.122 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0222e-05 2024-09-17T07:10:37.122 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.122 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.122 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:10:37.122 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.063485+0000", 2024-09-17T07:10:37.122 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.652e-06 2024-09-17T07:10:37.122 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.122 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.122 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:10:37.122 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.063526+0000", 2024-09-17T07:10:37.122 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.0704999999999997e-05 2024-09-17T07:10:37.123 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.123 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.123 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:10:37.123 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.063544+0000", 2024-09-17T07:10:37.123 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.8391999999999999e-05 2024-09-17T07:10:37.123 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.123 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.123 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:10:37.123 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.190179+0000", 2024-09-17T07:10:37.123 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.126635472 2024-09-17T07:10:37.123 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.124 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.124 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:10:37.124 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.190228+0000", 2024-09-17T07:10:37.124 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.8656000000000003e-05 2024-09-17T07:10:37.124 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:10:37.124 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:10:37.124 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:10:37.124 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.124 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.124 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.24929.0:101263 8.8 8:1ccf7aea:::benchmark_data_smithi100_102803_object101262:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e190)", 2024-09-17T07:10:37.124 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:09:17.067005+0000", 2024-09-17T07:10:37.124 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 79.974193241999998, 2024-09-17T07:10:37.124 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.13779218600000001, 2024-09-17T07:10:37.125 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:10:37.125 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:10:37.125 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:10:37.125 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.24929", 2024-09-17T07:10:37.125 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/564515582", 2024-09-17T07:10:37.125 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 101263 2024-09-17T07:10:37.125 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.125 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:10:37.125 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.125 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:10:37.125 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.067005+0000", 2024-09-17T07:10:37.125 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:10:37.125 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.126 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.126 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:10:37.126 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.067005+0000", 2024-09-17T07:10:37.126 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:10:37.126 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.126 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.126 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:10:37.126 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.067004+0000", 2024-09-17T07:10:37.126 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:10:37.126 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.126 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.126 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:10:37.126 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.067067+0000", 2024-09-17T07:10:37.127 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.3522000000000006e-05 2024-09-17T07:10:37.127 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.127 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.127 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:10:37.127 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.067079+0000", 2024-09-17T07:10:37.127 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.149e-05 2024-09-17T07:10:37.127 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.127 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.127 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:10:37.127 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.067085+0000", 2024-09-17T07:10:37.127 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.686e-06 2024-09-17T07:10:37.127 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.127 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.127 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:10:37.128 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.067126+0000", 2024-09-17T07:10:37.128 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.0868999999999998e-05 2024-09-17T07:10:37.128 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.128 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.128 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:10:37.128 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.067259+0000", 2024-09-17T07:10:37.128 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000132278 2024-09-17T07:10:37.128 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.128 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.128 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 3,7", 2024-09-17T07:10:37.128 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.190580+0000", 2024-09-17T07:10:37.128 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.123321809 2024-09-17T07:10:37.129 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.129 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.129 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:10:37.129 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.196628+0000", 2024-09-17T07:10:37.129 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0060476690000000003 2024-09-17T07:10:37.129 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.129 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.129 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:10:37.129 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.202068+0000", 2024-09-17T07:10:37.129 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0054404140000000002 2024-09-17T07:10:37.129 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.129 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.129 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:10:37.129 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.204715+0000", 2024-09-17T07:10:37.130 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.002646893 2024-09-17T07:10:37.130 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.130 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.130 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:10:37.130 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.204762+0000", 2024-09-17T07:10:37.130 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.6995e-05 2024-09-17T07:10:37.130 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.130 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.130 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:10:37.130 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.204798+0000", 2024-09-17T07:10:37.130 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.5219999999999998e-05 2024-09-17T07:10:37.130 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:10:37.130 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:10:37.131 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:10:37.131 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.131 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.131 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_repop(client.24929.0:101264 8.2 e190/168 8:4e035611:::benchmark_data_smithi100_102803_object101263:head v 190'6500, mlcod=190'6498)", 2024-09-17T07:10:37.131 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:09:17.068440+0000", 2024-09-17T07:10:37.131 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 79.972758115999994, 2024-09-17T07:10:37.131 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.12738667300000001, 2024-09-17T07:10:37.131 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:10:37.131 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:10:37.131 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:10:37.131 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.131 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:10:37.131 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.068440+0000", 2024-09-17T07:10:37.131 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:10:37.132 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.132 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.132 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:10:37.132 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.068440+0000", 2024-09-17T07:10:37.132 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:10:37.132 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.132 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.132 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:10:37.132 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.068440+0000", 2024-09-17T07:10:37.132 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999995 2024-09-17T07:10:37.132 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.132 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.132 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:10:37.133 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.068468+0000", 2024-09-17T07:10:37.133 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.8619000000000002e-05 2024-09-17T07:10:37.133 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.133 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.133 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:10:37.133 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.190184+0000", 2024-09-17T07:10:37.133 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.121715428 2024-09-17T07:10:37.133 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.133 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.133 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:10:37.133 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.190200+0000", 2024-09-17T07:10:37.133 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.5778999999999999e-05 2024-09-17T07:10:37.133 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.134 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.134 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:10:37.134 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.190793+0000", 2024-09-17T07:10:37.134 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00059346500000000005 2024-09-17T07:10:37.134 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.134 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.134 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:10:37.134 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.190812+0000", 2024-09-17T07:10:37.134 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.8542e-05 2024-09-17T07:10:37.134 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.134 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.134 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:10:37.134 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.195798+0000", 2024-09-17T07:10:37.135 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0049864999999999996 2024-09-17T07:10:37.135 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.135 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:10:37.136 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.136 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:10:37.136 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.195827+0000", 2024-09-17T07:10:37.136 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.9002000000000001e-05 2024-09-17T07:10:37.137 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:10:37.137 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:10:37.137 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:10:37.137 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.137 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.137 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_repop(client.24929.0:101265 8.a e190/168 8:5d99f2b0:::benchmark_data_smithi100_102803_object101264:head v 190'6135, mlcod=190'6133)", 2024-09-17T07:10:37.137 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:09:17.068939+0000", 2024-09-17T07:10:37.137 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 79.972259321999999, 2024-09-17T07:10:37.137 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.126853823, 2024-09-17T07:10:37.137 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:10:37.137 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:10:37.137 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:10:37.137 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.138 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:10:37.138 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.068939+0000", 2024-09-17T07:10:37.138 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:10:37.138 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.138 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.138 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:10:37.138 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.068939+0000", 2024-09-17T07:10:37.138 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:10:37.138 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.138 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.138 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:10:37.138 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.068939+0000", 2024-09-17T07:10:37.138 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999995 2024-09-17T07:10:37.139 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.139 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.139 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:10:37.139 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.068998+0000", 2024-09-17T07:10:37.139 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.897e-05 2024-09-17T07:10:37.139 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.139 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.139 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:10:37.139 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.190186+0000", 2024-09-17T07:10:37.139 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.12118796799999999 2024-09-17T07:10:37.139 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.139 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.139 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:10:37.139 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.190200+0000", 2024-09-17T07:10:37.140 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.4021e-05 2024-09-17T07:10:37.140 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.140 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.140 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:10:37.140 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.190280+0000", 2024-09-17T07:10:37.140 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.0657999999999999e-05 2024-09-17T07:10:37.140 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.140 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.140 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:10:37.140 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.190304+0000", 2024-09-17T07:10:37.140 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.3288000000000001e-05 2024-09-17T07:10:37.140 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.141 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.141 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:10:37.141 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.195690+0000", 2024-09-17T07:10:37.141 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0053862049999999998 2024-09-17T07:10:37.141 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.141 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.141 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:10:37.141 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:17.195793+0000", 2024-09-17T07:10:37.141 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000103293 2024-09-17T07:10:37.141 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:10:37.141 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:10:37.141 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:10:37.141 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.142 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.142 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.24929.0:108591 8.8 8:1397d059:::benchmark_data_smithi100_102803_object108590:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e193)", 2024-09-17T07:10:37.142 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:09:23.376742+0000", 2024-09-17T07:10:37.142 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 73.664456358999999, 2024-09-17T07:10:37.142 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.1395554159999999, 2024-09-17T07:10:37.142 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:10:37.142 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "reached pg", 2024-09-17T07:10:37.142 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:10:37.142 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.24929", 2024-09-17T07:10:37.142 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/564515582", 2024-09-17T07:10:37.142 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 108591 2024-09-17T07:10:37.142 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.142 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:10:37.143 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.143 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:10:37.143 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:23.376742+0000", 2024-09-17T07:10:37.143 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:10:37.143 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.143 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.143 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:10:37.143 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:23.376742+0000", 2024-09-17T07:10:37.143 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:10:37.143 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.143 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.143 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:10:37.143 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:23.376731+0000", 2024-09-17T07:10:37.143 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.999989 2024-09-17T07:10:37.144 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.144 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.144 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:10:37.144 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:23.376775+0000", 2024-09-17T07:10:37.144 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.3896999999999997e-05 2024-09-17T07:10:37.144 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.144 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.144 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:10:37.144 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:23.376786+0000", 2024-09-17T07:10:37.144 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.097e-05 2024-09-17T07:10:37.144 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.144 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.144 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:10:37.145 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:23.376793+0000", 2024-09-17T07:10:37.145 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.1439999999999997e-06 2024-09-17T07:10:37.145 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.145 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.145 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:10:37.145 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:23.376967+0000", 2024-09-17T07:10:37.145 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00017330700000000001 2024-09-17T07:10:37.145 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.145 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.145 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:10:37.145 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:23.377101+0000", 2024-09-17T07:10:37.145 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00013460800000000001 2024-09-17T07:10:37.145 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.146 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.146 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 3,7", 2024-09-17T07:10:37.146 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:23.377254+0000", 2024-09-17T07:10:37.146 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00015275799999999999 2024-09-17T07:10:37.146 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.146 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.146 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:10:37.146 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:23.382723+0000", 2024-09-17T07:10:37.146 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0054693010000000002 2024-09-17T07:10:37.146 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.146 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.146 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:10:37.146 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:23.383671+0000", 2024-09-17T07:10:37.146 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00094795999999999997 2024-09-17T07:10:37.147 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.147 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.147 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:10:37.147 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:23.398000+0000", 2024-09-17T07:10:37.147 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.014329206000000001 2024-09-17T07:10:37.147 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.147 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.147 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:10:37.147 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:23.398009+0000", 2024-09-17T07:10:37.147 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.1680000000000004e-06 2024-09-17T07:10:37.147 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.147 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.147 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:10:37.148 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:24.387642+0000", 2024-09-17T07:10:37.148 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.98963316199999996 2024-09-17T07:10:37.148 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.148 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.148 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:10:37.148 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:24.387670+0000", 2024-09-17T07:10:37.148 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.7736999999999999e-05 2024-09-17T07:10:37.148 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.148 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.148 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:10:37.148 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:25.516225+0000", 2024-09-17T07:10:37.148 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.128555242 2024-09-17T07:10:37.148 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.149 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.149 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:10:37.149 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:25.516298+0000", 2024-09-17T07:10:37.149 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.2893999999999997e-05 2024-09-17T07:10:37.149 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:10:37.149 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:10:37.149 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:10:37.149 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.149 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.149 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.24929.0:108596 8.0 8:00252b92:::benchmark_data_smithi100_102803_object108595:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e194)", 2024-09-17T07:10:37.149 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:09:23.383049+0000", 2024-09-17T07:10:37.149 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 73.658149401000003, 2024-09-17T07:10:37.149 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.0834633180000002, 2024-09-17T07:10:37.149 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:10:37.150 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "reached pg", 2024-09-17T07:10:37.150 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:10:37.150 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.24929", 2024-09-17T07:10:37.150 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/564515582", 2024-09-17T07:10:37.150 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 108596 2024-09-17T07:10:37.150 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.150 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:10:37.150 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.150 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:10:37.150 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:23.383049+0000", 2024-09-17T07:10:37.150 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:10:37.150 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.150 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.151 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:10:37.151 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:23.383049+0000", 2024-09-17T07:10:37.151 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:10:37.151 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.151 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.151 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:10:37.151 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:23.383047+0000", 2024-09-17T07:10:37.151 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999981 2024-09-17T07:10:37.151 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.151 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.151 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:10:37.151 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:23.383083+0000", 2024-09-17T07:10:37.151 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.5309999999999999e-05 2024-09-17T07:10:37.152 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.152 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.152 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:10:37.152 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:23.383095+0000", 2024-09-17T07:10:37.152 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.2218000000000001e-05 2024-09-17T07:10:37.152 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.152 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.152 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:10:37.152 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:23.383102+0000", 2024-09-17T07:10:37.152 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.6480000000000003e-06 2024-09-17T07:10:37.152 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.152 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.152 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:10:37.153 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:23.383617+0000", 2024-09-17T07:10:37.153 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00051466999999999995 2024-09-17T07:10:37.153 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.153 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.153 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op must wait for map", 2024-09-17T07:10:37.153 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:23.383624+0000", 2024-09-17T07:10:37.153 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.6039999999999998e-06 2024-09-17T07:10:37.153 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.153 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.153 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:10:37.153 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:23.398129+0000", 2024-09-17T07:10:37.153 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.014505074999999999 2024-09-17T07:10:37.153 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.154 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.154 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:10:37.154 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:23.398139+0000", 2024-09-17T07:10:37.154 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.8230000000000006e-06 2024-09-17T07:10:37.154 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.154 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.154 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:10:37.154 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:24.387765+0000", 2024-09-17T07:10:37.154 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.98962641299999998 2024-09-17T07:10:37.154 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.154 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.154 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:10:37.154 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:24.387773+0000", 2024-09-17T07:10:37.154 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.7700000000000001e-06 2024-09-17T07:10:37.155 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.155 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.155 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:10:37.155 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:25.464988+0000", 2024-09-17T07:10:37.155 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.07721541 2024-09-17T07:10:37.155 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.155 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.155 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:10:37.155 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:25.466512+0000", 2024-09-17T07:10:37.155 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0015242630000000001 2024-09-17T07:10:37.155 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:10:37.155 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:10:37.155 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:10:37.156 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.156 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.156 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.24929.0:108611 8.8 8:12d8f62d:::benchmark_data_smithi100_102803_object108610:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-09-17T07:10:37.156 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:09:23.425265+0000", 2024-09-17T07:10:37.156 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 73.615933287000004, 2024-09-17T07:10:37.156 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.1042026979999999, 2024-09-17T07:10:37.156 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:10:37.156 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:10:37.156 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:10:37.156 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.24929", 2024-09-17T07:10:37.156 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/564515582", 2024-09-17T07:10:37.156 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 108611 2024-09-17T07:10:37.156 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.157 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:10:37.157 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.157 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:10:37.157 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:23.425265+0000", 2024-09-17T07:10:37.157 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:10:37.157 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.157 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.157 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:10:37.157 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:23.425265+0000", 2024-09-17T07:10:37.157 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:10:37.157 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.157 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.157 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:10:37.158 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:23.425255+0000", 2024-09-17T07:10:37.158 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.99999 2024-09-17T07:10:37.158 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.158 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.158 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:10:37.158 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:23.425299+0000", 2024-09-17T07:10:37.158 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.3364e-05 2024-09-17T07:10:37.158 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.158 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.158 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:10:37.158 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:23.425313+0000", 2024-09-17T07:10:37.158 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.4285e-05 2024-09-17T07:10:37.158 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.159 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.159 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:10:37.159 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:23.425321+0000", 2024-09-17T07:10:37.159 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.8520000000000004e-06 2024-09-17T07:10:37.159 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.159 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.159 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:10:37.159 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:23.425384+0000", 2024-09-17T07:10:37.159 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.2910999999999994e-05 2024-09-17T07:10:37.159 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.159 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.159 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:10:37.159 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:23.425433+0000", 2024-09-17T07:10:37.159 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.9424e-05 2024-09-17T07:10:37.160 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.160 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.160 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:10:37.160 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:24.387703+0000", 2024-09-17T07:10:37.160 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.96227026000000004 2024-09-17T07:10:37.160 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.160 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.160 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:10:37.160 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:24.387712+0000", 2024-09-17T07:10:37.160 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.3420000000000008e-06 2024-09-17T07:10:37.160 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.160 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.160 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:10:37.161 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:25.516572+0000", 2024-09-17T07:10:37.161 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.128860677 2024-09-17T07:10:37.161 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.161 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.161 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:10:37.161 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:25.516696+0000", 2024-09-17T07:10:37.161 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000123619 2024-09-17T07:10:37.161 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.161 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.161 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,3,7", 2024-09-17T07:10:37.161 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:25.517429+0000", 2024-09-17T07:10:37.161 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00073301200000000001 2024-09-17T07:10:37.161 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.161 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.162 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:10:37.162 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:25.529046+0000", 2024-09-17T07:10:37.162 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.011617136 2024-09-17T07:10:37.162 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.162 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.162 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:10:37.162 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:25.529116+0000", 2024-09-17T07:10:37.162 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.9560999999999993e-05 2024-09-17T07:10:37.162 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.162 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.162 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:10:37.162 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:25.529127+0000", 2024-09-17T07:10:37.162 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.1124999999999999e-05 2024-09-17T07:10:37.163 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.163 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.163 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:10:37.163 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:25.529319+0000", 2024-09-17T07:10:37.163 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000192578 2024-09-17T07:10:37.163 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.163 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.163 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:10:37.163 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:25.529362+0000", 2024-09-17T07:10:37.163 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.2305000000000002e-05 2024-09-17T07:10:37.163 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.163 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.163 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:10:37.163 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:25.529468+0000", 2024-09-17T07:10:37.164 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000106241 2024-09-17T07:10:37.164 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:10:37.164 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:10:37.164 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:10:37.164 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.164 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.164 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.24929.0:108614 8.8 8:128b071c:::benchmark_data_smithi100_102803_object108613:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-09-17T07:10:37.164 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:09:23.429623+0000", 2024-09-17T07:10:37.164 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 73.611575172000002, 2024-09-17T07:10:37.164 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.1100672880000002, 2024-09-17T07:10:37.164 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:10:37.164 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:10:37.164 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:10:37.165 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.24929", 2024-09-17T07:10:37.165 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/564515582", 2024-09-17T07:10:37.165 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 108614 2024-09-17T07:10:37.165 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.165 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:10:37.165 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.165 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:10:37.165 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:23.429623+0000", 2024-09-17T07:10:37.165 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:10:37.165 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.165 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.165 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:10:37.165 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:23.429623+0000", 2024-09-17T07:10:37.166 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:10:37.166 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.166 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.166 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:10:37.166 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:23.429622+0000", 2024-09-17T07:10:37.166 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:10:37.166 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.166 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.166 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:10:37.166 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:23.429670+0000", 2024-09-17T07:10:37.166 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.8279999999999999e-05 2024-09-17T07:10:37.166 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.166 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.167 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:10:37.167 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:23.429682+0000", 2024-09-17T07:10:37.167 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.1844e-05 2024-09-17T07:10:37.167 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.167 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.167 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:10:37.167 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:23.429688+0000", 2024-09-17T07:10:37.167 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.9390000000000002e-06 2024-09-17T07:10:37.167 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.167 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.167 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:10:37.167 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:23.429731+0000", 2024-09-17T07:10:37.167 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.2537999999999998e-05 2024-09-17T07:10:37.167 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.168 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.168 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:10:37.168 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:23.429739+0000", 2024-09-17T07:10:37.168 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.0020000000000006e-06 2024-09-17T07:10:37.168 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.168 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.168 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:10:37.168 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:24.387734+0000", 2024-09-17T07:10:37.168 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.95799508 2024-09-17T07:10:37.168 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.168 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.168 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:10:37.168 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:24.387742+0000", 2024-09-17T07:10:37.169 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.7829999999999999e-06 2024-09-17T07:10:37.169 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.169 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.169 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:10:37.169 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:25.526272+0000", 2024-09-17T07:10:37.169 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.138530515 2024-09-17T07:10:37.169 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.169 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.169 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:10:37.169 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:25.526411+0000", 2024-09-17T07:10:37.169 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00013934099999999999 2024-09-17T07:10:37.169 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.169 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.170 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,3,7", 2024-09-17T07:10:37.170 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:25.526582+0000", 2024-09-17T07:10:37.170 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00017097299999999999 2024-09-17T07:10:37.170 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.170 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.170 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:10:37.170 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:25.532450+0000", 2024-09-17T07:10:37.170 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0058678360000000004 2024-09-17T07:10:37.170 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.170 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.170 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:10:37.170 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:25.534965+0000", 2024-09-17T07:10:37.170 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.002515041 2024-09-17T07:10:37.171 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.171 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.171 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:10:37.171 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:25.538338+0000", 2024-09-17T07:10:37.171 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0033725859999999999 2024-09-17T07:10:37.171 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.171 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.171 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:10:37.171 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:25.539475+0000", 2024-09-17T07:10:37.171 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.001137021 2024-09-17T07:10:37.171 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.171 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.171 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:10:37.172 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:25.539527+0000", 2024-09-17T07:10:37.172 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.1897000000000001e-05 2024-09-17T07:10:37.172 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.172 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.172 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:10:37.172 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:25.539691+0000", 2024-09-17T07:10:37.172 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00016390600000000001 2024-09-17T07:10:37.172 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:10:37.172 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:10:37.172 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:10:37.172 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.172 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.172 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.24929.0:108617 8.0 8:07b3f4db:::benchmark_data_smithi100_102803_object108616:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-09-17T07:10:37.173 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:09:23.433279+0000", 2024-09-17T07:10:37.173 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 73.607919237000004, 2024-09-17T07:10:37.173 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.0496449120000002, 2024-09-17T07:10:37.173 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:10:37.173 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:10:37.173 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:10:37.173 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.24929", 2024-09-17T07:10:37.173 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/564515582", 2024-09-17T07:10:37.173 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 108617 2024-09-17T07:10:37.173 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.173 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:10:37.173 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.173 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:10:37.173 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:23.433279+0000", 2024-09-17T07:10:37.174 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:10:37.174 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.174 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.174 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:10:37.174 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:23.433279+0000", 2024-09-17T07:10:37.174 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:10:37.174 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.174 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.174 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:10:37.174 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:23.433273+0000", 2024-09-17T07:10:37.174 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999938 2024-09-17T07:10:37.174 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.174 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.175 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:10:37.175 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:23.433323+0000", 2024-09-17T07:10:37.175 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.0145000000000002e-05 2024-09-17T07:10:37.175 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.175 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.175 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:10:37.175 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:23.433349+0000", 2024-09-17T07:10:37.175 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.5426999999999999e-05 2024-09-17T07:10:37.175 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.175 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.175 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:10:37.175 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:23.433362+0000", 2024-09-17T07:10:37.175 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.3576e-05 2024-09-17T07:10:37.175 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.176 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.176 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:10:37.176 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:23.433463+0000", 2024-09-17T07:10:37.176 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00010019800000000001 2024-09-17T07:10:37.176 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.176 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.176 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:10:37.176 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:23.433492+0000", 2024-09-17T07:10:37.176 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.9312e-05 2024-09-17T07:10:37.176 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.176 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.176 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:10:37.176 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:24.387795+0000", 2024-09-17T07:10:37.177 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.95430293099999997 2024-09-17T07:10:37.177 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.177 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.177 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:10:37.177 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:24.387803+0000", 2024-09-17T07:10:37.177 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.7100000000000007e-06 2024-09-17T07:10:37.177 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.177 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.177 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:10:37.177 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:25.468245+0000", 2024-09-17T07:10:37.177 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.080442036 2024-09-17T07:10:37.177 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.177 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.177 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:10:37.178 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:25.468859+0000", 2024-09-17T07:10:37.178 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00061439399999999999 2024-09-17T07:10:37.178 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.178 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.178 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 1,3,4", 2024-09-17T07:10:37.178 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:25.469133+0000", 2024-09-17T07:10:37.178 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00027381699999999998 2024-09-17T07:10:37.178 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.178 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.179 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:10:37.179 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:25.475887+0000", 2024-09-17T07:10:37.179 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0067536849999999997 2024-09-17T07:10:37.179 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.179 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.179 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:10:37.179 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:25.477255+0000", 2024-09-17T07:10:37.179 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.001368593 2024-09-17T07:10:37.179 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.179 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.179 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:10:37.179 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:25.478000+0000", 2024-09-17T07:10:37.179 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00074448799999999996 2024-09-17T07:10:37.180 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.180 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.180 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:10:37.180 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:25.482753+0000", 2024-09-17T07:10:37.180 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0047536089999999998 2024-09-17T07:10:37.180 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.180 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.180 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:10:37.180 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:25.482812+0000", 2024-09-17T07:10:37.180 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.9159000000000001e-05 2024-09-17T07:10:37.180 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:37.180 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:37.180 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:10:37.181 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:09:25.482924+0000", 2024-09-17T07:10:37.181 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00011185500000000001 2024-09-17T07:10:37.181 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:10:37.181 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:10:37.181 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:10:37.181 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:10:37.181 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:10:37.181 INFO:teuthology.orchestra.run.smithi135.stdout:} 2024-09-17T07:10:37.208 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:37.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:37.209 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:10:37.291 INFO:teuthology.orchestra.run.smithi100.stderr:set pool 8 pgp_num to 16 2024-09-17T07:10:37.310 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:10:37.315 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:10:37.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:37.353 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:37.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:10:37.454 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:10:37.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:37.488 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:37.488 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:10:37.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:37 smithi135 ceph-mon[33238]: 2.14 deep-scrub starts 2024-09-17T07:10:37.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:37 smithi135 ceph-mon[33238]: 2.14 deep-scrub ok 2024-09-17T07:10:37.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:37 smithi135 ceph-mon[33238]: 5.4 scrub starts 2024-09-17T07:10:37.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:37 smithi135 ceph-mon[33238]: 5.4 scrub ok 2024-09-17T07:10:37.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:37 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2020176286' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]: dispatch 2024-09-17T07:10:37.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:37 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]: dispatch 2024-09-17T07:10:37.589 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:10:37.623 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:37.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:37.624 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:10:37.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:37 smithi100 ceph-mon[29092]: 2.14 deep-scrub starts 2024-09-17T07:10:37.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:37 smithi100 ceph-mon[29092]: 2.14 deep-scrub ok 2024-09-17T07:10:37.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:37 smithi100 ceph-mon[29092]: 5.4 scrub starts 2024-09-17T07:10:37.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:37 smithi100 ceph-mon[29092]: 5.4 scrub ok 2024-09-17T07:10:37.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:37 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2020176286' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]: dispatch 2024-09-17T07:10:37.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:37 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]: dispatch 2024-09-17T07:10:37.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:37 smithi100 ceph-mon[39462]: 2.14 deep-scrub starts 2024-09-17T07:10:37.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:37 smithi100 ceph-mon[39462]: 2.14 deep-scrub ok 2024-09-17T07:10:37.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:37 smithi100 ceph-mon[39462]: 5.4 scrub starts 2024-09-17T07:10:37.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:37 smithi100 ceph-mon[39462]: 5.4 scrub ok 2024-09-17T07:10:37.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:37 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2020176286' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]: dispatch 2024-09-17T07:10:37.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:37 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]: dispatch 2024-09-17T07:10:37.725 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:10:37.760 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:37.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:37.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:10:37.862 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:10:37.899 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:37.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:37.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:10:38.001 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:10:38.037 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:38.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:38.038 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:10:38.139 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:10:38.176 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:38.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:38.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:10:38.278 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:10:38.293 INFO:teuthology.orchestra.run.smithi100.stderr:noscrub is set 2024-09-17T07:10:38.320 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:38.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:38.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:10:38.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:38 smithi100 ceph-mon[29092]: pgmap v371: 148 pgs: 1 active+remapped+backfilling, 4 active+clean+scrubbing, 143 active+clean; 5.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 928 op/s; 3566/244488 objects misplaced (1.459%); 20 MiB/s, 312 objects/s recovering 2024-09-17T07:10:38.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:38 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]': finished 2024-09-17T07:10:38.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:38 smithi100 ceph-mon[29092]: osdmap e212: 8 total, 8 up, 7 in 2024-09-17T07:10:38.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:38 smithi100 ceph-mon[29092]: 6.14 scrub starts 2024-09-17T07:10:38.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:38 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/589504844' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:10:38.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:38 smithi100 ceph-mon[29092]: 1.0 deep-scrub starts 2024-09-17T07:10:38.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:38 smithi100 ceph-mon[29092]: 1.0 deep-scrub ok 2024-09-17T07:10:38.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:38 smithi100 ceph-mon[39462]: pgmap v371: 148 pgs: 1 active+remapped+backfilling, 4 active+clean+scrubbing, 143 active+clean; 5.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 928 op/s; 3566/244488 objects misplaced (1.459%); 20 MiB/s, 312 objects/s recovering 2024-09-17T07:10:38.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:38 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]': finished 2024-09-17T07:10:38.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:38 smithi100 ceph-mon[39462]: osdmap e212: 8 total, 8 up, 7 in 2024-09-17T07:10:38.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:38 smithi100 ceph-mon[39462]: 6.14 scrub starts 2024-09-17T07:10:38.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:38 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/589504844' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:10:38.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:38 smithi100 ceph-mon[39462]: 1.0 deep-scrub starts 2024-09-17T07:10:38.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:38 smithi100 ceph-mon[39462]: 1.0 deep-scrub ok 2024-09-17T07:10:38.421 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:10:38.454 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:38.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:38.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:10:38.505 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:38 smithi135 ceph-mon[33238]: pgmap v371: 148 pgs: 1 active+remapped+backfilling, 4 active+clean+scrubbing, 143 active+clean; 5.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 928 op/s; 3566/244488 objects misplaced (1.459%); 20 MiB/s, 312 objects/s recovering 2024-09-17T07:10:38.505 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:38 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]': finished 2024-09-17T07:10:38.505 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:38 smithi135 ceph-mon[33238]: osdmap e212: 8 total, 8 up, 7 in 2024-09-17T07:10:38.505 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:38 smithi135 ceph-mon[33238]: 6.14 scrub starts 2024-09-17T07:10:38.505 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:38 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/589504844' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:10:38.505 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:38 smithi135 ceph-mon[33238]: 1.0 deep-scrub starts 2024-09-17T07:10:38.506 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:38 smithi135 ceph-mon[33238]: 1.0 deep-scrub ok 2024-09-17T07:10:38.556 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:10:38.583 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:38.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:38.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:10:38.609 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-17T07:10:38.685 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:10:38.711 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:38.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:38.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:10:38.793 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:10:38 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:10:38.813 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:10:38.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:38.860 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:38.860 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:10:38.937 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:10:38.962 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:10:38.993 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:38.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:38.994 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:10:39.095 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:10:39.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:39.143 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:39.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:10:39.244 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:10:39.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:39.289 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:39.289 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:10:39.390 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:10:39.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:39.414 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:39.414 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:10:39.437 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:39 smithi135 ceph-mon[33238]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:10:39.437 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:39 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/589504844' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T07:10:39.437 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:39 smithi135 ceph-mon[33238]: osdmap e213: 8 total, 8 up, 7 in 2024-09-17T07:10:39.437 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:39 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:10:39.438 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:39 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:10:39.438 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:39 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:10:39.438 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:39 smithi135 ceph-mon[33238]: pgmap v374: 148 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 146 active+clean; 4.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.48k op/s; 2718/228768 objects misplaced (1.188%); 9.1 MiB/s, 145 objects/s recovering 2024-09-17T07:10:39.515 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:10:39.547 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:39.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:39.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:10:39.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:39 smithi100 ceph-mon[29092]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:10:39.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:39 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/589504844' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T07:10:39.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:39 smithi100 ceph-mon[29092]: osdmap e213: 8 total, 8 up, 7 in 2024-09-17T07:10:39.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:39 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:10:39.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:39 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:10:39.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:39 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:10:39.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:39 smithi100 ceph-mon[29092]: pgmap v374: 148 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 146 active+clean; 4.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.48k op/s; 2718/228768 objects misplaced (1.188%); 9.1 MiB/s, 145 objects/s recovering 2024-09-17T07:10:39.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:39 smithi100 ceph-mon[39462]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:10:39.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:39 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/589504844' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T07:10:39.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:39 smithi100 ceph-mon[39462]: osdmap e213: 8 total, 8 up, 7 in 2024-09-17T07:10:39.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:39 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:10:39.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:39 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:10:39.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:39 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:10:39.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:39 smithi100 ceph-mon[39462]: pgmap v374: 148 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 146 active+clean; 4.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.48k op/s; 2718/228768 objects misplaced (1.188%); 9.1 MiB/s, 145 objects/s recovering 2024-09-17T07:10:39.648 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:10:39.684 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:39.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:39.685 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:10:39.786 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:10:39.807 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:39.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:39.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:10:39.909 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:10:39.946 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:39.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:39.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:10:40.048 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:10:40.079 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:40.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:40.080 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:10:40.180 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:10:40.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:40.211 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:40.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:10:40.312 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:10:40.341 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:40.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:40.342 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:10:40.443 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:10:40.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:40.479 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:40.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:10:40.537 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:40 smithi135 ceph-mon[33238]: osdmap e214: 8 total, 8 up, 7 in 2024-09-17T07:10:40.580 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:10:40.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:40.619 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:40.619 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:10:40.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:40 smithi100 ceph-mon[29092]: osdmap e214: 8 total, 8 up, 7 in 2024-09-17T07:10:40.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:40 smithi100 ceph-mon[39462]: osdmap e214: 8 total, 8 up, 7 in 2024-09-17T07:10:40.720 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:10:40.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:40.753 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:40.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:10:40.854 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:10:40.891 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:40.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:40.892 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:10:40.993 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:10:41.024 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:41.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:41.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:10:41.126 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:10:41.169 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:41.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:41.170 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:10:41.271 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:10:41.296 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:41.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:41.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:10:41.398 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:10:41.429 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:41.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:41.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:10:41.503 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:41 smithi135 ceph-mon[33238]: pgmap v376: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 4.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.08k op/s; 2718/225093 objects misplaced (1.208%); 6.3 MiB/s, 100 objects/s recovering 2024-09-17T07:10:41.531 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:10:41.561 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:41.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:41.562 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:10:41.563 INFO:teuthology.orchestra.run.smithi135.stdout:{ 2024-09-17T07:10:41.563 INFO:teuthology.orchestra.run.smithi135.stdout: "ops": [ 2024-09-17T07:10:41.563 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:41.563 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.24929.0:189620 8.5 8:a300f2b6:::benchmark_data_smithi100_102803_object59116:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-09-17T07:10:41.563 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:10:41.555135+0000", 2024-09-17T07:10:41.563 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 0.0050971209999999996, 2024-09-17T07:10:41.563 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0051289509999999996, 2024-09-17T07:10:41.563 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:10:41.563 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "waiting for sub ops", 2024-09-17T07:10:41.563 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:10:41.563 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.24929", 2024-09-17T07:10:41.563 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/564515582", 2024-09-17T07:10:41.564 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 189620 2024-09-17T07:10:41.564 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:41.564 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:10:41.564 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:41.564 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:10:41.564 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:41.555135+0000", 2024-09-17T07:10:41.564 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:10:41.564 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:41.564 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:41.564 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:10:41.564 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:41.555135+0000", 2024-09-17T07:10:41.564 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:10:41.564 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:41.565 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:41.565 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:10:41.565 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:41.555133+0000", 2024-09-17T07:10:41.565 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:10:41.565 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:41.565 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:41.565 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:10:41.565 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:41.555140+0000", 2024-09-17T07:10:41.565 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.5830000000000003e-06 2024-09-17T07:10:41.565 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:41.565 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:41.565 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:10:41.566 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:41.555151+0000", 2024-09-17T07:10:41.566 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.146e-05 2024-09-17T07:10:41.566 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:41.566 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:41.566 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:10:41.566 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:41.555158+0000", 2024-09-17T07:10:41.566 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.1380000000000001e-06 2024-09-17T07:10:41.566 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:41.566 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:41.566 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:10:41.566 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:41.555368+0000", 2024-09-17T07:10:41.566 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00021013999999999999 2024-09-17T07:10:41.566 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:41.567 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:41.567 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:10:41.567 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:41.555797+0000", 2024-09-17T07:10:41.567 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0004284 2024-09-17T07:10:41.567 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:41.567 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:41.567 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 4,7", 2024-09-17T07:10:41.567 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:41.556201+0000", 2024-09-17T07:10:41.567 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00040382100000000001 2024-09-17T07:10:41.567 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:10:41.567 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:10:41.567 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:10:41.567 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:41.568 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:41.568 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_repop(client.24929.0:189619 8.3 e214/168 8:c0849b85:::benchmark_data_smithi100_102803_object59115:head v 214'11633, mlcod=214'11631)", 2024-09-17T07:10:41.568 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:10:41.555862+0000", 2024-09-17T07:10:41.568 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 0.0043696830000000001, 2024-09-17T07:10:41.568 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0044867140000000002, 2024-09-17T07:10:41.568 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:10:41.568 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "started", 2024-09-17T07:10:41.568 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:10:41.568 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:41.568 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:10:41.568 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:41.555862+0000", 2024-09-17T07:10:41.568 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:10:41.568 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:41.568 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:41.569 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:10:41.569 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:41.555862+0000", 2024-09-17T07:10:41.569 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:10:41.569 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:41.569 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:41.569 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:10:41.569 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:41.555861+0000", 2024-09-17T07:10:41.569 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.999999 2024-09-17T07:10:41.569 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:41.569 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:41.569 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:10:41.569 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:41.555872+0000", 2024-09-17T07:10:41.570 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.1308000000000001e-05 2024-09-17T07:10:41.570 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:41.570 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:41.570 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:10:41.570 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:41.555883+0000", 2024-09-17T07:10:41.570 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.039e-05 2024-09-17T07:10:41.570 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:41.570 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:41.570 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:10:41.570 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:41.555889+0000", 2024-09-17T07:10:41.570 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.2439999999999998e-06 2024-09-17T07:10:41.570 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:41.570 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:41.570 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:10:41.571 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:41.556909+0000", 2024-09-17T07:10:41.571 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.001019998 2024-09-17T07:10:41.571 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:41.571 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:41.571 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:10:41.571 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:41.556962+0000", 2024-09-17T07:10:41.571 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.3149999999999998e-05 2024-09-17T07:10:41.571 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:10:41.571 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:10:41.571 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:10:41.571 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:41.571 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:41.571 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_repop(client.24929.0:189622 8.6 e214/168 8:62461917:::benchmark_data_smithi100_102803_object59118:head v 214'11936, mlcod=214'11934)", 2024-09-17T07:10:41.572 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:10:41.555878+0000", 2024-09-17T07:10:41.572 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 0.0043542349999999997, 2024-09-17T07:10:41.572 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.004531775, 2024-09-17T07:10:41.572 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:10:41.572 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "started", 2024-09-17T07:10:41.572 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:10:41.572 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:41.572 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:10:41.572 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:41.555878+0000", 2024-09-17T07:10:41.572 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:10:41.572 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:41.572 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:41.573 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:10:41.573 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:41.555878+0000", 2024-09-17T07:10:41.573 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:10:41.573 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:41.573 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:41.573 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:10:41.573 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:41.555877+0000", 2024-09-17T07:10:41.573 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.999999 2024-09-17T07:10:41.573 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:41.573 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:41.573 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:10:41.573 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:41.555887+0000", 2024-09-17T07:10:41.574 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.8449999999999998e-06 2024-09-17T07:10:41.574 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:41.574 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:41.574 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:10:41.574 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:41.555896+0000", 2024-09-17T07:10:41.574 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.3149999999999998e-06 2024-09-17T07:10:41.574 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:41.574 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:41.574 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:10:41.574 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:41.555902+0000", 2024-09-17T07:10:41.574 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.1580000000000002e-06 2024-09-17T07:10:41.574 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:41.574 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:41.575 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:10:41.575 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:41.558698+0000", 2024-09-17T07:10:41.575 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.002796055 2024-09-17T07:10:41.575 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:41.575 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:41.575 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:10:41.575 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:41.558718+0000", 2024-09-17T07:10:41.575 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.9975000000000001e-05 2024-09-17T07:10:41.575 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:10:41.575 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:10:41.575 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:10:41.575 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:41.576 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:41.576 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_repop(client.24929.0:189618 8.7 e214/206 8:e4d9374f:::benchmark_data_smithi100_102803_object59114:head v 214'11923, mlcod=214'11920)", 2024-09-17T07:10:41.576 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:10:41.556240+0000", 2024-09-17T07:10:41.576 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 0.0039916070000000003, 2024-09-17T07:10:41.576 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0042201280000000001, 2024-09-17T07:10:41.576 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:10:41.576 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "started", 2024-09-17T07:10:41.576 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:10:41.576 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:41.576 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:10:41.576 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:41.556240+0000", 2024-09-17T07:10:41.576 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:10:41.577 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:41.577 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:41.577 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:10:41.577 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:41.556240+0000", 2024-09-17T07:10:41.577 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:10:41.577 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:41.577 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:41.577 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:10:41.577 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:41.556240+0000", 2024-09-17T07:10:41.577 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999995 2024-09-17T07:10:41.577 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:41.577 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:41.577 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:10:41.578 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:41.556246+0000", 2024-09-17T07:10:41.578 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.268e-06 2024-09-17T07:10:41.578 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:41.578 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:41.578 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:10:41.578 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:41.556260+0000", 2024-09-17T07:10:41.578 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.431e-05 2024-09-17T07:10:41.578 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:41.578 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:41.578 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:10:41.578 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:41.556267+0000", 2024-09-17T07:10:41.578 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.8820000000000003e-06 2024-09-17T07:10:41.579 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:41.579 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:41.579 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:10:41.579 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:41.556327+0000", 2024-09-17T07:10:41.579 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.9583999999999997e-05 2024-09-17T07:10:41.579 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:41.579 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:41.579 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:10:41.579 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:41.556344+0000", 2024-09-17T07:10:41.579 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.7042999999999999e-05 2024-09-17T07:10:41.580 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:10:41.580 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:10:41.580 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:10:41.580 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:41.580 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:41.580 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_repop(client.24929.0:189621 8.7 e214/206 8:e9e88d76:::benchmark_data_smithi100_102803_object59117:head v 214'11924, mlcod=214'11920)", 2024-09-17T07:10:41.580 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:10:41.558198+0000", 2024-09-17T07:10:41.580 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 0.0020337440000000001, 2024-09-17T07:10:41.580 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0023148650000000002, 2024-09-17T07:10:41.580 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:10:41.580 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "started", 2024-09-17T07:10:41.580 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:10:41.581 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:41.581 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:10:41.581 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:41.558198+0000", 2024-09-17T07:10:41.581 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:10:41.581 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:41.581 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:41.581 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:10:41.581 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:41.558198+0000", 2024-09-17T07:10:41.581 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:10:41.581 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:41.581 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:41.581 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:10:41.581 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:41.558197+0000", 2024-09-17T07:10:41.582 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.999999 2024-09-17T07:10:41.582 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:41.582 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:41.582 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:10:41.582 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:41.558203+0000", 2024-09-17T07:10:41.582 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.3500000000000002e-06 2024-09-17T07:10:41.582 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:41.582 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:41.582 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:10:41.582 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:41.558217+0000", 2024-09-17T07:10:41.582 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.4051e-05 2024-09-17T07:10:41.582 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:41.582 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:41.583 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:10:41.583 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:41.558224+0000", 2024-09-17T07:10:41.583 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.5490000000000003e-06 2024-09-17T07:10:41.583 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:41.583 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:41.583 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:10:41.583 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:41.558878+0000", 2024-09-17T07:10:41.583 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00065380899999999999 2024-09-17T07:10:41.583 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:41.583 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:41.583 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:10:41.583 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:41.558925+0000", 2024-09-17T07:10:41.584 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.6768999999999999e-05 2024-09-17T07:10:41.584 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:10:41.584 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:10:41.584 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:10:41.584 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:10:41.584 INFO:teuthology.orchestra.run.smithi135.stdout: ], 2024-09-17T07:10:41.584 INFO:teuthology.orchestra.run.smithi135.stdout: "num_ops": 5 2024-09-17T07:10:41.584 INFO:teuthology.orchestra.run.smithi135.stdout:} 2024-09-17T07:10:41.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:41 smithi100 ceph-mon[29092]: pgmap v376: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 4.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.08k op/s; 2718/225093 objects misplaced (1.208%); 6.3 MiB/s, 100 objects/s recovering 2024-09-17T07:10:41.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:41 smithi100 ceph-mon[39462]: pgmap v376: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 4.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.08k op/s; 2718/225093 objects misplaced (1.208%); 6.3 MiB/s, 100 objects/s recovering 2024-09-17T07:10:41.663 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:10:41.698 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:41.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:41.699 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:10:41.800 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:10:41.839 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:41.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:41.840 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:10:41.940 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:10:41.983 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:41.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:41.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:10:42.085 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:10:42.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:42.117 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:42.117 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:10:42.218 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:10:42.249 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:42.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:42.250 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:10:42.273 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-17T07:10:42.351 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:10:42.385 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:42.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:42.386 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:10:42.486 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:10:42.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:42.541 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:42.542 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:10:42.630 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:10:42.643 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:10:42.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:42.694 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:42.694 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:10:42.795 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:10:42.828 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:42.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:42.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:10:42.930 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:10:42.967 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:42.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:42.968 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:10:43.068 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:10:43.107 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:43.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:43.108 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:10:43.151 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7] out_osds: [3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-17T07:10:43.151 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd reweight 0 1 2024-09-17T07:10:43.210 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:10:43.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:43.254 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:43.254 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:10:43.355 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:10:43.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:43.400 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:43.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:10:43.501 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:10:43.543 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:43.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:43.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:10:43.645 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:10:43.656 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:10:43.656 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:10:43.672 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:43.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:43.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:10:43.692 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1: {} 2024-09-17T07:10:43.692 INFO:teuthology.orchestra.run.smithi100.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:10:43.724 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:10:43.724 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:10:43.726 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:10:43.757 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:10:43.757 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:10:43.774 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:10:43.790 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:10:43.790 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:10:43.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:43.814 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:43.814 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:10:43.829 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:10:43.830 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:10:43.863 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:10:43.863 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:10:43.892 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:10:43.893 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:10:43.914 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:10:43.932 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-17T07:10:43.960 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:43.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:43.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:10:44.062 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:10:44.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:44.100 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:44.100 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:10:44.201 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:10:44.227 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:44.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:44.228 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:10:44.328 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:10:44.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:44.359 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:44.359 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:10:44.451 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:44 smithi135 ceph-mon[33238]: pgmap v377: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 4.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.21k op/s; 2718/222777 objects misplaced (1.220%); 6.3 MiB/s, 100 objects/s recovering 2024-09-17T07:10:44.451 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:44 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/267392219' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:10:44.460 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:10:44.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:44.496 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:44.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:10:44.540 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:44 smithi100 ceph-mon[29092]: pgmap v377: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 4.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.21k op/s; 2718/222777 objects misplaced (1.220%); 6.3 MiB/s, 100 objects/s recovering 2024-09-17T07:10:44.540 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:44 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/267392219' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:10:44.541 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:44 smithi100 ceph-mon[39462]: pgmap v377: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 4.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.21k op/s; 2718/222777 objects misplaced (1.220%); 6.3 MiB/s, 100 objects/s recovering 2024-09-17T07:10:44.541 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:44 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/267392219' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:10:44.598 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:10:44.619 INFO:teuthology.orchestra.run.smithi135.stdout:ERROR: (22) Invalid argument 2024-09-17T07:10:44.619 INFO:teuthology.orchestra.run.smithi135.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:10:44.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:44.631 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:44.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:10:44.732 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:10:44.773 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:44.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:44.774 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:10:44.875 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:10:44.911 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:44.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:44.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:10:45.013 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:10:45.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:45.048 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:45.048 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:10:45.149 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:10:45.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:45.177 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:45.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:10:45.278 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:10:45.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:45.312 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:45.312 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:10:45.413 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:10:45.444 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:45.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:45.445 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:10:45.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:45 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:10:45.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:45 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:10:45.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:45 smithi135 ceph-mon[33238]: osdmap e215: 8 total, 8 up, 7 in 2024-09-17T07:10:45.546 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:10:45.581 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:45.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:45.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:10:45.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:45 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:10:45.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:45 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:10:45.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:45 smithi100 ceph-mon[29092]: osdmap e215: 8 total, 8 up, 7 in 2024-09-17T07:10:45.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:45 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:10:45.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:45 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:10:45.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:45 smithi100 ceph-mon[39462]: osdmap e215: 8 total, 8 up, 7 in 2024-09-17T07:10:45.684 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:10:45.722 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:45.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:45.723 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:10:45.824 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:10:45.857 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:45.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:45.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:10:45.959 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:10:45.996 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:45.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:45.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:10:46.053 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:10:45 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:10:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:10:46.098 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:10:46.133 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:46.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:46.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:10:46.235 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:10:46.267 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:46.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:46.268 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:10:46.344 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:10:46.369 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:10:46.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:46 smithi100 ceph-mon[29092]: pgmap v379: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 4.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.32k op/s; 1989/207639 objects misplaced (0.958%); 11 MiB/s, 169 objects/s recovering 2024-09-17T07:10:46.376 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:46 smithi100 ceph-mon[39462]: pgmap v379: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 4.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.32k op/s; 1989/207639 objects misplaced (0.958%); 11 MiB/s, 169 objects/s recovering 2024-09-17T07:10:46.390 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:46.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:46.391 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:10:46.492 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:10:46.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:46.524 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:46.525 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:10:46.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:46 smithi135 ceph-mon[33238]: pgmap v379: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 4.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.32k op/s; 1989/207639 objects misplaced (0.958%); 11 MiB/s, 169 objects/s recovering 2024-09-17T07:10:46.625 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:10:46.661 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:46.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:46.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:10:46.763 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:10:46.797 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:46.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:46.798 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:10:46.898 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:10:46.932 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:46.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:46.933 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:10:47.034 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:10:47.058 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:47.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:47.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:10:47.161 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:10:47.194 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:47.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:47.195 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:10:47.296 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:10:47.331 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:47.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:47.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:10:47.433 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:10:47.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:47.459 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:47.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:10:47.560 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:10:47.598 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:47.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:47.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:10:47.700 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:10:47.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:47.739 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:47.739 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:10:47.840 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:10:47.867 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:10:47.881 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:47.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:47.882 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:10:47.983 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:10:48.012 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:48.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:48.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:10:48.114 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:10:48.147 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:48.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:48.148 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:10:48.249 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:10:48.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:48.285 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:48.285 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:10:48.387 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:10:48.424 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-17T07:10:48.426 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:48.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:48.427 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:10:48.516 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:48 smithi135 ceph-mon[33238]: pgmap v380: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 4.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 880 op/s; 1989/207639 objects misplaced (0.958%); 4.4 MiB/s, 70 objects/s recovering 2024-09-17T07:10:48.517 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:48 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:10:48.517 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:48 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:10:48.517 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:48 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [7, 1]}]: dispatch 2024-09-17T07:10:48.517 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:48 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 0]}]: dispatch 2024-09-17T07:10:48.517 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:48 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1e", "id": [1, 5]}]: dispatch 2024-09-17T07:10:48.524 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:48 smithi100 ceph-mon[29092]: pgmap v380: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 4.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 880 op/s; 1989/207639 objects misplaced (0.958%); 4.4 MiB/s, 70 objects/s recovering 2024-09-17T07:10:48.525 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:48 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:10:48.525 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:48 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:10:48.525 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:48 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [7, 1]}]: dispatch 2024-09-17T07:10:48.525 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:48 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 0]}]: dispatch 2024-09-17T07:10:48.525 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:48 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1e", "id": [1, 5]}]: dispatch 2024-09-17T07:10:48.525 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:48 smithi100 ceph-mon[39462]: pgmap v380: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 4.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 880 op/s; 1989/207639 objects misplaced (0.958%); 4.4 MiB/s, 70 objects/s recovering 2024-09-17T07:10:48.525 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:48 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:10:48.525 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:48 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:10:48.526 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:48 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [7, 1]}]: dispatch 2024-09-17T07:10:48.526 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:48 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 0]}]: dispatch 2024-09-17T07:10:48.526 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:48 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1e", "id": [1, 5]}]: dispatch 2024-09-17T07:10:48.528 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:10:48.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:48.573 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:48.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:10:48.675 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:10:48.710 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:48.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:48.711 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:10:48.748 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:10:48.793 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:10:48 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:10:48.812 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:10:48.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:48.841 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:48.841 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:10:48.942 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:10:48.969 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:48.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:48.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:10:49.071 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:10:49.118 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:49.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:49.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:10:49.198 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:10:49.220 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:10:49.248 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:49.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:49.249 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:10:49.350 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:10:49.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:49.381 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:49.381 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:10:49.482 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:10:49.504 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:49.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:49.505 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:10:49.606 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:10:49.627 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:49.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:49.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:10:49.664 INFO:teuthology.orchestra.run.smithi100.stderr:nodeep-scrub is set 2024-09-17T07:10:49.666 INFO:teuthology.orchestra.run.smithi100.stderr:reweighted osd.0 to 1 (10000) 2024-09-17T07:10:49.729 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:10:49.768 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:49.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:49.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:10:49.870 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:10:49.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:49 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:10:49.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:49 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [7, 1]}]': finished 2024-09-17T07:10:49.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:49 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 0]}]': finished 2024-09-17T07:10:49.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:49 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1e", "id": [1, 5]}]': finished 2024-09-17T07:10:49.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:49 smithi100 ceph-mon[39462]: osdmap e216: 8 total, 8 up, 7 in 2024-09-17T07:10:49.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:49 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2237612361' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-17T07:10:49.879 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:49 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-17T07:10:49.879 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:49 smithi100 ceph-mon[39462]: pgmap v382: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 3.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.34k op/s; 1282/192915 objects misplaced (0.665%); 8.7 MiB/s, 139 objects/s recovering 2024-09-17T07:10:49.879 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:49 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1019445682' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:10:49.879 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:49 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:10:49.879 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:49 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [7, 1]}]': finished 2024-09-17T07:10:49.879 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:49 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 0]}]': finished 2024-09-17T07:10:49.880 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:49 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1e", "id": [1, 5]}]': finished 2024-09-17T07:10:49.880 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:49 smithi100 ceph-mon[29092]: osdmap e216: 8 total, 8 up, 7 in 2024-09-17T07:10:49.880 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:49 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2237612361' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-17T07:10:49.880 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:49 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-17T07:10:49.880 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:49 smithi100 ceph-mon[29092]: pgmap v382: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 3.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.34k op/s; 1282/192915 objects misplaced (0.665%); 8.7 MiB/s, 139 objects/s recovering 2024-09-17T07:10:49.880 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:49 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1019445682' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:10:49.913 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:49.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:49.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:10:50.015 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:10:50.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:49 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:10:50.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:49 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [7, 1]}]': finished 2024-09-17T07:10:50.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:49 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 0]}]': finished 2024-09-17T07:10:50.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:49 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1e", "id": [1, 5]}]': finished 2024-09-17T07:10:50.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:49 smithi135 ceph-mon[33238]: osdmap e216: 8 total, 8 up, 7 in 2024-09-17T07:10:50.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:49 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2237612361' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-17T07:10:50.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:49 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-17T07:10:50.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:49 smithi135 ceph-mon[33238]: pgmap v382: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 3.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.34k op/s; 1282/192915 objects misplaced (0.665%); 8.7 MiB/s, 139 objects/s recovering 2024-09-17T07:10:50.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:49 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1019445682' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:10:50.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:50.049 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:50.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:10:50.150 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:10:50.185 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:50.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:50.187 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:10:50.288 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:10:50.326 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:50.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:50.327 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:10:50.428 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:10:50.468 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:50.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:50.469 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:10:50.570 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:10:50.591 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:50.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:50.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:10:50.693 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:10:50.726 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:50.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:50.727 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:10:50.828 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:10:50.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:50.857 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:50.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:10:50.865 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:50 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:10:50.865 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:50 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:10:50.865 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:50 smithi100 ceph-mon[29092]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:10:50.865 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:50 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-17T07:10:50.865 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:50 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1019445682' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T07:10:50.865 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:50 smithi100 ceph-mon[29092]: osdmap e217: 8 total, 8 up, 7 in 2024-09-17T07:10:50.865 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:50 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:10:50.866 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:50 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:10:50.866 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:50 smithi100 ceph-mon[39462]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:10:50.866 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:50 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-17T07:10:50.866 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:50 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1019445682' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T07:10:50.866 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:50 smithi100 ceph-mon[39462]: osdmap e217: 8 total, 8 up, 7 in 2024-09-17T07:10:50.910 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:50 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:10:50.910 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:50 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:10:50.910 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:50 smithi135 ceph-mon[33238]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:10:50.910 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:50 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-17T07:10:50.910 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:50 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1019445682' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T07:10:50.910 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:50 smithi135 ceph-mon[33238]: osdmap e217: 8 total, 8 up, 7 in 2024-09-17T07:10:50.959 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:10:50.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:50.994 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:50.994 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:10:51.095 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:10:51.128 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:51.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:51.130 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:10:51.231 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:10:51.263 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:51.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:51.264 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:10:51.365 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:10:51.398 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:51.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:51.399 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:10:51.499 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:10:51.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:51.531 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:51.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:10:51.632 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:10:51.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:51.668 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:51.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:10:51.769 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:10:51.814 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:51.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:51.815 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:10:51.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:51 smithi100 ceph-mon[29092]: pgmap v384: 148 pgs: 1 peering, 1 active+remapped+backfilling, 146 active+clean; 3.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 737 B/s rd, 0 B/s wr, 872 op/s; 1282/189477 objects misplaced (0.677%); 5.0 MiB/s, 79 objects/s recovering 2024-09-17T07:10:51.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:51 smithi100 ceph-mon[39462]: pgmap v384: 148 pgs: 1 peering, 1 active+remapped+backfilling, 146 active+clean; 3.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 737 B/s rd, 0 B/s wr, 872 op/s; 1282/189477 objects misplaced (0.677%); 5.0 MiB/s, 79 objects/s recovering 2024-09-17T07:10:51.916 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:10:51.945 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:51.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:51.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:10:51.969 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:51 smithi135 ceph-mon[33238]: pgmap v384: 148 pgs: 1 peering, 1 active+remapped+backfilling, 146 active+clean; 3.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 737 B/s rd, 0 B/s wr, 872 op/s; 1282/189477 objects misplaced (0.677%); 5.0 MiB/s, 79 objects/s recovering 2024-09-17T07:10:52.047 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:10:52.087 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:52.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:52.088 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:10:52.189 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:10:52.225 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:52.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:52.226 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:10:52.326 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:10:52.361 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:52.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:52.362 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:10:52.463 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:10:52.505 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:52.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:52.506 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:10:52.607 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:10:52.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:52.655 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:52.655 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:10:52.756 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:10:52.791 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:52.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:52.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:10:52.893 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:10:52.920 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:52.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:52.921 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:10:52.946 INFO:teuthology.orchestra.run.smithi135.stdout:ERROR: (22) Invalid argument 2024-09-17T07:10:52.947 INFO:teuthology.orchestra.run.smithi135.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:10:53.022 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:10:53.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:53.576 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:53.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:10:53.577 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd reweight 1 1 2024-09-17T07:10:53.634 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-17T07:10:54.368 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T07:10:54.369 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:10:54.370 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:10:54.370 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:10:54.370 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1: {} 2024-09-17T07:10:54.371 INFO:teuthology.orchestra.run.smithi100.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:10:54.371 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:10:54.371 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:10:54.372 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:10:54.372 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:10:54.372 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:10:54.373 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:10:54.373 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:10:54.373 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:10:54.373 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:10:54.374 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:10:54.374 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:10:54.374 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:10:54.375 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:10:54.375 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:54 smithi100 ceph-mon[29092]: pgmap v385: 148 pgs: 1 peering, 1 active+remapped+backfilling, 146 active+clean; 3.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 757 op/s; 1282/189477 objects misplaced (0.677%); 4.3 MiB/s, 69 objects/s recovering 2024-09-17T07:10:54.375 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:54 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/3074855320' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:10:54.375 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:54 smithi100 ceph-mon[39462]: pgmap v385: 148 pgs: 1 peering, 1 active+remapped+backfilling, 146 active+clean; 3.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 757 op/s; 1282/189477 objects misplaced (0.677%); 4.3 MiB/s, 69 objects/s recovering 2024-09-17T07:10:54.376 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:54 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/3074855320' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:10:54.405 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:54.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:54.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:10:54.507 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:10:54.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:54 smithi135 ceph-mon[33238]: pgmap v385: 148 pgs: 1 peering, 1 active+remapped+backfilling, 146 active+clean; 3.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 757 op/s; 1282/189477 objects misplaced (0.677%); 4.3 MiB/s, 69 objects/s recovering 2024-09-17T07:10:54.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:54 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/3074855320' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:10:54.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:54.548 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:54.548 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:10:54.649 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:10:54.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:54.690 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:54.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:10:54.716 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:10:54.792 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:10:54.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:54.829 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:54.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:10:54.930 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:10:54.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:54.970 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:54.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:10:55.071 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:10:55.108 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:55.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:55.109 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:10:55.210 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:10:55.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:55.238 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:55.238 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:10:55.339 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:10:55.377 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:55.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:55.378 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:10:55.478 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:10:55.513 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:55.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:55.514 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:10:55.615 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:10:55.655 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:55.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:55.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:10:55.757 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:10:55.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:55.786 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:55.787 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:10:55.887 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:10:55.931 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:55.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:55.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:10:56.033 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:10:56.095 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:56.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:56.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:10:56.157 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:10:55 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:10:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:10:56.196 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:10:56.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:56.231 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:56.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:10:56.332 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:10:56.365 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:56.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:56.366 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:10:56.440 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:56 smithi135 ceph-mon[33238]: pgmap v386: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 3.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.51k op/s; 634/171510 objects misplaced (0.370%); 8.7 MiB/s, 140 objects/s recovering 2024-09-17T07:10:56.466 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:56 smithi100 ceph-mon[29092]: pgmap v386: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 3.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.51k op/s; 634/171510 objects misplaced (0.370%); 8.7 MiB/s, 140 objects/s recovering 2024-09-17T07:10:56.466 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:56 smithi100 ceph-mon[39462]: pgmap v386: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 3.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.51k op/s; 634/171510 objects misplaced (0.370%); 8.7 MiB/s, 140 objects/s recovering 2024-09-17T07:10:56.467 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:10:56.473 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:10:56.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:56.508 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:56.508 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:10:56.602 INFO:teuthology.orchestra.run.smithi135.stdout:{ 2024-09-17T07:10:56.602 INFO:teuthology.orchestra.run.smithi135.stdout: "ops": [ 2024-09-17T07:10:56.602 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:56.602 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_repop(client.24929.0:208098 8.5 e217/168 8:ab27d806:::benchmark_data_smithi100_102803_object77594:head v 217'13223, mlcod=217'13219)", 2024-09-17T07:10:56.602 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:10:56.597880+0000", 2024-09-17T07:10:56.602 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 0.002984787, 2024-09-17T07:10:56.602 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0030156950000000001, 2024-09-17T07:10:56.602 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:10:56.602 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "started", 2024-09-17T07:10:56.603 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:10:56.603 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:56.603 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:10:56.603 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:56.597880+0000", 2024-09-17T07:10:56.603 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:10:56.603 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:56.603 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:56.603 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:10:56.603 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:56.597880+0000", 2024-09-17T07:10:56.603 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:10:56.603 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:56.603 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:56.603 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:10:56.604 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:56.597879+0000", 2024-09-17T07:10:56.604 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.999999 2024-09-17T07:10:56.604 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:56.604 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:56.604 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:10:56.604 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:56.597887+0000", 2024-09-17T07:10:56.604 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.0169999999999996e-06 2024-09-17T07:10:56.604 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:56.604 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:56.604 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:10:56.604 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:56.597903+0000", 2024-09-17T07:10:56.604 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.5985999999999999e-05 2024-09-17T07:10:56.604 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:56.605 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:56.605 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:10:56.605 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:56.597912+0000", 2024-09-17T07:10:56.605 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.5129999999999996e-06 2024-09-17T07:10:56.605 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:56.605 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:56.605 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:10:56.605 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:56.598069+0000", 2024-09-17T07:10:56.605 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00015773600000000001 2024-09-17T07:10:56.605 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:10:56.605 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:10:56.605 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:10:56.605 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:56.598091+0000", 2024-09-17T07:10:56.606 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.1790999999999999e-05 2024-09-17T07:10:56.606 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:10:56.606 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:10:56.606 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:10:56.606 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:10:56.606 INFO:teuthology.orchestra.run.smithi135.stdout: ], 2024-09-17T07:10:56.606 INFO:teuthology.orchestra.run.smithi135.stdout: "num_ops": 1 2024-09-17T07:10:56.606 INFO:teuthology.orchestra.run.smithi135.stdout:} 2024-09-17T07:10:56.608 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:10:56.638 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:56.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:56.640 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:10:56.740 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:10:56.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:56.784 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:56.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:10:56.885 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:10:56.917 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:56.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:56.918 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:10:57.019 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:10:57.052 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:57.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:57.053 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:10:57.154 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:10:57.167 INFO:teuthology.orchestra.run.smithi100.stderr:reweighted osd.1 to 1 (10000) 2024-09-17T07:10:57.191 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:57.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:57.192 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:10:57.292 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:10:57.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:57 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:10:57.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:57 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:10:57.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:57 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/3426904577' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-17T07:10:57.325 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:57.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:57.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:10:57.365 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:57 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:10:57.365 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:57 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:10:57.365 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:57 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/3426904577' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-17T07:10:57.366 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:57 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:10:57.366 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:57 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:10:57.366 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:57 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/3426904577' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-17T07:10:57.427 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:10:57.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:57.461 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:57.461 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:10:57.562 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:10:57.586 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-17T07:10:57.591 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:57.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:57.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:10:57.692 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:10:57.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:57.725 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:57.725 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:10:57.826 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:10:57.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:57.889 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:57.889 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:10:57.940 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:10:57.951 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:10:57.989 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:10:58.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:58.026 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:58.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:10:58.127 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:10:58.156 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:58.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:58.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:10:58.223 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd reweight 2 1 2024-09-17T07:10:58.256 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:58 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/3426904577' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-17T07:10:58.256 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:58 smithi100 ceph-mon[29092]: osdmap e218: 8 total, 8 up, 7 in 2024-09-17T07:10:58.256 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:10:58 smithi100 ceph-mon[29092]: pgmap v388: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 3.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 892 op/s; 634/171510 objects misplaced (0.370%); 4.4 MiB/s, 71 objects/s recovering 2024-09-17T07:10:58.256 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:58 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/3426904577' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-17T07:10:58.256 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:58 smithi100 ceph-mon[39462]: osdmap e218: 8 total, 8 up, 7 in 2024-09-17T07:10:58.256 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:10:58 smithi100 ceph-mon[39462]: pgmap v388: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 3.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 892 op/s; 634/171510 objects misplaced (0.370%); 4.4 MiB/s, 71 objects/s recovering 2024-09-17T07:10:58.258 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:10:58.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:58.310 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:58.310 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:10:58.353 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:58 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/3426904577' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-17T07:10:58.353 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:58 smithi135 ceph-mon[33238]: osdmap e218: 8 total, 8 up, 7 in 2024-09-17T07:10:58.353 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:10:58 smithi135 ceph-mon[33238]: pgmap v388: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 3.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 892 op/s; 634/171510 objects misplaced (0.370%); 4.4 MiB/s, 71 objects/s recovering 2024-09-17T07:10:58.411 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:10:58.444 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:58.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:58.445 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:10:58.546 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:10:58.578 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:58.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:58.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:10:58.680 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:10:58.709 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:58.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:58.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:10:58.749 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:10:58.787 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:10:58.793 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:10:58 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:10:58.811 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:10:58.855 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:58.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:58.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:10:58.957 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:10:58.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:58.994 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:58.994 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:10:59.095 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:10:59.152 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:59.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:59.153 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:10:59.254 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:10:59.281 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:59.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:59.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:10:59.383 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:10:59.407 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:59.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:59.408 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:10:59.509 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:10:59.530 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:59.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:59.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:10:59.632 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:10:59.654 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:10:59.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:59.655 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:10:59.756 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:10:59.777 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:59.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:59.778 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:10:59.790 INFO:teuthology.orchestra.run.smithi135.stdout:{ 2024-09-17T07:10:59.790 INFO:teuthology.orchestra.run.smithi135.stdout: "ops": [], 2024-09-17T07:10:59.790 INFO:teuthology.orchestra.run.smithi135.stdout: "complaint_time": 30, 2024-09-17T07:10:59.790 INFO:teuthology.orchestra.run.smithi135.stdout: "num_blocked_ops": 0 2024-09-17T07:10:59.790 INFO:teuthology.orchestra.run.smithi135.stdout:} 2024-09-17T07:10:59.878 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:10:59.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:10:59.900 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:10:59.900 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:11:00.002 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:11:00.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:00.023 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:00.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:11:00.124 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:11:00.146 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:00.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:00.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:11:00.196 INFO:teuthology.orchestra.run.smithi100.stderr:noscrub is unset 2024-09-17T07:11:00.248 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:11:00.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:00.287 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:00.287 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:11:00.388 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:11:00.419 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:00.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:00.420 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:11:00.428 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:00 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:11:00.428 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:00 smithi135 ceph-mon[33238]: pgmap v390: 148 pgs: 1 active+clean+remapped, 147 active+clean; 3.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.41k op/s; 9.0 MiB/s, 145 objects/s recovering 2024-09-17T07:11:00.428 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:00 smithi135 ceph-mon[33238]: osdmap e219: 8 total, 8 up, 7 in 2024-09-17T07:11:00.428 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:00 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1678012439' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:11:00.428 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:00 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:11:00.444 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:00 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:11:00.444 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:00 smithi100 ceph-mon[29092]: pgmap v390: 148 pgs: 1 active+clean+remapped, 147 active+clean; 3.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.41k op/s; 9.0 MiB/s, 145 objects/s recovering 2024-09-17T07:11:00.444 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:00 smithi100 ceph-mon[29092]: osdmap e219: 8 total, 8 up, 7 in 2024-09-17T07:11:00.445 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:00 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1678012439' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:11:00.445 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:00 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:11:00.445 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:00 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:11:00.445 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:00 smithi100 ceph-mon[39462]: pgmap v390: 148 pgs: 1 active+clean+remapped, 147 active+clean; 3.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.41k op/s; 9.0 MiB/s, 145 objects/s recovering 2024-09-17T07:11:00.445 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:00 smithi100 ceph-mon[39462]: osdmap e219: 8 total, 8 up, 7 in 2024-09-17T07:11:00.445 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:00 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1678012439' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:11:00.445 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:00 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:11:00.467 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-17T07:11:00.521 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:11:00.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:00.556 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:00.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:11:00.657 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:11:00.705 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:00.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:00.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:11:00.807 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:11:00.808 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:11:00.835 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:00.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:00.836 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:11:00.937 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:11:00.967 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:00.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:00.968 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:11:01.069 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:11:01.100 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:01.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:01.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:11:01.202 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:11:01.238 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:01.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:01.239 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:11:01.240 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:01 smithi135 ceph-mon[33238]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:11:01.240 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:01 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T07:11:01.240 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:01 smithi135 ceph-mon[33238]: osdmap e220: 8 total, 8 up, 7 in 2024-09-17T07:11:01.240 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:01 smithi135 ceph-mon[33238]: 5.c scrub starts 2024-09-17T07:11:01.240 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:01 smithi135 ceph-mon[33238]: 5.c scrub ok 2024-09-17T07:11:01.240 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:01 smithi135 ceph-mon[33238]: 6.1f scrub starts 2024-09-17T07:11:01.240 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:01 smithi135 ceph-mon[33238]: 6.1f scrub ok 2024-09-17T07:11:01.240 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:01 smithi135 ceph-mon[33238]: 2.1b scrub starts 2024-09-17T07:11:01.240 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:01 smithi135 ceph-mon[33238]: 2.1b scrub ok 2024-09-17T07:11:01.340 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:11:01.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:01.376 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:01.376 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:11:01.446 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:01 smithi100 ceph-mon[29092]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:11:01.446 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:01 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T07:11:01.446 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:01 smithi100 ceph-mon[29092]: osdmap e220: 8 total, 8 up, 7 in 2024-09-17T07:11:01.447 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:01 smithi100 ceph-mon[29092]: 5.c scrub starts 2024-09-17T07:11:01.447 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:01 smithi100 ceph-mon[29092]: 5.c scrub ok 2024-09-17T07:11:01.447 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:01 smithi100 ceph-mon[29092]: 6.1f scrub starts 2024-09-17T07:11:01.447 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:01 smithi100 ceph-mon[29092]: 6.1f scrub ok 2024-09-17T07:11:01.447 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:01 smithi100 ceph-mon[29092]: 2.1b scrub starts 2024-09-17T07:11:01.447 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:01 smithi100 ceph-mon[29092]: 2.1b scrub ok 2024-09-17T07:11:01.447 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:01 smithi100 ceph-mon[39462]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:11:01.448 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:01 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T07:11:01.448 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:01 smithi100 ceph-mon[39462]: osdmap e220: 8 total, 8 up, 7 in 2024-09-17T07:11:01.448 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:01 smithi100 ceph-mon[39462]: 5.c scrub starts 2024-09-17T07:11:01.448 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:01 smithi100 ceph-mon[39462]: 5.c scrub ok 2024-09-17T07:11:01.448 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:01 smithi100 ceph-mon[39462]: 6.1f scrub starts 2024-09-17T07:11:01.448 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:01 smithi100 ceph-mon[39462]: 6.1f scrub ok 2024-09-17T07:11:01.448 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:01 smithi100 ceph-mon[39462]: 2.1b scrub starts 2024-09-17T07:11:01.448 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:01 smithi100 ceph-mon[39462]: 2.1b scrub ok 2024-09-17T07:11:01.477 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:11:01.510 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:01.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:01.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:11:01.613 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:11:01.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:01.642 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:01.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:11:01.743 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:11:01.779 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:01.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:01.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:11:01.881 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:11:01.912 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:01.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:01.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:11:02.014 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:11:02.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:02.050 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:02.050 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:11:02.151 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:11:02.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:02.194 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:02.194 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:11:02.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:02 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:11:02.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:02 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:11:02.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:02 smithi135 ceph-mon[33238]: pgmap v392: 148 pgs: 1 active+clean+remapped, 147 active+clean; 3.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.09k op/s; 6.2 MiB/s, 98 objects/s recovering 2024-09-17T07:11:02.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:02 smithi135 ceph-mon[33238]: osdmap e221: 8 total, 8 up, 7 in 2024-09-17T07:11:02.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:02 smithi135 ceph-mon[33238]: 2.c scrub starts 2024-09-17T07:11:02.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:02 smithi135 ceph-mon[33238]: 2.c scrub ok 2024-09-17T07:11:02.295 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:11:02.323 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:02.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:02.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:11:02.426 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:11:02.451 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:02.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:02.452 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:11:02.536 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:02 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:11:02.536 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:02 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:11:02.537 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:02 smithi100 ceph-mon[29092]: pgmap v392: 148 pgs: 1 active+clean+remapped, 147 active+clean; 3.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.09k op/s; 6.2 MiB/s, 98 objects/s recovering 2024-09-17T07:11:02.537 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:02 smithi100 ceph-mon[29092]: osdmap e221: 8 total, 8 up, 7 in 2024-09-17T07:11:02.537 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:02 smithi100 ceph-mon[29092]: 2.c scrub starts 2024-09-17T07:11:02.537 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:02 smithi100 ceph-mon[29092]: 2.c scrub ok 2024-09-17T07:11:02.537 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:02 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:11:02.537 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:02 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:11:02.537 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:02 smithi100 ceph-mon[39462]: pgmap v392: 148 pgs: 1 active+clean+remapped, 147 active+clean; 3.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.09k op/s; 6.2 MiB/s, 98 objects/s recovering 2024-09-17T07:11:02.537 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:02 smithi100 ceph-mon[39462]: osdmap e221: 8 total, 8 up, 7 in 2024-09-17T07:11:02.538 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:02 smithi100 ceph-mon[39462]: 2.c scrub starts 2024-09-17T07:11:02.538 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:02 smithi100 ceph-mon[39462]: 2.c scrub ok 2024-09-17T07:11:02.553 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:11:02.587 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:02.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:02.588 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:11:02.689 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:11:02.727 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:02.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:02.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:11:02.825 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:11:02.826 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:11:02.829 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:11:02.853 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1: {} 2024-09-17T07:11:02.853 INFO:teuthology.orchestra.run.smithi100.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:11:02.865 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:02.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:02.866 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:11:02.882 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:11:02.882 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:11:02.907 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:11:02.907 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:11:02.942 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:11:02.942 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:11:02.967 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:11:02.977 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:11:02.977 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:11:02.999 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:02.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:03.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:11:03.001 INFO:teuthology.orchestra.run.smithi135.stdout:{ 2024-09-17T07:11:03.001 INFO:teuthology.orchestra.run.smithi135.stdout: "size": 20, 2024-09-17T07:11:03.001 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 600, 2024-09-17T07:11:03.001 INFO:teuthology.orchestra.run.smithi135.stdout: "ops": [ 2024-09-17T07:11:03.001 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.001 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:54898 7.5 7:a270837f:::benchmark_data_smithi100_78556_object54897:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e111)", 2024-09-17T07:11:03.001 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:04:57.340083+0000", 2024-09-17T07:11:03.001 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 365.65667717299999, 2024-09-17T07:11:03.001 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0127802889999999, 2024-09-17T07:11:03.002 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:11:03.002 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "reached pg", 2024-09-17T07:11:03.002 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:11:03.002 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:11:03.002 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:11:03.002 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 54898 2024-09-17T07:11:03.002 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.002 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:11:03.002 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.002 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:11:03.002 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:57.340083+0000", 2024-09-17T07:11:03.003 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:11:03.003 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.003 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.003 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:11:03.003 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:57.340083+0000", 2024-09-17T07:11:03.003 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:11:03.003 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.003 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.003 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:11:03.003 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:57.340081+0000", 2024-09-17T07:11:03.004 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999976 2024-09-17T07:11:03.004 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.004 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.004 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:11:03.004 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:57.340142+0000", 2024-09-17T07:11:03.004 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.1171000000000001e-05 2024-09-17T07:11:03.004 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.004 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.004 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:11:03.004 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:57.340169+0000", 2024-09-17T07:11:03.005 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.7288e-05 2024-09-17T07:11:03.005 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.005 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.005 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:11:03.005 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:57.340181+0000", 2024-09-17T07:11:03.005 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.1498000000000001e-05 2024-09-17T07:11:03.005 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.005 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.005 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:11:03.005 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:57.340249+0000", 2024-09-17T07:11:03.006 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.8008000000000003e-05 2024-09-17T07:11:03.006 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.006 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.006 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op must wait for map", 2024-09-17T07:11:03.006 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:57.340257+0000", 2024-09-17T07:11:03.006 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.4509999999999992e-06 2024-09-17T07:11:03.006 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.006 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.006 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:11:03.006 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:57.434348+0000", 2024-09-17T07:11:03.007 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.094091286999999996 2024-09-17T07:11:03.007 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.007 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.007 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:11:03.007 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:57.434358+0000", 2024-09-17T07:11:03.007 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.7599999999999997e-06 2024-09-17T07:11:03.007 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.007 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.007 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:11:03.007 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:58.352774+0000", 2024-09-17T07:11:03.007 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.91841545300000005 2024-09-17T07:11:03.007 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.008 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.008 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:11:03.008 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:58.352863+0000", 2024-09-17T07:11:03.008 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.9732000000000001e-05 2024-09-17T07:11:03.008 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:11:03.008 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:11:03.008 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:11:03.008 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.008 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.008 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:54904 7.18 7:188b301f:::benchmark_data_smithi100_78556_object54903:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-09-17T07:11:03.009 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:04:57.344410+0000", 2024-09-17T07:11:03.009 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 365.65235080799999, 2024-09-17T07:11:03.009 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.004638776, 2024-09-17T07:11:03.009 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:11:03.009 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:11:03.009 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:11:03.009 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:11:03.009 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:11:03.010 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 54904 2024-09-17T07:11:03.010 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.010 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:11:03.010 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.010 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:11:03.010 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:57.344410+0000", 2024-09-17T07:11:03.010 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:11:03.010 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.010 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.010 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:11:03.010 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:57.344410+0000", 2024-09-17T07:11:03.011 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:11:03.011 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.011 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.011 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:11:03.011 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:57.344407+0000", 2024-09-17T07:11:03.011 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999976 2024-09-17T07:11:03.011 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.011 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.011 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:11:03.011 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:57.344445+0000", 2024-09-17T07:11:03.012 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.7654000000000002e-05 2024-09-17T07:11:03.012 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.012 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.012 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:11:03.012 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:57.344460+0000", 2024-09-17T07:11:03.012 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.4781e-05 2024-09-17T07:11:03.012 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.012 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.012 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:11:03.012 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:57.344469+0000", 2024-09-17T07:11:03.012 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.4560000000000006e-06 2024-09-17T07:11:03.013 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.013 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.013 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:11:03.013 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:57.434942+0000", 2024-09-17T07:11:03.013 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.090472616000000006 2024-09-17T07:11:03.013 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.013 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.013 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:11:03.013 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:57.434950+0000", 2024-09-17T07:11:03.013 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.943e-06 2024-09-17T07:11:03.014 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.014 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.014 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:11:03.014 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:58.345420+0000", 2024-09-17T07:11:03.014 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.91047032299999997 2024-09-17T07:11:03.014 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.014 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.014 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:11:03.014 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:58.345549+0000", 2024-09-17T07:11:03.014 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00012873100000000001 2024-09-17T07:11:03.014 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.014 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.014 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 2,4", 2024-09-17T07:11:03.015 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:58.345685+0000", 2024-09-17T07:11:03.015 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000136656 2024-09-17T07:11:03.015 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.015 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.015 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:11:03.015 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:58.346784+0000", 2024-09-17T07:11:03.015 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0010982120000000001 2024-09-17T07:11:03.015 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.015 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.015 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:11:03.015 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:58.347480+0000", 2024-09-17T07:11:03.015 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00069644399999999995 2024-09-17T07:11:03.016 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.016 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.016 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:11:03.016 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:58.348950+0000", 2024-09-17T07:11:03.016 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.001470341 2024-09-17T07:11:03.016 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.016 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.016 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:11:03.016 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:58.348987+0000", 2024-09-17T07:11:03.016 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.6358e-05 2024-09-17T07:11:03.016 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.016 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.016 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:11:03.017 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:58.349048+0000", 2024-09-17T07:11:03.017 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.1471999999999997e-05 2024-09-17T07:11:03.017 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:11:03.017 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:11:03.017 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:11:03.017 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.017 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.017 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:54906 7.8 7:14bcdeaa:::benchmark_data_smithi100_78556_object54905:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-09-17T07:11:03.017 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:04:57.344533+0000", 2024-09-17T07:11:03.017 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 365.65222695800003, 2024-09-17T07:11:03.017 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.004326727, 2024-09-17T07:11:03.017 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:11:03.017 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:11:03.018 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:11:03.018 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:11:03.018 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:11:03.018 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 54906 2024-09-17T07:11:03.018 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.018 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:11:03.018 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.018 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:11:03.018 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:57.344533+0000", 2024-09-17T07:11:03.018 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:11:03.018 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.018 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.019 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:11:03.019 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:57.344533+0000", 2024-09-17T07:11:03.019 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:11:03.019 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.019 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.019 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:11:03.019 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:57.344533+0000", 2024-09-17T07:11:03.019 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.999999 2024-09-17T07:11:03.019 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.019 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.019 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:11:03.019 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:57.344588+0000", 2024-09-17T07:11:03.019 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.5139000000000002e-05 2024-09-17T07:11:03.020 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.020 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.020 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:11:03.020 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:57.344596+0000", 2024-09-17T07:11:03.020 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.8760000000000005e-06 2024-09-17T07:11:03.020 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.020 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.020 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:11:03.020 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:57.344602+0000", 2024-09-17T07:11:03.020 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.4999999999999996e-06 2024-09-17T07:11:03.020 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.020 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.021 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:11:03.021 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:57.432211+0000", 2024-09-17T07:11:03.021 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.087608723999999999 2024-09-17T07:11:03.021 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.021 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.021 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:11:03.021 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:57.432228+0000", 2024-09-17T07:11:03.021 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.6679000000000001e-05 2024-09-17T07:11:03.021 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.021 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.021 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:11:03.021 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:58.345376+0000", 2024-09-17T07:11:03.021 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.91314849200000003 2024-09-17T07:11:03.022 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.022 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.022 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:11:03.022 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:58.345502+0000", 2024-09-17T07:11:03.022 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00012606299999999999 2024-09-17T07:11:03.022 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.022 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.022 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 2,4", 2024-09-17T07:11:03.022 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:58.345646+0000", 2024-09-17T07:11:03.023 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00014381299999999999 2024-09-17T07:11:03.023 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.023 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.023 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:11:03.023 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:58.346790+0000", 2024-09-17T07:11:03.023 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0011440879999999999 2024-09-17T07:11:03.023 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.023 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.023 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:11:03.023 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:58.347629+0000", 2024-09-17T07:11:03.023 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00083862699999999999 2024-09-17T07:11:03.023 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.023 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.024 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:11:03.024 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:58.348755+0000", 2024-09-17T07:11:03.024 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0011261330000000001 2024-09-17T07:11:03.024 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.024 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.024 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:11:03.024 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:58.348821+0000", 2024-09-17T07:11:03.024 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.6103000000000001e-05 2024-09-17T07:11:03.024 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.024 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.024 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:11:03.024 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:58.348860+0000", 2024-09-17T07:11:03.024 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.9211000000000001e-05 2024-09-17T07:11:03.025 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:11:03.025 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:11:03.025 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:11:03.025 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.025 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.025 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:77574 7.11 7:8e8a8d68:::benchmark_data_smithi100_78556_object77573:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e122)", 2024-09-17T07:11:03.025 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:05:22.263616+0000", 2024-09-17T07:11:03.025 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 340.733144364, 2024-09-17T07:11:03.025 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0084467050000001, 2024-09-17T07:11:03.025 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:11:03.025 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "reached pg", 2024-09-17T07:11:03.025 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:11:03.026 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:11:03.026 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:11:03.026 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 77574 2024-09-17T07:11:03.026 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.026 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:11:03.026 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.026 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:11:03.026 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:22.263616+0000", 2024-09-17T07:11:03.026 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:11:03.026 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.026 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.026 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:11:03.026 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:22.263616+0000", 2024-09-17T07:11:03.027 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:11:03.027 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.027 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.027 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:11:03.027 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:22.263614+0000", 2024-09-17T07:11:03.027 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999976 2024-09-17T07:11:03.027 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.027 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.027 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:11:03.027 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:22.264165+0000", 2024-09-17T07:11:03.027 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00055141399999999996 2024-09-17T07:11:03.027 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.027 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.028 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:11:03.028 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:22.264178+0000", 2024-09-17T07:11:03.028 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.2911000000000001e-05 2024-09-17T07:11:03.028 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.028 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.028 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:11:03.028 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:22.264186+0000", 2024-09-17T07:11:03.028 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.0129999999999993e-06 2024-09-17T07:11:03.028 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.028 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.028 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:11:03.028 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:22.264230+0000", 2024-09-17T07:11:03.028 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.4564999999999998e-05 2024-09-17T07:11:03.029 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.029 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.029 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:11:03.029 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:22.264239+0000", 2024-09-17T07:11:03.029 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.5080000000000005e-06 2024-09-17T07:11:03.029 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.029 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.029 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:11:03.029 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.268253+0000", 2024-09-17T07:11:03.029 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0040142169999999 2024-09-17T07:11:03.029 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.029 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.030 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for missing object", 2024-09-17T07:11:03.030 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.268370+0000", 2024-09-17T07:11:03.030 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000116482 2024-09-17T07:11:03.030 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.030 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.030 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:11:03.030 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.271987+0000", 2024-09-17T07:11:03.030 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.003617744 2024-09-17T07:11:03.030 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.030 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.030 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:11:03.030 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.272063+0000", 2024-09-17T07:11:03.030 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.5209000000000004e-05 2024-09-17T07:11:03.031 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:11:03.031 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:11:03.031 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:11:03.031 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.031 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.031 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:77630 7.11 7:88aca6ac:::benchmark_data_smithi100_78556_object77629:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-09-17T07:11:03.031 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:05:22.343377+0000", 2024-09-17T07:11:03.031 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 340.65338335899997, 2024-09-17T07:11:03.031 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.93764683900000001, 2024-09-17T07:11:03.031 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:11:03.031 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:11:03.031 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:11:03.032 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:11:03.032 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:11:03.032 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 77630 2024-09-17T07:11:03.032 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.032 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:11:03.032 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.032 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:11:03.032 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:22.343377+0000", 2024-09-17T07:11:03.032 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:11:03.032 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.032 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.032 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:11:03.033 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:22.343377+0000", 2024-09-17T07:11:03.033 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:11:03.033 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.033 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.033 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:11:03.033 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:22.343375+0000", 2024-09-17T07:11:03.034 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:11:03.034 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:11:03.035 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:11:03.035 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.035 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.035 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:11:03.035 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:22.343436+0000", 2024-09-17T07:11:03.035 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.0801999999999998e-05 2024-09-17T07:11:03.035 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.035 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.035 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:11:03.035 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:22.343447+0000", 2024-09-17T07:11:03.035 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0499000000000001e-05 2024-09-17T07:11:03.036 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.036 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.036 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:11:03.036 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:22.343452+0000", 2024-09-17T07:11:03.036 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.6210000000000001e-06 2024-09-17T07:11:03.036 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.036 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.036 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:11:03.036 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:22.343508+0000", 2024-09-17T07:11:03.036 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.6059e-05 2024-09-17T07:11:03.036 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.036 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.037 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:11:03.037 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:22.343520+0000", 2024-09-17T07:11:03.037 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.1674999999999999e-05 2024-09-17T07:11:03.037 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.037 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.037 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:11:03.037 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.268676+0000", 2024-09-17T07:11:03.037 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.92515617299999997 2024-09-17T07:11:03.037 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.037 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.037 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:11:03.037 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.268770+0000", 2024-09-17T07:11:03.037 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.3239000000000006e-05 2024-09-17T07:11:03.038 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.038 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.038 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 3,6", 2024-09-17T07:11:03.038 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.268875+0000", 2024-09-17T07:11:03.038 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000105633 2024-09-17T07:11:03.038 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.038 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.038 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:11:03.038 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.272164+0000", 2024-09-17T07:11:03.038 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0032888100000000001 2024-09-17T07:11:03.038 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.038 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.039 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:11:03.039 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.272238+0000", 2024-09-17T07:11:03.039 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.3727999999999996e-05 2024-09-17T07:11:03.039 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.039 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.039 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:11:03.039 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.280903+0000", 2024-09-17T07:11:03.039 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0086651569999999997 2024-09-17T07:11:03.039 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.039 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.039 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:11:03.039 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.280934+0000", 2024-09-17T07:11:03.039 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.1238999999999997e-05 2024-09-17T07:11:03.040 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.040 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.040 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:11:03.040 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.281024+0000", 2024-09-17T07:11:03.040 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.9676e-05 2024-09-17T07:11:03.040 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:11:03.040 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:11:03.040 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:11:03.040 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.040 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.040 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:77650 7.11 7:8c8c0ca2:::benchmark_data_smithi100_78556_object77649:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-09-17T07:11:03.040 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:05:22.374462+0000", 2024-09-17T07:11:03.040 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 340.62229814300002, 2024-09-17T07:11:03.041 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.906699531, 2024-09-17T07:11:03.041 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:11:03.041 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:11:03.041 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:11:03.041 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:11:03.041 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:11:03.041 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 77650 2024-09-17T07:11:03.041 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.041 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:11:03.041 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.041 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:11:03.041 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:22.374462+0000", 2024-09-17T07:11:03.041 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:11:03.042 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.042 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.042 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:11:03.042 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:22.374462+0000", 2024-09-17T07:11:03.042 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:11:03.042 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.042 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.042 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:11:03.042 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:22.374461+0000", 2024-09-17T07:11:03.042 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.999999 2024-09-17T07:11:03.042 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.042 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.042 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:11:03.043 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:22.374607+0000", 2024-09-17T07:11:03.043 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00014552400000000001 2024-09-17T07:11:03.043 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.043 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.043 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:11:03.043 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:22.374615+0000", 2024-09-17T07:11:03.043 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.6950000000000006e-06 2024-09-17T07:11:03.043 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.043 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.043 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:11:03.043 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:22.374620+0000", 2024-09-17T07:11:03.043 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.8629999999999999e-06 2024-09-17T07:11:03.044 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.044 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.044 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:11:03.044 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:22.374667+0000", 2024-09-17T07:11:03.044 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.6422999999999999e-05 2024-09-17T07:11:03.044 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.044 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.044 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:11:03.044 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:22.374679+0000", 2024-09-17T07:11:03.044 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.2544999999999999e-05 2024-09-17T07:11:03.044 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.044 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.044 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:11:03.045 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.270028+0000", 2024-09-17T07:11:03.045 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.89534863600000003 2024-09-17T07:11:03.045 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.045 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.045 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:11:03.045 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.270112+0000", 2024-09-17T07:11:03.045 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.4559000000000001e-05 2024-09-17T07:11:03.045 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.045 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.045 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 3,6", 2024-09-17T07:11:03.045 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.270210+0000", 2024-09-17T07:11:03.045 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.7913999999999998e-05 2024-09-17T07:11:03.045 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.046 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.046 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:11:03.046 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.272170+0000", 2024-09-17T07:11:03.046 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0019594289999999999 2024-09-17T07:11:03.046 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.046 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.046 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:11:03.046 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.272291+0000", 2024-09-17T07:11:03.046 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000121678 2024-09-17T07:11:03.046 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.046 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.046 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:11:03.046 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.281061+0000", 2024-09-17T07:11:03.046 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0087690979999999995 2024-09-17T07:11:03.047 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.047 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.047 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:11:03.047 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.281088+0000", 2024-09-17T07:11:03.047 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.7861999999999999e-05 2024-09-17T07:11:03.047 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.047 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.047 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:11:03.047 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.281162+0000", 2024-09-17T07:11:03.047 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.3267000000000002e-05 2024-09-17T07:11:03.047 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:11:03.047 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:11:03.047 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:11:03.048 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.048 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.048 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:77688 7.11 7:8ad92a20:::benchmark_data_smithi100_78556_object77687:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-09-17T07:11:03.048 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:05:22.422717+0000", 2024-09-17T07:11:03.048 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 340.57404334799998, 2024-09-17T07:11:03.048 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.85855422400000003, 2024-09-17T07:11:03.048 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:11:03.048 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:11:03.048 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:11:03.048 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:11:03.048 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:11:03.048 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 77688 2024-09-17T07:11:03.049 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.049 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:11:03.049 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.049 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:11:03.053 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:22.422717+0000", 2024-09-17T07:11:03.053 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:11:03.053 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.053 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.053 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:11:03.053 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:22.422717+0000", 2024-09-17T07:11:03.053 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:11:03.053 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.053 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.053 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:11:03.053 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:22.422715+0000", 2024-09-17T07:11:03.053 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:11:03.054 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.054 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.054 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:11:03.054 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:22.422772+0000", 2024-09-17T07:11:03.054 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.6440999999999997e-05 2024-09-17T07:11:03.054 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.054 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.054 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:11:03.054 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:22.422786+0000", 2024-09-17T07:11:03.054 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.4603000000000001e-05 2024-09-17T07:11:03.054 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.054 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.054 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:11:03.055 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:22.422795+0000", 2024-09-17T07:11:03.055 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.5189999999999992e-06 2024-09-17T07:11:03.055 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.055 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.055 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:11:03.055 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:22.422849+0000", 2024-09-17T07:11:03.055 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.4049999999999999e-05 2024-09-17T07:11:03.055 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.055 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.055 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:11:03.055 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:22.422859+0000", 2024-09-17T07:11:03.055 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.5910000000000002e-06 2024-09-17T07:11:03.056 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.056 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.056 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:11:03.056 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.272328+0000", 2024-09-17T07:11:03.056 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.84946953999999997 2024-09-17T07:11:03.056 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.056 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.056 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:11:03.056 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.272461+0000", 2024-09-17T07:11:03.056 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00013322900000000001 2024-09-17T07:11:03.056 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.056 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.056 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 3,6", 2024-09-17T07:11:03.057 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.272640+0000", 2024-09-17T07:11:03.057 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000178464 2024-09-17T07:11:03.057 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.057 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.057 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:11:03.057 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.275257+0000", 2024-09-17T07:11:03.057 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0026168070000000001 2024-09-17T07:11:03.057 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.057 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.057 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:11:03.057 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.275268+0000", 2024-09-17T07:11:03.058 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.1212999999999999e-05 2024-09-17T07:11:03.058 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.058 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.058 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:11:03.058 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.281196+0000", 2024-09-17T07:11:03.058 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0059279459999999999 2024-09-17T07:11:03.058 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.058 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.058 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:11:03.058 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.281221+0000", 2024-09-17T07:11:03.058 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.5678e-05 2024-09-17T07:11:03.058 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.058 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.059 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:11:03.059 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.281271+0000", 2024-09-17T07:11:03.059 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.9771000000000002e-05 2024-09-17T07:11:03.059 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:11:03.059 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:11:03.059 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:11:03.059 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.059 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.059 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:77736 7.11 7:8ce7f145:::benchmark_data_smithi100_78556_object77735:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-09-17T07:11:03.059 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:05:22.474543+0000", 2024-09-17T07:11:03.059 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 340.52221690900001, 2024-09-17T07:11:03.059 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.81044233300000001, 2024-09-17T07:11:03.059 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:11:03.060 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:11:03.060 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:11:03.060 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:11:03.060 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:11:03.060 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 77736 2024-09-17T07:11:03.060 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.060 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:11:03.060 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.060 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:11:03.060 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:22.474543+0000", 2024-09-17T07:11:03.060 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:11:03.060 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.060 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.061 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:11:03.061 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:22.474543+0000", 2024-09-17T07:11:03.061 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:11:03.061 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.061 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.061 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:11:03.061 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:22.474542+0000", 2024-09-17T07:11:03.061 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.999999 2024-09-17T07:11:03.061 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.061 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.061 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:11:03.061 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:22.474602+0000", 2024-09-17T07:11:03.061 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.9852000000000002e-05 2024-09-17T07:11:03.062 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.062 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.062 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:11:03.062 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:22.474613+0000", 2024-09-17T07:11:03.062 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.1038e-05 2024-09-17T07:11:03.062 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.062 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.062 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:11:03.062 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:22.474619+0000", 2024-09-17T07:11:03.062 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.0530000000000003e-06 2024-09-17T07:11:03.062 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.062 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.062 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:11:03.063 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:22.474668+0000", 2024-09-17T07:11:03.063 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.8517999999999999e-05 2024-09-17T07:11:03.063 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.063 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.063 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:11:03.063 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:22.474675+0000", 2024-09-17T07:11:03.063 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.827e-06 2024-09-17T07:11:03.063 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.063 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.063 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:11:03.063 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.273464+0000", 2024-09-17T07:11:03.063 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.79878933799999996 2024-09-17T07:11:03.063 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.064 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.064 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:11:03.064 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.273563+0000", 2024-09-17T07:11:03.064 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.8820999999999995e-05 2024-09-17T07:11:03.064 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.064 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.064 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 3,6", 2024-09-17T07:11:03.064 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.273667+0000", 2024-09-17T07:11:03.064 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000103819 2024-09-17T07:11:03.064 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.064 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.064 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:11:03.064 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.275321+0000", 2024-09-17T07:11:03.065 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.001654109 2024-09-17T07:11:03.065 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.065 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.065 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:11:03.065 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.276551+0000", 2024-09-17T07:11:03.065 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.001230047 2024-09-17T07:11:03.065 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.065 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.065 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:11:03.065 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.284793+0000", 2024-09-17T07:11:03.065 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0082423849999999996 2024-09-17T07:11:03.065 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.065 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.066 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:11:03.066 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.284884+0000", 2024-09-17T07:11:03.066 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.1281999999999998e-05 2024-09-17T07:11:03.066 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.066 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.066 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:11:03.066 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.284986+0000", 2024-09-17T07:11:03.066 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00010141699999999999 2024-09-17T07:11:03.066 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:11:03.066 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:11:03.066 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:11:03.066 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.066 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.067 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:77737 7.11 7:8d47d682:::benchmark_data_smithi100_78556_object77736:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-09-17T07:11:03.067 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:05:22.478029+0000", 2024-09-17T07:11:03.067 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 340.518731006, 2024-09-17T07:11:03.067 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.81293265999999997, 2024-09-17T07:11:03.067 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:11:03.067 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:11:03.067 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:11:03.067 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:11:03.067 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:11:03.067 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 77737 2024-09-17T07:11:03.067 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.067 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:11:03.067 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.067 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:11:03.068 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:22.478029+0000", 2024-09-17T07:11:03.068 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:11:03.068 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.068 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.068 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:11:03.068 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:22.478029+0000", 2024-09-17T07:11:03.068 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:11:03.068 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.068 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.068 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:11:03.068 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:22.478026+0000", 2024-09-17T07:11:03.068 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999962 2024-09-17T07:11:03.069 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.069 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.069 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:11:03.069 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:22.478091+0000", 2024-09-17T07:11:03.069 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.5185999999999996e-05 2024-09-17T07:11:03.069 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.069 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.069 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:11:03.069 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:22.478102+0000", 2024-09-17T07:11:03.069 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.1477e-05 2024-09-17T07:11:03.069 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.070 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.070 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:11:03.070 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:22.478110+0000", 2024-09-17T07:11:03.070 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.6490000000000008e-06 2024-09-17T07:11:03.070 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.070 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.070 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:11:03.070 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:22.478178+0000", 2024-09-17T07:11:03.070 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.7701999999999997e-05 2024-09-17T07:11:03.070 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.070 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.071 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:11:03.071 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:22.478188+0000", 2024-09-17T07:11:03.071 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.9990000000000003e-06 2024-09-17T07:11:03.071 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.071 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.071 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:11:03.071 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.274455+0000", 2024-09-17T07:11:03.071 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.79626708700000004 2024-09-17T07:11:03.071 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.071 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.071 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:11:03.071 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.274551+0000", 2024-09-17T07:11:03.071 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.6096999999999998e-05 2024-09-17T07:11:03.072 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.072 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.072 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 3,6", 2024-09-17T07:11:03.072 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.274642+0000", 2024-09-17T07:11:03.072 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.1228000000000001e-05 2024-09-17T07:11:03.072 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.072 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.072 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:11:03.072 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.277999+0000", 2024-09-17T07:11:03.073 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0033572680000000001 2024-09-17T07:11:03.073 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.073 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.073 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:11:03.073 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.278011+0000", 2024-09-17T07:11:03.073 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.1766000000000001e-05 2024-09-17T07:11:03.073 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.073 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.074 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:11:03.074 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.290727+0000", 2024-09-17T07:11:03.074 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.012716032 2024-09-17T07:11:03.074 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.074 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.074 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:11:03.074 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.290846+0000", 2024-09-17T07:11:03.074 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00011924199999999999 2024-09-17T07:11:03.074 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.074 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.074 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:11:03.074 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.290962+0000", 2024-09-17T07:11:03.074 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000115573 2024-09-17T07:11:03.074 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:11:03.075 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:11:03.075 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:11:03.075 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.075 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.075 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:133492 7.17 7:e82e7f25:::benchmark_data_smithi100_78556_object22689:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-09-17T07:11:03.075 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:06:21.233865+0000", 2024-09-17T07:11:03.075 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 281.76289504900001, 2024-09-17T07:11:03.075 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0087981210000001, 2024-09-17T07:11:03.075 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:11:03.075 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:11:03.075 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:11:03.075 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:11:03.075 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:11:03.076 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 133492 2024-09-17T07:11:03.076 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.076 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:11:03.076 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.076 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:11:03.076 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.233865+0000", 2024-09-17T07:11:03.076 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:11:03.076 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.076 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.076 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:11:03.076 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.233865+0000", 2024-09-17T07:11:03.076 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:11:03.076 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.077 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.077 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:11:03.077 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.233864+0000", 2024-09-17T07:11:03.077 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:11:03.077 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.077 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.077 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:11:03.077 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.233870+0000", 2024-09-17T07:11:03.077 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.9739999999999999e-06 2024-09-17T07:11:03.077 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.077 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.077 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:11:03.077 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.233881+0000", 2024-09-17T07:11:03.078 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0978000000000001e-05 2024-09-17T07:11:03.078 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.078 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.078 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:11:03.078 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.233887+0000", 2024-09-17T07:11:03.078 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.8599999999999998e-06 2024-09-17T07:11:03.078 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.078 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.078 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:11:03.078 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.233943+0000", 2024-09-17T07:11:03.078 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.5872000000000003e-05 2024-09-17T07:11:03.078 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.078 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.078 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:11:03.079 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.233954+0000", 2024-09-17T07:11:03.079 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.1399e-05 2024-09-17T07:11:03.079 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.079 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.079 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:11:03.079 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.237945+0000", 2024-09-17T07:11:03.079 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0039914249999999 2024-09-17T07:11:03.079 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.079 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.079 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:11:03.079 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.238033+0000", 2024-09-17T07:11:03.079 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.7311999999999993e-05 2024-09-17T07:11:03.080 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.080 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.080 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 2,5", 2024-09-17T07:11:03.080 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.238263+0000", 2024-09-17T07:11:03.080 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000230316 2024-09-17T07:11:03.080 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.080 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.080 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:11:03.080 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.242538+0000", 2024-09-17T07:11:03.080 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0042753299999999999 2024-09-17T07:11:03.080 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.080 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.081 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:11:03.081 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.242550+0000", 2024-09-17T07:11:03.081 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.1325000000000001e-05 2024-09-17T07:11:03.081 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.081 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.081 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:11:03.081 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.242586+0000", 2024-09-17T07:11:03.081 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.6467999999999997e-05 2024-09-17T07:11:03.081 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.081 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.081 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:11:03.081 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.242628+0000", 2024-09-17T07:11:03.081 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.1780000000000003e-05 2024-09-17T07:11:03.082 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.082 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.082 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:11:03.082 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.242663+0000", 2024-09-17T07:11:03.082 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.5460999999999998e-05 2024-09-17T07:11:03.082 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:11:03.082 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:11:03.082 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:11:03.082 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.082 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.082 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:133561 7.17 7:ebede8e9:::benchmark_data_smithi100_78556_object22758:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-09-17T07:11:03.082 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:06:21.290634+0000", 2024-09-17T07:11:03.082 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 281.70612608300002, 2024-09-17T07:11:03.082 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.95504869599999997, 2024-09-17T07:11:03.083 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:11:03.083 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:11:03.083 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:11:03.083 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:11:03.083 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:11:03.083 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 133561 2024-09-17T07:11:03.083 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.083 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:11:03.083 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.084 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:11:03.084 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:11:03.085 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:11:03.085 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.290634+0000", 2024-09-17T07:11:03.085 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:11:03.085 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.085 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.085 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:11:03.085 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.290634+0000", 2024-09-17T07:11:03.086 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:11:03.086 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.086 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.086 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:11:03.086 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.290633+0000", 2024-09-17T07:11:03.086 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:11:03.086 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.086 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.086 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:11:03.086 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.290640+0000", 2024-09-17T07:11:03.086 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.922e-06 2024-09-17T07:11:03.086 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.087 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.087 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:11:03.087 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.290650+0000", 2024-09-17T07:11:03.087 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0404999999999999e-05 2024-09-17T07:11:03.087 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.087 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.087 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:11:03.087 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.290656+0000", 2024-09-17T07:11:03.087 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.429e-06 2024-09-17T07:11:03.087 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.087 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.087 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:11:03.087 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.290707+0000", 2024-09-17T07:11:03.087 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.1591999999999997e-05 2024-09-17T07:11:03.088 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.088 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.088 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:11:03.088 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.290718+0000", 2024-09-17T07:11:03.088 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0383999999999999e-05 2024-09-17T07:11:03.088 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.088 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.088 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:11:03.088 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.239856+0000", 2024-09-17T07:11:03.088 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.94913878799999996 2024-09-17T07:11:03.088 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.088 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.089 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:11:03.089 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.239976+0000", 2024-09-17T07:11:03.089 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000119456 2024-09-17T07:11:03.089 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.089 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.089 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 2,5", 2024-09-17T07:11:03.089 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.240193+0000", 2024-09-17T07:11:03.089 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00021685500000000001 2024-09-17T07:11:03.089 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.089 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.089 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:11:03.089 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.242807+0000", 2024-09-17T07:11:03.089 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0026144380000000002 2024-09-17T07:11:03.090 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.090 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.090 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:11:03.090 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.242858+0000", 2024-09-17T07:11:03.090 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.1082999999999999e-05 2024-09-17T07:11:03.090 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.090 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.090 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:11:03.090 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.245615+0000", 2024-09-17T07:11:03.090 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0027572529999999999 2024-09-17T07:11:03.090 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.090 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.091 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:11:03.091 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.245645+0000", 2024-09-17T07:11:03.091 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.9247999999999998e-05 2024-09-17T07:11:03.091 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.091 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.091 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:11:03.091 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.245683+0000", 2024-09-17T07:11:03.091 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.8303999999999998e-05 2024-09-17T07:11:03.091 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:11:03.091 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:11:03.091 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:11:03.091 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.091 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.092 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:133593 7.17 7:e9fbb2d9:::benchmark_data_smithi100_78556_object22790:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-09-17T07:11:03.092 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:06:21.311909+0000", 2024-09-17T07:11:03.092 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 281.68485130599998, 2024-09-17T07:11:03.092 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.93886656400000001, 2024-09-17T07:11:03.092 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:11:03.092 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:11:03.092 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:11:03.092 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:11:03.092 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:11:03.092 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 133593 2024-09-17T07:11:03.092 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.092 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:11:03.092 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.093 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:11:03.093 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.311909+0000", 2024-09-17T07:11:03.093 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:11:03.093 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.093 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.093 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:11:03.093 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.311909+0000", 2024-09-17T07:11:03.093 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:11:03.093 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.093 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.093 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:11:03.093 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.311908+0000", 2024-09-17T07:11:03.094 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.999999 2024-09-17T07:11:03.094 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.094 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.094 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:11:03.094 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.311913+0000", 2024-09-17T07:11:03.094 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.1789999999999999e-06 2024-09-17T07:11:03.094 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.094 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.094 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:11:03.094 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.311931+0000", 2024-09-17T07:11:03.094 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.7405e-05 2024-09-17T07:11:03.094 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.094 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.095 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:11:03.095 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.311937+0000", 2024-09-17T07:11:03.095 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.3350000000000003e-06 2024-09-17T07:11:03.095 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.095 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.095 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:11:03.095 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.312304+0000", 2024-09-17T07:11:03.095 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00036720900000000001 2024-09-17T07:11:03.095 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.095 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.095 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:11:03.095 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.312314+0000", 2024-09-17T07:11:03.095 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.4180000000000003e-06 2024-09-17T07:11:03.095 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.096 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.096 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:11:03.096 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.241295+0000", 2024-09-17T07:11:03.096 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.92898169799999997 2024-09-17T07:11:03.096 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.096 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.096 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:11:03.096 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.241380+0000", 2024-09-17T07:11:03.096 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.4628000000000004e-05 2024-09-17T07:11:03.096 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.096 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.096 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 2,5", 2024-09-17T07:11:03.097 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.241568+0000", 2024-09-17T07:11:03.097 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000187646 2024-09-17T07:11:03.097 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.097 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.097 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:11:03.097 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.242846+0000", 2024-09-17T07:11:03.097 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0012786049999999999 2024-09-17T07:11:03.097 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.097 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.097 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:11:03.097 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.242863+0000", 2024-09-17T07:11:03.097 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.6540000000000001e-05 2024-09-17T07:11:03.097 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.097 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.098 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:11:03.098 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.250690+0000", 2024-09-17T07:11:03.098 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0078269770000000006 2024-09-17T07:11:03.098 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.098 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.098 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:11:03.098 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.250737+0000", 2024-09-17T07:11:03.098 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.7018000000000003e-05 2024-09-17T07:11:03.098 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.098 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.098 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:11:03.098 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.250776+0000", 2024-09-17T07:11:03.098 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.8785999999999998e-05 2024-09-17T07:11:03.099 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:11:03.099 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:11:03.099 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:11:03.099 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.099 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.099 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:133623 7.17 7:efb043b2:::benchmark_data_smithi100_78556_object22820:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-09-17T07:11:03.099 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:06:21.333369+0000", 2024-09-17T07:11:03.099 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 281.66339108699998, 2024-09-17T07:11:03.099 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.91751769500000002, 2024-09-17T07:11:03.099 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:11:03.099 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:11:03.099 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:11:03.099 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:11:03.100 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:11:03.100 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 133623 2024-09-17T07:11:03.100 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.100 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:11:03.100 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.100 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:11:03.100 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.333369+0000", 2024-09-17T07:11:03.100 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:11:03.100 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.100 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.100 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:11:03.100 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.333369+0000", 2024-09-17T07:11:03.101 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:11:03.101 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.101 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.101 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:11:03.101 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.333369+0000", 2024-09-17T07:11:03.101 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999995 2024-09-17T07:11:03.101 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.101 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.101 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:11:03.101 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.333372+0000", 2024-09-17T07:11:03.101 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.039e-06 2024-09-17T07:11:03.101 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.101 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.102 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:11:03.102 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.333382+0000", 2024-09-17T07:11:03.102 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.9389999999999992e-06 2024-09-17T07:11:03.102 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.102 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.102 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:11:03.102 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.333388+0000", 2024-09-17T07:11:03.102 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.1059999999999996e-06 2024-09-17T07:11:03.102 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.102 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.102 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:11:03.102 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.333444+0000", 2024-09-17T07:11:03.102 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.6140999999999997e-05 2024-09-17T07:11:03.103 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.103 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.103 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:11:03.103 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.333453+0000", 2024-09-17T07:11:03.103 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.8759999999999994e-06 2024-09-17T07:11:03.103 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.103 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.103 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:11:03.103 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.242890+0000", 2024-09-17T07:11:03.103 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.90943727500000004 2024-09-17T07:11:03.103 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.103 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.103 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:11:03.104 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.242963+0000", 2024-09-17T07:11:03.104 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.2889e-05 2024-09-17T07:11:03.104 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.104 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.104 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 2,5", 2024-09-17T07:11:03.104 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.243109+0000", 2024-09-17T07:11:03.104 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00014637999999999999 2024-09-17T07:11:03.104 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.104 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.104 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:11:03.104 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.245732+0000", 2024-09-17T07:11:03.104 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.002622479 2024-09-17T07:11:03.104 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.105 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.105 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:11:03.105 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.245741+0000", 2024-09-17T07:11:03.105 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.3859999999999996e-06 2024-09-17T07:11:03.105 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.105 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.105 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:11:03.105 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.250809+0000", 2024-09-17T07:11:03.105 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0050681119999999996 2024-09-17T07:11:03.105 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.105 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.105 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:11:03.105 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.250852+0000", 2024-09-17T07:11:03.105 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.2858999999999997e-05 2024-09-17T07:11:03.106 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.106 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.106 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:11:03.106 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.250887+0000", 2024-09-17T07:11:03.106 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.4672000000000003e-05 2024-09-17T07:11:03.106 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:11:03.106 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:11:03.106 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:11:03.106 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.106 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.106 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:133647 7.17 7:ed38d1ee:::benchmark_data_smithi100_78556_object22844:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-09-17T07:11:03.106 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:06:21.354171+0000", 2024-09-17T07:11:03.107 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 281.64258901199997, 2024-09-17T07:11:03.107 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.90300700300000003, 2024-09-17T07:11:03.107 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:11:03.107 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:11:03.107 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:11:03.107 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:11:03.107 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:11:03.107 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 133647 2024-09-17T07:11:03.107 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.107 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:11:03.107 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.107 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:11:03.107 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.354171+0000", 2024-09-17T07:11:03.107 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:11:03.108 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.108 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.108 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:11:03.108 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.354171+0000", 2024-09-17T07:11:03.108 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:11:03.108 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.108 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.108 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:11:03.108 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.354170+0000", 2024-09-17T07:11:03.108 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:11:03.108 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.108 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.108 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:11:03.109 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.354176+0000", 2024-09-17T07:11:03.109 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.8199999999999999e-06 2024-09-17T07:11:03.109 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.109 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.109 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:11:03.109 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.354189+0000", 2024-09-17T07:11:03.109 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.2628e-05 2024-09-17T07:11:03.109 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.109 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.109 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:11:03.109 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.354196+0000", 2024-09-17T07:11:03.109 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.0269999999999998e-06 2024-09-17T07:11:03.109 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.110 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.110 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:11:03.110 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.354242+0000", 2024-09-17T07:11:03.110 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.5977999999999999e-05 2024-09-17T07:11:03.110 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.110 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.110 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:11:03.110 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.354253+0000", 2024-09-17T07:11:03.110 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0705e-05 2024-09-17T07:11:03.110 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.110 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.110 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:11:03.110 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.244353+0000", 2024-09-17T07:11:03.111 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.89009969200000005 2024-09-17T07:11:03.111 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.111 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.111 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:11:03.111 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.244453+0000", 2024-09-17T07:11:03.111 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000100108 2024-09-17T07:11:03.111 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.111 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.111 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 2,5", 2024-09-17T07:11:03.111 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.244627+0000", 2024-09-17T07:11:03.111 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00017430599999999999 2024-09-17T07:11:03.111 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.111 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.111 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:11:03.112 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.247973+0000", 2024-09-17T07:11:03.112 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0033458680000000001 2024-09-17T07:11:03.112 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.112 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.112 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:11:03.112 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.248021+0000", 2024-09-17T07:11:03.112 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.8018e-05 2024-09-17T07:11:03.112 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.112 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.112 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:11:03.112 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.257059+0000", 2024-09-17T07:11:03.112 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0090379129999999998 2024-09-17T07:11:03.112 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.113 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.113 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:11:03.113 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.257140+0000", 2024-09-17T07:11:03.113 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.1446999999999995e-05 2024-09-17T07:11:03.113 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.113 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.113 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:11:03.113 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.257178+0000", 2024-09-17T07:11:03.113 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.8148000000000001e-05 2024-09-17T07:11:03.113 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:11:03.113 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:11:03.113 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:11:03.113 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.114 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.114 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:133686 7.17 7:ec7660e3:::benchmark_data_smithi100_78556_object22883:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-09-17T07:11:03.114 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:06:21.385193+0000", 2024-09-17T07:11:03.114 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 281.61156777000002, 2024-09-17T07:11:03.114 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.87216530299999995, 2024-09-17T07:11:03.114 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:11:03.114 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:11:03.114 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:11:03.114 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:11:03.114 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:11:03.114 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 133686 2024-09-17T07:11:03.114 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.114 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:11:03.115 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.115 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:11:03.115 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.385193+0000", 2024-09-17T07:11:03.115 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:11:03.115 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.115 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.115 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:11:03.115 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.385193+0000", 2024-09-17T07:11:03.115 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:11:03.115 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.115 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.115 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:11:03.115 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.385191+0000", 2024-09-17T07:11:03.116 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:11:03.116 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.116 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.116 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:11:03.116 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.385199+0000", 2024-09-17T07:11:03.116 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.7100000000000007e-06 2024-09-17T07:11:03.116 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.116 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.116 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:11:03.116 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.385211+0000", 2024-09-17T07:11:03.116 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.2513e-05 2024-09-17T07:11:03.116 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.116 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.117 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:11:03.117 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.385220+0000", 2024-09-17T07:11:03.117 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.6379999999999993e-06 2024-09-17T07:11:03.117 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.117 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.117 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:11:03.117 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.385274+0000", 2024-09-17T07:11:03.117 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.4629999999999997e-05 2024-09-17T07:11:03.117 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.117 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.117 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:11:03.117 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.385284+0000", 2024-09-17T07:11:03.117 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.1910000000000006e-06 2024-09-17T07:11:03.118 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.118 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.118 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:11:03.118 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.245775+0000", 2024-09-17T07:11:03.118 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.86049110799999995 2024-09-17T07:11:03.118 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.118 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.118 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:11:03.118 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.245884+0000", 2024-09-17T07:11:03.118 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000109576 2024-09-17T07:11:03.118 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.118 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.118 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 2,5", 2024-09-17T07:11:03.119 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.246086+0000", 2024-09-17T07:11:03.119 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00020126700000000001 2024-09-17T07:11:03.119 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.119 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.119 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:11:03.119 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.248065+0000", 2024-09-17T07:11:03.119 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00197956 2024-09-17T07:11:03.119 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.119 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.119 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:11:03.119 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.248072+0000", 2024-09-17T07:11:03.119 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.9619999999999996e-06 2024-09-17T07:11:03.119 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.120 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.120 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:11:03.120 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.257235+0000", 2024-09-17T07:11:03.120 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0091628419999999992 2024-09-17T07:11:03.120 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.120 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.120 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:11:03.120 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.257319+0000", 2024-09-17T07:11:03.120 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.3988999999999998e-05 2024-09-17T07:11:03.120 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.120 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.120 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:11:03.121 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.257358+0000", 2024-09-17T07:11:03.121 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.892e-05 2024-09-17T07:11:03.121 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:11:03.121 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:11:03.121 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:11:03.121 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.121 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.121 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:133688 7.17 7:ef66ce5f:::benchmark_data_smithi100_78556_object22885:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-09-17T07:11:03.121 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:06:21.385272+0000", 2024-09-17T07:11:03.121 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 281.611488463, 2024-09-17T07:11:03.121 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.87961432500000003, 2024-09-17T07:11:03.121 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:11:03.121 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:11:03.122 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:11:03.122 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:11:03.122 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:11:03.122 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 133688 2024-09-17T07:11:03.122 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.122 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:11:03.122 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.122 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:11:03.122 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.385272+0000", 2024-09-17T07:11:03.122 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:11:03.122 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.122 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.122 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:11:03.123 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.385272+0000", 2024-09-17T07:11:03.123 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:11:03.123 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.123 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.123 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:11:03.123 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.385271+0000", 2024-09-17T07:11:03.123 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999995 2024-09-17T07:11:03.123 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.123 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.123 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:11:03.123 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.385275+0000", 2024-09-17T07:11:03.123 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.0629999999999999e-06 2024-09-17T07:11:03.123 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.124 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.124 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:11:03.124 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.385286+0000", 2024-09-17T07:11:03.124 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0455000000000001e-05 2024-09-17T07:11:03.124 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.124 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.124 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:11:03.124 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.385292+0000", 2024-09-17T07:11:03.124 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.1619999999999996e-06 2024-09-17T07:11:03.124 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.124 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.124 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:11:03.125 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.385355+0000", 2024-09-17T07:11:03.125 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.2841000000000004e-05 2024-09-17T07:11:03.125 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.125 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.125 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:11:03.125 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.385364+0000", 2024-09-17T07:11:03.125 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.0389999999999993e-06 2024-09-17T07:11:03.125 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.125 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.125 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:11:03.125 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.246954+0000", 2024-09-17T07:11:03.125 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.86158991100000004 2024-09-17T07:11:03.125 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.126 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.126 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:11:03.126 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.247027+0000", 2024-09-17T07:11:03.126 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.2847000000000002e-05 2024-09-17T07:11:03.126 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.126 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.126 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 2,5", 2024-09-17T07:11:03.126 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.247185+0000", 2024-09-17T07:11:03.126 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00015882200000000001 2024-09-17T07:11:03.126 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.126 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.126 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:11:03.126 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.250636+0000", 2024-09-17T07:11:03.126 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0034508130000000001 2024-09-17T07:11:03.127 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.127 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.127 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:11:03.127 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.250932+0000", 2024-09-17T07:11:03.127 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00029620800000000001 2024-09-17T07:11:03.127 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.127 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.127 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:11:03.127 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.264818+0000", 2024-09-17T07:11:03.127 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.013885784999999999 2024-09-17T07:11:03.127 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.127 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.127 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:11:03.128 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.264850+0000", 2024-09-17T07:11:03.128 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.1643999999999997e-05 2024-09-17T07:11:03.128 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.128 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.128 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:11:03.128 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.264886+0000", 2024-09-17T07:11:03.128 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.6295999999999999e-05 2024-09-17T07:11:03.128 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:11:03.128 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:11:03.128 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:11:03.128 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.128 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.128 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:133704 7.17 7:ea78a78f:::benchmark_data_smithi100_78556_object22901:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-09-17T07:11:03.129 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:06:21.399193+0000", 2024-09-17T07:11:03.129 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 281.59756728500003, 2024-09-17T07:11:03.129 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.86581602800000002, 2024-09-17T07:11:03.129 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:11:03.129 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:11:03.129 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:11:03.129 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:11:03.129 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:11:03.129 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 133704 2024-09-17T07:11:03.129 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.129 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:11:03.129 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.129 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:11:03.130 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.399193+0000", 2024-09-17T07:11:03.130 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:11:03.130 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.130 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.130 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:11:03.130 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.399193+0000", 2024-09-17T07:11:03.130 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:11:03.130 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.130 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.130 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:11:03.130 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.399192+0000", 2024-09-17T07:11:03.130 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:11:03.130 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.131 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.131 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:11:03.131 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.399198+0000", 2024-09-17T07:11:03.131 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.2489999999999998e-06 2024-09-17T07:11:03.131 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.131 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.131 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:11:03.131 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.399209+0000", 2024-09-17T07:11:03.131 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.1411e-05 2024-09-17T07:11:03.131 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.131 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.131 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:11:03.131 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.399216+0000", 2024-09-17T07:11:03.132 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.9190000000000002e-06 2024-09-17T07:11:03.132 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.132 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.132 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:11:03.132 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.399269+0000", 2024-09-17T07:11:03.132 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.2812999999999997e-05 2024-09-17T07:11:03.132 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.132 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.132 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:11:03.132 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.399278+0000", 2024-09-17T07:11:03.132 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.0960000000000007e-06 2024-09-17T07:11:03.132 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.132 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.133 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:11:03.133 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.248091+0000", 2024-09-17T07:11:03.133 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.84881321399999998 2024-09-17T07:11:03.133 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.133 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.133 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:11:03.133 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.248157+0000", 2024-09-17T07:11:03.133 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.5504000000000002e-05 2024-09-17T07:11:03.133 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.133 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.133 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 2,5", 2024-09-17T07:11:03.133 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.248302+0000", 2024-09-17T07:11:03.134 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000145277 2024-09-17T07:11:03.134 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.134 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.134 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:11:03.134 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.250894+0000", 2024-09-17T07:11:03.134 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0025922459999999999 2024-09-17T07:11:03.134 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.134 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.134 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:11:03.134 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.250965+0000", 2024-09-17T07:11:03.134 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.0650999999999998e-05 2024-09-17T07:11:03.134 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.134 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.134 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:11:03.135 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.264949+0000", 2024-09-17T07:11:03.135 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.013983923000000001 2024-09-17T07:11:03.135 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.135 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.135 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:11:03.135 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.264974+0000", 2024-09-17T07:11:03.135 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.4705000000000001e-05 2024-09-17T07:11:03.135 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.135 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.135 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:11:03.135 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.265009+0000", 2024-09-17T07:11:03.135 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.5357e-05 2024-09-17T07:11:03.135 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:11:03.136 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:11:03.136 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:11:03.136 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.136 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.136 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:133742 7.17 7:eeafcb5d:::benchmark_data_smithi100_78556_object22939:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-09-17T07:11:03.136 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:06:21.427739+0000", 2024-09-17T07:11:03.136 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 281.56902123399999, 2024-09-17T07:11:03.136 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.83739142099999997, 2024-09-17T07:11:03.136 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:11:03.136 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:11:03.136 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:11:03.136 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:11:03.136 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:11:03.137 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 133742 2024-09-17T07:11:03.137 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.137 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:11:03.137 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.137 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:11:03.137 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.427739+0000", 2024-09-17T07:11:03.137 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:11:03.137 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.137 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.137 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:11:03.137 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.427739+0000", 2024-09-17T07:11:03.137 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:11:03.137 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.138 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.138 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:11:03.138 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.427738+0000", 2024-09-17T07:11:03.138 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:11:03.138 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.138 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.138 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:11:03.138 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.427744+0000", 2024-09-17T07:11:03.138 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.365e-06 2024-09-17T07:11:03.138 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.138 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.138 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:11:03.138 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.427757+0000", 2024-09-17T07:11:03.139 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.2744e-05 2024-09-17T07:11:03.139 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.139 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.139 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:11:03.139 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.427764+0000", 2024-09-17T07:11:03.139 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.1180000000000003e-06 2024-09-17T07:11:03.139 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.139 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.139 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:11:03.139 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.427825+0000", 2024-09-17T07:11:03.139 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.0665999999999998e-05 2024-09-17T07:11:03.139 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.139 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.140 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:11:03.140 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.427837+0000", 2024-09-17T07:11:03.140 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.2694000000000001e-05 2024-09-17T07:11:03.140 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.140 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.140 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:11:03.140 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.249221+0000", 2024-09-17T07:11:03.140 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.82138384099999995 2024-09-17T07:11:03.140 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.140 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.140 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:11:03.140 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.249321+0000", 2024-09-17T07:11:03.140 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00010035900000000001 2024-09-17T07:11:03.141 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.141 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.141 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 2,5", 2024-09-17T07:11:03.141 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.249530+0000", 2024-09-17T07:11:03.141 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00020840799999999999 2024-09-17T07:11:03.141 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.141 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.141 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:11:03.141 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.250970+0000", 2024-09-17T07:11:03.141 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0014404940000000001 2024-09-17T07:11:03.141 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.141 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.141 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:11:03.141 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.250990+0000", 2024-09-17T07:11:03.142 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.9740000000000001e-05 2024-09-17T07:11:03.142 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.142 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.142 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:11:03.142 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.265046+0000", 2024-09-17T07:11:03.142 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.014055412999999999 2024-09-17T07:11:03.142 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.142 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.142 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:11:03.142 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.265094+0000", 2024-09-17T07:11:03.142 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.8690999999999998e-05 2024-09-17T07:11:03.142 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.143 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.143 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:11:03.143 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.265131+0000", 2024-09-17T07:11:03.143 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.6263999999999997e-05 2024-09-17T07:11:03.143 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:11:03.143 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:11:03.143 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:11:03.143 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.143 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.143 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:133756 7.17 7:ed7bd19c:::benchmark_data_smithi100_78556_object22953:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-09-17T07:11:03.143 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:06:21.438670+0000", 2024-09-17T07:11:03.143 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 281.55809052400002, 2024-09-17T07:11:03.143 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.82965550099999996, 2024-09-17T07:11:03.144 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:11:03.144 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:11:03.144 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:11:03.144 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:11:03.144 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:11:03.144 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 133756 2024-09-17T07:11:03.144 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.144 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:11:03.144 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.144 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:11:03.144 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.438670+0000", 2024-09-17T07:11:03.144 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:11:03.145 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.145 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.145 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:11:03.145 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.438670+0000", 2024-09-17T07:11:03.145 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:11:03.145 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.145 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.145 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:11:03.145 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.438668+0000", 2024-09-17T07:11:03.145 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:11:03.145 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.145 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.145 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:11:03.146 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.438674+0000", 2024-09-17T07:11:03.146 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.9390000000000002e-06 2024-09-17T07:11:03.146 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.146 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.146 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:11:03.146 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.438686+0000", 2024-09-17T07:11:03.146 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.1685000000000001e-05 2024-09-17T07:11:03.146 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.146 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.146 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:11:03.146 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.438694+0000", 2024-09-17T07:11:03.146 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.7959999999999996e-06 2024-09-17T07:11:03.147 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.147 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.147 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:11:03.147 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.438743+0000", 2024-09-17T07:11:03.147 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.9443000000000001e-05 2024-09-17T07:11:03.147 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.147 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.147 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:11:03.147 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.438750+0000", 2024-09-17T07:11:03.147 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.2029999999999997e-06 2024-09-17T07:11:03.147 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.147 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.147 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:11:03.148 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.251012+0000", 2024-09-17T07:11:03.148 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.81226268999999995 2024-09-17T07:11:03.148 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.148 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.148 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:11:03.148 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.251078+0000", 2024-09-17T07:11:03.148 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.5424000000000002e-05 2024-09-17T07:11:03.148 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.148 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.148 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 2,5", 2024-09-17T07:11:03.148 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.251225+0000", 2024-09-17T07:11:03.148 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00014769100000000001 2024-09-17T07:11:03.148 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.149 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.149 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:11:03.149 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.254317+0000", 2024-09-17T07:11:03.149 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0030916749999999999 2024-09-17T07:11:03.149 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.149 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.149 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:11:03.149 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.254363+0000", 2024-09-17T07:11:03.149 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.6045999999999999e-05 2024-09-17T07:11:03.149 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.149 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.149 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:11:03.149 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.268276+0000", 2024-09-17T07:11:03.150 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.013912590000000001 2024-09-17T07:11:03.150 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.150 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.150 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:11:03.150 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.268302+0000", 2024-09-17T07:11:03.150 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.6795e-05 2024-09-17T07:11:03.150 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.150 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.150 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:11:03.150 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.268325+0000", 2024-09-17T07:11:03.150 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.2883000000000001e-05 2024-09-17T07:11:03.150 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:11:03.150 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:11:03.151 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:11:03.151 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.151 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.151 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:133762 7.17 7:e8f37ebb:::benchmark_data_smithi100_78556_object22959:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-09-17T07:11:03.151 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:06:21.442731+0000", 2024-09-17T07:11:03.151 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 281.55402950199999, 2024-09-17T07:11:03.151 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.82569069399999995, 2024-09-17T07:11:03.151 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:11:03.151 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:11:03.151 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:11:03.151 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:11:03.151 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:11:03.151 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 133762 2024-09-17T07:11:03.152 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.152 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:11:03.152 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.152 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:11:03.152 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.442731+0000", 2024-09-17T07:11:03.152 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:11:03.152 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.152 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.152 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:11:03.152 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.442731+0000", 2024-09-17T07:11:03.152 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:11:03.152 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.153 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.153 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:11:03.153 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.442730+0000", 2024-09-17T07:11:03.153 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.999999 2024-09-17T07:11:03.153 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.153 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.153 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:11:03.153 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.442737+0000", 2024-09-17T07:11:03.153 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.4039999999999998e-06 2024-09-17T07:11:03.153 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.153 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.153 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:11:03.153 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.442748+0000", 2024-09-17T07:11:03.154 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0335e-05 2024-09-17T07:11:03.154 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.154 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.154 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:11:03.154 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.442753+0000", 2024-09-17T07:11:03.154 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.1760000000000001e-06 2024-09-17T07:11:03.154 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.154 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.154 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:11:03.154 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.442805+0000", 2024-09-17T07:11:03.154 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.2049999999999998e-05 2024-09-17T07:11:03.154 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.154 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.155 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:11:03.155 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:21.442816+0000", 2024-09-17T07:11:03.155 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.1079e-05 2024-09-17T07:11:03.155 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.155 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.155 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:11:03.155 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.252036+0000", 2024-09-17T07:11:03.155 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.80921993599999997 2024-09-17T07:11:03.155 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.155 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.155 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:11:03.155 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.252146+0000", 2024-09-17T07:11:03.155 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00011021799999999999 2024-09-17T07:11:03.156 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.156 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.156 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 2,5", 2024-09-17T07:11:03.156 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.252317+0000", 2024-09-17T07:11:03.156 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00017104500000000001 2024-09-17T07:11:03.156 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.156 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.156 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:11:03.156 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.254355+0000", 2024-09-17T07:11:03.156 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.002038322 2024-09-17T07:11:03.156 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.156 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.156 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:11:03.157 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.254370+0000", 2024-09-17T07:11:03.157 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.4285e-05 2024-09-17T07:11:03.158 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.158 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.158 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:11:03.158 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.268360+0000", 2024-09-17T07:11:03.158 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.013990677999999999 2024-09-17T07:11:03.158 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.158 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.158 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:11:03.158 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.268390+0000", 2024-09-17T07:11:03.158 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.0032e-05 2024-09-17T07:11:03.158 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:11:03.158 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:11:03.159 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:11:03.159 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:06:22.268422+0000", 2024-09-17T07:11:03.159 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.1095e-05 2024-09-17T07:11:03.159 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:11:03.159 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:11:03.159 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:11:03.159 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:11:03.159 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:11:03.159 INFO:teuthology.orchestra.run.smithi135.stdout:} 2024-09-17T07:11:03.159 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:11:03.191 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:03.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:03.192 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:11:03.293 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:11:03.339 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:03.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:03.340 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:11:03.441 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:11:03.464 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:03 smithi135 ceph-mon[33238]: 5.14 scrub starts 2024-09-17T07:11:03.464 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:03 smithi135 ceph-mon[33238]: 5.14 scrub ok 2024-09-17T07:11:03.464 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:03 smithi135 ceph-mon[33238]: 4.1b scrub starts 2024-09-17T07:11:03.464 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:03 smithi135 ceph-mon[33238]: 4.1b scrub ok 2024-09-17T07:11:03.464 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:03 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:11:03.464 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:03 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1289506520' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:11:03.475 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:03.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:03.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:11:03.577 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:11:03.610 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:03.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:03.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:11:03.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:03 smithi100 ceph-mon[29092]: 5.14 scrub starts 2024-09-17T07:11:03.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:03 smithi100 ceph-mon[29092]: 5.14 scrub ok 2024-09-17T07:11:03.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:03 smithi100 ceph-mon[29092]: 4.1b scrub starts 2024-09-17T07:11:03.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:03 smithi100 ceph-mon[29092]: 4.1b scrub ok 2024-09-17T07:11:03.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:03 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:11:03.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:03 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1289506520' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:11:03.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:03 smithi100 ceph-mon[39462]: 5.14 scrub starts 2024-09-17T07:11:03.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:03 smithi100 ceph-mon[39462]: 5.14 scrub ok 2024-09-17T07:11:03.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:03 smithi100 ceph-mon[39462]: 4.1b scrub starts 2024-09-17T07:11:03.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:03 smithi100 ceph-mon[39462]: 4.1b scrub ok 2024-09-17T07:11:03.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:03 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:11:03.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:03 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1289506520' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:11:03.712 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:11:03.742 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:03.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:03.743 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:11:03.844 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:11:03.883 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:03.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:03.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:11:03.895 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-17T07:11:03.985 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:11:04.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:04.015 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:04.015 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:11:04.020 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-17T07:11:04.116 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:11:04.182 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:04.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:04.184 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:11:04.284 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:11:04.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:04 smithi135 ceph-mon[33238]: 6.19 scrub starts 2024-09-17T07:11:04.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:04 smithi135 ceph-mon[33238]: 6.19 scrub ok 2024-09-17T07:11:04.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:04 smithi135 ceph-mon[33238]: pgmap v394: 148 pgs: 1 active+clean+remapped, 147 active+clean; 3.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.09k op/s; 6.2 MiB/s, 98 objects/s recovering 2024-09-17T07:11:04.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:04 smithi135 ceph-mon[33238]: 5.11 scrub starts 2024-09-17T07:11:04.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:04 smithi135 ceph-mon[33238]: 5.11 scrub ok 2024-09-17T07:11:04.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:04 smithi135 ceph-mon[33238]: 4.3 scrub starts 2024-09-17T07:11:04.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:04 smithi135 ceph-mon[33238]: 4.3 scrub ok 2024-09-17T07:11:04.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:04 smithi135 ceph-mon[33238]: 2.10 scrub starts 2024-09-17T07:11:04.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:04 smithi135 ceph-mon[33238]: 2.10 scrub ok 2024-09-17T07:11:04.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:04 smithi135 ceph-mon[33238]: 6.1a scrub starts 2024-09-17T07:11:04.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:04 smithi135 ceph-mon[33238]: 6.1a scrub ok 2024-09-17T07:11:04.300 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:04 smithi100 ceph-mon[29092]: 6.19 scrub starts 2024-09-17T07:11:04.300 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:04 smithi100 ceph-mon[29092]: 6.19 scrub ok 2024-09-17T07:11:04.300 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:04 smithi100 ceph-mon[29092]: pgmap v394: 148 pgs: 1 active+clean+remapped, 147 active+clean; 3.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.09k op/s; 6.2 MiB/s, 98 objects/s recovering 2024-09-17T07:11:04.300 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:04 smithi100 ceph-mon[29092]: 5.11 scrub starts 2024-09-17T07:11:04.300 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:04 smithi100 ceph-mon[29092]: 5.11 scrub ok 2024-09-17T07:11:04.300 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:04 smithi100 ceph-mon[29092]: 4.3 scrub starts 2024-09-17T07:11:04.300 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:04 smithi100 ceph-mon[29092]: 4.3 scrub ok 2024-09-17T07:11:04.300 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:04 smithi100 ceph-mon[29092]: 2.10 scrub starts 2024-09-17T07:11:04.300 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:04 smithi100 ceph-mon[29092]: 2.10 scrub ok 2024-09-17T07:11:04.300 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:04 smithi100 ceph-mon[29092]: 6.1a scrub starts 2024-09-17T07:11:04.301 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:04 smithi100 ceph-mon[29092]: 6.1a scrub ok 2024-09-17T07:11:04.301 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:04 smithi100 ceph-mon[39462]: 6.19 scrub starts 2024-09-17T07:11:04.301 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:04 smithi100 ceph-mon[39462]: 6.19 scrub ok 2024-09-17T07:11:04.301 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:04 smithi100 ceph-mon[39462]: pgmap v394: 148 pgs: 1 active+clean+remapped, 147 active+clean; 3.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.09k op/s; 6.2 MiB/s, 98 objects/s recovering 2024-09-17T07:11:04.301 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:04 smithi100 ceph-mon[39462]: 5.11 scrub starts 2024-09-17T07:11:04.301 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:04 smithi100 ceph-mon[39462]: 5.11 scrub ok 2024-09-17T07:11:04.301 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:04 smithi100 ceph-mon[39462]: 4.3 scrub starts 2024-09-17T07:11:04.301 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:04 smithi100 ceph-mon[39462]: 4.3 scrub ok 2024-09-17T07:11:04.301 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:04 smithi100 ceph-mon[39462]: 2.10 scrub starts 2024-09-17T07:11:04.301 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:04 smithi100 ceph-mon[39462]: 2.10 scrub ok 2024-09-17T07:11:04.302 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:04 smithi100 ceph-mon[39462]: 6.1a scrub starts 2024-09-17T07:11:04.302 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:04 smithi100 ceph-mon[39462]: 6.1a scrub ok 2024-09-17T07:11:04.322 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:04.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:04.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:11:04.424 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:11:04.453 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:04.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:04.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:11:04.555 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:11:04.589 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:04.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:04.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:11:04.691 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:11:04.724 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:11:04.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:04.744 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:04.744 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:11:04.782 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:11:04.845 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:11:04.881 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:04.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:04.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:11:04.983 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:11:05.018 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:05.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:05.019 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:11:05.119 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:11:05.149 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:05.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:05.150 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:11:05.222 INFO:teuthology.orchestra.run.smithi100.stderr:reweighted osd.2 to 1 (10000) 2024-09-17T07:11:05.251 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:11:05.290 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:05.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:05.291 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:11:05.392 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:11:05.425 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:05.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:05.426 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:11:05.528 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:11:05.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:05 smithi135 ceph-mon[33238]: 5.13 scrub starts 2024-09-17T07:11:05.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:05 smithi135 ceph-mon[33238]: 5.13 scrub ok 2024-09-17T07:11:05.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:05 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/3728604346' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-17T07:11:05.559 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:05.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:05.560 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:11:05.590 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:05 smithi100 ceph-mon[29092]: 5.13 scrub starts 2024-09-17T07:11:05.590 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:05 smithi100 ceph-mon[29092]: 5.13 scrub ok 2024-09-17T07:11:05.591 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:05 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/3728604346' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-17T07:11:05.591 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:05 smithi100 ceph-mon[39462]: 5.13 scrub starts 2024-09-17T07:11:05.591 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:05 smithi100 ceph-mon[39462]: 5.13 scrub ok 2024-09-17T07:11:05.591 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:05 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/3728604346' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-17T07:11:05.662 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:11:05.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:05.702 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:05.702 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:11:05.803 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:11:05.832 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:05.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:05.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:11:05.934 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:11:05.967 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:05.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:05.968 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:11:06.069 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:11:06.083 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:11:06.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:06.109 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:06.110 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:11:06.184 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:11:05 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:11:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:11:06.211 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:11:06.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:06.252 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:06.252 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:11:06.353 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:11:06.383 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:06.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:06.384 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:11:06.439 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:06 smithi100 ceph-mon[29092]: pgmap v395: 148 pgs: 148 active+clean; 2.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.14k op/s 2024-09-17T07:11:06.439 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:06 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/3728604346' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-17T07:11:06.439 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:06 smithi100 ceph-mon[29092]: osdmap e222: 8 total, 8 up, 7 in 2024-09-17T07:11:06.439 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:06 smithi100 ceph-mon[29092]: 2.f scrub starts 2024-09-17T07:11:06.439 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:06 smithi100 ceph-mon[29092]: 2.f scrub ok 2024-09-17T07:11:06.439 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:06 smithi100 ceph-mon[39462]: pgmap v395: 148 pgs: 148 active+clean; 2.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.14k op/s 2024-09-17T07:11:06.440 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:06 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/3728604346' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-17T07:11:06.440 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:06 smithi100 ceph-mon[39462]: osdmap e222: 8 total, 8 up, 7 in 2024-09-17T07:11:06.440 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:06 smithi100 ceph-mon[39462]: 2.f scrub starts 2024-09-17T07:11:06.440 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:06 smithi100 ceph-mon[39462]: 2.f scrub ok 2024-09-17T07:11:06.485 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:11:06.520 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:06.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:06.521 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:11:06.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:06 smithi135 ceph-mon[33238]: pgmap v395: 148 pgs: 148 active+clean; 2.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.14k op/s 2024-09-17T07:11:06.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:06 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/3728604346' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-17T07:11:06.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:06 smithi135 ceph-mon[33238]: osdmap e222: 8 total, 8 up, 7 in 2024-09-17T07:11:06.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:06 smithi135 ceph-mon[33238]: 2.f scrub starts 2024-09-17T07:11:06.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:06 smithi135 ceph-mon[33238]: 2.f scrub ok 2024-09-17T07:11:06.622 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:11:06.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:06.655 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:06.655 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:11:06.756 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:11:06.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:06.800 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:06.800 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:11:06.901 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:11:06.946 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:06.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:06.947 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:11:07.048 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:11:07.078 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:07.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:07.079 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:11:07.180 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:11:07.210 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:07.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:07.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:11:07.312 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:11:07.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:07.346 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:07.346 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:11:07.447 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:11:07.472 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:07.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:07.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:11:07.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:07 smithi135 ceph-mon[33238]: 2.18 scrub starts 2024-09-17T07:11:07.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:07 smithi135 ceph-mon[33238]: 2.18 scrub ok 2024-09-17T07:11:07.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:07 smithi135 ceph-mon[33238]: 8.7 scrub starts 2024-09-17T07:11:07.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:07 smithi135 ceph-mon[33238]: 6.18 scrub starts 2024-09-17T07:11:07.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:07 smithi135 ceph-mon[33238]: 6.18 scrub ok 2024-09-17T07:11:07.574 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:11:07.605 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:07.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:07.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:11:07.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:07 smithi100 ceph-mon[29092]: 2.18 scrub starts 2024-09-17T07:11:07.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:07 smithi100 ceph-mon[29092]: 2.18 scrub ok 2024-09-17T07:11:07.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:07 smithi100 ceph-mon[29092]: 8.7 scrub starts 2024-09-17T07:11:07.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:07 smithi100 ceph-mon[29092]: 6.18 scrub starts 2024-09-17T07:11:07.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:07 smithi100 ceph-mon[29092]: 6.18 scrub ok 2024-09-17T07:11:07.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:07 smithi100 ceph-mon[39462]: 2.18 scrub starts 2024-09-17T07:11:07.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:07 smithi100 ceph-mon[39462]: 2.18 scrub ok 2024-09-17T07:11:07.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:07 smithi100 ceph-mon[39462]: 8.7 scrub starts 2024-09-17T07:11:07.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:07 smithi100 ceph-mon[39462]: 6.18 scrub starts 2024-09-17T07:11:07.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:07 smithi100 ceph-mon[39462]: 6.18 scrub ok 2024-09-17T07:11:07.707 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:11:07.745 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:07.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:07.746 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:11:07.847 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:11:07.879 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:07.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:07.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:11:07.981 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:11:08.014 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:08.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:08.015 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:11:08.116 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:11:08.138 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:08.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:08.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:11:08.240 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:11:08.274 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:08.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:08.275 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:11:08.376 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:11:08.408 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:08.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:08.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:11:08.510 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:11:08.540 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:08.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:08.541 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:11:08.542 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:08 smithi135 ceph-mon[33238]: pgmap v397: 148 pgs: 148 active+clean; 2.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 731 B/s rd, 0 B/s wr, 800 op/s 2024-09-17T07:11:08.542 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:08 smithi135 ceph-mon[33238]: 8.7 scrub ok 2024-09-17T07:11:08.542 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:08 smithi135 ceph-mon[33238]: 2.6 scrub starts 2024-09-17T07:11:08.542 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:08 smithi135 ceph-mon[33238]: 2.6 scrub ok 2024-09-17T07:11:08.542 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:08 smithi135 ceph-mon[33238]: 5.5 scrub starts 2024-09-17T07:11:08.542 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:08 smithi135 ceph-mon[33238]: 5.5 scrub ok 2024-09-17T07:11:08.542 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:08 smithi135 ceph-mon[33238]: 4.10 scrub starts 2024-09-17T07:11:08.542 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:08 smithi135 ceph-mon[33238]: 4.10 scrub ok 2024-09-17T07:11:08.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:08 smithi100 ceph-mon[29092]: pgmap v397: 148 pgs: 148 active+clean; 2.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 731 B/s rd, 0 B/s wr, 800 op/s 2024-09-17T07:11:08.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:08 smithi100 ceph-mon[29092]: 8.7 scrub ok 2024-09-17T07:11:08.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:08 smithi100 ceph-mon[29092]: 2.6 scrub starts 2024-09-17T07:11:08.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:08 smithi100 ceph-mon[29092]: 2.6 scrub ok 2024-09-17T07:11:08.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:08 smithi100 ceph-mon[29092]: 5.5 scrub starts 2024-09-17T07:11:08.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:08 smithi100 ceph-mon[29092]: 5.5 scrub ok 2024-09-17T07:11:08.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:08 smithi100 ceph-mon[29092]: 4.10 scrub starts 2024-09-17T07:11:08.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:08 smithi100 ceph-mon[29092]: 4.10 scrub ok 2024-09-17T07:11:08.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:08 smithi100 ceph-mon[39462]: pgmap v397: 148 pgs: 148 active+clean; 2.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 731 B/s rd, 0 B/s wr, 800 op/s 2024-09-17T07:11:08.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:08 smithi100 ceph-mon[39462]: 8.7 scrub ok 2024-09-17T07:11:08.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:08 smithi100 ceph-mon[39462]: 2.6 scrub starts 2024-09-17T07:11:08.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:08 smithi100 ceph-mon[39462]: 2.6 scrub ok 2024-09-17T07:11:08.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:08 smithi100 ceph-mon[39462]: 5.5 scrub starts 2024-09-17T07:11:08.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:08 smithi100 ceph-mon[39462]: 5.5 scrub ok 2024-09-17T07:11:08.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:08 smithi100 ceph-mon[39462]: 4.10 scrub starts 2024-09-17T07:11:08.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:08 smithi100 ceph-mon[39462]: 4.10 scrub ok 2024-09-17T07:11:08.642 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:11:08.683 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:08.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:08.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:11:08.785 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:11:08.793 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:11:08 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:11:08.815 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:08.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:08.816 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:11:08.917 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:11:08.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:08.958 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:08.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:11:09.059 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:11:09.089 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:09.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:09.090 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:11:09.191 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:11:09.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:09.233 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:09.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:11:09.334 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:11:09.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:09.371 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:09.371 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:11:09.472 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:11:09.510 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:09.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:09.511 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:11:09.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:09 smithi135 ceph-mon[33238]: 5.15 scrub starts 2024-09-17T07:11:09.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:09 smithi135 ceph-mon[33238]: 5.15 scrub ok 2024-09-17T07:11:09.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:09 smithi135 ceph-mon[33238]: 6.14 scrub starts 2024-09-17T07:11:09.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:09 smithi135 ceph-mon[33238]: 6.14 scrub ok 2024-09-17T07:11:09.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:09 smithi135 ceph-mon[33238]: 5.a scrub starts 2024-09-17T07:11:09.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:09 smithi135 ceph-mon[33238]: 5.a scrub ok 2024-09-17T07:11:09.612 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:11:09.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:09 smithi100 ceph-mon[29092]: 5.15 scrub starts 2024-09-17T07:11:09.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:09 smithi100 ceph-mon[29092]: 5.15 scrub ok 2024-09-17T07:11:09.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:09 smithi100 ceph-mon[29092]: 6.14 scrub starts 2024-09-17T07:11:09.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:09 smithi100 ceph-mon[29092]: 6.14 scrub ok 2024-09-17T07:11:09.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:09 smithi100 ceph-mon[29092]: 5.a scrub starts 2024-09-17T07:11:09.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:09 smithi100 ceph-mon[29092]: 5.a scrub ok 2024-09-17T07:11:09.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:09 smithi100 ceph-mon[39462]: 5.15 scrub starts 2024-09-17T07:11:09.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:09 smithi100 ceph-mon[39462]: 5.15 scrub ok 2024-09-17T07:11:09.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:09 smithi100 ceph-mon[39462]: 6.14 scrub starts 2024-09-17T07:11:09.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:09 smithi100 ceph-mon[39462]: 6.14 scrub ok 2024-09-17T07:11:09.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:09 smithi100 ceph-mon[39462]: 5.a scrub starts 2024-09-17T07:11:09.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:09 smithi100 ceph-mon[39462]: 5.a scrub ok 2024-09-17T07:11:09.648 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:09.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:09.649 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:11:09.749 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:11:09.787 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:09.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:09.788 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:11:09.889 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:11:09.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:09.929 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:09.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:11:10.030 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:11:10.074 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:10.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:10.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:11:10.176 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:11:10.215 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:10.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:10.216 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:11:10.317 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:11:10.347 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:10.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:10.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:11:10.449 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:11:10.480 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:10.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:10.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:11:10.524 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:10 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:11:10.524 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:10 smithi100 ceph-mon[29092]: pgmap v398: 148 pgs: 148 active+clean; 2.4 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s 2024-09-17T07:11:10.524 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:10 smithi100 ceph-mon[29092]: 2.11 scrub starts 2024-09-17T07:11:10.524 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:10 smithi100 ceph-mon[29092]: 2.11 scrub ok 2024-09-17T07:11:10.524 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:10 smithi100 ceph-mon[29092]: 6.1e scrub starts 2024-09-17T07:11:10.524 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:10 smithi100 ceph-mon[29092]: 6.1e scrub ok 2024-09-17T07:11:10.525 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:10 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:11:10.525 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:10 smithi100 ceph-mon[39462]: pgmap v398: 148 pgs: 148 active+clean; 2.4 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s 2024-09-17T07:11:10.525 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:10 smithi100 ceph-mon[39462]: 2.11 scrub starts 2024-09-17T07:11:10.525 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:10 smithi100 ceph-mon[39462]: 2.11 scrub ok 2024-09-17T07:11:10.525 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:10 smithi100 ceph-mon[39462]: 6.1e scrub starts 2024-09-17T07:11:10.525 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:10 smithi100 ceph-mon[39462]: 6.1e scrub ok 2024-09-17T07:11:10.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:10 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:11:10.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:10 smithi135 ceph-mon[33238]: pgmap v398: 148 pgs: 148 active+clean; 2.4 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s 2024-09-17T07:11:10.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:10 smithi135 ceph-mon[33238]: 2.11 scrub starts 2024-09-17T07:11:10.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:10 smithi135 ceph-mon[33238]: 2.11 scrub ok 2024-09-17T07:11:10.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:10 smithi135 ceph-mon[33238]: 6.1e scrub starts 2024-09-17T07:11:10.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:10 smithi135 ceph-mon[33238]: 6.1e scrub ok 2024-09-17T07:11:10.582 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:11:10.612 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:10.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:10.613 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:11:10.714 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:11:10.754 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:10.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:10.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:11:10.856 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:11:10.896 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:10.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:10.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:11:10.998 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:11:11.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:11.038 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:11.038 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:11:11.139 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:11:11.173 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:11.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:11.174 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:11:11.275 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:11:11.304 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:11.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:11.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:11:11.406 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:11:11.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:11.428 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:11.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:11:11.529 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:11:11.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:11 smithi135 ceph-mon[33238]: 8.f scrub starts 2024-09-17T07:11:11.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:11 smithi135 ceph-mon[33238]: 8.5 scrub starts 2024-09-17T07:11:11.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:11 smithi135 ceph-mon[33238]: 6.1b scrub starts 2024-09-17T07:11:11.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:11 smithi135 ceph-mon[33238]: 6.1b scrub ok 2024-09-17T07:11:11.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:11 smithi135 ceph-mon[33238]: 8.b scrub starts 2024-09-17T07:11:11.570 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:11.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:11.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:11:11.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:11 smithi100 ceph-mon[29092]: 8.f scrub starts 2024-09-17T07:11:11.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:11 smithi100 ceph-mon[29092]: 8.5 scrub starts 2024-09-17T07:11:11.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:11 smithi100 ceph-mon[29092]: 6.1b scrub starts 2024-09-17T07:11:11.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:11 smithi100 ceph-mon[29092]: 6.1b scrub ok 2024-09-17T07:11:11.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:11 smithi100 ceph-mon[29092]: 8.b scrub starts 2024-09-17T07:11:11.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:11 smithi100 ceph-mon[39462]: 8.f scrub starts 2024-09-17T07:11:11.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:11 smithi100 ceph-mon[39462]: 8.5 scrub starts 2024-09-17T07:11:11.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:11 smithi100 ceph-mon[39462]: 6.1b scrub starts 2024-09-17T07:11:11.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:11 smithi100 ceph-mon[39462]: 6.1b scrub ok 2024-09-17T07:11:11.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:11 smithi100 ceph-mon[39462]: 8.b scrub starts 2024-09-17T07:11:11.672 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:11:11.703 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:11.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:11.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:11:11.804 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:11:11.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:11.835 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:11.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:11:11.936 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:11:11.968 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:11.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:11.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:11:12.012 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd reweight 4 1 2024-09-17T07:11:12.071 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:11:12.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:12.106 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:12.106 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:11:12.207 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:11:12.256 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:12.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:12.257 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:11:12.285 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:12 smithi100 ceph-mon[29092]: 8.f scrub ok 2024-09-17T07:11:12.285 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:12 smithi100 ceph-mon[29092]: 8.5 scrub ok 2024-09-17T07:11:12.285 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:12 smithi100 ceph-mon[29092]: 8.a scrub starts 2024-09-17T07:11:12.285 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:12 smithi100 ceph-mon[29092]: pgmap v399: 148 pgs: 148 active+clean; 2.3 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.25k op/s 2024-09-17T07:11:12.285 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:12 smithi100 ceph-mon[29092]: 6.b scrub starts 2024-09-17T07:11:12.285 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:12 smithi100 ceph-mon[29092]: 6.b scrub ok 2024-09-17T07:11:12.285 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:12 smithi100 ceph-mon[29092]: 8.b scrub ok 2024-09-17T07:11:12.357 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:11:12.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:12.395 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:12.396 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:11:12.497 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:11:12.534 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:12.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:12.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:11:12.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:12 smithi135 ceph-mon[33238]: 8.f scrub ok 2024-09-17T07:11:12.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:12 smithi135 ceph-mon[33238]: 8.5 scrub ok 2024-09-17T07:11:12.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:12 smithi135 ceph-mon[33238]: 8.a scrub starts 2024-09-17T07:11:12.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:12 smithi135 ceph-mon[33238]: pgmap v399: 148 pgs: 148 active+clean; 2.3 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.25k op/s 2024-09-17T07:11:12.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:12 smithi135 ceph-mon[33238]: 6.b scrub starts 2024-09-17T07:11:12.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:12 smithi135 ceph-mon[33238]: 6.b scrub ok 2024-09-17T07:11:12.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:12 smithi135 ceph-mon[33238]: 8.b scrub ok 2024-09-17T07:11:12.569 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:12 smithi100 ceph-mon[39462]: 8.f scrub ok 2024-09-17T07:11:12.570 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:12 smithi100 ceph-mon[39462]: 8.5 scrub ok 2024-09-17T07:11:12.570 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:12 smithi100 ceph-mon[39462]: 8.a scrub starts 2024-09-17T07:11:12.570 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:12 smithi100 ceph-mon[39462]: pgmap v399: 148 pgs: 148 active+clean; 2.3 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.25k op/s 2024-09-17T07:11:12.570 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:12 smithi100 ceph-mon[39462]: 6.b scrub starts 2024-09-17T07:11:12.570 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:12 smithi100 ceph-mon[39462]: 6.b scrub ok 2024-09-17T07:11:12.570 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:12 smithi100 ceph-mon[39462]: 8.b scrub ok 2024-09-17T07:11:12.635 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:11:12.673 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:12.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:12.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:11:12.775 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:11:12.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:12.804 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:12.804 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:11:12.805 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-17T07:11:12.805 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:11:12.906 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:11:12.939 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:12.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:12.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:11:13.041 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:11:13.080 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:13.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:13.082 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:11:13.183 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:11:13.217 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:13.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:13.218 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:11:13.319 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:11:13.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:13.344 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:13.344 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:11:13.447 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:11:13.487 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:13.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:13.488 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:11:13.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:13 smithi135 ceph-mon[33238]: 8.a scrub ok 2024-09-17T07:11:13.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:13 smithi135 ceph-mon[33238]: 8.0 scrub starts 2024-09-17T07:11:13.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:13 smithi135 ceph-mon[33238]: 5.1f scrub starts 2024-09-17T07:11:13.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:13 smithi135 ceph-mon[33238]: 5.1f scrub ok 2024-09-17T07:11:13.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:13 smithi135 ceph-mon[33238]: 8.0 scrub ok 2024-09-17T07:11:13.573 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:13 smithi100 ceph-mon[29092]: 8.a scrub ok 2024-09-17T07:11:13.573 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:13 smithi100 ceph-mon[29092]: 8.0 scrub starts 2024-09-17T07:11:13.573 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:13 smithi100 ceph-mon[29092]: 5.1f scrub starts 2024-09-17T07:11:13.573 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:13 smithi100 ceph-mon[29092]: 5.1f scrub ok 2024-09-17T07:11:13.573 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:13 smithi100 ceph-mon[29092]: 8.0 scrub ok 2024-09-17T07:11:13.573 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:13 smithi100 ceph-mon[39462]: 8.a scrub ok 2024-09-17T07:11:13.574 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:13 smithi100 ceph-mon[39462]: 8.0 scrub starts 2024-09-17T07:11:13.574 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:13 smithi100 ceph-mon[39462]: 5.1f scrub starts 2024-09-17T07:11:13.574 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:13 smithi100 ceph-mon[39462]: 5.1f scrub ok 2024-09-17T07:11:13.574 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:13 smithi100 ceph-mon[39462]: 8.0 scrub ok 2024-09-17T07:11:13.588 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:11:13.606 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-17T07:11:13.625 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:13.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:13.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:11:13.727 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:11:13.754 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:13.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:13.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:11:13.841 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:11:13.856 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:11:13.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:13.883 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:13.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:11:13.984 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:11:14.017 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:14.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:14.018 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:11:14.119 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:11:14.156 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:14.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:14.157 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:11:14.257 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:11:14.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:14.294 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:14.295 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:11:14.307 INFO:teuthology.orchestra.run.smithi100.stderr:nodeep-scrub is unset 2024-09-17T07:11:14.396 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:11:14.430 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:14.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:14.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:11:14.532 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:11:14.563 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:14.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:14.564 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:11:14.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:14 smithi100 ceph-mon[29092]: pgmap v400: 148 pgs: 148 active+clean; 2.3 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.24k op/s 2024-09-17T07:11:14.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:14 smithi100 ceph-mon[29092]: 8.6 scrub starts 2024-09-17T07:11:14.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:14 smithi100 ceph-mon[29092]: 2.b scrub starts 2024-09-17T07:11:14.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:14 smithi100 ceph-mon[29092]: 2.b scrub ok 2024-09-17T07:11:14.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:14 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/3185506309' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:11:14.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:14 smithi100 ceph-mon[39462]: pgmap v400: 148 pgs: 148 active+clean; 2.3 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.24k op/s 2024-09-17T07:11:14.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:14 smithi100 ceph-mon[39462]: 8.6 scrub starts 2024-09-17T07:11:14.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:14 smithi100 ceph-mon[39462]: 2.b scrub starts 2024-09-17T07:11:14.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:14 smithi100 ceph-mon[39462]: 2.b scrub ok 2024-09-17T07:11:14.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:14 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/3185506309' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:11:14.666 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:11:14.711 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:14.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:14.712 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:11:14.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:14 smithi135 ceph-mon[33238]: pgmap v400: 148 pgs: 148 active+clean; 2.3 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.24k op/s 2024-09-17T07:11:14.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:14 smithi135 ceph-mon[33238]: 8.6 scrub starts 2024-09-17T07:11:14.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:14 smithi135 ceph-mon[33238]: 2.b scrub starts 2024-09-17T07:11:14.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:14 smithi135 ceph-mon[33238]: 2.b scrub ok 2024-09-17T07:11:14.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:14 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/3185506309' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:11:14.812 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:11:14.849 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:14.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:14.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:11:14.951 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:11:14.977 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:14.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:14.978 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:11:15.079 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:11:15.102 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:15.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:15.103 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:11:15.204 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:11:15.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:15.239 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:15.239 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:11:15.340 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:11:15.370 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:15.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:15.371 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:11:15.472 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:11:15.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:15.509 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:15.509 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:11:15.610 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:11:15.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:15 smithi100 ceph-mon[29092]: 8.6 scrub ok 2024-09-17T07:11:15.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:15 smithi100 ceph-mon[29092]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T07:11:15.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:15 smithi100 ceph-mon[29092]: Cluster is now healthy 2024-09-17T07:11:15.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:15 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/3185506309' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T07:11:15.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:15 smithi100 ceph-mon[29092]: osdmap e223: 8 total, 8 up, 7 in 2024-09-17T07:11:15.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:15 smithi100 ceph-mon[29092]: 5.1c scrub starts 2024-09-17T07:11:15.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:15 smithi100 ceph-mon[29092]: 5.1c scrub ok 2024-09-17T07:11:15.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:15 smithi100 ceph-mon[29092]: pgmap v402: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.9 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.32k op/s 2024-09-17T07:11:15.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:15 smithi100 ceph-mon[39462]: 8.6 scrub ok 2024-09-17T07:11:15.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:15 smithi100 ceph-mon[39462]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T07:11:15.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:15 smithi100 ceph-mon[39462]: Cluster is now healthy 2024-09-17T07:11:15.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:15 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/3185506309' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T07:11:15.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:15 smithi100 ceph-mon[39462]: osdmap e223: 8 total, 8 up, 7 in 2024-09-17T07:11:15.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:15 smithi100 ceph-mon[39462]: 5.1c scrub starts 2024-09-17T07:11:15.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:15 smithi100 ceph-mon[39462]: 5.1c scrub ok 2024-09-17T07:11:15.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:15 smithi100 ceph-mon[39462]: pgmap v402: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.9 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.32k op/s 2024-09-17T07:11:15.640 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:15.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:15.641 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:11:15.741 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:11:15.775 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:15.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:15.776 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:11:15.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:15 smithi135 ceph-mon[33238]: 8.6 scrub ok 2024-09-17T07:11:15.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:15 smithi135 ceph-mon[33238]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T07:11:15.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:15 smithi135 ceph-mon[33238]: Cluster is now healthy 2024-09-17T07:11:15.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:15 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/3185506309' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T07:11:15.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:15 smithi135 ceph-mon[33238]: osdmap e223: 8 total, 8 up, 7 in 2024-09-17T07:11:15.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:15 smithi135 ceph-mon[33238]: 5.1c scrub starts 2024-09-17T07:11:15.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:15 smithi135 ceph-mon[33238]: 5.1c scrub ok 2024-09-17T07:11:15.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:15 smithi135 ceph-mon[33238]: pgmap v402: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.9 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.32k op/s 2024-09-17T07:11:15.876 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:11:15.908 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:15.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:15.914 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:11:16.015 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:11:16.053 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:16.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:16.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:11:16.126 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:11:15 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:11:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:11:16.155 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:11:16.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:16.189 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:16.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:11:16.290 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:11:16.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:16.322 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:16.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:11:16.423 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:11:16.467 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:16.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:16.469 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:11:16.569 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:11:16.599 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:16.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:16.601 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:11:16.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:16 smithi100 ceph-mon[29092]: 2.17 scrub starts 2024-09-17T07:11:16.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:16 smithi100 ceph-mon[29092]: 2.17 scrub ok 2024-09-17T07:11:16.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:16 smithi100 ceph-mon[29092]: 2.5 scrub starts 2024-09-17T07:11:16.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:16 smithi100 ceph-mon[29092]: 2.5 scrub ok 2024-09-17T07:11:16.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:16 smithi100 ceph-mon[39462]: 2.17 scrub starts 2024-09-17T07:11:16.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:16 smithi100 ceph-mon[39462]: 2.17 scrub ok 2024-09-17T07:11:16.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:16 smithi100 ceph-mon[39462]: 2.5 scrub starts 2024-09-17T07:11:16.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:16 smithi100 ceph-mon[39462]: 2.5 scrub ok 2024-09-17T07:11:16.672 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:11:16.678 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:11:16.702 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:11:16.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:16.742 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:16.743 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:11:16.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:16 smithi135 ceph-mon[33238]: 2.17 scrub starts 2024-09-17T07:11:16.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:16 smithi135 ceph-mon[33238]: 2.17 scrub ok 2024-09-17T07:11:16.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:16 smithi135 ceph-mon[33238]: 2.5 scrub starts 2024-09-17T07:11:16.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:16 smithi135 ceph-mon[33238]: 2.5 scrub ok 2024-09-17T07:11:16.845 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:11:16.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:16.880 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:16.880 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:11:16.981 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:11:17.010 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:17.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:17.011 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:11:17.112 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:11:17.145 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:17.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:17.146 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:11:17.247 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:11:17.281 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:17.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:17.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:11:17.383 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:11:17.397 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:17 smithi100 ceph-mon[39462]: 4.5 scrub starts 2024-09-17T07:11:17.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:17 smithi100 ceph-mon[39462]: 4.5 scrub ok 2024-09-17T07:11:17.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:17 smithi100 ceph-mon[39462]: 5.16 scrub starts 2024-09-17T07:11:17.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:17 smithi100 ceph-mon[39462]: 5.16 scrub ok 2024-09-17T07:11:17.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:17 smithi100 ceph-mon[39462]: 2.8 scrub starts 2024-09-17T07:11:17.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:17 smithi100 ceph-mon[39462]: 2.8 scrub ok 2024-09-17T07:11:17.398 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:17 smithi100 ceph-mon[39462]: pgmap v403: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.9 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.31k op/s 2024-09-17T07:11:17.398 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:17 smithi100 ceph-mon[29092]: 4.5 scrub starts 2024-09-17T07:11:17.398 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:17 smithi100 ceph-mon[29092]: 4.5 scrub ok 2024-09-17T07:11:17.399 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:17 smithi100 ceph-mon[29092]: 5.16 scrub starts 2024-09-17T07:11:17.399 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:17 smithi100 ceph-mon[29092]: 5.16 scrub ok 2024-09-17T07:11:17.399 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:17 smithi100 ceph-mon[29092]: 2.8 scrub starts 2024-09-17T07:11:17.399 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:17 smithi100 ceph-mon[29092]: 2.8 scrub ok 2024-09-17T07:11:17.399 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:17 smithi100 ceph-mon[29092]: pgmap v403: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.9 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.31k op/s 2024-09-17T07:11:17.417 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:17.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:17.418 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:11:17.518 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:11:17.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:17.548 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:17.548 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:11:17.649 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:11:17.683 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:17.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:17.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:11:17.785 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:11:17.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:17 smithi135 ceph-mon[33238]: 4.5 scrub starts 2024-09-17T07:11:17.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:17 smithi135 ceph-mon[33238]: 4.5 scrub ok 2024-09-17T07:11:17.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:17 smithi135 ceph-mon[33238]: 5.16 scrub starts 2024-09-17T07:11:17.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:17 smithi135 ceph-mon[33238]: 5.16 scrub ok 2024-09-17T07:11:17.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:17 smithi135 ceph-mon[33238]: 2.8 scrub starts 2024-09-17T07:11:17.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:17 smithi135 ceph-mon[33238]: 2.8 scrub ok 2024-09-17T07:11:17.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:17 smithi135 ceph-mon[33238]: pgmap v403: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.9 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.31k op/s 2024-09-17T07:11:17.820 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:17.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:17.821 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:11:17.922 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:11:17.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:17.960 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:17.961 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:11:18.061 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:11:18.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:18.097 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:18.098 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:11:18.198 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:11:18.238 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:18.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:18.240 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:11:18.341 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:11:18.372 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:18.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:18.374 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:11:18.475 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:11:18.506 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:18.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:18.507 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:11:18.553 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:18 smithi100 ceph-mon[29092]: 4.a scrub starts 2024-09-17T07:11:18.553 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:18 smithi100 ceph-mon[29092]: 4.a scrub ok 2024-09-17T07:11:18.553 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:18 smithi100 ceph-mon[29092]: 8.4 scrub starts 2024-09-17T07:11:18.553 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:18 smithi100 ceph-mon[29092]: 4.7 scrub starts 2024-09-17T07:11:18.553 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:18 smithi100 ceph-mon[29092]: 4.7 scrub ok 2024-09-17T07:11:18.553 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:18 smithi100 ceph-mon[29092]: 6.1c scrub starts 2024-09-17T07:11:18.553 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:18 smithi100 ceph-mon[29092]: 6.1c scrub ok 2024-09-17T07:11:18.553 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:18 smithi100 ceph-mon[29092]: 5.8 deep-scrub starts 2024-09-17T07:11:18.553 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:18 smithi100 ceph-mon[29092]: 5.8 deep-scrub ok 2024-09-17T07:11:18.553 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:18 smithi100 ceph-mon[29092]: 8.4 scrub ok 2024-09-17T07:11:18.554 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:18 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:11:18.554 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:18 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:11:18.554 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:18 smithi100 ceph-mon[39462]: 4.a scrub starts 2024-09-17T07:11:18.554 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:18 smithi100 ceph-mon[39462]: 4.a scrub ok 2024-09-17T07:11:18.554 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:18 smithi100 ceph-mon[39462]: 8.4 scrub starts 2024-09-17T07:11:18.554 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:18 smithi100 ceph-mon[39462]: 4.7 scrub starts 2024-09-17T07:11:18.554 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:18 smithi100 ceph-mon[39462]: 4.7 scrub ok 2024-09-17T07:11:18.554 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:18 smithi100 ceph-mon[39462]: 6.1c scrub starts 2024-09-17T07:11:18.555 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:18 smithi100 ceph-mon[39462]: 6.1c scrub ok 2024-09-17T07:11:18.555 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:18 smithi100 ceph-mon[39462]: 5.8 deep-scrub starts 2024-09-17T07:11:18.555 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:18 smithi100 ceph-mon[39462]: 5.8 deep-scrub ok 2024-09-17T07:11:18.555 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:18 smithi100 ceph-mon[39462]: 8.4 scrub ok 2024-09-17T07:11:18.555 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:18 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:11:18.556 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:18 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:11:18.608 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:11:18.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:18.645 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:18.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:11:18.704 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-17T07:11:18.746 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:11:18.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:18.770 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:18.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:11:18.793 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:11:18 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:11:18.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:18 smithi135 ceph-mon[33238]: 4.a scrub starts 2024-09-17T07:11:18.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:18 smithi135 ceph-mon[33238]: 4.a scrub ok 2024-09-17T07:11:18.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:18 smithi135 ceph-mon[33238]: 8.4 scrub starts 2024-09-17T07:11:18.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:18 smithi135 ceph-mon[33238]: 4.7 scrub starts 2024-09-17T07:11:18.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:18 smithi135 ceph-mon[33238]: 4.7 scrub ok 2024-09-17T07:11:18.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:18 smithi135 ceph-mon[33238]: 6.1c scrub starts 2024-09-17T07:11:18.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:18 smithi135 ceph-mon[33238]: 6.1c scrub ok 2024-09-17T07:11:18.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:18 smithi135 ceph-mon[33238]: 5.8 deep-scrub starts 2024-09-17T07:11:18.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:18 smithi135 ceph-mon[33238]: 5.8 deep-scrub ok 2024-09-17T07:11:18.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:18 smithi135 ceph-mon[33238]: 8.4 scrub ok 2024-09-17T07:11:18.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:18 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:11:18.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:18 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:11:18.871 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:11:18.906 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:18.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:18.907 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:11:19.008 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:11:19.049 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:19.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:19.053 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:11:19.153 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:11:19.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:19.215 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:19.216 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:11:19.316 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:11:19.350 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:19.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:19.352 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:11:19.452 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:11:19.486 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:19.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:19.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:11:19.588 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:11:19.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:19.621 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:19.621 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:11:19.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:19 smithi100 ceph-mon[29092]: 5.2 deep-scrub starts 2024-09-17T07:11:19.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:19 smithi100 ceph-mon[29092]: 5.2 deep-scrub ok 2024-09-17T07:11:19.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:19 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:11:19.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:19 smithi100 ceph-mon[29092]: pgmap v404: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.6 GiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.31k op/s 2024-09-17T07:11:19.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:19 smithi100 ceph-mon[39462]: 5.2 deep-scrub starts 2024-09-17T07:11:19.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:19 smithi100 ceph-mon[39462]: 5.2 deep-scrub ok 2024-09-17T07:11:19.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:19 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:11:19.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:19 smithi100 ceph-mon[39462]: pgmap v404: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.6 GiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.31k op/s 2024-09-17T07:11:19.722 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:11:19.762 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:19.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:19.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:11:19.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:19 smithi135 ceph-mon[33238]: 5.2 deep-scrub starts 2024-09-17T07:11:19.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:19 smithi135 ceph-mon[33238]: 5.2 deep-scrub ok 2024-09-17T07:11:19.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:19 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:11:19.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:19 smithi135 ceph-mon[33238]: pgmap v404: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.6 GiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.31k op/s 2024-09-17T07:11:19.816 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:11:19.864 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:11:19.898 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:19.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:19.899 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:11:20.000 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:11:20.040 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:20.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:20.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:11:20.142 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:11:20.183 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:20.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:20.185 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:11:20.285 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:11:20.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:20.322 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:20.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:11:20.423 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:11:20.445 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:20.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:20.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:11:20.547 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:11:20.577 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:20.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:20.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:11:20.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:20 smithi100 ceph-mon[29092]: 2.1a deep-scrub starts 2024-09-17T07:11:20.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:20 smithi100 ceph-mon[29092]: 2.1a deep-scrub ok 2024-09-17T07:11:20.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:20 smithi100 ceph-mon[29092]: 4.0 scrub starts 2024-09-17T07:11:20.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:20 smithi100 ceph-mon[29092]: 4.0 scrub ok 2024-09-17T07:11:20.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:20 smithi100 ceph-mon[29092]: 8.2 scrub starts 2024-09-17T07:11:20.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:20 smithi100 ceph-mon[39462]: 2.1a deep-scrub starts 2024-09-17T07:11:20.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:20 smithi100 ceph-mon[39462]: 2.1a deep-scrub ok 2024-09-17T07:11:20.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:20 smithi100 ceph-mon[39462]: 4.0 scrub starts 2024-09-17T07:11:20.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:20 smithi100 ceph-mon[39462]: 4.0 scrub ok 2024-09-17T07:11:20.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:20 smithi100 ceph-mon[39462]: 8.2 scrub starts 2024-09-17T07:11:20.678 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:11:20.712 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:20.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:20.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:11:20.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:20 smithi135 ceph-mon[33238]: 2.1a deep-scrub starts 2024-09-17T07:11:20.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:20 smithi135 ceph-mon[33238]: 2.1a deep-scrub ok 2024-09-17T07:11:20.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:20 smithi135 ceph-mon[33238]: 4.0 scrub starts 2024-09-17T07:11:20.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:20 smithi135 ceph-mon[33238]: 4.0 scrub ok 2024-09-17T07:11:20.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:20 smithi135 ceph-mon[33238]: 8.2 scrub starts 2024-09-17T07:11:20.814 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:11:20.852 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:20.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:20.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:11:20.954 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:11:20.991 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:20.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:20.992 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:11:21.093 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:11:21.134 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:21.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:21.135 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:11:21.236 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:11:21.269 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:21.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:21.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:11:21.371 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:11:21.410 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:21.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:21.412 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:11:21.468 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:21 smithi100 ceph-mon[29092]: 4.1f scrub starts 2024-09-17T07:11:21.469 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:21 smithi100 ceph-mon[29092]: 4.1f scrub ok 2024-09-17T07:11:21.469 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:21 smithi100 ceph-mon[29092]: 8.d scrub starts 2024-09-17T07:11:21.469 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:21 smithi100 ceph-mon[29092]: 8.2 scrub ok 2024-09-17T07:11:21.469 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:21 smithi100 ceph-mon[29092]: 8.3 scrub starts 2024-09-17T07:11:21.469 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:21 smithi100 ceph-mon[29092]: 4.11 scrub starts 2024-09-17T07:11:21.469 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:21 smithi100 ceph-mon[29092]: 4.11 scrub ok 2024-09-17T07:11:21.469 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:21 smithi100 ceph-mon[29092]: 8.d scrub ok 2024-09-17T07:11:21.469 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:21 smithi100 ceph-mon[29092]: 4.1a scrub starts 2024-09-17T07:11:21.469 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:21 smithi100 ceph-mon[29092]: 4.1a scrub ok 2024-09-17T07:11:21.469 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:21 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/3136855019' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-17T07:11:21.469 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:21 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-17T07:11:21.469 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:21 smithi100 ceph-mon[29092]: pgmap v405: 148 pgs: 148 active+clean; 1.5 GiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.30k op/s 2024-09-17T07:11:21.469 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:21 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1225878391' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:11:21.470 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:21 smithi100 ceph-mon[39462]: 4.1f scrub starts 2024-09-17T07:11:21.470 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:21 smithi100 ceph-mon[39462]: 4.1f scrub ok 2024-09-17T07:11:21.470 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:21 smithi100 ceph-mon[39462]: 8.d scrub starts 2024-09-17T07:11:21.470 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:21 smithi100 ceph-mon[39462]: 8.2 scrub ok 2024-09-17T07:11:21.470 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:21 smithi100 ceph-mon[39462]: 8.3 scrub starts 2024-09-17T07:11:21.470 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:21 smithi100 ceph-mon[39462]: 4.11 scrub starts 2024-09-17T07:11:21.470 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:21 smithi100 ceph-mon[39462]: 4.11 scrub ok 2024-09-17T07:11:21.470 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:21 smithi100 ceph-mon[39462]: 8.d scrub ok 2024-09-17T07:11:21.470 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:21 smithi100 ceph-mon[39462]: 4.1a scrub starts 2024-09-17T07:11:21.471 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:21 smithi100 ceph-mon[39462]: 4.1a scrub ok 2024-09-17T07:11:21.471 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:21 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/3136855019' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-17T07:11:21.471 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:21 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-17T07:11:21.471 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:21 smithi100 ceph-mon[39462]: pgmap v405: 148 pgs: 148 active+clean; 1.5 GiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.30k op/s 2024-09-17T07:11:21.471 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:21 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1225878391' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:11:21.512 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:11:21.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:21.540 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:21.540 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:11:21.641 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:11:21.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:21.687 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:21.687 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:11:21.788 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:11:21.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:21 smithi135 ceph-mon[33238]: 4.1f scrub starts 2024-09-17T07:11:21.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:21 smithi135 ceph-mon[33238]: 4.1f scrub ok 2024-09-17T07:11:21.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:21 smithi135 ceph-mon[33238]: 8.d scrub starts 2024-09-17T07:11:21.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:21 smithi135 ceph-mon[33238]: 8.2 scrub ok 2024-09-17T07:11:21.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:21 smithi135 ceph-mon[33238]: 8.3 scrub starts 2024-09-17T07:11:21.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:21 smithi135 ceph-mon[33238]: 4.11 scrub starts 2024-09-17T07:11:21.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:21 smithi135 ceph-mon[33238]: 4.11 scrub ok 2024-09-17T07:11:21.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:21 smithi135 ceph-mon[33238]: 8.d scrub ok 2024-09-17T07:11:21.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:21 smithi135 ceph-mon[33238]: 4.1a scrub starts 2024-09-17T07:11:21.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:21 smithi135 ceph-mon[33238]: 4.1a scrub ok 2024-09-17T07:11:21.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:21 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/3136855019' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-17T07:11:21.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:21 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-17T07:11:21.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:21 smithi135 ceph-mon[33238]: pgmap v405: 148 pgs: 148 active+clean; 1.5 GiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.30k op/s 2024-09-17T07:11:21.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:21 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1225878391' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:11:21.827 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:21.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:21.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:11:21.930 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:11:21.970 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:21.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:21.971 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:11:22.072 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:11:22.103 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:22.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:22.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:11:22.147 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:11:22.147 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1: {} 2024-09-17T07:11:22.147 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:11:22.147 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:11:22.147 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:11:22.147 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:11:22.147 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:11:22.147 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:11:22.148 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:11:22.148 INFO:teuthology.orchestra.run.smithi100.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:11:22.148 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:11:22.148 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:11:22.148 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:11:22.148 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:11:22.148 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:11:22.148 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:11:22.206 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:11:22.244 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:22.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:22.245 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:11:22.345 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:11:22.383 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:22.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:22.384 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:11:22.485 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:11:22.520 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:22 smithi100 ceph-mon[29092]: 8.3 scrub ok 2024-09-17T07:11:22.520 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:22 smithi100 ceph-mon[29092]: 8.8 scrub starts 2024-09-17T07:11:22.520 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:22 smithi100 ceph-mon[29092]: 4.12 scrub starts 2024-09-17T07:11:22.520 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:22 smithi100 ceph-mon[29092]: 4.12 scrub ok 2024-09-17T07:11:22.520 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:22 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-17T07:11:22.520 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:22 smithi100 ceph-mon[29092]: osdmap e224: 8 total, 8 up, 7 in 2024-09-17T07:11:22.520 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:22 smithi100 ceph-mon[29092]: 8.8 scrub ok 2024-09-17T07:11:22.521 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:22 smithi100 ceph-mon[39462]: 8.3 scrub ok 2024-09-17T07:11:22.521 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:22 smithi100 ceph-mon[39462]: 8.8 scrub starts 2024-09-17T07:11:22.521 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:22 smithi100 ceph-mon[39462]: 4.12 scrub starts 2024-09-17T07:11:22.521 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:22 smithi100 ceph-mon[39462]: 4.12 scrub ok 2024-09-17T07:11:22.521 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:22 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-17T07:11:22.521 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:22 smithi100 ceph-mon[39462]: osdmap e224: 8 total, 8 up, 7 in 2024-09-17T07:11:22.521 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:22 smithi100 ceph-mon[39462]: 8.8 scrub ok 2024-09-17T07:11:22.521 INFO:teuthology.orchestra.run.smithi100.stderr:reweighted osd.4 to 1 (10000) 2024-09-17T07:11:22.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:22.524 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:22.524 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:11:22.625 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:11:22.657 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:22.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:22.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:11:22.759 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:11:22.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:22 smithi135 ceph-mon[33238]: 8.3 scrub ok 2024-09-17T07:11:22.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:22 smithi135 ceph-mon[33238]: 8.8 scrub starts 2024-09-17T07:11:22.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:22 smithi135 ceph-mon[33238]: 4.12 scrub starts 2024-09-17T07:11:22.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:22 smithi135 ceph-mon[33238]: 4.12 scrub ok 2024-09-17T07:11:22.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:22 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-17T07:11:22.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:22 smithi135 ceph-mon[33238]: osdmap e224: 8 total, 8 up, 7 in 2024-09-17T07:11:22.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:22 smithi135 ceph-mon[33238]: 8.8 scrub ok 2024-09-17T07:11:22.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:22.795 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:22.795 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:11:22.896 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:11:22.926 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:22.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:22.927 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:11:23.028 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:11:23.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:23.068 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:23.069 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:11:23.169 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:11:23.204 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:23.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:23.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:11:23.306 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:11:23.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:23.349 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:23.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:11:23.450 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:11:23.488 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:23.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:23.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:11:23.590 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:11:23.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:23.624 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:23.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:11:23.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:23 smithi100 ceph-mon[29092]: pgmap v407: 148 pgs: 148 active+clean; 1.5 GiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 575 B/s rd, 0 B/s wr, 756 op/s 2024-09-17T07:11:23.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:23 smithi100 ceph-mon[39462]: pgmap v407: 148 pgs: 148 active+clean; 1.5 GiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 575 B/s rd, 0 B/s wr, 756 op/s 2024-09-17T07:11:23.726 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:11:23.762 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:23.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:23.763 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:11:23.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:23 smithi135 ceph-mon[33238]: pgmap v407: 148 pgs: 148 active+clean; 1.5 GiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 575 B/s rd, 0 B/s wr, 756 op/s 2024-09-17T07:11:23.864 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:11:23.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:23.894 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:23.894 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:11:23.995 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:11:24.034 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:24.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:24.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:11:24.136 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:11:24.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:24.173 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:24.173 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:11:24.274 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:11:24.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:24.307 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:24.307 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:11:24.408 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:11:24.444 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:24.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:24.445 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:11:24.545 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:11:25.017 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:24 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:11:25.017 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:24 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:11:25.018 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:24 smithi135 ceph-mon[33238]: 6.12 scrub starts 2024-09-17T07:11:25.018 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:24 smithi135 ceph-mon[33238]: 6.12 scrub ok 2024-09-17T07:11:25.018 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:24 smithi135 ceph-mon[33238]: osdmap e225: 8 total, 8 up, 7 in 2024-09-17T07:11:25.018 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:24 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:11:25.018 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:24 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:11:25.018 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:24 smithi100 ceph-mon[29092]: 6.12 scrub starts 2024-09-17T07:11:25.018 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:24 smithi100 ceph-mon[29092]: 6.12 scrub ok 2024-09-17T07:11:25.018 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:24 smithi100 ceph-mon[29092]: osdmap e225: 8 total, 8 up, 7 in 2024-09-17T07:11:25.018 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:24 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:11:25.019 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:24 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:11:25.019 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:24 smithi100 ceph-mon[39462]: 6.12 scrub starts 2024-09-17T07:11:25.019 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:24 smithi100 ceph-mon[39462]: 6.12 scrub ok 2024-09-17T07:11:25.019 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:24 smithi100 ceph-mon[39462]: osdmap e225: 8 total, 8 up, 7 in 2024-09-17T07:11:25.021 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd reweight 5 1 2024-09-17T07:11:25.021 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-17T07:11:25.021 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-17T07:11:25.021 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-17T07:11:25.021 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-17T07:11:25.021 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-17T07:11:25.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:25.102 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:25.102 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:11:25.203 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:11:25.241 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:25.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:25.242 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:11:25.343 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:11:25.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:25.382 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:25.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:11:25.483 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:11:25.519 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:25.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:25.520 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:11:25.621 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:11:25.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:25.665 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:25.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:11:25.766 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:11:25.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:25 smithi135 ceph-mon[33238]: 2.16 scrub starts 2024-09-17T07:11:25.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:25 smithi135 ceph-mon[33238]: 2.16 scrub ok 2024-09-17T07:11:25.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:25 smithi135 ceph-mon[33238]: 5.1a scrub starts 2024-09-17T07:11:25.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:25 smithi135 ceph-mon[33238]: 5.1a scrub ok 2024-09-17T07:11:25.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:25 smithi135 ceph-mon[33238]: pgmap v409: 148 pgs: 148 active+clean; 1.1 GiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.66k op/s 2024-09-17T07:11:25.802 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:25.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:25.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:11:25.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:25 smithi100 ceph-mon[29092]: 2.16 scrub starts 2024-09-17T07:11:25.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:25 smithi100 ceph-mon[29092]: 2.16 scrub ok 2024-09-17T07:11:25.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:25 smithi100 ceph-mon[29092]: 5.1a scrub starts 2024-09-17T07:11:25.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:25 smithi100 ceph-mon[29092]: 5.1a scrub ok 2024-09-17T07:11:25.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:25 smithi100 ceph-mon[29092]: pgmap v409: 148 pgs: 148 active+clean; 1.1 GiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.66k op/s 2024-09-17T07:11:25.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:25 smithi100 ceph-mon[39462]: 2.16 scrub starts 2024-09-17T07:11:25.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:25 smithi100 ceph-mon[39462]: 2.16 scrub ok 2024-09-17T07:11:25.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:25 smithi100 ceph-mon[39462]: 5.1a scrub starts 2024-09-17T07:11:25.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:25 smithi100 ceph-mon[39462]: 5.1a scrub ok 2024-09-17T07:11:25.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:25 smithi100 ceph-mon[39462]: pgmap v409: 148 pgs: 148 active+clean; 1.1 GiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.66k op/s 2024-09-17T07:11:25.904 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:11:25.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:25.935 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:25.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:11:26.036 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:11:26.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:26.079 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:26.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:11:26.180 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:11:26.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:26.213 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:26.214 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:11:26.314 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:11:26.347 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:26.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:26.348 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:11:26.376 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:11:25 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:11:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:11:26.449 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:11:26.483 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:26.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:26.484 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:11:26.585 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:11:26.614 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:26.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:26.615 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:11:26.716 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:11:26.755 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:26.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:26.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:11:26.786 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:26 smithi100 ceph-mon[29092]: 4.c scrub starts 2024-09-17T07:11:26.786 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:26 smithi100 ceph-mon[29092]: 4.c scrub ok 2024-09-17T07:11:26.786 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:26 smithi100 ceph-mon[29092]: 5.9 deep-scrub starts 2024-09-17T07:11:26.786 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:26 smithi100 ceph-mon[29092]: 5.9 deep-scrub ok 2024-09-17T07:11:26.786 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:26 smithi100 ceph-mon[39462]: 4.c scrub starts 2024-09-17T07:11:26.786 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:26 smithi100 ceph-mon[39462]: 4.c scrub ok 2024-09-17T07:11:26.786 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:26 smithi100 ceph-mon[39462]: 5.9 deep-scrub starts 2024-09-17T07:11:26.786 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:26 smithi100 ceph-mon[39462]: 5.9 deep-scrub ok 2024-09-17T07:11:26.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:26 smithi135 ceph-mon[33238]: 4.c scrub starts 2024-09-17T07:11:26.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:26 smithi135 ceph-mon[33238]: 4.c scrub ok 2024-09-17T07:11:26.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:26 smithi135 ceph-mon[33238]: 5.9 deep-scrub starts 2024-09-17T07:11:26.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:26 smithi135 ceph-mon[33238]: 5.9 deep-scrub ok 2024-09-17T07:11:26.820 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-17T07:11:26.857 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:11:26.888 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:26.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:26.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:11:26.989 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:11:27.020 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:11:27.030 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:27.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:27.031 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:11:27.131 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:11:27.159 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:27.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:27.160 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:11:27.262 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:11:27.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:27.301 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:27.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:11:27.402 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:11:27.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:27.443 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:27.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:11:27.544 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:11:27.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:27.578 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:27.578 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:11:27.679 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:11:27.715 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:27.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:27.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:11:27.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:27 smithi135 ceph-mon[33238]: 5.e scrub starts 2024-09-17T07:11:27.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:27 smithi135 ceph-mon[33238]: 5.e scrub ok 2024-09-17T07:11:27.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:27 smithi135 ceph-mon[33238]: 5.10 scrub starts 2024-09-17T07:11:27.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:27 smithi135 ceph-mon[33238]: 5.10 scrub ok 2024-09-17T07:11:27.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:27 smithi135 ceph-mon[33238]: 5.1 scrub starts 2024-09-17T07:11:27.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:27 smithi135 ceph-mon[33238]: 5.1 scrub ok 2024-09-17T07:11:27.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:27 smithi135 ceph-mon[33238]: pgmap v410: 148 pgs: 148 active+clean; 1.1 GiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 977 op/s 2024-09-17T07:11:27.817 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:11:27.846 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:27 smithi100 ceph-mon[29092]: 5.e scrub starts 2024-09-17T07:11:27.846 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:27 smithi100 ceph-mon[29092]: 5.e scrub ok 2024-09-17T07:11:27.846 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:27 smithi100 ceph-mon[29092]: 5.10 scrub starts 2024-09-17T07:11:27.846 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:27 smithi100 ceph-mon[29092]: 5.10 scrub ok 2024-09-17T07:11:27.846 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:27 smithi100 ceph-mon[29092]: 5.1 scrub starts 2024-09-17T07:11:27.846 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:27 smithi100 ceph-mon[29092]: 5.1 scrub ok 2024-09-17T07:11:27.847 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:27 smithi100 ceph-mon[29092]: pgmap v410: 148 pgs: 148 active+clean; 1.1 GiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 977 op/s 2024-09-17T07:11:27.847 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:27 smithi100 ceph-mon[39462]: 5.e scrub starts 2024-09-17T07:11:27.847 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:27 smithi100 ceph-mon[39462]: 5.e scrub ok 2024-09-17T07:11:27.847 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:27 smithi100 ceph-mon[39462]: 5.10 scrub starts 2024-09-17T07:11:27.847 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:27 smithi100 ceph-mon[39462]: 5.10 scrub ok 2024-09-17T07:11:27.847 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:27 smithi100 ceph-mon[39462]: 5.1 scrub starts 2024-09-17T07:11:27.847 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:27 smithi100 ceph-mon[39462]: 5.1 scrub ok 2024-09-17T07:11:27.847 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:27 smithi100 ceph-mon[39462]: pgmap v410: 148 pgs: 148 active+clean; 1.1 GiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 977 op/s 2024-09-17T07:11:27.854 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:27.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:27.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:11:27.956 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:11:27.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:27.990 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:27.990 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:11:28.091 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:11:28.122 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:28.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:28.123 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:11:28.223 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:11:28.262 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:28.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:28.263 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:11:28.364 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:11:28.402 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:28.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:28.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:11:28.504 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:11:28.533 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:28.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:28.534 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:11:28.604 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:11:28.606 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:11:28.635 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:11:28.673 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:28.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:28.675 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:11:28.691 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:28 smithi100 ceph-mon[29092]: 5.1b scrub starts 2024-09-17T07:11:28.691 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:28 smithi100 ceph-mon[29092]: 5.1b scrub ok 2024-09-17T07:11:28.691 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:28 smithi100 ceph-mon[29092]: 4.e scrub starts 2024-09-17T07:11:28.691 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:28 smithi100 ceph-mon[29092]: 4.e scrub ok 2024-09-17T07:11:28.692 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:28 smithi100 ceph-mon[39462]: 5.1b scrub starts 2024-09-17T07:11:28.692 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:28 smithi100 ceph-mon[39462]: 5.1b scrub ok 2024-09-17T07:11:28.692 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:28 smithi100 ceph-mon[39462]: 4.e scrub starts 2024-09-17T07:11:28.692 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:28 smithi100 ceph-mon[39462]: 4.e scrub ok 2024-09-17T07:11:28.776 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:11:28.793 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:11:28 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:11:28.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:28 smithi135 ceph-mon[33238]: 5.1b scrub starts 2024-09-17T07:11:28.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:28 smithi135 ceph-mon[33238]: 5.1b scrub ok 2024-09-17T07:11:28.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:28 smithi135 ceph-mon[33238]: 4.e scrub starts 2024-09-17T07:11:28.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:28 smithi135 ceph-mon[33238]: 4.e scrub ok 2024-09-17T07:11:28.815 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:28.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:28.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:11:28.917 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:11:28.948 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:28.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:28.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:11:29.050 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:11:29.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:29.086 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:29.086 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:11:29.187 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:11:29.218 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:29.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:29.220 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:11:29.320 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:11:29.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:29.356 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:29.356 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:11:29.457 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:11:29.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:29.489 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:29.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:11:29.590 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:11:29.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:29.632 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:29.632 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:11:29.696 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:29 smithi100 ceph-mon[29092]: 6.1 scrub starts 2024-09-17T07:11:29.696 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:29 smithi100 ceph-mon[29092]: 6.1 scrub ok 2024-09-17T07:11:29.696 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:29 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:11:29.696 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:29 smithi100 ceph-mon[29092]: pgmap v411: 148 pgs: 148 active+clean; 781 MiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.56k op/s 2024-09-17T07:11:29.696 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:29 smithi100 ceph-mon[39462]: 6.1 scrub starts 2024-09-17T07:11:29.696 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:29 smithi100 ceph-mon[39462]: 6.1 scrub ok 2024-09-17T07:11:29.696 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:29 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:11:29.697 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:29 smithi100 ceph-mon[39462]: pgmap v411: 148 pgs: 148 active+clean; 781 MiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.56k op/s 2024-09-17T07:11:29.733 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:11:29.762 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:29.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:29.763 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:11:29.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:29 smithi135 ceph-mon[33238]: 6.1 scrub starts 2024-09-17T07:11:29.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:29 smithi135 ceph-mon[33238]: 6.1 scrub ok 2024-09-17T07:11:29.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:29 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:11:29.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:29 smithi135 ceph-mon[33238]: pgmap v411: 148 pgs: 148 active+clean; 781 MiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.56k op/s 2024-09-17T07:11:29.864 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:11:29.896 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:29.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:29.897 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:11:29.940 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:11:29.999 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:11:30.041 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:30.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:30.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:11:30.143 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:11:30.192 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:30.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:30.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:11:30.294 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:11:30.330 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:30.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:30.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:11:30.426 INFO:teuthology.orchestra.run.smithi100.stderr:noscrub is set 2024-09-17T07:11:30.432 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:11:30.468 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:30.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:30.469 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:11:30.570 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:11:30.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:30.604 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:30.604 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:11:30.703 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:30 smithi100 ceph-mon[29092]: 8.9 scrub starts 2024-09-17T07:11:30.704 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:30 smithi100 ceph-mon[29092]: 8.9 scrub ok 2024-09-17T07:11:30.704 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:30 smithi100 ceph-mon[29092]: 2.d scrub starts 2024-09-17T07:11:30.704 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:30 smithi100 ceph-mon[29092]: 2.d scrub ok 2024-09-17T07:11:30.704 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:30 smithi100 ceph-mon[29092]: 6.e scrub starts 2024-09-17T07:11:30.704 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:30 smithi100 ceph-mon[29092]: 6.e scrub ok 2024-09-17T07:11:30.704 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:30 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2113657625' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:11:30.704 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:30 smithi100 ceph-mon[39462]: 8.9 scrub starts 2024-09-17T07:11:30.704 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:30 smithi100 ceph-mon[39462]: 8.9 scrub ok 2024-09-17T07:11:30.704 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:30 smithi100 ceph-mon[39462]: 2.d scrub starts 2024-09-17T07:11:30.705 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:30 smithi100 ceph-mon[39462]: 2.d scrub ok 2024-09-17T07:11:30.705 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:30 smithi100 ceph-mon[39462]: 6.e scrub starts 2024-09-17T07:11:30.705 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:30 smithi100 ceph-mon[39462]: 6.e scrub ok 2024-09-17T07:11:30.705 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:30 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2113657625' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:11:30.705 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:11:30.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:30.774 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:30.774 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:11:30.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:30 smithi135 ceph-mon[33238]: 8.9 scrub starts 2024-09-17T07:11:30.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:30 smithi135 ceph-mon[33238]: 8.9 scrub ok 2024-09-17T07:11:30.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:30 smithi135 ceph-mon[33238]: 2.d scrub starts 2024-09-17T07:11:30.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:30 smithi135 ceph-mon[33238]: 2.d scrub ok 2024-09-17T07:11:30.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:30 smithi135 ceph-mon[33238]: 6.e scrub starts 2024-09-17T07:11:30.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:30 smithi135 ceph-mon[33238]: 6.e scrub ok 2024-09-17T07:11:30.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:30 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2113657625' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:11:30.874 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:11:30.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:30.920 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:30.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:11:31.021 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:11:31.053 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:31.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:31.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:11:31.155 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:11:31.188 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:31.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:31.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:11:31.290 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:11:31.329 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:31.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:31.330 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:11:31.431 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:11:31.468 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:31.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:31.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:11:31.570 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:11:31.602 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:31.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:31.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:11:31.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:31 smithi100 ceph-mon[29092]: 8.1 deep-scrub starts 2024-09-17T07:11:31.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:31 smithi100 ceph-mon[29092]: 6.d scrub starts 2024-09-17T07:11:31.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:31 smithi100 ceph-mon[29092]: 6.d scrub ok 2024-09-17T07:11:31.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:31 smithi100 ceph-mon[29092]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:11:31.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:31 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2113657625' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T07:11:31.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:31 smithi100 ceph-mon[29092]: osdmap e226: 8 total, 8 up, 7 in 2024-09-17T07:11:31.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:31 smithi100 ceph-mon[29092]: pgmap v413: 148 pgs: 148 active+clean; 698 MiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.72k op/s 2024-09-17T07:11:31.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:31 smithi100 ceph-mon[39462]: 8.1 deep-scrub starts 2024-09-17T07:11:31.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:31 smithi100 ceph-mon[39462]: 6.d scrub starts 2024-09-17T07:11:31.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:31 smithi100 ceph-mon[39462]: 6.d scrub ok 2024-09-17T07:11:31.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:31 smithi100 ceph-mon[39462]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:11:31.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:31 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2113657625' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T07:11:31.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:31 smithi100 ceph-mon[39462]: osdmap e226: 8 total, 8 up, 7 in 2024-09-17T07:11:31.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:31 smithi100 ceph-mon[39462]: pgmap v413: 148 pgs: 148 active+clean; 698 MiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.72k op/s 2024-09-17T07:11:31.703 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:11:31.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:31.736 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:31.737 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:11:31.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:31 smithi135 ceph-mon[33238]: 8.1 deep-scrub starts 2024-09-17T07:11:31.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:31 smithi135 ceph-mon[33238]: 6.d scrub starts 2024-09-17T07:11:31.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:31 smithi135 ceph-mon[33238]: 6.d scrub ok 2024-09-17T07:11:31.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:31 smithi135 ceph-mon[33238]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:11:31.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:31 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2113657625' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T07:11:31.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:31 smithi135 ceph-mon[33238]: osdmap e226: 8 total, 8 up, 7 in 2024-09-17T07:11:31.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:31 smithi135 ceph-mon[33238]: pgmap v413: 148 pgs: 148 active+clean; 698 MiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.72k op/s 2024-09-17T07:11:31.837 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:11:31.871 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:31.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:31.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:11:31.973 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:11:32.006 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:32.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:32.007 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:11:32.108 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:11:32.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:32.144 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:32.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:11:32.245 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:11:32.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:32.283 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:32.283 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:11:32.384 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:11:32.426 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:32.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:32.427 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:11:32.528 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:11:32.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:32.567 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:32.567 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:11:32.597 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:32 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:11:32.597 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:32 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:11:32.598 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:32 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:11:32.598 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:32 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:11:32.668 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:11:32.710 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:32.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:32.711 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:11:32.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:32 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:11:32.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:32 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:11:32.812 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:11:32.854 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:32.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:32.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:11:32.956 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:11:32.983 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:32.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:32.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:11:33.086 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:11:33.130 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:33.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:33.131 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:11:33.232 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:11:33.264 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:33.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:33.265 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:11:33.366 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:11:33.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:33.395 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:33.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:11:33.496 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:11:33.524 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:33.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:33.526 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:11:33.627 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:11:33.655 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:33.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:33.656 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:11:33.757 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:11:33.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:33.791 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:33.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:11:33.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:33 smithi135 ceph-mon[33238]: osdmap e227: 8 total, 8 up, 7 in 2024-09-17T07:11:33.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:33 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:11:33.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:33 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:11:33.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:33 smithi135 ceph-mon[33238]: pgmap v415: 148 pgs: 148 active+clean; 698 MiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 903 op/s 2024-09-17T07:11:33.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:33 smithi100 ceph-mon[29092]: osdmap e227: 8 total, 8 up, 7 in 2024-09-17T07:11:33.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:33 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:11:33.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:33 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:11:33.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:33 smithi100 ceph-mon[29092]: pgmap v415: 148 pgs: 148 active+clean; 698 MiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 903 op/s 2024-09-17T07:11:33.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:33 smithi100 ceph-mon[39462]: osdmap e227: 8 total, 8 up, 7 in 2024-09-17T07:11:33.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:33 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:11:33.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:33 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:11:33.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:33 smithi100 ceph-mon[39462]: pgmap v415: 148 pgs: 148 active+clean; 698 MiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 903 op/s 2024-09-17T07:11:33.893 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:11:33.924 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:33.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:33.925 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:11:34.026 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:11:34.064 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:34.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:34.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:11:34.089 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-17T07:11:34.168 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:11:34.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:34.210 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:34.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:11:34.311 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:11:34.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:34.354 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:34.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:11:34.454 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:11:34.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:34.498 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:34.498 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:11:34.559 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:11:34.559 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1: {} 2024-09-17T07:11:34.560 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:11:34.560 INFO:teuthology.orchestra.run.smithi100.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:11:34.578 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:11:34.579 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:11:34.600 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:11:34.610 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:11:34.611 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:11:34.640 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:34.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:34.641 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:11:34.645 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:11:34.646 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:11:34.679 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:11:34.679 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:11:34.713 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:34 smithi100 ceph-mon[29092]: 8.1 deep-scrub ok 2024-09-17T07:11:34.714 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:34 smithi100 ceph-mon[39462]: 8.1 deep-scrub ok 2024-09-17T07:11:34.714 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:11:34.714 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:11:34.743 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:11:34.750 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:11:34.750 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:11:34.789 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:34.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:34.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:11:34.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:34 smithi135 ceph-mon[33238]: 8.1 deep-scrub ok 2024-09-17T07:11:34.821 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-17T07:11:34.821 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:11:34.893 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:11:34.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:34.922 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:34.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:11:35.023 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:11:35.065 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:35.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:35.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:11:35.167 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:11:35.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:35.202 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:35.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:11:35.303 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:11:35.341 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:35.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:35.342 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:11:35.443 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:11:35.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:35.493 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:35.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:11:35.594 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:11:35.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:35.627 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:35.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:11:35.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:35 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2519716350' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:11:35.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:35 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:11:35.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:35 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:11:35.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:35 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:11:35.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:35 smithi100 ceph-mon[29092]: pgmap v416: 148 pgs: 148 active+clean; 288 MiB data, 3.4 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.73k op/s 2024-09-17T07:11:35.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:35 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2519716350' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:11:35.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:35 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:11:35.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:35 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:11:35.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:35 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:11:35.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:35 smithi100 ceph-mon[39462]: pgmap v416: 148 pgs: 148 active+clean; 288 MiB data, 3.4 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.73k op/s 2024-09-17T07:11:35.728 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:11:35.763 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:35.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:35.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:11:35.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:35 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2519716350' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:11:35.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:35 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:11:35.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:35 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:11:35.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:35 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:11:35.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:35 smithi135 ceph-mon[33238]: pgmap v416: 148 pgs: 148 active+clean; 288 MiB data, 3.4 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.73k op/s 2024-09-17T07:11:35.865 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:11:35.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:35.898 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:35.898 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:11:35.999 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:11:36.029 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:36.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:36.030 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:11:36.126 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:11:35 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:11:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:11:36.131 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:11:36.173 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:36.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:36.174 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:11:36.275 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:11:36.307 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-17T07:11:36.313 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:36.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:36.314 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:11:36.415 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:11:36.449 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:36.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:36.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:11:36.551 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:11:36.583 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:36.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:36.585 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:11:36.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:36 smithi100 ceph-mon[29092]: osdmap e228: 8 total, 8 up, 7 in 2024-09-17T07:11:36.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:36 smithi100 ceph-mon[39462]: osdmap e228: 8 total, 8 up, 7 in 2024-09-17T07:11:36.685 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:11:36.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:36.722 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:36.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:11:36.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:36 smithi135 ceph-mon[33238]: osdmap e228: 8 total, 8 up, 7 in 2024-09-17T07:11:36.824 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:11:36.849 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:36.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:36.850 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:11:36.870 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:11:36.917 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Removed 130501 objects 2024-09-17T07:11:36.917 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Clean up completed and total clean up time :110.842 2024-09-17T07:11:36.951 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:11:36.971 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:36.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:36.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:11:37.073 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:11:37.095 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:37.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:37.096 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:11:37.197 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:11:37.218 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:37.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:37.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:11:37.320 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:11:37.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:37.343 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:37.343 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:11:37.444 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:11:37.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:37.467 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:37.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:11:37.470 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_1 2024-09-17T07:11:37.471 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd pool rm unique_pool_1 unique_pool_1 --yes-i-really-really-mean-it 2024-09-17T07:11:37.567 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:11:37.588 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:37.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:37.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:11:37.690 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:11:37.710 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:37 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:11:37.710 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:37 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:11:37.710 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:37 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:11:37.710 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:37 smithi100 ceph-mon[29092]: pgmap v418: 148 pgs: 148 active+clean; 288 MiB data, 3.4 GiB used, 622 GiB / 626 GiB avail; 754 B/s rd, 0 B/s wr, 968 op/s 2024-09-17T07:11:37.711 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:37 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:11:37.711 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:37 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:11:37.711 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:37 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:11:37.711 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:37 smithi100 ceph-mon[39462]: pgmap v418: 148 pgs: 148 active+clean; 288 MiB data, 3.4 GiB used, 622 GiB / 626 GiB avail; 754 B/s rd, 0 B/s wr, 968 op/s 2024-09-17T07:11:37.712 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:37.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:37.713 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:11:37.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:37 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:11:37.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:37 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:11:37.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:37 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:11:37.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:37 smithi135 ceph-mon[33238]: pgmap v418: 148 pgs: 148 active+clean; 288 MiB data, 3.4 GiB used, 622 GiB / 626 GiB avail; 754 B/s rd, 0 B/s wr, 968 op/s 2024-09-17T07:11:37.814 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:11:37.824 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:11:37.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:37.836 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:37.836 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:11:37.938 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:11:37.960 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:37.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:37.962 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:11:38.062 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:11:38.086 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:38.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:38.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:11:38.091 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:11:38.092 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:11:38.189 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:11:38.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:38.216 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:38.217 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:11:38.318 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:11:38.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:38.340 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:38.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:11:38.441 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:11:38.463 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:38.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:38.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:11:38.565 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:11:38.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:38.587 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:38.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:11:38.688 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:11:38.712 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:38.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:38.714 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:11:38.815 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:11:38.838 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:38.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:38.839 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:11:38.897 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:11:38 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:11:38.905 INFO:teuthology.orchestra.run.smithi100.stderr:reweighted osd.5 to 1 (10000) 2024-09-17T07:11:38.940 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:11:38.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:38.962 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:38.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:11:38.999 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:38 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1689883405' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-17T07:11:38.999 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:38 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-17T07:11:38.999 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:38 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1689883405' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-17T07:11:39.000 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:38 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-17T07:11:39.063 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:11:39.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:39.086 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:39.086 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:11:39.187 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:11:39.213 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:39.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:39.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:11:39.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:38 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1689883405' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-17T07:11:39.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:38 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-17T07:11:39.315 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:11:39.341 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:39.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:39.342 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:11:39.442 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:11:39.467 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:39.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:39.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:11:39.569 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:11:39.593 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:39.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:39.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:11:39.677 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd reweight 6 1 2024-09-17T07:11:39.695 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:11:39.720 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:39.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:39.721 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:11:39.822 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:11:39.843 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:39.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:39.844 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:11:39.946 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:11:39.967 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:39.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:39.968 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:11:40.069 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:11:40.096 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:40.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:40.097 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:11:40.158 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:39 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:11:40.158 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:39 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-17T07:11:40.158 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:39 smithi100 ceph-mon[39462]: osdmap e229: 8 total, 8 up, 7 in 2024-09-17T07:11:40.158 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:39 smithi100 ceph-mon[39462]: pgmap v420: 148 pgs: 148 active+clean; 41 MiB data, 2.6 GiB used, 623 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.56k op/s 2024-09-17T07:11:40.158 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:39 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:11:40.158 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:39 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-17T07:11:40.158 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:39 smithi100 ceph-mon[29092]: osdmap e229: 8 total, 8 up, 7 in 2024-09-17T07:11:40.158 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:39 smithi100 ceph-mon[29092]: pgmap v420: 148 pgs: 148 active+clean; 41 MiB data, 2.6 GiB used, 623 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.56k op/s 2024-09-17T07:11:40.198 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:11:40.222 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:40.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:40.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:11:40.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:39 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:11:40.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:39 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-17T07:11:40.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:39 smithi135 ceph-mon[33238]: osdmap e229: 8 total, 8 up, 7 in 2024-09-17T07:11:40.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:39 smithi135 ceph-mon[33238]: pgmap v420: 148 pgs: 148 active+clean; 41 MiB data, 2.6 GiB used, 623 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.56k op/s 2024-09-17T07:11:40.324 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:11:40.347 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:40.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:40.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:11:40.449 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:11:40.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:40.471 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:40.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:11:40.572 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:11:40.593 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:40.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:40.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:11:40.695 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:11:40.716 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:40.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:40.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:11:40.818 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:11:40.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:40.842 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:40.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:11:40.943 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:11:40.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:40.965 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:40.966 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:11:41.066 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:11:41.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:41.089 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:41.089 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:11:41.190 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:11:41.211 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:41.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:41.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:11:41.313 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:11:41.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:41.336 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:41.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:11:41.437 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:11:41.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:41.462 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:41.462 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:11:41.563 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:11:41.587 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:41.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:41.589 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:11:41.689 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:11:41.715 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:41.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:41.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:11:41.817 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:11:41.838 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:41.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:41.839 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:11:41.940 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:11:41.961 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:41.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:41.962 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:11:42.063 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:11:42.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:42.089 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:42.092 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:11:42.194 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:11:42.215 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:42.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:42.216 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:11:42.316 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:11:42.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:42.338 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:42.338 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:11:42.371 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:42 smithi100 ceph-mon[29092]: pgmap v421: 148 pgs: 148 active+clean; 582 KiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.40k op/s 2024-09-17T07:11:42.372 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:42 smithi100 ceph-mon[39462]: pgmap v421: 148 pgs: 148 active+clean; 582 KiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.40k op/s 2024-09-17T07:11:42.373 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-17T07:11:42.373 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:11:42.439 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:11:42.465 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:42.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:42.466 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:11:42.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:42 smithi135 ceph-mon[33238]: pgmap v421: 148 pgs: 148 active+clean; 582 KiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.40k op/s 2024-09-17T07:11:42.567 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:11:42.588 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:42.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:42.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:11:42.691 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:11:42.716 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:42.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:42.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:11:42.818 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:11:42.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:42.844 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:42.845 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:11:42.945 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:11:42.968 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:42.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:42.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:11:43.041 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-17T07:11:43.069 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:11:43.095 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:43.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:43.096 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:11:43.118 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:11:43.197 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:11:43.222 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:43.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:43.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:11:43.268 INFO:teuthology.orchestra.run.smithi100.stderr:nodeep-scrub is set 2024-09-17T07:11:43.279 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:43 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1804522943' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:11:43.279 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:43 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:11:43.279 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:43 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1804522943' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:11:43.279 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:43 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:11:43.324 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:11:43.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:43.349 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:43.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:11:43.450 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:11:43.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:43.475 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:43.475 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:11:43.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:43 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1804522943' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:11:43.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:43 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:11:43.577 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:11:43.601 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:43.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:43.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:11:43.703 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:11:43.726 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:43.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:43.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:11:43.828 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:11:43.850 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:43.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:43.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:11:43.952 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:11:43.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:43.977 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:43.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:11:44.079 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:11:44.101 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:44.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:44.103 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:11:44.204 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:11:44.228 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:44.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:44.229 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:11:44.330 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:11:44.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:44.352 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:44.353 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:11:44.453 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:11:44.478 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:44.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:44.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:11:44.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:44 smithi135 ceph-mon[33238]: pgmap v422: 148 pgs: 148 active+clean; 582 KiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 574 op/s 2024-09-17T07:11:44.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:44 smithi135 ceph-mon[33238]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:11:44.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:44 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T07:11:44.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:44 smithi135 ceph-mon[33238]: osdmap e230: 8 total, 8 up, 7 in 2024-09-17T07:11:44.580 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:11:44.604 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:44.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:44.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:11:44.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:44 smithi100 ceph-mon[29092]: pgmap v422: 148 pgs: 148 active+clean; 582 KiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 574 op/s 2024-09-17T07:11:44.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:44 smithi100 ceph-mon[29092]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:11:44.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:44 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T07:11:44.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:44 smithi100 ceph-mon[29092]: osdmap e230: 8 total, 8 up, 7 in 2024-09-17T07:11:44.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:44 smithi100 ceph-mon[39462]: pgmap v422: 148 pgs: 148 active+clean; 582 KiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 574 op/s 2024-09-17T07:11:44.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:44 smithi100 ceph-mon[39462]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:11:44.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:44 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T07:11:44.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:44 smithi100 ceph-mon[39462]: osdmap e230: 8 total, 8 up, 7 in 2024-09-17T07:11:44.706 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:11:44.730 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:44.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:44.731 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:11:44.832 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:11:44.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:44.858 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:44.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:11:44.959 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:11:44.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:44.984 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:44.984 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:11:45.085 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:11:45.108 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:45.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:45.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:11:45.210 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:11:45.232 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:45.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:45.233 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:11:45.295 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:11:45.295 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:11:45.310 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1: {} 2024-09-17T07:11:45.311 INFO:teuthology.orchestra.run.smithi100.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:11:45.326 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:11:45.326 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:11:45.334 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:11:45.342 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:11:45.342 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:11:45.359 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:11:45.359 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:11:45.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:45.360 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:45.360 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:11:45.375 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:11:45.375 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:11:45.391 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:11:45.391 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:11:45.407 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:11:45.408 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:11:45.461 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:11:45.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:45.483 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:45.483 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:11:45.584 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:11:45.609 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:45.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:45.610 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:11:45.711 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:11:45.736 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:45.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:45.737 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:11:45.838 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:11:45.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:45.865 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:45.865 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:11:45.966 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:11:45.988 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:45.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:45.989 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:11:46.089 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:11:46.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:46.112 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:46.113 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:11:46.117 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:11:45 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:11:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:11:46.214 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:11:46.235 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:46.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:46.236 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:11:46.337 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:11:46.363 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:46.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:46.364 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:11:46.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:46 smithi100 ceph-mon[29092]: pgmap v424: 148 pgs: 148 active+clean; 582 KiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 575 op/s 2024-09-17T07:11:46.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:46 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/4136306454' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:11:46.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:46 smithi100 ceph-mon[39462]: pgmap v424: 148 pgs: 148 active+clean; 582 KiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 575 op/s 2024-09-17T07:11:46.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:46 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/4136306454' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:11:46.465 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:11:46.486 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:46.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:46.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:11:46.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:46 smithi135 ceph-mon[33238]: pgmap v424: 148 pgs: 148 active+clean; 582 KiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 575 op/s 2024-09-17T07:11:46.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:46 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/4136306454' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:11:46.589 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:11:46.610 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:46.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:46.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:11:46.712 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:11:46.733 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:46.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:46.734 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:11:46.835 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:11:46.861 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:46.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:46.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:11:46.963 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:11:46.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:46.990 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:46.990 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:11:47.024 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:11:47.090 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:11:47.116 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:47.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:47.117 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:11:47.173 INFO:teuthology.orchestra.run.smithi100.stderr:pool 'unique_pool_1' removed 2024-09-17T07:11:47.218 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:11:47.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:47.259 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:47.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:11:47.288 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:47 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2001496150' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-17T07:11:47.288 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:47 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2001496150' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-17T07:11:47.360 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:11:47.381 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:47.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:47.382 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:11:47.483 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:11:47.509 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:47.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:47.510 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:11:47.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:47 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2001496150' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-17T07:11:47.611 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:11:47.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:47.634 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:47.634 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:11:47.735 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:11:47.758 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:47.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:47.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:11:47.860 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:11:47.882 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:47.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:47.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:11:47.984 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:11:48.006 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:48.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:48.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:11:48.029 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:11:48.107 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:11:48.129 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:48.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:48.130 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:11:48.139 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:11:48.232 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:11:48.254 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:48.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:48.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:11:48.356 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:11:48.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:48.379 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:48.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:11:48.481 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:11:48.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:48.504 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:48.504 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:11:48.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:48 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2001496150' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-09-17T07:11:48.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:48 smithi135 ceph-mon[33238]: osdmap e231: 8 total, 8 up, 7 in 2024-09-17T07:11:48.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:48 smithi135 ceph-mon[33238]: pgmap v426: 132 pgs: 132 active+clean; 582 KiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 511 B/s rd, 0 op/s 2024-09-17T07:11:48.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:48 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:11:48.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:48 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:11:48.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:48 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:11:48.577 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:48 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2001496150' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-09-17T07:11:48.577 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:48 smithi100 ceph-mon[29092]: osdmap e231: 8 total, 8 up, 7 in 2024-09-17T07:11:48.577 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:48 smithi100 ceph-mon[29092]: pgmap v426: 132 pgs: 132 active+clean; 582 KiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 511 B/s rd, 0 op/s 2024-09-17T07:11:48.577 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:48 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:11:48.577 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:48 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:11:48.577 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:48 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:11:48.577 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:48 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2001496150' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-09-17T07:11:48.577 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:48 smithi100 ceph-mon[39462]: osdmap e231: 8 total, 8 up, 7 in 2024-09-17T07:11:48.578 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:48 smithi100 ceph-mon[39462]: pgmap v426: 132 pgs: 132 active+clean; 582 KiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 511 B/s rd, 0 op/s 2024-09-17T07:11:48.578 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:48 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:11:48.578 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:48 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:11:48.578 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:48 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:11:48.598 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-17T07:11:48.599 INFO:tasks.radosbench:Beginning radosbench... 2024-09-17T07:11:48.599 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_2 2024-09-17T07:11:48.599 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd pool create unique_pool_2 16 2024-09-17T07:11:48.604 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:11:48.665 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:48.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:48.666 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:11:48.767 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:11:48.789 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:48.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:48.790 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:11:48.891 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:11:48.913 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:48.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:48.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:11:49.016 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:11:49.030 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T07:11:49.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:49.038 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:49.038 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:11:49.043 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:11:48 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:11:49.139 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:11:49.160 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:49.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:49.161 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:11:49.262 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:11:49.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:49.285 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:49.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:11:49.386 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:11:49.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:49.408 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:49.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:11:49.497 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:49 smithi100 ceph-mon[29092]: osdmap e232: 8 total, 8 up, 7 in 2024-09-17T07:11:49.498 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:49 smithi100 ceph-mon[39462]: osdmap e232: 8 total, 8 up, 7 in 2024-09-17T07:11:49.509 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:11:49.532 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:49.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:49.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:11:49.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:49 smithi135 ceph-mon[33238]: osdmap e232: 8 total, 8 up, 7 in 2024-09-17T07:11:49.634 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:11:49.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:49.657 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:49.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:11:49.758 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:11:49.781 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:49.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:49.782 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:11:49.841 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:11:49.882 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:11:49.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:49.905 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:49.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:11:50.006 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:11:50.029 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:50.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:50.030 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:11:50.130 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:11:50.153 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:50.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:50.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:11:50.254 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:11:50.276 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:50.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:50.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:11:50.378 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:11:50.416 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:50.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:50.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:11:50.498 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:50 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:11:50.498 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:50 smithi100 ceph-mon[39462]: pgmap v428: 132 pgs: 132 active+clean; 582 KiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-17T07:11:50.498 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:50 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:11:50.499 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:50 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:11:50.499 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:50 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:11:50.499 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:50 smithi100 ceph-mon[29092]: pgmap v428: 132 pgs: 132 active+clean; 582 KiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-17T07:11:50.499 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:50 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:11:50.499 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:50 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:11:50.519 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:11:50.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:50 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:11:50.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:50 smithi135 ceph-mon[33238]: pgmap v428: 132 pgs: 132 active+clean; 582 KiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-17T07:11:50.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:50 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:11:50.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:50 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:11:50.547 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:50.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:50.548 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:11:50.649 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:11:50.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:50.674 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:50.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:11:50.727 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:11:50.775 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:11:50.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:50.797 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:50.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:11:50.900 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:11:50.923 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:50.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:50.924 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:11:51.025 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:11:51.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:51.054 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:51.054 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:11:51.068 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-17T07:11:51.068 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-17T07:11:51.068 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-17T07:11:51.068 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-17T07:11:51.068 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-17T07:11:51.155 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:11:51.180 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:51.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:51.181 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:11:51.282 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:11:51.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:51.307 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:51.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:11:51.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:51 smithi100 ceph-mon[39462]: osdmap e233: 8 total, 8 up, 7 in 2024-09-17T07:11:51.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:51 smithi100 ceph-mon[29092]: osdmap e233: 8 total, 8 up, 7 in 2024-09-17T07:11:51.409 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:11:51.430 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:51.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:51.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:11:51.532 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:11:51.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:51 smithi135 ceph-mon[33238]: osdmap e233: 8 total, 8 up, 7 in 2024-09-17T07:11:51.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:51.555 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:51.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:11:51.656 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:11:51.678 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:51.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:51.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:11:51.780 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:11:51.806 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:51.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:51.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:11:51.908 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:11:51.934 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:51.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:51.936 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:11:52.036 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:11:52.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:52.059 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:52.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:11:52.160 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:11:52.182 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:52.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:52.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:11:52.231 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-17T07:11:52.284 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:11:52.306 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:52.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:52.307 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:11:52.408 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:11:52.432 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:52.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:52.433 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:11:52.436 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:52 smithi100 ceph-mon[29092]: pgmap v430: 132 pgs: 132 active+clean; 582 KiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 682 B/s rd, 0 op/s 2024-09-17T07:11:52.437 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:52 smithi100 ceph-mon[39462]: pgmap v430: 132 pgs: 132 active+clean; 582 KiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 682 B/s rd, 0 op/s 2024-09-17T07:11:52.534 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:11:52.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:52 smithi135 ceph-mon[33238]: pgmap v430: 132 pgs: 132 active+clean; 582 KiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 682 B/s rd, 0 op/s 2024-09-17T07:11:52.557 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:52.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:52.558 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:11:52.659 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:11:52.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:52.683 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:52.683 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:11:52.784 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:11:52.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:52.810 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:52.810 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:11:52.911 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:11:52.935 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:52.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:52.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:11:53.037 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:11:53.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:53.060 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:53.061 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:11:53.161 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:11:53.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:53.185 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:53.185 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:11:53.286 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:11:53.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:53.311 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:53.311 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:11:53.412 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:11:53.434 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:53.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:53.435 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:11:53.536 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:11:53.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:53.562 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:53.562 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:11:53.663 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:11:53.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:53.687 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:53.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:11:53.788 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:11:53.812 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:53.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:53.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:11:53.914 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:11:53.937 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:53.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:53.939 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:11:54.039 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:11:54.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:54.063 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:54.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:11:54.164 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:11:54.185 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:54.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:54.187 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:11:54.287 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:11:54.314 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:54.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:54.315 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:11:54.416 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:11:54.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:54.438 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:54.439 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:11:54.539 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:11:54.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:54 smithi135 ceph-mon[33238]: pgmap v431: 132 pgs: 132 active+clean; 582 KiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 848 B/s rd, 0 op/s 2024-09-17T07:11:54.564 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:54.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:54.565 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:11:54.568 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:54 smithi100 ceph-mon[29092]: pgmap v431: 132 pgs: 132 active+clean; 582 KiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 848 B/s rd, 0 op/s 2024-09-17T07:11:54.568 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:54 smithi100 ceph-mon[39462]: pgmap v431: 132 pgs: 132 active+clean; 582 KiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 848 B/s rd, 0 op/s 2024-09-17T07:11:54.666 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:11:54.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:54.689 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:54.689 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:11:54.790 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:11:54.816 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:54.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:54.817 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:11:54.873 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:11:54.881 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:11:54.918 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:11:54.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:54.942 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:54.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:11:55.043 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:11:55.069 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:55.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:55.070 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:11:55.170 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:11:55.195 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:55.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:55.197 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:11:55.270 INFO:teuthology.orchestra.run.smithi100.stderr:reweighted osd.6 to 1 (10000) 2024-09-17T07:11:55.297 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:11:55.321 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:55.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:55.322 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:11:55.386 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:55 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2853461033' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-17T07:11:55.386 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:55 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-17T07:11:55.386 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:55 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2853461033' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-17T07:11:55.387 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:55 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-17T07:11:55.423 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:11:55.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:55.445 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:55.446 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:11:55.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:55 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2853461033' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-17T07:11:55.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:55 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-17T07:11:55.547 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:11:55.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:55.572 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:55.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:11:55.674 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:11:55.699 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:55.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:55.700 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:11:55.801 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:11:55.862 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:55.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:55.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:11:55.964 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:11:55.990 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:55.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:55.991 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:11:56.085 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:11:55 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:11:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:11:56.091 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:11:56.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:56.118 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:56.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:11:56.219 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:11:56.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:56.242 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:56.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:11:56.343 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:11:56.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:56.366 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:56.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:11:56.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:56 smithi100 ceph-mon[29092]: pgmap v432: 132 pgs: 132 active+clean; 582 KiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:11:56.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:56 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-17T07:11:56.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:56 smithi100 ceph-mon[29092]: osdmap e234: 8 total, 8 up, 7 in 2024-09-17T07:11:56.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:56 smithi100 ceph-mon[39462]: pgmap v432: 132 pgs: 132 active+clean; 582 KiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:11:56.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:56 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-17T07:11:56.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:56 smithi100 ceph-mon[39462]: osdmap e234: 8 total, 8 up, 7 in 2024-09-17T07:11:56.467 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:11:56.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:56.489 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:56.489 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:11:56.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:56 smithi135 ceph-mon[33238]: pgmap v432: 132 pgs: 132 active+clean; 582 KiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:11:56.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:56 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-17T07:11:56.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:56 smithi135 ceph-mon[33238]: osdmap e234: 8 total, 8 up, 7 in 2024-09-17T07:11:56.590 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:11:56.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:56.616 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:56.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:11:56.717 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:11:56.740 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:56.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:56.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:11:56.842 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:11:56.863 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:56.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:56.865 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:11:56.965 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:11:56.991 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:56.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:56.992 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:11:57.093 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:11:57.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:57.117 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:57.117 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:11:57.218 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:11:57.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:57.244 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:57.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:11:57.306 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd reweight 7 1 2024-09-17T07:11:57.346 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:11:57.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:57.371 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:57.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:11:57.472 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:11:57.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:57.494 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:57.494 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:11:57.595 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:11:57.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:57.621 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:57.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:11:57.722 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:11:57.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:57.746 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:57.746 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:11:57.847 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:11:57.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:57.871 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:57.871 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:11:57.972 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:11:57.998 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:57.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:57.999 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:11:58.100 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:11:58.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:58.125 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:58.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:11:58.226 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:11:58.247 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:58.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:58.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:11:58.349 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:11:58.353 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:58 smithi100 ceph-mon[29092]: pgmap v434: 132 pgs: 132 active+clean; 582 KiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-09-17T07:11:58.353 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:58 smithi100 ceph-mon[39462]: pgmap v434: 132 pgs: 132 active+clean; 582 KiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-09-17T07:11:58.371 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:58.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:58.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:11:58.472 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:11:58.499 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:58.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:58.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:11:58.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:58 smithi135 ceph-mon[33238]: pgmap v434: 132 pgs: 132 active+clean; 582 KiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-09-17T07:11:58.601 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:11:58.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:58.623 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:58.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:11:58.724 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:11:58.745 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:58.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:58.746 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:11:58.847 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:11:58.876 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:58.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:58.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:11:58.978 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:11:59.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:59.000 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:59.001 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:11:59.043 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:11:58 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:11:59.102 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:11:59.127 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:59.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:59.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:11:59.230 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:11:59.255 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:59.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:59.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:11:59.286 INFO:teuthology.orchestra.run.smithi100.stderr:noscrub is unset 2024-09-17T07:11:59.357 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:11:59.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:59.380 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:59.381 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:11:59.481 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:11:59.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:59.504 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:59.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:11:59.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:11:59 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1135579757' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:11:59.606 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:11:59.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:11:59 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1135579757' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:11:59.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:11:59 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1135579757' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:11:59.630 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:11:59.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:59.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:11:59.732 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:11:59.755 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:59.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:59.756 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:11:59.856 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:11:59.879 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:11:59.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:11:59.881 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:11:59.981 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:12:00.007 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:00.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:00.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:12:00.109 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:12:00.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:00.131 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:00.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:12:00.232 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:12:00.258 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:00.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:00.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:12:00.360 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:12:00.381 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:00.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:00.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:12:00.483 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:12:00.509 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:00.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:00.510 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:12:00.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:00 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:12:00.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:00 smithi135 ceph-mon[33238]: pgmap v435: 132 pgs: 132 active+clean; 582 KiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T07:12:00.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:00 smithi135 ceph-mon[33238]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:12:00.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:00 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1135579757' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T07:12:00.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:00 smithi135 ceph-mon[33238]: osdmap e235: 8 total, 8 up, 7 in 2024-09-17T07:12:00.584 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:00 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:12:00.584 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:00 smithi100 ceph-mon[29092]: pgmap v435: 132 pgs: 132 active+clean; 582 KiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T07:12:00.584 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:00 smithi100 ceph-mon[29092]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:12:00.584 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:00 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1135579757' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T07:12:00.584 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:00 smithi100 ceph-mon[29092]: osdmap e235: 8 total, 8 up, 7 in 2024-09-17T07:12:00.585 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:00 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:12:00.585 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:00 smithi100 ceph-mon[39462]: pgmap v435: 132 pgs: 132 active+clean; 582 KiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T07:12:00.585 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:00 smithi100 ceph-mon[39462]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:12:00.585 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:00 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1135579757' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T07:12:00.585 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:00 smithi100 ceph-mon[39462]: osdmap e235: 8 total, 8 up, 7 in 2024-09-17T07:12:00.610 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:12:00.631 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:00.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:00.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:12:00.733 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:12:00.758 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:00.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:00.759 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:12:00.860 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:12:00.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:00.883 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:00.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:12:00.895 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-17T07:12:00.895 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:12:00.984 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:12:01.010 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:01.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:01.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:12:01.112 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:12:01.137 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:01.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:01.139 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:12:01.239 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:12:01.261 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:01.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:01.262 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:12:01.363 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:12:01.388 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:01.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:01.389 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:12:01.490 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:12:01.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:01.515 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:01.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:12:01.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:01 smithi135 ceph-mon[33238]: 5.d scrub starts 2024-09-17T07:12:01.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:01 smithi135 ceph-mon[33238]: 5.d scrub ok 2024-09-17T07:12:01.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:01 smithi135 ceph-mon[33238]: 4.1 scrub starts 2024-09-17T07:12:01.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:01 smithi135 ceph-mon[33238]: 4.1 scrub ok 2024-09-17T07:12:01.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:01 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2544464057' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:12:01.559 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:01 smithi100 ceph-mon[29092]: 5.d scrub starts 2024-09-17T07:12:01.559 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:01 smithi100 ceph-mon[29092]: 5.d scrub ok 2024-09-17T07:12:01.560 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:01 smithi100 ceph-mon[29092]: 4.1 scrub starts 2024-09-17T07:12:01.560 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:01 smithi100 ceph-mon[29092]: 4.1 scrub ok 2024-09-17T07:12:01.560 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:01 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2544464057' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:12:01.560 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:01 smithi100 ceph-mon[39462]: 5.d scrub starts 2024-09-17T07:12:01.560 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:01 smithi100 ceph-mon[39462]: 5.d scrub ok 2024-09-17T07:12:01.561 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:01 smithi100 ceph-mon[39462]: 4.1 scrub starts 2024-09-17T07:12:01.561 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:01 smithi100 ceph-mon[39462]: 4.1 scrub ok 2024-09-17T07:12:01.561 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:01 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2544464057' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:12:01.580 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-17T07:12:01.617 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:12:01.644 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:01.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:01.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:12:01.746 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:12:01.767 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:01.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:01.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:12:01.869 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:12:01.894 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:01.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:01.895 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:12:01.996 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:12:02.022 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:02.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:02.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:12:02.124 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:12:02.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:02.148 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:02.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:12:02.250 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:12:02.276 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:02.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:02.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:12:02.297 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:12:02.297 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1: {} 2024-09-17T07:12:02.297 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:12:02.297 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:12:02.297 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:12:02.298 INFO:teuthology.orchestra.run.smithi100.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:12:02.298 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:12:02.298 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:12:02.298 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:12:02.298 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:12:02.298 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:12:02.298 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:12:02.298 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:12:02.299 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:12:02.299 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:12:02.299 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:12:02.378 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:12:02.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:02.401 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:02.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:12:02.502 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:12:02.523 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:02.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:02.524 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:12:02.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:02 smithi135 ceph-mon[33238]: pgmap v437: 132 pgs: 132 active+clean; 582 KiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:12:02.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:02 smithi135 ceph-mon[33238]: 5.1d scrub starts 2024-09-17T07:12:02.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:02 smithi135 ceph-mon[33238]: 5.1d scrub ok 2024-09-17T07:12:02.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:02 smithi135 ceph-mon[33238]: 3.2 scrub starts 2024-09-17T07:12:02.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:02 smithi135 ceph-mon[33238]: 3.2 scrub ok 2024-09-17T07:12:02.551 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:02 smithi100 ceph-mon[39462]: pgmap v437: 132 pgs: 132 active+clean; 582 KiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:12:02.551 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:02 smithi100 ceph-mon[39462]: 5.1d scrub starts 2024-09-17T07:12:02.551 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:02 smithi100 ceph-mon[39462]: 5.1d scrub ok 2024-09-17T07:12:02.551 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:02 smithi100 ceph-mon[39462]: 3.2 scrub starts 2024-09-17T07:12:02.551 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:02 smithi100 ceph-mon[39462]: 3.2 scrub ok 2024-09-17T07:12:02.551 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:02 smithi100 ceph-mon[29092]: pgmap v437: 132 pgs: 132 active+clean; 582 KiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:12:02.551 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:02 smithi100 ceph-mon[29092]: 5.1d scrub starts 2024-09-17T07:12:02.552 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:02 smithi100 ceph-mon[29092]: 5.1d scrub ok 2024-09-17T07:12:02.552 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:02 smithi100 ceph-mon[29092]: 3.2 scrub starts 2024-09-17T07:12:02.552 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:02 smithi100 ceph-mon[29092]: 3.2 scrub ok 2024-09-17T07:12:02.624 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:12:02.650 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:02.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:02.651 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:12:02.751 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:12:02.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:02.778 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:02.778 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:12:02.879 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:12:02.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:02.905 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:02.905 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:12:03.006 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:12:03.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:03.033 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:03.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:12:03.133 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:12:03.159 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:03.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:03.160 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:12:03.183 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:12:03.261 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:12:03.282 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:03.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:03.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:12:03.384 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:12:03.405 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:03.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:03.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:12:03.453 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:03 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:12:03.453 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:03 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:12:03.507 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:12:03.528 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:03.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:03.529 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:12:03.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:03 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:12:03.630 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:12:03.657 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:03.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:03.658 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:12:03.759 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:12:03.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:03.784 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:03.784 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:12:03.885 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:12:03.910 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:03.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:03.911 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:12:04.012 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:12:04.034 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:04.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:04.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:12:04.137 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:12:04.162 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:04.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:04.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:12:04.193 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-17T07:12:04.263 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:12:04.284 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:04.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:04.285 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:12:04.303 INFO:teuthology.orchestra.run.smithi100.stderr:pool 'unique_pool_2' created 2024-09-17T07:12:04.386 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:12:04.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:04.411 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:04.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:12:04.512 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:12:04.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:04.536 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:04.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:12:04.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:04 smithi135 ceph-mon[33238]: 2.7 scrub starts 2024-09-17T07:12:04.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:04 smithi135 ceph-mon[33238]: 2.7 scrub ok 2024-09-17T07:12:04.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:04 smithi135 ceph-mon[33238]: pgmap v438: 132 pgs: 132 active+clean; 582 KiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-09-17T07:12:04.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:04 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/408369572' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-09-17T07:12:04.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:04 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-09-17T07:12:04.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:04 smithi135 ceph-mon[33238]: 4.17 scrub starts 2024-09-17T07:12:04.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:04 smithi135 ceph-mon[33238]: 4.17 scrub ok 2024-09-17T07:12:04.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:04 smithi135 ceph-mon[33238]: 5.3 scrub starts 2024-09-17T07:12:04.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:04 smithi135 ceph-mon[33238]: 5.3 scrub ok 2024-09-17T07:12:04.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:04 smithi135 ceph-mon[33238]: 6.6 scrub starts 2024-09-17T07:12:04.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:04 smithi135 ceph-mon[33238]: 6.6 scrub ok 2024-09-17T07:12:04.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:04 smithi135 ceph-mon[33238]: 5.b scrub starts 2024-09-17T07:12:04.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:04 smithi135 ceph-mon[33238]: 5.b scrub ok 2024-09-17T07:12:04.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:04 smithi100 ceph-mon[39462]: 2.7 scrub starts 2024-09-17T07:12:04.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:04 smithi100 ceph-mon[39462]: 2.7 scrub ok 2024-09-17T07:12:04.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:04 smithi100 ceph-mon[39462]: pgmap v438: 132 pgs: 132 active+clean; 582 KiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-09-17T07:12:04.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:04 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/408369572' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-09-17T07:12:04.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:04 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-09-17T07:12:04.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:04 smithi100 ceph-mon[39462]: 4.17 scrub starts 2024-09-17T07:12:04.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:04 smithi100 ceph-mon[39462]: 4.17 scrub ok 2024-09-17T07:12:04.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:04 smithi100 ceph-mon[39462]: 5.3 scrub starts 2024-09-17T07:12:04.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:04 smithi100 ceph-mon[39462]: 5.3 scrub ok 2024-09-17T07:12:04.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:04 smithi100 ceph-mon[39462]: 6.6 scrub starts 2024-09-17T07:12:04.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:04 smithi100 ceph-mon[39462]: 6.6 scrub ok 2024-09-17T07:12:04.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:04 smithi100 ceph-mon[39462]: 5.b scrub starts 2024-09-17T07:12:04.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:04 smithi100 ceph-mon[39462]: 5.b scrub ok 2024-09-17T07:12:04.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:04 smithi100 ceph-mon[29092]: 2.7 scrub starts 2024-09-17T07:12:04.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:04 smithi100 ceph-mon[29092]: 2.7 scrub ok 2024-09-17T07:12:04.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:04 smithi100 ceph-mon[29092]: pgmap v438: 132 pgs: 132 active+clean; 582 KiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-09-17T07:12:04.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:04 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/408369572' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-09-17T07:12:04.629 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:04 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-09-17T07:12:04.629 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:04 smithi100 ceph-mon[29092]: 4.17 scrub starts 2024-09-17T07:12:04.629 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:04 smithi100 ceph-mon[29092]: 4.17 scrub ok 2024-09-17T07:12:04.629 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:04 smithi100 ceph-mon[29092]: 5.3 scrub starts 2024-09-17T07:12:04.629 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:04 smithi100 ceph-mon[29092]: 5.3 scrub ok 2024-09-17T07:12:04.629 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:04 smithi100 ceph-mon[29092]: 6.6 scrub starts 2024-09-17T07:12:04.629 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:04 smithi100 ceph-mon[29092]: 6.6 scrub ok 2024-09-17T07:12:04.629 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:04 smithi100 ceph-mon[29092]: 5.b scrub starts 2024-09-17T07:12:04.629 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:04 smithi100 ceph-mon[29092]: 5.b scrub ok 2024-09-17T07:12:04.633 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:12:04.637 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:12:04.660 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:04.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:04.661 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:12:04.762 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:12:04.783 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:04.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:04.784 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:12:04.885 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:12:04.898 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:12:04.908 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:04.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:04.909 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:12:05.009 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:12:05.031 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd pool application enable unique_pool_2 rados --yes-i-really-mean-it || true 2024-09-17T07:12:05.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:05.034 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:05.034 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:12:05.134 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:12:05.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:05.157 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:05.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:12:05.258 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:12:05.278 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:05.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:05.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:12:05.380 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:12:05.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:05.403 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:05.403 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:12:05.504 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:12:05.527 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:05.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:05.527 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:12:05.602 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:05 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-09-17T07:12:05.603 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:05 smithi100 ceph-mon[29092]: osdmap e236: 8 total, 8 up, 7 in 2024-09-17T07:12:05.603 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:05 smithi100 ceph-mon[29092]: 2.4 scrub starts 2024-09-17T07:12:05.603 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:05 smithi100 ceph-mon[29092]: 2.4 scrub ok 2024-09-17T07:12:05.603 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:05 smithi100 ceph-mon[29092]: osdmap e237: 8 total, 8 up, 7 in 2024-09-17T07:12:05.603 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:05 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-09-17T07:12:05.603 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:05 smithi100 ceph-mon[39462]: osdmap e236: 8 total, 8 up, 7 in 2024-09-17T07:12:05.603 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:05 smithi100 ceph-mon[39462]: 2.4 scrub starts 2024-09-17T07:12:05.603 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:05 smithi100 ceph-mon[39462]: 2.4 scrub ok 2024-09-17T07:12:05.603 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:05 smithi100 ceph-mon[39462]: osdmap e237: 8 total, 8 up, 7 in 2024-09-17T07:12:05.628 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:12:05.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:05.651 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:05.651 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:12:05.752 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:12:05.775 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:05.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:05.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:12:05.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:05 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-09-17T07:12:05.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:05 smithi135 ceph-mon[33238]: osdmap e236: 8 total, 8 up, 7 in 2024-09-17T07:12:05.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:05 smithi135 ceph-mon[33238]: 2.4 scrub starts 2024-09-17T07:12:05.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:05 smithi135 ceph-mon[33238]: 2.4 scrub ok 2024-09-17T07:12:05.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:05 smithi135 ceph-mon[33238]: osdmap e237: 8 total, 8 up, 7 in 2024-09-17T07:12:05.832 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:12:05.848 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:12:05.871 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:12:05.877 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:12:05.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:05.902 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:05.902 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:12:06.003 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:12:06.027 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:06.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:06.028 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:12:06.128 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:12:06.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:06.152 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:06.152 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:12:06.196 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:12:05 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:12:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:12:06.253 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:12:06.276 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:06.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:06.277 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:12:06.378 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:12:06.401 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:06.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:06.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:12:06.502 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:12:06.525 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:06.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:06.526 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:12:06.626 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:12:06.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:06 smithi100 ceph-mon[29092]: pgmap v441: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:12:06.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:06 smithi100 ceph-mon[29092]: 6.11 scrub starts 2024-09-17T07:12:06.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:06 smithi100 ceph-mon[29092]: 6.11 scrub ok 2024-09-17T07:12:06.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:06 smithi100 ceph-mon[29092]: 5.6 scrub starts 2024-09-17T07:12:06.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:06 smithi100 ceph-mon[29092]: 5.6 scrub ok 2024-09-17T07:12:06.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:06 smithi100 ceph-mon[29092]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-17T07:12:06.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:06 smithi100 ceph-mon[29092]: osdmap e238: 8 total, 8 up, 7 in 2024-09-17T07:12:06.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:06 smithi100 ceph-mon[39462]: pgmap v441: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:12:06.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:06 smithi100 ceph-mon[39462]: 6.11 scrub starts 2024-09-17T07:12:06.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:06 smithi100 ceph-mon[39462]: 6.11 scrub ok 2024-09-17T07:12:06.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:06 smithi100 ceph-mon[39462]: 5.6 scrub starts 2024-09-17T07:12:06.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:06 smithi100 ceph-mon[39462]: 5.6 scrub ok 2024-09-17T07:12:06.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:06 smithi100 ceph-mon[39462]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-17T07:12:06.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:06 smithi100 ceph-mon[39462]: osdmap e238: 8 total, 8 up, 7 in 2024-09-17T07:12:06.650 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:06.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:06.651 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:12:06.751 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:12:06.774 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:06.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:06.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:12:06.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:06 smithi135 ceph-mon[33238]: pgmap v441: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:12:06.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:06 smithi135 ceph-mon[33238]: 6.11 scrub starts 2024-09-17T07:12:06.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:06 smithi135 ceph-mon[33238]: 6.11 scrub ok 2024-09-17T07:12:06.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:06 smithi135 ceph-mon[33238]: 5.6 scrub starts 2024-09-17T07:12:06.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:06 smithi135 ceph-mon[33238]: 5.6 scrub ok 2024-09-17T07:12:06.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:06 smithi135 ceph-mon[33238]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-17T07:12:06.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:06 smithi135 ceph-mon[33238]: osdmap e238: 8 total, 8 up, 7 in 2024-09-17T07:12:06.875 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:12:06.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:06.898 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:06.899 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:12:06.999 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:12:07.021 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:07.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:07.022 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:12:07.123 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:12:07.148 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:07.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:07.149 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:12:07.249 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:12:07.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:07.274 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:07.274 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:12:07.375 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:12:07.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:07.399 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:07.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:12:07.500 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:12:07.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:07.523 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:07.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:12:07.591 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:07 smithi100 ceph-mon[39462]: 6.15 scrub starts 2024-09-17T07:12:07.591 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:07 smithi100 ceph-mon[39462]: 6.15 scrub ok 2024-09-17T07:12:07.591 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:07 smithi100 ceph-mon[39462]: 6.10 scrub starts 2024-09-17T07:12:07.591 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:07 smithi100 ceph-mon[39462]: 6.10 scrub ok 2024-09-17T07:12:07.591 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:07 smithi100 ceph-mon[29092]: 6.15 scrub starts 2024-09-17T07:12:07.591 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:07 smithi100 ceph-mon[29092]: 6.15 scrub ok 2024-09-17T07:12:07.591 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:07 smithi100 ceph-mon[29092]: 6.10 scrub starts 2024-09-17T07:12:07.591 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:07 smithi100 ceph-mon[29092]: 6.10 scrub ok 2024-09-17T07:12:07.624 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:12:07.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:07.649 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:07.649 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:12:07.750 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:12:07.774 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:07.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:07.775 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:12:07.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:07 smithi135 ceph-mon[33238]: 6.15 scrub starts 2024-09-17T07:12:07.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:07 smithi135 ceph-mon[33238]: 6.15 scrub ok 2024-09-17T07:12:07.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:07 smithi135 ceph-mon[33238]: 6.10 scrub starts 2024-09-17T07:12:07.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:07 smithi135 ceph-mon[33238]: 6.10 scrub ok 2024-09-17T07:12:07.876 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:12:07.897 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:07.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:07.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:12:07.999 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:12:08.021 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:08.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:08.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:12:08.123 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:12:08.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:08.145 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:08.145 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:12:08.246 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:12:08.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:08.270 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:08.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:12:08.371 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:12:08.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:08.398 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:08.398 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:12:08.499 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:12:08.520 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:08.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:08.521 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:12:08.601 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:08 smithi135 ceph-mon[33238]: pgmap v443: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:12:08.602 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:08 smithi135 ceph-mon[33238]: 4.9 scrub starts 2024-09-17T07:12:08.602 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:08 smithi135 ceph-mon[33238]: 4.9 scrub ok 2024-09-17T07:12:08.622 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:12:08.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:08 smithi100 ceph-mon[29092]: pgmap v443: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:12:08.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:08 smithi100 ceph-mon[29092]: 4.9 scrub starts 2024-09-17T07:12:08.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:08 smithi100 ceph-mon[29092]: 4.9 scrub ok 2024-09-17T07:12:08.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:08 smithi100 ceph-mon[39462]: pgmap v443: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:12:08.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:08 smithi100 ceph-mon[39462]: 4.9 scrub starts 2024-09-17T07:12:08.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:08 smithi100 ceph-mon[39462]: 4.9 scrub ok 2024-09-17T07:12:08.647 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:08.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:08.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:12:08.749 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:12:08.774 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:08.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:08.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:12:08.875 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:12:08.896 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:08.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:08.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:12:08.960 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:12:08.960 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:12:08.975 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1: {} 2024-09-17T07:12:08.975 INFO:teuthology.orchestra.run.smithi100.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:12:08.991 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:12:08.991 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:12:08.998 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:12:09.006 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:12:09.006 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:12:09.021 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:09.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:09.022 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:12:09.022 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:12:09.022 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:12:09.039 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:12:09.039 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:12:09.043 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:12:08 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:12:09.056 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:12:09.056 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:12:09.072 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:12:09.072 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:12:09.123 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:12:09.144 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:09.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:09.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:12:09.246 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:12:09.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:09.268 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:09.269 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:12:09.369 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:12:09.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:09.391 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:09.391 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:12:09.492 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:12:09.514 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:09.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:09.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:12:09.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:09 smithi135 ceph-mon[33238]: 4.4 scrub starts 2024-09-17T07:12:09.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:09 smithi135 ceph-mon[33238]: 4.4 scrub ok 2024-09-17T07:12:09.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:09 smithi135 ceph-mon[33238]: 5.7 scrub starts 2024-09-17T07:12:09.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:09 smithi135 ceph-mon[33238]: 5.7 scrub ok 2024-09-17T07:12:09.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:09 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2507860805' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:12:09.615 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:12:09.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:09 smithi100 ceph-mon[29092]: 4.4 scrub starts 2024-09-17T07:12:09.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:09 smithi100 ceph-mon[29092]: 4.4 scrub ok 2024-09-17T07:12:09.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:09 smithi100 ceph-mon[29092]: 5.7 scrub starts 2024-09-17T07:12:09.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:09 smithi100 ceph-mon[29092]: 5.7 scrub ok 2024-09-17T07:12:09.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:09 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2507860805' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:12:09.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:09 smithi100 ceph-mon[39462]: 4.4 scrub starts 2024-09-17T07:12:09.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:09 smithi100 ceph-mon[39462]: 4.4 scrub ok 2024-09-17T07:12:09.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:09 smithi100 ceph-mon[39462]: 5.7 scrub starts 2024-09-17T07:12:09.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:09 smithi100 ceph-mon[39462]: 5.7 scrub ok 2024-09-17T07:12:09.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:09 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2507860805' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:12:09.638 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:09.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:09.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:12:09.739 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:12:09.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:09.762 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:09.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:12:09.863 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:12:09.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:09.886 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:09.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:12:09.987 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:12:10.007 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:10.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:10.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:12:10.109 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:12:10.131 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:10.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:10.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:12:10.228 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-17T07:12:10.229 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-17T07:12:10.229 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-17T07:12:10.229 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-17T07:12:10.232 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:12:10.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:10.254 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:10.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:12:10.355 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:12:10.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:10 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:12:10.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:10 smithi100 ceph-mon[29092]: pgmap v444: 148 pgs: 4 unknown, 144 active+clean; 582 KiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-17T07:12:10.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:10 smithi100 ceph-mon[29092]: 2.a scrub starts 2024-09-17T07:12:10.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:10 smithi100 ceph-mon[29092]: 2.a scrub ok 2024-09-17T07:12:10.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:10 smithi100 ceph-mon[29092]: 2.12 scrub starts 2024-09-17T07:12:10.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:10 smithi100 ceph-mon[29092]: 2.12 scrub ok 2024-09-17T07:12:10.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:10 smithi100 ceph-mon[29092]: osdmap e239: 8 total, 8 up, 7 in 2024-09-17T07:12:10.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:10 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:12:10.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:10 smithi100 ceph-mon[39462]: pgmap v444: 148 pgs: 4 unknown, 144 active+clean; 582 KiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-17T07:12:10.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:10 smithi100 ceph-mon[39462]: 2.a scrub starts 2024-09-17T07:12:10.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:10 smithi100 ceph-mon[39462]: 2.a scrub ok 2024-09-17T07:12:10.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:10 smithi100 ceph-mon[39462]: 2.12 scrub starts 2024-09-17T07:12:10.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:10 smithi100 ceph-mon[39462]: 2.12 scrub ok 2024-09-17T07:12:10.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:10 smithi100 ceph-mon[39462]: osdmap e239: 8 total, 8 up, 7 in 2024-09-17T07:12:10.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:10.379 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:10.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:12:10.480 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:12:10.501 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:10.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:10.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:12:10.603 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:12:10.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:10.626 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:10.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:12:10.727 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:12:10.749 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:10.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:10.750 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:12:10.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:10 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:12:10.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:10 smithi135 ceph-mon[33238]: pgmap v444: 148 pgs: 4 unknown, 144 active+clean; 582 KiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-17T07:12:10.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:10 smithi135 ceph-mon[33238]: 2.a scrub starts 2024-09-17T07:12:10.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:10 smithi135 ceph-mon[33238]: 2.a scrub ok 2024-09-17T07:12:10.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:10 smithi135 ceph-mon[33238]: 2.12 scrub starts 2024-09-17T07:12:10.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:10 smithi135 ceph-mon[33238]: 2.12 scrub ok 2024-09-17T07:12:10.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:10 smithi135 ceph-mon[33238]: osdmap e239: 8 total, 8 up, 7 in 2024-09-17T07:12:10.850 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:12:10.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:10.872 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:10.872 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:12:10.973 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:12:10.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:10.995 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:10.995 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:12:11.096 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:12:11.117 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:11.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:11.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:12:11.219 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:12:11.241 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:11.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:11.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:12:11.343 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:12:11.365 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:11.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:11.366 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:12:11.467 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:12:11.489 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:11.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:11.490 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:12:11.591 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:12:11.604 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:11 smithi100 ceph-mon[29092]: 2.19 scrub starts 2024-09-17T07:12:11.604 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:11 smithi100 ceph-mon[29092]: 2.19 scrub ok 2024-09-17T07:12:11.604 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:11 smithi100 ceph-mon[29092]: 2.1c scrub starts 2024-09-17T07:12:11.604 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:11 smithi100 ceph-mon[29092]: 2.1c scrub ok 2024-09-17T07:12:11.604 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:11 smithi100 ceph-mon[29092]: 4.15 scrub starts 2024-09-17T07:12:11.604 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:11 smithi100 ceph-mon[29092]: 4.15 scrub ok 2024-09-17T07:12:11.604 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:11 smithi100 ceph-mon[29092]: pgmap v446: 148 pgs: 148 active+clean; 582 KiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 846 B/s rd, 0 op/s 2024-09-17T07:12:11.605 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:11 smithi100 ceph-mon[39462]: 2.19 scrub starts 2024-09-17T07:12:11.605 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:11 smithi100 ceph-mon[39462]: 2.19 scrub ok 2024-09-17T07:12:11.605 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:11 smithi100 ceph-mon[39462]: 2.1c scrub starts 2024-09-17T07:12:11.605 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:11 smithi100 ceph-mon[39462]: 2.1c scrub ok 2024-09-17T07:12:11.605 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:11 smithi100 ceph-mon[39462]: 4.15 scrub starts 2024-09-17T07:12:11.605 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:11 smithi100 ceph-mon[39462]: 4.15 scrub ok 2024-09-17T07:12:11.605 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:11 smithi100 ceph-mon[39462]: pgmap v446: 148 pgs: 148 active+clean; 582 KiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 846 B/s rd, 0 op/s 2024-09-17T07:12:11.614 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:11.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:11.615 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:12:11.716 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:12:11.738 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:11.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:11.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:12:11.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:11 smithi135 ceph-mon[33238]: 2.19 scrub starts 2024-09-17T07:12:11.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:11 smithi135 ceph-mon[33238]: 2.19 scrub ok 2024-09-17T07:12:11.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:11 smithi135 ceph-mon[33238]: 2.1c scrub starts 2024-09-17T07:12:11.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:11 smithi135 ceph-mon[33238]: 2.1c scrub ok 2024-09-17T07:12:11.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:11 smithi135 ceph-mon[33238]: 4.15 scrub starts 2024-09-17T07:12:11.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:11 smithi135 ceph-mon[33238]: 4.15 scrub ok 2024-09-17T07:12:11.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:11 smithi135 ceph-mon[33238]: pgmap v446: 148 pgs: 148 active+clean; 582 KiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 846 B/s rd, 0 op/s 2024-09-17T07:12:11.840 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:12:11.862 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:11.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:11.863 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:12:11.964 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:12:11.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:11.988 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:11.988 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:12:12.089 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:12:12.111 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:12.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:12.112 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:12:12.213 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:12:12.234 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:12.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:12.235 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:12:12.336 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:12:12.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:12.358 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:12.358 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:12:12.459 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:12:12.481 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:12.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:12.482 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:12:12.573 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:12 smithi100 ceph-mon[29092]: 2.1d scrub starts 2024-09-17T07:12:12.573 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:12 smithi100 ceph-mon[29092]: 2.1d scrub ok 2024-09-17T07:12:12.573 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:12 smithi100 ceph-mon[29092]: 4.6 scrub starts 2024-09-17T07:12:12.573 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:12 smithi100 ceph-mon[29092]: 4.6 scrub ok 2024-09-17T07:12:12.573 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:12 smithi100 ceph-mon[29092]: 2.3 scrub starts 2024-09-17T07:12:12.573 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:12 smithi100 ceph-mon[29092]: 2.3 scrub ok 2024-09-17T07:12:12.573 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:12 smithi100 ceph-mon[39462]: 2.1d scrub starts 2024-09-17T07:12:12.574 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:12 smithi100 ceph-mon[39462]: 2.1d scrub ok 2024-09-17T07:12:12.574 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:12 smithi100 ceph-mon[39462]: 4.6 scrub starts 2024-09-17T07:12:12.574 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:12 smithi100 ceph-mon[39462]: 4.6 scrub ok 2024-09-17T07:12:12.574 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:12 smithi100 ceph-mon[39462]: 2.3 scrub starts 2024-09-17T07:12:12.574 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:12 smithi100 ceph-mon[39462]: 2.3 scrub ok 2024-09-17T07:12:12.583 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:12:12.605 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:12.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:12.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:12:12.706 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:12:12.728 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:12.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:12.729 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:12:12.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:12 smithi135 ceph-mon[33238]: 2.1d scrub starts 2024-09-17T07:12:12.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:12 smithi135 ceph-mon[33238]: 2.1d scrub ok 2024-09-17T07:12:12.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:12 smithi135 ceph-mon[33238]: 4.6 scrub starts 2024-09-17T07:12:12.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:12 smithi135 ceph-mon[33238]: 4.6 scrub ok 2024-09-17T07:12:12.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:12 smithi135 ceph-mon[33238]: 2.3 scrub starts 2024-09-17T07:12:12.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:12 smithi135 ceph-mon[33238]: 2.3 scrub ok 2024-09-17T07:12:12.830 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:12:12.852 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:12.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:12.853 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:12:12.954 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:12:12.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:12.976 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:12.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:12:13.077 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:12:13.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:13.099 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:13.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:12:13.200 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:12:13.221 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:13.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:13.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:12:13.322 INFO:teuthology.orchestra.run.smithi100.stderr:nodeep-scrub is unset 2024-09-17T07:12:13.322 INFO:teuthology.orchestra.run.smithi100.stderr:enabled application 'rados' on pool 'unique_pool_2' 2024-09-17T07:12:13.323 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:12:13.343 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:13.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:13.344 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:12:13.445 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:12:13.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:13.469 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:13.469 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:12:13.570 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:12:13.591 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:13.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:13.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:12:13.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:13 smithi100 ceph-mon[29092]: 4.14 scrub starts 2024-09-17T07:12:13.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:13 smithi100 ceph-mon[29092]: 4.14 scrub ok 2024-09-17T07:12:13.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:13 smithi100 ceph-mon[29092]: 6.5 scrub starts 2024-09-17T07:12:13.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:13 smithi100 ceph-mon[29092]: 6.5 scrub ok 2024-09-17T07:12:13.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:13 smithi100 ceph-mon[29092]: 4.b scrub starts 2024-09-17T07:12:13.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:13 smithi100 ceph-mon[29092]: 4.b scrub ok 2024-09-17T07:12:13.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:13 smithi100 ceph-mon[29092]: 5.18 scrub starts 2024-09-17T07:12:13.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:13 smithi100 ceph-mon[29092]: 5.18 scrub ok 2024-09-17T07:12:13.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:13 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/634362563' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-17T07:12:13.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:13 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-17T07:12:13.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:13 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1761578411' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:12:13.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:13 smithi100 ceph-mon[29092]: pgmap v447: 148 pgs: 148 active+clean; 582 KiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-09-17T07:12:13.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:13 smithi100 ceph-mon[39462]: 4.14 scrub starts 2024-09-17T07:12:13.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:13 smithi100 ceph-mon[39462]: 4.14 scrub ok 2024-09-17T07:12:13.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:13 smithi100 ceph-mon[39462]: 6.5 scrub starts 2024-09-17T07:12:13.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:13 smithi100 ceph-mon[39462]: 6.5 scrub ok 2024-09-17T07:12:13.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:13 smithi100 ceph-mon[39462]: 4.b scrub starts 2024-09-17T07:12:13.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:13 smithi100 ceph-mon[39462]: 4.b scrub ok 2024-09-17T07:12:13.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:13 smithi100 ceph-mon[39462]: 5.18 scrub starts 2024-09-17T07:12:13.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:13 smithi100 ceph-mon[39462]: 5.18 scrub ok 2024-09-17T07:12:13.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:13 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/634362563' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-17T07:12:13.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:13 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-17T07:12:13.630 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:13 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1761578411' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:12:13.630 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:13 smithi100 ceph-mon[39462]: pgmap v447: 148 pgs: 148 active+clean; 582 KiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-09-17T07:12:13.693 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:12:13.714 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:13.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:13.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:12:13.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:13 smithi135 ceph-mon[33238]: 4.14 scrub starts 2024-09-17T07:12:13.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:13 smithi135 ceph-mon[33238]: 4.14 scrub ok 2024-09-17T07:12:13.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:13 smithi135 ceph-mon[33238]: 6.5 scrub starts 2024-09-17T07:12:13.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:13 smithi135 ceph-mon[33238]: 6.5 scrub ok 2024-09-17T07:12:13.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:13 smithi135 ceph-mon[33238]: 4.b scrub starts 2024-09-17T07:12:13.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:13 smithi135 ceph-mon[33238]: 4.b scrub ok 2024-09-17T07:12:13.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:13 smithi135 ceph-mon[33238]: 5.18 scrub starts 2024-09-17T07:12:13.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:13 smithi135 ceph-mon[33238]: 5.18 scrub ok 2024-09-17T07:12:13.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:13 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/634362563' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-17T07:12:13.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:13 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-17T07:12:13.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:13 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1761578411' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:12:13.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:13 smithi135 ceph-mon[33238]: pgmap v447: 148 pgs: 148 active+clean; 582 KiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-09-17T07:12:13.817 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:12:13.839 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:13.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:13.840 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:12:13.941 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:12:13.963 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:13.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:13.964 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:12:14.064 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:12:14.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:14.088 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:14.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:12:14.189 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:12:14.210 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:14.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:14.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:12:14.312 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:12:14.325 INFO:teuthology.orchestra.run.smithi100.stderr:reweighted osd.7 to 1 (10000) 2024-09-17T07:12:14.325 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:14 smithi100 ceph-mon[29092]: 6.c scrub starts 2024-09-17T07:12:14.337 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:14.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:14.338 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:12:14.439 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:12:14.459 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:12:14.461 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:14.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:14.461 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:12:14.482 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-17T07:12:14.563 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:12:14.588 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:14.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:14.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:12:14.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:14 smithi100 ceph-mon[39462]: 6.c scrub starts 2024-09-17T07:12:14.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:14 smithi100 ceph-mon[39462]: 6.c scrub ok 2024-09-17T07:12:14.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:14 smithi100 ceph-mon[39462]: 4.f scrub starts 2024-09-17T07:12:14.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:14 smithi100 ceph-mon[39462]: 4.f scrub ok 2024-09-17T07:12:14.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:14 smithi100 ceph-mon[39462]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T07:12:14.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:14 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-17T07:12:14.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:14 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1761578411' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T07:12:14.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:14 smithi100 ceph-mon[39462]: osdmap e240: 8 total, 8 up, 7 in 2024-09-17T07:12:14.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:14 smithi100 ceph-mon[39462]: 2.1 scrub starts 2024-09-17T07:12:14.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:14 smithi100 ceph-mon[39462]: 2.1 scrub ok 2024-09-17T07:12:14.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:14 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1819794540' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-17T07:12:14.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:14 smithi100 ceph-mon[39462]: 4.13 scrub starts 2024-09-17T07:12:14.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:14 smithi100 ceph-mon[39462]: 4.13 scrub ok 2024-09-17T07:12:14.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:14 smithi100 ceph-mon[29092]: 6.c scrub ok 2024-09-17T07:12:14.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:14 smithi100 ceph-mon[29092]: 4.f scrub starts 2024-09-17T07:12:14.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:14 smithi100 ceph-mon[29092]: 4.f scrub ok 2024-09-17T07:12:14.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:14 smithi100 ceph-mon[29092]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T07:12:14.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:14 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-17T07:12:14.629 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:14 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1761578411' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T07:12:14.629 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:14 smithi100 ceph-mon[29092]: osdmap e240: 8 total, 8 up, 7 in 2024-09-17T07:12:14.629 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:14 smithi100 ceph-mon[29092]: 2.1 scrub starts 2024-09-17T07:12:14.629 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:14 smithi100 ceph-mon[29092]: 2.1 scrub ok 2024-09-17T07:12:14.629 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:14 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1819794540' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-17T07:12:14.629 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:14 smithi100 ceph-mon[29092]: 4.13 scrub starts 2024-09-17T07:12:14.629 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:14 smithi100 ceph-mon[29092]: 4.13 scrub ok 2024-09-17T07:12:14.691 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:12:14.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:14.714 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:14.714 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:12:14.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:14 smithi135 ceph-mon[33238]: 6.c scrub starts 2024-09-17T07:12:14.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:14 smithi135 ceph-mon[33238]: 6.c scrub ok 2024-09-17T07:12:14.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:14 smithi135 ceph-mon[33238]: 4.f scrub starts 2024-09-17T07:12:14.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:14 smithi135 ceph-mon[33238]: 4.f scrub ok 2024-09-17T07:12:14.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:14 smithi135 ceph-mon[33238]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T07:12:14.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:14 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-17T07:12:14.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:14 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1761578411' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T07:12:14.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:14 smithi135 ceph-mon[33238]: osdmap e240: 8 total, 8 up, 7 in 2024-09-17T07:12:14.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:14 smithi135 ceph-mon[33238]: 2.1 scrub starts 2024-09-17T07:12:14.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:14 smithi135 ceph-mon[33238]: 2.1 scrub ok 2024-09-17T07:12:14.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:14 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1819794540' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-17T07:12:14.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:14 smithi135 ceph-mon[33238]: 4.13 scrub starts 2024-09-17T07:12:14.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:14 smithi135 ceph-mon[33238]: 4.13 scrub ok 2024-09-17T07:12:14.815 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:12:14.837 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:14.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:14.838 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:12:14.939 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:12:14.961 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:14.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:14.962 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:12:15.063 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:12:15.086 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:15.087 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-17T07:12:15.087 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T07:12:15.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:15.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:12:15.189 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:12:15.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:15.213 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:15.213 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:12:15.314 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:12:15.336 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:15.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:15.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:12:15.437 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:12:15.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:15.461 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:15.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:12:15.562 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:12:15.585 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:15.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:15.586 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:12:15.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:15 smithi100 ceph-mon[29092]: 2.14 scrub starts 2024-09-17T07:12:15.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:15 smithi100 ceph-mon[29092]: 2.14 scrub ok 2024-09-17T07:12:15.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:15 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1819794540' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-17T07:12:15.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:15 smithi100 ceph-mon[29092]: osdmap e241: 8 total, 8 up, 7 in 2024-09-17T07:12:15.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:15 smithi100 ceph-mon[29092]: osdmap e242: 8 total, 8 up, 7 in 2024-09-17T07:12:15.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:15 smithi100 ceph-mon[29092]: pgmap v451: 148 pgs: 148 active+clean; 582 KiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 1013 B/s rd, 0 op/s 2024-09-17T07:12:15.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:15 smithi100 ceph-mon[39462]: 2.14 scrub starts 2024-09-17T07:12:15.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:15 smithi100 ceph-mon[39462]: 2.14 scrub ok 2024-09-17T07:12:15.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:15 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1819794540' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-17T07:12:15.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:15 smithi100 ceph-mon[39462]: osdmap e241: 8 total, 8 up, 7 in 2024-09-17T07:12:15.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:15 smithi100 ceph-mon[39462]: osdmap e242: 8 total, 8 up, 7 in 2024-09-17T07:12:15.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:15 smithi100 ceph-mon[39462]: pgmap v451: 148 pgs: 148 active+clean; 582 KiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 1013 B/s rd, 0 op/s 2024-09-17T07:12:15.686 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:12:15.708 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:15.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:15.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:12:15.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:15 smithi135 ceph-mon[33238]: 2.14 scrub starts 2024-09-17T07:12:15.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:15 smithi135 ceph-mon[33238]: 2.14 scrub ok 2024-09-17T07:12:15.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:15 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1819794540' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-17T07:12:15.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:15 smithi135 ceph-mon[33238]: osdmap e241: 8 total, 8 up, 7 in 2024-09-17T07:12:15.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:15 smithi135 ceph-mon[33238]: osdmap e242: 8 total, 8 up, 7 in 2024-09-17T07:12:15.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:15 smithi135 ceph-mon[33238]: pgmap v451: 148 pgs: 148 active+clean; 582 KiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 1013 B/s rd, 0 op/s 2024-09-17T07:12:15.810 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:12:15.833 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:15.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:15.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:12:15.935 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:12:15.957 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:15.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:15.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:12:16.059 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:12:16.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:16.083 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:16.083 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:12:16.099 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:12:15 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:12:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:12:16.184 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:12:16.186 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:12:16.186 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:12:16.206 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:12:16.212 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:16.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:16.213 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:12:16.314 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:12:16.339 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:16.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:16.340 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:12:16.355 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:16 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:12:16.355 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:16 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:12:16.356 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:16 smithi100 ceph-mon[29092]: 4.18 scrub starts 2024-09-17T07:12:16.356 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:16 smithi100 ceph-mon[29092]: 4.18 scrub ok 2024-09-17T07:12:16.356 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:16 smithi100 ceph-mon[29092]: 2.10 scrub starts 2024-09-17T07:12:16.356 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:16 smithi100 ceph-mon[29092]: 2.10 scrub ok 2024-09-17T07:12:16.356 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:16 smithi100 ceph-mon[29092]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-17T07:12:16.356 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:16 smithi100 ceph-mon[29092]: Cluster is now healthy 2024-09-17T07:12:16.356 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:16 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:12:16.356 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:16 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:12:16.356 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:16 smithi100 ceph-mon[39462]: 4.18 scrub starts 2024-09-17T07:12:16.356 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:16 smithi100 ceph-mon[39462]: 4.18 scrub ok 2024-09-17T07:12:16.357 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:16 smithi100 ceph-mon[39462]: 2.10 scrub starts 2024-09-17T07:12:16.357 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:16 smithi100 ceph-mon[39462]: 2.10 scrub ok 2024-09-17T07:12:16.357 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:16 smithi100 ceph-mon[39462]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-17T07:12:16.357 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:16 smithi100 ceph-mon[39462]: Cluster is now healthy 2024-09-17T07:12:16.441 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:12:16.463 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:16.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:16.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:12:16.565 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:12:16.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:16.588 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:16.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:12:16.689 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:12:16.715 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:16.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:16.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:12:16.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:16 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:12:16.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:16 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:12:16.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:16 smithi135 ceph-mon[33238]: 4.18 scrub starts 2024-09-17T07:12:16.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:16 smithi135 ceph-mon[33238]: 4.18 scrub ok 2024-09-17T07:12:16.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:16 smithi135 ceph-mon[33238]: 2.10 scrub starts 2024-09-17T07:12:16.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:16 smithi135 ceph-mon[33238]: 2.10 scrub ok 2024-09-17T07:12:16.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:16 smithi135 ceph-mon[33238]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-17T07:12:16.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:16 smithi135 ceph-mon[33238]: Cluster is now healthy 2024-09-17T07:12:16.817 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:12:16.843 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:16.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:16.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:12:16.944 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:12:16.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:16.967 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:16.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:12:17.068 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:12:17.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:17.095 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:17.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:12:17.159 DEBUG:teuthology.orchestra.run.smithi100:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_2 bench 90 write' 2024-09-17T07:12:17.162 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-17T07:12:17.163 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-17T07:12:17.196 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:12:17.212 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:hints = 1 2024-09-17T07:12:17.212 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-17T07:12:17.212 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Object prefix: benchmark_data_smithi100_128631 2024-09-17T07:12:17.213 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-17T07:12:17.213 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 0 0 0 0 0 0 - 0 2024-09-17T07:12:17.221 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:17.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:17.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:12:17.325 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:12:17.360 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:17.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:17.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:12:17.462 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:12:17.497 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:17.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:17.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:12:17.599 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:12:17.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:17 smithi100 ceph-mon[29092]: 2.0 scrub starts 2024-09-17T07:12:17.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:17 smithi100 ceph-mon[29092]: 2.0 scrub ok 2024-09-17T07:12:17.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:17 smithi100 ceph-mon[29092]: 6.17 scrub starts 2024-09-17T07:12:17.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:17 smithi100 ceph-mon[29092]: 6.17 scrub ok 2024-09-17T07:12:17.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:17 smithi100 ceph-mon[29092]: 4.16 scrub starts 2024-09-17T07:12:17.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:17 smithi100 ceph-mon[29092]: 4.16 scrub ok 2024-09-17T07:12:17.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:17 smithi100 ceph-mon[29092]: 2.c scrub starts 2024-09-17T07:12:17.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:17 smithi100 ceph-mon[29092]: 2.c scrub ok 2024-09-17T07:12:17.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:17 smithi100 ceph-mon[29092]: pgmap v452: 148 pgs: 148 active+clean; 582 KiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:12:17.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:17 smithi100 ceph-mon[39462]: 2.0 scrub starts 2024-09-17T07:12:17.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:17 smithi100 ceph-mon[39462]: 2.0 scrub ok 2024-09-17T07:12:17.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:17 smithi100 ceph-mon[39462]: 6.17 scrub starts 2024-09-17T07:12:17.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:17 smithi100 ceph-mon[39462]: 6.17 scrub ok 2024-09-17T07:12:17.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:17 smithi100 ceph-mon[39462]: 4.16 scrub starts 2024-09-17T07:12:17.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:17 smithi100 ceph-mon[39462]: 4.16 scrub ok 2024-09-17T07:12:17.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:17 smithi100 ceph-mon[39462]: 2.c scrub starts 2024-09-17T07:12:17.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:17 smithi100 ceph-mon[39462]: 2.c scrub ok 2024-09-17T07:12:17.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:17 smithi100 ceph-mon[39462]: pgmap v452: 148 pgs: 148 active+clean; 582 KiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:12:17.632 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:17.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:17.633 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:12:17.671 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-17T07:12:17.734 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:12:17.771 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:17.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:17.773 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:12:17.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:17 smithi135 ceph-mon[33238]: 2.0 scrub starts 2024-09-17T07:12:17.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:17 smithi135 ceph-mon[33238]: 2.0 scrub ok 2024-09-17T07:12:17.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:17 smithi135 ceph-mon[33238]: 6.17 scrub starts 2024-09-17T07:12:17.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:17 smithi135 ceph-mon[33238]: 6.17 scrub ok 2024-09-17T07:12:17.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:17 smithi135 ceph-mon[33238]: 4.16 scrub starts 2024-09-17T07:12:17.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:17 smithi135 ceph-mon[33238]: 4.16 scrub ok 2024-09-17T07:12:17.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:17 smithi135 ceph-mon[33238]: 2.c scrub starts 2024-09-17T07:12:17.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:17 smithi135 ceph-mon[33238]: 2.c scrub ok 2024-09-17T07:12:17.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:17 smithi135 ceph-mon[33238]: pgmap v452: 148 pgs: 148 active+clean; 582 KiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:12:17.873 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:12:17.909 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:17.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:17.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:12:18.010 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:12:18.045 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:18.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:18.046 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:12:18.147 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:12:18.181 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:18.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:18.182 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:12:18.213 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 1 15 3977 3962 247.612 247.625 0.00365411 0.00402515 2024-09-17T07:12:18.283 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:12:18.317 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:18.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:18.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:12:18.419 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:12:18.454 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:18.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:18.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:12:18.556 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:12:18.591 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:18.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:18.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:12:18.693 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:12:18.725 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:18.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:18.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:12:18.827 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:12:18.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:18.862 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:18.862 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:12:18.963 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:12:19.004 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:19.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:19.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:12:19.043 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:12:18 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:12:19.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:18 smithi135 ceph-mon[33238]: 2.11 scrub starts 2024-09-17T07:12:19.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:18 smithi135 ceph-mon[33238]: 2.11 scrub ok 2024-09-17T07:12:19.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:18 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:12:19.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:18 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:12:19.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:18 smithi135 ceph-mon[33238]: 5.17 deep-scrub starts 2024-09-17T07:12:19.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:18 smithi135 ceph-mon[33238]: 5.17 deep-scrub ok 2024-09-17T07:12:19.102 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:18 smithi100 ceph-mon[29092]: 2.11 scrub starts 2024-09-17T07:12:19.102 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:18 smithi100 ceph-mon[29092]: 2.11 scrub ok 2024-09-17T07:12:19.102 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:18 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:12:19.102 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:18 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:12:19.102 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:18 smithi100 ceph-mon[29092]: 5.17 deep-scrub starts 2024-09-17T07:12:19.102 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:18 smithi100 ceph-mon[29092]: 5.17 deep-scrub ok 2024-09-17T07:12:19.102 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:18 smithi100 ceph-mon[39462]: 2.11 scrub starts 2024-09-17T07:12:19.102 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:18 smithi100 ceph-mon[39462]: 2.11 scrub ok 2024-09-17T07:12:19.102 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:18 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:12:19.103 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:18 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:12:19.103 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:18 smithi100 ceph-mon[39462]: 5.17 deep-scrub starts 2024-09-17T07:12:19.103 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:18 smithi100 ceph-mon[39462]: 5.17 deep-scrub ok 2024-09-17T07:12:19.109 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:12:19.144 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:19.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:19.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:12:19.212 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 2 16 7643 7627 238.327 229.062 0.00420653 0.00418728 2024-09-17T07:12:19.246 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:12:19.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:19.282 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:19.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:12:19.383 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:12:19.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:19.419 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:19.419 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:12:19.520 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:12:19.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:19.561 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:19.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:12:19.662 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:12:19.685 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:19.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:19.686 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:12:19.787 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:12:19.822 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:19.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:19.823 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:12:19.924 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:12:19.964 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:19.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:19.968 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:12:19.988 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:19 smithi100 ceph-mon[29092]: 6.2 scrub starts 2024-09-17T07:12:19.988 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:19 smithi100 ceph-mon[29092]: 6.2 scrub ok 2024-09-17T07:12:19.988 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:19 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:12:19.989 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:19 smithi100 ceph-mon[29092]: 6.7 scrub starts 2024-09-17T07:12:19.989 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:19 smithi100 ceph-mon[29092]: 6.7 scrub ok 2024-09-17T07:12:19.989 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:19 smithi100 ceph-mon[29092]: pgmap v453: 148 pgs: 148 active+clean; 218 MiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1.7 KiB/s rd, 36 MiB/s wr, 580 op/s 2024-09-17T07:12:19.989 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:19 smithi100 ceph-mon[29092]: 4.2 scrub starts 2024-09-17T07:12:19.989 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:19 smithi100 ceph-mon[29092]: 4.2 scrub ok 2024-09-17T07:12:19.989 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:19 smithi100 ceph-mon[39462]: 6.2 scrub starts 2024-09-17T07:12:19.990 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:19 smithi100 ceph-mon[39462]: 6.2 scrub ok 2024-09-17T07:12:19.990 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:19 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:12:19.990 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:19 smithi100 ceph-mon[39462]: 6.7 scrub starts 2024-09-17T07:12:19.990 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:19 smithi100 ceph-mon[39462]: 6.7 scrub ok 2024-09-17T07:12:19.990 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:19 smithi100 ceph-mon[39462]: pgmap v453: 148 pgs: 148 active+clean; 218 MiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1.7 KiB/s rd, 36 MiB/s wr, 580 op/s 2024-09-17T07:12:19.990 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:19 smithi100 ceph-mon[39462]: 4.2 scrub starts 2024-09-17T07:12:19.990 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:19 smithi100 ceph-mon[39462]: 4.2 scrub ok 2024-09-17T07:12:20.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:19 smithi135 ceph-mon[33238]: 6.2 scrub starts 2024-09-17T07:12:20.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:19 smithi135 ceph-mon[33238]: 6.2 scrub ok 2024-09-17T07:12:20.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:19 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:12:20.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:19 smithi135 ceph-mon[33238]: 6.7 scrub starts 2024-09-17T07:12:20.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:19 smithi135 ceph-mon[33238]: 6.7 scrub ok 2024-09-17T07:12:20.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:19 smithi135 ceph-mon[33238]: pgmap v453: 148 pgs: 148 active+clean; 218 MiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1.7 KiB/s rd, 36 MiB/s wr, 580 op/s 2024-09-17T07:12:20.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:19 smithi135 ceph-mon[33238]: 4.2 scrub starts 2024-09-17T07:12:20.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:19 smithi135 ceph-mon[33238]: 4.2 scrub ok 2024-09-17T07:12:20.070 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:12:20.117 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:20.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:20.118 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:12:20.213 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 3 16 10269 10253 213.589 164.125 0.0114195 0.00464362 2024-09-17T07:12:20.219 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:12:20.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:20.245 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:20.245 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:12:20.346 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:12:20.376 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:20.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:20.380 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:12:20.481 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:12:20.518 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:20.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:20.519 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:12:20.579 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:12:20.580 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:12:20.613 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1: {} 2024-09-17T07:12:20.613 INFO:teuthology.orchestra.run.smithi100.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:12:20.621 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:12:20.651 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:12:20.651 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:12:20.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:20.658 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:20.658 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:12:20.685 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:12:20.685 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:12:20.719 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:12:20.720 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:12:20.752 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:12:20.752 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:12:20.759 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:12:20.784 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:12:20.785 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:12:20.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:20.793 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:20.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:12:20.818 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:12:20.818 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:12:20.894 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:12:20.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:20.918 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:20.918 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:12:21.018 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:12:21.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:20 smithi135 ceph-mon[33238]: 4.d scrub starts 2024-09-17T07:12:21.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:20 smithi135 ceph-mon[33238]: 4.d scrub ok 2024-09-17T07:12:21.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:20 smithi135 ceph-mon[33238]: 2.2 scrub starts 2024-09-17T07:12:21.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:20 smithi135 ceph-mon[33238]: 2.2 scrub ok 2024-09-17T07:12:21.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:20 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/482480018' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:12:21.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:21.055 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:21.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:12:21.126 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:20 smithi100 ceph-mon[29092]: 4.d scrub starts 2024-09-17T07:12:21.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:20 smithi100 ceph-mon[29092]: 4.d scrub ok 2024-09-17T07:12:21.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:20 smithi100 ceph-mon[29092]: 2.2 scrub starts 2024-09-17T07:12:21.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:20 smithi100 ceph-mon[29092]: 2.2 scrub ok 2024-09-17T07:12:21.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:20 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/482480018' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:12:21.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:20 smithi100 ceph-mon[39462]: 4.d scrub starts 2024-09-17T07:12:21.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:20 smithi100 ceph-mon[39462]: 4.d scrub ok 2024-09-17T07:12:21.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:20 smithi100 ceph-mon[39462]: 2.2 scrub starts 2024-09-17T07:12:21.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:20 smithi100 ceph-mon[39462]: 2.2 scrub ok 2024-09-17T07:12:21.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:20 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/482480018' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:12:21.156 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:12:21.192 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:21.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:21.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:12:21.213 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 4 16 12690 12674 198.017 151.312 0.00510336 0.00504353 2024-09-17T07:12:21.294 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:12:21.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:21.316 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:21.317 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:12:21.417 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:12:21.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:21.455 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:21.456 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:12:21.556 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:12:21.593 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:21.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:21.594 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:12:21.695 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:12:21.727 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:21.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:21.729 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:12:21.830 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:12:21.863 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:21.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:21.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:12:21.965 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:12:21.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:21.999 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:21.999 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:12:22.100 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:12:22.126 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:21 smithi100 ceph-mon[29092]: 2.e deep-scrub starts 2024-09-17T07:12:22.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:21 smithi100 ceph-mon[29092]: 2.e deep-scrub ok 2024-09-17T07:12:22.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:21 smithi100 ceph-mon[29092]: 2.1f scrub starts 2024-09-17T07:12:22.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:21 smithi100 ceph-mon[29092]: 2.1f scrub ok 2024-09-17T07:12:22.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:21 smithi100 ceph-mon[29092]: 4.a scrub starts 2024-09-17T07:12:22.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:21 smithi100 ceph-mon[29092]: 4.a scrub ok 2024-09-17T07:12:22.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:21 smithi100 ceph-mon[29092]: pgmap v454: 148 pgs: 148 active+clean; 335 MiB data, 3.2 GiB used, 623 GiB / 626 GiB avail; 2.8 KiB/s rd, 42 MiB/s wr, 681 op/s 2024-09-17T07:12:22.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:21 smithi100 ceph-mon[29092]: 6.1d scrub starts 2024-09-17T07:12:22.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:21 smithi100 ceph-mon[29092]: 6.1d scrub ok 2024-09-17T07:12:22.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:21 smithi100 ceph-mon[39462]: 2.e deep-scrub starts 2024-09-17T07:12:22.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:21 smithi100 ceph-mon[39462]: 2.e deep-scrub ok 2024-09-17T07:12:22.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:21 smithi100 ceph-mon[39462]: 2.1f scrub starts 2024-09-17T07:12:22.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:21 smithi100 ceph-mon[39462]: 2.1f scrub ok 2024-09-17T07:12:22.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:21 smithi100 ceph-mon[39462]: 4.a scrub starts 2024-09-17T07:12:22.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:21 smithi100 ceph-mon[39462]: 4.a scrub ok 2024-09-17T07:12:22.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:21 smithi100 ceph-mon[39462]: pgmap v454: 148 pgs: 148 active+clean; 335 MiB data, 3.2 GiB used, 623 GiB / 626 GiB avail; 2.8 KiB/s rd, 42 MiB/s wr, 681 op/s 2024-09-17T07:12:22.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:21 smithi100 ceph-mon[39462]: 6.1d scrub starts 2024-09-17T07:12:22.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:21 smithi100 ceph-mon[39462]: 6.1d scrub ok 2024-09-17T07:12:22.131 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:22.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:22.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:12:22.213 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 5 16 15649 15633 195.398 184.938 0.00517165 0.00511104 2024-09-17T07:12:22.233 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:12:22.266 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:22.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:22.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:12:22.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:21 smithi135 ceph-mon[33238]: 2.e deep-scrub starts 2024-09-17T07:12:22.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:21 smithi135 ceph-mon[33238]: 2.e deep-scrub ok 2024-09-17T07:12:22.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:21 smithi135 ceph-mon[33238]: 2.1f scrub starts 2024-09-17T07:12:22.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:21 smithi135 ceph-mon[33238]: 2.1f scrub ok 2024-09-17T07:12:22.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:21 smithi135 ceph-mon[33238]: 4.a scrub starts 2024-09-17T07:12:22.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:21 smithi135 ceph-mon[33238]: 4.a scrub ok 2024-09-17T07:12:22.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:21 smithi135 ceph-mon[33238]: pgmap v454: 148 pgs: 148 active+clean; 335 MiB data, 3.2 GiB used, 623 GiB / 626 GiB avail; 2.8 KiB/s rd, 42 MiB/s wr, 681 op/s 2024-09-17T07:12:22.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:21 smithi135 ceph-mon[33238]: 6.1d scrub starts 2024-09-17T07:12:22.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:21 smithi135 ceph-mon[33238]: 6.1d scrub ok 2024-09-17T07:12:22.368 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:12:22.407 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:22.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:22.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:12:22.510 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:12:22.547 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:22.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:22.548 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:12:22.648 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:12:22.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:22.686 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:22.687 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:12:22.788 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:12:22.825 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:22 smithi100 ceph-mon[39462]: 6.f scrub starts 2024-09-17T07:12:22.825 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:22 smithi100 ceph-mon[39462]: 6.f scrub ok 2024-09-17T07:12:22.825 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:22 smithi100 ceph-mon[39462]: 2.1a scrub starts 2024-09-17T07:12:22.825 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:22 smithi100 ceph-mon[39462]: 2.1a scrub ok 2024-09-17T07:12:22.825 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:22 smithi100 ceph-mon[39462]: 3.0 scrub starts 2024-09-17T07:12:22.825 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:22 smithi100 ceph-mon[39462]: 3.0 scrub ok 2024-09-17T07:12:22.826 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:22 smithi100 ceph-mon[29092]: 6.f scrub starts 2024-09-17T07:12:22.826 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:22 smithi100 ceph-mon[29092]: 6.f scrub ok 2024-09-17T07:12:22.826 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:22 smithi100 ceph-mon[29092]: 2.1a scrub starts 2024-09-17T07:12:22.826 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:22 smithi100 ceph-mon[29092]: 2.1a scrub ok 2024-09-17T07:12:22.826 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:22 smithi100 ceph-mon[29092]: 3.0 scrub starts 2024-09-17T07:12:22.826 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:22 smithi100 ceph-mon[29092]: 3.0 scrub ok 2024-09-17T07:12:22.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:22.827 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:22.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:12:22.928 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:12:22.945 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:12:22.960 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:22.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:22.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:12:23.001 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-17T07:12:23.001 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:12:23.039 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T07:12:23.043 INFO:teuthology.orchestra.run.smithi100.stderr:dumped all 2024-09-17T07:12:23.062 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:12:23.099 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:23.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:23.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:12:23.201 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:12:23.213 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 6 15 18480 18465 192.329 177 0.00897392 0.00519364 2024-09-17T07:12:23.235 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:23.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:23.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:12:23.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:22 smithi135 ceph-mon[33238]: 6.f scrub starts 2024-09-17T07:12:23.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:22 smithi135 ceph-mon[33238]: 6.f scrub ok 2024-09-17T07:12:23.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:22 smithi135 ceph-mon[33238]: 2.1a scrub starts 2024-09-17T07:12:23.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:22 smithi135 ceph-mon[33238]: 2.1a scrub ok 2024-09-17T07:12:23.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:22 smithi135 ceph-mon[33238]: 3.0 scrub starts 2024-09-17T07:12:23.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:22 smithi135 ceph-mon[33238]: 3.0 scrub ok 2024-09-17T07:12:23.337 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:12:23.371 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:23.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:23.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:12:23.473 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:12:23.507 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:23.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:23.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:12:23.609 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:12:23.644 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:23.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:23.645 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:12:23.746 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:12:23.777 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:23.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:23.778 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:12:23.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:23 smithi100 ceph-mon[29092]: 6.14 scrub starts 2024-09-17T07:12:23.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:23 smithi100 ceph-mon[29092]: 6.14 scrub ok 2024-09-17T07:12:23.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:23 smithi100 ceph-mon[29092]: 6.8 scrub starts 2024-09-17T07:12:23.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:23 smithi100 ceph-mon[29092]: 6.8 scrub ok 2024-09-17T07:12:23.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:23 smithi100 ceph-mon[29092]: 6.0 scrub starts 2024-09-17T07:12:23.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:23 smithi100 ceph-mon[29092]: 6.0 scrub ok 2024-09-17T07:12:23.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:23 smithi100 ceph-mon[29092]: 6.1f scrub starts 2024-09-17T07:12:23.879 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:23 smithi100 ceph-mon[29092]: 6.1f scrub ok 2024-09-17T07:12:23.879 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:23 smithi100 ceph-mon[29092]: from='client.15342 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:12:23.879 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:23 smithi100 ceph-mon[29092]: pgmap v455: 148 pgs: 148 active+clean; 335 MiB data, 3.2 GiB used, 623 GiB / 626 GiB avail; 1.9 KiB/s rd, 38 MiB/s wr, 604 op/s 2024-09-17T07:12:23.879 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:23 smithi100 ceph-mon[29092]: 5.12 scrub starts 2024-09-17T07:12:23.879 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:23 smithi100 ceph-mon[29092]: 5.12 scrub ok 2024-09-17T07:12:23.880 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:23 smithi100 ceph-mon[39462]: 6.14 scrub starts 2024-09-17T07:12:23.880 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:23 smithi100 ceph-mon[39462]: 6.14 scrub ok 2024-09-17T07:12:23.880 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:23 smithi100 ceph-mon[39462]: 6.8 scrub starts 2024-09-17T07:12:23.880 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:23 smithi100 ceph-mon[39462]: 6.8 scrub ok 2024-09-17T07:12:23.881 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:23 smithi100 ceph-mon[39462]: 6.0 scrub starts 2024-09-17T07:12:23.881 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:23 smithi100 ceph-mon[39462]: 6.0 scrub ok 2024-09-17T07:12:23.881 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:23 smithi100 ceph-mon[39462]: 6.1f scrub starts 2024-09-17T07:12:23.881 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:23 smithi100 ceph-mon[39462]: 6.1f scrub ok 2024-09-17T07:12:23.881 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:23 smithi100 ceph-mon[39462]: from='client.15342 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:12:23.881 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:23 smithi100 ceph-mon[39462]: pgmap v455: 148 pgs: 148 active+clean; 335 MiB data, 3.2 GiB used, 623 GiB / 626 GiB avail; 1.9 KiB/s rd, 38 MiB/s wr, 604 op/s 2024-09-17T07:12:23.883 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:23 smithi100 ceph-mon[39462]: 5.12 scrub starts 2024-09-17T07:12:23.883 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:23 smithi100 ceph-mon[39462]: 5.12 scrub ok 2024-09-17T07:12:23.883 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:12:23.918 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:23.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:23.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:12:24.020 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:12:24.052 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:24.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:24.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:12:24.154 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:12:24.193 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:24.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:24.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:12:24.213 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 7 16 21241 21225 189.494 172.5 0.00586657 0.00527266 2024-09-17T07:12:24.289 INFO:teuthology.orchestra.run.smithi100.stdout:{"pg_ready":true,"pg_map":{"version":454,"stamp":"2024-09-17T07:12:21.214717+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":351344918,"num_objects":5548,"num_object_clones":0,"num_object_copies":16644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5548,"num_whiteouts":0,"num_read":1527,"num_read_kb":1373,"num_write":5969,"num_write_kb":344987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":123,"num_bytes_recovered":1773876,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6003,"ondisk_log_size":6003,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":3306708,"kb_used_data":677908,"kb_used_omap":10,"kb_used_meta":2628725,"kb_avail":652909356,"statfs":{"total":671965249536,"available":668579180544,"internally_reserved":0,"allocated":694177792,"data_stored":690902463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11152,"internal_metadata":2691814512},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":350748672,"num_objects":5352,"num_object_clones":0,"num_object_copies":16056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5352,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":5360,"num_write_kb":342528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.896827"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":71,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-17T07:12:13.396591+0000","last_change":"2024-09-17T07:11:10.474903+0000","last_active":"2024-09-17T07:12:13.396591+0000","last_peered":"2024-09-17T07:12:13.396591+0000","last_clean":"2024-09-17T07:12:13.396591+0000","last_became_active":"2024-09-17T07:09:24.401344+0000","last_became_peered":"2024-09-17T07:09:24.401344+0000","last_unstale":"2024-09-17T07:12:13.396591+0000","last_undegraded":"2024-09-17T07:12:13.396591+0000","last_fullsized":"2024-09-17T07:12:13.396591+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:10.474778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:33.644728+0000","last_clean_scrub_stamp":"2024-09-17T07:11:10.474778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:18.994927+0000","scrub_duration":0.004995724,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":220,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-17T07:12:13.321038+0000","last_change":"2024-09-17T07:08:17.332759+0000","last_active":"2024-09-17T07:12:13.321038+0000","last_peered":"2024-09-17T07:12:13.321038+0000","last_clean":"2024-09-17T07:12:13.321038+0000","last_became_active":"2024-09-17T07:02:12.006754+0000","last_became_peered":"2024-09-17T07:02:12.006754+0000","last_unstale":"2024-09-17T07:12:13.321038+0000","last_undegraded":"2024-09-17T07:12:13.321038+0000","last_fullsized":"2024-09-17T07:12:13.321038+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:17.332631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:08:17.332631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023942220000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"75'15","reported_seq":113,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-17T07:12:13.321426+0000","last_change":"2024-09-17T07:09:24.419732+0000","last_active":"2024-09-17T07:12:13.321426+0000","last_peered":"2024-09-17T07:12:13.321426+0000","last_clean":"2024-09-17T07:12:13.321426+0000","last_became_active":"2024-09-17T07:09:24.402042+0000","last_became_peered":"2024-09-17T07:09:24.402042+0000","last_unstale":"2024-09-17T07:12:13.321426+0000","last_undegraded":"2024-09-17T07:12:13.321426+0000","last_fullsized":"2024-09-17T07:12:13.321426+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-09-17T07:08:28.845108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:08:28.845108+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":217,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-17T07:12:14.320746+0000","last_change":"2024-09-17T07:12:12.612582+0000","last_active":"2024-09-17T07:12:14.320746+0000","last_peered":"2024-09-17T07:12:14.320746+0000","last_clean":"2024-09-17T07:12:14.320746+0000","last_became_active":"2024-09-17T07:02:16.014928+0000","last_became_peered":"2024-09-17T07:02:16.014928+0000","last_unstale":"2024-09-17T07:12:14.320746+0000","last_undegraded":"2024-09-17T07:12:14.320746+0000","last_fullsized":"2024-09-17T07:12:14.320746+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:12.612502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:12:12.612502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:17.607137+0000","scrub_duration":0.0012142710000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":71,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-17T07:12:13.321006+0000","last_change":"2024-09-17T07:10:26.564948+0000","last_active":"2024-09-17T07:12:13.321006+0000","last_peered":"2024-09-17T07:12:13.321006+0000","last_clean":"2024-09-17T07:12:13.321006+0000","last_became_active":"2024-09-17T07:09:24.402568+0000","last_became_peered":"2024-09-17T07:09:24.402568+0000","last_unstale":"2024-09-17T07:12:13.321006+0000","last_undegraded":"2024-09-17T07:12:13.321006+0000","last_fullsized":"2024-09-17T07:12:13.321006+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:10:26.564872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:10:26.564872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0011084269999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"75'9","reported_seq":246,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-09-17T07:12:15.549126+0000","last_change":"2024-09-17T07:12:15.549204+0000","last_active":"2024-09-17T07:12:15.549126+0000","last_peered":"2024-09-17T07:12:15.549126+0000","last_clean":"2024-09-17T07:12:15.549126+0000","last_became_active":"2024-09-17T07:02:14.016244+0000","last_became_peered":"2024-09-17T07:02:14.016244+0000","last_unstale":"2024-09-17T07:12:15.549126+0000","last_undegraded":"2024-09-17T07:12:15.549126+0000","last_fullsized":"2024-09-17T07:12:15.549126+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:12:15.549113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:12:15.549113+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:18.330981+0000","scrub_duration":0.0025070779999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":216,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-17T07:12:14.323623+0000","last_change":"2024-09-17T07:09:25.123274+0000","last_active":"2024-09-17T07:12:14.323623+0000","last_peered":"2024-09-17T07:12:14.323623+0000","last_clean":"2024-09-17T07:12:14.323623+0000","last_became_active":"2024-09-17T07:02:16.014380+0000","last_became_peered":"2024-09-17T07:02:16.014380+0000","last_unstale":"2024-09-17T07:12:14.323623+0000","last_undegraded":"2024-09-17T07:12:14.323623+0000","last_fullsized":"2024-09-17T07:12:14.323623+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:25.123158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:09:25.123158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00166512,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":216,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-09-17T07:12:15.548095+0000","last_change":"2024-09-17T07:11:03.641887+0000","last_active":"2024-09-17T07:12:15.548095+0000","last_peered":"2024-09-17T07:12:15.548095+0000","last_clean":"2024-09-17T07:12:15.548095+0000","last_became_active":"2024-09-17T07:02:18.020790+0000","last_became_peered":"2024-09-17T07:02:18.020790+0000","last_unstale":"2024-09-17T07:12:15.548095+0000","last_undegraded":"2024-09-17T07:12:15.548095+0000","last_fullsized":"2024-09-17T07:12:15.548095+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:03.641779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:11:03.641779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0057007170000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":230,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-09-17T07:12:15.261232+0000","last_change":"2024-09-17T07:12:11.365357+0000","last_active":"2024-09-17T07:12:15.261232+0000","last_peered":"2024-09-17T07:12:15.261232+0000","last_clean":"2024-09-17T07:12:15.261232+0000","last_became_active":"2024-09-17T07:02:12.011129+0000","last_became_peered":"2024-09-17T07:02:12.011129+0000","last_unstale":"2024-09-17T07:12:15.261232+0000","last_undegraded":"2024-09-17T07:12:15.261232+0000","last_fullsized":"2024-09-17T07:12:15.261232+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:11.365269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:43.060102+0000","last_clean_scrub_stamp":"2024-09-17T07:12:11.365269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:35.406945+0000","scrub_duration":0.0012842960000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"75'5","reported_seq":245,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-09-17T07:12:15.548250+0000","last_change":"2024-09-17T07:11:02.661284+0000","last_active":"2024-09-17T07:12:15.548250+0000","last_peered":"2024-09-17T07:12:15.548250+0000","last_clean":"2024-09-17T07:12:15.548250+0000","last_became_active":"2024-09-17T07:09:24.396900+0000","last_became_peered":"2024-09-17T07:09:24.396900+0000","last_unstale":"2024-09-17T07:12:15.548250+0000","last_undegraded":"2024-09-17T07:12:15.548250+0000","last_fullsized":"2024-09-17T07:12:15.548250+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-09-17T07:11:02.661069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:11:02.661069+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:30.147338+0000","scrub_duration":0.0086442429999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":226,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-17T07:12:13.396433+0000","last_change":"2024-09-17T07:11:24.287105+0000","last_active":"2024-09-17T07:12:13.396433+0000","last_peered":"2024-09-17T07:12:13.396433+0000","last_clean":"2024-09-17T07:12:13.396433+0000","last_became_active":"2024-09-17T07:02:16.015401+0000","last_became_peered":"2024-09-17T07:02:16.015401+0000","last_unstale":"2024-09-17T07:12:13.396433+0000","last_undegraded":"2024-09-17T07:12:13.396433+0000","last_fullsized":"2024-09-17T07:12:13.396433+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:24.286986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:11:24.286986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:50.052847+0000","scrub_duration":0.0067853130000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":234,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-17T07:12:13.397734+0000","last_change":"2024-09-17T07:11:03.187420+0000","last_active":"2024-09-17T07:12:13.397734+0000","last_peered":"2024-09-17T07:12:13.397734+0000","last_clean":"2024-09-17T07:12:13.397734+0000","last_became_active":"2024-09-17T07:09:24.402616+0000","last_became_peered":"2024-09-17T07:09:24.402616+0000","last_unstale":"2024-09-17T07:12:13.397734+0000","last_undegraded":"2024-09-17T07:12:13.397734+0000","last_fullsized":"2024-09-17T07:12:13.397734+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:03.187280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:11:03.187280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0050576739999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":230,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-09-17T07:12:15.260822+0000","last_change":"2024-09-17T07:12:10.356268+0000","last_active":"2024-09-17T07:12:15.260822+0000","last_peered":"2024-09-17T07:12:15.260822+0000","last_clean":"2024-09-17T07:12:15.260822+0000","last_became_active":"2024-09-17T07:02:12.012234+0000","last_became_peered":"2024-09-17T07:02:12.012234+0000","last_unstale":"2024-09-17T07:12:15.260822+0000","last_undegraded":"2024-09-17T07:12:15.260822+0000","last_fullsized":"2024-09-17T07:12:15.260822+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:10.356011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:45.029247+0000","last_clean_scrub_stamp":"2024-09-17T07:12:10.356011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:12.798127+0000","scrub_duration":0.001613686,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"75'9","reported_seq":254,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-17T07:12:14.321135+0000","last_change":"2024-09-17T07:11:20.847222+0000","last_active":"2024-09-17T07:12:14.321135+0000","last_peered":"2024-09-17T07:12:14.321135+0000","last_clean":"2024-09-17T07:12:14.321135+0000","last_became_active":"2024-09-17T07:09:24.397954+0000","last_became_peered":"2024-09-17T07:09:24.397954+0000","last_unstale":"2024-09-17T07:12:14.321135+0000","last_undegraded":"2024-09-17T07:12:14.321135+0000","last_fullsized":"2024-09-17T07:12:14.321135+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:11:20.847071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:11:20.847071+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:45.513061+0000","scrub_duration":0.0051365669999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":226,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-09-17T07:12:15.164735+0000","last_change":"2024-09-17T07:11:27.107585+0000","last_active":"2024-09-17T07:12:15.164735+0000","last_peered":"2024-09-17T07:12:15.164735+0000","last_clean":"2024-09-17T07:12:15.164735+0000","last_became_active":"2024-09-17T07:02:16.023774+0000","last_became_peered":"2024-09-17T07:02:16.023774+0000","last_unstale":"2024-09-17T07:12:15.164735+0000","last_undegraded":"2024-09-17T07:12:15.164735+0000","last_fullsized":"2024-09-17T07:12:15.164735+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:27.107400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:28.096266+0000","last_clean_scrub_stamp":"2024-09-17T07:11:27.107400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:54.412659+0000","scrub_duration":0.0081054549999999993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":215,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-17T07:12:13.319517+0000","last_change":"2024-09-17T07:11:06.419278+0000","last_active":"2024-09-17T07:12:13.319517+0000","last_peered":"2024-09-17T07:12:13.319517+0000","last_clean":"2024-09-17T07:12:13.319517+0000","last_became_active":"2024-09-17T07:02:18.029178+0000","last_became_peered":"2024-09-17T07:02:18.029178+0000","last_unstale":"2024-09-17T07:12:13.319517+0000","last_undegraded":"2024-09-17T07:12:13.319517+0000","last_fullsized":"2024-09-17T07:12:13.319517+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:06.419134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:11:06.419134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:17.581608+0000","scrub_duration":0.0086498219999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.0","version":"242'310","reported_seq":325,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-09-17T07:12:18.548995+0000","last_change":"2024-09-17T07:12:05.418073+0000","last_active":"2024-09-17T07:12:18.548995+0000","last_peered":"2024-09-17T07:12:18.548995+0000","last_clean":"2024-09-17T07:12:18.548995+0000","last_became_active":"2024-09-17T07:12:05.417841+0000","last_became_peered":"2024-09-17T07:12:05.417841+0000","last_unstale":"2024-09-17T07:12:18.548995+0000","last_undegraded":"2024-09-17T07:12:18.548995+0000","last_fullsized":"2024-09-17T07:12:18.548995+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_clean_scrub_stamp":"2024-09-17T07:12:04.285115+0000","objects_scrubbed":0,"log_size":310,"log_dups_size":0,"ondisk_log_size":310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:09.864746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20316160,"num_objects":310,"num_object_clones":0,"num_object_copies":930,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":310,"num_write_kb":19840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"95'25","reported_seq":255,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-17T07:12:14.321383+0000","last_change":"2024-09-17T07:09:17.694556+0000","last_active":"2024-09-17T07:12:14.321383+0000","last_peered":"2024-09-17T07:12:14.321383+0000","last_clean":"2024-09-17T07:12:14.321383+0000","last_became_active":"2024-09-17T07:02:14.016270+0000","last_became_peered":"2024-09-17T07:02:14.016270+0000","last_unstale":"2024-09-17T07:12:14.321383+0000","last_undegraded":"2024-09-17T07:12:14.321383+0000","last_fullsized":"2024-09-17T07:12:14.321383+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'25","last_scrub_stamp":"2024-09-17T07:09:17.694236+0000","last_deep_scrub":"95'25","last_deep_scrub_stamp":"2024-09-17T07:04:09.450039+0000","last_clean_scrub_stamp":"2024-09-17T07:09:17.694236+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.014129604,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":232,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-17T07:12:13.396969+0000","last_change":"2024-09-17T07:11:13.480849+0000","last_active":"2024-09-17T07:12:13.396969+0000","last_peered":"2024-09-17T07:12:13.396969+0000","last_clean":"2024-09-17T07:12:13.396969+0000","last_became_active":"2024-09-17T07:02:12.011258+0000","last_became_peered":"2024-09-17T07:02:12.011258+0000","last_unstale":"2024-09-17T07:12:13.396969+0000","last_undegraded":"2024-09-17T07:12:13.396969+0000","last_fullsized":"2024-09-17T07:12:13.396969+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:13.480727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:11:13.480727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:31.316242+0000","scrub_duration":0.0080456680000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":216,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-17T07:12:14.322892+0000","last_change":"2024-09-17T07:11:00.256708+0000","last_active":"2024-09-17T07:12:14.322892+0000","last_peered":"2024-09-17T07:12:14.322892+0000","last_clean":"2024-09-17T07:12:14.322892+0000","last_became_active":"2024-09-17T07:02:16.018147+0000","last_became_peered":"2024-09-17T07:02:16.018147+0000","last_unstale":"2024-09-17T07:12:14.322892+0000","last_undegraded":"2024-09-17T07:12:14.322892+0000","last_fullsized":"2024-09-17T07:12:14.322892+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:00.256595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:11:00.256595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040770219999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":63,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-17T07:12:14.321411+0000","last_change":"2024-09-17T07:09:24.397784+0000","last_active":"2024-09-17T07:12:14.321411+0000","last_peered":"2024-09-17T07:12:14.321411+0000","last_clean":"2024-09-17T07:12:14.321411+0000","last_became_active":"2024-09-17T07:09:24.396334+0000","last_became_peered":"2024-09-17T07:09:24.396334+0000","last_unstale":"2024-09-17T07:12:14.321411+0000","last_undegraded":"2024-09-17T07:12:14.321411+0000","last_fullsized":"2024-09-17T07:12:14.321411+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:21.753939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:42.984088+0000","last_clean_scrub_stamp":"2024-09-17T07:09:21.753939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1","version":"242'257","reported_seq":272,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-09-17T07:12:18.240894+0000","last_change":"2024-09-17T07:12:05.172795+0000","last_active":"2024-09-17T07:12:18.240894+0000","last_peered":"2024-09-17T07:12:18.240894+0000","last_clean":"2024-09-17T07:12:18.240894+0000","last_became_active":"2024-09-17T07:12:05.172332+0000","last_became_peered":"2024-09-17T07:12:05.172332+0000","last_unstale":"2024-09-17T07:12:18.240894+0000","last_undegraded":"2024-09-17T07:12:18.240894+0000","last_fullsized":"2024-09-17T07:12:18.240894+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_clean_scrub_stamp":"2024-09-17T07:12:04.285115+0000","objects_scrubbed":0,"log_size":257,"log_dups_size":0,"ondisk_log_size":257,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:27.965693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16842752,"num_objects":257,"num_object_clones":0,"num_object_copies":771,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":257,"num_write_kb":16448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"176'15","reported_seq":271,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-17T07:12:14.321263+0000","last_change":"2024-09-17T07:11:24.918471+0000","last_active":"2024-09-17T07:12:14.321263+0000","last_peered":"2024-09-17T07:12:14.321263+0000","last_clean":"2024-09-17T07:12:14.321263+0000","last_became_active":"2024-09-17T07:09:24.399287+0000","last_became_peered":"2024-09-17T07:09:24.399287+0000","last_unstale":"2024-09-17T07:12:14.321263+0000","last_undegraded":"2024-09-17T07:12:14.321263+0000","last_fullsized":"2024-09-17T07:12:14.321263+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'15","last_scrub_stamp":"2024-09-17T07:11:24.918191+0000","last_deep_scrub":"176'15","last_deep_scrub_stamp":"2024-09-17T07:08:19.633791+0000","last_clean_scrub_stamp":"2024-09-17T07:11:24.918191+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:52.956372+0000","scrub_duration":0.0091725629999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":239,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-17T07:12:14.322800+0000","last_change":"2024-09-17T07:12:09.254527+0000","last_active":"2024-09-17T07:12:14.322800+0000","last_peered":"2024-09-17T07:12:14.322800+0000","last_clean":"2024-09-17T07:12:14.322800+0000","last_became_active":"2024-09-17T07:09:24.406973+0000","last_became_peered":"2024-09-17T07:09:24.406973+0000","last_unstale":"2024-09-17T07:12:14.322800+0000","last_undegraded":"2024-09-17T07:12:14.322800+0000","last_fullsized":"2024-09-17T07:12:14.322800+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:09.254452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:12:09.254452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:36.038881+0000","scrub_duration":0.0013322150000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"98'9","reported_seq":307,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-09-17T07:12:15.259894+0000","last_change":"2024-09-17T07:12:00.407366+0000","last_active":"2024-09-17T07:12:15.259894+0000","last_peered":"2024-09-17T07:12:15.259894+0000","last_clean":"2024-09-17T07:12:15.259894+0000","last_became_active":"2024-09-17T07:04:15.110797+0000","last_became_peered":"2024-09-17T07:04:15.110797+0000","last_unstale":"2024-09-17T07:12:15.259894+0000","last_undegraded":"2024-09-17T07:12:15.259894+0000","last_fullsized":"2024-09-17T07:12:15.259894+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-09-17T07:12:00.407262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:12:00.407262+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:14.851416+0000","scrub_duration":0.002024489,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":226,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-17T07:12:14.321428+0000","last_change":"2024-09-17T07:11:29.774296+0000","last_active":"2024-09-17T07:12:14.321428+0000","last_peered":"2024-09-17T07:12:14.321428+0000","last_clean":"2024-09-17T07:12:14.321428+0000","last_became_active":"2024-09-17T07:02:18.017460+0000","last_became_peered":"2024-09-17T07:02:18.017460+0000","last_unstale":"2024-09-17T07:12:14.321428+0000","last_undegraded":"2024-09-17T07:12:14.321428+0000","last_fullsized":"2024-09-17T07:12:14.321428+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:29.774144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:11:29.774144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:30.365963+0000","scrub_duration":0.013832928,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"242'360","reported_seq":375,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-09-17T07:12:18.550727+0000","last_change":"2024-09-17T07:12:05.416444+0000","last_active":"2024-09-17T07:12:18.550727+0000","last_peered":"2024-09-17T07:12:18.550727+0000","last_clean":"2024-09-17T07:12:18.550727+0000","last_became_active":"2024-09-17T07:12:05.416186+0000","last_became_peered":"2024-09-17T07:12:05.416186+0000","last_unstale":"2024-09-17T07:12:18.550727+0000","last_undegraded":"2024-09-17T07:12:18.550727+0000","last_fullsized":"2024-09-17T07:12:18.550727+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_clean_scrub_stamp":"2024-09-17T07:12:04.285115+0000","objects_scrubbed":0,"log_size":360,"log_dups_size":0,"ondisk_log_size":360,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:25.741377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23592960,"num_objects":360,"num_object_clones":0,"num_object_copies":1080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":360,"num_write_kb":23040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"75'15","reported_seq":278,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-17T07:12:14.323759+0000","last_change":"2024-09-17T07:12:13.306460+0000","last_active":"2024-09-17T07:12:14.323759+0000","last_peered":"2024-09-17T07:12:14.323759+0000","last_clean":"2024-09-17T07:12:14.323759+0000","last_became_active":"2024-09-17T07:09:24.407311+0000","last_became_peered":"2024-09-17T07:09:24.407311+0000","last_unstale":"2024-09-17T07:12:14.323759+0000","last_undegraded":"2024-09-17T07:12:14.323759+0000","last_fullsized":"2024-09-17T07:12:14.323759+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-09-17T07:12:13.306349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:12:13.306349+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:26.012489+0000","scrub_duration":0.0021853620000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":229,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-17T07:12:14.323787+0000","last_change":"2024-09-17T07:09:24.405992+0000","last_active":"2024-09-17T07:12:14.323787+0000","last_peered":"2024-09-17T07:12:14.323787+0000","last_clean":"2024-09-17T07:12:14.323787+0000","last_became_active":"2024-09-17T07:09:24.405655+0000","last_became_peered":"2024-09-17T07:09:24.405655+0000","last_unstale":"2024-09-17T07:12:14.323787+0000","last_undegraded":"2024-09-17T07:12:14.323787+0000","last_fullsized":"2024-09-17T07:12:14.323787+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:20.182256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:09:20.182256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034116369999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"75'8","reported_seq":346,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-09-17T07:12:15.548876+0000","last_change":"2024-09-17T07:11:25.896643+0000","last_active":"2024-09-17T07:12:15.548876+0000","last_peered":"2024-09-17T07:12:15.548876+0000","last_clean":"2024-09-17T07:12:15.548876+0000","last_became_active":"2024-09-17T07:02:16.021762+0000","last_became_peered":"2024-09-17T07:02:16.021762+0000","last_unstale":"2024-09-17T07:12:15.548876+0000","last_undegraded":"2024-09-17T07:12:15.548876+0000","last_fullsized":"2024-09-17T07:12:15.548876+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:11:25.896517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:11:25.896517+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:53.871973+0000","scrub_duration":0.01413413,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":224,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-17T07:12:13.398274+0000","last_change":"2024-09-17T07:11:30.065694+0000","last_active":"2024-09-17T07:12:13.398274+0000","last_peered":"2024-09-17T07:12:13.398274+0000","last_clean":"2024-09-17T07:12:13.398274+0000","last_became_active":"2024-09-17T07:02:18.025332+0000","last_became_peered":"2024-09-17T07:02:18.025332+0000","last_unstale":"2024-09-17T07:12:13.398274+0000","last_undegraded":"2024-09-17T07:12:13.398274+0000","last_fullsized":"2024-09-17T07:12:13.398274+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:30.065556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:11:30.065556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:38.369770+0000","scrub_duration":0.0046561759999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"242'285","reported_seq":299,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-09-17T07:12:18.365091+0000","last_change":"2024-09-17T07:12:05.174275+0000","last_active":"2024-09-17T07:12:18.365091+0000","last_peered":"2024-09-17T07:12:18.365091+0000","last_clean":"2024-09-17T07:12:18.365091+0000","last_became_active":"2024-09-17T07:12:05.174017+0000","last_became_peered":"2024-09-17T07:12:05.174017+0000","last_unstale":"2024-09-17T07:12:18.365091+0000","last_undegraded":"2024-09-17T07:12:18.365091+0000","last_fullsized":"2024-09-17T07:12:18.365091+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_clean_scrub_stamp":"2024-09-17T07:12:04.285115+0000","objects_scrubbed":0,"log_size":285,"log_dups_size":0,"ondisk_log_size":285,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:30.641723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18677760,"num_objects":285,"num_object_clones":0,"num_object_copies":855,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":285,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":285,"num_write_kb":18240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"75'11","reported_seq":249,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-17T07:12:14.321524+0000","last_change":"2024-09-17T07:11:27.816034+0000","last_active":"2024-09-17T07:12:14.321524+0000","last_peered":"2024-09-17T07:12:14.321524+0000","last_clean":"2024-09-17T07:12:14.321524+0000","last_became_active":"2024-09-17T07:02:14.010227+0000","last_became_peered":"2024-09-17T07:02:14.010227+0000","last_unstale":"2024-09-17T07:12:14.321524+0000","last_undegraded":"2024-09-17T07:12:14.321524+0000","last_fullsized":"2024-09-17T07:12:14.321524+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-09-17T07:11:27.815628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:11:27.815628+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:31.961816+0000","scrub_duration":0.0098845579999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":231,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-17T07:12:13.396173+0000","last_change":"2024-09-17T07:11:16.421038+0000","last_active":"2024-09-17T07:12:13.396173+0000","last_peered":"2024-09-17T07:12:13.396173+0000","last_clean":"2024-09-17T07:12:13.396173+0000","last_became_active":"2024-09-17T07:02:12.013182+0000","last_became_peered":"2024-09-17T07:02:12.013182+0000","last_unstale":"2024-09-17T07:12:13.396173+0000","last_undegraded":"2024-09-17T07:12:13.396173+0000","last_fullsized":"2024-09-17T07:12:13.396173+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:16.420367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:11:16.420367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:27.475359+0000","scrub_duration":0.0051482619999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":226,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-17T07:12:13.398441+0000","last_change":"2024-09-17T07:09:24.229155+0000","last_active":"2024-09-17T07:12:13.398441+0000","last_peered":"2024-09-17T07:12:13.398441+0000","last_clean":"2024-09-17T07:12:13.398441+0000","last_became_active":"2024-09-17T07:02:16.020224+0000","last_became_peered":"2024-09-17T07:02:16.020224+0000","last_unstale":"2024-09-17T07:12:13.398441+0000","last_undegraded":"2024-09-17T07:12:13.398441+0000","last_fullsized":"2024-09-17T07:12:13.398441+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:24.229069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:09:24.229069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001199566,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":81,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-17T07:12:14.974761+0000","last_change":"2024-09-17T07:12:12.704448+0000","last_active":"2024-09-17T07:12:14.974761+0000","last_peered":"2024-09-17T07:12:14.974761+0000","last_clean":"2024-09-17T07:12:14.974761+0000","last_became_active":"2024-09-17T07:09:24.408109+0000","last_became_peered":"2024-09-17T07:09:24.408109+0000","last_unstale":"2024-09-17T07:12:14.974761+0000","last_undegraded":"2024-09-17T07:12:14.974761+0000","last_fullsized":"2024-09-17T07:12:14.974761+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:12.704363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:12:12.704363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:17.195681+0000","scrub_duration":0.0011153510000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"242'255","reported_seq":270,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-09-17T07:12:18.242038+0000","last_change":"2024-09-17T07:12:05.174268+0000","last_active":"2024-09-17T07:12:18.242038+0000","last_peered":"2024-09-17T07:12:18.242038+0000","last_clean":"2024-09-17T07:12:18.242038+0000","last_became_active":"2024-09-17T07:12:05.173802+0000","last_became_peered":"2024-09-17T07:12:05.173802+0000","last_unstale":"2024-09-17T07:12:18.242038+0000","last_undegraded":"2024-09-17T07:12:18.242038+0000","last_fullsized":"2024-09-17T07:12:18.242038+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_clean_scrub_stamp":"2024-09-17T07:12:04.285115+0000","objects_scrubbed":0,"log_size":255,"log_dups_size":0,"ondisk_log_size":255,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:14.585969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16711680,"num_objects":255,"num_object_clones":0,"num_object_copies":765,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":255,"num_write_kb":16320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"75'14","reported_seq":240,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-17T07:12:14.320913+0000","last_change":"2024-09-17T07:12:00.638759+0000","last_active":"2024-09-17T07:12:14.320913+0000","last_peered":"2024-09-17T07:12:14.320913+0000","last_clean":"2024-09-17T07:12:14.320913+0000","last_became_active":"2024-09-17T07:02:14.007810+0000","last_became_peered":"2024-09-17T07:02:14.007810+0000","last_unstale":"2024-09-17T07:12:14.320913+0000","last_undegraded":"2024-09-17T07:12:14.320913+0000","last_fullsized":"2024-09-17T07:12:14.320913+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'14","last_scrub_stamp":"2024-09-17T07:12:00.638644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:12:00.638644+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:20.604094+0000","scrub_duration":0.002384817,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":252,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-17T07:12:14.975078+0000","last_change":"2024-09-17T07:12:02.695870+0000","last_active":"2024-09-17T07:12:14.975078+0000","last_peered":"2024-09-17T07:12:14.975078+0000","last_clean":"2024-09-17T07:12:14.975078+0000","last_became_active":"2024-09-17T07:02:12.011766+0000","last_became_peered":"2024-09-17T07:02:12.011766+0000","last_unstale":"2024-09-17T07:12:14.975078+0000","last_undegraded":"2024-09-17T07:12:14.975078+0000","last_fullsized":"2024-09-17T07:12:14.975078+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:02.695765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:09:23.181028+0000","last_clean_scrub_stamp":"2024-09-17T07:12:02.695765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:04.960155+0000","scrub_duration":0.0016435110000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"196'9","reported_seq":118,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-09-17T07:12:15.166294+0000","last_change":"2024-09-17T07:09:24.427192+0000","last_active":"2024-09-17T07:12:15.166294+0000","last_peered":"2024-09-17T07:12:15.166294+0000","last_clean":"2024-09-17T07:12:15.166294+0000","last_became_active":"2024-09-17T07:09:24.406056+0000","last_became_peered":"2024-09-17T07:09:24.406056+0000","last_unstale":"2024-09-17T07:12:15.166294+0000","last_undegraded":"2024-09-17T07:12:15.166294+0000","last_fullsized":"2024-09-17T07:12:15.166294+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:09:14.776794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:09:14.776794+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":224,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-17T07:12:13.396658+0000","last_change":"2024-09-17T07:10:33.727608+0000","last_active":"2024-09-17T07:12:13.396658+0000","last_peered":"2024-09-17T07:12:13.396658+0000","last_clean":"2024-09-17T07:12:13.396658+0000","last_became_active":"2024-09-17T07:02:18.364981+0000","last_became_peered":"2024-09-17T07:02:18.364981+0000","last_unstale":"2024-09-17T07:12:13.396658+0000","last_undegraded":"2024-09-17T07:12:13.396658+0000","last_fullsized":"2024-09-17T07:12:13.396658+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:10:33.727472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:10:33.727472+0000","last_clean_scrub_stamp":"2024-09-17T07:10:33.727472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0087379229999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.d","version":"242'269","reported_seq":283,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-09-17T07:12:18.361237+0000","last_change":"2024-09-17T07:12:05.177946+0000","last_active":"2024-09-17T07:12:18.361237+0000","last_peered":"2024-09-17T07:12:18.361237+0000","last_clean":"2024-09-17T07:12:18.361237+0000","last_became_active":"2024-09-17T07:12:05.177682+0000","last_became_peered":"2024-09-17T07:12:05.177682+0000","last_unstale":"2024-09-17T07:12:18.361237+0000","last_undegraded":"2024-09-17T07:12:18.361237+0000","last_fullsized":"2024-09-17T07:12:18.361237+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_clean_scrub_stamp":"2024-09-17T07:12:04.285115+0000","objects_scrubbed":0,"log_size":269,"log_dups_size":0,"ondisk_log_size":269,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:19.366886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17629184,"num_objects":269,"num_object_clones":0,"num_object_copies":807,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":269,"num_write_kb":17216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"75'18","reported_seq":132,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-17T07:12:13.396019+0000","last_change":"2024-09-17T07:11:19.363446+0000","last_active":"2024-09-17T07:12:13.396019+0000","last_peered":"2024-09-17T07:12:13.396019+0000","last_clean":"2024-09-17T07:12:13.396019+0000","last_became_active":"2024-09-17T07:09:24.403240+0000","last_became_peered":"2024-09-17T07:09:24.403240+0000","last_unstale":"2024-09-17T07:12:13.396019+0000","last_undegraded":"2024-09-17T07:12:13.396019+0000","last_fullsized":"2024-09-17T07:12:13.396019+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'18","last_scrub_stamp":"2024-09-17T07:11:19.363076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:11:19.363076+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:34.602689+0000","scrub_duration":0.010644855,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":33,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":230,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-17T07:12:14.323347+0000","last_change":"2024-09-17T07:11:07.244633+0000","last_active":"2024-09-17T07:12:14.323347+0000","last_peered":"2024-09-17T07:12:14.323347+0000","last_clean":"2024-09-17T07:12:14.323347+0000","last_became_active":"2024-09-17T07:02:12.005687+0000","last_became_peered":"2024-09-17T07:02:12.005687+0000","last_unstale":"2024-09-17T07:12:14.323347+0000","last_undegraded":"2024-09-17T07:12:14.323347+0000","last_fullsized":"2024-09-17T07:12:14.323347+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:07.244521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:11:07.244521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0062381399999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":226,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-17T07:12:14.320834+0000","last_change":"2024-09-17T07:11:26.837485+0000","last_active":"2024-09-17T07:12:14.320834+0000","last_peered":"2024-09-17T07:12:14.320834+0000","last_clean":"2024-09-17T07:12:14.320834+0000","last_became_active":"2024-09-17T07:09:24.393375+0000","last_became_peered":"2024-09-17T07:09:24.393375+0000","last_unstale":"2024-09-17T07:12:14.320834+0000","last_undegraded":"2024-09-17T07:12:14.320834+0000","last_fullsized":"2024-09-17T07:12:14.320834+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:26.837370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:11:26.837370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:27.420201+0000","scrub_duration":0.0042759720000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":224,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-17T07:12:14.321048+0000","last_change":"2024-09-17T07:09:24.399627+0000","last_active":"2024-09-17T07:12:14.321048+0000","last_peered":"2024-09-17T07:12:14.321048+0000","last_clean":"2024-09-17T07:12:14.321048+0000","last_became_active":"2024-09-17T07:09:24.398659+0000","last_became_peered":"2024-09-17T07:09:24.398659+0000","last_unstale":"2024-09-17T07:12:14.321048+0000","last_undegraded":"2024-09-17T07:12:14.321048+0000","last_fullsized":"2024-09-17T07:12:14.321048+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:25.588903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:08:25.588903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021632869999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.e","version":"242'308","reported_seq":322,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-09-17T07:12:18.349245+0000","last_change":"2024-09-17T07:12:05.175609+0000","last_active":"2024-09-17T07:12:18.349245+0000","last_peered":"2024-09-17T07:12:18.349245+0000","last_clean":"2024-09-17T07:12:18.349245+0000","last_became_active":"2024-09-17T07:12:05.175318+0000","last_became_peered":"2024-09-17T07:12:05.175318+0000","last_unstale":"2024-09-17T07:12:18.349245+0000","last_undegraded":"2024-09-17T07:12:18.349245+0000","last_fullsized":"2024-09-17T07:12:18.349245+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_clean_scrub_stamp":"2024-09-17T07:12:04.285115+0000","objects_scrubbed":0,"log_size":308,"log_dups_size":0,"ondisk_log_size":308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:21.393438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20185088,"num_objects":308,"num_object_clones":0,"num_object_copies":924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":308,"num_write_kb":19712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"242'21","reported_seq":258,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-09-17T07:12:19.175373+0000","last_change":"2024-09-17T07:11:03.438378+0000","last_active":"2024-09-17T07:12:19.175373+0000","last_peered":"2024-09-17T07:12:19.175373+0000","last_clean":"2024-09-17T07:12:19.175373+0000","last_became_active":"2024-09-17T07:02:14.027389+0000","last_became_peered":"2024-09-17T07:02:14.027389+0000","last_unstale":"2024-09-17T07:12:19.175373+0000","last_undegraded":"2024-09-17T07:12:19.175373+0000","last_fullsized":"2024-09-17T07:12:19.175373+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-09-17T07:11:03.438281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:11:03.438281+0000","objects_scrubbed":8,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:31.590106+0000","scrub_duration":0.006304324,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":28,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":231,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-17T07:12:13.397667+0000","last_change":"2024-09-17T07:11:15.424520+0000","last_active":"2024-09-17T07:12:13.397667+0000","last_peered":"2024-09-17T07:12:13.397667+0000","last_clean":"2024-09-17T07:12:13.397667+0000","last_became_active":"2024-09-17T07:02:12.011917+0000","last_became_peered":"2024-09-17T07:02:12.011917+0000","last_unstale":"2024-09-17T07:12:13.397667+0000","last_undegraded":"2024-09-17T07:12:13.397667+0000","last_fullsized":"2024-09-17T07:12:13.397667+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:15.424430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:24.072799+0000","last_clean_scrub_stamp":"2024-09-17T07:11:15.424430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:28.799868+0000","scrub_duration":0.0057333330000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":227,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-17T07:12:14.974699+0000","last_change":"2024-09-17T07:11:17.583846+0000","last_active":"2024-09-17T07:12:14.974699+0000","last_peered":"2024-09-17T07:12:14.974699+0000","last_clean":"2024-09-17T07:12:14.974699+0000","last_became_active":"2024-09-17T07:02:16.016199+0000","last_became_peered":"2024-09-17T07:02:16.016199+0000","last_unstale":"2024-09-17T07:12:14.974699+0000","last_undegraded":"2024-09-17T07:12:14.974699+0000","last_fullsized":"2024-09-17T07:12:14.974699+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:17.583712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:11:17.583712+0000","last_clean_scrub_stamp":"2024-09-17T07:11:17.583712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:26.565473+0000","scrub_duration":0.0064424299999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":224,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-17T07:12:14.323547+0000","last_change":"2024-09-17T07:11:28.050865+0000","last_active":"2024-09-17T07:12:14.323547+0000","last_peered":"2024-09-17T07:12:14.323547+0000","last_clean":"2024-09-17T07:12:14.323547+0000","last_became_active":"2024-09-17T07:02:18.364835+0000","last_became_peered":"2024-09-17T07:02:18.364835+0000","last_unstale":"2024-09-17T07:12:14.323547+0000","last_undegraded":"2024-09-17T07:12:14.323547+0000","last_fullsized":"2024-09-17T07:12:14.323547+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:28.050750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:11:28.050750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:47.779865+0000","scrub_duration":0.003023796,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.f","version":"242'273","reported_seq":286,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-09-17T07:12:18.302811+0000","last_change":"2024-09-17T07:12:05.418098+0000","last_active":"2024-09-17T07:12:18.302811+0000","last_peered":"2024-09-17T07:12:18.302811+0000","last_clean":"2024-09-17T07:12:18.302811+0000","last_became_active":"2024-09-17T07:12:05.417843+0000","last_became_peered":"2024-09-17T07:12:05.417843+0000","last_unstale":"2024-09-17T07:12:18.302811+0000","last_undegraded":"2024-09-17T07:12:18.302811+0000","last_fullsized":"2024-09-17T07:12:18.302811+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_clean_scrub_stamp":"2024-09-17T07:12:04.285115+0000","objects_scrubbed":0,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:32.459979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17891328,"num_objects":273,"num_object_clones":0,"num_object_copies":819,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":273,"num_write_kb":17472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"110'15","reported_seq":237,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-17T07:12:14.322684+0000","last_change":"2024-09-17T07:08:19.835062+0000","last_active":"2024-09-17T07:12:14.322684+0000","last_peered":"2024-09-17T07:12:14.322684+0000","last_clean":"2024-09-17T07:12:14.322684+0000","last_became_active":"2024-09-17T07:02:14.010637+0000","last_became_peered":"2024-09-17T07:02:14.010637+0000","last_unstale":"2024-09-17T07:12:14.322684+0000","last_undegraded":"2024-09-17T07:12:14.322684+0000","last_fullsized":"2024-09-17T07:12:14.322684+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'15","last_scrub_stamp":"2024-09-17T07:08:19.834380+0000","last_deep_scrub":"110'15","last_deep_scrub_stamp":"2024-09-17T07:04:57.091400+0000","last_clean_scrub_stamp":"2024-09-17T07:08:19.834380+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0055436599999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":230,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-17T07:12:14.323630+0000","last_change":"2024-09-17T07:12:04.314016+0000","last_active":"2024-09-17T07:12:14.323630+0000","last_peered":"2024-09-17T07:12:14.323630+0000","last_clean":"2024-09-17T07:12:14.323630+0000","last_became_active":"2024-09-17T07:02:12.010803+0000","last_became_peered":"2024-09-17T07:02:12.010803+0000","last_unstale":"2024-09-17T07:12:14.323630+0000","last_undegraded":"2024-09-17T07:12:14.323630+0000","last_fullsized":"2024-09-17T07:12:14.323630+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:04.313886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:25.147878+0000","last_clean_scrub_stamp":"2024-09-17T07:12:04.313886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:27.666148+0000","scrub_duration":0.0018928020000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"89'9","reported_seq":357,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-09-17T07:12:15.552766+0000","last_change":"2024-09-17T07:12:03.588884+0000","last_active":"2024-09-17T07:12:15.552766+0000","last_peered":"2024-09-17T07:12:15.552766+0000","last_clean":"2024-09-17T07:12:15.552766+0000","last_became_active":"2024-09-17T07:02:16.021360+0000","last_became_peered":"2024-09-17T07:02:16.021360+0000","last_unstale":"2024-09-17T07:12:15.552766+0000","last_undegraded":"2024-09-17T07:12:15.552766+0000","last_fullsized":"2024-09-17T07:12:15.552766+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'9","last_scrub_stamp":"2024-09-17T07:12:03.588797+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-09-17T07:03:29.961439+0000","last_clean_scrub_stamp":"2024-09-17T07:12:03.588797+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:21.994712+0000","scrub_duration":0.0018430440000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"75'1","reported_seq":228,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-17T07:12:13.319327+0000","last_change":"2024-09-17T07:09:24.411824+0000","last_active":"2024-09-17T07:12:13.319327+0000","last_peered":"2024-09-17T07:12:13.319327+0000","last_clean":"2024-09-17T07:12:13.319327+0000","last_became_active":"2024-09-17T07:09:24.399565+0000","last_became_peered":"2024-09-17T07:09:24.399565+0000","last_unstale":"2024-09-17T07:12:13.319327+0000","last_undegraded":"2024-09-17T07:12:13.319327+0000","last_fullsized":"2024-09-17T07:12:13.319327+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-17T07:08:25.355935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:08:25.355935+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0052992940000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.9","version":"242'457","reported_seq":471,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-09-17T07:12:19.249990+0000","last_change":"2024-09-17T07:12:05.175204+0000","last_active":"2024-09-17T07:12:19.249990+0000","last_peered":"2024-09-17T07:12:19.249990+0000","last_clean":"2024-09-17T07:12:19.249990+0000","last_became_active":"2024-09-17T07:12:05.174893+0000","last_became_peered":"2024-09-17T07:12:05.174893+0000","last_unstale":"2024-09-17T07:12:19.249990+0000","last_undegraded":"2024-09-17T07:12:19.249990+0000","last_fullsized":"2024-09-17T07:12:19.249990+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_clean_scrub_stamp":"2024-09-17T07:12:04.285115+0000","objects_scrubbed":0,"log_size":457,"log_dups_size":0,"ondisk_log_size":457,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:22.619349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29949952,"num_objects":457,"num_object_clones":0,"num_object_copies":1371,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":457,"num_write_kb":29248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"196'29","reported_seq":428,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-09-17T07:12:15.167911+0000","last_change":"2024-09-17T07:12:08.264127+0000","last_active":"2024-09-17T07:12:15.167911+0000","last_peered":"2024-09-17T07:12:15.167911+0000","last_clean":"2024-09-17T07:12:15.167911+0000","last_became_active":"2024-09-17T07:09:24.403267+0000","last_became_peered":"2024-09-17T07:09:24.403267+0000","last_unstale":"2024-09-17T07:12:15.167911+0000","last_undegraded":"2024-09-17T07:12:15.167911+0000","last_fullsized":"2024-09-17T07:12:15.167911+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'29","last_scrub_stamp":"2024-09-17T07:12:08.264016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:12:08.264016+0000","objects_scrubbed":10,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:30.766851+0000","scrub_duration":0.0026058610000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":230,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-17T07:12:13.398026+0000","last_change":"2024-09-17T07:09:19.253480+0000","last_active":"2024-09-17T07:12:13.398026+0000","last_peered":"2024-09-17T07:12:13.398026+0000","last_clean":"2024-09-17T07:12:13.398026+0000","last_became_active":"2024-09-17T07:02:12.008802+0000","last_became_peered":"2024-09-17T07:02:12.008802+0000","last_unstale":"2024-09-17T07:12:13.398026+0000","last_undegraded":"2024-09-17T07:12:13.398026+0000","last_fullsized":"2024-09-17T07:12:13.398026+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:19.253385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:09:19.253385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017535820000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":227,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-17T07:12:13.320628+0000","last_change":"2024-09-17T07:11:07.433543+0000","last_active":"2024-09-17T07:12:13.320628+0000","last_peered":"2024-09-17T07:12:13.320628+0000","last_clean":"2024-09-17T07:12:13.320628+0000","last_became_active":"2024-09-17T07:02:16.016606+0000","last_became_peered":"2024-09-17T07:02:16.016606+0000","last_unstale":"2024-09-17T07:12:13.320628+0000","last_undegraded":"2024-09-17T07:12:13.320628+0000","last_fullsized":"2024-09-17T07:12:13.320628+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:07.433420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:11:07.433420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012900124000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":71,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-17T07:12:14.321364+0000","last_change":"2024-09-17T07:12:03.666609+0000","last_active":"2024-09-17T07:12:14.321364+0000","last_peered":"2024-09-17T07:12:14.321364+0000","last_clean":"2024-09-17T07:12:14.321364+0000","last_became_active":"2024-09-17T07:09:24.398878+0000","last_became_peered":"2024-09-17T07:09:24.398878+0000","last_unstale":"2024-09-17T07:12:14.321364+0000","last_undegraded":"2024-09-17T07:12:14.321364+0000","last_fullsized":"2024-09-17T07:12:14.321364+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:03.666518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:12:03.666518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:28.852645+0000","scrub_duration":0.0015175169999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.a","version":"242'275","reported_seq":289,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-09-17T07:12:18.295937+0000","last_change":"2024-09-17T07:12:05.175424+0000","last_active":"2024-09-17T07:12:18.295937+0000","last_peered":"2024-09-17T07:12:18.295937+0000","last_clean":"2024-09-17T07:12:18.295937+0000","last_became_active":"2024-09-17T07:12:05.175109+0000","last_became_peered":"2024-09-17T07:12:05.175109+0000","last_unstale":"2024-09-17T07:12:18.295937+0000","last_undegraded":"2024-09-17T07:12:18.295937+0000","last_fullsized":"2024-09-17T07:12:18.295937+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_clean_scrub_stamp":"2024-09-17T07:12:04.285115+0000","objects_scrubbed":0,"log_size":275,"log_dups_size":0,"ondisk_log_size":275,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:23.729980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18022400,"num_objects":275,"num_object_clones":0,"num_object_copies":825,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":275,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":275,"num_write_kb":17600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"75'13","reported_seq":255,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-17T07:12:14.323836+0000","last_change":"2024-09-17T07:11:17.112766+0000","last_active":"2024-09-17T07:12:14.323836+0000","last_peered":"2024-09-17T07:12:14.323836+0000","last_clean":"2024-09-17T07:12:14.323836+0000","last_became_active":"2024-09-17T07:02:14.019838+0000","last_became_peered":"2024-09-17T07:02:14.019838+0000","last_unstale":"2024-09-17T07:12:14.323836+0000","last_undegraded":"2024-09-17T07:12:14.323836+0000","last_fullsized":"2024-09-17T07:12:14.323836+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'13","last_scrub_stamp":"2024-09-17T07:11:17.112635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:11:17.112635+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:21.494810+0000","scrub_duration":0.005881725,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"76'5","reported_seq":827,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-09-17T07:12:18.388462+0000","last_change":"2024-09-17T07:08:29.818116+0000","last_active":"2024-09-17T07:12:18.388462+0000","last_peered":"2024-09-17T07:12:18.388462+0000","last_clean":"2024-09-17T07:12:18.388462+0000","last_became_active":"2024-09-17T07:02:12.006883+0000","last_became_peered":"2024-09-17T07:02:12.006883+0000","last_unstale":"2024-09-17T07:12:18.388462+0000","last_undegraded":"2024-09-17T07:12:18.388462+0000","last_fullsized":"2024-09-17T07:12:18.388462+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'5","last_scrub_stamp":"2024-09-17T07:08:29.817785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:08:29.817785+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034070260000000001,"objects_trimmed":0,"snaptrim_duration":0.00033739799999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":599,"num_read_kb":594,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":72,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-17T07:12:13.536969+0000","last_change":"2024-09-17T07:12:13.536969+0000","last_active":"2024-09-17T07:12:13.536969+0000","last_peered":"2024-09-17T07:12:13.536969+0000","last_clean":"2024-09-17T07:12:13.536969+0000","last_became_active":"2024-09-17T07:09:24.392850+0000","last_became_peered":"2024-09-17T07:09:24.392850+0000","last_unstale":"2024-09-17T07:12:13.536969+0000","last_undegraded":"2024-09-17T07:12:13.536969+0000","last_fullsized":"2024-09-17T07:12:13.536969+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:13.536888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:23.833170+0000","last_clean_scrub_stamp":"2024-09-17T07:12:13.536888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:25.415883+0000","scrub_duration":0.001259055,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":239,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-17T07:12:14.322586+0000","last_change":"2024-09-17T07:12:06.049472+0000","last_active":"2024-09-17T07:12:14.322586+0000","last_peered":"2024-09-17T07:12:14.322586+0000","last_clean":"2024-09-17T07:12:14.322586+0000","last_became_active":"2024-09-17T07:02:16.014492+0000","last_became_peered":"2024-09-17T07:02:16.014492+0000","last_unstale":"2024-09-17T07:12:14.322586+0000","last_undegraded":"2024-09-17T07:12:14.322586+0000","last_fullsized":"2024-09-17T07:12:14.322586+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:06.049391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:12:06.049391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:34.526195+0000","scrub_duration":0.001322546,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":244,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-17T07:12:13.397471+0000","last_change":"2024-09-17T07:12:12.332042+0000","last_active":"2024-09-17T07:12:13.397471+0000","last_peered":"2024-09-17T07:12:13.397471+0000","last_clean":"2024-09-17T07:12:13.397471+0000","last_became_active":"2024-09-17T07:09:24.402704+0000","last_became_peered":"2024-09-17T07:09:24.402704+0000","last_unstale":"2024-09-17T07:12:13.397471+0000","last_undegraded":"2024-09-17T07:12:13.397471+0000","last_fullsized":"2024-09-17T07:12:13.397471+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:12.331949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:12:12.331949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:30.558308+0000","scrub_duration":0.0012857490000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"242'486","reported_seq":500,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-09-17T07:12:19.245580+0000","last_change":"2024-09-17T07:12:05.173856+0000","last_active":"2024-09-17T07:12:19.245580+0000","last_peered":"2024-09-17T07:12:19.245580+0000","last_clean":"2024-09-17T07:12:19.245580+0000","last_became_active":"2024-09-17T07:12:05.173534+0000","last_became_peered":"2024-09-17T07:12:05.173534+0000","last_unstale":"2024-09-17T07:12:19.245580+0000","last_undegraded":"2024-09-17T07:12:19.245580+0000","last_fullsized":"2024-09-17T07:12:19.245580+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_clean_scrub_stamp":"2024-09-17T07:12:04.285115+0000","objects_scrubbed":0,"log_size":486,"log_dups_size":0,"ondisk_log_size":486,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:29.636041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31850496,"num_objects":486,"num_object_clones":0,"num_object_copies":1458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":486,"num_write_kb":31104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"242'14","reported_seq":273,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-09-17T07:12:19.182071+0000","last_change":"2024-09-17T07:12:11.578753+0000","last_active":"2024-09-17T07:12:19.182071+0000","last_peered":"2024-09-17T07:12:19.182071+0000","last_clean":"2024-09-17T07:12:19.182071+0000","last_became_active":"2024-09-17T07:03:48.760405+0000","last_became_peered":"2024-09-17T07:03:48.760405+0000","last_unstale":"2024-09-17T07:12:19.182071+0000","last_undegraded":"2024-09-17T07:12:19.182071+0000","last_fullsized":"2024-09-17T07:12:19.182071+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-09-17T07:12:11.578647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:12:11.578647+0000","objects_scrubbed":6,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:30.740260+0000","scrub_duration":0.0020729070000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":21,"num_read_kb":15,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"70'1","reported_seq":239,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-17T07:12:13.321519+0000","last_change":"2024-09-17T07:09:24.415023+0000","last_active":"2024-09-17T07:12:13.321519+0000","last_peered":"2024-09-17T07:12:13.321519+0000","last_clean":"2024-09-17T07:12:13.321519+0000","last_became_active":"2024-09-17T07:09:24.402499+0000","last_became_peered":"2024-09-17T07:09:24.402499+0000","last_unstale":"2024-09-17T07:12:13.321519+0000","last_undegraded":"2024-09-17T07:12:13.321519+0000","last_fullsized":"2024-09-17T07:12:13.321519+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-17T07:09:22.249833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:09:22.249833+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0061513059999999996,"objects_trimmed":0,"snaptrim_duration":0.0032298629999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":231,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-09-17T07:12:16.273630+0000","last_change":"2024-09-17T07:12:16.273630+0000","last_active":"2024-09-17T07:12:16.273630+0000","last_peered":"2024-09-17T07:12:16.273630+0000","last_clean":"2024-09-17T07:12:16.273630+0000","last_became_active":"2024-09-17T07:02:12.014623+0000","last_became_peered":"2024-09-17T07:02:12.014623+0000","last_unstale":"2024-09-17T07:12:16.273630+0000","last_undegraded":"2024-09-17T07:12:16.273630+0000","last_fullsized":"2024-09-17T07:12:16.273630+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:16.273537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:12:16.273537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:41.553719+0000","scrub_duration":0.001748185,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":226,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-17T07:12:13.398444+0000","last_change":"2024-09-17T07:12:08.807635+0000","last_active":"2024-09-17T07:12:13.398444+0000","last_peered":"2024-09-17T07:12:13.398444+0000","last_clean":"2024-09-17T07:12:13.398444+0000","last_became_active":"2024-09-17T07:02:16.021561+0000","last_became_peered":"2024-09-17T07:02:16.021561+0000","last_unstale":"2024-09-17T07:12:13.398444+0000","last_undegraded":"2024-09-17T07:12:13.398444+0000","last_fullsized":"2024-09-17T07:12:13.398444+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:08.807531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:12:08.807531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:24.781268+0000","scrub_duration":0.0018440049999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":222,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-17T07:12:14.322826+0000","last_change":"2024-09-17T07:09:24.396448+0000","last_active":"2024-09-17T07:12:14.322826+0000","last_peered":"2024-09-17T07:12:14.322826+0000","last_clean":"2024-09-17T07:12:14.322826+0000","last_became_active":"2024-09-17T07:09:24.396154+0000","last_became_peered":"2024-09-17T07:09:24.396154+0000","last_unstale":"2024-09-17T07:12:14.322826+0000","last_undegraded":"2024-09-17T07:12:14.322826+0000","last_fullsized":"2024-09-17T07:12:14.322826+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:17.192238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:09:17.192238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.058290885000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"242'447","reported_seq":461,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-09-17T07:12:19.250077+0000","last_change":"2024-09-17T07:12:05.176237+0000","last_active":"2024-09-17T07:12:19.250077+0000","last_peered":"2024-09-17T07:12:19.250077+0000","last_clean":"2024-09-17T07:12:19.250077+0000","last_became_active":"2024-09-17T07:12:05.175944+0000","last_became_peered":"2024-09-17T07:12:05.175944+0000","last_unstale":"2024-09-17T07:12:19.250077+0000","last_undegraded":"2024-09-17T07:12:19.250077+0000","last_fullsized":"2024-09-17T07:12:19.250077+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_clean_scrub_stamp":"2024-09-17T07:12:04.285115+0000","objects_scrubbed":0,"log_size":447,"log_dups_size":0,"ondisk_log_size":447,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:21.517874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29294592,"num_objects":447,"num_object_clones":0,"num_object_copies":1341,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":447,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":447,"num_write_kb":28608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"75'14","reported_seq":252,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-17T07:12:14.975034+0000","last_change":"2024-09-17T07:11:15.463787+0000","last_active":"2024-09-17T07:12:14.975034+0000","last_peered":"2024-09-17T07:12:14.975034+0000","last_clean":"2024-09-17T07:12:14.975034+0000","last_became_active":"2024-09-17T07:02:14.028857+0000","last_became_peered":"2024-09-17T07:02:14.028857+0000","last_unstale":"2024-09-17T07:12:14.975034+0000","last_undegraded":"2024-09-17T07:12:14.975034+0000","last_fullsized":"2024-09-17T07:12:14.975034+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'14","last_scrub_stamp":"2024-09-17T07:11:15.463520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:11:15.463520+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:27.978530+0000","scrub_duration":0.0066150439999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"72'2","reported_seq":89,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-17T07:12:13.397930+0000","last_change":"2024-09-17T07:12:01.874553+0000","last_active":"2024-09-17T07:12:13.397930+0000","last_peered":"2024-09-17T07:12:13.397930+0000","last_clean":"2024-09-17T07:12:13.397930+0000","last_became_active":"2024-09-17T07:09:24.404532+0000","last_became_peered":"2024-09-17T07:09:24.404532+0000","last_unstale":"2024-09-17T07:12:13.397930+0000","last_undegraded":"2024-09-17T07:12:13.397930+0000","last_fullsized":"2024-09-17T07:12:13.397930+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'2","last_scrub_stamp":"2024-09-17T07:12:01.874430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:12:01.874430+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:16.309823+0000","scrub_duration":0.0018649420000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":241,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-17T07:12:13.398138+0000","last_change":"2024-09-17T07:12:11.818502+0000","last_active":"2024-09-17T07:12:13.398138+0000","last_peered":"2024-09-17T07:12:13.398138+0000","last_clean":"2024-09-17T07:12:13.398138+0000","last_became_active":"2024-09-17T07:02:12.004606+0000","last_became_peered":"2024-09-17T07:02:12.004606+0000","last_unstale":"2024-09-17T07:12:13.398138+0000","last_undegraded":"2024-09-17T07:12:13.398138+0000","last_fullsized":"2024-09-17T07:12:13.398138+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:11.818392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:12:11.818392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:15.462083+0000","scrub_duration":0.0014854499999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"212'130","reported_seq":481,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-17T07:12:13.396319+0000","last_change":"2024-09-17T07:10:37.684858+0000","last_active":"2024-09-17T07:12:13.396319+0000","last_peered":"2024-09-17T07:12:13.396319+0000","last_clean":"2024-09-17T07:12:13.396319+0000","last_became_active":"2024-09-17T07:02:03.020675+0000","last_became_peered":"2024-09-17T07:02:03.020675+0000","last_unstale":"2024-09-17T07:12:13.396319+0000","last_undegraded":"2024-09-17T07:12:13.396319+0000","last_fullsized":"2024-09-17T07:12:13.396319+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:10:37.684115+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:10:37.684115+0000","last_clean_scrub_stamp":"2024-09-17T07:10:37.684115+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.016130123,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":226,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-17T07:12:13.397210+0000","last_change":"2024-09-17T07:10:36.722143+0000","last_active":"2024-09-17T07:12:13.397210+0000","last_peered":"2024-09-17T07:12:13.397210+0000","last_clean":"2024-09-17T07:12:13.397210+0000","last_became_active":"2024-09-17T07:02:16.016031+0000","last_became_peered":"2024-09-17T07:02:16.016031+0000","last_unstale":"2024-09-17T07:12:13.397210+0000","last_undegraded":"2024-09-17T07:12:13.397210+0000","last_fullsized":"2024-09-17T07:12:13.397210+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:10:36.721691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:10:36.721691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0088554370000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":222,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-17T07:12:13.398582+0000","last_change":"2024-09-17T07:09:24.400356+0000","last_active":"2024-09-17T07:12:13.398582+0000","last_peered":"2024-09-17T07:12:13.398582+0000","last_clean":"2024-09-17T07:12:13.398582+0000","last_became_active":"2024-09-17T07:09:24.400041+0000","last_became_peered":"2024-09-17T07:09:24.400041+0000","last_unstale":"2024-09-17T07:12:13.398582+0000","last_undegraded":"2024-09-17T07:12:13.398582+0000","last_fullsized":"2024-09-17T07:12:13.398582+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:15.194496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:58.670571+0000","last_clean_scrub_stamp":"2024-09-17T07:09:15.194496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0091630700000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.7","version":"242'266","reported_seq":280,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-09-17T07:12:18.294477+0000","last_change":"2024-09-17T07:12:05.418324+0000","last_active":"2024-09-17T07:12:18.294477+0000","last_peered":"2024-09-17T07:12:18.294477+0000","last_clean":"2024-09-17T07:12:18.294477+0000","last_became_active":"2024-09-17T07:12:05.417969+0000","last_became_peered":"2024-09-17T07:12:05.417969+0000","last_unstale":"2024-09-17T07:12:18.294477+0000","last_undegraded":"2024-09-17T07:12:18.294477+0000","last_fullsized":"2024-09-17T07:12:18.294477+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_clean_scrub_stamp":"2024-09-17T07:12:04.285115+0000","objects_scrubbed":0,"log_size":266,"log_dups_size":0,"ondisk_log_size":266,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:04.730041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17432576,"num_objects":266,"num_object_clones":0,"num_object_copies":798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":266,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":266,"num_write_kb":17024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"75'19","reported_seq":260,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-17T07:12:14.974756+0000","last_change":"2024-09-17T07:11:16.489331+0000","last_active":"2024-09-17T07:12:14.974756+0000","last_peered":"2024-09-17T07:12:14.974756+0000","last_clean":"2024-09-17T07:12:14.974756+0000","last_became_active":"2024-09-17T07:02:14.027313+0000","last_became_peered":"2024-09-17T07:02:14.027313+0000","last_unstale":"2024-09-17T07:12:14.974756+0000","last_undegraded":"2024-09-17T07:12:14.974756+0000","last_fullsized":"2024-09-17T07:12:14.974756+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-09-17T07:11:16.489206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:11:16.489206+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:17.580670+0000","scrub_duration":0.0036743610000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":255,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-09-17T07:12:17.126848+0000","last_change":"2024-09-17T07:12:17.126950+0000","last_active":"2024-09-17T07:12:17.126848+0000","last_peered":"2024-09-17T07:12:17.126848+0000","last_clean":"2024-09-17T07:12:17.126848+0000","last_became_active":"2024-09-17T07:02:12.012037+0000","last_became_peered":"2024-09-17T07:02:12.012037+0000","last_unstale":"2024-09-17T07:12:17.126848+0000","last_undegraded":"2024-09-17T07:12:17.126848+0000","last_fullsized":"2024-09-17T07:12:17.126848+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:17.126825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:06:39.322535+0000","last_clean_scrub_stamp":"2024-09-17T07:12:17.126825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:20.306203+0000","scrub_duration":0.001773309,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":250,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-17T07:12:14.322697+0000","last_change":"2024-09-17T07:12:04.096069+0000","last_active":"2024-09-17T07:12:14.322697+0000","last_peered":"2024-09-17T07:12:14.322697+0000","last_clean":"2024-09-17T07:12:14.322697+0000","last_became_active":"2024-09-17T07:02:16.014631+0000","last_became_peered":"2024-09-17T07:02:16.014631+0000","last_unstale":"2024-09-17T07:12:14.322697+0000","last_undegraded":"2024-09-17T07:12:14.322697+0000","last_fullsized":"2024-09-17T07:12:14.322697+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:04.095985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:12:04.095985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:31.712454+0000","scrub_duration":0.0015433160000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":222,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-17T07:12:13.396403+0000","last_change":"2024-09-17T07:09:24.404437+0000","last_active":"2024-09-17T07:12:13.396403+0000","last_peered":"2024-09-17T07:12:13.396403+0000","last_clean":"2024-09-17T07:12:13.396403+0000","last_became_active":"2024-09-17T07:09:24.403904+0000","last_became_peered":"2024-09-17T07:09:24.403904+0000","last_unstale":"2024-09-17T07:12:13.396403+0000","last_undegraded":"2024-09-17T07:12:13.396403+0000","last_fullsized":"2024-09-17T07:12:13.396403+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:10.781244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:09:10.781244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010463343,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.6","version":"242'287","reported_seq":301,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-09-17T07:12:18.291127+0000","last_change":"2024-09-17T07:12:05.416479+0000","last_active":"2024-09-17T07:12:18.291127+0000","last_peered":"2024-09-17T07:12:18.291127+0000","last_clean":"2024-09-17T07:12:18.291127+0000","last_became_active":"2024-09-17T07:12:05.416160+0000","last_became_peered":"2024-09-17T07:12:05.416160+0000","last_unstale":"2024-09-17T07:12:18.291127+0000","last_undegraded":"2024-09-17T07:12:18.291127+0000","last_fullsized":"2024-09-17T07:12:18.291127+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_clean_scrub_stamp":"2024-09-17T07:12:04.285115+0000","objects_scrubbed":0,"log_size":287,"log_dups_size":0,"ondisk_log_size":287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:06.915351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18808832,"num_objects":287,"num_object_clones":0,"num_object_copies":861,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":287,"num_write_kb":18368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"75'9","reported_seq":245,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-17T07:12:13.319957+0000","last_change":"2024-09-17T07:12:12.585443+0000","last_active":"2024-09-17T07:12:13.319957+0000","last_peered":"2024-09-17T07:12:13.319957+0000","last_clean":"2024-09-17T07:12:13.319957+0000","last_became_active":"2024-09-17T07:02:14.014939+0000","last_became_peered":"2024-09-17T07:02:14.014939+0000","last_unstale":"2024-09-17T07:12:13.319957+0000","last_undegraded":"2024-09-17T07:12:13.319957+0000","last_fullsized":"2024-09-17T07:12:13.319957+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:12:12.585348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:12:12.585348+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:34.309086+0000","scrub_duration":0.001887445,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":239,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-17T07:12:14.322860+0000","last_change":"2024-09-17T07:11:29.068669+0000","last_active":"2024-09-17T07:12:14.322860+0000","last_peered":"2024-09-17T07:12:14.322860+0000","last_clean":"2024-09-17T07:12:14.322860+0000","last_became_active":"2024-09-17T07:09:24.391207+0000","last_became_peered":"2024-09-17T07:09:24.391207+0000","last_unstale":"2024-09-17T07:12:14.322860+0000","last_undegraded":"2024-09-17T07:12:14.322860+0000","last_fullsized":"2024-09-17T07:12:14.322860+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:29.068555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:06:30.038104+0000","last_clean_scrub_stamp":"2024-09-17T07:11:29.068555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:58.695021+0000","scrub_duration":0.0026579759999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":191,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-17T07:12:14.320922+0000","last_change":"2024-09-17T07:11:08.698817+0000","last_active":"2024-09-17T07:12:14.320922+0000","last_peered":"2024-09-17T07:12:14.320922+0000","last_clean":"2024-09-17T07:12:14.320922+0000","last_became_active":"2024-09-17T07:09:24.396642+0000","last_became_peered":"2024-09-17T07:09:24.396642+0000","last_unstale":"2024-09-17T07:12:14.320922+0000","last_undegraded":"2024-09-17T07:12:14.320922+0000","last_fullsized":"2024-09-17T07:12:14.320922+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:08.698703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:11:08.698703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:33.550225+0000","scrub_duration":0.0063428779999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":214,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-17T07:12:13.321514+0000","last_change":"2024-09-17T07:09:08.316161+0000","last_active":"2024-09-17T07:12:13.321514+0000","last_peered":"2024-09-17T07:12:13.321514+0000","last_clean":"2024-09-17T07:12:13.321514+0000","last_became_active":"2024-09-17T07:02:18.026994+0000","last_became_peered":"2024-09-17T07:02:18.026994+0000","last_unstale":"2024-09-17T07:12:13.321514+0000","last_undegraded":"2024-09-17T07:12:13.321514+0000","last_fullsized":"2024-09-17T07:12:13.321514+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:08.315842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:58.099849+0000","last_clean_scrub_stamp":"2024-09-17T07:09:08.315842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0081844919999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.5","version":"242'486","reported_seq":499,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-09-17T07:12:19.241292+0000","last_change":"2024-09-17T07:12:05.177132+0000","last_active":"2024-09-17T07:12:19.241292+0000","last_peered":"2024-09-17T07:12:19.241292+0000","last_clean":"2024-09-17T07:12:19.241292+0000","last_became_active":"2024-09-17T07:12:05.176729+0000","last_became_peered":"2024-09-17T07:12:05.176729+0000","last_unstale":"2024-09-17T07:12:19.241292+0000","last_undegraded":"2024-09-17T07:12:19.241292+0000","last_fullsized":"2024-09-17T07:12:19.241292+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_clean_scrub_stamp":"2024-09-17T07:12:04.285115+0000","objects_scrubbed":0,"log_size":486,"log_dups_size":0,"ondisk_log_size":486,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:24.650845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31850496,"num_objects":486,"num_object_clones":0,"num_object_copies":1458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":486,"num_write_kb":31104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"111'22","reported_seq":260,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-17T07:12:13.397526+0000","last_change":"2024-09-17T07:10:35.125689+0000","last_active":"2024-09-17T07:12:13.397526+0000","last_peered":"2024-09-17T07:12:13.397526+0000","last_clean":"2024-09-17T07:12:13.397526+0000","last_became_active":"2024-09-17T07:02:14.028392+0000","last_became_peered":"2024-09-17T07:02:14.028392+0000","last_unstale":"2024-09-17T07:12:13.397526+0000","last_undegraded":"2024-09-17T07:12:13.397526+0000","last_fullsized":"2024-09-17T07:12:13.397526+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'22","last_scrub_stamp":"2024-09-17T07:10:35.125303+0000","last_deep_scrub":"111'22","last_deep_scrub_stamp":"2024-09-17T07:04:57.500450+0000","last_clean_scrub_stamp":"2024-09-17T07:10:35.125303+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.019620391000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'2","reported_seq":102,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-17T07:12:13.397096+0000","last_change":"2024-09-17T07:09:24.428473+0000","last_active":"2024-09-17T07:12:13.397096+0000","last_peered":"2024-09-17T07:12:13.397096+0000","last_clean":"2024-09-17T07:12:13.397096+0000","last_became_active":"2024-09-17T07:09:24.402909+0000","last_became_peered":"2024-09-17T07:09:24.402909+0000","last_unstale":"2024-09-17T07:12:13.397096+0000","last_undegraded":"2024-09-17T07:12:13.397096+0000","last_fullsized":"2024-09-17T07:12:13.397096+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-09-17T07:09:19.720425+0000","last_deep_scrub":"89'2","last_deep_scrub_stamp":"2024-09-17T07:03:26.694366+0000","last_clean_scrub_stamp":"2024-09-17T07:09:19.720425+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":420,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":420,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"225'9","reported_seq":359,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-09-17T07:12:15.260357+0000","last_change":"2024-09-17T07:11:25.292700+0000","last_active":"2024-09-17T07:12:15.260357+0000","last_peered":"2024-09-17T07:12:15.260357+0000","last_clean":"2024-09-17T07:12:15.260357+0000","last_became_active":"2024-09-17T07:02:16.015418+0000","last_became_peered":"2024-09-17T07:02:16.015418+0000","last_unstale":"2024-09-17T07:12:15.260357+0000","last_undegraded":"2024-09-17T07:12:15.260357+0000","last_fullsized":"2024-09-17T07:12:15.260357+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"225'9","last_scrub_stamp":"2024-09-17T07:11:25.292565+0000","last_deep_scrub":"225'9","last_deep_scrub_stamp":"2024-09-17T07:11:25.292565+0000","last_clean_scrub_stamp":"2024-09-17T07:11:25.292565+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:54.115350+0000","scrub_duration":0.0070251389999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":224,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-17T07:12:13.397805+0000","last_change":"2024-09-17T07:10:29.121100+0000","last_active":"2024-09-17T07:12:13.397805+0000","last_peered":"2024-09-17T07:12:13.397805+0000","last_clean":"2024-09-17T07:12:13.397805+0000","last_became_active":"2024-09-17T07:02:18.363253+0000","last_became_peered":"2024-09-17T07:02:18.363253+0000","last_unstale":"2024-09-17T07:12:13.397805+0000","last_undegraded":"2024-09-17T07:12:13.397805+0000","last_fullsized":"2024-09-17T07:12:13.397805+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:10:29.120985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:10:29.120985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.017226297000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.4","version":"242'331","reported_seq":346,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-09-17T07:12:18.549399+0000","last_change":"2024-09-17T07:12:05.169555+0000","last_active":"2024-09-17T07:12:18.549399+0000","last_peered":"2024-09-17T07:12:18.549399+0000","last_clean":"2024-09-17T07:12:18.549399+0000","last_became_active":"2024-09-17T07:12:05.169197+0000","last_became_peered":"2024-09-17T07:12:05.169197+0000","last_unstale":"2024-09-17T07:12:18.549399+0000","last_undegraded":"2024-09-17T07:12:18.549399+0000","last_fullsized":"2024-09-17T07:12:18.549399+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_clean_scrub_stamp":"2024-09-17T07:12:04.285115+0000","objects_scrubbed":0,"log_size":331,"log_dups_size":0,"ondisk_log_size":331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:19.862533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21692416,"num_objects":331,"num_object_clones":0,"num_object_copies":993,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":331,"num_write_kb":21184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"150'17","reported_seq":300,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-17T07:12:14.320886+0000","last_change":"2024-09-17T07:12:07.649803+0000","last_active":"2024-09-17T07:12:14.320886+0000","last_peered":"2024-09-17T07:12:14.320886+0000","last_clean":"2024-09-17T07:12:14.320886+0000","last_became_active":"2024-09-17T07:10:49.723292+0000","last_became_peered":"2024-09-17T07:10:49.723292+0000","last_unstale":"2024-09-17T07:12:14.320886+0000","last_undegraded":"2024-09-17T07:12:14.320886+0000","last_fullsized":"2024-09-17T07:12:14.320886+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'17","last_scrub_stamp":"2024-09-17T07:12:07.649697+0000","last_deep_scrub":"150'17","last_deep_scrub_stamp":"2024-09-17T07:06:38.775788+0000","last_clean_scrub_stamp":"2024-09-17T07:12:07.649697+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:32.620327+0000","scrub_duration":0.002192238,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"68'2","reported_seq":244,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-17T07:12:14.321637+0000","last_change":"2024-09-17T07:11:05.636950+0000","last_active":"2024-09-17T07:12:14.321637+0000","last_peered":"2024-09-17T07:12:14.321637+0000","last_clean":"2024-09-17T07:12:14.321637+0000","last_became_active":"2024-09-17T07:10:49.727577+0000","last_became_peered":"2024-09-17T07:10:49.727577+0000","last_unstale":"2024-09-17T07:12:14.321637+0000","last_undegraded":"2024-09-17T07:12:14.321637+0000","last_fullsized":"2024-09-17T07:12:14.321637+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-09-17T07:11:05.636655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:11:05.636655+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:26.915955+0000","scrub_duration":0.013113257999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":191,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-17T07:12:13.319431+0000","last_change":"2024-09-17T07:11:17.479269+0000","last_active":"2024-09-17T07:12:13.319431+0000","last_peered":"2024-09-17T07:12:13.319431+0000","last_clean":"2024-09-17T07:12:13.319431+0000","last_became_active":"2024-09-17T07:04:15.111069+0000","last_became_peered":"2024-09-17T07:04:15.111069+0000","last_unstale":"2024-09-17T07:12:13.319431+0000","last_undegraded":"2024-09-17T07:12:13.319431+0000","last_fullsized":"2024-09-17T07:12:13.319431+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:17.479036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:11:17.479036+0000","last_clean_scrub_stamp":"2024-09-17T07:11:17.479036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:43.456685+0000","scrub_duration":0.0014966910000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":71,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-17T07:12:13.396903+0000","last_change":"2024-09-17T07:11:11.432231+0000","last_active":"2024-09-17T07:12:13.396903+0000","last_peered":"2024-09-17T07:12:13.396903+0000","last_clean":"2024-09-17T07:12:13.396903+0000","last_became_active":"2024-09-17T07:09:24.397669+0000","last_became_peered":"2024-09-17T07:09:24.397669+0000","last_unstale":"2024-09-17T07:12:13.396903+0000","last_undegraded":"2024-09-17T07:12:13.396903+0000","last_fullsized":"2024-09-17T07:12:13.396903+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:11.432111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:11:11.432111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:19.061983+0000","scrub_duration":0.0075412869999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"242'13","reported_seq":270,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-09-17T07:12:19.200461+0000","last_change":"2024-09-17T07:12:16.562906+0000","last_active":"2024-09-17T07:12:19.200461+0000","last_peered":"2024-09-17T07:12:19.200461+0000","last_clean":"2024-09-17T07:12:19.200461+0000","last_became_active":"2024-09-17T07:09:24.408875+0000","last_became_peered":"2024-09-17T07:09:24.408875+0000","last_unstale":"2024-09-17T07:12:19.200461+0000","last_undegraded":"2024-09-17T07:12:19.200461+0000","last_fullsized":"2024-09-17T07:12:19.200461+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:12:16.562818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:12:16.562818+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:26.004316+0000","scrub_duration":0.002298876,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":23,"num_read_kb":17,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":255,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-09-17T07:12:16.090135+0000","last_change":"2024-09-17T07:12:16.090212+0000","last_active":"2024-09-17T07:12:16.090135+0000","last_peered":"2024-09-17T07:12:16.090135+0000","last_clean":"2024-09-17T07:12:16.090135+0000","last_became_active":"2024-09-17T07:02:12.011327+0000","last_became_peered":"2024-09-17T07:02:12.011327+0000","last_unstale":"2024-09-17T07:12:16.090135+0000","last_undegraded":"2024-09-17T07:12:16.090135+0000","last_fullsized":"2024-09-17T07:12:16.090135+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:16.090110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:17.976267+0000","last_clean_scrub_stamp":"2024-09-17T07:12:16.090110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:41.373062+0000","scrub_duration":0.0012389359999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":70,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-09-17T07:12:18.367516+0000","last_change":"2024-09-17T07:12:18.367610+0000","last_active":"2024-09-17T07:12:18.367516+0000","last_peered":"2024-09-17T07:12:18.367516+0000","last_clean":"2024-09-17T07:12:18.367516+0000","last_became_active":"2024-09-17T07:09:24.407806+0000","last_became_peered":"2024-09-17T07:09:24.407806+0000","last_unstale":"2024-09-17T07:12:18.367516+0000","last_undegraded":"2024-09-17T07:12:18.367516+0000","last_fullsized":"2024-09-17T07:12:18.367516+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:18.367486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:18.367486+0000","last_clean_scrub_stamp":"2024-09-17T07:12:18.367486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:44.237351+0000","scrub_duration":0.0038802519999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":249,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-17T07:12:14.322873+0000","last_change":"2024-09-17T07:11:08.520927+0000","last_active":"2024-09-17T07:12:14.322873+0000","last_peered":"2024-09-17T07:12:14.322873+0000","last_clean":"2024-09-17T07:12:14.322873+0000","last_became_active":"2024-09-17T07:02:18.017825+0000","last_became_peered":"2024-09-17T07:02:18.017825+0000","last_unstale":"2024-09-17T07:12:14.322873+0000","last_undegraded":"2024-09-17T07:12:14.322873+0000","last_fullsized":"2024-09-17T07:12:14.322873+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:08.520823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:09:23.378805+0000","last_clean_scrub_stamp":"2024-09-17T07:11:08.520823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:18.882924+0000","scrub_duration":0.0057314200000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":243,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-09-17T07:12:17.648073+0000","last_change":"2024-09-17T07:12:17.648175+0000","last_active":"2024-09-17T07:12:17.648073+0000","last_peered":"2024-09-17T07:12:17.648073+0000","last_clean":"2024-09-17T07:12:17.648073+0000","last_became_active":"2024-09-17T07:02:12.008065+0000","last_became_peered":"2024-09-17T07:02:12.008065+0000","last_unstale":"2024-09-17T07:12:17.648073+0000","last_undegraded":"2024-09-17T07:12:17.648073+0000","last_fullsized":"2024-09-17T07:12:17.648073+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:17.648051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:12:17.648051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:33.995482+0000","scrub_duration":0.0030045190000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"75'6","reported_seq":99,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-17T07:12:13.396832+0000","last_change":"2024-09-17T07:12:03.380136+0000","last_active":"2024-09-17T07:12:13.396832+0000","last_peered":"2024-09-17T07:12:13.396832+0000","last_clean":"2024-09-17T07:12:13.396832+0000","last_became_active":"2024-09-17T07:09:24.405385+0000","last_became_peered":"2024-09-17T07:09:24.405385+0000","last_unstale":"2024-09-17T07:12:13.396832+0000","last_undegraded":"2024-09-17T07:12:13.396832+0000","last_fullsized":"2024-09-17T07:12:13.396832+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'6","last_scrub_stamp":"2024-09-17T07:12:03.380040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:12:03.380040+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:31.256725+0000","scrub_duration":0.0019381719999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":235,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-17T07:12:13.398417+0000","last_change":"2024-09-17T07:11:16.112334+0000","last_active":"2024-09-17T07:12:13.398417+0000","last_peered":"2024-09-17T07:12:13.398417+0000","last_clean":"2024-09-17T07:12:13.398417+0000","last_became_active":"2024-09-17T07:09:24.398922+0000","last_became_peered":"2024-09-17T07:09:24.398922+0000","last_unstale":"2024-09-17T07:12:13.398417+0000","last_undegraded":"2024-09-17T07:12:13.398417+0000","last_fullsized":"2024-09-17T07:12:13.398417+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:16.112226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:11:16.112226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:24.816384+0000","scrub_duration":0.0062666659999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":224,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-17T07:12:13.397420+0000","last_change":"2024-09-17T07:12:06.344966+0000","last_active":"2024-09-17T07:12:13.397420+0000","last_peered":"2024-09-17T07:12:13.397420+0000","last_clean":"2024-09-17T07:12:13.397420+0000","last_became_active":"2024-09-17T07:02:18.365419+0000","last_became_peered":"2024-09-17T07:02:18.365419+0000","last_unstale":"2024-09-17T07:12:13.397420+0000","last_undegraded":"2024-09-17T07:12:13.397420+0000","last_fullsized":"2024-09-17T07:12:13.397420+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:06.344833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:12:06.344833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:19.489450+0000","scrub_duration":0.002302817,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"75'10","reported_seq":110,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-17T07:12:14.323709+0000","last_change":"2024-09-17T07:12:12.288945+0000","last_active":"2024-09-17T07:12:14.323709+0000","last_peered":"2024-09-17T07:12:14.323709+0000","last_clean":"2024-09-17T07:12:14.323709+0000","last_became_active":"2024-09-17T07:09:24.405944+0000","last_became_peered":"2024-09-17T07:09:24.405944+0000","last_unstale":"2024-09-17T07:12:14.323709+0000","last_undegraded":"2024-09-17T07:12:14.323709+0000","last_fullsized":"2024-09-17T07:12:14.323709+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-09-17T07:12:12.288864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:12:12.288864+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:17.960594+0000","scrub_duration":0.0020069329999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":250,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-17T07:12:13.398006+0000","last_change":"2024-09-17T07:12:09.840697+0000","last_active":"2024-09-17T07:12:13.398006+0000","last_peered":"2024-09-17T07:12:13.398006+0000","last_clean":"2024-09-17T07:12:13.398006+0000","last_became_active":"2024-09-17T07:09:24.402355+0000","last_became_peered":"2024-09-17T07:09:24.402355+0000","last_unstale":"2024-09-17T07:12:13.398006+0000","last_undegraded":"2024-09-17T07:12:13.398006+0000","last_fullsized":"2024-09-17T07:12:13.398006+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:09.840613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:12:09.840613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:16.811598+0000","scrub_duration":0.0067048560000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"163'9","reported_seq":356,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-09-17T07:12:15.165399+0000","last_change":"2024-09-17T07:11:08.147847+0000","last_active":"2024-09-17T07:12:15.165399+0000","last_peered":"2024-09-17T07:12:15.165399+0000","last_clean":"2024-09-17T07:12:15.165399+0000","last_became_active":"2024-09-17T07:02:16.022583+0000","last_became_peered":"2024-09-17T07:02:16.022583+0000","last_unstale":"2024-09-17T07:12:15.165399+0000","last_undegraded":"2024-09-17T07:12:15.165399+0000","last_fullsized":"2024-09-17T07:12:15.165399+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'9","last_scrub_stamp":"2024-09-17T07:11:08.147466+0000","last_deep_scrub":"163'9","last_deep_scrub_stamp":"2024-09-17T07:07:34.933808+0000","last_clean_scrub_stamp":"2024-09-17T07:11:08.147466+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:29.005486+0000","scrub_duration":0.039751243999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":223,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-17T07:12:13.320604+0000","last_change":"2024-09-17T07:09:24.409388+0000","last_active":"2024-09-17T07:12:13.320604+0000","last_peered":"2024-09-17T07:12:13.320604+0000","last_clean":"2024-09-17T07:12:13.320604+0000","last_became_active":"2024-09-17T07:09:24.409051+0000","last_became_peered":"2024-09-17T07:09:24.409051+0000","last_unstale":"2024-09-17T07:12:13.320604+0000","last_undegraded":"2024-09-17T07:12:13.320604+0000","last_fullsized":"2024-09-17T07:12:13.320604+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:12.338453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:09:12.338453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0011356249999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"75'9","reported_seq":274,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-17T07:12:13.398448+0000","last_change":"2024-09-17T07:12:10.828225+0000","last_active":"2024-09-17T07:12:13.398448+0000","last_peered":"2024-09-17T07:12:13.398448+0000","last_clean":"2024-09-17T07:12:13.398448+0000","last_became_active":"2024-09-17T07:09:24.406333+0000","last_became_peered":"2024-09-17T07:09:24.406333+0000","last_unstale":"2024-09-17T07:12:13.398448+0000","last_undegraded":"2024-09-17T07:12:13.398448+0000","last_fullsized":"2024-09-17T07:12:13.398448+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:12:10.828114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:12:10.828114+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:28.226498+0000","scrub_duration":0.001735435,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":229,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-17T07:12:13.320162+0000","last_change":"2024-09-17T07:09:24.402987+0000","last_active":"2024-09-17T07:12:13.320162+0000","last_peered":"2024-09-17T07:12:13.320162+0000","last_clean":"2024-09-17T07:12:13.320162+0000","last_became_active":"2024-09-17T07:09:24.402704+0000","last_became_peered":"2024-09-17T07:09:24.402704+0000","last_unstale":"2024-09-17T07:12:13.320162+0000","last_undegraded":"2024-09-17T07:12:13.320162+0000","last_fullsized":"2024-09-17T07:12:13.320162+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:15.356187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:08:15.356187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015925290000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"196'9","reported_seq":130,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-09-17T07:12:15.260818+0000","last_change":"2024-09-17T07:11:02.536969+0000","last_active":"2024-09-17T07:12:15.260818+0000","last_peered":"2024-09-17T07:12:15.260818+0000","last_clean":"2024-09-17T07:12:15.260818+0000","last_became_active":"2024-09-17T07:09:24.406267+0000","last_became_peered":"2024-09-17T07:09:24.406267+0000","last_unstale":"2024-09-17T07:12:15.260818+0000","last_undegraded":"2024-09-17T07:12:15.260818+0000","last_fullsized":"2024-09-17T07:12:15.260818+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'9","last_scrub_stamp":"2024-09-17T07:11:02.536718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:11:02.536718+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:18.053847+0000","scrub_duration":0.0054409640000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":226,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-09-17T07:12:16.543274+0000","last_change":"2024-09-17T07:12:16.543360+0000","last_active":"2024-09-17T07:12:16.543274+0000","last_peered":"2024-09-17T07:12:16.543274+0000","last_clean":"2024-09-17T07:12:16.543274+0000","last_became_active":"2024-09-17T07:02:18.018716+0000","last_became_peered":"2024-09-17T07:02:18.018716+0000","last_unstale":"2024-09-17T07:12:16.543274+0000","last_undegraded":"2024-09-17T07:12:16.543274+0000","last_fullsized":"2024-09-17T07:12:16.543274+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:16.543257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:12:16.543257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:33.805404+0000","scrub_duration":0.001567354,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"75'9","reported_seq":244,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-17T07:12:14.322609+0000","last_change":"2024-09-17T07:11:21.246333+0000","last_active":"2024-09-17T07:12:14.322609+0000","last_peered":"2024-09-17T07:12:14.322609+0000","last_clean":"2024-09-17T07:12:14.322609+0000","last_became_active":"2024-09-17T07:02:14.010289+0000","last_became_peered":"2024-09-17T07:02:14.010289+0000","last_unstale":"2024-09-17T07:12:14.322609+0000","last_undegraded":"2024-09-17T07:12:14.322609+0000","last_fullsized":"2024-09-17T07:12:14.322609+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:11:21.246183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:11:21.246183+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:32.839381+0000","scrub_duration":0.005053564,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":251,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-17T07:12:14.974998+0000","last_change":"2024-09-17T07:12:13.677093+0000","last_active":"2024-09-17T07:12:14.974998+0000","last_peered":"2024-09-17T07:12:14.974998+0000","last_clean":"2024-09-17T07:12:14.974998+0000","last_became_active":"2024-09-17T07:09:24.408139+0000","last_became_peered":"2024-09-17T07:09:24.408139+0000","last_unstale":"2024-09-17T07:12:14.974998+0000","last_undegraded":"2024-09-17T07:12:14.974998+0000","last_fullsized":"2024-09-17T07:12:14.974998+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:13.677010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:10:36.334827+0000","last_clean_scrub_stamp":"2024-09-17T07:12:13.677010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:14.451688+0000","scrub_duration":0.0012308760000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":71,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-17T07:12:13.319678+0000","last_change":"2024-09-17T07:11:04.420764+0000","last_active":"2024-09-17T07:12:13.319678+0000","last_peered":"2024-09-17T07:12:13.319678+0000","last_clean":"2024-09-17T07:12:13.319678+0000","last_became_active":"2024-09-17T07:09:24.399116+0000","last_became_peered":"2024-09-17T07:09:24.399116+0000","last_unstale":"2024-09-17T07:12:13.319678+0000","last_undegraded":"2024-09-17T07:12:13.319678+0000","last_fullsized":"2024-09-17T07:12:13.319678+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:04.420639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:11:04.420639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011748066999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":225,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-17T07:12:13.319446+0000","last_change":"2024-09-17T07:12:06.573120+0000","last_active":"2024-09-17T07:12:13.319446+0000","last_peered":"2024-09-17T07:12:13.319446+0000","last_clean":"2024-09-17T07:12:13.319446+0000","last_became_active":"2024-09-17T07:02:18.027622+0000","last_became_peered":"2024-09-17T07:02:18.027622+0000","last_unstale":"2024-09-17T07:12:13.319446+0000","last_undegraded":"2024-09-17T07:12:13.319446+0000","last_fullsized":"2024-09-17T07:12:13.319446+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:06.573041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:12:06.573041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:11.909768+0000","scrub_duration":0.0024674409999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"75'11","reported_seq":249,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-17T07:12:14.320962+0000","last_change":"2024-09-17T07:12:13.566691+0000","last_active":"2024-09-17T07:12:14.320962+0000","last_peered":"2024-09-17T07:12:14.320962+0000","last_clean":"2024-09-17T07:12:14.320962+0000","last_became_active":"2024-09-17T07:02:14.016255+0000","last_became_peered":"2024-09-17T07:02:14.016255+0000","last_unstale":"2024-09-17T07:12:14.320962+0000","last_undegraded":"2024-09-17T07:12:14.320962+0000","last_fullsized":"2024-09-17T07:12:14.320962+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-09-17T07:12:13.566608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:12:13.566608+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:36.971016+0000","scrub_duration":0.0019196129999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"94'2","reported_seq":225,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-17T07:12:14.322781+0000","last_change":"2024-09-17T07:09:18.169710+0000","last_active":"2024-09-17T07:12:14.322781+0000","last_peered":"2024-09-17T07:12:14.322781+0000","last_clean":"2024-09-17T07:12:14.322781+0000","last_became_active":"2024-09-17T07:02:12.011461+0000","last_became_peered":"2024-09-17T07:02:12.011461+0000","last_unstale":"2024-09-17T07:12:14.322781+0000","last_undegraded":"2024-09-17T07:12:14.322781+0000","last_fullsized":"2024-09-17T07:12:14.322781+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-09-17T07:09:18.169580+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-09-17T07:07:31.717888+0000","last_clean_scrub_stamp":"2024-09-17T07:09:18.169580+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011743155,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":224,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-17T07:12:14.322481+0000","last_change":"2024-09-17T07:09:24.407995+0000","last_active":"2024-09-17T07:12:14.322481+0000","last_peered":"2024-09-17T07:12:14.322481+0000","last_clean":"2024-09-17T07:12:14.322481+0000","last_became_active":"2024-09-17T07:09:24.407362+0000","last_became_peered":"2024-09-17T07:09:24.407362+0000","last_unstale":"2024-09-17T07:12:14.322481+0000","last_undegraded":"2024-09-17T07:12:14.322481+0000","last_fullsized":"2024-09-17T07:12:14.322481+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:27.841486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:08:27.841486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037672679999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":70,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-17T07:12:13.321267+0000","last_change":"2024-09-17T07:12:05.555207+0000","last_active":"2024-09-17T07:12:13.321267+0000","last_peered":"2024-09-17T07:12:13.321267+0000","last_clean":"2024-09-17T07:12:13.321267+0000","last_became_active":"2024-09-17T07:09:24.400791+0000","last_became_peered":"2024-09-17T07:09:24.400791+0000","last_unstale":"2024-09-17T07:12:13.321267+0000","last_undegraded":"2024-09-17T07:12:13.321267+0000","last_fullsized":"2024-09-17T07:12:13.321267+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:05.555100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:12:05.555100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:14.820862+0000","scrub_duration":0.00158344,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"75'4","reported_seq":127,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-17T07:12:14.321870+0000","last_change":"2024-09-17T07:11:07.499806+0000","last_active":"2024-09-17T07:12:14.321870+0000","last_peered":"2024-09-17T07:12:14.321870+0000","last_clean":"2024-09-17T07:12:14.321870+0000","last_became_active":"2024-09-17T07:09:24.396368+0000","last_became_peered":"2024-09-17T07:09:24.396368+0000","last_unstale":"2024-09-17T07:12:14.321870+0000","last_undegraded":"2024-09-17T07:12:14.321870+0000","last_fullsized":"2024-09-17T07:12:14.321870+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'4","last_scrub_stamp":"2024-09-17T07:11:07.499679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:11:07.499679+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:36.469804+0000","scrub_duration":0.0043183819999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":230,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-17T07:12:13.398520+0000","last_change":"2024-09-17T07:11:24.083407+0000","last_active":"2024-09-17T07:12:13.398520+0000","last_peered":"2024-09-17T07:12:13.398520+0000","last_clean":"2024-09-17T07:12:13.398520+0000","last_became_active":"2024-09-17T07:02:12.008898+0000","last_became_peered":"2024-09-17T07:02:12.008898+0000","last_unstale":"2024-09-17T07:12:13.398520+0000","last_undegraded":"2024-09-17T07:12:13.398520+0000","last_fullsized":"2024-09-17T07:12:13.398520+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:24.083284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:11:24.083284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:40.949692+0000","scrub_duration":0.014006177999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":228,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-17T07:12:14.974516+0000","last_change":"2024-09-17T07:11:03.383292+0000","last_active":"2024-09-17T07:12:14.974516+0000","last_peered":"2024-09-17T07:12:14.974516+0000","last_clean":"2024-09-17T07:12:14.974516+0000","last_became_active":"2024-09-17T07:02:16.014982+0000","last_became_peered":"2024-09-17T07:02:16.014982+0000","last_unstale":"2024-09-17T07:12:14.974516+0000","last_undegraded":"2024-09-17T07:12:14.974516+0000","last_fullsized":"2024-09-17T07:12:14.974516+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:03.383039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:47.444972+0000","last_clean_scrub_stamp":"2024-09-17T07:11:03.383039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:26.830393+0000","scrub_duration":0.0015364230000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":224,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-17T07:12:13.396433+0000","last_change":"2024-09-17T07:11:23.302709+0000","last_active":"2024-09-17T07:12:13.396433+0000","last_peered":"2024-09-17T07:12:13.396433+0000","last_clean":"2024-09-17T07:12:13.396433+0000","last_became_active":"2024-09-17T07:02:18.021014+0000","last_became_peered":"2024-09-17T07:02:18.021014+0000","last_unstale":"2024-09-17T07:12:13.396433+0000","last_undegraded":"2024-09-17T07:12:13.396433+0000","last_fullsized":"2024-09-17T07:12:13.396433+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:23.302586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:11:23.302586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:45.028048+0000","scrub_duration":0.0077240449999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"75'11","reported_seq":113,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-17T07:12:13.396308+0000","last_change":"2024-09-17T07:11:20.321025+0000","last_active":"2024-09-17T07:12:13.396308+0000","last_peered":"2024-09-17T07:12:13.396308+0000","last_clean":"2024-09-17T07:12:13.396308+0000","last_became_active":"2024-09-17T07:09:24.404942+0000","last_became_peered":"2024-09-17T07:09:24.404942+0000","last_unstale":"2024-09-17T07:12:13.396308+0000","last_undegraded":"2024-09-17T07:12:13.396308+0000","last_fullsized":"2024-09-17T07:12:13.396308+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-09-17T07:11:20.320870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:11:20.320870+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:37.430071+0000","scrub_duration":0.013438802,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":242,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-17T07:12:14.975078+0000","last_change":"2024-09-17T07:11:14.431423+0000","last_active":"2024-09-17T07:12:14.975078+0000","last_peered":"2024-09-17T07:12:14.975078+0000","last_clean":"2024-09-17T07:12:14.975078+0000","last_became_active":"2024-09-17T07:02:12.010195+0000","last_became_peered":"2024-09-17T07:02:12.010195+0000","last_unstale":"2024-09-17T07:12:14.975078+0000","last_undegraded":"2024-09-17T07:12:14.975078+0000","last_fullsized":"2024-09-17T07:12:14.975078+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:14.431102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:11:14.431102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:37.926489+0000","scrub_duration":0.0070527250000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":226,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-17T07:12:13.396108+0000","last_change":"2024-09-17T07:11:26.259825+0000","last_active":"2024-09-17T07:12:13.396108+0000","last_peered":"2024-09-17T07:12:13.396108+0000","last_clean":"2024-09-17T07:12:13.396108+0000","last_became_active":"2024-09-17T07:02:16.016183+0000","last_became_peered":"2024-09-17T07:02:16.016183+0000","last_unstale":"2024-09-17T07:12:13.396108+0000","last_undegraded":"2024-09-17T07:12:13.396108+0000","last_fullsized":"2024-09-17T07:12:13.396108+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:26.258291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:11:26.258291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:37.661689+0000","scrub_duration":0.0034422369999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":70,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-17T07:12:13.319606+0000","last_change":"2024-09-17T07:10:34.515889+0000","last_active":"2024-09-17T07:12:13.319606+0000","last_peered":"2024-09-17T07:12:13.319606+0000","last_clean":"2024-09-17T07:12:13.319606+0000","last_became_active":"2024-09-17T07:09:24.392216+0000","last_became_peered":"2024-09-17T07:09:24.392216+0000","last_unstale":"2024-09-17T07:12:13.319606+0000","last_undegraded":"2024-09-17T07:12:13.319606+0000","last_fullsized":"2024-09-17T07:12:13.319606+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:10:34.515779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:10:34.515779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032588679999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1c","version":"75'1","reported_seq":224,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-17T07:12:13.397187+0000","last_change":"2024-09-17T07:11:17.398031+0000","last_active":"2024-09-17T07:12:13.397187+0000","last_peered":"2024-09-17T07:12:13.397187+0000","last_clean":"2024-09-17T07:12:13.397187+0000","last_became_active":"2024-09-17T07:02:18.021855+0000","last_became_peered":"2024-09-17T07:02:18.021855+0000","last_unstale":"2024-09-17T07:12:13.397187+0000","last_undegraded":"2024-09-17T07:12:13.397187+0000","last_fullsized":"2024-09-17T07:12:13.397187+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-17T07:11:17.396229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:11:17.396229+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:39.204660+0000","scrub_duration":0.0090123449999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":231,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-17T07:12:13.397526+0000","last_change":"2024-09-17T07:11:06.126289+0000","last_active":"2024-09-17T07:12:13.397526+0000","last_peered":"2024-09-17T07:12:13.397526+0000","last_clean":"2024-09-17T07:12:13.397526+0000","last_became_active":"2024-09-17T07:02:12.005329+0000","last_became_peered":"2024-09-17T07:02:12.005329+0000","last_unstale":"2024-09-17T07:12:13.397526+0000","last_undegraded":"2024-09-17T07:12:13.397526+0000","last_fullsized":"2024-09-17T07:12:13.397526+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:06.126127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:11:06.126127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:19.574156+0000","scrub_duration":0.001691456,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"75'10","reported_seq":276,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-09-17T07:12:15.552893+0000","last_change":"2024-09-17T07:10:49.754178+0000","last_active":"2024-09-17T07:12:15.552893+0000","last_peered":"2024-09-17T07:12:15.552893+0000","last_clean":"2024-09-17T07:12:15.552893+0000","last_became_active":"2024-09-17T07:10:49.730948+0000","last_became_peered":"2024-09-17T07:10:49.730948+0000","last_unstale":"2024-09-17T07:12:15.552893+0000","last_undegraded":"2024-09-17T07:12:15.552893+0000","last_fullsized":"2024-09-17T07:12:15.552893+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-09-17T07:09:13.341701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:09:13.341701+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.024611266999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"75'8","reported_seq":360,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-09-17T07:12:15.550207+0000","last_change":"2024-09-17T07:11:12.356249+0000","last_active":"2024-09-17T07:12:15.550207+0000","last_peered":"2024-09-17T07:12:15.550207+0000","last_clean":"2024-09-17T07:12:15.550207+0000","last_became_active":"2024-09-17T07:02:16.020560+0000","last_became_peered":"2024-09-17T07:02:16.020560+0000","last_unstale":"2024-09-17T07:12:15.550207+0000","last_undegraded":"2024-09-17T07:12:15.550207+0000","last_fullsized":"2024-09-17T07:12:15.550207+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:11:12.356129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:11:12.356129+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:39.144563+0000","scrub_duration":0.0063199399999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":213,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-09-17T07:12:15.166571+0000","last_change":"2024-09-17T07:08:34.803052+0000","last_active":"2024-09-17T07:12:15.166571+0000","last_peered":"2024-09-17T07:12:15.166571+0000","last_clean":"2024-09-17T07:12:15.166571+0000","last_became_active":"2024-09-17T07:02:18.021603+0000","last_became_peered":"2024-09-17T07:02:18.021603+0000","last_unstale":"2024-09-17T07:12:15.166571+0000","last_undegraded":"2024-09-17T07:12:15.166571+0000","last_fullsized":"2024-09-17T07:12:15.166571+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:08:34.802938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:08:34.802938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0022680489999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":82,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-17T07:12:14.974755+0000","last_change":"2024-09-17T07:12:09.738322+0000","last_active":"2024-09-17T07:12:14.974755+0000","last_peered":"2024-09-17T07:12:14.974755+0000","last_clean":"2024-09-17T07:12:14.974755+0000","last_became_active":"2024-09-17T07:09:24.403360+0000","last_became_peered":"2024-09-17T07:09:24.403360+0000","last_unstale":"2024-09-17T07:12:14.974755+0000","last_undegraded":"2024-09-17T07:12:14.974755+0000","last_fullsized":"2024-09-17T07:12:14.974755+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:09.738248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:10:29.325664+0000","last_clean_scrub_stamp":"2024-09-17T07:12:09.738248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:27.942295+0000","scrub_duration":0.0014227129999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"75'11","reported_seq":290,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-17T07:12:14.975006+0000","last_change":"2024-09-17T07:11:19.549613+0000","last_active":"2024-09-17T07:12:14.975006+0000","last_peered":"2024-09-17T07:12:14.975006+0000","last_clean":"2024-09-17T07:12:14.975006+0000","last_became_active":"2024-09-17T07:03:48.760390+0000","last_became_peered":"2024-09-17T07:03:48.760390+0000","last_unstale":"2024-09-17T07:12:14.975006+0000","last_undegraded":"2024-09-17T07:12:14.975006+0000","last_fullsized":"2024-09-17T07:12:14.975006+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-09-17T07:11:19.549119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:11:19.549119+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:26.240319+0000","scrub_duration":0.005408731,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":216,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-17T07:12:13.320721+0000","last_change":"2024-09-17T07:09:10.293301+0000","last_active":"2024-09-17T07:12:13.320721+0000","last_peered":"2024-09-17T07:12:13.320721+0000","last_clean":"2024-09-17T07:12:13.320721+0000","last_became_active":"2024-09-17T07:02:16.019701+0000","last_became_peered":"2024-09-17T07:02:16.019701+0000","last_unstale":"2024-09-17T07:12:13.320721+0000","last_undegraded":"2024-09-17T07:12:13.320721+0000","last_fullsized":"2024-09-17T07:12:13.320721+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:10.292615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:09:10.292615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010112249,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":215,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-17T07:12:14.321442+0000","last_change":"2024-09-17T07:11:09.721112+0000","last_active":"2024-09-17T07:12:14.321442+0000","last_peered":"2024-09-17T07:12:14.321442+0000","last_clean":"2024-09-17T07:12:14.321442+0000","last_became_active":"2024-09-17T07:02:18.363752+0000","last_became_peered":"2024-09-17T07:02:18.363752+0000","last_unstale":"2024-09-17T07:12:14.321442+0000","last_undegraded":"2024-09-17T07:12:14.321442+0000","last_fullsized":"2024-09-17T07:12:14.321442+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:09.720998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:11:09.720998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002692687,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":261,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-17T07:12:14.975020+0000","last_change":"2024-09-17T07:11:18.559991+0000","last_active":"2024-09-17T07:12:14.975020+0000","last_peered":"2024-09-17T07:12:14.975020+0000","last_clean":"2024-09-17T07:12:14.975020+0000","last_became_active":"2024-09-17T07:03:48.759384+0000","last_became_peered":"2024-09-17T07:03:48.759384+0000","last_unstale":"2024-09-17T07:12:14.975020+0000","last_undegraded":"2024-09-17T07:12:14.975020+0000","last_fullsized":"2024-09-17T07:12:14.975020+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:18.559877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:11:18.559877+0000","last_clean_scrub_stamp":"2024-09-17T07:11:18.559877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:18.776161+0000","scrub_duration":0.001515052,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"75'15","reported_seq":254,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-17T07:12:14.323410+0000","last_change":"2024-09-17T07:09:24.427421+0000","last_active":"2024-09-17T07:12:14.323410+0000","last_peered":"2024-09-17T07:12:14.323410+0000","last_clean":"2024-09-17T07:12:14.323410+0000","last_became_active":"2024-09-17T07:09:24.403030+0000","last_became_peered":"2024-09-17T07:09:24.403030+0000","last_unstale":"2024-09-17T07:12:14.323410+0000","last_undegraded":"2024-09-17T07:12:14.323410+0000","last_fullsized":"2024-09-17T07:12:14.323410+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-09-17T07:09:15.110940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:09:15.110940+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.017595989999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":226,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-09-17T07:12:15.166752+0000","last_change":"2024-09-17T07:12:01.366198+0000","last_active":"2024-09-17T07:12:15.166752+0000","last_peered":"2024-09-17T07:12:15.166752+0000","last_clean":"2024-09-17T07:12:15.166752+0000","last_became_active":"2024-09-17T07:02:16.015903+0000","last_became_peered":"2024-09-17T07:02:16.015903+0000","last_unstale":"2024-09-17T07:12:15.166752+0000","last_undegraded":"2024-09-17T07:12:15.166752+0000","last_fullsized":"2024-09-17T07:12:15.166752+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:01.366088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:25.794418+0000","last_clean_scrub_stamp":"2024-09-17T07:12:01.366088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:01.913633+0000","scrub_duration":0.001615116,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":72,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-17T07:12:14.974898+0000","last_change":"2024-09-17T07:11:00.320085+0000","last_active":"2024-09-17T07:12:14.974898+0000","last_peered":"2024-09-17T07:12:14.974898+0000","last_clean":"2024-09-17T07:12:14.974898+0000","last_became_active":"2024-09-17T07:09:24.402411+0000","last_became_peered":"2024-09-17T07:09:24.402411+0000","last_unstale":"2024-09-17T07:12:14.974898+0000","last_undegraded":"2024-09-17T07:12:14.974898+0000","last_fullsized":"2024-09-17T07:12:14.974898+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:00.319980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:11:00.319980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:21.654835+0000","scrub_duration":0.0047119570000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":72,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-17T07:12:13.396928+0000","last_change":"2024-09-17T07:11:00.563847+0000","last_active":"2024-09-17T07:12:13.396928+0000","last_peered":"2024-09-17T07:12:13.396928+0000","last_clean":"2024-09-17T07:12:13.396928+0000","last_became_active":"2024-09-17T07:09:24.404795+0000","last_became_peered":"2024-09-17T07:09:24.404795+0000","last_unstale":"2024-09-17T07:12:13.396928+0000","last_undegraded":"2024-09-17T07:12:13.396928+0000","last_fullsized":"2024-09-17T07:12:13.396928+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:00.563722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:11:00.563722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:26.978807+0000","scrub_duration":0.018069940999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"86'17","reported_seq":125,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-09-17T07:12:14.322638+0000","last_change":"2024-09-17T07:09:24.416689+0000","last_active":"2024-09-17T07:12:14.322638+0000","last_peered":"2024-09-17T07:12:14.322638+0000","last_clean":"2024-09-17T07:12:14.322638+0000","last_became_active":"2024-09-17T07:09:24.394476+0000","last_became_peered":"2024-09-17T07:09:24.394476+0000","last_unstale":"2024-09-17T07:12:14.322638+0000","last_undegraded":"2024-09-17T07:12:14.322638+0000","last_fullsized":"2024-09-17T07:12:14.322638+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'17","last_scrub_stamp":"2024-09-17T07:09:12.729002+0000","last_deep_scrub":"86'17","last_deep_scrub_stamp":"2024-09-17T07:03:18.241067+0000","last_clean_scrub_stamp":"2024-09-17T07:09:12.729002+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":237,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-09-17T07:12:15.548599+0000","last_change":"2024-09-17T07:11:14.763627+0000","last_active":"2024-09-17T07:12:15.548599+0000","last_peered":"2024-09-17T07:12:15.548599+0000","last_clean":"2024-09-17T07:12:15.548599+0000","last_became_active":"2024-09-17T07:09:24.393329+0000","last_became_peered":"2024-09-17T07:09:24.393329+0000","last_unstale":"2024-09-17T07:12:15.548599+0000","last_undegraded":"2024-09-17T07:12:15.548599+0000","last_fullsized":"2024-09-17T07:12:15.548599+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:14.763470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:11:14.763470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:39.147572+0000","scrub_duration":0.0044668709999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":350748672,"num_objects":5352,"num_object_clones":0,"num_object_copies":16056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5352,"num_write_kb":342528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":986185728,"data_stored":986185728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5352,"ondisk_log_size":5352,"up":48,"acting":48,"num_store_stats":7},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70,"ondisk_log_size":70,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":704,"num_read_kb":461,"num_write":425,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":109,"num_bytes_recovered":2228,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":13004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":435,"ondisk_log_size":435,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":599,"num_read_kb":594,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1386,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":12,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":4578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037887,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":557620,"kb_used_data":179500,"kb_used_omap":1,"kb_used_meta":378110,"kb_avail":93187532,"statfs":{"total":95995035648,"available":95424032768,"internally_reserved":0,"allocated":183808000,"data_stored":183354084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":387185099},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:11:17 2024","interfaces":[{"interface":"back","average":{"1min":1.4419999999999999,"5min":1.9359999999999999,"15min":1.4079999999999999},"min":{"1min":0.20699999999999999,"5min":0.192,"15min":0.192},"max":{"1min":8.6080000000000005,"5min":48.493000000000002,"15min":52.478999999999999},"last":0.32400000000000001},{"interface":"front","average":{"1min":2.5979999999999999,"5min":2.2490000000000001,"15min":1.5209999999999999},"min":{"1min":0.23899999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":19.216000000000001,"5min":48.738,"15min":52.871000000000002},"last":0.39400000000000002}]},{"osd":1,"last update":"Tue Sep 17 07:11:17 2024","interfaces":[{"interface":"back","average":{"1min":1.173,"5min":2.2759999999999998,"15min":1.1160000000000001},"min":{"1min":0.248,"5min":0.223,"15min":0.223},"max":{"1min":3.9209999999999998,"5min":73.549000000000007,"15min":73.549000000000007},"last":0.498},{"interface":"front","average":{"1min":1.1679999999999999,"5min":2.3799999999999999,"15min":1.169},"min":{"1min":0.26200000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":2.8980000000000001,"5min":73.116,"15min":73.116},"last":0.36699999999999999}]},{"osd":2,"last update":"Tue Sep 17 07:11:17 2024","interfaces":[{"interface":"back","average":{"1min":1.302,"5min":1.2509999999999999,"15min":0.94299999999999995},"min":{"1min":0.28899999999999998,"5min":0.245,"15min":0.23899999999999999},"max":{"1min":6.0090000000000003,"5min":22.376000000000001,"15min":52.055999999999997},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.85299999999999998,"5min":1.1040000000000001,"15min":0.94099999999999995},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":2.6349999999999998,"5min":22.298999999999999,"15min":52.064999999999998},"last":0.35399999999999998}]},{"osd":3,"last update":"Tue Sep 17 07:11:17 2024","interfaces":[{"interface":"back","average":{"1min":1.194,"5min":1.4339999999999999,"15min":1.141},"min":{"1min":0.317,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":6.0830000000000002,"5min":22.196000000000002,"15min":51.689},"last":0.625},{"interface":"front","average":{"1min":0.84699999999999998,"5min":1.488,"15min":1.224},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":2.6579999999999999,"5min":22.234000000000002,"15min":52.145000000000003},"last":0.61399999999999999}]},{"osd":4,"last update":"Tue Sep 17 07:11:17 2024","interfaces":[{"interface":"back","average":{"1min":1.1830000000000001,"5min":0.73899999999999999,"15min":0.53800000000000003},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.22800000000000001},"max":{"1min":4.9950000000000001,"5min":4.9950000000000001,"15min":4.9950000000000001},"last":0.23499999999999999},{"interface":"front","average":{"1min":1.621,"5min":0.85399999999999998,"15min":0.56599999999999995},"min":{"1min":0.26000000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":6.0549999999999997,"5min":6.0549999999999997,"15min":6.0549999999999997},"last":0.57099999999999995}]},{"osd":5,"last update":"Tue Sep 17 07:11:17 2024","interfaces":[{"interface":"back","average":{"1min":1.4590000000000001,"5min":0.78400000000000003,"15min":0.55400000000000005},"min":{"1min":0.253,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":5.1360000000000001,"5min":5.1360000000000001,"15min":5.1360000000000001},"last":0.28399999999999997},{"interface":"front","average":{"1min":1.2030000000000001,"5min":0.81999999999999995,"15min":0.57799999999999996},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":3.294,"5min":3.8639999999999999,"15min":3.8639999999999999},"last":0.27300000000000002}]},{"osd":6,"last update":"Tue Sep 17 07:11:17 2024","interfaces":[{"interface":"back","average":{"1min":2.6419999999999999,"5min":1.1220000000000001,"15min":0.70599999999999996},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.26100000000000001},"max":{"1min":25.363,"5min":25.363,"15min":25.363},"last":0.25900000000000001},{"interface":"front","average":{"1min":2.669,"5min":1.1120000000000001,"15min":0.69299999999999995},"min":{"1min":0.30099999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":23.571000000000002,"5min":23.571000000000002,"15min":23.571000000000002},"last":0.30399999999999999}]}]},{"osd":6,"up_from":50,"seq":214748364930,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":321748,"kb_used_data":24008,"kb_used_omap":1,"kb_used_meta":297726,"kb_avail":93423404,"statfs":{"total":95995035648,"available":95665565696,"internally_reserved":0,"allocated":24584192,"data_stored":24102221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":304871879},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:12:04 2024","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":1.667,"15min":1.5880000000000001},"min":{"1min":0.318,"5min":0.185,"15min":0.185},"max":{"1min":5.5010000000000003,"5min":24.106999999999999,"15min":75.323999999999998},"last":0.46200000000000002},{"interface":"front","average":{"1min":1.4630000000000001,"5min":1.6000000000000001,"15min":1.546},"min":{"1min":0.24299999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":5.6459999999999999,"5min":26.193000000000001,"15min":75.168999999999997},"last":0.439}]},{"osd":1,"last update":"Tue Sep 17 07:12:04 2024","interfaces":[{"interface":"back","average":{"1min":1.2649999999999999,"5min":1.101,"15min":0.72699999999999998},"min":{"1min":0.28399999999999997,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":5.0549999999999997,"5min":5.0549999999999997,"15min":5.0549999999999997},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.79600000000000004,"5min":1.0169999999999999,"15min":0.747},"min":{"1min":0.27300000000000002,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":2.9399999999999999,"5min":5.4930000000000003,"15min":5.9530000000000003},"last":0.41099999999999998}]},{"osd":2,"last update":"Tue Sep 17 07:12:04 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.91300000000000003,"15min":0.80200000000000005},"min":{"1min":0.314,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":2.984,"5min":5.1159999999999997,"15min":33.079000000000001},"last":0.35999999999999999},{"interface":"front","average":{"1min":1.095,"5min":0.876,"15min":0.79100000000000004},"min":{"1min":0.26700000000000002,"5min":0.254,"15min":0.23000000000000001},"max":{"1min":5.734,"5min":5.734,"15min":33.100000000000001},"last":0.48099999999999998}]},{"osd":3,"last update":"Tue Sep 17 07:12:04 2024","interfaces":[{"interface":"back","average":{"1min":1.0289999999999999,"5min":1.2010000000000001,"15min":1.1080000000000001},"min":{"1min":0.31900000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":5.0830000000000002,"5min":16.984000000000002,"15min":52.944000000000003},"last":0.373},{"interface":"front","average":{"1min":0.73199999999999998,"5min":1.2729999999999999,"15min":1.1279999999999999},"min":{"1min":0.32600000000000001,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":2.9649999999999999,"5min":16.745999999999999,"15min":52.747},"last":0.499}]},{"osd":4,"last update":"Tue Sep 17 07:12:04 2024","interfaces":[{"interface":"back","average":{"1min":1.2050000000000001,"5min":4.444,"15min":1.7609999999999999},"min":{"1min":0.23999999999999999,"5min":0.214,"15min":0.214},"max":{"1min":5.7130000000000001,"5min":260.77199999999999,"15min":260.77199999999999},"last":0.29499999999999998},{"interface":"front","average":{"1min":1.294,"5min":4.4829999999999997,"15min":1.79},"min":{"1min":0.24299999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":6.2140000000000004,"5min":260.69099999999997,"15min":260.69099999999997},"last":0.32700000000000001}]},{"osd":5,"last update":"Tue Sep 17 07:12:04 2024","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":1.159,"15min":0.67900000000000005},"min":{"1min":0.27600000000000002,"5min":0.26200000000000001,"15min":0.25900000000000001},"max":{"1min":6.3860000000000001,"5min":32.042000000000002,"15min":32.042000000000002},"last":0.34399999999999997},{"interface":"front","average":{"1min":1.135,"5min":1.3200000000000001,"15min":0.748},"min":{"1min":0.28999999999999998,"5min":0.251,"15min":0.251},"max":{"1min":5.1379999999999999,"5min":31.852,"15min":31.852},"last":0.42299999999999999}]},{"osd":7,"last update":"Tue Sep 17 07:11:20 2024","interfaces":[{"interface":"back","average":{"1min":1.619,"5min":1.0029999999999999,"15min":0.66500000000000004},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.23699999999999999},"max":{"1min":7.657,"5min":7.657,"15min":7.657},"last":0.51200000000000001},{"interface":"front","average":{"1min":1.528,"5min":0.93899999999999995,"15min":0.66800000000000004},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":7.1890000000000001,"5min":7.1890000000000001,"15min":7.1890000000000001},"last":0.39100000000000001}]}]},{"osd":5,"up_from":42,"seq":180388626567,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":422644,"kb_used_data":76524,"kb_used_omap":1,"kb_used_meta":346110,"kb_avail":93322508,"statfs":{"total":95995035648,"available":95562248192,"internally_reserved":0,"allocated":78360576,"data_stored":77906448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":354417094},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:11:37 2024","interfaces":[{"interface":"back","average":{"1min":1.1339999999999999,"5min":1.7050000000000001,"15min":1.4630000000000001},"min":{"1min":0.25900000000000001,"5min":0.23499999999999999,"15min":0.221},"max":{"1min":5.202,"5min":28.727,"15min":49.265000000000001},"last":0.308},{"interface":"front","average":{"1min":1.5529999999999999,"5min":1.716,"15min":1.5069999999999999},"min":{"1min":0.30199999999999999,"5min":0.219,"15min":0.219},"max":{"1min":5.6879999999999997,"5min":27.824999999999999,"15min":47.350000000000001},"last":0.35199999999999998}]},{"osd":1,"last update":"Tue Sep 17 07:11:37 2024","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":1.208,"15min":0.76700000000000002},"min":{"1min":0.29899999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":5.5529999999999999,"5min":20.329000000000001,"15min":20.329000000000001},"last":0.40200000000000002},{"interface":"front","average":{"1min":1.9039999999999999,"5min":1.4359999999999999,"15min":1.151},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":5.2969999999999997,"5min":20.408999999999999,"15min":20.408999999999999},"last":0.246}]},{"osd":2,"last update":"Tue Sep 17 07:11:37 2024","interfaces":[{"interface":"back","average":{"1min":1.508,"5min":1.512,"15min":0.96499999999999997},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.252},"max":{"1min":5.6719999999999997,"5min":20.486000000000001,"15min":20.486000000000001},"last":0.33200000000000002},{"interface":"front","average":{"1min":1.1060000000000001,"5min":1.337,"15min":0.877},"min":{"1min":0.29199999999999998,"5min":0.26300000000000001,"15min":0.23200000000000001},"max":{"1min":5.7789999999999999,"5min":23.925999999999998,"15min":23.925999999999998},"last":0.25800000000000001}]},{"osd":3,"last update":"Tue Sep 17 07:11:37 2024","interfaces":[{"interface":"back","average":{"1min":1.2689999999999999,"5min":2.2749999999999999,"15min":1.2170000000000001},"min":{"1min":0.29299999999999998,"5min":0.247,"15min":0.247},"max":{"1min":5.6970000000000001,"5min":68.978999999999999,"15min":68.978999999999999},"last":0.47199999999999998},{"interface":"front","average":{"1min":1.008,"5min":2.2719999999999998,"15min":1.1870000000000001},"min":{"1min":0.34499999999999997,"5min":0.253,"15min":0.253},"max":{"1min":5.6139999999999999,"5min":69.084000000000003,"15min":69.084000000000003},"last":0.372}]},{"osd":4,"last update":"Tue Sep 17 07:11:37 2024","interfaces":[{"interface":"back","average":{"1min":2.415,"5min":2.6699999999999999,"15min":1.153},"min":{"1min":0.27700000000000002,"5min":0.22,"15min":0.22},"max":{"1min":11.835000000000001,"5min":142.327,"15min":142.327},"last":0.28599999999999998},{"interface":"front","average":{"1min":1.911,"5min":2.5430000000000001,"15min":1.139},"min":{"1min":0.23999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":10.01,"5min":142.13200000000001,"15min":142.13200000000001},"last":0.38900000000000001}]},{"osd":6,"last update":"Tue Sep 17 07:11:56 2024","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.995,"15min":0.61699999999999999},"min":{"1min":0.30399999999999999,"5min":0.23499999999999999,"15min":0.23200000000000001},"max":{"1min":3.637,"5min":8.3149999999999995,"15min":8.3149999999999995},"last":0.45100000000000001},{"interface":"front","average":{"1min":1.1479999999999999,"5min":0.98399999999999999,"15min":0.63900000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.24099999999999999},"max":{"1min":6.4459999999999997,"5min":6.4459999999999997,"15min":6.4459999999999997},"last":0.435}]},{"osd":7,"last update":"Tue Sep 17 07:12:16 2024","interfaces":[{"interface":"back","average":{"1min":1.1930000000000001,"5min":1.0980000000000001,"15min":0.69699999999999995},"min":{"1min":0.34200000000000003,"5min":0.29199999999999998,"15min":0.29099999999999998},"max":{"1min":6.4130000000000003,"5min":6.4130000000000003,"15min":6.4130000000000003},"last":0.496},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.94299999999999995,"15min":0.628},"min":{"1min":0.32400000000000001,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":3.8530000000000002,"5min":5.641,"15min":5.641},"last":0.42399999999999999}]}]},{"osd":4,"up_from":33,"seq":141733920909,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":367804,"kb_used_data":21680,"kb_used_omap":1,"kb_used_meta":346110,"kb_avail":93377348,"statfs":{"total":95995035648,"available":95618404352,"internally_reserved":0,"allocated":22200320,"data_stored":21741196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":354417094},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:12:01 2024","interfaces":[{"interface":"back","average":{"1min":0.96099999999999997,"5min":2.323,"15min":1.8540000000000001},"min":{"1min":0.29799999999999999,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":2.4470000000000001,"5min":58.097999999999999,"15min":74.551000000000002},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.85099999999999998,"5min":2.577,"15min":2.012},"min":{"1min":0.21299999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":3.2280000000000002,"5min":58.383000000000003,"15min":74.778999999999996},"last":0.57199999999999995}]},{"osd":1,"last update":"Tue Sep 17 07:12:01 2024","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":2.7360000000000002,"15min":1.3140000000000001},"min":{"1min":0.251,"5min":0.23300000000000001,"15min":0.23000000000000001},"max":{"1min":6.9729999999999999,"5min":54.914999999999999,"15min":54.914999999999999},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.69599999999999995,"5min":2.3809999999999998,"15min":1.2170000000000001},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.22800000000000001},"max":{"1min":3.073,"5min":53.725000000000001,"15min":53.725000000000001},"last":0.56299999999999994}]},{"osd":2,"last update":"Tue Sep 17 07:12:01 2024","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":0.98499999999999999,"15min":0.94199999999999995},"min":{"1min":0.26900000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":6.8929999999999998,"5min":6.8929999999999998,"15min":49.651000000000003},"last":0.49299999999999999},{"interface":"front","average":{"1min":1.7110000000000001,"5min":1.1479999999999999,"15min":1.016},"min":{"1min":0.35199999999999998,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":16.003,"5min":16.003,"15min":48.813000000000002},"last":0.59099999999999997}]},{"osd":3,"last update":"Tue Sep 17 07:12:01 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":1.133,"15min":0.98099999999999998},"min":{"1min":0.28699999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":3.2080000000000002,"5min":17.872,"15min":39.042999999999999},"last":0.45400000000000001},{"interface":"front","average":{"1min":1.4330000000000001,"5min":1.2210000000000001,"15min":1.012},"min":{"1min":0.29899999999999999,"5min":0.253,"15min":0.253},"max":{"1min":15.970000000000001,"5min":17.908000000000001,"15min":38.920000000000002},"last":0.58099999999999996}]},{"osd":5,"last update":"Tue Sep 17 07:11:30 2024","interfaces":[{"interface":"back","average":{"1min":1.3959999999999999,"5min":1.073,"15min":0.63200000000000001},"min":{"1min":0.247,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":4.1609999999999996,"5min":11.486000000000001,"15min":11.486000000000001},"last":0.28100000000000003},{"interface":"front","average":{"1min":1.827,"5min":1.1699999999999999,"15min":0.67700000000000005},"min":{"1min":0.36699999999999999,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":5.8319999999999999,"5min":11.781000000000001,"15min":11.781000000000001},"last":0.32500000000000001}]},{"osd":6,"last update":"Tue Sep 17 07:11:58 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":1.349,"15min":0.74399999999999999},"min":{"1min":0.26200000000000001,"5min":0.218,"15min":0.218},"max":{"1min":2.7010000000000001,"5min":47.923999999999999,"15min":47.923999999999999},"last":0.26800000000000002},{"interface":"front","average":{"1min":1.2569999999999999,"5min":1.4490000000000001,"15min":0.77800000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26300000000000001},"max":{"1min":6.2140000000000004,"5min":47.600999999999999,"15min":47.600999999999999},"last":0.33900000000000002}]},{"osd":7,"last update":"Tue Sep 17 07:11:21 2024","interfaces":[{"interface":"back","average":{"1min":1.4079999999999999,"5min":1.45,"15min":0.81200000000000006},"min":{"1min":0.33300000000000002,"5min":0.28899999999999998,"15min":0.27600000000000002},"max":{"1min":5.8040000000000003,"5min":48.719000000000001,"15min":48.719000000000001},"last":0.35299999999999998},{"interface":"front","average":{"1min":1.202,"5min":1.3700000000000001,"15min":0.77800000000000002},"min":{"1min":0.36299999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":4.4279999999999999,"5min":50.527000000000001,"15min":50.527000000000001},"last":0.314}]}]},{"osd":3,"up_from":0,"seq":115964117135,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":18,"seq":77309411477,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":587640,"kb_used_data":79088,"kb_used_omap":1,"kb_used_meta":508542,"kb_avail":93157512,"statfs":{"total":95995035648,"available":95393292288,"internally_reserved":0,"allocated":80986112,"data_stored":80527832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":520747462},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:11:38 2024","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":2.2240000000000002,"15min":1.0660000000000001},"min":{"1min":0.19400000000000001,"5min":0.193,"15min":0.17599999999999999},"max":{"1min":11.262,"5min":79.715000000000003,"15min":79.715000000000003},"last":0.29599999999999999},{"interface":"front","average":{"1min":1.048,"5min":2.177,"15min":1.2110000000000001},"min":{"1min":0.217,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":3.141,"5min":79.691999999999993,"15min":79.691999999999993},"last":1.127}]},{"osd":1,"last update":"Tue Sep 17 07:11:38 2024","interfaces":[{"interface":"back","average":{"1min":2.718,"5min":2.8380000000000001,"15min":1.2989999999999999},"min":{"1min":0.218,"5min":0.184,"15min":0.184},"max":{"1min":29.434999999999999,"5min":79.659000000000006,"15min":79.659000000000006},"last":1.47},{"interface":"front","average":{"1min":2.6509999999999998,"5min":2.8199999999999998,"15min":1.2689999999999999},"min":{"1min":0.28899999999999998,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":29.411999999999999,"5min":80.155000000000001,"15min":80.155000000000001},"last":1.9610000000000001}]},{"osd":3,"last update":"Tue Sep 17 07:11:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.6259999999999999,"15min":0.93700000000000006},"min":{"1min":0.27100000000000002,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":5.2039999999999997,"5min":23.43,"15min":23.43},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.94999999999999996,"5min":1.546,"15min":1.0349999999999999},"min":{"1min":0.26000000000000001,"5min":0.223,"15min":0.223},"max":{"1min":4.5,"5min":23.408000000000001,"15min":23.408000000000001},"last":0.65300000000000002}]},{"osd":4,"last update":"Tue Sep 17 07:12:15 2024","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":1.012,"15min":0.81899999999999995},"min":{"1min":0.40799999999999997,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":4.867,"5min":5.944,"15min":5.944},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.94399999999999995,"5min":1.109,"15min":0.81899999999999995},"min":{"1min":0.43099999999999999,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":6.8529999999999998,"5min":6.8529999999999998,"15min":6.8529999999999998},"last":0.77300000000000002}]},{"osd":5,"last update":"Tue Sep 17 07:11:38 2024","interfaces":[{"interface":"back","average":{"1min":1.738,"5min":1.3600000000000001,"15min":0.93500000000000005},"min":{"1min":0.42299999999999999,"5min":0.26100000000000001,"15min":0.249},"max":{"1min":8.8870000000000005,"5min":8.8870000000000005,"15min":8.8870000000000005},"last":0.36899999999999999},{"interface":"front","average":{"1min":1.3320000000000001,"5min":1.181,"15min":0.84099999999999997},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":4.7919999999999998,"5min":7.1790000000000003,"15min":7.1790000000000003},"last":0.63500000000000001}]},{"osd":6,"last update":"Tue Sep 17 07:11:51 2024","interfaces":[{"interface":"back","average":{"1min":1.607,"5min":2.3540000000000001,"15min":1.2689999999999999},"min":{"1min":0.45400000000000001,"5min":0.29999999999999999,"15min":0.28999999999999998},"max":{"1min":6.899,"5min":101.587,"15min":101.587},"last":0.84499999999999997},{"interface":"front","average":{"1min":1.002,"5min":2.3849999999999998,"15min":1.3140000000000001},"min":{"1min":0.38900000000000001,"5min":0.30599999999999999,"15min":0.28299999999999997},"max":{"1min":2.9940000000000002,"5min":100.696,"15min":100.696},"last":0.40000000000000002}]},{"osd":7,"last update":"Tue Sep 17 07:11:27 2024","interfaces":[{"interface":"back","average":{"1min":1.2290000000000001,"5min":1.099,"15min":0.81799999999999995},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":4.8440000000000003,"5min":6.9450000000000003,"15min":6.9450000000000003},"last":0.67200000000000004},{"interface":"front","average":{"1min":1.387,"5min":1.3140000000000001,"15min":0.95499999999999996},"min":{"1min":0.377,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":6.9219999999999997,"5min":9.2919999999999998,"15min":9.2919999999999998},"last":0.86699999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575002,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":464732,"kb_used_data":166996,"kb_used_omap":1,"kb_used_meta":297726,"kb_avail":93280420,"statfs":{"total":95995035648,"available":95519150080,"internally_reserved":0,"allocated":171003904,"data_stored":170509914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":304871878},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:11:56 2024","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":1.667,"15min":1.036},"min":{"1min":0.192,"5min":0.192,"15min":0.155},"max":{"1min":3.6030000000000002,"5min":22.521000000000001,"15min":22.521000000000001},"last":0.214},{"interface":"front","average":{"1min":0.70799999999999996,"5min":1.333,"15min":0.95499999999999996},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.17000000000000001},"max":{"1min":4.335,"5min":22.471,"15min":22.471},"last":0.25600000000000001}]},{"osd":2,"last update":"Tue Sep 17 07:11:32 2024","interfaces":[{"interface":"back","average":{"1min":1.1579999999999999,"5min":1.3779999999999999,"15min":0.90600000000000003},"min":{"1min":0.218,"5min":0.183,"15min":0.183},"max":{"1min":4.6459999999999999,"5min":6.2130000000000001,"15min":11.055999999999999},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.97599999999999998,"5min":1.123,"15min":0.877},"min":{"1min":0.27500000000000002,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":4.1559999999999997,"5min":5.4269999999999996,"15min":9.3160000000000007},"last":0.33200000000000002}]},{"osd":3,"last update":"Tue Sep 17 07:11:56 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":1.1000000000000001,"15min":0.89600000000000002},"min":{"1min":0.308,"5min":0.223,"15min":0.219},"max":{"1min":4.1269999999999998,"5min":8.5510000000000002,"15min":8.5510000000000002},"last":0.313},{"interface":"front","average":{"1min":0.48999999999999999,"5min":1.405,"15min":0.97999999999999998},"min":{"1min":0.246,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":2.1869999999999998,"5min":8.6400000000000006,"15min":19.602},"last":0.23799999999999999}]},{"osd":4,"last update":"Tue Sep 17 07:12:14 2024","interfaces":[{"interface":"back","average":{"1min":1.3069999999999999,"5min":1.278,"15min":0.95699999999999996},"min":{"1min":0.374,"5min":0.25,"15min":0.25},"max":{"1min":4.71,"5min":9.8480000000000008,"15min":9.8480000000000008},"last":0.69199999999999995},{"interface":"front","average":{"1min":1.085,"5min":1.585,"15min":1.0349999999999999},"min":{"1min":0.32500000000000001,"5min":0.317,"15min":0.307},"max":{"1min":3.4369999999999998,"5min":27.097999999999999,"15min":27.097999999999999},"last":0.71199999999999997}]},{"osd":5,"last update":"Tue Sep 17 07:11:32 2024","interfaces":[{"interface":"back","average":{"1min":1.968,"5min":1.5029999999999999,"15min":1.004},"min":{"1min":0.35099999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":6.6790000000000003,"5min":16.84,"15min":16.84},"last":0.77200000000000002},{"interface":"front","average":{"1min":1.3460000000000001,"5min":1.7450000000000001,"15min":1.0529999999999999},"min":{"1min":0.42499999999999999,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":4.5300000000000002,"5min":16.949999999999999,"15min":16.949999999999999},"last":0.57499999999999996}]},{"osd":6,"last update":"Tue Sep 17 07:11:56 2024","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.5900000000000001,"15min":1.0009999999999999},"min":{"1min":0.34499999999999997,"5min":0.308,"15min":0.308},"max":{"1min":6.8040000000000003,"5min":9.1359999999999992,"15min":9.1359999999999992},"last":0.64600000000000002},{"interface":"front","average":{"1min":1.302,"5min":1.238,"15min":0.92200000000000004},"min":{"1min":0.39800000000000002,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":4.5810000000000004,"5min":7.1130000000000004,"15min":7.1130000000000004},"last":0.84599999999999997}]},{"osd":7,"last update":"Tue Sep 17 07:12:16 2024","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":1.5900000000000001,"15min":1.0529999999999999},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":4.5759999999999996,"5min":9.8179999999999996,"15min":9.8179999999999996},"last":0.81200000000000006},{"interface":"front","average":{"1min":1.3100000000000001,"5min":1.4870000000000001,"15min":1.026},"min":{"1min":0.35999999999999999,"5min":0.34100000000000003,"15min":0.32900000000000001},"max":{"1min":4.1600000000000001,"5min":6.2590000000000003,"15min":11.007999999999999},"last":0.623}]}]},{"osd":0,"up_from":8,"seq":34359738525,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":584520,"kb_used_data":130112,"kb_used_omap":1,"kb_used_meta":454398,"kb_avail":93160632,"statfs":{"total":95995035648,"available":95396487168,"internally_reserved":0,"allocated":133234688,"data_stored":132760768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":465304006},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 17 07:12:05 2024","interfaces":[{"interface":"back","average":{"1min":1.706,"5min":1.2250000000000001,"15min":0.85899999999999999},"min":{"1min":0.19500000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":21.876000000000001,"5min":21.876000000000001,"15min":21.876000000000001},"last":0.63100000000000001},{"interface":"front","average":{"1min":2.242,"5min":1.657,"15min":0.98999999999999999},"min":{"1min":0.28899999999999998,"5min":0.158,"15min":0.158},"max":{"1min":22.521999999999998,"5min":22.521999999999998,"15min":22.521999999999998},"last":0.40300000000000002}]},{"osd":2,"last update":"Tue Sep 17 07:11:32 2024","interfaces":[{"interface":"back","average":{"1min":1.173,"5min":1.413,"15min":1.073},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":6.3799999999999999,"5min":18.643999999999998,"15min":18.643999999999998},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.90300000000000002,"5min":1.4359999999999999,"15min":0.94799999999999995},"min":{"1min":0.219,"5min":0.219,"15min":0.19900000000000001},"max":{"1min":4.431,"5min":17.972999999999999,"15min":17.972999999999999},"last":0.23499999999999999}]},{"osd":3,"last update":"Tue Sep 17 07:12:05 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":1.137,"15min":0.97099999999999997},"min":{"1min":0.255,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":4.4619999999999997,"5min":11.598000000000001,"15min":18.629000000000001},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.95599999999999996,"15min":1.0069999999999999},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.23699999999999999},"max":{"1min":0.96899999999999997,"5min":5.4169999999999998,"15min":22.358000000000001},"last":0.28399999999999997}]},{"osd":4,"last update":"Tue Sep 17 07:12:13 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":1.042,"15min":1.0249999999999999},"min":{"1min":0.39400000000000002,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.611,"5min":19.146999999999998,"15min":19.146999999999998},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.60899999999999999,"5min":1.171,"15min":0.97499999999999998},"min":{"1min":0.28000000000000003,"5min":0.27400000000000002,"15min":0.255},"max":{"1min":1.621,"5min":19.158999999999999,"15min":19.158999999999999},"last":0.53400000000000003}]},{"osd":5,"last update":"Tue Sep 17 07:11:42 2024","interfaces":[{"interface":"back","average":{"1min":1.5669999999999999,"5min":1.6220000000000001,"15min":1.0469999999999999},"min":{"1min":0.40899999999999997,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":5.0730000000000004,"5min":18.841000000000001,"15min":18.841000000000001},"last":1.8360000000000001},{"interface":"front","average":{"1min":1.446,"5min":1.653,"15min":1.044},"min":{"1min":0.39000000000000001,"5min":0.30199999999999999,"15min":0.28599999999999998},"max":{"1min":4.3680000000000003,"5min":19.248000000000001,"15min":19.248000000000001},"last":0.495}]},{"osd":6,"last update":"Tue Sep 17 07:12:05 2024","interfaces":[{"interface":"back","average":{"1min":1.149,"5min":1.1100000000000001,"15min":0.94699999999999995},"min":{"1min":0.376,"5min":0.31900000000000001,"15min":0.30399999999999999},"max":{"1min":4.3630000000000004,"5min":5.2240000000000002,"15min":18.602},"last":1.4970000000000001},{"interface":"front","average":{"1min":0.81399999999999995,"5min":1.073,"15min":0.996},"min":{"1min":0.42299999999999999,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":2.3170000000000002,"5min":6.149,"15min":18.692},"last":0.50600000000000001}]},{"osd":7,"last update":"Tue Sep 17 07:12:18 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":1.085,"15min":1.3879999999999999},"min":{"1min":0.38400000000000001,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":2.0150000000000001,"5min":4.8490000000000002,"15min":46.183},"last":1.7010000000000001},{"interface":"front","average":{"1min":0.97899999999999998,"5min":1.1299999999999999,"15min":1.1240000000000001},"min":{"1min":0.46600000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":4.4029999999999996,"5min":5.0190000000000001,"15min":45.914999999999999},"last":0.46700000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":186515456,"data_stored":186515456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":198770688,"data_stored":198770688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":99221504,"data_stored":99221504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":69206016,"data_stored":69206016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":175702016,"data_stored":175702016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":52297728,"data_stored":52297728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204472320,"data_stored":204472320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T07:12:24.298 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T07:12:24.298 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:12:24.299 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:23 smithi135 ceph-mon[33238]: 6.14 scrub starts 2024-09-17T07:12:24.299 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:23 smithi135 ceph-mon[33238]: 6.14 scrub ok 2024-09-17T07:12:24.299 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:23 smithi135 ceph-mon[33238]: 6.8 scrub starts 2024-09-17T07:12:24.299 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:23 smithi135 ceph-mon[33238]: 6.8 scrub ok 2024-09-17T07:12:24.299 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:23 smithi135 ceph-mon[33238]: 6.0 scrub starts 2024-09-17T07:12:24.299 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:23 smithi135 ceph-mon[33238]: 6.0 scrub ok 2024-09-17T07:12:24.300 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:23 smithi135 ceph-mon[33238]: 6.1f scrub starts 2024-09-17T07:12:24.300 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:23 smithi135 ceph-mon[33238]: 6.1f scrub ok 2024-09-17T07:12:24.300 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:23 smithi135 ceph-mon[33238]: from='client.15342 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:12:24.300 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:23 smithi135 ceph-mon[33238]: pgmap v455: 148 pgs: 148 active+clean; 335 MiB data, 3.2 GiB used, 623 GiB / 626 GiB avail; 1.9 KiB/s rd, 38 MiB/s wr, 604 op/s 2024-09-17T07:12:24.300 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:23 smithi135 ceph-mon[33238]: 5.12 scrub starts 2024-09-17T07:12:24.300 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:23 smithi135 ceph-mon[33238]: 5.12 scrub ok 2024-09-17T07:12:24.317 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:12:24.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:24.335 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:24.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:12:24.435 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:12:24.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:24.475 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:24.476 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:12:24.576 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:12:24.609 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:24.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:24.610 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:12:24.711 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:12:24.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:24.746 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:24.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:12:24.847 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:12:24.850 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:24 smithi100 ceph-mon[29092]: 1.0 scrub starts 2024-09-17T07:12:24.851 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:24 smithi100 ceph-mon[29092]: 1.0 scrub ok 2024-09-17T07:12:24.851 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:24 smithi100 ceph-mon[29092]: 2.13 scrub starts 2024-09-17T07:12:24.851 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:24 smithi100 ceph-mon[29092]: 2.13 scrub ok 2024-09-17T07:12:24.851 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:24 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:12:24.851 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:24 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:12:24.851 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:24 smithi100 ceph-mon[29092]: osdmap e243: 8 total, 8 up, 7 in 2024-09-17T07:12:24.851 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:24 smithi100 ceph-mon[29092]: 4.1d scrub starts 2024-09-17T07:12:24.852 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:24 smithi100 ceph-mon[29092]: 4.1d scrub ok 2024-09-17T07:12:24.852 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:24 smithi100 ceph-mon[39462]: 1.0 scrub starts 2024-09-17T07:12:24.852 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:24 smithi100 ceph-mon[39462]: 1.0 scrub ok 2024-09-17T07:12:24.852 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:24 smithi100 ceph-mon[39462]: 2.13 scrub starts 2024-09-17T07:12:24.852 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:24 smithi100 ceph-mon[39462]: 2.13 scrub ok 2024-09-17T07:12:24.852 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:24 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:12:24.852 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:24 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:12:24.852 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:24 smithi100 ceph-mon[39462]: osdmap e243: 8 total, 8 up, 7 in 2024-09-17T07:12:24.852 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:24 smithi100 ceph-mon[39462]: 4.1d scrub starts 2024-09-17T07:12:24.852 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:24 smithi100 ceph-mon[39462]: 4.1d scrub ok 2024-09-17T07:12:24.883 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:24.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:24.884 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:12:24.985 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:12:25.021 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:25.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:25.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:12:25.123 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:12:25.158 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:25.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:25.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:12:25.213 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 8 16 23786 23770 185.689 159.062 0.00404231 0.00537997 2024-09-17T07:12:25.261 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:12:25.291 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:25.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:25.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:12:25.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:24 smithi135 ceph-mon[33238]: 1.0 scrub starts 2024-09-17T07:12:25.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:24 smithi135 ceph-mon[33238]: 1.0 scrub ok 2024-09-17T07:12:25.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:24 smithi135 ceph-mon[33238]: 2.13 scrub starts 2024-09-17T07:12:25.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:24 smithi135 ceph-mon[33238]: 2.13 scrub ok 2024-09-17T07:12:25.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:24 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:12:25.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:24 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:12:25.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:24 smithi135 ceph-mon[33238]: osdmap e243: 8 total, 8 up, 7 in 2024-09-17T07:12:25.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:24 smithi135 ceph-mon[33238]: 4.1d scrub starts 2024-09-17T07:12:25.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:24 smithi135 ceph-mon[33238]: 4.1d scrub ok 2024-09-17T07:12:25.393 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:12:25.427 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:25.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:25.428 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:12:25.529 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:12:25.562 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:25.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:25.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:12:25.619 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-17T07:12:25.663 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:12:25.693 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:25.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:25.694 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:12:25.795 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:12:25.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:25.836 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:25.836 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:12:25.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:25 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:12:25.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:25 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:12:25.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:25 smithi100 ceph-mon[29092]: 4.19 scrub starts 2024-09-17T07:12:25.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:25 smithi100 ceph-mon[29092]: 4.19 scrub ok 2024-09-17T07:12:25.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:25 smithi100 ceph-mon[29092]: 5.f scrub starts 2024-09-17T07:12:25.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:25 smithi100 ceph-mon[29092]: 5.f scrub ok 2024-09-17T07:12:25.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:25 smithi100 ceph-mon[29092]: osdmap e244: 8 total, 8 up, 7 in 2024-09-17T07:12:25.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:25 smithi100 ceph-mon[29092]: pgmap v458: 148 pgs: 148 active+clean; 1.2 GiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 55 KiB/s rd, 153 MiB/s wr, 2.53k op/s 2024-09-17T07:12:25.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:25 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:12:25.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:25 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:12:25.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:25 smithi100 ceph-mon[39462]: 4.19 scrub starts 2024-09-17T07:12:25.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:25 smithi100 ceph-mon[39462]: 4.19 scrub ok 2024-09-17T07:12:25.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:25 smithi100 ceph-mon[39462]: 5.f scrub starts 2024-09-17T07:12:25.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:25 smithi100 ceph-mon[39462]: 5.f scrub ok 2024-09-17T07:12:25.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:25 smithi100 ceph-mon[39462]: osdmap e244: 8 total, 8 up, 7 in 2024-09-17T07:12:25.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:25 smithi100 ceph-mon[39462]: pgmap v458: 148 pgs: 148 active+clean; 1.2 GiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 55 KiB/s rd, 153 MiB/s wr, 2.53k op/s 2024-09-17T07:12:25.937 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:12:25.974 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:25.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:25.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:12:26.076 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:12:26.109 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:26.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:26.110 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:12:26.167 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:12:25 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:12:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:12:26.212 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:12:26.213 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 9 15 26320 26305 182.657 158.438 0.00492074 0.00546924 2024-09-17T07:12:26.263 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:12:26.263 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:26.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:26.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:12:26.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:25 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:12:26.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:25 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:12:26.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:25 smithi135 ceph-mon[33238]: 4.19 scrub starts 2024-09-17T07:12:26.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:25 smithi135 ceph-mon[33238]: 4.19 scrub ok 2024-09-17T07:12:26.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:25 smithi135 ceph-mon[33238]: 5.f scrub starts 2024-09-17T07:12:26.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:25 smithi135 ceph-mon[33238]: 5.f scrub ok 2024-09-17T07:12:26.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:25 smithi135 ceph-mon[33238]: osdmap e244: 8 total, 8 up, 7 in 2024-09-17T07:12:26.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:25 smithi135 ceph-mon[33238]: pgmap v458: 148 pgs: 148 active+clean; 1.2 GiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 55 KiB/s rd, 153 MiB/s wr, 2.53k op/s 2024-09-17T07:12:26.310 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:12:26.365 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:12:26.398 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:26.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:26.400 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:12:26.500 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:12:26.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:26.538 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:26.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:12:26.639 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:12:26.675 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:26.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:26.676 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:12:26.777 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:12:26.812 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:26.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:26.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:12:26.914 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:12:26.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:26.951 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:26.951 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:12:27.024 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:26 smithi100 ceph-mon[39462]: 6.3 scrub starts 2024-09-17T07:12:27.024 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:26 smithi100 ceph-mon[39462]: 6.3 scrub ok 2024-09-17T07:12:27.024 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:26 smithi100 ceph-mon[39462]: 6.1a scrub starts 2024-09-17T07:12:27.024 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:26 smithi100 ceph-mon[39462]: 6.1a scrub ok 2024-09-17T07:12:27.024 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:26 smithi100 ceph-mon[39462]: 4.1e scrub starts 2024-09-17T07:12:27.025 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:26 smithi100 ceph-mon[39462]: 4.1e scrub ok 2024-09-17T07:12:27.025 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:26 smithi100 ceph-mon[29092]: 6.3 scrub starts 2024-09-17T07:12:27.025 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:26 smithi100 ceph-mon[29092]: 6.3 scrub ok 2024-09-17T07:12:27.025 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:26 smithi100 ceph-mon[29092]: 6.1a scrub starts 2024-09-17T07:12:27.025 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:26 smithi100 ceph-mon[29092]: 6.1a scrub ok 2024-09-17T07:12:27.025 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:26 smithi100 ceph-mon[29092]: 4.1e scrub starts 2024-09-17T07:12:27.025 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:26 smithi100 ceph-mon[29092]: 4.1e scrub ok 2024-09-17T07:12:27.052 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:12:27.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:27.090 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:27.090 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:12:27.191 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:12:27.214 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 10 16 28466 28450 177.797 134.062 0.00587252 0.00561743 2024-09-17T07:12:27.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:27.224 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:27.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:12:27.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:26 smithi135 ceph-mon[33238]: 6.3 scrub starts 2024-09-17T07:12:27.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:26 smithi135 ceph-mon[33238]: 6.3 scrub ok 2024-09-17T07:12:27.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:26 smithi135 ceph-mon[33238]: 6.1a scrub starts 2024-09-17T07:12:27.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:26 smithi135 ceph-mon[33238]: 6.1a scrub ok 2024-09-17T07:12:27.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:26 smithi135 ceph-mon[33238]: 4.1e scrub starts 2024-09-17T07:12:27.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:26 smithi135 ceph-mon[33238]: 4.1e scrub ok 2024-09-17T07:12:27.325 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:12:27.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:27.362 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:27.362 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:12:27.463 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:12:27.498 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:27.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:27.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:12:27.600 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:12:27.633 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:27.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:27.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:12:27.735 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:12:27.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:27.769 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:27.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:12:27.836 INFO:teuthology.orchestra.run.smithi100.stderr:noscrub is set 2024-09-17T07:12:27.871 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:12:27.906 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:27.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:27.907 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:12:28.008 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:12:28.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:28.030 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:28.030 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:12:28.126 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:27 smithi100 ceph-mon[29092]: 6.16 deep-scrub starts 2024-09-17T07:12:28.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:27 smithi100 ceph-mon[29092]: 6.16 deep-scrub ok 2024-09-17T07:12:28.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:27 smithi100 ceph-mon[29092]: pgmap v459: 148 pgs: 148 active+clean; 1.2 GiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 54 KiB/s rd, 125 MiB/s wr, 2.10k op/s 2024-09-17T07:12:28.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:27 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1482718199' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:12:28.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:27 smithi100 ceph-mon[39462]: 6.16 deep-scrub starts 2024-09-17T07:12:28.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:27 smithi100 ceph-mon[39462]: 6.16 deep-scrub ok 2024-09-17T07:12:28.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:27 smithi100 ceph-mon[39462]: pgmap v459: 148 pgs: 148 active+clean; 1.2 GiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 54 KiB/s rd, 125 MiB/s wr, 2.10k op/s 2024-09-17T07:12:28.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:27 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1482718199' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:12:28.131 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:12:28.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:28.168 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:28.168 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:12:28.213 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 11 16 30605 30589 173.786 133.688 0.00689071 0.0057482 2024-09-17T07:12:28.269 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:12:28.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:27 smithi135 ceph-mon[33238]: 6.16 deep-scrub starts 2024-09-17T07:12:28.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:27 smithi135 ceph-mon[33238]: 6.16 deep-scrub ok 2024-09-17T07:12:28.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:27 smithi135 ceph-mon[33238]: pgmap v459: 148 pgs: 148 active+clean; 1.2 GiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 54 KiB/s rd, 125 MiB/s wr, 2.10k op/s 2024-09-17T07:12:28.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:27 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1482718199' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:12:28.303 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:28.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:28.305 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:12:28.405 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:12:28.437 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:28.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:28.438 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:12:28.539 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:12:28.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:28.573 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:28.573 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:12:28.674 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:12:28.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:28.697 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:28.697 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:12:28.798 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:12:28.832 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:28.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:28.833 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:12:28.934 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:12:28.966 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:28.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:28.967 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:12:29.043 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:12:28 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:12:29.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:28 smithi135 ceph-mon[33238]: 6.1e scrub starts 2024-09-17T07:12:29.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:28 smithi135 ceph-mon[33238]: 6.1e scrub ok 2024-09-17T07:12:29.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:28 smithi135 ceph-mon[33238]: 4.1f scrub starts 2024-09-17T07:12:29.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:28 smithi135 ceph-mon[33238]: 4.1f scrub ok 2024-09-17T07:12:29.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:28 smithi135 ceph-mon[33238]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:12:29.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:28 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1482718199' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T07:12:29.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:28 smithi135 ceph-mon[33238]: osdmap e245: 8 total, 8 up, 7 in 2024-09-17T07:12:29.067 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:12:29.108 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:29.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:29.109 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:12:29.126 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:28 smithi100 ceph-mon[29092]: 6.1e scrub starts 2024-09-17T07:12:29.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:28 smithi100 ceph-mon[29092]: 6.1e scrub ok 2024-09-17T07:12:29.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:28 smithi100 ceph-mon[29092]: 4.1f scrub starts 2024-09-17T07:12:29.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:28 smithi100 ceph-mon[29092]: 4.1f scrub ok 2024-09-17T07:12:29.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:28 smithi100 ceph-mon[29092]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:12:29.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:28 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1482718199' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T07:12:29.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:28 smithi100 ceph-mon[29092]: osdmap e245: 8 total, 8 up, 7 in 2024-09-17T07:12:29.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:28 smithi100 ceph-mon[39462]: 6.1e scrub starts 2024-09-17T07:12:29.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:28 smithi100 ceph-mon[39462]: 6.1e scrub ok 2024-09-17T07:12:29.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:28 smithi100 ceph-mon[39462]: 4.1f scrub starts 2024-09-17T07:12:29.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:28 smithi100 ceph-mon[39462]: 4.1f scrub ok 2024-09-17T07:12:29.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:28 smithi100 ceph-mon[39462]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:12:29.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:28 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1482718199' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T07:12:29.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:28 smithi100 ceph-mon[39462]: osdmap e245: 8 total, 8 up, 7 in 2024-09-17T07:12:29.210 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:12:29.213 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 12 16 32810 32794 170.787 137.812 0.00347215 0.00584816 2024-09-17T07:12:29.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:29.236 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:29.236 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:12:29.337 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:12:29.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:29.360 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:29.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:12:29.461 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:12:29.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:29.502 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:29.502 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:12:29.603 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:12:29.637 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:29.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:29.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:12:29.739 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:12:29.773 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:29.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:29.774 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:12:29.875 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:12:29.913 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:29.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:29.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:12:30.015 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:12:30.050 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:30.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:30.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:12:30.126 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:29 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:12:30.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:29 smithi100 ceph-mon[29092]: pgmap v461: 148 pgs: 148 active+clean; 1.7 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 71 KiB/s rd, 242 MiB/s wr, 3.99k op/s 2024-09-17T07:12:30.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:29 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:12:30.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:29 smithi100 ceph-mon[39462]: pgmap v461: 148 pgs: 148 active+clean; 1.7 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 71 KiB/s rd, 242 MiB/s wr, 3.99k op/s 2024-09-17T07:12:30.152 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:12:30.191 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:30.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:30.193 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:12:30.213 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 13 15 34609 34594 166.303 112.5 0.012067 0.00600797 2024-09-17T07:12:30.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:29 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:12:30.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:29 smithi135 ceph-mon[33238]: pgmap v461: 148 pgs: 148 active+clean; 1.7 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 71 KiB/s rd, 242 MiB/s wr, 3.99k op/s 2024-09-17T07:12:30.294 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:12:30.325 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:30.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:30.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:12:30.428 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:12:30.460 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:30.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:30.461 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:12:30.562 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:12:30.598 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:30.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:30.600 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:12:30.701 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:12:30.736 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:30.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:30.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:12:30.838 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:12:30.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:30.859 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:30.860 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:12:30.960 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:12:30.992 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:30.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:30.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:12:31.093 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:12:31.125 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:31.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:31.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:12:31.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:30 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:12:31.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:30 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:12:31.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:30 smithi100 ceph-mon[39462]: osdmap e246: 8 total, 8 up, 7 in 2024-09-17T07:12:31.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:30 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:12:31.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:30 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:12:31.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:30 smithi100 ceph-mon[29092]: osdmap e246: 8 total, 8 up, 7 in 2024-09-17T07:12:31.214 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 14 16 36550 36534 163.084 121.25 0.00760512 0.006127 2024-09-17T07:12:31.227 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:12:31.266 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:31.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:31.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:12:31.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:30 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:12:31.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:30 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:12:31.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:30 smithi135 ceph-mon[33238]: osdmap e246: 8 total, 8 up, 7 in 2024-09-17T07:12:31.368 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:12:31.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:31.412 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:31.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:12:31.513 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:12:31.543 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:31.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:31.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:12:31.645 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:12:31.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:31.682 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:31.682 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:12:31.783 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:12:31.819 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:31.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:31.820 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:12:31.920 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:12:31.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:31.946 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:31.947 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:12:32.048 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:12:32.079 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:32.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:32.080 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:12:32.126 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:31 smithi100 ceph-mon[39462]: pgmap v463: 148 pgs: 148 active+clean; 1.9 GiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 800 B/s rd, 116 MiB/s wr, 1.86k op/s 2024-09-17T07:12:32.126 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:31 smithi100 ceph-mon[29092]: pgmap v463: 148 pgs: 148 active+clean; 1.9 GiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 800 B/s rd, 116 MiB/s wr, 1.86k op/s 2024-09-17T07:12:32.181 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:12:32.214 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 15 16 38547 38531 160.532 124.812 0.00348793 0.00622252 2024-09-17T07:12:32.217 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:32.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:32.218 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:12:32.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:31 smithi135 ceph-mon[33238]: pgmap v463: 148 pgs: 148 active+clean; 1.9 GiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 800 B/s rd, 116 MiB/s wr, 1.86k op/s 2024-09-17T07:12:32.320 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:12:32.354 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:32.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:32.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:12:32.455 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:12:32.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:32.490 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:32.490 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:12:32.591 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:12:32.627 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:32.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:32.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:12:32.730 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:12:32.766 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:32.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:32.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:12:32.868 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:12:32.900 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:32.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:32.902 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:12:33.002 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:12:33.037 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:33.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:33.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:12:33.138 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:12:33.171 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:33.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:33.172 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:12:33.214 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 16 16 40577 40561 158.428 126.875 0.00405351 0.00630743 2024-09-17T07:12:33.272 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:12:33.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:33.306 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:33.307 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:12:33.408 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:12:33.443 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:33.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:33.444 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:12:33.545 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:12:33.581 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:33.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:33.582 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:12:33.683 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:12:33.722 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:33.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:33.723 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:12:33.824 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:12:33.860 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:33.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:33.862 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:12:33.900 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:33 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:12:33.900 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:33 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:12:33.900 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:33 smithi100 ceph-mon[29092]: pgmap v464: 148 pgs: 148 active+clean; 1.9 GiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 639 B/s rd, 93 MiB/s wr, 1.49k op/s 2024-09-17T07:12:33.900 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:33 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:12:33.901 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:33 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:12:33.901 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:33 smithi100 ceph-mon[39462]: pgmap v464: 148 pgs: 148 active+clean; 1.9 GiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 639 B/s rd, 93 MiB/s wr, 1.49k op/s 2024-09-17T07:12:33.962 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:12:33.997 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:33.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:33.998 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:12:34.001 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-17T07:12:34.001 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-17T07:12:34.001 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-17T07:12:34.001 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-17T07:12:34.002 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.002 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.14805.0:79334 7.18 7:1cf1bf05:::benchmark_data_smithi100_78556_object79333:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-09-17T07:12:34.002 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:05:25.093334+0000", 2024-09-17T07:12:34.002 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 428.90548593199998, 2024-09-17T07:12:34.002 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.17747916599999999, 2024-09-17T07:12:34.002 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:12:34.002 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:12:34.002 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:12:34.002 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.14805", 2024-09-17T07:12:34.002 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:12:34.002 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 79334 2024-09-17T07:12:34.003 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.003 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:12:34.003 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.003 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:12:34.003 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:25.093334+0000", 2024-09-17T07:12:34.003 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:12:34.003 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.003 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.003 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:12:34.003 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:25.093334+0000", 2024-09-17T07:12:34.003 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:12:34.003 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.004 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.004 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:12:34.004 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:25.093333+0000", 2024-09-17T07:12:34.004 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999986 2024-09-17T07:12:34.004 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.004 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.004 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:12:34.004 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:25.093358+0000", 2024-09-17T07:12:34.004 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4613999999999998e-05 2024-09-17T07:12:34.004 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.004 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.004 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:12:34.005 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:25.093366+0000", 2024-09-17T07:12:34.005 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5590000000000006e-06 2024-09-17T07:12:34.005 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.005 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.005 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:12:34.005 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:25.093371+0000", 2024-09-17T07:12:34.005 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2630000000000003e-06 2024-09-17T07:12:34.005 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.005 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.005 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:12:34.005 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:25.100569+0000", 2024-09-17T07:12:34.006 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0071974109999999999 2024-09-17T07:12:34.006 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.006 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.006 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:12:34.006 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:25.100720+0000", 2024-09-17T07:12:34.006 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00015124999999999999 2024-09-17T07:12:34.006 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.006 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.006 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 0,6,7", 2024-09-17T07:12:34.006 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:25.100824+0000", 2024-09-17T07:12:34.006 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000104295 2024-09-17T07:12:34.006 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.006 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.007 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:12:34.007 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:25.102786+0000", 2024-09-17T07:12:34.007 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0019617670000000001 2024-09-17T07:12:34.007 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.007 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.007 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:12:34.007 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:25.103820+0000", 2024-09-17T07:12:34.007 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0010339139999999999 2024-09-17T07:12:34.007 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.007 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.007 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:12:34.007 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:25.111368+0000", 2024-09-17T07:12:34.008 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0075483929999999996 2024-09-17T07:12:34.008 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.008 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.008 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:12:34.008 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:25.270310+0000", 2024-09-17T07:12:34.008 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.15894150400000001 2024-09-17T07:12:34.008 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.008 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.008 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:12:34.008 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:25.270368+0000", 2024-09-17T07:12:34.008 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7936999999999998e-05 2024-09-17T07:12:34.009 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.009 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.009 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:12:34.009 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:25.270813+0000", 2024-09-17T07:12:34.009 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00044558899999999998 2024-09-17T07:12:34.009 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:12:34.009 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:12:34.009 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:12:34.009 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.009 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.009 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.14805.0:86266 7.e 7:701f0743:::benchmark_data_smithi100_78556_object86265:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-17T07:12:34.009 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:05:33.460112+0000", 2024-09-17T07:12:34.009 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 420.53870845199998, 2024-09-17T07:12:34.010 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.17481023600000001, 2024-09-17T07:12:34.010 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:12:34.010 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:12:34.010 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:12:34.010 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.14805", 2024-09-17T07:12:34.010 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:12:34.010 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 86266 2024-09-17T07:12:34.010 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.010 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:12:34.010 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.010 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:12:34.010 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:33.460112+0000", 2024-09-17T07:12:34.011 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:12:34.011 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.011 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.011 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:12:34.011 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:33.460112+0000", 2024-09-17T07:12:34.011 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:12:34.011 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.011 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.011 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:12:34.011 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:33.460110+0000", 2024-09-17T07:12:34.011 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999986 2024-09-17T07:12:34.011 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.011 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.012 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:12:34.012 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:33.460140+0000", 2024-09-17T07:12:34.012 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0119000000000001e-05 2024-09-17T07:12:34.012 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.012 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.012 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:12:34.012 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:33.460152+0000", 2024-09-17T07:12:34.012 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1854e-05 2024-09-17T07:12:34.012 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.012 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.012 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:12:34.012 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:33.460160+0000", 2024-09-17T07:12:34.012 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.3429999999999998e-06 2024-09-17T07:12:34.013 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.013 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.013 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:12:34.013 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:33.460206+0000", 2024-09-17T07:12:34.013 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6381000000000002e-05 2024-09-17T07:12:34.013 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.013 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.013 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:12:34.013 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:33.460325+0000", 2024-09-17T07:12:34.013 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000119122 2024-09-17T07:12:34.013 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.013 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.014 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 0,5", 2024-09-17T07:12:34.014 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:33.460618+0000", 2024-09-17T07:12:34.014 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000292658 2024-09-17T07:12:34.014 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.014 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.014 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:12:34.014 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:33.464720+0000", 2024-09-17T07:12:34.014 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0041026500000000002 2024-09-17T07:12:34.014 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.014 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.014 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:12:34.014 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:33.464730+0000", 2024-09-17T07:12:34.014 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7790000000000007e-06 2024-09-17T07:12:34.014 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.015 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.015 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:12:34.015 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:33.634831+0000", 2024-09-17T07:12:34.015 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.17010106699999999 2024-09-17T07:12:34.015 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.015 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.015 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:12:34.015 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:33.634884+0000", 2024-09-17T07:12:34.015 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3054999999999999e-05 2024-09-17T07:12:34.015 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.015 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.015 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:12:34.016 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:33.634922+0000", 2024-09-17T07:12:34.016 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7771999999999997e-05 2024-09-17T07:12:34.016 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:12:34.016 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:12:34.016 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:12:34.016 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.016 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.016 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.14805.0:86277 7.e 7:7e944240:::benchmark_data_smithi100_78556_object86276:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-17T07:12:34.016 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:05:33.468775+0000", 2024-09-17T07:12:34.016 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 420.53004561900002, 2024-09-17T07:12:34.017 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.17435674100000001, 2024-09-17T07:12:34.017 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:12:34.017 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:12:34.017 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:12:34.018 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.14805", 2024-09-17T07:12:34.018 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:12:34.018 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 86277 2024-09-17T07:12:34.018 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.018 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:12:34.018 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.018 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:12:34.018 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:33.468775+0000", 2024-09-17T07:12:34.018 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:12:34.018 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.018 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.018 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:12:34.018 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:33.468775+0000", 2024-09-17T07:12:34.019 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:12:34.019 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.019 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.019 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:12:34.019 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:33.468773+0000", 2024-09-17T07:12:34.019 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999986 2024-09-17T07:12:34.019 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.019 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.019 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:12:34.019 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:33.468799+0000", 2024-09-17T07:12:34.019 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5652000000000001e-05 2024-09-17T07:12:34.019 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.019 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.020 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:12:34.020 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:33.468811+0000", 2024-09-17T07:12:34.020 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2537999999999999e-05 2024-09-17T07:12:34.020 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.020 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.020 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:12:34.020 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:33.468819+0000", 2024-09-17T07:12:34.020 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.3730000000000004e-06 2024-09-17T07:12:34.020 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.020 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.020 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:12:34.020 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:33.468966+0000", 2024-09-17T07:12:34.021 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014728399999999999 2024-09-17T07:12:34.021 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.021 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.021 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:12:34.021 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:33.469100+0000", 2024-09-17T07:12:34.021 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00013421900000000001 2024-09-17T07:12:34.021 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.021 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.021 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 0,5", 2024-09-17T07:12:34.021 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:33.469242+0000", 2024-09-17T07:12:34.021 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014150999999999999 2024-09-17T07:12:34.021 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.022 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.022 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:12:34.022 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:33.471523+0000", 2024-09-17T07:12:34.022 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.002281894 2024-09-17T07:12:34.022 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.022 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.022 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:12:34.022 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:33.472285+0000", 2024-09-17T07:12:34.022 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00076177099999999995 2024-09-17T07:12:34.022 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.022 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.022 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:12:34.022 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:33.643030+0000", 2024-09-17T07:12:34.023 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.170744904 2024-09-17T07:12:34.023 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.023 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.023 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:12:34.023 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:33.643087+0000", 2024-09-17T07:12:34.023 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7407000000000001e-05 2024-09-17T07:12:34.023 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.023 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.023 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:12:34.023 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:33.643131+0000", 2024-09-17T07:12:34.023 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3841999999999998e-05 2024-09-17T07:12:34.023 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:12:34.023 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:12:34.024 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:12:34.024 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.024 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.024 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.14805.0:90717 7.e 7:79b8186d:::benchmark_data_smithi100_78556_object90716:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-09-17T07:12:34.024 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:05:38.170578+0000", 2024-09-17T07:12:34.024 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 415.8282423, 2024-09-17T07:12:34.024 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.21138723600000001, 2024-09-17T07:12:34.024 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:12:34.024 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:12:34.024 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:12:34.024 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.14805", 2024-09-17T07:12:34.024 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:12:34.024 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 90717 2024-09-17T07:12:34.025 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.025 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:12:34.025 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.025 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:12:34.025 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:38.170578+0000", 2024-09-17T07:12:34.025 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:12:34.025 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.025 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.025 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:12:34.025 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:38.170578+0000", 2024-09-17T07:12:34.025 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:12:34.025 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.026 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.026 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:12:34.026 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:38.170577+0000", 2024-09-17T07:12:34.026 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999986 2024-09-17T07:12:34.026 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.026 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.026 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:12:34.026 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:38.170596+0000", 2024-09-17T07:12:34.026 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.9398999999999998e-05 2024-09-17T07:12:34.026 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.026 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.026 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:12:34.026 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:38.170604+0000", 2024-09-17T07:12:34.027 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7999999999999999e-06 2024-09-17T07:12:34.027 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.027 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.027 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:12:34.027 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:38.170609+0000", 2024-09-17T07:12:34.027 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5389999999999998e-06 2024-09-17T07:12:34.027 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.027 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.027 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:12:34.027 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:38.185849+0000", 2024-09-17T07:12:34.027 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.01524006 2024-09-17T07:12:34.027 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.027 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.028 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:12:34.028 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:38.185978+0000", 2024-09-17T07:12:34.028 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000128932 2024-09-17T07:12:34.028 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.028 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.028 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 0,5", 2024-09-17T07:12:34.028 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:38.186068+0000", 2024-09-17T07:12:34.028 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.9214000000000003e-05 2024-09-17T07:12:34.028 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.028 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.028 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:12:34.028 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:38.196885+0000", 2024-09-17T07:12:34.028 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010817311 2024-09-17T07:12:34.029 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.029 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.029 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:12:34.029 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:38.380070+0000", 2024-09-17T07:12:34.029 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.18318483999999999 2024-09-17T07:12:34.029 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.029 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.029 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:12:34.029 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:38.381887+0000", 2024-09-17T07:12:34.029 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0018170700000000001 2024-09-17T07:12:34.029 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.029 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.030 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:12:34.030 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:38.381933+0000", 2024-09-17T07:12:34.030 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5908999999999997e-05 2024-09-17T07:12:34.030 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.030 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.030 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:12:34.030 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:38.381965+0000", 2024-09-17T07:12:34.030 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.2403000000000001e-05 2024-09-17T07:12:34.030 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:12:34.030 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:12:34.030 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:12:34.030 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.030 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.031 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "MOSDPGPush(7.19 128/114 [PushOp(7:98423951:::benchmark_data_smithi100_78556_object15363:head, version: 105'921, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:98423951:::benchmark_data_smithi100_78556_object15363:head@105'921, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:12:34.031 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:05:38.191939+0000", 2024-09-17T07:12:34.031 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 415.806881648, 2024-09-17T07:12:34.031 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.18722904300000001, 2024-09-17T07:12:34.031 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:12:34.031 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-17T07:12:34.031 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:12:34.031 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.031 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:12:34.031 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:38.191939+0000", 2024-09-17T07:12:34.031 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:12:34.031 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.032 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.032 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:12:34.032 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:38.191939+0000", 2024-09-17T07:12:34.032 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:12:34.032 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.032 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.032 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:12:34.032 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:38.191938+0000", 2024-09-17T07:12:34.032 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-17T07:12:34.032 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.032 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.032 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:12:34.033 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:38.191987+0000", 2024-09-17T07:12:34.033 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9314000000000003e-05 2024-09-17T07:12:34.033 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.033 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.033 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:12:34.033 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:38.192001+0000", 2024-09-17T07:12:34.033 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.3604000000000001e-05 2024-09-17T07:12:34.033 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.033 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.033 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:12:34.033 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:38.192013+0000", 2024-09-17T07:12:34.033 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1776999999999999e-05 2024-09-17T07:12:34.033 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.034 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.034 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:12:34.034 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:38.199047+0000", 2024-09-17T07:12:34.034 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0070342160000000003 2024-09-17T07:12:34.034 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.034 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.034 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:12:34.034 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:38.199058+0000", 2024-09-17T07:12:34.034 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.116e-05 2024-09-17T07:12:34.034 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.034 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.034 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:12:34.035 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:38.379168+0000", 2024-09-17T07:12:34.035 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.180109519 2024-09-17T07:12:34.035 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:12:34.035 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:12:34.035 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:12:34.035 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.035 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.035 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.14805.0:90715 7.19 e128/114 7:9d154c9f:::benchmark_data_smithi100_78556_object90714:head v 128'4395, mlcod=105'920)", 2024-09-17T07:12:34.035 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:05:38.192033+0000", 2024-09-17T07:12:34.035 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 415.80678769299999, 2024-09-17T07:12:34.035 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.18716384599999999, 2024-09-17T07:12:34.035 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:12:34.035 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:12:34.036 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:12:34.036 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.036 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:12:34.036 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:38.192033+0000", 2024-09-17T07:12:34.036 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:12:34.036 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.036 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.036 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:12:34.036 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:38.192033+0000", 2024-09-17T07:12:34.036 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:12:34.036 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.036 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.037 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:12:34.037 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:38.192032+0000", 2024-09-17T07:12:34.037 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-17T07:12:34.037 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.037 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.037 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:12:34.037 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:38.192055+0000", 2024-09-17T07:12:34.037 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2591000000000001e-05 2024-09-17T07:12:34.037 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.037 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.037 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:12:34.037 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:38.192060+0000", 2024-09-17T07:12:34.038 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7119999999999997e-06 2024-09-17T07:12:34.038 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.038 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.038 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:12:34.038 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:38.192065+0000", 2024-09-17T07:12:34.038 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1470000000000003e-06 2024-09-17T07:12:34.038 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.038 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.038 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:12:34.038 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:38.192499+0000", 2024-09-17T07:12:34.038 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000434652 2024-09-17T07:12:34.038 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.039 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.039 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:12:34.039 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:38.192522+0000", 2024-09-17T07:12:34.039 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2807999999999999e-05 2024-09-17T07:12:34.039 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.039 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.039 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:12:34.039 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:38.379171+0000", 2024-09-17T07:12:34.039 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.18664914299999999 2024-09-17T07:12:34.039 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.039 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.039 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:12:34.039 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:38.379196+0000", 2024-09-17T07:12:34.040 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5134e-05 2024-09-17T07:12:34.040 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:12:34.040 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:12:34.040 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:12:34.040 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.040 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.040 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "MOSDPGPush(7.14 128/114 [PushOp(7:2a1bf4e7:::benchmark_data_smithi100_78556_object587:head, version: 103'40, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:2a1bf4e7:::benchmark_data_smithi100_78556_object587:head@103'40, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-17T07:12:34.040 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:05:38.192069+0000", 2024-09-17T07:12:34.040 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 415.80675124800001, 2024-09-17T07:12:34.040 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.18638816, 2024-09-17T07:12:34.040 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:12:34.040 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-17T07:12:34.041 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:12:34.041 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.041 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:12:34.041 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:38.192069+0000", 2024-09-17T07:12:34.041 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:12:34.041 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.041 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.041 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:12:34.041 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:38.192069+0000", 2024-09-17T07:12:34.041 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:12:34.041 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.041 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.042 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:12:34.042 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:38.192069+0000", 2024-09-17T07:12:34.042 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-17T07:12:34.042 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.042 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.042 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:12:34.042 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:38.192094+0000", 2024-09-17T07:12:34.042 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5647e-05 2024-09-17T07:12:34.042 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.042 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.042 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:12:34.042 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:38.192104+0000", 2024-09-17T07:12:34.043 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.6579999999999997e-06 2024-09-17T07:12:34.043 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.043 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.043 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:12:34.043 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:38.192115+0000", 2024-09-17T07:12:34.043 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0961000000000001e-05 2024-09-17T07:12:34.043 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.043 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.043 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:12:34.043 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:38.195880+0000", 2024-09-17T07:12:34.043 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0037652649999999998 2024-09-17T07:12:34.043 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.044 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.044 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:12:34.044 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:38.195892+0000", 2024-09-17T07:12:34.044 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1625e-05 2024-09-17T07:12:34.044 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.044 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.044 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:12:34.044 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:38.378457+0000", 2024-09-17T07:12:34.044 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.18256524399999999 2024-09-17T07:12:34.044 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:12:34.044 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:12:34.044 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:12:34.044 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.045 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.045 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.14805.0:90712 7.14 e128/114 7:2e0cc038:::benchmark_data_smithi100_78556_object90711:head v 128'4556, mlcod=103'35)", 2024-09-17T07:12:34.045 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:05:38.192136+0000", 2024-09-17T07:12:34.045 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 415.80668427000001, 2024-09-17T07:12:34.045 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.18728404100000001, 2024-09-17T07:12:34.045 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:12:34.045 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:12:34.045 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:12:34.045 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.045 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:12:34.045 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:38.192136+0000", 2024-09-17T07:12:34.045 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:12:34.046 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.046 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.046 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:12:34.046 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:38.192136+0000", 2024-09-17T07:12:34.046 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:12:34.046 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.046 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.046 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:12:34.046 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:38.192136+0000", 2024-09-17T07:12:34.046 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-17T07:12:34.046 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.046 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.047 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:12:34.047 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:38.192158+0000", 2024-09-17T07:12:34.047 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2135000000000001e-05 2024-09-17T07:12:34.047 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.047 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.047 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:12:34.047 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:38.192163+0000", 2024-09-17T07:12:34.047 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4280000000000004e-06 2024-09-17T07:12:34.047 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.047 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.047 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:12:34.047 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:38.192168+0000", 2024-09-17T07:12:34.048 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3499999999999999e-06 2024-09-17T07:12:34.048 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.048 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.048 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:12:34.048 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:38.193181+0000", 2024-09-17T07:12:34.048 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0010133799999999999 2024-09-17T07:12:34.048 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.048 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.048 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:12:34.048 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:38.193206+0000", 2024-09-17T07:12:34.048 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4579e-05 2024-09-17T07:12:34.048 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.048 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.049 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:12:34.049 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:38.379397+0000", 2024-09-17T07:12:34.049 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.18619118600000001 2024-09-17T07:12:34.049 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.049 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.049 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:12:34.049 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:38.379420+0000", 2024-09-17T07:12:34.049 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3203000000000001e-05 2024-09-17T07:12:34.049 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:12:34.049 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:12:34.049 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:12:34.049 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.050 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.050 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.14805.0:90730 7.a e128/99 7:586940c8:::benchmark_data_smithi100_78556_object90729:head v 128'5591, mlcod=128'5586)", 2024-09-17T07:12:34.050 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:05:38.194608+0000", 2024-09-17T07:12:34.050 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 415.804212355, 2024-09-17T07:12:34.050 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.184593282, 2024-09-17T07:12:34.050 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:12:34.050 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:12:34.050 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:12:34.050 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.050 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:12:34.050 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:38.194608+0000", 2024-09-17T07:12:34.050 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:12:34.050 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.051 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.051 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:12:34.051 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:38.194608+0000", 2024-09-17T07:12:34.051 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:12:34.051 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.051 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.051 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:12:34.051 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:38.194607+0000", 2024-09-17T07:12:34.051 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999986 2024-09-17T07:12:34.051 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.051 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.051 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:12:34.052 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:38.194637+0000", 2024-09-17T07:12:34.052 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0099000000000001e-05 2024-09-17T07:12:34.052 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.052 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.052 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:12:34.052 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:38.194649+0000", 2024-09-17T07:12:34.052 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2553e-05 2024-09-17T07:12:34.052 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.052 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.052 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:12:34.052 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:38.194657+0000", 2024-09-17T07:12:34.052 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6369999999999999e-06 2024-09-17T07:12:34.053 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.053 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.053 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:12:34.053 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:38.194754+0000", 2024-09-17T07:12:34.053 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7535999999999996e-05 2024-09-17T07:12:34.053 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.053 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.053 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:12:34.053 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:38.194773+0000", 2024-09-17T07:12:34.053 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.8334000000000001e-05 2024-09-17T07:12:34.053 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.053 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.054 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:12:34.054 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:38.199192+0000", 2024-09-17T07:12:34.054 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0044188989999999996 2024-09-17T07:12:34.054 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.054 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.054 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:12:34.054 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:38.379201+0000", 2024-09-17T07:12:34.054 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.18000949299999999 2024-09-17T07:12:34.054 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:12:34.054 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:12:34.054 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:12:34.054 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.055 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.055 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.14805.0:90739 7.8 e129/111 7:1558f93f:::benchmark_data_smithi100_78556_object90738:head v 129'4529, mlcod=128'4527)", 2024-09-17T07:12:34.055 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:05:38.198297+0000", 2024-09-17T07:12:34.055 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 415.80052286599999, 2024-09-17T07:12:34.055 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.181563955, 2024-09-17T07:12:34.055 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:12:34.055 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:12:34.055 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:12:34.055 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.055 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:12:34.055 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:38.198297+0000", 2024-09-17T07:12:34.055 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:12:34.056 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.056 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.056 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:12:34.056 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:38.198297+0000", 2024-09-17T07:12:34.056 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:12:34.056 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.056 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.056 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:12:34.056 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:38.198297+0000", 2024-09-17T07:12:34.056 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-17T07:12:34.056 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.056 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.056 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:12:34.057 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:38.198358+0000", 2024-09-17T07:12:34.057 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1277000000000003e-05 2024-09-17T07:12:34.057 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.057 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.057 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:12:34.057 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:38.198367+0000", 2024-09-17T07:12:34.057 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.7329999999999992e-06 2024-09-17T07:12:34.057 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.057 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.057 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:12:34.057 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:38.198373+0000", 2024-09-17T07:12:34.057 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7000000000000002e-06 2024-09-17T07:12:34.058 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.058 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.058 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:12:34.058 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:38.377751+0000", 2024-09-17T07:12:34.058 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.179377553 2024-09-17T07:12:34.058 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.058 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.058 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:12:34.058 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:38.377776+0000", 2024-09-17T07:12:34.058 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5060000000000001e-05 2024-09-17T07:12:34.058 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.058 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.058 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:12:34.059 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:38.379803+0000", 2024-09-17T07:12:34.059 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0020269120000000001 2024-09-17T07:12:34.059 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.059 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.059 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:12:34.059 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:38.379861+0000", 2024-09-17T07:12:34.059 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8564999999999999e-05 2024-09-17T07:12:34.059 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:12:34.059 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:12:34.059 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:12:34.060 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.060 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.060 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "MOSDPGPushReply(7.13 128/114 [PushReplyOp(7:c859a6cd:::benchmark_data_smithi100_78556_object23308:head)])", 2024-09-17T07:12:34.060 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:05:38.200426+0000", 2024-09-17T07:12:34.060 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 415.79839439300002, 2024-09-17T07:12:34.060 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.18610089099999999, 2024-09-17T07:12:34.060 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:12:34.060 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-17T07:12:34.060 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:12:34.060 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.060 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:12:34.060 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:38.200426+0000", 2024-09-17T07:12:34.061 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:12:34.061 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.061 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.061 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:12:34.061 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:38.200426+0000", 2024-09-17T07:12:34.061 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:12:34.061 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.061 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.061 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:12:34.061 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:38.200425+0000", 2024-09-17T07:12:34.061 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-17T07:12:34.061 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.062 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.062 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:12:34.062 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:38.200432+0000", 2024-09-17T07:12:34.062 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4509999999999997e-06 2024-09-17T07:12:34.062 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.062 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.062 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:12:34.063 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:38.377593+0000", 2024-09-17T07:12:34.063 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.177161649 2024-09-17T07:12:34.063 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.063 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.063 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:12:34.063 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:38.377609+0000", 2024-09-17T07:12:34.063 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5758999999999999e-05 2024-09-17T07:12:34.063 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.063 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.064 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:12:34.064 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:38.385772+0000", 2024-09-17T07:12:34.064 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0081627020000000008 2024-09-17T07:12:34.064 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.064 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.064 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:12:34.064 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:38.386527+0000", 2024-09-17T07:12:34.064 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00075502100000000003 2024-09-17T07:12:34.064 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:12:34.064 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:12:34.064 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:12:34.065 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.065 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.065 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.14805.0:94234 7.e 7:7fa753a4:::benchmark_data_smithi100_78556_object94233:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-17T07:12:34.065 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:05:42.668390+0000", 2024-09-17T07:12:34.065 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 411.330430175, 2024-09-17T07:12:34.065 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.192483182, 2024-09-17T07:12:34.065 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:12:34.065 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:12:34.065 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:12:34.065 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.14805", 2024-09-17T07:12:34.065 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:12:34.065 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 94234 2024-09-17T07:12:34.066 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.066 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:12:34.066 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.066 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:12:34.066 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:42.668390+0000", 2024-09-17T07:12:34.066 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:12:34.066 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.066 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.066 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:12:34.066 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:42.668390+0000", 2024-09-17T07:12:34.066 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:12:34.067 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.067 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.067 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:12:34.067 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:42.668389+0000", 2024-09-17T07:12:34.067 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-17T07:12:34.067 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.067 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.067 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:12:34.067 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:42.668423+0000", 2024-09-17T07:12:34.067 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.4168000000000002e-05 2024-09-17T07:12:34.067 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.067 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.067 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:12:34.068 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:42.668434+0000", 2024-09-17T07:12:34.068 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0502999999999999e-05 2024-09-17T07:12:34.068 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.068 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.068 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:12:34.068 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:42.668440+0000", 2024-09-17T07:12:34.068 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6989999999999997e-06 2024-09-17T07:12:34.068 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.068 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.068 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:12:34.068 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:42.672883+0000", 2024-09-17T07:12:34.068 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0044427729999999997 2024-09-17T07:12:34.068 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.069 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.069 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:12:34.069 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:42.672996+0000", 2024-09-17T07:12:34.069 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00011244299999999999 2024-09-17T07:12:34.069 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.069 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.069 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 0,5", 2024-09-17T07:12:34.069 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:42.673137+0000", 2024-09-17T07:12:34.069 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014109499999999999 2024-09-17T07:12:34.069 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.069 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.069 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:12:34.069 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:42.677367+0000", 2024-09-17T07:12:34.070 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0042299870000000002 2024-09-17T07:12:34.070 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.070 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.070 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:12:34.070 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:42.677374+0000", 2024-09-17T07:12:34.070 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8430000000000004e-06 2024-09-17T07:12:34.070 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.070 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.070 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:12:34.070 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:42.860782+0000", 2024-09-17T07:12:34.070 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.183408077 2024-09-17T07:12:34.070 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.070 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.071 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:12:34.071 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:42.860835+0000", 2024-09-17T07:12:34.071 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3572000000000001e-05 2024-09-17T07:12:34.071 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.071 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.071 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:12:34.071 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:42.860873+0000", 2024-09-17T07:12:34.071 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8000999999999998e-05 2024-09-17T07:12:34.071 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:12:34.071 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:12:34.071 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:12:34.071 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.072 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.072 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.14805.0:94251 7.13 7:cff27f16:::benchmark_data_smithi100_78556_object94250:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-17T07:12:34.072 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:05:42.678917+0000", 2024-09-17T07:12:34.072 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 411.31990339800001, 2024-09-17T07:12:34.072 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.17902530699999999, 2024-09-17T07:12:34.072 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:12:34.072 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:12:34.072 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:12:34.072 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.14805", 2024-09-17T07:12:34.072 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:12:34.072 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 94251 2024-09-17T07:12:34.072 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.073 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:12:34.073 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.073 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:12:34.073 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:42.678917+0000", 2024-09-17T07:12:34.073 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:12:34.073 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.073 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.073 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:12:34.073 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:42.678917+0000", 2024-09-17T07:12:34.073 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:12:34.073 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.073 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.074 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:12:34.074 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:42.678915+0000", 2024-09-17T07:12:34.074 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999986 2024-09-17T07:12:34.074 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.074 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.074 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:12:34.074 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:42.678938+0000", 2024-09-17T07:12:34.074 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2603999999999999e-05 2024-09-17T07:12:34.074 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.074 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.074 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:12:34.074 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:42.678949+0000", 2024-09-17T07:12:34.074 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0607e-05 2024-09-17T07:12:34.075 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.075 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.075 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:12:34.075 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:42.678955+0000", 2024-09-17T07:12:34.075 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4400000000000002e-06 2024-09-17T07:12:34.075 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.075 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.075 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:12:34.075 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:42.679040+0000", 2024-09-17T07:12:34.075 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5091000000000002e-05 2024-09-17T07:12:34.075 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.075 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.075 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:12:34.076 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:42.679203+0000", 2024-09-17T07:12:34.076 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016316599999999999 2024-09-17T07:12:34.076 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.076 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.076 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 0,7", 2024-09-17T07:12:34.076 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:42.679392+0000", 2024-09-17T07:12:34.076 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00018854200000000001 2024-09-17T07:12:34.076 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.076 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.076 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:12:34.076 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:42.835673+0000", 2024-09-17T07:12:34.076 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.156281584 2024-09-17T07:12:34.077 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.077 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.077 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:12:34.077 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:42.835682+0000", 2024-09-17T07:12:34.077 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6440000000000006e-06 2024-09-17T07:12:34.077 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.077 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.077 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:12:34.077 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:42.857208+0000", 2024-09-17T07:12:34.077 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.021525638 2024-09-17T07:12:34.077 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.077 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.077 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:12:34.078 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:42.857283+0000", 2024-09-17T07:12:34.078 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4936000000000002e-05 2024-09-17T07:12:34.078 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.078 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.078 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:12:34.078 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:42.857942+0000", 2024-09-17T07:12:34.078 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00065951900000000001 2024-09-17T07:12:34.078 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:12:34.078 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:12:34.078 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:12:34.078 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.078 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.079 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.14805.0:94252 7.e 7:752146a9:::benchmark_data_smithi100_78556_object94251:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-17T07:12:34.079 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:05:42.678970+0000", 2024-09-17T07:12:34.079 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 411.31985004000001, 2024-09-17T07:12:34.079 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.18597355199999999, 2024-09-17T07:12:34.079 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:12:34.079 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:12:34.079 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:12:34.079 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.14805", 2024-09-17T07:12:34.079 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:12:34.079 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 94252 2024-09-17T07:12:34.079 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.079 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:12:34.079 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.080 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:12:34.080 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:42.678970+0000", 2024-09-17T07:12:34.080 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:12:34.080 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.080 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.080 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:12:34.080 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:42.678970+0000", 2024-09-17T07:12:34.080 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:12:34.080 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.080 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.080 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:12:34.080 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:42.678970+0000", 2024-09-17T07:12:34.081 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-17T07:12:34.081 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.081 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.081 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:12:34.081 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:42.678987+0000", 2024-09-17T07:12:34.081 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7413000000000001e-05 2024-09-17T07:12:34.081 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.081 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.081 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:12:34.081 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:42.678993+0000", 2024-09-17T07:12:34.081 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8459999999999996e-06 2024-09-17T07:12:34.081 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.082 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.082 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:12:34.082 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:42.678997+0000", 2024-09-17T07:12:34.082 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5290000000000002e-06 2024-09-17T07:12:34.082 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.082 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.082 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:12:34.082 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:42.679252+0000", 2024-09-17T07:12:34.082 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000254489 2024-09-17T07:12:34.082 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.082 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.082 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:12:34.082 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:42.679369+0000", 2024-09-17T07:12:34.083 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000117113 2024-09-17T07:12:34.083 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.083 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.083 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 0,5", 2024-09-17T07:12:34.083 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:42.679466+0000", 2024-09-17T07:12:34.083 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7194000000000005e-05 2024-09-17T07:12:34.083 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.083 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.083 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:12:34.083 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:42.835749+0000", 2024-09-17T07:12:34.083 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.15628266900000001 2024-09-17T07:12:34.083 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.084 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.084 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:12:34.084 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:42.835898+0000", 2024-09-17T07:12:34.084 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014890300000000001 2024-09-17T07:12:34.084 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.084 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.084 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:12:34.084 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:42.864849+0000", 2024-09-17T07:12:34.084 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.028951530999999999 2024-09-17T07:12:34.084 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.085 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.085 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:12:34.085 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:42.864905+0000", 2024-09-17T07:12:34.085 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5998000000000002e-05 2024-09-17T07:12:34.085 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.085 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.085 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:12:34.085 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:42.864944+0000", 2024-09-17T07:12:34.085 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8365000000000003e-05 2024-09-17T07:12:34.085 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:12:34.085 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:12:34.085 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:12:34.085 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.086 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.086 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.14805.0:94483 7.13 7:cedb9871:::benchmark_data_smithi100_78556_object94482:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-17T07:12:34.086 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:05:43.061040+0000", 2024-09-17T07:12:34.086 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 410.937780003, 2024-09-17T07:12:34.086 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.191785812, 2024-09-17T07:12:34.086 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:12:34.086 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:12:34.086 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:12:34.086 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.14805", 2024-09-17T07:12:34.086 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:12:34.086 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 94483 2024-09-17T07:12:34.086 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.086 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:12:34.087 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.087 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:12:34.087 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:43.061040+0000", 2024-09-17T07:12:34.087 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:12:34.087 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.087 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.087 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:12:34.087 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:43.061040+0000", 2024-09-17T07:12:34.087 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:12:34.087 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.087 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.087 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:12:34.088 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:43.061038+0000", 2024-09-17T07:12:34.088 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999976 2024-09-17T07:12:34.088 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.088 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.088 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:12:34.088 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:43.061071+0000", 2024-09-17T07:12:34.088 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.2727e-05 2024-09-17T07:12:34.088 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.088 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.088 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:12:34.088 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:43.061088+0000", 2024-09-17T07:12:34.088 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7589999999999999e-05 2024-09-17T07:12:34.088 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.089 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.089 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:12:34.089 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:43.061096+0000", 2024-09-17T07:12:34.089 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5689999999999999e-06 2024-09-17T07:12:34.089 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.089 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.089 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:12:34.089 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:43.061844+0000", 2024-09-17T07:12:34.089 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00074800400000000003 2024-09-17T07:12:34.089 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.089 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.089 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:12:34.089 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:43.061973+0000", 2024-09-17T07:12:34.090 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012960399999999999 2024-09-17T07:12:34.090 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.090 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.090 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 0,7", 2024-09-17T07:12:34.090 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:43.062111+0000", 2024-09-17T07:12:34.090 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00013746599999999999 2024-09-17T07:12:34.090 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.090 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.090 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:12:34.090 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:43.063784+0000", 2024-09-17T07:12:34.090 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001672991 2024-09-17T07:12:34.090 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.091 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.091 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:12:34.091 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:43.064767+0000", 2024-09-17T07:12:34.091 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00098304899999999995 2024-09-17T07:12:34.091 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.091 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.091 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:12:34.091 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:43.249987+0000", 2024-09-17T07:12:34.091 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.18522053899999999 2024-09-17T07:12:34.091 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.091 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.091 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:12:34.091 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:43.250828+0000", 2024-09-17T07:12:34.092 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00084073199999999996 2024-09-17T07:12:34.092 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.092 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.092 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:12:34.092 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:43.252826+0000", 2024-09-17T07:12:34.092 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0019978190000000001 2024-09-17T07:12:34.092 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:12:34.092 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:12:34.092 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:12:34.092 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.092 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.092 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.14805.0:94484 7.e 7:7b75f1c9:::benchmark_data_smithi100_78556_object94483:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-17T07:12:34.093 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:05:43.061118+0000", 2024-09-17T07:12:34.093 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 410.937702484, 2024-09-17T07:12:34.093 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.18942290000000001, 2024-09-17T07:12:34.093 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:12:34.093 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:12:34.093 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:12:34.093 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.14805", 2024-09-17T07:12:34.093 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:12:34.093 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 94484 2024-09-17T07:12:34.093 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.093 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:12:34.093 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.093 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:12:34.094 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:43.061118+0000", 2024-09-17T07:12:34.094 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:12:34.094 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.094 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.094 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:12:34.094 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:43.061118+0000", 2024-09-17T07:12:34.094 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:12:34.094 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.094 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.094 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:12:34.094 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:43.061117+0000", 2024-09-17T07:12:34.094 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-17T07:12:34.094 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.095 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.095 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:12:34.095 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:43.061143+0000", 2024-09-17T07:12:34.095 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5746e-05 2024-09-17T07:12:34.095 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.095 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.095 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:12:34.095 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:43.061150+0000", 2024-09-17T07:12:34.095 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9430000000000003e-06 2024-09-17T07:12:34.095 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.095 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.095 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:12:34.095 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:43.061155+0000", 2024-09-17T07:12:34.096 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0599999999999998e-06 2024-09-17T07:12:34.096 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.096 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.096 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:12:34.096 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:43.061336+0000", 2024-09-17T07:12:34.096 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00018128999999999999 2024-09-17T07:12:34.096 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.096 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.096 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:12:34.096 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:43.061447+0000", 2024-09-17T07:12:34.096 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000110833 2024-09-17T07:12:34.096 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.097 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.097 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 0,5", 2024-09-17T07:12:34.097 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:43.061577+0000", 2024-09-17T07:12:34.097 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000129942 2024-09-17T07:12:34.097 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.097 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.097 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:12:34.097 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:43.064833+0000", 2024-09-17T07:12:34.098 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0032554699999999999 2024-09-17T07:12:34.098 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.098 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.098 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:12:34.098 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:43.064988+0000", 2024-09-17T07:12:34.098 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00015527499999999999 2024-09-17T07:12:34.098 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.098 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.098 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:12:34.098 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:43.250138+0000", 2024-09-17T07:12:34.099 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.185149815 2024-09-17T07:12:34.099 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.099 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.099 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:12:34.099 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:43.250494+0000", 2024-09-17T07:12:34.099 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00035599399999999999 2024-09-17T07:12:34.099 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.099 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.099 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:12:34.099 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:43.250541+0000", 2024-09-17T07:12:34.099 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7015999999999998e-05 2024-09-17T07:12:34.099 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:12:34.100 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:12:34.100 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:12:34.100 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.100 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.100 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.14805.0:94492 7.e 7:7b451ade:::benchmark_data_smithi100_78556_object94491:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-17T07:12:34.100 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:05:43.072022+0000", 2024-09-17T07:12:34.100 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 410.92679783, 2024-09-17T07:12:34.100 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.178680491, 2024-09-17T07:12:34.100 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:12:34.100 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:12:34.100 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:12:34.100 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.14805", 2024-09-17T07:12:34.100 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:12:34.101 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 94492 2024-09-17T07:12:34.101 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.101 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:12:34.101 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.101 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:12:34.101 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:43.072022+0000", 2024-09-17T07:12:34.101 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:12:34.101 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.101 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.101 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:12:34.101 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:43.072022+0000", 2024-09-17T07:12:34.101 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:12:34.101 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.102 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.102 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:12:34.102 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:43.072020+0000", 2024-09-17T07:12:34.102 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999981 2024-09-17T07:12:34.102 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.102 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.102 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:12:34.102 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:43.072043+0000", 2024-09-17T07:12:34.102 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2904999999999998e-05 2024-09-17T07:12:34.102 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.102 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.102 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:12:34.102 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:43.072055+0000", 2024-09-17T07:12:34.103 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1878999999999999e-05 2024-09-17T07:12:34.103 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.103 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.103 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:12:34.103 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:43.072061+0000", 2024-09-17T07:12:34.103 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7089999999999999e-06 2024-09-17T07:12:34.103 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.103 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.103 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:12:34.103 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:43.072104+0000", 2024-09-17T07:12:34.103 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3211999999999999e-05 2024-09-17T07:12:34.103 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.103 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.104 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:12:34.104 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:43.072190+0000", 2024-09-17T07:12:34.104 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5575999999999995e-05 2024-09-17T07:12:34.104 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.104 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.104 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 0,5", 2024-09-17T07:12:34.104 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:43.072276+0000", 2024-09-17T07:12:34.104 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6043000000000003e-05 2024-09-17T07:12:34.104 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.104 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.104 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:12:34.104 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:43.074515+0000", 2024-09-17T07:12:34.104 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0022392599999999999 2024-09-17T07:12:34.105 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.105 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.105 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:12:34.105 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:43.074572+0000", 2024-09-17T07:12:34.105 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7003999999999998e-05 2024-09-17T07:12:34.105 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.105 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.105 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:12:34.105 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:43.250590+0000", 2024-09-17T07:12:34.105 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.17601846400000001 2024-09-17T07:12:34.105 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.105 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.105 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:12:34.106 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:43.250670+0000", 2024-09-17T07:12:34.106 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9767999999999999e-05 2024-09-17T07:12:34.106 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.106 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.106 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:12:34.106 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:43.250703+0000", 2024-09-17T07:12:34.106 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.2727e-05 2024-09-17T07:12:34.106 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:12:34.106 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:12:34.106 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:12:34.106 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.106 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.106 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.14805.0:95019 7.e 7:7e4bb94b:::benchmark_data_smithi100_78556_object95018:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-17T07:12:34.107 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:05:43.859894+0000", 2024-09-17T07:12:34.107 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 410.138926449, 2024-09-17T07:12:34.107 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.19306743800000001, 2024-09-17T07:12:34.107 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:12:34.107 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:12:34.107 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:12:34.107 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.14805", 2024-09-17T07:12:34.107 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:12:34.107 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 95019 2024-09-17T07:12:34.107 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.107 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:12:34.107 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.107 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:12:34.108 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:43.859894+0000", 2024-09-17T07:12:34.108 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:12:34.108 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.108 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.108 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:12:34.108 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:43.859894+0000", 2024-09-17T07:12:34.108 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:12:34.108 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.108 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.108 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:12:34.108 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:12:34.110 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:43.859893+0000", 2024-09-17T07:12:34.110 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-17T07:12:34.110 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.110 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.110 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:12:34.110 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:43.859914+0000", 2024-09-17T07:12:34.110 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1016999999999999e-05 2024-09-17T07:12:34.110 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.110 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.110 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:12:34.111 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:43.859922+0000", 2024-09-17T07:12:34.111 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9140000000000008e-06 2024-09-17T07:12:34.111 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.111 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.111 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:12:34.111 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:43.859928+0000", 2024-09-17T07:12:34.111 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7679999999999997e-06 2024-09-17T07:12:34.111 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.111 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.111 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:12:34.111 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:43.865917+0000", 2024-09-17T07:12:34.111 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0059894550000000003 2024-09-17T07:12:34.111 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.112 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.112 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:12:34.112 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:43.866023+0000", 2024-09-17T07:12:34.112 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000105357 2024-09-17T07:12:34.112 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.112 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.112 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 0,5", 2024-09-17T07:12:34.112 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:43.866157+0000", 2024-09-17T07:12:34.112 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00013432400000000001 2024-09-17T07:12:34.112 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.112 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.112 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:12:34.112 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:43.869050+0000", 2024-09-17T07:12:34.113 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0028935269999999999 2024-09-17T07:12:34.113 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.113 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.113 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:12:34.113 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:43.869266+0000", 2024-09-17T07:12:34.113 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00021562099999999999 2024-09-17T07:12:34.113 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.113 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.113 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:12:34.113 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:44.052869+0000", 2024-09-17T07:12:34.113 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.18360257699999999 2024-09-17T07:12:34.113 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.114 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.114 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:12:34.114 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:44.052920+0000", 2024-09-17T07:12:34.114 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1536000000000003e-05 2024-09-17T07:12:34.114 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.114 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.114 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:12:34.114 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:44.052961+0000", 2024-09-17T07:12:34.114 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1032000000000003e-05 2024-09-17T07:12:34.114 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:12:34.114 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:12:34.114 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:12:34.114 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.115 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.115 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.14805.0:95035 7.e 7:7c41d6d0:::benchmark_data_smithi100_78556_object95034:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-17T07:12:34.115 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:05:43.871991+0000", 2024-09-17T07:12:34.115 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 410.126829262, 2024-09-17T07:12:34.115 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.185888211, 2024-09-17T07:12:34.115 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:12:34.115 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:12:34.115 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:12:34.115 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.14805", 2024-09-17T07:12:34.115 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:12:34.115 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 95035 2024-09-17T07:12:34.115 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.115 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:12:34.116 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.116 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:12:34.116 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:43.871991+0000", 2024-09-17T07:12:34.116 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:12:34.116 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.116 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.116 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:12:34.116 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:43.871991+0000", 2024-09-17T07:12:34.116 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:12:34.116 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.116 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.116 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:12:34.117 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:43.871990+0000", 2024-09-17T07:12:34.117 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999986 2024-09-17T07:12:34.117 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.117 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.117 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:12:34.117 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:43.872008+0000", 2024-09-17T07:12:34.117 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.8606000000000001e-05 2024-09-17T07:12:34.117 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.117 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.117 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:12:34.117 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:43.872017+0000", 2024-09-17T07:12:34.118 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.1279999999999997e-06 2024-09-17T07:12:34.118 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.118 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.118 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:12:34.118 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:43.872023+0000", 2024-09-17T07:12:34.118 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3229999999999997e-06 2024-09-17T07:12:34.118 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.118 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.118 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:12:34.118 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:43.872059+0000", 2024-09-17T07:12:34.118 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6559e-05 2024-09-17T07:12:34.118 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.118 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.119 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:12:34.119 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:43.872160+0000", 2024-09-17T07:12:34.119 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000100443 2024-09-17T07:12:34.119 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.119 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.119 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 0,5", 2024-09-17T07:12:34.119 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:43.872246+0000", 2024-09-17T07:12:34.119 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6002999999999996e-05 2024-09-17T07:12:34.119 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.119 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.119 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:12:34.119 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:43.874389+0000", 2024-09-17T07:12:34.119 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0021434840000000002 2024-09-17T07:12:34.120 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.120 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.120 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:12:34.120 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:43.874405+0000", 2024-09-17T07:12:34.120 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5209e-05 2024-09-17T07:12:34.120 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.120 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.120 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:12:34.120 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:44.057751+0000", 2024-09-17T07:12:34.120 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.18334653100000001 2024-09-17T07:12:34.120 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.120 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.121 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:12:34.121 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:44.057843+0000", 2024-09-17T07:12:34.121 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.1467999999999999e-05 2024-09-17T07:12:34.121 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.121 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.121 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:12:34.121 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:44.057879+0000", 2024-09-17T07:12:34.121 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6653000000000003e-05 2024-09-17T07:12:34.121 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:12:34.121 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:12:34.121 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:12:34.121 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.121 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.122 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.14805.0:99069 7.e 7:763c661c:::benchmark_data_smithi100_78556_object99068:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-17T07:12:34.122 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:05:48.678324+0000", 2024-09-17T07:12:34.122 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 405.32049589500002, 2024-09-17T07:12:34.122 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.17728213700000001, 2024-09-17T07:12:34.124 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:12:34.124 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:12:34.124 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:12:34.124 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.14805", 2024-09-17T07:12:34.124 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:12:34.124 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 99069 2024-09-17T07:12:34.125 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.125 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:12:34.125 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.125 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:12:34.125 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:48.678324+0000", 2024-09-17T07:12:34.125 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:12:34.125 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.125 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.125 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:12:34.125 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:48.678324+0000", 2024-09-17T07:12:34.125 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:12:34.125 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.125 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.126 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:12:34.126 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:48.678323+0000", 2024-09-17T07:12:34.126 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999986 2024-09-17T07:12:34.126 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.126 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.126 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:12:34.126 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:48.678349+0000", 2024-09-17T07:12:34.126 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5831999999999999e-05 2024-09-17T07:12:34.126 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.126 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.126 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:12:34.126 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:48.678360+0000", 2024-09-17T07:12:34.126 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1684e-05 2024-09-17T07:12:34.127 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.127 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.127 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:12:34.127 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:48.678368+0000", 2024-09-17T07:12:34.127 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5310000000000004e-06 2024-09-17T07:12:34.127 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.127 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.127 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:12:34.127 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:48.679351+0000", 2024-09-17T07:12:34.127 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000982915 2024-09-17T07:12:34.127 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.127 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.128 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:12:34.128 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:48.679469+0000", 2024-09-17T07:12:34.128 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000118056 2024-09-17T07:12:34.128 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.128 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.128 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 0,5", 2024-09-17T07:12:34.128 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:48.679600+0000", 2024-09-17T07:12:34.128 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00013086900000000001 2024-09-17T07:12:34.128 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.128 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.128 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:12:34.128 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:48.682577+0000", 2024-09-17T07:12:34.128 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0029776659999999999 2024-09-17T07:12:34.129 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.129 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.129 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:12:34.129 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:48.685432+0000", 2024-09-17T07:12:34.129 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0028549510000000001 2024-09-17T07:12:34.129 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.129 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.129 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:12:34.129 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:48.855520+0000", 2024-09-17T07:12:34.129 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.17008757699999999 2024-09-17T07:12:34.129 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.129 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.130 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:12:34.130 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:48.855570+0000", 2024-09-17T07:12:34.130 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0126000000000001e-05 2024-09-17T07:12:34.130 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:12:34.130 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:12:34.130 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:12:34.130 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:48.855606+0000", 2024-09-17T07:12:34.130 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6550000000000001e-05 2024-09-17T07:12:34.130 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:12:34.130 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:12:34.130 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:12:34.130 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:12:34.131 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:12:34.131 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-17T07:12:34.131 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:12:34.131 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1: {} 2024-09-17T07:12:34.131 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:12:34.131 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:12:34.131 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:12:34.131 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:12:34.132 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:12:34.132 INFO:teuthology.orchestra.run.smithi100.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:12:34.132 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:12:34.132 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:12:34.132 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:12:34.132 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:12:34.132 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:12:34.133 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:12:34.141 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:12:34.141 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:12:34.143 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:34.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:34.144 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:12:34.214 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 17 16 42438 42422 155.95 116.312 0.00892994 0.00640793 2024-09-17T07:12:34.244 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:12:34.278 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:34.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:34.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:12:34.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:33 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:12:34.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:33 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:12:34.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:33 smithi135 ceph-mon[33238]: pgmap v464: 148 pgs: 148 active+clean; 1.9 GiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 639 B/s rd, 93 MiB/s wr, 1.49k op/s 2024-09-17T07:12:34.380 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:12:34.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:34.419 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:34.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:12:34.520 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:12:34.553 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:34.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:34.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:12:34.655 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:12:34.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:34.690 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:34.690 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:12:34.791 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:12:34.832 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:34.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:34.833 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:12:34.934 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:12:34.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:34.971 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:34.971 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:12:35.072 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:12:35.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:35.113 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:35.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:12:35.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:34 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/828821423' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:12:35.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:34 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/828821423' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:12:35.214 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:12:35.214 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 18 16 44355 44339 153.941 119.812 0.00997729 0.00649086 2024-09-17T07:12:35.254 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:35.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:35.255 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:12:35.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:34 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/828821423' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:12:35.356 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:12:35.393 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:35.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:35.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:12:35.494 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:12:35.526 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:35.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:35.527 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:12:35.614 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:12:35.627 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:12:35.662 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:35.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:35.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:12:35.764 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:12:35.810 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:35.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:35.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:12:35.913 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:12:35.949 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:35.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:35.950 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:12:36.051 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:12:36.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:36.095 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:36.095 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:12:36.096 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:35 smithi100 ceph-mon[29092]: pgmap v465: 148 pgs: 148 active+clean; 2.5 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 170 MiB/s wr, 2.72k op/s 2024-09-17T07:12:36.096 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:12:35 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:12:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:12:36.096 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:35 smithi100 ceph-mon[39462]: pgmap v465: 148 pgs: 148 active+clean; 2.5 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 170 MiB/s wr, 2.72k op/s 2024-09-17T07:12:36.098 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-17T07:12:36.197 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:12:36.214 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 19 16 46205 46189 151.923 115.625 0.00624455 0.00657648 2024-09-17T07:12:36.234 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:36.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:36.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:12:36.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:35 smithi135 ceph-mon[33238]: pgmap v465: 148 pgs: 148 active+clean; 2.5 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 170 MiB/s wr, 2.72k op/s 2024-09-17T07:12:36.337 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:12:36.369 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:36.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:36.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:12:36.471 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:12:36.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:36.516 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:36.516 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:12:36.617 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:12:36.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:36.659 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:36.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:12:36.760 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:12:36.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:36.794 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:36.795 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:12:36.896 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:12:36.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:36.930 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:36.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:12:37.031 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:12:37.064 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:37.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:37.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:12:37.126 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:36 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:12:37.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:36 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:12:37.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:36 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:12:37.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:36 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:12:37.166 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:12:37.201 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:37.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:37.202 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:12:37.214 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:2024-09-17T07:12:37.214900+0000 min lat: 0.00110426 max lat: 0.189997 avg lat: 0.00666675 2024-09-17T07:12:37.214 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-17T07:12:37.214 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 20 16 47983 47967 149.883 111.125 0.0147424 0.00666675 2024-09-17T07:12:37.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:36 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:12:37.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:36 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:12:37.304 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:12:37.332 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:37.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:37.333 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:12:37.434 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:12:37.472 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:37.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:37.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:12:37.555 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-17T07:12:37.574 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:12:37.645 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:37.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:37.646 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:12:37.747 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:12:37.806 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:37.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:37.807 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:12:37.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:37 smithi100 ceph-mon[29092]: osdmap e247: 8 total, 8 up, 7 in 2024-09-17T07:12:37.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:37 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:12:37.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:37 smithi100 ceph-mon[29092]: pgmap v467: 148 pgs: 148 active+clean; 2.5 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 639 B/s rd, 99 MiB/s wr, 1.59k op/s 2024-09-17T07:12:37.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:37 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:12:37.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:37 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:12:37.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:37 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:12:37.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:37 smithi100 ceph-mon[39462]: osdmap e247: 8 total, 8 up, 7 in 2024-09-17T07:12:37.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:37 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:12:37.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:37 smithi100 ceph-mon[39462]: pgmap v467: 148 pgs: 148 active+clean; 2.5 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 639 B/s rd, 99 MiB/s wr, 1.59k op/s 2024-09-17T07:12:37.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:37 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:12:37.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:37 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:12:37.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:37 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:12:37.908 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:12:37.946 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:37.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:37.947 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:12:38.048 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:12:38.078 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:38.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:38.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:12:38.145 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:12:38.180 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:12:38.194 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:37 smithi135 ceph-mon[33238]: osdmap e247: 8 total, 8 up, 7 in 2024-09-17T07:12:38.194 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:37 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:12:38.195 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:37 smithi135 ceph-mon[33238]: pgmap v467: 148 pgs: 148 active+clean; 2.5 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 639 B/s rd, 99 MiB/s wr, 1.59k op/s 2024-09-17T07:12:38.195 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:37 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:12:38.195 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:37 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:12:38.195 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:37 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:12:38.216 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 21 16 49584 49568 147.51 100.062 0.0132335 0.00677387 2024-09-17T07:12:38.236 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:38.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:38.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:12:38.342 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:12:38.374 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:38.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:38.375 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:12:38.475 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:12:38.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:38.510 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:38.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:12:38.611 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:12:38.644 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:38.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:38.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:12:38.746 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:12:38.783 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:38.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:38.784 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:12:38.884 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:12:38.914 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:38.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:38.915 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:12:39.016 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:12:39.043 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:12:38 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:12:39.046 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:39.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:39.047 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:12:39.148 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:12:39.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:39.182 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:39.183 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:12:39.214 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 22 16 51248 51232 145.532 104 0.049368 0.00685704 2024-09-17T07:12:39.283 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:12:39.318 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:39.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:39.319 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:12:39.420 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:12:39.457 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:39.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:39.459 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:12:39.559 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:12:39.600 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:39.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:39.601 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:12:39.643 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:12:39.702 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:12:39.739 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:39.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:39.740 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:12:39.841 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:12:39.877 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:39.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:39.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:12:40.070 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:12:40.071 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:12:40.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:40.109 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:40.109 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:12:40.210 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:12:40.214 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 23 16 52831 52815 143.505 98.9375 0.00730095 0.00696382 2024-09-17T07:12:40.240 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:40.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:40.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:12:40.342 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:12:40.354 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:40 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:12:40.354 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:40 smithi100 ceph-mon[29092]: pgmap v468: 148 pgs: 148 active+clean; 2.9 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.1 KiB/s rd, 110 MiB/s wr, 1.77k op/s 2024-09-17T07:12:40.354 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:40 smithi100 ceph-mon[29092]: osdmap e248: 8 total, 8 up, 7 in 2024-09-17T07:12:40.354 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:40 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:12:40.354 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:40 smithi100 ceph-mon[39462]: pgmap v468: 148 pgs: 148 active+clean; 2.9 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.1 KiB/s rd, 110 MiB/s wr, 1.77k op/s 2024-09-17T07:12:40.354 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:40 smithi100 ceph-mon[39462]: osdmap e248: 8 total, 8 up, 7 in 2024-09-17T07:12:40.374 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:40.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:40.375 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:12:40.476 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:12:40.509 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:40.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:40.510 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:12:40.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:40 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:12:40.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:40 smithi135 ceph-mon[33238]: pgmap v468: 148 pgs: 148 active+clean; 2.9 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.1 KiB/s rd, 110 MiB/s wr, 1.77k op/s 2024-09-17T07:12:40.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:40 smithi135 ceph-mon[33238]: osdmap e248: 8 total, 8 up, 7 in 2024-09-17T07:12:40.611 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:12:40.651 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:40.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:40.652 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:12:40.753 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:12:40.788 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:40.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:40.789 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:12:40.890 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:12:40.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:40.924 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:40.924 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:12:41.025 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:12:41.063 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:41.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:41.064 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:12:41.165 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:12:41.198 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:41.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:41.199 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:12:41.215 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 24 16 54482 54466 141.825 103.188 0.00924058 0.00704646 2024-09-17T07:12:41.300 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:12:41.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:41.340 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:41.340 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:12:41.441 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:12:41.478 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:41.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:41.479 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:12:41.580 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:12:41.606 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T07:12:41.609 INFO:teuthology.orchestra.run.smithi100.stderr:dumped all 2024-09-17T07:12:41.613 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:41.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:41.614 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:12:41.715 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:12:41.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:41.752 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:41.753 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:12:41.854 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:12:41.893 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:41.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:41.894 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:12:41.995 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:12:42.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:42.032 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:42.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:12:42.133 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:12:42.162 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-17T07:12:42.163 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-17T07:12:42.163 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-17T07:12:42.163 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-17T07:12:42.165 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:42.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:42.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:12:42.215 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 25 16 56047 56031 140.064 97.8125 0.0059625 0.00713453 2024-09-17T07:12:42.267 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:12:42.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:42.307 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:42.307 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:12:42.371 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:12:42.408 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:12:42.445 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:42.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:42.446 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:12:42.508 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:42 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:12:42.508 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:42 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:12:42.509 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:42 smithi100 ceph-mon[39462]: pgmap v470: 148 pgs: 148 active+clean; 3.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 147 MiB/s wr, 2.35k op/s 2024-09-17T07:12:42.509 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:42 smithi100 ceph-mon[39462]: osdmap e249: 8 total, 8 up, 7 in 2024-09-17T07:12:42.509 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:42 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:12:42.509 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:42 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:12:42.509 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:42 smithi100 ceph-mon[29092]: pgmap v470: 148 pgs: 148 active+clean; 3.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 147 MiB/s wr, 2.35k op/s 2024-09-17T07:12:42.509 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:42 smithi100 ceph-mon[29092]: osdmap e249: 8 total, 8 up, 7 in 2024-09-17T07:12:42.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:42 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:12:42.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:42 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:12:42.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:42 smithi135 ceph-mon[33238]: pgmap v470: 148 pgs: 148 active+clean; 3.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 147 MiB/s wr, 2.35k op/s 2024-09-17T07:12:42.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:42 smithi135 ceph-mon[33238]: osdmap e249: 8 total, 8 up, 7 in 2024-09-17T07:12:42.547 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:12:42.587 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:42.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:42.588 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:12:42.689 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:12:42.721 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:42.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:42.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:12:42.823 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:12:42.856 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:42.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:42.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:12:42.946 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-17T07:12:42.958 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:12:43.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:43.037 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:43.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:12:43.138 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:12:43.177 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:43.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:43.178 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:12:43.216 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 26 16 57642 57626 138.511 99.6875 0.00874087 0.00721485 2024-09-17T07:12:43.279 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:12:43.317 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:43.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:43.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:12:43.419 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:12:43.456 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:43.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:43.457 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:12:43.558 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:12:43.589 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:43.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:43.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:12:43.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:43 smithi100 ceph-mon[29092]: from='client.25214 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:12:43.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:43 smithi100 ceph-mon[39462]: from='client.25214 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:12:43.691 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:12:43.720 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:43.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:43.721 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:12:43.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:43 smithi135 ceph-mon[33238]: from='client.25214 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:12:43.822 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:12:43.858 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:43.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:43.859 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:12:43.874 INFO:teuthology.orchestra.run.smithi100.stdout:{"pg_ready":true,"pg_map":{"version":471,"stamp":"2024-09-17T07:12:41.290286+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3293714710,"num_objects":50445,"num_object_clones":0,"num_object_copies":151335,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50445,"num_whiteouts":0,"num_read":1964,"num_read_kb":1810,"num_write":51144,"num_write_kb":3218395,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":123,"num_bytes_recovered":1773876,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":51178,"ondisk_log_size":51178,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":12281672,"kb_used_data":9478904,"kb_used_omap":10,"kb_used_meta":2802677,"kb_avail":643934392,"statfs":{"total":671965249536,"available":659388817408,"internally_reserved":0,"allocated":9706397696,"data_stored":9702937707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11152,"internal_metadata":2869941360},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[8,3,11,10,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":586481664,"num_objects":8949,"num_object_clones":0,"num_object_copies":26847,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8949,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":8949,"num_write_kb":572736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.071940"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":76,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.858193+0000","last_change":"2024-09-17T07:11:10.474903+0000","last_active":"2024-09-17T07:12:29.858193+0000","last_peered":"2024-09-17T07:12:29.858193+0000","last_clean":"2024-09-17T07:12:29.858193+0000","last_became_active":"2024-09-17T07:09:24.401344+0000","last_became_peered":"2024-09-17T07:09:24.401344+0000","last_unstale":"2024-09-17T07:12:29.858193+0000","last_undegraded":"2024-09-17T07:12:29.858193+0000","last_fullsized":"2024-09-17T07:12:29.858193+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:10.474778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:33.644728+0000","last_clean_scrub_stamp":"2024-09-17T07:11:10.474778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004995724,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":237,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T07:12:36.837007+0000","last_change":"2024-09-17T07:12:20.616904+0000","last_active":"2024-09-17T07:12:36.837007+0000","last_peered":"2024-09-17T07:12:36.837007+0000","last_clean":"2024-09-17T07:12:36.837007+0000","last_became_active":"2024-09-17T07:02:12.006754+0000","last_became_peered":"2024-09-17T07:02:12.006754+0000","last_unstale":"2024-09-17T07:12:36.837007+0000","last_undegraded":"2024-09-17T07:12:36.837007+0000","last_fullsized":"2024-09-17T07:12:36.837007+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:20.616809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:12:20.616809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:28.462459+0000","scrub_duration":0.0024498060000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"242'27","reported_seq":147,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.863323+0000","last_change":"2024-09-17T07:12:24.639592+0000","last_active":"2024-09-17T07:12:29.863323+0000","last_peered":"2024-09-17T07:12:29.863323+0000","last_clean":"2024-09-17T07:12:29.863323+0000","last_became_active":"2024-09-17T07:09:24.402042+0000","last_became_peered":"2024-09-17T07:09:24.402042+0000","last_unstale":"2024-09-17T07:12:29.863323+0000","last_undegraded":"2024-09-17T07:12:29.863323+0000","last_fullsized":"2024-09-17T07:12:29.863323+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'27","last_scrub_stamp":"2024-09-17T07:12:24.639342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:12:24.639342+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:48.768911+0000","scrub_duration":0.0036508320000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":222,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.865747+0000","last_change":"2024-09-17T07:12:12.612582+0000","last_active":"2024-09-17T07:12:29.865747+0000","last_peered":"2024-09-17T07:12:29.865747+0000","last_clean":"2024-09-17T07:12:29.865747+0000","last_became_active":"2024-09-17T07:02:16.014928+0000","last_became_peered":"2024-09-17T07:02:16.014928+0000","last_unstale":"2024-09-17T07:12:29.865747+0000","last_undegraded":"2024-09-17T07:12:29.865747+0000","last_fullsized":"2024-09-17T07:12:29.865747+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:12.612502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:12:12.612502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:17.607137+0000","scrub_duration":0.0012142710000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":76,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.863618+0000","last_change":"2024-09-17T07:10:26.564948+0000","last_active":"2024-09-17T07:12:29.863618+0000","last_peered":"2024-09-17T07:12:29.863618+0000","last_clean":"2024-09-17T07:12:29.863618+0000","last_became_active":"2024-09-17T07:09:24.402568+0000","last_became_peered":"2024-09-17T07:09:24.402568+0000","last_unstale":"2024-09-17T07:12:29.863618+0000","last_undegraded":"2024-09-17T07:12:29.863618+0000","last_fullsized":"2024-09-17T07:12:29.863618+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:10:26.564872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:10:26.564872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0011084269999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"242'15","reported_seq":260,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.865824+0000","last_change":"2024-09-17T07:12:15.549204+0000","last_active":"2024-09-17T07:12:29.865824+0000","last_peered":"2024-09-17T07:12:29.865824+0000","last_clean":"2024-09-17T07:12:29.865824+0000","last_became_active":"2024-09-17T07:02:14.016244+0000","last_became_peered":"2024-09-17T07:02:14.016244+0000","last_unstale":"2024-09-17T07:12:29.865824+0000","last_undegraded":"2024-09-17T07:12:29.865824+0000","last_fullsized":"2024-09-17T07:12:29.865824+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:12:15.549113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:12:15.549113+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:18.330981+0000","scrub_duration":0.0025070779999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":221,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.862701+0000","last_change":"2024-09-17T07:09:25.123274+0000","last_active":"2024-09-17T07:12:29.862701+0000","last_peered":"2024-09-17T07:12:29.862701+0000","last_clean":"2024-09-17T07:12:29.862701+0000","last_became_active":"2024-09-17T07:02:16.014380+0000","last_became_peered":"2024-09-17T07:02:16.014380+0000","last_unstale":"2024-09-17T07:12:29.862701+0000","last_undegraded":"2024-09-17T07:12:29.862701+0000","last_fullsized":"2024-09-17T07:12:29.862701+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:25.123158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:09:25.123158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00166512,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":232,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.865990+0000","last_change":"2024-09-17T07:12:25.491058+0000","last_active":"2024-09-17T07:12:29.865990+0000","last_peered":"2024-09-17T07:12:29.865990+0000","last_clean":"2024-09-17T07:12:29.865990+0000","last_became_active":"2024-09-17T07:02:18.020790+0000","last_became_peered":"2024-09-17T07:02:18.020790+0000","last_unstale":"2024-09-17T07:12:29.865990+0000","last_undegraded":"2024-09-17T07:12:29.865990+0000","last_fullsized":"2024-09-17T07:12:29.865990+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:25.490932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:12:25.490932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:50.379522+0000","scrub_duration":0.0032974060000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":235,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.862684+0000","last_change":"2024-09-17T07:12:11.365357+0000","last_active":"2024-09-17T07:12:29.862684+0000","last_peered":"2024-09-17T07:12:29.862684+0000","last_clean":"2024-09-17T07:12:29.862684+0000","last_became_active":"2024-09-17T07:02:12.011129+0000","last_became_peered":"2024-09-17T07:02:12.011129+0000","last_unstale":"2024-09-17T07:12:29.862684+0000","last_undegraded":"2024-09-17T07:12:29.862684+0000","last_fullsized":"2024-09-17T07:12:29.862684+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:11.365269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:43.060102+0000","last_clean_scrub_stamp":"2024-09-17T07:12:11.365269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:35.406945+0000","scrub_duration":0.0012842960000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"242'7","reported_seq":253,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.866112+0000","last_change":"2024-09-17T07:11:02.661284+0000","last_active":"2024-09-17T07:12:29.866112+0000","last_peered":"2024-09-17T07:12:29.866112+0000","last_clean":"2024-09-17T07:12:29.866112+0000","last_became_active":"2024-09-17T07:09:24.396900+0000","last_became_peered":"2024-09-17T07:09:24.396900+0000","last_unstale":"2024-09-17T07:12:29.866112+0000","last_undegraded":"2024-09-17T07:12:29.866112+0000","last_fullsized":"2024-09-17T07:12:29.866112+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-09-17T07:11:02.661069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:11:02.661069+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0086442429999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":231,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.859613+0000","last_change":"2024-09-17T07:11:24.287105+0000","last_active":"2024-09-17T07:12:29.859613+0000","last_peered":"2024-09-17T07:12:29.859613+0000","last_clean":"2024-09-17T07:12:29.859613+0000","last_became_active":"2024-09-17T07:02:16.015401+0000","last_became_peered":"2024-09-17T07:02:16.015401+0000","last_unstale":"2024-09-17T07:12:29.859613+0000","last_undegraded":"2024-09-17T07:12:29.859613+0000","last_fullsized":"2024-09-17T07:12:29.859613+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:24.286986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:11:24.286986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:50.052847+0000","scrub_duration":0.0067853130000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":239,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.860462+0000","last_change":"2024-09-17T07:11:03.187420+0000","last_active":"2024-09-17T07:12:29.860462+0000","last_peered":"2024-09-17T07:12:29.860462+0000","last_clean":"2024-09-17T07:12:29.860462+0000","last_became_active":"2024-09-17T07:09:24.402616+0000","last_became_peered":"2024-09-17T07:09:24.402616+0000","last_unstale":"2024-09-17T07:12:29.860462+0000","last_undegraded":"2024-09-17T07:12:29.860462+0000","last_fullsized":"2024-09-17T07:12:29.860462+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:03.187280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:11:03.187280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0050576739999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":235,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.858898+0000","last_change":"2024-09-17T07:12:10.356268+0000","last_active":"2024-09-17T07:12:29.858898+0000","last_peered":"2024-09-17T07:12:29.858898+0000","last_clean":"2024-09-17T07:12:29.858898+0000","last_became_active":"2024-09-17T07:02:12.012234+0000","last_became_peered":"2024-09-17T07:02:12.012234+0000","last_unstale":"2024-09-17T07:12:29.858898+0000","last_undegraded":"2024-09-17T07:12:29.858898+0000","last_fullsized":"2024-09-17T07:12:29.858898+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:10.356011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:45.029247+0000","last_clean_scrub_stamp":"2024-09-17T07:12:10.356011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:12.798127+0000","scrub_duration":0.001613686,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"242'15","reported_seq":268,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.866192+0000","last_change":"2024-09-17T07:11:20.847222+0000","last_active":"2024-09-17T07:12:29.866192+0000","last_peered":"2024-09-17T07:12:29.866192+0000","last_clean":"2024-09-17T07:12:29.866192+0000","last_became_active":"2024-09-17T07:09:24.397954+0000","last_became_peered":"2024-09-17T07:09:24.397954+0000","last_unstale":"2024-09-17T07:12:29.866192+0000","last_undegraded":"2024-09-17T07:12:29.866192+0000","last_fullsized":"2024-09-17T07:12:29.866192+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:11:20.847071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:11:20.847071+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:45.513061+0000","scrub_duration":0.0051365669999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":231,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.862815+0000","last_change":"2024-09-17T07:11:27.107585+0000","last_active":"2024-09-17T07:12:29.862815+0000","last_peered":"2024-09-17T07:12:29.862815+0000","last_clean":"2024-09-17T07:12:29.862815+0000","last_became_active":"2024-09-17T07:02:16.023774+0000","last_became_peered":"2024-09-17T07:02:16.023774+0000","last_unstale":"2024-09-17T07:12:29.862815+0000","last_undegraded":"2024-09-17T07:12:29.862815+0000","last_fullsized":"2024-09-17T07:12:29.862815+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:27.107400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:28.096266+0000","last_clean_scrub_stamp":"2024-09-17T07:11:27.107400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:54.412659+0000","scrub_duration":0.0081054549999999993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":220,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.861643+0000","last_change":"2024-09-17T07:11:06.419278+0000","last_active":"2024-09-17T07:12:29.861643+0000","last_peered":"2024-09-17T07:12:29.861643+0000","last_clean":"2024-09-17T07:12:29.861643+0000","last_became_active":"2024-09-17T07:02:18.029178+0000","last_became_peered":"2024-09-17T07:02:18.029178+0000","last_unstale":"2024-09-17T07:12:29.861643+0000","last_undegraded":"2024-09-17T07:12:29.861643+0000","last_fullsized":"2024-09-17T07:12:29.861643+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:06.419134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:11:06.419134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0086498219999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.0","version":"247'3095","reported_seq":3114,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T07:12:38.565213+0000","last_change":"2024-09-17T07:12:05.418073+0000","last_active":"2024-09-17T07:12:38.565213+0000","last_peered":"2024-09-17T07:12:38.565213+0000","last_clean":"2024-09-17T07:12:38.565213+0000","last_became_active":"2024-09-17T07:12:05.417841+0000","last_became_peered":"2024-09-17T07:12:05.417841+0000","last_unstale":"2024-09-17T07:12:38.565213+0000","last_undegraded":"2024-09-17T07:12:38.565213+0000","last_fullsized":"2024-09-17T07:12:38.565213+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_clean_scrub_stamp":"2024-09-17T07:12:04.285115+0000","objects_scrubbed":0,"log_size":3095,"log_dups_size":0,"ondisk_log_size":3095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:09.864746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":202833920,"num_objects":3095,"num_object_clones":0,"num_object_copies":9285,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3095,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3095,"num_write_kb":198080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"242'39","reported_seq":292,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.866355+0000","last_change":"2024-09-17T07:12:19.487929+0000","last_active":"2024-09-17T07:12:29.866355+0000","last_peered":"2024-09-17T07:12:29.866355+0000","last_clean":"2024-09-17T07:12:29.866355+0000","last_became_active":"2024-09-17T07:02:14.016270+0000","last_became_peered":"2024-09-17T07:02:14.016270+0000","last_unstale":"2024-09-17T07:12:29.866355+0000","last_undegraded":"2024-09-17T07:12:29.866355+0000","last_fullsized":"2024-09-17T07:12:29.866355+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'27","last_scrub_stamp":"2024-09-17T07:12:19.487754+0000","last_deep_scrub":"95'25","last_deep_scrub_stamp":"2024-09-17T07:04:09.450039+0000","last_clean_scrub_stamp":"2024-09-17T07:12:19.487754+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:22.189722+0000","scrub_duration":0.0086853929999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":237,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.859026+0000","last_change":"2024-09-17T07:11:13.480849+0000","last_active":"2024-09-17T07:12:29.859026+0000","last_peered":"2024-09-17T07:12:29.859026+0000","last_clean":"2024-09-17T07:12:29.859026+0000","last_became_active":"2024-09-17T07:02:12.011258+0000","last_became_peered":"2024-09-17T07:02:12.011258+0000","last_unstale":"2024-09-17T07:12:29.859026+0000","last_undegraded":"2024-09-17T07:12:29.859026+0000","last_fullsized":"2024-09-17T07:12:29.859026+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:13.480727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:11:13.480727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0080456680000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":221,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.862192+0000","last_change":"2024-09-17T07:11:00.256708+0000","last_active":"2024-09-17T07:12:29.862192+0000","last_peered":"2024-09-17T07:12:29.862192+0000","last_clean":"2024-09-17T07:12:29.862192+0000","last_became_active":"2024-09-17T07:02:16.018147+0000","last_became_peered":"2024-09-17T07:02:16.018147+0000","last_unstale":"2024-09-17T07:12:29.862192+0000","last_undegraded":"2024-09-17T07:12:29.862192+0000","last_fullsized":"2024-09-17T07:12:29.862192+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:00.256595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:11:00.256595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040770219999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":79,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.866684+0000","last_change":"2024-09-17T07:12:21.520698+0000","last_active":"2024-09-17T07:12:29.866684+0000","last_peered":"2024-09-17T07:12:29.866684+0000","last_clean":"2024-09-17T07:12:29.866684+0000","last_became_active":"2024-09-17T07:09:24.396334+0000","last_became_peered":"2024-09-17T07:09:24.396334+0000","last_unstale":"2024-09-17T07:12:29.866684+0000","last_undegraded":"2024-09-17T07:12:29.866684+0000","last_fullsized":"2024-09-17T07:12:29.866684+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:21.520586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:42.984088+0000","last_clean_scrub_stamp":"2024-09-17T07:12:21.520586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:37.147324+0000","scrub_duration":0.0022027779999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1","version":"247'3018","reported_seq":3038,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T07:12:38.262687+0000","last_change":"2024-09-17T07:12:05.172795+0000","last_active":"2024-09-17T07:12:38.262687+0000","last_peered":"2024-09-17T07:12:38.262687+0000","last_clean":"2024-09-17T07:12:38.262687+0000","last_became_active":"2024-09-17T07:12:05.172332+0000","last_became_peered":"2024-09-17T07:12:05.172332+0000","last_unstale":"2024-09-17T07:12:38.262687+0000","last_undegraded":"2024-09-17T07:12:38.262687+0000","last_fullsized":"2024-09-17T07:12:38.262687+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_clean_scrub_stamp":"2024-09-17T07:12:04.285115+0000","objects_scrubbed":0,"log_size":3018,"log_dups_size":0,"ondisk_log_size":3018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:27.965693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197787648,"num_objects":3018,"num_object_clones":0,"num_object_copies":9054,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3018,"num_write_kb":193152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"242'25","reported_seq":291,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.866658+0000","last_change":"2024-09-17T07:11:24.918471+0000","last_active":"2024-09-17T07:12:29.866658+0000","last_peered":"2024-09-17T07:12:29.866658+0000","last_clean":"2024-09-17T07:12:29.866658+0000","last_became_active":"2024-09-17T07:09:24.399287+0000","last_became_peered":"2024-09-17T07:09:24.399287+0000","last_unstale":"2024-09-17T07:12:29.866658+0000","last_undegraded":"2024-09-17T07:12:29.866658+0000","last_fullsized":"2024-09-17T07:12:29.866658+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'15","last_scrub_stamp":"2024-09-17T07:11:24.918191+0000","last_deep_scrub":"176'15","last_deep_scrub_stamp":"2024-09-17T07:08:19.633791+0000","last_clean_scrub_stamp":"2024-09-17T07:11:24.918191+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:52.956372+0000","scrub_duration":0.0091725629999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":244,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.864302+0000","last_change":"2024-09-17T07:12:09.254527+0000","last_active":"2024-09-17T07:12:29.864302+0000","last_peered":"2024-09-17T07:12:29.864302+0000","last_clean":"2024-09-17T07:12:29.864302+0000","last_became_active":"2024-09-17T07:09:24.406973+0000","last_became_peered":"2024-09-17T07:09:24.406973+0000","last_unstale":"2024-09-17T07:12:29.864302+0000","last_undegraded":"2024-09-17T07:12:29.864302+0000","last_fullsized":"2024-09-17T07:12:29.864302+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:09.254452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:12:09.254452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:36.038881+0000","scrub_duration":0.0013322150000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"98'9","reported_seq":316,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:35.649094+0000","last_change":"2024-09-17T07:12:00.407366+0000","last_active":"2024-09-17T07:12:35.649094+0000","last_peered":"2024-09-17T07:12:35.649094+0000","last_clean":"2024-09-17T07:12:35.649094+0000","last_became_active":"2024-09-17T07:04:15.110797+0000","last_became_peered":"2024-09-17T07:04:15.110797+0000","last_unstale":"2024-09-17T07:12:35.649094+0000","last_undegraded":"2024-09-17T07:12:35.649094+0000","last_fullsized":"2024-09-17T07:12:35.649094+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-09-17T07:12:00.407262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:12:00.407262+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:14.851416+0000","scrub_duration":0.002024489,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":231,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.867887+0000","last_change":"2024-09-17T07:11:29.774296+0000","last_active":"2024-09-17T07:12:29.867887+0000","last_peered":"2024-09-17T07:12:29.867887+0000","last_clean":"2024-09-17T07:12:29.867887+0000","last_became_active":"2024-09-17T07:02:18.017460+0000","last_became_peered":"2024-09-17T07:02:18.017460+0000","last_unstale":"2024-09-17T07:12:29.867887+0000","last_undegraded":"2024-09-17T07:12:29.867887+0000","last_fullsized":"2024-09-17T07:12:29.867887+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:29.774144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:11:29.774144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.013832928,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"247'3251","reported_seq":3270,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T07:12:38.560843+0000","last_change":"2024-09-17T07:12:05.416444+0000","last_active":"2024-09-17T07:12:38.560843+0000","last_peered":"2024-09-17T07:12:38.560843+0000","last_clean":"2024-09-17T07:12:38.560843+0000","last_became_active":"2024-09-17T07:12:05.416186+0000","last_became_peered":"2024-09-17T07:12:05.416186+0000","last_unstale":"2024-09-17T07:12:38.560843+0000","last_undegraded":"2024-09-17T07:12:38.560843+0000","last_fullsized":"2024-09-17T07:12:38.560843+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_clean_scrub_stamp":"2024-09-17T07:12:04.285115+0000","objects_scrubbed":0,"log_size":3251,"log_dups_size":0,"ondisk_log_size":3251,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:25.741377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213057536,"num_objects":3251,"num_object_clones":0,"num_object_copies":9753,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3251,"num_write_kb":208064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"242'27","reported_seq":301,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.864792+0000","last_change":"2024-09-17T07:12:13.306460+0000","last_active":"2024-09-17T07:12:29.864792+0000","last_peered":"2024-09-17T07:12:29.864792+0000","last_clean":"2024-09-17T07:12:29.864792+0000","last_became_active":"2024-09-17T07:09:24.407311+0000","last_became_peered":"2024-09-17T07:09:24.407311+0000","last_unstale":"2024-09-17T07:12:29.864792+0000","last_undegraded":"2024-09-17T07:12:29.864792+0000","last_fullsized":"2024-09-17T07:12:29.864792+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-09-17T07:12:13.306349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:12:13.306349+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:26.012489+0000","scrub_duration":0.0021853620000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":234,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.862740+0000","last_change":"2024-09-17T07:09:24.405992+0000","last_active":"2024-09-17T07:12:29.862740+0000","last_peered":"2024-09-17T07:12:29.862740+0000","last_clean":"2024-09-17T07:12:29.862740+0000","last_became_active":"2024-09-17T07:09:24.405655+0000","last_became_peered":"2024-09-17T07:09:24.405655+0000","last_unstale":"2024-09-17T07:12:29.862740+0000","last_undegraded":"2024-09-17T07:12:29.862740+0000","last_fullsized":"2024-09-17T07:12:29.862740+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:20.182256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:09:20.182256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034116369999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"75'8","reported_seq":355,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:35.649746+0000","last_change":"2024-09-17T07:11:25.896643+0000","last_active":"2024-09-17T07:12:35.649746+0000","last_peered":"2024-09-17T07:12:35.649746+0000","last_clean":"2024-09-17T07:12:35.649746+0000","last_became_active":"2024-09-17T07:02:16.021762+0000","last_became_peered":"2024-09-17T07:02:16.021762+0000","last_unstale":"2024-09-17T07:12:35.649746+0000","last_undegraded":"2024-09-17T07:12:35.649746+0000","last_fullsized":"2024-09-17T07:12:35.649746+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:11:25.896517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:11:25.896517+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:53.871973+0000","scrub_duration":0.01413413,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":229,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.861030+0000","last_change":"2024-09-17T07:11:30.065694+0000","last_active":"2024-09-17T07:12:29.861030+0000","last_peered":"2024-09-17T07:12:29.861030+0000","last_clean":"2024-09-17T07:12:29.861030+0000","last_became_active":"2024-09-17T07:02:18.025332+0000","last_became_peered":"2024-09-17T07:02:18.025332+0000","last_unstale":"2024-09-17T07:12:29.861030+0000","last_undegraded":"2024-09-17T07:12:29.861030+0000","last_fullsized":"2024-09-17T07:12:29.861030+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:30.065556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:11:30.065556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:38.369770+0000","scrub_duration":0.0046561759999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"247'3118","reported_seq":3137,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T07:12:38.378086+0000","last_change":"2024-09-17T07:12:05.174275+0000","last_active":"2024-09-17T07:12:38.378086+0000","last_peered":"2024-09-17T07:12:38.378086+0000","last_clean":"2024-09-17T07:12:38.378086+0000","last_became_active":"2024-09-17T07:12:05.174017+0000","last_became_peered":"2024-09-17T07:12:05.174017+0000","last_unstale":"2024-09-17T07:12:38.378086+0000","last_undegraded":"2024-09-17T07:12:38.378086+0000","last_fullsized":"2024-09-17T07:12:38.378086+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_clean_scrub_stamp":"2024-09-17T07:12:04.285115+0000","objects_scrubbed":0,"log_size":3118,"log_dups_size":0,"ondisk_log_size":3118,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:30.641723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":204341248,"num_objects":3118,"num_object_clones":0,"num_object_copies":9354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3118,"num_write_kb":199552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"242'19","reported_seq":266,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.868045+0000","last_change":"2024-09-17T07:11:27.816034+0000","last_active":"2024-09-17T07:12:29.868045+0000","last_peered":"2024-09-17T07:12:29.868045+0000","last_clean":"2024-09-17T07:12:29.868045+0000","last_became_active":"2024-09-17T07:02:14.010227+0000","last_became_peered":"2024-09-17T07:02:14.010227+0000","last_unstale":"2024-09-17T07:12:29.868045+0000","last_undegraded":"2024-09-17T07:12:29.868045+0000","last_fullsized":"2024-09-17T07:12:29.868045+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-09-17T07:11:27.815628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:11:27.815628+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0098845579999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":236,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.858245+0000","last_change":"2024-09-17T07:11:16.421038+0000","last_active":"2024-09-17T07:12:29.858245+0000","last_peered":"2024-09-17T07:12:29.858245+0000","last_clean":"2024-09-17T07:12:29.858245+0000","last_became_active":"2024-09-17T07:02:12.013182+0000","last_became_peered":"2024-09-17T07:02:12.013182+0000","last_unstale":"2024-09-17T07:12:29.858245+0000","last_undegraded":"2024-09-17T07:12:29.858245+0000","last_fullsized":"2024-09-17T07:12:29.858245+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:16.420367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:11:16.420367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0051482619999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":242,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.863034+0000","last_change":"2024-09-17T07:12:24.732629+0000","last_active":"2024-09-17T07:12:29.863034+0000","last_peered":"2024-09-17T07:12:29.863034+0000","last_clean":"2024-09-17T07:12:29.863034+0000","last_became_active":"2024-09-17T07:02:16.020224+0000","last_became_peered":"2024-09-17T07:02:16.020224+0000","last_unstale":"2024-09-17T07:12:29.863034+0000","last_undegraded":"2024-09-17T07:12:29.863034+0000","last_fullsized":"2024-09-17T07:12:29.863034+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:24.732549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:12:24.732549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:26.236089+0000","scrub_duration":0.0013739589999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":86,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.870415+0000","last_change":"2024-09-17T07:12:12.704448+0000","last_active":"2024-09-17T07:12:29.870415+0000","last_peered":"2024-09-17T07:12:29.870415+0000","last_clean":"2024-09-17T07:12:29.870415+0000","last_became_active":"2024-09-17T07:09:24.408109+0000","last_became_peered":"2024-09-17T07:09:24.408109+0000","last_unstale":"2024-09-17T07:12:29.870415+0000","last_undegraded":"2024-09-17T07:12:29.870415+0000","last_fullsized":"2024-09-17T07:12:29.870415+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:12.704363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:12:12.704363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:17.195681+0000","scrub_duration":0.0011153510000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"247'3194","reported_seq":3214,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T07:12:38.232934+0000","last_change":"2024-09-17T07:12:05.174268+0000","last_active":"2024-09-17T07:12:38.232934+0000","last_peered":"2024-09-17T07:12:38.232934+0000","last_clean":"2024-09-17T07:12:38.232934+0000","last_became_active":"2024-09-17T07:12:05.173802+0000","last_became_peered":"2024-09-17T07:12:05.173802+0000","last_unstale":"2024-09-17T07:12:38.232934+0000","last_undegraded":"2024-09-17T07:12:38.232934+0000","last_fullsized":"2024-09-17T07:12:38.232934+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_clean_scrub_stamp":"2024-09-17T07:12:04.285115+0000","objects_scrubbed":0,"log_size":3194,"log_dups_size":0,"ondisk_log_size":3194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:14.585969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209321984,"num_objects":3194,"num_object_clones":0,"num_object_copies":9582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3194,"num_write_kb":204416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"242'28","reported_seq":267,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T07:12:36.848963+0000","last_change":"2024-09-17T07:12:00.638759+0000","last_active":"2024-09-17T07:12:36.848963+0000","last_peered":"2024-09-17T07:12:36.848963+0000","last_clean":"2024-09-17T07:12:36.848963+0000","last_became_active":"2024-09-17T07:02:14.007810+0000","last_became_peered":"2024-09-17T07:02:14.007810+0000","last_unstale":"2024-09-17T07:12:36.848963+0000","last_undegraded":"2024-09-17T07:12:36.848963+0000","last_fullsized":"2024-09-17T07:12:36.848963+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'14","last_scrub_stamp":"2024-09-17T07:12:00.638644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:12:00.638644+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:20.604094+0000","scrub_duration":0.002384817,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":257,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.870815+0000","last_change":"2024-09-17T07:12:02.695870+0000","last_active":"2024-09-17T07:12:29.870815+0000","last_peered":"2024-09-17T07:12:29.870815+0000","last_clean":"2024-09-17T07:12:29.870815+0000","last_became_active":"2024-09-17T07:02:12.011766+0000","last_became_peered":"2024-09-17T07:02:12.011766+0000","last_unstale":"2024-09-17T07:12:29.870815+0000","last_undegraded":"2024-09-17T07:12:29.870815+0000","last_fullsized":"2024-09-17T07:12:29.870815+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:02.695765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:09:23.181028+0000","last_clean_scrub_stamp":"2024-09-17T07:12:02.695765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:04.960155+0000","scrub_duration":0.0016435110000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"196'9","reported_seq":127,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:35.648772+0000","last_change":"2024-09-17T07:09:24.427192+0000","last_active":"2024-09-17T07:12:35.648772+0000","last_peered":"2024-09-17T07:12:35.648772+0000","last_clean":"2024-09-17T07:12:35.648772+0000","last_became_active":"2024-09-17T07:09:24.406056+0000","last_became_peered":"2024-09-17T07:09:24.406056+0000","last_unstale":"2024-09-17T07:12:35.648772+0000","last_undegraded":"2024-09-17T07:12:35.648772+0000","last_fullsized":"2024-09-17T07:12:35.648772+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:09:14.776794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:09:14.776794+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":240,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.858313+0000","last_change":"2024-09-17T07:12:25.194692+0000","last_active":"2024-09-17T07:12:29.858313+0000","last_peered":"2024-09-17T07:12:29.858313+0000","last_clean":"2024-09-17T07:12:29.858313+0000","last_became_active":"2024-09-17T07:02:18.364981+0000","last_became_peered":"2024-09-17T07:02:18.364981+0000","last_unstale":"2024-09-17T07:12:29.858313+0000","last_undegraded":"2024-09-17T07:12:29.858313+0000","last_fullsized":"2024-09-17T07:12:29.858313+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:25.194515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:10:33.727472+0000","last_clean_scrub_stamp":"2024-09-17T07:12:25.194515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:40.081616+0000","scrub_duration":0.002844943,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.d","version":"247'3120","reported_seq":3139,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T07:12:38.359510+0000","last_change":"2024-09-17T07:12:05.177946+0000","last_active":"2024-09-17T07:12:38.359510+0000","last_peered":"2024-09-17T07:12:38.359510+0000","last_clean":"2024-09-17T07:12:38.359510+0000","last_became_active":"2024-09-17T07:12:05.177682+0000","last_became_peered":"2024-09-17T07:12:05.177682+0000","last_unstale":"2024-09-17T07:12:38.359510+0000","last_undegraded":"2024-09-17T07:12:38.359510+0000","last_fullsized":"2024-09-17T07:12:38.359510+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_clean_scrub_stamp":"2024-09-17T07:12:04.285115+0000","objects_scrubbed":0,"log_size":3120,"log_dups_size":0,"ondisk_log_size":3120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:19.366886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":204472320,"num_objects":3120,"num_object_clones":0,"num_object_copies":9360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3120,"num_write_kb":199680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"242'30","reported_seq":155,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.858018+0000","last_change":"2024-09-17T07:11:19.363446+0000","last_active":"2024-09-17T07:12:29.858018+0000","last_peered":"2024-09-17T07:12:29.858018+0000","last_clean":"2024-09-17T07:12:29.858018+0000","last_became_active":"2024-09-17T07:09:24.403240+0000","last_became_peered":"2024-09-17T07:09:24.403240+0000","last_unstale":"2024-09-17T07:12:29.858018+0000","last_undegraded":"2024-09-17T07:12:29.858018+0000","last_fullsized":"2024-09-17T07:12:29.858018+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'18","last_scrub_stamp":"2024-09-17T07:11:19.363076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:11:19.363076+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010644855,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":235,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.862819+0000","last_change":"2024-09-17T07:11:07.244633+0000","last_active":"2024-09-17T07:12:29.862819+0000","last_peered":"2024-09-17T07:12:29.862819+0000","last_clean":"2024-09-17T07:12:29.862819+0000","last_became_active":"2024-09-17T07:02:12.005687+0000","last_became_peered":"2024-09-17T07:02:12.005687+0000","last_unstale":"2024-09-17T07:12:29.862819+0000","last_undegraded":"2024-09-17T07:12:29.862819+0000","last_fullsized":"2024-09-17T07:12:29.862819+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:07.244521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:11:07.244521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0062381399999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":232,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T07:12:36.848771+0000","last_change":"2024-09-17T07:11:26.837485+0000","last_active":"2024-09-17T07:12:36.848771+0000","last_peered":"2024-09-17T07:12:36.848771+0000","last_clean":"2024-09-17T07:12:36.848771+0000","last_became_active":"2024-09-17T07:09:24.393375+0000","last_became_peered":"2024-09-17T07:09:24.393375+0000","last_unstale":"2024-09-17T07:12:36.848771+0000","last_undegraded":"2024-09-17T07:12:36.848771+0000","last_fullsized":"2024-09-17T07:12:36.848771+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:26.837370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:11:26.837370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0042759720000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":240,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.866170+0000","last_change":"2024-09-17T07:12:18.466015+0000","last_active":"2024-09-17T07:12:29.866170+0000","last_peered":"2024-09-17T07:12:29.866170+0000","last_clean":"2024-09-17T07:12:29.866170+0000","last_became_active":"2024-09-17T07:09:24.398659+0000","last_became_peered":"2024-09-17T07:09:24.398659+0000","last_unstale":"2024-09-17T07:12:29.866170+0000","last_undegraded":"2024-09-17T07:12:29.866170+0000","last_fullsized":"2024-09-17T07:12:29.866170+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:18.465901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:12:18.465901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:47.260148+0000","scrub_duration":0.0024831419999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.e","version":"247'3061","reported_seq":3079,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T07:12:38.358323+0000","last_change":"2024-09-17T07:12:05.175609+0000","last_active":"2024-09-17T07:12:38.358323+0000","last_peered":"2024-09-17T07:12:38.358323+0000","last_clean":"2024-09-17T07:12:38.358323+0000","last_became_active":"2024-09-17T07:12:05.175318+0000","last_became_peered":"2024-09-17T07:12:05.175318+0000","last_unstale":"2024-09-17T07:12:38.358323+0000","last_undegraded":"2024-09-17T07:12:38.358323+0000","last_fullsized":"2024-09-17T07:12:38.358323+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_clean_scrub_stamp":"2024-09-17T07:12:04.285115+0000","objects_scrubbed":0,"log_size":3061,"log_dups_size":0,"ondisk_log_size":3061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:21.393438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":200605696,"num_objects":3061,"num_object_clones":0,"num_object_copies":9183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3061,"num_write_kb":195904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"242'29","reported_seq":276,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T07:12:36.837882+0000","last_change":"2024-09-17T07:11:03.438378+0000","last_active":"2024-09-17T07:12:36.837882+0000","last_peered":"2024-09-17T07:12:36.837882+0000","last_clean":"2024-09-17T07:12:36.837882+0000","last_became_active":"2024-09-17T07:02:14.027389+0000","last_became_peered":"2024-09-17T07:02:14.027389+0000","last_unstale":"2024-09-17T07:12:36.837882+0000","last_undegraded":"2024-09-17T07:12:36.837882+0000","last_fullsized":"2024-09-17T07:12:36.837882+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-09-17T07:11:03.438281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:11:03.438281+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.006304324,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":236,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.862759+0000","last_change":"2024-09-17T07:11:15.424520+0000","last_active":"2024-09-17T07:12:29.862759+0000","last_peered":"2024-09-17T07:12:29.862759+0000","last_clean":"2024-09-17T07:12:29.862759+0000","last_became_active":"2024-09-17T07:02:12.011917+0000","last_became_peered":"2024-09-17T07:02:12.011917+0000","last_unstale":"2024-09-17T07:12:29.862759+0000","last_undegraded":"2024-09-17T07:12:29.862759+0000","last_fullsized":"2024-09-17T07:12:29.862759+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:15.424430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:24.072799+0000","last_clean_scrub_stamp":"2024-09-17T07:11:15.424430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0057333330000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":232,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.870041+0000","last_change":"2024-09-17T07:11:17.583846+0000","last_active":"2024-09-17T07:12:29.870041+0000","last_peered":"2024-09-17T07:12:29.870041+0000","last_clean":"2024-09-17T07:12:29.870041+0000","last_became_active":"2024-09-17T07:02:16.016199+0000","last_became_peered":"2024-09-17T07:02:16.016199+0000","last_unstale":"2024-09-17T07:12:29.870041+0000","last_undegraded":"2024-09-17T07:12:29.870041+0000","last_fullsized":"2024-09-17T07:12:29.870041+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:17.583712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:11:17.583712+0000","last_clean_scrub_stamp":"2024-09-17T07:11:17.583712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0064424299999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":229,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.862218+0000","last_change":"2024-09-17T07:11:28.050865+0000","last_active":"2024-09-17T07:12:29.862218+0000","last_peered":"2024-09-17T07:12:29.862218+0000","last_clean":"2024-09-17T07:12:29.862218+0000","last_became_active":"2024-09-17T07:02:18.364835+0000","last_became_peered":"2024-09-17T07:02:18.364835+0000","last_unstale":"2024-09-17T07:12:29.862218+0000","last_undegraded":"2024-09-17T07:12:29.862218+0000","last_fullsized":"2024-09-17T07:12:29.862218+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:28.050750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:11:28.050750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:47.779865+0000","scrub_duration":0.003023796,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.f","version":"247'3190","reported_seq":3209,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T07:12:38.312488+0000","last_change":"2024-09-17T07:12:05.418098+0000","last_active":"2024-09-17T07:12:38.312488+0000","last_peered":"2024-09-17T07:12:38.312488+0000","last_clean":"2024-09-17T07:12:38.312488+0000","last_became_active":"2024-09-17T07:12:05.417843+0000","last_became_peered":"2024-09-17T07:12:05.417843+0000","last_unstale":"2024-09-17T07:12:38.312488+0000","last_undegraded":"2024-09-17T07:12:38.312488+0000","last_fullsized":"2024-09-17T07:12:38.312488+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_clean_scrub_stamp":"2024-09-17T07:12:04.285115+0000","objects_scrubbed":0,"log_size":3190,"log_dups_size":0,"ondisk_log_size":3190,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:32.459979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209059840,"num_objects":3190,"num_object_clones":0,"num_object_copies":9570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3190,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3190,"num_write_kb":204160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"242'25","reported_seq":269,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T07:12:36.843371+0000","last_change":"2024-09-17T07:12:19.410879+0000","last_active":"2024-09-17T07:12:36.843371+0000","last_peered":"2024-09-17T07:12:36.843371+0000","last_clean":"2024-09-17T07:12:36.843371+0000","last_became_active":"2024-09-17T07:02:14.010637+0000","last_became_peered":"2024-09-17T07:02:14.010637+0000","last_unstale":"2024-09-17T07:12:36.843371+0000","last_undegraded":"2024-09-17T07:12:36.843371+0000","last_fullsized":"2024-09-17T07:12:36.843371+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'17","last_scrub_stamp":"2024-09-17T07:12:19.410758+0000","last_deep_scrub":"110'15","last_deep_scrub_stamp":"2024-09-17T07:04:57.091400+0000","last_clean_scrub_stamp":"2024-09-17T07:12:19.410758+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:29.125099+0000","scrub_duration":0.0070558269999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":236,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T07:12:36.843350+0000","last_change":"2024-09-17T07:12:04.314016+0000","last_active":"2024-09-17T07:12:36.843350+0000","last_peered":"2024-09-17T07:12:36.843350+0000","last_clean":"2024-09-17T07:12:36.843350+0000","last_became_active":"2024-09-17T07:02:12.010803+0000","last_became_peered":"2024-09-17T07:02:12.010803+0000","last_unstale":"2024-09-17T07:12:36.843350+0000","last_undegraded":"2024-09-17T07:12:36.843350+0000","last_fullsized":"2024-09-17T07:12:36.843350+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:04.313886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:25.147878+0000","last_clean_scrub_stamp":"2024-09-17T07:12:04.313886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:27.666148+0000","scrub_duration":0.0018928020000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"89'9","reported_seq":366,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:35.648947+0000","last_change":"2024-09-17T07:12:03.588884+0000","last_active":"2024-09-17T07:12:35.648947+0000","last_peered":"2024-09-17T07:12:35.648947+0000","last_clean":"2024-09-17T07:12:35.648947+0000","last_became_active":"2024-09-17T07:02:16.021360+0000","last_became_peered":"2024-09-17T07:02:16.021360+0000","last_unstale":"2024-09-17T07:12:35.648947+0000","last_undegraded":"2024-09-17T07:12:35.648947+0000","last_fullsized":"2024-09-17T07:12:35.648947+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'9","last_scrub_stamp":"2024-09-17T07:12:03.588797+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-09-17T07:03:29.961439+0000","last_clean_scrub_stamp":"2024-09-17T07:12:03.588797+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:21.994712+0000","scrub_duration":0.0018430440000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"75'1","reported_seq":245,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.861572+0000","last_change":"2024-09-17T07:12:22.641805+0000","last_active":"2024-09-17T07:12:29.861572+0000","last_peered":"2024-09-17T07:12:29.861572+0000","last_clean":"2024-09-17T07:12:29.861572+0000","last_became_active":"2024-09-17T07:09:24.399565+0000","last_became_peered":"2024-09-17T07:09:24.399565+0000","last_unstale":"2024-09-17T07:12:29.861572+0000","last_undegraded":"2024-09-17T07:12:29.861572+0000","last_fullsized":"2024-09-17T07:12:29.861572+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-17T07:12:22.641641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:12:22.641641+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:32.278016+0000","scrub_duration":0.0043476369999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.9","version":"247'3220","reported_seq":3239,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T07:12:39.240249+0000","last_change":"2024-09-17T07:12:05.175204+0000","last_active":"2024-09-17T07:12:39.240249+0000","last_peered":"2024-09-17T07:12:39.240249+0000","last_clean":"2024-09-17T07:12:39.240249+0000","last_became_active":"2024-09-17T07:12:05.174893+0000","last_became_peered":"2024-09-17T07:12:05.174893+0000","last_unstale":"2024-09-17T07:12:39.240249+0000","last_undegraded":"2024-09-17T07:12:39.240249+0000","last_fullsized":"2024-09-17T07:12:39.240249+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_clean_scrub_stamp":"2024-09-17T07:12:04.285115+0000","objects_scrubbed":0,"log_size":3220,"log_dups_size":0,"ondisk_log_size":3220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:22.619349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":211025920,"num_objects":3220,"num_object_clones":0,"num_object_copies":9660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3220,"num_write_kb":206080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"242'41","reported_seq":456,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T07:12:36.843283+0000","last_change":"2024-09-17T07:12:08.264127+0000","last_active":"2024-09-17T07:12:36.843283+0000","last_peered":"2024-09-17T07:12:36.843283+0000","last_clean":"2024-09-17T07:12:36.843283+0000","last_became_active":"2024-09-17T07:09:24.403267+0000","last_became_peered":"2024-09-17T07:09:24.403267+0000","last_unstale":"2024-09-17T07:12:36.843283+0000","last_undegraded":"2024-09-17T07:12:36.843283+0000","last_fullsized":"2024-09-17T07:12:36.843283+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'29","last_scrub_stamp":"2024-09-17T07:12:08.264016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:12:08.264016+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:30.766851+0000","scrub_duration":0.0026058610000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":247,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T07:12:36.837208+0000","last_change":"2024-09-17T07:12:19.678216+0000","last_active":"2024-09-17T07:12:36.837208+0000","last_peered":"2024-09-17T07:12:36.837208+0000","last_clean":"2024-09-17T07:12:36.837208+0000","last_became_active":"2024-09-17T07:02:12.008802+0000","last_became_peered":"2024-09-17T07:02:12.008802+0000","last_unstale":"2024-09-17T07:12:36.837208+0000","last_undegraded":"2024-09-17T07:12:36.837208+0000","last_fullsized":"2024-09-17T07:12:36.837208+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:19.678145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:12:19.678145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:27.258606+0000","scrub_duration":0.0013514519999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":232,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.863154+0000","last_change":"2024-09-17T07:11:07.433543+0000","last_active":"2024-09-17T07:12:29.863154+0000","last_peered":"2024-09-17T07:12:29.863154+0000","last_clean":"2024-09-17T07:12:29.863154+0000","last_became_active":"2024-09-17T07:02:16.016606+0000","last_became_peered":"2024-09-17T07:02:16.016606+0000","last_unstale":"2024-09-17T07:12:29.863154+0000","last_undegraded":"2024-09-17T07:12:29.863154+0000","last_fullsized":"2024-09-17T07:12:29.863154+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:07.433420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:11:07.433420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012900124000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":76,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.867888+0000","last_change":"2024-09-17T07:12:03.666609+0000","last_active":"2024-09-17T07:12:29.867888+0000","last_peered":"2024-09-17T07:12:29.867888+0000","last_clean":"2024-09-17T07:12:29.867888+0000","last_became_active":"2024-09-17T07:09:24.398878+0000","last_became_peered":"2024-09-17T07:09:24.398878+0000","last_unstale":"2024-09-17T07:12:29.867888+0000","last_undegraded":"2024-09-17T07:12:29.867888+0000","last_fullsized":"2024-09-17T07:12:29.867888+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:03.666518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:12:03.666518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:28.852645+0000","scrub_duration":0.0015175169999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.a","version":"247'3069","reported_seq":3087,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T07:12:38.322859+0000","last_change":"2024-09-17T07:12:05.175424+0000","last_active":"2024-09-17T07:12:38.322859+0000","last_peered":"2024-09-17T07:12:38.322859+0000","last_clean":"2024-09-17T07:12:38.322859+0000","last_became_active":"2024-09-17T07:12:05.175109+0000","last_became_peered":"2024-09-17T07:12:05.175109+0000","last_unstale":"2024-09-17T07:12:38.322859+0000","last_undegraded":"2024-09-17T07:12:38.322859+0000","last_fullsized":"2024-09-17T07:12:38.322859+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_clean_scrub_stamp":"2024-09-17T07:12:04.285115+0000","objects_scrubbed":0,"log_size":3069,"log_dups_size":0,"ondisk_log_size":3069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:23.729980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201129984,"num_objects":3069,"num_object_clones":0,"num_object_copies":9207,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3069,"num_write_kb":196416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"242'17","reported_seq":266,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.864812+0000","last_change":"2024-09-17T07:11:17.112766+0000","last_active":"2024-09-17T07:12:29.864812+0000","last_peered":"2024-09-17T07:12:29.864812+0000","last_clean":"2024-09-17T07:12:29.864812+0000","last_became_active":"2024-09-17T07:02:14.019838+0000","last_became_peered":"2024-09-17T07:02:14.019838+0000","last_unstale":"2024-09-17T07:12:29.864812+0000","last_undegraded":"2024-09-17T07:12:29.864812+0000","last_fullsized":"2024-09-17T07:12:29.864812+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'13","last_scrub_stamp":"2024-09-17T07:11:17.112635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:11:17.112635+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.005881725,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"76'5","reported_seq":863,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T07:12:37.416801+0000","last_change":"2024-09-17T07:12:22.408620+0000","last_active":"2024-09-17T07:12:37.416801+0000","last_peered":"2024-09-17T07:12:37.416801+0000","last_clean":"2024-09-17T07:12:37.416801+0000","last_became_active":"2024-09-17T07:02:12.006883+0000","last_became_peered":"2024-09-17T07:02:12.006883+0000","last_unstale":"2024-09-17T07:12:37.416801+0000","last_undegraded":"2024-09-17T07:12:37.416801+0000","last_fullsized":"2024-09-17T07:12:37.416801+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'5","last_scrub_stamp":"2024-09-17T07:12:22.408498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:12:22.408498+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:38.147275+0000","scrub_duration":0.0032583909999999998,"objects_trimmed":0,"snaptrim_duration":0.00033739799999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":619,"num_read_kb":614,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":77,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.863400+0000","last_change":"2024-09-17T07:12:13.536969+0000","last_active":"2024-09-17T07:12:29.863400+0000","last_peered":"2024-09-17T07:12:29.863400+0000","last_clean":"2024-09-17T07:12:29.863400+0000","last_became_active":"2024-09-17T07:09:24.392850+0000","last_became_peered":"2024-09-17T07:09:24.392850+0000","last_unstale":"2024-09-17T07:12:29.863400+0000","last_undegraded":"2024-09-17T07:12:29.863400+0000","last_fullsized":"2024-09-17T07:12:29.863400+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:13.536888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:23.833170+0000","last_clean_scrub_stamp":"2024-09-17T07:12:13.536888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:25.415883+0000","scrub_duration":0.001259055,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":244,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.859113+0000","last_change":"2024-09-17T07:12:06.049472+0000","last_active":"2024-09-17T07:12:29.859113+0000","last_peered":"2024-09-17T07:12:29.859113+0000","last_clean":"2024-09-17T07:12:29.859113+0000","last_became_active":"2024-09-17T07:02:16.014492+0000","last_became_peered":"2024-09-17T07:02:16.014492+0000","last_unstale":"2024-09-17T07:12:29.859113+0000","last_undegraded":"2024-09-17T07:12:29.859113+0000","last_fullsized":"2024-09-17T07:12:29.859113+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:06.049391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:12:06.049391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:34.526195+0000","scrub_duration":0.001322546,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":249,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.860788+0000","last_change":"2024-09-17T07:12:12.332042+0000","last_active":"2024-09-17T07:12:29.860788+0000","last_peered":"2024-09-17T07:12:29.860788+0000","last_clean":"2024-09-17T07:12:29.860788+0000","last_became_active":"2024-09-17T07:09:24.402704+0000","last_became_peered":"2024-09-17T07:09:24.402704+0000","last_unstale":"2024-09-17T07:12:29.860788+0000","last_undegraded":"2024-09-17T07:12:29.860788+0000","last_fullsized":"2024-09-17T07:12:29.860788+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:12.331949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:12:12.331949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:30.558308+0000","scrub_duration":0.0012857490000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"247'3234","reported_seq":3253,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T07:12:39.243010+0000","last_change":"2024-09-17T07:12:05.173856+0000","last_active":"2024-09-17T07:12:39.243010+0000","last_peered":"2024-09-17T07:12:39.243010+0000","last_clean":"2024-09-17T07:12:39.243010+0000","last_became_active":"2024-09-17T07:12:05.173534+0000","last_became_peered":"2024-09-17T07:12:05.173534+0000","last_unstale":"2024-09-17T07:12:39.243010+0000","last_undegraded":"2024-09-17T07:12:39.243010+0000","last_fullsized":"2024-09-17T07:12:39.243010+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_clean_scrub_stamp":"2024-09-17T07:12:04.285115+0000","objects_scrubbed":0,"log_size":3234,"log_dups_size":0,"ondisk_log_size":3234,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:29.636041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":211943424,"num_objects":3234,"num_object_clones":0,"num_object_copies":9702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3234,"num_write_kb":206976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"242'24","reported_seq":293,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.863547+0000","last_change":"2024-09-17T07:12:11.578753+0000","last_active":"2024-09-17T07:12:29.863547+0000","last_peered":"2024-09-17T07:12:29.863547+0000","last_clean":"2024-09-17T07:12:29.863547+0000","last_became_active":"2024-09-17T07:03:48.760405+0000","last_became_peered":"2024-09-17T07:03:48.760405+0000","last_unstale":"2024-09-17T07:12:29.863547+0000","last_undegraded":"2024-09-17T07:12:29.863547+0000","last_fullsized":"2024-09-17T07:12:29.863547+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-09-17T07:12:11.578647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:12:11.578647+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:30.740260+0000","scrub_duration":0.0020729070000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"70'1","reported_seq":245,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T07:12:36.838494+0000","last_change":"2024-09-17T07:09:24.415023+0000","last_active":"2024-09-17T07:12:36.838494+0000","last_peered":"2024-09-17T07:12:36.838494+0000","last_clean":"2024-09-17T07:12:36.838494+0000","last_became_active":"2024-09-17T07:09:24.402499+0000","last_became_peered":"2024-09-17T07:09:24.402499+0000","last_unstale":"2024-09-17T07:12:36.838494+0000","last_undegraded":"2024-09-17T07:12:36.838494+0000","last_fullsized":"2024-09-17T07:12:36.838494+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-17T07:09:22.249833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:09:22.249833+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0061513059999999996,"objects_trimmed":0,"snaptrim_duration":0.0032298629999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":236,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.858327+0000","last_change":"2024-09-17T07:12:16.273630+0000","last_active":"2024-09-17T07:12:29.858327+0000","last_peered":"2024-09-17T07:12:29.858327+0000","last_clean":"2024-09-17T07:12:29.858327+0000","last_became_active":"2024-09-17T07:02:12.014623+0000","last_became_peered":"2024-09-17T07:02:12.014623+0000","last_unstale":"2024-09-17T07:12:29.858327+0000","last_undegraded":"2024-09-17T07:12:29.858327+0000","last_fullsized":"2024-09-17T07:12:29.858327+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:16.273537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:12:16.273537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:41.553719+0000","scrub_duration":0.001748185,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":231,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.863086+0000","last_change":"2024-09-17T07:12:08.807635+0000","last_active":"2024-09-17T07:12:29.863086+0000","last_peered":"2024-09-17T07:12:29.863086+0000","last_clean":"2024-09-17T07:12:29.863086+0000","last_became_active":"2024-09-17T07:02:16.021561+0000","last_became_peered":"2024-09-17T07:02:16.021561+0000","last_unstale":"2024-09-17T07:12:29.863086+0000","last_undegraded":"2024-09-17T07:12:29.863086+0000","last_fullsized":"2024-09-17T07:12:29.863086+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:08.807531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:12:08.807531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:24.781268+0000","scrub_duration":0.0018440049999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":227,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.862108+0000","last_change":"2024-09-17T07:09:24.396448+0000","last_active":"2024-09-17T07:12:29.862108+0000","last_peered":"2024-09-17T07:12:29.862108+0000","last_clean":"2024-09-17T07:12:29.862108+0000","last_became_active":"2024-09-17T07:09:24.396154+0000","last_became_peered":"2024-09-17T07:09:24.396154+0000","last_unstale":"2024-09-17T07:12:29.862108+0000","last_undegraded":"2024-09-17T07:12:29.862108+0000","last_fullsized":"2024-09-17T07:12:29.862108+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:17.192238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:09:17.192238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.058290885000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"247'3131","reported_seq":3150,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T07:12:39.245225+0000","last_change":"2024-09-17T07:12:05.176237+0000","last_active":"2024-09-17T07:12:39.245225+0000","last_peered":"2024-09-17T07:12:39.245225+0000","last_clean":"2024-09-17T07:12:39.245225+0000","last_became_active":"2024-09-17T07:12:05.175944+0000","last_became_peered":"2024-09-17T07:12:05.175944+0000","last_unstale":"2024-09-17T07:12:39.245225+0000","last_undegraded":"2024-09-17T07:12:39.245225+0000","last_fullsized":"2024-09-17T07:12:39.245225+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_clean_scrub_stamp":"2024-09-17T07:12:04.285115+0000","objects_scrubbed":0,"log_size":3131,"log_dups_size":0,"ondisk_log_size":3131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:21.517874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205193216,"num_objects":3131,"num_object_clones":0,"num_object_copies":9393,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3131,"num_write_kb":200384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"242'26","reported_seq":275,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.870844+0000","last_change":"2024-09-17T07:11:15.463787+0000","last_active":"2024-09-17T07:12:29.870844+0000","last_peered":"2024-09-17T07:12:29.870844+0000","last_clean":"2024-09-17T07:12:29.870844+0000","last_became_active":"2024-09-17T07:02:14.028857+0000","last_became_peered":"2024-09-17T07:02:14.028857+0000","last_unstale":"2024-09-17T07:12:29.870844+0000","last_undegraded":"2024-09-17T07:12:29.870844+0000","last_fullsized":"2024-09-17T07:12:29.870844+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'14","last_scrub_stamp":"2024-09-17T07:11:15.463520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:11:15.463520+0000","objects_scrubbed":7,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0066150439999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"72'2","reported_seq":94,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.860522+0000","last_change":"2024-09-17T07:12:01.874553+0000","last_active":"2024-09-17T07:12:29.860522+0000","last_peered":"2024-09-17T07:12:29.860522+0000","last_clean":"2024-09-17T07:12:29.860522+0000","last_became_active":"2024-09-17T07:09:24.404532+0000","last_became_peered":"2024-09-17T07:09:24.404532+0000","last_unstale":"2024-09-17T07:12:29.860522+0000","last_undegraded":"2024-09-17T07:12:29.860522+0000","last_fullsized":"2024-09-17T07:12:29.860522+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'2","last_scrub_stamp":"2024-09-17T07:12:01.874430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:12:01.874430+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:16.309823+0000","scrub_duration":0.0018649420000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":247,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T07:12:36.837023+0000","last_change":"2024-09-17T07:12:11.818502+0000","last_active":"2024-09-17T07:12:36.837023+0000","last_peered":"2024-09-17T07:12:36.837023+0000","last_clean":"2024-09-17T07:12:36.837023+0000","last_became_active":"2024-09-17T07:02:12.004606+0000","last_became_peered":"2024-09-17T07:02:12.004606+0000","last_unstale":"2024-09-17T07:12:36.837023+0000","last_undegraded":"2024-09-17T07:12:36.837023+0000","last_fullsized":"2024-09-17T07:12:36.837023+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:11.818392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:12:11.818392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:15.462083+0000","scrub_duration":0.0014854499999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"212'130","reported_seq":497,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.858871+0000","last_change":"2024-09-17T07:12:23.270434+0000","last_active":"2024-09-17T07:12:29.858871+0000","last_peered":"2024-09-17T07:12:29.858871+0000","last_clean":"2024-09-17T07:12:29.858871+0000","last_became_active":"2024-09-17T07:02:03.020675+0000","last_became_peered":"2024-09-17T07:02:03.020675+0000","last_unstale":"2024-09-17T07:12:29.858871+0000","last_undegraded":"2024-09-17T07:12:29.858871+0000","last_fullsized":"2024-09-17T07:12:29.858871+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:12:23.270099+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:10:37.684115+0000","last_clean_scrub_stamp":"2024-09-17T07:12:23.270099+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:27.199435+0000","scrub_duration":0.0032525319999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":231,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.858379+0000","last_change":"2024-09-17T07:10:36.722143+0000","last_active":"2024-09-17T07:12:29.858379+0000","last_peered":"2024-09-17T07:12:29.858379+0000","last_clean":"2024-09-17T07:12:29.858379+0000","last_became_active":"2024-09-17T07:02:16.016031+0000","last_became_peered":"2024-09-17T07:02:16.016031+0000","last_unstale":"2024-09-17T07:12:29.858379+0000","last_undegraded":"2024-09-17T07:12:29.858379+0000","last_fullsized":"2024-09-17T07:12:29.858379+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:10:36.721691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:10:36.721691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0088554370000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":238,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.863244+0000","last_change":"2024-09-17T07:12:18.707829+0000","last_active":"2024-09-17T07:12:29.863244+0000","last_peered":"2024-09-17T07:12:29.863244+0000","last_clean":"2024-09-17T07:12:29.863244+0000","last_became_active":"2024-09-17T07:09:24.400041+0000","last_became_peered":"2024-09-17T07:09:24.400041+0000","last_unstale":"2024-09-17T07:12:29.863244+0000","last_undegraded":"2024-09-17T07:12:29.863244+0000","last_fullsized":"2024-09-17T07:12:29.863244+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:18.707704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:58.670571+0000","last_clean_scrub_stamp":"2024-09-17T07:12:18.707704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:46.842558+0000","scrub_duration":0.0052717149999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.7","version":"247'3100","reported_seq":3119,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T07:12:38.329763+0000","last_change":"2024-09-17T07:12:05.418324+0000","last_active":"2024-09-17T07:12:38.329763+0000","last_peered":"2024-09-17T07:12:38.329763+0000","last_clean":"2024-09-17T07:12:38.329763+0000","last_became_active":"2024-09-17T07:12:05.417969+0000","last_became_peered":"2024-09-17T07:12:05.417969+0000","last_unstale":"2024-09-17T07:12:38.329763+0000","last_undegraded":"2024-09-17T07:12:38.329763+0000","last_fullsized":"2024-09-17T07:12:38.329763+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_clean_scrub_stamp":"2024-09-17T07:12:04.285115+0000","objects_scrubbed":0,"log_size":3100,"log_dups_size":0,"ondisk_log_size":3100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:04.730041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":203161600,"num_objects":3100,"num_object_clones":0,"num_object_copies":9300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3100,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3100,"num_write_kb":198400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"242'35","reported_seq":300,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.870963+0000","last_change":"2024-09-17T07:12:20.745763+0000","last_active":"2024-09-17T07:12:29.870963+0000","last_peered":"2024-09-17T07:12:29.870963+0000","last_clean":"2024-09-17T07:12:29.870963+0000","last_became_active":"2024-09-17T07:02:14.027313+0000","last_became_peered":"2024-09-17T07:02:14.027313+0000","last_unstale":"2024-09-17T07:12:29.870963+0000","last_undegraded":"2024-09-17T07:12:29.870963+0000","last_fullsized":"2024-09-17T07:12:29.870963+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'33","last_scrub_stamp":"2024-09-17T07:12:20.745628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:12:20.745628+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:28.059499+0000","scrub_duration":0.0047473180000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":260,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.858664+0000","last_change":"2024-09-17T07:12:17.126950+0000","last_active":"2024-09-17T07:12:29.858664+0000","last_peered":"2024-09-17T07:12:29.858664+0000","last_clean":"2024-09-17T07:12:29.858664+0000","last_became_active":"2024-09-17T07:02:12.012037+0000","last_became_peered":"2024-09-17T07:02:12.012037+0000","last_unstale":"2024-09-17T07:12:29.858664+0000","last_undegraded":"2024-09-17T07:12:29.858664+0000","last_fullsized":"2024-09-17T07:12:29.858664+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:17.126825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:06:39.322535+0000","last_clean_scrub_stamp":"2024-09-17T07:12:17.126825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:20.306203+0000","scrub_duration":0.001773309,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":255,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.858088+0000","last_change":"2024-09-17T07:12:04.096069+0000","last_active":"2024-09-17T07:12:29.858088+0000","last_peered":"2024-09-17T07:12:29.858088+0000","last_clean":"2024-09-17T07:12:29.858088+0000","last_became_active":"2024-09-17T07:02:16.014631+0000","last_became_peered":"2024-09-17T07:02:16.014631+0000","last_unstale":"2024-09-17T07:12:29.858088+0000","last_undegraded":"2024-09-17T07:12:29.858088+0000","last_fullsized":"2024-09-17T07:12:29.858088+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:04.095985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:12:04.095985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:31.712454+0000","scrub_duration":0.0015433160000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":238,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.858796+0000","last_change":"2024-09-17T07:12:22.309666+0000","last_active":"2024-09-17T07:12:29.858796+0000","last_peered":"2024-09-17T07:12:29.858796+0000","last_clean":"2024-09-17T07:12:29.858796+0000","last_became_active":"2024-09-17T07:09:24.403904+0000","last_became_peered":"2024-09-17T07:09:24.403904+0000","last_unstale":"2024-09-17T07:12:29.858796+0000","last_undegraded":"2024-09-17T07:12:29.858796+0000","last_fullsized":"2024-09-17T07:12:29.858796+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:22.309233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:12:22.309233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:34.704079+0000","scrub_duration":0.0042909669999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.6","version":"247'3105","reported_seq":3124,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T07:12:38.309385+0000","last_change":"2024-09-17T07:12:05.416479+0000","last_active":"2024-09-17T07:12:38.309385+0000","last_peered":"2024-09-17T07:12:38.309385+0000","last_clean":"2024-09-17T07:12:38.309385+0000","last_became_active":"2024-09-17T07:12:05.416160+0000","last_became_peered":"2024-09-17T07:12:05.416160+0000","last_unstale":"2024-09-17T07:12:38.309385+0000","last_undegraded":"2024-09-17T07:12:38.309385+0000","last_fullsized":"2024-09-17T07:12:38.309385+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_clean_scrub_stamp":"2024-09-17T07:12:04.285115+0000","objects_scrubbed":0,"log_size":3105,"log_dups_size":0,"ondisk_log_size":3105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:06.915351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":203489280,"num_objects":3105,"num_object_clones":0,"num_object_copies":9315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3105,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3105,"num_write_kb":198720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"242'15","reported_seq":259,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.862656+0000","last_change":"2024-09-17T07:12:12.585443+0000","last_active":"2024-09-17T07:12:29.862656+0000","last_peered":"2024-09-17T07:12:29.862656+0000","last_clean":"2024-09-17T07:12:29.862656+0000","last_became_active":"2024-09-17T07:02:14.014939+0000","last_became_peered":"2024-09-17T07:02:14.014939+0000","last_unstale":"2024-09-17T07:12:29.862656+0000","last_undegraded":"2024-09-17T07:12:29.862656+0000","last_fullsized":"2024-09-17T07:12:29.862656+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:12:12.585348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:12:12.585348+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:34.309086+0000","scrub_duration":0.001887445,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":244,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.864081+0000","last_change":"2024-09-17T07:11:29.068669+0000","last_active":"2024-09-17T07:12:29.864081+0000","last_peered":"2024-09-17T07:12:29.864081+0000","last_clean":"2024-09-17T07:12:29.864081+0000","last_became_active":"2024-09-17T07:09:24.391207+0000","last_became_peered":"2024-09-17T07:09:24.391207+0000","last_unstale":"2024-09-17T07:12:29.864081+0000","last_undegraded":"2024-09-17T07:12:29.864081+0000","last_fullsized":"2024-09-17T07:12:29.864081+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:29.068555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:06:30.038104+0000","last_clean_scrub_stamp":"2024-09-17T07:11:29.068555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:58.695021+0000","scrub_duration":0.0026579759999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":196,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.866044+0000","last_change":"2024-09-17T07:11:08.698817+0000","last_active":"2024-09-17T07:12:29.866044+0000","last_peered":"2024-09-17T07:12:29.866044+0000","last_clean":"2024-09-17T07:12:29.866044+0000","last_became_active":"2024-09-17T07:09:24.396642+0000","last_became_peered":"2024-09-17T07:09:24.396642+0000","last_unstale":"2024-09-17T07:12:29.866044+0000","last_undegraded":"2024-09-17T07:12:29.866044+0000","last_fullsized":"2024-09-17T07:12:29.866044+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:08.698703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:11:08.698703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0063428779999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":219,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.863587+0000","last_change":"2024-09-17T07:09:08.316161+0000","last_active":"2024-09-17T07:12:29.863587+0000","last_peered":"2024-09-17T07:12:29.863587+0000","last_clean":"2024-09-17T07:12:29.863587+0000","last_became_active":"2024-09-17T07:02:18.026994+0000","last_became_peered":"2024-09-17T07:02:18.026994+0000","last_unstale":"2024-09-17T07:12:29.863587+0000","last_undegraded":"2024-09-17T07:12:29.863587+0000","last_fullsized":"2024-09-17T07:12:29.863587+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:08.315842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:58.099849+0000","last_clean_scrub_stamp":"2024-09-17T07:09:08.315842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0081844919999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.5","version":"247'3213","reported_seq":3232,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T07:12:39.236192+0000","last_change":"2024-09-17T07:12:05.177132+0000","last_active":"2024-09-17T07:12:39.236192+0000","last_peered":"2024-09-17T07:12:39.236192+0000","last_clean":"2024-09-17T07:12:39.236192+0000","last_became_active":"2024-09-17T07:12:05.176729+0000","last_became_peered":"2024-09-17T07:12:05.176729+0000","last_unstale":"2024-09-17T07:12:39.236192+0000","last_undegraded":"2024-09-17T07:12:39.236192+0000","last_fullsized":"2024-09-17T07:12:39.236192+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_clean_scrub_stamp":"2024-09-17T07:12:04.285115+0000","objects_scrubbed":0,"log_size":3213,"log_dups_size":0,"ondisk_log_size":3213,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:24.650845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":210567168,"num_objects":3213,"num_object_clones":0,"num_object_copies":9639,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3213,"num_write_kb":205632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"242'34","reported_seq":283,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.861572+0000","last_change":"2024-09-17T07:10:35.125689+0000","last_active":"2024-09-17T07:12:29.861572+0000","last_peered":"2024-09-17T07:12:29.861572+0000","last_clean":"2024-09-17T07:12:29.861572+0000","last_became_active":"2024-09-17T07:02:14.028392+0000","last_became_peered":"2024-09-17T07:02:14.028392+0000","last_unstale":"2024-09-17T07:12:29.861572+0000","last_undegraded":"2024-09-17T07:12:29.861572+0000","last_fullsized":"2024-09-17T07:12:29.861572+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'22","last_scrub_stamp":"2024-09-17T07:10:35.125303+0000","last_deep_scrub":"111'22","last_deep_scrub_stamp":"2024-09-17T07:04:57.500450+0000","last_clean_scrub_stamp":"2024-09-17T07:10:35.125303+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.019620391000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'2","reported_seq":118,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.859734+0000","last_change":"2024-09-17T07:12:20.301388+0000","last_active":"2024-09-17T07:12:29.859734+0000","last_peered":"2024-09-17T07:12:29.859734+0000","last_clean":"2024-09-17T07:12:29.859734+0000","last_became_active":"2024-09-17T07:09:24.402909+0000","last_became_peered":"2024-09-17T07:09:24.402909+0000","last_unstale":"2024-09-17T07:12:29.859734+0000","last_undegraded":"2024-09-17T07:12:29.859734+0000","last_fullsized":"2024-09-17T07:12:29.859734+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-09-17T07:12:20.301263+0000","last_deep_scrub":"89'2","last_deep_scrub_stamp":"2024-09-17T07:12:20.301263+0000","last_clean_scrub_stamp":"2024-09-17T07:12:20.301263+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:38.286950+0000","scrub_duration":0.0065119280000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":420,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":420,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"225'9","reported_seq":368,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:35.649812+0000","last_change":"2024-09-17T07:11:25.292700+0000","last_active":"2024-09-17T07:12:35.649812+0000","last_peered":"2024-09-17T07:12:35.649812+0000","last_clean":"2024-09-17T07:12:35.649812+0000","last_became_active":"2024-09-17T07:02:16.015418+0000","last_became_peered":"2024-09-17T07:02:16.015418+0000","last_unstale":"2024-09-17T07:12:35.649812+0000","last_undegraded":"2024-09-17T07:12:35.649812+0000","last_fullsized":"2024-09-17T07:12:35.649812+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"225'9","last_scrub_stamp":"2024-09-17T07:11:25.292565+0000","last_deep_scrub":"225'9","last_deep_scrub_stamp":"2024-09-17T07:11:25.292565+0000","last_clean_scrub_stamp":"2024-09-17T07:11:25.292565+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:54.115350+0000","scrub_duration":0.0070251389999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":229,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.860248+0000","last_change":"2024-09-17T07:10:29.121100+0000","last_active":"2024-09-17T07:12:29.860248+0000","last_peered":"2024-09-17T07:12:29.860248+0000","last_clean":"2024-09-17T07:12:29.860248+0000","last_became_active":"2024-09-17T07:02:18.363253+0000","last_became_peered":"2024-09-17T07:02:18.363253+0000","last_unstale":"2024-09-17T07:12:29.860248+0000","last_undegraded":"2024-09-17T07:12:29.860248+0000","last_fullsized":"2024-09-17T07:12:29.860248+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:10:29.120985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:10:29.120985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.017226297000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.4","version":"247'3130","reported_seq":3150,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T07:12:38.527830+0000","last_change":"2024-09-17T07:12:05.169555+0000","last_active":"2024-09-17T07:12:38.527830+0000","last_peered":"2024-09-17T07:12:38.527830+0000","last_clean":"2024-09-17T07:12:38.527830+0000","last_became_active":"2024-09-17T07:12:05.169197+0000","last_became_peered":"2024-09-17T07:12:05.169197+0000","last_unstale":"2024-09-17T07:12:38.527830+0000","last_undegraded":"2024-09-17T07:12:38.527830+0000","last_fullsized":"2024-09-17T07:12:38.527830+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_clean_scrub_stamp":"2024-09-17T07:12:04.285115+0000","objects_scrubbed":0,"log_size":3130,"log_dups_size":0,"ondisk_log_size":3130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:19.862533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205127680,"num_objects":3130,"num_object_clones":0,"num_object_copies":9390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3130,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3130,"num_write_kb":200320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"242'23","reported_seq":314,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.866007+0000","last_change":"2024-09-17T07:12:07.649803+0000","last_active":"2024-09-17T07:12:29.866007+0000","last_peered":"2024-09-17T07:12:29.866007+0000","last_clean":"2024-09-17T07:12:29.866007+0000","last_became_active":"2024-09-17T07:10:49.723292+0000","last_became_peered":"2024-09-17T07:10:49.723292+0000","last_unstale":"2024-09-17T07:12:29.866007+0000","last_undegraded":"2024-09-17T07:12:29.866007+0000","last_fullsized":"2024-09-17T07:12:29.866007+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'17","last_scrub_stamp":"2024-09-17T07:12:07.649697+0000","last_deep_scrub":"150'17","last_deep_scrub_stamp":"2024-09-17T07:06:38.775788+0000","last_clean_scrub_stamp":"2024-09-17T07:12:07.649697+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:32.620327+0000","scrub_duration":0.002192238,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"68'2","reported_seq":249,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.867083+0000","last_change":"2024-09-17T07:11:05.636950+0000","last_active":"2024-09-17T07:12:29.867083+0000","last_peered":"2024-09-17T07:12:29.867083+0000","last_clean":"2024-09-17T07:12:29.867083+0000","last_became_active":"2024-09-17T07:10:49.727577+0000","last_became_peered":"2024-09-17T07:10:49.727577+0000","last_unstale":"2024-09-17T07:12:29.867083+0000","last_undegraded":"2024-09-17T07:12:29.867083+0000","last_fullsized":"2024-09-17T07:12:29.867083+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-09-17T07:11:05.636655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:11:05.636655+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.013113257999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":196,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.861566+0000","last_change":"2024-09-17T07:11:17.479269+0000","last_active":"2024-09-17T07:12:29.861566+0000","last_peered":"2024-09-17T07:12:29.861566+0000","last_clean":"2024-09-17T07:12:29.861566+0000","last_became_active":"2024-09-17T07:04:15.111069+0000","last_became_peered":"2024-09-17T07:04:15.111069+0000","last_unstale":"2024-09-17T07:12:29.861566+0000","last_undegraded":"2024-09-17T07:12:29.861566+0000","last_fullsized":"2024-09-17T07:12:29.861566+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:17.479036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:11:17.479036+0000","last_clean_scrub_stamp":"2024-09-17T07:11:17.479036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:43.456685+0000","scrub_duration":0.0014966910000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":76,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.858115+0000","last_change":"2024-09-17T07:11:11.432231+0000","last_active":"2024-09-17T07:12:29.858115+0000","last_peered":"2024-09-17T07:12:29.858115+0000","last_clean":"2024-09-17T07:12:29.858115+0000","last_became_active":"2024-09-17T07:09:24.397669+0000","last_became_peered":"2024-09-17T07:09:24.397669+0000","last_unstale":"2024-09-17T07:12:29.858115+0000","last_undegraded":"2024-09-17T07:12:29.858115+0000","last_fullsized":"2024-09-17T07:12:29.858115+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:11.432111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:11:11.432111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0075412869999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"242'15","reported_seq":278,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.863464+0000","last_change":"2024-09-17T07:12:16.562906+0000","last_active":"2024-09-17T07:12:29.863464+0000","last_peered":"2024-09-17T07:12:29.863464+0000","last_clean":"2024-09-17T07:12:29.863464+0000","last_became_active":"2024-09-17T07:09:24.408875+0000","last_became_peered":"2024-09-17T07:09:24.408875+0000","last_unstale":"2024-09-17T07:12:29.863464+0000","last_undegraded":"2024-09-17T07:12:29.863464+0000","last_fullsized":"2024-09-17T07:12:29.863464+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:12:16.562818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:12:16.562818+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:26.004316+0000","scrub_duration":0.002298876,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":260,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.857766+0000","last_change":"2024-09-17T07:12:16.090212+0000","last_active":"2024-09-17T07:12:29.857766+0000","last_peered":"2024-09-17T07:12:29.857766+0000","last_clean":"2024-09-17T07:12:29.857766+0000","last_became_active":"2024-09-17T07:02:12.011327+0000","last_became_peered":"2024-09-17T07:02:12.011327+0000","last_unstale":"2024-09-17T07:12:29.857766+0000","last_undegraded":"2024-09-17T07:12:29.857766+0000","last_fullsized":"2024-09-17T07:12:29.857766+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:16.090110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:17.976267+0000","last_clean_scrub_stamp":"2024-09-17T07:12:16.090110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:41.373062+0000","scrub_duration":0.0012389359999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":75,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.862905+0000","last_change":"2024-09-17T07:12:18.367610+0000","last_active":"2024-09-17T07:12:29.862905+0000","last_peered":"2024-09-17T07:12:29.862905+0000","last_clean":"2024-09-17T07:12:29.862905+0000","last_became_active":"2024-09-17T07:09:24.407806+0000","last_became_peered":"2024-09-17T07:09:24.407806+0000","last_unstale":"2024-09-17T07:12:29.862905+0000","last_undegraded":"2024-09-17T07:12:29.862905+0000","last_fullsized":"2024-09-17T07:12:29.862905+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:18.367486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:18.367486+0000","last_clean_scrub_stamp":"2024-09-17T07:12:18.367486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:44.237351+0000","scrub_duration":0.0038802519999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":265,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.858266+0000","last_change":"2024-09-17T07:12:22.178678+0000","last_active":"2024-09-17T07:12:29.858266+0000","last_peered":"2024-09-17T07:12:29.858266+0000","last_clean":"2024-09-17T07:12:29.858266+0000","last_became_active":"2024-09-17T07:02:18.017825+0000","last_became_peered":"2024-09-17T07:02:18.017825+0000","last_unstale":"2024-09-17T07:12:29.858266+0000","last_undegraded":"2024-09-17T07:12:29.858266+0000","last_fullsized":"2024-09-17T07:12:29.858266+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:22.178556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:09:23.378805+0000","last_clean_scrub_stamp":"2024-09-17T07:12:22.178556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:28.430654+0000","scrub_duration":0.0029804850000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":248,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.870648+0000","last_change":"2024-09-17T07:12:17.648175+0000","last_active":"2024-09-17T07:12:29.870648+0000","last_peered":"2024-09-17T07:12:29.870648+0000","last_clean":"2024-09-17T07:12:29.870648+0000","last_became_active":"2024-09-17T07:02:12.008065+0000","last_became_peered":"2024-09-17T07:02:12.008065+0000","last_unstale":"2024-09-17T07:12:29.870648+0000","last_undegraded":"2024-09-17T07:12:29.870648+0000","last_fullsized":"2024-09-17T07:12:29.870648+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:17.648051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:12:17.648051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:33.995482+0000","scrub_duration":0.0030045190000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"242'12","reported_seq":113,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.858954+0000","last_change":"2024-09-17T07:12:03.380136+0000","last_active":"2024-09-17T07:12:29.858954+0000","last_peered":"2024-09-17T07:12:29.858954+0000","last_clean":"2024-09-17T07:12:29.858954+0000","last_became_active":"2024-09-17T07:09:24.405385+0000","last_became_peered":"2024-09-17T07:09:24.405385+0000","last_unstale":"2024-09-17T07:12:29.858954+0000","last_undegraded":"2024-09-17T07:12:29.858954+0000","last_fullsized":"2024-09-17T07:12:29.858954+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'6","last_scrub_stamp":"2024-09-17T07:12:03.380040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:12:03.380040+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:31.256725+0000","scrub_duration":0.0019381719999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":240,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.861143+0000","last_change":"2024-09-17T07:11:16.112334+0000","last_active":"2024-09-17T07:12:29.861143+0000","last_peered":"2024-09-17T07:12:29.861143+0000","last_clean":"2024-09-17T07:12:29.861143+0000","last_became_active":"2024-09-17T07:09:24.398922+0000","last_became_peered":"2024-09-17T07:09:24.398922+0000","last_unstale":"2024-09-17T07:12:29.861143+0000","last_undegraded":"2024-09-17T07:12:29.861143+0000","last_fullsized":"2024-09-17T07:12:29.861143+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:16.112226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:11:16.112226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0062666659999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":229,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.860714+0000","last_change":"2024-09-17T07:12:06.344966+0000","last_active":"2024-09-17T07:12:29.860714+0000","last_peered":"2024-09-17T07:12:29.860714+0000","last_clean":"2024-09-17T07:12:29.860714+0000","last_became_active":"2024-09-17T07:02:18.365419+0000","last_became_peered":"2024-09-17T07:02:18.365419+0000","last_unstale":"2024-09-17T07:12:29.860714+0000","last_undegraded":"2024-09-17T07:12:29.860714+0000","last_fullsized":"2024-09-17T07:12:29.860714+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:06.344833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:12:06.344833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:19.489450+0000","scrub_duration":0.002302817,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"242'20","reported_seq":130,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.862277+0000","last_change":"2024-09-17T07:12:12.288945+0000","last_active":"2024-09-17T07:12:29.862277+0000","last_peered":"2024-09-17T07:12:29.862277+0000","last_clean":"2024-09-17T07:12:29.862277+0000","last_became_active":"2024-09-17T07:09:24.405944+0000","last_became_peered":"2024-09-17T07:09:24.405944+0000","last_unstale":"2024-09-17T07:12:29.862277+0000","last_undegraded":"2024-09-17T07:12:29.862277+0000","last_fullsized":"2024-09-17T07:12:29.862277+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-09-17T07:12:12.288864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:12:12.288864+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:17.960594+0000","scrub_duration":0.0020069329999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":255,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.860682+0000","last_change":"2024-09-17T07:12:09.840697+0000","last_active":"2024-09-17T07:12:29.860682+0000","last_peered":"2024-09-17T07:12:29.860682+0000","last_clean":"2024-09-17T07:12:29.860682+0000","last_became_active":"2024-09-17T07:09:24.402355+0000","last_became_peered":"2024-09-17T07:09:24.402355+0000","last_unstale":"2024-09-17T07:12:29.860682+0000","last_undegraded":"2024-09-17T07:12:29.860682+0000","last_fullsized":"2024-09-17T07:12:29.860682+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:09.840613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:12:09.840613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:16.811598+0000","scrub_duration":0.0067048560000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"163'9","reported_seq":365,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:35.650481+0000","last_change":"2024-09-17T07:11:08.147847+0000","last_active":"2024-09-17T07:12:35.650481+0000","last_peered":"2024-09-17T07:12:35.650481+0000","last_clean":"2024-09-17T07:12:35.650481+0000","last_became_active":"2024-09-17T07:02:16.022583+0000","last_became_peered":"2024-09-17T07:02:16.022583+0000","last_unstale":"2024-09-17T07:12:35.650481+0000","last_undegraded":"2024-09-17T07:12:35.650481+0000","last_fullsized":"2024-09-17T07:12:35.650481+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'9","last_scrub_stamp":"2024-09-17T07:11:08.147466+0000","last_deep_scrub":"163'9","last_deep_scrub_stamp":"2024-09-17T07:07:34.933808+0000","last_clean_scrub_stamp":"2024-09-17T07:11:08.147466+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.039751243999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":239,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.863011+0000","last_change":"2024-09-17T07:12:26.619980+0000","last_active":"2024-09-17T07:12:29.863011+0000","last_peered":"2024-09-17T07:12:29.863011+0000","last_clean":"2024-09-17T07:12:29.863011+0000","last_became_active":"2024-09-17T07:09:24.409051+0000","last_became_peered":"2024-09-17T07:09:24.409051+0000","last_unstale":"2024-09-17T07:12:29.863011+0000","last_undegraded":"2024-09-17T07:12:29.863011+0000","last_fullsized":"2024-09-17T07:12:29.863011+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:26.619893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:26.619893+0000","last_clean_scrub_stamp":"2024-09-17T07:12:26.619893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:34.198640+0000","scrub_duration":0.0044029580000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"242'15","reported_seq":288,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.862115+0000","last_change":"2024-09-17T07:12:10.828225+0000","last_active":"2024-09-17T07:12:29.862115+0000","last_peered":"2024-09-17T07:12:29.862115+0000","last_clean":"2024-09-17T07:12:29.862115+0000","last_became_active":"2024-09-17T07:09:24.406333+0000","last_became_peered":"2024-09-17T07:09:24.406333+0000","last_unstale":"2024-09-17T07:12:29.862115+0000","last_undegraded":"2024-09-17T07:12:29.862115+0000","last_fullsized":"2024-09-17T07:12:29.862115+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:12:10.828114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:12:10.828114+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:28.226498+0000","scrub_duration":0.001735435,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":245,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.862929+0000","last_change":"2024-09-17T07:12:23.597448+0000","last_active":"2024-09-17T07:12:29.862929+0000","last_peered":"2024-09-17T07:12:29.862929+0000","last_clean":"2024-09-17T07:12:29.862929+0000","last_became_active":"2024-09-17T07:09:24.402704+0000","last_became_peered":"2024-09-17T07:09:24.402704+0000","last_unstale":"2024-09-17T07:12:29.862929+0000","last_undegraded":"2024-09-17T07:12:29.862929+0000","last_fullsized":"2024-09-17T07:12:29.862929+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:23.597330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:12:23.597330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:47.442372+0000","scrub_duration":0.0032745959999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"196'9","reported_seq":139,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:35.649032+0000","last_change":"2024-09-17T07:11:02.536969+0000","last_active":"2024-09-17T07:12:35.649032+0000","last_peered":"2024-09-17T07:12:35.649032+0000","last_clean":"2024-09-17T07:12:35.649032+0000","last_became_active":"2024-09-17T07:09:24.406267+0000","last_became_peered":"2024-09-17T07:09:24.406267+0000","last_unstale":"2024-09-17T07:12:35.649032+0000","last_undegraded":"2024-09-17T07:12:35.649032+0000","last_fullsized":"2024-09-17T07:12:35.649032+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'9","last_scrub_stamp":"2024-09-17T07:11:02.536718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:11:02.536718+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0054409640000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":231,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.866782+0000","last_change":"2024-09-17T07:12:16.543360+0000","last_active":"2024-09-17T07:12:29.866782+0000","last_peered":"2024-09-17T07:12:29.866782+0000","last_clean":"2024-09-17T07:12:29.866782+0000","last_became_active":"2024-09-17T07:02:18.018716+0000","last_became_peered":"2024-09-17T07:02:18.018716+0000","last_unstale":"2024-09-17T07:12:29.866782+0000","last_undegraded":"2024-09-17T07:12:29.866782+0000","last_fullsized":"2024-09-17T07:12:29.866782+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:16.543257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:12:16.543257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:33.805404+0000","scrub_duration":0.001567354,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"242'15","reported_seq":258,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.863799+0000","last_change":"2024-09-17T07:11:21.246333+0000","last_active":"2024-09-17T07:12:29.863799+0000","last_peered":"2024-09-17T07:12:29.863799+0000","last_clean":"2024-09-17T07:12:29.863799+0000","last_became_active":"2024-09-17T07:02:14.010289+0000","last_became_peered":"2024-09-17T07:02:14.010289+0000","last_unstale":"2024-09-17T07:12:29.863799+0000","last_undegraded":"2024-09-17T07:12:29.863799+0000","last_fullsized":"2024-09-17T07:12:29.863799+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:11:21.246183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:11:21.246183+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.005053564,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":256,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.870543+0000","last_change":"2024-09-17T07:12:13.677093+0000","last_active":"2024-09-17T07:12:29.870543+0000","last_peered":"2024-09-17T07:12:29.870543+0000","last_clean":"2024-09-17T07:12:29.870543+0000","last_became_active":"2024-09-17T07:09:24.408139+0000","last_became_peered":"2024-09-17T07:09:24.408139+0000","last_unstale":"2024-09-17T07:12:29.870543+0000","last_undegraded":"2024-09-17T07:12:29.870543+0000","last_fullsized":"2024-09-17T07:12:29.870543+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:13.677010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:10:36.334827+0000","last_clean_scrub_stamp":"2024-09-17T07:12:13.677010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:14.451688+0000","scrub_duration":0.0012308760000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":76,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.861783+0000","last_change":"2024-09-17T07:11:04.420764+0000","last_active":"2024-09-17T07:12:29.861783+0000","last_peered":"2024-09-17T07:12:29.861783+0000","last_clean":"2024-09-17T07:12:29.861783+0000","last_became_active":"2024-09-17T07:09:24.399116+0000","last_became_peered":"2024-09-17T07:09:24.399116+0000","last_unstale":"2024-09-17T07:12:29.861783+0000","last_undegraded":"2024-09-17T07:12:29.861783+0000","last_fullsized":"2024-09-17T07:12:29.861783+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:04.420639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:11:04.420639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011748066999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":230,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.861467+0000","last_change":"2024-09-17T07:12:06.573120+0000","last_active":"2024-09-17T07:12:29.861467+0000","last_peered":"2024-09-17T07:12:29.861467+0000","last_clean":"2024-09-17T07:12:29.861467+0000","last_became_active":"2024-09-17T07:02:18.027622+0000","last_became_peered":"2024-09-17T07:02:18.027622+0000","last_unstale":"2024-09-17T07:12:29.861467+0000","last_undegraded":"2024-09-17T07:12:29.861467+0000","last_fullsized":"2024-09-17T07:12:29.861467+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:06.573041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:12:06.573041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:11.909768+0000","scrub_duration":0.0024674409999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"242'19","reported_seq":266,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.866192+0000","last_change":"2024-09-17T07:12:13.566691+0000","last_active":"2024-09-17T07:12:29.866192+0000","last_peered":"2024-09-17T07:12:29.866192+0000","last_clean":"2024-09-17T07:12:29.866192+0000","last_became_active":"2024-09-17T07:02:14.016255+0000","last_became_peered":"2024-09-17T07:02:14.016255+0000","last_unstale":"2024-09-17T07:12:29.866192+0000","last_undegraded":"2024-09-17T07:12:29.866192+0000","last_fullsized":"2024-09-17T07:12:29.866192+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-09-17T07:12:13.566608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:12:13.566608+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:36.971016+0000","scrub_duration":0.0019196129999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"94'2","reported_seq":230,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.864056+0000","last_change":"2024-09-17T07:09:18.169710+0000","last_active":"2024-09-17T07:12:29.864056+0000","last_peered":"2024-09-17T07:12:29.864056+0000","last_clean":"2024-09-17T07:12:29.864056+0000","last_became_active":"2024-09-17T07:02:12.011461+0000","last_became_peered":"2024-09-17T07:02:12.011461+0000","last_unstale":"2024-09-17T07:12:29.864056+0000","last_undegraded":"2024-09-17T07:12:29.864056+0000","last_fullsized":"2024-09-17T07:12:29.864056+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-09-17T07:09:18.169580+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-09-17T07:07:31.717888+0000","last_clean_scrub_stamp":"2024-09-17T07:09:18.169580+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011743155,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":240,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.862698+0000","last_change":"2024-09-17T07:12:23.390704+0000","last_active":"2024-09-17T07:12:29.862698+0000","last_peered":"2024-09-17T07:12:29.862698+0000","last_clean":"2024-09-17T07:12:29.862698+0000","last_became_active":"2024-09-17T07:09:24.407362+0000","last_became_peered":"2024-09-17T07:09:24.407362+0000","last_unstale":"2024-09-17T07:12:29.862698+0000","last_undegraded":"2024-09-17T07:12:29.862698+0000","last_fullsized":"2024-09-17T07:12:29.862698+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:23.390587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:12:23.390587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:45.619920+0000","scrub_duration":0.010514658,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":75,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.863185+0000","last_change":"2024-09-17T07:12:05.555207+0000","last_active":"2024-09-17T07:12:29.863185+0000","last_peered":"2024-09-17T07:12:29.863185+0000","last_clean":"2024-09-17T07:12:29.863185+0000","last_became_active":"2024-09-17T07:09:24.400791+0000","last_became_peered":"2024-09-17T07:09:24.400791+0000","last_unstale":"2024-09-17T07:12:29.863185+0000","last_undegraded":"2024-09-17T07:12:29.863185+0000","last_fullsized":"2024-09-17T07:12:29.863185+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:05.555100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:12:05.555100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:14.820862+0000","scrub_duration":0.00158344,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"242'8","reported_seq":138,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.859459+0000","last_change":"2024-09-17T07:11:07.499806+0000","last_active":"2024-09-17T07:12:29.859459+0000","last_peered":"2024-09-17T07:12:29.859459+0000","last_clean":"2024-09-17T07:12:29.859459+0000","last_became_active":"2024-09-17T07:09:24.396368+0000","last_became_peered":"2024-09-17T07:09:24.396368+0000","last_unstale":"2024-09-17T07:12:29.859459+0000","last_undegraded":"2024-09-17T07:12:29.859459+0000","last_fullsized":"2024-09-17T07:12:29.859459+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'4","last_scrub_stamp":"2024-09-17T07:11:07.499679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:11:07.499679+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0043183819999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":235,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.861600+0000","last_change":"2024-09-17T07:11:24.083407+0000","last_active":"2024-09-17T07:12:29.861600+0000","last_peered":"2024-09-17T07:12:29.861600+0000","last_clean":"2024-09-17T07:12:29.861600+0000","last_became_active":"2024-09-17T07:02:12.008898+0000","last_became_peered":"2024-09-17T07:02:12.008898+0000","last_unstale":"2024-09-17T07:12:29.861600+0000","last_undegraded":"2024-09-17T07:12:29.861600+0000","last_fullsized":"2024-09-17T07:12:29.861600+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:24.083284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:11:24.083284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:40.949692+0000","scrub_duration":0.014006177999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":233,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.870115+0000","last_change":"2024-09-17T07:11:03.383292+0000","last_active":"2024-09-17T07:12:29.870115+0000","last_peered":"2024-09-17T07:12:29.870115+0000","last_clean":"2024-09-17T07:12:29.870115+0000","last_became_active":"2024-09-17T07:02:16.014982+0000","last_became_peered":"2024-09-17T07:02:16.014982+0000","last_unstale":"2024-09-17T07:12:29.870115+0000","last_undegraded":"2024-09-17T07:12:29.870115+0000","last_fullsized":"2024-09-17T07:12:29.870115+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:03.383039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:47.444972+0000","last_clean_scrub_stamp":"2024-09-17T07:11:03.383039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015364230000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":229,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.859115+0000","last_change":"2024-09-17T07:11:23.302709+0000","last_active":"2024-09-17T07:12:29.859115+0000","last_peered":"2024-09-17T07:12:29.859115+0000","last_clean":"2024-09-17T07:12:29.859115+0000","last_became_active":"2024-09-17T07:02:18.021014+0000","last_became_peered":"2024-09-17T07:02:18.021014+0000","last_unstale":"2024-09-17T07:12:29.859115+0000","last_undegraded":"2024-09-17T07:12:29.859115+0000","last_fullsized":"2024-09-17T07:12:29.859115+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:23.302586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:11:23.302586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:45.028048+0000","scrub_duration":0.0077240449999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"242'19","reported_seq":130,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.859479+0000","last_change":"2024-09-17T07:11:20.321025+0000","last_active":"2024-09-17T07:12:29.859479+0000","last_peered":"2024-09-17T07:12:29.859479+0000","last_clean":"2024-09-17T07:12:29.859479+0000","last_became_active":"2024-09-17T07:09:24.404942+0000","last_became_peered":"2024-09-17T07:09:24.404942+0000","last_unstale":"2024-09-17T07:12:29.859479+0000","last_undegraded":"2024-09-17T07:12:29.859479+0000","last_fullsized":"2024-09-17T07:12:29.859479+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-09-17T07:11:20.320870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:11:20.320870+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:37.430071+0000","scrub_duration":0.013438802,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":247,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.870744+0000","last_change":"2024-09-17T07:11:14.431423+0000","last_active":"2024-09-17T07:12:29.870744+0000","last_peered":"2024-09-17T07:12:29.870744+0000","last_clean":"2024-09-17T07:12:29.870744+0000","last_became_active":"2024-09-17T07:02:12.010195+0000","last_became_peered":"2024-09-17T07:02:12.010195+0000","last_unstale":"2024-09-17T07:12:29.870744+0000","last_undegraded":"2024-09-17T07:12:29.870744+0000","last_fullsized":"2024-09-17T07:12:29.870744+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:14.431102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:11:14.431102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:37.926489+0000","scrub_duration":0.0070527250000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":231,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.857915+0000","last_change":"2024-09-17T07:11:26.259825+0000","last_active":"2024-09-17T07:12:29.857915+0000","last_peered":"2024-09-17T07:12:29.857915+0000","last_clean":"2024-09-17T07:12:29.857915+0000","last_became_active":"2024-09-17T07:02:16.016183+0000","last_became_peered":"2024-09-17T07:02:16.016183+0000","last_unstale":"2024-09-17T07:12:29.857915+0000","last_undegraded":"2024-09-17T07:12:29.857915+0000","last_fullsized":"2024-09-17T07:12:29.857915+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:26.258291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:11:26.258291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:37.661689+0000","scrub_duration":0.0034422369999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":75,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.861734+0000","last_change":"2024-09-17T07:10:34.515889+0000","last_active":"2024-09-17T07:12:29.861734+0000","last_peered":"2024-09-17T07:12:29.861734+0000","last_clean":"2024-09-17T07:12:29.861734+0000","last_became_active":"2024-09-17T07:09:24.392216+0000","last_became_peered":"2024-09-17T07:09:24.392216+0000","last_unstale":"2024-09-17T07:12:29.861734+0000","last_undegraded":"2024-09-17T07:12:29.861734+0000","last_fullsized":"2024-09-17T07:12:29.861734+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:10:34.515779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:10:34.515779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032588679999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1c","version":"75'1","reported_seq":230,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.858303+0000","last_change":"2024-09-17T07:11:17.398031+0000","last_active":"2024-09-17T07:12:29.858303+0000","last_peered":"2024-09-17T07:12:29.858303+0000","last_clean":"2024-09-17T07:12:29.858303+0000","last_became_active":"2024-09-17T07:02:18.021855+0000","last_became_peered":"2024-09-17T07:02:18.021855+0000","last_unstale":"2024-09-17T07:12:29.858303+0000","last_undegraded":"2024-09-17T07:12:29.858303+0000","last_fullsized":"2024-09-17T07:12:29.858303+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-17T07:11:17.396229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:11:17.396229+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:39.204660+0000","scrub_duration":0.0090123449999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":236,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.861643+0000","last_change":"2024-09-17T07:11:06.126289+0000","last_active":"2024-09-17T07:12:29.861643+0000","last_peered":"2024-09-17T07:12:29.861643+0000","last_clean":"2024-09-17T07:12:29.861643+0000","last_became_active":"2024-09-17T07:02:12.005329+0000","last_became_peered":"2024-09-17T07:02:12.005329+0000","last_unstale":"2024-09-17T07:12:29.861643+0000","last_undegraded":"2024-09-17T07:12:29.861643+0000","last_fullsized":"2024-09-17T07:12:29.861643+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:06.126127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:11:06.126127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001691456,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"242'20","reported_seq":307,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.863346+0000","last_change":"2024-09-17T07:12:25.625670+0000","last_active":"2024-09-17T07:12:29.863346+0000","last_peered":"2024-09-17T07:12:29.863346+0000","last_clean":"2024-09-17T07:12:29.863346+0000","last_became_active":"2024-09-17T07:10:49.730948+0000","last_became_peered":"2024-09-17T07:10:49.730948+0000","last_unstale":"2024-09-17T07:12:29.863346+0000","last_undegraded":"2024-09-17T07:12:29.863346+0000","last_fullsized":"2024-09-17T07:12:29.863346+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'20","last_scrub_stamp":"2024-09-17T07:12:25.625530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:12:25.625530+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:27.820616+0000","scrub_duration":0.0066279870000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"75'8","reported_seq":369,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:35.648828+0000","last_change":"2024-09-17T07:11:12.356249+0000","last_active":"2024-09-17T07:12:35.648828+0000","last_peered":"2024-09-17T07:12:35.648828+0000","last_clean":"2024-09-17T07:12:35.648828+0000","last_became_active":"2024-09-17T07:02:16.020560+0000","last_became_peered":"2024-09-17T07:02:16.020560+0000","last_unstale":"2024-09-17T07:12:35.648828+0000","last_undegraded":"2024-09-17T07:12:35.648828+0000","last_fullsized":"2024-09-17T07:12:35.648828+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:11:12.356129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:11:12.356129+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:39.144563+0000","scrub_duration":0.0063199399999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":229,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.862551+0000","last_change":"2024-09-17T07:12:21.397606+0000","last_active":"2024-09-17T07:12:29.862551+0000","last_peered":"2024-09-17T07:12:29.862551+0000","last_clean":"2024-09-17T07:12:29.862551+0000","last_became_active":"2024-09-17T07:02:18.021603+0000","last_became_peered":"2024-09-17T07:02:18.021603+0000","last_unstale":"2024-09-17T07:12:29.862551+0000","last_undegraded":"2024-09-17T07:12:29.862551+0000","last_fullsized":"2024-09-17T07:12:29.862551+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:21.397491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:12:21.397491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:42.061961+0000","scrub_duration":0.0045805669999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":87,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.870577+0000","last_change":"2024-09-17T07:12:09.738322+0000","last_active":"2024-09-17T07:12:29.870577+0000","last_peered":"2024-09-17T07:12:29.870577+0000","last_clean":"2024-09-17T07:12:29.870577+0000","last_became_active":"2024-09-17T07:09:24.403360+0000","last_became_peered":"2024-09-17T07:09:24.403360+0000","last_unstale":"2024-09-17T07:12:29.870577+0000","last_undegraded":"2024-09-17T07:12:29.870577+0000","last_fullsized":"2024-09-17T07:12:29.870577+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:09.738248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:10:29.325664+0000","last_clean_scrub_stamp":"2024-09-17T07:12:09.738248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:27.942295+0000","scrub_duration":0.0014227129999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"242'19","reported_seq":319,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T07:12:36.834844+0000","last_change":"2024-09-17T07:12:27.748152+0000","last_active":"2024-09-17T07:12:36.834844+0000","last_peered":"2024-09-17T07:12:36.834844+0000","last_clean":"2024-09-17T07:12:36.834844+0000","last_became_active":"2024-09-17T07:03:48.760390+0000","last_became_peered":"2024-09-17T07:03:48.760390+0000","last_unstale":"2024-09-17T07:12:36.834844+0000","last_undegraded":"2024-09-17T07:12:36.834844+0000","last_fullsized":"2024-09-17T07:12:36.834844+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'19","last_scrub_stamp":"2024-09-17T07:12:27.748022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:12:27.748022+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:53.737985+0000","scrub_duration":0.0036302800000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":222,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.863098+0000","last_change":"2024-09-17T07:09:10.293301+0000","last_active":"2024-09-17T07:12:29.863098+0000","last_peered":"2024-09-17T07:12:29.863098+0000","last_clean":"2024-09-17T07:12:29.863098+0000","last_became_active":"2024-09-17T07:02:16.019701+0000","last_became_peered":"2024-09-17T07:02:16.019701+0000","last_unstale":"2024-09-17T07:12:29.863098+0000","last_undegraded":"2024-09-17T07:12:29.863098+0000","last_fullsized":"2024-09-17T07:12:29.863098+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:10.292615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:09:10.292615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010112249,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":232,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.867967+0000","last_change":"2024-09-17T07:12:27.469985+0000","last_active":"2024-09-17T07:12:29.867967+0000","last_peered":"2024-09-17T07:12:29.867967+0000","last_clean":"2024-09-17T07:12:29.867967+0000","last_became_active":"2024-09-17T07:02:18.363752+0000","last_became_peered":"2024-09-17T07:02:18.363752+0000","last_unstale":"2024-09-17T07:12:29.867967+0000","last_undegraded":"2024-09-17T07:12:29.867967+0000","last_fullsized":"2024-09-17T07:12:29.867967+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:27.469698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:12:27.469698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:34.111455+0000","scrub_duration":0.0056145910000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":277,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.871090+0000","last_change":"2024-09-17T07:12:21.708679+0000","last_active":"2024-09-17T07:12:29.871090+0000","last_peered":"2024-09-17T07:12:29.871090+0000","last_clean":"2024-09-17T07:12:29.871090+0000","last_became_active":"2024-09-17T07:03:48.759384+0000","last_became_peered":"2024-09-17T07:03:48.759384+0000","last_unstale":"2024-09-17T07:12:29.871090+0000","last_undegraded":"2024-09-17T07:12:29.871090+0000","last_fullsized":"2024-09-17T07:12:29.871090+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:21.708543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:11:18.559877+0000","last_clean_scrub_stamp":"2024-09-17T07:12:21.708543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:38.909772+0000","scrub_duration":0.001526856,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"242'27","reported_seq":277,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.863395+0000","last_change":"2024-09-17T07:09:24.427421+0000","last_active":"2024-09-17T07:12:29.863395+0000","last_peered":"2024-09-17T07:12:29.863395+0000","last_clean":"2024-09-17T07:12:29.863395+0000","last_became_active":"2024-09-17T07:09:24.403030+0000","last_became_peered":"2024-09-17T07:09:24.403030+0000","last_unstale":"2024-09-17T07:12:29.863395+0000","last_undegraded":"2024-09-17T07:12:29.863395+0000","last_fullsized":"2024-09-17T07:12:29.863395+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-09-17T07:09:15.110940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:09:15.110940+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.017595989999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":231,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.863945+0000","last_change":"2024-09-17T07:12:01.366198+0000","last_active":"2024-09-17T07:12:29.863945+0000","last_peered":"2024-09-17T07:12:29.863945+0000","last_clean":"2024-09-17T07:12:29.863945+0000","last_became_active":"2024-09-17T07:02:16.015903+0000","last_became_peered":"2024-09-17T07:02:16.015903+0000","last_unstale":"2024-09-17T07:12:29.863945+0000","last_undegraded":"2024-09-17T07:12:29.863945+0000","last_fullsized":"2024-09-17T07:12:29.863945+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:01.366088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:25.794418+0000","last_clean_scrub_stamp":"2024-09-17T07:12:01.366088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:01.913633+0000","scrub_duration":0.001615116,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":88,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.870373+0000","last_change":"2024-09-17T07:12:22.734468+0000","last_active":"2024-09-17T07:12:29.870373+0000","last_peered":"2024-09-17T07:12:29.870373+0000","last_clean":"2024-09-17T07:12:29.870373+0000","last_became_active":"2024-09-17T07:09:24.402411+0000","last_became_peered":"2024-09-17T07:09:24.402411+0000","last_unstale":"2024-09-17T07:12:29.870373+0000","last_undegraded":"2024-09-17T07:12:29.870373+0000","last_fullsized":"2024-09-17T07:12:29.870373+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:22.734328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:12:22.734328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:24.175241+0000","scrub_duration":0.0038730150000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":77,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.858765+0000","last_change":"2024-09-17T07:11:00.563847+0000","last_active":"2024-09-17T07:12:29.858765+0000","last_peered":"2024-09-17T07:12:29.858765+0000","last_clean":"2024-09-17T07:12:29.858765+0000","last_became_active":"2024-09-17T07:09:24.404795+0000","last_became_peered":"2024-09-17T07:09:24.404795+0000","last_unstale":"2024-09-17T07:12:29.858765+0000","last_undegraded":"2024-09-17T07:12:29.858765+0000","last_fullsized":"2024-09-17T07:12:29.858765+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:00.563722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:11:00.563722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.018069940999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"242'23","reported_seq":150,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.863978+0000","last_change":"2024-09-17T07:12:24.341244+0000","last_active":"2024-09-17T07:12:29.863978+0000","last_peered":"2024-09-17T07:12:29.863978+0000","last_clean":"2024-09-17T07:12:29.863978+0000","last_became_active":"2024-09-17T07:09:24.394476+0000","last_became_peered":"2024-09-17T07:09:24.394476+0000","last_unstale":"2024-09-17T07:12:29.863978+0000","last_undegraded":"2024-09-17T07:12:29.863978+0000","last_fullsized":"2024-09-17T07:12:29.863978+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'23","last_scrub_stamp":"2024-09-17T07:12:24.341132+0000","last_deep_scrub":"86'17","last_deep_scrub_stamp":"2024-09-17T07:03:18.241067+0000","last_clean_scrub_stamp":"2024-09-17T07:12:24.341132+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:34.696830+0000","scrub_duration":0.0043568469999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":242,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T07:12:29.866812+0000","last_change":"2024-09-17T07:11:14.763627+0000","last_active":"2024-09-17T07:12:29.866812+0000","last_peered":"2024-09-17T07:12:29.866812+0000","last_clean":"2024-09-17T07:12:29.866812+0000","last_became_active":"2024-09-17T07:09:24.393329+0000","last_became_peered":"2024-09-17T07:09:24.393329+0000","last_unstale":"2024-09-17T07:12:29.866812+0000","last_undegraded":"2024-09-17T07:12:29.866812+0000","last_fullsized":"2024-09-17T07:12:29.866812+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:14.763470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:11:14.763470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:39.147572+0000","scrub_duration":0.0044668709999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":3293118464,"num_objects":50249,"num_object_clones":0,"num_object_copies":150747,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":50249,"num_write_kb":3215936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9844817920,"data_stored":9844817920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":50249,"ondisk_log_size":50249,"up":48,"acting":48,"num_store_stats":7},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70,"ondisk_log_size":70,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1121,"num_read_kb":878,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":109,"num_bytes_recovered":2228,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":13004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":713,"ondisk_log_size":713,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":619,"num_read_kb":614,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1386,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":12,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":4578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037891,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2588212,"kb_used_data":2178088,"kb_used_omap":1,"kb_used_meta":410110,"kb_avail":91156940,"statfs":{"total":95995035648,"available":93344706560,"internally_reserved":0,"allocated":2230362112,"data_stored":2229881816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":419953099},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,4,3,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:12:22 2024","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":1.8680000000000001,"15min":1.4319999999999999},"min":{"1min":0.315,"5min":0.192,"15min":0.192},"max":{"1min":4.907,"5min":48.493000000000002,"15min":52.478999999999999},"last":1.7809999999999999},{"interface":"front","average":{"1min":0.78000000000000003,"5min":2.2080000000000002,"15min":1.5389999999999999},"min":{"1min":0.36599999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":2.7519999999999998,"5min":48.738,"15min":52.871000000000002},"last":1.6399999999999999}]},{"osd":1,"last update":"Tue Sep 17 07:12:22 2024","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":2.1960000000000002,"15min":1.141},"min":{"1min":0.376,"5min":0.223,"15min":0.223},"max":{"1min":4.9569999999999999,"5min":73.549000000000007,"15min":73.549000000000007},"last":1.8240000000000001},{"interface":"front","average":{"1min":0.86399999999999999,"5min":2.3069999999999999,"15min":1.1910000000000001},"min":{"1min":0.308,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":3.2970000000000002,"5min":73.116,"15min":73.116},"last":1.669}]},{"osd":2,"last update":"Tue Sep 17 07:12:22 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":1.1930000000000001,"15min":0.96299999999999997},"min":{"1min":0.38400000000000001,"5min":0.245,"15min":0.23899999999999999},"max":{"1min":2.3330000000000002,"5min":22.376000000000001,"15min":52.055999999999997},"last":1.52},{"interface":"front","average":{"1min":0.91200000000000003,"5min":1.1000000000000001,"15min":0.96599999999999997},"min":{"1min":0.34599999999999997,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":4.9340000000000002,"5min":22.298999999999999,"15min":52.064999999999998},"last":1.847}]},{"osd":3,"last update":"Tue Sep 17 07:12:22 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":1.367,"15min":1.155},"min":{"1min":0.35599999999999998,"5min":0.29499999999999998,"15min":0.28799999999999998},"max":{"1min":2.2999999999999998,"5min":22.196000000000002,"15min":51.689},"last":1.5680000000000001},{"interface":"front","average":{"1min":0.92200000000000004,"5min":1.3640000000000001,"15min":1.2490000000000001},"min":{"1min":0.34599999999999997,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":4.8769999999999998,"5min":22.234000000000002,"15min":52.145000000000003},"last":1.867}]},{"osd":4,"last update":"Tue Sep 17 07:12:22 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.75,"15min":0.54500000000000004},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.22800000000000001},"max":{"1min":2.8700000000000001,"5min":4.9950000000000001,"15min":4.9950000000000001},"last":1.6930000000000001},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.82699999999999996,"15min":0.58199999999999996},"min":{"1min":0.28499999999999998,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.968,"5min":6.0549999999999997,"15min":6.0549999999999997},"last":1.5489999999999999}]},{"osd":5,"last update":"Tue Sep 17 07:12:22 2024","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.81799999999999995,"15min":0.57899999999999996},"min":{"1min":0.27100000000000002,"5min":0.253,"15min":0.23699999999999999},"max":{"1min":4.8390000000000004,"5min":5.1360000000000001,"15min":5.1360000000000001},"last":1.885},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.83399999999999996,"15min":0.59099999999999997},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":2.794,"5min":3.8639999999999999,"15min":3.8639999999999999},"last":1.7490000000000001}]},{"osd":6,"last update":"Tue Sep 17 07:12:22 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":1.1140000000000001,"15min":0.71399999999999997},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":2.8479999999999999,"5min":25.363,"15min":25.363},"last":1.7250000000000001},{"interface":"front","average":{"1min":0.76500000000000001,"5min":1.0820000000000001,"15min":0.71199999999999997},"min":{"1min":0.29599999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":2.6299999999999999,"5min":23.571000000000002,"15min":23.571000000000002},"last":1.587}]}]},{"osd":6,"up_from":50,"seq":214748364934,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":880528,"kb_used_data":578692,"kb_used_omap":1,"kb_used_meta":301822,"kb_avail":92864624,"statfs":{"total":95995035648,"available":95093374976,"internally_reserved":0,"allocated":592580608,"data_stored":592072257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":309066183},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:12:04 2024","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":1.667,"15min":1.5880000000000001},"min":{"1min":0.318,"5min":0.185,"15min":0.185},"max":{"1min":5.5010000000000003,"5min":24.106999999999999,"15min":75.323999999999998},"last":0.47299999999999998},{"interface":"front","average":{"1min":1.4630000000000001,"5min":1.6000000000000001,"15min":1.546},"min":{"1min":0.24299999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":5.6459999999999999,"5min":26.193000000000001,"15min":75.168999999999997},"last":0.42799999999999999}]},{"osd":1,"last update":"Tue Sep 17 07:12:04 2024","interfaces":[{"interface":"back","average":{"1min":1.2649999999999999,"5min":1.101,"15min":0.72699999999999998},"min":{"1min":0.28399999999999997,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":5.0549999999999997,"5min":5.0549999999999997,"15min":5.0549999999999997},"last":1.179},{"interface":"front","average":{"1min":0.79600000000000004,"5min":1.0169999999999999,"15min":0.747},"min":{"1min":0.27300000000000002,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":2.9399999999999999,"5min":5.4930000000000003,"15min":5.9530000000000003},"last":0.52500000000000002}]},{"osd":2,"last update":"Tue Sep 17 07:12:04 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.91300000000000003,"15min":0.80200000000000005},"min":{"1min":0.314,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":2.984,"5min":5.1159999999999997,"15min":33.079000000000001},"last":0.504},{"interface":"front","average":{"1min":1.095,"5min":0.876,"15min":0.79100000000000004},"min":{"1min":0.26700000000000002,"5min":0.254,"15min":0.23000000000000001},"max":{"1min":5.734,"5min":5.734,"15min":33.100000000000001},"last":0.40100000000000002}]},{"osd":3,"last update":"Tue Sep 17 07:12:04 2024","interfaces":[{"interface":"back","average":{"1min":1.0289999999999999,"5min":1.2010000000000001,"15min":1.1080000000000001},"min":{"1min":0.31900000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":5.0830000000000002,"5min":16.984000000000002,"15min":52.944000000000003},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.73199999999999998,"5min":1.2729999999999999,"15min":1.1279999999999999},"min":{"1min":0.32600000000000001,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":2.9649999999999999,"5min":16.745999999999999,"15min":52.747},"last":0.54200000000000004}]},{"osd":4,"last update":"Tue Sep 17 07:12:04 2024","interfaces":[{"interface":"back","average":{"1min":1.2050000000000001,"5min":4.444,"15min":1.7609999999999999},"min":{"1min":0.23999999999999999,"5min":0.214,"15min":0.214},"max":{"1min":5.7130000000000001,"5min":260.77199999999999,"15min":260.77199999999999},"last":0.36099999999999999},{"interface":"front","average":{"1min":1.294,"5min":4.4829999999999997,"15min":1.79},"min":{"1min":0.24299999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":6.2140000000000004,"5min":260.69099999999997,"15min":260.69099999999997},"last":0.35299999999999998}]},{"osd":5,"last update":"Tue Sep 17 07:12:04 2024","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":1.159,"15min":0.67900000000000005},"min":{"1min":0.27600000000000002,"5min":0.26200000000000001,"15min":0.25900000000000001},"max":{"1min":6.3860000000000001,"5min":32.042000000000002,"15min":32.042000000000002},"last":0.66100000000000003},{"interface":"front","average":{"1min":1.135,"5min":1.3200000000000001,"15min":0.748},"min":{"1min":0.28999999999999998,"5min":0.251,"15min":0.251},"max":{"1min":5.1379999999999999,"5min":31.852,"15min":31.852},"last":0.29599999999999999}]},{"osd":7,"last update":"Tue Sep 17 07:12:24 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.98299999999999998,"15min":0.67700000000000005},"min":{"1min":0.371,"5min":0.28100000000000003,"15min":0.23699999999999999},"max":{"1min":3.4100000000000001,"5min":7.657,"15min":7.657},"last":0.435},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.94599999999999995,"15min":0.68200000000000005},"min":{"1min":0.35599999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":3.4510000000000001,"5min":7.1890000000000001,"15min":7.1890000000000001},"last":0.46200000000000002}]}]},{"osd":5,"up_from":42,"seq":180388626571,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2337524,"kb_used_data":1945448,"kb_used_omap":1,"kb_used_meta":392062,"kb_avail":91407628,"statfs":{"total":95995035648,"available":93601411072,"internally_reserved":0,"allocated":1992138752,"data_stored":1991658244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":401471942},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,4,5],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:11:37 2024","interfaces":[{"interface":"back","average":{"1min":1.1339999999999999,"5min":1.7050000000000001,"15min":1.4630000000000001},"min":{"1min":0.25900000000000001,"5min":0.23499999999999999,"15min":0.221},"max":{"1min":5.202,"5min":28.727,"15min":49.265000000000001},"last":3.9729999999999999},{"interface":"front","average":{"1min":1.5529999999999999,"5min":1.716,"15min":1.5069999999999999},"min":{"1min":0.30199999999999999,"5min":0.219,"15min":0.219},"max":{"1min":5.6879999999999997,"5min":27.824999999999999,"15min":47.350000000000001},"last":1.929}]},{"osd":1,"last update":"Tue Sep 17 07:11:37 2024","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":1.208,"15min":0.76700000000000002},"min":{"1min":0.29899999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":5.5529999999999999,"5min":20.329000000000001,"15min":20.329000000000001},"last":0.92500000000000004},{"interface":"front","average":{"1min":1.9039999999999999,"5min":1.4359999999999999,"15min":1.151},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":5.2969999999999997,"5min":20.408999999999999,"15min":20.408999999999999},"last":1.4159999999999999}]},{"osd":2,"last update":"Tue Sep 17 07:11:37 2024","interfaces":[{"interface":"back","average":{"1min":1.508,"5min":1.512,"15min":0.96499999999999997},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.252},"max":{"1min":5.6719999999999997,"5min":20.486000000000001,"15min":20.486000000000001},"last":0.39900000000000002},{"interface":"front","average":{"1min":1.1060000000000001,"5min":1.337,"15min":0.877},"min":{"1min":0.29199999999999998,"5min":0.26300000000000001,"15min":0.23200000000000001},"max":{"1min":5.7789999999999999,"5min":23.925999999999998,"15min":23.925999999999998},"last":0.86699999999999999}]},{"osd":3,"last update":"Tue Sep 17 07:11:37 2024","interfaces":[{"interface":"back","average":{"1min":1.2689999999999999,"5min":2.2749999999999999,"15min":1.2170000000000001},"min":{"1min":0.29299999999999998,"5min":0.247,"15min":0.247},"max":{"1min":5.6970000000000001,"5min":68.978999999999999,"15min":68.978999999999999},"last":0.439},{"interface":"front","average":{"1min":1.008,"5min":2.2719999999999998,"15min":1.1870000000000001},"min":{"1min":0.34499999999999997,"5min":0.253,"15min":0.253},"max":{"1min":5.6139999999999999,"5min":69.084000000000003,"15min":69.084000000000003},"last":0.89900000000000002}]},{"osd":4,"last update":"Tue Sep 17 07:11:37 2024","interfaces":[{"interface":"back","average":{"1min":2.415,"5min":2.6699999999999999,"15min":1.153},"min":{"1min":0.27700000000000002,"5min":0.22,"15min":0.22},"max":{"1min":11.835000000000001,"5min":142.327,"15min":142.327},"last":1.96},{"interface":"front","average":{"1min":1.911,"5min":2.5430000000000001,"15min":1.139},"min":{"1min":0.23999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":10.01,"5min":142.13200000000001,"15min":142.13200000000001},"last":0.42099999999999999}]},{"osd":6,"last update":"Tue Sep 17 07:11:56 2024","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.995,"15min":0.61699999999999999},"min":{"1min":0.30399999999999999,"5min":0.23499999999999999,"15min":0.23200000000000001},"max":{"1min":3.637,"5min":8.3149999999999995,"15min":8.3149999999999995},"last":0.83099999999999996},{"interface":"front","average":{"1min":1.1479999999999999,"5min":0.98399999999999999,"15min":0.63900000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.24099999999999999},"max":{"1min":6.4459999999999997,"5min":6.4459999999999997,"15min":6.4459999999999997},"last":0.372}]},{"osd":7,"last update":"Tue Sep 17 07:12:16 2024","interfaces":[{"interface":"back","average":{"1min":1.1930000000000001,"5min":1.0980000000000001,"15min":0.69699999999999995},"min":{"1min":0.34200000000000003,"5min":0.29199999999999998,"15min":0.29099999999999998},"max":{"1min":6.4130000000000003,"5min":6.4130000000000003,"15min":6.4130000000000003},"last":0.95299999999999996},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.94299999999999995,"15min":0.628},"min":{"1min":0.32400000000000001,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":3.8530000000000002,"5min":5.641,"15min":5.641},"last":2.363}]}]},{"osd":4,"up_from":33,"seq":141733920913,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1151416,"kb_used_data":773292,"kb_used_omap":1,"kb_used_meta":378110,"kb_avail":92593736,"statfs":{"total":95995035648,"available":94815985664,"internally_reserved":0,"allocated":791851008,"data_stored":791365504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":387185094},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:12:01 2024","interfaces":[{"interface":"back","average":{"1min":0.96099999999999997,"5min":2.323,"15min":1.8540000000000001},"min":{"1min":0.29799999999999999,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":2.4470000000000001,"5min":58.097999999999999,"15min":74.551000000000002},"last":7.8700000000000001},{"interface":"front","average":{"1min":0.85099999999999998,"5min":2.577,"15min":2.012},"min":{"1min":0.21299999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":3.2280000000000002,"5min":58.383000000000003,"15min":74.778999999999996},"last":0.39600000000000002}]},{"osd":1,"last update":"Tue Sep 17 07:12:01 2024","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":2.7360000000000002,"15min":1.3140000000000001},"min":{"1min":0.251,"5min":0.23300000000000001,"15min":0.23000000000000001},"max":{"1min":6.9729999999999999,"5min":54.914999999999999,"15min":54.914999999999999},"last":7.0999999999999996},{"interface":"front","average":{"1min":0.69599999999999995,"5min":2.3809999999999998,"15min":1.2170000000000001},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.22800000000000001},"max":{"1min":3.073,"5min":53.725000000000001,"15min":53.725000000000001},"last":1.8149999999999999}]},{"osd":2,"last update":"Tue Sep 17 07:12:01 2024","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":0.98499999999999999,"15min":0.94199999999999995},"min":{"1min":0.26900000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":6.8929999999999998,"5min":6.8929999999999998,"15min":49.651000000000003},"last":1.05},{"interface":"front","average":{"1min":1.7110000000000001,"5min":1.1479999999999999,"15min":1.016},"min":{"1min":0.35199999999999998,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":16.003,"5min":16.003,"15min":48.813000000000002},"last":1.2629999999999999}]},{"osd":3,"last update":"Tue Sep 17 07:12:01 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":1.133,"15min":0.98099999999999998},"min":{"1min":0.28699999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":3.2080000000000002,"5min":17.872,"15min":39.042999999999999},"last":1.2310000000000001},{"interface":"front","average":{"1min":1.4330000000000001,"5min":1.2210000000000001,"15min":1.012},"min":{"1min":0.29899999999999999,"5min":0.253,"15min":0.253},"max":{"1min":15.970000000000001,"5min":17.908000000000001,"15min":38.920000000000002},"last":1.075}]},{"osd":5,"last update":"Tue Sep 17 07:12:32 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":1.1160000000000001,"15min":0.65700000000000003},"min":{"1min":0.28100000000000003,"5min":0.222,"15min":0.20799999999999999},"max":{"1min":3.2480000000000002,"5min":11.486000000000001,"15min":11.486000000000001},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.94699999999999995,"5min":1.252,"15min":0.71299999999999997},"min":{"1min":0.32500000000000001,"5min":0.26600000000000001,"15min":0.26100000000000001},"max":{"1min":3.806,"5min":11.781000000000001,"15min":11.781000000000001},"last":0.45400000000000001}]},{"osd":6,"last update":"Tue Sep 17 07:11:58 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":1.349,"15min":0.74399999999999999},"min":{"1min":0.26200000000000001,"5min":0.218,"15min":0.218},"max":{"1min":2.7010000000000001,"5min":47.923999999999999,"15min":47.923999999999999},"last":0.77400000000000002},{"interface":"front","average":{"1min":1.2569999999999999,"5min":1.4490000000000001,"15min":0.77800000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26300000000000001},"max":{"1min":6.2140000000000004,"5min":47.600999999999999,"15min":47.600999999999999},"last":0.56699999999999995}]},{"osd":7,"last update":"Tue Sep 17 07:12:23 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":1.427,"15min":0.82299999999999995},"min":{"1min":0.315,"5min":0.29299999999999998,"15min":0.27600000000000002},"max":{"1min":3.1509999999999998,"5min":48.719000000000001,"15min":48.719000000000001},"last":0.95599999999999996},{"interface":"front","average":{"1min":0.74399999999999999,"5min":1.375,"15min":0.79500000000000004},"min":{"1min":0.314,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":2.8639999999999999,"5min":50.527000000000001,"15min":50.527000000000001},"last":0.81799999999999995}]}]},{"osd":3,"up_from":0,"seq":115964117139,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":18,"seq":77309411481,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1495672,"kb_used_data":999404,"kb_used_omap":1,"kb_used_meta":496254,"kb_avail":92249480,"statfs":{"total":95995035648,"available":94463467520,"internally_reserved":0,"allocated":1023389696,"data_stored":1022905036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":508164550},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:11:38 2024","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":2.2240000000000002,"15min":1.0660000000000001},"min":{"1min":0.19400000000000001,"5min":0.193,"15min":0.17599999999999999},"max":{"1min":11.262,"5min":79.715000000000003,"15min":79.715000000000003},"last":0.17399999999999999},{"interface":"front","average":{"1min":1.048,"5min":2.177,"15min":1.2110000000000001},"min":{"1min":0.217,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":3.141,"5min":79.691999999999993,"15min":79.691999999999993},"last":0.64300000000000002}]},{"osd":1,"last update":"Tue Sep 17 07:11:38 2024","interfaces":[{"interface":"back","average":{"1min":2.718,"5min":2.8380000000000001,"15min":1.2989999999999999},"min":{"1min":0.218,"5min":0.184,"15min":0.184},"max":{"1min":29.434999999999999,"5min":79.659000000000006,"15min":79.659000000000006},"last":0.34300000000000003},{"interface":"front","average":{"1min":2.6509999999999998,"5min":2.8199999999999998,"15min":1.2689999999999999},"min":{"1min":0.28899999999999998,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":29.411999999999999,"5min":80.155000000000001,"15min":80.155000000000001},"last":1.361}]},{"osd":3,"last update":"Tue Sep 17 07:11:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.6259999999999999,"15min":0.93700000000000006},"min":{"1min":0.27100000000000002,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":5.2039999999999997,"5min":23.43,"15min":23.43},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.94999999999999996,"5min":1.546,"15min":1.0349999999999999},"min":{"1min":0.26000000000000001,"5min":0.223,"15min":0.223},"max":{"1min":4.5,"5min":23.408000000000001,"15min":23.408000000000001},"last":0.88}]},{"osd":4,"last update":"Tue Sep 17 07:12:15 2024","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":1.012,"15min":0.81899999999999995},"min":{"1min":0.40799999999999997,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":4.867,"5min":5.944,"15min":5.944},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.94399999999999995,"5min":1.109,"15min":0.81899999999999995},"min":{"1min":0.43099999999999999,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":6.8529999999999998,"5min":6.8529999999999998,"15min":6.8529999999999998},"last":0.45300000000000001}]},{"osd":5,"last update":"Tue Sep 17 07:11:38 2024","interfaces":[{"interface":"back","average":{"1min":1.738,"5min":1.3600000000000001,"15min":0.93500000000000005},"min":{"1min":0.42299999999999999,"5min":0.26100000000000001,"15min":0.249},"max":{"1min":8.8870000000000005,"5min":8.8870000000000005,"15min":8.8870000000000005},"last":0.90700000000000003},{"interface":"front","average":{"1min":1.3320000000000001,"5min":1.181,"15min":0.84099999999999997},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":4.7919999999999998,"5min":7.1790000000000003,"15min":7.1790000000000003},"last":0.73999999999999999}]},{"osd":6,"last update":"Tue Sep 17 07:11:51 2024","interfaces":[{"interface":"back","average":{"1min":1.607,"5min":2.3540000000000001,"15min":1.2689999999999999},"min":{"1min":0.45400000000000001,"5min":0.29999999999999999,"15min":0.28999999999999998},"max":{"1min":6.899,"5min":101.587,"15min":101.587},"last":0.376},{"interface":"front","average":{"1min":1.002,"5min":2.3849999999999998,"15min":1.3140000000000001},"min":{"1min":0.38900000000000001,"5min":0.30599999999999999,"15min":0.28299999999999997},"max":{"1min":2.9940000000000002,"5min":100.696,"15min":100.696},"last":0.93600000000000005}]},{"osd":7,"last update":"Tue Sep 17 07:12:30 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":1.0069999999999999,"15min":0.82699999999999996},"min":{"1min":0.35599999999999998,"5min":0.316,"15min":0.316},"max":{"1min":3.052,"5min":6.9450000000000003,"15min":6.9450000000000003},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.66100000000000003,"5min":1.0469999999999999,"15min":0.94699999999999995},"min":{"1min":0.40999999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.0740000000000001,"5min":6.9219999999999997,"15min":9.2919999999999998},"last":1.2549999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575006,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2130840,"kb_used_data":1792912,"kb_used_omap":1,"kb_used_meta":337918,"kb_avail":91614312,"statfs":{"total":95995035648,"available":93813055488,"internally_reserved":0,"allocated":1835941888,"data_stored":1835421518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":346028486},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:11:56 2024","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":1.667,"15min":1.036},"min":{"1min":0.192,"5min":0.192,"15min":0.155},"max":{"1min":3.6030000000000002,"5min":22.521000000000001,"15min":22.521000000000001},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.70799999999999996,"5min":1.333,"15min":0.95499999999999996},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.17000000000000001},"max":{"1min":4.335,"5min":22.471,"15min":22.471},"last":1.5620000000000001}]},{"osd":2,"last update":"Tue Sep 17 07:12:32 2024","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":1.1910000000000001,"15min":0.93600000000000005},"min":{"1min":0.23999999999999999,"5min":0.183,"15min":0.183},"max":{"1min":3.8929999999999998,"5min":6.2130000000000001,"15min":11.055999999999999},"last":0.88300000000000001},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.96699999999999997,"15min":0.89400000000000002},"min":{"1min":0.245,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":3.9199999999999999,"5min":5.242,"15min":9.3160000000000007},"last":1.6000000000000001}]},{"osd":3,"last update":"Tue Sep 17 07:11:56 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":1.1000000000000001,"15min":0.89600000000000002},"min":{"1min":0.308,"5min":0.223,"15min":0.219},"max":{"1min":4.1269999999999998,"5min":8.5510000000000002,"15min":8.5510000000000002},"last":1.7150000000000001},{"interface":"front","average":{"1min":0.48999999999999999,"5min":1.405,"15min":0.97999999999999998},"min":{"1min":0.246,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":2.1869999999999998,"5min":8.6400000000000006,"15min":19.602},"last":0.53700000000000003}]},{"osd":4,"last update":"Tue Sep 17 07:12:14 2024","interfaces":[{"interface":"back","average":{"1min":1.3069999999999999,"5min":1.278,"15min":0.95699999999999996},"min":{"1min":0.374,"5min":0.25,"15min":0.25},"max":{"1min":4.71,"5min":9.8480000000000008,"15min":9.8480000000000008},"last":1.744},{"interface":"front","average":{"1min":1.085,"5min":1.585,"15min":1.0349999999999999},"min":{"1min":0.32500000000000001,"5min":0.317,"15min":0.307},"max":{"1min":3.4369999999999998,"5min":27.097999999999999,"15min":27.097999999999999},"last":1.931}]},{"osd":5,"last update":"Tue Sep 17 07:12:32 2024","interfaces":[{"interface":"back","average":{"1min":1.0189999999999999,"5min":1.444,"15min":1.034},"min":{"1min":0.33400000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":4.0110000000000001,"5min":16.84,"15min":16.84},"last":2.2389999999999999},{"interface":"front","average":{"1min":0.69799999999999995,"5min":1.5600000000000001,"15min":1.0620000000000001},"min":{"1min":0.29199999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":3.0249999999999999,"5min":16.949999999999999,"15min":16.949999999999999},"last":2.2719999999999998}]},{"osd":6,"last update":"Tue Sep 17 07:11:56 2024","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.5900000000000001,"15min":1.0009999999999999},"min":{"1min":0.34499999999999997,"5min":0.308,"15min":0.308},"max":{"1min":6.8040000000000003,"5min":9.1359999999999992,"15min":9.1359999999999992},"last":1.9650000000000001},{"interface":"front","average":{"1min":1.302,"5min":1.238,"15min":0.92200000000000004},"min":{"1min":0.39800000000000002,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":4.5810000000000004,"5min":7.1130000000000004,"15min":7.1130000000000004},"last":1.7709999999999999}]},{"osd":7,"last update":"Tue Sep 17 07:12:16 2024","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":1.5900000000000001,"15min":1.0529999999999999},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":4.5759999999999996,"5min":9.8179999999999996,"15min":9.8179999999999996},"last":2.827},{"interface":"front","average":{"1min":1.3100000000000001,"5min":1.4870000000000001,"15min":1.026},"min":{"1min":0.35999999999999999,"5min":0.34100000000000003,"15min":0.32900000000000001},"max":{"1min":4.1600000000000001,"5min":6.2590000000000003,"15min":11.007999999999999},"last":0.71199999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738529,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1697480,"kb_used_data":1211068,"kb_used_omap":1,"kb_used_meta":486398,"kb_avail":92047672,"statfs":{"total":95995035648,"available":94256816128,"internally_reserved":0,"allocated":1240133632,"data_stored":1239633332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":498072006},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 17 07:12:05 2024","interfaces":[{"interface":"back","average":{"1min":1.706,"5min":1.2250000000000001,"15min":0.85899999999999999},"min":{"1min":0.19500000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":21.876000000000001,"5min":21.876000000000001,"15min":21.876000000000001},"last":0.78100000000000003},{"interface":"front","average":{"1min":2.242,"5min":1.657,"15min":0.98999999999999999},"min":{"1min":0.28899999999999998,"5min":0.158,"15min":0.158},"max":{"1min":22.521999999999998,"5min":22.521999999999998,"15min":22.521999999999998},"last":0.27100000000000002}]},{"osd":2,"last update":"Tue Sep 17 07:12:33 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":1.095,"15min":1.0860000000000001},"min":{"1min":0.23499999999999999,"5min":0.188,"15min":0.188},"max":{"1min":2.3690000000000002,"5min":6.3799999999999999,"15min":18.643999999999998},"last":0.224},{"interface":"front","average":{"1min":0.67000000000000004,"5min":1.002,"15min":0.96799999999999997},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":2.173,"5min":5.8440000000000003,"15min":17.972999999999999},"last":0.24099999999999999}]},{"osd":3,"last update":"Tue Sep 17 07:12:05 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":1.137,"15min":0.97099999999999997},"min":{"1min":0.255,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":4.4619999999999997,"5min":11.598000000000001,"15min":18.629000000000001},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.95599999999999996,"15min":1.0069999999999999},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.23699999999999999},"max":{"1min":0.96899999999999997,"5min":5.4169999999999998,"15min":22.358000000000001},"last":0.45800000000000002}]},{"osd":4,"last update":"Tue Sep 17 07:12:13 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":1.042,"15min":1.0249999999999999},"min":{"1min":0.39400000000000002,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.611,"5min":19.146999999999998,"15min":19.146999999999998},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.60899999999999999,"5min":1.171,"15min":0.97499999999999998},"min":{"1min":0.28000000000000003,"5min":0.27400000000000002,"15min":0.255},"max":{"1min":1.621,"5min":19.158999999999999,"15min":19.158999999999999},"last":0.314}]},{"osd":5,"last update":"Tue Sep 17 07:11:42 2024","interfaces":[{"interface":"back","average":{"1min":1.5669999999999999,"5min":1.6220000000000001,"15min":1.0469999999999999},"min":{"1min":0.40899999999999997,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":5.0730000000000004,"5min":18.841000000000001,"15min":18.841000000000001},"last":1.129},{"interface":"front","average":{"1min":1.446,"5min":1.653,"15min":1.044},"min":{"1min":0.39000000000000001,"5min":0.30199999999999999,"15min":0.28599999999999998},"max":{"1min":4.3680000000000003,"5min":19.248000000000001,"15min":19.248000000000001},"last":1.0549999999999999}]},{"osd":6,"last update":"Tue Sep 17 07:12:05 2024","interfaces":[{"interface":"back","average":{"1min":1.149,"5min":1.1100000000000001,"15min":0.94699999999999995},"min":{"1min":0.376,"5min":0.31900000000000001,"15min":0.30399999999999999},"max":{"1min":4.3630000000000004,"5min":5.2240000000000002,"15min":18.602},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.81399999999999995,"5min":1.073,"15min":0.996},"min":{"1min":0.42299999999999999,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":2.3170000000000002,"5min":6.149,"15min":18.692},"last":0.97399999999999998}]},{"osd":7,"last update":"Tue Sep 17 07:12:18 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":1.085,"15min":1.3879999999999999},"min":{"1min":0.38400000000000001,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":2.0150000000000001,"5min":4.8490000000000002,"15min":46.183},"last":1.3129999999999999},{"interface":"front","average":{"1min":0.97899999999999998,"5min":1.1299999999999999,"15min":1.1240000000000001},"min":{"1min":0.46600000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":4.4029999999999996,"5min":5.0190000000000001,"15min":45.914999999999999},"last":0.36499999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1265303552,"data_stored":1265303552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1852768256,"data_stored":1852768256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1031798784,"data_stored":1031798784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":809631744,"data_stored":809631744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2038366208,"data_stored":2038366208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":602996736,"data_stored":602996736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2243952640,"data_stored":2243952640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T07:12:43.882 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-17T07:12:43.882 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T07:12:43.883 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_2 2024-09-17T07:12:43.883 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T07:12:43.960 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:12:44.000 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:44.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:44.001 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:12:44.102 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:12:44.141 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:44.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:44.142 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:12:44.215 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 27 16 59189 59173 136.961 96.6875 0.0149574 0.00729652 2024-09-17T07:12:44.242 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:12:44.276 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:44.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:44.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:12:44.378 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:12:44.408 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:44.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:44.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:12:44.510 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:12:44.541 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:44.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:44.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:12:44.585 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:44 smithi100 ceph-mon[29092]: pgmap v472: 148 pgs: 148 active+clean; 3.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 800 B/s rd, 87 MiB/s wr, 1.40k op/s 2024-09-17T07:12:44.585 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:44 smithi100 ceph-mon[39462]: pgmap v472: 148 pgs: 148 active+clean; 3.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 800 B/s rd, 87 MiB/s wr, 1.40k op/s 2024-09-17T07:12:44.643 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:12:44.679 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:44.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:44.680 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:12:44.781 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:12:44.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:44 smithi135 ceph-mon[33238]: pgmap v472: 148 pgs: 148 active+clean; 3.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 800 B/s rd, 87 MiB/s wr, 1.40k op/s 2024-09-17T07:12:44.822 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:44.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:44.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:12:44.924 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:12:44.962 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:44.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:44.963 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:12:45.064 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:12:45.100 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:45.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:45.101 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:12:45.202 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:12:45.215 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 28 15 60700 60685 135.445 94.5 0.0224515 0.00737784 2024-09-17T07:12:45.236 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:45.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:45.237 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:12:45.338 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:12:45.375 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:45.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:45.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:12:45.477 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:12:45.520 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:45.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:45.521 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:12:45.622 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:12:45.657 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:45.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:45.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:12:45.760 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:12:45.782 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:45.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:45.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:12:45.883 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:12:45.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:45.915 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:45.915 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:12:46.016 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:12:46.040 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:46.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:46.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:12:46.142 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:12:46.150 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:12:45 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:12:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:12:46.183 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:46.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:46.184 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:12:46.216 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 29 16 62058 62042 133.699 84.8125 0.00814705 0.00747471 2024-09-17T07:12:46.285 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:12:46.317 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:46.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:46.318 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:12:46.421 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:12:46.461 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:46.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:46.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:12:46.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:46 smithi135 ceph-mon[33238]: osdmap e250: 8 total, 8 up, 7 in 2024-09-17T07:12:46.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:46 smithi135 ceph-mon[33238]: pgmap v474: 148 pgs: 148 active+clean; 3.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 852 B/s rd, 107 MiB/s wr, 1.72k op/s 2024-09-17T07:12:46.551 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:46 smithi100 ceph-mon[29092]: osdmap e250: 8 total, 8 up, 7 in 2024-09-17T07:12:46.551 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:46 smithi100 ceph-mon[29092]: pgmap v474: 148 pgs: 148 active+clean; 3.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 852 B/s rd, 107 MiB/s wr, 1.72k op/s 2024-09-17T07:12:46.551 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:46 smithi100 ceph-mon[39462]: osdmap e250: 8 total, 8 up, 7 in 2024-09-17T07:12:46.551 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:46 smithi100 ceph-mon[39462]: pgmap v474: 148 pgs: 148 active+clean; 3.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 852 B/s rd, 107 MiB/s wr, 1.72k op/s 2024-09-17T07:12:46.564 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:12:46.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:46.602 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:46.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:12:46.703 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:12:46.735 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:46.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:46.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:12:46.837 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:12:46.878 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:46.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:46.879 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:12:46.980 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:12:47.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:47.010 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:47.010 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:12:47.111 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:12:47.140 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:47.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:47.141 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:12:47.215 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 30 16 63555 63539 132.36 93.5625 0.00816095 0.00754982 2024-09-17T07:12:47.243 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:12:47.278 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:47.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:47.280 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:12:47.381 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:12:47.412 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:47.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:47.413 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:12:47.514 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:12:47.549 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:47.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:47.551 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:12:47.652 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:12:47.680 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:47.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:47.681 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:12:47.782 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:12:47.818 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:47.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:47.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:12:47.920 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:12:47.962 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:47.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:47.963 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:12:48.064 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:12:48.102 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:48.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:48.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:12:48.146 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:12:48.180 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:12:48.181 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:12:48.203 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:12:48.204 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:12:48.215 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 31 16 64977 64961 130.957 88.875 0.0107349 0.00763115 2024-09-17T07:12:48.222 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1: {} 2024-09-17T07:12:48.222 INFO:teuthology.orchestra.run.smithi100.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:12:48.256 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:12:48.257 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:12:48.260 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:48.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:48.261 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:12:48.286 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:12:48.287 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:12:48.287 INFO:teuthology.orchestra.run.smithi100.stderr:nodeep-scrub is set 2024-09-17T07:12:48.321 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:12:48.321 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:12:48.356 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:12:48.356 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:12:48.362 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:12:48.391 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:12:48.391 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:12:48.392 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:48 smithi100 ceph-mon[29092]: pgmap v475: 148 pgs: 148 active+clean; 3.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 725 B/s rd, 71 MiB/s wr, 1.14k op/s 2024-09-17T07:12:48.392 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:48 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:12:48.392 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:48 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1966505563' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:12:48.392 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:48 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:12:48.392 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:48 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/3976950264' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:12:48.392 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:48 smithi100 ceph-mon[39462]: pgmap v475: 148 pgs: 148 active+clean; 3.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 725 B/s rd, 71 MiB/s wr, 1.14k op/s 2024-09-17T07:12:48.392 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:48 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:12:48.393 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:48 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1966505563' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:12:48.393 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:48 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:12:48.393 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:48 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/3976950264' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:12:48.396 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:48.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:48.398 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:12:48.425 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:12:48.425 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:12:48.499 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:12:48.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:48.536 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:48.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:12:48.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:48 smithi135 ceph-mon[33238]: pgmap v475: 148 pgs: 148 active+clean; 3.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 725 B/s rd, 71 MiB/s wr, 1.14k op/s 2024-09-17T07:12:48.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:48 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:12:48.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:48 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1966505563' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:12:48.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:48 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:12:48.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:48 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/3976950264' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:12:48.638 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:12:48.668 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:48.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:48.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:12:48.770 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:12:48.800 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:48.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:48.801 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:12:48.902 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:12:48.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:48.938 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:48.938 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:12:49.039 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:12:49.043 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:12:48 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:12:49.069 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:49.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:49.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:12:49.171 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:12:49.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:49.210 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:49.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:12:49.215 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 32 16 66324 66308 129.495 84.1875 0.0103624 0.00771754 2024-09-17T07:12:49.311 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:12:49.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:49.347 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:49.348 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:12:49.448 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:12:49.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:49.486 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:49.486 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:12:49.587 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:12:49.619 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:49.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:49.620 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:12:49.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:49 smithi100 ceph-mon[29092]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:12:49.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:49 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T07:12:49.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:49 smithi100 ceph-mon[29092]: osdmap e251: 8 total, 8 up, 7 in 2024-09-17T07:12:49.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:49 smithi100 ceph-mon[39462]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:12:49.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:49 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T07:12:49.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:49 smithi100 ceph-mon[39462]: osdmap e251: 8 total, 8 up, 7 in 2024-09-17T07:12:49.721 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:12:49.746 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:49.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:49.748 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:12:49.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:49 smithi135 ceph-mon[33238]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:12:49.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:49 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T07:12:49.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:49 smithi135 ceph-mon[33238]: osdmap e251: 8 total, 8 up, 7 in 2024-09-17T07:12:49.849 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:12:49.881 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:49.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:49.883 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:12:49.984 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:12:50.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:50.018 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:50.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:12:50.119 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:12:50.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:50.165 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:50.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:12:50.216 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 33 16 67709 67693 128.194 86.5625 0.0101916 0.00779592 2024-09-17T07:12:50.265 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:12:50.298 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:50.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:50.299 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:12:50.400 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:12:50.439 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:50.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:50.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:12:50.541 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:12:50.576 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:50.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:50.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:12:50.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:50 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:12:50.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:50 smithi100 ceph-mon[29092]: pgmap v477: 148 pgs: 148 active+clean; 3.9 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 1.3 KiB/s rd, 107 MiB/s wr, 1.71k op/s 2024-09-17T07:12:50.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:50 smithi100 ceph-mon[29092]: osdmap e252: 8 total, 8 up, 7 in 2024-09-17T07:12:50.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:50 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:12:50.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:50 smithi100 ceph-mon[39462]: pgmap v477: 148 pgs: 148 active+clean; 3.9 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 1.3 KiB/s rd, 107 MiB/s wr, 1.71k op/s 2024-09-17T07:12:50.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:50 smithi100 ceph-mon[39462]: osdmap e252: 8 total, 8 up, 7 in 2024-09-17T07:12:50.678 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:12:50.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:50.729 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:50.729 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:12:50.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:50 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:12:50.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:50 smithi135 ceph-mon[33238]: pgmap v477: 148 pgs: 148 active+clean; 3.9 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 1.3 KiB/s rd, 107 MiB/s wr, 1.71k op/s 2024-09-17T07:12:50.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:50 smithi135 ceph-mon[33238]: osdmap e252: 8 total, 8 up, 7 in 2024-09-17T07:12:50.830 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:12:50.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:50.866 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:50.866 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:12:50.967 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:12:50.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:50.992 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:50.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:12:51.093 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:12:51.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:51.125 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:51.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:12:51.216 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 34 16 68846 68830 126.514 71.0625 0.00536624 0.00789954 2024-09-17T07:12:51.226 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:12:51.267 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:51.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:51.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:12:51.369 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:12:51.399 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:51.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:51.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:12:51.501 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:12:51.527 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:51.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:51.528 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:12:51.629 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:12:51.664 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:51.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:51.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:12:51.766 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:12:51.802 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:51.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:51.803 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:12:51.904 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:12:51.948 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:51.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:51.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:12:52.050 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:12:52.086 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:52.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:52.087 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:12:52.187 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:12:52.216 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 35 16 70126 70110 125.185 80 0.0167677 0.00798342 2024-09-17T07:12:52.230 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:52.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:52.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:12:52.332 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:12:52.361 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:52.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:52.362 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:12:52.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:52 smithi100 ceph-mon[29092]: pgmap v479: 148 pgs: 148 active+clean; 4.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 846 B/s rd, 75 MiB/s wr, 1.19k op/s 2024-09-17T07:12:52.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:52 smithi100 ceph-mon[39462]: pgmap v479: 148 pgs: 148 active+clean; 4.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 846 B/s rd, 75 MiB/s wr, 1.19k op/s 2024-09-17T07:12:52.463 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:12:52.492 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:52.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:52.493 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:12:52.594 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:12:52.629 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:52.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:52.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:12:52.731 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:12:52.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:52.770 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:52.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:12:52.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:52 smithi135 ceph-mon[33238]: pgmap v479: 148 pgs: 148 active+clean; 4.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 846 B/s rd, 75 MiB/s wr, 1.19k op/s 2024-09-17T07:12:52.810 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T07:12:52.871 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:12:52.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:52.910 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:52.911 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:12:53.011 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-17T07:12:53.011 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:12:53.011 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:12:53.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:53.046 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:53.047 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:12:53.147 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:12:53.179 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:53.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:53.180 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:12:53.216 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 36 16 71508 71492 124.106 86.375 0.0129665 0.00805237 2024-09-17T07:12:53.281 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:12:53.303 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:53.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:53.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:12:53.405 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:12:53.438 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:53.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:53.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:12:53.539 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:12:53.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:53.586 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:53.586 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:12:53.687 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:12:53.722 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:53.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:53.723 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:12:53.824 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:12:53.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:53.863 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:53.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:12:53.964 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:12:53.999 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:54.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:54.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:12:54.101 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:12:54.136 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:54.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:54.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:12:54.216 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 37 16 72740 72724 122.833 77 0.0161035 0.00813558 2024-09-17T07:12:54.237 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:12:54.274 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:54.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:54.275 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:12:54.376 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:12:54.412 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:54.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:54.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:12:54.514 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:12:54.523 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:12:54.548 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:54.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:54.549 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:12:54.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:54 smithi100 ceph-mon[29092]: pgmap v480: 148 pgs: 148 active+clean; 4.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 56 MiB/s wr, 902 op/s 2024-09-17T07:12:54.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:54 smithi100 ceph-mon[39462]: pgmap v480: 148 pgs: 148 active+clean; 4.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 56 MiB/s wr, 902 op/s 2024-09-17T07:12:54.650 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:12:54.692 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:54.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:54.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:12:54.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:54 smithi135 ceph-mon[33238]: pgmap v480: 148 pgs: 148 active+clean; 4.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 56 MiB/s wr, 902 op/s 2024-09-17T07:12:54.794 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:12:54.823 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:54.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:54.824 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:12:54.925 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:12:54.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:54.961 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:54.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:12:55.062 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:12:55.098 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:55.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:55.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:12:55.199 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:12:55.216 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 38 16 74037 74021 121.734 81.0625 0.00903032 0.00821027 2024-09-17T07:12:55.230 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:55.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:55.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:12:55.332 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:12:55.364 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:55.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:55.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:12:55.466 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:12:55.497 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:55.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:55.498 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:12:55.570 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:55 smithi100 ceph-mon[29092]: pgmap v481: 148 pgs: 148 active+clean; 4.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 107 MiB/s wr, 1.71k op/s 2024-09-17T07:12:55.570 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:55 smithi100 ceph-mon[39462]: pgmap v481: 148 pgs: 148 active+clean; 4.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 107 MiB/s wr, 1.71k op/s 2024-09-17T07:12:55.599 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:12:55.605 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-17T07:12:55.628 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:55.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:55.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:12:55.730 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:12:55.768 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:55.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:55.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:12:55.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:55 smithi135 ceph-mon[33238]: pgmap v481: 148 pgs: 148 active+clean; 4.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 107 MiB/s wr, 1.71k op/s 2024-09-17T07:12:55.870 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:12:55.919 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:55.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:55.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:12:56.021 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:12:56.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:56.061 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:56.061 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:12:56.162 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:12:56.218 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 39 16 75385 75369 120.765 84.25 0.0165013 0.00827505 2024-09-17T07:12:56.222 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:56.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:56.223 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:12:56.284 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:12:55 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:12:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:12:56.324 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:12:56.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:56.352 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:56.352 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:12:56.453 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:12:56.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:56.494 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:56.494 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:12:56.595 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:12:56.633 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:56.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:56.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:12:56.735 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:12:56.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:56.775 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:56.776 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:12:56.860 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:12:56.860 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:12:56.878 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:12:56.888 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:12:56.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:56.942 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:56.943 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:12:57.043 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:12:57.083 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:57.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:57.084 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:12:57.111 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T07:12:57.113 INFO:teuthology.orchestra.run.smithi100.stderr:dumped all 2024-09-17T07:12:57.184 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:12:57.218 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:2024-09-17T07:12:57.219131+0000 min lat: 0.00110426 max lat: 0.189997 avg lat: 0.00834365 2024-09-17T07:12:57.219 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-17T07:12:57.219 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 40 16 76690 76674 119.785 81.5625 0.0132527 0.00834365 2024-09-17T07:12:57.221 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:57.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:57.222 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:12:57.323 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:12:57.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:57.355 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:57.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:12:57.456 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:12:57.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:57.491 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:57.491 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:12:57.592 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:12:57.627 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:57.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:57.628 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:12:57.729 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:12:57.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:57.762 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:57.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:12:57.864 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:12:57.903 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:57.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:57.904 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:12:58.005 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:12:58.030 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:58.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:58.032 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:12:58.133 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:12:58.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:58.165 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:58.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:12:58.219 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 41 16 78040 78024 118.921 84.375 0.0131704 0.00840374 2024-09-17T07:12:58.266 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:12:58.298 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:58.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:58.299 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:12:58.400 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:12:58.429 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:58.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:58.430 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:12:58.531 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:12:58.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:58 smithi135 ceph-mon[33238]: from='client.25229 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:12:58.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:58 smithi135 ceph-mon[33238]: pgmap v482: 148 pgs: 148 active+clean; 4.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.1 KiB/s rd, 89 MiB/s wr, 1.43k op/s 2024-09-17T07:12:58.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:58 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:12:58.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:58 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:12:58.561 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:58.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:58.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:12:58.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:58 smithi100 ceph-mon[29092]: from='client.25229 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:12:58.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:58 smithi100 ceph-mon[29092]: pgmap v482: 148 pgs: 148 active+clean; 4.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.1 KiB/s rd, 89 MiB/s wr, 1.43k op/s 2024-09-17T07:12:58.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:58 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:12:58.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:58 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:12:58.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:58 smithi100 ceph-mon[39462]: from='client.25229 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:12:58.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:58 smithi100 ceph-mon[39462]: pgmap v482: 148 pgs: 148 active+clean; 4.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.1 KiB/s rd, 89 MiB/s wr, 1.43k op/s 2024-09-17T07:12:58.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:58 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:12:58.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:58 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:12:58.663 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:12:58.703 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:58.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:58.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:12:58.805 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:12:58.836 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:58.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:58.837 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:12:58.938 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:12:58.968 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:58.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:58.969 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:12:59.043 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:12:58 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:12:59.070 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:12:59.103 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:59.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:59.104 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:12:59.205 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:12:59.219 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 42 16 79407 79391 118.124 85.4375 0.00812252 0.00846086 2024-09-17T07:12:59.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:59.253 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:59.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:12:59.339 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:12:59 smithi100 ceph-mon[29092]: osdmap e253: 8 total, 8 up, 7 in 2024-09-17T07:12:59.340 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:12:59 smithi100 ceph-mon[39462]: osdmap e253: 8 total, 8 up, 7 in 2024-09-17T07:12:59.354 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:12:59.388 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:59.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:59.389 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:12:59.490 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:12:59.528 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:59.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:59.529 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:12:59.630 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:12:59.672 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:59.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:59.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:12:59.774 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:12:59.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:12:59 smithi135 ceph-mon[33238]: osdmap e253: 8 total, 8 up, 7 in 2024-09-17T07:12:59.806 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:12:59.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:59.807 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:12:59.908 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:12:59.939 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:12:59.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:12:59.940 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:13:00.041 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:13:00.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:00.085 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:00.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:13:00.186 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:13:00.219 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 43 16 80712 80696 117.273 81.5625 0.0148489 0.00852183 2024-09-17T07:13:00.221 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:00.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:00.222 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:13:00.323 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:13:00.357 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:00.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:00.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:13:00.459 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:13:00.492 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:00.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:00.493 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:13:00.594 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:13:00.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:00 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:13:00.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:00 smithi100 ceph-mon[29092]: pgmap v484: 148 pgs: 148 active+clean; 4.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.1 KiB/s rd, 79 MiB/s wr, 1.26k op/s 2024-09-17T07:13:00.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:00 smithi100 ceph-mon[29092]: osdmap e254: 8 total, 8 up, 7 in 2024-09-17T07:13:00.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:00 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:13:00.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:00 smithi100 ceph-mon[39462]: pgmap v484: 148 pgs: 148 active+clean; 4.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.1 KiB/s rd, 79 MiB/s wr, 1.26k op/s 2024-09-17T07:13:00.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:00 smithi100 ceph-mon[39462]: osdmap e254: 8 total, 8 up, 7 in 2024-09-17T07:13:00.631 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:00.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:00.633 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:13:00.734 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:13:00.768 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:00.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:00.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:13:00.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:00 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:13:00.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:00 smithi135 ceph-mon[33238]: pgmap v484: 148 pgs: 148 active+clean; 4.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.1 KiB/s rd, 79 MiB/s wr, 1.26k op/s 2024-09-17T07:13:00.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:00 smithi135 ceph-mon[33238]: osdmap e254: 8 total, 8 up, 7 in 2024-09-17T07:13:00.870 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:13:00.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:00.912 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:00.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:13:01.013 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:13:01.052 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:01.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:01.053 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:13:01.154 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:13:01.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:01.188 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:01.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:13:01.219 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 44 16 82013 81997 116.456 81.3125 0.0232004 0.00858164 2024-09-17T07:13:01.289 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:13:01.319 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:01.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:01.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:13:01.343 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:01 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:13:01.343 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:01 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:13:01.343 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:01 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:13:01.343 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:01 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:13:01.421 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:13:01.457 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:01.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:01.458 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:13:01.559 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:13:01.600 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:01.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:01.601 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:13:01.702 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:13:01.735 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:01.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:01.736 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:13:01.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:01 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:13:01.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:01 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:13:01.838 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:13:01.875 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:01.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:01.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:13:01.976 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:13:02.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:02.022 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:02.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:13:02.123 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:13:02.158 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:02.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:02.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:13:02.219 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 45 16 83265 83249 115.607 78.25 0.00975549 0.00864486 2024-09-17T07:13:02.261 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:13:02.299 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:02.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:02.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:13:02.327 INFO:teuthology.orchestra.run.smithi100.stderr:noscrub is unset 2024-09-17T07:13:02.401 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:13:02.434 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:02.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:02.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:13:02.536 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:13:02.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:02.573 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:02.573 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:13:02.619 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:02 smithi100 ceph-mon[29092]: pgmap v486: 148 pgs: 148 active+clean; 4.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 102 MiB/s wr, 1.64k op/s 2024-09-17T07:13:02.619 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:02 smithi100 ceph-mon[29092]: osdmap e255: 8 total, 8 up, 7 in 2024-09-17T07:13:02.619 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:02 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/4203459157' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:13:02.620 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:02 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:13:02.620 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:02 smithi100 ceph-mon[39462]: pgmap v486: 148 pgs: 148 active+clean; 4.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 102 MiB/s wr, 1.64k op/s 2024-09-17T07:13:02.620 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:02 smithi100 ceph-mon[39462]: osdmap e255: 8 total, 8 up, 7 in 2024-09-17T07:13:02.620 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:02 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/4203459157' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:13:02.620 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:02 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:13:02.675 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:13:02.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:02.711 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:02.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:13:02.779 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-17T07:13:02.779 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:13:02.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:02 smithi135 ceph-mon[33238]: pgmap v486: 148 pgs: 148 active+clean; 4.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 102 MiB/s wr, 1.64k op/s 2024-09-17T07:13:02.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:02 smithi135 ceph-mon[33238]: osdmap e255: 8 total, 8 up, 7 in 2024-09-17T07:13:02.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:02 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/4203459157' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:13:02.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:02 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:13:02.812 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:13:02.854 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:02.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:02.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:13:02.956 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:13:03.005 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:03.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:03.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:13:03.107 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:13:03.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:03.143 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:03.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:13:03.219 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 46 16 84511 84495 114.786 77.875 0.0114471 0.00870681 2024-09-17T07:13:03.245 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:13:03.278 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:03.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:03.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:13:03.307 INFO:teuthology.orchestra.run.smithi100.stdout:{"pg_ready":true,"pg_map":{"version":481,"stamp":"2024-09-17T07:12:55.225233+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4717680918,"num_objects":72173,"num_object_clones":0,"num_object_copies":216519,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72173,"num_whiteouts":0,"num_read":1979,"num_read_kb":1825,"num_write":72872,"num_write_kb":4608987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":123,"num_bytes_recovered":1773876,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71806,"ondisk_log_size":71806,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":16616724,"kb_used_data":13690972,"kb_used_omap":10,"kb_used_meta":2925685,"kb_avail":639599340,"statfs":{"total":671965249536,"available":654949724160,"internally_reserved":0,"allocated":14019555328,"data_stored":14015909629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11152,"internal_metadata":2995901552},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":898105344,"num_objects":13704,"num_object_clones":0,"num_object_copies":41112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13704,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":13704,"num_write_kb":877056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002706"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":81,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.296383+0000","last_change":"2024-09-17T07:11:10.474903+0000","last_active":"2024-09-17T07:12:48.296383+0000","last_peered":"2024-09-17T07:12:48.296383+0000","last_clean":"2024-09-17T07:12:48.296383+0000","last_became_active":"2024-09-17T07:09:24.401344+0000","last_became_peered":"2024-09-17T07:09:24.401344+0000","last_unstale":"2024-09-17T07:12:48.296383+0000","last_undegraded":"2024-09-17T07:12:48.296383+0000","last_fullsized":"2024-09-17T07:12:48.296383+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:10.474778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:33.644728+0000","last_clean_scrub_stamp":"2024-09-17T07:11:10.474778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004995724,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":242,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.288494+0000","last_change":"2024-09-17T07:12:20.616904+0000","last_active":"2024-09-17T07:12:48.288494+0000","last_peered":"2024-09-17T07:12:48.288494+0000","last_clean":"2024-09-17T07:12:48.288494+0000","last_became_active":"2024-09-17T07:02:12.006754+0000","last_became_peered":"2024-09-17T07:02:12.006754+0000","last_unstale":"2024-09-17T07:12:48.288494+0000","last_undegraded":"2024-09-17T07:12:48.288494+0000","last_fullsized":"2024-09-17T07:12:48.288494+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:20.616809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:12:20.616809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:28.462459+0000","scrub_duration":0.0024498060000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"242'27","reported_seq":152,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.289636+0000","last_change":"2024-09-17T07:12:24.639592+0000","last_active":"2024-09-17T07:12:48.289636+0000","last_peered":"2024-09-17T07:12:48.289636+0000","last_clean":"2024-09-17T07:12:48.289636+0000","last_became_active":"2024-09-17T07:09:24.402042+0000","last_became_peered":"2024-09-17T07:09:24.402042+0000","last_unstale":"2024-09-17T07:12:48.289636+0000","last_undegraded":"2024-09-17T07:12:48.289636+0000","last_fullsized":"2024-09-17T07:12:48.289636+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'27","last_scrub_stamp":"2024-09-17T07:12:24.639342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:12:24.639342+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:48.768911+0000","scrub_duration":0.0036508320000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":227,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.284325+0000","last_change":"2024-09-17T07:12:12.612582+0000","last_active":"2024-09-17T07:12:48.284325+0000","last_peered":"2024-09-17T07:12:48.284325+0000","last_clean":"2024-09-17T07:12:48.284325+0000","last_became_active":"2024-09-17T07:02:16.014928+0000","last_became_peered":"2024-09-17T07:02:16.014928+0000","last_unstale":"2024-09-17T07:12:48.284325+0000","last_undegraded":"2024-09-17T07:12:48.284325+0000","last_fullsized":"2024-09-17T07:12:48.284325+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:12.612502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:12:12.612502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:17.607137+0000","scrub_duration":0.0012142710000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":81,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.290967+0000","last_change":"2024-09-17T07:10:26.564948+0000","last_active":"2024-09-17T07:12:48.290967+0000","last_peered":"2024-09-17T07:12:48.290967+0000","last_clean":"2024-09-17T07:12:48.290967+0000","last_became_active":"2024-09-17T07:09:24.402568+0000","last_became_peered":"2024-09-17T07:09:24.402568+0000","last_unstale":"2024-09-17T07:12:48.290967+0000","last_undegraded":"2024-09-17T07:12:48.290967+0000","last_fullsized":"2024-09-17T07:12:48.290967+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:10:26.564872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:10:26.564872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0011084269999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"242'15","reported_seq":265,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.284390+0000","last_change":"2024-09-17T07:12:15.549204+0000","last_active":"2024-09-17T07:12:48.284390+0000","last_peered":"2024-09-17T07:12:48.284390+0000","last_clean":"2024-09-17T07:12:48.284390+0000","last_became_active":"2024-09-17T07:02:14.016244+0000","last_became_peered":"2024-09-17T07:02:14.016244+0000","last_unstale":"2024-09-17T07:12:48.284390+0000","last_undegraded":"2024-09-17T07:12:48.284390+0000","last_fullsized":"2024-09-17T07:12:48.284390+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:12:15.549113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:12:15.549113+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:18.330981+0000","scrub_duration":0.0025070779999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":226,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.300800+0000","last_change":"2024-09-17T07:09:25.123274+0000","last_active":"2024-09-17T07:12:48.300800+0000","last_peered":"2024-09-17T07:12:48.300800+0000","last_clean":"2024-09-17T07:12:48.300800+0000","last_became_active":"2024-09-17T07:02:16.014380+0000","last_became_peered":"2024-09-17T07:02:16.014380+0000","last_unstale":"2024-09-17T07:12:48.300800+0000","last_undegraded":"2024-09-17T07:12:48.300800+0000","last_fullsized":"2024-09-17T07:12:48.300800+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:25.123158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:09:25.123158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00166512,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":237,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.285386+0000","last_change":"2024-09-17T07:12:25.491058+0000","last_active":"2024-09-17T07:12:48.285386+0000","last_peered":"2024-09-17T07:12:48.285386+0000","last_clean":"2024-09-17T07:12:48.285386+0000","last_became_active":"2024-09-17T07:02:18.020790+0000","last_became_peered":"2024-09-17T07:02:18.020790+0000","last_unstale":"2024-09-17T07:12:48.285386+0000","last_undegraded":"2024-09-17T07:12:48.285386+0000","last_fullsized":"2024-09-17T07:12:48.285386+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:25.490932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:12:25.490932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:50.379522+0000","scrub_duration":0.0032974060000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":240,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.298038+0000","last_change":"2024-09-17T07:12:11.365357+0000","last_active":"2024-09-17T07:12:48.298038+0000","last_peered":"2024-09-17T07:12:48.298038+0000","last_clean":"2024-09-17T07:12:48.298038+0000","last_became_active":"2024-09-17T07:02:12.011129+0000","last_became_peered":"2024-09-17T07:02:12.011129+0000","last_unstale":"2024-09-17T07:12:48.298038+0000","last_undegraded":"2024-09-17T07:12:48.298038+0000","last_fullsized":"2024-09-17T07:12:48.298038+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:11.365269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:43.060102+0000","last_clean_scrub_stamp":"2024-09-17T07:12:11.365269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:35.406945+0000","scrub_duration":0.0012842960000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"242'7","reported_seq":258,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.285737+0000","last_change":"2024-09-17T07:11:02.661284+0000","last_active":"2024-09-17T07:12:48.285737+0000","last_peered":"2024-09-17T07:12:48.285737+0000","last_clean":"2024-09-17T07:12:48.285737+0000","last_became_active":"2024-09-17T07:09:24.396900+0000","last_became_peered":"2024-09-17T07:09:24.396900+0000","last_unstale":"2024-09-17T07:12:48.285737+0000","last_undegraded":"2024-09-17T07:12:48.285737+0000","last_fullsized":"2024-09-17T07:12:48.285737+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-09-17T07:11:02.661069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:11:02.661069+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0086442429999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":236,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.296835+0000","last_change":"2024-09-17T07:11:24.287105+0000","last_active":"2024-09-17T07:12:48.296835+0000","last_peered":"2024-09-17T07:12:48.296835+0000","last_clean":"2024-09-17T07:12:48.296835+0000","last_became_active":"2024-09-17T07:02:16.015401+0000","last_became_peered":"2024-09-17T07:02:16.015401+0000","last_unstale":"2024-09-17T07:12:48.296835+0000","last_undegraded":"2024-09-17T07:12:48.296835+0000","last_fullsized":"2024-09-17T07:12:48.296835+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:24.286986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:11:24.286986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0067853130000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":244,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.297019+0000","last_change":"2024-09-17T07:11:03.187420+0000","last_active":"2024-09-17T07:12:48.297019+0000","last_peered":"2024-09-17T07:12:48.297019+0000","last_clean":"2024-09-17T07:12:48.297019+0000","last_became_active":"2024-09-17T07:09:24.402616+0000","last_became_peered":"2024-09-17T07:09:24.402616+0000","last_unstale":"2024-09-17T07:12:48.297019+0000","last_undegraded":"2024-09-17T07:12:48.297019+0000","last_fullsized":"2024-09-17T07:12:48.297019+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:03.187280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:11:03.187280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0050576739999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":240,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.297121+0000","last_change":"2024-09-17T07:12:10.356268+0000","last_active":"2024-09-17T07:12:48.297121+0000","last_peered":"2024-09-17T07:12:48.297121+0000","last_clean":"2024-09-17T07:12:48.297121+0000","last_became_active":"2024-09-17T07:02:12.012234+0000","last_became_peered":"2024-09-17T07:02:12.012234+0000","last_unstale":"2024-09-17T07:12:48.297121+0000","last_undegraded":"2024-09-17T07:12:48.297121+0000","last_fullsized":"2024-09-17T07:12:48.297121+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:10.356011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:45.029247+0000","last_clean_scrub_stamp":"2024-09-17T07:12:10.356011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:12.798127+0000","scrub_duration":0.001613686,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"242'15","reported_seq":273,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.285561+0000","last_change":"2024-09-17T07:11:20.847222+0000","last_active":"2024-09-17T07:12:48.285561+0000","last_peered":"2024-09-17T07:12:48.285561+0000","last_clean":"2024-09-17T07:12:48.285561+0000","last_became_active":"2024-09-17T07:09:24.397954+0000","last_became_peered":"2024-09-17T07:09:24.397954+0000","last_unstale":"2024-09-17T07:12:48.285561+0000","last_undegraded":"2024-09-17T07:12:48.285561+0000","last_fullsized":"2024-09-17T07:12:48.285561+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:11:20.847071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:11:20.847071+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0051365669999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":236,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.297450+0000","last_change":"2024-09-17T07:11:27.107585+0000","last_active":"2024-09-17T07:12:48.297450+0000","last_peered":"2024-09-17T07:12:48.297450+0000","last_clean":"2024-09-17T07:12:48.297450+0000","last_became_active":"2024-09-17T07:02:16.023774+0000","last_became_peered":"2024-09-17T07:02:16.023774+0000","last_unstale":"2024-09-17T07:12:48.297450+0000","last_undegraded":"2024-09-17T07:12:48.297450+0000","last_fullsized":"2024-09-17T07:12:48.297450+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:27.107400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:28.096266+0000","last_clean_scrub_stamp":"2024-09-17T07:11:27.107400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:54.412659+0000","scrub_duration":0.0081054549999999993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":225,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.287970+0000","last_change":"2024-09-17T07:11:06.419278+0000","last_active":"2024-09-17T07:12:48.287970+0000","last_peered":"2024-09-17T07:12:48.287970+0000","last_clean":"2024-09-17T07:12:48.287970+0000","last_became_active":"2024-09-17T07:02:18.029178+0000","last_became_peered":"2024-09-17T07:02:18.029178+0000","last_unstale":"2024-09-17T07:12:48.287970+0000","last_undegraded":"2024-09-17T07:12:48.287970+0000","last_fullsized":"2024-09-17T07:12:48.287970+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:06.419134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:11:06.419134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0086498219999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.0","version":"252'4479","reported_seq":4504,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-17T07:12:53.621139+0000","last_change":"2024-09-17T07:12:05.418073+0000","last_active":"2024-09-17T07:12:53.621139+0000","last_peered":"2024-09-17T07:12:53.621139+0000","last_clean":"2024-09-17T07:12:53.621139+0000","last_became_active":"2024-09-17T07:12:05.417841+0000","last_became_peered":"2024-09-17T07:12:05.417841+0000","last_unstale":"2024-09-17T07:12:53.621139+0000","last_undegraded":"2024-09-17T07:12:53.621139+0000","last_fullsized":"2024-09-17T07:12:53.621139+0000","mapping_epoch":236,"log_start":"242'100","ondisk_log_start":"242'100","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_clean_scrub_stamp":"2024-09-17T07:12:04.285115+0000","objects_scrubbed":0,"log_size":4379,"log_dups_size":0,"ondisk_log_size":4379,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:09.864746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":293535744,"num_objects":4479,"num_object_clones":0,"num_object_copies":13437,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4479,"num_write_kb":286656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"242'39","reported_seq":297,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.285955+0000","last_change":"2024-09-17T07:12:19.487929+0000","last_active":"2024-09-17T07:12:48.285955+0000","last_peered":"2024-09-17T07:12:48.285955+0000","last_clean":"2024-09-17T07:12:48.285955+0000","last_became_active":"2024-09-17T07:02:14.016270+0000","last_became_peered":"2024-09-17T07:02:14.016270+0000","last_unstale":"2024-09-17T07:12:48.285955+0000","last_undegraded":"2024-09-17T07:12:48.285955+0000","last_fullsized":"2024-09-17T07:12:48.285955+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'27","last_scrub_stamp":"2024-09-17T07:12:19.487754+0000","last_deep_scrub":"95'25","last_deep_scrub_stamp":"2024-09-17T07:04:09.450039+0000","last_clean_scrub_stamp":"2024-09-17T07:12:19.487754+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:22.189722+0000","scrub_duration":0.0086853929999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":242,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.296945+0000","last_change":"2024-09-17T07:11:13.480849+0000","last_active":"2024-09-17T07:12:48.296945+0000","last_peered":"2024-09-17T07:12:48.296945+0000","last_clean":"2024-09-17T07:12:48.296945+0000","last_became_active":"2024-09-17T07:02:12.011258+0000","last_became_peered":"2024-09-17T07:02:12.011258+0000","last_unstale":"2024-09-17T07:12:48.296945+0000","last_undegraded":"2024-09-17T07:12:48.296945+0000","last_fullsized":"2024-09-17T07:12:48.296945+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:13.480727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:11:13.480727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0080456680000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":226,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.301339+0000","last_change":"2024-09-17T07:11:00.256708+0000","last_active":"2024-09-17T07:12:48.301339+0000","last_peered":"2024-09-17T07:12:48.301339+0000","last_clean":"2024-09-17T07:12:48.301339+0000","last_became_active":"2024-09-17T07:02:16.018147+0000","last_became_peered":"2024-09-17T07:02:16.018147+0000","last_unstale":"2024-09-17T07:12:48.301339+0000","last_undegraded":"2024-09-17T07:12:48.301339+0000","last_fullsized":"2024-09-17T07:12:48.301339+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:00.256595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:11:00.256595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040770219999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":84,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.286128+0000","last_change":"2024-09-17T07:12:21.520698+0000","last_active":"2024-09-17T07:12:48.286128+0000","last_peered":"2024-09-17T07:12:48.286128+0000","last_clean":"2024-09-17T07:12:48.286128+0000","last_became_active":"2024-09-17T07:09:24.396334+0000","last_became_peered":"2024-09-17T07:09:24.396334+0000","last_unstale":"2024-09-17T07:12:48.286128+0000","last_undegraded":"2024-09-17T07:12:48.286128+0000","last_fullsized":"2024-09-17T07:12:48.286128+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:21.520586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:42.984088+0000","last_clean_scrub_stamp":"2024-09-17T07:12:21.520586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:37.147324+0000","scrub_duration":0.0022027779999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1","version":"252'4410","reported_seq":4435,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-17T07:12:53.267910+0000","last_change":"2024-09-17T07:12:05.172795+0000","last_active":"2024-09-17T07:12:53.267910+0000","last_peered":"2024-09-17T07:12:53.267910+0000","last_clean":"2024-09-17T07:12:53.267910+0000","last_became_active":"2024-09-17T07:12:05.172332+0000","last_became_peered":"2024-09-17T07:12:05.172332+0000","last_unstale":"2024-09-17T07:12:53.267910+0000","last_undegraded":"2024-09-17T07:12:53.267910+0000","last_fullsized":"2024-09-17T07:12:53.267910+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_clean_scrub_stamp":"2024-09-17T07:12:04.285115+0000","objects_scrubbed":0,"log_size":4410,"log_dups_size":0,"ondisk_log_size":4410,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:27.965693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":289013760,"num_objects":4410,"num_object_clones":0,"num_object_copies":13230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4410,"num_write_kb":282240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"242'25","reported_seq":296,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.287223+0000","last_change":"2024-09-17T07:11:24.918471+0000","last_active":"2024-09-17T07:12:48.287223+0000","last_peered":"2024-09-17T07:12:48.287223+0000","last_clean":"2024-09-17T07:12:48.287223+0000","last_became_active":"2024-09-17T07:09:24.399287+0000","last_became_peered":"2024-09-17T07:09:24.399287+0000","last_unstale":"2024-09-17T07:12:48.287223+0000","last_undegraded":"2024-09-17T07:12:48.287223+0000","last_fullsized":"2024-09-17T07:12:48.287223+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'15","last_scrub_stamp":"2024-09-17T07:11:24.918191+0000","last_deep_scrub":"176'15","last_deep_scrub_stamp":"2024-09-17T07:08:19.633791+0000","last_clean_scrub_stamp":"2024-09-17T07:11:24.918191+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:52.956372+0000","scrub_duration":0.0091725629999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":249,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.301059+0000","last_change":"2024-09-17T07:12:09.254527+0000","last_active":"2024-09-17T07:12:48.301059+0000","last_peered":"2024-09-17T07:12:48.301059+0000","last_clean":"2024-09-17T07:12:48.301059+0000","last_became_active":"2024-09-17T07:09:24.406973+0000","last_became_peered":"2024-09-17T07:09:24.406973+0000","last_unstale":"2024-09-17T07:12:48.301059+0000","last_undegraded":"2024-09-17T07:12:48.301059+0000","last_fullsized":"2024-09-17T07:12:48.301059+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:09.254452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:12:09.254452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:36.038881+0000","scrub_duration":0.0013322150000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"98'9","reported_seq":324,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-17T07:12:50.181796+0000","last_change":"2024-09-17T07:12:00.407366+0000","last_active":"2024-09-17T07:12:50.181796+0000","last_peered":"2024-09-17T07:12:50.181796+0000","last_clean":"2024-09-17T07:12:50.181796+0000","last_became_active":"2024-09-17T07:04:15.110797+0000","last_became_peered":"2024-09-17T07:04:15.110797+0000","last_unstale":"2024-09-17T07:12:50.181796+0000","last_undegraded":"2024-09-17T07:12:50.181796+0000","last_fullsized":"2024-09-17T07:12:50.181796+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-09-17T07:12:00.407262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:12:00.407262+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:14.851416+0000","scrub_duration":0.002024489,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":236,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.286622+0000","last_change":"2024-09-17T07:11:29.774296+0000","last_active":"2024-09-17T07:12:48.286622+0000","last_peered":"2024-09-17T07:12:48.286622+0000","last_clean":"2024-09-17T07:12:48.286622+0000","last_became_active":"2024-09-17T07:02:18.017460+0000","last_became_peered":"2024-09-17T07:02:18.017460+0000","last_unstale":"2024-09-17T07:12:48.286622+0000","last_undegraded":"2024-09-17T07:12:48.286622+0000","last_fullsized":"2024-09-17T07:12:48.286622+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:29.774144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:11:29.774144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.013832928,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"252'4612","reported_seq":4637,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-17T07:12:53.546200+0000","last_change":"2024-09-17T07:12:05.416444+0000","last_active":"2024-09-17T07:12:53.546200+0000","last_peered":"2024-09-17T07:12:53.546200+0000","last_clean":"2024-09-17T07:12:53.546200+0000","last_became_active":"2024-09-17T07:12:05.416186+0000","last_became_peered":"2024-09-17T07:12:05.416186+0000","last_unstale":"2024-09-17T07:12:53.546200+0000","last_undegraded":"2024-09-17T07:12:53.546200+0000","last_fullsized":"2024-09-17T07:12:53.546200+0000","mapping_epoch":236,"log_start":"242'200","ondisk_log_start":"242'200","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_clean_scrub_stamp":"2024-09-17T07:12:04.285115+0000","objects_scrubbed":0,"log_size":4412,"log_dups_size":0,"ondisk_log_size":4412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:25.741377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":302252032,"num_objects":4612,"num_object_clones":0,"num_object_copies":13836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4612,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4612,"num_write_kb":295168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"242'27","reported_seq":306,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.309367+0000","last_change":"2024-09-17T07:12:13.306460+0000","last_active":"2024-09-17T07:12:48.309367+0000","last_peered":"2024-09-17T07:12:48.309367+0000","last_clean":"2024-09-17T07:12:48.309367+0000","last_became_active":"2024-09-17T07:09:24.407311+0000","last_became_peered":"2024-09-17T07:09:24.407311+0000","last_unstale":"2024-09-17T07:12:48.309367+0000","last_undegraded":"2024-09-17T07:12:48.309367+0000","last_fullsized":"2024-09-17T07:12:48.309367+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-09-17T07:12:13.306349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:12:13.306349+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:26.012489+0000","scrub_duration":0.0021853620000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":239,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.300932+0000","last_change":"2024-09-17T07:09:24.405992+0000","last_active":"2024-09-17T07:12:48.300932+0000","last_peered":"2024-09-17T07:12:48.300932+0000","last_clean":"2024-09-17T07:12:48.300932+0000","last_became_active":"2024-09-17T07:09:24.405655+0000","last_became_peered":"2024-09-17T07:09:24.405655+0000","last_unstale":"2024-09-17T07:12:48.300932+0000","last_undegraded":"2024-09-17T07:12:48.300932+0000","last_fullsized":"2024-09-17T07:12:48.300932+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:20.182256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:09:20.182256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034116369999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"75'8","reported_seq":363,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-17T07:12:50.185894+0000","last_change":"2024-09-17T07:11:25.896643+0000","last_active":"2024-09-17T07:12:50.185894+0000","last_peered":"2024-09-17T07:12:50.185894+0000","last_clean":"2024-09-17T07:12:50.185894+0000","last_became_active":"2024-09-17T07:02:16.021762+0000","last_became_peered":"2024-09-17T07:02:16.021762+0000","last_unstale":"2024-09-17T07:12:50.185894+0000","last_undegraded":"2024-09-17T07:12:50.185894+0000","last_fullsized":"2024-09-17T07:12:50.185894+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:11:25.896517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:11:25.896517+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:53.871973+0000","scrub_duration":0.01413413,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":234,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.298967+0000","last_change":"2024-09-17T07:11:30.065694+0000","last_active":"2024-09-17T07:12:48.298967+0000","last_peered":"2024-09-17T07:12:48.298967+0000","last_clean":"2024-09-17T07:12:48.298967+0000","last_became_active":"2024-09-17T07:02:18.025332+0000","last_became_peered":"2024-09-17T07:02:18.025332+0000","last_unstale":"2024-09-17T07:12:48.298967+0000","last_undegraded":"2024-09-17T07:12:48.298967+0000","last_fullsized":"2024-09-17T07:12:48.298967+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:30.065556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:11:30.065556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046561759999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"252'4436","reported_seq":4460,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-17T07:12:53.384189+0000","last_change":"2024-09-17T07:12:05.174275+0000","last_active":"2024-09-17T07:12:53.384189+0000","last_peered":"2024-09-17T07:12:53.384189+0000","last_clean":"2024-09-17T07:12:53.384189+0000","last_became_active":"2024-09-17T07:12:05.174017+0000","last_became_peered":"2024-09-17T07:12:05.174017+0000","last_unstale":"2024-09-17T07:12:53.384189+0000","last_undegraded":"2024-09-17T07:12:53.384189+0000","last_fullsized":"2024-09-17T07:12:53.384189+0000","mapping_epoch":236,"log_start":"242'200","ondisk_log_start":"242'200","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_clean_scrub_stamp":"2024-09-17T07:12:04.285115+0000","objects_scrubbed":0,"log_size":4236,"log_dups_size":0,"ondisk_log_size":4236,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:30.641723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":290717696,"num_objects":4436,"num_object_clones":0,"num_object_copies":13308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4436,"num_write_kb":283904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"242'19","reported_seq":271,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.286834+0000","last_change":"2024-09-17T07:11:27.816034+0000","last_active":"2024-09-17T07:12:48.286834+0000","last_peered":"2024-09-17T07:12:48.286834+0000","last_clean":"2024-09-17T07:12:48.286834+0000","last_became_active":"2024-09-17T07:02:14.010227+0000","last_became_peered":"2024-09-17T07:02:14.010227+0000","last_unstale":"2024-09-17T07:12:48.286834+0000","last_undegraded":"2024-09-17T07:12:48.286834+0000","last_fullsized":"2024-09-17T07:12:48.286834+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-09-17T07:11:27.815628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:11:27.815628+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0098845579999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":241,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.297055+0000","last_change":"2024-09-17T07:11:16.421038+0000","last_active":"2024-09-17T07:12:48.297055+0000","last_peered":"2024-09-17T07:12:48.297055+0000","last_clean":"2024-09-17T07:12:48.297055+0000","last_became_active":"2024-09-17T07:02:12.013182+0000","last_became_peered":"2024-09-17T07:02:12.013182+0000","last_unstale":"2024-09-17T07:12:48.297055+0000","last_undegraded":"2024-09-17T07:12:48.297055+0000","last_fullsized":"2024-09-17T07:12:48.297055+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:16.420367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:11:16.420367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0051482619999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":247,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.298307+0000","last_change":"2024-09-17T07:12:24.732629+0000","last_active":"2024-09-17T07:12:48.298307+0000","last_peered":"2024-09-17T07:12:48.298307+0000","last_clean":"2024-09-17T07:12:48.298307+0000","last_became_active":"2024-09-17T07:02:16.020224+0000","last_became_peered":"2024-09-17T07:02:16.020224+0000","last_unstale":"2024-09-17T07:12:48.298307+0000","last_undegraded":"2024-09-17T07:12:48.298307+0000","last_fullsized":"2024-09-17T07:12:48.298307+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:24.732549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:12:24.732549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:26.236089+0000","scrub_duration":0.0013739589999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":91,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.292789+0000","last_change":"2024-09-17T07:12:12.704448+0000","last_active":"2024-09-17T07:12:48.292789+0000","last_peered":"2024-09-17T07:12:48.292789+0000","last_clean":"2024-09-17T07:12:48.292789+0000","last_became_active":"2024-09-17T07:09:24.408109+0000","last_became_peered":"2024-09-17T07:09:24.408109+0000","last_unstale":"2024-09-17T07:12:48.292789+0000","last_undegraded":"2024-09-17T07:12:48.292789+0000","last_fullsized":"2024-09-17T07:12:48.292789+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:12.704363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:12:12.704363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:17.195681+0000","scrub_duration":0.0011153510000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"252'4540","reported_seq":4565,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-17T07:12:53.219120+0000","last_change":"2024-09-17T07:12:05.174268+0000","last_active":"2024-09-17T07:12:53.219120+0000","last_peered":"2024-09-17T07:12:53.219120+0000","last_clean":"2024-09-17T07:12:53.219120+0000","last_became_active":"2024-09-17T07:12:05.173802+0000","last_became_peered":"2024-09-17T07:12:05.173802+0000","last_unstale":"2024-09-17T07:12:53.219120+0000","last_undegraded":"2024-09-17T07:12:53.219120+0000","last_fullsized":"2024-09-17T07:12:53.219120+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_clean_scrub_stamp":"2024-09-17T07:12:04.285115+0000","objects_scrubbed":0,"log_size":4540,"log_dups_size":0,"ondisk_log_size":4540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:14.585969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":297533440,"num_objects":4540,"num_object_clones":0,"num_object_copies":13620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4540,"num_write_kb":290560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"242'28","reported_seq":272,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.287137+0000","last_change":"2024-09-17T07:12:00.638759+0000","last_active":"2024-09-17T07:12:48.287137+0000","last_peered":"2024-09-17T07:12:48.287137+0000","last_clean":"2024-09-17T07:12:48.287137+0000","last_became_active":"2024-09-17T07:02:14.007810+0000","last_became_peered":"2024-09-17T07:02:14.007810+0000","last_unstale":"2024-09-17T07:12:48.287137+0000","last_undegraded":"2024-09-17T07:12:48.287137+0000","last_fullsized":"2024-09-17T07:12:48.287137+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'14","last_scrub_stamp":"2024-09-17T07:12:00.638644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:12:00.638644+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:20.604094+0000","scrub_duration":0.002384817,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":262,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.293300+0000","last_change":"2024-09-17T07:12:02.695870+0000","last_active":"2024-09-17T07:12:48.293300+0000","last_peered":"2024-09-17T07:12:48.293300+0000","last_clean":"2024-09-17T07:12:48.293300+0000","last_became_active":"2024-09-17T07:02:12.011766+0000","last_became_peered":"2024-09-17T07:02:12.011766+0000","last_unstale":"2024-09-17T07:12:48.293300+0000","last_undegraded":"2024-09-17T07:12:48.293300+0000","last_fullsized":"2024-09-17T07:12:48.293300+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:02.695765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:09:23.181028+0000","last_clean_scrub_stamp":"2024-09-17T07:12:02.695765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:04.960155+0000","scrub_duration":0.0016435110000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"196'9","reported_seq":135,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-17T07:12:50.188019+0000","last_change":"2024-09-17T07:09:24.427192+0000","last_active":"2024-09-17T07:12:50.188019+0000","last_peered":"2024-09-17T07:12:50.188019+0000","last_clean":"2024-09-17T07:12:50.188019+0000","last_became_active":"2024-09-17T07:09:24.406056+0000","last_became_peered":"2024-09-17T07:09:24.406056+0000","last_unstale":"2024-09-17T07:12:50.188019+0000","last_undegraded":"2024-09-17T07:12:50.188019+0000","last_fullsized":"2024-09-17T07:12:50.188019+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:09:14.776794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:09:14.776794+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":246,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.405915+0000","last_change":"2024-09-17T07:12:25.194692+0000","last_active":"2024-09-17T07:12:48.405915+0000","last_peered":"2024-09-17T07:12:48.405915+0000","last_clean":"2024-09-17T07:12:48.405915+0000","last_became_active":"2024-09-17T07:02:18.364981+0000","last_became_peered":"2024-09-17T07:02:18.364981+0000","last_unstale":"2024-09-17T07:12:48.405915+0000","last_undegraded":"2024-09-17T07:12:48.405915+0000","last_fullsized":"2024-09-17T07:12:48.405915+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:25.194515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:10:33.727472+0000","last_clean_scrub_stamp":"2024-09-17T07:12:25.194515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:40.081616+0000","scrub_duration":0.002844943,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.d","version":"252'4485","reported_seq":4508,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-17T07:12:53.387579+0000","last_change":"2024-09-17T07:12:05.177946+0000","last_active":"2024-09-17T07:12:53.387579+0000","last_peered":"2024-09-17T07:12:53.387579+0000","last_clean":"2024-09-17T07:12:53.387579+0000","last_became_active":"2024-09-17T07:12:05.177682+0000","last_became_peered":"2024-09-17T07:12:05.177682+0000","last_unstale":"2024-09-17T07:12:53.387579+0000","last_undegraded":"2024-09-17T07:12:53.387579+0000","last_fullsized":"2024-09-17T07:12:53.387579+0000","mapping_epoch":236,"log_start":"242'300","ondisk_log_start":"242'300","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_clean_scrub_stamp":"2024-09-17T07:12:04.285115+0000","objects_scrubbed":0,"log_size":4185,"log_dups_size":0,"ondisk_log_size":4185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:19.366886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":293928960,"num_objects":4485,"num_object_clones":0,"num_object_copies":13455,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4485,"num_write_kb":287040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"242'30","reported_seq":160,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.296826+0000","last_change":"2024-09-17T07:11:19.363446+0000","last_active":"2024-09-17T07:12:48.296826+0000","last_peered":"2024-09-17T07:12:48.296826+0000","last_clean":"2024-09-17T07:12:48.296826+0000","last_became_active":"2024-09-17T07:09:24.403240+0000","last_became_peered":"2024-09-17T07:09:24.403240+0000","last_unstale":"2024-09-17T07:12:48.296826+0000","last_undegraded":"2024-09-17T07:12:48.296826+0000","last_fullsized":"2024-09-17T07:12:48.296826+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'18","last_scrub_stamp":"2024-09-17T07:11:19.363076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:11:19.363076+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010644855,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":240,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.308021+0000","last_change":"2024-09-17T07:11:07.244633+0000","last_active":"2024-09-17T07:12:48.308021+0000","last_peered":"2024-09-17T07:12:48.308021+0000","last_clean":"2024-09-17T07:12:48.308021+0000","last_became_active":"2024-09-17T07:02:12.005687+0000","last_became_peered":"2024-09-17T07:02:12.005687+0000","last_unstale":"2024-09-17T07:12:48.308021+0000","last_undegraded":"2024-09-17T07:12:48.308021+0000","last_fullsized":"2024-09-17T07:12:48.308021+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:07.244521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:11:07.244521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0062381399999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":237,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.286990+0000","last_change":"2024-09-17T07:11:26.837485+0000","last_active":"2024-09-17T07:12:48.286990+0000","last_peered":"2024-09-17T07:12:48.286990+0000","last_clean":"2024-09-17T07:12:48.286990+0000","last_became_active":"2024-09-17T07:09:24.393375+0000","last_became_peered":"2024-09-17T07:09:24.393375+0000","last_unstale":"2024-09-17T07:12:48.286990+0000","last_undegraded":"2024-09-17T07:12:48.286990+0000","last_fullsized":"2024-09-17T07:12:48.286990+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:26.837370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:11:26.837370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0042759720000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":246,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.285515+0000","last_change":"2024-09-17T07:12:18.466015+0000","last_active":"2024-09-17T07:12:48.285515+0000","last_peered":"2024-09-17T07:12:48.285515+0000","last_clean":"2024-09-17T07:12:48.285515+0000","last_became_active":"2024-09-17T07:09:24.398659+0000","last_became_peered":"2024-09-17T07:09:24.398659+0000","last_unstale":"2024-09-17T07:12:48.285515+0000","last_undegraded":"2024-09-17T07:12:48.285515+0000","last_fullsized":"2024-09-17T07:12:48.285515+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:18.465901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:12:18.465901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:47.260148+0000","scrub_duration":0.0024831419999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.e","version":"252'4419","reported_seq":4443,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-17T07:12:53.399133+0000","last_change":"2024-09-17T07:12:05.175609+0000","last_active":"2024-09-17T07:12:53.399133+0000","last_peered":"2024-09-17T07:12:53.399133+0000","last_clean":"2024-09-17T07:12:53.399133+0000","last_became_active":"2024-09-17T07:12:05.175318+0000","last_became_peered":"2024-09-17T07:12:05.175318+0000","last_unstale":"2024-09-17T07:12:53.399133+0000","last_undegraded":"2024-09-17T07:12:53.399133+0000","last_fullsized":"2024-09-17T07:12:53.399133+0000","mapping_epoch":236,"log_start":"242'200","ondisk_log_start":"242'200","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_clean_scrub_stamp":"2024-09-17T07:12:04.285115+0000","objects_scrubbed":0,"log_size":4219,"log_dups_size":0,"ondisk_log_size":4219,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:21.393438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":289603584,"num_objects":4419,"num_object_clones":0,"num_object_copies":13257,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4419,"num_write_kb":282816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"242'29","reported_seq":281,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.288386+0000","last_change":"2024-09-17T07:11:03.438378+0000","last_active":"2024-09-17T07:12:48.288386+0000","last_peered":"2024-09-17T07:12:48.288386+0000","last_clean":"2024-09-17T07:12:48.288386+0000","last_became_active":"2024-09-17T07:02:14.027389+0000","last_became_peered":"2024-09-17T07:02:14.027389+0000","last_unstale":"2024-09-17T07:12:48.288386+0000","last_undegraded":"2024-09-17T07:12:48.288386+0000","last_fullsized":"2024-09-17T07:12:48.288386+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-09-17T07:11:03.438281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:11:03.438281+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.006304324,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":242,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.298107+0000","last_change":"2024-09-17T07:11:15.424520+0000","last_active":"2024-09-17T07:12:48.298107+0000","last_peered":"2024-09-17T07:12:48.298107+0000","last_clean":"2024-09-17T07:12:48.298107+0000","last_became_active":"2024-09-17T07:02:12.011917+0000","last_became_peered":"2024-09-17T07:02:12.011917+0000","last_unstale":"2024-09-17T07:12:48.298107+0000","last_undegraded":"2024-09-17T07:12:48.298107+0000","last_fullsized":"2024-09-17T07:12:48.298107+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:15.424430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:24.072799+0000","last_clean_scrub_stamp":"2024-09-17T07:11:15.424430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0057333330000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":238,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.290467+0000","last_change":"2024-09-17T07:11:17.583846+0000","last_active":"2024-09-17T07:12:48.290467+0000","last_peered":"2024-09-17T07:12:48.290467+0000","last_clean":"2024-09-17T07:12:48.290467+0000","last_became_active":"2024-09-17T07:02:16.016199+0000","last_became_peered":"2024-09-17T07:02:16.016199+0000","last_unstale":"2024-09-17T07:12:48.290467+0000","last_undegraded":"2024-09-17T07:12:48.290467+0000","last_fullsized":"2024-09-17T07:12:48.290467+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:17.583712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:11:17.583712+0000","last_clean_scrub_stamp":"2024-09-17T07:11:17.583712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0064424299999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":235,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.300718+0000","last_change":"2024-09-17T07:11:28.050865+0000","last_active":"2024-09-17T07:12:48.300718+0000","last_peered":"2024-09-17T07:12:48.300718+0000","last_clean":"2024-09-17T07:12:48.300718+0000","last_became_active":"2024-09-17T07:02:18.364835+0000","last_became_peered":"2024-09-17T07:02:18.364835+0000","last_unstale":"2024-09-17T07:12:48.300718+0000","last_undegraded":"2024-09-17T07:12:48.300718+0000","last_fullsized":"2024-09-17T07:12:48.300718+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:28.050750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:11:28.050750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003023796,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.f","version":"252'4504","reported_seq":4528,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-17T07:12:53.256300+0000","last_change":"2024-09-17T07:12:05.418098+0000","last_active":"2024-09-17T07:12:53.256300+0000","last_peered":"2024-09-17T07:12:53.256300+0000","last_clean":"2024-09-17T07:12:53.256300+0000","last_became_active":"2024-09-17T07:12:05.417843+0000","last_became_peered":"2024-09-17T07:12:05.417843+0000","last_unstale":"2024-09-17T07:12:53.256300+0000","last_undegraded":"2024-09-17T07:12:53.256300+0000","last_fullsized":"2024-09-17T07:12:53.256300+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_clean_scrub_stamp":"2024-09-17T07:12:04.285115+0000","objects_scrubbed":0,"log_size":4504,"log_dups_size":0,"ondisk_log_size":4504,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:32.459979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":295174144,"num_objects":4504,"num_object_clones":0,"num_object_copies":13512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4504,"num_write_kb":288256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"242'25","reported_seq":274,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.300938+0000","last_change":"2024-09-17T07:12:19.410879+0000","last_active":"2024-09-17T07:12:48.300938+0000","last_peered":"2024-09-17T07:12:48.300938+0000","last_clean":"2024-09-17T07:12:48.300938+0000","last_became_active":"2024-09-17T07:02:14.010637+0000","last_became_peered":"2024-09-17T07:02:14.010637+0000","last_unstale":"2024-09-17T07:12:48.300938+0000","last_undegraded":"2024-09-17T07:12:48.300938+0000","last_fullsized":"2024-09-17T07:12:48.300938+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'17","last_scrub_stamp":"2024-09-17T07:12:19.410758+0000","last_deep_scrub":"110'15","last_deep_scrub_stamp":"2024-09-17T07:04:57.091400+0000","last_clean_scrub_stamp":"2024-09-17T07:12:19.410758+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:29.125099+0000","scrub_duration":0.0070558269999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":241,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.303861+0000","last_change":"2024-09-17T07:12:04.314016+0000","last_active":"2024-09-17T07:12:48.303861+0000","last_peered":"2024-09-17T07:12:48.303861+0000","last_clean":"2024-09-17T07:12:48.303861+0000","last_became_active":"2024-09-17T07:02:12.010803+0000","last_became_peered":"2024-09-17T07:02:12.010803+0000","last_unstale":"2024-09-17T07:12:48.303861+0000","last_undegraded":"2024-09-17T07:12:48.303861+0000","last_fullsized":"2024-09-17T07:12:48.303861+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:04.313886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:25.147878+0000","last_clean_scrub_stamp":"2024-09-17T07:12:04.313886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:27.666148+0000","scrub_duration":0.0018928020000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"89'9","reported_seq":375,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-17T07:12:50.186440+0000","last_change":"2024-09-17T07:12:03.588884+0000","last_active":"2024-09-17T07:12:50.186440+0000","last_peered":"2024-09-17T07:12:50.186440+0000","last_clean":"2024-09-17T07:12:50.186440+0000","last_became_active":"2024-09-17T07:02:16.021360+0000","last_became_peered":"2024-09-17T07:02:16.021360+0000","last_unstale":"2024-09-17T07:12:50.186440+0000","last_undegraded":"2024-09-17T07:12:50.186440+0000","last_fullsized":"2024-09-17T07:12:50.186440+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'9","last_scrub_stamp":"2024-09-17T07:12:03.588797+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-09-17T07:03:29.961439+0000","last_clean_scrub_stamp":"2024-09-17T07:12:03.588797+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:21.994712+0000","scrub_duration":0.0018430440000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"75'1","reported_seq":250,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.287803+0000","last_change":"2024-09-17T07:12:22.641805+0000","last_active":"2024-09-17T07:12:48.287803+0000","last_peered":"2024-09-17T07:12:48.287803+0000","last_clean":"2024-09-17T07:12:48.287803+0000","last_became_active":"2024-09-17T07:09:24.399565+0000","last_became_peered":"2024-09-17T07:09:24.399565+0000","last_unstale":"2024-09-17T07:12:48.287803+0000","last_undegraded":"2024-09-17T07:12:48.287803+0000","last_fullsized":"2024-09-17T07:12:48.287803+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-17T07:12:22.641641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:12:22.641641+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:32.278016+0000","scrub_duration":0.0043476369999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.9","version":"252'4609","reported_seq":4633,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-17T07:12:54.230600+0000","last_change":"2024-09-17T07:12:05.175204+0000","last_active":"2024-09-17T07:12:54.230600+0000","last_peered":"2024-09-17T07:12:54.230600+0000","last_clean":"2024-09-17T07:12:54.230600+0000","last_became_active":"2024-09-17T07:12:05.174893+0000","last_became_peered":"2024-09-17T07:12:05.174893+0000","last_unstale":"2024-09-17T07:12:54.230600+0000","last_undegraded":"2024-09-17T07:12:54.230600+0000","last_fullsized":"2024-09-17T07:12:54.230600+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_clean_scrub_stamp":"2024-09-17T07:12:04.285115+0000","objects_scrubbed":0,"log_size":4609,"log_dups_size":0,"ondisk_log_size":4609,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:22.619349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":302055424,"num_objects":4609,"num_object_clones":0,"num_object_copies":13827,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4609,"num_write_kb":294976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"242'41","reported_seq":464,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-17T07:12:50.191684+0000","last_change":"2024-09-17T07:12:08.264127+0000","last_active":"2024-09-17T07:12:50.191684+0000","last_peered":"2024-09-17T07:12:50.191684+0000","last_clean":"2024-09-17T07:12:50.191684+0000","last_became_active":"2024-09-17T07:09:24.403267+0000","last_became_peered":"2024-09-17T07:09:24.403267+0000","last_unstale":"2024-09-17T07:12:50.191684+0000","last_undegraded":"2024-09-17T07:12:50.191684+0000","last_fullsized":"2024-09-17T07:12:50.191684+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'29","last_scrub_stamp":"2024-09-17T07:12:08.264016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:12:08.264016+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:30.766851+0000","scrub_duration":0.0026058610000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":252,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.300128+0000","last_change":"2024-09-17T07:12:19.678216+0000","last_active":"2024-09-17T07:12:48.300128+0000","last_peered":"2024-09-17T07:12:48.300128+0000","last_clean":"2024-09-17T07:12:48.300128+0000","last_became_active":"2024-09-17T07:02:12.008802+0000","last_became_peered":"2024-09-17T07:02:12.008802+0000","last_unstale":"2024-09-17T07:12:48.300128+0000","last_undegraded":"2024-09-17T07:12:48.300128+0000","last_fullsized":"2024-09-17T07:12:48.300128+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:19.678145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:12:19.678145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:27.258606+0000","scrub_duration":0.0013514519999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":238,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:49.277223+0000","last_change":"2024-09-17T07:11:07.433543+0000","last_active":"2024-09-17T07:12:49.277223+0000","last_peered":"2024-09-17T07:12:49.277223+0000","last_clean":"2024-09-17T07:12:49.277223+0000","last_became_active":"2024-09-17T07:02:16.016606+0000","last_became_peered":"2024-09-17T07:02:16.016606+0000","last_unstale":"2024-09-17T07:12:49.277223+0000","last_undegraded":"2024-09-17T07:12:49.277223+0000","last_fullsized":"2024-09-17T07:12:49.277223+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:07.433420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:11:07.433420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012900124000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":81,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.286557+0000","last_change":"2024-09-17T07:12:03.666609+0000","last_active":"2024-09-17T07:12:48.286557+0000","last_peered":"2024-09-17T07:12:48.286557+0000","last_clean":"2024-09-17T07:12:48.286557+0000","last_became_active":"2024-09-17T07:09:24.398878+0000","last_became_peered":"2024-09-17T07:09:24.398878+0000","last_unstale":"2024-09-17T07:12:48.286557+0000","last_undegraded":"2024-09-17T07:12:48.286557+0000","last_fullsized":"2024-09-17T07:12:48.286557+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:03.666518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:12:03.666518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:28.852645+0000","scrub_duration":0.0015175169999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.a","version":"252'4452","reported_seq":4476,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-17T07:12:53.261613+0000","last_change":"2024-09-17T07:12:05.175424+0000","last_active":"2024-09-17T07:12:53.261613+0000","last_peered":"2024-09-17T07:12:53.261613+0000","last_clean":"2024-09-17T07:12:53.261613+0000","last_became_active":"2024-09-17T07:12:05.175109+0000","last_became_peered":"2024-09-17T07:12:05.175109+0000","last_unstale":"2024-09-17T07:12:53.261613+0000","last_undegraded":"2024-09-17T07:12:53.261613+0000","last_fullsized":"2024-09-17T07:12:53.261613+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_clean_scrub_stamp":"2024-09-17T07:12:04.285115+0000","objects_scrubbed":0,"log_size":4452,"log_dups_size":0,"ondisk_log_size":4452,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:23.729980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":291766272,"num_objects":4452,"num_object_clones":0,"num_object_copies":13356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4452,"num_write_kb":284928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"242'17","reported_seq":271,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.309450+0000","last_change":"2024-09-17T07:11:17.112766+0000","last_active":"2024-09-17T07:12:48.309450+0000","last_peered":"2024-09-17T07:12:48.309450+0000","last_clean":"2024-09-17T07:12:48.309450+0000","last_became_active":"2024-09-17T07:02:14.019838+0000","last_became_peered":"2024-09-17T07:02:14.019838+0000","last_unstale":"2024-09-17T07:12:48.309450+0000","last_undegraded":"2024-09-17T07:12:48.309450+0000","last_fullsized":"2024-09-17T07:12:48.309450+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'13","last_scrub_stamp":"2024-09-17T07:11:17.112635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:11:17.112635+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.005881725,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"76'5","reported_seq":883,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-17T07:12:52.450975+0000","last_change":"2024-09-17T07:12:22.408620+0000","last_active":"2024-09-17T07:12:52.450975+0000","last_peered":"2024-09-17T07:12:52.450975+0000","last_clean":"2024-09-17T07:12:52.450975+0000","last_became_active":"2024-09-17T07:02:12.006883+0000","last_became_peered":"2024-09-17T07:02:12.006883+0000","last_unstale":"2024-09-17T07:12:52.450975+0000","last_undegraded":"2024-09-17T07:12:52.450975+0000","last_fullsized":"2024-09-17T07:12:52.450975+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'5","last_scrub_stamp":"2024-09-17T07:12:22.408498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:12:22.408498+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:38.147275+0000","scrub_duration":0.0032583909999999998,"objects_trimmed":0,"snaptrim_duration":0.00033739799999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":634,"num_read_kb":629,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":82,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.289707+0000","last_change":"2024-09-17T07:12:13.536969+0000","last_active":"2024-09-17T07:12:48.289707+0000","last_peered":"2024-09-17T07:12:48.289707+0000","last_clean":"2024-09-17T07:12:48.289707+0000","last_became_active":"2024-09-17T07:09:24.392850+0000","last_became_peered":"2024-09-17T07:09:24.392850+0000","last_unstale":"2024-09-17T07:12:48.289707+0000","last_undegraded":"2024-09-17T07:12:48.289707+0000","last_fullsized":"2024-09-17T07:12:48.289707+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:13.536888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:23.833170+0000","last_clean_scrub_stamp":"2024-09-17T07:12:13.536888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:25.415883+0000","scrub_duration":0.001259055,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":249,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.302790+0000","last_change":"2024-09-17T07:12:06.049472+0000","last_active":"2024-09-17T07:12:48.302790+0000","last_peered":"2024-09-17T07:12:48.302790+0000","last_clean":"2024-09-17T07:12:48.302790+0000","last_became_active":"2024-09-17T07:02:16.014492+0000","last_became_peered":"2024-09-17T07:02:16.014492+0000","last_unstale":"2024-09-17T07:12:48.302790+0000","last_undegraded":"2024-09-17T07:12:48.302790+0000","last_fullsized":"2024-09-17T07:12:48.302790+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:06.049391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:12:06.049391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:34.526195+0000","scrub_duration":0.001322546,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":255,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.405993+0000","last_change":"2024-09-17T07:12:12.332042+0000","last_active":"2024-09-17T07:12:48.405993+0000","last_peered":"2024-09-17T07:12:48.405993+0000","last_clean":"2024-09-17T07:12:48.405993+0000","last_became_active":"2024-09-17T07:09:24.402704+0000","last_became_peered":"2024-09-17T07:09:24.402704+0000","last_unstale":"2024-09-17T07:12:48.405993+0000","last_undegraded":"2024-09-17T07:12:48.405993+0000","last_fullsized":"2024-09-17T07:12:48.405993+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:12.331949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:12:12.331949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:30.558308+0000","scrub_duration":0.0012857490000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"252'4585","reported_seq":4609,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-17T07:12:54.198770+0000","last_change":"2024-09-17T07:12:05.173856+0000","last_active":"2024-09-17T07:12:54.198770+0000","last_peered":"2024-09-17T07:12:54.198770+0000","last_clean":"2024-09-17T07:12:54.198770+0000","last_became_active":"2024-09-17T07:12:05.173534+0000","last_became_peered":"2024-09-17T07:12:05.173534+0000","last_unstale":"2024-09-17T07:12:54.198770+0000","last_undegraded":"2024-09-17T07:12:54.198770+0000","last_fullsized":"2024-09-17T07:12:54.198770+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_clean_scrub_stamp":"2024-09-17T07:12:04.285115+0000","objects_scrubbed":0,"log_size":4585,"log_dups_size":0,"ondisk_log_size":4585,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:29.636041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":300482560,"num_objects":4585,"num_object_clones":0,"num_object_copies":13755,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4585,"num_write_kb":293440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"242'24","reported_seq":298,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.290899+0000","last_change":"2024-09-17T07:12:11.578753+0000","last_active":"2024-09-17T07:12:48.290899+0000","last_peered":"2024-09-17T07:12:48.290899+0000","last_clean":"2024-09-17T07:12:48.290899+0000","last_became_active":"2024-09-17T07:03:48.760405+0000","last_became_peered":"2024-09-17T07:03:48.760405+0000","last_unstale":"2024-09-17T07:12:48.290899+0000","last_undegraded":"2024-09-17T07:12:48.290899+0000","last_fullsized":"2024-09-17T07:12:48.290899+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-09-17T07:12:11.578647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:12:11.578647+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:30.740260+0000","scrub_duration":0.0020729070000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"70'1","reported_seq":250,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.289948+0000","last_change":"2024-09-17T07:09:24.415023+0000","last_active":"2024-09-17T07:12:48.289948+0000","last_peered":"2024-09-17T07:12:48.289948+0000","last_clean":"2024-09-17T07:12:48.289948+0000","last_became_active":"2024-09-17T07:09:24.402499+0000","last_became_peered":"2024-09-17T07:09:24.402499+0000","last_unstale":"2024-09-17T07:12:48.289948+0000","last_undegraded":"2024-09-17T07:12:48.289948+0000","last_fullsized":"2024-09-17T07:12:48.289948+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-17T07:09:22.249833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:09:22.249833+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0061513059999999996,"objects_trimmed":0,"snaptrim_duration":0.0032298629999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":241,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.297890+0000","last_change":"2024-09-17T07:12:16.273630+0000","last_active":"2024-09-17T07:12:48.297890+0000","last_peered":"2024-09-17T07:12:48.297890+0000","last_clean":"2024-09-17T07:12:48.297890+0000","last_became_active":"2024-09-17T07:02:12.014623+0000","last_became_peered":"2024-09-17T07:02:12.014623+0000","last_unstale":"2024-09-17T07:12:48.297890+0000","last_undegraded":"2024-09-17T07:12:48.297890+0000","last_fullsized":"2024-09-17T07:12:48.297890+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:16.273537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:12:16.273537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:41.553719+0000","scrub_duration":0.001748185,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":236,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.298379+0000","last_change":"2024-09-17T07:12:08.807635+0000","last_active":"2024-09-17T07:12:48.298379+0000","last_peered":"2024-09-17T07:12:48.298379+0000","last_clean":"2024-09-17T07:12:48.298379+0000","last_became_active":"2024-09-17T07:02:16.021561+0000","last_became_peered":"2024-09-17T07:02:16.021561+0000","last_unstale":"2024-09-17T07:12:48.298379+0000","last_undegraded":"2024-09-17T07:12:48.298379+0000","last_fullsized":"2024-09-17T07:12:48.298379+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:08.807531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:12:08.807531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:24.781268+0000","scrub_duration":0.0018440049999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":233,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.301264+0000","last_change":"2024-09-17T07:09:24.396448+0000","last_active":"2024-09-17T07:12:48.301264+0000","last_peered":"2024-09-17T07:12:48.301264+0000","last_clean":"2024-09-17T07:12:48.301264+0000","last_became_active":"2024-09-17T07:09:24.396154+0000","last_became_peered":"2024-09-17T07:09:24.396154+0000","last_unstale":"2024-09-17T07:12:48.301264+0000","last_undegraded":"2024-09-17T07:12:48.301264+0000","last_fullsized":"2024-09-17T07:12:48.301264+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:17.192238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:09:17.192238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.058290885000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"252'4473","reported_seq":4497,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-17T07:12:54.230746+0000","last_change":"2024-09-17T07:12:05.176237+0000","last_active":"2024-09-17T07:12:54.230746+0000","last_peered":"2024-09-17T07:12:54.230746+0000","last_clean":"2024-09-17T07:12:54.230746+0000","last_became_active":"2024-09-17T07:12:05.175944+0000","last_became_peered":"2024-09-17T07:12:05.175944+0000","last_unstale":"2024-09-17T07:12:54.230746+0000","last_undegraded":"2024-09-17T07:12:54.230746+0000","last_fullsized":"2024-09-17T07:12:54.230746+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_clean_scrub_stamp":"2024-09-17T07:12:04.285115+0000","objects_scrubbed":0,"log_size":4473,"log_dups_size":0,"ondisk_log_size":4473,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:21.517874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":293142528,"num_objects":4473,"num_object_clones":0,"num_object_copies":13419,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4473,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4473,"num_write_kb":286272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"242'26","reported_seq":281,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.292261+0000","last_change":"2024-09-17T07:11:15.463787+0000","last_active":"2024-09-17T07:12:48.292261+0000","last_peered":"2024-09-17T07:12:48.292261+0000","last_clean":"2024-09-17T07:12:48.292261+0000","last_became_active":"2024-09-17T07:02:14.028857+0000","last_became_peered":"2024-09-17T07:02:14.028857+0000","last_unstale":"2024-09-17T07:12:48.292261+0000","last_undegraded":"2024-09-17T07:12:48.292261+0000","last_fullsized":"2024-09-17T07:12:48.292261+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'14","last_scrub_stamp":"2024-09-17T07:11:15.463520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:11:15.463520+0000","objects_scrubbed":7,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0066150439999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"72'2","reported_seq":99,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.299922+0000","last_change":"2024-09-17T07:12:01.874553+0000","last_active":"2024-09-17T07:12:48.299922+0000","last_peered":"2024-09-17T07:12:48.299922+0000","last_clean":"2024-09-17T07:12:48.299922+0000","last_became_active":"2024-09-17T07:09:24.404532+0000","last_became_peered":"2024-09-17T07:09:24.404532+0000","last_unstale":"2024-09-17T07:12:48.299922+0000","last_undegraded":"2024-09-17T07:12:48.299922+0000","last_fullsized":"2024-09-17T07:12:48.299922+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'2","last_scrub_stamp":"2024-09-17T07:12:01.874430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:12:01.874430+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:16.309823+0000","scrub_duration":0.0018649420000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":252,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.297622+0000","last_change":"2024-09-17T07:12:11.818502+0000","last_active":"2024-09-17T07:12:48.297622+0000","last_peered":"2024-09-17T07:12:48.297622+0000","last_clean":"2024-09-17T07:12:48.297622+0000","last_became_active":"2024-09-17T07:02:12.004606+0000","last_became_peered":"2024-09-17T07:02:12.004606+0000","last_unstale":"2024-09-17T07:12:48.297622+0000","last_undegraded":"2024-09-17T07:12:48.297622+0000","last_fullsized":"2024-09-17T07:12:48.297622+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:11.818392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:12:11.818392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:15.462083+0000","scrub_duration":0.0014854499999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"212'130","reported_seq":502,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.297830+0000","last_change":"2024-09-17T07:12:23.270434+0000","last_active":"2024-09-17T07:12:48.297830+0000","last_peered":"2024-09-17T07:12:48.297830+0000","last_clean":"2024-09-17T07:12:48.297830+0000","last_became_active":"2024-09-17T07:02:03.020675+0000","last_became_peered":"2024-09-17T07:02:03.020675+0000","last_unstale":"2024-09-17T07:12:48.297830+0000","last_undegraded":"2024-09-17T07:12:48.297830+0000","last_fullsized":"2024-09-17T07:12:48.297830+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:12:23.270099+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:10:37.684115+0000","last_clean_scrub_stamp":"2024-09-17T07:12:23.270099+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:27.199435+0000","scrub_duration":0.0032525319999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":237,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.297858+0000","last_change":"2024-09-17T07:10:36.722143+0000","last_active":"2024-09-17T07:12:48.297858+0000","last_peered":"2024-09-17T07:12:48.297858+0000","last_clean":"2024-09-17T07:12:48.297858+0000","last_became_active":"2024-09-17T07:02:16.016031+0000","last_became_peered":"2024-09-17T07:02:16.016031+0000","last_unstale":"2024-09-17T07:12:48.297858+0000","last_undegraded":"2024-09-17T07:12:48.297858+0000","last_fullsized":"2024-09-17T07:12:48.297858+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:10:36.721691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:10:36.721691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0088554370000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":243,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.298598+0000","last_change":"2024-09-17T07:12:18.707829+0000","last_active":"2024-09-17T07:12:48.298598+0000","last_peered":"2024-09-17T07:12:48.298598+0000","last_clean":"2024-09-17T07:12:48.298598+0000","last_became_active":"2024-09-17T07:09:24.400041+0000","last_became_peered":"2024-09-17T07:09:24.400041+0000","last_unstale":"2024-09-17T07:12:48.298598+0000","last_undegraded":"2024-09-17T07:12:48.298598+0000","last_fullsized":"2024-09-17T07:12:48.298598+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:18.707704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:58.670571+0000","last_clean_scrub_stamp":"2024-09-17T07:12:18.707704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:46.842558+0000","scrub_duration":0.0052717149999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.7","version":"252'4480","reported_seq":4504,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-17T07:12:53.268035+0000","last_change":"2024-09-17T07:12:05.418324+0000","last_active":"2024-09-17T07:12:53.268035+0000","last_peered":"2024-09-17T07:12:53.268035+0000","last_clean":"2024-09-17T07:12:53.268035+0000","last_became_active":"2024-09-17T07:12:05.417969+0000","last_became_peered":"2024-09-17T07:12:05.417969+0000","last_unstale":"2024-09-17T07:12:53.268035+0000","last_undegraded":"2024-09-17T07:12:53.268035+0000","last_fullsized":"2024-09-17T07:12:53.268035+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_clean_scrub_stamp":"2024-09-17T07:12:04.285115+0000","objects_scrubbed":0,"log_size":4480,"log_dups_size":0,"ondisk_log_size":4480,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:04.730041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":293601280,"num_objects":4480,"num_object_clones":0,"num_object_copies":13440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4480,"num_write_kb":286720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"242'35","reported_seq":305,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.290641+0000","last_change":"2024-09-17T07:12:20.745763+0000","last_active":"2024-09-17T07:12:48.290641+0000","last_peered":"2024-09-17T07:12:48.290641+0000","last_clean":"2024-09-17T07:12:48.290641+0000","last_became_active":"2024-09-17T07:02:14.027313+0000","last_became_peered":"2024-09-17T07:02:14.027313+0000","last_unstale":"2024-09-17T07:12:48.290641+0000","last_undegraded":"2024-09-17T07:12:48.290641+0000","last_fullsized":"2024-09-17T07:12:48.290641+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'33","last_scrub_stamp":"2024-09-17T07:12:20.745628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:12:20.745628+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:28.059499+0000","scrub_duration":0.0047473180000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":265,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.302563+0000","last_change":"2024-09-17T07:12:17.126950+0000","last_active":"2024-09-17T07:12:48.302563+0000","last_peered":"2024-09-17T07:12:48.302563+0000","last_clean":"2024-09-17T07:12:48.302563+0000","last_became_active":"2024-09-17T07:02:12.012037+0000","last_became_peered":"2024-09-17T07:02:12.012037+0000","last_unstale":"2024-09-17T07:12:48.302563+0000","last_undegraded":"2024-09-17T07:12:48.302563+0000","last_fullsized":"2024-09-17T07:12:48.302563+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:17.126825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:06:39.322535+0000","last_clean_scrub_stamp":"2024-09-17T07:12:17.126825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:20.306203+0000","scrub_duration":0.001773309,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":260,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.301270+0000","last_change":"2024-09-17T07:12:04.096069+0000","last_active":"2024-09-17T07:12:48.301270+0000","last_peered":"2024-09-17T07:12:48.301270+0000","last_clean":"2024-09-17T07:12:48.301270+0000","last_became_active":"2024-09-17T07:02:16.014631+0000","last_became_peered":"2024-09-17T07:02:16.014631+0000","last_unstale":"2024-09-17T07:12:48.301270+0000","last_undegraded":"2024-09-17T07:12:48.301270+0000","last_fullsized":"2024-09-17T07:12:48.301270+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:04.095985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:12:04.095985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:31.712454+0000","scrub_duration":0.0015433160000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":243,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.297743+0000","last_change":"2024-09-17T07:12:22.309666+0000","last_active":"2024-09-17T07:12:48.297743+0000","last_peered":"2024-09-17T07:12:48.297743+0000","last_clean":"2024-09-17T07:12:48.297743+0000","last_became_active":"2024-09-17T07:09:24.403904+0000","last_became_peered":"2024-09-17T07:09:24.403904+0000","last_unstale":"2024-09-17T07:12:48.297743+0000","last_undegraded":"2024-09-17T07:12:48.297743+0000","last_fullsized":"2024-09-17T07:12:48.297743+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:22.309233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:12:22.309233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:34.704079+0000","scrub_duration":0.0042909669999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.6","version":"252'4426","reported_seq":4450,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-17T07:12:53.240615+0000","last_change":"2024-09-17T07:12:05.416479+0000","last_active":"2024-09-17T07:12:53.240615+0000","last_peered":"2024-09-17T07:12:53.240615+0000","last_clean":"2024-09-17T07:12:53.240615+0000","last_became_active":"2024-09-17T07:12:05.416160+0000","last_became_peered":"2024-09-17T07:12:05.416160+0000","last_unstale":"2024-09-17T07:12:53.240615+0000","last_undegraded":"2024-09-17T07:12:53.240615+0000","last_fullsized":"2024-09-17T07:12:53.240615+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_clean_scrub_stamp":"2024-09-17T07:12:04.285115+0000","objects_scrubbed":0,"log_size":4426,"log_dups_size":0,"ondisk_log_size":4426,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:06.915351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":290062336,"num_objects":4426,"num_object_clones":0,"num_object_copies":13278,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4426,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4426,"num_write_kb":283264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"242'15","reported_seq":264,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.288312+0000","last_change":"2024-09-17T07:12:12.585443+0000","last_active":"2024-09-17T07:12:48.288312+0000","last_peered":"2024-09-17T07:12:48.288312+0000","last_clean":"2024-09-17T07:12:48.288312+0000","last_became_active":"2024-09-17T07:02:14.014939+0000","last_became_peered":"2024-09-17T07:02:14.014939+0000","last_unstale":"2024-09-17T07:12:48.288312+0000","last_undegraded":"2024-09-17T07:12:48.288312+0000","last_fullsized":"2024-09-17T07:12:48.288312+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:12:12.585348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:12:12.585348+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:34.309086+0000","scrub_duration":0.001887445,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":249,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.307004+0000","last_change":"2024-09-17T07:11:29.068669+0000","last_active":"2024-09-17T07:12:48.307004+0000","last_peered":"2024-09-17T07:12:48.307004+0000","last_clean":"2024-09-17T07:12:48.307004+0000","last_became_active":"2024-09-17T07:09:24.391207+0000","last_became_peered":"2024-09-17T07:09:24.391207+0000","last_unstale":"2024-09-17T07:12:48.307004+0000","last_undegraded":"2024-09-17T07:12:48.307004+0000","last_fullsized":"2024-09-17T07:12:48.307004+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:29.068555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:06:30.038104+0000","last_clean_scrub_stamp":"2024-09-17T07:11:29.068555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:58.695021+0000","scrub_duration":0.0026579759999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":201,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.285422+0000","last_change":"2024-09-17T07:11:08.698817+0000","last_active":"2024-09-17T07:12:48.285422+0000","last_peered":"2024-09-17T07:12:48.285422+0000","last_clean":"2024-09-17T07:12:48.285422+0000","last_became_active":"2024-09-17T07:09:24.396642+0000","last_became_peered":"2024-09-17T07:09:24.396642+0000","last_unstale":"2024-09-17T07:12:48.285422+0000","last_undegraded":"2024-09-17T07:12:48.285422+0000","last_fullsized":"2024-09-17T07:12:48.285422+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:08.698703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:11:08.698703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0063428779999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":224,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.289776+0000","last_change":"2024-09-17T07:09:08.316161+0000","last_active":"2024-09-17T07:12:48.289776+0000","last_peered":"2024-09-17T07:12:48.289776+0000","last_clean":"2024-09-17T07:12:48.289776+0000","last_became_active":"2024-09-17T07:02:18.026994+0000","last_became_peered":"2024-09-17T07:02:18.026994+0000","last_unstale":"2024-09-17T07:12:48.289776+0000","last_undegraded":"2024-09-17T07:12:48.289776+0000","last_fullsized":"2024-09-17T07:12:48.289776+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:08.315842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:58.099849+0000","last_clean_scrub_stamp":"2024-09-17T07:09:08.315842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0081844919999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.5","version":"252'4546","reported_seq":4570,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-17T07:12:54.266064+0000","last_change":"2024-09-17T07:12:05.177132+0000","last_active":"2024-09-17T07:12:54.266064+0000","last_peered":"2024-09-17T07:12:54.266064+0000","last_clean":"2024-09-17T07:12:54.266064+0000","last_became_active":"2024-09-17T07:12:05.176729+0000","last_became_peered":"2024-09-17T07:12:05.176729+0000","last_unstale":"2024-09-17T07:12:54.266064+0000","last_undegraded":"2024-09-17T07:12:54.266064+0000","last_fullsized":"2024-09-17T07:12:54.266064+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_clean_scrub_stamp":"2024-09-17T07:12:04.285115+0000","objects_scrubbed":0,"log_size":4546,"log_dups_size":0,"ondisk_log_size":4546,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:24.650845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":297926656,"num_objects":4546,"num_object_clones":0,"num_object_copies":13638,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4546,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4546,"num_write_kb":290944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"242'34","reported_seq":288,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.298024+0000","last_change":"2024-09-17T07:10:35.125689+0000","last_active":"2024-09-17T07:12:48.298024+0000","last_peered":"2024-09-17T07:12:48.298024+0000","last_clean":"2024-09-17T07:12:48.298024+0000","last_became_active":"2024-09-17T07:02:14.028392+0000","last_became_peered":"2024-09-17T07:02:14.028392+0000","last_unstale":"2024-09-17T07:12:48.298024+0000","last_undegraded":"2024-09-17T07:12:48.298024+0000","last_fullsized":"2024-09-17T07:12:48.298024+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'22","last_scrub_stamp":"2024-09-17T07:10:35.125303+0000","last_deep_scrub":"111'22","last_deep_scrub_stamp":"2024-09-17T07:04:57.500450+0000","last_clean_scrub_stamp":"2024-09-17T07:10:35.125303+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.019620391000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'2","reported_seq":123,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.297790+0000","last_change":"2024-09-17T07:12:20.301388+0000","last_active":"2024-09-17T07:12:48.297790+0000","last_peered":"2024-09-17T07:12:48.297790+0000","last_clean":"2024-09-17T07:12:48.297790+0000","last_became_active":"2024-09-17T07:09:24.402909+0000","last_became_peered":"2024-09-17T07:09:24.402909+0000","last_unstale":"2024-09-17T07:12:48.297790+0000","last_undegraded":"2024-09-17T07:12:48.297790+0000","last_fullsized":"2024-09-17T07:12:48.297790+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-09-17T07:12:20.301263+0000","last_deep_scrub":"89'2","last_deep_scrub_stamp":"2024-09-17T07:12:20.301263+0000","last_clean_scrub_stamp":"2024-09-17T07:12:20.301263+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:38.286950+0000","scrub_duration":0.0065119280000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":420,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":420,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"225'9","reported_seq":376,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-17T07:12:50.181290+0000","last_change":"2024-09-17T07:11:25.292700+0000","last_active":"2024-09-17T07:12:50.181290+0000","last_peered":"2024-09-17T07:12:50.181290+0000","last_clean":"2024-09-17T07:12:50.181290+0000","last_became_active":"2024-09-17T07:02:16.015418+0000","last_became_peered":"2024-09-17T07:02:16.015418+0000","last_unstale":"2024-09-17T07:12:50.181290+0000","last_undegraded":"2024-09-17T07:12:50.181290+0000","last_fullsized":"2024-09-17T07:12:50.181290+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"225'9","last_scrub_stamp":"2024-09-17T07:11:25.292565+0000","last_deep_scrub":"225'9","last_deep_scrub_stamp":"2024-09-17T07:11:25.292565+0000","last_clean_scrub_stamp":"2024-09-17T07:11:25.292565+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:54.115350+0000","scrub_duration":0.0070251389999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":234,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.299238+0000","last_change":"2024-09-17T07:10:29.121100+0000","last_active":"2024-09-17T07:12:48.299238+0000","last_peered":"2024-09-17T07:12:48.299238+0000","last_clean":"2024-09-17T07:12:48.299238+0000","last_became_active":"2024-09-17T07:02:18.363253+0000","last_became_peered":"2024-09-17T07:02:18.363253+0000","last_unstale":"2024-09-17T07:12:48.299238+0000","last_undegraded":"2024-09-17T07:12:48.299238+0000","last_fullsized":"2024-09-17T07:12:48.299238+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:10:29.120985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:10:29.120985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.017226297000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.4","version":"252'4521","reported_seq":4546,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-17T07:12:53.622877+0000","last_change":"2024-09-17T07:12:05.169555+0000","last_active":"2024-09-17T07:12:53.622877+0000","last_peered":"2024-09-17T07:12:53.622877+0000","last_clean":"2024-09-17T07:12:53.622877+0000","last_became_active":"2024-09-17T07:12:05.169197+0000","last_became_peered":"2024-09-17T07:12:05.169197+0000","last_unstale":"2024-09-17T07:12:53.622877+0000","last_undegraded":"2024-09-17T07:12:53.622877+0000","last_fullsized":"2024-09-17T07:12:53.622877+0000","mapping_epoch":236,"log_start":"242'100","ondisk_log_start":"242'100","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_clean_scrub_stamp":"2024-09-17T07:12:04.285115+0000","objects_scrubbed":0,"log_size":4421,"log_dups_size":0,"ondisk_log_size":4421,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:19.862533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":296288256,"num_objects":4521,"num_object_clones":0,"num_object_copies":13563,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4521,"num_write_kb":289344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"242'23","reported_seq":319,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.287064+0000","last_change":"2024-09-17T07:12:07.649803+0000","last_active":"2024-09-17T07:12:48.287064+0000","last_peered":"2024-09-17T07:12:48.287064+0000","last_clean":"2024-09-17T07:12:48.287064+0000","last_became_active":"2024-09-17T07:10:49.723292+0000","last_became_peered":"2024-09-17T07:10:49.723292+0000","last_unstale":"2024-09-17T07:12:48.287064+0000","last_undegraded":"2024-09-17T07:12:48.287064+0000","last_fullsized":"2024-09-17T07:12:48.287064+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'17","last_scrub_stamp":"2024-09-17T07:12:07.649697+0000","last_deep_scrub":"150'17","last_deep_scrub_stamp":"2024-09-17T07:06:38.775788+0000","last_clean_scrub_stamp":"2024-09-17T07:12:07.649697+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:32.620327+0000","scrub_duration":0.002192238,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"68'2","reported_seq":254,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.286443+0000","last_change":"2024-09-17T07:11:05.636950+0000","last_active":"2024-09-17T07:12:48.286443+0000","last_peered":"2024-09-17T07:12:48.286443+0000","last_clean":"2024-09-17T07:12:48.286443+0000","last_became_active":"2024-09-17T07:10:49.727577+0000","last_became_peered":"2024-09-17T07:10:49.727577+0000","last_unstale":"2024-09-17T07:12:48.286443+0000","last_undegraded":"2024-09-17T07:12:48.286443+0000","last_fullsized":"2024-09-17T07:12:48.286443+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-09-17T07:11:05.636655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:11:05.636655+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.013113257999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":201,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.287893+0000","last_change":"2024-09-17T07:11:17.479269+0000","last_active":"2024-09-17T07:12:48.287893+0000","last_peered":"2024-09-17T07:12:48.287893+0000","last_clean":"2024-09-17T07:12:48.287893+0000","last_became_active":"2024-09-17T07:04:15.111069+0000","last_became_peered":"2024-09-17T07:04:15.111069+0000","last_unstale":"2024-09-17T07:12:48.287893+0000","last_undegraded":"2024-09-17T07:12:48.287893+0000","last_fullsized":"2024-09-17T07:12:48.287893+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:17.479036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:11:17.479036+0000","last_clean_scrub_stamp":"2024-09-17T07:11:17.479036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014966910000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":81,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.296296+0000","last_change":"2024-09-17T07:11:11.432231+0000","last_active":"2024-09-17T07:12:48.296296+0000","last_peered":"2024-09-17T07:12:48.296296+0000","last_clean":"2024-09-17T07:12:48.296296+0000","last_became_active":"2024-09-17T07:09:24.397669+0000","last_became_peered":"2024-09-17T07:09:24.397669+0000","last_unstale":"2024-09-17T07:12:48.296296+0000","last_undegraded":"2024-09-17T07:12:48.296296+0000","last_fullsized":"2024-09-17T07:12:48.296296+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:11.432111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:11:11.432111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0075412869999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"242'15","reported_seq":283,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.290824+0000","last_change":"2024-09-17T07:12:16.562906+0000","last_active":"2024-09-17T07:12:48.290824+0000","last_peered":"2024-09-17T07:12:48.290824+0000","last_clean":"2024-09-17T07:12:48.290824+0000","last_became_active":"2024-09-17T07:09:24.408875+0000","last_became_peered":"2024-09-17T07:09:24.408875+0000","last_unstale":"2024-09-17T07:12:48.290824+0000","last_undegraded":"2024-09-17T07:12:48.290824+0000","last_fullsized":"2024-09-17T07:12:48.290824+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:12:16.562818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:12:16.562818+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:26.004316+0000","scrub_duration":0.002298876,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":265,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.303348+0000","last_change":"2024-09-17T07:12:16.090212+0000","last_active":"2024-09-17T07:12:48.303348+0000","last_peered":"2024-09-17T07:12:48.303348+0000","last_clean":"2024-09-17T07:12:48.303348+0000","last_became_active":"2024-09-17T07:02:12.011327+0000","last_became_peered":"2024-09-17T07:02:12.011327+0000","last_unstale":"2024-09-17T07:12:48.303348+0000","last_undegraded":"2024-09-17T07:12:48.303348+0000","last_fullsized":"2024-09-17T07:12:48.303348+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:16.090110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:17.976267+0000","last_clean_scrub_stamp":"2024-09-17T07:12:16.090110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:41.373062+0000","scrub_duration":0.0012389359999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":80,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.306449+0000","last_change":"2024-09-17T07:12:18.367610+0000","last_active":"2024-09-17T07:12:48.306449+0000","last_peered":"2024-09-17T07:12:48.306449+0000","last_clean":"2024-09-17T07:12:48.306449+0000","last_became_active":"2024-09-17T07:09:24.407806+0000","last_became_peered":"2024-09-17T07:09:24.407806+0000","last_unstale":"2024-09-17T07:12:48.306449+0000","last_undegraded":"2024-09-17T07:12:48.306449+0000","last_fullsized":"2024-09-17T07:12:48.306449+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:18.367486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:18.367486+0000","last_clean_scrub_stamp":"2024-09-17T07:12:18.367486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:44.237351+0000","scrub_duration":0.0038802519999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":270,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.301688+0000","last_change":"2024-09-17T07:12:22.178678+0000","last_active":"2024-09-17T07:12:48.301688+0000","last_peered":"2024-09-17T07:12:48.301688+0000","last_clean":"2024-09-17T07:12:48.301688+0000","last_became_active":"2024-09-17T07:02:18.017825+0000","last_became_peered":"2024-09-17T07:02:18.017825+0000","last_unstale":"2024-09-17T07:12:48.301688+0000","last_undegraded":"2024-09-17T07:12:48.301688+0000","last_fullsized":"2024-09-17T07:12:48.301688+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:22.178556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:09:23.378805+0000","last_clean_scrub_stamp":"2024-09-17T07:12:22.178556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:28.430654+0000","scrub_duration":0.0029804850000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":253,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.291321+0000","last_change":"2024-09-17T07:12:17.648175+0000","last_active":"2024-09-17T07:12:48.291321+0000","last_peered":"2024-09-17T07:12:48.291321+0000","last_clean":"2024-09-17T07:12:48.291321+0000","last_became_active":"2024-09-17T07:02:12.008065+0000","last_became_peered":"2024-09-17T07:02:12.008065+0000","last_unstale":"2024-09-17T07:12:48.291321+0000","last_undegraded":"2024-09-17T07:12:48.291321+0000","last_fullsized":"2024-09-17T07:12:48.291321+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:17.648051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:12:17.648051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:33.995482+0000","scrub_duration":0.0030045190000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"242'12","reported_seq":118,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.296736+0000","last_change":"2024-09-17T07:12:03.380136+0000","last_active":"2024-09-17T07:12:48.296736+0000","last_peered":"2024-09-17T07:12:48.296736+0000","last_clean":"2024-09-17T07:12:48.296736+0000","last_became_active":"2024-09-17T07:09:24.405385+0000","last_became_peered":"2024-09-17T07:09:24.405385+0000","last_unstale":"2024-09-17T07:12:48.296736+0000","last_undegraded":"2024-09-17T07:12:48.296736+0000","last_fullsized":"2024-09-17T07:12:48.296736+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'6","last_scrub_stamp":"2024-09-17T07:12:03.380040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:12:03.380040+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:31.256725+0000","scrub_duration":0.0019381719999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":245,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.298171+0000","last_change":"2024-09-17T07:11:16.112334+0000","last_active":"2024-09-17T07:12:48.298171+0000","last_peered":"2024-09-17T07:12:48.298171+0000","last_clean":"2024-09-17T07:12:48.298171+0000","last_became_active":"2024-09-17T07:09:24.398922+0000","last_became_peered":"2024-09-17T07:09:24.398922+0000","last_unstale":"2024-09-17T07:12:48.298171+0000","last_undegraded":"2024-09-17T07:12:48.298171+0000","last_fullsized":"2024-09-17T07:12:48.298171+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:16.112226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:11:16.112226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0062666659999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":234,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.296798+0000","last_change":"2024-09-17T07:12:06.344966+0000","last_active":"2024-09-17T07:12:48.296798+0000","last_peered":"2024-09-17T07:12:48.296798+0000","last_clean":"2024-09-17T07:12:48.296798+0000","last_became_active":"2024-09-17T07:02:18.365419+0000","last_became_peered":"2024-09-17T07:02:18.365419+0000","last_unstale":"2024-09-17T07:12:48.296798+0000","last_undegraded":"2024-09-17T07:12:48.296798+0000","last_fullsized":"2024-09-17T07:12:48.296798+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:06.344833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:12:06.344833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:19.489450+0000","scrub_duration":0.002302817,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"242'20","reported_seq":135,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.303473+0000","last_change":"2024-09-17T07:12:12.288945+0000","last_active":"2024-09-17T07:12:48.303473+0000","last_peered":"2024-09-17T07:12:48.303473+0000","last_clean":"2024-09-17T07:12:48.303473+0000","last_became_active":"2024-09-17T07:09:24.405944+0000","last_became_peered":"2024-09-17T07:09:24.405944+0000","last_unstale":"2024-09-17T07:12:48.303473+0000","last_undegraded":"2024-09-17T07:12:48.303473+0000","last_fullsized":"2024-09-17T07:12:48.303473+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-09-17T07:12:12.288864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:12:12.288864+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:17.960594+0000","scrub_duration":0.0020069329999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":260,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.300081+0000","last_change":"2024-09-17T07:12:09.840697+0000","last_active":"2024-09-17T07:12:48.300081+0000","last_peered":"2024-09-17T07:12:48.300081+0000","last_clean":"2024-09-17T07:12:48.300081+0000","last_became_active":"2024-09-17T07:09:24.402355+0000","last_became_peered":"2024-09-17T07:09:24.402355+0000","last_unstale":"2024-09-17T07:12:48.300081+0000","last_undegraded":"2024-09-17T07:12:48.300081+0000","last_fullsized":"2024-09-17T07:12:48.300081+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:09.840613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:12:09.840613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:16.811598+0000","scrub_duration":0.0067048560000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"163'9","reported_seq":373,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-17T07:12:50.176881+0000","last_change":"2024-09-17T07:11:08.147847+0000","last_active":"2024-09-17T07:12:50.176881+0000","last_peered":"2024-09-17T07:12:50.176881+0000","last_clean":"2024-09-17T07:12:50.176881+0000","last_became_active":"2024-09-17T07:02:16.022583+0000","last_became_peered":"2024-09-17T07:02:16.022583+0000","last_unstale":"2024-09-17T07:12:50.176881+0000","last_undegraded":"2024-09-17T07:12:50.176881+0000","last_fullsized":"2024-09-17T07:12:50.176881+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'9","last_scrub_stamp":"2024-09-17T07:11:08.147466+0000","last_deep_scrub":"163'9","last_deep_scrub_stamp":"2024-09-17T07:07:34.933808+0000","last_clean_scrub_stamp":"2024-09-17T07:11:08.147466+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.039751243999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":244,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.290538+0000","last_change":"2024-09-17T07:12:26.619980+0000","last_active":"2024-09-17T07:12:48.290538+0000","last_peered":"2024-09-17T07:12:48.290538+0000","last_clean":"2024-09-17T07:12:48.290538+0000","last_became_active":"2024-09-17T07:09:24.409051+0000","last_became_peered":"2024-09-17T07:09:24.409051+0000","last_unstale":"2024-09-17T07:12:48.290538+0000","last_undegraded":"2024-09-17T07:12:48.290538+0000","last_fullsized":"2024-09-17T07:12:48.290538+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:26.619893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:26.619893+0000","last_clean_scrub_stamp":"2024-09-17T07:12:26.619893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:34.198640+0000","scrub_duration":0.0044029580000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"242'15","reported_seq":293,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.299219+0000","last_change":"2024-09-17T07:12:10.828225+0000","last_active":"2024-09-17T07:12:48.299219+0000","last_peered":"2024-09-17T07:12:48.299219+0000","last_clean":"2024-09-17T07:12:48.299219+0000","last_became_active":"2024-09-17T07:09:24.406333+0000","last_became_peered":"2024-09-17T07:09:24.406333+0000","last_unstale":"2024-09-17T07:12:48.299219+0000","last_undegraded":"2024-09-17T07:12:48.299219+0000","last_fullsized":"2024-09-17T07:12:48.299219+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:12:10.828114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:12:10.828114+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:28.226498+0000","scrub_duration":0.001735435,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":250,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.288663+0000","last_change":"2024-09-17T07:12:23.597448+0000","last_active":"2024-09-17T07:12:48.288663+0000","last_peered":"2024-09-17T07:12:48.288663+0000","last_clean":"2024-09-17T07:12:48.288663+0000","last_became_active":"2024-09-17T07:09:24.402704+0000","last_became_peered":"2024-09-17T07:09:24.402704+0000","last_unstale":"2024-09-17T07:12:48.288663+0000","last_undegraded":"2024-09-17T07:12:48.288663+0000","last_fullsized":"2024-09-17T07:12:48.288663+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:23.597330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:12:23.597330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:47.442372+0000","scrub_duration":0.0032745959999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"196'9","reported_seq":147,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-17T07:12:50.183345+0000","last_change":"2024-09-17T07:11:02.536969+0000","last_active":"2024-09-17T07:12:50.183345+0000","last_peered":"2024-09-17T07:12:50.183345+0000","last_clean":"2024-09-17T07:12:50.183345+0000","last_became_active":"2024-09-17T07:09:24.406267+0000","last_became_peered":"2024-09-17T07:09:24.406267+0000","last_unstale":"2024-09-17T07:12:50.183345+0000","last_undegraded":"2024-09-17T07:12:50.183345+0000","last_fullsized":"2024-09-17T07:12:50.183345+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'9","last_scrub_stamp":"2024-09-17T07:11:02.536718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:11:02.536718+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0054409640000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":236,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.286199+0000","last_change":"2024-09-17T07:12:16.543360+0000","last_active":"2024-09-17T07:12:48.286199+0000","last_peered":"2024-09-17T07:12:48.286199+0000","last_clean":"2024-09-17T07:12:48.286199+0000","last_became_active":"2024-09-17T07:02:18.018716+0000","last_became_peered":"2024-09-17T07:02:18.018716+0000","last_unstale":"2024-09-17T07:12:48.286199+0000","last_undegraded":"2024-09-17T07:12:48.286199+0000","last_fullsized":"2024-09-17T07:12:48.286199+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:16.543257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:12:16.543257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:33.805404+0000","scrub_duration":0.001567354,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"242'15","reported_seq":263,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.300867+0000","last_change":"2024-09-17T07:11:21.246333+0000","last_active":"2024-09-17T07:12:48.300867+0000","last_peered":"2024-09-17T07:12:48.300867+0000","last_clean":"2024-09-17T07:12:48.300867+0000","last_became_active":"2024-09-17T07:02:14.010289+0000","last_became_peered":"2024-09-17T07:02:14.010289+0000","last_unstale":"2024-09-17T07:12:48.300867+0000","last_undegraded":"2024-09-17T07:12:48.300867+0000","last_fullsized":"2024-09-17T07:12:48.300867+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:11:21.246183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:11:21.246183+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.005053564,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":261,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.292901+0000","last_change":"2024-09-17T07:12:13.677093+0000","last_active":"2024-09-17T07:12:48.292901+0000","last_peered":"2024-09-17T07:12:48.292901+0000","last_clean":"2024-09-17T07:12:48.292901+0000","last_became_active":"2024-09-17T07:09:24.408139+0000","last_became_peered":"2024-09-17T07:09:24.408139+0000","last_unstale":"2024-09-17T07:12:48.292901+0000","last_undegraded":"2024-09-17T07:12:48.292901+0000","last_fullsized":"2024-09-17T07:12:48.292901+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:13.677010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:10:36.334827+0000","last_clean_scrub_stamp":"2024-09-17T07:12:13.677010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:14.451688+0000","scrub_duration":0.0012308760000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":81,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.287971+0000","last_change":"2024-09-17T07:11:04.420764+0000","last_active":"2024-09-17T07:12:48.287971+0000","last_peered":"2024-09-17T07:12:48.287971+0000","last_clean":"2024-09-17T07:12:48.287971+0000","last_became_active":"2024-09-17T07:09:24.399116+0000","last_became_peered":"2024-09-17T07:09:24.399116+0000","last_unstale":"2024-09-17T07:12:48.287971+0000","last_undegraded":"2024-09-17T07:12:48.287971+0000","last_fullsized":"2024-09-17T07:12:48.287971+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:04.420639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:11:04.420639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011748066999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":235,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.287715+0000","last_change":"2024-09-17T07:12:06.573120+0000","last_active":"2024-09-17T07:12:48.287715+0000","last_peered":"2024-09-17T07:12:48.287715+0000","last_clean":"2024-09-17T07:12:48.287715+0000","last_became_active":"2024-09-17T07:02:18.027622+0000","last_became_peered":"2024-09-17T07:02:18.027622+0000","last_unstale":"2024-09-17T07:12:48.287715+0000","last_undegraded":"2024-09-17T07:12:48.287715+0000","last_fullsized":"2024-09-17T07:12:48.287715+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:06.573041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:12:06.573041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:11.909768+0000","scrub_duration":0.0024674409999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"242'19","reported_seq":271,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.285807+0000","last_change":"2024-09-17T07:12:13.566691+0000","last_active":"2024-09-17T07:12:48.285807+0000","last_peered":"2024-09-17T07:12:48.285807+0000","last_clean":"2024-09-17T07:12:48.285807+0000","last_became_active":"2024-09-17T07:02:14.016255+0000","last_became_peered":"2024-09-17T07:02:14.016255+0000","last_unstale":"2024-09-17T07:12:48.285807+0000","last_undegraded":"2024-09-17T07:12:48.285807+0000","last_fullsized":"2024-09-17T07:12:48.285807+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-09-17T07:12:13.566608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:12:13.566608+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:36.971016+0000","scrub_duration":0.0019196129999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"94'2","reported_seq":235,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.306933+0000","last_change":"2024-09-17T07:09:18.169710+0000","last_active":"2024-09-17T07:12:48.306933+0000","last_peered":"2024-09-17T07:12:48.306933+0000","last_clean":"2024-09-17T07:12:48.306933+0000","last_became_active":"2024-09-17T07:02:12.011461+0000","last_became_peered":"2024-09-17T07:02:12.011461+0000","last_unstale":"2024-09-17T07:12:48.306933+0000","last_undegraded":"2024-09-17T07:12:48.306933+0000","last_fullsized":"2024-09-17T07:12:48.306933+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-09-17T07:09:18.169580+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-09-17T07:07:31.717888+0000","last_clean_scrub_stamp":"2024-09-17T07:09:18.169580+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011743155,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":245,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.300740+0000","last_change":"2024-09-17T07:12:23.390704+0000","last_active":"2024-09-17T07:12:48.300740+0000","last_peered":"2024-09-17T07:12:48.300740+0000","last_clean":"2024-09-17T07:12:48.300740+0000","last_became_active":"2024-09-17T07:09:24.407362+0000","last_became_peered":"2024-09-17T07:09:24.407362+0000","last_unstale":"2024-09-17T07:12:48.300740+0000","last_undegraded":"2024-09-17T07:12:48.300740+0000","last_fullsized":"2024-09-17T07:12:48.300740+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:23.390587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:12:23.390587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:45.619920+0000","scrub_duration":0.010514658,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":80,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.289509+0000","last_change":"2024-09-17T07:12:05.555207+0000","last_active":"2024-09-17T07:12:48.289509+0000","last_peered":"2024-09-17T07:12:48.289509+0000","last_clean":"2024-09-17T07:12:48.289509+0000","last_became_active":"2024-09-17T07:09:24.400791+0000","last_became_peered":"2024-09-17T07:09:24.400791+0000","last_unstale":"2024-09-17T07:12:48.289509+0000","last_undegraded":"2024-09-17T07:12:48.289509+0000","last_fullsized":"2024-09-17T07:12:48.289509+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:05.555100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:12:05.555100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:14.820862+0000","scrub_duration":0.00158344,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"242'8","reported_seq":143,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.303242+0000","last_change":"2024-09-17T07:11:07.499806+0000","last_active":"2024-09-17T07:12:48.303242+0000","last_peered":"2024-09-17T07:12:48.303242+0000","last_clean":"2024-09-17T07:12:48.303242+0000","last_became_active":"2024-09-17T07:09:24.396368+0000","last_became_peered":"2024-09-17T07:09:24.396368+0000","last_unstale":"2024-09-17T07:12:48.303242+0000","last_undegraded":"2024-09-17T07:12:48.303242+0000","last_fullsized":"2024-09-17T07:12:48.303242+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'4","last_scrub_stamp":"2024-09-17T07:11:07.499679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:11:07.499679+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0043183819999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":240,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.298438+0000","last_change":"2024-09-17T07:11:24.083407+0000","last_active":"2024-09-17T07:12:48.298438+0000","last_peered":"2024-09-17T07:12:48.298438+0000","last_clean":"2024-09-17T07:12:48.298438+0000","last_became_active":"2024-09-17T07:02:12.008898+0000","last_became_peered":"2024-09-17T07:02:12.008898+0000","last_unstale":"2024-09-17T07:12:48.298438+0000","last_undegraded":"2024-09-17T07:12:48.298438+0000","last_fullsized":"2024-09-17T07:12:48.298438+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:24.083284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:11:24.083284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.014006177999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":238,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.291009+0000","last_change":"2024-09-17T07:11:03.383292+0000","last_active":"2024-09-17T07:12:48.291009+0000","last_peered":"2024-09-17T07:12:48.291009+0000","last_clean":"2024-09-17T07:12:48.291009+0000","last_became_active":"2024-09-17T07:02:16.014982+0000","last_became_peered":"2024-09-17T07:02:16.014982+0000","last_unstale":"2024-09-17T07:12:48.291009+0000","last_undegraded":"2024-09-17T07:12:48.291009+0000","last_fullsized":"2024-09-17T07:12:48.291009+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:03.383039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:47.444972+0000","last_clean_scrub_stamp":"2024-09-17T07:11:03.383039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015364230000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":234,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.296590+0000","last_change":"2024-09-17T07:11:23.302709+0000","last_active":"2024-09-17T07:12:48.296590+0000","last_peered":"2024-09-17T07:12:48.296590+0000","last_clean":"2024-09-17T07:12:48.296590+0000","last_became_active":"2024-09-17T07:02:18.021014+0000","last_became_peered":"2024-09-17T07:02:18.021014+0000","last_unstale":"2024-09-17T07:12:48.296590+0000","last_undegraded":"2024-09-17T07:12:48.296590+0000","last_fullsized":"2024-09-17T07:12:48.296590+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:23.302586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:11:23.302586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0077240449999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"242'19","reported_seq":135,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.299110+0000","last_change":"2024-09-17T07:11:20.321025+0000","last_active":"2024-09-17T07:12:48.299110+0000","last_peered":"2024-09-17T07:12:48.299110+0000","last_clean":"2024-09-17T07:12:48.299110+0000","last_became_active":"2024-09-17T07:09:24.404942+0000","last_became_peered":"2024-09-17T07:09:24.404942+0000","last_unstale":"2024-09-17T07:12:48.299110+0000","last_undegraded":"2024-09-17T07:12:48.299110+0000","last_fullsized":"2024-09-17T07:12:48.299110+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-09-17T07:11:20.320870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:11:20.320870+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.013438802,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":252,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.293242+0000","last_change":"2024-09-17T07:11:14.431423+0000","last_active":"2024-09-17T07:12:48.293242+0000","last_peered":"2024-09-17T07:12:48.293242+0000","last_clean":"2024-09-17T07:12:48.293242+0000","last_became_active":"2024-09-17T07:02:12.010195+0000","last_became_peered":"2024-09-17T07:02:12.010195+0000","last_unstale":"2024-09-17T07:12:48.293242+0000","last_undegraded":"2024-09-17T07:12:48.293242+0000","last_fullsized":"2024-09-17T07:12:48.293242+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:14.431102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:11:14.431102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0070527250000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":236,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.296674+0000","last_change":"2024-09-17T07:11:26.259825+0000","last_active":"2024-09-17T07:12:48.296674+0000","last_peered":"2024-09-17T07:12:48.296674+0000","last_clean":"2024-09-17T07:12:48.296674+0000","last_became_active":"2024-09-17T07:02:16.016183+0000","last_became_peered":"2024-09-17T07:02:16.016183+0000","last_unstale":"2024-09-17T07:12:48.296674+0000","last_undegraded":"2024-09-17T07:12:48.296674+0000","last_fullsized":"2024-09-17T07:12:48.296674+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:26.258291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:11:26.258291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034422369999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":80,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.287639+0000","last_change":"2024-09-17T07:10:34.515889+0000","last_active":"2024-09-17T07:12:48.287639+0000","last_peered":"2024-09-17T07:12:48.287639+0000","last_clean":"2024-09-17T07:12:48.287639+0000","last_became_active":"2024-09-17T07:09:24.392216+0000","last_became_peered":"2024-09-17T07:09:24.392216+0000","last_unstale":"2024-09-17T07:12:48.287639+0000","last_undegraded":"2024-09-17T07:12:48.287639+0000","last_fullsized":"2024-09-17T07:12:48.287639+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:10:34.515779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:10:34.515779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032588679999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1c","version":"75'1","reported_seq":235,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.296891+0000","last_change":"2024-09-17T07:11:17.398031+0000","last_active":"2024-09-17T07:12:48.296891+0000","last_peered":"2024-09-17T07:12:48.296891+0000","last_clean":"2024-09-17T07:12:48.296891+0000","last_became_active":"2024-09-17T07:02:18.021855+0000","last_became_peered":"2024-09-17T07:02:18.021855+0000","last_unstale":"2024-09-17T07:12:48.296891+0000","last_undegraded":"2024-09-17T07:12:48.296891+0000","last_fullsized":"2024-09-17T07:12:48.296891+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-17T07:11:17.396229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:11:17.396229+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0090123449999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":241,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.298066+0000","last_change":"2024-09-17T07:11:06.126289+0000","last_active":"2024-09-17T07:12:48.298066+0000","last_peered":"2024-09-17T07:12:48.298066+0000","last_clean":"2024-09-17T07:12:48.298066+0000","last_became_active":"2024-09-17T07:02:12.005329+0000","last_became_peered":"2024-09-17T07:02:12.005329+0000","last_unstale":"2024-09-17T07:12:48.298066+0000","last_undegraded":"2024-09-17T07:12:48.298066+0000","last_fullsized":"2024-09-17T07:12:48.298066+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:06.126127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:11:06.126127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001691456,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"242'20","reported_seq":312,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.290746+0000","last_change":"2024-09-17T07:12:25.625670+0000","last_active":"2024-09-17T07:12:48.290746+0000","last_peered":"2024-09-17T07:12:48.290746+0000","last_clean":"2024-09-17T07:12:48.290746+0000","last_became_active":"2024-09-17T07:10:49.730948+0000","last_became_peered":"2024-09-17T07:10:49.730948+0000","last_unstale":"2024-09-17T07:12:48.290746+0000","last_undegraded":"2024-09-17T07:12:48.290746+0000","last_fullsized":"2024-09-17T07:12:48.290746+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'20","last_scrub_stamp":"2024-09-17T07:12:25.625530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:12:25.625530+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:27.820616+0000","scrub_duration":0.0066279870000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"75'8","reported_seq":378,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-17T07:12:50.188285+0000","last_change":"2024-09-17T07:11:12.356249+0000","last_active":"2024-09-17T07:12:50.188285+0000","last_peered":"2024-09-17T07:12:50.188285+0000","last_clean":"2024-09-17T07:12:50.188285+0000","last_became_active":"2024-09-17T07:02:16.020560+0000","last_became_peered":"2024-09-17T07:02:16.020560+0000","last_unstale":"2024-09-17T07:12:50.188285+0000","last_undegraded":"2024-09-17T07:12:50.188285+0000","last_fullsized":"2024-09-17T07:12:50.188285+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:11:12.356129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:11:12.356129+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0063199399999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":234,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.301213+0000","last_change":"2024-09-17T07:12:21.397606+0000","last_active":"2024-09-17T07:12:48.301213+0000","last_peered":"2024-09-17T07:12:48.301213+0000","last_clean":"2024-09-17T07:12:48.301213+0000","last_became_active":"2024-09-17T07:02:18.021603+0000","last_became_peered":"2024-09-17T07:02:18.021603+0000","last_unstale":"2024-09-17T07:12:48.301213+0000","last_undegraded":"2024-09-17T07:12:48.301213+0000","last_fullsized":"2024-09-17T07:12:48.301213+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:21.397491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:12:21.397491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:42.061961+0000","scrub_duration":0.0045805669999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":92,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.291247+0000","last_change":"2024-09-17T07:12:09.738322+0000","last_active":"2024-09-17T07:12:48.291247+0000","last_peered":"2024-09-17T07:12:48.291247+0000","last_clean":"2024-09-17T07:12:48.291247+0000","last_became_active":"2024-09-17T07:09:24.403360+0000","last_became_peered":"2024-09-17T07:09:24.403360+0000","last_unstale":"2024-09-17T07:12:48.291247+0000","last_undegraded":"2024-09-17T07:12:48.291247+0000","last_fullsized":"2024-09-17T07:12:48.291247+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:09.738248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:10:29.325664+0000","last_clean_scrub_stamp":"2024-09-17T07:12:09.738248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:27.942295+0000","scrub_duration":0.0014227129999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"242'19","reported_seq":324,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.293181+0000","last_change":"2024-09-17T07:12:27.748152+0000","last_active":"2024-09-17T07:12:48.293181+0000","last_peered":"2024-09-17T07:12:48.293181+0000","last_clean":"2024-09-17T07:12:48.293181+0000","last_became_active":"2024-09-17T07:03:48.760390+0000","last_became_peered":"2024-09-17T07:03:48.760390+0000","last_unstale":"2024-09-17T07:12:48.293181+0000","last_undegraded":"2024-09-17T07:12:48.293181+0000","last_fullsized":"2024-09-17T07:12:48.293181+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'19","last_scrub_stamp":"2024-09-17T07:12:27.748022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:12:27.748022+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:53.737985+0000","scrub_duration":0.0036302800000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":227,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.290670+0000","last_change":"2024-09-17T07:09:10.293301+0000","last_active":"2024-09-17T07:12:48.290670+0000","last_peered":"2024-09-17T07:12:48.290670+0000","last_clean":"2024-09-17T07:12:48.290670+0000","last_became_active":"2024-09-17T07:02:16.019701+0000","last_became_peered":"2024-09-17T07:02:16.019701+0000","last_unstale":"2024-09-17T07:12:48.290670+0000","last_undegraded":"2024-09-17T07:12:48.290670+0000","last_fullsized":"2024-09-17T07:12:48.290670+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:09:10.292615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:09:10.292615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010112249,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":237,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.286685+0000","last_change":"2024-09-17T07:12:27.469985+0000","last_active":"2024-09-17T07:12:48.286685+0000","last_peered":"2024-09-17T07:12:48.286685+0000","last_clean":"2024-09-17T07:12:48.286685+0000","last_became_active":"2024-09-17T07:02:18.363752+0000","last_became_peered":"2024-09-17T07:02:18.363752+0000","last_unstale":"2024-09-17T07:12:48.286685+0000","last_undegraded":"2024-09-17T07:12:48.286685+0000","last_fullsized":"2024-09-17T07:12:48.286685+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:27.469698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:12:27.469698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:34.111455+0000","scrub_duration":0.0056145910000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":282,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.290765+0000","last_change":"2024-09-17T07:12:21.708679+0000","last_active":"2024-09-17T07:12:48.290765+0000","last_peered":"2024-09-17T07:12:48.290765+0000","last_clean":"2024-09-17T07:12:48.290765+0000","last_became_active":"2024-09-17T07:03:48.759384+0000","last_became_peered":"2024-09-17T07:03:48.759384+0000","last_unstale":"2024-09-17T07:12:48.290765+0000","last_undegraded":"2024-09-17T07:12:48.290765+0000","last_fullsized":"2024-09-17T07:12:48.290765+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:21.708543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:11:18.559877+0000","last_clean_scrub_stamp":"2024-09-17T07:12:21.708543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:38.909772+0000","scrub_duration":0.001526856,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"242'27","reported_seq":282,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.303565+0000","last_change":"2024-09-17T07:09:24.427421+0000","last_active":"2024-09-17T07:12:48.303565+0000","last_peered":"2024-09-17T07:12:48.303565+0000","last_clean":"2024-09-17T07:12:48.303565+0000","last_became_active":"2024-09-17T07:09:24.403030+0000","last_became_peered":"2024-09-17T07:09:24.403030+0000","last_unstale":"2024-09-17T07:12:48.303565+0000","last_undegraded":"2024-09-17T07:12:48.303565+0000","last_fullsized":"2024-09-17T07:12:48.303565+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-09-17T07:09:15.110940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:09:15.110940+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.017595989999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":236,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.306726+0000","last_change":"2024-09-17T07:12:01.366198+0000","last_active":"2024-09-17T07:12:48.306726+0000","last_peered":"2024-09-17T07:12:48.306726+0000","last_clean":"2024-09-17T07:12:48.306726+0000","last_became_active":"2024-09-17T07:02:16.015903+0000","last_became_peered":"2024-09-17T07:02:16.015903+0000","last_unstale":"2024-09-17T07:12:48.306726+0000","last_undegraded":"2024-09-17T07:12:48.306726+0000","last_fullsized":"2024-09-17T07:12:48.306726+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:01.366088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:25.794418+0000","last_clean_scrub_stamp":"2024-09-17T07:12:01.366088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:01.913633+0000","scrub_duration":0.001615116,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":93,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.292128+0000","last_change":"2024-09-17T07:12:22.734468+0000","last_active":"2024-09-17T07:12:48.292128+0000","last_peered":"2024-09-17T07:12:48.292128+0000","last_clean":"2024-09-17T07:12:48.292128+0000","last_became_active":"2024-09-17T07:09:24.402411+0000","last_became_peered":"2024-09-17T07:09:24.402411+0000","last_unstale":"2024-09-17T07:12:48.292128+0000","last_undegraded":"2024-09-17T07:12:48.292128+0000","last_fullsized":"2024-09-17T07:12:48.292128+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:22.734328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:12:22.734328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:24.175241+0000","scrub_duration":0.0038730150000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":82,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.296825+0000","last_change":"2024-09-17T07:11:00.563847+0000","last_active":"2024-09-17T07:12:48.296825+0000","last_peered":"2024-09-17T07:12:48.296825+0000","last_clean":"2024-09-17T07:12:48.296825+0000","last_became_active":"2024-09-17T07:09:24.404795+0000","last_became_peered":"2024-09-17T07:09:24.404795+0000","last_unstale":"2024-09-17T07:12:48.296825+0000","last_undegraded":"2024-09-17T07:12:48.296825+0000","last_fullsized":"2024-09-17T07:12:48.296825+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:00.563722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:11:00.563722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.018069940999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"242'23","reported_seq":155,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.306804+0000","last_change":"2024-09-17T07:12:24.341244+0000","last_active":"2024-09-17T07:12:48.306804+0000","last_peered":"2024-09-17T07:12:48.306804+0000","last_clean":"2024-09-17T07:12:48.306804+0000","last_became_active":"2024-09-17T07:09:24.394476+0000","last_became_peered":"2024-09-17T07:09:24.394476+0000","last_unstale":"2024-09-17T07:12:48.306804+0000","last_undegraded":"2024-09-17T07:12:48.306804+0000","last_fullsized":"2024-09-17T07:12:48.306804+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'23","last_scrub_stamp":"2024-09-17T07:12:24.341132+0000","last_deep_scrub":"86'17","last_deep_scrub_stamp":"2024-09-17T07:03:18.241067+0000","last_clean_scrub_stamp":"2024-09-17T07:12:24.341132+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:34.696830+0000","scrub_duration":0.0043568469999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":247,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-17T07:12:48.287137+0000","last_change":"2024-09-17T07:11:14.763627+0000","last_active":"2024-09-17T07:12:48.287137+0000","last_peered":"2024-09-17T07:12:48.287137+0000","last_clean":"2024-09-17T07:12:48.287137+0000","last_became_active":"2024-09-17T07:09:24.393329+0000","last_became_peered":"2024-09-17T07:09:24.393329+0000","last_unstale":"2024-09-17T07:12:48.287137+0000","last_undegraded":"2024-09-17T07:12:48.287137+0000","last_fullsized":"2024-09-17T07:12:48.287137+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:14.763470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:11:14.763470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0044668709999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":4717084672,"num_objects":71977,"num_object_clones":0,"num_object_copies":215931,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71977,"num_write_kb":4606528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14123204608,"data_stored":14123204608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70877,"ondisk_log_size":70877,"up":48,"acting":48,"num_store_stats":7},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70,"ondisk_log_size":70,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1121,"num_read_kb":878,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":109,"num_bytes_recovered":2228,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":13004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":713,"ondisk_log_size":713,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":634,"num_read_kb":629,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1386,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":12,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":4578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037894,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3559916,"kb_used_data":3135844,"kb_used_omap":1,"kb_used_meta":424062,"kb_avail":90185236,"statfs":{"total":95995035648,"available":92349681664,"internally_reserved":0,"allocated":3211104256,"data_stored":3210597430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":434239947},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:12:22 2024","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":1.8680000000000001,"15min":1.4319999999999999},"min":{"1min":0.315,"5min":0.192,"15min":0.192},"max":{"1min":4.907,"5min":48.493000000000002,"15min":52.478999999999999},"last":0.875},{"interface":"front","average":{"1min":0.78000000000000003,"5min":2.2080000000000002,"15min":1.5389999999999999},"min":{"1min":0.36599999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":2.7519999999999998,"5min":48.738,"15min":52.871000000000002},"last":0.73999999999999999}]},{"osd":1,"last update":"Tue Sep 17 07:12:22 2024","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":2.1960000000000002,"15min":1.141},"min":{"1min":0.376,"5min":0.223,"15min":0.223},"max":{"1min":4.9569999999999999,"5min":73.549000000000007,"15min":73.549000000000007},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.86399999999999999,"5min":2.3069999999999999,"15min":1.1910000000000001},"min":{"1min":0.308,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":3.2970000000000002,"5min":73.116,"15min":73.116},"last":1.661}]},{"osd":2,"last update":"Tue Sep 17 07:12:22 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":1.1930000000000001,"15min":0.96299999999999997},"min":{"1min":0.38400000000000001,"5min":0.245,"15min":0.23899999999999999},"max":{"1min":2.3330000000000002,"5min":22.376000000000001,"15min":52.055999999999997},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.91200000000000003,"5min":1.1000000000000001,"15min":0.96599999999999997},"min":{"1min":0.34599999999999997,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":4.9340000000000002,"5min":22.298999999999999,"15min":52.064999999999998},"last":0.94699999999999995}]},{"osd":3,"last update":"Tue Sep 17 07:12:22 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":1.367,"15min":1.155},"min":{"1min":0.35599999999999998,"5min":0.29499999999999998,"15min":0.28799999999999998},"max":{"1min":2.2999999999999998,"5min":22.196000000000002,"15min":51.689},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.92200000000000004,"5min":1.3640000000000001,"15min":1.2490000000000001},"min":{"1min":0.34599999999999997,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":4.8769999999999998,"5min":22.234000000000002,"15min":52.145000000000003},"last":0.92300000000000004}]},{"osd":4,"last update":"Tue Sep 17 07:12:22 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.75,"15min":0.54500000000000004},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.22800000000000001},"max":{"1min":2.8700000000000001,"5min":4.9950000000000001,"15min":4.9950000000000001},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.82699999999999996,"15min":0.58199999999999996},"min":{"1min":0.28499999999999998,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.968,"5min":6.0549999999999997,"15min":6.0549999999999997},"last":0.40999999999999998}]},{"osd":5,"last update":"Tue Sep 17 07:12:22 2024","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.81799999999999995,"15min":0.57899999999999996},"min":{"1min":0.27100000000000002,"5min":0.253,"15min":0.23699999999999999},"max":{"1min":4.8390000000000004,"5min":5.1360000000000001,"15min":5.1360000000000001},"last":0.90100000000000002},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.83399999999999996,"15min":0.59099999999999997},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":2.794,"5min":3.8639999999999999,"15min":3.8639999999999999},"last":0.48099999999999998}]},{"osd":6,"last update":"Tue Sep 17 07:12:22 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":1.1140000000000001,"15min":0.71399999999999997},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":2.8479999999999999,"5min":25.363,"15min":25.363},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.76500000000000001,"5min":1.0820000000000001,"15min":0.71199999999999997},"min":{"1min":0.29599999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":2.6299999999999999,"5min":23.571000000000002,"15min":23.571000000000002},"last":0.38900000000000001}]}]},{"osd":6,"up_from":50,"seq":214748364937,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1170888,"kb_used_data":851008,"kb_used_omap":1,"kb_used_meta":319870,"kb_avail":92574264,"statfs":{"total":95995035648,"available":94796046336,"internally_reserved":0,"allocated":871432192,"data_stored":870897311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":327547335},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:12:04 2024","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":1.667,"15min":1.5880000000000001},"min":{"1min":0.318,"5min":0.185,"15min":0.185},"max":{"1min":5.5010000000000003,"5min":24.106999999999999,"15min":75.323999999999998},"last":0.28100000000000003},{"interface":"front","average":{"1min":1.4630000000000001,"5min":1.6000000000000001,"15min":1.546},"min":{"1min":0.24299999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":5.6459999999999999,"5min":26.193000000000001,"15min":75.168999999999997},"last":0.36399999999999999}]},{"osd":1,"last update":"Tue Sep 17 07:12:04 2024","interfaces":[{"interface":"back","average":{"1min":1.2649999999999999,"5min":1.101,"15min":0.72699999999999998},"min":{"1min":0.28399999999999997,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":5.0549999999999997,"5min":5.0549999999999997,"15min":5.0549999999999997},"last":14.720000000000001},{"interface":"front","average":{"1min":0.79600000000000004,"5min":1.0169999999999999,"15min":0.747},"min":{"1min":0.27300000000000002,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":2.9399999999999999,"5min":5.4930000000000003,"15min":5.9530000000000003},"last":14.699}]},{"osd":2,"last update":"Tue Sep 17 07:12:04 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.91300000000000003,"15min":0.80200000000000005},"min":{"1min":0.314,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":2.984,"5min":5.1159999999999997,"15min":33.079000000000001},"last":0.35599999999999998},{"interface":"front","average":{"1min":1.095,"5min":0.876,"15min":0.79100000000000004},"min":{"1min":0.26700000000000002,"5min":0.254,"15min":0.23000000000000001},"max":{"1min":5.734,"5min":5.734,"15min":33.100000000000001},"last":0.25600000000000001}]},{"osd":3,"last update":"Tue Sep 17 07:12:04 2024","interfaces":[{"interface":"back","average":{"1min":1.0289999999999999,"5min":1.2010000000000001,"15min":1.1080000000000001},"min":{"1min":0.31900000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":5.0830000000000002,"5min":16.984000000000002,"15min":52.944000000000003},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.73199999999999998,"5min":1.2729999999999999,"15min":1.1279999999999999},"min":{"1min":0.32600000000000001,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":2.9649999999999999,"5min":16.745999999999999,"15min":52.747},"last":0.33300000000000002}]},{"osd":4,"last update":"Tue Sep 17 07:12:04 2024","interfaces":[{"interface":"back","average":{"1min":1.2050000000000001,"5min":4.444,"15min":1.7609999999999999},"min":{"1min":0.23999999999999999,"5min":0.214,"15min":0.214},"max":{"1min":5.7130000000000001,"5min":260.77199999999999,"15min":260.77199999999999},"last":0.30599999999999999},{"interface":"front","average":{"1min":1.294,"5min":4.4829999999999997,"15min":1.79},"min":{"1min":0.24299999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":6.2140000000000004,"5min":260.69099999999997,"15min":260.69099999999997},"last":0.24199999999999999}]},{"osd":5,"last update":"Tue Sep 17 07:12:04 2024","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":1.159,"15min":0.67900000000000005},"min":{"1min":0.27600000000000002,"5min":0.26200000000000001,"15min":0.25900000000000001},"max":{"1min":6.3860000000000001,"5min":32.042000000000002,"15min":32.042000000000002},"last":0.34399999999999997},{"interface":"front","average":{"1min":1.135,"5min":1.3200000000000001,"15min":0.748},"min":{"1min":0.28999999999999998,"5min":0.251,"15min":0.251},"max":{"1min":5.1379999999999999,"5min":31.852,"15min":31.852},"last":0.29499999999999998}]},{"osd":7,"last update":"Tue Sep 17 07:12:24 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.98299999999999998,"15min":0.67700000000000005},"min":{"1min":0.371,"5min":0.28100000000000003,"15min":0.23699999999999999},"max":{"1min":3.4100000000000001,"5min":7.657,"15min":7.657},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.94599999999999995,"15min":0.68200000000000005},"min":{"1min":0.35599999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":3.4510000000000001,"5min":7.1890000000000001,"15min":7.1890000000000001},"last":0.38200000000000001}]}]},{"osd":5,"up_from":42,"seq":180388626574,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3255404,"kb_used_data":2832996,"kb_used_omap":1,"kb_used_meta":422398,"kb_avail":90489748,"statfs":{"total":95995035648,"available":92661501952,"internally_reserved":0,"allocated":2900987904,"data_stored":2900480866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":432536006},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:12:40 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":1.2789999999999999,"15min":1.4790000000000001},"min":{"1min":0.26000000000000001,"5min":0.251,"15min":0.221},"max":{"1min":3.9729999999999999,"5min":13.913,"15min":49.265000000000001},"last":1.8340000000000001},{"interface":"front","average":{"1min":0.97099999999999997,"5min":1.3580000000000001,"15min":1.5369999999999999},"min":{"1min":0.35199999999999998,"5min":0.219,"15min":0.219},"max":{"1min":3.9369999999999998,"5min":14.539999999999999,"15min":47.350000000000001},"last":0.29999999999999999}]},{"osd":1,"last update":"Tue Sep 17 07:12:40 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.97699999999999998,"15min":0.77100000000000002},"min":{"1min":0.33900000000000002,"5min":0.23799999999999999,"15min":0.23300000000000001},"max":{"1min":2.0329999999999999,"5min":5.5529999999999999,"15min":20.329000000000001},"last":1.8109999999999999},{"interface":"front","average":{"1min":0.879,"5min":1.2230000000000001,"15min":1.1299999999999999},"min":{"1min":0.246,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":2.327,"5min":5.931,"15min":20.408999999999999},"last":2.3599999999999999}]},{"osd":2,"last update":"Tue Sep 17 07:12:40 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":1.2190000000000001,"15min":0.96499999999999997},"min":{"1min":0.29199999999999998,"5min":0.25900000000000001,"15min":0.252},"max":{"1min":1.7250000000000001,"5min":6.242,"15min":20.486000000000001},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.60399999999999998,"5min":1.0229999999999999,"15min":0.88400000000000001},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.23200000000000001},"max":{"1min":1.958,"5min":7.1310000000000002,"15min":23.925999999999998},"last":1.8979999999999999}]},{"osd":3,"last update":"Tue Sep 17 07:12:40 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":1.036,"15min":1.222},"min":{"1min":0.374,"5min":0.247,"15min":0.247},"max":{"1min":1.696,"5min":6.5439999999999996,"15min":68.978999999999999},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.67700000000000005,"5min":1.0369999999999999,"15min":1.196},"min":{"1min":0.372,"5min":0.26800000000000002,"15min":0.253},"max":{"1min":2,"5min":6.4500000000000002,"15min":69.084000000000003},"last":1.8779999999999999}]},{"osd":4,"last update":"Tue Sep 17 07:12:40 2024","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":2.7229999999999999,"15min":1.1859999999999999},"min":{"1min":0.28599999999999998,"5min":0.23400000000000001,"15min":0.22},"max":{"1min":5.2240000000000002,"5min":142.327,"15min":142.327},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.56999999999999995,"5min":2.569,"15min":1.1479999999999999},"min":{"1min":0.28999999999999998,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":1.7629999999999999,"5min":142.13200000000001,"15min":142.13200000000001},"last":0.64400000000000002}]},{"osd":6,"last update":"Tue Sep 17 07:11:56 2024","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.995,"15min":0.61699999999999999},"min":{"1min":0.30399999999999999,"5min":0.23499999999999999,"15min":0.23200000000000001},"max":{"1min":3.637,"5min":8.3149999999999995,"15min":8.3149999999999995},"last":1.8580000000000001},{"interface":"front","average":{"1min":1.1479999999999999,"5min":0.98399999999999999,"15min":0.63900000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.24099999999999999},"max":{"1min":6.4459999999999997,"5min":6.4459999999999997,"15min":6.4459999999999997},"last":0.67100000000000004}]},{"osd":7,"last update":"Tue Sep 17 07:12:16 2024","interfaces":[{"interface":"back","average":{"1min":1.1930000000000001,"5min":1.0980000000000001,"15min":0.69699999999999995},"min":{"1min":0.34200000000000003,"5min":0.29199999999999998,"15min":0.29099999999999998},"max":{"1min":6.4130000000000003,"5min":6.4130000000000003,"15min":6.4130000000000003},"last":1.73},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.94299999999999995,"15min":0.628},"min":{"1min":0.32400000000000001,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":3.8530000000000002,"5min":5.641,"15min":5.641},"last":2.0289999999999999}]}]},{"osd":4,"up_from":33,"seq":141733920916,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1508272,"kb_used_data":1126056,"kb_used_omap":1,"kb_used_meta":382206,"kb_avail":92236880,"statfs":{"total":95995035648,"available":94450565120,"internally_reserved":0,"allocated":1153081344,"data_stored":1152569310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":391379398},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:12:01 2024","interfaces":[{"interface":"back","average":{"1min":0.96099999999999997,"5min":2.323,"15min":1.8540000000000001},"min":{"1min":0.29799999999999999,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":2.4470000000000001,"5min":58.097999999999999,"15min":74.551000000000002},"last":0.90700000000000003},{"interface":"front","average":{"1min":0.85099999999999998,"5min":2.577,"15min":2.012},"min":{"1min":0.21299999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":3.2280000000000002,"5min":58.383000000000003,"15min":74.778999999999996},"last":0.53100000000000003}]},{"osd":1,"last update":"Tue Sep 17 07:12:01 2024","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":2.7360000000000002,"15min":1.3140000000000001},"min":{"1min":0.251,"5min":0.23300000000000001,"15min":0.23000000000000001},"max":{"1min":6.9729999999999999,"5min":54.914999999999999,"15min":54.914999999999999},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.69599999999999995,"5min":2.3809999999999998,"15min":1.2170000000000001},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.22800000000000001},"max":{"1min":3.073,"5min":53.725000000000001,"15min":53.725000000000001},"last":0.86799999999999999}]},{"osd":2,"last update":"Tue Sep 17 07:12:01 2024","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":0.98499999999999999,"15min":0.94199999999999995},"min":{"1min":0.26900000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":6.8929999999999998,"5min":6.8929999999999998,"15min":49.651000000000003},"last":0.84299999999999997},{"interface":"front","average":{"1min":1.7110000000000001,"5min":1.1479999999999999,"15min":1.016},"min":{"1min":0.35199999999999998,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":16.003,"5min":16.003,"15min":48.813000000000002},"last":0.315}]},{"osd":3,"last update":"Tue Sep 17 07:12:01 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":1.133,"15min":0.98099999999999998},"min":{"1min":0.28699999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":3.2080000000000002,"5min":17.872,"15min":39.042999999999999},"last":1.327},{"interface":"front","average":{"1min":1.4330000000000001,"5min":1.2210000000000001,"15min":1.012},"min":{"1min":0.29899999999999999,"5min":0.253,"15min":0.253},"max":{"1min":15.970000000000001,"5min":17.908000000000001,"15min":38.920000000000002},"last":0.40899999999999997}]},{"osd":5,"last update":"Tue Sep 17 07:12:32 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":1.1160000000000001,"15min":0.65700000000000003},"min":{"1min":0.28100000000000003,"5min":0.222,"15min":0.20799999999999999},"max":{"1min":3.2480000000000002,"5min":11.486000000000001,"15min":11.486000000000001},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.94699999999999995,"5min":1.252,"15min":0.71299999999999997},"min":{"1min":0.32500000000000001,"5min":0.26600000000000001,"15min":0.26100000000000001},"max":{"1min":3.806,"5min":11.781000000000001,"15min":11.781000000000001},"last":1.375}]},{"osd":6,"last update":"Tue Sep 17 07:11:58 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":1.349,"15min":0.74399999999999999},"min":{"1min":0.26200000000000001,"5min":0.218,"15min":0.218},"max":{"1min":2.7010000000000001,"5min":47.923999999999999,"15min":47.923999999999999},"last":0.74399999999999999},{"interface":"front","average":{"1min":1.2569999999999999,"5min":1.4490000000000001,"15min":0.77800000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26300000000000001},"max":{"1min":6.2140000000000004,"5min":47.600999999999999,"15min":47.600999999999999},"last":0.88800000000000001}]},{"osd":7,"last update":"Tue Sep 17 07:12:23 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":1.427,"15min":0.82299999999999995},"min":{"1min":0.315,"5min":0.29299999999999998,"15min":0.27600000000000002},"max":{"1min":3.1509999999999998,"5min":48.719000000000001,"15min":48.719000000000001},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.74399999999999999,"5min":1.375,"15min":0.79500000000000004},"min":{"1min":0.314,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":2.8639999999999999,"5min":50.527000000000001,"15min":50.527000000000001},"last":1.5760000000000001}]}]},{"osd":3,"up_from":0,"seq":115964117142,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":18,"seq":77309411484,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1963632,"kb_used_data":1437032,"kb_used_omap":1,"kb_used_meta":526590,"kb_avail":91781520,"statfs":{"total":95995035648,"available":93984276480,"internally_reserved":0,"allocated":1471520768,"data_stored":1471009578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":539228614},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:12:38 2024","interfaces":[{"interface":"back","average":{"1min":0.30599999999999999,"5min":1.885,"15min":1.0629999999999999},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.70499999999999996,"5min":79.715000000000003,"15min":79.715000000000003},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.48399999999999999,"5min":1.931,"15min":1.22},"min":{"1min":0.26000000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":1.127,"5min":79.691999999999993,"15min":79.691999999999993},"last":0.307}]},{"osd":1,"last update":"Tue Sep 17 07:12:38 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":2.6309999999999998,"15min":1.3089999999999999},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":1.47,"5min":79.659000000000006,"15min":79.659000000000006},"last":15.717000000000001},{"interface":"front","average":{"1min":0.55200000000000005,"5min":2.613,"15min":1.2809999999999999},"min":{"1min":0.22900000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":1.9610000000000001,"5min":80.155000000000001,"15min":80.155000000000001},"last":15.640000000000001}]},{"osd":3,"last update":"Tue Sep 17 07:12:51 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":1.4870000000000001,"15min":0.93600000000000005},"min":{"1min":0.25600000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.90700000000000003,"5min":23.43,"15min":23.43},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.40899999999999997,"5min":1.377,"15min":1.0329999999999999},"min":{"1min":0.28000000000000003,"5min":0.223,"15min":0.223},"max":{"1min":0.88,"5min":23.408000000000001,"15min":23.408000000000001},"last":0.36199999999999999}]},{"osd":4,"last update":"Tue Sep 17 07:12:15 2024","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":1.012,"15min":0.81899999999999995},"min":{"1min":0.40799999999999997,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":4.867,"5min":5.944,"15min":5.944},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.94399999999999995,"5min":1.109,"15min":0.81899999999999995},"min":{"1min":0.43099999999999999,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":6.8529999999999998,"5min":6.8529999999999998,"15min":6.8529999999999998},"last":0.66800000000000004}]},{"osd":5,"last update":"Tue Sep 17 07:12:38 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":1.2050000000000001,"15min":0.93600000000000005},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.249},"max":{"1min":1.409,"5min":8.8870000000000005,"15min":8.8870000000000005},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.64100000000000001,"5min":1.085,"15min":0.84399999999999997},"min":{"1min":0.29399999999999998,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":1.7949999999999999,"5min":7.1790000000000003,"15min":7.1790000000000003},"last":0.67600000000000005}]},{"osd":6,"last update":"Tue Sep 17 07:12:51 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":2.306,"15min":1.2709999999999999},"min":{"1min":0.36599999999999999,"5min":0.29999999999999999,"15min":0.28999999999999998},"max":{"1min":1.7210000000000001,"5min":101.587,"15min":101.587},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.58799999999999997,"5min":2.274,"15min":1.3129999999999999},"min":{"1min":0.38800000000000001,"5min":0.30599999999999999,"15min":0.28299999999999997},"max":{"1min":0.98899999999999999,"5min":100.696,"15min":100.696},"last":0.65900000000000003}]},{"osd":7,"last update":"Tue Sep 17 07:12:30 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":1.0069999999999999,"15min":0.82699999999999996},"min":{"1min":0.35599999999999998,"5min":0.316,"15min":0.316},"max":{"1min":3.052,"5min":6.9450000000000003,"15min":6.9450000000000003},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.66100000000000003,"5min":1.0469999999999999,"15min":0.94699999999999995},"min":{"1min":0.40999999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.0740000000000001,"5min":6.9219999999999997,"15min":9.2919999999999998},"last":0.70499999999999996}]}]},{"osd":1,"up_from":13,"seq":55834575009,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2935572,"kb_used_data":2571404,"kb_used_omap":1,"kb_used_meta":364158,"kb_avail":90809580,"statfs":{"total":95995035648,"available":92989009920,"internally_reserved":0,"allocated":2633117696,"data_stored":2632570796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":372898246},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:11:56 2024","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":1.667,"15min":1.036},"min":{"1min":0.192,"5min":0.192,"15min":0.155},"max":{"1min":3.6030000000000002,"5min":22.521000000000001,"15min":22.521000000000001},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.70799999999999996,"5min":1.333,"15min":0.95499999999999996},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.17000000000000001},"max":{"1min":4.335,"5min":22.471,"15min":22.471},"last":0.27300000000000002}]},{"osd":2,"last update":"Tue Sep 17 07:12:32 2024","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":1.1910000000000001,"15min":0.93600000000000005},"min":{"1min":0.23999999999999999,"5min":0.183,"15min":0.183},"max":{"1min":3.8929999999999998,"5min":6.2130000000000001,"15min":11.055999999999999},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.96699999999999997,"15min":0.89400000000000002},"min":{"1min":0.245,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":3.9199999999999999,"5min":5.242,"15min":9.3160000000000007},"last":0.61599999999999999}]},{"osd":3,"last update":"Tue Sep 17 07:11:56 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":1.1000000000000001,"15min":0.89600000000000002},"min":{"1min":0.308,"5min":0.223,"15min":0.219},"max":{"1min":4.1269999999999998,"5min":8.5510000000000002,"15min":8.5510000000000002},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.48999999999999999,"5min":1.405,"15min":0.97999999999999998},"min":{"1min":0.246,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":2.1869999999999998,"5min":8.6400000000000006,"15min":19.602},"last":0.48799999999999999}]},{"osd":4,"last update":"Tue Sep 17 07:12:14 2024","interfaces":[{"interface":"back","average":{"1min":1.3069999999999999,"5min":1.278,"15min":0.95699999999999996},"min":{"1min":0.374,"5min":0.25,"15min":0.25},"max":{"1min":4.71,"5min":9.8480000000000008,"15min":9.8480000000000008},"last":0.67600000000000005},{"interface":"front","average":{"1min":1.085,"5min":1.585,"15min":1.0349999999999999},"min":{"1min":0.32500000000000001,"5min":0.317,"15min":0.307},"max":{"1min":3.4369999999999998,"5min":27.097999999999999,"15min":27.097999999999999},"last":0.63300000000000001}]},{"osd":5,"last update":"Tue Sep 17 07:12:32 2024","interfaces":[{"interface":"back","average":{"1min":1.0189999999999999,"5min":1.444,"15min":1.034},"min":{"1min":0.33400000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":4.0110000000000001,"5min":16.84,"15min":16.84},"last":1.5669999999999999},{"interface":"front","average":{"1min":0.69799999999999995,"5min":1.5600000000000001,"15min":1.0620000000000001},"min":{"1min":0.29199999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":3.0249999999999999,"5min":16.949999999999999,"15min":16.949999999999999},"last":1.4279999999999999}]},{"osd":6,"last update":"Tue Sep 17 07:11:56 2024","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.5900000000000001,"15min":1.0009999999999999},"min":{"1min":0.34499999999999997,"5min":0.308,"15min":0.308},"max":{"1min":6.8040000000000003,"5min":9.1359999999999992,"15min":9.1359999999999992},"last":0.79900000000000004},{"interface":"front","average":{"1min":1.302,"5min":1.238,"15min":0.92200000000000004},"min":{"1min":0.39800000000000002,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":4.5810000000000004,"5min":7.1130000000000004,"15min":7.1130000000000004},"last":0.74099999999999999}]},{"osd":7,"last update":"Tue Sep 17 07:12:16 2024","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":1.5900000000000001,"15min":1.0529999999999999},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":4.5759999999999996,"5min":9.8179999999999996,"15min":9.8179999999999996},"last":0.66400000000000003},{"interface":"front","average":{"1min":1.3100000000000001,"5min":1.4870000000000001,"15min":1.026},"min":{"1min":0.35999999999999999,"5min":0.34100000000000003,"15min":0.32900000000000001},"max":{"1min":4.1600000000000001,"5min":6.2590000000000003,"15min":11.007999999999999},"last":0.68999999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738532,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2223040,"kb_used_data":1736632,"kb_used_omap":1,"kb_used_meta":486398,"kb_avail":91522112,"statfs":{"total":95995035648,"available":93718642688,"internally_reserved":0,"allocated":1778311168,"data_stored":1777784338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":498072006},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 17 07:12:05 2024","interfaces":[{"interface":"back","average":{"1min":1.706,"5min":1.2250000000000001,"15min":0.85899999999999999},"min":{"1min":0.19500000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":21.876000000000001,"5min":21.876000000000001,"15min":21.876000000000001},"last":1.71},{"interface":"front","average":{"1min":2.242,"5min":1.657,"15min":0.98999999999999999},"min":{"1min":0.28899999999999998,"5min":0.158,"15min":0.158},"max":{"1min":22.521999999999998,"5min":22.521999999999998,"15min":22.521999999999998},"last":1.3160000000000001}]},{"osd":2,"last update":"Tue Sep 17 07:12:33 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":1.095,"15min":1.0860000000000001},"min":{"1min":0.23499999999999999,"5min":0.188,"15min":0.188},"max":{"1min":2.3690000000000002,"5min":6.3799999999999999,"15min":18.643999999999998},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.67000000000000004,"5min":1.002,"15min":0.96799999999999997},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":2.173,"5min":5.8440000000000003,"15min":17.972999999999999},"last":0.73699999999999999}]},{"osd":3,"last update":"Tue Sep 17 07:12:05 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":1.137,"15min":0.97099999999999997},"min":{"1min":0.255,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":4.4619999999999997,"5min":11.598000000000001,"15min":18.629000000000001},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.95599999999999996,"15min":1.0069999999999999},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.23699999999999999},"max":{"1min":0.96899999999999997,"5min":5.4169999999999998,"15min":22.358000000000001},"last":0.29999999999999999}]},{"osd":4,"last update":"Tue Sep 17 07:12:13 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":1.042,"15min":1.0249999999999999},"min":{"1min":0.39400000000000002,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.611,"5min":19.146999999999998,"15min":19.146999999999998},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.60899999999999999,"5min":1.171,"15min":0.97499999999999998},"min":{"1min":0.28000000000000003,"5min":0.27400000000000002,"15min":0.255},"max":{"1min":1.621,"5min":19.158999999999999,"15min":19.158999999999999},"last":0.874}]},{"osd":5,"last update":"Tue Sep 17 07:12:43 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":1.284,"15min":1.0569999999999999},"min":{"1min":0.39800000000000002,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":1.8360000000000001,"5min":18.841000000000001,"15min":18.841000000000001},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.98299999999999998,"5min":1.452,"15min":1.0740000000000001},"min":{"1min":0.44600000000000001,"5min":0.30199999999999999,"15min":0.28599999999999998},"max":{"1min":2.5219999999999998,"5min":19.248000000000001,"15min":19.248000000000001},"last":1.0720000000000001}]},{"osd":6,"last update":"Tue Sep 17 07:12:05 2024","interfaces":[{"interface":"back","average":{"1min":1.149,"5min":1.1100000000000001,"15min":0.94699999999999995},"min":{"1min":0.376,"5min":0.31900000000000001,"15min":0.30399999999999999},"max":{"1min":4.3630000000000004,"5min":5.2240000000000002,"15min":18.602},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.81399999999999995,"5min":1.073,"15min":0.996},"min":{"1min":0.42299999999999999,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":2.3170000000000002,"5min":6.149,"15min":18.692},"last":0.66100000000000003}]},{"osd":7,"last update":"Tue Sep 17 07:12:18 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":1.085,"15min":1.3879999999999999},"min":{"1min":0.38400000000000001,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":2.0150000000000001,"5min":4.8490000000000002,"15min":46.183},"last":0.94099999999999995},{"interface":"front","average":{"1min":0.97899999999999998,"5min":1.1299999999999999,"15min":1.1240000000000001},"min":{"1min":0.46600000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":4.4029999999999996,"5min":5.0190000000000001,"15min":45.914999999999999},"last":0.67700000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1795686400,"data_stored":1795686400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2651062272,"data_stored":2651062272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1473445888,"data_stored":1473445888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1170079744,"data_stored":1170079744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2929131520,"data_stored":2929131520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":875954176,"data_stored":875954176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3227844608,"data_stored":3227844608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T07:13:03.315 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd pool set unique_pool_2 pgp_num 16 2024-09-17T07:13:03.380 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:13:03.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:03.420 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:03.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:13:03.447 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:03 smithi100 ceph-mon[29092]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:13:03.447 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:03 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T07:13:03.447 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:03 smithi100 ceph-mon[29092]: osdmap e256: 8 total, 8 up, 7 in 2024-09-17T07:13:03.447 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:03 smithi100 ceph-mon[29092]: 6.9 scrub starts 2024-09-17T07:13:03.447 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:03 smithi100 ceph-mon[29092]: 6.9 scrub ok 2024-09-17T07:13:03.447 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:03 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:13:03.447 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:03 smithi100 ceph-mon[39462]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:13:03.448 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:03 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T07:13:03.448 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:03 smithi100 ceph-mon[39462]: osdmap e256: 8 total, 8 up, 7 in 2024-09-17T07:13:03.448 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:03 smithi100 ceph-mon[39462]: 6.9 scrub starts 2024-09-17T07:13:03.448 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:03 smithi100 ceph-mon[39462]: 6.9 scrub ok 2024-09-17T07:13:03.448 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:03 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:13:03.448 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:13:03.448 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:13:03.483 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1: {} 2024-09-17T07:13:03.484 INFO:teuthology.orchestra.run.smithi100.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:13:03.518 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:13:03.518 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:13:03.522 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:13:03.548 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:13:03.548 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:13:03.563 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:03.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:03.565 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:13:03.582 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:13:03.582 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:13:03.611 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:13:03.612 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:13:03.640 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:13:03.640 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:13:03.666 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:13:03.681 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:13:03.681 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:13:03.705 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:03.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:03.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:13:03.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:03 smithi135 ceph-mon[33238]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:13:03.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:03 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T07:13:03.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:03 smithi135 ceph-mon[33238]: osdmap e256: 8 total, 8 up, 7 in 2024-09-17T07:13:03.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:03 smithi135 ceph-mon[33238]: 6.9 scrub starts 2024-09-17T07:13:03.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:03 smithi135 ceph-mon[33238]: 6.9 scrub ok 2024-09-17T07:13:03.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:03 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:13:03.807 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:13:03.843 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:03.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:03.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:13:03.945 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:13:03.974 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:03.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:03.975 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:13:04.076 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:13:04.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:04.108 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:04.109 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:13:04.210 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:13:04.219 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 47 16 85821 85805 114.086 81.875 0.0146442 0.00876057 2024-09-17T07:13:04.237 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:04.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:04.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:13:04.339 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:13:04.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:04.370 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:04.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:13:04.471 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:13:04.499 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:04.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:04.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:13:04.601 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:13:04.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:04 smithi100 ceph-mon[29092]: 5.2 scrub starts 2024-09-17T07:13:04.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:04 smithi100 ceph-mon[29092]: 5.2 scrub ok 2024-09-17T07:13:04.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:04 smithi100 ceph-mon[29092]: pgmap v489: 148 pgs: 148 active+clean; 4.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.0 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-09-17T07:13:04.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:04 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/879821941' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:13:04.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:04 smithi100 ceph-mon[29092]: 5.4 scrub starts 2024-09-17T07:13:04.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:04 smithi100 ceph-mon[29092]: 5.4 scrub ok 2024-09-17T07:13:04.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:04 smithi100 ceph-mon[39462]: 5.2 scrub starts 2024-09-17T07:13:04.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:04 smithi100 ceph-mon[39462]: 5.2 scrub ok 2024-09-17T07:13:04.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:04 smithi100 ceph-mon[39462]: pgmap v489: 148 pgs: 148 active+clean; 4.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.0 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-09-17T07:13:04.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:04 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/879821941' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:13:04.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:04 smithi100 ceph-mon[39462]: 5.4 scrub starts 2024-09-17T07:13:04.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:04 smithi100 ceph-mon[39462]: 5.4 scrub ok 2024-09-17T07:13:04.635 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:04.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:04.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:13:04.738 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:13:04.767 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:04.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:04.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:13:04.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:04 smithi135 ceph-mon[33238]: 5.2 scrub starts 2024-09-17T07:13:04.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:04 smithi135 ceph-mon[33238]: 5.2 scrub ok 2024-09-17T07:13:04.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:04 smithi135 ceph-mon[33238]: pgmap v489: 148 pgs: 148 active+clean; 4.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.0 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-09-17T07:13:04.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:04 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/879821941' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:13:04.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:04 smithi135 ceph-mon[33238]: 5.4 scrub starts 2024-09-17T07:13:04.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:04 smithi135 ceph-mon[33238]: 5.4 scrub ok 2024-09-17T07:13:04.870 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:13:04.898 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:04.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:04.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:13:05.000 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:13:05.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:05.033 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:05.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:13:05.134 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:13:05.167 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:05.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:05.169 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:13:05.212 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-17T07:13:05.220 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 48 16 87138 87122 113.423 82.3125 0.0156893 0.00881167 2024-09-17T07:13:05.236 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:13:05.270 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:13:05.323 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:05.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:05.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:13:05.425 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:13:05.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:05.499 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:05.499 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:13:05.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:05 smithi135 ceph-mon[33238]: 2.15 scrub starts 2024-09-17T07:13:05.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:05 smithi135 ceph-mon[33238]: 2.15 scrub ok 2024-09-17T07:13:05.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:05 smithi135 ceph-mon[33238]: 5.1e scrub starts 2024-09-17T07:13:05.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:05 smithi135 ceph-mon[33238]: 2.f scrub starts 2024-09-17T07:13:05.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:05 smithi135 ceph-mon[33238]: 2.f scrub ok 2024-09-17T07:13:05.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:05 smithi135 ceph-mon[33238]: 5.1e scrub ok 2024-09-17T07:13:05.593 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:13:05.600 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:13:05.673 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:05 smithi100 ceph-mon[29092]: 2.15 scrub starts 2024-09-17T07:13:05.673 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:05 smithi100 ceph-mon[29092]: 2.15 scrub ok 2024-09-17T07:13:05.674 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:05 smithi100 ceph-mon[29092]: 5.1e scrub starts 2024-09-17T07:13:05.674 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:05 smithi100 ceph-mon[29092]: 2.f scrub starts 2024-09-17T07:13:05.674 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:05 smithi100 ceph-mon[29092]: 2.f scrub ok 2024-09-17T07:13:05.674 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:05 smithi100 ceph-mon[29092]: 5.1e scrub ok 2024-09-17T07:13:05.674 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:05 smithi100 ceph-mon[39462]: 2.15 scrub starts 2024-09-17T07:13:05.674 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:05 smithi100 ceph-mon[39462]: 2.15 scrub ok 2024-09-17T07:13:05.674 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:05 smithi100 ceph-mon[39462]: 5.1e scrub starts 2024-09-17T07:13:05.674 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:05 smithi100 ceph-mon[39462]: 2.f scrub starts 2024-09-17T07:13:05.674 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:05 smithi100 ceph-mon[39462]: 2.f scrub ok 2024-09-17T07:13:05.674 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:05 smithi100 ceph-mon[39462]: 5.1e scrub ok 2024-09-17T07:13:05.714 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:05.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:05.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:13:05.816 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:13:05.852 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:05.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:05.853 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:13:05.955 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:13:05.995 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:05.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:05.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:13:06.097 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:13:06.127 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:13:05 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:13:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:13:06.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:06.128 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:06.128 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:13:06.220 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 49 16 88326 88310 112.623 74.25 0.00592691 0.00887426 2024-09-17T07:13:06.229 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:13:06.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:06.263 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:06.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:13:06.364 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:13:06.393 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:06.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:06.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:13:06.424 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:06 smithi100 ceph-mon[29092]: pgmap v490: 148 pgs: 148 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 853 B/s rd, 85 MiB/s wr, 1.35k op/s 2024-09-17T07:13:06.424 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:06 smithi100 ceph-mon[29092]: 4.10 scrub starts 2024-09-17T07:13:06.425 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:06 smithi100 ceph-mon[29092]: 4.10 scrub ok 2024-09-17T07:13:06.425 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:06 smithi100 ceph-mon[29092]: 3.1 scrub starts 2024-09-17T07:13:06.425 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:06 smithi100 ceph-mon[29092]: 3.1 scrub ok 2024-09-17T07:13:06.425 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:06 smithi100 ceph-mon[29092]: 6.a scrub starts 2024-09-17T07:13:06.425 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:06 smithi100 ceph-mon[29092]: 6.a scrub ok 2024-09-17T07:13:06.425 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:06 smithi100 ceph-mon[39462]: pgmap v490: 148 pgs: 148 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 853 B/s rd, 85 MiB/s wr, 1.35k op/s 2024-09-17T07:13:06.425 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:06 smithi100 ceph-mon[39462]: 4.10 scrub starts 2024-09-17T07:13:06.425 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:06 smithi100 ceph-mon[39462]: 4.10 scrub ok 2024-09-17T07:13:06.425 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:06 smithi100 ceph-mon[39462]: 3.1 scrub starts 2024-09-17T07:13:06.425 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:06 smithi100 ceph-mon[39462]: 3.1 scrub ok 2024-09-17T07:13:06.426 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:06 smithi100 ceph-mon[39462]: 6.a scrub starts 2024-09-17T07:13:06.426 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:06 smithi100 ceph-mon[39462]: 6.a scrub ok 2024-09-17T07:13:06.495 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:13:06.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:06.525 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:06.525 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:13:06.626 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:13:06.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:06.656 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:06.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:13:06.713 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:13:06.757 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:13:06.771 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:06 smithi135 ceph-mon[33238]: pgmap v490: 148 pgs: 148 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 853 B/s rd, 85 MiB/s wr, 1.35k op/s 2024-09-17T07:13:06.771 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:06 smithi135 ceph-mon[33238]: 4.10 scrub starts 2024-09-17T07:13:06.772 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:06 smithi135 ceph-mon[33238]: 4.10 scrub ok 2024-09-17T07:13:06.772 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:06 smithi135 ceph-mon[33238]: 3.1 scrub starts 2024-09-17T07:13:06.772 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:06 smithi135 ceph-mon[33238]: 3.1 scrub ok 2024-09-17T07:13:06.772 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:06 smithi135 ceph-mon[33238]: 6.a scrub starts 2024-09-17T07:13:06.772 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:06 smithi135 ceph-mon[33238]: 6.a scrub ok 2024-09-17T07:13:06.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:06.797 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:06.797 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:13:06.898 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:13:06.937 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:06.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:06.939 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:13:07.040 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:13:07.072 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:07.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:07.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:13:07.173 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:13:07.220 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 50 16 89602 89586 111.966 79.75 0.0238118 0.00892607 2024-09-17T07:13:07.221 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:07.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:07.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:13:07.323 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:13:07.352 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:07.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:07.353 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:13:07.454 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:13:07.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:07.478 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:07.478 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:13:07.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:07 smithi135 ceph-mon[33238]: 5.11 scrub starts 2024-09-17T07:13:07.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:07 smithi135 ceph-mon[33238]: 5.11 scrub ok 2024-09-17T07:13:07.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:07 smithi135 ceph-mon[33238]: 5.1 scrub starts 2024-09-17T07:13:07.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:07 smithi135 ceph-mon[33238]: 5.1 scrub ok 2024-09-17T07:13:07.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:07 smithi135 ceph-mon[33238]: 6.13 scrub starts 2024-09-17T07:13:07.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:07 smithi135 ceph-mon[33238]: 6.13 scrub ok 2024-09-17T07:13:07.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:07 smithi135 ceph-mon[33238]: 4.8 scrub starts 2024-09-17T07:13:07.545 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:07 smithi135 ceph-mon[33238]: 4.8 scrub ok 2024-09-17T07:13:07.545 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:07 smithi135 ceph-mon[33238]: pgmap v491: 148 pgs: 148 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 726 B/s rd, 57 MiB/s wr, 913 op/s 2024-09-17T07:13:07.580 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:13:07.611 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:07.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:07.613 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:13:07.613 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:07 smithi100 ceph-mon[29092]: 5.11 scrub starts 2024-09-17T07:13:07.613 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:07 smithi100 ceph-mon[29092]: 5.11 scrub ok 2024-09-17T07:13:07.614 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:07 smithi100 ceph-mon[29092]: 5.1 scrub starts 2024-09-17T07:13:07.614 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:07 smithi100 ceph-mon[29092]: 5.1 scrub ok 2024-09-17T07:13:07.614 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:07 smithi100 ceph-mon[29092]: 6.13 scrub starts 2024-09-17T07:13:07.614 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:07 smithi100 ceph-mon[29092]: 6.13 scrub ok 2024-09-17T07:13:07.614 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:07 smithi100 ceph-mon[29092]: 4.8 scrub starts 2024-09-17T07:13:07.614 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:07 smithi100 ceph-mon[29092]: 4.8 scrub ok 2024-09-17T07:13:07.614 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:07 smithi100 ceph-mon[29092]: pgmap v491: 148 pgs: 148 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 726 B/s rd, 57 MiB/s wr, 913 op/s 2024-09-17T07:13:07.614 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:07 smithi100 ceph-mon[39462]: 5.11 scrub starts 2024-09-17T07:13:07.614 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:07 smithi100 ceph-mon[39462]: 5.11 scrub ok 2024-09-17T07:13:07.614 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:07 smithi100 ceph-mon[39462]: 5.1 scrub starts 2024-09-17T07:13:07.614 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:07 smithi100 ceph-mon[39462]: 5.1 scrub ok 2024-09-17T07:13:07.615 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:07 smithi100 ceph-mon[39462]: 6.13 scrub starts 2024-09-17T07:13:07.615 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:07 smithi100 ceph-mon[39462]: 6.13 scrub ok 2024-09-17T07:13:07.615 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:07 smithi100 ceph-mon[39462]: 4.8 scrub starts 2024-09-17T07:13:07.615 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:07 smithi100 ceph-mon[39462]: 4.8 scrub ok 2024-09-17T07:13:07.615 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:07 smithi100 ceph-mon[39462]: pgmap v491: 148 pgs: 148 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 726 B/s rd, 57 MiB/s wr, 913 op/s 2024-09-17T07:13:07.714 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:13:07.754 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:07.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:07.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:13:07.854 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-17T07:13:07.856 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:13:07.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:07.882 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:07.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:13:07.983 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:13:08.004 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:08.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:08.005 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:13:08.089 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:13:08.106 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:13:08.141 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:08.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:08.142 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:13:08.220 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 51 16 90574 90558 110.962 60.75 0.00550313 0.00900701 2024-09-17T07:13:08.242 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:13:08.288 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:08.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:08.289 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:13:08.389 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:13:08.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:08.431 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:08.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:13:08.497 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:08 smithi100 ceph-mon[29092]: 6.4 scrub starts 2024-09-17T07:13:08.497 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:08 smithi100 ceph-mon[29092]: 6.4 scrub ok 2024-09-17T07:13:08.497 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:08 smithi100 ceph-mon[29092]: 2.1e scrub starts 2024-09-17T07:13:08.497 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:08 smithi100 ceph-mon[29092]: 2.1e scrub ok 2024-09-17T07:13:08.498 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:08 smithi100 ceph-mon[39462]: 6.4 scrub starts 2024-09-17T07:13:08.498 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:08 smithi100 ceph-mon[39462]: 6.4 scrub ok 2024-09-17T07:13:08.498 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:08 smithi100 ceph-mon[39462]: 2.1e scrub starts 2024-09-17T07:13:08.498 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:08 smithi100 ceph-mon[39462]: 2.1e scrub ok 2024-09-17T07:13:08.532 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:13:08.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:08 smithi135 ceph-mon[33238]: 6.4 scrub starts 2024-09-17T07:13:08.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:08 smithi135 ceph-mon[33238]: 6.4 scrub ok 2024-09-17T07:13:08.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:08 smithi135 ceph-mon[33238]: 2.1e scrub starts 2024-09-17T07:13:08.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:08 smithi135 ceph-mon[33238]: 2.1e scrub ok 2024-09-17T07:13:08.562 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:08.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:08.564 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:13:08.577 INFO:teuthology.orchestra.run.smithi135.stdout:{ 2024-09-17T07:13:08.578 INFO:teuthology.orchestra.run.smithi135.stdout: "ops": [ 2024-09-17T07:13:08.578 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:08.578 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.15330.0:90725 9.c 9:3e266b5f:::benchmark_data_smithi100_128631_object90724:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e256)", 2024-09-17T07:13:08.578 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:13:08.363837+0000", 2024-09-17T07:13:08.578 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 0.00048942300000000001, 2024-09-17T07:13:08.578 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00052264900000000003, 2024-09-17T07:13:08.578 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:13:08.578 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "waiting for sub ops", 2024-09-17T07:13:08.578 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:13:08.578 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.15330", 2024-09-17T07:13:08.578 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/1361381749", 2024-09-17T07:13:08.578 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 90725 2024-09-17T07:13:08.579 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:08.579 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:13:08.579 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:08.579 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:13:08.579 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:08.363837+0000", 2024-09-17T07:13:08.579 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:08.579 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:08.579 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:08.579 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:13:08.579 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:08.363837+0000", 2024-09-17T07:13:08.579 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:08.579 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:08.579 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:08.580 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:13:08.580 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:08.363828+0000", 2024-09-17T07:13:08.580 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999905 2024-09-17T07:13:08.580 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:08.580 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:08.580 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:13:08.580 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:08.363868+0000", 2024-09-17T07:13:08.580 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.0420999999999998e-05 2024-09-17T07:13:08.580 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:08.580 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:08.580 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:13:08.580 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:08.363880+0000", 2024-09-17T07:13:08.581 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.2193e-05 2024-09-17T07:13:08.581 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:08.581 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:08.581 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:13:08.581 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:08.363898+0000", 2024-09-17T07:13:08.581 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.7606000000000001e-05 2024-09-17T07:13:08.581 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:08.581 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:08.581 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:08.581 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:08.363955+0000", 2024-09-17T07:13:08.581 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.6483000000000001e-05 2024-09-17T07:13:08.581 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:08.581 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:08.582 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:13:08.582 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:08.364072+0000", 2024-09-17T07:13:08.582 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000117649 2024-09-17T07:13:08.582 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:08.582 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:08.582 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,7", 2024-09-17T07:13:08.582 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:08.364202+0000", 2024-09-17T07:13:08.582 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000129472 2024-09-17T07:13:08.582 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:08.582 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:13:08.582 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:08.582 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:08.582 INFO:teuthology.orchestra.run.smithi135.stdout: ], 2024-09-17T07:13:08.583 INFO:teuthology.orchestra.run.smithi135.stdout: "num_ops": 1 2024-09-17T07:13:08.583 INFO:teuthology.orchestra.run.smithi135.stdout:} 2024-09-17T07:13:08.664 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:13:08.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:08.695 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:08.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:13:08.797 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:13:08.829 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:08.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:08.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:13:08.931 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:13:08.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:08.963 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:08.963 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:13:09.043 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:13:08 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:13:09.064 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:13:09.099 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:09.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:09.100 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:13:09.201 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:13:09.220 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 52 16 91776 91760 110.272 75.125 0.0154694 0.00906458 2024-09-17T07:13:09.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:09.230 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:09.231 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:13:09.331 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:13:09.348 INFO:teuthology.orchestra.run.smithi100.stderr:set pool 9 pgp_num to 16 2024-09-17T07:13:09.373 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:09.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:09.375 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:13:09.476 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:13:09.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:09.506 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:09.507 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:13:09.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:09 smithi135 ceph-mon[33238]: 4.5 scrub starts 2024-09-17T07:13:09.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:09 smithi135 ceph-mon[33238]: 4.5 scrub ok 2024-09-17T07:13:09.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:09 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1881385594' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "16"}]: dispatch 2024-09-17T07:13:09.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:09 smithi135 ceph-mon[33238]: 5.13 scrub starts 2024-09-17T07:13:09.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:09 smithi135 ceph-mon[33238]: 5.13 scrub ok 2024-09-17T07:13:09.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:09 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:13:09.545 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:09 smithi135 ceph-mon[33238]: pgmap v492: 148 pgs: 148 active+clean; 5.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 86 MiB/s wr, 1.37k op/s 2024-09-17T07:13:09.607 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:13:09.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:09 smithi100 ceph-mon[39462]: 4.5 scrub starts 2024-09-17T07:13:09.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:09 smithi100 ceph-mon[39462]: 4.5 scrub ok 2024-09-17T07:13:09.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:09 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1881385594' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "16"}]: dispatch 2024-09-17T07:13:09.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:09 smithi100 ceph-mon[39462]: 5.13 scrub starts 2024-09-17T07:13:09.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:09 smithi100 ceph-mon[39462]: 5.13 scrub ok 2024-09-17T07:13:09.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:09 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:13:09.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:09 smithi100 ceph-mon[39462]: pgmap v492: 148 pgs: 148 active+clean; 5.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 86 MiB/s wr, 1.37k op/s 2024-09-17T07:13:09.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:09 smithi100 ceph-mon[29092]: 4.5 scrub starts 2024-09-17T07:13:09.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:09 smithi100 ceph-mon[29092]: 4.5 scrub ok 2024-09-17T07:13:09.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:09 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1881385594' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "16"}]: dispatch 2024-09-17T07:13:09.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:09 smithi100 ceph-mon[29092]: 5.13 scrub starts 2024-09-17T07:13:09.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:09 smithi100 ceph-mon[29092]: 5.13 scrub ok 2024-09-17T07:13:09.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:09 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:13:09.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:09 smithi100 ceph-mon[29092]: pgmap v492: 148 pgs: 148 active+clean; 5.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 86 MiB/s wr, 1.37k op/s 2024-09-17T07:13:09.640 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:09.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:09.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:13:09.742 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:13:09.789 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:09.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:09.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:13:09.799 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-17T07:13:09.891 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:13:09.916 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:09.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:09.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:13:10.018 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:13:10.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:10.046 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:10.046 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:13:10.148 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:13:10.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:10.201 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:10.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:13:10.220 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 53 16 92933 92917 109.556 72.3125 0.0130656 0.00912254 2024-09-17T07:13:10.225 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:13:10.303 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:13:10.333 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:10.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:10.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:13:10.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:10 smithi100 ceph-mon[29092]: 4.1c scrub starts 2024-09-17T07:13:10.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:10 smithi100 ceph-mon[29092]: 4.1c scrub ok 2024-09-17T07:13:10.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:10 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1881385594' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "16"}]': finished 2024-09-17T07:13:10.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:10 smithi100 ceph-mon[29092]: osdmap e257: 8 total, 8 up, 7 in 2024-09-17T07:13:10.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:10 smithi100 ceph-mon[29092]: 5.5 scrub starts 2024-09-17T07:13:10.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:10 smithi100 ceph-mon[29092]: 5.5 scrub ok 2024-09-17T07:13:10.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:10 smithi100 ceph-mon[29092]: 6.19 scrub starts 2024-09-17T07:13:10.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:10 smithi100 ceph-mon[29092]: 6.19 scrub ok 2024-09-17T07:13:10.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:10 smithi100 ceph-mon[39462]: 4.1c scrub starts 2024-09-17T07:13:10.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:10 smithi100 ceph-mon[39462]: 4.1c scrub ok 2024-09-17T07:13:10.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:10 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1881385594' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "16"}]': finished 2024-09-17T07:13:10.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:10 smithi100 ceph-mon[39462]: osdmap e257: 8 total, 8 up, 7 in 2024-09-17T07:13:10.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:10 smithi100 ceph-mon[39462]: 5.5 scrub starts 2024-09-17T07:13:10.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:10 smithi100 ceph-mon[39462]: 5.5 scrub ok 2024-09-17T07:13:10.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:10 smithi100 ceph-mon[39462]: 6.19 scrub starts 2024-09-17T07:13:10.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:10 smithi100 ceph-mon[39462]: 6.19 scrub ok 2024-09-17T07:13:10.435 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:13:10.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:10.474 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:10.474 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:13:10.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:10 smithi135 ceph-mon[33238]: 4.1c scrub starts 2024-09-17T07:13:10.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:10 smithi135 ceph-mon[33238]: 4.1c scrub ok 2024-09-17T07:13:10.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:10 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1881385594' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "16"}]': finished 2024-09-17T07:13:10.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:10 smithi135 ceph-mon[33238]: osdmap e257: 8 total, 8 up, 7 in 2024-09-17T07:13:10.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:10 smithi135 ceph-mon[33238]: 5.5 scrub starts 2024-09-17T07:13:10.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:10 smithi135 ceph-mon[33238]: 5.5 scrub ok 2024-09-17T07:13:10.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:10 smithi135 ceph-mon[33238]: 6.19 scrub starts 2024-09-17T07:13:10.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:10 smithi135 ceph-mon[33238]: 6.19 scrub ok 2024-09-17T07:13:10.575 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:13:10.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:10.606 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:10.606 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:13:10.707 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:13:10.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:10.747 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:10.747 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:13:10.848 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:13:10.859 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:13:10.860 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1: {} 2024-09-17T07:13:10.860 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:13:10.860 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:13:10.860 INFO:teuthology.orchestra.run.smithi100.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:13:10.860 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:13:10.862 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:13:10.862 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:13:10.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:10.870 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:10.870 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:13:10.883 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:13:10.883 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:13:10.897 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:13:10.914 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:13:10.915 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:13:10.942 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:13:10.942 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:13:10.971 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:13:10.975 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:13:10.975 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:13:11.011 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:11.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:11.012 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:13:11.113 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:13:11.134 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:11.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:11.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:13:11.220 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 54 16 94011 93995 108.775 67.375 0.00734472 0.00918816 2024-09-17T07:13:11.236 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:13:11.266 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:11.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:11.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:13:11.368 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:13:11.396 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:11.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:11.397 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:13:11.498 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:13:11.525 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:11 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1518658827' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:13:11.525 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:11 smithi100 ceph-mon[29092]: 5.14 scrub starts 2024-09-17T07:13:11.525 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:11 smithi100 ceph-mon[29092]: 5.14 scrub ok 2024-09-17T07:13:11.525 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:11 smithi100 ceph-mon[29092]: pgmap v494: 148 pgs: 148 active+clean; 5.6 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1.1 KiB/s rd, 87 MiB/s wr, 1.40k op/s 2024-09-17T07:13:11.525 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:11 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1518658827' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:13:11.525 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:11 smithi100 ceph-mon[39462]: 5.14 scrub starts 2024-09-17T07:13:11.525 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:11 smithi100 ceph-mon[39462]: 5.14 scrub ok 2024-09-17T07:13:11.525 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:11 smithi100 ceph-mon[39462]: pgmap v494: 148 pgs: 148 active+clean; 5.6 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1.1 KiB/s rd, 87 MiB/s wr, 1.40k op/s 2024-09-17T07:13:11.528 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:11.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:11.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:13:11.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:11 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1518658827' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:13:11.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:11 smithi135 ceph-mon[33238]: 5.14 scrub starts 2024-09-17T07:13:11.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:11 smithi135 ceph-mon[33238]: 5.14 scrub ok 2024-09-17T07:13:11.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:11 smithi135 ceph-mon[33238]: pgmap v494: 148 pgs: 148 active+clean; 5.6 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1.1 KiB/s rd, 87 MiB/s wr, 1.40k op/s 2024-09-17T07:13:11.630 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:13:11.666 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:11.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:11.667 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:13:11.767 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:13:11.793 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:11.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:11.794 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:13:11.894 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:13:11.922 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:11.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:11.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:13:12.023 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:13:12.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:12.069 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:12.070 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:13:12.171 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:13:12.200 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:12.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:12.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:13:12.220 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 55 16 95159 95143 108.102 71.75 0.0102474 0.00924603 2024-09-17T07:13:12.302 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:13:12.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:12.335 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:12.335 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:13:12.436 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:13:12.481 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:12.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:12.482 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:13:12.583 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:13:12.606 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:12.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:12.607 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:13:12.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:12 smithi100 ceph-mon[29092]: 2.9 scrub starts 2024-09-17T07:13:12.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:12 smithi100 ceph-mon[29092]: 2.9 scrub ok 2024-09-17T07:13:12.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:12 smithi100 ceph-mon[29092]: 4.1b scrub starts 2024-09-17T07:13:12.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:12 smithi100 ceph-mon[29092]: 4.1b scrub ok 2024-09-17T07:13:12.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:12 smithi100 ceph-mon[29092]: 6.1b scrub starts 2024-09-17T07:13:12.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:12 smithi100 ceph-mon[29092]: 6.1b scrub ok 2024-09-17T07:13:12.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:12 smithi100 ceph-mon[39462]: 2.9 scrub starts 2024-09-17T07:13:12.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:12 smithi100 ceph-mon[39462]: 2.9 scrub ok 2024-09-17T07:13:12.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:12 smithi100 ceph-mon[39462]: 4.1b scrub starts 2024-09-17T07:13:12.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:12 smithi100 ceph-mon[39462]: 4.1b scrub ok 2024-09-17T07:13:12.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:12 smithi100 ceph-mon[39462]: 6.1b scrub starts 2024-09-17T07:13:12.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:12 smithi100 ceph-mon[39462]: 6.1b scrub ok 2024-09-17T07:13:12.708 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:13:12.730 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:12.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:12.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:13:12.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:12 smithi135 ceph-mon[33238]: 2.9 scrub starts 2024-09-17T07:13:12.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:12 smithi135 ceph-mon[33238]: 2.9 scrub ok 2024-09-17T07:13:12.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:12 smithi135 ceph-mon[33238]: 4.1b scrub starts 2024-09-17T07:13:12.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:12 smithi135 ceph-mon[33238]: 4.1b scrub ok 2024-09-17T07:13:12.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:12 smithi135 ceph-mon[33238]: 6.1b scrub starts 2024-09-17T07:13:12.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:12 smithi135 ceph-mon[33238]: 6.1b scrub ok 2024-09-17T07:13:12.832 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:13:12.861 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:12.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:12.862 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:13:12.963 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:13:12.992 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:12.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:12.994 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:13:13.094 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:13:13.123 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:13.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:13.124 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:13:13.220 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 56 16 96342 96326 107.491 73.9375 0.00750337 0.00929851 2024-09-17T07:13:13.225 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:13:13.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:13.269 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:13.269 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:13:13.370 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:13:13.407 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:13.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:13.408 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:13:13.508 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:13:13.555 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:13.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:13.556 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:13:13.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:13 smithi100 ceph-mon[39462]: 5.0 scrub starts 2024-09-17T07:13:13.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:13 smithi100 ceph-mon[39462]: 5.0 scrub ok 2024-09-17T07:13:13.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:13 smithi100 ceph-mon[39462]: 6.18 scrub starts 2024-09-17T07:13:13.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:13 smithi100 ceph-mon[39462]: 6.e scrub starts 2024-09-17T07:13:13.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:13 smithi100 ceph-mon[39462]: 6.18 scrub ok 2024-09-17T07:13:13.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:13 smithi100 ceph-mon[39462]: 6.e scrub ok 2024-09-17T07:13:13.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:13 smithi100 ceph-mon[39462]: pgmap v495: 148 pgs: 148 active+clean; 5.6 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1023 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-09-17T07:13:13.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:13 smithi100 ceph-mon[29092]: 5.0 scrub starts 2024-09-17T07:13:13.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:13 smithi100 ceph-mon[29092]: 5.0 scrub ok 2024-09-17T07:13:13.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:13 smithi100 ceph-mon[29092]: 6.18 scrub starts 2024-09-17T07:13:13.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:13 smithi100 ceph-mon[29092]: 6.e scrub starts 2024-09-17T07:13:13.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:13 smithi100 ceph-mon[29092]: 6.18 scrub ok 2024-09-17T07:13:13.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:13 smithi100 ceph-mon[29092]: 6.e scrub ok 2024-09-17T07:13:13.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:13 smithi100 ceph-mon[29092]: pgmap v495: 148 pgs: 148 active+clean; 5.6 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1023 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-09-17T07:13:13.656 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:13:13.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:13.683 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:13.683 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:13:13.784 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:13:13.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:13 smithi135 ceph-mon[33238]: 5.0 scrub starts 2024-09-17T07:13:13.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:13 smithi135 ceph-mon[33238]: 5.0 scrub ok 2024-09-17T07:13:13.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:13 smithi135 ceph-mon[33238]: 6.18 scrub starts 2024-09-17T07:13:13.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:13 smithi135 ceph-mon[33238]: 6.e scrub starts 2024-09-17T07:13:13.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:13 smithi135 ceph-mon[33238]: 6.18 scrub ok 2024-09-17T07:13:13.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:13 smithi135 ceph-mon[33238]: 6.e scrub ok 2024-09-17T07:13:13.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:13 smithi135 ceph-mon[33238]: pgmap v495: 148 pgs: 148 active+clean; 5.6 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1023 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-09-17T07:13:13.819 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:13.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:13.820 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:13:13.921 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:13:13.945 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:13.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:13.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:13:14.047 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:13:14.079 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:14.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:14.081 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:13:14.181 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:13:14.210 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:14.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:14.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:13:14.220 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 57 16 97554 97538 106.935 75.75 0.0203535 0.00934697 2024-09-17T07:13:14.312 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:13:14.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:14.366 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:14.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:13:14.433 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:13:14.462 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:14 smithi135 ceph-mon[33238]: 4.3 scrub starts 2024-09-17T07:13:14.462 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:14 smithi135 ceph-mon[33238]: 4.3 scrub ok 2024-09-17T07:13:14.462 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:14 smithi135 ceph-mon[33238]: 6.b scrub starts 2024-09-17T07:13:14.462 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:14 smithi135 ceph-mon[33238]: 6.b scrub ok 2024-09-17T07:13:14.468 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:13:14.514 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:14.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:14.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:13:14.616 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:13:14.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:14 smithi100 ceph-mon[29092]: 4.3 scrub starts 2024-09-17T07:13:14.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:14 smithi100 ceph-mon[29092]: 4.3 scrub ok 2024-09-17T07:13:14.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:14 smithi100 ceph-mon[29092]: 6.b scrub starts 2024-09-17T07:13:14.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:14 smithi100 ceph-mon[29092]: 6.b scrub ok 2024-09-17T07:13:14.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:14 smithi100 ceph-mon[39462]: 4.3 scrub starts 2024-09-17T07:13:14.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:14 smithi100 ceph-mon[39462]: 4.3 scrub ok 2024-09-17T07:13:14.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:14 smithi100 ceph-mon[39462]: 6.b scrub starts 2024-09-17T07:13:14.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:14 smithi100 ceph-mon[39462]: 6.b scrub ok 2024-09-17T07:13:14.656 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:14.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:14.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:13:14.757 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:13:14.779 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:14.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:14.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:13:14.881 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:13:14.906 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:14.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:14.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:13:14.908 INFO:teuthology.orchestra.run.smithi135.stdout:ERROR: (22) Invalid argument 2024-09-17T07:13:14.908 INFO:teuthology.orchestra.run.smithi135.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:13:15.009 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:13:15.041 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:15.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:15.042 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:13:15.136 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7] out_osds: [3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-17T07:13:15.137 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T07:13:15.137 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-17T07:13:15.137 INFO:tasks.thrashosds.thrasher:Setting osd 7 primary_affinity to 1.000000 2024-09-17T07:13:15.137 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 7 1 2024-09-17T07:13:15.144 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:13:15.222 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 58 16 98609 98593 106.227 65.9375 0.0345309 0.00940882 2024-09-17T07:13:15.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:15.247 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:15.247 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:13:15.348 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:13:15.388 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:15.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:15.389 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:13:15.490 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:13:15.515 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:15.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:15.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:13:15.518 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:13:15.589 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:13:15.617 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:13:15.630 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:15 smithi100 ceph-mon[39462]: 2.17 scrub starts 2024-09-17T07:13:15.630 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:15 smithi100 ceph-mon[39462]: 2.17 scrub ok 2024-09-17T07:13:15.630 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:15 smithi100 ceph-mon[39462]: pgmap v496: 148 pgs: 148 active+clean; 5.9 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 1023 B/s rd, 74 MiB/s wr, 1.18k op/s 2024-09-17T07:13:15.631 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:15 smithi100 ceph-mon[29092]: 2.17 scrub starts 2024-09-17T07:13:15.631 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:15 smithi100 ceph-mon[29092]: 2.17 scrub ok 2024-09-17T07:13:15.631 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:15 smithi100 ceph-mon[29092]: pgmap v496: 148 pgs: 148 active+clean; 5.9 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 1023 B/s rd, 74 MiB/s wr, 1.18k op/s 2024-09-17T07:13:15.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:15.657 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:15.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:13:15.758 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:13:15.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:15.780 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:15.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:13:15.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:15 smithi135 ceph-mon[33238]: 2.17 scrub starts 2024-09-17T07:13:15.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:15 smithi135 ceph-mon[33238]: 2.17 scrub ok 2024-09-17T07:13:15.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:15 smithi135 ceph-mon[33238]: pgmap v496: 148 pgs: 148 active+clean; 5.9 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 1023 B/s rd, 74 MiB/s wr, 1.18k op/s 2024-09-17T07:13:15.881 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:13:15.905 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:15.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:15.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:13:16.006 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:13:16.037 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:16.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:16.038 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:13:16.106 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:13:15 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:13:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:13:16.139 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:13:16.170 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:16.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:16.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:13:16.221 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 59 16 99461 99445 105.329 53.25 0.0100099 0.00948968 2024-09-17T07:13:16.272 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:13:16.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:16.304 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:16.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:13:16.386 INFO:teuthology.orchestra.run.smithi100.stderr:nodeep-scrub is unset 2024-09-17T07:13:16.405 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:13:16.428 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:16.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:16.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:13:16.530 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:13:16.569 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:16.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:16.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:13:16.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:16 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/911340815' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:13:16.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:16 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:13:16.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:16 smithi100 ceph-mon[29092]: 2.1b scrub starts 2024-09-17T07:13:16.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:16 smithi100 ceph-mon[29092]: 2.1b scrub ok 2024-09-17T07:13:16.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:16 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/911340815' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:13:16.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:16 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:13:16.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:16 smithi100 ceph-mon[39462]: 2.1b scrub starts 2024-09-17T07:13:16.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:16 smithi100 ceph-mon[39462]: 2.1b scrub ok 2024-09-17T07:13:16.671 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:13:16.703 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:16.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:16.704 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:13:16.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:16 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/911340815' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:13:16.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:16 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:13:16.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:16 smithi135 ceph-mon[33238]: 2.1b scrub starts 2024-09-17T07:13:16.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:16 smithi135 ceph-mon[33238]: 2.1b scrub ok 2024-09-17T07:13:16.805 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:13:16.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:16.847 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:16.847 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:13:16.948 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:13:16.996 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:16.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:16.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:13:17.098 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:13:17.136 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:17.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:17.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:13:17.221 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:2024-09-17T07:13:17.221370+0000 min lat: 0.00110426 max lat: 0.299974 avg lat: 0.00953815 2024-09-17T07:13:17.221 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-17T07:13:17.221 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 60 16 100630 100614 104.792 73.0625 0.00673235 0.00953815 2024-09-17T07:13:17.237 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:13:17.265 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:17.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:17.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:13:17.313 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-17T07:13:17.367 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:13:17.417 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:17.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:17.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:13:17.519 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:13:17.547 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:17 smithi135 ceph-mon[33238]: 5.1f scrub starts 2024-09-17T07:13:17.547 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:17 smithi135 ceph-mon[33238]: 5.1f scrub ok 2024-09-17T07:13:17.547 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:17 smithi135 ceph-mon[33238]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T07:13:17.547 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:17 smithi135 ceph-mon[33238]: Cluster is now healthy 2024-09-17T07:13:17.547 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:17 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T07:13:17.547 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:17 smithi135 ceph-mon[33238]: osdmap e258: 8 total, 8 up, 7 in 2024-09-17T07:13:17.547 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:17 smithi135 ceph-mon[33238]: pgmap v498: 148 pgs: 148 active+clean; 5.9 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 639 B/s rd, 56 MiB/s wr, 903 op/s 2024-09-17T07:13:17.569 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:17.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:17.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:13:17.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:17 smithi100 ceph-mon[29092]: 5.1f scrub starts 2024-09-17T07:13:17.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:17 smithi100 ceph-mon[29092]: 5.1f scrub ok 2024-09-17T07:13:17.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:17 smithi100 ceph-mon[29092]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T07:13:17.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:17 smithi100 ceph-mon[29092]: Cluster is now healthy 2024-09-17T07:13:17.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:17 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T07:13:17.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:17 smithi100 ceph-mon[29092]: osdmap e258: 8 total, 8 up, 7 in 2024-09-17T07:13:17.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:17 smithi100 ceph-mon[29092]: pgmap v498: 148 pgs: 148 active+clean; 5.9 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 639 B/s rd, 56 MiB/s wr, 903 op/s 2024-09-17T07:13:17.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:17 smithi100 ceph-mon[39462]: 5.1f scrub starts 2024-09-17T07:13:17.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:17 smithi100 ceph-mon[39462]: 5.1f scrub ok 2024-09-17T07:13:17.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:17 smithi100 ceph-mon[39462]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T07:13:17.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:17 smithi100 ceph-mon[39462]: Cluster is now healthy 2024-09-17T07:13:17.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:17 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T07:13:17.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:17 smithi100 ceph-mon[39462]: osdmap e258: 8 total, 8 up, 7 in 2024-09-17T07:13:17.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:17 smithi100 ceph-mon[39462]: pgmap v498: 148 pgs: 148 active+clean; 5.9 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 639 B/s rd, 56 MiB/s wr, 903 op/s 2024-09-17T07:13:17.671 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:13:17.693 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:13:17.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:17.736 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:17.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:13:17.837 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:13:17.876 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:17.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:17.878 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:13:17.978 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:13:18.038 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:18.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:18.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:13:18.140 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:13:18.178 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:18.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:18.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:13:18.221 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 61 16 101741 101725 104.212 69.4375 0.016394 0.00959136 2024-09-17T07:13:18.281 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:13:18.315 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:18.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:18.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:13:18.417 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:13:18.438 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:18.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:18.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:13:18.540 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:13:18.576 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:18.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:18.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:13:18.678 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:13:18.684 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:18 smithi100 ceph-mon[29092]: 2.7 scrub starts 2024-09-17T07:13:18.684 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:18 smithi100 ceph-mon[29092]: 2.7 scrub ok 2024-09-17T07:13:18.684 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:18 smithi100 ceph-mon[29092]: 5.19 scrub starts 2024-09-17T07:13:18.684 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:18 smithi100 ceph-mon[29092]: 5.19 scrub ok 2024-09-17T07:13:18.685 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:18 smithi100 ceph-mon[29092]: 4.e scrub starts 2024-09-17T07:13:18.685 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:18 smithi100 ceph-mon[29092]: 4.e scrub ok 2024-09-17T07:13:18.685 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:18 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:13:18.685 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:18 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:13:18.685 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:18 smithi100 ceph-mon[39462]: 2.7 scrub starts 2024-09-17T07:13:18.685 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:18 smithi100 ceph-mon[39462]: 2.7 scrub ok 2024-09-17T07:13:18.685 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:18 smithi100 ceph-mon[39462]: 5.19 scrub starts 2024-09-17T07:13:18.685 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:18 smithi100 ceph-mon[39462]: 5.19 scrub ok 2024-09-17T07:13:18.685 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:18 smithi100 ceph-mon[39462]: 4.e scrub starts 2024-09-17T07:13:18.685 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:18 smithi100 ceph-mon[39462]: 4.e scrub ok 2024-09-17T07:13:18.685 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:18 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:13:18.686 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:18 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:13:18.688 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:18 smithi135 ceph-mon[33238]: 2.7 scrub starts 2024-09-17T07:13:18.688 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:18 smithi135 ceph-mon[33238]: 2.7 scrub ok 2024-09-17T07:13:18.688 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:18 smithi135 ceph-mon[33238]: 5.19 scrub starts 2024-09-17T07:13:18.688 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:18 smithi135 ceph-mon[33238]: 5.19 scrub ok 2024-09-17T07:13:18.688 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:18 smithi135 ceph-mon[33238]: 4.e scrub starts 2024-09-17T07:13:18.688 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:18 smithi135 ceph-mon[33238]: 4.e scrub ok 2024-09-17T07:13:18.688 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:18 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:13:18.688 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:18 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:13:18.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:18.704 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:18.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:13:18.765 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:13:18.766 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:13:18.797 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1: {} 2024-09-17T07:13:18.797 INFO:teuthology.orchestra.run.smithi100.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:13:18.804 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:13:18.827 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:13:18.827 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:13:18.849 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:13:18.849 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:13:18.852 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:18.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:18.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:13:18.889 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:13:18.889 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:13:18.922 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:13:18.923 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:13:18.954 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:13:18.962 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:13:18.962 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:13:18.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:18.990 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:18.991 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:13:18.996 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:13:18.997 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:13:19.043 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:13:18 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:13:19.092 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:13:19.127 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:19.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:19.128 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:13:19.221 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 62 15 102930 102915 103.731 74.375 0.00627199 0.00963537 2024-09-17T07:13:19.229 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:13:19.264 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:19.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:19.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:13:19.365 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:13:19.379 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-17T07:13:19.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:19.402 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:19.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:13:19.503 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:13:19.537 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:19.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:19.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:13:19.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:19 smithi135 ceph-mon[33238]: 5.a scrub starts 2024-09-17T07:13:19.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:19 smithi135 ceph-mon[33238]: 5.a scrub ok 2024-09-17T07:13:19.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:19 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:13:19.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:19 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1619900628' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:13:19.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:19 smithi135 ceph-mon[33238]: 2.18 scrub starts 2024-09-17T07:13:19.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:19 smithi135 ceph-mon[33238]: 2.18 scrub ok 2024-09-17T07:13:19.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:19 smithi135 ceph-mon[33238]: pgmap v499: 148 pgs: 148 active+clean; 6.2 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 1.0 KiB/s rd, 62 MiB/s wr, 999 op/s 2024-09-17T07:13:19.596 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:19 smithi100 ceph-mon[29092]: 5.a scrub starts 2024-09-17T07:13:19.597 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:19 smithi100 ceph-mon[29092]: 5.a scrub ok 2024-09-17T07:13:19.597 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:19 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:13:19.597 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:19 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1619900628' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:13:19.597 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:19 smithi100 ceph-mon[29092]: 2.18 scrub starts 2024-09-17T07:13:19.597 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:19 smithi100 ceph-mon[29092]: 2.18 scrub ok 2024-09-17T07:13:19.597 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:19 smithi100 ceph-mon[29092]: pgmap v499: 148 pgs: 148 active+clean; 6.2 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 1.0 KiB/s rd, 62 MiB/s wr, 999 op/s 2024-09-17T07:13:19.597 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:19 smithi100 ceph-mon[39462]: 5.a scrub starts 2024-09-17T07:13:19.597 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:19 smithi100 ceph-mon[39462]: 5.a scrub ok 2024-09-17T07:13:19.597 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:19 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:13:19.597 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:19 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1619900628' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:13:19.598 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:19 smithi100 ceph-mon[39462]: 2.18 scrub starts 2024-09-17T07:13:19.598 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:19 smithi100 ceph-mon[39462]: 2.18 scrub ok 2024-09-17T07:13:19.598 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:19 smithi100 ceph-mon[39462]: pgmap v499: 148 pgs: 148 active+clean; 6.2 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 1.0 KiB/s rd, 62 MiB/s wr, 999 op/s 2024-09-17T07:13:19.639 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:13:19.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:19.691 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:19.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:13:19.792 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:13:19.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:19.841 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:19.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:13:19.942 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:13:19.975 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:19.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:19.976 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:13:20.077 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:13:20.110 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:20.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:20.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:13:20.212 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:13:20.221 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 63 16 104149 104133 103.292 76.125 0.00723493 0.00967601 2024-09-17T07:13:20.259 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:20.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:20.261 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:13:20.361 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:13:20.382 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:20.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:20.383 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:13:20.484 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:13:20.504 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:20 smithi135 ceph-mon[33238]: 5.16 scrub starts 2024-09-17T07:13:20.504 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:20 smithi135 ceph-mon[33238]: 5.16 scrub ok 2024-09-17T07:13:20.511 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:20.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:20.512 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:13:20.613 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:13:20.656 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:20.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:20.658 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:13:20.758 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:13:20.775 INFO:teuthology.orchestra.run.smithi135.stdout:{ 2024-09-17T07:13:20.775 INFO:teuthology.orchestra.run.smithi135.stdout: "size": 20, 2024-09-17T07:13:20.775 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 600, 2024-09-17T07:13:20.775 INFO:teuthology.orchestra.run.smithi135.stdout: "ops": [ 2024-09-17T07:13:20.775 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.775 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:54907 7.1 7:84086a4c:::benchmark_data_smithi100_78556_object54906:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-09-17T07:13:20.775 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:04:57.352937+0000", 2024-09-17T07:13:20.775 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 503.14698021499999, 2024-09-17T07:13:20.775 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.008768664, 2024-09-17T07:13:20.775 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:13:20.776 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:13:20.776 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:13:20.776 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:13:20.776 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:13:20.776 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 54907 2024-09-17T07:13:20.776 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.776 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:13:20.776 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.776 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:13:20.776 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:57.352937+0000", 2024-09-17T07:13:20.776 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:20.776 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.777 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.777 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:13:20.777 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:57.352937+0000", 2024-09-17T07:13:20.777 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:20.777 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.777 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.777 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:13:20.777 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:57.352933+0000", 2024-09-17T07:13:20.778 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999957 2024-09-17T07:13:20.778 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.778 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.778 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:13:20.778 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:57.353001+0000", 2024-09-17T07:13:20.778 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.7626000000000006e-05 2024-09-17T07:13:20.778 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.778 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.778 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:13:20.779 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:57.353029+0000", 2024-09-17T07:13:20.779 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.8904e-05 2024-09-17T07:13:20.779 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.779 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.779 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:13:20.779 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:57.353043+0000", 2024-09-17T07:13:20.779 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.308e-05 2024-09-17T07:13:20.779 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.779 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.779 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:20.779 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:57.437796+0000", 2024-09-17T07:13:20.780 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.084753001999999994 2024-09-17T07:13:20.780 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.780 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.780 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:13:20.780 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:57.437809+0000", 2024-09-17T07:13:20.780 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.3312999999999999e-05 2024-09-17T07:13:20.780 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.780 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.780 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:20.780 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:58.344080+0000", 2024-09-17T07:13:20.781 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.90627139400000001 2024-09-17T07:13:20.781 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.781 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.781 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:13:20.781 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:58.344251+0000", 2024-09-17T07:13:20.781 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000170977 2024-09-17T07:13:20.781 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.781 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.781 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 3,6", 2024-09-17T07:13:20.781 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:58.344425+0000", 2024-09-17T07:13:20.781 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000173933 2024-09-17T07:13:20.781 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.782 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.782 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:13:20.782 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:58.345775+0000", 2024-09-17T07:13:20.782 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0013498449999999999 2024-09-17T07:13:20.782 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.782 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.782 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:13:20.782 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:58.346094+0000", 2024-09-17T07:13:20.782 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00031945299999999998 2024-09-17T07:13:20.782 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.782 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.782 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:13:20.782 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:58.361572+0000", 2024-09-17T07:13:20.783 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.015477538000000001 2024-09-17T07:13:20.783 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.783 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.783 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:13:20.783 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:58.361669+0000", 2024-09-17T07:13:20.783 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.6887999999999998e-05 2024-09-17T07:13:20.783 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.783 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.783 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:13:20.783 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:58.361706+0000", 2024-09-17T07:13:20.783 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.6869999999999998e-05 2024-09-17T07:13:20.783 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:20.784 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:13:20.784 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:20.784 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.784 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.784 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:78080 7.16 7:6df83576:::benchmark_data_smithi100_78556_object78079:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-17T07:13:20.784 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:05:23.290266+0000", 2024-09-17T07:13:20.784 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 477.20965177900001, 2024-09-17T07:13:20.784 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.03028199, 2024-09-17T07:13:20.784 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:13:20.784 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:13:20.784 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:13:20.784 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:13:20.784 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:13:20.785 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 78080 2024-09-17T07:13:20.785 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.785 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:13:20.785 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.785 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:13:20.785 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.290266+0000", 2024-09-17T07:13:20.785 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:20.785 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.785 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.785 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:13:20.785 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.290266+0000", 2024-09-17T07:13:20.785 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:20.786 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.786 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.786 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:13:20.786 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.290263+0000", 2024-09-17T07:13:20.786 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999976 2024-09-17T07:13:20.786 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.786 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.786 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:13:20.786 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.290329+0000", 2024-09-17T07:13:20.786 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.5493000000000005e-05 2024-09-17T07:13:20.786 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.786 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.786 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:13:20.787 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.290351+0000", 2024-09-17T07:13:20.787 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.2245000000000001e-05 2024-09-17T07:13:20.787 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.787 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.787 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:13:20.787 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.290362+0000", 2024-09-17T07:13:20.787 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0838e-05 2024-09-17T07:13:20.787 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.787 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.787 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:20.787 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.290436+0000", 2024-09-17T07:13:20.787 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.4530999999999995e-05 2024-09-17T07:13:20.788 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.788 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.788 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:13:20.788 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.290449+0000", 2024-09-17T07:13:20.788 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.2791e-05 2024-09-17T07:13:20.788 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.788 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.788 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:20.788 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.305250+0000", 2024-09-17T07:13:20.788 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.014800766 2024-09-17T07:13:20.788 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.788 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.788 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:13:20.789 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.305369+0000", 2024-09-17T07:13:20.789 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000118658 2024-09-17T07:13:20.789 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.789 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.789 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,7", 2024-09-17T07:13:20.789 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.305526+0000", 2024-09-17T07:13:20.789 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00015784300000000001 2024-09-17T07:13:20.789 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.789 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.789 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:13:20.789 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.309043+0000", 2024-09-17T07:13:20.789 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0035167050000000002 2024-09-17T07:13:20.789 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.790 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.790 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:13:20.790 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.309240+0000", 2024-09-17T07:13:20.790 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00019642100000000001 2024-09-17T07:13:20.790 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.790 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.790 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:13:20.790 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.320476+0000", 2024-09-17T07:13:20.790 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.01123681 2024-09-17T07:13:20.790 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.790 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.790 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:13:20.791 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.320510+0000", 2024-09-17T07:13:20.791 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.3553000000000002e-05 2024-09-17T07:13:20.791 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.791 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.791 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:13:20.791 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.320547+0000", 2024-09-17T07:13:20.791 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.7527999999999997e-05 2024-09-17T07:13:20.791 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:20.791 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:13:20.791 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:20.791 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.791 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.791 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:78111 7.16 7:69dc329b:::benchmark_data_smithi100_78556_object78110:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-17T07:13:20.792 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:05:23.330298+0000", 2024-09-17T07:13:20.792 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 477.16961962200003, 2024-09-17T07:13:20.792 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.99037067999999995, 2024-09-17T07:13:20.792 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:13:20.792 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:13:20.792 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:13:20.792 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:13:20.792 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:13:20.792 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 78111 2024-09-17T07:13:20.792 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.792 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:13:20.792 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.793 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:13:20.793 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.330298+0000", 2024-09-17T07:13:20.793 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:20.793 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.793 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.793 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:13:20.793 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.330298+0000", 2024-09-17T07:13:20.793 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:20.793 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.793 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.793 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:13:20.793 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.330295+0000", 2024-09-17T07:13:20.793 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999976 2024-09-17T07:13:20.794 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.794 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.794 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:13:20.794 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.330335+0000", 2024-09-17T07:13:20.794 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.9816e-05 2024-09-17T07:13:20.794 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.795 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.795 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:13:20.795 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.330356+0000", 2024-09-17T07:13:20.795 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.0679e-05 2024-09-17T07:13:20.795 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.795 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.795 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:13:20.795 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.330366+0000", 2024-09-17T07:13:20.796 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0421000000000001e-05 2024-09-17T07:13:20.796 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.796 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.796 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:20.796 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.330453+0000", 2024-09-17T07:13:20.796 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.6465000000000006e-05 2024-09-17T07:13:20.796 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.796 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.796 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:13:20.796 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.330468+0000", 2024-09-17T07:13:20.796 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.4861999999999999e-05 2024-09-17T07:13:20.796 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.797 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.797 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:20.797 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.306866+0000", 2024-09-17T07:13:20.797 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.97639884300000002 2024-09-17T07:13:20.797 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.797 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.797 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:13:20.797 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.307049+0000", 2024-09-17T07:13:20.797 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00018273700000000001 2024-09-17T07:13:20.797 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.797 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.797 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,7", 2024-09-17T07:13:20.798 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.307202+0000", 2024-09-17T07:13:20.798 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00015266300000000001 2024-09-17T07:13:20.798 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.798 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.798 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:13:20.798 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.309154+0000", 2024-09-17T07:13:20.798 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0019523609999999999 2024-09-17T07:13:20.798 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.798 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.798 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:13:20.798 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.309279+0000", 2024-09-17T07:13:20.798 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00012443400000000001 2024-09-17T07:13:20.798 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.799 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.799 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:13:20.799 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.320587+0000", 2024-09-17T07:13:20.799 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.011307874000000001 2024-09-17T07:13:20.799 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.799 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.799 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:13:20.799 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.320632+0000", 2024-09-17T07:13:20.799 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.5847000000000003e-05 2024-09-17T07:13:20.799 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.799 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.799 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:13:20.799 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.320668+0000", 2024-09-17T07:13:20.800 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.5936999999999998e-05 2024-09-17T07:13:20.800 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:20.800 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:13:20.800 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:20.800 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.800 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.800 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:78143 7.16 7:6a829b84:::benchmark_data_smithi100_78556_object78142:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-17T07:13:20.800 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:05:23.360285+0000", 2024-09-17T07:13:20.800 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 477.13963262999999, 2024-09-17T07:13:20.800 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.96129615000000002, 2024-09-17T07:13:20.800 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:13:20.800 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:13:20.801 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:13:20.801 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:13:20.801 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:13:20.801 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 78143 2024-09-17T07:13:20.801 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.801 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:13:20.801 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.801 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:13:20.801 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.360285+0000", 2024-09-17T07:13:20.801 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:20.801 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.801 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.801 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:13:20.801 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.360285+0000", 2024-09-17T07:13:20.802 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:20.802 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.802 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.802 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:13:20.802 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.360282+0000", 2024-09-17T07:13:20.802 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999976 2024-09-17T07:13:20.802 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.802 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.802 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:13:20.802 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.360333+0000", 2024-09-17T07:13:20.802 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.1029000000000002e-05 2024-09-17T07:13:20.802 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.802 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.803 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:13:20.803 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.360354+0000", 2024-09-17T07:13:20.803 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.0392999999999999e-05 2024-09-17T07:13:20.803 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.803 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.803 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:13:20.803 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.360364+0000", 2024-09-17T07:13:20.803 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0336999999999999e-05 2024-09-17T07:13:20.803 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.803 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.803 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:20.804 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.360436+0000", 2024-09-17T07:13:20.804 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.1639999999999998e-05 2024-09-17T07:13:20.804 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.804 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.804 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:13:20.804 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.360448+0000", 2024-09-17T07:13:20.804 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.1704e-05 2024-09-17T07:13:20.804 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.804 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.804 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:20.804 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.308033+0000", 2024-09-17T07:13:20.805 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.94758547000000004 2024-09-17T07:13:20.805 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.805 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.805 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:13:20.805 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.308115+0000", 2024-09-17T07:13:20.805 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.1798000000000005e-05 2024-09-17T07:13:20.805 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.805 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.805 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,7", 2024-09-17T07:13:20.805 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.308243+0000", 2024-09-17T07:13:20.806 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00012773299999999999 2024-09-17T07:13:20.806 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.806 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.806 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:13:20.806 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.311329+0000", 2024-09-17T07:13:20.806 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0030862849999999998 2024-09-17T07:13:20.806 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.806 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.806 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:13:20.806 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.311371+0000", 2024-09-17T07:13:20.806 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.2502000000000001e-05 2024-09-17T07:13:20.806 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.807 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.807 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:13:20.807 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.321532+0000", 2024-09-17T07:13:20.807 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.010161129 2024-09-17T07:13:20.807 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.807 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.807 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:13:20.807 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.321562+0000", 2024-09-17T07:13:20.807 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.9882999999999998e-05 2024-09-17T07:13:20.807 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.807 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.807 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:13:20.807 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.321581+0000", 2024-09-17T07:13:20.808 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.8488e-05 2024-09-17T07:13:20.808 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:20.808 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:13:20.808 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:20.808 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.808 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.808 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:78146 7.16 7:6cf9446e:::benchmark_data_smithi100_78556_object78145:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-17T07:13:20.808 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:05:23.363433+0000", 2024-09-17T07:13:20.808 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 477.13648411299999, 2024-09-17T07:13:20.808 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.95826878400000004, 2024-09-17T07:13:20.808 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:13:20.808 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:13:20.808 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:13:20.809 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:13:20.809 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:13:20.809 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 78146 2024-09-17T07:13:20.809 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.809 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:13:20.809 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.809 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:13:20.809 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.363433+0000", 2024-09-17T07:13:20.809 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:20.809 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.809 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.809 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:13:20.810 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.363433+0000", 2024-09-17T07:13:20.810 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:20.810 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.810 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.810 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:13:20.810 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.363431+0000", 2024-09-17T07:13:20.810 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999981 2024-09-17T07:13:20.810 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.810 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.810 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:13:20.810 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.363469+0000", 2024-09-17T07:13:20.810 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.8065000000000002e-05 2024-09-17T07:13:20.811 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.811 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.811 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:13:20.811 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.363486+0000", 2024-09-17T07:13:20.811 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.6946e-05 2024-09-17T07:13:20.811 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.811 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.811 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:13:20.811 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.363493+0000", 2024-09-17T07:13:20.811 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.1550000000000001e-06 2024-09-17T07:13:20.811 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.811 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.812 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:20.812 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.363553+0000", 2024-09-17T07:13:20.812 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.0066999999999999e-05 2024-09-17T07:13:20.812 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.812 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.812 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:13:20.812 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.363565+0000", 2024-09-17T07:13:20.812 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.1960999999999999e-05 2024-09-17T07:13:20.812 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.812 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.812 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:20.812 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.309302+0000", 2024-09-17T07:13:20.813 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.94573720299999997 2024-09-17T07:13:20.813 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.813 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.813 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:13:20.813 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.309414+0000", 2024-09-17T07:13:20.813 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000111467 2024-09-17T07:13:20.813 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.813 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.813 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,7", 2024-09-17T07:13:20.813 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.309535+0000", 2024-09-17T07:13:20.813 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00012128799999999999 2024-09-17T07:13:20.813 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.813 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.814 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:13:20.814 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.311362+0000", 2024-09-17T07:13:20.814 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0018264939999999999 2024-09-17T07:13:20.814 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.814 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.814 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:13:20.814 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.311376+0000", 2024-09-17T07:13:20.814 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.4374e-05 2024-09-17T07:13:20.814 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.814 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.814 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:13:20.814 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.321625+0000", 2024-09-17T07:13:20.814 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.010249121999999999 2024-09-17T07:13:20.815 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.815 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.815 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:13:20.815 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.321668+0000", 2024-09-17T07:13:20.815 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.3028000000000002e-05 2024-09-17T07:13:20.815 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.815 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.815 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:13:20.815 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.321702+0000", 2024-09-17T07:13:20.815 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.3689000000000002e-05 2024-09-17T07:13:20.815 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:20.815 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:13:20.816 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:20.816 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.816 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.816 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:78148 7.16 7:6f8b784c:::benchmark_data_smithi100_78556_object78147:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-17T07:13:20.816 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:05:23.364154+0000", 2024-09-17T07:13:20.816 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 477.13576329300002, 2024-09-17T07:13:20.816 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.95766116599999995, 2024-09-17T07:13:20.816 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:13:20.816 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:13:20.816 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:13:20.816 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:13:20.816 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:13:20.816 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 78148 2024-09-17T07:13:20.817 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.817 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:13:20.817 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.817 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:13:20.817 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.364154+0000", 2024-09-17T07:13:20.817 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:20.817 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.817 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.817 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:13:20.817 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.364154+0000", 2024-09-17T07:13:20.817 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:20.818 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.818 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.818 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:13:20.818 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.364153+0000", 2024-09-17T07:13:20.818 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999995 2024-09-17T07:13:20.818 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.818 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.818 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:13:20.818 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.364258+0000", 2024-09-17T07:13:20.818 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00010417 2024-09-17T07:13:20.818 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.818 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.819 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:13:20.819 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.364267+0000", 2024-09-17T07:13:20.819 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.0129999999999999e-06 2024-09-17T07:13:20.819 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.819 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.819 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:13:20.819 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.364271+0000", 2024-09-17T07:13:20.819 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.5850000000000001e-06 2024-09-17T07:13:20.819 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.819 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.819 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:20.820 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.364314+0000", 2024-09-17T07:13:20.820 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.337e-05 2024-09-17T07:13:20.820 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.820 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.820 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:13:20.820 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.364324+0000", 2024-09-17T07:13:20.820 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.2790000000000004e-06 2024-09-17T07:13:20.820 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.820 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.820 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:20.821 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.310480+0000", 2024-09-17T07:13:20.821 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.94615585199999996 2024-09-17T07:13:20.821 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.821 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.821 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:13:20.821 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.310606+0000", 2024-09-17T07:13:20.821 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00012678200000000001 2024-09-17T07:13:20.821 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.821 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.821 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,7", 2024-09-17T07:13:20.821 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.310733+0000", 2024-09-17T07:13:20.822 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00012678200000000001 2024-09-17T07:13:20.822 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.822 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.822 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:13:20.822 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.313206+0000", 2024-09-17T07:13:20.822 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.002472442 2024-09-17T07:13:20.822 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.822 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.822 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:13:20.822 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.313211+0000", 2024-09-17T07:13:20.822 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.9620000000000001e-06 2024-09-17T07:13:20.823 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.823 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.823 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:13:20.823 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.321752+0000", 2024-09-17T07:13:20.823 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0085410239999999995 2024-09-17T07:13:20.823 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.823 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.823 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:13:20.823 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.321782+0000", 2024-09-17T07:13:20.823 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.0269000000000001e-05 2024-09-17T07:13:20.823 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.824 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.824 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:13:20.824 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.321815+0000", 2024-09-17T07:13:20.824 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.3275000000000002e-05 2024-09-17T07:13:20.824 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:20.824 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:13:20.824 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:20.824 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.824 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.824 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:78206 7.16 7:6d56ab44:::benchmark_data_smithi100_78556_object78205:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-17T07:13:20.824 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:05:23.442944+0000", 2024-09-17T07:13:20.825 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 477.05697376900002, 2024-09-17T07:13:20.825 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.87899096200000004, 2024-09-17T07:13:20.825 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:13:20.825 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:13:20.825 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:13:20.825 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:13:20.825 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:13:20.825 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 78206 2024-09-17T07:13:20.825 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.825 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:13:20.825 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.826 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:13:20.826 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.442944+0000", 2024-09-17T07:13:20.826 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:20.826 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.826 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.826 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:13:20.826 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.442944+0000", 2024-09-17T07:13:20.826 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:20.826 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.826 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.826 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:13:20.827 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.442942+0000", 2024-09-17T07:13:20.827 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:13:20.827 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.827 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.827 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:13:20.827 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.442976+0000", 2024-09-17T07:13:20.827 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.4471999999999998e-05 2024-09-17T07:13:20.827 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.827 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.827 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:13:20.827 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.442986+0000", 2024-09-17T07:13:20.827 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.1600000000000004e-06 2024-09-17T07:13:20.827 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.828 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.828 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:13:20.828 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.442992+0000", 2024-09-17T07:13:20.828 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.3040000000000001e-06 2024-09-17T07:13:20.828 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.828 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.828 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:20.828 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.443041+0000", 2024-09-17T07:13:20.828 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.8633e-05 2024-09-17T07:13:20.828 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.828 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.828 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:13:20.828 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.443049+0000", 2024-09-17T07:13:20.828 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.6789999999999994e-06 2024-09-17T07:13:20.829 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.829 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.829 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:20.829 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.311410+0000", 2024-09-17T07:13:20.829 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.86836050300000001 2024-09-17T07:13:20.829 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.829 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.829 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:13:20.830 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.311516+0000", 2024-09-17T07:13:20.830 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00010664500000000001 2024-09-17T07:13:20.830 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.830 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.830 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,7", 2024-09-17T07:13:20.830 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.311636+0000", 2024-09-17T07:13:20.830 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000119796 2024-09-17T07:13:20.830 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.830 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.830 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:13:20.830 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.313229+0000", 2024-09-17T07:13:20.830 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0015927389999999999 2024-09-17T07:13:20.831 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.831 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.831 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:13:20.831 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.313950+0000", 2024-09-17T07:13:20.831 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00072075800000000005 2024-09-17T07:13:20.831 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.831 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.831 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:13:20.831 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.321861+0000", 2024-09-17T07:13:20.831 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0079110919999999998 2024-09-17T07:13:20.831 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.831 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.831 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:13:20.832 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.321903+0000", 2024-09-17T07:13:20.832 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.1992e-05 2024-09-17T07:13:20.832 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.832 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.832 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:13:20.832 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.321934+0000", 2024-09-17T07:13:20.832 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.1678999999999999e-05 2024-09-17T07:13:20.832 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:20.832 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:13:20.832 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:20.832 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.833 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.833 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:78211 7.16 7:6dd6ed80:::benchmark_data_smithi100_78556_object78210:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-17T07:13:20.833 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:05:23.448482+0000", 2024-09-17T07:13:20.833 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 477.05143548799998, 2024-09-17T07:13:20.833 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.873569598, 2024-09-17T07:13:20.833 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:13:20.833 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:13:20.833 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:13:20.833 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:13:20.833 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:13:20.833 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 78211 2024-09-17T07:13:20.833 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.834 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:13:20.834 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.834 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:13:20.834 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.448482+0000", 2024-09-17T07:13:20.834 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:20.834 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.834 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.834 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:13:20.834 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.448482+0000", 2024-09-17T07:13:20.834 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:20.834 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.835 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.835 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:13:20.835 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.448480+0000", 2024-09-17T07:13:20.835 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999981 2024-09-17T07:13:20.835 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.835 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.835 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:13:20.835 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.448529+0000", 2024-09-17T07:13:20.835 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.8891999999999999e-05 2024-09-17T07:13:20.835 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.835 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.836 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:13:20.836 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.448548+0000", 2024-09-17T07:13:20.836 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.9093000000000002e-05 2024-09-17T07:13:20.836 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.836 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.836 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:13:20.836 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.448558+0000", 2024-09-17T07:13:20.836 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0064000000000001e-05 2024-09-17T07:13:20.836 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.836 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.836 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:20.836 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.448621+0000", 2024-09-17T07:13:20.837 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.3278e-05 2024-09-17T07:13:20.837 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.837 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.837 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:13:20.837 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:23.448632+0000", 2024-09-17T07:13:20.837 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0342e-05 2024-09-17T07:13:20.837 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.837 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.837 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:20.837 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.312467+0000", 2024-09-17T07:13:20.837 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.86383483800000005 2024-09-17T07:13:20.838 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.838 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.838 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:13:20.838 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.312562+0000", 2024-09-17T07:13:20.838 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.5843999999999999e-05 2024-09-17T07:13:20.838 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.838 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.838 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,7", 2024-09-17T07:13:20.838 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.312644+0000", 2024-09-17T07:13:20.838 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.1299999999999997e-05 2024-09-17T07:13:20.838 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.838 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.838 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:13:20.839 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.314813+0000", 2024-09-17T07:13:20.839 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0021691929999999998 2024-09-17T07:13:20.839 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.839 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.839 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:13:20.839 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.314822+0000", 2024-09-17T07:13:20.839 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.5730000000000008e-06 2024-09-17T07:13:20.839 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.839 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.839 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:13:20.839 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.321970+0000", 2024-09-17T07:13:20.840 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.007148906 2024-09-17T07:13:20.840 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.840 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.840 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:13:20.840 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.322004+0000", 2024-09-17T07:13:20.840 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.3183999999999999e-05 2024-09-17T07:13:20.840 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.840 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.840 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:13:20.840 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:24.322051+0000", 2024-09-17T07:13:20.840 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.7790000000000002e-05 2024-09-17T07:13:20.840 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:20.841 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:13:20.841 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:20.841 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.841 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.841 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.24929.0:164758 8.b 8:d5cb077d:::benchmark_data_smithi100_102803_object34254:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-09-17T07:13:20.841 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:10:17.219733+0000", 2024-09-17T07:13:20.841 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 183.28018453499999, 2024-09-17T07:13:20.841 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.98373762600000003, 2024-09-17T07:13:20.841 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:13:20.841 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:13:20.841 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:13:20.841 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.24929", 2024-09-17T07:13:20.842 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/564515582", 2024-09-17T07:13:20.842 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 164758 2024-09-17T07:13:20.842 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.842 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:13:20.842 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.842 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:13:20.842 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.219733+0000", 2024-09-17T07:13:20.842 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:20.842 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.842 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.842 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:13:20.843 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.219733+0000", 2024-09-17T07:13:20.843 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:20.843 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.843 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.843 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:13:20.843 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.219731+0000", 2024-09-17T07:13:20.843 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:13:20.843 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.843 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.843 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:13:20.843 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.219738+0000", 2024-09-17T07:13:20.844 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.6100000000000002e-06 2024-09-17T07:13:20.844 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.844 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.844 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:13:20.844 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.219754+0000", 2024-09-17T07:13:20.844 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.6098000000000001e-05 2024-09-17T07:13:20.844 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.844 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.844 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:13:20.844 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.219761+0000", 2024-09-17T07:13:20.844 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.5340000000000002e-06 2024-09-17T07:13:20.845 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.845 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.845 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:20.845 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.219814+0000", 2024-09-17T07:13:20.845 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.2556999999999998e-05 2024-09-17T07:13:20.845 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.845 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.845 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:13:20.845 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.219823+0000", 2024-09-17T07:13:20.845 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.4730000000000006e-06 2024-09-17T07:13:20.845 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.845 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.845 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:20.846 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.196970+0000", 2024-09-17T07:13:20.846 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.97714677999999999 2024-09-17T07:13:20.846 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.846 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.846 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:13:20.846 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.197099+0000", 2024-09-17T07:13:20.846 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000128703 2024-09-17T07:13:20.846 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.846 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.846 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 1,2", 2024-09-17T07:13:20.846 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.197226+0000", 2024-09-17T07:13:20.846 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00012752799999999999 2024-09-17T07:13:20.847 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.847 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.847 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:13:20.847 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.201464+0000", 2024-09-17T07:13:20.847 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0042372540000000002 2024-09-17T07:13:20.847 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.847 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.847 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:13:20.847 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.203242+0000", 2024-09-17T07:13:20.847 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.001778083 2024-09-17T07:13:20.847 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.847 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.847 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:13:20.848 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.203372+0000", 2024-09-17T07:13:20.848 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00013064199999999999 2024-09-17T07:13:20.848 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.848 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.848 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:13:20.848 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.203426+0000", 2024-09-17T07:13:20.848 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.3329999999999999e-05 2024-09-17T07:13:20.848 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.848 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.848 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:13:20.848 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.203470+0000", 2024-09-17T07:13:20.848 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.4638000000000002e-05 2024-09-17T07:13:20.849 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:20.849 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:13:20.849 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:20.849 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.849 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.849 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.24929.0:164774 8.b 8:d8c6bb1d:::benchmark_data_smithi100_102803_object34270:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-09-17T07:13:20.849 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:10:17.230982+0000", 2024-09-17T07:13:20.849 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 183.26893538499999, 2024-09-17T07:13:20.849 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.98469290399999998, 2024-09-17T07:13:20.849 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:13:20.849 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:13:20.849 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:13:20.850 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.24929", 2024-09-17T07:13:20.850 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/564515582", 2024-09-17T07:13:20.850 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 164774 2024-09-17T07:13:20.850 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.850 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:13:20.850 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.850 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:13:20.850 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.230982+0000", 2024-09-17T07:13:20.850 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:20.850 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.850 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.850 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:13:20.852 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.230982+0000", 2024-09-17T07:13:20.852 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:20.852 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.852 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.852 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:13:20.852 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.230980+0000", 2024-09-17T07:13:20.852 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:13:20.852 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.852 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.853 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:13:20.853 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.230986+0000", 2024-09-17T07:13:20.853 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.7139999999999998e-06 2024-09-17T07:13:20.853 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.853 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.853 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:13:20.853 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.230997+0000", 2024-09-17T07:13:20.853 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.1090000000000001e-05 2024-09-17T07:13:20.853 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.853 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.853 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:13:20.853 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.231004+0000", 2024-09-17T07:13:20.853 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.8249999999999999e-06 2024-09-17T07:13:20.854 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.854 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.854 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:20.854 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.231058+0000", 2024-09-17T07:13:20.854 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.3931000000000003e-05 2024-09-17T07:13:20.854 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.854 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.854 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:13:20.854 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.231066+0000", 2024-09-17T07:13:20.854 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.9689999999999994e-06 2024-09-17T07:13:20.854 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.854 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.854 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:20.855 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.199823+0000", 2024-09-17T07:13:20.855 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.96875726900000003 2024-09-17T07:13:20.855 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.855 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.855 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:13:20.855 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.199939+0000", 2024-09-17T07:13:20.855 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000115328 2024-09-17T07:13:20.855 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.855 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.855 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 1,2", 2024-09-17T07:13:20.855 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.200064+0000", 2024-09-17T07:13:20.855 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00012558900000000001 2024-09-17T07:13:20.855 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.856 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.856 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:13:20.856 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.205175+0000", 2024-09-17T07:13:20.856 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0051110360000000002 2024-09-17T07:13:20.856 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.856 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.856 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:13:20.856 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.207188+0000", 2024-09-17T07:13:20.856 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.002012626 2024-09-17T07:13:20.856 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.856 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.856 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:13:20.856 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.215606+0000", 2024-09-17T07:13:20.857 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0084185200000000005 2024-09-17T07:13:20.857 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.857 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.857 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:13:20.857 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.215650+0000", 2024-09-17T07:13:20.857 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.4061999999999999e-05 2024-09-17T07:13:20.857 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.857 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.857 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:13:20.857 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.215675+0000", 2024-09-17T07:13:20.857 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.4426999999999998e-05 2024-09-17T07:13:20.857 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:20.858 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:13:20.858 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:20.858 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.858 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.858 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.24929.0:164784 8.b 8:d2e02e23:::benchmark_data_smithi100_102803_object34280:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-09-17T07:13:20.858 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:10:17.240427+0000", 2024-09-17T07:13:20.858 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 183.25949022099999, 2024-09-17T07:13:20.858 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.98057870999999996, 2024-09-17T07:13:20.858 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:13:20.858 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:13:20.858 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:13:20.858 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.24929", 2024-09-17T07:13:20.859 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/564515582", 2024-09-17T07:13:20.859 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 164784 2024-09-17T07:13:20.859 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.859 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:13:20.859 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.859 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:13:20.859 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.240427+0000", 2024-09-17T07:13:20.859 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:20.859 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.859 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.859 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:13:20.859 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.240427+0000", 2024-09-17T07:13:20.859 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:20.860 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.860 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.860 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:13:20.860 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.240425+0000", 2024-09-17T07:13:20.860 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999981 2024-09-17T07:13:20.860 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.860 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.860 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:13:20.860 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.240434+0000", 2024-09-17T07:13:20.860 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.4780000000000008e-06 2024-09-17T07:13:20.860 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.860 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.860 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:13:20.861 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.240449+0000", 2024-09-17T07:13:20.861 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.5153999999999999e-05 2024-09-17T07:13:20.861 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.861 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.861 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:13:20.861 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.240457+0000", 2024-09-17T07:13:20.861 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.5009999999999998e-06 2024-09-17T07:13:20.861 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.861 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.861 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:20.861 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.245473+0000", 2024-09-17T07:13:20.861 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0050163769999999998 2024-09-17T07:13:20.862 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.862 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.862 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:13:20.862 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.245483+0000", 2024-09-17T07:13:20.862 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.9660000000000008e-06 2024-09-17T07:13:20.862 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.862 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.862 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:20.862 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.201492+0000", 2024-09-17T07:13:20.862 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.95600924600000003 2024-09-17T07:13:20.862 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.862 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.863 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:13:20.863 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.201577+0000", 2024-09-17T07:13:20.863 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.4685000000000007e-05 2024-09-17T07:13:20.863 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.863 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.863 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 1,2", 2024-09-17T07:13:20.863 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.201689+0000", 2024-09-17T07:13:20.863 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00011176 2024-09-17T07:13:20.863 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.863 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.863 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:13:20.863 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.205184+0000", 2024-09-17T07:13:20.863 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.003495593 2024-09-17T07:13:20.864 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.864 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.864 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:13:20.864 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.215528+0000", 2024-09-17T07:13:20.864 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.010343662999999999 2024-09-17T07:13:20.864 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.864 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.864 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:13:20.864 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.220925+0000", 2024-09-17T07:13:20.864 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.005396858 2024-09-17T07:13:20.864 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.864 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.865 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:13:20.865 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.220969+0000", 2024-09-17T07:13:20.865 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.4382999999999998e-05 2024-09-17T07:13:20.865 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.865 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.865 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:13:20.865 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.221006+0000", 2024-09-17T07:13:20.865 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.6736999999999997e-05 2024-09-17T07:13:20.865 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:20.865 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:13:20.865 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:20.865 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.866 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.866 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.24929.0:164821 8.b 8:d651d42e:::benchmark_data_smithi100_102803_object34317:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-09-17T07:13:20.866 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:10:17.292857+0000", 2024-09-17T07:13:20.866 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 183.20706025800001, 2024-09-17T07:13:20.866 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.92826142099999998, 2024-09-17T07:13:20.866 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:13:20.866 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:13:20.866 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:13:20.866 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.24929", 2024-09-17T07:13:20.866 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/564515582", 2024-09-17T07:13:20.866 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 164821 2024-09-17T07:13:20.866 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.866 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:13:20.866 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.867 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:13:20.867 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.292857+0000", 2024-09-17T07:13:20.867 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:20.867 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.867 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.867 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:13:20.867 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.292857+0000", 2024-09-17T07:13:20.867 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:20.867 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.867 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.867 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:13:20.867 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.292820+0000", 2024-09-17T07:13:20.868 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999628 2024-09-17T07:13:20.868 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.868 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.868 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:13:20.868 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.292933+0000", 2024-09-17T07:13:20.868 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000113736 2024-09-17T07:13:20.868 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.868 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.868 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:13:20.868 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.292945+0000", 2024-09-17T07:13:20.868 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.1202e-05 2024-09-17T07:13:20.868 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.868 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.869 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:13:20.869 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.292952+0000", 2024-09-17T07:13:20.869 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.7270000000000007e-06 2024-09-17T07:13:20.869 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.869 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.869 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:20.869 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.293004+0000", 2024-09-17T07:13:20.869 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.1286999999999999e-05 2024-09-17T07:13:20.869 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.869 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.869 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:13:20.869 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.293014+0000", 2024-09-17T07:13:20.870 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0142e-05 2024-09-17T07:13:20.870 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.870 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.870 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:20.870 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.203517+0000", 2024-09-17T07:13:20.870 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.91050304199999998 2024-09-17T07:13:20.870 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.870 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.870 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:13:20.870 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.203643+0000", 2024-09-17T07:13:20.870 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00012641099999999999 2024-09-17T07:13:20.870 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.870 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.871 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 1,2", 2024-09-17T07:13:20.871 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.203765+0000", 2024-09-17T07:13:20.871 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000121395 2024-09-17T07:13:20.871 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.871 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.871 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:13:20.871 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.210334+0000", 2024-09-17T07:13:20.871 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0065697999999999998 2024-09-17T07:13:20.871 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.871 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.871 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:13:20.871 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.215566+0000", 2024-09-17T07:13:20.871 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0052315310000000002 2024-09-17T07:13:20.871 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.872 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.872 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:13:20.872 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.221049+0000", 2024-09-17T07:13:20.872 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0054830199999999999 2024-09-17T07:13:20.872 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.872 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.872 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:13:20.872 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.221083+0000", 2024-09-17T07:13:20.872 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.4251000000000001e-05 2024-09-17T07:13:20.872 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.872 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.873 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:13:20.873 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.221118+0000", 2024-09-17T07:13:20.873 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.5296999999999998e-05 2024-09-17T07:13:20.873 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:20.873 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:13:20.873 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:20.873 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.873 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.873 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.24929.0:164830 8.b 8:d9eeacc1:::benchmark_data_smithi100_102803_object34326:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-09-17T07:13:20.873 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:10:17.302885+0000", 2024-09-17T07:13:20.873 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 183.197032334, 2024-09-17T07:13:20.873 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.91836632900000004, 2024-09-17T07:13:20.873 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:13:20.874 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:13:20.874 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:13:20.874 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.24929", 2024-09-17T07:13:20.874 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/564515582", 2024-09-17T07:13:20.874 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 164830 2024-09-17T07:13:20.874 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.874 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:13:20.874 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.874 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:13:20.874 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.302885+0000", 2024-09-17T07:13:20.874 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:20.874 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.874 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.875 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:13:20.875 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.302885+0000", 2024-09-17T07:13:20.875 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:20.875 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.875 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.875 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:13:20.875 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.302884+0000", 2024-09-17T07:13:20.875 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.999999 2024-09-17T07:13:20.875 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.875 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.875 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:13:20.875 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.302888+0000", 2024-09-17T07:13:20.875 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.1500000000000001e-06 2024-09-17T07:13:20.876 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.876 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.876 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:13:20.876 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.302895+0000", 2024-09-17T07:13:20.876 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.3409999999999997e-06 2024-09-17T07:13:20.876 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.876 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.876 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:13:20.876 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.302900+0000", 2024-09-17T07:13:20.876 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.7389999999999999e-06 2024-09-17T07:13:20.876 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.877 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.877 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:20.877 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.302996+0000", 2024-09-17T07:13:20.877 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.5439000000000006e-05 2024-09-17T07:13:20.877 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.877 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.877 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:13:20.877 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.303004+0000", 2024-09-17T07:13:20.877 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.3049999999999993e-06 2024-09-17T07:13:20.877 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.877 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.877 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:20.877 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.205220+0000", 2024-09-17T07:13:20.878 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.90221566900000005 2024-09-17T07:13:20.878 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.878 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.878 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:13:20.878 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.205341+0000", 2024-09-17T07:13:20.878 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00012173 2024-09-17T07:13:20.878 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.878 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.878 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 1,2", 2024-09-17T07:13:20.878 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.205454+0000", 2024-09-17T07:13:20.878 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000112272 2024-09-17T07:13:20.878 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.879 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.879 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:13:20.879 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.213812+0000", 2024-09-17T07:13:20.879 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0083582610000000005 2024-09-17T07:13:20.879 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.879 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.879 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:13:20.879 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.220826+0000", 2024-09-17T07:13:20.879 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0070144109999999999 2024-09-17T07:13:20.879 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.879 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.879 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:13:20.879 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.221160+0000", 2024-09-17T07:13:20.880 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000333436 2024-09-17T07:13:20.880 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.880 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.880 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:13:20.880 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.221201+0000", 2024-09-17T07:13:20.880 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.1505999999999999e-05 2024-09-17T07:13:20.880 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.880 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.880 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:13:20.880 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.221251+0000", 2024-09-17T07:13:20.880 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.0142999999999997e-05 2024-09-17T07:13:20.880 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:20.881 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:13:20.881 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:20.881 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.881 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.881 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.24929.0:164831 8.b 8:d6c25258:::benchmark_data_smithi100_102803_object34327:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-09-17T07:13:20.881 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:10:17.303020+0000", 2024-09-17T07:13:20.881 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 183.196896982, 2024-09-17T07:13:20.881 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.91836619399999997, 2024-09-17T07:13:20.881 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:13:20.881 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:13:20.881 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:13:20.881 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.24929", 2024-09-17T07:13:20.882 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/564515582", 2024-09-17T07:13:20.882 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 164831 2024-09-17T07:13:20.882 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.882 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:13:20.882 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.882 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:13:20.882 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.303020+0000", 2024-09-17T07:13:20.882 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:20.882 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.882 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.882 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:13:20.883 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.303020+0000", 2024-09-17T07:13:20.883 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:20.883 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.883 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.883 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:13:20.883 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.303020+0000", 2024-09-17T07:13:20.883 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999995 2024-09-17T07:13:20.883 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.883 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.883 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:13:20.883 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.303023+0000", 2024-09-17T07:13:20.883 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.2100000000000002e-06 2024-09-17T07:13:20.883 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.884 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.884 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:13:20.884 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.303029+0000", 2024-09-17T07:13:20.884 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.0809999999999999e-06 2024-09-17T07:13:20.884 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.884 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.884 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:13:20.884 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.303034+0000", 2024-09-17T07:13:20.884 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.9910000000000002e-06 2024-09-17T07:13:20.884 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.884 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.885 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:20.885 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.303118+0000", 2024-09-17T07:13:20.885 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.3605999999999999e-05 2024-09-17T07:13:20.885 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.885 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.885 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:13:20.885 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.303124+0000", 2024-09-17T07:13:20.885 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.2929999999999997e-06 2024-09-17T07:13:20.885 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.885 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.885 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:20.885 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.207218+0000", 2024-09-17T07:13:20.885 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.90409358399999995 2024-09-17T07:13:20.886 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.886 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.886 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:13:20.886 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.207352+0000", 2024-09-17T07:13:20.886 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00013440899999999999 2024-09-17T07:13:20.886 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.886 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.886 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 1,2", 2024-09-17T07:13:20.886 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.207482+0000", 2024-09-17T07:13:20.886 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000130097 2024-09-17T07:13:20.886 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.887 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.887 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:13:20.887 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.213820+0000", 2024-09-17T07:13:20.887 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0063380069999999997 2024-09-17T07:13:20.887 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.887 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.887 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:13:20.887 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.220880+0000", 2024-09-17T07:13:20.887 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0070600790000000004 2024-09-17T07:13:20.887 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.887 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.887 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:13:20.888 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.221313+0000", 2024-09-17T07:13:20.888 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00043324500000000002 2024-09-17T07:13:20.888 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.888 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.888 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:13:20.888 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.221353+0000", 2024-09-17T07:13:20.888 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.9934000000000001e-05 2024-09-17T07:13:20.888 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.888 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.888 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:13:20.888 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.221386+0000", 2024-09-17T07:13:20.888 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.3135e-05 2024-09-17T07:13:20.889 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:20.889 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:13:20.889 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:20.889 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.889 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.889 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.24929.0:164833 8.b 8:d91c7c28:::benchmark_data_smithi100_102803_object34329:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-09-17T07:13:20.889 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:10:17.303139+0000", 2024-09-17T07:13:20.889 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 183.19677821900001, 2024-09-17T07:13:20.889 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.92849971600000003, 2024-09-17T07:13:20.889 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:13:20.889 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:13:20.889 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:13:20.889 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.24929", 2024-09-17T07:13:20.890 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/564515582", 2024-09-17T07:13:20.890 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 164833 2024-09-17T07:13:20.890 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.890 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:13:20.890 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.890 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:13:20.890 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.303139+0000", 2024-09-17T07:13:20.890 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:20.890 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.890 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.890 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:13:20.890 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.303139+0000", 2024-09-17T07:13:20.890 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:20.890 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.891 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.891 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:13:20.891 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.303139+0000", 2024-09-17T07:13:20.891 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999995 2024-09-17T07:13:20.891 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.891 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.891 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:13:20.891 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.303142+0000", 2024-09-17T07:13:20.891 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.4359999999999998e-06 2024-09-17T07:13:20.891 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.891 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.891 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:13:20.891 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.303148+0000", 2024-09-17T07:13:20.892 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.2639999999999997e-06 2024-09-17T07:13:20.892 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.892 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.892 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:13:20.892 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.303153+0000", 2024-09-17T07:13:20.892 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.6779999999999999e-06 2024-09-17T07:13:20.892 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.892 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.892 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:20.892 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.303191+0000", 2024-09-17T07:13:20.892 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.8112999999999999e-05 2024-09-17T07:13:20.892 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.893 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.893 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:13:20.893 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.303196+0000", 2024-09-17T07:13:20.893 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.4580000000000001e-06 2024-09-17T07:13:20.893 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.893 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.893 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:20.893 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.208847+0000", 2024-09-17T07:13:20.893 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.90565048999999997 2024-09-17T07:13:20.893 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.893 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.893 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:13:20.893 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.208954+0000", 2024-09-17T07:13:20.894 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000107046 2024-09-17T07:13:20.894 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.894 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.894 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 1,2", 2024-09-17T07:13:20.894 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.209039+0000", 2024-09-17T07:13:20.894 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.5461000000000001e-05 2024-09-17T07:13:20.894 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.894 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.894 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:13:20.894 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.215573+0000", 2024-09-17T07:13:20.894 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0065331590000000002 2024-09-17T07:13:20.894 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.894 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.895 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:13:20.895 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.221425+0000", 2024-09-17T07:13:20.895 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0058528169999999997 2024-09-17T07:13:20.895 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.895 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.895 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:13:20.895 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.231554+0000", 2024-09-17T07:13:20.895 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.010129035 2024-09-17T07:13:20.895 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.895 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.895 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:13:20.895 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.231601+0000", 2024-09-17T07:13:20.895 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.6687999999999998e-05 2024-09-17T07:13:20.895 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.896 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.896 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:13:20.896 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.231639+0000", 2024-09-17T07:13:20.896 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.7604000000000001e-05 2024-09-17T07:13:20.896 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:20.896 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:13:20.896 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:20.896 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.896 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.896 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.24929.0:164844 8.b 8:d3d463b4:::benchmark_data_smithi100_102803_object34340:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-09-17T07:13:20.896 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:10:17.313421+0000", 2024-09-17T07:13:20.896 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 183.18649604999999, 2024-09-17T07:13:20.896 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.91846116300000002, 2024-09-17T07:13:20.897 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:13:20.897 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:13:20.897 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:13:20.897 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.24929", 2024-09-17T07:13:20.897 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/564515582", 2024-09-17T07:13:20.897 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 164844 2024-09-17T07:13:20.897 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.897 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:13:20.897 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.897 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:13:20.897 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.313421+0000", 2024-09-17T07:13:20.897 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:20.898 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.898 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.898 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:13:20.898 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.313421+0000", 2024-09-17T07:13:20.898 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:20.898 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.898 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.898 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:13:20.898 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.313420+0000", 2024-09-17T07:13:20.898 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:13:20.898 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.898 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.898 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:13:20.899 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.313424+0000", 2024-09-17T07:13:20.899 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.9709999999999998e-06 2024-09-17T07:13:20.899 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.899 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.899 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:13:20.899 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.313430+0000", 2024-09-17T07:13:20.899 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.3010000000000003e-06 2024-09-17T07:13:20.899 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.899 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.899 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:13:20.899 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.313435+0000", 2024-09-17T07:13:20.899 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.1540000000000001e-06 2024-09-17T07:13:20.899 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.900 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.900 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:20.900 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.313478+0000", 2024-09-17T07:13:20.900 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.2960999999999997e-05 2024-09-17T07:13:20.900 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.900 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.900 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:13:20.900 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.313486+0000", 2024-09-17T07:13:20.900 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.9219999999999997e-06 2024-09-17T07:13:20.900 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.900 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.900 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:20.901 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.210370+0000", 2024-09-17T07:13:20.901 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.89688354999999997 2024-09-17T07:13:20.901 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.901 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.901 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:13:20.901 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.210471+0000", 2024-09-17T07:13:20.901 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000101307 2024-09-17T07:13:20.901 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.901 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.901 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 1,2", 2024-09-17T07:13:20.901 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.210532+0000", 2024-09-17T07:13:20.901 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.0868000000000001e-05 2024-09-17T07:13:20.901 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.902 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.902 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:13:20.902 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.215579+0000", 2024-09-17T07:13:20.902 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.005046811 2024-09-17T07:13:20.902 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.902 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.902 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:13:20.902 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.221473+0000", 2024-09-17T07:13:20.902 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0058948120000000001 2024-09-17T07:13:20.902 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.902 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.902 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:13:20.903 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.231808+0000", 2024-09-17T07:13:20.903 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.010334622 2024-09-17T07:13:20.903 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.903 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.903 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:13:20.903 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.231848+0000", 2024-09-17T07:13:20.903 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.9780000000000002e-05 2024-09-17T07:13:20.903 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.903 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.903 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:13:20.903 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.231882+0000", 2024-09-17T07:13:20.903 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.4505999999999998e-05 2024-09-17T07:13:20.904 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:20.904 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:13:20.904 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:20.904 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.904 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.904 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.24929.0:164852 8.b 8:dc4d72d5:::benchmark_data_smithi100_102803_object34348:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-09-17T07:13:20.904 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:10:17.317343+0000", 2024-09-17T07:13:20.904 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 183.182574719, 2024-09-17T07:13:20.904 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.91462852100000003, 2024-09-17T07:13:20.904 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:13:20.905 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:13:20.905 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:13:20.905 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.24929", 2024-09-17T07:13:20.905 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/564515582", 2024-09-17T07:13:20.905 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 164852 2024-09-17T07:13:20.905 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.905 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:13:20.905 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.905 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:13:20.905 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.317343+0000", 2024-09-17T07:13:20.905 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:20.905 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.906 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.906 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:13:20.906 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.317343+0000", 2024-09-17T07:13:20.906 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:20.906 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.906 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.906 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:13:20.906 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.317341+0000", 2024-09-17T07:13:20.906 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999981 2024-09-17T07:13:20.906 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.906 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.907 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:13:20.907 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.317347+0000", 2024-09-17T07:13:20.907 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.4300000000000003e-06 2024-09-17T07:13:20.907 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.907 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.907 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:13:20.907 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.317361+0000", 2024-09-17T07:13:20.907 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.3392e-05 2024-09-17T07:13:20.907 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.907 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.907 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:13:20.907 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.317371+0000", 2024-09-17T07:13:20.908 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0421999999999999e-05 2024-09-17T07:13:20.908 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.908 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.908 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:20.908 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.317445+0000", 2024-09-17T07:13:20.908 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.4203999999999996e-05 2024-09-17T07:13:20.908 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.908 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.908 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:13:20.908 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.317453+0000", 2024-09-17T07:13:20.908 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.8690000000000004e-06 2024-09-17T07:13:20.908 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.908 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.909 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:20.909 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.212084+0000", 2024-09-17T07:13:20.909 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.89463097700000005 2024-09-17T07:13:20.909 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.909 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.909 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:13:20.909 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.212206+0000", 2024-09-17T07:13:20.909 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000122111 2024-09-17T07:13:20.909 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.909 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.909 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 1,2", 2024-09-17T07:13:20.909 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.212291+0000", 2024-09-17T07:13:20.909 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.4608e-05 2024-09-17T07:13:20.910 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.910 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.910 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:13:20.910 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.218235+0000", 2024-09-17T07:13:20.910 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0059445419999999997 2024-09-17T07:13:20.910 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.910 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.910 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:13:20.910 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.231721+0000", 2024-09-17T07:13:20.910 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.013485914999999999 2024-09-17T07:13:20.910 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.910 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.910 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:13:20.911 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.231913+0000", 2024-09-17T07:13:20.911 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000191344 2024-09-17T07:13:20.911 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.911 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.911 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:13:20.911 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.231943+0000", 2024-09-17T07:13:20.911 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.0685000000000002e-05 2024-09-17T07:13:20.911 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.911 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.911 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:13:20.911 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.231971+0000", 2024-09-17T07:13:20.911 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.7835e-05 2024-09-17T07:13:20.911 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:20.911 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:13:20.912 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:20.912 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.912 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.912 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.24929.0:164879 8.b 8:d659370b:::benchmark_data_smithi100_102803_object34375:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-09-17T07:13:20.912 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:10:17.346186+0000", 2024-09-17T07:13:20.912 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 183.15373152199999, 2024-09-17T07:13:20.912 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.88586965200000001, 2024-09-17T07:13:20.912 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:13:20.912 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:13:20.912 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:13:20.912 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.24929", 2024-09-17T07:13:20.912 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/564515582", 2024-09-17T07:13:20.913 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 164879 2024-09-17T07:13:20.913 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.913 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:13:20.913 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.913 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:13:20.913 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.346186+0000", 2024-09-17T07:13:20.913 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:20.913 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.913 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.913 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:13:20.913 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.346186+0000", 2024-09-17T07:13:20.913 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:20.914 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.914 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.914 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:13:20.914 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.346184+0000", 2024-09-17T07:13:20.914 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:13:20.914 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.914 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.914 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:13:20.914 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.346190+0000", 2024-09-17T07:13:20.914 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.412e-06 2024-09-17T07:13:20.914 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.914 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.915 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:13:20.915 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.346199+0000", 2024-09-17T07:13:20.915 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.4919999999999999e-06 2024-09-17T07:13:20.915 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.915 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.915 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:13:20.915 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.346205+0000", 2024-09-17T07:13:20.915 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.8749999999999997e-06 2024-09-17T07:13:20.915 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.915 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.915 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:20.915 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.346249+0000", 2024-09-17T07:13:20.915 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.4239000000000001e-05 2024-09-17T07:13:20.916 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.916 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.916 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:13:20.916 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.346258+0000", 2024-09-17T07:13:20.916 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.9050000000000003e-06 2024-09-17T07:13:20.916 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.916 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.916 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:20.916 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.213866+0000", 2024-09-17T07:13:20.916 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.86760820299999997 2024-09-17T07:13:20.916 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.916 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.917 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:13:20.917 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.213988+0000", 2024-09-17T07:13:20.917 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000121691 2024-09-17T07:13:20.917 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.917 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.917 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 1,2", 2024-09-17T07:13:20.917 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.214103+0000", 2024-09-17T07:13:20.917 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000114852 2024-09-17T07:13:20.917 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.917 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.917 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:13:20.917 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.220838+0000", 2024-09-17T07:13:20.918 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0067347470000000001 2024-09-17T07:13:20.918 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.918 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.918 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:13:20.918 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.231756+0000", 2024-09-17T07:13:20.918 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.010918673 2024-09-17T07:13:20.918 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.918 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.918 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:13:20.918 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.231999+0000", 2024-09-17T07:13:20.918 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00024308700000000001 2024-09-17T07:13:20.918 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.919 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.919 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:13:20.919 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.232029+0000", 2024-09-17T07:13:20.919 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.9258999999999999e-05 2024-09-17T07:13:20.919 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.919 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.919 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:13:20.919 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.232055+0000", 2024-09-17T07:13:20.919 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.6848000000000001e-05 2024-09-17T07:13:20.919 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:20.919 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:13:20.919 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:20.920 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.920 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.920 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.24929.0:164910 8.b 8:dc89b12b:::benchmark_data_smithi100_102803_object34406:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-09-17T07:13:20.920 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:10:17.380469+0000", 2024-09-17T07:13:20.920 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 183.11944820100001, 2024-09-17T07:13:20.920 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.87627316499999997, 2024-09-17T07:13:20.920 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:13:20.920 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:13:20.920 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:13:20.920 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.24929", 2024-09-17T07:13:20.920 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/564515582", 2024-09-17T07:13:20.920 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 164910 2024-09-17T07:13:20.921 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.921 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:13:20.921 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.921 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:13:20.921 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.380469+0000", 2024-09-17T07:13:20.921 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:20.921 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.921 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.921 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:13:20.921 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.380469+0000", 2024-09-17T07:13:20.921 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:20.921 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.922 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.922 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:13:20.922 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.380467+0000", 2024-09-17T07:13:20.922 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999981 2024-09-17T07:13:20.922 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.922 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.922 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:13:20.922 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.380474+0000", 2024-09-17T07:13:20.922 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.8689999999999998e-06 2024-09-17T07:13:20.922 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.922 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.922 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:13:20.922 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.380489+0000", 2024-09-17T07:13:20.923 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.4749999999999999e-05 2024-09-17T07:13:20.923 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.923 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.923 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:13:20.923 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.380497+0000", 2024-09-17T07:13:20.923 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.2889999999999998e-06 2024-09-17T07:13:20.923 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.923 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.923 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:20.923 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.380560+0000", 2024-09-17T07:13:20.923 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.3288999999999997e-05 2024-09-17T07:13:20.923 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.924 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.924 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:13:20.924 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.380569+0000", 2024-09-17T07:13:20.924 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.9279999999999999e-06 2024-09-17T07:13:20.924 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.924 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.924 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:20.924 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.221510+0000", 2024-09-17T07:13:20.924 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.84094051299999995 2024-09-17T07:13:20.924 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.924 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.924 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:13:20.924 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.221636+0000", 2024-09-17T07:13:20.925 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00012644800000000001 2024-09-17T07:13:20.925 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.925 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.925 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 1,2", 2024-09-17T07:13:20.925 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.221746+0000", 2024-09-17T07:13:20.925 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000109836 2024-09-17T07:13:20.925 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.925 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.925 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:13:20.925 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.231503+0000", 2024-09-17T07:13:20.925 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0097572619999999992 2024-09-17T07:13:20.925 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.926 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.926 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:13:20.926 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.232110+0000", 2024-09-17T07:13:20.926 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00060627700000000001 2024-09-17T07:13:20.926 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.926 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.926 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:13:20.926 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.256666+0000", 2024-09-17T07:13:20.926 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.024556069999999999 2024-09-17T07:13:20.926 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.926 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.927 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:13:20.927 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.256708+0000", 2024-09-17T07:13:20.927 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.2211e-05 2024-09-17T07:13:20.927 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.927 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.927 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:13:20.927 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.256742+0000", 2024-09-17T07:13:20.927 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.4199999999999998e-05 2024-09-17T07:13:20.927 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:20.927 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:13:20.927 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:20.927 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.928 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.928 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.24929.0:164913 8.b 8:dcec2487:::benchmark_data_smithi100_102803_object34409:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-09-17T07:13:20.928 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:10:17.383765+0000", 2024-09-17T07:13:20.928 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 183.116151884, 2024-09-17T07:13:20.928 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.87384644, 2024-09-17T07:13:20.928 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:13:20.928 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:13:20.928 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:13:20.928 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.24929", 2024-09-17T07:13:20.928 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/564515582", 2024-09-17T07:13:20.929 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 164913 2024-09-17T07:13:20.929 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.929 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:13:20.929 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.929 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:13:20.929 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.383765+0000", 2024-09-17T07:13:20.929 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:20.929 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.929 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.929 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:13:20.929 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.383765+0000", 2024-09-17T07:13:20.929 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:20.930 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.930 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.930 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:13:20.930 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.383764+0000", 2024-09-17T07:13:20.930 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:13:20.930 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.930 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.930 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:13:20.930 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.383771+0000", 2024-09-17T07:13:20.930 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.1640000000000004e-06 2024-09-17T07:13:20.930 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.930 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.930 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:13:20.931 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.383780+0000", 2024-09-17T07:13:20.931 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.2699999999999993e-06 2024-09-17T07:13:20.931 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.931 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.931 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:13:20.931 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.383786+0000", 2024-09-17T07:13:20.931 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.1989999999999997e-06 2024-09-17T07:13:20.931 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.931 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.931 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:20.931 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.383822+0000", 2024-09-17T07:13:20.932 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.5952999999999999e-05 2024-09-17T07:13:20.932 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.932 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.932 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:13:20.932 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.383827+0000", 2024-09-17T07:13:20.932 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.6879999999999996e-06 2024-09-17T07:13:20.932 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.932 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.932 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:20.932 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.224104+0000", 2024-09-17T07:13:20.932 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.84027645100000004 2024-09-17T07:13:20.932 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.932 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.933 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:13:20.933 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.224225+0000", 2024-09-17T07:13:20.933 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000121618 2024-09-17T07:13:20.933 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.933 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.933 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 1,2", 2024-09-17T07:13:20.933 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.224337+0000", 2024-09-17T07:13:20.933 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000111352 2024-09-17T07:13:20.933 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.933 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.933 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:13:20.933 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.231510+0000", 2024-09-17T07:13:20.933 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0071731060000000003 2024-09-17T07:13:20.934 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.934 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.934 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:13:20.934 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.232151+0000", 2024-09-17T07:13:20.934 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00064107699999999999 2024-09-17T07:13:20.934 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.934 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.934 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:13:20.934 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.257530+0000", 2024-09-17T07:13:20.934 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.025379235 2024-09-17T07:13:20.934 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.934 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.934 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:13:20.935 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.257575+0000", 2024-09-17T07:13:20.935 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.5263999999999999e-05 2024-09-17T07:13:20.935 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:20.935 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:20.935 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:13:20.935 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.257612+0000", 2024-09-17T07:13:20.935 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.6477999999999999e-05 2024-09-17T07:13:20.935 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:20.935 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:13:20.935 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:20.935 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:20.935 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:13:20.935 INFO:teuthology.orchestra.run.smithi135.stdout:} 2024-09-17T07:13:20.936 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:20 smithi100 ceph-mon[29092]: 5.16 scrub starts 2024-09-17T07:13:20.936 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:20 smithi100 ceph-mon[29092]: 5.16 scrub ok 2024-09-17T07:13:20.937 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:20 smithi100 ceph-mon[39462]: 5.16 scrub starts 2024-09-17T07:13:20.937 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:20 smithi100 ceph-mon[39462]: 5.16 scrub ok 2024-09-17T07:13:20.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:20.938 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:20.938 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:13:20.939 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:13:21.039 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:13:21.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:21.075 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:21.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:13:21.176 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:13:21.203 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:21.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:21.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:13:21.221 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 64 16 105199 105183 102.704 65.625 0.00482243 0.00973205 2024-09-17T07:13:21.306 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:13:21.347 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:21.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:21.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:13:21.449 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:13:21.493 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:21.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:21.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:13:21.595 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:13:21.634 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:21.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:21.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:13:21.665 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:21 smithi135 ceph-mon[33238]: 2.14 scrub starts 2024-09-17T07:13:21.665 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:21 smithi135 ceph-mon[33238]: 2.14 scrub ok 2024-09-17T07:13:21.665 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:21 smithi135 ceph-mon[33238]: 2.c scrub starts 2024-09-17T07:13:21.665 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:21 smithi135 ceph-mon[33238]: 2.c scrub ok 2024-09-17T07:13:21.665 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:21 smithi135 ceph-mon[33238]: 5.1c scrub starts 2024-09-17T07:13:21.665 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:21 smithi135 ceph-mon[33238]: 5.1c scrub ok 2024-09-17T07:13:21.666 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:21 smithi135 ceph-mon[33238]: 5.8 scrub starts 2024-09-17T07:13:21.666 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:21 smithi135 ceph-mon[33238]: 5.8 scrub ok 2024-09-17T07:13:21.666 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:21 smithi135 ceph-mon[33238]: pgmap v500: 148 pgs: 148 active+clean; 6.2 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1023 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-09-17T07:13:21.674 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:21 smithi100 ceph-mon[29092]: 2.14 scrub starts 2024-09-17T07:13:21.674 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:21 smithi100 ceph-mon[29092]: 2.14 scrub ok 2024-09-17T07:13:21.674 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:21 smithi100 ceph-mon[29092]: 2.c scrub starts 2024-09-17T07:13:21.674 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:21 smithi100 ceph-mon[29092]: 2.c scrub ok 2024-09-17T07:13:21.674 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:21 smithi100 ceph-mon[29092]: 5.1c scrub starts 2024-09-17T07:13:21.674 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:21 smithi100 ceph-mon[29092]: 5.1c scrub ok 2024-09-17T07:13:21.675 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:21 smithi100 ceph-mon[29092]: 5.8 scrub starts 2024-09-17T07:13:21.675 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:21 smithi100 ceph-mon[29092]: 5.8 scrub ok 2024-09-17T07:13:21.675 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:21 smithi100 ceph-mon[29092]: pgmap v500: 148 pgs: 148 active+clean; 6.2 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1023 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-09-17T07:13:21.675 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:21 smithi100 ceph-mon[39462]: 2.14 scrub starts 2024-09-17T07:13:21.675 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:21 smithi100 ceph-mon[39462]: 2.14 scrub ok 2024-09-17T07:13:21.675 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:21 smithi100 ceph-mon[39462]: 2.c scrub starts 2024-09-17T07:13:21.675 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:21 smithi100 ceph-mon[39462]: 2.c scrub ok 2024-09-17T07:13:21.675 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:21 smithi100 ceph-mon[39462]: 5.1c scrub starts 2024-09-17T07:13:21.675 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:21 smithi100 ceph-mon[39462]: 5.1c scrub ok 2024-09-17T07:13:21.675 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:21 smithi100 ceph-mon[39462]: 5.8 scrub starts 2024-09-17T07:13:21.675 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:21 smithi100 ceph-mon[39462]: 5.8 scrub ok 2024-09-17T07:13:21.675 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:21 smithi100 ceph-mon[39462]: pgmap v500: 148 pgs: 148 active+clean; 6.2 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1023 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-09-17T07:13:21.737 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:13:21.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:21.774 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:21.775 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:13:21.876 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:13:21.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:21.909 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:21.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:13:22.010 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:13:22.044 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:22.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:22.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:13:22.143 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-17T07:13:22.145 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:13:22.221 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 65 16 106420 106404 102.298 76.3125 0.0147264 0.00977045 2024-09-17T07:13:22.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:22.228 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:22.228 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:13:22.329 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:13:22.353 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:22.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:22.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:13:22.455 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:13:22.508 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:22.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:22.509 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:13:22.518 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:13:22.610 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:13:22.643 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:22.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:22.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:13:22.696 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:22 smithi135 ceph-mon[33238]: 6.10 scrub starts 2024-09-17T07:13:22.696 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:22 smithi135 ceph-mon[33238]: 6.10 scrub ok 2024-09-17T07:13:22.696 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:22 smithi135 ceph-mon[33238]: 2.8 scrub starts 2024-09-17T07:13:22.696 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:22 smithi135 ceph-mon[33238]: 2.8 scrub ok 2024-09-17T07:13:22.745 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:13:22.766 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:22.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:22.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:13:22.868 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:13:22.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:22 smithi100 ceph-mon[29092]: 6.10 scrub starts 2024-09-17T07:13:22.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:22 smithi100 ceph-mon[29092]: 6.10 scrub ok 2024-09-17T07:13:22.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:22 smithi100 ceph-mon[29092]: 2.8 scrub starts 2024-09-17T07:13:22.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:22 smithi100 ceph-mon[29092]: 2.8 scrub ok 2024-09-17T07:13:22.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:22 smithi100 ceph-mon[39462]: 6.10 scrub starts 2024-09-17T07:13:22.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:22 smithi100 ceph-mon[39462]: 6.10 scrub ok 2024-09-17T07:13:22.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:22 smithi100 ceph-mon[39462]: 2.8 scrub starts 2024-09-17T07:13:22.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:22 smithi100 ceph-mon[39462]: 2.8 scrub ok 2024-09-17T07:13:22.907 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:22.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:22.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:13:22.940 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:13:23.009 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:13:23.045 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:23.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:23.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:13:23.147 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:13:23.179 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:23.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:23.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:13:23.221 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 66 16 107508 107492 101.778 68 0.0172447 0.00982075 2024-09-17T07:13:23.281 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:13:23.317 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:23.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:23.318 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:13:23.419 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:13:23.426 INFO:teuthology.orchestra.run.smithi100.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-09-17T07:13:23.464 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:23.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:23.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:13:23.538 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:23 smithi100 ceph-mon[29092]: 6.c deep-scrub starts 2024-09-17T07:13:23.538 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:23 smithi100 ceph-mon[29092]: 6.c deep-scrub ok 2024-09-17T07:13:23.538 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:23 smithi100 ceph-mon[29092]: 4.1a scrub starts 2024-09-17T07:13:23.538 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:23 smithi100 ceph-mon[29092]: 4.1a scrub ok 2024-09-17T07:13:23.538 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:23 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/857986540' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-09-17T07:13:23.538 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:23 smithi100 ceph-mon[29092]: 5.15 scrub starts 2024-09-17T07:13:23.538 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:23 smithi100 ceph-mon[29092]: 5.15 scrub ok 2024-09-17T07:13:23.538 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:23 smithi100 ceph-mon[29092]: pgmap v501: 148 pgs: 148 active+clean; 6.2 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1023 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-09-17T07:13:23.539 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:23 smithi100 ceph-mon[39462]: 6.c deep-scrub starts 2024-09-17T07:13:23.539 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:23 smithi100 ceph-mon[39462]: 6.c deep-scrub ok 2024-09-17T07:13:23.539 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:23 smithi100 ceph-mon[39462]: 4.1a scrub starts 2024-09-17T07:13:23.540 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:23 smithi100 ceph-mon[39462]: 4.1a scrub ok 2024-09-17T07:13:23.540 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:23 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/857986540' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-09-17T07:13:23.540 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:23 smithi100 ceph-mon[39462]: 5.15 scrub starts 2024-09-17T07:13:23.540 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:23 smithi100 ceph-mon[39462]: 5.15 scrub ok 2024-09-17T07:13:23.540 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:23 smithi100 ceph-mon[39462]: pgmap v501: 148 pgs: 148 active+clean; 6.2 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1023 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-09-17T07:13:23.566 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:13:23.597 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:23.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:23.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:13:23.699 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:13:23.702 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:23 smithi135 ceph-mon[33238]: 6.c deep-scrub starts 2024-09-17T07:13:23.702 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:23 smithi135 ceph-mon[33238]: 6.c deep-scrub ok 2024-09-17T07:13:23.702 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:23 smithi135 ceph-mon[33238]: 4.1a scrub starts 2024-09-17T07:13:23.702 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:23 smithi135 ceph-mon[33238]: 4.1a scrub ok 2024-09-17T07:13:23.702 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:23 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/857986540' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-09-17T07:13:23.703 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:23 smithi135 ceph-mon[33238]: 5.15 scrub starts 2024-09-17T07:13:23.703 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:23 smithi135 ceph-mon[33238]: 5.15 scrub ok 2024-09-17T07:13:23.703 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:23 smithi135 ceph-mon[33238]: pgmap v501: 148 pgs: 148 active+clean; 6.2 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1023 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-09-17T07:13:23.741 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:23.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:23.742 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:13:23.843 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:13:23.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:23.894 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:23.895 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:13:23.995 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:13:24.032 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:24.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:24.034 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:13:24.060 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:13:24.134 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:13:24.167 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:24.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:24.168 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:13:24.221 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 67 16 108734 108718 101.402 76.625 0.0119416 0.00985742 2024-09-17T07:13:24.269 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:13:24.310 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:24.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:24.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:13:24.412 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:13:24.446 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:24.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:24.447 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:13:24.449 INFO:teuthology.orchestra.run.smithi135.stdout:{ 2024-09-17T07:13:24.449 INFO:teuthology.orchestra.run.smithi135.stdout: "ops": [ 2024-09-17T07:13:24.449 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.449 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.15330.0:108985 9.6 9:6c703b93:::benchmark_data_smithi100_128631_object108984:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e259)", 2024-09-17T07:13:24.449 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:13:24.437100+0000", 2024-09-17T07:13:24.449 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 0.0095780929999999993, 2024-09-17T07:13:24.449 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0096183580000000005, 2024-09-17T07:13:24.449 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:13:24.449 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "waiting for sub ops", 2024-09-17T07:13:24.450 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:13:24.450 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.15330", 2024-09-17T07:13:24.450 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/1361381749", 2024-09-17T07:13:24.450 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 108985 2024-09-17T07:13:24.450 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.450 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:13:24.450 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.450 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:13:24.450 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.437100+0000", 2024-09-17T07:13:24.450 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:24.450 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.450 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.451 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:13:24.451 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.437100+0000", 2024-09-17T07:13:24.451 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:24.451 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.451 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.451 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:13:24.451 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.437098+0000", 2024-09-17T07:13:24.451 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:13:24.451 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.452 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.452 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:13:24.452 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.437167+0000", 2024-09-17T07:13:24.452 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.8279000000000001e-05 2024-09-17T07:13:24.452 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.452 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.452 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:13:24.452 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.437182+0000", 2024-09-17T07:13:24.452 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.5031000000000001e-05 2024-09-17T07:13:24.452 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.452 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.452 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:13:24.453 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.437190+0000", 2024-09-17T07:13:24.453 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.4670000000000004e-06 2024-09-17T07:13:24.453 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.453 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.453 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:24.453 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.440041+0000", 2024-09-17T07:13:24.453 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0028507979999999999 2024-09-17T07:13:24.453 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.453 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.453 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:13:24.453 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.440155+0000", 2024-09-17T07:13:24.453 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000114237 2024-09-17T07:13:24.453 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.454 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.454 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 1,2", 2024-09-17T07:13:24.454 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.440283+0000", 2024-09-17T07:13:24.454 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00012828399999999999 2024-09-17T07:13:24.454 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.454 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.454 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:13:24.454 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.443085+0000", 2024-09-17T07:13:24.454 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.002801343 2024-09-17T07:13:24.454 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:24.454 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:13:24.455 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:24.455 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.455 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.455 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_repop(client.15330.0:108984 9.d e259/236 9:b5c9e686:::benchmark_data_smithi100_128631_object108983:head v 259'6796, mlcod=259'6792)", 2024-09-17T07:13:24.455 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:13:24.437364+0000", 2024-09-17T07:13:24.455 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 0.0093140770000000005, 2024-09-17T07:13:24.455 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0094467790000000006, 2024-09-17T07:13:24.455 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:13:24.455 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "started", 2024-09-17T07:13:24.456 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:13:24.456 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.456 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:13:24.456 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.437364+0000", 2024-09-17T07:13:24.456 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:24.456 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.456 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.456 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:13:24.456 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.437364+0000", 2024-09-17T07:13:24.456 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:24.456 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.456 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.456 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:13:24.457 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.437364+0000", 2024-09-17T07:13:24.457 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999995 2024-09-17T07:13:24.457 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.457 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.457 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:13:24.457 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.437386+0000", 2024-09-17T07:13:24.457 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.209e-05 2024-09-17T07:13:24.457 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.457 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.457 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:13:24.457 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.437391+0000", 2024-09-17T07:13:24.457 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.6219999999999997e-06 2024-09-17T07:13:24.458 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.458 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.458 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:13:24.458 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.437406+0000", 2024-09-17T07:13:24.458 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.4239999999999999e-05 2024-09-17T07:13:24.459 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.459 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.459 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:24.459 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.442329+0000", 2024-09-17T07:13:24.459 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0049236929999999998 2024-09-17T07:13:24.459 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.459 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.459 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:13:24.459 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.442348+0000", 2024-09-17T07:13:24.459 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.9120999999999999e-05 2024-09-17T07:13:24.459 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:24.459 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:13:24.460 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:24.460 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.460 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.460 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.15330.0:108986 9.a 9:5d9eb312:::benchmark_data_smithi100_128631_object108985:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e259)", 2024-09-17T07:13:24.460 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:13:24.438963+0000", 2024-09-17T07:13:24.460 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 0.0077146489999999996, 2024-09-17T07:13:24.460 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0079146800000000003, 2024-09-17T07:13:24.460 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:13:24.460 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "waiting for sub ops", 2024-09-17T07:13:24.460 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:13:24.460 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.15330", 2024-09-17T07:13:24.460 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/1361381749", 2024-09-17T07:13:24.460 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 108986 2024-09-17T07:13:24.461 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.461 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:13:24.461 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.461 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:13:24.461 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.438963+0000", 2024-09-17T07:13:24.461 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:24.461 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.461 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.461 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:13:24.461 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.438963+0000", 2024-09-17T07:13:24.461 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:24.461 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.462 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.462 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:13:24.462 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.438962+0000", 2024-09-17T07:13:24.462 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.999999 2024-09-17T07:13:24.462 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.462 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.462 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:13:24.462 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.438994+0000", 2024-09-17T07:13:24.462 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.1319000000000003e-05 2024-09-17T07:13:24.462 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.462 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.462 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:13:24.462 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.439005+0000", 2024-09-17T07:13:24.463 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.1207e-05 2024-09-17T07:13:24.463 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.463 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.463 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:13:24.463 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.439012+0000", 2024-09-17T07:13:24.463 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.5290000000000003e-06 2024-09-17T07:13:24.463 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.463 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.463 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:24.463 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.439166+0000", 2024-09-17T07:13:24.463 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00015378200000000001 2024-09-17T07:13:24.463 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.464 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.464 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:13:24.464 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.439307+0000", 2024-09-17T07:13:24.464 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00014088800000000001 2024-09-17T07:13:24.464 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.464 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.464 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 1,7", 2024-09-17T07:13:24.464 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.439471+0000", 2024-09-17T07:13:24.464 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000163988 2024-09-17T07:13:24.464 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.464 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.465 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:13:24.465 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.442999+0000", 2024-09-17T07:13:24.465 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0035281599999999998 2024-09-17T07:13:24.465 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:24.465 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:13:24.465 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:24.465 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.465 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.466 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.15330.0:108988 9.f 9:f827fd5e:::benchmark_data_smithi100_128631_object108987:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e259)", 2024-09-17T07:13:24.466 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:13:24.439024+0000", 2024-09-17T07:13:24.466 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 0.0076538279999999997, 2024-09-17T07:13:24.466 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0079389860000000003, 2024-09-17T07:13:24.466 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:13:24.467 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "waiting for sub ops", 2024-09-17T07:13:24.467 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:13:24.467 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.15330", 2024-09-17T07:13:24.467 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/1361381749", 2024-09-17T07:13:24.467 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 108988 2024-09-17T07:13:24.467 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.467 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:13:24.467 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.467 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:13:24.467 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.439024+0000", 2024-09-17T07:13:24.467 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:24.468 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.468 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.468 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:13:24.468 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.439024+0000", 2024-09-17T07:13:24.468 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:24.468 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.468 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.468 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:13:24.468 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.439024+0000", 2024-09-17T07:13:24.469 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999995 2024-09-17T07:13:24.469 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.469 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.469 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:13:24.469 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.439048+0000", 2024-09-17T07:13:24.469 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.3808999999999999e-05 2024-09-17T07:13:24.469 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.469 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.469 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:13:24.469 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.439054+0000", 2024-09-17T07:13:24.469 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.3620000000000002e-06 2024-09-17T07:13:24.470 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.470 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.470 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:13:24.470 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.439059+0000", 2024-09-17T07:13:24.470 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.5400000000000003e-06 2024-09-17T07:13:24.470 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.470 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.471 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:24.471 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.439375+0000", 2024-09-17T07:13:24.471 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00031578999999999998 2024-09-17T07:13:24.471 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.471 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.471 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:13:24.471 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.439504+0000", 2024-09-17T07:13:24.471 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00012841 2024-09-17T07:13:24.471 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.471 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.471 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 1,2", 2024-09-17T07:13:24.471 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.439637+0000", 2024-09-17T07:13:24.472 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00013304699999999999 2024-09-17T07:13:24.472 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.472 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.472 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:13:24.472 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.443011+0000", 2024-09-17T07:13:24.472 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0033743250000000001 2024-09-17T07:13:24.472 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:24.472 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:13:24.472 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:24.472 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.472 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.472 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.15330.0:108992 9.f 9:fe58bce7:::benchmark_data_smithi100_128631_object108991:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e259)", 2024-09-17T07:13:24.473 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:13:24.443287+0000", 2024-09-17T07:13:24.473 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 0.0033911779999999999, 2024-09-17T07:13:24.473 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0037630369999999999, 2024-09-17T07:13:24.473 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:13:24.473 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "waiting for sub ops", 2024-09-17T07:13:24.473 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:13:24.473 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.15330", 2024-09-17T07:13:24.473 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/1361381749", 2024-09-17T07:13:24.474 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 108992 2024-09-17T07:13:24.474 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.474 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:13:24.474 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.474 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:13:24.474 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.443287+0000", 2024-09-17T07:13:24.474 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:24.474 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.474 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.474 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:13:24.474 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.443287+0000", 2024-09-17T07:13:24.474 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:24.475 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.475 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.475 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:13:24.475 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.443285+0000", 2024-09-17T07:13:24.475 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:13:24.475 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.475 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.475 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:13:24.475 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.443316+0000", 2024-09-17T07:13:24.475 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.0358999999999999e-05 2024-09-17T07:13:24.475 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.475 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.476 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:13:24.476 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.443325+0000", 2024-09-17T07:13:24.476 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.2990000000000002e-06 2024-09-17T07:13:24.476 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.476 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.476 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:13:24.476 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.443332+0000", 2024-09-17T07:13:24.476 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.9110000000000004e-06 2024-09-17T07:13:24.476 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.476 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.476 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:24.476 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.443390+0000", 2024-09-17T07:13:24.476 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.8094000000000003e-05 2024-09-17T07:13:24.477 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.477 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.477 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:13:24.477 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.443523+0000", 2024-09-17T07:13:24.477 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00013297 2024-09-17T07:13:24.477 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.477 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.477 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 1,2", 2024-09-17T07:13:24.477 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.443783+0000", 2024-09-17T07:13:24.477 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00026027400000000001 2024-09-17T07:13:24.477 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:24.477 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:13:24.478 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:24.478 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.478 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.478 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.15330.0:108993 9.6 9:6400b02f:::benchmark_data_smithi100_128631_object108992:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e259)", 2024-09-17T07:13:24.478 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:13:24.443616+0000", 2024-09-17T07:13:24.478 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 0.0030619190000000002, 2024-09-17T07:13:24.478 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0035124470000000001, 2024-09-17T07:13:24.478 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:13:24.478 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "waiting for sub ops", 2024-09-17T07:13:24.478 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:13:24.478 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.15330", 2024-09-17T07:13:24.478 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/1361381749", 2024-09-17T07:13:24.478 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 108993 2024-09-17T07:13:24.479 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.479 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:13:24.479 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.479 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:13:24.479 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.443616+0000", 2024-09-17T07:13:24.479 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:24.479 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.479 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.479 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:13:24.480 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.443616+0000", 2024-09-17T07:13:24.480 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:24.480 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.480 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.480 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:13:24.480 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.443615+0000", 2024-09-17T07:13:24.480 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.999999 2024-09-17T07:13:24.480 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.480 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.480 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:13:24.480 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.443678+0000", 2024-09-17T07:13:24.480 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.2823000000000005e-05 2024-09-17T07:13:24.481 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.481 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.481 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:13:24.481 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.443692+0000", 2024-09-17T07:13:24.481 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.4052e-05 2024-09-17T07:13:24.481 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.481 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.481 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:13:24.481 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.443698+0000", 2024-09-17T07:13:24.481 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.1580000000000002e-06 2024-09-17T07:13:24.481 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.481 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.482 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:24.482 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.444126+0000", 2024-09-17T07:13:24.482 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000427883 2024-09-17T07:13:24.482 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.482 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.482 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:13:24.482 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.444241+0000", 2024-09-17T07:13:24.482 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00011493300000000001 2024-09-17T07:13:24.482 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.482 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.482 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 1,2", 2024-09-17T07:13:24.482 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.444366+0000", 2024-09-17T07:13:24.482 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00012476800000000001 2024-09-17T07:13:24.483 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:24.483 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:13:24.483 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:24.483 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.483 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.483 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_repop(client.15330.0:108991 9.9 e259/236 9:927d05f2:::benchmark_data_smithi100_128631_object108990:head v 259'6916, mlcod=259'6913)", 2024-09-17T07:13:24.483 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:13:24.443985+0000", 2024-09-17T07:13:24.483 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 0.0026926839999999999, 2024-09-17T07:13:24.483 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0032204780000000001, 2024-09-17T07:13:24.483 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:13:24.483 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "started", 2024-09-17T07:13:24.484 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:13:24.484 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.484 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:13:24.484 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.443985+0000", 2024-09-17T07:13:24.484 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:24.484 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.484 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.484 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:13:24.484 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.443985+0000", 2024-09-17T07:13:24.484 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:24.484 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.484 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.484 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:13:24.485 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.443984+0000", 2024-09-17T07:13:24.485 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.999999 2024-09-17T07:13:24.485 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.485 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.485 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:13:24.485 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.444017+0000", 2024-09-17T07:13:24.485 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.2808999999999997e-05 2024-09-17T07:13:24.485 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.485 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.485 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:13:24.485 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.444030+0000", 2024-09-17T07:13:24.485 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.2658e-05 2024-09-17T07:13:24.486 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.486 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.486 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:13:24.486 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.444036+0000", 2024-09-17T07:13:24.486 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.2060000000000004e-06 2024-09-17T07:13:24.486 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.486 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.486 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:24.486 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.444071+0000", 2024-09-17T07:13:24.486 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.4992e-05 2024-09-17T07:13:24.486 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.486 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.486 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:13:24.487 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.444094+0000", 2024-09-17T07:13:24.487 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.3159999999999998e-05 2024-09-17T07:13:24.487 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:24.487 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:13:24.487 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:24.487 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.487 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.487 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_repop_reply(client.15330.0:108988 9.f e259/236)", 2024-09-17T07:13:24.487 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:13:24.446490+0000", 2024-09-17T07:13:24.487 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 0.00018788299999999999, 2024-09-17T07:13:24.487 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00077108400000000001, 2024-09-17T07:13:24.487 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:13:24.488 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "queued for pg", 2024-09-17T07:13:24.488 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:13:24.488 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.488 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:13:24.488 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.446490+0000", 2024-09-17T07:13:24.488 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:24.488 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.488 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.488 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:13:24.488 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.446490+0000", 2024-09-17T07:13:24.488 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:24.488 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.489 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.489 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:13:24.489 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.446489+0000", 2024-09-17T07:13:24.489 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.999999 2024-09-17T07:13:24.489 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.489 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.489 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:13:24.489 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.446498+0000", 2024-09-17T07:13:24.489 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.8869999999999998e-06 2024-09-17T07:13:24.489 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.490 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.490 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:13:24.490 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.446508+0000", 2024-09-17T07:13:24.490 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0132000000000001e-05 2024-09-17T07:13:24.490 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.490 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.490 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:13:24.490 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.446515+0000", 2024-09-17T07:13:24.490 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.1160000000000001e-06 2024-09-17T07:13:24.490 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:24.490 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:13:24.490 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:24.490 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.491 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.491 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.15330.0:108995 9.f 9.574a2fcf (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e259)", 2024-09-17T07:13:24.491 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:13:24.446571+0000", 2024-09-17T07:13:24.491 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 0.000106606, 2024-09-17T07:13:24.491 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00073873099999999996, 2024-09-17T07:13:24.491 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:13:24.491 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "queued for pg", 2024-09-17T07:13:24.491 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:13:24.491 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.15330", 2024-09-17T07:13:24.491 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/1361381749", 2024-09-17T07:13:24.491 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 108995 2024-09-17T07:13:24.492 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.492 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:13:24.492 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.492 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:13:24.492 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.446571+0000", 2024-09-17T07:13:24.492 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:24.492 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.492 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.492 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:13:24.492 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.446571+0000", 2024-09-17T07:13:24.492 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:24.492 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.493 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.493 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:13:24.493 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.446570+0000", 2024-09-17T07:13:24.493 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:13:24.493 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.493 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.493 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:13:24.493 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.446621+0000", 2024-09-17T07:13:24.493 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.1079999999999999e-05 2024-09-17T07:13:24.493 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.493 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.493 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:13:24.493 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.446631+0000", 2024-09-17T07:13:24.494 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.8470000000000008e-06 2024-09-17T07:13:24.494 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.494 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.494 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:13:24.494 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.446638+0000", 2024-09-17T07:13:24.494 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.7889999999999997e-06 2024-09-17T07:13:24.494 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:24.494 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:13:24.494 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:24.494 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.494 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.494 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.15330.0:108996 9.6 9:64600a7a:::benchmark_data_smithi100_128631_object108995:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e259)", 2024-09-17T07:13:24.494 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:13:24.446872+0000", 2024-09-17T07:13:24.495 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 4294967295.9998059, 2024-09-17T07:13:24.495 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00050090400000000004, 2024-09-17T07:13:24.495 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:13:24.495 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "waiting for sub ops", 2024-09-17T07:13:24.495 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:13:24.495 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.15330", 2024-09-17T07:13:24.495 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/1361381749", 2024-09-17T07:13:24.495 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 108996 2024-09-17T07:13:24.495 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.495 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:13:24.495 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.495 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:13:24.496 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.446872+0000", 2024-09-17T07:13:24.496 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:24.496 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.496 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.496 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:13:24.496 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.446872+0000", 2024-09-17T07:13:24.496 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:24.496 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.496 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.496 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:13:24.497 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.446871+0000", 2024-09-17T07:13:24.497 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.999999 2024-09-17T07:13:24.497 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.497 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.497 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:13:24.497 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.446901+0000", 2024-09-17T07:13:24.497 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.9570999999999999e-05 2024-09-17T07:13:24.497 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.497 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.497 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:13:24.497 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.446910+0000", 2024-09-17T07:13:24.497 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.0100000000000001e-06 2024-09-17T07:13:24.497 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.498 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.498 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:13:24.498 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.446916+0000", 2024-09-17T07:13:24.498 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.6030000000000001e-06 2024-09-17T07:13:24.498 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.498 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.498 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:24.498 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.446958+0000", 2024-09-17T07:13:24.498 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.2082000000000001e-05 2024-09-17T07:13:24.498 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.498 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.498 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:13:24.498 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.447090+0000", 2024-09-17T07:13:24.499 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000131144 2024-09-17T07:13:24.499 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.499 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.499 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 1,2", 2024-09-17T07:13:24.499 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.447374+0000", 2024-09-17T07:13:24.499 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00028408399999999999 2024-09-17T07:13:24.499 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:24.499 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:13:24.499 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:24.499 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.499 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.499 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_repop_reply(client.15330.0:108986 9.a e259/236)", 2024-09-17T07:13:24.499 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:13:24.447388+0000", 2024-09-17T07:13:24.500 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 4294967295.9992895, 2024-09-17T07:13:24.500 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.1548000000000001e-05, 2024-09-17T07:13:24.500 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:13:24.500 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "queued for pg", 2024-09-17T07:13:24.500 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:13:24.500 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.500 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:13:24.500 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.447388+0000", 2024-09-17T07:13:24.500 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:24.501 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.501 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.501 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:13:24.501 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.447388+0000", 2024-09-17T07:13:24.501 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:24.501 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.501 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.501 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:13:24.501 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.447388+0000", 2024-09-17T07:13:24.501 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999995 2024-09-17T07:13:24.501 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.501 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.501 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:13:24.502 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.447392+0000", 2024-09-17T07:13:24.502 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.0300000000000004e-06 2024-09-17T07:13:24.502 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.502 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.502 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:13:24.502 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.447404+0000", 2024-09-17T07:13:24.502 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.2404e-05 2024-09-17T07:13:24.502 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:24.502 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:24.502 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:13:24.502 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:13:24.447410+0000", 2024-09-17T07:13:24.502 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.4589999999999997e-06 2024-09-17T07:13:24.502 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:24.503 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:13:24.503 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:24.503 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:24.503 INFO:teuthology.orchestra.run.smithi135.stdout: ], 2024-09-17T07:13:24.503 INFO:teuthology.orchestra.run.smithi135.stdout: "num_ops": 11 2024-09-17T07:13:24.503 INFO:teuthology.orchestra.run.smithi135.stdout:} 2024-09-17T07:13:24.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:24 smithi135 ceph-mon[33238]: 2.6 scrub starts 2024-09-17T07:13:24.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:24 smithi135 ceph-mon[33238]: 2.6 scrub ok 2024-09-17T07:13:24.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:24 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/857986540' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-17T07:13:24.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:24 smithi135 ceph-mon[33238]: osdmap e259: 8 total, 8 up, 7 in 2024-09-17T07:13:24.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:24 smithi135 ceph-mon[33238]: 6.11 scrub starts 2024-09-17T07:13:24.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:24 smithi135 ceph-mon[33238]: 4.c scrub starts 2024-09-17T07:13:24.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:24 smithi135 ceph-mon[33238]: 6.11 scrub ok 2024-09-17T07:13:24.545 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:24 smithi135 ceph-mon[33238]: 4.c scrub ok 2024-09-17T07:13:24.545 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:24 smithi135 ceph-mon[33238]: 2.5 scrub starts 2024-09-17T07:13:24.545 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:24 smithi135 ceph-mon[33238]: 2.5 scrub ok 2024-09-17T07:13:24.547 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:13:24.580 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:24.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:24.582 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:13:24.682 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:13:24.716 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:24.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:24.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:13:24.818 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:13:24.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:24.871 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:24.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:13:24.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:24 smithi100 ceph-mon[29092]: 2.6 scrub starts 2024-09-17T07:13:24.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:24 smithi100 ceph-mon[29092]: 2.6 scrub ok 2024-09-17T07:13:24.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:24 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/857986540' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-17T07:13:24.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:24 smithi100 ceph-mon[29092]: osdmap e259: 8 total, 8 up, 7 in 2024-09-17T07:13:24.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:24 smithi100 ceph-mon[29092]: 6.11 scrub starts 2024-09-17T07:13:24.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:24 smithi100 ceph-mon[29092]: 4.c scrub starts 2024-09-17T07:13:24.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:24 smithi100 ceph-mon[29092]: 6.11 scrub ok 2024-09-17T07:13:24.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:24 smithi100 ceph-mon[29092]: 4.c scrub ok 2024-09-17T07:13:24.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:24 smithi100 ceph-mon[29092]: 2.5 scrub starts 2024-09-17T07:13:24.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:24 smithi100 ceph-mon[29092]: 2.5 scrub ok 2024-09-17T07:13:24.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:24 smithi100 ceph-mon[39462]: 2.6 scrub starts 2024-09-17T07:13:24.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:24 smithi100 ceph-mon[39462]: 2.6 scrub ok 2024-09-17T07:13:24.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:24 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/857986540' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-17T07:13:24.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:24 smithi100 ceph-mon[39462]: osdmap e259: 8 total, 8 up, 7 in 2024-09-17T07:13:24.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:24 smithi100 ceph-mon[39462]: 6.11 scrub starts 2024-09-17T07:13:24.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:24 smithi100 ceph-mon[39462]: 4.c scrub starts 2024-09-17T07:13:24.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:24 smithi100 ceph-mon[39462]: 6.11 scrub ok 2024-09-17T07:13:24.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:24 smithi100 ceph-mon[39462]: 4.c scrub ok 2024-09-17T07:13:24.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:24 smithi100 ceph-mon[39462]: 2.5 scrub starts 2024-09-17T07:13:24.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:24 smithi100 ceph-mon[39462]: 2.5 scrub ok 2024-09-17T07:13:24.972 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:13:25.006 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:25.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:25.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:13:25.108 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:13:25.145 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:25.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:25.146 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:13:25.221 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 68 16 109885 109869 100.969 71.9375 0.00761458 0.00989835 2024-09-17T07:13:25.247 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:13:25.277 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:25.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:25.278 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:13:25.280 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-17T07:13:25.379 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:13:25.427 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:25.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:25.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:13:25.528 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:13:25.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:25 smithi135 ceph-mon[33238]: 5.e scrub starts 2024-09-17T07:13:25.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:25 smithi135 ceph-mon[33238]: 5.e scrub ok 2024-09-17T07:13:25.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:25 smithi135 ceph-mon[33238]: 9.8 scrub starts 2024-09-17T07:13:25.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:25 smithi135 ceph-mon[33238]: pgmap v503: 148 pgs: 148 active+clean; 6.6 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.1 KiB/s rd, 79 MiB/s wr, 1.27k op/s 2024-09-17T07:13:25.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:25.557 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:25.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:13:25.640 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:13:25.658 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:13:25.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:25.708 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:25.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:13:25.734 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:25 smithi100 ceph-mon[39462]: 5.e scrub starts 2024-09-17T07:13:25.734 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:25 smithi100 ceph-mon[39462]: 5.e scrub ok 2024-09-17T07:13:25.734 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:25 smithi100 ceph-mon[39462]: 9.8 scrub starts 2024-09-17T07:13:25.734 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:25 smithi100 ceph-mon[39462]: pgmap v503: 148 pgs: 148 active+clean; 6.6 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.1 KiB/s rd, 79 MiB/s wr, 1.27k op/s 2024-09-17T07:13:25.735 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:25 smithi100 ceph-mon[29092]: 5.e scrub starts 2024-09-17T07:13:25.735 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:25 smithi100 ceph-mon[29092]: 5.e scrub ok 2024-09-17T07:13:25.735 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:25 smithi100 ceph-mon[29092]: 9.8 scrub starts 2024-09-17T07:13:25.735 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:25 smithi100 ceph-mon[29092]: pgmap v503: 148 pgs: 148 active+clean; 6.6 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.1 KiB/s rd, 79 MiB/s wr, 1.27k op/s 2024-09-17T07:13:25.810 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:13:25.855 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:25.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:25.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:13:25.957 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:13:25.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:25.989 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:25.990 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:13:26.043 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:13:25 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:13:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:13:26.090 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:13:26.176 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:26.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:26.177 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:13:26.221 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 69 16 110998 110982 100.514 69.5625 0.0166984 0.00994409 2024-09-17T07:13:26.277 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:13:26.316 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:26.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:26.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:13:26.418 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:13:26.443 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:26.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:26.444 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:13:26.545 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:13:26.574 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:26.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:26.575 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:13:26.676 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:13:26.709 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:26.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:26.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:13:26.776 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:26 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:13:26.776 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:26 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:13:26.776 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:26 smithi100 ceph-mon[29092]: osdmap e260: 8 total, 8 up, 7 in 2024-09-17T07:13:26.776 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:26 smithi100 ceph-mon[29092]: 5.3 scrub starts 2024-09-17T07:13:26.776 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:26 smithi100 ceph-mon[29092]: 5.3 scrub ok 2024-09-17T07:13:26.776 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:26 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:13:26.776 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:26 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:13:26.776 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:26 smithi100 ceph-mon[29092]: 6.d scrub starts 2024-09-17T07:13:26.776 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:26 smithi100 ceph-mon[29092]: 6.d scrub ok 2024-09-17T07:13:26.776 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:26 smithi100 ceph-mon[29092]: 2.b scrub starts 2024-09-17T07:13:26.776 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:26 smithi100 ceph-mon[29092]: 2.b scrub ok 2024-09-17T07:13:26.777 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:26 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:13:26.777 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:26 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:13:26.777 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:26 smithi100 ceph-mon[39462]: osdmap e260: 8 total, 8 up, 7 in 2024-09-17T07:13:26.777 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:26 smithi100 ceph-mon[39462]: 5.3 scrub starts 2024-09-17T07:13:26.777 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:26 smithi100 ceph-mon[39462]: 5.3 scrub ok 2024-09-17T07:13:26.777 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:26 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:13:26.777 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:26 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:13:26.777 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:26 smithi100 ceph-mon[39462]: 6.d scrub starts 2024-09-17T07:13:26.777 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:26 smithi100 ceph-mon[39462]: 6.d scrub ok 2024-09-17T07:13:26.777 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:26 smithi100 ceph-mon[39462]: 2.b scrub starts 2024-09-17T07:13:26.777 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:26 smithi100 ceph-mon[39462]: 2.b scrub ok 2024-09-17T07:13:26.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:26 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:13:26.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:26 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:13:26.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:26 smithi135 ceph-mon[33238]: osdmap e260: 8 total, 8 up, 7 in 2024-09-17T07:13:26.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:26 smithi135 ceph-mon[33238]: 5.3 scrub starts 2024-09-17T07:13:26.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:26 smithi135 ceph-mon[33238]: 5.3 scrub ok 2024-09-17T07:13:26.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:26 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:13:26.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:26 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:13:26.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:26 smithi135 ceph-mon[33238]: 6.d scrub starts 2024-09-17T07:13:26.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:26 smithi135 ceph-mon[33238]: 6.d scrub ok 2024-09-17T07:13:26.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:26 smithi135 ceph-mon[33238]: 2.b scrub starts 2024-09-17T07:13:26.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:26 smithi135 ceph-mon[33238]: 2.b scrub ok 2024-09-17T07:13:26.811 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:13:26.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:26.855 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:26.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:13:26.956 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:13:26.989 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:26.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:26.990 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:13:27.091 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:13:27.140 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:27.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:27.142 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:13:27.222 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 70 16 112060 112044 100.026 66.375 0.00420679 0.00999202 2024-09-17T07:13:27.242 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:13:27.282 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:27.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:27.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:13:27.384 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:13:27.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:27.406 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:27.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:13:27.507 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:13:27.545 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:27.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:27.546 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:13:27.647 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:13:27.680 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:27.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:27.681 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:13:27.738 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:27 smithi135 ceph-mon[33238]: osdmap e261: 8 total, 8 up, 7 in 2024-09-17T07:13:27.738 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:27 smithi135 ceph-mon[33238]: 9.c scrub starts 2024-09-17T07:13:27.738 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:27 smithi135 ceph-mon[33238]: 9.8 scrub ok 2024-09-17T07:13:27.738 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:27 smithi135 ceph-mon[33238]: pgmap v506: 148 pgs: 148 active+clean; 6.6 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 853 B/s rd, 60 MiB/s wr, 965 op/s 2024-09-17T07:13:27.782 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:13:27.795 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:27 smithi100 ceph-mon[29092]: osdmap e261: 8 total, 8 up, 7 in 2024-09-17T07:13:27.795 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:27 smithi100 ceph-mon[29092]: 9.c scrub starts 2024-09-17T07:13:27.795 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:27 smithi100 ceph-mon[29092]: 9.8 scrub ok 2024-09-17T07:13:27.795 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:27 smithi100 ceph-mon[29092]: pgmap v506: 148 pgs: 148 active+clean; 6.6 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 853 B/s rd, 60 MiB/s wr, 965 op/s 2024-09-17T07:13:27.796 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:27 smithi100 ceph-mon[39462]: osdmap e261: 8 total, 8 up, 7 in 2024-09-17T07:13:27.796 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:27 smithi100 ceph-mon[39462]: 9.c scrub starts 2024-09-17T07:13:27.796 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:27 smithi100 ceph-mon[39462]: 9.8 scrub ok 2024-09-17T07:13:27.796 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:27 smithi100 ceph-mon[39462]: pgmap v506: 148 pgs: 148 active+clean; 6.6 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 853 B/s rd, 60 MiB/s wr, 965 op/s 2024-09-17T07:13:27.820 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:27.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:27.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:13:27.922 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:13:27.967 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:27.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:27.968 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:13:28.068 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:13:28.112 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:13:28.113 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:13:28.113 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:28.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:28.115 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:13:28.139 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1: {} 2024-09-17T07:13:28.139 INFO:teuthology.orchestra.run.smithi100.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:13:28.165 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:13:28.165 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:13:28.187 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:13:28.187 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:13:28.212 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:13:28.212 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:13:28.215 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:13:28.221 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 71 16 113098 113082 99.531 64.875 0.0140198 0.0100422 2024-09-17T07:13:28.236 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:13:28.237 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:13:28.262 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:13:28.263 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:13:28.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:28.264 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:28.264 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:13:28.295 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:13:28.296 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:13:28.365 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:13:28.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:28.403 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:28.403 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:13:28.504 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:13:28.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:28.556 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:28.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:13:28.657 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:13:28.696 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:28.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:28.697 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:13:28.699 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:28 smithi135 ceph-mon[33238]: 2.16 scrub starts 2024-09-17T07:13:28.699 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:28 smithi135 ceph-mon[33238]: 2.16 scrub ok 2024-09-17T07:13:28.699 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:28 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2994019419' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:13:28.791 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:28 smithi100 ceph-mon[29092]: 2.16 scrub starts 2024-09-17T07:13:28.791 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:28 smithi100 ceph-mon[29092]: 2.16 scrub ok 2024-09-17T07:13:28.791 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:28 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2994019419' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:13:28.792 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:28 smithi100 ceph-mon[39462]: 2.16 scrub starts 2024-09-17T07:13:28.792 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:28 smithi100 ceph-mon[39462]: 2.16 scrub ok 2024-09-17T07:13:28.792 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:28 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2994019419' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:13:28.797 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:13:28.824 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:28.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:28.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:13:28.873 INFO:teuthology.orchestra.run.smithi135.stdout:ERROR: (22) Invalid argument 2024-09-17T07:13:28.873 INFO:teuthology.orchestra.run.smithi135.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:13:28.926 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:13:28.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:28.954 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:28.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:13:29.044 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:13:28 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:13:29.056 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:13:29.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:29.097 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:29.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:13:29.198 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:13:29.222 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 72 16 114186 114170 99.093 68 0.0154487 0.0100869 2024-09-17T07:13:29.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:29.230 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:29.231 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:13:29.331 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:13:29.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:29.369 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:29.369 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:13:29.470 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:13:29.481 INFO:teuthology.orchestra.run.smithi100.stderr:noscrub is set 2024-09-17T07:13:29.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:29.507 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:29.507 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:13:29.608 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:13:29.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:29.639 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:29.639 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:13:29.740 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:13:29.748 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:29 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:13:29.749 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:29 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1793139852' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:13:29.749 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:29 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:13:29.749 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:29 smithi135 ceph-mon[33238]: 9.c scrub ok 2024-09-17T07:13:29.749 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:29 smithi135 ceph-mon[33238]: 5.1b scrub starts 2024-09-17T07:13:29.749 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:29 smithi135 ceph-mon[33238]: 5.1b scrub ok 2024-09-17T07:13:29.749 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:29 smithi135 ceph-mon[33238]: 4.0 scrub starts 2024-09-17T07:13:29.749 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:29 smithi135 ceph-mon[33238]: 4.0 scrub ok 2024-09-17T07:13:29.749 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:29 smithi135 ceph-mon[33238]: 6.1f scrub starts 2024-09-17T07:13:29.749 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:29 smithi135 ceph-mon[33238]: 6.1f scrub ok 2024-09-17T07:13:29.749 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:29 smithi135 ceph-mon[33238]: pgmap v507: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.8 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.7 KiB/s rd, 104 MiB/s wr, 1.66k op/s 2024-09-17T07:13:29.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:29.781 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:29.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:13:29.875 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:29 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:13:29.875 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:29 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1793139852' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:13:29.875 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:29 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:13:29.875 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:29 smithi100 ceph-mon[29092]: 9.c scrub ok 2024-09-17T07:13:29.875 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:29 smithi100 ceph-mon[29092]: 5.1b scrub starts 2024-09-17T07:13:29.875 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:29 smithi100 ceph-mon[29092]: 5.1b scrub ok 2024-09-17T07:13:29.875 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:29 smithi100 ceph-mon[29092]: 4.0 scrub starts 2024-09-17T07:13:29.875 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:29 smithi100 ceph-mon[29092]: 4.0 scrub ok 2024-09-17T07:13:29.875 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:29 smithi100 ceph-mon[29092]: 6.1f scrub starts 2024-09-17T07:13:29.875 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:29 smithi100 ceph-mon[29092]: 6.1f scrub ok 2024-09-17T07:13:29.875 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:29 smithi100 ceph-mon[29092]: pgmap v507: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.8 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.7 KiB/s rd, 104 MiB/s wr, 1.66k op/s 2024-09-17T07:13:29.876 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:29 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:13:29.876 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:29 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1793139852' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:13:29.876 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:29 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:13:29.876 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:29 smithi100 ceph-mon[39462]: 9.c scrub ok 2024-09-17T07:13:29.876 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:29 smithi100 ceph-mon[39462]: 5.1b scrub starts 2024-09-17T07:13:29.876 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:29 smithi100 ceph-mon[39462]: 5.1b scrub ok 2024-09-17T07:13:29.876 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:29 smithi100 ceph-mon[39462]: 4.0 scrub starts 2024-09-17T07:13:29.876 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:29 smithi100 ceph-mon[39462]: 4.0 scrub ok 2024-09-17T07:13:29.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:29 smithi100 ceph-mon[39462]: 6.1f scrub starts 2024-09-17T07:13:29.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:29 smithi100 ceph-mon[39462]: 6.1f scrub ok 2024-09-17T07:13:29.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:29 smithi100 ceph-mon[39462]: pgmap v507: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.8 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.7 KiB/s rd, 104 MiB/s wr, 1.66k op/s 2024-09-17T07:13:29.882 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:13:29.915 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:29.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:29.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:13:30.017 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:13:30.052 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:30.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:30.053 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:13:30.102 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-17T07:13:30.153 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:13:30.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:30.182 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:30.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:13:30.222 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 73 16 115478 115462 98.8417 80.75 0.00780871 0.0101127 2024-09-17T07:13:30.283 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:13:30.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:30.331 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:30.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:13:30.432 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:13:30.472 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:13:30.481 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:30.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:30.482 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:13:30.583 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:13:30.613 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:30.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:30.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:13:30.714 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:13:30.742 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:30.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:30.743 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:13:30.767 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:30 smithi135 ceph-mon[33238]: 6.14 scrub starts 2024-09-17T07:13:30.767 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:30 smithi135 ceph-mon[33238]: 6.14 scrub ok 2024-09-17T07:13:30.768 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:30 smithi135 ceph-mon[33238]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:13:30.768 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:30 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T07:13:30.768 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:30 smithi135 ceph-mon[33238]: osdmap e262: 8 total, 8 up, 7 in 2024-09-17T07:13:30.844 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:13:30.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:30 smithi100 ceph-mon[39462]: 6.14 scrub starts 2024-09-17T07:13:30.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:30 smithi100 ceph-mon[39462]: 6.14 scrub ok 2024-09-17T07:13:30.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:30 smithi100 ceph-mon[39462]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:13:30.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:30 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T07:13:30.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:30 smithi100 ceph-mon[39462]: osdmap e262: 8 total, 8 up, 7 in 2024-09-17T07:13:30.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:30 smithi100 ceph-mon[29092]: 6.14 scrub starts 2024-09-17T07:13:30.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:30 smithi100 ceph-mon[29092]: 6.14 scrub ok 2024-09-17T07:13:30.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:30 smithi100 ceph-mon[29092]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:13:30.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:30 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T07:13:30.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:30 smithi100 ceph-mon[29092]: osdmap e262: 8 total, 8 up, 7 in 2024-09-17T07:13:30.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:30.888 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:30.889 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:13:30.989 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:13:31.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:31.020 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:31.021 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:13:31.072 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7] out_osds: [3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-17T07:13:31.072 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T07:13:31.072 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-09-17T07:13:31.073 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-17T07:13:31.073 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-17T07:13:31.073 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.4 config set filestore_inject_stall 3 2024-09-17T07:13:31.122 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:13:31.129 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:13:31.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:31.168 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:31.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:13:31.222 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 74 16 116689 116673 98.5287 75.6875 0.0114857 0.0101447 2024-09-17T07:13:31.268 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:13:31.325 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:31.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:31.326 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:13:31.417 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:13:31.427 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:13:31.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:31.464 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:31.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:13:31.565 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:13:31.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:31.601 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:31.602 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:13:31.704 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:13:31.733 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:31.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:31.734 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:13:31.737 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:31 smithi100 ceph-mon[29092]: pgmap v509: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.9 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 853 B/s rd, 58 MiB/s wr, 923 op/s 2024-09-17T07:13:31.737 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:31 smithi100 ceph-mon[39462]: pgmap v509: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.9 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 853 B/s rd, 58 MiB/s wr, 923 op/s 2024-09-17T07:13:31.777 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:31 smithi135 ceph-mon[33238]: pgmap v509: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.9 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 853 B/s rd, 58 MiB/s wr, 923 op/s 2024-09-17T07:13:31.836 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:13:31.883 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:31.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:31.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:13:31.985 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:13:32.013 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:32.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:32.014 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:13:32.115 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:13:32.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:32.142 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:32.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:13:32.222 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 75 16 117883 117867 98.2099 74.625 0.0142926 0.0101777 2024-09-17T07:13:32.243 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:13:32.259 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-17T07:13:32.268 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:32.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:32.269 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:13:32.331 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:13:32.370 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:13:32.385 INFO:teuthology.orchestra.run.smithi135.stdout:ERROR: (22) Invalid argument 2024-09-17T07:13:32.385 INFO:teuthology.orchestra.run.smithi135.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:13:32.416 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:32.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:32.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:13:32.518 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:13:32.555 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:32.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:32.556 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:13:32.657 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:13:32.684 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:32.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:32.685 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:13:32.786 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:13:32.821 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:32.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:32.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:13:32.904 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:13:32.923 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:13:32.956 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:32.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:32.958 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:13:33.058 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:13:33.096 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:33.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:33.097 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:13:33.198 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:13:33.222 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 76 16 119080 119064 97.902 74.8125 0.00860323 0.0102093 2024-09-17T07:13:33.233 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:33.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:33.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:13:33.335 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:13:33.369 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:33.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:33.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:13:33.470 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:13:33.506 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:33.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:33.507 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:13:33.607 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:13:33.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:33.643 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:33.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:13:33.744 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:13:33.753 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-17T07:13:33.781 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:33.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:33.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:13:33.883 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:13:33.918 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:33.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:33.919 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:13:34.020 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:13:34.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:33 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:13:34.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:33 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:13:34.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:33 smithi135 ceph-mon[33238]: pgmap v510: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.9 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 655 B/s rd, 44 MiB/s wr, 710 op/s 2024-09-17T07:13:34.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:34.052 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:34.052 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:13:34.153 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:13:34.179 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:33 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:13:34.180 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:33 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:13:34.180 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:33 smithi100 ceph-mon[29092]: pgmap v510: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.9 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 655 B/s rd, 44 MiB/s wr, 710 op/s 2024-09-17T07:13:34.180 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:33 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:13:34.180 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:33 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:13:34.180 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:33 smithi100 ceph-mon[39462]: pgmap v510: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.9 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 655 B/s rd, 44 MiB/s wr, 710 op/s 2024-09-17T07:13:34.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:34.189 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:34.190 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:13:34.222 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 77 16 120279 120263 97.6037 74.9375 0.0131085 0.0102408 2024-09-17T07:13:34.257 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:13:34.257 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:13:34.286 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1: {} 2024-09-17T07:13:34.287 INFO:teuthology.orchestra.run.smithi100.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:13:34.290 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:13:34.312 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:13:34.312 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:13:34.338 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:13:34.339 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:13:34.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:34.343 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:34.344 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:13:34.369 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:13:34.369 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:13:34.406 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:13:34.406 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:13:34.436 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:13:34.436 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:13:34.444 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:13:34.469 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:13:34.470 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:13:34.485 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:34.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:34.486 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:13:34.586 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:13:34.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:34.622 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:34.622 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:13:34.723 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:13:34.759 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:34.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:34.760 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:13:34.861 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:13:34.891 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:34.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:34.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:13:34.993 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:13:35.017 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:35.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:35.018 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:13:35.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:34 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/24393260' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:13:35.043 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:13:34 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=sqlstore.transactions t=2024-09-17T07:13:34.720296402Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-17T07:13:35.044 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:13:34 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=cleanup t=2024-09-17T07:13:34.722710915Z level=info msg="Completed cleanup jobs" duration=13.150127ms 2024-09-17T07:13:35.119 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:13:35.126 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:34 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/24393260' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:13:35.126 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:34 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/24393260' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:13:35.153 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:35.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:35.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:13:35.222 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 78 16 121434 121418 97.2778 72.1875 0.0121558 0.0102747 2024-09-17T07:13:35.255 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:13:35.281 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:35.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:35.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:13:35.383 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:13:35.423 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:35.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:35.424 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:13:35.525 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:13:35.560 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:35.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:35.561 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:13:35.662 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:13:35.697 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:35.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:35.698 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:13:35.799 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:13:35.829 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:35.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:35.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:13:35.930 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:13:35.960 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:35.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:35.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:13:36.062 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:13:36.094 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:36.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:36.095 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:13:36.143 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:35 smithi100 ceph-mon[29092]: pgmap v511: 148 pgs: 148 active+clean; 7.3 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.1 KiB/s rd, 83 MiB/s wr, 1.32k op/s 2024-09-17T07:13:36.144 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:13:35 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:13:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:13:36.144 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:35 smithi100 ceph-mon[39462]: pgmap v511: 148 pgs: 148 active+clean; 7.3 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.1 KiB/s rd, 83 MiB/s wr, 1.32k op/s 2024-09-17T07:13:36.196 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:13:36.213 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:35 smithi135 ceph-mon[33238]: pgmap v511: 148 pgs: 148 active+clean; 7.3 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.1 KiB/s rd, 83 MiB/s wr, 1.32k op/s 2024-09-17T07:13:36.222 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 79 16 122591 122575 96.9617 72.3125 0.0202514 0.0103086 2024-09-17T07:13:36.238 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:36.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:36.240 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:13:36.340 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:13:36.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:36.362 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:36.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:13:36.463 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:13:36.489 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:36.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:36.490 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:13:36.591 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:13:36.633 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:36.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:36.634 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:13:36.734 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:13:36.763 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:36.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:36.765 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:13:36.865 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:13:36.897 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:36.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:36.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:13:36.917 INFO:teuthology.orchestra.run.smithi100.stderr:nodeep-scrub is set 2024-09-17T07:13:36.999 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:13:37.028 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:37.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:37.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:13:37.130 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:13:37.172 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:37.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:37.173 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:13:37.222 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:2024-09-17T07:13:37.223011+0000 min lat: 0.00110426 max lat: 0.299974 avg lat: 0.0103589 2024-09-17T07:13:37.223 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-17T07:13:37.223 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 80 16 123552 123536 96.5004 60.0625 0.0103353 0.0103589 2024-09-17T07:13:37.274 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:13:37.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:36 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/3978625106' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:13:37.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:36 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:13:37.295 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:36 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/3978625106' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:13:37.295 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:36 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:13:37.296 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:36 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/3978625106' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:13:37.296 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:36 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:13:37.310 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:37.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:37.311 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:13:37.412 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:13:37.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:37.443 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:37.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:13:37.537 INFO:teuthology.orchestra.run.smithi135.stdout:{ 2024-09-17T07:13:37.537 INFO:teuthology.orchestra.run.smithi135.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-17T07:13:37.538 INFO:teuthology.orchestra.run.smithi135.stdout:} 2024-09-17T07:13:37.544 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:13:37.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:37.580 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:37.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:13:37.614 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:13:37.682 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:13:37.719 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:37.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:37.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:13:37.821 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:13:37.856 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:37.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:37.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:13:37.896 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:13:37.958 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:13:37.979 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:37.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:37.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:13:38.081 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:13:38.118 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:38.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:38.119 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:13:38.200 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:37 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:13:38.200 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:37 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:13:38.201 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:37 smithi100 ceph-mon[39462]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:13:38.201 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:37 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T07:13:38.201 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:37 smithi100 ceph-mon[39462]: osdmap e263: 8 total, 8 up, 7 in 2024-09-17T07:13:38.201 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:37 smithi100 ceph-mon[39462]: pgmap v513: 148 pgs: 148 active+clean; 7.3 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 639 B/s rd, 58 MiB/s wr, 929 op/s 2024-09-17T07:13:38.201 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:37 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:13:38.201 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:37 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:13:38.201 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:37 smithi100 ceph-mon[29092]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:13:38.201 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:37 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T07:13:38.201 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:37 smithi100 ceph-mon[29092]: osdmap e263: 8 total, 8 up, 7 in 2024-09-17T07:13:38.202 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:37 smithi100 ceph-mon[29092]: pgmap v513: 148 pgs: 148 active+clean; 7.3 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 639 B/s rd, 58 MiB/s wr, 929 op/s 2024-09-17T07:13:38.220 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:13:38.223 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 81 16 124629 124613 96.14 67.3125 0.00448592 0.010396 2024-09-17T07:13:38.242 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:38.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:38.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:13:38.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:37 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:13:38.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:37 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:13:38.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:37 smithi135 ceph-mon[33238]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:13:38.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:37 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T07:13:38.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:37 smithi135 ceph-mon[33238]: osdmap e263: 8 total, 8 up, 7 in 2024-09-17T07:13:38.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:37 smithi135 ceph-mon[33238]: pgmap v513: 148 pgs: 148 active+clean; 7.3 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 639 B/s rd, 58 MiB/s wr, 929 op/s 2024-09-17T07:13:38.343 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:13:38.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:38.383 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:38.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:13:38.484 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:13:38.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:38.526 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:38.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:13:38.627 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:13:38.656 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:38.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:38.657 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:13:38.758 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:13:38.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:38.797 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:38.798 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:13:38.898 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:13:38.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:38.930 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:38.930 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:13:39.031 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:13:39.043 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:13:38 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:13:39.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:38 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:13:39.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:38 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:13:39.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:38 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:13:39.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:38 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:13:39.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:38 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:13:39.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:38 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:13:39.058 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:39.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:39.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:13:39.160 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:13:39.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:39.202 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:39.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:13:39.206 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:38 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:13:39.206 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:38 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:13:39.206 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:38 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:13:39.206 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:38 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:13:39.206 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:38 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:13:39.206 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:38 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:13:39.207 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:38 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:13:39.207 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:38 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:13:39.207 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:38 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:13:39.207 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:38 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:13:39.207 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:38 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:13:39.207 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:38 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:13:39.224 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 82 16 125622 125606 95.7233 62.0625 0.0115063 0.0104424 2024-09-17T07:13:39.299 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:13:39.303 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:13:39.358 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:39.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:39.359 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:13:39.460 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:13:39.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:39.490 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:39.490 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:13:39.591 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:13:39.630 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:39.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:39.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:13:39.732 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:13:39.772 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:39.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:39.774 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:13:39.789 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T07:13:39.826 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:13:39.826 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:13:39.858 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1: {} 2024-09-17T07:13:39.858 INFO:teuthology.orchestra.run.smithi100.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:13:39.874 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:13:39.884 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:13:39.884 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:13:39.907 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:39.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:39.908 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:13:39.913 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:13:39.913 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:13:39.948 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:13:39.949 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:13:39.991 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:13:39.991 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:13:40.010 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:13:40.022 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:13:40.022 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:13:40.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:39 smithi135 ceph-mon[33238]: Adjusting osd_memory_target on smithi100 to 3247M 2024-09-17T07:13:40.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:39 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:13:40.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:39 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:13:40.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:39 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:13:40.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:39 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:13:40.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:39 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:13:40.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:39 smithi135 ceph-mon[33238]: Adjusting osd_memory_target on smithi135 to 429.8M 2024-09-17T07:13:40.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:39 smithi135 ceph-mon[33238]: Unable to set osd_memory_target on smithi135 to 450693836: error parsing value: Value '450693836' is below minimum 939524096 2024-09-17T07:13:40.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:39 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:13:40.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:39 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:13:40.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:39 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:13:40.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:39 smithi135 ceph-mon[33238]: pgmap v514: 148 pgs: 148 active+clean; 7.5 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1.0 KiB/s rd, 65 MiB/s wr, 1.03k op/s 2024-09-17T07:13:40.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:39 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1198387032' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:13:40.051 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:40.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:40.052 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:13:40.056 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:13:40.056 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:13:40.057 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:39 smithi100 ceph-mon[29092]: Adjusting osd_memory_target on smithi100 to 3247M 2024-09-17T07:13:40.057 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:39 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:13:40.057 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:39 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:13:40.057 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:39 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:13:40.057 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:39 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:13:40.057 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:39 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:13:40.057 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:39 smithi100 ceph-mon[29092]: Adjusting osd_memory_target on smithi135 to 429.8M 2024-09-17T07:13:40.057 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:39 smithi100 ceph-mon[29092]: Unable to set osd_memory_target on smithi135 to 450693836: error parsing value: Value '450693836' is below minimum 939524096 2024-09-17T07:13:40.057 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:39 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:13:40.057 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:39 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:13:40.057 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:39 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:13:40.058 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:39 smithi100 ceph-mon[29092]: pgmap v514: 148 pgs: 148 active+clean; 7.5 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1.0 KiB/s rd, 65 MiB/s wr, 1.03k op/s 2024-09-17T07:13:40.058 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:39 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1198387032' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:13:40.058 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:39 smithi100 ceph-mon[39462]: Adjusting osd_memory_target on smithi100 to 3247M 2024-09-17T07:13:40.058 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:39 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:13:40.058 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:39 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:13:40.058 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:39 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:13:40.058 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:39 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:13:40.059 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:39 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:13:40.059 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:39 smithi100 ceph-mon[39462]: Adjusting osd_memory_target on smithi135 to 429.8M 2024-09-17T07:13:40.059 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:39 smithi100 ceph-mon[39462]: Unable to set osd_memory_target on smithi135 to 450693836: error parsing value: Value '450693836' is below minimum 939524096 2024-09-17T07:13:40.059 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:39 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:13:40.059 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:39 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:13:40.059 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:39 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:13:40.059 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:39 smithi100 ceph-mon[39462]: pgmap v514: 148 pgs: 148 active+clean; 7.5 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1.0 KiB/s rd, 65 MiB/s wr, 1.03k op/s 2024-09-17T07:13:40.059 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:39 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1198387032' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:13:40.153 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:13:40.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:40.191 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:40.191 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:13:40.224 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 83 16 126788 126772 95.4479 72.875 0.00854908 0.0104726 2024-09-17T07:13:40.292 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:13:40.330 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:40.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:40.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:13:40.344 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:13:40.431 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:13:40.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:40.470 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:40.470 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:13:40.571 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:13:40.596 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:40.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:40.597 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:13:40.698 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:13:40.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:40.728 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:40.728 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:13:40.829 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:13:40.855 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:40.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:40.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:13:40.958 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:13:41.002 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:41.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:41.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:13:41.104 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:13:41.130 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:41.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:41.131 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:13:41.224 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 84 16 127963 127947 95.1858 73.4375 0.0103718 0.0105011 2024-09-17T07:13:41.232 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:13:41.260 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:41.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:41.261 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:13:41.362 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:13:41.401 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:41.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:41.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:13:41.502 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:13:41.542 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:41.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:41.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:13:41.644 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:13:41.679 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:41.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:41.681 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:13:41.781 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:13:41.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:41.813 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:41.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:13:41.914 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:13:41.949 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:41.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:41.950 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:13:42.051 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:13:42.072 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:42.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:42.074 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:13:42.174 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:13:42.209 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:42.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:42.210 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:13:42.224 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 85 16 129036 129020 94.8549 67.0625 0.00598718 0.0105377 2024-09-17T07:13:42.310 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:13:42.351 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:42.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:42.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:13:42.355 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:42 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:13:42.355 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:42 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:13:42.355 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:42 smithi100 ceph-mon[29092]: osdmap e264: 8 total, 8 up, 7 in 2024-09-17T07:13:42.355 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:42 smithi100 ceph-mon[29092]: pgmap v516: 148 pgs: 148 active+clean; 7.6 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1.2 KiB/s rd, 89 MiB/s wr, 1.43k op/s 2024-09-17T07:13:42.355 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:42 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:13:42.356 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:42 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:13:42.356 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:42 smithi100 ceph-mon[39462]: osdmap e264: 8 total, 8 up, 7 in 2024-09-17T07:13:42.356 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:42 smithi100 ceph-mon[39462]: pgmap v516: 148 pgs: 148 active+clean; 7.6 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1.2 KiB/s rd, 89 MiB/s wr, 1.43k op/s 2024-09-17T07:13:42.452 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:13:42.491 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:42.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:42.492 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:13:42.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:42 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:13:42.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:42 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:13:42.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:42 smithi135 ceph-mon[33238]: osdmap e264: 8 total, 8 up, 7 in 2024-09-17T07:13:42.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:42 smithi135 ceph-mon[33238]: pgmap v516: 148 pgs: 148 active+clean; 7.6 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1.2 KiB/s rd, 89 MiB/s wr, 1.43k op/s 2024-09-17T07:13:42.593 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:13:42.620 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:42.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:42.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:13:42.722 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:13:42.749 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:42.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:42.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:13:42.767 INFO:teuthology.orchestra.run.smithi135.stdout:ERROR: (22) Invalid argument 2024-09-17T07:13:42.767 INFO:teuthology.orchestra.run.smithi135.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:13:42.850 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:13:42.889 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:42.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:42.890 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:13:42.981 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:13:42.992 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:13:43.056 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:43.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:43.057 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:13:43.158 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:13:43.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:43.200 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:43.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:13:43.225 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 86 16 130191 130175 94.5908 72.1875 0.00686228 0.0105675 2024-09-17T07:13:43.297 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:13:43.301 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:13:43.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:43.324 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:43.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:13:43.425 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:13:43.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:43.453 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:43.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:13:43.555 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:13:43.591 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:43.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:43.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:13:43.693 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:13:43.730 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:43.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:43.731 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:13:43.832 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:13:43.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:43.869 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:43.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:13:43.970 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:13:43.981 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-17T07:13:43.998 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:43.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:43.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:13:44.100 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:13:44.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:44.122 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:44.122 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:13:44.223 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:13:44.224 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 87 16 131138 131122 94.1839 59.1875 0.00455945 0.0106129 2024-09-17T07:13:44.249 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:44.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:44.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:13:44.315 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:13:44.316 INFO:teuthology.orchestra.run.smithi100.stderr:noscrub is unset 2024-09-17T07:13:44.349 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7] out_osds: [3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-17T07:13:44.349 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T07:13:44.350 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-09-17T07:13:44.350 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-17T07:13:44.350 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-17T07:13:44.350 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.6 config set filestore_inject_stall 3 2024-09-17T07:13:44.351 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:13:44.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:44.380 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:44.380 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:13:44.481 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:13:44.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:44.543 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:44.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:13:44.625 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:44 smithi135 ceph-mon[33238]: pgmap v517: 148 pgs: 148 active+clean; 7.6 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 639 B/s rd, 42 MiB/s wr, 669 op/s 2024-09-17T07:13:44.625 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:44 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/461254036' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:13:44.626 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:44 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:13:44.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:44 smithi100 ceph-mon[29092]: pgmap v517: 148 pgs: 148 active+clean; 7.6 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 639 B/s rd, 42 MiB/s wr, 669 op/s 2024-09-17T07:13:44.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:44 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/461254036' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:13:44.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:44 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:13:44.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:44 smithi100 ceph-mon[39462]: pgmap v517: 148 pgs: 148 active+clean; 7.6 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 639 B/s rd, 42 MiB/s wr, 669 op/s 2024-09-17T07:13:44.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:44 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/461254036' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:13:44.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:44 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:13:44.644 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:13:44.705 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:44.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:44.706 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:13:44.806 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:13:44.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:44.835 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:44.835 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:13:44.936 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:13:44.958 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:44.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:44.959 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:13:45.060 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:13:45.094 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:45.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:45.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:13:45.197 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:13:45.224 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 88 16 132085 132069 93.7861 59.1875 0.0265698 0.0106579 2024-09-17T07:13:45.231 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:45.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:45.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:13:45.333 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:13:45.368 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:45.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:45.369 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:13:45.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:45 smithi100 ceph-mon[29092]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:13:45.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:45 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T07:13:45.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:45 smithi100 ceph-mon[29092]: osdmap e265: 8 total, 8 up, 7 in 2024-09-17T07:13:45.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:45 smithi100 ceph-mon[29092]: 2.19 scrub starts 2024-09-17T07:13:45.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:45 smithi100 ceph-mon[29092]: 2.19 scrub ok 2024-09-17T07:13:45.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:45 smithi100 ceph-mon[29092]: 9.9 scrub starts 2024-09-17T07:13:45.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:45 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:13:45.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:45 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:13:45.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:45 smithi100 ceph-mon[29092]: 4.11 scrub starts 2024-09-17T07:13:45.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:45 smithi100 ceph-mon[29092]: 4.11 scrub ok 2024-09-17T07:13:45.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:45 smithi100 ceph-mon[39462]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:13:45.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:45 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T07:13:45.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:45 smithi100 ceph-mon[39462]: osdmap e265: 8 total, 8 up, 7 in 2024-09-17T07:13:45.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:45 smithi100 ceph-mon[39462]: 2.19 scrub starts 2024-09-17T07:13:45.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:45 smithi100 ceph-mon[39462]: 2.19 scrub ok 2024-09-17T07:13:45.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:45 smithi100 ceph-mon[39462]: 9.9 scrub starts 2024-09-17T07:13:45.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:45 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:13:45.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:45 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:13:45.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:45 smithi100 ceph-mon[39462]: 4.11 scrub starts 2024-09-17T07:13:45.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:45 smithi100 ceph-mon[39462]: 4.11 scrub ok 2024-09-17T07:13:45.471 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:13:45.507 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:45.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:45.509 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:13:45.609 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:13:45.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:45.649 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:45.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:13:45.750 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:13:45.783 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:45.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:45.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:13:45.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:45 smithi135 ceph-mon[33238]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:13:45.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:45 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T07:13:45.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:45 smithi135 ceph-mon[33238]: osdmap e265: 8 total, 8 up, 7 in 2024-09-17T07:13:45.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:45 smithi135 ceph-mon[33238]: 2.19 scrub starts 2024-09-17T07:13:45.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:45 smithi135 ceph-mon[33238]: 2.19 scrub ok 2024-09-17T07:13:45.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:45 smithi135 ceph-mon[33238]: 9.9 scrub starts 2024-09-17T07:13:45.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:45 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:13:45.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:45 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:13:45.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:45 smithi135 ceph-mon[33238]: 4.11 scrub starts 2024-09-17T07:13:45.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:45 smithi135 ceph-mon[33238]: 4.11 scrub ok 2024-09-17T07:13:45.885 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:13:45.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:45.920 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:45.920 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:13:46.021 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:13:46.042 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:46.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:46.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:13:46.126 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:13:45 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:13:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:13:46.144 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:13:46.181 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:46.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:46.182 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:13:46.225 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 89 16 133205 133189 93.5189 70 0.0111463 0.0106884 2024-09-17T07:13:46.229 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:13:46.283 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:13:46.313 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:46.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:46.314 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:13:46.415 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:13:46.449 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:46.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:46.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:13:46.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:46 smithi135 ceph-mon[33238]: pgmap v519: 148 pgs: 148 active+clean; 8.0 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 1.2 KiB/s rd, 85 MiB/s wr, 1.37k op/s 2024-09-17T07:13:46.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:46 smithi135 ceph-mon[33238]: osdmap e266: 8 total, 8 up, 7 in 2024-09-17T07:13:46.551 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:13:46.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:46.584 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:46.584 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:13:46.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:46 smithi100 ceph-mon[39462]: pgmap v519: 148 pgs: 148 active+clean; 8.0 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 1.2 KiB/s rd, 85 MiB/s wr, 1.37k op/s 2024-09-17T07:13:46.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:46 smithi100 ceph-mon[39462]: osdmap e266: 8 total, 8 up, 7 in 2024-09-17T07:13:46.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:46 smithi100 ceph-mon[29092]: pgmap v519: 148 pgs: 148 active+clean; 8.0 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 1.2 KiB/s rd, 85 MiB/s wr, 1.37k op/s 2024-09-17T07:13:46.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:46 smithi100 ceph-mon[29092]: osdmap e266: 8 total, 8 up, 7 in 2024-09-17T07:13:46.685 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:13:46.697 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:13:46.697 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:13:46.726 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:46.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:46.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:13:46.731 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1: {} 2024-09-17T07:13:46.731 INFO:teuthology.orchestra.run.smithi100.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:13:46.760 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:13:46.760 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:13:46.790 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:13:46.791 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:13:46.824 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:13:46.824 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:13:46.828 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:13:46.854 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:13:46.854 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:13:46.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:46.867 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:46.867 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:13:46.878 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:13:46.879 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:13:46.909 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:13:46.909 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:13:46.968 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:13:47.002 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:47.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:47.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:13:47.104 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:13:47.116 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-17T07:13:47.141 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:47.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:47.142 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:13:47.225 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Total time run: 90.0077 2024-09-17T07:13:47.225 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Total writes made: 134263 2024-09-17T07:13:47.225 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Write size: 65536 2024-09-17T07:13:47.225 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Object size: 65536 2024-09-17T07:13:47.226 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Bandwidth (MB/sec): 93.2303 2024-09-17T07:13:47.226 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Stddev Bandwidth: 36.5933 2024-09-17T07:13:47.226 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Max bandwidth (MB/sec): 247.625 2024-09-17T07:13:47.226 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Min bandwidth (MB/sec): 53.25 2024-09-17T07:13:47.226 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Average IOPS: 1491 2024-09-17T07:13:47.226 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Stddev IOPS: 585.492 2024-09-17T07:13:47.227 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Max IOPS: 3962 2024-09-17T07:13:47.227 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Min IOPS: 852 2024-09-17T07:13:47.227 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Average Latency(s): 0.0107224 2024-09-17T07:13:47.227 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Stddev Latency(s): 0.00909802 2024-09-17T07:13:47.227 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Max latency(s): 0.299974 2024-09-17T07:13:47.227 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Min latency(s): 0.00110426 2024-09-17T07:13:47.231 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Cleaning up (deleting benchmark objects) 2024-09-17T07:13:47.243 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:13:47.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:47.283 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:47.283 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:13:47.327 INFO:teuthology.orchestra.run.smithi135.stdout:{ 2024-09-17T07:13:47.327 INFO:teuthology.orchestra.run.smithi135.stdout: "ops": [], 2024-09-17T07:13:47.327 INFO:teuthology.orchestra.run.smithi135.stdout: "complaint_time": 30, 2024-09-17T07:13:47.327 INFO:teuthology.orchestra.run.smithi135.stdout: "num_blocked_ops": 0 2024-09-17T07:13:47.328 INFO:teuthology.orchestra.run.smithi135.stdout:} 2024-09-17T07:13:47.384 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:13:47.419 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:47.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:47.420 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:13:47.473 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:47 smithi135 ceph-mon[33238]: 5.18 scrub starts 2024-09-17T07:13:47.473 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:47 smithi135 ceph-mon[33238]: 5.18 scrub ok 2024-09-17T07:13:47.473 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:47 smithi135 ceph-mon[33238]: 9.5 scrub starts 2024-09-17T07:13:47.473 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:47 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2523522916' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:13:47.474 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:47 smithi135 ceph-mon[33238]: 5.10 scrub starts 2024-09-17T07:13:47.474 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:47 smithi135 ceph-mon[33238]: 5.10 scrub ok 2024-09-17T07:13:47.521 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:13:47.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:47.553 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:47.553 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:13:47.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:47 smithi100 ceph-mon[29092]: 5.18 scrub starts 2024-09-17T07:13:47.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:47 smithi100 ceph-mon[29092]: 5.18 scrub ok 2024-09-17T07:13:47.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:47 smithi100 ceph-mon[29092]: 9.5 scrub starts 2024-09-17T07:13:47.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:47 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2523522916' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:13:47.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:47 smithi100 ceph-mon[29092]: 5.10 scrub starts 2024-09-17T07:13:47.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:47 smithi100 ceph-mon[29092]: 5.10 scrub ok 2024-09-17T07:13:47.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:47 smithi100 ceph-mon[39462]: 5.18 scrub starts 2024-09-17T07:13:47.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:47 smithi100 ceph-mon[39462]: 5.18 scrub ok 2024-09-17T07:13:47.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:47 smithi100 ceph-mon[39462]: 9.5 scrub starts 2024-09-17T07:13:47.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:47 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2523522916' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:13:47.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:47 smithi100 ceph-mon[39462]: 5.10 scrub starts 2024-09-17T07:13:47.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:47 smithi100 ceph-mon[39462]: 5.10 scrub ok 2024-09-17T07:13:47.654 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:13:47.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:47.682 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:47.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:13:47.783 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:13:47.820 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:47.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:47.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:13:47.922 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:13:47.947 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:47.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:47.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:13:48.049 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:13:48.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:48.099 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:48.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:13:48.111 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:13:48.201 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:13:48.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:48.244 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:48.244 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:13:48.345 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:13:48.388 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:48.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:48.389 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:13:48.490 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:13:48.506 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:48 smithi100 ceph-mon[29092]: 5.c scrub starts 2024-09-17T07:13:48.506 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:48 smithi100 ceph-mon[29092]: 5.c scrub ok 2024-09-17T07:13:48.506 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:48 smithi100 ceph-mon[29092]: 9.9 scrub ok 2024-09-17T07:13:48.506 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:48 smithi100 ceph-mon[29092]: 5.b scrub starts 2024-09-17T07:13:48.506 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:48 smithi100 ceph-mon[29092]: 5.b scrub ok 2024-09-17T07:13:48.506 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:48 smithi100 ceph-mon[29092]: pgmap v521: 148 pgs: 148 active+clean; 8.0 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 845 B/s rd, 57 MiB/s wr, 920 op/s 2024-09-17T07:13:48.506 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:48 smithi100 ceph-mon[29092]: 9.1 scrub starts 2024-09-17T07:13:48.506 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:48 smithi100 ceph-mon[29092]: 4.7 scrub starts 2024-09-17T07:13:48.506 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:48 smithi100 ceph-mon[29092]: 4.7 scrub ok 2024-09-17T07:13:48.506 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:48 smithi100 ceph-mon[29092]: 4.18 scrub starts 2024-09-17T07:13:48.507 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:48 smithi100 ceph-mon[29092]: 4.18 scrub ok 2024-09-17T07:13:48.507 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:48 smithi100 ceph-mon[29092]: 9.7 scrub starts 2024-09-17T07:13:48.507 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:48 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:13:48.507 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:48 smithi100 ceph-mon[39462]: 5.c scrub starts 2024-09-17T07:13:48.507 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:48 smithi100 ceph-mon[39462]: 5.c scrub ok 2024-09-17T07:13:48.507 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:48 smithi100 ceph-mon[39462]: 9.9 scrub ok 2024-09-17T07:13:48.507 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:48 smithi100 ceph-mon[39462]: 5.b scrub starts 2024-09-17T07:13:48.507 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:48 smithi100 ceph-mon[39462]: 5.b scrub ok 2024-09-17T07:13:48.507 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:48 smithi100 ceph-mon[39462]: pgmap v521: 148 pgs: 148 active+clean; 8.0 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 845 B/s rd, 57 MiB/s wr, 920 op/s 2024-09-17T07:13:48.507 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:48 smithi100 ceph-mon[39462]: 9.1 scrub starts 2024-09-17T07:13:48.508 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:48 smithi100 ceph-mon[39462]: 4.7 scrub starts 2024-09-17T07:13:48.508 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:48 smithi100 ceph-mon[39462]: 4.7 scrub ok 2024-09-17T07:13:48.508 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:48 smithi100 ceph-mon[39462]: 4.18 scrub starts 2024-09-17T07:13:48.508 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:48 smithi100 ceph-mon[39462]: 4.18 scrub ok 2024-09-17T07:13:48.508 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:48 smithi100 ceph-mon[39462]: 9.7 scrub starts 2024-09-17T07:13:48.508 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:48 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:13:48.523 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:48 smithi135 ceph-mon[33238]: 5.c scrub starts 2024-09-17T07:13:48.524 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:48 smithi135 ceph-mon[33238]: 5.c scrub ok 2024-09-17T07:13:48.524 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:48 smithi135 ceph-mon[33238]: 9.9 scrub ok 2024-09-17T07:13:48.524 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:48 smithi135 ceph-mon[33238]: 5.b scrub starts 2024-09-17T07:13:48.524 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:48 smithi135 ceph-mon[33238]: 5.b scrub ok 2024-09-17T07:13:48.524 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:48 smithi135 ceph-mon[33238]: pgmap v521: 148 pgs: 148 active+clean; 8.0 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 845 B/s rd, 57 MiB/s wr, 920 op/s 2024-09-17T07:13:48.524 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:48 smithi135 ceph-mon[33238]: 9.1 scrub starts 2024-09-17T07:13:48.524 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:48 smithi135 ceph-mon[33238]: 4.7 scrub starts 2024-09-17T07:13:48.525 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:48 smithi135 ceph-mon[33238]: 4.7 scrub ok 2024-09-17T07:13:48.525 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:48 smithi135 ceph-mon[33238]: 4.18 scrub starts 2024-09-17T07:13:48.525 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:48 smithi135 ceph-mon[33238]: 4.18 scrub ok 2024-09-17T07:13:48.525 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:48 smithi135 ceph-mon[33238]: 9.7 scrub starts 2024-09-17T07:13:48.525 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:48 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:13:48.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:48.526 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:48.526 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:13:48.627 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:13:48.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:48.666 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:48.666 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:13:48.767 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:13:48.793 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:13:48 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:13:48.802 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:48.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:48.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:13:48.904 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:13:48.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:48.934 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:48.934 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:13:49.035 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:13:49.078 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:49.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:49.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:13:49.180 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:13:49.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:49.219 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:49.219 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:13:49.320 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:13:49.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:49.357 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:49.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:13:49.458 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:13:49.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:49.500 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:49.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:13:49.596 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:49 smithi100 ceph-mon[29092]: 5.6 scrub starts 2024-09-17T07:13:49.596 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:49 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:13:49.597 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:49 smithi100 ceph-mon[29092]: 9.6 scrub starts 2024-09-17T07:13:49.597 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:49 smithi100 ceph-mon[29092]: pgmap v522: 148 pgs: 3 active+clean+scrubbing, 145 active+clean; 8.1 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 1.4 KiB/s rd, 66 MiB/s wr, 1.19k op/s 2024-09-17T07:13:49.597 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:49 smithi100 ceph-mon[39462]: 5.6 scrub starts 2024-09-17T07:13:49.597 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:49 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:13:49.598 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:49 smithi100 ceph-mon[39462]: 9.6 scrub starts 2024-09-17T07:13:49.598 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:49 smithi100 ceph-mon[39462]: pgmap v522: 148 pgs: 3 active+clean+scrubbing, 145 active+clean; 8.1 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 1.4 KiB/s rd, 66 MiB/s wr, 1.19k op/s 2024-09-17T07:13:49.600 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:13:49.606 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:49 smithi135 ceph-mon[33238]: 5.6 scrub starts 2024-09-17T07:13:49.606 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:49 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:13:49.606 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:49 smithi135 ceph-mon[33238]: 9.6 scrub starts 2024-09-17T07:13:49.606 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:49 smithi135 ceph-mon[33238]: pgmap v522: 148 pgs: 3 active+clean+scrubbing, 145 active+clean; 8.1 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 1.4 KiB/s rd, 66 MiB/s wr, 1.19k op/s 2024-09-17T07:13:49.607 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-17T07:13:49.643 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:49.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:49.645 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:13:49.745 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:13:49.769 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:49.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:49.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:13:49.871 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:13:49.891 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:49.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:49.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:13:49.993 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:13:49.998 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:13:50.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:50.030 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:50.030 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:13:50.131 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:13:50.163 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:50.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:50.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:13:50.265 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:13:50.289 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:50.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:50.291 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:13:50.391 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:13:50.421 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:50.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:50.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:13:50.523 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:13:50.543 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:50.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:50.544 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:13:50.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:50 smithi100 ceph-mon[29092]: 4.12 scrub starts 2024-09-17T07:13:50.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:50 smithi100 ceph-mon[29092]: 4.12 scrub ok 2024-09-17T07:13:50.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:50 smithi100 ceph-mon[29092]: 4.a scrub starts 2024-09-17T07:13:50.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:50 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:13:50.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:50 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:13:50.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:50 smithi100 ceph-mon[29092]: 9.5 scrub ok 2024-09-17T07:13:50.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:50 smithi100 ceph-mon[29092]: osdmap e267: 8 total, 8 up, 7 in 2024-09-17T07:13:50.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:50 smithi100 ceph-mon[29092]: 4.1 scrub starts 2024-09-17T07:13:50.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:50 smithi100 ceph-mon[29092]: 4.1 scrub ok 2024-09-17T07:13:50.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:50 smithi100 ceph-mon[39462]: 4.12 scrub starts 2024-09-17T07:13:50.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:50 smithi100 ceph-mon[39462]: 4.12 scrub ok 2024-09-17T07:13:50.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:50 smithi100 ceph-mon[39462]: 4.a scrub starts 2024-09-17T07:13:50.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:50 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:13:50.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:50 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:13:50.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:50 smithi100 ceph-mon[39462]: 9.5 scrub ok 2024-09-17T07:13:50.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:50 smithi100 ceph-mon[39462]: osdmap e267: 8 total, 8 up, 7 in 2024-09-17T07:13:50.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:50 smithi100 ceph-mon[39462]: 4.1 scrub starts 2024-09-17T07:13:50.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:50 smithi100 ceph-mon[39462]: 4.1 scrub ok 2024-09-17T07:13:50.645 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:13:50.673 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:50.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:50.674 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:13:50.775 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:13:50.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:50 smithi135 ceph-mon[33238]: 4.12 scrub starts 2024-09-17T07:13:50.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:50 smithi135 ceph-mon[33238]: 4.12 scrub ok 2024-09-17T07:13:50.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:50 smithi135 ceph-mon[33238]: 4.a scrub starts 2024-09-17T07:13:50.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:50 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:13:50.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:50 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:13:50.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:50 smithi135 ceph-mon[33238]: 9.5 scrub ok 2024-09-17T07:13:50.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:50 smithi135 ceph-mon[33238]: osdmap e267: 8 total, 8 up, 7 in 2024-09-17T07:13:50.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:50 smithi135 ceph-mon[33238]: 4.1 scrub starts 2024-09-17T07:13:50.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:50 smithi135 ceph-mon[33238]: 4.1 scrub ok 2024-09-17T07:13:50.814 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:50.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:50.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:13:50.915 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:13:50.964 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:50.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:50.965 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:13:51.066 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:13:51.094 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:51.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:51.095 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:13:51.196 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:13:51.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:51.220 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:51.220 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:13:51.321 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:13:51.342 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:51.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:51.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:13:51.444 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:13:51.468 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:51.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:51.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:13:51.571 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:13:51.601 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:51.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:51.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:13:51.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:51 smithi100 ceph-mon[29092]: 2.11 scrub starts 2024-09-17T07:13:51.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:51 smithi100 ceph-mon[29092]: 2.11 scrub ok 2024-09-17T07:13:51.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:51 smithi100 ceph-mon[29092]: 2.1 scrub starts 2024-09-17T07:13:51.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:51 smithi100 ceph-mon[29092]: 2.1 scrub ok 2024-09-17T07:13:51.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:51 smithi100 ceph-mon[29092]: 2.3 scrub starts 2024-09-17T07:13:51.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:51 smithi100 ceph-mon[29092]: 2.3 scrub ok 2024-09-17T07:13:51.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:51 smithi100 ceph-mon[29092]: pgmap v524: 148 pgs: 3 active+clean+scrubbing, 145 active+clean; 8.1 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 885 B/s rd, 46 MiB/s wr, 952 op/s 2024-09-17T07:13:51.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:51 smithi100 ceph-mon[39462]: 2.11 scrub starts 2024-09-17T07:13:51.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:51 smithi100 ceph-mon[39462]: 2.11 scrub ok 2024-09-17T07:13:51.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:51 smithi100 ceph-mon[39462]: 2.1 scrub starts 2024-09-17T07:13:51.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:51 smithi100 ceph-mon[39462]: 2.1 scrub ok 2024-09-17T07:13:51.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:51 smithi100 ceph-mon[39462]: 2.3 scrub starts 2024-09-17T07:13:51.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:51 smithi100 ceph-mon[39462]: 2.3 scrub ok 2024-09-17T07:13:51.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:51 smithi100 ceph-mon[39462]: pgmap v524: 148 pgs: 3 active+clean+scrubbing, 145 active+clean; 8.1 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 885 B/s rd, 46 MiB/s wr, 952 op/s 2024-09-17T07:13:51.703 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:13:51.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:51 smithi135 ceph-mon[33238]: 2.11 scrub starts 2024-09-17T07:13:51.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:51 smithi135 ceph-mon[33238]: 2.11 scrub ok 2024-09-17T07:13:51.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:51 smithi135 ceph-mon[33238]: 2.1 scrub starts 2024-09-17T07:13:51.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:51 smithi135 ceph-mon[33238]: 2.1 scrub ok 2024-09-17T07:13:51.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:51 smithi135 ceph-mon[33238]: 2.3 scrub starts 2024-09-17T07:13:51.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:51 smithi135 ceph-mon[33238]: 2.3 scrub ok 2024-09-17T07:13:51.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:51 smithi135 ceph-mon[33238]: pgmap v524: 148 pgs: 3 active+clean+scrubbing, 145 active+clean; 8.1 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 885 B/s rd, 46 MiB/s wr, 952 op/s 2024-09-17T07:13:51.809 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:51.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:51.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:13:51.911 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:13:51.941 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:51.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:51.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:13:52.043 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:13:52.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:52.075 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:52.075 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:13:52.080 INFO:teuthology.orchestra.run.smithi135.stdout:{ 2024-09-17T07:13:52.080 INFO:teuthology.orchestra.run.smithi135.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-17T07:13:52.080 INFO:teuthology.orchestra.run.smithi135.stdout:} 2024-09-17T07:13:52.176 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:13:52.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:52.201 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:52.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:13:52.302 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:13:52.341 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:52.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:52.342 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:13:52.442 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:13:52.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:52.484 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:52.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:13:52.585 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:13:52.621 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:52.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:52.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:13:52.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:52 smithi100 ceph-mon[39462]: 6.1 scrub starts 2024-09-17T07:13:52.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:52 smithi100 ceph-mon[39462]: 6.1 scrub ok 2024-09-17T07:13:52.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:52 smithi100 ceph-mon[39462]: 9.1 scrub ok 2024-09-17T07:13:52.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:52 smithi100 ceph-mon[39462]: 4.16 scrub starts 2024-09-17T07:13:52.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:52 smithi100 ceph-mon[39462]: 4.16 scrub ok 2024-09-17T07:13:52.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:52 smithi100 ceph-mon[39462]: 9.6 scrub ok 2024-09-17T07:13:52.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:52 smithi100 ceph-mon[39462]: 9.7 scrub ok 2024-09-17T07:13:52.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:52 smithi100 ceph-mon[29092]: 6.1 scrub starts 2024-09-17T07:13:52.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:52 smithi100 ceph-mon[29092]: 6.1 scrub ok 2024-09-17T07:13:52.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:52 smithi100 ceph-mon[29092]: 9.1 scrub ok 2024-09-17T07:13:52.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:52 smithi100 ceph-mon[29092]: 4.16 scrub starts 2024-09-17T07:13:52.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:52 smithi100 ceph-mon[29092]: 4.16 scrub ok 2024-09-17T07:13:52.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:52 smithi100 ceph-mon[29092]: 9.6 scrub ok 2024-09-17T07:13:52.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:52 smithi100 ceph-mon[29092]: 9.7 scrub ok 2024-09-17T07:13:52.723 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:13:52.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:52.769 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:52.769 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:13:52.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:52 smithi135 ceph-mon[33238]: 6.1 scrub starts 2024-09-17T07:13:52.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:52 smithi135 ceph-mon[33238]: 6.1 scrub ok 2024-09-17T07:13:52.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:52 smithi135 ceph-mon[33238]: 9.1 scrub ok 2024-09-17T07:13:52.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:52 smithi135 ceph-mon[33238]: 4.16 scrub starts 2024-09-17T07:13:52.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:52 smithi135 ceph-mon[33238]: 4.16 scrub ok 2024-09-17T07:13:52.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:52 smithi135 ceph-mon[33238]: 9.6 scrub ok 2024-09-17T07:13:52.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:52 smithi135 ceph-mon[33238]: 9.7 scrub ok 2024-09-17T07:13:52.870 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:13:52.902 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:52.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:52.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:13:53.004 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:13:53.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:53.042 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:53.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:13:53.143 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:13:53.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:53.166 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:53.166 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:13:53.267 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:13:53.306 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:53.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:53.307 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:13:53.408 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:13:53.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:53.444 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:53.444 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:13:53.545 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:13:53.565 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:53.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:53.567 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:13:53.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:53 smithi100 ceph-mon[29092]: 4.d scrub starts 2024-09-17T07:13:53.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:53 smithi100 ceph-mon[29092]: 4.d scrub ok 2024-09-17T07:13:53.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:53 smithi100 ceph-mon[29092]: 4.1e scrub starts 2024-09-17T07:13:53.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:53 smithi100 ceph-mon[29092]: 4.1e scrub ok 2024-09-17T07:13:53.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:53 smithi100 ceph-mon[29092]: 2.10 scrub starts 2024-09-17T07:13:53.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:53 smithi100 ceph-mon[29092]: 2.10 scrub ok 2024-09-17T07:13:53.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:53 smithi100 ceph-mon[29092]: pgmap v525: 148 pgs: 3 active+clean+scrubbing, 145 active+clean; 8.1 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 767 B/s rd, 29 MiB/s wr, 653 op/s 2024-09-17T07:13:53.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:53 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2100821255' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:13:53.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:53 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:13:53.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:53 smithi100 ceph-mon[39462]: 4.d scrub starts 2024-09-17T07:13:53.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:53 smithi100 ceph-mon[39462]: 4.d scrub ok 2024-09-17T07:13:53.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:53 smithi100 ceph-mon[39462]: 4.1e scrub starts 2024-09-17T07:13:53.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:53 smithi100 ceph-mon[39462]: 4.1e scrub ok 2024-09-17T07:13:53.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:53 smithi100 ceph-mon[39462]: 2.10 scrub starts 2024-09-17T07:13:53.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:53 smithi100 ceph-mon[39462]: 2.10 scrub ok 2024-09-17T07:13:53.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:53 smithi100 ceph-mon[39462]: pgmap v525: 148 pgs: 3 active+clean+scrubbing, 145 active+clean; 8.1 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 767 B/s rd, 29 MiB/s wr, 653 op/s 2024-09-17T07:13:53.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:53 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2100821255' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:13:53.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:53 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:13:53.668 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:13:53.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:53.702 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:53.702 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:13:53.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:53 smithi135 ceph-mon[33238]: 4.d scrub starts 2024-09-17T07:13:53.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:53 smithi135 ceph-mon[33238]: 4.d scrub ok 2024-09-17T07:13:53.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:53 smithi135 ceph-mon[33238]: 4.1e scrub starts 2024-09-17T07:13:53.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:53 smithi135 ceph-mon[33238]: 4.1e scrub ok 2024-09-17T07:13:53.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:53 smithi135 ceph-mon[33238]: 2.10 scrub starts 2024-09-17T07:13:53.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:53 smithi135 ceph-mon[33238]: 2.10 scrub ok 2024-09-17T07:13:53.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:53 smithi135 ceph-mon[33238]: pgmap v525: 148 pgs: 3 active+clean+scrubbing, 145 active+clean; 8.1 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 767 B/s rd, 29 MiB/s wr, 653 op/s 2024-09-17T07:13:53.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:53 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2100821255' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:13:53.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:53 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:13:53.803 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:13:53.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:53.831 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:53.831 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:13:53.932 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:13:53.962 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:53.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:53.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:13:54.064 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:13:54.092 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:54.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:54.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:13:54.193 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:13:54.200 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:13:54.239 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:54.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:54.240 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:13:54.319 INFO:teuthology.orchestra.run.smithi100.stderr:nodeep-scrub is unset 2024-09-17T07:13:54.341 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:13:54.371 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:54.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:54.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:13:54.473 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:13:54.502 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:54.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:54.503 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:13:54.603 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:13:54.616 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:54 smithi135 ceph-mon[33238]: 2.1a scrub starts 2024-09-17T07:13:54.616 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:54 smithi135 ceph-mon[33238]: 2.1a scrub ok 2024-09-17T07:13:54.616 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:54 smithi135 ceph-mon[33238]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T07:13:54.616 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:54 smithi135 ceph-mon[33238]: Cluster is now healthy 2024-09-17T07:13:54.617 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:54 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T07:13:54.617 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:54 smithi135 ceph-mon[33238]: osdmap e268: 8 total, 8 up, 7 in 2024-09-17T07:13:54.617 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:54 smithi135 ceph-mon[33238]: 6.6 deep-scrub starts 2024-09-17T07:13:54.617 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:54 smithi135 ceph-mon[33238]: 6.6 deep-scrub ok 2024-09-17T07:13:54.617 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:54 smithi135 ceph-mon[33238]: 5.6 scrub starts 2024-09-17T07:13:54.617 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:54 smithi135 ceph-mon[33238]: 5.6 scrub ok 2024-09-17T07:13:54.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:54 smithi100 ceph-mon[29092]: 2.1a scrub starts 2024-09-17T07:13:54.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:54 smithi100 ceph-mon[29092]: 2.1a scrub ok 2024-09-17T07:13:54.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:54 smithi100 ceph-mon[29092]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T07:13:54.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:54 smithi100 ceph-mon[29092]: Cluster is now healthy 2024-09-17T07:13:54.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:54 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T07:13:54.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:54 smithi100 ceph-mon[29092]: osdmap e268: 8 total, 8 up, 7 in 2024-09-17T07:13:54.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:54 smithi100 ceph-mon[29092]: 6.6 deep-scrub starts 2024-09-17T07:13:54.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:54 smithi100 ceph-mon[29092]: 6.6 deep-scrub ok 2024-09-17T07:13:54.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:54 smithi100 ceph-mon[29092]: 5.6 scrub starts 2024-09-17T07:13:54.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:54 smithi100 ceph-mon[29092]: 5.6 scrub ok 2024-09-17T07:13:54.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:54 smithi100 ceph-mon[39462]: 2.1a scrub starts 2024-09-17T07:13:54.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:54 smithi100 ceph-mon[39462]: 2.1a scrub ok 2024-09-17T07:13:54.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:54 smithi100 ceph-mon[39462]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T07:13:54.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:54 smithi100 ceph-mon[39462]: Cluster is now healthy 2024-09-17T07:13:54.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:54 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T07:13:54.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:54 smithi100 ceph-mon[39462]: osdmap e268: 8 total, 8 up, 7 in 2024-09-17T07:13:54.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:54 smithi100 ceph-mon[39462]: 6.6 deep-scrub starts 2024-09-17T07:13:54.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:54 smithi100 ceph-mon[39462]: 6.6 deep-scrub ok 2024-09-17T07:13:54.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:54 smithi100 ceph-mon[39462]: 5.6 scrub starts 2024-09-17T07:13:54.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:54 smithi100 ceph-mon[39462]: 5.6 scrub ok 2024-09-17T07:13:54.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:54.629 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:54.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:13:54.730 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:13:54.760 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:54.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:54.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:13:54.862 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:13:54.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:54.899 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:54.899 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:13:55.000 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:13:55.027 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:55.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:55.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:13:55.129 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:13:55.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:55.159 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:55.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:13:55.260 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:13:55.282 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:55.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:55.283 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:13:55.384 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:13:55.413 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:55.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:55.415 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:13:55.515 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:13:55.538 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:13:55.545 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:55.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:55.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:13:55.625 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:55 smithi135 ceph-mon[33238]: 2.d deep-scrub starts 2024-09-17T07:13:55.625 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:55 smithi135 ceph-mon[33238]: 2.d deep-scrub ok 2024-09-17T07:13:55.625 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:55 smithi135 ceph-mon[33238]: 2.1f scrub starts 2024-09-17T07:13:55.625 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:55 smithi135 ceph-mon[33238]: 2.1f scrub ok 2024-09-17T07:13:55.625 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:55 smithi135 ceph-mon[33238]: 3.2 deep-scrub starts 2024-09-17T07:13:55.625 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:55 smithi135 ceph-mon[33238]: 3.2 deep-scrub ok 2024-09-17T07:13:55.625 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:55 smithi135 ceph-mon[33238]: pgmap v527: 148 pgs: 148 active+clean; 7.8 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 1.4 KiB/s rd, 29 MiB/s wr, 1.22k op/s 2024-09-17T07:13:55.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:55 smithi100 ceph-mon[39462]: 2.d deep-scrub starts 2024-09-17T07:13:55.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:55 smithi100 ceph-mon[39462]: 2.d deep-scrub ok 2024-09-17T07:13:55.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:55 smithi100 ceph-mon[39462]: 2.1f scrub starts 2024-09-17T07:13:55.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:55 smithi100 ceph-mon[39462]: 2.1f scrub ok 2024-09-17T07:13:55.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:55 smithi100 ceph-mon[39462]: 3.2 deep-scrub starts 2024-09-17T07:13:55.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:55 smithi100 ceph-mon[39462]: 3.2 deep-scrub ok 2024-09-17T07:13:55.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:55 smithi100 ceph-mon[39462]: pgmap v527: 148 pgs: 148 active+clean; 7.8 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 1.4 KiB/s rd, 29 MiB/s wr, 1.22k op/s 2024-09-17T07:13:55.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:55 smithi100 ceph-mon[29092]: 2.d deep-scrub starts 2024-09-17T07:13:55.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:55 smithi100 ceph-mon[29092]: 2.d deep-scrub ok 2024-09-17T07:13:55.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:55 smithi100 ceph-mon[29092]: 2.1f scrub starts 2024-09-17T07:13:55.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:55 smithi100 ceph-mon[29092]: 2.1f scrub ok 2024-09-17T07:13:55.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:55 smithi100 ceph-mon[29092]: 3.2 deep-scrub starts 2024-09-17T07:13:55.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:55 smithi100 ceph-mon[29092]: 3.2 deep-scrub ok 2024-09-17T07:13:55.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:55 smithi100 ceph-mon[29092]: pgmap v527: 148 pgs: 148 active+clean; 7.8 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 1.4 KiB/s rd, 29 MiB/s wr, 1.22k op/s 2024-09-17T07:13:55.647 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:13:55.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:55.676 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:55.676 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:13:55.777 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:13:55.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:55.813 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:55.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:13:55.914 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:13:55.943 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:55.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:55.944 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:13:56.045 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:13:56.058 INFO:teuthology.orchestra.run.smithi135.stdout:{ 2024-09-17T07:13:56.058 INFO:teuthology.orchestra.run.smithi135.stdout: "size": 20, 2024-09-17T07:13:56.058 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 600, 2024-09-17T07:13:56.058 INFO:teuthology.orchestra.run.smithi135.stdout: "ops": [ 2024-09-17T07:13:56.058 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.058 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:54899 7.7 7:e2e5199d:::benchmark_data_smithi100_78556_object54898:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e111)", 2024-09-17T07:13:56.059 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:04:57.339110+0000", 2024-09-17T07:13:56.059 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 538.715259543, 2024-09-17T07:13:56.059 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0186058010000001, 2024-09-17T07:13:56.059 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:13:56.059 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "reached pg", 2024-09-17T07:13:56.059 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:13:56.059 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:13:56.059 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:13:56.059 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 54899 2024-09-17T07:13:56.059 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.059 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:13:56.059 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.059 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:13:56.060 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:57.339110+0000", 2024-09-17T07:13:56.060 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:56.060 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.060 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.060 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:13:56.060 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:57.339110+0000", 2024-09-17T07:13:56.060 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:56.060 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.060 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.060 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:13:56.060 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:57.339108+0000", 2024-09-17T07:13:56.060 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999981 2024-09-17T07:13:56.061 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.061 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.061 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:13:56.061 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:57.339145+0000", 2024-09-17T07:13:56.061 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.6754000000000001e-05 2024-09-17T07:13:56.061 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.061 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.061 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:13:56.061 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:57.339156+0000", 2024-09-17T07:13:56.061 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.115e-05 2024-09-17T07:13:56.061 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.061 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.062 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:13:56.062 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:57.339164+0000", 2024-09-17T07:13:56.062 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.909e-06 2024-09-17T07:13:56.062 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.062 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.062 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:56.062 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:57.424983+0000", 2024-09-17T07:13:56.062 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.085818679999999994 2024-09-17T07:13:56.062 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.062 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.062 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:13:56.062 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:57.424994+0000", 2024-09-17T07:13:56.063 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.1432e-05 2024-09-17T07:13:56.063 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.063 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.063 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:56.063 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:58.357650+0000", 2024-09-17T07:13:56.063 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.93265618299999997 2024-09-17T07:13:56.063 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.063 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.063 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:13:56.063 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:58.357716+0000", 2024-09-17T07:13:56.063 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.5579000000000004e-05 2024-09-17T07:13:56.063 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:56.063 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:13:56.064 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:56.064 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.064 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.064 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:54910 7.17 7:e939e698:::benchmark_data_smithi100_78556_object54909:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-09-17T07:13:56.064 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:04:57.368094+0000", 2024-09-17T07:13:56.064 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 538.68627550099995, 2024-09-17T07:13:56.064 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0100409610000001, 2024-09-17T07:13:56.064 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:13:56.064 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:13:56.064 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:13:56.064 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:13:56.064 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:13:56.065 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 54910 2024-09-17T07:13:56.065 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.065 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:13:56.065 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.065 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:13:56.065 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:57.368094+0000", 2024-09-17T07:13:56.065 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:56.065 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.065 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.065 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:13:56.065 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:57.368094+0000", 2024-09-17T07:13:56.065 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:56.066 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.066 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.066 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:13:56.066 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:57.368091+0000", 2024-09-17T07:13:56.066 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999967 2024-09-17T07:13:56.066 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.066 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.066 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:13:56.066 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:57.368164+0000", 2024-09-17T07:13:56.066 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.3077999999999994e-05 2024-09-17T07:13:56.066 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.066 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.066 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:13:56.067 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:57.368211+0000", 2024-09-17T07:13:56.067 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.6356999999999997e-05 2024-09-17T07:13:56.067 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.067 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.067 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:13:56.067 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:57.368229+0000", 2024-09-17T07:13:56.067 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.8425999999999999e-05 2024-09-17T07:13:56.067 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.067 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.067 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:56.067 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:57.425154+0000", 2024-09-17T07:13:56.067 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.056925396000000003 2024-09-17T07:13:56.068 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.068 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.068 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:13:56.068 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:57.425162+0000", 2024-09-17T07:13:56.068 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.2599999999999999e-06 2024-09-17T07:13:56.068 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.068 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.068 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:56.068 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:58.368459+0000", 2024-09-17T07:13:56.068 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.94329758900000005 2024-09-17T07:13:56.068 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.068 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.068 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:13:56.069 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:58.368589+0000", 2024-09-17T07:13:56.069 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00013005100000000001 2024-09-17T07:13:56.069 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.069 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.069 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,3", 2024-09-17T07:13:56.069 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:58.368741+0000", 2024-09-17T07:13:56.069 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00015186299999999999 2024-09-17T07:13:56.069 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.069 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.069 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:13:56.069 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:58.369926+0000", 2024-09-17T07:13:56.069 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0011852900000000001 2024-09-17T07:13:56.070 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.070 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.070 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:13:56.070 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:58.374167+0000", 2024-09-17T07:13:56.070 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0042405339999999998 2024-09-17T07:13:56.070 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.070 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.070 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:13:56.070 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:58.378070+0000", 2024-09-17T07:13:56.070 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0039034460000000001 2024-09-17T07:13:56.070 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.070 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.070 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:13:56.071 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:58.378103+0000", 2024-09-17T07:13:56.071 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.2332999999999997e-05 2024-09-17T07:13:56.071 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.071 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.071 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:13:56.071 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:04:58.378135+0000", 2024-09-17T07:13:56.071 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.2515000000000003e-05 2024-09-17T07:13:56.071 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:56.071 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:13:56.071 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:56.071 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.071 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.072 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:90793 7.12 7:4bb4ecc8:::benchmark_data_smithi100_78556_object90792:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-17T07:13:56.072 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:05:38.427843+0000", 2024-09-17T07:13:56.072 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 497.62652661200002, 2024-09-17T07:13:56.072 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.78499230399999997, 2024-09-17T07:13:56.072 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:13:56.072 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:13:56.072 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:13:56.072 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:13:56.072 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:13:56.072 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 90793 2024-09-17T07:13:56.072 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.072 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:13:56.072 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.073 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:13:56.073 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.427843+0000", 2024-09-17T07:13:56.073 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:56.073 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.073 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.073 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:13:56.074 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.427843+0000", 2024-09-17T07:13:56.074 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:56.074 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.075 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.075 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:13:56.075 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.427842+0000", 2024-09-17T07:13:56.075 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.999999 2024-09-17T07:13:56.075 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.075 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.075 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:13:56.075 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.427922+0000", 2024-09-17T07:13:56.075 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.9587999999999998e-05 2024-09-17T07:13:56.075 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.075 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.075 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:13:56.075 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.427932+0000", 2024-09-17T07:13:56.076 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0176999999999999e-05 2024-09-17T07:13:56.076 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.076 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.076 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:13:56.076 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.427939+0000", 2024-09-17T07:13:56.076 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.7229999999999998e-06 2024-09-17T07:13:56.076 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.076 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.076 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:56.076 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.428022+0000", 2024-09-17T07:13:56.076 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.2738999999999995e-05 2024-09-17T07:13:56.076 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.077 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.077 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:13:56.077 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.428034+0000", 2024-09-17T07:13:56.077 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.2415999999999999e-05 2024-09-17T07:13:56.077 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.077 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.077 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:56.077 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.201180+0000", 2024-09-17T07:13:56.077 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.773146369 2024-09-17T07:13:56.077 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.077 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.077 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:13:56.077 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.201327+0000", 2024-09-17T07:13:56.078 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00014637799999999999 2024-09-17T07:13:56.078 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.078 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.078 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,2", 2024-09-17T07:13:56.078 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.201497+0000", 2024-09-17T07:13:56.078 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000170345 2024-09-17T07:13:56.078 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.078 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.078 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:13:56.078 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.204531+0000", 2024-09-17T07:13:56.078 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0030341790000000001 2024-09-17T07:13:56.078 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.079 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.079 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:13:56.079 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.207437+0000", 2024-09-17T07:13:56.079 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0029059569999999998 2024-09-17T07:13:56.079 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.079 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.079 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:13:56.079 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.212762+0000", 2024-09-17T07:13:56.079 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0053249609999999996 2024-09-17T07:13:56.079 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.079 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.079 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:13:56.079 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.212795+0000", 2024-09-17T07:13:56.080 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.2459999999999998e-05 2024-09-17T07:13:56.080 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.080 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.080 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:13:56.080 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.212835+0000", 2024-09-17T07:13:56.080 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.0763000000000003e-05 2024-09-17T07:13:56.080 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:56.080 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:13:56.080 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:56.080 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.080 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.080 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:90813 7.12 7:49534b43:::benchmark_data_smithi100_78556_object90812:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-17T07:13:56.081 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:05:38.447256+0000", 2024-09-17T07:13:56.081 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 497.607114013, 2024-09-17T07:13:56.081 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.76565891100000005, 2024-09-17T07:13:56.081 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:13:56.081 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:13:56.081 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:13:56.081 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:13:56.081 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:13:56.081 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 90813 2024-09-17T07:13:56.081 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.081 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:13:56.081 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.082 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:13:56.082 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.447256+0000", 2024-09-17T07:13:56.082 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:56.082 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.082 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.082 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:13:56.082 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.447256+0000", 2024-09-17T07:13:56.082 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:56.082 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.082 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.082 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:13:56.082 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.447253+0000", 2024-09-17T07:13:56.082 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999976 2024-09-17T07:13:56.083 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.083 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.083 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:13:56.083 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.447306+0000", 2024-09-17T07:13:56.083 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.2253000000000003e-05 2024-09-17T07:13:56.083 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.083 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.083 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:13:56.083 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.447318+0000", 2024-09-17T07:13:56.083 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.2143999999999999e-05 2024-09-17T07:13:56.083 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.083 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.084 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:13:56.084 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.447323+0000", 2024-09-17T07:13:56.084 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.3430000000000003e-06 2024-09-17T07:13:56.084 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.084 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.084 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:56.084 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.447374+0000", 2024-09-17T07:13:56.084 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.0676000000000001e-05 2024-09-17T07:13:56.084 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.084 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.084 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:13:56.084 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.447385+0000", 2024-09-17T07:13:56.084 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.1395e-05 2024-09-17T07:13:56.085 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.085 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.085 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:56.085 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.202849+0000", 2024-09-17T07:13:56.085 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.75546397600000004 2024-09-17T07:13:56.085 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.085 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.085 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:13:56.085 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.202930+0000", 2024-09-17T07:13:56.085 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.1080000000000003e-05 2024-09-17T07:13:56.085 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.085 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.085 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,2", 2024-09-17T07:13:56.086 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.203018+0000", 2024-09-17T07:13:56.086 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.7695999999999995e-05 2024-09-17T07:13:56.086 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.086 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.086 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:13:56.086 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.206503+0000", 2024-09-17T07:13:56.086 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.003484816 2024-09-17T07:13:56.086 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.086 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.086 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:13:56.086 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.207478+0000", 2024-09-17T07:13:56.086 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00097474300000000003 2024-09-17T07:13:56.087 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.087 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.087 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:13:56.087 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.212868+0000", 2024-09-17T07:13:56.087 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0053907130000000001 2024-09-17T07:13:56.087 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.087 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.087 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:13:56.087 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.212895+0000", 2024-09-17T07:13:56.087 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.6602e-05 2024-09-17T07:13:56.087 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.087 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.087 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:13:56.087 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.212915+0000", 2024-09-17T07:13:56.088 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.9724999999999998e-05 2024-09-17T07:13:56.088 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:56.088 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:13:56.088 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:56.088 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.088 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.088 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:90860 7.12 7:48288261:::benchmark_data_smithi100_78556_object90859:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-17T07:13:56.088 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:05:38.496803+0000", 2024-09-17T07:13:56.088 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 497.55756661300001, 2024-09-17T07:13:56.088 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.72173665899999995, 2024-09-17T07:13:56.088 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:13:56.088 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:13:56.088 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:13:56.089 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:13:56.089 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:13:56.089 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 90860 2024-09-17T07:13:56.089 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.089 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:13:56.089 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.089 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:13:56.089 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.496803+0000", 2024-09-17T07:13:56.089 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:56.089 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.089 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.089 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:13:56.090 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.496803+0000", 2024-09-17T07:13:56.090 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:56.090 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.090 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.090 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:13:56.090 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.496802+0000", 2024-09-17T07:13:56.090 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:13:56.090 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.090 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.090 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:13:56.090 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.496851+0000", 2024-09-17T07:13:56.090 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.9397999999999997e-05 2024-09-17T07:13:56.090 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.091 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.091 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:13:56.091 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.496863+0000", 2024-09-17T07:13:56.091 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.1633e-05 2024-09-17T07:13:56.091 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.091 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.091 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:13:56.091 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.496869+0000", 2024-09-17T07:13:56.092 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.8159999999999999e-06 2024-09-17T07:13:56.092 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.092 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.092 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:56.092 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.496924+0000", 2024-09-17T07:13:56.092 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.5458999999999999e-05 2024-09-17T07:13:56.092 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.092 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.092 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:13:56.092 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.496935+0000", 2024-09-17T07:13:56.092 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0658e-05 2024-09-17T07:13:56.093 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.093 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.093 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:56.093 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.203685+0000", 2024-09-17T07:13:56.093 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.706750604 2024-09-17T07:13:56.093 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.093 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.093 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:13:56.093 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.203780+0000", 2024-09-17T07:13:56.093 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.4462999999999993e-05 2024-09-17T07:13:56.093 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.093 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.094 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,2", 2024-09-17T07:13:56.094 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.203863+0000", 2024-09-17T07:13:56.094 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.3236000000000001e-05 2024-09-17T07:13:56.094 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.094 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.094 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:13:56.094 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.207445+0000", 2024-09-17T07:13:56.094 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0035815589999999998 2024-09-17T07:13:56.094 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.094 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.094 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:13:56.094 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.209736+0000", 2024-09-17T07:13:56.094 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0022916249999999998 2024-09-17T07:13:56.095 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.095 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.095 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:13:56.095 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.218451+0000", 2024-09-17T07:13:56.095 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0087142590000000002 2024-09-17T07:13:56.095 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.095 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.095 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:13:56.095 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.218499+0000", 2024-09-17T07:13:56.095 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.8624000000000001e-05 2024-09-17T07:13:56.095 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.095 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.096 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:13:56.096 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.218540+0000", 2024-09-17T07:13:56.096 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.0633999999999998e-05 2024-09-17T07:13:56.096 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:56.096 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:13:56.096 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:56.096 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.096 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.096 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:90870 7.12 7:4b253461:::benchmark_data_smithi100_78556_object90869:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-17T07:13:56.096 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:05:38.502763+0000", 2024-09-17T07:13:56.096 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 497.55160625100001, 2024-09-17T07:13:56.096 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.723378619, 2024-09-17T07:13:56.097 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:13:56.097 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:13:56.097 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:13:56.097 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:13:56.097 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:13:56.097 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 90870 2024-09-17T07:13:56.097 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.097 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:13:56.097 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.097 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:13:56.097 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.502763+0000", 2024-09-17T07:13:56.097 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:56.097 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.098 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.098 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:13:56.098 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.502763+0000", 2024-09-17T07:13:56.098 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:56.098 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.098 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.098 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:13:56.098 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.502763+0000", 2024-09-17T07:13:56.098 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.999999 2024-09-17T07:13:56.098 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.098 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.098 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:13:56.098 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.502862+0000", 2024-09-17T07:13:56.099 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.9003000000000001e-05 2024-09-17T07:13:56.099 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.099 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.099 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:13:56.099 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.502873+0000", 2024-09-17T07:13:56.099 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0919e-05 2024-09-17T07:13:56.099 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.099 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.099 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:13:56.099 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.502881+0000", 2024-09-17T07:13:56.099 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.3080000000000008e-06 2024-09-17T07:13:56.099 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.099 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.100 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:56.100 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.502962+0000", 2024-09-17T07:13:56.100 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.0871000000000005e-05 2024-09-17T07:13:56.100 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.100 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.100 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:13:56.100 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.502971+0000", 2024-09-17T07:13:56.100 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.4930000000000004e-06 2024-09-17T07:13:56.100 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.100 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.100 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:56.100 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.204570+0000", 2024-09-17T07:13:56.101 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.70159855100000001 2024-09-17T07:13:56.101 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.101 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.101 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:13:56.101 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.204687+0000", 2024-09-17T07:13:56.101 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00011735000000000001 2024-09-17T07:13:56.101 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.101 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.101 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,2", 2024-09-17T07:13:56.101 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.204770+0000", 2024-09-17T07:13:56.101 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.2836000000000004e-05 2024-09-17T07:13:56.101 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.101 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.102 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:13:56.102 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.207451+0000", 2024-09-17T07:13:56.102 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0026814059999999999 2024-09-17T07:13:56.102 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.102 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.102 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:13:56.102 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.209766+0000", 2024-09-17T07:13:56.102 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00231452 2024-09-17T07:13:56.102 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.102 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.102 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:13:56.102 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.226055+0000", 2024-09-17T07:13:56.102 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.016288830000000001 2024-09-17T07:13:56.103 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.103 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.103 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:13:56.103 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.226104+0000", 2024-09-17T07:13:56.103 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.8875999999999998e-05 2024-09-17T07:13:56.103 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.103 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.103 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:13:56.103 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.226142+0000", 2024-09-17T07:13:56.103 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.8529000000000003e-05 2024-09-17T07:13:56.103 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:56.103 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:13:56.104 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:56.104 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.104 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.104 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:90872 7.12 7:48b9e8b9:::benchmark_data_smithi100_78556_object90871:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-17T07:13:56.104 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:05:38.503420+0000", 2024-09-17T07:13:56.104 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 497.55095001799998, 2024-09-17T07:13:56.104 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.72282881499999996, 2024-09-17T07:13:56.104 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:13:56.104 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:13:56.104 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:13:56.104 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:13:56.104 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:13:56.105 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 90872 2024-09-17T07:13:56.105 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.105 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:13:56.105 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.105 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:13:56.105 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.503420+0000", 2024-09-17T07:13:56.105 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:56.105 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.105 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.105 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:13:56.105 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.503420+0000", 2024-09-17T07:13:56.105 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:56.106 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.106 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.106 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:13:56.106 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.503419+0000", 2024-09-17T07:13:56.106 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.999999 2024-09-17T07:13:56.106 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.106 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.106 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:13:56.106 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.503493+0000", 2024-09-17T07:13:56.106 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.3919999999999997e-05 2024-09-17T07:13:56.106 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.106 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.106 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:13:56.107 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.503505+0000", 2024-09-17T07:13:56.107 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.2248e-05 2024-09-17T07:13:56.107 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.107 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.107 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:13:56.107 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.503512+0000", 2024-09-17T07:13:56.107 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.9199999999999998e-06 2024-09-17T07:13:56.107 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.107 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.107 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:56.107 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.503564+0000", 2024-09-17T07:13:56.107 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.2142000000000003e-05 2024-09-17T07:13:56.108 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.108 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.108 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:13:56.108 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.503572+0000", 2024-09-17T07:13:56.108 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.3720000000000005e-06 2024-09-17T07:13:56.108 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.108 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.108 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:56.108 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.205473+0000", 2024-09-17T07:13:56.108 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.70190045599999995 2024-09-17T07:13:56.108 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.108 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.109 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:13:56.109 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.205577+0000", 2024-09-17T07:13:56.109 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00010372100000000001 2024-09-17T07:13:56.109 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.109 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.109 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,2", 2024-09-17T07:13:56.109 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.205712+0000", 2024-09-17T07:13:56.109 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00013573400000000001 2024-09-17T07:13:56.109 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.109 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.109 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:13:56.109 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.208223+0000", 2024-09-17T07:13:56.110 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0025110140000000002 2024-09-17T07:13:56.110 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.110 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.110 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:13:56.110 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.209806+0000", 2024-09-17T07:13:56.110 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0015828330000000001 2024-09-17T07:13:56.110 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.110 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.110 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:13:56.110 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.226184+0000", 2024-09-17T07:13:56.110 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.016377454 2024-09-17T07:13:56.110 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.110 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.111 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:13:56.111 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.226217+0000", 2024-09-17T07:13:56.111 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.3169e-05 2024-09-17T07:13:56.111 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.111 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.111 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:13:56.111 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.226249+0000", 2024-09-17T07:13:56.111 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.1807000000000002e-05 2024-09-17T07:13:56.111 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:56.111 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:13:56.111 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:56.111 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.112 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.112 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:90873 7.12 7:4f7608af:::benchmark_data_smithi100_78556_object90872:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-17T07:13:56.112 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:05:38.505355+0000", 2024-09-17T07:13:56.112 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 497.54901453600002, 2024-09-17T07:13:56.112 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.72099324600000003, 2024-09-17T07:13:56.112 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:13:56.112 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:13:56.112 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:13:56.112 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:13:56.112 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:13:56.112 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 90873 2024-09-17T07:13:56.112 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.112 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:13:56.113 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.113 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:13:56.113 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.505355+0000", 2024-09-17T07:13:56.113 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:56.113 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.113 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.113 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:13:56.113 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.505355+0000", 2024-09-17T07:13:56.113 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:56.113 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.113 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.113 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:13:56.113 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.505353+0000", 2024-09-17T07:13:56.114 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999981 2024-09-17T07:13:56.114 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.114 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.114 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:13:56.114 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.505426+0000", 2024-09-17T07:13:56.114 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.3030999999999999e-05 2024-09-17T07:13:56.114 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.114 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.114 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:13:56.114 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.505441+0000", 2024-09-17T07:13:56.114 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.4871999999999999e-05 2024-09-17T07:13:56.114 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.114 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.115 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:13:56.115 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.505456+0000", 2024-09-17T07:13:56.115 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.487e-05 2024-09-17T07:13:56.115 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.115 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.115 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:56.115 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.505512+0000", 2024-09-17T07:13:56.115 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.5949999999999998e-05 2024-09-17T07:13:56.115 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.115 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.115 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:13:56.115 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.505524+0000", 2024-09-17T07:13:56.115 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.1613e-05 2024-09-17T07:13:56.116 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.116 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.116 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:56.116 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.206533+0000", 2024-09-17T07:13:56.116 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.70100958300000005 2024-09-17T07:13:56.116 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.116 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.116 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:13:56.116 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.206627+0000", 2024-09-17T07:13:56.116 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.4192999999999998e-05 2024-09-17T07:13:56.116 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.116 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.117 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,2", 2024-09-17T07:13:56.117 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.206714+0000", 2024-09-17T07:13:56.117 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.6081999999999993e-05 2024-09-17T07:13:56.117 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.117 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.117 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:13:56.117 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.209706+0000", 2024-09-17T07:13:56.117 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0029928770000000001 2024-09-17T07:13:56.117 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.117 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.117 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:13:56.117 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.212641+0000", 2024-09-17T07:13:56.117 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0029341910000000001 2024-09-17T07:13:56.118 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.118 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.118 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:13:56.118 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.226285+0000", 2024-09-17T07:13:56.118 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.013644 2024-09-17T07:13:56.118 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.118 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.118 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:13:56.118 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.226319+0000", 2024-09-17T07:13:56.118 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.4233000000000002e-05 2024-09-17T07:13:56.118 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.118 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.119 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:13:56.119 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.226348+0000", 2024-09-17T07:13:56.119 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.9641999999999998e-05 2024-09-17T07:13:56.119 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:56.119 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:13:56.119 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:56.119 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.119 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.119 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:90894 7.12 7:499604d5:::benchmark_data_smithi100_78556_object90893:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-17T07:13:56.119 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:05:38.528712+0000", 2024-09-17T07:13:56.119 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 497.52565819099999, 2024-09-17T07:13:56.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:56.120 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:56.121 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:13:56.121 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:13:55 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:13:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:13:56.122 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.69785077299999998, 2024-09-17T07:13:56.122 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:13:56.122 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:13:56.122 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:13:56.123 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:13:56.123 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:13:56.123 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 90894 2024-09-17T07:13:56.123 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.123 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:13:56.123 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.123 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:13:56.123 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.528712+0000", 2024-09-17T07:13:56.123 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:56.123 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.123 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.123 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:13:56.124 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.528712+0000", 2024-09-17T07:13:56.124 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:56.124 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.124 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.124 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:13:56.124 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.528710+0000", 2024-09-17T07:13:56.124 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999981 2024-09-17T07:13:56.124 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.124 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.124 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:13:56.124 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.528748+0000", 2024-09-17T07:13:56.124 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.8711999999999998e-05 2024-09-17T07:13:56.125 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.125 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.125 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:13:56.125 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.528759+0000", 2024-09-17T07:13:56.125 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0056e-05 2024-09-17T07:13:56.125 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.125 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.125 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:13:56.125 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.528765+0000", 2024-09-17T07:13:56.125 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.6359999999999997e-06 2024-09-17T07:13:56.125 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.125 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.125 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:56.126 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.528814+0000", 2024-09-17T07:13:56.126 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.8872999999999998e-05 2024-09-17T07:13:56.126 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.126 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.126 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:13:56.126 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.528824+0000", 2024-09-17T07:13:56.126 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0145e-05 2024-09-17T07:13:56.126 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.126 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.126 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:56.126 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.207500+0000", 2024-09-17T07:13:56.126 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.67867577700000004 2024-09-17T07:13:56.126 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.127 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.127 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:13:56.127 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.207567+0000", 2024-09-17T07:13:56.127 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.6982000000000003e-05 2024-09-17T07:13:56.127 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.127 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.127 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,2", 2024-09-17T07:13:56.127 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.207644+0000", 2024-09-17T07:13:56.127 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.6905999999999999e-05 2024-09-17T07:13:56.127 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.127 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.127 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:13:56.128 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.209772+0000", 2024-09-17T07:13:56.128 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0021282419999999998 2024-09-17T07:13:56.128 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.128 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.128 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:13:56.128 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.212700+0000", 2024-09-17T07:13:56.128 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0029276889999999998 2024-09-17T07:13:56.128 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.128 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.128 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:13:56.128 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.226515+0000", 2024-09-17T07:13:56.128 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.013815403 2024-09-17T07:13:56.128 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.129 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.129 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:13:56.129 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.226539+0000", 2024-09-17T07:13:56.129 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.4128999999999999e-05 2024-09-17T07:13:56.129 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.129 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.129 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:13:56.129 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.226562+0000", 2024-09-17T07:13:56.129 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.3037999999999999e-05 2024-09-17T07:13:56.129 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:56.129 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:13:56.129 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:56.130 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.130 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.130 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:90928 7.12 7:4920d0db:::benchmark_data_smithi100_78556_object90927:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-17T07:13:56.130 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:05:38.563430+0000", 2024-09-17T07:13:56.130 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 497.49093947699998, 2024-09-17T07:13:56.130 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.66321884600000003, 2024-09-17T07:13:56.130 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:13:56.130 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:13:56.130 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:13:56.130 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:13:56.130 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:13:56.130 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 90928 2024-09-17T07:13:56.130 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.131 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:13:56.131 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.131 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:13:56.131 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.563430+0000", 2024-09-17T07:13:56.131 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:56.131 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.131 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.131 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:13:56.131 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.563430+0000", 2024-09-17T07:13:56.131 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:56.131 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.131 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.131 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:13:56.132 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.563428+0000", 2024-09-17T07:13:56.132 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999981 2024-09-17T07:13:56.132 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.132 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.132 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:13:56.132 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.563483+0000", 2024-09-17T07:13:56.132 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.4409e-05 2024-09-17T07:13:56.132 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.132 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.132 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:13:56.132 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.563493+0000", 2024-09-17T07:13:56.132 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.059e-05 2024-09-17T07:13:56.133 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.133 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.133 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:13:56.133 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.563500+0000", 2024-09-17T07:13:56.133 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.1129999999999997e-06 2024-09-17T07:13:56.133 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.133 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.133 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:56.133 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.563563+0000", 2024-09-17T07:13:56.133 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.3647999999999998e-05 2024-09-17T07:13:56.133 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.133 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.134 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:13:56.134 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.563574+0000", 2024-09-17T07:13:56.134 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.1260000000000001e-05 2024-09-17T07:13:56.134 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.134 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.134 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:56.134 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.208250+0000", 2024-09-17T07:13:56.134 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.64467564499999996 2024-09-17T07:13:56.134 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.134 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.134 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:13:56.134 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.208335+0000", 2024-09-17T07:13:56.134 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.4809999999999996e-05 2024-09-17T07:13:56.135 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.135 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.135 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,2", 2024-09-17T07:13:56.135 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.208426+0000", 2024-09-17T07:13:56.135 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.0944000000000002e-05 2024-09-17T07:13:56.135 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.135 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.135 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:13:56.135 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.209778+0000", 2024-09-17T07:13:56.135 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0013524609999999999 2024-09-17T07:13:56.135 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.135 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.135 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:13:56.136 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.212731+0000", 2024-09-17T07:13:56.136 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0029530670000000002 2024-09-17T07:13:56.136 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.136 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.136 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:13:56.136 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.226586+0000", 2024-09-17T07:13:56.136 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.013855012 2024-09-17T07:13:56.136 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.136 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.136 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:13:56.136 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.226613+0000", 2024-09-17T07:13:56.136 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.7013e-05 2024-09-17T07:13:56.137 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.137 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.137 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:13:56.137 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.226649+0000", 2024-09-17T07:13:56.137 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.57e-05 2024-09-17T07:13:56.137 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:56.137 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:13:56.137 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:56.137 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.137 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.137 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:90963 7.12 7:4e9f616d:::benchmark_data_smithi100_78556_object90962:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-17T07:13:56.137 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:05:38.599732+0000", 2024-09-17T07:13:56.137 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 497.45463819899999, 2024-09-17T07:13:56.138 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.636136068, 2024-09-17T07:13:56.138 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:13:56.138 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:13:56.138 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:13:56.138 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:13:56.138 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:13:56.138 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 90963 2024-09-17T07:13:56.138 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.138 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:13:56.138 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.138 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:13:56.138 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.599732+0000", 2024-09-17T07:13:56.138 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:56.139 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.139 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.139 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:13:56.139 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.599732+0000", 2024-09-17T07:13:56.139 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:56.140 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.140 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.140 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:13:56.140 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.599730+0000", 2024-09-17T07:13:56.140 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:13:56.140 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.140 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.140 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:13:56.140 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.599776+0000", 2024-09-17T07:13:56.140 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.5624000000000002e-05 2024-09-17T07:13:56.140 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.140 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.141 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:13:56.141 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.599792+0000", 2024-09-17T07:13:56.141 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.6042999999999999e-05 2024-09-17T07:13:56.141 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.141 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.141 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:13:56.141 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.599800+0000", 2024-09-17T07:13:56.141 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.1769999999999998e-06 2024-09-17T07:13:56.141 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.141 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.141 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:56.141 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.599856+0000", 2024-09-17T07:13:56.142 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.6582999999999997e-05 2024-09-17T07:13:56.142 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.142 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.142 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:13:56.142 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.599866+0000", 2024-09-17T07:13:56.142 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.3659999999999998e-06 2024-09-17T07:13:56.142 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.142 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.142 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:56.142 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.209005+0000", 2024-09-17T07:13:56.142 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.60913872499999999 2024-09-17T07:13:56.142 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.143 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.143 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:13:56.143 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.209072+0000", 2024-09-17T07:13:56.143 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.7325999999999999e-05 2024-09-17T07:13:56.143 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.143 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.143 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,2", 2024-09-17T07:13:56.143 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.209145+0000", 2024-09-17T07:13:56.143 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.2743000000000005e-05 2024-09-17T07:13:56.143 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.143 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.143 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:13:56.144 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.211831+0000", 2024-09-17T07:13:56.144 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0026865859999999999 2024-09-17T07:13:56.144 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.144 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.144 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:13:56.144 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.212951+0000", 2024-09-17T07:13:56.144 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.001119718 2024-09-17T07:13:56.144 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.144 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.144 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:13:56.144 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.235636+0000", 2024-09-17T07:13:56.144 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.022684708000000001 2024-09-17T07:13:56.145 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.145 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.145 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:13:56.145 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.235822+0000", 2024-09-17T07:13:56.145 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00018626999999999999 2024-09-17T07:13:56.145 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.145 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.145 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:13:56.145 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.235868+0000", 2024-09-17T07:13:56.145 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.5720000000000003e-05 2024-09-17T07:13:56.145 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:56.145 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:13:56.146 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:56.146 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.146 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.146 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:90966 7.12 7:4d208069:::benchmark_data_smithi100_78556_object90965:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-17T07:13:56.146 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:05:38.605986+0000", 2024-09-17T07:13:56.146 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 497.44838357600003, 2024-09-17T07:13:56.146 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.63002641199999998, 2024-09-17T07:13:56.146 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:13:56.146 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:13:56.146 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:13:56.146 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:13:56.146 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:13:56.146 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 90966 2024-09-17T07:13:56.147 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.147 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:13:56.147 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.147 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:13:56.147 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.605986+0000", 2024-09-17T07:13:56.147 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:56.147 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.147 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.147 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:13:56.147 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.605986+0000", 2024-09-17T07:13:56.147 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:56.147 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.147 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.148 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:13:56.148 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.605985+0000", 2024-09-17T07:13:56.148 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:13:56.148 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.148 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.148 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:13:56.148 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.606057+0000", 2024-09-17T07:13:56.148 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.2602999999999996e-05 2024-09-17T07:13:56.148 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.148 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.149 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:13:56.149 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.606071+0000", 2024-09-17T07:13:56.149 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.4341999999999999e-05 2024-09-17T07:13:56.149 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.149 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.149 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:13:56.149 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.606078+0000", 2024-09-17T07:13:56.149 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.5180000000000001e-06 2024-09-17T07:13:56.150 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.150 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.150 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:56.150 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.606133+0000", 2024-09-17T07:13:56.150 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.5234e-05 2024-09-17T07:13:56.150 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.150 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.150 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:13:56.150 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.606145+0000", 2024-09-17T07:13:56.150 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.2072000000000001e-05 2024-09-17T07:13:56.150 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.150 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.150 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:56.151 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.209827+0000", 2024-09-17T07:13:56.151 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.60368213999999998 2024-09-17T07:13:56.151 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.151 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.151 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:13:56.151 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.209896+0000", 2024-09-17T07:13:56.151 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.8246999999999999e-05 2024-09-17T07:13:56.151 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.151 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.151 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,2", 2024-09-17T07:13:56.151 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.210025+0000", 2024-09-17T07:13:56.151 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00012941999999999999 2024-09-17T07:13:56.152 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.152 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.152 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:13:56.152 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.212650+0000", 2024-09-17T07:13:56.152 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00262496 2024-09-17T07:13:56.152 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.152 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.152 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:13:56.152 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.217625+0000", 2024-09-17T07:13:56.152 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0049752490000000002 2024-09-17T07:13:56.152 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.152 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.153 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:13:56.153 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.235925+0000", 2024-09-17T07:13:56.153 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.018299536000000002 2024-09-17T07:13:56.153 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.153 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.153 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:13:56.153 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.235974+0000", 2024-09-17T07:13:56.153 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.8662999999999998e-05 2024-09-17T07:13:56.153 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.153 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.153 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:13:56.153 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.236013+0000", 2024-09-17T07:13:56.154 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.9050000000000001e-05 2024-09-17T07:13:56.154 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:56.154 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:13:56.154 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:56.154 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.154 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.154 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:91033 7.12 7:4d9e7439:::benchmark_data_smithi100_78556_object91032:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-17T07:13:56.154 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:05:38.703237+0000", 2024-09-17T07:13:56.154 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 497.351132264, 2024-09-17T07:13:56.154 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.53285939400000004, 2024-09-17T07:13:56.154 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:13:56.154 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:13:56.155 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:13:56.155 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:13:56.155 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:13:56.155 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 91033 2024-09-17T07:13:56.155 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.155 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:13:56.155 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.155 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:13:56.155 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.703237+0000", 2024-09-17T07:13:56.155 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:56.155 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.155 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.156 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:13:56.156 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.703237+0000", 2024-09-17T07:13:56.156 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:56.156 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.156 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.156 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:13:56.156 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.703236+0000", 2024-09-17T07:13:56.156 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999981 2024-09-17T07:13:56.156 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.156 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.156 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:13:56.156 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.703290+0000", 2024-09-17T07:13:56.157 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.4296000000000003e-05 2024-09-17T07:13:56.157 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.157 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.157 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:13:56.157 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.703306+0000", 2024-09-17T07:13:56.157 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.5642999999999999e-05 2024-09-17T07:13:56.157 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.157 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.157 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:13:56.158 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.703312+0000", 2024-09-17T07:13:56.158 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.4300000000000003e-06 2024-09-17T07:13:56.159 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.159 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.159 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:56.159 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.703362+0000", 2024-09-17T07:13:56.159 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.9867000000000002e-05 2024-09-17T07:13:56.159 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.159 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.159 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:13:56.159 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.703371+0000", 2024-09-17T07:13:56.159 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.9420000000000001e-06 2024-09-17T07:13:56.159 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.159 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.160 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:56.160 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.210878+0000", 2024-09-17T07:13:56.160 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.50750747399999996 2024-09-17T07:13:56.160 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.160 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.160 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:13:56.160 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.210980+0000", 2024-09-17T07:13:56.160 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00010117 2024-09-17T07:13:56.160 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.160 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.160 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,2", 2024-09-17T07:13:56.160 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.211100+0000", 2024-09-17T07:13:56.161 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000120254 2024-09-17T07:13:56.161 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.161 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.161 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:13:56.161 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.212738+0000", 2024-09-17T07:13:56.161 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.001638245 2024-09-17T07:13:56.161 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.161 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.161 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:13:56.161 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.217672+0000", 2024-09-17T07:13:56.161 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0049344089999999998 2024-09-17T07:13:56.161 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.162 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.162 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:13:56.162 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.236041+0000", 2024-09-17T07:13:56.162 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.018368922999999999 2024-09-17T07:13:56.162 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.162 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.162 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:13:56.162 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.236077+0000", 2024-09-17T07:13:56.162 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.5235999999999999e-05 2024-09-17T07:13:56.162 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.162 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.162 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:13:56.162 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.236097+0000", 2024-09-17T07:13:56.163 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.0236000000000001e-05 2024-09-17T07:13:56.163 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:56.163 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:13:56.163 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:56.163 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.163 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.163 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:91055 7.12 7:4bab7a93:::benchmark_data_smithi100_78556_object91054:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-17T07:13:56.163 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:05:38.799197+0000", 2024-09-17T07:13:56.163 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 497.25517266000003, 2024-09-17T07:13:56.163 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.43702168200000002, 2024-09-17T07:13:56.163 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:13:56.163 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:13:56.163 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:13:56.164 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:13:56.164 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:13:56.164 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 91055 2024-09-17T07:13:56.164 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.164 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:13:56.164 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.164 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:13:56.164 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.799197+0000", 2024-09-17T07:13:56.164 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:56.164 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.164 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.164 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:13:56.164 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.799197+0000", 2024-09-17T07:13:56.165 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:56.165 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.165 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.165 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:13:56.165 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.799195+0000", 2024-09-17T07:13:56.165 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999976 2024-09-17T07:13:56.165 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.165 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.165 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:13:56.165 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.799242+0000", 2024-09-17T07:13:56.165 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.6836999999999999e-05 2024-09-17T07:13:56.165 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.166 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.166 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:13:56.166 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.799262+0000", 2024-09-17T07:13:56.166 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.0701000000000001e-05 2024-09-17T07:13:56.166 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.166 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.166 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:13:56.166 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.799272+0000", 2024-09-17T07:13:56.166 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.7280000000000007e-06 2024-09-17T07:13:56.166 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.166 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.166 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:56.166 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.799342+0000", 2024-09-17T07:13:56.167 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.9871999999999998e-05 2024-09-17T07:13:56.167 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.167 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.167 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:13:56.167 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.799356+0000", 2024-09-17T07:13:56.167 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.3816e-05 2024-09-17T07:13:56.167 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.167 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.167 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:56.167 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.211858+0000", 2024-09-17T07:13:56.167 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.41250220100000001 2024-09-17T07:13:56.167 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.167 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.168 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:13:56.168 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.211924+0000", 2024-09-17T07:13:56.168 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.5924000000000001e-05 2024-09-17T07:13:56.168 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.168 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.168 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,2", 2024-09-17T07:13:56.168 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.212011+0000", 2024-09-17T07:13:56.168 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.7026000000000003e-05 2024-09-17T07:13:56.168 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.168 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.168 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:13:56.168 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.212924+0000", 2024-09-17T07:13:56.169 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00091303 2024-09-17T07:13:56.169 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.169 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.169 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:13:56.169 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.217747+0000", 2024-09-17T07:13:56.169 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0048232259999999999 2024-09-17T07:13:56.169 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.169 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.169 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:13:56.169 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.236140+0000", 2024-09-17T07:13:56.169 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.018392737999999999 2024-09-17T07:13:56.169 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.170 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.170 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:13:56.170 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.236182+0000", 2024-09-17T07:13:56.170 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.2129999999999998e-05 2024-09-17T07:13:56.170 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.170 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.170 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:13:56.170 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.236219+0000", 2024-09-17T07:13:56.170 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.667e-05 2024-09-17T07:13:56.170 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:56.170 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:13:56.170 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:56.170 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.171 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.171 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:91058 7.12 7:4ef33ee8:::benchmark_data_smithi100_78556_object91057:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-17T07:13:56.171 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:05:38.802806+0000", 2024-09-17T07:13:56.171 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 497.25156383900003, 2024-09-17T07:13:56.171 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.43349131200000002, 2024-09-17T07:13:56.171 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:13:56.171 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:13:56.171 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:13:56.171 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:13:56.171 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:13:56.171 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 91058 2024-09-17T07:13:56.171 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.172 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:13:56.172 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.172 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:13:56.172 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.802806+0000", 2024-09-17T07:13:56.172 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:56.172 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.172 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.172 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:13:56.172 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.802806+0000", 2024-09-17T07:13:56.172 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:56.172 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.172 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.173 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:13:56.173 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.802804+0000", 2024-09-17T07:13:56.173 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999981 2024-09-17T07:13:56.173 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.173 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.173 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:13:56.173 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.802859+0000", 2024-09-17T07:13:56.173 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.5124000000000003e-05 2024-09-17T07:13:56.173 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.173 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.173 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:13:56.173 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.802877+0000", 2024-09-17T07:13:56.174 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.8198000000000001e-05 2024-09-17T07:13:56.174 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.174 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.174 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:13:56.174 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.802886+0000", 2024-09-17T07:13:56.174 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.9029999999999993e-06 2024-09-17T07:13:56.174 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.174 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.174 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:56.174 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.802943+0000", 2024-09-17T07:13:56.174 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.6873999999999998e-05 2024-09-17T07:13:56.174 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.174 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.175 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:13:56.175 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.802957+0000", 2024-09-17T07:13:56.175 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.3492000000000001e-05 2024-09-17T07:13:56.175 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.175 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.175 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:56.175 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.212973+0000", 2024-09-17T07:13:56.175 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.41001628800000001 2024-09-17T07:13:56.175 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.175 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.175 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:13:56.175 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.213043+0000", 2024-09-17T07:13:56.176 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.9663e-05 2024-09-17T07:13:56.176 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.176 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.176 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,2", 2024-09-17T07:13:56.176 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.213122+0000", 2024-09-17T07:13:56.176 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.9629999999999995e-05 2024-09-17T07:13:56.176 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.176 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.176 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:13:56.176 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.217682+0000", 2024-09-17T07:13:56.176 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.004559479 2024-09-17T07:13:56.176 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.177 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.177 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:13:56.177 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.217785+0000", 2024-09-17T07:13:56.177 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000103597 2024-09-17T07:13:56.177 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.177 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.177 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:13:56.177 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.236252+0000", 2024-09-17T07:13:56.177 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.018466372000000002 2024-09-17T07:13:56.177 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.177 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.177 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:13:56.178 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.236277+0000", 2024-09-17T07:13:56.178 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.4828999999999999e-05 2024-09-17T07:13:56.178 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.178 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.178 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:13:56.178 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.236297+0000", 2024-09-17T07:13:56.178 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.0676999999999999e-05 2024-09-17T07:13:56.178 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:56.178 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:13:56.178 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:56.178 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.178 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.179 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:91075 7.12 7:4be87e86:::benchmark_data_smithi100_78556_object91074:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-17T07:13:56.179 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:05:38.831097+0000", 2024-09-17T07:13:56.179 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 497.223272662, 2024-09-17T07:13:56.179 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.40532509100000003, 2024-09-17T07:13:56.179 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:13:56.179 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:13:56.179 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:13:56.179 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:13:56.179 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:13:56.179 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 91075 2024-09-17T07:13:56.179 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.179 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:13:56.179 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.180 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:13:56.180 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.831097+0000", 2024-09-17T07:13:56.180 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:56.180 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.180 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.180 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:13:56.180 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.831097+0000", 2024-09-17T07:13:56.180 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:56.180 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.180 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.180 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:13:56.180 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.831096+0000", 2024-09-17T07:13:56.181 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.999999 2024-09-17T07:13:56.181 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.181 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.181 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:13:56.181 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.831150+0000", 2024-09-17T07:13:56.181 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.4138999999999998e-05 2024-09-17T07:13:56.181 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.181 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.181 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:13:56.181 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.831161+0000", 2024-09-17T07:13:56.181 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.1005000000000001e-05 2024-09-17T07:13:56.181 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.181 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.182 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:13:56.182 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.831166+0000", 2024-09-17T07:13:56.182 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.7060000000000003e-06 2024-09-17T07:13:56.182 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.182 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.182 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:56.182 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.831222+0000", 2024-09-17T07:13:56.182 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.5866999999999999e-05 2024-09-17T07:13:56.182 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.182 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.182 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:13:56.182 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.831234+0000", 2024-09-17T07:13:56.183 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.2119e-05 2024-09-17T07:13:56.183 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.183 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.183 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:56.183 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.213810+0000", 2024-09-17T07:13:56.183 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.38257537800000002 2024-09-17T07:13:56.183 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.183 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.183 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:13:56.183 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.213967+0000", 2024-09-17T07:13:56.183 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00015702199999999999 2024-09-17T07:13:56.183 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.183 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.184 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,2", 2024-09-17T07:13:56.184 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.214099+0000", 2024-09-17T07:13:56.184 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00013254900000000001 2024-09-17T07:13:56.184 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.184 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.184 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:13:56.184 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.217687+0000", 2024-09-17T07:13:56.184 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.003588141 2024-09-17T07:13:56.184 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.184 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.184 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:13:56.184 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.221425+0000", 2024-09-17T07:13:56.185 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0037375970000000001 2024-09-17T07:13:56.185 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.185 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.185 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:13:56.185 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.236337+0000", 2024-09-17T07:13:56.185 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.01491165 2024-09-17T07:13:56.185 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.185 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.186 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:13:56.186 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.236379+0000", 2024-09-17T07:13:56.186 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.2904999999999997e-05 2024-09-17T07:13:56.186 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.186 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.186 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:13:56.186 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.236422+0000", 2024-09-17T07:13:56.186 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.2760999999999999e-05 2024-09-17T07:13:56.186 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:56.186 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:13:56.187 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:56.187 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.187 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.187 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:91109 7.12 7:4c36141b:::benchmark_data_smithi100_78556_object91108:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-17T07:13:56.187 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:05:38.908421+0000", 2024-09-17T07:13:56.187 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 497.14594855899998, 2024-09-17T07:13:56.187 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.32807808900000002, 2024-09-17T07:13:56.187 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:13:56.187 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:13:56.187 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:13:56.187 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:13:56.187 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:13:56.188 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 91109 2024-09-17T07:13:56.188 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.188 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:13:56.188 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.188 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:13:56.188 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.908421+0000", 2024-09-17T07:13:56.188 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:56.188 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.188 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.188 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:13:56.188 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.908421+0000", 2024-09-17T07:13:56.188 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:56.189 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.189 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.189 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:13:56.189 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.908420+0000", 2024-09-17T07:13:56.189 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.999999 2024-09-17T07:13:56.189 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.189 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.189 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:13:56.189 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.908466+0000", 2024-09-17T07:13:56.189 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.6315e-05 2024-09-17T07:13:56.189 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.189 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.190 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:13:56.190 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.908477+0000", 2024-09-17T07:13:56.190 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0127e-05 2024-09-17T07:13:56.190 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.190 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.190 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:13:56.190 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.908484+0000", 2024-09-17T07:13:56.190 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.5190000000000003e-06 2024-09-17T07:13:56.190 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.190 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.190 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:56.190 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.908533+0000", 2024-09-17T07:13:56.191 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.8971000000000003e-05 2024-09-17T07:13:56.191 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.191 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.191 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:13:56.191 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:38.908541+0000", 2024-09-17T07:13:56.191 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.7190000000000001e-06 2024-09-17T07:13:56.191 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.191 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.191 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:56.191 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.215081+0000", 2024-09-17T07:13:56.191 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.30653980199999997 2024-09-17T07:13:56.191 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.192 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.192 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:13:56.192 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.215194+0000", 2024-09-17T07:13:56.192 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000113596 2024-09-17T07:13:56.192 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.192 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.192 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,2", 2024-09-17T07:13:56.192 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.215318+0000", 2024-09-17T07:13:56.192 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00012414699999999999 2024-09-17T07:13:56.192 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.192 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.192 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:13:56.192 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.217693+0000", 2024-09-17T07:13:56.193 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0023749029999999998 2024-09-17T07:13:56.193 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.193 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.193 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:13:56.193 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.223267+0000", 2024-09-17T07:13:56.193 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00557384 2024-09-17T07:13:56.193 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.193 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.193 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:13:56.193 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.236456+0000", 2024-09-17T07:13:56.193 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.013188942 2024-09-17T07:13:56.193 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.194 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.194 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:13:56.194 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.236479+0000", 2024-09-17T07:13:56.194 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.2785999999999998e-05 2024-09-17T07:13:56.194 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.194 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.194 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:13:56.194 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:39.236499+0000", 2024-09-17T07:13:56.194 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.0537999999999999e-05 2024-09-17T07:13:56.194 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:56.194 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:13:56.194 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:56.194 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.195 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.195 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:94245 7.7 7:e1cebfa6:::benchmark_data_smithi100_78556_object94244:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-17T07:13:56.195 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:05:42.675680+0000", 2024-09-17T07:13:56.195 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 493.37869014299997, 2024-09-17T07:13:56.195 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.21869654899999999, 2024-09-17T07:13:56.195 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:13:56.195 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:13:56.195 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:13:56.195 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:13:56.195 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:13:56.195 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 94245 2024-09-17T07:13:56.195 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.195 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:13:56.196 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.196 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:13:56.196 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:42.675680+0000", 2024-09-17T07:13:56.196 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:56.196 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.196 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.196 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:13:56.196 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:42.675680+0000", 2024-09-17T07:13:56.196 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:56.196 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.196 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.196 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:13:56.197 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:42.675678+0000", 2024-09-17T07:13:56.197 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:13:56.197 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.197 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.197 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:13:56.197 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:42.675741+0000", 2024-09-17T07:13:56.197 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.2485999999999998e-05 2024-09-17T07:13:56.197 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.197 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.197 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:13:56.197 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:42.675753+0000", 2024-09-17T07:13:56.197 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.269e-05 2024-09-17T07:13:56.197 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.198 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.198 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:13:56.198 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:42.675761+0000", 2024-09-17T07:13:56.198 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.2690000000000002e-06 2024-09-17T07:13:56.198 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.198 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.198 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:56.198 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:42.675812+0000", 2024-09-17T07:13:56.198 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.1041999999999997e-05 2024-09-17T07:13:56.199 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.199 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.199 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:13:56.199 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:42.675893+0000", 2024-09-17T07:13:56.199 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.0958000000000007e-05 2024-09-17T07:13:56.199 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.199 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.199 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,3", 2024-09-17T07:13:56.199 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:42.675975+0000", 2024-09-17T07:13:56.199 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.2306999999999996e-05 2024-09-17T07:13:56.199 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.199 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.200 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:13:56.200 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:42.677435+0000", 2024-09-17T07:13:56.200 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0014597099999999999 2024-09-17T07:13:56.200 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.200 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.200 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:13:56.200 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:42.679293+0000", 2024-09-17T07:13:56.200 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.001858405 2024-09-17T07:13:56.200 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.200 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.200 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:13:56.200 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:42.894281+0000", 2024-09-17T07:13:56.200 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.214988502 2024-09-17T07:13:56.201 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.201 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.201 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:13:56.201 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:42.894335+0000", 2024-09-17T07:13:56.201 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.3786000000000003e-05 2024-09-17T07:13:56.201 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.201 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.201 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:13:56.201 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:42.894376+0000", 2024-09-17T07:13:56.201 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.0914999999999997e-05 2024-09-17T07:13:56.201 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:56.202 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:13:56.202 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:56.202 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.202 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.202 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:94259 7.7 7:e0fabcbf:::benchmark_data_smithi100_78556_object94258:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-17T07:13:56.202 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:05:42.686358+0000", 2024-09-17T07:13:56.202 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 493.36801178100001, 2024-09-17T07:13:56.202 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.21047728800000001, 2024-09-17T07:13:56.202 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:13:56.202 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:13:56.202 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:13:56.202 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:13:56.202 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:13:56.203 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 94259 2024-09-17T07:13:56.203 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.203 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:13:56.203 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.203 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:13:56.203 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:42.686358+0000", 2024-09-17T07:13:56.203 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:56.203 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.203 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.203 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:13:56.203 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:42.686358+0000", 2024-09-17T07:13:56.203 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:56.203 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.204 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.204 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:13:56.204 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:42.686357+0000", 2024-09-17T07:13:56.204 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:13:56.204 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.204 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.204 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:13:56.204 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:42.686416+0000", 2024-09-17T07:13:56.204 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.9768000000000001e-05 2024-09-17T07:13:56.204 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.204 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.204 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:13:56.205 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:42.686428+0000", 2024-09-17T07:13:56.205 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.1208e-05 2024-09-17T07:13:56.205 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.205 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.205 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:13:56.205 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:42.686433+0000", 2024-09-17T07:13:56.205 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.5400000000000003e-06 2024-09-17T07:13:56.205 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.205 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.205 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:56.205 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:42.686472+0000", 2024-09-17T07:13:56.205 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.8516999999999997e-05 2024-09-17T07:13:56.206 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.206 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.206 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:13:56.206 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:42.686565+0000", 2024-09-17T07:13:56.206 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.3184999999999996e-05 2024-09-17T07:13:56.206 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.206 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.206 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,3", 2024-09-17T07:13:56.206 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:42.686663+0000", 2024-09-17T07:13:56.206 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.7617999999999999e-05 2024-09-17T07:13:56.207 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.207 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.207 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:13:56.207 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:42.687928+0000", 2024-09-17T07:13:56.207 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.001265146 2024-09-17T07:13:56.207 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.207 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.207 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:13:56.207 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:42.837296+0000", 2024-09-17T07:13:56.207 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.149368642 2024-09-17T07:13:56.207 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.207 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.208 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:13:56.208 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:42.896752+0000", 2024-09-17T07:13:56.208 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.059455988000000001 2024-09-17T07:13:56.208 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.208 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.208 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:13:56.208 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:42.896802+0000", 2024-09-17T07:13:56.208 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.0074000000000002e-05 2024-09-17T07:13:56.208 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.208 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.208 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:13:56.208 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:42.896835+0000", 2024-09-17T07:13:56.209 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.2904000000000002e-05 2024-09-17T07:13:56.209 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:56.209 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:13:56.209 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:56.209 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.209 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.209 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.14805.0:94479 7.7 7:e5cde295:::benchmark_data_smithi100_78556_object94478:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-17T07:13:56.209 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:05:43.059491+0000", 2024-09-17T07:13:56.209 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 492.99487916200002, 2024-09-17T07:13:56.209 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.197441328, 2024-09-17T07:13:56.209 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:13:56.209 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:13:56.209 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:13:56.210 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.14805", 2024-09-17T07:13:56.210 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:13:56.210 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 94479 2024-09-17T07:13:56.210 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.210 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:13:56.210 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.210 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:13:56.210 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:43.059491+0000", 2024-09-17T07:13:56.210 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:56.210 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.210 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.210 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:13:56.211 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:43.059491+0000", 2024-09-17T07:13:56.211 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:13:56.211 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.211 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.211 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:13:56.211 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:43.059489+0000", 2024-09-17T07:13:56.211 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:13:56.211 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.211 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.211 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:13:56.211 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:43.059547+0000", 2024-09-17T07:13:56.211 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.8136e-05 2024-09-17T07:13:56.211 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.212 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.212 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:13:56.212 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:43.059562+0000", 2024-09-17T07:13:56.212 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.4932999999999999e-05 2024-09-17T07:13:56.212 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.212 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.212 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:13:56.212 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:43.059570+0000", 2024-09-17T07:13:56.212 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.0190000000000006e-06 2024-09-17T07:13:56.212 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.212 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.212 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:13:56.212 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:43.059621+0000", 2024-09-17T07:13:56.213 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.0488999999999997e-05 2024-09-17T07:13:56.213 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.213 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.213 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:13:56.213 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:43.059732+0000", 2024-09-17T07:13:56.213 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000111722 2024-09-17T07:13:56.213 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.213 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.213 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,3", 2024-09-17T07:13:56.213 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:43.059860+0000", 2024-09-17T07:13:56.213 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00012725599999999999 2024-09-17T07:13:56.213 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.214 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.214 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:13:56.214 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:43.061565+0000", 2024-09-17T07:13:56.214 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.001705834 2024-09-17T07:13:56.214 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.214 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.214 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:13:56.214 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:43.065093+0000", 2024-09-17T07:13:56.214 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0035274249999999998 2024-09-17T07:13:56.214 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.214 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.214 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:13:56.214 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:43.256840+0000", 2024-09-17T07:13:56.215 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.191746839 2024-09-17T07:13:56.215 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.215 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.215 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:13:56.215 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:43.256894+0000", 2024-09-17T07:13:56.215 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.3903999999999998e-05 2024-09-17T07:13:56.215 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:13:56.215 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:13:56.215 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:13:56.215 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:05:43.256932+0000", 2024-09-17T07:13:56.215 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.8373e-05 2024-09-17T07:13:56.215 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:56.216 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:13:56.216 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:56.216 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:13:56.216 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:13:56.216 INFO:teuthology.orchestra.run.smithi135.stdout:} 2024-09-17T07:13:56.221 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:13:56.254 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:56.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:56.255 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:13:56.355 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:13:56.387 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:56.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:56.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:13:56.489 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:13:56.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:56.531 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:56.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:13:56.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:56 smithi135 ceph-mon[33238]: 4.1f scrub starts 2024-09-17T07:13:56.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:56 smithi135 ceph-mon[33238]: 4.1f scrub ok 2024-09-17T07:13:56.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:56 smithi135 ceph-mon[33238]: 2.12 scrub starts 2024-09-17T07:13:56.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:56 smithi135 ceph-mon[33238]: 2.12 scrub ok 2024-09-17T07:13:56.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:56 smithi100 ceph-mon[29092]: 4.1f scrub starts 2024-09-17T07:13:56.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:56 smithi100 ceph-mon[29092]: 4.1f scrub ok 2024-09-17T07:13:56.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:56 smithi100 ceph-mon[29092]: 2.12 scrub starts 2024-09-17T07:13:56.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:56 smithi100 ceph-mon[29092]: 2.12 scrub ok 2024-09-17T07:13:56.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:56 smithi100 ceph-mon[39462]: 4.1f scrub starts 2024-09-17T07:13:56.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:56 smithi100 ceph-mon[39462]: 4.1f scrub ok 2024-09-17T07:13:56.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:56 smithi100 ceph-mon[39462]: 2.12 scrub starts 2024-09-17T07:13:56.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:56 smithi100 ceph-mon[39462]: 2.12 scrub ok 2024-09-17T07:13:56.633 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:13:56.670 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:56.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:56.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:13:56.772 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:13:56.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:56.894 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:56.895 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:13:56.995 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:13:57.024 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:57.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:57.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:13:57.126 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:13:57.153 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:57.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:57.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:13:57.183 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-17T07:13:57.255 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:13:57.321 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:57.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:57.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:13:57.423 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:13:57.481 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:57.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:57.482 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:13:57.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:57 smithi135 ceph-mon[33238]: 9.b scrub starts 2024-09-17T07:13:57.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:57 smithi135 ceph-mon[33238]: 9.a scrub starts 2024-09-17T07:13:57.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:57 smithi135 ceph-mon[33238]: pgmap v528: 148 pgs: 148 active+clean; 7.8 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 639 B/s rd, 5.9 MiB/s wr, 719 op/s 2024-09-17T07:13:57.566 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:13:57.583 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:13:57.610 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:57.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:57.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:13:57.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:57 smithi100 ceph-mon[29092]: 9.b scrub starts 2024-09-17T07:13:57.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:57 smithi100 ceph-mon[29092]: 9.a scrub starts 2024-09-17T07:13:57.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:57 smithi100 ceph-mon[29092]: pgmap v528: 148 pgs: 148 active+clean; 7.8 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 639 B/s rd, 5.9 MiB/s wr, 719 op/s 2024-09-17T07:13:57.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:57 smithi100 ceph-mon[39462]: 9.b scrub starts 2024-09-17T07:13:57.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:57 smithi100 ceph-mon[39462]: 9.a scrub starts 2024-09-17T07:13:57.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:57 smithi100 ceph-mon[39462]: pgmap v528: 148 pgs: 148 active+clean; 7.8 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 639 B/s rd, 5.9 MiB/s wr, 719 op/s 2024-09-17T07:13:57.712 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:13:57.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:57.748 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:57.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:13:57.850 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:13:57.888 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:57.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:57.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:13:57.990 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:13:58.025 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:58.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:58.026 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:13:58.069 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:13:58.070 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:13:58.094 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1: {} 2024-09-17T07:13:58.095 INFO:teuthology.orchestra.run.smithi100.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:13:58.121 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:13:58.121 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:13:58.122 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-17T07:13:58.127 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:13:58.149 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:13:58.150 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:13:58.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:58.166 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:58.166 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:13:58.179 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:13:58.179 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:13:58.213 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:13:58.214 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:13:58.245 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:13:58.245 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:13:58.268 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:13:58.279 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:13:58.279 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:13:58.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:58.314 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:58.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:13:58.414 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:13:58.452 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:58.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:58.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:13:58.554 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:13:58.585 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:58.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:58.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:13:58.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:58 smithi100 ceph-mon[29092]: 6.1c scrub starts 2024-09-17T07:13:58.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:58 smithi100 ceph-mon[29092]: 6.1c scrub ok 2024-09-17T07:13:58.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:58 smithi100 ceph-mon[29092]: 4.a scrub starts 2024-09-17T07:13:58.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:58 smithi100 ceph-mon[29092]: 4.a scrub ok 2024-09-17T07:13:58.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:58 smithi100 ceph-mon[29092]: 5.7 scrub starts 2024-09-17T07:13:58.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:58 smithi100 ceph-mon[29092]: 5.7 scrub ok 2024-09-17T07:13:58.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:58 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/377386023' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:13:58.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:58 smithi100 ceph-mon[39462]: 6.1c scrub starts 2024-09-17T07:13:58.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:58 smithi100 ceph-mon[39462]: 6.1c scrub ok 2024-09-17T07:13:58.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:58 smithi100 ceph-mon[39462]: 4.a scrub starts 2024-09-17T07:13:58.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:58 smithi100 ceph-mon[39462]: 4.a scrub ok 2024-09-17T07:13:58.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:58 smithi100 ceph-mon[39462]: 5.7 scrub starts 2024-09-17T07:13:58.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:58 smithi100 ceph-mon[39462]: 5.7 scrub ok 2024-09-17T07:13:58.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:58 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/377386023' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:13:58.687 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:13:58.714 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:58.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:58.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:13:58.726 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:58 smithi135 ceph-mon[33238]: 6.1c scrub starts 2024-09-17T07:13:58.727 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:58 smithi135 ceph-mon[33238]: 6.1c scrub ok 2024-09-17T07:13:58.727 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:58 smithi135 ceph-mon[33238]: 4.a scrub starts 2024-09-17T07:13:58.727 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:58 smithi135 ceph-mon[33238]: 4.a scrub ok 2024-09-17T07:13:58.727 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:58 smithi135 ceph-mon[33238]: 5.7 scrub starts 2024-09-17T07:13:58.727 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:58 smithi135 ceph-mon[33238]: 5.7 scrub ok 2024-09-17T07:13:58.727 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:58 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/377386023' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:13:58.816 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:13:58.839 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:58.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:58.840 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:13:58.941 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:13:58.966 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:58.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:58.967 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:13:59.043 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:13:58 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:13:59.068 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:13:59.095 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:59.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:59.097 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:13:59.109 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7] out_osds: [3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-17T07:13:59.109 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T07:13:59.110 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T07:13:59.198 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:13:59.238 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:59.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:59.239 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:13:59.340 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:13:59.380 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:59.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:59.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:13:59.482 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:13:59.513 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:59.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:59.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:13:59.615 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:13:59.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:59 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:13:59.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:13:59 smithi100 ceph-mon[29092]: pgmap v529: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.6 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 901 op/s 2024-09-17T07:13:59.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:59 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:13:59.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:13:59 smithi100 ceph-mon[39462]: pgmap v529: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.6 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 901 op/s 2024-09-17T07:13:59.660 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:59.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:59.661 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:13:59.705 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:59 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:13:59.705 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:13:59 smithi135 ceph-mon[33238]: pgmap v529: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.6 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 901 op/s 2024-09-17T07:13:59.762 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:13:59.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:59.803 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:13:59.803 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:13:59.904 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:13:59.930 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:13:59.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:13:59.931 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:14:00.032 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:14:00.065 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:00.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:00.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:14:00.167 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:14:00.199 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:00.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:00.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:14:00.301 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:14:00.322 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:00.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:00.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:14:00.423 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:14:00.466 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:00.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:00.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:14:00.496 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:14:00.499 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:14:00.568 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:14:00.602 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:00.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:00.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:14:00.636 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:00 smithi100 ceph-mon[29092]: 9.b scrub ok 2024-09-17T07:14:00.637 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:00 smithi100 ceph-mon[29092]: 6.12 scrub starts 2024-09-17T07:14:00.637 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:00 smithi100 ceph-mon[29092]: 6.12 scrub ok 2024-09-17T07:14:00.637 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:00 smithi100 ceph-mon[29092]: 4.9 scrub starts 2024-09-17T07:14:00.637 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:00 smithi100 ceph-mon[29092]: 4.9 scrub ok 2024-09-17T07:14:00.637 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:00 smithi100 ceph-mon[29092]: 9.a scrub ok 2024-09-17T07:14:00.637 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:00 smithi100 ceph-mon[39462]: 9.b scrub ok 2024-09-17T07:14:00.637 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:00 smithi100 ceph-mon[39462]: 6.12 scrub starts 2024-09-17T07:14:00.637 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:00 smithi100 ceph-mon[39462]: 6.12 scrub ok 2024-09-17T07:14:00.638 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:00 smithi100 ceph-mon[39462]: 4.9 scrub starts 2024-09-17T07:14:00.638 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:00 smithi100 ceph-mon[39462]: 4.9 scrub ok 2024-09-17T07:14:00.638 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:00 smithi100 ceph-mon[39462]: 9.a scrub ok 2024-09-17T07:14:00.703 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:14:00.738 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:00.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:00.739 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:14:00.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:00 smithi135 ceph-mon[33238]: 9.b scrub ok 2024-09-17T07:14:00.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:00 smithi135 ceph-mon[33238]: 6.12 scrub starts 2024-09-17T07:14:00.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:00 smithi135 ceph-mon[33238]: 6.12 scrub ok 2024-09-17T07:14:00.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:00 smithi135 ceph-mon[33238]: 4.9 scrub starts 2024-09-17T07:14:00.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:00 smithi135 ceph-mon[33238]: 4.9 scrub ok 2024-09-17T07:14:00.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:00 smithi135 ceph-mon[33238]: 9.a scrub ok 2024-09-17T07:14:00.839 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:14:00.879 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:00.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:00.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:14:00.982 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:14:01.003 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:01.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:01.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:14:01.105 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:14:01.144 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:01.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:01.146 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:14:01.246 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:14:01.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:01.276 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:01.276 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:14:01.377 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:14:01.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:01.411 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:01.411 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:14:01.512 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:14:01.552 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:01.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:01.554 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:14:01.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:01 smithi100 ceph-mon[29092]: 5.1d scrub starts 2024-09-17T07:14:01.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:01 smithi100 ceph-mon[29092]: 5.1d scrub ok 2024-09-17T07:14:01.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:01 smithi100 ceph-mon[29092]: 4.6 deep-scrub starts 2024-09-17T07:14:01.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:01 smithi100 ceph-mon[29092]: 4.6 deep-scrub ok 2024-09-17T07:14:01.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:01 smithi100 ceph-mon[29092]: 5.f scrub starts 2024-09-17T07:14:01.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:01 smithi100 ceph-mon[29092]: 5.f scrub ok 2024-09-17T07:14:01.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:01 smithi100 ceph-mon[29092]: pgmap v530: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.5 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.04k op/s 2024-09-17T07:14:01.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:01 smithi100 ceph-mon[39462]: 5.1d scrub starts 2024-09-17T07:14:01.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:01 smithi100 ceph-mon[39462]: 5.1d scrub ok 2024-09-17T07:14:01.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:01 smithi100 ceph-mon[39462]: 4.6 deep-scrub starts 2024-09-17T07:14:01.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:01 smithi100 ceph-mon[39462]: 4.6 deep-scrub ok 2024-09-17T07:14:01.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:01 smithi100 ceph-mon[39462]: 5.f scrub starts 2024-09-17T07:14:01.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:01 smithi100 ceph-mon[39462]: 5.f scrub ok 2024-09-17T07:14:01.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:01 smithi100 ceph-mon[39462]: pgmap v530: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.5 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.04k op/s 2024-09-17T07:14:01.654 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:14:01.693 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:01.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:01.694 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:14:01.774 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:01 smithi135 ceph-mon[33238]: 5.1d scrub starts 2024-09-17T07:14:01.775 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:01 smithi135 ceph-mon[33238]: 5.1d scrub ok 2024-09-17T07:14:01.775 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:01 smithi135 ceph-mon[33238]: 4.6 deep-scrub starts 2024-09-17T07:14:01.775 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:01 smithi135 ceph-mon[33238]: 4.6 deep-scrub ok 2024-09-17T07:14:01.775 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:01 smithi135 ceph-mon[33238]: 5.f scrub starts 2024-09-17T07:14:01.775 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:01 smithi135 ceph-mon[33238]: 5.f scrub ok 2024-09-17T07:14:01.775 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:01 smithi135 ceph-mon[33238]: pgmap v530: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.5 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.04k op/s 2024-09-17T07:14:01.796 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:14:01.835 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:01.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:01.836 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:14:01.937 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:14:01.966 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:01.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:01.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:14:02.068 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:14:02.096 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:02.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:02.097 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:14:02.106 INFO:teuthology.orchestra.run.smithi135.stdout:ERROR: (22) Invalid argument 2024-09-17T07:14:02.107 INFO:teuthology.orchestra.run.smithi135.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:14:02.198 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:14:02.228 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:02.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:02.229 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:14:02.330 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:14:02.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:02.364 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:02.365 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:14:02.431 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:14:02.468 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:14:02.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:02.513 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:02.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:14:02.614 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:14:02.656 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:02.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:02.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:14:02.758 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:14:02.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:02.791 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:02.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:14:02.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:02 smithi135 ceph-mon[33238]: 9.0 scrub starts 2024-09-17T07:14:02.869 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-17T07:14:02.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:02 smithi100 ceph-mon[29092]: 9.0 scrub starts 2024-09-17T07:14:02.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:02 smithi100 ceph-mon[39462]: 9.0 scrub starts 2024-09-17T07:14:02.892 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:14:02.966 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:02.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:02.967 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:14:03.068 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:14:03.136 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:03.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:03.137 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:14:03.237 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:14:03.249 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:14:03.284 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:03.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:03.286 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:14:03.387 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:14:03.421 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:03.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:03.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:14:03.522 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:14:03.553 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:03.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:03.554 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:14:03.655 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:14:03.691 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:03.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:03.692 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:14:03.793 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:14:03.831 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:03.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:03.833 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:14:03.934 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:14:03.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:03.967 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:03.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:14:04.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:03 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:14:04.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:03 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:14:04.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:03 smithi135 ceph-mon[33238]: 2.2 scrub starts 2024-09-17T07:14:04.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:03 smithi135 ceph-mon[33238]: 2.2 scrub ok 2024-09-17T07:14:04.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:03 smithi135 ceph-mon[33238]: pgmap v531: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.5 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.04k op/s 2024-09-17T07:14:04.068 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:14:04.098 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:04.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:04.099 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:14:04.200 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:14:04.232 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:03 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:14:04.232 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:03 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:14:04.232 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:03 smithi100 ceph-mon[29092]: 2.2 scrub starts 2024-09-17T07:14:04.232 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:03 smithi100 ceph-mon[29092]: 2.2 scrub ok 2024-09-17T07:14:04.233 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:03 smithi100 ceph-mon[29092]: pgmap v531: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.5 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.04k op/s 2024-09-17T07:14:04.233 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:03 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:14:04.233 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:03 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:14:04.233 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:03 smithi100 ceph-mon[39462]: 2.2 scrub starts 2024-09-17T07:14:04.233 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:03 smithi100 ceph-mon[39462]: 2.2 scrub ok 2024-09-17T07:14:04.234 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:03 smithi100 ceph-mon[39462]: pgmap v531: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.5 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.04k op/s 2024-09-17T07:14:04.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:04.243 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:04.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:14:04.344 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:14:04.372 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:04.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:04.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:14:04.474 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:14:04.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:04.512 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:04.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:14:04.614 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:14:04.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:04.652 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:04.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:14:04.753 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:14:04.789 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:04.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:04.790 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:14:04.891 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:14:04.924 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:04.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:04.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:14:04.966 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:04 smithi135 ceph-mon[33238]: 6.0 deep-scrub starts 2024-09-17T07:14:04.966 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:04 smithi135 ceph-mon[33238]: 6.0 deep-scrub ok 2024-09-17T07:14:04.966 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:04 smithi135 ceph-mon[33238]: 6.17 scrub starts 2024-09-17T07:14:04.966 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:04 smithi135 ceph-mon[33238]: 6.17 scrub ok 2024-09-17T07:14:04.966 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:04 smithi135 ceph-mon[33238]: 9.0 scrub ok 2024-09-17T07:14:05.025 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:14:05.053 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:05.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:05.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:14:05.105 INFO:teuthology.orchestra.run.smithi135.stdout:ERROR: (22) Invalid argument 2024-09-17T07:14:05.106 INFO:teuthology.orchestra.run.smithi135.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:14:05.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:04 smithi100 ceph-mon[29092]: 6.0 deep-scrub starts 2024-09-17T07:14:05.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:04 smithi100 ceph-mon[29092]: 6.0 deep-scrub ok 2024-09-17T07:14:05.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:04 smithi100 ceph-mon[29092]: 6.17 scrub starts 2024-09-17T07:14:05.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:04 smithi100 ceph-mon[29092]: 6.17 scrub ok 2024-09-17T07:14:05.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:04 smithi100 ceph-mon[29092]: 9.0 scrub ok 2024-09-17T07:14:05.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:04 smithi100 ceph-mon[39462]: 6.0 deep-scrub starts 2024-09-17T07:14:05.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:04 smithi100 ceph-mon[39462]: 6.0 deep-scrub ok 2024-09-17T07:14:05.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:04 smithi100 ceph-mon[39462]: 6.17 scrub starts 2024-09-17T07:14:05.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:04 smithi100 ceph-mon[39462]: 6.17 scrub ok 2024-09-17T07:14:05.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:04 smithi100 ceph-mon[39462]: 9.0 scrub ok 2024-09-17T07:14:05.155 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:14:05.192 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:05.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:05.193 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:14:05.294 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:14:05.328 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:05.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:05.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:14:05.431 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:14:05.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:05.464 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:05.465 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:14:05.566 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:14:05.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:05.589 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:05.589 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:14:05.690 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:14:05.722 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:05.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:05.723 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:14:05.761 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-17T07:14:05.815 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T07:14:05.819 INFO:teuthology.orchestra.run.smithi100.stderr:dumped all 2024-09-17T07:14:05.823 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:14:05.856 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:05.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:05.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:14:05.958 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:14:05.991 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:05.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:05.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:14:06.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:05 smithi135 ceph-mon[33238]: 6.16 scrub starts 2024-09-17T07:14:06.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:05 smithi135 ceph-mon[33238]: 6.16 scrub ok 2024-09-17T07:14:06.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:05 smithi135 ceph-mon[33238]: 6.1e scrub starts 2024-09-17T07:14:06.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:05 smithi135 ceph-mon[33238]: 6.1e scrub ok 2024-09-17T07:14:06.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:05 smithi135 ceph-mon[33238]: pgmap v532: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.1 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.37k op/s 2024-09-17T07:14:06.045 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:05 smithi135 ceph-mon[33238]: 4.14 scrub starts 2024-09-17T07:14:06.045 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:05 smithi135 ceph-mon[33238]: 4.14 scrub ok 2024-09-17T07:14:06.093 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:14:06.120 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:05 smithi100 ceph-mon[29092]: 6.16 scrub starts 2024-09-17T07:14:06.120 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:05 smithi100 ceph-mon[29092]: 6.16 scrub ok 2024-09-17T07:14:06.120 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:05 smithi100 ceph-mon[29092]: 6.1e scrub starts 2024-09-17T07:14:06.121 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:05 smithi100 ceph-mon[29092]: 6.1e scrub ok 2024-09-17T07:14:06.121 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:05 smithi100 ceph-mon[29092]: pgmap v532: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.1 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.37k op/s 2024-09-17T07:14:06.121 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:05 smithi100 ceph-mon[29092]: 4.14 scrub starts 2024-09-17T07:14:06.121 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:05 smithi100 ceph-mon[29092]: 4.14 scrub ok 2024-09-17T07:14:06.121 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:14:05 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:14:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:14:06.121 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:05 smithi100 ceph-mon[39462]: 6.16 scrub starts 2024-09-17T07:14:06.122 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:05 smithi100 ceph-mon[39462]: 6.16 scrub ok 2024-09-17T07:14:06.122 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:05 smithi100 ceph-mon[39462]: 6.1e scrub starts 2024-09-17T07:14:06.122 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:05 smithi100 ceph-mon[39462]: 6.1e scrub ok 2024-09-17T07:14:06.122 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:05 smithi100 ceph-mon[39462]: pgmap v532: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.1 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.37k op/s 2024-09-17T07:14:06.122 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:05 smithi100 ceph-mon[39462]: 4.14 scrub starts 2024-09-17T07:14:06.122 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:05 smithi100 ceph-mon[39462]: 4.14 scrub ok 2024-09-17T07:14:06.146 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:06.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:06.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:14:06.162 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:14:06.248 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:14:06.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:06.284 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:06.284 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:14:06.385 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:14:06.424 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:06.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:06.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:14:06.526 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:14:06.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:06.566 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:06.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:14:06.667 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:14:06.694 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:06.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:06.696 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:14:06.771 INFO:teuthology.orchestra.run.smithi100.stdout:{"pg_ready":true,"pg_map":{"version":532,"stamp":"2024-09-17T07:14:05.250034+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7615682862,"num_objects":116394,"num_object_clones":0,"num_object_copies":349182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":116394,"num_whiteouts":0,"num_read":2050,"num_read_kb":1896,"num_write":153225,"num_write_kb":8595292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":123,"num_bytes_recovered":1773876,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75567,"ondisk_log_size":75567,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":25905800,"kb_used_data":22460688,"kb_used_omap":10,"kb_used_meta":3444981,"kb_avail":630310264,"statfs":{"total":671965249536,"available":645437710336,"internally_reserved":0,"allocated":22999744512,"data_stored":22997189643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11152,"internal_metadata":3527660656},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-1065091072,"num_objects":-16252,"num_object_clones":0,"num_object_copies":-48756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-16252,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":16252,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.891431"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":108,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:49.352413+0000","last_change":"2024-09-17T07:13:11.978300+0000","last_active":"2024-09-17T07:13:49.352413+0000","last_peered":"2024-09-17T07:13:49.352413+0000","last_clean":"2024-09-17T07:13:49.352413+0000","last_became_active":"2024-09-17T07:09:24.401344+0000","last_became_peered":"2024-09-17T07:09:24.401344+0000","last_unstale":"2024-09-17T07:13:49.352413+0000","last_undegraded":"2024-09-17T07:13:49.352413+0000","last_fullsized":"2024-09-17T07:13:49.352413+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:11.978185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:33.644728+0000","last_clean_scrub_stamp":"2024-09-17T07:13:11.978185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:30.369938+0000","scrub_duration":0.0074436570000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":269,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-17T07:13:54.566784+0000","last_change":"2024-09-17T07:13:54.566881+0000","last_active":"2024-09-17T07:13:54.566784+0000","last_peered":"2024-09-17T07:13:54.566784+0000","last_clean":"2024-09-17T07:13:54.566784+0000","last_became_active":"2024-09-17T07:02:12.006754+0000","last_became_peered":"2024-09-17T07:02:12.006754+0000","last_unstale":"2024-09-17T07:13:54.566784+0000","last_undegraded":"2024-09-17T07:13:54.566784+0000","last_fullsized":"2024-09-17T07:13:54.566784+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:54.566757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:13:54.566757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:15:14.266560+0000","scrub_duration":0.0074097110000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"242'27","reported_seq":168,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:49.343956+0000","last_change":"2024-09-17T07:12:24.639592+0000","last_active":"2024-09-17T07:13:49.343956+0000","last_peered":"2024-09-17T07:13:49.343956+0000","last_clean":"2024-09-17T07:13:49.343956+0000","last_became_active":"2024-09-17T07:09:24.402042+0000","last_became_peered":"2024-09-17T07:09:24.402042+0000","last_unstale":"2024-09-17T07:13:49.343956+0000","last_undegraded":"2024-09-17T07:13:49.343956+0000","last_fullsized":"2024-09-17T07:13:49.343956+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'27","last_scrub_stamp":"2024-09-17T07:12:24.639342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:12:24.639342+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0036508320000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":254,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:49.344305+0000","last_change":"2024-09-17T07:13:46.575280+0000","last_active":"2024-09-17T07:13:49.344305+0000","last_peered":"2024-09-17T07:13:49.344305+0000","last_clean":"2024-09-17T07:13:49.344305+0000","last_became_active":"2024-09-17T07:02:16.014928+0000","last_became_peered":"2024-09-17T07:02:16.014928+0000","last_unstale":"2024-09-17T07:13:49.344305+0000","last_undegraded":"2024-09-17T07:13:49.344305+0000","last_fullsized":"2024-09-17T07:13:49.344305+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:46.575137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:13:46.575137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:56.258658+0000","scrub_duration":0.020219906999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":109,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:49.345130+0000","last_change":"2024-09-17T07:13:07.630234+0000","last_active":"2024-09-17T07:13:49.345130+0000","last_peered":"2024-09-17T07:13:49.345130+0000","last_clean":"2024-09-17T07:13:49.345130+0000","last_became_active":"2024-09-17T07:09:24.402568+0000","last_became_peered":"2024-09-17T07:09:24.402568+0000","last_unstale":"2024-09-17T07:13:49.345130+0000","last_undegraded":"2024-09-17T07:13:49.345130+0000","last_fullsized":"2024-09-17T07:13:49.345130+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:07.630076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:13:07.630076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:09.126175+0000","scrub_duration":0.0017151130000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"242'15","reported_seq":292,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:49.344389+0000","last_change":"2024-09-17T07:13:47.538039+0000","last_active":"2024-09-17T07:13:49.344389+0000","last_peered":"2024-09-17T07:13:49.344389+0000","last_clean":"2024-09-17T07:13:49.344389+0000","last_became_active":"2024-09-17T07:02:14.016244+0000","last_became_peered":"2024-09-17T07:02:14.016244+0000","last_unstale":"2024-09-17T07:13:49.344389+0000","last_undegraded":"2024-09-17T07:13:49.344389+0000","last_fullsized":"2024-09-17T07:13:49.344389+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'15","last_scrub_stamp":"2024-09-17T07:13:47.537894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:13:47.537894+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:50.482649+0000","scrub_duration":0.0052352630000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":253,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:49.350291+0000","last_change":"2024-09-17T07:13:17.184439+0000","last_active":"2024-09-17T07:13:49.350291+0000","last_peered":"2024-09-17T07:13:49.350291+0000","last_clean":"2024-09-17T07:13:49.350291+0000","last_became_active":"2024-09-17T07:02:16.014380+0000","last_became_peered":"2024-09-17T07:02:16.014380+0000","last_unstale":"2024-09-17T07:13:49.350291+0000","last_undegraded":"2024-09-17T07:13:49.350291+0000","last_fullsized":"2024-09-17T07:13:49.350291+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:17.184312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:13:17.184312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:36.365464+0000","scrub_duration":0.0075305299999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":253,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:49.344649+0000","last_change":"2024-09-17T07:12:25.491058+0000","last_active":"2024-09-17T07:13:49.344649+0000","last_peered":"2024-09-17T07:13:49.344649+0000","last_clean":"2024-09-17T07:13:49.344649+0000","last_became_active":"2024-09-17T07:02:18.020790+0000","last_became_peered":"2024-09-17T07:02:18.020790+0000","last_unstale":"2024-09-17T07:13:49.344649+0000","last_undegraded":"2024-09-17T07:13:49.344649+0000","last_fullsized":"2024-09-17T07:13:49.344649+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:25.490932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:12:25.490932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032974060000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":256,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:49.348807+0000","last_change":"2024-09-17T07:12:11.365357+0000","last_active":"2024-09-17T07:13:49.348807+0000","last_peered":"2024-09-17T07:13:49.348807+0000","last_clean":"2024-09-17T07:13:49.348807+0000","last_became_active":"2024-09-17T07:02:12.011129+0000","last_became_peered":"2024-09-17T07:02:12.011129+0000","last_unstale":"2024-09-17T07:13:49.348807+0000","last_undegraded":"2024-09-17T07:13:49.348807+0000","last_fullsized":"2024-09-17T07:13:49.348807+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:11.365269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:43.060102+0000","last_clean_scrub_stamp":"2024-09-17T07:12:11.365269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0012842960000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"242'7","reported_seq":285,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:49.345221+0000","last_change":"2024-09-17T07:13:11.674190+0000","last_active":"2024-09-17T07:13:49.345221+0000","last_peered":"2024-09-17T07:13:49.345221+0000","last_clean":"2024-09-17T07:13:49.345221+0000","last_became_active":"2024-09-17T07:09:24.396900+0000","last_became_peered":"2024-09-17T07:09:24.396900+0000","last_unstale":"2024-09-17T07:13:49.345221+0000","last_undegraded":"2024-09-17T07:13:49.345221+0000","last_fullsized":"2024-09-17T07:13:49.345221+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'7","last_scrub_stamp":"2024-09-17T07:13:11.674044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:13:11.674044+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:12.175367+0000","scrub_duration":0.0043110969999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":252,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:49.348806+0000","last_change":"2024-09-17T07:11:24.287105+0000","last_active":"2024-09-17T07:13:49.348806+0000","last_peered":"2024-09-17T07:13:49.348806+0000","last_clean":"2024-09-17T07:13:49.348806+0000","last_became_active":"2024-09-17T07:02:16.015401+0000","last_became_peered":"2024-09-17T07:02:16.015401+0000","last_unstale":"2024-09-17T07:13:49.348806+0000","last_undegraded":"2024-09-17T07:13:49.348806+0000","last_fullsized":"2024-09-17T07:13:49.348806+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:11:24.286986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:11:24.286986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0067853130000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":271,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:49.348293+0000","last_change":"2024-09-17T07:13:09.848188+0000","last_active":"2024-09-17T07:13:49.348293+0000","last_peered":"2024-09-17T07:13:49.348293+0000","last_clean":"2024-09-17T07:13:49.348293+0000","last_became_active":"2024-09-17T07:09:24.402616+0000","last_became_peered":"2024-09-17T07:09:24.402616+0000","last_unstale":"2024-09-17T07:13:49.348293+0000","last_undegraded":"2024-09-17T07:13:49.348293+0000","last_fullsized":"2024-09-17T07:13:49.348293+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:09.848034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:13:09.848034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:25.887201+0000","scrub_duration":0.0087071409999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":256,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:49.352939+0000","last_change":"2024-09-17T07:12:10.356268+0000","last_active":"2024-09-17T07:13:49.352939+0000","last_peered":"2024-09-17T07:13:49.352939+0000","last_clean":"2024-09-17T07:13:49.352939+0000","last_became_active":"2024-09-17T07:02:12.012234+0000","last_became_peered":"2024-09-17T07:02:12.012234+0000","last_unstale":"2024-09-17T07:13:49.352939+0000","last_undegraded":"2024-09-17T07:13:49.352939+0000","last_fullsized":"2024-09-17T07:13:49.352939+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:10.356011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:45.029247+0000","last_clean_scrub_stamp":"2024-09-17T07:12:10.356011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001613686,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"242'15","reported_seq":300,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:49.344991+0000","last_change":"2024-09-17T07:13:22.568728+0000","last_active":"2024-09-17T07:13:49.344991+0000","last_peered":"2024-09-17T07:13:49.344991+0000","last_clean":"2024-09-17T07:13:49.344991+0000","last_became_active":"2024-09-17T07:09:24.397954+0000","last_became_peered":"2024-09-17T07:09:24.397954+0000","last_unstale":"2024-09-17T07:13:49.344991+0000","last_undegraded":"2024-09-17T07:13:49.344991+0000","last_fullsized":"2024-09-17T07:13:49.344991+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'15","last_scrub_stamp":"2024-09-17T07:13:22.568591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:13:22.568591+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:39.102740+0000","scrub_duration":0.0053569139999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":263,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:49.357460+0000","last_change":"2024-09-17T07:13:28.892860+0000","last_active":"2024-09-17T07:13:49.357460+0000","last_peered":"2024-09-17T07:13:49.357460+0000","last_clean":"2024-09-17T07:13:49.357460+0000","last_became_active":"2024-09-17T07:02:16.023774+0000","last_became_peered":"2024-09-17T07:02:16.023774+0000","last_unstale":"2024-09-17T07:13:49.357460+0000","last_undegraded":"2024-09-17T07:13:49.357460+0000","last_fullsized":"2024-09-17T07:13:49.357460+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:28.892735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:28.096266+0000","last_clean_scrub_stamp":"2024-09-17T07:13:28.892735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:37.546083+0000","scrub_duration":0.029775294000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":252,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:49.345515+0000","last_change":"2024-09-17T07:13:12.646509+0000","last_active":"2024-09-17T07:13:49.345515+0000","last_peered":"2024-09-17T07:13:49.345515+0000","last_clean":"2024-09-17T07:13:49.345515+0000","last_became_active":"2024-09-17T07:02:18.029178+0000","last_became_peered":"2024-09-17T07:02:18.029178+0000","last_unstale":"2024-09-17T07:13:49.345515+0000","last_undegraded":"2024-09-17T07:13:49.345515+0000","last_fullsized":"2024-09-17T07:13:49.345515+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:12.646348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:13:12.646348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:36.705609+0000","scrub_duration":0.0028829530000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.0","version":"268'9498","reported_seq":9639,"reported_epoch":268,"state":"active+clean+scrubbing","last_fresh":"2024-09-17T07:14:03.722987+0000","last_change":"2024-09-17T07:14:02.496605+0000","last_active":"2024-09-17T07:14:03.722987+0000","last_peered":"2024-09-17T07:14:03.722987+0000","last_clean":"2024-09-17T07:14:03.722987+0000","last_became_active":"2024-09-17T07:12:05.417841+0000","last_became_peered":"2024-09-17T07:12:05.417841+0000","last_unstale":"2024-09-17T07:14:03.722987+0000","last_undegraded":"2024-09-17T07:14:03.722987+0000","last_fullsized":"2024-09-17T07:14:03.722987+0000","mapping_epoch":236,"log_start":"254'5100","ondisk_log_start":"254'5100","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_clean_scrub_stamp":"2024-09-17T07:12:04.285115+0000","objects_scrubbed":4699,"log_size":4398,"log_dups_size":0,"ondisk_log_size":4398,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 1s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":478806016,"num_objects":7306,"num_object_clones":0,"num_object_copies":21918,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9498,"num_write_kb":537728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"242'39","reported_seq":325,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:52.396347+0000","last_change":"2024-09-17T07:13:52.396347+0000","last_active":"2024-09-17T07:13:52.396347+0000","last_peered":"2024-09-17T07:13:52.396347+0000","last_clean":"2024-09-17T07:13:52.396347+0000","last_became_active":"2024-09-17T07:02:14.016270+0000","last_became_peered":"2024-09-17T07:02:14.016270+0000","last_unstale":"2024-09-17T07:13:52.396347+0000","last_undegraded":"2024-09-17T07:13:52.396347+0000","last_fullsized":"2024-09-17T07:13:52.396347+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'39","last_scrub_stamp":"2024-09-17T07:13:52.396215+0000","last_deep_scrub":"95'25","last_deep_scrub_stamp":"2024-09-17T07:04:09.450039+0000","last_clean_scrub_stamp":"2024-09-17T07:13:52.396215+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:58.420584+0000","scrub_duration":0.0048956010000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":270,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:49.352783+0000","last_change":"2024-09-17T07:13:25.951693+0000","last_active":"2024-09-17T07:13:49.352783+0000","last_peered":"2024-09-17T07:13:49.352783+0000","last_clean":"2024-09-17T07:13:49.352783+0000","last_became_active":"2024-09-17T07:02:12.011258+0000","last_became_peered":"2024-09-17T07:02:12.011258+0000","last_unstale":"2024-09-17T07:13:49.352783+0000","last_undegraded":"2024-09-17T07:13:49.352783+0000","last_fullsized":"2024-09-17T07:13:49.352783+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:25.950509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:13:25.950509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:40.605297+0000","scrub_duration":0.0099987270000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":254,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:49.350561+0000","last_change":"2024-09-17T07:13:46.502789+0000","last_active":"2024-09-17T07:13:49.350561+0000","last_peered":"2024-09-17T07:13:49.350561+0000","last_clean":"2024-09-17T07:13:49.350561+0000","last_became_active":"2024-09-17T07:02:16.018147+0000","last_became_peered":"2024-09-17T07:02:16.018147+0000","last_unstale":"2024-09-17T07:13:49.350561+0000","last_undegraded":"2024-09-17T07:13:49.350561+0000","last_fullsized":"2024-09-17T07:13:49.350561+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:46.502641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:13:46.502641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:15:06.659907+0000","scrub_duration":0.0074335069999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":101,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-17T07:13:53.369265+0000","last_change":"2024-09-17T07:12:21.520698+0000","last_active":"2024-09-17T07:13:53.369265+0000","last_peered":"2024-09-17T07:13:53.369265+0000","last_clean":"2024-09-17T07:13:53.369265+0000","last_became_active":"2024-09-17T07:09:24.396334+0000","last_became_peered":"2024-09-17T07:09:24.396334+0000","last_unstale":"2024-09-17T07:13:53.369265+0000","last_undegraded":"2024-09-17T07:13:53.369265+0000","last_fullsized":"2024-09-17T07:13:53.369265+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:21.520586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:42.984088+0000","last_clean_scrub_stamp":"2024-09-17T07:12:21.520586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0022027779999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1","version":"268'9550","reported_seq":9773,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-17T07:14:03.281302+0000","last_change":"2024-09-17T07:13:50.572979+0000","last_active":"2024-09-17T07:14:03.281302+0000","last_peered":"2024-09-17T07:14:03.281302+0000","last_clean":"2024-09-17T07:14:03.281302+0000","last_became_active":"2024-09-17T07:12:05.172332+0000","last_became_peered":"2024-09-17T07:12:05.172332+0000","last_unstale":"2024-09-17T07:14:03.281302+0000","last_undegraded":"2024-09-17T07:14:03.281302+0000","last_fullsized":"2024-09-17T07:14:03.281302+0000","mapping_epoch":236,"log_start":"252'4400","ondisk_log_start":"252'4400","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'8637","last_scrub_stamp":"2024-09-17T07:13:50.572714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_clean_scrub_stamp":"2024-09-17T07:13:50.572714+0000","objects_scrubbed":8331,"log_size":5150,"log_dups_size":0,"ondisk_log_size":5150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:15:08.150375+0000","scrub_duration":3.1196512479999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":481165312,"num_objects":7342,"num_object_clones":0,"num_object_copies":22026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9550,"num_write_kb":540544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"242'25","reported_seq":324,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:49.345687+0000","last_change":"2024-09-17T07:13:23.563057+0000","last_active":"2024-09-17T07:13:49.345687+0000","last_peered":"2024-09-17T07:13:49.345687+0000","last_clean":"2024-09-17T07:13:49.345687+0000","last_became_active":"2024-09-17T07:09:24.399287+0000","last_became_peered":"2024-09-17T07:09:24.399287+0000","last_unstale":"2024-09-17T07:13:49.345687+0000","last_undegraded":"2024-09-17T07:13:49.345687+0000","last_fullsized":"2024-09-17T07:13:49.345687+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'25","last_scrub_stamp":"2024-09-17T07:13:23.562295+0000","last_deep_scrub":"176'15","last_deep_scrub_stamp":"2024-09-17T07:08:19.633791+0000","last_clean_scrub_stamp":"2024-09-17T07:13:23.562295+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:33.304402+0000","scrub_duration":0.010202764,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":266,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:49.351816+0000","last_change":"2024-09-17T07:12:09.254527+0000","last_active":"2024-09-17T07:13:49.351816+0000","last_peered":"2024-09-17T07:13:49.351816+0000","last_clean":"2024-09-17T07:13:49.351816+0000","last_became_active":"2024-09-17T07:09:24.406973+0000","last_became_peered":"2024-09-17T07:09:24.406973+0000","last_unstale":"2024-09-17T07:13:49.351816+0000","last_undegraded":"2024-09-17T07:13:49.351816+0000","last_fullsized":"2024-09-17T07:13:49.351816+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:09.254452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:12:09.254452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013322150000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"98'9","reported_seq":355,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-17T07:13:55.663577+0000","last_change":"2024-09-17T07:12:00.407366+0000","last_active":"2024-09-17T07:13:55.663577+0000","last_peered":"2024-09-17T07:13:55.663577+0000","last_clean":"2024-09-17T07:13:55.663577+0000","last_became_active":"2024-09-17T07:04:15.110797+0000","last_became_peered":"2024-09-17T07:04:15.110797+0000","last_unstale":"2024-09-17T07:13:55.663577+0000","last_undegraded":"2024-09-17T07:13:55.663577+0000","last_fullsized":"2024-09-17T07:13:55.663577+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-09-17T07:12:00.407262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:12:00.407262+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002024489,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":264,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:49.345941+0000","last_change":"2024-09-17T07:13:12.654035+0000","last_active":"2024-09-17T07:13:49.345941+0000","last_peered":"2024-09-17T07:13:49.345941+0000","last_clean":"2024-09-17T07:13:49.345941+0000","last_became_active":"2024-09-17T07:02:18.017460+0000","last_became_peered":"2024-09-17T07:02:18.017460+0000","last_unstale":"2024-09-17T07:13:49.345941+0000","last_undegraded":"2024-09-17T07:13:49.345941+0000","last_fullsized":"2024-09-17T07:13:49.345941+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:12.652782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:13:12.652782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:20.978346+0000","scrub_duration":0.0091841319999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"268'9692","reported_seq":9733,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-17T07:14:03.732999+0000","last_change":"2024-09-17T07:12:05.416444+0000","last_active":"2024-09-17T07:14:03.732999+0000","last_peered":"2024-09-17T07:14:03.732999+0000","last_clean":"2024-09-17T07:14:03.732999+0000","last_became_active":"2024-09-17T07:12:05.416186+0000","last_became_peered":"2024-09-17T07:12:05.416186+0000","last_unstale":"2024-09-17T07:14:03.732999+0000","last_undegraded":"2024-09-17T07:14:03.732999+0000","last_fullsized":"2024-09-17T07:14:03.732999+0000","mapping_epoch":236,"log_start":"255'5300","ondisk_log_start":"255'5300","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_clean_scrub_stamp":"2024-09-17T07:12:04.285115+0000","objects_scrubbed":0,"log_size":4392,"log_dups_size":0,"ondisk_log_size":4392,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":480772096,"num_objects":7336,"num_object_clones":0,"num_object_copies":22008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7336,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9692,"num_write_kb":544896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"242'27","reported_seq":323,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:49.351339+0000","last_change":"2024-09-17T07:12:13.306460+0000","last_active":"2024-09-17T07:13:49.351339+0000","last_peered":"2024-09-17T07:13:49.351339+0000","last_clean":"2024-09-17T07:13:49.351339+0000","last_became_active":"2024-09-17T07:09:24.407311+0000","last_became_peered":"2024-09-17T07:09:24.407311+0000","last_unstale":"2024-09-17T07:13:49.351339+0000","last_undegraded":"2024-09-17T07:13:49.351339+0000","last_fullsized":"2024-09-17T07:13:49.351339+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-09-17T07:12:13.306349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:12:13.306349+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021853620000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":267,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:49.350467+0000","last_change":"2024-09-17T07:13:11.141362+0000","last_active":"2024-09-17T07:13:49.350467+0000","last_peered":"2024-09-17T07:13:49.350467+0000","last_clean":"2024-09-17T07:13:49.350467+0000","last_became_active":"2024-09-17T07:09:24.405655+0000","last_became_peered":"2024-09-17T07:09:24.405655+0000","last_unstale":"2024-09-17T07:13:49.350467+0000","last_undegraded":"2024-09-17T07:13:49.350467+0000","last_fullsized":"2024-09-17T07:13:49.350467+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:11.141247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:13:11.141247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:13.900806+0000","scrub_duration":0.0047986260000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"75'8","reported_seq":405,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-17T07:13:55.663229+0000","last_change":"2024-09-17T07:13:24.551128+0000","last_active":"2024-09-17T07:13:55.663229+0000","last_peered":"2024-09-17T07:13:55.663229+0000","last_clean":"2024-09-17T07:13:55.663229+0000","last_became_active":"2024-09-17T07:02:16.021762+0000","last_became_peered":"2024-09-17T07:02:16.021762+0000","last_unstale":"2024-09-17T07:13:55.663229+0000","last_undegraded":"2024-09-17T07:13:55.663229+0000","last_fullsized":"2024-09-17T07:13:55.663229+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:13:24.550889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:13:24.550889+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:52.797078+0000","scrub_duration":0.013749008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":262,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:49.357296+0000","last_change":"2024-09-17T07:13:25.930031+0000","last_active":"2024-09-17T07:13:49.357296+0000","last_peered":"2024-09-17T07:13:49.357296+0000","last_clean":"2024-09-17T07:13:49.357296+0000","last_became_active":"2024-09-17T07:02:18.025332+0000","last_became_peered":"2024-09-17T07:02:18.025332+0000","last_unstale":"2024-09-17T07:13:49.357296+0000","last_undegraded":"2024-09-17T07:13:49.357296+0000","last_fullsized":"2024-09-17T07:13:49.357296+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:25.929670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:13:25.929670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:35.833012+0000","scrub_duration":0.0085846570000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"268'9437","reported_seq":9477,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-17T07:14:03.491527+0000","last_change":"2024-09-17T07:12:05.174275+0000","last_active":"2024-09-17T07:14:03.491527+0000","last_peered":"2024-09-17T07:14:03.491527+0000","last_clean":"2024-09-17T07:14:03.491527+0000","last_became_active":"2024-09-17T07:12:05.174017+0000","last_became_peered":"2024-09-17T07:12:05.174017+0000","last_unstale":"2024-09-17T07:14:03.491527+0000","last_undegraded":"2024-09-17T07:14:03.491527+0000","last_fullsized":"2024-09-17T07:14:03.491527+0000","mapping_epoch":236,"log_start":"256'5200","ondisk_log_start":"256'5200","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_clean_scrub_stamp":"2024-09-17T07:12:04.285115+0000","objects_scrubbed":0,"log_size":4237,"log_dups_size":0,"ondisk_log_size":4237,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":472186880,"num_objects":7205,"num_object_clones":0,"num_object_copies":21615,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9437,"num_write_kb":532544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"242'19","reported_seq":299,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:49.346147+0000","last_change":"2024-09-17T07:13:17.544177+0000","last_active":"2024-09-17T07:13:49.346147+0000","last_peered":"2024-09-17T07:13:49.346147+0000","last_clean":"2024-09-17T07:13:49.346147+0000","last_became_active":"2024-09-17T07:02:14.010227+0000","last_became_peered":"2024-09-17T07:02:14.010227+0000","last_unstale":"2024-09-17T07:13:49.346147+0000","last_undegraded":"2024-09-17T07:13:49.346147+0000","last_fullsized":"2024-09-17T07:13:49.346147+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'19","last_scrub_stamp":"2024-09-17T07:13:17.543751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:13:17.543751+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:23.718388+0000","scrub_duration":0.01494678,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":269,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:49.351446+0000","last_change":"2024-09-17T07:13:21.858251+0000","last_active":"2024-09-17T07:13:49.351446+0000","last_peered":"2024-09-17T07:13:49.351446+0000","last_clean":"2024-09-17T07:13:49.351446+0000","last_became_active":"2024-09-17T07:02:12.013182+0000","last_became_peered":"2024-09-17T07:02:12.013182+0000","last_unstale":"2024-09-17T07:13:49.351446+0000","last_undegraded":"2024-09-17T07:13:49.351446+0000","last_fullsized":"2024-09-17T07:13:49.351446+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:21.858127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:13:21.858127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:33.414152+0000","scrub_duration":0.0033279709999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":275,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-17T07:14:01.113896+0000","last_change":"2024-09-17T07:14:01.113998+0000","last_active":"2024-09-17T07:14:01.113896+0000","last_peered":"2024-09-17T07:14:01.113896+0000","last_clean":"2024-09-17T07:14:01.113896+0000","last_became_active":"2024-09-17T07:02:16.020224+0000","last_became_peered":"2024-09-17T07:02:16.020224+0000","last_unstale":"2024-09-17T07:14:01.113896+0000","last_undegraded":"2024-09-17T07:14:01.113896+0000","last_fullsized":"2024-09-17T07:14:01.113896+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:01.113874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:14:01.113874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:15:09.719677+0000","scrub_duration":0.027812278999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":119,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-17T07:13:53.375807+0000","last_change":"2024-09-17T07:13:22.187280+0000","last_active":"2024-09-17T07:13:53.375807+0000","last_peered":"2024-09-17T07:13:53.375807+0000","last_clean":"2024-09-17T07:13:53.375807+0000","last_became_active":"2024-09-17T07:09:24.408109+0000","last_became_peered":"2024-09-17T07:09:24.408109+0000","last_unstale":"2024-09-17T07:13:53.375807+0000","last_undegraded":"2024-09-17T07:13:53.375807+0000","last_fullsized":"2024-09-17T07:13:53.375807+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:22.187127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:13:22.187127+0000","last_clean_scrub_stamp":"2024-09-17T07:13:22.187127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:36.934260+0000","scrub_duration":0.011212194999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"268'9526","reported_seq":9719,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-17T07:14:03.355912+0000","last_change":"2024-09-17T07:13:28.810891+0000","last_active":"2024-09-17T07:14:03.355912+0000","last_peered":"2024-09-17T07:14:03.355912+0000","last_clean":"2024-09-17T07:14:03.355912+0000","last_became_active":"2024-09-17T07:12:05.173802+0000","last_became_peered":"2024-09-17T07:12:05.173802+0000","last_unstale":"2024-09-17T07:14:03.355912+0000","last_undegraded":"2024-09-17T07:14:03.355912+0000","last_fullsized":"2024-09-17T07:14:03.355912+0000","mapping_epoch":236,"log_start":"252'4500","ondisk_log_start":"252'4500","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'7086","last_scrub_stamp":"2024-09-17T07:13:28.810560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_clean_scrub_stamp":"2024-09-17T07:13:28.810560+0000","objects_scrubbed":7013,"log_size":5026,"log_dups_size":0,"ondisk_log_size":5026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:42.608639+0000","scrub_duration":2.281952134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":475922432,"num_objects":7262,"num_object_clones":0,"num_object_copies":21786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9526,"num_write_kb":537216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"242'28","reported_seq":299,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:49.474636+0000","last_change":"2024-09-17T07:13:49.474636+0000","last_active":"2024-09-17T07:13:49.474636+0000","last_peered":"2024-09-17T07:13:49.474636+0000","last_clean":"2024-09-17T07:13:49.474636+0000","last_became_active":"2024-09-17T07:02:14.007810+0000","last_became_peered":"2024-09-17T07:02:14.007810+0000","last_unstale":"2024-09-17T07:13:49.474636+0000","last_undegraded":"2024-09-17T07:13:49.474636+0000","last_fullsized":"2024-09-17T07:13:49.474636+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'28","last_scrub_stamp":"2024-09-17T07:13:49.474100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:13:49.474100+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:55.188004+0000","scrub_duration":0.01650393,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":290,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:49.355472+0000","last_change":"2024-09-17T07:13:17.132943+0000","last_active":"2024-09-17T07:13:49.355472+0000","last_peered":"2024-09-17T07:13:49.355472+0000","last_clean":"2024-09-17T07:13:49.355472+0000","last_became_active":"2024-09-17T07:02:12.011766+0000","last_became_peered":"2024-09-17T07:02:12.011766+0000","last_unstale":"2024-09-17T07:13:49.355472+0000","last_undegraded":"2024-09-17T07:13:49.355472+0000","last_fullsized":"2024-09-17T07:13:49.355472+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:17.132822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:09:23.181028+0000","last_clean_scrub_stamp":"2024-09-17T07:13:17.132822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:37.062282+0000","scrub_duration":0.0079643140000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"196'9","reported_seq":177,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-17T07:13:55.662961+0000","last_change":"2024-09-17T07:13:12.145244+0000","last_active":"2024-09-17T07:13:55.662961+0000","last_peered":"2024-09-17T07:13:55.662961+0000","last_clean":"2024-09-17T07:13:55.662961+0000","last_became_active":"2024-09-17T07:09:24.406056+0000","last_became_peered":"2024-09-17T07:09:24.406056+0000","last_unstale":"2024-09-17T07:13:55.662961+0000","last_undegraded":"2024-09-17T07:13:55.662961+0000","last_fullsized":"2024-09-17T07:13:55.662961+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'9","last_scrub_stamp":"2024-09-17T07:13:12.145158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:13:12.145158+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:36.583847+0000","scrub_duration":0.001480777,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":262,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:49.352482+0000","last_change":"2024-09-17T07:12:25.194692+0000","last_active":"2024-09-17T07:13:49.352482+0000","last_peered":"2024-09-17T07:13:49.352482+0000","last_clean":"2024-09-17T07:13:49.352482+0000","last_became_active":"2024-09-17T07:02:18.364981+0000","last_became_peered":"2024-09-17T07:02:18.364981+0000","last_unstale":"2024-09-17T07:13:49.352482+0000","last_undegraded":"2024-09-17T07:13:49.352482+0000","last_fullsized":"2024-09-17T07:13:49.352482+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:25.194515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:10:33.727472+0000","last_clean_scrub_stamp":"2024-09-17T07:12:25.194515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002844943,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.d","version":"268'9483","reported_seq":9524,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-17T07:14:03.513157+0000","last_change":"2024-09-17T07:12:05.177946+0000","last_active":"2024-09-17T07:14:03.513157+0000","last_peered":"2024-09-17T07:14:03.513157+0000","last_clean":"2024-09-17T07:14:03.513157+0000","last_became_active":"2024-09-17T07:12:05.177682+0000","last_became_peered":"2024-09-17T07:12:05.177682+0000","last_unstale":"2024-09-17T07:14:03.513157+0000","last_undegraded":"2024-09-17T07:14:03.513157+0000","last_fullsized":"2024-09-17T07:14:03.513157+0000","mapping_epoch":236,"log_start":"256'5300","ondisk_log_start":"256'5300","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_clean_scrub_stamp":"2024-09-17T07:12:04.285115+0000","objects_scrubbed":0,"log_size":4183,"log_dups_size":0,"ondisk_log_size":4183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":475529240,"num_objects":7257,"num_object_clones":0,"num_object_copies":21771,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7257,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9483,"num_write_kb":535617,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"242'30","reported_seq":188,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:49.351267+0000","last_change":"2024-09-17T07:13:28.997698+0000","last_active":"2024-09-17T07:13:49.351267+0000","last_peered":"2024-09-17T07:13:49.351267+0000","last_clean":"2024-09-17T07:13:49.351267+0000","last_became_active":"2024-09-17T07:09:24.403240+0000","last_became_peered":"2024-09-17T07:09:24.403240+0000","last_unstale":"2024-09-17T07:13:49.351267+0000","last_undegraded":"2024-09-17T07:13:49.351267+0000","last_fullsized":"2024-09-17T07:13:49.351267+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'30","last_scrub_stamp":"2024-09-17T07:13:28.997563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:13:28.997563+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:35.023378+0000","scrub_duration":0.0061961159999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":268,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:49.350733+0000","last_change":"2024-09-17T07:13:23.253165+0000","last_active":"2024-09-17T07:13:49.350733+0000","last_peered":"2024-09-17T07:13:49.350733+0000","last_clean":"2024-09-17T07:13:49.350733+0000","last_became_active":"2024-09-17T07:02:12.005687+0000","last_became_peered":"2024-09-17T07:02:12.005687+0000","last_unstale":"2024-09-17T07:13:49.350733+0000","last_undegraded":"2024-09-17T07:13:49.350733+0000","last_fullsized":"2024-09-17T07:13:49.350733+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:23.253036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:13:23.253036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:43.201094+0000","scrub_duration":0.002831362,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":264,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:49.348869+0000","last_change":"2024-09-17T07:13:06.579449+0000","last_active":"2024-09-17T07:13:49.348869+0000","last_peered":"2024-09-17T07:13:49.348869+0000","last_clean":"2024-09-17T07:13:49.348869+0000","last_became_active":"2024-09-17T07:09:24.393375+0000","last_became_peered":"2024-09-17T07:09:24.393375+0000","last_unstale":"2024-09-17T07:13:49.348869+0000","last_undegraded":"2024-09-17T07:13:49.348869+0000","last_fullsized":"2024-09-17T07:13:49.348869+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:06.578911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:13:06.578911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:23.699140+0000","scrub_duration":0.0074995440000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":262,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:49.344910+0000","last_change":"2024-09-17T07:12:18.466015+0000","last_active":"2024-09-17T07:13:49.344910+0000","last_peered":"2024-09-17T07:13:49.344910+0000","last_clean":"2024-09-17T07:13:49.344910+0000","last_became_active":"2024-09-17T07:09:24.398659+0000","last_became_peered":"2024-09-17T07:09:24.398659+0000","last_unstale":"2024-09-17T07:13:49.344910+0000","last_undegraded":"2024-09-17T07:13:49.344910+0000","last_fullsized":"2024-09-17T07:13:49.344910+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:18.465901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:12:18.465901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0024831419999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.e","version":"268'9537","reported_seq":9577,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-17T07:14:03.528333+0000","last_change":"2024-09-17T07:12:05.175609+0000","last_active":"2024-09-17T07:14:03.528333+0000","last_peered":"2024-09-17T07:14:03.528333+0000","last_clean":"2024-09-17T07:14:03.528333+0000","last_became_active":"2024-09-17T07:12:05.175318+0000","last_became_peered":"2024-09-17T07:12:05.175318+0000","last_unstale":"2024-09-17T07:14:03.528333+0000","last_undegraded":"2024-09-17T07:14:03.528333+0000","last_fullsized":"2024-09-17T07:14:03.528333+0000","mapping_epoch":236,"log_start":"256'5300","ondisk_log_start":"256'5300","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_clean_scrub_stamp":"2024-09-17T07:12:04.285115+0000","objects_scrubbed":0,"log_size":4237,"log_dups_size":0,"ondisk_log_size":4237,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":481230848,"num_objects":7343,"num_object_clones":0,"num_object_copies":22029,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9537,"num_write_kb":540160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"242'29","reported_seq":308,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:49.345150+0000","last_change":"2024-09-17T07:13:13.626759+0000","last_active":"2024-09-17T07:13:49.345150+0000","last_peered":"2024-09-17T07:13:49.345150+0000","last_clean":"2024-09-17T07:13:49.345150+0000","last_became_active":"2024-09-17T07:02:14.027389+0000","last_became_peered":"2024-09-17T07:02:14.027389+0000","last_unstale":"2024-09-17T07:13:49.345150+0000","last_undegraded":"2024-09-17T07:13:49.345150+0000","last_fullsized":"2024-09-17T07:13:49.345150+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'29","last_scrub_stamp":"2024-09-17T07:13:13.626559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:13:13.626559+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:26.067016+0000","scrub_duration":0.013908154000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":269,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:49.348883+0000","last_change":"2024-09-17T07:13:23.945293+0000","last_active":"2024-09-17T07:13:49.348883+0000","last_peered":"2024-09-17T07:13:49.348883+0000","last_clean":"2024-09-17T07:13:49.348883+0000","last_became_active":"2024-09-17T07:02:12.011917+0000","last_became_peered":"2024-09-17T07:02:12.011917+0000","last_unstale":"2024-09-17T07:13:49.348883+0000","last_undegraded":"2024-09-17T07:13:49.348883+0000","last_fullsized":"2024-09-17T07:13:49.348883+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:23.945157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:24.072799+0000","last_clean_scrub_stamp":"2024-09-17T07:13:23.945157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:43.607995+0000","scrub_duration":0.0029434130000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":265,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:49.353994+0000","last_change":"2024-09-17T07:13:02.907379+0000","last_active":"2024-09-17T07:13:49.353994+0000","last_peered":"2024-09-17T07:13:49.353994+0000","last_clean":"2024-09-17T07:13:49.353994+0000","last_became_active":"2024-09-17T07:02:16.016199+0000","last_became_peered":"2024-09-17T07:02:16.016199+0000","last_unstale":"2024-09-17T07:13:49.353994+0000","last_undegraded":"2024-09-17T07:13:49.353994+0000","last_fullsized":"2024-09-17T07:13:49.353994+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:02.907196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:11:17.583712+0000","last_clean_scrub_stamp":"2024-09-17T07:13:02.907196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:09.093088+0000","scrub_duration":0.0052105759999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":262,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:50.514634+0000","last_change":"2024-09-17T07:13:50.514634+0000","last_active":"2024-09-17T07:13:50.514634+0000","last_peered":"2024-09-17T07:13:50.514634+0000","last_clean":"2024-09-17T07:13:50.514634+0000","last_became_active":"2024-09-17T07:02:18.364835+0000","last_became_peered":"2024-09-17T07:02:18.364835+0000","last_unstale":"2024-09-17T07:13:50.514634+0000","last_undegraded":"2024-09-17T07:13:50.514634+0000","last_fullsized":"2024-09-17T07:13:50.514634+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:50.514525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:13:50.514525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:15:05.831575+0000","scrub_duration":0.010637423,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.f","version":"268'9534","reported_seq":9573,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-17T07:14:03.487951+0000","last_change":"2024-09-17T07:12:05.418098+0000","last_active":"2024-09-17T07:14:03.487951+0000","last_peered":"2024-09-17T07:14:03.487951+0000","last_clean":"2024-09-17T07:14:03.487951+0000","last_became_active":"2024-09-17T07:12:05.417843+0000","last_became_peered":"2024-09-17T07:12:05.417843+0000","last_unstale":"2024-09-17T07:14:03.487951+0000","last_undegraded":"2024-09-17T07:14:03.487951+0000","last_fullsized":"2024-09-17T07:14:03.487951+0000","mapping_epoch":236,"log_start":"253'4900","ondisk_log_start":"253'4900","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_clean_scrub_stamp":"2024-09-17T07:12:04.285115+0000","objects_scrubbed":0,"log_size":4634,"log_dups_size":0,"ondisk_log_size":4634,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":475267072,"num_objects":7252,"num_object_clones":0,"num_object_copies":21756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9534,"num_write_kb":537152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"242'25","reported_seq":290,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:49.351740+0000","last_change":"2024-09-17T07:12:19.410879+0000","last_active":"2024-09-17T07:13:49.351740+0000","last_peered":"2024-09-17T07:13:49.351740+0000","last_clean":"2024-09-17T07:13:49.351740+0000","last_became_active":"2024-09-17T07:02:14.010637+0000","last_became_peered":"2024-09-17T07:02:14.010637+0000","last_unstale":"2024-09-17T07:13:49.351740+0000","last_undegraded":"2024-09-17T07:13:49.351740+0000","last_fullsized":"2024-09-17T07:13:49.351740+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'17","last_scrub_stamp":"2024-09-17T07:12:19.410758+0000","last_deep_scrub":"110'15","last_deep_scrub_stamp":"2024-09-17T07:04:57.091400+0000","last_clean_scrub_stamp":"2024-09-17T07:12:19.410758+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0070558269999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":257,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:49.351017+0000","last_change":"2024-09-17T07:12:04.314016+0000","last_active":"2024-09-17T07:13:49.351017+0000","last_peered":"2024-09-17T07:13:49.351017+0000","last_clean":"2024-09-17T07:13:49.351017+0000","last_became_active":"2024-09-17T07:02:12.010803+0000","last_became_peered":"2024-09-17T07:02:12.010803+0000","last_unstale":"2024-09-17T07:13:49.351017+0000","last_undegraded":"2024-09-17T07:13:49.351017+0000","last_fullsized":"2024-09-17T07:13:49.351017+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:04.313886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:25.147878+0000","last_clean_scrub_stamp":"2024-09-17T07:12:04.313886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0018928020000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"89'9","reported_seq":416,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-17T07:13:55.662868+0000","last_change":"2024-09-17T07:13:25.608693+0000","last_active":"2024-09-17T07:13:55.662868+0000","last_peered":"2024-09-17T07:13:55.662868+0000","last_clean":"2024-09-17T07:13:55.662868+0000","last_became_active":"2024-09-17T07:02:16.021360+0000","last_became_peered":"2024-09-17T07:02:16.021360+0000","last_unstale":"2024-09-17T07:13:55.662868+0000","last_undegraded":"2024-09-17T07:13:55.662868+0000","last_fullsized":"2024-09-17T07:13:55.662868+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'9","last_scrub_stamp":"2024-09-17T07:13:25.608457+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-09-17T07:03:29.961439+0000","last_clean_scrub_stamp":"2024-09-17T07:13:25.608457+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:50.846059+0000","scrub_duration":0.0077181510000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"268'2","reported_seq":278,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-17T07:14:03.609164+0000","last_change":"2024-09-17T07:14:03.609273+0000","last_active":"2024-09-17T07:14:03.609164+0000","last_peered":"2024-09-17T07:14:03.609164+0000","last_clean":"2024-09-17T07:14:03.609164+0000","last_became_active":"2024-09-17T07:09:24.399565+0000","last_became_peered":"2024-09-17T07:09:24.399565+0000","last_unstale":"2024-09-17T07:14:03.609164+0000","last_undegraded":"2024-09-17T07:14:03.609164+0000","last_fullsized":"2024-09-17T07:14:03.609164+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'2","last_scrub_stamp":"2024-09-17T07:14:03.609131+0000","last_deep_scrub":"268'2","last_deep_scrub_stamp":"2024-09-17T07:14:03.609131+0000","last_clean_scrub_stamp":"2024-09-17T07:14:03.609131+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:15:28.766122+0000","scrub_duration":0.011205872,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.9","version":"268'9730","reported_seq":9951,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-17T07:14:04.312083+0000","last_change":"2024-09-17T07:13:47.191287+0000","last_active":"2024-09-17T07:14:04.312083+0000","last_peered":"2024-09-17T07:14:04.312083+0000","last_clean":"2024-09-17T07:14:04.312083+0000","last_became_active":"2024-09-17T07:12:05.174893+0000","last_became_peered":"2024-09-17T07:12:05.174893+0000","last_unstale":"2024-09-17T07:14:04.312083+0000","last_undegraded":"2024-09-17T07:14:04.312083+0000","last_fullsized":"2024-09-17T07:14:04.312083+0000","mapping_epoch":236,"log_start":"252'4700","ondisk_log_start":"252'4700","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"266'8545","last_scrub_stamp":"2024-09-17T07:13:47.191084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_clean_scrub_stamp":"2024-09-17T07:13:47.191084+0000","objects_scrubbed":8451,"log_size":5030,"log_dups_size":0,"ondisk_log_size":5030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:53.808772+0000","scrub_duration":2.5276497170000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482869248,"num_objects":7368,"num_object_clones":0,"num_object_copies":22104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7368,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9730,"num_write_kb":547136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"242'41","reported_seq":494,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-17T07:13:55.662923+0000","last_change":"2024-09-17T07:12:08.264127+0000","last_active":"2024-09-17T07:13:55.662923+0000","last_peered":"2024-09-17T07:13:55.662923+0000","last_clean":"2024-09-17T07:13:55.662923+0000","last_became_active":"2024-09-17T07:09:24.403267+0000","last_became_peered":"2024-09-17T07:09:24.403267+0000","last_unstale":"2024-09-17T07:13:55.662923+0000","last_undegraded":"2024-09-17T07:13:55.662923+0000","last_fullsized":"2024-09-17T07:13:55.662923+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'29","last_scrub_stamp":"2024-09-17T07:12:08.264016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:12:08.264016+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026058610000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":279,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-17T07:14:03.045734+0000","last_change":"2024-09-17T07:14:03.046151+0000","last_active":"2024-09-17T07:14:03.045734+0000","last_peered":"2024-09-17T07:14:03.045734+0000","last_clean":"2024-09-17T07:14:03.045734+0000","last_became_active":"2024-09-17T07:02:12.008802+0000","last_became_peered":"2024-09-17T07:02:12.008802+0000","last_unstale":"2024-09-17T07:14:03.045734+0000","last_undegraded":"2024-09-17T07:14:03.045734+0000","last_fullsized":"2024-09-17T07:14:03.045734+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:03.045709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:14:03.045709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:15:18.281913+0000","scrub_duration":0.0049245929999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":265,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:49.344432+0000","last_change":"2024-09-17T07:13:09.682306+0000","last_active":"2024-09-17T07:13:49.344432+0000","last_peered":"2024-09-17T07:13:49.344432+0000","last_clean":"2024-09-17T07:13:49.344432+0000","last_became_active":"2024-09-17T07:02:16.016606+0000","last_became_peered":"2024-09-17T07:02:16.016606+0000","last_unstale":"2024-09-17T07:13:49.344432+0000","last_undegraded":"2024-09-17T07:13:49.344432+0000","last_fullsized":"2024-09-17T07:13:49.344432+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:09.682171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:13:09.682171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:21.062791+0000","scrub_duration":0.0022092190000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":109,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-17T07:13:53.406687+0000","last_change":"2024-09-17T07:13:53.406787+0000","last_active":"2024-09-17T07:13:53.406687+0000","last_peered":"2024-09-17T07:13:53.406687+0000","last_clean":"2024-09-17T07:13:53.406687+0000","last_became_active":"2024-09-17T07:09:24.398878+0000","last_became_peered":"2024-09-17T07:09:24.398878+0000","last_unstale":"2024-09-17T07:13:53.406687+0000","last_undegraded":"2024-09-17T07:13:53.406687+0000","last_fullsized":"2024-09-17T07:13:53.406687+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:53.406661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:13:53.406661+0000","last_clean_scrub_stamp":"2024-09-17T07:13:53.406661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:15:05.060705+0000","scrub_duration":0.0074403789999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.a","version":"268'9398","reported_seq":9599,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-17T07:14:03.474420+0000","last_change":"2024-09-17T07:13:59.891840+0000","last_active":"2024-09-17T07:14:03.474420+0000","last_peered":"2024-09-17T07:14:03.474420+0000","last_clean":"2024-09-17T07:14:03.474420+0000","last_became_active":"2024-09-17T07:12:05.175109+0000","last_became_peered":"2024-09-17T07:12:05.175109+0000","last_unstale":"2024-09-17T07:14:03.474420+0000","last_undegraded":"2024-09-17T07:14:03.474420+0000","last_fullsized":"2024-09-17T07:14:03.474420+0000","mapping_epoch":236,"log_start":"252'4800","ondisk_log_start":"252'4800","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'9122","last_scrub_stamp":"2024-09-17T07:13:59.891657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_clean_scrub_stamp":"2024-09-17T07:13:59.891657+0000","objects_scrubbed":7549,"log_size":4598,"log_dups_size":0,"ondisk_log_size":4598,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:15:14.153676+0000","scrub_duration":2.9016005169999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":470155264,"num_objects":7174,"num_object_clones":0,"num_object_copies":21522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7174,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9398,"num_write_kb":530304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"242'17","reported_seq":299,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:49.351410+0000","last_change":"2024-09-17T07:13:47.468495+0000","last_active":"2024-09-17T07:13:49.351410+0000","last_peered":"2024-09-17T07:13:49.351410+0000","last_clean":"2024-09-17T07:13:49.351410+0000","last_became_active":"2024-09-17T07:02:14.019838+0000","last_became_peered":"2024-09-17T07:02:14.019838+0000","last_unstale":"2024-09-17T07:13:49.351410+0000","last_undegraded":"2024-09-17T07:13:49.351410+0000","last_fullsized":"2024-09-17T07:13:49.351410+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'17","last_scrub_stamp":"2024-09-17T07:13:47.468399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:13:47.468399+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:15:03.608044+0000","scrub_duration":0.0060398329999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"76'5","reported_seq":969,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-17T07:14:02.601844+0000","last_change":"2024-09-17T07:12:22.408620+0000","last_active":"2024-09-17T07:14:02.601844+0000","last_peered":"2024-09-17T07:14:02.601844+0000","last_clean":"2024-09-17T07:14:02.601844+0000","last_became_active":"2024-09-17T07:02:12.006883+0000","last_became_peered":"2024-09-17T07:02:12.006883+0000","last_unstale":"2024-09-17T07:14:02.601844+0000","last_undegraded":"2024-09-17T07:14:02.601844+0000","last_fullsized":"2024-09-17T07:14:02.601844+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'5","last_scrub_stamp":"2024-09-17T07:12:22.408498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:12:22.408498+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032583909999999998,"objects_trimmed":0,"snaptrim_duration":0.00033739799999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":704,"num_read_kb":699,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":110,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:50.549466+0000","last_change":"2024-09-17T07:13:50.549466+0000","last_active":"2024-09-17T07:13:50.549466+0000","last_peered":"2024-09-17T07:13:50.549466+0000","last_clean":"2024-09-17T07:13:50.549466+0000","last_became_active":"2024-09-17T07:09:24.392850+0000","last_became_peered":"2024-09-17T07:09:24.392850+0000","last_unstale":"2024-09-17T07:13:50.549466+0000","last_undegraded":"2024-09-17T07:13:50.549466+0000","last_fullsized":"2024-09-17T07:13:50.549466+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:50.549369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:23.833170+0000","last_clean_scrub_stamp":"2024-09-17T07:13:50.549369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:50.944032+0000","scrub_duration":0.0013250060000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":278,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-17T07:13:54.117382+0000","last_change":"2024-09-17T07:13:54.117489+0000","last_active":"2024-09-17T07:13:54.117382+0000","last_peered":"2024-09-17T07:13:54.117382+0000","last_clean":"2024-09-17T07:13:54.117382+0000","last_became_active":"2024-09-17T07:02:16.014492+0000","last_became_peered":"2024-09-17T07:02:16.014492+0000","last_unstale":"2024-09-17T07:13:54.117382+0000","last_undegraded":"2024-09-17T07:13:54.117382+0000","last_fullsized":"2024-09-17T07:13:54.117382+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:54.117350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:13:54.117350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:15:21.006161+0000","scrub_duration":0.015188548,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":271,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:49.348419+0000","last_change":"2024-09-17T07:12:12.332042+0000","last_active":"2024-09-17T07:13:49.348419+0000","last_peered":"2024-09-17T07:13:49.348419+0000","last_clean":"2024-09-17T07:13:49.348419+0000","last_became_active":"2024-09-17T07:09:24.402704+0000","last_became_peered":"2024-09-17T07:09:24.402704+0000","last_unstale":"2024-09-17T07:13:49.348419+0000","last_undegraded":"2024-09-17T07:13:49.348419+0000","last_fullsized":"2024-09-17T07:13:49.348419+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:12.331949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:12:12.331949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0012857490000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"268'9425","reported_seq":9628,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-17T07:14:04.333149+0000","last_change":"2024-09-17T07:13:59.013583+0000","last_active":"2024-09-17T07:14:04.333149+0000","last_peered":"2024-09-17T07:14:04.333149+0000","last_clean":"2024-09-17T07:14:04.333149+0000","last_became_active":"2024-09-17T07:12:05.173534+0000","last_became_peered":"2024-09-17T07:12:05.173534+0000","last_unstale":"2024-09-17T07:14:04.333149+0000","last_undegraded":"2024-09-17T07:14:04.333149+0000","last_fullsized":"2024-09-17T07:14:04.333149+0000","mapping_epoch":236,"log_start":"252'4400","ondisk_log_start":"252'4400","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'9015","last_scrub_stamp":"2024-09-17T07:13:59.013457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_clean_scrub_stamp":"2024-09-17T07:13:59.013457+0000","objects_scrubbed":7572,"log_size":5025,"log_dups_size":0,"ondisk_log_size":5025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:15:28.079676+0000","scrub_duration":2.3824500519999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":463929344,"num_objects":7079,"num_object_clones":0,"num_object_copies":21237,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7079,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9425,"num_write_kb":528128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"268'30","reported_seq":332,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-17T07:14:00.599691+0000","last_change":"2024-09-17T07:14:00.599815+0000","last_active":"2024-09-17T07:14:00.599691+0000","last_peered":"2024-09-17T07:14:00.599691+0000","last_clean":"2024-09-17T07:14:00.599691+0000","last_became_active":"2024-09-17T07:03:48.760405+0000","last_became_peered":"2024-09-17T07:03:48.760405+0000","last_unstale":"2024-09-17T07:14:00.599691+0000","last_undegraded":"2024-09-17T07:14:00.599691+0000","last_fullsized":"2024-09-17T07:14:00.599691+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'30","last_scrub_stamp":"2024-09-17T07:14:00.599656+0000","last_deep_scrub":"268'30","last_deep_scrub_stamp":"2024-09-17T07:14:00.599656+0000","last_clean_scrub_stamp":"2024-09-17T07:14:00.599656+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:15:24.087868+0000","scrub_duration":0.030045246000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"70'1","reported_seq":277,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:49.344135+0000","last_change":"2024-09-17T07:13:05.610360+0000","last_active":"2024-09-17T07:13:49.344135+0000","last_peered":"2024-09-17T07:13:49.344135+0000","last_clean":"2024-09-17T07:13:49.344135+0000","last_became_active":"2024-09-17T07:09:24.402499+0000","last_became_peered":"2024-09-17T07:09:24.402499+0000","last_unstale":"2024-09-17T07:13:49.344135+0000","last_undegraded":"2024-09-17T07:13:49.344135+0000","last_fullsized":"2024-09-17T07:13:49.344135+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-17T07:13:05.610235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:13:05.610235+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:15.487535+0000","scrub_duration":0.0063184210000000003,"objects_trimmed":0,"snaptrim_duration":0.0032298629999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":257,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:49.351507+0000","last_change":"2024-09-17T07:12:16.273630+0000","last_active":"2024-09-17T07:13:49.351507+0000","last_peered":"2024-09-17T07:13:49.351507+0000","last_clean":"2024-09-17T07:13:49.351507+0000","last_became_active":"2024-09-17T07:02:12.014623+0000","last_became_peered":"2024-09-17T07:02:12.014623+0000","last_unstale":"2024-09-17T07:13:49.351507+0000","last_undegraded":"2024-09-17T07:13:49.351507+0000","last_fullsized":"2024-09-17T07:13:49.351507+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:16.273537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:12:16.273537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001748185,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":264,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-17T07:13:58.015877+0000","last_change":"2024-09-17T07:13:58.021130+0000","last_active":"2024-09-17T07:13:58.015877+0000","last_peered":"2024-09-17T07:13:58.015877+0000","last_clean":"2024-09-17T07:13:58.015877+0000","last_became_active":"2024-09-17T07:02:16.021561+0000","last_became_peered":"2024-09-17T07:02:16.021561+0000","last_unstale":"2024-09-17T07:13:58.015877+0000","last_undegraded":"2024-09-17T07:13:58.015877+0000","last_fullsized":"2024-09-17T07:13:58.015877+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:58.015853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:13:58.015853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:15:07.906016+0000","scrub_duration":0.0048227230000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":260,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:49.350488+0000","last_change":"2024-09-17T07:13:07.152353+0000","last_active":"2024-09-17T07:13:49.350488+0000","last_peered":"2024-09-17T07:13:49.350488+0000","last_clean":"2024-09-17T07:13:49.350488+0000","last_became_active":"2024-09-17T07:09:24.396154+0000","last_became_peered":"2024-09-17T07:09:24.396154+0000","last_unstale":"2024-09-17T07:13:49.350488+0000","last_undegraded":"2024-09-17T07:13:49.350488+0000","last_fullsized":"2024-09-17T07:13:49.350488+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:07.152198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:13:07.152198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:20.576466+0000","scrub_duration":0.0050730200000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"268'9489","reported_seq":9675,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-17T07:14:04.320525+0000","last_change":"2024-09-17T07:13:26.677354+0000","last_active":"2024-09-17T07:14:04.320525+0000","last_peered":"2024-09-17T07:14:04.320525+0000","last_clean":"2024-09-17T07:14:04.320525+0000","last_became_active":"2024-09-17T07:12:05.175944+0000","last_became_peered":"2024-09-17T07:12:05.175944+0000","last_unstale":"2024-09-17T07:14:04.320525+0000","last_undegraded":"2024-09-17T07:14:04.320525+0000","last_fullsized":"2024-09-17T07:14:04.320525+0000","mapping_epoch":236,"log_start":"252'4400","ondisk_log_start":"252'4400","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'6923","last_scrub_stamp":"2024-09-17T07:13:26.677050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_clean_scrub_stamp":"2024-09-17T07:13:26.677050+0000","objects_scrubbed":6842,"log_size":5089,"log_dups_size":0,"ondisk_log_size":5089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:40.632843+0000","scrub_duration":2.0920533560000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":473890816,"num_objects":7231,"num_object_clones":0,"num_object_copies":21693,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7231,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9489,"num_write_kb":535040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"242'26","reported_seq":308,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:49.354862+0000","last_change":"2024-09-17T07:13:07.955600+0000","last_active":"2024-09-17T07:13:49.354862+0000","last_peered":"2024-09-17T07:13:49.354862+0000","last_clean":"2024-09-17T07:13:49.354862+0000","last_became_active":"2024-09-17T07:02:14.028857+0000","last_became_peered":"2024-09-17T07:02:14.028857+0000","last_unstale":"2024-09-17T07:13:49.354862+0000","last_undegraded":"2024-09-17T07:13:49.354862+0000","last_fullsized":"2024-09-17T07:13:49.354862+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'26","last_scrub_stamp":"2024-09-17T07:13:07.955499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:13:07.955499+0000","objects_scrubbed":7,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:13.891062+0000","scrub_duration":0.0019340029999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"268'3","reported_seq":128,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-17T07:13:54.949372+0000","last_change":"2024-09-17T07:13:54.949565+0000","last_active":"2024-09-17T07:13:54.949372+0000","last_peered":"2024-09-17T07:13:54.949372+0000","last_clean":"2024-09-17T07:13:54.949372+0000","last_became_active":"2024-09-17T07:09:24.404532+0000","last_became_peered":"2024-09-17T07:09:24.404532+0000","last_unstale":"2024-09-17T07:13:54.949372+0000","last_undegraded":"2024-09-17T07:13:54.949372+0000","last_fullsized":"2024-09-17T07:13:54.949372+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'3","last_scrub_stamp":"2024-09-17T07:13:54.949345+0000","last_deep_scrub":"268'3","last_deep_scrub_stamp":"2024-09-17T07:13:54.949345+0000","last_clean_scrub_stamp":"2024-09-17T07:13:54.949345+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:15:04.643915+0000","scrub_duration":0.0091390389999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":279,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:50.914454+0000","last_change":"2024-09-17T07:13:50.914454+0000","last_active":"2024-09-17T07:13:50.914454+0000","last_peered":"2024-09-17T07:13:50.914454+0000","last_clean":"2024-09-17T07:13:50.914454+0000","last_became_active":"2024-09-17T07:02:12.004606+0000","last_became_peered":"2024-09-17T07:02:12.004606+0000","last_unstale":"2024-09-17T07:13:50.914454+0000","last_undegraded":"2024-09-17T07:13:50.914454+0000","last_fullsized":"2024-09-17T07:13:50.914454+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:50.914276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:13:50.914276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:15:16.796810+0000","scrub_duration":0.0071842440000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"212'130","reported_seq":518,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:49.351911+0000","last_change":"2024-09-17T07:12:23.270434+0000","last_active":"2024-09-17T07:13:49.351911+0000","last_peered":"2024-09-17T07:13:49.351911+0000","last_clean":"2024-09-17T07:13:49.351911+0000","last_became_active":"2024-09-17T07:02:03.020675+0000","last_became_peered":"2024-09-17T07:02:03.020675+0000","last_unstale":"2024-09-17T07:13:49.351911+0000","last_undegraded":"2024-09-17T07:13:49.351911+0000","last_fullsized":"2024-09-17T07:13:49.351911+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:12:23.270099+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:10:37.684115+0000","last_clean_scrub_stamp":"2024-09-17T07:12:23.270099+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032525319999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":264,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:49.352817+0000","last_change":"2024-09-17T07:13:04.014087+0000","last_active":"2024-09-17T07:13:49.352817+0000","last_peered":"2024-09-17T07:13:49.352817+0000","last_clean":"2024-09-17T07:13:49.352817+0000","last_became_active":"2024-09-17T07:02:16.016031+0000","last_became_peered":"2024-09-17T07:02:16.016031+0000","last_unstale":"2024-09-17T07:13:49.352817+0000","last_undegraded":"2024-09-17T07:13:49.352817+0000","last_fullsized":"2024-09-17T07:13:49.352817+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:04.013978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:13:04.013978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:22.588833+0000","scrub_duration":0.004290149,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":260,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:49.356646+0000","last_change":"2024-09-17T07:12:18.707829+0000","last_active":"2024-09-17T07:13:49.356646+0000","last_peered":"2024-09-17T07:13:49.356646+0000","last_clean":"2024-09-17T07:13:49.356646+0000","last_became_active":"2024-09-17T07:09:24.400041+0000","last_became_peered":"2024-09-17T07:09:24.400041+0000","last_unstale":"2024-09-17T07:13:49.356646+0000","last_undegraded":"2024-09-17T07:13:49.356646+0000","last_fullsized":"2024-09-17T07:13:49.356646+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:18.707704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:58.670571+0000","last_clean_scrub_stamp":"2024-09-17T07:12:18.707704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0052717149999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.7","version":"268'9509","reported_seq":9727,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-17T07:14:03.495193+0000","last_change":"2024-09-17T07:13:51.581427+0000","last_active":"2024-09-17T07:14:03.495193+0000","last_peered":"2024-09-17T07:14:03.495193+0000","last_clean":"2024-09-17T07:14:03.495193+0000","last_became_active":"2024-09-17T07:12:05.417969+0000","last_became_peered":"2024-09-17T07:12:05.417969+0000","last_unstale":"2024-09-17T07:14:03.495193+0000","last_undegraded":"2024-09-17T07:14:03.495193+0000","last_fullsized":"2024-09-17T07:14:03.495193+0000","mapping_epoch":236,"log_start":"253'4900","ondisk_log_start":"253'4900","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'8654","last_scrub_stamp":"2024-09-17T07:13:51.581134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_clean_scrub_stamp":"2024-09-17T07:13:51.581134+0000","objects_scrubbed":8284,"log_size":4609,"log_dups_size":0,"ondisk_log_size":4609,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:57.200245+0000","scrub_duration":3.7707355740000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":480706560,"num_objects":7335,"num_object_clones":0,"num_object_copies":22005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9509,"num_write_kb":539008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"242'35","reported_seq":334,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-17T07:13:57.396897+0000","last_change":"2024-09-17T07:13:57.397022+0000","last_active":"2024-09-17T07:13:57.396897+0000","last_peered":"2024-09-17T07:13:57.396897+0000","last_clean":"2024-09-17T07:13:57.396897+0000","last_became_active":"2024-09-17T07:02:14.027313+0000","last_became_peered":"2024-09-17T07:02:14.027313+0000","last_unstale":"2024-09-17T07:13:57.396897+0000","last_undegraded":"2024-09-17T07:13:57.396897+0000","last_fullsized":"2024-09-17T07:13:57.396897+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'35","last_scrub_stamp":"2024-09-17T07:13:57.396869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:13:57.396869+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:15:11.009865+0000","scrub_duration":0.0060406619999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":293,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:49.342405+0000","last_change":"2024-09-17T07:13:20.403259+0000","last_active":"2024-09-17T07:13:49.342405+0000","last_peered":"2024-09-17T07:13:49.342405+0000","last_clean":"2024-09-17T07:13:49.342405+0000","last_became_active":"2024-09-17T07:02:12.012037+0000","last_became_peered":"2024-09-17T07:02:12.012037+0000","last_unstale":"2024-09-17T07:13:49.342405+0000","last_undegraded":"2024-09-17T07:13:49.342405+0000","last_fullsized":"2024-09-17T07:13:49.342405+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:20.403127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:06:39.322535+0000","last_clean_scrub_stamp":"2024-09-17T07:13:20.403127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:23.286605+0000","scrub_duration":0.030498408000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":288,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:49.341755+0000","last_change":"2024-09-17T07:13:47.237776+0000","last_active":"2024-09-17T07:13:49.341755+0000","last_peered":"2024-09-17T07:13:49.341755+0000","last_clean":"2024-09-17T07:13:49.341755+0000","last_became_active":"2024-09-17T07:02:16.014631+0000","last_became_peered":"2024-09-17T07:02:16.014631+0000","last_unstale":"2024-09-17T07:13:49.341755+0000","last_undegraded":"2024-09-17T07:13:49.341755+0000","last_fullsized":"2024-09-17T07:13:49.341755+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:47.237658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:13:47.237658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:15:09.941611+0000","scrub_duration":0.0030177730000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":260,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:49.352024+0000","last_change":"2024-09-17T07:12:22.309666+0000","last_active":"2024-09-17T07:13:49.352024+0000","last_peered":"2024-09-17T07:13:49.352024+0000","last_clean":"2024-09-17T07:13:49.352024+0000","last_became_active":"2024-09-17T07:09:24.403904+0000","last_became_peered":"2024-09-17T07:09:24.403904+0000","last_unstale":"2024-09-17T07:13:49.352024+0000","last_undegraded":"2024-09-17T07:13:49.352024+0000","last_fullsized":"2024-09-17T07:13:49.352024+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:22.309233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:12:22.309233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0042909669999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.6","version":"268'9442","reported_seq":9655,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-17T07:14:03.459347+0000","last_change":"2024-09-17T07:13:51.557600+0000","last_active":"2024-09-17T07:14:03.459347+0000","last_peered":"2024-09-17T07:14:03.459347+0000","last_clean":"2024-09-17T07:14:03.459347+0000","last_became_active":"2024-09-17T07:12:05.416160+0000","last_became_peered":"2024-09-17T07:12:05.416160+0000","last_unstale":"2024-09-17T07:14:03.459347+0000","last_undegraded":"2024-09-17T07:14:03.459347+0000","last_fullsized":"2024-09-17T07:14:03.459347+0000","mapping_epoch":236,"log_start":"252'4800","ondisk_log_start":"252'4800","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'8613","last_scrub_stamp":"2024-09-17T07:13:51.557295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_clean_scrub_stamp":"2024-09-17T07:13:51.557295+0000","objects_scrubbed":8141,"log_size":4642,"log_dups_size":0,"ondisk_log_size":4642,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:54.172738+0000","scrub_duration":2.6637870449999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":475267072,"num_objects":7252,"num_object_clones":0,"num_object_copies":21756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9442,"num_write_kb":534208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"242'15","reported_seq":281,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:49.345005+0000","last_change":"2024-09-17T07:12:12.585443+0000","last_active":"2024-09-17T07:13:49.345005+0000","last_peered":"2024-09-17T07:13:49.345005+0000","last_clean":"2024-09-17T07:13:49.345005+0000","last_became_active":"2024-09-17T07:02:14.014939+0000","last_became_peered":"2024-09-17T07:02:14.014939+0000","last_unstale":"2024-09-17T07:13:49.345005+0000","last_undegraded":"2024-09-17T07:13:49.345005+0000","last_fullsized":"2024-09-17T07:13:49.345005+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:12:12.585348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:12:12.585348+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001887445,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":277,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-17T07:13:53.607222+0000","last_change":"2024-09-17T07:13:53.607329+0000","last_active":"2024-09-17T07:13:53.607222+0000","last_peered":"2024-09-17T07:13:53.607222+0000","last_clean":"2024-09-17T07:13:53.607222+0000","last_became_active":"2024-09-17T07:09:24.391207+0000","last_became_peered":"2024-09-17T07:09:24.391207+0000","last_unstale":"2024-09-17T07:13:53.607222+0000","last_undegraded":"2024-09-17T07:13:53.607222+0000","last_fullsized":"2024-09-17T07:13:53.607222+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:53.593078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:13:53.593078+0000","last_clean_scrub_stamp":"2024-09-17T07:13:53.593078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:57.667036+0000","scrub_duration":0.0025217759999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":229,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:49.344726+0000","last_change":"2024-09-17T07:13:18.562324+0000","last_active":"2024-09-17T07:13:49.344726+0000","last_peered":"2024-09-17T07:13:49.344726+0000","last_clean":"2024-09-17T07:13:49.344726+0000","last_became_active":"2024-09-17T07:09:24.396642+0000","last_became_peered":"2024-09-17T07:09:24.396642+0000","last_unstale":"2024-09-17T07:13:49.344726+0000","last_undegraded":"2024-09-17T07:13:49.344726+0000","last_fullsized":"2024-09-17T07:13:49.344726+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:18.562183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:13:18.562183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:19.864802+0000","scrub_duration":0.0046756590000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":252,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:49.344110+0000","last_change":"2024-09-17T07:13:02.638619+0000","last_active":"2024-09-17T07:13:49.344110+0000","last_peered":"2024-09-17T07:13:49.344110+0000","last_clean":"2024-09-17T07:13:49.344110+0000","last_became_active":"2024-09-17T07:02:18.026994+0000","last_became_peered":"2024-09-17T07:02:18.026994+0000","last_unstale":"2024-09-17T07:13:49.344110+0000","last_undegraded":"2024-09-17T07:13:49.344110+0000","last_fullsized":"2024-09-17T07:13:49.344110+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:02.638486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:58.099849+0000","last_clean_scrub_stamp":"2024-09-17T07:13:02.638486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:13.244812+0000","scrub_duration":0.0036574849999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.5","version":"268'9515","reported_seq":9732,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-17T07:14:04.333215+0000","last_change":"2024-09-17T07:13:49.219004+0000","last_active":"2024-09-17T07:14:04.333215+0000","last_peered":"2024-09-17T07:14:04.333215+0000","last_clean":"2024-09-17T07:14:04.333215+0000","last_became_active":"2024-09-17T07:12:05.176729+0000","last_became_peered":"2024-09-17T07:12:05.176729+0000","last_unstale":"2024-09-17T07:14:04.333215+0000","last_undegraded":"2024-09-17T07:14:04.333215+0000","last_fullsized":"2024-09-17T07:14:04.333215+0000","mapping_epoch":236,"log_start":"252'4500","ondisk_log_start":"252'4500","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"266'8455","last_scrub_stamp":"2024-09-17T07:13:49.218864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_clean_scrub_stamp":"2024-09-17T07:13:49.218864+0000","objects_scrubbed":8267,"log_size":5015,"log_dups_size":0,"ondisk_log_size":5015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:15:04.494765+0000","scrub_duration":2.6139698149999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":467861504,"num_objects":7139,"num_object_clones":0,"num_object_copies":21417,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7139,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9515,"num_write_kb":532928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"242'34","reported_seq":316,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:49.347336+0000","last_change":"2024-09-17T07:13:06.840286+0000","last_active":"2024-09-17T07:13:49.347336+0000","last_peered":"2024-09-17T07:13:49.347336+0000","last_clean":"2024-09-17T07:13:49.347336+0000","last_became_active":"2024-09-17T07:02:14.028392+0000","last_became_peered":"2024-09-17T07:02:14.028392+0000","last_unstale":"2024-09-17T07:13:49.347336+0000","last_undegraded":"2024-09-17T07:13:49.347336+0000","last_fullsized":"2024-09-17T07:13:49.347336+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'34","last_scrub_stamp":"2024-09-17T07:13:06.839894+0000","last_deep_scrub":"111'22","last_deep_scrub_stamp":"2024-09-17T07:04:57.500450+0000","last_clean_scrub_stamp":"2024-09-17T07:13:06.839894+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:16.089955+0000","scrub_duration":0.015966121999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'2","reported_seq":140,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-17T07:13:53.380098+0000","last_change":"2024-09-17T07:12:20.301388+0000","last_active":"2024-09-17T07:13:53.380098+0000","last_peered":"2024-09-17T07:13:53.380098+0000","last_clean":"2024-09-17T07:13:53.380098+0000","last_became_active":"2024-09-17T07:09:24.402909+0000","last_became_peered":"2024-09-17T07:09:24.402909+0000","last_unstale":"2024-09-17T07:13:53.380098+0000","last_undegraded":"2024-09-17T07:13:53.380098+0000","last_fullsized":"2024-09-17T07:13:53.380098+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-09-17T07:12:20.301263+0000","last_deep_scrub":"89'2","last_deep_scrub_stamp":"2024-09-17T07:12:20.301263+0000","last_clean_scrub_stamp":"2024-09-17T07:12:20.301263+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0065119280000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":420,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":420,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"225'9","reported_seq":407,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-17T07:13:55.668533+0000","last_change":"2024-09-17T07:11:25.292700+0000","last_active":"2024-09-17T07:13:55.668533+0000","last_peered":"2024-09-17T07:13:55.668533+0000","last_clean":"2024-09-17T07:13:55.668533+0000","last_became_active":"2024-09-17T07:02:16.015418+0000","last_became_peered":"2024-09-17T07:02:16.015418+0000","last_unstale":"2024-09-17T07:13:55.668533+0000","last_undegraded":"2024-09-17T07:13:55.668533+0000","last_fullsized":"2024-09-17T07:13:55.668533+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"225'9","last_scrub_stamp":"2024-09-17T07:11:25.292565+0000","last_deep_scrub":"225'9","last_deep_scrub_stamp":"2024-09-17T07:11:25.292565+0000","last_clean_scrub_stamp":"2024-09-17T07:11:25.292565+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0070251389999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":262,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:49.347956+0000","last_change":"2024-09-17T07:13:05.783707+0000","last_active":"2024-09-17T07:13:49.347956+0000","last_peered":"2024-09-17T07:13:49.347956+0000","last_clean":"2024-09-17T07:13:49.347956+0000","last_became_active":"2024-09-17T07:02:18.363253+0000","last_became_peered":"2024-09-17T07:02:18.363253+0000","last_unstale":"2024-09-17T07:13:49.347956+0000","last_undegraded":"2024-09-17T07:13:49.347956+0000","last_fullsized":"2024-09-17T07:13:49.347956+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:05.783527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:13:05.783527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:13.271530+0000","scrub_duration":0.0026723879999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.4","version":"268'9565","reported_seq":9606,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-17T07:14:03.719377+0000","last_change":"2024-09-17T07:12:05.169555+0000","last_active":"2024-09-17T07:14:03.719377+0000","last_peered":"2024-09-17T07:14:03.719377+0000","last_clean":"2024-09-17T07:14:03.719377+0000","last_became_active":"2024-09-17T07:12:05.169197+0000","last_became_peered":"2024-09-17T07:12:05.169197+0000","last_unstale":"2024-09-17T07:14:03.719377+0000","last_undegraded":"2024-09-17T07:14:03.719377+0000","last_fullsized":"2024-09-17T07:14:03.719377+0000","mapping_epoch":236,"log_start":"255'5200","ondisk_log_start":"255'5200","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:04.285115+0000","last_clean_scrub_stamp":"2024-09-17T07:12:04.285115+0000","objects_scrubbed":0,"log_size":4365,"log_dups_size":0,"ondisk_log_size":4365,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":479526912,"num_objects":7317,"num_object_clones":0,"num_object_copies":21951,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7317,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9565,"num_write_kb":540224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"242'23","reported_seq":347,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-17T07:13:59.558342+0000","last_change":"2024-09-17T07:13:59.558578+0000","last_active":"2024-09-17T07:13:59.558342+0000","last_peered":"2024-09-17T07:13:59.558342+0000","last_clean":"2024-09-17T07:13:59.558342+0000","last_became_active":"2024-09-17T07:10:49.723292+0000","last_became_peered":"2024-09-17T07:10:49.723292+0000","last_unstale":"2024-09-17T07:13:59.558342+0000","last_undegraded":"2024-09-17T07:13:59.558342+0000","last_fullsized":"2024-09-17T07:13:59.558342+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'23","last_scrub_stamp":"2024-09-17T07:13:59.558320+0000","last_deep_scrub":"150'17","last_deep_scrub_stamp":"2024-09-17T07:06:38.775788+0000","last_clean_scrub_stamp":"2024-09-17T07:13:59.558320+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:15:26.597564+0000","scrub_duration":0.0092780950000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"68'2","reported_seq":282,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:49.348544+0000","last_change":"2024-09-17T07:13:04.603384+0000","last_active":"2024-09-17T07:13:49.348544+0000","last_peered":"2024-09-17T07:13:49.348544+0000","last_clean":"2024-09-17T07:13:49.348544+0000","last_became_active":"2024-09-17T07:10:49.727577+0000","last_became_peered":"2024-09-17T07:10:49.727577+0000","last_unstale":"2024-09-17T07:13:49.348544+0000","last_undegraded":"2024-09-17T07:13:49.348544+0000","last_fullsized":"2024-09-17T07:13:49.348544+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-09-17T07:13:04.603238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:13:04.603238+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:26.025309+0000","scrub_duration":0.0053842450000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":229,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:49.343943+0000","last_change":"2024-09-17T07:13:20.617968+0000","last_active":"2024-09-17T07:13:49.343943+0000","last_peered":"2024-09-17T07:13:49.343943+0000","last_clean":"2024-09-17T07:13:49.343943+0000","last_became_active":"2024-09-17T07:04:15.111069+0000","last_became_peered":"2024-09-17T07:04:15.111069+0000","last_unstale":"2024-09-17T07:13:49.343943+0000","last_undegraded":"2024-09-17T07:13:49.343943+0000","last_fullsized":"2024-09-17T07:13:49.343943+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:20.617815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:11:17.479036+0000","last_clean_scrub_stamp":"2024-09-17T07:13:20.617815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:50.583241+0000","scrub_duration":0.0014980270000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":109,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-17T07:13:53.374476+0000","last_change":"2024-09-17T07:13:13.938089+0000","last_active":"2024-09-17T07:13:53.374476+0000","last_peered":"2024-09-17T07:13:53.374476+0000","last_clean":"2024-09-17T07:13:53.374476+0000","last_became_active":"2024-09-17T07:09:24.397669+0000","last_became_peered":"2024-09-17T07:09:24.397669+0000","last_unstale":"2024-09-17T07:13:53.374476+0000","last_undegraded":"2024-09-17T07:13:53.374476+0000","last_fullsized":"2024-09-17T07:13:53.374476+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:13.937979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:13:13.937979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:31.692708+0000","scrub_duration":0.018487173999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"242'15","reported_seq":310,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:51.542389+0000","last_change":"2024-09-17T07:13:51.542389+0000","last_active":"2024-09-17T07:13:51.542389+0000","last_peered":"2024-09-17T07:13:51.542389+0000","last_clean":"2024-09-17T07:13:51.542389+0000","last_became_active":"2024-09-17T07:09:24.408875+0000","last_became_peered":"2024-09-17T07:09:24.408875+0000","last_unstale":"2024-09-17T07:13:51.542389+0000","last_undegraded":"2024-09-17T07:13:51.542389+0000","last_fullsized":"2024-09-17T07:13:51.542389+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'15","last_scrub_stamp":"2024-09-17T07:13:51.542263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:13:51.542263+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:15:02.038752+0000","scrub_duration":0.0072768679999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":293,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-17T07:13:53.361873+0000","last_change":"2024-09-17T07:13:53.059080+0000","last_active":"2024-09-17T07:13:53.361873+0000","last_peered":"2024-09-17T07:13:53.361873+0000","last_clean":"2024-09-17T07:13:53.361873+0000","last_became_active":"2024-09-17T07:02:12.011327+0000","last_became_peered":"2024-09-17T07:02:12.011327+0000","last_unstale":"2024-09-17T07:13:53.361873+0000","last_undegraded":"2024-09-17T07:13:53.361873+0000","last_fullsized":"2024-09-17T07:13:53.361873+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:53.058820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:17.976267+0000","last_clean_scrub_stamp":"2024-09-17T07:13:53.058820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:15:17.871770+0000","scrub_duration":0.0016061910000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":96,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:49.351101+0000","last_change":"2024-09-17T07:12:18.367610+0000","last_active":"2024-09-17T07:13:49.351101+0000","last_peered":"2024-09-17T07:13:49.351101+0000","last_clean":"2024-09-17T07:13:49.351101+0000","last_became_active":"2024-09-17T07:09:24.407806+0000","last_became_peered":"2024-09-17T07:09:24.407806+0000","last_unstale":"2024-09-17T07:13:49.351101+0000","last_undegraded":"2024-09-17T07:13:49.351101+0000","last_fullsized":"2024-09-17T07:13:49.351101+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:18.367486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:18.367486+0000","last_clean_scrub_stamp":"2024-09-17T07:12:18.367486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0038802519999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":298,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-17T07:13:53.362312+0000","last_change":"2024-09-17T07:13:29.243604+0000","last_active":"2024-09-17T07:13:53.362312+0000","last_peered":"2024-09-17T07:13:53.362312+0000","last_clean":"2024-09-17T07:13:53.362312+0000","last_became_active":"2024-09-17T07:02:18.017825+0000","last_became_peered":"2024-09-17T07:02:18.017825+0000","last_unstale":"2024-09-17T07:13:53.362312+0000","last_undegraded":"2024-09-17T07:13:53.362312+0000","last_fullsized":"2024-09-17T07:13:53.362312+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:29.243451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:09:23.378805+0000","last_clean_scrub_stamp":"2024-09-17T07:13:29.243451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:50.025771+0000","scrub_duration":0.0060882920000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":281,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:53.303713+0000","last_change":"2024-09-17T07:13:49.568222+0000","last_active":"2024-09-17T07:13:53.303713+0000","last_peered":"2024-09-17T07:13:53.303713+0000","last_clean":"2024-09-17T07:13:53.303713+0000","last_became_active":"2024-09-17T07:02:12.008065+0000","last_became_peered":"2024-09-17T07:02:12.008065+0000","last_unstale":"2024-09-17T07:13:53.303713+0000","last_undegraded":"2024-09-17T07:13:53.303713+0000","last_fullsized":"2024-09-17T07:13:53.303713+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:49.566797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:13:49.566797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:15:16.056425+0000","scrub_duration":0.021302588000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"242'12","reported_seq":134,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:49.348305+0000","last_change":"2024-09-17T07:12:03.380136+0000","last_active":"2024-09-17T07:13:49.348305+0000","last_peered":"2024-09-17T07:13:49.348305+0000","last_clean":"2024-09-17T07:13:49.348305+0000","last_became_active":"2024-09-17T07:09:24.405385+0000","last_became_peered":"2024-09-17T07:09:24.405385+0000","last_unstale":"2024-09-17T07:13:49.348305+0000","last_undegraded":"2024-09-17T07:13:49.348305+0000","last_fullsized":"2024-09-17T07:13:49.348305+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'6","last_scrub_stamp":"2024-09-17T07:12:03.380040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:12:03.380040+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019381719999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":272,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:49.350308+0000","last_change":"2024-09-17T07:13:19.780144+0000","last_active":"2024-09-17T07:13:49.350308+0000","last_peered":"2024-09-17T07:13:49.350308+0000","last_clean":"2024-09-17T07:13:49.350308+0000","last_became_active":"2024-09-17T07:09:24.398922+0000","last_became_peered":"2024-09-17T07:09:24.398922+0000","last_unstale":"2024-09-17T07:13:49.350308+0000","last_undegraded":"2024-09-17T07:13:49.350308+0000","last_fullsized":"2024-09-17T07:13:49.350308+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:19.780027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:13:19.780027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:34.795763+0000","scrub_duration":0.0044369229999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":250,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:49.348335+0000","last_change":"2024-09-17T07:12:06.344966+0000","last_active":"2024-09-17T07:13:49.348335+0000","last_peered":"2024-09-17T07:13:49.348335+0000","last_clean":"2024-09-17T07:13:49.348335+0000","last_became_active":"2024-09-17T07:02:18.365419+0000","last_became_peered":"2024-09-17T07:02:18.365419+0000","last_unstale":"2024-09-17T07:13:49.348335+0000","last_undegraded":"2024-09-17T07:13:49.348335+0000","last_fullsized":"2024-09-17T07:13:49.348335+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:06.344833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:12:06.344833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002302817,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"242'20","reported_seq":151,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:49.350631+0000","last_change":"2024-09-17T07:12:12.288945+0000","last_active":"2024-09-17T07:13:49.350631+0000","last_peered":"2024-09-17T07:13:49.350631+0000","last_clean":"2024-09-17T07:13:49.350631+0000","last_became_active":"2024-09-17T07:09:24.405944+0000","last_became_peered":"2024-09-17T07:09:24.405944+0000","last_unstale":"2024-09-17T07:13:49.350631+0000","last_undegraded":"2024-09-17T07:13:49.350631+0000","last_fullsized":"2024-09-17T07:13:49.350631+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-09-17T07:12:12.288864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:12:12.288864+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0020069329999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":288,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-17T07:13:55.971118+0000","last_change":"2024-09-17T07:13:55.972896+0000","last_active":"2024-09-17T07:13:55.971118+0000","last_peered":"2024-09-17T07:13:55.971118+0000","last_clean":"2024-09-17T07:13:55.971118+0000","last_became_active":"2024-09-17T07:09:24.402355+0000","last_became_peered":"2024-09-17T07:09:24.402355+0000","last_unstale":"2024-09-17T07:13:55.971118+0000","last_undegraded":"2024-09-17T07:13:55.971118+0000","last_fullsized":"2024-09-17T07:13:55.971118+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:55.971099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:13:55.971099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:15:09.440558+0000","scrub_duration":0.0068125299999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"163'9","reported_seq":414,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-17T07:13:55.666220+0000","last_change":"2024-09-17T07:13:22.888883+0000","last_active":"2024-09-17T07:13:55.666220+0000","last_peered":"2024-09-17T07:13:55.666220+0000","last_clean":"2024-09-17T07:13:55.666220+0000","last_became_active":"2024-09-17T07:02:16.022583+0000","last_became_peered":"2024-09-17T07:02:16.022583+0000","last_unstale":"2024-09-17T07:13:55.666220+0000","last_undegraded":"2024-09-17T07:13:55.666220+0000","last_fullsized":"2024-09-17T07:13:55.666220+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'9","last_scrub_stamp":"2024-09-17T07:13:22.888752+0000","last_deep_scrub":"163'9","last_deep_scrub_stamp":"2024-09-17T07:07:34.933808+0000","last_clean_scrub_stamp":"2024-09-17T07:13:22.888752+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:33.311423+0000","scrub_duration":0.019535219999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":260,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:49.343900+0000","last_change":"2024-09-17T07:12:26.619980+0000","last_active":"2024-09-17T07:13:49.343900+0000","last_peered":"2024-09-17T07:13:49.343900+0000","last_clean":"2024-09-17T07:13:49.343900+0000","last_became_active":"2024-09-17T07:09:24.409051+0000","last_became_peered":"2024-09-17T07:09:24.409051+0000","last_unstale":"2024-09-17T07:13:49.343900+0000","last_undegraded":"2024-09-17T07:13:49.343900+0000","last_fullsized":"2024-09-17T07:13:49.343900+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:26.619893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:26.619893+0000","last_clean_scrub_stamp":"2024-09-17T07:12:26.619893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0044029580000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"242'15","reported_seq":309,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:49.357583+0000","last_change":"2024-09-17T07:12:10.828225+0000","last_active":"2024-09-17T07:13:49.357583+0000","last_peered":"2024-09-17T07:13:49.357583+0000","last_clean":"2024-09-17T07:13:49.357583+0000","last_became_active":"2024-09-17T07:09:24.406333+0000","last_became_peered":"2024-09-17T07:09:24.406333+0000","last_unstale":"2024-09-17T07:13:49.357583+0000","last_undegraded":"2024-09-17T07:13:49.357583+0000","last_fullsized":"2024-09-17T07:13:49.357583+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-09-17T07:12:10.828114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:12:10.828114+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001735435,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":267,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-17T07:13:53.370100+0000","last_change":"2024-09-17T07:12:23.597448+0000","last_active":"2024-09-17T07:13:53.370100+0000","last_peered":"2024-09-17T07:13:53.370100+0000","last_clean":"2024-09-17T07:13:53.370100+0000","last_became_active":"2024-09-17T07:09:24.402704+0000","last_became_peered":"2024-09-17T07:09:24.402704+0000","last_unstale":"2024-09-17T07:13:53.370100+0000","last_undegraded":"2024-09-17T07:13:53.370100+0000","last_fullsized":"2024-09-17T07:13:53.370100+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:23.597330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:12:23.597330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032745959999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"196'9","reported_seq":188,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-17T07:13:55.664065+0000","last_change":"2024-09-17T07:13:10.965977+0000","last_active":"2024-09-17T07:13:55.664065+0000","last_peered":"2024-09-17T07:13:55.664065+0000","last_clean":"2024-09-17T07:13:55.664065+0000","last_became_active":"2024-09-17T07:09:24.406267+0000","last_became_peered":"2024-09-17T07:09:24.406267+0000","last_unstale":"2024-09-17T07:13:55.664065+0000","last_undegraded":"2024-09-17T07:13:55.664065+0000","last_fullsized":"2024-09-17T07:13:55.664065+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'9","last_scrub_stamp":"2024-09-17T07:13:10.965855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:13:10.965855+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:38.419605+0000","scrub_duration":0.0088433720000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":252,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:49.346301+0000","last_change":"2024-09-17T07:12:16.543360+0000","last_active":"2024-09-17T07:13:49.346301+0000","last_peered":"2024-09-17T07:13:49.346301+0000","last_clean":"2024-09-17T07:13:49.346301+0000","last_became_active":"2024-09-17T07:02:18.018716+0000","last_became_peered":"2024-09-17T07:02:18.018716+0000","last_unstale":"2024-09-17T07:13:49.346301+0000","last_undegraded":"2024-09-17T07:13:49.346301+0000","last_fullsized":"2024-09-17T07:13:49.346301+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:16.543257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:12:16.543257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001567354,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"242'15","reported_seq":291,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-17T07:13:53.366053+0000","last_change":"2024-09-17T07:13:48.500828+0000","last_active":"2024-09-17T07:13:53.366053+0000","last_peered":"2024-09-17T07:13:53.366053+0000","last_clean":"2024-09-17T07:13:53.366053+0000","last_became_active":"2024-09-17T07:02:14.010289+0000","last_became_peered":"2024-09-17T07:02:14.010289+0000","last_unstale":"2024-09-17T07:13:53.366053+0000","last_undegraded":"2024-09-17T07:13:53.366053+0000","last_fullsized":"2024-09-17T07:13:53.366053+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'15","last_scrub_stamp":"2024-09-17T07:13:48.500257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:13:48.500257+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:15:08.277886+0000","scrub_duration":0.0054642730000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":289,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-17T07:13:53.375933+0000","last_change":"2024-09-17T07:13:20.150502+0000","last_active":"2024-09-17T07:13:53.375933+0000","last_peered":"2024-09-17T07:13:53.375933+0000","last_clean":"2024-09-17T07:13:53.375933+0000","last_became_active":"2024-09-17T07:09:24.408139+0000","last_became_peered":"2024-09-17T07:09:24.408139+0000","last_unstale":"2024-09-17T07:13:53.375933+0000","last_undegraded":"2024-09-17T07:13:53.375933+0000","last_fullsized":"2024-09-17T07:13:53.375933+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:20.150317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:10:36.334827+0000","last_clean_scrub_stamp":"2024-09-17T07:13:20.150317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:40.903864+0000","scrub_duration":0.0030784160000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":109,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-17T07:13:53.369784+0000","last_change":"2024-09-17T07:13:08.661580+0000","last_active":"2024-09-17T07:13:53.369784+0000","last_peered":"2024-09-17T07:13:53.369784+0000","last_clean":"2024-09-17T07:13:53.369784+0000","last_became_active":"2024-09-17T07:09:24.399116+0000","last_became_peered":"2024-09-17T07:09:24.399116+0000","last_unstale":"2024-09-17T07:13:53.369784+0000","last_undegraded":"2024-09-17T07:13:53.369784+0000","last_fullsized":"2024-09-17T07:13:53.369784+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:08.661455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:13:08.661455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:25.169159+0000","scrub_duration":0.0032948439999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":263,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-17T07:13:53.369449+0000","last_change":"2024-09-17T07:13:21.583247+0000","last_active":"2024-09-17T07:13:53.369449+0000","last_peered":"2024-09-17T07:13:53.369449+0000","last_clean":"2024-09-17T07:13:53.369449+0000","last_became_active":"2024-09-17T07:02:18.027622+0000","last_became_peered":"2024-09-17T07:02:18.027622+0000","last_unstale":"2024-09-17T07:13:53.369449+0000","last_undegraded":"2024-09-17T07:13:53.369449+0000","last_fullsized":"2024-09-17T07:13:53.369449+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:21.583164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:13:21.583164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:26.090061+0000","scrub_duration":0.0016901679999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"242'19","reported_seq":288,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-17T07:13:53.369564+0000","last_change":"2024-09-17T07:12:13.566691+0000","last_active":"2024-09-17T07:13:53.369564+0000","last_peered":"2024-09-17T07:13:53.369564+0000","last_clean":"2024-09-17T07:13:53.369564+0000","last_became_active":"2024-09-17T07:02:14.016255+0000","last_became_peered":"2024-09-17T07:02:14.016255+0000","last_unstale":"2024-09-17T07:13:53.369564+0000","last_undegraded":"2024-09-17T07:13:53.369564+0000","last_fullsized":"2024-09-17T07:13:53.369564+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-09-17T07:12:13.566608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:12:13.566608+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019196129999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"94'2","reported_seq":262,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:49.351007+0000","last_change":"2024-09-17T07:13:04.126170+0000","last_active":"2024-09-17T07:13:49.351007+0000","last_peered":"2024-09-17T07:13:49.351007+0000","last_clean":"2024-09-17T07:13:49.351007+0000","last_became_active":"2024-09-17T07:02:12.011461+0000","last_became_peered":"2024-09-17T07:02:12.011461+0000","last_unstale":"2024-09-17T07:13:49.351007+0000","last_undegraded":"2024-09-17T07:13:49.351007+0000","last_fullsized":"2024-09-17T07:13:49.351007+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-09-17T07:13:04.125999+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-09-17T07:07:31.717888+0000","last_clean_scrub_stamp":"2024-09-17T07:13:04.125999+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:16.465038+0000","scrub_duration":0.0036426029999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":262,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-17T07:13:53.365925+0000","last_change":"2024-09-17T07:12:23.390704+0000","last_active":"2024-09-17T07:13:53.365925+0000","last_peered":"2024-09-17T07:13:53.365925+0000","last_clean":"2024-09-17T07:13:53.365925+0000","last_became_active":"2024-09-17T07:09:24.407362+0000","last_became_peered":"2024-09-17T07:09:24.407362+0000","last_unstale":"2024-09-17T07:13:53.365925+0000","last_undegraded":"2024-09-17T07:13:53.365925+0000","last_fullsized":"2024-09-17T07:13:53.365925+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:23.390587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:12:23.390587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010514658,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":108,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-17T07:13:53.369656+0000","last_change":"2024-09-17T07:13:23.553834+0000","last_active":"2024-09-17T07:13:53.369656+0000","last_peered":"2024-09-17T07:13:53.369656+0000","last_clean":"2024-09-17T07:13:53.369656+0000","last_became_active":"2024-09-17T07:09:24.400791+0000","last_became_peered":"2024-09-17T07:09:24.400791+0000","last_unstale":"2024-09-17T07:13:53.369656+0000","last_undegraded":"2024-09-17T07:13:53.369656+0000","last_fullsized":"2024-09-17T07:13:53.369656+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:23.553740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:13:23.553740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:38.946109+0000","scrub_duration":0.0041723180000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"242'8","reported_seq":170,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:49.341250+0000","last_change":"2024-09-17T07:13:05.267715+0000","last_active":"2024-09-17T07:13:49.341250+0000","last_peered":"2024-09-17T07:13:49.341250+0000","last_clean":"2024-09-17T07:13:49.341250+0000","last_became_active":"2024-09-17T07:09:24.396368+0000","last_became_peered":"2024-09-17T07:09:24.396368+0000","last_unstale":"2024-09-17T07:13:49.341250+0000","last_undegraded":"2024-09-17T07:13:49.341250+0000","last_fullsized":"2024-09-17T07:13:49.341250+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'8","last_scrub_stamp":"2024-09-17T07:13:05.267508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:13:05.267508+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:31.598473+0000","scrub_duration":0.0058992599999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":267,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:49.350736+0000","last_change":"2024-09-17T07:13:27.876879+0000","last_active":"2024-09-17T07:13:49.350736+0000","last_peered":"2024-09-17T07:13:49.350736+0000","last_clean":"2024-09-17T07:13:49.350736+0000","last_became_active":"2024-09-17T07:02:12.008898+0000","last_became_peered":"2024-09-17T07:02:12.008898+0000","last_unstale":"2024-09-17T07:13:49.350736+0000","last_undegraded":"2024-09-17T07:13:49.350736+0000","last_fullsized":"2024-09-17T07:13:49.350736+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:27.876490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:13:27.876490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:51.317177+0000","scrub_duration":0.007015673,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":266,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-17T07:13:53.375072+0000","last_change":"2024-09-17T07:13:05.966082+0000","last_active":"2024-09-17T07:13:53.375072+0000","last_peered":"2024-09-17T07:13:53.375072+0000","last_clean":"2024-09-17T07:13:53.375072+0000","last_became_active":"2024-09-17T07:02:16.014982+0000","last_became_peered":"2024-09-17T07:02:16.014982+0000","last_unstale":"2024-09-17T07:13:53.375072+0000","last_undegraded":"2024-09-17T07:13:53.375072+0000","last_fullsized":"2024-09-17T07:13:53.375072+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:05.965827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:47.444972+0000","last_clean_scrub_stamp":"2024-09-17T07:13:05.965827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:19.035886+0000","scrub_duration":0.024001009,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":262,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-17T07:13:59.042437+0000","last_change":"2024-09-17T07:13:59.042437+0000","last_active":"2024-09-17T07:13:59.042437+0000","last_peered":"2024-09-17T07:13:59.042437+0000","last_clean":"2024-09-17T07:13:59.042437+0000","last_became_active":"2024-09-17T07:02:18.021014+0000","last_became_peered":"2024-09-17T07:02:18.021014+0000","last_unstale":"2024-09-17T07:13:59.042437+0000","last_undegraded":"2024-09-17T07:13:59.042437+0000","last_fullsized":"2024-09-17T07:13:59.042437+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:59.042322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:13:59.042322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:15:00.217092+0000","scrub_duration":0.0076271120000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"242'19","reported_seq":162,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:49.351416+0000","last_change":"2024-09-17T07:13:45.076965+0000","last_active":"2024-09-17T07:13:49.351416+0000","last_peered":"2024-09-17T07:13:49.351416+0000","last_clean":"2024-09-17T07:13:49.351416+0000","last_became_active":"2024-09-17T07:09:24.404942+0000","last_became_peered":"2024-09-17T07:09:24.404942+0000","last_unstale":"2024-09-17T07:13:49.351416+0000","last_undegraded":"2024-09-17T07:13:49.351416+0000","last_fullsized":"2024-09-17T07:13:49.351416+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'19","last_scrub_stamp":"2024-09-17T07:13:45.076322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:13:45.076322+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:15:14.843078+0000","scrub_duration":0.0088383200000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":279,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:49.355398+0000","last_change":"2024-09-17T07:13:14.042792+0000","last_active":"2024-09-17T07:13:49.355398+0000","last_peered":"2024-09-17T07:13:49.355398+0000","last_clean":"2024-09-17T07:13:49.355398+0000","last_became_active":"2024-09-17T07:02:12.010195+0000","last_became_peered":"2024-09-17T07:02:12.010195+0000","last_unstale":"2024-09-17T07:13:49.355398+0000","last_undegraded":"2024-09-17T07:13:49.355398+0000","last_fullsized":"2024-09-17T07:13:49.355398+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:14.042643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:13:14.042643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:37.124641+0000","scrub_duration":0.013121302,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":264,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-17T07:13:53.373520+0000","last_change":"2024-09-17T07:13:47.027302+0000","last_active":"2024-09-17T07:13:53.373520+0000","last_peered":"2024-09-17T07:13:53.373520+0000","last_clean":"2024-09-17T07:13:53.373520+0000","last_became_active":"2024-09-17T07:02:16.016183+0000","last_became_peered":"2024-09-17T07:02:16.016183+0000","last_unstale":"2024-09-17T07:13:53.373520+0000","last_undegraded":"2024-09-17T07:13:53.373520+0000","last_fullsized":"2024-09-17T07:13:53.373520+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:47.027189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:13:47.027189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:48.242739+0000","scrub_duration":0.0024966390000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":108,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-17T07:13:53.369710+0000","last_change":"2024-09-17T07:13:06.639710+0000","last_active":"2024-09-17T07:13:53.369710+0000","last_peered":"2024-09-17T07:13:53.369710+0000","last_clean":"2024-09-17T07:13:53.369710+0000","last_became_active":"2024-09-17T07:09:24.392216+0000","last_became_peered":"2024-09-17T07:09:24.392216+0000","last_unstale":"2024-09-17T07:13:53.369710+0000","last_undegraded":"2024-09-17T07:13:53.369710+0000","last_fullsized":"2024-09-17T07:13:53.369710+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:06.639268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:13:06.639268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:07.756802+0000","scrub_duration":0.0018494749999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1c","version":"75'1","reported_seq":262,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-17T07:13:56.976193+0000","last_change":"2024-09-17T07:13:56.976301+0000","last_active":"2024-09-17T07:13:56.976193+0000","last_peered":"2024-09-17T07:13:56.976193+0000","last_clean":"2024-09-17T07:13:56.976193+0000","last_became_active":"2024-09-17T07:02:18.021855+0000","last_became_peered":"2024-09-17T07:02:18.021855+0000","last_unstale":"2024-09-17T07:13:56.976193+0000","last_undegraded":"2024-09-17T07:13:56.976193+0000","last_fullsized":"2024-09-17T07:13:56.976193+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-17T07:13:56.976159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:13:56.976159+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:15:26.442712+0000","scrub_duration":0.0053255050000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":268,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:49.347418+0000","last_change":"2024-09-17T07:13:18.740495+0000","last_active":"2024-09-17T07:13:49.347418+0000","last_peered":"2024-09-17T07:13:49.347418+0000","last_clean":"2024-09-17T07:13:49.347418+0000","last_became_active":"2024-09-17T07:02:12.005329+0000","last_became_peered":"2024-09-17T07:02:12.005329+0000","last_unstale":"2024-09-17T07:13:49.347418+0000","last_undegraded":"2024-09-17T07:13:49.347418+0000","last_fullsized":"2024-09-17T07:13:49.347418+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:18.740322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:13:18.740322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:40.698190+0000","scrub_duration":0.0045334520000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"242'20","reported_seq":339,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:52.546409+0000","last_change":"2024-09-17T07:13:52.546409+0000","last_active":"2024-09-17T07:13:52.546409+0000","last_peered":"2024-09-17T07:13:52.546409+0000","last_clean":"2024-09-17T07:13:52.546409+0000","last_became_active":"2024-09-17T07:10:49.730948+0000","last_became_peered":"2024-09-17T07:10:49.730948+0000","last_unstale":"2024-09-17T07:13:52.546409+0000","last_undegraded":"2024-09-17T07:13:52.546409+0000","last_fullsized":"2024-09-17T07:13:52.546409+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'20","last_scrub_stamp":"2024-09-17T07:13:52.546261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:13:52.546261+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:15:05.373826+0000","scrub_duration":0.0080850769999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"75'8","reported_seq":419,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-17T07:13:55.662875+0000","last_change":"2024-09-17T07:13:16.083435+0000","last_active":"2024-09-17T07:13:55.662875+0000","last_peered":"2024-09-17T07:13:55.662875+0000","last_clean":"2024-09-17T07:13:55.662875+0000","last_became_active":"2024-09-17T07:02:16.020560+0000","last_became_peered":"2024-09-17T07:02:16.020560+0000","last_unstale":"2024-09-17T07:13:55.662875+0000","last_undegraded":"2024-09-17T07:13:55.662875+0000","last_fullsized":"2024-09-17T07:13:55.662875+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:13:16.083313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:13:16.083313+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:36.805408+0000","scrub_duration":0.0026743069999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":250,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:49.350572+0000","last_change":"2024-09-17T07:12:21.397606+0000","last_active":"2024-09-17T07:13:49.350572+0000","last_peered":"2024-09-17T07:13:49.350572+0000","last_clean":"2024-09-17T07:13:49.350572+0000","last_became_active":"2024-09-17T07:02:18.021603+0000","last_became_peered":"2024-09-17T07:02:18.021603+0000","last_unstale":"2024-09-17T07:13:49.350572+0000","last_undegraded":"2024-09-17T07:13:49.350572+0000","last_fullsized":"2024-09-17T07:13:49.350572+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:21.397491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:12:21.397491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045805669999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":119,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:49.364311+0000","last_change":"2024-09-17T07:13:44.383770+0000","last_active":"2024-09-17T07:13:49.364311+0000","last_peered":"2024-09-17T07:13:49.364311+0000","last_clean":"2024-09-17T07:13:49.364311+0000","last_became_active":"2024-09-17T07:09:24.403360+0000","last_became_peered":"2024-09-17T07:09:24.403360+0000","last_unstale":"2024-09-17T07:13:49.364311+0000","last_undegraded":"2024-09-17T07:13:49.364311+0000","last_fullsized":"2024-09-17T07:13:49.364311+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:44.383642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:10:29.325664+0000","last_clean_scrub_stamp":"2024-09-17T07:13:44.383642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:15:07.947843+0000","scrub_duration":0.0021007909999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"242'19","reported_seq":351,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-17T07:13:54.506576+0000","last_change":"2024-09-17T07:13:54.506697+0000","last_active":"2024-09-17T07:13:54.506576+0000","last_peered":"2024-09-17T07:13:54.506576+0000","last_clean":"2024-09-17T07:13:54.506576+0000","last_became_active":"2024-09-17T07:03:48.760390+0000","last_became_peered":"2024-09-17T07:03:48.760390+0000","last_unstale":"2024-09-17T07:13:54.506576+0000","last_undegraded":"2024-09-17T07:13:54.506576+0000","last_fullsized":"2024-09-17T07:13:54.506576+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'19","last_scrub_stamp":"2024-09-17T07:13:54.506548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:13:54.506548+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:15:09.858340+0000","scrub_duration":0.0060171460000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":254,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:49.344482+0000","last_change":"2024-09-17T07:13:04.609663+0000","last_active":"2024-09-17T07:13:49.344482+0000","last_peered":"2024-09-17T07:13:49.344482+0000","last_clean":"2024-09-17T07:13:49.344482+0000","last_became_active":"2024-09-17T07:02:16.019701+0000","last_became_peered":"2024-09-17T07:02:16.019701+0000","last_unstale":"2024-09-17T07:13:49.344482+0000","last_undegraded":"2024-09-17T07:13:49.344482+0000","last_fullsized":"2024-09-17T07:13:49.344482+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:04.609543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:13:04.609543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:07.171894+0000","scrub_duration":0.012760591,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":253,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:49.346006+0000","last_change":"2024-09-17T07:12:27.469985+0000","last_active":"2024-09-17T07:13:49.346006+0000","last_peered":"2024-09-17T07:13:49.346006+0000","last_clean":"2024-09-17T07:13:49.346006+0000","last_became_active":"2024-09-17T07:02:18.363752+0000","last_became_peered":"2024-09-17T07:02:18.363752+0000","last_unstale":"2024-09-17T07:13:49.346006+0000","last_undegraded":"2024-09-17T07:13:49.346006+0000","last_fullsized":"2024-09-17T07:13:49.346006+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:27.469698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:12:27.469698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0056145910000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":309,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:52.510459+0000","last_change":"2024-09-17T07:13:52.510459+0000","last_active":"2024-09-17T07:13:52.510459+0000","last_peered":"2024-09-17T07:13:52.510459+0000","last_clean":"2024-09-17T07:13:52.510459+0000","last_became_active":"2024-09-17T07:03:48.759384+0000","last_became_peered":"2024-09-17T07:03:48.759384+0000","last_unstale":"2024-09-17T07:13:52.510459+0000","last_undegraded":"2024-09-17T07:13:52.510459+0000","last_fullsized":"2024-09-17T07:13:52.510459+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:52.510313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:11:18.559877+0000","last_clean_scrub_stamp":"2024-09-17T07:13:52.510313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:15:07.519418+0000","scrub_duration":0.0042802059999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"242'27","reported_seq":309,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:49.350768+0000","last_change":"2024-09-17T07:13:09.143615+0000","last_active":"2024-09-17T07:13:49.350768+0000","last_peered":"2024-09-17T07:13:49.350768+0000","last_clean":"2024-09-17T07:13:49.350768+0000","last_became_active":"2024-09-17T07:09:24.403030+0000","last_became_peered":"2024-09-17T07:09:24.403030+0000","last_unstale":"2024-09-17T07:13:49.350768+0000","last_undegraded":"2024-09-17T07:13:49.350768+0000","last_fullsized":"2024-09-17T07:13:49.350768+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'27","last_scrub_stamp":"2024-09-17T07:13:09.143488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:13:09.143488+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:15.278478+0000","scrub_duration":0.0029473339999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":263,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-17T07:14:00.509306+0000","last_change":"2024-09-17T07:14:00.509698+0000","last_active":"2024-09-17T07:14:00.509306+0000","last_peered":"2024-09-17T07:14:00.509306+0000","last_clean":"2024-09-17T07:14:00.509306+0000","last_became_active":"2024-09-17T07:02:16.015903+0000","last_became_peered":"2024-09-17T07:02:16.015903+0000","last_unstale":"2024-09-17T07:14:00.509306+0000","last_undegraded":"2024-09-17T07:14:00.509306+0000","last_fullsized":"2024-09-17T07:14:00.509306+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:00.509283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:25.794418+0000","last_clean_scrub_stamp":"2024-09-17T07:14:00.509283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:15:05.250735+0000","scrub_duration":0.0073526269999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":121,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:49.355231+0000","last_change":"2024-09-17T07:13:29.181828+0000","last_active":"2024-09-17T07:13:49.355231+0000","last_peered":"2024-09-17T07:13:49.355231+0000","last_clean":"2024-09-17T07:13:49.355231+0000","last_became_active":"2024-09-17T07:09:24.402411+0000","last_became_peered":"2024-09-17T07:09:24.402411+0000","last_unstale":"2024-09-17T07:13:49.355231+0000","last_undegraded":"2024-09-17T07:13:49.355231+0000","last_fullsized":"2024-09-17T07:13:49.355231+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:29.181697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:13:29.181697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:37.812878+0000","scrub_duration":0.0069966179999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":109,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:49.352714+0000","last_change":"2024-09-17T07:13:15.914281+0000","last_active":"2024-09-17T07:13:49.352714+0000","last_peered":"2024-09-17T07:13:49.352714+0000","last_clean":"2024-09-17T07:13:49.352714+0000","last_became_active":"2024-09-17T07:09:24.404795+0000","last_became_peered":"2024-09-17T07:09:24.404795+0000","last_unstale":"2024-09-17T07:13:49.352714+0000","last_undegraded":"2024-09-17T07:13:49.352714+0000","last_fullsized":"2024-09-17T07:13:49.352714+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:15.914144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:13:15.914144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:29.135207+0000","scrub_duration":0.059987944000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"242'23","reported_seq":171,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:49.350881+0000","last_change":"2024-09-17T07:12:24.341244+0000","last_active":"2024-09-17T07:13:49.350881+0000","last_peered":"2024-09-17T07:13:49.350881+0000","last_clean":"2024-09-17T07:13:49.350881+0000","last_became_active":"2024-09-17T07:09:24.394476+0000","last_became_peered":"2024-09-17T07:09:24.394476+0000","last_unstale":"2024-09-17T07:13:49.350881+0000","last_undegraded":"2024-09-17T07:13:49.350881+0000","last_fullsized":"2024-09-17T07:13:49.350881+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'23","last_scrub_stamp":"2024-09-17T07:12:24.341132+0000","last_deep_scrub":"86'17","last_deep_scrub_stamp":"2024-09-17T07:03:18.241067+0000","last_clean_scrub_stamp":"2024-09-17T07:12:24.341132+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0043568469999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":274,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-17T07:13:49.345609+0000","last_change":"2024-09-17T07:13:20.568579+0000","last_active":"2024-09-17T07:13:49.345609+0000","last_peered":"2024-09-17T07:13:49.345609+0000","last_clean":"2024-09-17T07:13:49.345609+0000","last_became_active":"2024-09-17T07:09:24.393329+0000","last_became_peered":"2024-09-17T07:09:24.393329+0000","last_unstale":"2024-09-17T07:13:49.345609+0000","last_undegraded":"2024-09-17T07:13:49.345609+0000","last_fullsized":"2024-09-17T07:13:49.345609+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:20.568314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:13:20.568314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:46.285324+0000","scrub_duration":0.016901157999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":7615086616,"num_objects":116198,"num_object_clones":0,"num_object_copies":348594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":116198,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":152330,"num_write_kb":8592833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":22868865024,"data_stored":22868852808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74630,"ondisk_log_size":74630,"up":48,"acting":48,"num_store_stats":7},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70,"ondisk_log_size":70,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1121,"num_read_kb":878,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":109,"num_bytes_recovered":2228,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":13004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":719,"ondisk_log_size":719,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":704,"num_read_kb":699,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1386,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":12,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":4578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037908,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5683080,"kb_used_data":5156416,"kb_used_omap":1,"kb_used_meta":526590,"kb_avail":88062072,"statfs":{"total":95995035648,"available":90175561728,"internally_reserved":0,"allocated":5280169984,"data_stored":5279816664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":539228619},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:13:23 2024","interfaces":[{"interface":"back","average":{"1min":1.3320000000000001,"5min":1.464,"15min":1.4850000000000001},"min":{"1min":0.22600000000000001,"5min":0.20599999999999999,"15min":0.192},"max":{"1min":6.2489999999999997,"5min":41.783999999999999,"15min":52.478999999999999},"last":0.48999999999999999},{"interface":"front","average":{"1min":1.069,"5min":1.762,"15min":1.5760000000000001},"min":{"1min":0.255,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":3.75,"5min":42.149999999999999,"15min":52.871000000000002},"last":0.40799999999999997}]},{"osd":1,"last update":"Tue Sep 17 07:13:23 2024","interfaces":[{"interface":"back","average":{"1min":2.4140000000000001,"5min":2.5590000000000002,"15min":1.268},"min":{"1min":0.28100000000000003,"5min":0.223,"15min":0.223},"max":{"1min":15.801,"5min":73.549000000000007,"15min":73.549000000000007},"last":0.46200000000000002},{"interface":"front","average":{"1min":1.659,"5min":2.5049999999999999,"15min":1.2669999999999999},"min":{"1min":0.27800000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":13.989000000000001,"5min":73.116,"15min":73.116},"last":0.43099999999999999}]},{"osd":2,"last update":"Tue Sep 17 07:13:23 2024","interfaces":[{"interface":"back","average":{"1min":1.331,"5min":1.056,"15min":1.02},"min":{"1min":0.26800000000000002,"5min":0.26600000000000001,"15min":0.23899999999999999},"max":{"1min":4.8460000000000001,"5min":6.0090000000000003,"15min":52.055999999999997},"last":0.50700000000000001},{"interface":"front","average":{"1min":1.365,"5min":0.93100000000000005,"15min":1.0209999999999999},"min":{"1min":0.33000000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":6.2880000000000003,"5min":6.2880000000000003,"15min":52.064999999999998},"last":0.51900000000000002}]},{"osd":3,"last update":"Tue Sep 17 07:13:23 2024","interfaces":[{"interface":"back","average":{"1min":1.5069999999999999,"5min":1.26,"15min":1.22},"min":{"1min":0.312,"5min":0.29499999999999998,"15min":0.28799999999999998},"max":{"1min":5.835,"5min":11.167,"15min":51.689},"last":0.55600000000000005},{"interface":"front","average":{"1min":1.5860000000000001,"5min":1.1000000000000001,"15min":1.3180000000000001},"min":{"1min":0.35499999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":6.3380000000000001,"5min":6.3380000000000001,"15min":52.145000000000003},"last":0.57299999999999995}]},{"osd":4,"last update":"Tue Sep 17 07:13:23 2024","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":0.88100000000000001,"15min":0.59799999999999998},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":8.5730000000000004,"5min":8.5730000000000004,"15min":8.5730000000000004},"last":0.35299999999999998},{"interface":"front","average":{"1min":1.5820000000000001,"5min":1.032,"15min":0.65900000000000003},"min":{"1min":0.307,"5min":0.26000000000000001,"15min":0.24399999999999999},"max":{"1min":4.8840000000000003,"5min":6.0549999999999997,"15min":6.0549999999999997},"last":0.53300000000000003}]},{"osd":5,"last update":"Tue Sep 17 07:13:23 2024","interfaces":[{"interface":"back","average":{"1min":3.1349999999999998,"5min":1.3280000000000001,"15min":0.75900000000000001},"min":{"1min":0.29099999999999998,"5min":0.253,"15min":0.23699999999999999},"max":{"1min":29.428999999999998,"5min":29.428999999999998,"15min":29.428999999999998},"last":0.54700000000000004},{"interface":"front","average":{"1min":3.266,"5min":1.3700000000000001,"15min":0.77800000000000002},"min":{"1min":0.26000000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":30.059999999999999,"5min":30.059999999999999,"15min":30.059999999999999},"last":4.4989999999999997}]},{"osd":6,"last update":"Tue Sep 17 07:13:23 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.2689999999999999,"15min":0.77300000000000002},"min":{"1min":0.37,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":8.5329999999999995,"5min":25.363,"15min":25.363},"last":0.38100000000000001},{"interface":"front","average":{"1min":1.504,"5min":1.258,"15min":0.78000000000000003},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.27500000000000002},"max":{"1min":5.8799999999999999,"5min":23.571000000000002,"15min":23.571000000000002},"last":0.82999999999999996}]}]},{"osd":6,"up_from":50,"seq":214748364951,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1762016,"kb_used_data":1415896,"kb_used_omap":1,"kb_used_meta":346110,"kb_avail":91983136,"statfs":{"total":95995035648,"available":94190731264,"internally_reserved":0,"allocated":1449877504,"data_stored":1449500201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":354417095},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:13:06 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":1.3129999999999999,"15min":1.591},"min":{"1min":0.248,"5min":0.185,"15min":0.185},"max":{"1min":1.653,"5min":24.106999999999999,"15min":75.323999999999998},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.63700000000000001,"5min":1.349,"15min":1.556},"min":{"1min":0.24399999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":2.9940000000000002,"5min":26.193000000000001,"15min":75.168999999999997},"last":0.26800000000000002}]},{"osd":1,"last update":"Tue Sep 17 07:13:06 2024","interfaces":[{"interface":"back","average":{"1min":1.8180000000000001,"5min":1.284,"15min":0.81499999999999995},"min":{"1min":0.32500000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":14.720000000000001,"5min":14.720000000000001,"15min":14.720000000000001},"last":0.33700000000000002},{"interface":"front","average":{"1min":1.748,"5min":1.2070000000000001,"15min":0.82799999999999996},"min":{"1min":0.32900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":14.699,"5min":14.699,"15min":14.699},"last":0.57099999999999995}]},{"osd":2,"last update":"Tue Sep 17 07:13:06 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.89700000000000002,"15min":0.81200000000000006},"min":{"1min":0.28599999999999998,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":1.73,"5min":5.1159999999999997,"15min":33.079000000000001},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.86299999999999999,"15min":0.80500000000000005},"min":{"1min":0.25600000000000001,"5min":0.254,"15min":0.23000000000000001},"max":{"1min":3.0710000000000002,"5min":5.734,"15min":33.100000000000001},"last":0.30299999999999999}]},{"osd":3,"last update":"Tue Sep 17 07:13:06 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":1.014,"15min":1.1279999999999999},"min":{"1min":0.26800000000000002,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":3.2000000000000002,"5min":5.0830000000000002,"15min":52.944000000000003},"last":0.374},{"interface":"front","average":{"1min":0.77400000000000002,"5min":1.0469999999999999,"15min":1.143},"min":{"1min":0.33300000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":2.101,"5min":5.3200000000000003,"15min":52.747},"last":0.76600000000000001}]},{"osd":4,"last update":"Tue Sep 17 07:13:06 2024","interfaces":[{"interface":"back","average":{"1min":0.877,"5min":4.5330000000000004,"15min":1.792},"min":{"1min":0.27100000000000002,"5min":0.214,"15min":0.214},"max":{"1min":3.8580000000000001,"5min":260.77199999999999,"15min":260.77199999999999},"last":2.0459999999999998},{"interface":"front","average":{"1min":0.76500000000000001,"5min":4.5220000000000002,"15min":1.8140000000000001},"min":{"1min":0.24199999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":3.222,"5min":260.69099999999997,"15min":260.69099999999997},"last":3.7759999999999998}]},{"osd":5,"last update":"Tue Sep 17 07:13:06 2024","interfaces":[{"interface":"back","average":{"1min":1.212,"5min":1.2949999999999999,"15min":0.73099999999999998},"min":{"1min":0.30499999999999999,"5min":0.26200000000000001,"15min":0.25900000000000001},"max":{"1min":5.1180000000000003,"5min":32.042000000000002,"15min":32.042000000000002},"last":1.911},{"interface":"front","average":{"1min":0.80200000000000005,"5min":1.3580000000000001,"15min":0.77300000000000002},"min":{"1min":0.29499999999999998,"5min":0.251,"15min":0.251},"max":{"1min":3.355,"5min":31.852,"15min":31.852},"last":0.39900000000000002}]},{"osd":7,"last update":"Tue Sep 17 07:13:27 2024","interfaces":[{"interface":"back","average":{"1min":1.444,"5min":1.159,"15min":0.74099999999999999},"min":{"1min":0.32400000000000001,"5min":0.28100000000000003,"15min":0.23699999999999999},"max":{"1min":4.8159999999999998,"5min":7.657,"15min":7.657},"last":0.71399999999999997},{"interface":"front","average":{"1min":1.3660000000000001,"5min":1.107,"15min":0.73799999999999999},"min":{"1min":0.38200000000000001,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":5.3879999999999999,"5min":7.1890000000000001,"15min":7.1890000000000001},"last":0.751}]}]},{"osd":5,"up_from":42,"seq":180388626588,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5193160,"kb_used_data":4666560,"kb_used_omap":1,"kb_used_meta":526590,"kb_avail":88551992,"statfs":{"total":95995035648,"available":90677239808,"internally_reserved":0,"allocated":4778557440,"data_stored":4778203908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":539228614},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:13:44 2024","interfaces":[{"interface":"back","average":{"1min":1.264,"5min":1.4350000000000001,"15min":1.5289999999999999},"min":{"1min":0.224,"5min":0.224,"15min":0.221},"max":{"1min":4.2329999999999997,"5min":13.913,"15min":49.265000000000001},"last":0.47799999999999998},{"interface":"front","average":{"1min":1.9610000000000001,"5min":1.633,"15min":1.6339999999999999},"min":{"1min":0.26000000000000001,"5min":0.219,"15min":0.219},"max":{"1min":6.7149999999999999,"5min":14.539999999999999,"15min":47.350000000000001},"last":0.378}]},{"osd":1,"last update":"Tue Sep 17 07:13:44 2024","interfaces":[{"interface":"back","average":{"1min":1.2889999999999999,"5min":1.111,"15min":0.82099999999999995},"min":{"1min":0.253,"5min":0.23799999999999999,"15min":0.23300000000000001},"max":{"1min":4.258,"5min":5.5529999999999999,"15min":20.329000000000001},"last":0.58999999999999997},{"interface":"front","average":{"1min":2.1709999999999998,"5min":1.542,"15min":1.1950000000000001},"min":{"1min":0.29799999999999999,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":6.7560000000000002,"5min":6.7560000000000002,"15min":20.408999999999999},"last":0.41099999999999998}]},{"osd":2,"last update":"Tue Sep 17 07:13:44 2024","interfaces":[{"interface":"back","average":{"1min":1.458,"5min":1.3839999999999999,"15min":1.0249999999999999},"min":{"1min":0.27400000000000002,"5min":0.25900000000000001,"15min":0.252},"max":{"1min":4.0570000000000004,"5min":6.242,"15min":20.486000000000001},"last":1.2909999999999999},{"interface":"front","average":{"1min":1.3160000000000001,"5min":1.1679999999999999,"15min":0.93799999999999994},"min":{"1min":0.28100000000000003,"5min":0.25800000000000001,"15min":0.23200000000000001},"max":{"1min":4.3079999999999998,"5min":7.1310000000000002,"15min":23.925999999999998},"last":0.50800000000000001}]},{"osd":3,"last update":"Tue Sep 17 07:13:44 2024","interfaces":[{"interface":"back","average":{"1min":1.4890000000000001,"5min":1.196,"15min":1.284},"min":{"1min":0.308,"5min":0.247,"15min":0.247},"max":{"1min":4.0960000000000001,"5min":6.5439999999999996,"15min":68.978999999999999},"last":1.327},{"interface":"front","average":{"1min":1.4019999999999999,"5min":1.1739999999999999,"15min":1.2529999999999999},"min":{"1min":0.31900000000000001,"5min":0.26800000000000002,"15min":0.253},"max":{"1min":4.327,"5min":6.4500000000000002,"15min":69.084000000000003},"last":0.55100000000000005}]},{"osd":4,"last update":"Tue Sep 17 07:13:44 2024","interfaces":[{"interface":"back","average":{"1min":2.6110000000000002,"5min":3.141,"15min":1.333},"min":{"1min":0.32400000000000001,"5min":0.23400000000000001,"15min":0.22},"max":{"1min":6.798,"5min":142.327,"15min":142.327},"last":0.45400000000000001},{"interface":"front","average":{"1min":2.012,"5min":2.8719999999999999,"15min":1.2529999999999999},"min":{"1min":0.34499999999999997,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":6.1920000000000002,"5min":142.13200000000001,"15min":142.13200000000001},"last":1.357}]},{"osd":6,"last update":"Tue Sep 17 07:13:58 2024","interfaces":[{"interface":"back","average":{"1min":1.827,"5min":1.26,"15min":0.72999999999999998},"min":{"1min":0.29399999999999998,"5min":0.26100000000000001,"15min":0.23200000000000001},"max":{"1min":6.8419999999999996,"5min":8.3149999999999995,"15min":8.3149999999999995},"last":0.53100000000000003},{"interface":"front","average":{"1min":1.5580000000000001,"5min":1.1970000000000001,"15min":0.73799999999999999},"min":{"1min":0.35699999999999998,"5min":0.27200000000000002,"15min":0.24099999999999999},"max":{"1min":4.1859999999999999,"5min":6.4459999999999997,"15min":6.4459999999999997},"last":2.016}]},{"osd":7,"last update":"Tue Sep 17 07:13:18 2024","interfaces":[{"interface":"back","average":{"1min":1.9330000000000001,"5min":1.349,"15min":0.79300000000000004},"min":{"1min":0.53000000000000003,"5min":0.29199999999999998,"15min":0.29099999999999998},"max":{"1min":5.8159999999999998,"5min":6.4130000000000003,"15min":6.4130000000000003},"last":1.391},{"interface":"front","average":{"1min":1.5469999999999999,"5min":1.1200000000000001,"15min":0.69999999999999996},"min":{"1min":0.32800000000000001,"5min":0.29899999999999999,"15min":0.27100000000000002},"max":{"1min":3.2570000000000001,"5min":5.641,"15min":5.641},"last":0.57099999999999995}]}]},{"osd":4,"up_from":33,"seq":141733920930,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2303624,"kb_used_data":1889408,"kb_used_omap":1,"kb_used_meta":414206,"kb_avail":91441528,"statfs":{"total":95995035648,"available":93636124672,"internally_reserved":0,"allocated":1934753792,"data_stored":1934399336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":424147398},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:13:06 2024","interfaces":[{"interface":"back","average":{"1min":1.7549999999999999,"5min":1.9870000000000001,"15min":1.9319999999999999},"min":{"1min":0.217,"5min":0.217,"15min":0.21299999999999999},"max":{"1min":18.434999999999999,"5min":55.310000000000002,"15min":74.551000000000002},"last":0.38600000000000001},{"interface":"front","average":{"1min":1.538,"5min":2.1320000000000001,"15min":2.073},"min":{"1min":0.28399999999999997,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":18.521000000000001,"5min":53.771000000000001,"15min":74.778999999999996},"last":0.48199999999999998}]},{"osd":1,"last update":"Tue Sep 17 07:13:06 2024","interfaces":[{"interface":"back","average":{"1min":1.857,"5min":2.964,"15min":1.3999999999999999},"min":{"1min":0.24399999999999999,"5min":0.23300000000000001,"15min":0.23000000000000001},"max":{"1min":18.661000000000001,"5min":54.914999999999999,"15min":54.914999999999999},"last":1.232},{"interface":"front","average":{"1min":2.0590000000000002,"5min":2.6709999999999998,"15min":1.3140000000000001},"min":{"1min":0.33000000000000002,"5min":0.23300000000000001,"15min":0.22800000000000001},"max":{"1min":21.931999999999999,"5min":53.725000000000001,"15min":53.725000000000001},"last":0.77200000000000002}]},{"osd":2,"last update":"Tue Sep 17 07:13:06 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":1.002,"15min":0.94299999999999995},"min":{"1min":0.25700000000000001,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.2869999999999999,"5min":6.8929999999999998,"15min":49.651000000000003},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.67600000000000005,"5min":1.111,"15min":1.022},"min":{"1min":0.28699999999999998,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":1.6639999999999999,"5min":16.003,"15min":48.813000000000002},"last":0.35299999999999998}]},{"osd":3,"last update":"Tue Sep 17 07:13:06 2024","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":1.1950000000000001,"15min":0.998},"min":{"1min":0.41399999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":2.0950000000000002,"5min":17.872,"15min":39.042999999999999},"last":0.375},{"interface":"front","average":{"1min":0.623,"5min":1.1799999999999999,"15min":1.016},"min":{"1min":0.309,"5min":0.253,"15min":0.253},"max":{"1min":1.4750000000000001,"5min":17.908000000000001,"15min":38.920000000000002},"last":0.42699999999999999}]},{"osd":5,"last update":"Tue Sep 17 07:13:33 2024","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":1.2470000000000001,"15min":0.71499999999999997},"min":{"1min":0.28399999999999997,"5min":0.222,"15min":0.20799999999999999},"max":{"1min":4.5289999999999999,"5min":11.486000000000001,"15min":11.486000000000001},"last":0.70999999999999996},{"interface":"front","average":{"1min":1.464,"5min":1.4139999999999999,"15min":0.78400000000000003},"min":{"1min":0.29899999999999999,"5min":0.27500000000000002,"15min":0.26100000000000001},"max":{"1min":4.3780000000000001,"5min":11.781000000000001,"15min":11.781000000000001},"last":0.84499999999999997}]},{"osd":6,"last update":"Tue Sep 17 07:13:01 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.83199999999999996,"15min":0.75600000000000001},"min":{"1min":0.26800000000000002,"5min":0.218,"15min":0.218},"max":{"1min":1.4390000000000001,"5min":4.3620000000000001,"15min":47.923999999999999},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.92100000000000004,"15min":0.79200000000000004},"min":{"1min":0.33100000000000002,"5min":0.26700000000000002,"15min":0.26300000000000001},"max":{"1min":1.403,"5min":6.2140000000000004,"15min":47.600999999999999},"last":0.443}]},{"osd":7,"last update":"Tue Sep 17 07:13:25 2024","interfaces":[{"interface":"back","average":{"1min":1.21,"5min":1.0569999999999999,"15min":0.87},"min":{"1min":0.34100000000000003,"5min":0.29299999999999998,"15min":0.27600000000000002},"max":{"1min":3.403,"5min":5.8040000000000003,"15min":48.719000000000001},"last":0.32300000000000001},{"interface":"front","average":{"1min":1.373,"5min":1.0349999999999999,"15min":0.85499999999999998},"min":{"1min":0.317,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":4.3159999999999998,"5min":4.4279999999999999,"15min":50.527000000000001},"last":0.54500000000000004}]}]},{"osd":3,"up_from":0,"seq":115964117156,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":18,"seq":77309411498,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2940552,"kb_used_data":2345856,"kb_used_omap":1,"kb_used_meta":594686,"kb_avail":90804600,"statfs":{"total":95995035648,"available":92983910400,"internally_reserved":0,"allocated":2402156544,"data_stored":2401802932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":608958918},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:13:42 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":1.9159999999999999,"15min":1.0880000000000001},"min":{"1min":0.19,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":4.0039999999999996,"5min":79.715000000000003,"15min":79.715000000000003},"last":0.20200000000000001},{"interface":"front","average":{"1min":0.53700000000000003,"5min":1.9279999999999999,"15min":1.2330000000000001},"min":{"1min":0.221,"5min":0.217,"15min":0.19900000000000001},"max":{"1min":1.5049999999999999,"5min":79.691999999999993,"15min":79.691999999999993},"last":0.41099999999999998}]},{"osd":1,"last update":"Tue Sep 17 07:13:42 2024","interfaces":[{"interface":"back","average":{"1min":1.7789999999999999,"5min":2.8730000000000002,"15min":1.4039999999999999},"min":{"1min":0.22800000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":15.717000000000001,"5min":79.659000000000006,"15min":79.659000000000006},"last":0.27000000000000002},{"interface":"front","average":{"1min":1.508,"5min":2.794,"15min":1.357},"min":{"1min":0.20899999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":15.640000000000001,"5min":80.155000000000001,"15min":80.155000000000001},"last":0.255}]},{"osd":3,"last update":"Tue Sep 17 07:13:52 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":1.3859999999999999,"15min":0.95899999999999996},"min":{"1min":0.245,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":4.0469999999999997,"5min":23.43,"15min":23.43},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.48899999999999999,"5min":1.2050000000000001,"15min":1.0369999999999999},"min":{"1min":0.23499999999999999,"5min":0.223,"15min":0.223},"max":{"1min":1.2709999999999999,"5min":23.408000000000001,"15min":23.408000000000001},"last":0.35999999999999999}]},{"osd":4,"last update":"Tue Sep 17 07:13:17 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":1.038,"15min":0.82399999999999995},"min":{"1min":0.28899999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":2.202,"5min":5.944,"15min":5.944},"last":0.37},{"interface":"front","average":{"1min":0.69499999999999995,"5min":1.119,"15min":0.82599999999999996},"min":{"1min":0.28999999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":2.2170000000000001,"5min":6.8529999999999998,"15min":6.8529999999999998},"last":0.46500000000000002}]},{"osd":5,"last update":"Tue Sep 17 07:13:42 2024","interfaces":[{"interface":"back","average":{"1min":2.0859999999999999,"5min":1.492,"15min":1.0349999999999999},"min":{"1min":0.34899999999999998,"5min":0.26900000000000002,"15min":0.249},"max":{"1min":15.276999999999999,"5min":15.276999999999999,"15min":15.276999999999999},"last":1.8680000000000001},{"interface":"front","average":{"1min":2.0169999999999999,"5min":1.3720000000000001,"15min":0.93799999999999994},"min":{"1min":0.35399999999999998,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":15.561999999999999,"5min":15.561999999999999,"15min":15.561999999999999},"last":0.307}]},{"osd":6,"last update":"Tue Sep 17 07:13:52 2024","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":2.3570000000000002,"15min":1.298},"min":{"1min":0.32100000000000001,"5min":0.29999999999999999,"15min":0.28999999999999998},"max":{"1min":4.1959999999999997,"5min":101.587,"15min":101.587},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.67700000000000005,"5min":2.2400000000000002,"15min":1.319},"min":{"1min":0.376,"5min":0.30599999999999999,"15min":0.28299999999999997},"max":{"1min":2.081,"5min":100.696,"15min":100.696},"last":0.51500000000000001}]},{"osd":7,"last update":"Tue Sep 17 07:13:33 2024","interfaces":[{"interface":"back","average":{"1min":1.236,"5min":1.1439999999999999,"15min":0.872},"min":{"1min":0.41499999999999998,"5min":0.316,"15min":0.316},"max":{"1min":4.1500000000000004,"5min":6.9450000000000003,"15min":6.9450000000000003},"last":4.3220000000000001},{"interface":"front","average":{"1min":1.421,"5min":1.2130000000000001,"15min":0.98999999999999999},"min":{"1min":0.40200000000000002,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":4.8470000000000004,"5min":6.9219999999999997,"15min":9.2919999999999998},"last":1.095}]}]},{"osd":1,"up_from":13,"seq":55834575023,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4654320,"kb_used_data":4194152,"kb_used_omap":1,"kb_used_meta":460158,"kb_avail":89090832,"statfs":{"total":95995035648,"available":91229011968,"internally_reserved":0,"allocated":4294811648,"data_stored":4294418254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":471202246},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:13:59 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":1.571,"15min":1.109},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.155},"max":{"1min":3.4870000000000001,"5min":22.521000000000001,"15min":22.521000000000001},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.41799999999999998,"5min":1.1830000000000001,"15min":0.997},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.65100000000000002,"5min":22.471,"15min":22.471},"last":0.40799999999999997}]},{"osd":2,"last update":"Tue Sep 17 07:13:35 2024","interfaces":[{"interface":"back","average":{"1min":1.502,"5min":1.3680000000000001,"15min":1.0089999999999999},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":6.1920000000000002,"5min":6.2130000000000001,"15min":11.055999999999999},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.75600000000000001,"5min":1.0249999999999999,"15min":0.91600000000000004},"min":{"1min":0.28599999999999998,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":2.7370000000000001,"5min":5.242,"15min":9.3160000000000007},"last":0.379}]},{"osd":3,"last update":"Tue Sep 17 07:13:59 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":1.01,"15min":0.92600000000000005},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.219},"max":{"1min":0.59499999999999997,"5min":8.5510000000000002,"15min":8.5510000000000002},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.42099999999999999,"5min":1.0600000000000001,"15min":0.99099999999999999},"min":{"1min":0.22500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.72399999999999998,"5min":8.6400000000000006,"15min":19.602},"last":0.35999999999999999}]},{"osd":4,"last update":"Tue Sep 17 07:13:16 2024","interfaces":[{"interface":"back","average":{"1min":1.2250000000000001,"5min":1.401,"15min":1},"min":{"1min":0.44900000000000001,"5min":0.25,"15min":0.25},"max":{"1min":3.9540000000000002,"5min":9.8480000000000008,"15min":9.8480000000000008},"last":2.0089999999999999},{"interface":"front","average":{"1min":0.93000000000000005,"5min":1.5940000000000001,"15min":1.0589999999999999},"min":{"1min":0.32700000000000001,"5min":0.317,"15min":0.307},"max":{"1min":2.2690000000000001,"5min":27.097999999999999,"15min":27.097999999999999},"last":0.44700000000000001}]},{"osd":5,"last update":"Tue Sep 17 07:13:35 2024","interfaces":[{"interface":"back","average":{"1min":1.6339999999999999,"5min":1.6539999999999999,"15min":1.105},"min":{"1min":0.36299999999999999,"5min":0.33400000000000002,"15min":0.27100000000000002},"max":{"1min":4.4400000000000004,"5min":16.84,"15min":16.84},"last":2.5369999999999999},{"interface":"front","average":{"1min":1.0629999999999999,"5min":1.627,"15min":1.095},"min":{"1min":0.29899999999999999,"5min":0.29199999999999998,"15min":0.28899999999999998},"max":{"1min":2.2719999999999998,"5min":16.949999999999999,"15min":16.949999999999999},"last":0.63800000000000001}]},{"osd":6,"last update":"Tue Sep 17 07:13:59 2024","interfaces":[{"interface":"back","average":{"1min":1.373,"5min":1.921,"15min":1.248},"min":{"1min":0.32300000000000001,"5min":0.308,"15min":0.308},"max":{"1min":3.9180000000000001,"5min":50.497,"15min":50.497},"last":1.038},{"interface":"front","average":{"1min":0.97999999999999998,"5min":1.7629999999999999,"15min":1.159},"min":{"1min":0.39900000000000002,"5min":0.35099999999999998,"15min":0.28899999999999998},"max":{"1min":3.5569999999999999,"5min":50.793999999999997,"15min":50.793999999999997},"last":2.0830000000000002}]},{"osd":7,"last update":"Tue Sep 17 07:13:17 2024","interfaces":[{"interface":"back","average":{"1min":1.1779999999999999,"5min":1.6379999999999999,"15min":1.0940000000000001},"min":{"1min":0.47999999999999998,"5min":0.314,"15min":0.314},"max":{"1min":2.827,"5min":9.8179999999999996,"15min":9.8179999999999996},"last":0.61199999999999999},{"interface":"front","average":{"1min":2.1429999999999998,"5min":1.7769999999999999,"15min":1.131},"min":{"1min":0.40899999999999997,"5min":0.34100000000000003,"15min":0.32900000000000001},"max":{"1min":7.5839999999999996,"5min":7.5839999999999996,"15min":11.007999999999999},"last":1.71}]}]},{"osd":0,"up_from":8,"seq":34359738546,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3369048,"kb_used_data":2792400,"kb_used_omap":1,"kb_used_meta":576638,"kb_avail":90376104,"statfs":{"total":95995035648,"available":92545130496,"internally_reserved":0,"allocated":2859417600,"data_stored":2859048348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":590477766},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 17 07:13:10 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":1.151,"15min":0.86799999999999999},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":1.71,"5min":21.876000000000001,"15min":21.876000000000001},"last":0.439},{"interface":"front","average":{"1min":1.1279999999999999,"5min":1.575,"15min":1.0429999999999999},"min":{"1min":0.27100000000000002,"5min":0.158,"15min":0.158},"max":{"1min":5.9960000000000004,"5min":22.521999999999998,"15min":22.521999999999998},"last":0.58199999999999996}]},{"osd":2,"last update":"Tue Sep 17 07:13:38 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":1.0369999999999999,"15min":1.0980000000000001},"min":{"1min":0.20999999999999999,"5min":0.188,"15min":0.188},"max":{"1min":2.2810000000000001,"5min":6.3799999999999999,"15min":18.643999999999998},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.89200000000000002,"15min":0.98299999999999998},"min":{"1min":0.23799999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":1.9610000000000001,"5min":5.8440000000000003,"15min":17.972999999999999},"last":0.33300000000000002}]},{"osd":3,"last update":"Tue Sep 17 07:13:10 2024","interfaces":[{"interface":"back","average":{"1min":0.97599999999999998,"5min":0.96999999999999997,"15min":1.0069999999999999},"min":{"1min":0.255,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":5.5880000000000001,"5min":5.5880000000000001,"15min":18.629000000000001},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.80600000000000005,"15min":1.006},"min":{"1min":0.25700000000000001,"5min":0.24199999999999999,"15min":0.23699999999999999},"max":{"1min":1.2230000000000001,"5min":3.9100000000000001,"15min":22.358000000000001},"last":1.3819999999999999}]},{"osd":4,"last update":"Tue Sep 17 07:13:14 2024","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":1.071,"15min":1.044},"min":{"1min":0.29799999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":2.7170000000000001,"5min":19.146999999999998,"15min":19.146999999999998},"last":0.52800000000000002},{"interface":"front","average":{"1min":1.056,"5min":1.1739999999999999,"15min":1.0089999999999999},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.255},"max":{"1min":2.7559999999999998,"5min":19.158999999999999,"15min":19.158999999999999},"last":0.47099999999999997}]},{"osd":5,"last update":"Tue Sep 17 07:13:48 2024","interfaces":[{"interface":"back","average":{"1min":1.204,"5min":1.3160000000000001,"15min":1.101},"min":{"1min":0.35499999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":3.609,"5min":18.841000000000001,"15min":18.841000000000001},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.98299999999999998,"5min":1.339,"15min":1.103},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":3.415,"5min":19.248000000000001,"15min":19.248000000000001},"last":2.335}]},{"osd":6,"last update":"Tue Sep 17 07:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.014,"5min":1.075,"15min":0.97799999999999998},"min":{"1min":0.308,"5min":0.308,"15min":0.30399999999999999},"max":{"1min":2.1469999999999998,"5min":4.3630000000000004,"15min":18.602},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.98099999999999998,"15min":1.0089999999999999},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.33900000000000002},"max":{"1min":1.9410000000000001,"5min":3.7650000000000001,"15min":18.692},"last":0.60099999999999998}]},{"osd":7,"last update":"Tue Sep 17 07:13:18 2024","interfaces":[{"interface":"back","average":{"1min":1.7789999999999999,"5min":1.218,"15min":1.427},"min":{"1min":0.30599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":5.9509999999999996,"5min":5.9509999999999996,"15min":46.183},"last":0.49399999999999999},{"interface":"front","average":{"1min":1.532,"5min":1.238,"15min":1.1890000000000001},"min":{"1min":0.36499999999999999,"5min":0.33600000000000002,"15min":0.32000000000000001},"max":{"1min":6.1079999999999997,"5min":6.1079999999999997,"15min":45.914999999999999},"last":0.41499999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2836135936,"data_stored":2836135936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4281864192,"data_stored":4281860120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2393767936,"data_stored":2393767936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1918763008,"data_stored":1918763008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4755099648,"data_stored":4755095576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1431044096,"data_stored":1431044096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":5252190208,"data_stored":5252186136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T07:14:06.796 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:14:06.817 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:14:06.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:06.835 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:06.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:14:06.936 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:14:06.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:06.967 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:06.967 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:14:07.068 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:14:07.136 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:06 smithi100 ceph-mon[29092]: 4.13 scrub starts 2024-09-17T07:14:07.136 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:06 smithi100 ceph-mon[29092]: from='client.25316 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:14:07.136 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:06 smithi100 ceph-mon[29092]: 4.13 scrub ok 2024-09-17T07:14:07.136 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:06 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1964452076' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:14:07.136 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:06 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:14:07.136 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:06 smithi100 ceph-mon[39462]: 4.13 scrub starts 2024-09-17T07:14:07.136 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:06 smithi100 ceph-mon[39462]: from='client.25316 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:14:07.137 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:06 smithi100 ceph-mon[39462]: 4.13 scrub ok 2024-09-17T07:14:07.137 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:06 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1964452076' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:14:07.137 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:06 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:14:07.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:07.174 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:07.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:14:07.216 INFO:teuthology.orchestra.run.smithi100.stderr:noscrub is set 2024-09-17T07:14:07.275 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:14:07.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:06 smithi135 ceph-mon[33238]: 4.13 scrub starts 2024-09-17T07:14:07.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:06 smithi135 ceph-mon[33238]: from='client.25316 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:14:07.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:06 smithi135 ceph-mon[33238]: 4.13 scrub ok 2024-09-17T07:14:07.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:06 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1964452076' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:14:07.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:06 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:14:07.309 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:07.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:07.310 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:14:07.411 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:14:07.450 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:07.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:07.452 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:14:07.553 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:14:07.585 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:07.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:07.586 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:14:07.687 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:14:07.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:07.726 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:07.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:14:07.827 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:14:07.857 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:07.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:07.858 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:14:07.959 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:14:07.990 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:07.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:07.991 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:14:08.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:07 smithi135 ceph-mon[33238]: 4.b deep-scrub starts 2024-09-17T07:14:08.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:07 smithi135 ceph-mon[33238]: 4.b deep-scrub ok 2024-09-17T07:14:08.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:07 smithi135 ceph-mon[33238]: 4.15 scrub starts 2024-09-17T07:14:08.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:07 smithi135 ceph-mon[33238]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:14:08.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:07 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T07:14:08.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:07 smithi135 ceph-mon[33238]: osdmap e269: 8 total, 8 up, 7 in 2024-09-17T07:14:08.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:07 smithi135 ceph-mon[33238]: pgmap v534: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.1 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.20k op/s 2024-09-17T07:14:08.091 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:14:08.120 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:08.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:08.121 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:14:08.222 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:14:08.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:08.254 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:08.254 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:14:08.355 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:14:08.376 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:07 smithi100 ceph-mon[39462]: 4.b deep-scrub starts 2024-09-17T07:14:08.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:07 smithi100 ceph-mon[39462]: 4.b deep-scrub ok 2024-09-17T07:14:08.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:07 smithi100 ceph-mon[39462]: 4.15 scrub starts 2024-09-17T07:14:08.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:07 smithi100 ceph-mon[39462]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:14:08.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:07 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T07:14:08.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:07 smithi100 ceph-mon[39462]: osdmap e269: 8 total, 8 up, 7 in 2024-09-17T07:14:08.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:07 smithi100 ceph-mon[39462]: pgmap v534: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.1 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.20k op/s 2024-09-17T07:14:08.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:07 smithi100 ceph-mon[29092]: 4.b deep-scrub starts 2024-09-17T07:14:08.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:07 smithi100 ceph-mon[29092]: 4.b deep-scrub ok 2024-09-17T07:14:08.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:07 smithi100 ceph-mon[29092]: 4.15 scrub starts 2024-09-17T07:14:08.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:07 smithi100 ceph-mon[29092]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:14:08.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:07 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T07:14:08.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:07 smithi100 ceph-mon[29092]: osdmap e269: 8 total, 8 up, 7 in 2024-09-17T07:14:08.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:07 smithi100 ceph-mon[29092]: pgmap v534: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.1 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.20k op/s 2024-09-17T07:14:08.390 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:08.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:08.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:14:08.492 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:14:08.525 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:08.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:08.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:14:08.627 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:14:08.655 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:08.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:08.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:14:08.757 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:14:08.785 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:08.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:08.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:14:08.793 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:14:08 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:14:08.887 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:14:08.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:08.933 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:08.933 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:14:09.033 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:14:09.071 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:09.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:09.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:14:09.110 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:08 smithi135 ceph-mon[33238]: 4.15 scrub ok 2024-09-17T07:14:09.173 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:14:09.201 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:08 smithi100 ceph-mon[29092]: 4.15 scrub ok 2024-09-17T07:14:09.201 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:08 smithi100 ceph-mon[39462]: 4.15 scrub ok 2024-09-17T07:14:09.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:09.208 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:09.208 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:14:09.309 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:14:09.337 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:09.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:09.338 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:14:09.439 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:14:09.471 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:09.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:09.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:14:09.502 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:14:09.503 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:14:09.529 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1: {} 2024-09-17T07:14:09.530 INFO:teuthology.orchestra.run.smithi100.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:14:09.555 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:14:09.555 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:14:09.574 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:14:09.583 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:14:09.583 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:14:09.605 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:09.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:09.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:14:09.621 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:14:09.622 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:14:09.645 INFO:teuthology.orchestra.run.smithi135.stdout:ERROR: (22) Invalid argument 2024-09-17T07:14:09.645 INFO:teuthology.orchestra.run.smithi135.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:14:09.648 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:14:09.649 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:14:09.677 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:14:09.678 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:14:09.707 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:14:09.713 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:14:09.713 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:14:09.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:09.746 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:09.746 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:14:09.847 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:14:09.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:09.882 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:09.883 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:14:09.983 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:14:10.019 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:10.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:10.020 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:14:10.117 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:09 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:14:10.117 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:09 smithi100 ceph-mon[29092]: pgmap v535: 148 pgs: 148 active+clean; 6.8 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.22k op/s 2024-09-17T07:14:10.117 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:09 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1591873582' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:14:10.118 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:09 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:14:10.118 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:09 smithi100 ceph-mon[39462]: pgmap v535: 148 pgs: 148 active+clean; 6.8 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.22k op/s 2024-09-17T07:14:10.118 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:09 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1591873582' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:14:10.121 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:14:10.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:10.159 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:10.159 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:14:10.202 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:09 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:14:10.202 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:09 smithi135 ceph-mon[33238]: pgmap v535: 148 pgs: 148 active+clean; 6.8 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.22k op/s 2024-09-17T07:14:10.203 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:09 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1591873582' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:14:10.259 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:14:10.299 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:10.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:10.300 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:14:10.402 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:14:10.432 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:10.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:10.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:14:10.535 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:14:10.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:10.557 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:10.557 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:14:10.660 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:14:10.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:10.696 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:10.697 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:14:10.797 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:14:10.829 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:10.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:10.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:14:10.931 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:14:10.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:10.969 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:10.969 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:14:10.987 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-17T07:14:11.070 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:14:11.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:11.104 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:11.105 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:14:11.205 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:14:11.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:11.241 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:11.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:14:11.289 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:14:11.342 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:14:11.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:11.374 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:11.374 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:14:11.477 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:14:11.498 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-17T07:14:11.500 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:11.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:11.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:14:11.602 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:14:11.638 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:11.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:11.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:14:11.740 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:14:11.766 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:11.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:11.767 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:14:11.786 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7] out_osds: [3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-17T07:14:11.787 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd reweight 0 1 2024-09-17T07:14:11.868 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:14:11.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:11.910 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:11.911 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:14:12.011 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:14:12.044 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:12.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:12.045 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:14:12.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:11 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:14:12.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:11 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:14:12.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:11 smithi100 ceph-mon[29092]: osdmap e270: 8 total, 8 up, 7 in 2024-09-17T07:14:12.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:11 smithi100 ceph-mon[29092]: pgmap v537: 148 pgs: 148 active+clean; 6.7 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.53k op/s 2024-09-17T07:14:12.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:11 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:14:12.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:11 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:14:12.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:11 smithi100 ceph-mon[39462]: osdmap e270: 8 total, 8 up, 7 in 2024-09-17T07:14:12.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:11 smithi100 ceph-mon[39462]: pgmap v537: 148 pgs: 148 active+clean; 6.7 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.53k op/s 2024-09-17T07:14:12.146 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:14:12.188 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:12.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:12.189 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:14:12.290 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:14:12.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:11 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:14:12.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:11 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:14:12.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:11 smithi135 ceph-mon[33238]: osdmap e270: 8 total, 8 up, 7 in 2024-09-17T07:14:12.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:11 smithi135 ceph-mon[33238]: pgmap v537: 148 pgs: 148 active+clean; 6.7 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.53k op/s 2024-09-17T07:14:12.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:12.319 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:12.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:14:12.420 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:14:12.454 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:12.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:12.455 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:14:12.471 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:14:12.556 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:14:12.584 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:12.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:12.585 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:14:12.686 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:14:12.724 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:12.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:12.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:14:12.826 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:14:12.869 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:12.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:12.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:14:12.971 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:14:13.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:13.009 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:13.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:14:13.113 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:14:13.142 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:13.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:13.143 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:14:13.244 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:14:13.280 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:13.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:13.281 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:14:13.382 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:14:13.413 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:13.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:13.414 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:14:13.515 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:14:13.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:13.546 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:13.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:14:13.647 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:14:13.673 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:13.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:13.674 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:14:13.775 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:14:13.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:13.810 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:13.810 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:14:13.877 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:14:13.891 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:14:13.904 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:14:13.911 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:14:13.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:13.941 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:13.942 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:14:14.042 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:14:14.081 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:14.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:14.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:14:14.183 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:14:14.219 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:14.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:14.220 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:14:14.321 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:14:14.359 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:14.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:14.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:14:14.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:14 smithi100 ceph-mon[29092]: pgmap v538: 148 pgs: 148 active+clean; 6.7 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 768 op/s 2024-09-17T07:14:14.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:14 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:14:14.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:14 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:14:14.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:14 smithi100 ceph-mon[39462]: pgmap v538: 148 pgs: 148 active+clean; 6.7 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 768 op/s 2024-09-17T07:14:14.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:14 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:14:14.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:14 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:14:14.461 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:14:14.493 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:14.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:14.494 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:14:14.596 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:14:14.626 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:14.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:14.627 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:14:14.728 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:14:14.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:14.762 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:14.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:14:14.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:14 smithi135 ceph-mon[33238]: pgmap v538: 148 pgs: 148 active+clean; 6.7 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 768 op/s 2024-09-17T07:14:14.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:14 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:14:14.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:14 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:14:14.863 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:14:14.893 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:14.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:14.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:14:14.995 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:14:15.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:15.038 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:15.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:14:15.140 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:14:15.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:15.176 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:15.176 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:14:15.277 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:14:15.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:15.309 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:15.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:14:15.410 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:14:15.443 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:15.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:15.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:14:15.545 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:14:15.574 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:15.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:15.575 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:14:15.612 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:15 smithi100 ceph-mon[29092]: osdmap e271: 8 total, 8 up, 7 in 2024-09-17T07:14:15.612 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:15 smithi100 ceph-mon[39462]: osdmap e271: 8 total, 8 up, 7 in 2024-09-17T07:14:15.676 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:14:15.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:15.706 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:15.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:14:15.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:15 smithi135 ceph-mon[33238]: osdmap e271: 8 total, 8 up, 7 in 2024-09-17T07:14:15.807 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:14:15.838 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:15.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:15.839 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:14:15.940 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:14:15.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:15.980 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:15.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:14:16.081 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:14:16.117 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:16.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:16.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:14:16.126 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:14:15 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:14:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:14:16.219 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:14:16.247 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:16.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:16.248 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:14:16.343 INFO:teuthology.orchestra.run.smithi100.stderr:nodeep-scrub is set 2024-09-17T07:14:16.349 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:14:16.385 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:16.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:16.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:14:16.487 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:14:16.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:16.527 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:16.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:14:16.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:16 smithi100 ceph-mon[29092]: pgmap v540: 148 pgs: 148 active+clean; 6.3 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.57k op/s 2024-09-17T07:14:16.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:16 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/3804956421' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:14:16.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:16 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:14:16.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:16 smithi100 ceph-mon[39462]: pgmap v540: 148 pgs: 148 active+clean; 6.3 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.57k op/s 2024-09-17T07:14:16.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:16 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/3804956421' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:14:16.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:16 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:14:16.629 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:14:16.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:16.667 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:16.667 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:14:16.768 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:14:16.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:16 smithi135 ceph-mon[33238]: pgmap v540: 148 pgs: 148 active+clean; 6.3 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.57k op/s 2024-09-17T07:14:16.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:16 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/3804956421' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:14:16.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:16 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:14:16.797 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:16.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:16.799 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:14:16.899 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:14:16.926 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:16.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:16.928 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:14:17.028 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:14:17.068 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:17.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:17.069 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:14:17.170 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:14:17.211 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:17.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:17.212 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:14:17.313 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:14:17.350 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:17.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:17.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:14:17.452 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:14:17.485 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:17.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:17.486 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:14:17.587 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:14:17.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:17.623 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:17.623 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:14:17.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:17 smithi100 ceph-mon[29092]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:14:17.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:17 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T07:14:17.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:17 smithi100 ceph-mon[29092]: osdmap e272: 8 total, 8 up, 7 in 2024-09-17T07:14:17.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:17 smithi100 ceph-mon[39462]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:14:17.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:17 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T07:14:17.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:17 smithi100 ceph-mon[39462]: osdmap e272: 8 total, 8 up, 7 in 2024-09-17T07:14:17.725 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:14:17.755 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:17.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:17.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:14:17.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:17 smithi135 ceph-mon[33238]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:14:17.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:17 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T07:14:17.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:17 smithi135 ceph-mon[33238]: osdmap e272: 8 total, 8 up, 7 in 2024-09-17T07:14:17.857 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:14:17.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:17.899 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:17.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:14:18.000 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:14:18.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:18.030 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:18.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:14:18.131 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:14:18.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:18.169 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:18.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:14:18.270 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:14:18.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:18.308 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:18.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:14:18.409 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:14:18.427 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:18 smithi100 ceph-mon[29092]: pgmap v542: 148 pgs: 148 active+clean; 6.3 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 811 B/s rd, 0 B/s wr, 1.02k op/s 2024-09-17T07:14:18.427 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:18 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:14:18.427 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:18 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:14:18.427 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:18 smithi100 ceph-mon[39462]: pgmap v542: 148 pgs: 148 active+clean; 6.3 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 811 B/s rd, 0 B/s wr, 1.02k op/s 2024-09-17T07:14:18.427 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:18 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:14:18.427 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:18 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:14:18.443 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:18.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:18.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:14:18.469 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-17T07:14:18.469 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-17T07:14:18.470 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:18.470 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.15330.0:171410 9.5 9:a0ec1303:::benchmark_data_smithi100_128631_object37144:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e272)", 2024-09-17T07:14:18.470 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:14:18.461859+0000", 2024-09-17T07:14:18.470 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.0058565930000000002, 2024-09-17T07:14:18.470 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0058866989999999996, 2024-09-17T07:14:18.470 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:14:18.470 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "waiting for sub ops", 2024-09-17T07:14:18.470 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:14:18.470 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.15330", 2024-09-17T07:14:18.471 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/1361381749", 2024-09-17T07:14:18.471 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 171410 2024-09-17T07:14:18.471 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:18.471 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:14:18.471 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:18.472 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:14:18.472 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:14:18.461859+0000", 2024-09-17T07:14:18.472 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:14:18.472 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:18.472 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:18.472 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:14:18.472 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:14:18.461859+0000", 2024-09-17T07:14:18.472 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:14:18.472 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:18.472 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:18.473 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:14:18.473 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:14:18.461857+0000", 2024-09-17T07:14:18.473 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999986 2024-09-17T07:14:18.473 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:18.473 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:18.473 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:14:18.473 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:14:18.461863+0000", 2024-09-17T07:14:18.473 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.84e-06 2024-09-17T07:14:18.473 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:18.473 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:18.473 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:14:18.473 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:14:18.461873+0000", 2024-09-17T07:14:18.473 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3379999999999994e-06 2024-09-17T07:14:18.474 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:18.474 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:18.474 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:14:18.474 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:14:18.461879+0000", 2024-09-17T07:14:18.474 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8140000000000003e-06 2024-09-17T07:14:18.474 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:18.474 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:18.474 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:14:18.474 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:14:18.461928+0000", 2024-09-17T07:14:18.474 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8118999999999998e-05 2024-09-17T07:14:18.474 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:18.474 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:18.475 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:14:18.475 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:14:18.462033+0000", 2024-09-17T07:14:18.475 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000105845 2024-09-17T07:14:18.475 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:18.475 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:18.475 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 5,7", 2024-09-17T07:14:18.475 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:14:18.462158+0000", 2024-09-17T07:14:18.475 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012493000000000001 2024-09-17T07:14:18.475 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:18.475 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:18.475 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:14:18.475 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:14:18.466642+0000", 2024-09-17T07:14:18.475 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0044835209999999999 2024-09-17T07:14:18.476 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:18.476 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:18.476 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:14:18.476 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:14:18.467072+0000", 2024-09-17T07:14:18.476 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00042988100000000001 2024-09-17T07:14:18.476 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:18.476 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:18.476 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:14:18.476 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:14:18.467755+0000", 2024-09-17T07:14:18.476 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00068314600000000004 2024-09-17T07:14:18.476 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:14:18.476 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:14:18.477 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:14:18.477 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:18.477 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:18.477 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.15330.0:171414 9.8 9:1ff54dcb:::benchmark_data_smithi100_128631_object37148:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e272)", 2024-09-17T07:14:18.477 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:14:18.462884+0000", 2024-09-17T07:14:18.477 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.004831938, 2024-09-17T07:14:18.477 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0049595230000000004, 2024-09-17T07:14:18.477 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:14:18.477 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:14:18.477 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:14:18.477 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.15330", 2024-09-17T07:14:18.477 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/1361381749", 2024-09-17T07:14:18.478 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 171414 2024-09-17T07:14:18.478 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:18.478 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:14:18.478 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:18.478 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:14:18.478 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:14:18.462884+0000", 2024-09-17T07:14:18.478 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:14:18.478 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:18.478 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:18.478 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:14:18.478 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:14:18.462884+0000", 2024-09-17T07:14:18.478 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:14:18.479 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:18.479 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:18.479 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:14:18.479 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:14:18.462883+0000", 2024-09-17T07:14:18.479 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-17T07:14:18.479 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:18.479 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:18.479 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:14:18.479 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:14:18.462887+0000", 2024-09-17T07:14:18.479 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8280000000000002e-06 2024-09-17T07:14:18.479 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:18.480 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:18.480 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:14:18.480 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:14:18.462898+0000", 2024-09-17T07:14:18.480 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0311e-05 2024-09-17T07:14:18.480 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:18.480 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:18.480 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:14:18.480 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:14:18.462904+0000", 2024-09-17T07:14:18.480 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6359999999999997e-06 2024-09-17T07:14:18.480 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:18.480 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:18.480 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:14:18.481 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:14:18.462943+0000", 2024-09-17T07:14:18.481 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8155000000000003e-05 2024-09-17T07:14:18.481 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:18.481 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:18.481 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:14:18.481 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:14:18.463047+0000", 2024-09-17T07:14:18.481 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00010405500000000001 2024-09-17T07:14:18.481 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:18.481 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:18.481 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 5,7", 2024-09-17T07:14:18.481 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:14:18.463159+0000", 2024-09-17T07:14:18.481 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00011206500000000001 2024-09-17T07:14:18.482 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:18.482 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:18.482 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:14:18.482 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:14:18.467718+0000", 2024-09-17T07:14:18.482 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0045590530000000004 2024-09-17T07:14:18.482 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:18.482 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:18.482 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:14:18.482 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:14:18.467767+0000", 2024-09-17T07:14:18.482 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8918999999999997e-05 2024-09-17T07:14:18.482 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:18.482 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:18.482 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:14:18.483 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:14:18.467779+0000", 2024-09-17T07:14:18.483 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2495e-05 2024-09-17T07:14:18.483 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:18.483 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:18.483 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:14:18.483 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:14:18.467815+0000", 2024-09-17T07:14:18.483 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5849999999999997e-05 2024-09-17T07:14:18.483 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:14:18.483 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:14:18.483 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:14:18.483 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:18.483 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:18.483 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.15330.0:171413 9.c e272/236 9:373305e3:::benchmark_data_smithi100_128631_object37147:head v 272'10799, mlcod=272'10797)", 2024-09-17T07:14:18.484 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:14:18.463768+0000", 2024-09-17T07:14:18.484 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.0039471899999999997, 2024-09-17T07:14:18.484 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.004170254, 2024-09-17T07:14:18.484 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:14:18.484 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-17T07:14:18.484 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:14:18.484 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:18.484 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:14:18.484 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:14:18.463768+0000", 2024-09-17T07:14:18.484 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:14:18.484 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:18.484 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:18.484 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:14:18.485 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:14:18.463768+0000", 2024-09-17T07:14:18.485 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:14:18.485 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:18.485 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:18.485 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:14:18.485 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:14:18.463768+0000", 2024-09-17T07:14:18.485 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-17T07:14:18.485 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:18.485 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:18.485 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:14:18.485 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:14:18.463773+0000", 2024-09-17T07:14:18.485 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6230000000000002e-06 2024-09-17T07:14:18.486 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:18.486 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:18.486 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:14:18.486 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:14:18.463785+0000", 2024-09-17T07:14:18.486 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1938e-05 2024-09-17T07:14:18.486 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:18.486 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:18.487 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:14:18.487 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:14:18.463791+0000", 2024-09-17T07:14:18.487 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7930000000000003e-06 2024-09-17T07:14:18.487 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:18.487 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:18.487 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:14:18.487 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:14:18.463841+0000", 2024-09-17T07:14:18.487 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9660000000000002e-05 2024-09-17T07:14:18.487 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:18.487 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:18.487 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:14:18.487 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:14:18.463858+0000", 2024-09-17T07:14:18.488 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7737e-05 2024-09-17T07:14:18.488 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:14:18.488 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:14:18.488 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:14:18.488 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:18.488 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:18.488 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop_reply(client.15330.0:171410 9.5 e272/236 ondisk, result = 0)", 2024-09-17T07:14:18.488 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:14:18.467646+0000", 2024-09-17T07:14:18.488 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 6.9808999999999995e-05, 2024-09-17T07:14:18.488 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00034290200000000002, 2024-09-17T07:14:18.489 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:14:18.489 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-17T07:14:18.489 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:14:18.489 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:18.489 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:14:18.489 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:14:18.467646+0000", 2024-09-17T07:14:18.489 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:14:18.489 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:18.489 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:18.489 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:14:18.489 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:14:18.467646+0000", 2024-09-17T07:14:18.489 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:14:18.489 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:18.490 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:18.490 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:14:18.490 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:14:18.467645+0000", 2024-09-17T07:14:18.490 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-17T07:14:18.490 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:18.490 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:18.490 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:14:18.490 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:14:18.467648+0000", 2024-09-17T07:14:18.490 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7070000000000001e-06 2024-09-17T07:14:18.490 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:18.490 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:18.490 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:14:18.491 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:14:18.467654+0000", 2024-09-17T07:14:18.491 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7010000000000001e-06 2024-09-17T07:14:18.491 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:18.491 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:18.491 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:14:18.491 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:14:18.467659+0000", 2024-09-17T07:14:18.491 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5990000000000001e-06 2024-09-17T07:14:18.491 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:18.491 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:18.491 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:14:18.491 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:14:18.467743+0000", 2024-09-17T07:14:18.491 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3972000000000002e-05 2024-09-17T07:14:18.491 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:18.492 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:18.492 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:14:18.492 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:14:18.467750+0000", 2024-09-17T07:14:18.492 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5239999999999997e-06 2024-09-17T07:14:18.492 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:18.492 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:18.492 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:14:18.492 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:14:18.467989+0000", 2024-09-17T07:14:18.492 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00023870900000000001 2024-09-17T07:14:18.492 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:14:18.492 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:14:18.492 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:14:18.493 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:18.493 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:18.493 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.15330.0:171417 9.5 9:a0443b57:::benchmark_data_smithi100_128631_object37151:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e272)", 2024-09-17T07:14:18.493 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:14:18.467717+0000", 2024-09-17T07:14:18.493 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4294967295.9999986, 2024-09-17T07:14:18.493 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00034051300000000002, 2024-09-17T07:14:18.493 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:14:18.493 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-17T07:14:18.493 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:14:18.493 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.15330", 2024-09-17T07:14:18.493 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/1361381749", 2024-09-17T07:14:18.493 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 171417 2024-09-17T07:14:18.494 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:18.494 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:14:18.494 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:18.494 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:14:18.494 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:14:18.467717+0000", 2024-09-17T07:14:18.494 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:14:18.494 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:18.494 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:18.494 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:14:18.494 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:14:18.467717+0000", 2024-09-17T07:14:18.494 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:14:18.494 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:18.494 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:18.495 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:14:18.495 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:14:18.467715+0000", 2024-09-17T07:14:18.495 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999981 2024-09-17T07:14:18.495 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:18.495 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:18.495 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:14:18.495 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:14:18.467722+0000", 2024-09-17T07:14:18.495 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9650000000000002e-06 2024-09-17T07:14:18.495 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:18.495 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:18.495 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:14:18.495 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:14:18.467732+0000", 2024-09-17T07:14:18.495 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0397000000000001e-05 2024-09-17T07:14:18.495 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:18.496 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:18.496 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:14:18.496 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:14:18.467739+0000", 2024-09-17T07:14:18.496 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5180000000000001e-06 2024-09-17T07:14:18.496 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:18.496 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:18.496 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:14:18.496 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:14:18.468007+0000", 2024-09-17T07:14:18.496 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00026836900000000002 2024-09-17T07:14:18.496 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:14:18.496 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:14:18.496 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:14:18.497 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:18.497 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:18.497 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.15330.0:171406 9.4 e272/236 9:26d1d5d8:::benchmark_data_smithi100_128631_object37140:head v 272'10786, mlcod=272'10784)", 2024-09-17T07:14:18.497 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:14:18.461118+0000", 2024-09-17T07:14:18.497 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.0065973480000000003, 2024-09-17T07:14:18.497 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.006994292, 2024-09-17T07:14:18.497 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:14:18.497 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-17T07:14:18.497 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:14:18.497 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:18.497 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:14:18.497 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:14:18.461118+0000", 2024-09-17T07:14:18.497 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:14:18.498 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:18.498 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:18.498 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:14:18.498 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:14:18.461118+0000", 2024-09-17T07:14:18.498 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:14:18.498 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:18.498 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:18.498 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:14:18.498 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:14:18.461117+0000", 2024-09-17T07:14:18.498 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-17T07:14:18.498 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:18.498 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:18.499 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:14:18.499 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:14:18.461130+0000", 2024-09-17T07:14:18.499 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2646000000000001e-05 2024-09-17T07:14:18.499 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:18.499 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:18.499 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:14:18.499 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:14:18.461143+0000", 2024-09-17T07:14:18.499 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.329e-05 2024-09-17T07:14:18.499 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:18.499 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:18.499 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:14:18.499 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:14:18.461149+0000", 2024-09-17T07:14:18.499 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.387e-06 2024-09-17T07:14:18.500 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:18.500 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:18.500 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:14:18.500 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:14:18.461185+0000", 2024-09-17T07:14:18.500 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5506000000000002e-05 2024-09-17T07:14:18.500 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:18.500 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:18.500 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:14:18.500 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:14:18.461205+0000", 2024-09-17T07:14:18.500 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0207000000000002e-05 2024-09-17T07:14:18.500 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:14:18.501 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:14:18.501 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:14:18.501 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:14:18.501 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-17T07:14:18.501 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 6 2024-09-17T07:14:18.501 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-17T07:14:18.545 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:14:18.574 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:18.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:18.576 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:14:18.677 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:14:18.702 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:18.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:18.703 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:14:18.762 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:18 smithi135 ceph-mon[33238]: pgmap v542: 148 pgs: 148 active+clean; 6.3 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 811 B/s rd, 0 B/s wr, 1.02k op/s 2024-09-17T07:14:18.762 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:18 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:14:18.762 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:18 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:14:18.805 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:14:18.837 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:18.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:18.839 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:14:18.939 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:14:18.971 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:18.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:18.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:14:19.045 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:14:18 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:14:19.073 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:14:19.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:19.097 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:19.098 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:14:19.198 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:14:19.231 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:19.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:19.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:14:19.333 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:14:19.369 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:19.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:19.371 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:14:19.472 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:14:19.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:19.506 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:19.506 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:14:19.607 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:14:19.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:19.637 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:19.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:14:19.738 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:14:19.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:19.759 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:19.760 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:14:19.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:19 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:14:19.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:19 smithi135 ceph-mon[33238]: pgmap v543: 148 pgs: 148 active+clean; 6.0 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.45k op/s 2024-09-17T07:14:19.860 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:14:19.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:19 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:14:19.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:19 smithi100 ceph-mon[29092]: pgmap v543: 148 pgs: 148 active+clean; 6.0 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.45k op/s 2024-09-17T07:14:19.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:19 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:14:19.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:19 smithi100 ceph-mon[39462]: pgmap v543: 148 pgs: 148 active+clean; 6.0 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.45k op/s 2024-09-17T07:14:19.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:19.891 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:19.891 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:14:19.992 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:14:20.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:20.014 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:20.015 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:14:20.116 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:14:20.148 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:20.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:20.149 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:14:20.194 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T07:14:20.249 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:14:20.283 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:20.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:20.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:14:20.385 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:14:20.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:20.429 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:20.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:14:20.530 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:14:20.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:20.565 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:20.565 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:14:20.666 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:14:20.709 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:20.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:20.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:14:20.814 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:14:20.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:20.848 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:20.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:14:20.949 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:14:20.978 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:20.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:20.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:14:21.080 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:14:21.115 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:21.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:21.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:14:21.217 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:14:21.254 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:21.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:21.255 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:14:21.356 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:14:21.398 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:21.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:21.399 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:14:21.500 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:14:21.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:21.534 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:21.534 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:14:21.635 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:14:21.666 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:21.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:21.667 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:14:21.768 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:14:21.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:21.808 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:21.808 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:14:21.909 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:14:21.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:21.950 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:21.950 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:14:22.051 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:14:22.091 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:22.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:22.092 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:14:22.193 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:14:22.215 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:22.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:22.216 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:14:22.317 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:14:22.355 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:22.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:22.357 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:14:22.457 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:14:22.488 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:22.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:22.489 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:14:22.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:22 smithi135 ceph-mon[33238]: pgmap v544: 148 pgs: 148 active+clean; 5.9 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.67k op/s 2024-09-17T07:14:22.590 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:14:22.626 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:22.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:22 smithi100 ceph-mon[29092]: pgmap v544: 148 pgs: 148 active+clean; 5.9 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.67k op/s 2024-09-17T07:14:22.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:22 smithi100 ceph-mon[39462]: pgmap v544: 148 pgs: 148 active+clean; 5.9 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.67k op/s 2024-09-17T07:14:22.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:22.628 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:14:22.728 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:14:22.757 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:22.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:22.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:14:22.859 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:14:22.891 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:22.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:22.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:14:22.993 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:14:23.030 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:23.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:23.031 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:14:23.132 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:14:23.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:23.168 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:23.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:14:23.210 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-17T07:14:23.269 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:14:23.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:23.303 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:23.303 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:14:23.404 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:14:23.441 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:23.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:23.442 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:14:23.543 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:14:23.578 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:23.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:23.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:14:23.680 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:14:23.713 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:23.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:23.714 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:14:23.815 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:14:23.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:23.855 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:23.856 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:14:23.956 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:14:23.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:24.000 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:24.000 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:14:24.101 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:14:24.137 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:24.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:24.138 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:14:24.239 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:14:24.267 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:24.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:24.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:14:24.369 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:14:24.403 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:24.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:24.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:14:24.505 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:14:24.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:24.550 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:24.550 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:14:24.578 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:24 smithi100 ceph-mon[39462]: pgmap v545: 148 pgs: 148 active+clean; 5.9 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 572 B/s rd, 0 B/s wr, 948 op/s 2024-09-17T07:14:24.579 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:24 smithi100 ceph-mon[29092]: pgmap v545: 148 pgs: 148 active+clean; 5.9 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 572 B/s rd, 0 B/s wr, 948 op/s 2024-09-17T07:14:24.650 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:14:24.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:24.688 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:24.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:14:24.789 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:14:24.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:24 smithi135 ceph-mon[33238]: pgmap v545: 148 pgs: 148 active+clean; 5.9 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 572 B/s rd, 0 B/s wr, 948 op/s 2024-09-17T07:14:24.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:24.829 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:24.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:14:24.833 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:14:24.834 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:14:24.860 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1: {} 2024-09-17T07:14:24.860 INFO:teuthology.orchestra.run.smithi100.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:14:24.886 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:14:24.886 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:14:24.913 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:14:24.913 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:14:24.930 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:14:24.942 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:14:24.942 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:14:24.965 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:24.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:24.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:14:24.978 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:14:24.978 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:14:25.009 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:14:25.009 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:14:25.039 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:14:25.040 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:14:25.067 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:14:25.107 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:25.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:25.108 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:14:25.209 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:14:25.255 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:25.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:25.256 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:14:25.356 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:14:25.391 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:25.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:25.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:14:25.493 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:14:25.529 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:25.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:25.530 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:14:25.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:25 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/574259629' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:14:25.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:25 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/574259629' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:14:25.631 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:14:25.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:25.669 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:25.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:14:25.770 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:14:25.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:25 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/574259629' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:14:25.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:25.808 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:25.808 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:14:25.909 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:14:25.948 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:25.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:25.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:14:26.050 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:14:26.081 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:26.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:26.082 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:14:26.183 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:14:26.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:26.216 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:26.216 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:14:26.317 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:14:26.328 INFO:teuthology.orchestra.run.smithi100.stderr:reweighted osd.0 to 1 (10000) 2024-09-17T07:14:26.329 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:14:25 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:14:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:14:26.356 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:26.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:26.357 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:14:26.458 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:14:26.488 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:26.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:26.490 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:14:26.590 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:14:26.620 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:26.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:26.621 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:14:26.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:26 smithi100 ceph-mon[29092]: pgmap v546: 148 pgs: 148 active+clean; 5.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.34k op/s 2024-09-17T07:14:26.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:26 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/3591530297' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-17T07:14:26.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:26 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-17T07:14:26.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:26 smithi100 ceph-mon[39462]: pgmap v546: 148 pgs: 148 active+clean; 5.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.34k op/s 2024-09-17T07:14:26.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:26 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/3591530297' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-17T07:14:26.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:26 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-17T07:14:26.721 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:14:26.759 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:26.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:26.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:14:26.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:26 smithi135 ceph-mon[33238]: pgmap v546: 148 pgs: 148 active+clean; 5.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.34k op/s 2024-09-17T07:14:26.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:26 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/3591530297' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-17T07:14:26.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:26 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-17T07:14:26.864 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:14:26.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:26.900 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:26.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:14:27.001 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:14:27.031 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:27.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:27.032 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:14:27.134 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:14:27.135 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:14:27.164 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:27.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:27.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:14:27.266 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:14:27.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:27.300 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:27.300 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:14:27.401 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:14:27.442 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:27.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:27.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:14:27.544 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:14:27.687 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:27 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-17T07:14:27.688 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:27 smithi100 ceph-mon[29092]: osdmap e273: 8 total, 8 up, 7 in 2024-09-17T07:14:27.688 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:27 smithi100 ceph-mon[29092]: pgmap v548: 148 pgs: 148 active+clean; 5.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.34k op/s 2024-09-17T07:14:27.688 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:27 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-17T07:14:27.689 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:27 smithi100 ceph-mon[39462]: osdmap e273: 8 total, 8 up, 7 in 2024-09-17T07:14:27.689 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:27 smithi100 ceph-mon[39462]: pgmap v548: 148 pgs: 148 active+clean; 5.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.34k op/s 2024-09-17T07:14:27.717 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:27.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:27.720 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:14:27.743 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd reweight 1 1 2024-09-17T07:14:27.777 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:14:27.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:27 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-17T07:14:27.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:27 smithi135 ceph-mon[33238]: osdmap e273: 8 total, 8 up, 7 in 2024-09-17T07:14:27.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:27 smithi135 ceph-mon[33238]: pgmap v548: 148 pgs: 148 active+clean; 5.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.34k op/s 2024-09-17T07:14:27.821 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:14:27.861 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:27.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:27.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:14:27.963 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:14:28.005 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:28.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:28.006 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:14:28.107 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:14:28.150 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:28.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:28.151 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:14:28.252 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:14:28.284 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:28.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:28.285 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:14:28.386 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:14:28.421 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:28.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:28.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:14:28.523 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:14:28.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:28.558 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:28.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:14:28.659 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:14:28.687 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:28.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:28.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:14:28.789 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:14:28.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:28.820 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:28.820 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:14:28.895 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:14:28.921 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:14:28.960 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:28.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:28.961 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:14:29.043 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:14:28 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:14:29.062 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:14:29.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:29.102 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:29.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:14:29.203 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:14:29.238 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:29.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:29.239 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:14:29.340 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:14:29.376 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:29.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:29.377 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:14:29.478 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:14:29.513 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:29.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:29.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:14:29.615 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:14:29.653 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:29.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:29.654 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:14:29.755 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:14:29.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:29.792 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:29.793 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:14:29.893 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:14:29.930 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:29.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:29.931 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:14:30.032 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:14:30.063 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:30.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:30.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:14:30.165 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:14:30.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:30.195 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:30.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:14:30.296 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:14:30.332 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:30.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:30.333 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:14:30.385 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:14:30.435 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:14:30.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:30.468 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:30.468 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:14:30.569 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:14:30.607 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:30.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:30.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:14:30.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:30 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:14:30.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:30 smithi100 ceph-mon[29092]: pgmap v549: 148 pgs: 148 active+clean; 5.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.33k op/s 2024-09-17T07:14:30.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:30 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:14:30.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:30 smithi100 ceph-mon[39462]: pgmap v549: 148 pgs: 148 active+clean; 5.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.33k op/s 2024-09-17T07:14:30.708 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:14:30.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:30.741 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:30.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:14:30.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:30 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:14:30.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:30 smithi135 ceph-mon[33238]: pgmap v549: 148 pgs: 148 active+clean; 5.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.33k op/s 2024-09-17T07:14:30.842 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:14:30.871 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:30.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:30.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:14:30.973 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:14:31.011 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:31.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:31.012 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:14:31.113 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:14:31.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:31.144 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:31.144 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:14:31.245 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:14:31.269 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:31.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:31.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:14:31.370 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:14:31.405 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:31.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:31.406 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:14:31.507 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:14:31.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:31.541 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:31.541 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:14:31.642 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:14:31.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:31.675 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:31.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:14:31.776 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:14:31.811 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:31.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:31.812 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:14:31.913 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:14:31.942 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:31.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:31.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:14:32.044 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:14:32.074 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:32.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:32.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:14:32.175 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:14:32.197 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:32.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:32.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:14:32.298 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:14:32.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:32.329 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:32.329 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:14:32.430 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:14:32.467 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:32.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:32.468 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:14:32.569 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:14:32.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:32.591 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:32.591 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:14:32.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:32 smithi100 ceph-mon[29092]: pgmap v550: 148 pgs: 148 active+clean; 5.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.33k op/s 2024-09-17T07:14:32.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:32 smithi100 ceph-mon[39462]: pgmap v550: 148 pgs: 148 active+clean; 5.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.33k op/s 2024-09-17T07:14:32.692 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:14:32.714 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:32.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:32.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:14:32.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:32 smithi135 ceph-mon[33238]: pgmap v550: 148 pgs: 148 active+clean; 5.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.33k op/s 2024-09-17T07:14:32.818 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:14:32.853 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:32.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:32.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:14:32.955 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:14:32.985 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:32.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:32.986 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:14:33.087 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:14:33.103 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-17T07:14:33.103 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:14:33.117 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:33.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:33.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:14:33.219 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:14:33.246 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:33.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:33.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:14:33.348 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:14:33.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:33 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:14:33.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:33 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:14:33.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:33 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:14:33.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:33 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:14:33.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:33 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:14:33.379 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:33 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:14:33.383 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:33.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:33.384 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:14:33.485 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:14:33.517 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:33.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:33.518 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:14:33.619 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:14:33.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:33.655 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:33.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:14:33.756 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:14:33.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:33 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:14:33.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:33 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:14:33.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:33 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:14:33.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:33.795 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:33.795 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:14:33.896 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:14:33.930 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:33.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:33.931 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:14:34.032 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:14:34.070 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:34.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:34.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:14:34.172 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:14:34.183 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-17T07:14:34.200 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:14:34.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:34.211 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:34.211 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:14:34.313 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:14:34.348 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:34.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:34.349 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:14:34.450 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:14:34.480 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:34.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:34.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:14:34.501 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:34 smithi100 ceph-mon[29092]: pgmap v551: 148 pgs: 148 active+clean; 5.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.33k op/s 2024-09-17T07:14:34.501 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:34 smithi100 ceph-mon[29092]: osdmap e274: 8 total, 8 up, 7 in 2024-09-17T07:14:34.501 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:34 smithi100 ceph-mon[39462]: pgmap v551: 148 pgs: 148 active+clean; 5.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.33k op/s 2024-09-17T07:14:34.501 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:34 smithi100 ceph-mon[39462]: osdmap e274: 8 total, 8 up, 7 in 2024-09-17T07:14:34.583 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:14:34.610 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:34.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:34.611 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:14:34.712 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:14:34.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:34.751 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:34.752 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:14:34.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:34 smithi135 ceph-mon[33238]: pgmap v551: 148 pgs: 148 active+clean; 5.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.33k op/s 2024-09-17T07:14:34.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:34 smithi135 ceph-mon[33238]: osdmap e274: 8 total, 8 up, 7 in 2024-09-17T07:14:34.852 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:14:34.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:34.887 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:34.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:14:34.988 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:14:35.025 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:35.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:35.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:14:35.127 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:14:35.159 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:35.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:35.160 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:14:35.261 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:14:35.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:35.301 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:35.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:14:35.402 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:14:35.435 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:35.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:35.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:14:35.537 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:14:35.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:35.565 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:35.566 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:14:35.666 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:14:35.707 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:35.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:35.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:14:35.808 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:14:35.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:35.839 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:35.839 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:14:35.940 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:14:35.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:35.978 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:35.978 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:14:36.079 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:14:36.106 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:36.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:36.107 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:14:36.208 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:14:36.239 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:36.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:36.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:14:36.333 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:14:35 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:14:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:14:36.341 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:14:36.373 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:36.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:36.374 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:14:36.474 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:14:36.508 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:36.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:36.509 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:14:36.610 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:14:36.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:36 smithi100 ceph-mon[29092]: pgmap v553: 148 pgs: 148 active+clean; 4.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.51k op/s 2024-09-17T07:14:36.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:36 smithi100 ceph-mon[39462]: pgmap v553: 148 pgs: 148 active+clean; 4.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.51k op/s 2024-09-17T07:14:36.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:36.641 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:36.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:14:36.742 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:14:36.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:36.782 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:36.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:14:36.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:36 smithi135 ceph-mon[33238]: pgmap v553: 148 pgs: 148 active+clean; 4.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.51k op/s 2024-09-17T07:14:36.883 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:14:36.913 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:36.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:36.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:14:37.015 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:14:37.045 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:37.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:37.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:14:37.147 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:14:37.181 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:37.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:37.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:14:37.283 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:14:37.323 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:37.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:37.325 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:14:37.346 INFO:teuthology.orchestra.run.smithi100.stderr:noscrub is unset 2024-09-17T07:14:37.425 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:14:37.463 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:37.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:37.464 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:14:37.565 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:14:37.598 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:37.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:37.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:14:37.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:37 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/351556601' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:14:37.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:37 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:14:37.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:37 smithi100 ceph-mon[29092]: pgmap v554: 148 pgs: 148 active+clean; 4.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.35k op/s 2024-09-17T07:14:37.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:37 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/351556601' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:14:37.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:37 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:14:37.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:37 smithi100 ceph-mon[39462]: pgmap v554: 148 pgs: 148 active+clean; 4.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.35k op/s 2024-09-17T07:14:37.699 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:14:37.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:37.733 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:37.738 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:14:37.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:37 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/351556601' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:14:37.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:37 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:14:37.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:37 smithi135 ceph-mon[33238]: pgmap v554: 148 pgs: 148 active+clean; 4.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.35k op/s 2024-09-17T07:14:37.839 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:14:37.873 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:37.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:37.874 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:14:37.975 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:14:38.005 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:38.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:38.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:14:38.108 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:14:38.140 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:38.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:38.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:14:38.245 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:14:38.277 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:38.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:38.278 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:14:38.379 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:14:38.417 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:38.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:38.418 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:14:38.519 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:14:38.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:38.555 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:38.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:14:38.577 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:14:38.577 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:14:38.606 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1: {} 2024-09-17T07:14:38.606 INFO:teuthology.orchestra.run.smithi100.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:14:38.606 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:38 smithi100 ceph-mon[29092]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:14:38.606 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:38 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T07:14:38.606 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:38 smithi100 ceph-mon[29092]: osdmap e275: 8 total, 8 up, 7 in 2024-09-17T07:14:38.607 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:38 smithi100 ceph-mon[39462]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:14:38.607 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:38 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T07:14:38.607 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:38 smithi100 ceph-mon[39462]: osdmap e275: 8 total, 8 up, 7 in 2024-09-17T07:14:38.638 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:14:38.639 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:14:38.656 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:14:38.673 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:14:38.674 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:14:38.674 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:14:38.689 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:38.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:38.690 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:14:38.694 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:38 smithi135 ceph-mon[33238]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:14:38.694 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:38 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T07:14:38.694 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:38 smithi135 ceph-mon[33238]: osdmap e275: 8 total, 8 up, 7 in 2024-09-17T07:14:38.705 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:14:38.705 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:14:38.734 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:14:38.734 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:14:38.764 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:14:38.765 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:14:38.790 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:14:38.797 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:14:38.797 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:14:38.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:38.828 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:38.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:14:38.929 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:14:38.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:38.964 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:38.964 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:14:39.043 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:14:38 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:14:39.065 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:14:39.090 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:39.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:39.091 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:14:39.192 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:14:39.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:39.222 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:39.223 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:14:39.323 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:14:39.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:39.370 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:39.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:14:39.373 INFO:teuthology.orchestra.run.smithi100.stderr:reweighted osd.1 to 1 (10000) 2024-09-17T07:14:39.471 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:14:39.498 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:39.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:39.499 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:14:39.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:39 smithi135 ceph-mon[33238]: 5.1a scrub starts 2024-09-17T07:14:39.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:39 smithi135 ceph-mon[33238]: 5.1a scrub ok 2024-09-17T07:14:39.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:39 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/3196192721' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:14:39.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:39 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1092331504' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-17T07:14:39.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:39 smithi135 ceph-mon[33238]: 6.f scrub starts 2024-09-17T07:14:39.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:39 smithi135 ceph-mon[33238]: 6.f scrub ok 2024-09-17T07:14:39.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:39 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:14:39.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:39 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:14:39.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:39 smithi135 ceph-mon[33238]: pgmap v556: 148 pgs: 148 active+clean; 4.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.47k op/s 2024-09-17T07:14:39.600 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:14:39.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:39 smithi100 ceph-mon[29092]: 5.1a scrub starts 2024-09-17T07:14:39.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:39 smithi100 ceph-mon[29092]: 5.1a scrub ok 2024-09-17T07:14:39.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:39 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/3196192721' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:14:39.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:39 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1092331504' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-17T07:14:39.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:39 smithi100 ceph-mon[29092]: 6.f scrub starts 2024-09-17T07:14:39.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:39 smithi100 ceph-mon[29092]: 6.f scrub ok 2024-09-17T07:14:39.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:39 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:14:39.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:39 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:14:39.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:39 smithi100 ceph-mon[29092]: pgmap v556: 148 pgs: 148 active+clean; 4.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.47k op/s 2024-09-17T07:14:39.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:39 smithi100 ceph-mon[39462]: 5.1a scrub starts 2024-09-17T07:14:39.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:39 smithi100 ceph-mon[39462]: 5.1a scrub ok 2024-09-17T07:14:39.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:39 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/3196192721' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:14:39.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:39 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1092331504' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-17T07:14:39.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:39 smithi100 ceph-mon[39462]: 6.f scrub starts 2024-09-17T07:14:39.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:39 smithi100 ceph-mon[39462]: 6.f scrub ok 2024-09-17T07:14:39.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:39 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:14:39.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:39 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:14:39.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:39 smithi100 ceph-mon[39462]: pgmap v556: 148 pgs: 148 active+clean; 4.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.47k op/s 2024-09-17T07:14:39.636 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:39.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:39.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:14:39.738 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:14:39.764 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:39.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:39.765 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:14:39.866 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:14:39.892 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:39.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:39.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:14:39.994 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:14:40.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:40.026 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:40.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:14:40.074 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-17T07:14:40.126 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:14:40.159 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:40.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:40.160 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:14:40.261 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:14:40.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:40.295 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:40.295 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:14:40.396 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:14:40.423 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:40.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:40.424 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:14:40.524 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:14:40.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:40.548 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:40.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:14:40.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:40 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1092331504' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-17T07:14:40.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:40 smithi100 ceph-mon[29092]: osdmap e276: 8 total, 8 up, 7 in 2024-09-17T07:14:40.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:40 smithi100 ceph-mon[29092]: 6.2 scrub starts 2024-09-17T07:14:40.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:40 smithi100 ceph-mon[29092]: 6.2 scrub ok 2024-09-17T07:14:40.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:40 smithi100 ceph-mon[29092]: 2.c scrub starts 2024-09-17T07:14:40.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:40 smithi100 ceph-mon[29092]: 2.c scrub ok 2024-09-17T07:14:40.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:40 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1092331504' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-17T07:14:40.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:40 smithi100 ceph-mon[39462]: osdmap e276: 8 total, 8 up, 7 in 2024-09-17T07:14:40.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:40 smithi100 ceph-mon[39462]: 6.2 scrub starts 2024-09-17T07:14:40.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:40 smithi100 ceph-mon[39462]: 6.2 scrub ok 2024-09-17T07:14:40.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:40 smithi100 ceph-mon[39462]: 2.c scrub starts 2024-09-17T07:14:40.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:40 smithi100 ceph-mon[39462]: 2.c scrub ok 2024-09-17T07:14:40.649 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:14:40.677 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:40.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:40.679 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:14:40.779 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:14:40.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:40 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1092331504' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-17T07:14:40.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:40 smithi135 ceph-mon[33238]: osdmap e276: 8 total, 8 up, 7 in 2024-09-17T07:14:40.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:40 smithi135 ceph-mon[33238]: 6.2 scrub starts 2024-09-17T07:14:40.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:40 smithi135 ceph-mon[33238]: 6.2 scrub ok 2024-09-17T07:14:40.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:40 smithi135 ceph-mon[33238]: 2.c scrub starts 2024-09-17T07:14:40.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:40 smithi135 ceph-mon[33238]: 2.c scrub ok 2024-09-17T07:14:40.809 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:40.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:40.810 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:14:40.911 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:14:40.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:40.938 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:40.939 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:14:41.040 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:14:41.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:41.073 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:41.073 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:14:41.174 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:14:41.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:41.209 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:41.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:14:41.310 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:14:41.342 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:41.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:41.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:14:41.444 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:14:41.483 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:41.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:41.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:14:41.585 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:14:41.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:41.622 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:41.622 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:14:41.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:41 smithi100 ceph-mon[29092]: pgmap v558: 148 pgs: 148 active+clean; 4.3 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.64k op/s 2024-09-17T07:14:41.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:41 smithi100 ceph-mon[39462]: pgmap v558: 148 pgs: 148 active+clean; 4.3 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.64k op/s 2024-09-17T07:14:41.723 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:14:41.759 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:41.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:41.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:14:41.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:41 smithi135 ceph-mon[33238]: pgmap v558: 148 pgs: 148 active+clean; 4.3 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.64k op/s 2024-09-17T07:14:41.862 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:14:41.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:41.894 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:41.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:14:41.995 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:14:42.032 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:42.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:42.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:14:42.134 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:14:42.165 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:42.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:42.167 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:14:42.267 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:14:42.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:42.289 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:42.290 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:14:42.390 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:14:42.422 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:42.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:42.424 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:14:42.524 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:14:42.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:42.563 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:42.563 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:14:42.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:42 smithi100 ceph-mon[39462]: 9.4 scrub starts 2024-09-17T07:14:42.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:42 smithi100 ceph-mon[39462]: 2.13 scrub starts 2024-09-17T07:14:42.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:42 smithi100 ceph-mon[39462]: 2.13 scrub ok 2024-09-17T07:14:42.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:42 smithi100 ceph-mon[39462]: 6.1a scrub starts 2024-09-17T07:14:42.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:42 smithi100 ceph-mon[39462]: 6.1a scrub ok 2024-09-17T07:14:42.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:42 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:14:42.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:42 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:14:42.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:42 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:14:42.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:42 smithi100 ceph-mon[29092]: 9.4 scrub starts 2024-09-17T07:14:42.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:42 smithi100 ceph-mon[29092]: 2.13 scrub starts 2024-09-17T07:14:42.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:42 smithi100 ceph-mon[29092]: 2.13 scrub ok 2024-09-17T07:14:42.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:42 smithi100 ceph-mon[29092]: 6.1a scrub starts 2024-09-17T07:14:42.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:42 smithi100 ceph-mon[29092]: 6.1a scrub ok 2024-09-17T07:14:42.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:42 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:14:42.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:42 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:14:42.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:42 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:14:42.663 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:14:42.692 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:42.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:42.693 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:14:42.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:42 smithi135 ceph-mon[33238]: 9.4 scrub starts 2024-09-17T07:14:42.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:42 smithi135 ceph-mon[33238]: 2.13 scrub starts 2024-09-17T07:14:42.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:42 smithi135 ceph-mon[33238]: 2.13 scrub ok 2024-09-17T07:14:42.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:42 smithi135 ceph-mon[33238]: 6.1a scrub starts 2024-09-17T07:14:42.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:42 smithi135 ceph-mon[33238]: 6.1a scrub ok 2024-09-17T07:14:42.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:42 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:14:42.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:42 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:14:42.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:42 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:14:42.794 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:14:42.835 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:42.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:42.836 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:14:42.937 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:14:42.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:42.973 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:42.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:14:43.074 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:14:43.082 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd reweight 2 1 2024-09-17T07:14:43.118 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:43.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:43.120 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:14:43.220 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:14:43.252 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:43.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:43.253 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:14:43.354 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:14:43.385 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:43.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:43.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:14:43.488 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:14:43.516 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:43.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:43.517 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:14:43.569 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:14:43.620 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:14:43.652 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:43.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:43.653 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:14:43.706 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:43 smithi100 ceph-mon[39462]: 5.2 scrub starts 2024-09-17T07:14:43.706 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:43 smithi100 ceph-mon[39462]: 5.2 scrub ok 2024-09-17T07:14:43.706 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:43 smithi100 ceph-mon[39462]: 5.9 scrub starts 2024-09-17T07:14:43.706 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:43 smithi100 ceph-mon[39462]: 5.9 scrub ok 2024-09-17T07:14:43.706 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:43 smithi100 ceph-mon[39462]: 9.4 scrub ok 2024-09-17T07:14:43.706 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:43 smithi100 ceph-mon[39462]: 4.1b scrub starts 2024-09-17T07:14:43.707 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:43 smithi100 ceph-mon[39462]: 4.1b scrub ok 2024-09-17T07:14:43.707 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:43 smithi100 ceph-mon[39462]: 4.10 scrub starts 2024-09-17T07:14:43.707 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:43 smithi100 ceph-mon[39462]: 4.10 scrub ok 2024-09-17T07:14:43.707 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:43 smithi100 ceph-mon[39462]: pgmap v559: 148 pgs: 148 active+clean; 4.3 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 846 op/s 2024-09-17T07:14:43.707 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:43 smithi100 ceph-mon[29092]: 5.2 scrub starts 2024-09-17T07:14:43.707 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:43 smithi100 ceph-mon[29092]: 5.2 scrub ok 2024-09-17T07:14:43.707 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:43 smithi100 ceph-mon[29092]: 5.9 scrub starts 2024-09-17T07:14:43.707 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:43 smithi100 ceph-mon[29092]: 5.9 scrub ok 2024-09-17T07:14:43.707 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:43 smithi100 ceph-mon[29092]: 9.4 scrub ok 2024-09-17T07:14:43.707 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:43 smithi100 ceph-mon[29092]: 4.1b scrub starts 2024-09-17T07:14:43.707 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:43 smithi100 ceph-mon[29092]: 4.1b scrub ok 2024-09-17T07:14:43.708 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:43 smithi100 ceph-mon[29092]: 4.10 scrub starts 2024-09-17T07:14:43.708 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:43 smithi100 ceph-mon[29092]: 4.10 scrub ok 2024-09-17T07:14:43.708 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:43 smithi100 ceph-mon[29092]: pgmap v559: 148 pgs: 148 active+clean; 4.3 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 846 op/s 2024-09-17T07:14:43.753 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:14:43.792 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:43.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:43.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:14:43.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:43 smithi135 ceph-mon[33238]: 5.2 scrub starts 2024-09-17T07:14:43.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:43 smithi135 ceph-mon[33238]: 5.2 scrub ok 2024-09-17T07:14:43.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:43 smithi135 ceph-mon[33238]: 5.9 scrub starts 2024-09-17T07:14:43.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:43 smithi135 ceph-mon[33238]: 5.9 scrub ok 2024-09-17T07:14:43.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:43 smithi135 ceph-mon[33238]: 9.4 scrub ok 2024-09-17T07:14:43.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:43 smithi135 ceph-mon[33238]: 4.1b scrub starts 2024-09-17T07:14:43.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:43 smithi135 ceph-mon[33238]: 4.1b scrub ok 2024-09-17T07:14:43.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:43 smithi135 ceph-mon[33238]: 4.10 scrub starts 2024-09-17T07:14:43.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:43 smithi135 ceph-mon[33238]: 4.10 scrub ok 2024-09-17T07:14:43.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:43 smithi135 ceph-mon[33238]: pgmap v559: 148 pgs: 148 active+clean; 4.3 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 846 op/s 2024-09-17T07:14:43.861 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:14:43.894 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:14:43.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:43.928 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:43.929 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:14:44.030 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:14:44.071 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:44.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:44.072 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:14:44.173 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:14:44.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:44.202 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:44.202 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:14:44.303 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:14:44.332 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:44.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:44.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:14:44.434 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:14:44.471 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:44.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:44.472 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:14:44.573 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:14:44.603 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:44.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:44.604 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:14:44.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:44 smithi100 ceph-mon[29092]: 4.5 scrub starts 2024-09-17T07:14:44.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:44 smithi100 ceph-mon[39462]: 4.5 scrub starts 2024-09-17T07:14:44.694 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:44 smithi135 ceph-mon[33238]: 4.5 scrub starts 2024-09-17T07:14:44.705 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:14:44.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:44.734 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:44.734 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:14:44.835 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:14:44.861 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:44.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:44.862 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:14:44.963 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:14:44.997 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:44.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:44.998 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:14:45.100 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:14:45.140 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:45.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:45.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:14:45.242 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:14:45.275 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:45.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:45.276 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:14:45.331 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-17T07:14:45.331 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-17T07:14:45.331 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-17T07:14:45.331 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-17T07:14:45.332 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.332 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.14805.0:54890 7.14 7:2f55ae45:::benchmark_data_smithi100_78556_object54889:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e111)", 2024-09-17T07:14:45.332 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:04:57.343812+0000", 2024-09-17T07:14:45.332 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 587.98466056100006, 2024-09-17T07:14:45.332 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0203684420000001, 2024-09-17T07:14:45.332 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:14:45.332 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-17T07:14:45.332 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:14:45.332 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.14805", 2024-09-17T07:14:45.332 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:14:45.332 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 54890 2024-09-17T07:14:45.333 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.333 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:14:45.333 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.333 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:14:45.333 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.343812+0000", 2024-09-17T07:14:45.333 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:14:45.333 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.333 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.333 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:14:45.334 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.343812+0000", 2024-09-17T07:14:45.334 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:14:45.334 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.334 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.334 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:14:45.334 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.343811+0000", 2024-09-17T07:14:45.334 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-17T07:14:45.334 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.334 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.334 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:14:45.334 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.343835+0000", 2024-09-17T07:14:45.334 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3618e-05 2024-09-17T07:14:45.335 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.335 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.335 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:14:45.335 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.343841+0000", 2024-09-17T07:14:45.335 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9440000000000002e-06 2024-09-17T07:14:45.335 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.335 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.335 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:14:45.335 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.343845+0000", 2024-09-17T07:14:45.336 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5870000000000003e-06 2024-09-17T07:14:45.336 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.336 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.336 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:14:45.336 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.485155+0000", 2024-09-17T07:14:45.336 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.141309198 2024-09-17T07:14:45.336 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.336 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.336 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-17T07:14:45.336 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.485163+0000", 2024-09-17T07:14:45.336 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0909999999999993e-06 2024-09-17T07:14:45.336 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.337 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.337 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:14:45.337 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:58.356461+0000", 2024-09-17T07:14:45.337 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.871298605 2024-09-17T07:14:45.337 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.337 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.337 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for degraded object", 2024-09-17T07:14:45.337 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:58.359041+0000", 2024-09-17T07:14:45.337 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0025792749999999998 2024-09-17T07:14:45.337 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.337 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.337 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:14:45.338 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:58.364109+0000", 2024-09-17T07:14:45.338 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.005068197 2024-09-17T07:14:45.338 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.338 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.338 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:14:45.338 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:58.364180+0000", 2024-09-17T07:14:45.338 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1373999999999998e-05 2024-09-17T07:14:45.338 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:14:45.338 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:14:45.338 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:14:45.339 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.339 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.339 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.14805.0:54891 7.16 7:691d1cb1:::benchmark_data_smithi100_78556_object54890:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e111)", 2024-09-17T07:14:45.339 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:04:57.343856+0000", 2024-09-17T07:14:45.339 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 587.98461594100002, 2024-09-17T07:14:45.339 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0321880130000001, 2024-09-17T07:14:45.339 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:14:45.339 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-17T07:14:45.339 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:14:45.339 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.14805", 2024-09-17T07:14:45.339 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:14:45.339 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 54891 2024-09-17T07:14:45.339 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.340 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:14:45.340 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.340 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:14:45.340 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.343856+0000", 2024-09-17T07:14:45.340 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:14:45.340 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.340 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.340 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:14:45.341 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.343856+0000", 2024-09-17T07:14:45.341 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:14:45.341 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.341 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.341 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:14:45.341 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.343856+0000", 2024-09-17T07:14:45.341 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-17T07:14:45.341 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.342 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.342 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:14:45.342 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.343879+0000", 2024-09-17T07:14:45.342 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.302e-05 2024-09-17T07:14:45.342 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.342 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.342 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:14:45.342 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.343885+0000", 2024-09-17T07:14:45.342 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2339999999999999e-06 2024-09-17T07:14:45.343 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.343 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.343 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:14:45.343 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.343890+0000", 2024-09-17T07:14:45.343 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8359999999999999e-06 2024-09-17T07:14:45.344 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.344 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.344 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:14:45.344 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.486174+0000", 2024-09-17T07:14:45.344 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.14228400799999999 2024-09-17T07:14:45.344 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.344 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.344 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-17T07:14:45.344 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.486189+0000", 2024-09-17T07:14:45.344 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4939999999999999e-05 2024-09-17T07:14:45.344 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.344 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.345 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:14:45.345 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:58.375979+0000", 2024-09-17T07:14:45.345 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.88978975800000004 2024-09-17T07:14:45.345 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.345 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.345 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:14:45.345 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:58.376044+0000", 2024-09-17T07:14:45.346 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5617999999999994e-05 2024-09-17T07:14:45.346 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:14:45.346 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:14:45.346 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:14:45.346 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.346 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.346 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.14805.0:54895 7.6 7:63ba05c1:::benchmark_data_smithi100_78556_object54894:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e111)", 2024-09-17T07:14:45.346 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:04:57.343895+0000", 2024-09-17T07:14:45.347 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 587.98457727799996, 2024-09-17T07:14:45.347 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0193373509999999, 2024-09-17T07:14:45.347 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:14:45.347 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-17T07:14:45.347 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:14:45.347 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.14805", 2024-09-17T07:14:45.347 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:14:45.347 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 54895 2024-09-17T07:14:45.347 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.347 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:14:45.347 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.347 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:14:45.347 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.343895+0000", 2024-09-17T07:14:45.348 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:14:45.348 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.348 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.348 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:14:45.348 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.343895+0000", 2024-09-17T07:14:45.348 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:14:45.348 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.348 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.348 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:14:45.348 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.343895+0000", 2024-09-17T07:14:45.348 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-17T07:14:45.348 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.349 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.349 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:14:45.349 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.343917+0000", 2024-09-17T07:14:45.349 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1951e-05 2024-09-17T07:14:45.349 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.349 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.349 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:14:45.349 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.343922+0000", 2024-09-17T07:14:45.349 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4430000000000002e-06 2024-09-17T07:14:45.349 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.350 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.350 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:14:45.350 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.343926+0000", 2024-09-17T07:14:45.350 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.296e-06 2024-09-17T07:14:45.352 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.352 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.353 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:14:45.353 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.344913+0000", 2024-09-17T07:14:45.353 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00098710500000000006 2024-09-17T07:14:45.353 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.353 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.353 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op must wait for map", 2024-09-17T07:14:45.353 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.344918+0000", 2024-09-17T07:14:45.353 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0159999999999999e-06 2024-09-17T07:14:45.353 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.353 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.353 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:14:45.353 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.482949+0000", 2024-09-17T07:14:45.354 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.13803019799999999 2024-09-17T07:14:45.354 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.354 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.354 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-17T07:14:45.354 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.482961+0000", 2024-09-17T07:14:45.354 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2052000000000001e-05 2024-09-17T07:14:45.354 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.355 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.355 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:14:45.355 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:58.363163+0000", 2024-09-17T07:14:45.355 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.88020196299999998 2024-09-17T07:14:45.355 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.355 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.355 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:14:45.355 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:58.363232+0000", 2024-09-17T07:14:45.355 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9688000000000001e-05 2024-09-17T07:14:45.356 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:14:45.356 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:14:45.356 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:14:45.356 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.356 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.356 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.14805.0:54896 7.16 7:68fa3457:::benchmark_data_smithi100_78556_object54895:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e111)", 2024-09-17T07:14:45.356 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:04:57.343930+0000", 2024-09-17T07:14:45.356 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 587.98454199499997, 2024-09-17T07:14:45.356 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.035847894, 2024-09-17T07:14:45.357 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:14:45.357 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-17T07:14:45.357 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:14:45.357 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.14805", 2024-09-17T07:14:45.357 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:14:45.357 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 54896 2024-09-17T07:14:45.357 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.357 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:14:45.358 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.358 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:14:45.358 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.343930+0000", 2024-09-17T07:14:45.358 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:14:45.358 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.358 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.358 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:14:45.358 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.343930+0000", 2024-09-17T07:14:45.359 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:14:45.359 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.359 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.359 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:14:45.359 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.343930+0000", 2024-09-17T07:14:45.359 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-17T07:14:45.359 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.359 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.359 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:14:45.359 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.343994+0000", 2024-09-17T07:14:45.359 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4443999999999995e-05 2024-09-17T07:14:45.359 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.360 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.360 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:14:45.360 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.344000+0000", 2024-09-17T07:14:45.360 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7270000000000004e-06 2024-09-17T07:14:45.360 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.360 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.360 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:14:45.360 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.344005+0000", 2024-09-17T07:14:45.361 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.527e-06 2024-09-17T07:14:45.361 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.361 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.361 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:14:45.361 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.486209+0000", 2024-09-17T07:14:45.361 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.14220397200000001 2024-09-17T07:14:45.361 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.361 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.361 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-17T07:14:45.362 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.486214+0000", 2024-09-17T07:14:45.362 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8250000000000001e-06 2024-09-17T07:14:45.362 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.362 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.362 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:14:45.363 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:58.376324+0000", 2024-09-17T07:14:45.363 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.89010998200000002 2024-09-17T07:14:45.363 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.363 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.363 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for degraded object", 2024-09-17T07:14:45.363 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:58.377026+0000", 2024-09-17T07:14:45.363 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00070122200000000004 2024-09-17T07:14:45.363 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.364 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.364 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:14:45.364 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:58.379701+0000", 2024-09-17T07:14:45.364 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0026749579999999998 2024-09-17T07:14:45.364 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.364 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.364 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:14:45.364 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:58.379778+0000", 2024-09-17T07:14:45.364 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7606000000000002e-05 2024-09-17T07:14:45.364 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:14:45.364 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:14:45.365 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:14:45.365 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.365 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.365 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.14805.0:54905 7.14 7:2f70dbc4:::benchmark_data_smithi100_78556_object54904:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-09-17T07:14:45.365 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:04:57.346340+0000", 2024-09-17T07:14:45.365 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 587.982132643, 2024-09-17T07:14:45.365 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0173668300000001, 2024-09-17T07:14:45.365 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:14:45.366 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:14:45.366 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:14:45.366 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.14805", 2024-09-17T07:14:45.366 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:14:45.366 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 54905 2024-09-17T07:14:45.366 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.366 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:14:45.366 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.366 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:14:45.366 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.346340+0000", 2024-09-17T07:14:45.366 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:14:45.366 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.366 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.367 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:14:45.367 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.346340+0000", 2024-09-17T07:14:45.367 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:14:45.367 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.367 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.367 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:14:45.367 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.346338+0000", 2024-09-17T07:14:45.367 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999986 2024-09-17T07:14:45.367 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.367 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.368 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:14:45.368 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.346366+0000", 2024-09-17T07:14:45.368 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.811e-05 2024-09-17T07:14:45.368 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.368 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.369 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:14:45.369 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.346377+0000", 2024-09-17T07:14:45.369 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1100999999999999e-05 2024-09-17T07:14:45.369 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.369 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.369 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:14:45.369 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.346388+0000", 2024-09-17T07:14:45.369 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0282000000000001e-05 2024-09-17T07:14:45.369 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.370 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.370 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:14:45.370 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.485178+0000", 2024-09-17T07:14:45.370 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.13879050700000001 2024-09-17T07:14:45.370 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.370 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.370 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-17T07:14:45.370 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:57.485183+0000", 2024-09-17T07:14:45.370 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0329999999999999e-06 2024-09-17T07:14:45.370 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.370 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.370 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:14:45.370 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:58.359332+0000", 2024-09-17T07:14:45.371 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.87414892399999999 2024-09-17T07:14:45.371 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.371 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.371 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:14:45.371 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:58.359468+0000", 2024-09-17T07:14:45.371 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000136162 2024-09-17T07:14:45.371 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.371 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.371 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 1,4", 2024-09-17T07:14:45.371 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:58.359621+0000", 2024-09-17T07:14:45.371 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000153051 2024-09-17T07:14:45.371 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.371 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.372 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:14:45.372 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:58.362451+0000", 2024-09-17T07:14:45.372 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0028294259999999999 2024-09-17T07:14:45.372 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.372 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.372 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:14:45.372 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:58.363057+0000", 2024-09-17T07:14:45.372 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00060569200000000001 2024-09-17T07:14:45.372 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.372 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.372 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:14:45.373 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:58.363591+0000", 2024-09-17T07:14:45.373 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00053405499999999997 2024-09-17T07:14:45.373 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.373 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.373 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:14:45.373 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:58.363631+0000", 2024-09-17T07:14:45.373 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0664999999999998e-05 2024-09-17T07:14:45.373 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.374 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.374 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:14:45.374 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:04:58.363707+0000", 2024-09-17T07:14:45.374 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5242999999999998e-05 2024-09-17T07:14:45.374 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:14:45.374 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:14:45.374 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:14:45.374 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.374 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.374 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.14805.0:84002 7.18 7:1e146e87:::benchmark_data_smithi100_78556_object84001:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-09-17T07:14:45.374 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:05:30.141882+0000", 2024-09-17T07:14:45.374 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 555.18659051999998, 2024-09-17T07:14:45.375 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0090834440000001, 2024-09-17T07:14:45.375 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:14:45.375 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:14:45.375 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:14:45.375 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.14805", 2024-09-17T07:14:45.375 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:14:45.375 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 84002 2024-09-17T07:14:45.375 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.375 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:14:45.375 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.375 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:14:45.375 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.141882+0000", 2024-09-17T07:14:45.376 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:14:45.376 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.376 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.376 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:14:45.376 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.141882+0000", 2024-09-17T07:14:45.376 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:14:45.376 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.376 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.376 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:14:45.376 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.141880+0000", 2024-09-17T07:14:45.376 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999981 2024-09-17T07:14:45.376 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.377 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.377 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:14:45.377 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.141913+0000", 2024-09-17T07:14:45.377 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3009999999999997e-05 2024-09-17T07:14:45.377 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.377 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.377 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:14:45.377 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.141926+0000", 2024-09-17T07:14:45.377 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2680999999999999e-05 2024-09-17T07:14:45.377 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.377 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.377 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:14:45.378 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.141934+0000", 2024-09-17T07:14:45.378 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9850000000000006e-06 2024-09-17T07:14:45.378 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.378 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.378 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:14:45.378 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.145584+0000", 2024-09-17T07:14:45.378 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.003650665 2024-09-17T07:14:45.378 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.378 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.378 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-17T07:14:45.378 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.145594+0000", 2024-09-17T07:14:45.378 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.8570000000000007e-06 2024-09-17T07:14:45.378 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.379 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.379 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:14:45.379 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.147036+0000", 2024-09-17T07:14:45.379 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.001441904 2024-09-17T07:14:45.379 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.379 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.379 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:14:45.379 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.147199+0000", 2024-09-17T07:14:45.379 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016296799999999999 2024-09-17T07:14:45.379 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.379 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.379 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6,7", 2024-09-17T07:14:45.379 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.147348+0000", 2024-09-17T07:14:45.380 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014852800000000001 2024-09-17T07:14:45.380 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.380 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.380 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:14:45.380 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.150802+0000", 2024-09-17T07:14:45.380 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.003454188 2024-09-17T07:14:45.380 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.380 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.380 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:14:45.380 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.150845+0000", 2024-09-17T07:14:45.381 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3349000000000001e-05 2024-09-17T07:14:45.381 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.381 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.381 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:14:45.381 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.150853+0000", 2024-09-17T07:14:45.381 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3590000000000008e-06 2024-09-17T07:14:45.381 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.381 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.381 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:14:45.381 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.150924+0000", 2024-09-17T07:14:45.382 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0389999999999995e-05 2024-09-17T07:14:45.382 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.382 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.382 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:14:45.382 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.150965+0000", 2024-09-17T07:14:45.382 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1443000000000003e-05 2024-09-17T07:14:45.382 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:14:45.382 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:14:45.382 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:14:45.382 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.382 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.383 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.14805.0:84044 7.18 7:1f7aa994:::benchmark_data_smithi100_78556_object84043:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-09-17T07:14:45.383 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:05:30.182888+0000", 2024-09-17T07:14:45.383 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 555.14558423999995, 2024-09-17T07:14:45.383 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.97126389700000004, 2024-09-17T07:14:45.383 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:14:45.383 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:14:45.383 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:14:45.383 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.14805", 2024-09-17T07:14:45.383 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:14:45.384 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 84044 2024-09-17T07:14:45.384 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.384 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:14:45.384 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.384 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:14:45.384 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.182888+0000", 2024-09-17T07:14:45.384 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:14:45.384 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.384 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.384 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:14:45.384 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.182888+0000", 2024-09-17T07:14:45.384 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:14:45.385 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.385 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.385 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:14:45.385 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.182887+0000", 2024-09-17T07:14:45.385 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-17T07:14:45.385 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.385 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.386 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:14:45.386 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.182908+0000", 2024-09-17T07:14:45.386 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.086e-05 2024-09-17T07:14:45.386 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.386 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.386 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:14:45.386 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.182915+0000", 2024-09-17T07:14:45.387 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5270000000000001e-06 2024-09-17T07:14:45.387 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.387 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.387 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:14:45.387 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.182921+0000", 2024-09-17T07:14:45.387 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.305e-06 2024-09-17T07:14:45.387 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.387 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.387 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:14:45.388 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.183176+0000", 2024-09-17T07:14:45.388 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00025525999999999999 2024-09-17T07:14:45.388 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.388 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.388 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-17T07:14:45.388 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.183186+0000", 2024-09-17T07:14:45.388 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9170000000000001e-06 2024-09-17T07:14:45.388 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.388 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.389 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:14:45.389 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.149202+0000", 2024-09-17T07:14:45.389 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.96601636199999996 2024-09-17T07:14:45.389 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.389 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.389 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:14:45.389 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.149325+0000", 2024-09-17T07:14:45.389 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012303300000000001 2024-09-17T07:14:45.389 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.389 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.389 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6,7", 2024-09-17T07:14:45.390 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.149466+0000", 2024-09-17T07:14:45.390 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014086700000000001 2024-09-17T07:14:45.390 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.390 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.390 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:14:45.390 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.151097+0000", 2024-09-17T07:14:45.390 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001630793 2024-09-17T07:14:45.390 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.390 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.390 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:14:45.390 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.151156+0000", 2024-09-17T07:14:45.391 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9160000000000003e-05 2024-09-17T07:14:45.391 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.391 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.391 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:14:45.392 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.154052+0000", 2024-09-17T07:14:45.392 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0028960169999999999 2024-09-17T07:14:45.392 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.392 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.392 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:14:45.392 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.154100+0000", 2024-09-17T07:14:45.392 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8062000000000002e-05 2024-09-17T07:14:45.393 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.393 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.393 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:14:45.393 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.154152+0000", 2024-09-17T07:14:45.393 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1833000000000003e-05 2024-09-17T07:14:45.393 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:14:45.393 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:14:45.394 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:14:45.394 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.394 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.394 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.14805.0:84080 7.18 7:1999f9fa:::benchmark_data_smithi100_78556_object84079:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-09-17T07:14:45.394 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:05:30.213543+0000", 2024-09-17T07:14:45.394 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 555.11492910100003, 2024-09-17T07:14:45.394 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.94079719699999997, 2024-09-17T07:14:45.395 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:14:45.395 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:14:45.395 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:14:45.395 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.14805", 2024-09-17T07:14:45.395 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:14:45.395 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 84080 2024-09-17T07:14:45.395 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.395 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:14:45.395 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.395 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:14:45.395 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.213543+0000", 2024-09-17T07:14:45.396 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:14:45.396 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.396 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.396 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:14:45.396 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.213543+0000", 2024-09-17T07:14:45.396 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:14:45.396 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.396 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.396 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:14:45.397 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.213542+0000", 2024-09-17T07:14:45.397 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-17T07:14:45.397 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.397 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.397 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:14:45.397 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.213564+0000", 2024-09-17T07:14:45.397 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1260999999999999e-05 2024-09-17T07:14:45.397 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.398 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.398 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:14:45.398 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.213574+0000", 2024-09-17T07:14:45.398 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0071000000000001e-05 2024-09-17T07:14:45.398 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.399 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.399 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:14:45.399 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.213580+0000", 2024-09-17T07:14:45.399 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.421e-06 2024-09-17T07:14:45.399 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.399 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.399 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:14:45.400 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.213839+0000", 2024-09-17T07:14:45.400 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00025876000000000002 2024-09-17T07:14:45.400 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.400 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.400 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-17T07:14:45.400 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.213849+0000", 2024-09-17T07:14:45.400 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.8400000000000007e-06 2024-09-17T07:14:45.400 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.400 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.400 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:14:45.400 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.151200+0000", 2024-09-17T07:14:45.400 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.93735180600000001 2024-09-17T07:14:45.401 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.401 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.401 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:14:45.401 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.151318+0000", 2024-09-17T07:14:45.401 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000117757 2024-09-17T07:14:45.401 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.401 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.401 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6,7", 2024-09-17T07:14:45.401 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.151475+0000", 2024-09-17T07:14:45.401 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00015659499999999999 2024-09-17T07:14:45.401 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.401 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.401 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:14:45.402 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.154041+0000", 2024-09-17T07:14:45.402 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0025661619999999999 2024-09-17T07:14:45.402 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.402 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.402 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:14:45.402 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.154190+0000", 2024-09-17T07:14:45.402 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014864300000000001 2024-09-17T07:14:45.402 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.402 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.402 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:14:45.402 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.154196+0000", 2024-09-17T07:14:45.403 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5039999999999999e-06 2024-09-17T07:14:45.403 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.403 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.403 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:14:45.403 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.154234+0000", 2024-09-17T07:14:45.403 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7562999999999999e-05 2024-09-17T07:14:45.403 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.403 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.403 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:14:45.403 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.154340+0000", 2024-09-17T07:14:45.403 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000106777 2024-09-17T07:14:45.403 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:14:45.404 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:14:45.404 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:14:45.404 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.404 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.404 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.14805.0:84111 7.18 7:1dd58e1f:::benchmark_data_smithi100_78556_object84110:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-09-17T07:14:45.404 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:05:30.240539+0000", 2024-09-17T07:14:45.404 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 555.08793326700004, 2024-09-17T07:14:45.404 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.91743996999999999, 2024-09-17T07:14:45.404 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:14:45.404 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:14:45.404 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:14:45.404 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.14805", 2024-09-17T07:14:45.404 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:14:45.405 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 84111 2024-09-17T07:14:45.405 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.405 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:14:45.405 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.405 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:14:45.405 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.240539+0000", 2024-09-17T07:14:45.405 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:14:45.405 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.405 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.406 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:14:45.406 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.240539+0000", 2024-09-17T07:14:45.406 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:14:45.406 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.406 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.406 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:14:45.406 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.240537+0000", 2024-09-17T07:14:45.406 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999986 2024-09-17T07:14:45.406 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.406 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.406 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:14:45.406 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.240565+0000", 2024-09-17T07:14:45.406 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7404000000000001e-05 2024-09-17T07:14:45.406 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.407 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.407 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:14:45.407 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.299225+0000", 2024-09-17T07:14:45.407 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.058660122000000002 2024-09-17T07:14:45.407 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.407 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.407 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:14:45.407 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.299235+0000", 2024-09-17T07:14:45.407 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7370000000000001e-06 2024-09-17T07:14:45.407 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.407 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.407 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:14:45.407 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.299678+0000", 2024-09-17T07:14:45.408 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0004437 2024-09-17T07:14:45.408 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.408 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.408 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-17T07:14:45.408 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.299687+0000", 2024-09-17T07:14:45.408 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.0650000000000005e-06 2024-09-17T07:14:45.408 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.408 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.408 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:14:45.408 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.152594+0000", 2024-09-17T07:14:45.409 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.852906991 2024-09-17T07:14:45.409 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.409 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.409 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:14:45.409 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.152698+0000", 2024-09-17T07:14:45.409 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000103175 2024-09-17T07:14:45.409 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.409 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.409 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6,7", 2024-09-17T07:14:45.409 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.152844+0000", 2024-09-17T07:14:45.409 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014609599999999999 2024-09-17T07:14:45.410 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.410 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.410 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:14:45.410 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.154641+0000", 2024-09-17T07:14:45.410 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0017969069999999999 2024-09-17T07:14:45.410 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.411 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.411 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:14:45.411 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.157865+0000", 2024-09-17T07:14:45.411 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0032239489999999998 2024-09-17T07:14:45.411 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.411 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.412 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:14:45.412 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.157900+0000", 2024-09-17T07:14:45.412 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5244999999999999e-05 2024-09-17T07:14:45.412 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.412 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.412 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:14:45.413 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.157945+0000", 2024-09-17T07:14:45.413 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5407000000000001e-05 2024-09-17T07:14:45.413 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.413 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.413 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:14:45.413 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.157979+0000", 2024-09-17T07:14:45.414 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3825000000000002e-05 2024-09-17T07:14:45.414 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:14:45.414 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:14:45.414 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:14:45.414 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.415 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.415 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.14805.0:84132 7.18 7:1cbb02d1:::benchmark_data_smithi100_78556_object84131:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-09-17T07:14:45.415 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:05:30.316148+0000", 2024-09-17T07:14:45.415 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 555.01232404799998, 2024-09-17T07:14:45.415 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.84196913299999998, 2024-09-17T07:14:45.415 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:14:45.416 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:14:45.416 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:14:45.416 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.14805", 2024-09-17T07:14:45.416 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:14:45.416 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 84132 2024-09-17T07:14:45.416 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.417 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:14:45.417 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.417 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:14:45.417 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.316148+0000", 2024-09-17T07:14:45.417 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:14:45.418 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.418 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.418 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:14:45.418 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.316148+0000", 2024-09-17T07:14:45.418 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:14:45.418 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.419 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.419 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:14:45.419 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.316146+0000", 2024-09-17T07:14:45.419 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999981 2024-09-17T07:14:45.419 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.420 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.420 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:14:45.420 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.316171+0000", 2024-09-17T07:14:45.420 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4522e-05 2024-09-17T07:14:45.420 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.420 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.421 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:14:45.421 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.316182+0000", 2024-09-17T07:14:45.421 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0648e-05 2024-09-17T07:14:45.421 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.421 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.422 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:14:45.422 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.316189+0000", 2024-09-17T07:14:45.422 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.729e-06 2024-09-17T07:14:45.422 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.422 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.422 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:14:45.423 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.316243+0000", 2024-09-17T07:14:45.423 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3483000000000003e-05 2024-09-17T07:14:45.423 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.423 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.423 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-17T07:14:45.423 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.316252+0000", 2024-09-17T07:14:45.424 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.4690000000000003e-06 2024-09-17T07:14:45.424 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.424 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.424 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:14:45.424 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.154692+0000", 2024-09-17T07:14:45.424 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.83843984199999999 2024-09-17T07:14:45.425 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.425 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.425 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:14:45.425 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.154852+0000", 2024-09-17T07:14:45.425 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00015942800000000001 2024-09-17T07:14:45.426 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.426 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.426 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6,7", 2024-09-17T07:14:45.426 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.155014+0000", 2024-09-17T07:14:45.426 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016287499999999999 2024-09-17T07:14:45.426 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.427 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.427 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:14:45.427 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.157988+0000", 2024-09-17T07:14:45.427 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0029739699999999998 2024-09-17T07:14:45.427 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.427 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.428 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:14:45.428 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.158021+0000", 2024-09-17T07:14:45.428 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.2101999999999999e-05 2024-09-17T07:14:45.428 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.428 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.429 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:14:45.429 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.158049+0000", 2024-09-17T07:14:45.429 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8255999999999998e-05 2024-09-17T07:14:45.429 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.429 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.430 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:14:45.430 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.158088+0000", 2024-09-17T07:14:45.430 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9319000000000001e-05 2024-09-17T07:14:45.430 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.431 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.432 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:14:45.432 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.158117+0000", 2024-09-17T07:14:45.432 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9268000000000002e-05 2024-09-17T07:14:45.432 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:14:45.432 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:14:45.433 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:14:45.433 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.433 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.433 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.14805.0:84172 7.18 7:1ab41362:::benchmark_data_smithi100_78556_object84171:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-09-17T07:14:45.433 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:05:30.354283+0000", 2024-09-17T07:14:45.433 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 554.97418961799997, 2024-09-17T07:14:45.434 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.81521177700000003, 2024-09-17T07:14:45.434 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:14:45.434 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:14:45.434 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:14:45.434 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.14805", 2024-09-17T07:14:45.434 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:14:45.435 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 84172 2024-09-17T07:14:45.435 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.435 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:14:45.435 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.435 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:14:45.436 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.354283+0000", 2024-09-17T07:14:45.436 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:14:45.436 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.436 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.436 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:14:45.436 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.354283+0000", 2024-09-17T07:14:45.436 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:14:45.437 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.437 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.437 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:14:45.437 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.354282+0000", 2024-09-17T07:14:45.437 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-17T07:14:45.437 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.437 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.438 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:14:45.438 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.354301+0000", 2024-09-17T07:14:45.438 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.9596e-05 2024-09-17T07:14:45.438 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.438 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.438 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:14:45.438 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.354310+0000", 2024-09-17T07:14:45.438 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.9020000000000005e-06 2024-09-17T07:14:45.439 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.439 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.439 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:14:45.439 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.354316+0000", 2024-09-17T07:14:45.439 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0800000000000002e-06 2024-09-17T07:14:45.439 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.439 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.439 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:14:45.440 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.354362+0000", 2024-09-17T07:14:45.440 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5775000000000001e-05 2024-09-17T07:14:45.440 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.440 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.440 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-17T07:14:45.440 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.354371+0000", 2024-09-17T07:14:45.440 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8340000000000005e-06 2024-09-17T07:14:45.441 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.441 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.441 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:14:45.441 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.156387+0000", 2024-09-17T07:14:45.441 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.80201631500000004 2024-09-17T07:14:45.441 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.441 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.441 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:14:45.442 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.156505+0000", 2024-09-17T07:14:45.442 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00011775199999999999 2024-09-17T07:14:45.442 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.442 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:14:45.444 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.444 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6,7", 2024-09-17T07:14:45.445 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.156639+0000", 2024-09-17T07:14:45.445 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00013390099999999999 2024-09-17T07:14:45.445 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.445 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.445 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:14:45.445 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.158165+0000", 2024-09-17T07:14:45.445 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0015256549999999999 2024-09-17T07:14:45.445 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.445 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.446 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:14:45.446 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.158198+0000", 2024-09-17T07:14:45.446 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3921000000000003e-05 2024-09-17T07:14:45.446 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.446 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.446 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:14:45.446 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.169374+0000", 2024-09-17T07:14:45.446 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.011175345 2024-09-17T07:14:45.446 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.446 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.447 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:14:45.447 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.169437+0000", 2024-09-17T07:14:45.447 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3408e-05 2024-09-17T07:14:45.447 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.447 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.447 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:14:45.448 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.169494+0000", 2024-09-17T07:14:45.448 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7234000000000001e-05 2024-09-17T07:14:45.448 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:14:45.448 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:14:45.448 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:14:45.448 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.449 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.449 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.14805.0:84182 7.18 7:1f6d53df:::benchmark_data_smithi100_78556_object84181:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-09-17T07:14:45.449 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:05:30.363797+0000", 2024-09-17T07:14:45.449 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 554.96467508800004, 2024-09-17T07:14:45.449 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.80578439000000002, 2024-09-17T07:14:45.449 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:14:45.450 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:14:45.450 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:14:45.450 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.14805", 2024-09-17T07:14:45.450 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:14:45.450 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 84182 2024-09-17T07:14:45.450 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.450 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:14:45.450 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.450 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:14:45.451 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.363797+0000", 2024-09-17T07:14:45.451 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:14:45.451 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.451 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.451 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:14:45.451 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.363797+0000", 2024-09-17T07:14:45.451 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:14:45.451 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.451 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.451 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:14:45.451 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.363796+0000", 2024-09-17T07:14:45.452 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999986 2024-09-17T07:14:45.452 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.452 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.452 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:14:45.452 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.363819+0000", 2024-09-17T07:14:45.452 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3042000000000001e-05 2024-09-17T07:14:45.452 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.452 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.452 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:14:45.453 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.363831+0000", 2024-09-17T07:14:45.453 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2014999999999999e-05 2024-09-17T07:14:45.453 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.453 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.453 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:14:45.453 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.363839+0000", 2024-09-17T07:14:45.453 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6660000000000008e-06 2024-09-17T07:14:45.453 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.453 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.453 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:14:45.454 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.363896+0000", 2024-09-17T07:14:45.454 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7153e-05 2024-09-17T07:14:45.454 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.454 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.454 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-17T07:14:45.454 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.363907+0000", 2024-09-17T07:14:45.454 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0774000000000001e-05 2024-09-17T07:14:45.454 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.454 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.454 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:14:45.454 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.158220+0000", 2024-09-17T07:14:45.454 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.79431372200000006 2024-09-17T07:14:45.455 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.455 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.455 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:14:45.455 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.158642+0000", 2024-09-17T07:14:45.455 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000421848 2024-09-17T07:14:45.455 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.455 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.455 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6,7", 2024-09-17T07:14:45.455 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.158758+0000", 2024-09-17T07:14:45.455 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00011592200000000001 2024-09-17T07:14:45.455 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.455 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.456 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:14:45.456 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.162687+0000", 2024-09-17T07:14:45.456 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0039287560000000003 2024-09-17T07:14:45.456 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.456 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.456 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:14:45.456 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.164120+0000", 2024-09-17T07:14:45.456 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0014329919999999999 2024-09-17T07:14:45.456 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.456 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.456 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:14:45.456 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.169507+0000", 2024-09-17T07:14:45.457 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.005387101 2024-09-17T07:14:45.457 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.457 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.457 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:14:45.457 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.169550+0000", 2024-09-17T07:14:45.457 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2852999999999998e-05 2024-09-17T07:14:45.457 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.457 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.457 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:14:45.457 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.169582+0000", 2024-09-17T07:14:45.457 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1789999999999999e-05 2024-09-17T07:14:45.457 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:14:45.458 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:14:45.458 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:14:45.458 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.458 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.458 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.14805.0:84203 7.18 7:1a6f3cfc:::benchmark_data_smithi100_78556_object84202:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-09-17T07:14:45.458 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:05:30.391006+0000", 2024-09-17T07:14:45.458 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 554.93746616999999, 2024-09-17T07:14:45.458 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.77878669899999997, 2024-09-17T07:14:45.458 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:14:45.458 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:14:45.458 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:14:45.458 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.14805", 2024-09-17T07:14:45.458 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:14:45.459 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 84203 2024-09-17T07:14:45.459 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.459 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:14:45.459 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.459 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:14:45.459 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.391006+0000", 2024-09-17T07:14:45.459 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:14:45.459 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.459 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.459 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:14:45.459 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.391006+0000", 2024-09-17T07:14:45.459 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:14:45.459 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.460 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.460 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:14:45.460 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.391004+0000", 2024-09-17T07:14:45.460 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999981 2024-09-17T07:14:45.460 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.460 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.460 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:14:45.460 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.391027+0000", 2024-09-17T07:14:45.461 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2889999999999999e-05 2024-09-17T07:14:45.461 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.461 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.461 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:14:45.461 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.391042+0000", 2024-09-17T07:14:45.461 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4609e-05 2024-09-17T07:14:45.462 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.462 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.462 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:14:45.462 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.391049+0000", 2024-09-17T07:14:45.462 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4769999999999997e-06 2024-09-17T07:14:45.462 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.463 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.463 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:14:45.463 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.391107+0000", 2024-09-17T07:14:45.463 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7457999999999998e-05 2024-09-17T07:14:45.463 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.464 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.464 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-17T07:14:45.464 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.391131+0000", 2024-09-17T07:14:45.464 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4032e-05 2024-09-17T07:14:45.464 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.464 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.465 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:14:45.465 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.161034+0000", 2024-09-17T07:14:45.465 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.769902906 2024-09-17T07:14:45.465 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.465 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.466 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:14:45.466 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.161155+0000", 2024-09-17T07:14:45.466 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000121673 2024-09-17T07:14:45.466 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.466 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.466 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6,7", 2024-09-17T07:14:45.467 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.161292+0000", 2024-09-17T07:14:45.467 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000136277 2024-09-17T07:14:45.467 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.467 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.467 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:14:45.467 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.164234+0000", 2024-09-17T07:14:45.468 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.002942414 2024-09-17T07:14:45.468 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.468 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.468 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:14:45.468 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.166530+0000", 2024-09-17T07:14:45.468 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00229583 2024-09-17T07:14:45.469 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.469 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.469 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:14:45.469 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.169712+0000", 2024-09-17T07:14:45.469 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0031819280000000001 2024-09-17T07:14:45.469 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.469 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.470 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:14:45.470 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.169756+0000", 2024-09-17T07:14:45.470 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4131999999999997e-05 2024-09-17T07:14:45.470 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.470 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.470 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:14:45.471 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.169793+0000", 2024-09-17T07:14:45.471 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6931999999999998e-05 2024-09-17T07:14:45.471 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:14:45.471 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:14:45.471 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:14:45.471 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.472 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.472 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.14805.0:84230 7.18 7:1ef434f2:::benchmark_data_smithi100_78556_object84229:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-09-17T07:14:45.472 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:05:30.414950+0000", 2024-09-17T07:14:45.472 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 554.91352264600005, 2024-09-17T07:14:45.472 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.75492697600000003, 2024-09-17T07:14:45.472 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:14:45.473 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:14:45.473 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:14:45.473 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.14805", 2024-09-17T07:14:45.473 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:14:45.473 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 84230 2024-09-17T07:14:45.473 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.473 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:14:45.473 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.474 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:14:45.474 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.414950+0000", 2024-09-17T07:14:45.474 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:14:45.474 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.474 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.474 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:14:45.474 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.414950+0000", 2024-09-17T07:14:45.475 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:14:45.475 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.475 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.475 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:14:45.475 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.414949+0000", 2024-09-17T07:14:45.475 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-17T07:14:45.476 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.476 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.476 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:14:45.476 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.414972+0000", 2024-09-17T07:14:45.476 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3638999999999998e-05 2024-09-17T07:14:45.476 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.476 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.477 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:14:45.477 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.414987+0000", 2024-09-17T07:14:45.477 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5209e-05 2024-09-17T07:14:45.477 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.477 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.478 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:14:45.478 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.414995+0000", 2024-09-17T07:14:45.478 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2389999999999996e-06 2024-09-17T07:14:45.478 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.478 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.478 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:14:45.479 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.415044+0000", 2024-09-17T07:14:45.479 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9471000000000001e-05 2024-09-17T07:14:45.479 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.479 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.479 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-17T07:14:45.480 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.415198+0000", 2024-09-17T07:14:45.480 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00015380400000000001 2024-09-17T07:14:45.480 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.480 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.480 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:14:45.481 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.162713+0000", 2024-09-17T07:14:45.481 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.74751521099999996 2024-09-17T07:14:45.481 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.481 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.481 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:14:45.482 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.162813+0000", 2024-09-17T07:14:45.482 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9946999999999997e-05 2024-09-17T07:14:45.482 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.482 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.482 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6,7", 2024-09-17T07:14:45.482 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.162923+0000", 2024-09-17T07:14:45.482 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00010950699999999999 2024-09-17T07:14:45.482 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.483 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.483 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:14:45.483 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.166506+0000", 2024-09-17T07:14:45.483 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.003582928 2024-09-17T07:14:45.483 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.483 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.483 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:14:45.483 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.169699+0000", 2024-09-17T07:14:45.483 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0031935420000000002 2024-09-17T07:14:45.483 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.483 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.484 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:14:45.484 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.169803+0000", 2024-09-17T07:14:45.484 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000103566 2024-09-17T07:14:45.484 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.484 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.484 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:14:45.484 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.169844+0000", 2024-09-17T07:14:45.484 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.121e-05 2024-09-17T07:14:45.484 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.484 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.485 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:14:45.485 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.169877+0000", 2024-09-17T07:14:45.485 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.2808999999999997e-05 2024-09-17T07:14:45.485 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:14:45.485 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:14:45.485 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:14:45.485 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.485 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.485 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.14805.0:84239 7.18 7:1d5db2f1:::benchmark_data_smithi100_78556_object84238:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-09-17T07:14:45.485 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:05:30.428155+0000", 2024-09-17T07:14:45.485 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 554.90031759600004, 2024-09-17T07:14:45.486 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.74195996500000005, 2024-09-17T07:14:45.486 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:14:45.486 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:14:45.486 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:14:45.486 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.14805", 2024-09-17T07:14:45.486 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:14:45.486 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 84239 2024-09-17T07:14:45.486 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.486 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:14:45.486 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.486 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:14:45.486 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.428155+0000", 2024-09-17T07:14:45.487 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:14:45.487 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.487 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.487 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:14:45.487 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.428155+0000", 2024-09-17T07:14:45.487 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:14:45.487 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.487 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.487 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:14:45.487 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.428153+0000", 2024-09-17T07:14:45.487 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999981 2024-09-17T07:14:45.487 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.487 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.488 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:14:45.488 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.428175+0000", 2024-09-17T07:14:45.488 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2569999999999999e-05 2024-09-17T07:14:45.488 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.488 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.488 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:14:45.488 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.428186+0000", 2024-09-17T07:14:45.488 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0429e-05 2024-09-17T07:14:45.488 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.488 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.488 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:14:45.488 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.428193+0000", 2024-09-17T07:14:45.489 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8940000000000004e-06 2024-09-17T07:14:45.489 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.489 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.489 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:14:45.489 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.428243+0000", 2024-09-17T07:14:45.489 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0494000000000001e-05 2024-09-17T07:14:45.489 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.489 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.489 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-17T07:14:45.490 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.428253+0000", 2024-09-17T07:14:45.490 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7860000000000008e-06 2024-09-17T07:14:45.490 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.490 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.490 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:14:45.490 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.164256+0000", 2024-09-17T07:14:45.490 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.73600358600000004 2024-09-17T07:14:45.490 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.490 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.491 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:14:45.491 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.164377+0000", 2024-09-17T07:14:45.491 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012071 2024-09-17T07:14:45.491 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.491 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.491 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6,7", 2024-09-17T07:14:45.492 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.164531+0000", 2024-09-17T07:14:45.492 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00015407600000000001 2024-09-17T07:14:45.492 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.492 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.492 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:14:45.492 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.169651+0000", 2024-09-17T07:14:45.493 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0051196640000000003 2024-09-17T07:14:45.493 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.493 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.493 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:14:45.493 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.169887+0000", 2024-09-17T07:14:45.493 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00023590500000000001 2024-09-17T07:14:45.493 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.493 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.494 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:14:45.494 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.170038+0000", 2024-09-17T07:14:45.494 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00015090500000000001 2024-09-17T07:14:45.494 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.494 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.494 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:14:45.494 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.170081+0000", 2024-09-17T07:14:45.494 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3464000000000003e-05 2024-09-17T07:14:45.494 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.494 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.494 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:14:45.495 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.170115+0000", 2024-09-17T07:14:45.495 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3553000000000002e-05 2024-09-17T07:14:45.495 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:14:45.495 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:14:45.495 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:14:45.495 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.495 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.495 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.14805.0:84247 7.18 7:1f13e6b2:::benchmark_data_smithi100_78556_object84246:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-09-17T07:14:45.496 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:05:30.435367+0000", 2024-09-17T07:14:45.496 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 554.89310510799999, 2024-09-17T07:14:45.496 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.74378478199999998, 2024-09-17T07:14:45.496 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:14:45.496 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:14:45.496 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:14:45.497 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.14805", 2024-09-17T07:14:45.497 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:14:45.497 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 84247 2024-09-17T07:14:45.497 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.497 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:14:45.497 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.498 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:14:45.498 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.435367+0000", 2024-09-17T07:14:45.498 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:14:45.499 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.499 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.499 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:14:45.499 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.435367+0000", 2024-09-17T07:14:45.499 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:14:45.499 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.500 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.500 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:14:45.500 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.435365+0000", 2024-09-17T07:14:45.500 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999981 2024-09-17T07:14:45.500 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.500 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.501 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:14:45.501 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.435404+0000", 2024-09-17T07:14:45.501 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8220000000000003e-05 2024-09-17T07:14:45.501 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.501 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.501 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:14:45.501 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.435416+0000", 2024-09-17T07:14:45.501 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.205e-05 2024-09-17T07:14:45.502 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.502 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.502 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:14:45.502 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.435423+0000", 2024-09-17T07:14:45.502 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1760000000000004e-06 2024-09-17T07:14:45.502 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.502 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.502 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:14:45.502 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.435482+0000", 2024-09-17T07:14:45.503 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8857000000000002e-05 2024-09-17T07:14:45.503 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.505 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.505 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-17T07:14:45.505 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.435492+0000", 2024-09-17T07:14:45.505 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7610000000000002e-06 2024-09-17T07:14:45.505 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.506 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.506 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:14:45.506 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.166549+0000", 2024-09-17T07:14:45.506 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.73105739300000006 2024-09-17T07:14:45.506 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.506 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.507 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:14:45.507 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.166642+0000", 2024-09-17T07:14:45.507 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.2769000000000006e-05 2024-09-17T07:14:45.507 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.507 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.507 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6,7", 2024-09-17T07:14:45.507 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.167044+0000", 2024-09-17T07:14:45.508 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00040220799999999998 2024-09-17T07:14:45.508 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.508 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.508 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:14:45.508 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.169894+0000", 2024-09-17T07:14:45.508 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.002849697 2024-09-17T07:14:45.508 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.508 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.508 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:14:45.508 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.170001+0000", 2024-09-17T07:14:45.509 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000107262 2024-09-17T07:14:45.509 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.509 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.509 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:14:45.509 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.179069+0000", 2024-09-17T07:14:45.509 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0090679720000000005 2024-09-17T07:14:45.510 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.510 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.510 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:14:45.510 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.179110+0000", 2024-09-17T07:14:45.510 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1483000000000002e-05 2024-09-17T07:14:45.510 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.510 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.510 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:14:45.510 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.179152+0000", 2024-09-17T07:14:45.510 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1672999999999999e-05 2024-09-17T07:14:45.510 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:14:45.510 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:14:45.511 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:14:45.511 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.511 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.511 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.14805.0:84302 7.18 7:1819d156:::benchmark_data_smithi100_78556_object84301:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-09-17T07:14:45.511 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:05:30.494328+0000", 2024-09-17T07:14:45.511 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 554.83414478300006, 2024-09-17T07:14:45.511 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.68497140400000001, 2024-09-17T07:14:45.511 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:14:45.511 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:14:45.511 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:14:45.511 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.14805", 2024-09-17T07:14:45.511 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:14:45.511 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 84302 2024-09-17T07:14:45.512 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.512 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:14:45.512 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.512 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:14:45.512 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.494328+0000", 2024-09-17T07:14:45.512 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:14:45.512 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.512 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.512 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:14:45.512 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.494328+0000", 2024-09-17T07:14:45.512 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:14:45.513 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.513 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.513 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:14:45.513 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.494327+0000", 2024-09-17T07:14:45.513 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-17T07:14:45.513 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.513 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.513 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:14:45.513 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.494346+0000", 2024-09-17T07:14:45.513 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.9245e-05 2024-09-17T07:14:45.513 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.513 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.513 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:14:45.514 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.494355+0000", 2024-09-17T07:14:45.514 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3249999999999997e-06 2024-09-17T07:14:45.514 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.514 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.514 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:14:45.514 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.494360+0000", 2024-09-17T07:14:45.514 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9320000000000004e-06 2024-09-17T07:14:45.514 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.514 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.515 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:14:45.515 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.494400+0000", 2024-09-17T07:14:45.515 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9918e-05 2024-09-17T07:14:45.515 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.515 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.515 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-17T07:14:45.515 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.494408+0000", 2024-09-17T07:14:45.515 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6140000000000003e-06 2024-09-17T07:14:45.515 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.516 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.516 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:14:45.516 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.170148+0000", 2024-09-17T07:14:45.516 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.67574066899999996 2024-09-17T07:14:45.516 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.516 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.516 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:14:45.516 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.170241+0000", 2024-09-17T07:14:45.516 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.2353000000000002e-05 2024-09-17T07:14:45.517 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.517 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.517 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6,7", 2024-09-17T07:14:45.517 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.170352+0000", 2024-09-17T07:14:45.517 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000111702 2024-09-17T07:14:45.517 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.517 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.517 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:14:45.517 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.179010+0000", 2024-09-17T07:14:45.517 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0086577300000000006 2024-09-17T07:14:45.517 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.517 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.517 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:14:45.518 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.179019+0000", 2024-09-17T07:14:45.518 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.7309999999999999e-06 2024-09-17T07:14:45.518 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.518 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.518 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:14:45.518 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.179228+0000", 2024-09-17T07:14:45.518 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00020955299999999999 2024-09-17T07:14:45.518 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.518 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.518 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:14:45.518 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.179268+0000", 2024-09-17T07:14:45.518 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9848999999999997e-05 2024-09-17T07:14:45.519 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.519 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.519 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:14:45.519 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.179299+0000", 2024-09-17T07:14:45.519 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0614000000000002e-05 2024-09-17T07:14:45.519 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:14:45.519 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:14:45.519 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:14:45.519 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.519 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.519 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.14805.0:84355 7.18 7:1becf356:::benchmark_data_smithi100_78556_object84354:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-09-17T07:14:45.519 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:05:30.572233+0000", 2024-09-17T07:14:45.520 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 554.75623936399995, 2024-09-17T07:14:45.520 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.60719921300000002, 2024-09-17T07:14:45.520 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:14:45.520 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:14:45.520 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:14:45.520 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.14805", 2024-09-17T07:14:45.520 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/516149216", 2024-09-17T07:14:45.520 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 84355 2024-09-17T07:14:45.520 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.520 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:14:45.520 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.521 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:14:45.521 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.572233+0000", 2024-09-17T07:14:45.521 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:14:45.521 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.521 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.521 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:14:45.521 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.572233+0000", 2024-09-17T07:14:45.521 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:14:45.521 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.521 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.521 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:14:45.521 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.572231+0000", 2024-09-17T07:14:45.522 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999981 2024-09-17T07:14:45.522 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.522 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.522 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:14:45.522 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.572297+0000", 2024-09-17T07:14:45.522 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5572000000000002e-05 2024-09-17T07:14:45.522 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.522 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.522 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:14:45.522 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.572474+0000", 2024-09-17T07:14:45.522 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000177102 2024-09-17T07:14:45.522 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.522 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.523 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:14:45.523 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.572486+0000", 2024-09-17T07:14:45.523 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1865000000000001e-05 2024-09-17T07:14:45.523 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.523 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.523 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:14:45.523 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.572539+0000", 2024-09-17T07:14:45.523 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3007000000000003e-05 2024-09-17T07:14:45.523 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.523 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.523 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-17T07:14:45.523 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:30.572550+0000", 2024-09-17T07:14:45.523 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1779e-05 2024-09-17T07:14:45.524 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.524 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.524 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:14:45.524 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.171857+0000", 2024-09-17T07:14:45.524 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.59930705299999998 2024-09-17T07:14:45.524 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.524 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.524 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:14:45.525 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.171963+0000", 2024-09-17T07:14:45.525 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00010545200000000001 2024-09-17T07:14:45.525 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.525 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.525 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6,7", 2024-09-17T07:14:45.525 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.172088+0000", 2024-09-17T07:14:45.525 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000125017 2024-09-17T07:14:45.525 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.525 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.525 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:14:45.525 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.179025+0000", 2024-09-17T07:14:45.525 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0069367259999999998 2024-09-17T07:14:45.525 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.526 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.526 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:14:45.526 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.179193+0000", 2024-09-17T07:14:45.526 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016800499999999999 2024-09-17T07:14:45.526 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.526 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.526 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:14:45.526 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.179366+0000", 2024-09-17T07:14:45.526 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00017284399999999999 2024-09-17T07:14:45.527 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.527 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.527 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:14:45.527 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.179402+0000", 2024-09-17T07:14:45.527 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6442000000000001e-05 2024-09-17T07:14:45.527 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.527 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.527 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:14:45.527 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:05:31.179432+0000", 2024-09-17T07:14:45.527 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0199999999999999e-05 2024-09-17T07:14:45.527 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:14:45.528 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:14:45.528 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:14:45.528 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.528 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.528 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.24929.0:108604 8.c 8:3e7c26a2:::benchmark_data_smithi100_102803_object108603:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-09-17T07:14:45.528 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:09:23.413427+0000", 2024-09-17T07:14:45.528 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 321.91504566899999, 2024-09-17T07:14:45.528 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0862750060000002, 2024-09-17T07:14:45.529 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:14:45.529 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:14:45.529 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:14:45.529 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.24929", 2024-09-17T07:14:45.529 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/564515582", 2024-09-17T07:14:45.529 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 108604 2024-09-17T07:14:45.529 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.529 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:14:45.529 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.529 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:14:45.529 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.413427+0000", 2024-09-17T07:14:45.529 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:14:45.529 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.530 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.530 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:14:45.530 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.413427+0000", 2024-09-17T07:14:45.530 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:14:45.530 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.530 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.530 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:14:45.530 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.413425+0000", 2024-09-17T07:14:45.530 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999986 2024-09-17T07:14:45.530 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.531 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.531 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:14:45.531 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.413454+0000", 2024-09-17T07:14:45.531 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9417999999999999e-05 2024-09-17T07:14:45.531 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.531 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.531 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:14:45.531 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.413467+0000", 2024-09-17T07:14:45.532 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2363e-05 2024-09-17T07:14:45.532 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.532 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.532 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:14:45.532 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.413475+0000", 2024-09-17T07:14:45.532 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1240000000000005e-06 2024-09-17T07:14:45.532 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.532 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.533 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:14:45.533 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.413727+0000", 2024-09-17T07:14:45.533 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00025191299999999999 2024-09-17T07:14:45.533 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.533 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.533 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-17T07:14:45.533 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.413735+0000", 2024-09-17T07:14:45.533 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6030000000000005e-06 2024-09-17T07:14:45.534 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.534 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.534 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:14:45.534 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:24.385461+0000", 2024-09-17T07:14:45.534 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.97172562100000004 2024-09-17T07:14:45.534 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.534 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.534 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-17T07:14:45.535 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:24.385472+0000", 2024-09-17T07:14:45.535 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.118e-05 2024-09-17T07:14:45.535 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.535 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.535 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:14:45.535 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:25.471023+0000", 2024-09-17T07:14:45.535 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.085550276 2024-09-17T07:14:45.535 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.536 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.536 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:14:45.536 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:25.471146+0000", 2024-09-17T07:14:45.536 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000123437 2024-09-17T07:14:45.536 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.536 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.536 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 3,5,7", 2024-09-17T07:14:45.536 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:25.471678+0000", 2024-09-17T07:14:45.537 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00053228200000000002 2024-09-17T07:14:45.537 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.537 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.537 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:14:45.537 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:25.483053+0000", 2024-09-17T07:14:45.537 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.011375031000000001 2024-09-17T07:14:45.538 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.538 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.538 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:14:45.538 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:25.483228+0000", 2024-09-17T07:14:45.538 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00017469600000000001 2024-09-17T07:14:45.538 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.538 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.538 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:14:45.539 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:25.483332+0000", 2024-09-17T07:14:45.539 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000103733 2024-09-17T07:14:45.539 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.539 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.539 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:14:45.539 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:25.499421+0000", 2024-09-17T07:14:45.539 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.016089678 2024-09-17T07:14:45.539 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.539 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.539 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:14:45.539 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:25.499581+0000", 2024-09-17T07:14:45.540 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016000899999999999 2024-09-17T07:14:45.540 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.540 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.540 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:14:45.540 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:25.499702+0000", 2024-09-17T07:14:45.540 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000120277 2024-09-17T07:14:45.540 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:14:45.540 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:14:45.540 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:14:45.540 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.540 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.540 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.24380.0:542 4.16 4:6fe6cc47:notif::queues_list_object:head [omap-get-keys] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e184)", 2024-09-17T07:14:45.541 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:09:23.652772+0000", 2024-09-17T07:14:45.541 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 321.67569984199997, 2024-09-17T07:14:45.541 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.75680366099999996, 2024-09-17T07:14:45.541 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:14:45.541 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-17T07:14:45.541 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:14:45.541 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.24380", 2024-09-17T07:14:45.541 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/1334829920", 2024-09-17T07:14:45.541 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 542 2024-09-17T07:14:45.541 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.541 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:14:45.541 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.541 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:14:45.542 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.652772+0000", 2024-09-17T07:14:45.542 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:14:45.542 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.542 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.542 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:14:45.542 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.652772+0000", 2024-09-17T07:14:45.542 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:14:45.542 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.542 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.542 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:14:45.542 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.652770+0000", 2024-09-17T07:14:45.542 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999976 2024-09-17T07:14:45.542 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.543 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.543 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:14:45.543 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.652783+0000", 2024-09-17T07:14:45.543 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2670999999999999e-05 2024-09-17T07:14:45.543 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.543 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.543 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:14:45.543 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.652802+0000", 2024-09-17T07:14:45.543 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.8947e-05 2024-09-17T07:14:45.543 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.543 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.543 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:14:45.543 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.652811+0000", 2024-09-17T07:14:45.543 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5230000000000001e-06 2024-09-17T07:14:45.544 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.544 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.544 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:14:45.544 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.652907+0000", 2024-09-17T07:14:45.544 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5879999999999997e-05 2024-09-17T07:14:45.544 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.544 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.544 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-17T07:14:45.544 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.652920+0000", 2024-09-17T07:14:45.544 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2986000000000001e-05 2024-09-17T07:14:45.544 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.544 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.544 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:14:45.545 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:24.409477+0000", 2024-09-17T07:14:45.545 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.75655701500000005 2024-09-17T07:14:45.545 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:14:45.545 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:14:45.545 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:14:45.545 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:24.409576+0000", 2024-09-17T07:14:45.545 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.8919999999999995e-05 2024-09-17T07:14:45.545 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:14:45.545 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:14:45.545 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:14:45.545 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:14:45.545 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:14:45.545 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-17T07:14:45.546 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:45 smithi100 ceph-mon[29092]: 4.5 scrub ok 2024-09-17T07:14:45.546 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:45 smithi100 ceph-mon[29092]: 2.1c scrub starts 2024-09-17T07:14:45.547 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:45 smithi100 ceph-mon[29092]: 2.1c scrub ok 2024-09-17T07:14:45.547 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:45 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:14:45.547 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:45 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:14:45.547 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:45 smithi100 ceph-mon[29092]: pgmap v560: 148 pgs: 148 active+clean; 3.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.64k op/s 2024-09-17T07:14:45.547 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:45 smithi100 ceph-mon[39462]: 4.5 scrub ok 2024-09-17T07:14:45.547 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:45 smithi100 ceph-mon[39462]: 2.1c scrub starts 2024-09-17T07:14:45.547 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:45 smithi100 ceph-mon[39462]: 2.1c scrub ok 2024-09-17T07:14:45.547 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:45 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:14:45.547 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:45 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:14:45.547 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:45 smithi100 ceph-mon[39462]: pgmap v560: 148 pgs: 148 active+clean; 3.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.64k op/s 2024-09-17T07:14:45.561 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:14:45.584 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:45.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:45.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:14:45.686 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:14:45.723 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:45.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:45.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:14:45.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:45 smithi135 ceph-mon[33238]: 4.5 scrub ok 2024-09-17T07:14:45.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:45 smithi135 ceph-mon[33238]: 2.1c scrub starts 2024-09-17T07:14:45.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:45 smithi135 ceph-mon[33238]: 2.1c scrub ok 2024-09-17T07:14:45.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:45 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:14:45.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:45 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:14:45.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:45 smithi135 ceph-mon[33238]: pgmap v560: 148 pgs: 148 active+clean; 3.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.64k op/s 2024-09-17T07:14:45.824 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:14:45.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:45.864 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:45.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:14:45.965 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:14:46.001 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:46.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:46.002 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:14:46.103 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:14:46.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:46.134 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:46.134 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:14:46.235 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:14:46.279 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:46.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:46.280 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:14:46.376 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:14:45 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:14:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:14:46.381 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:14:46.421 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:46.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:46.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:14:46.523 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:14:46.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:46 smithi135 ceph-mon[33238]: 9.2 scrub starts 2024-09-17T07:14:46.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:46 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:14:46.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:46 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:14:46.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:46 smithi135 ceph-mon[33238]: osdmap e277: 8 total, 8 up, 7 in 2024-09-17T07:14:46.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:46 smithi135 ceph-mon[33238]: 5.11 scrub starts 2024-09-17T07:14:46.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:46 smithi135 ceph-mon[33238]: 5.a scrub starts 2024-09-17T07:14:46.545 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:46 smithi135 ceph-mon[33238]: 5.a scrub ok 2024-09-17T07:14:46.545 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:46 smithi135 ceph-mon[33238]: 9.f scrub starts 2024-09-17T07:14:46.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:46.553 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:46.553 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:14:46.654 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:14:46.694 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:46.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:46.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:14:46.731 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:46 smithi100 ceph-mon[29092]: 9.2 scrub starts 2024-09-17T07:14:46.731 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:46 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:14:46.731 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:46 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:14:46.731 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:46 smithi100 ceph-mon[29092]: osdmap e277: 8 total, 8 up, 7 in 2024-09-17T07:14:46.731 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:46 smithi100 ceph-mon[29092]: 5.11 scrub starts 2024-09-17T07:14:46.731 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:46 smithi100 ceph-mon[29092]: 5.a scrub starts 2024-09-17T07:14:46.731 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:46 smithi100 ceph-mon[29092]: 5.a scrub ok 2024-09-17T07:14:46.731 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:46 smithi100 ceph-mon[29092]: 9.f scrub starts 2024-09-17T07:14:46.732 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:46 smithi100 ceph-mon[39462]: 9.2 scrub starts 2024-09-17T07:14:46.732 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:46 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:14:46.732 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:46 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:14:46.732 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:46 smithi100 ceph-mon[39462]: osdmap e277: 8 total, 8 up, 7 in 2024-09-17T07:14:46.732 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:46 smithi100 ceph-mon[39462]: 5.11 scrub starts 2024-09-17T07:14:46.732 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:46 smithi100 ceph-mon[39462]: 5.a scrub starts 2024-09-17T07:14:46.732 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:46 smithi100 ceph-mon[39462]: 5.a scrub ok 2024-09-17T07:14:46.732 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:46 smithi100 ceph-mon[39462]: 9.f scrub starts 2024-09-17T07:14:46.765 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-17T07:14:46.796 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:14:46.834 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:46.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:46.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:14:46.936 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:14:46.970 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:46.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:46.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:14:47.071 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:14:47.110 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:47.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:47.111 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:14:47.212 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:14:47.249 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:47.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:47.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:14:47.351 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:14:47.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:47.389 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:47.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:14:47.491 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:14:47.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:47.526 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:47.526 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:14:47.627 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:14:47.657 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:47.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:47.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:14:47.759 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:14:47.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:47.788 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:47.788 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:14:47.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:47 smithi135 ceph-mon[33238]: 5.11 scrub ok 2024-09-17T07:14:47.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:47 smithi135 ceph-mon[33238]: 4.f scrub starts 2024-09-17T07:14:47.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:47 smithi135 ceph-mon[33238]: 4.f scrub ok 2024-09-17T07:14:47.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:47 smithi135 ceph-mon[33238]: 9.2 scrub ok 2024-09-17T07:14:47.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:47 smithi135 ceph-mon[33238]: 5.d scrub starts 2024-09-17T07:14:47.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:47 smithi135 ceph-mon[33238]: 5.d scrub ok 2024-09-17T07:14:47.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:47 smithi135 ceph-mon[33238]: 5.1f scrub starts 2024-09-17T07:14:47.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:47 smithi135 ceph-mon[33238]: pgmap v562: 148 pgs: 148 active+clean; 3.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1k op/s 2024-09-17T07:14:47.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:47 smithi100 ceph-mon[29092]: 5.11 scrub ok 2024-09-17T07:14:47.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:47 smithi100 ceph-mon[29092]: 4.f scrub starts 2024-09-17T07:14:47.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:47 smithi100 ceph-mon[29092]: 4.f scrub ok 2024-09-17T07:14:47.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:47 smithi100 ceph-mon[29092]: 9.2 scrub ok 2024-09-17T07:14:47.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:47 smithi100 ceph-mon[29092]: 5.d scrub starts 2024-09-17T07:14:47.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:47 smithi100 ceph-mon[29092]: 5.d scrub ok 2024-09-17T07:14:47.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:47 smithi100 ceph-mon[29092]: 5.1f scrub starts 2024-09-17T07:14:47.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:47 smithi100 ceph-mon[29092]: pgmap v562: 148 pgs: 148 active+clean; 3.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1k op/s 2024-09-17T07:14:47.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:47 smithi100 ceph-mon[39462]: 5.11 scrub ok 2024-09-17T07:14:47.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:47 smithi100 ceph-mon[39462]: 4.f scrub starts 2024-09-17T07:14:47.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:47 smithi100 ceph-mon[39462]: 4.f scrub ok 2024-09-17T07:14:47.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:47 smithi100 ceph-mon[39462]: 9.2 scrub ok 2024-09-17T07:14:47.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:47 smithi100 ceph-mon[39462]: 5.d scrub starts 2024-09-17T07:14:47.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:47 smithi100 ceph-mon[39462]: 5.d scrub ok 2024-09-17T07:14:47.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:47 smithi100 ceph-mon[39462]: 5.1f scrub starts 2024-09-17T07:14:47.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:47 smithi100 ceph-mon[39462]: pgmap v562: 148 pgs: 148 active+clean; 3.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1k op/s 2024-09-17T07:14:47.889 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:14:47.916 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:47.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:47.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:14:48.017 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:14:48.048 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:48.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:48.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:14:48.149 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:14:48.177 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:48.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:48.178 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:14:48.279 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:14:48.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:48.323 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:48.323 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:14:48.424 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:14:48.425 INFO:teuthology.orchestra.run.smithi100.stderr:nodeep-scrub is unset 2024-09-17T07:14:48.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:48.458 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:48.458 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:14:48.559 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:14:48.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:48.592 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:48.592 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:14:48.693 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:14:48.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:48.723 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:48.723 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:14:48.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:48 smithi135 ceph-mon[33238]: 5.1f scrub ok 2024-09-17T07:14:48.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:48 smithi135 ceph-mon[33238]: 9.d scrub starts 2024-09-17T07:14:48.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:48 smithi135 ceph-mon[33238]: 9.f scrub ok 2024-09-17T07:14:48.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:48 smithi135 ceph-mon[33238]: 4.19 scrub starts 2024-09-17T07:14:48.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:48 smithi135 ceph-mon[33238]: 4.19 scrub ok 2024-09-17T07:14:48.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:48 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1409199331' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:14:48.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:48 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:14:48.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:48 smithi135 ceph-mon[33238]: 6.7 scrub starts 2024-09-17T07:14:48.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:48 smithi135 ceph-mon[33238]: 6.7 scrub ok 2024-09-17T07:14:48.824 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:14:48.855 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:48.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:48.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:14:48.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:48 smithi100 ceph-mon[29092]: 5.1f scrub ok 2024-09-17T07:14:48.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:48 smithi100 ceph-mon[29092]: 9.d scrub starts 2024-09-17T07:14:48.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:48 smithi100 ceph-mon[29092]: 9.f scrub ok 2024-09-17T07:14:48.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:48 smithi100 ceph-mon[29092]: 4.19 scrub starts 2024-09-17T07:14:48.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:48 smithi100 ceph-mon[29092]: 4.19 scrub ok 2024-09-17T07:14:48.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:48 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1409199331' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:14:48.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:48 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:14:48.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:48 smithi100 ceph-mon[29092]: 6.7 scrub starts 2024-09-17T07:14:48.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:48 smithi100 ceph-mon[29092]: 6.7 scrub ok 2024-09-17T07:14:48.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:48 smithi100 ceph-mon[39462]: 5.1f scrub ok 2024-09-17T07:14:48.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:48 smithi100 ceph-mon[39462]: 9.d scrub starts 2024-09-17T07:14:48.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:48 smithi100 ceph-mon[39462]: 9.f scrub ok 2024-09-17T07:14:48.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:48 smithi100 ceph-mon[39462]: 4.19 scrub starts 2024-09-17T07:14:48.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:48 smithi100 ceph-mon[39462]: 4.19 scrub ok 2024-09-17T07:14:48.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:48 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1409199331' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:14:48.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:48 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:14:48.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:48 smithi100 ceph-mon[39462]: 6.7 scrub starts 2024-09-17T07:14:48.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:48 smithi100 ceph-mon[39462]: 6.7 scrub ok 2024-09-17T07:14:48.957 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:14:48.987 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:48.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:48.988 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:14:49.089 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:14:49.128 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:49.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:49.129 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:14:49.194 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:14:49.230 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:14:49.259 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:49.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:49.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:14:49.293 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:14:48 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:14:49.361 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:14:49.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:49.394 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:49.394 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:14:49.495 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:14:49.529 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:49.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:49.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:14:49.631 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:14:49.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:49.663 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:49.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:14:49.720 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:49 smithi100 ceph-mon[29092]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T07:14:49.720 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:49 smithi100 ceph-mon[29092]: Cluster is now healthy 2024-09-17T07:14:49.720 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:49 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1409199331' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T07:14:49.720 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:49 smithi100 ceph-mon[29092]: osdmap e278: 8 total, 8 up, 7 in 2024-09-17T07:14:49.720 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:49 smithi100 ceph-mon[29092]: 9.d scrub ok 2024-09-17T07:14:49.721 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:49 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:14:49.721 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:49 smithi100 ceph-mon[29092]: 6.a scrub starts 2024-09-17T07:14:49.721 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:49 smithi100 ceph-mon[29092]: 6.a scrub ok 2024-09-17T07:14:49.721 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:49 smithi100 ceph-mon[29092]: pgmap v564: 148 pgs: 148 active+clean; 3.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.38k op/s 2024-09-17T07:14:49.721 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:49 smithi100 ceph-mon[39462]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T07:14:49.721 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:49 smithi100 ceph-mon[39462]: Cluster is now healthy 2024-09-17T07:14:49.721 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:49 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1409199331' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T07:14:49.721 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:49 smithi100 ceph-mon[39462]: osdmap e278: 8 total, 8 up, 7 in 2024-09-17T07:14:49.721 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:49 smithi100 ceph-mon[39462]: 9.d scrub ok 2024-09-17T07:14:49.721 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:49 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:14:49.722 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:49 smithi100 ceph-mon[39462]: 6.a scrub starts 2024-09-17T07:14:49.722 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:49 smithi100 ceph-mon[39462]: 6.a scrub ok 2024-09-17T07:14:49.722 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:49 smithi100 ceph-mon[39462]: pgmap v564: 148 pgs: 148 active+clean; 3.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.38k op/s 2024-09-17T07:14:49.765 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:14:49.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:49.789 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:49.789 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:14:49.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:49 smithi135 ceph-mon[33238]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T07:14:49.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:49 smithi135 ceph-mon[33238]: Cluster is now healthy 2024-09-17T07:14:49.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:49 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1409199331' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T07:14:49.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:49 smithi135 ceph-mon[33238]: osdmap e278: 8 total, 8 up, 7 in 2024-09-17T07:14:49.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:49 smithi135 ceph-mon[33238]: 9.d scrub ok 2024-09-17T07:14:49.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:49 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:14:49.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:49 smithi135 ceph-mon[33238]: 6.a scrub starts 2024-09-17T07:14:49.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:49 smithi135 ceph-mon[33238]: 6.a scrub ok 2024-09-17T07:14:49.795 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:49 smithi135 ceph-mon[33238]: pgmap v564: 148 pgs: 148 active+clean; 3.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.38k op/s 2024-09-17T07:14:49.796 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:14:49.890 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:14:49.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:49.913 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:49.913 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:14:50.014 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:14:50.050 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:50.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:50.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:14:50.152 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:14:50.190 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:50.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:50.191 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:14:50.292 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:14:50.325 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:50.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:50.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:14:50.426 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:14:50.465 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:50.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:50.466 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:14:50.567 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:14:50.599 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:50.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:50.600 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:14:50.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:50 smithi100 ceph-mon[39462]: 5.1e scrub starts 2024-09-17T07:14:50.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:50 smithi100 ceph-mon[39462]: 5.1e scrub ok 2024-09-17T07:14:50.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:50 smithi100 ceph-mon[39462]: 4.8 deep-scrub starts 2024-09-17T07:14:50.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:50 smithi100 ceph-mon[39462]: 4.8 deep-scrub ok 2024-09-17T07:14:50.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:50 smithi100 ceph-mon[29092]: 5.1e scrub starts 2024-09-17T07:14:50.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:50 smithi100 ceph-mon[29092]: 5.1e scrub ok 2024-09-17T07:14:50.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:50 smithi100 ceph-mon[29092]: 4.8 deep-scrub starts 2024-09-17T07:14:50.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:50 smithi100 ceph-mon[29092]: 4.8 deep-scrub ok 2024-09-17T07:14:50.701 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:14:50.738 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:50.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:50.740 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:14:50.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:50 smithi135 ceph-mon[33238]: 5.1e scrub starts 2024-09-17T07:14:50.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:50 smithi135 ceph-mon[33238]: 5.1e scrub ok 2024-09-17T07:14:50.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:50 smithi135 ceph-mon[33238]: 4.8 deep-scrub starts 2024-09-17T07:14:50.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:50 smithi135 ceph-mon[33238]: 4.8 deep-scrub ok 2024-09-17T07:14:50.840 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:14:50.882 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:50.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:50.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:14:50.984 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:14:51.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:51.018 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:51.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:14:51.120 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:14:51.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:51.155 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:51.156 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:14:51.257 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:14:51.290 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:51.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:51.292 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:14:51.393 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:14:51.433 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:51.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:51.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:14:51.439 INFO:teuthology.orchestra.run.smithi100.stderr:reweighted osd.2 to 1 (10000) 2024-09-17T07:14:51.534 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:14:51.572 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:51.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:51.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:14:51.674 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:14:51.711 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:51.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:51.712 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:14:51.758 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-17T07:14:51.785 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:51 smithi100 ceph-mon[29092]: 2.4 scrub starts 2024-09-17T07:14:51.786 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:51 smithi100 ceph-mon[29092]: 2.4 scrub ok 2024-09-17T07:14:51.786 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:51 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:14:51.786 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:51 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:14:51.786 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:51 smithi100 ceph-mon[29092]: 6.e deep-scrub starts 2024-09-17T07:14:51.786 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:51 smithi100 ceph-mon[29092]: 6.e deep-scrub ok 2024-09-17T07:14:51.786 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:51 smithi100 ceph-mon[29092]: pgmap v565: 148 pgs: 148 active+clean; 3.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.57k op/s 2024-09-17T07:14:51.786 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:51 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2806585259' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-17T07:14:51.786 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:51 smithi100 ceph-mon[39462]: 2.4 scrub starts 2024-09-17T07:14:51.786 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:51 smithi100 ceph-mon[39462]: 2.4 scrub ok 2024-09-17T07:14:51.786 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:51 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:14:51.786 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:51 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:14:51.787 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:51 smithi100 ceph-mon[39462]: 6.e deep-scrub starts 2024-09-17T07:14:51.787 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:51 smithi100 ceph-mon[39462]: 6.e deep-scrub ok 2024-09-17T07:14:51.787 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:51 smithi100 ceph-mon[39462]: pgmap v565: 148 pgs: 148 active+clean; 3.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.57k op/s 2024-09-17T07:14:51.787 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:51 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2806585259' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-17T07:14:51.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:51 smithi135 ceph-mon[33238]: 2.4 scrub starts 2024-09-17T07:14:51.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:51 smithi135 ceph-mon[33238]: 2.4 scrub ok 2024-09-17T07:14:51.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:51 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:14:51.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:51 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:14:51.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:51 smithi135 ceph-mon[33238]: 6.e deep-scrub starts 2024-09-17T07:14:51.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:51 smithi135 ceph-mon[33238]: 6.e deep-scrub ok 2024-09-17T07:14:51.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:51 smithi135 ceph-mon[33238]: pgmap v565: 148 pgs: 148 active+clean; 3.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.57k op/s 2024-09-17T07:14:51.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:51 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2806585259' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-17T07:14:51.812 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:14:51.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:51.849 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:51.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:14:51.950 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:14:51.982 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:51.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:51.983 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:14:52.084 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:14:52.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:52.121 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:52.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:14:52.222 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:14:52.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:52.264 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:52.264 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:14:52.365 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:14:52.393 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:52.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:52.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:14:52.495 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:14:52.523 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:52.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:52.525 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:14:52.625 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:14:52.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:52.657 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:52.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:14:52.733 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:52 smithi100 ceph-mon[29092]: 4.2 scrub starts 2024-09-17T07:14:52.733 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:52 smithi100 ceph-mon[29092]: 4.2 scrub ok 2024-09-17T07:14:52.733 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:52 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2806585259' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-17T07:14:52.734 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:52 smithi100 ceph-mon[29092]: osdmap e279: 8 total, 8 up, 7 in 2024-09-17T07:14:52.734 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:52 smithi100 ceph-mon[29092]: 6.13 scrub starts 2024-09-17T07:14:52.734 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:52 smithi100 ceph-mon[29092]: 6.13 scrub ok 2024-09-17T07:14:52.734 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:52 smithi100 ceph-mon[29092]: 6.14 scrub starts 2024-09-17T07:14:52.734 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:52 smithi100 ceph-mon[29092]: 6.14 scrub ok 2024-09-17T07:14:52.734 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:52 smithi100 ceph-mon[39462]: 4.2 scrub starts 2024-09-17T07:14:52.734 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:52 smithi100 ceph-mon[39462]: 4.2 scrub ok 2024-09-17T07:14:52.734 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:52 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2806585259' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-17T07:14:52.734 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:52 smithi100 ceph-mon[39462]: osdmap e279: 8 total, 8 up, 7 in 2024-09-17T07:14:52.735 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:52 smithi100 ceph-mon[39462]: 6.13 scrub starts 2024-09-17T07:14:52.735 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:52 smithi100 ceph-mon[39462]: 6.13 scrub ok 2024-09-17T07:14:52.735 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:52 smithi100 ceph-mon[39462]: 6.14 scrub starts 2024-09-17T07:14:52.735 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:52 smithi100 ceph-mon[39462]: 6.14 scrub ok 2024-09-17T07:14:52.758 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:14:52.788 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:52.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:52.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:14:52.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:52 smithi135 ceph-mon[33238]: 4.2 scrub starts 2024-09-17T07:14:52.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:52 smithi135 ceph-mon[33238]: 4.2 scrub ok 2024-09-17T07:14:52.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:52 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2806585259' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-17T07:14:52.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:52 smithi135 ceph-mon[33238]: osdmap e279: 8 total, 8 up, 7 in 2024-09-17T07:14:52.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:52 smithi135 ceph-mon[33238]: 6.13 scrub starts 2024-09-17T07:14:52.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:52 smithi135 ceph-mon[33238]: 6.13 scrub ok 2024-09-17T07:14:52.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:52 smithi135 ceph-mon[33238]: 6.14 scrub starts 2024-09-17T07:14:52.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:52 smithi135 ceph-mon[33238]: 6.14 scrub ok 2024-09-17T07:14:52.889 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:14:52.928 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:52.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:52.929 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:14:53.030 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:14:53.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:53.065 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:53.065 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:14:53.166 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:14:53.202 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:53.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:53.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:14:53.303 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:14:53.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:53.343 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:53.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:14:53.444 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:14:53.477 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:53.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:53.478 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:14:53.579 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:14:53.618 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:53.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:53.619 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:14:53.672 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:14:53.720 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:14:53.761 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:53.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:53.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:14:53.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:53 smithi135 ceph-mon[33238]: 6.c scrub starts 2024-09-17T07:14:53.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:53 smithi135 ceph-mon[33238]: 6.c scrub ok 2024-09-17T07:14:53.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:53 smithi135 ceph-mon[33238]: 6.15 scrub starts 2024-09-17T07:14:53.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:53 smithi135 ceph-mon[33238]: 6.15 scrub ok 2024-09-17T07:14:53.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:53 smithi135 ceph-mon[33238]: 6.19 scrub starts 2024-09-17T07:14:53.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:53 smithi135 ceph-mon[33238]: 6.19 scrub ok 2024-09-17T07:14:53.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:53 smithi135 ceph-mon[33238]: 5.1 scrub starts 2024-09-17T07:14:53.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:53 smithi135 ceph-mon[33238]: 5.1 scrub ok 2024-09-17T07:14:53.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:53 smithi135 ceph-mon[33238]: 5.15 scrub starts 2024-09-17T07:14:53.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:53 smithi135 ceph-mon[33238]: 5.15 scrub ok 2024-09-17T07:14:53.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:53 smithi135 ceph-mon[33238]: pgmap v567: 148 pgs: 148 active+clean; 3.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 651 B/s rd, 0 B/s wr, 794 op/s 2024-09-17T07:14:53.800 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:53 smithi100 ceph-mon[29092]: 6.c scrub starts 2024-09-17T07:14:53.800 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:53 smithi100 ceph-mon[29092]: 6.c scrub ok 2024-09-17T07:14:53.800 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:53 smithi100 ceph-mon[29092]: 6.15 scrub starts 2024-09-17T07:14:53.801 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:53 smithi100 ceph-mon[29092]: 6.15 scrub ok 2024-09-17T07:14:53.801 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:53 smithi100 ceph-mon[29092]: 6.19 scrub starts 2024-09-17T07:14:53.801 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:53 smithi100 ceph-mon[29092]: 6.19 scrub ok 2024-09-17T07:14:53.801 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:53 smithi100 ceph-mon[29092]: 5.1 scrub starts 2024-09-17T07:14:53.801 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:53 smithi100 ceph-mon[29092]: 5.1 scrub ok 2024-09-17T07:14:53.801 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:53 smithi100 ceph-mon[29092]: 5.15 scrub starts 2024-09-17T07:14:53.801 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:53 smithi100 ceph-mon[29092]: 5.15 scrub ok 2024-09-17T07:14:53.801 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:53 smithi100 ceph-mon[29092]: pgmap v567: 148 pgs: 148 active+clean; 3.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 651 B/s rd, 0 B/s wr, 794 op/s 2024-09-17T07:14:53.802 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:53 smithi100 ceph-mon[39462]: 6.c scrub starts 2024-09-17T07:14:53.802 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:53 smithi100 ceph-mon[39462]: 6.c scrub ok 2024-09-17T07:14:53.802 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:53 smithi100 ceph-mon[39462]: 6.15 scrub starts 2024-09-17T07:14:53.802 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:53 smithi100 ceph-mon[39462]: 6.15 scrub ok 2024-09-17T07:14:53.802 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:53 smithi100 ceph-mon[39462]: 6.19 scrub starts 2024-09-17T07:14:53.802 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:53 smithi100 ceph-mon[39462]: 6.19 scrub ok 2024-09-17T07:14:53.802 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:53 smithi100 ceph-mon[39462]: 5.1 scrub starts 2024-09-17T07:14:53.802 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:53 smithi100 ceph-mon[39462]: 5.1 scrub ok 2024-09-17T07:14:53.802 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:53 smithi100 ceph-mon[39462]: 5.15 scrub starts 2024-09-17T07:14:53.802 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:53 smithi100 ceph-mon[39462]: 5.15 scrub ok 2024-09-17T07:14:53.802 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:53 smithi100 ceph-mon[39462]: pgmap v567: 148 pgs: 148 active+clean; 3.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 651 B/s rd, 0 B/s wr, 794 op/s 2024-09-17T07:14:53.864 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:14:53.899 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:53.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:53.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:14:54.001 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:14:54.037 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:54.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:54.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:14:54.139 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:14:54.174 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:54.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:54.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:14:54.257 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd reweight 4 1 2024-09-17T07:14:54.276 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:14:54.314 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:54.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:54.315 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:14:54.416 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:14:54.454 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:54.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:54.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:14:54.557 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:14:54.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:54.601 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:54.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:14:54.702 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:14:54.734 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:54.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:54.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:14:54.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:54 smithi135 ceph-mon[33238]: 2.7 scrub starts 2024-09-17T07:14:54.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:54 smithi135 ceph-mon[33238]: 2.7 scrub ok 2024-09-17T07:14:54.822 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:54 smithi100 ceph-mon[29092]: 2.7 scrub starts 2024-09-17T07:14:54.822 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:54 smithi100 ceph-mon[29092]: 2.7 scrub ok 2024-09-17T07:14:54.822 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:54 smithi100 ceph-mon[39462]: 2.7 scrub starts 2024-09-17T07:14:54.822 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:54 smithi100 ceph-mon[39462]: 2.7 scrub ok 2024-09-17T07:14:54.837 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:14:54.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:54.873 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:54.874 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:14:54.974 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:14:55.013 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:55.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:55.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:14:55.115 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:14:55.146 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:55.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:55.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:14:55.248 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:14:55.283 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:55.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:55.284 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:14:55.385 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:14:55.415 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:55.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:55.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:14:55.516 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:14:55.549 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:55.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:55.551 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:14:55.652 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:14:55.685 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:55.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:55.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:14:55.787 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:14:55.808 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:55.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:55.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:14:55.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:55 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2915357619' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:14:55.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:55 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:14:55.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:55 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:14:55.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:55 smithi100 ceph-mon[29092]: pgmap v568: 148 pgs: 148 active+clean; 3.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.59k op/s 2024-09-17T07:14:55.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:55 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2915357619' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:14:55.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:55 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:14:55.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:55 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:14:55.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:55 smithi100 ceph-mon[39462]: pgmap v568: 148 pgs: 148 active+clean; 3.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.59k op/s 2024-09-17T07:14:55.910 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:14:55.946 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:55.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:55.947 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:14:56.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:55 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2915357619' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:14:56.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:55 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:14:56.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:55 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:14:56.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:55 smithi135 ceph-mon[33238]: pgmap v568: 148 pgs: 148 active+clean; 3.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.59k op/s 2024-09-17T07:14:56.048 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:14:56.075 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:56.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:56.076 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:14:56.150 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:14:55 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:14:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:14:56.177 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:14:56.204 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:56.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:56.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:14:56.306 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:14:56.333 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:56.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:56.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:14:56.435 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:14:56.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:56.476 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:56.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:14:56.578 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:14:56.605 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:56.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:56.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:14:56.707 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:14:56.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:56.737 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:56.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:14:56.838 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:14:56.865 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:56.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:56.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:14:56.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:56 smithi100 ceph-mon[29092]: 5.16 scrub starts 2024-09-17T07:14:56.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:56 smithi100 ceph-mon[29092]: 5.16 scrub ok 2024-09-17T07:14:56.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:56 smithi100 ceph-mon[29092]: 4.e scrub starts 2024-09-17T07:14:56.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:56 smithi100 ceph-mon[29092]: 4.e scrub ok 2024-09-17T07:14:56.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:56 smithi100 ceph-mon[29092]: osdmap e280: 8 total, 8 up, 7 in 2024-09-17T07:14:56.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:56 smithi100 ceph-mon[29092]: 9.e deep-scrub starts 2024-09-17T07:14:56.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:56 smithi100 ceph-mon[29092]: 6.d scrub starts 2024-09-17T07:14:56.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:56 smithi100 ceph-mon[29092]: 6.d scrub ok 2024-09-17T07:14:56.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:56 smithi100 ceph-mon[39462]: 5.16 scrub starts 2024-09-17T07:14:56.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:56 smithi100 ceph-mon[39462]: 5.16 scrub ok 2024-09-17T07:14:56.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:56 smithi100 ceph-mon[39462]: 4.e scrub starts 2024-09-17T07:14:56.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:56 smithi100 ceph-mon[39462]: 4.e scrub ok 2024-09-17T07:14:56.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:56 smithi100 ceph-mon[39462]: osdmap e280: 8 total, 8 up, 7 in 2024-09-17T07:14:56.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:56 smithi100 ceph-mon[39462]: 9.e deep-scrub starts 2024-09-17T07:14:56.879 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:56 smithi100 ceph-mon[39462]: 6.d scrub starts 2024-09-17T07:14:56.879 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:56 smithi100 ceph-mon[39462]: 6.d scrub ok 2024-09-17T07:14:56.967 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:14:56.992 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:56.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:56.993 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:14:57.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:56 smithi135 ceph-mon[33238]: 5.16 scrub starts 2024-09-17T07:14:57.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:56 smithi135 ceph-mon[33238]: 5.16 scrub ok 2024-09-17T07:14:57.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:56 smithi135 ceph-mon[33238]: 4.e scrub starts 2024-09-17T07:14:57.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:56 smithi135 ceph-mon[33238]: 4.e scrub ok 2024-09-17T07:14:57.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:56 smithi135 ceph-mon[33238]: osdmap e280: 8 total, 8 up, 7 in 2024-09-17T07:14:57.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:56 smithi135 ceph-mon[33238]: 9.e deep-scrub starts 2024-09-17T07:14:57.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:56 smithi135 ceph-mon[33238]: 6.d scrub starts 2024-09-17T07:14:57.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:56 smithi135 ceph-mon[33238]: 6.d scrub ok 2024-09-17T07:14:57.094 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:14:57.130 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:57.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:57.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:14:57.231 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:14:57.258 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:57.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:57.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:14:57.360 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:14:57.399 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:57.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:57.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:14:57.501 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:14:57.528 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:57.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:57.529 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:14:57.630 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:14:57.654 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:57.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:57.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:14:57.756 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:14:57.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:57.787 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:57.788 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:14:57.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:57 smithi100 ceph-mon[29092]: 2.f scrub starts 2024-09-17T07:14:57.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:57 smithi100 ceph-mon[29092]: 2.f scrub ok 2024-09-17T07:14:57.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:57 smithi100 ceph-mon[29092]: 1.0 deep-scrub starts 2024-09-17T07:14:57.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:57 smithi100 ceph-mon[29092]: 1.0 deep-scrub ok 2024-09-17T07:14:57.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:57 smithi100 ceph-mon[29092]: pgmap v570: 148 pgs: 148 active+clean; 3.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 511 B/s rd, 0 B/s wr, 1.00k op/s 2024-09-17T07:14:57.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:57 smithi100 ceph-mon[39462]: 2.f scrub starts 2024-09-17T07:14:57.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:57 smithi100 ceph-mon[39462]: 2.f scrub ok 2024-09-17T07:14:57.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:57 smithi100 ceph-mon[39462]: 1.0 deep-scrub starts 2024-09-17T07:14:57.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:57 smithi100 ceph-mon[39462]: 1.0 deep-scrub ok 2024-09-17T07:14:57.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:57 smithi100 ceph-mon[39462]: pgmap v570: 148 pgs: 148 active+clean; 3.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 511 B/s rd, 0 B/s wr, 1.00k op/s 2024-09-17T07:14:57.889 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:14:57.919 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:57.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:57.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:14:58.021 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:14:58.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:57 smithi135 ceph-mon[33238]: 2.f scrub starts 2024-09-17T07:14:58.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:57 smithi135 ceph-mon[33238]: 2.f scrub ok 2024-09-17T07:14:58.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:57 smithi135 ceph-mon[33238]: 1.0 deep-scrub starts 2024-09-17T07:14:58.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:57 smithi135 ceph-mon[33238]: 1.0 deep-scrub ok 2024-09-17T07:14:58.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:57 smithi135 ceph-mon[33238]: pgmap v570: 148 pgs: 148 active+clean; 3.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 511 B/s rd, 0 B/s wr, 1.00k op/s 2024-09-17T07:14:58.049 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:58.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:58.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:14:58.151 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:14:58.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:58.189 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:58.189 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:14:58.290 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:14:58.329 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:58.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:58.330 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:14:58.431 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:14:58.460 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:58.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:58.462 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:14:58.562 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:14:58.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:58.598 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:58.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:14:58.699 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:14:58.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:58.732 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:58.732 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:14:58.806 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:58 smithi135 ceph-mon[33238]: 4.4 scrub starts 2024-09-17T07:14:58.806 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:58 smithi135 ceph-mon[33238]: 4.4 scrub ok 2024-09-17T07:14:58.806 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:58 smithi135 ceph-mon[33238]: 2.1e scrub starts 2024-09-17T07:14:58.806 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:58 smithi135 ceph-mon[33238]: 2.1e scrub ok 2024-09-17T07:14:58.833 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:14:58.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:58.871 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:58.872 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:14:58.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:58 smithi100 ceph-mon[29092]: 4.4 scrub starts 2024-09-17T07:14:58.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:58 smithi100 ceph-mon[29092]: 4.4 scrub ok 2024-09-17T07:14:58.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:58 smithi100 ceph-mon[29092]: 2.1e scrub starts 2024-09-17T07:14:58.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:58 smithi100 ceph-mon[29092]: 2.1e scrub ok 2024-09-17T07:14:58.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:58 smithi100 ceph-mon[39462]: 4.4 scrub starts 2024-09-17T07:14:58.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:58 smithi100 ceph-mon[39462]: 4.4 scrub ok 2024-09-17T07:14:58.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:58 smithi100 ceph-mon[39462]: 2.1e scrub starts 2024-09-17T07:14:58.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:58 smithi100 ceph-mon[39462]: 2.1e scrub ok 2024-09-17T07:14:58.973 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:14:59.008 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:59.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:59.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:14:59.110 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:14:59.141 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:59.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:59.142 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:14:59.243 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:14:59.271 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:59.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:59.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:14:59.293 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:14:58 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:14:59.373 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:14:59.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:59.406 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:59.407 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:14:59.507 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:14:59.543 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:59.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:59.544 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:14:59.566 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:14:59.566 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1: {} 2024-09-17T07:14:59.567 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:14:59.567 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:14:59.567 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:14:59.567 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:14:59.567 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:14:59.567 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:14:59.567 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:14:59.567 INFO:teuthology.orchestra.run.smithi100.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:14:59.567 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:14:59.568 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:14:59.568 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:14:59.568 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:14:59.568 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:14:59.568 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:14:59.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:59 smithi100 ceph-mon[39462]: 4.1d scrub starts 2024-09-17T07:14:59.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:59 smithi100 ceph-mon[39462]: 4.1d scrub ok 2024-09-17T07:14:59.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:59 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:14:59.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:59 smithi100 ceph-mon[39462]: 5.1b scrub starts 2024-09-17T07:14:59.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:59 smithi100 ceph-mon[39462]: 5.1b scrub ok 2024-09-17T07:14:59.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:14:59 smithi100 ceph-mon[39462]: pgmap v571: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 2.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.37k op/s 2024-09-17T07:14:59.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:59 smithi100 ceph-mon[29092]: 4.1d scrub starts 2024-09-17T07:14:59.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:59 smithi100 ceph-mon[29092]: 4.1d scrub ok 2024-09-17T07:14:59.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:59 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:14:59.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:59 smithi100 ceph-mon[29092]: 5.1b scrub starts 2024-09-17T07:14:59.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:59 smithi100 ceph-mon[29092]: 5.1b scrub ok 2024-09-17T07:14:59.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:14:59 smithi100 ceph-mon[29092]: pgmap v571: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 2.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.37k op/s 2024-09-17T07:14:59.645 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:14:59.672 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:59.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:59.673 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:14:59.774 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:14:59.817 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:14:59.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:59.818 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:14:59.919 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:14:59.948 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:14:59.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:14:59.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:15:00.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:59 smithi135 ceph-mon[33238]: 4.1d scrub starts 2024-09-17T07:15:00.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:59 smithi135 ceph-mon[33238]: 4.1d scrub ok 2024-09-17T07:15:00.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:59 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:15:00.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:59 smithi135 ceph-mon[33238]: 5.1b scrub starts 2024-09-17T07:15:00.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:59 smithi135 ceph-mon[33238]: 5.1b scrub ok 2024-09-17T07:15:00.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:14:59 smithi135 ceph-mon[33238]: pgmap v571: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 2.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.37k op/s 2024-09-17T07:15:00.050 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:15:00.086 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:00.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:00.087 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:15:00.188 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:15:00.222 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:00.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:00.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:15:00.323 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:15:00.359 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:00.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:00.360 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:15:00.461 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:15:00.490 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:00.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:00.491 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:15:00.592 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:15:00.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:00.623 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:00.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:15:00.724 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:15:00.754 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:00.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:00.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:15:00.856 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:15:00.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:00 smithi100 ceph-mon[29092]: 2.a scrub starts 2024-09-17T07:15:00.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:00 smithi100 ceph-mon[29092]: 2.a scrub ok 2024-09-17T07:15:00.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:00 smithi100 ceph-mon[29092]: 2.18 scrub starts 2024-09-17T07:15:00.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:00 smithi100 ceph-mon[29092]: 2.18 scrub ok 2024-09-17T07:15:00.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:00 smithi100 ceph-mon[29092]: 2.17 scrub starts 2024-09-17T07:15:00.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:00 smithi100 ceph-mon[29092]: 2.17 scrub ok 2024-09-17T07:15:00.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:00 smithi100 ceph-mon[39462]: 2.a scrub starts 2024-09-17T07:15:00.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:00 smithi100 ceph-mon[39462]: 2.a scrub ok 2024-09-17T07:15:00.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:00 smithi100 ceph-mon[39462]: 2.18 scrub starts 2024-09-17T07:15:00.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:00 smithi100 ceph-mon[39462]: 2.18 scrub ok 2024-09-17T07:15:00.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:00 smithi100 ceph-mon[39462]: 2.17 scrub starts 2024-09-17T07:15:00.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:00 smithi100 ceph-mon[39462]: 2.17 scrub ok 2024-09-17T07:15:00.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:00.895 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:00.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:15:00.904 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-17T07:15:00.904 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:15:00.997 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:15:01.025 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:01.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:01.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:15:01.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:00 smithi135 ceph-mon[33238]: 2.a scrub starts 2024-09-17T07:15:01.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:00 smithi135 ceph-mon[33238]: 2.a scrub ok 2024-09-17T07:15:01.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:00 smithi135 ceph-mon[33238]: 2.18 scrub starts 2024-09-17T07:15:01.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:00 smithi135 ceph-mon[33238]: 2.18 scrub ok 2024-09-17T07:15:01.045 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:00 smithi135 ceph-mon[33238]: 2.17 scrub starts 2024-09-17T07:15:01.045 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:00 smithi135 ceph-mon[33238]: 2.17 scrub ok 2024-09-17T07:15:01.127 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:15:01.157 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:01.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:01.158 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:15:01.259 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:15:01.289 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:01.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:01.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:15:01.391 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:15:01.427 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:01.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:01.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:15:01.529 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:15:01.556 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:01.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:01.557 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:15:01.600 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-17T07:15:01.658 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:15:01.690 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:01.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:01.691 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:15:01.791 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:15:01.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:01.823 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:01.823 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:15:01.851 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:01 smithi100 ceph-mon[29092]: 6.9 scrub starts 2024-09-17T07:15:01.852 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:01 smithi100 ceph-mon[29092]: 6.9 scrub ok 2024-09-17T07:15:01.852 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:01 smithi100 ceph-mon[29092]: 2.16 scrub starts 2024-09-17T07:15:01.852 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:01 smithi100 ceph-mon[29092]: 2.16 scrub ok 2024-09-17T07:15:01.852 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:01 smithi100 ceph-mon[29092]: pgmap v572: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 2.7 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 937 B/s rd, 0 B/s wr, 1.26k op/s 2024-09-17T07:15:01.852 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:01 smithi100 ceph-mon[39462]: 6.9 scrub starts 2024-09-17T07:15:01.852 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:01 smithi100 ceph-mon[39462]: 6.9 scrub ok 2024-09-17T07:15:01.852 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:01 smithi100 ceph-mon[39462]: 2.16 scrub starts 2024-09-17T07:15:01.852 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:01 smithi100 ceph-mon[39462]: 2.16 scrub ok 2024-09-17T07:15:01.852 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:01 smithi100 ceph-mon[39462]: pgmap v572: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 2.7 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 937 B/s rd, 0 B/s wr, 1.26k op/s 2024-09-17T07:15:01.924 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:15:01.957 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:01.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:01.958 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:15:02.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:01 smithi135 ceph-mon[33238]: 6.9 scrub starts 2024-09-17T07:15:02.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:01 smithi135 ceph-mon[33238]: 6.9 scrub ok 2024-09-17T07:15:02.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:01 smithi135 ceph-mon[33238]: 2.16 scrub starts 2024-09-17T07:15:02.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:01 smithi135 ceph-mon[33238]: 2.16 scrub ok 2024-09-17T07:15:02.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:01 smithi135 ceph-mon[33238]: pgmap v572: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 2.7 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 937 B/s rd, 0 B/s wr, 1.26k op/s 2024-09-17T07:15:02.059 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:15:02.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:02.098 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:02.098 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:15:02.199 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:15:02.245 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:02.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:02.246 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:15:02.276 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:15:02.289 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:15:02.348 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:15:02.389 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:02.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:02.390 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:15:02.491 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:15:02.526 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:02.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:02.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:15:02.576 INFO:teuthology.orchestra.run.smithi100.stderr:noscrub is set 2024-09-17T07:15:02.628 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:15:02.658 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:02.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:02.659 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:15:02.683 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:02 smithi100 ceph-mon[29092]: 3.1 scrub starts 2024-09-17T07:15:02.683 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:02 smithi100 ceph-mon[29092]: 3.1 scrub ok 2024-09-17T07:15:02.683 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:02 smithi100 ceph-mon[29092]: 6.5 deep-scrub starts 2024-09-17T07:15:02.683 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:02 smithi100 ceph-mon[29092]: 6.5 deep-scrub ok 2024-09-17T07:15:02.683 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:02 smithi100 ceph-mon[29092]: 6.1f scrub starts 2024-09-17T07:15:02.683 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:02 smithi100 ceph-mon[29092]: 6.1f scrub ok 2024-09-17T07:15:02.683 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:02 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1305094754' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:15:02.684 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:02 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:15:02.684 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:02 smithi100 ceph-mon[39462]: 3.1 scrub starts 2024-09-17T07:15:02.684 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:02 smithi100 ceph-mon[39462]: 3.1 scrub ok 2024-09-17T07:15:02.684 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:02 smithi100 ceph-mon[39462]: 6.5 deep-scrub starts 2024-09-17T07:15:02.684 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:02 smithi100 ceph-mon[39462]: 6.5 deep-scrub ok 2024-09-17T07:15:02.684 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:02 smithi100 ceph-mon[39462]: 6.1f scrub starts 2024-09-17T07:15:02.684 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:02 smithi100 ceph-mon[39462]: 6.1f scrub ok 2024-09-17T07:15:02.684 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:02 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1305094754' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:15:02.684 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:02 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:15:02.760 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:15:02.800 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:02.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:02.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:15:02.902 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:15:02.941 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:02.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:02.943 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:15:03.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:02 smithi135 ceph-mon[33238]: 3.1 scrub starts 2024-09-17T07:15:03.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:02 smithi135 ceph-mon[33238]: 3.1 scrub ok 2024-09-17T07:15:03.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:02 smithi135 ceph-mon[33238]: 6.5 deep-scrub starts 2024-09-17T07:15:03.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:02 smithi135 ceph-mon[33238]: 6.5 deep-scrub ok 2024-09-17T07:15:03.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:02 smithi135 ceph-mon[33238]: 6.1f scrub starts 2024-09-17T07:15:03.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:02 smithi135 ceph-mon[33238]: 6.1f scrub ok 2024-09-17T07:15:03.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:02 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1305094754' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:15:03.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:02 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:15:03.045 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:15:03.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:03.085 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:03.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:15:03.186 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:15:03.229 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:03.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:03.230 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:15:03.331 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:15:03.370 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:03.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:03.372 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:15:03.473 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:15:03.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:03.505 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:03.506 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:15:03.606 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:15:03.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:03 smithi100 ceph-mon[29092]: 4.c scrub starts 2024-09-17T07:15:03.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:03 smithi100 ceph-mon[29092]: 4.c scrub ok 2024-09-17T07:15:03.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:03 smithi100 ceph-mon[29092]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:15:03.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:03 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T07:15:03.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:03 smithi100 ceph-mon[29092]: osdmap e281: 8 total, 8 up, 7 in 2024-09-17T07:15:03.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:03 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:15:03.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:03 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:15:03.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:03 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:15:03.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:03 smithi100 ceph-mon[29092]: pgmap v574: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 2.7 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 745 op/s 2024-09-17T07:15:03.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:03 smithi100 ceph-mon[39462]: 4.c scrub starts 2024-09-17T07:15:03.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:03 smithi100 ceph-mon[39462]: 4.c scrub ok 2024-09-17T07:15:03.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:03 smithi100 ceph-mon[39462]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:15:03.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:03 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T07:15:03.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:03 smithi100 ceph-mon[39462]: osdmap e281: 8 total, 8 up, 7 in 2024-09-17T07:15:03.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:03 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:15:03.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:03 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:15:03.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:03 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:15:03.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:03 smithi100 ceph-mon[39462]: pgmap v574: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 2.7 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 745 op/s 2024-09-17T07:15:03.637 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:03.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:03.638 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:15:03.739 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:15:03.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:03.770 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:03.771 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:15:03.871 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:15:03.898 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:03.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:03.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:15:04.000 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:15:04.002 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:15:04.032 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:04.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:04.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:15:04.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:03 smithi135 ceph-mon[33238]: 4.c scrub starts 2024-09-17T07:15:04.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:03 smithi135 ceph-mon[33238]: 4.c scrub ok 2024-09-17T07:15:04.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:03 smithi135 ceph-mon[33238]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:15:04.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:03 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T07:15:04.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:03 smithi135 ceph-mon[33238]: osdmap e281: 8 total, 8 up, 7 in 2024-09-17T07:15:04.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:03 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:15:04.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:03 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:15:04.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:03 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:15:04.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:03 smithi135 ceph-mon[33238]: pgmap v574: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 2.7 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 745 op/s 2024-09-17T07:15:04.133 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:15:04.168 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:04.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:04.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:15:04.270 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:15:04.302 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:04.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:04.303 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:15:04.404 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:15:04.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:04.446 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:04.446 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:15:04.547 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:15:04.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:04.586 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:04.586 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:15:04.687 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:15:04.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:04.723 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:04.724 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:15:04.824 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:15:04.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:04.857 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:04.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:15:04.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:04 smithi100 ceph-mon[29092]: osdmap e282: 8 total, 8 up, 7 in 2024-09-17T07:15:04.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:04 smithi100 ceph-mon[39462]: osdmap e282: 8 total, 8 up, 7 in 2024-09-17T07:15:04.958 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:15:04.993 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:04.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:04.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:15:05.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:04 smithi135 ceph-mon[33238]: osdmap e282: 8 total, 8 up, 7 in 2024-09-17T07:15:05.095 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:15:05.133 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:05.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:05.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:15:05.235 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:15:05.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:05.279 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:05.279 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:15:05.380 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:15:05.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:05.415 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:05.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:15:05.516 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:15:05.549 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:05.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:05.550 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:15:05.651 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:15:05.683 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:05.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:05.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:15:05.785 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:15:05.818 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:05.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:05.819 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:15:05.920 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:15:05.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:05.944 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:05.944 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:15:06.045 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:15:06.073 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:06.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:06.074 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:15:06.174 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:15:06.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:06.207 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:06.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:15:06.236 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:15:05 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:15:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:15:06.243 INFO:teuthology.orchestra.run.smithi100.stderr:reweighted osd.4 to 1 (10000) 2024-09-17T07:15:06.308 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:15:06.354 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:06.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:06.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:15:06.456 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:15:06.489 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:06.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:06.490 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:15:06.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:06 smithi135 ceph-mon[33238]: pgmap v576: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 2.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.48k op/s 2024-09-17T07:15:06.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:06 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/4052731702' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-17T07:15:06.591 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:15:06.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:06.620 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:06.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:15:06.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:06 smithi100 ceph-mon[29092]: pgmap v576: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 2.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.48k op/s 2024-09-17T07:15:06.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:06 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/4052731702' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-17T07:15:06.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:06 smithi100 ceph-mon[39462]: pgmap v576: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 2.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.48k op/s 2024-09-17T07:15:06.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:06 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/4052731702' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-17T07:15:06.722 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:15:06.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:06.754 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:06.755 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:15:06.856 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:15:06.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:06.877 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:06.878 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:15:06.948 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-17T07:15:06.978 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:15:07.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:07.029 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:07.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:15:07.130 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:15:07.170 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:07.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:07.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:15:07.272 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:15:07.295 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:07.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:07.296 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:15:07.397 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:15:07.418 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:07.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:07.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:15:07.520 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:15:07.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:07 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/4052731702' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-17T07:15:07.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:07 smithi135 ceph-mon[33238]: osdmap e283: 8 total, 8 up, 7 in 2024-09-17T07:15:07.543 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:07.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:07.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:15:07.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:07 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/4052731702' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-17T07:15:07.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:07 smithi100 ceph-mon[29092]: osdmap e283: 8 total, 8 up, 7 in 2024-09-17T07:15:07.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:07 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/4052731702' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-17T07:15:07.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:07 smithi100 ceph-mon[39462]: osdmap e283: 8 total, 8 up, 7 in 2024-09-17T07:15:07.645 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:15:07.682 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:07.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:07.683 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:15:07.784 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:15:07.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:07.821 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:07.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:15:07.922 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:15:07.957 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:07.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:07.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:15:08.059 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:15:08.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:08.097 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:08.097 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:15:08.198 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:15:08.227 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:08.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:08.228 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:15:08.329 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:15:08.365 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:08.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:08.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:15:08.418 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd reweight 5 1 2024-09-17T07:15:08.448 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:15:08.467 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:15:08.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:08.533 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:08.534 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:15:08.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:08 smithi135 ceph-mon[33238]: pgmap v578: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 2.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 980 op/s 2024-09-17T07:15:08.577 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:08 smithi100 ceph-mon[29092]: pgmap v578: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 2.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 980 op/s 2024-09-17T07:15:08.578 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:08 smithi100 ceph-mon[39462]: pgmap v578: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 2.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 980 op/s 2024-09-17T07:15:08.634 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:15:08.670 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:08.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:08.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:15:08.772 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:15:08.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:08.806 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:08.807 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:15:08.907 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:15:08.942 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:08.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:08.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:15:09.044 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:15:09.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:09.082 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:09.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:15:09.097 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-17T07:15:09.097 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:15:09.183 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:15:09.190 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:15:08 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:15:09.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:09.211 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:09.211 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:15:09.312 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:15:09.343 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:15:09.359 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:09.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:09.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:15:09.461 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:15:09.500 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:09.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:09.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:15:09.603 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:15:09.642 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:09.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:09.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:15:09.744 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:15:09.784 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:09.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:09.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:15:09.886 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:15:09.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:09.923 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:09.924 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:15:10.024 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:15:10.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:10.056 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:10.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:15:10.067 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-17T07:15:10.158 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:15:10.196 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:10.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:10.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:15:10.298 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:15:10.332 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:10.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:10.334 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:15:10.434 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:15:10.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:10.469 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:10.469 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:15:10.570 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:15:10.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:10.601 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:10.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:15:10.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:10 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:15:10.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:10 smithi100 ceph-mon[29092]: pgmap v579: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 2.1 GiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.49k op/s 2024-09-17T07:15:10.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:10 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:15:10.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:10 smithi100 ceph-mon[39462]: pgmap v579: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 2.1 GiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.49k op/s 2024-09-17T07:15:10.703 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:15:10.732 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:10.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:10.733 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:15:10.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:10 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:15:10.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:10 smithi135 ceph-mon[33238]: pgmap v579: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 2.1 GiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.49k op/s 2024-09-17T07:15:10.834 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:15:10.862 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:10.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:10.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:15:10.964 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:15:11.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:11.004 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:11.004 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:15:11.105 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:15:11.126 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:11.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:11.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:15:11.228 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:15:11.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:11.257 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:11.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:15:11.358 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:15:11.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:11.402 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:11.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:15:11.503 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:15:11.532 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:11.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:11.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:15:11.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:11 smithi100 ceph-mon[29092]: 9.e deep-scrub ok 2024-09-17T07:15:11.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:11 smithi100 ceph-mon[39462]: 9.e deep-scrub ok 2024-09-17T07:15:11.634 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:15:11.678 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:11.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:11.679 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:15:11.780 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:15:11.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:11 smithi135 ceph-mon[33238]: 9.e deep-scrub ok 2024-09-17T07:15:11.804 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:15:11.819 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:11.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:11.820 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:15:11.921 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:15:11.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:11.958 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:11.958 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:15:12.059 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:15:12.094 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:12.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:12.095 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:15:12.196 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:15:12.230 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:12.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:12.231 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:15:12.332 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:15:12.369 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:12.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:12.370 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:15:12.471 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:15:12.502 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:12.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:12.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:15:12.604 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:15:12.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:12 smithi100 ceph-mon[29092]: pgmap v580: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 2.1 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.42k op/s 2024-09-17T07:15:12.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:12 smithi100 ceph-mon[39462]: pgmap v580: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 2.1 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.42k op/s 2024-09-17T07:15:12.640 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:12.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:12.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:15:12.742 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:15:12.777 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:12.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:12.778 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:15:12.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:12 smithi135 ceph-mon[33238]: pgmap v580: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 2.1 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.42k op/s 2024-09-17T07:15:12.880 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:15:12.909 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:12.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:12.910 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:15:13.011 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:15:13.047 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:13.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:13.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:15:13.149 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:15:13.181 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:13.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:13.182 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:15:13.283 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:15:13.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:13.322 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:13.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:15:13.423 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:15:13.459 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:13.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:13.461 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:15:13.559 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:15:13.559 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:15:13.561 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:15:13.591 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1: {} 2024-09-17T07:15:13.591 INFO:teuthology.orchestra.run.smithi100.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:15:13.596 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:13.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:13.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:15:13.620 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:15:13.620 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:15:13.649 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:15:13.649 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:15:13.683 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:15:13.683 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:15:13.698 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:15:13.710 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:15:13.710 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:15:13.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:13.738 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:13.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:15:13.746 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:15:13.746 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:15:13.769 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:15:13.770 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:15:13.839 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:15:13.887 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:13.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:13.888 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:15:13.989 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:15:14.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:14.023 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:14.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:15:14.124 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:15:14.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:14.162 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:14.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:15:14.262 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:15:14.309 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:14.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:14.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:15:14.411 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:15:14.433 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:14.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:14.434 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:15:14.535 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:15:14.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:14.571 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:14.571 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:15:14.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:14 smithi100 ceph-mon[29092]: pgmap v581: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 2.1 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.10k op/s 2024-09-17T07:15:14.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:14 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2471390561' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:15:14.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:14 smithi100 ceph-mon[39462]: pgmap v581: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 2.1 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.10k op/s 2024-09-17T07:15:14.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:14 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2471390561' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:15:14.673 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:15:14.700 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:14.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:14.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:15:14.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:14 smithi135 ceph-mon[33238]: pgmap v581: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 2.1 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.10k op/s 2024-09-17T07:15:14.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:14 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2471390561' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:15:14.802 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:15:14.839 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:14.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:14.840 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:15:14.941 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:15:14.974 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:14.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:14.975 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:15:15.076 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:15:15.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:15.112 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:15.112 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:15:15.214 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:15:15.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:15.249 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:15.249 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:15:15.350 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:15:15.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:15.391 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:15.391 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:15:15.492 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:15:15.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:15.535 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:15.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:15:15.614 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:15 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1242555856' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-17T07:15:15.614 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:15 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-17T07:15:15.614 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:15 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1242555856' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-17T07:15:15.614 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:15 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-17T07:15:15.636 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:15:15.667 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:15.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:15.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:15:15.769 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:15:15.781 INFO:teuthology.orchestra.run.smithi100.stderr:reweighted osd.5 to 1 (10000) 2024-09-17T07:15:15.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:15 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1242555856' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-17T07:15:15.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:15 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-17T07:15:15.817 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:15.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:15.819 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:15:15.919 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:15:15.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:15.955 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:15.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:15:16.056 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:15:16.087 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:16.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:16.088 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:15:16.188 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:15:16.223 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:16.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:16.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:15:16.325 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:15:16.352 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:15:15 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:15:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:15:16.361 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:16.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:16.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:15:16.439 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd reweight 6 1 2024-09-17T07:15:16.463 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:15:16.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:16.516 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:16.516 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:15:16.617 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:15:16.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:16 smithi100 ceph-mon[29092]: pgmap v582: 148 pgs: 148 active+clean; 1.7 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.17k op/s 2024-09-17T07:15:16.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:16 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-17T07:15:16.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:16 smithi100 ceph-mon[29092]: osdmap e284: 8 total, 8 up, 7 in 2024-09-17T07:15:16.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:16 smithi100 ceph-mon[39462]: pgmap v582: 148 pgs: 148 active+clean; 1.7 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.17k op/s 2024-09-17T07:15:16.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:16 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-17T07:15:16.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:16 smithi100 ceph-mon[39462]: osdmap e284: 8 total, 8 up, 7 in 2024-09-17T07:15:16.660 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:16.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:16.661 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:15:16.762 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:15:16.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:16 smithi135 ceph-mon[33238]: pgmap v582: 148 pgs: 148 active+clean; 1.7 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.17k op/s 2024-09-17T07:15:16.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:16 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-17T07:15:16.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:16 smithi135 ceph-mon[33238]: osdmap e284: 8 total, 8 up, 7 in 2024-09-17T07:15:16.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:16.798 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:16.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:15:16.899 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:15:16.934 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:16.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:16.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:15:16.996 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:15:17.036 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:15:17.078 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:17.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:17.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:15:17.180 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:15:17.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:17.220 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:17.221 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:15:17.321 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:15:17.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:17.356 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:17.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:15:17.365 INFO:teuthology.orchestra.run.smithi100.stderr:nodeep-scrub is set 2024-09-17T07:15:17.458 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:15:17.488 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:17.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:17.489 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:15:17.590 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:15:17.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:17 smithi100 ceph-mon[29092]: pgmap v584: 148 pgs: 148 active+clean; 1.7 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.17k op/s 2024-09-17T07:15:17.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:17 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1042384334' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:15:17.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:17 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:15:17.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:17 smithi100 ceph-mon[39462]: pgmap v584: 148 pgs: 148 active+clean; 1.7 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.17k op/s 2024-09-17T07:15:17.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:17 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1042384334' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:15:17.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:17 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:15:17.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:17.629 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:17.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:15:17.730 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:15:17.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:17.762 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:17.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:15:17.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:17 smithi135 ceph-mon[33238]: pgmap v584: 148 pgs: 148 active+clean; 1.7 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.17k op/s 2024-09-17T07:15:17.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:17 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1042384334' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:15:17.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:17 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:15:17.863 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:15:17.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:17.898 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:17.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:15:17.999 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:15:18.039 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:18.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:18.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:15:18.141 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:15:18.157 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:15:18.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:18.178 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:18.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:15:18.279 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:15:18.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:18.309 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:18.310 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:15:18.410 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:15:18.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:18.452 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:18.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:15:18.553 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:15:18.583 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:18.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:18.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:15:18.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:18 smithi100 ceph-mon[29092]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:15:18.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:18 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T07:15:18.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:18 smithi100 ceph-mon[29092]: osdmap e285: 8 total, 8 up, 7 in 2024-09-17T07:15:18.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:18 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:15:18.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:18 smithi100 ceph-mon[39462]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:15:18.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:18 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T07:15:18.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:18 smithi100 ceph-mon[39462]: osdmap e285: 8 total, 8 up, 7 in 2024-09-17T07:15:18.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:18 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:15:18.686 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:15:18.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:18.722 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:18.722 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:15:18.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:18 smithi135 ceph-mon[33238]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:15:18.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:18 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T07:15:18.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:18 smithi135 ceph-mon[33238]: osdmap e285: 8 total, 8 up, 7 in 2024-09-17T07:15:18.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:18 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:15:18.823 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:15:18.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:18.858 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:18.858 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:15:18.959 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:15:18.998 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:18.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:18.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:15:19.100 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:15:19.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:19.141 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:19.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:15:19.242 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:15:19.274 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:19.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:19.275 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:15:19.293 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:15:18 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:15:19.376 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:15:19.409 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:19.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:19.410 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:15:19.457 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:15:19.511 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:15:19.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:19.543 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:19.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:15:19.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:19 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:15:19.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:19 smithi100 ceph-mon[29092]: pgmap v586: 148 pgs: 148 active+clean; 1.4 GiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.43k op/s 2024-09-17T07:15:19.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:19 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:15:19.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:19 smithi100 ceph-mon[39462]: pgmap v586: 148 pgs: 148 active+clean; 1.4 GiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.43k op/s 2024-09-17T07:15:19.643 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:15:19.682 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:19.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:19.683 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:15:19.784 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:15:19.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:19 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:15:19.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:19 smithi135 ceph-mon[33238]: pgmap v586: 148 pgs: 148 active+clean; 1.4 GiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.43k op/s 2024-09-17T07:15:19.819 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:19.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:19.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:15:19.921 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:15:19.955 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:19.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:19.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:15:20.057 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:15:20.097 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:20.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:20.098 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:15:20.134 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T07:15:20.199 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:15:20.234 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:20.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:20.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:15:20.336 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:15:20.371 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:20.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:20.372 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:15:20.473 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:15:20.515 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:20.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:20.516 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:15:20.616 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:15:20.651 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:20.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:20.652 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:15:20.752 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:15:20.786 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:20.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:20.787 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:15:20.888 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:15:20.921 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:20.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:20.922 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:15:21.023 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:15:21.055 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:21.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:21.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:15:21.157 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:15:21.191 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:21.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:21.192 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:15:21.293 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:15:21.313 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:21.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:21.315 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:15:21.415 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:15:21.453 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:21.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:21.454 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:15:21.555 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:15:21.584 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:21.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:21.585 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:15:21.686 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:15:21.727 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:21.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:21.728 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:15:21.829 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:15:21.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:21.861 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:21.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:15:21.962 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:15:21.994 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:21.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:21.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:15:22.095 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:15:22.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:22.117 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:22.117 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:15:22.218 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:15:22.240 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:22.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:22.241 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:15:22.341 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:15:22.378 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:22.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:22.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:15:22.480 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:15:22.509 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:22.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:22.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:15:22.611 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:15:22.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:22 smithi100 ceph-mon[29092]: pgmap v587: 148 pgs: 148 active+clean; 1.3 GiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.65k op/s 2024-09-17T07:15:22.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:22 smithi100 ceph-mon[39462]: pgmap v587: 148 pgs: 148 active+clean; 1.3 GiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.65k op/s 2024-09-17T07:15:22.646 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:22.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:22.647 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:15:22.748 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:15:22.783 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:22.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:22.784 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:15:22.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:22 smithi135 ceph-mon[33238]: pgmap v587: 148 pgs: 148 active+clean; 1.3 GiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.65k op/s 2024-09-17T07:15:22.885 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:15:22.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:22.929 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:22.929 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:15:23.030 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:15:23.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:23.072 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:23.072 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:15:23.173 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:15:23.203 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:23.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:23.204 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:15:23.305 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:15:23.348 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:23.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:23.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:15:23.450 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:15:23.489 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:23.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:23.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:15:23.591 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:15:23.621 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:23.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:23.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:15:23.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:23 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:15:23.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:23 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:15:23.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:23 smithi100 ceph-mon[39462]: osdmap e286: 8 total, 8 up, 7 in 2024-09-17T07:15:23.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:23 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:15:23.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:23 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:15:23.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:23 smithi100 ceph-mon[29092]: osdmap e286: 8 total, 8 up, 7 in 2024-09-17T07:15:23.723 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:15:23.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:23.760 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:23.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:15:23.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:23 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:15:23.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:23 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:15:23.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:23 smithi135 ceph-mon[33238]: osdmap e286: 8 total, 8 up, 7 in 2024-09-17T07:15:23.839 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:15:23.862 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:15:23.878 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-17T07:15:23.878 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-17T07:15:23.878 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-17T07:15:23.878 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-17T07:15:23.878 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.879 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.24929.0:108584 8.7 8:e1cd5908:::benchmark_data_smithi100_102803_object108583:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e193)", 2024-09-17T07:15:23.879 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:09:23.370549+0000", 2024-09-17T07:15:23.879 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 360.502056294, 2024-09-17T07:15:23.879 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1232894089999998, 2024-09-17T07:15:23.879 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:15:23.879 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-17T07:15:23.879 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:15:23.879 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.24929", 2024-09-17T07:15:23.879 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/564515582", 2024-09-17T07:15:23.879 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 108584 2024-09-17T07:15:23.879 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.879 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:15:23.880 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.880 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:15:23.880 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.370549+0000", 2024-09-17T07:15:23.880 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:15:23.880 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.880 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.880 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:15:23.880 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.370549+0000", 2024-09-17T07:15:23.880 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:15:23.880 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.880 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.881 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:15:23.881 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.370547+0000", 2024-09-17T07:15:23.881 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999981 2024-09-17T07:15:23.881 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.881 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.881 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:15:23.881 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.370572+0000", 2024-09-17T07:15:23.881 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4349e-05 2024-09-17T07:15:23.881 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.881 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.881 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:15:23.881 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.370590+0000", 2024-09-17T07:15:23.882 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.8717999999999999e-05 2024-09-17T07:15:23.882 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.882 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.882 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:15:23.882 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.370599+0000", 2024-09-17T07:15:23.882 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.4600000000000003e-06 2024-09-17T07:15:23.882 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.882 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.882 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:15:23.882 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.370651+0000", 2024-09-17T07:15:23.882 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2138999999999997e-05 2024-09-17T07:15:23.883 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.883 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.883 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:15:23.883 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.370807+0000", 2024-09-17T07:15:23.883 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000155985 2024-09-17T07:15:23.883 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.883 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.883 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 3,6", 2024-09-17T07:15:23.883 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.370983+0000", 2024-09-17T07:15:23.883 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00017630799999999999 2024-09-17T07:15:23.883 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.883 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.884 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:15:23.884 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.380345+0000", 2024-09-17T07:15:23.884 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.009362202 2024-09-17T07:15:23.884 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.884 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.884 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:15:23.884 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.389703+0000", 2024-09-17T07:15:23.884 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0093574169999999998 2024-09-17T07:15:23.884 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.884 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.884 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:15:23.885 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.418524+0000", 2024-09-17T07:15:23.885 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.028821352000000001 2024-09-17T07:15:23.885 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.885 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.885 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-17T07:15:23.885 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.418536+0000", 2024-09-17T07:15:23.885 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1304999999999999e-05 2024-09-17T07:15:23.885 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.885 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.885 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:15:23.885 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:24.400394+0000", 2024-09-17T07:15:23.885 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.98185840400000002 2024-09-17T07:15:23.886 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.886 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.886 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-17T07:15:23.886 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:24.400406+0000", 2024-09-17T07:15:23.886 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1963e-05 2024-09-17T07:15:23.886 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.886 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.886 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:15:23.886 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:25.493774+0000", 2024-09-17T07:15:23.886 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0933686149999999 2024-09-17T07:15:23.886 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.886 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.886 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:15:23.887 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:25.493838+0000", 2024-09-17T07:15:23.887 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3930000000000006e-05 2024-09-17T07:15:23.887 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:15:23.887 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:15:23.887 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:15:23.887 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.887 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.887 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.24929.0:108573 8.f 8:f7d53c2a:::benchmark_data_smithi100_102803_object108572:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e194)", 2024-09-17T07:15:23.887 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:09:23.382980+0000", 2024-09-17T07:15:23.887 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 360.489625386, 2024-09-17T07:15:23.887 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0796412069999999, 2024-09-17T07:15:23.887 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:15:23.888 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-17T07:15:23.888 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:15:23.888 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.24929", 2024-09-17T07:15:23.888 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/564515582", 2024-09-17T07:15:23.888 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 108573 2024-09-17T07:15:23.888 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.888 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:15:23.888 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.888 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:15:23.888 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.382980+0000", 2024-09-17T07:15:23.888 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:15:23.888 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.888 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.889 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:15:23.889 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.382980+0000", 2024-09-17T07:15:23.889 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:15:23.889 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.889 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.889 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:15:23.889 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.382979+0000", 2024-09-17T07:15:23.890 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999986 2024-09-17T07:15:23.890 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.890 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.890 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:15:23.890 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.383002+0000", 2024-09-17T07:15:23.890 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3292999999999999e-05 2024-09-17T07:15:23.890 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.890 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.890 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:15:23.890 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.383011+0000", 2024-09-17T07:15:23.890 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5680000000000005e-06 2024-09-17T07:15:23.890 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.891 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.891 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:15:23.891 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.383018+0000", 2024-09-17T07:15:23.891 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5350000000000001e-06 2024-09-17T07:15:23.891 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.891 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.891 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:15:23.891 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.392305+0000", 2024-09-17T07:15:23.891 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0092874080000000005 2024-09-17T07:15:23.891 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.892 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.892 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op must wait for map", 2024-09-17T07:15:23.892 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.392312+0000", 2024-09-17T07:15:23.892 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6540000000000002e-06 2024-09-17T07:15:23.892 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.892 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.892 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:15:23.892 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.420628+0000", 2024-09-17T07:15:23.892 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.028316386999999998 2024-09-17T07:15:23.893 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.893 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.893 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-17T07:15:23.893 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.420638+0000", 2024-09-17T07:15:23.893 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.2779999999999999e-06 2024-09-17T07:15:23.893 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.893 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.893 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:15:23.893 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:24.397467+0000", 2024-09-17T07:15:23.893 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.97682960900000004 2024-09-17T07:15:23.893 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.894 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.894 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-17T07:15:23.894 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:24.397480+0000", 2024-09-17T07:15:23.894 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2296000000000001e-05 2024-09-17T07:15:23.894 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.894 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.894 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:15:23.894 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:25.461022+0000", 2024-09-17T07:15:23.894 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.063542405 2024-09-17T07:15:23.895 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.895 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.895 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:15:23.895 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:25.462621+0000", 2024-09-17T07:15:23.895 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0015991709999999999 2024-09-17T07:15:23.895 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:15:23.895 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:15:23.895 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:15:23.895 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.895 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.896 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.24929.0:108585 8.f 8:f6331868:::benchmark_data_smithi100_102803_object108584:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e194)", 2024-09-17T07:15:23.896 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:09:23.383027+0000", 2024-09-17T07:15:23.896 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 360.48957828699997, 2024-09-17T07:15:23.896 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0799240299999999, 2024-09-17T07:15:23.896 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:15:23.896 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-17T07:15:23.896 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:15:23.896 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.24929", 2024-09-17T07:15:23.896 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/564515582", 2024-09-17T07:15:23.896 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 108585 2024-09-17T07:15:23.897 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.897 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:15:23.897 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.897 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:15:23.897 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.383027+0000", 2024-09-17T07:15:23.897 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:15:23.897 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.897 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.897 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:15:23.897 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.383027+0000", 2024-09-17T07:15:23.898 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:15:23.898 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.898 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.898 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:15:23.898 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.383026+0000", 2024-09-17T07:15:23.898 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-17T07:15:23.898 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.898 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.898 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:15:23.898 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.383046+0000", 2024-09-17T07:15:23.898 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.9256999999999999e-05 2024-09-17T07:15:23.898 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.898 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.899 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:15:23.899 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.383052+0000", 2024-09-17T07:15:23.899 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5520000000000001e-06 2024-09-17T07:15:23.899 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.899 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.899 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:15:23.899 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.383058+0000", 2024-09-17T07:15:23.899 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.631e-06 2024-09-17T07:15:23.899 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.899 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.899 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:15:23.899 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.392339+0000", 2024-09-17T07:15:23.899 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0092806099999999999 2024-09-17T07:15:23.900 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.900 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.900 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting_for_map not empty", 2024-09-17T07:15:23.900 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.392343+0000", 2024-09-17T07:15:23.900 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9830000000000004e-06 2024-09-17T07:15:23.900 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.900 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.900 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:15:23.900 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.420668+0000", 2024-09-17T07:15:23.900 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.028324479 2024-09-17T07:15:23.900 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.900 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.900 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-17T07:15:23.901 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.420685+0000", 2024-09-17T07:15:23.901 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.6784999999999999e-05 2024-09-17T07:15:23.901 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.901 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.901 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:15:23.901 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:24.400655+0000", 2024-09-17T07:15:23.901 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.97997047100000001 2024-09-17T07:15:23.901 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.901 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.901 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-17T07:15:23.901 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:24.400666+0000", 2024-09-17T07:15:23.901 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0876000000000001e-05 2024-09-17T07:15:23.902 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.902 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.902 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:15:23.902 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:25.462891+0000", 2024-09-17T07:15:23.902 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.062224788 2024-09-17T07:15:23.902 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.902 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.902 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:15:23.902 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:25.462951+0000", 2024-09-17T07:15:23.903 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0152999999999998e-05 2024-09-17T07:15:23.903 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:15:23.903 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:15:23.903 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:15:23.903 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.903 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.903 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.24929.0:108587 8.f 8:f0b91aee:::benchmark_data_smithi100_102803_object108586:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e194)", 2024-09-17T07:15:23.903 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:09:23.383062+0000", 2024-09-17T07:15:23.903 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 360.48954281200002, 2024-09-17T07:15:23.903 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0800082120000001, 2024-09-17T07:15:23.903 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:15:23.903 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-17T07:15:23.903 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:15:23.904 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.24929", 2024-09-17T07:15:23.904 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/564515582", 2024-09-17T07:15:23.904 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 108587 2024-09-17T07:15:23.904 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.904 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:15:23.904 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.904 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:15:23.904 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.383062+0000", 2024-09-17T07:15:23.904 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:15:23.904 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.904 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.904 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:15:23.905 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.383062+0000", 2024-09-17T07:15:23.905 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:15:23.905 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.905 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.905 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:15:23.905 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.383062+0000", 2024-09-17T07:15:23.905 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-17T07:15:23.905 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.905 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.905 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:15:23.905 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.383109+0000", 2024-09-17T07:15:23.905 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7175000000000001e-05 2024-09-17T07:15:23.905 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.906 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.906 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:15:23.906 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.383115+0000", 2024-09-17T07:15:23.906 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1990000000000003e-06 2024-09-17T07:15:23.906 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.906 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.906 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:15:23.906 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.383120+0000", 2024-09-17T07:15:23.906 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9269999999999996e-06 2024-09-17T07:15:23.906 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.906 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.906 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:15:23.907 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.420712+0000", 2024-09-17T07:15:23.907 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.037591319999999998 2024-09-17T07:15:23.907 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.907 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.907 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-17T07:15:23.907 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.420718+0000", 2024-09-17T07:15:23.907 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6139999999999997e-06 2024-09-17T07:15:23.907 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.907 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.907 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:15:23.907 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:24.400687+0000", 2024-09-17T07:15:23.907 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.97996898399999999 2024-09-17T07:15:23.907 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.907 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.908 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-17T07:15:23.908 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:24.400695+0000", 2024-09-17T07:15:23.908 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4780000000000002e-06 2024-09-17T07:15:23.908 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.908 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.908 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:15:23.908 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:25.462994+0000", 2024-09-17T07:15:23.908 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0622987770000001 2024-09-17T07:15:23.908 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.908 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.908 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:15:23.908 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:25.463071+0000", 2024-09-17T07:15:23.908 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7152999999999998e-05 2024-09-17T07:15:23.909 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:15:23.909 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:15:23.909 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:15:23.909 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.909 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.909 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.24929.0:108590 8.f 8:f0e823d3:::benchmark_data_smithi100_102803_object108589:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e194)", 2024-09-17T07:15:23.909 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:09:23.383147+0000", 2024-09-17T07:15:23.909 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 360.48945800500002, 2024-09-17T07:15:23.909 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0800091310000002, 2024-09-17T07:15:23.909 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:15:23.909 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-17T07:15:23.909 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:15:23.909 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.24929", 2024-09-17T07:15:23.910 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/564515582", 2024-09-17T07:15:23.910 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 108590 2024-09-17T07:15:23.910 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.910 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:15:23.910 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.910 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:15:23.910 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.383147+0000", 2024-09-17T07:15:23.910 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:15:23.910 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.910 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.910 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:15:23.910 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.383147+0000", 2024-09-17T07:15:23.911 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:15:23.911 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.912 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.912 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:15:23.912 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.383147+0000", 2024-09-17T07:15:23.912 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-17T07:15:23.912 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.912 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.912 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:15:23.912 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.383191+0000", 2024-09-17T07:15:23.912 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4088000000000002e-05 2024-09-17T07:15:23.912 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.912 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.913 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:15:23.913 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.383197+0000", 2024-09-17T07:15:23.913 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.336e-06 2024-09-17T07:15:23.913 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.913 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.913 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:15:23.913 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.383202+0000", 2024-09-17T07:15:23.913 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1100000000000002e-06 2024-09-17T07:15:23.913 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.913 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.913 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:15:23.913 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.420760+0000", 2024-09-17T07:15:23.913 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.037558151999999997 2024-09-17T07:15:23.914 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.914 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.914 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-17T07:15:23.914 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.420768+0000", 2024-09-17T07:15:23.914 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2690000000000002e-06 2024-09-17T07:15:23.914 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.914 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.914 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:15:23.914 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:24.400714+0000", 2024-09-17T07:15:23.914 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.979946655 2024-09-17T07:15:23.914 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.914 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.914 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-17T07:15:23.915 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:24.400722+0000", 2024-09-17T07:15:23.915 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5259999999999996e-06 2024-09-17T07:15:23.915 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.915 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.915 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:15:23.915 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:25.463092+0000", 2024-09-17T07:15:23.915 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0623703069999999 2024-09-17T07:15:23.915 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.915 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.915 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:15:23.915 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:25.463156+0000", 2024-09-17T07:15:23.915 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4221999999999996e-05 2024-09-17T07:15:23.915 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:15:23.916 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:15:23.916 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:15:23.916 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.916 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.916 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.24929.0:108599 8.7 8:e4c6d75b:::benchmark_data_smithi100_102803_object108598:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-09-17T07:15:23.916 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:09:23.403863+0000", 2024-09-17T07:15:23.916 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 360.46874268599998, 2024-09-17T07:15:23.916 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.112761989, 2024-09-17T07:15:23.916 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:15:23.916 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:15:23.916 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:15:23.916 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.24929", 2024-09-17T07:15:23.917 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/564515582", 2024-09-17T07:15:23.917 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 108599 2024-09-17T07:15:23.917 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.917 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:15:23.917 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.917 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:15:23.917 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.403863+0000", 2024-09-17T07:15:23.917 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:15:23.917 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.917 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.917 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:15:23.917 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.403863+0000", 2024-09-17T07:15:23.917 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:15:23.918 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.918 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.918 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:15:23.918 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.403861+0000", 2024-09-17T07:15:23.918 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999986 2024-09-17T07:15:23.918 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.918 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.918 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:15:23.918 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.403891+0000", 2024-09-17T07:15:23.918 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9487000000000001e-05 2024-09-17T07:15:23.918 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.918 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.918 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:15:23.919 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.406808+0000", 2024-09-17T07:15:23.919 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00291779 2024-09-17T07:15:23.919 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.919 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.919 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:15:23.919 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.406820+0000", 2024-09-17T07:15:23.919 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1847e-05 2024-09-17T07:15:23.919 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.919 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.919 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:15:23.919 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.420739+0000", 2024-09-17T07:15:23.919 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.013919000000000001 2024-09-17T07:15:23.919 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.920 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.920 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-17T07:15:23.920 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.420757+0000", 2024-09-17T07:15:23.920 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7499999999999998e-05 2024-09-17T07:15:23.920 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.920 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.920 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:15:23.920 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:24.400440+0000", 2024-09-17T07:15:23.920 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.979683462 2024-09-17T07:15:23.920 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.920 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.920 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-17T07:15:23.920 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:24.400448+0000", 2024-09-17T07:15:23.921 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2660000000000001e-06 2024-09-17T07:15:23.921 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.921 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.921 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:15:23.921 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:25.494556+0000", 2024-09-17T07:15:23.921 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.094107838 2024-09-17T07:15:23.921 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.921 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.921 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:15:23.921 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:25.494665+0000", 2024-09-17T07:15:23.921 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00010876 2024-09-17T07:15:23.921 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.921 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.922 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 2,3,6", 2024-09-17T07:15:23.922 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:25.495261+0000", 2024-09-17T07:15:23.922 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00059620799999999998 2024-09-17T07:15:23.922 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.922 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.922 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:15:23.922 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:25.507051+0000", 2024-09-17T07:15:23.922 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.011789624 2024-09-17T07:15:23.922 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.922 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.922 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:15:23.922 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:25.507064+0000", 2024-09-17T07:15:23.922 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2816000000000001e-05 2024-09-17T07:15:23.923 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.923 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.923 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:15:23.923 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:25.514254+0000", 2024-09-17T07:15:23.923 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.007190498 2024-09-17T07:15:23.923 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.923 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.923 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:15:23.923 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:25.516389+0000", 2024-09-17T07:15:23.923 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0021347179999999999 2024-09-17T07:15:23.923 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.924 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.924 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:15:23.924 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:25.516492+0000", 2024-09-17T07:15:23.924 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000103299 2024-09-17T07:15:23.924 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.924 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.924 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:15:23.924 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:25.516625+0000", 2024-09-17T07:15:23.924 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000132395 2024-09-17T07:15:23.924 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:15:23.925 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:15:23.925 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:15:23.925 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.925 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.925 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.24929.0:108600 8.f 8:f10e8358:::benchmark_data_smithi100_102803_object108599:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-09-17T07:15:23.925 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:09:23.406831+0000", 2024-09-17T07:15:23.925 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 360.46577432800001, 2024-09-17T07:15:23.925 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0873082639999998, 2024-09-17T07:15:23.925 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:15:23.925 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:15:23.925 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:15:23.925 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.24929", 2024-09-17T07:15:23.925 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/564515582", 2024-09-17T07:15:23.926 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 108600 2024-09-17T07:15:23.926 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.926 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:15:23.926 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.926 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:15:23.926 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.406831+0000", 2024-09-17T07:15:23.926 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:15:23.926 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.926 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.927 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:15:23.927 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.406831+0000", 2024-09-17T07:15:23.927 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:15:23.927 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.927 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.927 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:15:23.927 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.406830+0000", 2024-09-17T07:15:23.927 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-17T07:15:23.927 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.927 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.927 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:15:23.927 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.406863+0000", 2024-09-17T07:15:23.928 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3556999999999997e-05 2024-09-17T07:15:23.928 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.928 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.928 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:15:23.928 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.406873+0000", 2024-09-17T07:15:23.928 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.1649999999999995e-06 2024-09-17T07:15:23.928 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.928 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.928 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:15:23.929 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.406878+0000", 2024-09-17T07:15:23.929 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.626e-06 2024-09-17T07:15:23.929 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.929 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.929 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:15:23.929 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.420793+0000", 2024-09-17T07:15:23.929 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.013915214 2024-09-17T07:15:23.929 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.929 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.929 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-17T07:15:23.929 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.420801+0000", 2024-09-17T07:15:23.930 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8080000000000005e-06 2024-09-17T07:15:23.930 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.930 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.930 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:15:23.930 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:24.400743+0000", 2024-09-17T07:15:23.930 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.97994139300000005 2024-09-17T07:15:23.930 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.930 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.930 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-17T07:15:23.931 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:24.400750+0000", 2024-09-17T07:15:23.931 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1849999999999998e-06 2024-09-17T07:15:23.931 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.931 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.931 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:15:23.931 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:25.463357+0000", 2024-09-17T07:15:23.931 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.062607522 2024-09-17T07:15:23.931 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.931 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.931 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:15:23.931 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:25.463947+0000", 2024-09-17T07:15:23.932 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00058976299999999998 2024-09-17T07:15:23.932 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.932 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.932 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 2,3,4", 2024-09-17T07:15:23.932 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:25.464250+0000", 2024-09-17T07:15:23.932 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00030318800000000003 2024-09-17T07:15:23.932 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.932 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.932 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:15:23.932 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:25.473223+0000", 2024-09-17T07:15:23.933 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0089722799999999991 2024-09-17T07:15:23.933 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.933 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.933 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:15:23.933 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:25.476258+0000", 2024-09-17T07:15:23.933 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0030358590000000001 2024-09-17T07:15:23.933 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.933 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.933 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:15:23.934 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:25.478274+0000", 2024-09-17T07:15:23.934 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0020160780000000001 2024-09-17T07:15:23.934 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.934 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.934 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:15:23.934 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:25.493956+0000", 2024-09-17T07:15:23.934 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.015681106 2024-09-17T07:15:23.934 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.935 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.935 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:15:23.935 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:25.494013+0000", 2024-09-17T07:15:23.935 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7352000000000002e-05 2024-09-17T07:15:23.935 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.935 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.935 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:15:23.935 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:25.494139+0000", 2024-09-17T07:15:23.935 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012631900000000001 2024-09-17T07:15:23.935 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:15:23.936 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:15:23.936 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:15:23.936 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.936 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.936 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.24929.0:108601 8.f 8:fb3ae19a:::benchmark_data_smithi100_102803_object108600:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-09-17T07:15:23.936 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:09:23.407878+0000", 2024-09-17T07:15:23.936 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 360.46472749200001, 2024-09-17T07:15:23.936 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.086463835, 2024-09-17T07:15:23.936 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:15:23.936 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:15:23.936 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:15:23.937 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.24929", 2024-09-17T07:15:23.937 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/564515582", 2024-09-17T07:15:23.937 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 108601 2024-09-17T07:15:23.937 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.937 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:15:23.937 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.937 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:15:23.937 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.407878+0000", 2024-09-17T07:15:23.937 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:15:23.937 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.937 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.937 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:15:23.937 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.407878+0000", 2024-09-17T07:15:23.938 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:15:23.938 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.938 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.938 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:15:23.938 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.407876+0000", 2024-09-17T07:15:23.938 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999986 2024-09-17T07:15:23.938 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.938 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.938 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:15:23.938 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.407929+0000", 2024-09-17T07:15:23.938 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2711999999999999e-05 2024-09-17T07:15:23.938 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.938 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.939 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:15:23.939 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.407940+0000", 2024-09-17T07:15:23.939 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.063e-05 2024-09-17T07:15:23.939 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.939 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.939 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:15:23.939 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.407947+0000", 2024-09-17T07:15:23.939 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.413e-06 2024-09-17T07:15:23.939 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.939 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.939 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:15:23.939 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.420826+0000", 2024-09-17T07:15:23.940 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.012878367 2024-09-17T07:15:23.940 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.940 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.940 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-17T07:15:23.940 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.420832+0000", 2024-09-17T07:15:23.940 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7560000000000002e-06 2024-09-17T07:15:23.940 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.940 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.940 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:15:23.941 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:24.400769+0000", 2024-09-17T07:15:23.941 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.97993688000000001 2024-09-17T07:15:23.941 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.941 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.941 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-17T07:15:23.941 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:24.400776+0000", 2024-09-17T07:15:23.941 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8569999999999997e-06 2024-09-17T07:15:23.941 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.941 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.941 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:15:23.941 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:25.470203+0000", 2024-09-17T07:15:23.941 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.069427106 2024-09-17T07:15:23.941 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.942 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.942 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:15:23.942 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:25.470342+0000", 2024-09-17T07:15:23.942 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00013921699999999999 2024-09-17T07:15:23.942 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.942 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.942 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 2,3,4", 2024-09-17T07:15:23.942 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:25.470516+0000", 2024-09-17T07:15:23.942 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00017379599999999999 2024-09-17T07:15:23.942 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.942 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.942 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:15:23.942 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:25.478306+0000", 2024-09-17T07:15:23.943 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0077898109999999998 2024-09-17T07:15:23.943 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.943 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.943 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:15:23.943 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:25.478976+0000", 2024-09-17T07:15:23.943 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00067018600000000002 2024-09-17T07:15:23.943 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.943 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.943 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:15:23.943 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:25.480110+0000", 2024-09-17T07:15:23.943 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0011333840000000001 2024-09-17T07:15:23.943 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.943 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.944 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:15:23.944 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:25.494180+0000", 2024-09-17T07:15:23.944 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.014070918 2024-09-17T07:15:23.944 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.944 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.944 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:15:23.944 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:25.494229+0000", 2024-09-17T07:15:23.944 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8887000000000001e-05 2024-09-17T07:15:23.944 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.944 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.944 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:15:23.944 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:25.494342+0000", 2024-09-17T07:15:23.945 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00011229 2024-09-17T07:15:23.945 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:15:23.945 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:15:23.945 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:15:23.945 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.945 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.945 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.24929.0:108602 8.f 8:fff6ec9d:::benchmark_data_smithi100_102803_object108601:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-09-17T07:15:23.945 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:09:23.407954+0000", 2024-09-17T07:15:23.945 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 360.46465138899998, 2024-09-17T07:15:23.945 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0866440530000001, 2024-09-17T07:15:23.945 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:15:23.945 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:15:23.945 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:15:23.946 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.24929", 2024-09-17T07:15:23.946 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/564515582", 2024-09-17T07:15:23.946 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 108602 2024-09-17T07:15:23.946 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.946 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:15:23.946 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.946 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:15:23.946 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.407954+0000", 2024-09-17T07:15:23.946 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:15:23.946 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.946 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.946 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:15:23.947 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.407954+0000", 2024-09-17T07:15:23.947 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:15:23.947 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.947 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.947 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:15:23.947 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.407953+0000", 2024-09-17T07:15:23.947 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-17T07:15:23.947 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.947 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.947 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:15:23.947 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.407997+0000", 2024-09-17T07:15:23.947 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3859000000000001e-05 2024-09-17T07:15:23.947 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.948 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.948 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:15:23.948 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.408004+0000", 2024-09-17T07:15:23.948 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6309999999999997e-06 2024-09-17T07:15:23.948 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.948 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.948 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:15:23.948 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.408009+0000", 2024-09-17T07:15:23.948 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3770000000000004e-06 2024-09-17T07:15:23.948 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.948 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.948 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:15:23.949 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.420856+0000", 2024-09-17T07:15:23.949 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0128468 2024-09-17T07:15:23.949 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.949 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.949 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-17T07:15:23.949 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.420863+0000", 2024-09-17T07:15:23.949 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.216e-06 2024-09-17T07:15:23.949 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.949 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.949 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:15:23.949 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:24.400796+0000", 2024-09-17T07:15:23.949 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.97993322400000005 2024-09-17T07:15:23.949 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.950 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.950 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-17T07:15:23.950 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:24.400820+0000", 2024-09-17T07:15:23.950 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4015999999999998e-05 2024-09-17T07:15:23.950 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.950 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.950 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:15:23.950 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:25.473502+0000", 2024-09-17T07:15:23.950 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0726811599999999 2024-09-17T07:15:23.950 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.950 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.950 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:15:23.950 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:25.473667+0000", 2024-09-17T07:15:23.951 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016558299999999999 2024-09-17T07:15:23.951 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.951 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.951 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 2,3,4", 2024-09-17T07:15:23.951 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:25.473854+0000", 2024-09-17T07:15:23.951 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000186761 2024-09-17T07:15:23.951 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.951 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.951 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:15:23.951 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:25.478446+0000", 2024-09-17T07:15:23.951 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0045921080000000001 2024-09-17T07:15:23.951 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.952 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.952 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:15:23.952 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:25.478982+0000", 2024-09-17T07:15:23.952 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000536281 2024-09-17T07:15:23.952 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.952 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.952 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:15:23.952 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:25.480596+0000", 2024-09-17T07:15:23.952 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0016133549999999999 2024-09-17T07:15:23.952 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.952 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.952 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:15:23.952 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:25.494386+0000", 2024-09-17T07:15:23.953 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.013790257 2024-09-17T07:15:23.953 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.953 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.953 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:15:23.953 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:25.494460+0000", 2024-09-17T07:15:23.953 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4189000000000004e-05 2024-09-17T07:15:23.953 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.953 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.953 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:15:23.953 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:25.494598+0000", 2024-09-17T07:15:23.953 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00013783299999999999 2024-09-17T07:15:23.953 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:15:23.953 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:15:23.954 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:15:23.954 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.954 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.954 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.24929.0:108615 8.7 8:e66dae25:::benchmark_data_smithi100_102803_object108614:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-09-17T07:15:23.954 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:09:23.429585+0000", 2024-09-17T07:15:23.954 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 360.44302055000003, 2024-09-17T07:15:23.954 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.087252238, 2024-09-17T07:15:23.954 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:15:23.954 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:15:23.954 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:15:23.954 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.24929", 2024-09-17T07:15:23.954 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/564515582", 2024-09-17T07:15:23.954 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 108615 2024-09-17T07:15:23.955 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.955 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:15:23.955 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.955 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:15:23.955 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.429585+0000", 2024-09-17T07:15:23.955 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:15:23.955 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.955 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.955 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:15:23.955 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.429585+0000", 2024-09-17T07:15:23.955 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:15:23.955 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.955 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.956 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:15:23.956 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.429583+0000", 2024-09-17T07:15:23.956 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999986 2024-09-17T07:15:23.956 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.956 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.956 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:15:23.956 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.429609+0000", 2024-09-17T07:15:23.956 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.6129e-05 2024-09-17T07:15:23.956 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.956 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.956 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:15:23.956 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.429619+0000", 2024-09-17T07:15:23.956 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.8949999999999993e-06 2024-09-17T07:15:23.957 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.957 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.957 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:15:23.957 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.429626+0000", 2024-09-17T07:15:23.957 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5100000000000004e-06 2024-09-17T07:15:23.957 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.957 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.957 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:15:23.957 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.429674+0000", 2024-09-17T07:15:23.957 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8262e-05 2024-09-17T07:15:23.957 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.957 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.957 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-17T07:15:23.958 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:23.429686+0000", 2024-09-17T07:15:23.958 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2179e-05 2024-09-17T07:15:23.958 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.958 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.958 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:15:23.958 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:24.400860+0000", 2024-09-17T07:15:23.958 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.97117404600000001 2024-09-17T07:15:23.958 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.958 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.958 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-17T07:15:23.958 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:24.400871+0000", 2024-09-17T07:15:23.958 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1025e-05 2024-09-17T07:15:23.958 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.959 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.959 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:15:23.959 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:25.503333+0000", 2024-09-17T07:15:23.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:23.960 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:23.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:15:23.961 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1024616329999999 2024-09-17T07:15:23.961 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.961 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.961 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:15:23.962 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:25.503445+0000", 2024-09-17T07:15:23.962 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000111779 2024-09-17T07:15:23.962 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.962 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.962 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 2,3,6", 2024-09-17T07:15:23.962 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:25.503621+0000", 2024-09-17T07:15:23.962 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000175985 2024-09-17T07:15:23.962 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.963 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.963 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:15:23.963 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:25.507080+0000", 2024-09-17T07:15:23.963 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0034589849999999999 2024-09-17T07:15:23.963 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.963 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.963 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:15:23.963 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:25.514346+0000", 2024-09-17T07:15:23.963 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0072661449999999999 2024-09-17T07:15:23.963 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.963 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.963 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:15:23.964 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:25.514581+0000", 2024-09-17T07:15:23.964 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000234995 2024-09-17T07:15:23.964 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.964 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.964 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:15:23.964 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:25.516686+0000", 2024-09-17T07:15:23.964 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0021048289999999999 2024-09-17T07:15:23.964 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.964 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.964 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:15:23.964 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:25.516743+0000", 2024-09-17T07:15:23.964 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6789000000000001e-05 2024-09-17T07:15:23.965 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.965 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.965 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:15:23.965 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:09:25.516837+0000", 2024-09-17T07:15:23.965 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.4476999999999997e-05 2024-09-17T07:15:23.965 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:15:23.965 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:15:23.965 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:15:23.965 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.965 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.965 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.24929.0:211519 8.f 8:f76f566c:::benchmark_data_smithi100_102803_object81015:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e218)", 2024-09-17T07:15:23.965 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:10:59.187009+0000", 2024-09-17T07:15:23.965 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 264.68559615100003, 2024-09-17T07:15:23.966 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.032661445, 2024-09-17T07:15:23.966 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:15:23.966 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-17T07:15:23.966 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:15:23.966 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.24929", 2024-09-17T07:15:23.966 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/564515582", 2024-09-17T07:15:23.966 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 211519 2024-09-17T07:15:23.966 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.966 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:15:23.966 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.966 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:15:23.966 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.187009+0000", 2024-09-17T07:15:23.966 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:15:23.967 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.967 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.967 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:15:23.967 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.187009+0000", 2024-09-17T07:15:23.967 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:15:23.967 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.967 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.967 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:15:23.967 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.187008+0000", 2024-09-17T07:15:23.967 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-17T07:15:23.967 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.967 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.968 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:15:23.968 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.187014+0000", 2024-09-17T07:15:23.968 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3369999999999999e-06 2024-09-17T07:15:23.968 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.968 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.968 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:15:23.968 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.187022+0000", 2024-09-17T07:15:23.968 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3149999999999992e-06 2024-09-17T07:15:23.968 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.968 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.968 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:15:23.968 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.187028+0000", 2024-09-17T07:15:23.968 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.353e-06 2024-09-17T07:15:23.969 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.969 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.969 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:15:23.969 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.187070+0000", 2024-09-17T07:15:23.969 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1405000000000001e-05 2024-09-17T07:15:23.969 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.969 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.969 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:15:23.969 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.187268+0000", 2024-09-17T07:15:23.969 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000197775 2024-09-17T07:15:23.969 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.969 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.969 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 2,3,4", 2024-09-17T07:15:23.970 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.187403+0000", 2024-09-17T07:15:23.970 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000135372 2024-09-17T07:15:23.970 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.970 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.970 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:15:23.970 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.190220+0000", 2024-09-17T07:15:23.970 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0028173209999999998 2024-09-17T07:15:23.970 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.970 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.970 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:15:23.970 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.200079+0000", 2024-09-17T07:15:23.970 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0098588530000000008 2024-09-17T07:15:23.971 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.971 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.971 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-17T07:15:23.971 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.200095+0000", 2024-09-17T07:15:23.971 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5463000000000001e-05 2024-09-17T07:15:23.971 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.971 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.971 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:15:23.971 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:11:00.219598+0000", 2024-09-17T07:15:23.971 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0195031109999999 2024-09-17T07:15:23.971 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.971 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.971 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:15:23.972 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:11:00.219671+0000", 2024-09-17T07:15:23.972 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2964000000000001e-05 2024-09-17T07:15:23.972 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:15:23.972 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:15:23.972 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:15:23.972 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.972 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.972 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.24929.0:211550 8.f 8:f6793428:::benchmark_data_smithi100_102803_object81046:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e219)", 2024-09-17T07:15:23.972 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:10:59.219639+0000", 2024-09-17T07:15:23.972 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 264.65296590200001, 2024-09-17T07:15:23.972 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.007437761, 2024-09-17T07:15:23.972 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:15:23.972 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:15:23.973 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:15:23.973 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.24929", 2024-09-17T07:15:23.973 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/564515582", 2024-09-17T07:15:23.973 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 211550 2024-09-17T07:15:23.973 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.973 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:15:23.973 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.973 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:15:23.973 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.219639+0000", 2024-09-17T07:15:23.973 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:15:23.973 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.973 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.974 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:15:23.974 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.219639+0000", 2024-09-17T07:15:23.974 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:15:23.974 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.974 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.974 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:15:23.974 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.219638+0000", 2024-09-17T07:15:23.974 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-17T07:15:23.974 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.974 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.974 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:15:23.974 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.219644+0000", 2024-09-17T07:15:23.974 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7069999999999997e-06 2024-09-17T07:15:23.974 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.975 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.975 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:15:23.975 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.219656+0000", 2024-09-17T07:15:23.975 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.154e-05 2024-09-17T07:15:23.975 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.975 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.975 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:15:23.975 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.219661+0000", 2024-09-17T07:15:23.975 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1159999999999998e-06 2024-09-17T07:15:23.975 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.975 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.975 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:15:23.976 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.219695+0000", 2024-09-17T07:15:23.976 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3952000000000003e-05 2024-09-17T07:15:23.976 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.976 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.976 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-17T07:15:23.976 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.219701+0000", 2024-09-17T07:15:23.976 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5119999999999997e-06 2024-09-17T07:15:23.976 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.976 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.976 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:15:23.976 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:11:00.219995+0000", 2024-09-17T07:15:23.976 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0002942939999999 2024-09-17T07:15:23.976 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.977 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.977 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:15:23.977 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:11:00.220114+0000", 2024-09-17T07:15:23.977 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000118808 2024-09-17T07:15:23.977 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.977 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.977 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 2,4", 2024-09-17T07:15:23.977 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:11:00.220200+0000", 2024-09-17T07:15:23.977 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6199999999999995e-05 2024-09-17T07:15:23.977 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.977 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.977 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:15:23.978 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:11:00.226930+0000", 2024-09-17T07:15:23.978 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0067294090000000004 2024-09-17T07:15:23.978 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.978 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.978 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:15:23.978 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:11:00.226984+0000", 2024-09-17T07:15:23.978 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3825000000000001e-05 2024-09-17T07:15:23.978 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.978 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.978 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:15:23.978 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:11:00.227010+0000", 2024-09-17T07:15:23.978 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.6038e-05 2024-09-17T07:15:23.979 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.979 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.979 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:15:23.979 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:11:00.227039+0000", 2024-09-17T07:15:23.979 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9601e-05 2024-09-17T07:15:23.979 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.979 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.979 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:15:23.979 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:11:00.227077+0000", 2024-09-17T07:15:23.979 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7880000000000003e-05 2024-09-17T07:15:23.979 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:15:23.979 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:15:23.980 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:15:23.980 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.980 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.980 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.24929.0:211554 8.f 8:f3602104:::benchmark_data_smithi100_102803_object81050:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e219)", 2024-09-17T07:15:23.980 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:10:59.222856+0000", 2024-09-17T07:15:23.980 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 264.64974906100002, 2024-09-17T07:15:23.980 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0043365790000001, 2024-09-17T07:15:23.980 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:15:23.980 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:15:23.980 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:15:23.980 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.24929", 2024-09-17T07:15:23.980 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/564515582", 2024-09-17T07:15:23.981 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 211554 2024-09-17T07:15:23.981 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.981 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:15:23.981 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.981 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:15:23.981 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.222856+0000", 2024-09-17T07:15:23.981 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:15:23.981 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.981 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.981 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:15:23.981 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.222856+0000", 2024-09-17T07:15:23.981 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:15:23.982 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.982 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.982 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:15:23.982 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.222855+0000", 2024-09-17T07:15:23.982 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-17T07:15:23.982 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.982 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.982 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:15:23.982 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.222861+0000", 2024-09-17T07:15:23.982 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0809999999999999e-06 2024-09-17T07:15:23.982 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.982 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.982 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:15:23.983 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.222873+0000", 2024-09-17T07:15:23.983 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2150000000000001e-05 2024-09-17T07:15:23.983 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.983 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.983 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:15:23.983 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.222879+0000", 2024-09-17T07:15:23.983 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9399999999999999e-06 2024-09-17T07:15:23.983 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.983 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.983 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:15:23.983 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.222926+0000", 2024-09-17T07:15:23.983 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6891999999999998e-05 2024-09-17T07:15:23.983 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.984 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.984 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-17T07:15:23.984 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.222935+0000", 2024-09-17T07:15:23.984 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.969e-06 2024-09-17T07:15:23.984 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.984 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.984 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:15:23.984 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:11:00.222348+0000", 2024-09-17T07:15:23.984 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.99941278 2024-09-17T07:15:23.984 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.984 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.984 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:15:23.985 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:11:00.222477+0000", 2024-09-17T07:15:23.985 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012879999999999999 2024-09-17T07:15:23.985 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.985 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.985 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 2,4", 2024-09-17T07:15:23.985 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:11:00.222601+0000", 2024-09-17T07:15:23.985 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012441399999999999 2024-09-17T07:15:23.985 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.985 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.985 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:15:23.985 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:11:00.226977+0000", 2024-09-17T07:15:23.985 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0043759910000000001 2024-09-17T07:15:23.985 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.986 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.986 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:15:23.986 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:11:00.226987+0000", 2024-09-17T07:15:23.986 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0103e-05 2024-09-17T07:15:23.986 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.986 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.986 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:15:23.986 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:11:00.227139+0000", 2024-09-17T07:15:23.986 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00015154200000000001 2024-09-17T07:15:23.986 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.986 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.986 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:15:23.986 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:11:00.227161+0000", 2024-09-17T07:15:23.987 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2274000000000001e-05 2024-09-17T07:15:23.987 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.987 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.987 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:15:23.987 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:11:00.227193+0000", 2024-09-17T07:15:23.987 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1622999999999999e-05 2024-09-17T07:15:23.987 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:15:23.987 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:15:23.987 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:15:23.987 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.987 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.987 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.24929.0:211561 8.f 8:f1730719:::benchmark_data_smithi100_102803_object81057:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e219)", 2024-09-17T07:15:23.988 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:10:59.224630+0000", 2024-09-17T07:15:23.988 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 264.64797544800001, 2024-09-17T07:15:23.988 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.002674525, 2024-09-17T07:15:23.988 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:15:23.988 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:15:23.988 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:15:23.988 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.24929", 2024-09-17T07:15:23.988 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/564515582", 2024-09-17T07:15:23.988 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 211561 2024-09-17T07:15:23.988 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.988 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:15:23.988 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.988 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:15:23.989 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.224630+0000", 2024-09-17T07:15:23.989 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:15:23.989 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.989 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.989 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:15:23.989 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.224630+0000", 2024-09-17T07:15:23.989 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:15:23.989 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.989 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.989 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:15:23.993 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.224629+0000", 2024-09-17T07:15:23.993 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-17T07:15:23.993 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.993 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.993 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:15:23.993 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.224633+0000", 2024-09-17T07:15:23.993 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7699999999999999e-06 2024-09-17T07:15:23.993 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.993 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.994 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:15:23.994 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.224641+0000", 2024-09-17T07:15:23.994 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.4579999999999993e-06 2024-09-17T07:15:23.994 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.994 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.994 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:15:23.994 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.224647+0000", 2024-09-17T07:15:23.994 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4609999999999999e-06 2024-09-17T07:15:23.994 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.994 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.994 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:15:23.994 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.224683+0000", 2024-09-17T07:15:23.994 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6621000000000001e-05 2024-09-17T07:15:23.995 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.995 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.995 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-17T07:15:23.995 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.224692+0000", 2024-09-17T07:15:23.995 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2949999999999994e-06 2024-09-17T07:15:23.995 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.995 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.995 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:15:23.995 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:11:00.224509+0000", 2024-09-17T07:15:23.995 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.99981752400000001 2024-09-17T07:15:23.995 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.995 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.996 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:15:23.996 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:11:00.224646+0000", 2024-09-17T07:15:23.996 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00013640000000000001 2024-09-17T07:15:23.996 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.996 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.996 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 2,4", 2024-09-17T07:15:23.996 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:11:00.224777+0000", 2024-09-17T07:15:23.996 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000131018 2024-09-17T07:15:23.996 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.996 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.996 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:15:23.996 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:11:00.227107+0000", 2024-09-17T07:15:23.997 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.002330216 2024-09-17T07:15:23.997 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.997 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.997 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:15:23.997 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:11:00.227111+0000", 2024-09-17T07:15:23.997 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1200000000000004e-06 2024-09-17T07:15:23.997 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.997 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.997 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:15:23.997 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:11:00.227237+0000", 2024-09-17T07:15:23.997 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012634000000000001 2024-09-17T07:15:23.997 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.998 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.998 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:15:23.998 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:11:00.227271+0000", 2024-09-17T07:15:23.998 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3331000000000003e-05 2024-09-17T07:15:23.998 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.998 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.998 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:15:23.998 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:11:00.227304+0000", 2024-09-17T07:15:23.998 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3662999999999999e-05 2024-09-17T07:15:23.998 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:15:23.998 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:15:23.998 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:15:23.999 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:23.999 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:23.999 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.24929.0:211568 8.f 8:fbb8deae:::benchmark_data_smithi100_102803_object81064:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e219)", 2024-09-17T07:15:23.999 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:10:59.232824+0000", 2024-09-17T07:15:23.999 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 264.63978152300001, 2024-09-17T07:15:23.999 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.99982723799999995, 2024-09-17T07:15:23.999 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:15:23.999 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:15:23.999 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:15:23.999 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.24929", 2024-09-17T07:15:23.999 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/564515582", 2024-09-17T07:15:23.999 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 211568 2024-09-17T07:15:24.000 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.000 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:15:24.000 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.000 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:15:24.000 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.232824+0000", 2024-09-17T07:15:24.000 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:15:24.000 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.000 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.000 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:15:24.000 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.232824+0000", 2024-09-17T07:15:24.000 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:15:24.000 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.001 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.001 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:15:24.001 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.232822+0000", 2024-09-17T07:15:24.001 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999986 2024-09-17T07:15:24.001 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.001 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.001 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:15:24.001 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.232829+0000", 2024-09-17T07:15:24.001 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0929999999999999e-06 2024-09-17T07:15:24.001 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.001 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.001 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:15:24.002 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.232933+0000", 2024-09-17T07:15:24.002 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00010426500000000001 2024-09-17T07:15:24.002 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.002 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.002 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:15:24.002 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.232942+0000", 2024-09-17T07:15:24.002 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3549999999999994e-06 2024-09-17T07:15:24.002 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.002 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.002 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:15:24.002 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.232992+0000", 2024-09-17T07:15:24.002 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9376000000000003e-05 2024-09-17T07:15:24.002 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.003 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.003 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-17T07:15:24.003 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.233001+0000", 2024-09-17T07:15:24.003 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3160000000000003e-06 2024-09-17T07:15:24.003 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.003 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.003 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:15:24.003 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:11:00.227356+0000", 2024-09-17T07:15:24.003 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.99435491600000003 2024-09-17T07:15:24.003 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.003 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.003 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:15:24.003 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:11:00.227454+0000", 2024-09-17T07:15:24.004 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7807999999999996e-05 2024-09-17T07:15:24.004 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.004 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.004 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 2,4", 2024-09-17T07:15:24.004 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:11:00.227527+0000", 2024-09-17T07:15:24.004 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.3137000000000001e-05 2024-09-17T07:15:24.004 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.004 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.004 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:15:24.004 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:11:00.232485+0000", 2024-09-17T07:15:24.004 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0049583630000000004 2024-09-17T07:15:24.004 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.005 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.005 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:15:24.005 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:11:00.232531+0000", 2024-09-17T07:15:24.005 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5957000000000001e-05 2024-09-17T07:15:24.005 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.005 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.005 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:15:24.005 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:11:00.232590+0000", 2024-09-17T07:15:24.005 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8652e-05 2024-09-17T07:15:24.005 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.005 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.005 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:15:24.005 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:11:00.232625+0000", 2024-09-17T07:15:24.006 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.4820999999999997e-05 2024-09-17T07:15:24.006 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.006 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.006 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:15:24.006 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:11:00.232651+0000", 2024-09-17T07:15:24.006 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.6463000000000001e-05 2024-09-17T07:15:24.006 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:15:24.006 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:15:24.006 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:15:24.006 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.006 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.006 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.24929.0:211579 8.f 8:f56bd06a:::benchmark_data_smithi100_102803_object81075:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e219)", 2024-09-17T07:15:24.006 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:10:59.239309+0000", 2024-09-17T07:15:24.007 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 264.63329670799999, 2024-09-17T07:15:24.007 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.99757258999999998, 2024-09-17T07:15:24.007 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:15:24.007 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:15:24.007 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:15:24.007 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.24929", 2024-09-17T07:15:24.007 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/564515582", 2024-09-17T07:15:24.007 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 211579 2024-09-17T07:15:24.007 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.007 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:15:24.007 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.007 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:15:24.007 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.239309+0000", 2024-09-17T07:15:24.008 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:15:24.008 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.008 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.008 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:15:24.008 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.239309+0000", 2024-09-17T07:15:24.008 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:15:24.008 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.008 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.008 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:15:24.008 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.239307+0000", 2024-09-17T07:15:24.008 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999986 2024-09-17T07:15:24.008 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.008 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.009 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:15:24.009 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.239313+0000", 2024-09-17T07:15:24.009 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0399999999999998e-06 2024-09-17T07:15:24.009 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.009 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.009 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:15:24.009 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.239327+0000", 2024-09-17T07:15:24.009 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4165000000000001e-05 2024-09-17T07:15:24.009 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.009 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.009 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:15:24.009 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.239334+0000", 2024-09-17T07:15:24.009 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0859999999999996e-06 2024-09-17T07:15:24.010 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.010 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.010 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:15:24.010 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.239375+0000", 2024-09-17T07:15:24.010 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0995999999999998e-05 2024-09-17T07:15:24.010 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.010 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.010 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-17T07:15:24.010 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.239384+0000", 2024-09-17T07:15:24.010 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3329999999999997e-06 2024-09-17T07:15:24.010 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.010 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.011 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:15:24.011 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:11:00.229941+0000", 2024-09-17T07:15:24.011 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.99055742999999996 2024-09-17T07:15:24.011 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.011 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.011 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:15:24.011 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:11:00.230078+0000", 2024-09-17T07:15:24.011 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00013677399999999999 2024-09-17T07:15:24.011 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.011 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.011 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 2,4", 2024-09-17T07:15:24.011 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:11:00.230204+0000", 2024-09-17T07:15:24.011 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012565800000000001 2024-09-17T07:15:24.012 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.012 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.012 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:15:24.012 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:11:00.232581+0000", 2024-09-17T07:15:24.012 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.002377592 2024-09-17T07:15:24.012 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.012 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.012 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:15:24.012 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:11:00.236761+0000", 2024-09-17T07:15:24.012 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0041795979999999996 2024-09-17T07:15:24.012 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.012 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.012 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:15:24.013 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:11:00.236795+0000", 2024-09-17T07:15:24.013 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.4177000000000002e-05 2024-09-17T07:15:24.013 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.013 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.013 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:15:24.013 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:11:00.236843+0000", 2024-09-17T07:15:24.013 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.774e-05 2024-09-17T07:15:24.013 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.013 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.013 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:15:24.013 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:11:00.236881+0000", 2024-09-17T07:15:24.013 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8420000000000001e-05 2024-09-17T07:15:24.014 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:15:24.014 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:15:24.014 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:15:24.014 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.014 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.014 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.24929.0:211597 8.f 8:f99f2f81:::benchmark_data_smithi100_102803_object81093:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e219)", 2024-09-17T07:15:24.014 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:10:59.256810+0000", 2024-09-17T07:15:24.014 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 264.61579572199997, 2024-09-17T07:15:24.014 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.98020336600000002, 2024-09-17T07:15:24.014 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:15:24.014 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:15:24.014 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:15:24.014 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.24929", 2024-09-17T07:15:24.015 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/564515582", 2024-09-17T07:15:24.015 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 211597 2024-09-17T07:15:24.015 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.015 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:15:24.015 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.015 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:15:24.015 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.256810+0000", 2024-09-17T07:15:24.015 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:15:24.015 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.015 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.015 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:15:24.015 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.256810+0000", 2024-09-17T07:15:24.015 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:15:24.016 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.016 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.016 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:15:24.016 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.256808+0000", 2024-09-17T07:15:24.016 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-17T07:15:24.016 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.016 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.016 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:15:24.016 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.256813+0000", 2024-09-17T07:15:24.016 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7820000000000001e-06 2024-09-17T07:15:24.016 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.016 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.016 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:15:24.017 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.256821+0000", 2024-09-17T07:15:24.017 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1170000000000004e-06 2024-09-17T07:15:24.017 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.017 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.017 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:15:24.017 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.256828+0000", 2024-09-17T07:15:24.017 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9489999999999999e-06 2024-09-17T07:15:24.017 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.017 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.017 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:15:24.017 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.256869+0000", 2024-09-17T07:15:24.017 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0688000000000001e-05 2024-09-17T07:15:24.017 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.018 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.018 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-17T07:15:24.018 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.256877+0000", 2024-09-17T07:15:24.018 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.14e-06 2024-09-17T07:15:24.018 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.018 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.018 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:15:24.018 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:11:00.232681+0000", 2024-09-17T07:15:24.018 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.97580388699999998 2024-09-17T07:15:24.018 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.018 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.018 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:15:24.018 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:11:00.232787+0000", 2024-09-17T07:15:24.019 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00010566299999999999 2024-09-17T07:15:24.019 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.019 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.019 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 2,4", 2024-09-17T07:15:24.019 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:11:00.232858+0000", 2024-09-17T07:15:24.019 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1070999999999998e-05 2024-09-17T07:15:24.019 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.019 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.019 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:15:24.019 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:11:00.236891+0000", 2024-09-17T07:15:24.019 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0040330310000000003 2024-09-17T07:15:24.019 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.020 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.020 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:15:24.020 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:11:00.236921+0000", 2024-09-17T07:15:24.020 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0122000000000001e-05 2024-09-17T07:15:24.020 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.020 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.020 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:15:24.020 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:11:00.236950+0000", 2024-09-17T07:15:24.020 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9496999999999999e-05 2024-09-17T07:15:24.020 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.020 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.020 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:15:24.020 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:11:00.236983+0000", 2024-09-17T07:15:24.021 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.2608999999999999e-05 2024-09-17T07:15:24.021 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.021 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.021 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:15:24.021 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:11:00.237013+0000", 2024-09-17T07:15:24.021 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9901000000000001e-05 2024-09-17T07:15:24.021 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:15:24.021 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:15:24.021 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:15:24.021 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.021 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.022 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.24929.0:211640 8.f 8:fe4059ac:::benchmark_data_smithi100_102803_object81136:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e219)", 2024-09-17T07:15:24.022 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:10:59.291168+0000", 2024-09-17T07:15:24.022 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 264.58143720599998, 2024-09-17T07:15:24.022 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.95191181800000002, 2024-09-17T07:15:24.022 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:15:24.022 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:15:24.022 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:15:24.022 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.24929", 2024-09-17T07:15:24.022 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/564515582", 2024-09-17T07:15:24.022 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 211640 2024-09-17T07:15:24.022 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.022 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:15:24.022 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.023 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:15:24.023 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.291168+0000", 2024-09-17T07:15:24.023 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:15:24.023 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.023 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.023 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:15:24.023 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.291168+0000", 2024-09-17T07:15:24.023 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:15:24.023 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.023 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.023 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:15:24.023 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.291167+0000", 2024-09-17T07:15:24.023 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-17T07:15:24.024 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.024 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.024 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:15:24.024 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.291172+0000", 2024-09-17T07:15:24.024 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3290000000000001e-06 2024-09-17T07:15:24.024 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.024 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.024 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:15:24.024 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.291182+0000", 2024-09-17T07:15:24.024 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7990000000000005e-06 2024-09-17T07:15:24.024 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.024 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.025 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:15:24.025 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.291189+0000", 2024-09-17T07:15:24.025 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8220000000000001e-06 2024-09-17T07:15:24.025 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.025 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.025 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:15:24.025 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.291226+0000", 2024-09-17T07:15:24.025 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6465999999999999e-05 2024-09-17T07:15:24.025 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.025 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.025 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-17T07:15:24.025 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.291233+0000", 2024-09-17T07:15:24.025 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4370000000000001e-06 2024-09-17T07:15:24.026 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.026 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.026 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:15:24.026 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:11:00.234697+0000", 2024-09-17T07:15:24.027 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.94346365200000004 2024-09-17T07:15:24.027 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.027 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.027 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:15:24.027 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:11:00.234833+0000", 2024-09-17T07:15:24.027 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00013602699999999999 2024-09-17T07:15:24.027 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.027 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.027 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 2,4", 2024-09-17T07:15:24.027 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:11:00.234980+0000", 2024-09-17T07:15:24.027 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014682200000000001 2024-09-17T07:15:24.027 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.028 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.028 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:15:24.028 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:11:00.242914+0000", 2024-09-17T07:15:24.028 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0079340929999999997 2024-09-17T07:15:24.028 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.028 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.028 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:15:24.028 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:11:00.242965+0000", 2024-09-17T07:15:24.028 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1150000000000003e-05 2024-09-17T07:15:24.028 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.028 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.028 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:15:24.029 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:11:00.242975+0000", 2024-09-17T07:15:24.029 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.009e-05 2024-09-17T07:15:24.029 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.029 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.029 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:15:24.029 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:11:00.243040+0000", 2024-09-17T07:15:24.029 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5468000000000004e-05 2024-09-17T07:15:24.029 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.029 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.029 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:15:24.029 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:11:00.243080+0000", 2024-09-17T07:15:24.029 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9598999999999998e-05 2024-09-17T07:15:24.029 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:15:24.030 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:15:24.030 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:15:24.030 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.030 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.030 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.24929.0:211649 8.f 8:f5873bba:::benchmark_data_smithi100_102803_object81145:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e219)", 2024-09-17T07:15:24.030 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:10:59.300298+0000", 2024-09-17T07:15:24.030 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 264.572307249, 2024-09-17T07:15:24.030 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.94293216599999996, 2024-09-17T07:15:24.030 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:15:24.030 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:15:24.030 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:15:24.030 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.24929", 2024-09-17T07:15:24.031 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/564515582", 2024-09-17T07:15:24.031 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 211649 2024-09-17T07:15:24.031 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.031 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:15:24.031 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.031 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:15:24.031 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.300298+0000", 2024-09-17T07:15:24.031 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:15:24.031 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.031 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.031 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:15:24.031 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.300298+0000", 2024-09-17T07:15:24.031 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:15:24.032 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.032 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.032 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:15:24.032 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.300297+0000", 2024-09-17T07:15:24.032 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999986 2024-09-17T07:15:24.032 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.032 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.032 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:15:24.032 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.300304+0000", 2024-09-17T07:15:24.032 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7889999999999997e-06 2024-09-17T07:15:24.032 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.032 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.032 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:15:24.033 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.300318+0000", 2024-09-17T07:15:24.033 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4229999999999999e-05 2024-09-17T07:15:24.033 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.033 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.033 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:15:24.033 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.300326+0000", 2024-09-17T07:15:24.033 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1310000000000006e-06 2024-09-17T07:15:24.033 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.033 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.033 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:15:24.033 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.300371+0000", 2024-09-17T07:15:24.033 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5627000000000002e-05 2024-09-17T07:15:24.034 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.034 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.034 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-17T07:15:24.034 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.300378+0000", 2024-09-17T07:15:24.034 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3640000000000004e-06 2024-09-17T07:15:24.034 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.034 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.034 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:15:24.034 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:11:00.237043+0000", 2024-09-17T07:15:24.034 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.93666494700000003 2024-09-17T07:15:24.034 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.034 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.034 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:15:24.035 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:11:00.237156+0000", 2024-09-17T07:15:24.035 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00011332699999999999 2024-09-17T07:15:24.035 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.035 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.035 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 2,4", 2024-09-17T07:15:24.035 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:11:00.237258+0000", 2024-09-17T07:15:24.035 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000101785 2024-09-17T07:15:24.035 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.035 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.035 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:15:24.035 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:11:00.243112+0000", 2024-09-17T07:15:24.035 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0058541340000000004 2024-09-17T07:15:24.035 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.036 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.036 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:15:24.036 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:11:00.243121+0000", 2024-09-17T07:15:24.036 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.4589999999999998e-06 2024-09-17T07:15:24.036 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.036 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.036 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:15:24.036 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:11:00.243159+0000", 2024-09-17T07:15:24.036 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8287999999999997e-05 2024-09-17T07:15:24.036 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.036 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.036 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:15:24.036 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:11:00.243198+0000", 2024-09-17T07:15:24.037 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9060000000000002e-05 2024-09-17T07:15:24.037 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.037 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.037 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:15:24.037 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:11:00.243230+0000", 2024-09-17T07:15:24.037 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.2366000000000001e-05 2024-09-17T07:15:24.037 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:15:24.037 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:15:24.037 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:15:24.037 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.037 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.037 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.24929.0:211679 8.f 8:fd2392a9:::benchmark_data_smithi100_102803_object81175:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e219)", 2024-09-17T07:15:24.037 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:10:59.324628+0000", 2024-09-17T07:15:24.038 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 264.54797709600001, 2024-09-17T07:15:24.038 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.93129354099999995, 2024-09-17T07:15:24.038 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:15:24.038 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:15:24.038 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:15:24.038 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.24929", 2024-09-17T07:15:24.038 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/564515582", 2024-09-17T07:15:24.038 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 211679 2024-09-17T07:15:24.038 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.038 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:15:24.038 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.038 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:15:24.038 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.324628+0000", 2024-09-17T07:15:24.039 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:15:24.039 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.039 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.039 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:15:24.039 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.324628+0000", 2024-09-17T07:15:24.039 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:15:24.039 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.039 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.039 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:15:24.039 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.324627+0000", 2024-09-17T07:15:24.039 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999986 2024-09-17T07:15:24.039 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.039 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.040 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:15:24.040 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.324633+0000", 2024-09-17T07:15:24.040 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3980000000000004e-06 2024-09-17T07:15:24.040 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.040 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.040 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:15:24.040 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.324645+0000", 2024-09-17T07:15:24.040 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1338999999999999e-05 2024-09-17T07:15:24.040 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.040 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.040 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:15:24.040 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.324651+0000", 2024-09-17T07:15:24.040 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5189999999999998e-06 2024-09-17T07:15:24.041 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.041 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.041 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:15:24.041 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.324691+0000", 2024-09-17T07:15:24.041 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0278999999999998e-05 2024-09-17T07:15:24.041 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.041 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.041 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-17T07:15:24.041 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:10:59.324697+0000", 2024-09-17T07:15:24.041 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5380000000000002e-06 2024-09-17T07:15:24.041 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.041 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.041 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:15:24.042 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:11:00.239374+0000", 2024-09-17T07:15:24.042 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.91467675199999998 2024-09-17T07:15:24.042 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.042 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.042 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:15:24.042 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:11:00.239494+0000", 2024-09-17T07:15:24.042 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000120392 2024-09-17T07:15:24.042 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.042 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.042 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 2,4", 2024-09-17T07:15:24.042 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:11:00.239895+0000", 2024-09-17T07:15:24.042 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00040044400000000002 2024-09-17T07:15:24.042 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.043 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.043 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:15:24.043 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:11:00.243258+0000", 2024-09-17T07:15:24.043 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0033637699999999999 2024-09-17T07:15:24.043 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.043 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.043 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:15:24.043 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:11:00.243290+0000", 2024-09-17T07:15:24.043 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1983000000000002e-05 2024-09-17T07:15:24.043 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.043 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.043 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:15:24.043 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:11:00.255602+0000", 2024-09-17T07:15:24.044 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.012311715000000001 2024-09-17T07:15:24.044 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.044 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.044 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:15:24.044 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:11:00.255879+0000", 2024-09-17T07:15:24.044 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00027728300000000002 2024-09-17T07:15:24.044 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:15:24.044 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:15:24.044 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:15:24.044 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:11:00.255922+0000", 2024-09-17T07:15:24.044 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2466999999999998e-05 2024-09-17T07:15:24.044 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:15:24.044 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:15:24.045 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:15:24.045 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:15:24.045 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:15:24.045 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-17T07:15:24.061 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:15:24.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:24.104 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:24.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:15:24.205 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:15:24.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:24.238 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:24.238 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:15:24.339 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:15:24.375 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:24.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:24.376 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:15:24.477 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:15:24.515 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:24.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:24.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:15:24.600 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:24 smithi100 ceph-mon[29092]: pgmap v589: 148 pgs: 148 active+clean; 1.3 GiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 645 B/s rd, 0 B/s wr, 867 op/s 2024-09-17T07:15:24.601 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:24 smithi100 ceph-mon[39462]: pgmap v589: 148 pgs: 148 active+clean; 1.3 GiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 645 B/s rd, 0 B/s wr, 867 op/s 2024-09-17T07:15:24.617 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:15:24.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:24.654 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:24.654 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:15:24.755 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:15:24.785 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:24.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:24.786 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:15:24.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:24 smithi135 ceph-mon[33238]: pgmap v589: 148 pgs: 148 active+clean; 1.3 GiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 645 B/s rd, 0 B/s wr, 867 op/s 2024-09-17T07:15:24.888 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:15:24.917 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:24.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:24.918 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:15:25.019 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:15:25.056 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:25.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:25.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:15:25.158 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:15:25.190 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:25.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:25.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:15:25.292 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:15:25.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:25.322 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:25.322 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:15:25.423 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:15:25.462 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:25.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:25.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:15:25.564 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:15:25.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:25.603 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:25.604 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:15:25.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:25 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:15:25.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:25 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:15:25.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:25 smithi100 ceph-mon[29092]: pgmap v590: 148 pgs: 148 active+clean; 886 MiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.65k op/s 2024-09-17T07:15:25.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:25 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:15:25.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:25 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:15:25.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:25 smithi100 ceph-mon[39462]: pgmap v590: 148 pgs: 148 active+clean; 886 MiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.65k op/s 2024-09-17T07:15:25.705 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:15:25.742 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:25.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:25.743 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:15:25.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:25 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:15:25.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:25 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:15:25.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:25 smithi135 ceph-mon[33238]: pgmap v590: 148 pgs: 148 active+clean; 886 MiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.65k op/s 2024-09-17T07:15:25.794 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:15:25.844 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:15:25.871 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:25.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:25.872 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:15:25.973 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:15:26.002 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:26.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:26.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:15:26.104 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:15:26.141 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:26.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:26.142 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:15:26.243 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:15:26.276 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:26.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:26.278 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:15:26.348 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:15:25 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:15:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:15:26.379 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:15:26.390 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-17T07:15:26.415 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:26.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:26.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:15:26.517 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:15:26.552 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:26.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:26.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:15:26.620 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:26 smithi100 ceph-mon[29092]: osdmap e287: 8 total, 8 up, 7 in 2024-09-17T07:15:26.620 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:26 smithi100 ceph-mon[39462]: osdmap e287: 8 total, 8 up, 7 in 2024-09-17T07:15:26.653 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:15:26.685 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:26.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:26.686 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:15:26.787 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:15:26.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:26 smithi135 ceph-mon[33238]: osdmap e287: 8 total, 8 up, 7 in 2024-09-17T07:15:26.813 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:26.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:26.814 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:15:26.915 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:15:26.941 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:26.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:26.942 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:15:27.043 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:15:27.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:27.087 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:27.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:15:27.187 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:15:27.216 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:27.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:27.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:15:27.318 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:15:27.351 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:27.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:27.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:15:27.453 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:15:27.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:27.494 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:27.494 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:15:27.595 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:15:27.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:27 smithi100 ceph-mon[29092]: pgmap v592: 148 pgs: 148 active+clean; 886 MiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.01k op/s 2024-09-17T07:15:27.627 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:27.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:27 smithi100 ceph-mon[39462]: pgmap v592: 148 pgs: 148 active+clean; 886 MiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.01k op/s 2024-09-17T07:15:27.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:27.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:15:27.729 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:15:27.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:27.762 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:27.762 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:15:27.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:27 smithi135 ceph-mon[33238]: pgmap v592: 148 pgs: 148 active+clean; 886 MiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.01k op/s 2024-09-17T07:15:27.863 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:15:27.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:27.895 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:27.895 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:15:27.996 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:15:28.029 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:28.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:28.030 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:15:28.131 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:15:28.160 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:28.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:28.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:15:28.262 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:15:28.303 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:28.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:28.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:15:28.405 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:15:28.437 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:28.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:28.438 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:15:28.538 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:15:28.574 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:28.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:28.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:15:28.676 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:15:28.713 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:28.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:28.714 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:15:28.815 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:15:28.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:28.854 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:28.854 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:15:28.955 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:15:28.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:28.985 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:28.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:15:29.086 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:15:29.120 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:29.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:29.121 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:15:29.222 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:15:29.246 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:29.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:29.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:15:29.498 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:15:28 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:15:29.598 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:15:29.634 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:29.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:29.635 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:15:29.736 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:15:29.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:29.764 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:29.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:15:29.865 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:15:29.898 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:29.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:29.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:15:30.000 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:15:30.041 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:30.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:30.043 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:15:30.143 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:15:30.179 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:30.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:30.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:15:30.281 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:15:30.314 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:30.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:30.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:15:30.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:30 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:15:30.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:30 smithi100 ceph-mon[29092]: pgmap v593: 148 pgs: 148 active+clean; 569 MiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.42k op/s 2024-09-17T07:15:30.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:30 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:15:30.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:30 smithi100 ceph-mon[39462]: pgmap v593: 148 pgs: 148 active+clean; 569 MiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.42k op/s 2024-09-17T07:15:30.417 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:15:30.454 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:30.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:30.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:15:30.555 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:15:30.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:30.594 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:30.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:15:30.695 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:15:30.728 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:30.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:30.729 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:15:30.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:30 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:15:30.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:30 smithi135 ceph-mon[33238]: pgmap v593: 148 pgs: 148 active+clean; 569 MiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.42k op/s 2024-09-17T07:15:30.830 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:15:30.861 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:30.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:30.862 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:15:30.963 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:15:31.004 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:31.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:31.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:15:31.037 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:15:31.037 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1: {} 2024-09-17T07:15:31.037 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:15:31.037 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:15:31.038 INFO:teuthology.orchestra.run.smithi100.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:15:31.038 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:15:31.062 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:15:31.062 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:15:31.091 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:15:31.092 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:15:31.106 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:15:31.132 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:15:31.132 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:15:31.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:31.147 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:31.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:15:31.173 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:15:31.173 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:15:31.206 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:15:31.207 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:15:31.248 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:15:31.284 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:31.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:31.286 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:15:31.387 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:15:31.418 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:31.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:31.419 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:15:31.462 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:31 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1577740864' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:15:31.462 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:31 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1577740864' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:15:31.520 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:15:31.554 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:31.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:31.555 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:15:31.656 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:15:31.687 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:31.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:31.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:15:31.789 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:15:31.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:31 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1577740864' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:15:31.823 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:31.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:31.824 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:15:31.925 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:15:31.964 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:31.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:31.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:15:32.066 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:15:32.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:32.099 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:32.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:15:32.200 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:15:32.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:32.233 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:32.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:15:32.335 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:15:32.351 INFO:teuthology.orchestra.run.smithi100.stderr:noscrub is unset 2024-09-17T07:15:32.371 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:32.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:32.372 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:15:32.473 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:15:32.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:32.507 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:32.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:15:32.525 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:32 smithi100 ceph-mon[29092]: pgmap v594: 148 pgs: 148 active+clean; 466 MiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.46k op/s 2024-09-17T07:15:32.525 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:32 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2948498362' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:15:32.526 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:32 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:15:32.527 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:32 smithi100 ceph-mon[39462]: pgmap v594: 148 pgs: 148 active+clean; 466 MiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.46k op/s 2024-09-17T07:15:32.527 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:32 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2948498362' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:15:32.527 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:32 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:15:32.537 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:15:32.608 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:15:32.642 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:32.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:32.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:15:32.744 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:15:32.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:32.785 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:32.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:15:32.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:32 smithi135 ceph-mon[33238]: pgmap v594: 148 pgs: 148 active+clean; 466 MiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.46k op/s 2024-09-17T07:15:32.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:32 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2948498362' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:15:32.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:32 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:15:32.886 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:15:32.913 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:32.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:32.914 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:15:33.015 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:15:33.044 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:33.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:33.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:15:33.146 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:15:33.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:33.184 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:33.185 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:15:33.285 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:15:33.315 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:33.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:33.316 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:15:33.355 INFO:teuthology.orchestra.run.smithi100.stderr:reweighted osd.6 to 1 (10000) 2024-09-17T07:15:33.417 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:15:33.447 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:33.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:33.448 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:15:33.508 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:33 smithi100 ceph-mon[29092]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:15:33.508 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:33 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T07:15:33.508 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:33 smithi100 ceph-mon[29092]: osdmap e288: 8 total, 8 up, 7 in 2024-09-17T07:15:33.508 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:33 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1900735562' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-17T07:15:33.508 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:33 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-17T07:15:33.509 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:33 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:15:33.509 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:33 smithi100 ceph-mon[39462]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:15:33.509 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:33 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T07:15:33.509 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:33 smithi100 ceph-mon[39462]: osdmap e288: 8 total, 8 up, 7 in 2024-09-17T07:15:33.509 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:33 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1900735562' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-17T07:15:33.509 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:33 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-17T07:15:33.509 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:33 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:15:33.549 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:15:33.589 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:33.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:33.590 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:15:33.691 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:15:33.723 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:33.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:33.724 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:15:33.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:33 smithi135 ceph-mon[33238]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:15:33.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:33 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T07:15:33.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:33 smithi135 ceph-mon[33238]: osdmap e288: 8 total, 8 up, 7 in 2024-09-17T07:15:33.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:33 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1900735562' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-17T07:15:33.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:33 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-17T07:15:33.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:33 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:15:33.825 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:15:33.852 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:33.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:33.853 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:15:33.954 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:15:33.986 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:33.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:33.987 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:15:34.088 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:15:34.122 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:34.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:34.123 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:15:34.223 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:15:34.253 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:34.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:34.254 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:15:34.276 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:15:34.355 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:15:34.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:34.399 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:34.399 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:15:34.419 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Removed 134263 objects 2024-09-17T07:15:34.419 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Clean up completed and total clean up time :107.188 2024-09-17T07:15:34.500 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:15:34.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:34.533 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:34.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:15:34.564 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:34 smithi100 ceph-mon[39462]: pgmap v596: 148 pgs: 148 active+clean; 466 MiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 840 op/s 2024-09-17T07:15:34.564 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:34 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-17T07:15:34.565 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:34 smithi100 ceph-mon[39462]: osdmap e289: 8 total, 8 up, 7 in 2024-09-17T07:15:34.565 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:34 smithi100 ceph-mon[29092]: pgmap v596: 148 pgs: 148 active+clean; 466 MiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 840 op/s 2024-09-17T07:15:34.565 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:34 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-17T07:15:34.565 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:34 smithi100 ceph-mon[29092]: osdmap e289: 8 total, 8 up, 7 in 2024-09-17T07:15:34.634 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:15:34.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:34.657 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:34.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:15:34.758 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:15:34.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:34.780 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:34.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:15:34.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:34 smithi135 ceph-mon[33238]: pgmap v596: 148 pgs: 148 active+clean; 466 MiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 840 op/s 2024-09-17T07:15:34.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:34 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-17T07:15:34.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:34 smithi135 ceph-mon[33238]: osdmap e289: 8 total, 8 up, 7 in 2024-09-17T07:15:34.882 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:15:34.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:34.905 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:34.905 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:15:35.006 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:15:35.028 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:35.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:35.029 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:15:35.130 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:15:35.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:35.152 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:35.152 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:15:35.230 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-17T07:15:35.253 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:15:35.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:35.295 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:35.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:15:35.397 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:15:35.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:35.419 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:35.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:15:35.521 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:15:35.545 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:35.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:35.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:15:35.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:35 smithi100 ceph-mon[29092]: 2.14 scrub starts 2024-09-17T07:15:35.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:35 smithi100 ceph-mon[29092]: 2.14 scrub ok 2024-09-17T07:15:35.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:35 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:15:35.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:35 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:15:35.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:35 smithi100 ceph-mon[39462]: 2.14 scrub starts 2024-09-17T07:15:35.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:35 smithi100 ceph-mon[39462]: 2.14 scrub ok 2024-09-17T07:15:35.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:35 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:15:35.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:35 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:15:35.646 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:15:35.670 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:35.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:35.672 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:15:35.772 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:15:35.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:35 smithi135 ceph-mon[33238]: 2.14 scrub starts 2024-09-17T07:15:35.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:35 smithi135 ceph-mon[33238]: 2.14 scrub ok 2024-09-17T07:15:35.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:35 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:15:35.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:35 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:15:35.794 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:35.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:35.795 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:15:35.896 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:15:35.902 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd reweight 7 1 2024-09-17T07:15:35.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:35.919 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:35.919 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:15:36.020 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:15:36.041 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:36.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:36.042 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:15:36.143 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:15:36.165 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:36.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:36.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:15:36.167 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:15:35 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:15:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:15:36.198 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-17T07:15:36.198 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:15:36.267 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:15:36.292 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:36.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:36.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:15:36.393 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:15:36.408 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:15:36.415 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:36.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:36.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:15:36.517 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:15:36.528 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:36 smithi100 ceph-mon[29092]: pgmap v598: 148 pgs: 148 active+clean; 48 MiB data, 3.9 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.68k op/s 2024-09-17T07:15:36.528 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:36 smithi100 ceph-mon[29092]: osdmap e290: 8 total, 8 up, 7 in 2024-09-17T07:15:36.529 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:36 smithi100 ceph-mon[39462]: pgmap v598: 148 pgs: 148 active+clean; 48 MiB data, 3.9 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.68k op/s 2024-09-17T07:15:36.529 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:36 smithi100 ceph-mon[39462]: osdmap e290: 8 total, 8 up, 7 in 2024-09-17T07:15:36.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:36.540 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:36.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:15:36.641 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:15:36.663 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:36.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:36.664 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:15:36.691 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:15:36.765 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:15:36.788 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:36.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:36.789 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:15:36.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:36 smithi135 ceph-mon[33238]: pgmap v598: 148 pgs: 148 active+clean; 48 MiB data, 3.9 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.68k op/s 2024-09-17T07:15:36.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:36 smithi135 ceph-mon[33238]: osdmap e290: 8 total, 8 up, 7 in 2024-09-17T07:15:36.890 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:15:36.911 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:36.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:36.913 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:15:37.014 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:15:37.039 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:37.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:37.040 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:15:37.141 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:15:37.165 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:37.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:37.166 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:15:37.267 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:15:37.291 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:37.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:37.292 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:15:37.393 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:15:37.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:37.416 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:37.416 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:15:37.517 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:15:37.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:37.544 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:37.544 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:15:37.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:37 smithi100 ceph-mon[29092]: 2.1a scrub starts 2024-09-17T07:15:37.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:37 smithi100 ceph-mon[29092]: 2.1a scrub ok 2024-09-17T07:15:37.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:37 smithi100 ceph-mon[29092]: 4.1a scrub starts 2024-09-17T07:15:37.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:37 smithi100 ceph-mon[29092]: 4.1a scrub ok 2024-09-17T07:15:37.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:37 smithi100 ceph-mon[29092]: 9.6 scrub starts 2024-09-17T07:15:37.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:37 smithi100 ceph-mon[29092]: 9.6 scrub ok 2024-09-17T07:15:37.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:37 smithi100 ceph-mon[29092]: 5.5 scrub starts 2024-09-17T07:15:37.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:37 smithi100 ceph-mon[29092]: 5.5 scrub ok 2024-09-17T07:15:37.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:37 smithi100 ceph-mon[29092]: 5.b scrub starts 2024-09-17T07:15:37.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:37 smithi100 ceph-mon[29092]: 5.b scrub ok 2024-09-17T07:15:37.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:37 smithi100 ceph-mon[29092]: pgmap v600: 148 pgs: 148 active+clean; 48 MiB data, 3.9 GiB used, 622 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.12k op/s 2024-09-17T07:15:37.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:37 smithi100 ceph-mon[39462]: 2.1a scrub starts 2024-09-17T07:15:37.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:37 smithi100 ceph-mon[39462]: 2.1a scrub ok 2024-09-17T07:15:37.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:37 smithi100 ceph-mon[39462]: 4.1a scrub starts 2024-09-17T07:15:37.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:37 smithi100 ceph-mon[39462]: 4.1a scrub ok 2024-09-17T07:15:37.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:37 smithi100 ceph-mon[39462]: 9.6 scrub starts 2024-09-17T07:15:37.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:37 smithi100 ceph-mon[39462]: 9.6 scrub ok 2024-09-17T07:15:37.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:37 smithi100 ceph-mon[39462]: 5.5 scrub starts 2024-09-17T07:15:37.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:37 smithi100 ceph-mon[39462]: 5.5 scrub ok 2024-09-17T07:15:37.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:37 smithi100 ceph-mon[39462]: 5.b scrub starts 2024-09-17T07:15:37.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:37 smithi100 ceph-mon[39462]: 5.b scrub ok 2024-09-17T07:15:37.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:37 smithi100 ceph-mon[39462]: pgmap v600: 148 pgs: 148 active+clean; 48 MiB data, 3.9 GiB used, 622 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.12k op/s 2024-09-17T07:15:37.645 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:15:37.668 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:37.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:37.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:15:37.769 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:15:37.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:37 smithi135 ceph-mon[33238]: 2.1a scrub starts 2024-09-17T07:15:37.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:37 smithi135 ceph-mon[33238]: 2.1a scrub ok 2024-09-17T07:15:37.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:37 smithi135 ceph-mon[33238]: 4.1a scrub starts 2024-09-17T07:15:37.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:37 smithi135 ceph-mon[33238]: 4.1a scrub ok 2024-09-17T07:15:37.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:37 smithi135 ceph-mon[33238]: 9.6 scrub starts 2024-09-17T07:15:37.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:37 smithi135 ceph-mon[33238]: 9.6 scrub ok 2024-09-17T07:15:37.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:37 smithi135 ceph-mon[33238]: 5.5 scrub starts 2024-09-17T07:15:37.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:37 smithi135 ceph-mon[33238]: 5.5 scrub ok 2024-09-17T07:15:37.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:37 smithi135 ceph-mon[33238]: 5.b scrub starts 2024-09-17T07:15:37.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:37 smithi135 ceph-mon[33238]: 5.b scrub ok 2024-09-17T07:15:37.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:37 smithi135 ceph-mon[33238]: pgmap v600: 148 pgs: 148 active+clean; 48 MiB data, 3.9 GiB used, 622 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.12k op/s 2024-09-17T07:15:37.795 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:37.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:37.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:15:37.897 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:15:37.923 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:37.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:37.924 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:15:38.025 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:15:38.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:38.051 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:38.052 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:15:38.153 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:15:38.174 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:38.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:38.176 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:15:38.276 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:15:38.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:38.322 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:38.322 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:15:38.423 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:15:38.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:38.446 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:38.446 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:15:38.547 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:15:38.557 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_2 2024-09-17T07:15:38.558 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd pool rm unique_pool_2 unique_pool_2 --yes-i-really-really-mean-it 2024-09-17T07:15:38.570 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:15:38.572 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1: {} 2024-09-17T07:15:38.572 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:15:38.572 INFO:teuthology.orchestra.run.smithi100.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:15:38.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:38.573 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:38.573 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:15:38.574 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:15:38.575 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:15:38.591 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:15:38.591 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:15:38.607 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:15:38.608 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:15:38.608 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:38 smithi100 ceph-mon[29092]: 2.19 scrub starts 2024-09-17T07:15:38.608 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:38 smithi100 ceph-mon[29092]: 2.19 scrub ok 2024-09-17T07:15:38.608 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:38 smithi100 ceph-mon[29092]: 3.0 scrub starts 2024-09-17T07:15:38.608 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:38 smithi100 ceph-mon[29092]: 3.0 scrub ok 2024-09-17T07:15:38.608 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:38 smithi100 ceph-mon[29092]: 9.c scrub starts 2024-09-17T07:15:38.608 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:38 smithi100 ceph-mon[29092]: 9.c scrub ok 2024-09-17T07:15:38.609 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:38 smithi100 ceph-mon[29092]: 5.13 scrub starts 2024-09-17T07:15:38.609 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:38 smithi100 ceph-mon[29092]: 5.13 scrub ok 2024-09-17T07:15:38.609 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:38 smithi100 ceph-mon[39462]: 2.19 scrub starts 2024-09-17T07:15:38.609 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:38 smithi100 ceph-mon[39462]: 2.19 scrub ok 2024-09-17T07:15:38.609 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:38 smithi100 ceph-mon[39462]: 3.0 scrub starts 2024-09-17T07:15:38.609 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:38 smithi100 ceph-mon[39462]: 3.0 scrub ok 2024-09-17T07:15:38.609 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:38 smithi100 ceph-mon[39462]: 9.c scrub starts 2024-09-17T07:15:38.609 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:38 smithi100 ceph-mon[39462]: 9.c scrub ok 2024-09-17T07:15:38.609 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:38 smithi100 ceph-mon[39462]: 5.13 scrub starts 2024-09-17T07:15:38.609 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:38 smithi100 ceph-mon[39462]: 5.13 scrub ok 2024-09-17T07:15:38.624 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:15:38.624 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:15:38.641 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:15:38.641 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:15:38.658 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:15:38.658 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:15:38.674 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:15:38.694 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:38 smithi135 ceph-mon[33238]: 2.19 scrub starts 2024-09-17T07:15:38.695 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:38 smithi135 ceph-mon[33238]: 2.19 scrub ok 2024-09-17T07:15:38.695 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:38 smithi135 ceph-mon[33238]: 3.0 scrub starts 2024-09-17T07:15:38.695 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:38 smithi135 ceph-mon[33238]: 3.0 scrub ok 2024-09-17T07:15:38.695 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:38 smithi135 ceph-mon[33238]: 9.c scrub starts 2024-09-17T07:15:38.695 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:38 smithi135 ceph-mon[33238]: 9.c scrub ok 2024-09-17T07:15:38.695 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:38 smithi135 ceph-mon[33238]: 5.13 scrub starts 2024-09-17T07:15:38.695 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:38 smithi135 ceph-mon[33238]: 5.13 scrub ok 2024-09-17T07:15:38.698 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:38.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:38.699 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:15:38.800 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:15:38.821 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:38.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:38.823 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:15:38.923 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:15:38.945 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:38.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:38.946 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:15:39.043 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:15:38 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:15:39.048 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:15:39.074 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:39.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:39.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:15:39.176 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:15:39.199 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:39.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:39.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:15:39.301 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:15:39.325 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:39.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:39.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:15:39.427 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:15:39.445 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:39 smithi100 ceph-mon[39462]: 6.1d scrub starts 2024-09-17T07:15:39.445 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:39 smithi100 ceph-mon[39462]: 6.1d scrub ok 2024-09-17T07:15:39.445 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:39 smithi100 ceph-mon[39462]: 5.1c scrub starts 2024-09-17T07:15:39.445 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:39 smithi100 ceph-mon[39462]: 5.1c scrub ok 2024-09-17T07:15:39.445 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:39 smithi100 ceph-mon[39462]: 9.7 scrub starts 2024-09-17T07:15:39.445 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:39 smithi100 ceph-mon[39462]: 9.7 scrub ok 2024-09-17T07:15:39.445 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:39 smithi100 ceph-mon[39462]: 4.3 scrub starts 2024-09-17T07:15:39.445 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:39 smithi100 ceph-mon[39462]: 4.3 scrub ok 2024-09-17T07:15:39.445 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:39 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2549389109' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:15:39.445 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:39 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:15:39.446 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:39 smithi100 ceph-mon[39462]: pgmap v601: 148 pgs: 148 active+clean; 582 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.08k op/s 2024-09-17T07:15:39.446 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:39 smithi100 ceph-mon[29092]: 6.1d scrub starts 2024-09-17T07:15:39.446 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:39 smithi100 ceph-mon[29092]: 6.1d scrub ok 2024-09-17T07:15:39.446 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:39 smithi100 ceph-mon[29092]: 5.1c scrub starts 2024-09-17T07:15:39.446 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:39 smithi100 ceph-mon[29092]: 5.1c scrub ok 2024-09-17T07:15:39.446 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:39 smithi100 ceph-mon[29092]: 9.7 scrub starts 2024-09-17T07:15:39.446 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:39 smithi100 ceph-mon[29092]: 9.7 scrub ok 2024-09-17T07:15:39.446 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:39 smithi100 ceph-mon[29092]: 4.3 scrub starts 2024-09-17T07:15:39.446 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:39 smithi100 ceph-mon[29092]: 4.3 scrub ok 2024-09-17T07:15:39.446 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:39 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2549389109' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:15:39.446 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:39 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:15:39.447 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:39 smithi100 ceph-mon[29092]: pgmap v601: 148 pgs: 148 active+clean; 582 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.08k op/s 2024-09-17T07:15:39.451 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:39.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:39.452 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:15:39.553 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:15:39.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:39.576 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:39.576 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:15:39.677 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:15:39.701 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:39.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:39.702 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:15:39.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:39 smithi135 ceph-mon[33238]: 6.1d scrub starts 2024-09-17T07:15:39.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:39 smithi135 ceph-mon[33238]: 6.1d scrub ok 2024-09-17T07:15:39.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:39 smithi135 ceph-mon[33238]: 5.1c scrub starts 2024-09-17T07:15:39.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:39 smithi135 ceph-mon[33238]: 5.1c scrub ok 2024-09-17T07:15:39.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:39 smithi135 ceph-mon[33238]: 9.7 scrub starts 2024-09-17T07:15:39.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:39 smithi135 ceph-mon[33238]: 9.7 scrub ok 2024-09-17T07:15:39.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:39 smithi135 ceph-mon[33238]: 4.3 scrub starts 2024-09-17T07:15:39.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:39 smithi135 ceph-mon[33238]: 4.3 scrub ok 2024-09-17T07:15:39.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:39 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2549389109' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:15:39.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:39 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:15:39.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:39 smithi135 ceph-mon[33238]: pgmap v601: 148 pgs: 148 active+clean; 582 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.08k op/s 2024-09-17T07:15:39.802 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:15:39.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:39.825 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:39.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:15:39.927 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:15:39.953 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:39.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:39.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:15:40.054 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:15:40.079 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:40.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:40.080 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:15:40.181 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:15:40.204 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:40.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:40.205 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:15:40.306 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:15:40.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:40.330 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:40.330 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:15:40.336 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:15:40.431 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:15:40.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:40.454 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:40.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:15:40.486 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-17T07:15:40.556 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:15:40.577 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:40.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:40.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:15:40.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:40 smithi100 ceph-mon[29092]: 9.3 scrub starts 2024-09-17T07:15:40.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:40 smithi100 ceph-mon[29092]: 9.3 scrub ok 2024-09-17T07:15:40.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:40 smithi100 ceph-mon[29092]: 4.18 scrub starts 2024-09-17T07:15:40.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:40 smithi100 ceph-mon[29092]: 4.18 scrub ok 2024-09-17T07:15:40.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:40 smithi100 ceph-mon[29092]: 2.10 scrub starts 2024-09-17T07:15:40.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:40 smithi100 ceph-mon[29092]: 2.10 scrub ok 2024-09-17T07:15:40.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:40 smithi100 ceph-mon[39462]: 9.3 scrub starts 2024-09-17T07:15:40.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:40 smithi100 ceph-mon[39462]: 9.3 scrub ok 2024-09-17T07:15:40.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:40 smithi100 ceph-mon[39462]: 4.18 scrub starts 2024-09-17T07:15:40.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:40 smithi100 ceph-mon[39462]: 4.18 scrub ok 2024-09-17T07:15:40.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:40 smithi100 ceph-mon[39462]: 2.10 scrub starts 2024-09-17T07:15:40.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:40 smithi100 ceph-mon[39462]: 2.10 scrub ok 2024-09-17T07:15:40.679 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:15:40.701 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:40.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:40.702 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:15:40.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:40 smithi135 ceph-mon[33238]: 9.3 scrub starts 2024-09-17T07:15:40.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:40 smithi135 ceph-mon[33238]: 9.3 scrub ok 2024-09-17T07:15:40.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:40 smithi135 ceph-mon[33238]: 4.18 scrub starts 2024-09-17T07:15:40.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:40 smithi135 ceph-mon[33238]: 4.18 scrub ok 2024-09-17T07:15:40.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:40 smithi135 ceph-mon[33238]: 2.10 scrub starts 2024-09-17T07:15:40.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:40 smithi135 ceph-mon[33238]: 2.10 scrub ok 2024-09-17T07:15:40.802 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:15:40.824 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:40.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:40.825 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:15:40.926 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:15:40.949 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:40.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:40.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:15:41.051 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:15:41.072 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:41.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:41.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:15:41.174 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:15:41.198 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:41.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:41.199 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:15:41.299 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:15:41.321 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:41.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:41.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:15:41.422 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:15:41.444 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:41.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:41.445 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:15:41.546 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:15:41.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:41.570 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:41.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:15:41.609 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:41 smithi100 ceph-mon[29092]: 5.e scrub starts 2024-09-17T07:15:41.609 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:41 smithi100 ceph-mon[29092]: 5.e scrub ok 2024-09-17T07:15:41.609 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:41 smithi100 ceph-mon[29092]: 6.10 scrub starts 2024-09-17T07:15:41.609 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:41 smithi100 ceph-mon[29092]: 6.10 scrub ok 2024-09-17T07:15:41.609 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:41 smithi100 ceph-mon[29092]: 5.6 scrub starts 2024-09-17T07:15:41.610 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:41 smithi100 ceph-mon[29092]: 5.6 scrub ok 2024-09-17T07:15:41.610 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:41 smithi100 ceph-mon[29092]: pgmap v602: 148 pgs: 148 active+clean; 582 KiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 932 op/s 2024-09-17T07:15:41.610 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:41 smithi100 ceph-mon[39462]: 5.e scrub starts 2024-09-17T07:15:41.610 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:41 smithi100 ceph-mon[39462]: 5.e scrub ok 2024-09-17T07:15:41.610 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:41 smithi100 ceph-mon[39462]: 6.10 scrub starts 2024-09-17T07:15:41.610 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:41 smithi100 ceph-mon[39462]: 6.10 scrub ok 2024-09-17T07:15:41.610 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:41 smithi100 ceph-mon[39462]: 5.6 scrub starts 2024-09-17T07:15:41.610 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:41 smithi100 ceph-mon[39462]: 5.6 scrub ok 2024-09-17T07:15:41.610 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:41 smithi100 ceph-mon[39462]: pgmap v602: 148 pgs: 148 active+clean; 582 KiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 932 op/s 2024-09-17T07:15:41.671 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:15:41.693 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:41.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:41.695 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:15:41.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:41 smithi135 ceph-mon[33238]: 5.e scrub starts 2024-09-17T07:15:41.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:41 smithi135 ceph-mon[33238]: 5.e scrub ok 2024-09-17T07:15:41.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:41 smithi135 ceph-mon[33238]: 6.10 scrub starts 2024-09-17T07:15:41.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:41 smithi135 ceph-mon[33238]: 6.10 scrub ok 2024-09-17T07:15:41.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:41 smithi135 ceph-mon[33238]: 5.6 scrub starts 2024-09-17T07:15:41.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:41 smithi135 ceph-mon[33238]: 5.6 scrub ok 2024-09-17T07:15:41.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:41 smithi135 ceph-mon[33238]: pgmap v602: 148 pgs: 148 active+clean; 582 KiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 932 op/s 2024-09-17T07:15:41.795 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:15:41.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:41.818 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:41.818 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:15:41.919 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:15:41.941 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:41.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:41.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:15:42.043 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:15:42.065 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:42.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:42.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:15:42.167 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:15:42.189 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:42.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:42.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:15:42.267 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:15:42.291 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:15:42.312 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:42.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:42.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:15:42.413 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:15:42.436 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:42.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:42.437 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:15:42.538 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:15:42.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:42.561 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:42.561 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:15:42.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:42 smithi100 ceph-mon[29092]: 9.1 scrub starts 2024-09-17T07:15:42.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:42 smithi100 ceph-mon[29092]: 9.1 scrub ok 2024-09-17T07:15:42.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:42 smithi100 ceph-mon[29092]: 5.17 scrub starts 2024-09-17T07:15:42.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:42 smithi100 ceph-mon[29092]: 5.17 scrub ok 2024-09-17T07:15:42.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:42 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:15:42.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:42 smithi100 ceph-mon[39462]: 9.1 scrub starts 2024-09-17T07:15:42.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:42 smithi100 ceph-mon[39462]: 9.1 scrub ok 2024-09-17T07:15:42.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:42 smithi100 ceph-mon[39462]: 5.17 scrub starts 2024-09-17T07:15:42.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:42 smithi100 ceph-mon[39462]: 5.17 scrub ok 2024-09-17T07:15:42.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:42 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:15:42.662 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:15:42.684 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:42.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:42.685 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:15:42.785 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:15:42.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:42 smithi135 ceph-mon[33238]: 9.1 scrub starts 2024-09-17T07:15:42.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:42 smithi135 ceph-mon[33238]: 9.1 scrub ok 2024-09-17T07:15:42.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:42 smithi135 ceph-mon[33238]: 5.17 scrub starts 2024-09-17T07:15:42.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:42 smithi135 ceph-mon[33238]: 5.17 scrub ok 2024-09-17T07:15:42.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:42 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:15:42.807 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:42.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:42.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:15:42.829 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:15:42.909 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:15:42.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:42.932 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:42.932 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:15:43.033 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:15:43.057 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:43.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:43.058 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:15:43.159 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:15:43.181 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:43.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:43.182 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:15:43.282 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:15:43.305 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:43.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:43.307 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:15:43.380 INFO:teuthology.orchestra.run.smithi100.stderr:reweighted osd.7 to 1 (10000) 2024-09-17T07:15:43.408 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:15:43.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:43.433 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:43.433 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:15:43.535 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:15:43.557 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:43.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:43.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:15:43.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:43 smithi100 ceph-mon[29092]: 4.1f scrub starts 2024-09-17T07:15:43.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:43 smithi100 ceph-mon[29092]: 4.1f scrub ok 2024-09-17T07:15:43.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:43 smithi100 ceph-mon[29092]: 5.12 scrub starts 2024-09-17T07:15:43.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:43 smithi100 ceph-mon[29092]: 5.12 scrub ok 2024-09-17T07:15:43.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:43 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2645963785' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-17T07:15:43.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:43 smithi100 ceph-mon[29092]: pgmap v603: 148 pgs: 148 active+clean; 582 KiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 751 op/s 2024-09-17T07:15:43.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:43 smithi100 ceph-mon[39462]: 4.1f scrub starts 2024-09-17T07:15:43.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:43 smithi100 ceph-mon[39462]: 4.1f scrub ok 2024-09-17T07:15:43.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:43 smithi100 ceph-mon[39462]: 5.12 scrub starts 2024-09-17T07:15:43.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:43 smithi100 ceph-mon[39462]: 5.12 scrub ok 2024-09-17T07:15:43.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:43 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2645963785' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-17T07:15:43.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:43 smithi100 ceph-mon[39462]: pgmap v603: 148 pgs: 148 active+clean; 582 KiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 751 op/s 2024-09-17T07:15:43.659 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:15:43.682 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:43.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:43.683 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:15:43.784 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:15:43.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:43 smithi135 ceph-mon[33238]: 4.1f scrub starts 2024-09-17T07:15:43.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:43 smithi135 ceph-mon[33238]: 4.1f scrub ok 2024-09-17T07:15:43.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:43 smithi135 ceph-mon[33238]: 5.12 scrub starts 2024-09-17T07:15:43.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:43 smithi135 ceph-mon[33238]: 5.12 scrub ok 2024-09-17T07:15:43.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:43 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2645963785' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-17T07:15:43.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:43 smithi135 ceph-mon[33238]: pgmap v603: 148 pgs: 148 active+clean; 582 KiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 751 op/s 2024-09-17T07:15:43.806 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:43.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:43.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:15:43.908 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:15:43.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:43.930 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:43.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:15:44.030 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:15:44.031 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:15:44.053 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:44.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:44.054 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:15:44.154 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:15:44.176 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:44.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:44.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:15:44.278 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:15:44.300 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:44.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:44.301 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:15:44.388 INFO:teuthology.orchestra.run.smithi100.stderr:nodeep-scrub is unset 2024-09-17T07:15:44.402 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:15:44.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:44.425 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:44.425 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:15:44.526 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:15:44.548 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:44.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:44.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:15:44.651 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:15:44.672 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:44.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:44.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:15:44.731 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:44 smithi100 ceph-mon[29092]: 4.1 scrub starts 2024-09-17T07:15:44.731 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:44 smithi100 ceph-mon[29092]: 4.1 scrub ok 2024-09-17T07:15:44.731 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:44 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2645963785' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-17T07:15:44.731 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:44 smithi100 ceph-mon[29092]: osdmap e291: 8 total, 8 up, 7 in 2024-09-17T07:15:44.731 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:44 smithi100 ceph-mon[29092]: 3.2 scrub starts 2024-09-17T07:15:44.731 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:44 smithi100 ceph-mon[29092]: 3.2 scrub ok 2024-09-17T07:15:44.731 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:44 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/240856045' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:15:44.731 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:44 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:15:44.731 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:44 smithi100 ceph-mon[39462]: 4.1 scrub starts 2024-09-17T07:15:44.732 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:44 smithi100 ceph-mon[39462]: 4.1 scrub ok 2024-09-17T07:15:44.732 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:44 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2645963785' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-17T07:15:44.732 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:44 smithi100 ceph-mon[39462]: osdmap e291: 8 total, 8 up, 7 in 2024-09-17T07:15:44.732 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:44 smithi100 ceph-mon[39462]: 3.2 scrub starts 2024-09-17T07:15:44.732 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:44 smithi100 ceph-mon[39462]: 3.2 scrub ok 2024-09-17T07:15:44.732 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:44 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/240856045' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:15:44.732 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:44 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:15:44.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:44 smithi135 ceph-mon[33238]: 4.1 scrub starts 2024-09-17T07:15:44.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:44 smithi135 ceph-mon[33238]: 4.1 scrub ok 2024-09-17T07:15:44.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:44 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2645963785' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-17T07:15:44.759 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:44 smithi135 ceph-mon[33238]: osdmap e291: 8 total, 8 up, 7 in 2024-09-17T07:15:44.759 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:44 smithi135 ceph-mon[33238]: 3.2 scrub starts 2024-09-17T07:15:44.759 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:44 smithi135 ceph-mon[33238]: 3.2 scrub ok 2024-09-17T07:15:44.759 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:44 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/240856045' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:15:44.759 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:44 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:15:44.774 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:15:44.796 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:44.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:44.797 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:15:44.899 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:15:44.920 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:44.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:44.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:15:45.022 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:15:45.044 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:45.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:45.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:15:45.146 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:15:45.167 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:45.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:45.168 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:15:45.269 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:15:45.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:45.291 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:45.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:15:45.392 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:15:45.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:45.414 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:45.414 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:15:45.515 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:15:45.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:45.538 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:45.538 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:15:45.640 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:15:45.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:45.662 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:45.662 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:15:45.729 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:45 smithi100 ceph-mon[29092]: 4.a scrub starts 2024-09-17T07:15:45.729 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:45 smithi100 ceph-mon[29092]: 4.a scrub ok 2024-09-17T07:15:45.730 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:45 smithi100 ceph-mon[29092]: 4.17 scrub starts 2024-09-17T07:15:45.730 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:45 smithi100 ceph-mon[29092]: 4.17 scrub ok 2024-09-17T07:15:45.730 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:45 smithi100 ceph-mon[29092]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T07:15:45.730 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:45 smithi100 ceph-mon[29092]: Cluster is now healthy 2024-09-17T07:15:45.730 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:45 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T07:15:45.730 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:45 smithi100 ceph-mon[29092]: osdmap e292: 8 total, 8 up, 7 in 2024-09-17T07:15:45.730 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:45 smithi100 ceph-mon[29092]: pgmap v606: 148 pgs: 148 active+clean; 582 KiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 96 op/s 2024-09-17T07:15:45.730 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:45 smithi100 ceph-mon[39462]: 4.a scrub starts 2024-09-17T07:15:45.730 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:45 smithi100 ceph-mon[39462]: 4.a scrub ok 2024-09-17T07:15:45.730 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:45 smithi100 ceph-mon[39462]: 4.17 scrub starts 2024-09-17T07:15:45.730 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:45 smithi100 ceph-mon[39462]: 4.17 scrub ok 2024-09-17T07:15:45.731 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:45 smithi100 ceph-mon[39462]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T07:15:45.731 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:45 smithi100 ceph-mon[39462]: Cluster is now healthy 2024-09-17T07:15:45.731 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:45 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T07:15:45.731 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:45 smithi100 ceph-mon[39462]: osdmap e292: 8 total, 8 up, 7 in 2024-09-17T07:15:45.731 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:45 smithi100 ceph-mon[39462]: pgmap v606: 148 pgs: 148 active+clean; 582 KiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 96 op/s 2024-09-17T07:15:45.762 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:15:45.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:45.785 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:45.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:15:45.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:45 smithi135 ceph-mon[33238]: 4.a scrub starts 2024-09-17T07:15:45.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:45 smithi135 ceph-mon[33238]: 4.a scrub ok 2024-09-17T07:15:45.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:45 smithi135 ceph-mon[33238]: 4.17 scrub starts 2024-09-17T07:15:45.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:45 smithi135 ceph-mon[33238]: 4.17 scrub ok 2024-09-17T07:15:45.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:45 smithi135 ceph-mon[33238]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T07:15:45.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:45 smithi135 ceph-mon[33238]: Cluster is now healthy 2024-09-17T07:15:45.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:45 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T07:15:45.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:45 smithi135 ceph-mon[33238]: osdmap e292: 8 total, 8 up, 7 in 2024-09-17T07:15:45.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:45 smithi135 ceph-mon[33238]: pgmap v606: 148 pgs: 148 active+clean; 582 KiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 96 op/s 2024-09-17T07:15:45.887 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:15:45.908 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:45.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:45.909 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:15:46.010 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:15:46.032 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:46.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:46.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:15:46.127 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:15:45 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:15:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:15:46.136 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:15:46.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:46.159 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:46.160 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:15:46.261 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:15:46.283 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:46.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:46.284 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:15:46.385 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:15:46.408 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:46.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:46.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:15:46.510 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:15:46.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:46.532 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:46.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:15:46.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:46 smithi100 ceph-mon[39462]: 2.9 scrub starts 2024-09-17T07:15:46.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:46 smithi100 ceph-mon[39462]: 2.9 scrub ok 2024-09-17T07:15:46.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:46 smithi100 ceph-mon[39462]: 5.18 scrub starts 2024-09-17T07:15:46.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:46 smithi100 ceph-mon[39462]: 5.18 scrub ok 2024-09-17T07:15:46.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:46 smithi100 ceph-mon[39462]: 6.8 deep-scrub starts 2024-09-17T07:15:46.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:46 smithi100 ceph-mon[39462]: 6.8 deep-scrub ok 2024-09-17T07:15:46.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:46 smithi100 ceph-mon[39462]: 5.7 scrub starts 2024-09-17T07:15:46.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:46 smithi100 ceph-mon[39462]: 5.7 scrub ok 2024-09-17T07:15:46.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:46 smithi100 ceph-mon[39462]: 6.18 scrub starts 2024-09-17T07:15:46.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:46 smithi100 ceph-mon[39462]: 6.18 scrub ok 2024-09-17T07:15:46.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:46 smithi100 ceph-mon[29092]: 2.9 scrub starts 2024-09-17T07:15:46.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:46 smithi100 ceph-mon[29092]: 2.9 scrub ok 2024-09-17T07:15:46.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:46 smithi100 ceph-mon[29092]: 5.18 scrub starts 2024-09-17T07:15:46.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:46 smithi100 ceph-mon[29092]: 5.18 scrub ok 2024-09-17T07:15:46.629 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:46 smithi100 ceph-mon[29092]: 6.8 deep-scrub starts 2024-09-17T07:15:46.629 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:46 smithi100 ceph-mon[29092]: 6.8 deep-scrub ok 2024-09-17T07:15:46.629 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:46 smithi100 ceph-mon[29092]: 5.7 scrub starts 2024-09-17T07:15:46.629 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:46 smithi100 ceph-mon[29092]: 5.7 scrub ok 2024-09-17T07:15:46.629 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:46 smithi100 ceph-mon[29092]: 6.18 scrub starts 2024-09-17T07:15:46.629 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:46 smithi100 ceph-mon[29092]: 6.18 scrub ok 2024-09-17T07:15:46.633 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:15:46.650 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:46 smithi135 ceph-mon[33238]: 2.9 scrub starts 2024-09-17T07:15:46.650 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:46 smithi135 ceph-mon[33238]: 2.9 scrub ok 2024-09-17T07:15:46.650 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:46 smithi135 ceph-mon[33238]: 5.18 scrub starts 2024-09-17T07:15:46.650 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:46 smithi135 ceph-mon[33238]: 5.18 scrub ok 2024-09-17T07:15:46.650 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:46 smithi135 ceph-mon[33238]: 6.8 deep-scrub starts 2024-09-17T07:15:46.650 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:46 smithi135 ceph-mon[33238]: 6.8 deep-scrub ok 2024-09-17T07:15:46.650 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:46 smithi135 ceph-mon[33238]: 5.7 scrub starts 2024-09-17T07:15:46.650 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:46 smithi135 ceph-mon[33238]: 5.7 scrub ok 2024-09-17T07:15:46.650 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:46 smithi135 ceph-mon[33238]: 6.18 scrub starts 2024-09-17T07:15:46.650 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:46 smithi135 ceph-mon[33238]: 6.18 scrub ok 2024-09-17T07:15:46.655 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:46.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:46.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:15:46.757 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:15:46.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:46.779 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:46.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:15:46.880 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:15:46.901 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:46.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:46.902 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:15:47.003 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:15:47.023 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:47.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:47.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:15:47.125 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:15:47.148 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:47.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:47.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:15:47.206 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-17T07:15:47.206 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-17T07:15:47.206 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-17T07:15:47.206 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-17T07:15:47.206 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-17T07:15:47.250 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:15:47.272 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:47.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:47.273 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:15:47.374 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:15:47.395 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:47.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:47.396 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:15:47.497 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:15:47.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:47.520 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:47.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:15:47.541 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-17T07:15:47.621 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:15:47.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:47 smithi100 ceph-mon[29092]: 4.1c scrub starts 2024-09-17T07:15:47.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:47 smithi100 ceph-mon[29092]: 4.1c scrub ok 2024-09-17T07:15:47.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:47 smithi100 ceph-mon[29092]: 4.d scrub starts 2024-09-17T07:15:47.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:47 smithi100 ceph-mon[29092]: 4.d scrub ok 2024-09-17T07:15:47.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:47 smithi100 ceph-mon[29092]: 2.1d deep-scrub starts 2024-09-17T07:15:47.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:47 smithi100 ceph-mon[29092]: 2.1d deep-scrub ok 2024-09-17T07:15:47.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:47 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:15:47.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:47 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:15:47.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:47 smithi100 ceph-mon[29092]: pgmap v607: 148 pgs: 148 active+clean; 582 KiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-09-17T07:15:47.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:47 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:15:47.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:47 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:15:47.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:47 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:15:47.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:47 smithi100 ceph-mon[39462]: 4.1c scrub starts 2024-09-17T07:15:47.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:47 smithi100 ceph-mon[39462]: 4.1c scrub ok 2024-09-17T07:15:47.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:47 smithi100 ceph-mon[39462]: 4.d scrub starts 2024-09-17T07:15:47.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:47 smithi100 ceph-mon[39462]: 4.d scrub ok 2024-09-17T07:15:47.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:47 smithi100 ceph-mon[39462]: 2.1d deep-scrub starts 2024-09-17T07:15:47.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:47 smithi100 ceph-mon[39462]: 2.1d deep-scrub ok 2024-09-17T07:15:47.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:47 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:15:47.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:47 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:15:47.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:47 smithi100 ceph-mon[39462]: pgmap v607: 148 pgs: 148 active+clean; 582 KiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-09-17T07:15:47.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:47 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:15:47.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:47 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:15:47.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:47 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:15:47.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:47.642 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:47.642 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:15:47.744 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:15:47.765 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:47.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:47.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:15:47.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:47 smithi135 ceph-mon[33238]: 4.1c scrub starts 2024-09-17T07:15:47.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:47 smithi135 ceph-mon[33238]: 4.1c scrub ok 2024-09-17T07:15:47.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:47 smithi135 ceph-mon[33238]: 4.d scrub starts 2024-09-17T07:15:47.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:47 smithi135 ceph-mon[33238]: 4.d scrub ok 2024-09-17T07:15:47.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:47 smithi135 ceph-mon[33238]: 2.1d deep-scrub starts 2024-09-17T07:15:47.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:47 smithi135 ceph-mon[33238]: 2.1d deep-scrub ok 2024-09-17T07:15:47.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:47 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:15:47.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:47 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:15:47.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:47 smithi135 ceph-mon[33238]: pgmap v607: 148 pgs: 148 active+clean; 582 KiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-09-17T07:15:47.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:47 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:15:47.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:47 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:15:47.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:47 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:15:47.868 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:15:47.890 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:47.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:47.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:15:47.956 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-17T07:15:47.993 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:15:48.020 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:48.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:48.022 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:15:48.123 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:15:48.143 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:48.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:48.144 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:15:48.245 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:15:48.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:48.269 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:48.269 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:15:48.370 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:15:48.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:48.393 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:48.393 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:15:48.472 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:15:48.491 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-17T07:15:48.491 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T07:15:48.494 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:15:48.555 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:48.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:48.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:15:48.633 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:48 smithi100 ceph-mon[29092]: 2.11 deep-scrub starts 2024-09-17T07:15:48.633 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:48 smithi100 ceph-mon[29092]: 2.11 deep-scrub ok 2024-09-17T07:15:48.633 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:48 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:15:48.633 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:48 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:15:48.634 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:48 smithi100 ceph-mon[39462]: 2.11 deep-scrub starts 2024-09-17T07:15:48.634 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:48 smithi100 ceph-mon[39462]: 2.11 deep-scrub ok 2024-09-17T07:15:48.634 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:48 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:15:48.634 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:48 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:15:48.657 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:15:48.681 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:48.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:48.683 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:15:48.784 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:15:48.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:48 smithi135 ceph-mon[33238]: 2.11 deep-scrub starts 2024-09-17T07:15:48.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:48 smithi135 ceph-mon[33238]: 2.11 deep-scrub ok 2024-09-17T07:15:48.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:48 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:15:48.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:48 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:15:48.804 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:48.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:48.805 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:15:48.906 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:15:48.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:48.931 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:48.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:15:49.033 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:15:49.054 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:49.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:49.056 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:15:49.157 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:15:49.178 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:49.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:49.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:15:49.280 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:15:49.293 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:15:48 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:15:49.301 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:49.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:49.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:15:49.403 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:15:49.426 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:49.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:49.428 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:15:49.529 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:15:49.550 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:49.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:49.551 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:15:49.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:49 smithi100 ceph-mon[29092]: 2.e scrub starts 2024-09-17T07:15:49.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:49 smithi100 ceph-mon[29092]: 2.e scrub ok 2024-09-17T07:15:49.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:49 smithi100 ceph-mon[29092]: 2.12 scrub starts 2024-09-17T07:15:49.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:49 smithi100 ceph-mon[29092]: 2.12 scrub ok 2024-09-17T07:15:49.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:49 smithi100 ceph-mon[29092]: 2.c scrub starts 2024-09-17T07:15:49.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:49 smithi100 ceph-mon[29092]: 2.c scrub ok 2024-09-17T07:15:49.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:49 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:15:49.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:49 smithi100 ceph-mon[29092]: pgmap v608: 148 pgs: 148 active+clean; 582 KiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:15:49.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:49 smithi100 ceph-mon[39462]: 2.e scrub starts 2024-09-17T07:15:49.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:49 smithi100 ceph-mon[39462]: 2.e scrub ok 2024-09-17T07:15:49.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:49 smithi100 ceph-mon[39462]: 2.12 scrub starts 2024-09-17T07:15:49.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:49 smithi100 ceph-mon[39462]: 2.12 scrub ok 2024-09-17T07:15:49.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:49 smithi100 ceph-mon[39462]: 2.c scrub starts 2024-09-17T07:15:49.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:49 smithi100 ceph-mon[39462]: 2.c scrub ok 2024-09-17T07:15:49.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:49 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:15:49.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:49 smithi100 ceph-mon[39462]: pgmap v608: 148 pgs: 148 active+clean; 582 KiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:15:49.652 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:15:49.675 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:49.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:49.676 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:15:49.777 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:15:49.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:49 smithi135 ceph-mon[33238]: 2.e scrub starts 2024-09-17T07:15:49.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:49 smithi135 ceph-mon[33238]: 2.e scrub ok 2024-09-17T07:15:49.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:49 smithi135 ceph-mon[33238]: 2.12 scrub starts 2024-09-17T07:15:49.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:49 smithi135 ceph-mon[33238]: 2.12 scrub ok 2024-09-17T07:15:49.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:49 smithi135 ceph-mon[33238]: 2.c scrub starts 2024-09-17T07:15:49.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:49 smithi135 ceph-mon[33238]: 2.c scrub ok 2024-09-17T07:15:49.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:49 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:15:49.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:49 smithi135 ceph-mon[33238]: pgmap v608: 148 pgs: 148 active+clean; 582 KiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:15:49.798 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:49.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:49.798 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:15:49.899 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:15:49.921 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:49.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:49.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:15:50.023 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:15:50.056 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:50.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:50.058 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:15:50.158 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:15:50.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:50.184 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:50.184 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:15:50.285 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:15:50.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:50.307 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:50.307 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:15:50.408 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:15:50.431 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:50.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:50.432 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:15:50.480 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:50 smithi100 ceph-mon[29092]: 2.15 scrub starts 2024-09-17T07:15:50.480 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:50 smithi100 ceph-mon[29092]: 2.15 scrub ok 2024-09-17T07:15:50.480 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:50 smithi100 ceph-mon[29092]: 6.11 deep-scrub starts 2024-09-17T07:15:50.480 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:50 smithi100 ceph-mon[29092]: 6.11 deep-scrub ok 2024-09-17T07:15:50.480 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:50 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:15:50.480 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:50 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:15:50.480 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:50 smithi100 ceph-mon[39462]: 2.15 scrub starts 2024-09-17T07:15:50.481 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:50 smithi100 ceph-mon[39462]: 2.15 scrub ok 2024-09-17T07:15:50.481 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:50 smithi100 ceph-mon[39462]: 6.11 deep-scrub starts 2024-09-17T07:15:50.481 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:50 smithi100 ceph-mon[39462]: 6.11 deep-scrub ok 2024-09-17T07:15:50.481 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:50 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:15:50.481 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:50 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:15:50.532 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:15:50.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:50.554 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:50.555 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:15:50.589 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:15:50.655 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:15:50.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:50.679 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:50.679 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:15:50.780 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:15:50.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:50 smithi135 ceph-mon[33238]: 2.15 scrub starts 2024-09-17T07:15:50.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:50 smithi135 ceph-mon[33238]: 2.15 scrub ok 2024-09-17T07:15:50.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:50 smithi135 ceph-mon[33238]: 6.11 deep-scrub starts 2024-09-17T07:15:50.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:50 smithi135 ceph-mon[33238]: 6.11 deep-scrub ok 2024-09-17T07:15:50.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:50 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:15:50.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:50 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:15:50.802 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:50.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:50.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:15:50.904 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:15:50.927 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:50.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:50.928 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:15:51.028 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:15:51.053 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:51.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:51.054 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:15:51.155 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:15:51.178 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:51.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:51.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:15:51.280 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:15:51.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:51.303 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:51.304 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:15:51.405 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:15:51.405 INFO:teuthology.orchestra.run.smithi100.stderr:pool 'unique_pool_2' removed 2024-09-17T07:15:51.428 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:51.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:51.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:15:51.529 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:15:51.561 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:51.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:51.562 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:15:51.588 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:15:51.588 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1: {} 2024-09-17T07:15:51.588 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:15:51.588 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:15:51.588 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:15:51.588 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:15:51.588 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:15:51.589 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:15:51.589 INFO:teuthology.orchestra.run.smithi100.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:15:51.589 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:15:51.589 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:15:51.589 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:15:51.589 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:15:51.589 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:15:51.590 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:15:51.590 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:15:51.590 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:51 smithi100 ceph-mon[29092]: osdmap e293: 8 total, 8 up, 7 in 2024-09-17T07:15:51.590 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:51 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2808343564' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-17T07:15:51.590 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:51 smithi100 ceph-mon[29092]: 9.8 deep-scrub starts 2024-09-17T07:15:51.590 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:51 smithi100 ceph-mon[29092]: 9.8 deep-scrub ok 2024-09-17T07:15:51.590 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:51 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/3538496113' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:15:51.591 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:51 smithi100 ceph-mon[29092]: pgmap v610: 148 pgs: 148 active+clean; 582 KiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-17T07:15:51.591 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:51 smithi100 ceph-mon[39462]: osdmap e293: 8 total, 8 up, 7 in 2024-09-17T07:15:51.591 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:51 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2808343564' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-17T07:15:51.591 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:51 smithi100 ceph-mon[39462]: 9.8 deep-scrub starts 2024-09-17T07:15:51.591 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:51 smithi100 ceph-mon[39462]: 9.8 deep-scrub ok 2024-09-17T07:15:51.591 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:51 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/3538496113' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:15:51.591 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:51 smithi100 ceph-mon[39462]: pgmap v610: 148 pgs: 148 active+clean; 582 KiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-17T07:15:51.663 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:15:51.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:51.685 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:51.685 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:15:51.786 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:15:51.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:51 smithi135 ceph-mon[33238]: osdmap e293: 8 total, 8 up, 7 in 2024-09-17T07:15:51.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:51 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2808343564' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-17T07:15:51.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:51 smithi135 ceph-mon[33238]: 9.8 deep-scrub starts 2024-09-17T07:15:51.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:51 smithi135 ceph-mon[33238]: 9.8 deep-scrub ok 2024-09-17T07:15:51.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:51 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/3538496113' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:15:51.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:51 smithi135 ceph-mon[33238]: pgmap v610: 148 pgs: 148 active+clean; 582 KiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-17T07:15:51.807 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:51.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:51.809 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:15:51.909 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:15:51.932 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:51.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:51.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:15:52.034 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:15:52.056 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:52.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:52.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:15:52.158 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:15:52.180 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:52.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:52.181 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:15:52.281 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:15:52.304 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:52.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:52.305 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:15:52.406 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:15:52.428 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:52.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:52.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:15:52.530 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:15:52.551 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:52.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:52.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:15:52.564 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:15:52.564 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:15:52.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:52 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2808343564' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-09-17T07:15:52.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:52 smithi100 ceph-mon[29092]: osdmap e294: 8 total, 8 up, 7 in 2024-09-17T07:15:52.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:52 smithi100 ceph-mon[29092]: 5.8 scrub starts 2024-09-17T07:15:52.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:52 smithi100 ceph-mon[29092]: 5.8 scrub ok 2024-09-17T07:15:52.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:52 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2808343564' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-09-17T07:15:52.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:52 smithi100 ceph-mon[39462]: osdmap e294: 8 total, 8 up, 7 in 2024-09-17T07:15:52.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:52 smithi100 ceph-mon[39462]: 5.8 scrub starts 2024-09-17T07:15:52.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:52 smithi100 ceph-mon[39462]: 5.8 scrub ok 2024-09-17T07:15:52.653 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:15:52.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:52.675 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:52.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:15:52.776 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:15:52.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:52 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2808343564' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-09-17T07:15:52.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:52 smithi135 ceph-mon[33238]: osdmap e294: 8 total, 8 up, 7 in 2024-09-17T07:15:52.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:52 smithi135 ceph-mon[33238]: 5.8 scrub starts 2024-09-17T07:15:52.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:52 smithi135 ceph-mon[33238]: 5.8 scrub ok 2024-09-17T07:15:52.798 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:52.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:52.799 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:15:52.900 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:15:52.921 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:52.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:52.923 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:15:53.023 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:15:53.045 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:53.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:53.046 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:15:53.146 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:15:53.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:53.169 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:53.169 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:15:53.270 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:15:53.293 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:53.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:53.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:15:53.395 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:15:53.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:53.419 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:53.419 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:15:53.424 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-17T07:15:53.425 INFO:tasks.radosbench:Beginning radosbench... 2024-09-17T07:15:53.425 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_3 2024-09-17T07:15:53.425 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd pool create unique_pool_3 16 2024-09-17T07:15:53.520 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:15:53.543 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:53.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:53.544 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:15:53.644 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:15:53.666 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:53.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:53.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:15:53.688 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:53 smithi100 ceph-mon[29092]: 6.4 scrub starts 2024-09-17T07:15:53.689 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:53 smithi100 ceph-mon[29092]: 6.4 scrub ok 2024-09-17T07:15:53.689 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:53 smithi100 ceph-mon[29092]: 6.3 scrub starts 2024-09-17T07:15:53.689 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:53 smithi100 ceph-mon[29092]: 6.3 scrub ok 2024-09-17T07:15:53.689 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:53 smithi100 ceph-mon[29092]: pgmap v612: 132 pgs: 132 active+clean; 582 KiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 511 B/s rd, 0 op/s 2024-09-17T07:15:53.689 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:53 smithi100 ceph-mon[39462]: 6.4 scrub starts 2024-09-17T07:15:53.689 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:53 smithi100 ceph-mon[39462]: 6.4 scrub ok 2024-09-17T07:15:53.689 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:53 smithi100 ceph-mon[39462]: 6.3 scrub starts 2024-09-17T07:15:53.689 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:53 smithi100 ceph-mon[39462]: 6.3 scrub ok 2024-09-17T07:15:53.689 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:53 smithi100 ceph-mon[39462]: pgmap v612: 132 pgs: 132 active+clean; 582 KiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 511 B/s rd, 0 op/s 2024-09-17T07:15:53.768 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:15:53.789 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:53.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:53.790 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:15:53.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:53 smithi135 ceph-mon[33238]: 6.4 scrub starts 2024-09-17T07:15:53.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:53 smithi135 ceph-mon[33238]: 6.4 scrub ok 2024-09-17T07:15:53.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:53 smithi135 ceph-mon[33238]: 6.3 scrub starts 2024-09-17T07:15:53.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:53 smithi135 ceph-mon[33238]: 6.3 scrub ok 2024-09-17T07:15:53.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:53 smithi135 ceph-mon[33238]: pgmap v612: 132 pgs: 132 active+clean; 582 KiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 511 B/s rd, 0 op/s 2024-09-17T07:15:53.891 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:15:53.913 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:53.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:53.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:15:54.015 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:15:54.037 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:54.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:54.038 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:15:54.139 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:15:54.159 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:54.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:54.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:15:54.261 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:15:54.284 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:54.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:54.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:15:54.323 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:15:54.385 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:15:54.408 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:54.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:54.409 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:15:54.510 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:15:54.533 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:54.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:54.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:15:54.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:54 smithi100 ceph-mon[29092]: 5.f scrub starts 2024-09-17T07:15:54.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:54 smithi100 ceph-mon[29092]: 5.f scrub ok 2024-09-17T07:15:54.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:54 smithi100 ceph-mon[29092]: 5.3 deep-scrub starts 2024-09-17T07:15:54.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:54 smithi100 ceph-mon[29092]: 5.3 deep-scrub ok 2024-09-17T07:15:54.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:54 smithi100 ceph-mon[39462]: 5.f scrub starts 2024-09-17T07:15:54.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:54 smithi100 ceph-mon[39462]: 5.f scrub ok 2024-09-17T07:15:54.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:54 smithi100 ceph-mon[39462]: 5.3 deep-scrub starts 2024-09-17T07:15:54.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:54 smithi100 ceph-mon[39462]: 5.3 deep-scrub ok 2024-09-17T07:15:54.638 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:15:54.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:54.660 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:54.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:15:54.762 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:15:54.784 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:54.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:54.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:15:54.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:54 smithi135 ceph-mon[33238]: 5.f scrub starts 2024-09-17T07:15:54.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:54 smithi135 ceph-mon[33238]: 5.f scrub ok 2024-09-17T07:15:54.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:54 smithi135 ceph-mon[33238]: 5.3 deep-scrub starts 2024-09-17T07:15:54.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:54 smithi135 ceph-mon[33238]: 5.3 deep-scrub ok 2024-09-17T07:15:54.886 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:15:54.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:54.912 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:54.913 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:15:55.014 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:15:55.036 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:55.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:55.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:15:55.137 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:15:55.164 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:55.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:55.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:15:55.265 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:15:55.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:55.292 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:55.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:15:55.393 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:15:55.419 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:55.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:55.421 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:15:55.483 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:55 smithi100 ceph-mon[29092]: 5.19 scrub starts 2024-09-17T07:15:55.483 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:55 smithi100 ceph-mon[29092]: 5.19 scrub ok 2024-09-17T07:15:55.483 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:55 smithi100 ceph-mon[29092]: pgmap v613: 132 pgs: 132 active+clean; 582 KiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T07:15:55.483 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:55 smithi100 ceph-mon[39462]: 5.19 scrub starts 2024-09-17T07:15:55.483 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:55 smithi100 ceph-mon[39462]: 5.19 scrub ok 2024-09-17T07:15:55.484 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:55 smithi100 ceph-mon[39462]: pgmap v613: 132 pgs: 132 active+clean; 582 KiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T07:15:55.521 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:15:55.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:55.547 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:55.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:15:55.583 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T07:15:55.586 INFO:teuthology.orchestra.run.smithi100.stderr:dumped all 2024-09-17T07:15:55.648 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:15:55.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:55.674 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:55.674 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:15:55.776 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:15:55.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:55 smithi135 ceph-mon[33238]: 5.19 scrub starts 2024-09-17T07:15:55.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:55 smithi135 ceph-mon[33238]: 5.19 scrub ok 2024-09-17T07:15:55.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:55 smithi135 ceph-mon[33238]: pgmap v613: 132 pgs: 132 active+clean; 582 KiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T07:15:55.801 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:55.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:55.802 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:15:55.903 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:15:55.929 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:55.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:55.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:15:56.031 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:15:56.057 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:56.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:56.058 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:15:56.159 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:15:56.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:56.183 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:56.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:15:56.252 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:15:55 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:15:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:15:56.284 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:15:56.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:56.310 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:56.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:15:56.412 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:15:56.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:56.436 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:56.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:15:56.537 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:15:56.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:56.561 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:56.562 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:15:56.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:56 smithi100 ceph-mon[29092]: 5.11 deep-scrub starts 2024-09-17T07:15:56.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:56 smithi100 ceph-mon[29092]: 5.11 deep-scrub ok 2024-09-17T07:15:56.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:56 smithi100 ceph-mon[29092]: 6.6 scrub starts 2024-09-17T07:15:56.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:56 smithi100 ceph-mon[29092]: 6.6 scrub ok 2024-09-17T07:15:56.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:56 smithi100 ceph-mon[29092]: 2.0 deep-scrub starts 2024-09-17T07:15:56.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:56 smithi100 ceph-mon[29092]: 2.0 deep-scrub ok 2024-09-17T07:15:56.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:56 smithi100 ceph-mon[29092]: 2.3 scrub starts 2024-09-17T07:15:56.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:56 smithi100 ceph-mon[29092]: 2.3 scrub ok 2024-09-17T07:15:56.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:56 smithi100 ceph-mon[29092]: from='client.15603 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:15:56.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:56 smithi100 ceph-mon[39462]: 5.11 deep-scrub starts 2024-09-17T07:15:56.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:56 smithi100 ceph-mon[39462]: 5.11 deep-scrub ok 2024-09-17T07:15:56.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:56 smithi100 ceph-mon[39462]: 6.6 scrub starts 2024-09-17T07:15:56.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:56 smithi100 ceph-mon[39462]: 6.6 scrub ok 2024-09-17T07:15:56.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:56 smithi100 ceph-mon[39462]: 2.0 deep-scrub starts 2024-09-17T07:15:56.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:56 smithi100 ceph-mon[39462]: 2.0 deep-scrub ok 2024-09-17T07:15:56.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:56 smithi100 ceph-mon[39462]: 2.3 scrub starts 2024-09-17T07:15:56.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:56 smithi100 ceph-mon[39462]: 2.3 scrub ok 2024-09-17T07:15:56.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:56 smithi100 ceph-mon[39462]: from='client.15603 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:15:56.663 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:15:56.686 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:56.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:56.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:15:56.790 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:15:56.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:56 smithi135 ceph-mon[33238]: 5.11 deep-scrub starts 2024-09-17T07:15:56.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:56 smithi135 ceph-mon[33238]: 5.11 deep-scrub ok 2024-09-17T07:15:56.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:56 smithi135 ceph-mon[33238]: 6.6 scrub starts 2024-09-17T07:15:56.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:56 smithi135 ceph-mon[33238]: 6.6 scrub ok 2024-09-17T07:15:56.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:56 smithi135 ceph-mon[33238]: 2.0 deep-scrub starts 2024-09-17T07:15:56.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:56 smithi135 ceph-mon[33238]: 2.0 deep-scrub ok 2024-09-17T07:15:56.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:56 smithi135 ceph-mon[33238]: 2.3 scrub starts 2024-09-17T07:15:56.795 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:56 smithi135 ceph-mon[33238]: 2.3 scrub ok 2024-09-17T07:15:56.795 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:56 smithi135 ceph-mon[33238]: from='client.15603 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:15:56.812 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:56.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:56.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:15:56.914 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:15:56.937 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:56.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:56.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:15:57.039 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:15:57.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:57.063 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:57.063 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:15:57.164 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:15:57.186 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:57.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:57.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:15:57.289 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:15:57.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:57.312 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:57.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:15:57.413 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:15:57.435 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:57.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:57.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:15:57.537 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:15:57.559 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:57.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:57.560 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:15:57.661 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:15:57.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:57.706 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:57.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:15:57.713 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:57 smithi100 ceph-mon[29092]: 5.0 scrub starts 2024-09-17T07:15:57.713 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:57 smithi100 ceph-mon[29092]: 5.0 scrub ok 2024-09-17T07:15:57.713 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:57 smithi100 ceph-mon[29092]: 5.4 scrub starts 2024-09-17T07:15:57.714 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:57 smithi100 ceph-mon[29092]: 5.4 scrub ok 2024-09-17T07:15:57.714 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:57 smithi100 ceph-mon[29092]: 2.1 scrub starts 2024-09-17T07:15:57.714 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:57 smithi100 ceph-mon[29092]: 2.1 scrub ok 2024-09-17T07:15:57.714 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:57 smithi100 ceph-mon[29092]: pgmap v614: 132 pgs: 132 active+clean; 582 KiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 511 B/s rd, 0 op/s 2024-09-17T07:15:57.714 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:57 smithi100 ceph-mon[39462]: 5.0 scrub starts 2024-09-17T07:15:57.714 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:57 smithi100 ceph-mon[39462]: 5.0 scrub ok 2024-09-17T07:15:57.714 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:57 smithi100 ceph-mon[39462]: 5.4 scrub starts 2024-09-17T07:15:57.714 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:57 smithi100 ceph-mon[39462]: 5.4 scrub ok 2024-09-17T07:15:57.714 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:57 smithi100 ceph-mon[39462]: 2.1 scrub starts 2024-09-17T07:15:57.714 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:57 smithi100 ceph-mon[39462]: 2.1 scrub ok 2024-09-17T07:15:57.714 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:57 smithi100 ceph-mon[39462]: pgmap v614: 132 pgs: 132 active+clean; 582 KiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 511 B/s rd, 0 op/s 2024-09-17T07:15:57.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:57 smithi135 ceph-mon[33238]: 5.0 scrub starts 2024-09-17T07:15:57.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:57 smithi135 ceph-mon[33238]: 5.0 scrub ok 2024-09-17T07:15:57.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:57 smithi135 ceph-mon[33238]: 5.4 scrub starts 2024-09-17T07:15:57.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:57 smithi135 ceph-mon[33238]: 5.4 scrub ok 2024-09-17T07:15:57.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:57 smithi135 ceph-mon[33238]: 2.1 scrub starts 2024-09-17T07:15:57.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:57 smithi135 ceph-mon[33238]: 2.1 scrub ok 2024-09-17T07:15:57.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:57 smithi135 ceph-mon[33238]: pgmap v614: 132 pgs: 132 active+clean; 582 KiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 511 B/s rd, 0 op/s 2024-09-17T07:15:57.807 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:15:57.813 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-17T07:15:57.813 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:15:57.830 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:57.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:57.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:15:57.932 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:15:57.955 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:57.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:57.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:15:58.057 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:15:58.078 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:58.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:58.079 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:15:58.180 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:15:58.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:58.203 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:58.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:15:58.304 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:15:58.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:58.325 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:58.326 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:15:58.426 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:15:58.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:58.452 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:58.452 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:15:58.553 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:15:58.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:58.576 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:58.576 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:15:58.585 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:58 smithi100 ceph-mon[29092]: 5.2 scrub starts 2024-09-17T07:15:58.585 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:58 smithi100 ceph-mon[29092]: 5.2 scrub ok 2024-09-17T07:15:58.585 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:58 smithi100 ceph-mon[29092]: 2.6 deep-scrub starts 2024-09-17T07:15:58.585 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:58 smithi100 ceph-mon[29092]: 2.6 deep-scrub ok 2024-09-17T07:15:58.585 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:58 smithi100 ceph-mon[29092]: 6.1e scrub starts 2024-09-17T07:15:58.586 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:58 smithi100 ceph-mon[29092]: 6.1e scrub ok 2024-09-17T07:15:58.586 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:58 smithi100 ceph-mon[39462]: 5.2 scrub starts 2024-09-17T07:15:58.586 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:58 smithi100 ceph-mon[39462]: 5.2 scrub ok 2024-09-17T07:15:58.586 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:58 smithi100 ceph-mon[39462]: 2.6 deep-scrub starts 2024-09-17T07:15:58.586 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:58 smithi100 ceph-mon[39462]: 2.6 deep-scrub ok 2024-09-17T07:15:58.586 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:58 smithi100 ceph-mon[39462]: 6.1e scrub starts 2024-09-17T07:15:58.586 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:58 smithi100 ceph-mon[39462]: 6.1e scrub ok 2024-09-17T07:15:58.606 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-17T07:15:58.677 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:15:58.710 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:58.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:58.711 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:15:58.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:58 smithi135 ceph-mon[33238]: 5.2 scrub starts 2024-09-17T07:15:58.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:58 smithi135 ceph-mon[33238]: 5.2 scrub ok 2024-09-17T07:15:58.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:58 smithi135 ceph-mon[33238]: 2.6 deep-scrub starts 2024-09-17T07:15:58.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:58 smithi135 ceph-mon[33238]: 2.6 deep-scrub ok 2024-09-17T07:15:58.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:58 smithi135 ceph-mon[33238]: 6.1e scrub starts 2024-09-17T07:15:58.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:58 smithi135 ceph-mon[33238]: 6.1e scrub ok 2024-09-17T07:15:58.812 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:15:58.838 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:58.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:58.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:15:58.941 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:15:58.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:58.968 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:58.968 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:15:59.069 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:15:59.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:59.093 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:59.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:15:59.102 INFO:teuthology.orchestra.run.smithi100.stdout:{"pg_ready":true,"pg_map":{"version":613,"stamp":"2024-09-17T07:15:55.280669+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596246,"num_objects":196,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":2158,"num_read_kb":2004,"num_write":895,"num_write_kb":2459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":123,"num_bytes_recovered":1773876,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":941,"ondisk_log_size":941,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":401,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":3809048,"kb_used_data":7088,"kb_used_omap":10,"kb_used_meta":3801845,"kb_avail":652407016,"statfs":{"total":671965249536,"available":668064784384,"internally_reserved":0,"allocated":7258112,"data_stored":4546297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11152,"internal_metadata":3893089392},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":68,"apply_latency_ms":68,"commit_latency_ns":68000000,"apply_latency_ns":68000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":-268528,"num_write_kb":-8592833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001852"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":135,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.079827+0000","last_change":"2024-09-17T07:13:11.978300+0000","last_active":"2024-09-17T07:15:51.079827+0000","last_peered":"2024-09-17T07:15:51.079827+0000","last_clean":"2024-09-17T07:15:51.079827+0000","last_became_active":"2024-09-17T07:09:24.401344+0000","last_became_peered":"2024-09-17T07:09:24.401344+0000","last_unstale":"2024-09-17T07:15:51.079827+0000","last_undegraded":"2024-09-17T07:15:51.079827+0000","last_fullsized":"2024-09-17T07:15:51.079827+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:11.978185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:33.644728+0000","last_clean_scrub_stamp":"2024-09-17T07:13:11.978185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0074436570000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":296,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.078264+0000","last_change":"2024-09-17T07:13:54.566881+0000","last_active":"2024-09-17T07:15:51.078264+0000","last_peered":"2024-09-17T07:15:51.078264+0000","last_clean":"2024-09-17T07:15:51.078264+0000","last_became_active":"2024-09-17T07:02:12.006754+0000","last_became_peered":"2024-09-17T07:02:12.006754+0000","last_unstale":"2024-09-17T07:15:51.078264+0000","last_undegraded":"2024-09-17T07:15:51.078264+0000","last_fullsized":"2024-09-17T07:15:51.078264+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:54.566757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:13:54.566757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0074097110000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"242'27","reported_seq":206,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.078066+0000","last_change":"2024-09-17T07:14:47.661623+0000","last_active":"2024-09-17T07:15:51.078066+0000","last_peered":"2024-09-17T07:15:51.078066+0000","last_clean":"2024-09-17T07:15:51.078066+0000","last_became_active":"2024-09-17T07:09:24.402042+0000","last_became_peered":"2024-09-17T07:09:24.402042+0000","last_unstale":"2024-09-17T07:15:51.078066+0000","last_undegraded":"2024-09-17T07:15:51.078066+0000","last_fullsized":"2024-09-17T07:15:51.078066+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'27","last_scrub_stamp":"2024-09-17T07:14:47.661187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:14:47.661187+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:05.925671+0000","scrub_duration":0.010587222,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":292,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:50.394001+0000","last_change":"2024-09-17T07:15:45.261193+0000","last_active":"2024-09-17T07:15:50.394001+0000","last_peered":"2024-09-17T07:15:50.394001+0000","last_clean":"2024-09-17T07:15:50.394001+0000","last_became_active":"2024-09-17T07:02:16.014928+0000","last_became_peered":"2024-09-17T07:02:16.014928+0000","last_unstale":"2024-09-17T07:15:50.394001+0000","last_undegraded":"2024-09-17T07:15:50.394001+0000","last_fullsized":"2024-09-17T07:15:50.394001+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:45.261117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:15:45.261117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:00.371170+0000","scrub_duration":0.001477482,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":147,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.079187+0000","last_change":"2024-09-17T07:14:57.520400+0000","last_active":"2024-09-17T07:15:51.079187+0000","last_peered":"2024-09-17T07:15:51.079187+0000","last_clean":"2024-09-17T07:15:51.079187+0000","last_became_active":"2024-09-17T07:09:24.402568+0000","last_became_peered":"2024-09-17T07:09:24.402568+0000","last_unstale":"2024-09-17T07:15:51.079187+0000","last_undegraded":"2024-09-17T07:15:51.079187+0000","last_fullsized":"2024-09-17T07:15:51.079187+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:57.520290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:14:57.520290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:15.653444+0000","scrub_duration":0.001539143,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"242'15","reported_seq":330,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:50.394052+0000","last_change":"2024-09-17T07:15:39.306155+0000","last_active":"2024-09-17T07:15:50.394052+0000","last_peered":"2024-09-17T07:15:50.394052+0000","last_clean":"2024-09-17T07:15:50.394052+0000","last_became_active":"2024-09-17T07:02:14.016244+0000","last_became_peered":"2024-09-17T07:02:14.016244+0000","last_unstale":"2024-09-17T07:15:50.394052+0000","last_undegraded":"2024-09-17T07:15:50.394052+0000","last_fullsized":"2024-09-17T07:15:50.394052+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'15","last_scrub_stamp":"2024-09-17T07:15:39.306043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:15:39.306043+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:57.372120+0000","scrub_duration":0.002471029,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":280,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:50.395450+0000","last_change":"2024-09-17T07:13:17.184439+0000","last_active":"2024-09-17T07:15:50.395450+0000","last_peered":"2024-09-17T07:15:50.395450+0000","last_clean":"2024-09-17T07:15:50.395450+0000","last_became_active":"2024-09-17T07:02:16.014380+0000","last_became_peered":"2024-09-17T07:02:16.014380+0000","last_unstale":"2024-09-17T07:15:50.395450+0000","last_undegraded":"2024-09-17T07:15:50.395450+0000","last_fullsized":"2024-09-17T07:15:50.395450+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:17.184312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:13:17.184312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0075305299999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":291,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:50.394170+0000","last_change":"2024-09-17T07:14:41.708103+0000","last_active":"2024-09-17T07:15:50.394170+0000","last_peered":"2024-09-17T07:15:50.394170+0000","last_clean":"2024-09-17T07:15:50.394170+0000","last_became_active":"2024-09-17T07:02:18.020790+0000","last_became_peered":"2024-09-17T07:02:18.020790+0000","last_unstale":"2024-09-17T07:15:50.394170+0000","last_undegraded":"2024-09-17T07:15:50.394170+0000","last_fullsized":"2024-09-17T07:15:50.394170+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:41.707969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:14:41.707969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:15:51.720634+0000","scrub_duration":0.014931757,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":294,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.079637+0000","last_change":"2024-09-17T07:15:46.312474+0000","last_active":"2024-09-17T07:15:51.079637+0000","last_peered":"2024-09-17T07:15:51.079637+0000","last_clean":"2024-09-17T07:15:51.079637+0000","last_became_active":"2024-09-17T07:02:12.011129+0000","last_became_peered":"2024-09-17T07:02:12.011129+0000","last_unstale":"2024-09-17T07:15:51.079637+0000","last_undegraded":"2024-09-17T07:15:51.079637+0000","last_fullsized":"2024-09-17T07:15:51.079637+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:46.312371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:46.312371+0000","last_clean_scrub_stamp":"2024-09-17T07:15:46.312371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:06.831380+0000","scrub_duration":0.0015176510000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"242'7","reported_seq":323,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:50.394328+0000","last_change":"2024-09-17T07:14:42.653845+0000","last_active":"2024-09-17T07:15:50.394328+0000","last_peered":"2024-09-17T07:15:50.394328+0000","last_clean":"2024-09-17T07:15:50.394328+0000","last_became_active":"2024-09-17T07:09:24.396900+0000","last_became_peered":"2024-09-17T07:09:24.396900+0000","last_unstale":"2024-09-17T07:15:50.394328+0000","last_undegraded":"2024-09-17T07:15:50.394328+0000","last_fullsized":"2024-09-17T07:15:50.394328+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'7","last_scrub_stamp":"2024-09-17T07:14:42.653705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:14:42.653705+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:15:56.219399+0000","scrub_duration":0.0050149230000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":290,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.080184+0000","last_change":"2024-09-17T07:14:38.000519+0000","last_active":"2024-09-17T07:15:51.080184+0000","last_peered":"2024-09-17T07:15:51.080184+0000","last_clean":"2024-09-17T07:15:51.080184+0000","last_became_active":"2024-09-17T07:02:16.015401+0000","last_became_peered":"2024-09-17T07:02:16.015401+0000","last_unstale":"2024-09-17T07:15:51.080184+0000","last_undegraded":"2024-09-17T07:15:51.080184+0000","last_fullsized":"2024-09-17T07:15:51.080184+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:37.997296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:14:37.997296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045486420000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":309,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.078911+0000","last_change":"2024-09-17T07:14:52.168308+0000","last_active":"2024-09-17T07:15:51.078911+0000","last_peered":"2024-09-17T07:15:51.078911+0000","last_clean":"2024-09-17T07:15:51.078911+0000","last_became_active":"2024-09-17T07:09:24.402616+0000","last_became_peered":"2024-09-17T07:09:24.402616+0000","last_unstale":"2024-09-17T07:15:51.078911+0000","last_undegraded":"2024-09-17T07:15:51.078911+0000","last_fullsized":"2024-09-17T07:15:51.078911+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:52.168181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:14:52.168181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:03.969726+0000","scrub_duration":0.0056501399999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":294,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.078800+0000","last_change":"2024-09-17T07:14:44.076126+0000","last_active":"2024-09-17T07:15:51.078800+0000","last_peered":"2024-09-17T07:15:51.078800+0000","last_clean":"2024-09-17T07:15:51.078800+0000","last_became_active":"2024-09-17T07:02:12.012234+0000","last_became_peered":"2024-09-17T07:02:12.012234+0000","last_unstale":"2024-09-17T07:15:51.078800+0000","last_undegraded":"2024-09-17T07:15:51.078800+0000","last_fullsized":"2024-09-17T07:15:51.078800+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:44.076012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:45.029247+0000","last_clean_scrub_stamp":"2024-09-17T07:14:44.076012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.015917521,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"242'15","reported_seq":338,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:50.394354+0000","last_change":"2024-09-17T07:15:36.373317+0000","last_active":"2024-09-17T07:15:50.394354+0000","last_peered":"2024-09-17T07:15:50.394354+0000","last_clean":"2024-09-17T07:15:50.394354+0000","last_became_active":"2024-09-17T07:09:24.397954+0000","last_became_peered":"2024-09-17T07:09:24.397954+0000","last_unstale":"2024-09-17T07:15:50.394354+0000","last_undegraded":"2024-09-17T07:15:50.394354+0000","last_fullsized":"2024-09-17T07:15:50.394354+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'15","last_scrub_stamp":"2024-09-17T07:15:36.373220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:15:36.373220+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:37.392728+0000","scrub_duration":0.0019745959999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":301,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.080021+0000","last_change":"2024-09-17T07:14:59.140862+0000","last_active":"2024-09-17T07:15:51.080021+0000","last_peered":"2024-09-17T07:15:51.080021+0000","last_clean":"2024-09-17T07:15:51.080021+0000","last_became_active":"2024-09-17T07:02:16.023774+0000","last_became_peered":"2024-09-17T07:02:16.023774+0000","last_unstale":"2024-09-17T07:15:51.080021+0000","last_undegraded":"2024-09-17T07:15:51.080021+0000","last_fullsized":"2024-09-17T07:15:51.080021+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:59.140673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:28.096266+0000","last_clean_scrub_stamp":"2024-09-17T07:14:59.140673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:23.008311+0000","scrub_duration":0.013656697000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":290,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.077527+0000","last_change":"2024-09-17T07:15:45.483700+0000","last_active":"2024-09-17T07:15:51.077527+0000","last_peered":"2024-09-17T07:15:51.077527+0000","last_clean":"2024-09-17T07:15:51.077527+0000","last_became_active":"2024-09-17T07:02:18.029178+0000","last_became_peered":"2024-09-17T07:02:18.029178+0000","last_unstale":"2024-09-17T07:15:51.077527+0000","last_undegraded":"2024-09-17T07:15:51.077527+0000","last_fullsized":"2024-09-17T07:15:51.077527+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:45.483622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:15:45.483622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:12.943923+0000","scrub_duration":0.001342359,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"242'39","reported_seq":363,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:50.394695+0000","last_change":"2024-09-17T07:15:46.226654+0000","last_active":"2024-09-17T07:15:50.394695+0000","last_peered":"2024-09-17T07:15:50.394695+0000","last_clean":"2024-09-17T07:15:50.394695+0000","last_became_active":"2024-09-17T07:02:14.016270+0000","last_became_peered":"2024-09-17T07:02:14.016270+0000","last_unstale":"2024-09-17T07:15:50.394695+0000","last_undegraded":"2024-09-17T07:15:50.394695+0000","last_fullsized":"2024-09-17T07:15:50.394695+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'39","last_scrub_stamp":"2024-09-17T07:15:46.226554+0000","last_deep_scrub":"95'25","last_deep_scrub_stamp":"2024-09-17T07:04:09.450039+0000","last_clean_scrub_stamp":"2024-09-17T07:15:46.226554+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:02.790391+0000","scrub_duration":0.0027061799999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":297,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.080230+0000","last_change":"2024-09-17T07:13:25.951693+0000","last_active":"2024-09-17T07:15:51.080230+0000","last_peered":"2024-09-17T07:15:51.080230+0000","last_clean":"2024-09-17T07:15:51.080230+0000","last_became_active":"2024-09-17T07:02:12.011258+0000","last_became_peered":"2024-09-17T07:02:12.011258+0000","last_unstale":"2024-09-17T07:15:51.080230+0000","last_undegraded":"2024-09-17T07:15:51.080230+0000","last_fullsized":"2024-09-17T07:15:51.080230+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:25.950509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:13:25.950509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0099987270000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":281,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:50.396709+0000","last_change":"2024-09-17T07:13:46.502789+0000","last_active":"2024-09-17T07:15:50.396709+0000","last_peered":"2024-09-17T07:15:50.396709+0000","last_clean":"2024-09-17T07:15:50.396709+0000","last_became_active":"2024-09-17T07:02:16.018147+0000","last_became_peered":"2024-09-17T07:02:16.018147+0000","last_unstale":"2024-09-17T07:15:50.396709+0000","last_undegraded":"2024-09-17T07:15:50.396709+0000","last_fullsized":"2024-09-17T07:15:50.396709+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:46.502641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:13:46.502641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0074335069999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":138,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:50.394602+0000","last_change":"2024-09-17T07:14:38.777656+0000","last_active":"2024-09-17T07:15:50.394602+0000","last_peered":"2024-09-17T07:15:50.394602+0000","last_clean":"2024-09-17T07:15:50.394602+0000","last_became_active":"2024-09-17T07:09:24.396334+0000","last_became_peered":"2024-09-17T07:09:24.396334+0000","last_unstale":"2024-09-17T07:15:50.394602+0000","last_undegraded":"2024-09-17T07:15:50.394602+0000","last_fullsized":"2024-09-17T07:15:50.394602+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:38.777538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:42.984088+0000","last_clean_scrub_stamp":"2024-09-17T07:14:38.777538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003669088,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"242'25","reported_seq":362,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:50.394759+0000","last_change":"2024-09-17T07:15:02.535674+0000","last_active":"2024-09-17T07:15:50.394759+0000","last_peered":"2024-09-17T07:15:50.394759+0000","last_clean":"2024-09-17T07:15:50.394759+0000","last_became_active":"2024-09-17T07:09:24.399287+0000","last_became_peered":"2024-09-17T07:09:24.399287+0000","last_unstale":"2024-09-17T07:15:50.394759+0000","last_undegraded":"2024-09-17T07:15:50.394759+0000","last_fullsized":"2024-09-17T07:15:50.394759+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'25","last_scrub_stamp":"2024-09-17T07:15:02.532371+0000","last_deep_scrub":"176'15","last_deep_scrub_stamp":"2024-09-17T07:08:19.633791+0000","last_clean_scrub_stamp":"2024-09-17T07:15:02.532371+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:25.841604+0000","scrub_duration":0.012712606,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":304,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:50.396953+0000","last_change":"2024-09-17T07:14:59.697649+0000","last_active":"2024-09-17T07:15:50.396953+0000","last_peered":"2024-09-17T07:15:50.396953+0000","last_clean":"2024-09-17T07:15:50.396953+0000","last_became_active":"2024-09-17T07:09:24.406973+0000","last_became_peered":"2024-09-17T07:09:24.406973+0000","last_unstale":"2024-09-17T07:15:50.396953+0000","last_undegraded":"2024-09-17T07:15:50.396953+0000","last_fullsized":"2024-09-17T07:15:50.396953+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:59.697560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:14:59.697560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:19.402327+0000","scrub_duration":0.003544859,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"98'9","reported_seq":415,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.079173+0000","last_change":"2024-09-17T07:14:46.126177+0000","last_active":"2024-09-17T07:15:51.079173+0000","last_peered":"2024-09-17T07:15:51.079173+0000","last_clean":"2024-09-17T07:15:51.079173+0000","last_became_active":"2024-09-17T07:04:15.110797+0000","last_became_peered":"2024-09-17T07:04:15.110797+0000","last_unstale":"2024-09-17T07:15:51.079173+0000","last_undegraded":"2024-09-17T07:15:51.079173+0000","last_fullsized":"2024-09-17T07:15:51.079173+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-09-17T07:14:46.126045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:14:46.126045+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:11.141681+0000","scrub_duration":0.014185467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":302,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.400663+0000","last_change":"2024-09-17T07:14:50.554927+0000","last_active":"2024-09-17T07:15:51.400663+0000","last_peered":"2024-09-17T07:15:51.400663+0000","last_clean":"2024-09-17T07:15:51.400663+0000","last_became_active":"2024-09-17T07:02:18.017460+0000","last_became_peered":"2024-09-17T07:02:18.017460+0000","last_unstale":"2024-09-17T07:15:51.400663+0000","last_undegraded":"2024-09-17T07:15:51.400663+0000","last_fullsized":"2024-09-17T07:15:51.400663+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:50.554591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:14:50.554591+0000","last_clean_scrub_stamp":"2024-09-17T07:14:50.554591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:14.252478+0000","scrub_duration":0.0067125969999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"242'27","reported_seq":361,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:50.396475+0000","last_change":"2024-09-17T07:14:45.832566+0000","last_active":"2024-09-17T07:15:50.396475+0000","last_peered":"2024-09-17T07:15:50.396475+0000","last_clean":"2024-09-17T07:15:50.396475+0000","last_became_active":"2024-09-17T07:09:24.407311+0000","last_became_peered":"2024-09-17T07:09:24.407311+0000","last_unstale":"2024-09-17T07:15:50.396475+0000","last_undegraded":"2024-09-17T07:15:50.396475+0000","last_fullsized":"2024-09-17T07:15:50.396475+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'27","last_scrub_stamp":"2024-09-17T07:14:45.831734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:14:45.831734+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0075085209999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":305,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:50.395690+0000","last_change":"2024-09-17T07:15:44.897692+0000","last_active":"2024-09-17T07:15:50.395690+0000","last_peered":"2024-09-17T07:15:50.395690+0000","last_clean":"2024-09-17T07:15:50.395690+0000","last_became_active":"2024-09-17T07:09:24.405655+0000","last_became_peered":"2024-09-17T07:09:24.405655+0000","last_unstale":"2024-09-17T07:15:50.395690+0000","last_undegraded":"2024-09-17T07:15:50.395690+0000","last_fullsized":"2024-09-17T07:15:50.395690+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:44.897611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:15:44.897611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:51.136422+0000","scrub_duration":0.0030827340000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"75'8","reported_seq":465,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:50.683290+0000","last_change":"2024-09-17T07:15:40.259961+0000","last_active":"2024-09-17T07:15:50.683290+0000","last_peered":"2024-09-17T07:15:50.683290+0000","last_clean":"2024-09-17T07:15:50.683290+0000","last_became_active":"2024-09-17T07:02:16.021762+0000","last_became_peered":"2024-09-17T07:02:16.021762+0000","last_unstale":"2024-09-17T07:15:50.683290+0000","last_undegraded":"2024-09-17T07:15:50.683290+0000","last_fullsized":"2024-09-17T07:15:50.683290+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:15:40.259852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:15:40.259852+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:44.641715+0000","scrub_duration":0.001523993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":300,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.079910+0000","last_change":"2024-09-17T07:14:56.146344+0000","last_active":"2024-09-17T07:15:51.079910+0000","last_peered":"2024-09-17T07:15:51.079910+0000","last_clean":"2024-09-17T07:15:51.079910+0000","last_became_active":"2024-09-17T07:02:18.025332+0000","last_became_peered":"2024-09-17T07:02:18.025332+0000","last_unstale":"2024-09-17T07:15:51.079910+0000","last_undegraded":"2024-09-17T07:15:51.079910+0000","last_fullsized":"2024-09-17T07:15:51.079910+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:56.146098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:14:56.146098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:15:57.984867+0000","scrub_duration":0.0054479960000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"242'19","reported_seq":337,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:50.394763+0000","last_change":"2024-09-17T07:14:55.490025+0000","last_active":"2024-09-17T07:15:50.394763+0000","last_peered":"2024-09-17T07:15:50.394763+0000","last_clean":"2024-09-17T07:15:50.394763+0000","last_became_active":"2024-09-17T07:02:14.010227+0000","last_became_peered":"2024-09-17T07:02:14.010227+0000","last_unstale":"2024-09-17T07:15:50.394763+0000","last_undegraded":"2024-09-17T07:15:50.394763+0000","last_fullsized":"2024-09-17T07:15:50.394763+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'19","last_scrub_stamp":"2024-09-17T07:14:55.489874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:14:55.489874+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:23.724406+0000","scrub_duration":0.0045924440000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":296,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.080577+0000","last_change":"2024-09-17T07:13:21.858251+0000","last_active":"2024-09-17T07:15:51.080577+0000","last_peered":"2024-09-17T07:15:51.080577+0000","last_clean":"2024-09-17T07:15:51.080577+0000","last_became_active":"2024-09-17T07:02:12.013182+0000","last_became_peered":"2024-09-17T07:02:12.013182+0000","last_unstale":"2024-09-17T07:15:51.080577+0000","last_undegraded":"2024-09-17T07:15:51.080577+0000","last_fullsized":"2024-09-17T07:15:51.080577+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:21.858127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:13:21.858127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033279709999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":313,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-17T07:15:53.424831+0000","last_change":"2024-09-17T07:15:53.424831+0000","last_active":"2024-09-17T07:15:53.424831+0000","last_peered":"2024-09-17T07:15:53.424831+0000","last_clean":"2024-09-17T07:15:53.424831+0000","last_became_active":"2024-09-17T07:02:16.020224+0000","last_became_peered":"2024-09-17T07:02:16.020224+0000","last_unstale":"2024-09-17T07:15:53.424831+0000","last_undegraded":"2024-09-17T07:15:53.424831+0000","last_fullsized":"2024-09-17T07:15:53.424831+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:53.424736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:15:53.424736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:53.871415+0000","scrub_duration":0.001270806,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":156,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.078699+0000","last_change":"2024-09-17T07:14:51.530956+0000","last_active":"2024-09-17T07:15:51.078699+0000","last_peered":"2024-09-17T07:15:51.078699+0000","last_clean":"2024-09-17T07:15:51.078699+0000","last_became_active":"2024-09-17T07:09:24.408109+0000","last_became_peered":"2024-09-17T07:09:24.408109+0000","last_unstale":"2024-09-17T07:15:51.078699+0000","last_undegraded":"2024-09-17T07:15:51.078699+0000","last_fullsized":"2024-09-17T07:15:51.078699+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:51.530841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:13:22.187127+0000","last_clean_scrub_stamp":"2024-09-17T07:14:51.530841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:17.322714+0000","scrub_duration":0.0027489939999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"242'28","reported_seq":337,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:50.394135+0000","last_change":"2024-09-17T07:15:43.270581+0000","last_active":"2024-09-17T07:15:50.394135+0000","last_peered":"2024-09-17T07:15:50.394135+0000","last_clean":"2024-09-17T07:15:50.394135+0000","last_became_active":"2024-09-17T07:02:14.007810+0000","last_became_peered":"2024-09-17T07:02:14.007810+0000","last_unstale":"2024-09-17T07:15:50.394135+0000","last_undegraded":"2024-09-17T07:15:50.394135+0000","last_fullsized":"2024-09-17T07:15:50.394135+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'28","last_scrub_stamp":"2024-09-17T07:15:43.270473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:15:43.270473+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:07.514992+0000","scrub_duration":0.0022765149999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":328,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.079653+0000","last_change":"2024-09-17T07:14:53.544359+0000","last_active":"2024-09-17T07:15:51.079653+0000","last_peered":"2024-09-17T07:15:51.079653+0000","last_clean":"2024-09-17T07:15:51.079653+0000","last_became_active":"2024-09-17T07:02:12.011766+0000","last_became_peered":"2024-09-17T07:02:12.011766+0000","last_unstale":"2024-09-17T07:15:51.079653+0000","last_undegraded":"2024-09-17T07:15:51.079653+0000","last_fullsized":"2024-09-17T07:15:51.079653+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:53.544229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:09:23.181028+0000","last_clean_scrub_stamp":"2024-09-17T07:14:53.544229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:15:57.913689+0000","scrub_duration":0.011986663999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"196'9","reported_seq":226,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:50.683117+0000","last_change":"2024-09-17T07:13:12.145244+0000","last_active":"2024-09-17T07:15:50.683117+0000","last_peered":"2024-09-17T07:15:50.683117+0000","last_clean":"2024-09-17T07:15:50.683117+0000","last_became_active":"2024-09-17T07:09:24.406056+0000","last_became_peered":"2024-09-17T07:09:24.406056+0000","last_unstale":"2024-09-17T07:15:50.683117+0000","last_undegraded":"2024-09-17T07:15:50.683117+0000","last_fullsized":"2024-09-17T07:15:50.683117+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'9","last_scrub_stamp":"2024-09-17T07:13:12.145158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:13:12.145158+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001480777,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":300,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-17T07:15:52.378408+0000","last_change":"2024-09-17T07:15:52.378479+0000","last_active":"2024-09-17T07:15:52.378408+0000","last_peered":"2024-09-17T07:15:52.378408+0000","last_clean":"2024-09-17T07:15:52.378408+0000","last_became_active":"2024-09-17T07:02:18.364981+0000","last_became_peered":"2024-09-17T07:02:18.364981+0000","last_unstale":"2024-09-17T07:15:52.378408+0000","last_undegraded":"2024-09-17T07:15:52.378408+0000","last_fullsized":"2024-09-17T07:15:52.378408+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:52.378384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:10:33.727472+0000","last_clean_scrub_stamp":"2024-09-17T07:15:52.378384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:07.710387+0000","scrub_duration":0.001396736,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"242'30","reported_seq":215,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.080473+0000","last_change":"2024-09-17T07:13:28.997698+0000","last_active":"2024-09-17T07:15:51.080473+0000","last_peered":"2024-09-17T07:15:51.080473+0000","last_clean":"2024-09-17T07:15:51.080473+0000","last_became_active":"2024-09-17T07:09:24.403240+0000","last_became_peered":"2024-09-17T07:09:24.403240+0000","last_unstale":"2024-09-17T07:15:51.080473+0000","last_undegraded":"2024-09-17T07:15:51.080473+0000","last_fullsized":"2024-09-17T07:15:51.080473+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'30","last_scrub_stamp":"2024-09-17T07:13:28.997563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:13:28.997563+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0061961159999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":295,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:50.397416+0000","last_change":"2024-09-17T07:13:23.253165+0000","last_active":"2024-09-17T07:15:50.397416+0000","last_peered":"2024-09-17T07:15:50.397416+0000","last_clean":"2024-09-17T07:15:50.397416+0000","last_became_active":"2024-09-17T07:02:12.005687+0000","last_became_peered":"2024-09-17T07:02:12.005687+0000","last_unstale":"2024-09-17T07:15:50.397416+0000","last_undegraded":"2024-09-17T07:15:50.397416+0000","last_fullsized":"2024-09-17T07:15:50.397416+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:23.253036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:13:23.253036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002831362,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":302,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:50.394043+0000","last_change":"2024-09-17T07:14:52.606726+0000","last_active":"2024-09-17T07:15:50.394043+0000","last_peered":"2024-09-17T07:15:50.394043+0000","last_clean":"2024-09-17T07:15:50.394043+0000","last_became_active":"2024-09-17T07:09:24.393375+0000","last_became_peered":"2024-09-17T07:09:24.393375+0000","last_unstale":"2024-09-17T07:15:50.394043+0000","last_undegraded":"2024-09-17T07:15:50.394043+0000","last_fullsized":"2024-09-17T07:15:50.394043+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:52.606401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:14:52.606401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:05.230017+0000","scrub_duration":0.0068634849999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":300,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:50.394298+0000","last_change":"2024-09-17T07:14:39.758541+0000","last_active":"2024-09-17T07:15:50.394298+0000","last_peered":"2024-09-17T07:15:50.394298+0000","last_clean":"2024-09-17T07:15:50.394298+0000","last_became_active":"2024-09-17T07:09:24.398659+0000","last_became_peered":"2024-09-17T07:09:24.398659+0000","last_unstale":"2024-09-17T07:15:50.394298+0000","last_undegraded":"2024-09-17T07:15:50.394298+0000","last_fullsized":"2024-09-17T07:15:50.394298+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:39.758427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:14:39.758427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:04.633730+0000","scrub_duration":0.0019183889999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"242'29","reported_seq":346,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.078009+0000","last_change":"2024-09-17T07:15:38.507145+0000","last_active":"2024-09-17T07:15:51.078009+0000","last_peered":"2024-09-17T07:15:51.078009+0000","last_clean":"2024-09-17T07:15:51.078009+0000","last_became_active":"2024-09-17T07:02:14.027389+0000","last_became_peered":"2024-09-17T07:02:14.027389+0000","last_unstale":"2024-09-17T07:15:51.078009+0000","last_undegraded":"2024-09-17T07:15:51.078009+0000","last_fullsized":"2024-09-17T07:15:51.078009+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'29","last_scrub_stamp":"2024-09-17T07:15:38.507042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:15:38.507042+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:40.074077+0000","scrub_duration":0.0024959420000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":296,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.079711+0000","last_change":"2024-09-17T07:13:23.945293+0000","last_active":"2024-09-17T07:15:51.079711+0000","last_peered":"2024-09-17T07:15:51.079711+0000","last_clean":"2024-09-17T07:15:51.079711+0000","last_became_active":"2024-09-17T07:02:12.011917+0000","last_became_peered":"2024-09-17T07:02:12.011917+0000","last_unstale":"2024-09-17T07:15:51.079711+0000","last_undegraded":"2024-09-17T07:15:51.079711+0000","last_fullsized":"2024-09-17T07:15:51.079711+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:23.945157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:24.072799+0000","last_clean_scrub_stamp":"2024-09-17T07:13:23.945157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0029434130000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":303,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.079448+0000","last_change":"2024-09-17T07:14:41.506985+0000","last_active":"2024-09-17T07:15:51.079448+0000","last_peered":"2024-09-17T07:15:51.079448+0000","last_clean":"2024-09-17T07:15:51.079448+0000","last_became_active":"2024-09-17T07:02:16.016199+0000","last_became_peered":"2024-09-17T07:02:16.016199+0000","last_unstale":"2024-09-17T07:15:51.079448+0000","last_undegraded":"2024-09-17T07:15:51.079448+0000","last_fullsized":"2024-09-17T07:15:51.079448+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:41.506851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:11:17.583712+0000","last_clean_scrub_stamp":"2024-09-17T07:14:41.506851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:15:54.688256+0000","scrub_duration":0.0087014999999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":289,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:50.395549+0000","last_change":"2024-09-17T07:13:50.514634+0000","last_active":"2024-09-17T07:15:50.395549+0000","last_peered":"2024-09-17T07:15:50.395549+0000","last_clean":"2024-09-17T07:15:50.395549+0000","last_became_active":"2024-09-17T07:02:18.364835+0000","last_became_peered":"2024-09-17T07:02:18.364835+0000","last_unstale":"2024-09-17T07:15:50.395549+0000","last_undegraded":"2024-09-17T07:15:50.395549+0000","last_fullsized":"2024-09-17T07:15:50.395549+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:50.514525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:13:50.514525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010637423,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"242'25","reported_seq":328,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:50.396834+0000","last_change":"2024-09-17T07:14:50.794400+0000","last_active":"2024-09-17T07:15:50.396834+0000","last_peered":"2024-09-17T07:15:50.396834+0000","last_clean":"2024-09-17T07:15:50.396834+0000","last_became_active":"2024-09-17T07:02:14.010637+0000","last_became_peered":"2024-09-17T07:02:14.010637+0000","last_unstale":"2024-09-17T07:15:50.396834+0000","last_undegraded":"2024-09-17T07:15:50.396834+0000","last_fullsized":"2024-09-17T07:15:50.396834+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'25","last_scrub_stamp":"2024-09-17T07:14:50.794200+0000","last_deep_scrub":"110'15","last_deep_scrub_stamp":"2024-09-17T07:04:57.091400+0000","last_clean_scrub_stamp":"2024-09-17T07:14:50.794200+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0058730839999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":295,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:50.397095+0000","last_change":"2024-09-17T07:14:49.765133+0000","last_active":"2024-09-17T07:15:50.397095+0000","last_peered":"2024-09-17T07:15:50.397095+0000","last_clean":"2024-09-17T07:15:50.397095+0000","last_became_active":"2024-09-17T07:02:12.010803+0000","last_became_peered":"2024-09-17T07:02:12.010803+0000","last_unstale":"2024-09-17T07:15:50.397095+0000","last_undegraded":"2024-09-17T07:15:50.397095+0000","last_fullsized":"2024-09-17T07:15:50.397095+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:49.764885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:25.147878+0000","last_clean_scrub_stamp":"2024-09-17T07:14:49.764885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:08.217541+0000","scrub_duration":0.002419587,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"89'9","reported_seq":476,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-17T07:15:53.512795+0000","last_change":"2024-09-17T07:15:53.512860+0000","last_active":"2024-09-17T07:15:53.512795+0000","last_peered":"2024-09-17T07:15:53.512795+0000","last_clean":"2024-09-17T07:15:53.512795+0000","last_became_active":"2024-09-17T07:02:16.021360+0000","last_became_peered":"2024-09-17T07:02:16.021360+0000","last_unstale":"2024-09-17T07:15:53.512795+0000","last_undegraded":"2024-09-17T07:15:53.512795+0000","last_fullsized":"2024-09-17T07:15:53.512795+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'9","last_scrub_stamp":"2024-09-17T07:15:53.512777+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-09-17T07:15:53.512777+0000","last_clean_scrub_stamp":"2024-09-17T07:15:53.512777+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:59.615040+0000","scrub_duration":0.0020298600000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"268'2","reported_seq":305,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.077369+0000","last_change":"2024-09-17T07:14:03.609273+0000","last_active":"2024-09-17T07:15:51.077369+0000","last_peered":"2024-09-17T07:15:51.077369+0000","last_clean":"2024-09-17T07:15:51.077369+0000","last_became_active":"2024-09-17T07:09:24.399565+0000","last_became_peered":"2024-09-17T07:09:24.399565+0000","last_unstale":"2024-09-17T07:15:51.077369+0000","last_undegraded":"2024-09-17T07:15:51.077369+0000","last_fullsized":"2024-09-17T07:15:51.077369+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'2","last_scrub_stamp":"2024-09-17T07:14:03.609131+0000","last_deep_scrub":"268'2","last_deep_scrub_stamp":"2024-09-17T07:14:03.609131+0000","last_clean_scrub_stamp":"2024-09-17T07:14:03.609131+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011205872,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"242'41","reported_seq":554,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:50.683159+0000","last_change":"2024-09-17T07:14:56.700945+0000","last_active":"2024-09-17T07:15:50.683159+0000","last_peered":"2024-09-17T07:15:50.683159+0000","last_clean":"2024-09-17T07:15:50.683159+0000","last_became_active":"2024-09-17T07:09:24.403267+0000","last_became_peered":"2024-09-17T07:09:24.403267+0000","last_unstale":"2024-09-17T07:15:50.683159+0000","last_undegraded":"2024-09-17T07:15:50.683159+0000","last_fullsized":"2024-09-17T07:15:50.683159+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'41","last_scrub_stamp":"2024-09-17T07:14:56.700858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:14:56.700858+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:15:56.905149+0000","scrub_duration":0.002102558,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":306,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.079720+0000","last_change":"2024-09-17T07:14:03.046151+0000","last_active":"2024-09-17T07:15:51.079720+0000","last_peered":"2024-09-17T07:15:51.079720+0000","last_clean":"2024-09-17T07:15:51.079720+0000","last_became_active":"2024-09-17T07:02:12.008802+0000","last_became_peered":"2024-09-17T07:02:12.008802+0000","last_unstale":"2024-09-17T07:15:51.079720+0000","last_undegraded":"2024-09-17T07:15:51.079720+0000","last_fullsized":"2024-09-17T07:15:51.079720+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:03.045709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:14:03.045709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0049245929999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":303,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.078242+0000","last_change":"2024-09-17T07:15:36.523759+0000","last_active":"2024-09-17T07:15:51.078242+0000","last_peered":"2024-09-17T07:15:51.078242+0000","last_clean":"2024-09-17T07:15:51.078242+0000","last_became_active":"2024-09-17T07:02:16.016606+0000","last_became_peered":"2024-09-17T07:02:16.016606+0000","last_unstale":"2024-09-17T07:15:51.078242+0000","last_undegraded":"2024-09-17T07:15:51.078242+0000","last_fullsized":"2024-09-17T07:15:51.078242+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:36.523676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:15:36.523676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:58.160341+0000","scrub_duration":0.0017370549999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":135,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:50.394649+0000","last_change":"2024-09-17T07:13:53.406787+0000","last_active":"2024-09-17T07:15:50.394649+0000","last_peered":"2024-09-17T07:15:50.394649+0000","last_clean":"2024-09-17T07:15:50.394649+0000","last_became_active":"2024-09-17T07:09:24.398878+0000","last_became_peered":"2024-09-17T07:09:24.398878+0000","last_unstale":"2024-09-17T07:15:50.394649+0000","last_undegraded":"2024-09-17T07:15:50.394649+0000","last_fullsized":"2024-09-17T07:15:50.394649+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:53.406661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:13:53.406661+0000","last_clean_scrub_stamp":"2024-09-17T07:13:53.406661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0074403789999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.7","version":"242'17","reported_seq":326,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:50.396475+0000","last_change":"2024-09-17T07:13:47.468495+0000","last_active":"2024-09-17T07:15:50.396475+0000","last_peered":"2024-09-17T07:15:50.396475+0000","last_clean":"2024-09-17T07:15:50.396475+0000","last_became_active":"2024-09-17T07:02:14.019838+0000","last_became_peered":"2024-09-17T07:02:14.019838+0000","last_unstale":"2024-09-17T07:15:50.396475+0000","last_undegraded":"2024-09-17T07:15:50.396475+0000","last_fullsized":"2024-09-17T07:15:50.396475+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'17","last_scrub_stamp":"2024-09-17T07:13:47.468399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:13:47.468399+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0060398329999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"76'5","reported_seq":1116,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.963501+0000","last_change":"2024-09-17T07:15:36.901698+0000","last_active":"2024-09-17T07:15:51.963501+0000","last_peered":"2024-09-17T07:15:51.963501+0000","last_clean":"2024-09-17T07:15:51.963501+0000","last_became_active":"2024-09-17T07:02:12.006883+0000","last_became_peered":"2024-09-17T07:02:12.006883+0000","last_unstale":"2024-09-17T07:15:51.963501+0000","last_undegraded":"2024-09-17T07:15:51.963501+0000","last_fullsized":"2024-09-17T07:15:51.963501+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'5","last_scrub_stamp":"2024-09-17T07:15:36.901601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:15:36.901601+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:56.163207+0000","scrub_duration":0.002331086,"objects_trimmed":0,"snaptrim_duration":0.00033739799999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":813,"num_read_kb":808,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":137,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.078150+0000","last_change":"2024-09-17T07:13:50.549466+0000","last_active":"2024-09-17T07:15:51.078150+0000","last_peered":"2024-09-17T07:15:51.078150+0000","last_clean":"2024-09-17T07:15:51.078150+0000","last_became_active":"2024-09-17T07:09:24.392850+0000","last_became_peered":"2024-09-17T07:09:24.392850+0000","last_unstale":"2024-09-17T07:15:51.078150+0000","last_undegraded":"2024-09-17T07:15:51.078150+0000","last_fullsized":"2024-09-17T07:15:51.078150+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:50.549369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:23.833170+0000","last_clean_scrub_stamp":"2024-09-17T07:13:50.549369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013250060000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":316,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.078641+0000","last_change":"2024-09-17T07:15:40.663837+0000","last_active":"2024-09-17T07:15:51.078641+0000","last_peered":"2024-09-17T07:15:51.078641+0000","last_clean":"2024-09-17T07:15:51.078641+0000","last_became_active":"2024-09-17T07:02:16.014492+0000","last_became_peered":"2024-09-17T07:02:16.014492+0000","last_unstale":"2024-09-17T07:15:51.078641+0000","last_undegraded":"2024-09-17T07:15:51.078641+0000","last_fullsized":"2024-09-17T07:15:51.078641+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:40.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:15:40.663759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:44.968568+0000","scrub_duration":0.0013265250000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":309,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.079090+0000","last_change":"2024-09-17T07:15:02.062776+0000","last_active":"2024-09-17T07:15:51.079090+0000","last_peered":"2024-09-17T07:15:51.079090+0000","last_clean":"2024-09-17T07:15:51.079090+0000","last_became_active":"2024-09-17T07:09:24.402704+0000","last_became_peered":"2024-09-17T07:09:24.402704+0000","last_unstale":"2024-09-17T07:15:51.079090+0000","last_undegraded":"2024-09-17T07:15:51.079090+0000","last_fullsized":"2024-09-17T07:15:51.079090+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:02.062618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:02.062618+0000","last_clean_scrub_stamp":"2024-09-17T07:15:02.062618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:23.850877+0000","scrub_duration":0.0042833460000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"268'30","reported_seq":359,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.079156+0000","last_change":"2024-09-17T07:14:00.599815+0000","last_active":"2024-09-17T07:15:51.079156+0000","last_peered":"2024-09-17T07:15:51.079156+0000","last_clean":"2024-09-17T07:15:51.079156+0000","last_became_active":"2024-09-17T07:03:48.760405+0000","last_became_peered":"2024-09-17T07:03:48.760405+0000","last_unstale":"2024-09-17T07:15:51.079156+0000","last_undegraded":"2024-09-17T07:15:51.079156+0000","last_fullsized":"2024-09-17T07:15:51.079156+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'30","last_scrub_stamp":"2024-09-17T07:14:00.599656+0000","last_deep_scrub":"268'30","last_deep_scrub_stamp":"2024-09-17T07:14:00.599656+0000","last_clean_scrub_stamp":"2024-09-17T07:14:00.599656+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.030045246000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"70'1","reported_seq":315,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.078492+0000","last_change":"2024-09-17T07:15:01.575863+0000","last_active":"2024-09-17T07:15:51.078492+0000","last_peered":"2024-09-17T07:15:51.078492+0000","last_clean":"2024-09-17T07:15:51.078492+0000","last_became_active":"2024-09-17T07:09:24.402499+0000","last_became_peered":"2024-09-17T07:09:24.402499+0000","last_unstale":"2024-09-17T07:15:51.078492+0000","last_undegraded":"2024-09-17T07:15:51.078492+0000","last_fullsized":"2024-09-17T07:15:51.078492+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-17T07:15:01.575747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:15:01.575747+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:07.093189+0000","scrub_duration":0.0037109349999999998,"objects_trimmed":0,"snaptrim_duration":0.0032298629999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":284,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.080603+0000","last_change":"2024-09-17T07:12:16.273630+0000","last_active":"2024-09-17T07:15:51.080603+0000","last_peered":"2024-09-17T07:15:51.080603+0000","last_clean":"2024-09-17T07:15:51.080603+0000","last_became_active":"2024-09-17T07:02:12.014623+0000","last_became_peered":"2024-09-17T07:02:12.014623+0000","last_unstale":"2024-09-17T07:15:51.080603+0000","last_undegraded":"2024-09-17T07:15:51.080603+0000","last_fullsized":"2024-09-17T07:15:51.080603+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:12:16.273537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:12:16.273537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001748185,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":302,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.078676+0000","last_change":"2024-09-17T07:15:45.447122+0000","last_active":"2024-09-17T07:15:51.078676+0000","last_peered":"2024-09-17T07:15:51.078676+0000","last_clean":"2024-09-17T07:15:51.078676+0000","last_became_active":"2024-09-17T07:02:16.021561+0000","last_became_peered":"2024-09-17T07:02:16.021561+0000","last_unstale":"2024-09-17T07:15:51.078676+0000","last_undegraded":"2024-09-17T07:15:51.078676+0000","last_fullsized":"2024-09-17T07:15:51.078676+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:45.447029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:15:45.447029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:59.805276+0000","scrub_duration":0.001990463,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":298,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.877380+0000","last_change":"2024-09-17T07:15:51.877450+0000","last_active":"2024-09-17T07:15:51.877380+0000","last_peered":"2024-09-17T07:15:51.877380+0000","last_clean":"2024-09-17T07:15:51.877380+0000","last_became_active":"2024-09-17T07:09:24.396154+0000","last_became_peered":"2024-09-17T07:09:24.396154+0000","last_unstale":"2024-09-17T07:15:51.877380+0000","last_undegraded":"2024-09-17T07:15:51.877380+0000","last_fullsized":"2024-09-17T07:15:51.877380+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:51.877367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:15:51.877367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:03.493027+0000","scrub_duration":0.001552423,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"242'26","reported_seq":346,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.079158+0000","last_change":"2024-09-17T07:14:43.500757+0000","last_active":"2024-09-17T07:15:51.079158+0000","last_peered":"2024-09-17T07:15:51.079158+0000","last_clean":"2024-09-17T07:15:51.079158+0000","last_became_active":"2024-09-17T07:02:14.028857+0000","last_became_peered":"2024-09-17T07:02:14.028857+0000","last_unstale":"2024-09-17T07:15:51.079158+0000","last_undegraded":"2024-09-17T07:15:51.079158+0000","last_fullsized":"2024-09-17T07:15:51.079158+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'26","last_scrub_stamp":"2024-09-17T07:14:43.500621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:14:43.500621+0000","objects_scrubbed":7,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:06.894697+0000","scrub_duration":0.032308936000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"268'3","reported_seq":166,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.079483+0000","last_change":"2024-09-17T07:15:43.516879+0000","last_active":"2024-09-17T07:15:51.079483+0000","last_peered":"2024-09-17T07:15:51.079483+0000","last_clean":"2024-09-17T07:15:51.079483+0000","last_became_active":"2024-09-17T07:09:24.404532+0000","last_became_peered":"2024-09-17T07:09:24.404532+0000","last_unstale":"2024-09-17T07:15:51.079483+0000","last_undegraded":"2024-09-17T07:15:51.079483+0000","last_fullsized":"2024-09-17T07:15:51.079483+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'3","last_scrub_stamp":"2024-09-17T07:15:43.516714+0000","last_deep_scrub":"268'3","last_deep_scrub_stamp":"2024-09-17T07:13:54.949345+0000","last_clean_scrub_stamp":"2024-09-17T07:15:43.516714+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:05.399081+0000","scrub_duration":0.002071711,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":306,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.080200+0000","last_change":"2024-09-17T07:13:50.914454+0000","last_active":"2024-09-17T07:15:51.080200+0000","last_peered":"2024-09-17T07:15:51.080200+0000","last_clean":"2024-09-17T07:15:51.080200+0000","last_became_active":"2024-09-17T07:02:12.004606+0000","last_became_peered":"2024-09-17T07:02:12.004606+0000","last_unstale":"2024-09-17T07:15:51.080200+0000","last_undegraded":"2024-09-17T07:15:51.080200+0000","last_fullsized":"2024-09-17T07:15:51.080200+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:50.914276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:13:50.914276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0071842440000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"212'130","reported_seq":556,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.080725+0000","last_change":"2024-09-17T07:14:57.052759+0000","last_active":"2024-09-17T07:15:51.080725+0000","last_peered":"2024-09-17T07:15:51.080725+0000","last_clean":"2024-09-17T07:15:51.080725+0000","last_became_active":"2024-09-17T07:02:03.020675+0000","last_became_peered":"2024-09-17T07:02:03.020675+0000","last_unstale":"2024-09-17T07:15:51.080725+0000","last_undegraded":"2024-09-17T07:15:51.080725+0000","last_fullsized":"2024-09-17T07:15:51.080725+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:14:57.052505+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:14:57.052505+0000","last_clean_scrub_stamp":"2024-09-17T07:14:57.052505+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:01.924960+0000","scrub_duration":0.022122466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":291,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.078711+0000","last_change":"2024-09-17T07:13:04.014087+0000","last_active":"2024-09-17T07:15:51.078711+0000","last_peered":"2024-09-17T07:15:51.078711+0000","last_clean":"2024-09-17T07:15:51.078711+0000","last_became_active":"2024-09-17T07:02:16.016031+0000","last_became_peered":"2024-09-17T07:02:16.016031+0000","last_unstale":"2024-09-17T07:15:51.078711+0000","last_undegraded":"2024-09-17T07:15:51.078711+0000","last_fullsized":"2024-09-17T07:15:51.078711+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:04.013978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:13:04.013978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004290149,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":298,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.078908+0000","last_change":"2024-09-17T07:14:48.021809+0000","last_active":"2024-09-17T07:15:51.078908+0000","last_peered":"2024-09-17T07:15:51.078908+0000","last_clean":"2024-09-17T07:15:51.078908+0000","last_became_active":"2024-09-17T07:09:24.400041+0000","last_became_peered":"2024-09-17T07:09:24.400041+0000","last_unstale":"2024-09-17T07:15:51.078908+0000","last_undegraded":"2024-09-17T07:15:51.078908+0000","last_fullsized":"2024-09-17T07:15:51.078908+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:48.021592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:58.670571+0000","last_clean_scrub_stamp":"2024-09-17T07:14:48.021592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:08.077034+0000","scrub_duration":0.0064844619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"242'35","reported_seq":372,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.079647+0000","last_change":"2024-09-17T07:15:43.665940+0000","last_active":"2024-09-17T07:15:51.079647+0000","last_peered":"2024-09-17T07:15:51.079647+0000","last_clean":"2024-09-17T07:15:51.079647+0000","last_became_active":"2024-09-17T07:02:14.027313+0000","last_became_peered":"2024-09-17T07:02:14.027313+0000","last_unstale":"2024-09-17T07:15:51.079647+0000","last_undegraded":"2024-09-17T07:15:51.079647+0000","last_fullsized":"2024-09-17T07:15:51.079647+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'35","last_scrub_stamp":"2024-09-17T07:15:43.665829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:15:43.665829+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:00.032507+0000","scrub_duration":0.0048325160000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":342,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.078768+0000","last_change":"2024-09-17T07:15:48.685572+0000","last_active":"2024-09-17T07:15:51.078768+0000","last_peered":"2024-09-17T07:15:51.078768+0000","last_clean":"2024-09-17T07:15:51.078768+0000","last_became_active":"2024-09-17T07:02:12.012037+0000","last_became_peered":"2024-09-17T07:02:12.012037+0000","last_unstale":"2024-09-17T07:15:51.078768+0000","last_undegraded":"2024-09-17T07:15:51.078768+0000","last_fullsized":"2024-09-17T07:15:51.078768+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:48.685498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:06:39.322535+0000","last_clean_scrub_stamp":"2024-09-17T07:15:48.685498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:17.814720+0000","scrub_duration":0.001437528,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":326,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.078395+0000","last_change":"2024-09-17T07:15:36.567808+0000","last_active":"2024-09-17T07:15:51.078395+0000","last_peered":"2024-09-17T07:15:51.078395+0000","last_clean":"2024-09-17T07:15:51.078395+0000","last_became_active":"2024-09-17T07:02:16.014631+0000","last_became_peered":"2024-09-17T07:02:16.014631+0000","last_unstale":"2024-09-17T07:15:51.078395+0000","last_undegraded":"2024-09-17T07:15:51.078395+0000","last_fullsized":"2024-09-17T07:15:51.078395+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:36.567726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:15:36.567726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:44.026430+0000","scrub_duration":0.0014186050000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":298,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.080686+0000","last_change":"2024-09-17T07:15:45.275553+0000","last_active":"2024-09-17T07:15:51.080686+0000","last_peered":"2024-09-17T07:15:51.080686+0000","last_clean":"2024-09-17T07:15:51.080686+0000","last_became_active":"2024-09-17T07:09:24.403904+0000","last_became_peered":"2024-09-17T07:09:24.403904+0000","last_unstale":"2024-09-17T07:15:51.080686+0000","last_undegraded":"2024-09-17T07:15:51.080686+0000","last_fullsized":"2024-09-17T07:15:51.080686+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:45.275473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:45.275473+0000","last_clean_scrub_stamp":"2024-09-17T07:15:45.275473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:11.590023+0000","scrub_duration":0.0014472759999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"268'19","reported_seq":323,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.077939+0000","last_change":"2024-09-17T07:14:06.634500+0000","last_active":"2024-09-17T07:15:51.077939+0000","last_peered":"2024-09-17T07:15:51.077939+0000","last_clean":"2024-09-17T07:15:51.077939+0000","last_became_active":"2024-09-17T07:02:14.014939+0000","last_became_peered":"2024-09-17T07:02:14.014939+0000","last_unstale":"2024-09-17T07:15:51.077939+0000","last_undegraded":"2024-09-17T07:15:51.077939+0000","last_fullsized":"2024-09-17T07:15:51.077939+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'19","last_scrub_stamp":"2024-09-17T07:14:06.634334+0000","last_deep_scrub":"268'19","last_deep_scrub_stamp":"2024-09-17T07:14:06.634334+0000","last_clean_scrub_stamp":"2024-09-17T07:14:06.634334+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0088943289999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":304,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:50.397478+0000","last_change":"2024-09-17T07:13:53.607329+0000","last_active":"2024-09-17T07:15:50.397478+0000","last_peered":"2024-09-17T07:15:50.397478+0000","last_clean":"2024-09-17T07:15:50.397478+0000","last_became_active":"2024-09-17T07:09:24.391207+0000","last_became_peered":"2024-09-17T07:09:24.391207+0000","last_unstale":"2024-09-17T07:15:50.397478+0000","last_undegraded":"2024-09-17T07:15:50.397478+0000","last_fullsized":"2024-09-17T07:15:50.397478+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:53.593078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:13:53.593078+0000","last_clean_scrub_stamp":"2024-09-17T07:13:53.593078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0025217759999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":267,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:50.394169+0000","last_change":"2024-09-17T07:14:45.625006+0000","last_active":"2024-09-17T07:15:50.394169+0000","last_peered":"2024-09-17T07:15:50.394169+0000","last_clean":"2024-09-17T07:15:50.394169+0000","last_became_active":"2024-09-17T07:09:24.396642+0000","last_became_peered":"2024-09-17T07:09:24.396642+0000","last_unstale":"2024-09-17T07:15:50.394169+0000","last_undegraded":"2024-09-17T07:15:50.394169+0000","last_fullsized":"2024-09-17T07:15:50.394169+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:45.624886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:14:45.624886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:15:58.276677+0000","scrub_duration":0.0061816429999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":290,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.078263+0000","last_change":"2024-09-17T07:15:00.557150+0000","last_active":"2024-09-17T07:15:51.078263+0000","last_peered":"2024-09-17T07:15:51.078263+0000","last_clean":"2024-09-17T07:15:51.078263+0000","last_became_active":"2024-09-17T07:02:18.026994+0000","last_became_peered":"2024-09-17T07:02:18.026994+0000","last_unstale":"2024-09-17T07:15:51.078263+0000","last_undegraded":"2024-09-17T07:15:51.078263+0000","last_fullsized":"2024-09-17T07:15:51.078263+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:00.557042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:58.099849+0000","last_clean_scrub_stamp":"2024-09-17T07:15:00.557042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:11.269725+0000","scrub_duration":0.0065216600000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"242'34","reported_seq":354,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.078062+0000","last_change":"2024-09-17T07:14:50.103842+0000","last_active":"2024-09-17T07:15:51.078062+0000","last_peered":"2024-09-17T07:15:51.078062+0000","last_clean":"2024-09-17T07:15:51.078062+0000","last_became_active":"2024-09-17T07:02:14.028392+0000","last_became_peered":"2024-09-17T07:02:14.028392+0000","last_unstale":"2024-09-17T07:15:51.078062+0000","last_undegraded":"2024-09-17T07:15:51.078062+0000","last_fullsized":"2024-09-17T07:15:51.078062+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'34","last_scrub_stamp":"2024-09-17T07:14:50.103539+0000","last_deep_scrub":"242'34","last_deep_scrub_stamp":"2024-09-17T07:14:50.103539+0000","last_clean_scrub_stamp":"2024-09-17T07:14:50.103539+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:08.014973+0000","scrub_duration":0.016600602999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'2","reported_seq":177,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.079386+0000","last_change":"2024-09-17T07:15:48.378000+0000","last_active":"2024-09-17T07:15:51.079386+0000","last_peered":"2024-09-17T07:15:51.079386+0000","last_clean":"2024-09-17T07:15:51.079386+0000","last_became_active":"2024-09-17T07:09:24.402909+0000","last_became_peered":"2024-09-17T07:09:24.402909+0000","last_unstale":"2024-09-17T07:15:51.079386+0000","last_undegraded":"2024-09-17T07:15:51.079386+0000","last_fullsized":"2024-09-17T07:15:51.079386+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-09-17T07:15:48.377886+0000","last_deep_scrub":"89'2","last_deep_scrub_stamp":"2024-09-17T07:12:20.301263+0000","last_clean_scrub_stamp":"2024-09-17T07:15:48.377886+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:50.626969+0000","scrub_duration":0.001635079,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":420,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":420,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"225'9","reported_seq":467,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.078719+0000","last_change":"2024-09-17T07:14:42.018236+0000","last_active":"2024-09-17T07:15:51.078719+0000","last_peered":"2024-09-17T07:15:51.078719+0000","last_clean":"2024-09-17T07:15:51.078719+0000","last_became_active":"2024-09-17T07:02:16.015418+0000","last_became_peered":"2024-09-17T07:02:16.015418+0000","last_unstale":"2024-09-17T07:15:51.078719+0000","last_undegraded":"2024-09-17T07:15:51.078719+0000","last_fullsized":"2024-09-17T07:15:51.078719+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"225'9","last_scrub_stamp":"2024-09-17T07:14:42.017656+0000","last_deep_scrub":"225'9","last_deep_scrub_stamp":"2024-09-17T07:11:25.292565+0000","last_clean_scrub_stamp":"2024-09-17T07:14:42.017656+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0089186300000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":300,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.079232+0000","last_change":"2024-09-17T07:14:49.061957+0000","last_active":"2024-09-17T07:15:51.079232+0000","last_peered":"2024-09-17T07:15:51.079232+0000","last_clean":"2024-09-17T07:15:51.079232+0000","last_became_active":"2024-09-17T07:02:18.363253+0000","last_became_peered":"2024-09-17T07:02:18.363253+0000","last_unstale":"2024-09-17T07:15:51.079232+0000","last_undegraded":"2024-09-17T07:15:51.079232+0000","last_fullsized":"2024-09-17T07:15:51.079232+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:49.061804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:14:49.061804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:15:53.632074+0000","scrub_duration":0.0048663639999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"242'23","reported_seq":374,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:50.394114+0000","last_change":"2024-09-17T07:13:59.558578+0000","last_active":"2024-09-17T07:15:50.394114+0000","last_peered":"2024-09-17T07:15:50.394114+0000","last_clean":"2024-09-17T07:15:50.394114+0000","last_became_active":"2024-09-17T07:10:49.723292+0000","last_became_peered":"2024-09-17T07:10:49.723292+0000","last_unstale":"2024-09-17T07:15:50.394114+0000","last_undegraded":"2024-09-17T07:15:50.394114+0000","last_fullsized":"2024-09-17T07:15:50.394114+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'23","last_scrub_stamp":"2024-09-17T07:13:59.558320+0000","last_deep_scrub":"150'17","last_deep_scrub_stamp":"2024-09-17T07:06:38.775788+0000","last_clean_scrub_stamp":"2024-09-17T07:13:59.558320+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0092780950000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"68'2","reported_seq":320,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:50.394931+0000","last_change":"2024-09-17T07:14:56.533910+0000","last_active":"2024-09-17T07:15:50.394931+0000","last_peered":"2024-09-17T07:15:50.394931+0000","last_clean":"2024-09-17T07:15:50.394931+0000","last_became_active":"2024-09-17T07:10:49.727577+0000","last_became_peered":"2024-09-17T07:10:49.727577+0000","last_unstale":"2024-09-17T07:15:50.394931+0000","last_undegraded":"2024-09-17T07:15:50.394931+0000","last_fullsized":"2024-09-17T07:15:50.394931+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-09-17T07:14:56.533798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:14:56.533798+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:10.946077+0000","scrub_duration":0.003207115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":267,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.586943+0000","last_change":"2024-09-17T07:15:51.587007+0000","last_active":"2024-09-17T07:15:51.586943+0000","last_peered":"2024-09-17T07:15:51.586943+0000","last_clean":"2024-09-17T07:15:51.586943+0000","last_became_active":"2024-09-17T07:04:15.111069+0000","last_became_peered":"2024-09-17T07:04:15.111069+0000","last_unstale":"2024-09-17T07:15:51.586943+0000","last_undegraded":"2024-09-17T07:15:51.586943+0000","last_fullsized":"2024-09-17T07:15:51.586943+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:51.586929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:11:17.479036+0000","last_clean_scrub_stamp":"2024-09-17T07:15:51.586929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:55.589024+0000","scrub_duration":0.001274435,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":135,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.079815+0000","last_change":"2024-09-17T07:13:13.938089+0000","last_active":"2024-09-17T07:15:51.079815+0000","last_peered":"2024-09-17T07:15:51.079815+0000","last_clean":"2024-09-17T07:15:51.079815+0000","last_became_active":"2024-09-17T07:09:24.397669+0000","last_became_peered":"2024-09-17T07:09:24.397669+0000","last_unstale":"2024-09-17T07:15:51.079815+0000","last_undegraded":"2024-09-17T07:15:51.079815+0000","last_fullsized":"2024-09-17T07:15:51.079815+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:13.937979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:13:13.937979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.018487173999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"242'15","reported_seq":337,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.078252+0000","last_change":"2024-09-17T07:13:51.542389+0000","last_active":"2024-09-17T07:15:51.078252+0000","last_peered":"2024-09-17T07:15:51.078252+0000","last_clean":"2024-09-17T07:15:51.078252+0000","last_became_active":"2024-09-17T07:09:24.408875+0000","last_became_peered":"2024-09-17T07:09:24.408875+0000","last_unstale":"2024-09-17T07:15:51.078252+0000","last_undegraded":"2024-09-17T07:15:51.078252+0000","last_fullsized":"2024-09-17T07:15:51.078252+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'15","last_scrub_stamp":"2024-09-17T07:13:51.542263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:13:51.542263+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0072768679999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":330,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.077818+0000","last_change":"2024-09-17T07:15:39.634792+0000","last_active":"2024-09-17T07:15:51.077818+0000","last_peered":"2024-09-17T07:15:51.077818+0000","last_clean":"2024-09-17T07:15:51.077818+0000","last_became_active":"2024-09-17T07:02:12.011327+0000","last_became_peered":"2024-09-17T07:02:12.011327+0000","last_unstale":"2024-09-17T07:15:51.077818+0000","last_undegraded":"2024-09-17T07:15:51.077818+0000","last_fullsized":"2024-09-17T07:15:51.077818+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:39.634698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:17.976267+0000","last_clean_scrub_stamp":"2024-09-17T07:15:39.634698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:44.039723+0000","scrub_duration":0.00117202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":134,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:50.396317+0000","last_change":"2024-09-17T07:15:40.845904+0000","last_active":"2024-09-17T07:15:50.396317+0000","last_peered":"2024-09-17T07:15:50.396317+0000","last_clean":"2024-09-17T07:15:50.396317+0000","last_became_active":"2024-09-17T07:09:24.407806+0000","last_became_peered":"2024-09-17T07:09:24.407806+0000","last_unstale":"2024-09-17T07:15:50.396317+0000","last_undegraded":"2024-09-17T07:15:50.396317+0000","last_fullsized":"2024-09-17T07:15:50.396317+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:40.845816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:18.367486+0000","last_clean_scrub_stamp":"2024-09-17T07:15:40.845816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:53.615891+0000","scrub_duration":0.0013977639999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":335,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.078475+0000","last_change":"2024-09-17T07:14:52.084362+0000","last_active":"2024-09-17T07:15:51.078475+0000","last_peered":"2024-09-17T07:15:51.078475+0000","last_clean":"2024-09-17T07:15:51.078475+0000","last_became_active":"2024-09-17T07:02:18.017825+0000","last_became_peered":"2024-09-17T07:02:18.017825+0000","last_unstale":"2024-09-17T07:15:51.078475+0000","last_undegraded":"2024-09-17T07:15:51.078475+0000","last_fullsized":"2024-09-17T07:15:51.078475+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:52.084273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:09:23.378805+0000","last_clean_scrub_stamp":"2024-09-17T07:14:52.084273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:14.268291+0000","scrub_duration":0.0039317479999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":318,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.079714+0000","last_change":"2024-09-17T07:15:46.645165+0000","last_active":"2024-09-17T07:15:51.079714+0000","last_peered":"2024-09-17T07:15:51.079714+0000","last_clean":"2024-09-17T07:15:51.079714+0000","last_became_active":"2024-09-17T07:02:12.008065+0000","last_became_peered":"2024-09-17T07:02:12.008065+0000","last_unstale":"2024-09-17T07:15:51.079714+0000","last_undegraded":"2024-09-17T07:15:51.079714+0000","last_fullsized":"2024-09-17T07:15:51.079714+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:46.645080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:46.645080+0000","last_clean_scrub_stamp":"2024-09-17T07:15:46.645080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:59.909153+0000","scrub_duration":0.0015661049999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"242'12","reported_seq":172,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.078387+0000","last_change":"2024-09-17T07:15:44.249955+0000","last_active":"2024-09-17T07:15:51.078387+0000","last_peered":"2024-09-17T07:15:51.078387+0000","last_clean":"2024-09-17T07:15:51.078387+0000","last_became_active":"2024-09-17T07:09:24.405385+0000","last_became_peered":"2024-09-17T07:09:24.405385+0000","last_unstale":"2024-09-17T07:15:51.078387+0000","last_undegraded":"2024-09-17T07:15:51.078387+0000","last_fullsized":"2024-09-17T07:15:51.078387+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'12","last_scrub_stamp":"2024-09-17T07:15:44.249858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:15:44.249858+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:47.251325+0000","scrub_duration":0.0022006600000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":310,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.079327+0000","last_change":"2024-09-17T07:14:55.191458+0000","last_active":"2024-09-17T07:15:51.079327+0000","last_peered":"2024-09-17T07:15:51.079327+0000","last_clean":"2024-09-17T07:15:51.079327+0000","last_became_active":"2024-09-17T07:09:24.398922+0000","last_became_peered":"2024-09-17T07:09:24.398922+0000","last_unstale":"2024-09-17T07:15:51.079327+0000","last_undegraded":"2024-09-17T07:15:51.079327+0000","last_fullsized":"2024-09-17T07:15:51.079327+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:55.191252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:14:55.191252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:02.113664+0000","scrub_duration":0.0090150360000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":288,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.079008+0000","last_change":"2024-09-17T07:14:52.058961+0000","last_active":"2024-09-17T07:15:51.079008+0000","last_peered":"2024-09-17T07:15:51.079008+0000","last_clean":"2024-09-17T07:15:51.079008+0000","last_became_active":"2024-09-17T07:02:18.365419+0000","last_became_peered":"2024-09-17T07:02:18.365419+0000","last_unstale":"2024-09-17T07:15:51.079008+0000","last_undegraded":"2024-09-17T07:15:51.079008+0000","last_fullsized":"2024-09-17T07:15:51.079008+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:52.058847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:14:52.058847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:19.721921+0000","scrub_duration":0.0054926549999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"242'20","reported_seq":189,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:50.396784+0000","last_change":"2024-09-17T07:14:05.528864+0000","last_active":"2024-09-17T07:15:50.396784+0000","last_peered":"2024-09-17T07:15:50.396784+0000","last_clean":"2024-09-17T07:15:50.396784+0000","last_became_active":"2024-09-17T07:09:24.405944+0000","last_became_peered":"2024-09-17T07:09:24.405944+0000","last_unstale":"2024-09-17T07:15:50.396784+0000","last_undegraded":"2024-09-17T07:15:50.396784+0000","last_fullsized":"2024-09-17T07:15:50.396784+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'20","last_scrub_stamp":"2024-09-17T07:14:05.528769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:14:05.528769+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.015483452,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":325,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.079650+0000","last_change":"2024-09-17T07:15:48.405105+0000","last_active":"2024-09-17T07:15:51.079650+0000","last_peered":"2024-09-17T07:15:51.079650+0000","last_clean":"2024-09-17T07:15:51.079650+0000","last_became_active":"2024-09-17T07:09:24.402355+0000","last_became_peered":"2024-09-17T07:09:24.402355+0000","last_unstale":"2024-09-17T07:15:51.079650+0000","last_undegraded":"2024-09-17T07:15:51.079650+0000","last_fullsized":"2024-09-17T07:15:51.079650+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:48.405027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:15:48.405027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:06.134372+0000","scrub_duration":0.0014214710000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"163'9","reported_seq":474,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.079974+0000","last_change":"2024-09-17T07:14:53.140257+0000","last_active":"2024-09-17T07:15:51.079974+0000","last_peered":"2024-09-17T07:15:51.079974+0000","last_clean":"2024-09-17T07:15:51.079974+0000","last_became_active":"2024-09-17T07:02:16.022583+0000","last_became_peered":"2024-09-17T07:02:16.022583+0000","last_unstale":"2024-09-17T07:15:51.079974+0000","last_undegraded":"2024-09-17T07:15:51.079974+0000","last_fullsized":"2024-09-17T07:15:51.079974+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'9","last_scrub_stamp":"2024-09-17T07:14:53.140133+0000","last_deep_scrub":"163'9","last_deep_scrub_stamp":"2024-09-17T07:07:34.933808+0000","last_clean_scrub_stamp":"2024-09-17T07:14:53.140133+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:15.885880+0000","scrub_duration":0.0088819989999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":298,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.077941+0000","last_change":"2024-09-17T07:14:04.624250+0000","last_active":"2024-09-17T07:15:51.077941+0000","last_peered":"2024-09-17T07:15:51.077941+0000","last_clean":"2024-09-17T07:15:51.077941+0000","last_became_active":"2024-09-17T07:09:24.409051+0000","last_became_peered":"2024-09-17T07:09:24.409051+0000","last_unstale":"2024-09-17T07:15:51.077941+0000","last_undegraded":"2024-09-17T07:15:51.077941+0000","last_fullsized":"2024-09-17T07:15:51.077941+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:04.624118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:26.619893+0000","last_clean_scrub_stamp":"2024-09-17T07:14:04.624118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0028490479999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"242'15","reported_seq":347,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.080062+0000","last_change":"2024-09-17T07:14:07.138526+0000","last_active":"2024-09-17T07:15:51.080062+0000","last_peered":"2024-09-17T07:15:51.080062+0000","last_clean":"2024-09-17T07:15:51.080062+0000","last_became_active":"2024-09-17T07:09:24.406333+0000","last_became_peered":"2024-09-17T07:09:24.406333+0000","last_unstale":"2024-09-17T07:15:51.080062+0000","last_undegraded":"2024-09-17T07:15:51.080062+0000","last_fullsized":"2024-09-17T07:15:51.080062+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'15","last_scrub_stamp":"2024-09-17T07:14:07.138401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:14:07.138401+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.038794863999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":304,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.078070+0000","last_change":"2024-09-17T07:14:41.618598+0000","last_active":"2024-09-17T07:15:51.078070+0000","last_peered":"2024-09-17T07:15:51.078070+0000","last_clean":"2024-09-17T07:15:51.078070+0000","last_became_active":"2024-09-17T07:09:24.402704+0000","last_became_peered":"2024-09-17T07:09:24.402704+0000","last_unstale":"2024-09-17T07:15:51.078070+0000","last_undegraded":"2024-09-17T07:15:51.078070+0000","last_fullsized":"2024-09-17T07:15:51.078070+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:41.618524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:14:41.618524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:05.162355+0000","scrub_duration":0.0025348459999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"196'9","reported_seq":237,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.078802+0000","last_change":"2024-09-17T07:13:10.965977+0000","last_active":"2024-09-17T07:15:51.078802+0000","last_peered":"2024-09-17T07:15:51.078802+0000","last_clean":"2024-09-17T07:15:51.078802+0000","last_became_active":"2024-09-17T07:09:24.406267+0000","last_became_peered":"2024-09-17T07:09:24.406267+0000","last_unstale":"2024-09-17T07:15:51.078802+0000","last_undegraded":"2024-09-17T07:15:51.078802+0000","last_fullsized":"2024-09-17T07:15:51.078802+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'9","last_scrub_stamp":"2024-09-17T07:13:10.965855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:13:10.965855+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0088433720000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":290,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:50.394650+0000","last_change":"2024-09-17T07:14:03.699173+0000","last_active":"2024-09-17T07:15:50.394650+0000","last_peered":"2024-09-17T07:15:50.394650+0000","last_clean":"2024-09-17T07:15:50.394650+0000","last_became_active":"2024-09-17T07:02:18.018716+0000","last_became_peered":"2024-09-17T07:02:18.018716+0000","last_unstale":"2024-09-17T07:15:50.394650+0000","last_undegraded":"2024-09-17T07:15:50.394650+0000","last_fullsized":"2024-09-17T07:15:50.394650+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:03.699048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:14:03.699048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0099202749999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"242'15","reported_seq":317,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:50.396758+0000","last_change":"2024-09-17T07:13:48.500828+0000","last_active":"2024-09-17T07:15:50.396758+0000","last_peered":"2024-09-17T07:15:50.396758+0000","last_clean":"2024-09-17T07:15:50.396758+0000","last_became_active":"2024-09-17T07:02:14.010289+0000","last_became_peered":"2024-09-17T07:02:14.010289+0000","last_unstale":"2024-09-17T07:15:50.396758+0000","last_undegraded":"2024-09-17T07:15:50.396758+0000","last_fullsized":"2024-09-17T07:15:50.396758+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'15","last_scrub_stamp":"2024-09-17T07:13:48.500257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:13:48.500257+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0054642730000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":326,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.078834+0000","last_change":"2024-09-17T07:15:33.622783+0000","last_active":"2024-09-17T07:15:51.078834+0000","last_peered":"2024-09-17T07:15:51.078834+0000","last_clean":"2024-09-17T07:15:51.078834+0000","last_became_active":"2024-09-17T07:09:24.408139+0000","last_became_peered":"2024-09-17T07:09:24.408139+0000","last_unstale":"2024-09-17T07:15:51.078834+0000","last_undegraded":"2024-09-17T07:15:51.078834+0000","last_fullsized":"2024-09-17T07:15:51.078834+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:33.622666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:10:36.334827+0000","last_clean_scrub_stamp":"2024-09-17T07:15:33.622666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:03.203848+0000","scrub_duration":0.002139064,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":146,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.077624+0000","last_change":"2024-09-17T07:15:37.523361+0000","last_active":"2024-09-17T07:15:51.077624+0000","last_peered":"2024-09-17T07:15:51.077624+0000","last_clean":"2024-09-17T07:15:51.077624+0000","last_became_active":"2024-09-17T07:09:24.399116+0000","last_became_peered":"2024-09-17T07:09:24.399116+0000","last_unstale":"2024-09-17T07:15:51.077624+0000","last_undegraded":"2024-09-17T07:15:51.077624+0000","last_fullsized":"2024-09-17T07:15:51.077624+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:37.523283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:15:37.523283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:50.803530+0000","scrub_duration":0.0013475850000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":300,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.077369+0000","last_change":"2024-09-17T07:15:40.540428+0000","last_active":"2024-09-17T07:15:51.077369+0000","last_peered":"2024-09-17T07:15:51.077369+0000","last_clean":"2024-09-17T07:15:51.077369+0000","last_became_active":"2024-09-17T07:02:18.027622+0000","last_became_peered":"2024-09-17T07:02:18.027622+0000","last_unstale":"2024-09-17T07:15:51.077369+0000","last_undegraded":"2024-09-17T07:15:51.077369+0000","last_fullsized":"2024-09-17T07:15:51.077369+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:40.540336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:15:40.540336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:46.617341+0000","scrub_duration":0.001212982,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"242'19","reported_seq":325,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:50.394479+0000","last_change":"2024-09-17T07:14:05.729683+0000","last_active":"2024-09-17T07:15:50.394479+0000","last_peered":"2024-09-17T07:15:50.394479+0000","last_clean":"2024-09-17T07:15:50.394479+0000","last_became_active":"2024-09-17T07:02:14.016255+0000","last_became_peered":"2024-09-17T07:02:14.016255+0000","last_unstale":"2024-09-17T07:15:50.394479+0000","last_undegraded":"2024-09-17T07:15:50.394479+0000","last_fullsized":"2024-09-17T07:15:50.394479+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'19","last_scrub_stamp":"2024-09-17T07:14:05.729535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:14:05.729535+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.008660559,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"94'2","reported_seq":300,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:50.397482+0000","last_change":"2024-09-17T07:15:48.907903+0000","last_active":"2024-09-17T07:15:50.397482+0000","last_peered":"2024-09-17T07:15:50.397482+0000","last_clean":"2024-09-17T07:15:50.397482+0000","last_became_active":"2024-09-17T07:02:12.011461+0000","last_became_peered":"2024-09-17T07:02:12.011461+0000","last_unstale":"2024-09-17T07:15:50.397482+0000","last_undegraded":"2024-09-17T07:15:50.397482+0000","last_fullsized":"2024-09-17T07:15:50.397482+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-09-17T07:15:48.907821+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-09-17T07:07:31.717888+0000","last_clean_scrub_stamp":"2024-09-17T07:15:48.907821+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:00.875203+0000","scrub_duration":0.001503307,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":299,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:50.396628+0000","last_change":"2024-09-17T07:15:41.858178+0000","last_active":"2024-09-17T07:15:50.396628+0000","last_peered":"2024-09-17T07:15:50.396628+0000","last_clean":"2024-09-17T07:15:50.396628+0000","last_became_active":"2024-09-17T07:09:24.407362+0000","last_became_peered":"2024-09-17T07:09:24.407362+0000","last_unstale":"2024-09-17T07:15:50.396628+0000","last_undegraded":"2024-09-17T07:15:50.396628+0000","last_fullsized":"2024-09-17T07:15:50.396628+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:41.858093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:15:41.858093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:47.679821+0000","scrub_duration":0.0020998420000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":145,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.077666+0000","last_change":"2024-09-17T07:15:49.573768+0000","last_active":"2024-09-17T07:15:51.077666+0000","last_peered":"2024-09-17T07:15:51.077666+0000","last_clean":"2024-09-17T07:15:51.077666+0000","last_became_active":"2024-09-17T07:09:24.400791+0000","last_became_peered":"2024-09-17T07:09:24.400791+0000","last_unstale":"2024-09-17T07:15:51.077666+0000","last_undegraded":"2024-09-17T07:15:51.077666+0000","last_fullsized":"2024-09-17T07:15:51.077666+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:49.573666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:49.573666+0000","last_clean_scrub_stamp":"2024-09-17T07:15:49.573666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:18.941100+0000","scrub_duration":0.0017548970000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"242'8","reported_seq":208,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.077436+0000","last_change":"2024-09-17T07:14:43.060402+0000","last_active":"2024-09-17T07:15:51.077436+0000","last_peered":"2024-09-17T07:15:51.077436+0000","last_clean":"2024-09-17T07:15:51.077436+0000","last_became_active":"2024-09-17T07:09:24.396368+0000","last_became_peered":"2024-09-17T07:09:24.396368+0000","last_unstale":"2024-09-17T07:15:51.077436+0000","last_undegraded":"2024-09-17T07:15:51.077436+0000","last_fullsized":"2024-09-17T07:15:51.077436+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'8","last_scrub_stamp":"2024-09-17T07:14:43.059953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:14:43.059953+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:06.391559+0000","scrub_duration":0.0051998909999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":305,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.079636+0000","last_change":"2024-09-17T07:15:01.141244+0000","last_active":"2024-09-17T07:15:51.079636+0000","last_peered":"2024-09-17T07:15:51.079636+0000","last_clean":"2024-09-17T07:15:51.079636+0000","last_became_active":"2024-09-17T07:02:12.008898+0000","last_became_peered":"2024-09-17T07:02:12.008898+0000","last_unstale":"2024-09-17T07:15:51.079636+0000","last_undegraded":"2024-09-17T07:15:51.079636+0000","last_fullsized":"2024-09-17T07:15:51.079636+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:01.140327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:15:01.140327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:04.563392+0000","scrub_duration":0.0060829439999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":303,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.079350+0000","last_change":"2024-09-17T07:14:45.506205+0000","last_active":"2024-09-17T07:15:51.079350+0000","last_peered":"2024-09-17T07:15:51.079350+0000","last_clean":"2024-09-17T07:15:51.079350+0000","last_became_active":"2024-09-17T07:02:16.014982+0000","last_became_peered":"2024-09-17T07:02:16.014982+0000","last_unstale":"2024-09-17T07:15:51.079350+0000","last_undegraded":"2024-09-17T07:15:51.079350+0000","last_fullsized":"2024-09-17T07:15:51.079350+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:45.505869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:47.444972+0000","last_clean_scrub_stamp":"2024-09-17T07:14:45.505869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:15:52.708354+0000","scrub_duration":0.033706858999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":288,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.080111+0000","last_change":"2024-09-17T07:13:59.042437+0000","last_active":"2024-09-17T07:15:51.080111+0000","last_peered":"2024-09-17T07:15:51.080111+0000","last_clean":"2024-09-17T07:15:51.080111+0000","last_became_active":"2024-09-17T07:02:18.021014+0000","last_became_peered":"2024-09-17T07:02:18.021014+0000","last_unstale":"2024-09-17T07:15:51.080111+0000","last_undegraded":"2024-09-17T07:15:51.080111+0000","last_fullsized":"2024-09-17T07:15:51.080111+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:59.042322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:13:59.042322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0076271120000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"242'19","reported_seq":189,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.078880+0000","last_change":"2024-09-17T07:13:45.076965+0000","last_active":"2024-09-17T07:15:51.078880+0000","last_peered":"2024-09-17T07:15:51.078880+0000","last_clean":"2024-09-17T07:15:51.078880+0000","last_became_active":"2024-09-17T07:09:24.404942+0000","last_became_peered":"2024-09-17T07:09:24.404942+0000","last_unstale":"2024-09-17T07:15:51.078880+0000","last_undegraded":"2024-09-17T07:15:51.078880+0000","last_fullsized":"2024-09-17T07:15:51.078880+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'19","last_scrub_stamp":"2024-09-17T07:13:45.076322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:13:45.076322+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0088383200000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":317,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.079584+0000","last_change":"2024-09-17T07:15:00.436459+0000","last_active":"2024-09-17T07:15:51.079584+0000","last_peered":"2024-09-17T07:15:51.079584+0000","last_clean":"2024-09-17T07:15:51.079584+0000","last_became_active":"2024-09-17T07:02:12.010195+0000","last_became_peered":"2024-09-17T07:02:12.010195+0000","last_unstale":"2024-09-17T07:15:51.079584+0000","last_undegraded":"2024-09-17T07:15:51.079584+0000","last_fullsized":"2024-09-17T07:15:51.079584+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:00.436330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:15:00.436330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:03.413807+0000","scrub_duration":0.0037388289999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":290,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.080318+0000","last_change":"2024-09-17T07:13:47.027302+0000","last_active":"2024-09-17T07:15:51.080318+0000","last_peered":"2024-09-17T07:15:51.080318+0000","last_clean":"2024-09-17T07:15:51.080318+0000","last_became_active":"2024-09-17T07:02:16.016183+0000","last_became_peered":"2024-09-17T07:02:16.016183+0000","last_unstale":"2024-09-17T07:15:51.080318+0000","last_undegraded":"2024-09-17T07:15:51.080318+0000","last_fullsized":"2024-09-17T07:15:51.080318+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:47.027189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:13:47.027189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0024966390000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":145,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.077910+0000","last_change":"2024-09-17T07:14:51.603973+0000","last_active":"2024-09-17T07:15:51.077910+0000","last_peered":"2024-09-17T07:15:51.077910+0000","last_clean":"2024-09-17T07:15:51.077910+0000","last_became_active":"2024-09-17T07:09:24.392216+0000","last_became_peered":"2024-09-17T07:09:24.392216+0000","last_unstale":"2024-09-17T07:15:51.077910+0000","last_undegraded":"2024-09-17T07:15:51.077910+0000","last_fullsized":"2024-09-17T07:15:51.077910+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:51.603818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:14:51.603818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:13.344567+0000","scrub_duration":0.0041420099999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1c","version":"75'1","reported_seq":289,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.078616+0000","last_change":"2024-09-17T07:13:56.976301+0000","last_active":"2024-09-17T07:15:51.078616+0000","last_peered":"2024-09-17T07:15:51.078616+0000","last_clean":"2024-09-17T07:15:51.078616+0000","last_became_active":"2024-09-17T07:02:18.021855+0000","last_became_peered":"2024-09-17T07:02:18.021855+0000","last_unstale":"2024-09-17T07:15:51.078616+0000","last_undegraded":"2024-09-17T07:15:51.078616+0000","last_fullsized":"2024-09-17T07:15:51.078616+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-17T07:13:56.976159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:13:56.976159+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0053255050000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":306,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.078145+0000","last_change":"2024-09-17T07:15:00.144722+0000","last_active":"2024-09-17T07:15:51.078145+0000","last_peered":"2024-09-17T07:15:51.078145+0000","last_clean":"2024-09-17T07:15:51.078145+0000","last_became_active":"2024-09-17T07:02:12.005329+0000","last_became_peered":"2024-09-17T07:02:12.005329+0000","last_unstale":"2024-09-17T07:15:51.078145+0000","last_undegraded":"2024-09-17T07:15:51.078145+0000","last_fullsized":"2024-09-17T07:15:51.078145+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:00.144598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:15:00.144598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:06.979069+0000","scrub_duration":0.0076347749999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"242'20","reported_seq":366,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.078167+0000","last_change":"2024-09-17T07:13:52.546409+0000","last_active":"2024-09-17T07:15:51.078167+0000","last_peered":"2024-09-17T07:15:51.078167+0000","last_clean":"2024-09-17T07:15:51.078167+0000","last_became_active":"2024-09-17T07:10:49.730948+0000","last_became_peered":"2024-09-17T07:10:49.730948+0000","last_unstale":"2024-09-17T07:15:51.078167+0000","last_undegraded":"2024-09-17T07:15:51.078167+0000","last_fullsized":"2024-09-17T07:15:51.078167+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'20","last_scrub_stamp":"2024-09-17T07:13:52.546261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:13:52.546261+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0080850769999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"75'8","reported_seq":479,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.079433+0000","last_change":"2024-09-17T07:14:46.490248+0000","last_active":"2024-09-17T07:15:51.079433+0000","last_peered":"2024-09-17T07:15:51.079433+0000","last_clean":"2024-09-17T07:15:51.079433+0000","last_became_active":"2024-09-17T07:02:16.020560+0000","last_became_peered":"2024-09-17T07:02:16.020560+0000","last_unstale":"2024-09-17T07:15:51.079433+0000","last_undegraded":"2024-09-17T07:15:51.079433+0000","last_fullsized":"2024-09-17T07:15:51.079433+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:14:46.490111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:14:46.490111+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:15:56.963491+0000","scrub_duration":0.011893111,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":288,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:50.397191+0000","last_change":"2024-09-17T07:15:37.885645+0000","last_active":"2024-09-17T07:15:50.397191+0000","last_peered":"2024-09-17T07:15:50.397191+0000","last_clean":"2024-09-17T07:15:50.397191+0000","last_became_active":"2024-09-17T07:02:18.021603+0000","last_became_peered":"2024-09-17T07:02:18.021603+0000","last_unstale":"2024-09-17T07:15:50.397191+0000","last_undegraded":"2024-09-17T07:15:50.397191+0000","last_fullsized":"2024-09-17T07:15:50.397191+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:37.885570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:15:37.885570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:07.813041+0000","scrub_duration":0.001703174,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":157,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.079643+0000","last_change":"2024-09-17T07:15:36.653927+0000","last_active":"2024-09-17T07:15:51.079643+0000","last_peered":"2024-09-17T07:15:51.079643+0000","last_clean":"2024-09-17T07:15:51.079643+0000","last_became_active":"2024-09-17T07:09:24.403360+0000","last_became_peered":"2024-09-17T07:09:24.403360+0000","last_unstale":"2024-09-17T07:15:51.079643+0000","last_undegraded":"2024-09-17T07:15:51.079643+0000","last_fullsized":"2024-09-17T07:15:51.079643+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:36.653817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:10:29.325664+0000","last_clean_scrub_stamp":"2024-09-17T07:15:36.653817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:55.913678+0000","scrub_duration":0.0014304299999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"242'19","reported_seq":389,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.079511+0000","last_change":"2024-09-17T07:15:41.670675+0000","last_active":"2024-09-17T07:15:51.079511+0000","last_peered":"2024-09-17T07:15:51.079511+0000","last_clean":"2024-09-17T07:15:51.079511+0000","last_became_active":"2024-09-17T07:03:48.760390+0000","last_became_peered":"2024-09-17T07:03:48.760390+0000","last_unstale":"2024-09-17T07:15:51.079511+0000","last_undegraded":"2024-09-17T07:15:51.079511+0000","last_fullsized":"2024-09-17T07:15:51.079511+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'19","last_scrub_stamp":"2024-09-17T07:15:41.670531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:15:41.670531+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:00.891108+0000","scrub_duration":0.0022660979999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":292,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.078070+0000","last_change":"2024-09-17T07:14:49.639583+0000","last_active":"2024-09-17T07:15:51.078070+0000","last_peered":"2024-09-17T07:15:51.078070+0000","last_clean":"2024-09-17T07:15:51.078070+0000","last_became_active":"2024-09-17T07:02:16.019701+0000","last_became_peered":"2024-09-17T07:02:16.019701+0000","last_unstale":"2024-09-17T07:15:51.078070+0000","last_undegraded":"2024-09-17T07:15:51.078070+0000","last_fullsized":"2024-09-17T07:15:51.078070+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:49.639480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:14:49.639480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:12.299880+0000","scrub_duration":0.003512032,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":291,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:50.394686+0000","last_change":"2024-09-17T07:14:04.726753+0000","last_active":"2024-09-17T07:15:50.394686+0000","last_peered":"2024-09-17T07:15:50.394686+0000","last_clean":"2024-09-17T07:15:50.394686+0000","last_became_active":"2024-09-17T07:02:18.363752+0000","last_became_peered":"2024-09-17T07:02:18.363752+0000","last_unstale":"2024-09-17T07:15:50.394686+0000","last_undegraded":"2024-09-17T07:15:50.394686+0000","last_fullsized":"2024-09-17T07:15:50.394686+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:04.726216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:14:04.726216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0080957649999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":347,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.079787+0000","last_change":"2024-09-17T07:15:35.670909+0000","last_active":"2024-09-17T07:15:51.079787+0000","last_peered":"2024-09-17T07:15:51.079787+0000","last_clean":"2024-09-17T07:15:51.079787+0000","last_became_active":"2024-09-17T07:03:48.759384+0000","last_became_peered":"2024-09-17T07:03:48.759384+0000","last_unstale":"2024-09-17T07:15:51.079787+0000","last_undegraded":"2024-09-17T07:15:51.079787+0000","last_fullsized":"2024-09-17T07:15:51.079787+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:35.670737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:11:18.559877+0000","last_clean_scrub_stamp":"2024-09-17T07:15:35.670737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:43.644043+0000","scrub_duration":0.0013632939999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"242'27","reported_seq":347,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:50.396864+0000","last_change":"2024-09-17T07:15:45.855529+0000","last_active":"2024-09-17T07:15:50.396864+0000","last_peered":"2024-09-17T07:15:50.396864+0000","last_clean":"2024-09-17T07:15:50.396864+0000","last_became_active":"2024-09-17T07:09:24.403030+0000","last_became_peered":"2024-09-17T07:09:24.403030+0000","last_unstale":"2024-09-17T07:15:50.396864+0000","last_undegraded":"2024-09-17T07:15:50.396864+0000","last_fullsized":"2024-09-17T07:15:50.396864+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'27","last_scrub_stamp":"2024-09-17T07:15:45.855430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:15:45.855430+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:00.331869+0000","scrub_duration":0.0021742689999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":290,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:50.397324+0000","last_change":"2024-09-17T07:14:00.509698+0000","last_active":"2024-09-17T07:15:50.397324+0000","last_peered":"2024-09-17T07:15:50.397324+0000","last_clean":"2024-09-17T07:15:50.397324+0000","last_became_active":"2024-09-17T07:02:16.015903+0000","last_became_peered":"2024-09-17T07:02:16.015903+0000","last_unstale":"2024-09-17T07:15:50.397324+0000","last_undegraded":"2024-09-17T07:15:50.397324+0000","last_fullsized":"2024-09-17T07:15:50.397324+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:00.509283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:25.794418+0000","last_clean_scrub_stamp":"2024-09-17T07:14:00.509283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0073526269999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":159,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.078206+0000","last_change":"2024-09-17T07:15:02.473582+0000","last_active":"2024-09-17T07:15:51.078206+0000","last_peered":"2024-09-17T07:15:51.078206+0000","last_clean":"2024-09-17T07:15:51.078206+0000","last_became_active":"2024-09-17T07:09:24.402411+0000","last_became_peered":"2024-09-17T07:09:24.402411+0000","last_unstale":"2024-09-17T07:15:51.078206+0000","last_undegraded":"2024-09-17T07:15:51.078206+0000","last_fullsized":"2024-09-17T07:15:51.078206+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:02.473467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:15:02.473467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:14.391965+0000","scrub_duration":0.001733163,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":136,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:51.080154+0000","last_change":"2024-09-17T07:13:15.914281+0000","last_active":"2024-09-17T07:15:51.080154+0000","last_peered":"2024-09-17T07:15:51.080154+0000","last_clean":"2024-09-17T07:15:51.080154+0000","last_became_active":"2024-09-17T07:09:24.404795+0000","last_became_peered":"2024-09-17T07:09:24.404795+0000","last_unstale":"2024-09-17T07:15:51.080154+0000","last_undegraded":"2024-09-17T07:15:51.080154+0000","last_fullsized":"2024-09-17T07:15:51.080154+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:15.914144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:13:15.914144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.059987944000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"242'23","reported_seq":209,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:50.397381+0000","last_change":"2024-09-17T07:14:58.689224+0000","last_active":"2024-09-17T07:15:50.397381+0000","last_peered":"2024-09-17T07:15:50.397381+0000","last_clean":"2024-09-17T07:15:50.397381+0000","last_became_active":"2024-09-17T07:09:24.394476+0000","last_became_peered":"2024-09-17T07:09:24.394476+0000","last_unstale":"2024-09-17T07:15:50.397381+0000","last_undegraded":"2024-09-17T07:15:50.397381+0000","last_fullsized":"2024-09-17T07:15:50.397381+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'23","last_scrub_stamp":"2024-09-17T07:14:58.689120+0000","last_deep_scrub":"86'17","last_deep_scrub_stamp":"2024-09-17T07:03:18.241067+0000","last_clean_scrub_stamp":"2024-09-17T07:14:58.689120+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:08.797039+0000","scrub_duration":0.0066193850000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":312,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-17T07:15:50.394801+0000","last_change":"2024-09-17T07:15:38.338684+0000","last_active":"2024-09-17T07:15:50.394801+0000","last_peered":"2024-09-17T07:15:50.394801+0000","last_clean":"2024-09-17T07:15:50.394801+0000","last_became_active":"2024-09-17T07:09:24.393329+0000","last_became_peered":"2024-09-17T07:09:24.393329+0000","last_unstale":"2024-09-17T07:15:50.394801+0000","last_undegraded":"2024-09-17T07:15:50.394801+0000","last_fullsized":"2024-09-17T07:15:50.394801+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:38.338552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:15:38.338552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:58.182182+0000","scrub_duration":0.0017509940000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70,"ondisk_log_size":70,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1121,"num_read_kb":878,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":109,"num_bytes_recovered":2228,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":13004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":723,"ondisk_log_size":723,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":813,"num_read_kb":808,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1386,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":12,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":4578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037930,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":609984,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":608638,"kb_avail":93135168,"statfs":{"total":95995035648,"available":95370412032,"internally_reserved":0,"allocated":1359872,"data_stored":986458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":623245771},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:15:24 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":1.244,"15min":1.585},"min":{"1min":0.29699999999999999,"5min":0.20699999999999999,"15min":0.192},"max":{"1min":2.867,"5min":8.6080000000000005,"15min":52.478999999999999},"last":0.67600000000000005},{"interface":"front","average":{"1min":1.2809999999999999,"5min":1.4370000000000001,"15min":1.6910000000000001},"min":{"1min":0.246,"5min":0.23899999999999999,"15min":0.22600000000000001},"max":{"1min":4.766,"5min":19.216000000000001,"15min":52.871000000000002},"last":0.56100000000000005}]},{"osd":1,"last update":"Tue Sep 17 07:15:24 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":1.4510000000000001,"15min":1.3859999999999999},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.223},"max":{"1min":1.573,"5min":15.801,"15min":73.549000000000007},"last":0.69599999999999995},{"interface":"front","average":{"1min":1.3779999999999999,"5min":1.2969999999999999,"15min":1.383},"min":{"1min":0.29699999999999999,"5min":0.26200000000000001,"15min":0.24099999999999999},"max":{"1min":4.7359999999999998,"5min":13.989000000000001,"15min":73.116},"last":0.59499999999999997}]},{"osd":2,"last update":"Tue Sep 17 07:15:24 2024","interfaces":[{"interface":"back","average":{"1min":1.3620000000000001,"5min":1.1759999999999999,"15min":1.121},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":5.0209999999999999,"5min":6.4900000000000002,"15min":52.055999999999997},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.77700000000000002,"5min":1.165,"15min":1.1279999999999999},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":4.9320000000000004,"5min":8.1639999999999997,"15min":52.064999999999998},"last":0.57499999999999996}]},{"osd":3,"last update":"Tue Sep 17 07:15:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2869999999999999,"5min":1.1799999999999999,"15min":1.3140000000000001},"min":{"1min":0.35899999999999999,"5min":0.29199999999999998,"15min":0.28799999999999998},"max":{"1min":5.0439999999999996,"5min":6.4640000000000004,"15min":51.689},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.92400000000000004,"5min":1.224,"15min":1.429},"min":{"1min":0.32900000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":5.1520000000000001,"5min":8.1839999999999993,"15min":52.145000000000003},"last":0.60799999999999998}]},{"osd":4,"last update":"Tue Sep 17 07:15:24 2024","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.1459999999999999,"15min":0.72099999999999997},"min":{"1min":0.31900000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":4.6950000000000003,"5min":8.5730000000000004,"15min":8.5730000000000004},"last":0.46200000000000002},{"interface":"front","average":{"1min":1.1890000000000001,"5min":1.2989999999999999,"15min":0.77700000000000002},"min":{"1min":0.30399999999999999,"5min":0.26000000000000001,"15min":0.24399999999999999},"max":{"1min":4.9569999999999999,"5min":6.5129999999999999,"15min":6.5129999999999999},"last":0.42899999999999999}]},{"osd":5,"last update":"Tue Sep 17 07:15:24 2024","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.806,"15min":0.94299999999999995},"min":{"1min":0.27900000000000003,"5min":0.253,"15min":0.23699999999999999},"max":{"1min":2.895,"5min":29.428999999999998,"15min":29.428999999999998},"last":0.39200000000000002},{"interface":"front","average":{"1min":1.8069999999999999,"5min":1.8839999999999999,"15min":1.0029999999999999},"min":{"1min":0.375,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":4.7930000000000001,"5min":30.059999999999999,"15min":30.059999999999999},"last":0.41199999999999998}]},{"osd":6,"last update":"Tue Sep 17 07:15:24 2024","interfaces":[{"interface":"back","average":{"1min":1.7030000000000001,"5min":1.5640000000000001,"15min":0.92000000000000004},"min":{"1min":0.34699999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":6.3339999999999996,"5min":25.363,"15min":25.363},"last":0.442},{"interface":"front","average":{"1min":1.423,"5min":1.607,"15min":0.92100000000000004},"min":{"1min":0.41799999999999998,"5min":0.28399999999999997,"15min":0.27500000000000002},"max":{"1min":4.9889999999999999,"5min":23.571000000000002,"15min":23.571000000000002},"last":0.47099999999999997}]}]},{"osd":6,"up_from":50,"seq":214748364973,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":371292,"kb_used_data":1356,"kb_used_omap":1,"kb_used_meta":369918,"kb_avail":93373860,"statfs":{"total":95995035648,"available":95614832640,"internally_reserved":0,"allocated":1388544,"data_stored":987075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":378796487},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:15:08 2024","interfaces":[{"interface":"back","average":{"1min":5.8029999999999999,"5min":1.946,"15min":1.988},"min":{"1min":0.20200000000000001,"5min":0.185,"15min":0.185},"max":{"1min":126.902,"5min":126.902,"15min":126.902},"last":0.46200000000000002},{"interface":"front","average":{"1min":6.3780000000000001,"5min":2.0259999999999998,"15min":1.9490000000000001},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20799999999999999},"max":{"1min":127.06399999999999,"5min":127.06399999999999,"15min":127.06399999999999},"last":0.52500000000000002}]},{"osd":1,"last update":"Tue Sep 17 07:15:08 2024","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":1.1379999999999999,"15min":0.83999999999999997},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22600000000000001},"max":{"1min":2.395,"5min":14.720000000000001,"15min":14.720000000000001},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.67900000000000005,"5min":1.1240000000000001,"15min":0.88300000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.22900000000000001},"max":{"1min":3.0990000000000002,"5min":14.699,"15min":14.699},"last":0.41699999999999998}]},{"osd":2,"last update":"Tue Sep 17 07:15:08 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.81499999999999995,"15min":0.84599999999999997},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":3.1360000000000001,"5min":4.9039999999999999,"15min":33.079000000000001},"last":0.51000000000000001},{"interface":"front","average":{"1min":1.1040000000000001,"5min":0.873,"15min":0.85199999999999998},"min":{"1min":0.26300000000000001,"5min":0.25600000000000001,"15min":0.23000000000000001},"max":{"1min":4.8399999999999999,"5min":5.734,"15min":33.100000000000001},"last":0.44400000000000001}]},{"osd":3,"last update":"Tue Sep 17 07:15:08 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.81599999999999995,"15min":1.149},"min":{"1min":0.307,"5min":0.24399999999999999,"15min":0.23200000000000001},"max":{"1min":2.48,"5min":5.0830000000000002,"15min":52.944000000000003},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.94999999999999996,"15min":1.212},"min":{"1min":0.28599999999999998,"5min":0.28499999999999998,"15min":0.27700000000000002},"max":{"1min":5.7690000000000001,"5min":5.7690000000000001,"15min":52.747},"last":0.56100000000000005}]},{"osd":4,"last update":"Tue Sep 17 07:15:08 2024","interfaces":[{"interface":"back","average":{"1min":1.744,"5min":4.7640000000000002,"15min":1.931},"min":{"1min":0.28899999999999998,"5min":0.214,"15min":0.214},"max":{"1min":7.1050000000000004,"5min":260.77199999999999,"15min":260.77199999999999},"last":0.32200000000000001},{"interface":"front","average":{"1min":1.536,"5min":4.7229999999999999,"15min":1.929},"min":{"1min":0.28899999999999998,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":4.5449999999999999,"5min":260.69099999999997,"15min":260.69099999999997},"last":0.502}]},{"osd":5,"last update":"Tue Sep 17 07:15:08 2024","interfaces":[{"interface":"back","average":{"1min":2.073,"5min":1.8620000000000001,"15min":0.97099999999999997},"min":{"1min":0.33100000000000002,"5min":0.26200000000000001,"15min":0.25900000000000001},"max":{"1min":8.0500000000000007,"5min":32.042000000000002,"15min":32.042000000000002},"last":0.38300000000000001},{"interface":"front","average":{"1min":1.573,"5min":1.7849999999999999,"15min":0.97499999999999998},"min":{"1min":0.27200000000000002,"5min":0.251,"15min":0.251},"max":{"1min":5.032,"5min":31.852,"15min":31.852},"last":0.432}]},{"osd":7,"last update":"Tue Sep 17 07:15:34 2024","interfaces":[{"interface":"back","average":{"1min":2.1230000000000002,"5min":1.48,"15min":0.91900000000000004},"min":{"1min":0.47999999999999998,"5min":0.28100000000000003,"15min":0.23699999999999999},"max":{"1min":6.2110000000000003,"5min":7.657,"15min":7.657},"last":0.54800000000000004},{"interface":"front","average":{"1min":2.1269999999999998,"5min":1.4470000000000001,"15min":0.90900000000000003},"min":{"1min":0.39800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":6.0519999999999996,"5min":7.1890000000000001,"15min":7.1890000000000001},"last":0.48999999999999999}]}]},{"osd":5,"up_from":42,"seq":180388626610,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":581504,"kb_used_data":752,"kb_used_omap":1,"kb_used_meta":580734,"kb_avail":93163648,"statfs":{"total":95995035648,"available":95399575552,"internally_reserved":0,"allocated":770048,"data_stored":396422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":594672070},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:15:47 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.94399999999999995,"15min":1.5629999999999999},"min":{"1min":0.218,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":1.823,"5min":5.202,"15min":49.265000000000001},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.84999999999999998,"5min":1.206,"15min":1.669},"min":{"1min":0.309,"5min":0.221,"15min":0.219},"max":{"1min":4.649,"5min":6.7149999999999999,"15min":47.350000000000001},"last":0.51000000000000001}]},{"osd":1,"last update":"Tue Sep 17 07:15:47 2024","interfaces":[{"interface":"back","average":{"1min":1.4319999999999999,"5min":1.0509999999999999,"15min":0.90100000000000002},"min":{"1min":0.30499999999999999,"5min":0.22,"15min":0.22},"max":{"1min":13.711,"5min":13.711,"15min":20.329000000000001},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.99399999999999999,"5min":1.3320000000000001,"15min":1.149},"min":{"1min":0.32000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4.617,"5min":6.7560000000000002,"15min":20.408999999999999},"last":0.67200000000000004}]},{"osd":2,"last update":"Tue Sep 17 07:15:47 2024","interfaces":[{"interface":"back","average":{"1min":1.254,"5min":1.125,"15min":1.0880000000000001},"min":{"1min":0.29299999999999998,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":6.3419999999999996,"5min":6.3419999999999996,"15min":20.486000000000001},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.92200000000000004,"15min":0.97499999999999998},"min":{"1min":0.317,"5min":0.25800000000000001,"15min":0.23200000000000001},"max":{"1min":1.8440000000000001,"5min":5.7789999999999999,"15min":23.925999999999998},"last":0.64500000000000002}]},{"osd":3,"last update":"Tue Sep 17 07:15:47 2024","interfaces":[{"interface":"back","average":{"1min":1.9710000000000001,"5min":1.353,"15min":1.4359999999999999},"min":{"1min":0.32100000000000001,"5min":0.27100000000000002,"15min":0.247},"max":{"1min":13.882,"5min":13.882,"15min":68.978999999999999},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.93500000000000005,"15min":1.2849999999999999},"min":{"1min":0.38400000000000001,"5min":0.29999999999999999,"15min":0.253},"max":{"1min":1.863,"5min":5.6139999999999999,"15min":69.084000000000003},"last":0.51900000000000002}]},{"osd":4,"last update":"Tue Sep 17 07:15:47 2024","interfaces":[{"interface":"back","average":{"1min":1.3979999999999999,"5min":1.821,"15min":1.4930000000000001},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.22},"max":{"1min":4.569,"5min":14.036,"15min":142.327},"last":0.30399999999999999},{"interface":"front","average":{"1min":1.4219999999999999,"5min":1.5229999999999999,"15min":1.4039999999999999},"min":{"1min":0.25,"5min":0.23999999999999999,"15min":0.20999999999999999},"max":{"1min":6.415,"5min":13.478999999999999,"15min":142.13200000000001},"last":0.35099999999999998}]},{"osd":6,"last update":"Tue Sep 17 07:15:01 2024","interfaces":[{"interface":"back","average":{"1min":0.98599999999999999,"5min":1.28,"15min":0.76800000000000002},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.23200000000000001},"max":{"1min":3.0049999999999999,"5min":8.3149999999999995,"15min":8.3149999999999995},"last":0.32700000000000001},{"interface":"front","average":{"1min":1.4319999999999999,"5min":1.294,"15min":0.80400000000000005},"min":{"1min":0.32400000000000001,"5min":0.27200000000000002,"15min":0.24099999999999999},"max":{"1min":6.3860000000000001,"5min":6.4459999999999997,"15min":6.4459999999999997},"last":0.38400000000000001}]},{"osd":7,"last update":"Tue Sep 17 07:15:24 2024","interfaces":[{"interface":"back","average":{"1min":1.976,"5min":1.742,"15min":1.012},"min":{"1min":0.35299999999999998,"5min":0.29599999999999999,"15min":0.29099999999999998},"max":{"1min":6.8869999999999996,"5min":6.8869999999999996,"15min":6.8869999999999996},"last":0.40400000000000003},{"interface":"front","average":{"1min":1.2050000000000001,"5min":1.397,"15min":0.84799999999999998},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.27100000000000002},"max":{"1min":3.4329999999999998,"5min":5.641,"15min":5.641},"last":0.36699999999999999}]}]},{"osd":4,"up_from":33,"seq":141733920952,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":455172,"kb_used_data":756,"kb_used_omap":1,"kb_used_meta":454398,"kb_avail":93289980,"statfs":{"total":95995035648,"available":95528939520,"internally_reserved":0,"allocated":774144,"data_stored":395522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":465304006},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:15:06 2024","interfaces":[{"interface":"back","average":{"1min":1.4890000000000001,"5min":1.355,"15min":2.0209999999999999},"min":{"1min":0.215,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":5.4500000000000002,"5min":18.434999999999999,"15min":74.551000000000002},"last":0.58699999999999997},{"interface":"front","average":{"1min":1.0329999999999999,"5min":1.333,"15min":2.125},"min":{"1min":0.22700000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":3.1819999999999999,"5min":18.521000000000001,"15min":74.778999999999996},"last":0.53300000000000003}]},{"osd":1,"last update":"Tue Sep 17 07:15:06 2024","interfaces":[{"interface":"back","average":{"1min":1.3109999999999999,"5min":2.125,"15min":1.484},"min":{"1min":0.29199999999999998,"5min":0.24399999999999999,"15min":0.23000000000000001},"max":{"1min":5.3010000000000002,"5min":44.984000000000002,"15min":54.914999999999999},"last":0.54700000000000004},{"interface":"front","average":{"1min":1.8540000000000001,"5min":2.1739999999999999,"15min":1.4359999999999999},"min":{"1min":0.39300000000000002,"5min":0.23300000000000001,"15min":0.22800000000000001},"max":{"1min":5.4109999999999996,"5min":45.353999999999999,"15min":53.725000000000001},"last":0.40699999999999997}]},{"osd":2,"last update":"Tue Sep 17 07:15:06 2024","interfaces":[{"interface":"back","average":{"1min":1.6499999999999999,"5min":1.167,"15min":1.075},"min":{"1min":0.308,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":5.4980000000000002,"5min":6.8929999999999998,"15min":49.651000000000003},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.63700000000000001,"5min":1.0920000000000001,"15min":1.0469999999999999},"min":{"1min":0.27500000000000002,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":2.1869999999999998,"5min":16.003,"15min":48.813000000000002},"last":0.70999999999999996}]},{"osd":3,"last update":"Tue Sep 17 07:15:06 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.94199999999999995,"15min":1.016},"min":{"1min":0.30599999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":1.272,"5min":4.2599999999999998,"15min":39.042999999999999},"last":0.56100000000000005},{"interface":"front","average":{"1min":1.2989999999999999,"5min":1.171,"15min":1.105},"min":{"1min":0.32200000000000001,"5min":0.27100000000000002,"15min":0.253},"max":{"1min":5.2699999999999996,"5min":15.970000000000001,"15min":38.920000000000002},"last":0.47699999999999998}]},{"osd":5,"last update":"Tue Sep 17 07:15:34 2024","interfaces":[{"interface":"back","average":{"1min":1.012,"5min":1.252,"15min":0.85399999999999998},"min":{"1min":0.28399999999999997,"5min":0.247,"15min":0.20799999999999999},"max":{"1min":2.7370000000000001,"5min":7.1909999999999998,"15min":11.486000000000001},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.84199999999999997,"5min":1.244,"15min":0.86299999999999999},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.26100000000000001},"max":{"1min":2.0169999999999999,"5min":5.9870000000000001,"15min":11.781000000000001},"last":0.63300000000000001}]},{"osd":6,"last update":"Tue Sep 17 07:15:06 2024","interfaces":[{"interface":"back","average":{"1min":1.4419999999999999,"5min":1.125,"15min":0.90200000000000002},"min":{"1min":0.30099999999999999,"5min":0.218,"15min":0.218},"max":{"1min":5.734,"5min":5.734,"15min":47.923999999999999},"last":0.34999999999999998},{"interface":"front","average":{"1min":1.9570000000000001,"5min":1.4690000000000001,"15min":1.0069999999999999},"min":{"1min":0.36199999999999999,"5min":0.26700000000000002,"15min":0.26300000000000001},"max":{"1min":6.7670000000000003,"5min":10.051,"15min":47.600999999999999},"last":0.52000000000000002}]},{"osd":7,"last update":"Tue Sep 17 07:15:25 2024","interfaces":[{"interface":"back","average":{"1min":1.544,"5min":1.3109999999999999,"15min":1.022},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":6.5140000000000002,"5min":7.1550000000000002,"15min":48.719000000000001},"last":0.5},{"interface":"front","average":{"1min":3.0249999999999999,"5min":1.637,"15min":1.115},"min":{"1min":0.44900000000000001,"5min":0.313,"15min":0.28499999999999998},"max":{"1min":6.3410000000000002,"5min":6.3410000000000002,"15min":50.527000000000001},"last":0.57499999999999996}]}]},{"osd":3,"up_from":0,"seq":115964117178,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":18,"seq":77309411520,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":631552,"kb_used_data":756,"kb_used_omap":1,"kb_used_meta":630782,"kb_avail":93113600,"statfs":{"total":95995035648,"available":95348326400,"internally_reserved":0,"allocated":774144,"data_stored":396366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":645921222},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:15:43 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":1.466,"15min":1.3779999999999999},"min":{"1min":0.17499999999999999,"5min":0.159,"15min":0.159},"max":{"1min":1.498,"5min":83.793999999999997,"15min":83.793999999999997},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.38100000000000001,"5min":1.3899999999999999,"15min":1.512},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":0.67800000000000005,"5min":83.588999999999999,"15min":83.588999999999999},"last":0.27300000000000002}]},{"osd":1,"last update":"Tue Sep 17 07:15:43 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":1.7949999999999999,"15min":1.623},"min":{"1min":0.23699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":1.6459999999999999,"5min":47.101999999999997,"15min":79.659000000000006},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.55000000000000004,"5min":1.6990000000000001,"15min":1.5600000000000001},"min":{"1min":0.22700000000000001,"5min":0.20899999999999999,"15min":0.20200000000000001},"max":{"1min":1.8049999999999999,"5min":45.831000000000003,"15min":80.155000000000001},"last":0.30499999999999999}]},{"osd":3,"last update":"Tue Sep 17 07:14:54 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":1.0349999999999999,"15min":0.97999999999999998},"min":{"1min":0.22,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":4.6449999999999996,"5min":23.43,"15min":23.43},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.503,"5min":0.92700000000000005,"15min":1.042},"min":{"1min":0.245,"5min":0.223,"15min":0.223},"max":{"1min":1.6739999999999999,"5min":23.408000000000001,"15min":23.408000000000001},"last":0.32100000000000001}]},{"osd":4,"last update":"Tue Sep 17 07:15:21 2024","interfaces":[{"interface":"back","average":{"1min":1.218,"5min":0.97799999999999998,"15min":0.89500000000000002},"min":{"1min":0.26100000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":6.8449999999999998,"5min":6.8449999999999998,"15min":6.8449999999999998},"last":0.46800000000000003},{"interface":"front","average":{"1min":1.4330000000000001,"5min":1.097,"15min":0.93200000000000005},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":6.8609999999999998,"5min":6.8609999999999998,"15min":6.8609999999999998},"last":0.45800000000000002}]},{"osd":5,"last update":"Tue Sep 17 07:15:43 2024","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":1.3779999999999999,"15min":1.119},"min":{"1min":0.33600000000000002,"5min":0.26900000000000002,"15min":0.249},"max":{"1min":4.0960000000000001,"5min":15.276999999999999,"15min":15.276999999999999},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.96699999999999997,"5min":1.1930000000000001,"15min":0.99099999999999999},"min":{"1min":0.35899999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":2.048,"5min":15.561999999999999,"15min":15.561999999999999},"last":0.40500000000000003}]},{"osd":6,"last update":"Tue Sep 17 07:14:54 2024","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":1.071,"15min":1.321},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.28999999999999998},"max":{"1min":6.0389999999999997,"5min":6.899,"15min":101.587},"last":0.379},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.97499999999999998,"15min":1.323},"min":{"1min":0.32300000000000001,"5min":0.317,"15min":0.28299999999999997},"max":{"1min":1.7490000000000001,"5min":12.339,"15min":100.696},"last":0.44800000000000001}]},{"osd":7,"last update":"Tue Sep 17 07:15:39 2024","interfaces":[{"interface":"back","average":{"1min":1.212,"5min":1.25,"15min":1},"min":{"1min":0.36699999999999999,"5min":0.316,"15min":0.316},"max":{"1min":2.629,"5min":7.8099999999999996,"15min":7.8099999999999996},"last":0.48999999999999999},{"interface":"front","average":{"1min":1.3220000000000001,"5min":1.272,"15min":1.0780000000000001},"min":{"1min":0.38,"5min":0.34999999999999998,"15min":0.32900000000000001},"max":{"1min":3.2429999999999999,"5min":6.9219999999999997,"15min":9.2919999999999998},"last":0.42899999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575045,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":541348,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":540542,"kb_avail":93203804,"statfs":{"total":95995035648,"available":95440695296,"internally_reserved":0,"allocated":811008,"data_stored":397520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":553515462},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:15:00 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":1.139,"15min":1.1259999999999999},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":2.8039999999999998,"5min":22.521000000000001,"15min":22.521000000000001},"last":0.214},{"interface":"front","average":{"1min":0.80800000000000005,"5min":1.0409999999999999,"15min":1.0269999999999999},"min":{"1min":0.17399999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":2.992,"5min":22.471,"15min":22.471},"last":0.31}]},{"osd":2,"last update":"Tue Sep 17 07:15:44 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.91000000000000003,"15min":1.024},"min":{"1min":0.21299999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":2.6360000000000001,"5min":6.1920000000000002,"15min":11.055999999999999},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.73899999999999999,"15min":0.94799999999999995},"min":{"1min":0.22900000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":1.3160000000000001,"5min":4.1559999999999997,"15min":9.3160000000000007},"last":0.23799999999999999}]},{"osd":3,"last update":"Tue Sep 17 07:15:00 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.91800000000000004,"15min":0.94299999999999995},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.219},"max":{"1min":3.0419999999999998,"5min":8.5510000000000002,"15min":8.5510000000000002},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.66600000000000004,"15min":0.99099999999999999},"min":{"1min":0.23999999999999999,"5min":0.22500000000000001,"15min":0.19500000000000001},"max":{"1min":1.496,"5min":7.2009999999999996,"15min":19.602},"last":0.32100000000000001}]},{"osd":4,"last update":"Tue Sep 17 07:15:20 2024","interfaces":[{"interface":"back","average":{"1min":1.1339999999999999,"5min":1.4670000000000001,"15min":1.129},"min":{"1min":0.36199999999999999,"5min":0.314,"15min":0.25},"max":{"1min":2.5659999999999998,"5min":19.088999999999999,"15min":19.088999999999999},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.98999999999999999,"5min":1.2010000000000001,"15min":1.1759999999999999},"min":{"1min":0.36799999999999999,"5min":0.317,"15min":0.307},"max":{"1min":2.6459999999999999,"5min":18.959,"15min":27.097999999999999},"last":0.67600000000000005}]},{"osd":5,"last update":"Tue Sep 17 07:15:44 2024","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.47,"15min":1.2110000000000001},"min":{"1min":0.34799999999999998,"5min":0.28899999999999998,"15min":0.27100000000000002},"max":{"1min":3.6869999999999998,"5min":6.6790000000000003,"15min":16.84},"last":0.47399999999999998},{"interface":"front","average":{"1min":1.1559999999999999,"5min":1.069,"15min":1.169},"min":{"1min":0.29299999999999998,"5min":0.29199999999999998,"15min":0.28899999999999998},"max":{"1min":6.2560000000000002,"5min":6.2560000000000002,"15min":16.949999999999999},"last":0.55700000000000005}]},{"osd":6,"last update":"Tue Sep 17 07:15:00 2024","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":1.667,"15min":1.272},"min":{"1min":0.315,"5min":0.315,"15min":0.308},"max":{"1min":2.3559999999999999,"5min":50.497,"15min":50.497},"last":0.48299999999999998},{"interface":"front","average":{"1min":1.141,"5min":1.6899999999999999,"15min":1.196},"min":{"1min":0.376,"5min":0.376,"15min":0.28899999999999998},"max":{"1min":3.5760000000000001,"5min":50.793999999999997,"15min":50.793999999999997},"last":0.61399999999999999}]},{"osd":7,"last update":"Tue Sep 17 07:15:20 2024","interfaces":[{"interface":"back","average":{"1min":1.655,"5min":1.343,"15min":1.1919999999999999},"min":{"1min":0.435,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":3.6120000000000001,"5min":9.8179999999999996,"15min":9.8179999999999996},"last":0.65700000000000003},{"interface":"front","average":{"1min":1.464,"5min":1.6319999999999999,"15min":1.2410000000000001},"min":{"1min":0.35099999999999998,"5min":0.317,"15min":0.317},"max":{"1min":4.1219999999999999,"5min":7.5839999999999996,"15min":11.007999999999999},"last":0.50800000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738568,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":618196,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":616830,"kb_avail":93126956,"statfs":{"total":95995035648,"available":95362002944,"internally_reserved":0,"allocated":1380352,"data_stored":986934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":631634374},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 17 07:15:18 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.89700000000000002,"15min":0.89500000000000002},"min":{"1min":0.254,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":1.518,"5min":21.876000000000001,"15min":21.876000000000001},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.56399999999999995,"5min":1.2330000000000001,"15min":1.081},"min":{"1min":0.17899999999999999,"5min":0.158,"15min":0.158},"max":{"1min":2.2599999999999998,"5min":22.521999999999998,"15min":22.521999999999998},"last":0.39700000000000002}]},{"osd":2,"last update":"Tue Sep 17 07:15:41 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.63100000000000001,"15min":1.1060000000000001},"min":{"1min":0.23599999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":1.327,"5min":6.3799999999999999,"15min":18.643999999999998},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.65900000000000003,"15min":1.0109999999999999},"min":{"1min":0.22,"5min":0.189,"15min":0.189},"max":{"1min":1.9850000000000001,"5min":4.431,"15min":17.972999999999999},"last":0.379}]},{"osd":3,"last update":"Tue Sep 17 07:15:18 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.68700000000000006,"15min":1.0109999999999999},"min":{"1min":0.23599999999999999,"5min":0.23400000000000001,"15min":0.21199999999999999},"max":{"1min":0.629,"5min":5.5880000000000001,"15min":18.629000000000001},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.58899999999999997,"15min":1.018},"min":{"1min":0.255,"5min":0.24199999999999999,"15min":0.23699999999999999},"max":{"1min":1.1339999999999999,"5min":3.9100000000000001,"15min":22.358000000000001},"last":0.35899999999999999}]},{"osd":4,"last update":"Tue Sep 17 07:15:18 2024","interfaces":[{"interface":"back","average":{"1min":1.056,"5min":1.127,"15min":1.1180000000000001},"min":{"1min":0.32600000000000001,"5min":0.29799999999999999,"15min":0.29099999999999998},"max":{"1min":4.0590000000000002,"5min":19.146999999999998,"15min":19.146999999999998},"last":0.495},{"interface":"front","average":{"1min":0.90900000000000003,"5min":1.151,"15min":1.0780000000000001},"min":{"1min":0.28199999999999997,"5min":0.26500000000000001,"15min":0.255},"max":{"1min":4.0449999999999999,"5min":19.158999999999999,"15min":19.158999999999999},"last":0.53300000000000003}]},{"osd":5,"last update":"Tue Sep 17 07:15:53 2024","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":1.1220000000000001,"15min":1.1699999999999999},"min":{"1min":0.35899999999999999,"5min":0.35499999999999998,"15min":0.28899999999999998},"max":{"1min":4.0350000000000001,"5min":5.0730000000000004,"15min":18.841000000000001},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.67900000000000005,"5min":1.032,"15min":1.1479999999999999},"min":{"1min":0.30099999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":1.9910000000000001,"5min":4.3680000000000003,"15min":19.248000000000001},"last":0.5}]},{"osd":6,"last update":"Tue Sep 17 07:15:18 2024","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.97899999999999998,"15min":1.016},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":4.6260000000000003,"5min":4.6260000000000003,"15min":18.602},"last":0.52500000000000002},{"interface":"front","average":{"1min":1.0289999999999999,"5min":0.98499999999999999,"15min":1.0569999999999999},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":4.4889999999999999,"5min":4.4889999999999999,"15min":18.692},"last":0.48099999999999998}]},{"osd":7,"last update":"Tue Sep 17 07:15:19 2024","interfaces":[{"interface":"back","average":{"1min":1.5740000000000001,"5min":1.4550000000000001,"15min":1.478},"min":{"1min":0.38600000000000001,"5min":0.30599999999999999,"15min":0.29599999999999999},"max":{"1min":4.4020000000000001,"5min":5.9509999999999996,"15min":46.183},"last":0.55700000000000005},{"interface":"front","average":{"1min":1.45,"5min":1.3899999999999999,"15min":1.3140000000000001},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.32000000000000001},"max":{"1min":4.0650000000000004,"5min":6.1079999999999997,"15min":45.914999999999999},"last":0.46200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T07:15:59.111 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T07:15:59.195 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:15:59.216 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:59.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:59.217 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:15:59.293 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:15:58 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:15:59.318 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:15:59.339 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:59.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:59.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:15:59.441 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:15:59.464 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:59.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:59.466 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:15:59.472 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:15:59.514 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:15:59.514 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:15:59.567 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:15:59.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:59.589 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:59.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:15:59.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:59 smithi100 ceph-mon[29092]: 5.1f scrub starts 2024-09-17T07:15:59.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:59 smithi100 ceph-mon[29092]: 5.1f scrub ok 2024-09-17T07:15:59.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:59 smithi100 ceph-mon[29092]: 4.9 scrub starts 2024-09-17T07:15:59.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:59 smithi100 ceph-mon[29092]: 4.9 scrub ok 2024-09-17T07:15:59.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:59 smithi100 ceph-mon[29092]: 4.16 scrub starts 2024-09-17T07:15:59.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:59 smithi100 ceph-mon[29092]: 4.16 scrub ok 2024-09-17T07:15:59.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:59 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:15:59.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:15:59 smithi100 ceph-mon[29092]: pgmap v615: 132 pgs: 132 active+clean; 582 KiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-17T07:15:59.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:59 smithi100 ceph-mon[39462]: 5.1f scrub starts 2024-09-17T07:15:59.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:59 smithi100 ceph-mon[39462]: 5.1f scrub ok 2024-09-17T07:15:59.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:59 smithi100 ceph-mon[39462]: 4.9 scrub starts 2024-09-17T07:15:59.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:59 smithi100 ceph-mon[39462]: 4.9 scrub ok 2024-09-17T07:15:59.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:59 smithi100 ceph-mon[39462]: 4.16 scrub starts 2024-09-17T07:15:59.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:59 smithi100 ceph-mon[39462]: 4.16 scrub ok 2024-09-17T07:15:59.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:59 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:15:59.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:15:59 smithi100 ceph-mon[39462]: pgmap v615: 132 pgs: 132 active+clean; 582 KiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-17T07:15:59.690 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:15:59.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:59.713 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:59.713 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:15:59.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:59 smithi135 ceph-mon[33238]: 5.1f scrub starts 2024-09-17T07:15:59.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:59 smithi135 ceph-mon[33238]: 5.1f scrub ok 2024-09-17T07:15:59.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:59 smithi135 ceph-mon[33238]: 4.9 scrub starts 2024-09-17T07:15:59.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:59 smithi135 ceph-mon[33238]: 4.9 scrub ok 2024-09-17T07:15:59.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:59 smithi135 ceph-mon[33238]: 4.16 scrub starts 2024-09-17T07:15:59.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:59 smithi135 ceph-mon[33238]: 4.16 scrub ok 2024-09-17T07:15:59.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:59 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:15:59.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:15:59 smithi135 ceph-mon[33238]: pgmap v615: 132 pgs: 132 active+clean; 582 KiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-17T07:15:59.814 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:15:59.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:59.836 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:15:59.836 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:15:59.937 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:15:59.963 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:15:59.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:15:59.964 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:16:00.065 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:16:00.087 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:00.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:00.088 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:16:00.189 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:16:00.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:00.705 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:00.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:16:00.705 INFO:teuthology.orchestra.run.smithi100.stderr:noscrub is set 2024-09-17T07:16:00.728 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:00 smithi100 ceph-mon[29092]: 2.7 scrub starts 2024-09-17T07:16:00.729 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:00 smithi100 ceph-mon[29092]: 2.7 scrub ok 2024-09-17T07:16:00.729 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:00 smithi100 ceph-mon[29092]: 2.d scrub starts 2024-09-17T07:16:00.729 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:00 smithi100 ceph-mon[29092]: 2.d scrub ok 2024-09-17T07:16:00.729 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:00 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1374014666' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:16:00.729 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:00 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:16:00.729 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:00 smithi100 ceph-mon[39462]: 2.7 scrub starts 2024-09-17T07:16:00.729 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:00 smithi100 ceph-mon[39462]: 2.7 scrub ok 2024-09-17T07:16:00.729 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:00 smithi100 ceph-mon[39462]: 2.d scrub starts 2024-09-17T07:16:00.729 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:00 smithi100 ceph-mon[39462]: 2.d scrub ok 2024-09-17T07:16:00.730 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:00 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1374014666' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:16:00.730 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:00 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:16:00.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:00 smithi135 ceph-mon[33238]: 2.7 scrub starts 2024-09-17T07:16:00.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:00 smithi135 ceph-mon[33238]: 2.7 scrub ok 2024-09-17T07:16:00.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:00 smithi135 ceph-mon[33238]: 2.d scrub starts 2024-09-17T07:16:00.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:00 smithi135 ceph-mon[33238]: 2.d scrub ok 2024-09-17T07:16:00.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:00 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1374014666' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:16:00.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:00 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:16:00.806 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:16:00.832 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:00.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:00.833 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:16:00.933 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:16:00.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:00.956 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:00.956 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:16:01.057 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:16:01.082 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:01.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:01.083 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:16:01.184 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:16:01.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:01.206 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:01.206 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:16:01.308 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:16:01.334 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:01.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:01.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:16:01.436 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:16:01.461 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:01.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:01.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:16:01.563 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:16:01.584 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:01.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:01.585 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:16:01.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:01 smithi100 ceph-mon[29092]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:16:01.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:01 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T07:16:01.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:01 smithi100 ceph-mon[29092]: osdmap e295: 8 total, 8 up, 7 in 2024-09-17T07:16:01.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:01 smithi100 ceph-mon[29092]: 4.1e deep-scrub starts 2024-09-17T07:16:01.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:01 smithi100 ceph-mon[29092]: 4.1e deep-scrub ok 2024-09-17T07:16:01.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:01 smithi100 ceph-mon[29092]: pgmap v617: 132 pgs: 132 active+clean; 582 KiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-17T07:16:01.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:01 smithi100 ceph-mon[39462]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:16:01.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:01 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T07:16:01.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:01 smithi100 ceph-mon[39462]: osdmap e295: 8 total, 8 up, 7 in 2024-09-17T07:16:01.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:01 smithi100 ceph-mon[39462]: 4.1e deep-scrub starts 2024-09-17T07:16:01.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:01 smithi100 ceph-mon[39462]: 4.1e deep-scrub ok 2024-09-17T07:16:01.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:01 smithi100 ceph-mon[39462]: pgmap v617: 132 pgs: 132 active+clean; 582 KiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-17T07:16:01.686 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:16:01.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:01.710 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:01.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:16:01.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:01 smithi135 ceph-mon[33238]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:16:01.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:01 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T07:16:01.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:01 smithi135 ceph-mon[33238]: osdmap e295: 8 total, 8 up, 7 in 2024-09-17T07:16:01.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:01 smithi135 ceph-mon[33238]: 4.1e deep-scrub starts 2024-09-17T07:16:01.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:01 smithi135 ceph-mon[33238]: 4.1e deep-scrub ok 2024-09-17T07:16:01.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:01 smithi135 ceph-mon[33238]: pgmap v617: 132 pgs: 132 active+clean; 582 KiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-17T07:16:01.811 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:16:01.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:01.835 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:01.835 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:16:01.936 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:16:01.958 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:01.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:01.959 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:16:02.060 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:16:02.086 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:02.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:02.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:16:02.188 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:16:02.211 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:02.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:02.212 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:16:02.313 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:16:02.336 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:02.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:02.337 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:16:02.438 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:16:02.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:02.461 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:02.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:16:02.562 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:16:02.584 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:02.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:02.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:16:02.686 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:16:02.707 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:02.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:02.708 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:16:02.750 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:02 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:16:02.750 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:02 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:16:02.751 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:02 smithi100 ceph-mon[29092]: osdmap e296: 8 total, 8 up, 7 in 2024-09-17T07:16:02.751 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:02 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:16:02.751 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:02 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:16:02.751 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:02 smithi100 ceph-mon[39462]: osdmap e296: 8 total, 8 up, 7 in 2024-09-17T07:16:02.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:02 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:16:02.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:02 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:16:02.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:02 smithi135 ceph-mon[33238]: osdmap e296: 8 total, 8 up, 7 in 2024-09-17T07:16:02.810 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:16:02.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:02.831 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:02.832 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:16:02.936 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:16:02.958 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:02.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:02.959 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:16:03.060 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:16:03.082 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:03.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:03.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:16:03.184 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:16:03.205 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:03.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:03.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:16:03.266 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-17T07:16:03.307 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:16:03.329 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:03.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:03.330 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:16:03.431 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:16:03.454 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:03.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:03.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:16:03.556 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:16:03.578 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:03.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:03.579 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:16:03.680 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:16:03.701 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:03.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:03.702 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:16:03.803 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:16:03.830 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:03.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:03.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:16:03.931 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:16:03.953 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:03.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:03.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:16:04.038 INFO:teuthology.orchestra.run.smithi100.stderr:pool 'unique_pool_3' created 2024-09-17T07:16:04.055 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:16:04.078 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:04.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:04.079 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:16:04.180 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:16:04.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:04.205 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:04.205 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:16:04.280 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:04 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:16:04.280 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:04 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:16:04.280 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:04 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/40632733' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-09-17T07:16:04.281 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:04 smithi100 ceph-mon[39462]: pgmap v619: 132 pgs: 132 active+clean; 582 KiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-09-17T07:16:04.281 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:04 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:16:04.281 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:04 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:16:04.281 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:04 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/40632733' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-09-17T07:16:04.281 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:04 smithi100 ceph-mon[29092]: pgmap v619: 132 pgs: 132 active+clean; 582 KiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-09-17T07:16:04.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:04 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:16:04.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:04 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:16:04.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:04 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/40632733' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-09-17T07:16:04.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:04 smithi135 ceph-mon[33238]: pgmap v619: 132 pgs: 132 active+clean; 582 KiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-09-17T07:16:04.307 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:16:04.330 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:04.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:04.331 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:16:04.431 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:16:04.454 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:04.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:04.456 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:16:04.556 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:16:04.578 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:04.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:04.579 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:16:04.680 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:16:04.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:04.702 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:04.702 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:16:04.787 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-17T07:16:04.787 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:16:04.803 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:16:04.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:04.827 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:04.828 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:16:04.928 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:16:04.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:04.951 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:04.951 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:16:05.052 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:16:05.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:05.077 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:05.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:16:05.179 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:16:05.200 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:05.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:05.201 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:16:05.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:05 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/40632733' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-09-17T07:16:05.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:05 smithi135 ceph-mon[33238]: osdmap e297: 8 total, 8 up, 7 in 2024-09-17T07:16:05.303 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:16:05.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:05.326 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:05.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:16:05.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:05 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/40632733' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-09-17T07:16:05.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:05 smithi100 ceph-mon[29092]: osdmap e297: 8 total, 8 up, 7 in 2024-09-17T07:16:05.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:05 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/40632733' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-09-17T07:16:05.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:05 smithi100 ceph-mon[39462]: osdmap e297: 8 total, 8 up, 7 in 2024-09-17T07:16:05.427 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:16:05.450 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:05.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:05.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:16:05.473 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd pool application enable unique_pool_3 rados --yes-i-really-mean-it || true 2024-09-17T07:16:05.474 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:16:05.551 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:16:05.572 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:05.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:05.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:16:05.674 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:16:05.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:05.696 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:05.696 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:16:05.797 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:16:05.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:05.819 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:05.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:16:05.920 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:16:05.944 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:05.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:05.945 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:16:06.046 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:16:06.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:06.070 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:06.070 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:16:06.171 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:16:06.193 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:06.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:06.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:16:06.266 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:06 smithi100 ceph-mon[29092]: osdmap e298: 8 total, 8 up, 7 in 2024-09-17T07:16:06.266 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:06 smithi100 ceph-mon[29092]: pgmap v622: 148 pgs: 3 creating+peering, 13 unknown, 132 active+clean; 582 KiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-17T07:16:06.266 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:16:05 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:16:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:16:06.267 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:06 smithi100 ceph-mon[39462]: osdmap e298: 8 total, 8 up, 7 in 2024-09-17T07:16:06.267 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:06 smithi100 ceph-mon[39462]: pgmap v622: 148 pgs: 3 creating+peering, 13 unknown, 132 active+clean; 582 KiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-17T07:16:06.295 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:16:06.316 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:06.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:06.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:16:06.418 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:16:06.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:06.440 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:06.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:16:06.498 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:16:06.499 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:16:06.514 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1: {} 2024-09-17T07:16:06.515 INFO:teuthology.orchestra.run.smithi100.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:16:06.535 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:16:06.535 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:16:06.541 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:16:06.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:06 smithi135 ceph-mon[33238]: osdmap e298: 8 total, 8 up, 7 in 2024-09-17T07:16:06.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:06 smithi135 ceph-mon[33238]: pgmap v622: 148 pgs: 3 creating+peering, 13 unknown, 132 active+clean; 582 KiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-17T07:16:06.551 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:16:06.551 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:16:06.563 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:06.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:06.564 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:16:06.568 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:16:06.568 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:16:06.585 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:16:06.585 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:16:06.602 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:16:06.602 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:16:06.621 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:16:06.621 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:16:06.665 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:16:06.686 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:06.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:06.687 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:16:06.788 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:16:06.809 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:06.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:06.810 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:16:06.911 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:16:06.937 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:06.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:06.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:16:07.039 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:16:07.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:07.062 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:07.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:16:07.071 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:07 smithi100 ceph-mon[29092]: osdmap e299: 8 total, 8 up, 7 in 2024-09-17T07:16:07.071 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:07 smithi100 ceph-mon[29092]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-17T07:16:07.071 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:07 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1903568432' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:16:07.071 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:07 smithi100 ceph-mon[39462]: osdmap e299: 8 total, 8 up, 7 in 2024-09-17T07:16:07.072 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:07 smithi100 ceph-mon[39462]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-17T07:16:07.072 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:07 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1903568432' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:16:07.162 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:16:07.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:07.185 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:07.186 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:16:07.287 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:16:07.311 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:07.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:07.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:16:07.413 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:16:07.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:07.438 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:07.438 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:16:07.539 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:16:07.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:07 smithi135 ceph-mon[33238]: osdmap e299: 8 total, 8 up, 7 in 2024-09-17T07:16:07.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:07 smithi135 ceph-mon[33238]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-17T07:16:07.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:07 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1903568432' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:16:07.564 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:07.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:07.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:16:07.650 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:16:07.666 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:16:07.691 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:07.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:07.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:16:07.793 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:16:07.817 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:07.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:07.818 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:16:07.919 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:16:07.943 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:07.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:07.944 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:16:08.045 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:16:08.067 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:08.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:08.068 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:16:08.119 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-17T07:16:08.170 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:16:08.194 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:08.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:08.195 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:16:08.296 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:16:08.320 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:08.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:08.321 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:16:08.347 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:08 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:16:08.347 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:08 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:16:08.347 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:08 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:16:08.347 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:08 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:16:08.347 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:08 smithi100 ceph-mon[29092]: osdmap e300: 8 total, 8 up, 7 in 2024-09-17T07:16:08.347 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:08 smithi100 ceph-mon[29092]: pgmap v625: 148 pgs: 3 creating+peering, 13 unknown, 132 active+clean; 582 KiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:16:08.347 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:08 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:16:08.347 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:08 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:16:08.348 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:08 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:16:08.348 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:08 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:16:08.348 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:08 smithi100 ceph-mon[39462]: osdmap e300: 8 total, 8 up, 7 in 2024-09-17T07:16:08.348 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:08 smithi100 ceph-mon[39462]: pgmap v625: 148 pgs: 3 creating+peering, 13 unknown, 132 active+clean; 582 KiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:16:08.421 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:16:08.444 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:08.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:08.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:16:08.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:08 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:16:08.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:08 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:16:08.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:08 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:16:08.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:08 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:16:08.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:08 smithi135 ceph-mon[33238]: osdmap e300: 8 total, 8 up, 7 in 2024-09-17T07:16:08.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:08 smithi135 ceph-mon[33238]: pgmap v625: 148 pgs: 3 creating+peering, 13 unknown, 132 active+clean; 582 KiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:16:08.545 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:16:08.568 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:08.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:08.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:16:08.670 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:16:08.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:08.694 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:08.694 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:16:08.795 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:16:08.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:08.818 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:08.818 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:16:08.919 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:16:08.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:08.941 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:08.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:16:09.042 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:16:09.063 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:09.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:09.064 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:16:09.165 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:16:09.186 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:09.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:09.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:16:09.288 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:16:09.293 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:16:08 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:16:09.310 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:09.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:09.311 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:16:09.411 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:16:09.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:09.434 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:09.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:16:09.536 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:16:09.559 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:09.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:09.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:16:09.567 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:16:09.592 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:16:09.637 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:16:09.661 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:16:09.702 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:09.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:09.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:16:09.804 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:16:09.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:09.828 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:09.828 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:16:09.929 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:16:09.951 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:09.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:09.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:16:10.052 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:16:10.074 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:10.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:10.075 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:16:10.176 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:16:10.197 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:10.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:10.199 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:16:10.300 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:16:10.321 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:10.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:10.323 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:16:10.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:10 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:16:10.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:10 smithi100 ceph-mon[29092]: pgmap v626: 148 pgs: 3 creating+peering, 145 active+clean; 582 KiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 974 B/s rd, 0 op/s 2024-09-17T07:16:10.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:10 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:16:10.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:10 smithi100 ceph-mon[39462]: pgmap v626: 148 pgs: 3 creating+peering, 145 active+clean; 582 KiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 974 B/s rd, 0 op/s 2024-09-17T07:16:10.424 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:16:10.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:10.446 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:10.446 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:16:10.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:10 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:16:10.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:10 smithi135 ceph-mon[33238]: pgmap v626: 148 pgs: 3 creating+peering, 145 active+clean; 582 KiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 974 B/s rd, 0 op/s 2024-09-17T07:16:10.547 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:16:10.568 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:10.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:10.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:16:10.670 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:16:10.691 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:10.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:10.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:16:10.793 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:16:10.814 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:10.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:10.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:16:10.916 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:16:10.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:10.939 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:10.939 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:16:11.040 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:16:11.063 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:11.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:11.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:16:11.164 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:16:11.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:11.187 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:11.187 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:16:11.288 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:16:11.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:11.311 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:11.311 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:16:11.412 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:16:11.435 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:11.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:11.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:16:11.537 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:16:11.559 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:11.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:11.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:16:11.625 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T07:16:11.627 INFO:teuthology.orchestra.run.smithi100.stderr:dumped all 2024-09-17T07:16:11.661 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:16:11.682 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:11.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:11.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:16:11.784 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:16:11.806 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:11.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:11.808 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:16:11.908 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:16:11.929 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:11.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:11.930 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:16:12.031 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:16:12.055 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:12.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:12.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:16:12.157 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:16:12.179 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:12.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:12.180 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:16:12.281 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:16:12.303 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:12.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:12.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:16:12.405 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:16:12.427 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:12.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:12.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:16:12.529 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:16:12.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:12.552 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:12.552 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:16:12.599 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:12 smithi100 ceph-mon[29092]: pgmap v627: 148 pgs: 148 active+clean; 582 KiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 820 B/s rd, 0 op/s 2024-09-17T07:16:12.599 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:12 smithi100 ceph-mon[39462]: pgmap v627: 148 pgs: 148 active+clean; 582 KiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 820 B/s rd, 0 op/s 2024-09-17T07:16:12.653 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:16:12.675 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:12.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:12.676 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:16:12.776 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:16:12.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:12 smithi135 ceph-mon[33238]: pgmap v627: 148 pgs: 148 active+clean; 582 KiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 820 B/s rd, 0 op/s 2024-09-17T07:16:12.798 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:12.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:12.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:16:12.899 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:16:12.922 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:12.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:12.923 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:16:13.002 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-17T07:16:13.002 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-17T07:16:13.002 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-17T07:16:13.002 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-17T07:16:13.002 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-17T07:16:13.024 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:16:13.046 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:13.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:13.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:16:13.149 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:16:13.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:13.171 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:13.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:16:13.272 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:16:13.294 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:13.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:13.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:16:13.396 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:16:13.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:13.419 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:13.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:16:13.520 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:16:13.542 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:13.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:13.543 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:16:13.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:13 smithi100 ceph-mon[29092]: from='client.25475 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:16:13.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:13 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:16:13.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:13 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:16:13.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:13 smithi100 ceph-mon[39462]: from='client.25475 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:16:13.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:13 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:16:13.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:13 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:16:13.644 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:16:13.664 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:13.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:13.666 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:16:13.767 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:16:13.789 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:13.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:13.790 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:16:13.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:13 smithi135 ceph-mon[33238]: from='client.25475 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:16:13.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:13 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:16:13.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:13 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:16:13.891 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:16:13.913 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:13.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:13.914 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:16:14.015 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:16:14.037 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:14.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:14.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:16:14.072 INFO:teuthology.orchestra.run.smithi100.stdout:{"pg_ready":true,"pg_map":{"version":627,"stamp":"2024-09-17T07:16:11.284542+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596246,"num_objects":196,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":2173,"num_read_kb":2019,"num_write":895,"num_write_kb":2459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":123,"num_bytes_recovered":1773876,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":946,"ondisk_log_size":946,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":434,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":3809508,"kb_used_data":7544,"kb_used_omap":10,"kb_used_meta":3801845,"kb_avail":652406556,"statfs":{"total":671965249536,"available":668064313344,"internally_reserved":0,"allocated":7725056,"data_stored":4811690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11152,"internal_metadata":3893089392},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.242763"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":141,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.074537+0000","last_change":"2024-09-17T07:13:11.978300+0000","last_active":"2024-09-17T07:16:06.074537+0000","last_peered":"2024-09-17T07:16:06.074537+0000","last_clean":"2024-09-17T07:16:06.074537+0000","last_became_active":"2024-09-17T07:09:24.401344+0000","last_became_peered":"2024-09-17T07:09:24.401344+0000","last_unstale":"2024-09-17T07:16:06.074537+0000","last_undegraded":"2024-09-17T07:16:06.074537+0000","last_fullsized":"2024-09-17T07:16:06.074537+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:11.978185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:33.644728+0000","last_clean_scrub_stamp":"2024-09-17T07:13:11.978185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0074436570000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":302,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.073657+0000","last_change":"2024-09-17T07:13:54.566881+0000","last_active":"2024-09-17T07:16:06.073657+0000","last_peered":"2024-09-17T07:16:06.073657+0000","last_clean":"2024-09-17T07:16:06.073657+0000","last_became_active":"2024-09-17T07:02:12.006754+0000","last_became_peered":"2024-09-17T07:02:12.006754+0000","last_unstale":"2024-09-17T07:16:06.073657+0000","last_undegraded":"2024-09-17T07:16:06.073657+0000","last_fullsized":"2024-09-17T07:16:06.073657+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:54.566757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:13:54.566757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0074097110000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"242'27","reported_seq":212,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.073130+0000","last_change":"2024-09-17T07:14:47.661623+0000","last_active":"2024-09-17T07:16:06.073130+0000","last_peered":"2024-09-17T07:16:06.073130+0000","last_clean":"2024-09-17T07:16:06.073130+0000","last_became_active":"2024-09-17T07:09:24.402042+0000","last_became_peered":"2024-09-17T07:09:24.402042+0000","last_unstale":"2024-09-17T07:16:06.073130+0000","last_undegraded":"2024-09-17T07:16:06.073130+0000","last_fullsized":"2024-09-17T07:16:06.073130+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'27","last_scrub_stamp":"2024-09-17T07:14:47.661187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:14:47.661187+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010587222,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":298,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.042275+0000","last_change":"2024-09-17T07:15:45.261193+0000","last_active":"2024-09-17T07:16:06.042275+0000","last_peered":"2024-09-17T07:16:06.042275+0000","last_clean":"2024-09-17T07:16:06.042275+0000","last_became_active":"2024-09-17T07:02:16.014928+0000","last_became_peered":"2024-09-17T07:02:16.014928+0000","last_unstale":"2024-09-17T07:16:06.042275+0000","last_undegraded":"2024-09-17T07:16:06.042275+0000","last_fullsized":"2024-09-17T07:16:06.042275+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:45.261117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:15:45.261117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:00.371170+0000","scrub_duration":0.001477482,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":153,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.073736+0000","last_change":"2024-09-17T07:14:57.520400+0000","last_active":"2024-09-17T07:16:06.073736+0000","last_peered":"2024-09-17T07:16:06.073736+0000","last_clean":"2024-09-17T07:16:06.073736+0000","last_became_active":"2024-09-17T07:09:24.402568+0000","last_became_peered":"2024-09-17T07:09:24.402568+0000","last_unstale":"2024-09-17T07:16:06.073736+0000","last_undegraded":"2024-09-17T07:16:06.073736+0000","last_fullsized":"2024-09-17T07:16:06.073736+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:57.520290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:14:57.520290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:15.653444+0000","scrub_duration":0.001539143,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"242'15","reported_seq":336,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.042290+0000","last_change":"2024-09-17T07:15:39.306155+0000","last_active":"2024-09-17T07:16:06.042290+0000","last_peered":"2024-09-17T07:16:06.042290+0000","last_clean":"2024-09-17T07:16:06.042290+0000","last_became_active":"2024-09-17T07:02:14.016244+0000","last_became_peered":"2024-09-17T07:02:14.016244+0000","last_unstale":"2024-09-17T07:16:06.042290+0000","last_undegraded":"2024-09-17T07:16:06.042290+0000","last_fullsized":"2024-09-17T07:16:06.042290+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'15","last_scrub_stamp":"2024-09-17T07:15:39.306043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:15:39.306043+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:57.372120+0000","scrub_duration":0.002471029,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":297,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.047928+0000","last_change":"2024-09-17T07:15:53.881909+0000","last_active":"2024-09-17T07:16:06.047928+0000","last_peered":"2024-09-17T07:16:06.047928+0000","last_clean":"2024-09-17T07:16:06.047928+0000","last_became_active":"2024-09-17T07:02:16.014380+0000","last_became_peered":"2024-09-17T07:02:16.014380+0000","last_unstale":"2024-09-17T07:16:06.047928+0000","last_undegraded":"2024-09-17T07:16:06.047928+0000","last_fullsized":"2024-09-17T07:16:06.047928+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:53.881813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:15:53.881813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:00.007241+0000","scrub_duration":0.001414806,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":297,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.042526+0000","last_change":"2024-09-17T07:14:41.708103+0000","last_active":"2024-09-17T07:16:06.042526+0000","last_peered":"2024-09-17T07:16:06.042526+0000","last_clean":"2024-09-17T07:16:06.042526+0000","last_became_active":"2024-09-17T07:02:18.020790+0000","last_became_peered":"2024-09-17T07:02:18.020790+0000","last_unstale":"2024-09-17T07:16:06.042526+0000","last_undegraded":"2024-09-17T07:16:06.042526+0000","last_fullsized":"2024-09-17T07:16:06.042526+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:41.707969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:14:41.707969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.014931757,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":300,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.075391+0000","last_change":"2024-09-17T07:15:46.312474+0000","last_active":"2024-09-17T07:16:06.075391+0000","last_peered":"2024-09-17T07:16:06.075391+0000","last_clean":"2024-09-17T07:16:06.075391+0000","last_became_active":"2024-09-17T07:02:12.011129+0000","last_became_peered":"2024-09-17T07:02:12.011129+0000","last_unstale":"2024-09-17T07:16:06.075391+0000","last_undegraded":"2024-09-17T07:16:06.075391+0000","last_fullsized":"2024-09-17T07:16:06.075391+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:46.312371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:46.312371+0000","last_clean_scrub_stamp":"2024-09-17T07:15:46.312371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:06.831380+0000","scrub_duration":0.0015176510000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"242'7","reported_seq":329,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.042550+0000","last_change":"2024-09-17T07:14:42.653845+0000","last_active":"2024-09-17T07:16:06.042550+0000","last_peered":"2024-09-17T07:16:06.042550+0000","last_clean":"2024-09-17T07:16:06.042550+0000","last_became_active":"2024-09-17T07:09:24.396900+0000","last_became_peered":"2024-09-17T07:09:24.396900+0000","last_unstale":"2024-09-17T07:16:06.042550+0000","last_undegraded":"2024-09-17T07:16:06.042550+0000","last_fullsized":"2024-09-17T07:16:06.042550+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'7","last_scrub_stamp":"2024-09-17T07:14:42.653705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:14:42.653705+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0050149230000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":296,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.073785+0000","last_change":"2024-09-17T07:14:38.000519+0000","last_active":"2024-09-17T07:16:06.073785+0000","last_peered":"2024-09-17T07:16:06.073785+0000","last_clean":"2024-09-17T07:16:06.073785+0000","last_became_active":"2024-09-17T07:02:16.015401+0000","last_became_peered":"2024-09-17T07:02:16.015401+0000","last_unstale":"2024-09-17T07:16:06.073785+0000","last_undegraded":"2024-09-17T07:16:06.073785+0000","last_fullsized":"2024-09-17T07:16:06.073785+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:37.997296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:14:37.997296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045486420000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":315,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.073546+0000","last_change":"2024-09-17T07:14:52.168308+0000","last_active":"2024-09-17T07:16:06.073546+0000","last_peered":"2024-09-17T07:16:06.073546+0000","last_clean":"2024-09-17T07:16:06.073546+0000","last_became_active":"2024-09-17T07:09:24.402616+0000","last_became_peered":"2024-09-17T07:09:24.402616+0000","last_unstale":"2024-09-17T07:16:06.073546+0000","last_undegraded":"2024-09-17T07:16:06.073546+0000","last_fullsized":"2024-09-17T07:16:06.073546+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:52.168181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:14:52.168181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0056501399999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":300,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.074560+0000","last_change":"2024-09-17T07:14:44.076126+0000","last_active":"2024-09-17T07:16:06.074560+0000","last_peered":"2024-09-17T07:16:06.074560+0000","last_clean":"2024-09-17T07:16:06.074560+0000","last_became_active":"2024-09-17T07:02:12.012234+0000","last_became_peered":"2024-09-17T07:02:12.012234+0000","last_unstale":"2024-09-17T07:16:06.074560+0000","last_undegraded":"2024-09-17T07:16:06.074560+0000","last_fullsized":"2024-09-17T07:16:06.074560+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:44.076012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:45.029247+0000","last_clean_scrub_stamp":"2024-09-17T07:14:44.076012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.015917521,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"242'15","reported_seq":344,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.042736+0000","last_change":"2024-09-17T07:15:36.373317+0000","last_active":"2024-09-17T07:16:06.042736+0000","last_peered":"2024-09-17T07:16:06.042736+0000","last_clean":"2024-09-17T07:16:06.042736+0000","last_became_active":"2024-09-17T07:09:24.397954+0000","last_became_peered":"2024-09-17T07:09:24.397954+0000","last_unstale":"2024-09-17T07:16:06.042736+0000","last_undegraded":"2024-09-17T07:16:06.042736+0000","last_fullsized":"2024-09-17T07:16:06.042736+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'15","last_scrub_stamp":"2024-09-17T07:15:36.373220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:15:36.373220+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:37.392728+0000","scrub_duration":0.0019745959999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":307,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.074713+0000","last_change":"2024-09-17T07:14:59.140862+0000","last_active":"2024-09-17T07:16:06.074713+0000","last_peered":"2024-09-17T07:16:06.074713+0000","last_clean":"2024-09-17T07:16:06.074713+0000","last_became_active":"2024-09-17T07:02:16.023774+0000","last_became_peered":"2024-09-17T07:02:16.023774+0000","last_unstale":"2024-09-17T07:16:06.074713+0000","last_undegraded":"2024-09-17T07:16:06.074713+0000","last_fullsized":"2024-09-17T07:16:06.074713+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:59.140673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:28.096266+0000","last_clean_scrub_stamp":"2024-09-17T07:14:59.140673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:23.008311+0000","scrub_duration":0.013656697000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":296,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.072938+0000","last_change":"2024-09-17T07:15:45.483700+0000","last_active":"2024-09-17T07:16:06.072938+0000","last_peered":"2024-09-17T07:16:06.072938+0000","last_clean":"2024-09-17T07:16:06.072938+0000","last_became_active":"2024-09-17T07:02:18.029178+0000","last_became_peered":"2024-09-17T07:02:18.029178+0000","last_unstale":"2024-09-17T07:16:06.072938+0000","last_undegraded":"2024-09-17T07:16:06.072938+0000","last_fullsized":"2024-09-17T07:16:06.072938+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:45.483622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:15:45.483622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:12.943923+0000","scrub_duration":0.001342359,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":12,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-17T07:16:08.305704+0000","last_change":"2024-09-17T07:16:05.058011+0000","last_active":"2024-09-17T07:16:08.305704+0000","last_peered":"2024-09-17T07:16:08.305704+0000","last_clean":"2024-09-17T07:16:08.305704+0000","last_became_active":"2024-09-17T07:16:05.057730+0000","last_became_peered":"2024-09-17T07:16:05.057730+0000","last_unstale":"2024-09-17T07:16:08.305704+0000","last_undegraded":"2024-09-17T07:16:08.305704+0000","last_fullsized":"2024-09-17T07:16:08.305704+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":297,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:13.826432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"242'39","reported_seq":369,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.043000+0000","last_change":"2024-09-17T07:15:46.226654+0000","last_active":"2024-09-17T07:16:06.043000+0000","last_peered":"2024-09-17T07:16:06.043000+0000","last_clean":"2024-09-17T07:16:06.043000+0000","last_became_active":"2024-09-17T07:02:14.016270+0000","last_became_peered":"2024-09-17T07:02:14.016270+0000","last_unstale":"2024-09-17T07:16:06.043000+0000","last_undegraded":"2024-09-17T07:16:06.043000+0000","last_fullsized":"2024-09-17T07:16:06.043000+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'39","last_scrub_stamp":"2024-09-17T07:15:46.226554+0000","last_deep_scrub":"95'25","last_deep_scrub_stamp":"2024-09-17T07:04:09.450039+0000","last_clean_scrub_stamp":"2024-09-17T07:15:46.226554+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:02.790391+0000","scrub_duration":0.0027061799999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":303,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.074893+0000","last_change":"2024-09-17T07:13:25.951693+0000","last_active":"2024-09-17T07:16:06.074893+0000","last_peered":"2024-09-17T07:16:06.074893+0000","last_clean":"2024-09-17T07:16:06.074893+0000","last_became_active":"2024-09-17T07:02:12.011258+0000","last_became_peered":"2024-09-17T07:02:12.011258+0000","last_unstale":"2024-09-17T07:16:06.074893+0000","last_undegraded":"2024-09-17T07:16:06.074893+0000","last_fullsized":"2024-09-17T07:16:06.074893+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:25.950509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:13:25.950509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0099987270000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":287,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.048823+0000","last_change":"2024-09-17T07:13:46.502789+0000","last_active":"2024-09-17T07:16:06.048823+0000","last_peered":"2024-09-17T07:16:06.048823+0000","last_clean":"2024-09-17T07:16:06.048823+0000","last_became_active":"2024-09-17T07:02:16.018147+0000","last_became_peered":"2024-09-17T07:02:16.018147+0000","last_unstale":"2024-09-17T07:16:06.048823+0000","last_undegraded":"2024-09-17T07:16:06.048823+0000","last_fullsized":"2024-09-17T07:16:06.048823+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:46.502641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:13:46.502641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0074335069999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":144,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.043116+0000","last_change":"2024-09-17T07:14:38.777656+0000","last_active":"2024-09-17T07:16:06.043116+0000","last_peered":"2024-09-17T07:16:06.043116+0000","last_clean":"2024-09-17T07:16:06.043116+0000","last_became_active":"2024-09-17T07:09:24.396334+0000","last_became_peered":"2024-09-17T07:09:24.396334+0000","last_unstale":"2024-09-17T07:16:06.043116+0000","last_undegraded":"2024-09-17T07:16:06.043116+0000","last_fullsized":"2024-09-17T07:16:06.043116+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:38.777538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:42.984088+0000","last_clean_scrub_stamp":"2024-09-17T07:14:38.777538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003669088,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":12,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-17T07:16:08.052559+0000","last_change":"2024-09-17T07:16:05.048650+0000","last_active":"2024-09-17T07:16:08.052559+0000","last_peered":"2024-09-17T07:16:08.052559+0000","last_clean":"2024-09-17T07:16:08.052559+0000","last_became_active":"2024-09-17T07:16:05.048357+0000","last_became_peered":"2024-09-17T07:16:05.048357+0000","last_unstale":"2024-09-17T07:16:08.052559+0000","last_undegraded":"2024-09-17T07:16:08.052559+0000","last_fullsized":"2024-09-17T07:16:08.052559+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":297,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:17.709243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"242'25","reported_seq":368,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.042791+0000","last_change":"2024-09-17T07:15:02.535674+0000","last_active":"2024-09-17T07:16:06.042791+0000","last_peered":"2024-09-17T07:16:06.042791+0000","last_clean":"2024-09-17T07:16:06.042791+0000","last_became_active":"2024-09-17T07:09:24.399287+0000","last_became_peered":"2024-09-17T07:09:24.399287+0000","last_unstale":"2024-09-17T07:16:06.042791+0000","last_undegraded":"2024-09-17T07:16:06.042791+0000","last_fullsized":"2024-09-17T07:16:06.042791+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'25","last_scrub_stamp":"2024-09-17T07:15:02.532371+0000","last_deep_scrub":"176'15","last_deep_scrub_stamp":"2024-09-17T07:08:19.633791+0000","last_clean_scrub_stamp":"2024-09-17T07:15:02.532371+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:25.841604+0000","scrub_duration":0.012712606,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":310,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.048672+0000","last_change":"2024-09-17T07:14:59.697649+0000","last_active":"2024-09-17T07:16:06.048672+0000","last_peered":"2024-09-17T07:16:06.048672+0000","last_clean":"2024-09-17T07:16:06.048672+0000","last_became_active":"2024-09-17T07:09:24.406973+0000","last_became_peered":"2024-09-17T07:09:24.406973+0000","last_unstale":"2024-09-17T07:16:06.048672+0000","last_undegraded":"2024-09-17T07:16:06.048672+0000","last_fullsized":"2024-09-17T07:16:06.048672+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:59.697560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:14:59.697560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:19.402327+0000","scrub_duration":0.003544859,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"98'9","reported_seq":424,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.075208+0000","last_change":"2024-09-17T07:14:46.126177+0000","last_active":"2024-09-17T07:16:06.075208+0000","last_peered":"2024-09-17T07:16:06.075208+0000","last_clean":"2024-09-17T07:16:06.075208+0000","last_became_active":"2024-09-17T07:04:15.110797+0000","last_became_peered":"2024-09-17T07:04:15.110797+0000","last_unstale":"2024-09-17T07:16:06.075208+0000","last_undegraded":"2024-09-17T07:16:06.075208+0000","last_fullsized":"2024-09-17T07:16:06.075208+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-09-17T07:14:46.126045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:14:46.126045+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:11.141681+0000","scrub_duration":0.014185467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":308,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.043017+0000","last_change":"2024-09-17T07:14:50.554927+0000","last_active":"2024-09-17T07:16:06.043017+0000","last_peered":"2024-09-17T07:16:06.043017+0000","last_clean":"2024-09-17T07:16:06.043017+0000","last_became_active":"2024-09-17T07:02:18.017460+0000","last_became_peered":"2024-09-17T07:02:18.017460+0000","last_unstale":"2024-09-17T07:16:06.043017+0000","last_undegraded":"2024-09-17T07:16:06.043017+0000","last_fullsized":"2024-09-17T07:16:06.043017+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:50.554591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:14:50.554591+0000","last_clean_scrub_stamp":"2024-09-17T07:14:50.554591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:14.252478+0000","scrub_duration":0.0067125969999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":12,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-17T07:16:07.224248+0000","last_change":"2024-09-17T07:16:05.051123+0000","last_active":"2024-09-17T07:16:07.224248+0000","last_peered":"2024-09-17T07:16:07.224248+0000","last_clean":"2024-09-17T07:16:07.224248+0000","last_became_active":"2024-09-17T07:16:05.050872+0000","last_became_peered":"2024-09-17T07:16:05.050872+0000","last_unstale":"2024-09-17T07:16:07.224248+0000","last_undegraded":"2024-09-17T07:16:07.224248+0000","last_fullsized":"2024-09-17T07:16:07.224248+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":297,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:05.259471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"242'27","reported_seq":367,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.049035+0000","last_change":"2024-09-17T07:14:45.832566+0000","last_active":"2024-09-17T07:16:06.049035+0000","last_peered":"2024-09-17T07:16:06.049035+0000","last_clean":"2024-09-17T07:16:06.049035+0000","last_became_active":"2024-09-17T07:09:24.407311+0000","last_became_peered":"2024-09-17T07:09:24.407311+0000","last_unstale":"2024-09-17T07:16:06.049035+0000","last_undegraded":"2024-09-17T07:16:06.049035+0000","last_fullsized":"2024-09-17T07:16:06.049035+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'27","last_scrub_stamp":"2024-09-17T07:14:45.831734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:14:45.831734+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0075085209999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":311,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.047991+0000","last_change":"2024-09-17T07:15:44.897692+0000","last_active":"2024-09-17T07:16:06.047991+0000","last_peered":"2024-09-17T07:16:06.047991+0000","last_clean":"2024-09-17T07:16:06.047991+0000","last_became_active":"2024-09-17T07:09:24.405655+0000","last_became_peered":"2024-09-17T07:09:24.405655+0000","last_unstale":"2024-09-17T07:16:06.047991+0000","last_undegraded":"2024-09-17T07:16:06.047991+0000","last_fullsized":"2024-09-17T07:16:06.047991+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:44.897611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:15:44.897611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:51.136422+0000","scrub_duration":0.0030827340000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"75'8","reported_seq":474,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.043125+0000","last_change":"2024-09-17T07:15:40.259961+0000","last_active":"2024-09-17T07:16:06.043125+0000","last_peered":"2024-09-17T07:16:06.043125+0000","last_clean":"2024-09-17T07:16:06.043125+0000","last_became_active":"2024-09-17T07:02:16.021762+0000","last_became_peered":"2024-09-17T07:02:16.021762+0000","last_unstale":"2024-09-17T07:16:06.043125+0000","last_undegraded":"2024-09-17T07:16:06.043125+0000","last_fullsized":"2024-09-17T07:16:06.043125+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:15:40.259852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:15:40.259852+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:44.641715+0000","scrub_duration":0.001523993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":306,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.074870+0000","last_change":"2024-09-17T07:14:56.146344+0000","last_active":"2024-09-17T07:16:06.074870+0000","last_peered":"2024-09-17T07:16:06.074870+0000","last_clean":"2024-09-17T07:16:06.074870+0000","last_became_active":"2024-09-17T07:02:18.025332+0000","last_became_peered":"2024-09-17T07:02:18.025332+0000","last_unstale":"2024-09-17T07:16:06.074870+0000","last_undegraded":"2024-09-17T07:16:06.074870+0000","last_fullsized":"2024-09-17T07:16:06.074870+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:56.146098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:14:56.146098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0054479960000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":11,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.073269+0000","last_change":"2024-09-17T07:16:05.056703+0000","last_active":"2024-09-17T07:16:06.073269+0000","last_peered":"2024-09-17T07:16:06.073269+0000","last_clean":"2024-09-17T07:16:06.073269+0000","last_became_active":"2024-09-17T07:16:05.056428+0000","last_became_peered":"2024-09-17T07:16:05.056428+0000","last_unstale":"2024-09-17T07:16:06.073269+0000","last_undegraded":"2024-09-17T07:16:06.073269+0000","last_fullsized":"2024-09-17T07:16:06.073269+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":297,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:17.159975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"242'19","reported_seq":343,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.043217+0000","last_change":"2024-09-17T07:14:55.490025+0000","last_active":"2024-09-17T07:16:06.043217+0000","last_peered":"2024-09-17T07:16:06.043217+0000","last_clean":"2024-09-17T07:16:06.043217+0000","last_became_active":"2024-09-17T07:02:14.010227+0000","last_became_peered":"2024-09-17T07:02:14.010227+0000","last_unstale":"2024-09-17T07:16:06.043217+0000","last_undegraded":"2024-09-17T07:16:06.043217+0000","last_fullsized":"2024-09-17T07:16:06.043217+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'19","last_scrub_stamp":"2024-09-17T07:14:55.489874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:14:55.489874+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:23.724406+0000","scrub_duration":0.0045924440000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":302,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.073600+0000","last_change":"2024-09-17T07:13:21.858251+0000","last_active":"2024-09-17T07:16:06.073600+0000","last_peered":"2024-09-17T07:16:06.073600+0000","last_clean":"2024-09-17T07:16:06.073600+0000","last_became_active":"2024-09-17T07:02:12.013182+0000","last_became_peered":"2024-09-17T07:02:12.013182+0000","last_unstale":"2024-09-17T07:16:06.073600+0000","last_undegraded":"2024-09-17T07:16:06.073600+0000","last_fullsized":"2024-09-17T07:16:06.073600+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:21.858127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:13:21.858127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033279709999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":319,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.075289+0000","last_change":"2024-09-17T07:15:53.424831+0000","last_active":"2024-09-17T07:16:06.075289+0000","last_peered":"2024-09-17T07:16:06.075289+0000","last_clean":"2024-09-17T07:16:06.075289+0000","last_became_active":"2024-09-17T07:02:16.020224+0000","last_became_peered":"2024-09-17T07:02:16.020224+0000","last_unstale":"2024-09-17T07:16:06.075289+0000","last_undegraded":"2024-09-17T07:16:06.075289+0000","last_fullsized":"2024-09-17T07:16:06.075289+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:53.424736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:15:53.424736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:53.871415+0000","scrub_duration":0.001270806,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":161,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-17T07:16:05.047732+0000","last_change":"2024-09-17T07:14:51.530956+0000","last_active":"2024-09-17T07:16:05.047732+0000","last_peered":"2024-09-17T07:16:05.047732+0000","last_clean":"2024-09-17T07:16:05.047732+0000","last_became_active":"2024-09-17T07:09:24.408109+0000","last_became_peered":"2024-09-17T07:09:24.408109+0000","last_unstale":"2024-09-17T07:16:05.047732+0000","last_undegraded":"2024-09-17T07:16:05.047732+0000","last_fullsized":"2024-09-17T07:16:05.047732+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:51.530841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:13:22.187127+0000","last_clean_scrub_stamp":"2024-09-17T07:14:51.530841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:17.322714+0000","scrub_duration":0.0027489939999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":11,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.075944+0000","last_change":"2024-09-17T07:16:05.052296+0000","last_active":"2024-09-17T07:16:06.075944+0000","last_peered":"2024-09-17T07:16:06.075944+0000","last_clean":"2024-09-17T07:16:06.075944+0000","last_became_active":"2024-09-17T07:16:05.052037+0000","last_became_peered":"2024-09-17T07:16:05.052037+0000","last_unstale":"2024-09-17T07:16:06.075944+0000","last_undegraded":"2024-09-17T07:16:06.075944+0000","last_fullsized":"2024-09-17T07:16:06.075944+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":297,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:13.511460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"242'28","reported_seq":343,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.042424+0000","last_change":"2024-09-17T07:15:43.270581+0000","last_active":"2024-09-17T07:16:06.042424+0000","last_peered":"2024-09-17T07:16:06.042424+0000","last_clean":"2024-09-17T07:16:06.042424+0000","last_became_active":"2024-09-17T07:02:14.007810+0000","last_became_peered":"2024-09-17T07:02:14.007810+0000","last_unstale":"2024-09-17T07:16:06.042424+0000","last_undegraded":"2024-09-17T07:16:06.042424+0000","last_fullsized":"2024-09-17T07:16:06.042424+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'28","last_scrub_stamp":"2024-09-17T07:15:43.270473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:15:43.270473+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:07.514992+0000","scrub_duration":0.0022765149999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":344,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-17T07:16:05.048569+0000","last_change":"2024-09-17T07:15:58.575503+0000","last_active":"2024-09-17T07:16:05.048569+0000","last_peered":"2024-09-17T07:16:05.048569+0000","last_clean":"2024-09-17T07:16:05.048569+0000","last_became_active":"2024-09-17T07:02:12.011766+0000","last_became_peered":"2024-09-17T07:02:12.011766+0000","last_unstale":"2024-09-17T07:16:05.048569+0000","last_undegraded":"2024-09-17T07:16:05.048569+0000","last_fullsized":"2024-09-17T07:16:05.048569+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:58.575409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:09:23.181028+0000","last_clean_scrub_stamp":"2024-09-17T07:15:58.575409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:05.657241+0000","scrub_duration":0.0012800719999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"196'9","reported_seq":246,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.047484+0000","last_change":"2024-09-17T07:15:55.813983+0000","last_active":"2024-09-17T07:16:06.047484+0000","last_peered":"2024-09-17T07:16:06.047484+0000","last_clean":"2024-09-17T07:16:06.047484+0000","last_became_active":"2024-09-17T07:09:24.406056+0000","last_became_peered":"2024-09-17T07:09:24.406056+0000","last_unstale":"2024-09-17T07:16:06.047484+0000","last_undegraded":"2024-09-17T07:16:06.047484+0000","last_fullsized":"2024-09-17T07:16:06.047484+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'9","last_scrub_stamp":"2024-09-17T07:15:55.813882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:15:55.813882+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:56.978977+0000","scrub_duration":0.001931716,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":306,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.074617+0000","last_change":"2024-09-17T07:15:52.378479+0000","last_active":"2024-09-17T07:16:06.074617+0000","last_peered":"2024-09-17T07:16:06.074617+0000","last_clean":"2024-09-17T07:16:06.074617+0000","last_became_active":"2024-09-17T07:02:18.364981+0000","last_became_peered":"2024-09-17T07:02:18.364981+0000","last_unstale":"2024-09-17T07:16:06.074617+0000","last_undegraded":"2024-09-17T07:16:06.074617+0000","last_fullsized":"2024-09-17T07:16:06.074617+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:52.378384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:10:33.727472+0000","last_clean_scrub_stamp":"2024-09-17T07:15:52.378384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:07.710387+0000","scrub_duration":0.001396736,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":12,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-17T07:16:07.225412+0000","last_change":"2024-09-17T07:16:05.054368+0000","last_active":"2024-09-17T07:16:07.225412+0000","last_peered":"2024-09-17T07:16:07.225412+0000","last_clean":"2024-09-17T07:16:07.225412+0000","last_became_active":"2024-09-17T07:16:05.054109+0000","last_became_peered":"2024-09-17T07:16:05.054109+0000","last_unstale":"2024-09-17T07:16:07.225412+0000","last_undegraded":"2024-09-17T07:16:07.225412+0000","last_fullsized":"2024-09-17T07:16:07.225412+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":297,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:25.923906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"242'30","reported_seq":221,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.073491+0000","last_change":"2024-09-17T07:13:28.997698+0000","last_active":"2024-09-17T07:16:06.073491+0000","last_peered":"2024-09-17T07:16:06.073491+0000","last_clean":"2024-09-17T07:16:06.073491+0000","last_became_active":"2024-09-17T07:09:24.403240+0000","last_became_peered":"2024-09-17T07:09:24.403240+0000","last_unstale":"2024-09-17T07:16:06.073491+0000","last_undegraded":"2024-09-17T07:16:06.073491+0000","last_fullsized":"2024-09-17T07:16:06.073491+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'30","last_scrub_stamp":"2024-09-17T07:13:28.997563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:13:28.997563+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0061961159999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":312,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.048743+0000","last_change":"2024-09-17T07:15:56.848779+0000","last_active":"2024-09-17T07:16:06.048743+0000","last_peered":"2024-09-17T07:16:06.048743+0000","last_clean":"2024-09-17T07:16:06.048743+0000","last_became_active":"2024-09-17T07:02:12.005687+0000","last_became_peered":"2024-09-17T07:02:12.005687+0000","last_unstale":"2024-09-17T07:16:06.048743+0000","last_undegraded":"2024-09-17T07:16:06.048743+0000","last_fullsized":"2024-09-17T07:16:06.048743+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:56.848689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:56.848689+0000","last_clean_scrub_stamp":"2024-09-17T07:15:56.848689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:25.793491+0000","scrub_duration":0.0020256440000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":308,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.042336+0000","last_change":"2024-09-17T07:14:52.606726+0000","last_active":"2024-09-17T07:16:06.042336+0000","last_peered":"2024-09-17T07:16:06.042336+0000","last_clean":"2024-09-17T07:16:06.042336+0000","last_became_active":"2024-09-17T07:09:24.393375+0000","last_became_peered":"2024-09-17T07:09:24.393375+0000","last_unstale":"2024-09-17T07:16:06.042336+0000","last_undegraded":"2024-09-17T07:16:06.042336+0000","last_fullsized":"2024-09-17T07:16:06.042336+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:52.606401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:14:52.606401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0068634849999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":306,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.042678+0000","last_change":"2024-09-17T07:14:39.758541+0000","last_active":"2024-09-17T07:16:06.042678+0000","last_peered":"2024-09-17T07:16:06.042678+0000","last_clean":"2024-09-17T07:16:06.042678+0000","last_became_active":"2024-09-17T07:09:24.398659+0000","last_became_peered":"2024-09-17T07:09:24.398659+0000","last_unstale":"2024-09-17T07:16:06.042678+0000","last_undegraded":"2024-09-17T07:16:06.042678+0000","last_fullsized":"2024-09-17T07:16:06.042678+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:39.758427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:14:39.758427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019183889999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":11,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.074835+0000","last_change":"2024-09-17T07:16:05.052409+0000","last_active":"2024-09-17T07:16:06.074835+0000","last_peered":"2024-09-17T07:16:06.074835+0000","last_clean":"2024-09-17T07:16:06.074835+0000","last_became_active":"2024-09-17T07:16:05.052153+0000","last_became_peered":"2024-09-17T07:16:05.052153+0000","last_unstale":"2024-09-17T07:16:06.074835+0000","last_undegraded":"2024-09-17T07:16:06.074835+0000","last_fullsized":"2024-09-17T07:16:06.074835+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":297,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:17.269415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"242'29","reported_seq":352,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.073571+0000","last_change":"2024-09-17T07:15:38.507145+0000","last_active":"2024-09-17T07:16:06.073571+0000","last_peered":"2024-09-17T07:16:06.073571+0000","last_clean":"2024-09-17T07:16:06.073571+0000","last_became_active":"2024-09-17T07:02:14.027389+0000","last_became_peered":"2024-09-17T07:02:14.027389+0000","last_unstale":"2024-09-17T07:16:06.073571+0000","last_undegraded":"2024-09-17T07:16:06.073571+0000","last_fullsized":"2024-09-17T07:16:06.073571+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'29","last_scrub_stamp":"2024-09-17T07:15:38.507042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:15:38.507042+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:40.074077+0000","scrub_duration":0.0024959420000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":302,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.075546+0000","last_change":"2024-09-17T07:13:23.945293+0000","last_active":"2024-09-17T07:16:06.075546+0000","last_peered":"2024-09-17T07:16:06.075546+0000","last_clean":"2024-09-17T07:16:06.075546+0000","last_became_active":"2024-09-17T07:02:12.011917+0000","last_became_peered":"2024-09-17T07:02:12.011917+0000","last_unstale":"2024-09-17T07:16:06.075546+0000","last_undegraded":"2024-09-17T07:16:06.075546+0000","last_fullsized":"2024-09-17T07:16:06.075546+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:23.945157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:24.072799+0000","last_clean_scrub_stamp":"2024-09-17T07:13:23.945157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0029434130000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":319,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-17T07:16:05.047820+0000","last_change":"2024-09-17T07:15:56.525463+0000","last_active":"2024-09-17T07:16:05.047820+0000","last_peered":"2024-09-17T07:16:05.047820+0000","last_clean":"2024-09-17T07:16:05.047820+0000","last_became_active":"2024-09-17T07:02:16.016199+0000","last_became_peered":"2024-09-17T07:02:16.016199+0000","last_unstale":"2024-09-17T07:16:05.047820+0000","last_undegraded":"2024-09-17T07:16:05.047820+0000","last_fullsized":"2024-09-17T07:16:05.047820+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:56.525361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:11:17.583712+0000","last_clean_scrub_stamp":"2024-09-17T07:15:56.525361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:12.314220+0000","scrub_duration":0.0013436679999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":295,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.047884+0000","last_change":"2024-09-17T07:13:50.514634+0000","last_active":"2024-09-17T07:16:06.047884+0000","last_peered":"2024-09-17T07:16:06.047884+0000","last_clean":"2024-09-17T07:16:06.047884+0000","last_became_active":"2024-09-17T07:02:18.364835+0000","last_became_peered":"2024-09-17T07:02:18.364835+0000","last_unstale":"2024-09-17T07:16:06.047884+0000","last_undegraded":"2024-09-17T07:16:06.047884+0000","last_fullsized":"2024-09-17T07:16:06.047884+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:50.514525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:13:50.514525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010637423,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":12,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-17T07:16:08.499432+0000","last_change":"2024-09-17T07:16:05.057855+0000","last_active":"2024-09-17T07:16:08.499432+0000","last_peered":"2024-09-17T07:16:08.499432+0000","last_clean":"2024-09-17T07:16:08.499432+0000","last_became_active":"2024-09-17T07:16:05.057588+0000","last_became_peered":"2024-09-17T07:16:05.057588+0000","last_unstale":"2024-09-17T07:16:08.499432+0000","last_undegraded":"2024-09-17T07:16:08.499432+0000","last_fullsized":"2024-09-17T07:16:08.499432+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":297,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:20.816387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"242'25","reported_seq":334,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.048450+0000","last_change":"2024-09-17T07:14:50.794400+0000","last_active":"2024-09-17T07:16:06.048450+0000","last_peered":"2024-09-17T07:16:06.048450+0000","last_clean":"2024-09-17T07:16:06.048450+0000","last_became_active":"2024-09-17T07:02:14.010637+0000","last_became_peered":"2024-09-17T07:02:14.010637+0000","last_unstale":"2024-09-17T07:16:06.048450+0000","last_undegraded":"2024-09-17T07:16:06.048450+0000","last_fullsized":"2024-09-17T07:16:06.048450+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'25","last_scrub_stamp":"2024-09-17T07:14:50.794200+0000","last_deep_scrub":"110'15","last_deep_scrub_stamp":"2024-09-17T07:04:57.091400+0000","last_clean_scrub_stamp":"2024-09-17T07:14:50.794200+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0058730839999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":301,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.049050+0000","last_change":"2024-09-17T07:14:49.765133+0000","last_active":"2024-09-17T07:16:06.049050+0000","last_peered":"2024-09-17T07:16:06.049050+0000","last_clean":"2024-09-17T07:16:06.049050+0000","last_became_active":"2024-09-17T07:02:12.010803+0000","last_became_peered":"2024-09-17T07:02:12.010803+0000","last_unstale":"2024-09-17T07:16:06.049050+0000","last_undegraded":"2024-09-17T07:16:06.049050+0000","last_fullsized":"2024-09-17T07:16:06.049050+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:49.764885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:25.147878+0000","last_clean_scrub_stamp":"2024-09-17T07:14:49.764885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:08.217541+0000","scrub_duration":0.002419587,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"89'9","reported_seq":485,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.073395+0000","last_change":"2024-09-17T07:15:53.512860+0000","last_active":"2024-09-17T07:16:06.073395+0000","last_peered":"2024-09-17T07:16:06.073395+0000","last_clean":"2024-09-17T07:16:06.073395+0000","last_became_active":"2024-09-17T07:02:16.021360+0000","last_became_peered":"2024-09-17T07:02:16.021360+0000","last_unstale":"2024-09-17T07:16:06.073395+0000","last_undegraded":"2024-09-17T07:16:06.073395+0000","last_fullsized":"2024-09-17T07:16:06.073395+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'9","last_scrub_stamp":"2024-09-17T07:15:53.512777+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-09-17T07:15:53.512777+0000","last_clean_scrub_stamp":"2024-09-17T07:15:53.512777+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:59.615040+0000","scrub_duration":0.0020298600000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"268'2","reported_seq":311,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.072856+0000","last_change":"2024-09-17T07:14:03.609273+0000","last_active":"2024-09-17T07:16:06.072856+0000","last_peered":"2024-09-17T07:16:06.072856+0000","last_clean":"2024-09-17T07:16:06.072856+0000","last_became_active":"2024-09-17T07:09:24.399565+0000","last_became_peered":"2024-09-17T07:09:24.399565+0000","last_unstale":"2024-09-17T07:16:06.072856+0000","last_undegraded":"2024-09-17T07:16:06.072856+0000","last_fullsized":"2024-09-17T07:16:06.072856+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'2","last_scrub_stamp":"2024-09-17T07:14:03.609131+0000","last_deep_scrub":"268'2","last_deep_scrub_stamp":"2024-09-17T07:14:03.609131+0000","last_clean_scrub_stamp":"2024-09-17T07:14:03.609131+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011205872,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":11,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.074424+0000","last_change":"2024-09-17T07:16:05.055275+0000","last_active":"2024-09-17T07:16:06.074424+0000","last_peered":"2024-09-17T07:16:06.074424+0000","last_clean":"2024-09-17T07:16:06.074424+0000","last_became_active":"2024-09-17T07:16:05.055008+0000","last_became_peered":"2024-09-17T07:16:05.055008+0000","last_unstale":"2024-09-17T07:16:06.074424+0000","last_undegraded":"2024-09-17T07:16:06.074424+0000","last_fullsized":"2024-09-17T07:16:06.074424+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":297,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:20.759555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"242'41","reported_seq":563,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.049057+0000","last_change":"2024-09-17T07:14:56.700945+0000","last_active":"2024-09-17T07:16:06.049057+0000","last_peered":"2024-09-17T07:16:06.049057+0000","last_clean":"2024-09-17T07:16:06.049057+0000","last_became_active":"2024-09-17T07:09:24.403267+0000","last_became_peered":"2024-09-17T07:09:24.403267+0000","last_unstale":"2024-09-17T07:16:06.049057+0000","last_undegraded":"2024-09-17T07:16:06.049057+0000","last_fullsized":"2024-09-17T07:16:06.049057+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'41","last_scrub_stamp":"2024-09-17T07:14:56.700858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:14:56.700858+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002102558,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":312,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.075248+0000","last_change":"2024-09-17T07:14:03.046151+0000","last_active":"2024-09-17T07:16:06.075248+0000","last_peered":"2024-09-17T07:16:06.075248+0000","last_clean":"2024-09-17T07:16:06.075248+0000","last_became_active":"2024-09-17T07:02:12.008802+0000","last_became_peered":"2024-09-17T07:02:12.008802+0000","last_unstale":"2024-09-17T07:16:06.075248+0000","last_undegraded":"2024-09-17T07:16:06.075248+0000","last_fullsized":"2024-09-17T07:16:06.075248+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:03.045709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:14:03.045709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0049245929999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":309,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.073508+0000","last_change":"2024-09-17T07:15:36.523759+0000","last_active":"2024-09-17T07:16:06.073508+0000","last_peered":"2024-09-17T07:16:06.073508+0000","last_clean":"2024-09-17T07:16:06.073508+0000","last_became_active":"2024-09-17T07:02:16.016606+0000","last_became_peered":"2024-09-17T07:02:16.016606+0000","last_unstale":"2024-09-17T07:16:06.073508+0000","last_undegraded":"2024-09-17T07:16:06.073508+0000","last_fullsized":"2024-09-17T07:16:06.073508+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:36.523676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:15:36.523676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:58.160341+0000","scrub_duration":0.0017370549999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":153,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.043014+0000","last_change":"2024-09-17T07:15:55.123577+0000","last_active":"2024-09-17T07:16:06.043014+0000","last_peered":"2024-09-17T07:16:06.043014+0000","last_clean":"2024-09-17T07:16:06.043014+0000","last_became_active":"2024-09-17T07:09:24.398878+0000","last_became_peered":"2024-09-17T07:09:24.398878+0000","last_unstale":"2024-09-17T07:16:06.043014+0000","last_undegraded":"2024-09-17T07:16:06.043014+0000","last_fullsized":"2024-09-17T07:16:06.043014+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:55.123464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:13:53.406661+0000","last_clean_scrub_stamp":"2024-09-17T07:15:55.123464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:03.173520+0000","scrub_duration":0.0014856559999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":12,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-17T07:16:08.305288+0000","last_change":"2024-09-17T07:16:05.057109+0000","last_active":"2024-09-17T07:16:08.305288+0000","last_peered":"2024-09-17T07:16:08.305288+0000","last_clean":"2024-09-17T07:16:08.305288+0000","last_became_active":"2024-09-17T07:16:05.056719+0000","last_became_peered":"2024-09-17T07:16:05.056719+0000","last_unstale":"2024-09-17T07:16:08.305288+0000","last_undegraded":"2024-09-17T07:16:08.305288+0000","last_fullsized":"2024-09-17T07:16:08.305288+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":297,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:09.010134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"242'17","reported_seq":332,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.049038+0000","last_change":"2024-09-17T07:13:47.468495+0000","last_active":"2024-09-17T07:16:06.049038+0000","last_peered":"2024-09-17T07:16:06.049038+0000","last_clean":"2024-09-17T07:16:06.049038+0000","last_became_active":"2024-09-17T07:02:14.019838+0000","last_became_peered":"2024-09-17T07:02:14.019838+0000","last_unstale":"2024-09-17T07:16:06.049038+0000","last_undegraded":"2024-09-17T07:16:06.049038+0000","last_fullsized":"2024-09-17T07:16:06.049038+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'17","last_scrub_stamp":"2024-09-17T07:13:47.468399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:13:47.468399+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0060398329999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"76'5","reported_seq":1137,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:07.986001+0000","last_change":"2024-09-17T07:15:36.901698+0000","last_active":"2024-09-17T07:16:07.986001+0000","last_peered":"2024-09-17T07:16:07.986001+0000","last_clean":"2024-09-17T07:16:07.986001+0000","last_became_active":"2024-09-17T07:02:12.006883+0000","last_became_peered":"2024-09-17T07:02:12.006883+0000","last_unstale":"2024-09-17T07:16:07.986001+0000","last_undegraded":"2024-09-17T07:16:07.986001+0000","last_fullsized":"2024-09-17T07:16:07.986001+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'5","last_scrub_stamp":"2024-09-17T07:15:36.901601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:15:36.901601+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:56.163207+0000","scrub_duration":0.002331086,"objects_trimmed":0,"snaptrim_duration":0.00033739799999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":828,"num_read_kb":823,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":154,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.073130+0000","last_change":"2024-09-17T07:15:56.534623+0000","last_active":"2024-09-17T07:16:06.073130+0000","last_peered":"2024-09-17T07:16:06.073130+0000","last_clean":"2024-09-17T07:16:06.073130+0000","last_became_active":"2024-09-17T07:09:24.392850+0000","last_became_peered":"2024-09-17T07:09:24.392850+0000","last_unstale":"2024-09-17T07:16:06.073130+0000","last_undegraded":"2024-09-17T07:16:06.073130+0000","last_fullsized":"2024-09-17T07:16:06.073130+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:56.534547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:23.833170+0000","last_clean_scrub_stamp":"2024-09-17T07:15:56.534547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:15.585405+0000","scrub_duration":0.0011018950000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":322,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.047500+0000","last_change":"2024-09-17T07:15:40.663837+0000","last_active":"2024-09-17T07:16:06.047500+0000","last_peered":"2024-09-17T07:16:06.047500+0000","last_clean":"2024-09-17T07:16:06.047500+0000","last_became_active":"2024-09-17T07:02:16.014492+0000","last_became_peered":"2024-09-17T07:02:16.014492+0000","last_unstale":"2024-09-17T07:16:06.047500+0000","last_undegraded":"2024-09-17T07:16:06.047500+0000","last_fullsized":"2024-09-17T07:16:06.047500+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:40.663759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:15:40.663759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:44.968568+0000","scrub_duration":0.0013265250000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":315,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.075134+0000","last_change":"2024-09-17T07:15:02.062776+0000","last_active":"2024-09-17T07:16:06.075134+0000","last_peered":"2024-09-17T07:16:06.075134+0000","last_clean":"2024-09-17T07:16:06.075134+0000","last_became_active":"2024-09-17T07:09:24.402704+0000","last_became_peered":"2024-09-17T07:09:24.402704+0000","last_unstale":"2024-09-17T07:16:06.075134+0000","last_undegraded":"2024-09-17T07:16:06.075134+0000","last_fullsized":"2024-09-17T07:16:06.075134+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:02.062618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:02.062618+0000","last_clean_scrub_stamp":"2024-09-17T07:15:02.062618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:23.850877+0000","scrub_duration":0.0042833460000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":12,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.073088+0000","last_change":"2024-09-17T07:16:05.054358+0000","last_active":"2024-09-17T07:16:06.073088+0000","last_peered":"2024-09-17T07:16:06.073088+0000","last_clean":"2024-09-17T07:16:06.073088+0000","last_became_active":"2024-09-17T07:16:05.053996+0000","last_became_peered":"2024-09-17T07:16:05.053996+0000","last_unstale":"2024-09-17T07:16:06.073088+0000","last_undegraded":"2024-09-17T07:16:06.073088+0000","last_fullsized":"2024-09-17T07:16:06.073088+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":297,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:31.098761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"268'30","reported_seq":365,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.073716+0000","last_change":"2024-09-17T07:14:00.599815+0000","last_active":"2024-09-17T07:16:06.073716+0000","last_peered":"2024-09-17T07:16:06.073716+0000","last_clean":"2024-09-17T07:16:06.073716+0000","last_became_active":"2024-09-17T07:03:48.760405+0000","last_became_peered":"2024-09-17T07:03:48.760405+0000","last_unstale":"2024-09-17T07:16:06.073716+0000","last_undegraded":"2024-09-17T07:16:06.073716+0000","last_fullsized":"2024-09-17T07:16:06.073716+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'30","last_scrub_stamp":"2024-09-17T07:14:00.599656+0000","last_deep_scrub":"268'30","last_deep_scrub_stamp":"2024-09-17T07:14:00.599656+0000","last_clean_scrub_stamp":"2024-09-17T07:14:00.599656+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.030045246000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"70'1","reported_seq":321,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.073291+0000","last_change":"2024-09-17T07:15:01.575863+0000","last_active":"2024-09-17T07:16:06.073291+0000","last_peered":"2024-09-17T07:16:06.073291+0000","last_clean":"2024-09-17T07:16:06.073291+0000","last_became_active":"2024-09-17T07:09:24.402499+0000","last_became_peered":"2024-09-17T07:09:24.402499+0000","last_unstale":"2024-09-17T07:16:06.073291+0000","last_undegraded":"2024-09-17T07:16:06.073291+0000","last_fullsized":"2024-09-17T07:16:06.073291+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-17T07:15:01.575747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:15:01.575747+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037109349999999998,"objects_trimmed":0,"snaptrim_duration":0.0032298629999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":301,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.073822+0000","last_change":"2024-09-17T07:15:55.321315+0000","last_active":"2024-09-17T07:16:06.073822+0000","last_peered":"2024-09-17T07:16:06.073822+0000","last_clean":"2024-09-17T07:16:06.073822+0000","last_became_active":"2024-09-17T07:02:12.014623+0000","last_became_peered":"2024-09-17T07:02:12.014623+0000","last_unstale":"2024-09-17T07:16:06.073822+0000","last_undegraded":"2024-09-17T07:16:06.073822+0000","last_fullsized":"2024-09-17T07:16:06.073822+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:55.321202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:55.321202+0000","last_clean_scrub_stamp":"2024-09-17T07:15:55.321202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:22.161490+0000","scrub_duration":0.0014716950000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":308,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.075296+0000","last_change":"2024-09-17T07:15:45.447122+0000","last_active":"2024-09-17T07:16:06.075296+0000","last_peered":"2024-09-17T07:16:06.075296+0000","last_clean":"2024-09-17T07:16:06.075296+0000","last_became_active":"2024-09-17T07:02:16.021561+0000","last_became_peered":"2024-09-17T07:02:16.021561+0000","last_unstale":"2024-09-17T07:16:06.075296+0000","last_undegraded":"2024-09-17T07:16:06.075296+0000","last_fullsized":"2024-09-17T07:16:06.075296+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:45.447029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:15:45.447029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:59.805276+0000","scrub_duration":0.001990463,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":304,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.048805+0000","last_change":"2024-09-17T07:15:51.877450+0000","last_active":"2024-09-17T07:16:06.048805+0000","last_peered":"2024-09-17T07:16:06.048805+0000","last_clean":"2024-09-17T07:16:06.048805+0000","last_became_active":"2024-09-17T07:09:24.396154+0000","last_became_peered":"2024-09-17T07:09:24.396154+0000","last_unstale":"2024-09-17T07:16:06.048805+0000","last_undegraded":"2024-09-17T07:16:06.048805+0000","last_fullsized":"2024-09-17T07:16:06.048805+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:51.877367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:15:51.877367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:03.493027+0000","scrub_duration":0.001552423,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":11,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.073794+0000","last_change":"2024-09-17T07:16:05.051669+0000","last_active":"2024-09-17T07:16:06.073794+0000","last_peered":"2024-09-17T07:16:06.073794+0000","last_clean":"2024-09-17T07:16:06.073794+0000","last_became_active":"2024-09-17T07:16:05.051277+0000","last_became_peered":"2024-09-17T07:16:05.051277+0000","last_unstale":"2024-09-17T07:16:06.073794+0000","last_undegraded":"2024-09-17T07:16:06.073794+0000","last_fullsized":"2024-09-17T07:16:06.073794+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":297,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:25.863440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"242'26","reported_seq":351,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-17T07:16:05.049568+0000","last_change":"2024-09-17T07:14:43.500757+0000","last_active":"2024-09-17T07:16:05.049568+0000","last_peered":"2024-09-17T07:16:05.049568+0000","last_clean":"2024-09-17T07:16:05.049568+0000","last_became_active":"2024-09-17T07:02:14.028857+0000","last_became_peered":"2024-09-17T07:02:14.028857+0000","last_unstale":"2024-09-17T07:16:05.049568+0000","last_undegraded":"2024-09-17T07:16:05.049568+0000","last_fullsized":"2024-09-17T07:16:05.049568+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'26","last_scrub_stamp":"2024-09-17T07:14:43.500621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:14:43.500621+0000","objects_scrubbed":7,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:06.894697+0000","scrub_duration":0.032308936000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"268'3","reported_seq":172,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.075075+0000","last_change":"2024-09-17T07:15:43.516879+0000","last_active":"2024-09-17T07:16:06.075075+0000","last_peered":"2024-09-17T07:16:06.075075+0000","last_clean":"2024-09-17T07:16:06.075075+0000","last_became_active":"2024-09-17T07:09:24.404532+0000","last_became_peered":"2024-09-17T07:09:24.404532+0000","last_unstale":"2024-09-17T07:16:06.075075+0000","last_undegraded":"2024-09-17T07:16:06.075075+0000","last_fullsized":"2024-09-17T07:16:06.075075+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'3","last_scrub_stamp":"2024-09-17T07:15:43.516714+0000","last_deep_scrub":"268'3","last_deep_scrub_stamp":"2024-09-17T07:13:54.949345+0000","last_clean_scrub_stamp":"2024-09-17T07:15:43.516714+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:05.399081+0000","scrub_duration":0.002071711,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":323,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.074628+0000","last_change":"2024-09-17T07:15:55.432933+0000","last_active":"2024-09-17T07:16:06.074628+0000","last_peered":"2024-09-17T07:16:06.074628+0000","last_clean":"2024-09-17T07:16:06.074628+0000","last_became_active":"2024-09-17T07:02:12.004606+0000","last_became_peered":"2024-09-17T07:02:12.004606+0000","last_unstale":"2024-09-17T07:16:06.074628+0000","last_undegraded":"2024-09-17T07:16:06.074628+0000","last_fullsized":"2024-09-17T07:16:06.074628+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:55.432818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:15:55.432818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:23.510512+0000","scrub_duration":0.001411254,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"212'130","reported_seq":563,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-17T07:16:08.498876+0000","last_change":"2024-09-17T07:14:57.052759+0000","last_active":"2024-09-17T07:16:08.498876+0000","last_peered":"2024-09-17T07:16:08.498876+0000","last_clean":"2024-09-17T07:16:08.498876+0000","last_became_active":"2024-09-17T07:02:03.020675+0000","last_became_peered":"2024-09-17T07:02:03.020675+0000","last_unstale":"2024-09-17T07:16:08.498876+0000","last_undegraded":"2024-09-17T07:16:08.498876+0000","last_fullsized":"2024-09-17T07:16:08.498876+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:14:57.052505+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:14:57.052505+0000","last_clean_scrub_stamp":"2024-09-17T07:14:57.052505+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.022122466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":308,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.074349+0000","last_change":"2024-09-17T07:15:56.366761+0000","last_active":"2024-09-17T07:16:06.074349+0000","last_peered":"2024-09-17T07:16:06.074349+0000","last_clean":"2024-09-17T07:16:06.074349+0000","last_became_active":"2024-09-17T07:02:16.016031+0000","last_became_peered":"2024-09-17T07:02:16.016031+0000","last_unstale":"2024-09-17T07:16:06.074349+0000","last_undegraded":"2024-09-17T07:16:06.074349+0000","last_fullsized":"2024-09-17T07:16:06.074349+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:56.366669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:15:56.366669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:20.727553+0000","scrub_duration":0.0015953269999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":304,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.075392+0000","last_change":"2024-09-17T07:14:48.021809+0000","last_active":"2024-09-17T07:16:06.075392+0000","last_peered":"2024-09-17T07:16:06.075392+0000","last_clean":"2024-09-17T07:16:06.075392+0000","last_became_active":"2024-09-17T07:09:24.400041+0000","last_became_peered":"2024-09-17T07:09:24.400041+0000","last_unstale":"2024-09-17T07:16:06.075392+0000","last_undegraded":"2024-09-17T07:16:06.075392+0000","last_fullsized":"2024-09-17T07:16:06.075392+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:48.021592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:58.670571+0000","last_clean_scrub_stamp":"2024-09-17T07:14:48.021592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0064844619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":12,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.073306+0000","last_change":"2024-09-17T07:16:05.050800+0000","last_active":"2024-09-17T07:16:06.073306+0000","last_peered":"2024-09-17T07:16:06.073306+0000","last_clean":"2024-09-17T07:16:06.073306+0000","last_became_active":"2024-09-17T07:16:05.050503+0000","last_became_peered":"2024-09-17T07:16:05.050503+0000","last_unstale":"2024-09-17T07:16:06.073306+0000","last_undegraded":"2024-09-17T07:16:06.073306+0000","last_fullsized":"2024-09-17T07:16:06.073306+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":297,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:30.553238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"242'35","reported_seq":377,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-17T07:16:05.048042+0000","last_change":"2024-09-17T07:15:43.665940+0000","last_active":"2024-09-17T07:16:05.048042+0000","last_peered":"2024-09-17T07:16:05.048042+0000","last_clean":"2024-09-17T07:16:05.048042+0000","last_became_active":"2024-09-17T07:02:14.027313+0000","last_became_peered":"2024-09-17T07:02:14.027313+0000","last_unstale":"2024-09-17T07:16:05.048042+0000","last_undegraded":"2024-09-17T07:16:05.048042+0000","last_fullsized":"2024-09-17T07:16:05.048042+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'35","last_scrub_stamp":"2024-09-17T07:15:43.665829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:15:43.665829+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:00.032507+0000","scrub_duration":0.0048325160000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":348,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.047320+0000","last_change":"2024-09-17T07:15:48.685572+0000","last_active":"2024-09-17T07:16:06.047320+0000","last_peered":"2024-09-17T07:16:06.047320+0000","last_clean":"2024-09-17T07:16:06.047320+0000","last_became_active":"2024-09-17T07:02:12.012037+0000","last_became_peered":"2024-09-17T07:02:12.012037+0000","last_unstale":"2024-09-17T07:16:06.047320+0000","last_undegraded":"2024-09-17T07:16:06.047320+0000","last_fullsized":"2024-09-17T07:16:06.047320+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:48.685498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:06:39.322535+0000","last_clean_scrub_stamp":"2024-09-17T07:15:48.685498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:17.814720+0000","scrub_duration":0.001437528,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":332,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.047047+0000","last_change":"2024-09-17T07:15:36.567808+0000","last_active":"2024-09-17T07:16:06.047047+0000","last_peered":"2024-09-17T07:16:06.047047+0000","last_clean":"2024-09-17T07:16:06.047047+0000","last_became_active":"2024-09-17T07:02:16.014631+0000","last_became_peered":"2024-09-17T07:02:16.014631+0000","last_unstale":"2024-09-17T07:16:06.047047+0000","last_undegraded":"2024-09-17T07:16:06.047047+0000","last_fullsized":"2024-09-17T07:16:06.047047+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:36.567726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:15:36.567726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:44.026430+0000","scrub_duration":0.0014186050000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":304,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.073902+0000","last_change":"2024-09-17T07:15:45.275553+0000","last_active":"2024-09-17T07:16:06.073902+0000","last_peered":"2024-09-17T07:16:06.073902+0000","last_clean":"2024-09-17T07:16:06.073902+0000","last_became_active":"2024-09-17T07:09:24.403904+0000","last_became_peered":"2024-09-17T07:09:24.403904+0000","last_unstale":"2024-09-17T07:16:06.073902+0000","last_undegraded":"2024-09-17T07:16:06.073902+0000","last_fullsized":"2024-09-17T07:16:06.073902+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:45.275473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:45.275473+0000","last_clean_scrub_stamp":"2024-09-17T07:15:45.275473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:11.590023+0000","scrub_duration":0.0014472759999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":12,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.073369+0000","last_change":"2024-09-17T07:16:05.056347+0000","last_active":"2024-09-17T07:16:06.073369+0000","last_peered":"2024-09-17T07:16:06.073369+0000","last_clean":"2024-09-17T07:16:06.073369+0000","last_became_active":"2024-09-17T07:16:05.056100+0000","last_became_peered":"2024-09-17T07:16:05.056100+0000","last_unstale":"2024-09-17T07:16:06.073369+0000","last_undegraded":"2024-09-17T07:16:06.073369+0000","last_fullsized":"2024-09-17T07:16:06.073369+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":297,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:30.058146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"268'19","reported_seq":329,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.073478+0000","last_change":"2024-09-17T07:14:06.634500+0000","last_active":"2024-09-17T07:16:06.073478+0000","last_peered":"2024-09-17T07:16:06.073478+0000","last_clean":"2024-09-17T07:16:06.073478+0000","last_became_active":"2024-09-17T07:02:14.014939+0000","last_became_peered":"2024-09-17T07:02:14.014939+0000","last_unstale":"2024-09-17T07:16:06.073478+0000","last_undegraded":"2024-09-17T07:16:06.073478+0000","last_fullsized":"2024-09-17T07:16:06.073478+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'19","last_scrub_stamp":"2024-09-17T07:14:06.634334+0000","last_deep_scrub":"268'19","last_deep_scrub_stamp":"2024-09-17T07:14:06.634334+0000","last_clean_scrub_stamp":"2024-09-17T07:14:06.634334+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0088943289999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":321,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.049111+0000","last_change":"2024-09-17T07:15:58.886377+0000","last_active":"2024-09-17T07:16:06.049111+0000","last_peered":"2024-09-17T07:16:06.049111+0000","last_clean":"2024-09-17T07:16:06.049111+0000","last_became_active":"2024-09-17T07:09:24.391207+0000","last_became_peered":"2024-09-17T07:09:24.391207+0000","last_unstale":"2024-09-17T07:16:06.049111+0000","last_undegraded":"2024-09-17T07:16:06.049111+0000","last_fullsized":"2024-09-17T07:16:06.049111+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:58.886294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:13:53.593078+0000","last_clean_scrub_stamp":"2024-09-17T07:15:58.886294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:13.318718+0000","scrub_duration":0.0012514449999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":273,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.042572+0000","last_change":"2024-09-17T07:14:45.625006+0000","last_active":"2024-09-17T07:16:06.042572+0000","last_peered":"2024-09-17T07:16:06.042572+0000","last_clean":"2024-09-17T07:16:06.042572+0000","last_became_active":"2024-09-17T07:09:24.396642+0000","last_became_peered":"2024-09-17T07:09:24.396642+0000","last_unstale":"2024-09-17T07:16:06.042572+0000","last_undegraded":"2024-09-17T07:16:06.042572+0000","last_fullsized":"2024-09-17T07:16:06.042572+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:45.624886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:14:45.624886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0061816429999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":296,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.073266+0000","last_change":"2024-09-17T07:15:00.557150+0000","last_active":"2024-09-17T07:16:06.073266+0000","last_peered":"2024-09-17T07:16:06.073266+0000","last_clean":"2024-09-17T07:16:06.073266+0000","last_became_active":"2024-09-17T07:02:18.026994+0000","last_became_peered":"2024-09-17T07:02:18.026994+0000","last_unstale":"2024-09-17T07:16:06.073266+0000","last_undegraded":"2024-09-17T07:16:06.073266+0000","last_fullsized":"2024-09-17T07:16:06.073266+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:00.557042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:58.099849+0000","last_clean_scrub_stamp":"2024-09-17T07:15:00.557042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:11.269725+0000","scrub_duration":0.0065216600000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":12,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-17T07:16:08.305987+0000","last_change":"2024-09-17T07:16:05.058034+0000","last_active":"2024-09-17T07:16:08.305987+0000","last_peered":"2024-09-17T07:16:08.305987+0000","last_clean":"2024-09-17T07:16:08.305987+0000","last_became_active":"2024-09-17T07:16:05.057758+0000","last_became_peered":"2024-09-17T07:16:05.057758+0000","last_unstale":"2024-09-17T07:16:08.305987+0000","last_undegraded":"2024-09-17T07:16:08.305987+0000","last_fullsized":"2024-09-17T07:16:08.305987+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":297,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:13.541473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"242'34","reported_seq":360,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.073267+0000","last_change":"2024-09-17T07:14:50.103842+0000","last_active":"2024-09-17T07:16:06.073267+0000","last_peered":"2024-09-17T07:16:06.073267+0000","last_clean":"2024-09-17T07:16:06.073267+0000","last_became_active":"2024-09-17T07:02:14.028392+0000","last_became_peered":"2024-09-17T07:02:14.028392+0000","last_unstale":"2024-09-17T07:16:06.073267+0000","last_undegraded":"2024-09-17T07:16:06.073267+0000","last_fullsized":"2024-09-17T07:16:06.073267+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'34","last_scrub_stamp":"2024-09-17T07:14:50.103539+0000","last_deep_scrub":"242'34","last_deep_scrub_stamp":"2024-09-17T07:14:50.103539+0000","last_clean_scrub_stamp":"2024-09-17T07:14:50.103539+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.016600602999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'2","reported_seq":183,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.073975+0000","last_change":"2024-09-17T07:15:48.378000+0000","last_active":"2024-09-17T07:16:06.073975+0000","last_peered":"2024-09-17T07:16:06.073975+0000","last_clean":"2024-09-17T07:16:06.073975+0000","last_became_active":"2024-09-17T07:09:24.402909+0000","last_became_peered":"2024-09-17T07:09:24.402909+0000","last_unstale":"2024-09-17T07:16:06.073975+0000","last_undegraded":"2024-09-17T07:16:06.073975+0000","last_fullsized":"2024-09-17T07:16:06.073975+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-09-17T07:15:48.377886+0000","last_deep_scrub":"89'2","last_deep_scrub_stamp":"2024-09-17T07:12:20.301263+0000","last_clean_scrub_stamp":"2024-09-17T07:15:48.377886+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:50.626969+0000","scrub_duration":0.001635079,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":420,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":420,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"225'9","reported_seq":476,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.074146+0000","last_change":"2024-09-17T07:14:42.018236+0000","last_active":"2024-09-17T07:16:06.074146+0000","last_peered":"2024-09-17T07:16:06.074146+0000","last_clean":"2024-09-17T07:16:06.074146+0000","last_became_active":"2024-09-17T07:02:16.015418+0000","last_became_peered":"2024-09-17T07:02:16.015418+0000","last_unstale":"2024-09-17T07:16:06.074146+0000","last_undegraded":"2024-09-17T07:16:06.074146+0000","last_fullsized":"2024-09-17T07:16:06.074146+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"225'9","last_scrub_stamp":"2024-09-17T07:14:42.017656+0000","last_deep_scrub":"225'9","last_deep_scrub_stamp":"2024-09-17T07:11:25.292565+0000","last_clean_scrub_stamp":"2024-09-17T07:14:42.017656+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0089186300000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":306,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.074824+0000","last_change":"2024-09-17T07:14:49.061957+0000","last_active":"2024-09-17T07:16:06.074824+0000","last_peered":"2024-09-17T07:16:06.074824+0000","last_clean":"2024-09-17T07:16:06.074824+0000","last_became_active":"2024-09-17T07:02:18.363253+0000","last_became_peered":"2024-09-17T07:02:18.363253+0000","last_unstale":"2024-09-17T07:16:06.074824+0000","last_undegraded":"2024-09-17T07:16:06.074824+0000","last_fullsized":"2024-09-17T07:16:06.074824+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:49.061804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:14:49.061804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0048663639999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":11,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.075770+0000","last_change":"2024-09-17T07:16:05.054694+0000","last_active":"2024-09-17T07:16:06.075770+0000","last_peered":"2024-09-17T07:16:06.075770+0000","last_clean":"2024-09-17T07:16:06.075770+0000","last_became_active":"2024-09-17T07:16:05.054306+0000","last_became_peered":"2024-09-17T07:16:05.054306+0000","last_unstale":"2024-09-17T07:16:06.075770+0000","last_undegraded":"2024-09-17T07:16:06.075770+0000","last_fullsized":"2024-09-17T07:16:06.075770+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":297,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:14.628931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"242'23","reported_seq":391,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.042404+0000","last_change":"2024-09-17T07:15:58.176415+0000","last_active":"2024-09-17T07:16:06.042404+0000","last_peered":"2024-09-17T07:16:06.042404+0000","last_clean":"2024-09-17T07:16:06.042404+0000","last_became_active":"2024-09-17T07:10:49.723292+0000","last_became_peered":"2024-09-17T07:10:49.723292+0000","last_unstale":"2024-09-17T07:16:06.042404+0000","last_undegraded":"2024-09-17T07:16:06.042404+0000","last_fullsized":"2024-09-17T07:16:06.042404+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'23","last_scrub_stamp":"2024-09-17T07:15:58.176287+0000","last_deep_scrub":"150'17","last_deep_scrub_stamp":"2024-09-17T07:06:38.775788+0000","last_clean_scrub_stamp":"2024-09-17T07:15:58.176287+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:09.296262+0000","scrub_duration":0.0024399420000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"68'2","reported_seq":326,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.043325+0000","last_change":"2024-09-17T07:14:56.533910+0000","last_active":"2024-09-17T07:16:06.043325+0000","last_peered":"2024-09-17T07:16:06.043325+0000","last_clean":"2024-09-17T07:16:06.043325+0000","last_became_active":"2024-09-17T07:10:49.727577+0000","last_became_peered":"2024-09-17T07:10:49.727577+0000","last_unstale":"2024-09-17T07:16:06.043325+0000","last_undegraded":"2024-09-17T07:16:06.043325+0000","last_fullsized":"2024-09-17T07:16:06.043325+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-09-17T07:14:56.533798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:14:56.533798+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:10.946077+0000","scrub_duration":0.003207115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":273,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.072879+0000","last_change":"2024-09-17T07:15:51.587007+0000","last_active":"2024-09-17T07:16:06.072879+0000","last_peered":"2024-09-17T07:16:06.072879+0000","last_clean":"2024-09-17T07:16:06.072879+0000","last_became_active":"2024-09-17T07:04:15.111069+0000","last_became_peered":"2024-09-17T07:04:15.111069+0000","last_unstale":"2024-09-17T07:16:06.072879+0000","last_undegraded":"2024-09-17T07:16:06.072879+0000","last_fullsized":"2024-09-17T07:16:06.072879+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:51.586929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:11:17.479036+0000","last_clean_scrub_stamp":"2024-09-17T07:15:51.586929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:55.589024+0000","scrub_duration":0.001274435,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":141,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.074454+0000","last_change":"2024-09-17T07:13:13.938089+0000","last_active":"2024-09-17T07:16:06.074454+0000","last_peered":"2024-09-17T07:16:06.074454+0000","last_clean":"2024-09-17T07:16:06.074454+0000","last_became_active":"2024-09-17T07:09:24.397669+0000","last_became_peered":"2024-09-17T07:09:24.397669+0000","last_unstale":"2024-09-17T07:16:06.074454+0000","last_undegraded":"2024-09-17T07:16:06.074454+0000","last_fullsized":"2024-09-17T07:16:06.074454+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:13.937979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:13:13.937979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.018487173999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"242'15","reported_seq":354,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.073684+0000","last_change":"2024-09-17T07:15:58.508698+0000","last_active":"2024-09-17T07:16:06.073684+0000","last_peered":"2024-09-17T07:16:06.073684+0000","last_clean":"2024-09-17T07:16:06.073684+0000","last_became_active":"2024-09-17T07:09:24.408875+0000","last_became_peered":"2024-09-17T07:09:24.408875+0000","last_unstale":"2024-09-17T07:16:06.073684+0000","last_undegraded":"2024-09-17T07:16:06.073684+0000","last_fullsized":"2024-09-17T07:16:06.073684+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'15","last_scrub_stamp":"2024-09-17T07:15:58.508607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:15:58.508607+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:28.355513+0000","scrub_duration":0.0022156900000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":336,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.046669+0000","last_change":"2024-09-17T07:15:39.634792+0000","last_active":"2024-09-17T07:16:06.046669+0000","last_peered":"2024-09-17T07:16:06.046669+0000","last_clean":"2024-09-17T07:16:06.046669+0000","last_became_active":"2024-09-17T07:02:12.011327+0000","last_became_peered":"2024-09-17T07:02:12.011327+0000","last_unstale":"2024-09-17T07:16:06.046669+0000","last_undegraded":"2024-09-17T07:16:06.046669+0000","last_fullsized":"2024-09-17T07:16:06.046669+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:39.634698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:17.976267+0000","last_clean_scrub_stamp":"2024-09-17T07:15:39.634698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:44.039723+0000","scrub_duration":0.00117202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":140,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.048893+0000","last_change":"2024-09-17T07:15:40.845904+0000","last_active":"2024-09-17T07:16:06.048893+0000","last_peered":"2024-09-17T07:16:06.048893+0000","last_clean":"2024-09-17T07:16:06.048893+0000","last_became_active":"2024-09-17T07:09:24.407806+0000","last_became_peered":"2024-09-17T07:09:24.407806+0000","last_unstale":"2024-09-17T07:16:06.048893+0000","last_undegraded":"2024-09-17T07:16:06.048893+0000","last_fullsized":"2024-09-17T07:16:06.048893+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:40.845816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:18.367486+0000","last_clean_scrub_stamp":"2024-09-17T07:15:40.845816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:53.615891+0000","scrub_duration":0.0013977639999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":341,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.046976+0000","last_change":"2024-09-17T07:14:52.084362+0000","last_active":"2024-09-17T07:16:06.046976+0000","last_peered":"2024-09-17T07:16:06.046976+0000","last_clean":"2024-09-17T07:16:06.046976+0000","last_became_active":"2024-09-17T07:02:18.017825+0000","last_became_peered":"2024-09-17T07:02:18.017825+0000","last_unstale":"2024-09-17T07:16:06.046976+0000","last_undegraded":"2024-09-17T07:16:06.046976+0000","last_fullsized":"2024-09-17T07:16:06.046976+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:52.084273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:09:23.378805+0000","last_clean_scrub_stamp":"2024-09-17T07:14:52.084273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:14.268291+0000","scrub_duration":0.0039317479999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":323,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-17T07:16:05.049875+0000","last_change":"2024-09-17T07:15:46.645165+0000","last_active":"2024-09-17T07:16:05.049875+0000","last_peered":"2024-09-17T07:16:05.049875+0000","last_clean":"2024-09-17T07:16:05.049875+0000","last_became_active":"2024-09-17T07:02:12.008065+0000","last_became_peered":"2024-09-17T07:02:12.008065+0000","last_unstale":"2024-09-17T07:16:05.049875+0000","last_undegraded":"2024-09-17T07:16:05.049875+0000","last_fullsized":"2024-09-17T07:16:05.049875+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:46.645080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:46.645080+0000","last_clean_scrub_stamp":"2024-09-17T07:15:46.645080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:59.909153+0000","scrub_duration":0.0015661049999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"242'12","reported_seq":178,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.075702+0000","last_change":"2024-09-17T07:15:44.249955+0000","last_active":"2024-09-17T07:16:06.075702+0000","last_peered":"2024-09-17T07:16:06.075702+0000","last_clean":"2024-09-17T07:16:06.075702+0000","last_became_active":"2024-09-17T07:09:24.405385+0000","last_became_peered":"2024-09-17T07:09:24.405385+0000","last_unstale":"2024-09-17T07:16:06.075702+0000","last_undegraded":"2024-09-17T07:16:06.075702+0000","last_fullsized":"2024-09-17T07:16:06.075702+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'12","last_scrub_stamp":"2024-09-17T07:15:44.249858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:15:44.249858+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:47.251325+0000","scrub_duration":0.0022006600000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":316,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.074668+0000","last_change":"2024-09-17T07:14:55.191458+0000","last_active":"2024-09-17T07:16:06.074668+0000","last_peered":"2024-09-17T07:16:06.074668+0000","last_clean":"2024-09-17T07:16:06.074668+0000","last_became_active":"2024-09-17T07:09:24.398922+0000","last_became_peered":"2024-09-17T07:09:24.398922+0000","last_unstale":"2024-09-17T07:16:06.074668+0000","last_undegraded":"2024-09-17T07:16:06.074668+0000","last_fullsized":"2024-09-17T07:16:06.074668+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:55.191252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:14:55.191252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0090150360000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":294,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.075061+0000","last_change":"2024-09-17T07:14:52.058961+0000","last_active":"2024-09-17T07:16:06.075061+0000","last_peered":"2024-09-17T07:16:06.075061+0000","last_clean":"2024-09-17T07:16:06.075061+0000","last_became_active":"2024-09-17T07:02:18.365419+0000","last_became_peered":"2024-09-17T07:02:18.365419+0000","last_unstale":"2024-09-17T07:16:06.075061+0000","last_undegraded":"2024-09-17T07:16:06.075061+0000","last_fullsized":"2024-09-17T07:16:06.075061+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:52.058847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:14:52.058847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:19.721921+0000","scrub_duration":0.0054926549999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"242'20","reported_seq":195,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.048945+0000","last_change":"2024-09-17T07:14:05.528864+0000","last_active":"2024-09-17T07:16:06.048945+0000","last_peered":"2024-09-17T07:16:06.048945+0000","last_clean":"2024-09-17T07:16:06.048945+0000","last_became_active":"2024-09-17T07:09:24.405944+0000","last_became_peered":"2024-09-17T07:09:24.405944+0000","last_unstale":"2024-09-17T07:16:06.048945+0000","last_undegraded":"2024-09-17T07:16:06.048945+0000","last_fullsized":"2024-09-17T07:16:06.048945+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'20","last_scrub_stamp":"2024-09-17T07:14:05.528769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:14:05.528769+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.015483452,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":331,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.075162+0000","last_change":"2024-09-17T07:15:48.405105+0000","last_active":"2024-09-17T07:16:06.075162+0000","last_peered":"2024-09-17T07:16:06.075162+0000","last_clean":"2024-09-17T07:16:06.075162+0000","last_became_active":"2024-09-17T07:09:24.402355+0000","last_became_peered":"2024-09-17T07:09:24.402355+0000","last_unstale":"2024-09-17T07:16:06.075162+0000","last_undegraded":"2024-09-17T07:16:06.075162+0000","last_fullsized":"2024-09-17T07:16:06.075162+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:48.405027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:15:48.405027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:06.134372+0000","scrub_duration":0.0014214710000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"163'9","reported_seq":483,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.075004+0000","last_change":"2024-09-17T07:14:53.140257+0000","last_active":"2024-09-17T07:16:06.075004+0000","last_peered":"2024-09-17T07:16:06.075004+0000","last_clean":"2024-09-17T07:16:06.075004+0000","last_became_active":"2024-09-17T07:02:16.022583+0000","last_became_peered":"2024-09-17T07:02:16.022583+0000","last_unstale":"2024-09-17T07:16:06.075004+0000","last_undegraded":"2024-09-17T07:16:06.075004+0000","last_fullsized":"2024-09-17T07:16:06.075004+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'9","last_scrub_stamp":"2024-09-17T07:14:53.140133+0000","last_deep_scrub":"163'9","last_deep_scrub_stamp":"2024-09-17T07:07:34.933808+0000","last_clean_scrub_stamp":"2024-09-17T07:14:53.140133+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:15.885880+0000","scrub_duration":0.0088819989999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":304,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.073484+0000","last_change":"2024-09-17T07:14:04.624250+0000","last_active":"2024-09-17T07:16:06.073484+0000","last_peered":"2024-09-17T07:16:06.073484+0000","last_clean":"2024-09-17T07:16:06.073484+0000","last_became_active":"2024-09-17T07:09:24.409051+0000","last_became_peered":"2024-09-17T07:09:24.409051+0000","last_unstale":"2024-09-17T07:16:06.073484+0000","last_undegraded":"2024-09-17T07:16:06.073484+0000","last_fullsized":"2024-09-17T07:16:06.073484+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:04.624118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:26.619893+0000","last_clean_scrub_stamp":"2024-09-17T07:14:04.624118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0028490479999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"242'15","reported_seq":353,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.075084+0000","last_change":"2024-09-17T07:14:07.138526+0000","last_active":"2024-09-17T07:16:06.075084+0000","last_peered":"2024-09-17T07:16:06.075084+0000","last_clean":"2024-09-17T07:16:06.075084+0000","last_became_active":"2024-09-17T07:09:24.406333+0000","last_became_peered":"2024-09-17T07:09:24.406333+0000","last_unstale":"2024-09-17T07:16:06.075084+0000","last_undegraded":"2024-09-17T07:16:06.075084+0000","last_fullsized":"2024-09-17T07:16:06.075084+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'15","last_scrub_stamp":"2024-09-17T07:14:07.138401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:14:07.138401+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.038794863999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":310,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.073683+0000","last_change":"2024-09-17T07:14:41.618598+0000","last_active":"2024-09-17T07:16:06.073683+0000","last_peered":"2024-09-17T07:16:06.073683+0000","last_clean":"2024-09-17T07:16:06.073683+0000","last_became_active":"2024-09-17T07:09:24.402704+0000","last_became_peered":"2024-09-17T07:09:24.402704+0000","last_unstale":"2024-09-17T07:16:06.073683+0000","last_undegraded":"2024-09-17T07:16:06.073683+0000","last_fullsized":"2024-09-17T07:16:06.073683+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:41.618524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:14:41.618524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0025348459999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"196'9","reported_seq":246,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.074504+0000","last_change":"2024-09-17T07:13:10.965977+0000","last_active":"2024-09-17T07:16:06.074504+0000","last_peered":"2024-09-17T07:16:06.074504+0000","last_clean":"2024-09-17T07:16:06.074504+0000","last_became_active":"2024-09-17T07:09:24.406267+0000","last_became_peered":"2024-09-17T07:09:24.406267+0000","last_unstale":"2024-09-17T07:16:06.074504+0000","last_undegraded":"2024-09-17T07:16:06.074504+0000","last_fullsized":"2024-09-17T07:16:06.074504+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'9","last_scrub_stamp":"2024-09-17T07:13:10.965855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:13:10.965855+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0088433720000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":296,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.043217+0000","last_change":"2024-09-17T07:14:03.699173+0000","last_active":"2024-09-17T07:16:06.043217+0000","last_peered":"2024-09-17T07:16:06.043217+0000","last_clean":"2024-09-17T07:16:06.043217+0000","last_became_active":"2024-09-17T07:02:18.018716+0000","last_became_peered":"2024-09-17T07:02:18.018716+0000","last_unstale":"2024-09-17T07:16:06.043217+0000","last_undegraded":"2024-09-17T07:16:06.043217+0000","last_fullsized":"2024-09-17T07:16:06.043217+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:03.699048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:14:03.699048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0099202749999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"242'15","reported_seq":323,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.048446+0000","last_change":"2024-09-17T07:13:48.500828+0000","last_active":"2024-09-17T07:16:06.048446+0000","last_peered":"2024-09-17T07:16:06.048446+0000","last_clean":"2024-09-17T07:16:06.048446+0000","last_became_active":"2024-09-17T07:02:14.010289+0000","last_became_peered":"2024-09-17T07:02:14.010289+0000","last_unstale":"2024-09-17T07:16:06.048446+0000","last_undegraded":"2024-09-17T07:16:06.048446+0000","last_fullsized":"2024-09-17T07:16:06.048446+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'15","last_scrub_stamp":"2024-09-17T07:13:48.500257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:13:48.500257+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0054642730000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":331,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-17T07:16:05.048270+0000","last_change":"2024-09-17T07:15:33.622783+0000","last_active":"2024-09-17T07:16:05.048270+0000","last_peered":"2024-09-17T07:16:05.048270+0000","last_clean":"2024-09-17T07:16:05.048270+0000","last_became_active":"2024-09-17T07:09:24.408139+0000","last_became_peered":"2024-09-17T07:09:24.408139+0000","last_unstale":"2024-09-17T07:16:05.048270+0000","last_undegraded":"2024-09-17T07:16:05.048270+0000","last_fullsized":"2024-09-17T07:16:05.048270+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:33.622666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:10:36.334827+0000","last_clean_scrub_stamp":"2024-09-17T07:15:33.622666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:03.203848+0000","scrub_duration":0.002139064,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":152,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.073024+0000","last_change":"2024-09-17T07:15:37.523361+0000","last_active":"2024-09-17T07:16:06.073024+0000","last_peered":"2024-09-17T07:16:06.073024+0000","last_clean":"2024-09-17T07:16:06.073024+0000","last_became_active":"2024-09-17T07:09:24.399116+0000","last_became_peered":"2024-09-17T07:09:24.399116+0000","last_unstale":"2024-09-17T07:16:06.073024+0000","last_undegraded":"2024-09-17T07:16:06.073024+0000","last_fullsized":"2024-09-17T07:16:06.073024+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:37.523283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:15:37.523283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:50.803530+0000","scrub_duration":0.0013475850000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":306,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.072822+0000","last_change":"2024-09-17T07:15:40.540428+0000","last_active":"2024-09-17T07:16:06.072822+0000","last_peered":"2024-09-17T07:16:06.072822+0000","last_clean":"2024-09-17T07:16:06.072822+0000","last_became_active":"2024-09-17T07:02:18.027622+0000","last_became_peered":"2024-09-17T07:02:18.027622+0000","last_unstale":"2024-09-17T07:16:06.072822+0000","last_undegraded":"2024-09-17T07:16:06.072822+0000","last_fullsized":"2024-09-17T07:16:06.072822+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:40.540336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:15:40.540336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:46.617341+0000","scrub_duration":0.001212982,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"242'19","reported_seq":331,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.042829+0000","last_change":"2024-09-17T07:14:05.729683+0000","last_active":"2024-09-17T07:16:06.042829+0000","last_peered":"2024-09-17T07:16:06.042829+0000","last_clean":"2024-09-17T07:16:06.042829+0000","last_became_active":"2024-09-17T07:02:14.016255+0000","last_became_peered":"2024-09-17T07:02:14.016255+0000","last_unstale":"2024-09-17T07:16:06.042829+0000","last_undegraded":"2024-09-17T07:16:06.042829+0000","last_fullsized":"2024-09-17T07:16:06.042829+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'19","last_scrub_stamp":"2024-09-17T07:14:05.729535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:14:05.729535+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.008660559,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"94'2","reported_seq":306,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.049065+0000","last_change":"2024-09-17T07:15:48.907903+0000","last_active":"2024-09-17T07:16:06.049065+0000","last_peered":"2024-09-17T07:16:06.049065+0000","last_clean":"2024-09-17T07:16:06.049065+0000","last_became_active":"2024-09-17T07:02:12.011461+0000","last_became_peered":"2024-09-17T07:02:12.011461+0000","last_unstale":"2024-09-17T07:16:06.049065+0000","last_undegraded":"2024-09-17T07:16:06.049065+0000","last_fullsized":"2024-09-17T07:16:06.049065+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-09-17T07:15:48.907821+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-09-17T07:07:31.717888+0000","last_clean_scrub_stamp":"2024-09-17T07:15:48.907821+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:00.875203+0000","scrub_duration":0.001503307,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":305,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.048341+0000","last_change":"2024-09-17T07:15:41.858178+0000","last_active":"2024-09-17T07:16:06.048341+0000","last_peered":"2024-09-17T07:16:06.048341+0000","last_clean":"2024-09-17T07:16:06.048341+0000","last_became_active":"2024-09-17T07:09:24.407362+0000","last_became_peered":"2024-09-17T07:09:24.407362+0000","last_unstale":"2024-09-17T07:16:06.048341+0000","last_undegraded":"2024-09-17T07:16:06.048341+0000","last_fullsized":"2024-09-17T07:16:06.048341+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:41.858093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:15:41.858093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:47.679821+0000","scrub_duration":0.0020998420000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":151,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.073026+0000","last_change":"2024-09-17T07:15:49.573768+0000","last_active":"2024-09-17T07:16:06.073026+0000","last_peered":"2024-09-17T07:16:06.073026+0000","last_clean":"2024-09-17T07:16:06.073026+0000","last_became_active":"2024-09-17T07:09:24.400791+0000","last_became_peered":"2024-09-17T07:09:24.400791+0000","last_unstale":"2024-09-17T07:16:06.073026+0000","last_undegraded":"2024-09-17T07:16:06.073026+0000","last_fullsized":"2024-09-17T07:16:06.073026+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:49.573666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:49.573666+0000","last_clean_scrub_stamp":"2024-09-17T07:15:49.573666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:18.941100+0000","scrub_duration":0.0017548970000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"242'8","reported_seq":214,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.046539+0000","last_change":"2024-09-17T07:14:43.060402+0000","last_active":"2024-09-17T07:16:06.046539+0000","last_peered":"2024-09-17T07:16:06.046539+0000","last_clean":"2024-09-17T07:16:06.046539+0000","last_became_active":"2024-09-17T07:09:24.396368+0000","last_became_peered":"2024-09-17T07:09:24.396368+0000","last_unstale":"2024-09-17T07:16:06.046539+0000","last_undegraded":"2024-09-17T07:16:06.046539+0000","last_fullsized":"2024-09-17T07:16:06.046539+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'8","last_scrub_stamp":"2024-09-17T07:14:43.059953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:14:43.059953+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0051998909999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":311,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.075067+0000","last_change":"2024-09-17T07:15:01.141244+0000","last_active":"2024-09-17T07:16:06.075067+0000","last_peered":"2024-09-17T07:16:06.075067+0000","last_clean":"2024-09-17T07:16:06.075067+0000","last_became_active":"2024-09-17T07:02:12.008898+0000","last_became_peered":"2024-09-17T07:02:12.008898+0000","last_unstale":"2024-09-17T07:16:06.075067+0000","last_undegraded":"2024-09-17T07:16:06.075067+0000","last_fullsized":"2024-09-17T07:16:06.075067+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:01.140327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:15:01.140327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0060829439999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":319,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-17T07:16:05.049694+0000","last_change":"2024-09-17T07:15:54.554064+0000","last_active":"2024-09-17T07:16:05.049694+0000","last_peered":"2024-09-17T07:16:05.049694+0000","last_clean":"2024-09-17T07:16:05.049694+0000","last_became_active":"2024-09-17T07:02:16.014982+0000","last_became_peered":"2024-09-17T07:02:16.014982+0000","last_unstale":"2024-09-17T07:16:05.049694+0000","last_undegraded":"2024-09-17T07:16:05.049694+0000","last_fullsized":"2024-09-17T07:16:05.049694+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:54.553970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:54.553970+0000","last_clean_scrub_stamp":"2024-09-17T07:15:54.553970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:11.981217+0000","scrub_duration":0.0012697520000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":294,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.073707+0000","last_change":"2024-09-17T07:13:59.042437+0000","last_active":"2024-09-17T07:16:06.073707+0000","last_peered":"2024-09-17T07:16:06.073707+0000","last_clean":"2024-09-17T07:16:06.073707+0000","last_became_active":"2024-09-17T07:02:18.021014+0000","last_became_peered":"2024-09-17T07:02:18.021014+0000","last_unstale":"2024-09-17T07:16:06.073707+0000","last_undegraded":"2024-09-17T07:16:06.073707+0000","last_fullsized":"2024-09-17T07:16:06.073707+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:59.042322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:13:59.042322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0076271120000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"242'19","reported_seq":195,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.074235+0000","last_change":"2024-09-17T07:13:45.076965+0000","last_active":"2024-09-17T07:16:06.074235+0000","last_peered":"2024-09-17T07:16:06.074235+0000","last_clean":"2024-09-17T07:16:06.074235+0000","last_became_active":"2024-09-17T07:09:24.404942+0000","last_became_peered":"2024-09-17T07:09:24.404942+0000","last_unstale":"2024-09-17T07:16:06.074235+0000","last_undegraded":"2024-09-17T07:16:06.074235+0000","last_fullsized":"2024-09-17T07:16:06.074235+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'19","last_scrub_stamp":"2024-09-17T07:13:45.076322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:13:45.076322+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0088383200000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":322,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-17T07:16:05.048487+0000","last_change":"2024-09-17T07:15:00.436459+0000","last_active":"2024-09-17T07:16:05.048487+0000","last_peered":"2024-09-17T07:16:05.048487+0000","last_clean":"2024-09-17T07:16:05.048487+0000","last_became_active":"2024-09-17T07:02:12.010195+0000","last_became_peered":"2024-09-17T07:02:12.010195+0000","last_unstale":"2024-09-17T07:16:05.048487+0000","last_undegraded":"2024-09-17T07:16:05.048487+0000","last_fullsized":"2024-09-17T07:16:05.048487+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:00.436330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:15:00.436330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037388289999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":296,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.073276+0000","last_change":"2024-09-17T07:13:47.027302+0000","last_active":"2024-09-17T07:16:06.073276+0000","last_peered":"2024-09-17T07:16:06.073276+0000","last_clean":"2024-09-17T07:16:06.073276+0000","last_became_active":"2024-09-17T07:02:16.016183+0000","last_became_peered":"2024-09-17T07:02:16.016183+0000","last_unstale":"2024-09-17T07:16:06.073276+0000","last_undegraded":"2024-09-17T07:16:06.073276+0000","last_fullsized":"2024-09-17T07:16:06.073276+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:47.027189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:13:47.027189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0024966390000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":151,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.073114+0000","last_change":"2024-09-17T07:14:51.603973+0000","last_active":"2024-09-17T07:16:06.073114+0000","last_peered":"2024-09-17T07:16:06.073114+0000","last_clean":"2024-09-17T07:16:06.073114+0000","last_became_active":"2024-09-17T07:09:24.392216+0000","last_became_peered":"2024-09-17T07:09:24.392216+0000","last_unstale":"2024-09-17T07:16:06.073114+0000","last_undegraded":"2024-09-17T07:16:06.073114+0000","last_fullsized":"2024-09-17T07:16:06.073114+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:51.603818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:14:51.603818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:13.344567+0000","scrub_duration":0.0041420099999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1c","version":"75'1","reported_seq":295,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.074271+0000","last_change":"2024-09-17T07:13:56.976301+0000","last_active":"2024-09-17T07:16:06.074271+0000","last_peered":"2024-09-17T07:16:06.074271+0000","last_clean":"2024-09-17T07:16:06.074271+0000","last_became_active":"2024-09-17T07:02:18.021855+0000","last_became_peered":"2024-09-17T07:02:18.021855+0000","last_unstale":"2024-09-17T07:16:06.074271+0000","last_undegraded":"2024-09-17T07:16:06.074271+0000","last_fullsized":"2024-09-17T07:16:06.074271+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-17T07:13:56.976159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:13:56.976159+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0053255050000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":312,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.073351+0000","last_change":"2024-09-17T07:15:00.144722+0000","last_active":"2024-09-17T07:16:06.073351+0000","last_peered":"2024-09-17T07:16:06.073351+0000","last_clean":"2024-09-17T07:16:06.073351+0000","last_became_active":"2024-09-17T07:02:12.005329+0000","last_became_peered":"2024-09-17T07:02:12.005329+0000","last_unstale":"2024-09-17T07:16:06.073351+0000","last_undegraded":"2024-09-17T07:16:06.073351+0000","last_fullsized":"2024-09-17T07:16:06.073351+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:00.144598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:15:00.144598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0076347749999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"294'25","reported_seq":388,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.073592+0000","last_change":"2024-09-17T07:16:00.471391+0000","last_active":"2024-09-17T07:16:06.073592+0000","last_peered":"2024-09-17T07:16:06.073592+0000","last_clean":"2024-09-17T07:16:06.073592+0000","last_became_active":"2024-09-17T07:10:49.730948+0000","last_became_peered":"2024-09-17T07:10:49.730948+0000","last_unstale":"2024-09-17T07:16:06.073592+0000","last_undegraded":"2024-09-17T07:16:06.073592+0000","last_fullsized":"2024-09-17T07:16:06.073592+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'25","last_scrub_stamp":"2024-09-17T07:16:00.471311+0000","last_deep_scrub":"294'25","last_deep_scrub_stamp":"2024-09-17T07:16:00.471311+0000","last_clean_scrub_stamp":"2024-09-17T07:16:00.471311+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:10.547342+0000","scrub_duration":0.0040525370000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"75'8","reported_seq":498,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-17T07:16:05.685845+0000","last_change":"2024-09-17T07:15:57.561805+0000","last_active":"2024-09-17T07:16:05.685845+0000","last_peered":"2024-09-17T07:16:05.685845+0000","last_clean":"2024-09-17T07:16:05.685845+0000","last_became_active":"2024-09-17T07:02:16.020560+0000","last_became_peered":"2024-09-17T07:02:16.020560+0000","last_unstale":"2024-09-17T07:16:05.685845+0000","last_undegraded":"2024-09-17T07:16:05.685845+0000","last_fullsized":"2024-09-17T07:16:05.685845+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:15:57.561693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:15:57.561693+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:06.265158+0000","scrub_duration":0.0015318759999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":294,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.048728+0000","last_change":"2024-09-17T07:15:37.885645+0000","last_active":"2024-09-17T07:16:06.048728+0000","last_peered":"2024-09-17T07:16:06.048728+0000","last_clean":"2024-09-17T07:16:06.048728+0000","last_became_active":"2024-09-17T07:02:18.021603+0000","last_became_peered":"2024-09-17T07:02:18.021603+0000","last_unstale":"2024-09-17T07:16:06.048728+0000","last_undegraded":"2024-09-17T07:16:06.048728+0000","last_fullsized":"2024-09-17T07:16:06.048728+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:37.885570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:15:37.885570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:07.813041+0000","scrub_duration":0.001703174,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":162,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-17T07:16:05.049861+0000","last_change":"2024-09-17T07:15:36.653927+0000","last_active":"2024-09-17T07:16:05.049861+0000","last_peered":"2024-09-17T07:16:05.049861+0000","last_clean":"2024-09-17T07:16:05.049861+0000","last_became_active":"2024-09-17T07:09:24.403360+0000","last_became_peered":"2024-09-17T07:09:24.403360+0000","last_unstale":"2024-09-17T07:16:05.049861+0000","last_undegraded":"2024-09-17T07:16:05.049861+0000","last_fullsized":"2024-09-17T07:16:05.049861+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:36.653817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:10:29.325664+0000","last_clean_scrub_stamp":"2024-09-17T07:15:36.653817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:55.913678+0000","scrub_duration":0.0014304299999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"242'19","reported_seq":394,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-17T07:16:05.048409+0000","last_change":"2024-09-17T07:15:41.670675+0000","last_active":"2024-09-17T07:16:05.048409+0000","last_peered":"2024-09-17T07:16:05.048409+0000","last_clean":"2024-09-17T07:16:05.048409+0000","last_became_active":"2024-09-17T07:03:48.760390+0000","last_became_peered":"2024-09-17T07:03:48.760390+0000","last_unstale":"2024-09-17T07:16:05.048409+0000","last_undegraded":"2024-09-17T07:16:05.048409+0000","last_fullsized":"2024-09-17T07:16:05.048409+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'19","last_scrub_stamp":"2024-09-17T07:15:41.670531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:15:41.670531+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:00.891108+0000","scrub_duration":0.0022660979999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":298,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.073582+0000","last_change":"2024-09-17T07:14:49.639583+0000","last_active":"2024-09-17T07:16:06.073582+0000","last_peered":"2024-09-17T07:16:06.073582+0000","last_clean":"2024-09-17T07:16:06.073582+0000","last_became_active":"2024-09-17T07:02:16.019701+0000","last_became_peered":"2024-09-17T07:02:16.019701+0000","last_unstale":"2024-09-17T07:16:06.073582+0000","last_undegraded":"2024-09-17T07:16:06.073582+0000","last_fullsized":"2024-09-17T07:16:06.073582+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:49.639480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:14:49.639480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:12.299880+0000","scrub_duration":0.003512032,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":308,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.043120+0000","last_change":"2024-09-17T07:15:57.151512+0000","last_active":"2024-09-17T07:16:06.043120+0000","last_peered":"2024-09-17T07:16:06.043120+0000","last_clean":"2024-09-17T07:16:06.043120+0000","last_became_active":"2024-09-17T07:02:18.363752+0000","last_became_peered":"2024-09-17T07:02:18.363752+0000","last_unstale":"2024-09-17T07:16:06.043120+0000","last_undegraded":"2024-09-17T07:16:06.043120+0000","last_fullsized":"2024-09-17T07:16:06.043120+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:57.151431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:15:57.151431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:19.350004+0000","scrub_duration":0.0012311939999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":352,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-17T07:16:05.048173+0000","last_change":"2024-09-17T07:15:35.670909+0000","last_active":"2024-09-17T07:16:05.048173+0000","last_peered":"2024-09-17T07:16:05.048173+0000","last_clean":"2024-09-17T07:16:05.048173+0000","last_became_active":"2024-09-17T07:03:48.759384+0000","last_became_peered":"2024-09-17T07:03:48.759384+0000","last_unstale":"2024-09-17T07:16:05.048173+0000","last_undegraded":"2024-09-17T07:16:05.048173+0000","last_fullsized":"2024-09-17T07:16:05.048173+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:35.670737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:11:18.559877+0000","last_clean_scrub_stamp":"2024-09-17T07:15:35.670737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:43.644043+0000","scrub_duration":0.0013632939999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"242'27","reported_seq":353,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.048953+0000","last_change":"2024-09-17T07:15:45.855529+0000","last_active":"2024-09-17T07:16:06.048953+0000","last_peered":"2024-09-17T07:16:06.048953+0000","last_clean":"2024-09-17T07:16:06.048953+0000","last_became_active":"2024-09-17T07:09:24.403030+0000","last_became_peered":"2024-09-17T07:09:24.403030+0000","last_unstale":"2024-09-17T07:16:06.048953+0000","last_undegraded":"2024-09-17T07:16:06.048953+0000","last_fullsized":"2024-09-17T07:16:06.048953+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'27","last_scrub_stamp":"2024-09-17T07:15:45.855430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:15:45.855430+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:00.331869+0000","scrub_duration":0.0021742689999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":296,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.048950+0000","last_change":"2024-09-17T07:14:00.509698+0000","last_active":"2024-09-17T07:16:06.048950+0000","last_peered":"2024-09-17T07:16:06.048950+0000","last_clean":"2024-09-17T07:16:06.048950+0000","last_became_active":"2024-09-17T07:02:16.015903+0000","last_became_peered":"2024-09-17T07:02:16.015903+0000","last_unstale":"2024-09-17T07:16:06.048950+0000","last_undegraded":"2024-09-17T07:16:06.048950+0000","last_fullsized":"2024-09-17T07:16:06.048950+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:00.509283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:25.794418+0000","last_clean_scrub_stamp":"2024-09-17T07:14:00.509283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0073526269999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":164,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-09-17T07:16:05.048172+0000","last_change":"2024-09-17T07:15:02.473582+0000","last_active":"2024-09-17T07:16:05.048172+0000","last_peered":"2024-09-17T07:16:05.048172+0000","last_clean":"2024-09-17T07:16:05.048172+0000","last_became_active":"2024-09-17T07:09:24.402411+0000","last_became_peered":"2024-09-17T07:09:24.402411+0000","last_unstale":"2024-09-17T07:16:05.048172+0000","last_undegraded":"2024-09-17T07:16:05.048172+0000","last_fullsized":"2024-09-17T07:16:05.048172+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:02.473467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:15:02.473467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:14.391965+0000","scrub_duration":0.001733163,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":142,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.074881+0000","last_change":"2024-09-17T07:13:15.914281+0000","last_active":"2024-09-17T07:16:06.074881+0000","last_peered":"2024-09-17T07:16:06.074881+0000","last_clean":"2024-09-17T07:16:06.074881+0000","last_became_active":"2024-09-17T07:09:24.404795+0000","last_became_peered":"2024-09-17T07:09:24.404795+0000","last_unstale":"2024-09-17T07:16:06.074881+0000","last_undegraded":"2024-09-17T07:16:06.074881+0000","last_fullsized":"2024-09-17T07:16:06.074881+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:15.914144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:13:15.914144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.059987944000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"242'23","reported_seq":215,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.049033+0000","last_change":"2024-09-17T07:14:58.689224+0000","last_active":"2024-09-17T07:16:06.049033+0000","last_peered":"2024-09-17T07:16:06.049033+0000","last_clean":"2024-09-17T07:16:06.049033+0000","last_became_active":"2024-09-17T07:09:24.394476+0000","last_became_peered":"2024-09-17T07:09:24.394476+0000","last_unstale":"2024-09-17T07:16:06.049033+0000","last_undegraded":"2024-09-17T07:16:06.049033+0000","last_fullsized":"2024-09-17T07:16:06.049033+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'23","last_scrub_stamp":"2024-09-17T07:14:58.689120+0000","last_deep_scrub":"86'17","last_deep_scrub_stamp":"2024-09-17T07:03:18.241067+0000","last_clean_scrub_stamp":"2024-09-17T07:14:58.689120+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:08.797039+0000","scrub_duration":0.0066193850000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":318,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-17T07:16:06.042722+0000","last_change":"2024-09-17T07:15:38.338684+0000","last_active":"2024-09-17T07:16:06.042722+0000","last_peered":"2024-09-17T07:16:06.042722+0000","last_clean":"2024-09-17T07:16:06.042722+0000","last_became_active":"2024-09-17T07:09:24.393329+0000","last_became_peered":"2024-09-17T07:09:24.393329+0000","last_unstale":"2024-09-17T07:16:06.042722+0000","last_undegraded":"2024-09-17T07:16:06.042722+0000","last_fullsized":"2024-09-17T07:16:06.042722+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:38.338552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:15:38.338552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:58.182182+0000","scrub_duration":0.0017509940000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":48,"acting":48,"num_store_stats":7},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70,"ondisk_log_size":70,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1121,"num_read_kb":878,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":109,"num_bytes_recovered":2228,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":13004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":728,"ondisk_log_size":728,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":828,"num_read_kb":823,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1386,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":12,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":4578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037933,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":610040,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":608638,"kb_avail":93135112,"statfs":{"total":95995035648,"available":95370354688,"internally_reserved":0,"allocated":1421312,"data_stored":1021346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":623245771},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:15:24 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":1.244,"15min":1.585},"min":{"1min":0.29699999999999999,"5min":0.20699999999999999,"15min":0.192},"max":{"1min":2.867,"5min":8.6080000000000005,"15min":52.478999999999999},"last":0.54900000000000004},{"interface":"front","average":{"1min":1.2809999999999999,"5min":1.4370000000000001,"15min":1.6910000000000001},"min":{"1min":0.246,"5min":0.23899999999999999,"15min":0.22600000000000001},"max":{"1min":4.766,"5min":19.216000000000001,"15min":52.871000000000002},"last":0.44400000000000001}]},{"osd":1,"last update":"Tue Sep 17 07:15:24 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":1.4510000000000001,"15min":1.3859999999999999},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.223},"max":{"1min":1.573,"5min":15.801,"15min":73.549000000000007},"last":0.36499999999999999},{"interface":"front","average":{"1min":1.3779999999999999,"5min":1.2969999999999999,"15min":1.383},"min":{"1min":0.29699999999999999,"5min":0.26200000000000001,"15min":0.24099999999999999},"max":{"1min":4.7359999999999998,"5min":13.989000000000001,"15min":73.116},"last":0.53700000000000003}]},{"osd":2,"last update":"Tue Sep 17 07:15:24 2024","interfaces":[{"interface":"back","average":{"1min":1.3620000000000001,"5min":1.1759999999999999,"15min":1.121},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":5.0209999999999999,"5min":6.4900000000000002,"15min":52.055999999999997},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.77700000000000002,"5min":1.165,"15min":1.1279999999999999},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":4.9320000000000004,"5min":8.1639999999999997,"15min":52.064999999999998},"last":0.51500000000000001}]},{"osd":3,"last update":"Tue Sep 17 07:15:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2869999999999999,"5min":1.1799999999999999,"15min":1.3140000000000001},"min":{"1min":0.35899999999999999,"5min":0.29199999999999998,"15min":0.28799999999999998},"max":{"1min":5.0439999999999996,"5min":6.4640000000000004,"15min":51.689},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.92400000000000004,"5min":1.224,"15min":1.429},"min":{"1min":0.32900000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":5.1520000000000001,"5min":8.1839999999999993,"15min":52.145000000000003},"last":0.57099999999999995}]},{"osd":4,"last update":"Tue Sep 17 07:15:24 2024","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.1459999999999999,"15min":0.72099999999999997},"min":{"1min":0.31900000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":4.6950000000000003,"5min":8.5730000000000004,"15min":8.5730000000000004},"last":0.56000000000000005},{"interface":"front","average":{"1min":1.1890000000000001,"5min":1.2989999999999999,"15min":0.77700000000000002},"min":{"1min":0.30399999999999999,"5min":0.26000000000000001,"15min":0.24399999999999999},"max":{"1min":4.9569999999999999,"5min":6.5129999999999999,"15min":6.5129999999999999},"last":0.32200000000000001}]},{"osd":5,"last update":"Tue Sep 17 07:15:24 2024","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.806,"15min":0.94299999999999995},"min":{"1min":0.27900000000000003,"5min":0.253,"15min":0.23699999999999999},"max":{"1min":2.895,"5min":29.428999999999998,"15min":29.428999999999998},"last":0.46000000000000002},{"interface":"front","average":{"1min":1.8069999999999999,"5min":1.8839999999999999,"15min":1.0029999999999999},"min":{"1min":0.375,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":4.7930000000000001,"5min":30.059999999999999,"15min":30.059999999999999},"last":0.58199999999999996}]},{"osd":6,"last update":"Tue Sep 17 07:15:24 2024","interfaces":[{"interface":"back","average":{"1min":1.7030000000000001,"5min":1.5640000000000001,"15min":0.92000000000000004},"min":{"1min":0.34699999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":6.3339999999999996,"5min":25.363,"15min":25.363},"last":0.49099999999999999},{"interface":"front","average":{"1min":1.423,"5min":1.607,"15min":0.92100000000000004},"min":{"1min":0.41799999999999998,"5min":0.28399999999999997,"15min":0.27500000000000002},"max":{"1min":4.9889999999999999,"5min":23.571000000000002,"15min":23.571000000000002},"last":0.502}]}]},{"osd":6,"up_from":50,"seq":214748364976,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":371352,"kb_used_data":1416,"kb_used_omap":1,"kb_used_meta":369918,"kb_avail":93373800,"statfs":{"total":95995035648,"available":95614771200,"internally_reserved":0,"allocated":1449984,"data_stored":1021963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":378796487},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:15:08 2024","interfaces":[{"interface":"back","average":{"1min":5.8029999999999999,"5min":1.946,"15min":1.988},"min":{"1min":0.20200000000000001,"5min":0.185,"15min":0.185},"max":{"1min":126.902,"5min":126.902,"15min":126.902},"last":0.36399999999999999},{"interface":"front","average":{"1min":6.3780000000000001,"5min":2.0259999999999998,"15min":1.9490000000000001},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20799999999999999},"max":{"1min":127.06399999999999,"5min":127.06399999999999,"15min":127.06399999999999},"last":0.40500000000000003}]},{"osd":1,"last update":"Tue Sep 17 07:15:08 2024","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":1.1379999999999999,"15min":0.83999999999999997},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22600000000000001},"max":{"1min":2.395,"5min":14.720000000000001,"15min":14.720000000000001},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.67900000000000005,"5min":1.1240000000000001,"15min":0.88300000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.22900000000000001},"max":{"1min":3.0990000000000002,"5min":14.699,"15min":14.699},"last":0.39400000000000002}]},{"osd":2,"last update":"Tue Sep 17 07:15:08 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.81499999999999995,"15min":0.84599999999999997},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":3.1360000000000001,"5min":4.9039999999999999,"15min":33.079000000000001},"last":0.32600000000000001},{"interface":"front","average":{"1min":1.1040000000000001,"5min":0.873,"15min":0.85199999999999998},"min":{"1min":0.26300000000000001,"5min":0.25600000000000001,"15min":0.23000000000000001},"max":{"1min":4.8399999999999999,"5min":5.734,"15min":33.100000000000001},"last":0.371}]},{"osd":3,"last update":"Tue Sep 17 07:15:08 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.81599999999999995,"15min":1.149},"min":{"1min":0.307,"5min":0.24399999999999999,"15min":0.23200000000000001},"max":{"1min":2.48,"5min":5.0830000000000002,"15min":52.944000000000003},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.94999999999999996,"15min":1.212},"min":{"1min":0.28599999999999998,"5min":0.28499999999999998,"15min":0.27700000000000002},"max":{"1min":5.7690000000000001,"5min":5.7690000000000001,"15min":52.747},"last":0.442}]},{"osd":4,"last update":"Tue Sep 17 07:15:08 2024","interfaces":[{"interface":"back","average":{"1min":1.744,"5min":4.7640000000000002,"15min":1.931},"min":{"1min":0.28899999999999998,"5min":0.214,"15min":0.214},"max":{"1min":7.1050000000000004,"5min":260.77199999999999,"15min":260.77199999999999},"last":0.28499999999999998},{"interface":"front","average":{"1min":1.536,"5min":4.7229999999999999,"15min":1.929},"min":{"1min":0.28899999999999998,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":4.5449999999999999,"5min":260.69099999999997,"15min":260.69099999999997},"last":0.52400000000000002}]},{"osd":5,"last update":"Tue Sep 17 07:15:08 2024","interfaces":[{"interface":"back","average":{"1min":2.073,"5min":1.8620000000000001,"15min":0.97099999999999997},"min":{"1min":0.33100000000000002,"5min":0.26200000000000001,"15min":0.25900000000000001},"max":{"1min":8.0500000000000007,"5min":32.042000000000002,"15min":32.042000000000002},"last":0.34000000000000002},{"interface":"front","average":{"1min":1.573,"5min":1.7849999999999999,"15min":0.97499999999999998},"min":{"1min":0.27200000000000002,"5min":0.251,"15min":0.251},"max":{"1min":5.032,"5min":31.852,"15min":31.852},"last":0.50700000000000001}]},{"osd":7,"last update":"Tue Sep 17 07:15:34 2024","interfaces":[{"interface":"back","average":{"1min":2.1230000000000002,"5min":1.48,"15min":0.91900000000000004},"min":{"1min":0.47999999999999998,"5min":0.28100000000000003,"15min":0.23699999999999999},"max":{"1min":6.2110000000000003,"5min":7.657,"15min":7.657},"last":0.54000000000000004},{"interface":"front","average":{"1min":2.1269999999999998,"5min":1.4470000000000001,"15min":0.90900000000000003},"min":{"1min":0.39800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":6.0519999999999996,"5min":7.1890000000000001,"15min":7.1890000000000001},"last":0.46400000000000002}]}]},{"osd":5,"up_from":42,"seq":180388626613,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":581564,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":580734,"kb_avail":93163588,"statfs":{"total":95995035648,"available":95399514112,"internally_reserved":0,"allocated":831488,"data_stored":431310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":594672070},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:15:47 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.94399999999999995,"15min":1.5629999999999999},"min":{"1min":0.218,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":1.823,"5min":5.202,"15min":49.265000000000001},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.84999999999999998,"5min":1.206,"15min":1.669},"min":{"1min":0.309,"5min":0.221,"15min":0.219},"max":{"1min":4.649,"5min":6.7149999999999999,"15min":47.350000000000001},"last":0.41399999999999998}]},{"osd":1,"last update":"Tue Sep 17 07:15:47 2024","interfaces":[{"interface":"back","average":{"1min":1.4319999999999999,"5min":1.0509999999999999,"15min":0.90100000000000002},"min":{"1min":0.30499999999999999,"5min":0.22,"15min":0.22},"max":{"1min":13.711,"5min":13.711,"15min":20.329000000000001},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.99399999999999999,"5min":1.3320000000000001,"15min":1.149},"min":{"1min":0.32000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4.617,"5min":6.7560000000000002,"15min":20.408999999999999},"last":0.371}]},{"osd":2,"last update":"Tue Sep 17 07:15:47 2024","interfaces":[{"interface":"back","average":{"1min":1.254,"5min":1.125,"15min":1.0880000000000001},"min":{"1min":0.29299999999999998,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":6.3419999999999996,"5min":6.3419999999999996,"15min":20.486000000000001},"last":0.496},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.92200000000000004,"15min":0.97499999999999998},"min":{"1min":0.317,"5min":0.25800000000000001,"15min":0.23200000000000001},"max":{"1min":1.8440000000000001,"5min":5.7789999999999999,"15min":23.925999999999998},"last":0.42599999999999999}]},{"osd":3,"last update":"Tue Sep 17 07:15:47 2024","interfaces":[{"interface":"back","average":{"1min":1.9710000000000001,"5min":1.353,"15min":1.4359999999999999},"min":{"1min":0.32100000000000001,"5min":0.27100000000000002,"15min":0.247},"max":{"1min":13.882,"5min":13.882,"15min":68.978999999999999},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.93500000000000005,"15min":1.2849999999999999},"min":{"1min":0.38400000000000001,"5min":0.29999999999999999,"15min":0.253},"max":{"1min":1.863,"5min":5.6139999999999999,"15min":69.084000000000003},"last":0.48399999999999999}]},{"osd":4,"last update":"Tue Sep 17 07:15:47 2024","interfaces":[{"interface":"back","average":{"1min":1.3979999999999999,"5min":1.821,"15min":1.4930000000000001},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.22},"max":{"1min":4.569,"5min":14.036,"15min":142.327},"last":0.32900000000000001},{"interface":"front","average":{"1min":1.4219999999999999,"5min":1.5229999999999999,"15min":1.4039999999999999},"min":{"1min":0.25,"5min":0.23999999999999999,"15min":0.20999999999999999},"max":{"1min":6.415,"5min":13.478999999999999,"15min":142.13200000000001},"last":0.39500000000000002}]},{"osd":6,"last update":"Tue Sep 17 07:16:04 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":1.004,"15min":0.78200000000000003},"min":{"1min":0.32700000000000001,"5min":0.27600000000000002,"15min":0.23200000000000001},"max":{"1min":1.631,"5min":6.8419999999999996,"15min":8.3149999999999995},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.88500000000000001,"5min":1.169,"15min":0.83299999999999996},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.24099999999999999},"max":{"1min":2.9550000000000001,"5min":6.4459999999999997,"15min":6.4459999999999997},"last":0.51700000000000002}]},{"osd":7,"last update":"Tue Sep 17 07:15:24 2024","interfaces":[{"interface":"back","average":{"1min":1.976,"5min":1.742,"15min":1.012},"min":{"1min":0.35299999999999998,"5min":0.29599999999999999,"15min":0.29099999999999998},"max":{"1min":6.8869999999999996,"5min":6.8869999999999996,"15min":6.8869999999999996},"last":0.439},{"interface":"front","average":{"1min":1.2050000000000001,"5min":1.397,"15min":0.84799999999999998},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.27100000000000002},"max":{"1min":3.4329999999999998,"5min":5.641,"15min":5.641},"last":0.52900000000000003}]}]},{"osd":4,"up_from":33,"seq":141733920955,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":455248,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":454398,"kb_avail":93289904,"statfs":{"total":95995035648,"available":95528861696,"internally_reserved":0,"allocated":847872,"data_stored":437469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":465304006},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:15:06 2024","interfaces":[{"interface":"back","average":{"1min":1.4890000000000001,"5min":1.355,"15min":2.0209999999999999},"min":{"1min":0.215,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":5.4500000000000002,"5min":18.434999999999999,"15min":74.551000000000002},"last":0.44500000000000001},{"interface":"front","average":{"1min":1.0329999999999999,"5min":1.333,"15min":2.125},"min":{"1min":0.22700000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":3.1819999999999999,"5min":18.521000000000001,"15min":74.778999999999996},"last":0.52300000000000002}]},{"osd":1,"last update":"Tue Sep 17 07:15:06 2024","interfaces":[{"interface":"back","average":{"1min":1.3109999999999999,"5min":2.125,"15min":1.484},"min":{"1min":0.29199999999999998,"5min":0.24399999999999999,"15min":0.23000000000000001},"max":{"1min":5.3010000000000002,"5min":44.984000000000002,"15min":54.914999999999999},"last":0.32900000000000001},{"interface":"front","average":{"1min":1.8540000000000001,"5min":2.1739999999999999,"15min":1.4359999999999999},"min":{"1min":0.39300000000000002,"5min":0.23300000000000001,"15min":0.22800000000000001},"max":{"1min":5.4109999999999996,"5min":45.353999999999999,"15min":53.725000000000001},"last":0.35199999999999998}]},{"osd":2,"last update":"Tue Sep 17 07:15:06 2024","interfaces":[{"interface":"back","average":{"1min":1.6499999999999999,"5min":1.167,"15min":1.075},"min":{"1min":0.308,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":5.4980000000000002,"5min":6.8929999999999998,"15min":49.651000000000003},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.63700000000000001,"5min":1.0920000000000001,"15min":1.0469999999999999},"min":{"1min":0.27500000000000002,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":2.1869999999999998,"5min":16.003,"15min":48.813000000000002},"last":0.49299999999999999}]},{"osd":3,"last update":"Tue Sep 17 07:15:06 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.94199999999999995,"15min":1.016},"min":{"1min":0.30599999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":1.272,"5min":4.2599999999999998,"15min":39.042999999999999},"last":0.64000000000000001},{"interface":"front","average":{"1min":1.2989999999999999,"5min":1.171,"15min":1.105},"min":{"1min":0.32200000000000001,"5min":0.27100000000000002,"15min":0.253},"max":{"1min":5.2699999999999996,"5min":15.970000000000001,"15min":38.920000000000002},"last":0.47299999999999998}]},{"osd":5,"last update":"Tue Sep 17 07:15:34 2024","interfaces":[{"interface":"back","average":{"1min":1.012,"5min":1.252,"15min":0.85399999999999998},"min":{"1min":0.28399999999999997,"5min":0.247,"15min":0.20799999999999999},"max":{"1min":2.7370000000000001,"5min":7.1909999999999998,"15min":11.486000000000001},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.84199999999999997,"5min":1.244,"15min":0.86299999999999999},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.26100000000000001},"max":{"1min":2.0169999999999999,"5min":5.9870000000000001,"15min":11.781000000000001},"last":0.58499999999999996}]},{"osd":6,"last update":"Tue Sep 17 07:15:06 2024","interfaces":[{"interface":"back","average":{"1min":1.4419999999999999,"5min":1.125,"15min":0.90200000000000002},"min":{"1min":0.30099999999999999,"5min":0.218,"15min":0.218},"max":{"1min":5.734,"5min":5.734,"15min":47.923999999999999},"last":0.371},{"interface":"front","average":{"1min":1.9570000000000001,"5min":1.4690000000000001,"15min":1.0069999999999999},"min":{"1min":0.36199999999999999,"5min":0.26700000000000002,"15min":0.26300000000000001},"max":{"1min":6.7670000000000003,"5min":10.051,"15min":47.600999999999999},"last":0.56399999999999995}]},{"osd":7,"last update":"Tue Sep 17 07:15:25 2024","interfaces":[{"interface":"back","average":{"1min":1.544,"5min":1.3109999999999999,"15min":1.022},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":6.5140000000000002,"5min":7.1550000000000002,"15min":48.719000000000001},"last":0.61599999999999999},{"interface":"front","average":{"1min":3.0249999999999999,"5min":1.637,"15min":1.115},"min":{"1min":0.44900000000000001,"5min":0.313,"15min":0.28499999999999998},"max":{"1min":6.3410000000000002,"5min":6.3410000000000002,"15min":50.527000000000001},"last":0.53400000000000003}]}]},{"osd":3,"up_from":0,"seq":115964117181,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":18,"seq":77309411523,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":631624,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":630782,"kb_avail":93113528,"statfs":{"total":95995035648,"available":95348252672,"internally_reserved":0,"allocated":847872,"data_stored":438313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":645921222},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:15:43 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":1.466,"15min":1.3779999999999999},"min":{"1min":0.17499999999999999,"5min":0.159,"15min":0.159},"max":{"1min":1.498,"5min":83.793999999999997,"15min":83.793999999999997},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.38100000000000001,"5min":1.3899999999999999,"15min":1.512},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":0.67800000000000005,"5min":83.588999999999999,"15min":83.588999999999999},"last":0.45600000000000002}]},{"osd":1,"last update":"Tue Sep 17 07:15:43 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":1.7949999999999999,"15min":1.623},"min":{"1min":0.23699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":1.6459999999999999,"5min":47.101999999999997,"15min":79.659000000000006},"last":0.505},{"interface":"front","average":{"1min":0.55000000000000004,"5min":1.6990000000000001,"15min":1.5600000000000001},"min":{"1min":0.22700000000000001,"5min":0.20899999999999999,"15min":0.20200000000000001},"max":{"1min":1.8049999999999999,"5min":45.831000000000003,"15min":80.155000000000001},"last":0.39400000000000002}]},{"osd":3,"last update":"Tue Sep 17 07:15:58 2024","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.70399999999999996,"15min":0.97899999999999998},"min":{"1min":0.23599999999999999,"5min":0.22,"15min":0.20100000000000001},"max":{"1min":1.5740000000000001,"5min":5.2039999999999997,"15min":23.43},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.55200000000000005,"15min":1.0409999999999999},"min":{"1min":0.26400000000000001,"5min":0.23499999999999999,"15min":0.223},"max":{"1min":0.83599999999999997,"5min":4.5,"15min":23.408000000000001},"last":0.47599999999999998}]},{"osd":4,"last update":"Tue Sep 17 07:15:21 2024","interfaces":[{"interface":"back","average":{"1min":1.218,"5min":0.97799999999999998,"15min":0.89500000000000002},"min":{"1min":0.26100000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":6.8449999999999998,"5min":6.8449999999999998,"15min":6.8449999999999998},"last":0.61199999999999999},{"interface":"front","average":{"1min":1.4330000000000001,"5min":1.097,"15min":0.93200000000000005},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":6.8609999999999998,"5min":6.8609999999999998,"15min":6.8609999999999998},"last":0.60399999999999998}]},{"osd":5,"last update":"Tue Sep 17 07:15:43 2024","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":1.3779999999999999,"15min":1.119},"min":{"1min":0.33600000000000002,"5min":0.26900000000000002,"15min":0.249},"max":{"1min":4.0960000000000001,"5min":15.276999999999999,"15min":15.276999999999999},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.96699999999999997,"5min":1.1930000000000001,"15min":0.99099999999999999},"min":{"1min":0.35899999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":2.048,"5min":15.561999999999999,"15min":15.561999999999999},"last":0.56000000000000005}]},{"osd":6,"last update":"Tue Sep 17 07:15:58 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.98199999999999998,"15min":1.3280000000000001},"min":{"1min":0.36099999999999999,"5min":0.29499999999999998,"15min":0.28999999999999998},"max":{"1min":2.431,"5min":6.899,"15min":101.587},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.755,"15min":1.3400000000000001},"min":{"1min":0.38100000000000001,"5min":0.32300000000000001,"15min":0.28299999999999997},"max":{"1min":3.2519999999999998,"5min":3.2519999999999998,"15min":100.696},"last":0.65800000000000003}]},{"osd":7,"last update":"Tue Sep 17 07:15:39 2024","interfaces":[{"interface":"back","average":{"1min":1.212,"5min":1.25,"15min":1},"min":{"1min":0.36699999999999999,"5min":0.316,"15min":0.316},"max":{"1min":2.629,"5min":7.8099999999999996,"15min":7.8099999999999996},"last":0.64300000000000002},{"interface":"front","average":{"1min":1.3220000000000001,"5min":1.272,"15min":1.0780000000000001},"min":{"1min":0.38,"5min":0.34999999999999998,"15min":0.32900000000000001},"max":{"1min":3.2429999999999999,"5min":6.9219999999999997,"15min":9.2919999999999998},"last":0.63200000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575048,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":541408,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":540542,"kb_avail":93203744,"statfs":{"total":95995035648,"available":95440633856,"internally_reserved":0,"allocated":872448,"data_stored":432408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":553515462},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:16:00 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.73699999999999999,"15min":1.139},"min":{"1min":0.183,"5min":0.155,"15min":0.155},"max":{"1min":2.6070000000000002,"5min":4.5780000000000003,"15min":22.521000000000001},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.67500000000000004,"15min":1.0369999999999999},"min":{"1min":0.247,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.198,"5min":4.335,"15min":22.471},"last":0.40000000000000002}]},{"osd":2,"last update":"Tue Sep 17 07:15:44 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.91000000000000003,"15min":1.024},"min":{"1min":0.21299999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":2.6360000000000001,"5min":6.1920000000000002,"15min":11.055999999999999},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.73899999999999999,"15min":0.94799999999999995},"min":{"1min":0.22900000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":1.3160000000000001,"5min":4.1559999999999997,"15min":9.3160000000000007},"last":0.371}]},{"osd":3,"last update":"Tue Sep 17 07:16:00 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.67800000000000005,"15min":0.94999999999999996},"min":{"1min":0.27400000000000002,"5min":0.25900000000000001,"15min":0.219},"max":{"1min":2.4409999999999998,"5min":4.1269999999999998,"15min":8.5510000000000002},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.371,"5min":0.47399999999999998,"15min":0.98599999999999999},"min":{"1min":0.247,"5min":0.22500000000000001,"15min":0.19500000000000001},"max":{"1min":0.627,"5min":2.8199999999999998,"15min":19.602},"last":0.436}]},{"osd":4,"last update":"Tue Sep 17 07:15:20 2024","interfaces":[{"interface":"back","average":{"1min":1.1339999999999999,"5min":1.4670000000000001,"15min":1.129},"min":{"1min":0.36199999999999999,"5min":0.314,"15min":0.25},"max":{"1min":2.5659999999999998,"5min":19.088999999999999,"15min":19.088999999999999},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.98999999999999999,"5min":1.2010000000000001,"15min":1.1759999999999999},"min":{"1min":0.36799999999999999,"5min":0.317,"15min":0.307},"max":{"1min":2.6459999999999999,"5min":18.959,"15min":27.097999999999999},"last":0.67700000000000005}]},{"osd":5,"last update":"Tue Sep 17 07:15:44 2024","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.47,"15min":1.2110000000000001},"min":{"1min":0.34799999999999998,"5min":0.28899999999999998,"15min":0.27100000000000002},"max":{"1min":3.6869999999999998,"5min":6.6790000000000003,"15min":16.84},"last":0.53400000000000003},{"interface":"front","average":{"1min":1.1559999999999999,"5min":1.069,"15min":1.169},"min":{"1min":0.29299999999999998,"5min":0.29199999999999998,"15min":0.28899999999999998},"max":{"1min":6.2560000000000002,"5min":6.2560000000000002,"15min":16.949999999999999},"last":0.621}]},{"osd":6,"last update":"Tue Sep 17 07:16:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":1.5149999999999999,"15min":1.274},"min":{"1min":0.33100000000000002,"5min":0.315,"15min":0.308},"max":{"1min":1.127,"5min":50.497,"15min":50.497},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.99099999999999999,"5min":1.6379999999999999,"15min":1.222},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.28899999999999998},"max":{"1min":3.593,"5min":50.793999999999997,"15min":50.793999999999997},"last":0.58999999999999997}]},{"osd":7,"last update":"Tue Sep 17 07:15:20 2024","interfaces":[{"interface":"back","average":{"1min":1.655,"5min":1.343,"15min":1.1919999999999999},"min":{"1min":0.435,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":3.6120000000000001,"5min":9.8179999999999996,"15min":9.8179999999999996},"last":0.65000000000000002},{"interface":"front","average":{"1min":1.464,"5min":1.6319999999999999,"15min":1.2410000000000001},"min":{"1min":0.35099999999999998,"5min":0.317,"15min":0.317},"max":{"1min":4.1219999999999999,"5min":7.5839999999999996,"15min":11.007999999999999},"last":0.68799999999999994}]}]},{"osd":0,"up_from":8,"seq":34359738571,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":618272,"kb_used_data":1420,"kb_used_omap":1,"kb_used_meta":616830,"kb_avail":93126880,"statfs":{"total":95995035648,"available":95361925120,"internally_reserved":0,"allocated":1454080,"data_stored":1028881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":631634374},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 17 07:15:18 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.89700000000000002,"15min":0.89500000000000002},"min":{"1min":0.254,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":1.518,"5min":21.876000000000001,"15min":21.876000000000001},"last":0.249},{"interface":"front","average":{"1min":0.56399999999999995,"5min":1.2330000000000001,"15min":1.081},"min":{"1min":0.17899999999999999,"5min":0.158,"15min":0.158},"max":{"1min":2.2599999999999998,"5min":22.521999999999998,"15min":22.521999999999998},"last":0.33200000000000002}]},{"osd":2,"last update":"Tue Sep 17 07:15:41 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.63100000000000001,"15min":1.1060000000000001},"min":{"1min":0.23599999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":1.327,"5min":6.3799999999999999,"15min":18.643999999999998},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.65900000000000003,"15min":1.0109999999999999},"min":{"1min":0.22,"5min":0.189,"15min":0.189},"max":{"1min":1.9850000000000001,"5min":4.431,"15min":17.972999999999999},"last":0.32000000000000001}]},{"osd":3,"last update":"Tue Sep 17 07:15:18 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.68700000000000006,"15min":1.0109999999999999},"min":{"1min":0.23599999999999999,"5min":0.23400000000000001,"15min":0.21199999999999999},"max":{"1min":0.629,"5min":5.5880000000000001,"15min":18.629000000000001},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.58899999999999997,"15min":1.018},"min":{"1min":0.255,"5min":0.24199999999999999,"15min":0.23699999999999999},"max":{"1min":1.1339999999999999,"5min":3.9100000000000001,"15min":22.358000000000001},"last":0.30299999999999999}]},{"osd":4,"last update":"Tue Sep 17 07:15:18 2024","interfaces":[{"interface":"back","average":{"1min":1.056,"5min":1.127,"15min":1.1180000000000001},"min":{"1min":0.32600000000000001,"5min":0.29799999999999999,"15min":0.29099999999999998},"max":{"1min":4.0590000000000002,"5min":19.146999999999998,"15min":19.146999999999998},"last":0.501},{"interface":"front","average":{"1min":0.90900000000000003,"5min":1.151,"15min":1.0780000000000001},"min":{"1min":0.28199999999999997,"5min":0.26500000000000001,"15min":0.255},"max":{"1min":4.0449999999999999,"5min":19.158999999999999,"15min":19.158999999999999},"last":0.41999999999999998}]},{"osd":5,"last update":"Tue Sep 17 07:15:53 2024","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":1.1220000000000001,"15min":1.1699999999999999},"min":{"1min":0.35899999999999999,"5min":0.35499999999999998,"15min":0.28899999999999998},"max":{"1min":4.0350000000000001,"5min":5.0730000000000004,"15min":18.841000000000001},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.67900000000000005,"5min":1.032,"15min":1.1479999999999999},"min":{"1min":0.30099999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":1.9910000000000001,"5min":4.3680000000000003,"15min":19.248000000000001},"last":0.48799999999999999}]},{"osd":6,"last update":"Tue Sep 17 07:15:18 2024","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.97899999999999998,"15min":1.016},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":4.6260000000000003,"5min":4.6260000000000003,"15min":18.602},"last":0.47699999999999998},{"interface":"front","average":{"1min":1.0289999999999999,"5min":0.98499999999999999,"15min":1.0569999999999999},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":4.4889999999999999,"5min":4.4889999999999999,"15min":18.692},"last":0.56899999999999995}]},{"osd":7,"last update":"Tue Sep 17 07:15:19 2024","interfaces":[{"interface":"back","average":{"1min":1.5740000000000001,"5min":1.4550000000000001,"15min":1.478},"min":{"1min":0.38600000000000001,"5min":0.30599999999999999,"15min":0.29599999999999999},"max":{"1min":4.4020000000000001,"5min":5.9509999999999996,"15min":46.183},"last":0.51300000000000001},{"interface":"front","average":{"1min":1.45,"5min":1.3899999999999999,"15min":1.3140000000000001},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.32000000000000001},"max":{"1min":4.0650000000000004,"5min":6.1079999999999997,"15min":45.914999999999999},"last":0.55900000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T07:16:14.080 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-17T07:16:14.080 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T07:16:14.080 INFO:tasks.thrashosds.thrasher:Killing osd 1, live_osds are [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-17T07:16:14.081 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-09-17T07:16:14.081 DEBUG:teuthology.orchestra.run.smithi100:> sudo systemctl stop ceph-e848267c-74c1-11ef-bceb-c7b262605968@osd.1 2024-09-17T07:16:14.138 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:16:14.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:14.162 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:14.162 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:16:14.264 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:16:14.287 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:14.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:14.292 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:16:14.314 INFO:journalctl@ceph.osd.1.smithi100.stdout:Sep 17 07:16:14 smithi100 systemd[1]: Stopping Ceph osd.1 for e848267c-74c1-11ef-bceb-c7b262605968... 2024-09-17T07:16:14.393 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:16:14.416 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:14.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:14.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:16:14.518 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:16:14.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:14.540 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:14.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:16:14.602 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:14 smithi100 ceph-mon[29092]: pgmap v628: 148 pgs: 148 active+clean; 582 KiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-09-17T07:16:14.602 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:14 smithi100 ceph-mon[29092]: osdmap e301: 8 total, 8 up, 7 in 2024-09-17T07:16:14.603 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:14 smithi100 ceph-mon[39462]: pgmap v628: 148 pgs: 148 active+clean; 582 KiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-09-17T07:16:14.603 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:14 smithi100 ceph-mon[39462]: osdmap e301: 8 total, 8 up, 7 in 2024-09-17T07:16:14.641 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:16:14.663 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:14.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:14.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:16:14.764 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:16:14.786 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:14.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:14.787 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:16:14.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:14 smithi135 ceph-mon[33238]: pgmap v628: 148 pgs: 148 active+clean; 582 KiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-09-17T07:16:14.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:14 smithi135 ceph-mon[33238]: osdmap e301: 8 total, 8 up, 7 in 2024-09-17T07:16:14.888 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:16:14.909 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:14.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:14.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:16:15.011 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:16:15.034 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:15.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:15.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:16:15.136 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:16:15.137 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-17T07:16:15.159 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:15.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:15.160 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:16:15.160 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:16:15.261 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:16:15.283 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:15.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:15.283 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:16:15.361 INFO:teuthology.orchestra.run.smithi100.stderr:nodeep-scrub is set 2024-09-17T07:16:15.373 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:15 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2338473294' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:16:15.373 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:15 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:16:15.373 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:15 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2338473294' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:16:15.373 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:15 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:16:15.385 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:16:15.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:15.407 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:15.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:16:15.508 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:16:15.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:15.531 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:15.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:16:15.632 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:16:15.653 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:15.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:15.654 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:16:15.755 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:16:15.777 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:15.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:15.778 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:16:15.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:15 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2338473294' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:16:15.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:15 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:16:15.879 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:16:15.905 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:15.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:15.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:16:16.007 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:16:16.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:16.030 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:16.031 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:16:16.111 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:16:15 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:16:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:16:16.111 INFO:journalctl@ceph.osd.1.smithi100.stdout:Sep 17 07:16:15 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-1[48943]: 2024-09-17T07:16:15.974+0000 7f475c3b0640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T07:16:16.111 INFO:journalctl@ceph.osd.1.smithi100.stdout:Sep 17 07:16:15 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-1[48943]: 2024-09-17T07:16:15.974+0000 7f475c3b0640 -1 osd.1 302 *** Got signal Terminated *** 2024-09-17T07:16:16.112 INFO:journalctl@ceph.osd.1.smithi100.stdout:Sep 17 07:16:15 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-1[48943]: 2024-09-17T07:16:15.974+0000 7f475c3b0640 -1 osd.1 302 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T07:16:16.132 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:16:16.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:16.155 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:16.155 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:16:16.256 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:16:16.278 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:16.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:16.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:16:16.364 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:16 smithi100 ceph-mon[29092]: pgmap v630: 148 pgs: 148 active+clean; 582 KiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:16:16.364 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:16 smithi100 ceph-mon[29092]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:16:16.364 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:16 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T07:16:16.364 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:16 smithi100 ceph-mon[29092]: osdmap e302: 8 total, 8 up, 7 in 2024-09-17T07:16:16.364 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:16 smithi100 ceph-mon[29092]: osd.1 marked itself down and dead 2024-09-17T07:16:16.364 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:16 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2503614878' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-17T07:16:16.364 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:16 smithi100 ceph-mon[39462]: pgmap v630: 148 pgs: 148 active+clean; 582 KiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:16:16.365 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:16 smithi100 ceph-mon[39462]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:16:16.365 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:16 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T07:16:16.365 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:16 smithi100 ceph-mon[39462]: osdmap e302: 8 total, 8 up, 7 in 2024-09-17T07:16:16.365 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:16 smithi100 ceph-mon[39462]: osd.1 marked itself down and dead 2024-09-17T07:16:16.365 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:16 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2503614878' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-17T07:16:16.365 INFO:teuthology.orchestra.run.smithi100.stderr:enabled application 'rados' on pool 'unique_pool_3' 2024-09-17T07:16:16.381 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:16:16.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:16.406 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:16.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:16:16.507 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:16:16.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:16.529 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:16.529 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:16:16.630 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:16:16.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:16.653 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:16.654 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:16:16.754 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:16:16.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:16.778 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:16.778 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:16:16.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:16 smithi135 ceph-mon[33238]: pgmap v630: 148 pgs: 148 active+clean; 582 KiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:16:16.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:16 smithi135 ceph-mon[33238]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:16:16.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:16 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T07:16:16.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:16 smithi135 ceph-mon[33238]: osdmap e302: 8 total, 8 up, 7 in 2024-09-17T07:16:16.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:16 smithi135 ceph-mon[33238]: osd.1 marked itself down and dead 2024-09-17T07:16:16.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:16 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2503614878' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-17T07:16:16.881 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:16:16.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:16.903 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:16.904 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:16:17.005 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:16:17.027 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:17.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:17.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:16:17.130 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:16:17.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:17.152 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:17.152 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:16:17.253 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:16:17.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:17.276 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:17.276 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:16:17.291 INFO:journalctl@ceph.osd.1.smithi100.stdout:Sep 17 07:16:17 smithi100 podman[154652]: 2024-09-17 07:16:17.031411598 +0000 UTC m=+2.282915832 container died 1d8852ff87f4ec212ad314ddb42e58ebf940b657ab784dc7cb84ff2bdfb5e1d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-1, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=reef-19e2a2b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T07:16:17.378 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:16:17.400 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:17.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:17.401 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:16:17.502 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:16:17.523 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:17.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:17.524 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:16:17.551 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:17 smithi100 ceph-mon[29092]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T07:16:17.551 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:17 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2503614878' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-17T07:16:17.552 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:17 smithi100 ceph-mon[29092]: osdmap e303: 8 total, 7 up, 7 in 2024-09-17T07:16:17.552 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:17 smithi100 ceph-mon[39462]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T07:16:17.552 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:17 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2503614878' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-17T07:16:17.552 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:17 smithi100 ceph-mon[39462]: osdmap e303: 8 total, 7 up, 7 in 2024-09-17T07:16:17.625 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:16:17.645 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:17.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:17.646 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:16:17.747 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:16:17.770 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:17.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:17.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:16:17.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:17 smithi135 ceph-mon[33238]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T07:16:17.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:17 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2503614878' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-17T07:16:17.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:17 smithi135 ceph-mon[33238]: osdmap e303: 8 total, 7 up, 7 in 2024-09-17T07:16:17.872 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:16:17.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:17.894 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:17.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:16:17.996 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:16:18.017 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:18.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:18.018 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:16:18.119 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:16:18.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:18.142 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:18.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:16:18.243 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:16:18.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:18.266 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:18.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:16:18.367 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:16:18.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:18.389 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:18.390 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:16:18.390 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:18 smithi100 ceph-mon[29092]: pgmap v633: 148 pgs: 25 stale+active+clean, 123 active+clean; 582 KiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:16:18.390 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:18 smithi100 ceph-mon[29092]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-17T07:16:18.391 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:18 smithi100 ceph-mon[29092]: osdmap e304: 8 total, 7 up, 7 in 2024-09-17T07:16:18.391 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:18 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:16:18.391 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:18 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:16:18.391 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:18 smithi100 ceph-mon[39462]: pgmap v633: 148 pgs: 25 stale+active+clean, 123 active+clean; 582 KiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:16:18.391 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:18 smithi100 ceph-mon[39462]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-17T07:16:18.391 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:18 smithi100 ceph-mon[39462]: osdmap e304: 8 total, 7 up, 7 in 2024-09-17T07:16:18.391 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:18 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:16:18.391 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:18 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:16:18.428 DEBUG:teuthology.orchestra.run.smithi100:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_3 bench 90 write' 2024-09-17T07:16:18.430 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-17T07:16:18.430 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-17T07:16:18.490 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:16:18.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:18.512 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:18.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:16:18.613 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:16:18.623 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:hints = 1 2024-09-17T07:16:18.623 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-17T07:16:18.623 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Object prefix: benchmark_data_smithi100_155039 2024-09-17T07:16:18.624 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-17T07:16:18.624 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 0 0 0 0 0 0 - 0 2024-09-17T07:16:18.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:18.645 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:18.645 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:16:18.746 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:16:18.782 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:18.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:18.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:16:18.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:18 smithi135 ceph-mon[33238]: pgmap v633: 148 pgs: 25 stale+active+clean, 123 active+clean; 582 KiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:16:18.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:18 smithi135 ceph-mon[33238]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-17T07:16:18.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:18 smithi135 ceph-mon[33238]: osdmap e304: 8 total, 7 up, 7 in 2024-09-17T07:16:18.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:18 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:16:18.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:18 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:16:18.884 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:16:18.919 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:18.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:18.920 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:16:19.021 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:16:19.060 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:19.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:19.061 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:16:19.126 INFO:journalctl@ceph.osd.1.smithi100.stdout:Sep 17 07:16:19 smithi100 podman[154652]: 2024-09-17 07:16:19.025462492 +0000 UTC m=+4.276966733 container cleanup 1d8852ff87f4ec212ad314ddb42e58ebf940b657ab784dc7cb84ff2bdfb5e1d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-1, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=reef-19e2a2b, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD) 2024-09-17T07:16:19.127 INFO:journalctl@ceph.osd.1.smithi100.stdout:Sep 17 07:16:19 smithi100 bash[154652]: ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-1 2024-09-17T07:16:19.161 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:16:19.199 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:19.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:19.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:16:19.293 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:16:18 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:16:19.301 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:16:19.339 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:19.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:19.341 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:16:19.349 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:16:19.442 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:16:19.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:19.484 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:19.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:16:19.585 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:16:19.621 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:19.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:19.622 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:16:19.624 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 1 16 3799 3783 236.425 236.438 0.00717806 0.00421637 2024-09-17T07:16:19.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:19 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:16:19.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:19 smithi100 ceph-mon[29092]: pgmap v635: 148 pgs: 32 active+undersized, 2 stale+active+clean, 22 active+undersized+degraded, 92 active+clean; 5.5 MiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 852 KiB/s wr, 13 op/s; 97/825 objects degraded (11.758%) 2024-09-17T07:16:19.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:19 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:16:19.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:19 smithi100 ceph-mon[39462]: pgmap v635: 148 pgs: 32 active+undersized, 2 stale+active+clean, 22 active+undersized+degraded, 92 active+clean; 5.5 MiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 852 KiB/s wr, 13 op/s; 97/825 objects degraded (11.758%) 2024-09-17T07:16:19.627 INFO:journalctl@ceph.osd.1.smithi100.stdout:Sep 17 07:16:19 smithi100 podman[154931]: 2024-09-17 07:16:19.285710996 +0000 UTC m=+2.288148681 container remove 1d8852ff87f4ec212ad314ddb42e58ebf940b657ab784dc7cb84ff2bdfb5e1d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-1, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=reef-19e2a2b, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True) 2024-09-17T07:16:19.724 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:16:19.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:19.753 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:19.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:16:19.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:19 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:16:19.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:19 smithi135 ceph-mon[33238]: pgmap v635: 148 pgs: 32 active+undersized, 2 stale+active+clean, 22 active+undersized+degraded, 92 active+clean; 5.5 MiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 852 KiB/s wr, 13 op/s; 97/825 objects degraded (11.758%) 2024-09-17T07:16:19.854 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:16:19.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:19.882 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:19.883 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:16:19.932 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T07:16:19.983 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:16:20.020 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:20.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:20.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:16:20.122 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:16:20.156 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:20.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:20.158 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:16:20.259 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:16:20.304 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:20.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:20.305 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:16:20.406 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:16:20.442 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:20.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:20.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:16:20.544 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:16:20.575 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:20.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:20.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:16:20.624 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 2 16 6829 6813 212.892 189.375 0.00405296 0.00468597 2024-09-17T07:16:20.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:20 smithi100 ceph-mon[29092]: Health check failed: Degraded data redundancy: 97/825 objects degraded (11.758%), 22 pgs degraded (PG_DEGRADED) 2024-09-17T07:16:20.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:20 smithi100 ceph-mon[29092]: osdmap e305: 8 total, 7 up, 7 in 2024-09-17T07:16:20.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:20 smithi100 ceph-mon[39462]: Health check failed: Degraded data redundancy: 97/825 objects degraded (11.758%), 22 pgs degraded (PG_DEGRADED) 2024-09-17T07:16:20.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:20 smithi100 ceph-mon[39462]: osdmap e305: 8 total, 7 up, 7 in 2024-09-17T07:16:20.677 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:16:20.715 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:20.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:20.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:16:20.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:20 smithi135 ceph-mon[33238]: Health check failed: Degraded data redundancy: 97/825 objects degraded (11.758%), 22 pgs degraded (PG_DEGRADED) 2024-09-17T07:16:20.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:20 smithi135 ceph-mon[33238]: osdmap e305: 8 total, 7 up, 7 in 2024-09-17T07:16:20.817 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:16:20.828 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:16:20.829 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:16:20.829 INFO:teuthology.orchestra.run.smithi100.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-17T07:16:20.852 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:20.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:20.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:16:20.863 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:16:20.863 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:16:20.892 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:16:20.892 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:16:20.924 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:16:20.925 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:16:20.953 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:16:20.954 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:16:20.954 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:16:20.984 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:16:20.984 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:16:20.989 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:20.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:20.990 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:16:21.012 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:16:21.012 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:16:21.091 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:16:21.120 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:21.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:21.121 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:16:21.222 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:16:21.259 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:21.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:21.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:16:21.361 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:16:21.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:21.394 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:21.394 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:16:21.495 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:16:21.533 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:21.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:21.534 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:16:21.624 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 3 16 10157 10141 211.255 208 0.0018402 0.00472573 2024-09-17T07:16:21.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:21 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/396478192' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:16:21.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:21 smithi100 ceph-mon[29092]: pgmap v637: 148 pgs: 39 active+undersized, 25 active+undersized+degraded, 84 active+clean; 45 MiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 7.5 MiB/s wr, 120 op/s; 323/2730 objects degraded (11.832%) 2024-09-17T07:16:21.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:21 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/396478192' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:16:21.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:21 smithi100 ceph-mon[39462]: pgmap v637: 148 pgs: 39 active+undersized, 25 active+undersized+degraded, 84 active+clean; 45 MiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 7.5 MiB/s wr, 120 op/s; 323/2730 objects degraded (11.832%) 2024-09-17T07:16:21.635 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:16:21.673 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:21.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:21.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:16:21.775 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:16:21.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:21 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/396478192' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:16:21.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:21 smithi135 ceph-mon[33238]: pgmap v637: 148 pgs: 39 active+undersized, 25 active+undersized+degraded, 84 active+clean; 45 MiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 7.5 MiB/s wr, 120 op/s; 323/2730 objects degraded (11.832%) 2024-09-17T07:16:21.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:21.814 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:21.814 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:16:21.915 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:16:21.951 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:21.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:21.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:16:22.053 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:16:22.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:22.077 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:22.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:16:22.178 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:16:22.214 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:22.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:22.215 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:16:22.316 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:16:22.353 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:22.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:22.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:16:22.381 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-17T07:16:22.381 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:16:22.455 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:16:22.487 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:22.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:22.488 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:16:22.589 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:16:22.624 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 4 16 13306 13290 207.641 196.812 0.00545717 0.00480797 2024-09-17T07:16:22.626 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:22.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:22.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:16:22.728 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:16:22.764 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:22.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:22.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:16:22.826 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:16:22.827 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:16:22.865 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:16:22.903 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:22.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:22.904 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:16:23.004 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:16:23.034 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:23.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:23.035 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:16:23.136 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:16:23.167 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:23.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:23.169 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:16:23.269 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:16:23.305 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:23.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:23.306 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:16:23.407 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:16:23.443 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:23.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:23.444 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:16:23.522 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-17T07:16:23.545 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:16:23.613 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:23.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:23.614 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:16:23.625 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 5 15 16329 16314 203.889 189 0.00707624 0.00489784 2024-09-17T07:16:23.631 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:16:23.715 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:16:23.763 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:23.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:23.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:16:23.865 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:16:23.896 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:16:23.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:23.897 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:23.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:16:23.998 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:16:24.028 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:24.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:24.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:16:24.133 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:16:24.170 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:24.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:24.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:16:24.272 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:16:24.307 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:24.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:24.309 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:16:24.409 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:16:24.440 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:24.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:24.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:16:24.542 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:16:24.573 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:24.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:24.575 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:16:24.625 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 6 16 19013 18997 197.854 167.688 0.005654 0.00504904 2024-09-17T07:16:24.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:24 smithi100 ceph-mon[29092]: pgmap v638: 148 pgs: 39 active+undersized, 25 active+undersized+degraded, 84 active+clean; 45 MiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 6.4 MiB/s wr, 103 op/s; 323/2730 objects degraded (11.832%) 2024-09-17T07:16:24.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:24 smithi100 ceph-mon[39462]: pgmap v638: 148 pgs: 39 active+undersized, 25 active+undersized+degraded, 84 active+clean; 45 MiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 6.4 MiB/s wr, 103 op/s; 323/2730 objects degraded (11.832%) 2024-09-17T07:16:24.627 INFO:journalctl@ceph.osd.1.smithi100.stdout:Sep 17 07:16:24 smithi100 podman[155563]: 2024-09-17 07:16:24.405730811 +0000 UTC m=+0.457495209 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9 2024-09-17T07:16:24.627 INFO:journalctl@ceph.osd.1.smithi100.stdout:Sep 17 07:16:24 smithi100 podman[155563]: 2024-09-17 07:16:24.52801738 +0000 UTC m=+0.579781764 container create a71fca5079fd27759205c852d95ffcf5aee09cd33dee775580478ba0d06eff33 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-1-deactivate, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, RELEASE=reef-19e2a2b, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_CLEAN=True) 2024-09-17T07:16:24.676 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:16:24.707 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:24.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:24.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:16:24.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:24 smithi135 ceph-mon[33238]: pgmap v638: 148 pgs: 39 active+undersized, 25 active+undersized+degraded, 84 active+clean; 45 MiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 6.4 MiB/s wr, 103 op/s; 323/2730 objects degraded (11.832%) 2024-09-17T07:16:24.809 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:16:24.828 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:16:24.839 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:24.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:24.840 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:16:24.915 INFO:journalctl@ceph.osd.1.smithi100.stdout:Sep 17 07:16:24 smithi100 podman[155563]: 2024-09-17 07:16:24.806559549 +0000 UTC m=+0.858323950 container init a71fca5079fd27759205c852d95ffcf5aee09cd33dee775580478ba0d06eff33 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-1-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.37.2, RELEASE=reef-19e2a2b, org.label-schema.schema-version=1.0, ceph=True) 2024-09-17T07:16:24.915 INFO:journalctl@ceph.osd.1.smithi100.stdout:Sep 17 07:16:24 smithi100 podman[155563]: 2024-09-17 07:16:24.811636248 +0000 UTC m=+0.863400639 container start a71fca5079fd27759205c852d95ffcf5aee09cd33dee775580478ba0d06eff33 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-1-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, RELEASE=reef-19e2a2b, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-09-17T07:16:24.941 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:16:24.978 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:24.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:24.979 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:16:25.080 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:16:25.120 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:25.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:25.121 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:16:25.222 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:16:25.260 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:25.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:25.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:16:25.269 INFO:journalctl@ceph.osd.1.smithi100.stdout:Sep 17 07:16:24 smithi100 podman[155563]: 2024-09-17 07:16:24.914766767 +0000 UTC m=+0.966531154 container attach a71fca5079fd27759205c852d95ffcf5aee09cd33dee775580478ba0d06eff33 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-1-deactivate, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=reef-19e2a2b, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T07:16:25.348 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:25 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 323/2730 objects degraded (11.832%), 25 pgs degraded (PG_DEGRADED) 2024-09-17T07:16:25.348 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:25 smithi135 ceph-mon[33238]: osdmap e306: 8 total, 7 up, 7 in 2024-09-17T07:16:25.362 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:16:25.402 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:25.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:25.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:16:25.504 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:16:25.534 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:25.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:25.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:16:25.625 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 7 16 21550 21534 192.239 158.562 0.00211971 0.00518893 2024-09-17T07:16:25.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:25 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 323/2730 objects degraded (11.832%), 25 pgs degraded (PG_DEGRADED) 2024-09-17T07:16:25.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:25 smithi100 ceph-mon[29092]: osdmap e306: 8 total, 7 up, 7 in 2024-09-17T07:16:25.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:25 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 323/2730 objects degraded (11.832%), 25 pgs degraded (PG_DEGRADED) 2024-09-17T07:16:25.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:25 smithi100 ceph-mon[39462]: osdmap e306: 8 total, 7 up, 7 in 2024-09-17T07:16:25.636 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:16:25.661 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:25.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:25.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:16:25.763 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:16:25.802 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:25.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:25.803 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:16:25.893 INFO:teuthology.orchestra.run.smithi135.stdout:ERROR: (22) Invalid argument 2024-09-17T07:16:25.894 INFO:teuthology.orchestra.run.smithi135.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:16:25.904 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:16:25.945 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:25.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:25.946 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:16:26.048 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:16:26.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:26.085 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:26.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:16:26.186 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:16:26.221 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:26.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:26.222 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:16:26.323 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:16:26.346 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:26.347 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:16:25 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:16:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:16:26.347 INFO:journalctl@ceph.osd.1.smithi100.stdout:Sep 17 07:16:26 smithi100 podman[155563]: 2024-09-17 07:16:26.099497516 +0000 UTC m=+2.151261916 container died a71fca5079fd27759205c852d95ffcf5aee09cd33dee775580478ba0d06eff33 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-1-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.37.2, RELEASE=reef-19e2a2b, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240731) 2024-09-17T07:16:26.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:26.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:16:26.449 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:16:26.476 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:26.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:26.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:16:26.578 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:16:26.617 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:26.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:26.618 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:16:26.625 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 8 16 23426 23410 182.865 117.25 0.00366008 0.00546197 2024-09-17T07:16:26.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:26 smithi100 ceph-mon[29092]: pgmap v640: 148 pgs: 36 active+undersized, 28 active+undersized+degraded, 84 active+clean; 1.0 GiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 130 MiB/s wr, 2.08k op/s; 4102/50025 objects degraded (8.200%) 2024-09-17T07:16:26.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:26 smithi100 ceph-mon[39462]: pgmap v640: 148 pgs: 36 active+undersized, 28 active+undersized+degraded, 84 active+clean; 1.0 GiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 130 MiB/s wr, 2.08k op/s; 4102/50025 objects degraded (8.200%) 2024-09-17T07:16:26.719 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:16:26.747 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:26.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:26.748 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:16:26.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:26 smithi135 ceph-mon[33238]: pgmap v640: 148 pgs: 36 active+undersized, 28 active+undersized+degraded, 84 active+clean; 1.0 GiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 130 MiB/s wr, 2.08k op/s; 4102/50025 objects degraded (8.200%) 2024-09-17T07:16:26.849 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:16:26.884 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:26.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:26.885 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:16:26.985 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:16:27.023 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:27.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:27.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:16:27.124 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:16:27.159 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:27.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:27.160 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:16:27.260 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:16:27.302 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:27.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:27.303 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:16:27.404 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:16:27.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:27.432 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:27.432 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:16:27.533 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:16:27.562 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:27.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:27.563 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:16:27.625 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 9 16 25473 25457 176.761 127.938 0.00526476 0.00565178 2024-09-17T07:16:27.664 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:16:27.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:27.704 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:27.704 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:16:27.805 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:16:27.828 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:27.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:27.830 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:16:27.930 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:16:27.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:27.963 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:27.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:16:28.064 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:16:28.085 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:28.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:28.086 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:16:28.110 DEBUG:teuthology.orchestra.run.smithi100:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e848267c-74c1-11ef-bceb-c7b262605968@osd.1.service' 2024-09-17T07:16:28.186 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:16:28.213 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:28.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:28.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:16:28.286 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T07:16:28.286 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-09-17T07:16:28.286 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.1 2024-09-17T07:16:28.286 INFO:tasks.thrashosds.thrasher:remote for osd 1 is ubuntu@smithi100.front.sepia.ceph.com 2024-09-17T07:16:28.287 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -n osd.1 --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --op list-pgs 2024-09-17T07:16:28.316 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:16:28.353 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:28.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:28.354 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:16:28.455 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:16:28.482 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:28.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:28.484 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:16:28.584 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:16:28.618 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:28.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:28.619 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:16:28.625 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 10 16 27120 27104 169.378 102.938 0.00222639 0.00589205 2024-09-17T07:16:28.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:28 smithi100 ceph-mon[29092]: pgmap v641: 148 pgs: 36 active+undersized, 28 active+undersized+degraded, 84 active+clean; 1.0 GiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 383 B/s rd, 128 MiB/s wr, 2.05k op/s; 4102/50025 objects degraded (8.200%) 2024-09-17T07:16:28.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:28 smithi100 ceph-mon[39462]: pgmap v641: 148 pgs: 36 active+undersized, 28 active+undersized+degraded, 84 active+clean; 1.0 GiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 383 B/s rd, 128 MiB/s wr, 2.05k op/s; 4102/50025 objects degraded (8.200%) 2024-09-17T07:16:28.720 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:16:28.756 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:28.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:28.758 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:16:28.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:28 smithi135 ceph-mon[33238]: pgmap v641: 148 pgs: 36 active+undersized, 28 active+undersized+degraded, 84 active+clean; 1.0 GiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 383 B/s rd, 128 MiB/s wr, 2.05k op/s; 4102/50025 objects degraded (8.200%) 2024-09-17T07:16:28.859 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:16:28.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:28.880 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:28.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:16:28.900 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/osd.1/config 2024-09-17T07:16:28.981 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:16:29.003 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:29.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:29.004 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:16:29.105 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:16:29.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:29.144 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:29.144 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:16:29.245 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:16:29.266 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:29.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:29.267 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:16:29.293 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:16:28 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:16:29.368 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:16:29.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:29.401 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:29.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:16:29.502 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:16:29.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:29.525 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:29.525 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:16:29.626 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 11 16 28303 28287 160.7 73.9375 0.00205465 0.00609789 2024-09-17T07:16:29.626 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:16:29.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:29.650 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:29.650 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:16:29.751 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:16:29.774 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:29.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:29.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:16:29.876 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:16:29.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:29.914 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:29.914 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:16:30.016 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:16:30.037 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:30.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:30.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:16:30.139 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:16:30.175 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:30.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:30.176 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:16:30.234 INFO:teuthology.orchestra.run.smithi100.stderr:noscrub is unset 2024-09-17T07:16:30.277 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:16:30.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:30.307 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:30.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:16:30.409 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:16:30.439 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:30.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:30.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:16:30.542 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:16:30.574 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:30.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:30.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:16:30.615 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:30 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:16:30.615 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:30 smithi100 ceph-mon[29092]: pgmap v642: 148 pgs: 36 active+undersized, 28 active+undersized+degraded, 84 active+clean; 1.6 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 1016 B/s rd, 171 MiB/s wr, 2.73k op/s; 6239/76956 objects degraded (8.107%) 2024-09-17T07:16:30.615 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:30 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1550258098' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:16:30.615 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:30 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:16:30.616 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:30 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 6239/76956 objects degraded (8.107%), 28 pgs degraded (PG_DEGRADED) 2024-09-17T07:16:30.616 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:30 smithi100 ceph-mon[29092]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:16:30.616 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:30 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T07:16:30.616 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:30 smithi100 ceph-mon[29092]: osdmap e307: 8 total, 7 up, 7 in 2024-09-17T07:16:30.616 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:30 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:16:30.616 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:30 smithi100 ceph-mon[39462]: pgmap v642: 148 pgs: 36 active+undersized, 28 active+undersized+degraded, 84 active+clean; 1.6 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 1016 B/s rd, 171 MiB/s wr, 2.73k op/s; 6239/76956 objects degraded (8.107%) 2024-09-17T07:16:30.616 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:30 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1550258098' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:16:30.616 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:30 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:16:30.616 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:30 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 6239/76956 objects degraded (8.107%), 28 pgs degraded (PG_DEGRADED) 2024-09-17T07:16:30.616 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:30 smithi100 ceph-mon[39462]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:16:30.617 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:30 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T07:16:30.617 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:30 smithi100 ceph-mon[39462]: osdmap e307: 8 total, 7 up, 7 in 2024-09-17T07:16:30.625 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 12 16 30000 29984 156.147 106.062 0.00637666 0.00639862 2024-09-17T07:16:30.654 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:16:30.676 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:16:30.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:30.712 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:30.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:16:30.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:30 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:16:30.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:30 smithi135 ceph-mon[33238]: pgmap v642: 148 pgs: 36 active+undersized, 28 active+undersized+degraded, 84 active+clean; 1.6 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 1016 B/s rd, 171 MiB/s wr, 2.73k op/s; 6239/76956 objects degraded (8.107%) 2024-09-17T07:16:30.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:30 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1550258098' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:16:30.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:30 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:16:30.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:30 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 6239/76956 objects degraded (8.107%), 28 pgs degraded (PG_DEGRADED) 2024-09-17T07:16:30.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:30 smithi135 ceph-mon[33238]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:16:30.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:30 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T07:16:30.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:30 smithi135 ceph-mon[33238]: osdmap e307: 8 total, 7 up, 7 in 2024-09-17T07:16:30.813 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:16:30.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:30.835 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:30.836 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:16:30.938 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:16:30.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:30.960 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:30.960 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:16:31.061 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:16:31.082 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:31.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:31.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:16:31.184 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:16:31.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:31.218 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:31.218 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:16:31.319 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:16:31.356 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:31.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:31.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:16:31.458 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:16:31.489 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:31.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:31.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:16:31.591 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:16:31.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:31.613 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:31.613 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:16:31.625 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 13 16 31290 31274 150.337 80.625 0.134074 0.00664225 2024-09-17T07:16:31.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:31 smithi100 ceph-mon[29092]: 2.1f scrub starts 2024-09-17T07:16:31.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:31 smithi100 ceph-mon[29092]: 2.1f scrub ok 2024-09-17T07:16:31.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:31 smithi100 ceph-mon[29092]: 6.17 scrub starts 2024-09-17T07:16:31.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:31 smithi100 ceph-mon[29092]: 6.17 scrub ok 2024-09-17T07:16:31.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:31 smithi100 ceph-mon[29092]: pgmap v644: 148 pgs: 36 active+undersized, 28 active+undersized+degraded, 84 active+clean; 1.7 GiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 1.1 KiB/s rd, 207 MiB/s wr, 3.32k op/s; 6866/82347 objects degraded (8.338%) 2024-09-17T07:16:31.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:31 smithi100 ceph-mon[39462]: 2.1f scrub starts 2024-09-17T07:16:31.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:31 smithi100 ceph-mon[39462]: 2.1f scrub ok 2024-09-17T07:16:31.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:31 smithi100 ceph-mon[39462]: 6.17 scrub starts 2024-09-17T07:16:31.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:31 smithi100 ceph-mon[39462]: 6.17 scrub ok 2024-09-17T07:16:31.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:31 smithi100 ceph-mon[39462]: pgmap v644: 148 pgs: 36 active+undersized, 28 active+undersized+degraded, 84 active+clean; 1.7 GiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 1.1 KiB/s rd, 207 MiB/s wr, 3.32k op/s; 6866/82347 objects degraded (8.338%) 2024-09-17T07:16:31.714 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:16:31.743 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:31.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:31.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:16:31.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:31 smithi135 ceph-mon[33238]: 2.1f scrub starts 2024-09-17T07:16:31.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:31 smithi135 ceph-mon[33238]: 2.1f scrub ok 2024-09-17T07:16:31.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:31 smithi135 ceph-mon[33238]: 6.17 scrub starts 2024-09-17T07:16:31.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:31 smithi135 ceph-mon[33238]: 6.17 scrub ok 2024-09-17T07:16:31.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:31 smithi135 ceph-mon[33238]: pgmap v644: 148 pgs: 36 active+undersized, 28 active+undersized+degraded, 84 active+clean; 1.7 GiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 1.1 KiB/s rd, 207 MiB/s wr, 3.32k op/s; 6866/82347 objects degraded (8.338%) 2024-09-17T07:16:31.845 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:16:31.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:31.884 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:31.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:16:31.984 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-17T07:16:31.984 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:16:32.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:32.024 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:32.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:16:32.125 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:16:32.153 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:32.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:32.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:16:32.255 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:16:32.280 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:32.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:32.281 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:16:32.365 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:16:32.382 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:16:32.405 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:32.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:32.406 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:16:32.507 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:16:32.529 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:32.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:32.530 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:16:32.626 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 14 16 33061 33045 147.504 110.688 0.00278332 0.00677375 2024-09-17T07:16:32.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:32 smithi100 ceph-mon[29092]: 2.1b scrub starts 2024-09-17T07:16:32.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:32 smithi100 ceph-mon[29092]: 2.1b scrub ok 2024-09-17T07:16:32.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:32 smithi100 ceph-mon[39462]: 2.1b scrub starts 2024-09-17T07:16:32.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:32 smithi100 ceph-mon[39462]: 2.1b scrub ok 2024-09-17T07:16:32.631 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:16:32.668 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:32.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:32.669 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:16:32.770 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:16:32.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:32 smithi135 ceph-mon[33238]: 2.1b scrub starts 2024-09-17T07:16:32.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:32 smithi135 ceph-mon[33238]: 2.1b scrub ok 2024-09-17T07:16:32.809 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:32.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:32.810 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:16:32.911 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:16:32.947 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:32.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:32.948 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:16:33.049 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:16:33.082 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:33.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:33.083 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:16:33.184 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:16:33.216 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:33.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:33.217 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:16:33.318 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:16:33.360 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:33.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:33.361 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:16:33.462 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:16:33.488 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:33.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:33.489 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:16:33.590 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:16:33.614 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:33.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:33.615 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:16:33.626 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 15 16 34861 34845 145.17 112.5 0.00331991 0.00686875 2024-09-17T07:16:33.716 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:16:33.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:33.751 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:33.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:16:33.852 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:16:33.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:33.887 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:33.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:16:33.888 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:16:33.889 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:16:33.889 INFO:teuthology.orchestra.run.smithi100.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-17T07:16:33.921 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:16:33.921 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:16:33.950 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:16:33.950 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:16:33.980 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:16:33.981 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:16:33.988 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:16:34.007 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:16:34.008 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:16:34.027 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:34.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:34.028 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:16:34.082 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:16:34.082 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:16:34.105 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:16:34.106 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:16:34.128 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:16:34.155 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:34.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:34.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:16:34.257 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:16:34.288 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:34.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:34.289 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:16:34.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:34 smithi135 ceph-mon[33238]: 2.17 scrub starts 2024-09-17T07:16:34.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:34 smithi135 ceph-mon[33238]: 2.17 scrub ok 2024-09-17T07:16:34.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:34 smithi135 ceph-mon[33238]: 4.10 scrub starts 2024-09-17T07:16:34.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:34 smithi135 ceph-mon[33238]: 4.10 scrub ok 2024-09-17T07:16:34.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:34 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:16:34.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:34 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:16:34.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:34 smithi135 ceph-mon[33238]: pgmap v645: 148 pgs: 36 active+undersized, 28 active+undersized+degraded, 84 active+clean; 1.7 GiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 635 B/s rd, 84 MiB/s wr, 1.34k op/s; 6866/82347 objects degraded (8.338%) 2024-09-17T07:16:34.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:34 smithi135 ceph-mon[33238]: 2.b scrub starts 2024-09-17T07:16:34.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:34 smithi135 ceph-mon[33238]: 2.b scrub ok 2024-09-17T07:16:34.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:34 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/413575511' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:16:34.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:34 smithi100 ceph-mon[29092]: 2.17 scrub starts 2024-09-17T07:16:34.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:34 smithi100 ceph-mon[29092]: 2.17 scrub ok 2024-09-17T07:16:34.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:34 smithi100 ceph-mon[29092]: 4.10 scrub starts 2024-09-17T07:16:34.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:34 smithi100 ceph-mon[29092]: 4.10 scrub ok 2024-09-17T07:16:34.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:34 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:16:34.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:34 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:16:34.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:34 smithi100 ceph-mon[29092]: pgmap v645: 148 pgs: 36 active+undersized, 28 active+undersized+degraded, 84 active+clean; 1.7 GiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 635 B/s rd, 84 MiB/s wr, 1.34k op/s; 6866/82347 objects degraded (8.338%) 2024-09-17T07:16:34.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:34 smithi100 ceph-mon[29092]: 2.b scrub starts 2024-09-17T07:16:34.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:34 smithi100 ceph-mon[29092]: 2.b scrub ok 2024-09-17T07:16:34.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:34 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/413575511' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:16:34.379 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:34 smithi100 ceph-mon[39462]: 2.17 scrub starts 2024-09-17T07:16:34.379 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:34 smithi100 ceph-mon[39462]: 2.17 scrub ok 2024-09-17T07:16:34.379 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:34 smithi100 ceph-mon[39462]: 4.10 scrub starts 2024-09-17T07:16:34.379 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:34 smithi100 ceph-mon[39462]: 4.10 scrub ok 2024-09-17T07:16:34.380 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:34 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:16:34.380 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:34 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:16:34.380 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:34 smithi100 ceph-mon[39462]: pgmap v645: 148 pgs: 36 active+undersized, 28 active+undersized+degraded, 84 active+clean; 1.7 GiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 635 B/s rd, 84 MiB/s wr, 1.34k op/s; 6866/82347 objects degraded (8.338%) 2024-09-17T07:16:34.380 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:34 smithi100 ceph-mon[39462]: 2.b scrub starts 2024-09-17T07:16:34.380 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:34 smithi100 ceph-mon[39462]: 2.b scrub ok 2024-09-17T07:16:34.380 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:34 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/413575511' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:16:34.390 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:16:34.428 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:34.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:34.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:16:34.520 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-17T07:16:34.530 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:16:34.599 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:34.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:34.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:16:34.626 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 16 16 36469 36453 142.378 100.5 0.00218263 0.00701573 2024-09-17T07:16:34.702 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:16:34.736 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:34.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:34.738 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:16:34.839 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:16:34.878 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:34.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:34.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:16:34.980 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:16:35.001 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:16:35.002 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:16:35.013 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:35.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:35.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:16:35.115 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:16:35.135 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:16:35.138 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:35.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:35.139 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:16:35.240 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:16:35.265 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:35 smithi100 ceph-mon[29092]: 4.5 scrub starts 2024-09-17T07:16:35.266 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:35 smithi100 ceph-mon[29092]: 4.5 scrub ok 2024-09-17T07:16:35.266 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:35 smithi100 ceph-mon[29092]: 6.14 scrub starts 2024-09-17T07:16:35.266 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:35 smithi100 ceph-mon[29092]: 6.14 scrub ok 2024-09-17T07:16:35.266 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:35 smithi100 ceph-mon[29092]: 4.15 scrub starts 2024-09-17T07:16:35.266 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:35 smithi100 ceph-mon[29092]: 4.15 scrub ok 2024-09-17T07:16:35.266 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:35 smithi100 ceph-mon[39462]: 4.5 scrub starts 2024-09-17T07:16:35.266 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:35 smithi100 ceph-mon[39462]: 4.5 scrub ok 2024-09-17T07:16:35.267 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:35 smithi100 ceph-mon[39462]: 6.14 scrub starts 2024-09-17T07:16:35.267 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:35 smithi100 ceph-mon[39462]: 6.14 scrub ok 2024-09-17T07:16:35.267 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:35 smithi100 ceph-mon[39462]: 4.15 scrub starts 2024-09-17T07:16:35.267 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:35 smithi100 ceph-mon[39462]: 4.15 scrub ok 2024-09-17T07:16:35.278 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:35.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:35.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:16:35.380 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:16:35.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:35.402 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:35.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:16:35.504 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:16:35.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:35.542 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:35.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:16:35.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:35 smithi135 ceph-mon[33238]: 4.5 scrub starts 2024-09-17T07:16:35.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:35 smithi135 ceph-mon[33238]: 4.5 scrub ok 2024-09-17T07:16:35.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:35 smithi135 ceph-mon[33238]: 6.14 scrub starts 2024-09-17T07:16:35.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:35 smithi135 ceph-mon[33238]: 6.14 scrub ok 2024-09-17T07:16:35.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:35 smithi135 ceph-mon[33238]: 4.15 scrub starts 2024-09-17T07:16:35.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:35 smithi135 ceph-mon[33238]: 4.15 scrub ok 2024-09-17T07:16:35.626 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 17 16 38245 38229 140.532 111 0.0104563 0.00711047 2024-09-17T07:16:35.643 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:16:35.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:35.680 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:35.680 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:16:35.781 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:16:35.811 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:35.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:35.812 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:16:35.913 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:16:35.950 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:35.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:35.951 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:16:36.052 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:16:36.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:36.088 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:36.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:16:36.189 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:16:36.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:36.211 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:36.211 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:16:36.309 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:16:35 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:16:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:16:36.310 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:36 smithi100 ceph-mon[39462]: 6.f scrub starts 2024-09-17T07:16:36.310 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:36 smithi100 ceph-mon[39462]: 6.f scrub ok 2024-09-17T07:16:36.310 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:36 smithi100 ceph-mon[39462]: 6.16 scrub starts 2024-09-17T07:16:36.310 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:36 smithi100 ceph-mon[39462]: 6.16 scrub ok 2024-09-17T07:16:36.310 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:36 smithi100 ceph-mon[39462]: 6.a scrub starts 2024-09-17T07:16:36.310 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:36 smithi100 ceph-mon[39462]: 6.a scrub ok 2024-09-17T07:16:36.310 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:36 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 6866/82347 objects degraded (8.338%), 28 pgs degraded (PG_DEGRADED) 2024-09-17T07:16:36.310 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:36 smithi100 ceph-mon[39462]: osdmap e308: 8 total, 7 up, 7 in 2024-09-17T07:16:36.311 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:36 smithi100 ceph-mon[39462]: pgmap v647: 148 pgs: 36 active+undersized, 28 active+undersized+degraded, 84 active+clean; 2.1 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 145 MiB/s wr, 2.32k op/s; 8822/105609 objects degraded (8.353%) 2024-09-17T07:16:36.311 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:36 smithi100 ceph-mon[29092]: 6.f scrub starts 2024-09-17T07:16:36.311 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:36 smithi100 ceph-mon[29092]: 6.f scrub ok 2024-09-17T07:16:36.311 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:36 smithi100 ceph-mon[29092]: 6.16 scrub starts 2024-09-17T07:16:36.311 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:36 smithi100 ceph-mon[29092]: 6.16 scrub ok 2024-09-17T07:16:36.311 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:36 smithi100 ceph-mon[29092]: 6.a scrub starts 2024-09-17T07:16:36.311 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:36 smithi100 ceph-mon[29092]: 6.a scrub ok 2024-09-17T07:16:36.311 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:36 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 6866/82347 objects degraded (8.338%), 28 pgs degraded (PG_DEGRADED) 2024-09-17T07:16:36.311 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:36 smithi100 ceph-mon[29092]: osdmap e308: 8 total, 7 up, 7 in 2024-09-17T07:16:36.311 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:36 smithi100 ceph-mon[29092]: pgmap v647: 148 pgs: 36 active+undersized, 28 active+undersized+degraded, 84 active+clean; 2.1 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 145 MiB/s wr, 2.32k op/s; 8822/105609 objects degraded (8.353%) 2024-09-17T07:16:36.312 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:16:36.349 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:36.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:36.350 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:16:36.451 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:16:36.488 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:36.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:36.489 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:16:36.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:36 smithi135 ceph-mon[33238]: 6.f scrub starts 2024-09-17T07:16:36.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:36 smithi135 ceph-mon[33238]: 6.f scrub ok 2024-09-17T07:16:36.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:36 smithi135 ceph-mon[33238]: 6.16 scrub starts 2024-09-17T07:16:36.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:36 smithi135 ceph-mon[33238]: 6.16 scrub ok 2024-09-17T07:16:36.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:36 smithi135 ceph-mon[33238]: 6.a scrub starts 2024-09-17T07:16:36.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:36 smithi135 ceph-mon[33238]: 6.a scrub ok 2024-09-17T07:16:36.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:36 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 6866/82347 objects degraded (8.338%), 28 pgs degraded (PG_DEGRADED) 2024-09-17T07:16:36.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:36 smithi135 ceph-mon[33238]: osdmap e308: 8 total, 7 up, 7 in 2024-09-17T07:16:36.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:36 smithi135 ceph-mon[33238]: pgmap v647: 148 pgs: 36 active+undersized, 28 active+undersized+degraded, 84 active+clean; 2.1 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 145 MiB/s wr, 2.32k op/s; 8822/105609 objects degraded (8.353%) 2024-09-17T07:16:36.589 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:16:36.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:36.620 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:36.620 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:16:36.626 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 18 16 39801 39785 138.127 97.25 0.0031592 0.00723039 2024-09-17T07:16:36.722 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:16:36.757 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:36.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:36.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:16:36.859 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:16:36.882 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:36.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:36.883 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:16:36.984 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:16:37.003 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:16:37.020 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:37.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:37.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:16:37.122 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:16:37.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:37.156 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:37.157 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:16:37.257 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:16:37.289 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:37.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:37.290 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:16:37.362 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:16:37.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:37 smithi100 ceph-mon[29092]: 5.a scrub starts 2024-09-17T07:16:37.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:37 smithi100 ceph-mon[29092]: 5.a scrub ok 2024-09-17T07:16:37.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:37 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:16:37.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:37 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:16:37.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:37 smithi100 ceph-mon[39462]: 5.a scrub starts 2024-09-17T07:16:37.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:37 smithi100 ceph-mon[39462]: 5.a scrub ok 2024-09-17T07:16:37.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:37 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:16:37.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:37 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:16:37.390 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:16:37.442 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:37.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:37.443 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:16:37.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:37 smithi135 ceph-mon[33238]: 5.a scrub starts 2024-09-17T07:16:37.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:37 smithi135 ceph-mon[33238]: 5.a scrub ok 2024-09-17T07:16:37.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:37 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:16:37.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:37 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:16:37.544 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:16:37.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:37.570 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:37.570 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:16:37.626 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 19 16 41104 41088 135.143 81.4375 0.00275377 0.00735587 2024-09-17T07:16:37.672 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:16:37.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:37.695 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:37.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:16:37.796 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:16:37.841 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:37.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:37.842 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:16:37.942 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:16:37.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:37.987 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:37.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:16:38.087 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:16:38.131 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:38.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:38.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:16:38.233 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:16:38.255 INFO:teuthology.orchestra.run.smithi100.stderr:nodeep-scrub is unset 2024-09-17T07:16:38.267 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:38.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:38.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:16:38.369 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:16:38.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:38 smithi100 ceph-mon[29092]: 6.2 scrub starts 2024-09-17T07:16:38.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:38 smithi100 ceph-mon[29092]: 6.2 scrub ok 2024-09-17T07:16:38.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:38 smithi100 ceph-mon[29092]: 2.16 scrub starts 2024-09-17T07:16:38.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:38 smithi100 ceph-mon[29092]: 2.16 scrub ok 2024-09-17T07:16:38.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:38 smithi100 ceph-mon[29092]: osdmap e309: 8 total, 7 up, 7 in 2024-09-17T07:16:38.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:38 smithi100 ceph-mon[29092]: pgmap v649: 148 pgs: 36 active+undersized, 28 active+undersized+degraded, 84 active+clean; 2.1 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 725 B/s rd, 69 MiB/s wr, 1.10k op/s; 8822/105609 objects degraded (8.353%) 2024-09-17T07:16:38.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:38 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2471334432' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:16:38.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:38 smithi100 ceph-mon[29092]: 6.c scrub starts 2024-09-17T07:16:38.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:38 smithi100 ceph-mon[29092]: 6.c scrub ok 2024-09-17T07:16:38.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:38 smithi100 ceph-mon[39462]: 6.2 scrub starts 2024-09-17T07:16:38.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:38 smithi100 ceph-mon[39462]: 6.2 scrub ok 2024-09-17T07:16:38.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:38 smithi100 ceph-mon[39462]: 2.16 scrub starts 2024-09-17T07:16:38.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:38 smithi100 ceph-mon[39462]: 2.16 scrub ok 2024-09-17T07:16:38.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:38 smithi100 ceph-mon[39462]: osdmap e309: 8 total, 7 up, 7 in 2024-09-17T07:16:38.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:38 smithi100 ceph-mon[39462]: pgmap v649: 148 pgs: 36 active+undersized, 28 active+undersized+degraded, 84 active+clean; 2.1 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 725 B/s rd, 69 MiB/s wr, 1.10k op/s; 8822/105609 objects degraded (8.353%) 2024-09-17T07:16:38.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:38 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2471334432' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:16:38.379 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:38 smithi100 ceph-mon[39462]: 6.c scrub starts 2024-09-17T07:16:38.379 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:38 smithi100 ceph-mon[39462]: 6.c scrub ok 2024-09-17T07:16:38.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:38.408 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:38.408 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:16:38.444 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:38 smithi135 ceph-mon[33238]: 6.2 scrub starts 2024-09-17T07:16:38.445 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:38 smithi135 ceph-mon[33238]: 6.2 scrub ok 2024-09-17T07:16:38.445 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:38 smithi135 ceph-mon[33238]: 2.16 scrub starts 2024-09-17T07:16:38.445 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:38 smithi135 ceph-mon[33238]: 2.16 scrub ok 2024-09-17T07:16:38.445 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:38 smithi135 ceph-mon[33238]: osdmap e309: 8 total, 7 up, 7 in 2024-09-17T07:16:38.445 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:38 smithi135 ceph-mon[33238]: pgmap v649: 148 pgs: 36 active+undersized, 28 active+undersized+degraded, 84 active+clean; 2.1 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 725 B/s rd, 69 MiB/s wr, 1.10k op/s; 8822/105609 objects degraded (8.353%) 2024-09-17T07:16:38.445 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:38 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2471334432' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:16:38.445 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:38 smithi135 ceph-mon[33238]: 6.c scrub starts 2024-09-17T07:16:38.446 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:38 smithi135 ceph-mon[33238]: 6.c scrub ok 2024-09-17T07:16:38.508 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:16:38.542 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:38.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:38.543 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:16:38.626 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:2024-09-17T07:16:38.626727+0000 min lat: 0.00112621 max lat: 0.332984 avg lat: 0.00751705 2024-09-17T07:16:38.626 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-17T07:16:38.626 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 20 16 42565 42549 132.951 91.3125 0.0076262 0.00751705 2024-09-17T07:16:38.644 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:16:38.681 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:38.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:38.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:16:38.783 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:16:38.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:38.806 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:38.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:16:38.907 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:16:38.947 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:38.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:38.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:16:39.050 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:16:39.088 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:39.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:39.090 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:16:39.093 INFO:teuthology.orchestra.run.smithi100.stdout:6.d 2024-09-17T07:16:39.094 INFO:teuthology.orchestra.run.smithi100.stdout:6.b 2024-09-17T07:16:39.094 INFO:teuthology.orchestra.run.smithi100.stdout:6.4 2024-09-17T07:16:39.094 INFO:teuthology.orchestra.run.smithi100.stdout:4.4 2024-09-17T07:16:39.094 INFO:teuthology.orchestra.run.smithi100.stdout:6.19 2024-09-17T07:16:39.094 INFO:teuthology.orchestra.run.smithi100.stdout:4.12 2024-09-17T07:16:39.095 INFO:teuthology.orchestra.run.smithi100.stdout:4.0 2024-09-17T07:16:39.095 INFO:teuthology.orchestra.run.smithi100.stdout:2.f 2024-09-17T07:16:39.095 INFO:teuthology.orchestra.run.smithi100.stdout:2.0 2024-09-17T07:16:39.095 INFO:teuthology.orchestra.run.smithi100.stdout:5.13 2024-09-17T07:16:39.095 INFO:teuthology.orchestra.run.smithi100.stdout:4.18 2024-09-17T07:16:39.095 INFO:teuthology.orchestra.run.smithi100.stdout:2.1 2024-09-17T07:16:39.095 INFO:teuthology.orchestra.run.smithi100.stdout:4.13 2024-09-17T07:16:39.095 INFO:teuthology.orchestra.run.smithi100.stdout:5.14 2024-09-17T07:16:39.095 INFO:teuthology.orchestra.run.smithi100.stdout:2.9 2024-09-17T07:16:39.096 INFO:teuthology.orchestra.run.smithi100.stdout:10.0 2024-09-17T07:16:39.096 INFO:teuthology.orchestra.run.smithi100.stdout:2.a 2024-09-17T07:16:39.096 INFO:teuthology.orchestra.run.smithi100.stdout:5.15 2024-09-17T07:16:39.096 INFO:teuthology.orchestra.run.smithi100.stdout:5.1a 2024-09-17T07:16:39.096 INFO:teuthology.orchestra.run.smithi100.stdout:5.7 2024-09-17T07:16:39.096 INFO:teuthology.orchestra.run.smithi100.stdout:10.2 2024-09-17T07:16:39.096 INFO:teuthology.orchestra.run.smithi100.stdout:4.1b 2024-09-17T07:16:39.096 INFO:teuthology.orchestra.run.smithi100.stdout:4.2 2024-09-17T07:16:39.096 INFO:teuthology.orchestra.run.smithi100.stdout:6.10 2024-09-17T07:16:39.096 INFO:teuthology.orchestra.run.smithi100.stdout:2.10 2024-09-17T07:16:39.097 INFO:teuthology.orchestra.run.smithi100.stdout:4.f 2024-09-17T07:16:39.097 INFO:teuthology.orchestra.run.smithi100.stdout:6.e 2024-09-17T07:16:39.097 INFO:teuthology.orchestra.run.smithi100.stdout:10.4 2024-09-17T07:16:39.097 INFO:teuthology.orchestra.run.smithi100.stdout:4.7 2024-09-17T07:16:39.097 INFO:teuthology.orchestra.run.smithi100.stdout:2.4 2024-09-17T07:16:39.097 INFO:teuthology.orchestra.run.smithi100.stdout:4.e 2024-09-17T07:16:39.097 INFO:teuthology.orchestra.run.smithi100.stdout:2.11 2024-09-17T07:16:39.097 INFO:teuthology.orchestra.run.smithi100.stdout:5.1d 2024-09-17T07:16:39.098 INFO:teuthology.orchestra.run.smithi100.stdout:4.1c 2024-09-17T07:16:39.098 INFO:teuthology.orchestra.run.smithi100.stdout:10.b 2024-09-17T07:16:39.098 INFO:teuthology.orchestra.run.smithi100.stdout:5.18 2024-09-17T07:16:39.098 INFO:teuthology.orchestra.run.smithi100.stdout:2.15 2024-09-17T07:16:39.098 INFO:teuthology.orchestra.run.smithi100.stdout:4.17 2024-09-17T07:16:39.098 INFO:teuthology.orchestra.run.smithi100.stdout:5.0 2024-09-17T07:16:39.098 INFO:teuthology.orchestra.run.smithi100.stdout:2.8 2024-09-17T07:16:39.099 INFO:teuthology.orchestra.run.smithi100.stdout:3.0 2024-09-17T07:16:39.099 INFO:teuthology.orchestra.run.smithi100.stdout:5.19 2024-09-17T07:16:39.099 INFO:teuthology.orchestra.run.smithi100.stdout:5.5 2024-09-17T07:16:39.099 INFO:teuthology.orchestra.run.smithi100.stdout:4.14 2024-09-17T07:16:39.099 INFO:teuthology.orchestra.run.smithi100.stdout:2.d 2024-09-17T07:16:39.099 INFO:teuthology.orchestra.run.smithi100.stdout:6.1 2024-09-17T07:16:39.099 INFO:teuthology.orchestra.run.smithi100.stdout:2.6 2024-09-17T07:16:39.099 INFO:teuthology.orchestra.run.smithi100.stdout:4.a 2024-09-17T07:16:39.100 INFO:teuthology.orchestra.run.smithi100.stdout:4.b 2024-09-17T07:16:39.100 INFO:teuthology.orchestra.run.smithi100.stdout:4.c 2024-09-17T07:16:39.100 INFO:teuthology.orchestra.run.smithi100.stdout:4.1d 2024-09-17T07:16:39.100 INFO:teuthology.orchestra.run.smithi100.stdout:4.d 2024-09-17T07:16:39.100 INFO:teuthology.orchestra.run.smithi100.stdout:6.7 2024-09-17T07:16:39.100 INFO:teuthology.orchestra.run.smithi100.stdout:5.12 2024-09-17T07:16:39.100 INFO:teuthology.orchestra.run.smithi100.stdout:5.16 2024-09-17T07:16:39.100 INFO:teuthology.orchestra.run.smithi100.stdout:5.17 2024-09-17T07:16:39.101 INFO:teuthology.orchestra.run.smithi100.stdout:5.8 2024-09-17T07:16:39.101 INFO:teuthology.orchestra.run.smithi100.stdout:5.c 2024-09-17T07:16:39.101 INFO:teuthology.orchestra.run.smithi100.stdout:2.2 2024-09-17T07:16:39.101 INFO:teuthology.orchestra.run.smithi100.stdout:6.18 2024-09-17T07:16:39.101 INFO:teuthology.orchestra.run.smithi100.stdout:6.1a 2024-09-17T07:16:39.101 INFO:teuthology.orchestra.run.smithi100.stdout:6.1b 2024-09-17T07:16:39.101 INFO:teuthology.orchestra.run.smithi100.stdout:6.1d 2024-09-17T07:16:39.102 INFO:teuthology.orchestra.run.smithi100.stdout:6.1f 2024-09-17T07:16:39.191 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:16:39.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:39.231 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:39.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:16:39.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:39 smithi135 ceph-mon[33238]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T07:16:39.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:39 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2471334432' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T07:16:39.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:39 smithi135 ceph-mon[33238]: osdmap e310: 8 total, 7 up, 7 in 2024-09-17T07:16:39.294 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:16:38 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:16:39.332 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:16:39.366 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:39.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:39.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:16:39.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:39 smithi100 ceph-mon[29092]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T07:16:39.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:39 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2471334432' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T07:16:39.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:39 smithi100 ceph-mon[29092]: osdmap e310: 8 total, 7 up, 7 in 2024-09-17T07:16:39.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:39 smithi100 ceph-mon[39462]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T07:16:39.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:39 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2471334432' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T07:16:39.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:39 smithi100 ceph-mon[39462]: osdmap e310: 8 total, 7 up, 7 in 2024-09-17T07:16:39.468 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:16:39.507 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:39.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:39.508 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:16:39.609 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:16:39.626 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 21 16 44155 44139 131.351 99.375 0.00851349 0.00760882 2024-09-17T07:16:39.642 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:39.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:39.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:16:39.744 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:16:39.784 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:39.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:39.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:16:39.885 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:16:39.912 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:39.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:39.913 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:16:40.014 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:16:40.056 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:40.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:40.057 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:16:40.161 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:16:40.199 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:40.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:40.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:16:40.301 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:16:40.329 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:40.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:40.330 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:16:40.341 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:40 smithi100 ceph-mon[39462]: 5.1 scrub starts 2024-09-17T07:16:40.341 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:40 smithi100 ceph-mon[39462]: 5.1 scrub ok 2024-09-17T07:16:40.341 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:40 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:16:40.341 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:40 smithi100 ceph-mon[39462]: 2.18 scrub starts 2024-09-17T07:16:40.341 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:40 smithi100 ceph-mon[39462]: 2.18 scrub ok 2024-09-17T07:16:40.341 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:40 smithi100 ceph-mon[39462]: pgmap v651: 148 pgs: 36 active+undersized, 28 active+undersized+degraded, 84 active+clean; 2.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.7 KiB/s rd, 145 MiB/s wr, 2.32k op/s; 10252/124035 objects degraded (8.265%) 2024-09-17T07:16:40.341 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:40 smithi100 ceph-mon[29092]: 5.1 scrub starts 2024-09-17T07:16:40.341 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:40 smithi100 ceph-mon[29092]: 5.1 scrub ok 2024-09-17T07:16:40.342 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:40 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:16:40.342 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:40 smithi100 ceph-mon[29092]: 2.18 scrub starts 2024-09-17T07:16:40.342 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:40 smithi100 ceph-mon[29092]: 2.18 scrub ok 2024-09-17T07:16:40.342 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:40 smithi100 ceph-mon[29092]: pgmap v651: 148 pgs: 36 active+undersized, 28 active+undersized+degraded, 84 active+clean; 2.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.7 KiB/s rd, 145 MiB/s wr, 2.32k op/s; 10252/124035 objects degraded (8.265%) 2024-09-17T07:16:40.432 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:16:40.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:40.469 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:40.469 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:16:40.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:40 smithi135 ceph-mon[33238]: 5.1 scrub starts 2024-09-17T07:16:40.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:40 smithi135 ceph-mon[33238]: 5.1 scrub ok 2024-09-17T07:16:40.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:40 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:16:40.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:40 smithi135 ceph-mon[33238]: 2.18 scrub starts 2024-09-17T07:16:40.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:40 smithi135 ceph-mon[33238]: 2.18 scrub ok 2024-09-17T07:16:40.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:40 smithi135 ceph-mon[33238]: pgmap v651: 148 pgs: 36 active+undersized, 28 active+undersized+degraded, 84 active+clean; 2.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.7 KiB/s rd, 145 MiB/s wr, 2.32k op/s; 10252/124035 objects degraded (8.265%) 2024-09-17T07:16:40.570 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:16:40.598 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:40.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:40.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:16:40.626 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 22 16 45533 45517 129.295 86.125 0.00338207 0.00772946 2024-09-17T07:16:40.700 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:16:40.727 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:40.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:40.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:16:40.828 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:16:40.864 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:40.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:40.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:16:40.967 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:16:40.992 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:40.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:40.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:16:41.094 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:16:41.132 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:41.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:41.133 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:16:41.234 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:16:41.262 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:41.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:41.264 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:16:41.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:41 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:16:41.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:41 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:16:41.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:41 smithi135 ceph-mon[33238]: osdmap e311: 8 total, 7 up, 7 in 2024-09-17T07:16:41.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:41 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 10252/124035 objects degraded (8.265%), 28 pgs degraded (PG_DEGRADED) 2024-09-17T07:16:41.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:41 smithi135 ceph-mon[33238]: osdmap e312: 8 total, 7 up, 7 in 2024-09-17T07:16:41.331 INFO:teuthology.orchestra.run.smithi135.stdout:{ 2024-09-17T07:16:41.331 INFO:teuthology.orchestra.run.smithi135.stdout: "ops": [], 2024-09-17T07:16:41.331 INFO:teuthology.orchestra.run.smithi135.stdout: "complaint_time": 30, 2024-09-17T07:16:41.331 INFO:teuthology.orchestra.run.smithi135.stdout: "num_blocked_ops": 0 2024-09-17T07:16:41.331 INFO:teuthology.orchestra.run.smithi135.stdout:} 2024-09-17T07:16:41.364 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:16:41.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:41 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:16:41.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:41 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:16:41.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:41 smithi100 ceph-mon[29092]: osdmap e311: 8 total, 7 up, 7 in 2024-09-17T07:16:41.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:41 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 10252/124035 objects degraded (8.265%), 28 pgs degraded (PG_DEGRADED) 2024-09-17T07:16:41.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:41 smithi100 ceph-mon[29092]: osdmap e312: 8 total, 7 up, 7 in 2024-09-17T07:16:41.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:41 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:16:41.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:41 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:16:41.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:41 smithi100 ceph-mon[39462]: osdmap e311: 8 total, 7 up, 7 in 2024-09-17T07:16:41.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:41 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 10252/124035 objects degraded (8.265%), 28 pgs degraded (PG_DEGRADED) 2024-09-17T07:16:41.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:41 smithi100 ceph-mon[39462]: osdmap e312: 8 total, 7 up, 7 in 2024-09-17T07:16:41.395 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:41.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:41.396 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:16:41.420 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:16:41.420 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:16:41.420 INFO:teuthology.orchestra.run.smithi100.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-17T07:16:41.445 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:16:41.445 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:16:41.467 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:16:41.467 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:16:41.497 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:16:41.498 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:16:41.498 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:16:41.521 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:16:41.522 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:16:41.544 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:41.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:41.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:16:41.550 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:16:41.551 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:16:41.573 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:16:41.574 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:16:41.626 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 23 16 46973 46957 127.586 90 0.0022391 0.00780999 2024-09-17T07:16:41.646 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:16:41.667 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:41.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:41.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:16:41.769 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:16:41.808 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:41.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:41.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:16:41.911 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:16:41.954 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:41.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:41.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:16:42.056 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:16:42.085 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:42.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:42.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:16:42.187 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:16:42.216 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:42.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:42.217 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:16:42.318 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:16:42.358 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:42.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:42.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:16:42.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:42 smithi100 ceph-mon[29092]: pgmap v654: 148 pgs: 36 active+undersized, 28 active+undersized+degraded, 84 active+clean; 2.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 117 MiB/s wr, 1.88k op/s; 10735/128421 objects degraded (8.359%) 2024-09-17T07:16:42.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:42 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/474455685' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:16:42.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:42 smithi100 ceph-mon[39462]: pgmap v654: 148 pgs: 36 active+undersized, 28 active+undersized+degraded, 84 active+clean; 2.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 117 MiB/s wr, 1.88k op/s; 10735/128421 objects degraded (8.359%) 2024-09-17T07:16:42.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:42 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/474455685' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:16:42.459 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:16:42.496 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:42.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:42.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:16:42.505 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -n osd.1 --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --op export --pgid 6.1 --file /var/log/ceph/exp.6.1.1 2024-09-17T07:16:42.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:42 smithi135 ceph-mon[33238]: pgmap v654: 148 pgs: 36 active+undersized, 28 active+undersized+degraded, 84 active+clean; 2.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 117 MiB/s wr, 1.88k op/s; 10735/128421 objects degraded (8.359%) 2024-09-17T07:16:42.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:42 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/474455685' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:16:42.598 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:16:42.627 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 24 16 48455 48439 126.129 92.625 0.0125178 0.00792259 2024-09-17T07:16:42.642 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-17T07:16:42.646 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:42.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:42.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:16:42.748 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:16:42.775 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:42.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:42.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:16:42.838 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-17T07:16:42.877 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:16:42.909 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:42.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:42.911 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:16:43.012 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:16:43.021 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:16:43.022 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:16:43.028 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/osd.1/config 2024-09-17T07:16:43.071 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:16:43.076 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:43.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:43.077 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:16:43.178 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:16:43.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:43.222 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:43.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:16:43.323 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:16:43.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:43.359 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:43.359 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:16:43.460 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:16:43.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:43.490 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:43.490 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:16:43.591 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:16:43.627 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 25 16 50036 50020 125.036 98.8125 0.00705996 0.00799296 2024-09-17T07:16:43.627 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:43.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:43.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:16:43.729 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:16:43.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:43.766 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:43.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:16:43.810 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:16:43.867 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:16:43.904 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:43.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:43.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:16:44.006 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:16:44.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:44.034 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:44.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:16:44.135 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:16:44.163 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:44.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:44.164 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:16:44.264 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:16:44.297 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:44.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:44.298 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:16:44.399 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:16:44.436 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:44.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:44.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:16:44.538 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:16:44.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:44 smithi135 ceph-mon[33238]: pgmap v655: 148 pgs: 36 active+undersized, 28 active+undersized+degraded, 84 active+clean; 2.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 853 B/s rd, 79 MiB/s wr, 1.27k op/s; 10735/128421 objects degraded (8.359%) 2024-09-17T07:16:44.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:44 smithi135 ceph-mon[33238]: 4.6 scrub starts 2024-09-17T07:16:44.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:44 smithi135 ceph-mon[33238]: 4.6 scrub ok 2024-09-17T07:16:44.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:44 smithi135 ceph-mon[33238]: 4.8 scrub starts 2024-09-17T07:16:44.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:44 smithi135 ceph-mon[33238]: 4.8 scrub ok 2024-09-17T07:16:44.562 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:44.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:44.563 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:16:44.624 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:44 smithi100 ceph-mon[39462]: pgmap v655: 148 pgs: 36 active+undersized, 28 active+undersized+degraded, 84 active+clean; 2.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 853 B/s rd, 79 MiB/s wr, 1.27k op/s; 10735/128421 objects degraded (8.359%) 2024-09-17T07:16:44.624 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:44 smithi100 ceph-mon[39462]: 4.6 scrub starts 2024-09-17T07:16:44.625 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:44 smithi100 ceph-mon[39462]: 4.6 scrub ok 2024-09-17T07:16:44.625 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:44 smithi100 ceph-mon[39462]: 4.8 scrub starts 2024-09-17T07:16:44.625 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:44 smithi100 ceph-mon[39462]: 4.8 scrub ok 2024-09-17T07:16:44.625 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:44 smithi100 ceph-mon[29092]: pgmap v655: 148 pgs: 36 active+undersized, 28 active+undersized+degraded, 84 active+clean; 2.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 853 B/s rd, 79 MiB/s wr, 1.27k op/s; 10735/128421 objects degraded (8.359%) 2024-09-17T07:16:44.625 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:44 smithi100 ceph-mon[29092]: 4.6 scrub starts 2024-09-17T07:16:44.625 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:44 smithi100 ceph-mon[29092]: 4.6 scrub ok 2024-09-17T07:16:44.625 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:44 smithi100 ceph-mon[29092]: 4.8 scrub starts 2024-09-17T07:16:44.625 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:44 smithi100 ceph-mon[29092]: 4.8 scrub ok 2024-09-17T07:16:44.626 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 26 16 51616 51600 124.025 98.75 0.00229248 0.00804431 2024-09-17T07:16:44.664 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:16:44.690 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:44.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:44.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:16:44.792 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:16:44.822 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:44.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:44.823 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:16:44.924 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:16:44.951 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:44.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:44.952 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:16:45.023 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:16:45.052 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:16:45.100 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:45.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:45.102 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:16:45.203 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:16:45.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:45.239 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:45.239 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:16:45.340 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:16:45.370 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:45.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:45.372 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:16:45.416 INFO:teuthology.orchestra.run.smithi100.stderr:Exporting 6.1 info 6.1( empty local-lis/les=73/74 n=0 ec=73/73 lis/c=73/73 les/c/f=74/74/0 sis=73) 2024-09-17T07:16:45.424 INFO:teuthology.orchestra.run.smithi100.stderr:Export successful 2024-09-17T07:16:45.473 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:16:45.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:45.511 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:45.511 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:16:45.612 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:16:45.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:45 smithi100 ceph-mon[39462]: 4.1a deep-scrub starts 2024-09-17T07:16:45.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:45 smithi100 ceph-mon[39462]: 4.1a deep-scrub ok 2024-09-17T07:16:45.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:45 smithi100 ceph-mon[39462]: 2.1a scrub starts 2024-09-17T07:16:45.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:45 smithi100 ceph-mon[39462]: 2.1a scrub ok 2024-09-17T07:16:45.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:45 smithi100 ceph-mon[39462]: 5.b scrub starts 2024-09-17T07:16:45.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:45 smithi100 ceph-mon[39462]: 5.b scrub ok 2024-09-17T07:16:45.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:45 smithi100 ceph-mon[39462]: 5.1b scrub starts 2024-09-17T07:16:45.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:45 smithi100 ceph-mon[39462]: 5.1b scrub ok 2024-09-17T07:16:45.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:45 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 10735/128421 objects degraded (8.359%), 28 pgs degraded (PG_DEGRADED) 2024-09-17T07:16:45.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:45 smithi100 ceph-mon[29092]: 4.1a deep-scrub starts 2024-09-17T07:16:45.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:45 smithi100 ceph-mon[29092]: 4.1a deep-scrub ok 2024-09-17T07:16:45.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:45 smithi100 ceph-mon[29092]: 2.1a scrub starts 2024-09-17T07:16:45.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:45 smithi100 ceph-mon[29092]: 2.1a scrub ok 2024-09-17T07:16:45.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:45 smithi100 ceph-mon[29092]: 5.b scrub starts 2024-09-17T07:16:45.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:45 smithi100 ceph-mon[29092]: 5.b scrub ok 2024-09-17T07:16:45.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:45 smithi100 ceph-mon[29092]: 5.1b scrub starts 2024-09-17T07:16:45.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:45 smithi100 ceph-mon[29092]: 5.1b scrub ok 2024-09-17T07:16:45.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:45 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 10735/128421 objects degraded (8.359%), 28 pgs degraded (PG_DEGRADED) 2024-09-17T07:16:45.628 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 27 16 53003 52987 122.642 86.6875 0.00329537 0.00813718 2024-09-17T07:16:45.633 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:45.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:45.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:16:45.735 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:16:45.757 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:45.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:45.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:16:45.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:45 smithi135 ceph-mon[33238]: 4.1a deep-scrub starts 2024-09-17T07:16:45.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:45 smithi135 ceph-mon[33238]: 4.1a deep-scrub ok 2024-09-17T07:16:45.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:45 smithi135 ceph-mon[33238]: 2.1a scrub starts 2024-09-17T07:16:45.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:45 smithi135 ceph-mon[33238]: 2.1a scrub ok 2024-09-17T07:16:45.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:45 smithi135 ceph-mon[33238]: 5.b scrub starts 2024-09-17T07:16:45.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:45 smithi135 ceph-mon[33238]: 5.b scrub ok 2024-09-17T07:16:45.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:45 smithi135 ceph-mon[33238]: 5.1b scrub starts 2024-09-17T07:16:45.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:45 smithi135 ceph-mon[33238]: 5.1b scrub ok 2024-09-17T07:16:45.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:45 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 10735/128421 objects degraded (8.359%), 28 pgs degraded (PG_DEGRADED) 2024-09-17T07:16:45.860 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:16:45.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:45.882 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:45.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:16:45.983 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:16:46.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:46.021 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:46.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:16:46.121 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:16:46.158 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:46.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:46.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:16:46.169 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:16:45 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:16:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:16:46.260 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:16:46.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:46.282 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:46.282 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:16:46.383 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:16:46.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:46.405 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:46.405 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:16:46.506 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:16:46.540 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:46.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:46.541 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:16:46.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:46 smithi100 ceph-mon[29092]: pgmap v656: 148 pgs: 36 active+undersized, 28 active+undersized+degraded, 84 active+clean; 3.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.4 KiB/s rd, 135 MiB/s wr, 2.15k op/s; 12631/151092 objects degraded (8.360%) 2024-09-17T07:16:46.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:46 smithi100 ceph-mon[39462]: pgmap v656: 148 pgs: 36 active+undersized, 28 active+undersized+degraded, 84 active+clean; 3.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.4 KiB/s rd, 135 MiB/s wr, 2.15k op/s; 12631/151092 objects degraded (8.360%) 2024-09-17T07:16:46.628 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 28 16 53705 53689 119.828 43.875 0.00791959 0.0083403 2024-09-17T07:16:46.642 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:16:46.668 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:46.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:46.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:16:46.770 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:16:46.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:46.792 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:46.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:16:46.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:46 smithi135 ceph-mon[33238]: pgmap v656: 148 pgs: 36 active+undersized, 28 active+undersized+degraded, 84 active+clean; 3.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.4 KiB/s rd, 135 MiB/s wr, 2.15k op/s; 12631/151092 objects degraded (8.360%) 2024-09-17T07:16:46.894 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:16:46.932 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:46.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:46.934 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:16:47.034 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:16:47.065 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:47.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:47.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:16:47.168 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:16:47.195 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:47.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:47.196 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:16:47.297 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:16:47.335 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:47.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:47.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:16:47.438 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:16:47.470 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:47.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:47.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:16:47.572 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:16:47.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:47.602 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:47.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:16:47.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:47 smithi100 ceph-mon[29092]: 5.6 scrub starts 2024-09-17T07:16:47.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:47 smithi100 ceph-mon[29092]: 5.6 scrub ok 2024-09-17T07:16:47.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:47 smithi100 ceph-mon[39462]: 5.6 scrub starts 2024-09-17T07:16:47.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:47 smithi100 ceph-mon[39462]: 5.6 scrub ok 2024-09-17T07:16:47.627 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 29 16 54986 54970 118.457 80.0625 0.0104345 0.00843712 2024-09-17T07:16:47.669 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:47 smithi135 ceph-mon[33238]: 5.6 scrub starts 2024-09-17T07:16:47.669 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:47 smithi135 ceph-mon[33238]: 5.6 scrub ok 2024-09-17T07:16:47.703 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:16:47.728 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:47.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:47.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:16:47.830 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:16:47.845 INFO:teuthology.orchestra.run.smithi135.stdout:ERROR: (22) Invalid argument 2024-09-17T07:16:47.846 INFO:teuthology.orchestra.run.smithi135.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:16:47.853 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -n osd.1 --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --force --op remove --pgid 6.1 2024-09-17T07:16:47.864 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:47.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:47.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:16:47.966 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:16:48.024 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:48.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:48.026 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:16:48.126 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:16:48.162 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:48.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:48.163 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:16:48.264 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:16:48.309 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:48.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:48.310 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:16:48.356 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:16:48.399 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/osd.1/config 2024-09-17T07:16:48.410 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:16:48.441 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:48.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:48.443 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:16:48.536 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-17T07:16:48.544 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:16:48.577 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:48.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:48.579 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:16:48.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:48 smithi100 ceph-mon[29092]: 6.0 scrub starts 2024-09-17T07:16:48.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:48 smithi100 ceph-mon[29092]: 6.0 scrub ok 2024-09-17T07:16:48.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:48 smithi100 ceph-mon[29092]: pgmap v657: 148 pgs: 36 active+undersized, 28 active+undersized+degraded, 84 active+clean; 3.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 639 B/s rd, 70 MiB/s wr, 1.13k op/s; 12631/151092 objects degraded (8.360%) 2024-09-17T07:16:48.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:48 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:16:48.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:48 smithi100 ceph-mon[29092]: 2.5 deep-scrub starts 2024-09-17T07:16:48.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:48 smithi100 ceph-mon[29092]: 2.5 deep-scrub ok 2024-09-17T07:16:48.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:48 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:16:48.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:48 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:16:48.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:48 smithi100 ceph-mon[39462]: 6.0 scrub starts 2024-09-17T07:16:48.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:48 smithi100 ceph-mon[39462]: 6.0 scrub ok 2024-09-17T07:16:48.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:48 smithi100 ceph-mon[39462]: pgmap v657: 148 pgs: 36 active+undersized, 28 active+undersized+degraded, 84 active+clean; 3.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 639 B/s rd, 70 MiB/s wr, 1.13k op/s; 12631/151092 objects degraded (8.360%) 2024-09-17T07:16:48.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:48 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:16:48.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:48 smithi100 ceph-mon[39462]: 2.5 deep-scrub starts 2024-09-17T07:16:48.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:48 smithi100 ceph-mon[39462]: 2.5 deep-scrub ok 2024-09-17T07:16:48.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:48 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:16:48.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:48 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:16:48.628 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 30 16 56383 56367 117.419 87.3125 0.00981428 0.00851099 2024-09-17T07:16:48.680 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:16:48.714 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:48.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:48.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:16:48.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:48 smithi135 ceph-mon[33238]: 6.0 scrub starts 2024-09-17T07:16:48.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:48 smithi135 ceph-mon[33238]: 6.0 scrub ok 2024-09-17T07:16:48.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:48 smithi135 ceph-mon[33238]: pgmap v657: 148 pgs: 36 active+undersized, 28 active+undersized+degraded, 84 active+clean; 3.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 639 B/s rd, 70 MiB/s wr, 1.13k op/s; 12631/151092 objects degraded (8.360%) 2024-09-17T07:16:48.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:48 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:16:48.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:48 smithi135 ceph-mon[33238]: 2.5 deep-scrub starts 2024-09-17T07:16:48.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:48 smithi135 ceph-mon[33238]: 2.5 deep-scrub ok 2024-09-17T07:16:48.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:48 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:16:48.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:48 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:16:48.815 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:16:48.849 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:48.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:48.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:16:48.903 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:16:48.951 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:16:48.982 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:48.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:48.983 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:16:49.084 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:16:49.113 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:49.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:49.114 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:16:49.210 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:16:48 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:16:49.215 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:16:49.257 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:49.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:49.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:16:49.359 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:16:49.409 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:49.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:49.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:16:49.511 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:16:49.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:49 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:16:49.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:49 smithi135 ceph-mon[33238]: pgmap v658: 148 pgs: 36 active+undersized, 28 active+undersized+degraded, 84 active+clean; 3.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.1 KiB/s rd, 86 MiB/s wr, 1.38k op/s; 13821/166608 objects degraded (8.296%) 2024-09-17T07:16:49.551 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:49.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:49.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:16:49.629 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:49 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:16:49.629 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:49 smithi100 ceph-mon[29092]: pgmap v658: 148 pgs: 36 active+undersized, 28 active+undersized+degraded, 84 active+clean; 3.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.1 KiB/s rd, 86 MiB/s wr, 1.38k op/s; 13821/166608 objects degraded (8.296%) 2024-09-17T07:16:49.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:49 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:16:49.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:49 smithi100 ceph-mon[39462]: pgmap v658: 148 pgs: 36 active+undersized, 28 active+undersized+degraded, 84 active+clean; 3.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.1 KiB/s rd, 86 MiB/s wr, 1.38k op/s; 13821/166608 objects degraded (8.296%) 2024-09-17T07:16:49.630 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 31 16 57739 57723 116.365 84.75 0.00225932 0.00858447 2024-09-17T07:16:49.653 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:16:49.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:49.697 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:49.697 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:16:49.798 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:16:49.836 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:49.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:49.837 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:16:49.938 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:16:49.982 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:49.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:49.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:16:50.083 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:16:50.114 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:50.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:50.115 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:16:50.215 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:16:50.244 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:50.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:50.245 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:16:50.346 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:16:50.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:50.370 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:50.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:16:50.405 INFO:teuthology.orchestra.run.smithi100.stderr:noscrub is set 2024-09-17T07:16:50.471 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:16:50.510 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:50.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:50.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:16:50.612 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:16:50.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:50 smithi100 ceph-mon[39462]: 5.e scrub starts 2024-09-17T07:16:50.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:50 smithi100 ceph-mon[39462]: 5.e scrub ok 2024-09-17T07:16:50.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:50 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/724125834' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:16:50.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:50 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:16:50.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:50 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:16:50.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:50 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:16:50.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:50 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 13821/166608 objects degraded (8.296%), 28 pgs degraded (PG_DEGRADED) 2024-09-17T07:16:50.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:50 smithi100 ceph-mon[29092]: 5.e scrub starts 2024-09-17T07:16:50.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:50 smithi100 ceph-mon[29092]: 5.e scrub ok 2024-09-17T07:16:50.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:50 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/724125834' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:16:50.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:50 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:16:50.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:50 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:16:50.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:50 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:16:50.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:50 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 13821/166608 objects degraded (8.296%), 28 pgs degraded (PG_DEGRADED) 2024-09-17T07:16:50.628 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 32 16 59151 59135 115.486 88.25 0.0123824 0.00865312 2024-09-17T07:16:50.653 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:50 smithi135 ceph-mon[33238]: 5.e scrub starts 2024-09-17T07:16:50.653 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:50 smithi135 ceph-mon[33238]: 5.e scrub ok 2024-09-17T07:16:50.653 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:50 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/724125834' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:16:50.653 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:50 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:16:50.653 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:50 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:16:50.653 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:50 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:16:50.653 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:50 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 13821/166608 objects degraded (8.296%), 28 pgs degraded (PG_DEGRADED) 2024-09-17T07:16:50.655 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:50.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:50.656 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:16:50.757 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:16:50.784 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:50.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:50.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:16:50.886 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:16:50.915 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:50.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:50.916 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:16:51.016 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:16:51.062 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:51.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:51.064 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:16:51.165 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:16:51.192 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:51.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:51.193 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:16:51.294 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:16:51.319 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:51.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:51.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:16:51.421 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:16:51.450 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:51.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:51.451 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:16:51.541 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:51 smithi135 ceph-mon[33238]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:16:51.541 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:51 smithi135 ceph-mon[33238]: 2.13 scrub starts 2024-09-17T07:16:51.541 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:51 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T07:16:51.541 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:51 smithi135 ceph-mon[33238]: osdmap e313: 8 total, 7 up, 7 in 2024-09-17T07:16:51.541 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:51 smithi135 ceph-mon[33238]: pgmap v660: 148 pgs: 36 active+undersized, 28 active+undersized+degraded, 84 active+clean; 3.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1023 B/s rd, 87 MiB/s wr, 1.39k op/s; 14206/170028 objects degraded (8.355%) 2024-09-17T07:16:51.550 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:51 smithi100 ceph-mon[29092]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:16:51.550 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:51 smithi100 ceph-mon[29092]: 2.13 scrub starts 2024-09-17T07:16:51.550 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:51 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T07:16:51.550 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:51 smithi100 ceph-mon[29092]: osdmap e313: 8 total, 7 up, 7 in 2024-09-17T07:16:51.550 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:51 smithi100 ceph-mon[29092]: pgmap v660: 148 pgs: 36 active+undersized, 28 active+undersized+degraded, 84 active+clean; 3.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1023 B/s rd, 87 MiB/s wr, 1.39k op/s; 14206/170028 objects degraded (8.355%) 2024-09-17T07:16:51.551 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:51 smithi100 ceph-mon[39462]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:16:51.551 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:51 smithi100 ceph-mon[39462]: 2.13 scrub starts 2024-09-17T07:16:51.551 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:51 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T07:16:51.551 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:51 smithi100 ceph-mon[39462]: osdmap e313: 8 total, 7 up, 7 in 2024-09-17T07:16:51.551 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:51 smithi100 ceph-mon[39462]: pgmap v660: 148 pgs: 36 active+undersized, 28 active+undersized+degraded, 84 active+clean; 3.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1023 B/s rd, 87 MiB/s wr, 1.39k op/s; 14206/170028 objects degraded (8.355%) 2024-09-17T07:16:51.552 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:16:51.592 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:51.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:51.594 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:16:51.627 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 33 16 60563 60547 114.66 88.25 0.00837843 0.00871621 2024-09-17T07:16:51.630 INFO:teuthology.orchestra.run.smithi135.stdout:ERROR: (22) Invalid argument 2024-09-17T07:16:51.630 INFO:teuthology.orchestra.run.smithi135.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:16:51.694 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:16:51.734 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:51.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:51.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:16:51.836 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:16:51.866 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:51.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:51.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:16:51.968 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:16:52.005 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:52.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:52.006 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:16:52.092 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:16:52.092 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:16:52.093 INFO:teuthology.orchestra.run.smithi100.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-17T07:16:52.107 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:16:52.119 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:16:52.119 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:16:52.143 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:16:52.143 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:16:52.145 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:52.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:52.146 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:16:52.176 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:16:52.177 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:16:52.202 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:16:52.202 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:16:52.227 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:16:52.228 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:16:52.247 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:16:52.251 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:16:52.251 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:16:52.274 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:52.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:52.276 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:16:52.377 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:16:52.395 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-17T07:16:52.417 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:52.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:52.418 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:16:52.520 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:16:52.546 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:52.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:52.547 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:16:52.627 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 34 16 61918 61902 113.779 84.6875 0.0171018 0.00878407 2024-09-17T07:16:52.648 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:16:52.675 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:52.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:52.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:16:52.743 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:52 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/484059851' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:16:52.743 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:52 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/484059851' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:16:52.777 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:16:52.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:52 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/484059851' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:16:52.817 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:16:52.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:52.828 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:52.828 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:16:52.929 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:16:52.955 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:52.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:52.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:16:53.057 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:16:53.095 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:53.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:53.096 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:16:53.197 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:16:53.199 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:16:53.199 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:16:53.237 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:53.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:53.238 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:16:53.339 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:16:53.379 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:53.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:53.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:16:53.481 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:16:53.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:53.513 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:53.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:16:53.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:53 smithi135 ceph-mon[33238]: pgmap v661: 148 pgs: 36 active+undersized, 28 active+undersized+degraded, 84 active+clean; 3.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1023 B/s rd, 87 MiB/s wr, 1.39k op/s; 14206/170028 objects degraded (8.355%) 2024-09-17T07:16:53.614 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:16:53.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:53 smithi100 ceph-mon[29092]: pgmap v661: 148 pgs: 36 active+undersized, 28 active+undersized+degraded, 84 active+clean; 3.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1023 B/s rd, 87 MiB/s wr, 1.39k op/s; 14206/170028 objects degraded (8.355%) 2024-09-17T07:16:53.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:53 smithi100 ceph-mon[39462]: pgmap v661: 148 pgs: 36 active+undersized, 28 active+undersized+degraded, 84 active+clean; 3.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1023 B/s rd, 87 MiB/s wr, 1.39k op/s; 14206/170028 objects degraded (8.355%) 2024-09-17T07:16:53.627 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 35 16 63301 63285 112.997 86.4375 0.00707883 0.00884488 2024-09-17T07:16:53.645 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:53.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:53.646 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:16:53.747 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:16:53.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:53.791 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:53.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:16:53.892 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:16:53.931 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:53.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:53.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:16:54.033 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:16:54.065 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:54.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:54.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:16:54.167 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:16:54.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:54.193 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:54.193 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:16:54.244 INFO:teuthology.orchestra.run.smithi100.stdout: marking collection for removal 2024-09-17T07:16:54.245 INFO:teuthology.orchestra.run.smithi100.stdout:setting '_remove' omap key 2024-09-17T07:16:54.245 INFO:teuthology.orchestra.run.smithi100.stdout:finish_remove_pgs 6.1_head removing 6.1 2024-09-17T07:16:54.250 INFO:teuthology.orchestra.run.smithi100.stdout:Remove successful 2024-09-17T07:16:54.294 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:16:54.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:54.344 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:54.344 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:16:54.445 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:16:54.486 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:54.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:54.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:16:54.588 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:16:54.622 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:54.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:54.624 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:16:54.627 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 36 16 64661 64645 112.219 85 0.0139578 0.00890653 2024-09-17T07:16:54.725 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:16:54.754 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:54.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:54.755 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:16:54.783 INFO:teuthology.orchestra.run.smithi135.stdout:{ 2024-09-17T07:16:54.783 INFO:teuthology.orchestra.run.smithi135.stdout: "ops": [], 2024-09-17T07:16:54.783 INFO:teuthology.orchestra.run.smithi135.stdout: "complaint_time": 30, 2024-09-17T07:16:54.783 INFO:teuthology.orchestra.run.smithi135.stdout: "num_blocked_ops": 0 2024-09-17T07:16:54.783 INFO:teuthology.orchestra.run.smithi135.stdout:} 2024-09-17T07:16:54.855 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:16:54.884 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:54.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:54.885 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:16:54.986 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:16:55.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:55.017 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:55.017 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:16:55.118 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:16:55.162 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:55.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:55.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:16:55.201 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:16:55.264 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:16:55.294 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:55.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:55.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:16:55.396 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:16:55.431 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:55.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:55.432 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:16:55.533 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:16:55.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:55 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 14206/170028 objects degraded (8.355%), 28 pgs degraded (PG_DEGRADED) 2024-09-17T07:16:55.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:55 smithi135 ceph-mon[33238]: osdmap e314: 8 total, 7 up, 7 in 2024-09-17T07:16:55.550 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:55 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 14206/170028 objects degraded (8.355%), 28 pgs degraded (PG_DEGRADED) 2024-09-17T07:16:55.550 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:55 smithi100 ceph-mon[29092]: osdmap e314: 8 total, 7 up, 7 in 2024-09-17T07:16:55.551 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:55 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 14206/170028 objects degraded (8.355%), 28 pgs degraded (PG_DEGRADED) 2024-09-17T07:16:55.551 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:55 smithi100 ceph-mon[39462]: osdmap e314: 8 total, 7 up, 7 in 2024-09-17T07:16:55.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:55.581 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:55.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:16:55.583 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -n osd.1 --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.6.1.1 2024-09-17T07:16:55.628 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 37 16 65942 65926 111.35 80.0625 0.0105316 0.00897585 2024-09-17T07:16:55.668 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:16:55.682 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:16:55.738 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:55.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:55.740 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:16:55.840 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:16:55.870 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:55.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:55.871 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:16:55.972 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:16:55.984 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/osd.1/config 2024-09-17T07:16:56.007 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:56.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:56.008 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:16:56.105 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:16:55 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:16:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:16:56.109 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:16:56.145 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:56.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:56.146 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:16:56.247 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:16:56.274 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:56.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:56.275 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:16:56.376 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:16:56.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:56 smithi100 ceph-mon[29092]: pgmap v663: 148 pgs: 1 active+clean+scrubbing, 36 active+undersized, 28 active+undersized+degraded, 83 active+clean; 3.9 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 103 MiB/s wr, 1.66k op/s; 15880/190806 objects degraded (8.323%) 2024-09-17T07:16:56.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:56 smithi100 ceph-mon[39462]: pgmap v663: 148 pgs: 1 active+clean+scrubbing, 36 active+undersized, 28 active+undersized+degraded, 83 active+clean; 3.9 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 103 MiB/s wr, 1.66k op/s; 15880/190806 objects degraded (8.323%) 2024-09-17T07:16:56.418 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:56.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:56.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:16:56.520 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:16:56.553 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:56.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:56.554 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:16:56.594 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-17T07:16:56.628 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 38 16 67170 67154 110.439 76.75 0.00954168 0.00904995 2024-09-17T07:16:56.655 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:16:56.683 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:56.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:56.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:16:56.788 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:16:56.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:56 smithi135 ceph-mon[33238]: pgmap v663: 148 pgs: 1 active+clean+scrubbing, 36 active+undersized, 28 active+undersized+degraded, 83 active+clean; 3.9 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 103 MiB/s wr, 1.66k op/s; 15880/190806 objects degraded (8.323%) 2024-09-17T07:16:56.817 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:56.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:56.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:16:56.919 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:16:56.948 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:56.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:56.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:16:57.050 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:16:57.092 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:57.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:57.094 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:16:57.195 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:16:57.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:57.231 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:57.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:16:57.332 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:16:57.359 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:57.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:57.360 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:16:57.461 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:16:57.501 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:57.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:57.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:16:57.605 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:16:57.628 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 39 16 68382 68366 109.55 75.75 0.0113662 0.00912336 2024-09-17T07:16:57.651 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:57.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:57.653 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:16:57.753 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:16:57.795 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:57.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:57.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:16:57.896 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:16:57.930 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:57.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:57.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:16:58.032 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:16:58.055 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:58.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:58.056 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:16:58.109 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:16:58.157 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:16:58.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:58.182 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:58.182 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:16:58.283 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:16:58.319 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:58.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:58.320 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:16:58.420 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:16:58.461 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:58.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:58.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:16:58.563 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:16:58.597 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:58.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:58.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:16:58.624 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:16:58 smithi135 ceph-mon[33238]: pgmap v664: 148 pgs: 1 active+clean+scrubbing, 36 active+undersized, 28 active+undersized+degraded, 83 active+clean; 3.9 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 63 MiB/s wr, 1.01k op/s; 15880/190806 objects degraded (8.323%) 2024-09-17T07:16:58.629 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:16:58 smithi100 ceph-mon[29092]: pgmap v664: 148 pgs: 1 active+clean+scrubbing, 36 active+undersized, 28 active+undersized+degraded, 83 active+clean; 3.9 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 63 MiB/s wr, 1.01k op/s; 15880/190806 objects degraded (8.323%) 2024-09-17T07:16:58.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:16:58 smithi100 ceph-mon[39462]: pgmap v664: 148 pgs: 1 active+clean+scrubbing, 36 active+undersized, 28 active+undersized+degraded, 83 active+clean; 3.9 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 63 MiB/s wr, 1.01k op/s; 15880/190806 objects degraded (8.323%) 2024-09-17T07:16:58.630 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:2024-09-17T07:16:58.628531+0000 min lat: 0.00112621 max lat: 0.332984 avg lat: 0.00921612 2024-09-17T07:16:58.630 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-17T07:16:58.630 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 40 16 69430 69414 108.448 65.5 0.0122898 0.00921612 2024-09-17T07:16:58.632 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-17T07:16:58.699 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:16:58.726 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:58.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:58.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:16:58.828 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:16:58.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:58.870 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:58.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:16:58.971 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:16:58.999 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:59.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:59.001 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:16:59.058 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:16:59.102 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:16:59.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:59.134 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:59.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:16:59.197 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:16:58 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:16:59.235 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:16:59.275 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:59.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:59.276 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:16:59.377 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:16:59.420 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:59.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:59.421 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:16:59.522 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:16:59.556 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:59.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:59.558 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:16:59.628 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 41 16 70649 70633 107.661 76.1875 0.0097959 0.00928319 2024-09-17T07:16:59.659 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:16:59.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:59.686 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:59.687 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:16:59.788 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:16:59.834 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:16:59.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:59.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:16:59.936 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:16:59.986 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:16:59.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:16:59.987 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:17:00.006 INFO:teuthology.orchestra.run.smithi100.stdout:get_pg_num_history pg_num_history pg_num_history(e302 pg_nums {1={20=1},2={67=32},3={67=3},4={69=32},5={71=32},6={73=32},9={236=16},10={297=16}} deleted_pools 294,9) 2024-09-17T07:17:00.006 INFO:teuthology.orchestra.run.smithi100.stdout:Importing pgid 6.1 2024-09-17T07:17:00.007 INFO:teuthology.orchestra.run.smithi100.stdout:write_pg epoch 267 info 6.1( empty local-lis/les=73/74 n=0 ec=73/73 lis/c=73/73 les/c/f=74/74/0 sis=73) 2024-09-17T07:17:00.007 INFO:teuthology.orchestra.run.smithi100.stdout:Import successful 2024-09-17T07:17:00.089 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:17:00.132 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:00.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:00.133 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:17:00.234 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:17:00.262 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:17:00.263 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:17:00.263 INFO:teuthology.orchestra.run.smithi100.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-17T07:17:00.276 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:00.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:00.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:17:00.284 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:17:00.285 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:17:00.308 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:17:00.309 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:17:00.333 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:17:00.333 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:17:00.362 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:17:00.363 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:17:00.378 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:17:00.400 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:17:00.400 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:17:00.427 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:17:00.427 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:17:00.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:00.428 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:00.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:17:00.480 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:00 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:17:00.480 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:00 smithi135 ceph-mon[33238]: pgmap v665: 148 pgs: 1 active+clean+scrubbing, 36 active+undersized, 28 active+undersized+degraded, 83 active+clean; 4.2 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 1.1 KiB/s rd, 84 MiB/s wr, 1.34k op/s; 17010/205803 objects degraded (8.265%) 2024-09-17T07:17:00.480 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:00 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 17010/205803 objects degraded (8.265%), 28 pgs degraded (PG_DEGRADED) 2024-09-17T07:17:00.480 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:00 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2657985152' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:17:00.529 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:17:00.569 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:00.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:00.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:17:00.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:00 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:17:00.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:00 smithi100 ceph-mon[39462]: pgmap v665: 148 pgs: 1 active+clean+scrubbing, 36 active+undersized, 28 active+undersized+degraded, 83 active+clean; 4.2 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 1.1 KiB/s rd, 84 MiB/s wr, 1.34k op/s; 17010/205803 objects degraded (8.265%) 2024-09-17T07:17:00.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:00 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 17010/205803 objects degraded (8.265%), 28 pgs degraded (PG_DEGRADED) 2024-09-17T07:17:00.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:00 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2657985152' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:17:00.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:00 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:17:00.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:00 smithi100 ceph-mon[29092]: pgmap v665: 148 pgs: 1 active+clean+scrubbing, 36 active+undersized, 28 active+undersized+degraded, 83 active+clean; 4.2 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 1.1 KiB/s rd, 84 MiB/s wr, 1.34k op/s; 17010/205803 objects degraded (8.265%) 2024-09-17T07:17:00.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:00 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 17010/205803 objects degraded (8.265%), 28 pgs degraded (PG_DEGRADED) 2024-09-17T07:17:00.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:00 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2657985152' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:17:00.628 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 42 16 71926 71910 106.998 79.8125 0.0110282 0.00934123 2024-09-17T07:17:00.671 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:17:00.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:00.708 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:00.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:17:00.809 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:17:00.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:00.845 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:00.845 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:17:00.946 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:17:00.987 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:00.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:00.988 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:17:00.994 INFO:teuthology.orchestra.run.smithi135.stdout:ERROR: (22) Invalid argument 2024-09-17T07:17:00.994 INFO:teuthology.orchestra.run.smithi135.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:17:01.090 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:17:01.124 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:01.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:01.125 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:17:01.186 DEBUG:teuthology.orchestra.run.smithi100:> sudo rm -f /var/log/ceph/e848267c-74c1-11ef-bceb-c7b262605968/exp.6.1.1 2024-09-17T07:17:01.226 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:17:01.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:01.255 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:01.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:17:01.356 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:17:01.385 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:01.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:01.386 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:17:01.489 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:17:01.515 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:01.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:01.516 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:17:01.617 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:17:01.628 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 43 16 73171 73155 106.319 77.8125 0.00912495 0.00940057 2024-09-17T07:17:01.657 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:01.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:01.658 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:17:01.759 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:17:01.786 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:01.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:01.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:17:01.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:01 smithi135 ceph-mon[33238]: pgmap v666: 148 pgs: 36 active+undersized, 28 active+undersized+degraded, 84 active+clean; 4.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1023 B/s rd, 82 MiB/s wr, 1.31k op/s; 17393/209226 objects degraded (8.313%) 2024-09-17T07:17:01.794 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:01 smithi100 ceph-mon[29092]: pgmap v666: 148 pgs: 36 active+undersized, 28 active+undersized+degraded, 84 active+clean; 4.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1023 B/s rd, 82 MiB/s wr, 1.31k op/s; 17393/209226 objects degraded (8.313%) 2024-09-17T07:17:01.794 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:01 smithi100 ceph-mon[39462]: pgmap v666: 148 pgs: 36 active+undersized, 28 active+undersized+degraded, 84 active+clean; 4.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1023 B/s rd, 82 MiB/s wr, 1.31k op/s; 17393/209226 objects degraded (8.313%) 2024-09-17T07:17:01.843 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-17T07:17:01.888 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:17:01.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:01.914 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:01.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:17:02.015 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:17:02.040 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:02.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:02.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:17:02.142 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:17:02.192 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:02.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:02.193 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:17:02.238 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:17:02.294 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:17:02.351 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:02.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:02.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:17:02.453 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:17:02.480 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:02.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:02.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:17:02.582 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:17:02.620 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:02.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:02.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:17:02.628 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 44 16 74286 74270 105.487 69.6875 0.0149639 0.00947479 2024-09-17T07:17:02.722 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:17:02.747 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:02.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:02.749 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:17:02.850 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:17:02.889 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:02.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:02.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:17:02.991 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:17:03.017 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:03.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:03.018 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:17:03.119 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:17:03.153 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:03.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:03.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:17:03.255 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:17:03.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:03.295 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:03.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:17:03.396 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:17:03.423 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:03.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:03.424 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:17:03.525 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:17:03.551 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:03.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:03.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:17:03.611 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:17:03.611 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:17:03.628 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 45 16 75479 75463 104.799 74.5625 0.00547706 0.00953727 2024-09-17T07:17:03.653 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:17:03.683 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:03.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:03.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:17:03.785 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:17:03.823 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:03.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:03.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:17:03.926 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:17:03.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:03.958 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:03.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:17:04.059 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:17:04.080 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:04.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:04.081 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:17:04.106 INFO:teuthology.orchestra.run.smithi100.stderr:nodeep-scrub is set 2024-09-17T07:17:04.182 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:17:04.214 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:04.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:04.216 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:17:04.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:04 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:17:04.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:04 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:17:04.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:04 smithi135 ceph-mon[33238]: pgmap v667: 148 pgs: 36 active+undersized, 28 active+undersized+degraded, 84 active+clean; 4.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1023 B/s rd, 82 MiB/s wr, 1.31k op/s; 17393/209226 objects degraded (8.313%) 2024-09-17T07:17:04.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:04 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2366181158' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:17:04.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:04 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:17:04.316 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:17:04.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:04.354 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:04.354 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:17:04.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:04 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:17:04.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:04 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:17:04.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:04 smithi100 ceph-mon[29092]: pgmap v667: 148 pgs: 36 active+undersized, 28 active+undersized+degraded, 84 active+clean; 4.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1023 B/s rd, 82 MiB/s wr, 1.31k op/s; 17393/209226 objects degraded (8.313%) 2024-09-17T07:17:04.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:04 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2366181158' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:17:04.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:04 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:17:04.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:04 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:17:04.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:04 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:17:04.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:04 smithi100 ceph-mon[39462]: pgmap v667: 148 pgs: 36 active+undersized, 28 active+undersized+degraded, 84 active+clean; 4.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1023 B/s rd, 82 MiB/s wr, 1.31k op/s; 17393/209226 objects degraded (8.313%) 2024-09-17T07:17:04.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:04 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2366181158' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:17:04.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:04 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:17:04.455 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:17:04.478 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:04.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:04.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:17:04.580 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:17:04.617 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:04.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:04.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:17:04.628 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 46 16 76488 76472 103.892 63.0625 0.0116918 0.00962058 2024-09-17T07:17:04.719 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:17:04.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:04.748 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:04.749 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:17:04.849 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:17:04.894 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:04.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:04.896 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:17:04.996 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:17:05.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:05.029 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:05.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:17:05.130 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:17:05.163 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:05.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:05.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:17:05.265 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:17:05.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:05.303 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:05.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:17:05.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:05 smithi100 ceph-mon[29092]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:17:05.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:05 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T07:17:05.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:05 smithi100 ceph-mon[29092]: osdmap e315: 8 total, 7 up, 7 in 2024-09-17T07:17:05.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:05 smithi100 ceph-mon[39462]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:17:05.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:05 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T07:17:05.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:05 smithi100 ceph-mon[39462]: osdmap e315: 8 total, 7 up, 7 in 2024-09-17T07:17:05.405 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:17:05.425 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:05.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:05.426 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:17:05.527 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:17:05.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:05 smithi135 ceph-mon[33238]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:17:05.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:05 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T07:17:05.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:05 smithi135 ceph-mon[33238]: osdmap e315: 8 total, 7 up, 7 in 2024-09-17T07:17:05.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:05.557 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:05.558 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:17:05.612 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:17:05.628 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 47 16 77546 77530 103.088 66.125 0.00916559 0.00969596 2024-09-17T07:17:05.658 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:17:05.691 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:05.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:05.692 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:17:05.797 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:17:05.828 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:05.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:05.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:17:05.888 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:17:05.930 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:17:05.970 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:05.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:05.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:17:06.018 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:17:05 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:17:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:17:06.072 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:17:06.095 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:06.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:06.096 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:17:06.196 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:17:06.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:06.235 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:06.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:17:06.262 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7] out_osds: [3] dead_osds: [1] live_osds: [0, 2, 3, 4, 5, 6, 7] 2024-09-17T07:17:06.262 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T07:17:06.262 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_3 2024-09-17T07:17:06.263 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T07:17:06.291 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:06 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 17393/209226 objects degraded (8.313%), 28 pgs degraded (PG_DEGRADED) 2024-09-17T07:17:06.291 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:06 smithi100 ceph-mon[29092]: osdmap e316: 8 total, 7 up, 7 in 2024-09-17T07:17:06.291 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:06 smithi100 ceph-mon[29092]: pgmap v670: 148 pgs: 36 active+undersized, 28 active+undersized+degraded, 84 active+clean; 4.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.52k op/s; 18870/227268 objects degraded (8.303%) 2024-09-17T07:17:06.291 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:06 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:17:06.291 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:06 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:17:06.291 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:06 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:17:06.291 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:06 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:17:06.291 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:06 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:17:06.292 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:06 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 17393/209226 objects degraded (8.313%), 28 pgs degraded (PG_DEGRADED) 2024-09-17T07:17:06.292 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:06 smithi100 ceph-mon[39462]: osdmap e316: 8 total, 7 up, 7 in 2024-09-17T07:17:06.292 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:06 smithi100 ceph-mon[39462]: pgmap v670: 148 pgs: 36 active+undersized, 28 active+undersized+degraded, 84 active+clean; 4.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.52k op/s; 18870/227268 objects degraded (8.303%) 2024-09-17T07:17:06.292 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:06 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:17:06.292 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:06 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:17:06.292 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:06 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:17:06.292 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:06 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:17:06.292 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:06 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:17:06.336 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:17:06.368 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:06.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:06.373 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:17:06.474 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:17:06.506 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:06.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:06.507 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:17:06.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:06 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 17393/209226 objects degraded (8.313%), 28 pgs degraded (PG_DEGRADED) 2024-09-17T07:17:06.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:06 smithi135 ceph-mon[33238]: osdmap e316: 8 total, 7 up, 7 in 2024-09-17T07:17:06.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:06 smithi135 ceph-mon[33238]: pgmap v670: 148 pgs: 36 active+undersized, 28 active+undersized+degraded, 84 active+clean; 4.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.52k op/s; 18870/227268 objects degraded (8.303%) 2024-09-17T07:17:06.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:06 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:17:06.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:06 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:17:06.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:06 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:17:06.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:06 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:17:06.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:06 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:17:06.607 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:17:06.629 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 48 16 78611 78595 102.327 66.5625 0.00266274 0.00975361 2024-09-17T07:17:06.630 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:06.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:06.631 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:17:06.690 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:17:06.731 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:17:06.752 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:06.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:06.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:17:06.854 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:17:06.880 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:06.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:06.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:17:06.983 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:17:07.012 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T07:17:07.020 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:07.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:07.021 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:17:07.121 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:17:07.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:07.143 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:07.144 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:17:07.244 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:17:07.269 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:07.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:07.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:17:07.371 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:17:07.373 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:17:07.413 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:07.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:07.414 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:17:07.515 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:17:07.556 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:07.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:07.557 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:17:07.629 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 49 16 79555 79539 101.443 59 0.0170554 0.00985325 2024-09-17T07:17:07.657 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:17:07.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:07.706 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:07.707 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:17:07.807 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:17:07.835 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:07.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:07.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:17:07.938 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:17:07.966 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:07.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:07.968 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:17:08.069 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:17:08.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:08.097 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:08.097 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:17:08.198 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:17:08.224 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:08.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:08.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:17:08.327 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:17:08.368 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:08.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:08.369 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:17:08.405 INFO:teuthology.orchestra.run.smithi135.stdout:ERROR: (22) Invalid argument 2024-09-17T07:17:08.405 INFO:teuthology.orchestra.run.smithi135.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:17:08.471 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:17:08.492 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:08.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:08.493 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:17:08.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:08 smithi135 ceph-mon[33238]: pgmap v671: 148 pgs: 36 active+undersized, 28 active+undersized+degraded, 84 active+clean; 4.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 639 B/s rd, 56 MiB/s wr, 894 op/s; 18870/227268 objects degraded (8.303%) 2024-09-17T07:17:08.595 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:17:08.621 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:08.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:08.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:17:08.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:08 smithi100 ceph-mon[29092]: pgmap v671: 148 pgs: 36 active+undersized, 28 active+undersized+degraded, 84 active+clean; 4.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 639 B/s rd, 56 MiB/s wr, 894 op/s; 18870/227268 objects degraded (8.303%) 2024-09-17T07:17:08.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:08 smithi100 ceph-mon[39462]: pgmap v671: 148 pgs: 36 active+undersized, 28 active+undersized+degraded, 84 active+clean; 4.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 639 B/s rd, 56 MiB/s wr, 894 op/s; 18870/227268 objects degraded (8.303%) 2024-09-17T07:17:08.628 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 50 16 80596 80580 100.715 65.0625 0.0135522 0.0099244 2024-09-17T07:17:08.723 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:17:08.765 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:08.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:08.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:17:08.868 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:17:08.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:08.901 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:08.901 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:17:09.002 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:17:09.042 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:09.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:09.043 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:17:09.143 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:17:09.145 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-17T07:17:09.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:09.228 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:09.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:17:09.232 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:17:08 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:17:09.328 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:17:09.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:09.366 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:09.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:17:09.467 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:17:09.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:09.530 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:09.530 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:17:09.561 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:17:09.629 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 51 16 81716 81700 100.113 70 0.00821758 0.00998387 2024-09-17T07:17:09.630 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:17:09.662 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:09.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:09.663 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:17:09.764 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:17:09.805 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:09.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:09.806 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:17:09.907 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:17:09.944 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:09.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:09.945 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:17:10.046 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:17:10.075 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:10.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:10.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:17:10.177 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:17:10.210 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:10.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:10.211 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:17:10.312 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:17:10.340 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:10.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:10.341 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:17:10.442 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:17:10.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:10.489 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:10.489 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:17:10.590 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:17:10.619 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:10.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:10.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:17:10.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:10 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:17:10.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:10 smithi100 ceph-mon[29092]: pgmap v672: 148 pgs: 36 active+undersized, 28 active+undersized+degraded, 84 active+clean; 4.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.28k op/s; 19841/239862 objects degraded (8.272%) 2024-09-17T07:17:10.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:10 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 19841/239862 objects degraded (8.272%), 28 pgs degraded (PG_DEGRADED) 2024-09-17T07:17:10.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:10 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:17:10.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:10 smithi100 ceph-mon[39462]: pgmap v672: 148 pgs: 36 active+undersized, 28 active+undersized+degraded, 84 active+clean; 4.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.28k op/s; 19841/239862 objects degraded (8.272%) 2024-09-17T07:17:10.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:10 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 19841/239862 objects degraded (8.272%), 28 pgs degraded (PG_DEGRADED) 2024-09-17T07:17:10.629 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 52 16 82989 82973 99.7175 79.5625 0.00630567 0.0100235 2024-09-17T07:17:10.721 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:17:10.748 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:10 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:17:10.748 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:10 smithi135 ceph-mon[33238]: pgmap v672: 148 pgs: 36 active+undersized, 28 active+undersized+degraded, 84 active+clean; 4.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.28k op/s; 19841/239862 objects degraded (8.272%) 2024-09-17T07:17:10.748 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:10 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 19841/239862 objects degraded (8.272%), 28 pgs degraded (PG_DEGRADED) 2024-09-17T07:17:10.758 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:10.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:10.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:17:10.860 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:17:10.908 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:10.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:10.909 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:17:11.010 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:17:11.035 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:17:11.036 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:17:11.036 INFO:teuthology.orchestra.run.smithi100.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-17T07:17:11.046 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:11.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:11.047 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:17:11.066 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:17:11.066 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:17:11.089 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:17:11.089 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:17:11.111 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:17:11.111 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:17:11.148 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:17:11.151 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:17:11.151 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:17:11.177 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:11.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:11.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:17:11.185 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:17:11.185 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:17:11.219 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:17:11.219 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:17:11.280 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:17:11.394 INFO:teuthology.orchestra.run.smithi135.stdout:{ 2024-09-17T07:17:11.395 INFO:teuthology.orchestra.run.smithi135.stdout: "ops": [], 2024-09-17T07:17:11.395 INFO:teuthology.orchestra.run.smithi135.stdout: "complaint_time": 30, 2024-09-17T07:17:11.395 INFO:teuthology.orchestra.run.smithi135.stdout: "num_blocked_ops": 0 2024-09-17T07:17:11.395 INFO:teuthology.orchestra.run.smithi135.stdout:} 2024-09-17T07:17:11.520 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:11.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:11.521 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:17:11.549 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:11 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1192791365' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:17:11.549 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:11 smithi100 ceph-mon[29092]: pgmap v673: 148 pgs: 36 active+undersized, 28 active+undersized+degraded, 84 active+clean; 4.9 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 87 MiB/s wr, 1.40k op/s; 20179/242775 objects degraded (8.312%) 2024-09-17T07:17:11.549 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:11 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1192791365' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:17:11.549 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:11 smithi100 ceph-mon[39462]: pgmap v673: 148 pgs: 36 active+undersized, 28 active+undersized+degraded, 84 active+clean; 4.9 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 87 MiB/s wr, 1.40k op/s; 20179/242775 objects degraded (8.312%) 2024-09-17T07:17:11.622 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:17:11.629 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 53 16 84042 84026 99.0767 65.8125 0.00919163 0.0100888 2024-09-17T07:17:11.651 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:11.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:11.652 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:17:11.753 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:17:11.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:11 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1192791365' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:17:11.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:11 smithi135 ceph-mon[33238]: pgmap v673: 148 pgs: 36 active+undersized, 28 active+undersized+degraded, 84 active+clean; 4.9 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 87 MiB/s wr, 1.40k op/s; 20179/242775 objects degraded (8.312%) 2024-09-17T07:17:11.795 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:11.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:11.796 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:17:11.897 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:17:11.936 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:11.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:11.938 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:17:11.943 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T07:17:11.950 INFO:teuthology.orchestra.run.smithi100.stderr:dumped all 2024-09-17T07:17:12.038 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:17:12.079 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:12.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:12.080 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:17:12.151 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-17T07:17:12.182 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:17:12.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:12.243 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:12.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:17:12.344 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:17:12.369 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:12.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:12.370 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:17:12.470 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:17:12.494 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:12.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:12.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:17:12.533 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:17:12.542 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:12 smithi100 ceph-mon[29092]: from='client.15714 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:17:12.542 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:12 smithi100 ceph-mon[39462]: from='client.15714 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:17:12.596 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:17:12.630 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 54 16 85219 85203 98.6041 73.5625 0.00817607 0.0101368 2024-09-17T07:17:12.642 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:12.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:12.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:17:12.678 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:12 smithi135 ceph-mon[33238]: from='client.15714 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:17:12.744 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:17:12.776 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:12.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:12.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:17:12.878 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:17:12.906 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:12.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:12.907 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:17:13.008 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:17:13.033 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:13.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:13.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:17:13.135 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:17:13.162 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:13.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:13.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:17:13.264 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:17:13.300 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:13.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:13.301 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:17:13.402 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:17:13.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:13.438 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:13.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:17:13.539 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:17:13.581 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:13.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:13.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:17:13.630 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 55 15 86459 86444 98.2215 77.5625 0.0108833 0.0101761 2024-09-17T07:17:13.683 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:17:13.721 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:13.723 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:13 smithi135 ceph-mon[33238]: pgmap v674: 148 pgs: 36 active+undersized, 28 active+undersized+degraded, 84 active+clean; 4.9 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 556 B/s rd, 42 MiB/s wr, 680 op/s; 20179/242775 objects degraded (8.312%) 2024-09-17T07:17:13.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:13.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:17:13.752 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:13 smithi100 ceph-mon[29092]: pgmap v674: 148 pgs: 36 active+undersized, 28 active+undersized+degraded, 84 active+clean; 4.9 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 556 B/s rd, 42 MiB/s wr, 680 op/s; 20179/242775 objects degraded (8.312%) 2024-09-17T07:17:13.753 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:13 smithi100 ceph-mon[39462]: pgmap v674: 148 pgs: 36 active+undersized, 28 active+undersized+degraded, 84 active+clean; 4.9 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 556 B/s rd, 42 MiB/s wr, 680 op/s; 20179/242775 objects degraded (8.312%) 2024-09-17T07:17:13.783 INFO:teuthology.orchestra.run.smithi100.stdout:{"pg_ready":true,"pg_map":{"version":673,"stamp":"2024-09-17T07:17:11.302492+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5291251990,"num_objects":80925,"num_object_clones":0,"num_object_copies":242775,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20179,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80925,"num_whiteouts":0,"num_read":1394,"num_read_kb":1240,"num_write":81624,"num_write_kb":5169115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":1773876,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77283,"ondisk_log_size":77283,"up":380,"acting":380,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":18339560,"kb_used_data":14191400,"kb_used_omap":10,"kb_used_meta":4148085,"kb_avail":637876504,"statfs":{"total":671965249536,"available":653185540096,"internally_reserved":0,"allocated":14531993600,"data_stored":14528560766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11152,"internal_metadata":4247639152},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":732889088,"num_objects":11183,"num_object_clones":0,"num_object_copies":33549,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2786,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11183,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":11183,"num_write_kb":715712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002310"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":164,"reported_epoch":315,"state":"active+undersized","last_fresh":"2024-09-17T07:17:04.110790+0000","last_change":"2024-09-17T07:16:17.379311+0000","last_active":"2024-09-17T07:17:04.110790+0000","last_peered":"2024-09-17T07:17:04.110790+0000","last_clean":"2024-09-17T07:16:15.590051+0000","last_became_active":"2024-09-17T07:16:17.379311+0000","last_became_peered":"2024-09-17T07:16:17.379311+0000","last_unstale":"2024-09-17T07:17:04.110790+0000","last_undegraded":"2024-09-17T07:17:04.110790+0000","last_fullsized":"2024-09-17T07:16:17.372051+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:11.978185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:33.644728+0000","last_clean_scrub_stamp":"2024-09-17T07:13:11.978185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0074436570000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":330,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-17T07:17:04.104012+0000","last_change":"2024-09-17T07:16:30.401351+0000","last_active":"2024-09-17T07:17:04.104012+0000","last_peered":"2024-09-17T07:17:04.104012+0000","last_clean":"2024-09-17T07:17:04.104012+0000","last_became_active":"2024-09-17T07:02:12.006754+0000","last_became_peered":"2024-09-17T07:02:12.006754+0000","last_unstale":"2024-09-17T07:17:04.104012+0000","last_undegraded":"2024-09-17T07:17:04.104012+0000","last_fullsized":"2024-09-17T07:17:04.104012+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:30.401191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:16:30.401191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:51.746677+0000","scrub_duration":0.0038520239999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"242'27","reported_seq":229,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-17T07:17:04.103752+0000","last_change":"2024-09-17T07:14:47.661623+0000","last_active":"2024-09-17T07:17:04.103752+0000","last_peered":"2024-09-17T07:17:04.103752+0000","last_clean":"2024-09-17T07:17:04.103752+0000","last_became_active":"2024-09-17T07:09:24.402042+0000","last_became_peered":"2024-09-17T07:09:24.402042+0000","last_unstale":"2024-09-17T07:17:04.103752+0000","last_undegraded":"2024-09-17T07:17:04.103752+0000","last_fullsized":"2024-09-17T07:17:04.103752+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'27","last_scrub_stamp":"2024-09-17T07:14:47.661187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:14:47.661187+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010587222,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":321,"reported_epoch":315,"state":"active+undersized","last_fresh":"2024-09-17T07:17:04.111702+0000","last_change":"2024-09-17T07:16:17.380766+0000","last_active":"2024-09-17T07:17:04.111702+0000","last_peered":"2024-09-17T07:17:04.111702+0000","last_clean":"2024-09-17T07:16:15.357790+0000","last_became_active":"2024-09-17T07:16:17.380766+0000","last_became_peered":"2024-09-17T07:16:17.380766+0000","last_unstale":"2024-09-17T07:17:04.111702+0000","last_undegraded":"2024-09-17T07:17:04.111702+0000","last_fullsized":"2024-09-17T07:16:17.364849+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:45.261117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:15:45.261117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:14.862846+0000","scrub_duration":0.001477482,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":169,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-17T07:17:04.104094+0000","last_change":"2024-09-17T07:14:57.520400+0000","last_active":"2024-09-17T07:17:04.104094+0000","last_peered":"2024-09-17T07:17:04.104094+0000","last_clean":"2024-09-17T07:17:04.104094+0000","last_became_active":"2024-09-17T07:09:24.402568+0000","last_became_peered":"2024-09-17T07:09:24.402568+0000","last_unstale":"2024-09-17T07:17:04.104094+0000","last_undegraded":"2024-09-17T07:17:04.104094+0000","last_fullsized":"2024-09-17T07:17:04.104094+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:57.520290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:14:57.520290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001539143,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"242'15","reported_seq":359,"reported_epoch":315,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:17:04.111833+0000","last_change":"2024-09-17T07:16:17.370326+0000","last_active":"2024-09-17T07:17:04.111833+0000","last_peered":"2024-09-17T07:17:04.111833+0000","last_clean":"2024-09-17T07:16:15.357813+0000","last_became_active":"2024-09-17T07:16:17.370326+0000","last_became_peered":"2024-09-17T07:16:17.370326+0000","last_unstale":"2024-09-17T07:17:04.111833+0000","last_undegraded":"2024-09-17T07:16:17.364616+0000","last_fullsized":"2024-09-17T07:16:17.364511+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'15","last_scrub_stamp":"2024-09-17T07:15:39.306043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:15:39.306043+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002471029,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":4}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":21,"reported_epoch":315,"state":"active+undersized","last_fresh":"2024-09-17T07:17:04.116957+0000","last_change":"2024-09-17T07:16:17.379569+0000","last_active":"2024-09-17T07:17:04.116957+0000","last_peered":"2024-09-17T07:17:04.116957+0000","last_clean":"2024-09-17T07:02:14.999435+0000","last_became_active":"2024-09-17T07:16:17.379569+0000","last_became_peered":"2024-09-17T07:16:17.379569+0000","last_unstale":"2024-09-17T07:17:04.116957+0000","last_undegraded":"2024-09-17T07:17:04.116957+0000","last_fullsized":"2024-09-17T07:16:17.369924+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:53.881813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:15:53.881813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:16.389981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":319,"reported_epoch":315,"state":"active+undersized","last_fresh":"2024-09-17T07:17:04.113055+0000","last_change":"2024-09-17T07:16:17.374605+0000","last_active":"2024-09-17T07:17:04.113055+0000","last_peered":"2024-09-17T07:17:04.113055+0000","last_clean":"2024-09-17T07:16:15.357982+0000","last_became_active":"2024-09-17T07:16:17.374605+0000","last_became_peered":"2024-09-17T07:16:17.374605+0000","last_unstale":"2024-09-17T07:17:04.113055+0000","last_undegraded":"2024-09-17T07:17:04.113055+0000","last_fullsized":"2024-09-17T07:16:17.366300+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:41.707969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:14:41.707969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.014931757,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":316,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-17T07:17:04.117332+0000","last_change":"2024-09-17T07:15:46.312474+0000","last_active":"2024-09-17T07:17:04.117332+0000","last_peered":"2024-09-17T07:17:04.117332+0000","last_clean":"2024-09-17T07:17:04.117332+0000","last_became_active":"2024-09-17T07:02:12.011129+0000","last_became_peered":"2024-09-17T07:02:12.011129+0000","last_unstale":"2024-09-17T07:17:04.117332+0000","last_undegraded":"2024-09-17T07:17:04.117332+0000","last_fullsized":"2024-09-17T07:17:04.117332+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:46.312371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:46.312371+0000","last_clean_scrub_stamp":"2024-09-17T07:15:46.312371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:06.831380+0000","scrub_duration":0.0015176510000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"242'7","reported_seq":351,"reported_epoch":315,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:17:04.111793+0000","last_change":"2024-09-17T07:16:17.373486+0000","last_active":"2024-09-17T07:17:04.111793+0000","last_peered":"2024-09-17T07:17:04.111793+0000","last_clean":"2024-09-17T07:16:15.358036+0000","last_became_active":"2024-09-17T07:16:17.373486+0000","last_became_peered":"2024-09-17T07:16:17.373486+0000","last_unstale":"2024-09-17T07:17:04.111793+0000","last_undegraded":"2024-09-17T07:16:17.366228+0000","last_fullsized":"2024-09-17T07:16:17.365631+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'7","last_scrub_stamp":"2024-09-17T07:14:42.653705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:14:42.653705+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0050149230000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[{"shards":"2,4","objects":2}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":318,"reported_epoch":315,"state":"active+undersized","last_fresh":"2024-09-17T07:17:04.110377+0000","last_change":"2024-09-17T07:16:17.382323+0000","last_active":"2024-09-17T07:17:04.110377+0000","last_peered":"2024-09-17T07:17:04.110377+0000","last_clean":"2024-09-17T07:16:15.589989+0000","last_became_active":"2024-09-17T07:16:17.382323+0000","last_became_peered":"2024-09-17T07:16:17.382323+0000","last_unstale":"2024-09-17T07:17:04.110377+0000","last_undegraded":"2024-09-17T07:17:04.110377+0000","last_fullsized":"2024-09-17T07:16:17.371260+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:37.997296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:14:37.997296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045486420000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":338,"reported_epoch":315,"state":"active+undersized","last_fresh":"2024-09-17T07:17:04.117071+0000","last_change":"2024-09-17T07:16:17.374238+0000","last_active":"2024-09-17T07:17:04.117071+0000","last_peered":"2024-09-17T07:17:04.117071+0000","last_clean":"2024-09-17T07:16:16.170346+0000","last_became_active":"2024-09-17T07:16:17.374238+0000","last_became_peered":"2024-09-17T07:16:17.374238+0000","last_unstale":"2024-09-17T07:17:04.117071+0000","last_undegraded":"2024-09-17T07:17:04.117071+0000","last_fullsized":"2024-09-17T07:16:17.367446+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:52.168181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:14:52.168181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0056501399999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":316,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-17T07:17:04.112336+0000","last_change":"2024-09-17T07:14:44.076126+0000","last_active":"2024-09-17T07:17:04.112336+0000","last_peered":"2024-09-17T07:17:04.112336+0000","last_clean":"2024-09-17T07:17:04.112336+0000","last_became_active":"2024-09-17T07:02:12.012234+0000","last_became_peered":"2024-09-17T07:02:12.012234+0000","last_unstale":"2024-09-17T07:17:04.112336+0000","last_undegraded":"2024-09-17T07:17:04.112336+0000","last_fullsized":"2024-09-17T07:17:04.112336+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:44.076012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:45.029247+0000","last_clean_scrub_stamp":"2024-09-17T07:14:44.076012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.015917521,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"312'19","reported_seq":375,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-17T07:17:04.113426+0000","last_change":"2024-09-17T07:16:44.293061+0000","last_active":"2024-09-17T07:17:04.113426+0000","last_peered":"2024-09-17T07:17:04.113426+0000","last_clean":"2024-09-17T07:17:04.113426+0000","last_became_active":"2024-09-17T07:09:24.397954+0000","last_became_peered":"2024-09-17T07:09:24.397954+0000","last_unstale":"2024-09-17T07:17:04.113426+0000","last_undegraded":"2024-09-17T07:17:04.113426+0000","last_fullsized":"2024-09-17T07:17:04.113426+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"312'19","last_scrub_stamp":"2024-09-17T07:16:44.292911+0000","last_deep_scrub":"312'19","last_deep_scrub_stamp":"2024-09-17T07:16:44.292911+0000","last_clean_scrub_stamp":"2024-09-17T07:16:44.292911+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:49.152333+0000","scrub_duration":0.013103178,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":334,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-17T07:17:04.116433+0000","last_change":"2024-09-17T07:16:45.164526+0000","last_active":"2024-09-17T07:17:04.116433+0000","last_peered":"2024-09-17T07:17:04.116433+0000","last_clean":"2024-09-17T07:17:04.116433+0000","last_became_active":"2024-09-17T07:02:16.023774+0000","last_became_peered":"2024-09-17T07:02:16.023774+0000","last_unstale":"2024-09-17T07:17:04.116433+0000","last_undegraded":"2024-09-17T07:17:04.116433+0000","last_fullsized":"2024-09-17T07:17:04.116433+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:45.164367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:28.096266+0000","last_clean_scrub_stamp":"2024-09-17T07:16:45.164367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:10.271781+0000","scrub_duration":0.0026790630000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":319,"reported_epoch":315,"state":"active+undersized","last_fresh":"2024-09-17T07:17:04.102508+0000","last_change":"2024-09-17T07:16:17.378534+0000","last_active":"2024-09-17T07:17:04.102508+0000","last_peered":"2024-09-17T07:17:04.102508+0000","last_clean":"2024-09-17T07:16:15.494529+0000","last_became_active":"2024-09-17T07:16:17.378534+0000","last_became_peered":"2024-09-17T07:16:17.378534+0000","last_unstale":"2024-09-17T07:17:04.102508+0000","last_undegraded":"2024-09-17T07:17:04.102508+0000","last_fullsized":"2024-09-17T07:16:17.365363+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:45.483622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:15:45.483622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001342359,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3","version":"316'4964","reported_seq":4992,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-17T07:17:08.664025+0000","last_change":"2024-09-17T07:16:05.058011+0000","last_active":"2024-09-17T07:17:08.664025+0000","last_peered":"2024-09-17T07:17:08.664025+0000","last_clean":"2024-09-17T07:17:08.664025+0000","last_became_active":"2024-09-17T07:16:05.057730+0000","last_became_peered":"2024-09-17T07:16:05.057730+0000","last_unstale":"2024-09-17T07:17:08.664025+0000","last_undegraded":"2024-09-17T07:17:08.664025+0000","last_fullsized":"2024-09-17T07:17:08.664025+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":297,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":4964,"log_dups_size":0,"ondisk_log_size":4964,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:13.826432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":325320704,"num_objects":4964,"num_object_clones":0,"num_object_copies":14892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4964,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4964,"num_write_kb":317696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"242'39","reported_seq":391,"reported_epoch":315,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:17:04.112236+0000","last_change":"2024-09-17T07:16:17.374805+0000","last_active":"2024-09-17T07:17:04.112236+0000","last_peered":"2024-09-17T07:17:04.112236+0000","last_clean":"2024-09-17T07:16:15.358504+0000","last_became_active":"2024-09-17T07:16:17.374805+0000","last_became_peered":"2024-09-17T07:16:17.374805+0000","last_unstale":"2024-09-17T07:17:04.112236+0000","last_undegraded":"2024-09-17T07:16:17.367444+0000","last_fullsized":"2024-09-17T07:16:17.367331+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'39","last_scrub_stamp":"2024-09-17T07:15:46.226554+0000","last_deep_scrub":"95'25","last_deep_scrub_stamp":"2024-09-17T07:04:09.450039+0000","last_clean_scrub_stamp":"2024-09-17T07:15:46.226554+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:08.525283+0000","scrub_duration":0.0027061799999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[{"shards":"2,4","objects":8}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":330,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-17T07:17:04.111225+0000","last_change":"2024-09-17T07:16:33.658452+0000","last_active":"2024-09-17T07:17:04.111225+0000","last_peered":"2024-09-17T07:17:04.111225+0000","last_clean":"2024-09-17T07:17:04.111225+0000","last_became_active":"2024-09-17T07:02:12.011258+0000","last_became_peered":"2024-09-17T07:02:12.011258+0000","last_unstale":"2024-09-17T07:17:04.111225+0000","last_undegraded":"2024-09-17T07:17:04.111225+0000","last_fullsized":"2024-09-17T07:17:04.111225+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:33.658293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:16:33.658293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:46.850647+0000","scrub_duration":0.044680275999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":21,"reported_epoch":315,"state":"active+undersized","last_fresh":"2024-09-17T07:17:04.112244+0000","last_change":"2024-09-17T07:16:17.372955+0000","last_active":"2024-09-17T07:17:04.112244+0000","last_peered":"2024-09-17T07:17:04.112244+0000","last_clean":"2024-09-17T07:02:14.999435+0000","last_became_active":"2024-09-17T07:16:17.372955+0000","last_became_peered":"2024-09-17T07:16:17.372955+0000","last_unstale":"2024-09-17T07:17:04.112244+0000","last_undegraded":"2024-09-17T07:17:04.112244+0000","last_fullsized":"2024-09-17T07:16:17.365245+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:46.502641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:13:46.502641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":172,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-17T07:17:04.112761+0000","last_change":"2024-09-17T07:16:34.161259+0000","last_active":"2024-09-17T07:17:04.112761+0000","last_peered":"2024-09-17T07:17:04.112761+0000","last_clean":"2024-09-17T07:17:04.112761+0000","last_became_active":"2024-09-17T07:09:24.396334+0000","last_became_peered":"2024-09-17T07:09:24.396334+0000","last_unstale":"2024-09-17T07:17:04.112761+0000","last_undegraded":"2024-09-17T07:17:04.112761+0000","last_fullsized":"2024-09-17T07:17:04.112761+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:34.161180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:42.984088+0000","last_clean_scrub_stamp":"2024-09-17T07:16:34.161180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:36.434870+0000","scrub_duration":0.001181355,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2","version":"316'4904","reported_seq":4925,"reported_epoch":316,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:17:08.518664+0000","last_change":"2024-09-17T07:16:18.633028+0000","last_active":"2024-09-17T07:17:08.518664+0000","last_peered":"2024-09-17T07:17:08.518664+0000","last_clean":"2024-09-17T07:16:04.025356+0000","last_became_active":"2024-09-17T07:16:17.372517+0000","last_became_peered":"2024-09-17T07:16:17.372517+0000","last_unstale":"2024-09-17T07:17:08.518664+0000","last_undegraded":"2024-09-17T07:16:18.629077+0000","last_fullsized":"2024-09-17T07:16:17.365507+0000","mapping_epoch":303,"log_start":"304'300","ondisk_log_start":"304'300","created":297,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":4604,"log_dups_size":0,"ondisk_log_size":4604,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:23.117974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":321388544,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4904,"num_write_kb":313856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[{"shards":"0,4","objects":4904}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"242'25","reported_seq":390,"reported_epoch":315,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:17:04.112450+0000","last_change":"2024-09-17T07:16:17.381025+0000","last_active":"2024-09-17T07:17:04.112450+0000","last_peered":"2024-09-17T07:17:04.112450+0000","last_clean":"2024-09-17T07:16:15.358419+0000","last_became_active":"2024-09-17T07:16:17.381025+0000","last_became_peered":"2024-09-17T07:16:17.381025+0000","last_unstale":"2024-09-17T07:17:04.112450+0000","last_undegraded":"2024-09-17T07:16:17.367122+0000","last_fullsized":"2024-09-17T07:16:17.365635+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'25","last_scrub_stamp":"2024-09-17T07:15:02.532371+0000","last_deep_scrub":"176'15","last_deep_scrub_stamp":"2024-09-17T07:08:19.633791+0000","last_clean_scrub_stamp":"2024-09-17T07:15:02.532371+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012712606,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":193,"reported_epoch":315,"state":"active+undersized","last_fresh":"2024-09-17T07:17:04.110489+0000","last_change":"2024-09-17T07:16:17.379662+0000","last_active":"2024-09-17T07:17:04.110489+0000","last_peered":"2024-09-17T07:17:04.110489+0000","last_clean":"2024-09-17T07:09:22.372452+0000","last_became_active":"2024-09-17T07:16:17.379662+0000","last_became_peered":"2024-09-17T07:16:17.379662+0000","last_unstale":"2024-09-17T07:17:04.110489+0000","last_undegraded":"2024-09-17T07:17:04.110489+0000","last_fullsized":"2024-09-17T07:16:17.370523+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:59.697560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:14:59.697560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0065215869999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.d","version":"98'9","reported_seq":452,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-17T07:17:05.271963+0000","last_change":"2024-09-17T07:14:46.126177+0000","last_active":"2024-09-17T07:17:05.271963+0000","last_peered":"2024-09-17T07:17:05.271963+0000","last_clean":"2024-09-17T07:17:05.271963+0000","last_became_active":"2024-09-17T07:04:15.110797+0000","last_became_peered":"2024-09-17T07:04:15.110797+0000","last_unstale":"2024-09-17T07:17:05.271963+0000","last_undegraded":"2024-09-17T07:17:05.271963+0000","last_fullsized":"2024-09-17T07:17:05.271963+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-09-17T07:14:46.126045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:14:46.126045+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.014185467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":330,"reported_epoch":315,"state":"active+undersized","last_fresh":"2024-09-17T07:17:04.113647+0000","last_change":"2024-09-17T07:16:17.373630+0000","last_active":"2024-09-17T07:17:04.113647+0000","last_peered":"2024-09-17T07:17:04.113647+0000","last_clean":"2024-09-17T07:16:15.358526+0000","last_became_active":"2024-09-17T07:16:17.373630+0000","last_became_peered":"2024-09-17T07:16:17.373630+0000","last_unstale":"2024-09-17T07:17:04.113647+0000","last_undegraded":"2024-09-17T07:17:04.113647+0000","last_fullsized":"2024-09-17T07:16:17.369047+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:50.554591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:14:50.554591+0000","last_clean_scrub_stamp":"2024-09-17T07:14:50.554591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0067125969999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1","version":"316'5244","reported_seq":5271,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-17T07:17:08.535499+0000","last_change":"2024-09-17T07:16:05.051123+0000","last_active":"2024-09-17T07:17:08.535499+0000","last_peered":"2024-09-17T07:17:08.535499+0000","last_clean":"2024-09-17T07:17:08.535499+0000","last_became_active":"2024-09-17T07:16:05.050872+0000","last_became_peered":"2024-09-17T07:16:05.050872+0000","last_unstale":"2024-09-17T07:17:08.535499+0000","last_undegraded":"2024-09-17T07:17:08.535499+0000","last_fullsized":"2024-09-17T07:17:08.535499+0000","mapping_epoch":297,"log_start":"305'600","ondisk_log_start":"305'600","created":297,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":4644,"log_dups_size":0,"ondisk_log_size":4644,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":343670784,"num_objects":5244,"num_object_clones":0,"num_object_copies":15732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5244,"num_write_kb":335616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"242'27","reported_seq":315,"reported_epoch":315,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:17:04.112963+0000","last_change":"2024-09-17T07:16:17.382238+0000","last_active":"2024-09-17T07:17:04.112963+0000","last_peered":"2024-09-17T07:17:04.112963+0000","last_clean":"2024-09-17T07:12:20.775697+0000","last_became_active":"2024-09-17T07:16:17.382238+0000","last_became_peered":"2024-09-17T07:16:17.382238+0000","last_unstale":"2024-09-17T07:17:04.112963+0000","last_undegraded":"2024-09-17T07:16:17.365468+0000","last_fullsized":"2024-09-17T07:16:17.365334+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'27","last_scrub_stamp":"2024-09-17T07:14:45.831734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:14:45.831734+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021853620000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[{"shards":"4,7","objects":7}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":194,"reported_epoch":315,"state":"active+undersized","last_fresh":"2024-09-17T07:17:04.111192+0000","last_change":"2024-09-17T07:16:17.378599+0000","last_active":"2024-09-17T07:17:04.111192+0000","last_peered":"2024-09-17T07:17:04.111192+0000","last_clean":"2024-09-17T07:09:22.371958+0000","last_became_active":"2024-09-17T07:16:17.378599+0000","last_became_peered":"2024-09-17T07:16:17.378599+0000","last_unstale":"2024-09-17T07:17:04.111192+0000","last_undegraded":"2024-09-17T07:17:04.111192+0000","last_fullsized":"2024-09-17T07:16:17.373236+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:44.897611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:15:44.897611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:08.126551+0000","scrub_duration":0.0034116369999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.e","version":"75'8","reported_seq":513,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-17T07:17:05.261220+0000","last_change":"2024-09-17T07:16:49.331611+0000","last_active":"2024-09-17T07:17:05.261220+0000","last_peered":"2024-09-17T07:17:05.261220+0000","last_clean":"2024-09-17T07:17:05.261220+0000","last_became_active":"2024-09-17T07:02:16.021762+0000","last_became_peered":"2024-09-17T07:02:16.021762+0000","last_unstale":"2024-09-17T07:17:05.261220+0000","last_undegraded":"2024-09-17T07:17:05.261220+0000","last_fullsized":"2024-09-17T07:17:05.261220+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:16:49.331407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:16:49.331407+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:57.973202+0000","scrub_duration":0.0033687029999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":328,"reported_epoch":315,"state":"active+undersized","last_fresh":"2024-09-17T07:17:04.115814+0000","last_change":"2024-09-17T07:16:17.374897+0000","last_active":"2024-09-17T07:17:04.115814+0000","last_peered":"2024-09-17T07:17:04.115814+0000","last_clean":"2024-09-17T07:16:16.170588+0000","last_became_active":"2024-09-17T07:16:17.374897+0000","last_became_peered":"2024-09-17T07:16:17.374897+0000","last_unstale":"2024-09-17T07:17:04.115814+0000","last_undegraded":"2024-09-17T07:17:04.115814+0000","last_fullsized":"2024-09-17T07:16:17.368895+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:56.146098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:14:56.146098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0054479960000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.0","version":"316'5019","reported_seq":5052,"reported_epoch":316,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:17:08.400631+0000","last_change":"2024-09-17T07:16:18.637531+0000","last_active":"2024-09-17T07:17:08.400631+0000","last_peered":"2024-09-17T07:17:08.400631+0000","last_clean":"2024-09-17T07:16:14.245624+0000","last_became_active":"2024-09-17T07:16:17.372069+0000","last_became_peered":"2024-09-17T07:16:17.372069+0000","last_unstale":"2024-09-17T07:17:08.400631+0000","last_undegraded":"2024-09-17T07:16:18.630601+0000","last_fullsized":"2024-09-17T07:16:17.362738+0000","mapping_epoch":303,"log_start":"304'200","ondisk_log_start":"304'200","created":297,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":4819,"log_dups_size":0,"ondisk_log_size":4819,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:19.243451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328925184,"num_objects":5019,"num_object_clones":0,"num_object_copies":15057,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5019,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5019,"num_write_kb":321216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[{"shards":"4,6","objects":5019}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"242'19","reported_seq":365,"reported_epoch":315,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:17:04.113884+0000","last_change":"2024-09-17T07:16:17.382184+0000","last_active":"2024-09-17T07:17:04.113884+0000","last_peered":"2024-09-17T07:17:04.113884+0000","last_clean":"2024-09-17T07:16:15.358681+0000","last_became_active":"2024-09-17T07:16:17.382184+0000","last_became_peered":"2024-09-17T07:16:17.382184+0000","last_unstale":"2024-09-17T07:17:04.113884+0000","last_undegraded":"2024-09-17T07:16:17.369179+0000","last_fullsized":"2024-09-17T07:16:17.369087+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'19","last_scrub_stamp":"2024-09-17T07:14:55.489874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:14:55.489874+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045924440000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":324,"reported_epoch":315,"state":"active+undersized","last_fresh":"2024-09-17T07:17:04.115742+0000","last_change":"2024-09-17T07:16:17.375239+0000","last_active":"2024-09-17T07:17:04.115742+0000","last_peered":"2024-09-17T07:17:04.115742+0000","last_clean":"2024-09-17T07:16:15.590006+0000","last_became_active":"2024-09-17T07:16:17.375239+0000","last_became_peered":"2024-09-17T07:16:17.375239+0000","last_unstale":"2024-09-17T07:17:04.115742+0000","last_undegraded":"2024-09-17T07:17:04.115742+0000","last_fullsized":"2024-09-17T07:16:17.372272+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:21.858127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:13:21.858127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033279709999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":335,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-17T07:17:04.115440+0000","last_change":"2024-09-17T07:15:53.424831+0000","last_active":"2024-09-17T07:17:04.115440+0000","last_peered":"2024-09-17T07:17:04.115440+0000","last_clean":"2024-09-17T07:17:04.115440+0000","last_became_active":"2024-09-17T07:02:16.020224+0000","last_became_peered":"2024-09-17T07:02:16.020224+0000","last_unstale":"2024-09-17T07:17:04.115440+0000","last_undegraded":"2024-09-17T07:17:04.115440+0000","last_fullsized":"2024-09-17T07:17:04.115440+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:53.424736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:15:53.424736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001270806,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":189,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-17T07:17:04.113574+0000","last_change":"2024-09-17T07:16:37.587187+0000","last_active":"2024-09-17T07:17:04.113574+0000","last_peered":"2024-09-17T07:17:04.113574+0000","last_clean":"2024-09-17T07:17:04.113574+0000","last_became_active":"2024-09-17T07:09:24.408109+0000","last_became_peered":"2024-09-17T07:09:24.408109+0000","last_unstale":"2024-09-17T07:17:04.113574+0000","last_undegraded":"2024-09-17T07:17:04.113574+0000","last_fullsized":"2024-09-17T07:17:04.113574+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:37.587106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:13:22.187127+0000","last_clean_scrub_stamp":"2024-09-17T07:16:37.587106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:59.564692+0000","scrub_duration":0.001128609,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.f","version":"316'5084","reported_seq":5111,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-17T07:17:08.373140+0000","last_change":"2024-09-17T07:16:05.052296+0000","last_active":"2024-09-17T07:17:08.373140+0000","last_peered":"2024-09-17T07:17:08.373140+0000","last_clean":"2024-09-17T07:17:08.373140+0000","last_became_active":"2024-09-17T07:16:05.052037+0000","last_became_peered":"2024-09-17T07:16:05.052037+0000","last_unstale":"2024-09-17T07:17:08.373140+0000","last_undegraded":"2024-09-17T07:17:08.373140+0000","last_fullsized":"2024-09-17T07:17:08.373140+0000","mapping_epoch":297,"log_start":"304'300","ondisk_log_start":"304'300","created":297,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":4784,"log_dups_size":0,"ondisk_log_size":4784,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:13.511460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":333185024,"num_objects":5084,"num_object_clones":0,"num_object_copies":15252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5084,"num_write_kb":325376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"242'28","reported_seq":360,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-17T07:17:04.109748+0000","last_change":"2024-09-17T07:15:43.270581+0000","last_active":"2024-09-17T07:17:04.109748+0000","last_peered":"2024-09-17T07:17:04.109748+0000","last_clean":"2024-09-17T07:17:04.109748+0000","last_became_active":"2024-09-17T07:02:14.007810+0000","last_became_peered":"2024-09-17T07:02:14.007810+0000","last_unstale":"2024-09-17T07:17:04.109748+0000","last_undegraded":"2024-09-17T07:17:04.109748+0000","last_fullsized":"2024-09-17T07:17:04.109748+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'28","last_scrub_stamp":"2024-09-17T07:15:43.270473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:15:43.270473+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:07.514992+0000","scrub_duration":0.0022765149999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":360,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-17T07:17:04.115260+0000","last_change":"2024-09-17T07:15:58.575503+0000","last_active":"2024-09-17T07:17:04.115260+0000","last_peered":"2024-09-17T07:17:04.115260+0000","last_clean":"2024-09-17T07:17:04.115260+0000","last_became_active":"2024-09-17T07:02:12.011766+0000","last_became_peered":"2024-09-17T07:02:12.011766+0000","last_unstale":"2024-09-17T07:17:04.115260+0000","last_undegraded":"2024-09-17T07:17:04.115260+0000","last_fullsized":"2024-09-17T07:17:04.115260+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:58.575409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:09:23.181028+0000","last_clean_scrub_stamp":"2024-09-17T07:15:58.575409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:05.657241+0000","scrub_duration":0.0012800719999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"304'10","reported_seq":52,"reported_epoch":316,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:17:05.258830+0000","last_change":"2024-09-17T07:16:17.371678+0000","last_active":"2024-09-17T07:17:05.258830+0000","last_peered":"2024-09-17T07:17:05.258830+0000","last_clean":"2024-09-17T07:02:19.145975+0000","last_became_active":"2024-09-17T07:16:17.371678+0000","last_became_peered":"2024-09-17T07:16:17.371678+0000","last_unstale":"2024-09-17T07:17:05.258830+0000","last_undegraded":"2024-09-17T07:16:17.366242+0000","last_fullsized":"2024-09-17T07:16:17.365352+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'9","last_scrub_stamp":"2024-09-17T07:15:55.813882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:15:55.813882+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:18.800760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[{"shards":"0,4","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":323,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-17T07:17:04.110901+0000","last_change":"2024-09-17T07:15:52.378479+0000","last_active":"2024-09-17T07:17:04.110901+0000","last_peered":"2024-09-17T07:17:04.110901+0000","last_clean":"2024-09-17T07:17:04.110901+0000","last_became_active":"2024-09-17T07:02:18.364981+0000","last_became_peered":"2024-09-17T07:02:18.364981+0000","last_unstale":"2024-09-17T07:17:04.110901+0000","last_undegraded":"2024-09-17T07:17:04.110901+0000","last_fullsized":"2024-09-17T07:17:04.110901+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:52.378384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:10:33.727472+0000","last_clean_scrub_stamp":"2024-09-17T07:15:52.378384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:07.710387+0000","scrub_duration":0.001396736,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"316'4975","reported_seq":5003,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-17T07:17:08.530224+0000","last_change":"2024-09-17T07:16:05.054368+0000","last_active":"2024-09-17T07:17:08.530224+0000","last_peered":"2024-09-17T07:17:08.530224+0000","last_clean":"2024-09-17T07:17:08.530224+0000","last_became_active":"2024-09-17T07:16:05.054109+0000","last_became_peered":"2024-09-17T07:16:05.054109+0000","last_unstale":"2024-09-17T07:17:08.530224+0000","last_undegraded":"2024-09-17T07:17:08.530224+0000","last_fullsized":"2024-09-17T07:17:08.530224+0000","mapping_epoch":297,"log_start":"305'400","ondisk_log_start":"305'400","created":297,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":4575,"log_dups_size":0,"ondisk_log_size":4575,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:25.923906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":326041600,"num_objects":4975,"num_object_clones":0,"num_object_copies":14925,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4975,"num_write_kb":318400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"242'30","reported_seq":243,"reported_epoch":315,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:17:04.115526+0000","last_change":"2024-09-17T07:16:17.378001+0000","last_active":"2024-09-17T07:17:04.115526+0000","last_peered":"2024-09-17T07:17:04.115526+0000","last_clean":"2024-09-17T07:16:15.589841+0000","last_became_active":"2024-09-17T07:16:17.378001+0000","last_became_peered":"2024-09-17T07:16:17.378001+0000","last_unstale":"2024-09-17T07:17:04.115526+0000","last_undegraded":"2024-09-17T07:16:17.372487+0000","last_fullsized":"2024-09-17T07:16:17.372376+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'30","last_scrub_stamp":"2024-09-17T07:13:28.997563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:13:28.997563+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0061961159999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[{"shards":"0,7","objects":8}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":21,"reported_epoch":315,"state":"active+undersized","last_fresh":"2024-09-17T07:17:04.114383+0000","last_change":"2024-09-17T07:16:17.376547+0000","last_active":"2024-09-17T07:17:04.114383+0000","last_peered":"2024-09-17T07:17:04.114383+0000","last_clean":"2024-09-17T07:02:10.991363+0000","last_became_active":"2024-09-17T07:16:17.376547+0000","last_became_peered":"2024-09-17T07:16:17.376547+0000","last_unstale":"2024-09-17T07:17:04.114383+0000","last_undegraded":"2024-09-17T07:17:04.114383+0000","last_fullsized":"2024-09-17T07:16:17.364886+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:56.848689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:56.848689+0000","last_clean_scrub_stamp":"2024-09-17T07:15:56.848689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:18.966156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":336,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-17T07:17:04.109612+0000","last_change":"2024-09-17T07:16:38.184957+0000","last_active":"2024-09-17T07:17:04.109612+0000","last_peered":"2024-09-17T07:17:04.109612+0000","last_clean":"2024-09-17T07:17:04.109612+0000","last_became_active":"2024-09-17T07:09:24.393375+0000","last_became_peered":"2024-09-17T07:09:24.393375+0000","last_unstale":"2024-09-17T07:17:04.109612+0000","last_undegraded":"2024-09-17T07:17:04.109612+0000","last_fullsized":"2024-09-17T07:17:04.109612+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:38.184818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:16:38.184818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:55.885718+0000","scrub_duration":0.0076357700000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":334,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-17T07:17:04.113345+0000","last_change":"2024-09-17T07:16:36.162077+0000","last_active":"2024-09-17T07:17:04.113345+0000","last_peered":"2024-09-17T07:17:04.113345+0000","last_clean":"2024-09-17T07:17:04.113345+0000","last_became_active":"2024-09-17T07:09:24.398659+0000","last_became_peered":"2024-09-17T07:09:24.398659+0000","last_unstale":"2024-09-17T07:17:04.113345+0000","last_undegraded":"2024-09-17T07:17:04.113345+0000","last_fullsized":"2024-09-17T07:17:04.113345+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:36.161622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:16:36.161622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:03.772691+0000","scrub_duration":0.0062812850000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.d","version":"316'5070","reported_seq":5097,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-17T07:17:08.287756+0000","last_change":"2024-09-17T07:16:05.052409+0000","last_active":"2024-09-17T07:17:08.287756+0000","last_peered":"2024-09-17T07:17:08.287756+0000","last_clean":"2024-09-17T07:17:08.287756+0000","last_became_active":"2024-09-17T07:16:05.052153+0000","last_became_peered":"2024-09-17T07:16:05.052153+0000","last_unstale":"2024-09-17T07:17:08.287756+0000","last_undegraded":"2024-09-17T07:17:08.287756+0000","last_fullsized":"2024-09-17T07:17:08.287756+0000","mapping_epoch":297,"log_start":"304'300","ondisk_log_start":"304'300","created":297,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":4770,"log_dups_size":0,"ondisk_log_size":4770,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:17.269415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332267520,"num_objects":5070,"num_object_clones":0,"num_object_copies":15210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5070,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5070,"num_write_kb":324480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"242'29","reported_seq":369,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-17T07:17:04.103897+0000","last_change":"2024-09-17T07:15:38.507145+0000","last_active":"2024-09-17T07:17:04.103897+0000","last_peered":"2024-09-17T07:17:04.103897+0000","last_clean":"2024-09-17T07:17:04.103897+0000","last_became_active":"2024-09-17T07:02:14.027389+0000","last_became_peered":"2024-09-17T07:02:14.027389+0000","last_unstale":"2024-09-17T07:17:04.103897+0000","last_undegraded":"2024-09-17T07:17:04.103897+0000","last_fullsized":"2024-09-17T07:17:04.103897+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'29","last_scrub_stamp":"2024-09-17T07:15:38.507042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:15:38.507042+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0024959420000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":330,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-17T07:17:04.117401+0000","last_change":"2024-09-17T07:16:47.631557+0000","last_active":"2024-09-17T07:17:04.117401+0000","last_peered":"2024-09-17T07:17:04.117401+0000","last_clean":"2024-09-17T07:17:04.117401+0000","last_became_active":"2024-09-17T07:02:12.011917+0000","last_became_peered":"2024-09-17T07:02:12.011917+0000","last_unstale":"2024-09-17T07:17:04.117401+0000","last_undegraded":"2024-09-17T07:17:04.117401+0000","last_fullsized":"2024-09-17T07:17:04.117401+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:47.631387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:47.631387+0000","last_clean_scrub_stamp":"2024-09-17T07:16:47.631387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:12.824261+0000","scrub_duration":0.006082372,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":336,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-17T07:17:04.113914+0000","last_change":"2024-09-17T07:15:56.525463+0000","last_active":"2024-09-17T07:17:04.113914+0000","last_peered":"2024-09-17T07:17:04.113914+0000","last_clean":"2024-09-17T07:17:04.113914+0000","last_became_active":"2024-09-17T07:02:16.016199+0000","last_became_peered":"2024-09-17T07:02:16.016199+0000","last_unstale":"2024-09-17T07:17:04.113914+0000","last_undegraded":"2024-09-17T07:17:04.113914+0000","last_fullsized":"2024-09-17T07:17:04.113914+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:56.525361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:11:17.583712+0000","last_clean_scrub_stamp":"2024-09-17T07:15:56.525361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:12.314220+0000","scrub_duration":0.0013436679999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":21,"reported_epoch":315,"state":"active+undersized","last_fresh":"2024-09-17T07:17:04.111001+0000","last_change":"2024-09-17T07:16:17.368186+0000","last_active":"2024-09-17T07:17:04.111001+0000","last_peered":"2024-09-17T07:17:04.111001+0000","last_clean":"2024-09-17T07:02:17.002904+0000","last_became_active":"2024-09-17T07:16:17.368186+0000","last_became_peered":"2024-09-17T07:16:17.368186+0000","last_unstale":"2024-09-17T07:17:04.111001+0000","last_undegraded":"2024-09-17T07:17:04.111001+0000","last_fullsized":"2024-09-17T07:16:17.362908+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:50.514525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:13:50.514525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.c","version":"316'5089","reported_seq":5117,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-17T07:17:08.895961+0000","last_change":"2024-09-17T07:16:05.057855+0000","last_active":"2024-09-17T07:17:08.895961+0000","last_peered":"2024-09-17T07:17:08.895961+0000","last_clean":"2024-09-17T07:17:08.895961+0000","last_became_active":"2024-09-17T07:16:05.057588+0000","last_became_peered":"2024-09-17T07:16:05.057588+0000","last_unstale":"2024-09-17T07:17:08.895961+0000","last_undegraded":"2024-09-17T07:17:08.895961+0000","last_fullsized":"2024-09-17T07:17:08.895961+0000","mapping_epoch":297,"log_start":"305'500","ondisk_log_start":"305'500","created":297,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":4589,"log_dups_size":0,"ondisk_log_size":4589,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:20.816387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":333512704,"num_objects":5089,"num_object_clones":0,"num_object_copies":15267,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5089,"num_write_kb":325696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"242'25","reported_seq":282,"reported_epoch":315,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:17:04.111571+0000","last_change":"2024-09-17T07:16:17.376630+0000","last_active":"2024-09-17T07:17:04.111571+0000","last_peered":"2024-09-17T07:17:04.111571+0000","last_clean":"2024-09-17T07:12:20.656798+0000","last_became_active":"2024-09-17T07:16:17.376630+0000","last_became_peered":"2024-09-17T07:16:17.376630+0000","last_unstale":"2024-09-17T07:17:04.111571+0000","last_undegraded":"2024-09-17T07:16:17.368029+0000","last_fullsized":"2024-09-17T07:16:17.367816+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'25","last_scrub_stamp":"2024-09-17T07:14:50.794200+0000","last_deep_scrub":"110'15","last_deep_scrub_stamp":"2024-09-17T07:04:57.091400+0000","last_clean_scrub_stamp":"2024-09-17T07:14:50.794200+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0070558269999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[{"shards":"4,5","objects":5}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":315,"state":"active+undersized","last_fresh":"2024-09-17T07:17:04.105207+0000","last_change":"2024-09-17T07:16:17.378414+0000","last_active":"2024-09-17T07:17:04.105207+0000","last_peered":"2024-09-17T07:17:04.105207+0000","last_clean":"2024-09-17T07:02:10.991363+0000","last_became_active":"2024-09-17T07:16:17.378414+0000","last_became_peered":"2024-09-17T07:16:17.378414+0000","last_unstale":"2024-09-17T07:17:04.105207+0000","last_undegraded":"2024-09-17T07:17:04.105207+0000","last_fullsized":"2024-09-17T07:16:17.365679+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:49.764885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:25.147878+0000","last_clean_scrub_stamp":"2024-09-17T07:14:49.764885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"89'9","reported_seq":514,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-17T07:17:05.243553+0000","last_change":"2024-09-17T07:15:53.512860+0000","last_active":"2024-09-17T07:17:05.243553+0000","last_peered":"2024-09-17T07:17:05.243553+0000","last_clean":"2024-09-17T07:17:05.243553+0000","last_became_active":"2024-09-17T07:02:16.021360+0000","last_became_peered":"2024-09-17T07:02:16.021360+0000","last_unstale":"2024-09-17T07:17:05.243553+0000","last_undegraded":"2024-09-17T07:17:05.243553+0000","last_fullsized":"2024-09-17T07:17:05.243553+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'9","last_scrub_stamp":"2024-09-17T07:15:53.512777+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-09-17T07:15:53.512777+0000","last_clean_scrub_stamp":"2024-09-17T07:15:53.512777+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0020298600000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"268'2","reported_seq":339,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-17T07:17:04.102379+0000","last_change":"2024-09-17T07:16:46.448318+0000","last_active":"2024-09-17T07:17:04.102379+0000","last_peered":"2024-09-17T07:17:04.102379+0000","last_clean":"2024-09-17T07:17:04.102379+0000","last_became_active":"2024-09-17T07:09:24.399565+0000","last_became_peered":"2024-09-17T07:09:24.399565+0000","last_unstale":"2024-09-17T07:17:04.102379+0000","last_undegraded":"2024-09-17T07:17:04.102379+0000","last_fullsized":"2024-09-17T07:17:04.102379+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'2","last_scrub_stamp":"2024-09-17T07:16:46.448196+0000","last_deep_scrub":"268'2","last_deep_scrub_stamp":"2024-09-17T07:14:03.609131+0000","last_clean_scrub_stamp":"2024-09-17T07:16:46.448196+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:12.835807+0000","scrub_duration":0.00244297,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.a","version":"316'4911","reported_seq":4937,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-17T07:17:08.382177+0000","last_change":"2024-09-17T07:16:05.055275+0000","last_active":"2024-09-17T07:17:08.382177+0000","last_peered":"2024-09-17T07:17:08.382177+0000","last_clean":"2024-09-17T07:17:08.382177+0000","last_became_active":"2024-09-17T07:16:05.055008+0000","last_became_peered":"2024-09-17T07:16:05.055008+0000","last_unstale":"2024-09-17T07:17:08.382177+0000","last_undegraded":"2024-09-17T07:17:08.382177+0000","last_fullsized":"2024-09-17T07:17:08.382177+0000","mapping_epoch":297,"log_start":"304'100","ondisk_log_start":"304'100","created":297,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":4811,"log_dups_size":0,"ondisk_log_size":4811,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:20.759555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":321847296,"num_objects":4911,"num_object_clones":0,"num_object_copies":14733,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4911,"num_write_kb":314304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"304'42","reported_seq":477,"reported_epoch":316,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:17:05.243942+0000","last_change":"2024-09-17T07:16:17.368142+0000","last_active":"2024-09-17T07:17:05.243942+0000","last_peered":"2024-09-17T07:17:05.243942+0000","last_clean":"2024-09-17T07:12:20.666266+0000","last_became_active":"2024-09-17T07:16:17.368142+0000","last_became_peered":"2024-09-17T07:16:17.368142+0000","last_unstale":"2024-09-17T07:17:05.243942+0000","last_undegraded":"2024-09-17T07:16:17.363645+0000","last_fullsized":"2024-09-17T07:16:17.363541+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'41","last_scrub_stamp":"2024-09-17T07:14:56.700858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:14:56.700858+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026058610000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[{"shards":"2,6","objects":10}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":335,"reported_epoch":315,"state":"active+undersized","last_fresh":"2024-09-17T07:17:04.111922+0000","last_change":"2024-09-17T07:16:17.371561+0000","last_active":"2024-09-17T07:17:04.111922+0000","last_peered":"2024-09-17T07:17:04.111922+0000","last_clean":"2024-09-17T07:16:16.170577+0000","last_became_active":"2024-09-17T07:16:17.371561+0000","last_became_peered":"2024-09-17T07:16:17.371561+0000","last_unstale":"2024-09-17T07:17:04.111922+0000","last_undegraded":"2024-09-17T07:17:04.111922+0000","last_fullsized":"2024-09-17T07:16:17.366918+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:03.045709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:14:03.045709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0049245929999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":332,"reported_epoch":315,"state":"active+undersized","last_fresh":"2024-09-17T07:17:04.104609+0000","last_change":"2024-09-17T07:16:17.375376+0000","last_active":"2024-09-17T07:17:04.104609+0000","last_peered":"2024-09-17T07:17:04.104609+0000","last_clean":"2024-09-17T07:16:15.495109+0000","last_became_active":"2024-09-17T07:16:17.375376+0000","last_became_peered":"2024-09-17T07:16:17.375376+0000","last_unstale":"2024-09-17T07:17:04.104609+0000","last_undegraded":"2024-09-17T07:17:04.104609+0000","last_fullsized":"2024-09-17T07:16:17.364236+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:36.523676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:15:36.523676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017370549999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":169,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-17T07:17:04.112837+0000","last_change":"2024-09-17T07:15:55.123577+0000","last_active":"2024-09-17T07:17:04.112837+0000","last_peered":"2024-09-17T07:17:04.112837+0000","last_clean":"2024-09-17T07:17:04.112837+0000","last_became_active":"2024-09-17T07:09:24.398878+0000","last_became_peered":"2024-09-17T07:09:24.398878+0000","last_unstale":"2024-09-17T07:17:04.112837+0000","last_undegraded":"2024-09-17T07:17:04.112837+0000","last_fullsized":"2024-09-17T07:17:04.112837+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:55.123464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:13:53.406661+0000","last_clean_scrub_stamp":"2024-09-17T07:15:55.123464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014856559999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.9","version":"316'4974","reported_seq":5002,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-17T07:17:08.679260+0000","last_change":"2024-09-17T07:16:05.057109+0000","last_active":"2024-09-17T07:17:08.679260+0000","last_peered":"2024-09-17T07:17:08.679260+0000","last_clean":"2024-09-17T07:17:08.679260+0000","last_became_active":"2024-09-17T07:16:05.056719+0000","last_became_peered":"2024-09-17T07:16:05.056719+0000","last_unstale":"2024-09-17T07:17:08.679260+0000","last_undegraded":"2024-09-17T07:17:08.679260+0000","last_fullsized":"2024-09-17T07:17:08.679260+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":297,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":4974,"log_dups_size":0,"ondisk_log_size":4974,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:09.010134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":325976064,"num_objects":4974,"num_object_clones":0,"num_object_copies":14922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4974,"num_write_kb":318336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"242'17","reported_seq":280,"reported_epoch":315,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:17:04.115652+0000","last_change":"2024-09-17T07:16:17.371977+0000","last_active":"2024-09-17T07:17:04.115652+0000","last_peered":"2024-09-17T07:17:04.115652+0000","last_clean":"2024-09-17T07:12:20.284331+0000","last_became_active":"2024-09-17T07:16:17.371977+0000","last_became_peered":"2024-09-17T07:16:17.371977+0000","last_unstale":"2024-09-17T07:17:04.115652+0000","last_undegraded":"2024-09-17T07:16:17.367143+0000","last_fullsized":"2024-09-17T07:16:17.367038+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'17","last_scrub_stamp":"2024-09-17T07:13:47.468399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:13:47.468399+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.005881725,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[{"shards":"0,5","objects":5}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"76'5","reported_seq":101,"reported_epoch":316,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:17:06.700834+0000","last_change":"2024-09-17T07:16:17.368338+0000","last_active":"2024-09-17T07:17:06.700834+0000","last_peered":"2024-09-17T07:17:06.700834+0000","last_clean":"2024-09-17T07:02:23.985427+0000","last_became_active":"2024-09-17T07:16:17.368338+0000","last_became_peered":"2024-09-17T07:16:17.368338+0000","last_unstale":"2024-09-17T07:17:06.700834+0000","last_undegraded":"2024-09-17T07:16:17.363185+0000","last_fullsized":"2024-09-17T07:16:17.363086+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'5","last_scrub_stamp":"2024-09-17T07:15:36.901601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:15:36.901601+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033739799999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":57,"num_read_kb":52,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[{"shards":"2,6","objects":1}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":176,"reported_epoch":315,"state":"active+undersized","last_fresh":"2024-09-17T07:17:04.103842+0000","last_change":"2024-09-17T07:16:17.367238+0000","last_active":"2024-09-17T07:17:04.103842+0000","last_peered":"2024-09-17T07:17:04.103842+0000","last_clean":"2024-09-17T07:16:15.494771+0000","last_became_active":"2024-09-17T07:16:17.367238+0000","last_became_peered":"2024-09-17T07:16:17.367238+0000","last_unstale":"2024-09-17T07:17:04.103842+0000","last_undegraded":"2024-09-17T07:17:04.103842+0000","last_fullsized":"2024-09-17T07:16:17.364334+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:56.534547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:23.833170+0000","last_clean_scrub_stamp":"2024-09-17T07:15:56.534547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:07.380435+0000","scrub_duration":0.0011018950000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":348,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-17T07:17:04.104475+0000","last_change":"2024-09-17T07:16:46.778669+0000","last_active":"2024-09-17T07:17:04.104475+0000","last_peered":"2024-09-17T07:17:04.104475+0000","last_clean":"2024-09-17T07:17:04.104475+0000","last_became_active":"2024-09-17T07:02:16.014492+0000","last_became_peered":"2024-09-17T07:02:16.014492+0000","last_unstale":"2024-09-17T07:17:04.104475+0000","last_undegraded":"2024-09-17T07:17:04.104475+0000","last_fullsized":"2024-09-17T07:17:04.104475+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:46.778591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:16:46.778591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:53.201381+0000","scrub_duration":0.0013415079999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":332,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-17T07:17:04.117094+0000","last_change":"2024-09-17T07:15:02.062776+0000","last_active":"2024-09-17T07:17:04.117094+0000","last_peered":"2024-09-17T07:17:04.117094+0000","last_clean":"2024-09-17T07:17:04.117094+0000","last_became_active":"2024-09-17T07:09:24.402704+0000","last_became_peered":"2024-09-17T07:09:24.402704+0000","last_unstale":"2024-09-17T07:17:04.117094+0000","last_undegraded":"2024-09-17T07:17:04.117094+0000","last_fullsized":"2024-09-17T07:17:04.117094+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:02.062618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:02.062618+0000","last_clean_scrub_stamp":"2024-09-17T07:15:02.062618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0042833460000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"316'5123","reported_seq":5151,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-17T07:17:09.537145+0000","last_change":"2024-09-17T07:16:05.054358+0000","last_active":"2024-09-17T07:17:09.537145+0000","last_peered":"2024-09-17T07:17:09.537145+0000","last_clean":"2024-09-17T07:17:09.537145+0000","last_became_active":"2024-09-17T07:16:05.053996+0000","last_became_peered":"2024-09-17T07:16:05.053996+0000","last_unstale":"2024-09-17T07:17:09.537145+0000","last_undegraded":"2024-09-17T07:17:09.537145+0000","last_fullsized":"2024-09-17T07:17:09.537145+0000","mapping_epoch":297,"log_start":"305'400","ondisk_log_start":"305'400","created":297,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":4723,"log_dups_size":0,"ondisk_log_size":4723,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:31.098761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335740928,"num_objects":5123,"num_object_clones":0,"num_object_copies":15369,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5123,"num_write_kb":327872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"268'30","reported_seq":392,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-17T07:17:04.103649+0000","last_change":"2024-09-17T07:16:43.375355+0000","last_active":"2024-09-17T07:17:04.103649+0000","last_peered":"2024-09-17T07:17:04.103649+0000","last_clean":"2024-09-17T07:17:04.103649+0000","last_became_active":"2024-09-17T07:03:48.760405+0000","last_became_peered":"2024-09-17T07:03:48.760405+0000","last_unstale":"2024-09-17T07:17:04.103649+0000","last_undegraded":"2024-09-17T07:17:04.103649+0000","last_fullsized":"2024-09-17T07:17:04.103649+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'30","last_scrub_stamp":"2024-09-17T07:16:43.375250+0000","last_deep_scrub":"268'30","last_deep_scrub_stamp":"2024-09-17T07:14:00.599656+0000","last_clean_scrub_stamp":"2024-09-17T07:16:43.375250+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:48.191852+0000","scrub_duration":0.006604922,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"70'1","reported_seq":338,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-17T07:17:04.104018+0000","last_change":"2024-09-17T07:15:01.575863+0000","last_active":"2024-09-17T07:17:04.104018+0000","last_peered":"2024-09-17T07:17:04.104018+0000","last_clean":"2024-09-17T07:17:04.104018+0000","last_became_active":"2024-09-17T07:09:24.402499+0000","last_became_peered":"2024-09-17T07:09:24.402499+0000","last_unstale":"2024-09-17T07:17:04.104018+0000","last_undegraded":"2024-09-17T07:17:04.104018+0000","last_fullsized":"2024-09-17T07:17:04.104018+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-17T07:15:01.575747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:15:01.575747+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037109349999999998,"objects_trimmed":0,"snaptrim_duration":0.0032298629999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":324,"reported_epoch":315,"state":"active+undersized","last_fresh":"2024-09-17T07:17:04.115906+0000","last_change":"2024-09-17T07:16:17.375018+0000","last_active":"2024-09-17T07:17:04.115906+0000","last_peered":"2024-09-17T07:17:04.115906+0000","last_clean":"2024-09-17T07:16:15.590088+0000","last_became_active":"2024-09-17T07:16:17.375018+0000","last_became_peered":"2024-09-17T07:16:17.375018+0000","last_unstale":"2024-09-17T07:17:04.115906+0000","last_undegraded":"2024-09-17T07:17:04.115906+0000","last_fullsized":"2024-09-17T07:16:17.370688+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:55.321202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:55.321202+0000","last_clean_scrub_stamp":"2024-09-17T07:15:55.321202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:13.360620+0000","scrub_duration":0.0014716950000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":330,"reported_epoch":315,"state":"active+undersized","last_fresh":"2024-09-17T07:17:04.115521+0000","last_change":"2024-09-17T07:16:17.375774+0000","last_active":"2024-09-17T07:17:04.115521+0000","last_peered":"2024-09-17T07:17:04.115521+0000","last_clean":"2024-09-17T07:16:16.169907+0000","last_became_active":"2024-09-17T07:16:17.375774+0000","last_became_peered":"2024-09-17T07:16:17.375774+0000","last_unstale":"2024-09-17T07:17:04.115521+0000","last_undegraded":"2024-09-17T07:17:04.115521+0000","last_fullsized":"2024-09-17T07:16:17.368453+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:45.447029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:15:45.447029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001990463,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":187,"reported_epoch":315,"state":"active+undersized","last_fresh":"2024-09-17T07:17:04.114814+0000","last_change":"2024-09-17T07:16:17.372859+0000","last_active":"2024-09-17T07:17:04.114814+0000","last_peered":"2024-09-17T07:17:04.114814+0000","last_clean":"2024-09-17T07:09:22.371903+0000","last_became_active":"2024-09-17T07:16:17.372859+0000","last_became_peered":"2024-09-17T07:16:17.372859+0000","last_unstale":"2024-09-17T07:17:04.114814+0000","last_undegraded":"2024-09-17T07:17:04.114814+0000","last_fullsized":"2024-09-17T07:16:17.367296+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:51.877367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:15:51.877367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.058290885000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.b","version":"316'5063","reported_seq":5095,"reported_epoch":316,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:17:08.390623+0000","last_change":"2024-09-17T07:16:18.636164+0000","last_active":"2024-09-17T07:17:08.390623+0000","last_peered":"2024-09-17T07:17:08.390623+0000","last_clean":"2024-09-17T07:16:14.246161+0000","last_became_active":"2024-09-17T07:16:17.365889+0000","last_became_peered":"2024-09-17T07:16:17.365889+0000","last_unstale":"2024-09-17T07:17:08.390623+0000","last_undegraded":"2024-09-17T07:16:18.628529+0000","last_fullsized":"2024-09-17T07:16:17.362973+0000","mapping_epoch":303,"log_start":"304'300","ondisk_log_start":"304'300","created":297,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":4763,"log_dups_size":0,"ondisk_log_size":4763,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:19.972769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":331808768,"num_objects":5063,"num_object_clones":0,"num_object_copies":15189,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5063,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5063,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5063,"num_write_kb":324032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[{"shards":"0,6","objects":5063}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"242'26","reported_seq":379,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-17T07:17:04.123325+0000","last_change":"2024-09-17T07:16:33.667044+0000","last_active":"2024-09-17T07:17:04.123325+0000","last_peered":"2024-09-17T07:17:04.123325+0000","last_clean":"2024-09-17T07:17:04.123325+0000","last_became_active":"2024-09-17T07:02:14.028857+0000","last_became_peered":"2024-09-17T07:02:14.028857+0000","last_unstale":"2024-09-17T07:17:04.123325+0000","last_undegraded":"2024-09-17T07:17:04.123325+0000","last_fullsized":"2024-09-17T07:17:04.123325+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'26","last_scrub_stamp":"2024-09-17T07:16:33.666684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:16:33.666684+0000","objects_scrubbed":7,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:38.577635+0000","scrub_duration":0.013353596000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"268'3","reported_seq":188,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-17T07:17:04.111702+0000","last_change":"2024-09-17T07:15:43.516879+0000","last_active":"2024-09-17T07:17:04.111702+0000","last_peered":"2024-09-17T07:17:04.111702+0000","last_clean":"2024-09-17T07:17:04.111702+0000","last_became_active":"2024-09-17T07:09:24.404532+0000","last_became_peered":"2024-09-17T07:09:24.404532+0000","last_unstale":"2024-09-17T07:17:04.111702+0000","last_undegraded":"2024-09-17T07:17:04.111702+0000","last_fullsized":"2024-09-17T07:17:04.111702+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'3","last_scrub_stamp":"2024-09-17T07:15:43.516714+0000","last_deep_scrub":"268'3","last_deep_scrub_stamp":"2024-09-17T07:13:54.949345+0000","last_clean_scrub_stamp":"2024-09-17T07:15:43.516714+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:05.399081+0000","scrub_duration":0.002071711,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":340,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-17T07:17:04.116766+0000","last_change":"2024-09-17T07:15:55.432933+0000","last_active":"2024-09-17T07:17:04.116766+0000","last_peered":"2024-09-17T07:17:04.116766+0000","last_clean":"2024-09-17T07:17:04.116766+0000","last_became_active":"2024-09-17T07:02:12.004606+0000","last_became_peered":"2024-09-17T07:02:12.004606+0000","last_unstale":"2024-09-17T07:17:04.116766+0000","last_undegraded":"2024-09-17T07:17:04.116766+0000","last_fullsized":"2024-09-17T07:17:04.116766+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:55.432818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:15:55.432818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:23.510512+0000","scrub_duration":0.001411254,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"212'130","reported_seq":579,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-17T07:17:04.116075+0000","last_change":"2024-09-17T07:14:57.052759+0000","last_active":"2024-09-17T07:17:04.116075+0000","last_peered":"2024-09-17T07:17:04.116075+0000","last_clean":"2024-09-17T07:17:04.116075+0000","last_became_active":"2024-09-17T07:02:03.020675+0000","last_became_peered":"2024-09-17T07:02:03.020675+0000","last_unstale":"2024-09-17T07:17:04.116075+0000","last_undegraded":"2024-09-17T07:17:04.116075+0000","last_fullsized":"2024-09-17T07:17:04.116075+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:14:57.052505+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:14:57.052505+0000","last_clean_scrub_stamp":"2024-09-17T07:14:57.052505+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.022122466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":325,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-17T07:17:04.110205+0000","last_change":"2024-09-17T07:15:56.366761+0000","last_active":"2024-09-17T07:17:04.110205+0000","last_peered":"2024-09-17T07:17:04.110205+0000","last_clean":"2024-09-17T07:17:04.110205+0000","last_became_active":"2024-09-17T07:02:16.016031+0000","last_became_peered":"2024-09-17T07:02:16.016031+0000","last_unstale":"2024-09-17T07:17:04.110205+0000","last_undegraded":"2024-09-17T07:17:04.110205+0000","last_fullsized":"2024-09-17T07:17:04.110205+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:56.366669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:15:56.366669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:20.727553+0000","scrub_duration":0.0015953269999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":326,"reported_epoch":315,"state":"active+undersized","last_fresh":"2024-09-17T07:17:04.115954+0000","last_change":"2024-09-17T07:16:17.377095+0000","last_active":"2024-09-17T07:17:04.115954+0000","last_peered":"2024-09-17T07:17:04.115954+0000","last_clean":"2024-09-17T07:16:16.170840+0000","last_became_active":"2024-09-17T07:16:17.377095+0000","last_became_peered":"2024-09-17T07:16:17.377095+0000","last_unstale":"2024-09-17T07:17:04.115954+0000","last_undegraded":"2024-09-17T07:17:04.115954+0000","last_fullsized":"2024-09-17T07:16:17.370787+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:48.021592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:58.670571+0000","last_clean_scrub_stamp":"2024-09-17T07:14:48.021592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0064844619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.4","version":"316'5085","reported_seq":5119,"reported_epoch":316,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:17:09.527396+0000","last_change":"2024-09-17T07:16:18.635996+0000","last_active":"2024-09-17T07:17:09.527396+0000","last_peered":"2024-09-17T07:17:09.527396+0000","last_clean":"2024-09-17T07:16:15.495005+0000","last_became_active":"2024-09-17T07:16:17.375662+0000","last_became_peered":"2024-09-17T07:16:17.375662+0000","last_unstale":"2024-09-17T07:17:09.527396+0000","last_undegraded":"2024-09-17T07:16:18.629481+0000","last_fullsized":"2024-09-17T07:16:17.364553+0000","mapping_epoch":303,"log_start":"304'300","ondisk_log_start":"304'300","created":297,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":4785,"log_dups_size":0,"ondisk_log_size":4785,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":333250560,"num_objects":5085,"num_object_clones":0,"num_object_copies":15255,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5085,"num_write_kb":325440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0,4","objects":5085}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"242'35","reported_seq":399,"reported_epoch":315,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:17:04.114404+0000","last_change":"2024-09-17T07:16:17.376246+0000","last_active":"2024-09-17T07:17:04.114404+0000","last_peered":"2024-09-17T07:17:04.114404+0000","last_clean":"2024-09-17T07:16:14.244770+0000","last_became_active":"2024-09-17T07:16:17.376246+0000","last_became_peered":"2024-09-17T07:16:17.376246+0000","last_unstale":"2024-09-17T07:17:04.114404+0000","last_undegraded":"2024-09-17T07:16:17.363365+0000","last_fullsized":"2024-09-17T07:16:17.363257+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'35","last_scrub_stamp":"2024-09-17T07:15:43.665829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:15:43.665829+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:06.182386+0000","scrub_duration":0.0048325160000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[{"shards":"6,7","objects":9}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":363,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-17T07:17:04.104368+0000","last_change":"2024-09-17T07:15:48.685572+0000","last_active":"2024-09-17T07:17:04.104368+0000","last_peered":"2024-09-17T07:17:04.104368+0000","last_clean":"2024-09-17T07:17:04.104368+0000","last_became_active":"2024-09-17T07:02:12.012037+0000","last_became_peered":"2024-09-17T07:02:12.012037+0000","last_unstale":"2024-09-17T07:17:04.104368+0000","last_undegraded":"2024-09-17T07:17:04.104368+0000","last_fullsized":"2024-09-17T07:17:04.104368+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:48.685498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:06:39.322535+0000","last_clean_scrub_stamp":"2024-09-17T07:15:48.685498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:17.814720+0000","scrub_duration":0.001437528,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":358,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-17T07:17:04.103854+0000","last_change":"2024-09-17T07:16:44.719372+0000","last_active":"2024-09-17T07:17:04.103854+0000","last_peered":"2024-09-17T07:17:04.103854+0000","last_clean":"2024-09-17T07:17:04.103854+0000","last_became_active":"2024-09-17T07:02:16.014631+0000","last_became_peered":"2024-09-17T07:02:16.014631+0000","last_unstale":"2024-09-17T07:17:04.103854+0000","last_undegraded":"2024-09-17T07:17:04.103854+0000","last_fullsized":"2024-09-17T07:17:04.103854+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:44.719264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:16:44.719264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:03.887933+0000","scrub_duration":0.0036441289999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":320,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-17T07:17:04.116004+0000","last_change":"2024-09-17T07:15:45.275553+0000","last_active":"2024-09-17T07:17:04.116004+0000","last_peered":"2024-09-17T07:17:04.116004+0000","last_clean":"2024-09-17T07:17:04.116004+0000","last_became_active":"2024-09-17T07:09:24.403904+0000","last_became_peered":"2024-09-17T07:09:24.403904+0000","last_unstale":"2024-09-17T07:17:04.116004+0000","last_undegraded":"2024-09-17T07:17:04.116004+0000","last_fullsized":"2024-09-17T07:17:04.116004+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:45.275473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:45.275473+0000","last_clean_scrub_stamp":"2024-09-17T07:15:45.275473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:11.590023+0000","scrub_duration":0.0014472759999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"316'5119","reported_seq":5147,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-17T07:17:09.514428+0000","last_change":"2024-09-17T07:16:05.056347+0000","last_active":"2024-09-17T07:17:09.514428+0000","last_peered":"2024-09-17T07:17:09.514428+0000","last_clean":"2024-09-17T07:17:09.514428+0000","last_became_active":"2024-09-17T07:16:05.056100+0000","last_became_peered":"2024-09-17T07:16:05.056100+0000","last_unstale":"2024-09-17T07:17:09.514428+0000","last_undegraded":"2024-09-17T07:17:09.514428+0000","last_fullsized":"2024-09-17T07:17:09.514428+0000","mapping_epoch":297,"log_start":"305'400","ondisk_log_start":"305'400","created":297,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":4719,"log_dups_size":0,"ondisk_log_size":4719,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:30.058146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335478784,"num_objects":5119,"num_object_clones":0,"num_object_copies":15357,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5119,"num_write_kb":327616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"268'19","reported_seq":351,"reported_epoch":315,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:17:04.103777+0000","last_change":"2024-09-17T07:16:17.375651+0000","last_active":"2024-09-17T07:17:04.103777+0000","last_peered":"2024-09-17T07:17:04.103777+0000","last_clean":"2024-09-17T07:16:15.494820+0000","last_became_active":"2024-09-17T07:16:17.375651+0000","last_became_peered":"2024-09-17T07:16:17.375651+0000","last_unstale":"2024-09-17T07:17:04.103777+0000","last_undegraded":"2024-09-17T07:16:17.364532+0000","last_fullsized":"2024-09-17T07:16:17.364433+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'19","last_scrub_stamp":"2024-09-17T07:14:06.634334+0000","last_deep_scrub":"268'19","last_deep_scrub_stamp":"2024-09-17T07:14:06.634334+0000","last_clean_scrub_stamp":"2024-09-17T07:14:06.634334+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0088943289999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0,4","objects":4}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":193,"reported_epoch":315,"state":"active+undersized","last_fresh":"2024-09-17T07:17:04.112501+0000","last_change":"2024-09-17T07:16:17.374110+0000","last_active":"2024-09-17T07:17:04.112501+0000","last_peered":"2024-09-17T07:17:04.112501+0000","last_clean":"2024-09-17T07:09:22.372513+0000","last_became_active":"2024-09-17T07:16:17.374110+0000","last_became_peered":"2024-09-17T07:16:17.374110+0000","last_unstale":"2024-09-17T07:17:04.112501+0000","last_undegraded":"2024-09-17T07:17:04.112501+0000","last_fullsized":"2024-09-17T07:16:17.367137+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:58.886294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:13:53.593078+0000","last_clean_scrub_stamp":"2024-09-17T07:15:58.886294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:27.370927+0000","scrub_duration":0.005853245,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":300,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-17T07:17:04.113165+0000","last_change":"2024-09-17T07:16:35.152814+0000","last_active":"2024-09-17T07:17:04.113165+0000","last_peered":"2024-09-17T07:17:04.113165+0000","last_clean":"2024-09-17T07:17:04.113165+0000","last_became_active":"2024-09-17T07:09:24.396642+0000","last_became_peered":"2024-09-17T07:09:24.396642+0000","last_unstale":"2024-09-17T07:17:04.113165+0000","last_undegraded":"2024-09-17T07:17:04.113165+0000","last_fullsized":"2024-09-17T07:17:04.113165+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:35.152744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:16:35.152744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:04.541036+0000","scrub_duration":0.0022724160000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":312,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-17T07:17:04.103967+0000","last_change":"2024-09-17T07:15:00.557150+0000","last_active":"2024-09-17T07:17:04.103967+0000","last_peered":"2024-09-17T07:17:04.103967+0000","last_clean":"2024-09-17T07:17:04.103967+0000","last_became_active":"2024-09-17T07:02:18.026994+0000","last_became_peered":"2024-09-17T07:02:18.026994+0000","last_unstale":"2024-09-17T07:17:04.103967+0000","last_undegraded":"2024-09-17T07:17:04.103967+0000","last_fullsized":"2024-09-17T07:17:04.103967+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:00.557042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:58.099849+0000","last_clean_scrub_stamp":"2024-09-17T07:15:00.557042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0065216600000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.6","version":"316'5001","reported_seq":5026,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-17T07:17:08.714960+0000","last_change":"2024-09-17T07:16:05.058034+0000","last_active":"2024-09-17T07:17:08.714960+0000","last_peered":"2024-09-17T07:17:08.714960+0000","last_clean":"2024-09-17T07:17:08.714960+0000","last_became_active":"2024-09-17T07:16:05.057758+0000","last_became_peered":"2024-09-17T07:16:05.057758+0000","last_unstale":"2024-09-17T07:17:08.714960+0000","last_undegraded":"2024-09-17T07:17:08.714960+0000","last_fullsized":"2024-09-17T07:17:08.714960+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":297,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":5001,"log_dups_size":0,"ondisk_log_size":5001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:13.541473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327745536,"num_objects":5001,"num_object_clones":0,"num_object_copies":15003,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5001,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5001,"num_write_kb":320064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"242'34","reported_seq":387,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-17T07:17:04.111340+0000","last_change":"2024-09-17T07:16:44.126984+0000","last_active":"2024-09-17T07:17:04.111340+0000","last_peered":"2024-09-17T07:17:04.111340+0000","last_clean":"2024-09-17T07:17:04.111340+0000","last_became_active":"2024-09-17T07:02:14.028392+0000","last_became_peered":"2024-09-17T07:02:14.028392+0000","last_unstale":"2024-09-17T07:17:04.111340+0000","last_undegraded":"2024-09-17T07:17:04.111340+0000","last_fullsized":"2024-09-17T07:17:04.111340+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'34","last_scrub_stamp":"2024-09-17T07:16:44.126830+0000","last_deep_scrub":"242'34","last_deep_scrub_stamp":"2024-09-17T07:14:50.103539+0000","last_clean_scrub_stamp":"2024-09-17T07:16:44.126830+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:57.642107+0000","scrub_duration":0.008313473,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'2","reported_seq":200,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-17T07:17:04.111804+0000","last_change":"2024-09-17T07:15:48.378000+0000","last_active":"2024-09-17T07:17:04.111804+0000","last_peered":"2024-09-17T07:17:04.111804+0000","last_clean":"2024-09-17T07:17:04.111804+0000","last_became_active":"2024-09-17T07:09:24.402909+0000","last_became_peered":"2024-09-17T07:09:24.402909+0000","last_unstale":"2024-09-17T07:17:04.111804+0000","last_undegraded":"2024-09-17T07:17:04.111804+0000","last_fullsized":"2024-09-17T07:17:04.111804+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-09-17T07:15:48.377886+0000","last_deep_scrub":"89'2","last_deep_scrub_stamp":"2024-09-17T07:12:20.301263+0000","last_clean_scrub_stamp":"2024-09-17T07:15:48.377886+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001635079,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":420,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":420,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"225'9","reported_seq":504,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-17T07:17:05.261619+0000","last_change":"2024-09-17T07:14:42.018236+0000","last_active":"2024-09-17T07:17:05.261619+0000","last_peered":"2024-09-17T07:17:05.261619+0000","last_clean":"2024-09-17T07:17:05.261619+0000","last_became_active":"2024-09-17T07:02:16.015418+0000","last_became_peered":"2024-09-17T07:02:16.015418+0000","last_unstale":"2024-09-17T07:17:05.261619+0000","last_undegraded":"2024-09-17T07:17:05.261619+0000","last_fullsized":"2024-09-17T07:17:05.261619+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"225'9","last_scrub_stamp":"2024-09-17T07:14:42.017656+0000","last_deep_scrub":"225'9","last_deep_scrub_stamp":"2024-09-17T07:11:25.292565+0000","last_clean_scrub_stamp":"2024-09-17T07:14:42.017656+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0089186300000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":333,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-17T07:17:04.111317+0000","last_change":"2024-09-17T07:16:35.068863+0000","last_active":"2024-09-17T07:17:04.111317+0000","last_peered":"2024-09-17T07:17:04.111317+0000","last_clean":"2024-09-17T07:17:04.111317+0000","last_became_active":"2024-09-17T07:02:18.363253+0000","last_became_peered":"2024-09-17T07:02:18.363253+0000","last_unstale":"2024-09-17T07:17:04.111317+0000","last_undegraded":"2024-09-17T07:17:04.111317+0000","last_fullsized":"2024-09-17T07:17:04.111317+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:35.068712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:16:35.068712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:47.164438+0000","scrub_duration":0.0062141719999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.7","version":"316'5104","reported_seq":5131,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-17T07:17:08.396631+0000","last_change":"2024-09-17T07:16:05.054694+0000","last_active":"2024-09-17T07:17:08.396631+0000","last_peered":"2024-09-17T07:17:08.396631+0000","last_clean":"2024-09-17T07:17:08.396631+0000","last_became_active":"2024-09-17T07:16:05.054306+0000","last_became_peered":"2024-09-17T07:16:05.054306+0000","last_unstale":"2024-09-17T07:17:08.396631+0000","last_undegraded":"2024-09-17T07:17:08.396631+0000","last_fullsized":"2024-09-17T07:17:08.396631+0000","mapping_epoch":297,"log_start":"304'300","ondisk_log_start":"304'300","created":297,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":4804,"log_dups_size":0,"ondisk_log_size":4804,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:14.628931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334495744,"num_objects":5104,"num_object_clones":0,"num_object_copies":15312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5104,"num_write_kb":326656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"242'23","reported_seq":407,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-17T07:17:04.109681+0000","last_change":"2024-09-17T07:15:58.176415+0000","last_active":"2024-09-17T07:17:04.109681+0000","last_peered":"2024-09-17T07:17:04.109681+0000","last_clean":"2024-09-17T07:17:04.109681+0000","last_became_active":"2024-09-17T07:10:49.723292+0000","last_became_peered":"2024-09-17T07:10:49.723292+0000","last_unstale":"2024-09-17T07:17:04.109681+0000","last_undegraded":"2024-09-17T07:17:04.109681+0000","last_fullsized":"2024-09-17T07:17:04.109681+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'23","last_scrub_stamp":"2024-09-17T07:15:58.176287+0000","last_deep_scrub":"150'17","last_deep_scrub_stamp":"2024-09-17T07:06:38.775788+0000","last_clean_scrub_stamp":"2024-09-17T07:15:58.176287+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:09.296262+0000","scrub_duration":0.0024399420000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"68'2","reported_seq":348,"reported_epoch":315,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:17:04.113217+0000","last_change":"2024-09-17T07:16:17.371234+0000","last_active":"2024-09-17T07:17:04.113217+0000","last_peered":"2024-09-17T07:17:04.113217+0000","last_clean":"2024-09-17T07:16:15.358653+0000","last_became_active":"2024-09-17T07:16:17.371234+0000","last_became_peered":"2024-09-17T07:16:17.371234+0000","last_unstale":"2024-09-17T07:17:04.113217+0000","last_undegraded":"2024-09-17T07:16:17.366653+0000","last_fullsized":"2024-09-17T07:16:17.366079+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-09-17T07:14:56.533798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:14:56.533798+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003207115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[{"shards":"0,4","objects":2}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":295,"reported_epoch":315,"state":"active+undersized","last_fresh":"2024-09-17T07:17:04.102471+0000","last_change":"2024-09-17T07:16:17.367290+0000","last_active":"2024-09-17T07:17:04.102471+0000","last_peered":"2024-09-17T07:17:04.102471+0000","last_clean":"2024-09-17T07:16:15.494380+0000","last_became_active":"2024-09-17T07:16:17.367290+0000","last_became_peered":"2024-09-17T07:16:17.367290+0000","last_unstale":"2024-09-17T07:17:04.102471+0000","last_undegraded":"2024-09-17T07:17:04.102471+0000","last_fullsized":"2024-09-17T07:16:17.363484+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:51.586929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:11:17.479036+0000","last_clean_scrub_stamp":"2024-09-17T07:15:51.586929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:16.003170+0000","scrub_duration":0.001274435,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":164,"reported_epoch":315,"state":"active+undersized","last_fresh":"2024-09-17T07:17:04.110653+0000","last_change":"2024-09-17T07:16:17.379791+0000","last_active":"2024-09-17T07:17:04.110653+0000","last_peered":"2024-09-17T07:17:04.110653+0000","last_clean":"2024-09-17T07:16:15.590199+0000","last_became_active":"2024-09-17T07:16:17.379791+0000","last_became_peered":"2024-09-17T07:16:17.379791+0000","last_unstale":"2024-09-17T07:17:04.110653+0000","last_undegraded":"2024-09-17T07:17:04.110653+0000","last_fullsized":"2024-09-17T07:16:17.372333+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:13.937979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:13:13.937979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.018487173999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"242'15","reported_seq":371,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-17T07:17:04.103574+0000","last_change":"2024-09-17T07:15:58.508698+0000","last_active":"2024-09-17T07:17:04.103574+0000","last_peered":"2024-09-17T07:17:04.103574+0000","last_clean":"2024-09-17T07:17:04.103574+0000","last_became_active":"2024-09-17T07:09:24.408875+0000","last_became_peered":"2024-09-17T07:09:24.408875+0000","last_unstale":"2024-09-17T07:17:04.103574+0000","last_undegraded":"2024-09-17T07:17:04.103574+0000","last_fullsized":"2024-09-17T07:17:04.103574+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'15","last_scrub_stamp":"2024-09-17T07:15:58.508607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:15:58.508607+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:28.355513+0000","scrub_duration":0.0022156900000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":358,"reported_epoch":315,"state":"active+undersized","last_fresh":"2024-09-17T07:17:04.103519+0000","last_change":"2024-09-17T07:16:17.367792+0000","last_active":"2024-09-17T07:17:04.103519+0000","last_peered":"2024-09-17T07:17:04.103519+0000","last_clean":"2024-09-17T07:16:14.244722+0000","last_became_active":"2024-09-17T07:16:17.367792+0000","last_became_peered":"2024-09-17T07:16:17.367792+0000","last_unstale":"2024-09-17T07:17:04.103519+0000","last_undegraded":"2024-09-17T07:17:04.103519+0000","last_fullsized":"2024-09-17T07:16:17.366356+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:39.634698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:17.976267+0000","last_clean_scrub_stamp":"2024-09-17T07:15:39.634698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00117202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":23,"reported_epoch":315,"state":"active+undersized","last_fresh":"2024-09-17T07:17:04.112051+0000","last_change":"2024-09-17T07:16:17.378272+0000","last_active":"2024-09-17T07:17:04.112051+0000","last_peered":"2024-09-17T07:17:04.112051+0000","last_clean":"2024-09-17T07:02:14.999435+0000","last_became_active":"2024-09-17T07:16:17.378272+0000","last_became_peered":"2024-09-17T07:16:17.378272+0000","last_unstale":"2024-09-17T07:17:04.112051+0000","last_undegraded":"2024-09-17T07:17:04.112051+0000","last_fullsized":"2024-09-17T07:16:17.373119+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:40.845816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:18.367486+0000","last_clean_scrub_stamp":"2024-09-17T07:15:40.845816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":368,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-17T07:17:04.103804+0000","last_change":"2024-09-17T07:16:33.749048+0000","last_active":"2024-09-17T07:17:04.103804+0000","last_peered":"2024-09-17T07:17:04.103804+0000","last_clean":"2024-09-17T07:17:04.103804+0000","last_became_active":"2024-09-17T07:02:18.017825+0000","last_became_peered":"2024-09-17T07:02:18.017825+0000","last_unstale":"2024-09-17T07:17:04.103804+0000","last_undegraded":"2024-09-17T07:17:04.103804+0000","last_fullsized":"2024-09-17T07:17:04.103804+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:33.748908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:09:23.378805+0000","last_clean_scrub_stamp":"2024-09-17T07:16:33.748908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:36.947226+0000","scrub_duration":0.0073832719999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":346,"reported_epoch":315,"state":"active+undersized","last_fresh":"2024-09-17T07:17:04.111406+0000","last_change":"2024-09-17T07:16:17.375999+0000","last_active":"2024-09-17T07:17:04.111406+0000","last_peered":"2024-09-17T07:17:04.111406+0000","last_clean":"2024-09-17T07:16:14.245868+0000","last_became_active":"2024-09-17T07:16:17.375999+0000","last_became_peered":"2024-09-17T07:16:17.375999+0000","last_unstale":"2024-09-17T07:17:04.111406+0000","last_undegraded":"2024-09-17T07:17:04.111406+0000","last_fullsized":"2024-09-17T07:16:17.363383+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:46.645080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:46.645080+0000","last_clean_scrub_stamp":"2024-09-17T07:15:46.645080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:11.635584+0000","scrub_duration":0.0015661049999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"242'12","reported_seq":201,"reported_epoch":315,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:17:04.112000+0000","last_change":"2024-09-17T07:16:17.379665+0000","last_active":"2024-09-17T07:17:04.112000+0000","last_peered":"2024-09-17T07:17:04.112000+0000","last_clean":"2024-09-17T07:16:15.590425+0000","last_became_active":"2024-09-17T07:16:17.379665+0000","last_became_peered":"2024-09-17T07:16:17.379665+0000","last_unstale":"2024-09-17T07:17:04.112000+0000","last_undegraded":"2024-09-17T07:16:17.373659+0000","last_fullsized":"2024-09-17T07:16:17.373552+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'12","last_scrub_stamp":"2024-09-17T07:15:44.249858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:15:44.249858+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0022006600000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":3}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":339,"reported_epoch":315,"state":"active+undersized","last_fresh":"2024-09-17T07:17:04.115285+0000","last_change":"2024-09-17T07:16:17.377495+0000","last_active":"2024-09-17T07:17:04.115285+0000","last_peered":"2024-09-17T07:17:04.115285+0000","last_clean":"2024-09-17T07:16:16.170775+0000","last_became_active":"2024-09-17T07:16:17.377495+0000","last_became_peered":"2024-09-17T07:16:17.377495+0000","last_unstale":"2024-09-17T07:17:04.115285+0000","last_undegraded":"2024-09-17T07:17:04.115285+0000","last_fullsized":"2024-09-17T07:16:17.367529+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:55.191252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:14:55.191252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0090150360000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":311,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-17T07:17:04.117024+0000","last_change":"2024-09-17T07:14:52.058961+0000","last_active":"2024-09-17T07:17:04.117024+0000","last_peered":"2024-09-17T07:17:04.117024+0000","last_clean":"2024-09-17T07:17:04.117024+0000","last_became_active":"2024-09-17T07:02:18.365419+0000","last_became_peered":"2024-09-17T07:02:18.365419+0000","last_unstale":"2024-09-17T07:17:04.117024+0000","last_undegraded":"2024-09-17T07:17:04.117024+0000","last_fullsized":"2024-09-17T07:17:04.117024+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:52.058847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:14:52.058847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0054926549999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"242'20","reported_seq":144,"reported_epoch":315,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:17:04.111835+0000","last_change":"2024-09-17T07:16:17.380213+0000","last_active":"2024-09-17T07:17:04.111835+0000","last_peered":"2024-09-17T07:17:04.111835+0000","last_clean":"2024-09-17T07:12:20.630919+0000","last_became_active":"2024-09-17T07:16:17.380213+0000","last_became_peered":"2024-09-17T07:16:17.380213+0000","last_unstale":"2024-09-17T07:17:04.111835+0000","last_undegraded":"2024-09-17T07:16:17.375354+0000","last_fullsized":"2024-09-17T07:16:17.375256+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'20","last_scrub_stamp":"2024-09-17T07:14:05.528769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:14:05.528769+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0020069329999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":5}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":348,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-17T07:17:04.111778+0000","last_change":"2024-09-17T07:15:48.405105+0000","last_active":"2024-09-17T07:17:04.111778+0000","last_peered":"2024-09-17T07:17:04.111778+0000","last_clean":"2024-09-17T07:17:04.111778+0000","last_became_active":"2024-09-17T07:09:24.402355+0000","last_became_peered":"2024-09-17T07:09:24.402355+0000","last_unstale":"2024-09-17T07:17:04.111778+0000","last_undegraded":"2024-09-17T07:17:04.111778+0000","last_fullsized":"2024-09-17T07:17:04.111778+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:48.405027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:15:48.405027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:06.134372+0000","scrub_duration":0.0014214710000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"304'10","reported_seq":519,"reported_epoch":316,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:17:05.246147+0000","last_change":"2024-09-17T07:16:17.373566+0000","last_active":"2024-09-17T07:17:05.246147+0000","last_peered":"2024-09-17T07:17:05.246147+0000","last_clean":"2024-09-17T07:16:16.169875+0000","last_became_active":"2024-09-17T07:16:17.373566+0000","last_became_peered":"2024-09-17T07:16:17.373566+0000","last_unstale":"2024-09-17T07:17:05.246147+0000","last_undegraded":"2024-09-17T07:16:17.367771+0000","last_fullsized":"2024-09-17T07:16:17.367624+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'9","last_scrub_stamp":"2024-09-17T07:14:53.140133+0000","last_deep_scrub":"163'9","last_deep_scrub_stamp":"2024-09-17T07:07:34.933808+0000","last_clean_scrub_stamp":"2024-09-17T07:14:53.140133+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0088819989999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[{"shards":"0,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":332,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-17T07:17:04.103298+0000","last_change":"2024-09-17T07:16:34.364362+0000","last_active":"2024-09-17T07:17:04.103298+0000","last_peered":"2024-09-17T07:17:04.103298+0000","last_clean":"2024-09-17T07:17:04.103298+0000","last_became_active":"2024-09-17T07:09:24.409051+0000","last_became_peered":"2024-09-17T07:09:24.409051+0000","last_unstale":"2024-09-17T07:17:04.103298+0000","last_undegraded":"2024-09-17T07:17:04.103298+0000","last_fullsized":"2024-09-17T07:17:04.103298+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:34.363904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:26.619893+0000","last_clean_scrub_stamp":"2024-09-17T07:16:34.363904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:01.999702+0000","scrub_duration":0.0046574870000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"242'15","reported_seq":381,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-17T07:17:04.116104+0000","last_change":"2024-09-17T07:16:34.100665+0000","last_active":"2024-09-17T07:17:04.116104+0000","last_peered":"2024-09-17T07:17:04.116104+0000","last_clean":"2024-09-17T07:17:04.116104+0000","last_became_active":"2024-09-17T07:09:24.406333+0000","last_became_peered":"2024-09-17T07:09:24.406333+0000","last_unstale":"2024-09-17T07:17:04.116104+0000","last_undegraded":"2024-09-17T07:17:04.116104+0000","last_fullsized":"2024-09-17T07:17:04.116104+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'15","last_scrub_stamp":"2024-09-17T07:16:34.100540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:16:34.100540+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:53.725587+0000","scrub_duration":0.0033371669999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":331,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-17T07:17:04.104114+0000","last_change":"2024-09-17T07:16:55.240834+0000","last_active":"2024-09-17T07:17:04.104114+0000","last_peered":"2024-09-17T07:17:04.104114+0000","last_clean":"2024-09-17T07:17:04.104114+0000","last_became_active":"2024-09-17T07:09:24.402704+0000","last_became_peered":"2024-09-17T07:09:24.402704+0000","last_unstale":"2024-09-17T07:17:04.104114+0000","last_undegraded":"2024-09-17T07:17:04.104114+0000","last_fullsized":"2024-09-17T07:17:04.104114+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:41.618524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:14:41.618524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0025348459999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"304'10","reported_seq":282,"reported_epoch":316,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:17:05.272118+0000","last_change":"2024-09-17T07:16:17.380279+0000","last_active":"2024-09-17T07:17:05.272118+0000","last_peered":"2024-09-17T07:17:05.272118+0000","last_clean":"2024-09-17T07:16:15.687380+0000","last_became_active":"2024-09-17T07:16:17.380279+0000","last_became_peered":"2024-09-17T07:16:17.380279+0000","last_unstale":"2024-09-17T07:17:05.272118+0000","last_undegraded":"2024-09-17T07:16:17.375037+0000","last_fullsized":"2024-09-17T07:16:17.374926+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'9","last_scrub_stamp":"2024-09-17T07:13:10.965855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:13:10.965855+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0088433720000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":324,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-17T07:17:04.112833+0000","last_change":"2024-09-17T07:16:31.078582+0000","last_active":"2024-09-17T07:17:04.112833+0000","last_peered":"2024-09-17T07:17:04.112833+0000","last_clean":"2024-09-17T07:17:04.112833+0000","last_became_active":"2024-09-17T07:02:18.018716+0000","last_became_peered":"2024-09-17T07:02:18.018716+0000","last_unstale":"2024-09-17T07:17:04.112833+0000","last_undegraded":"2024-09-17T07:17:04.112833+0000","last_fullsized":"2024-09-17T07:17:04.112833+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:31.078504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:16:31.078504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:36.707922+0000","scrub_duration":0.0014265700000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"242'15","reported_seq":272,"reported_epoch":315,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:17:04.113999+0000","last_change":"2024-09-17T07:16:17.368982+0000","last_active":"2024-09-17T07:17:04.113999+0000","last_peered":"2024-09-17T07:17:04.113999+0000","last_clean":"2024-09-17T07:12:20.132949+0000","last_became_active":"2024-09-17T07:16:17.368982+0000","last_became_peered":"2024-09-17T07:16:17.368982+0000","last_unstale":"2024-09-17T07:17:04.113999+0000","last_undegraded":"2024-09-17T07:16:17.363353+0000","last_fullsized":"2024-09-17T07:16:17.363248+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'15","last_scrub_stamp":"2024-09-17T07:13:48.500257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:13:48.500257+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.005053564,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[{"shards":"2,6","objects":4}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":348,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-17T07:17:04.113739+0000","last_change":"2024-09-17T07:15:33.622783+0000","last_active":"2024-09-17T07:17:04.113739+0000","last_peered":"2024-09-17T07:17:04.113739+0000","last_clean":"2024-09-17T07:17:04.113739+0000","last_became_active":"2024-09-17T07:09:24.408139+0000","last_became_peered":"2024-09-17T07:09:24.408139+0000","last_unstale":"2024-09-17T07:17:04.113739+0000","last_undegraded":"2024-09-17T07:17:04.113739+0000","last_fullsized":"2024-09-17T07:17:04.113739+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:33.622666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:10:36.334827+0000","last_clean_scrub_stamp":"2024-09-17T07:15:33.622666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002139064,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":175,"reported_epoch":315,"state":"active+undersized","last_fresh":"2024-09-17T07:17:04.102802+0000","last_change":"2024-09-17T07:16:17.373522+0000","last_active":"2024-09-17T07:17:04.102802+0000","last_peered":"2024-09-17T07:17:04.102802+0000","last_clean":"2024-09-17T07:16:15.494646+0000","last_became_active":"2024-09-17T07:16:17.373522+0000","last_became_peered":"2024-09-17T07:16:17.373522+0000","last_unstale":"2024-09-17T07:17:04.102802+0000","last_undegraded":"2024-09-17T07:17:04.102802+0000","last_fullsized":"2024-09-17T07:16:17.364098+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:37.523283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:15:37.523283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013475850000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":329,"reported_epoch":315,"state":"active+undersized","last_fresh":"2024-09-17T07:17:04.102381+0000","last_change":"2024-09-17T07:16:17.371270+0000","last_active":"2024-09-17T07:17:04.102381+0000","last_peered":"2024-09-17T07:17:04.102381+0000","last_clean":"2024-09-17T07:16:15.494422+0000","last_became_active":"2024-09-17T07:16:17.371270+0000","last_became_peered":"2024-09-17T07:16:17.371270+0000","last_unstale":"2024-09-17T07:17:04.102381+0000","last_undegraded":"2024-09-17T07:17:04.102381+0000","last_fullsized":"2024-09-17T07:16:17.363365+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:40.540336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:15:40.540336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001212982,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"242'19","reported_seq":354,"reported_epoch":315,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:17:04.111934+0000","last_change":"2024-09-17T07:16:17.382006+0000","last_active":"2024-09-17T07:17:04.111934+0000","last_peered":"2024-09-17T07:17:04.111934+0000","last_clean":"2024-09-17T07:16:15.358100+0000","last_became_active":"2024-09-17T07:16:17.382006+0000","last_became_peered":"2024-09-17T07:16:17.382006+0000","last_unstale":"2024-09-17T07:17:04.111934+0000","last_undegraded":"2024-09-17T07:16:17.365274+0000","last_fullsized":"2024-09-17T07:16:17.365168+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'19","last_scrub_stamp":"2024-09-17T07:14:05.729535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:14:05.729535+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.008660559,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"94'2","reported_seq":66,"reported_epoch":315,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:17:04.116186+0000","last_change":"2024-09-17T07:16:17.374418+0000","last_active":"2024-09-17T07:17:04.116186+0000","last_peered":"2024-09-17T07:17:04.116186+0000","last_clean":"2024-09-17T07:04:06.025673+0000","last_became_active":"2024-09-17T07:16:17.374418+0000","last_became_peered":"2024-09-17T07:16:17.374418+0000","last_unstale":"2024-09-17T07:17:04.116186+0000","last_undegraded":"2024-09-17T07:16:17.367359+0000","last_fullsized":"2024-09-17T07:16:17.367242+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-09-17T07:15:48.907821+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-09-17T07:07:31.717888+0000","last_clean_scrub_stamp":"2024-09-17T07:15:48.907821+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[{"shards":"0,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":189,"reported_epoch":315,"state":"active+undersized","last_fresh":"2024-09-17T07:17:04.111452+0000","last_change":"2024-09-17T07:16:17.378918+0000","last_active":"2024-09-17T07:17:04.111452+0000","last_peered":"2024-09-17T07:17:04.111452+0000","last_clean":"2024-09-17T07:09:22.372509+0000","last_became_active":"2024-09-17T07:16:17.378918+0000","last_became_peered":"2024-09-17T07:16:17.378918+0000","last_unstale":"2024-09-17T07:17:04.111452+0000","last_undegraded":"2024-09-17T07:17:04.111452+0000","last_fullsized":"2024-09-17T07:16:17.367596+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:41.858093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:15:41.858093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037672679999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":168,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-17T07:17:04.103594+0000","last_change":"2024-09-17T07:15:49.573768+0000","last_active":"2024-09-17T07:17:04.103594+0000","last_peered":"2024-09-17T07:17:04.103594+0000","last_clean":"2024-09-17T07:17:04.103594+0000","last_became_active":"2024-09-17T07:09:24.400791+0000","last_became_peered":"2024-09-17T07:09:24.400791+0000","last_unstale":"2024-09-17T07:17:04.103594+0000","last_undegraded":"2024-09-17T07:17:04.103594+0000","last_fullsized":"2024-09-17T07:17:04.103594+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:49.573666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:49.573666+0000","last_clean_scrub_stamp":"2024-09-17T07:15:49.573666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:18.941100+0000","scrub_duration":0.0017548970000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"242'8","reported_seq":241,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-17T07:17:04.102437+0000","last_change":"2024-09-17T07:16:32.727879+0000","last_active":"2024-09-17T07:17:04.102437+0000","last_peered":"2024-09-17T07:17:04.102437+0000","last_clean":"2024-09-17T07:17:04.102437+0000","last_became_active":"2024-09-17T07:09:24.396368+0000","last_became_peered":"2024-09-17T07:09:24.396368+0000","last_unstale":"2024-09-17T07:17:04.102437+0000","last_undegraded":"2024-09-17T07:17:04.102437+0000","last_fullsized":"2024-09-17T07:17:04.102437+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'8","last_scrub_stamp":"2024-09-17T07:16:32.727780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:16:32.727780+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:45.467928+0000","scrub_duration":0.0051147279999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":339,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-17T07:17:04.115522+0000","last_change":"2024-09-17T07:16:37.093569+0000","last_active":"2024-09-17T07:17:04.115522+0000","last_peered":"2024-09-17T07:17:04.115522+0000","last_clean":"2024-09-17T07:17:04.115522+0000","last_became_active":"2024-09-17T07:02:12.008898+0000","last_became_peered":"2024-09-17T07:02:12.008898+0000","last_unstale":"2024-09-17T07:17:04.115522+0000","last_undegraded":"2024-09-17T07:17:04.115522+0000","last_fullsized":"2024-09-17T07:17:04.115522+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:37.093460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:16:37.093460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:55.997553+0000","scrub_duration":0.0048915060000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":336,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-17T07:17:04.111120+0000","last_change":"2024-09-17T07:15:54.554064+0000","last_active":"2024-09-17T07:17:04.111120+0000","last_peered":"2024-09-17T07:17:04.111120+0000","last_clean":"2024-09-17T07:17:04.111120+0000","last_became_active":"2024-09-17T07:02:16.014982+0000","last_became_peered":"2024-09-17T07:02:16.014982+0000","last_unstale":"2024-09-17T07:17:04.111120+0000","last_undegraded":"2024-09-17T07:17:04.111120+0000","last_fullsized":"2024-09-17T07:17:04.111120+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:54.553970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:54.553970+0000","last_clean_scrub_stamp":"2024-09-17T07:15:54.553970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:11.981217+0000","scrub_duration":0.0012697520000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":311,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-17T07:17:04.110276+0000","last_change":"2024-09-17T07:13:59.042437+0000","last_active":"2024-09-17T07:17:04.110276+0000","last_peered":"2024-09-17T07:17:04.110276+0000","last_clean":"2024-09-17T07:17:04.110276+0000","last_became_active":"2024-09-17T07:02:18.021014+0000","last_became_peered":"2024-09-17T07:02:18.021014+0000","last_unstale":"2024-09-17T07:17:04.110276+0000","last_undegraded":"2024-09-17T07:17:04.110276+0000","last_fullsized":"2024-09-17T07:17:04.110276+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:59.042322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:13:59.042322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0076271120000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"242'19","reported_seq":212,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-17T07:17:04.111056+0000","last_change":"2024-09-17T07:13:45.076965+0000","last_active":"2024-09-17T07:17:04.111056+0000","last_peered":"2024-09-17T07:17:04.111056+0000","last_clean":"2024-09-17T07:17:04.111056+0000","last_became_active":"2024-09-17T07:09:24.404942+0000","last_became_peered":"2024-09-17T07:09:24.404942+0000","last_unstale":"2024-09-17T07:17:04.111056+0000","last_undegraded":"2024-09-17T07:17:04.111056+0000","last_fullsized":"2024-09-17T07:17:04.111056+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'19","last_scrub_stamp":"2024-09-17T07:13:45.076322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:13:45.076322+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0088383200000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":350,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-17T07:17:04.115124+0000","last_change":"2024-09-17T07:16:32.647214+0000","last_active":"2024-09-17T07:17:04.115124+0000","last_peered":"2024-09-17T07:17:04.115124+0000","last_clean":"2024-09-17T07:17:04.115124+0000","last_became_active":"2024-09-17T07:02:12.010195+0000","last_became_peered":"2024-09-17T07:02:12.010195+0000","last_unstale":"2024-09-17T07:17:04.115124+0000","last_undegraded":"2024-09-17T07:17:04.115124+0000","last_fullsized":"2024-09-17T07:17:04.115124+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:32.647069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:16:32.647069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:02.148267+0000","scrub_duration":0.011908693,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":313,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-17T07:17:04.115332+0000","last_change":"2024-09-17T07:13:47.027302+0000","last_active":"2024-09-17T07:17:04.115332+0000","last_peered":"2024-09-17T07:17:04.115332+0000","last_clean":"2024-09-17T07:17:04.115332+0000","last_became_active":"2024-09-17T07:02:16.016183+0000","last_became_peered":"2024-09-17T07:02:16.016183+0000","last_unstale":"2024-09-17T07:17:04.115332+0000","last_undegraded":"2024-09-17T07:17:04.115332+0000","last_fullsized":"2024-09-17T07:17:04.115332+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:47.027189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:13:47.027189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0024966390000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":168,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-17T07:17:04.102715+0000","last_change":"2024-09-17T07:14:51.603973+0000","last_active":"2024-09-17T07:17:04.102715+0000","last_peered":"2024-09-17T07:17:04.102715+0000","last_clean":"2024-09-17T07:17:04.102715+0000","last_became_active":"2024-09-17T07:09:24.392216+0000","last_became_peered":"2024-09-17T07:09:24.392216+0000","last_unstale":"2024-09-17T07:17:04.102715+0000","last_undegraded":"2024-09-17T07:17:04.102715+0000","last_fullsized":"2024-09-17T07:17:04.102715+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:51.603818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:14:51.603818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0041420099999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1c","version":"75'1","reported_seq":311,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-17T07:17:04.111724+0000","last_change":"2024-09-17T07:13:56.976301+0000","last_active":"2024-09-17T07:17:04.111724+0000","last_peered":"2024-09-17T07:17:04.111724+0000","last_clean":"2024-09-17T07:17:04.111724+0000","last_became_active":"2024-09-17T07:02:18.021855+0000","last_became_peered":"2024-09-17T07:02:18.021855+0000","last_unstale":"2024-09-17T07:17:04.111724+0000","last_undegraded":"2024-09-17T07:17:04.111724+0000","last_fullsized":"2024-09-17T07:17:04.111724+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-17T07:13:56.976159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:13:56.976159+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0053255050000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":340,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-17T07:17:04.111425+0000","last_change":"2024-09-17T07:16:39.053518+0000","last_active":"2024-09-17T07:17:04.111425+0000","last_peered":"2024-09-17T07:17:04.111425+0000","last_clean":"2024-09-17T07:17:04.111425+0000","last_became_active":"2024-09-17T07:02:12.005329+0000","last_became_peered":"2024-09-17T07:02:12.005329+0000","last_unstale":"2024-09-17T07:17:04.111425+0000","last_undegraded":"2024-09-17T07:17:04.111425+0000","last_fullsized":"2024-09-17T07:17:04.111425+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:39.053386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:16:39.053386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:56.666430+0000","scrub_duration":0.0044426509999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"294'25","reported_seq":405,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-17T07:17:04.103499+0000","last_change":"2024-09-17T07:16:00.471391+0000","last_active":"2024-09-17T07:17:04.103499+0000","last_peered":"2024-09-17T07:17:04.103499+0000","last_clean":"2024-09-17T07:17:04.103499+0000","last_became_active":"2024-09-17T07:10:49.730948+0000","last_became_peered":"2024-09-17T07:10:49.730948+0000","last_unstale":"2024-09-17T07:17:04.103499+0000","last_undegraded":"2024-09-17T07:17:04.103499+0000","last_fullsized":"2024-09-17T07:17:04.103499+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'25","last_scrub_stamp":"2024-09-17T07:16:00.471311+0000","last_deep_scrub":"294'25","last_deep_scrub_stamp":"2024-09-17T07:16:00.471311+0000","last_clean_scrub_stamp":"2024-09-17T07:16:00.471311+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:10.547342+0000","scrub_duration":0.0040525370000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"75'8","reported_seq":527,"reported_epoch":316,"state":"active+clean","last_fresh":"2024-09-17T07:17:05.248552+0000","last_change":"2024-09-17T07:15:57.561805+0000","last_active":"2024-09-17T07:17:05.248552+0000","last_peered":"2024-09-17T07:17:05.248552+0000","last_clean":"2024-09-17T07:17:05.248552+0000","last_became_active":"2024-09-17T07:02:16.020560+0000","last_became_peered":"2024-09-17T07:02:16.020560+0000","last_unstale":"2024-09-17T07:17:05.248552+0000","last_undegraded":"2024-09-17T07:17:05.248552+0000","last_fullsized":"2024-09-17T07:17:05.248552+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:15:57.561693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:15:57.561693+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:06.265158+0000","scrub_duration":0.0015318759999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":21,"reported_epoch":315,"state":"active+undersized","last_fresh":"2024-09-17T07:17:04.115678+0000","last_change":"2024-09-17T07:16:17.375358+0000","last_active":"2024-09-17T07:17:04.115678+0000","last_peered":"2024-09-17T07:17:04.115678+0000","last_clean":"2024-09-17T07:02:17.002904+0000","last_became_active":"2024-09-17T07:16:17.375358+0000","last_became_peered":"2024-09-17T07:16:17.375358+0000","last_unstale":"2024-09-17T07:17:04.115678+0000","last_undegraded":"2024-09-17T07:17:04.115678+0000","last_fullsized":"2024-09-17T07:16:17.368647+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:37.885570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:15:37.885570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":179,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-17T07:17:04.111319+0000","last_change":"2024-09-17T07:15:36.653927+0000","last_active":"2024-09-17T07:17:04.111319+0000","last_peered":"2024-09-17T07:17:04.111319+0000","last_clean":"2024-09-17T07:17:04.111319+0000","last_became_active":"2024-09-17T07:09:24.403360+0000","last_became_peered":"2024-09-17T07:09:24.403360+0000","last_unstale":"2024-09-17T07:17:04.111319+0000","last_undegraded":"2024-09-17T07:17:04.111319+0000","last_fullsized":"2024-09-17T07:17:04.111319+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:36.653817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:10:29.325664+0000","last_clean_scrub_stamp":"2024-09-17T07:15:36.653817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014304299999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"242'19","reported_seq":411,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-17T07:17:04.115059+0000","last_change":"2024-09-17T07:15:41.670675+0000","last_active":"2024-09-17T07:17:04.115059+0000","last_peered":"2024-09-17T07:17:04.115059+0000","last_clean":"2024-09-17T07:17:04.115059+0000","last_became_active":"2024-09-17T07:03:48.760390+0000","last_became_peered":"2024-09-17T07:03:48.760390+0000","last_unstale":"2024-09-17T07:17:04.115059+0000","last_undegraded":"2024-09-17T07:17:04.115059+0000","last_fullsized":"2024-09-17T07:17:04.115059+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'19","last_scrub_stamp":"2024-09-17T07:15:41.670531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:15:41.670531+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0022660979999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":315,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-17T07:17:04.103425+0000","last_change":"2024-09-17T07:14:49.639583+0000","last_active":"2024-09-17T07:17:04.103425+0000","last_peered":"2024-09-17T07:17:04.103425+0000","last_clean":"2024-09-17T07:17:04.103425+0000","last_became_active":"2024-09-17T07:02:16.019701+0000","last_became_peered":"2024-09-17T07:02:16.019701+0000","last_unstale":"2024-09-17T07:17:04.103425+0000","last_undegraded":"2024-09-17T07:17:04.103425+0000","last_fullsized":"2024-09-17T07:17:04.103425+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:49.639480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:14:49.639480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003512032,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":325,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-17T07:17:04.113741+0000","last_change":"2024-09-17T07:15:57.151512+0000","last_active":"2024-09-17T07:17:04.113741+0000","last_peered":"2024-09-17T07:17:04.113741+0000","last_clean":"2024-09-17T07:17:04.113741+0000","last_became_active":"2024-09-17T07:02:18.363752+0000","last_became_peered":"2024-09-17T07:02:18.363752+0000","last_unstale":"2024-09-17T07:17:04.113741+0000","last_undegraded":"2024-09-17T07:17:04.113741+0000","last_fullsized":"2024-09-17T07:17:04.113741+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:57.151431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:15:57.151431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:19.350004+0000","scrub_duration":0.0012311939999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":379,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-17T07:17:04.114565+0000","last_change":"2024-09-17T07:16:44.559141+0000","last_active":"2024-09-17T07:17:04.114565+0000","last_peered":"2024-09-17T07:17:04.114565+0000","last_clean":"2024-09-17T07:17:04.114565+0000","last_became_active":"2024-09-17T07:03:48.759384+0000","last_became_peered":"2024-09-17T07:03:48.759384+0000","last_unstale":"2024-09-17T07:17:04.114565+0000","last_undegraded":"2024-09-17T07:17:04.114565+0000","last_fullsized":"2024-09-17T07:17:04.114565+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:44.558901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:11:18.559877+0000","last_clean_scrub_stamp":"2024-09-17T07:16:44.558901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:53.237416+0000","scrub_duration":0.031473045999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"242'27","reported_seq":291,"reported_epoch":315,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:17:04.102735+0000","last_change":"2024-09-17T07:16:17.368008+0000","last_active":"2024-09-17T07:17:04.102735+0000","last_peered":"2024-09-17T07:17:04.102735+0000","last_clean":"2024-09-17T07:12:20.337498+0000","last_became_active":"2024-09-17T07:16:17.368008+0000","last_became_peered":"2024-09-17T07:16:17.368008+0000","last_unstale":"2024-09-17T07:17:04.102735+0000","last_undegraded":"2024-09-17T07:16:17.365648+0000","last_fullsized":"2024-09-17T07:16:17.365550+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'27","last_scrub_stamp":"2024-09-17T07:15:45.855430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:15:45.855430+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.017595989999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":7}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":21,"reported_epoch":315,"state":"active+undersized","last_fresh":"2024-09-17T07:17:04.111809+0000","last_change":"2024-09-17T07:16:17.371494+0000","last_active":"2024-09-17T07:17:04.111809+0000","last_peered":"2024-09-17T07:17:04.111809+0000","last_clean":"2024-09-17T07:02:14.999435+0000","last_became_active":"2024-09-17T07:16:17.371494+0000","last_became_peered":"2024-09-17T07:16:17.371494+0000","last_unstale":"2024-09-17T07:17:04.111809+0000","last_undegraded":"2024-09-17T07:17:04.111809+0000","last_fullsized":"2024-09-17T07:16:17.366463+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:00.509283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:25.794418+0000","last_clean_scrub_stamp":"2024-09-17T07:14:00.509283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":186,"reported_epoch":315,"state":"active+undersized","last_fresh":"2024-09-17T07:17:04.114801+0000","last_change":"2024-09-17T07:16:17.377951+0000","last_active":"2024-09-17T07:17:04.114801+0000","last_peered":"2024-09-17T07:17:04.114801+0000","last_clean":"2024-09-17T07:16:14.246741+0000","last_became_active":"2024-09-17T07:16:17.377951+0000","last_became_peered":"2024-09-17T07:16:17.377951+0000","last_unstale":"2024-09-17T07:17:04.114801+0000","last_undegraded":"2024-09-17T07:17:04.114801+0000","last_fullsized":"2024-09-17T07:16:17.364654+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:02.473467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:15:02.473467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001733163,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":170,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-17T07:17:04.111158+0000","last_change":"2024-09-17T07:16:31.651379+0000","last_active":"2024-09-17T07:17:04.111158+0000","last_peered":"2024-09-17T07:17:04.111158+0000","last_clean":"2024-09-17T07:17:04.111158+0000","last_became_active":"2024-09-17T07:09:24.404795+0000","last_became_peered":"2024-09-17T07:09:24.404795+0000","last_unstale":"2024-09-17T07:17:04.111158+0000","last_undegraded":"2024-09-17T07:17:04.111158+0000","last_fullsized":"2024-09-17T07:17:04.111158+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:31.651247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:16:31.651247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:33.630557+0000","scrub_duration":0.0075767350000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"242'23","reported_seq":153,"reported_epoch":315,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:17:04.111955+0000","last_change":"2024-09-17T07:16:17.380972+0000","last_active":"2024-09-17T07:17:04.111955+0000","last_peered":"2024-09-17T07:17:04.111955+0000","last_clean":"2024-09-17T07:12:20.742449+0000","last_became_active":"2024-09-17T07:16:17.380972+0000","last_became_peered":"2024-09-17T07:16:17.380972+0000","last_unstale":"2024-09-17T07:17:04.111955+0000","last_undegraded":"2024-09-17T07:16:17.366757+0000","last_fullsized":"2024-09-17T07:16:17.366645+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'23","last_scrub_stamp":"2024-09-17T07:14:58.689120+0000","last_deep_scrub":"86'17","last_deep_scrub_stamp":"2024-09-17T07:03:18.241067+0000","last_clean_scrub_stamp":"2024-09-17T07:14:58.689120+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":334,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-17T07:17:04.112354+0000","last_change":"2024-09-17T07:15:38.338684+0000","last_active":"2024-09-17T07:17:04.112354+0000","last_peered":"2024-09-17T07:17:04.112354+0000","last_clean":"2024-09-17T07:17:04.112354+0000","last_became_active":"2024-09-17T07:09:24.393329+0000","last_became_peered":"2024-09-17T07:09:24.393329+0000","last_unstale":"2024-09-17T07:17:04.112354+0000","last_undegraded":"2024-09-17T07:17:04.112354+0000","last_fullsized":"2024-09-17T07:17:04.112354+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:38.338552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:15:38.338552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017509940000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":5290655744,"num_objects":80729,"num_object_clones":0,"num_object_copies":242187,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20071,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":80729,"num_write_kb":5166656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14568652800,"data_stored":14568652800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76329,"ondisk_log_size":76329,"up":44,"acting":44,"num_store_stats":7},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":83,"acting":83,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73,"ondisk_log_size":73,"up":81,"acting":81,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":101,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1113,"num_read_kb":870,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":109,"num_bytes_recovered":2228,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":13004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":733,"ondisk_log_size":733,"up":78,"acting":78,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":57,"num_read_kb":52,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":8,"acting":8,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1386,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":12,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":4578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":83,"acting":83,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037945,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2270400,"kb_used_data":1629752,"kb_used_omap":1,"kb_used_meta":640638,"kb_avail":91474752,"statfs":{"total":95995035648,"available":93670146048,"internally_reserved":0,"allocated":1668866048,"data_stored":1668377761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":656013771},"hb_peers":[0,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:16:28 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":1.077,"15min":1.5900000000000001},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.192},"max":{"1min":2.4590000000000001,"5min":8.0800000000000001,"15min":52.478999999999999},"last":0.495},{"interface":"front","average":{"1min":0.84799999999999998,"5min":1.087,"15min":1.7130000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":4.2729999999999997,"5min":6.8120000000000003,"15min":52.871000000000002},"last":0.55100000000000005}]},{"osd":1,"last update":"Tue Sep 17 07:15:24 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":1.4510000000000001,"15min":1.3859999999999999},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.223},"max":{"1min":1.573,"5min":15.801,"15min":73.549000000000007},"last":0.65300000000000002},{"interface":"front","average":{"1min":1.3779999999999999,"5min":1.2969999999999999,"15min":1.383},"min":{"1min":0.29699999999999999,"5min":0.26200000000000001,"15min":0.24099999999999999},"max":{"1min":4.7359999999999998,"5min":13.989000000000001,"15min":73.116},"last":0.71499999999999997}]},{"osd":2,"last update":"Tue Sep 17 07:16:28 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":1.0489999999999999,"15min":1.133},"min":{"1min":0.28399999999999997,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":2.819,"5min":6.4900000000000002,"15min":52.055999999999997},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.64600000000000002,"5min":1.1240000000000001,"15min":1.135},"min":{"1min":0.251,"5min":0.224,"15min":0.224},"max":{"1min":2.5049999999999999,"5min":8.1639999999999997,"15min":52.064999999999998},"last":0.46600000000000003}]},{"osd":3,"last update":"Tue Sep 17 07:16:28 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":1.0820000000000001,"15min":1.3260000000000001},"min":{"1min":0.307,"5min":0.29199999999999998,"15min":0.28799999999999998},"max":{"1min":2.8490000000000002,"5min":6.4640000000000004,"15min":51.689},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.67100000000000004,"5min":1.1890000000000001,"15min":1.4370000000000001},"min":{"1min":0.371,"5min":0.27000000000000002,"15min":0.26500000000000001},"max":{"1min":2.5329999999999999,"5min":8.1839999999999993,"15min":52.145000000000003},"last":1.3440000000000001}]},{"osd":4,"last update":"Tue Sep 17 07:16:28 2024","interfaces":[{"interface":"back","average":{"1min":1.024,"5min":1.115,"15min":0.76000000000000001},"min":{"1min":0.23100000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":4.2489999999999997,"5min":8.5730000000000004,"15min":8.5730000000000004},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.92300000000000004,"5min":1.159,"15min":0.81000000000000005},"min":{"1min":0.28699999999999998,"5min":0.28499999999999998,"15min":0.24399999999999999},"max":{"1min":5.3250000000000002,"5min":6.5129999999999999,"15min":6.5129999999999999},"last":0.371}]},{"osd":5,"last update":"Tue Sep 17 07:16:28 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":1.655,"15min":0.96099999999999997},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":3.0129999999999999,"5min":29.428999999999998,"15min":29.428999999999998},"last":1.931},{"interface":"front","average":{"1min":1.113,"5min":1.8660000000000001,"15min":1.046},"min":{"1min":0.27500000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":7.7629999999999999,"5min":30.059999999999999,"15min":30.059999999999999},"last":0.41599999999999998}]},{"osd":6,"last update":"Tue Sep 17 07:16:28 2024","interfaces":[{"interface":"back","average":{"1min":0.98299999999999998,"5min":1.232,"15min":0.95199999999999996},"min":{"1min":0.32200000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":4.3540000000000001,"5min":8.5329999999999995,"15min":25.363},"last":2.8090000000000002},{"interface":"front","average":{"1min":0.93600000000000005,"5min":1.2609999999999999,"15min":0.95099999999999996},"min":{"1min":0.33000000000000002,"5min":0.28399999999999997,"15min":0.27500000000000002},"max":{"1min":4.2949999999999999,"5min":6.5030000000000001,"15min":23.571000000000002},"last":3.0990000000000002}]}]},{"osd":6,"up_from":50,"seq":214748364988,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2677916,"kb_used_data":2253844,"kb_used_omap":1,"kb_used_meta":424062,"kb_avail":91067236,"statfs":{"total":95995035648,"available":93252849664,"internally_reserved":0,"allocated":2307936256,"data_stored":2307419914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":434239943},"hb_peers":[0,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:16:10 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":1.827,"15min":1.9910000000000001},"min":{"1min":0.24399999999999999,"5min":0.20200000000000001,"15min":0.185},"max":{"1min":2.2759999999999998,"5min":126.902,"15min":126.902},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.58699999999999997,"5min":1.9119999999999999,"15min":1.9550000000000001},"min":{"1min":0.27900000000000003,"5min":0.20899999999999999,"15min":0.20799999999999999},"max":{"1min":2.6259999999999999,"5min":127.06399999999999,"15min":127.06399999999999},"last":1.7130000000000001}]},{"osd":1,"last update":"Tue Sep 17 07:16:10 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.996,"15min":0.84399999999999997},"min":{"1min":0.23799999999999999,"5min":0.22900000000000001,"15min":0.22600000000000001},"max":{"1min":2.2999999999999998,"5min":14.720000000000001,"15min":14.720000000000001},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.622,"5min":1.0109999999999999,"15min":0.88800000000000001},"min":{"1min":0.29099999999999998,"5min":0.23400000000000001,"15min":0.22900000000000001},"max":{"1min":2.323,"5min":14.699,"15min":14.699},"last":0.46600000000000003}]},{"osd":2,"last update":"Tue Sep 17 07:16:10 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.71399999999999997,"15min":0.84799999999999998},"min":{"1min":0.28699999999999998,"5min":0.219,"15min":0.219},"max":{"1min":2.2469999999999999,"5min":4.9039999999999999,"15min":33.079000000000001},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.82299999999999995,"15min":0.85399999999999998},"min":{"1min":0.26800000000000002,"5min":0.25600000000000001,"15min":0.23000000000000001},"max":{"1min":2.5880000000000001,"5min":5.734,"15min":33.100000000000001},"last":1.6870000000000001}]},{"osd":3,"last update":"Tue Sep 17 07:16:10 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.74299999999999999,"15min":1.159},"min":{"1min":0.31900000000000001,"5min":0.26800000000000002,"15min":0.23200000000000001},"max":{"1min":2.3570000000000002,"5min":5.0830000000000002,"15min":52.944000000000003},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.83999999999999997,"15min":1.2150000000000001},"min":{"1min":0.33800000000000002,"5min":0.28599999999999998,"15min":0.27700000000000002},"max":{"1min":2.2959999999999998,"5min":5.7690000000000001,"15min":52.747},"last":0.80100000000000005}]},{"osd":4,"last update":"Tue Sep 17 07:16:10 2024","interfaces":[{"interface":"back","average":{"1min":0.95899999999999996,"5min":1.194,"15min":1.9670000000000001},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.214},"max":{"1min":4.8390000000000004,"5min":7.1050000000000004,"15min":260.77199999999999},"last":1.764},{"interface":"front","average":{"1min":0.63900000000000001,"5min":1.0449999999999999,"15min":1.944},"min":{"1min":0.30299999999999999,"5min":0.24199999999999999,"15min":0.19800000000000001},"max":{"1min":2.726,"5min":6.2140000000000004,"15min":260.69099999999997},"last":0.72499999999999998}]},{"osd":5,"last update":"Tue Sep 17 07:16:10 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":1.512,"15min":0.997},"min":{"1min":0.30199999999999999,"5min":0.27600000000000002,"15min":0.25900000000000001},"max":{"1min":4.5110000000000001,"5min":11.488,"15min":32.042000000000002},"last":1.746},{"interface":"front","average":{"1min":0.76800000000000002,"5min":1.319,"15min":0.998},"min":{"1min":0.27600000000000002,"5min":0.27200000000000002,"15min":0.251},"max":{"1min":3.1219999999999999,"5min":10.244999999999999,"15min":31.852},"last":0.70399999999999996}]},{"osd":7,"last update":"Tue Sep 17 07:16:35 2024","interfaces":[{"interface":"back","average":{"1min":7.258,"5min":2.6080000000000001,"15min":1.3700000000000001},"min":{"1min":0.246,"5min":0.246,"15min":0.23699999999999999},"max":{"1min":131.95699999999999,"5min":131.95699999999999,"15min":131.95699999999999},"last":0.74299999999999999},{"interface":"front","average":{"1min":7.2329999999999997,"5min":2.5880000000000001,"15min":1.3560000000000001},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26800000000000002},"max":{"1min":131.833,"5min":131.833,"15min":131.833},"last":1.7290000000000001}]}]},{"osd":5,"up_from":42,"seq":180388626625,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2218176,"kb_used_data":1591480,"kb_used_omap":1,"kb_used_meta":626686,"kb_avail":91526976,"statfs":{"total":95995035648,"available":93723623424,"internally_reserved":0,"allocated":1629675520,"data_stored":1629187021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":641726918},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:16:52 2024","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":1.5880000000000001},"min":{"1min":0.30399999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":2.8980000000000001,"5min":4.2329999999999997,"15min":49.265000000000001},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.54800000000000004,"5min":1.0049999999999999,"15min":1.6719999999999999},"min":{"1min":0.26300000000000001,"5min":0.221,"15min":0.219},"max":{"1min":1.7090000000000001,"5min":6.7149999999999999,"15min":47.350000000000001},"last":2.1469999999999998}]},{"osd":1,"last update":"Tue Sep 17 07:15:47 2024","interfaces":[{"interface":"back","average":{"1min":1.4319999999999999,"5min":1.0509999999999999,"15min":0.90100000000000002},"min":{"1min":0.30499999999999999,"5min":0.22,"15min":0.22},"max":{"1min":13.711,"5min":13.711,"15min":20.329000000000001},"last":0.437},{"interface":"front","average":{"1min":0.99399999999999999,"5min":1.3320000000000001,"15min":1.149},"min":{"1min":0.32000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4.617,"5min":6.7560000000000002,"15min":20.408999999999999},"last":0.42199999999999999}]},{"osd":2,"last update":"Tue Sep 17 07:16:52 2024","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.97099999999999997,"15min":1.099},"min":{"1min":0.255,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":2.8730000000000002,"5min":6.3419999999999996,"15min":20.486000000000001},"last":0.23200000000000001},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.85899999999999999,"15min":0.99399999999999999},"min":{"1min":0.25900000000000001,"5min":0.25800000000000001,"15min":0.23200000000000001},"max":{"1min":2.875,"5min":4.3079999999999998,"15min":23.925999999999998},"last":0.44500000000000001}]},{"osd":3,"last update":"Tue Sep 17 07:16:52 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":1.2490000000000001,"15min":1.4490000000000001},"min":{"1min":0.30299999999999999,"5min":0.27100000000000002,"15min":0.247},"max":{"1min":2.9140000000000001,"5min":13.882,"15min":68.978999999999999},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.88900000000000001,"15min":1.3},"min":{"1min":0.34599999999999997,"5min":0.29999999999999999,"15min":0.253},"max":{"1min":2.8399999999999999,"5min":4.327,"15min":69.084000000000003},"last":0.42099999999999999}]},{"osd":4,"last update":"Tue Sep 17 07:16:52 2024","interfaces":[{"interface":"back","average":{"1min":1.101,"5min":1.5580000000000001,"15min":1.54},"min":{"1min":0.254,"5min":0.23200000000000001,"15min":0.22},"max":{"1min":7.1950000000000003,"5min":14.036,"15min":142.327},"last":5.8099999999999996},{"interface":"front","average":{"1min":1.296,"5min":1.3999999999999999,"15min":1.4610000000000001},"min":{"1min":0.32700000000000001,"5min":0.25,"15min":0.20999999999999999},"max":{"1min":5.1269999999999998,"5min":13.478999999999999,"15min":142.13200000000001},"last":1.9330000000000001}]},{"osd":6,"last update":"Tue Sep 17 07:17:07 2024","interfaces":[{"interface":"back","average":{"1min":1.4099999999999999,"5min":1.117,"15min":0.84799999999999998},"min":{"1min":0.29799999999999999,"5min":0.27600000000000002,"15min":0.23200000000000001},"max":{"1min":5.4649999999999999,"5min":6.8419999999999996,"15min":8.3149999999999995},"last":0.58399999999999996},{"interface":"front","average":{"1min":1.75,"5min":1.2889999999999999,"15min":0.92000000000000004},"min":{"1min":0.36799999999999999,"5min":0.26700000000000002,"15min":0.24099999999999999},"max":{"1min":8.4489999999999998,"5min":8.4489999999999998,"15min":8.4489999999999998},"last":1.8979999999999999}]},{"osd":7,"last update":"Tue Sep 17 07:16:28 2024","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":1.6439999999999999,"15min":1.0369999999999999},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":4.3970000000000002,"5min":6.8869999999999996,"15min":6.8869999999999996},"last":0.316},{"interface":"front","average":{"1min":0.754,"5min":1.2569999999999999,"15min":0.86699999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.27100000000000002},"max":{"1min":4.7919999999999998,"5min":4.9260000000000002,"15min":5.641},"last":0.46500000000000002}]}]},{"osd":4,"up_from":33,"seq":141733920967,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3782660,"kb_used_data":3214204,"kb_used_omap":1,"kb_used_meta":568446,"kb_avail":89962492,"statfs":{"total":95995035648,"available":92121591808,"internally_reserved":0,"allocated":3291344896,"data_stored":3290851401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":582089158},"hb_peers":[0,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:16:11 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":1.1419999999999999,"15min":2.016},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":1.052,"5min":18.434999999999999,"15min":74.551000000000002},"last":0.439},{"interface":"front","average":{"1min":0.505,"5min":0.98099999999999998,"15min":2.117},"min":{"1min":0.24299999999999999,"5min":0.21299999999999999,"15min":0.20699999999999999},"max":{"1min":0.93200000000000005,"5min":18.521000000000001,"15min":74.778999999999996},"last":2.4889999999999999}]},{"osd":1,"last update":"Tue Sep 17 07:16:11 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":1.2070000000000001,"15min":1.4850000000000001},"min":{"1min":0.252,"5min":0.24399999999999999,"15min":0.23000000000000001},"max":{"1min":1.736,"5min":18.661000000000001,"15min":54.914999999999999},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.71499999999999997,"5min":1.3,"15min":1.444},"min":{"1min":0.246,"5min":0.23300000000000001,"15min":0.22800000000000001},"max":{"1min":3.7770000000000001,"5min":21.931999999999999,"15min":53.725000000000001},"last":0.443}]},{"osd":2,"last update":"Tue Sep 17 07:16:11 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":1.0389999999999999,"15min":1.069},"min":{"1min":0.26700000000000002,"5min":0.25700000000000001,"15min":0.23799999999999999},"max":{"1min":0.88700000000000001,"5min":6.8929999999999998,"15min":49.651000000000003},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.89100000000000001,"15min":1.0409999999999999},"min":{"1min":0.28599999999999998,"5min":0.27500000000000002,"15min":0.23599999999999999},"max":{"1min":0.92700000000000005,"5min":16.003,"15min":48.813000000000002},"last":2.5699999999999998}]},{"osd":3,"last update":"Tue Sep 17 07:16:11 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.76500000000000001,"15min":1.008},"min":{"1min":0.30499999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.95099999999999996,"5min":3.2189999999999999,"15min":39.042999999999999},"last":2.5470000000000002},{"interface":"front","average":{"1min":0.63400000000000001,"5min":1.032,"15min":1.1100000000000001},"min":{"1min":0.33900000000000002,"5min":0.28999999999999998,"15min":0.253},"max":{"1min":1.768,"5min":15.970000000000001,"15min":38.920000000000002},"last":0.63400000000000001}]},{"osd":5,"last update":"Tue Sep 17 07:16:34 2024","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":1.1559999999999999,"15min":0.89000000000000001},"min":{"1min":0.317,"5min":0.28100000000000003,"15min":0.20799999999999999},"max":{"1min":3.6509999999999998,"5min":7.1909999999999998,"15min":11.486000000000001},"last":2.5230000000000001},{"interface":"front","average":{"1min":0.96599999999999997,"5min":1.0720000000000001,"15min":0.90000000000000002},"min":{"1min":0.28999999999999998,"5min":0.27800000000000002,"15min":0.26100000000000001},"max":{"1min":4.5110000000000001,"5min":5.9870000000000001,"15min":11.781000000000001},"last":2.589}]},{"osd":6,"last update":"Tue Sep 17 07:16:11 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.97299999999999998,"15min":0.91300000000000003},"min":{"1min":0.27900000000000003,"5min":0.26200000000000001,"15min":0.218},"max":{"1min":1.7050000000000001,"5min":5.734,"15min":47.923999999999999},"last":2.29},{"interface":"front","average":{"1min":0.51800000000000002,"5min":1.2809999999999999,"15min":1.014},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":1.0249999999999999,"5min":10.051,"15min":47.600999999999999},"last":0.54200000000000004}]},{"osd":7,"last update":"Tue Sep 17 07:16:27 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":1.155,"15min":1.03},"min":{"1min":0.31900000000000001,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":1.8480000000000001,"5min":7.1550000000000002,"15min":48.719000000000001},"last":7.1200000000000001},{"interface":"front","average":{"1min":0.53400000000000003,"5min":1.504,"15min":1.119},"min":{"1min":0.33400000000000002,"5min":0.313,"15min":0.28499999999999998},"max":{"1min":1.0940000000000001,"5min":6.3410000000000002,"15min":50.527000000000001},"last":0.68200000000000005}]}]},{"osd":3,"up_from":0,"seq":115964117193,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":18,"seq":77309411535,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2898436,"kb_used_data":2231548,"kb_used_omap":1,"kb_used_meta":666878,"kb_avail":90846716,"statfs":{"total":95995035648,"available":93027037184,"internally_reserved":0,"allocated":2285105152,"data_stored":2284612501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":682883526},"hb_peers":[0,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:16:46 2024","interfaces":[{"interface":"back","average":{"1min":8.0630000000000006,"5min":2.8239999999999998,"15min":1.893},"min":{"1min":0.17000000000000001,"5min":0.159,"15min":0.159},"max":{"1min":131.09800000000001,"5min":131.09800000000001,"15min":131.09800000000001},"last":0.182},{"interface":"front","average":{"1min":8.0389999999999997,"5min":2.7879999999999998,"15min":2.0249999999999999},"min":{"1min":0.219,"5min":0.185,"15min":0.185},"max":{"1min":131.24700000000001,"5min":131.24700000000001,"15min":131.24700000000001},"last":0.23499999999999999}]},{"osd":1,"last update":"Tue Sep 17 07:15:43 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":1.7949999999999999,"15min":1.623},"min":{"1min":0.23699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":1.6459999999999999,"5min":47.101999999999997,"15min":79.659000000000006},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.55000000000000004,"5min":1.6990000000000001,"15min":1.5600000000000001},"min":{"1min":0.22700000000000001,"5min":0.20899999999999999,"15min":0.20200000000000001},"max":{"1min":1.8049999999999999,"5min":45.831000000000003,"15min":80.155000000000001},"last":0.41399999999999998}]},{"osd":3,"last update":"Tue Sep 17 07:17:01 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.55700000000000005,"15min":0.97599999999999998},"min":{"1min":0.23599999999999999,"5min":0.22,"15min":0.20100000000000001},"max":{"1min":0.71299999999999997,"5min":4.6449999999999996,"15min":23.43},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.34699999999999998,"5min":0.432,"15min":1.036},"min":{"1min":0.24199999999999999,"5min":0.23499999999999999,"15min":0.223},"max":{"1min":0.56499999999999995,"5min":1.6739999999999999,"15min":23.408000000000001},"last":0.26900000000000002}]},{"osd":4,"last update":"Tue Sep 17 07:16:22 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.89700000000000002,"15min":0.90100000000000002},"min":{"1min":0.36099999999999999,"5min":0.26100000000000001,"15min":0.26000000000000001},"max":{"1min":2.8809999999999998,"5min":6.8449999999999998,"15min":6.8449999999999998},"last":1.2769999999999999},{"interface":"front","average":{"1min":0.66100000000000003,"5min":1.0149999999999999,"15min":0.93700000000000006},"min":{"1min":0.32800000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":2.8940000000000001,"5min":6.8609999999999998,"15min":6.8609999999999998},"last":1.3}]},{"osd":5,"last update":"Tue Sep 17 07:16:46 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":1.133,"15min":1.1140000000000001},"min":{"1min":0.28199999999999997,"5min":0.26900000000000002,"15min":0.249},"max":{"1min":0.98199999999999998,"5min":15.276999999999999,"15min":15.276999999999999},"last":1.091},{"interface":"front","average":{"1min":0.46700000000000003,"5min":1.02,"15min":0.98199999999999998},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.27600000000000002},"max":{"1min":0.73299999999999998,"5min":15.561999999999999,"15min":15.561999999999999},"last":0.311}]},{"osd":6,"last update":"Tue Sep 17 07:17:01 2024","interfaces":[{"interface":"back","average":{"1min":1.8340000000000001,"5min":1.0269999999999999,"15min":1.4099999999999999},"min":{"1min":0.32400000000000001,"5min":0.29499999999999998,"15min":0.28999999999999998},"max":{"1min":20.494,"5min":20.494,"15min":101.587},"last":2.262},{"interface":"front","average":{"1min":2.3809999999999998,"5min":1.03,"15min":1.4590000000000001},"min":{"1min":0.35399999999999998,"5min":0.32300000000000001,"15min":0.28299999999999997},"max":{"1min":26.922000000000001,"5min":26.922000000000001,"15min":100.696},"last":0.68500000000000005}]},{"osd":7,"last update":"Tue Sep 17 07:16:41 2024","interfaces":[{"interface":"back","average":{"1min":5.7519999999999998,"5min":2.1539999999999999,"15min":1.3460000000000001},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.316},"max":{"1min":94.269999999999996,"5min":94.269999999999996,"15min":94.269999999999996},"last":0.35999999999999999},{"interface":"front","average":{"1min":5.8250000000000002,"5min":2.1600000000000001,"15min":1.415},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.32900000000000001},"max":{"1min":94.634,"5min":94.634,"15min":94.634},"last":2.1669999999999998}]}]},{"osd":1,"up_from":13,"seq":55834575049,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":541424,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":540542,"kb_avail":93203728,"statfs":{"total":95995035648,"available":95440617472,"internally_reserved":0,"allocated":884736,"data_stored":439467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":553515462},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:16:00 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.73699999999999999,"15min":1.139},"min":{"1min":0.183,"5min":0.155,"15min":0.155},"max":{"1min":2.6070000000000002,"5min":4.5780000000000003,"15min":22.521000000000001},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.67500000000000004,"15min":1.0369999999999999},"min":{"1min":0.247,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.198,"5min":4.335,"15min":22.471},"last":0.377}]},{"osd":2,"last update":"Tue Sep 17 07:15:44 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.91000000000000003,"15min":1.024},"min":{"1min":0.21299999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":2.6360000000000001,"5min":6.1920000000000002,"15min":11.055999999999999},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.73899999999999999,"15min":0.94799999999999995},"min":{"1min":0.22900000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":1.3160000000000001,"5min":4.1559999999999997,"15min":9.3160000000000007},"last":0.30599999999999999}]},{"osd":3,"last update":"Tue Sep 17 07:16:00 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.67800000000000005,"15min":0.94999999999999996},"min":{"1min":0.27400000000000002,"5min":0.25900000000000001,"15min":0.219},"max":{"1min":2.4409999999999998,"5min":4.1269999999999998,"15min":8.5510000000000002},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.371,"5min":0.47399999999999998,"15min":0.98599999999999999},"min":{"1min":0.247,"5min":0.22500000000000001,"15min":0.19500000000000001},"max":{"1min":0.627,"5min":2.8199999999999998,"15min":19.602},"last":0.35299999999999998}]},{"osd":4,"last update":"Tue Sep 17 07:15:20 2024","interfaces":[{"interface":"back","average":{"1min":1.1339999999999999,"5min":1.4670000000000001,"15min":1.129},"min":{"1min":0.36199999999999999,"5min":0.314,"15min":0.25},"max":{"1min":2.5659999999999998,"5min":19.088999999999999,"15min":19.088999999999999},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.98999999999999999,"5min":1.2010000000000001,"15min":1.1759999999999999},"min":{"1min":0.36799999999999999,"5min":0.317,"15min":0.307},"max":{"1min":2.6459999999999999,"5min":18.959,"15min":27.097999999999999},"last":0.499}]},{"osd":5,"last update":"Tue Sep 17 07:15:44 2024","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.47,"15min":1.2110000000000001},"min":{"1min":0.34799999999999998,"5min":0.28899999999999998,"15min":0.27100000000000002},"max":{"1min":3.6869999999999998,"5min":6.6790000000000003,"15min":16.84},"last":0.51000000000000001},{"interface":"front","average":{"1min":1.1559999999999999,"5min":1.069,"15min":1.169},"min":{"1min":0.29299999999999998,"5min":0.29199999999999998,"15min":0.28899999999999998},"max":{"1min":6.2560000000000002,"5min":6.2560000000000002,"15min":16.949999999999999},"last":0.41299999999999998}]},{"osd":6,"last update":"Tue Sep 17 07:16:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":1.5149999999999999,"15min":1.274},"min":{"1min":0.33100000000000002,"5min":0.315,"15min":0.308},"max":{"1min":1.127,"5min":50.497,"15min":50.497},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.99099999999999999,"5min":1.6379999999999999,"15min":1.222},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.28899999999999998},"max":{"1min":3.593,"5min":50.793999999999997,"15min":50.793999999999997},"last":0.48499999999999999}]},{"osd":7,"last update":"Tue Sep 17 07:15:20 2024","interfaces":[{"interface":"back","average":{"1min":1.655,"5min":1.343,"15min":1.1919999999999999},"min":{"1min":0.435,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":3.6120000000000001,"5min":9.8179999999999996,"15min":9.8179999999999996},"last":0.441},{"interface":"front","average":{"1min":1.464,"5min":1.6319999999999999,"15min":1.2410000000000001},"min":{"1min":0.35099999999999998,"5min":0.317,"15min":0.317},"max":{"1min":4.1219999999999999,"5min":7.5839999999999996,"15min":11.007999999999999},"last":0.57999999999999996}]}]},{"osd":0,"up_from":8,"seq":34359738583,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3950548,"kb_used_data":3269708,"kb_used_omap":1,"kb_used_meta":680830,"kb_avail":89794604,"statfs":{"total":95995035648,"available":91949674496,"internally_reserved":0,"allocated":3348180992,"data_stored":3347672701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":697170374},"hb_peers":[2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 17 07:15:18 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.89700000000000002,"15min":0.89500000000000002},"min":{"1min":0.254,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":1.518,"5min":21.876000000000001,"15min":21.876000000000001},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.56399999999999995,"5min":1.2330000000000001,"15min":1.081},"min":{"1min":0.17899999999999999,"5min":0.158,"15min":0.158},"max":{"1min":2.2599999999999998,"5min":22.521999999999998,"15min":22.521999999999998},"last":0.33700000000000002}]},{"osd":2,"last update":"Tue Sep 17 07:16:42 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.47199999999999998,"15min":1.1060000000000001},"min":{"1min":0.20899999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":0.68200000000000005,"5min":2.3690000000000002,"15min":18.643999999999998},"last":0.20999999999999999},{"interface":"front","average":{"1min":0.32600000000000001,"5min":0.54300000000000004,"15min":1.0089999999999999},"min":{"1min":0.224,"5min":0.189,"15min":0.189},"max":{"1min":0.48899999999999999,"5min":2.3730000000000002,"15min":17.972999999999999},"last":0.27100000000000002}]},{"osd":3,"last update":"Tue Sep 17 07:16:22 2024","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.61899999999999999,"15min":1.0109999999999999},"min":{"1min":0.27700000000000002,"5min":0.23400000000000001,"15min":0.21199999999999999},"max":{"1min":1.2929999999999999,"5min":5.5880000000000001,"15min":18.629000000000001},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.435,"5min":0.47299999999999998,"15min":1.018},"min":{"1min":0.27600000000000002,"5min":0.24199999999999999,"15min":0.23699999999999999},"max":{"1min":1.294,"5min":1.7090000000000001,"15min":22.358000000000001},"last":0.34100000000000003}]},{"osd":4,"last update":"Tue Sep 17 07:16:22 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.86699999999999999,"15min":1.123},"min":{"1min":0.317,"5min":0.29799999999999999,"15min":0.29099999999999998},"max":{"1min":1.673,"5min":4.0590000000000002,"15min":19.146999999999998},"last":0.377},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.89800000000000002,"15min":1.0880000000000001},"min":{"1min":0.32100000000000001,"5min":0.26500000000000001,"15min":0.255},"max":{"1min":1.7150000000000001,"5min":4.0449999999999999,"15min":19.158999999999999},"last":0.40999999999999998}]},{"osd":5,"last update":"Tue Sep 17 07:16:54 2024","interfaces":[{"interface":"back","average":{"1min":14.298999999999999,"5min":3.6680000000000001,"15min":2.0870000000000002},"min":{"1min":0.35699999999999998,"5min":0.35499999999999998,"15min":0.28899999999999998},"max":{"1min":216.83699999999999,"5min":216.83699999999999,"15min":216.83699999999999},"last":1.8919999999999999},{"interface":"front","average":{"1min":14.337999999999999,"5min":3.6110000000000002,"15min":2.0680000000000001},"min":{"1min":0.34200000000000003,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":216.816,"5min":216.816,"15min":216.816},"last":0.45500000000000002}]},{"osd":6,"last update":"Tue Sep 17 07:16:22 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.90100000000000002,"15min":1.0249999999999999},"min":{"1min":0.35299999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":2.762,"5min":4.6260000000000003,"15min":18.602},"last":147.13200000000001},{"interface":"front","average":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":1.077},"min":{"1min":0.376,"5min":0.311,"15min":0.311},"max":{"1min":5.4420000000000002,"5min":5.4420000000000002,"15min":18.692},"last":147.172}]},{"osd":7,"last update":"Tue Sep 17 07:16:22 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":1.262,"15min":1.4399999999999999},"min":{"1min":0.36599999999999999,"5min":0.30599999999999999,"15min":0.29599999999999999},"max":{"1min":1.3580000000000001,"5min":5.9509999999999996,"15min":46.183},"last":0.434},{"interface":"front","average":{"1min":0.65300000000000002,"5min":1.234,"15min":1.3200000000000001},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.32000000000000001},"max":{"1min":2.6680000000000001,"5min":6.1079999999999997,"15min":45.914999999999999},"last":0.53100000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3348299776,"data_stored":3348299776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2305163264,"data_stored":2305163264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3298033664,"data_stored":3298033664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1637744640,"data_stored":1637744640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2308571136,"data_stored":2308571136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1670840320,"data_stored":1670840320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T07:17:13.791 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd pool set unique_pool_3 pgp_num 16 2024-09-17T07:17:13.824 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:17:13.863 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:13.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:13.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:17:13.965 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:17:13.994 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:13.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:13.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:17:14.095 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:17:14.121 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:14.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:14.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:17:14.223 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:17:14.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:14.267 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:14.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:17:14.367 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:17:14.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:14.407 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:14.408 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:17:14.509 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:17:14.558 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:14.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:14.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:17:14.630 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 56 15 87681 87666 97.8313 76.375 0.00824876 0.0102173 2024-09-17T07:17:14.660 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:17:14.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:14.699 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:14.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:17:14.707 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:17:14.707 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:17:14.800 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:17:14.824 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:14.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:14.825 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:17:14.926 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:17:14.967 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:14.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:14.968 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:17:15.069 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:17:15.094 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:15.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:15.095 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:17:15.140 INFO:teuthology.orchestra.run.smithi135.stdout:{ 2024-09-17T07:17:15.140 INFO:teuthology.orchestra.run.smithi135.stdout: "size": 20, 2024-09-17T07:17:15.140 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 600, 2024-09-17T07:17:15.140 INFO:teuthology.orchestra.run.smithi135.stdout: "ops": [ 2024-09-17T07:17:15.141 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.141 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.15330.0:261544 9.3 9:cd5b6b21:::benchmark_data_smithi100_128631_object127278:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e287)", 2024-09-17T07:17:15.141 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:15:29.219219+0000", 2024-09-17T07:17:15.141 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 105.918032075, 2024-09-17T07:17:15.141 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.15732822599999999, 2024-09-17T07:17:15.141 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:17:15.141 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:17:15.141 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:17:15.141 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.15330", 2024-09-17T07:17:15.141 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/1361381749", 2024-09-17T07:17:15.141 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 261544 2024-09-17T07:17:15.141 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.142 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:17:15.142 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.142 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:17:15.142 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.219219+0000", 2024-09-17T07:17:15.142 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:17:15.142 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.142 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.142 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:17:15.142 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.219219+0000", 2024-09-17T07:17:15.142 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:17:15.142 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.142 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.142 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:17:15.143 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.219217+0000", 2024-09-17T07:17:15.143 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:17:15.143 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.143 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.143 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:17:15.143 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.219222+0000", 2024-09-17T07:17:15.143 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.0590000000000002e-06 2024-09-17T07:17:15.143 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.143 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.143 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:17:15.143 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.219234+0000", 2024-09-17T07:17:15.143 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.1729000000000001e-05 2024-09-17T07:17:15.143 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.144 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.144 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:17:15.144 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.219241+0000", 2024-09-17T07:17:15.144 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.5819999999999998e-06 2024-09-17T07:17:15.144 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.144 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.144 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:17:15.144 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.219280+0000", 2024-09-17T07:17:15.144 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.9678000000000002e-05 2024-09-17T07:17:15.144 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.144 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.145 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:17:15.145 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.219368+0000", 2024-09-17T07:17:15.145 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.7573999999999999e-05 2024-09-17T07:17:15.145 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.145 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.145 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 1,5", 2024-09-17T07:17:15.145 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.219445+0000", 2024-09-17T07:17:15.145 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.6570999999999996e-05 2024-09-17T07:17:15.145 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.145 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.145 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:17:15.145 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.220998+0000", 2024-09-17T07:17:15.145 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0015535869999999999 2024-09-17T07:17:15.146 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.146 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.146 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:17:15.146 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.221216+0000", 2024-09-17T07:17:15.146 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00021739900000000001 2024-09-17T07:17:15.146 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.146 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.146 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:17:15.146 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.376457+0000", 2024-09-17T07:17:15.146 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.155241037 2024-09-17T07:17:15.146 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.146 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.146 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:17:15.147 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.376511+0000", 2024-09-17T07:17:15.147 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.4154999999999999e-05 2024-09-17T07:17:15.147 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.147 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.147 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:17:15.147 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.376547+0000", 2024-09-17T07:17:15.147 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.6208999999999998e-05 2024-09-17T07:17:15.147 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:17:15.147 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:17:15.147 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:17:15.147 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.147 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.147 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.15330.0:261545 9.e 9:738b7e79:::benchmark_data_smithi100_128631_object127279:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e287)", 2024-09-17T07:17:15.148 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:15:29.239103+0000", 2024-09-17T07:17:15.148 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 105.89814775799999, 2024-09-17T07:17:15.148 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.14040370499999999, 2024-09-17T07:17:15.148 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:17:15.148 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:17:15.148 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:17:15.148 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.15330", 2024-09-17T07:17:15.148 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/1361381749", 2024-09-17T07:17:15.148 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 261545 2024-09-17T07:17:15.148 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.148 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:17:15.148 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.149 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:17:15.149 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.239103+0000", 2024-09-17T07:17:15.149 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:17:15.149 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.149 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.149 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:17:15.149 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.239103+0000", 2024-09-17T07:17:15.149 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:17:15.149 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.149 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.149 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:17:15.149 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.239102+0000", 2024-09-17T07:17:15.150 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:17:15.150 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.150 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.150 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:17:15.150 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.239112+0000", 2024-09-17T07:17:15.150 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0088999999999999e-05 2024-09-17T07:17:15.150 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.150 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.150 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:17:15.150 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.239125+0000", 2024-09-17T07:17:15.150 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.3502e-05 2024-09-17T07:17:15.150 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.150 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.150 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:17:15.151 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.239132+0000", 2024-09-17T07:17:15.151 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.1169999999999998e-06 2024-09-17T07:17:15.151 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.151 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.151 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:17:15.151 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.239192+0000", 2024-09-17T07:17:15.151 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.0173999999999997e-05 2024-09-17T07:17:15.151 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.151 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.151 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:17:15.151 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.239375+0000", 2024-09-17T07:17:15.151 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00018231200000000001 2024-09-17T07:17:15.151 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.152 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.152 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 1,4", 2024-09-17T07:17:15.152 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.239517+0000", 2024-09-17T07:17:15.152 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000142249 2024-09-17T07:17:15.152 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.152 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.152 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:17:15.152 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.246202+0000", 2024-09-17T07:17:15.152 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0066851860000000001 2024-09-17T07:17:15.152 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.152 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.152 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:17:15.153 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.247753+0000", 2024-09-17T07:17:15.153 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.001551045 2024-09-17T07:17:15.153 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.153 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.153 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:17:15.153 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.379425+0000", 2024-09-17T07:17:15.153 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.131671275 2024-09-17T07:17:15.153 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.153 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.153 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:17:15.153 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.379467+0000", 2024-09-17T07:17:15.153 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.2522999999999999e-05 2024-09-17T07:17:15.154 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.154 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.154 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:17:15.154 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.379507+0000", 2024-09-17T07:17:15.154 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.9767000000000001e-05 2024-09-17T07:17:15.154 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:17:15.154 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:17:15.154 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:17:15.154 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.154 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.154 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.15330.0:261547 9.e 9:7ee19f44:::benchmark_data_smithi100_128631_object127281:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e287)", 2024-09-17T07:17:15.154 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:15:29.239186+0000", 2024-09-17T07:17:15.154 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 105.898064508, 2024-09-17T07:17:15.155 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.140419135, 2024-09-17T07:17:15.155 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:17:15.155 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:17:15.155 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:17:15.155 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.15330", 2024-09-17T07:17:15.155 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/1361381749", 2024-09-17T07:17:15.155 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 261547 2024-09-17T07:17:15.155 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.155 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:17:15.155 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.156 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:17:15.156 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.239186+0000", 2024-09-17T07:17:15.156 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:17:15.156 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.157 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.157 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:17:15.157 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.239186+0000", 2024-09-17T07:17:15.157 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:17:15.157 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.157 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.157 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:17:15.157 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.239186+0000", 2024-09-17T07:17:15.157 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999995 2024-09-17T07:17:15.157 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.157 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.157 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:17:15.158 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.239189+0000", 2024-09-17T07:17:15.158 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.2619999999999999e-06 2024-09-17T07:17:15.158 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.158 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.158 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:17:15.158 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.239195+0000", 2024-09-17T07:17:15.158 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.2450000000000003e-06 2024-09-17T07:17:15.158 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.158 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.158 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:17:15.158 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.239201+0000", 2024-09-17T07:17:15.158 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.1959999999999999e-06 2024-09-17T07:17:15.159 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.159 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.159 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:17:15.159 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.243600+0000", 2024-09-17T07:17:15.159 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0043995040000000003 2024-09-17T07:17:15.159 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.159 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.159 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:17:15.159 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.243726+0000", 2024-09-17T07:17:15.159 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00012610799999999999 2024-09-17T07:17:15.159 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.159 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.159 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 1,4", 2024-09-17T07:17:15.159 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.243841+0000", 2024-09-17T07:17:15.160 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000115098 2024-09-17T07:17:15.160 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.160 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.160 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:17:15.160 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.246672+0000", 2024-09-17T07:17:15.160 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0028311260000000002 2024-09-17T07:17:15.160 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.160 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.160 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:17:15.160 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.247793+0000", 2024-09-17T07:17:15.160 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0011204279999999999 2024-09-17T07:17:15.160 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.161 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.161 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:17:15.161 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.379547+0000", 2024-09-17T07:17:15.161 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.13175411000000001 2024-09-17T07:17:15.161 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.161 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.161 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:17:15.161 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.379577+0000", 2024-09-17T07:17:15.161 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.0199e-05 2024-09-17T07:17:15.161 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.161 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.161 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:17:15.161 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.379605+0000", 2024-09-17T07:17:15.162 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.8342000000000001e-05 2024-09-17T07:17:15.162 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:17:15.162 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:17:15.162 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:17:15.162 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.162 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.162 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.15330.0:261549 9.3 9:c6efe766:::benchmark_data_smithi100_128631_object127283:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e287)", 2024-09-17T07:17:15.162 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:15:29.240257+0000", 2024-09-17T07:17:15.162 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 105.89699398, 2024-09-17T07:17:15.162 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.13638078100000001, 2024-09-17T07:17:15.162 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:17:15.162 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:17:15.162 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:17:15.162 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.15330", 2024-09-17T07:17:15.163 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/1361381749", 2024-09-17T07:17:15.163 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 261549 2024-09-17T07:17:15.163 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.163 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:17:15.163 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.163 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:17:15.163 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.240257+0000", 2024-09-17T07:17:15.163 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:17:15.163 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.163 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.163 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:17:15.163 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.240257+0000", 2024-09-17T07:17:15.163 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:17:15.163 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.164 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.164 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:17:15.164 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.240256+0000", 2024-09-17T07:17:15.164 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.999999 2024-09-17T07:17:15.164 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.164 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.164 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:17:15.164 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.240263+0000", 2024-09-17T07:17:15.164 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.9580000000000001e-06 2024-09-17T07:17:15.164 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.164 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.165 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:17:15.165 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.240274+0000", 2024-09-17T07:17:15.165 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0738e-05 2024-09-17T07:17:15.165 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.165 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.165 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:17:15.165 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.240280+0000", 2024-09-17T07:17:15.165 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.8220000000000001e-06 2024-09-17T07:17:15.165 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.165 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.165 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:17:15.165 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.241670+0000", 2024-09-17T07:17:15.166 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0013893639999999999 2024-09-17T07:17:15.166 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.166 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.166 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:17:15.166 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.241807+0000", 2024-09-17T07:17:15.166 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00013672399999999999 2024-09-17T07:17:15.166 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.166 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.166 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 1,5", 2024-09-17T07:17:15.166 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.241912+0000", 2024-09-17T07:17:15.166 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000105783 2024-09-17T07:17:15.166 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.166 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.167 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:17:15.167 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.244576+0000", 2024-09-17T07:17:15.167 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0026634330000000002 2024-09-17T07:17:15.167 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.167 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.167 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:17:15.167 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.247577+0000", 2024-09-17T07:17:15.167 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0030012250000000002 2024-09-17T07:17:15.167 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.167 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.167 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:17:15.167 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.376579+0000", 2024-09-17T07:17:15.167 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.12900204100000001 2024-09-17T07:17:15.168 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.168 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.168 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:17:15.168 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.376603+0000", 2024-09-17T07:17:15.168 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.4250999999999999e-05 2024-09-17T07:17:15.168 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.168 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.168 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:17:15.168 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.376638+0000", 2024-09-17T07:17:15.168 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.4360999999999998e-05 2024-09-17T07:17:15.168 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:17:15.168 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:17:15.168 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:17:15.169 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.169 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.169 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.15330.0:261558 9.3 9:c6241716:::benchmark_data_smithi100_128631_object127292:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e287)", 2024-09-17T07:17:15.169 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:15:29.253417+0000", 2024-09-17T07:17:15.169 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 105.883833829, 2024-09-17T07:17:15.169 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.12615304099999999, 2024-09-17T07:17:15.169 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:17:15.169 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:17:15.169 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:17:15.169 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.15330", 2024-09-17T07:17:15.169 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/1361381749", 2024-09-17T07:17:15.169 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 261558 2024-09-17T07:17:15.170 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.170 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:17:15.170 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.170 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:17:15.170 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.253417+0000", 2024-09-17T07:17:15.170 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:17:15.170 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.170 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.170 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:17:15.170 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.253417+0000", 2024-09-17T07:17:15.170 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:17:15.170 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.170 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.170 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:17:15.171 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.253416+0000", 2024-09-17T07:17:15.171 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:17:15.171 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.171 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.171 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:17:15.171 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.253423+0000", 2024-09-17T07:17:15.171 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.2529999999999998e-06 2024-09-17T07:17:15.171 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.171 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.171 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:17:15.171 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.253436+0000", 2024-09-17T07:17:15.171 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.276e-05 2024-09-17T07:17:15.171 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.171 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.172 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:17:15.172 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.253443+0000", 2024-09-17T07:17:15.172 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.9289999999999998e-06 2024-09-17T07:17:15.172 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.172 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.172 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:17:15.172 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.253511+0000", 2024-09-17T07:17:15.172 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.7854000000000005e-05 2024-09-17T07:17:15.172 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.172 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.172 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:17:15.172 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.253664+0000", 2024-09-17T07:17:15.172 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00015305000000000001 2024-09-17T07:17:15.173 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.173 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.173 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 1,5", 2024-09-17T07:17:15.173 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.253772+0000", 2024-09-17T07:17:15.173 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000107197 2024-09-17T07:17:15.173 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.173 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.173 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:17:15.173 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.256497+0000", 2024-09-17T07:17:15.173 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0027255249999999999 2024-09-17T07:17:15.173 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.173 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.174 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:17:15.174 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.256593+0000", 2024-09-17T07:17:15.174 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.6161000000000003e-05 2024-09-17T07:17:15.174 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.174 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.174 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:17:15.174 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.379496+0000", 2024-09-17T07:17:15.174 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.122902548 2024-09-17T07:17:15.175 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.175 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.175 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:17:15.175 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.379542+0000", 2024-09-17T07:17:15.175 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.6483000000000002e-05 2024-09-17T07:17:15.175 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.175 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.175 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:17:15.175 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.379570+0000", 2024-09-17T07:17:15.175 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.7813999999999998e-05 2024-09-17T07:17:15.175 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:17:15.176 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:17:15.176 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:17:15.176 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.176 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.176 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.15330.0:261563 9.d 9:b8546f76:::benchmark_data_smithi100_128631_object127297:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e287)", 2024-09-17T07:17:15.176 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:15:29.256944+0000", 2024-09-17T07:17:15.176 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 105.88030655, 2024-09-17T07:17:15.176 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.11946464499999999, 2024-09-17T07:17:15.176 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:17:15.176 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:17:15.176 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:17:15.176 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.15330", 2024-09-17T07:17:15.177 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/1361381749", 2024-09-17T07:17:15.177 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 261563 2024-09-17T07:17:15.177 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.177 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:17:15.177 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.177 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:17:15.177 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.256944+0000", 2024-09-17T07:17:15.177 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:17:15.177 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.177 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.177 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:17:15.177 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.256944+0000", 2024-09-17T07:17:15.177 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:17:15.178 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.178 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.178 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:17:15.178 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.256943+0000", 2024-09-17T07:17:15.178 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:17:15.178 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.178 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.178 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:17:15.178 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.256951+0000", 2024-09-17T07:17:15.178 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.7400000000000004e-06 2024-09-17T07:17:15.178 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.178 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.179 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:17:15.179 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.256959+0000", 2024-09-17T07:17:15.179 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.6989999999999992e-06 2024-09-17T07:17:15.179 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.179 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.179 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:17:15.179 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.256966+0000", 2024-09-17T07:17:15.179 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.0650000000000004e-06 2024-09-17T07:17:15.179 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.179 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.179 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:17:15.179 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.257004+0000", 2024-09-17T07:17:15.179 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.8325999999999999e-05 2024-09-17T07:17:15.180 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.180 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.180 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:17:15.180 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.257095+0000", 2024-09-17T07:17:15.180 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.1113000000000007e-05 2024-09-17T07:17:15.180 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.180 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.180 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 1,5", 2024-09-17T07:17:15.180 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.257159+0000", 2024-09-17T07:17:15.180 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.4172999999999997e-05 2024-09-17T07:17:15.180 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.180 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.180 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:17:15.181 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.259067+0000", 2024-09-17T07:17:15.181 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0019074140000000001 2024-09-17T07:17:15.181 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.181 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.181 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:17:15.181 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.259501+0000", 2024-09-17T07:17:15.181 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00043439599999999998 2024-09-17T07:17:15.181 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.181 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.181 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:17:15.181 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.376319+0000", 2024-09-17T07:17:15.181 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.116818097 2024-09-17T07:17:15.181 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.182 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.182 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:17:15.182 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.376363+0000", 2024-09-17T07:17:15.182 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.4002999999999999e-05 2024-09-17T07:17:15.182 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.182 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.182 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:17:15.182 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.376409+0000", 2024-09-17T07:17:15.182 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.5915999999999999e-05 2024-09-17T07:17:15.182 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:17:15.182 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:17:15.182 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:17:15.182 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.183 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.183 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.24338.0:2015 5.14 5:2b04a3e9:::notify.6:head [watch reconnect cookie 94163966834176 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e303)", 2024-09-17T07:17:15.183 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:16:16.974119+0000", 2024-09-17T07:17:15.183 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 58.163131788000001, 2024-09-17T07:17:15.183 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.40769392999999998, 2024-09-17T07:17:15.183 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:17:15.183 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "started", 2024-09-17T07:17:15.183 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:17:15.183 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.24338", 2024-09-17T07:17:15.183 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/1969590951", 2024-09-17T07:17:15.183 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 2015 2024-09-17T07:17:15.183 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.183 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:17:15.183 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.184 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:17:15.184 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:16.974119+0000", 2024-09-17T07:17:15.184 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:17:15.184 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.184 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.184 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:17:15.184 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:16.974119+0000", 2024-09-17T07:17:15.184 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:17:15.184 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.184 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.184 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:17:15.184 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:16.974117+0000", 2024-09-17T07:17:15.184 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:17:15.185 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.185 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.185 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:17:15.185 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:16.974125+0000", 2024-09-17T07:17:15.185 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.5580000000000003e-06 2024-09-17T07:17:15.185 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.185 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.185 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:17:15.185 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:16.974137+0000", 2024-09-17T07:17:15.185 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.1970000000000001e-05 2024-09-17T07:17:15.185 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.185 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.185 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:17:15.186 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:16.974145+0000", 2024-09-17T07:17:15.186 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.2069999999999995e-06 2024-09-17T07:17:15.186 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.186 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.186 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:17:15.186 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:16.974228+0000", 2024-09-17T07:17:15.186 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.2814999999999999e-05 2024-09-17T07:17:15.186 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.186 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.186 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:17:15.186 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:16.974237+0000", 2024-09-17T07:17:15.186 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.1730000000000001e-06 2024-09-17T07:17:15.186 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.186 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.187 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:17:15.187 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:17.380641+0000", 2024-09-17T07:17:15.187 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.40640414499999999 2024-09-17T07:17:15.187 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.187 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.187 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:17:15.187 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:17.380797+0000", 2024-09-17T07:17:15.187 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00015604700000000001 2024-09-17T07:17:15.187 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.187 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.187 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:17:15.187 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:17.381813+0000", 2024-09-17T07:17:15.187 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0010156640000000001 2024-09-17T07:17:15.188 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:17:15.188 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:17:15.188 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:17:15.188 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.188 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.188 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.15648.0:4529 10.c 10:3d626f42:::benchmark_data_smithi100_155039_object4528:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e304)", 2024-09-17T07:17:15.188 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:16:19.862694+0000", 2024-09-17T07:17:15.188 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 55.274557078999997, 2024-09-17T07:17:15.188 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.112655033, 2024-09-17T07:17:15.188 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:17:15.188 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:17:15.188 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:17:15.189 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.15648", 2024-09-17T07:17:15.189 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/1684940078", 2024-09-17T07:17:15.189 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 4529 2024-09-17T07:17:15.189 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.189 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:17:15.189 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.189 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:17:15.189 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:19.862694+0000", 2024-09-17T07:17:15.189 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:17:15.189 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.189 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.189 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:17:15.190 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:19.862694+0000", 2024-09-17T07:17:15.190 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:17:15.190 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.190 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.190 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:17:15.190 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:19.862693+0000", 2024-09-17T07:17:15.190 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999995 2024-09-17T07:17:15.190 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.190 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.190 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:17:15.190 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:19.862852+0000", 2024-09-17T07:17:15.190 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00015929 2024-09-17T07:17:15.190 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.191 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.191 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:17:15.191 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:19.972412+0000", 2024-09-17T07:17:15.191 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.10955962 2024-09-17T07:17:15.191 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.191 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.191 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:17:15.191 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:19.972431+0000", 2024-09-17T07:17:15.191 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.9338999999999999e-05 2024-09-17T07:17:15.191 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.191 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.191 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:17:15.192 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:19.973219+0000", 2024-09-17T07:17:15.192 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00078767400000000003 2024-09-17T07:17:15.192 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.192 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.192 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:17:15.192 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:19.973419+0000", 2024-09-17T07:17:15.192 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00019963199999999999 2024-09-17T07:17:15.192 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.192 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.192 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,4", 2024-09-17T07:17:15.192 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:19.973560+0000", 2024-09-17T07:17:15.192 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00014079899999999999 2024-09-17T07:17:15.192 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.193 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.193 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:17:15.193 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:19.975064+0000", 2024-09-17T07:17:15.193 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0015042460000000001 2024-09-17T07:17:15.193 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.193 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.193 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:17:15.193 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:19.975177+0000", 2024-09-17T07:17:15.193 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000113657 2024-09-17T07:17:15.193 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.193 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.193 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:17:15.193 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:19.975217+0000", 2024-09-17T07:17:15.193 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.9289999999999998e-05 2024-09-17T07:17:15.194 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.194 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.194 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:17:15.194 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:19.975314+0000", 2024-09-17T07:17:15.194 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.7588000000000001e-05 2024-09-17T07:17:15.194 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.194 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.194 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:17:15.194 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:19.975349+0000", 2024-09-17T07:17:15.194 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.4536999999999998e-05 2024-09-17T07:17:15.194 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:17:15.194 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:17:15.194 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:17:15.195 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.195 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.195 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.15648.0:35682 10.c 10:30e91945:::benchmark_data_smithi100_155039_object35681:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e307)", 2024-09-17T07:17:15.195 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:16:34.107429+0000", 2024-09-17T07:17:15.195 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 41.029821767000001, 2024-09-17T07:17:15.195 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.14480166899999999, 2024-09-17T07:17:15.195 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:17:15.195 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:17:15.195 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:17:15.195 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.15648", 2024-09-17T07:17:15.195 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/1684940078", 2024-09-17T07:17:15.195 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 35682 2024-09-17T07:17:15.195 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.196 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:17:15.196 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.196 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:17:15.196 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.107429+0000", 2024-09-17T07:17:15.196 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:17:15.196 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.196 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.196 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:17:15.196 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.107429+0000", 2024-09-17T07:17:15.196 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:17:15.196 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.196 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.196 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:17:15.197 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.107427+0000", 2024-09-17T07:17:15.197 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:17:15.197 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.197 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.197 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:17:15.197 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.107465+0000", 2024-09-17T07:17:15.197 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.7874999999999999e-05 2024-09-17T07:17:15.197 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.197 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.197 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:17:15.197 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.107477+0000", 2024-09-17T07:17:15.197 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.1868999999999999e-05 2024-09-17T07:17:15.197 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.198 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.198 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:17:15.198 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.107484+0000", 2024-09-17T07:17:15.198 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.6229999999999999e-06 2024-09-17T07:17:15.198 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.198 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.198 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:17:15.198 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.107536+0000", 2024-09-17T07:17:15.198 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.2098999999999997e-05 2024-09-17T07:17:15.198 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.198 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.198 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:17:15.198 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.107661+0000", 2024-09-17T07:17:15.198 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000125489 2024-09-17T07:17:15.199 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.199 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.199 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,4", 2024-09-17T07:17:15.199 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.107784+0000", 2024-09-17T07:17:15.199 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000122337 2024-09-17T07:17:15.199 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.199 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.199 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:17:15.199 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.109446+0000", 2024-09-17T07:17:15.199 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0016624260000000001 2024-09-17T07:17:15.199 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.199 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.199 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:17:15.199 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.112271+0000", 2024-09-17T07:17:15.200 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0028247300000000001 2024-09-17T07:17:15.200 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.200 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.200 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:17:15.200 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.252127+0000", 2024-09-17T07:17:15.200 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.13985611100000001 2024-09-17T07:17:15.200 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.200 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.200 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:17:15.200 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.252186+0000", 2024-09-17T07:17:15.200 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.9398999999999998e-05 2024-09-17T07:17:15.200 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.200 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.201 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:17:15.201 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.252231+0000", 2024-09-17T07:17:15.201 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:17:15.202 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.4311999999999999e-05 2024-09-17T07:17:15.202 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:17:15.203 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:17:15.203 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:17:15.203 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.203 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.203 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.15648.0:35686 10.c 10:34ba8be8:::benchmark_data_smithi100_155039_object35685:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e307)", 2024-09-17T07:17:15.203 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:16:34.109726+0000", 2024-09-17T07:17:15.203 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 41.027525105999999, 2024-09-17T07:17:15.203 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.142680421, 2024-09-17T07:17:15.203 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:17:15.203 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:17:15.203 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:17:15.203 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.15648", 2024-09-17T07:17:15.203 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/1684940078", 2024-09-17T07:17:15.204 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 35686 2024-09-17T07:17:15.204 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.204 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:17:15.204 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.204 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:17:15.204 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.109726+0000", 2024-09-17T07:17:15.204 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:17:15.204 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.204 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.204 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:17:15.204 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.109726+0000", 2024-09-17T07:17:15.204 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:17:15.205 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.205 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.205 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:17:15.205 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.109724+0000", 2024-09-17T07:17:15.205 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:17:15.205 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.205 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.205 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:17:15.205 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.109826+0000", 2024-09-17T07:17:15.205 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00010228700000000001 2024-09-17T07:17:15.205 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.205 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.206 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:17:15.206 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.109838+0000", 2024-09-17T07:17:15.206 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.1552000000000001e-05 2024-09-17T07:17:15.206 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.206 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.206 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:17:15.206 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.109845+0000", 2024-09-17T07:17:15.206 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.2309999999999999e-06 2024-09-17T07:17:15.206 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.206 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.206 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:17:15.206 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.109901+0000", 2024-09-17T07:17:15.207 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.6066999999999997e-05 2024-09-17T07:17:15.207 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.207 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.207 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:17:15.207 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.110036+0000", 2024-09-17T07:17:15.207 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00013461699999999999 2024-09-17T07:17:15.207 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.207 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.207 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,4", 2024-09-17T07:17:15.207 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.110178+0000", 2024-09-17T07:17:15.207 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00014169599999999999 2024-09-17T07:17:15.207 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.207 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.208 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:17:15.208 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.112477+0000", 2024-09-17T07:17:15.208 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.002299789 2024-09-17T07:17:15.208 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.208 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.208 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:17:15.208 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.252287+0000", 2024-09-17T07:17:15.208 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.139809288 2024-09-17T07:17:15.208 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.208 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.208 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:17:15.208 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.252320+0000", 2024-09-17T07:17:15.209 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.2997000000000003e-05 2024-09-17T07:17:15.209 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.209 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.209 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:17:15.209 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.252361+0000", 2024-09-17T07:17:15.209 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.1289999999999999e-05 2024-09-17T07:17:15.209 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.209 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.209 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:17:15.209 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.252406+0000", 2024-09-17T07:17:15.209 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.5164000000000003e-05 2024-09-17T07:17:15.209 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:17:15.209 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:17:15.210 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:17:15.210 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.210 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.210 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_repop(client.15648.0:35694 10.5 e307/297 10:ac915533:::benchmark_data_smithi100_155039_object35693:head v 307'2270, mlcod=307'2267)", 2024-09-17T07:17:15.210 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:16:34.114311+0000", 2024-09-17T07:17:15.210 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 41.022939811000001, 2024-09-17T07:17:15.210 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.13783398799999999, 2024-09-17T07:17:15.210 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:17:15.210 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:17:15.210 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:17:15.210 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.210 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:17:15.211 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.114311+0000", 2024-09-17T07:17:15.211 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:17:15.211 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.211 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.211 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:17:15.211 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.114311+0000", 2024-09-17T07:17:15.211 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:17:15.211 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.211 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.211 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:17:15.211 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.114310+0000", 2024-09-17T07:17:15.211 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.999999 2024-09-17T07:17:15.211 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.212 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.212 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:17:15.212 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.114359+0000", 2024-09-17T07:17:15.212 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.8751000000000001e-05 2024-09-17T07:17:15.212 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.212 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.212 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:17:15.212 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.114367+0000", 2024-09-17T07:17:15.212 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.7389999999999999e-06 2024-09-17T07:17:15.213 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.213 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.213 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:17:15.213 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.114373+0000", 2024-09-17T07:17:15.213 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.8830000000000003e-06 2024-09-17T07:17:15.213 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.213 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.213 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:17:15.213 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.114430+0000", 2024-09-17T07:17:15.213 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.7729999999999998e-05 2024-09-17T07:17:15.213 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.213 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.213 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:17:15.214 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.114447+0000", 2024-09-17T07:17:15.214 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.6922000000000002e-05 2024-09-17T07:17:15.214 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.214 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.214 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:17:15.214 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.252110+0000", 2024-09-17T07:17:15.214 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.13766310700000001 2024-09-17T07:17:15.214 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.214 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.214 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:17:15.214 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.252145+0000", 2024-09-17T07:17:15.214 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.4737999999999998e-05 2024-09-17T07:17:15.214 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:17:15.215 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:17:15.215 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:17:15.215 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.215 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.215 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_repop(client.15648.0:35695 10.8 e307/297 10:1d682dd1:::benchmark_data_smithi100_155039_object35694:head v 307'2210, mlcod=307'2208)", 2024-09-17T07:17:15.215 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:16:34.115493+0000", 2024-09-17T07:17:15.215 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 41.021757536999999, 2024-09-17T07:17:15.215 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.140152415, 2024-09-17T07:17:15.215 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:17:15.215 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:17:15.215 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:17:15.215 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.215 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:17:15.216 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.115493+0000", 2024-09-17T07:17:15.216 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:17:15.216 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.216 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.216 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:17:15.216 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.115493+0000", 2024-09-17T07:17:15.216 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:17:15.216 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.216 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.216 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:17:15.216 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.115492+0000", 2024-09-17T07:17:15.216 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.999999 2024-09-17T07:17:15.216 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.217 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.217 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:17:15.217 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.115533+0000", 2024-09-17T07:17:15.217 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.0343999999999999e-05 2024-09-17T07:17:15.217 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.217 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.217 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:17:15.217 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.115543+0000", 2024-09-17T07:17:15.217 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0543999999999999e-05 2024-09-17T07:17:15.217 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.217 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.217 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:17:15.217 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.115550+0000", 2024-09-17T07:17:15.218 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.8589999999999999e-06 2024-09-17T07:17:15.218 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.218 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.218 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:17:15.218 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.252125+0000", 2024-09-17T07:17:15.218 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.136574959 2024-09-17T07:17:15.218 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.218 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.218 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:17:15.218 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.252149+0000", 2024-09-17T07:17:15.218 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.41e-05 2024-09-17T07:17:15.218 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.218 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.219 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:17:15.219 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.255610+0000", 2024-09-17T07:17:15.219 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.003460664 2024-09-17T07:17:15.219 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.219 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.219 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:17:15.219 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.255646+0000", 2024-09-17T07:17:15.219 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.5893999999999999e-05 2024-09-17T07:17:15.219 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:17:15.219 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:17:15.219 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:17:15.219 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.220 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.220 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_repop_reply(client.15648.0:35682 10.c e307/297 ondisk, result = 0)", 2024-09-17T07:17:15.220 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:16:34.115580+0000", 2024-09-17T07:17:15.220 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 41.021670755000002, 2024-09-17T07:17:15.220 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.136660329, 2024-09-17T07:17:15.220 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:17:15.220 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "started", 2024-09-17T07:17:15.220 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:17:15.220 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.220 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:17:15.220 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.115580+0000", 2024-09-17T07:17:15.220 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:17:15.220 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.220 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.221 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:17:15.221 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.115580+0000", 2024-09-17T07:17:15.221 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:17:15.221 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.221 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.221 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:17:15.221 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.115580+0000", 2024-09-17T07:17:15.221 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999995 2024-09-17T07:17:15.221 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.221 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.221 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:17:15.221 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.115583+0000", 2024-09-17T07:17:15.221 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.932e-06 2024-09-17T07:17:15.222 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.222 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.222 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:17:15.222 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.115589+0000", 2024-09-17T07:17:15.222 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.268e-06 2024-09-17T07:17:15.222 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.222 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.222 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:17:15.222 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.115593+0000", 2024-09-17T07:17:15.222 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.1910000000000002e-06 2024-09-17T07:17:15.222 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.222 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.223 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:17:15.223 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.252100+0000", 2024-09-17T07:17:15.223 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.136506719 2024-09-17T07:17:15.223 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.223 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.223 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:17:15.223 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.252119+0000", 2024-09-17T07:17:15.223 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.9295000000000001e-05 2024-09-17T07:17:15.223 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.223 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.223 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:17:15.223 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.252240+0000", 2024-09-17T07:17:15.223 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000121373 2024-09-17T07:17:15.223 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:17:15.224 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:17:15.224 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:17:15.224 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.224 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.224 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_repop_reply(client.15648.0:35686 10.c e307/297 ondisk, result = 0)", 2024-09-17T07:17:15.224 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:16:34.115604+0000", 2024-09-17T07:17:15.224 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 41.021647066, 2024-09-17T07:17:15.224 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.13668735000000001, 2024-09-17T07:17:15.224 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:17:15.224 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "started", 2024-09-17T07:17:15.224 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:17:15.224 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.225 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:17:15.225 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.115604+0000", 2024-09-17T07:17:15.225 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:17:15.225 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.225 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.225 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:17:15.225 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.115604+0000", 2024-09-17T07:17:15.225 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:17:15.225 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.225 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.225 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:17:15.225 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.115604+0000", 2024-09-17T07:17:15.226 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967296 2024-09-17T07:17:15.226 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.226 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.226 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:17:15.226 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.115605+0000", 2024-09-17T07:17:15.226 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.8059999999999999e-06 2024-09-17T07:17:15.226 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.226 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.226 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:17:15.226 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.115610+0000", 2024-09-17T07:17:15.226 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.1330000000000001e-06 2024-09-17T07:17:15.226 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.226 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.227 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:17:15.227 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.115614+0000", 2024-09-17T07:17:15.227 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.1060000000000002e-06 2024-09-17T07:17:15.227 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.227 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.227 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:17:15.227 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.252271+0000", 2024-09-17T07:17:15.227 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.136657587 2024-09-17T07:17:15.227 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.227 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.227 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:17:15.227 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.252280+0000", 2024-09-17T07:17:15.228 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.1679999999999993e-06 2024-09-17T07:17:15.228 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.228 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.228 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:17:15.228 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.252291+0000", 2024-09-17T07:17:15.228 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0747999999999999e-05 2024-09-17T07:17:15.228 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:17:15.228 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:17:15.228 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:17:15.228 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.228 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.228 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_repop(client.15648.0:35693 10.9 e307/297 10:9eefa6fa:::benchmark_data_smithi100_155039_object35692:head v 307'2219, mlcod=307'2217)", 2024-09-17T07:17:15.228 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:16:34.115874+0000", 2024-09-17T07:17:15.229 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 41.021376871000001, 2024-09-17T07:17:15.229 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.13958446399999999, 2024-09-17T07:17:15.229 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:17:15.229 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:17:15.229 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:17:15.229 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.229 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:17:15.229 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.115874+0000", 2024-09-17T07:17:15.229 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:17:15.229 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.229 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.229 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:17:15.229 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.115874+0000", 2024-09-17T07:17:15.230 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:17:15.230 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.230 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.230 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:17:15.230 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.115873+0000", 2024-09-17T07:17:15.230 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.999999 2024-09-17T07:17:15.230 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.230 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.230 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:17:15.230 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.115898+0000", 2024-09-17T07:17:15.230 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.4515000000000002e-05 2024-09-17T07:17:15.230 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.230 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.230 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:17:15.231 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.252099+0000", 2024-09-17T07:17:15.231 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.136200925 2024-09-17T07:17:15.231 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.231 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.231 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:17:15.231 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.252111+0000", 2024-09-17T07:17:15.231 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.2330999999999999e-05 2024-09-17T07:17:15.231 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.231 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.231 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:17:15.231 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.252171+0000", 2024-09-17T07:17:15.231 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.0492000000000003e-05 2024-09-17T07:17:15.231 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.232 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.232 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:17:15.232 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.252194+0000", 2024-09-17T07:17:15.232 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.2235e-05 2024-09-17T07:17:15.232 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.232 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.232 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:17:15.232 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.255423+0000", 2024-09-17T07:17:15.232 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00322887 2024-09-17T07:17:15.232 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.232 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.232 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:17:15.232 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.255458+0000", 2024-09-17T07:17:15.233 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.5898000000000001e-05 2024-09-17T07:17:15.233 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:17:15.233 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:17:15.233 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:17:15.233 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.233 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.233 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_repop_reply(client.15648.0:35686 10.c e307/297 ondisk, result = 0)", 2024-09-17T07:17:15.233 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:16:34.117905+0000", 2024-09-17T07:17:15.233 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 41.019345797, 2024-09-17T07:17:15.233 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.134507613, 2024-09-17T07:17:15.233 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:17:15.233 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "started", 2024-09-17T07:17:15.233 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:17:15.234 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.234 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:17:15.234 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.117905+0000", 2024-09-17T07:17:15.234 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:17:15.234 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.234 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.234 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:17:15.234 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.117905+0000", 2024-09-17T07:17:15.234 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:17:15.234 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.235 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.235 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:17:15.235 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.117904+0000", 2024-09-17T07:17:15.235 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.999999 2024-09-17T07:17:15.235 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.235 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.236 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:17:15.236 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.117911+0000", 2024-09-17T07:17:15.236 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.8349999999999998e-06 2024-09-17T07:17:15.236 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.236 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.236 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:17:15.236 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.252100+0000", 2024-09-17T07:17:15.236 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.13418920600000001 2024-09-17T07:17:15.236 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.236 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.236 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:17:15.236 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.252111+0000", 2024-09-17T07:17:15.237 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0672e-05 2024-09-17T07:17:15.237 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.237 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.237 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:17:15.237 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.252308+0000", 2024-09-17T07:17:15.237 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00019745299999999999 2024-09-17T07:17:15.237 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.237 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.237 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:17:15.237 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.252315+0000", 2024-09-17T07:17:15.237 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.3840000000000002e-06 2024-09-17T07:17:15.237 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.237 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.238 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:17:15.238 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:34.252413+0000", 2024-09-17T07:17:15.238 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.7918999999999995e-05 2024-09-17T07:17:15.238 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:17:15.238 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:17:15.238 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:17:15.238 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.238 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.238 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.15648.0:41085 10.c 10:3c85a0dc:::benchmark_data_smithi100_155039_object41084:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e309)", 2024-09-17T07:17:15.238 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:16:37.514983+0000", 2024-09-17T07:17:15.238 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 37.622268007000002, 2024-09-17T07:17:15.238 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.20283764600000001, 2024-09-17T07:17:15.239 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:17:15.239 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:17:15.239 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:17:15.239 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.15648", 2024-09-17T07:17:15.239 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/1684940078", 2024-09-17T07:17:15.239 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 41085 2024-09-17T07:17:15.239 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.239 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:17:15.239 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.239 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:17:15.239 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.514983+0000", 2024-09-17T07:17:15.239 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:17:15.239 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.239 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.240 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:17:15.240 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.514983+0000", 2024-09-17T07:17:15.240 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:17:15.240 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.240 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.240 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:17:15.240 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.514981+0000", 2024-09-17T07:17:15.240 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999981 2024-09-17T07:17:15.240 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.240 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.240 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:17:15.240 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.515020+0000", 2024-09-17T07:17:15.241 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.8779000000000003e-05 2024-09-17T07:17:15.241 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.241 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.241 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:17:15.241 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.515034+0000", 2024-09-17T07:17:15.241 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.3658e-05 2024-09-17T07:17:15.241 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.241 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.241 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:17:15.241 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.515040+0000", 2024-09-17T07:17:15.241 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.9029999999999998e-06 2024-09-17T07:17:15.241 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.241 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.242 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:17:15.242 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.515164+0000", 2024-09-17T07:17:15.242 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00012357700000000001 2024-09-17T07:17:15.242 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.242 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.242 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:17:15.242 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.515305+0000", 2024-09-17T07:17:15.242 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00014111500000000001 2024-09-17T07:17:15.242 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.243 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.243 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,4", 2024-09-17T07:17:15.243 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.515451+0000", 2024-09-17T07:17:15.243 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000145623 2024-09-17T07:17:15.243 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.243 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.243 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:17:15.243 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.517446+0000", 2024-09-17T07:17:15.243 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.001995447 2024-09-17T07:17:15.243 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.243 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.243 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:17:15.243 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.518673+0000", 2024-09-17T07:17:15.244 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0012263199999999999 2024-09-17T07:17:15.244 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.244 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.244 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:17:15.244 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.717735+0000", 2024-09-17T07:17:15.244 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.19906210499999999 2024-09-17T07:17:15.244 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.244 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.244 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:17:15.244 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.717784+0000", 2024-09-17T07:17:15.244 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.9632000000000002e-05 2024-09-17T07:17:15.244 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.245 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.245 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:17:15.245 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.717821+0000", 2024-09-17T07:17:15.245 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.6251000000000002e-05 2024-09-17T07:17:15.245 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:17:15.245 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:17:15.245 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:17:15.245 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.245 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.245 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.15648.0:41086 10.c 10:3dd3bd63:::benchmark_data_smithi100_155039_object41085:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e309)", 2024-09-17T07:17:15.245 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:16:37.515054+0000", 2024-09-17T07:17:15.245 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 37.622196811999999, 2024-09-17T07:17:15.245 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.20290586399999999, 2024-09-17T07:17:15.245 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:17:15.246 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:17:15.246 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:17:15.246 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.15648", 2024-09-17T07:17:15.246 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/1684940078", 2024-09-17T07:17:15.246 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 41086 2024-09-17T07:17:15.246 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.246 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:17:15.246 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.246 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:17:15.246 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.515054+0000", 2024-09-17T07:17:15.246 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:17:15.246 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.246 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.247 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:17:15.247 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.515054+0000", 2024-09-17T07:17:15.247 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:17:15.247 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.247 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.247 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:17:15.247 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.515054+0000", 2024-09-17T07:17:15.247 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999995 2024-09-17T07:17:15.247 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.247 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.247 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:17:15.248 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.515083+0000", 2024-09-17T07:17:15.248 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.9658e-05 2024-09-17T07:17:15.248 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.248 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.248 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:17:15.248 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.515089+0000", 2024-09-17T07:17:15.248 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.0780000000000001e-06 2024-09-17T07:17:15.248 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.248 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.249 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:17:15.249 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.515095+0000", 2024-09-17T07:17:15.249 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.7400000000000001e-06 2024-09-17T07:17:15.249 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.249 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.249 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:17:15.249 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.517474+0000", 2024-09-17T07:17:15.249 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0023792890000000001 2024-09-17T07:17:15.249 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.249 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.249 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:17:15.249 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.517605+0000", 2024-09-17T07:17:15.249 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000130197 2024-09-17T07:17:15.251 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.251 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.251 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,4", 2024-09-17T07:17:15.251 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.517726+0000", 2024-09-17T07:17:15.251 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000121136 2024-09-17T07:17:15.251 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.251 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.251 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:17:15.251 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.518794+0000", 2024-09-17T07:17:15.251 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0010687419999999999 2024-09-17T07:17:15.251 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.251 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.252 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:17:15.252 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.519810+0000", 2024-09-17T07:17:15.252 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0010154369999999999 2024-09-17T07:17:15.252 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.252 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.252 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:17:15.252 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.717855+0000", 2024-09-17T07:17:15.252 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.198045429 2024-09-17T07:17:15.252 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.252 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.252 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:17:15.252 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.717896+0000", 2024-09-17T07:17:15.253 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.1090000000000001e-05 2024-09-17T07:17:15.253 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.253 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.253 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:17:15.253 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.717960+0000", 2024-09-17T07:17:15.253 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.3567999999999999e-05 2024-09-17T07:17:15.253 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:17:15.253 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:17:15.253 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:17:15.253 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.253 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.254 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.15648.0:41096 10.c 10:3f502d12:::benchmark_data_smithi100_155039_object41095:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e309)", 2024-09-17T07:17:15.254 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:16:37.520264+0000", 2024-09-17T07:17:15.254 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 37.616986769999997, 2024-09-17T07:17:15.254 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.197820209, 2024-09-17T07:17:15.254 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:17:15.254 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:17:15.254 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:17:15.254 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.15648", 2024-09-17T07:17:15.254 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/1684940078", 2024-09-17T07:17:15.254 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 41096 2024-09-17T07:17:15.254 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.254 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:17:15.254 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.255 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:17:15.255 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.520264+0000", 2024-09-17T07:17:15.255 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:17:15.255 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.255 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.255 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:17:15.255 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.520264+0000", 2024-09-17T07:17:15.255 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:17:15.255 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.255 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.255 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:17:15.255 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.520263+0000", 2024-09-17T07:17:15.255 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.999999 2024-09-17T07:17:15.256 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.256 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.256 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:17:15.256 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.520526+0000", 2024-09-17T07:17:15.256 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00026263099999999999 2024-09-17T07:17:15.256 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.256 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.256 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:17:15.256 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.520537+0000", 2024-09-17T07:17:15.256 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0849000000000001e-05 2024-09-17T07:17:15.256 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.256 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.256 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:17:15.257 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.520544+0000", 2024-09-17T07:17:15.257 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.1060000000000002e-06 2024-09-17T07:17:15.257 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.257 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.257 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:17:15.257 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.520577+0000", 2024-09-17T07:17:15.257 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.3840999999999997e-05 2024-09-17T07:17:15.257 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.257 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.257 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:17:15.257 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.520668+0000", 2024-09-17T07:17:15.257 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.0229999999999995e-05 2024-09-17T07:17:15.257 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.258 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.258 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,4", 2024-09-17T07:17:15.258 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.520746+0000", 2024-09-17T07:17:15.258 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.8368000000000006e-05 2024-09-17T07:17:15.258 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.258 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.258 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:17:15.258 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.521868+0000", 2024-09-17T07:17:15.258 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.001121637 2024-09-17T07:17:15.258 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.258 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.258 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:17:15.258 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.522774+0000", 2024-09-17T07:17:15.259 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00090647000000000004 2024-09-17T07:17:15.259 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.259 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.259 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:17:15.259 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.717999+0000", 2024-09-17T07:17:15.260 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.195225327 2024-09-17T07:17:15.260 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.260 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.260 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:17:15.260 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.718048+0000", 2024-09-17T07:17:15.260 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.8568e-05 2024-09-17T07:17:15.260 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.260 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.260 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:17:15.260 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.718084+0000", 2024-09-17T07:17:15.260 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.6257000000000002e-05 2024-09-17T07:17:15.261 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:17:15.261 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:17:15.261 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:17:15.261 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.261 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.261 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.15648.0:41098 10.c 10:39aafbae:::benchmark_data_smithi100_155039_object41097:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e309)", 2024-09-17T07:17:15.261 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:16:37.522841+0000", 2024-09-17T07:17:15.261 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 37.614409840999997, 2024-09-17T07:17:15.261 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.199064503, 2024-09-17T07:17:15.261 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:17:15.261 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:17:15.261 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:17:15.261 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.15648", 2024-09-17T07:17:15.262 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/1684940078", 2024-09-17T07:17:15.262 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 41098 2024-09-17T07:17:15.262 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.262 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:17:15.262 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.262 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:17:15.262 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.522841+0000", 2024-09-17T07:17:15.262 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:17:15.262 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.262 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.262 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:17:15.262 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.522841+0000", 2024-09-17T07:17:15.262 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:17:15.263 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.263 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.263 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:17:15.263 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.522840+0000", 2024-09-17T07:17:15.263 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.999999 2024-09-17T07:17:15.263 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.263 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.263 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:17:15.263 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.522898+0000", 2024-09-17T07:17:15.263 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.7596999999999998e-05 2024-09-17T07:17:15.263 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.263 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.263 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:17:15.263 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.522904+0000", 2024-09-17T07:17:15.264 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.5239999999999997e-06 2024-09-17T07:17:15.264 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.264 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.264 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:17:15.264 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.522908+0000", 2024-09-17T07:17:15.264 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.7579999999999998e-06 2024-09-17T07:17:15.264 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.264 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.264 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:17:15.264 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.522935+0000", 2024-09-17T07:17:15.264 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.7546e-05 2024-09-17T07:17:15.264 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.264 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.265 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:17:15.265 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.546598+0000", 2024-09-17T07:17:15.265 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.023662882999999999 2024-09-17T07:17:15.265 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.265 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.265 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,4", 2024-09-17T07:17:15.265 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.546741+0000", 2024-09-17T07:17:15.265 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000142659 2024-09-17T07:17:15.265 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.265 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.265 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:17:15.265 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.548657+0000", 2024-09-17T07:17:15.265 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0019164270000000001 2024-09-17T07:17:15.266 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.266 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.266 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:17:15.266 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.548891+0000", 2024-09-17T07:17:15.266 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000233899 2024-09-17T07:17:15.266 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.266 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.266 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:17:15.266 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.721694+0000", 2024-09-17T07:17:15.266 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.17280299399999999 2024-09-17T07:17:15.266 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.266 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.266 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:17:15.267 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.721789+0000", 2024-09-17T07:17:15.267 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.4996999999999999e-05 2024-09-17T07:17:15.267 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:17:15.267 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:17:15.267 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:17:15.267 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.721906+0000", 2024-09-17T07:17:15.267 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00011618899999999999 2024-09-17T07:17:15.267 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:17:15.267 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:17:15.267 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:17:15.267 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:17:15.267 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:17:15.267 INFO:teuthology.orchestra.run.smithi135.stdout:} 2024-09-17T07:17:15.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:15.268 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:15.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:17:15.286 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:17:15.369 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:17:15.415 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:15.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:15.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:17:15.517 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:17:15.545 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:15.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:15.546 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:17:15.630 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 57 16 88809 88793 97.3506 70.4375 0.00861422 0.0102669 2024-09-17T07:17:15.647 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:17:15.680 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:15.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:15.681 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:17:15.782 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:17:15.804 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:15.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:15.805 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:17:15.906 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:17:15.946 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:15.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:15.947 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:17:16.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:15 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:17:16.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:15 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:17:16.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:15 smithi135 ceph-mon[33238]: osdmap e317: 8 total, 7 up, 7 in 2024-09-17T07:17:16.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:15 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 20179/242775 objects degraded (8.312%), 28 pgs degraded (PG_DEGRADED) 2024-09-17T07:17:16.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:15 smithi135 ceph-mon[33238]: osdmap e318: 8 total, 7 up, 7 in 2024-09-17T07:17:16.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:15 smithi135 ceph-mon[33238]: pgmap v677: 148 pgs: 36 active+undersized, 28 active+undersized+degraded, 84 active+clean; 5.3 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 86 MiB/s wr, 1.38k op/s; 21685/260313 objects degraded (8.330%) 2024-09-17T07:17:16.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:15 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2235012611' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:17:16.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:15 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:17:16.047 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:17:16.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:16.093 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:16.093 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:17:16.126 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:15 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:17:16.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:15 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:17:16.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:15 smithi100 ceph-mon[39462]: osdmap e317: 8 total, 7 up, 7 in 2024-09-17T07:17:16.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:15 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 20179/242775 objects degraded (8.312%), 28 pgs degraded (PG_DEGRADED) 2024-09-17T07:17:16.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:15 smithi100 ceph-mon[39462]: osdmap e318: 8 total, 7 up, 7 in 2024-09-17T07:17:16.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:15 smithi100 ceph-mon[39462]: pgmap v677: 148 pgs: 36 active+undersized, 28 active+undersized+degraded, 84 active+clean; 5.3 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 86 MiB/s wr, 1.38k op/s; 21685/260313 objects degraded (8.330%) 2024-09-17T07:17:16.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:15 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2235012611' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:17:16.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:15 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:17:16.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:15 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:17:16.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:15 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:17:16.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:15 smithi100 ceph-mon[29092]: osdmap e317: 8 total, 7 up, 7 in 2024-09-17T07:17:16.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:15 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 20179/242775 objects degraded (8.312%), 28 pgs degraded (PG_DEGRADED) 2024-09-17T07:17:16.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:15 smithi100 ceph-mon[29092]: osdmap e318: 8 total, 7 up, 7 in 2024-09-17T07:17:16.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:15 smithi100 ceph-mon[29092]: pgmap v677: 148 pgs: 36 active+undersized, 28 active+undersized+degraded, 84 active+clean; 5.3 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 86 MiB/s wr, 1.38k op/s; 21685/260313 objects degraded (8.330%) 2024-09-17T07:17:16.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:15 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2235012611' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:17:16.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:15 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:17:16.129 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:17:15 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:17:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:17:16.193 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:17:16.220 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:16.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:16.221 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:17:16.265 INFO:teuthology.orchestra.run.smithi100.stderr:noscrub is unset 2024-09-17T07:17:16.322 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:17:16.350 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:16.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:16.352 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:17:16.452 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:17:16.480 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:16.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:16.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:17:16.582 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:17:16.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:16.625 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:16.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:17:16.630 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 58 16 89955 89939 96.9069 71.625 0.0087388 0.0103149 2024-09-17T07:17:16.685 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-17T07:17:16.709 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:17:16.726 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:17:16.763 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:16.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:16.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:17:16.864 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:17:16.876 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:16 smithi100 ceph-mon[39462]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:17:16.876 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:16 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T07:17:16.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:16 smithi100 ceph-mon[39462]: osdmap e319: 8 total, 7 up, 7 in 2024-09-17T07:17:16.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:16 smithi100 ceph-mon[29092]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:17:16.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:16 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T07:17:16.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:16 smithi100 ceph-mon[29092]: osdmap e319: 8 total, 7 up, 7 in 2024-09-17T07:17:16.903 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:16.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:16.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:17:17.005 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:17:17.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:16 smithi135 ceph-mon[33238]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:17:17.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:16 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T07:17:17.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:16 smithi135 ceph-mon[33238]: osdmap e319: 8 total, 7 up, 7 in 2024-09-17T07:17:17.046 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:17.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:17.047 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:17:17.148 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:17:17.172 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:17.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:17.173 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:17:17.274 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:17:17.307 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:17.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:17.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:17:17.410 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:17:17.438 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:17.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:17.442 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:17:17.542 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:17:17.583 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:17.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:17.584 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:17:17.630 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 59 16 91202 91186 96.5853 77.9375 0.00805565 0.0103492 2024-09-17T07:17:17.685 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:17:17.711 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:17.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:17.713 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:17:17.814 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:17:17.853 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:17.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:17.854 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:17:17.955 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:17:17.980 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:17.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:17.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:17:18.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:17 smithi135 ceph-mon[33238]: 2.19 scrub starts 2024-09-17T07:17:18.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:17 smithi135 ceph-mon[33238]: 2.19 scrub ok 2024-09-17T07:17:18.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:17 smithi135 ceph-mon[33238]: pgmap v679: 148 pgs: 36 active+undersized, 28 active+undersized+degraded, 84 active+clean; 5.3 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 853 B/s rd, 61 MiB/s wr, 974 op/s; 21685/260313 objects degraded (8.330%) 2024-09-17T07:17:18.082 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:17:18.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:18.105 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:18.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:17:18.126 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:17 smithi100 ceph-mon[29092]: 2.19 scrub starts 2024-09-17T07:17:18.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:17 smithi100 ceph-mon[29092]: 2.19 scrub ok 2024-09-17T07:17:18.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:17 smithi100 ceph-mon[29092]: pgmap v679: 148 pgs: 36 active+undersized, 28 active+undersized+degraded, 84 active+clean; 5.3 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 853 B/s rd, 61 MiB/s wr, 974 op/s; 21685/260313 objects degraded (8.330%) 2024-09-17T07:17:18.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:17 smithi100 ceph-mon[39462]: 2.19 scrub starts 2024-09-17T07:17:18.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:17 smithi100 ceph-mon[39462]: 2.19 scrub ok 2024-09-17T07:17:18.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:17 smithi100 ceph-mon[39462]: pgmap v679: 148 pgs: 36 active+undersized, 28 active+undersized+degraded, 84 active+clean; 5.3 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 853 B/s rd, 61 MiB/s wr, 974 op/s; 21685/260313 objects degraded (8.330%) 2024-09-17T07:17:18.206 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:17:18.233 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:18.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:18.234 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:17:18.335 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:17:18.373 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:18.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:18.374 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:17:18.475 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:17:18.516 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:18.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:18.517 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:17:18.618 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:17:18.630 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:2024-09-17T07:17:18.630698+0000 min lat: 0.00112621 max lat: 0.332984 avg lat: 0.0103829 2024-09-17T07:17:18.630 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-17T07:17:18.630 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 60 16 92441 92425 96.2661 77.4375 0.0118311 0.0103829 2024-09-17T07:17:18.643 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:18.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:18.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:17:18.745 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:17:18.774 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:18.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:18.775 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:17:18.859 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:18 smithi100 ceph-mon[29092]: 5.1c scrub starts 2024-09-17T07:17:18.859 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:18 smithi100 ceph-mon[29092]: 5.1c scrub ok 2024-09-17T07:17:18.859 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:18 smithi100 ceph-mon[29092]: 4.1f scrub starts 2024-09-17T07:17:18.859 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:18 smithi100 ceph-mon[29092]: 4.1f scrub ok 2024-09-17T07:17:18.859 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:18 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:17:18.859 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:18 smithi100 ceph-mon[39462]: 5.1c scrub starts 2024-09-17T07:17:18.859 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:18 smithi100 ceph-mon[39462]: 5.1c scrub ok 2024-09-17T07:17:18.859 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:18 smithi100 ceph-mon[39462]: 4.1f scrub starts 2024-09-17T07:17:18.860 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:18 smithi100 ceph-mon[39462]: 4.1f scrub ok 2024-09-17T07:17:18.860 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:18 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:17:18.876 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:17:18.916 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:18.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:18.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:17:18.963 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:18 smithi135 ceph-mon[33238]: 5.1c scrub starts 2024-09-17T07:17:18.963 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:18 smithi135 ceph-mon[33238]: 5.1c scrub ok 2024-09-17T07:17:18.964 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:18 smithi135 ceph-mon[33238]: 4.1f scrub starts 2024-09-17T07:17:18.964 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:18 smithi135 ceph-mon[33238]: 4.1f scrub ok 2024-09-17T07:17:18.964 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:18 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:17:19.017 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:17:19.062 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:19.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:19.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:17:19.164 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:17:19.205 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:19.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:19.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:17:19.233 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:17:19.263 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:17:19.293 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:17:18 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:17:19.308 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:17:19.357 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:19.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:19.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:17:19.459 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:17:19.498 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:19.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:19.499 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:17:19.600 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:17:19.630 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 61 16 93652 93636 95.9286 75.6875 0.0152397 0.0104198 2024-09-17T07:17:19.639 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:19.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:19.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:17:19.713 INFO:teuthology.orchestra.run.smithi100.stderr:set pool 10 pgp_num to 16 2024-09-17T07:17:19.741 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:17:19.782 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:19.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:19.783 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:17:19.884 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:17:19.927 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:19.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:19.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:17:20.028 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:17:20.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:19 smithi135 ceph-mon[33238]: 6.6 scrub starts 2024-09-17T07:17:20.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:19 smithi135 ceph-mon[33238]: 6.6 scrub ok 2024-09-17T07:17:20.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:19 smithi135 ceph-mon[33238]: 2.14 scrub starts 2024-09-17T07:17:20.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:19 smithi135 ceph-mon[33238]: 2.14 scrub ok 2024-09-17T07:17:20.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:19 smithi135 ceph-mon[33238]: 5.f scrub starts 2024-09-17T07:17:20.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:19 smithi135 ceph-mon[33238]: 5.f scrub ok 2024-09-17T07:17:20.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:19 smithi135 ceph-mon[33238]: 2.13 scrub starts 2024-09-17T07:17:20.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:19 smithi135 ceph-mon[33238]: 2.13 scrub ok 2024-09-17T07:17:20.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:19 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:17:20.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:19 smithi135 ceph-mon[33238]: pgmap v680: 148 pgs: 36 active+undersized, 28 active+undersized+degraded, 84 active+clean; 5.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.7 KiB/s rd, 112 MiB/s wr, 1.80k op/s; 22799/275100 objects degraded (8.288%) 2024-09-17T07:17:20.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:19 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2878037950' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "16"}]: dispatch 2024-09-17T07:17:20.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:19 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "16"}]: dispatch 2024-09-17T07:17:20.049 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:20.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:20.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:17:20.068 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:19 smithi100 ceph-mon[29092]: 6.6 scrub starts 2024-09-17T07:17:20.068 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:19 smithi100 ceph-mon[29092]: 6.6 scrub ok 2024-09-17T07:17:20.068 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:19 smithi100 ceph-mon[29092]: 2.14 scrub starts 2024-09-17T07:17:20.068 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:19 smithi100 ceph-mon[29092]: 2.14 scrub ok 2024-09-17T07:17:20.068 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:19 smithi100 ceph-mon[29092]: 5.f scrub starts 2024-09-17T07:17:20.068 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:19 smithi100 ceph-mon[29092]: 5.f scrub ok 2024-09-17T07:17:20.069 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:19 smithi100 ceph-mon[29092]: 2.13 scrub starts 2024-09-17T07:17:20.069 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:19 smithi100 ceph-mon[29092]: 2.13 scrub ok 2024-09-17T07:17:20.069 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:19 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:17:20.069 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:19 smithi100 ceph-mon[29092]: pgmap v680: 148 pgs: 36 active+undersized, 28 active+undersized+degraded, 84 active+clean; 5.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.7 KiB/s rd, 112 MiB/s wr, 1.80k op/s; 22799/275100 objects degraded (8.288%) 2024-09-17T07:17:20.069 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:19 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2878037950' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "16"}]: dispatch 2024-09-17T07:17:20.069 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:19 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "16"}]: dispatch 2024-09-17T07:17:20.069 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:19 smithi100 ceph-mon[39462]: 6.6 scrub starts 2024-09-17T07:17:20.069 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:19 smithi100 ceph-mon[39462]: 6.6 scrub ok 2024-09-17T07:17:20.069 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:19 smithi100 ceph-mon[39462]: 2.14 scrub starts 2024-09-17T07:17:20.069 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:19 smithi100 ceph-mon[39462]: 2.14 scrub ok 2024-09-17T07:17:20.069 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:19 smithi100 ceph-mon[39462]: 5.f scrub starts 2024-09-17T07:17:20.070 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:19 smithi100 ceph-mon[39462]: 5.f scrub ok 2024-09-17T07:17:20.070 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:19 smithi100 ceph-mon[39462]: 2.13 scrub starts 2024-09-17T07:17:20.070 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:19 smithi100 ceph-mon[39462]: 2.13 scrub ok 2024-09-17T07:17:20.070 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:19 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:17:20.070 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:19 smithi100 ceph-mon[39462]: pgmap v680: 148 pgs: 36 active+undersized, 28 active+undersized+degraded, 84 active+clean; 5.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.7 KiB/s rd, 112 MiB/s wr, 1.80k op/s; 22799/275100 objects degraded (8.288%) 2024-09-17T07:17:20.070 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:19 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2878037950' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "16"}]: dispatch 2024-09-17T07:17:20.070 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:19 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "16"}]: dispatch 2024-09-17T07:17:20.150 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:17:20.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:20.186 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:20.187 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:17:20.287 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:17:20.316 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:20.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:20.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:17:20.418 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:17:20.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:20.446 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:20.447 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:17:20.547 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:17:20.590 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:20.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:20.591 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:17:20.630 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 62 16 94786 94770 95.5245 70.875 0.0120767 0.0104633 2024-09-17T07:17:20.692 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:17:20.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:20.733 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:20.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:17:20.834 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:17:20.861 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:20.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:20.862 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:17:20.890 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-17T07:17:20.963 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:17:20.994 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:20.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:20.995 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:17:21.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:20 smithi135 ceph-mon[33238]: 2.7 scrub starts 2024-09-17T07:17:21.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:20 smithi135 ceph-mon[33238]: 2.7 scrub ok 2024-09-17T07:17:21.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:20 smithi135 ceph-mon[33238]: 3.2 scrub starts 2024-09-17T07:17:21.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:20 smithi135 ceph-mon[33238]: 3.2 scrub ok 2024-09-17T07:17:21.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:20 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "16"}]': finished 2024-09-17T07:17:21.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:20 smithi135 ceph-mon[33238]: osdmap e320: 8 total, 7 up, 7 in 2024-09-17T07:17:21.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:20 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 22799/275100 objects degraded (8.288%), 28 pgs degraded, 64 pgs undersized (PG_DEGRADED) 2024-09-17T07:17:21.096 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:17:21.126 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:20 smithi100 ceph-mon[39462]: 2.7 scrub starts 2024-09-17T07:17:21.126 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:20 smithi100 ceph-mon[39462]: 2.7 scrub ok 2024-09-17T07:17:21.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:20 smithi100 ceph-mon[39462]: 3.2 scrub starts 2024-09-17T07:17:21.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:20 smithi100 ceph-mon[39462]: 3.2 scrub ok 2024-09-17T07:17:21.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:20 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "16"}]': finished 2024-09-17T07:17:21.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:20 smithi100 ceph-mon[39462]: osdmap e320: 8 total, 7 up, 7 in 2024-09-17T07:17:21.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:20 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 22799/275100 objects degraded (8.288%), 28 pgs degraded, 64 pgs undersized (PG_DEGRADED) 2024-09-17T07:17:21.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:20 smithi100 ceph-mon[29092]: 2.7 scrub starts 2024-09-17T07:17:21.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:20 smithi100 ceph-mon[29092]: 2.7 scrub ok 2024-09-17T07:17:21.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:20 smithi100 ceph-mon[29092]: 3.2 scrub starts 2024-09-17T07:17:21.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:20 smithi100 ceph-mon[29092]: 3.2 scrub ok 2024-09-17T07:17:21.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:20 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "16"}]': finished 2024-09-17T07:17:21.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:20 smithi100 ceph-mon[29092]: osdmap e320: 8 total, 7 up, 7 in 2024-09-17T07:17:21.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:20 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 22799/275100 objects degraded (8.288%), 28 pgs degraded, 64 pgs undersized (PG_DEGRADED) 2024-09-17T07:17:21.137 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:21.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:21.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:17:21.238 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:17:21.265 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:21.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:21.266 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:17:21.367 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:17:21.415 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:21.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:21.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:17:21.517 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:17:21.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:21.560 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:21.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:17:21.630 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 63 16 95947 95931 95.1599 72.5625 0.00548204 0.010504 2024-09-17T07:17:21.661 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:17:21.694 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:21.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:21.695 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:17:21.796 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:17:21.822 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:21.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:21.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:17:21.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:21 smithi100 ceph-mon[29092]: 10.1 scrub starts 2024-09-17T07:17:21.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:21 smithi100 ceph-mon[29092]: 2.12 scrub starts 2024-09-17T07:17:21.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:21 smithi100 ceph-mon[29092]: 2.12 scrub ok 2024-09-17T07:17:21.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:21 smithi100 ceph-mon[29092]: 2.c scrub starts 2024-09-17T07:17:21.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:21 smithi100 ceph-mon[29092]: 2.c scrub ok 2024-09-17T07:17:21.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:21 smithi100 ceph-mon[29092]: pgmap v682: 148 pgs: 36 active+undersized, 28 active+undersized+degraded, 84 active+clean; 5.7 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 844 B/s rd, 62 MiB/s wr, 989 op/s; 23176/278298 objects degraded (8.328%) 2024-09-17T07:17:21.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:21 smithi100 ceph-mon[39462]: 10.1 scrub starts 2024-09-17T07:17:21.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:21 smithi100 ceph-mon[39462]: 2.12 scrub starts 2024-09-17T07:17:21.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:21 smithi100 ceph-mon[39462]: 2.12 scrub ok 2024-09-17T07:17:21.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:21 smithi100 ceph-mon[39462]: 2.c scrub starts 2024-09-17T07:17:21.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:21 smithi100 ceph-mon[39462]: 2.c scrub ok 2024-09-17T07:17:21.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:21 smithi100 ceph-mon[39462]: pgmap v682: 148 pgs: 36 active+undersized, 28 active+undersized+degraded, 84 active+clean; 5.7 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 844 B/s rd, 62 MiB/s wr, 989 op/s; 23176/278298 objects degraded (8.328%) 2024-09-17T07:17:21.923 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:17:21.948 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:21.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:21.949 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:17:22.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:21 smithi135 ceph-mon[33238]: 10.1 scrub starts 2024-09-17T07:17:22.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:21 smithi135 ceph-mon[33238]: 2.12 scrub starts 2024-09-17T07:17:22.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:21 smithi135 ceph-mon[33238]: 2.12 scrub ok 2024-09-17T07:17:22.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:21 smithi135 ceph-mon[33238]: 2.c scrub starts 2024-09-17T07:17:22.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:21 smithi135 ceph-mon[33238]: 2.c scrub ok 2024-09-17T07:17:22.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:21 smithi135 ceph-mon[33238]: pgmap v682: 148 pgs: 36 active+undersized, 28 active+undersized+degraded, 84 active+clean; 5.7 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 844 B/s rd, 62 MiB/s wr, 989 op/s; 23176/278298 objects degraded (8.328%) 2024-09-17T07:17:22.050 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:17:22.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:22.095 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:22.095 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:17:22.196 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:17:22.226 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:22.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:22.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:17:22.328 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:17:22.354 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:22.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:22.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:17:22.456 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:17:22.486 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:22.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:22.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:17:22.588 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:17:22.630 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 64 16 96996 96980 94.6974 65.5625 0.013617 0.0105547 2024-09-17T07:17:22.633 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:22.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:22.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:17:22.735 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:17:22.779 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:22.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:22.780 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:17:22.882 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:17:22.925 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:22.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:22.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:17:23.027 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:17:23.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:22 smithi135 ceph-mon[33238]: 4.19 scrub starts 2024-09-17T07:17:23.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:22 smithi135 ceph-mon[33238]: 4.19 scrub ok 2024-09-17T07:17:23.068 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:23.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:23.069 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:17:23.109 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:22 smithi100 ceph-mon[39462]: 4.19 scrub starts 2024-09-17T07:17:23.109 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:22 smithi100 ceph-mon[39462]: 4.19 scrub ok 2024-09-17T07:17:23.110 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:22 smithi100 ceph-mon[29092]: 4.19 scrub starts 2024-09-17T07:17:23.110 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:22 smithi100 ceph-mon[29092]: 4.19 scrub ok 2024-09-17T07:17:23.169 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:17:23.210 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:23.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:23.211 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:17:23.237 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:17:23.248 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:17:23.249 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:17:23.249 INFO:teuthology.orchestra.run.smithi100.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-17T07:17:23.277 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:17:23.278 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:17:23.301 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:17:23.302 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:17:23.312 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:17:23.337 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:17:23.338 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:17:23.343 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:23.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:23.345 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:17:23.366 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:17:23.366 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:17:23.395 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:17:23.395 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:17:23.428 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:17:23.429 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:17:23.445 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:17:23.485 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:23.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:23.486 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:17:23.586 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:17:23.617 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:23.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:23.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:17:23.631 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 65 16 98197 98181 94.3952 75.0625 0.00815403 0.010589 2024-09-17T07:17:23.718 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:17:23.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:23.757 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:23.757 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:17:23.859 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:17:23.894 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:23.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:23.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:17:23.996 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:17:24.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:24.026 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:24.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:17:24.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:23 smithi135 ceph-mon[33238]: 10.1 scrub ok 2024-09-17T07:17:24.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:23 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:17:24.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:23 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:17:24.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:23 smithi135 ceph-mon[33238]: 5.10 scrub starts 2024-09-17T07:17:24.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:23 smithi135 ceph-mon[33238]: 5.10 scrub ok 2024-09-17T07:17:24.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:23 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/421005706' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:17:24.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:23 smithi135 ceph-mon[33238]: pgmap v683: 148 pgs: 36 active+undersized, 28 active+undersized+degraded, 84 active+clean; 5.7 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 639 B/s rd, 47 MiB/s wr, 749 op/s; 23176/278298 objects degraded (8.328%) 2024-09-17T07:17:24.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:23 smithi100 ceph-mon[29092]: 10.1 scrub ok 2024-09-17T07:17:24.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:23 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:17:24.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:23 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:17:24.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:23 smithi100 ceph-mon[29092]: 5.10 scrub starts 2024-09-17T07:17:24.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:23 smithi100 ceph-mon[29092]: 5.10 scrub ok 2024-09-17T07:17:24.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:23 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/421005706' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:17:24.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:23 smithi100 ceph-mon[29092]: pgmap v683: 148 pgs: 36 active+undersized, 28 active+undersized+degraded, 84 active+clean; 5.7 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 639 B/s rd, 47 MiB/s wr, 749 op/s; 23176/278298 objects degraded (8.328%) 2024-09-17T07:17:24.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:23 smithi100 ceph-mon[39462]: 10.1 scrub ok 2024-09-17T07:17:24.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:23 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:17:24.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:23 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:17:24.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:23 smithi100 ceph-mon[39462]: 5.10 scrub starts 2024-09-17T07:17:24.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:23 smithi100 ceph-mon[39462]: 5.10 scrub ok 2024-09-17T07:17:24.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:23 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/421005706' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:17:24.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:23 smithi100 ceph-mon[39462]: pgmap v683: 148 pgs: 36 active+undersized, 28 active+undersized+degraded, 84 active+clean; 5.7 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 639 B/s rd, 47 MiB/s wr, 749 op/s; 23176/278298 objects degraded (8.328%) 2024-09-17T07:17:24.128 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:17:24.168 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:24.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:24.169 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:17:24.270 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:17:24.273 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:17:24.273 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:17:24.303 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:24.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:24.304 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:17:24.405 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:17:24.437 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:24.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:24.438 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:17:24.488 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-17T07:17:24.488 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:17:24.539 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:17:24.565 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:24.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:24.566 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:17:24.631 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 66 15 99371 99356 94.0776 73.4375 0.00718743 0.0106246 2024-09-17T07:17:24.667 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:17:24.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:24.713 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:24.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:17:24.814 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:17:24.851 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:24.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:24.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:17:24.872 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:24 smithi100 ceph-mon[29092]: 10.9 scrub starts 2024-09-17T07:17:24.872 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:24 smithi100 ceph-mon[29092]: osdmap e321: 8 total, 7 up, 7 in 2024-09-17T07:17:24.872 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:24 smithi100 ceph-mon[39462]: 10.9 scrub starts 2024-09-17T07:17:24.872 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:24 smithi100 ceph-mon[39462]: osdmap e321: 8 total, 7 up, 7 in 2024-09-17T07:17:24.953 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:17:24.989 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:24.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:24.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:17:25.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:24 smithi135 ceph-mon[33238]: 10.9 scrub starts 2024-09-17T07:17:25.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:24 smithi135 ceph-mon[33238]: osdmap e321: 8 total, 7 up, 7 in 2024-09-17T07:17:25.091 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:17:25.134 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:25.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:25.135 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:17:25.236 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:17:25.269 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:25.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:25.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:17:25.371 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:17:25.393 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:25.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:25.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:17:25.495 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:17:25.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:25.530 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:25.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:17:25.631 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 67 16 100246 100230 93.4887 54.625 0.00766885 0.0106916 2024-09-17T07:17:25.631 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:17:25.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:25.665 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:25.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:17:25.766 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:17:25.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:25.793 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:25.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:17:25.894 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:17:25.920 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:25.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:25.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:17:25.978 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-17T07:17:26.008 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:17:25 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:17:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:17:26.009 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:25 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 23176/278298 objects degraded (8.328%), 28 pgs degraded, 64 pgs undersized (PG_DEGRADED) 2024-09-17T07:17:26.009 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:25 smithi100 ceph-mon[29092]: osdmap e322: 8 total, 7 up, 7 in 2024-09-17T07:17:26.009 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:25 smithi100 ceph-mon[29092]: pgmap v686: 148 pgs: 1 active+clean+scrubbing, 36 active+undersized, 28 active+undersized+degraded, 83 active+clean; 6.0 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 852 B/s rd, 71 MiB/s wr, 1.14k op/s; 24607/295641 objects degraded (8.323%) 2024-09-17T07:17:26.009 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:25 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 23176/278298 objects degraded (8.328%), 28 pgs degraded, 64 pgs undersized (PG_DEGRADED) 2024-09-17T07:17:26.009 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:25 smithi100 ceph-mon[39462]: osdmap e322: 8 total, 7 up, 7 in 2024-09-17T07:17:26.009 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:25 smithi100 ceph-mon[39462]: pgmap v686: 148 pgs: 1 active+clean+scrubbing, 36 active+undersized, 28 active+undersized+degraded, 83 active+clean; 6.0 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 852 B/s rd, 71 MiB/s wr, 1.14k op/s; 24607/295641 objects degraded (8.323%) 2024-09-17T07:17:26.023 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:17:26.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:25 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 23176/278298 objects degraded (8.328%), 28 pgs degraded, 64 pgs undersized (PG_DEGRADED) 2024-09-17T07:17:26.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:25 smithi135 ceph-mon[33238]: osdmap e322: 8 total, 7 up, 7 in 2024-09-17T07:17:26.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:25 smithi135 ceph-mon[33238]: pgmap v686: 148 pgs: 1 active+clean+scrubbing, 36 active+undersized, 28 active+undersized+degraded, 83 active+clean; 6.0 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 852 B/s rd, 71 MiB/s wr, 1.14k op/s; 24607/295641 objects degraded (8.323%) 2024-09-17T07:17:26.058 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:26.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:26.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:17:26.160 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:17:26.189 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:26.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:26.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:17:26.274 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:17:26.291 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:17:26.327 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:26.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:26.328 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:17:26.429 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:17:26.467 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:26.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:26.468 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:17:26.569 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:17:26.570 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:17:26.607 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:26.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:26.608 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:17:26.631 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 68 16 101528 101512 93.2921 80.125 0.0104571 0.0107151 2024-09-17T07:17:26.709 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:17:26.738 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:26.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:26.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:17:26.834 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:17:26.840 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:17:26.874 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:26.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:26.875 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:17:26.935 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:26 smithi100 ceph-mon[39462]: 5.1f scrub starts 2024-09-17T07:17:26.935 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:26 smithi100 ceph-mon[39462]: 5.1f scrub ok 2024-09-17T07:17:26.935 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:26 smithi100 ceph-mon[39462]: 10.9 scrub ok 2024-09-17T07:17:26.935 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:26 smithi100 ceph-mon[39462]: 3.1 scrub starts 2024-09-17T07:17:26.935 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:26 smithi100 ceph-mon[39462]: 3.1 scrub ok 2024-09-17T07:17:26.936 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:26 smithi100 ceph-mon[29092]: 5.1f scrub starts 2024-09-17T07:17:26.936 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:26 smithi100 ceph-mon[29092]: 5.1f scrub ok 2024-09-17T07:17:26.936 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:26 smithi100 ceph-mon[29092]: 10.9 scrub ok 2024-09-17T07:17:26.936 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:26 smithi100 ceph-mon[29092]: 3.1 scrub starts 2024-09-17T07:17:26.936 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:26 smithi100 ceph-mon[29092]: 3.1 scrub ok 2024-09-17T07:17:26.975 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:17:27.023 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:27.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:27.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:17:27.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:26 smithi135 ceph-mon[33238]: 5.1f scrub starts 2024-09-17T07:17:27.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:26 smithi135 ceph-mon[33238]: 5.1f scrub ok 2024-09-17T07:17:27.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:26 smithi135 ceph-mon[33238]: 10.9 scrub ok 2024-09-17T07:17:27.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:26 smithi135 ceph-mon[33238]: 3.1 scrub starts 2024-09-17T07:17:27.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:26 smithi135 ceph-mon[33238]: 3.1 scrub ok 2024-09-17T07:17:27.125 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:17:27.155 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:27.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:27.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:17:27.218 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7] out_osds: [3] dead_osds: [1] live_osds: [0, 2, 3, 4, 5, 6, 7] 2024-09-17T07:17:27.218 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T07:17:27.218 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_3 2024-09-17T07:17:27.219 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T07:17:27.258 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:17:27.295 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:27.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:27.296 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:17:27.397 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:17:27.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:27.423 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:27.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:17:27.523 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:17:27.560 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:27.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:27.561 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:17:27.631 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 69 16 102760 102744 93.0559 77 0.00846464 0.010742 2024-09-17T07:17:27.662 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:17:27.691 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:27.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:27.693 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:17:27.794 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:17:27.825 INFO:teuthology.orchestra.run.smithi100.stderr:nodeep-scrub is unset 2024-09-17T07:17:27.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:27.833 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:27.833 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:17:27.934 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:17:27.963 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:27.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:27.964 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:17:28.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:27 smithi135 ceph-mon[33238]: pgmap v687: 148 pgs: 1 active+clean+scrubbing, 36 active+undersized, 28 active+undersized+degraded, 83 active+clean; 6.0 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 673 B/s rd, 48 MiB/s wr, 761 op/s; 24607/295641 objects degraded (8.323%) 2024-09-17T07:17:28.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:27 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1533329112' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:17:28.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:27 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:17:28.065 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:17:28.091 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:28.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:28.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:17:28.126 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:27 smithi100 ceph-mon[29092]: pgmap v687: 148 pgs: 1 active+clean+scrubbing, 36 active+undersized, 28 active+undersized+degraded, 83 active+clean; 6.0 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 673 B/s rd, 48 MiB/s wr, 761 op/s; 24607/295641 objects degraded (8.323%) 2024-09-17T07:17:28.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:27 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1533329112' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:17:28.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:27 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:17:28.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:27 smithi100 ceph-mon[39462]: pgmap v687: 148 pgs: 1 active+clean+scrubbing, 36 active+undersized, 28 active+undersized+degraded, 83 active+clean; 6.0 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 673 B/s rd, 48 MiB/s wr, 761 op/s; 24607/295641 objects degraded (8.323%) 2024-09-17T07:17:28.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:27 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1533329112' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:17:28.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:27 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:17:28.194 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:17:28.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:28.237 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:28.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:17:28.338 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:17:28.381 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:28.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:28.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:17:28.483 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:17:28.515 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:28.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:28.517 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:17:28.617 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:17:28.631 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 70 16 104021 104005 92.8523 78.8125 0.0131263 0.0107648 2024-09-17T07:17:28.652 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:28.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:28.653 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:17:28.754 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:17:28.785 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:28.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:28.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:17:28.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:28 smithi100 ceph-mon[29092]: 6.9 scrub starts 2024-09-17T07:17:28.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:28 smithi100 ceph-mon[29092]: 6.9 scrub ok 2024-09-17T07:17:28.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:28 smithi100 ceph-mon[29092]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T07:17:28.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:28 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T07:17:28.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:28 smithi100 ceph-mon[29092]: osdmap e323: 8 total, 7 up, 7 in 2024-09-17T07:17:28.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:28 smithi100 ceph-mon[29092]: 6.12 scrub starts 2024-09-17T07:17:28.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:28 smithi100 ceph-mon[29092]: 6.12 scrub ok 2024-09-17T07:17:28.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:28 smithi100 ceph-mon[39462]: 6.9 scrub starts 2024-09-17T07:17:28.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:28 smithi100 ceph-mon[39462]: 6.9 scrub ok 2024-09-17T07:17:28.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:28 smithi100 ceph-mon[39462]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T07:17:28.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:28 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T07:17:28.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:28 smithi100 ceph-mon[39462]: osdmap e323: 8 total, 7 up, 7 in 2024-09-17T07:17:28.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:28 smithi100 ceph-mon[39462]: 6.12 scrub starts 2024-09-17T07:17:28.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:28 smithi100 ceph-mon[39462]: 6.12 scrub ok 2024-09-17T07:17:28.886 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:17:28.915 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:28.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:28.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:17:29.017 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:17:29.043 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:17:28 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:17:29.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:28 smithi135 ceph-mon[33238]: 6.9 scrub starts 2024-09-17T07:17:29.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:28 smithi135 ceph-mon[33238]: 6.9 scrub ok 2024-09-17T07:17:29.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:28 smithi135 ceph-mon[33238]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T07:17:29.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:28 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T07:17:29.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:28 smithi135 ceph-mon[33238]: osdmap e323: 8 total, 7 up, 7 in 2024-09-17T07:17:29.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:28 smithi135 ceph-mon[33238]: 6.12 scrub starts 2024-09-17T07:17:29.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:28 smithi135 ceph-mon[33238]: 6.12 scrub ok 2024-09-17T07:17:29.045 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:29.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:29.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:17:29.147 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:17:29.171 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:29.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:29.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:17:29.273 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:17:29.315 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:29.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:29.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:17:29.417 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:17:29.458 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:29.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:29.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:17:29.560 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:17:29.604 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:29.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:29.606 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:17:29.631 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 71 16 105216 105200 92.5964 74.6875 0.00845808 0.0107956 2024-09-17T07:17:29.706 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:17:29.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:29.744 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:29.745 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:17:29.845 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:17:29.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:29.877 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:29.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:17:29.978 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:17:30.003 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:30.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:30.004 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:17:30.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:29 smithi135 ceph-mon[33238]: 10.6 scrub starts 2024-09-17T07:17:30.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:29 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:17:30.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:29 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:17:30.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:29 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:17:30.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:29 smithi135 ceph-mon[33238]: pgmap v689: 148 pgs: 1 active+clean+scrubbing, 36 active+undersized, 28 active+undersized+degraded, 83 active+clean; 6.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.7 KiB/s rd, 110 MiB/s wr, 1.76k op/s; 25715/309966 objects degraded (8.296%) 2024-09-17T07:17:30.105 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:17:30.126 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:29 smithi100 ceph-mon[39462]: 10.6 scrub starts 2024-09-17T07:17:30.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:29 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:17:30.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:29 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:17:30.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:29 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:17:30.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:29 smithi100 ceph-mon[39462]: pgmap v689: 148 pgs: 1 active+clean+scrubbing, 36 active+undersized, 28 active+undersized+degraded, 83 active+clean; 6.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.7 KiB/s rd, 110 MiB/s wr, 1.76k op/s; 25715/309966 objects degraded (8.296%) 2024-09-17T07:17:30.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:29 smithi100 ceph-mon[29092]: 10.6 scrub starts 2024-09-17T07:17:30.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:29 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:17:30.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:29 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:17:30.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:29 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:17:30.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:29 smithi100 ceph-mon[29092]: pgmap v689: 148 pgs: 1 active+clean+scrubbing, 36 active+undersized, 28 active+undersized+degraded, 83 active+clean; 6.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.7 KiB/s rd, 110 MiB/s wr, 1.76k op/s; 25715/309966 objects degraded (8.296%) 2024-09-17T07:17:30.146 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:30.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:30.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:17:30.248 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:17:30.287 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:30.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:30.288 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:17:30.388 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:17:30.414 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:30.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:30.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:17:30.516 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:17:30.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:30.543 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:30.543 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:17:30.631 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 72 16 106346 106330 92.2912 70.625 0.0255707 0.0108296 2024-09-17T07:17:30.644 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:17:30.668 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:30.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:30.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:17:30.770 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:17:30.797 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:30.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:30.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:17:30.813 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:30 smithi100 ceph-mon[29092]: osdmap e324: 8 total, 7 up, 7 in 2024-09-17T07:17:30.813 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:30 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:17:30.813 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:30 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:17:30.813 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:30 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 25715/309966 objects degraded (8.296%), 28 pgs degraded, 64 pgs undersized (PG_DEGRADED) 2024-09-17T07:17:30.813 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:30 smithi100 ceph-mon[39462]: osdmap e324: 8 total, 7 up, 7 in 2024-09-17T07:17:30.814 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:30 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:17:30.814 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:30 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:17:30.814 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:30 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 25715/309966 objects degraded (8.296%), 28 pgs degraded, 64 pgs undersized (PG_DEGRADED) 2024-09-17T07:17:30.900 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:17:30.943 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:30.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:30.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:17:31.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:30 smithi135 ceph-mon[33238]: osdmap e324: 8 total, 7 up, 7 in 2024-09-17T07:17:31.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:30 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:17:31.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:30 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:17:31.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:30 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 25715/309966 objects degraded (8.296%), 28 pgs degraded, 64 pgs undersized (PG_DEGRADED) 2024-09-17T07:17:31.045 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:17:31.062 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:17:31.065 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:31.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:31.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:17:31.066 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:17:31.067 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:17:31.067 INFO:teuthology.orchestra.run.smithi100.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-17T07:17:31.091 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:17:31.092 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:17:31.116 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:17:31.116 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:17:31.146 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:17:31.146 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:17:31.167 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:17:31.172 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:17:31.172 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:17:31.202 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:17:31.202 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:17:31.206 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:31.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:31.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:17:31.231 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:17:31.232 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:17:31.306 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-17T07:17:31.306 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-17T07:17:31.306 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-17T07:17:31.306 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-17T07:17:31.306 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-17T07:17:31.308 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:17:31.339 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:31.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:31.341 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:17:31.441 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:17:31.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:31.480 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:31.481 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:17:31.581 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:17:31.610 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:31.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:31.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:17:31.631 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 73 16 107328 107312 91.8676 61.375 0.00529172 0.0108797 2024-09-17T07:17:31.712 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:17:31.748 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:31.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:31.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:17:31.850 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:17:31.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:31 smithi100 ceph-mon[29092]: 10.3 deep-scrub starts 2024-09-17T07:17:31.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:31 smithi100 ceph-mon[29092]: osdmap e325: 8 total, 7 up, 7 in 2024-09-17T07:17:31.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:31 smithi100 ceph-mon[29092]: 10.6 scrub ok 2024-09-17T07:17:31.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:31 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/673464966' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:17:31.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:31 smithi100 ceph-mon[29092]: 4.1 scrub starts 2024-09-17T07:17:31.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:31 smithi100 ceph-mon[29092]: 4.1 scrub ok 2024-09-17T07:17:31.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:31 smithi100 ceph-mon[29092]: pgmap v692: 148 pgs: 1 active+clean+scrubbing, 36 active+undersized, 28 active+undersized+degraded, 83 active+clean; 6.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 852 B/s rd, 61 MiB/s wr, 974 op/s; 26080/313176 objects degraded (8.328%) 2024-09-17T07:17:31.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:31 smithi100 ceph-mon[39462]: 10.3 deep-scrub starts 2024-09-17T07:17:31.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:31 smithi100 ceph-mon[39462]: osdmap e325: 8 total, 7 up, 7 in 2024-09-17T07:17:31.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:31 smithi100 ceph-mon[39462]: 10.6 scrub ok 2024-09-17T07:17:31.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:31 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/673464966' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:17:31.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:31 smithi100 ceph-mon[39462]: 4.1 scrub starts 2024-09-17T07:17:31.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:31 smithi100 ceph-mon[39462]: 4.1 scrub ok 2024-09-17T07:17:31.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:31 smithi100 ceph-mon[39462]: pgmap v692: 148 pgs: 1 active+clean+scrubbing, 36 active+undersized, 28 active+undersized+degraded, 83 active+clean; 6.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 852 B/s rd, 61 MiB/s wr, 974 op/s; 26080/313176 objects degraded (8.328%) 2024-09-17T07:17:31.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:31.879 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:31.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:17:31.956 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:17:31.957 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:17:31.979 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:17:32.021 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:32.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:32.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:17:32.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:31 smithi135 ceph-mon[33238]: 10.3 deep-scrub starts 2024-09-17T07:17:32.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:31 smithi135 ceph-mon[33238]: osdmap e325: 8 total, 7 up, 7 in 2024-09-17T07:17:32.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:31 smithi135 ceph-mon[33238]: 10.6 scrub ok 2024-09-17T07:17:32.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:31 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/673464966' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:17:32.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:31 smithi135 ceph-mon[33238]: 4.1 scrub starts 2024-09-17T07:17:32.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:31 smithi135 ceph-mon[33238]: 4.1 scrub ok 2024-09-17T07:17:32.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:31 smithi135 ceph-mon[33238]: pgmap v692: 148 pgs: 1 active+clean+scrubbing, 36 active+undersized, 28 active+undersized+degraded, 83 active+clean; 6.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 852 B/s rd, 61 MiB/s wr, 974 op/s; 26080/313176 objects degraded (8.328%) 2024-09-17T07:17:32.123 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:17:32.148 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:32.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:32.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:17:32.250 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:17:32.285 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:32.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:32.286 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:17:32.387 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:17:32.430 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:32.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:32.431 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:17:32.532 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:17:32.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:32.579 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:32.579 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:17:32.631 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 74 16 108434 108418 91.5602 69.125 0.0123743 0.0109165 2024-09-17T07:17:32.680 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:17:32.719 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:32.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:32.720 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:17:32.821 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:17:32.848 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:32.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:32.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:17:32.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:32 smithi100 ceph-mon[29092]: 5.1e scrub starts 2024-09-17T07:17:32.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:32 smithi100 ceph-mon[29092]: 5.1e scrub ok 2024-09-17T07:17:32.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:32 smithi100 ceph-mon[39462]: 5.1e scrub starts 2024-09-17T07:17:32.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:32 smithi100 ceph-mon[39462]: 5.1e scrub ok 2024-09-17T07:17:32.950 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:17:32.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:32.980 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:32.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:17:33.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:32 smithi135 ceph-mon[33238]: 5.1e scrub starts 2024-09-17T07:17:33.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:32 smithi135 ceph-mon[33238]: 5.1e scrub ok 2024-09-17T07:17:33.081 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:17:33.106 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:33.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:33.107 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:17:33.208 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:17:33.234 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:33.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:33.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:17:33.336 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:17:33.364 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:33.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:33.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:17:33.466 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:17:33.582 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:33.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:33.583 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:17:33.631 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 75 16 109543 109527 91.2635 69.3125 0.0433094 0.0109526 2024-09-17T07:17:33.684 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:17:33.711 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:33.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:33.712 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:17:33.813 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:17:33.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:33.849 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:33.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:17:33.950 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:17:33.958 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:17:33.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:33.985 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:33.985 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:17:34.086 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:17:34.121 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:34.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:34.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:17:34.224 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:17:34.257 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:34.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:34.258 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:17:34.359 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:17:34.376 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:34 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:17:34.376 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:34 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:17:34.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:34 smithi100 ceph-mon[39462]: pgmap v693: 148 pgs: 1 active+clean+scrubbing, 36 active+undersized, 28 active+undersized+degraded, 83 active+clean; 6.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 853 B/s rd, 61 MiB/s wr, 974 op/s; 26080/313176 objects degraded (8.328%) 2024-09-17T07:17:34.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:34 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:17:34.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:34 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:17:34.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:34 smithi100 ceph-mon[29092]: pgmap v693: 148 pgs: 1 active+clean+scrubbing, 36 active+undersized, 28 active+undersized+degraded, 83 active+clean; 6.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 853 B/s rd, 61 MiB/s wr, 974 op/s; 26080/313176 objects degraded (8.328%) 2024-09-17T07:17:34.395 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:34.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:34.396 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:17:34.497 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:17:34.523 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:34.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:34.524 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:17:34.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:34 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:17:34.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:34 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:17:34.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:34 smithi135 ceph-mon[33238]: pgmap v693: 148 pgs: 1 active+clean+scrubbing, 36 active+undersized, 28 active+undersized+degraded, 83 active+clean; 6.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 853 B/s rd, 61 MiB/s wr, 974 op/s; 26080/313176 objects degraded (8.328%) 2024-09-17T07:17:34.625 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:17:34.630 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-17T07:17:34.633 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 76 16 110668 110652 90.9878 70.3125 0.00534629 0.0109859 2024-09-17T07:17:34.667 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:34.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:34.668 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:17:34.764 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-17T07:17:34.769 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:17:34.803 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:34.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:34.804 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:17:34.807 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:17:34.905 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:17:34.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:34.936 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:34.936 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:17:35.037 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:17:35.063 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:35.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:35.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:17:35.164 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:17:35.196 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:35 smithi100 ceph-mon[29092]: 4.11 scrub starts 2024-09-17T07:17:35.197 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:35 smithi100 ceph-mon[29092]: 4.11 scrub ok 2024-09-17T07:17:35.197 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:35 smithi100 ceph-mon[29092]: 4.9 scrub starts 2024-09-17T07:17:35.197 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:35 smithi100 ceph-mon[29092]: 4.9 scrub ok 2024-09-17T07:17:35.197 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:35 smithi100 ceph-mon[29092]: 5.11 scrub starts 2024-09-17T07:17:35.197 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:35 smithi100 ceph-mon[29092]: 5.11 scrub ok 2024-09-17T07:17:35.197 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:35 smithi100 ceph-mon[39462]: 4.11 scrub starts 2024-09-17T07:17:35.197 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:35 smithi100 ceph-mon[39462]: 4.11 scrub ok 2024-09-17T07:17:35.197 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:35 smithi100 ceph-mon[39462]: 4.9 scrub starts 2024-09-17T07:17:35.197 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:35 smithi100 ceph-mon[39462]: 4.9 scrub ok 2024-09-17T07:17:35.197 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:35 smithi100 ceph-mon[39462]: 5.11 scrub starts 2024-09-17T07:17:35.198 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:35 smithi100 ceph-mon[39462]: 5.11 scrub ok 2024-09-17T07:17:35.200 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:35.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:35.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:17:35.303 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:17:35.344 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:35.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:35.345 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:17:35.446 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:17:35.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:35.485 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:35.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:17:35.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:35 smithi135 ceph-mon[33238]: 4.11 scrub starts 2024-09-17T07:17:35.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:35 smithi135 ceph-mon[33238]: 4.11 scrub ok 2024-09-17T07:17:35.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:35 smithi135 ceph-mon[33238]: 4.9 scrub starts 2024-09-17T07:17:35.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:35 smithi135 ceph-mon[33238]: 4.9 scrub ok 2024-09-17T07:17:35.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:35 smithi135 ceph-mon[33238]: 5.11 scrub starts 2024-09-17T07:17:35.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:35 smithi135 ceph-mon[33238]: 5.11 scrub ok 2024-09-17T07:17:35.586 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:17:35.612 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:35.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:35.613 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:17:35.631 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 77 16 111697 111681 90.6413 64.3125 0.00964864 0.0110254 2024-09-17T07:17:35.715 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:17:35.742 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:35.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:35.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:17:35.843 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:17:35.874 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:35.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:35.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:17:35.976 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:17:36.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:36.019 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:36.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:17:36.120 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:17:36.147 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:36.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:36.149 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:17:36.249 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:17:36.272 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:36.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:36.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:17:36.339 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:17:35 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:17:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:17:36.340 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:36 smithi100 ceph-mon[29092]: 6.13 scrub starts 2024-09-17T07:17:36.340 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:36 smithi100 ceph-mon[29092]: 6.13 scrub ok 2024-09-17T07:17:36.340 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:36 smithi100 ceph-mon[29092]: 2.3 scrub starts 2024-09-17T07:17:36.340 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:36 smithi100 ceph-mon[29092]: 2.3 scrub ok 2024-09-17T07:17:36.340 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:36 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 26080/313176 objects degraded (8.328%), 28 pgs degraded, 64 pgs undersized (PG_DEGRADED) 2024-09-17T07:17:36.340 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:36 smithi100 ceph-mon[29092]: osdmap e326: 8 total, 7 up, 7 in 2024-09-17T07:17:36.340 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:36 smithi100 ceph-mon[29092]: pgmap v695: 148 pgs: 1 active+clean+scrubbing+deep, 36 active+undersized, 28 active+undersized+degraded, 83 active+clean; 6.7 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 852 B/s rd, 69 MiB/s wr, 1.10k op/s; 27465/329739 objects degraded (8.329%) 2024-09-17T07:17:36.341 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:36 smithi100 ceph-mon[39462]: 6.13 scrub starts 2024-09-17T07:17:36.341 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:36 smithi100 ceph-mon[39462]: 6.13 scrub ok 2024-09-17T07:17:36.341 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:36 smithi100 ceph-mon[39462]: 2.3 scrub starts 2024-09-17T07:17:36.341 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:36 smithi100 ceph-mon[39462]: 2.3 scrub ok 2024-09-17T07:17:36.341 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:36 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 26080/313176 objects degraded (8.328%), 28 pgs degraded, 64 pgs undersized (PG_DEGRADED) 2024-09-17T07:17:36.341 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:36 smithi100 ceph-mon[39462]: osdmap e326: 8 total, 7 up, 7 in 2024-09-17T07:17:36.341 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:36 smithi100 ceph-mon[39462]: pgmap v695: 148 pgs: 1 active+clean+scrubbing+deep, 36 active+undersized, 28 active+undersized+degraded, 83 active+clean; 6.7 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 852 B/s rd, 69 MiB/s wr, 1.10k op/s; 27465/329739 objects degraded (8.329%) 2024-09-17T07:17:36.374 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:17:36.402 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:36.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:36.403 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:17:36.504 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:17:36.529 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:36.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:36.530 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:17:36.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:36 smithi135 ceph-mon[33238]: 6.13 scrub starts 2024-09-17T07:17:36.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:36 smithi135 ceph-mon[33238]: 6.13 scrub ok 2024-09-17T07:17:36.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:36 smithi135 ceph-mon[33238]: 2.3 scrub starts 2024-09-17T07:17:36.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:36 smithi135 ceph-mon[33238]: 2.3 scrub ok 2024-09-17T07:17:36.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:36 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 26080/313176 objects degraded (8.328%), 28 pgs degraded, 64 pgs undersized (PG_DEGRADED) 2024-09-17T07:17:36.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:36 smithi135 ceph-mon[33238]: osdmap e326: 8 total, 7 up, 7 in 2024-09-17T07:17:36.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:36 smithi135 ceph-mon[33238]: pgmap v695: 148 pgs: 1 active+clean+scrubbing+deep, 36 active+undersized, 28 active+undersized+degraded, 83 active+clean; 6.7 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 852 B/s rd, 69 MiB/s wr, 1.10k op/s; 27465/329739 objects degraded (8.329%) 2024-09-17T07:17:36.631 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:17:36.632 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 78 16 112727 112711 90.3045 64.375 0.01104 0.0110691 2024-09-17T07:17:36.673 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:36.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:36.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:17:36.775 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:17:36.810 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:36.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:36.812 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:17:36.913 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:17:36.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:36.940 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:36.940 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:17:37.041 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:17:37.085 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:37.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:37.086 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:17:37.187 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:17:37.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:37.225 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:37.226 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:17:37.285 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:17:37.306 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:17:37.326 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:17:37.368 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:37.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:37.369 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:17:37.470 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:17:37.506 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:37.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:37.507 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:17:37.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:37 smithi135 ceph-mon[33238]: 2.1e deep-scrub starts 2024-09-17T07:17:37.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:37 smithi135 ceph-mon[33238]: 2.1e deep-scrub ok 2024-09-17T07:17:37.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:37 smithi135 ceph-mon[33238]: 6.1e scrub starts 2024-09-17T07:17:37.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:37 smithi135 ceph-mon[33238]: 6.1e scrub ok 2024-09-17T07:17:37.567 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:37 smithi100 ceph-mon[29092]: 2.1e deep-scrub starts 2024-09-17T07:17:37.568 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:37 smithi100 ceph-mon[29092]: 2.1e deep-scrub ok 2024-09-17T07:17:37.568 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:37 smithi100 ceph-mon[29092]: 6.1e scrub starts 2024-09-17T07:17:37.568 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:37 smithi100 ceph-mon[29092]: 6.1e scrub ok 2024-09-17T07:17:37.568 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:37 smithi100 ceph-mon[39462]: 2.1e deep-scrub starts 2024-09-17T07:17:37.568 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:37 smithi100 ceph-mon[39462]: 2.1e deep-scrub ok 2024-09-17T07:17:37.568 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:37 smithi100 ceph-mon[39462]: 6.1e scrub starts 2024-09-17T07:17:37.568 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:37 smithi100 ceph-mon[39462]: 6.1e scrub ok 2024-09-17T07:17:37.608 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:17:37.631 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 79 16 113819 113803 90.0252 68.25 0.0192798 0.0111015 2024-09-17T07:17:37.636 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:37.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:37.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:17:37.737 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:17:37.776 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:37.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:37.777 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:17:37.878 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:17:37.908 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:37.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:37.910 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:17:38.011 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:17:38.051 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:38.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:38.052 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:17:38.153 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:17:38.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:38.189 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:38.189 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:17:38.290 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:17:38.314 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T07:17:38.318 INFO:teuthology.orchestra.run.smithi100.stderr:dumped all 2024-09-17T07:17:38.328 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:38.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:38.330 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:17:38.431 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:17:38.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:38.460 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:38.460 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:17:38.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:38 smithi135 ceph-mon[33238]: 6.1c deep-scrub starts 2024-09-17T07:17:38.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:38 smithi135 ceph-mon[33238]: 6.1c deep-scrub ok 2024-09-17T07:17:38.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:38 smithi135 ceph-mon[33238]: pgmap v696: 148 pgs: 1 active+clean+scrubbing+deep, 36 active+undersized, 28 active+undersized+degraded, 83 active+clean; 6.7 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 677 B/s rd, 46 MiB/s wr, 731 op/s; 27465/329739 objects degraded (8.329%) 2024-09-17T07:17:38.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:38 smithi135 ceph-mon[33238]: 6.14 scrub starts 2024-09-17T07:17:38.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:38 smithi135 ceph-mon[33238]: 6.14 scrub ok 2024-09-17T07:17:38.561 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:17:38.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:38.608 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:38.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:17:38.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:38 smithi100 ceph-mon[29092]: 6.1c deep-scrub starts 2024-09-17T07:17:38.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:38 smithi100 ceph-mon[29092]: 6.1c deep-scrub ok 2024-09-17T07:17:38.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:38 smithi100 ceph-mon[29092]: pgmap v696: 148 pgs: 1 active+clean+scrubbing+deep, 36 active+undersized, 28 active+undersized+degraded, 83 active+clean; 6.7 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 677 B/s rd, 46 MiB/s wr, 731 op/s; 27465/329739 objects degraded (8.329%) 2024-09-17T07:17:38.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:38 smithi100 ceph-mon[29092]: 6.14 scrub starts 2024-09-17T07:17:38.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:38 smithi100 ceph-mon[29092]: 6.14 scrub ok 2024-09-17T07:17:38.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:38 smithi100 ceph-mon[39462]: 6.1c deep-scrub starts 2024-09-17T07:17:38.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:38 smithi100 ceph-mon[39462]: 6.1c deep-scrub ok 2024-09-17T07:17:38.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:38 smithi100 ceph-mon[39462]: pgmap v696: 148 pgs: 1 active+clean+scrubbing+deep, 36 active+undersized, 28 active+undersized+degraded, 83 active+clean; 6.7 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 677 B/s rd, 46 MiB/s wr, 731 op/s; 27465/329739 objects degraded (8.329%) 2024-09-17T07:17:38.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:38 smithi100 ceph-mon[39462]: 6.14 scrub starts 2024-09-17T07:17:38.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:38 smithi100 ceph-mon[39462]: 6.14 scrub ok 2024-09-17T07:17:38.632 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:2024-09-17T07:17:38.632286+0000 min lat: 0.00112621 max lat: 0.332984 avg lat: 0.0111351 2024-09-17T07:17:38.632 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-17T07:17:38.632 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 80 16 114912 114896 89.7538 68.3125 0.0166827 0.0111351 2024-09-17T07:17:38.710 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:17:38.752 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:38.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:38.753 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:17:38.854 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:17:38.880 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:38.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:38.881 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:17:38.982 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:17:39.026 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:39.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:39.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:17:39.128 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:17:39.170 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:39.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:39.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:17:39.272 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:17:39.278 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:17:38 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:17:39.300 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:39.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:39.301 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:17:39.402 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:17:39.428 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:39.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:39.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:17:39.530 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:17:39.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:39 smithi135 ceph-mon[33238]: from='client.25589 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:17:39.569 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:39.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:39.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:17:39.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:39 smithi100 ceph-mon[29092]: from='client.25589 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:17:39.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:39 smithi100 ceph-mon[39462]: from='client.25589 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:17:39.631 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 81 16 115900 115884 89.408 61.75 0.00567954 0.01118 2024-09-17T07:17:39.671 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:17:39.699 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:39.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:39.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:17:39.801 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:17:39.840 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:39.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:39.841 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:17:39.944 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:17:39.972 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:39.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:39.973 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:17:40.074 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:17:40.102 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:40.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:40.103 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:17:40.204 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:17:40.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:40.226 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:40.227 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:17:40.328 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:17:40.361 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:40.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:40.362 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:17:40.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:40 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:17:40.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:40 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:17:40.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:40 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:17:40.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:40 smithi100 ceph-mon[29092]: osdmap e327: 8 total, 7 up, 7 in 2024-09-17T07:17:40.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:40 smithi100 ceph-mon[29092]: pgmap v698: 148 pgs: 1 active+clean+scrubbing+deep, 36 active+undersized, 28 active+undersized+degraded, 83 active+clean; 7.0 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 77 MiB/s wr, 1.23k op/s; 28470/342735 objects degraded (8.307%) 2024-09-17T07:17:40.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:40 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 27465/329739 objects degraded (8.329%), 28 pgs degraded, 64 pgs undersized (PG_DEGRADED) 2024-09-17T07:17:40.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:40 smithi100 ceph-mon[29092]: osdmap e328: 8 total, 7 up, 7 in 2024-09-17T07:17:40.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:40 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:17:40.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:40 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:17:40.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:40 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:17:40.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:40 smithi100 ceph-mon[39462]: osdmap e327: 8 total, 7 up, 7 in 2024-09-17T07:17:40.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:40 smithi100 ceph-mon[39462]: pgmap v698: 148 pgs: 1 active+clean+scrubbing+deep, 36 active+undersized, 28 active+undersized+degraded, 83 active+clean; 7.0 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 77 MiB/s wr, 1.23k op/s; 28470/342735 objects degraded (8.307%) 2024-09-17T07:17:40.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:40 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 27465/329739 objects degraded (8.329%), 28 pgs degraded, 64 pgs undersized (PG_DEGRADED) 2024-09-17T07:17:40.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:40 smithi100 ceph-mon[39462]: osdmap e328: 8 total, 7 up, 7 in 2024-09-17T07:17:40.463 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:17:40.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:40.501 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:40.501 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:17:40.602 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:17:40.632 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 82 16 116783 116767 88.9906 55.1875 0.00535981 0.0112326 2024-09-17T07:17:40.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:40.644 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:40.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:17:40.744 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:17:40.765 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:17:40.765 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:17:40.766 INFO:teuthology.orchestra.run.smithi100.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-17T07:17:40.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:40.786 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:40.786 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:17:40.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:40 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:17:40.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:40 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:17:40.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:40 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:17:40.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:40 smithi135 ceph-mon[33238]: osdmap e327: 8 total, 7 up, 7 in 2024-09-17T07:17:40.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:40 smithi135 ceph-mon[33238]: pgmap v698: 148 pgs: 1 active+clean+scrubbing+deep, 36 active+undersized, 28 active+undersized+degraded, 83 active+clean; 7.0 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 77 MiB/s wr, 1.23k op/s; 28470/342735 objects degraded (8.307%) 2024-09-17T07:17:40.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:40 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 27465/329739 objects degraded (8.329%), 28 pgs degraded, 64 pgs undersized (PG_DEGRADED) 2024-09-17T07:17:40.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:40 smithi135 ceph-mon[33238]: osdmap e328: 8 total, 7 up, 7 in 2024-09-17T07:17:40.795 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:17:40.796 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:17:40.819 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:17:40.819 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:17:40.865 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:17:40.866 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:17:40.887 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:17:40.896 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:17:40.897 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:17:40.920 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:40.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:40.922 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:17:40.925 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:17:40.925 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:17:40.952 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:17:40.953 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:17:40.987 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-17T07:17:40.987 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:17:41.022 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:17:41.052 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:41.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:41.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:17:41.154 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:17:41.184 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:41.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:41.185 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:17:41.286 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:17:41.319 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:41.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:41.320 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:17:41.421 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:17:41.464 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:41.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:41.466 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:17:41.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:41 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/4162955346' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:17:41.567 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:17:41.596 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:41.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:41.597 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:17:41.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:41 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/4162955346' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:17:41.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:41 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/4162955346' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:17:41.632 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 83 16 117819 117803 88.6985 64.75 0.00414 0.0112692 2024-09-17T07:17:41.699 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:17:41.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:41.736 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:41.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:17:41.837 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:17:41.873 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:41.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:41.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:17:41.975 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:17:42.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:42.012 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:42.012 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:17:42.113 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:17:42.141 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:42.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:42.142 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:17:42.243 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:17:42.270 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:42.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:42.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:17:42.372 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:17:42.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:42.418 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:42.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:17:42.519 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:17:42.561 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:42.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:42.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:17:42.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:42 smithi100 ceph-mon[39462]: 4.3 scrub starts 2024-09-17T07:17:42.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:42 smithi100 ceph-mon[39462]: 4.3 scrub ok 2024-09-17T07:17:42.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:42 smithi100 ceph-mon[39462]: pgmap v700: 148 pgs: 1 active+clean+scrubbing+deep, 36 active+undersized, 28 active+undersized+degraded, 83 active+clean; 7.0 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 843 B/s rd, 55 MiB/s wr, 876 op/s; 28809/345672 objects degraded (8.334%) 2024-09-17T07:17:42.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:42 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:17:42.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:42 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:17:42.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:42 smithi100 ceph-mon[29092]: 4.3 scrub starts 2024-09-17T07:17:42.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:42 smithi100 ceph-mon[29092]: 4.3 scrub ok 2024-09-17T07:17:42.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:42 smithi100 ceph-mon[29092]: pgmap v700: 148 pgs: 1 active+clean+scrubbing+deep, 36 active+undersized, 28 active+undersized+degraded, 83 active+clean; 7.0 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 843 B/s rd, 55 MiB/s wr, 876 op/s; 28809/345672 objects degraded (8.334%) 2024-09-17T07:17:42.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:42 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:17:42.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:42 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:17:42.632 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 84 16 118920 118904 88.4617 68.8125 0.00529133 0.0112957 2024-09-17T07:17:42.663 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:17:42.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:42.701 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:42.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:17:42.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:42 smithi135 ceph-mon[33238]: 4.3 scrub starts 2024-09-17T07:17:42.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:42 smithi135 ceph-mon[33238]: 4.3 scrub ok 2024-09-17T07:17:42.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:42 smithi135 ceph-mon[33238]: pgmap v700: 148 pgs: 1 active+clean+scrubbing+deep, 36 active+undersized, 28 active+undersized+degraded, 83 active+clean; 7.0 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 843 B/s rd, 55 MiB/s wr, 876 op/s; 28809/345672 objects degraded (8.334%) 2024-09-17T07:17:42.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:42 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:17:42.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:42 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:17:42.802 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:17:42.839 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:42.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:42.840 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:17:42.915 INFO:teuthology.orchestra.run.smithi100.stdout:{"pg_ready":true,"pg_map":{"version":696,"stamp":"2024-09-17T07:17:37.311711+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7191009558,"num_objects":109913,"num_object_clones":0,"num_object_copies":329739,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27465,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":109913,"num_whiteouts":0,"num_read":1419,"num_read_kb":1265,"num_write":110612,"num_write_kb":7024347,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":1773876,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77311,"ondisk_log_size":77311,"up":380,"acting":380,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":23569272,"kb_used_data":19257904,"kb_used_omap":10,"kb_used_meta":4311285,"kb_avail":632646792,"statfs":{"total":671965249536,"available":647830315008,"internally_reserved":0,"allocated":19720093696,"data_stored":19716377038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11152,"internal_metadata":4414755952},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[9,2,3,7,9,10,11,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":361824256,"num_objects":5521,"num_object_clones":0,"num_object_copies":16563,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1385,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5521,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":5521,"num_write_kb":353344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.555731"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":173,"reported_epoch":324,"state":"active+undersized","last_fresh":"2024-09-17T07:17:29.760529+0000","last_change":"2024-09-17T07:16:17.379311+0000","last_active":"2024-09-17T07:17:29.760529+0000","last_peered":"2024-09-17T07:17:29.760529+0000","last_clean":"2024-09-17T07:16:15.590051+0000","last_became_active":"2024-09-17T07:16:17.379311+0000","last_became_peered":"2024-09-17T07:16:17.379311+0000","last_unstale":"2024-09-17T07:17:29.760529+0000","last_undegraded":"2024-09-17T07:17:29.760529+0000","last_fullsized":"2024-09-17T07:16:17.372051+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:11.978185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:33.644728+0000","last_clean_scrub_stamp":"2024-09-17T07:13:11.978185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0074436570000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":339,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-09-17T07:17:29.761229+0000","last_change":"2024-09-17T07:16:30.401351+0000","last_active":"2024-09-17T07:17:29.761229+0000","last_peered":"2024-09-17T07:17:29.761229+0000","last_clean":"2024-09-17T07:17:29.761229+0000","last_became_active":"2024-09-17T07:02:12.006754+0000","last_became_peered":"2024-09-17T07:02:12.006754+0000","last_unstale":"2024-09-17T07:17:29.761229+0000","last_undegraded":"2024-09-17T07:17:29.761229+0000","last_fullsized":"2024-09-17T07:17:29.761229+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:30.401191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:16:30.401191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:51.746677+0000","scrub_duration":0.0038520239999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"242'27","reported_seq":249,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-09-17T07:17:29.760088+0000","last_change":"2024-09-17T07:17:21.548879+0000","last_active":"2024-09-17T07:17:29.760088+0000","last_peered":"2024-09-17T07:17:29.760088+0000","last_clean":"2024-09-17T07:17:29.760088+0000","last_became_active":"2024-09-17T07:09:24.402042+0000","last_became_peered":"2024-09-17T07:09:24.402042+0000","last_unstale":"2024-09-17T07:17:29.760088+0000","last_undegraded":"2024-09-17T07:17:29.760088+0000","last_fullsized":"2024-09-17T07:17:29.760088+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'27","last_scrub_stamp":"2024-09-17T07:17:21.548761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:17:21.548761+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:22.146018+0000","scrub_duration":0.0067518939999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":330,"reported_epoch":324,"state":"active+undersized","last_fresh":"2024-09-17T07:17:29.768525+0000","last_change":"2024-09-17T07:16:17.380766+0000","last_active":"2024-09-17T07:17:29.768525+0000","last_peered":"2024-09-17T07:17:29.768525+0000","last_clean":"2024-09-17T07:16:15.357790+0000","last_became_active":"2024-09-17T07:16:17.380766+0000","last_became_peered":"2024-09-17T07:16:17.380766+0000","last_unstale":"2024-09-17T07:17:29.768525+0000","last_undegraded":"2024-09-17T07:17:29.768525+0000","last_fullsized":"2024-09-17T07:16:17.364849+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:45.261117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:15:45.261117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001477482,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":179,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-09-17T07:17:29.761317+0000","last_change":"2024-09-17T07:14:57.520400+0000","last_active":"2024-09-17T07:17:29.761317+0000","last_peered":"2024-09-17T07:17:29.761317+0000","last_clean":"2024-09-17T07:17:29.761317+0000","last_became_active":"2024-09-17T07:09:24.402568+0000","last_became_peered":"2024-09-17T07:09:24.402568+0000","last_unstale":"2024-09-17T07:17:29.761317+0000","last_undegraded":"2024-09-17T07:17:29.761317+0000","last_fullsized":"2024-09-17T07:17:29.761317+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:57.520290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:14:57.520290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001539143,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"242'15","reported_seq":368,"reported_epoch":324,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:17:29.768648+0000","last_change":"2024-09-17T07:16:17.370326+0000","last_active":"2024-09-17T07:17:29.768648+0000","last_peered":"2024-09-17T07:17:29.768648+0000","last_clean":"2024-09-17T07:16:15.357813+0000","last_became_active":"2024-09-17T07:16:17.370326+0000","last_became_peered":"2024-09-17T07:16:17.370326+0000","last_unstale":"2024-09-17T07:17:29.768648+0000","last_undegraded":"2024-09-17T07:16:17.364616+0000","last_fullsized":"2024-09-17T07:16:17.364511+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'15","last_scrub_stamp":"2024-09-17T07:15:39.306043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:15:39.306043+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002471029,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":4}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":30,"reported_epoch":324,"state":"active+undersized","last_fresh":"2024-09-17T07:17:29.767906+0000","last_change":"2024-09-17T07:16:17.379569+0000","last_active":"2024-09-17T07:17:29.767906+0000","last_peered":"2024-09-17T07:17:29.767906+0000","last_clean":"2024-09-17T07:02:14.999435+0000","last_became_active":"2024-09-17T07:16:17.379569+0000","last_became_peered":"2024-09-17T07:16:17.379569+0000","last_unstale":"2024-09-17T07:17:29.767906+0000","last_undegraded":"2024-09-17T07:17:29.767906+0000","last_fullsized":"2024-09-17T07:16:17.369924+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:53.881813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:15:53.881813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":329,"reported_epoch":324,"state":"active+undersized","last_fresh":"2024-09-17T07:17:29.768222+0000","last_change":"2024-09-17T07:16:17.374605+0000","last_active":"2024-09-17T07:17:29.768222+0000","last_peered":"2024-09-17T07:17:29.768222+0000","last_clean":"2024-09-17T07:16:15.357982+0000","last_became_active":"2024-09-17T07:16:17.374605+0000","last_became_peered":"2024-09-17T07:16:17.374605+0000","last_unstale":"2024-09-17T07:17:29.768222+0000","last_undegraded":"2024-09-17T07:17:29.768222+0000","last_fullsized":"2024-09-17T07:16:17.366300+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:41.707969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:14:41.707969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.014931757,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":326,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-09-17T07:17:29.762373+0000","last_change":"2024-09-17T07:15:46.312474+0000","last_active":"2024-09-17T07:17:29.762373+0000","last_peered":"2024-09-17T07:17:29.762373+0000","last_clean":"2024-09-17T07:17:29.762373+0000","last_became_active":"2024-09-17T07:02:12.011129+0000","last_became_peered":"2024-09-17T07:02:12.011129+0000","last_unstale":"2024-09-17T07:17:29.762373+0000","last_undegraded":"2024-09-17T07:17:29.762373+0000","last_fullsized":"2024-09-17T07:17:29.762373+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:46.312371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:46.312371+0000","last_clean_scrub_stamp":"2024-09-17T07:15:46.312371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015176510000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"242'7","reported_seq":361,"reported_epoch":324,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:17:29.769010+0000","last_change":"2024-09-17T07:16:17.373486+0000","last_active":"2024-09-17T07:17:29.769010+0000","last_peered":"2024-09-17T07:17:29.769010+0000","last_clean":"2024-09-17T07:16:15.358036+0000","last_became_active":"2024-09-17T07:16:17.373486+0000","last_became_peered":"2024-09-17T07:16:17.373486+0000","last_unstale":"2024-09-17T07:17:29.769010+0000","last_undegraded":"2024-09-17T07:16:17.366228+0000","last_fullsized":"2024-09-17T07:16:17.365631+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'7","last_scrub_stamp":"2024-09-17T07:14:42.653705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:14:42.653705+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0050149230000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[{"shards":"2,4","objects":2}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":328,"reported_epoch":324,"state":"active+undersized","last_fresh":"2024-09-17T07:17:29.760477+0000","last_change":"2024-09-17T07:16:17.382323+0000","last_active":"2024-09-17T07:17:29.760477+0000","last_peered":"2024-09-17T07:17:29.760477+0000","last_clean":"2024-09-17T07:16:15.589989+0000","last_became_active":"2024-09-17T07:16:17.382323+0000","last_became_peered":"2024-09-17T07:16:17.382323+0000","last_unstale":"2024-09-17T07:17:29.760477+0000","last_undegraded":"2024-09-17T07:17:29.760477+0000","last_fullsized":"2024-09-17T07:16:17.371260+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:37.997296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:14:37.997296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045486420000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":347,"reported_epoch":324,"state":"active+undersized","last_fresh":"2024-09-17T07:17:29.768026+0000","last_change":"2024-09-17T07:16:17.374238+0000","last_active":"2024-09-17T07:17:29.768026+0000","last_peered":"2024-09-17T07:17:29.768026+0000","last_clean":"2024-09-17T07:16:16.170346+0000","last_became_active":"2024-09-17T07:16:17.374238+0000","last_became_peered":"2024-09-17T07:16:17.374238+0000","last_unstale":"2024-09-17T07:17:29.768026+0000","last_undegraded":"2024-09-17T07:17:29.768026+0000","last_fullsized":"2024-09-17T07:16:17.367446+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:52.168181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:14:52.168181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0056501399999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":326,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-09-17T07:17:29.766058+0000","last_change":"2024-09-17T07:14:44.076126+0000","last_active":"2024-09-17T07:17:29.766058+0000","last_peered":"2024-09-17T07:17:29.766058+0000","last_clean":"2024-09-17T07:17:29.766058+0000","last_became_active":"2024-09-17T07:02:12.012234+0000","last_became_peered":"2024-09-17T07:02:12.012234+0000","last_unstale":"2024-09-17T07:17:29.766058+0000","last_undegraded":"2024-09-17T07:17:29.766058+0000","last_fullsized":"2024-09-17T07:17:29.766058+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:44.076012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:45.029247+0000","last_clean_scrub_stamp":"2024-09-17T07:14:44.076012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.015917521,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"312'19","reported_seq":385,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-09-17T07:17:29.768647+0000","last_change":"2024-09-17T07:16:44.293061+0000","last_active":"2024-09-17T07:17:29.768647+0000","last_peered":"2024-09-17T07:17:29.768647+0000","last_clean":"2024-09-17T07:17:29.768647+0000","last_became_active":"2024-09-17T07:09:24.397954+0000","last_became_peered":"2024-09-17T07:09:24.397954+0000","last_unstale":"2024-09-17T07:17:29.768647+0000","last_undegraded":"2024-09-17T07:17:29.768647+0000","last_fullsized":"2024-09-17T07:17:29.768647+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"312'19","last_scrub_stamp":"2024-09-17T07:16:44.292911+0000","last_deep_scrub":"312'19","last_deep_scrub_stamp":"2024-09-17T07:16:44.292911+0000","last_clean_scrub_stamp":"2024-09-17T07:16:44.292911+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:49.152333+0000","scrub_duration":0.013103178,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":344,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-09-17T07:17:29.766041+0000","last_change":"2024-09-17T07:16:45.164526+0000","last_active":"2024-09-17T07:17:29.766041+0000","last_peered":"2024-09-17T07:17:29.766041+0000","last_clean":"2024-09-17T07:17:29.766041+0000","last_became_active":"2024-09-17T07:02:16.023774+0000","last_became_peered":"2024-09-17T07:02:16.023774+0000","last_unstale":"2024-09-17T07:17:29.766041+0000","last_undegraded":"2024-09-17T07:17:29.766041+0000","last_fullsized":"2024-09-17T07:17:29.766041+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:45.164367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:28.096266+0000","last_clean_scrub_stamp":"2024-09-17T07:16:45.164367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:10.271781+0000","scrub_duration":0.0026790630000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":328,"reported_epoch":324,"state":"active+undersized","last_fresh":"2024-09-17T07:17:29.759900+0000","last_change":"2024-09-17T07:16:17.378534+0000","last_active":"2024-09-17T07:17:29.759900+0000","last_peered":"2024-09-17T07:17:29.759900+0000","last_clean":"2024-09-17T07:16:15.494529+0000","last_became_active":"2024-09-17T07:16:17.378534+0000","last_became_peered":"2024-09-17T07:16:17.378534+0000","last_unstale":"2024-09-17T07:17:29.759900+0000","last_undegraded":"2024-09-17T07:17:29.759900+0000","last_fullsized":"2024-09-17T07:16:17.365363+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:45.483622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:15:45.483622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001342359,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3","version":"325'7188","reported_seq":7283,"reported_epoch":325,"state":"active+clean+scrubbing+deep","last_fresh":"2024-09-17T07:17:33.739905+0000","last_change":"2024-09-17T07:17:30.485220+0000","last_active":"2024-09-17T07:17:33.739905+0000","last_peered":"2024-09-17T07:17:33.739905+0000","last_clean":"2024-09-17T07:17:33.739905+0000","last_became_active":"2024-09-17T07:16:05.057730+0000","last_became_peered":"2024-09-17T07:16:05.057730+0000","last_unstale":"2024-09-17T07:17:33.739905+0000","last_undegraded":"2024-09-17T07:17:33.739905+0000","last_fullsized":"2024-09-17T07:17:33.739905+0000","mapping_epoch":297,"log_start":"307'2200","ondisk_log_start":"307'2200","created":297,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":661,"log_size":4988,"log_dups_size":0,"ondisk_log_size":4988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 3s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":442236928,"num_objects":6748,"num_object_clones":0,"num_object_copies":20244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6748,"num_write_kb":431872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"242'39","reported_seq":400,"reported_epoch":324,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:17:29.770773+0000","last_change":"2024-09-17T07:16:17.374805+0000","last_active":"2024-09-17T07:17:29.770773+0000","last_peered":"2024-09-17T07:17:29.770773+0000","last_clean":"2024-09-17T07:16:15.358504+0000","last_became_active":"2024-09-17T07:16:17.374805+0000","last_became_peered":"2024-09-17T07:16:17.374805+0000","last_unstale":"2024-09-17T07:17:29.770773+0000","last_undegraded":"2024-09-17T07:16:17.367444+0000","last_fullsized":"2024-09-17T07:16:17.367331+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'39","last_scrub_stamp":"2024-09-17T07:15:46.226554+0000","last_deep_scrub":"95'25","last_deep_scrub_stamp":"2024-09-17T07:04:09.450039+0000","last_clean_scrub_stamp":"2024-09-17T07:15:46.226554+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0027061799999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[{"shards":"2,4","objects":8}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":339,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-09-17T07:17:29.760975+0000","last_change":"2024-09-17T07:16:33.658452+0000","last_active":"2024-09-17T07:17:29.760975+0000","last_peered":"2024-09-17T07:17:29.760975+0000","last_clean":"2024-09-17T07:17:29.760975+0000","last_became_active":"2024-09-17T07:02:12.011258+0000","last_became_peered":"2024-09-17T07:02:12.011258+0000","last_unstale":"2024-09-17T07:17:29.760975+0000","last_undegraded":"2024-09-17T07:17:29.760975+0000","last_fullsized":"2024-09-17T07:17:29.760975+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:33.658293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:16:33.658293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:46.850647+0000","scrub_duration":0.044680275999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":30,"reported_epoch":324,"state":"active+undersized","last_fresh":"2024-09-17T07:17:29.774634+0000","last_change":"2024-09-17T07:16:17.372955+0000","last_active":"2024-09-17T07:17:29.774634+0000","last_peered":"2024-09-17T07:17:29.774634+0000","last_clean":"2024-09-17T07:02:14.999435+0000","last_became_active":"2024-09-17T07:16:17.372955+0000","last_became_peered":"2024-09-17T07:16:17.372955+0000","last_unstale":"2024-09-17T07:17:29.774634+0000","last_undegraded":"2024-09-17T07:17:29.774634+0000","last_fullsized":"2024-09-17T07:16:17.365245+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:46.502641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:13:46.502641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":181,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-09-17T07:17:29.769519+0000","last_change":"2024-09-17T07:16:34.161259+0000","last_active":"2024-09-17T07:17:29.769519+0000","last_peered":"2024-09-17T07:17:29.769519+0000","last_clean":"2024-09-17T07:17:29.769519+0000","last_became_active":"2024-09-17T07:09:24.396334+0000","last_became_peered":"2024-09-17T07:09:24.396334+0000","last_unstale":"2024-09-17T07:17:29.769519+0000","last_undegraded":"2024-09-17T07:17:29.769519+0000","last_fullsized":"2024-09-17T07:17:29.769519+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:34.161180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:42.984088+0000","last_clean_scrub_stamp":"2024-09-17T07:16:34.161180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:36.434870+0000","scrub_duration":0.001181355,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2","version":"325'6742","reported_seq":6772,"reported_epoch":325,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:17:33.596849+0000","last_change":"2024-09-17T07:16:18.633028+0000","last_active":"2024-09-17T07:17:33.596849+0000","last_peered":"2024-09-17T07:17:33.596849+0000","last_clean":"2024-09-17T07:16:04.025356+0000","last_became_active":"2024-09-17T07:16:17.372517+0000","last_became_peered":"2024-09-17T07:16:17.372517+0000","last_unstale":"2024-09-17T07:17:33.596849+0000","last_undegraded":"2024-09-17T07:16:18.629077+0000","last_fullsized":"2024-09-17T07:16:17.365507+0000","mapping_epoch":303,"log_start":"307'2100","ondisk_log_start":"307'2100","created":297,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":4642,"log_dups_size":0,"ondisk_log_size":4642,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":441843712,"num_objects":6742,"num_object_clones":0,"num_object_copies":20226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6742,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6742,"num_write_kb":431488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[{"shards":"0,4","objects":6742}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"242'25","reported_seq":399,"reported_epoch":324,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:17:29.774835+0000","last_change":"2024-09-17T07:16:17.381025+0000","last_active":"2024-09-17T07:17:29.774835+0000","last_peered":"2024-09-17T07:17:29.774835+0000","last_clean":"2024-09-17T07:16:15.358419+0000","last_became_active":"2024-09-17T07:16:17.381025+0000","last_became_peered":"2024-09-17T07:16:17.381025+0000","last_unstale":"2024-09-17T07:17:29.774835+0000","last_undegraded":"2024-09-17T07:16:17.367122+0000","last_fullsized":"2024-09-17T07:16:17.365635+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'25","last_scrub_stamp":"2024-09-17T07:15:02.532371+0000","last_deep_scrub":"176'15","last_deep_scrub_stamp":"2024-09-17T07:08:19.633791+0000","last_clean_scrub_stamp":"2024-09-17T07:15:02.532371+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012712606,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":202,"reported_epoch":324,"state":"active+undersized","last_fresh":"2024-09-17T07:17:29.760729+0000","last_change":"2024-09-17T07:16:17.379662+0000","last_active":"2024-09-17T07:17:29.760729+0000","last_peered":"2024-09-17T07:17:29.760729+0000","last_clean":"2024-09-17T07:09:22.372452+0000","last_became_active":"2024-09-17T07:16:17.379662+0000","last_became_peered":"2024-09-17T07:16:17.379662+0000","last_unstale":"2024-09-17T07:17:29.760729+0000","last_undegraded":"2024-09-17T07:17:29.760729+0000","last_fullsized":"2024-09-17T07:16:17.370523+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:59.697560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:14:59.697560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0065215869999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.d","version":"98'9","reported_seq":466,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-09-17T07:17:30.701155+0000","last_change":"2024-09-17T07:14:46.126177+0000","last_active":"2024-09-17T07:17:30.701155+0000","last_peered":"2024-09-17T07:17:30.701155+0000","last_clean":"2024-09-17T07:17:30.701155+0000","last_became_active":"2024-09-17T07:04:15.110797+0000","last_became_peered":"2024-09-17T07:04:15.110797+0000","last_unstale":"2024-09-17T07:17:30.701155+0000","last_undegraded":"2024-09-17T07:17:30.701155+0000","last_fullsized":"2024-09-17T07:17:30.701155+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-09-17T07:14:46.126045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:14:46.126045+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.014185467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":339,"reported_epoch":324,"state":"active+undersized","last_fresh":"2024-09-17T07:17:29.777023+0000","last_change":"2024-09-17T07:16:17.373630+0000","last_active":"2024-09-17T07:17:29.777023+0000","last_peered":"2024-09-17T07:17:29.777023+0000","last_clean":"2024-09-17T07:16:15.358526+0000","last_became_active":"2024-09-17T07:16:17.373630+0000","last_became_peered":"2024-09-17T07:16:17.373630+0000","last_unstale":"2024-09-17T07:17:29.777023+0000","last_undegraded":"2024-09-17T07:17:29.777023+0000","last_fullsized":"2024-09-17T07:16:17.369047+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:50.554591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:14:50.554591+0000","last_clean_scrub_stamp":"2024-09-17T07:14:50.554591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0067125969999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1","version":"325'6979","reported_seq":7149,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-17T07:17:33.577855+0000","last_change":"2024-09-17T07:17:22.401562+0000","last_active":"2024-09-17T07:17:33.577855+0000","last_peered":"2024-09-17T07:17:33.577855+0000","last_clean":"2024-09-17T07:17:33.577855+0000","last_became_active":"2024-09-17T07:16:05.050872+0000","last_became_peered":"2024-09-17T07:16:05.050872+0000","last_unstale":"2024-09-17T07:17:33.577855+0000","last_undegraded":"2024-09-17T07:17:33.577855+0000","last_fullsized":"2024-09-17T07:17:33.577855+0000","mapping_epoch":297,"log_start":"307'2400","ondisk_log_start":"307'2400","created":297,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'6219","last_scrub_stamp":"2024-09-17T07:17:22.401249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:17:22.401249+0000","objects_scrubbed":6144,"log_size":4579,"log_dups_size":0,"ondisk_log_size":4579,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:48.803491+0000","scrub_duration":2.0885702689999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":457375744,"num_objects":6979,"num_object_clones":0,"num_object_copies":20937,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6979,"num_write_kb":446656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"242'27","reported_seq":324,"reported_epoch":324,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:17:29.769698+0000","last_change":"2024-09-17T07:16:17.382238+0000","last_active":"2024-09-17T07:17:29.769698+0000","last_peered":"2024-09-17T07:17:29.769698+0000","last_clean":"2024-09-17T07:12:20.775697+0000","last_became_active":"2024-09-17T07:16:17.382238+0000","last_became_peered":"2024-09-17T07:16:17.382238+0000","last_unstale":"2024-09-17T07:17:29.769698+0000","last_undegraded":"2024-09-17T07:16:17.365468+0000","last_fullsized":"2024-09-17T07:16:17.365334+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'27","last_scrub_stamp":"2024-09-17T07:14:45.831734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:14:45.831734+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021853620000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[{"shards":"4,7","objects":7}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":203,"reported_epoch":324,"state":"active+undersized","last_fresh":"2024-09-17T07:17:29.763133+0000","last_change":"2024-09-17T07:16:17.378599+0000","last_active":"2024-09-17T07:17:29.763133+0000","last_peered":"2024-09-17T07:17:29.763133+0000","last_clean":"2024-09-17T07:09:22.371958+0000","last_became_active":"2024-09-17T07:16:17.378599+0000","last_became_peered":"2024-09-17T07:16:17.378599+0000","last_unstale":"2024-09-17T07:17:29.763133+0000","last_undegraded":"2024-09-17T07:17:29.763133+0000","last_fullsized":"2024-09-17T07:16:17.373236+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:44.897611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:15:44.897611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034116369999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.e","version":"75'8","reported_seq":527,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-09-17T07:17:30.702857+0000","last_change":"2024-09-17T07:16:49.331611+0000","last_active":"2024-09-17T07:17:30.702857+0000","last_peered":"2024-09-17T07:17:30.702857+0000","last_clean":"2024-09-17T07:17:30.702857+0000","last_became_active":"2024-09-17T07:02:16.021762+0000","last_became_peered":"2024-09-17T07:02:16.021762+0000","last_unstale":"2024-09-17T07:17:30.702857+0000","last_undegraded":"2024-09-17T07:17:30.702857+0000","last_fullsized":"2024-09-17T07:17:30.702857+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:16:49.331407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:16:49.331407+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:57.973202+0000","scrub_duration":0.0033687029999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":337,"reported_epoch":324,"state":"active+undersized","last_fresh":"2024-09-17T07:17:29.766967+0000","last_change":"2024-09-17T07:16:17.374897+0000","last_active":"2024-09-17T07:17:29.766967+0000","last_peered":"2024-09-17T07:17:29.766967+0000","last_clean":"2024-09-17T07:16:16.170588+0000","last_became_active":"2024-09-17T07:16:17.374897+0000","last_became_peered":"2024-09-17T07:16:17.374897+0000","last_unstale":"2024-09-17T07:17:29.766967+0000","last_undegraded":"2024-09-17T07:17:29.766967+0000","last_fullsized":"2024-09-17T07:16:17.368895+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:56.146098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:14:56.146098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0054479960000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.0","version":"325'6849","reported_seq":6891,"reported_epoch":325,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:17:33.466531+0000","last_change":"2024-09-17T07:16:18.637531+0000","last_active":"2024-09-17T07:17:33.466531+0000","last_peered":"2024-09-17T07:17:33.466531+0000","last_clean":"2024-09-17T07:16:14.245624+0000","last_became_active":"2024-09-17T07:16:17.372069+0000","last_became_peered":"2024-09-17T07:16:17.372069+0000","last_unstale":"2024-09-17T07:17:33.466531+0000","last_undegraded":"2024-09-17T07:16:18.630601+0000","last_fullsized":"2024-09-17T07:16:17.362738+0000","mapping_epoch":303,"log_start":"307'2000","ondisk_log_start":"307'2000","created":297,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":4849,"log_dups_size":0,"ondisk_log_size":4849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":448856064,"num_objects":6849,"num_object_clones":0,"num_object_copies":20547,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6849,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6849,"num_write_kb":438336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[{"shards":"4,6","objects":6849}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"242'19","reported_seq":374,"reported_epoch":324,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:17:29.778094+0000","last_change":"2024-09-17T07:16:17.382184+0000","last_active":"2024-09-17T07:17:29.778094+0000","last_peered":"2024-09-17T07:17:29.778094+0000","last_clean":"2024-09-17T07:16:15.358681+0000","last_became_active":"2024-09-17T07:16:17.382184+0000","last_became_peered":"2024-09-17T07:16:17.382184+0000","last_unstale":"2024-09-17T07:17:29.778094+0000","last_undegraded":"2024-09-17T07:16:17.369179+0000","last_fullsized":"2024-09-17T07:16:17.369087+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'19","last_scrub_stamp":"2024-09-17T07:14:55.489874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:14:55.489874+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045924440000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":334,"reported_epoch":324,"state":"active+undersized","last_fresh":"2024-09-17T07:17:29.760416+0000","last_change":"2024-09-17T07:16:17.375239+0000","last_active":"2024-09-17T07:17:29.760416+0000","last_peered":"2024-09-17T07:17:29.760416+0000","last_clean":"2024-09-17T07:16:15.590006+0000","last_became_active":"2024-09-17T07:16:17.375239+0000","last_became_peered":"2024-09-17T07:16:17.375239+0000","last_unstale":"2024-09-17T07:17:29.760416+0000","last_undegraded":"2024-09-17T07:17:29.760416+0000","last_fullsized":"2024-09-17T07:16:17.372272+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:21.858127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:13:21.858127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033279709999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":355,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-09-17T07:17:29.766939+0000","last_change":"2024-09-17T07:17:18.435029+0000","last_active":"2024-09-17T07:17:29.766939+0000","last_peered":"2024-09-17T07:17:29.766939+0000","last_clean":"2024-09-17T07:17:29.766939+0000","last_became_active":"2024-09-17T07:02:16.020224+0000","last_became_peered":"2024-09-17T07:02:16.020224+0000","last_unstale":"2024-09-17T07:17:29.766939+0000","last_undegraded":"2024-09-17T07:17:29.766939+0000","last_fullsized":"2024-09-17T07:17:29.766939+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:17:18.434919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:17:18.434919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:23.948987+0000","scrub_duration":0.0044010059999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":198,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-09-17T07:17:29.772671+0000","last_change":"2024-09-17T07:16:37.587187+0000","last_active":"2024-09-17T07:17:29.772671+0000","last_peered":"2024-09-17T07:17:29.772671+0000","last_clean":"2024-09-17T07:17:29.772671+0000","last_became_active":"2024-09-17T07:09:24.408109+0000","last_became_peered":"2024-09-17T07:09:24.408109+0000","last_unstale":"2024-09-17T07:17:29.772671+0000","last_undegraded":"2024-09-17T07:17:29.772671+0000","last_fullsized":"2024-09-17T07:17:29.772671+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:37.587106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:13:22.187127+0000","last_clean_scrub_stamp":"2024-09-17T07:16:37.587106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:59.564692+0000","scrub_duration":0.001128609,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.f","version":"325'6956","reported_seq":6992,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-17T07:17:33.414230+0000","last_change":"2024-09-17T07:16:05.052296+0000","last_active":"2024-09-17T07:17:33.414230+0000","last_peered":"2024-09-17T07:17:33.414230+0000","last_clean":"2024-09-17T07:17:33.414230+0000","last_became_active":"2024-09-17T07:16:05.052037+0000","last_became_peered":"2024-09-17T07:16:05.052037+0000","last_unstale":"2024-09-17T07:17:33.414230+0000","last_undegraded":"2024-09-17T07:17:33.414230+0000","last_fullsized":"2024-09-17T07:17:33.414230+0000","mapping_epoch":297,"log_start":"307'2100","ondisk_log_start":"307'2100","created":297,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":4856,"log_dups_size":0,"ondisk_log_size":4856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":455868416,"num_objects":6956,"num_object_clones":0,"num_object_copies":20868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6956,"num_write_kb":445184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"242'28","reported_seq":380,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-17T07:17:31.148934+0000","last_change":"2024-09-17T07:17:31.149145+0000","last_active":"2024-09-17T07:17:31.148934+0000","last_peered":"2024-09-17T07:17:31.148934+0000","last_clean":"2024-09-17T07:17:31.148934+0000","last_became_active":"2024-09-17T07:02:14.007810+0000","last_became_peered":"2024-09-17T07:02:14.007810+0000","last_unstale":"2024-09-17T07:17:31.148934+0000","last_undegraded":"2024-09-17T07:17:31.148934+0000","last_fullsized":"2024-09-17T07:17:31.148934+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'28","last_scrub_stamp":"2024-09-17T07:17:31.148909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:17:31.148909+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:44.025720+0000","scrub_duration":0.013898221000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":381,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-09-17T07:17:29.773950+0000","last_change":"2024-09-17T07:17:19.376369+0000","last_active":"2024-09-17T07:17:29.773950+0000","last_peered":"2024-09-17T07:17:29.773950+0000","last_clean":"2024-09-17T07:17:29.773950+0000","last_became_active":"2024-09-17T07:02:12.011766+0000","last_became_peered":"2024-09-17T07:02:12.011766+0000","last_unstale":"2024-09-17T07:17:29.773950+0000","last_undegraded":"2024-09-17T07:17:29.773950+0000","last_fullsized":"2024-09-17T07:17:29.773950+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:17:19.375818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:09:23.181028+0000","last_clean_scrub_stamp":"2024-09-17T07:17:19.375818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:43.381176+0000","scrub_duration":0.019104131999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"304'10","reported_seq":66,"reported_epoch":324,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:17:30.703470+0000","last_change":"2024-09-17T07:16:17.371678+0000","last_active":"2024-09-17T07:17:30.703470+0000","last_peered":"2024-09-17T07:17:30.703470+0000","last_clean":"2024-09-17T07:02:19.145975+0000","last_became_active":"2024-09-17T07:16:17.371678+0000","last_became_peered":"2024-09-17T07:16:17.371678+0000","last_unstale":"2024-09-17T07:17:30.703470+0000","last_undegraded":"2024-09-17T07:16:17.366242+0000","last_fullsized":"2024-09-17T07:16:17.365352+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'9","last_scrub_stamp":"2024-09-17T07:15:55.813882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:15:55.813882+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[{"shards":"0,4","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":332,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-09-17T07:17:29.760639+0000","last_change":"2024-09-17T07:15:52.378479+0000","last_active":"2024-09-17T07:17:29.760639+0000","last_peered":"2024-09-17T07:17:29.760639+0000","last_clean":"2024-09-17T07:17:29.760639+0000","last_became_active":"2024-09-17T07:02:18.364981+0000","last_became_peered":"2024-09-17T07:02:18.364981+0000","last_unstale":"2024-09-17T07:17:29.760639+0000","last_undegraded":"2024-09-17T07:17:29.760639+0000","last_fullsized":"2024-09-17T07:17:29.760639+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:52.378384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:10:33.727472+0000","last_clean_scrub_stamp":"2024-09-17T07:15:52.378384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001396736,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"325'6810","reported_seq":6847,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-17T07:17:33.566244+0000","last_change":"2024-09-17T07:16:05.054368+0000","last_active":"2024-09-17T07:17:33.566244+0000","last_peered":"2024-09-17T07:17:33.566244+0000","last_clean":"2024-09-17T07:17:33.566244+0000","last_became_active":"2024-09-17T07:16:05.054109+0000","last_became_peered":"2024-09-17T07:16:05.054109+0000","last_unstale":"2024-09-17T07:17:33.566244+0000","last_undegraded":"2024-09-17T07:17:33.566244+0000","last_fullsized":"2024-09-17T07:17:33.566244+0000","mapping_epoch":297,"log_start":"307'2200","ondisk_log_start":"307'2200","created":297,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":4610,"log_dups_size":0,"ondisk_log_size":4610,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":446300160,"num_objects":6810,"num_object_clones":0,"num_object_copies":20430,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6810,"num_write_kb":435840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"242'30","reported_seq":253,"reported_epoch":324,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:17:29.760283+0000","last_change":"2024-09-17T07:16:17.378001+0000","last_active":"2024-09-17T07:17:29.760283+0000","last_peered":"2024-09-17T07:17:29.760283+0000","last_clean":"2024-09-17T07:16:15.589841+0000","last_became_active":"2024-09-17T07:16:17.378001+0000","last_became_peered":"2024-09-17T07:16:17.378001+0000","last_unstale":"2024-09-17T07:17:29.760283+0000","last_undegraded":"2024-09-17T07:16:17.372487+0000","last_fullsized":"2024-09-17T07:16:17.372376+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'30","last_scrub_stamp":"2024-09-17T07:13:28.997563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:13:28.997563+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0061961159999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[{"shards":"0,7","objects":8}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":30,"reported_epoch":324,"state":"active+undersized","last_fresh":"2024-09-17T07:17:29.773457+0000","last_change":"2024-09-17T07:16:17.376547+0000","last_active":"2024-09-17T07:17:29.773457+0000","last_peered":"2024-09-17T07:17:29.773457+0000","last_clean":"2024-09-17T07:02:10.991363+0000","last_became_active":"2024-09-17T07:16:17.376547+0000","last_became_peered":"2024-09-17T07:16:17.376547+0000","last_unstale":"2024-09-17T07:17:29.773457+0000","last_undegraded":"2024-09-17T07:17:29.773457+0000","last_fullsized":"2024-09-17T07:16:17.364886+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:56.848689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:56.848689+0000","last_clean_scrub_stamp":"2024-09-17T07:15:56.848689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":345,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-09-17T07:17:29.767971+0000","last_change":"2024-09-17T07:16:38.184957+0000","last_active":"2024-09-17T07:17:29.767971+0000","last_peered":"2024-09-17T07:17:29.767971+0000","last_clean":"2024-09-17T07:17:29.767971+0000","last_became_active":"2024-09-17T07:09:24.393375+0000","last_became_peered":"2024-09-17T07:09:24.393375+0000","last_unstale":"2024-09-17T07:17:29.767971+0000","last_undegraded":"2024-09-17T07:17:29.767971+0000","last_fullsized":"2024-09-17T07:17:29.767971+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:38.184818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:16:38.184818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:55.885718+0000","scrub_duration":0.0076357700000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":343,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-09-17T07:17:29.768560+0000","last_change":"2024-09-17T07:16:36.162077+0000","last_active":"2024-09-17T07:17:29.768560+0000","last_peered":"2024-09-17T07:17:29.768560+0000","last_clean":"2024-09-17T07:17:29.768560+0000","last_became_active":"2024-09-17T07:09:24.398659+0000","last_became_peered":"2024-09-17T07:09:24.398659+0000","last_unstale":"2024-09-17T07:17:29.768560+0000","last_undegraded":"2024-09-17T07:17:29.768560+0000","last_fullsized":"2024-09-17T07:17:29.768560+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:36.161622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:16:36.161622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:03.772691+0000","scrub_duration":0.0062812850000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.d","version":"325'6875","reported_seq":6911,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-17T07:17:33.420328+0000","last_change":"2024-09-17T07:16:05.052409+0000","last_active":"2024-09-17T07:17:33.420328+0000","last_peered":"2024-09-17T07:17:33.420328+0000","last_clean":"2024-09-17T07:17:33.420328+0000","last_became_active":"2024-09-17T07:16:05.052153+0000","last_became_peered":"2024-09-17T07:16:05.052153+0000","last_unstale":"2024-09-17T07:17:33.420328+0000","last_undegraded":"2024-09-17T07:17:33.420328+0000","last_fullsized":"2024-09-17T07:17:33.420328+0000","mapping_epoch":297,"log_start":"307'2100","ondisk_log_start":"307'2100","created":297,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":4775,"log_dups_size":0,"ondisk_log_size":4775,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":450560000,"num_objects":6875,"num_object_clones":0,"num_object_copies":20625,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6875,"num_write_kb":440000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"242'29","reported_seq":378,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-09-17T07:17:29.760642+0000","last_change":"2024-09-17T07:15:38.507145+0000","last_active":"2024-09-17T07:17:29.760642+0000","last_peered":"2024-09-17T07:17:29.760642+0000","last_clean":"2024-09-17T07:17:29.760642+0000","last_became_active":"2024-09-17T07:02:14.027389+0000","last_became_peered":"2024-09-17T07:02:14.027389+0000","last_unstale":"2024-09-17T07:17:29.760642+0000","last_undegraded":"2024-09-17T07:17:29.760642+0000","last_fullsized":"2024-09-17T07:17:29.760642+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'29","last_scrub_stamp":"2024-09-17T07:15:38.507042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:15:38.507042+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0024959420000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":339,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-09-17T07:17:29.762443+0000","last_change":"2024-09-17T07:16:47.631557+0000","last_active":"2024-09-17T07:17:29.762443+0000","last_peered":"2024-09-17T07:17:29.762443+0000","last_clean":"2024-09-17T07:17:29.762443+0000","last_became_active":"2024-09-17T07:02:12.011917+0000","last_became_peered":"2024-09-17T07:02:12.011917+0000","last_unstale":"2024-09-17T07:17:29.762443+0000","last_undegraded":"2024-09-17T07:17:29.762443+0000","last_fullsized":"2024-09-17T07:17:29.762443+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:47.631387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:47.631387+0000","last_clean_scrub_stamp":"2024-09-17T07:16:47.631387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:12.824261+0000","scrub_duration":0.006082372,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":345,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-09-17T07:17:29.773252+0000","last_change":"2024-09-17T07:15:56.525463+0000","last_active":"2024-09-17T07:17:29.773252+0000","last_peered":"2024-09-17T07:17:29.773252+0000","last_clean":"2024-09-17T07:17:29.773252+0000","last_became_active":"2024-09-17T07:02:16.016199+0000","last_became_peered":"2024-09-17T07:02:16.016199+0000","last_unstale":"2024-09-17T07:17:29.773252+0000","last_undegraded":"2024-09-17T07:17:29.773252+0000","last_fullsized":"2024-09-17T07:17:29.773252+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:56.525361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:11:17.583712+0000","last_clean_scrub_stamp":"2024-09-17T07:15:56.525361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013436679999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":30,"reported_epoch":324,"state":"active+undersized","last_fresh":"2024-09-17T07:17:29.772574+0000","last_change":"2024-09-17T07:16:17.368186+0000","last_active":"2024-09-17T07:17:29.772574+0000","last_peered":"2024-09-17T07:17:29.772574+0000","last_clean":"2024-09-17T07:02:17.002904+0000","last_became_active":"2024-09-17T07:16:17.368186+0000","last_became_peered":"2024-09-17T07:16:17.368186+0000","last_unstale":"2024-09-17T07:17:29.772574+0000","last_undegraded":"2024-09-17T07:17:29.772574+0000","last_fullsized":"2024-09-17T07:16:17.362908+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:50.514525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:13:50.514525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.c","version":"325'6904","reported_seq":6941,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-17T07:17:33.910887+0000","last_change":"2024-09-17T07:16:05.057855+0000","last_active":"2024-09-17T07:17:33.910887+0000","last_peered":"2024-09-17T07:17:33.910887+0000","last_clean":"2024-09-17T07:17:33.910887+0000","last_became_active":"2024-09-17T07:16:05.057588+0000","last_became_peered":"2024-09-17T07:16:05.057588+0000","last_unstale":"2024-09-17T07:17:33.910887+0000","last_undegraded":"2024-09-17T07:17:33.910887+0000","last_fullsized":"2024-09-17T07:17:33.910887+0000","mapping_epoch":297,"log_start":"307'2300","ondisk_log_start":"307'2300","created":297,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":4604,"log_dups_size":0,"ondisk_log_size":4604,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452460544,"num_objects":6904,"num_object_clones":0,"num_object_copies":20712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6904,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6904,"num_write_kb":441856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"242'25","reported_seq":291,"reported_epoch":324,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:17:29.767673+0000","last_change":"2024-09-17T07:16:17.376630+0000","last_active":"2024-09-17T07:17:29.767673+0000","last_peered":"2024-09-17T07:17:29.767673+0000","last_clean":"2024-09-17T07:12:20.656798+0000","last_became_active":"2024-09-17T07:16:17.376630+0000","last_became_peered":"2024-09-17T07:16:17.376630+0000","last_unstale":"2024-09-17T07:17:29.767673+0000","last_undegraded":"2024-09-17T07:16:17.368029+0000","last_fullsized":"2024-09-17T07:16:17.367816+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'25","last_scrub_stamp":"2024-09-17T07:14:50.794200+0000","last_deep_scrub":"110'15","last_deep_scrub_stamp":"2024-09-17T07:04:57.091400+0000","last_clean_scrub_stamp":"2024-09-17T07:14:50.794200+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0070558269999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[{"shards":"4,5","objects":5}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":30,"reported_epoch":324,"state":"active+undersized","last_fresh":"2024-09-17T07:17:29.764230+0000","last_change":"2024-09-17T07:16:17.378414+0000","last_active":"2024-09-17T07:17:29.764230+0000","last_peered":"2024-09-17T07:17:29.764230+0000","last_clean":"2024-09-17T07:02:10.991363+0000","last_became_active":"2024-09-17T07:16:17.378414+0000","last_became_peered":"2024-09-17T07:16:17.378414+0000","last_unstale":"2024-09-17T07:17:29.764230+0000","last_undegraded":"2024-09-17T07:17:29.764230+0000","last_fullsized":"2024-09-17T07:16:17.365679+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:49.764885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:25.147878+0000","last_clean_scrub_stamp":"2024-09-17T07:14:49.764885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"89'9","reported_seq":528,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-09-17T07:17:30.700626+0000","last_change":"2024-09-17T07:15:53.512860+0000","last_active":"2024-09-17T07:17:30.700626+0000","last_peered":"2024-09-17T07:17:30.700626+0000","last_clean":"2024-09-17T07:17:30.700626+0000","last_became_active":"2024-09-17T07:02:16.021360+0000","last_became_peered":"2024-09-17T07:02:16.021360+0000","last_unstale":"2024-09-17T07:17:30.700626+0000","last_undegraded":"2024-09-17T07:17:30.700626+0000","last_fullsized":"2024-09-17T07:17:30.700626+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'9","last_scrub_stamp":"2024-09-17T07:15:53.512777+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-09-17T07:15:53.512777+0000","last_clean_scrub_stamp":"2024-09-17T07:15:53.512777+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0020298600000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"268'2","reported_seq":348,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-09-17T07:17:29.759808+0000","last_change":"2024-09-17T07:16:46.448318+0000","last_active":"2024-09-17T07:17:29.759808+0000","last_peered":"2024-09-17T07:17:29.759808+0000","last_clean":"2024-09-17T07:17:29.759808+0000","last_became_active":"2024-09-17T07:09:24.399565+0000","last_became_peered":"2024-09-17T07:09:24.399565+0000","last_unstale":"2024-09-17T07:17:29.759808+0000","last_undegraded":"2024-09-17T07:17:29.759808+0000","last_fullsized":"2024-09-17T07:17:29.759808+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'2","last_scrub_stamp":"2024-09-17T07:16:46.448196+0000","last_deep_scrub":"268'2","last_deep_scrub_stamp":"2024-09-17T07:14:03.609131+0000","last_clean_scrub_stamp":"2024-09-17T07:16:46.448196+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:12.835807+0000","scrub_duration":0.00244297,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.a","version":"325'6758","reported_seq":6794,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-17T07:17:33.444539+0000","last_change":"2024-09-17T07:16:05.055275+0000","last_active":"2024-09-17T07:17:33.444539+0000","last_peered":"2024-09-17T07:17:33.444539+0000","last_clean":"2024-09-17T07:17:33.444539+0000","last_became_active":"2024-09-17T07:16:05.055008+0000","last_became_peered":"2024-09-17T07:16:05.055008+0000","last_unstale":"2024-09-17T07:17:33.444539+0000","last_undegraded":"2024-09-17T07:17:33.444539+0000","last_fullsized":"2024-09-17T07:17:33.444539+0000","mapping_epoch":297,"log_start":"307'1900","ondisk_log_start":"307'1900","created":297,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":4858,"log_dups_size":0,"ondisk_log_size":4858,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":442892288,"num_objects":6758,"num_object_clones":0,"num_object_copies":20274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6758,"num_write_kb":432512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"304'42","reported_seq":491,"reported_epoch":324,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:17:30.700561+0000","last_change":"2024-09-17T07:16:17.368142+0000","last_active":"2024-09-17T07:17:30.700561+0000","last_peered":"2024-09-17T07:17:30.700561+0000","last_clean":"2024-09-17T07:12:20.666266+0000","last_became_active":"2024-09-17T07:16:17.368142+0000","last_became_peered":"2024-09-17T07:16:17.368142+0000","last_unstale":"2024-09-17T07:17:30.700561+0000","last_undegraded":"2024-09-17T07:16:17.363645+0000","last_fullsized":"2024-09-17T07:16:17.363541+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'41","last_scrub_stamp":"2024-09-17T07:14:56.700858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:14:56.700858+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026058610000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[{"shards":"2,6","objects":10}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":344,"reported_epoch":324,"state":"active+undersized","last_fresh":"2024-09-17T07:17:29.768011+0000","last_change":"2024-09-17T07:16:17.371561+0000","last_active":"2024-09-17T07:17:29.768011+0000","last_peered":"2024-09-17T07:17:29.768011+0000","last_clean":"2024-09-17T07:16:16.170577+0000","last_became_active":"2024-09-17T07:16:17.371561+0000","last_became_peered":"2024-09-17T07:16:17.371561+0000","last_unstale":"2024-09-17T07:17:29.768011+0000","last_undegraded":"2024-09-17T07:17:29.768011+0000","last_fullsized":"2024-09-17T07:16:17.366918+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:03.045709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:14:03.045709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0049245929999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":341,"reported_epoch":324,"state":"active+undersized","last_fresh":"2024-09-17T07:17:29.760737+0000","last_change":"2024-09-17T07:16:17.375376+0000","last_active":"2024-09-17T07:17:29.760737+0000","last_peered":"2024-09-17T07:17:29.760737+0000","last_clean":"2024-09-17T07:16:15.495109+0000","last_became_active":"2024-09-17T07:16:17.375376+0000","last_became_peered":"2024-09-17T07:16:17.375376+0000","last_unstale":"2024-09-17T07:17:29.760737+0000","last_undegraded":"2024-09-17T07:17:29.760737+0000","last_fullsized":"2024-09-17T07:16:17.364236+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:36.523676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:15:36.523676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017370549999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":189,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-09-17T07:17:29.776922+0000","last_change":"2024-09-17T07:17:18.312624+0000","last_active":"2024-09-17T07:17:29.776922+0000","last_peered":"2024-09-17T07:17:29.776922+0000","last_clean":"2024-09-17T07:17:29.776922+0000","last_became_active":"2024-09-17T07:09:24.398878+0000","last_became_peered":"2024-09-17T07:09:24.398878+0000","last_unstale":"2024-09-17T07:17:29.776922+0000","last_undegraded":"2024-09-17T07:17:29.776922+0000","last_fullsized":"2024-09-17T07:17:29.776922+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:17:18.312499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:13:53.406661+0000","last_clean_scrub_stamp":"2024-09-17T07:17:18.312499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:38.339565+0000","scrub_duration":0.0030147720000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.9","version":"325'6782","reported_seq":6957,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-17T07:17:33.722007+0000","last_change":"2024-09-17T07:17:25.671223+0000","last_active":"2024-09-17T07:17:33.722007+0000","last_peered":"2024-09-17T07:17:33.722007+0000","last_clean":"2024-09-17T07:17:33.722007+0000","last_became_active":"2024-09-17T07:16:05.056719+0000","last_became_peered":"2024-09-17T07:16:05.056719+0000","last_unstale":"2024-09-17T07:17:33.722007+0000","last_undegraded":"2024-09-17T07:17:33.722007+0000","last_fullsized":"2024-09-17T07:17:33.722007+0000","mapping_epoch":297,"log_start":"306'1800","ondisk_log_start":"306'1800","created":297,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"322'6203","last_scrub_stamp":"2024-09-17T07:17:25.670907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:17:25.670907+0000","objects_scrubbed":6134,"log_size":4982,"log_dups_size":0,"ondisk_log_size":4982,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:31.101590+0000","scrub_duration":2.260616706,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":444465152,"num_objects":6782,"num_object_clones":0,"num_object_copies":20346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6782,"num_write_kb":434048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"242'17","reported_seq":289,"reported_epoch":324,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:17:29.767294+0000","last_change":"2024-09-17T07:16:17.371977+0000","last_active":"2024-09-17T07:17:29.767294+0000","last_peered":"2024-09-17T07:17:29.767294+0000","last_clean":"2024-09-17T07:12:20.284331+0000","last_became_active":"2024-09-17T07:16:17.371977+0000","last_became_peered":"2024-09-17T07:16:17.371977+0000","last_unstale":"2024-09-17T07:17:29.767294+0000","last_undegraded":"2024-09-17T07:16:17.367143+0000","last_fullsized":"2024-09-17T07:16:17.367038+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'17","last_scrub_stamp":"2024-09-17T07:13:47.468399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:13:47.468399+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.005881725,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[{"shards":"0,5","objects":5}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"76'5","reported_seq":135,"reported_epoch":325,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:17:31.747887+0000","last_change":"2024-09-17T07:16:17.368338+0000","last_active":"2024-09-17T07:17:31.747887+0000","last_peered":"2024-09-17T07:17:31.747887+0000","last_clean":"2024-09-17T07:02:23.985427+0000","last_became_active":"2024-09-17T07:16:17.368338+0000","last_became_peered":"2024-09-17T07:16:17.368338+0000","last_unstale":"2024-09-17T07:17:31.747887+0000","last_undegraded":"2024-09-17T07:16:17.363185+0000","last_fullsized":"2024-09-17T07:16:17.363086+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'5","last_scrub_stamp":"2024-09-17T07:15:36.901601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:15:36.901601+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033739799999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":82,"num_read_kb":77,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[{"shards":"2,6","objects":1}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":185,"reported_epoch":324,"state":"active+undersized","last_fresh":"2024-09-17T07:17:29.760136+0000","last_change":"2024-09-17T07:16:17.367238+0000","last_active":"2024-09-17T07:17:29.760136+0000","last_peered":"2024-09-17T07:17:29.760136+0000","last_clean":"2024-09-17T07:16:15.494771+0000","last_became_active":"2024-09-17T07:16:17.367238+0000","last_became_peered":"2024-09-17T07:16:17.367238+0000","last_unstale":"2024-09-17T07:17:29.760136+0000","last_undegraded":"2024-09-17T07:17:29.760136+0000","last_fullsized":"2024-09-17T07:16:17.364334+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:56.534547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:23.833170+0000","last_clean_scrub_stamp":"2024-09-17T07:15:56.534547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0011018950000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":358,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-09-17T07:17:29.766018+0000","last_change":"2024-09-17T07:16:46.778669+0000","last_active":"2024-09-17T07:17:29.766018+0000","last_peered":"2024-09-17T07:17:29.766018+0000","last_clean":"2024-09-17T07:17:29.766018+0000","last_became_active":"2024-09-17T07:02:16.014492+0000","last_became_peered":"2024-09-17T07:02:16.014492+0000","last_unstale":"2024-09-17T07:17:29.766018+0000","last_undegraded":"2024-09-17T07:17:29.766018+0000","last_fullsized":"2024-09-17T07:17:29.766018+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:46.778591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:16:46.778591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:53.201381+0000","scrub_duration":0.0013415079999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":341,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-09-17T07:17:29.762132+0000","last_change":"2024-09-17T07:15:02.062776+0000","last_active":"2024-09-17T07:17:29.762132+0000","last_peered":"2024-09-17T07:17:29.762132+0000","last_clean":"2024-09-17T07:17:29.762132+0000","last_became_active":"2024-09-17T07:09:24.402704+0000","last_became_peered":"2024-09-17T07:09:24.402704+0000","last_unstale":"2024-09-17T07:17:29.762132+0000","last_undegraded":"2024-09-17T07:17:29.762132+0000","last_fullsized":"2024-09-17T07:17:29.762132+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:02.062618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:02.062618+0000","last_clean_scrub_stamp":"2024-09-17T07:15:02.062618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0042833460000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"325'6910","reported_seq":6947,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-17T07:17:34.545325+0000","last_change":"2024-09-17T07:16:05.054358+0000","last_active":"2024-09-17T07:17:34.545325+0000","last_peered":"2024-09-17T07:17:34.545325+0000","last_clean":"2024-09-17T07:17:34.545325+0000","last_became_active":"2024-09-17T07:16:05.053996+0000","last_became_peered":"2024-09-17T07:16:05.053996+0000","last_unstale":"2024-09-17T07:17:34.545325+0000","last_undegraded":"2024-09-17T07:17:34.545325+0000","last_fullsized":"2024-09-17T07:17:34.545325+0000","mapping_epoch":297,"log_start":"307'2200","ondisk_log_start":"307'2200","created":297,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":4710,"log_dups_size":0,"ondisk_log_size":4710,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452853760,"num_objects":6910,"num_object_clones":0,"num_object_copies":20730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6910,"num_write_kb":442240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"268'30","reported_seq":402,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-09-17T07:17:29.761317+0000","last_change":"2024-09-17T07:16:43.375355+0000","last_active":"2024-09-17T07:17:29.761317+0000","last_peered":"2024-09-17T07:17:29.761317+0000","last_clean":"2024-09-17T07:17:29.761317+0000","last_became_active":"2024-09-17T07:03:48.760405+0000","last_became_peered":"2024-09-17T07:03:48.760405+0000","last_unstale":"2024-09-17T07:17:29.761317+0000","last_undegraded":"2024-09-17T07:17:29.761317+0000","last_fullsized":"2024-09-17T07:17:29.761317+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'30","last_scrub_stamp":"2024-09-17T07:16:43.375250+0000","last_deep_scrub":"268'30","last_deep_scrub_stamp":"2024-09-17T07:14:00.599656+0000","last_clean_scrub_stamp":"2024-09-17T07:16:43.375250+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:48.191852+0000","scrub_duration":0.006604922,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"70'1","reported_seq":358,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-09-17T07:17:29.760267+0000","last_change":"2024-09-17T07:17:26.441837+0000","last_active":"2024-09-17T07:17:29.760267+0000","last_peered":"2024-09-17T07:17:29.760267+0000","last_clean":"2024-09-17T07:17:29.760267+0000","last_became_active":"2024-09-17T07:09:24.402499+0000","last_became_peered":"2024-09-17T07:09:24.402499+0000","last_unstale":"2024-09-17T07:17:29.760267+0000","last_undegraded":"2024-09-17T07:17:29.760267+0000","last_fullsized":"2024-09-17T07:17:29.760267+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-17T07:17:26.441670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:17:26.441670+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:29.168291+0000","scrub_duration":0.0054314109999999997,"objects_trimmed":0,"snaptrim_duration":0.0032298629999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":333,"reported_epoch":324,"state":"active+undersized","last_fresh":"2024-09-17T07:17:29.760514+0000","last_change":"2024-09-17T07:16:17.375018+0000","last_active":"2024-09-17T07:17:29.760514+0000","last_peered":"2024-09-17T07:17:29.760514+0000","last_clean":"2024-09-17T07:16:15.590088+0000","last_became_active":"2024-09-17T07:16:17.375018+0000","last_became_peered":"2024-09-17T07:16:17.375018+0000","last_unstale":"2024-09-17T07:17:29.760514+0000","last_undegraded":"2024-09-17T07:17:29.760514+0000","last_fullsized":"2024-09-17T07:16:17.370688+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:55.321202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:55.321202+0000","last_clean_scrub_stamp":"2024-09-17T07:15:55.321202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014716950000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":340,"reported_epoch":324,"state":"active+undersized","last_fresh":"2024-09-17T07:17:29.766733+0000","last_change":"2024-09-17T07:16:17.375774+0000","last_active":"2024-09-17T07:17:29.766733+0000","last_peered":"2024-09-17T07:17:29.766733+0000","last_clean":"2024-09-17T07:16:16.169907+0000","last_became_active":"2024-09-17T07:16:17.375774+0000","last_became_peered":"2024-09-17T07:16:17.375774+0000","last_unstale":"2024-09-17T07:17:29.766733+0000","last_undegraded":"2024-09-17T07:17:29.766733+0000","last_fullsized":"2024-09-17T07:16:17.368453+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:45.447029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:15:45.447029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001990463,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":196,"reported_epoch":324,"state":"active+undersized","last_fresh":"2024-09-17T07:17:29.766551+0000","last_change":"2024-09-17T07:16:17.372859+0000","last_active":"2024-09-17T07:17:29.766551+0000","last_peered":"2024-09-17T07:17:29.766551+0000","last_clean":"2024-09-17T07:09:22.371903+0000","last_became_active":"2024-09-17T07:16:17.372859+0000","last_became_peered":"2024-09-17T07:16:17.372859+0000","last_unstale":"2024-09-17T07:17:29.766551+0000","last_undegraded":"2024-09-17T07:17:29.766551+0000","last_fullsized":"2024-09-17T07:16:17.367296+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:51.877367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:15:51.877367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.058290885000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.b","version":"325'6877","reported_seq":6919,"reported_epoch":325,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:17:33.454702+0000","last_change":"2024-09-17T07:16:18.636164+0000","last_active":"2024-09-17T07:17:33.454702+0000","last_peered":"2024-09-17T07:17:33.454702+0000","last_clean":"2024-09-17T07:16:14.246161+0000","last_became_active":"2024-09-17T07:16:17.365889+0000","last_became_peered":"2024-09-17T07:16:17.365889+0000","last_unstale":"2024-09-17T07:17:33.454702+0000","last_undegraded":"2024-09-17T07:16:18.628529+0000","last_fullsized":"2024-09-17T07:16:17.362973+0000","mapping_epoch":303,"log_start":"307'2100","ondisk_log_start":"307'2100","created":297,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":4777,"log_dups_size":0,"ondisk_log_size":4777,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":450691072,"num_objects":6877,"num_object_clones":0,"num_object_copies":20631,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6877,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6877,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6877,"num_write_kb":440128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[{"shards":"0,6","objects":6877}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"242'26","reported_seq":388,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-09-17T07:17:29.772947+0000","last_change":"2024-09-17T07:16:33.667044+0000","last_active":"2024-09-17T07:17:29.772947+0000","last_peered":"2024-09-17T07:17:29.772947+0000","last_clean":"2024-09-17T07:17:29.772947+0000","last_became_active":"2024-09-17T07:02:14.028857+0000","last_became_peered":"2024-09-17T07:02:14.028857+0000","last_unstale":"2024-09-17T07:17:29.772947+0000","last_undegraded":"2024-09-17T07:17:29.772947+0000","last_fullsized":"2024-09-17T07:17:29.772947+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'26","last_scrub_stamp":"2024-09-17T07:16:33.666684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:16:33.666684+0000","objects_scrubbed":7,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:38.577635+0000","scrub_duration":0.013353596000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"268'3","reported_seq":208,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-09-17T07:17:29.767797+0000","last_change":"2024-09-17T07:17:19.453842+0000","last_active":"2024-09-17T07:17:29.767797+0000","last_peered":"2024-09-17T07:17:29.767797+0000","last_clean":"2024-09-17T07:17:29.767797+0000","last_became_active":"2024-09-17T07:09:24.404532+0000","last_became_peered":"2024-09-17T07:09:24.404532+0000","last_unstale":"2024-09-17T07:17:29.767797+0000","last_undegraded":"2024-09-17T07:17:29.767797+0000","last_fullsized":"2024-09-17T07:17:29.767797+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'3","last_scrub_stamp":"2024-09-17T07:17:19.444444+0000","last_deep_scrub":"268'3","last_deep_scrub_stamp":"2024-09-17T07:13:54.949345+0000","last_clean_scrub_stamp":"2024-09-17T07:17:19.444444+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:20.398187+0000","scrub_duration":0.0048257380000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":349,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-09-17T07:17:29.766554+0000","last_change":"2024-09-17T07:15:55.432933+0000","last_active":"2024-09-17T07:17:29.766554+0000","last_peered":"2024-09-17T07:17:29.766554+0000","last_clean":"2024-09-17T07:17:29.766554+0000","last_became_active":"2024-09-17T07:02:12.004606+0000","last_became_peered":"2024-09-17T07:02:12.004606+0000","last_unstale":"2024-09-17T07:17:29.766554+0000","last_undegraded":"2024-09-17T07:17:29.766554+0000","last_fullsized":"2024-09-17T07:17:29.766554+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:55.432818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:15:55.432818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001411254,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"212'130","reported_seq":588,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-09-17T07:17:29.760700+0000","last_change":"2024-09-17T07:14:57.052759+0000","last_active":"2024-09-17T07:17:29.760700+0000","last_peered":"2024-09-17T07:17:29.760700+0000","last_clean":"2024-09-17T07:17:29.760700+0000","last_became_active":"2024-09-17T07:02:03.020675+0000","last_became_peered":"2024-09-17T07:02:03.020675+0000","last_unstale":"2024-09-17T07:17:29.760700+0000","last_undegraded":"2024-09-17T07:17:29.760700+0000","last_fullsized":"2024-09-17T07:17:29.760700+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:14:57.052505+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:14:57.052505+0000","last_clean_scrub_stamp":"2024-09-17T07:14:57.052505+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.022122466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":334,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-09-17T07:17:29.765852+0000","last_change":"2024-09-17T07:15:56.366761+0000","last_active":"2024-09-17T07:17:29.765852+0000","last_peered":"2024-09-17T07:17:29.765852+0000","last_clean":"2024-09-17T07:17:29.765852+0000","last_became_active":"2024-09-17T07:02:16.016031+0000","last_became_peered":"2024-09-17T07:02:16.016031+0000","last_unstale":"2024-09-17T07:17:29.765852+0000","last_undegraded":"2024-09-17T07:17:29.765852+0000","last_fullsized":"2024-09-17T07:17:29.765852+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:56.366669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:15:56.366669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015953269999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":336,"reported_epoch":324,"state":"active+undersized","last_fresh":"2024-09-17T07:17:29.767027+0000","last_change":"2024-09-17T07:16:17.377095+0000","last_active":"2024-09-17T07:17:29.767027+0000","last_peered":"2024-09-17T07:17:29.767027+0000","last_clean":"2024-09-17T07:16:16.170840+0000","last_became_active":"2024-09-17T07:16:17.377095+0000","last_became_peered":"2024-09-17T07:16:17.377095+0000","last_unstale":"2024-09-17T07:17:29.767027+0000","last_undegraded":"2024-09-17T07:17:29.767027+0000","last_fullsized":"2024-09-17T07:16:17.370787+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:48.021592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:58.670571+0000","last_clean_scrub_stamp":"2024-09-17T07:14:48.021592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0064844619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.4","version":"325'6889","reported_seq":6932,"reported_epoch":325,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:17:34.525890+0000","last_change":"2024-09-17T07:16:18.635996+0000","last_active":"2024-09-17T07:17:34.525890+0000","last_peered":"2024-09-17T07:17:34.525890+0000","last_clean":"2024-09-17T07:16:15.495005+0000","last_became_active":"2024-09-17T07:16:17.375662+0000","last_became_peered":"2024-09-17T07:16:17.375662+0000","last_unstale":"2024-09-17T07:17:34.525890+0000","last_undegraded":"2024-09-17T07:16:18.629481+0000","last_fullsized":"2024-09-17T07:16:17.364553+0000","mapping_epoch":303,"log_start":"307'2200","ondisk_log_start":"307'2200","created":297,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":4689,"log_dups_size":0,"ondisk_log_size":4689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":451477504,"num_objects":6889,"num_object_clones":0,"num_object_copies":20667,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6889,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6889,"num_write_kb":440896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0,4","objects":6889}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"242'35","reported_seq":409,"reported_epoch":324,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:17:29.773637+0000","last_change":"2024-09-17T07:16:17.376246+0000","last_active":"2024-09-17T07:17:29.773637+0000","last_peered":"2024-09-17T07:17:29.773637+0000","last_clean":"2024-09-17T07:16:14.244770+0000","last_became_active":"2024-09-17T07:16:17.376246+0000","last_became_peered":"2024-09-17T07:16:17.376246+0000","last_unstale":"2024-09-17T07:17:29.773637+0000","last_undegraded":"2024-09-17T07:16:17.363365+0000","last_fullsized":"2024-09-17T07:16:17.363257+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'35","last_scrub_stamp":"2024-09-17T07:15:43.665829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:15:43.665829+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0048325160000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[{"shards":"6,7","objects":9}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":383,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-09-17T07:17:29.766196+0000","last_change":"2024-09-17T07:17:20.848821+0000","last_active":"2024-09-17T07:17:29.766196+0000","last_peered":"2024-09-17T07:17:29.766196+0000","last_clean":"2024-09-17T07:17:29.766196+0000","last_became_active":"2024-09-17T07:02:12.012037+0000","last_became_peered":"2024-09-17T07:02:12.012037+0000","last_unstale":"2024-09-17T07:17:29.766196+0000","last_undegraded":"2024-09-17T07:17:29.766196+0000","last_fullsized":"2024-09-17T07:17:29.766196+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:17:20.848743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:06:39.322535+0000","last_clean_scrub_stamp":"2024-09-17T07:17:20.848743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:38.587002+0000","scrub_duration":0.001349062,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":367,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-09-17T07:17:29.766500+0000","last_change":"2024-09-17T07:16:44.719372+0000","last_active":"2024-09-17T07:17:29.766500+0000","last_peered":"2024-09-17T07:17:29.766500+0000","last_clean":"2024-09-17T07:17:29.766500+0000","last_became_active":"2024-09-17T07:02:16.014631+0000","last_became_peered":"2024-09-17T07:02:16.014631+0000","last_unstale":"2024-09-17T07:17:29.766500+0000","last_undegraded":"2024-09-17T07:17:29.766500+0000","last_fullsized":"2024-09-17T07:17:29.766500+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:44.719264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:16:44.719264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:03.887933+0000","scrub_duration":0.0036441289999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":330,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-17T07:17:30.779372+0000","last_change":"2024-09-17T07:15:45.275553+0000","last_active":"2024-09-17T07:17:30.779372+0000","last_peered":"2024-09-17T07:17:30.779372+0000","last_clean":"2024-09-17T07:17:30.779372+0000","last_became_active":"2024-09-17T07:09:24.403904+0000","last_became_peered":"2024-09-17T07:09:24.403904+0000","last_unstale":"2024-09-17T07:17:30.779372+0000","last_undegraded":"2024-09-17T07:17:30.779372+0000","last_fullsized":"2024-09-17T07:17:30.779372+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:45.275473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:45.275473+0000","last_clean_scrub_stamp":"2024-09-17T07:15:45.275473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014472759999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"325'6948","reported_seq":6985,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-17T07:17:34.543621+0000","last_change":"2024-09-17T07:16:05.056347+0000","last_active":"2024-09-17T07:17:34.543621+0000","last_peered":"2024-09-17T07:17:34.543621+0000","last_clean":"2024-09-17T07:17:34.543621+0000","last_became_active":"2024-09-17T07:16:05.056100+0000","last_became_peered":"2024-09-17T07:16:05.056100+0000","last_unstale":"2024-09-17T07:17:34.543621+0000","last_undegraded":"2024-09-17T07:17:34.543621+0000","last_fullsized":"2024-09-17T07:17:34.543621+0000","mapping_epoch":297,"log_start":"307'2200","ondisk_log_start":"307'2200","created":297,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":4748,"log_dups_size":0,"ondisk_log_size":4748,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":455344128,"num_objects":6948,"num_object_clones":0,"num_object_copies":20844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6948,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6948,"num_write_kb":444672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"268'19","reported_seq":360,"reported_epoch":324,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:17:29.760573+0000","last_change":"2024-09-17T07:16:17.375651+0000","last_active":"2024-09-17T07:17:29.760573+0000","last_peered":"2024-09-17T07:17:29.760573+0000","last_clean":"2024-09-17T07:16:15.494820+0000","last_became_active":"2024-09-17T07:16:17.375651+0000","last_became_peered":"2024-09-17T07:16:17.375651+0000","last_unstale":"2024-09-17T07:17:29.760573+0000","last_undegraded":"2024-09-17T07:16:17.364532+0000","last_fullsized":"2024-09-17T07:16:17.364433+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'19","last_scrub_stamp":"2024-09-17T07:14:06.634334+0000","last_deep_scrub":"268'19","last_deep_scrub_stamp":"2024-09-17T07:14:06.634334+0000","last_clean_scrub_stamp":"2024-09-17T07:14:06.634334+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0088943289999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0,4","objects":4}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":202,"reported_epoch":324,"state":"active+undersized","last_fresh":"2024-09-17T07:17:29.770641+0000","last_change":"2024-09-17T07:16:17.374110+0000","last_active":"2024-09-17T07:17:29.770641+0000","last_peered":"2024-09-17T07:17:29.770641+0000","last_clean":"2024-09-17T07:09:22.372513+0000","last_became_active":"2024-09-17T07:16:17.374110+0000","last_became_peered":"2024-09-17T07:16:17.374110+0000","last_unstale":"2024-09-17T07:17:29.770641+0000","last_undegraded":"2024-09-17T07:17:29.770641+0000","last_fullsized":"2024-09-17T07:16:17.367137+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:58.886294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:13:53.593078+0000","last_clean_scrub_stamp":"2024-09-17T07:15:58.886294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.005853245,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":310,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-09-17T07:17:29.768324+0000","last_change":"2024-09-17T07:16:35.152814+0000","last_active":"2024-09-17T07:17:29.768324+0000","last_peered":"2024-09-17T07:17:29.768324+0000","last_clean":"2024-09-17T07:17:29.768324+0000","last_became_active":"2024-09-17T07:09:24.396642+0000","last_became_peered":"2024-09-17T07:09:24.396642+0000","last_unstale":"2024-09-17T07:17:29.768324+0000","last_undegraded":"2024-09-17T07:17:29.768324+0000","last_fullsized":"2024-09-17T07:17:29.768324+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:35.152744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:16:35.152744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:04.541036+0000","scrub_duration":0.0022724160000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":333,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-17T07:17:30.774105+0000","last_change":"2024-09-17T07:17:27.482509+0000","last_active":"2024-09-17T07:17:30.774105+0000","last_peered":"2024-09-17T07:17:30.774105+0000","last_clean":"2024-09-17T07:17:30.774105+0000","last_became_active":"2024-09-17T07:02:18.026994+0000","last_became_peered":"2024-09-17T07:02:18.026994+0000","last_unstale":"2024-09-17T07:17:30.774105+0000","last_undegraded":"2024-09-17T07:17:30.774105+0000","last_fullsized":"2024-09-17T07:17:30.774105+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:17:27.482401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:58.099849+0000","last_clean_scrub_stamp":"2024-09-17T07:17:27.482401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:42.245140+0000","scrub_duration":0.0020589919999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.6","version":"325'6824","reported_seq":7005,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-17T07:17:33.636782+0000","last_change":"2024-09-17T07:17:30.778260+0000","last_active":"2024-09-17T07:17:33.636782+0000","last_peered":"2024-09-17T07:17:33.636782+0000","last_clean":"2024-09-17T07:17:33.636782+0000","last_became_active":"2024-09-17T07:16:05.057758+0000","last_became_peered":"2024-09-17T07:16:05.057758+0000","last_unstale":"2024-09-17T07:17:33.636782+0000","last_undegraded":"2024-09-17T07:17:33.636782+0000","last_fullsized":"2024-09-17T07:17:33.636782+0000","mapping_epoch":297,"log_start":"307'1900","ondisk_log_start":"307'1900","created":297,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"324'6643","last_scrub_stamp":"2024-09-17T07:17:30.776520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:17:30.776520+0000","objects_scrubbed":6552,"log_size":4924,"log_dups_size":0,"ondisk_log_size":4924,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:42.108478+0000","scrub_duration":2.27785153,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":447217664,"num_objects":6824,"num_object_clones":0,"num_object_copies":20472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6824,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6824,"num_write_kb":436736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"242'34","reported_seq":397,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-09-17T07:17:29.766104+0000","last_change":"2024-09-17T07:16:44.126984+0000","last_active":"2024-09-17T07:17:29.766104+0000","last_peered":"2024-09-17T07:17:29.766104+0000","last_clean":"2024-09-17T07:17:29.766104+0000","last_became_active":"2024-09-17T07:02:14.028392+0000","last_became_peered":"2024-09-17T07:02:14.028392+0000","last_unstale":"2024-09-17T07:17:29.766104+0000","last_undegraded":"2024-09-17T07:17:29.766104+0000","last_fullsized":"2024-09-17T07:17:29.766104+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'34","last_scrub_stamp":"2024-09-17T07:16:44.126830+0000","last_deep_scrub":"242'34","last_deep_scrub_stamp":"2024-09-17T07:14:50.103539+0000","last_clean_scrub_stamp":"2024-09-17T07:16:44.126830+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:57.642107+0000","scrub_duration":0.008313473,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'2","reported_seq":209,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-09-17T07:17:29.761571+0000","last_change":"2024-09-17T07:15:48.378000+0000","last_active":"2024-09-17T07:17:29.761571+0000","last_peered":"2024-09-17T07:17:29.761571+0000","last_clean":"2024-09-17T07:17:29.761571+0000","last_became_active":"2024-09-17T07:09:24.402909+0000","last_became_peered":"2024-09-17T07:09:24.402909+0000","last_unstale":"2024-09-17T07:17:29.761571+0000","last_undegraded":"2024-09-17T07:17:29.761571+0000","last_fullsized":"2024-09-17T07:17:29.761571+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-09-17T07:15:48.377886+0000","last_deep_scrub":"89'2","last_deep_scrub_stamp":"2024-09-17T07:12:20.301263+0000","last_clean_scrub_stamp":"2024-09-17T07:15:48.377886+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001635079,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":420,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":420,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"225'9","reported_seq":519,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-09-17T07:17:30.702003+0000","last_change":"2024-09-17T07:14:42.018236+0000","last_active":"2024-09-17T07:17:30.702003+0000","last_peered":"2024-09-17T07:17:30.702003+0000","last_clean":"2024-09-17T07:17:30.702003+0000","last_became_active":"2024-09-17T07:02:16.015418+0000","last_became_peered":"2024-09-17T07:02:16.015418+0000","last_unstale":"2024-09-17T07:17:30.702003+0000","last_undegraded":"2024-09-17T07:17:30.702003+0000","last_fullsized":"2024-09-17T07:17:30.702003+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"225'9","last_scrub_stamp":"2024-09-17T07:14:42.017656+0000","last_deep_scrub":"225'9","last_deep_scrub_stamp":"2024-09-17T07:11:25.292565+0000","last_clean_scrub_stamp":"2024-09-17T07:14:42.017656+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0089186300000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":343,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-17T07:17:30.777040+0000","last_change":"2024-09-17T07:16:35.068863+0000","last_active":"2024-09-17T07:17:30.777040+0000","last_peered":"2024-09-17T07:17:30.777040+0000","last_clean":"2024-09-17T07:17:30.777040+0000","last_became_active":"2024-09-17T07:02:18.363253+0000","last_became_peered":"2024-09-17T07:02:18.363253+0000","last_unstale":"2024-09-17T07:17:30.777040+0000","last_undegraded":"2024-09-17T07:17:30.777040+0000","last_fullsized":"2024-09-17T07:17:30.777040+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:35.068712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:16:35.068712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:47.164438+0000","scrub_duration":0.0062141719999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.7","version":"325'6866","reported_seq":6902,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-17T07:17:33.429762+0000","last_change":"2024-09-17T07:16:05.054694+0000","last_active":"2024-09-17T07:17:33.429762+0000","last_peered":"2024-09-17T07:17:33.429762+0000","last_clean":"2024-09-17T07:17:33.429762+0000","last_became_active":"2024-09-17T07:16:05.054306+0000","last_became_peered":"2024-09-17T07:16:05.054306+0000","last_unstale":"2024-09-17T07:17:33.429762+0000","last_undegraded":"2024-09-17T07:17:33.429762+0000","last_fullsized":"2024-09-17T07:17:33.429762+0000","mapping_epoch":297,"log_start":"307'2100","ondisk_log_start":"307'2100","created":297,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":4766,"log_dups_size":0,"ondisk_log_size":4766,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":449970176,"num_objects":6866,"num_object_clones":0,"num_object_copies":20598,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6866,"num_write_kb":439424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"242'23","reported_seq":417,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-09-17T07:17:29.768002+0000","last_change":"2024-09-17T07:15:58.176415+0000","last_active":"2024-09-17T07:17:29.768002+0000","last_peered":"2024-09-17T07:17:29.768002+0000","last_clean":"2024-09-17T07:17:29.768002+0000","last_became_active":"2024-09-17T07:10:49.723292+0000","last_became_peered":"2024-09-17T07:10:49.723292+0000","last_unstale":"2024-09-17T07:17:29.768002+0000","last_undegraded":"2024-09-17T07:17:29.768002+0000","last_fullsized":"2024-09-17T07:17:29.768002+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'23","last_scrub_stamp":"2024-09-17T07:15:58.176287+0000","last_deep_scrub":"150'17","last_deep_scrub_stamp":"2024-09-17T07:06:38.775788+0000","last_clean_scrub_stamp":"2024-09-17T07:15:58.176287+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0024399420000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"68'2","reported_seq":357,"reported_epoch":324,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:17:29.769905+0000","last_change":"2024-09-17T07:16:17.371234+0000","last_active":"2024-09-17T07:17:29.769905+0000","last_peered":"2024-09-17T07:17:29.769905+0000","last_clean":"2024-09-17T07:16:15.358653+0000","last_became_active":"2024-09-17T07:16:17.371234+0000","last_became_peered":"2024-09-17T07:16:17.371234+0000","last_unstale":"2024-09-17T07:17:29.769905+0000","last_undegraded":"2024-09-17T07:16:17.366653+0000","last_fullsized":"2024-09-17T07:16:17.366079+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-09-17T07:14:56.533798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:14:56.533798+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003207115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[{"shards":"0,4","objects":2}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":305,"reported_epoch":324,"state":"active+undersized","last_fresh":"2024-09-17T07:17:29.759866+0000","last_change":"2024-09-17T07:16:17.367290+0000","last_active":"2024-09-17T07:17:29.759866+0000","last_peered":"2024-09-17T07:17:29.759866+0000","last_clean":"2024-09-17T07:16:15.494380+0000","last_became_active":"2024-09-17T07:16:17.367290+0000","last_became_peered":"2024-09-17T07:16:17.367290+0000","last_unstale":"2024-09-17T07:17:29.759866+0000","last_undegraded":"2024-09-17T07:17:29.759866+0000","last_fullsized":"2024-09-17T07:16:17.363484+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:51.586929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:11:17.479036+0000","last_clean_scrub_stamp":"2024-09-17T07:15:51.586929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001274435,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":173,"reported_epoch":324,"state":"active+undersized","last_fresh":"2024-09-17T07:17:29.760380+0000","last_change":"2024-09-17T07:16:17.379791+0000","last_active":"2024-09-17T07:17:29.760380+0000","last_peered":"2024-09-17T07:17:29.760380+0000","last_clean":"2024-09-17T07:16:15.590199+0000","last_became_active":"2024-09-17T07:16:17.379791+0000","last_became_peered":"2024-09-17T07:16:17.379791+0000","last_unstale":"2024-09-17T07:17:29.760380+0000","last_undegraded":"2024-09-17T07:17:29.760380+0000","last_fullsized":"2024-09-17T07:16:17.372333+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:13.937979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:13:13.937979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.018487173999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"242'15","reported_seq":380,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-09-17T07:17:29.761227+0000","last_change":"2024-09-17T07:15:58.508698+0000","last_active":"2024-09-17T07:17:29.761227+0000","last_peered":"2024-09-17T07:17:29.761227+0000","last_clean":"2024-09-17T07:17:29.761227+0000","last_became_active":"2024-09-17T07:09:24.408875+0000","last_became_peered":"2024-09-17T07:09:24.408875+0000","last_unstale":"2024-09-17T07:17:29.761227+0000","last_undegraded":"2024-09-17T07:17:29.761227+0000","last_fullsized":"2024-09-17T07:17:29.761227+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'15","last_scrub_stamp":"2024-09-17T07:15:58.508607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:15:58.508607+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0022156900000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":367,"reported_epoch":324,"state":"active+undersized","last_fresh":"2024-09-17T07:17:29.766976+0000","last_change":"2024-09-17T07:16:17.367792+0000","last_active":"2024-09-17T07:17:29.766976+0000","last_peered":"2024-09-17T07:17:29.766976+0000","last_clean":"2024-09-17T07:16:14.244722+0000","last_became_active":"2024-09-17T07:16:17.367792+0000","last_became_peered":"2024-09-17T07:16:17.367792+0000","last_unstale":"2024-09-17T07:17:29.766976+0000","last_undegraded":"2024-09-17T07:17:29.766976+0000","last_fullsized":"2024-09-17T07:16:17.366356+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:39.634698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:17.976267+0000","last_clean_scrub_stamp":"2024-09-17T07:15:39.634698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00117202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":32,"reported_epoch":324,"state":"active+undersized","last_fresh":"2024-09-17T07:17:29.761974+0000","last_change":"2024-09-17T07:16:17.378272+0000","last_active":"2024-09-17T07:17:29.761974+0000","last_peered":"2024-09-17T07:17:29.761974+0000","last_clean":"2024-09-17T07:02:14.999435+0000","last_became_active":"2024-09-17T07:16:17.378272+0000","last_became_peered":"2024-09-17T07:16:17.378272+0000","last_unstale":"2024-09-17T07:17:29.761974+0000","last_undegraded":"2024-09-17T07:17:29.761974+0000","last_fullsized":"2024-09-17T07:16:17.373119+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:40.845816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:18.367486+0000","last_clean_scrub_stamp":"2024-09-17T07:15:40.845816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":377,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-09-17T07:17:29.765747+0000","last_change":"2024-09-17T07:16:33.749048+0000","last_active":"2024-09-17T07:17:29.765747+0000","last_peered":"2024-09-17T07:17:29.765747+0000","last_clean":"2024-09-17T07:17:29.765747+0000","last_became_active":"2024-09-17T07:02:18.017825+0000","last_became_peered":"2024-09-17T07:02:18.017825+0000","last_unstale":"2024-09-17T07:17:29.765747+0000","last_undegraded":"2024-09-17T07:17:29.765747+0000","last_fullsized":"2024-09-17T07:17:29.765747+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:33.748908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:09:23.378805+0000","last_clean_scrub_stamp":"2024-09-17T07:16:33.748908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:36.947226+0000","scrub_duration":0.0073832719999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":355,"reported_epoch":324,"state":"active+undersized","last_fresh":"2024-09-17T07:17:29.772919+0000","last_change":"2024-09-17T07:16:17.375999+0000","last_active":"2024-09-17T07:17:29.772919+0000","last_peered":"2024-09-17T07:17:29.772919+0000","last_clean":"2024-09-17T07:16:14.245868+0000","last_became_active":"2024-09-17T07:16:17.375999+0000","last_became_peered":"2024-09-17T07:16:17.375999+0000","last_unstale":"2024-09-17T07:17:29.772919+0000","last_undegraded":"2024-09-17T07:17:29.772919+0000","last_fullsized":"2024-09-17T07:16:17.363383+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:46.645080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:46.645080+0000","last_clean_scrub_stamp":"2024-09-17T07:15:46.645080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015661049999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"242'12","reported_seq":210,"reported_epoch":324,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:17:29.761853+0000","last_change":"2024-09-17T07:16:17.379665+0000","last_active":"2024-09-17T07:17:29.761853+0000","last_peered":"2024-09-17T07:17:29.761853+0000","last_clean":"2024-09-17T07:16:15.590425+0000","last_became_active":"2024-09-17T07:16:17.379665+0000","last_became_peered":"2024-09-17T07:16:17.379665+0000","last_unstale":"2024-09-17T07:17:29.761853+0000","last_undegraded":"2024-09-17T07:16:17.373659+0000","last_fullsized":"2024-09-17T07:16:17.373552+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'12","last_scrub_stamp":"2024-09-17T07:15:44.249858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:15:44.249858+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0022006600000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":3}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":348,"reported_epoch":324,"state":"active+undersized","last_fresh":"2024-09-17T07:17:29.771527+0000","last_change":"2024-09-17T07:16:17.377495+0000","last_active":"2024-09-17T07:17:29.771527+0000","last_peered":"2024-09-17T07:17:29.771527+0000","last_clean":"2024-09-17T07:16:16.170775+0000","last_became_active":"2024-09-17T07:16:17.377495+0000","last_became_peered":"2024-09-17T07:16:17.377495+0000","last_unstale":"2024-09-17T07:17:29.771527+0000","last_undegraded":"2024-09-17T07:17:29.771527+0000","last_fullsized":"2024-09-17T07:16:17.367529+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:55.191252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:14:55.191252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0090150360000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":320,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-09-17T07:17:29.762059+0000","last_change":"2024-09-17T07:14:52.058961+0000","last_active":"2024-09-17T07:17:29.762059+0000","last_peered":"2024-09-17T07:17:29.762059+0000","last_clean":"2024-09-17T07:17:29.762059+0000","last_became_active":"2024-09-17T07:02:18.365419+0000","last_became_peered":"2024-09-17T07:02:18.365419+0000","last_unstale":"2024-09-17T07:17:29.762059+0000","last_undegraded":"2024-09-17T07:17:29.762059+0000","last_fullsized":"2024-09-17T07:17:29.762059+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:52.058847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:14:52.058847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0054926549999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"242'20","reported_seq":153,"reported_epoch":324,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:17:29.765909+0000","last_change":"2024-09-17T07:16:17.380213+0000","last_active":"2024-09-17T07:17:29.765909+0000","last_peered":"2024-09-17T07:17:29.765909+0000","last_clean":"2024-09-17T07:12:20.630919+0000","last_became_active":"2024-09-17T07:16:17.380213+0000","last_became_peered":"2024-09-17T07:16:17.380213+0000","last_unstale":"2024-09-17T07:17:29.765909+0000","last_undegraded":"2024-09-17T07:16:17.375354+0000","last_fullsized":"2024-09-17T07:16:17.375256+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'20","last_scrub_stamp":"2024-09-17T07:14:05.528769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:14:05.528769+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0020069329999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":5}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":368,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-09-17T07:17:29.767871+0000","last_change":"2024-09-17T07:17:20.425482+0000","last_active":"2024-09-17T07:17:29.767871+0000","last_peered":"2024-09-17T07:17:29.767871+0000","last_clean":"2024-09-17T07:17:29.767871+0000","last_became_active":"2024-09-17T07:09:24.402355+0000","last_became_peered":"2024-09-17T07:09:24.402355+0000","last_unstale":"2024-09-17T07:17:29.767871+0000","last_undegraded":"2024-09-17T07:17:29.767871+0000","last_fullsized":"2024-09-17T07:17:29.767871+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:17:20.425212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:17:20.425212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:34.381390+0000","scrub_duration":0.0020024069999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"304'10","reported_seq":533,"reported_epoch":324,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:17:30.700523+0000","last_change":"2024-09-17T07:16:17.373566+0000","last_active":"2024-09-17T07:17:30.700523+0000","last_peered":"2024-09-17T07:17:30.700523+0000","last_clean":"2024-09-17T07:16:16.169875+0000","last_became_active":"2024-09-17T07:16:17.373566+0000","last_became_peered":"2024-09-17T07:16:17.373566+0000","last_unstale":"2024-09-17T07:17:30.700523+0000","last_undegraded":"2024-09-17T07:16:17.367771+0000","last_fullsized":"2024-09-17T07:16:17.367624+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'9","last_scrub_stamp":"2024-09-17T07:14:53.140133+0000","last_deep_scrub":"163'9","last_deep_scrub_stamp":"2024-09-17T07:07:34.933808+0000","last_clean_scrub_stamp":"2024-09-17T07:14:53.140133+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0088819989999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[{"shards":"0,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":341,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-09-17T07:17:29.760927+0000","last_change":"2024-09-17T07:16:34.364362+0000","last_active":"2024-09-17T07:17:29.760927+0000","last_peered":"2024-09-17T07:17:29.760927+0000","last_clean":"2024-09-17T07:17:29.760927+0000","last_became_active":"2024-09-17T07:09:24.409051+0000","last_became_peered":"2024-09-17T07:09:24.409051+0000","last_unstale":"2024-09-17T07:17:29.760927+0000","last_undegraded":"2024-09-17T07:17:29.760927+0000","last_fullsized":"2024-09-17T07:17:29.760927+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:34.363904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:26.619893+0000","last_clean_scrub_stamp":"2024-09-17T07:16:34.363904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:01.999702+0000","scrub_duration":0.0046574870000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"242'15","reported_seq":390,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-09-17T07:17:29.767256+0000","last_change":"2024-09-17T07:16:34.100665+0000","last_active":"2024-09-17T07:17:29.767256+0000","last_peered":"2024-09-17T07:17:29.767256+0000","last_clean":"2024-09-17T07:17:29.767256+0000","last_became_active":"2024-09-17T07:09:24.406333+0000","last_became_peered":"2024-09-17T07:09:24.406333+0000","last_unstale":"2024-09-17T07:17:29.767256+0000","last_undegraded":"2024-09-17T07:17:29.767256+0000","last_fullsized":"2024-09-17T07:17:29.767256+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'15","last_scrub_stamp":"2024-09-17T07:16:34.100540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:16:34.100540+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:53.725587+0000","scrub_duration":0.0033371669999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":351,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-09-17T07:17:29.760726+0000","last_change":"2024-09-17T07:17:18.561114+0000","last_active":"2024-09-17T07:17:29.760726+0000","last_peered":"2024-09-17T07:17:29.760726+0000","last_clean":"2024-09-17T07:17:29.760726+0000","last_became_active":"2024-09-17T07:09:24.402704+0000","last_became_peered":"2024-09-17T07:09:24.402704+0000","last_unstale":"2024-09-17T07:17:29.760726+0000","last_undegraded":"2024-09-17T07:17:29.760726+0000","last_fullsized":"2024-09-17T07:17:29.760726+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:17:18.560995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:17:18.560995+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:43.212331+0000","scrub_duration":0.0064484570000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"304'10","reported_seq":296,"reported_epoch":324,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:17:30.702403+0000","last_change":"2024-09-17T07:16:17.380279+0000","last_active":"2024-09-17T07:17:30.702403+0000","last_peered":"2024-09-17T07:17:30.702403+0000","last_clean":"2024-09-17T07:16:15.687380+0000","last_became_active":"2024-09-17T07:16:17.380279+0000","last_became_peered":"2024-09-17T07:16:17.380279+0000","last_unstale":"2024-09-17T07:17:30.702403+0000","last_undegraded":"2024-09-17T07:16:17.375037+0000","last_fullsized":"2024-09-17T07:16:17.374926+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'9","last_scrub_stamp":"2024-09-17T07:13:10.965855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:13:10.965855+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0088433720000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":333,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-09-17T07:17:29.769581+0000","last_change":"2024-09-17T07:16:31.078582+0000","last_active":"2024-09-17T07:17:29.769581+0000","last_peered":"2024-09-17T07:17:29.769581+0000","last_clean":"2024-09-17T07:17:29.769581+0000","last_became_active":"2024-09-17T07:02:18.018716+0000","last_became_peered":"2024-09-17T07:02:18.018716+0000","last_unstale":"2024-09-17T07:17:29.769581+0000","last_undegraded":"2024-09-17T07:17:29.769581+0000","last_fullsized":"2024-09-17T07:17:29.769581+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:31.078504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:16:31.078504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:36.707922+0000","scrub_duration":0.0014265700000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"242'15","reported_seq":281,"reported_epoch":324,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:17:29.773344+0000","last_change":"2024-09-17T07:16:17.368982+0000","last_active":"2024-09-17T07:17:29.773344+0000","last_peered":"2024-09-17T07:17:29.773344+0000","last_clean":"2024-09-17T07:12:20.132949+0000","last_became_active":"2024-09-17T07:16:17.368982+0000","last_became_peered":"2024-09-17T07:16:17.368982+0000","last_unstale":"2024-09-17T07:17:29.773344+0000","last_undegraded":"2024-09-17T07:16:17.363353+0000","last_fullsized":"2024-09-17T07:16:17.363248+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'15","last_scrub_stamp":"2024-09-17T07:13:48.500257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:13:48.500257+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.005053564,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[{"shards":"2,6","objects":4}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":368,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-09-17T07:17:29.772810+0000","last_change":"2024-09-17T07:17:18.347374+0000","last_active":"2024-09-17T07:17:29.772810+0000","last_peered":"2024-09-17T07:17:29.772810+0000","last_clean":"2024-09-17T07:17:29.772810+0000","last_became_active":"2024-09-17T07:09:24.408139+0000","last_became_peered":"2024-09-17T07:09:24.408139+0000","last_unstale":"2024-09-17T07:17:29.772810+0000","last_undegraded":"2024-09-17T07:17:29.772810+0000","last_fullsized":"2024-09-17T07:17:29.772810+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:17:18.347257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:10:36.334827+0000","last_clean_scrub_stamp":"2024-09-17T07:17:18.347257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:21.549853+0000","scrub_duration":0.013321,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":184,"reported_epoch":324,"state":"active+undersized","last_fresh":"2024-09-17T07:17:29.760441+0000","last_change":"2024-09-17T07:16:17.373522+0000","last_active":"2024-09-17T07:17:29.760441+0000","last_peered":"2024-09-17T07:17:29.760441+0000","last_clean":"2024-09-17T07:16:15.494646+0000","last_became_active":"2024-09-17T07:16:17.373522+0000","last_became_peered":"2024-09-17T07:16:17.373522+0000","last_unstale":"2024-09-17T07:17:29.760441+0000","last_undegraded":"2024-09-17T07:17:29.760441+0000","last_fullsized":"2024-09-17T07:16:17.364098+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:37.523283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:15:37.523283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013475850000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":338,"reported_epoch":324,"state":"active+undersized","last_fresh":"2024-09-17T07:17:29.759743+0000","last_change":"2024-09-17T07:16:17.371270+0000","last_active":"2024-09-17T07:17:29.759743+0000","last_peered":"2024-09-17T07:17:29.759743+0000","last_clean":"2024-09-17T07:16:15.494422+0000","last_became_active":"2024-09-17T07:16:17.371270+0000","last_became_peered":"2024-09-17T07:16:17.371270+0000","last_unstale":"2024-09-17T07:17:29.759743+0000","last_undegraded":"2024-09-17T07:17:29.759743+0000","last_fullsized":"2024-09-17T07:16:17.363365+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:40.540336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:15:40.540336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001212982,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"242'19","reported_seq":363,"reported_epoch":324,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:17:29.769140+0000","last_change":"2024-09-17T07:16:17.382006+0000","last_active":"2024-09-17T07:17:29.769140+0000","last_peered":"2024-09-17T07:17:29.769140+0000","last_clean":"2024-09-17T07:16:15.358100+0000","last_became_active":"2024-09-17T07:16:17.382006+0000","last_became_peered":"2024-09-17T07:16:17.382006+0000","last_unstale":"2024-09-17T07:17:29.769140+0000","last_undegraded":"2024-09-17T07:16:17.365274+0000","last_fullsized":"2024-09-17T07:16:17.365168+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'19","last_scrub_stamp":"2024-09-17T07:14:05.729535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:14:05.729535+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.008660559,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"94'2","reported_seq":75,"reported_epoch":324,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:17:29.767346+0000","last_change":"2024-09-17T07:16:17.374418+0000","last_active":"2024-09-17T07:17:29.767346+0000","last_peered":"2024-09-17T07:17:29.767346+0000","last_clean":"2024-09-17T07:04:06.025673+0000","last_became_active":"2024-09-17T07:16:17.374418+0000","last_became_peered":"2024-09-17T07:16:17.374418+0000","last_unstale":"2024-09-17T07:17:29.767346+0000","last_undegraded":"2024-09-17T07:16:17.367359+0000","last_fullsized":"2024-09-17T07:16:17.367242+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-09-17T07:15:48.907821+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-09-17T07:07:31.717888+0000","last_clean_scrub_stamp":"2024-09-17T07:15:48.907821+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[{"shards":"0,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":198,"reported_epoch":324,"state":"active+undersized","last_fresh":"2024-09-17T07:17:29.767554+0000","last_change":"2024-09-17T07:16:17.378918+0000","last_active":"2024-09-17T07:17:29.767554+0000","last_peered":"2024-09-17T07:17:29.767554+0000","last_clean":"2024-09-17T07:09:22.372509+0000","last_became_active":"2024-09-17T07:16:17.378918+0000","last_became_peered":"2024-09-17T07:16:17.378918+0000","last_unstale":"2024-09-17T07:17:29.767554+0000","last_undegraded":"2024-09-17T07:17:29.767554+0000","last_fullsized":"2024-09-17T07:16:17.367596+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:41.858093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:15:41.858093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037672679999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":177,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-09-17T07:17:29.760027+0000","last_change":"2024-09-17T07:15:49.573768+0000","last_active":"2024-09-17T07:17:29.760027+0000","last_peered":"2024-09-17T07:17:29.760027+0000","last_clean":"2024-09-17T07:17:29.760027+0000","last_became_active":"2024-09-17T07:09:24.400791+0000","last_became_peered":"2024-09-17T07:09:24.400791+0000","last_unstale":"2024-09-17T07:17:29.760027+0000","last_undegraded":"2024-09-17T07:17:29.760027+0000","last_fullsized":"2024-09-17T07:17:29.760027+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:49.573666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:49.573666+0000","last_clean_scrub_stamp":"2024-09-17T07:15:49.573666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017548970000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"242'8","reported_seq":250,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-09-17T07:17:29.766665+0000","last_change":"2024-09-17T07:16:32.727879+0000","last_active":"2024-09-17T07:17:29.766665+0000","last_peered":"2024-09-17T07:17:29.766665+0000","last_clean":"2024-09-17T07:17:29.766665+0000","last_became_active":"2024-09-17T07:09:24.396368+0000","last_became_peered":"2024-09-17T07:09:24.396368+0000","last_unstale":"2024-09-17T07:17:29.766665+0000","last_undegraded":"2024-09-17T07:17:29.766665+0000","last_fullsized":"2024-09-17T07:17:29.766665+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'8","last_scrub_stamp":"2024-09-17T07:16:32.727780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:16:32.727780+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:45.467928+0000","scrub_duration":0.0051147279999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":348,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-09-17T07:17:29.771634+0000","last_change":"2024-09-17T07:16:37.093569+0000","last_active":"2024-09-17T07:17:29.771634+0000","last_peered":"2024-09-17T07:17:29.771634+0000","last_clean":"2024-09-17T07:17:29.771634+0000","last_became_active":"2024-09-17T07:02:12.008898+0000","last_became_peered":"2024-09-17T07:02:12.008898+0000","last_unstale":"2024-09-17T07:17:29.771634+0000","last_undegraded":"2024-09-17T07:17:29.771634+0000","last_fullsized":"2024-09-17T07:17:29.771634+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:37.093460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:16:37.093460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:55.997553+0000","scrub_duration":0.0048915060000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":345,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-09-17T07:17:29.772676+0000","last_change":"2024-09-17T07:15:54.554064+0000","last_active":"2024-09-17T07:17:29.772676+0000","last_peered":"2024-09-17T07:17:29.772676+0000","last_clean":"2024-09-17T07:17:29.772676+0000","last_became_active":"2024-09-17T07:02:16.014982+0000","last_became_peered":"2024-09-17T07:02:16.014982+0000","last_unstale":"2024-09-17T07:17:29.772676+0000","last_undegraded":"2024-09-17T07:17:29.772676+0000","last_fullsized":"2024-09-17T07:17:29.772676+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:54.553970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:54.553970+0000","last_clean_scrub_stamp":"2024-09-17T07:15:54.553970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0012697520000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":331,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-09-17T07:17:29.760375+0000","last_change":"2024-09-17T07:17:27.977910+0000","last_active":"2024-09-17T07:17:29.760375+0000","last_peered":"2024-09-17T07:17:29.760375+0000","last_clean":"2024-09-17T07:17:29.760375+0000","last_became_active":"2024-09-17T07:02:18.021014+0000","last_became_peered":"2024-09-17T07:02:18.021014+0000","last_unstale":"2024-09-17T07:17:29.760375+0000","last_undegraded":"2024-09-17T07:17:29.760375+0000","last_fullsized":"2024-09-17T07:17:29.760375+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:17:27.977803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:17:27.977803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:29.675833+0000","scrub_duration":0.002005036,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"242'19","reported_seq":221,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-09-17T07:17:29.763696+0000","last_change":"2024-09-17T07:13:45.076965+0000","last_active":"2024-09-17T07:17:29.763696+0000","last_peered":"2024-09-17T07:17:29.763696+0000","last_clean":"2024-09-17T07:17:29.763696+0000","last_became_active":"2024-09-17T07:09:24.404942+0000","last_became_peered":"2024-09-17T07:09:24.404942+0000","last_unstale":"2024-09-17T07:17:29.763696+0000","last_undegraded":"2024-09-17T07:17:29.763696+0000","last_fullsized":"2024-09-17T07:17:29.763696+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'19","last_scrub_stamp":"2024-09-17T07:13:45.076322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:13:45.076322+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0088383200000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":359,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-09-17T07:17:29.773805+0000","last_change":"2024-09-17T07:16:32.647214+0000","last_active":"2024-09-17T07:17:29.773805+0000","last_peered":"2024-09-17T07:17:29.773805+0000","last_clean":"2024-09-17T07:17:29.773805+0000","last_became_active":"2024-09-17T07:02:12.010195+0000","last_became_peered":"2024-09-17T07:02:12.010195+0000","last_unstale":"2024-09-17T07:17:29.773805+0000","last_undegraded":"2024-09-17T07:17:29.773805+0000","last_fullsized":"2024-09-17T07:17:29.773805+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:32.647069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:16:32.647069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:02.148267+0000","scrub_duration":0.011908693,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":333,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-09-17T07:17:29.760180+0000","last_change":"2024-09-17T07:17:22.916134+0000","last_active":"2024-09-17T07:17:29.760180+0000","last_peered":"2024-09-17T07:17:29.760180+0000","last_clean":"2024-09-17T07:17:29.760180+0000","last_became_active":"2024-09-17T07:02:16.016183+0000","last_became_peered":"2024-09-17T07:02:16.016183+0000","last_unstale":"2024-09-17T07:17:29.760180+0000","last_undegraded":"2024-09-17T07:17:29.760180+0000","last_fullsized":"2024-09-17T07:17:29.760180+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:17:22.915543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:17:22.915543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:39.898743+0000","scrub_duration":0.0086489989999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":188,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-17T07:17:34.551521+0000","last_change":"2024-09-17T07:17:34.551610+0000","last_active":"2024-09-17T07:17:34.551521+0000","last_peered":"2024-09-17T07:17:34.551521+0000","last_clean":"2024-09-17T07:17:34.551521+0000","last_became_active":"2024-09-17T07:09:24.392216+0000","last_became_peered":"2024-09-17T07:09:24.392216+0000","last_unstale":"2024-09-17T07:17:34.551521+0000","last_undegraded":"2024-09-17T07:17:34.551521+0000","last_fullsized":"2024-09-17T07:17:34.551521+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:17:34.551498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:17:34.551498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:38.895494+0000","scrub_duration":0.002777912,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1c","version":"75'1","reported_seq":321,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-09-17T07:17:29.765804+0000","last_change":"2024-09-17T07:13:56.976301+0000","last_active":"2024-09-17T07:17:29.765804+0000","last_peered":"2024-09-17T07:17:29.765804+0000","last_clean":"2024-09-17T07:17:29.765804+0000","last_became_active":"2024-09-17T07:02:18.021855+0000","last_became_peered":"2024-09-17T07:02:18.021855+0000","last_unstale":"2024-09-17T07:17:29.765804+0000","last_undegraded":"2024-09-17T07:17:29.765804+0000","last_fullsized":"2024-09-17T07:17:29.765804+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-09-17T07:13:56.976159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:13:56.976159+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0053255050000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":349,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-09-17T07:17:29.766173+0000","last_change":"2024-09-17T07:16:39.053518+0000","last_active":"2024-09-17T07:17:29.766173+0000","last_peered":"2024-09-17T07:17:29.766173+0000","last_clean":"2024-09-17T07:17:29.766173+0000","last_became_active":"2024-09-17T07:02:12.005329+0000","last_became_peered":"2024-09-17T07:02:12.005329+0000","last_unstale":"2024-09-17T07:17:29.766173+0000","last_undegraded":"2024-09-17T07:17:29.766173+0000","last_fullsized":"2024-09-17T07:17:29.766173+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:39.053386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:16:39.053386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:56.666430+0000","scrub_duration":0.0044426509999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"294'25","reported_seq":414,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-09-17T07:17:29.761151+0000","last_change":"2024-09-17T07:16:00.471391+0000","last_active":"2024-09-17T07:17:29.761151+0000","last_peered":"2024-09-17T07:17:29.761151+0000","last_clean":"2024-09-17T07:17:29.761151+0000","last_became_active":"2024-09-17T07:10:49.730948+0000","last_became_peered":"2024-09-17T07:10:49.730948+0000","last_unstale":"2024-09-17T07:17:29.761151+0000","last_undegraded":"2024-09-17T07:17:29.761151+0000","last_fullsized":"2024-09-17T07:17:29.761151+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'25","last_scrub_stamp":"2024-09-17T07:16:00.471311+0000","last_deep_scrub":"294'25","last_deep_scrub_stamp":"2024-09-17T07:16:00.471311+0000","last_clean_scrub_stamp":"2024-09-17T07:16:00.471311+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040525370000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"75'8","reported_seq":552,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-09-17T07:17:30.706066+0000","last_change":"2024-09-17T07:17:25.451002+0000","last_active":"2024-09-17T07:17:30.706066+0000","last_peered":"2024-09-17T07:17:30.706066+0000","last_clean":"2024-09-17T07:17:30.706066+0000","last_became_active":"2024-09-17T07:02:16.020560+0000","last_became_peered":"2024-09-17T07:02:16.020560+0000","last_unstale":"2024-09-17T07:17:30.706066+0000","last_undegraded":"2024-09-17T07:17:30.706066+0000","last_fullsized":"2024-09-17T07:17:30.706066+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:17:25.450919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:17:25.450919+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:53.609078+0000","scrub_duration":0.0013309369999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":30,"reported_epoch":324,"state":"active+undersized","last_fresh":"2024-09-17T07:17:29.766955+0000","last_change":"2024-09-17T07:16:17.375358+0000","last_active":"2024-09-17T07:17:29.766955+0000","last_peered":"2024-09-17T07:17:29.766955+0000","last_clean":"2024-09-17T07:02:17.002904+0000","last_became_active":"2024-09-17T07:16:17.375358+0000","last_became_peered":"2024-09-17T07:16:17.375358+0000","last_unstale":"2024-09-17T07:17:29.766955+0000","last_undegraded":"2024-09-17T07:17:29.766955+0000","last_fullsized":"2024-09-17T07:16:17.368647+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:37.885570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:15:37.885570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":199,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-09-17T07:17:29.772845+0000","last_change":"2024-09-17T07:17:16.329912+0000","last_active":"2024-09-17T07:17:29.772845+0000","last_peered":"2024-09-17T07:17:29.772845+0000","last_clean":"2024-09-17T07:17:29.772845+0000","last_became_active":"2024-09-17T07:09:24.403360+0000","last_became_peered":"2024-09-17T07:09:24.403360+0000","last_unstale":"2024-09-17T07:17:29.772845+0000","last_undegraded":"2024-09-17T07:17:29.772845+0000","last_fullsized":"2024-09-17T07:17:29.772845+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:17:16.329797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:10:29.325664+0000","last_clean_scrub_stamp":"2024-09-17T07:17:16.329797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:28.864230+0000","scrub_duration":0.0033990230000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"242'19","reported_seq":431,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-09-17T07:17:29.773727+0000","last_change":"2024-09-17T07:17:17.376514+0000","last_active":"2024-09-17T07:17:29.773727+0000","last_peered":"2024-09-17T07:17:29.773727+0000","last_clean":"2024-09-17T07:17:29.773727+0000","last_became_active":"2024-09-17T07:03:48.760390+0000","last_became_peered":"2024-09-17T07:03:48.760390+0000","last_unstale":"2024-09-17T07:17:29.773727+0000","last_undegraded":"2024-09-17T07:17:29.773727+0000","last_fullsized":"2024-09-17T07:17:29.773727+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'19","last_scrub_stamp":"2024-09-17T07:17:17.375166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:17:17.375166+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:41.472827+0000","scrub_duration":0.014344902,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":335,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-17T07:17:31.538827+0000","last_change":"2024-09-17T07:17:31.538898+0000","last_active":"2024-09-17T07:17:31.538827+0000","last_peered":"2024-09-17T07:17:31.538827+0000","last_clean":"2024-09-17T07:17:31.538827+0000","last_became_active":"2024-09-17T07:02:16.019701+0000","last_became_peered":"2024-09-17T07:02:16.019701+0000","last_unstale":"2024-09-17T07:17:31.538827+0000","last_undegraded":"2024-09-17T07:17:31.538827+0000","last_fullsized":"2024-09-17T07:17:31.538827+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:17:31.538812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:17:31.538812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:46.566309+0000","scrub_duration":0.0014632390000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":334,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-09-17T07:17:29.777121+0000","last_change":"2024-09-17T07:15:57.151512+0000","last_active":"2024-09-17T07:17:29.777121+0000","last_peered":"2024-09-17T07:17:29.777121+0000","last_clean":"2024-09-17T07:17:29.777121+0000","last_became_active":"2024-09-17T07:02:18.363752+0000","last_became_peered":"2024-09-17T07:02:18.363752+0000","last_unstale":"2024-09-17T07:17:29.777121+0000","last_undegraded":"2024-09-17T07:17:29.777121+0000","last_fullsized":"2024-09-17T07:17:29.777121+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:57.151431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:15:57.151431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0012311939999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":389,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-09-17T07:17:29.773825+0000","last_change":"2024-09-17T07:16:44.559141+0000","last_active":"2024-09-17T07:17:29.773825+0000","last_peered":"2024-09-17T07:17:29.773825+0000","last_clean":"2024-09-17T07:17:29.773825+0000","last_became_active":"2024-09-17T07:03:48.759384+0000","last_became_peered":"2024-09-17T07:03:48.759384+0000","last_unstale":"2024-09-17T07:17:29.773825+0000","last_undegraded":"2024-09-17T07:17:29.773825+0000","last_fullsized":"2024-09-17T07:17:29.773825+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:44.558901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:11:18.559877+0000","last_clean_scrub_stamp":"2024-09-17T07:16:44.558901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:53.237416+0000","scrub_duration":0.031473045999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"242'27","reported_seq":300,"reported_epoch":324,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:17:29.760959+0000","last_change":"2024-09-17T07:16:17.368008+0000","last_active":"2024-09-17T07:17:29.760959+0000","last_peered":"2024-09-17T07:17:29.760959+0000","last_clean":"2024-09-17T07:12:20.337498+0000","last_became_active":"2024-09-17T07:16:17.368008+0000","last_became_peered":"2024-09-17T07:16:17.368008+0000","last_unstale":"2024-09-17T07:17:29.760959+0000","last_undegraded":"2024-09-17T07:16:17.365648+0000","last_fullsized":"2024-09-17T07:16:17.365550+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'27","last_scrub_stamp":"2024-09-17T07:15:45.855430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:15:45.855430+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.017595989999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":7}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":30,"reported_epoch":324,"state":"active+undersized","last_fresh":"2024-09-17T07:17:29.770173+0000","last_change":"2024-09-17T07:16:17.371494+0000","last_active":"2024-09-17T07:17:29.770173+0000","last_peered":"2024-09-17T07:17:29.770173+0000","last_clean":"2024-09-17T07:02:14.999435+0000","last_became_active":"2024-09-17T07:16:17.371494+0000","last_became_peered":"2024-09-17T07:16:17.371494+0000","last_unstale":"2024-09-17T07:17:29.770173+0000","last_undegraded":"2024-09-17T07:17:29.770173+0000","last_fullsized":"2024-09-17T07:16:17.366463+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:00.509283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:25.794418+0000","last_clean_scrub_stamp":"2024-09-17T07:14:00.509283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":195,"reported_epoch":324,"state":"active+undersized","last_fresh":"2024-09-17T07:17:29.773409+0000","last_change":"2024-09-17T07:16:17.377951+0000","last_active":"2024-09-17T07:17:29.773409+0000","last_peered":"2024-09-17T07:17:29.773409+0000","last_clean":"2024-09-17T07:16:14.246741+0000","last_became_active":"2024-09-17T07:16:17.377951+0000","last_became_peered":"2024-09-17T07:16:17.377951+0000","last_unstale":"2024-09-17T07:17:29.773409+0000","last_undegraded":"2024-09-17T07:17:29.773409+0000","last_fullsized":"2024-09-17T07:16:17.364654+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:02.473467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:15:02.473467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001733163,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":179,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-09-17T07:17:29.760913+0000","last_change":"2024-09-17T07:16:31.651379+0000","last_active":"2024-09-17T07:17:29.760913+0000","last_peered":"2024-09-17T07:17:29.760913+0000","last_clean":"2024-09-17T07:17:29.760913+0000","last_became_active":"2024-09-17T07:09:24.404795+0000","last_became_peered":"2024-09-17T07:09:24.404795+0000","last_unstale":"2024-09-17T07:17:29.760913+0000","last_undegraded":"2024-09-17T07:17:29.760913+0000","last_fullsized":"2024-09-17T07:17:29.760913+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:31.651247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:16:31.651247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:17:33.630557+0000","scrub_duration":0.0075767350000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"242'23","reported_seq":162,"reported_epoch":324,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:17:29.770282+0000","last_change":"2024-09-17T07:16:17.380972+0000","last_active":"2024-09-17T07:17:29.770282+0000","last_peered":"2024-09-17T07:17:29.770282+0000","last_clean":"2024-09-17T07:12:20.742449+0000","last_became_active":"2024-09-17T07:16:17.380972+0000","last_became_peered":"2024-09-17T07:16:17.380972+0000","last_unstale":"2024-09-17T07:17:29.770282+0000","last_undegraded":"2024-09-17T07:16:17.366757+0000","last_fullsized":"2024-09-17T07:16:17.366645+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'23","last_scrub_stamp":"2024-09-17T07:14:58.689120+0000","last_deep_scrub":"86'17","last_deep_scrub_stamp":"2024-09-17T07:03:18.241067+0000","last_clean_scrub_stamp":"2024-09-17T07:14:58.689120+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":355,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-09-17T07:17:29.774743+0000","last_change":"2024-09-17T07:17:17.309054+0000","last_active":"2024-09-17T07:17:29.774743+0000","last_peered":"2024-09-17T07:17:29.774743+0000","last_clean":"2024-09-17T07:17:29.774743+0000","last_became_active":"2024-09-17T07:09:24.393329+0000","last_became_peered":"2024-09-17T07:09:24.393329+0000","last_unstale":"2024-09-17T07:17:29.774743+0000","last_undegraded":"2024-09-17T07:17:29.774743+0000","last_fullsized":"2024-09-17T07:17:29.774743+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:17:17.308762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:17:17.308762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:25.154038+0000","scrub_duration":0.023046534,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":7190413312,"num_objects":109717,"num_object_clones":0,"num_object_copies":329151,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27357,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":109717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109717,"num_write_kb":7021888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19787350016,"data_stored":19787350016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76357,"ondisk_log_size":76357,"up":44,"acting":44,"num_store_stats":7},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":83,"acting":83,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73,"ondisk_log_size":73,"up":81,"acting":81,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":101,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1113,"num_read_kb":870,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":109,"num_bytes_recovered":2228,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":13004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":733,"ondisk_log_size":733,"up":78,"acting":78,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":77,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":8,"acting":8,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1386,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":12,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":4578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":83,"acting":83,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037950,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2857136,"kb_used_data":2186148,"kb_used_omap":1,"kb_used_meta":670974,"kb_avail":90888016,"statfs":{"total":95995035648,"available":93069328384,"internally_reserved":0,"allocated":2238615552,"data_stored":2238079961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":687077835},"hb_peers":[0,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:17:30 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":1.0429999999999999,"15min":1.603},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.192},"max":{"1min":3.194,"5min":8.0800000000000001,"15min":52.478999999999999},"last":2.3730000000000002},{"interface":"front","average":{"1min":0.73899999999999999,"5min":1.079,"15min":1.728},"min":{"1min":0.29399999999999998,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":2.1499999999999999,"5min":6.8120000000000003,"15min":52.871000000000002},"last":0.35399999999999998}]},{"osd":1,"last update":"Tue Sep 17 07:15:24 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":1.4510000000000001,"15min":1.3859999999999999},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.223},"max":{"1min":1.573,"5min":15.801,"15min":73.549000000000007},"last":0.65300000000000002},{"interface":"front","average":{"1min":1.3779999999999999,"5min":1.2969999999999999,"15min":1.383},"min":{"1min":0.29699999999999999,"5min":0.26200000000000001,"15min":0.24099999999999999},"max":{"1min":4.7359999999999998,"5min":13.989000000000001,"15min":73.116},"last":0.71499999999999997}]},{"osd":2,"last update":"Tue Sep 17 07:17:30 2024","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.1359999999999999,"15min":1.1830000000000001},"min":{"1min":0.25,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4.0490000000000004,"5min":6.4900000000000002,"15min":52.055999999999997},"last":0.40000000000000002},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.161,"15min":1.1719999999999999},"min":{"1min":0.23200000000000001,"5min":0.224,"15min":0.224},"max":{"1min":8.5869999999999997,"5min":8.5869999999999997,"15min":52.064999999999998},"last":2.4670000000000001}]},{"osd":3,"last update":"Tue Sep 17 07:17:30 2024","interfaces":[{"interface":"back","average":{"1min":1.2669999999999999,"5min":1.1890000000000001,"15min":1.3759999999999999},"min":{"1min":0.30299999999999999,"5min":0.29199999999999998,"15min":0.28799999999999998},"max":{"1min":4.0970000000000004,"5min":6.4640000000000004,"15min":51.689},"last":0.42399999999999999},{"interface":"front","average":{"1min":1.214,"5min":1.2470000000000001,"15min":1.482},"min":{"1min":0.35399999999999998,"5min":0.27000000000000002,"15min":0.26500000000000001},"max":{"1min":8.5340000000000007,"5min":8.5340000000000007,"15min":52.145000000000003},"last":2.504}]},{"osd":4,"last update":"Tue Sep 17 07:17:30 2024","interfaces":[{"interface":"back","average":{"1min":2.4350000000000001,"5min":1.49,"15min":0.89200000000000002},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":21.140999999999998,"5min":21.140999999999998,"15min":21.140999999999998},"last":1.3160000000000001},{"interface":"front","average":{"1min":3.3380000000000001,"5min":1.694,"15min":1.004},"min":{"1min":0.35299999999999998,"5min":0.28699999999999998,"15min":0.24399999999999999},"max":{"1min":19.420000000000002,"5min":19.420000000000002,"15min":19.420000000000002},"last":1.3500000000000001}]},{"osd":5,"last update":"Tue Sep 17 07:17:30 2024","interfaces":[{"interface":"back","average":{"1min":2.2029999999999998,"5min":1.9330000000000001,"15min":1.079},"min":{"1min":0.28799999999999998,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":19.384,"5min":29.428999999999998,"15min":29.428999999999998},"last":2.4209999999999998},{"interface":"front","average":{"1min":1.794,"5min":2.0939999999999999,"15min":1.135},"min":{"1min":0.33500000000000002,"5min":0.26000000000000001,"15min":0.25700000000000001},"max":{"1min":19.483000000000001,"5min":30.059999999999999,"15min":30.059999999999999},"last":0.314}]},{"osd":6,"last update":"Tue Sep 17 07:17:30 2024","interfaces":[{"interface":"back","average":{"1min":3.2650000000000001,"5min":1.7629999999999999,"15min":1.137},"min":{"1min":0.40100000000000002,"5min":0.32200000000000001,"15min":0.25900000000000001},"max":{"1min":22.221,"5min":22.221,"15min":25.363},"last":3.2930000000000001},{"interface":"front","average":{"1min":4.1269999999999998,"5min":1.9330000000000001,"15min":1.194},"min":{"1min":0.35999999999999999,"5min":0.28399999999999997,"15min":0.27500000000000002},"max":{"1min":22.202000000000002,"5min":22.202000000000002,"15min":23.571000000000002},"last":2.2320000000000002}]}]},{"osd":6,"up_from":50,"seq":214748364993,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3520200,"kb_used_data":3069888,"kb_used_omap":1,"kb_used_meta":450302,"kb_avail":90224952,"statfs":{"total":95995035648,"available":92390350848,"internally_reserved":0,"allocated":3143565312,"data_stored":3143001666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":461109703},"hb_peers":[0,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:17:11 2024","interfaces":[{"interface":"back","average":{"1min":1.3149999999999999,"5min":1.8720000000000001,"15min":2.0449999999999999},"min":{"1min":0.33700000000000002,"5min":0.20200000000000001,"15min":0.185},"max":{"1min":3.9359999999999999,"5min":126.902,"15min":126.902},"last":2.286},{"interface":"front","average":{"1min":0.61899999999999999,"5min":1.744,"15min":1.964},"min":{"1min":0.22,"5min":0.20899999999999999,"15min":0.20799999999999999},"max":{"1min":1.7130000000000001,"5min":127.06399999999999,"15min":127.06399999999999},"last":0.28799999999999998}]},{"osd":1,"last update":"Tue Sep 17 07:16:10 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.996,"15min":0.84399999999999997},"min":{"1min":0.23799999999999999,"5min":0.22900000000000001,"15min":0.22600000000000001},"max":{"1min":2.2999999999999998,"5min":14.720000000000001,"15min":14.720000000000001},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.622,"5min":1.0109999999999999,"15min":0.88800000000000001},"min":{"1min":0.29099999999999998,"5min":0.23400000000000001,"15min":0.22900000000000001},"max":{"1min":2.323,"5min":14.699,"15min":14.699},"last":0.46600000000000003}]},{"osd":2,"last update":"Tue Sep 17 07:17:11 2024","interfaces":[{"interface":"back","average":{"1min":1.3700000000000001,"5min":0.83099999999999996,"15min":0.90500000000000003},"min":{"1min":0.316,"5min":0.219,"15min":0.219},"max":{"1min":4.0590000000000002,"5min":4.9039999999999999,"15min":33.079000000000001},"last":2.4540000000000002},{"interface":"front","average":{"1min":0.75,"5min":0.754,"15min":0.86899999999999999},"min":{"1min":0.31,"5min":0.25600000000000001,"15min":0.23000000000000001},"max":{"1min":2.5630000000000002,"5min":4.8399999999999999,"15min":33.100000000000001},"last":1.8}]},{"osd":3,"last update":"Tue Sep 17 07:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.70799999999999996,"15min":1.1839999999999999},"min":{"1min":0.27000000000000002,"5min":0.26800000000000002,"15min":0.23200000000000001},"max":{"1min":3.2200000000000002,"5min":3.2200000000000002,"15min":52.944000000000003},"last":2.1499999999999999},{"interface":"front","average":{"1min":1.694,"5min":1.032,"15min":1.292},"min":{"1min":0.35399999999999998,"5min":0.28599999999999998,"15min":0.27700000000000002},"max":{"1min":4.9980000000000002,"5min":5.7690000000000001,"15min":52.747},"last":2.4830000000000001}]},{"osd":4,"last update":"Tue Sep 17 07:17:11 2024","interfaces":[{"interface":"back","average":{"1min":1.2669999999999999,"5min":1.2070000000000001,"15min":2.0230000000000001},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":4.2619999999999996,"5min":7.1050000000000004,"15min":260.77199999999999},"last":1.9530000000000001},{"interface":"front","average":{"1min":1.1319999999999999,"5min":1.0129999999999999,"15min":1.992},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.19800000000000001},"max":{"1min":3.968,"5min":4.5449999999999999,"15min":260.69099999999997},"last":3.6739999999999999}]},{"osd":5,"last update":"Tue Sep 17 07:17:11 2024","interfaces":[{"interface":"back","average":{"1min":2.1669999999999998,"5min":1.726,"15min":1.113},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":9.0649999999999995,"5min":11.488,"15min":32.042000000000002},"last":1.9319999999999999},{"interface":"front","average":{"1min":1.9870000000000001,"5min":1.49,"15min":1.1020000000000001},"min":{"1min":0.30499999999999999,"5min":0.27200000000000002,"15min":0.251},"max":{"1min":9.4800000000000004,"5min":10.244999999999999,"15min":31.852},"last":2.0910000000000002}]},{"osd":7,"last update":"Tue Sep 17 07:16:35 2024","interfaces":[{"interface":"back","average":{"1min":7.258,"5min":2.6080000000000001,"15min":1.3700000000000001},"min":{"1min":0.246,"5min":0.246,"15min":0.23699999999999999},"max":{"1min":131.95699999999999,"5min":131.95699999999999,"15min":131.95699999999999},"last":2.2130000000000001},{"interface":"front","average":{"1min":7.2329999999999997,"5min":2.5880000000000001,"15min":1.3560000000000001},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26800000000000002},"max":{"1min":131.833,"5min":131.833,"15min":131.833},"last":1.9079999999999999}]}]},{"osd":5,"up_from":42,"seq":180388626630,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2798700,"kb_used_data":2172004,"kb_used_omap":1,"kb_used_meta":626686,"kb_avail":90946452,"statfs":{"total":95995035648,"available":93129166848,"internally_reserved":0,"allocated":2224132096,"data_stored":2223596293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":641726918},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,5,4,10,11,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:16:52 2024","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":1.5880000000000001},"min":{"1min":0.30399999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":2.8980000000000001,"5min":4.2329999999999997,"15min":49.265000000000001},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.54800000000000004,"5min":1.0049999999999999,"15min":1.6719999999999999},"min":{"1min":0.26300000000000001,"5min":0.221,"15min":0.219},"max":{"1min":1.7090000000000001,"5min":6.7149999999999999,"15min":47.350000000000001},"last":2.661}]},{"osd":1,"last update":"Tue Sep 17 07:15:47 2024","interfaces":[{"interface":"back","average":{"1min":1.4319999999999999,"5min":1.0509999999999999,"15min":0.90100000000000002},"min":{"1min":0.30499999999999999,"5min":0.22,"15min":0.22},"max":{"1min":13.711,"5min":13.711,"15min":20.329000000000001},"last":0.437},{"interface":"front","average":{"1min":0.99399999999999999,"5min":1.3320000000000001,"15min":1.149},"min":{"1min":0.32000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4.617,"5min":6.7560000000000002,"15min":20.408999999999999},"last":0.42199999999999999}]},{"osd":2,"last update":"Tue Sep 17 07:16:52 2024","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.97099999999999997,"15min":1.099},"min":{"1min":0.255,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":2.8730000000000002,"5min":6.3419999999999996,"15min":20.486000000000001},"last":0.308},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.85899999999999999,"15min":0.99399999999999999},"min":{"1min":0.25900000000000001,"5min":0.25800000000000001,"15min":0.23200000000000001},"max":{"1min":2.875,"5min":4.3079999999999998,"15min":23.925999999999998},"last":0.374}]},{"osd":3,"last update":"Tue Sep 17 07:16:52 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":1.2490000000000001,"15min":1.4490000000000001},"min":{"1min":0.30299999999999999,"5min":0.27100000000000002,"15min":0.247},"max":{"1min":2.9140000000000001,"5min":13.882,"15min":68.978999999999999},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.88900000000000001,"15min":1.3},"min":{"1min":0.34599999999999997,"5min":0.29999999999999999,"15min":0.253},"max":{"1min":2.8399999999999999,"5min":4.327,"15min":69.084000000000003},"last":0.34799999999999998}]},{"osd":4,"last update":"Tue Sep 17 07:16:52 2024","interfaces":[{"interface":"back","average":{"1min":1.101,"5min":1.5580000000000001,"15min":1.54},"min":{"1min":0.254,"5min":0.23200000000000001,"15min":0.22},"max":{"1min":7.1950000000000003,"5min":14.036,"15min":142.327},"last":3.4089999999999998},{"interface":"front","average":{"1min":1.296,"5min":1.3999999999999999,"15min":1.4610000000000001},"min":{"1min":0.32700000000000001,"5min":0.25,"15min":0.20999999999999999},"max":{"1min":5.1269999999999998,"5min":13.478999999999999,"15min":142.13200000000001},"last":0.432}]},{"osd":6,"last update":"Tue Sep 17 07:17:07 2024","interfaces":[{"interface":"back","average":{"1min":1.4099999999999999,"5min":1.117,"15min":0.84799999999999998},"min":{"1min":0.29799999999999999,"5min":0.27600000000000002,"15min":0.23200000000000001},"max":{"1min":5.4649999999999999,"5min":6.8419999999999996,"15min":8.3149999999999995},"last":3.1749999999999998},{"interface":"front","average":{"1min":1.75,"5min":1.2889999999999999,"15min":0.92000000000000004},"min":{"1min":0.36799999999999999,"5min":0.26700000000000002,"15min":0.24099999999999999},"max":{"1min":8.4489999999999998,"5min":8.4489999999999998,"15min":8.4489999999999998},"last":1.4690000000000001}]},{"osd":7,"last update":"Tue Sep 17 07:17:30 2024","interfaces":[{"interface":"back","average":{"1min":1.129,"5min":1.631,"15min":1.0800000000000001},"min":{"1min":0.28399999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":2.9409999999999998,"5min":6.8869999999999996,"15min":6.8869999999999996},"last":0.60499999999999998},{"interface":"front","average":{"1min":1.3029999999999999,"5min":1.3480000000000001,"15min":0.92300000000000004},"min":{"1min":0.34200000000000003,"5min":0.28100000000000003,"15min":0.27100000000000002},"max":{"1min":3.7610000000000001,"5min":4.9260000000000002,"15min":5.641},"last":0.32500000000000001}]}]},{"osd":4,"up_from":33,"seq":141733920972,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4974128,"kb_used_data":4361384,"kb_used_omap":1,"kb_used_meta":612734,"kb_avail":88771024,"statfs":{"total":95995035648,"available":90901528576,"internally_reserved":0,"allocated":4466057216,"data_stored":4465516417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":627440070},"hb_peers":[0,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,2,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:17:15 2024","interfaces":[{"interface":"back","average":{"1min":1.1699999999999999,"5min":1.1839999999999999,"15min":2.0550000000000002},"min":{"1min":0.23000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":5.8239999999999998,"5min":18.434999999999999,"15min":74.551000000000002},"last":0.48399999999999999},{"interface":"front","average":{"1min":2.5219999999999998,"5min":1.3149999999999999,"15min":2.2440000000000002},"min":{"1min":0.28599999999999998,"5min":0.22700000000000001,"15min":0.20699999999999999},"max":{"1min":16.619,"5min":18.521000000000001,"15min":74.778999999999996},"last":2.242}]},{"osd":1,"last update":"Tue Sep 17 07:16:11 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":1.2070000000000001,"15min":1.4850000000000001},"min":{"1min":0.252,"5min":0.24399999999999999,"15min":0.23000000000000001},"max":{"1min":1.736,"5min":18.661000000000001,"15min":54.914999999999999},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.71499999999999997,"5min":1.3,"15min":1.444},"min":{"1min":0.246,"5min":0.23300000000000001,"15min":0.22800000000000001},"max":{"1min":3.7770000000000001,"5min":21.931999999999999,"15min":53.725000000000001},"last":0.443}]},{"osd":2,"last update":"Tue Sep 17 07:17:15 2024","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.0669999999999999,"15min":1.1000000000000001},"min":{"1min":0.26700000000000002,"5min":0.25700000000000001,"15min":0.23799999999999999},"max":{"1min":5.7949999999999999,"5min":5.7949999999999999,"15min":49.651000000000003},"last":0.44800000000000001},{"interface":"front","average":{"1min":2.5019999999999998,"5min":1.0489999999999999,"15min":1.1679999999999999},"min":{"1min":0.28699999999999998,"5min":0.27500000000000002,"15min":0.23599999999999999},"max":{"1min":16.655999999999999,"5min":16.655999999999999,"15min":48.813000000000002},"last":2.302}]},{"osd":3,"last update":"Tue Sep 17 07:17:15 2024","interfaces":[{"interface":"back","average":{"1min":1.5760000000000001,"5min":0.89000000000000001,"15min":1.0720000000000001},"min":{"1min":0.317,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":3.831,"5min":3.831,"15min":39.042999999999999},"last":2.274},{"interface":"front","average":{"1min":1.7989999999999999,"5min":1.1060000000000001,"15min":1.1919999999999999},"min":{"1min":0.35599999999999998,"5min":0.28999999999999998,"15min":0.253},"max":{"1min":16.111000000000001,"5min":16.111000000000001,"15min":38.920000000000002},"last":9.0329999999999995}]},{"osd":5,"last update":"Tue Sep 17 07:16:34 2024","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":1.1559999999999999,"15min":0.89000000000000001},"min":{"1min":0.317,"5min":0.28100000000000003,"15min":0.20799999999999999},"max":{"1min":3.6509999999999998,"5min":7.1909999999999998,"15min":11.486000000000001},"last":5.3780000000000001},{"interface":"front","average":{"1min":0.96599999999999997,"5min":1.0720000000000001,"15min":0.90000000000000002},"min":{"1min":0.28999999999999998,"5min":0.27800000000000002,"15min":0.26100000000000001},"max":{"1min":4.5110000000000001,"5min":5.9870000000000001,"15min":11.781000000000001},"last":2.1930000000000001}]},{"osd":6,"last update":"Tue Sep 17 07:17:15 2024","interfaces":[{"interface":"back","average":{"1min":1.8080000000000001,"5min":1.1850000000000001,"15min":1.0069999999999999},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.218},"max":{"1min":7.1420000000000003,"5min":7.1420000000000003,"15min":47.923999999999999},"last":8.9619999999999997},{"interface":"front","average":{"1min":1.885,"5min":1.407,"15min":1.113},"min":{"1min":0.32600000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":5.774,"5min":10.051,"15min":47.600999999999999},"last":0.71899999999999997}]},{"osd":7,"last update":"Tue Sep 17 07:17:31 2024","interfaces":[{"interface":"back","average":{"1min":2.6480000000000001,"5min":1.5529999999999999,"15min":1.173},"min":{"1min":0.318,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":7.1200000000000001,"5min":7.1550000000000002,"15min":48.719000000000001},"last":2.4039999999999999},{"interface":"front","average":{"1min":1.7110000000000001,"5min":1.6970000000000001,"15min":1.2010000000000001},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.28499999999999998},"max":{"1min":9.0030000000000001,"5min":9.0030000000000001,"15min":50.527000000000001},"last":9.0030000000000001}]}]},{"osd":3,"up_from":0,"seq":115964117198,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":18,"seq":77309411540,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3729524,"kb_used_data":3040488,"kb_used_omap":1,"kb_used_meta":689022,"kb_avail":90015628,"statfs":{"total":95995035648,"available":92176003072,"internally_reserved":0,"allocated":3113459712,"data_stored":3112919757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":705558982},"hb_peers":[0,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:16:46 2024","interfaces":[{"interface":"back","average":{"1min":8.0630000000000006,"5min":2.8239999999999998,"15min":1.893},"min":{"1min":0.17000000000000001,"5min":0.159,"15min":0.159},"max":{"1min":131.09800000000001,"5min":131.09800000000001,"15min":131.09800000000001},"last":0.32400000000000001},{"interface":"front","average":{"1min":8.0389999999999997,"5min":2.7879999999999998,"15min":2.0249999999999999},"min":{"1min":0.219,"5min":0.185,"15min":0.185},"max":{"1min":131.24700000000001,"5min":131.24700000000001,"15min":131.24700000000001},"last":0.25}]},{"osd":1,"last update":"Tue Sep 17 07:15:43 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":1.7949999999999999,"15min":1.623},"min":{"1min":0.23699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":1.6459999999999999,"5min":47.101999999999997,"15min":79.659000000000006},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.55000000000000004,"5min":1.6990000000000001,"15min":1.5600000000000001},"min":{"1min":0.22700000000000001,"5min":0.20899999999999999,"15min":0.20200000000000001},"max":{"1min":1.8049999999999999,"5min":45.831000000000003,"15min":80.155000000000001},"last":0.41399999999999998}]},{"osd":3,"last update":"Tue Sep 17 07:17:01 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.55700000000000005,"15min":0.97599999999999998},"min":{"1min":0.23599999999999999,"5min":0.22,"15min":0.20100000000000001},"max":{"1min":0.71299999999999997,"5min":4.6449999999999996,"15min":23.43},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.34699999999999998,"5min":0.432,"15min":1.036},"min":{"1min":0.24199999999999999,"5min":0.23499999999999999,"15min":0.223},"max":{"1min":0.56499999999999995,"5min":1.6739999999999999,"15min":23.408000000000001},"last":0.27700000000000002}]},{"osd":4,"last update":"Tue Sep 17 07:17:27 2024","interfaces":[{"interface":"back","average":{"1min":1.212,"5min":0.94299999999999995,"15min":0.94399999999999995},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":5.8099999999999996,"5min":6.8449999999999998,"15min":6.8449999999999998},"last":1.6890000000000001},{"interface":"front","average":{"1min":1.2030000000000001,"5min":1.0669999999999999,"15min":0.97799999999999998},"min":{"1min":0.32300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":5.7999999999999998,"5min":6.8609999999999998,"15min":6.8609999999999998},"last":1.714}]},{"osd":5,"last update":"Tue Sep 17 07:16:46 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":1.133,"15min":1.1140000000000001},"min":{"1min":0.28199999999999997,"5min":0.26900000000000002,"15min":0.249},"max":{"1min":0.98199999999999998,"5min":15.276999999999999,"15min":15.276999999999999},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.46700000000000003,"5min":1.02,"15min":0.98199999999999998},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.27600000000000002},"max":{"1min":0.73299999999999998,"5min":15.561999999999999,"15min":15.561999999999999},"last":0.309}]},{"osd":6,"last update":"Tue Sep 17 07:17:01 2024","interfaces":[{"interface":"back","average":{"1min":1.8340000000000001,"5min":1.0269999999999999,"15min":1.4099999999999999},"min":{"1min":0.32400000000000001,"5min":0.29499999999999998,"15min":0.28999999999999998},"max":{"1min":20.494,"5min":20.494,"15min":101.587},"last":0.375},{"interface":"front","average":{"1min":2.3809999999999998,"5min":1.03,"15min":1.4590000000000001},"min":{"1min":0.35399999999999998,"5min":0.32300000000000001,"15min":0.28299999999999997},"max":{"1min":26.922000000000001,"5min":26.922000000000001,"15min":100.696},"last":0.48599999999999999}]},{"osd":7,"last update":"Tue Sep 17 07:16:41 2024","interfaces":[{"interface":"back","average":{"1min":5.7519999999999998,"5min":2.1539999999999999,"15min":1.3460000000000001},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.316},"max":{"1min":94.269999999999996,"5min":94.269999999999996,"15min":94.269999999999996},"last":0.36099999999999999},{"interface":"front","average":{"1min":5.8250000000000002,"5min":2.1600000000000001,"15min":1.415},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.32900000000000001},"max":{"1min":94.634,"5min":94.634,"15min":94.634},"last":0.56100000000000005}]}]},{"osd":1,"up_from":13,"seq":55834575049,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":541424,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":540542,"kb_avail":93203728,"statfs":{"total":95995035648,"available":95440617472,"internally_reserved":0,"allocated":884736,"data_stored":439467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":553515462},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:16:00 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.73699999999999999,"15min":1.139},"min":{"1min":0.183,"5min":0.155,"15min":0.155},"max":{"1min":2.6070000000000002,"5min":4.5780000000000003,"15min":22.521000000000001},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.67500000000000004,"15min":1.0369999999999999},"min":{"1min":0.247,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.198,"5min":4.335,"15min":22.471},"last":0.377}]},{"osd":2,"last update":"Tue Sep 17 07:15:44 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.91000000000000003,"15min":1.024},"min":{"1min":0.21299999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":2.6360000000000001,"5min":6.1920000000000002,"15min":11.055999999999999},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.73899999999999999,"15min":0.94799999999999995},"min":{"1min":0.22900000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":1.3160000000000001,"5min":4.1559999999999997,"15min":9.3160000000000007},"last":0.30599999999999999}]},{"osd":3,"last update":"Tue Sep 17 07:16:00 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.67800000000000005,"15min":0.94999999999999996},"min":{"1min":0.27400000000000002,"5min":0.25900000000000001,"15min":0.219},"max":{"1min":2.4409999999999998,"5min":4.1269999999999998,"15min":8.5510000000000002},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.371,"5min":0.47399999999999998,"15min":0.98599999999999999},"min":{"1min":0.247,"5min":0.22500000000000001,"15min":0.19500000000000001},"max":{"1min":0.627,"5min":2.8199999999999998,"15min":19.602},"last":0.35299999999999998}]},{"osd":4,"last update":"Tue Sep 17 07:15:20 2024","interfaces":[{"interface":"back","average":{"1min":1.1339999999999999,"5min":1.4670000000000001,"15min":1.129},"min":{"1min":0.36199999999999999,"5min":0.314,"15min":0.25},"max":{"1min":2.5659999999999998,"5min":19.088999999999999,"15min":19.088999999999999},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.98999999999999999,"5min":1.2010000000000001,"15min":1.1759999999999999},"min":{"1min":0.36799999999999999,"5min":0.317,"15min":0.307},"max":{"1min":2.6459999999999999,"5min":18.959,"15min":27.097999999999999},"last":0.499}]},{"osd":5,"last update":"Tue Sep 17 07:15:44 2024","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.47,"15min":1.2110000000000001},"min":{"1min":0.34799999999999998,"5min":0.28899999999999998,"15min":0.27100000000000002},"max":{"1min":3.6869999999999998,"5min":6.6790000000000003,"15min":16.84},"last":0.51000000000000001},{"interface":"front","average":{"1min":1.1559999999999999,"5min":1.069,"15min":1.169},"min":{"1min":0.29299999999999998,"5min":0.29199999999999998,"15min":0.28899999999999998},"max":{"1min":6.2560000000000002,"5min":6.2560000000000002,"15min":16.949999999999999},"last":0.41299999999999998}]},{"osd":6,"last update":"Tue Sep 17 07:16:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":1.5149999999999999,"15min":1.274},"min":{"1min":0.33100000000000002,"5min":0.315,"15min":0.308},"max":{"1min":1.127,"5min":50.497,"15min":50.497},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.99099999999999999,"5min":1.6379999999999999,"15min":1.222},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.28899999999999998},"max":{"1min":3.593,"5min":50.793999999999997,"15min":50.793999999999997},"last":0.48499999999999999}]},{"osd":7,"last update":"Tue Sep 17 07:15:20 2024","interfaces":[{"interface":"back","average":{"1min":1.655,"5min":1.343,"15min":1.1919999999999999},"min":{"1min":0.435,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":3.6120000000000001,"5min":9.8179999999999996,"15min":9.8179999999999996},"last":0.441},{"interface":"front","average":{"1min":1.464,"5min":1.6319999999999999,"15min":1.2410000000000001},"min":{"1min":0.35099999999999998,"5min":0.317,"15min":0.317},"max":{"1min":4.1219999999999999,"5min":7.5839999999999996,"15min":11.007999999999999},"last":0.57999999999999996}]}]},{"osd":0,"up_from":8,"seq":34359738588,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5148160,"kb_used_data":4427128,"kb_used_omap":1,"kb_used_meta":721022,"kb_avail":88596992,"statfs":{"total":95995035648,"available":90723319808,"internally_reserved":0,"allocated":4533379072,"data_stored":4532823477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":738326982},"hb_peers":[2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 17 07:15:18 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.89700000000000002,"15min":0.89500000000000002},"min":{"1min":0.254,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":1.518,"5min":21.876000000000001,"15min":21.876000000000001},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.56399999999999995,"5min":1.2330000000000001,"15min":1.081},"min":{"1min":0.17899999999999999,"5min":0.158,"15min":0.158},"max":{"1min":2.2599999999999998,"5min":22.521999999999998,"15min":22.521999999999998},"last":0.33700000000000002}]},{"osd":2,"last update":"Tue Sep 17 07:16:42 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.47199999999999998,"15min":1.1060000000000001},"min":{"1min":0.20899999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":0.68200000000000005,"5min":2.3690000000000002,"15min":18.643999999999998},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.32600000000000001,"5min":0.54300000000000004,"15min":1.0089999999999999},"min":{"1min":0.224,"5min":0.189,"15min":0.189},"max":{"1min":0.48899999999999999,"5min":2.3730000000000002,"15min":17.972999999999999},"last":0.76600000000000001}]},{"osd":3,"last update":"Tue Sep 17 07:17:22 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.57799999999999996,"15min":1.016},"min":{"1min":0.23599999999999999,"5min":0.23400000000000001,"15min":0.21199999999999999},"max":{"1min":3.2000000000000002,"5min":5.5880000000000001,"15min":18.629000000000001},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.49299999999999999,"15min":1.0249999999999999},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":3.1190000000000002,"5min":3.1190000000000002,"15min":22.358000000000001},"last":0.28299999999999997}]},{"osd":4,"last update":"Tue Sep 17 07:17:22 2024","interfaces":[{"interface":"back","average":{"1min":1.149,"5min":0.97099999999999997,"15min":1.1619999999999999},"min":{"1min":0.30499999999999999,"5min":0.29799999999999999,"15min":0.29099999999999998},"max":{"1min":5.5990000000000002,"5min":5.5990000000000002,"15min":19.146999999999998},"last":0.33700000000000002},{"interface":"front","average":{"1min":1.1659999999999999,"5min":1.0089999999999999,"15min":1.1299999999999999},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.255},"max":{"1min":5.6100000000000003,"5min":5.6100000000000003,"15min":19.158999999999999},"last":0.98199999999999998}]},{"osd":5,"last update":"Tue Sep 17 07:16:54 2024","interfaces":[{"interface":"back","average":{"1min":14.298999999999999,"5min":3.6680000000000001,"15min":2.0870000000000002},"min":{"1min":0.35699999999999998,"5min":0.35499999999999998,"15min":0.28899999999999998},"max":{"1min":216.83699999999999,"5min":216.83699999999999,"15min":216.83699999999999},"last":0.316},{"interface":"front","average":{"1min":14.337999999999999,"5min":3.6110000000000002,"15min":2.0680000000000001},"min":{"1min":0.34200000000000003,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":216.816,"5min":216.816,"15min":216.816},"last":1.1399999999999999}]},{"osd":6,"last update":"Tue Sep 17 07:17:22 2024","interfaces":[{"interface":"back","average":{"1min":8.0429999999999993,"5min":2.2789999999999999,"15min":1.5249999999999999},"min":{"1min":0.32200000000000001,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":147.13200000000001,"5min":147.13200000000001,"15min":147.13200000000001},"last":0.84399999999999997},{"interface":"front","average":{"1min":7.8419999999999996,"5min":2.2509999999999999,"15min":1.5629999999999999},"min":{"1min":0.32800000000000001,"5min":0.311,"15min":0.311},"max":{"1min":147.172,"5min":147.172,"15min":147.172},"last":0.35699999999999998}]},{"osd":7,"last update":"Tue Sep 17 07:17:22 2024","interfaces":[{"interface":"back","average":{"1min":1.177,"5min":1.351,"15min":1.4379999999999999},"min":{"1min":0.314,"5min":0.30599999999999999,"15min":0.29599999999999999},"max":{"1min":4.484,"5min":5.9509999999999996,"15min":46.183},"last":1.0089999999999999},{"interface":"front","average":{"1min":1.1619999999999999,"5min":1.27,"15min":1.3600000000000001},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32000000000000001},"max":{"1min":5.2149999999999999,"5min":6.1079999999999997,"15min":45.914999999999999},"last":0.88400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4532142080,"data_stored":4532142080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3134521344,"data_stored":3134521344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4488953856,"data_stored":4488953856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2229665792,"data_stored":2229665792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3144941568,"data_stored":3144941568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2257125376,"data_stored":2257125376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T07:17:42.923 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-17T07:17:42.923 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd pool set unique_pool_3 pg_num 26 2024-09-17T07:17:42.941 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:17:42.979 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:42.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:42.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:17:43.081 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:17:43.113 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:43.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:43.114 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:17:43.215 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:17:43.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:43.251 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:43.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:17:43.352 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:17:43.385 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:43.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:43.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:17:43.488 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:17:43.508 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:43 smithi100 ceph-mon[29092]: 5.3 scrub starts 2024-09-17T07:17:43.508 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:43 smithi100 ceph-mon[29092]: 5.3 scrub ok 2024-09-17T07:17:43.508 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:43 smithi100 ceph-mon[29092]: osdmap e329: 8 total, 7 up, 7 in 2024-09-17T07:17:43.508 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:43 smithi100 ceph-mon[29092]: 5.2 scrub starts 2024-09-17T07:17:43.508 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:43 smithi100 ceph-mon[29092]: 5.2 scrub ok 2024-09-17T07:17:43.508 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:43 smithi100 ceph-mon[29092]: 10.e deep-scrub starts 2024-09-17T07:17:43.509 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:43 smithi100 ceph-mon[39462]: 5.3 scrub starts 2024-09-17T07:17:43.509 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:43 smithi100 ceph-mon[39462]: 5.3 scrub ok 2024-09-17T07:17:43.509 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:43 smithi100 ceph-mon[39462]: osdmap e329: 8 total, 7 up, 7 in 2024-09-17T07:17:43.509 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:43 smithi100 ceph-mon[39462]: 5.2 scrub starts 2024-09-17T07:17:43.509 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:43 smithi100 ceph-mon[39462]: 5.2 scrub ok 2024-09-17T07:17:43.509 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:43 smithi100 ceph-mon[39462]: 10.e deep-scrub starts 2024-09-17T07:17:43.522 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:43.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:43.524 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:17:43.537 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:17:43.537 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:17:43.624 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:17:43.632 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 85 16 119875 119859 88.1231 59.6875 0.00555017 0.0113418 2024-09-17T07:17:43.664 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:43.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:43.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:17:43.766 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:17:43.791 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:43.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:43 smithi135 ceph-mon[33238]: 5.3 scrub starts 2024-09-17T07:17:43.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:43 smithi135 ceph-mon[33238]: 5.3 scrub ok 2024-09-17T07:17:43.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:43 smithi135 ceph-mon[33238]: osdmap e329: 8 total, 7 up, 7 in 2024-09-17T07:17:43.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:43 smithi135 ceph-mon[33238]: 5.2 scrub starts 2024-09-17T07:17:43.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:43 smithi135 ceph-mon[33238]: 5.2 scrub ok 2024-09-17T07:17:43.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:43 smithi135 ceph-mon[33238]: 10.e deep-scrub starts 2024-09-17T07:17:43.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:43.796 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:17:43.897 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:17:43.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:43.936 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:43.936 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:17:44.037 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:17:44.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:44.074 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:44.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:17:44.175 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:17:44.203 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-17T07:17:44.210 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:44.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:44.211 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:17:44.312 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:17:44.349 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:44.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:44.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:17:44.451 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:17:44.456 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:44 smithi100 ceph-mon[29092]: pgmap v702: 148 pgs: 1 active+clean+scrubbing+deep, 36 active+undersized, 28 active+undersized+degraded, 83 active+clean; 7.0 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 853 B/s rd, 55 MiB/s wr, 885 op/s; 28809/345672 objects degraded (8.334%) 2024-09-17T07:17:44.456 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:44 smithi100 ceph-mon[29092]: 10.f deep-scrub starts 2024-09-17T07:17:44.457 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:44 smithi100 ceph-mon[39462]: pgmap v702: 148 pgs: 1 active+clean+scrubbing+deep, 36 active+undersized, 28 active+undersized+degraded, 83 active+clean; 7.0 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 853 B/s rd, 55 MiB/s wr, 885 op/s; 28809/345672 objects degraded (8.334%) 2024-09-17T07:17:44.457 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:44 smithi100 ceph-mon[39462]: 10.f deep-scrub starts 2024-09-17T07:17:44.507 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:44.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:44.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:17:44.608 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:17:44.632 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 86 16 120561 120545 87.5969 42.875 0.0143897 0.0114101 2024-09-17T07:17:44.645 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:44.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:44.646 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:17:44.716 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:17:44.747 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:17:44.760 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:17:44.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:44.776 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:44.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:17:44.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:44 smithi135 ceph-mon[33238]: pgmap v702: 148 pgs: 1 active+clean+scrubbing+deep, 36 active+undersized, 28 active+undersized+degraded, 83 active+clean; 7.0 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 853 B/s rd, 55 MiB/s wr, 885 op/s; 28809/345672 objects degraded (8.334%) 2024-09-17T07:17:44.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:44 smithi135 ceph-mon[33238]: 10.f deep-scrub starts 2024-09-17T07:17:44.878 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:17:44.909 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:44.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:44.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:17:45.011 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:17:45.038 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:45.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:45.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:17:45.140 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:17:45.166 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:45.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:45.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:17:45.250 INFO:teuthology.orchestra.run.smithi100.stderr:noscrub is set 2024-09-17T07:17:45.268 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:17:45.299 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:45.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:45.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:17:45.376 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:45 smithi100 ceph-mon[39462]: 4.1e scrub starts 2024-09-17T07:17:45.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:45 smithi100 ceph-mon[39462]: 4.1e scrub ok 2024-09-17T07:17:45.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:45 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/3659241302' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:17:45.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:45 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 28809/345672 objects degraded (8.334%), 28 pgs degraded, 64 pgs undersized (PG_DEGRADED) 2024-09-17T07:17:45.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:45 smithi100 ceph-mon[39462]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:17:45.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:45 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/3659241302' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T07:17:45.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:45 smithi100 ceph-mon[39462]: osdmap e330: 8 total, 7 up, 7 in 2024-09-17T07:17:45.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:45 smithi100 ceph-mon[29092]: 4.1e scrub starts 2024-09-17T07:17:45.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:45 smithi100 ceph-mon[29092]: 4.1e scrub ok 2024-09-17T07:17:45.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:45 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/3659241302' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:17:45.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:45 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 28809/345672 objects degraded (8.334%), 28 pgs degraded, 64 pgs undersized (PG_DEGRADED) 2024-09-17T07:17:45.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:45 smithi100 ceph-mon[29092]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:17:45.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:45 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/3659241302' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T07:17:45.378 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:45 smithi100 ceph-mon[29092]: osdmap e330: 8 total, 7 up, 7 in 2024-09-17T07:17:45.400 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:17:45.422 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:45.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:45.423 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:17:45.523 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:17:45.539 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:17:45.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:45.548 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:45.548 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:17:45.632 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 87 16 121211 121195 87.057 40.625 0.00350573 0.011479 2024-09-17T07:17:45.650 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:17:45.678 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:45.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:45.679 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:17:45.780 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:17:45.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:45 smithi135 ceph-mon[33238]: 4.1e scrub starts 2024-09-17T07:17:45.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:45 smithi135 ceph-mon[33238]: 4.1e scrub ok 2024-09-17T07:17:45.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:45 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/3659241302' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:17:45.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:45 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 28809/345672 objects degraded (8.334%), 28 pgs degraded, 64 pgs undersized (PG_DEGRADED) 2024-09-17T07:17:45.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:45 smithi135 ceph-mon[33238]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:17:45.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:45 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/3659241302' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T07:17:45.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:45 smithi135 ceph-mon[33238]: osdmap e330: 8 total, 7 up, 7 in 2024-09-17T07:17:45.806 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:45.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:45.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:17:45.908 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:17:45.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:45.947 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:45.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:17:46.048 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:17:46.073 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:46.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:46.074 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:17:46.175 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:17:46.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:46.209 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:46.210 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:17:46.310 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:17:46.316 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:17:45 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:17:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:17:46.356 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:46.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:46.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:17:46.458 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:17:46.488 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:46.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:46.489 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:17:46.584 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:46 smithi100 ceph-mon[29092]: pgmap v704: 148 pgs: 3 active+clean+scrubbing+deep, 36 active+undersized, 28 active+undersized+degraded, 81 active+clean; 7.3 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 853 B/s rd, 62 MiB/s wr, 990 op/s; 30040/360552 objects degraded (8.332%) 2024-09-17T07:17:46.584 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:46 smithi100 ceph-mon[39462]: pgmap v704: 148 pgs: 3 active+clean+scrubbing+deep, 36 active+undersized, 28 active+undersized+degraded, 81 active+clean; 7.3 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 853 B/s rd, 62 MiB/s wr, 990 op/s; 30040/360552 objects degraded (8.332%) 2024-09-17T07:17:46.591 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:17:46.620 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:46.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:46.621 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:17:46.632 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 88 16 121941 121925 86.5862 45.625 0.00552317 0.0115406 2024-09-17T07:17:46.721 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:17:46.744 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:46.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:46.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:17:46.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:46 smithi135 ceph-mon[33238]: pgmap v704: 148 pgs: 3 active+clean+scrubbing+deep, 36 active+undersized, 28 active+undersized+degraded, 81 active+clean; 7.3 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 853 B/s rd, 62 MiB/s wr, 990 op/s; 30040/360552 objects degraded (8.332%) 2024-09-17T07:17:46.847 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:17:46.872 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:46.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:46.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:17:46.974 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:17:47.008 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:47.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:47.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:17:47.113 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:17:47.150 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:47.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:47.151 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:17:47.252 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:17:47.282 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:47.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:47.283 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:17:47.384 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:17:47.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:47.421 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:47.421 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:17:47.522 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:17:47.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:47.553 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:47.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:17:47.633 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 89 16 122673 122657 86.1273 45.75 0.00476917 0.0116043 2024-09-17T07:17:47.654 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:17:47.680 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:47.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:47.681 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:17:47.782 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:17:47.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:47.807 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:47.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:17:47.908 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:17:47.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:47.948 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:47.949 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:17:48.049 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:17:48.087 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:48.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:48.088 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:17:48.189 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:17:48.225 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:48.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:48.226 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:17:48.327 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:17:48.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:48.354 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:48.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:17:48.455 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:17:48.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:48.488 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:48.488 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:17:48.517 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:17:48.588 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:17:48.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:48.614 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:48.615 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:17:48.618 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:48 smithi100 ceph-mon[39462]: pgmap v705: 148 pgs: 3 active+clean+scrubbing+deep, 36 active+undersized, 28 active+undersized+degraded, 81 active+clean; 7.3 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 724 B/s rd, 44 MiB/s wr, 702 op/s; 30040/360552 objects degraded (8.332%) 2024-09-17T07:17:48.619 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:48 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:17:48.619 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:48 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:17:48.619 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:48 smithi100 ceph-mon[29092]: pgmap v705: 148 pgs: 3 active+clean+scrubbing+deep, 36 active+undersized, 28 active+undersized+degraded, 81 active+clean; 7.3 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 724 B/s rd, 44 MiB/s wr, 702 op/s; 30040/360552 objects degraded (8.332%) 2024-09-17T07:17:48.619 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:48 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:17:48.619 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:48 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:17:48.632 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 90 5 123345 123340 85.6446 42.6875 0.0134626 0.0116708 2024-09-17T07:17:48.715 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:17:48.747 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:48.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:48.748 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:17:48.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:48 smithi135 ceph-mon[33238]: pgmap v705: 148 pgs: 3 active+clean+scrubbing+deep, 36 active+undersized, 28 active+undersized+degraded, 81 active+clean; 7.3 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 724 B/s rd, 44 MiB/s wr, 702 op/s; 30040/360552 objects degraded (8.332%) 2024-09-17T07:17:48.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:48 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:17:48.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:48 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:17:48.849 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:17:48.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:48.875 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:48.875 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:17:48.977 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:17:49.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:49.005 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:49.005 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:17:49.106 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:17:49.128 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:49.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:49.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:17:49.230 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:17:49.257 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:49.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:49.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:17:49.293 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:17:48 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:17:49.360 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:17:49.390 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:49.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:49.391 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:17:49.492 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:17:49.496 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-17T07:17:49.519 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:49.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:49.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:17:49.621 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:17:49.633 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Total time run: 90.045 2024-09-17T07:17:49.633 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Total writes made: 123345 2024-09-17T07:17:49.633 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Write size: 65536 2024-09-17T07:17:49.633 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Object size: 65536 2024-09-17T07:17:49.633 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Bandwidth (MB/sec): 85.6135 2024-09-17T07:17:49.633 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Stddev Bandwidth: 35.8497 2024-09-17T07:17:49.633 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Max bandwidth (MB/sec): 236.438 2024-09-17T07:17:49.633 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Min bandwidth (MB/sec): 40.625 2024-09-17T07:17:49.633 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Average IOPS: 1369 2024-09-17T07:17:49.634 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Stddev IOPS: 573.595 2024-09-17T07:17:49.634 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Max IOPS: 3783 2024-09-17T07:17:49.634 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Min IOPS: 650 2024-09-17T07:17:49.634 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Average Latency(s): 0.0116733 2024-09-17T07:17:49.634 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Stddev Latency(s): 0.0138065 2024-09-17T07:17:49.634 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Max latency(s): 0.332984 2024-09-17T07:17:49.634 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Min latency(s): 0.00112621 2024-09-17T07:17:49.639 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Cleaning up (deleting benchmark objects) 2024-09-17T07:17:49.651 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:49.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:49.652 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:17:49.752 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:17:49.781 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:49.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:49.782 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:17:49.883 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:17:49.910 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:49.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:49.911 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:17:50.012 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:17:50.037 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:50.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:50.038 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:17:50.139 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:17:50.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:50.197 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:50.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:17:50.298 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:17:50.336 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:50.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:50.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:17:50.342 INFO:teuthology.orchestra.run.smithi100.stderr:set pool 10 pg_num to 26 2024-09-17T07:17:50.437 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:17:50.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:50.476 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:50.476 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:17:50.486 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:50 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:17:50.486 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:50 smithi100 ceph-mon[29092]: pgmap v706: 148 pgs: 3 active+clean+scrubbing+deep, 36 active+undersized, 28 active+undersized+degraded, 81 active+clean; 7.5 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.2 KiB/s rd, 60 MiB/s wr, 959 op/s; 30668/368670 objects degraded (8.319%) 2024-09-17T07:17:50.486 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:50 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2084210645' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]: dispatch 2024-09-17T07:17:50.486 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:50 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]: dispatch 2024-09-17T07:17:50.486 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:50 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 30668/368670 objects degraded (8.319%), 28 pgs degraded, 64 pgs undersized (PG_DEGRADED) 2024-09-17T07:17:50.487 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:50 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:17:50.487 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:50 smithi100 ceph-mon[39462]: pgmap v706: 148 pgs: 3 active+clean+scrubbing+deep, 36 active+undersized, 28 active+undersized+degraded, 81 active+clean; 7.5 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.2 KiB/s rd, 60 MiB/s wr, 959 op/s; 30668/368670 objects degraded (8.319%) 2024-09-17T07:17:50.487 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:50 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2084210645' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]: dispatch 2024-09-17T07:17:50.487 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:50 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]: dispatch 2024-09-17T07:17:50.487 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:50 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 30668/368670 objects degraded (8.319%), 28 pgs degraded, 64 pgs undersized (PG_DEGRADED) 2024-09-17T07:17:50.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:50 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:17:50.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:50 smithi135 ceph-mon[33238]: pgmap v706: 148 pgs: 3 active+clean+scrubbing+deep, 36 active+undersized, 28 active+undersized+degraded, 81 active+clean; 7.5 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.2 KiB/s rd, 60 MiB/s wr, 959 op/s; 30668/368670 objects degraded (8.319%) 2024-09-17T07:17:50.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:50 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2084210645' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]: dispatch 2024-09-17T07:17:50.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:50 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]: dispatch 2024-09-17T07:17:50.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:50 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 30668/368670 objects degraded (8.319%), 28 pgs degraded, 64 pgs undersized (PG_DEGRADED) 2024-09-17T07:17:50.577 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:17:50.615 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:50.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:50.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:17:50.717 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:17:50.757 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:50.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:50.758 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:17:50.859 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:17:50.899 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:50.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:50.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:17:51.001 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:17:51.028 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:51.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:51.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:17:51.130 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:17:51.167 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:51.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:51.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:17:51.269 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:17:51.302 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:51.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:51.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:17:51.405 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:17:51.496 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:51.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:51.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:17:51.598 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:17:51.620 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:51.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:51.621 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:17:51.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:51 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]': finished 2024-09-17T07:17:51.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:51 smithi100 ceph-mon[29092]: osdmap e331: 8 total, 7 up, 7 in 2024-09-17T07:17:51.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:51 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-09-17T07:17:51.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:51 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]': finished 2024-09-17T07:17:51.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:51 smithi100 ceph-mon[39462]: osdmap e331: 8 total, 7 up, 7 in 2024-09-17T07:17:51.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:51 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-09-17T07:17:51.722 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:17:51.747 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:51.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:51.749 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:17:51.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:51 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]': finished 2024-09-17T07:17:51.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:51 smithi135 ceph-mon[33238]: osdmap e331: 8 total, 7 up, 7 in 2024-09-17T07:17:51.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:51 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-09-17T07:17:51.850 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:17:51.875 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:51.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:51.876 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:17:51.977 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:17:51.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:52.000 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:52.000 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:17:52.100 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:17:52.122 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:52.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:52.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:17:52.224 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:17:52.250 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:52.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:52.251 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:17:52.352 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:17:52.383 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:52.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:52.384 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:17:52.391 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:17:52.409 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-17T07:17:52.410 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:17:52.410 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:52 smithi100 ceph-mon[29092]: pgmap v708: 148 pgs: 3 active+clean+scrubbing+deep, 36 active+undersized, 28 active+undersized+degraded, 81 active+clean; 7.5 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.2 KiB/s rd, 64 MiB/s wr, 1.02k op/s; 30828/370239 objects degraded (8.327%) 2024-09-17T07:17:52.410 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:52 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-17T07:17:52.410 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:52 smithi100 ceph-mon[29092]: osdmap e332: 8 total, 7 up, 7 in 2024-09-17T07:17:52.410 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:52 smithi100 ceph-mon[39462]: pgmap v708: 148 pgs: 3 active+clean+scrubbing+deep, 36 active+undersized, 28 active+undersized+degraded, 81 active+clean; 7.5 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.2 KiB/s rd, 64 MiB/s wr, 1.02k op/s; 30828/370239 objects degraded (8.327%) 2024-09-17T07:17:52.410 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:52 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-17T07:17:52.411 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:52 smithi100 ceph-mon[39462]: osdmap e332: 8 total, 7 up, 7 in 2024-09-17T07:17:52.485 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:17:52.521 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:52.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:52.522 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:17:52.623 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:17:52.657 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:52.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:52.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:17:52.758 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:17:52.785 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:52.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:52.786 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:17:52.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:52 smithi135 ceph-mon[33238]: pgmap v708: 148 pgs: 3 active+clean+scrubbing+deep, 36 active+undersized, 28 active+undersized+degraded, 81 active+clean; 7.5 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.2 KiB/s rd, 64 MiB/s wr, 1.02k op/s; 30828/370239 objects degraded (8.327%) 2024-09-17T07:17:52.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:52 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-17T07:17:52.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:52 smithi135 ceph-mon[33238]: osdmap e332: 8 total, 7 up, 7 in 2024-09-17T07:17:52.887 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:17:52.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:52.920 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:52.920 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:17:53.021 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:17:53.047 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:53.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:53.048 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:17:53.077 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-17T07:17:53.149 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:17:53.174 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:53.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:53.175 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:17:53.276 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:17:53.303 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:53.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:53.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:17:53.405 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:17:53.430 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:53.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:53.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:17:53.532 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:17:53.557 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:53.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:53.558 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:17:53.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:53 smithi100 ceph-mon[29092]: osdmap e333: 8 total, 7 up, 7 in 2024-09-17T07:17:53.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:53 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-09-17T07:17:53.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:53 smithi100 ceph-mon[39462]: osdmap e333: 8 total, 7 up, 7 in 2024-09-17T07:17:53.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:53 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-09-17T07:17:53.658 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:17:53.682 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:53.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:53.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:17:53.784 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:17:53.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:53 smithi135 ceph-mon[33238]: osdmap e333: 8 total, 7 up, 7 in 2024-09-17T07:17:53.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:53 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-09-17T07:17:53.810 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:53.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:53.811 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:17:53.912 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:17:53.933 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:53.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:53.934 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:17:54.035 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:17:54.061 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:54.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:54.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:17:54.163 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:17:54.185 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:54.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:54.186 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:17:54.288 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:17:54.313 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:54.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:54.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:17:54.415 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:17:54.437 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:54.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:54.438 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:17:54.539 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:17:54.562 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:54.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:54.563 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:17:54.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:54 smithi100 ceph-mon[39462]: pgmap v711: 158 pgs: 8 unknown, 3 active+clean+scrubbing+deep, 36 active+undersized, 29 active+undersized+degraded, 82 active+clean; 7.5 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 853 B/s rd, 35 MiB/s wr, 639 op/s; 30738/369117 objects degraded (8.327%); 0 B/s, 0 objects/s recovering 2024-09-17T07:17:54.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:54 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-17T07:17:54.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:54 smithi100 ceph-mon[39462]: osdmap e334: 8 total, 7 up, 7 in 2024-09-17T07:17:54.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:54 smithi100 ceph-mon[29092]: pgmap v711: 158 pgs: 8 unknown, 3 active+clean+scrubbing+deep, 36 active+undersized, 29 active+undersized+degraded, 82 active+clean; 7.5 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 853 B/s rd, 35 MiB/s wr, 639 op/s; 30738/369117 objects degraded (8.327%); 0 B/s, 0 objects/s recovering 2024-09-17T07:17:54.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:54 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-17T07:17:54.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:54 smithi100 ceph-mon[29092]: osdmap e334: 8 total, 7 up, 7 in 2024-09-17T07:17:54.664 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:17:54.690 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:54.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:54.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:17:54.786 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:17:54.786 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:17:54.786 INFO:teuthology.orchestra.run.smithi100.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-17T07:17:54.792 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:17:54.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:54 smithi135 ceph-mon[33238]: pgmap v711: 158 pgs: 8 unknown, 3 active+clean+scrubbing+deep, 36 active+undersized, 29 active+undersized+degraded, 82 active+clean; 7.5 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 853 B/s rd, 35 MiB/s wr, 639 op/s; 30738/369117 objects degraded (8.327%); 0 B/s, 0 objects/s recovering 2024-09-17T07:17:54.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:54 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-17T07:17:54.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:54 smithi135 ceph-mon[33238]: osdmap e334: 8 total, 7 up, 7 in 2024-09-17T07:17:54.794 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:17:54.809 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:17:54.809 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:17:54.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:54.816 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:54.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:17:54.831 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:17:54.831 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:17:54.848 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:17:54.849 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:17:54.868 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:17:54.868 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:17:54.885 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:17:54.885 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:17:54.903 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:17:54.904 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:17:54.917 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:17:54.942 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:54.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:54.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:17:55.044 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:17:55.067 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:55.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:55.068 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:17:55.169 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:17:55.197 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:55.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:55.199 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:17:55.299 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:17:55.363 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:55.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:55.364 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:17:55.465 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:17:55.494 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:55.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:55.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:17:55.596 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:17:55.626 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:55.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:55.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:17:55.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:55 smithi100 ceph-mon[29092]: osdmap e335: 8 total, 7 up, 7 in 2024-09-17T07:17:55.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:55 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/3932482237' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:17:55.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:55 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 30738/369117 objects degraded (8.327%), 29 pgs degraded, 63 pgs undersized (PG_DEGRADED) 2024-09-17T07:17:55.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:55 smithi100 ceph-mon[29092]: osdmap e336: 8 total, 7 up, 7 in 2024-09-17T07:17:55.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:55 smithi100 ceph-mon[29092]: 10.19 starting backfill to osd.0 from (0'0,0'0] MAX to 333'7741 2024-09-17T07:17:55.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:55 smithi100 ceph-mon[29092]: 10.13 starting backfill to osd.7 from (0'0,0'0] MAX to 333'10781 2024-09-17T07:17:55.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:55 smithi100 ceph-mon[29092]: 10.18 starting backfill to osd.6 from (0'0,0'0] MAX to 333'7828 2024-09-17T07:17:55.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:55 smithi100 ceph-mon[29092]: 10.17 starting backfill to osd.4 from (0'0,0'0] MAX to 333'7876 2024-09-17T07:17:55.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:55 smithi100 ceph-mon[29092]: 10.19 starting backfill to osd.4 from (0'0,0'0] MAX to 333'7741 2024-09-17T07:17:55.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:55 smithi100 ceph-mon[29092]: pgmap v715: 158 pgs: 4 remapped+peering, 6 unknown, 2 active+clean+scrubbing+deep, 36 active+undersized, 28 active+undersized+degraded, 82 active+clean; 6.0 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s; 18958/295947 objects degraded (6.406%); 0 B/s, 0 objects/s recovering 2024-09-17T07:17:55.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:55 smithi100 ceph-mon[39462]: osdmap e335: 8 total, 7 up, 7 in 2024-09-17T07:17:55.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:55 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/3932482237' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:17:55.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:55 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 30738/369117 objects degraded (8.327%), 29 pgs degraded, 63 pgs undersized (PG_DEGRADED) 2024-09-17T07:17:55.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:55 smithi100 ceph-mon[39462]: osdmap e336: 8 total, 7 up, 7 in 2024-09-17T07:17:55.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:55 smithi100 ceph-mon[39462]: 10.19 starting backfill to osd.0 from (0'0,0'0] MAX to 333'7741 2024-09-17T07:17:55.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:55 smithi100 ceph-mon[39462]: 10.13 starting backfill to osd.7 from (0'0,0'0] MAX to 333'10781 2024-09-17T07:17:55.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:55 smithi100 ceph-mon[39462]: 10.18 starting backfill to osd.6 from (0'0,0'0] MAX to 333'7828 2024-09-17T07:17:55.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:55 smithi100 ceph-mon[39462]: 10.17 starting backfill to osd.4 from (0'0,0'0] MAX to 333'7876 2024-09-17T07:17:55.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:55 smithi100 ceph-mon[39462]: 10.19 starting backfill to osd.4 from (0'0,0'0] MAX to 333'7741 2024-09-17T07:17:55.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:55 smithi100 ceph-mon[39462]: pgmap v715: 158 pgs: 4 remapped+peering, 6 unknown, 2 active+clean+scrubbing+deep, 36 active+undersized, 28 active+undersized+degraded, 82 active+clean; 6.0 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s; 18958/295947 objects degraded (6.406%); 0 B/s, 0 objects/s recovering 2024-09-17T07:17:55.725 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:17:55.726 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:17:55.727 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:17:55.757 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:55.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:55.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:17:55.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:55 smithi135 ceph-mon[33238]: osdmap e335: 8 total, 7 up, 7 in 2024-09-17T07:17:55.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:55 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/3932482237' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:17:55.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:55 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 30738/369117 objects degraded (8.327%), 29 pgs degraded, 63 pgs undersized (PG_DEGRADED) 2024-09-17T07:17:55.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:55 smithi135 ceph-mon[33238]: osdmap e336: 8 total, 7 up, 7 in 2024-09-17T07:17:55.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:55 smithi135 ceph-mon[33238]: 10.19 starting backfill to osd.0 from (0'0,0'0] MAX to 333'7741 2024-09-17T07:17:55.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:55 smithi135 ceph-mon[33238]: 10.13 starting backfill to osd.7 from (0'0,0'0] MAX to 333'10781 2024-09-17T07:17:55.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:55 smithi135 ceph-mon[33238]: 10.18 starting backfill to osd.6 from (0'0,0'0] MAX to 333'7828 2024-09-17T07:17:55.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:55 smithi135 ceph-mon[33238]: 10.17 starting backfill to osd.4 from (0'0,0'0] MAX to 333'7876 2024-09-17T07:17:55.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:55 smithi135 ceph-mon[33238]: 10.19 starting backfill to osd.4 from (0'0,0'0] MAX to 333'7741 2024-09-17T07:17:55.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:55 smithi135 ceph-mon[33238]: pgmap v715: 158 pgs: 4 remapped+peering, 6 unknown, 2 active+clean+scrubbing+deep, 36 active+undersized, 28 active+undersized+degraded, 82 active+clean; 6.0 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s; 18958/295947 objects degraded (6.406%); 0 B/s, 0 objects/s recovering 2024-09-17T07:17:55.859 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:17:55.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:55.895 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:55.895 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:17:55.953 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:17:55 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:17:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:17:55.995 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:17:56.027 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:56.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:56.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:17:56.129 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:17:56.170 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:56.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:56.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:17:56.272 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:17:56.303 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:56.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:56.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:17:56.405 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:17:56.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:56.432 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:56.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:17:56.533 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:17:56.560 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:56.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:56.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:17:56.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:56 smithi100 ceph-mon[29092]: 10.14 starting backfill to osd.2 from (0'0,0'0] MAX to 333'7793 2024-09-17T07:17:56.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:56 smithi100 ceph-mon[29092]: 10.10 starting backfill to osd.7 from (0'0,0'0] MAX to 333'7829 2024-09-17T07:17:56.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:56 smithi100 ceph-mon[29092]: 10.12 starting backfill to osd.2 from (0'0,0'0] MAX to 333'7756 2024-09-17T07:17:56.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:56 smithi100 ceph-mon[29092]: 10.11 starting backfill to osd.5 from (0'0,0'0] MAX to 333'7976 2024-09-17T07:17:56.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:56 smithi100 ceph-mon[29092]: 10.16 starting backfill to osd.6 from (0'0,0'0] MAX to 333'7768 2024-09-17T07:17:56.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:56 smithi100 ceph-mon[29092]: 10.15 starting backfill to osd.5 from (0'0,0'0] MAX to 333'7828 2024-09-17T07:17:56.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:56 smithi100 ceph-mon[29092]: 10.12 starting backfill to osd.5 from (0'0,0'0] MAX to 333'7756 2024-09-17T07:17:56.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:56 smithi100 ceph-mon[29092]: 10.12 starting backfill to osd.6 from (0'0,0'0] MAX to 333'7756 2024-09-17T07:17:56.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:56 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1690398657' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:17:56.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:56 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:17:56.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:56 smithi100 ceph-mon[39462]: 10.14 starting backfill to osd.2 from (0'0,0'0] MAX to 333'7793 2024-09-17T07:17:56.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:56 smithi100 ceph-mon[39462]: 10.10 starting backfill to osd.7 from (0'0,0'0] MAX to 333'7829 2024-09-17T07:17:56.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:56 smithi100 ceph-mon[39462]: 10.12 starting backfill to osd.2 from (0'0,0'0] MAX to 333'7756 2024-09-17T07:17:56.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:56 smithi100 ceph-mon[39462]: 10.11 starting backfill to osd.5 from (0'0,0'0] MAX to 333'7976 2024-09-17T07:17:56.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:56 smithi100 ceph-mon[39462]: 10.16 starting backfill to osd.6 from (0'0,0'0] MAX to 333'7768 2024-09-17T07:17:56.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:56 smithi100 ceph-mon[39462]: 10.15 starting backfill to osd.5 from (0'0,0'0] MAX to 333'7828 2024-09-17T07:17:56.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:56 smithi100 ceph-mon[39462]: 10.12 starting backfill to osd.5 from (0'0,0'0] MAX to 333'7756 2024-09-17T07:17:56.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:56 smithi100 ceph-mon[39462]: 10.12 starting backfill to osd.6 from (0'0,0'0] MAX to 333'7756 2024-09-17T07:17:56.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:56 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1690398657' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:17:56.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:56 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:17:56.662 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:17:56.702 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:56.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:56.704 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:17:56.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:56 smithi135 ceph-mon[33238]: 10.14 starting backfill to osd.2 from (0'0,0'0] MAX to 333'7793 2024-09-17T07:17:56.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:56 smithi135 ceph-mon[33238]: 10.10 starting backfill to osd.7 from (0'0,0'0] MAX to 333'7829 2024-09-17T07:17:56.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:56 smithi135 ceph-mon[33238]: 10.12 starting backfill to osd.2 from (0'0,0'0] MAX to 333'7756 2024-09-17T07:17:56.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:56 smithi135 ceph-mon[33238]: 10.11 starting backfill to osd.5 from (0'0,0'0] MAX to 333'7976 2024-09-17T07:17:56.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:56 smithi135 ceph-mon[33238]: 10.16 starting backfill to osd.6 from (0'0,0'0] MAX to 333'7768 2024-09-17T07:17:56.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:56 smithi135 ceph-mon[33238]: 10.15 starting backfill to osd.5 from (0'0,0'0] MAX to 333'7828 2024-09-17T07:17:56.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:56 smithi135 ceph-mon[33238]: 10.12 starting backfill to osd.5 from (0'0,0'0] MAX to 333'7756 2024-09-17T07:17:56.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:56 smithi135 ceph-mon[33238]: 10.12 starting backfill to osd.6 from (0'0,0'0] MAX to 333'7756 2024-09-17T07:17:56.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:56 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1690398657' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:17:56.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:56 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:17:56.804 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:17:56.825 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:56.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:56.826 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:17:56.927 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:17:56.957 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:56.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:56.958 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:17:57.059 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:17:57.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:57.097 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:57.097 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:17:57.182 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7] out_osds: [3] dead_osds: [1] live_osds: [0, 2, 3, 4, 5, 6, 7] 2024-09-17T07:17:57.182 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T07:17:57.182 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-17T07:17:57.183 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-17T07:17:57.199 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:17:57.237 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:57.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:57.238 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:17:57.255 INFO:teuthology.orchestra.run.smithi100.stderr:nodeep-scrub is set 2024-09-17T07:17:57.339 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:17:57.360 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:57.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:57.361 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:17:57.462 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:17:57.492 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:57.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:57.493 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:17:57.594 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:17:57.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:57 smithi100 ceph-mon[29092]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:17:57.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:57 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T07:17:57.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:57 smithi100 ceph-mon[29092]: osdmap e337: 8 total, 7 up, 7 in 2024-09-17T07:17:57.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:57 smithi100 ceph-mon[29092]: pgmap v717: 158 pgs: 4 remapped+peering, 6 unknown, 2 active+clean+scrubbing+deep, 36 active+undersized, 28 active+undersized+degraded, 82 active+clean; 6.0 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s; 18958/295947 objects degraded (6.406%) 2024-09-17T07:17:57.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:57 smithi100 ceph-mon[39462]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:17:57.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:57 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T07:17:57.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:57 smithi100 ceph-mon[39462]: osdmap e337: 8 total, 7 up, 7 in 2024-09-17T07:17:57.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:57 smithi100 ceph-mon[39462]: pgmap v717: 158 pgs: 4 remapped+peering, 6 unknown, 2 active+clean+scrubbing+deep, 36 active+undersized, 28 active+undersized+degraded, 82 active+clean; 6.0 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s; 18958/295947 objects degraded (6.406%) 2024-09-17T07:17:57.629 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:57.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:57.630 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:17:57.727 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:17:57.731 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:17:57.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:57 smithi135 ceph-mon[33238]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:17:57.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:57 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T07:17:57.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:57 smithi135 ceph-mon[33238]: osdmap e337: 8 total, 7 up, 7 in 2024-09-17T07:17:57.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:57 smithi135 ceph-mon[33238]: pgmap v717: 158 pgs: 4 remapped+peering, 6 unknown, 2 active+clean+scrubbing+deep, 36 active+undersized, 28 active+undersized+degraded, 82 active+clean; 6.0 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s; 18958/295947 objects degraded (6.406%) 2024-09-17T07:17:57.809 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:57.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:57.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:17:57.911 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:17:57.950 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:57.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:57.951 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:17:58.051 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:17:58.081 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:58.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:58.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:17:58.182 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:17:58.219 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:58.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:58.220 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:17:58.321 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:17:58.350 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:58.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:58.351 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:17:58.452 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:17:58.481 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:58.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:58.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:17:58.583 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:17:58.614 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:58.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:58.615 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:17:58.716 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:17:58.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:58.747 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:17:58.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:17:58.848 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:17:58.889 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:58.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:58.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:17:58.991 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:17:59.028 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:59.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:59.030 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:17:59.131 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:17:59.171 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:59.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:59.173 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:17:59.209 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:17:59.210 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-17T07:17:59.210 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-17T07:17:59.211 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-17T07:17:59.211 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-17T07:17:59.211 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-17T07:17:59.273 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:17:59.293 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:17:59 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:17:59.312 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:59.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:59.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:17:59.414 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:17:59.455 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:59.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:59.456 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:17:59.557 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:17:59.577 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:59 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:17:59.577 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:59 smithi100 ceph-mon[29092]: pgmap v718: 158 pgs: 3 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 4 remapped+peering, 36 active+undersized, 1 active+remapped+backfill_toofull, 28 active+undersized+degraded, 84 active+clean; 7.3 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.8 KiB/s rd, 140 KiB/s wr, 680 op/s; 24975/357090 objects degraded (6.994%); 35623/357090 objects misplaced (9.976%); 26 MiB/s, 412 objects/s recovering 2024-09-17T07:17:59.577 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:17:59 smithi100 ceph-mon[29092]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-17T07:17:59.577 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:59 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:17:59.578 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:59 smithi100 ceph-mon[39462]: pgmap v718: 158 pgs: 3 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 4 remapped+peering, 36 active+undersized, 1 active+remapped+backfill_toofull, 28 active+undersized+degraded, 84 active+clean; 7.3 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.8 KiB/s rd, 140 KiB/s wr, 680 op/s; 24975/357090 objects degraded (6.994%); 35623/357090 objects misplaced (9.976%); 26 MiB/s, 412 objects/s recovering 2024-09-17T07:17:59.578 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:17:59 smithi100 ceph-mon[39462]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-17T07:17:59.594 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:59.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:59.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:17:59.696 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:17:59.734 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:59.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:59.735 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:17:59.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:59 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:17:59.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:59 smithi135 ceph-mon[33238]: pgmap v718: 158 pgs: 3 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 4 remapped+peering, 36 active+undersized, 1 active+remapped+backfill_toofull, 28 active+undersized+degraded, 84 active+clean; 7.3 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.8 KiB/s rd, 140 KiB/s wr, 680 op/s; 24975/357090 objects degraded (6.994%); 35623/357090 objects misplaced (9.976%); 26 MiB/s, 412 objects/s recovering 2024-09-17T07:17:59.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:17:59 smithi135 ceph-mon[33238]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-17T07:17:59.836 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:17:59.870 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:17:59.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:17:59.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:17:59.972 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:18:00.003 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:00.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:00.004 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:18:00.105 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:18:00.136 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:00.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:00.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:18:00.222 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:18:00.238 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:18:00.259 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-17T07:18:00.275 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:00.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:00.277 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:18:00.378 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:18:00.420 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:00.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:00.421 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:18:00.522 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:18:00.524 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:00 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 24975/357090 objects degraded (6.994%), 30 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-09-17T07:18:00.524 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:00 smithi100 ceph-mon[29092]: osdmap e338: 8 total, 7 up, 7 in 2024-09-17T07:18:00.525 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:00 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 24975/357090 objects degraded (6.994%), 30 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-09-17T07:18:00.525 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:00 smithi100 ceph-mon[39462]: osdmap e338: 8 total, 7 up, 7 in 2024-09-17T07:18:00.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:00.559 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:00.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:18:00.654 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T07:18:00.660 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:18:00.733 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:00.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:00.734 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:18:00.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:00 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 24975/357090 objects degraded (6.994%), 30 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-09-17T07:18:00.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:00 smithi135 ceph-mon[33238]: osdmap e338: 8 total, 7 up, 7 in 2024-09-17T07:18:00.835 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:18:00.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:00.875 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:00.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:18:00.976 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:18:01.008 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:01.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:01.009 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:18:01.109 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:18:01.167 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:01.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:01.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:18:01.269 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:18:01.309 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:01.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:01.310 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:18:01.411 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:18:01.446 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:01.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:01.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:18:01.479 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:01 smithi100 ceph-mon[29092]: pgmap v720: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 5 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 36 active+undersized, 2 active+remapped+backfill_toofull, 28 active+undersized+degraded, 84 active+clean; 7.2 GiB data, 24 GiB used, 601 GiB / 626 GiB avail; 843 B/s rd, 237 MiB/s wr, 4.39k op/s; 28503/354267 objects degraded (8.046%); 59860/354267 objects misplaced (16.897%); 39 MiB/s, 623 objects/s recovering 2024-09-17T07:18:01.480 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:01 smithi100 ceph-mon[39462]: pgmap v720: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 5 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 36 active+undersized, 2 active+remapped+backfill_toofull, 28 active+undersized+degraded, 84 active+clean; 7.2 GiB data, 24 GiB used, 601 GiB / 626 GiB avail; 843 B/s rd, 237 MiB/s wr, 4.39k op/s; 28503/354267 objects degraded (8.046%); 59860/354267 objects misplaced (16.897%); 39 MiB/s, 623 objects/s recovering 2024-09-17T07:18:01.548 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:18:01.578 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:01.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:01.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:18:01.680 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:18:01.710 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:01.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:01.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:18:01.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:01 smithi135 ceph-mon[33238]: pgmap v720: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 5 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 36 active+undersized, 2 active+remapped+backfill_toofull, 28 active+undersized+degraded, 84 active+clean; 7.2 GiB data, 24 GiB used, 601 GiB / 626 GiB avail; 843 B/s rd, 237 MiB/s wr, 4.39k op/s; 28503/354267 objects degraded (8.046%); 59860/354267 objects misplaced (16.897%); 39 MiB/s, 623 objects/s recovering 2024-09-17T07:18:01.813 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:18:01.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:01.853 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:01.853 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:18:01.954 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:18:01.996 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:01.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:01.997 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:18:02.099 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:18:02.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:02.131 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:02.131 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:18:02.233 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:18:02.264 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:02.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:02.265 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:18:02.366 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:18:02.406 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:02.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:02.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:18:02.508 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:18:02.554 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:02.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:02.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:18:02.656 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:18:02.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:02.688 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:02.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:18:02.759 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:18:02.762 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:18:02.762 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:18:02.765 INFO:teuthology.orchestra.run.smithi100.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-17T07:18:02.782 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:18:02.789 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:18:02.803 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:18:02.803 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:18:02.821 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:02.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:02.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:18:02.831 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:18:02.831 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:18:02.832 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:02 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1579482879' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:18:02.832 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:02 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1579482879' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:18:02.863 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:18:02.863 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:18:02.899 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:18:02.899 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:18:02.924 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:18:02.928 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:18:02.929 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:18:02.957 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:18:02.957 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:18:02.964 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:02.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:02.965 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:18:03.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:02 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1579482879' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:18:03.066 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:18:03.100 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:03.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:03.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:18:03.202 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:18:03.233 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:03.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:03.234 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:18:03.335 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:18:03.372 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:03.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:03.373 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:18:03.473 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:18:03.509 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:03.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:03.510 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:18:03.611 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:18:03.642 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:03.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:03.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:18:03.744 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:18:03.773 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:03.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:03.774 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:18:03.875 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:18:03.906 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:03.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:03.908 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:18:04.008 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:18:04.055 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:04.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:04.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:18:04.088 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:18:04.089 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:18:04.156 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:18:04.195 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:04.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:04.196 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:18:04.297 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:18:04.315 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:04 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:18:04.315 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:04 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:18:04.315 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:04 smithi100 ceph-mon[29092]: pgmap v721: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 5 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 36 active+undersized, 2 active+remapped+backfill_toofull, 28 active+undersized+degraded, 84 active+clean; 7.0 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 639 B/s rd, 180 MiB/s wr, 3.66k op/s; 26602/346338 objects degraded (7.681%); 55374/346338 objects misplaced (15.988%); 59 MiB/s, 951 objects/s recovering 2024-09-17T07:18:04.316 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:04 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:18:04.316 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:04 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:18:04.316 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:04 smithi100 ceph-mon[39462]: pgmap v721: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 5 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 36 active+undersized, 2 active+remapped+backfill_toofull, 28 active+undersized+degraded, 84 active+clean; 7.0 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 639 B/s rd, 180 MiB/s wr, 3.66k op/s; 26602/346338 objects degraded (7.681%); 55374/346338 objects misplaced (15.988%); 59 MiB/s, 951 objects/s recovering 2024-09-17T07:18:04.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:04.331 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:04.332 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:18:04.433 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:18:04.784 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:04 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:18:04.784 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:04 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:18:04.784 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:04 smithi135 ceph-mon[33238]: pgmap v721: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 5 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 36 active+undersized, 2 active+remapped+backfill_toofull, 28 active+undersized+degraded, 84 active+clean; 7.0 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 639 B/s rd, 180 MiB/s wr, 3.66k op/s; 26602/346338 objects degraded (7.681%); 55374/346338 objects misplaced (15.988%); 59 MiB/s, 951 objects/s recovering 2024-09-17T07:18:04.818 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:04.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:04.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:18:04.893 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T07:18:04.894 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-17T07:18:04.894 INFO:teuthology.orchestra.run.smithi100.stdout: "epoch": 338, 2024-09-17T07:18:04.894 INFO:teuthology.orchestra.run.smithi100.stdout: "fsid": "e848267c-74c1-11ef-bceb-c7b262605968", 2024-09-17T07:18:04.894 INFO:teuthology.orchestra.run.smithi100.stdout: "created": "2024-09-17T06:57:05.010532+0000", 2024-09-17T07:18:04.894 INFO:teuthology.orchestra.run.smithi100.stdout: "modified": "2024-09-17T07:18:00.248814+0000", 2024-09-17T07:18:04.894 INFO:teuthology.orchestra.run.smithi100.stdout: "last_up_change": "2024-09-17T07:16:16.355276+0000", 2024-09-17T07:18:04.894 INFO:teuthology.orchestra.run.smithi100.stdout: "last_in_change": "2024-09-17T07:09:23.364026+0000", 2024-09-17T07:18:04.894 INFO:teuthology.orchestra.run.smithi100.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-17T07:18:04.894 INFO:teuthology.orchestra.run.smithi100.stdout: "flags_num": 5806080, 2024-09-17T07:18:04.894 INFO:teuthology.orchestra.run.smithi100.stdout: "flags_set": [ 2024-09-17T07:18:04.894 INFO:teuthology.orchestra.run.smithi100.stdout: "nodeep-scrub", 2024-09-17T07:18:04.895 INFO:teuthology.orchestra.run.smithi100.stdout: "noscrub", 2024-09-17T07:18:04.895 INFO:teuthology.orchestra.run.smithi100.stdout: "pglog_hardlimit", 2024-09-17T07:18:04.895 INFO:teuthology.orchestra.run.smithi100.stdout: "purged_snapdirs", 2024-09-17T07:18:04.895 INFO:teuthology.orchestra.run.smithi100.stdout: "recovery_deletes", 2024-09-17T07:18:04.895 INFO:teuthology.orchestra.run.smithi100.stdout: "sortbitwise" 2024-09-17T07:18:04.895 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-17T07:18:04.895 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_version": 18, 2024-09-17T07:18:04.895 INFO:teuthology.orchestra.run.smithi100.stdout: "full_ratio": 0.94999998807907104, 2024-09-17T07:18:04.895 INFO:teuthology.orchestra.run.smithi100.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-17T07:18:04.895 INFO:teuthology.orchestra.run.smithi100.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-17T07:18:04.895 INFO:teuthology.orchestra.run.smithi100.stdout: "cluster_snapshot": "", 2024-09-17T07:18:04.895 INFO:teuthology.orchestra.run.smithi100.stdout: "pool_max": 10, 2024-09-17T07:18:04.895 INFO:teuthology.orchestra.run.smithi100.stdout: "max_osd": 8, 2024-09-17T07:18:04.896 INFO:teuthology.orchestra.run.smithi100.stdout: "require_min_compat_client": "luminous", 2024-09-17T07:18:04.896 INFO:teuthology.orchestra.run.smithi100.stdout: "min_compat_client": "luminous", 2024-09-17T07:18:04.896 INFO:teuthology.orchestra.run.smithi100.stdout: "require_osd_release": "reef", 2024-09-17T07:18:04.896 INFO:teuthology.orchestra.run.smithi100.stdout: "allow_crimson": false, 2024-09-17T07:18:04.896 INFO:teuthology.orchestra.run.smithi100.stdout: "pools": [ 2024-09-17T07:18:04.896 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:18:04.896 INFO:teuthology.orchestra.run.smithi100.stdout: "pool": 1, 2024-09-17T07:18:04.896 INFO:teuthology.orchestra.run.smithi100.stdout: "pool_name": ".mgr", 2024-09-17T07:18:04.896 INFO:teuthology.orchestra.run.smithi100.stdout: "create_time": "2024-09-17T07:00:08.816136+0000", 2024-09-17T07:18:04.896 INFO:teuthology.orchestra.run.smithi100.stdout: "flags": 1, 2024-09-17T07:18:04.896 INFO:teuthology.orchestra.run.smithi100.stdout: "flags_names": "hashpspool", 2024-09-17T07:18:04.896 INFO:teuthology.orchestra.run.smithi100.stdout: "type": 1, 2024-09-17T07:18:04.897 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 3, 2024-09-17T07:18:04.897 INFO:teuthology.orchestra.run.smithi100.stdout: "min_size": 2, 2024-09-17T07:18:04.897 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_rule": 0, 2024-09-17T07:18:04.897 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_count": 0, 2024-09-17T07:18:04.897 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_target": 0, 2024-09-17T07:18:04.897 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T07:18:04.897 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T07:18:04.897 INFO:teuthology.orchestra.run.smithi100.stdout: "object_hash": 2, 2024-09-17T07:18:04.897 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_autoscale_mode": "off", 2024-09-17T07:18:04.897 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num": 1, 2024-09-17T07:18:04.897 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_placement_num": 1, 2024-09-17T07:18:04.897 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_placement_num_target": 1, 2024-09-17T07:18:04.897 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num_target": 1, 2024-09-17T07:18:04.898 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num_pending": 1, 2024-09-17T07:18:04.898 INFO:teuthology.orchestra.run.smithi100.stdout: "last_pg_merge_meta": { 2024-09-17T07:18:04.898 INFO:teuthology.orchestra.run.smithi100.stdout: "source_pgid": "0.0", 2024-09-17T07:18:04.898 INFO:teuthology.orchestra.run.smithi100.stdout: "ready_epoch": 0, 2024-09-17T07:18:04.898 INFO:teuthology.orchestra.run.smithi100.stdout: "last_epoch_started": 0, 2024-09-17T07:18:04.898 INFO:teuthology.orchestra.run.smithi100.stdout: "last_epoch_clean": 0, 2024-09-17T07:18:04.898 INFO:teuthology.orchestra.run.smithi100.stdout: "source_version": "0'0", 2024-09-17T07:18:04.898 INFO:teuthology.orchestra.run.smithi100.stdout: "target_version": "0'0" 2024-09-17T07:18:04.898 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.898 INFO:teuthology.orchestra.run.smithi100.stdout: "last_change": "205", 2024-09-17T07:18:04.898 INFO:teuthology.orchestra.run.smithi100.stdout: "last_force_op_resend": "0", 2024-09-17T07:18:04.898 INFO:teuthology.orchestra.run.smithi100.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-17T07:18:04.899 INFO:teuthology.orchestra.run.smithi100.stdout: "last_force_op_resend_preluminous": "0", 2024-09-17T07:18:04.899 INFO:teuthology.orchestra.run.smithi100.stdout: "auid": 0, 2024-09-17T07:18:04.899 INFO:teuthology.orchestra.run.smithi100.stdout: "snap_mode": "selfmanaged", 2024-09-17T07:18:04.899 INFO:teuthology.orchestra.run.smithi100.stdout: "snap_seq": 0, 2024-09-17T07:18:04.899 INFO:teuthology.orchestra.run.smithi100.stdout: "snap_epoch": 0, 2024-09-17T07:18:04.899 INFO:teuthology.orchestra.run.smithi100.stdout: "pool_snaps": [], 2024-09-17T07:18:04.899 INFO:teuthology.orchestra.run.smithi100.stdout: "removed_snaps": "[]", 2024-09-17T07:18:04.899 INFO:teuthology.orchestra.run.smithi100.stdout: "quota_max_bytes": 0, 2024-09-17T07:18:04.899 INFO:teuthology.orchestra.run.smithi100.stdout: "quota_max_objects": 0, 2024-09-17T07:18:04.900 INFO:teuthology.orchestra.run.smithi100.stdout: "tiers": [], 2024-09-17T07:18:04.900 INFO:teuthology.orchestra.run.smithi100.stdout: "tier_of": -1, 2024-09-17T07:18:04.900 INFO:teuthology.orchestra.run.smithi100.stdout: "read_tier": -1, 2024-09-17T07:18:04.900 INFO:teuthology.orchestra.run.smithi100.stdout: "write_tier": -1, 2024-09-17T07:18:04.900 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_mode": "none", 2024-09-17T07:18:04.900 INFO:teuthology.orchestra.run.smithi100.stdout: "target_max_bytes": 0, 2024-09-17T07:18:04.900 INFO:teuthology.orchestra.run.smithi100.stdout: "target_max_objects": 0, 2024-09-17T07:18:04.900 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T07:18:04.900 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T07:18:04.900 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T07:18:04.900 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_min_flush_age": 0, 2024-09-17T07:18:04.900 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_min_evict_age": 0, 2024-09-17T07:18:04.901 INFO:teuthology.orchestra.run.smithi100.stdout: "erasure_code_profile": "", 2024-09-17T07:18:04.901 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_params": { 2024-09-17T07:18:04.901 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "none" 2024-09-17T07:18:04.901 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.901 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_period": 0, 2024-09-17T07:18:04.901 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_count": 0, 2024-09-17T07:18:04.901 INFO:teuthology.orchestra.run.smithi100.stdout: "use_gmt_hitset": true, 2024-09-17T07:18:04.901 INFO:teuthology.orchestra.run.smithi100.stdout: "min_read_recency_for_promote": 0, 2024-09-17T07:18:04.901 INFO:teuthology.orchestra.run.smithi100.stdout: "min_write_recency_for_promote": 0, 2024-09-17T07:18:04.901 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T07:18:04.901 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_search_last_n": 0, 2024-09-17T07:18:04.901 INFO:teuthology.orchestra.run.smithi100.stdout: "grade_table": [], 2024-09-17T07:18:04.902 INFO:teuthology.orchestra.run.smithi100.stdout: "stripe_width": 0, 2024-09-17T07:18:04.902 INFO:teuthology.orchestra.run.smithi100.stdout: "expected_num_objects": 0, 2024-09-17T07:18:04.902 INFO:teuthology.orchestra.run.smithi100.stdout: "fast_read": false, 2024-09-17T07:18:04.902 INFO:teuthology.orchestra.run.smithi100.stdout: "options": { 2024-09-17T07:18:04.902 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num_max": 32, 2024-09-17T07:18:04.902 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num_min": 1 2024-09-17T07:18:04.902 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.902 INFO:teuthology.orchestra.run.smithi100.stdout: "application_metadata": { 2024-09-17T07:18:04.902 INFO:teuthology.orchestra.run.smithi100.stdout: "mgr": {} 2024-09-17T07:18:04.902 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.902 INFO:teuthology.orchestra.run.smithi100.stdout: "read_balance": { 2024-09-17T07:18:04.902 INFO:teuthology.orchestra.run.smithi100.stdout: "score_acting": 7.8899998664855957, 2024-09-17T07:18:04.902 INFO:teuthology.orchestra.run.smithi100.stdout: "score_stable": 7.8899998664855957, 2024-09-17T07:18:04.903 INFO:teuthology.orchestra.run.smithi100.stdout: "optimal_score": 0.37999999523162842, 2024-09-17T07:18:04.903 INFO:teuthology.orchestra.run.smithi100.stdout: "raw_score_acting": 3, 2024-09-17T07:18:04.903 INFO:teuthology.orchestra.run.smithi100.stdout: "raw_score_stable": 3, 2024-09-17T07:18:04.903 INFO:teuthology.orchestra.run.smithi100.stdout: "primary_affinity_weighted": 0.89598274230957031, 2024-09-17T07:18:04.903 INFO:teuthology.orchestra.run.smithi100.stdout: "average_primary_affinity": 0.89999997615814209, 2024-09-17T07:18:04.903 INFO:teuthology.orchestra.run.smithi100.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-09-17T07:18:04.903 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:18:04.903 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.903 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:18:04.903 INFO:teuthology.orchestra.run.smithi100.stdout: "pool": 2, 2024-09-17T07:18:04.903 INFO:teuthology.orchestra.run.smithi100.stdout: "pool_name": ".rgw.root", 2024-09-17T07:18:04.903 INFO:teuthology.orchestra.run.smithi100.stdout: "create_time": "2024-09-17T07:02:10.759826+0000", 2024-09-17T07:18:04.904 INFO:teuthology.orchestra.run.smithi100.stdout: "flags": 1, 2024-09-17T07:18:04.904 INFO:teuthology.orchestra.run.smithi100.stdout: "flags_names": "hashpspool", 2024-09-17T07:18:04.904 INFO:teuthology.orchestra.run.smithi100.stdout: "type": 1, 2024-09-17T07:18:04.904 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 3, 2024-09-17T07:18:04.904 INFO:teuthology.orchestra.run.smithi100.stdout: "min_size": 2, 2024-09-17T07:18:04.904 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_rule": 0, 2024-09-17T07:18:04.904 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_count": 0, 2024-09-17T07:18:04.904 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_target": 0, 2024-09-17T07:18:04.904 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T07:18:04.904 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T07:18:04.904 INFO:teuthology.orchestra.run.smithi100.stdout: "object_hash": 2, 2024-09-17T07:18:04.904 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_autoscale_mode": "off", 2024-09-17T07:18:04.905 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num": 32, 2024-09-17T07:18:04.905 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_placement_num": 32, 2024-09-17T07:18:04.905 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_placement_num_target": 32, 2024-09-17T07:18:04.905 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num_target": 32, 2024-09-17T07:18:04.905 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num_pending": 32, 2024-09-17T07:18:04.905 INFO:teuthology.orchestra.run.smithi100.stdout: "last_pg_merge_meta": { 2024-09-17T07:18:04.905 INFO:teuthology.orchestra.run.smithi100.stdout: "source_pgid": "0.0", 2024-09-17T07:18:04.905 INFO:teuthology.orchestra.run.smithi100.stdout: "ready_epoch": 0, 2024-09-17T07:18:04.905 INFO:teuthology.orchestra.run.smithi100.stdout: "last_epoch_started": 0, 2024-09-17T07:18:04.905 INFO:teuthology.orchestra.run.smithi100.stdout: "last_epoch_clean": 0, 2024-09-17T07:18:04.905 INFO:teuthology.orchestra.run.smithi100.stdout: "source_version": "0'0", 2024-09-17T07:18:04.905 INFO:teuthology.orchestra.run.smithi100.stdout: "target_version": "0'0" 2024-09-17T07:18:04.906 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.906 INFO:teuthology.orchestra.run.smithi100.stdout: "last_change": "69", 2024-09-17T07:18:04.906 INFO:teuthology.orchestra.run.smithi100.stdout: "last_force_op_resend": "0", 2024-09-17T07:18:04.906 INFO:teuthology.orchestra.run.smithi100.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-17T07:18:04.906 INFO:teuthology.orchestra.run.smithi100.stdout: "last_force_op_resend_preluminous": "0", 2024-09-17T07:18:04.906 INFO:teuthology.orchestra.run.smithi100.stdout: "auid": 0, 2024-09-17T07:18:04.906 INFO:teuthology.orchestra.run.smithi100.stdout: "snap_mode": "selfmanaged", 2024-09-17T07:18:04.906 INFO:teuthology.orchestra.run.smithi100.stdout: "snap_seq": 0, 2024-09-17T07:18:04.906 INFO:teuthology.orchestra.run.smithi100.stdout: "snap_epoch": 0, 2024-09-17T07:18:04.906 INFO:teuthology.orchestra.run.smithi100.stdout: "pool_snaps": [], 2024-09-17T07:18:04.906 INFO:teuthology.orchestra.run.smithi100.stdout: "removed_snaps": "[]", 2024-09-17T07:18:04.906 INFO:teuthology.orchestra.run.smithi100.stdout: "quota_max_bytes": 0, 2024-09-17T07:18:04.906 INFO:teuthology.orchestra.run.smithi100.stdout: "quota_max_objects": 0, 2024-09-17T07:18:04.907 INFO:teuthology.orchestra.run.smithi100.stdout: "tiers": [], 2024-09-17T07:18:04.907 INFO:teuthology.orchestra.run.smithi100.stdout: "tier_of": -1, 2024-09-17T07:18:04.907 INFO:teuthology.orchestra.run.smithi100.stdout: "read_tier": -1, 2024-09-17T07:18:04.907 INFO:teuthology.orchestra.run.smithi100.stdout: "write_tier": -1, 2024-09-17T07:18:04.907 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_mode": "none", 2024-09-17T07:18:04.907 INFO:teuthology.orchestra.run.smithi100.stdout: "target_max_bytes": 0, 2024-09-17T07:18:04.907 INFO:teuthology.orchestra.run.smithi100.stdout: "target_max_objects": 0, 2024-09-17T07:18:04.907 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T07:18:04.907 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T07:18:04.907 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T07:18:04.907 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_min_flush_age": 0, 2024-09-17T07:18:04.907 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_min_evict_age": 0, 2024-09-17T07:18:04.907 INFO:teuthology.orchestra.run.smithi100.stdout: "erasure_code_profile": "", 2024-09-17T07:18:04.908 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_params": { 2024-09-17T07:18:04.908 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "none" 2024-09-17T07:18:04.908 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.908 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_period": 0, 2024-09-17T07:18:04.908 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_count": 0, 2024-09-17T07:18:04.908 INFO:teuthology.orchestra.run.smithi100.stdout: "use_gmt_hitset": true, 2024-09-17T07:18:04.908 INFO:teuthology.orchestra.run.smithi100.stdout: "min_read_recency_for_promote": 0, 2024-09-17T07:18:04.908 INFO:teuthology.orchestra.run.smithi100.stdout: "min_write_recency_for_promote": 0, 2024-09-17T07:18:04.908 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T07:18:04.908 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_search_last_n": 0, 2024-09-17T07:18:04.908 INFO:teuthology.orchestra.run.smithi100.stdout: "grade_table": [], 2024-09-17T07:18:04.908 INFO:teuthology.orchestra.run.smithi100.stdout: "stripe_width": 0, 2024-09-17T07:18:04.909 INFO:teuthology.orchestra.run.smithi100.stdout: "expected_num_objects": 0, 2024-09-17T07:18:04.909 INFO:teuthology.orchestra.run.smithi100.stdout: "fast_read": false, 2024-09-17T07:18:04.909 INFO:teuthology.orchestra.run.smithi100.stdout: "options": {}, 2024-09-17T07:18:04.909 INFO:teuthology.orchestra.run.smithi100.stdout: "application_metadata": { 2024-09-17T07:18:04.909 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw": {} 2024-09-17T07:18:04.909 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.909 INFO:teuthology.orchestra.run.smithi100.stdout: "read_balance": { 2024-09-17T07:18:04.909 INFO:teuthology.orchestra.run.smithi100.stdout: "score_acting": 2.5099999904632568, 2024-09-17T07:18:04.909 INFO:teuthology.orchestra.run.smithi100.stdout: "score_stable": 2.5099999904632568, 2024-09-17T07:18:04.909 INFO:teuthology.orchestra.run.smithi100.stdout: "optimal_score": 0.75, 2024-09-17T07:18:04.909 INFO:teuthology.orchestra.run.smithi100.stdout: "raw_score_acting": 1.8799999952316284, 2024-09-17T07:18:04.909 INFO:teuthology.orchestra.run.smithi100.stdout: "raw_score_stable": 1.8799999952316284, 2024-09-17T07:18:04.909 INFO:teuthology.orchestra.run.smithi100.stdout: "primary_affinity_weighted": 0.89598274230957031, 2024-09-17T07:18:04.910 INFO:teuthology.orchestra.run.smithi100.stdout: "average_primary_affinity": 0.89999997615814209, 2024-09-17T07:18:04.910 INFO:teuthology.orchestra.run.smithi100.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-09-17T07:18:04.910 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:18:04.910 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.910 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:18:04.910 INFO:teuthology.orchestra.run.smithi100.stdout: "pool": 3, 2024-09-17T07:18:04.910 INFO:teuthology.orchestra.run.smithi100.stdout: "pool_name": "datapool", 2024-09-17T07:18:04.910 INFO:teuthology.orchestra.run.smithi100.stdout: "create_time": "2024-09-17T07:02:10.991289+0000", 2024-09-17T07:18:04.910 INFO:teuthology.orchestra.run.smithi100.stdout: "flags": 8193, 2024-09-17T07:18:04.910 INFO:teuthology.orchestra.run.smithi100.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-17T07:18:04.910 INFO:teuthology.orchestra.run.smithi100.stdout: "type": 1, 2024-09-17T07:18:04.910 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 3, 2024-09-17T07:18:04.910 INFO:teuthology.orchestra.run.smithi100.stdout: "min_size": 2, 2024-09-17T07:18:04.911 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_rule": 0, 2024-09-17T07:18:04.911 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_count": 0, 2024-09-17T07:18:04.911 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_target": 0, 2024-09-17T07:18:04.911 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T07:18:04.911 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T07:18:04.911 INFO:teuthology.orchestra.run.smithi100.stdout: "object_hash": 2, 2024-09-17T07:18:04.911 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_autoscale_mode": "off", 2024-09-17T07:18:04.911 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num": 3, 2024-09-17T07:18:04.911 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_placement_num": 3, 2024-09-17T07:18:04.911 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_placement_num_target": 3, 2024-09-17T07:18:04.911 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num_target": 3, 2024-09-17T07:18:04.911 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num_pending": 3, 2024-09-17T07:18:04.911 INFO:teuthology.orchestra.run.smithi100.stdout: "last_pg_merge_meta": { 2024-09-17T07:18:04.912 INFO:teuthology.orchestra.run.smithi100.stdout: "source_pgid": "0.0", 2024-09-17T07:18:04.912 INFO:teuthology.orchestra.run.smithi100.stdout: "ready_epoch": 0, 2024-09-17T07:18:04.912 INFO:teuthology.orchestra.run.smithi100.stdout: "last_epoch_started": 0, 2024-09-17T07:18:04.912 INFO:teuthology.orchestra.run.smithi100.stdout: "last_epoch_clean": 0, 2024-09-17T07:18:04.912 INFO:teuthology.orchestra.run.smithi100.stdout: "source_version": "0'0", 2024-09-17T07:18:04.912 INFO:teuthology.orchestra.run.smithi100.stdout: "target_version": "0'0" 2024-09-17T07:18:04.913 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.913 INFO:teuthology.orchestra.run.smithi100.stdout: "last_change": "72", 2024-09-17T07:18:04.913 INFO:teuthology.orchestra.run.smithi100.stdout: "last_force_op_resend": "0", 2024-09-17T07:18:04.913 INFO:teuthology.orchestra.run.smithi100.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-17T07:18:04.913 INFO:teuthology.orchestra.run.smithi100.stdout: "last_force_op_resend_preluminous": "0", 2024-09-17T07:18:04.913 INFO:teuthology.orchestra.run.smithi100.stdout: "auid": 0, 2024-09-17T07:18:04.913 INFO:teuthology.orchestra.run.smithi100.stdout: "snap_mode": "selfmanaged", 2024-09-17T07:18:04.913 INFO:teuthology.orchestra.run.smithi100.stdout: "snap_seq": 3, 2024-09-17T07:18:04.913 INFO:teuthology.orchestra.run.smithi100.stdout: "snap_epoch": 72, 2024-09-17T07:18:04.913 INFO:teuthology.orchestra.run.smithi100.stdout: "pool_snaps": [], 2024-09-17T07:18:04.914 INFO:teuthology.orchestra.run.smithi100.stdout: "removed_snaps": "[]", 2024-09-17T07:18:04.914 INFO:teuthology.orchestra.run.smithi100.stdout: "quota_max_bytes": 0, 2024-09-17T07:18:04.914 INFO:teuthology.orchestra.run.smithi100.stdout: "quota_max_objects": 0, 2024-09-17T07:18:04.914 INFO:teuthology.orchestra.run.smithi100.stdout: "tiers": [], 2024-09-17T07:18:04.914 INFO:teuthology.orchestra.run.smithi100.stdout: "tier_of": -1, 2024-09-17T07:18:04.914 INFO:teuthology.orchestra.run.smithi100.stdout: "read_tier": -1, 2024-09-17T07:18:04.914 INFO:teuthology.orchestra.run.smithi100.stdout: "write_tier": -1, 2024-09-17T07:18:04.914 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_mode": "none", 2024-09-17T07:18:04.914 INFO:teuthology.orchestra.run.smithi100.stdout: "target_max_bytes": 0, 2024-09-17T07:18:04.914 INFO:teuthology.orchestra.run.smithi100.stdout: "target_max_objects": 0, 2024-09-17T07:18:04.914 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T07:18:04.914 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T07:18:04.914 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T07:18:04.915 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_min_flush_age": 0, 2024-09-17T07:18:04.915 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_min_evict_age": 0, 2024-09-17T07:18:04.915 INFO:teuthology.orchestra.run.smithi100.stdout: "erasure_code_profile": "", 2024-09-17T07:18:04.915 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_params": { 2024-09-17T07:18:04.915 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "none" 2024-09-17T07:18:04.915 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.915 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_period": 0, 2024-09-17T07:18:04.915 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_count": 0, 2024-09-17T07:18:04.915 INFO:teuthology.orchestra.run.smithi100.stdout: "use_gmt_hitset": true, 2024-09-17T07:18:04.915 INFO:teuthology.orchestra.run.smithi100.stdout: "min_read_recency_for_promote": 0, 2024-09-17T07:18:04.915 INFO:teuthology.orchestra.run.smithi100.stdout: "min_write_recency_for_promote": 0, 2024-09-17T07:18:04.915 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T07:18:04.915 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_search_last_n": 0, 2024-09-17T07:18:04.916 INFO:teuthology.orchestra.run.smithi100.stdout: "grade_table": [], 2024-09-17T07:18:04.916 INFO:teuthology.orchestra.run.smithi100.stdout: "stripe_width": 0, 2024-09-17T07:18:04.916 INFO:teuthology.orchestra.run.smithi100.stdout: "expected_num_objects": 0, 2024-09-17T07:18:04.916 INFO:teuthology.orchestra.run.smithi100.stdout: "fast_read": false, 2024-09-17T07:18:04.916 INFO:teuthology.orchestra.run.smithi100.stdout: "options": {}, 2024-09-17T07:18:04.916 INFO:teuthology.orchestra.run.smithi100.stdout: "application_metadata": { 2024-09-17T07:18:04.916 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd": {} 2024-09-17T07:18:04.916 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.916 INFO:teuthology.orchestra.run.smithi100.stdout: "read_balance": { 2024-09-17T07:18:04.916 INFO:teuthology.orchestra.run.smithi100.stdout: "score_acting": 2.6500000953674316, 2024-09-17T07:18:04.916 INFO:teuthology.orchestra.run.smithi100.stdout: "score_stable": 2.6500000953674316, 2024-09-17T07:18:04.916 INFO:teuthology.orchestra.run.smithi100.stdout: "optimal_score": 0.62999999523162842, 2024-09-17T07:18:04.916 INFO:teuthology.orchestra.run.smithi100.stdout: "raw_score_acting": 1.6699999570846558, 2024-09-17T07:18:04.917 INFO:teuthology.orchestra.run.smithi100.stdout: "raw_score_stable": 1.6699999570846558, 2024-09-17T07:18:04.917 INFO:teuthology.orchestra.run.smithi100.stdout: "primary_affinity_weighted": 0.89598274230957031, 2024-09-17T07:18:04.917 INFO:teuthology.orchestra.run.smithi100.stdout: "average_primary_affinity": 0.89999997615814209, 2024-09-17T07:18:04.917 INFO:teuthology.orchestra.run.smithi100.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-09-17T07:18:04.917 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:18:04.917 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.917 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:18:04.917 INFO:teuthology.orchestra.run.smithi100.stdout: "pool": 4, 2024-09-17T07:18:04.917 INFO:teuthology.orchestra.run.smithi100.stdout: "pool_name": "default.rgw.log", 2024-09-17T07:18:04.917 INFO:teuthology.orchestra.run.smithi100.stdout: "create_time": "2024-09-17T07:02:12.291462+0000", 2024-09-17T07:18:04.917 INFO:teuthology.orchestra.run.smithi100.stdout: "flags": 1, 2024-09-17T07:18:04.917 INFO:teuthology.orchestra.run.smithi100.stdout: "flags_names": "hashpspool", 2024-09-17T07:18:04.917 INFO:teuthology.orchestra.run.smithi100.stdout: "type": 1, 2024-09-17T07:18:04.918 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 3, 2024-09-17T07:18:04.918 INFO:teuthology.orchestra.run.smithi100.stdout: "min_size": 2, 2024-09-17T07:18:04.918 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_rule": 0, 2024-09-17T07:18:04.918 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_count": 0, 2024-09-17T07:18:04.918 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_target": 0, 2024-09-17T07:18:04.918 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T07:18:04.918 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T07:18:04.918 INFO:teuthology.orchestra.run.smithi100.stdout: "object_hash": 2, 2024-09-17T07:18:04.918 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_autoscale_mode": "off", 2024-09-17T07:18:04.918 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num": 32, 2024-09-17T07:18:04.918 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_placement_num": 32, 2024-09-17T07:18:04.918 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_placement_num_target": 32, 2024-09-17T07:18:04.918 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num_target": 32, 2024-09-17T07:18:04.919 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num_pending": 32, 2024-09-17T07:18:04.919 INFO:teuthology.orchestra.run.smithi100.stdout: "last_pg_merge_meta": { 2024-09-17T07:18:04.919 INFO:teuthology.orchestra.run.smithi100.stdout: "source_pgid": "0.0", 2024-09-17T07:18:04.919 INFO:teuthology.orchestra.run.smithi100.stdout: "ready_epoch": 0, 2024-09-17T07:18:04.919 INFO:teuthology.orchestra.run.smithi100.stdout: "last_epoch_started": 0, 2024-09-17T07:18:04.919 INFO:teuthology.orchestra.run.smithi100.stdout: "last_epoch_clean": 0, 2024-09-17T07:18:04.919 INFO:teuthology.orchestra.run.smithi100.stdout: "source_version": "0'0", 2024-09-17T07:18:04.919 INFO:teuthology.orchestra.run.smithi100.stdout: "target_version": "0'0" 2024-09-17T07:18:04.919 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.919 INFO:teuthology.orchestra.run.smithi100.stdout: "last_change": "71", 2024-09-17T07:18:04.919 INFO:teuthology.orchestra.run.smithi100.stdout: "last_force_op_resend": "0", 2024-09-17T07:18:04.919 INFO:teuthology.orchestra.run.smithi100.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-17T07:18:04.919 INFO:teuthology.orchestra.run.smithi100.stdout: "last_force_op_resend_preluminous": "0", 2024-09-17T07:18:04.920 INFO:teuthology.orchestra.run.smithi100.stdout: "auid": 0, 2024-09-17T07:18:04.920 INFO:teuthology.orchestra.run.smithi100.stdout: "snap_mode": "selfmanaged", 2024-09-17T07:18:04.920 INFO:teuthology.orchestra.run.smithi100.stdout: "snap_seq": 0, 2024-09-17T07:18:04.920 INFO:teuthology.orchestra.run.smithi100.stdout: "snap_epoch": 0, 2024-09-17T07:18:04.920 INFO:teuthology.orchestra.run.smithi100.stdout: "pool_snaps": [], 2024-09-17T07:18:04.920 INFO:teuthology.orchestra.run.smithi100.stdout: "removed_snaps": "[]", 2024-09-17T07:18:04.920 INFO:teuthology.orchestra.run.smithi100.stdout: "quota_max_bytes": 0, 2024-09-17T07:18:04.920 INFO:teuthology.orchestra.run.smithi100.stdout: "quota_max_objects": 0, 2024-09-17T07:18:04.920 INFO:teuthology.orchestra.run.smithi100.stdout: "tiers": [], 2024-09-17T07:18:04.920 INFO:teuthology.orchestra.run.smithi100.stdout: "tier_of": -1, 2024-09-17T07:18:04.920 INFO:teuthology.orchestra.run.smithi100.stdout: "read_tier": -1, 2024-09-17T07:18:04.920 INFO:teuthology.orchestra.run.smithi100.stdout: "write_tier": -1, 2024-09-17T07:18:04.921 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_mode": "none", 2024-09-17T07:18:04.921 INFO:teuthology.orchestra.run.smithi100.stdout: "target_max_bytes": 0, 2024-09-17T07:18:04.921 INFO:teuthology.orchestra.run.smithi100.stdout: "target_max_objects": 0, 2024-09-17T07:18:04.921 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T07:18:04.921 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T07:18:04.921 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T07:18:04.921 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_min_flush_age": 0, 2024-09-17T07:18:04.921 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_min_evict_age": 0, 2024-09-17T07:18:04.921 INFO:teuthology.orchestra.run.smithi100.stdout: "erasure_code_profile": "", 2024-09-17T07:18:04.921 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_params": { 2024-09-17T07:18:04.921 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "none" 2024-09-17T07:18:04.921 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.921 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_period": 0, 2024-09-17T07:18:04.922 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_count": 0, 2024-09-17T07:18:04.922 INFO:teuthology.orchestra.run.smithi100.stdout: "use_gmt_hitset": true, 2024-09-17T07:18:04.922 INFO:teuthology.orchestra.run.smithi100.stdout: "min_read_recency_for_promote": 0, 2024-09-17T07:18:04.922 INFO:teuthology.orchestra.run.smithi100.stdout: "min_write_recency_for_promote": 0, 2024-09-17T07:18:04.922 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T07:18:04.922 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_search_last_n": 0, 2024-09-17T07:18:04.922 INFO:teuthology.orchestra.run.smithi100.stdout: "grade_table": [], 2024-09-17T07:18:04.922 INFO:teuthology.orchestra.run.smithi100.stdout: "stripe_width": 0, 2024-09-17T07:18:04.922 INFO:teuthology.orchestra.run.smithi100.stdout: "expected_num_objects": 0, 2024-09-17T07:18:04.922 INFO:teuthology.orchestra.run.smithi100.stdout: "fast_read": false, 2024-09-17T07:18:04.922 INFO:teuthology.orchestra.run.smithi100.stdout: "options": {}, 2024-09-17T07:18:04.922 INFO:teuthology.orchestra.run.smithi100.stdout: "application_metadata": { 2024-09-17T07:18:04.922 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw": {} 2024-09-17T07:18:04.923 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.923 INFO:teuthology.orchestra.run.smithi100.stdout: "read_balance": { 2024-09-17T07:18:04.923 INFO:teuthology.orchestra.run.smithi100.stdout: "score_acting": 2.75, 2024-09-17T07:18:04.923 INFO:teuthology.orchestra.run.smithi100.stdout: "score_stable": 2.75, 2024-09-17T07:18:04.923 INFO:teuthology.orchestra.run.smithi100.stdout: "optimal_score": 0.75, 2024-09-17T07:18:04.923 INFO:teuthology.orchestra.run.smithi100.stdout: "raw_score_acting": 2.059999942779541, 2024-09-17T07:18:04.923 INFO:teuthology.orchestra.run.smithi100.stdout: "raw_score_stable": 2.059999942779541, 2024-09-17T07:18:04.923 INFO:teuthology.orchestra.run.smithi100.stdout: "primary_affinity_weighted": 0.89598274230957031, 2024-09-17T07:18:04.923 INFO:teuthology.orchestra.run.smithi100.stdout: "average_primary_affinity": 0.89999997615814209, 2024-09-17T07:18:04.923 INFO:teuthology.orchestra.run.smithi100.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-09-17T07:18:04.923 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:18:04.923 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.923 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:18:04.924 INFO:teuthology.orchestra.run.smithi100.stdout: "pool": 5, 2024-09-17T07:18:04.924 INFO:teuthology.orchestra.run.smithi100.stdout: "pool_name": "default.rgw.control", 2024-09-17T07:18:04.924 INFO:teuthology.orchestra.run.smithi100.stdout: "create_time": "2024-09-17T07:02:14.253214+0000", 2024-09-17T07:18:04.924 INFO:teuthology.orchestra.run.smithi100.stdout: "flags": 1, 2024-09-17T07:18:04.924 INFO:teuthology.orchestra.run.smithi100.stdout: "flags_names": "hashpspool", 2024-09-17T07:18:04.924 INFO:teuthology.orchestra.run.smithi100.stdout: "type": 1, 2024-09-17T07:18:04.924 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 3, 2024-09-17T07:18:04.924 INFO:teuthology.orchestra.run.smithi100.stdout: "min_size": 2, 2024-09-17T07:18:04.924 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_rule": 0, 2024-09-17T07:18:04.924 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_count": 0, 2024-09-17T07:18:04.924 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_target": 0, 2024-09-17T07:18:04.924 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T07:18:04.924 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T07:18:04.925 INFO:teuthology.orchestra.run.smithi100.stdout: "object_hash": 2, 2024-09-17T07:18:04.925 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_autoscale_mode": "off", 2024-09-17T07:18:04.925 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num": 32, 2024-09-17T07:18:04.925 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_placement_num": 32, 2024-09-17T07:18:04.925 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_placement_num_target": 32, 2024-09-17T07:18:04.925 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num_target": 32, 2024-09-17T07:18:04.925 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num_pending": 32, 2024-09-17T07:18:04.925 INFO:teuthology.orchestra.run.smithi100.stdout: "last_pg_merge_meta": { 2024-09-17T07:18:04.925 INFO:teuthology.orchestra.run.smithi100.stdout: "source_pgid": "0.0", 2024-09-17T07:18:04.925 INFO:teuthology.orchestra.run.smithi100.stdout: "ready_epoch": 0, 2024-09-17T07:18:04.925 INFO:teuthology.orchestra.run.smithi100.stdout: "last_epoch_started": 0, 2024-09-17T07:18:04.925 INFO:teuthology.orchestra.run.smithi100.stdout: "last_epoch_clean": 0, 2024-09-17T07:18:04.925 INFO:teuthology.orchestra.run.smithi100.stdout: "source_version": "0'0", 2024-09-17T07:18:04.926 INFO:teuthology.orchestra.run.smithi100.stdout: "target_version": "0'0" 2024-09-17T07:18:04.926 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.926 INFO:teuthology.orchestra.run.smithi100.stdout: "last_change": "73", 2024-09-17T07:18:04.926 INFO:teuthology.orchestra.run.smithi100.stdout: "last_force_op_resend": "0", 2024-09-17T07:18:04.926 INFO:teuthology.orchestra.run.smithi100.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-17T07:18:04.926 INFO:teuthology.orchestra.run.smithi100.stdout: "last_force_op_resend_preluminous": "0", 2024-09-17T07:18:04.926 INFO:teuthology.orchestra.run.smithi100.stdout: "auid": 0, 2024-09-17T07:18:04.926 INFO:teuthology.orchestra.run.smithi100.stdout: "snap_mode": "selfmanaged", 2024-09-17T07:18:04.926 INFO:teuthology.orchestra.run.smithi100.stdout: "snap_seq": 0, 2024-09-17T07:18:04.926 INFO:teuthology.orchestra.run.smithi100.stdout: "snap_epoch": 0, 2024-09-17T07:18:04.926 INFO:teuthology.orchestra.run.smithi100.stdout: "pool_snaps": [], 2024-09-17T07:18:04.926 INFO:teuthology.orchestra.run.smithi100.stdout: "removed_snaps": "[]", 2024-09-17T07:18:04.926 INFO:teuthology.orchestra.run.smithi100.stdout: "quota_max_bytes": 0, 2024-09-17T07:18:04.926 INFO:teuthology.orchestra.run.smithi100.stdout: "quota_max_objects": 0, 2024-09-17T07:18:04.927 INFO:teuthology.orchestra.run.smithi100.stdout: "tiers": [], 2024-09-17T07:18:04.927 INFO:teuthology.orchestra.run.smithi100.stdout: "tier_of": -1, 2024-09-17T07:18:04.927 INFO:teuthology.orchestra.run.smithi100.stdout: "read_tier": -1, 2024-09-17T07:18:04.927 INFO:teuthology.orchestra.run.smithi100.stdout: "write_tier": -1, 2024-09-17T07:18:04.927 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_mode": "none", 2024-09-17T07:18:04.927 INFO:teuthology.orchestra.run.smithi100.stdout: "target_max_bytes": 0, 2024-09-17T07:18:04.927 INFO:teuthology.orchestra.run.smithi100.stdout: "target_max_objects": 0, 2024-09-17T07:18:04.927 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T07:18:04.927 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T07:18:04.927 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T07:18:04.927 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_min_flush_age": 0, 2024-09-17T07:18:04.927 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_min_evict_age": 0, 2024-09-17T07:18:04.927 INFO:teuthology.orchestra.run.smithi100.stdout: "erasure_code_profile": "", 2024-09-17T07:18:04.928 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_params": { 2024-09-17T07:18:04.928 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "none" 2024-09-17T07:18:04.928 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.928 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_period": 0, 2024-09-17T07:18:04.928 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_count": 0, 2024-09-17T07:18:04.928 INFO:teuthology.orchestra.run.smithi100.stdout: "use_gmt_hitset": true, 2024-09-17T07:18:04.928 INFO:teuthology.orchestra.run.smithi100.stdout: "min_read_recency_for_promote": 0, 2024-09-17T07:18:04.928 INFO:teuthology.orchestra.run.smithi100.stdout: "min_write_recency_for_promote": 0, 2024-09-17T07:18:04.928 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T07:18:04.928 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_search_last_n": 0, 2024-09-17T07:18:04.928 INFO:teuthology.orchestra.run.smithi100.stdout: "grade_table": [], 2024-09-17T07:18:04.928 INFO:teuthology.orchestra.run.smithi100.stdout: "stripe_width": 0, 2024-09-17T07:18:04.928 INFO:teuthology.orchestra.run.smithi100.stdout: "expected_num_objects": 0, 2024-09-17T07:18:04.929 INFO:teuthology.orchestra.run.smithi100.stdout: "fast_read": false, 2024-09-17T07:18:04.929 INFO:teuthology.orchestra.run.smithi100.stdout: "options": {}, 2024-09-17T07:18:04.929 INFO:teuthology.orchestra.run.smithi100.stdout: "application_metadata": { 2024-09-17T07:18:04.929 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw": {} 2024-09-17T07:18:04.929 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.929 INFO:teuthology.orchestra.run.smithi100.stdout: "read_balance": { 2024-09-17T07:18:04.929 INFO:teuthology.orchestra.run.smithi100.stdout: "score_acting": 2, 2024-09-17T07:18:04.929 INFO:teuthology.orchestra.run.smithi100.stdout: "score_stable": 2, 2024-09-17T07:18:04.929 INFO:teuthology.orchestra.run.smithi100.stdout: "optimal_score": 0.75, 2024-09-17T07:18:04.929 INFO:teuthology.orchestra.run.smithi100.stdout: "raw_score_acting": 1.5, 2024-09-17T07:18:04.929 INFO:teuthology.orchestra.run.smithi100.stdout: "raw_score_stable": 1.5, 2024-09-17T07:18:04.929 INFO:teuthology.orchestra.run.smithi100.stdout: "primary_affinity_weighted": 0.89598274230957031, 2024-09-17T07:18:04.929 INFO:teuthology.orchestra.run.smithi100.stdout: "average_primary_affinity": 0.89999997615814209, 2024-09-17T07:18:04.929 INFO:teuthology.orchestra.run.smithi100.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-09-17T07:18:04.930 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:18:04.930 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.930 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:18:04.930 INFO:teuthology.orchestra.run.smithi100.stdout: "pool": 6, 2024-09-17T07:18:04.930 INFO:teuthology.orchestra.run.smithi100.stdout: "pool_name": "default.rgw.meta", 2024-09-17T07:18:04.930 INFO:teuthology.orchestra.run.smithi100.stdout: "create_time": "2024-09-17T07:02:16.310620+0000", 2024-09-17T07:18:04.930 INFO:teuthology.orchestra.run.smithi100.stdout: "flags": 1, 2024-09-17T07:18:04.930 INFO:teuthology.orchestra.run.smithi100.stdout: "flags_names": "hashpspool", 2024-09-17T07:18:04.930 INFO:teuthology.orchestra.run.smithi100.stdout: "type": 1, 2024-09-17T07:18:04.930 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 3, 2024-09-17T07:18:04.930 INFO:teuthology.orchestra.run.smithi100.stdout: "min_size": 2, 2024-09-17T07:18:04.930 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_rule": 0, 2024-09-17T07:18:04.930 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_count": 0, 2024-09-17T07:18:04.931 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_target": 0, 2024-09-17T07:18:04.931 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T07:18:04.931 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T07:18:04.931 INFO:teuthology.orchestra.run.smithi100.stdout: "object_hash": 2, 2024-09-17T07:18:04.931 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_autoscale_mode": "off", 2024-09-17T07:18:04.931 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num": 32, 2024-09-17T07:18:04.931 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_placement_num": 32, 2024-09-17T07:18:04.931 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_placement_num_target": 32, 2024-09-17T07:18:04.931 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num_target": 32, 2024-09-17T07:18:04.931 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num_pending": 32, 2024-09-17T07:18:04.931 INFO:teuthology.orchestra.run.smithi100.stdout: "last_pg_merge_meta": { 2024-09-17T07:18:04.931 INFO:teuthology.orchestra.run.smithi100.stdout: "source_pgid": "0.0", 2024-09-17T07:18:04.932 INFO:teuthology.orchestra.run.smithi100.stdout: "ready_epoch": 0, 2024-09-17T07:18:04.932 INFO:teuthology.orchestra.run.smithi100.stdout: "last_epoch_started": 0, 2024-09-17T07:18:04.932 INFO:teuthology.orchestra.run.smithi100.stdout: "last_epoch_clean": 0, 2024-09-17T07:18:04.932 INFO:teuthology.orchestra.run.smithi100.stdout: "source_version": "0'0", 2024-09-17T07:18:04.932 INFO:teuthology.orchestra.run.smithi100.stdout: "target_version": "0'0" 2024-09-17T07:18:04.932 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.932 INFO:teuthology.orchestra.run.smithi100.stdout: "last_change": "75", 2024-09-17T07:18:04.932 INFO:teuthology.orchestra.run.smithi100.stdout: "last_force_op_resend": "0", 2024-09-17T07:18:04.932 INFO:teuthology.orchestra.run.smithi100.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-17T07:18:04.932 INFO:teuthology.orchestra.run.smithi100.stdout: "last_force_op_resend_preluminous": "0", 2024-09-17T07:18:04.932 INFO:teuthology.orchestra.run.smithi100.stdout: "auid": 0, 2024-09-17T07:18:04.932 INFO:teuthology.orchestra.run.smithi100.stdout: "snap_mode": "selfmanaged", 2024-09-17T07:18:04.933 INFO:teuthology.orchestra.run.smithi100.stdout: "snap_seq": 0, 2024-09-17T07:18:04.933 INFO:teuthology.orchestra.run.smithi100.stdout: "snap_epoch": 0, 2024-09-17T07:18:04.933 INFO:teuthology.orchestra.run.smithi100.stdout: "pool_snaps": [], 2024-09-17T07:18:04.933 INFO:teuthology.orchestra.run.smithi100.stdout: "removed_snaps": "[]", 2024-09-17T07:18:04.933 INFO:teuthology.orchestra.run.smithi100.stdout: "quota_max_bytes": 0, 2024-09-17T07:18:04.933 INFO:teuthology.orchestra.run.smithi100.stdout: "quota_max_objects": 0, 2024-09-17T07:18:04.933 INFO:teuthology.orchestra.run.smithi100.stdout: "tiers": [], 2024-09-17T07:18:04.933 INFO:teuthology.orchestra.run.smithi100.stdout: "tier_of": -1, 2024-09-17T07:18:04.933 INFO:teuthology.orchestra.run.smithi100.stdout: "read_tier": -1, 2024-09-17T07:18:04.933 INFO:teuthology.orchestra.run.smithi100.stdout: "write_tier": -1, 2024-09-17T07:18:04.933 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_mode": "none", 2024-09-17T07:18:04.933 INFO:teuthology.orchestra.run.smithi100.stdout: "target_max_bytes": 0, 2024-09-17T07:18:04.934 INFO:teuthology.orchestra.run.smithi100.stdout: "target_max_objects": 0, 2024-09-17T07:18:04.934 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T07:18:04.934 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T07:18:04.934 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T07:18:04.934 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_min_flush_age": 0, 2024-09-17T07:18:04.934 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_min_evict_age": 0, 2024-09-17T07:18:04.934 INFO:teuthology.orchestra.run.smithi100.stdout: "erasure_code_profile": "", 2024-09-17T07:18:04.934 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_params": { 2024-09-17T07:18:04.934 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "none" 2024-09-17T07:18:04.934 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.934 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_period": 0, 2024-09-17T07:18:04.934 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_count": 0, 2024-09-17T07:18:04.934 INFO:teuthology.orchestra.run.smithi100.stdout: "use_gmt_hitset": true, 2024-09-17T07:18:04.935 INFO:teuthology.orchestra.run.smithi100.stdout: "min_read_recency_for_promote": 0, 2024-09-17T07:18:04.935 INFO:teuthology.orchestra.run.smithi100.stdout: "min_write_recency_for_promote": 0, 2024-09-17T07:18:04.935 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T07:18:04.935 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_search_last_n": 0, 2024-09-17T07:18:04.935 INFO:teuthology.orchestra.run.smithi100.stdout: "grade_table": [], 2024-09-17T07:18:04.935 INFO:teuthology.orchestra.run.smithi100.stdout: "stripe_width": 0, 2024-09-17T07:18:04.935 INFO:teuthology.orchestra.run.smithi100.stdout: "expected_num_objects": 0, 2024-09-17T07:18:04.935 INFO:teuthology.orchestra.run.smithi100.stdout: "fast_read": false, 2024-09-17T07:18:04.935 INFO:teuthology.orchestra.run.smithi100.stdout: "options": { 2024-09-17T07:18:04.935 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_autoscale_bias": 4 2024-09-17T07:18:04.935 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.935 INFO:teuthology.orchestra.run.smithi100.stdout: "application_metadata": { 2024-09-17T07:18:04.936 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw": {} 2024-09-17T07:18:04.936 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.936 INFO:teuthology.orchestra.run.smithi100.stdout: "read_balance": { 2024-09-17T07:18:04.936 INFO:teuthology.orchestra.run.smithi100.stdout: "score_acting": 2, 2024-09-17T07:18:04.936 INFO:teuthology.orchestra.run.smithi100.stdout: "score_stable": 2, 2024-09-17T07:18:04.936 INFO:teuthology.orchestra.run.smithi100.stdout: "optimal_score": 0.75, 2024-09-17T07:18:04.936 INFO:teuthology.orchestra.run.smithi100.stdout: "raw_score_acting": 1.5, 2024-09-17T07:18:04.936 INFO:teuthology.orchestra.run.smithi100.stdout: "raw_score_stable": 1.5, 2024-09-17T07:18:04.936 INFO:teuthology.orchestra.run.smithi100.stdout: "primary_affinity_weighted": 0.89598274230957031, 2024-09-17T07:18:04.936 INFO:teuthology.orchestra.run.smithi100.stdout: "average_primary_affinity": 0.89999997615814209, 2024-09-17T07:18:04.936 INFO:teuthology.orchestra.run.smithi100.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-09-17T07:18:04.936 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:18:04.937 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.937 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:18:04.937 INFO:teuthology.orchestra.run.smithi100.stdout: "pool": 10, 2024-09-17T07:18:04.937 INFO:teuthology.orchestra.run.smithi100.stdout: "pool_name": "unique_pool_3", 2024-09-17T07:18:04.937 INFO:teuthology.orchestra.run.smithi100.stdout: "create_time": "2024-09-17T07:16:03.505171+0000", 2024-09-17T07:18:04.937 INFO:teuthology.orchestra.run.smithi100.stdout: "flags": 1, 2024-09-17T07:18:04.937 INFO:teuthology.orchestra.run.smithi100.stdout: "flags_names": "hashpspool", 2024-09-17T07:18:04.937 INFO:teuthology.orchestra.run.smithi100.stdout: "type": 1, 2024-09-17T07:18:04.937 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 3, 2024-09-17T07:18:04.937 INFO:teuthology.orchestra.run.smithi100.stdout: "min_size": 2, 2024-09-17T07:18:04.937 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_rule": 0, 2024-09-17T07:18:04.937 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_count": 0, 2024-09-17T07:18:04.938 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_target": 0, 2024-09-17T07:18:04.938 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T07:18:04.938 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T07:18:04.938 INFO:teuthology.orchestra.run.smithi100.stdout: "object_hash": 2, 2024-09-17T07:18:04.938 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_autoscale_mode": "off", 2024-09-17T07:18:04.938 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num": 26, 2024-09-17T07:18:04.938 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_placement_num": 26, 2024-09-17T07:18:04.938 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_placement_num_target": 26, 2024-09-17T07:18:04.938 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num_target": 26, 2024-09-17T07:18:04.938 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num_pending": 26, 2024-09-17T07:18:04.938 INFO:teuthology.orchestra.run.smithi100.stdout: "last_pg_merge_meta": { 2024-09-17T07:18:04.938 INFO:teuthology.orchestra.run.smithi100.stdout: "source_pgid": "0.0", 2024-09-17T07:18:04.939 INFO:teuthology.orchestra.run.smithi100.stdout: "ready_epoch": 0, 2024-09-17T07:18:04.939 INFO:teuthology.orchestra.run.smithi100.stdout: "last_epoch_started": 0, 2024-09-17T07:18:04.939 INFO:teuthology.orchestra.run.smithi100.stdout: "last_epoch_clean": 0, 2024-09-17T07:18:04.939 INFO:teuthology.orchestra.run.smithi100.stdout: "source_version": "0'0", 2024-09-17T07:18:04.939 INFO:teuthology.orchestra.run.smithi100.stdout: "target_version": "0'0" 2024-09-17T07:18:04.939 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.939 INFO:teuthology.orchestra.run.smithi100.stdout: "last_change": "334", 2024-09-17T07:18:04.939 INFO:teuthology.orchestra.run.smithi100.stdout: "last_force_op_resend": "0", 2024-09-17T07:18:04.939 INFO:teuthology.orchestra.run.smithi100.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-17T07:18:04.939 INFO:teuthology.orchestra.run.smithi100.stdout: "last_force_op_resend_preluminous": "332", 2024-09-17T07:18:04.939 INFO:teuthology.orchestra.run.smithi100.stdout: "auid": 0, 2024-09-17T07:18:04.939 INFO:teuthology.orchestra.run.smithi100.stdout: "snap_mode": "selfmanaged", 2024-09-17T07:18:04.939 INFO:teuthology.orchestra.run.smithi100.stdout: "snap_seq": 0, 2024-09-17T07:18:04.940 INFO:teuthology.orchestra.run.smithi100.stdout: "snap_epoch": 0, 2024-09-17T07:18:04.940 INFO:teuthology.orchestra.run.smithi100.stdout: "pool_snaps": [], 2024-09-17T07:18:04.940 INFO:teuthology.orchestra.run.smithi100.stdout: "removed_snaps": "[]", 2024-09-17T07:18:04.940 INFO:teuthology.orchestra.run.smithi100.stdout: "quota_max_bytes": 0, 2024-09-17T07:18:04.940 INFO:teuthology.orchestra.run.smithi100.stdout: "quota_max_objects": 0, 2024-09-17T07:18:04.940 INFO:teuthology.orchestra.run.smithi100.stdout: "tiers": [], 2024-09-17T07:18:04.940 INFO:teuthology.orchestra.run.smithi100.stdout: "tier_of": -1, 2024-09-17T07:18:04.940 INFO:teuthology.orchestra.run.smithi100.stdout: "read_tier": -1, 2024-09-17T07:18:04.940 INFO:teuthology.orchestra.run.smithi100.stdout: "write_tier": -1, 2024-09-17T07:18:04.940 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_mode": "none", 2024-09-17T07:18:04.940 INFO:teuthology.orchestra.run.smithi100.stdout: "target_max_bytes": 0, 2024-09-17T07:18:04.940 INFO:teuthology.orchestra.run.smithi100.stdout: "target_max_objects": 0, 2024-09-17T07:18:04.941 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T07:18:04.941 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T07:18:04.941 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T07:18:04.941 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_min_flush_age": 0, 2024-09-17T07:18:04.941 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_min_evict_age": 0, 2024-09-17T07:18:04.941 INFO:teuthology.orchestra.run.smithi100.stdout: "erasure_code_profile": "", 2024-09-17T07:18:04.941 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_params": { 2024-09-17T07:18:04.941 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "none" 2024-09-17T07:18:04.941 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.941 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_period": 0, 2024-09-17T07:18:04.941 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_count": 0, 2024-09-17T07:18:04.941 INFO:teuthology.orchestra.run.smithi100.stdout: "use_gmt_hitset": true, 2024-09-17T07:18:04.942 INFO:teuthology.orchestra.run.smithi100.stdout: "min_read_recency_for_promote": 0, 2024-09-17T07:18:04.942 INFO:teuthology.orchestra.run.smithi100.stdout: "min_write_recency_for_promote": 0, 2024-09-17T07:18:04.942 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T07:18:04.942 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_search_last_n": 0, 2024-09-17T07:18:04.942 INFO:teuthology.orchestra.run.smithi100.stdout: "grade_table": [], 2024-09-17T07:18:04.942 INFO:teuthology.orchestra.run.smithi100.stdout: "stripe_width": 0, 2024-09-17T07:18:04.942 INFO:teuthology.orchestra.run.smithi100.stdout: "expected_num_objects": 0, 2024-09-17T07:18:04.942 INFO:teuthology.orchestra.run.smithi100.stdout: "fast_read": false, 2024-09-17T07:18:04.942 INFO:teuthology.orchestra.run.smithi100.stdout: "options": {}, 2024-09-17T07:18:04.942 INFO:teuthology.orchestra.run.smithi100.stdout: "application_metadata": { 2024-09-17T07:18:04.942 INFO:teuthology.orchestra.run.smithi100.stdout: "rados": {} 2024-09-17T07:18:04.942 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.942 INFO:teuthology.orchestra.run.smithi100.stdout: "read_balance": { 2024-09-17T07:18:04.942 INFO:teuthology.orchestra.run.smithi100.stdout: "score_acting": 2.1600000858306885, 2024-09-17T07:18:04.943 INFO:teuthology.orchestra.run.smithi100.stdout: "score_stable": 2.4700000286102295, 2024-09-17T07:18:04.943 INFO:teuthology.orchestra.run.smithi100.stdout: "optimal_score": 0.75, 2024-09-17T07:18:04.943 INFO:teuthology.orchestra.run.smithi100.stdout: "raw_score_acting": 1.6200000047683716, 2024-09-17T07:18:04.943 INFO:teuthology.orchestra.run.smithi100.stdout: "raw_score_stable": 1.8500000238418579, 2024-09-17T07:18:04.943 INFO:teuthology.orchestra.run.smithi100.stdout: "primary_affinity_weighted": 0.89598274230957031, 2024-09-17T07:18:04.943 INFO:teuthology.orchestra.run.smithi100.stdout: "average_primary_affinity": 0.89999997615814209, 2024-09-17T07:18:04.943 INFO:teuthology.orchestra.run.smithi100.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-09-17T07:18:04.943 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:18:04.943 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:18:04.943 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-17T07:18:04.943 INFO:teuthology.orchestra.run.smithi100.stdout: "osds": [ 2024-09-17T07:18:04.943 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:18:04.944 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": 0, 2024-09-17T07:18:04.944 INFO:teuthology.orchestra.run.smithi100.stdout: "uuid": "27bee9ec-05e6-406c-8bf1-168098da0ec7", 2024-09-17T07:18:04.944 INFO:teuthology.orchestra.run.smithi100.stdout: "up": 1, 2024-09-17T07:18:04.944 INFO:teuthology.orchestra.run.smithi100.stdout: "in": 1, 2024-09-17T07:18:04.944 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 1, 2024-09-17T07:18:04.944 INFO:teuthology.orchestra.run.smithi100.stdout: "primary_affinity": 1, 2024-09-17T07:18:04.944 INFO:teuthology.orchestra.run.smithi100.stdout: "last_clean_begin": 0, 2024-09-17T07:18:04.944 INFO:teuthology.orchestra.run.smithi100.stdout: "last_clean_end": 0, 2024-09-17T07:18:04.944 INFO:teuthology.orchestra.run.smithi100.stdout: "up_from": 8, 2024-09-17T07:18:04.944 INFO:teuthology.orchestra.run.smithi100.stdout: "up_thru": 335, 2024-09-17T07:18:04.944 INFO:teuthology.orchestra.run.smithi100.stdout: "down_at": 0, 2024-09-17T07:18:04.944 INFO:teuthology.orchestra.run.smithi100.stdout: "lost_at": 0, 2024-09-17T07:18:04.945 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-17T07:18:04.945 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:18:04.945 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:18:04.945 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:18:04.945 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6801", 2024-09-17T07:18:04.945 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 3341017455 2024-09-17T07:18:04.945 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:18:04.945 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:18:04.945 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.945 INFO:teuthology.orchestra.run.smithi100.stdout: "cluster_addrs": { 2024-09-17T07:18:04.945 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:18:04.945 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:18:04.945 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:18:04.946 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6802", 2024-09-17T07:18:04.946 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 3341017455 2024-09-17T07:18:04.946 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:18:04.946 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:18:04.946 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.946 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_back_addrs": { 2024-09-17T07:18:04.946 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:18:04.946 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:18:04.946 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:18:04.946 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6804", 2024-09-17T07:18:04.947 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 3341017455 2024-09-17T07:18:04.947 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:18:04.947 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:18:04.947 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.947 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_front_addrs": { 2024-09-17T07:18:04.947 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:18:04.947 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:18:04.947 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:18:04.947 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6803", 2024-09-17T07:18:04.947 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 3341017455 2024-09-17T07:18:04.947 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:18:04.947 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:18:04.947 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.948 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.100:6801/3341017455", 2024-09-17T07:18:04.948 INFO:teuthology.orchestra.run.smithi100.stdout: "cluster_addr": "172.21.15.100:6802/3341017455", 2024-09-17T07:18:04.948 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_back_addr": "172.21.15.100:6804/3341017455", 2024-09-17T07:18:04.948 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_front_addr": "172.21.15.100:6803/3341017455", 2024-09-17T07:18:04.948 INFO:teuthology.orchestra.run.smithi100.stdout: "state": [ 2024-09-17T07:18:04.948 INFO:teuthology.orchestra.run.smithi100.stdout: "exists", 2024-09-17T07:18:04.948 INFO:teuthology.orchestra.run.smithi100.stdout: "up" 2024-09-17T07:18:04.948 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:18:04.948 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.948 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:18:04.948 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": 1, 2024-09-17T07:18:04.948 INFO:teuthology.orchestra.run.smithi100.stdout: "uuid": "b1a30c9c-eeaf-478e-9b62-2d63ad45873d", 2024-09-17T07:18:04.948 INFO:teuthology.orchestra.run.smithi100.stdout: "up": 0, 2024-09-17T07:18:04.949 INFO:teuthology.orchestra.run.smithi100.stdout: "in": 1, 2024-09-17T07:18:04.949 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 1, 2024-09-17T07:18:04.949 INFO:teuthology.orchestra.run.smithi100.stdout: "primary_affinity": 1, 2024-09-17T07:18:04.949 INFO:teuthology.orchestra.run.smithi100.stdout: "last_clean_begin": 0, 2024-09-17T07:18:04.949 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:18:04.950 INFO:teuthology.orchestra.run.smithi100.stdout: "last_clean_end": 0, 2024-09-17T07:18:04.950 INFO:teuthology.orchestra.run.smithi100.stdout: "up_from": 13, 2024-09-17T07:18:04.950 INFO:teuthology.orchestra.run.smithi100.stdout: "up_thru": 297, 2024-09-17T07:18:04.950 INFO:teuthology.orchestra.run.smithi100.stdout: "down_at": 303, 2024-09-17T07:18:04.950 INFO:teuthology.orchestra.run.smithi100.stdout: "lost_at": 0, 2024-09-17T07:18:04.951 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-17T07:18:04.951 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:18:04.951 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:18:04.951 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:18:04.951 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6805", 2024-09-17T07:18:04.951 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 1634750965 2024-09-17T07:18:04.951 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:18:04.951 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:18:04.951 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.951 INFO:teuthology.orchestra.run.smithi100.stdout: "cluster_addrs": { 2024-09-17T07:18:04.951 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:18:04.951 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:18:04.951 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:18:04.952 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6806", 2024-09-17T07:18:04.952 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 1634750965 2024-09-17T07:18:04.952 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:18:04.952 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:18:04.952 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.952 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_back_addrs": { 2024-09-17T07:18:04.952 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:18:04.952 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:18:04.952 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:18:04.952 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6808", 2024-09-17T07:18:04.952 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 1634750965 2024-09-17T07:18:04.952 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:18:04.953 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:18:04.953 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.953 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_front_addrs": { 2024-09-17T07:18:04.953 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:18:04.953 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:18:04.953 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:18:04.953 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6807", 2024-09-17T07:18:04.953 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 1634750965 2024-09-17T07:18:04.953 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:18:04.953 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:18:04.953 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.953 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.100:6805/1634750965", 2024-09-17T07:18:04.953 INFO:teuthology.orchestra.run.smithi100.stdout: "cluster_addr": "172.21.15.100:6806/1634750965", 2024-09-17T07:18:04.954 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_back_addr": "172.21.15.100:6808/1634750965", 2024-09-17T07:18:04.954 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_front_addr": "172.21.15.100:6807/1634750965", 2024-09-17T07:18:04.954 INFO:teuthology.orchestra.run.smithi100.stdout: "state": [ 2024-09-17T07:18:04.954 INFO:teuthology.orchestra.run.smithi100.stdout: "exists" 2024-09-17T07:18:04.954 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:18:04.954 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.954 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:18:04.954 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": 2, 2024-09-17T07:18:04.954 INFO:teuthology.orchestra.run.smithi100.stdout: "uuid": "ce542e0d-9915-465e-aa11-7b3ce13f694f", 2024-09-17T07:18:04.954 INFO:teuthology.orchestra.run.smithi100.stdout: "up": 1, 2024-09-17T07:18:04.954 INFO:teuthology.orchestra.run.smithi100.stdout: "in": 1, 2024-09-17T07:18:04.954 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 1, 2024-09-17T07:18:04.954 INFO:teuthology.orchestra.run.smithi100.stdout: "primary_affinity": 0.1678619384765625, 2024-09-17T07:18:04.955 INFO:teuthology.orchestra.run.smithi100.stdout: "last_clean_begin": 0, 2024-09-17T07:18:04.955 INFO:teuthology.orchestra.run.smithi100.stdout: "last_clean_end": 0, 2024-09-17T07:18:04.955 INFO:teuthology.orchestra.run.smithi100.stdout: "up_from": 18, 2024-09-17T07:18:04.955 INFO:teuthology.orchestra.run.smithi100.stdout: "up_thru": 335, 2024-09-17T07:18:04.955 INFO:teuthology.orchestra.run.smithi100.stdout: "down_at": 0, 2024-09-17T07:18:04.955 INFO:teuthology.orchestra.run.smithi100.stdout: "lost_at": 0, 2024-09-17T07:18:04.955 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-17T07:18:04.955 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:18:04.955 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:18:04.955 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:18:04.955 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6809", 2024-09-17T07:18:04.955 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 714579846 2024-09-17T07:18:04.956 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:18:04.956 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:18:04.956 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.956 INFO:teuthology.orchestra.run.smithi100.stdout: "cluster_addrs": { 2024-09-17T07:18:04.956 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:18:04.956 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:18:04.956 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:18:04.956 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6810", 2024-09-17T07:18:04.956 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 714579846 2024-09-17T07:18:04.956 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:18:04.956 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:18:04.956 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.956 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_back_addrs": { 2024-09-17T07:18:04.957 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:18:04.957 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:18:04.957 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:18:04.957 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6812", 2024-09-17T07:18:04.957 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 714579846 2024-09-17T07:18:04.957 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:18:04.957 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:18:04.957 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.957 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_front_addrs": { 2024-09-17T07:18:04.957 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:18:04.957 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:18:04.957 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:18:04.957 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6811", 2024-09-17T07:18:04.958 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 714579846 2024-09-17T07:18:04.958 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:18:04.958 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:18:04.958 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.958 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.100:6809/714579846", 2024-09-17T07:18:04.958 INFO:teuthology.orchestra.run.smithi100.stdout: "cluster_addr": "172.21.15.100:6810/714579846", 2024-09-17T07:18:04.958 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_back_addr": "172.21.15.100:6812/714579846", 2024-09-17T07:18:04.958 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_front_addr": "172.21.15.100:6811/714579846", 2024-09-17T07:18:04.958 INFO:teuthology.orchestra.run.smithi100.stdout: "state": [ 2024-09-17T07:18:04.958 INFO:teuthology.orchestra.run.smithi100.stdout: "exists", 2024-09-17T07:18:04.958 INFO:teuthology.orchestra.run.smithi100.stdout: "up" 2024-09-17T07:18:04.958 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:18:04.959 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.959 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:18:04.959 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": 3, 2024-09-17T07:18:04.959 INFO:teuthology.orchestra.run.smithi100.stdout: "uuid": "a1257575-3c90-41de-88f6-3960744351b1", 2024-09-17T07:18:04.959 INFO:teuthology.orchestra.run.smithi100.stdout: "up": 1, 2024-09-17T07:18:04.959 INFO:teuthology.orchestra.run.smithi100.stdout: "in": 0, 2024-09-17T07:18:04.959 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 0, 2024-09-17T07:18:04.959 INFO:teuthology.orchestra.run.smithi100.stdout: "primary_affinity": 1, 2024-09-17T07:18:04.959 INFO:teuthology.orchestra.run.smithi100.stdout: "last_clean_begin": 0, 2024-09-17T07:18:04.959 INFO:teuthology.orchestra.run.smithi100.stdout: "last_clean_end": 0, 2024-09-17T07:18:04.959 INFO:teuthology.orchestra.run.smithi100.stdout: "up_from": 27, 2024-09-17T07:18:04.959 INFO:teuthology.orchestra.run.smithi100.stdout: "up_thru": 168, 2024-09-17T07:18:04.959 INFO:teuthology.orchestra.run.smithi100.stdout: "down_at": 0, 2024-09-17T07:18:04.960 INFO:teuthology.orchestra.run.smithi100.stdout: "lost_at": 0, 2024-09-17T07:18:04.960 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-17T07:18:04.960 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:18:04.960 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:18:04.960 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:18:04.960 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6813", 2024-09-17T07:18:04.960 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 2315956572 2024-09-17T07:18:04.960 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:18:04.960 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:18:04.960 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.960 INFO:teuthology.orchestra.run.smithi100.stdout: "cluster_addrs": { 2024-09-17T07:18:04.960 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:18:04.960 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:18:04.961 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:18:04.961 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6814", 2024-09-17T07:18:04.961 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 2315956572 2024-09-17T07:18:04.961 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:18:04.961 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:18:04.961 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.961 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_back_addrs": { 2024-09-17T07:18:04.961 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:18:04.961 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:18:04.961 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:18:04.961 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6816", 2024-09-17T07:18:04.961 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 2315956572 2024-09-17T07:18:04.962 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:18:04.962 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:18:04.962 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.962 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_front_addrs": { 2024-09-17T07:18:04.962 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:18:04.962 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:18:04.962 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:18:04.962 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6815", 2024-09-17T07:18:04.962 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 2315956572 2024-09-17T07:18:04.962 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:18:04.962 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:18:04.962 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.962 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.100:6813/2315956572", 2024-09-17T07:18:04.963 INFO:teuthology.orchestra.run.smithi100.stdout: "cluster_addr": "172.21.15.100:6814/2315956572", 2024-09-17T07:18:04.963 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_back_addr": "172.21.15.100:6816/2315956572", 2024-09-17T07:18:04.963 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_front_addr": "172.21.15.100:6815/2315956572", 2024-09-17T07:18:04.963 INFO:teuthology.orchestra.run.smithi100.stdout: "state": [ 2024-09-17T07:18:04.963 INFO:teuthology.orchestra.run.smithi100.stdout: "exists", 2024-09-17T07:18:04.963 INFO:teuthology.orchestra.run.smithi100.stdout: "up" 2024-09-17T07:18:04.963 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:18:04.963 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.963 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:18:04.963 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": 4, 2024-09-17T07:18:04.963 INFO:teuthology.orchestra.run.smithi100.stdout: "uuid": "e44e8922-2c5e-44e3-b556-d4dfbc499649", 2024-09-17T07:18:04.963 INFO:teuthology.orchestra.run.smithi100.stdout: "up": 1, 2024-09-17T07:18:04.963 INFO:teuthology.orchestra.run.smithi100.stdout: "in": 1, 2024-09-17T07:18:04.964 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 1, 2024-09-17T07:18:04.964 INFO:teuthology.orchestra.run.smithi100.stdout: "primary_affinity": 1, 2024-09-17T07:18:04.964 INFO:teuthology.orchestra.run.smithi100.stdout: "last_clean_begin": 0, 2024-09-17T07:18:04.964 INFO:teuthology.orchestra.run.smithi100.stdout: "last_clean_end": 0, 2024-09-17T07:18:04.964 INFO:teuthology.orchestra.run.smithi100.stdout: "up_from": 33, 2024-09-17T07:18:04.964 INFO:teuthology.orchestra.run.smithi100.stdout: "up_thru": 335, 2024-09-17T07:18:04.964 INFO:teuthology.orchestra.run.smithi100.stdout: "down_at": 0, 2024-09-17T07:18:04.964 INFO:teuthology.orchestra.run.smithi100.stdout: "lost_at": 0, 2024-09-17T07:18:04.964 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-17T07:18:04.964 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:18:04.964 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:18:04.964 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:18:04.964 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.135:6800", 2024-09-17T07:18:04.965 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 1148820922 2024-09-17T07:18:04.965 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:18:04.965 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:18:04.965 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.965 INFO:teuthology.orchestra.run.smithi100.stdout: "cluster_addrs": { 2024-09-17T07:18:04.965 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:18:04.965 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:18:04.965 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:18:04.965 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.135:6801", 2024-09-17T07:18:04.965 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 1148820922 2024-09-17T07:18:04.965 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:18:04.965 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:18:04.966 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.966 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_back_addrs": { 2024-09-17T07:18:04.966 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:18:04.966 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:18:04.966 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:18:04.966 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.135:6803", 2024-09-17T07:18:04.966 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 1148820922 2024-09-17T07:18:04.966 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:18:04.966 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:18:04.966 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.966 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_front_addrs": { 2024-09-17T07:18:04.966 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:18:04.966 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:18:04.967 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:18:04.967 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.135:6802", 2024-09-17T07:18:04.967 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 1148820922 2024-09-17T07:18:04.967 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:18:04.967 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:18:04.967 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.967 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.135:6800/1148820922", 2024-09-17T07:18:04.967 INFO:teuthology.orchestra.run.smithi100.stdout: "cluster_addr": "172.21.15.135:6801/1148820922", 2024-09-17T07:18:04.967 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_back_addr": "172.21.15.135:6803/1148820922", 2024-09-17T07:18:04.967 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_front_addr": "172.21.15.135:6802/1148820922", 2024-09-17T07:18:04.967 INFO:teuthology.orchestra.run.smithi100.stdout: "state": [ 2024-09-17T07:18:04.967 INFO:teuthology.orchestra.run.smithi100.stdout: "exists", 2024-09-17T07:18:04.967 INFO:teuthology.orchestra.run.smithi100.stdout: "up" 2024-09-17T07:18:04.968 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:18:04.968 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.968 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:18:04.968 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": 5, 2024-09-17T07:18:04.968 INFO:teuthology.orchestra.run.smithi100.stdout: "uuid": "91e4a362-fcdb-41b9-b54e-716591ba6d79", 2024-09-17T07:18:04.968 INFO:teuthology.orchestra.run.smithi100.stdout: "up": 1, 2024-09-17T07:18:04.968 INFO:teuthology.orchestra.run.smithi100.stdout: "in": 1, 2024-09-17T07:18:04.968 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 1, 2024-09-17T07:18:04.968 INFO:teuthology.orchestra.run.smithi100.stdout: "primary_affinity": 1, 2024-09-17T07:18:04.968 INFO:teuthology.orchestra.run.smithi100.stdout: "last_clean_begin": 0, 2024-09-17T07:18:04.968 INFO:teuthology.orchestra.run.smithi100.stdout: "last_clean_end": 0, 2024-09-17T07:18:04.968 INFO:teuthology.orchestra.run.smithi100.stdout: "up_from": 42, 2024-09-17T07:18:04.968 INFO:teuthology.orchestra.run.smithi100.stdout: "up_thru": 334, 2024-09-17T07:18:04.969 INFO:teuthology.orchestra.run.smithi100.stdout: "down_at": 0, 2024-09-17T07:18:04.969 INFO:teuthology.orchestra.run.smithi100.stdout: "lost_at": 0, 2024-09-17T07:18:04.969 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-17T07:18:04.969 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:18:04.969 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:18:04.969 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:18:04.969 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.135:6804", 2024-09-17T07:18:04.969 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 2256520893 2024-09-17T07:18:04.969 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:18:04.969 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:18:04.969 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.969 INFO:teuthology.orchestra.run.smithi100.stdout: "cluster_addrs": { 2024-09-17T07:18:04.970 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:18:04.970 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:18:04.970 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:18:04.970 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.135:6805", 2024-09-17T07:18:04.970 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 2256520893 2024-09-17T07:18:04.970 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:18:04.970 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:18:04.970 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.970 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_back_addrs": { 2024-09-17T07:18:04.970 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:18:04.970 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:18:04.970 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:18:04.970 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.135:6807", 2024-09-17T07:18:04.971 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 2256520893 2024-09-17T07:18:04.971 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:18:04.971 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:18:04.971 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.971 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_front_addrs": { 2024-09-17T07:18:04.971 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:18:04.971 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:18:04.971 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:18:04.971 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.135:6806", 2024-09-17T07:18:04.971 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 2256520893 2024-09-17T07:18:04.971 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:18:04.971 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:18:04.971 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.972 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.135:6804/2256520893", 2024-09-17T07:18:04.972 INFO:teuthology.orchestra.run.smithi100.stdout: "cluster_addr": "172.21.15.135:6805/2256520893", 2024-09-17T07:18:04.972 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_back_addr": "172.21.15.135:6807/2256520893", 2024-09-17T07:18:04.972 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_front_addr": "172.21.15.135:6806/2256520893", 2024-09-17T07:18:04.972 INFO:teuthology.orchestra.run.smithi100.stdout: "state": [ 2024-09-17T07:18:04.972 INFO:teuthology.orchestra.run.smithi100.stdout: "exists", 2024-09-17T07:18:04.972 INFO:teuthology.orchestra.run.smithi100.stdout: "up" 2024-09-17T07:18:04.972 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:18:04.972 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.972 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:18:04.972 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": 6, 2024-09-17T07:18:04.972 INFO:teuthology.orchestra.run.smithi100.stdout: "uuid": "de799f64-ebd2-4a46-91f6-c2f276bee21f", 2024-09-17T07:18:04.973 INFO:teuthology.orchestra.run.smithi100.stdout: "up": 1, 2024-09-17T07:18:04.973 INFO:teuthology.orchestra.run.smithi100.stdout: "in": 1, 2024-09-17T07:18:04.973 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 1, 2024-09-17T07:18:04.973 INFO:teuthology.orchestra.run.smithi100.stdout: "primary_affinity": 1, 2024-09-17T07:18:04.973 INFO:teuthology.orchestra.run.smithi100.stdout: "last_clean_begin": 0, 2024-09-17T07:18:04.973 INFO:teuthology.orchestra.run.smithi100.stdout: "last_clean_end": 0, 2024-09-17T07:18:04.973 INFO:teuthology.orchestra.run.smithi100.stdout: "up_from": 50, 2024-09-17T07:18:04.973 INFO:teuthology.orchestra.run.smithi100.stdout: "up_thru": 334, 2024-09-17T07:18:04.973 INFO:teuthology.orchestra.run.smithi100.stdout: "down_at": 0, 2024-09-17T07:18:04.973 INFO:teuthology.orchestra.run.smithi100.stdout: "lost_at": 0, 2024-09-17T07:18:04.973 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-17T07:18:04.973 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:18:04.973 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:18:04.974 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:18:04.974 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.135:6808", 2024-09-17T07:18:04.974 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 2750184152 2024-09-17T07:18:04.974 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:18:04.974 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:18:04.974 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.974 INFO:teuthology.orchestra.run.smithi100.stdout: "cluster_addrs": { 2024-09-17T07:18:04.974 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:18:04.974 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:18:04.974 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:18:04.974 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.135:6809", 2024-09-17T07:18:04.974 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 2750184152 2024-09-17T07:18:04.975 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:18:04.975 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:18:04.975 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.975 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_back_addrs": { 2024-09-17T07:18:04.975 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:18:04.975 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:18:04.975 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:18:04.975 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.135:6811", 2024-09-17T07:18:04.975 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 2750184152 2024-09-17T07:18:04.975 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:18:04.975 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:18:04.975 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.975 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_front_addrs": { 2024-09-17T07:18:04.976 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:18:04.976 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:18:04.976 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:18:04.976 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.135:6810", 2024-09-17T07:18:04.976 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 2750184152 2024-09-17T07:18:04.976 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:18:04.976 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:18:04.976 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.976 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.135:6808/2750184152", 2024-09-17T07:18:04.976 INFO:teuthology.orchestra.run.smithi100.stdout: "cluster_addr": "172.21.15.135:6809/2750184152", 2024-09-17T07:18:04.976 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_back_addr": "172.21.15.135:6811/2750184152", 2024-09-17T07:18:04.976 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_front_addr": "172.21.15.135:6810/2750184152", 2024-09-17T07:18:04.976 INFO:teuthology.orchestra.run.smithi100.stdout: "state": [ 2024-09-17T07:18:04.977 INFO:teuthology.orchestra.run.smithi100.stdout: "exists", 2024-09-17T07:18:04.977 INFO:teuthology.orchestra.run.smithi100.stdout: "up" 2024-09-17T07:18:04.977 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:18:04.977 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.977 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:18:04.977 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": 7, 2024-09-17T07:18:04.977 INFO:teuthology.orchestra.run.smithi100.stdout: "uuid": "830932c7-5de1-44ff-a840-e53a6e859bda", 2024-09-17T07:18:04.977 INFO:teuthology.orchestra.run.smithi100.stdout: "up": 1, 2024-09-17T07:18:04.977 INFO:teuthology.orchestra.run.smithi100.stdout: "in": 1, 2024-09-17T07:18:04.977 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 1, 2024-09-17T07:18:04.977 INFO:teuthology.orchestra.run.smithi100.stdout: "primary_affinity": 1, 2024-09-17T07:18:04.977 INFO:teuthology.orchestra.run.smithi100.stdout: "last_clean_begin": 0, 2024-09-17T07:18:04.977 INFO:teuthology.orchestra.run.smithi100.stdout: "last_clean_end": 0, 2024-09-17T07:18:04.978 INFO:teuthology.orchestra.run.smithi100.stdout: "up_from": 60, 2024-09-17T07:18:04.978 INFO:teuthology.orchestra.run.smithi100.stdout: "up_thru": 334, 2024-09-17T07:18:04.978 INFO:teuthology.orchestra.run.smithi100.stdout: "down_at": 0, 2024-09-17T07:18:04.978 INFO:teuthology.orchestra.run.smithi100.stdout: "lost_at": 0, 2024-09-17T07:18:04.978 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-17T07:18:04.978 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:18:04.978 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:18:04.978 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:18:04.978 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.135:6812", 2024-09-17T07:18:04.978 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 160304773 2024-09-17T07:18:04.978 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:18:04.978 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:18:04.978 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.979 INFO:teuthology.orchestra.run.smithi100.stdout: "cluster_addrs": { 2024-09-17T07:18:04.979 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:18:04.979 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:18:04.979 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:18:04.979 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.135:6813", 2024-09-17T07:18:04.979 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 160304773 2024-09-17T07:18:04.979 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:18:04.979 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:18:04.979 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.979 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_back_addrs": { 2024-09-17T07:18:04.979 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:18:04.979 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:18:04.980 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:18:04.980 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.135:6815", 2024-09-17T07:18:04.980 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 160304773 2024-09-17T07:18:04.980 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:18:04.980 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:18:04.980 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.980 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_front_addrs": { 2024-09-17T07:18:04.980 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:18:04.980 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:18:04.980 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:18:04.980 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.135:6814", 2024-09-17T07:18:04.980 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 160304773 2024-09-17T07:18:04.980 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:18:04.981 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:18:04.981 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.981 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.135:6812/160304773", 2024-09-17T07:18:04.981 INFO:teuthology.orchestra.run.smithi100.stdout: "cluster_addr": "172.21.15.135:6813/160304773", 2024-09-17T07:18:04.981 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_back_addr": "172.21.15.135:6815/160304773", 2024-09-17T07:18:04.981 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_front_addr": "172.21.15.135:6814/160304773", 2024-09-17T07:18:04.981 INFO:teuthology.orchestra.run.smithi100.stdout: "state": [ 2024-09-17T07:18:04.981 INFO:teuthology.orchestra.run.smithi100.stdout: "exists", 2024-09-17T07:18:04.981 INFO:teuthology.orchestra.run.smithi100.stdout: "up" 2024-09-17T07:18:04.981 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:18:04.981 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:18:04.981 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-17T07:18:04.981 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_xinfo": [ 2024-09-17T07:18:04.982 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:18:04.982 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": 0, 2024-09-17T07:18:04.982 INFO:teuthology.orchestra.run.smithi100.stdout: "down_stamp": "0.000000", 2024-09-17T07:18:04.982 INFO:teuthology.orchestra.run.smithi100.stdout: "laggy_probability": 0, 2024-09-17T07:18:04.982 INFO:teuthology.orchestra.run.smithi100.stdout: "laggy_interval": 0, 2024-09-17T07:18:04.982 INFO:teuthology.orchestra.run.smithi100.stdout: "features": 4540138322906710015, 2024-09-17T07:18:04.982 INFO:teuthology.orchestra.run.smithi100.stdout: "old_weight": 0, 2024-09-17T07:18:04.982 INFO:teuthology.orchestra.run.smithi100.stdout: "last_purged_snaps_scrub": "2024-09-17T07:17:28.778213+0000", 2024-09-17T07:18:04.982 INFO:teuthology.orchestra.run.smithi100.stdout: "dead_epoch": 0 2024-09-17T07:18:04.982 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.982 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:18:04.982 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": 1, 2024-09-17T07:18:04.983 INFO:teuthology.orchestra.run.smithi100.stdout: "down_stamp": "0.000000", 2024-09-17T07:18:04.983 INFO:teuthology.orchestra.run.smithi100.stdout: "laggy_probability": 0, 2024-09-17T07:18:04.983 INFO:teuthology.orchestra.run.smithi100.stdout: "laggy_interval": 0, 2024-09-17T07:18:04.983 INFO:teuthology.orchestra.run.smithi100.stdout: "features": 4540138322906710015, 2024-09-17T07:18:04.983 INFO:teuthology.orchestra.run.smithi100.stdout: "old_weight": 0, 2024-09-17T07:18:04.983 INFO:teuthology.orchestra.run.smithi100.stdout: "last_purged_snaps_scrub": "2024-09-17T07:15:49.673764+0000", 2024-09-17T07:18:04.983 INFO:teuthology.orchestra.run.smithi100.stdout: "dead_epoch": 302 2024-09-17T07:18:04.983 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.983 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:18:04.983 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": 2, 2024-09-17T07:18:04.983 INFO:teuthology.orchestra.run.smithi100.stdout: "down_stamp": "0.000000", 2024-09-17T07:18:04.983 INFO:teuthology.orchestra.run.smithi100.stdout: "laggy_probability": 0, 2024-09-17T07:18:04.983 INFO:teuthology.orchestra.run.smithi100.stdout: "laggy_interval": 0, 2024-09-17T07:18:04.984 INFO:teuthology.orchestra.run.smithi100.stdout: "features": 4540138322906710015, 2024-09-17T07:18:04.984 INFO:teuthology.orchestra.run.smithi100.stdout: "old_weight": 0, 2024-09-17T07:18:04.984 INFO:teuthology.orchestra.run.smithi100.stdout: "last_purged_snaps_scrub": "2024-09-17T07:16:50.095599+0000", 2024-09-17T07:18:04.984 INFO:teuthology.orchestra.run.smithi100.stdout: "dead_epoch": 0 2024-09-17T07:18:04.984 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.984 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:18:04.984 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": 3, 2024-09-17T07:18:04.984 INFO:teuthology.orchestra.run.smithi100.stdout: "down_stamp": "0.000000", 2024-09-17T07:18:04.984 INFO:teuthology.orchestra.run.smithi100.stdout: "laggy_probability": 0, 2024-09-17T07:18:04.984 INFO:teuthology.orchestra.run.smithi100.stdout: "laggy_interval": 0, 2024-09-17T07:18:04.984 INFO:teuthology.orchestra.run.smithi100.stdout: "features": 4540138322906710015, 2024-09-17T07:18:04.984 INFO:teuthology.orchestra.run.smithi100.stdout: "old_weight": 65536, 2024-09-17T07:18:04.984 INFO:teuthology.orchestra.run.smithi100.stdout: "last_purged_snaps_scrub": "2024-09-17T07:17:14.621911+0000", 2024-09-17T07:18:04.985 INFO:teuthology.orchestra.run.smithi100.stdout: "dead_epoch": 0 2024-09-17T07:18:04.985 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.985 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:18:04.985 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": 4, 2024-09-17T07:18:04.985 INFO:teuthology.orchestra.run.smithi100.stdout: "down_stamp": "0.000000", 2024-09-17T07:18:04.985 INFO:teuthology.orchestra.run.smithi100.stdout: "laggy_probability": 0, 2024-09-17T07:18:04.985 INFO:teuthology.orchestra.run.smithi100.stdout: "laggy_interval": 0, 2024-09-17T07:18:04.985 INFO:teuthology.orchestra.run.smithi100.stdout: "features": 4540138322906710015, 2024-09-17T07:18:04.985 INFO:teuthology.orchestra.run.smithi100.stdout: "old_weight": 0, 2024-09-17T07:18:04.985 INFO:teuthology.orchestra.run.smithi100.stdout: "last_purged_snaps_scrub": "2024-09-17T07:17:41.847503+0000", 2024-09-17T07:18:04.985 INFO:teuthology.orchestra.run.smithi100.stdout: "dead_epoch": 0 2024-09-17T07:18:04.985 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.986 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:18:04.986 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": 5, 2024-09-17T07:18:04.986 INFO:teuthology.orchestra.run.smithi100.stdout: "down_stamp": "0.000000", 2024-09-17T07:18:04.986 INFO:teuthology.orchestra.run.smithi100.stdout: "laggy_probability": 0, 2024-09-17T07:18:04.986 INFO:teuthology.orchestra.run.smithi100.stdout: "laggy_interval": 0, 2024-09-17T07:18:04.986 INFO:teuthology.orchestra.run.smithi100.stdout: "features": 4540138322906710015, 2024-09-17T07:18:04.986 INFO:teuthology.orchestra.run.smithi100.stdout: "old_weight": 0, 2024-09-17T07:18:04.986 INFO:teuthology.orchestra.run.smithi100.stdout: "last_purged_snaps_scrub": "2024-09-17T07:17:22.883378+0000", 2024-09-17T07:18:04.986 INFO:teuthology.orchestra.run.smithi100.stdout: "dead_epoch": 0 2024-09-17T07:18:04.986 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.986 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:18:04.986 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": 6, 2024-09-17T07:18:04.986 INFO:teuthology.orchestra.run.smithi100.stdout: "down_stamp": "0.000000", 2024-09-17T07:18:04.986 INFO:teuthology.orchestra.run.smithi100.stdout: "laggy_probability": 0, 2024-09-17T07:18:04.987 INFO:teuthology.orchestra.run.smithi100.stdout: "laggy_interval": 0, 2024-09-17T07:18:04.987 INFO:teuthology.orchestra.run.smithi100.stdout: "features": 4540138322906710015, 2024-09-17T07:18:04.987 INFO:teuthology.orchestra.run.smithi100.stdout: "old_weight": 0, 2024-09-17T07:18:04.987 INFO:teuthology.orchestra.run.smithi100.stdout: "last_purged_snaps_scrub": "2024-09-17T07:17:30.012275+0000", 2024-09-17T07:18:04.987 INFO:teuthology.orchestra.run.smithi100.stdout: "dead_epoch": 0 2024-09-17T07:18:04.987 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.987 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:18:04.987 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": 7, 2024-09-17T07:18:04.987 INFO:teuthology.orchestra.run.smithi100.stdout: "down_stamp": "0.000000", 2024-09-17T07:18:04.987 INFO:teuthology.orchestra.run.smithi100.stdout: "laggy_probability": 0, 2024-09-17T07:18:04.987 INFO:teuthology.orchestra.run.smithi100.stdout: "laggy_interval": 0, 2024-09-17T07:18:04.987 INFO:teuthology.orchestra.run.smithi100.stdout: "features": 4540138322906710015, 2024-09-17T07:18:04.987 INFO:teuthology.orchestra.run.smithi100.stdout: "old_weight": 0, 2024-09-17T07:18:04.988 INFO:teuthology.orchestra.run.smithi100.stdout: "last_purged_snaps_scrub": "2024-09-17T07:17:38.758280+0000", 2024-09-17T07:18:04.988 INFO:teuthology.orchestra.run.smithi100.stdout: "dead_epoch": 0 2024-09-17T07:18:04.988 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:18:04.988 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-17T07:18:04.988 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_upmap": [], 2024-09-17T07:18:04.988 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_upmap_items": [ 2024-09-17T07:18:04.988 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:18:04.988 INFO:teuthology.orchestra.run.smithi100.stdout: "pgid": "2.f", 2024-09-17T07:18:04.988 INFO:teuthology.orchestra.run.smithi100.stdout: "mappings": [ 2024-09-17T07:18:04.988 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:18:04.988 INFO:teuthology.orchestra.run.smithi100.stdout: "from": 7, 2024-09-17T07:18:04.988 INFO:teuthology.orchestra.run.smithi100.stdout: "to": 1 2024-09-17T07:18:04.988 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:18:04.989 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:18:04.989 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.989 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:18:04.989 INFO:teuthology.orchestra.run.smithi100.stdout: "pgid": "2.1a", 2024-09-17T07:18:04.989 INFO:teuthology.orchestra.run.smithi100.stdout: "mappings": [ 2024-09-17T07:18:04.989 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:18:04.989 INFO:teuthology.orchestra.run.smithi100.stdout: "from": 7, 2024-09-17T07:18:04.989 INFO:teuthology.orchestra.run.smithi100.stdout: "to": 2 2024-09-17T07:18:04.989 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:18:04.989 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:18:04.989 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.989 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:18:04.989 INFO:teuthology.orchestra.run.smithi100.stdout: "pgid": "4.6", 2024-09-17T07:18:04.990 INFO:teuthology.orchestra.run.smithi100.stdout: "mappings": [ 2024-09-17T07:18:04.990 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:18:04.990 INFO:teuthology.orchestra.run.smithi100.stdout: "from": 1, 2024-09-17T07:18:04.990 INFO:teuthology.orchestra.run.smithi100.stdout: "to": 5 2024-09-17T07:18:04.990 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:18:04.990 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:18:04.990 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.990 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:18:04.990 INFO:teuthology.orchestra.run.smithi100.stdout: "pgid": "4.9", 2024-09-17T07:18:04.990 INFO:teuthology.orchestra.run.smithi100.stdout: "mappings": [ 2024-09-17T07:18:04.990 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:18:04.990 INFO:teuthology.orchestra.run.smithi100.stdout: "from": 1, 2024-09-17T07:18:04.990 INFO:teuthology.orchestra.run.smithi100.stdout: "to": 0 2024-09-17T07:18:04.990 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:18:04.991 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:18:04.991 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.991 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:18:04.991 INFO:teuthology.orchestra.run.smithi100.stdout: "pgid": "4.10", 2024-09-17T07:18:04.991 INFO:teuthology.orchestra.run.smithi100.stdout: "mappings": [ 2024-09-17T07:18:04.991 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:18:04.991 INFO:teuthology.orchestra.run.smithi100.stdout: "from": 1, 2024-09-17T07:18:04.991 INFO:teuthology.orchestra.run.smithi100.stdout: "to": 2 2024-09-17T07:18:04.991 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:18:04.991 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:18:04.991 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.991 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:18:04.991 INFO:teuthology.orchestra.run.smithi100.stdout: "pgid": "4.1e", 2024-09-17T07:18:04.992 INFO:teuthology.orchestra.run.smithi100.stdout: "mappings": [ 2024-09-17T07:18:04.992 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:18:04.992 INFO:teuthology.orchestra.run.smithi100.stdout: "from": 1, 2024-09-17T07:18:04.992 INFO:teuthology.orchestra.run.smithi100.stdout: "to": 5 2024-09-17T07:18:04.992 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:18:04.992 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:18:04.992 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.992 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:18:04.992 INFO:teuthology.orchestra.run.smithi100.stdout: "pgid": "4.1f", 2024-09-17T07:18:04.992 INFO:teuthology.orchestra.run.smithi100.stdout: "mappings": [ 2024-09-17T07:18:04.992 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:18:04.992 INFO:teuthology.orchestra.run.smithi100.stdout: "from": 1, 2024-09-17T07:18:04.993 INFO:teuthology.orchestra.run.smithi100.stdout: "to": 2 2024-09-17T07:18:04.993 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:18:04.993 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:18:04.993 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:18:04.993 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-17T07:18:04.993 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_upmap_primaries": [], 2024-09-17T07:18:04.993 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_temp": [ 2024-09-17T07:18:04.993 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:18:04.993 INFO:teuthology.orchestra.run.smithi100.stdout: "pgid": "10.10", 2024-09-17T07:18:04.993 INFO:teuthology.orchestra.run.smithi100.stdout: "osds": [ 2024-09-17T07:18:04.993 INFO:teuthology.orchestra.run.smithi100.stdout: 4, 2024-09-17T07:18:04.993 INFO:teuthology.orchestra.run.smithi100.stdout: 6 2024-09-17T07:18:04.993 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:18:04.994 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.994 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:18:04.994 INFO:teuthology.orchestra.run.smithi100.stdout: "pgid": "10.11", 2024-09-17T07:18:04.994 INFO:teuthology.orchestra.run.smithi100.stdout: "osds": [ 2024-09-17T07:18:04.994 INFO:teuthology.orchestra.run.smithi100.stdout: 0, 2024-09-17T07:18:04.994 INFO:teuthology.orchestra.run.smithi100.stdout: 7, 2024-09-17T07:18:04.994 INFO:teuthology.orchestra.run.smithi100.stdout: 4 2024-09-17T07:18:04.994 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:18:04.994 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.994 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:18:04.994 INFO:teuthology.orchestra.run.smithi100.stdout: "pgid": "10.12", 2024-09-17T07:18:04.994 INFO:teuthology.orchestra.run.smithi100.stdout: "osds": [ 2024-09-17T07:18:04.994 INFO:teuthology.orchestra.run.smithi100.stdout: 0, 2024-09-17T07:18:04.994 INFO:teuthology.orchestra.run.smithi100.stdout: 4 2024-09-17T07:18:04.995 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:18:04.995 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.995 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:18:04.995 INFO:teuthology.orchestra.run.smithi100.stdout: "pgid": "10.13", 2024-09-17T07:18:04.995 INFO:teuthology.orchestra.run.smithi100.stdout: "osds": [ 2024-09-17T07:18:04.995 INFO:teuthology.orchestra.run.smithi100.stdout: 2, 2024-09-17T07:18:04.995 INFO:teuthology.orchestra.run.smithi100.stdout: 4, 2024-09-17T07:18:04.995 INFO:teuthology.orchestra.run.smithi100.stdout: 5 2024-09-17T07:18:04.995 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:18:04.995 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.995 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:18:04.995 INFO:teuthology.orchestra.run.smithi100.stdout: "pgid": "10.14", 2024-09-17T07:18:04.995 INFO:teuthology.orchestra.run.smithi100.stdout: "osds": [ 2024-09-17T07:18:04.996 INFO:teuthology.orchestra.run.smithi100.stdout: 4, 2024-09-17T07:18:04.996 INFO:teuthology.orchestra.run.smithi100.stdout: 0 2024-09-17T07:18:04.996 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:18:04.996 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.996 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:18:04.996 INFO:teuthology.orchestra.run.smithi100.stdout: "pgid": "10.15", 2024-09-17T07:18:04.996 INFO:teuthology.orchestra.run.smithi100.stdout: "osds": [ 2024-09-17T07:18:04.996 INFO:teuthology.orchestra.run.smithi100.stdout: 0, 2024-09-17T07:18:04.996 INFO:teuthology.orchestra.run.smithi100.stdout: 2, 2024-09-17T07:18:04.996 INFO:teuthology.orchestra.run.smithi100.stdout: 7 2024-09-17T07:18:04.996 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:18:04.996 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.997 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:18:04.997 INFO:teuthology.orchestra.run.smithi100.stdout: "pgid": "10.16", 2024-09-17T07:18:04.997 INFO:teuthology.orchestra.run.smithi100.stdout: "osds": [ 2024-09-17T07:18:04.997 INFO:teuthology.orchestra.run.smithi100.stdout: 0, 2024-09-17T07:18:04.997 INFO:teuthology.orchestra.run.smithi100.stdout: 5, 2024-09-17T07:18:04.997 INFO:teuthology.orchestra.run.smithi100.stdout: 4 2024-09-17T07:18:04.997 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:18:04.997 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.997 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:18:04.997 INFO:teuthology.orchestra.run.smithi100.stdout: "pgid": "10.17", 2024-09-17T07:18:04.997 INFO:teuthology.orchestra.run.smithi100.stdout: "osds": [ 2024-09-17T07:18:04.997 INFO:teuthology.orchestra.run.smithi100.stdout: 2, 2024-09-17T07:18:04.997 INFO:teuthology.orchestra.run.smithi100.stdout: 6, 2024-09-17T07:18:04.998 INFO:teuthology.orchestra.run.smithi100.stdout: 5 2024-09-17T07:18:04.998 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:18:04.998 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.998 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:18:04.998 INFO:teuthology.orchestra.run.smithi100.stdout: "pgid": "10.18", 2024-09-17T07:18:04.998 INFO:teuthology.orchestra.run.smithi100.stdout: "osds": [ 2024-09-17T07:18:04.998 INFO:teuthology.orchestra.run.smithi100.stdout: 2, 2024-09-17T07:18:04.998 INFO:teuthology.orchestra.run.smithi100.stdout: 0, 2024-09-17T07:18:04.998 INFO:teuthology.orchestra.run.smithi100.stdout: 7 2024-09-17T07:18:04.998 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:18:04.998 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:04.998 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:18:04.998 INFO:teuthology.orchestra.run.smithi100.stdout: "pgid": "10.19", 2024-09-17T07:18:04.999 INFO:teuthology.orchestra.run.smithi100.stdout: "osds": [ 2024-09-17T07:18:04.999 INFO:teuthology.orchestra.run.smithi100.stdout: 2, 2024-09-17T07:18:04.999 INFO:teuthology.orchestra.run.smithi100.stdout: 5, 2024-09-17T07:18:04.999 INFO:teuthology.orchestra.run.smithi100.stdout: 7 2024-09-17T07:18:04.999 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:18:04.999 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:18:04.999 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-17T07:18:04.999 INFO:teuthology.orchestra.run.smithi100.stdout: "primary_temp": [], 2024-09-17T07:18:04.999 INFO:teuthology.orchestra.run.smithi100.stdout: "blocklist": { 2024-09-17T07:18:04.999 INFO:teuthology.orchestra.run.smithi100.stdout: "172.21.15.100:0/3796664909": "2024-09-18T06:57:35.678309+0000", 2024-09-17T07:18:04.999 INFO:teuthology.orchestra.run.smithi100.stdout: "172.21.15.100:0/2225586217": "2024-09-18T07:02:46.981486+0000", 2024-09-17T07:18:04.999 INFO:teuthology.orchestra.run.smithi100.stdout: "172.21.15.100:0/1626706829": "2024-09-18T06:57:35.678309+0000", 2024-09-17T07:18:04.999 INFO:teuthology.orchestra.run.smithi100.stdout: "172.21.15.100:0/3452172206": "2024-09-18T06:57:35.678309+0000", 2024-09-17T07:18:05.000 INFO:teuthology.orchestra.run.smithi100.stdout: "172.21.15.100:6800/3992296558": "2024-09-18T06:57:58.526309+0000", 2024-09-17T07:18:05.000 INFO:teuthology.orchestra.run.smithi100.stdout: "172.21.15.100:6800/3498574214": "2024-09-18T06:57:35.678309+0000", 2024-09-17T07:18:05.000 INFO:teuthology.orchestra.run.smithi100.stdout: "172.21.15.100:0/1365215411": "2024-09-18T06:57:58.526309+0000", 2024-09-17T07:18:05.000 INFO:teuthology.orchestra.run.smithi100.stdout: "172.21.15.100:0/3362169179": "2024-09-18T07:02:46.981486+0000", 2024-09-17T07:18:05.000 INFO:teuthology.orchestra.run.smithi100.stdout: "172.21.15.100:0/2573779025": "2024-09-18T07:02:46.981486+0000", 2024-09-17T07:18:05.000 INFO:teuthology.orchestra.run.smithi100.stdout: "172.21.15.100:0/380524491": "2024-09-18T07:02:46.981486+0000", 2024-09-17T07:18:05.000 INFO:teuthology.orchestra.run.smithi100.stdout: "172.21.15.100:0/1657291976": "2024-09-18T06:57:58.526309+0000", 2024-09-17T07:18:05.000 INFO:teuthology.orchestra.run.smithi100.stdout: "172.21.15.100:0/1676314112": "2024-09-18T07:02:46.981486+0000", 2024-09-17T07:18:05.000 INFO:teuthology.orchestra.run.smithi100.stdout: "172.21.15.100:0/1255197255": "2024-09-18T06:57:58.526309+0000", 2024-09-17T07:18:05.000 INFO:teuthology.orchestra.run.smithi100.stdout: "172.21.15.100:6800/3472658181": "2024-09-18T07:02:46.981486+0000" 2024-09-17T07:18:05.000 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:05.000 INFO:teuthology.orchestra.run.smithi100.stdout: "range_blocklist": {}, 2024-09-17T07:18:05.000 INFO:teuthology.orchestra.run.smithi100.stdout: "erasure_code_profiles": { 2024-09-17T07:18:05.001 INFO:teuthology.orchestra.run.smithi100.stdout: "default": { 2024-09-17T07:18:05.001 INFO:teuthology.orchestra.run.smithi100.stdout: "crush-failure-domain": "osd", 2024-09-17T07:18:05.001 INFO:teuthology.orchestra.run.smithi100.stdout: "k": "2", 2024-09-17T07:18:05.001 INFO:teuthology.orchestra.run.smithi100.stdout: "m": "1", 2024-09-17T07:18:05.001 INFO:teuthology.orchestra.run.smithi100.stdout: "plugin": "jerasure", 2024-09-17T07:18:05.001 INFO:teuthology.orchestra.run.smithi100.stdout: "technique": "reed_sol_van" 2024-09-17T07:18:05.001 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:18:05.001 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:18:05.001 INFO:teuthology.orchestra.run.smithi100.stdout: "removed_snaps_queue": [], 2024-09-17T07:18:05.001 INFO:teuthology.orchestra.run.smithi100.stdout: "new_removed_snaps": [], 2024-09-17T07:18:05.001 INFO:teuthology.orchestra.run.smithi100.stdout: "new_purged_snaps": [], 2024-09-17T07:18:05.001 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_node_flags": {}, 2024-09-17T07:18:05.001 INFO:teuthology.orchestra.run.smithi100.stdout: "device_class_flags": {}, 2024-09-17T07:18:05.002 INFO:teuthology.orchestra.run.smithi100.stdout: "stretch_mode": { 2024-09-17T07:18:05.002 INFO:teuthology.orchestra.run.smithi100.stdout: "stretch_mode_enabled": false, 2024-09-17T07:18:05.002 INFO:teuthology.orchestra.run.smithi100.stdout: "stretch_bucket_count": 0, 2024-09-17T07:18:05.002 INFO:teuthology.orchestra.run.smithi100.stdout: "degraded_stretch_mode": 0, 2024-09-17T07:18:05.002 INFO:teuthology.orchestra.run.smithi100.stdout: "recovering_stretch_mode": 0, 2024-09-17T07:18:05.002 INFO:teuthology.orchestra.run.smithi100.stdout: "stretch_mode_bucket": 0 2024-09-17T07:18:05.002 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:18:05.002 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-17T07:18:05.032 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:05.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:05.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:18:05.136 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:18:05.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:05.168 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:05.169 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:18:05.269 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:18:05.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:05.315 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:05.315 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:18:05.416 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:18:05.440 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:05.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:05.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:18:05.544 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:18:05.545 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:05 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1873033489' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-17T07:18:05.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:05.583 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:05.584 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:18:05.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:05 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1873033489' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-17T07:18:05.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:05 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1873033489' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-17T07:18:05.684 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:18:05.715 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:05.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:05.716 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:18:05.818 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:18:05.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:05.844 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:05.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:18:05.945 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:18:05.993 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:05.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:05.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:18:06.090 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:18:06.095 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:18:06.164 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:18:05 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:18:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:18:06.167 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:06 smithi100 ceph-mon[29092]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-17T07:18:06.167 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:06 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 26602/346338 objects degraded (7.681%), 31 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-09-17T07:18:06.167 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:06 smithi100 ceph-mon[29092]: pgmap v722: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 5 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 36 active+undersized, 2 active+remapped+backfill_toofull, 28 active+undersized+degraded, 84 active+clean; 6.8 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.2 KiB/s rd, 179 MiB/s wr, 4.14k op/s; 25634/333915 objects degraded (7.677%); 52567/333915 objects misplaced (15.743%); 71 MiB/s, 1.14k objects/s recovering 2024-09-17T07:18:06.167 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:06 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:18:06.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:06.174 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:06.174 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:18:06.275 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:18:06.304 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:06.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:06.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:18:06.406 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:18:06.434 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:06.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:06.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:18:06.536 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:18:06.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:06 smithi135 ceph-mon[33238]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-17T07:18:06.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:06 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 26602/346338 objects degraded (7.681%), 31 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-09-17T07:18:06.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:06 smithi135 ceph-mon[33238]: pgmap v722: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 5 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 36 active+undersized, 2 active+remapped+backfill_toofull, 28 active+undersized+degraded, 84 active+clean; 6.8 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.2 KiB/s rd, 179 MiB/s wr, 4.14k op/s; 25634/333915 objects degraded (7.677%); 52567/333915 objects misplaced (15.743%); 71 MiB/s, 1.14k objects/s recovering 2024-09-17T07:18:06.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:06 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:18:06.572 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:06.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:06.573 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:18:06.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:06 smithi100 ceph-mon[39462]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-17T07:18:06.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:06 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 26602/346338 objects degraded (7.681%), 31 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-09-17T07:18:06.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:06 smithi100 ceph-mon[39462]: pgmap v722: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 5 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 36 active+undersized, 2 active+remapped+backfill_toofull, 28 active+undersized+degraded, 84 active+clean; 6.8 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.2 KiB/s rd, 179 MiB/s wr, 4.14k op/s; 25634/333915 objects degraded (7.677%); 52567/333915 objects misplaced (15.743%); 71 MiB/s, 1.14k objects/s recovering 2024-09-17T07:18:06.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:06 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:18:06.674 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:18:06.709 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:06.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:06.710 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:18:06.762 INFO:tasks.thrashosds.thrasher:j is {'epoch': 338, 'fsid': 'e848267c-74c1-11ef-bceb-c7b262605968', 'created': '2024-09-17T06:57:05.010532+0000', 'modified': '2024-09-17T07:18:00.248814+0000', 'last_up_change': '2024-09-17T07:16:16.355276+0000', 'last_in_change': '2024-09-17T07:09:23.364026+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 10, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'reef', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-17T07:00:08.816136+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '205', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.8959827423095703, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-09-17T07:02:10.759826+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_acting': 2.509999990463257, 'score_stable': 2.509999990463257, 'optimal_score': 0.75, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.8959827423095703, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 3, 'pool_name': 'datapool', 'create_time': '2024-09-17T07:02:10.991289+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 72, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.6299999952316284, 'raw_score_acting': 1.6699999570846558, 'raw_score_stable': 1.6699999570846558, 'primary_affinity_weighted': 0.8959827423095703, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-17T07:02:12.291462+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_acting': 2.75, 'score_stable': 2.75, 'optimal_score': 0.75, 'raw_score_acting': 2.059999942779541, 'raw_score_stable': 2.059999942779541, 'primary_affinity_weighted': 0.8959827423095703, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-17T07:02:14.253214+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '73', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 0.75, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.8959827423095703, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-17T07:02:16.310620+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 0.75, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.8959827423095703, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 10, 'pool_name': 'unique_pool_3', 'create_time': '2024-09-17T07:16:03.505171+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 26, 'pg_placement_num': 26, 'pg_placement_num_target': 26, 'pg_num_target': 26, 'pg_num_pending': 26, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '334', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '332', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_acting': 2.1600000858306885, 'score_stable': 2.4700000286102295, 'optimal_score': 0.75, 'raw_score_acting': 1.6200000047683716, 'raw_score_stable': 1.850000023841858, 'primary_affinity_weighted': 0.8959827423095703, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}], 'osds': [{'osd': 0, 'uuid': '27bee9ec-05e6-406c-8bf1-168098da0ec7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 335, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6801', 'nonce': 3341017455}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6802', 'nonce': 3341017455}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6804', 'nonce': 3341017455}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6803', 'nonce': 3341017455}]}, 'public_addr': '172.21.15.100:6801/3341017455', 'cluster_addr': '172.21.15.100:6802/3341017455', 'heartbeat_back_addr': '172.21.15.100:6804/3341017455', 'heartbeat_front_addr': '172.21.15.100:6803/3341017455', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'b1a30c9c-eeaf-478e-9b62-2d63ad45873d', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 297, 'down_at': 303, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6805', 'nonce': 1634750965}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6806', 'nonce': 1634750965}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6808', 'nonce': 1634750965}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6807', 'nonce': 1634750965}]}, 'public_addr': '172.21.15.100:6805/1634750965', 'cluster_addr': '172.21.15.100:6806/1634750965', 'heartbeat_back_addr': '172.21.15.100:6808/1634750965', 'heartbeat_front_addr': '172.21.15.100:6807/1634750965', 'state': ['exists']}, {'osd': 2, 'uuid': 'ce542e0d-9915-465e-aa11-7b3ce13f694f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.1678619384765625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 335, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6809', 'nonce': 714579846}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6810', 'nonce': 714579846}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6812', 'nonce': 714579846}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6811', 'nonce': 714579846}]}, 'public_addr': '172.21.15.100:6809/714579846', 'cluster_addr': '172.21.15.100:6810/714579846', 'heartbeat_back_addr': '172.21.15.100:6812/714579846', 'heartbeat_front_addr': '172.21.15.100:6811/714579846', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'a1257575-3c90-41de-88f6-3960744351b1', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 27, 'up_thru': 168, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6813', 'nonce': 2315956572}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6814', 'nonce': 2315956572}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6816', 'nonce': 2315956572}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6815', 'nonce': 2315956572}]}, 'public_addr': '172.21.15.100:6813/2315956572', 'cluster_addr': '172.21.15.100:6814/2315956572', 'heartbeat_back_addr': '172.21.15.100:6816/2315956572', 'heartbeat_front_addr': '172.21.15.100:6815/2315956572', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'e44e8922-2c5e-44e3-b556-d4dfbc499649', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 33, 'up_thru': 335, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.135:6800', 'nonce': 1148820922}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.135:6801', 'nonce': 1148820922}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.135:6803', 'nonce': 1148820922}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.135:6802', 'nonce': 1148820922}]}, 'public_addr': '172.21.15.135:6800/1148820922', 'cluster_addr': '172.21.15.135:6801/1148820922', 'heartbeat_back_addr': '172.21.15.135:6803/1148820922', 'heartbeat_front_addr': '172.21.15.135:6802/1148820922', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '91e4a362-fcdb-41b9-b54e-716591ba6d79', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 42, 'up_thru': 334, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.135:6804', 'nonce': 2256520893}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.135:6805', 'nonce': 2256520893}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.135:6807', 'nonce': 2256520893}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.135:6806', 'nonce': 2256520893}]}, 'public_addr': '172.21.15.135:6804/2256520893', 'cluster_addr': '172.21.15.135:6805/2256520893', 'heartbeat_back_addr': '172.21.15.135:6807/2256520893', 'heartbeat_front_addr': '172.21.15.135:6806/2256520893', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'de799f64-ebd2-4a46-91f6-c2f276bee21f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 50, 'up_thru': 334, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.135:6808', 'nonce': 2750184152}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.135:6809', 'nonce': 2750184152}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.135:6811', 'nonce': 2750184152}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.135:6810', 'nonce': 2750184152}]}, 'public_addr': '172.21.15.135:6808/2750184152', 'cluster_addr': '172.21.15.135:6809/2750184152', 'heartbeat_back_addr': '172.21.15.135:6811/2750184152', 'heartbeat_front_addr': '172.21.15.135:6810/2750184152', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '830932c7-5de1-44ff-a840-e53a6e859bda', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 60, 'up_thru': 334, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.135:6812', 'nonce': 160304773}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.135:6813', 'nonce': 160304773}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.135:6815', 'nonce': 160304773}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.135:6814', 'nonce': 160304773}]}, 'public_addr': '172.21.15.135:6812/160304773', 'cluster_addr': '172.21.15.135:6813/160304773', 'heartbeat_back_addr': '172.21.15.135:6815/160304773', 'heartbeat_front_addr': '172.21.15.135:6814/160304773', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T07:17:28.778213+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T07:15:49.673764+0000', 'dead_epoch': 302}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T07:16:50.095599+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-17T07:17:14.621911+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T07:17:41.847503+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T07:17:22.883378+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T07:17:30.012275+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T07:17:38.758280+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '2.f', 'mappings': [{'from': 7, 'to': 1}]}, {'pgid': '2.1a', 'mappings': [{'from': 7, 'to': 2}]}, {'pgid': '4.6', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.9', 'mappings': [{'from': 1, 'to': 0}]}, {'pgid': '4.10', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1e', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.1f', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '10.10', 'osds': [4, 6]}, {'pgid': '10.11', 'osds': [0, 7, 4]}, {'pgid': '10.12', 'osds': [0, 4]}, {'pgid': '10.13', 'osds': [2, 4, 5]}, {'pgid': '10.14', 'osds': [4, 0]}, {'pgid': '10.15', 'osds': [0, 2, 7]}, {'pgid': '10.16', 'osds': [0, 5, 4]}, {'pgid': '10.17', 'osds': [2, 6, 5]}, {'pgid': '10.18', 'osds': [2, 0, 7]}, {'pgid': '10.19', 'osds': [2, 5, 7]}], 'primary_temp': [], 'blocklist': {'172.21.15.100:0/3796664909': '2024-09-18T06:57:35.678309+0000', '172.21.15.100:0/2225586217': '2024-09-18T07:02:46.981486+0000', '172.21.15.100:0/1626706829': '2024-09-18T06:57:35.678309+0000', '172.21.15.100:0/3452172206': '2024-09-18T06:57:35.678309+0000', '172.21.15.100:6800/3992296558': '2024-09-18T06:57:58.526309+0000', '172.21.15.100:6800/3498574214': '2024-09-18T06:57:35.678309+0000', '172.21.15.100:0/1365215411': '2024-09-18T06:57:58.526309+0000', '172.21.15.100:0/3362169179': '2024-09-18T07:02:46.981486+0000', '172.21.15.100:0/2573779025': '2024-09-18T07:02:46.981486+0000', '172.21.15.100:0/380524491': '2024-09-18T07:02:46.981486+0000', '172.21.15.100:0/1657291976': '2024-09-18T06:57:58.526309+0000', '172.21.15.100:0/1676314112': '2024-09-18T07:02:46.981486+0000', '172.21.15.100:0/1255197255': '2024-09-18T06:57:58.526309+0000', '172.21.15.100:6800/3472658181': '2024-09-18T07:02:46.981486+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-17T07:18:06.762 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T07:18:06.812 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:18:06.855 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:06.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:06.856 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:18:06.957 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:18:06.998 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:06.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:06.999 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:18:07.100 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:18:07.132 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:07.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:07.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:18:07.234 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:18:07.271 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:07.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:07.272 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:18:07.373 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:18:07.410 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:07.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:07.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:18:07.512 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:18:07.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:07 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:18:07.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:07 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:18:07.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:07 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:18:07.551 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:07.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:07.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:18:07.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:07 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:18:07.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:07 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:18:07.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:07 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:18:07.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:07 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:18:07.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:07 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:18:07.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:07 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:18:07.653 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:18:07.690 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:07.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:07.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:18:07.792 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:18:07.838 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:07.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:07.839 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:18:07.939 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:18:07.978 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:07.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:07.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:18:08.080 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:18:08.118 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:08.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:08.119 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:18:08.220 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:18:08.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:08.260 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:08.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:18:08.362 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:18:08.398 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:08.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:08.399 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:18:08.501 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:18:08.529 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:08.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:08.530 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:18:08.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:08 smithi135 ceph-mon[33238]: pgmap v723: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 5 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 36 active+undersized, 2 active+remapped+backfill_toofull, 28 active+undersized+degraded, 84 active+clean; 6.8 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1023 B/s rd, 144 MiB/s wr, 3.34k op/s; 25634/333915 objects degraded (7.677%); 52567/333915 objects misplaced (15.743%); 57 MiB/s, 915 objects/s recovering 2024-09-17T07:18:08.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:08 smithi100 ceph-mon[29092]: pgmap v723: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 5 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 36 active+undersized, 2 active+remapped+backfill_toofull, 28 active+undersized+degraded, 84 active+clean; 6.8 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1023 B/s rd, 144 MiB/s wr, 3.34k op/s; 25634/333915 objects degraded (7.677%); 52567/333915 objects misplaced (15.743%); 57 MiB/s, 915 objects/s recovering 2024-09-17T07:18:08.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:08 smithi100 ceph-mon[39462]: pgmap v723: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 5 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 36 active+undersized, 2 active+remapped+backfill_toofull, 28 active+undersized+degraded, 84 active+clean; 6.8 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1023 B/s rd, 144 MiB/s wr, 3.34k op/s; 25634/333915 objects degraded (7.677%); 52567/333915 objects misplaced (15.743%); 57 MiB/s, 915 objects/s recovering 2024-09-17T07:18:08.631 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:18:08.661 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:08.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:08.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:18:08.763 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:18:08.802 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:08.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:08.803 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:18:08.904 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:18:08.932 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:08.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:08.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:18:09.034 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:18:09.067 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:09.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:09.069 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:18:09.158 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-17T07:18:09.158 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:18:09.170 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:18:09.211 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:09.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:09.212 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:18:09.271 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:18:09 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:18:09.313 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:18:09.350 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:09.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:09.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:18:09.452 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:18:09.486 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:09.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:09.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:18:09.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:09 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:18:09.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:09 smithi135 ceph-mon[33238]: pgmap v724: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 5 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 36 active+undersized, 2 active+remapped+backfill_toofull, 28 active+undersized+degraded, 84 active+clean; 6.5 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.29k op/s; 23241/318264 objects degraded (7.302%); 48512/318264 objects misplaced (15.243%); 64 MiB/s, 1.03k objects/s recovering 2024-09-17T07:18:09.588 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:18:09.615 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:09.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:09.616 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:18:09.717 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:18:09.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:09.760 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:09.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:18:09.861 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:18:09.873 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:09 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:18:09.873 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:09 smithi100 ceph-mon[29092]: pgmap v724: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 5 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 36 active+undersized, 2 active+remapped+backfill_toofull, 28 active+undersized+degraded, 84 active+clean; 6.5 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.29k op/s; 23241/318264 objects degraded (7.302%); 48512/318264 objects misplaced (15.243%); 64 MiB/s, 1.03k objects/s recovering 2024-09-17T07:18:09.873 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:09 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:18:09.873 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:09 smithi100 ceph-mon[39462]: pgmap v724: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 5 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 36 active+undersized, 2 active+remapped+backfill_toofull, 28 active+undersized+degraded, 84 active+clean; 6.5 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.29k op/s; 23241/318264 objects degraded (7.302%); 48512/318264 objects misplaced (15.243%); 64 MiB/s, 1.03k objects/s recovering 2024-09-17T07:18:09.911 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:09.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:09.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:18:10.013 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:18:10.041 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:10.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:10.042 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:18:10.143 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:18:10.145 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:18:10.145 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:18:10.188 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-17T07:18:10.214 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:10.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:10.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:18:10.315 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:18:10.351 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:10.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:10.352 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:18:10.453 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:18:10.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:10.487 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:10.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:18:10.588 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:18:10.627 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:10.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:10.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:18:10.729 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:18:10.747 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:10 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 23241/318264 objects degraded (7.302%), 31 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-09-17T07:18:10.747 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:10 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 23241/318264 objects degraded (7.302%), 31 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-09-17T07:18:10.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:10.768 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:10.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:18:10.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:10 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 23241/318264 objects degraded (7.302%), 31 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-09-17T07:18:10.870 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:18:10.899 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:10.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:10.900 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:18:11.001 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:18:11.030 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:11.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:11.031 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:18:11.132 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:18:11.168 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:11.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:11.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:18:11.270 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:18:11.301 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:11.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:11.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:18:11.403 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:18:11.448 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:11.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:11.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:18:11.550 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:18:11.587 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:11.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:11.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:18:11.689 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:18:11.718 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:11.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:11.719 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:18:11.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:11 smithi135 ceph-mon[33238]: pgmap v725: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 5 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 36 active+undersized, 2 active+remapped+backfill_toofull, 28 active+undersized+degraded, 84 active+clean; 6.4 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 924 B/s rd, 0 B/s wr, 1.21k op/s; 22817/313977 objects degraded (7.267%); 45908/313977 objects misplaced (14.621%); 59 MiB/s, 940 objects/s recovering 2024-09-17T07:18:11.820 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:18:11.857 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:11.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:11.858 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:18:11.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:11 smithi100 ceph-mon[29092]: pgmap v725: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 5 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 36 active+undersized, 2 active+remapped+backfill_toofull, 28 active+undersized+degraded, 84 active+clean; 6.4 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 924 B/s rd, 0 B/s wr, 1.21k op/s; 22817/313977 objects degraded (7.267%); 45908/313977 objects misplaced (14.621%); 59 MiB/s, 940 objects/s recovering 2024-09-17T07:18:11.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:11 smithi100 ceph-mon[39462]: pgmap v725: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 5 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 36 active+undersized, 2 active+remapped+backfill_toofull, 28 active+undersized+degraded, 84 active+clean; 6.4 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 924 B/s rd, 0 B/s wr, 1.21k op/s; 22817/313977 objects degraded (7.267%); 45908/313977 objects misplaced (14.621%); 59 MiB/s, 940 objects/s recovering 2024-09-17T07:18:11.959 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:18:11.989 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:11.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:11.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:18:12.091 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:18:12.117 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:12.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:12.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:18:12.219 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:18:12.249 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:12.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:12.250 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:18:12.351 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:18:12.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:12.373 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:12.373 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:18:12.474 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:18:12.503 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:12.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:12.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:18:12.605 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:18:12.635 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:12.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:12.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:18:12.737 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:18:12.769 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:12.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:12.770 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:18:12.871 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:18:12.901 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:12.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:12.902 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:18:13.003 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:18:13.030 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:13.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:13.031 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:18:13.132 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:18:13.162 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:13.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:13.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:18:13.264 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:18:13.293 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:13.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:13.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:18:13.395 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:18:13.425 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:13.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:13.426 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:18:13.527 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:18:13.556 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:13.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:13.558 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:18:13.659 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:18:13.697 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:13.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:13.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:18:13.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:13 smithi135 ceph-mon[33238]: osdmap e339: 8 total, 7 up, 7 in 2024-09-17T07:18:13.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:13 smithi135 ceph-mon[33238]: pgmap v727: 158 pgs: 1 remapped+peering, 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 36 active+undersized, 2 active+remapped+backfill_toofull, 28 active+undersized+degraded, 84 active+clean; 6.2 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.29k op/s; 21013/307692 objects degraded (6.829%); 39247/307692 objects misplaced (12.755%); 64 MiB/s, 1.02k objects/s recovering 2024-09-17T07:18:13.799 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:18:13.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:13.835 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:13.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:18:13.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:13 smithi100 ceph-mon[29092]: osdmap e339: 8 total, 7 up, 7 in 2024-09-17T07:18:13.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:13 smithi100 ceph-mon[29092]: pgmap v727: 158 pgs: 1 remapped+peering, 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 36 active+undersized, 2 active+remapped+backfill_toofull, 28 active+undersized+degraded, 84 active+clean; 6.2 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.29k op/s; 21013/307692 objects degraded (6.829%); 39247/307692 objects misplaced (12.755%); 64 MiB/s, 1.02k objects/s recovering 2024-09-17T07:18:13.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:13 smithi100 ceph-mon[39462]: osdmap e339: 8 total, 7 up, 7 in 2024-09-17T07:18:13.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:13 smithi100 ceph-mon[39462]: pgmap v727: 158 pgs: 1 remapped+peering, 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 36 active+undersized, 2 active+remapped+backfill_toofull, 28 active+undersized+degraded, 84 active+clean; 6.2 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.29k op/s; 21013/307692 objects degraded (6.829%); 39247/307692 objects misplaced (12.755%); 64 MiB/s, 1.02k objects/s recovering 2024-09-17T07:18:13.937 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:18:13.977 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:13.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:13.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:18:14.079 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:18:14.106 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:14.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:14.107 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:18:14.208 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:18:14.230 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:14.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:14.231 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:18:14.332 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:18:14.361 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:14.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:14.362 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:18:14.463 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:18:14.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:14.500 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:14.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:18:14.601 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:18:14.632 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:14.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:14.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:18:14.733 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:18:14.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:14.758 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:14.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:18:14.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:14 smithi135 ceph-mon[33238]: osdmap e340: 8 total, 7 up, 7 in 2024-09-17T07:18:14.859 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:18:14.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:14 smithi100 ceph-mon[29092]: osdmap e340: 8 total, 7 up, 7 in 2024-09-17T07:18:14.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:14 smithi100 ceph-mon[39462]: osdmap e340: 8 total, 7 up, 7 in 2024-09-17T07:18:14.901 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:14.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:14.902 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:18:15.003 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:18:15.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:15.045 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:15.045 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:18:15.146 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:18:15.184 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:15.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:15.185 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:18:15.286 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:18:15.308 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:15.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:15.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:18:15.410 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:18:15.441 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:15.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:15.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:18:15.544 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:18:15.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:15.582 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:15.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:18:15.683 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:18:15.716 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:15.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:15.717 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:18:15.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:15 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 21013/307692 objects degraded (6.829%), 31 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-09-17T07:18:15.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:15 smithi135 ceph-mon[33238]: osdmap e341: 8 total, 7 up, 7 in 2024-09-17T07:18:15.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:15 smithi135 ceph-mon[33238]: pgmap v730: 158 pgs: 1 remapped+peering, 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 36 active+undersized, 2 active+remapped+backfill_toofull, 28 active+undersized+degraded, 84 active+clean; 6.1 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.12k op/s; 20154/298098 objects degraded (6.761%); 36669/298098 objects misplaced (12.301%); 71 MiB/s, 1.13k objects/s recovering 2024-09-17T07:18:15.818 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:18:15.851 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:15.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:15.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:18:15.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:15 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 21013/307692 objects degraded (6.829%), 31 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-09-17T07:18:15.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:15 smithi100 ceph-mon[29092]: osdmap e341: 8 total, 7 up, 7 in 2024-09-17T07:18:15.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:15 smithi100 ceph-mon[29092]: pgmap v730: 158 pgs: 1 remapped+peering, 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 36 active+undersized, 2 active+remapped+backfill_toofull, 28 active+undersized+degraded, 84 active+clean; 6.1 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.12k op/s; 20154/298098 objects degraded (6.761%); 36669/298098 objects misplaced (12.301%); 71 MiB/s, 1.13k objects/s recovering 2024-09-17T07:18:15.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:15 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 21013/307692 objects degraded (6.829%), 31 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-09-17T07:18:15.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:15 smithi100 ceph-mon[39462]: osdmap e341: 8 total, 7 up, 7 in 2024-09-17T07:18:15.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:15 smithi100 ceph-mon[39462]: pgmap v730: 158 pgs: 1 remapped+peering, 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 36 active+undersized, 2 active+remapped+backfill_toofull, 28 active+undersized+degraded, 84 active+clean; 6.1 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.12k op/s; 20154/298098 objects degraded (6.761%); 36669/298098 objects misplaced (12.301%); 71 MiB/s, 1.13k objects/s recovering 2024-09-17T07:18:15.953 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:18:15.978 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:15.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:15.979 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:18:16.080 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:18:16.107 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:16.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:16.108 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:18:16.209 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:18:16.235 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:16.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:16.236 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:18:16.336 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:18:16.360 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:16.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:16.361 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:18:16.376 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:18:15 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:18:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:18:16.463 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:18:16.484 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:16.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:16.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:18:16.586 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:18:16.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:16.621 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:16.621 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:18:16.722 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:18:16.747 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:16.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:16.748 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:18:16.849 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:18:16.881 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:16.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:16.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:18:16.983 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:18:17.004 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:17.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:17.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:18:17.106 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:18:17.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:17.131 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:17.131 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:18:17.232 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:18:17.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:17.256 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:17.256 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:18:17.357 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:18:17.379 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:17.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:17.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:18:17.481 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:18:17.501 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:17.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:17.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:18:17.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:17 smithi135 ceph-mon[33238]: osdmap e342: 8 total, 7 up, 7 in 2024-09-17T07:18:17.604 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:18:17.625 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:17.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:17.626 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:18:17.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:17 smithi100 ceph-mon[29092]: osdmap e342: 8 total, 7 up, 7 in 2024-09-17T07:18:17.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:17 smithi100 ceph-mon[39462]: osdmap e342: 8 total, 7 up, 7 in 2024-09-17T07:18:17.728 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:18:17.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:17.753 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:17.753 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:18:17.854 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:18:17.880 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:17.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:17.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:18:17.982 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:18:18.008 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:18.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:18.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:18:18.110 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:18:18.136 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:18.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:18.137 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:18:18.238 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:18:18.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:18.263 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:18.264 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:18:18.365 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:18:18.405 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:18.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:18.407 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:18:18.508 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:18:18.527 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:18 smithi100 ceph-mon[29092]: osdmap e343: 8 total, 7 up, 7 in 2024-09-17T07:18:18.527 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:18 smithi100 ceph-mon[29092]: pgmap v733: 158 pgs: 1 remapped+peering, 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 36 active+undersized, 2 active+remapped+backfill_toofull, 28 active+undersized+degraded, 84 active+clean; 6.1 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 800 op/s; 20154/298098 objects degraded (6.761%); 36669/298098 objects misplaced (12.301%); 27 MiB/s, 438 objects/s recovering 2024-09-17T07:18:18.527 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:18 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:18:18.527 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:18 smithi100 ceph-mon[39462]: osdmap e343: 8 total, 7 up, 7 in 2024-09-17T07:18:18.528 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:18 smithi100 ceph-mon[39462]: pgmap v733: 158 pgs: 1 remapped+peering, 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 36 active+undersized, 2 active+remapped+backfill_toofull, 28 active+undersized+degraded, 84 active+clean; 6.1 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 800 op/s; 20154/298098 objects degraded (6.761%); 36669/298098 objects misplaced (12.301%); 27 MiB/s, 438 objects/s recovering 2024-09-17T07:18:18.528 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:18 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:18:18.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:18 smithi135 ceph-mon[33238]: osdmap e343: 8 total, 7 up, 7 in 2024-09-17T07:18:18.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:18 smithi135 ceph-mon[33238]: pgmap v733: 158 pgs: 1 remapped+peering, 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 36 active+undersized, 2 active+remapped+backfill_toofull, 28 active+undersized+degraded, 84 active+clean; 6.1 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 800 op/s; 20154/298098 objects degraded (6.761%); 36669/298098 objects misplaced (12.301%); 27 MiB/s, 438 objects/s recovering 2024-09-17T07:18:18.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:18 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:18:18.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:18.545 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:18.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:18:18.646 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:18:18.689 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:18.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:18.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:18:18.791 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:18:18.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:18.824 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:18.824 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:18:18.924 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:18:18.954 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:18.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:18.955 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:18:19.056 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:18:19.093 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:19.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:19.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:18:19.195 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:18:19.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:19.226 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:19.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:18:19.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:19 smithi135 ceph-mon[33238]: osdmap e344: 8 total, 7 up, 7 in 2024-09-17T07:18:19.293 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:18:19 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:18:19.327 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:18:19.364 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:19.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:19.365 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:18:19.465 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:18:19.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:19.498 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:19.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:18:19.599 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:18:19.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:19 smithi100 ceph-mon[29092]: osdmap e344: 8 total, 7 up, 7 in 2024-09-17T07:18:19.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:19 smithi100 ceph-mon[39462]: osdmap e344: 8 total, 7 up, 7 in 2024-09-17T07:18:19.630 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:19.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:19.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:18:19.732 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:18:19.753 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:19.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:19.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:18:19.854 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:18:19.892 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:19.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:19.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:18:19.994 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:18:20.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:20.029 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:20.029 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:18:20.130 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:18:20.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:20.182 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:20.182 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:18:20.283 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:18:20.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:20.320 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:20.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:18:20.421 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:18:20.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:20.451 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:20.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:18:20.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:20 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:18:20.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:20 smithi135 ceph-mon[33238]: pgmap v735: 158 pgs: 3 remapped+peering, 2 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 36 active+undersized, 2 active+remapped+backfill_toofull, 28 active+undersized+degraded, 86 active+clean; 5.9 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 689 op/s; 18242/289686 objects degraded (6.297%); 27227/289686 objects misplaced (9.399%); 62 MiB/s, 987 objects/s recovering 2024-09-17T07:18:20.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:20 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 20154/298098 objects degraded (6.761%), 31 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-09-17T07:18:20.552 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:18:20.589 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:20.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:20.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:18:20.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:20 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:18:20.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:20 smithi100 ceph-mon[29092]: pgmap v735: 158 pgs: 3 remapped+peering, 2 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 36 active+undersized, 2 active+remapped+backfill_toofull, 28 active+undersized+degraded, 86 active+clean; 5.9 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 689 op/s; 18242/289686 objects degraded (6.297%); 27227/289686 objects misplaced (9.399%); 62 MiB/s, 987 objects/s recovering 2024-09-17T07:18:20.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:20 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 20154/298098 objects degraded (6.761%), 31 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-09-17T07:18:20.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:20 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:18:20.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:20 smithi100 ceph-mon[39462]: pgmap v735: 158 pgs: 3 remapped+peering, 2 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 36 active+undersized, 2 active+remapped+backfill_toofull, 28 active+undersized+degraded, 86 active+clean; 5.9 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 689 op/s; 18242/289686 objects degraded (6.297%); 27227/289686 objects misplaced (9.399%); 62 MiB/s, 987 objects/s recovering 2024-09-17T07:18:20.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:20 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 20154/298098 objects degraded (6.761%), 31 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-09-17T07:18:20.691 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:18:20.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:20.729 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:20.729 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:18:20.830 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:18:20.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:20.868 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:20.869 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:18:20.969 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:18:21.009 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:21.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:21.010 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:18:21.111 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:18:21.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:21.150 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:21.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:18:21.251 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:18:21.282 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:21.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:21.283 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:18:21.314 INFO:teuthology.orchestra.run.smithi100.stderr:noscrub is unset 2024-09-17T07:18:21.384 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:18:21.412 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:21.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:21.413 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:18:21.514 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:18:21.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:21.543 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:21.543 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:18:21.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:21 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:18:21.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:21 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:18:21.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:21 smithi100 ceph-mon[29092]: osdmap e345: 8 total, 7 up, 7 in 2024-09-17T07:18:21.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:21 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/3668673580' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:18:21.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:21 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:18:21.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:21 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:18:21.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:21 smithi100 ceph-mon[39462]: osdmap e345: 8 total, 7 up, 7 in 2024-09-17T07:18:21.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:21 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/3668673580' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:18:21.644 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:18:21.677 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:21.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:21.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:18:21.779 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:18:21.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:21 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:18:21.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:21 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:18:21.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:21 smithi135 ceph-mon[33238]: osdmap e345: 8 total, 7 up, 7 in 2024-09-17T07:18:21.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:21 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/3668673580' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:18:21.800 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:21.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:21.801 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:18:21.902 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:18:21.931 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:21.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:21.933 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:18:22.033 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:18:22.060 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:22.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:22.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:18:22.162 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:18:22.196 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:22.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:22.197 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:18:22.298 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:18:22.326 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:22.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:22.327 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:18:22.428 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:18:22.466 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:22.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:22.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:18:22.568 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:18:22.605 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:22.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:22.606 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:18:22.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:22 smithi100 ceph-mon[39462]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:18:22.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:22 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/3668673580' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T07:18:22.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:22 smithi100 ceph-mon[39462]: osdmap e346: 8 total, 7 up, 7 in 2024-09-17T07:18:22.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:22 smithi100 ceph-mon[39462]: pgmap v738: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 remapped+peering, 2 active+clean+remapped, 36 active+undersized, 2 active+remapped+backfill_toofull, 28 active+undersized+degraded, 86 active+clean; 5.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 845 op/s; 17989/287796 objects degraded (6.251%); 26767/287796 objects misplaced (9.301%); 62 MiB/s, 988 objects/s recovering 2024-09-17T07:18:22.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:22 smithi100 ceph-mon[29092]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:18:22.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:22 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/3668673580' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T07:18:22.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:22 smithi100 ceph-mon[29092]: osdmap e346: 8 total, 7 up, 7 in 2024-09-17T07:18:22.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:22 smithi100 ceph-mon[29092]: pgmap v738: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 remapped+peering, 2 active+clean+remapped, 36 active+undersized, 2 active+remapped+backfill_toofull, 28 active+undersized+degraded, 86 active+clean; 5.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 845 op/s; 17989/287796 objects degraded (6.251%); 26767/287796 objects misplaced (9.301%); 62 MiB/s, 988 objects/s recovering 2024-09-17T07:18:22.707 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:18:22.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:22.748 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:22.748 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:18:22.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:22 smithi135 ceph-mon[33238]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:18:22.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:22 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/3668673580' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T07:18:22.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:22 smithi135 ceph-mon[33238]: osdmap e346: 8 total, 7 up, 7 in 2024-09-17T07:18:22.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:22 smithi135 ceph-mon[33238]: pgmap v738: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 remapped+peering, 2 active+clean+remapped, 36 active+undersized, 2 active+remapped+backfill_toofull, 28 active+undersized+degraded, 86 active+clean; 5.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 845 op/s; 17989/287796 objects degraded (6.251%); 26767/287796 objects misplaced (9.301%); 62 MiB/s, 988 objects/s recovering 2024-09-17T07:18:22.849 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:18:22.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:22.889 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:22.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:18:22.990 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:18:23.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:23.017 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:23.017 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:18:23.118 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:18:23.150 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:23.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:23.152 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:18:23.252 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:18:23.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:23.290 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:23.290 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:18:23.391 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:18:23.419 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:23.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:23.420 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:18:23.521 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:18:23.546 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:23.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:23.548 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:18:23.648 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:18:23.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:23.688 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:23.688 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:18:23.789 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:18:23.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:23 smithi135 ceph-mon[33238]: pgmap v739: 158 pgs: 1 active+clean+scrubbing, 1 active+undersized+degraded+remapped+backfill_toofull, 5 active+clean+remapped, 36 active+undersized, 2 active+remapped+backfill_toofull, 28 active+undersized+degraded, 85 active+clean; 5.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 963 op/s; 17781/280764 objects degraded (6.333%); 34570/280764 objects misplaced (12.313%); 42 MiB/s, 669 objects/s recovering 2024-09-17T07:18:23.824 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:23.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:23.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:18:23.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:23 smithi100 ceph-mon[29092]: pgmap v739: 158 pgs: 1 active+clean+scrubbing, 1 active+undersized+degraded+remapped+backfill_toofull, 5 active+clean+remapped, 36 active+undersized, 2 active+remapped+backfill_toofull, 28 active+undersized+degraded, 85 active+clean; 5.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 963 op/s; 17781/280764 objects degraded (6.333%); 34570/280764 objects misplaced (12.313%); 42 MiB/s, 669 objects/s recovering 2024-09-17T07:18:23.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:23 smithi100 ceph-mon[39462]: pgmap v739: 158 pgs: 1 active+clean+scrubbing, 1 active+undersized+degraded+remapped+backfill_toofull, 5 active+clean+remapped, 36 active+undersized, 2 active+remapped+backfill_toofull, 28 active+undersized+degraded, 85 active+clean; 5.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 963 op/s; 17781/280764 objects degraded (6.333%); 34570/280764 objects misplaced (12.313%); 42 MiB/s, 669 objects/s recovering 2024-09-17T07:18:23.925 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:18:23.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:23.962 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:23.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:18:24.063 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:18:24.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:24.099 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:24.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:18:24.200 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:18:24.236 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:24.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:24.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:18:24.338 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:18:24.365 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:24.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:24.366 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:18:24.467 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:18:24.506 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:24.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:24.507 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:18:24.608 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:18:24.645 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:24.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:24.647 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:18:24.747 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:18:24.783 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:24.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:24.784 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:18:24.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:24 smithi135 ceph-mon[33238]: 4.10 scrub starts 2024-09-17T07:18:24.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:24 smithi135 ceph-mon[33238]: 4.10 scrub ok 2024-09-17T07:18:24.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:24 smithi135 ceph-mon[33238]: 6.11 scrub starts 2024-09-17T07:18:24.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:24 smithi135 ceph-mon[33238]: 6.11 scrub ok 2024-09-17T07:18:24.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:24 smithi135 ceph-mon[33238]: 10.e scrub starts 2024-09-17T07:18:24.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:24 smithi100 ceph-mon[29092]: 4.10 scrub starts 2024-09-17T07:18:24.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:24 smithi100 ceph-mon[29092]: 4.10 scrub ok 2024-09-17T07:18:24.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:24 smithi100 ceph-mon[29092]: 6.11 scrub starts 2024-09-17T07:18:24.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:24 smithi100 ceph-mon[29092]: 6.11 scrub ok 2024-09-17T07:18:24.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:24 smithi100 ceph-mon[29092]: 10.e scrub starts 2024-09-17T07:18:24.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:24 smithi100 ceph-mon[39462]: 4.10 scrub starts 2024-09-17T07:18:24.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:24 smithi100 ceph-mon[39462]: 4.10 scrub ok 2024-09-17T07:18:24.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:24 smithi100 ceph-mon[39462]: 6.11 scrub starts 2024-09-17T07:18:24.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:24 smithi100 ceph-mon[39462]: 6.11 scrub ok 2024-09-17T07:18:24.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:24 smithi100 ceph-mon[39462]: 10.e scrub starts 2024-09-17T07:18:24.885 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:18:24.913 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:24.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:24.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:18:24.949 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:18:25.015 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:18:25.052 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:25.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:25.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:18:25.153 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:18:25.187 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:25.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:25.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:18:25.290 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:18:25.318 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:25.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:25.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:18:25.420 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:18:25.462 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:25.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:25.464 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:18:25.564 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:18:25.600 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:25.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:25.601 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:18:25.681 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:18:25.681 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:18:25.682 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:25 smithi100 ceph-mon[29092]: 5.9 scrub starts 2024-09-17T07:18:25.682 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:25 smithi100 ceph-mon[29092]: 5.9 scrub ok 2024-09-17T07:18:25.682 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:25 smithi100 ceph-mon[29092]: 4.16 scrub starts 2024-09-17T07:18:25.682 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:25 smithi100 ceph-mon[29092]: 4.16 scrub ok 2024-09-17T07:18:25.682 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:25 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 17781/280764 objects degraded (6.333%), 29 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-09-17T07:18:25.682 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:25 smithi100 ceph-mon[29092]: pgmap v740: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+scrubbing, 5 active+clean+remapped, 36 active+undersized, 2 active+remapped+backfill_toofull, 28 active+undersized+degraded, 85 active+clean; 5.4 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.25k op/s; 16722/267945 objects degraded (6.241%); 33516/267945 objects misplaced (12.509%); 6.2 MiB/s, 99 objects/s recovering 2024-09-17T07:18:25.682 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:25 smithi100 ceph-mon[39462]: 5.9 scrub starts 2024-09-17T07:18:25.682 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:25 smithi100 ceph-mon[39462]: 5.9 scrub ok 2024-09-17T07:18:25.683 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:25 smithi100 ceph-mon[39462]: 4.16 scrub starts 2024-09-17T07:18:25.683 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:25 smithi100 ceph-mon[39462]: 4.16 scrub ok 2024-09-17T07:18:25.683 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:25 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 17781/280764 objects degraded (6.333%), 29 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-09-17T07:18:25.683 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:25 smithi100 ceph-mon[39462]: pgmap v740: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+scrubbing, 5 active+clean+remapped, 36 active+undersized, 2 active+remapped+backfill_toofull, 28 active+undersized+degraded, 85 active+clean; 5.4 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.25k op/s; 16722/267945 objects degraded (6.241%); 33516/267945 objects misplaced (12.509%); 6.2 MiB/s, 99 objects/s recovering 2024-09-17T07:18:25.683 INFO:teuthology.orchestra.run.smithi100.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-17T07:18:25.701 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:18:25.708 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:18:25.708 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:18:25.733 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:18:25.733 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:18:25.739 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:25.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:25.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:18:25.762 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:18:25.762 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:18:25.792 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:18:25.792 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:18:25.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:25 smithi135 ceph-mon[33238]: 5.9 scrub starts 2024-09-17T07:18:25.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:25 smithi135 ceph-mon[33238]: 5.9 scrub ok 2024-09-17T07:18:25.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:25 smithi135 ceph-mon[33238]: 4.16 scrub starts 2024-09-17T07:18:25.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:25 smithi135 ceph-mon[33238]: 4.16 scrub ok 2024-09-17T07:18:25.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:25 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 17781/280764 objects degraded (6.333%), 29 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-09-17T07:18:25.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:25 smithi135 ceph-mon[33238]: pgmap v740: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+scrubbing, 5 active+clean+remapped, 36 active+undersized, 2 active+remapped+backfill_toofull, 28 active+undersized+degraded, 85 active+clean; 5.4 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.25k op/s; 16722/267945 objects degraded (6.241%); 33516/267945 objects misplaced (12.509%); 6.2 MiB/s, 99 objects/s recovering 2024-09-17T07:18:25.823 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:18:25.823 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:18:25.841 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:18:25.863 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:18:25.864 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:18:25.877 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:25.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:25.879 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:18:25.944 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:18:25 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:18:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:18:25.980 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:18:26.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:26.012 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:26.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:18:26.113 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:18:26.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:26.144 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:26.144 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:18:26.245 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:18:26.282 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:26.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:26.283 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:18:26.384 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:18:26.433 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:26.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:26.434 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:18:26.534 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:26 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/3922910924' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:18:26.534 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:26 smithi100 ceph-mon[29092]: 10.e scrub ok 2024-09-17T07:18:26.535 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:26 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/3922910924' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:18:26.535 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:26 smithi100 ceph-mon[39462]: 10.e scrub ok 2024-09-17T07:18:26.535 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:18:26.575 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:26.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:26.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:18:26.682 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:18:26.684 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T07:18:26.687 INFO:teuthology.orchestra.run.smithi100.stderr:dumped all 2024-09-17T07:18:26.730 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:26.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:26.732 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:18:26.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:26 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/3922910924' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:18:26.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:26 smithi135 ceph-mon[33238]: 10.e scrub ok 2024-09-17T07:18:26.832 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:18:26.868 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:26.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:26.869 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:18:26.970 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:18:26.998 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:26.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:26.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:18:27.100 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:18:27.130 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:27.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:27.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:18:27.232 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:18:27.262 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:27.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:27.263 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:18:27.364 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:18:27.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:27.403 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:27.403 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:18:27.452 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:18:27.453 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:18:27.504 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:18:27.532 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:27.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:27.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:18:27.634 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:18:27.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:27.667 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:27.667 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:18:27.763 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:27 smithi100 ceph-mon[29092]: 2.1c scrub starts 2024-09-17T07:18:27.763 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:27 smithi100 ceph-mon[29092]: from='client.15804 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:18:27.763 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:27 smithi100 ceph-mon[29092]: pgmap v741: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+scrubbing, 5 active+clean+remapped, 36 active+undersized, 2 active+remapped+backfill_toofull, 28 active+undersized+degraded, 85 active+clean; 5.4 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 906 op/s; 16722/267945 objects degraded (6.241%); 33516/267945 objects misplaced (12.509%) 2024-09-17T07:18:27.763 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:27 smithi100 ceph-mon[39462]: 2.1c scrub starts 2024-09-17T07:18:27.763 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:27 smithi100 ceph-mon[39462]: from='client.15804 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:18:27.763 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:27 smithi100 ceph-mon[39462]: pgmap v741: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+scrubbing, 5 active+clean+remapped, 36 active+undersized, 2 active+remapped+backfill_toofull, 28 active+undersized+degraded, 85 active+clean; 5.4 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 906 op/s; 16722/267945 objects degraded (6.241%); 33516/267945 objects misplaced (12.509%) 2024-09-17T07:18:27.768 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:18:27.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:27 smithi135 ceph-mon[33238]: 2.1c scrub starts 2024-09-17T07:18:27.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:27 smithi135 ceph-mon[33238]: from='client.15804 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:18:27.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:27 smithi135 ceph-mon[33238]: pgmap v741: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+scrubbing, 5 active+clean+remapped, 36 active+undersized, 2 active+remapped+backfill_toofull, 28 active+undersized+degraded, 85 active+clean; 5.4 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 906 op/s; 16722/267945 objects degraded (6.241%); 33516/267945 objects misplaced (12.509%) 2024-09-17T07:18:27.797 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:27.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:27.798 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:18:27.899 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:18:27.929 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:27.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:27.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:18:28.031 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:18:28.064 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:28.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:28.066 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:18:28.127 INFO:teuthology.orchestra.run.smithi100.stdout:{"pg_ready":true,"pg_map":{"version":740,"stamp":"2024-09-17T07:18:25.331508+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5841033518,"num_objects":89315,"num_object_clones":0,"num_object_copies":267945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16722,"num_objects_misplaced":33516,"num_objects_unfound":0,"num_objects_dirty":89315,"num_whiteouts":0,"num_read":1470,"num_read_kb":1316,"num_write":158468,"num_write_kb":7896540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23699,"num_bytes_recovered":1545343284,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":143900,"ondisk_log_size":143900,"up":403,"acting":409,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":479,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":21049636,"kb_used_data":16194980,"kb_used_omap":10,"kb_used_meta":4854389,"kb_avail":635166428,"statfs":{"total":671965249536,"available":650410422272,"internally_reserved":0,"allocated":16583659520,"data_stored":16579293086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11152,"internal_metadata":4970894448},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-576847872,"num_objects":-8802,"num_object_clones":0,"num_object_copies":-26406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1841,"num_objects_misplaced":5428,"num_objects_unfound":0,"num_objects_dirty":-8802,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":8802,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":704,"num_bytes_recovered":46137344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.056063"},"pg_stats":[{"pgid":"10.17","version":"346'8865","reported_seq":15601,"reported_epoch":346,"state":"active+clean+remapped","last_fresh":"2024-09-17T07:18:23.084406+0000","last_change":"2024-09-17T07:18:18.296127+0000","last_active":"2024-09-17T07:18:23.084406+0000","last_peered":"2024-09-17T07:18:23.084406+0000","last_clean":"2024-09-17T07:18:23.084406+0000","last_became_active":"2024-09-17T07:18:18.283245+0000","last_became_peered":"2024-09-17T07:18:18.283245+0000","last_unstale":"2024-09-17T07:18:23.084406+0000","last_undegraded":"2024-09-17T07:18:23.084406+0000","last_fullsized":"2024-09-17T07:18:23.084406+0000","mapping_epoch":343,"log_start":"312'3000","ondisk_log_start":"312'3000","created":332,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":5865,"log_dups_size":0,"ondisk_log_size":5865,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185860096,"num_objects":2836,"num_object_clones":0,"num_object_copies":8508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2836,"num_objects_unfound":0,"num_objects_dirty":2836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4943,"num_write_kb":248928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3327,"num_bytes_recovered":218038272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6,2],"avail_no_missing":["4","2","6"],"object_location_counts":[{"shards":"2,4,6","objects":2836}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":194,"reported_epoch":345,"state":"active+undersized","last_fresh":"2024-09-17T07:18:20.313877+0000","last_change":"2024-09-17T07:16:17.379311+0000","last_active":"2024-09-17T07:18:20.313877+0000","last_peered":"2024-09-17T07:18:20.313877+0000","last_clean":"2024-09-17T07:16:15.590051+0000","last_became_active":"2024-09-17T07:16:17.379311+0000","last_became_peered":"2024-09-17T07:16:17.379311+0000","last_unstale":"2024-09-17T07:18:20.313877+0000","last_undegraded":"2024-09-17T07:18:20.313877+0000","last_fullsized":"2024-09-17T07:16:17.372051+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:11.978185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:33.644728+0000","last_clean_scrub_stamp":"2024-09-17T07:13:11.978185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0074436570000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":360,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-17T07:18:20.312344+0000","last_change":"2024-09-17T07:16:30.401351+0000","last_active":"2024-09-17T07:18:20.312344+0000","last_peered":"2024-09-17T07:18:20.312344+0000","last_clean":"2024-09-17T07:18:20.312344+0000","last_became_active":"2024-09-17T07:02:12.006754+0000","last_became_peered":"2024-09-17T07:02:12.006754+0000","last_unstale":"2024-09-17T07:18:20.312344+0000","last_undegraded":"2024-09-17T07:18:20.312344+0000","last_fullsized":"2024-09-17T07:18:20.312344+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:30.401191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:16:30.401191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0038520239999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"242'27","reported_seq":270,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-17T07:18:20.313979+0000","last_change":"2024-09-17T07:17:21.548879+0000","last_active":"2024-09-17T07:18:20.313979+0000","last_peered":"2024-09-17T07:18:20.313979+0000","last_clean":"2024-09-17T07:18:20.313979+0000","last_became_active":"2024-09-17T07:09:24.402042+0000","last_became_peered":"2024-09-17T07:09:24.402042+0000","last_unstale":"2024-09-17T07:18:20.313979+0000","last_undegraded":"2024-09-17T07:18:20.313979+0000","last_fullsized":"2024-09-17T07:18:20.313979+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'27","last_scrub_stamp":"2024-09-17T07:17:21.548761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:17:21.548761+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:22.146018+0000","scrub_duration":0.0067518939999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":351,"reported_epoch":345,"state":"active+undersized","last_fresh":"2024-09-17T07:18:20.307850+0000","last_change":"2024-09-17T07:16:17.380766+0000","last_active":"2024-09-17T07:18:20.307850+0000","last_peered":"2024-09-17T07:18:20.307850+0000","last_clean":"2024-09-17T07:16:15.357790+0000","last_became_active":"2024-09-17T07:16:17.380766+0000","last_became_peered":"2024-09-17T07:16:17.380766+0000","last_unstale":"2024-09-17T07:18:20.307850+0000","last_undegraded":"2024-09-17T07:18:20.307850+0000","last_fullsized":"2024-09-17T07:16:17.364849+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:45.261117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:15:45.261117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001477482,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.16","version":"346'8698","reported_seq":15497,"reported_epoch":346,"state":"active+clean+remapped","last_fresh":"2024-09-17T07:18:23.647544+0000","last_change":"2024-09-17T07:18:18.299552+0000","last_active":"2024-09-17T07:18:23.647544+0000","last_peered":"2024-09-17T07:18:23.647544+0000","last_clean":"2024-09-17T07:18:23.647544+0000","last_became_active":"2024-09-17T07:18:18.284748+0000","last_became_peered":"2024-09-17T07:18:18.284748+0000","last_unstale":"2024-09-17T07:18:23.647544+0000","last_undegraded":"2024-09-17T07:18:23.647544+0000","last_fullsized":"2024-09-17T07:18:23.647544+0000","mapping_epoch":343,"log_start":"312'2800","ondisk_log_start":"312'2800","created":332,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":5,"last_scrub":"324'6643","last_scrub_stamp":"2024-09-17T07:17:30.776520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:17:30.776520+0000","objects_scrubbed":6552,"log_size":5898,"log_dups_size":0,"ondisk_log_size":5898,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2.27785153,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185139200,"num_objects":2825,"num_object_clones":0,"num_object_copies":8475,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2825,"num_objects_unfound":0,"num_objects_dirty":2825,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4829,"num_write_kb":244928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3287,"num_bytes_recovered":215416832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5,0],"avail_no_missing":["6","0","5"],"object_location_counts":[{"shards":"0,5,6","objects":2825}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":211,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-17T07:18:20.313392+0000","last_change":"2024-09-17T07:17:35.507852+0000","last_active":"2024-09-17T07:18:20.313392+0000","last_peered":"2024-09-17T07:18:20.313392+0000","last_clean":"2024-09-17T07:18:20.313392+0000","last_became_active":"2024-09-17T07:09:24.402568+0000","last_became_peered":"2024-09-17T07:09:24.402568+0000","last_unstale":"2024-09-17T07:18:20.313392+0000","last_undegraded":"2024-09-17T07:18:20.313392+0000","last_fullsized":"2024-09-17T07:18:20.313392+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:17:35.507742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:17:35.507742+0000","last_clean_scrub_stamp":"2024-09-17T07:17:35.507742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:19:01.894837+0000","scrub_duration":0.00224683,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"242'15","reported_seq":389,"reported_epoch":345,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:18:20.308003+0000","last_change":"2024-09-17T07:16:17.370326+0000","last_active":"2024-09-17T07:18:20.308003+0000","last_peered":"2024-09-17T07:18:20.308003+0000","last_clean":"2024-09-17T07:16:15.357813+0000","last_became_active":"2024-09-17T07:16:17.370326+0000","last_became_peered":"2024-09-17T07:16:17.370326+0000","last_unstale":"2024-09-17T07:18:20.308003+0000","last_undegraded":"2024-09-17T07:16:17.364616+0000","last_fullsized":"2024-09-17T07:16:17.364511+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'15","last_scrub_stamp":"2024-09-17T07:15:39.306043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:15:39.306043+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002471029,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":4}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":51,"reported_epoch":345,"state":"active+undersized","last_fresh":"2024-09-17T07:18:20.316162+0000","last_change":"2024-09-17T07:16:17.379569+0000","last_active":"2024-09-17T07:18:20.316162+0000","last_peered":"2024-09-17T07:18:20.316162+0000","last_clean":"2024-09-17T07:02:14.999435+0000","last_became_active":"2024-09-17T07:16:17.379569+0000","last_became_peered":"2024-09-17T07:16:17.379569+0000","last_unstale":"2024-09-17T07:18:20.316162+0000","last_undegraded":"2024-09-17T07:18:20.316162+0000","last_fullsized":"2024-09-17T07:16:17.369924+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:53.881813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:15:53.881813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":350,"reported_epoch":345,"state":"active+undersized","last_fresh":"2024-09-17T07:18:20.310781+0000","last_change":"2024-09-17T07:16:17.374605+0000","last_active":"2024-09-17T07:18:20.310781+0000","last_peered":"2024-09-17T07:18:20.310781+0000","last_clean":"2024-09-17T07:16:15.357982+0000","last_became_active":"2024-09-17T07:16:17.374605+0000","last_became_peered":"2024-09-17T07:16:17.374605+0000","last_unstale":"2024-09-17T07:18:20.310781+0000","last_undegraded":"2024-09-17T07:18:20.310781+0000","last_fullsized":"2024-09-17T07:16:17.366300+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:41.707969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:14:41.707969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.014931757,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.15","version":"346'8840","reported_seq":8917,"reported_epoch":346,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-17T07:18:24.514799+0000","last_change":"2024-09-17T07:17:55.363314+0000","last_active":"2024-09-17T07:18:24.514799+0000","last_peered":"2024-09-17T07:18:24.514799+0000","last_clean":"2024-09-17T07:17:53.298813+0000","last_became_active":"2024-09-17T07:17:55.351989+0000","last_became_peered":"2024-09-17T07:17:55.351989+0000","last_unstale":"2024-09-17T07:18:24.514799+0000","last_undegraded":"2024-09-17T07:18:24.514799+0000","last_fullsized":"2024-09-17T07:18:24.514799+0000","mapping_epoch":335,"log_start":"312'3100","ondisk_log_start":"312'3100","created":332,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":5740,"log_dups_size":0,"ondisk_log_size":5740,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":181501952,"num_objects":2770,"num_object_clones":0,"num_object_copies":8310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5540,"num_objects_unfound":0,"num_objects_dirty":2770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4941,"num_write_kb":246720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[0,2,7],"avail_no_missing":["0","2","7"],"object_location_counts":[{"shards":"0,2,7","objects":2770}],"blocked_by":[],"up_primary":5,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":347,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-17T07:18:20.318172+0000","last_change":"2024-09-17T07:15:46.312474+0000","last_active":"2024-09-17T07:18:20.318172+0000","last_peered":"2024-09-17T07:18:20.318172+0000","last_clean":"2024-09-17T07:18:20.318172+0000","last_became_active":"2024-09-17T07:02:12.011129+0000","last_became_peered":"2024-09-17T07:02:12.011129+0000","last_unstale":"2024-09-17T07:18:20.318172+0000","last_undegraded":"2024-09-17T07:18:20.318172+0000","last_fullsized":"2024-09-17T07:18:20.318172+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:46.312371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:46.312371+0000","last_clean_scrub_stamp":"2024-09-17T07:15:46.312371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015176510000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"242'7","reported_seq":382,"reported_epoch":345,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:18:20.308060+0000","last_change":"2024-09-17T07:16:17.373486+0000","last_active":"2024-09-17T07:18:20.308060+0000","last_peered":"2024-09-17T07:18:20.308060+0000","last_clean":"2024-09-17T07:16:15.358036+0000","last_became_active":"2024-09-17T07:16:17.373486+0000","last_became_peered":"2024-09-17T07:16:17.373486+0000","last_unstale":"2024-09-17T07:18:20.308060+0000","last_undegraded":"2024-09-17T07:16:17.366228+0000","last_fullsized":"2024-09-17T07:16:17.365631+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'7","last_scrub_stamp":"2024-09-17T07:14:42.653705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:14:42.653705+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0050149230000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[{"shards":"2,4","objects":2}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":349,"reported_epoch":345,"state":"active+undersized","last_fresh":"2024-09-17T07:18:20.313890+0000","last_change":"2024-09-17T07:16:17.382323+0000","last_active":"2024-09-17T07:18:20.313890+0000","last_peered":"2024-09-17T07:18:20.313890+0000","last_clean":"2024-09-17T07:16:15.589989+0000","last_became_active":"2024-09-17T07:16:17.382323+0000","last_became_peered":"2024-09-17T07:16:17.382323+0000","last_unstale":"2024-09-17T07:18:20.313890+0000","last_undegraded":"2024-09-17T07:18:20.313890+0000","last_fullsized":"2024-09-17T07:16:17.371260+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:37.997296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:14:37.997296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045486420000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":368,"reported_epoch":345,"state":"active+undersized","last_fresh":"2024-09-17T07:18:20.313432+0000","last_change":"2024-09-17T07:16:17.374238+0000","last_active":"2024-09-17T07:18:20.313432+0000","last_peered":"2024-09-17T07:18:20.313432+0000","last_clean":"2024-09-17T07:16:16.170346+0000","last_became_active":"2024-09-17T07:16:17.374238+0000","last_became_peered":"2024-09-17T07:16:17.374238+0000","last_unstale":"2024-09-17T07:18:20.313432+0000","last_undegraded":"2024-09-17T07:18:20.313432+0000","last_fullsized":"2024-09-17T07:16:17.367446+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:52.168181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:14:52.168181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0056501399999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.14","version":"346'8746","reported_seq":15576,"reported_epoch":346,"state":"active+clean+remapped","last_fresh":"2024-09-17T07:18:23.105608+0000","last_change":"2024-09-17T07:18:18.296134+0000","last_active":"2024-09-17T07:18:23.105608+0000","last_peered":"2024-09-17T07:18:23.105608+0000","last_clean":"2024-09-17T07:18:23.105608+0000","last_became_active":"2024-09-17T07:18:18.281656+0000","last_became_peered":"2024-09-17T07:18:18.281656+0000","last_unstale":"2024-09-17T07:18:23.105608+0000","last_undegraded":"2024-09-17T07:18:23.105608+0000","last_fullsized":"2024-09-17T07:18:23.105608+0000","mapping_epoch":343,"log_start":"312'3000","ondisk_log_start":"312'3000","created":332,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":5746,"log_dups_size":0,"ondisk_log_size":5746,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184156160,"num_objects":2810,"num_object_clones":0,"num_object_copies":8430,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2810,"num_objects_unfound":0,"num_objects_dirty":2810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4868,"num_write_kb":245696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3370,"num_bytes_recovered":220856320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2,0],"avail_no_missing":["4","0","2"],"object_location_counts":[{"shards":"0,2,4","objects":2810}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":347,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-17T07:18:20.317666+0000","last_change":"2024-09-17T07:14:44.076126+0000","last_active":"2024-09-17T07:18:20.317666+0000","last_peered":"2024-09-17T07:18:20.317666+0000","last_clean":"2024-09-17T07:18:20.317666+0000","last_became_active":"2024-09-17T07:02:12.012234+0000","last_became_peered":"2024-09-17T07:02:12.012234+0000","last_unstale":"2024-09-17T07:18:20.317666+0000","last_undegraded":"2024-09-17T07:18:20.317666+0000","last_fullsized":"2024-09-17T07:18:20.317666+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:44.076012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:45.029247+0000","last_clean_scrub_stamp":"2024-09-17T07:14:44.076012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.015917521,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"312'19","reported_seq":406,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-17T07:18:20.310609+0000","last_change":"2024-09-17T07:16:44.293061+0000","last_active":"2024-09-17T07:18:20.310609+0000","last_peered":"2024-09-17T07:18:20.310609+0000","last_clean":"2024-09-17T07:18:20.310609+0000","last_became_active":"2024-09-17T07:09:24.397954+0000","last_became_peered":"2024-09-17T07:09:24.397954+0000","last_unstale":"2024-09-17T07:18:20.310609+0000","last_undegraded":"2024-09-17T07:18:20.310609+0000","last_fullsized":"2024-09-17T07:18:20.310609+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"312'19","last_scrub_stamp":"2024-09-17T07:16:44.292911+0000","last_deep_scrub":"312'19","last_deep_scrub_stamp":"2024-09-17T07:16:44.292911+0000","last_clean_scrub_stamp":"2024-09-17T07:16:44.292911+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.013103178,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":365,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-17T07:18:20.312411+0000","last_change":"2024-09-17T07:16:45.164526+0000","last_active":"2024-09-17T07:18:20.312411+0000","last_peered":"2024-09-17T07:18:20.312411+0000","last_clean":"2024-09-17T07:18:20.312411+0000","last_became_active":"2024-09-17T07:02:16.023774+0000","last_became_peered":"2024-09-17T07:02:16.023774+0000","last_unstale":"2024-09-17T07:18:20.312411+0000","last_undegraded":"2024-09-17T07:18:20.312411+0000","last_fullsized":"2024-09-17T07:18:20.312411+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:45.164367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:28.096266+0000","last_clean_scrub_stamp":"2024-09-17T07:16:45.164367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026790630000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":349,"reported_epoch":345,"state":"active+undersized","last_fresh":"2024-09-17T07:18:20.312873+0000","last_change":"2024-09-17T07:16:17.378534+0000","last_active":"2024-09-17T07:18:20.312873+0000","last_peered":"2024-09-17T07:18:20.312873+0000","last_clean":"2024-09-17T07:16:15.494529+0000","last_became_active":"2024-09-17T07:16:17.378534+0000","last_became_peered":"2024-09-17T07:16:17.378534+0000","last_unstale":"2024-09-17T07:18:20.312873+0000","last_undegraded":"2024-09-17T07:18:20.312873+0000","last_fullsized":"2024-09-17T07:16:17.365363+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:45.483622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:15:45.483622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001342359,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3","version":"346'11788","reported_seq":12189,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-17T07:18:23.828422+0000","last_change":"2024-09-17T07:17:52.413004+0000","last_active":"2024-09-17T07:18:23.828422+0000","last_peered":"2024-09-17T07:18:23.828422+0000","last_clean":"2024-09-17T07:18:23.828422+0000","last_became_active":"2024-09-17T07:17:52.409230+0000","last_became_peered":"2024-09-17T07:17:52.409230+0000","last_unstale":"2024-09-17T07:18:23.828422+0000","last_undegraded":"2024-09-17T07:18:23.828422+0000","last_fullsized":"2024-09-17T07:18:23.828422+0000","mapping_epoch":297,"log_start":"320'5801","ondisk_log_start":"320'5801","created":297,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":4348,"log_size":5987,"log_dups_size":0,"ondisk_log_size":5987,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":178028544,"num_objects":2716,"num_object_clones":0,"num_object_copies":8148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4852,"num_write_kb":242208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"242'39","reported_seq":422,"reported_epoch":345,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:18:20.312366+0000","last_change":"2024-09-17T07:16:17.374805+0000","last_active":"2024-09-17T07:18:20.312366+0000","last_peered":"2024-09-17T07:18:20.312366+0000","last_clean":"2024-09-17T07:16:15.358504+0000","last_became_active":"2024-09-17T07:16:17.374805+0000","last_became_peered":"2024-09-17T07:16:17.374805+0000","last_unstale":"2024-09-17T07:18:20.312366+0000","last_undegraded":"2024-09-17T07:16:17.367444+0000","last_fullsized":"2024-09-17T07:16:17.367331+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'39","last_scrub_stamp":"2024-09-17T07:15:46.226554+0000","last_deep_scrub":"95'25","last_deep_scrub_stamp":"2024-09-17T07:04:09.450039+0000","last_clean_scrub_stamp":"2024-09-17T07:15:46.226554+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0027061799999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[{"shards":"2,4","objects":8}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":361,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-17T07:18:20.315152+0000","last_change":"2024-09-17T07:16:33.658452+0000","last_active":"2024-09-17T07:18:20.315152+0000","last_peered":"2024-09-17T07:18:20.315152+0000","last_clean":"2024-09-17T07:18:20.315152+0000","last_became_active":"2024-09-17T07:02:12.011258+0000","last_became_peered":"2024-09-17T07:02:12.011258+0000","last_unstale":"2024-09-17T07:18:20.315152+0000","last_undegraded":"2024-09-17T07:18:20.315152+0000","last_fullsized":"2024-09-17T07:18:20.315152+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:33.658293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:16:33.658293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.044680275999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":51,"reported_epoch":345,"state":"active+undersized","last_fresh":"2024-09-17T07:18:20.308345+0000","last_change":"2024-09-17T07:16:17.372955+0000","last_active":"2024-09-17T07:18:20.308345+0000","last_peered":"2024-09-17T07:18:20.308345+0000","last_clean":"2024-09-17T07:02:14.999435+0000","last_became_active":"2024-09-17T07:16:17.372955+0000","last_became_peered":"2024-09-17T07:16:17.372955+0000","last_unstale":"2024-09-17T07:18:20.308345+0000","last_undegraded":"2024-09-17T07:18:20.308345+0000","last_fullsized":"2024-09-17T07:16:17.365245+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:46.502641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:13:46.502641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":202,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-17T07:18:20.307955+0000","last_change":"2024-09-17T07:16:34.161259+0000","last_active":"2024-09-17T07:18:20.307955+0000","last_peered":"2024-09-17T07:18:20.307955+0000","last_clean":"2024-09-17T07:18:20.307955+0000","last_became_active":"2024-09-17T07:09:24.396334+0000","last_became_peered":"2024-09-17T07:09:24.396334+0000","last_unstale":"2024-09-17T07:18:20.307955+0000","last_undegraded":"2024-09-17T07:18:20.307955+0000","last_fullsized":"2024-09-17T07:18:20.307955+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:34.161180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:42.984088+0000","last_clean_scrub_stamp":"2024-09-17T07:16:34.161180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001181355,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2","version":"346'8719","reported_seq":8777,"reported_epoch":346,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:18:23.140596+0000","last_change":"2024-09-17T07:17:52.402716+0000","last_active":"2024-09-17T07:18:23.140596+0000","last_peered":"2024-09-17T07:18:23.140596+0000","last_clean":"2024-09-17T07:16:04.025356+0000","last_became_active":"2024-09-17T07:17:52.402716+0000","last_became_peered":"2024-09-17T07:17:52.402716+0000","last_unstale":"2024-09-17T07:18:23.140596+0000","last_undegraded":"2024-09-17T07:17:52.372244+0000","last_fullsized":"2024-09-17T07:17:52.361979+0000","mapping_epoch":303,"log_start":"312'3100","ondisk_log_start":"312'3100","created":297,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":5619,"log_dups_size":0,"ondisk_log_size":5619,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184254464,"num_objects":2811,"num_object_clones":0,"num_object_copies":8433,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2811,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2811,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4850,"num_write_kb":245184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[{"shards":"0,4","objects":2811}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"242'25","reported_seq":421,"reported_epoch":345,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:18:20.310736+0000","last_change":"2024-09-17T07:16:17.381025+0000","last_active":"2024-09-17T07:18:20.310736+0000","last_peered":"2024-09-17T07:18:20.310736+0000","last_clean":"2024-09-17T07:16:15.358419+0000","last_became_active":"2024-09-17T07:16:17.381025+0000","last_became_peered":"2024-09-17T07:16:17.381025+0000","last_unstale":"2024-09-17T07:18:20.310736+0000","last_undegraded":"2024-09-17T07:16:17.367122+0000","last_fullsized":"2024-09-17T07:16:17.365635+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'25","last_scrub_stamp":"2024-09-17T07:15:02.532371+0000","last_deep_scrub":"176'15","last_deep_scrub_stamp":"2024-09-17T07:08:19.633791+0000","last_clean_scrub_stamp":"2024-09-17T07:15:02.532371+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012712606,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":223,"reported_epoch":345,"state":"active+undersized","last_fresh":"2024-09-17T07:18:20.314179+0000","last_change":"2024-09-17T07:16:17.379662+0000","last_active":"2024-09-17T07:18:20.314179+0000","last_peered":"2024-09-17T07:18:20.314179+0000","last_clean":"2024-09-17T07:09:22.372452+0000","last_became_active":"2024-09-17T07:16:17.379662+0000","last_became_peered":"2024-09-17T07:16:17.379662+0000","last_unstale":"2024-09-17T07:18:20.314179+0000","last_undegraded":"2024-09-17T07:18:20.314179+0000","last_fullsized":"2024-09-17T07:16:17.370523+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:59.697560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:14:59.697560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0065215869999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.d","version":"98'9","reported_seq":498,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-17T07:18:20.709594+0000","last_change":"2024-09-17T07:14:46.126177+0000","last_active":"2024-09-17T07:18:20.709594+0000","last_peered":"2024-09-17T07:18:20.709594+0000","last_clean":"2024-09-17T07:18:20.709594+0000","last_became_active":"2024-09-17T07:04:15.110797+0000","last_became_peered":"2024-09-17T07:04:15.110797+0000","last_unstale":"2024-09-17T07:18:20.709594+0000","last_undegraded":"2024-09-17T07:18:20.709594+0000","last_fullsized":"2024-09-17T07:18:20.709594+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-09-17T07:14:46.126045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:14:46.126045+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.014185467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":361,"reported_epoch":345,"state":"active+undersized","last_fresh":"2024-09-17T07:18:20.308692+0000","last_change":"2024-09-17T07:16:17.373630+0000","last_active":"2024-09-17T07:18:20.308692+0000","last_peered":"2024-09-17T07:18:20.308692+0000","last_clean":"2024-09-17T07:16:15.358526+0000","last_became_active":"2024-09-17T07:16:17.373630+0000","last_became_peered":"2024-09-17T07:16:17.373630+0000","last_unstale":"2024-09-17T07:18:20.308692+0000","last_undegraded":"2024-09-17T07:18:20.308692+0000","last_fullsized":"2024-09-17T07:16:17.369047+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:50.554591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:14:50.554591+0000","last_clean_scrub_stamp":"2024-09-17T07:14:50.554591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0067125969999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1","version":"346'9055","reported_seq":9260,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-17T07:18:23.119992+0000","last_change":"2024-09-17T07:17:52.414213+0000","last_active":"2024-09-17T07:18:23.119992+0000","last_peered":"2024-09-17T07:18:23.119992+0000","last_clean":"2024-09-17T07:18:23.119992+0000","last_became_active":"2024-09-17T07:17:52.402933+0000","last_became_peered":"2024-09-17T07:17:52.402933+0000","last_unstale":"2024-09-17T07:18:23.119992+0000","last_undegraded":"2024-09-17T07:18:23.119992+0000","last_fullsized":"2024-09-17T07:18:23.119992+0000","mapping_epoch":297,"log_start":"312'3400","ondisk_log_start":"312'3400","created":297,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'6219","last_scrub_stamp":"2024-09-17T07:17:22.401249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:17:22.401249+0000","objects_scrubbed":6144,"log_size":5655,"log_dups_size":0,"ondisk_log_size":5655,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2.0885702689999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":183369728,"num_objects":2798,"num_object_clones":0,"num_object_copies":8394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5078,"num_write_kb":252032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"242'27","reported_seq":345,"reported_epoch":345,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:18:20.312759+0000","last_change":"2024-09-17T07:16:17.382238+0000","last_active":"2024-09-17T07:18:20.312759+0000","last_peered":"2024-09-17T07:18:20.312759+0000","last_clean":"2024-09-17T07:12:20.775697+0000","last_became_active":"2024-09-17T07:16:17.382238+0000","last_became_peered":"2024-09-17T07:16:17.382238+0000","last_unstale":"2024-09-17T07:18:20.312759+0000","last_undegraded":"2024-09-17T07:16:17.365468+0000","last_fullsized":"2024-09-17T07:16:17.365334+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'27","last_scrub_stamp":"2024-09-17T07:14:45.831734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:14:45.831734+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021853620000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[{"shards":"4,7","objects":7}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":224,"reported_epoch":345,"state":"active+undersized","last_fresh":"2024-09-17T07:18:20.315068+0000","last_change":"2024-09-17T07:16:17.378599+0000","last_active":"2024-09-17T07:18:20.315068+0000","last_peered":"2024-09-17T07:18:20.315068+0000","last_clean":"2024-09-17T07:09:22.371958+0000","last_became_active":"2024-09-17T07:16:17.378599+0000","last_became_peered":"2024-09-17T07:16:17.378599+0000","last_unstale":"2024-09-17T07:18:20.315068+0000","last_undegraded":"2024-09-17T07:18:20.315068+0000","last_fullsized":"2024-09-17T07:16:17.373236+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:44.897611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:15:44.897611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034116369999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.e","version":"75'8","reported_seq":559,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-17T07:18:20.714179+0000","last_change":"2024-09-17T07:16:49.331611+0000","last_active":"2024-09-17T07:18:20.714179+0000","last_peered":"2024-09-17T07:18:20.714179+0000","last_clean":"2024-09-17T07:18:20.714179+0000","last_became_active":"2024-09-17T07:02:16.021762+0000","last_became_peered":"2024-09-17T07:02:16.021762+0000","last_unstale":"2024-09-17T07:18:20.714179+0000","last_undegraded":"2024-09-17T07:18:20.714179+0000","last_fullsized":"2024-09-17T07:18:20.714179+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:16:49.331407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:16:49.331407+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033687029999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":359,"reported_epoch":345,"state":"active+undersized","last_fresh":"2024-09-17T07:18:20.312952+0000","last_change":"2024-09-17T07:16:17.374897+0000","last_active":"2024-09-17T07:18:20.312952+0000","last_peered":"2024-09-17T07:18:20.312952+0000","last_clean":"2024-09-17T07:16:16.170588+0000","last_became_active":"2024-09-17T07:16:17.374897+0000","last_became_peered":"2024-09-17T07:16:17.374897+0000","last_unstale":"2024-09-17T07:18:20.312952+0000","last_undegraded":"2024-09-17T07:18:20.312952+0000","last_fullsized":"2024-09-17T07:16:17.368895+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:56.146098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:14:56.146098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0054479960000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.0","version":"346'8830","reported_seq":8901,"reported_epoch":346,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:18:23.606543+0000","last_change":"2024-09-17T07:17:52.385485+0000","last_active":"2024-09-17T07:18:23.606543+0000","last_peered":"2024-09-17T07:18:23.606543+0000","last_clean":"2024-09-17T07:16:14.245624+0000","last_became_active":"2024-09-17T07:17:52.385485+0000","last_became_peered":"2024-09-17T07:17:52.385485+0000","last_unstale":"2024-09-17T07:18:23.606543+0000","last_undegraded":"2024-09-17T07:17:52.365146+0000","last_fullsized":"2024-09-17T07:17:52.365038+0000","mapping_epoch":303,"log_start":"312'3000","ondisk_log_start":"312'3000","created":297,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":5830,"log_dups_size":0,"ondisk_log_size":5830,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":182517760,"num_objects":2785,"num_object_clones":0,"num_object_copies":8355,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2785,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2785,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4933,"num_write_kb":246976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[{"shards":"4,6","objects":2785}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"242'19","reported_seq":396,"reported_epoch":345,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:18:20.308737+0000","last_change":"2024-09-17T07:16:17.382184+0000","last_active":"2024-09-17T07:18:20.308737+0000","last_peered":"2024-09-17T07:18:20.308737+0000","last_clean":"2024-09-17T07:16:15.358681+0000","last_became_active":"2024-09-17T07:16:17.382184+0000","last_became_peered":"2024-09-17T07:16:17.382184+0000","last_unstale":"2024-09-17T07:18:20.308737+0000","last_undegraded":"2024-09-17T07:16:17.369179+0000","last_fullsized":"2024-09-17T07:16:17.369087+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'19","last_scrub_stamp":"2024-09-17T07:14:55.489874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:14:55.489874+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045924440000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":355,"reported_epoch":345,"state":"active+undersized","last_fresh":"2024-09-17T07:18:20.314530+0000","last_change":"2024-09-17T07:16:17.375239+0000","last_active":"2024-09-17T07:18:20.314530+0000","last_peered":"2024-09-17T07:18:20.314530+0000","last_clean":"2024-09-17T07:16:15.590006+0000","last_became_active":"2024-09-17T07:16:17.375239+0000","last_became_peered":"2024-09-17T07:16:17.375239+0000","last_unstale":"2024-09-17T07:18:20.314530+0000","last_undegraded":"2024-09-17T07:18:20.314530+0000","last_fullsized":"2024-09-17T07:16:17.372272+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:21.858127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:13:21.858127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033279709999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":377,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-17T07:18:20.313650+0000","last_change":"2024-09-17T07:17:18.435029+0000","last_active":"2024-09-17T07:18:20.313650+0000","last_peered":"2024-09-17T07:18:20.313650+0000","last_clean":"2024-09-17T07:18:20.313650+0000","last_became_active":"2024-09-17T07:02:16.020224+0000","last_became_peered":"2024-09-17T07:02:16.020224+0000","last_unstale":"2024-09-17T07:18:20.313650+0000","last_undegraded":"2024-09-17T07:18:20.313650+0000","last_fullsized":"2024-09-17T07:18:20.313650+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:17:18.434919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:17:18.434919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:23.948987+0000","scrub_duration":0.0044010059999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":219,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-17T07:18:20.317601+0000","last_change":"2024-09-17T07:16:37.587187+0000","last_active":"2024-09-17T07:18:20.317601+0000","last_peered":"2024-09-17T07:18:20.317601+0000","last_clean":"2024-09-17T07:18:20.317601+0000","last_became_active":"2024-09-17T07:09:24.408109+0000","last_became_peered":"2024-09-17T07:09:24.408109+0000","last_unstale":"2024-09-17T07:18:20.317601+0000","last_undegraded":"2024-09-17T07:18:20.317601+0000","last_fullsized":"2024-09-17T07:18:20.317601+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:37.587106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:13:22.187127+0000","last_clean_scrub_stamp":"2024-09-17T07:16:37.587106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001128609,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.f","version":"346'12926","reported_seq":13290,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-17T07:18:23.657164+0000","last_change":"2024-09-17T07:17:57.325680+0000","last_active":"2024-09-17T07:18:23.657164+0000","last_peered":"2024-09-17T07:18:23.657164+0000","last_clean":"2024-09-17T07:18:23.657164+0000","last_became_active":"2024-09-17T07:16:05.052037+0000","last_became_peered":"2024-09-17T07:16:05.052037+0000","last_unstale":"2024-09-17T07:18:23.657164+0000","last_undegraded":"2024-09-17T07:18:23.657164+0000","last_fullsized":"2024-09-17T07:18:23.657164+0000","mapping_epoch":297,"log_start":"330'8453","ondisk_log_start":"330'8453","created":297,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":3774,"log_size":4473,"log_dups_size":0,"ondisk_log_size":4473,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":367788032,"num_objects":5612,"num_object_clones":0,"num_object_copies":16836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5612,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9992,"num_write_kb":499328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"242'28","reported_seq":401,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-17T07:18:20.307671+0000","last_change":"2024-09-17T07:17:31.149145+0000","last_active":"2024-09-17T07:18:20.307671+0000","last_peered":"2024-09-17T07:18:20.307671+0000","last_clean":"2024-09-17T07:18:20.307671+0000","last_became_active":"2024-09-17T07:02:14.007810+0000","last_became_peered":"2024-09-17T07:02:14.007810+0000","last_unstale":"2024-09-17T07:18:20.307671+0000","last_undegraded":"2024-09-17T07:18:20.307671+0000","last_fullsized":"2024-09-17T07:18:20.307671+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'28","last_scrub_stamp":"2024-09-17T07:17:31.148909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:17:31.148909+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:44.025720+0000","scrub_duration":0.013898221000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":402,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-17T07:18:20.321162+0000","last_change":"2024-09-17T07:17:19.376369+0000","last_active":"2024-09-17T07:18:20.321162+0000","last_peered":"2024-09-17T07:18:20.321162+0000","last_clean":"2024-09-17T07:18:20.321162+0000","last_became_active":"2024-09-17T07:02:12.011766+0000","last_became_peered":"2024-09-17T07:02:12.011766+0000","last_unstale":"2024-09-17T07:18:20.321162+0000","last_undegraded":"2024-09-17T07:18:20.321162+0000","last_fullsized":"2024-09-17T07:18:20.321162+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:17:19.375818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:09:23.181028+0000","last_clean_scrub_stamp":"2024-09-17T07:17:19.375818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:43.381176+0000","scrub_duration":0.019104131999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"304'10","reported_seq":98,"reported_epoch":346,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:18:21.323557+0000","last_change":"2024-09-17T07:16:17.371678+0000","last_active":"2024-09-17T07:18:21.323557+0000","last_peered":"2024-09-17T07:18:21.323557+0000","last_clean":"2024-09-17T07:02:19.145975+0000","last_became_active":"2024-09-17T07:16:17.371678+0000","last_became_peered":"2024-09-17T07:16:17.371678+0000","last_unstale":"2024-09-17T07:18:21.323557+0000","last_undegraded":"2024-09-17T07:16:17.366242+0000","last_fullsized":"2024-09-17T07:16:17.365352+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'9","last_scrub_stamp":"2024-09-17T07:15:55.813882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:15:55.813882+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[{"shards":"0,4","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":353,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-17T07:18:20.314406+0000","last_change":"2024-09-17T07:15:52.378479+0000","last_active":"2024-09-17T07:18:20.314406+0000","last_peered":"2024-09-17T07:18:20.314406+0000","last_clean":"2024-09-17T07:18:20.314406+0000","last_became_active":"2024-09-17T07:02:18.364981+0000","last_became_peered":"2024-09-17T07:02:18.364981+0000","last_unstale":"2024-09-17T07:18:20.314406+0000","last_undegraded":"2024-09-17T07:18:20.314406+0000","last_fullsized":"2024-09-17T07:18:20.314406+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:52.378384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:10:33.727472+0000","last_clean_scrub_stamp":"2024-09-17T07:15:52.378384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001396736,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"346'12721","reported_seq":13098,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-17T07:18:23.096764+0000","last_change":"2024-09-17T07:17:57.447654+0000","last_active":"2024-09-17T07:18:23.096764+0000","last_peered":"2024-09-17T07:18:23.096764+0000","last_clean":"2024-09-17T07:18:23.096764+0000","last_became_active":"2024-09-17T07:16:05.054109+0000","last_became_peered":"2024-09-17T07:16:05.054109+0000","last_unstale":"2024-09-17T07:18:23.096764+0000","last_undegraded":"2024-09-17T07:18:23.096764+0000","last_fullsized":"2024-09-17T07:18:23.096764+0000","mapping_epoch":297,"log_start":"330'8491","ondisk_log_start":"330'8491","created":297,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":3924,"log_size":4230,"log_dups_size":0,"ondisk_log_size":4230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":365690880,"num_objects":5580,"num_object_clones":0,"num_object_copies":16740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9728,"num_write_kb":489856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"242'30","reported_seq":274,"reported_epoch":345,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:18:20.314228+0000","last_change":"2024-09-17T07:16:17.378001+0000","last_active":"2024-09-17T07:18:20.314228+0000","last_peered":"2024-09-17T07:18:20.314228+0000","last_clean":"2024-09-17T07:16:15.589841+0000","last_became_active":"2024-09-17T07:16:17.378001+0000","last_became_peered":"2024-09-17T07:16:17.378001+0000","last_unstale":"2024-09-17T07:18:20.314228+0000","last_undegraded":"2024-09-17T07:16:17.372487+0000","last_fullsized":"2024-09-17T07:16:17.372376+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'30","last_scrub_stamp":"2024-09-17T07:13:28.997563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:13:28.997563+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0061961159999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[{"shards":"0,7","objects":8}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":51,"reported_epoch":345,"state":"active+undersized","last_fresh":"2024-09-17T07:18:20.319424+0000","last_change":"2024-09-17T07:16:17.376547+0000","last_active":"2024-09-17T07:18:20.319424+0000","last_peered":"2024-09-17T07:18:20.319424+0000","last_clean":"2024-09-17T07:02:10.991363+0000","last_became_active":"2024-09-17T07:16:17.376547+0000","last_became_peered":"2024-09-17T07:16:17.376547+0000","last_unstale":"2024-09-17T07:18:20.319424+0000","last_undegraded":"2024-09-17T07:18:20.319424+0000","last_fullsized":"2024-09-17T07:16:17.364886+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:56.848689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:56.848689+0000","last_clean_scrub_stamp":"2024-09-17T07:15:56.848689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":366,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-17T07:18:20.307590+0000","last_change":"2024-09-17T07:16:38.184957+0000","last_active":"2024-09-17T07:18:20.307590+0000","last_peered":"2024-09-17T07:18:20.307590+0000","last_clean":"2024-09-17T07:18:20.307590+0000","last_became_active":"2024-09-17T07:09:24.393375+0000","last_became_peered":"2024-09-17T07:09:24.393375+0000","last_unstale":"2024-09-17T07:18:20.307590+0000","last_undegraded":"2024-09-17T07:18:20.307590+0000","last_fullsized":"2024-09-17T07:18:20.307590+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:38.184818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:16:38.184818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0076357700000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":364,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-17T07:18:20.310530+0000","last_change":"2024-09-17T07:16:36.162077+0000","last_active":"2024-09-17T07:18:20.310530+0000","last_peered":"2024-09-17T07:18:20.310530+0000","last_clean":"2024-09-17T07:18:20.310530+0000","last_became_active":"2024-09-17T07:09:24.398659+0000","last_became_peered":"2024-09-17T07:09:24.398659+0000","last_unstale":"2024-09-17T07:18:20.310530+0000","last_undegraded":"2024-09-17T07:18:20.310530+0000","last_fullsized":"2024-09-17T07:18:20.310530+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:36.161622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:16:36.161622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0062812850000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.d","version":"346'9918","reported_seq":9974,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-17T07:18:23.644182+0000","last_change":"2024-09-17T07:16:05.052409+0000","last_active":"2024-09-17T07:18:23.644182+0000","last_peered":"2024-09-17T07:18:23.644182+0000","last_clean":"2024-09-17T07:18:23.644182+0000","last_became_active":"2024-09-17T07:16:05.052153+0000","last_became_peered":"2024-09-17T07:16:05.052153+0000","last_unstale":"2024-09-17T07:18:23.644182+0000","last_undegraded":"2024-09-17T07:18:23.644182+0000","last_fullsized":"2024-09-17T07:18:23.644182+0000","mapping_epoch":297,"log_start":"316'5430","ondisk_log_start":"316'5430","created":297,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":4488,"log_dups_size":0,"ondisk_log_size":4488,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":365625368,"num_objects":5580,"num_object_clones":0,"num_object_copies":16740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5580,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9918,"num_write_kb":495873,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"242'29","reported_seq":410,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-17T07:18:20.313334+0000","last_change":"2024-09-17T07:17:40.599358+0000","last_active":"2024-09-17T07:18:20.313334+0000","last_peered":"2024-09-17T07:18:20.313334+0000","last_clean":"2024-09-17T07:18:20.313334+0000","last_became_active":"2024-09-17T07:02:14.027389+0000","last_became_peered":"2024-09-17T07:02:14.027389+0000","last_unstale":"2024-09-17T07:18:20.313334+0000","last_undegraded":"2024-09-17T07:18:20.313334+0000","last_fullsized":"2024-09-17T07:18:20.313334+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'29","last_scrub_stamp":"2024-09-17T07:17:40.599184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:17:40.599184+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:44.414394+0000","scrub_duration":0.073497912999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":360,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-17T07:18:20.318243+0000","last_change":"2024-09-17T07:16:47.631557+0000","last_active":"2024-09-17T07:18:20.318243+0000","last_peered":"2024-09-17T07:18:20.318243+0000","last_clean":"2024-09-17T07:18:20.318243+0000","last_became_active":"2024-09-17T07:02:12.011917+0000","last_became_peered":"2024-09-17T07:02:12.011917+0000","last_unstale":"2024-09-17T07:18:20.318243+0000","last_undegraded":"2024-09-17T07:18:20.318243+0000","last_fullsized":"2024-09-17T07:18:20.318243+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:47.631387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:47.631387+0000","last_clean_scrub_stamp":"2024-09-17T07:16:47.631387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.006082372,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":377,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-17T07:18:20.317677+0000","last_change":"2024-09-17T07:17:42.360268+0000","last_active":"2024-09-17T07:18:20.317677+0000","last_peered":"2024-09-17T07:18:20.317677+0000","last_clean":"2024-09-17T07:18:20.317677+0000","last_became_active":"2024-09-17T07:02:16.016199+0000","last_became_peered":"2024-09-17T07:02:16.016199+0000","last_unstale":"2024-09-17T07:18:20.317677+0000","last_undegraded":"2024-09-17T07:18:20.317677+0000","last_fullsized":"2024-09-17T07:18:20.317677+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:17:42.360104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:11:17.583712+0000","last_clean_scrub_stamp":"2024-09-17T07:17:42.360104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:54.611733+0000","scrub_duration":0.0062057919999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":52,"reported_epoch":346,"state":"active+undersized","last_fresh":"2024-09-17T07:18:21.316577+0000","last_change":"2024-09-17T07:16:17.368186+0000","last_active":"2024-09-17T07:18:21.316577+0000","last_peered":"2024-09-17T07:18:21.316577+0000","last_clean":"2024-09-17T07:02:17.002904+0000","last_became_active":"2024-09-17T07:16:17.368186+0000","last_became_peered":"2024-09-17T07:16:17.368186+0000","last_unstale":"2024-09-17T07:18:21.316577+0000","last_undegraded":"2024-09-17T07:18:21.316577+0000","last_fullsized":"2024-09-17T07:16:17.362908+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:50.514525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:13:50.514525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.c","version":"346'9838","reported_seq":9896,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-17T07:18:23.013093+0000","last_change":"2024-09-17T07:16:05.057855+0000","last_active":"2024-09-17T07:18:23.013093+0000","last_peered":"2024-09-17T07:18:23.013093+0000","last_clean":"2024-09-17T07:18:23.013093+0000","last_became_active":"2024-09-17T07:16:05.057588+0000","last_became_peered":"2024-09-17T07:16:05.057588+0000","last_unstale":"2024-09-17T07:18:23.013093+0000","last_undegraded":"2024-09-17T07:18:23.013093+0000","last_fullsized":"2024-09-17T07:18:23.013093+0000","mapping_epoch":297,"log_start":"318'5566","ondisk_log_start":"318'5566","created":297,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":4272,"log_dups_size":0,"ondisk_log_size":4272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371064832,"num_objects":5662,"num_object_clones":0,"num_object_copies":16986,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9838,"num_write_kb":496000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"242'25","reported_seq":313,"reported_epoch":346,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:18:21.330763+0000","last_change":"2024-09-17T07:16:17.376630+0000","last_active":"2024-09-17T07:18:21.330763+0000","last_peered":"2024-09-17T07:18:21.330763+0000","last_clean":"2024-09-17T07:12:20.656798+0000","last_became_active":"2024-09-17T07:16:17.376630+0000","last_became_peered":"2024-09-17T07:16:17.376630+0000","last_unstale":"2024-09-17T07:18:21.330763+0000","last_undegraded":"2024-09-17T07:16:17.368029+0000","last_fullsized":"2024-09-17T07:16:17.367816+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'25","last_scrub_stamp":"2024-09-17T07:14:50.794200+0000","last_deep_scrub":"110'15","last_deep_scrub_stamp":"2024-09-17T07:04:57.091400+0000","last_clean_scrub_stamp":"2024-09-17T07:14:50.794200+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0070558269999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[{"shards":"4,5","objects":5}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":52,"reported_epoch":346,"state":"active+undersized","last_fresh":"2024-09-17T07:18:21.315919+0000","last_change":"2024-09-17T07:16:17.378414+0000","last_active":"2024-09-17T07:18:21.315919+0000","last_peered":"2024-09-17T07:18:21.315919+0000","last_clean":"2024-09-17T07:02:10.991363+0000","last_became_active":"2024-09-17T07:16:17.378414+0000","last_became_peered":"2024-09-17T07:16:17.378414+0000","last_unstale":"2024-09-17T07:18:21.315919+0000","last_undegraded":"2024-09-17T07:18:21.315919+0000","last_fullsized":"2024-09-17T07:16:17.365679+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:49.764885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:25.147878+0000","last_clean_scrub_stamp":"2024-09-17T07:14:49.764885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"89'9","reported_seq":570,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-17T07:18:20.710896+0000","last_change":"2024-09-17T07:17:41.595535+0000","last_active":"2024-09-17T07:18:20.710896+0000","last_peered":"2024-09-17T07:18:20.710896+0000","last_clean":"2024-09-17T07:18:20.710896+0000","last_became_active":"2024-09-17T07:02:16.021360+0000","last_became_peered":"2024-09-17T07:02:16.021360+0000","last_unstale":"2024-09-17T07:18:20.710896+0000","last_undegraded":"2024-09-17T07:18:20.710896+0000","last_fullsized":"2024-09-17T07:18:20.710896+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'9","last_scrub_stamp":"2024-09-17T07:17:41.595032+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-09-17T07:15:53.512777+0000","last_clean_scrub_stamp":"2024-09-17T07:17:41.595032+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:19:05.063868+0000","scrub_duration":0.031478313000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"268'2","reported_seq":370,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-17T07:18:21.313042+0000","last_change":"2024-09-17T07:16:46.448318+0000","last_active":"2024-09-17T07:18:21.313042+0000","last_peered":"2024-09-17T07:18:21.313042+0000","last_clean":"2024-09-17T07:18:21.313042+0000","last_became_active":"2024-09-17T07:09:24.399565+0000","last_became_peered":"2024-09-17T07:09:24.399565+0000","last_unstale":"2024-09-17T07:18:21.313042+0000","last_undegraded":"2024-09-17T07:18:21.313042+0000","last_fullsized":"2024-09-17T07:18:21.313042+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'2","last_scrub_stamp":"2024-09-17T07:16:46.448196+0000","last_deep_scrub":"268'2","last_deep_scrub_stamp":"2024-09-17T07:14:03.609131+0000","last_clean_scrub_stamp":"2024-09-17T07:16:46.448196+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00244297,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.a","version":"346'9761","reported_seq":9819,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-17T07:18:23.650135+0000","last_change":"2024-09-17T07:16:05.055275+0000","last_active":"2024-09-17T07:18:23.650135+0000","last_peered":"2024-09-17T07:18:23.650135+0000","last_clean":"2024-09-17T07:18:23.650135+0000","last_became_active":"2024-09-17T07:16:05.055008+0000","last_became_peered":"2024-09-17T07:16:05.055008+0000","last_unstale":"2024-09-17T07:18:23.650135+0000","last_undegraded":"2024-09-17T07:18:23.650135+0000","last_fullsized":"2024-09-17T07:18:23.650135+0000","mapping_epoch":297,"log_start":"316'5250","ondisk_log_start":"316'5250","created":297,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":4511,"log_dups_size":0,"ondisk_log_size":4511,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":368771072,"num_objects":5627,"num_object_clones":0,"num_object_copies":16881,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9761,"num_write_kb":492416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"304'42","reported_seq":523,"reported_epoch":346,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:18:21.317088+0000","last_change":"2024-09-17T07:16:17.368142+0000","last_active":"2024-09-17T07:18:21.317088+0000","last_peered":"2024-09-17T07:18:21.317088+0000","last_clean":"2024-09-17T07:12:20.666266+0000","last_became_active":"2024-09-17T07:16:17.368142+0000","last_became_peered":"2024-09-17T07:16:17.368142+0000","last_unstale":"2024-09-17T07:18:21.317088+0000","last_undegraded":"2024-09-17T07:16:17.363645+0000","last_fullsized":"2024-09-17T07:16:17.363541+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'41","last_scrub_stamp":"2024-09-17T07:14:56.700858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:14:56.700858+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026058610000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[{"shards":"2,6","objects":10}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":365,"reported_epoch":345,"state":"active+undersized","last_fresh":"2024-09-17T07:18:20.313894+0000","last_change":"2024-09-17T07:16:17.371561+0000","last_active":"2024-09-17T07:18:20.313894+0000","last_peered":"2024-09-17T07:18:20.313894+0000","last_clean":"2024-09-17T07:16:16.170577+0000","last_became_active":"2024-09-17T07:16:17.371561+0000","last_became_peered":"2024-09-17T07:16:17.371561+0000","last_unstale":"2024-09-17T07:18:20.313894+0000","last_undegraded":"2024-09-17T07:18:20.313894+0000","last_fullsized":"2024-09-17T07:16:17.366918+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:03.045709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:14:03.045709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0049245929999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":362,"reported_epoch":345,"state":"active+undersized","last_fresh":"2024-09-17T07:18:20.312929+0000","last_change":"2024-09-17T07:16:17.375376+0000","last_active":"2024-09-17T07:18:20.312929+0000","last_peered":"2024-09-17T07:18:20.312929+0000","last_clean":"2024-09-17T07:16:15.495109+0000","last_became_active":"2024-09-17T07:16:17.375376+0000","last_became_peered":"2024-09-17T07:16:17.375376+0000","last_unstale":"2024-09-17T07:18:20.312929+0000","last_undegraded":"2024-09-17T07:18:20.312929+0000","last_fullsized":"2024-09-17T07:16:17.364236+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:36.523676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:15:36.523676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017370549999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":211,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-17T07:18:20.308437+0000","last_change":"2024-09-17T07:17:18.312624+0000","last_active":"2024-09-17T07:18:20.308437+0000","last_peered":"2024-09-17T07:18:20.308437+0000","last_clean":"2024-09-17T07:18:20.308437+0000","last_became_active":"2024-09-17T07:09:24.398878+0000","last_became_peered":"2024-09-17T07:09:24.398878+0000","last_unstale":"2024-09-17T07:18:20.308437+0000","last_undegraded":"2024-09-17T07:18:20.308437+0000","last_fullsized":"2024-09-17T07:18:20.308437+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:17:18.312499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:13:53.406661+0000","last_clean_scrub_stamp":"2024-09-17T07:17:18.312499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:38.339565+0000","scrub_duration":0.0030147720000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.9","version":"346'8774","reported_seq":8980,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-17T07:18:23.840262+0000","last_change":"2024-09-17T07:17:52.391749+0000","last_active":"2024-09-17T07:18:23.840262+0000","last_peered":"2024-09-17T07:18:23.840262+0000","last_clean":"2024-09-17T07:18:23.840262+0000","last_became_active":"2024-09-17T07:17:52.386125+0000","last_became_peered":"2024-09-17T07:17:52.386125+0000","last_unstale":"2024-09-17T07:18:23.840262+0000","last_undegraded":"2024-09-17T07:18:23.840262+0000","last_fullsized":"2024-09-17T07:18:23.840262+0000","mapping_epoch":297,"log_start":"310'2700","ondisk_log_start":"310'2700","created":297,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":0,"last_scrub":"322'6203","last_scrub_stamp":"2024-09-17T07:17:25.670907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:17:25.670907+0000","objects_scrubbed":6134,"log_size":6074,"log_dups_size":0,"ondisk_log_size":6074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2.260616706,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":178094080,"num_objects":2717,"num_object_clones":0,"num_object_copies":8151,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4920,"num_write_kb":244416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"242'17","reported_seq":310,"reported_epoch":345,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:18:20.313902+0000","last_change":"2024-09-17T07:16:17.371977+0000","last_active":"2024-09-17T07:18:20.313902+0000","last_peered":"2024-09-17T07:18:20.313902+0000","last_clean":"2024-09-17T07:12:20.284331+0000","last_became_active":"2024-09-17T07:16:17.371977+0000","last_became_peered":"2024-09-17T07:16:17.371977+0000","last_unstale":"2024-09-17T07:18:20.313902+0000","last_undegraded":"2024-09-17T07:16:17.367143+0000","last_fullsized":"2024-09-17T07:16:17.367038+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'17","last_scrub_stamp":"2024-09-17T07:13:47.468399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:13:47.468399+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.005881725,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[{"shards":"0,5","objects":5}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"76'5","reported_seq":207,"reported_epoch":346,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:18:21.866725+0000","last_change":"2024-09-17T07:16:17.368338+0000","last_active":"2024-09-17T07:18:21.866725+0000","last_peered":"2024-09-17T07:18:21.866725+0000","last_clean":"2024-09-17T07:02:23.985427+0000","last_became_active":"2024-09-17T07:16:17.368338+0000","last_became_peered":"2024-09-17T07:16:17.368338+0000","last_unstale":"2024-09-17T07:18:21.866725+0000","last_undegraded":"2024-09-17T07:16:17.363185+0000","last_fullsized":"2024-09-17T07:16:17.363086+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'5","last_scrub_stamp":"2024-09-17T07:15:36.901601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:15:36.901601+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033739799999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":132,"num_read_kb":127,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[{"shards":"2,6","objects":1}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":207,"reported_epoch":345,"state":"active+undersized","last_fresh":"2024-09-17T07:18:20.314055+0000","last_change":"2024-09-17T07:16:17.367238+0000","last_active":"2024-09-17T07:18:20.314055+0000","last_peered":"2024-09-17T07:18:20.314055+0000","last_clean":"2024-09-17T07:16:15.494771+0000","last_became_active":"2024-09-17T07:16:17.367238+0000","last_became_peered":"2024-09-17T07:16:17.367238+0000","last_unstale":"2024-09-17T07:18:20.314055+0000","last_undegraded":"2024-09-17T07:18:20.314055+0000","last_fullsized":"2024-09-17T07:16:17.364334+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:56.534547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:23.833170+0000","last_clean_scrub_stamp":"2024-09-17T07:15:56.534547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0011018950000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":379,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-17T07:18:20.310174+0000","last_change":"2024-09-17T07:16:46.778669+0000","last_active":"2024-09-17T07:18:20.310174+0000","last_peered":"2024-09-17T07:18:20.310174+0000","last_clean":"2024-09-17T07:18:20.310174+0000","last_became_active":"2024-09-17T07:02:16.014492+0000","last_became_peered":"2024-09-17T07:02:16.014492+0000","last_unstale":"2024-09-17T07:18:20.310174+0000","last_undegraded":"2024-09-17T07:18:20.310174+0000","last_fullsized":"2024-09-17T07:18:20.310174+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:46.778591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:16:46.778591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013415079999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":362,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-17T07:18:20.315164+0000","last_change":"2024-09-17T07:15:02.062776+0000","last_active":"2024-09-17T07:18:20.315164+0000","last_peered":"2024-09-17T07:18:20.315164+0000","last_clean":"2024-09-17T07:18:20.315164+0000","last_became_active":"2024-09-17T07:09:24.402704+0000","last_became_peered":"2024-09-17T07:09:24.402704+0000","last_unstale":"2024-09-17T07:18:20.315164+0000","last_undegraded":"2024-09-17T07:18:20.315164+0000","last_fullsized":"2024-09-17T07:18:20.315164+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:02.062618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:02.062618+0000","last_clean_scrub_stamp":"2024-09-17T07:15:02.062618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0042833460000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"346'8850","reported_seq":8919,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-17T07:18:24.545865+0000","last_change":"2024-09-17T07:17:52.375144+0000","last_active":"2024-09-17T07:18:24.545865+0000","last_peered":"2024-09-17T07:18:24.545865+0000","last_clean":"2024-09-17T07:18:24.545865+0000","last_became_active":"2024-09-17T07:17:52.372680+0000","last_became_peered":"2024-09-17T07:17:52.372680+0000","last_unstale":"2024-09-17T07:18:24.545865+0000","last_undegraded":"2024-09-17T07:18:24.545865+0000","last_fullsized":"2024-09-17T07:18:24.545865+0000","mapping_epoch":297,"log_start":"312'3100","ondisk_log_start":"312'3100","created":297,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":5750,"log_dups_size":0,"ondisk_log_size":5750,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":181272576,"num_objects":2766,"num_object_clones":0,"num_object_copies":8298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4953,"num_write_kb":247008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"268'30","reported_seq":423,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-17T07:18:20.313323+0000","last_change":"2024-09-17T07:16:43.375355+0000","last_active":"2024-09-17T07:18:20.313323+0000","last_peered":"2024-09-17T07:18:20.313323+0000","last_clean":"2024-09-17T07:18:20.313323+0000","last_became_active":"2024-09-17T07:03:48.760405+0000","last_became_peered":"2024-09-17T07:03:48.760405+0000","last_unstale":"2024-09-17T07:18:20.313323+0000","last_undegraded":"2024-09-17T07:18:20.313323+0000","last_fullsized":"2024-09-17T07:18:20.313323+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'30","last_scrub_stamp":"2024-09-17T07:16:43.375250+0000","last_deep_scrub":"268'30","last_deep_scrub_stamp":"2024-09-17T07:14:00.599656+0000","last_clean_scrub_stamp":"2024-09-17T07:16:43.375250+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.006604922,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"70'1","reported_seq":379,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-17T07:18:20.314208+0000","last_change":"2024-09-17T07:17:26.441837+0000","last_active":"2024-09-17T07:18:20.314208+0000","last_peered":"2024-09-17T07:18:20.314208+0000","last_clean":"2024-09-17T07:18:20.314208+0000","last_became_active":"2024-09-17T07:09:24.402499+0000","last_became_peered":"2024-09-17T07:09:24.402499+0000","last_unstale":"2024-09-17T07:18:20.314208+0000","last_undegraded":"2024-09-17T07:18:20.314208+0000","last_fullsized":"2024-09-17T07:18:20.314208+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-17T07:17:26.441670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:17:26.441670+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:29.168291+0000","scrub_duration":0.0054314109999999997,"objects_trimmed":0,"snaptrim_duration":0.0032298629999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":355,"reported_epoch":346,"state":"active+undersized","last_fresh":"2024-09-17T07:18:21.322563+0000","last_change":"2024-09-17T07:16:17.375018+0000","last_active":"2024-09-17T07:18:21.322563+0000","last_peered":"2024-09-17T07:18:21.322563+0000","last_clean":"2024-09-17T07:16:15.590088+0000","last_became_active":"2024-09-17T07:16:17.375018+0000","last_became_peered":"2024-09-17T07:16:17.375018+0000","last_unstale":"2024-09-17T07:18:21.322563+0000","last_undegraded":"2024-09-17T07:18:21.322563+0000","last_fullsized":"2024-09-17T07:16:17.370688+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:55.321202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:55.321202+0000","last_clean_scrub_stamp":"2024-09-17T07:15:55.321202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014716950000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":361,"reported_epoch":345,"state":"active+undersized","last_fresh":"2024-09-17T07:18:20.313737+0000","last_change":"2024-09-17T07:16:17.375774+0000","last_active":"2024-09-17T07:18:20.313737+0000","last_peered":"2024-09-17T07:18:20.313737+0000","last_clean":"2024-09-17T07:16:16.169907+0000","last_became_active":"2024-09-17T07:16:17.375774+0000","last_became_peered":"2024-09-17T07:16:17.375774+0000","last_unstale":"2024-09-17T07:18:20.313737+0000","last_undegraded":"2024-09-17T07:18:20.313737+0000","last_fullsized":"2024-09-17T07:16:17.368453+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:45.447029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:15:45.447029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001990463,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":218,"reported_epoch":346,"state":"active+undersized","last_fresh":"2024-09-17T07:18:21.326374+0000","last_change":"2024-09-17T07:16:17.372859+0000","last_active":"2024-09-17T07:18:21.326374+0000","last_peered":"2024-09-17T07:18:21.326374+0000","last_clean":"2024-09-17T07:09:22.371903+0000","last_became_active":"2024-09-17T07:16:17.372859+0000","last_became_peered":"2024-09-17T07:16:17.372859+0000","last_unstale":"2024-09-17T07:18:21.326374+0000","last_undegraded":"2024-09-17T07:18:21.326374+0000","last_fullsized":"2024-09-17T07:16:17.367296+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:51.877367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:15:51.877367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.058290885000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.b","version":"346'9825","reported_seq":9889,"reported_epoch":346,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:18:23.640195+0000","last_change":"2024-09-17T07:16:18.636164+0000","last_active":"2024-09-17T07:18:23.640195+0000","last_peered":"2024-09-17T07:18:23.640195+0000","last_clean":"2024-09-17T07:16:14.246161+0000","last_became_active":"2024-09-17T07:16:17.365889+0000","last_became_peered":"2024-09-17T07:16:17.365889+0000","last_unstale":"2024-09-17T07:18:23.640195+0000","last_undegraded":"2024-09-17T07:16:18.628529+0000","last_fullsized":"2024-09-17T07:16:17.362973+0000","mapping_epoch":303,"log_start":"316'5357","ondisk_log_start":"316'5357","created":297,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":4468,"log_dups_size":0,"ondisk_log_size":4468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":364576768,"num_objects":5563,"num_object_clones":0,"num_object_copies":16689,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5563,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9825,"num_write_kb":492416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[{"shards":"0,6","objects":5563}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"242'26","reported_seq":409,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-17T07:18:20.318206+0000","last_change":"2024-09-17T07:16:33.667044+0000","last_active":"2024-09-17T07:18:20.318206+0000","last_peered":"2024-09-17T07:18:20.318206+0000","last_clean":"2024-09-17T07:18:20.318206+0000","last_became_active":"2024-09-17T07:02:14.028857+0000","last_became_peered":"2024-09-17T07:02:14.028857+0000","last_unstale":"2024-09-17T07:18:20.318206+0000","last_undegraded":"2024-09-17T07:18:20.318206+0000","last_fullsized":"2024-09-17T07:18:20.318206+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'26","last_scrub_stamp":"2024-09-17T07:16:33.666684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:16:33.666684+0000","objects_scrubbed":7,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.013353596000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"268'3","reported_seq":230,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-17T07:18:20.313751+0000","last_change":"2024-09-17T07:17:19.453842+0000","last_active":"2024-09-17T07:18:20.313751+0000","last_peered":"2024-09-17T07:18:20.313751+0000","last_clean":"2024-09-17T07:18:20.313751+0000","last_became_active":"2024-09-17T07:09:24.404532+0000","last_became_peered":"2024-09-17T07:09:24.404532+0000","last_unstale":"2024-09-17T07:18:20.313751+0000","last_undegraded":"2024-09-17T07:18:20.313751+0000","last_fullsized":"2024-09-17T07:18:20.313751+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'3","last_scrub_stamp":"2024-09-17T07:17:19.444444+0000","last_deep_scrub":"268'3","last_deep_scrub_stamp":"2024-09-17T07:13:54.949345+0000","last_clean_scrub_stamp":"2024-09-17T07:17:19.444444+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0048257380000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":381,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-17T07:18:20.312693+0000","last_change":"2024-09-17T07:17:34.639287+0000","last_active":"2024-09-17T07:18:20.312693+0000","last_peered":"2024-09-17T07:18:20.312693+0000","last_clean":"2024-09-17T07:18:20.312693+0000","last_became_active":"2024-09-17T07:02:12.004606+0000","last_became_peered":"2024-09-17T07:02:12.004606+0000","last_unstale":"2024-09-17T07:18:20.312693+0000","last_undegraded":"2024-09-17T07:18:20.312693+0000","last_fullsized":"2024-09-17T07:18:20.312693+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:17:34.638733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:17:34.638733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:56.860917+0000","scrub_duration":0.064159545999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"212'130","reported_seq":610,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-17T07:18:20.314781+0000","last_change":"2024-09-17T07:14:57.052759+0000","last_active":"2024-09-17T07:18:20.314781+0000","last_peered":"2024-09-17T07:18:20.314781+0000","last_clean":"2024-09-17T07:18:20.314781+0000","last_became_active":"2024-09-17T07:02:03.020675+0000","last_became_peered":"2024-09-17T07:02:03.020675+0000","last_unstale":"2024-09-17T07:18:20.314781+0000","last_undegraded":"2024-09-17T07:18:20.314781+0000","last_fullsized":"2024-09-17T07:18:20.314781+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:14:57.052505+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:14:57.052505+0000","last_clean_scrub_stamp":"2024-09-17T07:14:57.052505+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.022122466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":355,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-17T07:18:20.317272+0000","last_change":"2024-09-17T07:15:56.366761+0000","last_active":"2024-09-17T07:18:20.317272+0000","last_peered":"2024-09-17T07:18:20.317272+0000","last_clean":"2024-09-17T07:18:20.317272+0000","last_became_active":"2024-09-17T07:02:16.016031+0000","last_became_peered":"2024-09-17T07:02:16.016031+0000","last_unstale":"2024-09-17T07:18:20.317272+0000","last_undegraded":"2024-09-17T07:18:20.317272+0000","last_fullsized":"2024-09-17T07:18:20.317272+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:56.366669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:15:56.366669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015953269999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":357,"reported_epoch":345,"state":"active+undersized","last_fresh":"2024-09-17T07:18:20.314083+0000","last_change":"2024-09-17T07:16:17.377095+0000","last_active":"2024-09-17T07:18:20.314083+0000","last_peered":"2024-09-17T07:18:20.314083+0000","last_clean":"2024-09-17T07:16:16.170840+0000","last_became_active":"2024-09-17T07:16:17.377095+0000","last_became_peered":"2024-09-17T07:16:17.377095+0000","last_unstale":"2024-09-17T07:18:20.314083+0000","last_undegraded":"2024-09-17T07:18:20.314083+0000","last_fullsized":"2024-09-17T07:16:17.370787+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:48.021592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:58.670571+0000","last_clean_scrub_stamp":"2024-09-17T07:14:48.021592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0064844619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.4","version":"346'8855","reported_seq":8927,"reported_epoch":346,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:18:24.572401+0000","last_change":"2024-09-17T07:17:52.376990+0000","last_active":"2024-09-17T07:18:24.572401+0000","last_peered":"2024-09-17T07:18:24.572401+0000","last_clean":"2024-09-17T07:16:15.495005+0000","last_became_active":"2024-09-17T07:17:52.376990+0000","last_became_peered":"2024-09-17T07:17:52.376990+0000","last_unstale":"2024-09-17T07:18:24.572401+0000","last_undegraded":"2024-09-17T07:17:52.365057+0000","last_fullsized":"2024-09-17T07:17:52.364949+0000","mapping_epoch":303,"log_start":"312'3000","ondisk_log_start":"312'3000","created":297,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":5855,"log_dups_size":0,"ondisk_log_size":5855,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":177012736,"num_objects":2701,"num_object_clones":0,"num_object_copies":8103,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2701,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2701,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4977,"num_write_kb":245696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0,4","objects":2701}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"242'35","reported_seq":430,"reported_epoch":345,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:18:20.318117+0000","last_change":"2024-09-17T07:16:17.376246+0000","last_active":"2024-09-17T07:18:20.318117+0000","last_peered":"2024-09-17T07:18:20.318117+0000","last_clean":"2024-09-17T07:16:14.244770+0000","last_became_active":"2024-09-17T07:16:17.376246+0000","last_became_peered":"2024-09-17T07:16:17.376246+0000","last_unstale":"2024-09-17T07:18:20.318117+0000","last_undegraded":"2024-09-17T07:16:17.363365+0000","last_fullsized":"2024-09-17T07:16:17.363257+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'35","last_scrub_stamp":"2024-09-17T07:15:43.665829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:15:43.665829+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0048325160000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[{"shards":"6,7","objects":9}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":405,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-17T07:18:20.309575+0000","last_change":"2024-09-17T07:17:20.848821+0000","last_active":"2024-09-17T07:18:20.309575+0000","last_peered":"2024-09-17T07:18:20.309575+0000","last_clean":"2024-09-17T07:18:20.309575+0000","last_became_active":"2024-09-17T07:02:12.012037+0000","last_became_peered":"2024-09-17T07:02:12.012037+0000","last_unstale":"2024-09-17T07:18:20.309575+0000","last_undegraded":"2024-09-17T07:18:20.309575+0000","last_fullsized":"2024-09-17T07:18:20.309575+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:17:20.848743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:06:39.322535+0000","last_clean_scrub_stamp":"2024-09-17T07:17:20.848743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:38.587002+0000","scrub_duration":0.001349062,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":389,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-17T07:18:20.307030+0000","last_change":"2024-09-17T07:16:44.719372+0000","last_active":"2024-09-17T07:18:20.307030+0000","last_peered":"2024-09-17T07:18:20.307030+0000","last_clean":"2024-09-17T07:18:20.307030+0000","last_became_active":"2024-09-17T07:02:16.014631+0000","last_became_peered":"2024-09-17T07:02:16.014631+0000","last_unstale":"2024-09-17T07:18:20.307030+0000","last_undegraded":"2024-09-17T07:18:20.307030+0000","last_fullsized":"2024-09-17T07:18:20.307030+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:44.719264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:16:44.719264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0036441289999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":351,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-17T07:18:20.314701+0000","last_change":"2024-09-17T07:15:45.275553+0000","last_active":"2024-09-17T07:18:20.314701+0000","last_peered":"2024-09-17T07:18:20.314701+0000","last_clean":"2024-09-17T07:18:20.314701+0000","last_became_active":"2024-09-17T07:09:24.403904+0000","last_became_peered":"2024-09-17T07:09:24.403904+0000","last_unstale":"2024-09-17T07:18:20.314701+0000","last_undegraded":"2024-09-17T07:18:20.314701+0000","last_fullsized":"2024-09-17T07:18:20.314701+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:45.275473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:45.275473+0000","last_clean_scrub_stamp":"2024-09-17T07:15:45.275473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014472759999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"346'8936","reported_seq":9005,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-17T07:18:24.533707+0000","last_change":"2024-09-17T07:17:52.379453+0000","last_active":"2024-09-17T07:18:24.533707+0000","last_peered":"2024-09-17T07:18:24.533707+0000","last_clean":"2024-09-17T07:18:24.533707+0000","last_became_active":"2024-09-17T07:17:52.375445+0000","last_became_peered":"2024-09-17T07:17:52.375445+0000","last_unstale":"2024-09-17T07:18:24.533707+0000","last_undegraded":"2024-09-17T07:18:24.533707+0000","last_fullsized":"2024-09-17T07:18:24.533707+0000","mapping_epoch":297,"log_start":"312'3100","ondisk_log_start":"312'3100","created":297,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":5836,"log_dups_size":0,"ondisk_log_size":5836,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":175210496,"num_objects":2673,"num_object_clones":0,"num_object_copies":8019,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5036,"num_write_kb":246720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"268'19","reported_seq":382,"reported_epoch":345,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:18:20.313183+0000","last_change":"2024-09-17T07:16:17.375651+0000","last_active":"2024-09-17T07:18:20.313183+0000","last_peered":"2024-09-17T07:18:20.313183+0000","last_clean":"2024-09-17T07:16:15.494820+0000","last_became_active":"2024-09-17T07:16:17.375651+0000","last_became_peered":"2024-09-17T07:16:17.375651+0000","last_unstale":"2024-09-17T07:18:20.313183+0000","last_undegraded":"2024-09-17T07:16:17.364532+0000","last_fullsized":"2024-09-17T07:16:17.364433+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'19","last_scrub_stamp":"2024-09-17T07:14:06.634334+0000","last_deep_scrub":"268'19","last_deep_scrub_stamp":"2024-09-17T07:14:06.634334+0000","last_clean_scrub_stamp":"2024-09-17T07:14:06.634334+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0088943289999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0,4","objects":4}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":223,"reported_epoch":345,"state":"active+undersized","last_fresh":"2024-09-17T07:18:20.308504+0000","last_change":"2024-09-17T07:16:17.374110+0000","last_active":"2024-09-17T07:18:20.308504+0000","last_peered":"2024-09-17T07:18:20.308504+0000","last_clean":"2024-09-17T07:09:22.372513+0000","last_became_active":"2024-09-17T07:16:17.374110+0000","last_became_peered":"2024-09-17T07:16:17.374110+0000","last_unstale":"2024-09-17T07:18:20.308504+0000","last_undegraded":"2024-09-17T07:18:20.308504+0000","last_fullsized":"2024-09-17T07:16:17.367137+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:58.886294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:13:53.593078+0000","last_clean_scrub_stamp":"2024-09-17T07:15:58.886294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.005853245,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":331,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-17T07:18:20.310238+0000","last_change":"2024-09-17T07:16:35.152814+0000","last_active":"2024-09-17T07:18:20.310238+0000","last_peered":"2024-09-17T07:18:20.310238+0000","last_clean":"2024-09-17T07:18:20.310238+0000","last_became_active":"2024-09-17T07:09:24.396642+0000","last_became_peered":"2024-09-17T07:09:24.396642+0000","last_unstale":"2024-09-17T07:18:20.310238+0000","last_undegraded":"2024-09-17T07:18:20.310238+0000","last_fullsized":"2024-09-17T07:18:20.310238+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:35.152744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:16:35.152744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0022724160000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":354,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-17T07:18:20.314151+0000","last_change":"2024-09-17T07:17:27.482509+0000","last_active":"2024-09-17T07:18:20.314151+0000","last_peered":"2024-09-17T07:18:20.314151+0000","last_clean":"2024-09-17T07:18:20.314151+0000","last_became_active":"2024-09-17T07:02:18.026994+0000","last_became_peered":"2024-09-17T07:02:18.026994+0000","last_unstale":"2024-09-17T07:18:20.314151+0000","last_undegraded":"2024-09-17T07:18:20.314151+0000","last_fullsized":"2024-09-17T07:18:20.314151+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:17:27.482401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:58.099849+0000","last_clean_scrub_stamp":"2024-09-17T07:17:27.482401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:42.245140+0000","scrub_duration":0.0020589919999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.6","version":"346'8786","reported_seq":8997,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-17T07:18:23.822499+0000","last_change":"2024-09-17T07:17:52.404390+0000","last_active":"2024-09-17T07:18:23.822499+0000","last_peered":"2024-09-17T07:18:23.822499+0000","last_clean":"2024-09-17T07:18:23.822499+0000","last_became_active":"2024-09-17T07:17:52.398777+0000","last_became_peered":"2024-09-17T07:17:52.398777+0000","last_unstale":"2024-09-17T07:18:23.822499+0000","last_undegraded":"2024-09-17T07:18:23.822499+0000","last_fullsized":"2024-09-17T07:18:23.822499+0000","mapping_epoch":297,"log_start":"312'2800","ondisk_log_start":"312'2800","created":297,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":0,"last_scrub":"324'6643","last_scrub_stamp":"2024-09-17T07:17:30.776520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:17:30.776520+0000","objects_scrubbed":6552,"log_size":5986,"log_dups_size":0,"ondisk_log_size":5986,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2.27785153,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":179372032,"num_objects":2737,"num_object_clones":0,"num_object_copies":8211,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2737,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4917,"num_write_kb":244928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"242'34","reported_seq":418,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-17T07:18:20.311360+0000","last_change":"2024-09-17T07:16:44.126984+0000","last_active":"2024-09-17T07:18:20.311360+0000","last_peered":"2024-09-17T07:18:20.311360+0000","last_clean":"2024-09-17T07:18:20.311360+0000","last_became_active":"2024-09-17T07:02:14.028392+0000","last_became_peered":"2024-09-17T07:02:14.028392+0000","last_unstale":"2024-09-17T07:18:20.311360+0000","last_undegraded":"2024-09-17T07:18:20.311360+0000","last_fullsized":"2024-09-17T07:18:20.311360+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'34","last_scrub_stamp":"2024-09-17T07:16:44.126830+0000","last_deep_scrub":"242'34","last_deep_scrub_stamp":"2024-09-17T07:14:50.103539+0000","last_clean_scrub_stamp":"2024-09-17T07:16:44.126830+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.008313473,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'2","reported_seq":230,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-17T07:18:20.316542+0000","last_change":"2024-09-17T07:15:48.378000+0000","last_active":"2024-09-17T07:18:20.316542+0000","last_peered":"2024-09-17T07:18:20.316542+0000","last_clean":"2024-09-17T07:18:20.316542+0000","last_became_active":"2024-09-17T07:09:24.402909+0000","last_became_peered":"2024-09-17T07:09:24.402909+0000","last_unstale":"2024-09-17T07:18:20.316542+0000","last_undegraded":"2024-09-17T07:18:20.316542+0000","last_fullsized":"2024-09-17T07:18:20.316542+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-09-17T07:15:48.377886+0000","last_deep_scrub":"89'2","last_deep_scrub_stamp":"2024-09-17T07:12:20.301263+0000","last_clean_scrub_stamp":"2024-09-17T07:15:48.377886+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001635079,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":420,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":420,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"225'9","reported_seq":550,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-17T07:18:20.713029+0000","last_change":"2024-09-17T07:14:42.018236+0000","last_active":"2024-09-17T07:18:20.713029+0000","last_peered":"2024-09-17T07:18:20.713029+0000","last_clean":"2024-09-17T07:18:20.713029+0000","last_became_active":"2024-09-17T07:02:16.015418+0000","last_became_peered":"2024-09-17T07:02:16.015418+0000","last_unstale":"2024-09-17T07:18:20.713029+0000","last_undegraded":"2024-09-17T07:18:20.713029+0000","last_fullsized":"2024-09-17T07:18:20.713029+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"225'9","last_scrub_stamp":"2024-09-17T07:14:42.017656+0000","last_deep_scrub":"225'9","last_deep_scrub_stamp":"2024-09-17T07:11:25.292565+0000","last_clean_scrub_stamp":"2024-09-17T07:14:42.017656+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0089186300000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":364,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-17T07:18:20.313459+0000","last_change":"2024-09-17T07:16:35.068863+0000","last_active":"2024-09-17T07:18:20.313459+0000","last_peered":"2024-09-17T07:18:20.313459+0000","last_clean":"2024-09-17T07:18:20.313459+0000","last_became_active":"2024-09-17T07:02:18.363253+0000","last_became_peered":"2024-09-17T07:02:18.363253+0000","last_unstale":"2024-09-17T07:18:20.313459+0000","last_undegraded":"2024-09-17T07:18:20.313459+0000","last_fullsized":"2024-09-17T07:18:20.313459+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:35.068712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:16:35.068712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0062141719999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.7","version":"346'8895","reported_seq":8963,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-17T07:18:23.643413+0000","last_change":"2024-09-17T07:17:52.454061+0000","last_active":"2024-09-17T07:18:23.643413+0000","last_peered":"2024-09-17T07:18:23.643413+0000","last_clean":"2024-09-17T07:18:23.643413+0000","last_became_active":"2024-09-17T07:17:52.402213+0000","last_became_peered":"2024-09-17T07:17:52.402213+0000","last_unstale":"2024-09-17T07:18:23.643413+0000","last_undegraded":"2024-09-17T07:18:23.643413+0000","last_fullsized":"2024-09-17T07:18:23.643413+0000","mapping_epoch":297,"log_start":"312'3000","ondisk_log_start":"312'3000","created":297,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":5895,"log_dups_size":0,"ondisk_log_size":5895,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":183894016,"num_objects":2806,"num_object_clones":0,"num_object_copies":8418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4973,"num_write_kb":248928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"242'23","reported_seq":449,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-17T07:18:20.307638+0000","last_change":"2024-09-17T07:17:34.108758+0000","last_active":"2024-09-17T07:18:20.307638+0000","last_peered":"2024-09-17T07:18:20.307638+0000","last_clean":"2024-09-17T07:18:20.307638+0000","last_became_active":"2024-09-17T07:10:49.723292+0000","last_became_peered":"2024-09-17T07:10:49.723292+0000","last_unstale":"2024-09-17T07:18:20.307638+0000","last_undegraded":"2024-09-17T07:18:20.307638+0000","last_fullsized":"2024-09-17T07:18:20.307638+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'23","last_scrub_stamp":"2024-09-17T07:17:34.108619+0000","last_deep_scrub":"150'17","last_deep_scrub_stamp":"2024-09-17T07:06:38.775788+0000","last_clean_scrub_stamp":"2024-09-17T07:17:34.108619+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:57.460011+0000","scrub_duration":0.018319322999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"68'2","reported_seq":379,"reported_epoch":345,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:18:20.311067+0000","last_change":"2024-09-17T07:16:17.371234+0000","last_active":"2024-09-17T07:18:20.311067+0000","last_peered":"2024-09-17T07:18:20.311067+0000","last_clean":"2024-09-17T07:16:15.358653+0000","last_became_active":"2024-09-17T07:16:17.371234+0000","last_became_peered":"2024-09-17T07:16:17.371234+0000","last_unstale":"2024-09-17T07:18:20.311067+0000","last_undegraded":"2024-09-17T07:16:17.366653+0000","last_fullsized":"2024-09-17T07:16:17.366079+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-09-17T07:14:56.533798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:14:56.533798+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003207115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[{"shards":"0,4","objects":2}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":326,"reported_epoch":345,"state":"active+undersized","last_fresh":"2024-09-17T07:18:20.311874+0000","last_change":"2024-09-17T07:16:17.367290+0000","last_active":"2024-09-17T07:18:20.311874+0000","last_peered":"2024-09-17T07:18:20.311874+0000","last_clean":"2024-09-17T07:16:15.494380+0000","last_became_active":"2024-09-17T07:16:17.367290+0000","last_became_peered":"2024-09-17T07:16:17.367290+0000","last_unstale":"2024-09-17T07:18:20.311874+0000","last_undegraded":"2024-09-17T07:18:20.311874+0000","last_fullsized":"2024-09-17T07:16:17.363484+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:51.586929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:11:17.479036+0000","last_clean_scrub_stamp":"2024-09-17T07:15:51.586929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001274435,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":194,"reported_epoch":345,"state":"active+undersized","last_fresh":"2024-09-17T07:18:20.313718+0000","last_change":"2024-09-17T07:16:17.379791+0000","last_active":"2024-09-17T07:18:20.313718+0000","last_peered":"2024-09-17T07:18:20.313718+0000","last_clean":"2024-09-17T07:16:15.590199+0000","last_became_active":"2024-09-17T07:16:17.379791+0000","last_became_peered":"2024-09-17T07:16:17.379791+0000","last_unstale":"2024-09-17T07:18:20.313718+0000","last_undegraded":"2024-09-17T07:18:20.313718+0000","last_fullsized":"2024-09-17T07:16:17.372333+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:13.937979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:13:13.937979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.018487173999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.18","version":"346'8793","reported_seq":15644,"reported_epoch":346,"state":"active+clean+remapped","last_fresh":"2024-09-17T07:18:22.991852+0000","last_change":"2024-09-17T07:18:18.290402+0000","last_active":"2024-09-17T07:18:22.991852+0000","last_peered":"2024-09-17T07:18:22.991852+0000","last_clean":"2024-09-17T07:18:22.991852+0000","last_became_active":"2024-09-17T07:18:18.281655+0000","last_became_peered":"2024-09-17T07:18:18.281655+0000","last_unstale":"2024-09-17T07:18:22.991852+0000","last_undegraded":"2024-09-17T07:18:22.991852+0000","last_fullsized":"2024-09-17T07:18:22.991852+0000","mapping_epoch":343,"log_start":"312'3100","ondisk_log_start":"312'3100","created":332,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":5693,"log_dups_size":0,"ondisk_log_size":5693,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185008128,"num_objects":2823,"num_object_clones":0,"num_object_copies":8469,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2823,"num_objects_unfound":0,"num_objects_dirty":2823,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4896,"num_write_kb":247008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3381,"num_bytes_recovered":221577216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6,0],"avail_no_missing":["2","0","6"],"object_location_counts":[{"shards":"0,2,6","objects":2823}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"242'15","reported_seq":412,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-17T07:18:24.505018+0000","last_change":"2024-09-17T07:18:24.505103+0000","last_active":"2024-09-17T07:18:24.505018+0000","last_peered":"2024-09-17T07:18:24.505018+0000","last_clean":"2024-09-17T07:18:24.505018+0000","last_became_active":"2024-09-17T07:09:24.408875+0000","last_became_peered":"2024-09-17T07:09:24.408875+0000","last_unstale":"2024-09-17T07:18:24.505018+0000","last_undegraded":"2024-09-17T07:18:24.505018+0000","last_fullsized":"2024-09-17T07:18:24.505018+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'15","last_scrub_stamp":"2024-09-17T07:18:24.505002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:18:24.505002+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:19:35.852600+0000","scrub_duration":0.002195809,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":388,"reported_epoch":345,"state":"active+undersized","last_fresh":"2024-09-17T07:18:20.306874+0000","last_change":"2024-09-17T07:16:17.367792+0000","last_active":"2024-09-17T07:18:20.306874+0000","last_peered":"2024-09-17T07:18:20.306874+0000","last_clean":"2024-09-17T07:16:14.244722+0000","last_became_active":"2024-09-17T07:16:17.367792+0000","last_became_peered":"2024-09-17T07:16:17.367792+0000","last_unstale":"2024-09-17T07:18:20.306874+0000","last_undegraded":"2024-09-17T07:18:20.306874+0000","last_fullsized":"2024-09-17T07:16:17.366356+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:39.634698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:17.976267+0000","last_clean_scrub_stamp":"2024-09-17T07:15:39.634698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00117202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":53,"reported_epoch":345,"state":"active+undersized","last_fresh":"2024-09-17T07:18:20.315191+0000","last_change":"2024-09-17T07:16:17.378272+0000","last_active":"2024-09-17T07:18:20.315191+0000","last_peered":"2024-09-17T07:18:20.315191+0000","last_clean":"2024-09-17T07:02:14.999435+0000","last_became_active":"2024-09-17T07:16:17.378272+0000","last_became_peered":"2024-09-17T07:16:17.378272+0000","last_unstale":"2024-09-17T07:18:20.315191+0000","last_undegraded":"2024-09-17T07:18:20.315191+0000","last_fullsized":"2024-09-17T07:16:17.373119+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:40.845816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:18.367486+0000","last_clean_scrub_stamp":"2024-09-17T07:15:40.845816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":409,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-17T07:18:20.306407+0000","last_change":"2024-09-17T07:17:37.827552+0000","last_active":"2024-09-17T07:18:20.306407+0000","last_peered":"2024-09-17T07:18:20.306407+0000","last_clean":"2024-09-17T07:18:20.306407+0000","last_became_active":"2024-09-17T07:02:18.017825+0000","last_became_peered":"2024-09-17T07:02:18.017825+0000","last_unstale":"2024-09-17T07:18:20.306407+0000","last_undegraded":"2024-09-17T07:18:20.306407+0000","last_fullsized":"2024-09-17T07:18:20.306407+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:17:37.827296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:09:23.378805+0000","last_clean_scrub_stamp":"2024-09-17T07:17:37.827296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:19:02.444738+0000","scrub_duration":0.0050232260000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"346'8683","reported_seq":8898,"reported_epoch":346,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-17T07:18:23.002915+0000","last_change":"2024-09-17T07:17:55.381142+0000","last_active":"2024-09-17T07:18:23.002915+0000","last_peered":"2024-09-17T07:18:23.002915+0000","last_clean":"2024-09-17T07:17:53.318487+0000","last_became_active":"2024-09-17T07:17:55.370266+0000","last_became_peered":"2024-09-17T07:17:55.370266+0000","last_unstale":"2024-09-17T07:18:23.002915+0000","last_undegraded":"2024-09-17T07:18:23.002915+0000","last_fullsized":"2024-09-17T07:18:23.002915+0000","mapping_epoch":335,"log_start":"310'2700","ondisk_log_start":"310'2700","created":332,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":5,"last_scrub":"322'6203","last_scrub_stamp":"2024-09-17T07:17:25.670907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:17:25.670907+0000","objects_scrubbed":6134,"log_size":5983,"log_dups_size":0,"ondisk_log_size":5983,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2.260616706,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184057856,"num_objects":2809,"num_object_clones":0,"num_object_copies":8427,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8427,"num_objects_unfound":0,"num_objects_dirty":2809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4830,"num_write_kb":244416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[2,5,7],"avail_no_missing":["2","5","7"],"object_location_counts":[{"shards":"2,5,7","objects":2809}],"blocked_by":[],"up_primary":4,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":376,"reported_epoch":345,"state":"active+undersized","last_fresh":"2024-09-17T07:18:20.318171+0000","last_change":"2024-09-17T07:16:17.375999+0000","last_active":"2024-09-17T07:18:20.318171+0000","last_peered":"2024-09-17T07:18:20.318171+0000","last_clean":"2024-09-17T07:16:14.245868+0000","last_became_active":"2024-09-17T07:16:17.375999+0000","last_became_peered":"2024-09-17T07:16:17.375999+0000","last_unstale":"2024-09-17T07:18:20.318171+0000","last_undegraded":"2024-09-17T07:18:20.318171+0000","last_fullsized":"2024-09-17T07:16:17.363383+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:46.645080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:46.645080+0000","last_clean_scrub_stamp":"2024-09-17T07:15:46.645080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015661049999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"242'12","reported_seq":231,"reported_epoch":345,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:18:20.314988+0000","last_change":"2024-09-17T07:16:17.379665+0000","last_active":"2024-09-17T07:18:20.314988+0000","last_peered":"2024-09-17T07:18:20.314988+0000","last_clean":"2024-09-17T07:16:15.590425+0000","last_became_active":"2024-09-17T07:16:17.379665+0000","last_became_peered":"2024-09-17T07:16:17.379665+0000","last_unstale":"2024-09-17T07:18:20.314988+0000","last_undegraded":"2024-09-17T07:16:17.373659+0000","last_fullsized":"2024-09-17T07:16:17.373552+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'12","last_scrub_stamp":"2024-09-17T07:15:44.249858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:15:44.249858+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0022006600000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":3}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":369,"reported_epoch":345,"state":"active+undersized","last_fresh":"2024-09-17T07:18:20.313608+0000","last_change":"2024-09-17T07:16:17.377495+0000","last_active":"2024-09-17T07:18:20.313608+0000","last_peered":"2024-09-17T07:18:20.313608+0000","last_clean":"2024-09-17T07:16:16.170775+0000","last_became_active":"2024-09-17T07:16:17.377495+0000","last_became_peered":"2024-09-17T07:16:17.377495+0000","last_unstale":"2024-09-17T07:18:20.313608+0000","last_undegraded":"2024-09-17T07:18:20.313608+0000","last_fullsized":"2024-09-17T07:16:17.367529+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:55.191252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:14:55.191252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0090150360000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":341,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-17T07:18:20.315091+0000","last_change":"2024-09-17T07:14:52.058961+0000","last_active":"2024-09-17T07:18:20.315091+0000","last_peered":"2024-09-17T07:18:20.315091+0000","last_clean":"2024-09-17T07:18:20.315091+0000","last_became_active":"2024-09-17T07:02:18.365419+0000","last_became_peered":"2024-09-17T07:02:18.365419+0000","last_unstale":"2024-09-17T07:18:20.315091+0000","last_undegraded":"2024-09-17T07:18:20.315091+0000","last_fullsized":"2024-09-17T07:18:20.315091+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:52.058847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:14:52.058847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0054926549999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"242'20","reported_seq":174,"reported_epoch":345,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:18:20.317365+0000","last_change":"2024-09-17T07:16:17.380213+0000","last_active":"2024-09-17T07:18:20.317365+0000","last_peered":"2024-09-17T07:18:20.317365+0000","last_clean":"2024-09-17T07:12:20.630919+0000","last_became_active":"2024-09-17T07:16:17.380213+0000","last_became_peered":"2024-09-17T07:16:17.380213+0000","last_unstale":"2024-09-17T07:18:20.317365+0000","last_undegraded":"2024-09-17T07:16:17.375354+0000","last_fullsized":"2024-09-17T07:16:17.375256+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'20","last_scrub_stamp":"2024-09-17T07:14:05.528769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:14:05.528769+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0020069329999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":5}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":389,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-17T07:18:20.314073+0000","last_change":"2024-09-17T07:17:20.425482+0000","last_active":"2024-09-17T07:18:20.314073+0000","last_peered":"2024-09-17T07:18:20.314073+0000","last_clean":"2024-09-17T07:18:20.314073+0000","last_became_active":"2024-09-17T07:09:24.402355+0000","last_became_peered":"2024-09-17T07:09:24.402355+0000","last_unstale":"2024-09-17T07:18:20.314073+0000","last_undegraded":"2024-09-17T07:18:20.314073+0000","last_fullsized":"2024-09-17T07:18:20.314073+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:17:20.425212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:17:20.425212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:34.381390+0000","scrub_duration":0.0020024069999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"304'10","reported_seq":564,"reported_epoch":345,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:18:20.709672+0000","last_change":"2024-09-17T07:16:17.373566+0000","last_active":"2024-09-17T07:18:20.709672+0000","last_peered":"2024-09-17T07:18:20.709672+0000","last_clean":"2024-09-17T07:16:16.169875+0000","last_became_active":"2024-09-17T07:16:17.373566+0000","last_became_peered":"2024-09-17T07:16:17.373566+0000","last_unstale":"2024-09-17T07:18:20.709672+0000","last_undegraded":"2024-09-17T07:16:17.367771+0000","last_fullsized":"2024-09-17T07:16:17.367624+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'9","last_scrub_stamp":"2024-09-17T07:14:53.140133+0000","last_deep_scrub":"163'9","last_deep_scrub_stamp":"2024-09-17T07:07:34.933808+0000","last_clean_scrub_stamp":"2024-09-17T07:14:53.140133+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0088819989999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[{"shards":"0,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":362,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-17T07:18:20.312763+0000","last_change":"2024-09-17T07:16:34.364362+0000","last_active":"2024-09-17T07:18:20.312763+0000","last_peered":"2024-09-17T07:18:20.312763+0000","last_clean":"2024-09-17T07:18:20.312763+0000","last_became_active":"2024-09-17T07:09:24.409051+0000","last_became_peered":"2024-09-17T07:09:24.409051+0000","last_unstale":"2024-09-17T07:18:20.312763+0000","last_undegraded":"2024-09-17T07:18:20.312763+0000","last_fullsized":"2024-09-17T07:18:20.312763+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:34.363904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:26.619893+0000","last_clean_scrub_stamp":"2024-09-17T07:16:34.363904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046574870000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"242'15","reported_seq":411,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-17T07:18:20.313184+0000","last_change":"2024-09-17T07:16:34.100665+0000","last_active":"2024-09-17T07:18:20.313184+0000","last_peered":"2024-09-17T07:18:20.313184+0000","last_clean":"2024-09-17T07:18:20.313184+0000","last_became_active":"2024-09-17T07:09:24.406333+0000","last_became_peered":"2024-09-17T07:09:24.406333+0000","last_unstale":"2024-09-17T07:18:20.313184+0000","last_undegraded":"2024-09-17T07:18:20.313184+0000","last_fullsized":"2024-09-17T07:18:20.313184+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'15","last_scrub_stamp":"2024-09-17T07:16:34.100540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:16:34.100540+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033371669999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":372,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-17T07:18:20.313461+0000","last_change":"2024-09-17T07:17:18.561114+0000","last_active":"2024-09-17T07:18:20.313461+0000","last_peered":"2024-09-17T07:18:20.313461+0000","last_clean":"2024-09-17T07:18:20.313461+0000","last_became_active":"2024-09-17T07:09:24.402704+0000","last_became_peered":"2024-09-17T07:09:24.402704+0000","last_unstale":"2024-09-17T07:18:20.313461+0000","last_undegraded":"2024-09-17T07:18:20.313461+0000","last_fullsized":"2024-09-17T07:18:20.313461+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:17:18.560995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:17:18.560995+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:43.212331+0000","scrub_duration":0.0064484570000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"304'10","reported_seq":327,"reported_epoch":345,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:18:20.711183+0000","last_change":"2024-09-17T07:16:17.380279+0000","last_active":"2024-09-17T07:18:20.711183+0000","last_peered":"2024-09-17T07:18:20.711183+0000","last_clean":"2024-09-17T07:16:15.687380+0000","last_became_active":"2024-09-17T07:16:17.380279+0000","last_became_peered":"2024-09-17T07:16:17.380279+0000","last_unstale":"2024-09-17T07:18:20.711183+0000","last_undegraded":"2024-09-17T07:16:17.375037+0000","last_fullsized":"2024-09-17T07:16:17.374926+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'9","last_scrub_stamp":"2024-09-17T07:13:10.965855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:13:10.965855+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0088433720000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":354,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-17T07:18:20.308005+0000","last_change":"2024-09-17T07:16:31.078582+0000","last_active":"2024-09-17T07:18:20.308005+0000","last_peered":"2024-09-17T07:18:20.308005+0000","last_clean":"2024-09-17T07:18:20.308005+0000","last_became_active":"2024-09-17T07:02:18.018716+0000","last_became_peered":"2024-09-17T07:02:18.018716+0000","last_unstale":"2024-09-17T07:18:20.308005+0000","last_undegraded":"2024-09-17T07:18:20.308005+0000","last_fullsized":"2024-09-17T07:18:20.308005+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:31.078504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:16:31.078504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014265700000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"242'15","reported_seq":302,"reported_epoch":345,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:18:20.317767+0000","last_change":"2024-09-17T07:16:17.368982+0000","last_active":"2024-09-17T07:18:20.317767+0000","last_peered":"2024-09-17T07:18:20.317767+0000","last_clean":"2024-09-17T07:12:20.132949+0000","last_became_active":"2024-09-17T07:16:17.368982+0000","last_became_peered":"2024-09-17T07:16:17.368982+0000","last_unstale":"2024-09-17T07:18:20.317767+0000","last_undegraded":"2024-09-17T07:16:17.363353+0000","last_fullsized":"2024-09-17T07:16:17.363248+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'15","last_scrub_stamp":"2024-09-17T07:13:48.500257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:13:48.500257+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.005053564,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[{"shards":"2,6","objects":4}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":389,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-17T07:18:20.317751+0000","last_change":"2024-09-17T07:17:18.347374+0000","last_active":"2024-09-17T07:18:20.317751+0000","last_peered":"2024-09-17T07:18:20.317751+0000","last_clean":"2024-09-17T07:18:20.317751+0000","last_became_active":"2024-09-17T07:09:24.408139+0000","last_became_peered":"2024-09-17T07:09:24.408139+0000","last_unstale":"2024-09-17T07:18:20.317751+0000","last_undegraded":"2024-09-17T07:18:20.317751+0000","last_fullsized":"2024-09-17T07:18:20.317751+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:17:18.347257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:10:36.334827+0000","last_clean_scrub_stamp":"2024-09-17T07:17:18.347257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:21.549853+0000","scrub_duration":0.013321,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":205,"reported_epoch":345,"state":"active+undersized","last_fresh":"2024-09-17T07:18:20.312532+0000","last_change":"2024-09-17T07:16:17.373522+0000","last_active":"2024-09-17T07:18:20.312532+0000","last_peered":"2024-09-17T07:18:20.312532+0000","last_clean":"2024-09-17T07:16:15.494646+0000","last_became_active":"2024-09-17T07:16:17.373522+0000","last_became_peered":"2024-09-17T07:16:17.373522+0000","last_unstale":"2024-09-17T07:18:20.312532+0000","last_undegraded":"2024-09-17T07:18:20.312532+0000","last_fullsized":"2024-09-17T07:16:17.364098+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:37.523283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:15:37.523283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013475850000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":359,"reported_epoch":345,"state":"active+undersized","last_fresh":"2024-09-17T07:18:20.311700+0000","last_change":"2024-09-17T07:16:17.371270+0000","last_active":"2024-09-17T07:18:20.311700+0000","last_peered":"2024-09-17T07:18:20.311700+0000","last_clean":"2024-09-17T07:16:15.494422+0000","last_became_active":"2024-09-17T07:16:17.371270+0000","last_became_peered":"2024-09-17T07:16:17.371270+0000","last_unstale":"2024-09-17T07:18:20.311700+0000","last_undegraded":"2024-09-17T07:18:20.311700+0000","last_fullsized":"2024-09-17T07:16:17.363365+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:40.540336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:15:40.540336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001212982,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"242'19","reported_seq":384,"reported_epoch":345,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:18:20.308258+0000","last_change":"2024-09-17T07:16:17.382006+0000","last_active":"2024-09-17T07:18:20.308258+0000","last_peered":"2024-09-17T07:18:20.308258+0000","last_clean":"2024-09-17T07:16:15.358100+0000","last_became_active":"2024-09-17T07:16:17.382006+0000","last_became_peered":"2024-09-17T07:16:17.382006+0000","last_unstale":"2024-09-17T07:18:20.308258+0000","last_undegraded":"2024-09-17T07:16:17.365274+0000","last_fullsized":"2024-09-17T07:16:17.365168+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'19","last_scrub_stamp":"2024-09-17T07:14:05.729535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:14:05.729535+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.008660559,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"94'2","reported_seq":96,"reported_epoch":345,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:18:20.313274+0000","last_change":"2024-09-17T07:16:17.374418+0000","last_active":"2024-09-17T07:18:20.313274+0000","last_peered":"2024-09-17T07:18:20.313274+0000","last_clean":"2024-09-17T07:04:06.025673+0000","last_became_active":"2024-09-17T07:16:17.374418+0000","last_became_peered":"2024-09-17T07:16:17.374418+0000","last_unstale":"2024-09-17T07:18:20.313274+0000","last_undegraded":"2024-09-17T07:16:17.367359+0000","last_fullsized":"2024-09-17T07:16:17.367242+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-09-17T07:15:48.907821+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-09-17T07:07:31.717888+0000","last_clean_scrub_stamp":"2024-09-17T07:15:48.907821+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[{"shards":"0,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":219,"reported_epoch":345,"state":"active+undersized","last_fresh":"2024-09-17T07:18:20.313599+0000","last_change":"2024-09-17T07:16:17.378918+0000","last_active":"2024-09-17T07:18:20.313599+0000","last_peered":"2024-09-17T07:18:20.313599+0000","last_clean":"2024-09-17T07:09:22.372509+0000","last_became_active":"2024-09-17T07:16:17.378918+0000","last_became_peered":"2024-09-17T07:16:17.378918+0000","last_unstale":"2024-09-17T07:18:20.313599+0000","last_undegraded":"2024-09-17T07:18:20.313599+0000","last_fullsized":"2024-09-17T07:16:17.367596+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:41.858093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:15:41.858093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037672679999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":209,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-17T07:18:23.515605+0000","last_change":"2024-09-17T07:18:23.515698+0000","last_active":"2024-09-17T07:18:23.515605+0000","last_peered":"2024-09-17T07:18:23.515605+0000","last_clean":"2024-09-17T07:18:23.515605+0000","last_became_active":"2024-09-17T07:09:24.400791+0000","last_became_peered":"2024-09-17T07:09:24.400791+0000","last_unstale":"2024-09-17T07:18:23.515605+0000","last_undegraded":"2024-09-17T07:18:23.515605+0000","last_fullsized":"2024-09-17T07:18:23.515605+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:18:23.515582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:49.573666+0000","last_clean_scrub_stamp":"2024-09-17T07:18:23.515582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:19:29.875635+0000","scrub_duration":0.0022931869999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"242'8","reported_seq":272,"reported_epoch":346,"state":"active+clean+scrubbing","last_fresh":"2024-09-17T07:18:21.317556+0000","last_change":"2024-09-17T07:18:23.016029+0000","last_active":"2024-09-17T07:18:21.317556+0000","last_peered":"2024-09-17T07:18:21.317556+0000","last_clean":"2024-09-17T07:18:21.317556+0000","last_became_active":"2024-09-17T07:09:24.396368+0000","last_became_peered":"2024-09-17T07:09:24.396368+0000","last_unstale":"2024-09-17T07:18:21.317556+0000","last_undegraded":"2024-09-17T07:18:21.317556+0000","last_fullsized":"2024-09-17T07:18:21.317556+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'8","last_scrub_stamp":"2024-09-17T07:16:32.727780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:16:32.727780+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0051147279999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":369,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-17T07:18:20.313921+0000","last_change":"2024-09-17T07:16:37.093569+0000","last_active":"2024-09-17T07:18:20.313921+0000","last_peered":"2024-09-17T07:18:20.313921+0000","last_clean":"2024-09-17T07:18:20.313921+0000","last_became_active":"2024-09-17T07:02:12.008898+0000","last_became_peered":"2024-09-17T07:02:12.008898+0000","last_unstale":"2024-09-17T07:18:20.313921+0000","last_undegraded":"2024-09-17T07:18:20.313921+0000","last_fullsized":"2024-09-17T07:18:20.313921+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:37.093460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:16:37.093460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0048915060000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":377,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-17T07:18:20.316651+0000","last_change":"2024-09-17T07:17:34.366051+0000","last_active":"2024-09-17T07:18:20.316651+0000","last_peered":"2024-09-17T07:18:20.316651+0000","last_clean":"2024-09-17T07:18:20.316651+0000","last_became_active":"2024-09-17T07:02:16.014982+0000","last_became_peered":"2024-09-17T07:02:16.014982+0000","last_unstale":"2024-09-17T07:18:20.316651+0000","last_undegraded":"2024-09-17T07:18:20.316651+0000","last_fullsized":"2024-09-17T07:18:20.316651+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:17:34.365905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:54.553970+0000","last_clean_scrub_stamp":"2024-09-17T07:17:34.365905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:19:02.007920+0000","scrub_duration":0.0089364449999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":352,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-17T07:18:20.313799+0000","last_change":"2024-09-17T07:17:27.977910+0000","last_active":"2024-09-17T07:18:20.313799+0000","last_peered":"2024-09-17T07:18:20.313799+0000","last_clean":"2024-09-17T07:18:20.313799+0000","last_became_active":"2024-09-17T07:02:18.021014+0000","last_became_peered":"2024-09-17T07:02:18.021014+0000","last_unstale":"2024-09-17T07:18:20.313799+0000","last_undegraded":"2024-09-17T07:18:20.313799+0000","last_fullsized":"2024-09-17T07:18:20.313799+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:17:27.977803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:17:27.977803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:29.675833+0000","scrub_duration":0.002005036,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"242'19","reported_seq":253,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-17T07:18:20.314947+0000","last_change":"2024-09-17T07:17:33.982980+0000","last_active":"2024-09-17T07:18:20.314947+0000","last_peered":"2024-09-17T07:18:20.314947+0000","last_clean":"2024-09-17T07:18:20.314947+0000","last_became_active":"2024-09-17T07:09:24.404942+0000","last_became_peered":"2024-09-17T07:09:24.404942+0000","last_unstale":"2024-09-17T07:18:20.314947+0000","last_undegraded":"2024-09-17T07:18:20.314947+0000","last_fullsized":"2024-09-17T07:18:20.314947+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'19","last_scrub_stamp":"2024-09-17T07:17:33.982784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:17:33.982784+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:38.944965+0000","scrub_duration":0.0043671170000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":380,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-17T07:18:20.321023+0000","last_change":"2024-09-17T07:16:32.647214+0000","last_active":"2024-09-17T07:18:20.321023+0000","last_peered":"2024-09-17T07:18:20.321023+0000","last_clean":"2024-09-17T07:18:20.321023+0000","last_became_active":"2024-09-17T07:02:12.010195+0000","last_became_peered":"2024-09-17T07:02:12.010195+0000","last_unstale":"2024-09-17T07:18:20.321023+0000","last_undegraded":"2024-09-17T07:18:20.321023+0000","last_fullsized":"2024-09-17T07:18:20.321023+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:32.647069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:16:32.647069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011908693,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":354,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-17T07:18:20.314036+0000","last_change":"2024-09-17T07:17:22.916134+0000","last_active":"2024-09-17T07:18:20.314036+0000","last_peered":"2024-09-17T07:18:20.314036+0000","last_clean":"2024-09-17T07:18:20.314036+0000","last_became_active":"2024-09-17T07:02:16.016183+0000","last_became_peered":"2024-09-17T07:02:16.016183+0000","last_unstale":"2024-09-17T07:18:20.314036+0000","last_undegraded":"2024-09-17T07:18:20.314036+0000","last_fullsized":"2024-09-17T07:18:20.314036+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:17:22.915543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:17:22.915543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:39.898743+0000","scrub_duration":0.0086489989999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":209,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-17T07:18:20.312441+0000","last_change":"2024-09-17T07:17:34.551610+0000","last_active":"2024-09-17T07:18:20.312441+0000","last_peered":"2024-09-17T07:18:20.312441+0000","last_clean":"2024-09-17T07:18:20.312441+0000","last_became_active":"2024-09-17T07:09:24.392216+0000","last_became_peered":"2024-09-17T07:09:24.392216+0000","last_unstale":"2024-09-17T07:18:20.312441+0000","last_undegraded":"2024-09-17T07:18:20.312441+0000","last_fullsized":"2024-09-17T07:18:20.312441+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:17:34.551498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:17:34.551498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:38.895494+0000","scrub_duration":0.002777912,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.10","version":"346'8829","reported_seq":15657,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-17T07:18:23.111959+0000","last_change":"2024-09-17T07:18:17.281071+0000","last_active":"2024-09-17T07:18:23.111959+0000","last_peered":"2024-09-17T07:18:23.111959+0000","last_clean":"2024-09-17T07:18:23.111959+0000","last_became_active":"2024-09-17T07:18:17.272736+0000","last_became_peered":"2024-09-17T07:18:17.272736+0000","last_unstale":"2024-09-17T07:18:23.111959+0000","last_undegraded":"2024-09-17T07:18:23.111959+0000","last_fullsized":"2024-09-17T07:18:23.111959+0000","mapping_epoch":342,"log_start":"312'3000","ondisk_log_start":"312'3000","created":332,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":5829,"log_dups_size":0,"ondisk_log_size":5829,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":182583296,"num_objects":2786,"num_object_clones":0,"num_object_copies":8358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4932,"num_write_kb":246976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3368,"num_bytes_recovered":220659712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"326'2","reported_seq":354,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-17T07:18:20.317193+0000","last_change":"2024-09-17T07:17:36.974642+0000","last_active":"2024-09-17T07:18:20.317193+0000","last_peered":"2024-09-17T07:18:20.317193+0000","last_clean":"2024-09-17T07:18:20.317193+0000","last_became_active":"2024-09-17T07:02:18.021855+0000","last_became_peered":"2024-09-17T07:02:18.021855+0000","last_unstale":"2024-09-17T07:18:20.317193+0000","last_undegraded":"2024-09-17T07:18:20.317193+0000","last_fullsized":"2024-09-17T07:18:20.317193+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'2","last_scrub_stamp":"2024-09-17T07:17:36.974358+0000","last_deep_scrub":"326'2","last_deep_scrub_stamp":"2024-09-17T07:17:36.974358+0000","last_clean_scrub_stamp":"2024-09-17T07:17:36.974358+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:37.640310+0000","scrub_duration":0.015149954,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":370,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-17T07:18:20.311446+0000","last_change":"2024-09-17T07:16:39.053518+0000","last_active":"2024-09-17T07:18:20.311446+0000","last_peered":"2024-09-17T07:18:20.311446+0000","last_clean":"2024-09-17T07:18:20.311446+0000","last_became_active":"2024-09-17T07:02:12.005329+0000","last_became_peered":"2024-09-17T07:02:12.005329+0000","last_unstale":"2024-09-17T07:18:20.311446+0000","last_undegraded":"2024-09-17T07:18:20.311446+0000","last_fullsized":"2024-09-17T07:18:20.311446+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:39.053386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:16:39.053386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0044426509999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"294'25","reported_seq":447,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-17T07:18:20.313096+0000","last_change":"2024-09-17T07:17:43.574192+0000","last_active":"2024-09-17T07:18:20.313096+0000","last_peered":"2024-09-17T07:18:20.313096+0000","last_clean":"2024-09-17T07:18:20.313096+0000","last_became_active":"2024-09-17T07:10:49.730948+0000","last_became_peered":"2024-09-17T07:10:49.730948+0000","last_unstale":"2024-09-17T07:18:20.313096+0000","last_undegraded":"2024-09-17T07:18:20.313096+0000","last_fullsized":"2024-09-17T07:18:20.313096+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'25","last_scrub_stamp":"2024-09-17T07:17:43.574077+0000","last_deep_scrub":"294'25","last_deep_scrub_stamp":"2024-09-17T07:16:00.471311+0000","last_clean_scrub_stamp":"2024-09-17T07:17:43.574077+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:19:00.914807+0000","scrub_duration":0.0071385140000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"75'8","reported_seq":583,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-17T07:18:20.711343+0000","last_change":"2024-09-17T07:17:25.451002+0000","last_active":"2024-09-17T07:18:20.711343+0000","last_peered":"2024-09-17T07:18:20.711343+0000","last_clean":"2024-09-17T07:18:20.711343+0000","last_became_active":"2024-09-17T07:02:16.020560+0000","last_became_peered":"2024-09-17T07:02:16.020560+0000","last_unstale":"2024-09-17T07:18:20.711343+0000","last_undegraded":"2024-09-17T07:18:20.711343+0000","last_fullsized":"2024-09-17T07:18:20.711343+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:17:25.450919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:17:25.450919+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:53.609078+0000","scrub_duration":0.0013309369999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.11","version":"346'9028","reported_seq":16210,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-17T07:18:23.862373+0000","last_change":"2024-09-17T07:18:16.281801+0000","last_active":"2024-09-17T07:18:23.862373+0000","last_peered":"2024-09-17T07:18:23.862373+0000","last_clean":"2024-09-17T07:18:23.862373+0000","last_became_active":"2024-09-17T07:18:16.270511+0000","last_became_peered":"2024-09-17T07:18:16.270511+0000","last_unstale":"2024-09-17T07:18:23.862373+0000","last_undegraded":"2024-09-17T07:18:23.862373+0000","last_fullsized":"2024-09-17T07:18:23.862373+0000","mapping_epoch":341,"log_start":"312'3400","ondisk_log_start":"312'3400","created":332,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":5,"last_scrub":"320'6219","last_scrub_stamp":"2024-09-17T07:17:22.401249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:17:22.401249+0000","objects_scrubbed":6144,"log_size":5628,"log_dups_size":0,"ondisk_log_size":5628,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2.0885702689999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185139200,"num_objects":2825,"num_object_clones":0,"num_object_copies":8475,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2825,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5051,"num_write_kb":252032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3482,"num_bytes_recovered":228196352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":51,"reported_epoch":345,"state":"active+undersized","last_fresh":"2024-09-17T07:18:20.312902+0000","last_change":"2024-09-17T07:16:17.375358+0000","last_active":"2024-09-17T07:18:20.312902+0000","last_peered":"2024-09-17T07:18:20.312902+0000","last_clean":"2024-09-17T07:02:17.002904+0000","last_became_active":"2024-09-17T07:16:17.375358+0000","last_became_peered":"2024-09-17T07:16:17.375358+0000","last_unstale":"2024-09-17T07:18:20.312902+0000","last_undegraded":"2024-09-17T07:18:20.312902+0000","last_fullsized":"2024-09-17T07:16:17.368647+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:37.885570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:15:37.885570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":220,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-17T07:18:20.318085+0000","last_change":"2024-09-17T07:17:16.329912+0000","last_active":"2024-09-17T07:18:20.318085+0000","last_peered":"2024-09-17T07:18:20.318085+0000","last_clean":"2024-09-17T07:18:20.318085+0000","last_became_active":"2024-09-17T07:09:24.403360+0000","last_became_peered":"2024-09-17T07:09:24.403360+0000","last_unstale":"2024-09-17T07:18:20.318085+0000","last_undegraded":"2024-09-17T07:18:20.318085+0000","last_fullsized":"2024-09-17T07:18:20.318085+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:17:16.329797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:10:29.325664+0000","last_clean_scrub_stamp":"2024-09-17T07:17:16.329797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:28.864230+0000","scrub_duration":0.0033990230000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"242'19","reported_seq":452,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-17T07:18:20.320961+0000","last_change":"2024-09-17T07:17:17.376514+0000","last_active":"2024-09-17T07:18:20.320961+0000","last_peered":"2024-09-17T07:18:20.320961+0000","last_clean":"2024-09-17T07:18:20.320961+0000","last_became_active":"2024-09-17T07:03:48.760390+0000","last_became_peered":"2024-09-17T07:03:48.760390+0000","last_unstale":"2024-09-17T07:18:20.320961+0000","last_undegraded":"2024-09-17T07:18:20.320961+0000","last_fullsized":"2024-09-17T07:18:20.320961+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'19","last_scrub_stamp":"2024-09-17T07:17:17.375166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:17:17.375166+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:41.472827+0000","scrub_duration":0.014344902,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":357,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-17T07:18:21.314423+0000","last_change":"2024-09-17T07:17:31.538898+0000","last_active":"2024-09-17T07:18:21.314423+0000","last_peered":"2024-09-17T07:18:21.314423+0000","last_clean":"2024-09-17T07:18:21.314423+0000","last_became_active":"2024-09-17T07:02:16.019701+0000","last_became_peered":"2024-09-17T07:02:16.019701+0000","last_unstale":"2024-09-17T07:18:21.314423+0000","last_undegraded":"2024-09-17T07:18:21.314423+0000","last_fullsized":"2024-09-17T07:18:21.314423+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:17:31.538812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:17:31.538812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:46.566309+0000","scrub_duration":0.0014632390000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.12","version":"346'8777","reported_seq":8848,"reported_epoch":346,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2024-09-17T07:18:24.549401+0000","last_change":"2024-09-17T07:17:55.367927+0000","last_active":"2024-09-17T07:18:24.549401+0000","last_peered":"2024-09-17T07:18:24.549401+0000","last_clean":"2024-09-17T07:16:04.025356+0000","last_became_active":"2024-09-17T07:17:55.362534+0000","last_became_peered":"2024-09-17T07:17:55.362534+0000","last_unstale":"2024-09-17T07:18:24.549401+0000","last_undegraded":"2024-09-17T07:17:55.278241+0000","last_fullsized":"2024-09-17T07:17:55.254538+0000","mapping_epoch":335,"log_start":"312'3100","ondisk_log_start":"312'3100","created":332,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":5677,"log_dups_size":0,"ondisk_log_size":5677,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180453376,"num_objects":2754,"num_object_clones":0,"num_object_copies":8262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2754,"num_objects_misplaced":5508,"num_objects_unfound":0,"num_objects_dirty":2754,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4909,"num_write_kb":245184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0,4","objects":2754}],"blocked_by":[],"up_primary":6,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":366,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-17T07:18:20.308556+0000","last_change":"2024-09-17T07:17:37.138113+0000","last_active":"2024-09-17T07:18:20.308556+0000","last_peered":"2024-09-17T07:18:20.308556+0000","last_clean":"2024-09-17T07:18:20.308556+0000","last_became_active":"2024-09-17T07:02:18.363752+0000","last_became_peered":"2024-09-17T07:02:18.363752+0000","last_unstale":"2024-09-17T07:18:20.308556+0000","last_undegraded":"2024-09-17T07:18:20.308556+0000","last_fullsized":"2024-09-17T07:18:20.308556+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:17:37.137989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:17:37.137989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:41.283275+0000","scrub_duration":0.012195245,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":410,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-17T07:18:20.318412+0000","last_change":"2024-09-17T07:16:44.559141+0000","last_active":"2024-09-17T07:18:20.318412+0000","last_peered":"2024-09-17T07:18:20.318412+0000","last_clean":"2024-09-17T07:18:20.318412+0000","last_became_active":"2024-09-17T07:03:48.759384+0000","last_became_peered":"2024-09-17T07:03:48.759384+0000","last_unstale":"2024-09-17T07:18:20.318412+0000","last_undegraded":"2024-09-17T07:18:20.318412+0000","last_fullsized":"2024-09-17T07:18:20.318412+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:44.558901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:11:18.559877+0000","last_clean_scrub_stamp":"2024-09-17T07:16:44.558901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.031473045999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"242'27","reported_seq":321,"reported_epoch":345,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:18:20.313584+0000","last_change":"2024-09-17T07:16:17.368008+0000","last_active":"2024-09-17T07:18:20.313584+0000","last_peered":"2024-09-17T07:18:20.313584+0000","last_clean":"2024-09-17T07:12:20.337498+0000","last_became_active":"2024-09-17T07:16:17.368008+0000","last_became_peered":"2024-09-17T07:16:17.368008+0000","last_unstale":"2024-09-17T07:18:20.313584+0000","last_undegraded":"2024-09-17T07:16:17.365648+0000","last_fullsized":"2024-09-17T07:16:17.365550+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'27","last_scrub_stamp":"2024-09-17T07:15:45.855430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:15:45.855430+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.017595989999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":7}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":51,"reported_epoch":345,"state":"active+undersized","last_fresh":"2024-09-17T07:18:20.307979+0000","last_change":"2024-09-17T07:16:17.371494+0000","last_active":"2024-09-17T07:18:20.307979+0000","last_peered":"2024-09-17T07:18:20.307979+0000","last_clean":"2024-09-17T07:02:14.999435+0000","last_became_active":"2024-09-17T07:16:17.371494+0000","last_became_peered":"2024-09-17T07:16:17.371494+0000","last_unstale":"2024-09-17T07:18:20.307979+0000","last_undegraded":"2024-09-17T07:18:20.307979+0000","last_fullsized":"2024-09-17T07:16:17.366463+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:00.509283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:25.794418+0000","last_clean_scrub_stamp":"2024-09-17T07:14:00.509283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.13","version":"346'11758","reported_seq":18914,"reported_epoch":346,"state":"active+clean+remapped","last_fresh":"2024-09-17T07:18:22.974894+0000","last_change":"2024-09-17T07:18:13.461884+0000","last_active":"2024-09-17T07:18:22.974894+0000","last_peered":"2024-09-17T07:18:22.974894+0000","last_clean":"2024-09-17T07:18:22.974894+0000","last_became_active":"2024-09-17T07:18:13.450785+0000","last_became_peered":"2024-09-17T07:18:13.450785+0000","last_unstale":"2024-09-17T07:18:22.974894+0000","last_undegraded":"2024-09-17T07:18:22.974894+0000","last_fullsized":"2024-09-17T07:18:22.974894+0000","mapping_epoch":339,"log_start":"320'5801","ondisk_log_start":"320'5801","created":332,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":4348,"log_size":5957,"log_dups_size":0,"ondisk_log_size":5957,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":179994624,"num_objects":2747,"num_object_clones":0,"num_object_copies":8241,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2747,"num_objects_unfound":0,"num_objects_dirty":2747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4823,"num_write_kb":242208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3360,"num_bytes_recovered":218824704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2,4],"avail_no_missing":["7","2","4"],"object_location_counts":[{"shards":"2,4,7","objects":2747}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":217,"reported_epoch":345,"state":"active+undersized","last_fresh":"2024-09-17T07:18:20.320639+0000","last_change":"2024-09-17T07:16:17.377951+0000","last_active":"2024-09-17T07:18:20.320639+0000","last_peered":"2024-09-17T07:18:20.320639+0000","last_clean":"2024-09-17T07:16:14.246741+0000","last_became_active":"2024-09-17T07:16:17.377951+0000","last_became_peered":"2024-09-17T07:16:17.377951+0000","last_unstale":"2024-09-17T07:18:20.320639+0000","last_undegraded":"2024-09-17T07:18:20.320639+0000","last_fullsized":"2024-09-17T07:16:17.364654+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:02.473467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:15:02.473467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001733163,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":200,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-17T07:18:20.315038+0000","last_change":"2024-09-17T07:16:31.651379+0000","last_active":"2024-09-17T07:18:20.315038+0000","last_peered":"2024-09-17T07:18:20.315038+0000","last_clean":"2024-09-17T07:18:20.315038+0000","last_became_active":"2024-09-17T07:09:24.404795+0000","last_became_peered":"2024-09-17T07:09:24.404795+0000","last_unstale":"2024-09-17T07:18:20.315038+0000","last_undegraded":"2024-09-17T07:18:20.315038+0000","last_fullsized":"2024-09-17T07:18:20.315038+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:31.651247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:16:31.651247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0075767350000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"242'23","reported_seq":183,"reported_epoch":345,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:18:20.308144+0000","last_change":"2024-09-17T07:16:17.380972+0000","last_active":"2024-09-17T07:18:20.308144+0000","last_peered":"2024-09-17T07:18:20.308144+0000","last_clean":"2024-09-17T07:12:20.742449+0000","last_became_active":"2024-09-17T07:16:17.380972+0000","last_became_peered":"2024-09-17T07:16:17.380972+0000","last_unstale":"2024-09-17T07:18:20.308144+0000","last_undegraded":"2024-09-17T07:16:17.366757+0000","last_fullsized":"2024-09-17T07:16:17.366645+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'23","last_scrub_stamp":"2024-09-17T07:14:58.689120+0000","last_deep_scrub":"86'17","last_deep_scrub_stamp":"2024-09-17T07:03:18.241067+0000","last_clean_scrub_stamp":"2024-09-17T07:14:58.689120+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":376,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-09-17T07:18:20.308504+0000","last_change":"2024-09-17T07:17:17.309054+0000","last_active":"2024-09-17T07:18:20.308504+0000","last_peered":"2024-09-17T07:18:20.308504+0000","last_clean":"2024-09-17T07:18:20.308504+0000","last_became_active":"2024-09-17T07:09:24.393329+0000","last_became_peered":"2024-09-17T07:09:24.393329+0000","last_unstale":"2024-09-17T07:18:20.308504+0000","last_undegraded":"2024-09-17T07:18:20.308504+0000","last_fullsized":"2024-09-17T07:18:20.308504+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:17:17.308762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:17:17.308762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:25.154038+0000","scrub_duration":0.023046534,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":26,"stat_sum":{"num_bytes":5840437272,"num_objects":89119,"num_object_clones":0,"num_object_copies":267357,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16614,"num_objects_misplaced":33516,"num_objects_unfound":0,"num_objects_dirty":89119,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":157573,"num_write_kb":7894081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23577,"num_bytes_recovered":1543569408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16444895232,"data_stored":16444883016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":142945,"ondisk_log_size":142945,"up":67,"acting":73,"num_store_stats":7},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":83,"acting":83,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73,"ondisk_log_size":73,"up":81,"acting":81,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":101,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1113,"num_read_kb":870,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":109,"num_bytes_recovered":2228,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":13004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":733,"ondisk_log_size":733,"up":78,"acting":78,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":132,"num_read_kb":127,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":8,"acting":8,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1386,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":12,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":4578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":83,"acting":83,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037960,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2715688,"kb_used_data":1984800,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":91029464,"statfs":{"total":95995035648,"available":93214171136,"internally_reserved":0,"allocated":2032435200,"data_stored":2031793349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":748419531},"hb_peers":[0,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:17:30 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":1.0429999999999999,"15min":1.603},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.192},"max":{"1min":3.194,"5min":8.0800000000000001,"15min":52.478999999999999},"last":24.739999999999998},{"interface":"front","average":{"1min":0.73899999999999999,"5min":1.079,"15min":1.728},"min":{"1min":0.29399999999999998,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":2.1499999999999999,"5min":6.8120000000000003,"15min":52.871000000000002},"last":24.768000000000001}]},{"osd":1,"last update":"Tue Sep 17 07:15:24 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":1.4510000000000001,"15min":1.3859999999999999},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.223},"max":{"1min":1.573,"5min":15.801,"15min":73.549000000000007},"last":0.65300000000000002},{"interface":"front","average":{"1min":1.3779999999999999,"5min":1.2969999999999999,"15min":1.383},"min":{"1min":0.29699999999999999,"5min":0.26200000000000001,"15min":0.24099999999999999},"max":{"1min":4.7359999999999998,"5min":13.989000000000001,"15min":73.116},"last":0.71499999999999997}]},{"osd":2,"last update":"Tue Sep 17 07:17:30 2024","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.1359999999999999,"15min":1.1830000000000001},"min":{"1min":0.25,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4.0490000000000004,"5min":6.4900000000000002,"15min":52.055999999999997},"last":0.34300000000000003},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.161,"15min":1.1719999999999999},"min":{"1min":0.23200000000000001,"5min":0.224,"15min":0.224},"max":{"1min":8.5869999999999997,"5min":8.5869999999999997,"15min":52.064999999999998},"last":0.41099999999999998}]},{"osd":3,"last update":"Tue Sep 17 07:17:30 2024","interfaces":[{"interface":"back","average":{"1min":1.2669999999999999,"5min":1.1890000000000001,"15min":1.3759999999999999},"min":{"1min":0.30299999999999999,"5min":0.29199999999999998,"15min":0.28799999999999998},"max":{"1min":4.0970000000000004,"5min":6.4640000000000004,"15min":51.689},"last":0.378},{"interface":"front","average":{"1min":1.214,"5min":1.2470000000000001,"15min":1.482},"min":{"1min":0.35399999999999998,"5min":0.27000000000000002,"15min":0.26500000000000001},"max":{"1min":8.5340000000000007,"5min":8.5340000000000007,"15min":52.145000000000003},"last":0.47399999999999998}]},{"osd":4,"last update":"Tue Sep 17 07:17:30 2024","interfaces":[{"interface":"back","average":{"1min":2.4350000000000001,"5min":1.49,"15min":0.89200000000000002},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":21.140999999999998,"5min":21.140999999999998,"15min":21.140999999999998},"last":0.27300000000000002},{"interface":"front","average":{"1min":3.3380000000000001,"5min":1.694,"15min":1.004},"min":{"1min":0.35299999999999998,"5min":0.28699999999999998,"15min":0.24399999999999999},"max":{"1min":19.420000000000002,"5min":19.420000000000002,"15min":19.420000000000002},"last":0.23599999999999999}]},{"osd":5,"last update":"Tue Sep 17 07:17:30 2024","interfaces":[{"interface":"back","average":{"1min":2.2029999999999998,"5min":1.9330000000000001,"15min":1.079},"min":{"1min":0.28799999999999998,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":19.384,"5min":29.428999999999998,"15min":29.428999999999998},"last":0.35099999999999998},{"interface":"front","average":{"1min":1.794,"5min":2.0939999999999999,"15min":1.135},"min":{"1min":0.33500000000000002,"5min":0.26000000000000001,"15min":0.25700000000000001},"max":{"1min":19.483000000000001,"5min":30.059999999999999,"15min":30.059999999999999},"last":0.32300000000000001}]},{"osd":6,"last update":"Tue Sep 17 07:17:30 2024","interfaces":[{"interface":"back","average":{"1min":3.2650000000000001,"5min":1.7629999999999999,"15min":1.137},"min":{"1min":0.40100000000000002,"5min":0.32200000000000001,"15min":0.25900000000000001},"max":{"1min":22.221,"5min":22.221,"15min":25.363},"last":0.36399999999999999},{"interface":"front","average":{"1min":4.1269999999999998,"5min":1.9330000000000001,"15min":1.194},"min":{"1min":0.35999999999999999,"5min":0.28399999999999997,"15min":0.27500000000000002},"max":{"1min":22.202000000000002,"5min":22.202000000000002,"15min":23.571000000000002},"last":0.30499999999999999}]}]},{"osd":6,"up_from":50,"seq":214748365003,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3508552,"kb_used_data":2904000,"kb_used_omap":1,"kb_used_meta":604542,"kb_avail":90236600,"statfs":{"total":95995035648,"available":92402278400,"internally_reserved":0,"allocated":2973696000,"data_stored":2973022022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":619051463},"hb_peers":[0,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:18:15 2024","interfaces":[{"interface":"back","average":{"1min":1.6419999999999999,"5min":2.0899999999999999,"15min":2.121},"min":{"1min":0.218,"5min":0.20200000000000001,"15min":0.185},"max":{"1min":5.2229999999999999,"5min":126.902,"15min":126.902},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.876,"5min":1.7909999999999999,"15min":1.99},"min":{"1min":0.23200000000000001,"5min":0.20899999999999999,"15min":0.20799999999999999},"max":{"1min":2.7919999999999998,"5min":127.06399999999999,"15min":127.06399999999999},"last":0.29099999999999998}]},{"osd":1,"last update":"Tue Sep 17 07:16:10 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.996,"15min":0.84399999999999997},"min":{"1min":0.23799999999999999,"5min":0.22900000000000001,"15min":0.22600000000000001},"max":{"1min":2.2999999999999998,"5min":14.720000000000001,"15min":14.720000000000001},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.622,"5min":1.0109999999999999,"15min":0.88800000000000001},"min":{"1min":0.29099999999999998,"5min":0.23400000000000001,"15min":0.22900000000000001},"max":{"1min":2.323,"5min":14.699,"15min":14.699},"last":0.46600000000000003}]},{"osd":2,"last update":"Tue Sep 17 07:18:15 2024","interfaces":[{"interface":"back","average":{"1min":1.742,"5min":1.0449999999999999,"15min":0.98599999999999999},"min":{"1min":0.251,"5min":0.219,"15min":0.219},"max":{"1min":5.266,"5min":5.266,"15min":33.079000000000001},"last":0.48299999999999998},{"interface":"front","average":{"1min":1.0369999999999999,"5min":0.81499999999999995,"15min":0.90400000000000003},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":2.8380000000000001,"5min":4.8399999999999999,"15min":33.100000000000001},"last":0.318}]},{"osd":3,"last update":"Tue Sep 17 07:18:15 2024","interfaces":[{"interface":"back","average":{"1min":1.643,"5min":0.879,"15min":1.2609999999999999},"min":{"1min":0.27500000000000002,"5min":0.27000000000000002,"15min":0.23200000000000001},"max":{"1min":4.9320000000000004,"5min":4.9320000000000004,"15min":52.944000000000003},"last":0.41499999999999998},{"interface":"front","average":{"1min":1.7130000000000001,"5min":1.22,"15min":1.3700000000000001},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":5.2949999999999999,"5min":5.7690000000000001,"15min":52.747},"last":0.52700000000000002}]},{"osd":4,"last update":"Tue Sep 17 07:18:15 2024","interfaces":[{"interface":"back","average":{"1min":2.0070000000000001,"5min":1.4330000000000001,"15min":2.129},"min":{"1min":0.27900000000000003,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":5.4800000000000004,"5min":7.1050000000000004,"15min":260.77199999999999},"last":0.92600000000000005},{"interface":"front","average":{"1min":3.2509999999999999,"5min":1.51,"15min":2.1819999999999999},"min":{"1min":0.36299999999999999,"5min":0.23300000000000001,"15min":0.19800000000000001},"max":{"1min":8.7929999999999993,"5min":8.7929999999999993,"15min":260.69099999999997},"last":0.996}]},{"osd":5,"last update":"Tue Sep 17 07:18:15 2024","interfaces":[{"interface":"back","average":{"1min":2.0649999999999999,"5min":1.8959999999999999,"15min":1.2230000000000001},"min":{"1min":0.32900000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":10.201000000000001,"5min":11.488,"15min":32.042000000000002},"last":0.96299999999999997},{"interface":"front","average":{"1min":2.0219999999999998,"5min":1.734,"15min":1.208},"min":{"1min":0.44,"5min":0.27200000000000002,"15min":0.251},"max":{"1min":5.5540000000000003,"5min":10.244999999999999,"15min":31.852},"last":0.39500000000000002}]},{"osd":7,"last update":"Tue Sep 17 07:17:36 2024","interfaces":[{"interface":"back","average":{"1min":1.6419999999999999,"5min":2.8010000000000002,"15min":1.446},"min":{"1min":0.29399999999999998,"5min":0.246,"15min":0.23699999999999999},"max":{"1min":3.8690000000000002,"5min":131.95699999999999,"15min":131.95699999999999},"last":1.3740000000000001},{"interface":"front","average":{"1min":1.377,"5min":2.7170000000000001,"15min":1.413},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":5.5279999999999996,"5min":131.833,"15min":131.833},"last":1.2889999999999999}]}]},{"osd":5,"up_from":42,"seq":180388626640,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2309672,"kb_used_data":1614752,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":91435480,"statfs":{"total":95995035648,"available":93629931520,"internally_reserved":0,"allocated":1653506048,"data_stored":1652863985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":711457222},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":0.92500000000000004,"15min":1.615},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":4.1900000000000004,"5min":4.2329999999999997,"15min":49.265000000000001},"last":0.28100000000000003},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.02,"15min":1.708},"min":{"1min":0.25800000000000001,"5min":0.221,"15min":0.219},"max":{"1min":4.7939999999999996,"5min":6.7149999999999999,"15min":47.350000000000001},"last":0.29499999999999998}]},{"osd":1,"last update":"Tue Sep 17 07:15:47 2024","interfaces":[{"interface":"back","average":{"1min":1.4319999999999999,"5min":1.0509999999999999,"15min":0.90100000000000002},"min":{"1min":0.30499999999999999,"5min":0.22,"15min":0.22},"max":{"1min":13.711,"5min":13.711,"15min":20.329000000000001},"last":0.437},{"interface":"front","average":{"1min":0.99399999999999999,"5min":1.3320000000000001,"15min":1.149},"min":{"1min":0.32000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4.617,"5min":6.7560000000000002,"15min":20.408999999999999},"last":0.42199999999999999}]},{"osd":2,"last update":"Tue Sep 17 07:17:55 2024","interfaces":[{"interface":"back","average":{"1min":1.248,"5min":1.105,"15min":1.145},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":4.4160000000000004,"5min":6.3419999999999996,"15min":20.486000000000001},"last":0.312},{"interface":"front","average":{"1min":1.1990000000000001,"5min":0.97799999999999998,"15min":1.04},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":7.71,"5min":7.71,"15min":23.925999999999998},"last":0.35799999999999998}]},{"osd":3,"last update":"Tue Sep 17 07:17:55 2024","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":1.333,"15min":1.482},"min":{"1min":0.27900000000000003,"5min":0.27100000000000002,"15min":0.247},"max":{"1min":3.3580000000000001,"5min":13.882,"15min":68.978999999999999},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.755,"5min":0.90400000000000003,"15min":1.3140000000000001},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.253},"max":{"1min":2.0720000000000001,"5min":4.327,"15min":69.084000000000003},"last":0.38800000000000001}]},{"osd":4,"last update":"Tue Sep 17 07:17:55 2024","interfaces":[{"interface":"back","average":{"1min":2.8929999999999998,"5min":1.9590000000000001,"15min":1.706},"min":{"1min":0.29899999999999999,"5min":0.23200000000000001,"15min":0.22},"max":{"1min":15.157,"5min":15.157,"15min":142.327},"last":96.668000000000006},{"interface":"front","average":{"1min":2.3239999999999998,"5min":1.7509999999999999,"15min":1.587},"min":{"1min":0.32800000000000001,"5min":0.25,"15min":0.20999999999999999},"max":{"1min":14.629,"5min":14.629,"15min":142.13200000000001},"last":96.801000000000002}]},{"osd":6,"last update":"Tue Sep 17 07:18:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1779999999999999,"5min":1.208,"15min":0.89800000000000002},"min":{"1min":0.245,"5min":0.245,"15min":0.23200000000000001},"max":{"1min":3.1749999999999998,"5min":6.8419999999999996,"15min":8.3149999999999995},"last":96.686000000000007},{"interface":"front","average":{"1min":1.9239999999999999,"5min":1.5089999999999999,"15min":1.018},"min":{"1min":0.29799999999999999,"5min":0.26700000000000002,"15min":0.24099999999999999},"max":{"1min":3.5750000000000002,"5min":8.4489999999999998,"15min":8.4489999999999998},"last":96.834999999999994}]},{"osd":7,"last update":"Tue Sep 17 07:17:30 2024","interfaces":[{"interface":"back","average":{"1min":1.129,"5min":1.631,"15min":1.0800000000000001},"min":{"1min":0.28399999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":2.9409999999999998,"5min":6.8869999999999996,"15min":6.8869999999999996},"last":0.373},{"interface":"front","average":{"1min":1.3029999999999999,"5min":1.3480000000000001,"15min":0.92300000000000004},"min":{"1min":0.34200000000000003,"5min":0.28100000000000003,"15min":0.27100000000000002},"max":{"1min":3.7610000000000001,"5min":4.9260000000000002,"15min":5.641},"last":0.32500000000000001}]}]},{"osd":4,"up_from":33,"seq":141733920982,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4155312,"kb_used_data":3470376,"kb_used_omap":1,"kb_used_meta":684926,"kb_avail":89589840,"statfs":{"total":95995035648,"available":91739996160,"internally_reserved":0,"allocated":3553665024,"data_stored":3553013893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":701364678},"hb_peers":[0,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:18:20 2024","interfaces":[{"interface":"back","average":{"1min":1.51,"5min":1.135,"15min":2.1230000000000002},"min":{"1min":0.26700000000000002,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":9.2750000000000004,"5min":9.2750000000000004,"15min":74.551000000000002},"last":1.383},{"interface":"front","average":{"1min":1.6839999999999999,"5min":1.345,"15min":2.3210000000000002},"min":{"1min":0.26800000000000002,"5min":0.22700000000000001,"15min":0.20699999999999999},"max":{"1min":9.9969999999999999,"5min":16.619,"15min":74.778999999999996},"last":1.5740000000000001}]},{"osd":1,"last update":"Tue Sep 17 07:16:11 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":1.2070000000000001,"15min":1.4850000000000001},"min":{"1min":0.252,"5min":0.24399999999999999,"15min":0.23000000000000001},"max":{"1min":1.736,"5min":18.661000000000001,"15min":54.914999999999999},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.71499999999999997,"5min":1.3,"15min":1.444},"min":{"1min":0.246,"5min":0.23300000000000001,"15min":0.22800000000000001},"max":{"1min":3.7770000000000001,"5min":21.931999999999999,"15min":53.725000000000001},"last":0.443}]},{"osd":2,"last update":"Tue Sep 17 07:18:20 2024","interfaces":[{"interface":"back","average":{"1min":1.5880000000000001,"5min":1.262,"15min":1.1739999999999999},"min":{"1min":0.30599999999999999,"5min":0.26700000000000002,"15min":0.23799999999999999},"max":{"1min":8.4710000000000001,"5min":8.4710000000000001,"15min":49.651000000000003},"last":1.415},{"interface":"front","average":{"1min":1.353,"5min":1.1850000000000001,"15min":1.224},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.23599999999999999},"max":{"1min":6.1840000000000002,"5min":16.655999999999999,"15min":48.813000000000002},"last":1.5449999999999999}]},{"osd":3,"last update":"Tue Sep 17 07:18:20 2024","interfaces":[{"interface":"back","average":{"1min":1.141,"5min":0.94399999999999995,"15min":1.115},"min":{"1min":0.318,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":4.0990000000000002,"5min":4.0990000000000002,"15min":39.042999999999999},"last":1.595},{"interface":"front","average":{"1min":1.1559999999999999,"5min":1.212,"15min":1.234},"min":{"1min":0.35099999999999998,"5min":0.28999999999999998,"15min":0.253},"max":{"1min":9.0329999999999995,"5min":16.111000000000001,"15min":38.920000000000002},"last":0.35199999999999998}]},{"osd":5,"last update":"Tue Sep 17 07:17:35 2024","interfaces":[{"interface":"back","average":{"1min":2.395,"5min":1.4830000000000001,"15min":1.024},"min":{"1min":0.34000000000000002,"5min":0.28399999999999997,"15min":0.20799999999999999},"max":{"1min":5.4500000000000002,"5min":7.1909999999999998,"15min":11.486000000000001},"last":1.516},{"interface":"front","average":{"1min":2.1160000000000001,"5min":1.306,"15min":1.0149999999999999},"min":{"1min":0.48799999999999999,"5min":0.27800000000000002,"15min":0.26100000000000001},"max":{"1min":4.9610000000000003,"5min":5.9870000000000001,"15min":11.781000000000001},"last":2.9129999999999998}]},{"osd":6,"last update":"Tue Sep 17 07:18:20 2024","interfaces":[{"interface":"back","average":{"1min":1.871,"5min":1.4430000000000001,"15min":1.1060000000000001},"min":{"1min":0.28999999999999998,"5min":0.24299999999999999,"15min":0.218},"max":{"1min":8.9619999999999997,"5min":8.9619999999999997,"15min":47.923999999999999},"last":7.3360000000000003},{"interface":"front","average":{"1min":1.55,"5min":1.5940000000000001,"15min":1.1890000000000001},"min":{"1min":0.38900000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":4.899,"5min":10.051,"15min":47.600999999999999},"last":3.9100000000000001}]},{"osd":7,"last update":"Tue Sep 17 07:17:31 2024","interfaces":[{"interface":"back","average":{"1min":2.6480000000000001,"5min":1.5529999999999999,"15min":1.173},"min":{"1min":0.318,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":7.1200000000000001,"5min":7.1550000000000002,"15min":48.719000000000001},"last":1.617},{"interface":"front","average":{"1min":1.7110000000000001,"5min":1.6970000000000001,"15min":1.2010000000000001},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.28499999999999998},"max":{"1min":9.0030000000000001,"5min":9.0030000000000001,"15min":50.527000000000001},"last":7.2910000000000004}]}]},{"osd":3,"up_from":0,"seq":115964117208,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":18,"seq":77309411550,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3475628,"kb_used_data":2696356,"kb_used_omap":1,"kb_used_meta":779262,"kb_avail":90269524,"statfs":{"total":95995035648,"available":92435992576,"internally_reserved":0,"allocated":2761068544,"data_stored":2760422329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":797964742},"hb_peers":[0,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:17:50 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":2.8759999999999999,"15min":1.905},"min":{"1min":0.158,"5min":0.158,"15min":0.158},"max":{"1min":4.1829999999999998,"5min":131.09800000000001,"15min":131.09800000000001},"last":0.307},{"interface":"front","average":{"1min":0.49199999999999999,"5min":2.79,"15min":2.0299999999999998},"min":{"1min":0.223,"5min":0.185,"15min":0.185},"max":{"1min":3.5830000000000002,"5min":131.24700000000001,"15min":131.24700000000001},"last":0.36399999999999999}]},{"osd":1,"last update":"Tue Sep 17 07:15:43 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":1.7949999999999999,"15min":1.623},"min":{"1min":0.23699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":1.6459999999999999,"5min":47.101999999999997,"15min":79.659000000000006},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.55000000000000004,"5min":1.6990000000000001,"15min":1.5600000000000001},"min":{"1min":0.22700000000000001,"5min":0.20899999999999999,"15min":0.20200000000000001},"max":{"1min":1.8049999999999999,"5min":45.831000000000003,"15min":80.155000000000001},"last":0.41399999999999998}]},{"osd":3,"last update":"Tue Sep 17 07:18:05 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.56899999999999995,"15min":0.97899999999999998},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20100000000000001},"max":{"1min":2.2160000000000002,"5min":4.6449999999999996,"15min":23.43},"last":0.23200000000000001},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.439,"15min":1.036},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":2.121,"5min":2.121,"15min":23.408000000000001},"last":0.28000000000000003}]},{"osd":4,"last update":"Tue Sep 17 07:17:27 2024","interfaces":[{"interface":"back","average":{"1min":1.212,"5min":0.94299999999999995,"15min":0.94399999999999995},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":5.8099999999999996,"5min":6.8449999999999998,"15min":6.8449999999999998},"last":0.29299999999999998},{"interface":"front","average":{"1min":1.2030000000000001,"5min":1.0669999999999999,"15min":0.97799999999999998},"min":{"1min":0.32300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":5.7999999999999998,"5min":6.8609999999999998,"15min":6.8609999999999998},"last":0.47799999999999998}]},{"osd":5,"last update":"Tue Sep 17 07:17:50 2024","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":1.2669999999999999,"15min":1.159},"min":{"1min":0.29499999999999998,"5min":0.26900000000000002,"15min":0.249},"max":{"1min":5.0919999999999996,"5min":15.276999999999999,"15min":15.276999999999999},"last":0.94899999999999995},{"interface":"front","average":{"1min":1.177,"5min":1.1279999999999999,"15min":1.0209999999999999},"min":{"1min":0.309,"5min":0.28000000000000003,"15min":0.27600000000000002},"max":{"1min":3.29,"5min":15.561999999999999,"15min":15.561999999999999},"last":0.32600000000000001}]},{"osd":6,"last update":"Tue Sep 17 07:18:05 2024","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.163,"15min":1.458},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":6.5609999999999999,"5min":20.494,"15min":101.587},"last":1.014},{"interface":"front","average":{"1min":1.643,"5min":1.2410000000000001,"15min":1.5289999999999999},"min":{"1min":0.34100000000000003,"5min":0.32300000000000001,"15min":0.28299999999999997},"max":{"1min":5.5019999999999998,"5min":26.922000000000001,"15min":100.696},"last":0.55200000000000005}]},{"osd":7,"last update":"Tue Sep 17 07:17:43 2024","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":2.2040000000000002,"15min":1.3720000000000001},"min":{"1min":0.32700000000000001,"5min":0.32000000000000001,"15min":0.316},"max":{"1min":4.2839999999999998,"5min":94.269999999999996,"15min":94.269999999999996},"last":1.105},{"interface":"front","average":{"1min":0.97599999999999998,"5min":2.2229999999999999,"15min":1.4279999999999999},"min":{"1min":0.373,"5min":0.34300000000000003,"15min":0.32900000000000001},"max":{"1min":2.5750000000000002,"5min":94.634,"15min":94.634},"last":0.41999999999999998}]}]},{"osd":1,"up_from":13,"seq":55834575049,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":541424,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":540542,"kb_avail":93203728,"statfs":{"total":95995035648,"available":95440617472,"internally_reserved":0,"allocated":884736,"data_stored":439467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":553515462},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:16:00 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.73699999999999999,"15min":1.139},"min":{"1min":0.183,"5min":0.155,"15min":0.155},"max":{"1min":2.6070000000000002,"5min":4.5780000000000003,"15min":22.521000000000001},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.67500000000000004,"15min":1.0369999999999999},"min":{"1min":0.247,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.198,"5min":4.335,"15min":22.471},"last":0.377}]},{"osd":2,"last update":"Tue Sep 17 07:15:44 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.91000000000000003,"15min":1.024},"min":{"1min":0.21299999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":2.6360000000000001,"5min":6.1920000000000002,"15min":11.055999999999999},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.73899999999999999,"15min":0.94799999999999995},"min":{"1min":0.22900000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":1.3160000000000001,"5min":4.1559999999999997,"15min":9.3160000000000007},"last":0.30599999999999999}]},{"osd":3,"last update":"Tue Sep 17 07:16:00 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.67800000000000005,"15min":0.94999999999999996},"min":{"1min":0.27400000000000002,"5min":0.25900000000000001,"15min":0.219},"max":{"1min":2.4409999999999998,"5min":4.1269999999999998,"15min":8.5510000000000002},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.371,"5min":0.47399999999999998,"15min":0.98599999999999999},"min":{"1min":0.247,"5min":0.22500000000000001,"15min":0.19500000000000001},"max":{"1min":0.627,"5min":2.8199999999999998,"15min":19.602},"last":0.35299999999999998}]},{"osd":4,"last update":"Tue Sep 17 07:15:20 2024","interfaces":[{"interface":"back","average":{"1min":1.1339999999999999,"5min":1.4670000000000001,"15min":1.129},"min":{"1min":0.36199999999999999,"5min":0.314,"15min":0.25},"max":{"1min":2.5659999999999998,"5min":19.088999999999999,"15min":19.088999999999999},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.98999999999999999,"5min":1.2010000000000001,"15min":1.1759999999999999},"min":{"1min":0.36799999999999999,"5min":0.317,"15min":0.307},"max":{"1min":2.6459999999999999,"5min":18.959,"15min":27.097999999999999},"last":0.499}]},{"osd":5,"last update":"Tue Sep 17 07:15:44 2024","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.47,"15min":1.2110000000000001},"min":{"1min":0.34799999999999998,"5min":0.28899999999999998,"15min":0.27100000000000002},"max":{"1min":3.6869999999999998,"5min":6.6790000000000003,"15min":16.84},"last":0.51000000000000001},{"interface":"front","average":{"1min":1.1559999999999999,"5min":1.069,"15min":1.169},"min":{"1min":0.29299999999999998,"5min":0.29199999999999998,"15min":0.28899999999999998},"max":{"1min":6.2560000000000002,"5min":6.2560000000000002,"15min":16.949999999999999},"last":0.41299999999999998}]},{"osd":6,"last update":"Tue Sep 17 07:16:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":1.5149999999999999,"15min":1.274},"min":{"1min":0.33100000000000002,"5min":0.315,"15min":0.308},"max":{"1min":1.127,"5min":50.497,"15min":50.497},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.99099999999999999,"5min":1.6379999999999999,"15min":1.222},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.28899999999999998},"max":{"1min":3.593,"5min":50.793999999999997,"15min":50.793999999999997},"last":0.48499999999999999}]},{"osd":7,"last update":"Tue Sep 17 07:15:20 2024","interfaces":[{"interface":"back","average":{"1min":1.655,"5min":1.343,"15min":1.1919999999999999},"min":{"1min":0.435,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":3.6120000000000001,"5min":9.8179999999999996,"15min":9.8179999999999996},"last":0.441},{"interface":"front","average":{"1min":1.464,"5min":1.6319999999999999,"15min":1.2410000000000001},"min":{"1min":0.35099999999999998,"5min":0.317,"15min":0.317},"max":{"1min":4.1219999999999999,"5min":7.5839999999999996,"15min":11.007999999999999},"last":0.57999999999999996}]}]},{"osd":0,"up_from":8,"seq":34359738598,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4343360,"kb_used_data":3523832,"kb_used_omap":1,"kb_used_meta":819454,"kb_avail":89401792,"statfs":{"total":95995035648,"available":91547435008,"internally_reserved":0,"allocated":3608403968,"data_stored":3607738041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":839121350},"hb_peers":[2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 17 07:15:18 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.89700000000000002,"15min":0.89500000000000002},"min":{"1min":0.254,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":1.518,"5min":21.876000000000001,"15min":21.876000000000001},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.56399999999999995,"5min":1.2330000000000001,"15min":1.081},"min":{"1min":0.17899999999999999,"5min":0.158,"15min":0.158},"max":{"1min":2.2599999999999998,"5min":22.521999999999998,"15min":22.521999999999998},"last":0.33700000000000002}]},{"osd":2,"last update":"Tue Sep 17 07:17:47 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.47499999999999998,"15min":1.117},"min":{"1min":0.155,"5min":0.155,"15min":0.155},"max":{"1min":2.6520000000000001,"5min":2.6520000000000001,"15min":18.643999999999998},"last":0.156},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.51500000000000001,"15min":1.018},"min":{"1min":0.19600000000000001,"5min":0.189,"15min":0.189},"max":{"1min":3.1760000000000002,"5min":3.1760000000000002,"15min":17.972999999999999},"last":0.20399999999999999}]},{"osd":3,"last update":"Tue Sep 17 07:18:24 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.46700000000000003,"15min":1.018},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.21199999999999999},"max":{"1min":1.7030000000000001,"5min":3.2000000000000002,"15min":18.629000000000001},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.46800000000000003,"15min":1.0189999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.42699999999999999,"5min":3.1190000000000002,"15min":22.358000000000001},"last":0.317}]},{"osd":4,"last update":"Tue Sep 17 07:18:24 2024","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":0.98699999999999999,"15min":1.1479999999999999},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":5.3179999999999996,"5min":5.5990000000000002,"15min":19.146999999999998},"last":0.67000000000000004},{"interface":"front","average":{"1min":1.3,"5min":1.0580000000000001,"15min":1.1799999999999999},"min":{"1min":0.29499999999999998,"5min":0.26200000000000001,"15min":0.255},"max":{"1min":6.2329999999999997,"5min":6.2329999999999997,"15min":19.158999999999999},"last":0.29499999999999998}]},{"osd":5,"last update":"Tue Sep 17 07:17:55 2024","interfaces":[{"interface":"back","average":{"1min":1.0489999999999999,"5min":3.7370000000000001,"15min":2.121},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":2.8399999999999999,"5min":216.83699999999999,"15min":216.83699999999999},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.88700000000000001,"5min":3.5920000000000001,"15min":2.0910000000000002},"min":{"1min":0.33400000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":3.6789999999999998,"5min":216.816,"15min":216.816},"last":4.4050000000000002}]},{"osd":6,"last update":"Tue Sep 17 07:18:24 2024","interfaces":[{"interface":"back","average":{"1min":1.103,"5min":2.2970000000000002,"15min":1.5620000000000001},"min":{"1min":0.33900000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":3.3889999999999998,"5min":147.13200000000001,"15min":147.13200000000001},"last":1.149},{"interface":"front","average":{"1min":0.76300000000000001,"5min":2.2549999999999999,"15min":1.577},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":2.1309999999999998,"5min":147.172,"15min":147.172},"last":0.36099999999999999}]},{"osd":7,"last update":"Tue Sep 17 07:18:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2070000000000001,"5min":1.236,"15min":1.4390000000000001},"min":{"1min":0.40799999999999997,"5min":0.314,"15min":0.29599999999999999},"max":{"1min":4.3490000000000002,"5min":4.484,"15min":46.183},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.96999999999999997,"5min":1.1579999999999999,"15min":1.3879999999999999},"min":{"1min":0.38800000000000001,"5min":0.32600000000000001,"15min":0.32000000000000001},"max":{"1min":3.528,"5min":5.2149999999999999,"15min":45.914999999999999},"last":2.1840000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3601666048,"data_stored":3601661976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2759524352,"data_stored":2759524352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3494776832,"data_stored":3494772760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1638072320,"data_stored":1638072320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2927562752,"data_stored":2927558680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2023292928,"data_stored":2023292928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T07:18:28.137 INFO:tasks.thrashosds.thrasher:Setting 10.17 to [7, 6, 3, 4, 2, 5] 2024-09-17T07:18:28.137 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '10.17', '7', '6', '3', '4', '2', '5'] 2024-09-17T07:18:28.137 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd pg-upmap-items 10.17 7 6 3 4 2 5 2024-09-17T07:18:28.167 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:18:28.205 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:28.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:28.206 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:18:28.306 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:18:28.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:28.346 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:28.346 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:18:28.447 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:18:28.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:28.488 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:28.489 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:18:28.590 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:18:28.625 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:28.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:28.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:18:28.689 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:28 smithi100 ceph-mon[29092]: 10.f scrub starts 2024-09-17T07:18:28.689 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:28 smithi100 ceph-mon[39462]: 10.f scrub starts 2024-09-17T07:18:28.728 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:18:28.763 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:28.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:28.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:18:28.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:28 smithi135 ceph-mon[33238]: 10.f scrub starts 2024-09-17T07:18:28.865 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:18:28.903 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:28.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:28.904 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:18:29.005 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:18:29.034 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:29.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:29.035 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:18:29.137 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:18:29.163 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:29.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:29.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:18:29.265 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:18:29.294 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:18:29 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:18:29.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:29.305 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:29.305 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:18:29.406 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:18:29.435 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:29.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:29.437 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:18:29.453 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:18:29.538 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:18:29.581 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:29.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:29.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:18:29.683 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:18:29.728 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:29.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:29.729 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:18:29.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:29 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:18:29.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:29 smithi135 ceph-mon[33238]: pgmap v742: 158 pgs: 1 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+scrubbing, 5 active+clean+remapped, 36 active+undersized, 1 active+remapped+backfill_toofull, 28 active+undersized+degraded, 85 active+clean; 5.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.41k op/s; 15811/249654 objects degraded (6.333%); 30959/249654 objects misplaced (12.401%); 2.4 MiB/s, 39 objects/s recovering 2024-09-17T07:18:29.830 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:18:29.846 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:18:29.851 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:29 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:18:29.851 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:29 smithi100 ceph-mon[29092]: pgmap v742: 158 pgs: 1 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+scrubbing, 5 active+clean+remapped, 36 active+undersized, 1 active+remapped+backfill_toofull, 28 active+undersized+degraded, 85 active+clean; 5.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.41k op/s; 15811/249654 objects degraded (6.333%); 30959/249654 objects misplaced (12.401%); 2.4 MiB/s, 39 objects/s recovering 2024-09-17T07:18:29.852 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:29 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:18:29.852 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:29 smithi100 ceph-mon[39462]: pgmap v742: 158 pgs: 1 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+scrubbing, 5 active+clean+remapped, 36 active+undersized, 1 active+remapped+backfill_toofull, 28 active+undersized+degraded, 85 active+clean; 5.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.41k op/s; 15811/249654 objects degraded (6.333%); 30959/249654 objects misplaced (12.401%); 2.4 MiB/s, 39 objects/s recovering 2024-09-17T07:18:29.864 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:29.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:29.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:18:29.966 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:18:30.004 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:30.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:30.005 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:18:30.106 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:18:30.141 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:30.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:30.142 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:18:30.243 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:18:30.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:30.282 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:30.282 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:18:30.383 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:18:30.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:30.419 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:30.419 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:18:30.520 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:18:30.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:30.564 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:30.565 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:18:30.565 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-17T07:18:30.565 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-17T07:18:30.565 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-17T07:18:30.565 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-17T07:18:30.665 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:18:30.704 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:30.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:30.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:18:30.718 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-17T07:18:30.752 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:30 smithi100 ceph-mon[29092]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-17T07:18:30.752 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:30 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 15811/249654 objects degraded (6.333%), 29 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-09-17T07:18:30.753 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:30 smithi100 ceph-mon[39462]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-17T07:18:30.753 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:30 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 15811/249654 objects degraded (6.333%), 29 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-09-17T07:18:30.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:30 smithi135 ceph-mon[33238]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-17T07:18:30.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:30 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 15811/249654 objects degraded (6.333%), 29 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-09-17T07:18:30.807 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:18:30.816 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:18:30.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:30.864 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:30.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:18:30.966 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:18:30.996 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:30.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:30.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:18:31.098 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:18:31.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:31.126 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:31.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:18:31.227 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:18:31.259 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:31.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:31.260 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:18:31.361 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:18:31.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:31.396 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:31.396 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:18:31.497 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:18:31.535 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:31.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:31.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:18:31.637 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:18:31.675 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:31.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:31.676 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:18:31.777 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:18:31.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:31 smithi135 ceph-mon[33238]: 10.d scrub starts 2024-09-17T07:18:31.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:31 smithi135 ceph-mon[33238]: pgmap v743: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfilling, 1 active+clean+scrubbing, 5 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 85 active+clean; 5.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1021 B/s rd, 0 B/s wr, 1.38k op/s; 15366/246204 objects degraded (6.241%); 29269/246204 objects misplaced (11.888%); 7.3 MiB/s, 116 objects/s recovering 2024-09-17T07:18:31.815 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:31.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:31.817 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:18:31.849 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:31 smithi100 ceph-mon[29092]: 10.d scrub starts 2024-09-17T07:18:31.849 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:31 smithi100 ceph-mon[29092]: pgmap v743: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfilling, 1 active+clean+scrubbing, 5 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 85 active+clean; 5.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1021 B/s rd, 0 B/s wr, 1.38k op/s; 15366/246204 objects degraded (6.241%); 29269/246204 objects misplaced (11.888%); 7.3 MiB/s, 116 objects/s recovering 2024-09-17T07:18:31.849 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:31 smithi100 ceph-mon[39462]: 10.d scrub starts 2024-09-17T07:18:31.849 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:31 smithi100 ceph-mon[39462]: pgmap v743: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfilling, 1 active+clean+scrubbing, 5 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 85 active+clean; 5.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1021 B/s rd, 0 B/s wr, 1.38k op/s; 15366/246204 objects degraded (6.241%); 29269/246204 objects misplaced (11.888%); 7.3 MiB/s, 116 objects/s recovering 2024-09-17T07:18:31.882 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-17T07:18:31.917 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:18:31.957 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:31.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:31.958 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:18:32.059 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:18:32.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:32.096 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:32.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:18:32.197 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:18:32.238 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:32.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:32.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:18:32.340 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:18:32.370 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:32.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:32.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:18:32.472 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:18:32.502 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:32.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:32.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:18:32.603 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:18:32.638 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:32.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:32.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:18:32.740 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:18:32.778 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:32.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:32.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:18:32.880 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:18:32.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:32.914 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:32.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:18:33.015 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:18:33.042 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:33.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:33.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:18:33.083 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:18:33.144 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:18:33.174 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:33.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:33.175 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:18:33.276 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:18:33.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:33.307 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:33.307 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:18:33.408 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:18:33.449 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:33.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:33.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:18:33.504 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:33 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:18:33.505 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:33 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:18:33.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:33 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:18:33.550 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:18:33.583 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:33.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:33.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:18:33.684 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:18:33.712 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:33.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:33.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:18:33.814 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:18:33.846 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:33.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:33.847 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:18:33.893 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:18:33.893 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:18:33.894 INFO:teuthology.orchestra.run.smithi100.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-17T07:18:33.923 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:18:33.923 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:18:33.946 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:18:33.946 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:18:33.948 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:18:33.980 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:33.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:33.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:18:33.983 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:18:33.983 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:18:34.019 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:18:34.020 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:18:34.051 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:18:34.051 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:18:34.080 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:18:34.080 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:18:34.081 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:18:34.113 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:34.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:34.114 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:18:34.215 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:18:34.241 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:18:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=sqlstore.transactions t=2024-09-17T07:18:33.89061249Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-17T07:18:34.252 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:34.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:34.253 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:18:34.353 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:18:34.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:34 smithi100 ceph-mon[29092]: pgmap v744: 158 pgs: 1 active+clean+scrubbing, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfilling, 5 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 85 active+clean; 4.8 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.38k op/s; 15143/238251 objects degraded (6.356%); 26443/238251 objects misplaced (11.099%); 9.7 MiB/s, 154 objects/s recovering 2024-09-17T07:18:34.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:34 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1787940830' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:18:34.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:34 smithi100 ceph-mon[39462]: pgmap v744: 158 pgs: 1 active+clean+scrubbing, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfilling, 5 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 85 active+clean; 4.8 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.38k op/s; 15143/238251 objects degraded (6.356%); 26443/238251 objects misplaced (11.099%); 9.7 MiB/s, 154 objects/s recovering 2024-09-17T07:18:34.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:34 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1787940830' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:18:34.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:34.397 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:34.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:18:34.498 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:18:34.536 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:34.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:34.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:18:34.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:34 smithi135 ceph-mon[33238]: pgmap v744: 158 pgs: 1 active+clean+scrubbing, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfilling, 5 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 85 active+clean; 4.8 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.38k op/s; 15143/238251 objects degraded (6.356%); 26443/238251 objects misplaced (11.099%); 9.7 MiB/s, 154 objects/s recovering 2024-09-17T07:18:34.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:34 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1787940830' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:18:34.638 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:18:34.678 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:34.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:34.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:18:34.780 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:18:34.814 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:34.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:34.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:18:34.916 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:18:34.947 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:34.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:34.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:18:35.049 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:18:35.092 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:35.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:35.093 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:18:35.194 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:18:35.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:35.231 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:35.232 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:18:35.332 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:18:35.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:35.362 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:35.363 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:18:35.464 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:18:35.492 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:35.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:35.493 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:18:35.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:35 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:18:35.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:35 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:18:35.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:35 smithi135 ceph-mon[33238]: osdmap e347: 8 total, 7 up, 7 in 2024-09-17T07:18:35.564 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:35 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:18:35.564 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:35 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:18:35.565 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:35 smithi100 ceph-mon[29092]: osdmap e347: 8 total, 7 up, 7 in 2024-09-17T07:18:35.565 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:35 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:18:35.565 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:35 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:18:35.566 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:35 smithi100 ceph-mon[39462]: osdmap e347: 8 total, 7 up, 7 in 2024-09-17T07:18:35.594 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:18:35.680 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:18:35.681 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:18:35.681 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:18:35.720 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:35.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:35.721 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:18:35.822 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:18:35.849 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:35.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:35.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:18:35.951 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:18:35.992 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:35.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:35.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:18:36.094 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:18:36.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:36.130 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:36.131 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:18:36.231 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:18:36.243 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:18:35 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:18:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:18:36.262 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:36.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:36.263 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:18:36.364 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:18:36.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:36.401 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:36.401 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:18:36.502 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:18:36.510 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:36 smithi100 ceph-mon[29092]: 1.0 scrub starts 2024-09-17T07:18:36.510 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:36 smithi100 ceph-mon[29092]: 10.a scrub starts 2024-09-17T07:18:36.510 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:36 smithi100 ceph-mon[29092]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-17T07:18:36.510 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:36 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 15143/238251 objects degraded (6.356%), 29 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-09-17T07:18:36.510 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:36 smithi100 ceph-mon[29092]: osdmap e348: 8 total, 7 up, 7 in 2024-09-17T07:18:36.510 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:36 smithi100 ceph-mon[29092]: pgmap v747: 158 pgs: 1 active+clean+scrubbing, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfilling, 5 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 85 active+clean; 4.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.79k op/s; 14047/224994 objects degraded (6.243%); 24532/224994 objects misplaced (10.903%); 22 MiB/s, 347 objects/s recovering 2024-09-17T07:18:36.510 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:36 smithi100 ceph-mon[39462]: 1.0 scrub starts 2024-09-17T07:18:36.510 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:36 smithi100 ceph-mon[39462]: 10.a scrub starts 2024-09-17T07:18:36.510 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:36 smithi100 ceph-mon[39462]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-17T07:18:36.510 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:36 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 15143/238251 objects degraded (6.356%), 29 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-09-17T07:18:36.511 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:36 smithi100 ceph-mon[39462]: osdmap e348: 8 total, 7 up, 7 in 2024-09-17T07:18:36.511 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:36 smithi100 ceph-mon[39462]: pgmap v747: 158 pgs: 1 active+clean+scrubbing, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfilling, 5 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 85 active+clean; 4.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.79k op/s; 14047/224994 objects degraded (6.243%); 24532/224994 objects misplaced (10.903%); 22 MiB/s, 347 objects/s recovering 2024-09-17T07:18:36.532 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:36.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:36.533 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:18:36.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:36 smithi135 ceph-mon[33238]: 1.0 scrub starts 2024-09-17T07:18:36.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:36 smithi135 ceph-mon[33238]: 10.a scrub starts 2024-09-17T07:18:36.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:36 smithi135 ceph-mon[33238]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-17T07:18:36.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:36 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 15143/238251 objects degraded (6.356%), 29 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-09-17T07:18:36.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:36 smithi135 ceph-mon[33238]: osdmap e348: 8 total, 7 up, 7 in 2024-09-17T07:18:36.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:36 smithi135 ceph-mon[33238]: pgmap v747: 158 pgs: 1 active+clean+scrubbing, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfilling, 5 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 85 active+clean; 4.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.79k op/s; 14047/224994 objects degraded (6.243%); 24532/224994 objects misplaced (10.903%); 22 MiB/s, 347 objects/s recovering 2024-09-17T07:18:36.634 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:18:36.677 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:36.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:36.678 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:18:36.779 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:18:36.823 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:36.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:36.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:18:36.926 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:18:36.960 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:36.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:36.961 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:18:37.062 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:18:37.098 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:37.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:37.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:18:37.200 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:18:37.234 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:37.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:37.235 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:18:37.272 INFO:teuthology.orchestra.run.smithi100.stderr:nodeep-scrub is unset 2024-09-17T07:18:37.335 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:18:37.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:37.364 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:37.364 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:18:37.465 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:18:37.495 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:37.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:37.496 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:18:37.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:37 smithi135 ceph-mon[33238]: 4.5 scrub starts 2024-09-17T07:18:37.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:37 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1801598681' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:18:37.580 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:37 smithi100 ceph-mon[29092]: 4.5 scrub starts 2024-09-17T07:18:37.580 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:37 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1801598681' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:18:37.580 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:37 smithi100 ceph-mon[39462]: 4.5 scrub starts 2024-09-17T07:18:37.580 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:37 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1801598681' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:18:37.597 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:18:37.627 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:37.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:37.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:18:37.682 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:18:37.729 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:18:37.770 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:37.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:37.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:18:37.872 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:18:37.913 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:37.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:37.915 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:18:38.016 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:18:38.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:38.053 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:38.054 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:18:38.154 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:18:38.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:38.185 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:38.185 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:18:38.286 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:18:38.306 INFO:teuthology.orchestra.run.smithi100.stderr:set 10.17 pg_upmap_items mapping to [7->6,3->4,2->5] 2024-09-17T07:18:38.327 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:38.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:38.328 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:18:38.429 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:18:38.465 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:38.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:38.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:18:38.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:38 smithi135 ceph-mon[33238]: 5.d scrub starts 2024-09-17T07:18:38.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:38 smithi135 ceph-mon[33238]: 10.7 scrub starts 2024-09-17T07:18:38.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:38 smithi135 ceph-mon[33238]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T07:18:38.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:38 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1801598681' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T07:18:38.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:38 smithi135 ceph-mon[33238]: osdmap e349: 8 total, 7 up, 7 in 2024-09-17T07:18:38.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:38 smithi135 ceph-mon[33238]: pgmap v749: 158 pgs: 1 active+clean+scrubbing, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfilling, 5 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 85 active+clean; 4.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.18k op/s; 14047/224994 objects degraded (6.243%); 24532/224994 objects misplaced (10.903%); 17 MiB/s, 269 objects/s recovering 2024-09-17T07:18:38.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:38 smithi135 ceph-mon[33238]: 6.15 scrub starts 2024-09-17T07:18:38.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:38 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/3455638736' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "10.17", "id": [7, 6, 3, 4, 2, 5]}]: dispatch 2024-09-17T07:18:38.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:38 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "10.17", "id": [7, 6, 3, 4, 2, 5]}]: dispatch 2024-09-17T07:18:38.567 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:18:38.595 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:38.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:38.596 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:18:38.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:38 smithi100 ceph-mon[29092]: 5.d scrub starts 2024-09-17T07:18:38.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:38 smithi100 ceph-mon[29092]: 10.7 scrub starts 2024-09-17T07:18:38.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:38 smithi100 ceph-mon[29092]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T07:18:38.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:38 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1801598681' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T07:18:38.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:38 smithi100 ceph-mon[29092]: osdmap e349: 8 total, 7 up, 7 in 2024-09-17T07:18:38.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:38 smithi100 ceph-mon[29092]: pgmap v749: 158 pgs: 1 active+clean+scrubbing, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfilling, 5 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 85 active+clean; 4.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.18k op/s; 14047/224994 objects degraded (6.243%); 24532/224994 objects misplaced (10.903%); 17 MiB/s, 269 objects/s recovering 2024-09-17T07:18:38.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:38 smithi100 ceph-mon[29092]: 6.15 scrub starts 2024-09-17T07:18:38.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:38 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/3455638736' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "10.17", "id": [7, 6, 3, 4, 2, 5]}]: dispatch 2024-09-17T07:18:38.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:38 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "10.17", "id": [7, 6, 3, 4, 2, 5]}]: dispatch 2024-09-17T07:18:38.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:38 smithi100 ceph-mon[39462]: 5.d scrub starts 2024-09-17T07:18:38.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:38 smithi100 ceph-mon[39462]: 10.7 scrub starts 2024-09-17T07:18:38.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:38 smithi100 ceph-mon[39462]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T07:18:38.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:38 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1801598681' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T07:18:38.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:38 smithi100 ceph-mon[39462]: osdmap e349: 8 total, 7 up, 7 in 2024-09-17T07:18:38.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:38 smithi100 ceph-mon[39462]: pgmap v749: 158 pgs: 1 active+clean+scrubbing, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfilling, 5 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 85 active+clean; 4.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.18k op/s; 14047/224994 objects degraded (6.243%); 24532/224994 objects misplaced (10.903%); 17 MiB/s, 269 objects/s recovering 2024-09-17T07:18:38.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:38 smithi100 ceph-mon[39462]: 6.15 scrub starts 2024-09-17T07:18:38.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:38 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/3455638736' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "10.17", "id": [7, 6, 3, 4, 2, 5]}]: dispatch 2024-09-17T07:18:38.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:38 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "10.17", "id": [7, 6, 3, 4, 2, 5]}]: dispatch 2024-09-17T07:18:38.697 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:18:38.733 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:38.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:38.734 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:18:38.835 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:18:38.870 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:38.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:38.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:18:38.972 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:18:39.012 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:39.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:39.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:18:39.113 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:18:39.149 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:39.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:39.151 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:18:39.252 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:18:39.281 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:39.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:39.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:18:39.383 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:18:39.415 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:39.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:39.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:18:39.486 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:39 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "10.17", "id": [7, 6, 3, 4, 2, 5]}]': finished 2024-09-17T07:18:39.486 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:39 smithi100 ceph-mon[29092]: osdmap e350: 8 total, 7 up, 7 in 2024-09-17T07:18:39.486 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:39 smithi100 ceph-mon[29092]: 6.5 scrub starts 2024-09-17T07:18:39.487 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:39 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "10.17", "id": [7, 6, 3, 4, 2, 5]}]': finished 2024-09-17T07:18:39.487 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:39 smithi100 ceph-mon[39462]: osdmap e350: 8 total, 7 up, 7 in 2024-09-17T07:18:39.487 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:39 smithi100 ceph-mon[39462]: 6.5 scrub starts 2024-09-17T07:18:39.517 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:18:39.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:39 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "10.17", "id": [7, 6, 3, 4, 2, 5]}]': finished 2024-09-17T07:18:39.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:39 smithi135 ceph-mon[33238]: osdmap e350: 8 total, 7 up, 7 in 2024-09-17T07:18:39.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:39 smithi135 ceph-mon[33238]: 6.5 scrub starts 2024-09-17T07:18:39.544 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:18:39 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:18:39.550 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:39.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:39.551 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:18:39.634 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-17T07:18:39.634 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:18:39.652 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:18:39.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:39.682 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:39.683 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:18:39.784 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:18:39.810 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:39.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:39.811 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:18:39.912 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:18:39.937 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:39.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:39.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:18:40.038 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:18:40.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:40.060 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:40.060 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:18:40.161 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:18:40.184 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:40.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:40.185 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:18:40.286 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:18:40.315 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:40.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:40.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:18:40.417 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:18:40.445 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:40.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:40.446 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:18:40.547 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:18:40.582 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:40.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:40.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:18:40.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:40 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:18:40.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:40 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:18:40.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:40 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:18:40.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:40 smithi100 ceph-mon[29092]: osdmap e351: 8 total, 7 up, 7 in 2024-09-17T07:18:40.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:40 smithi100 ceph-mon[29092]: pgmap v752: 158 pgs: 1 active+clean+scrubbing, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfilling, 5 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 85 active+clean; 4.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.44k op/s; 13172/207381 objects degraded (6.352%); 21767/207381 objects misplaced (10.496%); 10 MiB/s, 161 objects/s recovering 2024-09-17T07:18:40.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:40 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 14047/224994 objects degraded (6.243%), 29 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-09-17T07:18:40.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:40 smithi100 ceph-mon[29092]: osdmap e352: 8 total, 7 up, 7 in 2024-09-17T07:18:40.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:40 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:18:40.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:40 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:18:40.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:40 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:18:40.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:40 smithi100 ceph-mon[39462]: osdmap e351: 8 total, 7 up, 7 in 2024-09-17T07:18:40.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:40 smithi100 ceph-mon[39462]: pgmap v752: 158 pgs: 1 active+clean+scrubbing, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfilling, 5 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 85 active+clean; 4.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.44k op/s; 13172/207381 objects degraded (6.352%); 21767/207381 objects misplaced (10.496%); 10 MiB/s, 161 objects/s recovering 2024-09-17T07:18:40.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:40 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 14047/224994 objects degraded (6.243%), 29 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-09-17T07:18:40.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:40 smithi100 ceph-mon[39462]: osdmap e352: 8 total, 7 up, 7 in 2024-09-17T07:18:40.683 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:18:40.722 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:40.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:40.723 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:18:40.796 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:40 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:18:40.797 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:40 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:18:40.797 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:40 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:18:40.797 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:40 smithi135 ceph-mon[33238]: osdmap e351: 8 total, 7 up, 7 in 2024-09-17T07:18:40.797 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:40 smithi135 ceph-mon[33238]: pgmap v752: 158 pgs: 1 active+clean+scrubbing, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfilling, 5 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 85 active+clean; 4.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.44k op/s; 13172/207381 objects degraded (6.352%); 21767/207381 objects misplaced (10.496%); 10 MiB/s, 161 objects/s recovering 2024-09-17T07:18:40.797 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:40 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 14047/224994 objects degraded (6.243%), 29 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-09-17T07:18:40.797 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:40 smithi135 ceph-mon[33238]: osdmap e352: 8 total, 7 up, 7 in 2024-09-17T07:18:40.823 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:18:40.861 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:40.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:40.862 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:18:40.963 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:18:40.996 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:40.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:40.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:18:41.098 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:18:41.135 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:41.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:41.136 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:18:41.237 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:18:41.277 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:41.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:41.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:18:41.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:41 smithi100 ceph-mon[29092]: 2.1a deep-scrub starts 2024-09-17T07:18:41.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:41 smithi100 ceph-mon[29092]: 4.6 scrub starts 2024-09-17T07:18:41.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:41 smithi100 ceph-mon[29092]: 4.6 scrub ok 2024-09-17T07:18:41.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:41 smithi100 ceph-mon[39462]: 2.1a deep-scrub starts 2024-09-17T07:18:41.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:41 smithi100 ceph-mon[39462]: 4.6 scrub starts 2024-09-17T07:18:41.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:41 smithi100 ceph-mon[39462]: 4.6 scrub ok 2024-09-17T07:18:41.379 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:18:41.405 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:41.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:41.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:18:41.507 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:18:41.537 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:41.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:41.538 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:18:41.639 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:18:41.663 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:41.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:41.664 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:18:41.757 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:18:41.765 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:18:41.781 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-17T07:18:41.789 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:41.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:41.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:18:41.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:41 smithi135 ceph-mon[33238]: 2.1a deep-scrub starts 2024-09-17T07:18:41.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:41 smithi135 ceph-mon[33238]: 4.6 scrub starts 2024-09-17T07:18:41.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:41 smithi135 ceph-mon[33238]: 4.6 scrub ok 2024-09-17T07:18:41.892 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:18:41.914 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:41.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:41.915 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:18:42.016 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:18:42.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:42.038 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:42.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:18:42.139 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:18:42.160 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:42.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:42.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:18:42.262 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:18:42.283 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:42.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:42.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:18:42.385 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:18:42.425 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:42.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:42.426 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:18:42.527 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:18:42.566 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:42.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:42.567 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:18:42.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:42 smithi100 ceph-mon[29092]: 6.c scrub starts 2024-09-17T07:18:42.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:42 smithi100 ceph-mon[29092]: 6.c scrub ok 2024-09-17T07:18:42.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:42 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:18:42.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:42 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:18:42.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:42 smithi100 ceph-mon[29092]: osdmap e353: 8 total, 7 up, 7 in 2024-09-17T07:18:42.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:42 smithi100 ceph-mon[29092]: pgmap v755: 158 pgs: 1 active+clean+scrubbing, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfilling, 5 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 85 active+clean; 4.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.69k op/s; 12698/204678 objects degraded (6.204%); 21307/204678 objects misplaced (10.410%); 10 MiB/s, 164 objects/s recovering 2024-09-17T07:18:42.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:42 smithi100 ceph-mon[29092]: 6.a deep-scrub starts 2024-09-17T07:18:42.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:42 smithi100 ceph-mon[29092]: 6.a deep-scrub ok 2024-09-17T07:18:42.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:42 smithi100 ceph-mon[29092]: 10.5 scrub starts 2024-09-17T07:18:42.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:42 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:18:42.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:42 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:18:42.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:42 smithi100 ceph-mon[29092]: 6.f scrub starts 2024-09-17T07:18:42.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:42 smithi100 ceph-mon[29092]: 6.f scrub ok 2024-09-17T07:18:42.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:42 smithi100 ceph-mon[39462]: 6.c scrub starts 2024-09-17T07:18:42.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:42 smithi100 ceph-mon[39462]: 6.c scrub ok 2024-09-17T07:18:42.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:42 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:18:42.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:42 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:18:42.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:42 smithi100 ceph-mon[39462]: osdmap e353: 8 total, 7 up, 7 in 2024-09-17T07:18:42.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:42 smithi100 ceph-mon[39462]: pgmap v755: 158 pgs: 1 active+clean+scrubbing, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfilling, 5 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 85 active+clean; 4.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.69k op/s; 12698/204678 objects degraded (6.204%); 21307/204678 objects misplaced (10.410%); 10 MiB/s, 164 objects/s recovering 2024-09-17T07:18:42.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:42 smithi100 ceph-mon[39462]: 6.a deep-scrub starts 2024-09-17T07:18:42.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:42 smithi100 ceph-mon[39462]: 6.a deep-scrub ok 2024-09-17T07:18:42.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:42 smithi100 ceph-mon[39462]: 10.5 scrub starts 2024-09-17T07:18:42.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:42 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:18:42.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:42 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:18:42.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:42 smithi100 ceph-mon[39462]: 6.f scrub starts 2024-09-17T07:18:42.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:42 smithi100 ceph-mon[39462]: 6.f scrub ok 2024-09-17T07:18:42.668 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:18:42.709 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:42.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:42.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:18:42.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:42 smithi135 ceph-mon[33238]: 6.c scrub starts 2024-09-17T07:18:42.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:42 smithi135 ceph-mon[33238]: 6.c scrub ok 2024-09-17T07:18:42.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:42 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:18:42.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:42 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:18:42.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:42 smithi135 ceph-mon[33238]: osdmap e353: 8 total, 7 up, 7 in 2024-09-17T07:18:42.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:42 smithi135 ceph-mon[33238]: pgmap v755: 158 pgs: 1 active+clean+scrubbing, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfilling, 5 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 85 active+clean; 4.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.69k op/s; 12698/204678 objects degraded (6.204%); 21307/204678 objects misplaced (10.410%); 10 MiB/s, 164 objects/s recovering 2024-09-17T07:18:42.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:42 smithi135 ceph-mon[33238]: 6.a deep-scrub starts 2024-09-17T07:18:42.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:42 smithi135 ceph-mon[33238]: 6.a deep-scrub ok 2024-09-17T07:18:42.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:42 smithi135 ceph-mon[33238]: 10.5 scrub starts 2024-09-17T07:18:42.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:42 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:18:42.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:42 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:18:42.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:42 smithi135 ceph-mon[33238]: 6.f scrub starts 2024-09-17T07:18:42.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:42 smithi135 ceph-mon[33238]: 6.f scrub ok 2024-09-17T07:18:42.811 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:18:42.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:42.849 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:42.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:18:42.950 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:18:42.989 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:42.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:42.990 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:18:43.091 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:18:43.121 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:43.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:43.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:18:43.150 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-17T07:18:43.223 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:18:43.256 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:43.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:43.257 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:18:43.357 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:18:43.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:43.389 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:43.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:18:43.490 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:18:43.506 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:43 smithi100 ceph-mon[29092]: 10.5 scrub ok 2024-09-17T07:18:43.507 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:43 smithi100 ceph-mon[29092]: 2.17 deep-scrub starts 2024-09-17T07:18:43.507 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:43 smithi100 ceph-mon[29092]: 2.17 deep-scrub ok 2024-09-17T07:18:43.507 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:43 smithi100 ceph-mon[29092]: 5.6 deep-scrub starts 2024-09-17T07:18:43.507 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:43 smithi100 ceph-mon[29092]: 5.6 deep-scrub ok 2024-09-17T07:18:43.507 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:43 smithi100 ceph-mon[29092]: osdmap e354: 8 total, 7 up, 7 in 2024-09-17T07:18:43.507 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:43 smithi100 ceph-mon[29092]: 6.17 scrub starts 2024-09-17T07:18:43.507 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:43 smithi100 ceph-mon[29092]: 6.17 scrub ok 2024-09-17T07:18:43.508 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:43 smithi100 ceph-mon[39462]: 10.5 scrub ok 2024-09-17T07:18:43.508 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:43 smithi100 ceph-mon[39462]: 2.17 deep-scrub starts 2024-09-17T07:18:43.508 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:43 smithi100 ceph-mon[39462]: 2.17 deep-scrub ok 2024-09-17T07:18:43.508 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:43 smithi100 ceph-mon[39462]: 5.6 deep-scrub starts 2024-09-17T07:18:43.508 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:43 smithi100 ceph-mon[39462]: 5.6 deep-scrub ok 2024-09-17T07:18:43.508 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:43 smithi100 ceph-mon[39462]: osdmap e354: 8 total, 7 up, 7 in 2024-09-17T07:18:43.508 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:43 smithi100 ceph-mon[39462]: 6.17 scrub starts 2024-09-17T07:18:43.509 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:43 smithi100 ceph-mon[39462]: 6.17 scrub ok 2024-09-17T07:18:43.524 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:43.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:43.525 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:18:43.573 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:18:43.626 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:18:43.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:43.662 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:43.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:18:43.764 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:18:43.772 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:18:43.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:43 smithi135 ceph-mon[33238]: 10.5 scrub ok 2024-09-17T07:18:43.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:43 smithi135 ceph-mon[33238]: 2.17 deep-scrub starts 2024-09-17T07:18:43.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:43 smithi135 ceph-mon[33238]: 2.17 deep-scrub ok 2024-09-17T07:18:43.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:43 smithi135 ceph-mon[33238]: 5.6 deep-scrub starts 2024-09-17T07:18:43.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:43 smithi135 ceph-mon[33238]: 5.6 deep-scrub ok 2024-09-17T07:18:43.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:43 smithi135 ceph-mon[33238]: osdmap e354: 8 total, 7 up, 7 in 2024-09-17T07:18:43.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:43 smithi135 ceph-mon[33238]: 6.17 scrub starts 2024-09-17T07:18:43.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:43 smithi135 ceph-mon[33238]: 6.17 scrub ok 2024-09-17T07:18:43.801 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:43.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:43.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:18:43.903 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:18:43.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:43.946 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:43.946 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:18:44.047 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:18:44.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:44.073 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:44.074 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:18:44.174 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:18:44.212 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:44.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:44.213 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:18:44.314 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:18:44.341 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:44.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:44.343 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:18:44.444 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:18:44.468 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:44.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:44.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:18:44.570 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:18:44.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:44.609 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:44.610 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:18:44.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:44 smithi100 ceph-mon[39462]: pgmap v757: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfilling, 6 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 86 active+clean; 4.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 728 op/s; 12541/198591 objects degraded (6.315%); 19216/198591 objects misplaced (9.676%); 9.0 MiB/s, 144 objects/s recovering 2024-09-17T07:18:44.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:44 smithi100 ceph-mon[39462]: 10.8 scrub starts 2024-09-17T07:18:44.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:44 smithi100 ceph-mon[29092]: pgmap v757: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfilling, 6 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 86 active+clean; 4.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 728 op/s; 12541/198591 objects degraded (6.315%); 19216/198591 objects misplaced (9.676%); 9.0 MiB/s, 144 objects/s recovering 2024-09-17T07:18:44.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:44 smithi100 ceph-mon[29092]: 10.8 scrub starts 2024-09-17T07:18:44.710 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:18:44.748 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:44.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:44.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:18:44.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:44 smithi135 ceph-mon[33238]: pgmap v757: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfilling, 6 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 86 active+clean; 4.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 728 op/s; 12541/198591 objects degraded (6.315%); 19216/198591 objects misplaced (9.676%); 9.0 MiB/s, 144 objects/s recovering 2024-09-17T07:18:44.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:44 smithi135 ceph-mon[33238]: 10.8 scrub starts 2024-09-17T07:18:44.851 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:18:44.879 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:44.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:44.880 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:18:44.981 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:18:45.017 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:45.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:45.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:18:45.064 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7] out_osds: [3] dead_osds: [1] live_osds: [0, 2, 3, 4, 5, 6, 7] 2024-09-17T07:18:45.064 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T07:18:45.065 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_3 2024-09-17T07:18:45.065 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T07:18:45.119 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:18:45.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:45.160 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:45.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:18:45.261 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:18:45.303 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:45.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:45.304 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:18:45.405 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:18:45.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:45.442 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:45.442 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:18:45.543 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:18:45.571 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:45.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:45.572 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:18:45.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:45 smithi100 ceph-mon[29092]: 2.e scrub starts 2024-09-17T07:18:45.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:45 smithi100 ceph-mon[29092]: 2.e scrub ok 2024-09-17T07:18:45.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:45 smithi100 ceph-mon[29092]: 10.16 scrub starts 2024-09-17T07:18:45.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:45 smithi100 ceph-mon[29092]: 10.8 scrub ok 2024-09-17T07:18:45.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:45 smithi100 ceph-mon[29092]: 5.b scrub starts 2024-09-17T07:18:45.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:45 smithi100 ceph-mon[29092]: 5.b scrub ok 2024-09-17T07:18:45.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:45 smithi100 ceph-mon[29092]: 10.16 scrub ok 2024-09-17T07:18:45.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:45 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 12541/198591 objects degraded (6.315%), 29 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-09-17T07:18:45.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:45 smithi100 ceph-mon[39462]: 2.e scrub starts 2024-09-17T07:18:45.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:45 smithi100 ceph-mon[39462]: 2.e scrub ok 2024-09-17T07:18:45.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:45 smithi100 ceph-mon[39462]: 10.16 scrub starts 2024-09-17T07:18:45.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:45 smithi100 ceph-mon[39462]: 10.8 scrub ok 2024-09-17T07:18:45.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:45 smithi100 ceph-mon[39462]: 5.b scrub starts 2024-09-17T07:18:45.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:45 smithi100 ceph-mon[39462]: 5.b scrub ok 2024-09-17T07:18:45.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:45 smithi100 ceph-mon[39462]: 10.16 scrub ok 2024-09-17T07:18:45.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:45 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 12541/198591 objects degraded (6.315%), 29 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-09-17T07:18:45.673 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:18:45.705 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:45.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:45.706 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:18:45.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:45 smithi135 ceph-mon[33238]: 2.e scrub starts 2024-09-17T07:18:45.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:45 smithi135 ceph-mon[33238]: 2.e scrub ok 2024-09-17T07:18:45.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:45 smithi135 ceph-mon[33238]: 10.16 scrub starts 2024-09-17T07:18:45.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:45 smithi135 ceph-mon[33238]: 10.8 scrub ok 2024-09-17T07:18:45.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:45 smithi135 ceph-mon[33238]: 5.b scrub starts 2024-09-17T07:18:45.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:45 smithi135 ceph-mon[33238]: 5.b scrub ok 2024-09-17T07:18:45.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:45 smithi135 ceph-mon[33238]: 10.16 scrub ok 2024-09-17T07:18:45.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:45 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 12541/198591 objects degraded (6.315%), 29 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-09-17T07:18:45.807 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:18:45.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:45.853 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:45.853 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:18:45.954 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:18:45.983 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:45.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:45.984 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:18:46.085 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:18:46.106 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:46.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:46.107 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:18:46.129 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:18:45 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:18:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:18:46.209 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:18:46.248 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:46.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:46.249 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:18:46.350 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:18:46.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:46.388 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:46.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:18:46.489 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:18:46.527 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:46.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:46.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:18:46.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:46 smithi100 ceph-mon[39462]: 2.1d scrub starts 2024-09-17T07:18:46.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:46 smithi100 ceph-mon[39462]: 2.1d scrub ok 2024-09-17T07:18:46.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:46 smithi100 ceph-mon[39462]: pgmap v758: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 7 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 86 active+clean; 3.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 11758/188229 objects degraded (6.247%); 17700/188229 objects misplaced (9.403%); 13 MiB/s, 203 objects/s recovering 2024-09-17T07:18:46.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:46 smithi100 ceph-mon[29092]: 2.1d scrub starts 2024-09-17T07:18:46.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:46 smithi100 ceph-mon[29092]: 2.1d scrub ok 2024-09-17T07:18:46.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:46 smithi100 ceph-mon[29092]: pgmap v758: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 7 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 86 active+clean; 3.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 11758/188229 objects degraded (6.247%); 17700/188229 objects misplaced (9.403%); 13 MiB/s, 203 objects/s recovering 2024-09-17T07:18:46.629 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:18:46.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:46.663 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:46.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:18:46.764 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:18:46.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:46 smithi135 ceph-mon[33238]: 2.1d scrub starts 2024-09-17T07:18:46.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:46 smithi135 ceph-mon[33238]: 2.1d scrub ok 2024-09-17T07:18:46.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:46 smithi135 ceph-mon[33238]: pgmap v758: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 7 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 86 active+clean; 3.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 11758/188229 objects degraded (6.247%); 17700/188229 objects misplaced (9.403%); 13 MiB/s, 203 objects/s recovering 2024-09-17T07:18:46.800 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:46.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:46.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:18:46.902 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:18:46.924 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:46.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:46.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:18:47.026 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:18:47.053 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:47.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:47.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:18:47.155 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:18:47.182 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:47.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:47.183 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:18:47.284 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:18:47.321 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:47.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:47.322 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:18:47.423 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:18:47.465 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:47.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:47.466 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:18:47.567 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:18:47.594 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:47.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:47.595 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:18:47.612 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:47 smithi100 ceph-mon[29092]: 2.1f scrub starts 2024-09-17T07:18:47.612 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:47 smithi100 ceph-mon[29092]: 10.9 scrub starts 2024-09-17T07:18:47.612 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:47 smithi100 ceph-mon[29092]: 2.1f scrub ok 2024-09-17T07:18:47.612 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:47 smithi100 ceph-mon[29092]: 10.9 scrub ok 2024-09-17T07:18:47.612 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:47 smithi100 ceph-mon[39462]: 2.1f scrub starts 2024-09-17T07:18:47.612 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:47 smithi100 ceph-mon[39462]: 10.9 scrub starts 2024-09-17T07:18:47.612 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:47 smithi100 ceph-mon[39462]: 2.1f scrub ok 2024-09-17T07:18:47.613 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:47 smithi100 ceph-mon[39462]: 10.9 scrub ok 2024-09-17T07:18:47.696 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:18:47.732 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:47.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:47.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:18:47.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:47 smithi135 ceph-mon[33238]: 2.1f scrub starts 2024-09-17T07:18:47.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:47 smithi135 ceph-mon[33238]: 10.9 scrub starts 2024-09-17T07:18:47.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:47 smithi135 ceph-mon[33238]: 2.1f scrub ok 2024-09-17T07:18:47.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:47 smithi135 ceph-mon[33238]: 10.9 scrub ok 2024-09-17T07:18:47.834 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:18:47.861 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:47.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:47.862 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:18:47.963 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:18:48.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:48.008 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:48.008 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:18:48.109 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:18:48.138 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:48.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:48.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:18:48.240 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:18:48.265 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:48.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:48.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:18:48.368 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:18:48.412 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:48.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:48.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:18:48.444 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:48 smithi100 ceph-mon[29092]: 6.3 scrub starts 2024-09-17T07:18:48.444 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:48 smithi100 ceph-mon[29092]: 6.3 scrub ok 2024-09-17T07:18:48.445 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:48 smithi100 ceph-mon[29092]: pgmap v759: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 7 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 86 active+clean; 3.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 723 B/s rd, 0 op/s; 11758/188229 objects degraded (6.247%); 17700/188229 objects misplaced (9.403%); 11 MiB/s, 172 objects/s recovering 2024-09-17T07:18:48.445 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:48 smithi100 ceph-mon[29092]: 6.16 scrub starts 2024-09-17T07:18:48.445 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:48 smithi100 ceph-mon[29092]: 6.16 scrub ok 2024-09-17T07:18:48.445 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:48 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:18:48.445 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:48 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:18:48.445 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:48 smithi100 ceph-mon[39462]: 6.3 scrub starts 2024-09-17T07:18:48.445 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:48 smithi100 ceph-mon[39462]: 6.3 scrub ok 2024-09-17T07:18:48.446 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:48 smithi100 ceph-mon[39462]: pgmap v759: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 7 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 86 active+clean; 3.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 723 B/s rd, 0 op/s; 11758/188229 objects degraded (6.247%); 17700/188229 objects misplaced (9.403%); 11 MiB/s, 172 objects/s recovering 2024-09-17T07:18:48.446 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:48 smithi100 ceph-mon[39462]: 6.16 scrub starts 2024-09-17T07:18:48.446 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:48 smithi100 ceph-mon[39462]: 6.16 scrub ok 2024-09-17T07:18:48.446 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:48 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:18:48.446 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:48 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:18:48.514 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:18:48.542 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:48.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:48.543 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:18:48.644 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:18:48.682 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:48.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:48.683 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:18:48.784 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:18:48.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:48 smithi135 ceph-mon[33238]: 6.3 scrub starts 2024-09-17T07:18:48.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:48 smithi135 ceph-mon[33238]: 6.3 scrub ok 2024-09-17T07:18:48.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:48 smithi135 ceph-mon[33238]: pgmap v759: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 7 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 86 active+clean; 3.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 723 B/s rd, 0 op/s; 11758/188229 objects degraded (6.247%); 17700/188229 objects misplaced (9.403%); 11 MiB/s, 172 objects/s recovering 2024-09-17T07:18:48.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:48 smithi135 ceph-mon[33238]: 6.16 scrub starts 2024-09-17T07:18:48.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:48 smithi135 ceph-mon[33238]: 6.16 scrub ok 2024-09-17T07:18:48.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:48 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:18:48.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:48 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:18:48.824 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:48.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:48.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:18:48.926 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:18:48.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:48.967 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:48.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:18:49.068 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:18:49.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:49.101 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:49.101 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:18:49.202 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:18:49.217 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:18:49.217 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:18:49.217 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:18:49.217 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:18:49.217 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:18:49.217 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:18:49.217 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:18:49.218 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:18:49.218 INFO:teuthology.orchestra.run.smithi100.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-17T07:18:49.218 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:18:49.218 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:18:49.218 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:18:49.218 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:18:49.218 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:18:49.218 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:18:49.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:49.247 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:49.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:18:49.348 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:18:49.373 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:18:49 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:18:49.395 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:49.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:49.396 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:18:49.496 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:18:49.536 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:49.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:49.537 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:18:49.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:49 smithi100 ceph-mon[29092]: 6.8 scrub starts 2024-09-17T07:18:49.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:49 smithi100 ceph-mon[29092]: 6.8 scrub ok 2024-09-17T07:18:49.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:49 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/4201119666' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:18:49.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:49 smithi100 ceph-mon[29092]: 4.1a scrub starts 2024-09-17T07:18:49.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:49 smithi100 ceph-mon[29092]: 4.1a scrub ok 2024-09-17T07:18:49.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:49 smithi100 ceph-mon[39462]: 6.8 scrub starts 2024-09-17T07:18:49.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:49 smithi100 ceph-mon[39462]: 6.8 scrub ok 2024-09-17T07:18:49.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:49 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/4201119666' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:18:49.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:49 smithi100 ceph-mon[39462]: 4.1a scrub starts 2024-09-17T07:18:49.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:49 smithi100 ceph-mon[39462]: 4.1a scrub ok 2024-09-17T07:18:49.638 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:18:49.669 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:49.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:49.670 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:18:49.771 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:18:49.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:49 smithi135 ceph-mon[33238]: 6.8 scrub starts 2024-09-17T07:18:49.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:49 smithi135 ceph-mon[33238]: 6.8 scrub ok 2024-09-17T07:18:49.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:49 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/4201119666' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:18:49.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:49 smithi135 ceph-mon[33238]: 4.1a scrub starts 2024-09-17T07:18:49.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:49 smithi135 ceph-mon[33238]: 4.1a scrub ok 2024-09-17T07:18:49.798 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:49.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:49.799 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:18:49.900 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:18:49.928 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:49.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:49.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:18:50.023 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:18:50.030 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:18:50.063 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:50.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:50.064 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:18:50.165 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:18:50.204 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:50.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:50.205 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:18:50.306 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:18:50.351 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:50.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:50.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:18:50.453 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:18:50.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:50.494 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:50.494 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:18:50.586 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:18:50.587 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:18:50.596 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:18:50.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:50 smithi100 ceph-mon[29092]: 5.4 deep-scrub starts 2024-09-17T07:18:50.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:50 smithi100 ceph-mon[29092]: 5.4 deep-scrub ok 2024-09-17T07:18:50.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:50 smithi100 ceph-mon[29092]: 10.18 scrub starts 2024-09-17T07:18:50.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:50 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:18:50.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:50 smithi100 ceph-mon[29092]: 10.3 scrub starts 2024-09-17T07:18:50.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:50 smithi100 ceph-mon[29092]: pgmap v760: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 7 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 86 active+clean; 3.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 10924/170814 objects degraded (6.395%); 16248/170814 objects misplaced (9.512%) 2024-09-17T07:18:50.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:50 smithi100 ceph-mon[29092]: 10.18 scrub ok 2024-09-17T07:18:50.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:50 smithi100 ceph-mon[29092]: 6.0 scrub starts 2024-09-17T07:18:50.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:50 smithi100 ceph-mon[29092]: 6.0 scrub ok 2024-09-17T07:18:50.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:50 smithi100 ceph-mon[29092]: 10.3 scrub ok 2024-09-17T07:18:50.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:50 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 10924/170814 objects degraded (6.395%), 29 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-09-17T07:18:50.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:50 smithi100 ceph-mon[39462]: 5.4 deep-scrub starts 2024-09-17T07:18:50.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:50 smithi100 ceph-mon[39462]: 5.4 deep-scrub ok 2024-09-17T07:18:50.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:50 smithi100 ceph-mon[39462]: 10.18 scrub starts 2024-09-17T07:18:50.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:50 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:18:50.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:50 smithi100 ceph-mon[39462]: 10.3 scrub starts 2024-09-17T07:18:50.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:50 smithi100 ceph-mon[39462]: pgmap v760: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 7 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 86 active+clean; 3.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 10924/170814 objects degraded (6.395%); 16248/170814 objects misplaced (9.512%) 2024-09-17T07:18:50.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:50 smithi100 ceph-mon[39462]: 10.18 scrub ok 2024-09-17T07:18:50.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:50 smithi100 ceph-mon[39462]: 6.0 scrub starts 2024-09-17T07:18:50.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:50 smithi100 ceph-mon[39462]: 6.0 scrub ok 2024-09-17T07:18:50.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:50 smithi100 ceph-mon[39462]: 10.3 scrub ok 2024-09-17T07:18:50.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:50 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 10924/170814 objects degraded (6.395%), 29 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-09-17T07:18:50.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:50.638 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:50.638 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:18:50.739 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:18:50.767 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:50.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:50.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:18:50.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:50 smithi135 ceph-mon[33238]: 5.4 deep-scrub starts 2024-09-17T07:18:50.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:50 smithi135 ceph-mon[33238]: 5.4 deep-scrub ok 2024-09-17T07:18:50.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:50 smithi135 ceph-mon[33238]: 10.18 scrub starts 2024-09-17T07:18:50.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:50 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:18:50.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:50 smithi135 ceph-mon[33238]: 10.3 scrub starts 2024-09-17T07:18:50.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:50 smithi135 ceph-mon[33238]: pgmap v760: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 7 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 86 active+clean; 3.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 10924/170814 objects degraded (6.395%); 16248/170814 objects misplaced (9.512%) 2024-09-17T07:18:50.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:50 smithi135 ceph-mon[33238]: 10.18 scrub ok 2024-09-17T07:18:50.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:50 smithi135 ceph-mon[33238]: 6.0 scrub starts 2024-09-17T07:18:50.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:50 smithi135 ceph-mon[33238]: 6.0 scrub ok 2024-09-17T07:18:50.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:50 smithi135 ceph-mon[33238]: 10.3 scrub ok 2024-09-17T07:18:50.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:50 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 10924/170814 objects degraded (6.395%), 29 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-09-17T07:18:50.870 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:18:50.906 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:50.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:50.908 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:18:51.009 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:18:51.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:51.046 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:51.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:18:51.148 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:18:51.183 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:51.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:51.184 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:18:51.256 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-17T07:18:51.257 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-17T07:18:51.257 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-17T07:18:51.257 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [] 2024-09-17T07:18:51.257 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-17T07:18:51.285 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:18:51.331 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:51.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:51.332 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:18:51.433 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:18:51.460 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:51.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:51.461 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:18:51.562 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:18:51.598 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:51.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:51.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:18:51.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:51 smithi100 ceph-mon[29092]: 10.c scrub starts 2024-09-17T07:18:51.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:51 smithi100 ceph-mon[29092]: 2.c scrub starts 2024-09-17T07:18:51.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:51 smithi100 ceph-mon[29092]: 2.c scrub ok 2024-09-17T07:18:51.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:51 smithi100 ceph-mon[39462]: 10.c scrub starts 2024-09-17T07:18:51.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:51 smithi100 ceph-mon[39462]: 2.c scrub starts 2024-09-17T07:18:51.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:51 smithi100 ceph-mon[39462]: 2.c scrub ok 2024-09-17T07:18:51.701 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:18:51.730 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:51.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:51.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:18:51.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:51 smithi135 ceph-mon[33238]: 10.c scrub starts 2024-09-17T07:18:51.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:51 smithi135 ceph-mon[33238]: 2.c scrub starts 2024-09-17T07:18:51.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:51 smithi135 ceph-mon[33238]: 2.c scrub ok 2024-09-17T07:18:51.832 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:18:51.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:51.860 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:51.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:18:51.961 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:18:52.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:52.001 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:52.001 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:18:52.102 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:18:52.128 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:52.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:52.129 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:18:52.230 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:18:52.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:52.260 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:52.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:18:52.361 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:18:52.399 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:52.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:52.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:18:52.501 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:18:52.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:52.534 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:52.534 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:18:52.588 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:18:52.635 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:18:52.663 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:52.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:52.664 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:18:52.765 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:18:52.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:52 smithi135 ceph-mon[33238]: 10.c scrub ok 2024-09-17T07:18:52.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:52 smithi135 ceph-mon[33238]: pgmap v761: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 7 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 86 active+clean; 3.4 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s; 10510/168285 objects degraded (6.245%); 15854/168285 objects misplaced (9.421%) 2024-09-17T07:18:52.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:52 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:18:52.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:52 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:18:52.798 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:52.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:52.799 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:18:52.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:52 smithi100 ceph-mon[39462]: 10.c scrub ok 2024-09-17T07:18:52.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:52 smithi100 ceph-mon[39462]: pgmap v761: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 7 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 86 active+clean; 3.4 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s; 10510/168285 objects degraded (6.245%); 15854/168285 objects misplaced (9.421%) 2024-09-17T07:18:52.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:52 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:18:52.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:52 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:18:52.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:52 smithi100 ceph-mon[29092]: 10.c scrub ok 2024-09-17T07:18:52.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:52 smithi100 ceph-mon[29092]: pgmap v761: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 7 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 86 active+clean; 3.4 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s; 10510/168285 objects degraded (6.245%); 15854/168285 objects misplaced (9.421%) 2024-09-17T07:18:52.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:52 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:18:52.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:52 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:18:52.900 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:18:52.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:52.938 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:52.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:18:53.039 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:18:53.075 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:53.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:53.076 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:18:53.177 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:18:53.203 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:53.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:53.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:18:53.305 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:18:53.349 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:53.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:53.350 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:18:53.451 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:18:53.479 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:53.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:53.481 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:18:53.581 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:18:53.609 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:53.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:53.610 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:18:53.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:53 smithi100 ceph-mon[29092]: osdmap e355: 8 total, 7 up, 7 in 2024-09-17T07:18:53.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:53 smithi100 ceph-mon[29092]: pgmap v763: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 7 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 86 active+clean; 3.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s; 10319/160605 objects degraded (6.425%); 14760/160605 objects misplaced (9.190%) 2024-09-17T07:18:53.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:53 smithi100 ceph-mon[39462]: osdmap e355: 8 total, 7 up, 7 in 2024-09-17T07:18:53.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:53 smithi100 ceph-mon[39462]: pgmap v763: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 7 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 86 active+clean; 3.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s; 10319/160605 objects degraded (6.425%); 14760/160605 objects misplaced (9.190%) 2024-09-17T07:18:53.712 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:18:53.753 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:53.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:53.754 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:18:53.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:53 smithi135 ceph-mon[33238]: osdmap e355: 8 total, 7 up, 7 in 2024-09-17T07:18:53.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:53 smithi135 ceph-mon[33238]: pgmap v763: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 7 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 86 active+clean; 3.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s; 10319/160605 objects degraded (6.425%); 14760/160605 objects misplaced (9.190%) 2024-09-17T07:18:53.828 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-17T07:18:53.855 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:18:53.917 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:53.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:53.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:18:54.019 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:18:54.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:54.064 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:54.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:18:54.165 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:18:54.222 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:54.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:54.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:18:54.224 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:18:54.325 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:18:54.353 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:54.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:54.354 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:18:54.455 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:18:54.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:54.484 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:54.484 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:18:54.586 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:18:54.610 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:54.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:54.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:18:54.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:54 smithi100 ceph-mon[29092]: 4.19 scrub starts 2024-09-17T07:18:54.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:54 smithi100 ceph-mon[29092]: 4.19 scrub ok 2024-09-17T07:18:54.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:54 smithi100 ceph-mon[39462]: 4.19 scrub starts 2024-09-17T07:18:54.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:54 smithi100 ceph-mon[39462]: 4.19 scrub ok 2024-09-17T07:18:54.713 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:18:54.739 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:54.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:54.741 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:18:54.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:54 smithi135 ceph-mon[33238]: 4.19 scrub starts 2024-09-17T07:18:54.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:54 smithi135 ceph-mon[33238]: 4.19 scrub ok 2024-09-17T07:18:54.842 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:18:54.874 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:54.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:54.875 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:18:54.975 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:18:55.010 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:55.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:55.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:18:55.112 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:18:55.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:55.151 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:55.152 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:18:55.253 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:18:55.292 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:55.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:55.293 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:18:55.394 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:18:55.431 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:55.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:55.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:18:55.533 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:18:55.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:55 smithi135 ceph-mon[33238]: 2.14 scrub starts 2024-09-17T07:18:55.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:55 smithi135 ceph-mon[33238]: 2.14 scrub ok 2024-09-17T07:18:55.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:55 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:18:55.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:55 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:18:55.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:55 smithi135 ceph-mon[33238]: 10.6 deep-scrub starts 2024-09-17T07:18:55.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:55 smithi135 ceph-mon[33238]: osdmap e356: 8 total, 7 up, 7 in 2024-09-17T07:18:55.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:55 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 10319/160605 objects degraded (6.425%), 29 pgs degraded, 64 pgs undersized (PG_DEGRADED) 2024-09-17T07:18:55.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:55 smithi135 ceph-mon[33238]: pgmap v765: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 7 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 86 active+clean; 3.0 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 9307/147981 objects degraded (6.289%); 13936/147981 objects misplaced (9.417%) 2024-09-17T07:18:55.565 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:55.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:55.566 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:18:55.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:55 smithi100 ceph-mon[29092]: 2.14 scrub starts 2024-09-17T07:18:55.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:55 smithi100 ceph-mon[29092]: 2.14 scrub ok 2024-09-17T07:18:55.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:55 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:18:55.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:55 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:18:55.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:55 smithi100 ceph-mon[29092]: 10.6 deep-scrub starts 2024-09-17T07:18:55.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:55 smithi100 ceph-mon[29092]: osdmap e356: 8 total, 7 up, 7 in 2024-09-17T07:18:55.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:55 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 10319/160605 objects degraded (6.425%), 29 pgs degraded, 64 pgs undersized (PG_DEGRADED) 2024-09-17T07:18:55.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:55 smithi100 ceph-mon[29092]: pgmap v765: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 7 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 86 active+clean; 3.0 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 9307/147981 objects degraded (6.289%); 13936/147981 objects misplaced (9.417%) 2024-09-17T07:18:55.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:55 smithi100 ceph-mon[39462]: 2.14 scrub starts 2024-09-17T07:18:55.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:55 smithi100 ceph-mon[39462]: 2.14 scrub ok 2024-09-17T07:18:55.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:55 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:18:55.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:55 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:18:55.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:55 smithi100 ceph-mon[39462]: 10.6 deep-scrub starts 2024-09-17T07:18:55.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:55 smithi100 ceph-mon[39462]: osdmap e356: 8 total, 7 up, 7 in 2024-09-17T07:18:55.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:55 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 10319/160605 objects degraded (6.425%), 29 pgs degraded, 64 pgs undersized (PG_DEGRADED) 2024-09-17T07:18:55.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:55 smithi100 ceph-mon[39462]: pgmap v765: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 7 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 86 active+clean; 3.0 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 9307/147981 objects degraded (6.289%); 13936/147981 objects misplaced (9.417%) 2024-09-17T07:18:55.667 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:18:55.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:55.696 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:55.696 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:18:55.766 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:18:55.797 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:18:55.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:55.826 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:55.827 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:18:55.927 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:18:55.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:55.971 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:55.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:18:56.032 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:18:55 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:18:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:18:56.073 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:18:56.117 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:56.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:56.118 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:18:56.219 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:18:56.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:56.256 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:56.256 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:18:56.357 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:18:56.386 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:56.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:56.387 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:18:56.423 INFO:teuthology.orchestra.run.smithi100.stderr:noscrub is set 2024-09-17T07:18:56.489 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:18:56.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:56.518 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:56.518 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:18:56.619 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:18:56.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:56 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/887734423' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:18:56.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:56 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:18:56.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:56 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/887734423' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:18:56.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:56 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:18:56.659 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:56.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:56.660 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:18:56.762 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:18:56.792 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:56.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:56.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:18:56.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:56 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/887734423' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:18:56.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:56 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:18:56.894 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:18:56.934 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:56.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:56.935 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:18:57.035 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:18:57.075 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:57.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:57.076 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:18:57.108 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T07:18:57.111 INFO:teuthology.orchestra.run.smithi100.stderr:dumped all 2024-09-17T07:18:57.177 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:18:57.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:57.212 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:57.213 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:18:57.314 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:18:57.353 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:57.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:57.354 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:18:57.455 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:18:57.484 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:57.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:57.485 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:18:57.586 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:18:57.621 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:57.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:57.622 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:18:57.723 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:18:57.758 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:57.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:57.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:18:57.765 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:57 smithi100 ceph-mon[29092]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:18:57.765 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:57 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T07:18:57.765 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:57 smithi100 ceph-mon[29092]: osdmap e357: 8 total, 7 up, 7 in 2024-09-17T07:18:57.765 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:57 smithi100 ceph-mon[29092]: from='client.25679 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:18:57.765 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:57 smithi100 ceph-mon[29092]: pgmap v767: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 7 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 86 active+clean; 3.0 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 852 B/s rd, 0 op/s; 9307/147981 objects degraded (6.289%); 13936/147981 objects misplaced (9.417%) 2024-09-17T07:18:57.765 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:57 smithi100 ceph-mon[39462]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:18:57.765 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:57 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T07:18:57.765 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:57 smithi100 ceph-mon[39462]: osdmap e357: 8 total, 7 up, 7 in 2024-09-17T07:18:57.765 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:57 smithi100 ceph-mon[39462]: from='client.25679 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:18:57.765 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:57 smithi100 ceph-mon[39462]: pgmap v767: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 7 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 86 active+clean; 3.0 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 852 B/s rd, 0 op/s; 9307/147981 objects degraded (6.289%); 13936/147981 objects misplaced (9.417%) 2024-09-17T07:18:57.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:57 smithi135 ceph-mon[33238]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:18:57.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:57 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T07:18:57.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:57 smithi135 ceph-mon[33238]: osdmap e357: 8 total, 7 up, 7 in 2024-09-17T07:18:57.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:57 smithi135 ceph-mon[33238]: from='client.25679 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:18:57.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:57 smithi135 ceph-mon[33238]: pgmap v767: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 7 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 86 active+clean; 3.0 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 852 B/s rd, 0 op/s; 9307/147981 objects degraded (6.289%); 13936/147981 objects misplaced (9.417%) 2024-09-17T07:18:57.860 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:18:57.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:57.888 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:57.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:18:57.989 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:18:58.015 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:58.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:58.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:18:58.117 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:18:58.156 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:58.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:58.157 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:18:58.258 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:18:58.284 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:58.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:58.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:18:58.386 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:18:58.416 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:58.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:58.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:18:58.518 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:18:58.546 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:58.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:58.547 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:18:58.648 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:18:58.675 INFO:teuthology.orchestra.run.smithi100.stdout:{"pg_ready":true,"pg_map":{"version":766,"stamp":"2024-09-17T07:18:56.419693+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3220281646,"num_objects":49327,"num_object_clones":0,"num_object_copies":147981,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9307,"num_objects_misplaced":13936,"num_objects_unfound":0,"num_objects_dirty":49327,"num_whiteouts":0,"num_read":1500,"num_read_kb":1346,"num_write":163498,"num_write_kb":6424252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21696,"num_bytes_recovered":1414074676,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155147,"ondisk_log_size":155147,"up":403,"acting":409,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":476,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":13994236,"kb_used_data":8950956,"kb_used_omap":10,"kb_used_meta":5043061,"kb_avail":642221828,"statfs":{"total":671965249536,"available":657635151872,"internally_reserved":0,"allocated":9165778944,"data_stored":9161141676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11152,"internal_metadata":5164094576},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,1,3,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-498827264,"num_objects":-7611,"num_object_clones":0,"num_object_copies":-22833,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1617,"num_objects_misplaced":-2312,"num_objects_unfound":0,"num_objects_dirty":-7611,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":-4272,"num_write_kb":-489216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-3381,"num_bytes_recovered":-221577216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.077867"},"pg_stats":[{"pgid":"10.17","version":"355'10140","reported_seq":16885,"reported_epoch":355,"state":"active+clean+remapped","last_fresh":"2024-09-17T07:18:53.188496+0000","last_change":"2024-09-17T07:18:18.296127+0000","last_active":"2024-09-17T07:18:53.188496+0000","last_peered":"2024-09-17T07:18:53.188496+0000","last_clean":"2024-09-17T07:18:53.188496+0000","last_became_active":"2024-09-17T07:18:18.283245+0000","last_became_peered":"2024-09-17T07:18:18.283245+0000","last_unstale":"2024-09-17T07:18:53.188496+0000","last_undegraded":"2024-09-17T07:18:53.188496+0000","last_fullsized":"2024-09-17T07:18:53.188496+0000","mapping_epoch":343,"log_start":"313'4134","ondisk_log_start":"313'4134","created":332,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":6006,"log_dups_size":0,"ondisk_log_size":6006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102301696,"num_objects":1561,"num_object_clones":0,"num_object_copies":4683,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1561,"num_objects_unfound":0,"num_objects_dirty":1561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6218,"num_write_kb":248928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3327,"num_bytes_recovered":218038272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6,2],"avail_no_missing":["4","2","6"],"object_location_counts":[{"shards":"2,4,6","objects":1561}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":203,"reported_epoch":354,"state":"active+undersized","last_fresh":"2024-09-17T07:18:42.338075+0000","last_change":"2024-09-17T07:16:17.379311+0000","last_active":"2024-09-17T07:18:42.338075+0000","last_peered":"2024-09-17T07:18:42.338075+0000","last_clean":"2024-09-17T07:16:15.590051+0000","last_became_active":"2024-09-17T07:16:17.379311+0000","last_became_peered":"2024-09-17T07:16:17.379311+0000","last_unstale":"2024-09-17T07:18:42.338075+0000","last_undegraded":"2024-09-17T07:18:42.338075+0000","last_fullsized":"2024-09-17T07:16:17.372051+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:11.978185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:33.644728+0000","last_clean_scrub_stamp":"2024-09-17T07:13:11.978185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0074436570000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":382,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-17T07:18:54.404177+0000","last_change":"2024-09-17T07:18:46.393416+0000","last_active":"2024-09-17T07:18:54.404177+0000","last_peered":"2024-09-17T07:18:54.404177+0000","last_clean":"2024-09-17T07:18:54.404177+0000","last_became_active":"2024-09-17T07:02:12.006754+0000","last_became_peered":"2024-09-17T07:02:12.006754+0000","last_unstale":"2024-09-17T07:18:54.404177+0000","last_undegraded":"2024-09-17T07:18:54.404177+0000","last_fullsized":"2024-09-17T07:18:54.404177+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:18:46.393298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:18:46.393298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:19:47.980212+0000","scrub_duration":0.0058972349999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"242'27","reported_seq":291,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-17T07:18:53.477464+0000","last_change":"2024-09-17T07:18:53.477464+0000","last_active":"2024-09-17T07:18:53.477464+0000","last_peered":"2024-09-17T07:18:53.477464+0000","last_clean":"2024-09-17T07:18:53.477464+0000","last_became_active":"2024-09-17T07:09:24.402042+0000","last_became_peered":"2024-09-17T07:09:24.402042+0000","last_unstale":"2024-09-17T07:18:53.477464+0000","last_undegraded":"2024-09-17T07:18:53.477464+0000","last_fullsized":"2024-09-17T07:18:53.477464+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'27","last_scrub_stamp":"2024-09-17T07:18:53.476878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:18:53.476878+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:20:00.592721+0000","scrub_duration":0.0037504449999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":360,"reported_epoch":354,"state":"active+undersized","last_fresh":"2024-09-17T07:18:42.339365+0000","last_change":"2024-09-17T07:16:17.380766+0000","last_active":"2024-09-17T07:18:42.339365+0000","last_peered":"2024-09-17T07:18:42.339365+0000","last_clean":"2024-09-17T07:16:15.357790+0000","last_became_active":"2024-09-17T07:16:17.380766+0000","last_became_peered":"2024-09-17T07:16:17.380766+0000","last_unstale":"2024-09-17T07:18:42.339365+0000","last_undegraded":"2024-09-17T07:18:42.339365+0000","last_fullsized":"2024-09-17T07:16:17.364849+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:45.261117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:15:45.261117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001477482,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.16","version":"355'9943","reported_seq":16801,"reported_epoch":355,"state":"active+clean+remapped","last_fresh":"2024-09-17T07:18:53.727239+0000","last_change":"2024-09-17T07:18:44.528932+0000","last_active":"2024-09-17T07:18:53.727239+0000","last_peered":"2024-09-17T07:18:53.727239+0000","last_clean":"2024-09-17T07:18:53.727239+0000","last_became_active":"2024-09-17T07:18:18.284748+0000","last_became_peered":"2024-09-17T07:18:18.284748+0000","last_unstale":"2024-09-17T07:18:53.727239+0000","last_undegraded":"2024-09-17T07:18:53.727239+0000","last_fullsized":"2024-09-17T07:18:53.727239+0000","mapping_epoch":343,"log_start":"312'3338","ondisk_log_start":"312'3338","created":332,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":5,"last_scrub":"354'9544","last_scrub_stamp":"2024-09-17T07:18:44.528535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:18:44.528535+0000","objects_scrubbed":1943,"log_size":6605,"log_dups_size":0,"ondisk_log_size":6605,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:19:55.239183+0000","scrub_duration":0.61748898900000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100597760,"num_objects":1535,"num_object_clones":0,"num_object_copies":4605,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1535,"num_objects_unfound":0,"num_objects_dirty":1535,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5,0],"avail_no_missing":["6","0","5"],"object_location_counts":[{"shards":"0,5,6","objects":1535}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":221,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-17T07:18:52.407430+0000","last_change":"2024-09-17T07:17:35.507852+0000","last_active":"2024-09-17T07:18:52.407430+0000","last_peered":"2024-09-17T07:18:52.407430+0000","last_clean":"2024-09-17T07:18:52.407430+0000","last_became_active":"2024-09-17T07:09:24.402568+0000","last_became_peered":"2024-09-17T07:09:24.402568+0000","last_unstale":"2024-09-17T07:18:52.407430+0000","last_undegraded":"2024-09-17T07:18:52.407430+0000","last_fullsized":"2024-09-17T07:18:52.407430+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:17:35.507742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:17:35.507742+0000","last_clean_scrub_stamp":"2024-09-17T07:17:35.507742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:19:01.894837+0000","scrub_duration":0.00224683,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"242'15","reported_seq":398,"reported_epoch":354,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:18:42.339479+0000","last_change":"2024-09-17T07:16:17.370326+0000","last_active":"2024-09-17T07:18:42.339479+0000","last_peered":"2024-09-17T07:18:42.339479+0000","last_clean":"2024-09-17T07:16:15.357813+0000","last_became_active":"2024-09-17T07:16:17.370326+0000","last_became_peered":"2024-09-17T07:16:17.370326+0000","last_unstale":"2024-09-17T07:18:42.339479+0000","last_undegraded":"2024-09-17T07:16:17.364616+0000","last_fullsized":"2024-09-17T07:16:17.364511+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'15","last_scrub_stamp":"2024-09-17T07:15:39.306043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:15:39.306043+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002471029,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":4}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":61,"reported_epoch":354,"state":"active+undersized","last_fresh":"2024-09-17T07:18:43.908541+0000","last_change":"2024-09-17T07:16:17.379569+0000","last_active":"2024-09-17T07:18:43.908541+0000","last_peered":"2024-09-17T07:18:43.908541+0000","last_clean":"2024-09-17T07:02:14.999435+0000","last_became_active":"2024-09-17T07:16:17.379569+0000","last_became_peered":"2024-09-17T07:16:17.379569+0000","last_unstale":"2024-09-17T07:18:43.908541+0000","last_undegraded":"2024-09-17T07:18:43.908541+0000","last_fullsized":"2024-09-17T07:16:17.369924+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:53.881813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:15:53.881813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":359,"reported_epoch":354,"state":"active+undersized","last_fresh":"2024-09-17T07:18:42.339496+0000","last_change":"2024-09-17T07:16:17.374605+0000","last_active":"2024-09-17T07:18:42.339496+0000","last_peered":"2024-09-17T07:18:42.339496+0000","last_clean":"2024-09-17T07:16:15.357982+0000","last_became_active":"2024-09-17T07:16:17.374605+0000","last_became_peered":"2024-09-17T07:16:17.374605+0000","last_unstale":"2024-09-17T07:18:42.339496+0000","last_undegraded":"2024-09-17T07:18:42.339496+0000","last_fullsized":"2024-09-17T07:16:17.366300+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:41.707969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:14:41.707969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.014931757,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.15","version":"355'10042","reported_seq":14899,"reported_epoch":355,"state":"active+clean+remapped","last_fresh":"2024-09-17T07:18:53.860649+0000","last_change":"2024-09-17T07:18:40.282908+0000","last_active":"2024-09-17T07:18:53.860649+0000","last_peered":"2024-09-17T07:18:53.860649+0000","last_clean":"2024-09-17T07:18:53.860649+0000","last_became_active":"2024-09-17T07:18:40.268677+0000","last_became_peered":"2024-09-17T07:18:40.268677+0000","last_unstale":"2024-09-17T07:18:53.860649+0000","last_undegraded":"2024-09-17T07:18:53.860649+0000","last_fullsized":"2024-09-17T07:18:53.860649+0000","mapping_epoch":351,"log_start":"312'3100","ondisk_log_start":"312'3100","created":332,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":6942,"log_dups_size":0,"ondisk_log_size":6942,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102727680,"num_objects":1568,"num_object_clones":0,"num_object_copies":4704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1568,"num_objects_unfound":0,"num_objects_dirty":1568,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6143,"num_write_kb":246720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2379,"num_bytes_recovered":155910144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2,0],"avail_no_missing":["5","0","2"],"object_location_counts":[{"shards":"0,2,5","objects":1568}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":367,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-17T07:18:44.849429+0000","last_change":"2024-09-17T07:18:44.849429+0000","last_active":"2024-09-17T07:18:44.849429+0000","last_peered":"2024-09-17T07:18:44.849429+0000","last_clean":"2024-09-17T07:18:44.849429+0000","last_became_active":"2024-09-17T07:02:12.011129+0000","last_became_peered":"2024-09-17T07:02:12.011129+0000","last_unstale":"2024-09-17T07:18:44.849429+0000","last_undegraded":"2024-09-17T07:18:44.849429+0000","last_fullsized":"2024-09-17T07:18:44.849429+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:18:44.849299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:46.312371+0000","last_clean_scrub_stamp":"2024-09-17T07:18:44.849299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:20:14.794474+0000","scrub_duration":0.005351766,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"242'7","reported_seq":391,"reported_epoch":354,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:18:42.340106+0000","last_change":"2024-09-17T07:16:17.373486+0000","last_active":"2024-09-17T07:18:42.340106+0000","last_peered":"2024-09-17T07:18:42.340106+0000","last_clean":"2024-09-17T07:16:15.358036+0000","last_became_active":"2024-09-17T07:16:17.373486+0000","last_became_peered":"2024-09-17T07:16:17.373486+0000","last_unstale":"2024-09-17T07:18:42.340106+0000","last_undegraded":"2024-09-17T07:16:17.366228+0000","last_fullsized":"2024-09-17T07:16:17.365631+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'7","last_scrub_stamp":"2024-09-17T07:14:42.653705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:14:42.653705+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0050149230000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[{"shards":"2,4","objects":2}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":358,"reported_epoch":354,"state":"active+undersized","last_fresh":"2024-09-17T07:18:42.337768+0000","last_change":"2024-09-17T07:16:17.382323+0000","last_active":"2024-09-17T07:18:42.337768+0000","last_peered":"2024-09-17T07:18:42.337768+0000","last_clean":"2024-09-17T07:16:15.589989+0000","last_became_active":"2024-09-17T07:16:17.382323+0000","last_became_peered":"2024-09-17T07:16:17.382323+0000","last_unstale":"2024-09-17T07:18:42.337768+0000","last_undegraded":"2024-09-17T07:18:42.337768+0000","last_fullsized":"2024-09-17T07:16:17.371260+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:37.997296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:14:37.997296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045486420000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":377,"reported_epoch":354,"state":"active+undersized","last_fresh":"2024-09-17T07:18:42.370519+0000","last_change":"2024-09-17T07:16:17.374238+0000","last_active":"2024-09-17T07:18:42.370519+0000","last_peered":"2024-09-17T07:18:42.370519+0000","last_clean":"2024-09-17T07:16:16.170346+0000","last_became_active":"2024-09-17T07:16:17.374238+0000","last_became_peered":"2024-09-17T07:16:17.374238+0000","last_unstale":"2024-09-17T07:18:42.370519+0000","last_undegraded":"2024-09-17T07:18:42.370519+0000","last_fullsized":"2024-09-17T07:16:17.367446+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:52.168181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:14:52.168181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0056501399999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.14","version":"355'10007","reported_seq":16846,"reported_epoch":355,"state":"active+clean+remapped","last_fresh":"2024-09-17T07:18:53.169988+0000","last_change":"2024-09-17T07:18:18.296134+0000","last_active":"2024-09-17T07:18:53.169988+0000","last_peered":"2024-09-17T07:18:53.169988+0000","last_clean":"2024-09-17T07:18:53.169988+0000","last_became_active":"2024-09-17T07:18:18.281656+0000","last_became_peered":"2024-09-17T07:18:18.281656+0000","last_unstale":"2024-09-17T07:18:53.169988+0000","last_undegraded":"2024-09-17T07:18:53.169988+0000","last_fullsized":"2024-09-17T07:18:53.169988+0000","mapping_epoch":343,"log_start":"313'3942","ondisk_log_start":"313'3942","created":332,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":6065,"log_dups_size":0,"ondisk_log_size":6065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":101515264,"num_objects":1549,"num_object_clones":0,"num_object_copies":4647,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1549,"num_objects_unfound":0,"num_objects_dirty":1549,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6129,"num_write_kb":245696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3370,"num_bytes_recovered":220856320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2,0],"avail_no_missing":["4","0","2"],"object_location_counts":[{"shards":"0,2,4","objects":1549}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":357,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-17T07:18:42.338615+0000","last_change":"2024-09-17T07:18:34.258935+0000","last_active":"2024-09-17T07:18:42.338615+0000","last_peered":"2024-09-17T07:18:42.338615+0000","last_clean":"2024-09-17T07:18:42.338615+0000","last_became_active":"2024-09-17T07:02:12.012234+0000","last_became_peered":"2024-09-17T07:02:12.012234+0000","last_unstale":"2024-09-17T07:18:42.338615+0000","last_undegraded":"2024-09-17T07:18:42.338615+0000","last_fullsized":"2024-09-17T07:18:42.338615+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:44.076012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:45.029247+0000","last_clean_scrub_stamp":"2024-09-17T07:14:44.076012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.015917521,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"312'19","reported_seq":426,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-17T07:18:49.194511+0000","last_change":"2024-09-17T07:18:49.194511+0000","last_active":"2024-09-17T07:18:49.194511+0000","last_peered":"2024-09-17T07:18:49.194511+0000","last_clean":"2024-09-17T07:18:49.194511+0000","last_became_active":"2024-09-17T07:09:24.397954+0000","last_became_peered":"2024-09-17T07:09:24.397954+0000","last_unstale":"2024-09-17T07:18:49.194511+0000","last_undegraded":"2024-09-17T07:18:49.194511+0000","last_fullsized":"2024-09-17T07:18:49.194511+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"312'19","last_scrub_stamp":"2024-09-17T07:18:49.194375+0000","last_deep_scrub":"312'19","last_deep_scrub_stamp":"2024-09-17T07:16:44.292911+0000","last_clean_scrub_stamp":"2024-09-17T07:18:49.194375+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:19:49.607719+0000","scrub_duration":0.0031889269999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":374,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-17T07:18:42.368843+0000","last_change":"2024-09-17T07:16:45.164526+0000","last_active":"2024-09-17T07:18:42.368843+0000","last_peered":"2024-09-17T07:18:42.368843+0000","last_clean":"2024-09-17T07:18:42.368843+0000","last_became_active":"2024-09-17T07:02:16.023774+0000","last_became_peered":"2024-09-17T07:02:16.023774+0000","last_unstale":"2024-09-17T07:18:42.368843+0000","last_undegraded":"2024-09-17T07:18:42.368843+0000","last_fullsized":"2024-09-17T07:18:42.368843+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:45.164367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:28.096266+0000","last_clean_scrub_stamp":"2024-09-17T07:16:45.164367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026790630000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":359,"reported_epoch":355,"state":"active+undersized","last_fresh":"2024-09-17T07:18:52.406165+0000","last_change":"2024-09-17T07:16:17.378534+0000","last_active":"2024-09-17T07:18:52.406165+0000","last_peered":"2024-09-17T07:18:52.406165+0000","last_clean":"2024-09-17T07:16:15.494529+0000","last_became_active":"2024-09-17T07:16:17.378534+0000","last_became_peered":"2024-09-17T07:16:17.378534+0000","last_unstale":"2024-09-17T07:18:52.406165+0000","last_undegraded":"2024-09-17T07:18:52.406165+0000","last_fullsized":"2024-09-17T07:16:17.365363+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:45.483622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:15:45.483622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001342359,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3","version":"355'13051","reported_seq":13507,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-17T07:18:53.910662+0000","last_change":"2024-09-17T07:18:49.791732+0000","last_active":"2024-09-17T07:18:53.910662+0000","last_peered":"2024-09-17T07:18:53.910662+0000","last_clean":"2024-09-17T07:18:53.910662+0000","last_became_active":"2024-09-17T07:17:52.409230+0000","last_became_peered":"2024-09-17T07:17:52.409230+0000","last_unstale":"2024-09-17T07:18:53.910662+0000","last_undegraded":"2024-09-17T07:18:53.910662+0000","last_fullsized":"2024-09-17T07:18:53.910662+0000","mapping_epoch":297,"log_start":"326'7507","ondisk_log_start":"326'7507","created":297,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":0,"last_scrub":"354'12865","last_scrub_stamp":"2024-09-17T07:18:49.791465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:18:49.791465+0000","objects_scrubbed":1684,"log_size":5544,"log_dups_size":0,"ondisk_log_size":5544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:19:56.371309+0000","scrub_duration":0.44981474399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":97845248,"num_objects":1493,"num_object_clones":0,"num_object_copies":4479,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":191,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"242'39","reported_seq":431,"reported_epoch":354,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:18:42.340247+0000","last_change":"2024-09-17T07:16:17.374805+0000","last_active":"2024-09-17T07:18:42.340247+0000","last_peered":"2024-09-17T07:18:42.340247+0000","last_clean":"2024-09-17T07:16:15.358504+0000","last_became_active":"2024-09-17T07:16:17.374805+0000","last_became_peered":"2024-09-17T07:16:17.374805+0000","last_unstale":"2024-09-17T07:18:42.340247+0000","last_undegraded":"2024-09-17T07:16:17.367444+0000","last_fullsized":"2024-09-17T07:16:17.367331+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'39","last_scrub_stamp":"2024-09-17T07:15:46.226554+0000","last_deep_scrub":"95'25","last_deep_scrub_stamp":"2024-09-17T07:04:09.450039+0000","last_clean_scrub_stamp":"2024-09-17T07:15:46.226554+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0027061799999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[{"shards":"2,4","objects":8}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":370,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-17T07:18:42.338434+0000","last_change":"2024-09-17T07:16:33.658452+0000","last_active":"2024-09-17T07:18:42.338434+0000","last_peered":"2024-09-17T07:18:42.338434+0000","last_clean":"2024-09-17T07:18:42.338434+0000","last_became_active":"2024-09-17T07:02:12.011258+0000","last_became_peered":"2024-09-17T07:02:12.011258+0000","last_unstale":"2024-09-17T07:18:42.338434+0000","last_undegraded":"2024-09-17T07:18:42.338434+0000","last_fullsized":"2024-09-17T07:18:42.338434+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:33.658293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:16:33.658293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.044680275999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":61,"reported_epoch":354,"state":"active+undersized","last_fresh":"2024-09-17T07:18:42.340164+0000","last_change":"2024-09-17T07:16:17.372955+0000","last_active":"2024-09-17T07:18:42.340164+0000","last_peered":"2024-09-17T07:18:42.340164+0000","last_clean":"2024-09-17T07:02:14.999435+0000","last_became_active":"2024-09-17T07:16:17.372955+0000","last_became_peered":"2024-09-17T07:16:17.372955+0000","last_unstale":"2024-09-17T07:18:42.340164+0000","last_undegraded":"2024-09-17T07:18:42.340164+0000","last_fullsized":"2024-09-17T07:16:17.365245+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:46.502641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:13:46.502641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":223,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-17T07:18:42.340427+0000","last_change":"2024-09-17T07:18:42.105243+0000","last_active":"2024-09-17T07:18:42.340427+0000","last_peered":"2024-09-17T07:18:42.340427+0000","last_clean":"2024-09-17T07:18:42.340427+0000","last_became_active":"2024-09-17T07:09:24.396334+0000","last_became_peered":"2024-09-17T07:09:24.396334+0000","last_unstale":"2024-09-17T07:18:42.340427+0000","last_undegraded":"2024-09-17T07:18:42.340427+0000","last_fullsized":"2024-09-17T07:18:42.340427+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:18:42.105173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:42.984088+0000","last_clean_scrub_stamp":"2024-09-17T07:18:42.105173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:20:09.737648+0000","scrub_duration":0.0012123559999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2","version":"355'9921","reported_seq":9988,"reported_epoch":355,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:18:53.228865+0000","last_change":"2024-09-17T07:17:52.402716+0000","last_active":"2024-09-17T07:18:53.228865+0000","last_peered":"2024-09-17T07:18:53.228865+0000","last_clean":"2024-09-17T07:16:04.025356+0000","last_became_active":"2024-09-17T07:17:52.402716+0000","last_became_peered":"2024-09-17T07:17:52.402716+0000","last_unstale":"2024-09-17T07:18:53.228865+0000","last_undegraded":"2024-09-17T07:17:52.372244+0000","last_fullsized":"2024-09-17T07:17:52.361979+0000","mapping_epoch":303,"log_start":"313'3784","ondisk_log_start":"313'3784","created":297,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":6137,"log_dups_size":0,"ondisk_log_size":6137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105480192,"num_objects":1609,"num_object_clones":0,"num_object_copies":4827,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1609,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6052,"num_write_kb":245184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[{"shards":"0,4","objects":1609}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"242'25","reported_seq":430,"reported_epoch":354,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:18:42.340424+0000","last_change":"2024-09-17T07:16:17.381025+0000","last_active":"2024-09-17T07:18:42.340424+0000","last_peered":"2024-09-17T07:18:42.340424+0000","last_clean":"2024-09-17T07:16:15.358419+0000","last_became_active":"2024-09-17T07:16:17.381025+0000","last_became_peered":"2024-09-17T07:16:17.381025+0000","last_unstale":"2024-09-17T07:18:42.340424+0000","last_undegraded":"2024-09-17T07:16:17.367122+0000","last_fullsized":"2024-09-17T07:16:17.365635+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'25","last_scrub_stamp":"2024-09-17T07:15:02.532371+0000","last_deep_scrub":"176'15","last_deep_scrub_stamp":"2024-09-17T07:08:19.633791+0000","last_clean_scrub_stamp":"2024-09-17T07:15:02.532371+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012712606,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":233,"reported_epoch":354,"state":"active+undersized","last_fresh":"2024-09-17T07:18:42.337978+0000","last_change":"2024-09-17T07:16:17.379662+0000","last_active":"2024-09-17T07:18:42.337978+0000","last_peered":"2024-09-17T07:18:42.337978+0000","last_clean":"2024-09-17T07:09:22.372452+0000","last_became_active":"2024-09-17T07:16:17.379662+0000","last_became_peered":"2024-09-17T07:16:17.379662+0000","last_unstale":"2024-09-17T07:18:42.337978+0000","last_undegraded":"2024-09-17T07:18:42.337978+0000","last_fullsized":"2024-09-17T07:16:17.370523+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:59.697560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:14:59.697560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0065215869999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.d","version":"98'9","reported_seq":514,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-17T07:18:50.714662+0000","last_change":"2024-09-17T07:18:37.282353+0000","last_active":"2024-09-17T07:18:50.714662+0000","last_peered":"2024-09-17T07:18:50.714662+0000","last_clean":"2024-09-17T07:18:50.714662+0000","last_became_active":"2024-09-17T07:04:15.110797+0000","last_became_peered":"2024-09-17T07:04:15.110797+0000","last_unstale":"2024-09-17T07:18:50.714662+0000","last_undegraded":"2024-09-17T07:18:50.714662+0000","last_fullsized":"2024-09-17T07:18:50.714662+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-09-17T07:14:46.126045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:14:46.126045+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.014185467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":370,"reported_epoch":354,"state":"active+undersized","last_fresh":"2024-09-17T07:18:42.340401+0000","last_change":"2024-09-17T07:16:17.373630+0000","last_active":"2024-09-17T07:18:42.340401+0000","last_peered":"2024-09-17T07:18:42.340401+0000","last_clean":"2024-09-17T07:16:15.358526+0000","last_became_active":"2024-09-17T07:16:17.373630+0000","last_became_peered":"2024-09-17T07:16:17.373630+0000","last_unstale":"2024-09-17T07:18:42.340401+0000","last_undegraded":"2024-09-17T07:18:42.340401+0000","last_fullsized":"2024-09-17T07:16:17.369047+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:50.554591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:14:50.554591+0000","last_clean_scrub_stamp":"2024-09-17T07:14:50.554591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0067125969999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1","version":"355'10347","reported_seq":10561,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-17T07:18:53.218380+0000","last_change":"2024-09-17T07:17:52.414213+0000","last_active":"2024-09-17T07:18:53.218380+0000","last_peered":"2024-09-17T07:18:53.218380+0000","last_clean":"2024-09-17T07:18:53.218380+0000","last_became_active":"2024-09-17T07:17:52.402933+0000","last_became_peered":"2024-09-17T07:17:52.402933+0000","last_unstale":"2024-09-17T07:18:53.218380+0000","last_undegraded":"2024-09-17T07:18:53.218380+0000","last_fullsized":"2024-09-17T07:18:53.218380+0000","mapping_epoch":297,"log_start":"314'4404","ondisk_log_start":"314'4404","created":297,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'6219","last_scrub_stamp":"2024-09-17T07:17:22.401249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:17:22.401249+0000","objects_scrubbed":6144,"log_size":5943,"log_dups_size":0,"ondisk_log_size":5943,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2.0885702689999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98697216,"num_objects":1506,"num_object_clones":0,"num_object_copies":4518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6370,"num_write_kb":252032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"242'27","reported_seq":355,"reported_epoch":354,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:18:42.340650+0000","last_change":"2024-09-17T07:16:17.382238+0000","last_active":"2024-09-17T07:18:42.340650+0000","last_peered":"2024-09-17T07:18:42.340650+0000","last_clean":"2024-09-17T07:12:20.775697+0000","last_became_active":"2024-09-17T07:16:17.382238+0000","last_became_peered":"2024-09-17T07:16:17.382238+0000","last_unstale":"2024-09-17T07:18:42.340650+0000","last_undegraded":"2024-09-17T07:16:17.365468+0000","last_fullsized":"2024-09-17T07:16:17.365334+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'27","last_scrub_stamp":"2024-09-17T07:14:45.831734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:14:45.831734+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021853620000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[{"shards":"4,7","objects":7}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":234,"reported_epoch":354,"state":"active+undersized","last_fresh":"2024-09-17T07:18:42.337903+0000","last_change":"2024-09-17T07:16:17.378599+0000","last_active":"2024-09-17T07:18:42.337903+0000","last_peered":"2024-09-17T07:18:42.337903+0000","last_clean":"2024-09-17T07:09:22.371958+0000","last_became_active":"2024-09-17T07:16:17.378599+0000","last_became_peered":"2024-09-17T07:16:17.378599+0000","last_unstale":"2024-09-17T07:18:42.337903+0000","last_undegraded":"2024-09-17T07:18:42.337903+0000","last_fullsized":"2024-09-17T07:16:17.373236+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:44.897611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:15:44.897611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034116369999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.e","version":"75'8","reported_seq":574,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-17T07:18:50.715926+0000","last_change":"2024-09-17T07:16:49.331611+0000","last_active":"2024-09-17T07:18:50.715926+0000","last_peered":"2024-09-17T07:18:50.715926+0000","last_clean":"2024-09-17T07:18:50.715926+0000","last_became_active":"2024-09-17T07:02:16.021762+0000","last_became_peered":"2024-09-17T07:02:16.021762+0000","last_unstale":"2024-09-17T07:18:50.715926+0000","last_undegraded":"2024-09-17T07:18:50.715926+0000","last_fullsized":"2024-09-17T07:18:50.715926+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:16:49.331407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:16:49.331407+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033687029999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":368,"reported_epoch":354,"state":"active+undersized","last_fresh":"2024-09-17T07:18:42.369293+0000","last_change":"2024-09-17T07:16:17.374897+0000","last_active":"2024-09-17T07:18:42.369293+0000","last_peered":"2024-09-17T07:18:42.369293+0000","last_clean":"2024-09-17T07:16:16.170588+0000","last_became_active":"2024-09-17T07:16:17.374897+0000","last_became_peered":"2024-09-17T07:16:17.374897+0000","last_unstale":"2024-09-17T07:18:42.369293+0000","last_undegraded":"2024-09-17T07:18:42.369293+0000","last_fullsized":"2024-09-17T07:16:17.368895+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:56.146098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:14:56.146098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0054479960000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.0","version":"355'10047","reported_seq":10127,"reported_epoch":355,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:18:53.710738+0000","last_change":"2024-09-17T07:17:52.385485+0000","last_active":"2024-09-17T07:18:53.710738+0000","last_peered":"2024-09-17T07:18:53.710738+0000","last_clean":"2024-09-17T07:16:14.245624+0000","last_became_active":"2024-09-17T07:17:52.385485+0000","last_became_peered":"2024-09-17T07:17:52.385485+0000","last_unstale":"2024-09-17T07:18:53.710738+0000","last_undegraded":"2024-09-17T07:17:52.365146+0000","last_fullsized":"2024-09-17T07:17:52.365038+0000","mapping_epoch":303,"log_start":"312'3371","ondisk_log_start":"312'3371","created":297,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":6676,"log_dups_size":0,"ondisk_log_size":6676,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102760448,"num_objects":1568,"num_object_clones":0,"num_object_copies":4704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1568,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1568,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6150,"num_write_kb":246976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[{"shards":"4,6","objects":1568}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"242'19","reported_seq":405,"reported_epoch":354,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:18:42.340558+0000","last_change":"2024-09-17T07:16:17.382184+0000","last_active":"2024-09-17T07:18:42.340558+0000","last_peered":"2024-09-17T07:18:42.340558+0000","last_clean":"2024-09-17T07:16:15.358681+0000","last_became_active":"2024-09-17T07:16:17.382184+0000","last_became_peered":"2024-09-17T07:16:17.382184+0000","last_unstale":"2024-09-17T07:18:42.340558+0000","last_undegraded":"2024-09-17T07:16:17.369179+0000","last_fullsized":"2024-09-17T07:16:17.369087+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'19","last_scrub_stamp":"2024-09-17T07:14:55.489874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:14:55.489874+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045924440000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":364,"reported_epoch":354,"state":"active+undersized","last_fresh":"2024-09-17T07:18:42.337659+0000","last_change":"2024-09-17T07:16:17.375239+0000","last_active":"2024-09-17T07:18:42.337659+0000","last_peered":"2024-09-17T07:18:42.337659+0000","last_clean":"2024-09-17T07:16:15.590006+0000","last_became_active":"2024-09-17T07:16:17.375239+0000","last_became_peered":"2024-09-17T07:16:17.375239+0000","last_unstale":"2024-09-17T07:18:42.337659+0000","last_undegraded":"2024-09-17T07:18:42.337659+0000","last_fullsized":"2024-09-17T07:16:17.372272+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:21.858127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:13:21.858127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033279709999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":386,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-17T07:18:42.368668+0000","last_change":"2024-09-17T07:17:18.435029+0000","last_active":"2024-09-17T07:18:42.368668+0000","last_peered":"2024-09-17T07:18:42.368668+0000","last_clean":"2024-09-17T07:18:42.368668+0000","last_became_active":"2024-09-17T07:02:16.020224+0000","last_became_peered":"2024-09-17T07:02:16.020224+0000","last_unstale":"2024-09-17T07:18:42.368668+0000","last_undegraded":"2024-09-17T07:18:42.368668+0000","last_fullsized":"2024-09-17T07:18:42.368668+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:17:18.434919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:17:18.434919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0044010059999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":240,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-17T07:18:42.360887+0000","last_change":"2024-09-17T07:18:40.926042+0000","last_active":"2024-09-17T07:18:42.360887+0000","last_peered":"2024-09-17T07:18:42.360887+0000","last_clean":"2024-09-17T07:18:42.360887+0000","last_became_active":"2024-09-17T07:09:24.408109+0000","last_became_peered":"2024-09-17T07:09:24.408109+0000","last_unstale":"2024-09-17T07:18:42.360887+0000","last_undegraded":"2024-09-17T07:18:42.360887+0000","last_fullsized":"2024-09-17T07:18:42.360887+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:18:40.925918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:13:22.187127+0000","last_clean_scrub_stamp":"2024-09-17T07:18:40.925918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:20:00.910570+0000","scrub_duration":0.0076924289999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.f","version":"355'15432","reported_seq":15806,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-17T07:18:53.735712+0000","last_change":"2024-09-17T07:18:27.858457+0000","last_active":"2024-09-17T07:18:53.735712+0000","last_peered":"2024-09-17T07:18:53.735712+0000","last_clean":"2024-09-17T07:18:53.735712+0000","last_became_active":"2024-09-17T07:16:05.052037+0000","last_became_peered":"2024-09-17T07:16:05.052037+0000","last_unstale":"2024-09-17T07:18:53.735712+0000","last_undegraded":"2024-09-17T07:18:53.735712+0000","last_fullsized":"2024-09-17T07:18:53.735712+0000","mapping_epoch":297,"log_start":"336'10953","ondisk_log_start":"336'10953","created":297,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":3774,"log_size":4479,"log_dups_size":0,"ondisk_log_size":4479,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":203554816,"num_objects":3106,"num_object_clones":0,"num_object_copies":9318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12498,"num_write_kb":499328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"242'28","reported_seq":411,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-17T07:18:52.402033+0000","last_change":"2024-09-17T07:17:31.149145+0000","last_active":"2024-09-17T07:18:52.402033+0000","last_peered":"2024-09-17T07:18:52.402033+0000","last_clean":"2024-09-17T07:18:52.402033+0000","last_became_active":"2024-09-17T07:02:14.007810+0000","last_became_peered":"2024-09-17T07:02:14.007810+0000","last_unstale":"2024-09-17T07:18:52.402033+0000","last_undegraded":"2024-09-17T07:18:52.402033+0000","last_fullsized":"2024-09-17T07:18:52.402033+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'28","last_scrub_stamp":"2024-09-17T07:17:31.148909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:17:31.148909+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.013898221000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":411,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-17T07:18:42.361407+0000","last_change":"2024-09-17T07:17:19.376369+0000","last_active":"2024-09-17T07:18:42.361407+0000","last_peered":"2024-09-17T07:18:42.361407+0000","last_clean":"2024-09-17T07:18:42.361407+0000","last_became_active":"2024-09-17T07:02:12.011766+0000","last_became_peered":"2024-09-17T07:02:12.011766+0000","last_unstale":"2024-09-17T07:18:42.361407+0000","last_undegraded":"2024-09-17T07:18:42.361407+0000","last_fullsized":"2024-09-17T07:18:42.361407+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:17:19.375818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:09:23.181028+0000","last_clean_scrub_stamp":"2024-09-17T07:17:19.375818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.019104131999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"304'10","reported_seq":113,"reported_epoch":354,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:18:50.715956+0000","last_change":"2024-09-17T07:16:17.371678+0000","last_active":"2024-09-17T07:18:50.715956+0000","last_peered":"2024-09-17T07:18:50.715956+0000","last_clean":"2024-09-17T07:02:19.145975+0000","last_became_active":"2024-09-17T07:16:17.371678+0000","last_became_peered":"2024-09-17T07:16:17.371678+0000","last_unstale":"2024-09-17T07:18:50.715956+0000","last_undegraded":"2024-09-17T07:16:17.366242+0000","last_fullsized":"2024-09-17T07:16:17.365352+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'9","last_scrub_stamp":"2024-09-17T07:15:55.813882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:15:55.813882+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[{"shards":"0,4","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":373,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-17T07:18:46.921494+0000","last_change":"2024-09-17T07:18:46.921494+0000","last_active":"2024-09-17T07:18:46.921494+0000","last_peered":"2024-09-17T07:18:46.921494+0000","last_clean":"2024-09-17T07:18:46.921494+0000","last_became_active":"2024-09-17T07:02:18.364981+0000","last_became_peered":"2024-09-17T07:02:18.364981+0000","last_unstale":"2024-09-17T07:18:46.921494+0000","last_undegraded":"2024-09-17T07:18:46.921494+0000","last_fullsized":"2024-09-17T07:18:46.921494+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:18:46.921412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:10:33.727472+0000","last_clean_scrub_stamp":"2024-09-17T07:18:46.921412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:19:52.740617+0000","scrub_duration":0.0012782189999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"355'15191","reported_seq":15696,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-17T07:18:53.216736+0000","last_change":"2024-09-17T07:18:26.134904+0000","last_active":"2024-09-17T07:18:53.216736+0000","last_peered":"2024-09-17T07:18:53.216736+0000","last_clean":"2024-09-17T07:18:53.216736+0000","last_became_active":"2024-09-17T07:16:05.054109+0000","last_became_peered":"2024-09-17T07:16:05.054109+0000","last_unstale":"2024-09-17T07:18:53.216736+0000","last_undegraded":"2024-09-17T07:18:53.216736+0000","last_fullsized":"2024-09-17T07:18:53.216736+0000","mapping_epoch":297,"log_start":"337'10991","ondisk_log_start":"337'10991","created":297,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'12995","last_scrub_stamp":"2024-09-17T07:18:26.134704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:18:26.134704+0000","objects_scrubbed":5398,"log_size":4200,"log_dups_size":0,"ondisk_log_size":4200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:19:30.219027+0000","scrub_duration":1.955922511,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":203816960,"num_objects":3110,"num_object_clones":0,"num_object_copies":9330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12198,"num_write_kb":489856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"242'30","reported_seq":283,"reported_epoch":354,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:18:42.337503+0000","last_change":"2024-09-17T07:16:17.378001+0000","last_active":"2024-09-17T07:18:42.337503+0000","last_peered":"2024-09-17T07:18:42.337503+0000","last_clean":"2024-09-17T07:16:15.589841+0000","last_became_active":"2024-09-17T07:16:17.378001+0000","last_became_peered":"2024-09-17T07:16:17.378001+0000","last_unstale":"2024-09-17T07:18:42.337503+0000","last_undegraded":"2024-09-17T07:16:17.372487+0000","last_fullsized":"2024-09-17T07:16:17.372376+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'30","last_scrub_stamp":"2024-09-17T07:13:28.997563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:13:28.997563+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0061961159999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[{"shards":"0,7","objects":8}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":61,"reported_epoch":354,"state":"active+undersized","last_fresh":"2024-09-17T07:18:42.361260+0000","last_change":"2024-09-17T07:16:17.376547+0000","last_active":"2024-09-17T07:18:42.361260+0000","last_peered":"2024-09-17T07:18:42.361260+0000","last_clean":"2024-09-17T07:02:10.991363+0000","last_became_active":"2024-09-17T07:16:17.376547+0000","last_became_peered":"2024-09-17T07:16:17.376547+0000","last_unstale":"2024-09-17T07:18:42.361260+0000","last_undegraded":"2024-09-17T07:18:42.361260+0000","last_fullsized":"2024-09-17T07:16:17.364886+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:56.848689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:56.848689+0000","last_clean_scrub_stamp":"2024-09-17T07:15:56.848689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":376,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-17T07:18:52.401883+0000","last_change":"2024-09-17T07:16:38.184957+0000","last_active":"2024-09-17T07:18:52.401883+0000","last_peered":"2024-09-17T07:18:52.401883+0000","last_clean":"2024-09-17T07:18:52.401883+0000","last_became_active":"2024-09-17T07:09:24.393375+0000","last_became_peered":"2024-09-17T07:09:24.393375+0000","last_unstale":"2024-09-17T07:18:52.401883+0000","last_undegraded":"2024-09-17T07:18:52.401883+0000","last_fullsized":"2024-09-17T07:18:52.401883+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:38.184818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:16:38.184818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0076357700000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":373,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-17T07:18:42.339694+0000","last_change":"2024-09-17T07:16:36.162077+0000","last_active":"2024-09-17T07:18:42.339694+0000","last_peered":"2024-09-17T07:18:42.339694+0000","last_clean":"2024-09-17T07:18:42.339694+0000","last_became_active":"2024-09-17T07:09:24.398659+0000","last_became_peered":"2024-09-17T07:09:24.398659+0000","last_unstale":"2024-09-17T07:18:42.339694+0000","last_undegraded":"2024-09-17T07:18:42.339694+0000","last_fullsized":"2024-09-17T07:18:42.339694+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:36.161622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:16:36.161622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0062812850000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.d","version":"355'12412","reported_seq":12481,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-17T07:18:53.712063+0000","last_change":"2024-09-17T07:18:29.882659+0000","last_active":"2024-09-17T07:18:53.712063+0000","last_peered":"2024-09-17T07:18:53.712063+0000","last_clean":"2024-09-17T07:18:53.712063+0000","last_became_active":"2024-09-17T07:16:05.052153+0000","last_became_peered":"2024-09-17T07:16:05.052153+0000","last_unstale":"2024-09-17T07:18:53.712063+0000","last_undegraded":"2024-09-17T07:18:53.712063+0000","last_fullsized":"2024-09-17T07:18:53.712063+0000","mapping_epoch":297,"log_start":"336'7930","ondisk_log_start":"336'7930","created":297,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":202178584,"num_objects":3086,"num_object_clones":0,"num_object_copies":9258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3086,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12412,"num_write_kb":495873,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"242'29","reported_seq":421,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-17T07:18:54.404543+0000","last_change":"2024-09-17T07:17:40.599358+0000","last_active":"2024-09-17T07:18:54.404543+0000","last_peered":"2024-09-17T07:18:54.404543+0000","last_clean":"2024-09-17T07:18:54.404543+0000","last_became_active":"2024-09-17T07:02:14.027389+0000","last_became_peered":"2024-09-17T07:02:14.027389+0000","last_unstale":"2024-09-17T07:18:54.404543+0000","last_undegraded":"2024-09-17T07:18:54.404543+0000","last_fullsized":"2024-09-17T07:18:54.404543+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'29","last_scrub_stamp":"2024-09-17T07:17:40.599184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:17:40.599184+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.073497912999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":369,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-17T07:18:42.338898+0000","last_change":"2024-09-17T07:16:47.631557+0000","last_active":"2024-09-17T07:18:42.338898+0000","last_peered":"2024-09-17T07:18:42.338898+0000","last_clean":"2024-09-17T07:18:42.338898+0000","last_became_active":"2024-09-17T07:02:12.011917+0000","last_became_peered":"2024-09-17T07:02:12.011917+0000","last_unstale":"2024-09-17T07:18:42.338898+0000","last_undegraded":"2024-09-17T07:18:42.338898+0000","last_fullsized":"2024-09-17T07:18:42.338898+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:47.631387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:47.631387+0000","last_clean_scrub_stamp":"2024-09-17T07:16:47.631387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.006082372,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":386,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-17T07:18:42.360365+0000","last_change":"2024-09-17T07:17:42.360268+0000","last_active":"2024-09-17T07:18:42.360365+0000","last_peered":"2024-09-17T07:18:42.360365+0000","last_clean":"2024-09-17T07:18:42.360365+0000","last_became_active":"2024-09-17T07:02:16.016199+0000","last_became_peered":"2024-09-17T07:02:16.016199+0000","last_unstale":"2024-09-17T07:18:42.360365+0000","last_undegraded":"2024-09-17T07:18:42.360365+0000","last_fullsized":"2024-09-17T07:18:42.360365+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:17:42.360104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:11:17.583712+0000","last_clean_scrub_stamp":"2024-09-17T07:17:42.360104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:54.611733+0000","scrub_duration":0.0062057919999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":61,"reported_epoch":354,"state":"active+undersized","last_fresh":"2024-09-17T07:18:42.360753+0000","last_change":"2024-09-17T07:16:17.368186+0000","last_active":"2024-09-17T07:18:42.360753+0000","last_peered":"2024-09-17T07:18:42.360753+0000","last_clean":"2024-09-17T07:02:17.002904+0000","last_became_active":"2024-09-17T07:16:17.368186+0000","last_became_peered":"2024-09-17T07:16:17.368186+0000","last_unstale":"2024-09-17T07:18:42.360753+0000","last_undegraded":"2024-09-17T07:18:42.360753+0000","last_fullsized":"2024-09-17T07:16:17.362908+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:50.514525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:13:50.514525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.c","version":"355'12386","reported_seq":12531,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-17T07:18:53.027984+0000","last_change":"2024-09-17T07:18:50.965143+0000","last_active":"2024-09-17T07:18:53.027984+0000","last_peered":"2024-09-17T07:18:53.027984+0000","last_clean":"2024-09-17T07:18:53.027984+0000","last_became_active":"2024-09-17T07:16:05.057588+0000","last_became_peered":"2024-09-17T07:16:05.057588+0000","last_unstale":"2024-09-17T07:18:53.027984+0000","last_undegraded":"2024-09-17T07:18:53.027984+0000","last_fullsized":"2024-09-17T07:18:53.027984+0000","mapping_epoch":297,"log_start":"336'7966","ondisk_log_start":"336'7966","created":297,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"354'12224","last_scrub_stamp":"2024-09-17T07:18:50.964766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:18:50.964766+0000","objects_scrubbed":3319,"log_size":4420,"log_dups_size":0,"ondisk_log_size":4420,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:20:01.236328+0000","scrub_duration":1.073020833,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":204079104,"num_objects":3114,"num_object_clones":0,"num_object_copies":9342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3114,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12386,"num_write_kb":496000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"242'25","reported_seq":322,"reported_epoch":354,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:18:42.370603+0000","last_change":"2024-09-17T07:16:17.376630+0000","last_active":"2024-09-17T07:18:42.370603+0000","last_peered":"2024-09-17T07:18:42.370603+0000","last_clean":"2024-09-17T07:12:20.656798+0000","last_became_active":"2024-09-17T07:16:17.376630+0000","last_became_peered":"2024-09-17T07:16:17.376630+0000","last_unstale":"2024-09-17T07:18:42.370603+0000","last_undegraded":"2024-09-17T07:16:17.368029+0000","last_fullsized":"2024-09-17T07:16:17.367816+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'25","last_scrub_stamp":"2024-09-17T07:14:50.794200+0000","last_deep_scrub":"110'15","last_deep_scrub_stamp":"2024-09-17T07:04:57.091400+0000","last_clean_scrub_stamp":"2024-09-17T07:14:50.794200+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0070558269999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[{"shards":"4,5","objects":5}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":62,"reported_epoch":355,"state":"active+undersized","last_fresh":"2024-09-17T07:18:52.411259+0000","last_change":"2024-09-17T07:16:17.378414+0000","last_active":"2024-09-17T07:18:52.411259+0000","last_peered":"2024-09-17T07:18:52.411259+0000","last_clean":"2024-09-17T07:02:10.991363+0000","last_became_active":"2024-09-17T07:16:17.378414+0000","last_became_peered":"2024-09-17T07:16:17.378414+0000","last_unstale":"2024-09-17T07:18:52.411259+0000","last_undegraded":"2024-09-17T07:18:52.411259+0000","last_fullsized":"2024-09-17T07:16:17.365679+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:49.764885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:25.147878+0000","last_clean_scrub_stamp":"2024-09-17T07:14:49.764885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"89'9","reported_seq":586,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-17T07:18:52.408319+0000","last_change":"2024-09-17T07:17:41.595535+0000","last_active":"2024-09-17T07:18:52.408319+0000","last_peered":"2024-09-17T07:18:52.408319+0000","last_clean":"2024-09-17T07:18:52.408319+0000","last_became_active":"2024-09-17T07:02:16.021360+0000","last_became_peered":"2024-09-17T07:02:16.021360+0000","last_unstale":"2024-09-17T07:18:52.408319+0000","last_undegraded":"2024-09-17T07:18:52.408319+0000","last_fullsized":"2024-09-17T07:18:52.408319+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'9","last_scrub_stamp":"2024-09-17T07:17:41.595032+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-09-17T07:15:53.512777+0000","last_clean_scrub_stamp":"2024-09-17T07:17:41.595032+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:19:05.063868+0000","scrub_duration":0.031478313000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"268'2","reported_seq":391,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-17T07:18:52.393762+0000","last_change":"2024-09-17T07:18:49.469622+0000","last_active":"2024-09-17T07:18:52.393762+0000","last_peered":"2024-09-17T07:18:52.393762+0000","last_clean":"2024-09-17T07:18:52.393762+0000","last_became_active":"2024-09-17T07:09:24.399565+0000","last_became_peered":"2024-09-17T07:09:24.399565+0000","last_unstale":"2024-09-17T07:18:52.393762+0000","last_undegraded":"2024-09-17T07:18:52.393762+0000","last_fullsized":"2024-09-17T07:18:52.393762+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'2","last_scrub_stamp":"2024-09-17T07:18:49.469494+0000","last_deep_scrub":"268'2","last_deep_scrub_stamp":"2024-09-17T07:14:03.609131+0000","last_clean_scrub_stamp":"2024-09-17T07:18:49.469494+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:20:01.585713+0000","scrub_duration":0.0037058479999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.a","version":"355'12243","reported_seq":12308,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-17T07:18:53.749101+0000","last_change":"2024-09-17T07:18:35.037259+0000","last_active":"2024-09-17T07:18:53.749101+0000","last_peered":"2024-09-17T07:18:53.749101+0000","last_clean":"2024-09-17T07:18:53.749101+0000","last_became_active":"2024-09-17T07:16:05.055008+0000","last_became_peered":"2024-09-17T07:16:05.055008+0000","last_unstale":"2024-09-17T07:18:53.749101+0000","last_undegraded":"2024-09-17T07:18:53.749101+0000","last_fullsized":"2024-09-17T07:18:53.749101+0000","mapping_epoch":297,"log_start":"331'7750","ondisk_log_start":"331'7750","created":297,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":4493,"log_dups_size":0,"ondisk_log_size":4493,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":206110720,"num_objects":3145,"num_object_clones":0,"num_object_copies":9435,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3145,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12243,"num_write_kb":492416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"304'42","reported_seq":538,"reported_epoch":354,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:18:50.714660+0000","last_change":"2024-09-17T07:16:17.368142+0000","last_active":"2024-09-17T07:18:50.714660+0000","last_peered":"2024-09-17T07:18:50.714660+0000","last_clean":"2024-09-17T07:12:20.666266+0000","last_became_active":"2024-09-17T07:16:17.368142+0000","last_became_peered":"2024-09-17T07:16:17.368142+0000","last_unstale":"2024-09-17T07:18:50.714660+0000","last_undegraded":"2024-09-17T07:16:17.363645+0000","last_fullsized":"2024-09-17T07:16:17.363541+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'41","last_scrub_stamp":"2024-09-17T07:14:56.700858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:14:56.700858+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026058610000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[{"shards":"2,6","objects":10}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":375,"reported_epoch":355,"state":"active+undersized","last_fresh":"2024-09-17T07:18:52.409664+0000","last_change":"2024-09-17T07:16:17.371561+0000","last_active":"2024-09-17T07:18:52.409664+0000","last_peered":"2024-09-17T07:18:52.409664+0000","last_clean":"2024-09-17T07:16:16.170577+0000","last_became_active":"2024-09-17T07:16:17.371561+0000","last_became_peered":"2024-09-17T07:16:17.371561+0000","last_unstale":"2024-09-17T07:18:52.409664+0000","last_undegraded":"2024-09-17T07:18:52.409664+0000","last_fullsized":"2024-09-17T07:16:17.366918+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:03.045709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:14:03.045709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0049245929999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":372,"reported_epoch":355,"state":"active+undersized","last_fresh":"2024-09-17T07:18:52.408809+0000","last_change":"2024-09-17T07:16:17.375376+0000","last_active":"2024-09-17T07:18:52.408809+0000","last_peered":"2024-09-17T07:18:52.408809+0000","last_clean":"2024-09-17T07:16:15.495109+0000","last_became_active":"2024-09-17T07:16:17.375376+0000","last_became_peered":"2024-09-17T07:16:17.375376+0000","last_unstale":"2024-09-17T07:18:52.408809+0000","last_undegraded":"2024-09-17T07:18:52.408809+0000","last_fullsized":"2024-09-17T07:16:17.364236+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:36.523676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:15:36.523676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017370549999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":220,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-17T07:18:42.340210+0000","last_change":"2024-09-17T07:17:18.312624+0000","last_active":"2024-09-17T07:18:42.340210+0000","last_peered":"2024-09-17T07:18:42.340210+0000","last_clean":"2024-09-17T07:18:42.340210+0000","last_became_active":"2024-09-17T07:09:24.398878+0000","last_became_peered":"2024-09-17T07:09:24.398878+0000","last_unstale":"2024-09-17T07:18:42.340210+0000","last_undegraded":"2024-09-17T07:18:42.340210+0000","last_fullsized":"2024-09-17T07:18:42.340210+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:17:18.312499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:13:53.406661+0000","last_clean_scrub_stamp":"2024-09-17T07:17:18.312499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0030147720000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.9","version":"355'10014","reported_seq":10277,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-17T07:18:53.883002+0000","last_change":"2024-09-17T07:18:46.761729+0000","last_active":"2024-09-17T07:18:53.883002+0000","last_peered":"2024-09-17T07:18:53.883002+0000","last_clean":"2024-09-17T07:18:53.883002+0000","last_became_active":"2024-09-17T07:17:52.386125+0000","last_became_peered":"2024-09-17T07:17:52.386125+0000","last_unstale":"2024-09-17T07:18:53.883002+0000","last_undegraded":"2024-09-17T07:18:53.883002+0000","last_fullsized":"2024-09-17T07:18:53.883002+0000","mapping_epoch":297,"log_start":"312'3084","ondisk_log_start":"312'3084","created":297,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":0,"last_scrub":"354'9713","last_scrub_stamp":"2024-09-17T07:18:46.761585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:18:46.761585+0000","objects_scrubbed":1810,"log_size":6930,"log_dups_size":0,"ondisk_log_size":6930,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:20:09.045403+0000","scrub_duration":0.36938599100000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98500608,"num_objects":1503,"num_object_clones":0,"num_object_copies":4509,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":307,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"242'17","reported_seq":320,"reported_epoch":354,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:18:42.369966+0000","last_change":"2024-09-17T07:16:17.371977+0000","last_active":"2024-09-17T07:18:42.369966+0000","last_peered":"2024-09-17T07:18:42.369966+0000","last_clean":"2024-09-17T07:12:20.284331+0000","last_became_active":"2024-09-17T07:16:17.371977+0000","last_became_peered":"2024-09-17T07:16:17.371977+0000","last_unstale":"2024-09-17T07:18:42.369966+0000","last_undegraded":"2024-09-17T07:16:17.367143+0000","last_fullsized":"2024-09-17T07:16:17.367038+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'17","last_scrub_stamp":"2024-09-17T07:13:47.468399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:13:47.468399+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.005881725,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[{"shards":"0,5","objects":5}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"76'5","reported_seq":246,"reported_epoch":355,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:18:52.407015+0000","last_change":"2024-09-17T07:16:17.368338+0000","last_active":"2024-09-17T07:18:52.407015+0000","last_peered":"2024-09-17T07:18:52.407015+0000","last_clean":"2024-09-17T07:02:23.985427+0000","last_became_active":"2024-09-17T07:16:17.368338+0000","last_became_peered":"2024-09-17T07:16:17.368338+0000","last_unstale":"2024-09-17T07:18:52.407015+0000","last_undegraded":"2024-09-17T07:16:17.363185+0000","last_fullsized":"2024-09-17T07:16:17.363086+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'5","last_scrub_stamp":"2024-09-17T07:15:36.901601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:15:36.901601+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033739799999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":162,"num_read_kb":157,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[{"shards":"2,6","objects":1}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":217,"reported_epoch":355,"state":"active+undersized","last_fresh":"2024-09-17T07:18:52.407229+0000","last_change":"2024-09-17T07:16:17.367238+0000","last_active":"2024-09-17T07:18:52.407229+0000","last_peered":"2024-09-17T07:18:52.407229+0000","last_clean":"2024-09-17T07:16:15.494771+0000","last_became_active":"2024-09-17T07:16:17.367238+0000","last_became_peered":"2024-09-17T07:16:17.367238+0000","last_unstale":"2024-09-17T07:18:52.407229+0000","last_undegraded":"2024-09-17T07:18:52.407229+0000","last_fullsized":"2024-09-17T07:16:17.364334+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:56.534547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:23.833170+0000","last_clean_scrub_stamp":"2024-09-17T07:15:56.534547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0011018950000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":399,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-17T07:18:42.338467+0000","last_change":"2024-09-17T07:18:42.051741+0000","last_active":"2024-09-17T07:18:42.338467+0000","last_peered":"2024-09-17T07:18:42.338467+0000","last_clean":"2024-09-17T07:18:42.338467+0000","last_became_active":"2024-09-17T07:02:16.014492+0000","last_became_peered":"2024-09-17T07:02:16.014492+0000","last_unstale":"2024-09-17T07:18:42.338467+0000","last_undegraded":"2024-09-17T07:18:42.338467+0000","last_fullsized":"2024-09-17T07:18:42.338467+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:18:42.051667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:18:42.051667+0000","last_clean_scrub_stamp":"2024-09-17T07:18:42.051667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:19:51.231165+0000","scrub_duration":0.0011360980000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":372,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-17T07:18:42.338349+0000","last_change":"2024-09-17T07:18:39.328796+0000","last_active":"2024-09-17T07:18:42.338349+0000","last_peered":"2024-09-17T07:18:42.338349+0000","last_clean":"2024-09-17T07:18:42.338349+0000","last_became_active":"2024-09-17T07:09:24.402704+0000","last_became_peered":"2024-09-17T07:09:24.402704+0000","last_unstale":"2024-09-17T07:18:42.338349+0000","last_undegraded":"2024-09-17T07:18:42.338349+0000","last_fullsized":"2024-09-17T07:18:42.338349+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:02.062618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:02.062618+0000","last_clean_scrub_stamp":"2024-09-17T07:15:02.062618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0042833460000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"356'10121","reported_seq":10250,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-17T07:18:54.566974+0000","last_change":"2024-09-17T07:18:43.933340+0000","last_active":"2024-09-17T07:18:54.566974+0000","last_peered":"2024-09-17T07:18:54.566974+0000","last_clean":"2024-09-17T07:18:54.566974+0000","last_became_active":"2024-09-17T07:17:52.372680+0000","last_became_peered":"2024-09-17T07:17:52.372680+0000","last_unstale":"2024-09-17T07:18:54.566974+0000","last_undegraded":"2024-09-17T07:18:54.566974+0000","last_fullsized":"2024-09-17T07:18:54.566974+0000","mapping_epoch":297,"log_start":"313'3970","ondisk_log_start":"313'3970","created":297,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":0,"last_scrub":"354'9680","last_scrub_stamp":"2024-09-17T07:18:43.933160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:18:43.933160+0000","objects_scrubbed":1923,"log_size":6151,"log_dups_size":0,"ondisk_log_size":6151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:19:58.555529+0000","scrub_duration":0.45499974300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96665600,"num_objects":1475,"num_object_clones":0,"num_object_copies":4425,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":448,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"268'30","reported_seq":444,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-17T07:18:52.407379+0000","last_change":"2024-09-17T07:18:40.426710+0000","last_active":"2024-09-17T07:18:52.407379+0000","last_peered":"2024-09-17T07:18:52.407379+0000","last_clean":"2024-09-17T07:18:52.407379+0000","last_became_active":"2024-09-17T07:03:48.760405+0000","last_became_peered":"2024-09-17T07:03:48.760405+0000","last_unstale":"2024-09-17T07:18:52.407379+0000","last_undegraded":"2024-09-17T07:18:52.407379+0000","last_fullsized":"2024-09-17T07:18:52.407379+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'30","last_scrub_stamp":"2024-09-17T07:18:40.426597+0000","last_deep_scrub":"268'30","last_deep_scrub_stamp":"2024-09-17T07:14:00.599656+0000","last_clean_scrub_stamp":"2024-09-17T07:18:40.426597+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:19:44.097951+0000","scrub_duration":0.0031470769999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"70'1","reported_seq":390,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-17T07:18:54.404942+0000","last_change":"2024-09-17T07:17:26.441837+0000","last_active":"2024-09-17T07:18:54.404942+0000","last_peered":"2024-09-17T07:18:54.404942+0000","last_clean":"2024-09-17T07:18:54.404942+0000","last_became_active":"2024-09-17T07:09:24.402499+0000","last_became_peered":"2024-09-17T07:09:24.402499+0000","last_unstale":"2024-09-17T07:18:54.404942+0000","last_undegraded":"2024-09-17T07:18:54.404942+0000","last_fullsized":"2024-09-17T07:18:54.404942+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-17T07:17:26.441670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:17:26.441670+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0054314109999999997,"objects_trimmed":0,"snaptrim_duration":0.0032298629999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":364,"reported_epoch":354,"state":"active+undersized","last_fresh":"2024-09-17T07:18:42.337863+0000","last_change":"2024-09-17T07:16:17.375018+0000","last_active":"2024-09-17T07:18:42.337863+0000","last_peered":"2024-09-17T07:18:42.337863+0000","last_clean":"2024-09-17T07:16:15.590088+0000","last_became_active":"2024-09-17T07:16:17.375018+0000","last_became_peered":"2024-09-17T07:16:17.375018+0000","last_unstale":"2024-09-17T07:18:42.337863+0000","last_undegraded":"2024-09-17T07:18:42.337863+0000","last_fullsized":"2024-09-17T07:16:17.370688+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:55.321202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:55.321202+0000","last_clean_scrub_stamp":"2024-09-17T07:15:55.321202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014716950000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":370,"reported_epoch":354,"state":"active+undersized","last_fresh":"2024-09-17T07:18:42.369594+0000","last_change":"2024-09-17T07:16:17.375774+0000","last_active":"2024-09-17T07:18:42.369594+0000","last_peered":"2024-09-17T07:18:42.369594+0000","last_clean":"2024-09-17T07:16:16.169907+0000","last_became_active":"2024-09-17T07:16:17.375774+0000","last_became_peered":"2024-09-17T07:16:17.375774+0000","last_unstale":"2024-09-17T07:18:42.369594+0000","last_undegraded":"2024-09-17T07:18:42.369594+0000","last_fullsized":"2024-09-17T07:16:17.368453+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:45.447029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:15:45.447029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001990463,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":227,"reported_epoch":354,"state":"active+undersized","last_fresh":"2024-09-17T07:18:42.365729+0000","last_change":"2024-09-17T07:16:17.372859+0000","last_active":"2024-09-17T07:18:42.365729+0000","last_peered":"2024-09-17T07:18:42.365729+0000","last_clean":"2024-09-17T07:09:22.371903+0000","last_became_active":"2024-09-17T07:16:17.372859+0000","last_became_peered":"2024-09-17T07:16:17.372859+0000","last_unstale":"2024-09-17T07:18:42.365729+0000","last_undegraded":"2024-09-17T07:18:42.365729+0000","last_fullsized":"2024-09-17T07:16:17.367296+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:51.877367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:15:51.877367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.058290885000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.b","version":"355'12378","reported_seq":12451,"reported_epoch":355,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:18:53.711628+0000","last_change":"2024-09-17T07:16:18.636164+0000","last_active":"2024-09-17T07:18:53.711628+0000","last_peered":"2024-09-17T07:18:53.711628+0000","last_clean":"2024-09-17T07:16:14.246161+0000","last_became_active":"2024-09-17T07:16:17.365889+0000","last_became_peered":"2024-09-17T07:16:17.365889+0000","last_unstale":"2024-09-17T07:18:53.711628+0000","last_undegraded":"2024-09-17T07:16:18.628529+0000","last_fullsized":"2024-09-17T07:16:17.362973+0000","mapping_epoch":303,"log_start":"337'7957","ondisk_log_start":"337'7957","created":297,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":4421,"log_dups_size":0,"ondisk_log_size":4421,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197263360,"num_objects":3010,"num_object_clones":0,"num_object_copies":9030,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3010,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12378,"num_write_kb":492416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[{"shards":"0,6","objects":3010}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"242'26","reported_seq":419,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-17T07:18:42.361139+0000","last_change":"2024-09-17T07:18:37.273399+0000","last_active":"2024-09-17T07:18:42.361139+0000","last_peered":"2024-09-17T07:18:42.361139+0000","last_clean":"2024-09-17T07:18:42.361139+0000","last_became_active":"2024-09-17T07:02:14.028857+0000","last_became_peered":"2024-09-17T07:02:14.028857+0000","last_unstale":"2024-09-17T07:18:42.361139+0000","last_undegraded":"2024-09-17T07:18:42.361139+0000","last_fullsized":"2024-09-17T07:18:42.361139+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'26","last_scrub_stamp":"2024-09-17T07:16:33.666684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:16:33.666684+0000","objects_scrubbed":7,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.013353596000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"268'3","reported_seq":239,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-17T07:18:42.369735+0000","last_change":"2024-09-17T07:17:19.453842+0000","last_active":"2024-09-17T07:18:42.369735+0000","last_peered":"2024-09-17T07:18:42.369735+0000","last_clean":"2024-09-17T07:18:42.369735+0000","last_became_active":"2024-09-17T07:09:24.404532+0000","last_became_peered":"2024-09-17T07:09:24.404532+0000","last_unstale":"2024-09-17T07:18:42.369735+0000","last_undegraded":"2024-09-17T07:18:42.369735+0000","last_fullsized":"2024-09-17T07:18:42.369735+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'3","last_scrub_stamp":"2024-09-17T07:17:19.444444+0000","last_deep_scrub":"268'3","last_deep_scrub_stamp":"2024-09-17T07:13:54.949345+0000","last_clean_scrub_stamp":"2024-09-17T07:17:19.444444+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0048257380000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":391,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-17T07:18:52.408938+0000","last_change":"2024-09-17T07:17:34.639287+0000","last_active":"2024-09-17T07:18:52.408938+0000","last_peered":"2024-09-17T07:18:52.408938+0000","last_clean":"2024-09-17T07:18:52.408938+0000","last_became_active":"2024-09-17T07:02:12.004606+0000","last_became_peered":"2024-09-17T07:02:12.004606+0000","last_unstale":"2024-09-17T07:18:52.408938+0000","last_undegraded":"2024-09-17T07:18:52.408938+0000","last_fullsized":"2024-09-17T07:18:52.408938+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:17:34.638733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:17:34.638733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:56.860917+0000","scrub_duration":0.064159545999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"212'130","reported_seq":620,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-17T07:18:42.337911+0000","last_change":"2024-09-17T07:18:35.277546+0000","last_active":"2024-09-17T07:18:42.337911+0000","last_peered":"2024-09-17T07:18:42.337911+0000","last_clean":"2024-09-17T07:18:42.337911+0000","last_became_active":"2024-09-17T07:02:03.020675+0000","last_became_peered":"2024-09-17T07:02:03.020675+0000","last_unstale":"2024-09-17T07:18:42.337911+0000","last_undegraded":"2024-09-17T07:18:42.337911+0000","last_fullsized":"2024-09-17T07:18:42.337911+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:14:57.052505+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:14:57.052505+0000","last_clean_scrub_stamp":"2024-09-17T07:14:57.052505+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.022122466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":375,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-17T07:18:48.891849+0000","last_change":"2024-09-17T07:18:48.891849+0000","last_active":"2024-09-17T07:18:48.891849+0000","last_peered":"2024-09-17T07:18:48.891849+0000","last_clean":"2024-09-17T07:18:48.891849+0000","last_became_active":"2024-09-17T07:02:16.016031+0000","last_became_peered":"2024-09-17T07:02:16.016031+0000","last_unstale":"2024-09-17T07:18:48.891849+0000","last_undegraded":"2024-09-17T07:18:48.891849+0000","last_fullsized":"2024-09-17T07:18:48.891849+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:18:48.891136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:18:48.891136+0000","last_clean_scrub_stamp":"2024-09-17T07:18:48.891136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:20:18.076925+0000","scrub_duration":0.010734751000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":366,"reported_epoch":354,"state":"active+undersized","last_fresh":"2024-09-17T07:18:42.369721+0000","last_change":"2024-09-17T07:16:17.377095+0000","last_active":"2024-09-17T07:18:42.369721+0000","last_peered":"2024-09-17T07:18:42.369721+0000","last_clean":"2024-09-17T07:16:16.170840+0000","last_became_active":"2024-09-17T07:16:17.377095+0000","last_became_peered":"2024-09-17T07:16:17.377095+0000","last_unstale":"2024-09-17T07:18:42.369721+0000","last_undegraded":"2024-09-17T07:18:42.369721+0000","last_fullsized":"2024-09-17T07:16:17.370787+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:48.021592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:58.670571+0000","last_clean_scrub_stamp":"2024-09-17T07:14:48.021592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0064844619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.4","version":"356'10074","reported_seq":10156,"reported_epoch":356,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:18:54.537701+0000","last_change":"2024-09-17T07:17:52.376990+0000","last_active":"2024-09-17T07:18:54.537701+0000","last_peered":"2024-09-17T07:18:54.537701+0000","last_clean":"2024-09-17T07:16:15.495005+0000","last_became_active":"2024-09-17T07:17:52.376990+0000","last_became_peered":"2024-09-17T07:17:52.376990+0000","last_unstale":"2024-09-17T07:18:54.537701+0000","last_undegraded":"2024-09-17T07:17:52.365057+0000","last_fullsized":"2024-09-17T07:17:52.364949+0000","mapping_epoch":303,"log_start":"313'3919","ondisk_log_start":"313'3919","created":297,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":6155,"log_dups_size":0,"ondisk_log_size":6155,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":97124352,"num_objects":1482,"num_object_clones":0,"num_object_copies":4446,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1482,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6196,"num_write_kb":245696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0,4","objects":1482}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"242'35","reported_seq":439,"reported_epoch":354,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:18:42.360688+0000","last_change":"2024-09-17T07:16:17.376246+0000","last_active":"2024-09-17T07:18:42.360688+0000","last_peered":"2024-09-17T07:18:42.360688+0000","last_clean":"2024-09-17T07:16:14.244770+0000","last_became_active":"2024-09-17T07:16:17.376246+0000","last_became_peered":"2024-09-17T07:16:17.376246+0000","last_unstale":"2024-09-17T07:18:42.360688+0000","last_undegraded":"2024-09-17T07:16:17.363365+0000","last_fullsized":"2024-09-17T07:16:17.363257+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'35","last_scrub_stamp":"2024-09-17T07:15:43.665829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:15:43.665829+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0048325160000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[{"shards":"6,7","objects":9}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":425,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-17T07:18:50.077819+0000","last_change":"2024-09-17T07:18:50.077819+0000","last_active":"2024-09-17T07:18:50.077819+0000","last_peered":"2024-09-17T07:18:50.077819+0000","last_clean":"2024-09-17T07:18:50.077819+0000","last_became_active":"2024-09-17T07:02:12.012037+0000","last_became_peered":"2024-09-17T07:02:12.012037+0000","last_unstale":"2024-09-17T07:18:50.077819+0000","last_undegraded":"2024-09-17T07:18:50.077819+0000","last_fullsized":"2024-09-17T07:18:50.077819+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:18:50.077716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:06:39.322535+0000","last_clean_scrub_stamp":"2024-09-17T07:18:50.077716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:20:14.706290+0000","scrub_duration":0.0025671019999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":409,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-17T07:18:44.061764+0000","last_change":"2024-09-17T07:18:44.061764+0000","last_active":"2024-09-17T07:18:44.061764+0000","last_peered":"2024-09-17T07:18:44.061764+0000","last_clean":"2024-09-17T07:18:44.061764+0000","last_became_active":"2024-09-17T07:02:16.014631+0000","last_became_peered":"2024-09-17T07:02:16.014631+0000","last_unstale":"2024-09-17T07:18:44.061764+0000","last_undegraded":"2024-09-17T07:18:44.061764+0000","last_fullsized":"2024-09-17T07:18:44.061764+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:18:44.061681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:18:44.061681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:19:47.840600+0000","scrub_duration":0.00146547,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":371,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-17T07:18:47.881259+0000","last_change":"2024-09-17T07:18:47.881259+0000","last_active":"2024-09-17T07:18:47.881259+0000","last_peered":"2024-09-17T07:18:47.881259+0000","last_clean":"2024-09-17T07:18:47.881259+0000","last_became_active":"2024-09-17T07:09:24.403904+0000","last_became_peered":"2024-09-17T07:09:24.403904+0000","last_unstale":"2024-09-17T07:18:47.881259+0000","last_undegraded":"2024-09-17T07:18:47.881259+0000","last_fullsized":"2024-09-17T07:18:47.881259+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:18:47.880943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:45.275473+0000","last_clean_scrub_stamp":"2024-09-17T07:18:47.880943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:19:54.108267+0000","scrub_duration":0.0039294250000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"356'10186","reported_seq":10315,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-17T07:18:54.550956+0000","last_change":"2024-09-17T07:18:41.742362+0000","last_active":"2024-09-17T07:18:54.550956+0000","last_peered":"2024-09-17T07:18:54.550956+0000","last_clean":"2024-09-17T07:18:54.550956+0000","last_became_active":"2024-09-17T07:17:52.375445+0000","last_became_peered":"2024-09-17T07:17:52.375445+0000","last_unstale":"2024-09-17T07:18:54.550956+0000","last_undegraded":"2024-09-17T07:18:54.550956+0000","last_fullsized":"2024-09-17T07:18:54.550956+0000","mapping_epoch":297,"log_start":"313'4056","ondisk_log_start":"313'4056","created":297,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":0,"last_scrub":"353'9682","last_scrub_stamp":"2024-09-17T07:18:41.742224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:18:41.742224+0000","objects_scrubbed":1968,"log_size":6130,"log_dups_size":0,"ondisk_log_size":6130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:19:57.931343+0000","scrub_duration":0.28187726499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":95551488,"num_objects":1458,"num_object_clones":0,"num_object_copies":4374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":510,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"268'19","reported_seq":392,"reported_epoch":355,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:18:52.408341+0000","last_change":"2024-09-17T07:16:17.375651+0000","last_active":"2024-09-17T07:18:52.408341+0000","last_peered":"2024-09-17T07:18:52.408341+0000","last_clean":"2024-09-17T07:16:15.494820+0000","last_became_active":"2024-09-17T07:16:17.375651+0000","last_became_peered":"2024-09-17T07:16:17.375651+0000","last_unstale":"2024-09-17T07:18:52.408341+0000","last_undegraded":"2024-09-17T07:16:17.364532+0000","last_fullsized":"2024-09-17T07:16:17.364433+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'19","last_scrub_stamp":"2024-09-17T07:14:06.634334+0000","last_deep_scrub":"268'19","last_deep_scrub_stamp":"2024-09-17T07:14:06.634334+0000","last_clean_scrub_stamp":"2024-09-17T07:14:06.634334+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0088943289999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0,4","objects":4}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":233,"reported_epoch":354,"state":"active+undersized","last_fresh":"2024-09-17T07:18:42.340468+0000","last_change":"2024-09-17T07:16:17.374110+0000","last_active":"2024-09-17T07:18:42.340468+0000","last_peered":"2024-09-17T07:18:42.340468+0000","last_clean":"2024-09-17T07:09:22.372513+0000","last_became_active":"2024-09-17T07:16:17.374110+0000","last_became_peered":"2024-09-17T07:16:17.374110+0000","last_unstale":"2024-09-17T07:18:42.340468+0000","last_undegraded":"2024-09-17T07:18:42.340468+0000","last_fullsized":"2024-09-17T07:16:17.367137+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:58.886294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:13:53.593078+0000","last_clean_scrub_stamp":"2024-09-17T07:15:58.886294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.005853245,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":340,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-17T07:18:42.339527+0000","last_change":"2024-09-17T07:16:35.152814+0000","last_active":"2024-09-17T07:18:42.339527+0000","last_peered":"2024-09-17T07:18:42.339527+0000","last_clean":"2024-09-17T07:18:42.339527+0000","last_became_active":"2024-09-17T07:09:24.396642+0000","last_became_peered":"2024-09-17T07:09:24.396642+0000","last_unstale":"2024-09-17T07:18:42.339527+0000","last_undegraded":"2024-09-17T07:18:42.339527+0000","last_fullsized":"2024-09-17T07:18:42.339527+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:35.152744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:16:35.152744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0022724160000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":364,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-17T07:18:52.407335+0000","last_change":"2024-09-17T07:17:27.482509+0000","last_active":"2024-09-17T07:18:52.407335+0000","last_peered":"2024-09-17T07:18:52.407335+0000","last_clean":"2024-09-17T07:18:52.407335+0000","last_became_active":"2024-09-17T07:02:18.026994+0000","last_became_peered":"2024-09-17T07:02:18.026994+0000","last_unstale":"2024-09-17T07:18:52.407335+0000","last_undegraded":"2024-09-17T07:18:52.407335+0000","last_fullsized":"2024-09-17T07:18:52.407335+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:17:27.482401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:58.099849+0000","last_clean_scrub_stamp":"2024-09-17T07:17:27.482401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0020589919999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.6","version":"355'10052","reported_seq":10272,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-17T07:18:53.854173+0000","last_change":"2024-09-17T07:17:52.404390+0000","last_active":"2024-09-17T07:18:53.854173+0000","last_peered":"2024-09-17T07:18:53.854173+0000","last_clean":"2024-09-17T07:18:53.854173+0000","last_became_active":"2024-09-17T07:17:52.398777+0000","last_became_peered":"2024-09-17T07:17:52.398777+0000","last_unstale":"2024-09-17T07:18:53.854173+0000","last_undegraded":"2024-09-17T07:18:53.854173+0000","last_fullsized":"2024-09-17T07:18:53.854173+0000","mapping_epoch":297,"log_start":"312'3111","ondisk_log_start":"312'3111","created":297,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":0,"last_scrub":"324'6643","last_scrub_stamp":"2024-09-17T07:17:30.776520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:17:30.776520+0000","objects_scrubbed":6552,"log_size":6941,"log_dups_size":0,"ondisk_log_size":6941,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2.27785153,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96403456,"num_objects":1471,"num_object_clones":0,"num_object_copies":4413,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1471,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6183,"num_write_kb":244928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"242'34","reported_seq":427,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-17T07:18:42.365217+0000","last_change":"2024-09-17T07:16:44.126984+0000","last_active":"2024-09-17T07:18:42.365217+0000","last_peered":"2024-09-17T07:18:42.365217+0000","last_clean":"2024-09-17T07:18:42.365217+0000","last_became_active":"2024-09-17T07:02:14.028392+0000","last_became_peered":"2024-09-17T07:02:14.028392+0000","last_unstale":"2024-09-17T07:18:42.365217+0000","last_undegraded":"2024-09-17T07:18:42.365217+0000","last_fullsized":"2024-09-17T07:18:42.365217+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'34","last_scrub_stamp":"2024-09-17T07:16:44.126830+0000","last_deep_scrub":"242'34","last_deep_scrub_stamp":"2024-09-17T07:14:50.103539+0000","last_clean_scrub_stamp":"2024-09-17T07:16:44.126830+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.008313473,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'2","reported_seq":251,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-17T07:18:43.861957+0000","last_change":"2024-09-17T07:18:43.861957+0000","last_active":"2024-09-17T07:18:43.861957+0000","last_peered":"2024-09-17T07:18:43.861957+0000","last_clean":"2024-09-17T07:18:43.861957+0000","last_became_active":"2024-09-17T07:09:24.402909+0000","last_became_peered":"2024-09-17T07:09:24.402909+0000","last_unstale":"2024-09-17T07:18:43.861957+0000","last_undegraded":"2024-09-17T07:18:43.861957+0000","last_fullsized":"2024-09-17T07:18:43.861957+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-09-17T07:18:43.861842+0000","last_deep_scrub":"89'2","last_deep_scrub_stamp":"2024-09-17T07:12:20.301263+0000","last_clean_scrub_stamp":"2024-09-17T07:18:43.861842+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:19:59.925131+0000","scrub_duration":0.0028010019999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":420,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":420,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"225'9","reported_seq":576,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-17T07:18:50.715047+0000","last_change":"2024-09-17T07:18:23.789276+0000","last_active":"2024-09-17T07:18:50.715047+0000","last_peered":"2024-09-17T07:18:50.715047+0000","last_clean":"2024-09-17T07:18:50.715047+0000","last_became_active":"2024-09-17T07:02:16.015418+0000","last_became_peered":"2024-09-17T07:02:16.015418+0000","last_unstale":"2024-09-17T07:18:50.715047+0000","last_undegraded":"2024-09-17T07:18:50.715047+0000","last_fullsized":"2024-09-17T07:18:50.715047+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"225'9","last_scrub_stamp":"2024-09-17T07:18:23.788769+0000","last_deep_scrub":"225'9","last_deep_scrub_stamp":"2024-09-17T07:11:25.292565+0000","last_clean_scrub_stamp":"2024-09-17T07:18:23.788769+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:19:43.054958+0000","scrub_duration":0.006761141,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":384,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-17T07:18:42.365895+0000","last_change":"2024-09-17T07:18:41.386243+0000","last_active":"2024-09-17T07:18:42.365895+0000","last_peered":"2024-09-17T07:18:42.365895+0000","last_clean":"2024-09-17T07:18:42.365895+0000","last_became_active":"2024-09-17T07:02:18.363253+0000","last_became_peered":"2024-09-17T07:02:18.363253+0000","last_unstale":"2024-09-17T07:18:42.365895+0000","last_undegraded":"2024-09-17T07:18:42.365895+0000","last_fullsized":"2024-09-17T07:18:42.365895+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:18:41.385928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:18:41.385928+0000","last_clean_scrub_stamp":"2024-09-17T07:18:41.385928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:20:04.219102+0000","scrub_duration":0.0023027270000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.7","version":"355'10184","reported_seq":10262,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-17T07:18:53.738366+0000","last_change":"2024-09-17T07:18:37.046901+0000","last_active":"2024-09-17T07:18:53.738366+0000","last_peered":"2024-09-17T07:18:53.738366+0000","last_clean":"2024-09-17T07:18:53.738366+0000","last_became_active":"2024-09-17T07:17:52.402213+0000","last_became_peered":"2024-09-17T07:17:52.402213+0000","last_unstale":"2024-09-17T07:18:53.738366+0000","last_undegraded":"2024-09-17T07:18:53.738366+0000","last_fullsized":"2024-09-17T07:18:53.738366+0000","mapping_epoch":297,"log_start":"312'3663","ondisk_log_start":"312'3663","created":297,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":6521,"log_dups_size":0,"ondisk_log_size":6521,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":99418112,"num_objects":1517,"num_object_clones":0,"num_object_copies":4551,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6262,"num_write_kb":248928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"242'23","reported_seq":458,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-17T07:18:42.339855+0000","last_change":"2024-09-17T07:17:34.108758+0000","last_active":"2024-09-17T07:18:42.339855+0000","last_peered":"2024-09-17T07:18:42.339855+0000","last_clean":"2024-09-17T07:18:42.339855+0000","last_became_active":"2024-09-17T07:10:49.723292+0000","last_became_peered":"2024-09-17T07:10:49.723292+0000","last_unstale":"2024-09-17T07:18:42.339855+0000","last_undegraded":"2024-09-17T07:18:42.339855+0000","last_fullsized":"2024-09-17T07:18:42.339855+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'23","last_scrub_stamp":"2024-09-17T07:17:34.108619+0000","last_deep_scrub":"150'17","last_deep_scrub_stamp":"2024-09-17T07:06:38.775788+0000","last_clean_scrub_stamp":"2024-09-17T07:17:34.108619+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:57.460011+0000","scrub_duration":0.018319322999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"68'2","reported_seq":388,"reported_epoch":354,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:18:42.340813+0000","last_change":"2024-09-17T07:16:17.371234+0000","last_active":"2024-09-17T07:18:42.340813+0000","last_peered":"2024-09-17T07:18:42.340813+0000","last_clean":"2024-09-17T07:16:15.358653+0000","last_became_active":"2024-09-17T07:16:17.371234+0000","last_became_peered":"2024-09-17T07:16:17.371234+0000","last_unstale":"2024-09-17T07:18:42.340813+0000","last_undegraded":"2024-09-17T07:16:17.366653+0000","last_fullsized":"2024-09-17T07:16:17.366079+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-09-17T07:14:56.533798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:14:56.533798+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003207115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[{"shards":"0,4","objects":2}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":336,"reported_epoch":355,"state":"active+undersized","last_fresh":"2024-09-17T07:18:52.406149+0000","last_change":"2024-09-17T07:16:17.367290+0000","last_active":"2024-09-17T07:18:52.406149+0000","last_peered":"2024-09-17T07:18:52.406149+0000","last_clean":"2024-09-17T07:16:15.494380+0000","last_became_active":"2024-09-17T07:16:17.367290+0000","last_became_peered":"2024-09-17T07:16:17.367290+0000","last_unstale":"2024-09-17T07:18:52.406149+0000","last_undegraded":"2024-09-17T07:18:52.406149+0000","last_fullsized":"2024-09-17T07:16:17.363484+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:51.586929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:11:17.479036+0000","last_clean_scrub_stamp":"2024-09-17T07:15:51.586929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001274435,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":204,"reported_epoch":354,"state":"active+undersized","last_fresh":"2024-09-17T07:18:42.338037+0000","last_change":"2024-09-17T07:16:17.379791+0000","last_active":"2024-09-17T07:18:42.338037+0000","last_peered":"2024-09-17T07:18:42.338037+0000","last_clean":"2024-09-17T07:16:15.590199+0000","last_became_active":"2024-09-17T07:16:17.379791+0000","last_became_peered":"2024-09-17T07:16:17.379791+0000","last_unstale":"2024-09-17T07:18:42.338037+0000","last_undegraded":"2024-09-17T07:18:42.338037+0000","last_fullsized":"2024-09-17T07:16:17.372333+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:13.937979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:13:13.937979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.018487173999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.18","version":"355'10072","reported_seq":16978,"reported_epoch":355,"state":"active+clean+remapped","last_fresh":"2024-09-17T07:18:52.986823+0000","last_change":"2024-09-17T07:18:49.456041+0000","last_active":"2024-09-17T07:18:52.986823+0000","last_peered":"2024-09-17T07:18:52.986823+0000","last_clean":"2024-09-17T07:18:52.986823+0000","last_became_active":"2024-09-17T07:18:18.281655+0000","last_became_peered":"2024-09-17T07:18:18.281655+0000","last_unstale":"2024-09-17T07:18:52.986823+0000","last_undegraded":"2024-09-17T07:18:52.986823+0000","last_fullsized":"2024-09-17T07:18:52.986823+0000","mapping_epoch":343,"log_start":"312'3426","ondisk_log_start":"312'3426","created":332,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":5,"last_scrub":"354'9921","last_scrub_stamp":"2024-09-17T07:18:49.455799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:18:49.455799+0000","objects_scrubbed":1720,"log_size":6646,"log_dups_size":0,"ondisk_log_size":6646,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:20:17.992458+0000","scrub_duration":0.42320338699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102498304,"num_objects":1564,"num_object_clones":0,"num_object_copies":4692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1564,"num_objects_unfound":0,"num_objects_dirty":1564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":156,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6,0],"avail_no_missing":["2","0","6"],"object_location_counts":[{"shards":"0,2,6","objects":1564}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"242'15","reported_seq":422,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-17T07:18:52.407297+0000","last_change":"2024-09-17T07:18:24.505103+0000","last_active":"2024-09-17T07:18:52.407297+0000","last_peered":"2024-09-17T07:18:52.407297+0000","last_clean":"2024-09-17T07:18:52.407297+0000","last_became_active":"2024-09-17T07:09:24.408875+0000","last_became_peered":"2024-09-17T07:09:24.408875+0000","last_unstale":"2024-09-17T07:18:52.407297+0000","last_undegraded":"2024-09-17T07:18:52.407297+0000","last_fullsized":"2024-09-17T07:18:52.407297+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'15","last_scrub_stamp":"2024-09-17T07:18:24.505002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:18:24.505002+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:19:35.852600+0000","scrub_duration":0.002195809,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":398,"reported_epoch":354,"state":"active+undersized","last_fresh":"2024-09-17T07:18:42.338428+0000","last_change":"2024-09-17T07:16:17.367792+0000","last_active":"2024-09-17T07:18:42.338428+0000","last_peered":"2024-09-17T07:18:42.338428+0000","last_clean":"2024-09-17T07:16:14.244722+0000","last_became_active":"2024-09-17T07:16:17.367792+0000","last_became_peered":"2024-09-17T07:16:17.367792+0000","last_unstale":"2024-09-17T07:18:42.338428+0000","last_undegraded":"2024-09-17T07:18:42.338428+0000","last_fullsized":"2024-09-17T07:16:17.366356+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:39.634698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:17.976267+0000","last_clean_scrub_stamp":"2024-09-17T07:15:39.634698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00117202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":63,"reported_epoch":354,"state":"active+undersized","last_fresh":"2024-09-17T07:18:43.025205+0000","last_change":"2024-09-17T07:16:17.378272+0000","last_active":"2024-09-17T07:18:43.025205+0000","last_peered":"2024-09-17T07:18:43.025205+0000","last_clean":"2024-09-17T07:02:14.999435+0000","last_became_active":"2024-09-17T07:16:17.378272+0000","last_became_peered":"2024-09-17T07:16:17.378272+0000","last_unstale":"2024-09-17T07:18:43.025205+0000","last_undegraded":"2024-09-17T07:18:43.025205+0000","last_fullsized":"2024-09-17T07:16:17.373119+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:40.845816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:18.367486+0000","last_clean_scrub_stamp":"2024-09-17T07:15:40.845816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":419,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-17T07:18:43.031562+0000","last_change":"2024-09-17T07:17:37.827552+0000","last_active":"2024-09-17T07:18:43.031562+0000","last_peered":"2024-09-17T07:18:43.031562+0000","last_clean":"2024-09-17T07:18:43.031562+0000","last_became_active":"2024-09-17T07:02:18.017825+0000","last_became_peered":"2024-09-17T07:02:18.017825+0000","last_unstale":"2024-09-17T07:18:43.031562+0000","last_undegraded":"2024-09-17T07:18:43.031562+0000","last_fullsized":"2024-09-17T07:18:43.031562+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:17:37.827296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:09:23.378805+0000","last_clean_scrub_stamp":"2024-09-17T07:17:37.827296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:19:02.444738+0000","scrub_duration":0.0050232260000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"355'9926","reported_seq":17022,"reported_epoch":355,"state":"active+clean+remapped","last_fresh":"2024-09-17T07:18:53.229174+0000","last_change":"2024-09-17T07:18:42.357433+0000","last_active":"2024-09-17T07:18:53.229174+0000","last_peered":"2024-09-17T07:18:53.229174+0000","last_clean":"2024-09-17T07:18:53.229174+0000","last_became_active":"2024-09-17T07:18:42.341643+0000","last_became_peered":"2024-09-17T07:18:42.341643+0000","last_unstale":"2024-09-17T07:18:53.229174+0000","last_undegraded":"2024-09-17T07:18:53.229174+0000","last_fullsized":"2024-09-17T07:18:53.229174+0000","mapping_epoch":353,"log_start":"313'3851","ondisk_log_start":"313'3851","created":332,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"322'6203","last_scrub_stamp":"2024-09-17T07:17:25.670907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:17:25.670907+0000","objects_scrubbed":6134,"log_size":6075,"log_dups_size":0,"ondisk_log_size":6075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2.260616706,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102596608,"num_objects":1566,"num_object_clones":0,"num_object_copies":4698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1566,"num_objects_unfound":0,"num_objects_dirty":1566,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6073,"num_write_kb":244416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2286,"num_bytes_recovered":149815296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0,2],"avail_no_missing":["4","0","2"],"object_location_counts":[{"shards":"0,2,4","objects":1566}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":386,"reported_epoch":354,"state":"active+undersized","last_fresh":"2024-09-17T07:18:42.360670+0000","last_change":"2024-09-17T07:16:17.375999+0000","last_active":"2024-09-17T07:18:42.360670+0000","last_peered":"2024-09-17T07:18:42.360670+0000","last_clean":"2024-09-17T07:16:14.245868+0000","last_became_active":"2024-09-17T07:16:17.375999+0000","last_became_peered":"2024-09-17T07:16:17.375999+0000","last_unstale":"2024-09-17T07:18:42.360670+0000","last_undegraded":"2024-09-17T07:18:42.360670+0000","last_fullsized":"2024-09-17T07:16:17.363383+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:46.645080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:46.645080+0000","last_clean_scrub_stamp":"2024-09-17T07:15:46.645080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015661049999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"242'12","reported_seq":240,"reported_epoch":354,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:18:42.338655+0000","last_change":"2024-09-17T07:16:17.379665+0000","last_active":"2024-09-17T07:18:42.338655+0000","last_peered":"2024-09-17T07:18:42.338655+0000","last_clean":"2024-09-17T07:16:15.590425+0000","last_became_active":"2024-09-17T07:16:17.379665+0000","last_became_peered":"2024-09-17T07:16:17.379665+0000","last_unstale":"2024-09-17T07:18:42.338655+0000","last_undegraded":"2024-09-17T07:16:17.373659+0000","last_fullsized":"2024-09-17T07:16:17.373552+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'12","last_scrub_stamp":"2024-09-17T07:15:44.249858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:15:44.249858+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0022006600000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":3}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":378,"reported_epoch":354,"state":"active+undersized","last_fresh":"2024-09-17T07:18:42.369647+0000","last_change":"2024-09-17T07:16:17.377495+0000","last_active":"2024-09-17T07:18:42.369647+0000","last_peered":"2024-09-17T07:18:42.369647+0000","last_clean":"2024-09-17T07:16:16.170775+0000","last_became_active":"2024-09-17T07:16:17.377495+0000","last_became_peered":"2024-09-17T07:16:17.377495+0000","last_unstale":"2024-09-17T07:18:42.369647+0000","last_undegraded":"2024-09-17T07:18:42.369647+0000","last_fullsized":"2024-09-17T07:16:17.367529+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:55.191252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:14:55.191252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0090150360000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":351,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-17T07:18:42.338714+0000","last_change":"2024-09-17T07:18:38.319273+0000","last_active":"2024-09-17T07:18:42.338714+0000","last_peered":"2024-09-17T07:18:42.338714+0000","last_clean":"2024-09-17T07:18:42.338714+0000","last_became_active":"2024-09-17T07:02:18.365419+0000","last_became_peered":"2024-09-17T07:02:18.365419+0000","last_unstale":"2024-09-17T07:18:42.338714+0000","last_undegraded":"2024-09-17T07:18:42.338714+0000","last_fullsized":"2024-09-17T07:18:42.338714+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:52.058847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:14:52.058847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0054926549999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"242'20","reported_seq":184,"reported_epoch":354,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:18:43.025057+0000","last_change":"2024-09-17T07:16:17.380213+0000","last_active":"2024-09-17T07:18:43.025057+0000","last_peered":"2024-09-17T07:18:43.025057+0000","last_clean":"2024-09-17T07:12:20.630919+0000","last_became_active":"2024-09-17T07:16:17.380213+0000","last_became_peered":"2024-09-17T07:16:17.380213+0000","last_unstale":"2024-09-17T07:18:43.025057+0000","last_undegraded":"2024-09-17T07:16:17.375354+0000","last_fullsized":"2024-09-17T07:16:17.375256+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'20","last_scrub_stamp":"2024-09-17T07:14:05.528769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:14:05.528769+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0020069329999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":5}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":399,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-17T07:18:42.369835+0000","last_change":"2024-09-17T07:17:20.425482+0000","last_active":"2024-09-17T07:18:42.369835+0000","last_peered":"2024-09-17T07:18:42.369835+0000","last_clean":"2024-09-17T07:18:42.369835+0000","last_became_active":"2024-09-17T07:09:24.402355+0000","last_became_peered":"2024-09-17T07:09:24.402355+0000","last_unstale":"2024-09-17T07:18:42.369835+0000","last_undegraded":"2024-09-17T07:18:42.369835+0000","last_fullsized":"2024-09-17T07:18:42.369835+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:17:20.425212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:17:20.425212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0020024069999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"304'10","reported_seq":579,"reported_epoch":354,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:18:50.714584+0000","last_change":"2024-09-17T07:16:17.373566+0000","last_active":"2024-09-17T07:18:50.714584+0000","last_peered":"2024-09-17T07:18:50.714584+0000","last_clean":"2024-09-17T07:16:16.169875+0000","last_became_active":"2024-09-17T07:16:17.373566+0000","last_became_peered":"2024-09-17T07:16:17.373566+0000","last_unstale":"2024-09-17T07:18:50.714584+0000","last_undegraded":"2024-09-17T07:16:17.367771+0000","last_fullsized":"2024-09-17T07:16:17.367624+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'9","last_scrub_stamp":"2024-09-17T07:14:53.140133+0000","last_deep_scrub":"163'9","last_deep_scrub_stamp":"2024-09-17T07:07:34.933808+0000","last_clean_scrub_stamp":"2024-09-17T07:14:53.140133+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0088819989999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[{"shards":"0,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":383,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-17T07:18:52.407072+0000","last_change":"2024-09-17T07:18:47.392268+0000","last_active":"2024-09-17T07:18:52.407072+0000","last_peered":"2024-09-17T07:18:52.407072+0000","last_clean":"2024-09-17T07:18:52.407072+0000","last_became_active":"2024-09-17T07:09:24.409051+0000","last_became_peered":"2024-09-17T07:09:24.409051+0000","last_unstale":"2024-09-17T07:18:52.407072+0000","last_undegraded":"2024-09-17T07:18:52.407072+0000","last_fullsized":"2024-09-17T07:18:52.407072+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:18:47.392182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:26.619893+0000","last_clean_scrub_stamp":"2024-09-17T07:18:47.392182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:20:03.629550+0000","scrub_duration":0.0053055120000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"242'15","reported_seq":420,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-17T07:18:42.369443+0000","last_change":"2024-09-17T07:16:34.100665+0000","last_active":"2024-09-17T07:18:42.369443+0000","last_peered":"2024-09-17T07:18:42.369443+0000","last_clean":"2024-09-17T07:18:42.369443+0000","last_became_active":"2024-09-17T07:09:24.406333+0000","last_became_peered":"2024-09-17T07:09:24.406333+0000","last_unstale":"2024-09-17T07:18:42.369443+0000","last_undegraded":"2024-09-17T07:18:42.369443+0000","last_fullsized":"2024-09-17T07:18:42.369443+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'15","last_scrub_stamp":"2024-09-17T07:16:34.100540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:16:34.100540+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033371669999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":383,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-17T07:18:52.408473+0000","last_change":"2024-09-17T07:17:18.561114+0000","last_active":"2024-09-17T07:18:52.408473+0000","last_peered":"2024-09-17T07:18:52.408473+0000","last_clean":"2024-09-17T07:18:52.408473+0000","last_became_active":"2024-09-17T07:09:24.402704+0000","last_became_peered":"2024-09-17T07:09:24.402704+0000","last_unstale":"2024-09-17T07:18:52.408473+0000","last_undegraded":"2024-09-17T07:18:52.408473+0000","last_fullsized":"2024-09-17T07:18:52.408473+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:17:18.560995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:17:18.560995+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0064484570000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"304'10","reported_seq":343,"reported_epoch":355,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:18:52.408320+0000","last_change":"2024-09-17T07:16:17.380279+0000","last_active":"2024-09-17T07:18:52.408320+0000","last_peered":"2024-09-17T07:18:52.408320+0000","last_clean":"2024-09-17T07:16:15.687380+0000","last_became_active":"2024-09-17T07:16:17.380279+0000","last_became_peered":"2024-09-17T07:16:17.380279+0000","last_unstale":"2024-09-17T07:18:52.408320+0000","last_undegraded":"2024-09-17T07:16:17.375037+0000","last_fullsized":"2024-09-17T07:16:17.374926+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'9","last_scrub_stamp":"2024-09-17T07:13:10.965855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:13:10.965855+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0088433720000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":374,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-17T07:18:43.122873+0000","last_change":"2024-09-17T07:18:43.122873+0000","last_active":"2024-09-17T07:18:43.122873+0000","last_peered":"2024-09-17T07:18:43.122873+0000","last_clean":"2024-09-17T07:18:43.122873+0000","last_became_active":"2024-09-17T07:02:18.018716+0000","last_became_peered":"2024-09-17T07:02:18.018716+0000","last_unstale":"2024-09-17T07:18:43.122873+0000","last_undegraded":"2024-09-17T07:18:43.122873+0000","last_fullsized":"2024-09-17T07:18:43.122873+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:18:43.121570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:18:43.121570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:19:44.821007+0000","scrub_duration":0.010220139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"242'15","reported_seq":312,"reported_epoch":354,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:18:42.360471+0000","last_change":"2024-09-17T07:16:17.368982+0000","last_active":"2024-09-17T07:18:42.360471+0000","last_peered":"2024-09-17T07:18:42.360471+0000","last_clean":"2024-09-17T07:12:20.132949+0000","last_became_active":"2024-09-17T07:16:17.368982+0000","last_became_peered":"2024-09-17T07:16:17.368982+0000","last_unstale":"2024-09-17T07:18:42.360471+0000","last_undegraded":"2024-09-17T07:16:17.363353+0000","last_fullsized":"2024-09-17T07:16:17.363248+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'15","last_scrub_stamp":"2024-09-17T07:13:48.500257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:13:48.500257+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.005053564,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[{"shards":"2,6","objects":4}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":399,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-17T07:18:42.361037+0000","last_change":"2024-09-17T07:17:18.347374+0000","last_active":"2024-09-17T07:18:42.361037+0000","last_peered":"2024-09-17T07:18:42.361037+0000","last_clean":"2024-09-17T07:18:42.361037+0000","last_became_active":"2024-09-17T07:09:24.408139+0000","last_became_peered":"2024-09-17T07:09:24.408139+0000","last_unstale":"2024-09-17T07:18:42.361037+0000","last_undegraded":"2024-09-17T07:18:42.361037+0000","last_fullsized":"2024-09-17T07:18:42.361037+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:17:18.347257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:10:36.334827+0000","last_clean_scrub_stamp":"2024-09-17T07:17:18.347257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.013321,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":216,"reported_epoch":355,"state":"active+undersized","last_fresh":"2024-09-17T07:18:52.406882+0000","last_change":"2024-09-17T07:16:17.373522+0000","last_active":"2024-09-17T07:18:52.406882+0000","last_peered":"2024-09-17T07:18:52.406882+0000","last_clean":"2024-09-17T07:16:15.494646+0000","last_became_active":"2024-09-17T07:16:17.373522+0000","last_became_peered":"2024-09-17T07:16:17.373522+0000","last_unstale":"2024-09-17T07:18:52.406882+0000","last_undegraded":"2024-09-17T07:18:52.406882+0000","last_fullsized":"2024-09-17T07:16:17.364098+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:37.523283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:15:37.523283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013475850000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":370,"reported_epoch":355,"state":"active+undersized","last_fresh":"2024-09-17T07:18:52.393755+0000","last_change":"2024-09-17T07:16:17.371270+0000","last_active":"2024-09-17T07:18:52.393755+0000","last_peered":"2024-09-17T07:18:52.393755+0000","last_clean":"2024-09-17T07:16:15.494422+0000","last_became_active":"2024-09-17T07:16:17.371270+0000","last_became_peered":"2024-09-17T07:16:17.371270+0000","last_unstale":"2024-09-17T07:18:52.393755+0000","last_undegraded":"2024-09-17T07:18:52.393755+0000","last_fullsized":"2024-09-17T07:16:17.363365+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:40.540336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:15:40.540336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001212982,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"242'19","reported_seq":394,"reported_epoch":354,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:18:42.339855+0000","last_change":"2024-09-17T07:16:17.382006+0000","last_active":"2024-09-17T07:18:42.339855+0000","last_peered":"2024-09-17T07:18:42.339855+0000","last_clean":"2024-09-17T07:16:15.358100+0000","last_became_active":"2024-09-17T07:16:17.382006+0000","last_became_peered":"2024-09-17T07:16:17.382006+0000","last_unstale":"2024-09-17T07:18:42.339855+0000","last_undegraded":"2024-09-17T07:16:17.365274+0000","last_fullsized":"2024-09-17T07:16:17.365168+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'19","last_scrub_stamp":"2024-09-17T07:14:05.729535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:14:05.729535+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.008660559,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"94'2","reported_seq":106,"reported_epoch":354,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:18:43.912338+0000","last_change":"2024-09-17T07:16:17.374418+0000","last_active":"2024-09-17T07:18:43.912338+0000","last_peered":"2024-09-17T07:18:43.912338+0000","last_clean":"2024-09-17T07:04:06.025673+0000","last_became_active":"2024-09-17T07:16:17.374418+0000","last_became_peered":"2024-09-17T07:16:17.374418+0000","last_unstale":"2024-09-17T07:18:43.912338+0000","last_undegraded":"2024-09-17T07:16:17.367359+0000","last_fullsized":"2024-09-17T07:16:17.367242+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-09-17T07:15:48.907821+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-09-17T07:07:31.717888+0000","last_clean_scrub_stamp":"2024-09-17T07:15:48.907821+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[{"shards":"0,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":229,"reported_epoch":354,"state":"active+undersized","last_fresh":"2024-09-17T07:18:42.369537+0000","last_change":"2024-09-17T07:16:17.378918+0000","last_active":"2024-09-17T07:18:42.369537+0000","last_peered":"2024-09-17T07:18:42.369537+0000","last_clean":"2024-09-17T07:09:22.372509+0000","last_became_active":"2024-09-17T07:16:17.378918+0000","last_became_peered":"2024-09-17T07:16:17.378918+0000","last_unstale":"2024-09-17T07:18:42.369537+0000","last_undegraded":"2024-09-17T07:18:42.369537+0000","last_fullsized":"2024-09-17T07:16:17.367596+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:41.858093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:15:41.858093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037672679999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":220,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-17T07:18:52.406952+0000","last_change":"2024-09-17T07:18:23.515698+0000","last_active":"2024-09-17T07:18:52.406952+0000","last_peered":"2024-09-17T07:18:52.406952+0000","last_clean":"2024-09-17T07:18:52.406952+0000","last_became_active":"2024-09-17T07:09:24.400791+0000","last_became_peered":"2024-09-17T07:09:24.400791+0000","last_unstale":"2024-09-17T07:18:52.406952+0000","last_undegraded":"2024-09-17T07:18:52.406952+0000","last_fullsized":"2024-09-17T07:18:52.406952+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:18:23.515582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:49.573666+0000","last_clean_scrub_stamp":"2024-09-17T07:18:23.515582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:19:29.875635+0000","scrub_duration":0.0022931869999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"242'8","reported_seq":292,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-17T07:18:48.031916+0000","last_change":"2024-09-17T07:18:23.038181+0000","last_active":"2024-09-17T07:18:48.031916+0000","last_peered":"2024-09-17T07:18:48.031916+0000","last_clean":"2024-09-17T07:18:48.031916+0000","last_became_active":"2024-09-17T07:09:24.396368+0000","last_became_peered":"2024-09-17T07:09:24.396368+0000","last_unstale":"2024-09-17T07:18:48.031916+0000","last_undegraded":"2024-09-17T07:18:48.031916+0000","last_fullsized":"2024-09-17T07:18:48.031916+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'8","last_scrub_stamp":"2024-09-17T07:18:23.038036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:18:23.038036+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:19:48.354894+0000","scrub_duration":0.021850082999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":378,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-17T07:18:42.369947+0000","last_change":"2024-09-17T07:16:37.093569+0000","last_active":"2024-09-17T07:18:42.369947+0000","last_peered":"2024-09-17T07:18:42.369947+0000","last_clean":"2024-09-17T07:18:42.369947+0000","last_became_active":"2024-09-17T07:02:12.008898+0000","last_became_peered":"2024-09-17T07:02:12.008898+0000","last_unstale":"2024-09-17T07:18:42.369947+0000","last_undegraded":"2024-09-17T07:18:42.369947+0000","last_fullsized":"2024-09-17T07:18:42.369947+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:37.093460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:16:37.093460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0048915060000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":387,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-17T07:18:42.360323+0000","last_change":"2024-09-17T07:17:34.366051+0000","last_active":"2024-09-17T07:18:42.360323+0000","last_peered":"2024-09-17T07:18:42.360323+0000","last_clean":"2024-09-17T07:18:42.360323+0000","last_became_active":"2024-09-17T07:02:16.014982+0000","last_became_peered":"2024-09-17T07:02:16.014982+0000","last_unstale":"2024-09-17T07:18:42.360323+0000","last_undegraded":"2024-09-17T07:18:42.360323+0000","last_fullsized":"2024-09-17T07:18:42.360323+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:17:34.365905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:54.553970+0000","last_clean_scrub_stamp":"2024-09-17T07:17:34.365905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:19:02.007920+0000","scrub_duration":0.0089364449999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":362,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-17T07:18:43.024956+0000","last_change":"2024-09-17T07:17:27.977910+0000","last_active":"2024-09-17T07:18:43.024956+0000","last_peered":"2024-09-17T07:18:43.024956+0000","last_clean":"2024-09-17T07:18:43.024956+0000","last_became_active":"2024-09-17T07:02:18.021014+0000","last_became_peered":"2024-09-17T07:02:18.021014+0000","last_unstale":"2024-09-17T07:18:43.024956+0000","last_undegraded":"2024-09-17T07:18:43.024956+0000","last_fullsized":"2024-09-17T07:18:43.024956+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:17:27.977803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:17:27.977803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002005036,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"242'19","reported_seq":263,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-17T07:18:48.025750+0000","last_change":"2024-09-17T07:17:33.982980+0000","last_active":"2024-09-17T07:18:48.025750+0000","last_peered":"2024-09-17T07:18:48.025750+0000","last_clean":"2024-09-17T07:18:48.025750+0000","last_became_active":"2024-09-17T07:09:24.404942+0000","last_became_peered":"2024-09-17T07:09:24.404942+0000","last_unstale":"2024-09-17T07:18:48.025750+0000","last_undegraded":"2024-09-17T07:18:48.025750+0000","last_fullsized":"2024-09-17T07:18:48.025750+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'19","last_scrub_stamp":"2024-09-17T07:17:33.982784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:17:33.982784+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0043671170000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":400,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-17T07:18:42.361354+0000","last_change":"2024-09-17T07:18:41.951465+0000","last_active":"2024-09-17T07:18:42.361354+0000","last_peered":"2024-09-17T07:18:42.361354+0000","last_clean":"2024-09-17T07:18:42.361354+0000","last_became_active":"2024-09-17T07:02:12.010195+0000","last_became_peered":"2024-09-17T07:02:12.010195+0000","last_unstale":"2024-09-17T07:18:42.361354+0000","last_undegraded":"2024-09-17T07:18:42.361354+0000","last_fullsized":"2024-09-17T07:18:42.361354+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:18:41.951382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:18:41.951382+0000","last_clean_scrub_stamp":"2024-09-17T07:18:41.951382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:20:02.248914+0000","scrub_duration":0.001516739,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":364,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-17T07:18:42.337391+0000","last_change":"2024-09-17T07:17:22.916134+0000","last_active":"2024-09-17T07:18:42.337391+0000","last_peered":"2024-09-17T07:18:42.337391+0000","last_clean":"2024-09-17T07:18:42.337391+0000","last_became_active":"2024-09-17T07:02:16.016183+0000","last_became_peered":"2024-09-17T07:02:16.016183+0000","last_unstale":"2024-09-17T07:18:42.337391+0000","last_undegraded":"2024-09-17T07:18:42.337391+0000","last_fullsized":"2024-09-17T07:18:42.337391+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:17:22.915543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:17:22.915543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0086489989999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":220,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-17T07:18:52.406789+0000","last_change":"2024-09-17T07:17:34.551610+0000","last_active":"2024-09-17T07:18:52.406789+0000","last_peered":"2024-09-17T07:18:52.406789+0000","last_clean":"2024-09-17T07:18:52.406789+0000","last_became_active":"2024-09-17T07:09:24.392216+0000","last_became_peered":"2024-09-17T07:09:24.392216+0000","last_unstale":"2024-09-17T07:18:52.406789+0000","last_undegraded":"2024-09-17T07:18:52.406789+0000","last_fullsized":"2024-09-17T07:18:52.406789+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:17:34.551498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:17:34.551498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002777912,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.10","version":"355'10080","reported_seq":16917,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-17T07:18:53.170173+0000","last_change":"2024-09-17T07:18:17.281071+0000","last_active":"2024-09-17T07:18:53.170173+0000","last_peered":"2024-09-17T07:18:53.170173+0000","last_clean":"2024-09-17T07:18:53.170173+0000","last_became_active":"2024-09-17T07:18:17.272736+0000","last_became_peered":"2024-09-17T07:18:17.272736+0000","last_unstale":"2024-09-17T07:18:53.170173+0000","last_undegraded":"2024-09-17T07:18:53.170173+0000","last_fullsized":"2024-09-17T07:18:53.170173+0000","mapping_epoch":342,"log_start":"313'4083","ondisk_log_start":"313'4083","created":332,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":5997,"log_dups_size":0,"ondisk_log_size":5997,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100597760,"num_objects":1535,"num_object_clones":0,"num_object_copies":4605,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1535,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6183,"num_write_kb":246976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3368,"num_bytes_recovered":220659712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"326'2","reported_seq":363,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-17T07:18:42.338309+0000","last_change":"2024-09-17T07:17:36.974642+0000","last_active":"2024-09-17T07:18:42.338309+0000","last_peered":"2024-09-17T07:18:42.338309+0000","last_clean":"2024-09-17T07:18:42.338309+0000","last_became_active":"2024-09-17T07:02:18.021855+0000","last_became_peered":"2024-09-17T07:02:18.021855+0000","last_unstale":"2024-09-17T07:18:42.338309+0000","last_undegraded":"2024-09-17T07:18:42.338309+0000","last_fullsized":"2024-09-17T07:18:42.338309+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'2","last_scrub_stamp":"2024-09-17T07:17:36.974358+0000","last_deep_scrub":"326'2","last_deep_scrub_stamp":"2024-09-17T07:17:36.974358+0000","last_clean_scrub_stamp":"2024-09-17T07:17:36.974358+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.015149954,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":379,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-17T07:18:42.365295+0000","last_change":"2024-09-17T07:16:39.053518+0000","last_active":"2024-09-17T07:18:42.365295+0000","last_peered":"2024-09-17T07:18:42.365295+0000","last_clean":"2024-09-17T07:18:42.365295+0000","last_became_active":"2024-09-17T07:02:12.005329+0000","last_became_peered":"2024-09-17T07:02:12.005329+0000","last_unstale":"2024-09-17T07:18:42.365295+0000","last_undegraded":"2024-09-17T07:18:42.365295+0000","last_fullsized":"2024-09-17T07:18:42.365295+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:39.053386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:16:39.053386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0044426509999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"294'25","reported_seq":457,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-17T07:18:52.407281+0000","last_change":"2024-09-17T07:17:43.574192+0000","last_active":"2024-09-17T07:18:52.407281+0000","last_peered":"2024-09-17T07:18:52.407281+0000","last_clean":"2024-09-17T07:18:52.407281+0000","last_became_active":"2024-09-17T07:10:49.730948+0000","last_became_peered":"2024-09-17T07:10:49.730948+0000","last_unstale":"2024-09-17T07:18:52.407281+0000","last_undegraded":"2024-09-17T07:18:52.407281+0000","last_fullsized":"2024-09-17T07:18:52.407281+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'25","last_scrub_stamp":"2024-09-17T07:17:43.574077+0000","last_deep_scrub":"294'25","last_deep_scrub_stamp":"2024-09-17T07:16:00.471311+0000","last_clean_scrub_stamp":"2024-09-17T07:17:43.574077+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:19:00.914807+0000","scrub_duration":0.0071385140000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"75'8","reported_seq":598,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-17T07:18:50.714855+0000","last_change":"2024-09-17T07:17:25.451002+0000","last_active":"2024-09-17T07:18:50.714855+0000","last_peered":"2024-09-17T07:18:50.714855+0000","last_clean":"2024-09-17T07:18:50.714855+0000","last_became_active":"2024-09-17T07:02:16.020560+0000","last_became_peered":"2024-09-17T07:02:16.020560+0000","last_unstale":"2024-09-17T07:18:50.714855+0000","last_undegraded":"2024-09-17T07:18:50.714855+0000","last_fullsized":"2024-09-17T07:18:50.714855+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:17:25.450919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:17:25.450919+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:53.609078+0000","scrub_duration":0.0013309369999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.11","version":"355'10316","reported_seq":17507,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-17T07:18:53.859788+0000","last_change":"2024-09-17T07:18:16.281801+0000","last_active":"2024-09-17T07:18:53.859788+0000","last_peered":"2024-09-17T07:18:53.859788+0000","last_clean":"2024-09-17T07:18:53.859788+0000","last_became_active":"2024-09-17T07:18:16.270511+0000","last_became_peered":"2024-09-17T07:18:16.270511+0000","last_unstale":"2024-09-17T07:18:53.859788+0000","last_undegraded":"2024-09-17T07:18:53.859788+0000","last_fullsized":"2024-09-17T07:18:53.859788+0000","mapping_epoch":341,"log_start":"312'3457","ondisk_log_start":"312'3457","created":332,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":5,"last_scrub":"320'6219","last_scrub_stamp":"2024-09-17T07:17:22.401249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:17:22.401249+0000","objects_scrubbed":6144,"log_size":6859,"log_dups_size":0,"ondisk_log_size":6859,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2.0885702689999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100728832,"num_objects":1537,"num_object_clones":0,"num_object_copies":4611,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6339,"num_write_kb":252032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3482,"num_bytes_recovered":228196352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":61,"reported_epoch":354,"state":"active+undersized","last_fresh":"2024-09-17T07:18:48.916990+0000","last_change":"2024-09-17T07:16:17.375358+0000","last_active":"2024-09-17T07:18:48.916990+0000","last_peered":"2024-09-17T07:18:48.916990+0000","last_clean":"2024-09-17T07:02:17.002904+0000","last_became_active":"2024-09-17T07:16:17.375358+0000","last_became_peered":"2024-09-17T07:16:17.375358+0000","last_unstale":"2024-09-17T07:18:48.916990+0000","last_undegraded":"2024-09-17T07:18:48.916990+0000","last_fullsized":"2024-09-17T07:16:17.368647+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:37.885570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:15:37.885570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":229,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-17T07:18:42.360572+0000","last_change":"2024-09-17T07:17:16.329912+0000","last_active":"2024-09-17T07:18:42.360572+0000","last_peered":"2024-09-17T07:18:42.360572+0000","last_clean":"2024-09-17T07:18:42.360572+0000","last_became_active":"2024-09-17T07:09:24.403360+0000","last_became_peered":"2024-09-17T07:09:24.403360+0000","last_unstale":"2024-09-17T07:18:42.360572+0000","last_undegraded":"2024-09-17T07:18:42.360572+0000","last_fullsized":"2024-09-17T07:18:42.360572+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:17:16.329797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:10:29.325664+0000","last_clean_scrub_stamp":"2024-09-17T07:17:16.329797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033990230000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"242'19","reported_seq":462,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-17T07:18:52.406887+0000","last_change":"2024-09-17T07:17:17.376514+0000","last_active":"2024-09-17T07:18:52.406887+0000","last_peered":"2024-09-17T07:18:52.406887+0000","last_clean":"2024-09-17T07:18:52.406887+0000","last_became_active":"2024-09-17T07:03:48.760390+0000","last_became_peered":"2024-09-17T07:03:48.760390+0000","last_unstale":"2024-09-17T07:18:52.406887+0000","last_undegraded":"2024-09-17T07:18:52.406887+0000","last_fullsized":"2024-09-17T07:18:52.406887+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'19","last_scrub_stamp":"2024-09-17T07:17:17.375166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:17:17.375166+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.014344902,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":367,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-17T07:18:52.407219+0000","last_change":"2024-09-17T07:17:31.538898+0000","last_active":"2024-09-17T07:18:52.407219+0000","last_peered":"2024-09-17T07:18:52.407219+0000","last_clean":"2024-09-17T07:18:52.407219+0000","last_became_active":"2024-09-17T07:02:16.019701+0000","last_became_peered":"2024-09-17T07:02:16.019701+0000","last_unstale":"2024-09-17T07:18:52.407219+0000","last_undegraded":"2024-09-17T07:18:52.407219+0000","last_fullsized":"2024-09-17T07:18:52.407219+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:17:31.538812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:17:31.538812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014632390000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.12","version":"356'10001","reported_seq":10086,"reported_epoch":356,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2024-09-17T07:18:54.525443+0000","last_change":"2024-09-17T07:18:25.370704+0000","last_active":"2024-09-17T07:18:54.525443+0000","last_peered":"2024-09-17T07:18:54.525443+0000","last_clean":"2024-09-17T07:16:04.025356+0000","last_became_active":"2024-09-17T07:17:55.362534+0000","last_became_peered":"2024-09-17T07:17:55.362534+0000","last_unstale":"2024-09-17T07:18:54.525443+0000","last_undegraded":"2024-09-17T07:17:55.278241+0000","last_fullsized":"2024-09-17T07:17:55.254538+0000","mapping_epoch":335,"log_start":"313'3798","ondisk_log_start":"313'3798","created":332,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":6203,"log_dups_size":0,"ondisk_log_size":6203,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100237312,"num_objects":1530,"num_object_clones":0,"num_object_copies":4590,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1530,"num_objects_misplaced":3060,"num_objects_unfound":0,"num_objects_dirty":1530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6133,"num_write_kb":245184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0,4","objects":1530}],"blocked_by":[],"up_primary":6,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":376,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-17T07:18:42.340411+0000","last_change":"2024-09-17T07:17:37.138113+0000","last_active":"2024-09-17T07:18:42.340411+0000","last_peered":"2024-09-17T07:18:42.340411+0000","last_clean":"2024-09-17T07:18:42.340411+0000","last_became_active":"2024-09-17T07:02:18.363752+0000","last_became_peered":"2024-09-17T07:02:18.363752+0000","last_unstale":"2024-09-17T07:18:42.340411+0000","last_undegraded":"2024-09-17T07:18:42.340411+0000","last_fullsized":"2024-09-17T07:18:42.340411+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:17:37.137989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:17:37.137989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012195245,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":420,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-17T07:18:42.360853+0000","last_change":"2024-09-17T07:18:40.285093+0000","last_active":"2024-09-17T07:18:42.360853+0000","last_peered":"2024-09-17T07:18:42.360853+0000","last_clean":"2024-09-17T07:18:42.360853+0000","last_became_active":"2024-09-17T07:03:48.759384+0000","last_became_peered":"2024-09-17T07:03:48.759384+0000","last_unstale":"2024-09-17T07:18:42.360853+0000","last_undegraded":"2024-09-17T07:18:42.360853+0000","last_fullsized":"2024-09-17T07:18:42.360853+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:44.558901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:11:18.559877+0000","last_clean_scrub_stamp":"2024-09-17T07:16:44.558901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.031473045999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"242'27","reported_seq":332,"reported_epoch":355,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:18:52.410904+0000","last_change":"2024-09-17T07:16:17.368008+0000","last_active":"2024-09-17T07:18:52.410904+0000","last_peered":"2024-09-17T07:18:52.410904+0000","last_clean":"2024-09-17T07:12:20.337498+0000","last_became_active":"2024-09-17T07:16:17.368008+0000","last_became_peered":"2024-09-17T07:16:17.368008+0000","last_unstale":"2024-09-17T07:18:52.410904+0000","last_undegraded":"2024-09-17T07:16:17.365648+0000","last_fullsized":"2024-09-17T07:16:17.365550+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'27","last_scrub_stamp":"2024-09-17T07:15:45.855430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:15:45.855430+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.017595989999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":7}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":61,"reported_epoch":354,"state":"active+undersized","last_fresh":"2024-09-17T07:18:48.225679+0000","last_change":"2024-09-17T07:16:17.371494+0000","last_active":"2024-09-17T07:18:48.225679+0000","last_peered":"2024-09-17T07:18:48.225679+0000","last_clean":"2024-09-17T07:02:14.999435+0000","last_became_active":"2024-09-17T07:16:17.371494+0000","last_became_peered":"2024-09-17T07:16:17.371494+0000","last_unstale":"2024-09-17T07:18:48.225679+0000","last_undegraded":"2024-09-17T07:18:48.225679+0000","last_fullsized":"2024-09-17T07:16:17.366463+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:00.509283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:25.794418+0000","last_clean_scrub_stamp":"2024-09-17T07:14:00.509283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.13","version":"355'12972","reported_seq":20137,"reported_epoch":355,"state":"active+clean+remapped","last_fresh":"2024-09-17T07:18:52.975262+0000","last_change":"2024-09-17T07:18:13.461884+0000","last_active":"2024-09-17T07:18:52.975262+0000","last_peered":"2024-09-17T07:18:52.975262+0000","last_clean":"2024-09-17T07:18:52.975262+0000","last_became_active":"2024-09-17T07:18:13.450785+0000","last_became_peered":"2024-09-17T07:18:13.450785+0000","last_unstale":"2024-09-17T07:18:52.975262+0000","last_undegraded":"2024-09-17T07:18:52.975262+0000","last_fullsized":"2024-09-17T07:18:52.975262+0000","mapping_epoch":339,"log_start":"320'5801","ondisk_log_start":"320'5801","created":332,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":4348,"log_size":7171,"log_dups_size":0,"ondisk_log_size":7171,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100433920,"num_objects":1533,"num_object_clones":0,"num_object_copies":4599,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1533,"num_objects_unfound":0,"num_objects_dirty":1533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6037,"num_write_kb":242208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3360,"num_bytes_recovered":218824704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2,4],"avail_no_missing":["7","2","4"],"object_location_counts":[{"shards":"2,4,7","objects":1533}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":226,"reported_epoch":354,"state":"active+undersized","last_fresh":"2024-09-17T07:18:42.361200+0000","last_change":"2024-09-17T07:16:17.377951+0000","last_active":"2024-09-17T07:18:42.361200+0000","last_peered":"2024-09-17T07:18:42.361200+0000","last_clean":"2024-09-17T07:16:14.246741+0000","last_became_active":"2024-09-17T07:16:17.377951+0000","last_became_peered":"2024-09-17T07:16:17.377951+0000","last_unstale":"2024-09-17T07:18:42.361200+0000","last_undegraded":"2024-09-17T07:18:42.361200+0000","last_fullsized":"2024-09-17T07:16:17.364654+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:02.473467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:15:02.473467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001733163,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":209,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-17T07:18:42.338360+0000","last_change":"2024-09-17T07:16:31.651379+0000","last_active":"2024-09-17T07:18:42.338360+0000","last_peered":"2024-09-17T07:18:42.338360+0000","last_clean":"2024-09-17T07:18:42.338360+0000","last_became_active":"2024-09-17T07:09:24.404795+0000","last_became_peered":"2024-09-17T07:09:24.404795+0000","last_unstale":"2024-09-17T07:18:42.338360+0000","last_undegraded":"2024-09-17T07:18:42.338360+0000","last_fullsized":"2024-09-17T07:18:42.338360+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:31.651247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:16:31.651247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0075767350000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"242'23","reported_seq":193,"reported_epoch":354,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:18:48.225731+0000","last_change":"2024-09-17T07:16:17.380972+0000","last_active":"2024-09-17T07:18:48.225731+0000","last_peered":"2024-09-17T07:18:48.225731+0000","last_clean":"2024-09-17T07:12:20.742449+0000","last_became_active":"2024-09-17T07:16:17.380972+0000","last_became_peered":"2024-09-17T07:16:17.380972+0000","last_unstale":"2024-09-17T07:18:48.225731+0000","last_undegraded":"2024-09-17T07:16:17.366757+0000","last_fullsized":"2024-09-17T07:16:17.366645+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'23","last_scrub_stamp":"2024-09-17T07:14:58.689120+0000","last_deep_scrub":"86'17","last_deep_scrub_stamp":"2024-09-17T07:03:18.241067+0000","last_clean_scrub_stamp":"2024-09-17T07:14:58.689120+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":385,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-17T07:18:42.340322+0000","last_change":"2024-09-17T07:17:17.309054+0000","last_active":"2024-09-17T07:18:42.340322+0000","last_peered":"2024-09-17T07:18:42.340322+0000","last_clean":"2024-09-17T07:18:42.340322+0000","last_became_active":"2024-09-17T07:09:24.393329+0000","last_became_peered":"2024-09-17T07:09:24.393329+0000","last_unstale":"2024-09-17T07:18:42.340322+0000","last_undegraded":"2024-09-17T07:18:42.340322+0000","last_fullsized":"2024-09-17T07:18:42.340322+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:17:17.308762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:17:17.308762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.023046534,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":26,"stat_sum":{"num_bytes":3219685400,"num_objects":49131,"num_object_clones":0,"num_object_copies":147393,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9199,"num_objects_misplaced":13936,"num_objects_unfound":0,"num_objects_dirty":49131,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":162603,"num_write_kb":6421793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21574,"num_bytes_recovered":1412300800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9047322624,"data_stored":9047310408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154192,"ondisk_log_size":154192,"up":67,"acting":73,"num_store_stats":7},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":83,"acting":83,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73,"ondisk_log_size":73,"up":81,"acting":81,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":101,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1113,"num_read_kb":870,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":109,"num_bytes_recovered":2228,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":13004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":733,"ondisk_log_size":733,"up":78,"acting":78,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":162,"num_read_kb":157,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":8,"acting":8,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1386,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":12,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":4578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":83,"acting":83,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037966,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1625876,"kb_used_data":894988,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":92119276,"statfs":{"total":95995035648,"available":94330138624,"internally_reserved":0,"allocated":916467712,"data_stored":915780722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":748419531},"hb_peers":[0,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:18:32 2024","interfaces":[{"interface":"back","average":{"1min":3.0750000000000002,"5min":1.3919999999999999,"15min":1.772},"min":{"1min":0.20699999999999999,"5min":0.19700000000000001,"15min":0.192},"max":{"1min":24.739999999999998,"5min":24.739999999999998,"15min":52.478999999999999},"last":0.747},{"interface":"front","average":{"1min":2.6030000000000002,"5min":1.3859999999999999,"15min":1.8680000000000001},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":24.768000000000001,"5min":24.768000000000001,"15min":52.871000000000002},"last":0.53600000000000003}]},{"osd":1,"last update":"Tue Sep 17 07:15:24 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":1.4510000000000001,"15min":1.3859999999999999},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.223},"max":{"1min":1.573,"5min":15.801,"15min":73.549000000000007},"last":0.65300000000000002},{"interface":"front","average":{"1min":1.3779999999999999,"5min":1.2969999999999999,"15min":1.383},"min":{"1min":0.29699999999999999,"5min":0.26200000000000001,"15min":0.24099999999999999},"max":{"1min":4.7359999999999998,"5min":13.989000000000001,"15min":73.116},"last":0.71499999999999997}]},{"osd":2,"last update":"Tue Sep 17 07:18:32 2024","interfaces":[{"interface":"back","average":{"1min":1.8440000000000001,"5min":1.2390000000000001,"15min":1.274},"min":{"1min":0.28599999999999998,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":10.057,"5min":10.057,"15min":52.055999999999997},"last":0.38200000000000001},{"interface":"front","average":{"1min":1.216,"5min":1.131,"15min":1.2170000000000001},"min":{"1min":0.247,"5min":0.224,"15min":0.224},"max":{"1min":9.843,"5min":9.843,"15min":52.064999999999998},"last":0.72199999999999998}]},{"osd":3,"last update":"Tue Sep 17 07:18:32 2024","interfaces":[{"interface":"back","average":{"1min":2.02,"5min":1.292,"15min":1.4750000000000001},"min":{"1min":0.316,"5min":0.29199999999999998,"15min":0.28799999999999998},"max":{"1min":10.122999999999999,"5min":10.122999999999999,"15min":51.689},"last":0.58999999999999997},{"interface":"front","average":{"1min":1.4830000000000001,"5min":1.2270000000000001,"15min":1.544},"min":{"1min":0.33800000000000002,"5min":0.27000000000000002,"15min":0.26500000000000001},"max":{"1min":12.465999999999999,"5min":12.465999999999999,"15min":52.145000000000003},"last":0.68400000000000005}]},{"osd":4,"last update":"Tue Sep 17 07:18:32 2024","interfaces":[{"interface":"back","average":{"1min":1.7649999999999999,"5min":1.5960000000000001,"15min":0.97999999999999998},"min":{"1min":0.27300000000000002,"5min":0.222,"15min":0.222},"max":{"1min":12.679,"5min":21.140999999999998,"15min":21.140999999999998},"last":0.371},{"interface":"front","average":{"1min":2.4209999999999998,"5min":1.8620000000000001,"15min":1.1359999999999999},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":11.984999999999999,"5min":19.420000000000002,"15min":19.420000000000002},"last":1.343}]},{"osd":5,"last update":"Tue Sep 17 07:18:32 2024","interfaces":[{"interface":"back","average":{"1min":1.8300000000000001,"5min":1.6719999999999999,"15min":1.173},"min":{"1min":0.315,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":12.429,"5min":19.384,"15min":29.428999999999998},"last":3.8069999999999999},{"interface":"front","average":{"1min":1.6399999999999999,"5min":1.7689999999999999,"15min":1.214},"min":{"1min":0.314,"5min":0.27500000000000002,"15min":0.25700000000000001},"max":{"1min":12.606,"5min":19.483000000000001,"15min":30.059999999999999},"last":1.361}]},{"osd":6,"last update":"Tue Sep 17 07:18:32 2024","interfaces":[{"interface":"back","average":{"1min":2.2120000000000002,"5min":1.9299999999999999,"15min":1.2509999999999999},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":12.641,"5min":22.221,"15min":25.363},"last":0.40300000000000002},{"interface":"front","average":{"1min":2.5529999999999999,"5min":2.1429999999999998,"15min":1.3320000000000001},"min":{"1min":0.30499999999999999,"5min":0.29099999999999998,"15min":0.27500000000000002},"max":{"1min":10.499000000000001,"5min":22.202000000000002,"15min":23.571000000000002},"last":0.86699999999999999}]}]},{"osd":6,"up_from":50,"seq":214748365009,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2249208,"kb_used_data":1618412,"kb_used_omap":1,"kb_used_meta":630782,"kb_avail":91495944,"statfs":{"total":95995035648,"available":93691846656,"internally_reserved":0,"allocated":1657253888,"data_stored":1656534771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":645921223},"hb_peers":[0,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:18:15 2024","interfaces":[{"interface":"back","average":{"1min":1.6419999999999999,"5min":2.0899999999999999,"15min":2.121},"min":{"1min":0.218,"5min":0.20200000000000001,"15min":0.185},"max":{"1min":5.2229999999999999,"5min":126.902,"15min":126.902},"last":1.456},{"interface":"front","average":{"1min":0.876,"5min":1.7909999999999999,"15min":1.99},"min":{"1min":0.23200000000000001,"5min":0.20899999999999999,"15min":0.20799999999999999},"max":{"1min":2.7919999999999998,"5min":127.06399999999999,"15min":127.06399999999999},"last":0.307}]},{"osd":1,"last update":"Tue Sep 17 07:16:10 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.996,"15min":0.84399999999999997},"min":{"1min":0.23799999999999999,"5min":0.22900000000000001,"15min":0.22600000000000001},"max":{"1min":2.2999999999999998,"5min":14.720000000000001,"15min":14.720000000000001},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.622,"5min":1.0109999999999999,"15min":0.88800000000000001},"min":{"1min":0.29099999999999998,"5min":0.23400000000000001,"15min":0.22900000000000001},"max":{"1min":2.323,"5min":14.699,"15min":14.699},"last":0.46600000000000003}]},{"osd":2,"last update":"Tue Sep 17 07:18:15 2024","interfaces":[{"interface":"back","average":{"1min":1.742,"5min":1.0449999999999999,"15min":0.98599999999999999},"min":{"1min":0.251,"5min":0.219,"15min":0.219},"max":{"1min":5.266,"5min":5.266,"15min":33.079000000000001},"last":1.5149999999999999},{"interface":"front","average":{"1min":1.0369999999999999,"5min":0.81499999999999995,"15min":0.90400000000000003},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":2.8380000000000001,"5min":4.8399999999999999,"15min":33.100000000000001},"last":1.744}]},{"osd":3,"last update":"Tue Sep 17 07:18:15 2024","interfaces":[{"interface":"back","average":{"1min":1.643,"5min":0.879,"15min":1.2609999999999999},"min":{"1min":0.27500000000000002,"5min":0.27000000000000002,"15min":0.23200000000000001},"max":{"1min":4.9320000000000004,"5min":4.9320000000000004,"15min":52.944000000000003},"last":2.1339999999999999},{"interface":"front","average":{"1min":1.7130000000000001,"5min":1.22,"15min":1.3700000000000001},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":5.2949999999999999,"5min":5.7690000000000001,"15min":52.747},"last":1.492}]},{"osd":4,"last update":"Tue Sep 17 07:18:15 2024","interfaces":[{"interface":"back","average":{"1min":2.0070000000000001,"5min":1.4330000000000001,"15min":2.129},"min":{"1min":0.27900000000000003,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":5.4800000000000004,"5min":7.1050000000000004,"15min":260.77199999999999},"last":0.54500000000000004},{"interface":"front","average":{"1min":3.2509999999999999,"5min":1.51,"15min":2.1819999999999999},"min":{"1min":0.36299999999999999,"5min":0.23300000000000001,"15min":0.19800000000000001},"max":{"1min":8.7929999999999993,"5min":8.7929999999999993,"15min":260.69099999999997},"last":2.2149999999999999}]},{"osd":5,"last update":"Tue Sep 17 07:18:15 2024","interfaces":[{"interface":"back","average":{"1min":2.0649999999999999,"5min":1.8959999999999999,"15min":1.2230000000000001},"min":{"1min":0.32900000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":10.201000000000001,"5min":11.488,"15min":32.042000000000002},"last":1.7669999999999999},{"interface":"front","average":{"1min":2.0219999999999998,"5min":1.734,"15min":1.208},"min":{"1min":0.44,"5min":0.27200000000000002,"15min":0.251},"max":{"1min":5.5540000000000003,"5min":10.244999999999999,"15min":31.852},"last":2.1949999999999998}]},{"osd":7,"last update":"Tue Sep 17 07:18:38 2024","interfaces":[{"interface":"back","average":{"1min":2.2999999999999998,"5min":2.972,"15min":1.5669999999999999},"min":{"1min":0.41199999999999998,"5min":0.246,"15min":0.23699999999999999},"max":{"1min":6.8179999999999996,"5min":131.95699999999999,"15min":131.95699999999999},"last":2.1699999999999999},{"interface":"front","average":{"1min":1.3340000000000001,"5min":2.7109999999999999,"15min":1.4670000000000001},"min":{"1min":0.34499999999999997,"5min":0.252,"15min":0.252},"max":{"1min":5.8070000000000004,"5min":131.833,"15min":131.833},"last":1.7190000000000001}]}]},{"osd":5,"up_from":42,"seq":180388626646,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1636440,"kb_used_data":893260,"kb_used_omap":1,"kb_used_meta":743166,"kb_avail":92108712,"statfs":{"total":95995035648,"available":94319321088,"internally_reserved":0,"allocated":914698240,"data_stored":914011038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":761002438},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":0.92500000000000004,"15min":1.615},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":4.1900000000000004,"5min":4.2329999999999997,"15min":49.265000000000001},"last":0.74099999999999999},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.02,"15min":1.708},"min":{"1min":0.25800000000000001,"5min":0.221,"15min":0.219},"max":{"1min":4.7939999999999996,"5min":6.7149999999999999,"15min":47.350000000000001},"last":0.24399999999999999}]},{"osd":1,"last update":"Tue Sep 17 07:15:47 2024","interfaces":[{"interface":"back","average":{"1min":1.4319999999999999,"5min":1.0509999999999999,"15min":0.90100000000000002},"min":{"1min":0.30499999999999999,"5min":0.22,"15min":0.22},"max":{"1min":13.711,"5min":13.711,"15min":20.329000000000001},"last":0.437},{"interface":"front","average":{"1min":0.99399999999999999,"5min":1.3320000000000001,"15min":1.149},"min":{"1min":0.32000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4.617,"5min":6.7560000000000002,"15min":20.408999999999999},"last":0.42199999999999999}]},{"osd":2,"last update":"Tue Sep 17 07:17:55 2024","interfaces":[{"interface":"back","average":{"1min":1.248,"5min":1.105,"15min":1.145},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":4.4160000000000004,"5min":6.3419999999999996,"15min":20.486000000000001},"last":0.59099999999999997},{"interface":"front","average":{"1min":1.1990000000000001,"5min":0.97799999999999998,"15min":1.04},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":7.71,"5min":7.71,"15min":23.925999999999998},"last":0.76900000000000002}]},{"osd":3,"last update":"Tue Sep 17 07:17:55 2024","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":1.333,"15min":1.482},"min":{"1min":0.27900000000000003,"5min":0.27100000000000002,"15min":0.247},"max":{"1min":3.3580000000000001,"5min":13.882,"15min":68.978999999999999},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.755,"5min":0.90400000000000003,"15min":1.3140000000000001},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.253},"max":{"1min":2.0720000000000001,"5min":4.327,"15min":69.084000000000003},"last":0.81699999999999995}]},{"osd":4,"last update":"Tue Sep 17 07:17:55 2024","interfaces":[{"interface":"back","average":{"1min":2.8929999999999998,"5min":1.9590000000000001,"15min":1.706},"min":{"1min":0.29899999999999999,"5min":0.23200000000000001,"15min":0.22},"max":{"1min":15.157,"5min":15.157,"15min":142.327},"last":2.246},{"interface":"front","average":{"1min":2.3239999999999998,"5min":1.7509999999999999,"15min":1.587},"min":{"1min":0.32800000000000001,"5min":0.25,"15min":0.20999999999999999},"max":{"1min":14.629,"5min":14.629,"15min":142.13200000000001},"last":0.67100000000000004}]},{"osd":6,"last update":"Tue Sep 17 07:18:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1779999999999999,"5min":1.208,"15min":0.89800000000000002},"min":{"1min":0.245,"5min":0.245,"15min":0.23200000000000001},"max":{"1min":3.1749999999999998,"5min":6.8419999999999996,"15min":8.3149999999999995},"last":0.70599999999999996},{"interface":"front","average":{"1min":1.9239999999999999,"5min":1.5089999999999999,"15min":1.018},"min":{"1min":0.29799999999999999,"5min":0.26700000000000002,"15min":0.24099999999999999},"max":{"1min":3.5750000000000002,"5min":8.4489999999999998,"15min":8.4489999999999998},"last":0.55900000000000005}]},{"osd":7,"last update":"Tue Sep 17 07:18:31 2024","interfaces":[{"interface":"back","average":{"1min":1.6399999999999999,"5min":1.573,"15min":1.157},"min":{"1min":0.35499999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":3.5960000000000001,"5min":6.8869999999999996,"15min":6.8869999999999996},"last":2.157},{"interface":"front","average":{"1min":1.7609999999999999,"5min":1.391,"15min":1.01},"min":{"1min":0.32500000000000001,"5min":0.28100000000000003,"15min":0.27100000000000002},"max":{"1min":6.3639999999999999,"5min":6.3639999999999999,"15min":6.3639999999999999},"last":0.79500000000000004}]}]},{"osd":4,"up_from":33,"seq":141733920988,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2728800,"kb_used_data":1993812,"kb_used_omap":1,"kb_used_meta":734974,"kb_avail":91016352,"statfs":{"total":95995035648,"available":93200744448,"internally_reserved":0,"allocated":2041663488,"data_stored":2040967218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":752613830},"hb_peers":[0,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:18:20 2024","interfaces":[{"interface":"back","average":{"1min":1.51,"5min":1.135,"15min":2.1230000000000002},"min":{"1min":0.26700000000000002,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":9.2750000000000004,"5min":9.2750000000000004,"15min":74.551000000000002},"last":2.2770000000000001},{"interface":"front","average":{"1min":1.6839999999999999,"5min":1.345,"15min":2.3210000000000002},"min":{"1min":0.26800000000000002,"5min":0.22700000000000001,"15min":0.20699999999999999},"max":{"1min":9.9969999999999999,"5min":16.619,"15min":74.778999999999996},"last":0.36099999999999999}]},{"osd":1,"last update":"Tue Sep 17 07:16:11 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":1.2070000000000001,"15min":1.4850000000000001},"min":{"1min":0.252,"5min":0.24399999999999999,"15min":0.23000000000000001},"max":{"1min":1.736,"5min":18.661000000000001,"15min":54.914999999999999},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.71499999999999997,"5min":1.3,"15min":1.444},"min":{"1min":0.246,"5min":0.23300000000000001,"15min":0.22800000000000001},"max":{"1min":3.7770000000000001,"5min":21.931999999999999,"15min":53.725000000000001},"last":0.443}]},{"osd":2,"last update":"Tue Sep 17 07:18:20 2024","interfaces":[{"interface":"back","average":{"1min":1.5880000000000001,"5min":1.262,"15min":1.1739999999999999},"min":{"1min":0.30599999999999999,"5min":0.26700000000000002,"15min":0.23799999999999999},"max":{"1min":8.4710000000000001,"5min":8.4710000000000001,"15min":49.651000000000003},"last":2.3079999999999998},{"interface":"front","average":{"1min":1.353,"5min":1.1850000000000001,"15min":1.224},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.23599999999999999},"max":{"1min":6.1840000000000002,"5min":16.655999999999999,"15min":48.813000000000002},"last":0.38900000000000001}]},{"osd":3,"last update":"Tue Sep 17 07:18:20 2024","interfaces":[{"interface":"back","average":{"1min":1.141,"5min":0.94399999999999995,"15min":1.115},"min":{"1min":0.318,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":4.0990000000000002,"5min":4.0990000000000002,"15min":39.042999999999999},"last":0.42799999999999999},{"interface":"front","average":{"1min":1.1559999999999999,"5min":1.212,"15min":1.234},"min":{"1min":0.35099999999999998,"5min":0.28999999999999998,"15min":0.253},"max":{"1min":9.0329999999999995,"5min":16.111000000000001,"15min":38.920000000000002},"last":0.87}]},{"osd":5,"last update":"Tue Sep 17 07:18:35 2024","interfaces":[{"interface":"back","average":{"1min":7.8630000000000004,"5min":2.8069999999999999,"15min":1.518},"min":{"1min":0.249,"5min":0.249,"15min":0.20799999999999999},"max":{"1min":147.61099999999999,"5min":147.61099999999999,"15min":147.61099999999999},"last":0.47199999999999998},{"interface":"front","average":{"1min":7.9269999999999996,"5min":2.5979999999999999,"15min":1.5109999999999999},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.26100000000000001},"max":{"1min":147.74700000000001,"5min":147.74700000000001,"15min":147.74700000000001},"last":0.498}]},{"osd":6,"last update":"Tue Sep 17 07:18:20 2024","interfaces":[{"interface":"back","average":{"1min":1.871,"5min":1.4430000000000001,"15min":1.1060000000000001},"min":{"1min":0.28999999999999998,"5min":0.24299999999999999,"15min":0.218},"max":{"1min":8.9619999999999997,"5min":8.9619999999999997,"15min":47.923999999999999},"last":1.006},{"interface":"front","average":{"1min":1.55,"5min":1.5940000000000001,"15min":1.1890000000000001},"min":{"1min":0.38900000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":4.899,"5min":10.051,"15min":47.600999999999999},"last":4.7629999999999999}]},{"osd":7,"last update":"Tue Sep 17 07:18:33 2024","interfaces":[{"interface":"back","average":{"1min":1.1200000000000001,"5min":1.5349999999999999,"15min":1.2150000000000001},"min":{"1min":0.315,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":3.157,"5min":7.1550000000000002,"15min":48.719000000000001},"last":0.65200000000000002},{"interface":"front","average":{"1min":1.5229999999999999,"5min":1.7270000000000001,"15min":1.2709999999999999},"min":{"1min":0.32200000000000001,"5min":0.29199999999999998,"15min":0.28499999999999998},"max":{"1min":7.2910000000000004,"5min":9.0030000000000001,"15min":50.527000000000001},"last":0.83999999999999997}]}]},{"osd":3,"up_from":0,"seq":115964117214,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":18,"seq":77309411556,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2316620,"kb_used_data":1527428,"kb_used_omap":1,"kb_used_meta":789118,"kb_avail":91428532,"statfs":{"total":95995035648,"available":93622816768,"internally_reserved":0,"allocated":1564086272,"data_stored":1563399924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":808057286},"hb_peers":[0,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":2.8279999999999998,"15min":1.913},"min":{"1min":0.154,"5min":0.154,"15min":0.154},"max":{"1min":2.2890000000000001,"5min":131.09800000000001,"15min":131.09800000000001},"last":0.21099999999999999},{"interface":"front","average":{"1min":0.42099999999999999,"5min":2.7669999999999999,"15min":2.0350000000000001},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":1.0960000000000001,"5min":131.24700000000001,"15min":131.24700000000001},"last":0.35399999999999998}]},{"osd":1,"last update":"Tue Sep 17 07:15:43 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":1.7949999999999999,"15min":1.623},"min":{"1min":0.23699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":1.6459999999999999,"5min":47.101999999999997,"15min":79.659000000000006},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.55000000000000004,"5min":1.6990000000000001,"15min":1.5600000000000001},"min":{"1min":0.22700000000000001,"5min":0.20899999999999999,"15min":0.20200000000000001},"max":{"1min":1.8049999999999999,"5min":45.831000000000003,"15min":80.155000000000001},"last":0.41399999999999998}]},{"osd":3,"last update":"Tue Sep 17 07:18:05 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.56899999999999995,"15min":0.97899999999999998},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20100000000000001},"max":{"1min":2.2160000000000002,"5min":4.6449999999999996,"15min":23.43},"last":0.24199999999999999},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.439,"15min":1.036},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":2.121,"5min":2.121,"15min":23.408000000000001},"last":0.40799999999999997}]},{"osd":4,"last update":"Tue Sep 17 07:18:30 2024","interfaces":[{"interface":"back","average":{"1min":1.042,"5min":1.022,"15min":0.97599999999999998},"min":{"1min":0.247,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":5.351,"5min":6.8449999999999998,"15min":6.8449999999999998},"last":1.3700000000000001},{"interface":"front","average":{"1min":1.8080000000000001,"5min":1.2889999999999999,"15min":1.0600000000000001},"min":{"1min":0.28000000000000003,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":19.119,"5min":19.119,"15min":19.119},"last":1.379}]},{"osd":5,"last update":"Tue Sep 17 07:18:52 2024","interfaces":[{"interface":"back","average":{"1min":1.595,"5min":1.169,"15min":1.226},"min":{"1min":0.27900000000000003,"5min":0.26900000000000002,"15min":0.249},"max":{"1min":7.2229999999999999,"5min":7.2229999999999999,"15min":15.276999999999999},"last":1.621},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.82199999999999995,"15min":1.02},"min":{"1min":0.312,"5min":0.28000000000000003,"15min":0.27600000000000002},"max":{"1min":1.3400000000000001,"5min":4.577,"15min":15.561999999999999},"last":0.373}]},{"osd":6,"last update":"Tue Sep 17 07:18:05 2024","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.163,"15min":1.458},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":6.5609999999999999,"5min":20.494,"15min":101.587},"last":2.1589999999999998},{"interface":"front","average":{"1min":1.643,"5min":1.2410000000000001,"15min":1.5289999999999999},"min":{"1min":0.34100000000000003,"5min":0.32300000000000001,"15min":0.28299999999999997},"max":{"1min":5.5019999999999998,"5min":26.922000000000001,"15min":100.696},"last":2.9830000000000001}]},{"osd":7,"last update":"Tue Sep 17 07:18:48 2024","interfaces":[{"interface":"back","average":{"1min":1.1699999999999999,"5min":2.1909999999999998,"15min":1.4119999999999999},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":4.7759999999999998,"5min":94.269999999999996,"15min":94.269999999999996},"last":1.486},{"interface":"front","average":{"1min":1.0229999999999999,"5min":2.1429999999999998,"15min":1.444},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":4.8090000000000002,"5min":94.634,"15min":94.634},"last":0.42599999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575049,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":541424,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":540542,"kb_avail":93203728,"statfs":{"total":95995035648,"available":95440617472,"internally_reserved":0,"allocated":884736,"data_stored":439467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":553515462},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:16:00 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.73699999999999999,"15min":1.139},"min":{"1min":0.183,"5min":0.155,"15min":0.155},"max":{"1min":2.6070000000000002,"5min":4.5780000000000003,"15min":22.521000000000001},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.67500000000000004,"15min":1.0369999999999999},"min":{"1min":0.247,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.198,"5min":4.335,"15min":22.471},"last":0.377}]},{"osd":2,"last update":"Tue Sep 17 07:15:44 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.91000000000000003,"15min":1.024},"min":{"1min":0.21299999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":2.6360000000000001,"5min":6.1920000000000002,"15min":11.055999999999999},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.73899999999999999,"15min":0.94799999999999995},"min":{"1min":0.22900000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":1.3160000000000001,"5min":4.1559999999999997,"15min":9.3160000000000007},"last":0.30599999999999999}]},{"osd":3,"last update":"Tue Sep 17 07:16:00 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.67800000000000005,"15min":0.94999999999999996},"min":{"1min":0.27400000000000002,"5min":0.25900000000000001,"15min":0.219},"max":{"1min":2.4409999999999998,"5min":4.1269999999999998,"15min":8.5510000000000002},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.371,"5min":0.47399999999999998,"15min":0.98599999999999999},"min":{"1min":0.247,"5min":0.22500000000000001,"15min":0.19500000000000001},"max":{"1min":0.627,"5min":2.8199999999999998,"15min":19.602},"last":0.35299999999999998}]},{"osd":4,"last update":"Tue Sep 17 07:15:20 2024","interfaces":[{"interface":"back","average":{"1min":1.1339999999999999,"5min":1.4670000000000001,"15min":1.129},"min":{"1min":0.36199999999999999,"5min":0.314,"15min":0.25},"max":{"1min":2.5659999999999998,"5min":19.088999999999999,"15min":19.088999999999999},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.98999999999999999,"5min":1.2010000000000001,"15min":1.1759999999999999},"min":{"1min":0.36799999999999999,"5min":0.317,"15min":0.307},"max":{"1min":2.6459999999999999,"5min":18.959,"15min":27.097999999999999},"last":0.499}]},{"osd":5,"last update":"Tue Sep 17 07:15:44 2024","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.47,"15min":1.2110000000000001},"min":{"1min":0.34799999999999998,"5min":0.28899999999999998,"15min":0.27100000000000002},"max":{"1min":3.6869999999999998,"5min":6.6790000000000003,"15min":16.84},"last":0.51000000000000001},{"interface":"front","average":{"1min":1.1559999999999999,"5min":1.069,"15min":1.169},"min":{"1min":0.29299999999999998,"5min":0.29199999999999998,"15min":0.28899999999999998},"max":{"1min":6.2560000000000002,"5min":6.2560000000000002,"15min":16.949999999999999},"last":0.41299999999999998}]},{"osd":6,"last update":"Tue Sep 17 07:16:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":1.5149999999999999,"15min":1.274},"min":{"1min":0.33100000000000002,"5min":0.315,"15min":0.308},"max":{"1min":1.127,"5min":50.497,"15min":50.497},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.99099999999999999,"5min":1.6379999999999999,"15min":1.222},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.28899999999999998},"max":{"1min":3.593,"5min":50.793999999999997,"15min":50.793999999999997},"last":0.48499999999999999}]},{"osd":7,"last update":"Tue Sep 17 07:15:20 2024","interfaces":[{"interface":"back","average":{"1min":1.655,"5min":1.343,"15min":1.1919999999999999},"min":{"1min":0.435,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":3.6120000000000001,"5min":9.8179999999999996,"15min":9.8179999999999996},"last":0.441},{"interface":"front","average":{"1min":1.464,"5min":1.6319999999999999,"15min":1.2410000000000001},"min":{"1min":0.35099999999999998,"5min":0.317,"15min":0.317},"max":{"1min":4.1219999999999999,"5min":7.5839999999999996,"15min":11.007999999999999},"last":0.57999999999999996}]}]},{"osd":0,"up_from":8,"seq":34359738604,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2895868,"kb_used_data":2022192,"kb_used_omap":1,"kb_used_meta":873598,"kb_avail":90849284,"statfs":{"total":95995035648,"available":93029666816,"internally_reserved":0,"allocated":2070724608,"data_stored":2070008536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":894564806},"hb_peers":[2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 17 07:15:18 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.89700000000000002,"15min":0.89500000000000002},"min":{"1min":0.254,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":1.518,"5min":21.876000000000001,"15min":21.876000000000001},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.56399999999999995,"5min":1.2330000000000001,"15min":1.081},"min":{"1min":0.17899999999999999,"5min":0.158,"15min":0.158},"max":{"1min":2.2599999999999998,"5min":22.521999999999998,"15min":22.521999999999998},"last":0.33700000000000002}]},{"osd":2,"last update":"Tue Sep 17 07:18:48 2024","interfaces":[{"interface":"back","average":{"1min":0.312,"5min":0.42699999999999999,"15min":1.113},"min":{"1min":0.156,"5min":0.155,"15min":0.155},"max":{"1min":1.117,"5min":2.6520000000000001,"15min":18.643999999999998},"last":0.19},{"interface":"front","average":{"1min":0.307,"5min":0.46200000000000002,"15min":1.0129999999999999},"min":{"1min":0.20399999999999999,"5min":0.189,"15min":0.189},"max":{"1min":0.624,"5min":3.1760000000000002,"15min":17.972999999999999},"last":0.20000000000000001}]},{"osd":3,"last update":"Tue Sep 17 07:18:24 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.46700000000000003,"15min":1.018},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.21199999999999999},"max":{"1min":1.7030000000000001,"5min":3.2000000000000002,"15min":18.629000000000001},"last":0.23799999999999999},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.46800000000000003,"15min":1.0189999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.42699999999999999,"5min":3.1190000000000002,"15min":22.358000000000001},"last":0.248}]},{"osd":4,"last update":"Tue Sep 17 07:18:24 2024","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":0.98699999999999999,"15min":1.1479999999999999},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":5.3179999999999996,"5min":5.5990000000000002,"15min":19.146999999999998},"last":0.48399999999999999},{"interface":"front","average":{"1min":1.3,"5min":1.0580000000000001,"15min":1.1799999999999999},"min":{"1min":0.29499999999999998,"5min":0.26200000000000001,"15min":0.255},"max":{"1min":6.2329999999999997,"5min":6.2329999999999997,"15min":19.158999999999999},"last":0.45200000000000001}]},{"osd":5,"last update":"Tue Sep 17 07:17:55 2024","interfaces":[{"interface":"back","average":{"1min":1.0489999999999999,"5min":3.7370000000000001,"15min":2.121},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":2.8399999999999999,"5min":216.83699999999999,"15min":216.83699999999999},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.88700000000000001,"5min":3.5920000000000001,"15min":2.0910000000000002},"min":{"1min":0.33400000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":3.6789999999999998,"5min":216.816,"15min":216.816},"last":0.41599999999999998}]},{"osd":6,"last update":"Tue Sep 17 07:18:24 2024","interfaces":[{"interface":"back","average":{"1min":1.103,"5min":2.2970000000000002,"15min":1.5620000000000001},"min":{"1min":0.33900000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":3.3889999999999998,"5min":147.13200000000001,"15min":147.13200000000001},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.76300000000000001,"5min":2.2549999999999999,"15min":1.577},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":2.1309999999999998,"5min":147.172,"15min":147.172},"last":0.312}]},{"osd":7,"last update":"Tue Sep 17 07:18:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2070000000000001,"5min":1.236,"15min":1.4390000000000001},"min":{"1min":0.40799999999999997,"5min":0.314,"15min":0.29599999999999999},"max":{"1min":4.3490000000000002,"5min":4.484,"15min":46.183},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.96999999999999997,"5min":1.1579999999999999,"15min":1.3879999999999999},"min":{"1min":0.38800000000000001,"5min":0.32600000000000001,"15min":0.32000000000000001},"max":{"1min":3.528,"5min":5.2149999999999999,"15min":45.914999999999999},"last":0.52700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2063142912,"data_stored":2063138840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1525153792,"data_stored":1525153792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2037256192,"data_stored":2037252120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":898957312,"data_stored":898957312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1612582912,"data_stored":1612578840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":910229504,"data_stored":910229504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T07:18:58.683 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-17T07:18:58.684 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd pool set unique_pool_3 pg_num 36 2024-09-17T07:18:58.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:58.685 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:58.685 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:18:58.786 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:18:58.812 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:58.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:58.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:18:58.913 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:18:58.955 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:58.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:58.956 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:18:59.057 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:18:59.097 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:59.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:59.098 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:18:59.199 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:18:59.224 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:59.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:59.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:18:59.251 INFO:teuthology.orchestra.run.smithi135.stdout:{ 2024-09-17T07:18:59.251 INFO:teuthology.orchestra.run.smithi135.stdout: "ops": [ 2024-09-17T07:18:59.251 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:18:59.251 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_repop(osd.5.0:171214 10.6 e357/332 10:64fda379:::benchmark_data_smithi100_155039_object122415:head v 357'11113, mlcod=357'11102)", 2024-09-17T07:18:59.252 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:18:59.245780+0000", 2024-09-17T07:18:59.252 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 0.0013189359999999999, 2024-09-17T07:18:59.252 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00134698, 2024-09-17T07:18:59.252 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:18:59.252 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "started", 2024-09-17T07:18:59.252 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:18:59.252 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:18:59.252 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:18:59.252 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:18:59.245780+0000", 2024-09-17T07:18:59.252 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:18:59.252 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:18:59.253 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:18:59.253 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:18:59.253 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:18:59.245780+0000", 2024-09-17T07:18:59.253 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:18:59.253 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:18:59.253 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:18:59.253 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:18:59.253 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:18:59.245779+0000", 2024-09-17T07:18:59.253 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.999999 2024-09-17T07:18:59.253 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:18:59.253 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:18:59.253 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:18:59.254 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:18:59.245786+0000", 2024-09-17T07:18:59.254 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.8530000000000003e-06 2024-09-17T07:18:59.254 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:18:59.254 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:18:59.254 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:18:59.254 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:18:59.245795+0000", 2024-09-17T07:18:59.254 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.7799999999999995e-06 2024-09-17T07:18:59.254 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:18:59.254 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:18:59.254 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:18:59.254 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:18:59.245802+0000", 2024-09-17T07:18:59.254 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.6649999999999997e-06 2024-09-17T07:18:59.254 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:18:59.255 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:18:59.255 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:18:59.255 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:18:59.245837+0000", 2024-09-17T07:18:59.255 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.4700999999999999e-05 2024-09-17T07:18:59.255 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:18:59.255 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:18:59.255 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:18:59.255 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:18:59.245855+0000", 2024-09-17T07:18:59.255 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.8464000000000001e-05 2024-09-17T07:18:59.255 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:18:59.255 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:18:59.255 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:18:59.255 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:18:59.256 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:18:59.256 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_repop(osd.5.0:171216 10.6 e357/332)", 2024-09-17T07:18:59.256 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:18:59.245814+0000", 2024-09-17T07:18:59.256 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 0.00128447, 2024-09-17T07:18:59.256 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0013742699999999999, 2024-09-17T07:18:59.256 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:18:59.256 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "queued for pg", 2024-09-17T07:18:59.256 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:18:59.256 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:18:59.256 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:18:59.256 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:18:59.245814+0000", 2024-09-17T07:18:59.256 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:18:59.256 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:18:59.257 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:18:59.257 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:18:59.257 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:18:59.245814+0000", 2024-09-17T07:18:59.257 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:18:59.257 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:18:59.257 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:18:59.257 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:18:59.257 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:18:59.245814+0000", 2024-09-17T07:18:59.257 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967296 2024-09-17T07:18:59.257 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:18:59.257 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:18:59.257 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:18:59.258 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:18:59.245818+0000", 2024-09-17T07:18:59.258 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.8190000000000002e-06 2024-09-17T07:18:59.258 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:18:59.258 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:18:59.258 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:18:59.258 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:18:59.245822+0000", 2024-09-17T07:18:59.258 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.6399999999999996e-06 2024-09-17T07:18:59.258 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:18:59.258 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:18:59.258 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:18:59.258 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:18:59.245826+0000", 2024-09-17T07:18:59.258 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.9129999999999996e-06 2024-09-17T07:18:59.258 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:18:59.259 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:18:59.259 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:18:59.259 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:18:59.259 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:18:59.259 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.15648.0:203548 10.e 10.9a1cdace (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e357)", 2024-09-17T07:18:59.259 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:18:59.246329+0000", 2024-09-17T07:18:59.259 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 0.00076950399999999996, 2024-09-17T07:18:59.259 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00090119899999999999, 2024-09-17T07:18:59.259 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:18:59.259 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "queued for pg", 2024-09-17T07:18:59.259 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:18:59.259 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.15648", 2024-09-17T07:18:59.260 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/1684940078", 2024-09-17T07:18:59.260 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 203548 2024-09-17T07:18:59.260 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:18:59.260 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:18:59.260 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:18:59.260 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:18:59.260 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:18:59.246329+0000", 2024-09-17T07:18:59.260 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:18:59.260 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:18:59.260 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:18:59.260 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:18:59.260 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:18:59.246329+0000", 2024-09-17T07:18:59.261 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:18:59.261 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:18:59.261 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:18:59.261 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:18:59.261 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:18:59.246328+0000", 2024-09-17T07:18:59.261 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:18:59.261 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:18:59.261 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:18:59.261 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:18:59.261 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:18:59.246333+0000", 2024-09-17T07:18:59.261 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.553e-06 2024-09-17T07:18:59.261 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:18:59.262 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:18:59.262 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:18:59.262 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:18:59.246349+0000", 2024-09-17T07:18:59.262 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.5722e-05 2024-09-17T07:18:59.262 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:18:59.262 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:18:59.262 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:18:59.262 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:18:59.246356+0000", 2024-09-17T07:18:59.262 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.6090000000000003e-06 2024-09-17T07:18:59.262 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:18:59.262 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:18:59.262 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:18:59.262 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:18:59.262 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:18:59.263 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.15648.0:203540 10.19 10:9d142495:::benchmark_data_smithi100_155039_object80192:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e357)", 2024-09-17T07:18:59.263 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:18:59.242942+0000", 2024-09-17T07:18:59.263 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 0.0041562650000000001, 2024-09-17T07:18:59.263 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0043468980000000001, 2024-09-17T07:18:59.263 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:18:59.263 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "waiting for sub ops", 2024-09-17T07:18:59.263 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:18:59.263 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.15648", 2024-09-17T07:18:59.263 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/1684940078", 2024-09-17T07:18:59.263 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 203540 2024-09-17T07:18:59.263 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:18:59.263 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:18:59.263 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:18:59.264 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:18:59.264 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:18:59.242942+0000", 2024-09-17T07:18:59.264 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:18:59.264 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:18:59.264 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:18:59.264 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:18:59.264 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:18:59.242942+0000", 2024-09-17T07:18:59.264 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:18:59.264 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:18:59.264 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:18:59.264 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:18:59.264 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:18:59.242940+0000", 2024-09-17T07:18:59.265 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999981 2024-09-17T07:18:59.265 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:18:59.265 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:18:59.265 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:18:59.265 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:18:59.242951+0000", 2024-09-17T07:18:59.265 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0305e-05 2024-09-17T07:18:59.265 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:18:59.265 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:18:59.265 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:18:59.265 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:18:59.242963+0000", 2024-09-17T07:18:59.265 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.2293000000000001e-05 2024-09-17T07:18:59.265 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:18:59.266 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:18:59.266 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:18:59.266 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:18:59.242970+0000", 2024-09-17T07:18:59.266 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.3300000000000001e-06 2024-09-17T07:18:59.266 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:18:59.266 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:18:59.266 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:18:59.266 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:18:59.243030+0000", 2024-09-17T07:18:59.266 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.9787999999999997e-05 2024-09-17T07:18:59.266 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:18:59.266 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:18:59.266 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:18:59.267 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:18:59.243155+0000", 2024-09-17T07:18:59.267 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00012464799999999999 2024-09-17T07:18:59.267 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:18:59.267 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:18:59.267 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,2", 2024-09-17T07:18:59.267 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:18:59.243569+0000", 2024-09-17T07:18:59.267 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000414249 2024-09-17T07:18:59.267 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:18:59.267 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:18:59.267 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:18:59.267 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:18:59.246425+0000", 2024-09-17T07:18:59.267 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0028557589999999998 2024-09-17T07:18:59.268 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:18:59.268 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:18:59.268 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:18:59.268 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:18:59.268 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:18:59.268 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.15648.0:203541 10.2 10:44b54876:::benchmark_data_smithi100_155039_object80193:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e357)", 2024-09-17T07:18:59.268 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:18:59.242987+0000", 2024-09-17T07:18:59.268 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 0.0041113579999999999, 2024-09-17T07:18:59.268 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0043804830000000001, 2024-09-17T07:18:59.268 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:18:59.268 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "waiting for sub ops", 2024-09-17T07:18:59.268 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:18:59.269 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.15648", 2024-09-17T07:18:59.269 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/1684940078", 2024-09-17T07:18:59.269 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 203541 2024-09-17T07:18:59.269 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:18:59.269 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:18:59.269 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:18:59.269 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:18:59.269 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:18:59.242987+0000", 2024-09-17T07:18:59.269 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:18:59.269 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:18:59.269 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:18:59.269 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:18:59.269 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:18:59.242987+0000", 2024-09-17T07:18:59.270 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:18:59.270 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:18:59.270 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:18:59.270 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:18:59.270 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:18:59.242987+0000", 2024-09-17T07:18:59.270 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999995 2024-09-17T07:18:59.270 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:18:59.270 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:18:59.270 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:18:59.270 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:18:59.242990+0000", 2024-09-17T07:18:59.270 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.8020000000000002e-06 2024-09-17T07:18:59.270 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:18:59.270 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:18:59.271 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:18:59.271 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:18:59.243003+0000", 2024-09-17T07:18:59.271 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.2244e-05 2024-09-17T07:18:59.271 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:18:59.271 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:18:59.271 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:18:59.271 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:18:59.243008+0000", 2024-09-17T07:18:59.271 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.8459999999999996e-06 2024-09-17T07:18:59.271 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:18:59.271 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:18:59.271 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:18:59.271 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:18:59.243055+0000", 2024-09-17T07:18:59.272 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.6638999999999998e-05 2024-09-17T07:18:59.272 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:18:59.272 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:18:59.272 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:18:59.272 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:18:59.243175+0000", 2024-09-17T07:18:59.272 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00011955600000000001 2024-09-17T07:18:59.272 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:18:59.272 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:18:59.272 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0", 2024-09-17T07:18:59.272 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:18:59.243569+0000", 2024-09-17T07:18:59.272 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00039417800000000002 2024-09-17T07:18:59.272 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:18:59.272 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:18:59.273 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:18:59.273 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:18:59.273 INFO:teuthology.orchestra.run.smithi135.stdout: ], 2024-09-17T07:18:59.273 INFO:teuthology.orchestra.run.smithi135.stdout: "num_ops": 5 2024-09-17T07:18:59.273 INFO:teuthology.orchestra.run.smithi135.stdout:} 2024-09-17T07:18:59.326 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:18:59.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:59.366 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:59.366 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:18:59.429 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:18:59 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:18:59.467 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:18:59.504 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:59.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:59.505 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:18:59.606 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:18:59.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:59 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:18:59.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:18:59 smithi100 ceph-mon[29092]: pgmap v768: 158 pgs: 1 active+clean+scrubbing+deep, 1 active+undersized+degraded+remapped+backfill_toofull, 7 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 85 active+clean; 2.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.4 KiB/s rd, 1 op/s; 8642/133917 objects degraded (6.453%); 12760/133917 objects misplaced (9.528%) 2024-09-17T07:18:59.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:59 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:18:59.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:18:59 smithi100 ceph-mon[39462]: pgmap v768: 158 pgs: 1 active+clean+scrubbing+deep, 1 active+undersized+degraded+remapped+backfill_toofull, 7 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 85 active+clean; 2.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.4 KiB/s rd, 1 op/s; 8642/133917 objects degraded (6.453%); 12760/133917 objects misplaced (9.528%) 2024-09-17T07:18:59.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:59.645 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:59.645 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:18:59.683 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:59 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:18:59.683 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:18:59 smithi135 ceph-mon[33238]: pgmap v768: 158 pgs: 1 active+clean+scrubbing+deep, 1 active+undersized+degraded+remapped+backfill_toofull, 7 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 85 active+clean; 2.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.4 KiB/s rd, 1 op/s; 8642/133917 objects degraded (6.453%); 12760/133917 objects misplaced (9.528%) 2024-09-17T07:18:59.746 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:18:59.779 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:18:59.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:59.780 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:18:59.825 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:18:59.881 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:18:59.907 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:18:59.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:18:59.908 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:19:00.009 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:19:00.039 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:00.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:00.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:19:00.142 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:19:00.179 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:00.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:00.180 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:19:00.281 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:19:00.307 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:00.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:00.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:19:00.409 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:19:00.450 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:00.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:00.451 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:19:00.552 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:19:00.581 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:00.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:00.582 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:19:00.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:00 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 8642/133917 objects degraded (6.453%), 29 pgs degraded, 65 pgs undersized (PG_DEGRADED) 2024-09-17T07:19:00.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:00 smithi100 ceph-mon[39462]: osdmap e358: 8 total, 7 up, 7 in 2024-09-17T07:19:00.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:00 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 8642/133917 objects degraded (6.453%), 29 pgs degraded, 65 pgs undersized (PG_DEGRADED) 2024-09-17T07:19:00.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:00 smithi100 ceph-mon[29092]: osdmap e358: 8 total, 7 up, 7 in 2024-09-17T07:19:00.683 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:19:00.710 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:00.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:00.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:19:00.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:00 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 8642/133917 objects degraded (6.453%), 29 pgs degraded, 65 pgs undersized (PG_DEGRADED) 2024-09-17T07:19:00.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:00 smithi135 ceph-mon[33238]: osdmap e358: 8 total, 7 up, 7 in 2024-09-17T07:19:00.813 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:19:00.840 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:00.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:00.841 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:19:00.942 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:19:00.980 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:00.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:00.981 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:19:01.082 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:19:01.116 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:01.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:01.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:19:01.218 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:19:01.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:01.259 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:01.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:19:01.275 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-17T07:19:01.360 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:19:01.393 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:01.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:01.394 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:19:01.495 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:19:01.529 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:01.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:01.530 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:19:01.631 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:19:01.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:01.671 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:01.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:19:01.720 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:01 smithi100 ceph-mon[29092]: pgmap v770: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+clean+scrubbing+deep, 7 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 85 active+clean; 2.7 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 736 B/s rd, 0 B/s wr, 884 op/s; 7829/132024 objects degraded (5.930%); 11452/132024 objects misplaced (8.674%); 4.6 MiB/s, 74 objects/s recovering 2024-09-17T07:19:01.720 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:01 smithi100 ceph-mon[39462]: pgmap v770: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+clean+scrubbing+deep, 7 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 85 active+clean; 2.7 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 736 B/s rd, 0 B/s wr, 884 op/s; 7829/132024 objects degraded (5.930%); 11452/132024 objects misplaced (8.674%); 4.6 MiB/s, 74 objects/s recovering 2024-09-17T07:19:01.773 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:19:01.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:01 smithi135 ceph-mon[33238]: pgmap v770: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+clean+scrubbing+deep, 7 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 85 active+clean; 2.7 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 736 B/s rd, 0 B/s wr, 884 op/s; 7829/132024 objects degraded (5.930%); 11452/132024 objects misplaced (8.674%); 4.6 MiB/s, 74 objects/s recovering 2024-09-17T07:19:01.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:01.816 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:01.817 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:19:01.904 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-17T07:19:01.918 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:19:01.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:01.993 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:01.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:19:02.094 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:19:02.137 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:02.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:02.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:19:02.239 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:19:02.247 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:19:02.247 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:19:02.247 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:19:02.247 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:19:02.247 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:19:02.247 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:19:02.247 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:19:02.247 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:19:02.247 INFO:teuthology.orchestra.run.smithi100.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-17T07:19:02.247 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:19:02.248 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:19:02.248 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:19:02.248 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:19:02.248 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:19:02.248 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:19:02.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:02.296 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:02.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:19:02.325 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:19:02.396 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:19:02.436 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:02.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:02.437 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:19:02.454 INFO:teuthology.orchestra.run.smithi100.stderr:set pool 10 pg_num to 36 2024-09-17T07:19:02.538 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:19:02.566 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:02.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:02.567 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:19:02.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:02 smithi100 ceph-mon[29092]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-17T07:19:02.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:02 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/3959381065' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:19:02.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:02 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2437720984' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "36"}]: dispatch 2024-09-17T07:19:02.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:02 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "36"}]: dispatch 2024-09-17T07:19:02.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:02 smithi100 ceph-mon[39462]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-17T07:19:02.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:02 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/3959381065' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:19:02.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:02 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2437720984' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "36"}]: dispatch 2024-09-17T07:19:02.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:02 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "36"}]: dispatch 2024-09-17T07:19:02.668 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:19:02.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:02.704 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:02.704 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:19:02.718 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:02 smithi135 ceph-mon[33238]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-17T07:19:02.718 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:02 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/3959381065' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:19:02.718 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:02 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2437720984' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "36"}]: dispatch 2024-09-17T07:19:02.719 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:02 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "36"}]: dispatch 2024-09-17T07:19:02.805 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:19:02.834 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:02.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:02.835 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:19:02.936 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:19:02.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:02.976 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:02.976 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:19:03.078 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:19:03.118 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:03.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:03.119 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:19:03.220 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:19:03.265 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:03.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:03.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:19:03.367 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:19:03.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:03.389 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:03.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:19:03.490 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:19:03.534 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:03.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:03.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:19:03.636 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:19:03.672 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:03.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:03.673 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:19:03.774 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:19:03.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:03 smithi135 ceph-mon[33238]: 10.6 deep-scrub ok 2024-09-17T07:19:03.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:03 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "36"}]': finished 2024-09-17T07:19:03.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:03 smithi135 ceph-mon[33238]: osdmap e359: 8 total, 7 up, 7 in 2024-09-17T07:19:03.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:03 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:19:03.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:03 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:19:03.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:03 smithi135 ceph-mon[33238]: pgmap v772: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+clean+scrubbing+deep, 7 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 85 active+clean; 2.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 738 B/s rd, 0 B/s wr, 1.04k op/s; 7642/126369 objects degraded (6.047%); 10667/126369 objects misplaced (8.441%); 4.7 MiB/s, 74 objects/s recovering 2024-09-17T07:19:03.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:03 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "36"}]: dispatch 2024-09-17T07:19:03.796 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:03.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:03.797 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:19:03.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:03 smithi100 ceph-mon[29092]: 10.6 deep-scrub ok 2024-09-17T07:19:03.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:03 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "36"}]': finished 2024-09-17T07:19:03.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:03 smithi100 ceph-mon[29092]: osdmap e359: 8 total, 7 up, 7 in 2024-09-17T07:19:03.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:03 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:19:03.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:03 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:19:03.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:03 smithi100 ceph-mon[29092]: pgmap v772: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+clean+scrubbing+deep, 7 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 85 active+clean; 2.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 738 B/s rd, 0 B/s wr, 1.04k op/s; 7642/126369 objects degraded (6.047%); 10667/126369 objects misplaced (8.441%); 4.7 MiB/s, 74 objects/s recovering 2024-09-17T07:19:03.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:03 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "36"}]: dispatch 2024-09-17T07:19:03.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:03 smithi100 ceph-mon[39462]: 10.6 deep-scrub ok 2024-09-17T07:19:03.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:03 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "36"}]': finished 2024-09-17T07:19:03.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:03 smithi100 ceph-mon[39462]: osdmap e359: 8 total, 7 up, 7 in 2024-09-17T07:19:03.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:03 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:19:03.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:03 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:19:03.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:03 smithi100 ceph-mon[39462]: pgmap v772: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+clean+scrubbing+deep, 7 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 85 active+clean; 2.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 738 B/s rd, 0 B/s wr, 1.04k op/s; 7642/126369 objects degraded (6.047%); 10667/126369 objects misplaced (8.441%); 4.7 MiB/s, 74 objects/s recovering 2024-09-17T07:19:03.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:03 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "36"}]: dispatch 2024-09-17T07:19:03.898 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:19:03.921 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:03.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:03.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:19:04.023 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:19:04.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:04.052 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:04.052 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:19:04.153 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:19:04.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:04.194 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:04.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:19:04.295 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:19:04.340 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:04.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:04.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:19:04.441 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:19:04.479 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:04.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:04.480 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:19:04.581 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:19:04.602 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:04.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:04.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:19:04.704 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:19:04.725 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:04.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:04.726 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:19:04.827 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:19:04.847 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:04.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:04.849 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:19:04.949 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:19:04.972 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:04.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:04.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:19:05.073 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:19:05.097 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:05.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:05.098 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:19:05.199 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:19:05.224 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:05.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:05.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:19:05.325 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:19:05.351 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:05.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:05.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:19:05.453 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:19:05.491 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:05.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:05.492 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:19:05.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:05 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "36"}]': finished 2024-09-17T07:19:05.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:05 smithi135 ceph-mon[33238]: osdmap e360: 8 total, 7 up, 7 in 2024-09-17T07:19:05.593 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:19:05.626 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:05.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:05 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "36"}]': finished 2024-09-17T07:19:05.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:05 smithi100 ceph-mon[29092]: osdmap e360: 8 total, 7 up, 7 in 2024-09-17T07:19:05.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:05 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "36"}]': finished 2024-09-17T07:19:05.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:05 smithi100 ceph-mon[39462]: osdmap e360: 8 total, 7 up, 7 in 2024-09-17T07:19:05.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:05.628 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:19:05.729 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:19:05.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:05.767 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:05.768 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:19:05.772 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:19:05.868 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:19:05.909 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:05.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:05.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:19:06.011 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:19:06.032 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:06.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:06.033 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:19:06.127 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:19:05 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:19:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:19:06.134 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:19:06.176 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:06.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:06.178 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:19:06.254 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:19:06.255 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:19:06.278 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:19:06.311 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:06.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:06.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:19:06.413 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:19:06.436 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:06.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:06.437 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:19:06.493 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:06 smithi100 ceph-mon[29092]: osdmap e361: 8 total, 7 up, 7 in 2024-09-17T07:19:06.493 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:06 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 7642/126369 objects degraded (6.047%), 29 pgs degraded, 65 pgs undersized (PG_DEGRADED) 2024-09-17T07:19:06.493 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:06 smithi100 ceph-mon[29092]: osdmap e362: 8 total, 7 up, 7 in 2024-09-17T07:19:06.493 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:06 smithi100 ceph-mon[29092]: pgmap v776: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 10 unknown, 7 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 86 active+clean; 2.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 6410/116844 objects degraded (5.486%); 9167/116844 objects misplaced (7.846%); 7.9 MiB/s, 125 objects/s recovering 2024-09-17T07:19:06.493 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:06 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "36"}]: dispatch 2024-09-17T07:19:06.494 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:06 smithi100 ceph-mon[39462]: osdmap e361: 8 total, 7 up, 7 in 2024-09-17T07:19:06.494 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:06 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 7642/126369 objects degraded (6.047%), 29 pgs degraded, 65 pgs undersized (PG_DEGRADED) 2024-09-17T07:19:06.494 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:06 smithi100 ceph-mon[39462]: osdmap e362: 8 total, 7 up, 7 in 2024-09-17T07:19:06.494 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:06 smithi100 ceph-mon[39462]: pgmap v776: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 10 unknown, 7 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 86 active+clean; 2.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 6410/116844 objects degraded (5.486%); 9167/116844 objects misplaced (7.846%); 7.9 MiB/s, 125 objects/s recovering 2024-09-17T07:19:06.494 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:06 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "36"}]: dispatch 2024-09-17T07:19:06.539 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:19:06.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:06 smithi135 ceph-mon[33238]: osdmap e361: 8 total, 7 up, 7 in 2024-09-17T07:19:06.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:06 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 7642/126369 objects degraded (6.047%), 29 pgs degraded, 65 pgs undersized (PG_DEGRADED) 2024-09-17T07:19:06.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:06 smithi135 ceph-mon[33238]: osdmap e362: 8 total, 7 up, 7 in 2024-09-17T07:19:06.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:06 smithi135 ceph-mon[33238]: pgmap v776: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 10 unknown, 7 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 86 active+clean; 2.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 6410/116844 objects degraded (5.486%); 9167/116844 objects misplaced (7.846%); 7.9 MiB/s, 125 objects/s recovering 2024-09-17T07:19:06.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:06 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "36"}]: dispatch 2024-09-17T07:19:06.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:06.560 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:06.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:19:06.661 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:19:06.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:06.683 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:06.683 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:19:06.784 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:19:06.806 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:06.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:06.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:19:06.908 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:19:06.930 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:06.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:06.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:19:07.032 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:19:07.053 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:07.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:07.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:19:07.155 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:19:07.178 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:07.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:07.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:19:07.280 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:19:07.308 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:07.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:07.309 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:19:07.410 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:19:07.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:07.432 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:07.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:19:07.533 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:19:07.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:07 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "36"}]': finished 2024-09-17T07:19:07.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:07 smithi135 ceph-mon[33238]: osdmap e363: 8 total, 7 up, 7 in 2024-09-17T07:19:07.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:07 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:19:07.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:07.555 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:07.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:19:07.613 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:07 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "36"}]': finished 2024-09-17T07:19:07.613 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:07 smithi100 ceph-mon[29092]: osdmap e363: 8 total, 7 up, 7 in 2024-09-17T07:19:07.614 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:07 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:19:07.614 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:07 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "36"}]': finished 2024-09-17T07:19:07.614 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:07 smithi100 ceph-mon[39462]: osdmap e363: 8 total, 7 up, 7 in 2024-09-17T07:19:07.614 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:07 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:19:07.656 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:19:07.678 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:07.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:07.679 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:19:07.779 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:19:07.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:07.802 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:07.802 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:19:07.903 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:19:07.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:07.926 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:07.926 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:19:08.027 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:19:08.050 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:08.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:08.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:19:08.152 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:19:08.174 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:08.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:08.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:19:08.256 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:19:08.276 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:19:08.302 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:08.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:08.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:19:08.308 INFO:teuthology.orchestra.run.smithi100.stderr:nodeep-scrub is set 2024-09-17T07:19:08.404 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:19:08.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:08.457 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:08.457 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:19:08.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:08 smithi135 ceph-mon[33238]: osdmap e364: 8 total, 7 up, 7 in 2024-09-17T07:19:08.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:08 smithi135 ceph-mon[33238]: pgmap v779: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 10 unknown, 7 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 86 active+clean; 2.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 198 op/s; 6410/116844 objects degraded (5.486%); 9167/116844 objects misplaced (7.846%); 10 MiB/s, 160 objects/s recovering 2024-09-17T07:19:08.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:08 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:19:08.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:08 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:19:08.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:08 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/3566208024' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:19:08.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:08 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:19:08.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:08 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:19:08.558 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:19:08.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:08.602 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:08.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:19:08.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:08 smithi100 ceph-mon[29092]: osdmap e364: 8 total, 7 up, 7 in 2024-09-17T07:19:08.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:08 smithi100 ceph-mon[29092]: pgmap v779: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 10 unknown, 7 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 86 active+clean; 2.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 198 op/s; 6410/116844 objects degraded (5.486%); 9167/116844 objects misplaced (7.846%); 10 MiB/s, 160 objects/s recovering 2024-09-17T07:19:08.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:08 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:19:08.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:08 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:19:08.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:08 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/3566208024' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:19:08.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:08 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:19:08.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:08 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:19:08.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:08 smithi100 ceph-mon[39462]: osdmap e364: 8 total, 7 up, 7 in 2024-09-17T07:19:08.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:08 smithi100 ceph-mon[39462]: pgmap v779: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 10 unknown, 7 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 86 active+clean; 2.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 198 op/s; 6410/116844 objects degraded (5.486%); 9167/116844 objects misplaced (7.846%); 10 MiB/s, 160 objects/s recovering 2024-09-17T07:19:08.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:08 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:19:08.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:08 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:19:08.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:08 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/3566208024' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:19:08.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:08 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:19:08.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:08 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:19:08.703 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:19:08.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:08.746 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:08.746 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:19:08.847 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:19:08.894 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:08.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:08.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:19:08.995 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:19:09.044 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:09.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:09.046 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:19:09.089 INFO:teuthology.orchestra.run.smithi135.stdout:ERROR: (22) Invalid argument 2024-09-17T07:19:09.089 INFO:teuthology.orchestra.run.smithi135.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:19:09.091 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:19:09.146 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:19:09.171 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:19:09 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:19:09.180 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:09.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:09.181 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:19:09.282 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:19:09.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:09.316 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:09.316 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:19:09.417 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:19:09.423 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:09 smithi135 ceph-mon[33238]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:19:09.424 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:09 smithi135 ceph-mon[33238]: 10.23 starting backfill to osd.7 from (0'0,0'0] MAX to 362'13385 2024-09-17T07:19:09.424 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:09 smithi135 ceph-mon[33238]: 10.1e starting backfill to osd.0 from (0'0,0'0] MAX to 362'15953 2024-09-17T07:19:09.424 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:09 smithi135 ceph-mon[33238]: 10.1f starting backfill to osd.7 from (0'0,0'0] MAX to 362'16154 2024-09-17T07:19:09.424 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:09 smithi135 ceph-mon[33238]: 10.22 starting backfill to osd.5 from (0'0,0'0] MAX to 362'10335 2024-09-17T07:19:09.424 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:09 smithi135 ceph-mon[33238]: 10.1b starting backfill to osd.4 from (0'0,0'0] MAX to 362'13064 2024-09-17T07:19:09.424 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:09 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/3566208024' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T07:19:09.424 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:09 smithi135 ceph-mon[33238]: osdmap e365: 8 total, 7 up, 7 in 2024-09-17T07:19:09.424 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:09 smithi135 ceph-mon[33238]: 10.20 starting backfill to osd.0 from (0'0,0'0] MAX to 362'10406 2024-09-17T07:19:09.424 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:09 smithi135 ceph-mon[33238]: 10.21 starting backfill to osd.2 from (0'0,0'0] MAX to 362'10699 2024-09-17T07:19:09.424 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:09 smithi135 ceph-mon[33238]: 10.1c starting backfill to osd.6 from (0'0,0'0] MAX to 362'13160 2024-09-17T07:19:09.424 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:09 smithi135 ceph-mon[33238]: 10.1e starting backfill to osd.5 from (0'0,0'0] MAX to 362'15953 2024-09-17T07:19:09.424 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:09 smithi135 ceph-mon[33238]: 10.1b starting backfill to osd.5 from (0'0,0'0] MAX to 362'13064 2024-09-17T07:19:09.424 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:09 smithi135 ceph-mon[33238]: 10.20 starting backfill to osd.2 from (0'0,0'0] MAX to 362'10406 2024-09-17T07:19:09.425 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:09 smithi135 ceph-mon[33238]: 10.22 starting backfill to osd.7 from (0'0,0'0] MAX to 362'10335 2024-09-17T07:19:09.425 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:09 smithi135 ceph-mon[33238]: 10.1b starting backfill to osd.7 from (0'0,0'0] MAX to 362'13064 2024-09-17T07:19:09.425 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:09 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:19:09.425 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:09 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:19:09.425 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:09 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:19:09.448 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:09.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:09.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:19:09.550 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:19:09.590 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:09.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:09.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:19:09.599 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:09 smithi100 ceph-mon[29092]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:19:09.599 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:09 smithi100 ceph-mon[29092]: 10.23 starting backfill to osd.7 from (0'0,0'0] MAX to 362'13385 2024-09-17T07:19:09.599 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:09 smithi100 ceph-mon[29092]: 10.1e starting backfill to osd.0 from (0'0,0'0] MAX to 362'15953 2024-09-17T07:19:09.599 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:09 smithi100 ceph-mon[29092]: 10.1f starting backfill to osd.7 from (0'0,0'0] MAX to 362'16154 2024-09-17T07:19:09.600 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:09 smithi100 ceph-mon[29092]: 10.22 starting backfill to osd.5 from (0'0,0'0] MAX to 362'10335 2024-09-17T07:19:09.600 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:09 smithi100 ceph-mon[29092]: 10.1b starting backfill to osd.4 from (0'0,0'0] MAX to 362'13064 2024-09-17T07:19:09.600 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:09 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/3566208024' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T07:19:09.600 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:09 smithi100 ceph-mon[29092]: osdmap e365: 8 total, 7 up, 7 in 2024-09-17T07:19:09.600 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:09 smithi100 ceph-mon[29092]: 10.20 starting backfill to osd.0 from (0'0,0'0] MAX to 362'10406 2024-09-17T07:19:09.600 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:09 smithi100 ceph-mon[29092]: 10.21 starting backfill to osd.2 from (0'0,0'0] MAX to 362'10699 2024-09-17T07:19:09.600 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:09 smithi100 ceph-mon[29092]: 10.1c starting backfill to osd.6 from (0'0,0'0] MAX to 362'13160 2024-09-17T07:19:09.600 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:09 smithi100 ceph-mon[29092]: 10.1e starting backfill to osd.5 from (0'0,0'0] MAX to 362'15953 2024-09-17T07:19:09.600 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:09 smithi100 ceph-mon[29092]: 10.1b starting backfill to osd.5 from (0'0,0'0] MAX to 362'13064 2024-09-17T07:19:09.600 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:09 smithi100 ceph-mon[29092]: 10.20 starting backfill to osd.2 from (0'0,0'0] MAX to 362'10406 2024-09-17T07:19:09.600 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:09 smithi100 ceph-mon[29092]: 10.22 starting backfill to osd.7 from (0'0,0'0] MAX to 362'10335 2024-09-17T07:19:09.600 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:09 smithi100 ceph-mon[29092]: 10.1b starting backfill to osd.7 from (0'0,0'0] MAX to 362'13064 2024-09-17T07:19:09.600 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:09 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:19:09.601 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:09 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:19:09.601 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:09 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:19:09.601 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:09 smithi100 ceph-mon[39462]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:19:09.601 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:09 smithi100 ceph-mon[39462]: 10.23 starting backfill to osd.7 from (0'0,0'0] MAX to 362'13385 2024-09-17T07:19:09.601 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:09 smithi100 ceph-mon[39462]: 10.1e starting backfill to osd.0 from (0'0,0'0] MAX to 362'15953 2024-09-17T07:19:09.601 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:09 smithi100 ceph-mon[39462]: 10.1f starting backfill to osd.7 from (0'0,0'0] MAX to 362'16154 2024-09-17T07:19:09.601 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:09 smithi100 ceph-mon[39462]: 10.22 starting backfill to osd.5 from (0'0,0'0] MAX to 362'10335 2024-09-17T07:19:09.601 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:09 smithi100 ceph-mon[39462]: 10.1b starting backfill to osd.4 from (0'0,0'0] MAX to 362'13064 2024-09-17T07:19:09.601 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:09 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/3566208024' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T07:19:09.601 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:09 smithi100 ceph-mon[39462]: osdmap e365: 8 total, 7 up, 7 in 2024-09-17T07:19:09.601 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:09 smithi100 ceph-mon[39462]: 10.20 starting backfill to osd.0 from (0'0,0'0] MAX to 362'10406 2024-09-17T07:19:09.601 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:09 smithi100 ceph-mon[39462]: 10.21 starting backfill to osd.2 from (0'0,0'0] MAX to 362'10699 2024-09-17T07:19:09.602 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:09 smithi100 ceph-mon[39462]: 10.1c starting backfill to osd.6 from (0'0,0'0] MAX to 362'13160 2024-09-17T07:19:09.602 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:09 smithi100 ceph-mon[39462]: 10.1e starting backfill to osd.5 from (0'0,0'0] MAX to 362'15953 2024-09-17T07:19:09.602 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:09 smithi100 ceph-mon[39462]: 10.1b starting backfill to osd.5 from (0'0,0'0] MAX to 362'13064 2024-09-17T07:19:09.602 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:09 smithi100 ceph-mon[39462]: 10.20 starting backfill to osd.2 from (0'0,0'0] MAX to 362'10406 2024-09-17T07:19:09.602 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:09 smithi100 ceph-mon[39462]: 10.22 starting backfill to osd.7 from (0'0,0'0] MAX to 362'10335 2024-09-17T07:19:09.602 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:09 smithi100 ceph-mon[39462]: 10.1b starting backfill to osd.7 from (0'0,0'0] MAX to 362'13064 2024-09-17T07:19:09.602 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:09 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:19:09.602 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:09 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:19:09.602 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:09 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:19:09.692 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:19:09.731 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:09.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:09.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:19:09.833 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:19:09.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:09.876 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:09.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:19:09.977 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:19:10.014 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-17T07:19:10.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:10.021 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:10.021 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:19:10.122 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:19:10.171 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:10.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:10.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:19:10.274 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:19:10.308 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:10.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:10.310 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:19:10.390 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:19:10.410 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:19:10.439 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:10.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:10.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:19:10.541 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:19:10.572 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:10.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:10.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:19:10.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:10 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:19:10.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:10 smithi100 ceph-mon[29092]: pgmap v781: 168 pgs: 6 remapped+peering, 3 unknown, 7 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 88 active+clean; 2.1 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 3526/101898 objects degraded (3.460%); 8055/101898 objects misplaced (7.905%); 4.5 MiB/s, 71 objects/s recovering 2024-09-17T07:19:10.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:10 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 6410/116844 objects degraded (5.486%), 29 pgs degraded, 65 pgs undersized (PG_DEGRADED) 2024-09-17T07:19:10.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:10 smithi100 ceph-mon[29092]: osdmap e366: 8 total, 7 up, 7 in 2024-09-17T07:19:10.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:10 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:19:10.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:10 smithi100 ceph-mon[39462]: pgmap v781: 168 pgs: 6 remapped+peering, 3 unknown, 7 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 88 active+clean; 2.1 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 3526/101898 objects degraded (3.460%); 8055/101898 objects misplaced (7.905%); 4.5 MiB/s, 71 objects/s recovering 2024-09-17T07:19:10.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:10 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 6410/116844 objects degraded (5.486%), 29 pgs degraded, 65 pgs undersized (PG_DEGRADED) 2024-09-17T07:19:10.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:10 smithi100 ceph-mon[39462]: osdmap e366: 8 total, 7 up, 7 in 2024-09-17T07:19:10.674 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:19:10.719 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:10.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:10.720 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:19:10.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:10 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:19:10.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:10 smithi135 ceph-mon[33238]: pgmap v781: 168 pgs: 6 remapped+peering, 3 unknown, 7 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 88 active+clean; 2.1 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 3526/101898 objects degraded (3.460%); 8055/101898 objects misplaced (7.905%); 4.5 MiB/s, 71 objects/s recovering 2024-09-17T07:19:10.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:10 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 6410/116844 objects degraded (5.486%), 29 pgs degraded, 65 pgs undersized (PG_DEGRADED) 2024-09-17T07:19:10.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:10 smithi135 ceph-mon[33238]: osdmap e366: 8 total, 7 up, 7 in 2024-09-17T07:19:10.805 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7] out_osds: [3] dead_osds: [1] live_osds: [0, 2, 3, 4, 5, 6, 7] 2024-09-17T07:19:10.805 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T07:19:10.805 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-09-17T07:19:10.805 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T07:19:10.821 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:19:10.852 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:10.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:10.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:19:10.954 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:19:10.982 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:10.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:10.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:19:11.069 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T07:19:11.084 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:19:11.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:11.125 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:11.125 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:19:11.226 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:19:11.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:11.265 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:11.265 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:19:11.366 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:19:11.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:11.397 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:11.397 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:19:11.498 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:19:11.541 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:11.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:11.542 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:19:11.643 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:19:11.675 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:11.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:11.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:19:11.701 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:11 smithi100 ceph-mon[39462]: pgmap v783: 168 pgs: 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 6 remapped+peering, 7 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 88 active+clean; 2.2 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 1009 B/s rd, 0 B/s wr, 490 op/s; 4890/109392 objects degraded (4.470%); 13105/109392 objects misplaced (11.980%); 8.4 MiB/s, 134 objects/s recovering 2024-09-17T07:19:11.701 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:11 smithi100 ceph-mon[29092]: pgmap v783: 168 pgs: 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 6 remapped+peering, 7 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 88 active+clean; 2.2 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 1009 B/s rd, 0 B/s wr, 490 op/s; 4890/109392 objects degraded (4.470%); 13105/109392 objects misplaced (11.980%); 8.4 MiB/s, 134 objects/s recovering 2024-09-17T07:19:11.777 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:19:11.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:11 smithi135 ceph-mon[33238]: pgmap v783: 168 pgs: 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 6 remapped+peering, 7 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 88 active+clean; 2.2 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 1009 B/s rd, 0 B/s wr, 490 op/s; 4890/109392 objects degraded (4.470%); 13105/109392 objects misplaced (11.980%); 8.4 MiB/s, 134 objects/s recovering 2024-09-17T07:19:11.818 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:11.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:11.819 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:19:11.920 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:19:11.975 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:11.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:11.976 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:19:12.077 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:19:12.111 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:19:12.111 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:19:12.117 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:12.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:12.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:19:12.181 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:19:12.182 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:19:12.182 INFO:teuthology.orchestra.run.smithi100.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-17T07:19:12.211 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:19:12.212 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:19:12.219 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:19:12.238 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:19:12.239 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:19:12.252 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:12.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:12.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:19:12.279 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:19:12.279 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:19:12.311 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:19:12.311 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:19:12.344 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:19:12.344 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:19:12.355 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:19:12.375 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:19:12.375 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:19:12.390 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:12.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:12.391 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:19:12.492 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:19:12.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:12.541 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:12.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:19:12.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:12 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/298163531' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:19:12.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:12 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/298163531' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:19:12.642 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:19:12.661 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:12 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/298163531' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:19:12.671 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:12.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:12.672 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:19:12.773 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:19:12.810 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:12.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:12.811 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:19:12.840 INFO:teuthology.orchestra.run.smithi135.stdout:{ 2024-09-17T07:19:12.840 INFO:teuthology.orchestra.run.smithi135.stdout: "size": 20, 2024-09-17T07:19:12.840 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 600, 2024-09-17T07:19:12.840 INFO:teuthology.orchestra.run.smithi135.stdout: "ops": [ 2024-09-17T07:19:12.840 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.840 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.24929.0:164758 8.b 8:d5cb077d:::benchmark_data_smithi100_102803_object34254:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-09-17T07:19:12.840 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:10:17.219733+0000", 2024-09-17T07:19:12.840 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 535.61664601899997, 2024-09-17T07:19:12.840 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.98373762600000003, 2024-09-17T07:19:12.840 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:19:12.841 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:19:12.841 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:19:12.841 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.24929", 2024-09-17T07:19:12.841 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/564515582", 2024-09-17T07:19:12.841 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 164758 2024-09-17T07:19:12.841 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.841 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:19:12.841 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.841 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:19:12.841 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.219733+0000", 2024-09-17T07:19:12.841 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:12.841 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.842 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.842 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:19:12.842 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.219733+0000", 2024-09-17T07:19:12.842 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:12.842 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.842 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.842 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:19:12.842 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.219731+0000", 2024-09-17T07:19:12.842 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:19:12.842 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.842 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.842 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:19:12.842 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.219738+0000", 2024-09-17T07:19:12.842 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.6100000000000002e-06 2024-09-17T07:19:12.843 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.843 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.843 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:19:12.843 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.219754+0000", 2024-09-17T07:19:12.843 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.6098000000000001e-05 2024-09-17T07:19:12.843 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.843 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.843 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:19:12.843 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.219761+0000", 2024-09-17T07:19:12.843 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.5340000000000002e-06 2024-09-17T07:19:12.843 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.843 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.844 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:12.844 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.219814+0000", 2024-09-17T07:19:12.844 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.2556999999999998e-05 2024-09-17T07:19:12.844 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.844 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.844 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:19:12.844 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.219823+0000", 2024-09-17T07:19:12.844 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.4730000000000006e-06 2024-09-17T07:19:12.844 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.844 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.844 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:12.844 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.196970+0000", 2024-09-17T07:19:12.845 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.97714677999999999 2024-09-17T07:19:12.845 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.845 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.845 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:19:12.845 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.197099+0000", 2024-09-17T07:19:12.845 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000128703 2024-09-17T07:19:12.845 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.845 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.845 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 1,2", 2024-09-17T07:19:12.845 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.197226+0000", 2024-09-17T07:19:12.845 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00012752799999999999 2024-09-17T07:19:12.845 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.845 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.846 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:19:12.846 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.201464+0000", 2024-09-17T07:19:12.846 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0042372540000000002 2024-09-17T07:19:12.846 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.846 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.846 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:12.846 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.203242+0000", 2024-09-17T07:19:12.846 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.001778083 2024-09-17T07:19:12.846 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.846 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.846 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:12.846 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.203372+0000", 2024-09-17T07:19:12.847 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00013064199999999999 2024-09-17T07:19:12.847 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.847 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.847 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:19:12.847 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.203426+0000", 2024-09-17T07:19:12.847 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.3329999999999999e-05 2024-09-17T07:19:12.847 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.847 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.847 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:19:12.847 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.203470+0000", 2024-09-17T07:19:12.847 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.4638000000000002e-05 2024-09-17T07:19:12.847 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:12.848 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:19:12.848 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:12.848 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.848 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.848 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.24929.0:164774 8.b 8:d8c6bb1d:::benchmark_data_smithi100_102803_object34270:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-09-17T07:19:12.848 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:10:17.230982+0000", 2024-09-17T07:19:12.848 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 535.60539686899995, 2024-09-17T07:19:12.848 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.98469290399999998, 2024-09-17T07:19:12.848 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:19:12.848 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:19:12.848 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:19:12.848 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.24929", 2024-09-17T07:19:12.848 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/564515582", 2024-09-17T07:19:12.849 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 164774 2024-09-17T07:19:12.849 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.849 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:19:12.849 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.849 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:19:12.849 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.230982+0000", 2024-09-17T07:19:12.849 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:12.849 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.849 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.849 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:19:12.849 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.230982+0000", 2024-09-17T07:19:12.849 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:12.850 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.850 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.850 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:19:12.850 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.230980+0000", 2024-09-17T07:19:12.850 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:19:12.850 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.850 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.850 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:19:12.850 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.230986+0000", 2024-09-17T07:19:12.850 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.7139999999999998e-06 2024-09-17T07:19:12.850 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.850 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.851 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:19:12.851 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.230997+0000", 2024-09-17T07:19:12.851 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.1090000000000001e-05 2024-09-17T07:19:12.851 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.851 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.851 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:19:12.851 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.231004+0000", 2024-09-17T07:19:12.851 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.8249999999999999e-06 2024-09-17T07:19:12.851 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.851 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.851 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:12.851 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.231058+0000", 2024-09-17T07:19:12.851 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.3931000000000003e-05 2024-09-17T07:19:12.852 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.852 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.852 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:19:12.852 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.231066+0000", 2024-09-17T07:19:12.852 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.9689999999999994e-06 2024-09-17T07:19:12.852 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.852 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.852 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:12.852 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.199823+0000", 2024-09-17T07:19:12.852 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.96875726900000003 2024-09-17T07:19:12.852 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.852 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.853 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:19:12.853 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.199939+0000", 2024-09-17T07:19:12.853 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000115328 2024-09-17T07:19:12.853 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.853 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.853 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 1,2", 2024-09-17T07:19:12.853 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.200064+0000", 2024-09-17T07:19:12.853 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00012558900000000001 2024-09-17T07:19:12.853 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.853 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.853 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:19:12.853 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.205175+0000", 2024-09-17T07:19:12.853 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0051110360000000002 2024-09-17T07:19:12.853 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.854 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.854 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:12.854 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.207188+0000", 2024-09-17T07:19:12.854 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.002012626 2024-09-17T07:19:12.854 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.854 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.854 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:12.854 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.215606+0000", 2024-09-17T07:19:12.854 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0084185200000000005 2024-09-17T07:19:12.854 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.854 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.854 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:19:12.855 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.215650+0000", 2024-09-17T07:19:12.855 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.4061999999999999e-05 2024-09-17T07:19:12.855 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.855 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.855 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:19:12.856 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.215675+0000", 2024-09-17T07:19:12.856 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.4426999999999998e-05 2024-09-17T07:19:12.856 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:12.856 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:19:12.856 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:12.856 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.856 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.856 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.24929.0:164784 8.b 8:d2e02e23:::benchmark_data_smithi100_102803_object34280:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-09-17T07:19:12.856 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:10:17.240427+0000", 2024-09-17T07:19:12.856 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 535.59595170499995, 2024-09-17T07:19:12.856 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.98057870999999996, 2024-09-17T07:19:12.857 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:19:12.857 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:19:12.857 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:19:12.857 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.24929", 2024-09-17T07:19:12.857 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/564515582", 2024-09-17T07:19:12.857 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 164784 2024-09-17T07:19:12.857 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.857 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:19:12.857 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.857 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:19:12.857 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.240427+0000", 2024-09-17T07:19:12.857 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:12.858 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.858 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.858 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:19:12.858 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.240427+0000", 2024-09-17T07:19:12.858 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:12.858 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.858 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.858 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:19:12.858 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.240425+0000", 2024-09-17T07:19:12.858 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999981 2024-09-17T07:19:12.858 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.858 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.858 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:19:12.859 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.240434+0000", 2024-09-17T07:19:12.859 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.4780000000000008e-06 2024-09-17T07:19:12.859 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.859 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.859 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:19:12.859 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.240449+0000", 2024-09-17T07:19:12.859 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.5153999999999999e-05 2024-09-17T07:19:12.859 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.859 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.859 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:19:12.859 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.240457+0000", 2024-09-17T07:19:12.859 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.5009999999999998e-06 2024-09-17T07:19:12.859 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.860 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.860 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:12.860 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.245473+0000", 2024-09-17T07:19:12.860 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0050163769999999998 2024-09-17T07:19:12.860 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.860 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.860 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:19:12.860 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.245483+0000", 2024-09-17T07:19:12.860 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.9660000000000008e-06 2024-09-17T07:19:12.860 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.860 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.861 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:12.861 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.201492+0000", 2024-09-17T07:19:12.861 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.95600924600000003 2024-09-17T07:19:12.861 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.861 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.861 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:19:12.861 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.201577+0000", 2024-09-17T07:19:12.861 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.4685000000000007e-05 2024-09-17T07:19:12.861 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.861 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.861 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 1,2", 2024-09-17T07:19:12.861 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.201689+0000", 2024-09-17T07:19:12.861 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00011176 2024-09-17T07:19:12.862 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.862 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.862 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:19:12.862 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.205184+0000", 2024-09-17T07:19:12.862 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.003495593 2024-09-17T07:19:12.862 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.862 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.862 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:12.862 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.215528+0000", 2024-09-17T07:19:12.862 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.010343662999999999 2024-09-17T07:19:12.862 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.862 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.863 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:12.863 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.220925+0000", 2024-09-17T07:19:12.863 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.005396858 2024-09-17T07:19:12.863 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.863 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.863 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:19:12.863 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.220969+0000", 2024-09-17T07:19:12.863 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.4382999999999998e-05 2024-09-17T07:19:12.863 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.863 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.863 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:19:12.863 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.221006+0000", 2024-09-17T07:19:12.864 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.6736999999999997e-05 2024-09-17T07:19:12.864 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:12.864 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:19:12.864 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:12.864 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.864 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.864 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.24929.0:164821 8.b 8:d651d42e:::benchmark_data_smithi100_102803_object34317:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-09-17T07:19:12.864 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:10:17.292857+0000", 2024-09-17T07:19:12.864 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 535.54352174200005, 2024-09-17T07:19:12.864 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.92826142099999998, 2024-09-17T07:19:12.864 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:19:12.864 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:19:12.865 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:19:12.865 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.24929", 2024-09-17T07:19:12.865 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/564515582", 2024-09-17T07:19:12.865 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 164821 2024-09-17T07:19:12.865 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.865 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:19:12.865 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.865 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:19:12.865 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.292857+0000", 2024-09-17T07:19:12.865 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:12.865 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.865 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.866 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:19:12.866 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.292857+0000", 2024-09-17T07:19:12.866 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:12.866 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.866 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.866 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:19:12.866 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.292820+0000", 2024-09-17T07:19:12.866 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999628 2024-09-17T07:19:12.866 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.866 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.866 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:19:12.866 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.292933+0000", 2024-09-17T07:19:12.867 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000113736 2024-09-17T07:19:12.867 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.867 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.867 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:19:12.867 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.292945+0000", 2024-09-17T07:19:12.867 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.1202e-05 2024-09-17T07:19:12.867 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.867 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.867 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:19:12.867 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.292952+0000", 2024-09-17T07:19:12.867 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.7270000000000007e-06 2024-09-17T07:19:12.867 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.867 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.868 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:12.868 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.293004+0000", 2024-09-17T07:19:12.868 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.1286999999999999e-05 2024-09-17T07:19:12.868 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.868 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.868 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:19:12.868 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.293014+0000", 2024-09-17T07:19:12.868 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0142e-05 2024-09-17T07:19:12.868 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.868 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.868 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:12.868 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.203517+0000", 2024-09-17T07:19:12.869 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.91050304199999998 2024-09-17T07:19:12.869 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.869 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.869 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:19:12.869 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.203643+0000", 2024-09-17T07:19:12.869 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00012641099999999999 2024-09-17T07:19:12.869 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.869 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.869 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 1,2", 2024-09-17T07:19:12.869 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.203765+0000", 2024-09-17T07:19:12.869 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000121395 2024-09-17T07:19:12.869 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.869 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.870 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:19:12.870 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.210334+0000", 2024-09-17T07:19:12.870 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0065697999999999998 2024-09-17T07:19:12.870 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.870 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.870 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:12.870 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.215566+0000", 2024-09-17T07:19:12.870 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0052315310000000002 2024-09-17T07:19:12.870 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.870 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.870 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:12.870 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.221049+0000", 2024-09-17T07:19:12.870 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0054830199999999999 2024-09-17T07:19:12.871 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.871 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.871 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:19:12.871 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.221083+0000", 2024-09-17T07:19:12.871 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.4251000000000001e-05 2024-09-17T07:19:12.871 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.871 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.871 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:19:12.871 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.221118+0000", 2024-09-17T07:19:12.871 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.5296999999999998e-05 2024-09-17T07:19:12.871 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:12.871 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:19:12.872 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:12.872 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.872 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.872 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.24929.0:164830 8.b 8:d9eeacc1:::benchmark_data_smithi100_102803_object34326:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-09-17T07:19:12.872 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:10:17.302885+0000", 2024-09-17T07:19:12.872 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 535.53349381800001, 2024-09-17T07:19:12.872 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.91836632900000004, 2024-09-17T07:19:12.872 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:19:12.872 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:19:12.872 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:19:12.872 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.24929", 2024-09-17T07:19:12.872 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/564515582", 2024-09-17T07:19:12.872 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 164830 2024-09-17T07:19:12.873 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.873 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:19:12.873 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.873 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:19:12.873 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.302885+0000", 2024-09-17T07:19:12.873 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:12.873 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.873 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.873 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:19:12.873 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.302885+0000", 2024-09-17T07:19:12.873 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:12.873 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.873 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.874 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:19:12.874 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.302884+0000", 2024-09-17T07:19:12.874 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.999999 2024-09-17T07:19:12.874 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.874 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.874 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:19:12.874 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.302888+0000", 2024-09-17T07:19:12.874 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.1500000000000001e-06 2024-09-17T07:19:12.874 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.874 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.874 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:19:12.874 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.302895+0000", 2024-09-17T07:19:12.875 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.3409999999999997e-06 2024-09-17T07:19:12.875 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.875 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.875 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:19:12.875 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.302900+0000", 2024-09-17T07:19:12.875 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.7389999999999999e-06 2024-09-17T07:19:12.875 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.875 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.875 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:12.875 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.302996+0000", 2024-09-17T07:19:12.875 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.5439000000000006e-05 2024-09-17T07:19:12.875 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.876 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.876 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:19:12.876 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.303004+0000", 2024-09-17T07:19:12.876 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.3049999999999993e-06 2024-09-17T07:19:12.876 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.876 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.876 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:12.876 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.205220+0000", 2024-09-17T07:19:12.876 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.90221566900000005 2024-09-17T07:19:12.876 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.876 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.876 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:19:12.877 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.205341+0000", 2024-09-17T07:19:12.877 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00012173 2024-09-17T07:19:12.877 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.877 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.877 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 1,2", 2024-09-17T07:19:12.877 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.205454+0000", 2024-09-17T07:19:12.877 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000112272 2024-09-17T07:19:12.877 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.877 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.877 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:19:12.877 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.213812+0000", 2024-09-17T07:19:12.877 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0083582610000000005 2024-09-17T07:19:12.877 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.878 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.878 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:12.878 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.220826+0000", 2024-09-17T07:19:12.878 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0070144109999999999 2024-09-17T07:19:12.878 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.878 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.878 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:12.878 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.221160+0000", 2024-09-17T07:19:12.878 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000333436 2024-09-17T07:19:12.878 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.878 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.878 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:19:12.878 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.221201+0000", 2024-09-17T07:19:12.879 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.1505999999999999e-05 2024-09-17T07:19:12.879 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.879 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.879 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:19:12.879 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.221251+0000", 2024-09-17T07:19:12.879 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.0142999999999997e-05 2024-09-17T07:19:12.879 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:12.879 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:19:12.879 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:12.879 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.879 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.879 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.24929.0:164831 8.b 8:d6c25258:::benchmark_data_smithi100_102803_object34327:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-09-17T07:19:12.879 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:10:17.303020+0000", 2024-09-17T07:19:12.880 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 535.53335846599998, 2024-09-17T07:19:12.880 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.91836619399999997, 2024-09-17T07:19:12.880 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:19:12.880 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:19:12.880 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:19:12.880 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.24929", 2024-09-17T07:19:12.880 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/564515582", 2024-09-17T07:19:12.880 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 164831 2024-09-17T07:19:12.880 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.880 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:19:12.880 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.880 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:19:12.880 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.303020+0000", 2024-09-17T07:19:12.881 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:12.881 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.881 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.881 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:19:12.881 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.303020+0000", 2024-09-17T07:19:12.881 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:12.881 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.881 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.881 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:19:12.881 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.303020+0000", 2024-09-17T07:19:12.881 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999995 2024-09-17T07:19:12.881 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.881 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.882 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:19:12.882 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.303023+0000", 2024-09-17T07:19:12.882 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.2100000000000002e-06 2024-09-17T07:19:12.882 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.882 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.882 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:19:12.882 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.303029+0000", 2024-09-17T07:19:12.882 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.0809999999999999e-06 2024-09-17T07:19:12.882 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.882 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.882 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:19:12.882 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.303034+0000", 2024-09-17T07:19:12.882 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.9910000000000002e-06 2024-09-17T07:19:12.883 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.883 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.883 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:12.883 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.303118+0000", 2024-09-17T07:19:12.883 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.3605999999999999e-05 2024-09-17T07:19:12.883 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.883 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.883 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:19:12.883 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.303124+0000", 2024-09-17T07:19:12.883 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.2929999999999997e-06 2024-09-17T07:19:12.883 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.883 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.883 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:12.884 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.207218+0000", 2024-09-17T07:19:12.884 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.90409358399999995 2024-09-17T07:19:12.884 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.884 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.884 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:19:12.884 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.207352+0000", 2024-09-17T07:19:12.884 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00013440899999999999 2024-09-17T07:19:12.884 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.884 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.884 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 1,2", 2024-09-17T07:19:12.884 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.207482+0000", 2024-09-17T07:19:12.884 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000130097 2024-09-17T07:19:12.884 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.885 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.885 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:19:12.885 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.213820+0000", 2024-09-17T07:19:12.885 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0063380069999999997 2024-09-17T07:19:12.885 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.885 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.885 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:12.885 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.220880+0000", 2024-09-17T07:19:12.885 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0070600790000000004 2024-09-17T07:19:12.885 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.885 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.885 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:12.885 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.221313+0000", 2024-09-17T07:19:12.886 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00043324500000000002 2024-09-17T07:19:12.886 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.886 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.886 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:19:12.886 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.221353+0000", 2024-09-17T07:19:12.886 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.9934000000000001e-05 2024-09-17T07:19:12.886 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.886 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.886 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:19:12.886 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.221386+0000", 2024-09-17T07:19:12.886 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.3135e-05 2024-09-17T07:19:12.886 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:12.886 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:19:12.887 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:12.887 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.887 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.887 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.24929.0:164833 8.b 8:d91c7c28:::benchmark_data_smithi100_102803_object34329:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-09-17T07:19:12.887 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:10:17.303139+0000", 2024-09-17T07:19:12.887 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 535.53323970300005, 2024-09-17T07:19:12.887 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.92849971600000003, 2024-09-17T07:19:12.887 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:19:12.887 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:19:12.887 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:19:12.887 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.24929", 2024-09-17T07:19:12.887 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/564515582", 2024-09-17T07:19:12.888 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 164833 2024-09-17T07:19:12.888 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.888 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:19:12.888 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.888 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:19:12.888 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.303139+0000", 2024-09-17T07:19:12.888 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:12.888 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.888 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.888 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:19:12.888 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.303139+0000", 2024-09-17T07:19:12.888 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:12.889 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.889 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.889 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:19:12.889 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.303139+0000", 2024-09-17T07:19:12.889 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999995 2024-09-17T07:19:12.889 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.889 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.889 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:19:12.889 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.303142+0000", 2024-09-17T07:19:12.889 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.4359999999999998e-06 2024-09-17T07:19:12.889 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.889 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.889 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:19:12.890 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.303148+0000", 2024-09-17T07:19:12.890 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.2639999999999997e-06 2024-09-17T07:19:12.890 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.890 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.890 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:19:12.890 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.303153+0000", 2024-09-17T07:19:12.890 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.6779999999999999e-06 2024-09-17T07:19:12.890 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.890 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.890 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:12.890 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.303191+0000", 2024-09-17T07:19:12.890 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.8112999999999999e-05 2024-09-17T07:19:12.891 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.891 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.891 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:19:12.891 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.303196+0000", 2024-09-17T07:19:12.891 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.4580000000000001e-06 2024-09-17T07:19:12.891 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.891 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.891 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:12.891 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.208847+0000", 2024-09-17T07:19:12.891 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.90565048999999997 2024-09-17T07:19:12.891 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.891 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.891 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:19:12.892 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.208954+0000", 2024-09-17T07:19:12.892 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000107046 2024-09-17T07:19:12.892 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.892 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.892 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 1,2", 2024-09-17T07:19:12.892 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.209039+0000", 2024-09-17T07:19:12.892 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.5461000000000001e-05 2024-09-17T07:19:12.892 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.892 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.892 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:19:12.892 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.215573+0000", 2024-09-17T07:19:12.893 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0065331590000000002 2024-09-17T07:19:12.893 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.893 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.893 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:12.893 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.221425+0000", 2024-09-17T07:19:12.893 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0058528169999999997 2024-09-17T07:19:12.893 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.893 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.893 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:12.893 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.231554+0000", 2024-09-17T07:19:12.893 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.010129035 2024-09-17T07:19:12.893 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.893 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.894 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:19:12.894 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.231601+0000", 2024-09-17T07:19:12.894 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.6687999999999998e-05 2024-09-17T07:19:12.894 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.894 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.894 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:19:12.894 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.231639+0000", 2024-09-17T07:19:12.894 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.7604000000000001e-05 2024-09-17T07:19:12.894 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:12.894 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:19:12.894 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:12.894 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.895 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.895 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.24929.0:164844 8.b 8:d3d463b4:::benchmark_data_smithi100_102803_object34340:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-09-17T07:19:12.895 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:10:17.313421+0000", 2024-09-17T07:19:12.895 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 535.52295753400006, 2024-09-17T07:19:12.895 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.91846116300000002, 2024-09-17T07:19:12.895 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:19:12.895 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:19:12.895 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:19:12.895 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.24929", 2024-09-17T07:19:12.895 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/564515582", 2024-09-17T07:19:12.895 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 164844 2024-09-17T07:19:12.895 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.896 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:19:12.896 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.896 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:19:12.896 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.313421+0000", 2024-09-17T07:19:12.896 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:12.896 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.896 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.896 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:19:12.896 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.313421+0000", 2024-09-17T07:19:12.896 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:12.896 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.896 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.897 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:19:12.897 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.313420+0000", 2024-09-17T07:19:12.897 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:19:12.897 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.897 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.897 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:19:12.897 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.313424+0000", 2024-09-17T07:19:12.897 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.9709999999999998e-06 2024-09-17T07:19:12.897 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.897 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.897 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:19:12.897 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.313430+0000", 2024-09-17T07:19:12.897 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.3010000000000003e-06 2024-09-17T07:19:12.898 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.898 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.898 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:19:12.898 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.313435+0000", 2024-09-17T07:19:12.898 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.1540000000000001e-06 2024-09-17T07:19:12.898 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.898 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.898 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:12.898 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.313478+0000", 2024-09-17T07:19:12.898 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.2960999999999997e-05 2024-09-17T07:19:12.898 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.898 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.899 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:19:12.899 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.313486+0000", 2024-09-17T07:19:12.899 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.9219999999999997e-06 2024-09-17T07:19:12.899 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.899 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.899 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:12.899 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.210370+0000", 2024-09-17T07:19:12.899 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.89688354999999997 2024-09-17T07:19:12.899 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.899 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.899 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:19:12.899 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.210471+0000", 2024-09-17T07:19:12.900 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000101307 2024-09-17T07:19:12.900 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.900 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.900 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 1,2", 2024-09-17T07:19:12.900 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.210532+0000", 2024-09-17T07:19:12.900 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.0868000000000001e-05 2024-09-17T07:19:12.900 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.900 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.900 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:19:12.900 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.215579+0000", 2024-09-17T07:19:12.900 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.005046811 2024-09-17T07:19:12.900 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.900 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.901 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:12.901 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.221473+0000", 2024-09-17T07:19:12.901 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0058948120000000001 2024-09-17T07:19:12.901 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.901 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.901 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:12.901 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.231808+0000", 2024-09-17T07:19:12.902 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.010334622 2024-09-17T07:19:12.902 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.902 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.902 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:19:12.902 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.231848+0000", 2024-09-17T07:19:12.902 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.9780000000000002e-05 2024-09-17T07:19:12.902 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.903 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.903 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:19:12.903 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.231882+0000", 2024-09-17T07:19:12.903 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.4505999999999998e-05 2024-09-17T07:19:12.903 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:12.903 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:19:12.903 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:12.903 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.903 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.903 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.24929.0:164852 8.b 8:dc4d72d5:::benchmark_data_smithi100_102803_object34348:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-09-17T07:19:12.903 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:10:17.317343+0000", 2024-09-17T07:19:12.903 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 535.51903620300004, 2024-09-17T07:19:12.903 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.91462852100000003, 2024-09-17T07:19:12.904 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:19:12.904 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:19:12.904 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:19:12.904 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.24929", 2024-09-17T07:19:12.904 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/564515582", 2024-09-17T07:19:12.904 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 164852 2024-09-17T07:19:12.904 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.904 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:19:12.904 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.904 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:19:12.904 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.317343+0000", 2024-09-17T07:19:12.904 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:12.904 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.905 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.905 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:19:12.905 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.317343+0000", 2024-09-17T07:19:12.905 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:12.905 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.905 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.905 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:19:12.905 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.317341+0000", 2024-09-17T07:19:12.905 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999981 2024-09-17T07:19:12.905 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.905 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.905 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:19:12.906 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.317347+0000", 2024-09-17T07:19:12.906 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.4300000000000003e-06 2024-09-17T07:19:12.906 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.906 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.906 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:19:12.906 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.317361+0000", 2024-09-17T07:19:12.906 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.3392e-05 2024-09-17T07:19:12.906 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.906 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.906 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:19:12.906 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.317371+0000", 2024-09-17T07:19:12.906 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0421999999999999e-05 2024-09-17T07:19:12.906 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.907 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.907 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:12.907 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.317445+0000", 2024-09-17T07:19:12.907 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.4203999999999996e-05 2024-09-17T07:19:12.907 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.907 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.907 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:19:12.907 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.317453+0000", 2024-09-17T07:19:12.907 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.8690000000000004e-06 2024-09-17T07:19:12.907 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.907 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.907 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:12.907 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.212084+0000", 2024-09-17T07:19:12.908 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.89463097700000005 2024-09-17T07:19:12.908 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.908 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.908 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:19:12.908 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.212206+0000", 2024-09-17T07:19:12.908 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000122111 2024-09-17T07:19:12.908 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.908 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.908 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 1,2", 2024-09-17T07:19:12.908 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.212291+0000", 2024-09-17T07:19:12.908 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.4608e-05 2024-09-17T07:19:12.908 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.908 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.909 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:19:12.909 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.218235+0000", 2024-09-17T07:19:12.909 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0059445419999999997 2024-09-17T07:19:12.909 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.909 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.909 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:12.909 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.231721+0000", 2024-09-17T07:19:12.909 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.013485914999999999 2024-09-17T07:19:12.909 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.909 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.909 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:12.909 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.231913+0000", 2024-09-17T07:19:12.909 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000191344 2024-09-17T07:19:12.910 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.910 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.910 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:19:12.910 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.231943+0000", 2024-09-17T07:19:12.910 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.0685000000000002e-05 2024-09-17T07:19:12.910 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.910 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.910 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:19:12.910 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.231971+0000", 2024-09-17T07:19:12.910 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.7835e-05 2024-09-17T07:19:12.910 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:12.910 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:19:12.910 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:12.911 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.911 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.911 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.24929.0:164879 8.b 8:d659370b:::benchmark_data_smithi100_102803_object34375:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-09-17T07:19:12.911 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:10:17.346186+0000", 2024-09-17T07:19:12.911 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 535.49019300600003, 2024-09-17T07:19:12.911 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.88586965200000001, 2024-09-17T07:19:12.911 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:19:12.911 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:19:12.911 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:19:12.911 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.24929", 2024-09-17T07:19:12.911 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/564515582", 2024-09-17T07:19:12.911 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 164879 2024-09-17T07:19:12.911 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.912 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:19:12.912 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.912 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:19:12.912 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.346186+0000", 2024-09-17T07:19:12.912 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:12.912 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.912 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.912 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:19:12.912 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.346186+0000", 2024-09-17T07:19:12.912 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:12.912 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.912 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.913 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:19:12.913 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.346184+0000", 2024-09-17T07:19:12.913 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:19:12.913 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.913 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.913 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:19:12.913 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.346190+0000", 2024-09-17T07:19:12.913 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.412e-06 2024-09-17T07:19:12.913 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.913 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.913 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:19:12.913 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.346199+0000", 2024-09-17T07:19:12.913 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.4919999999999999e-06 2024-09-17T07:19:12.914 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.914 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.914 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:19:12.914 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.346205+0000", 2024-09-17T07:19:12.914 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.8749999999999997e-06 2024-09-17T07:19:12.914 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.914 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.914 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:12.914 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.346249+0000", 2024-09-17T07:19:12.914 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.4239000000000001e-05 2024-09-17T07:19:12.914 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.914 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.914 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:19:12.915 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.346258+0000", 2024-09-17T07:19:12.915 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.9050000000000003e-06 2024-09-17T07:19:12.915 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.915 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.915 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:12.915 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.213866+0000", 2024-09-17T07:19:12.915 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.86760820299999997 2024-09-17T07:19:12.915 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.915 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.915 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:19:12.915 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.213988+0000", 2024-09-17T07:19:12.915 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000121691 2024-09-17T07:19:12.915 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.915 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.916 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 1,2", 2024-09-17T07:19:12.916 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.214103+0000", 2024-09-17T07:19:12.916 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000114852 2024-09-17T07:19:12.916 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.916 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.916 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:19:12.916 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.220838+0000", 2024-09-17T07:19:12.916 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0067347470000000001 2024-09-17T07:19:12.916 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.916 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.916 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:12.916 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.231756+0000", 2024-09-17T07:19:12.917 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.010918673 2024-09-17T07:19:12.917 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.917 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.917 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:12.917 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.231999+0000", 2024-09-17T07:19:12.917 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00024308700000000001 2024-09-17T07:19:12.917 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.917 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.917 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:19:12.917 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.232029+0000", 2024-09-17T07:19:12.917 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.9258999999999999e-05 2024-09-17T07:19:12.917 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.918 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.918 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:19:12.918 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.232055+0000", 2024-09-17T07:19:12.918 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.6848000000000001e-05 2024-09-17T07:19:12.918 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:12.918 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:19:12.918 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:12.918 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.918 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.918 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.24929.0:164910 8.b 8:dc89b12b:::benchmark_data_smithi100_102803_object34406:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-09-17T07:19:12.918 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:10:17.380469+0000", 2024-09-17T07:19:12.918 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 535.45590968500005, 2024-09-17T07:19:12.918 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.87627316499999997, 2024-09-17T07:19:12.919 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:19:12.919 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:19:12.919 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:19:12.919 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.24929", 2024-09-17T07:19:12.919 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/564515582", 2024-09-17T07:19:12.919 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 164910 2024-09-17T07:19:12.919 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.919 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:19:12.919 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.919 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:19:12.919 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.380469+0000", 2024-09-17T07:19:12.919 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:12.920 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.920 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.920 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:19:12.920 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.380469+0000", 2024-09-17T07:19:12.920 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:12.920 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.920 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.920 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:19:12.920 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.380467+0000", 2024-09-17T07:19:12.920 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999981 2024-09-17T07:19:12.920 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.921 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.921 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:19:12.921 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.380474+0000", 2024-09-17T07:19:12.921 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.8689999999999998e-06 2024-09-17T07:19:12.921 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.921 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.921 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:19:12.921 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.380489+0000", 2024-09-17T07:19:12.921 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.4749999999999999e-05 2024-09-17T07:19:12.921 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.921 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.921 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:19:12.921 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.380497+0000", 2024-09-17T07:19:12.922 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.2889999999999998e-06 2024-09-17T07:19:12.922 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.922 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.922 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:12.922 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.380560+0000", 2024-09-17T07:19:12.922 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.3288999999999997e-05 2024-09-17T07:19:12.922 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.922 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.922 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:19:12.922 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.380569+0000", 2024-09-17T07:19:12.922 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.9279999999999999e-06 2024-09-17T07:19:12.922 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.922 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.923 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:12.923 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.221510+0000", 2024-09-17T07:19:12.923 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.84094051299999995 2024-09-17T07:19:12.923 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.923 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.923 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:19:12.923 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.221636+0000", 2024-09-17T07:19:12.923 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00012644800000000001 2024-09-17T07:19:12.923 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.923 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.923 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 1,2", 2024-09-17T07:19:12.923 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.221746+0000", 2024-09-17T07:19:12.923 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000109836 2024-09-17T07:19:12.924 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.924 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.924 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:19:12.924 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.231503+0000", 2024-09-17T07:19:12.924 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0097572619999999992 2024-09-17T07:19:12.924 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.924 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.924 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:12.924 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.232110+0000", 2024-09-17T07:19:12.924 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00060627700000000001 2024-09-17T07:19:12.924 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.924 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.924 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:12.925 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.256666+0000", 2024-09-17T07:19:12.925 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.024556069999999999 2024-09-17T07:19:12.925 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.925 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.925 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:19:12.925 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.256708+0000", 2024-09-17T07:19:12.925 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.2211e-05 2024-09-17T07:19:12.925 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.925 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.925 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:19:12.925 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.256742+0000", 2024-09-17T07:19:12.925 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.4199999999999998e-05 2024-09-17T07:19:12.925 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:12.925 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:19:12.926 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:12.926 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.926 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.926 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.24929.0:164913 8.b 8:dcec2487:::benchmark_data_smithi100_102803_object34409:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-09-17T07:19:12.926 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:10:17.383765+0000", 2024-09-17T07:19:12.926 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 535.45261336800002, 2024-09-17T07:19:12.926 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.87384644, 2024-09-17T07:19:12.926 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:19:12.926 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:19:12.926 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:19:12.926 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.24929", 2024-09-17T07:19:12.926 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/564515582", 2024-09-17T07:19:12.926 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 164913 2024-09-17T07:19:12.926 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.927 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:19:12.927 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.927 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:19:12.927 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.383765+0000", 2024-09-17T07:19:12.927 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:12.927 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.927 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.927 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:19:12.927 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.383765+0000", 2024-09-17T07:19:12.927 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:12.927 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.927 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.927 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:19:12.928 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.383764+0000", 2024-09-17T07:19:12.928 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:19:12.928 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.928 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.928 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:19:12.928 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.383771+0000", 2024-09-17T07:19:12.928 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.1640000000000004e-06 2024-09-17T07:19:12.928 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.928 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.928 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:19:12.928 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.383780+0000", 2024-09-17T07:19:12.928 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.2699999999999993e-06 2024-09-17T07:19:12.929 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.929 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.929 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:19:12.929 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.383786+0000", 2024-09-17T07:19:12.929 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.1989999999999997e-06 2024-09-17T07:19:12.929 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.929 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.929 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:12.929 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.383822+0000", 2024-09-17T07:19:12.929 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.5952999999999999e-05 2024-09-17T07:19:12.929 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.929 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.930 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:19:12.930 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:17.383827+0000", 2024-09-17T07:19:12.930 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.6879999999999996e-06 2024-09-17T07:19:12.930 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.930 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.930 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:12.930 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.224104+0000", 2024-09-17T07:19:12.930 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.84027645100000004 2024-09-17T07:19:12.930 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.930 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.930 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:19:12.930 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.224225+0000", 2024-09-17T07:19:12.930 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000121618 2024-09-17T07:19:12.930 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.931 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.931 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 1,2", 2024-09-17T07:19:12.931 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.224337+0000", 2024-09-17T07:19:12.931 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000111352 2024-09-17T07:19:12.931 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.931 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.931 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:19:12.931 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.231510+0000", 2024-09-17T07:19:12.931 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0071731060000000003 2024-09-17T07:19:12.931 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.931 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.931 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:12.931 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.232151+0000", 2024-09-17T07:19:12.931 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00064107699999999999 2024-09-17T07:19:12.932 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.932 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.932 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:12.932 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.257530+0000", 2024-09-17T07:19:12.932 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.025379235 2024-09-17T07:19:12.932 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.932 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.932 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:19:12.932 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.257575+0000", 2024-09-17T07:19:12.932 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.5263999999999999e-05 2024-09-17T07:19:12.932 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.932 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.933 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:19:12.933 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:10:18.257612+0000", 2024-09-17T07:19:12.933 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.6477999999999999e-05 2024-09-17T07:19:12.933 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:12.933 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:19:12.933 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:12.933 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.933 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.933 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.15330.0:252181 9.c 9:3230c420:::benchmark_data_smithi100_128631_object117915:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e285)", 2024-09-17T07:19:12.933 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:15:22.043524+0000", 2024-09-17T07:19:12.933 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 230.792854369, 2024-09-17T07:19:12.933 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.24149841399999999, 2024-09-17T07:19:12.933 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:19:12.934 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:19:12.934 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:19:12.934 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.15330", 2024-09-17T07:19:12.934 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/1361381749", 2024-09-17T07:19:12.934 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 252181 2024-09-17T07:19:12.934 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.934 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:19:12.934 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.934 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:19:12.934 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:22.043524+0000", 2024-09-17T07:19:12.934 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:12.934 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.934 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.935 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:19:12.935 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:22.043524+0000", 2024-09-17T07:19:12.935 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:12.935 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.935 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.935 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:19:12.935 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:22.043524+0000", 2024-09-17T07:19:12.935 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999995 2024-09-17T07:19:12.935 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.935 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.935 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:19:12.935 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:22.043527+0000", 2024-09-17T07:19:12.935 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.4300000000000002e-06 2024-09-17T07:19:12.936 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.936 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.936 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:19:12.936 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:22.043534+0000", 2024-09-17T07:19:12.936 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.8360000000000003e-06 2024-09-17T07:19:12.936 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.936 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.936 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:19:12.936 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:22.043539+0000", 2024-09-17T07:19:12.936 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.8969999999999999e-06 2024-09-17T07:19:12.936 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.936 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.936 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:12.937 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:22.047874+0000", 2024-09-17T07:19:12.937 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0043346970000000002 2024-09-17T07:19:12.937 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.937 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.937 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:19:12.937 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:22.048006+0000", 2024-09-17T07:19:12.937 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000132772 2024-09-17T07:19:12.937 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.937 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.937 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,7", 2024-09-17T07:19:12.937 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:22.048120+0000", 2024-09-17T07:19:12.937 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000113744 2024-09-17T07:19:12.938 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.938 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.938 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:19:12.938 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:22.051043+0000", 2024-09-17T07:19:12.938 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.002922413 2024-09-17T07:19:12.938 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.938 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.938 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:12.938 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:22.053005+0000", 2024-09-17T07:19:12.938 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0019624540000000002 2024-09-17T07:19:12.938 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.938 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.938 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:12.939 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:22.284959+0000", 2024-09-17T07:19:12.939 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.231954407 2024-09-17T07:19:12.939 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.939 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.939 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:19:12.939 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:22.284994+0000", 2024-09-17T07:19:12.939 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.4588999999999997e-05 2024-09-17T07:19:12.939 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.939 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.939 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:19:12.939 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:22.285023+0000", 2024-09-17T07:19:12.939 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.8753999999999999e-05 2024-09-17T07:19:12.939 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:12.940 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:19:12.940 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:12.940 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.940 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.940 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.24338.0:2012 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 94163966828416 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e303)", 2024-09-17T07:19:12.940 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:16:16.974110+0000", 2024-09-17T07:19:12.940 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 175.86226894999999, 2024-09-17T07:19:12.940 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.40324416499999999, 2024-09-17T07:19:12.940 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:19:12.940 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "started", 2024-09-17T07:19:12.940 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:19:12.940 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.24338", 2024-09-17T07:19:12.940 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/1969590951", 2024-09-17T07:19:12.941 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 2012 2024-09-17T07:19:12.941 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.941 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:19:12.941 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.941 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:19:12.941 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:16.974110+0000", 2024-09-17T07:19:12.941 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:12.941 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.941 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.941 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:19:12.941 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:16.974110+0000", 2024-09-17T07:19:12.941 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:12.942 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.942 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.942 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:19:12.942 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:16.974108+0000", 2024-09-17T07:19:12.942 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999976 2024-09-17T07:19:12.942 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.942 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.942 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:19:12.942 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:16.974120+0000", 2024-09-17T07:19:12.942 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.1963999999999999e-05 2024-09-17T07:19:12.942 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.942 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.942 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:19:12.943 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:16.974134+0000", 2024-09-17T07:19:12.943 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.4089e-05 2024-09-17T07:19:12.943 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.943 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.943 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:19:12.943 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:16.974144+0000", 2024-09-17T07:19:12.943 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0533000000000001e-05 2024-09-17T07:19:12.943 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.943 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.943 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:12.943 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:16.974228+0000", 2024-09-17T07:19:12.943 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.4149999999999999e-05 2024-09-17T07:19:12.943 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.943 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.944 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:19:12.944 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:16.974238+0000", 2024-09-17T07:19:12.944 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.2320000000000007e-06 2024-09-17T07:19:12.944 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.944 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.944 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:12.944 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:17.372016+0000", 2024-09-17T07:19:12.944 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.39777825900000002 2024-09-17T07:19:12.944 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.944 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.944 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:19:12.944 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:17.372156+0000", 2024-09-17T07:19:12.945 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00013977 2024-09-17T07:19:12.945 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.945 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.945 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:19:12.945 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:17.377354+0000", 2024-09-17T07:19:12.945 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0051983480000000002 2024-09-17T07:19:12.945 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:12.945 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:19:12.945 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:12.945 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.945 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.945 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_repop(client.15648.0:41085 10.c e309/297 10:3c85a0dc:::benchmark_data_smithi100_155039_object41084:head v 309'2579, mlcod=309'2576)", 2024-09-17T07:19:12.945 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:16:37.515816+0000", 2024-09-17T07:19:12.946 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 155.32056278600001, 2024-09-17T07:19:12.946 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.20139325799999999, 2024-09-17T07:19:12.946 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:19:12.946 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:19:12.946 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:19:12.946 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.946 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:19:12.946 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.515816+0000", 2024-09-17T07:19:12.946 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:12.946 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.946 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.946 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:19:12.946 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.515816+0000", 2024-09-17T07:19:12.947 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:19:12.948 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:12.948 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.948 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.948 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:19:12.948 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.515815+0000", 2024-09-17T07:19:12.948 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999995 2024-09-17T07:19:12.948 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.948 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.949 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:19:12.949 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.515839+0000", 2024-09-17T07:19:12.949 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.3320999999999999e-05 2024-09-17T07:19:12.949 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.949 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.949 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:19:12.949 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.710058+0000", 2024-09-17T07:19:12.949 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.19421917699999999 2024-09-17T07:19:12.949 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.949 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.949 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:19:12.949 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.710071+0000", 2024-09-17T07:19:12.949 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.2896e-05 2024-09-17T07:19:12.950 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.950 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.950 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:12.950 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.710135+0000", 2024-09-17T07:19:12.950 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.3689999999999995e-05 2024-09-17T07:19:12.950 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.950 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.950 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:19:12.950 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.710156+0000", 2024-09-17T07:19:12.950 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.1379e-05 2024-09-17T07:19:12.950 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.950 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.950 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:19:12.951 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.717182+0000", 2024-09-17T07:19:12.951 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0070264899999999998 2024-09-17T07:19:12.951 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.951 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.951 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:19:12.951 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.717209+0000", 2024-09-17T07:19:12.951 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.6846999999999999e-05 2024-09-17T07:19:12.951 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:12.951 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:19:12.951 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:12.951 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.951 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.952 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.15648.0:41091 10.1 10:8987a173:::benchmark_data_smithi100_155039_object41090:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e309)", 2024-09-17T07:19:12.952 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:16:37.517770+0000", 2024-09-17T07:19:12.952 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 155.31860909, 2024-09-17T07:19:12.952 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.200763778, 2024-09-17T07:19:12.952 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:19:12.952 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:19:12.952 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:19:12.952 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.15648", 2024-09-17T07:19:12.952 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/1684940078", 2024-09-17T07:19:12.952 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 41091 2024-09-17T07:19:12.952 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.952 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:19:12.952 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.953 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:19:12.953 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.517770+0000", 2024-09-17T07:19:12.953 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:12.953 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.953 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.953 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:19:12.953 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.517770+0000", 2024-09-17T07:19:12.953 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:12.953 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.953 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.953 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:19:12.953 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.517768+0000", 2024-09-17T07:19:12.953 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:19:12.954 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.954 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.954 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:19:12.954 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.517810+0000", 2024-09-17T07:19:12.954 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.1838000000000002e-05 2024-09-17T07:19:12.954 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.954 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.954 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:19:12.954 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.710075+0000", 2024-09-17T07:19:12.954 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.192264723 2024-09-17T07:19:12.954 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.954 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.954 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:19:12.955 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.710091+0000", 2024-09-17T07:19:12.955 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.5841e-05 2024-09-17T07:19:12.955 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.955 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.955 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:12.955 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.711695+0000", 2024-09-17T07:19:12.955 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0016048740000000001 2024-09-17T07:19:12.955 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.955 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.955 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:19:12.955 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.711889+0000", 2024-09-17T07:19:12.955 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00019338 2024-09-17T07:19:12.955 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.956 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.956 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,7", 2024-09-17T07:19:12.956 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.712039+0000", 2024-09-17T07:19:12.956 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000150316 2024-09-17T07:19:12.956 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.956 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.956 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:12.956 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.718385+0000", 2024-09-17T07:19:12.956 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0063457959999999999 2024-09-17T07:19:12.956 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.956 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.956 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:19:12.956 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.718392+0000", 2024-09-17T07:19:12.957 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.939e-06 2024-09-17T07:19:12.957 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.957 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.957 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:12.957 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.718440+0000", 2024-09-17T07:19:12.957 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.8384000000000003e-05 2024-09-17T07:19:12.957 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.957 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.957 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:19:12.957 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.718484+0000", 2024-09-17T07:19:12.957 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.3766000000000001e-05 2024-09-17T07:19:12.957 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.957 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.958 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:19:12.958 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.718533+0000", 2024-09-17T07:19:12.958 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.9443000000000001e-05 2024-09-17T07:19:12.958 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:12.958 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:19:12.958 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:12.958 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.958 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.958 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.15648.0:83760 10.1 10:8647247b:::benchmark_data_smithi100_155039_object83759:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e316)", 2024-09-17T07:19:12.958 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:17:11.263326+0000", 2024-09-17T07:19:12.958 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 121.573053031, 2024-09-17T07:19:12.958 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.22141090499999999, 2024-09-17T07:19:12.958 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:19:12.959 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:19:12.959 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:19:12.959 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.15648", 2024-09-17T07:19:12.959 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/1684940078", 2024-09-17T07:19:12.959 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 83760 2024-09-17T07:19:12.959 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.959 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:19:12.959 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.959 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:19:12.959 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:17:11.263326+0000", 2024-09-17T07:19:12.959 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:12.959 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.959 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.960 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:19:12.960 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:17:11.263326+0000", 2024-09-17T07:19:12.960 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:12.960 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.960 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.960 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:19:12.960 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:17:11.263324+0000", 2024-09-17T07:19:12.960 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:19:12.960 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.960 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.960 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:19:12.960 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:17:11.273203+0000", 2024-09-17T07:19:12.960 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0098783459999999997 2024-09-17T07:19:12.961 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.961 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.961 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:19:12.961 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:17:11.273212+0000", 2024-09-17T07:19:12.961 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.4150000000000005e-06 2024-09-17T07:19:12.961 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.961 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.961 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:19:12.961 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:17:11.273219+0000", 2024-09-17T07:19:12.961 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.2389999999999996e-06 2024-09-17T07:19:12.961 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.961 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.961 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:12.962 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:17:11.273276+0000", 2024-09-17T07:19:12.962 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.7037000000000003e-05 2024-09-17T07:19:12.962 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.962 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.962 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:19:12.962 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:17:11.273408+0000", 2024-09-17T07:19:12.962 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000131435 2024-09-17T07:19:12.962 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.962 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.962 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,7", 2024-09-17T07:19:12.962 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:17:11.273544+0000", 2024-09-17T07:19:12.962 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00013591300000000001 2024-09-17T07:19:12.962 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.963 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.963 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:12.963 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:17:11.276856+0000", 2024-09-17T07:19:12.963 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.003312068 2024-09-17T07:19:12.963 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.963 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.963 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:19:12.963 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:17:11.277096+0000", 2024-09-17T07:19:12.963 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000240051 2024-09-17T07:19:12.963 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.963 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.963 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:12.964 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:17:11.484668+0000", 2024-09-17T07:19:12.964 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.20757231700000001 2024-09-17T07:19:12.964 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.964 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.964 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:19:12.964 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:17:11.484705+0000", 2024-09-17T07:19:12.964 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.7206999999999998e-05 2024-09-17T07:19:12.964 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.964 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.964 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:19:12.964 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:17:11.484737+0000", 2024-09-17T07:19:12.964 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.1325000000000002e-05 2024-09-17T07:19:12.964 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:12.965 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:19:12.965 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:12.965 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.965 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.965 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.15648.0:83762 10.2 10:48aa8dc0:::benchmark_data_smithi100_155039_object83761:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e316)", 2024-09-17T07:19:12.965 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:17:11.273234+0000", 2024-09-17T07:19:12.965 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 121.56314478, 2024-09-17T07:19:12.965 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.211662237, 2024-09-17T07:19:12.965 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:19:12.965 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:19:12.965 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:19:12.965 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.15648", 2024-09-17T07:19:12.966 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/1684940078", 2024-09-17T07:19:12.966 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 83762 2024-09-17T07:19:12.966 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.966 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:19:12.966 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.966 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:19:12.966 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:17:11.273234+0000", 2024-09-17T07:19:12.966 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:12.966 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.966 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.966 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:19:12.966 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:17:11.273234+0000", 2024-09-17T07:19:12.966 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:12.967 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.967 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.967 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:19:12.967 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:17:11.273233+0000", 2024-09-17T07:19:12.967 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.999999 2024-09-17T07:19:12.967 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.967 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.967 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:19:12.967 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:17:11.292441+0000", 2024-09-17T07:19:12.967 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.019207990000000001 2024-09-17T07:19:12.967 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.967 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.967 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:19:12.968 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:17:11.292460+0000", 2024-09-17T07:19:12.968 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.8546000000000002e-05 2024-09-17T07:19:12.968 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.968 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.968 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:19:12.968 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:17:11.292468+0000", 2024-09-17T07:19:12.968 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.0269999999999995e-06 2024-09-17T07:19:12.968 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.968 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.968 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:12.968 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:17:11.292545+0000", 2024-09-17T07:19:12.968 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.7453999999999994e-05 2024-09-17T07:19:12.969 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.969 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.969 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:19:12.969 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:17:11.292770+0000", 2024-09-17T07:19:12.969 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00022537300000000001 2024-09-17T07:19:12.969 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.969 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.969 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0", 2024-09-17T07:19:12.969 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:17:11.292954+0000", 2024-09-17T07:19:12.969 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000183406 2024-09-17T07:19:12.969 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.969 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.969 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:19:12.970 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:17:11.296390+0000", 2024-09-17T07:19:12.970 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0034362450000000001 2024-09-17T07:19:12.970 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.970 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.970 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:12.970 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:17:11.484754+0000", 2024-09-17T07:19:12.970 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.18836391699999999 2024-09-17T07:19:12.970 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.970 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.970 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:19:12.970 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:17:11.484802+0000", 2024-09-17T07:19:12.970 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.7645999999999997e-05 2024-09-17T07:19:12.970 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.970 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.971 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:19:12.971 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:17:11.484896+0000", 2024-09-17T07:19:12.971 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.4523000000000003e-05 2024-09-17T07:19:12.971 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:12.971 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:19:12.971 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:12.971 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.971 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.971 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.15648.0:127022 10.e 10:73c3c914:::benchmark_data_smithi100_155039_object3674:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e336)", 2024-09-17T07:19:12.971 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:17:57.179302+0000", 2024-09-17T07:19:12.971 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 75.657077178999998, 2024-09-17T07:19:12.971 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.25327506300000002, 2024-09-17T07:19:12.971 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:19:12.972 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:19:12.972 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:19:12.972 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.15648", 2024-09-17T07:19:12.972 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/1684940078", 2024-09-17T07:19:12.972 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 127022 2024-09-17T07:19:12.972 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.972 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:19:12.972 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.972 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:19:12.972 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:17:57.179302+0000", 2024-09-17T07:19:12.972 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:12.972 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.972 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.973 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:19:12.973 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:17:57.179302+0000", 2024-09-17T07:19:12.973 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:12.973 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.973 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.973 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:19:12.973 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:17:57.179300+0000", 2024-09-17T07:19:12.973 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.999999 2024-09-17T07:19:12.973 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.973 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.973 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:19:12.973 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:17:57.179308+0000", 2024-09-17T07:19:12.973 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.8699999999999992e-06 2024-09-17T07:19:12.974 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.974 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.974 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:19:12.974 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:17:57.179317+0000", 2024-09-17T07:19:12.974 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.8510000000000005e-06 2024-09-17T07:19:12.974 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.974 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.974 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:19:12.974 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:17:57.179324+0000", 2024-09-17T07:19:12.974 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.4749999999999998e-06 2024-09-17T07:19:12.974 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.974 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.974 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:12.975 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:17:57.247082+0000", 2024-09-17T07:19:12.975 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.067758556999999997 2024-09-17T07:19:12.975 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.975 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.975 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:19:12.975 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:17:57.247210+0000", 2024-09-17T07:19:12.975 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000127883 2024-09-17T07:19:12.975 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.975 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.975 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 2,6", 2024-09-17T07:19:12.975 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:17:57.247343+0000", 2024-09-17T07:19:12.975 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00013321200000000001 2024-09-17T07:19:12.975 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.976 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.976 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:19:12.976 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:17:57.257879+0000", 2024-09-17T07:19:12.976 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.010536106 2024-09-17T07:19:12.976 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.976 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.976 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:12.976 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:17:57.267937+0000", 2024-09-17T07:19:12.976 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.010058037000000001 2024-09-17T07:19:12.976 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.976 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.976 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:12.976 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:17:57.432493+0000", 2024-09-17T07:19:12.977 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.16455605800000001 2024-09-17T07:19:12.977 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.977 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.977 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:19:12.977 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:17:57.432539+0000", 2024-09-17T07:19:12.977 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.5707000000000001e-05 2024-09-17T07:19:12.977 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.977 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.977 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:19:12.977 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:17:57.432577+0000", 2024-09-17T07:19:12.977 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.7477e-05 2024-09-17T07:19:12.977 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:12.977 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:19:12.978 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:12.978 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.978 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.978 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.15648.0:127052 10.e 10:7c35c682:::benchmark_data_smithi100_155039_object3704:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e336)", 2024-09-17T07:19:12.978 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:17:57.229169+0000", 2024-09-17T07:19:12.978 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 75.607209776999994, 2024-09-17T07:19:12.978 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.20349905300000001, 2024-09-17T07:19:12.978 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:19:12.978 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:19:12.978 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:19:12.978 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.15648", 2024-09-17T07:19:12.978 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/1684940078", 2024-09-17T07:19:12.978 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 127052 2024-09-17T07:19:12.979 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.979 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:19:12.979 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.979 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:19:12.979 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:17:57.229169+0000", 2024-09-17T07:19:12.979 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:12.979 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.979 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.979 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:19:12.979 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:17:57.229169+0000", 2024-09-17T07:19:12.979 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:12.979 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.979 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.980 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:19:12.980 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:17:57.229168+0000", 2024-09-17T07:19:12.980 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:19:12.980 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.980 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.980 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:19:12.980 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:17:57.229174+0000", 2024-09-17T07:19:12.980 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.4439999999999996e-06 2024-09-17T07:19:12.980 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.980 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.980 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:19:12.980 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:17:57.229188+0000", 2024-09-17T07:19:12.980 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.4423999999999999e-05 2024-09-17T07:19:12.980 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.981 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.981 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:19:12.981 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:17:57.229196+0000", 2024-09-17T07:19:12.981 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.1600000000000001e-06 2024-09-17T07:19:12.981 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.981 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.981 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:12.981 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:17:57.249816+0000", 2024-09-17T07:19:12.981 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.020620472000000001 2024-09-17T07:19:12.981 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.981 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.981 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:19:12.981 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:17:57.249954+0000", 2024-09-17T07:19:12.982 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00013742100000000001 2024-09-17T07:19:12.982 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.982 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.982 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 2,6", 2024-09-17T07:19:12.982 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:17:57.250077+0000", 2024-09-17T07:19:12.982 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00012339299999999999 2024-09-17T07:19:12.982 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.982 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.982 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:19:12.982 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:17:57.257885+0000", 2024-09-17T07:19:12.982 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.007808494 2024-09-17T07:19:12.982 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.982 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.983 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:12.983 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:17:57.267966+0000", 2024-09-17T07:19:12.983 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.010080861999999999 2024-09-17T07:19:12.983 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.983 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.983 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:12.983 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:17:57.432607+0000", 2024-09-17T07:19:12.983 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.16464032300000001 2024-09-17T07:19:12.983 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.983 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.983 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:19:12.983 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:17:57.432639+0000", 2024-09-17T07:19:12.983 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.2388999999999998e-05 2024-09-17T07:19:12.984 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:12.984 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:12.984 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:19:12.984 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:17:57.432668+0000", 2024-09-17T07:19:12.984 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.9068999999999999e-05 2024-09-17T07:19:12.984 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:12.984 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:19:12.984 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:12.984 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:12.984 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:19:12.984 INFO:teuthology.orchestra.run.smithi135.stdout:} 2024-09-17T07:19:13.020 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:13.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:13.021 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:19:13.122 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:19:13.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:13.167 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:13.167 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:19:13.268 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:19:13.304 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:13.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:13.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:19:13.406 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:19:13.441 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:13.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:13.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:19:13.543 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:19:13.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:13.582 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:13.582 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:19:13.683 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:19:13.716 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:13.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:13.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:19:13.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:13 smithi135 ceph-mon[33238]: osdmap e367: 8 total, 7 up, 7 in 2024-09-17T07:19:13.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:13 smithi135 ceph-mon[33238]: pgmap v785: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+remapped+backfilling, 1 peering, 2 active+undersized+degraded+remapped+backfilling, 8 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 88 active+clean; 2.1 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 843 op/s; 5294/101679 objects degraded (5.207%); 16643/101679 objects misplaced (16.368%); 27 MiB/s, 431 objects/s recovering 2024-09-17T07:19:13.818 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:19:13.852 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:13.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:13.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:19:13.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:13 smithi100 ceph-mon[29092]: osdmap e367: 8 total, 7 up, 7 in 2024-09-17T07:19:13.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:13 smithi100 ceph-mon[29092]: pgmap v785: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+remapped+backfilling, 1 peering, 2 active+undersized+degraded+remapped+backfilling, 8 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 88 active+clean; 2.1 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 843 op/s; 5294/101679 objects degraded (5.207%); 16643/101679 objects misplaced (16.368%); 27 MiB/s, 431 objects/s recovering 2024-09-17T07:19:13.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:13 smithi100 ceph-mon[39462]: osdmap e367: 8 total, 7 up, 7 in 2024-09-17T07:19:13.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:13 smithi100 ceph-mon[39462]: pgmap v785: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+remapped+backfilling, 1 peering, 2 active+undersized+degraded+remapped+backfilling, 8 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 88 active+clean; 2.1 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 843 op/s; 5294/101679 objects degraded (5.207%); 16643/101679 objects misplaced (16.368%); 27 MiB/s, 431 objects/s recovering 2024-09-17T07:19:13.954 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:19:13.984 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:13.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:13.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:19:14.045 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-17T07:19:14.086 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:19:14.120 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:14.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:14.121 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:19:14.222 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:19:14.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:14.248 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:14.248 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:19:14.290 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:19:14.349 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:19:14.376 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:14.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:14.378 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:19:14.482 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:19:14.518 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:14.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:14.519 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:19:14.620 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:19:14.654 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:14.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:14.655 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:19:14.676 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:14 smithi135 ceph-mon[33238]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-17T07:19:14.677 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:14 smithi135 ceph-mon[33238]: osdmap e368: 8 total, 7 up, 7 in 2024-09-17T07:19:14.756 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:19:14.781 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:14.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:14.782 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:19:14.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:14 smithi100 ceph-mon[29092]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-17T07:19:14.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:14 smithi100 ceph-mon[29092]: osdmap e368: 8 total, 7 up, 7 in 2024-09-17T07:19:14.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:14 smithi100 ceph-mon[39462]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-17T07:19:14.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:14 smithi100 ceph-mon[39462]: osdmap e368: 8 total, 7 up, 7 in 2024-09-17T07:19:14.882 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:19:14.905 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:14.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:14.906 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:19:15.006 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:19:15.029 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:15.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:15.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:19:15.130 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:19:15.151 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:15.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:15.152 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:19:15.253 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:19:15.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:15.279 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:15.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:19:15.381 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:19:15.406 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:15.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:15.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:19:15.508 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:19:15.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:15.530 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:15.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:19:15.602 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:19:15.603 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:19:15.631 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:19:15.654 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:15.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:15.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:19:15.657 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:15 smithi135 ceph-mon[33238]: osdmap e369: 8 total, 7 up, 7 in 2024-09-17T07:19:15.657 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:15 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 5294/101679 objects degraded (5.207%), 31 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-09-17T07:19:15.657 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:15 smithi135 ceph-mon[33238]: osdmap e370: 8 total, 7 up, 7 in 2024-09-17T07:19:15.657 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:15 smithi135 ceph-mon[33238]: pgmap v789: 168 pgs: 1 remapped+peering, 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfilling, 1 peering, 8 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 88 active+clean; 1.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.49k op/s; 3597/91557 objects degraded (3.929%); 13732/91557 objects misplaced (14.998%); 46 MiB/s, 739 objects/s recovering 2024-09-17T07:19:15.756 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:19:15.777 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:15.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:15.778 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:19:15.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:15 smithi100 ceph-mon[29092]: osdmap e369: 8 total, 7 up, 7 in 2024-09-17T07:19:15.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:15 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 5294/101679 objects degraded (5.207%), 31 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-09-17T07:19:15.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:15 smithi100 ceph-mon[29092]: osdmap e370: 8 total, 7 up, 7 in 2024-09-17T07:19:15.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:15 smithi100 ceph-mon[29092]: pgmap v789: 168 pgs: 1 remapped+peering, 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfilling, 1 peering, 8 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 88 active+clean; 1.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.49k op/s; 3597/91557 objects degraded (3.929%); 13732/91557 objects misplaced (14.998%); 46 MiB/s, 739 objects/s recovering 2024-09-17T07:19:15.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:15 smithi100 ceph-mon[39462]: osdmap e369: 8 total, 7 up, 7 in 2024-09-17T07:19:15.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:15 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 5294/101679 objects degraded (5.207%), 31 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-09-17T07:19:15.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:15 smithi100 ceph-mon[39462]: osdmap e370: 8 total, 7 up, 7 in 2024-09-17T07:19:15.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:15 smithi100 ceph-mon[39462]: pgmap v789: 168 pgs: 1 remapped+peering, 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfilling, 1 peering, 8 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 88 active+clean; 1.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.49k op/s; 3597/91557 objects degraded (3.929%); 13732/91557 objects misplaced (14.998%); 46 MiB/s, 739 objects/s recovering 2024-09-17T07:19:15.879 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:19:15.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:15.901 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:15.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:19:16.002 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:19:16.023 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:16.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:16.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:19:16.116 INFO:teuthology.orchestra.run.smithi135.stdout:{ 2024-09-17T07:19:16.116 INFO:teuthology.orchestra.run.smithi135.stdout: "ops": [], 2024-09-17T07:19:16.116 INFO:teuthology.orchestra.run.smithi135.stdout: "num_ops": 0 2024-09-17T07:19:16.116 INFO:teuthology.orchestra.run.smithi135.stdout:} 2024-09-17T07:19:16.125 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:19:16.147 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:16.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:16.148 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:19:16.227 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:19:15 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:19:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:19:16.249 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:19:16.270 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:16.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:16.271 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:19:16.372 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:19:16.412 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:16.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:16.413 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:19:16.514 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:19:16.548 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:16.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:16.549 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:19:16.650 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:19:16.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:16.682 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:16.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:19:16.783 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:19:16.819 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:16.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:16.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:19:16.921 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:19:16.958 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:16.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:16.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:19:17.012 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T07:19:17.015 INFO:teuthology.orchestra.run.smithi100.stderr:dumped all 2024-09-17T07:19:17.059 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:19:17.095 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:17.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:17.096 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:19:17.196 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:19:17.227 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:17.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:17.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:19:17.328 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:19:17.356 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:17.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:17.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:19:17.458 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:19:17.484 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:17.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:17.485 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:19:17.493 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-17T07:19:17.527 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:17 smithi135 ceph-mon[33238]: osdmap e371: 8 total, 7 up, 7 in 2024-09-17T07:19:17.586 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:19:17.604 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:19:17.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:17 smithi100 ceph-mon[29092]: osdmap e371: 8 total, 7 up, 7 in 2024-09-17T07:19:17.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:17 smithi100 ceph-mon[39462]: osdmap e371: 8 total, 7 up, 7 in 2024-09-17T07:19:17.637 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:17.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:17.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:19:17.739 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:19:17.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:17.800 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:17.801 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:19:17.846 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:19:17.897 INFO:teuthology.orchestra.run.smithi100.stdout:{"pg_ready":true,"pg_map":{"version":790,"stamp":"2024-09-17T07:19:16.272107+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1987680558,"num_objects":30519,"num_object_clones":0,"num_object_copies":91557,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3597,"num_objects_misplaced":13732,"num_objects_unfound":0,"num_objects_dirty":30519,"num_whiteouts":0,"num_read":1520,"num_read_kb":1366,"num_write":176026,"num_write_kb":6179324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26357,"num_bytes_recovered":1719341364,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":186146,"ondisk_log_size":186146,"up":428,"acting":438,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":515,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":11242984,"kb_used_data":6000972,"kb_used_omap":10,"kb_used_meta":5241589,"kb_avail":644973080,"statfs":{"total":671965249536,"available":660452433920,"internally_reserved":0,"allocated":6144995328,"data_stored":6140009100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11152,"internal_metadata":5367387248},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,2,1,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-389611520,"num_objects":-5945,"num_object_clones":0,"num_object_copies":-17835,"num_objects_missing_on_primary":0,"num_objects_missing":-1,"num_objects_degraded":-1293,"num_objects_misplaced":627,"num_objects_unfound":0,"num_objects_dirty":-5945,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":5945,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2959,"num_bytes_recovered":193855488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.838787"},"pg_stats":[{"pgid":"10.23","version":"367'13464","reported_seq":14009,"reported_epoch":367,"state":"active+remapped+backfilling","last_fresh":"2024-09-17T07:19:13.037215+0000","last_change":"2024-09-17T07:19:12.866679+0000","last_active":"2024-09-17T07:19:13.037215+0000","last_peered":"2024-09-17T07:19:13.037215+0000","last_clean":"2024-09-17T07:19:06.102950+0000","last_became_active":"2024-09-17T07:19:08.441703+0000","last_became_peered":"2024-09-17T07:19:08.441703+0000","last_unstale":"2024-09-17T07:19:13.037215+0000","last_undegraded":"2024-09-17T07:19:13.037215+0000","last_fullsized":"2024-09-17T07:19:13.037215+0000","mapping_epoch":364,"log_start":"326'7867","ondisk_log_start":"326'7867","created":360,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":6,"last_scrub":"354'12865","last_scrub_stamp":"2024-09-17T07:18:49.791465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:18:49.791465+0000","objects_scrubbed":1684,"log_size":5597,"log_dups_size":0,"ondisk_log_size":5597,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.44981474399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32112640,"num_objects":490,"num_object_clones":0,"num_object_copies":1470,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":461,"num_objects_unfound":0,"num_objects_dirty":490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":352,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1966080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[2,4,5],"avail_no_missing":["2","4","5"],"object_location_counts":[{"shards":"2,4,5","objects":490}],"blocked_by":[],"up_primary":7,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.22","version":"367'10441","reported_seq":12061,"reported_epoch":368,"state":"remapped+peering","last_fresh":"2024-09-17T07:19:13.462542+0000","last_change":"2024-09-17T07:19:13.462542+0000","last_active":"2024-09-17T07:19:13.462542+0000","last_peered":"2024-09-17T07:19:13.214483+0000","last_clean":"2024-09-17T07:16:04.025356+0000","last_became_active":"2024-09-17T07:19:08.406007+0000","last_became_peered":"2024-09-17T07:19:08.406007+0000","last_unstale":"2024-09-17T07:19:13.462542+0000","last_undegraded":"2024-09-17T07:19:13.462542+0000","last_fullsized":"2024-09-17T07:19:13.462542+0000","mapping_epoch":368,"log_start":"314'4549","ondisk_log_start":"314'4549","created":360,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":5892,"log_dups_size":0,"ondisk_log_size":5892,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31408128,"num_objects":479,"num_object_clones":0,"num_object_copies":1437,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3352,"num_write_kb":122592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":508,"num_bytes_recovered":33292288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.21","version":"366'10781","reported_seq":12110,"reported_epoch":367,"state":"peering","last_fresh":"2024-09-17T07:19:12.440193+0000","last_change":"2024-09-17T07:19:12.440354+0000","last_active":"2024-09-17T07:19:12.440193+0000","last_peered":"2024-09-17T07:19:12.437525+0000","last_clean":"2024-09-17T07:19:12.437525+0000","last_became_active":"2024-09-17T07:19:08.358190+0000","last_became_peered":"2024-09-17T07:19:08.358190+0000","last_unstale":"2024-09-17T07:19:12.440193+0000","last_undegraded":"2024-09-17T07:19:12.440193+0000","last_fullsized":"2024-09-17T07:19:12.440193+0000","mapping_epoch":367,"log_start":"316'5056","ondisk_log_start":"316'5056","created":360,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":6,"last_scrub":"320'6219","last_scrub_stamp":"2024-09-17T07:17:22.401249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:17:22.401249+0000","objects_scrubbed":6144,"log_size":5725,"log_dups_size":0,"ondisk_log_size":5725,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2.0885702689999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31653888,"num_objects":483,"num_object_clones":0,"num_object_copies":1449,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3455,"num_write_kb":126016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":540,"num_bytes_recovered":35323904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,2,7],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.20","version":"367'10500","reported_seq":10607,"reported_epoch":367,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2024-09-17T07:19:13.232853+0000","last_change":"2024-09-17T07:19:08.407333+0000","last_active":"2024-09-17T07:19:13.232853+0000","last_peered":"2024-09-17T07:19:13.232853+0000","last_clean":"2024-09-17T07:16:14.245624+0000","last_became_active":"2024-09-17T07:19:08.386176+0000","last_became_peered":"2024-09-17T07:19:08.386176+0000","last_unstale":"2024-09-17T07:19:13.232853+0000","last_undegraded":"2024-09-17T07:19:08.327640+0000","last_fullsized":"2024-09-17T07:19:08.308549+0000","mapping_epoch":364,"log_start":"313'4095","ondisk_log_start":"313'4095","created":360,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":6405,"log_dups_size":0,"ondisk_log_size":6405,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32669696,"num_objects":499,"num_object_clones":0,"num_object_copies":1497,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":499,"num_objects_misplaced":998,"num_objects_unfound":0,"num_objects_dirty":499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3361,"num_write_kb":123488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":499}],"blocked_by":[],"up_primary":0,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.17","version":"367'10759","reported_seq":17516,"reported_epoch":367,"state":"active+clean+remapped","last_fresh":"2024-09-17T07:19:13.080792+0000","last_change":"2024-09-17T07:18:18.296127+0000","last_active":"2024-09-17T07:19:13.080792+0000","last_peered":"2024-09-17T07:19:13.080792+0000","last_clean":"2024-09-17T07:19:13.080792+0000","last_became_active":"2024-09-17T07:18:18.283245+0000","last_became_peered":"2024-09-17T07:18:18.283245+0000","last_unstale":"2024-09-17T07:19:13.080792+0000","last_undegraded":"2024-09-17T07:19:13.080792+0000","last_fullsized":"2024-09-17T07:19:13.080792+0000","mapping_epoch":343,"log_start":"320'6161","ondisk_log_start":"320'6161","created":332,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":4598,"log_dups_size":0,"ondisk_log_size":4598,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61734912,"num_objects":942,"num_object_clones":0,"num_object_copies":2826,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":942,"num_objects_unfound":0,"num_objects_dirty":942,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6837,"num_write_kb":248928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3327,"num_bytes_recovered":218038272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6,2],"avail_no_missing":["4","2","6"],"object_location_counts":[{"shards":"2,4,6","objects":942}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":215,"reported_epoch":366,"state":"active+undersized","last_fresh":"2024-09-17T07:19:10.279227+0000","last_change":"2024-09-17T07:16:17.379311+0000","last_active":"2024-09-17T07:19:10.279227+0000","last_peered":"2024-09-17T07:19:10.279227+0000","last_clean":"2024-09-17T07:16:15.590051+0000","last_became_active":"2024-09-17T07:16:17.379311+0000","last_became_peered":"2024-09-17T07:16:17.379311+0000","last_unstale":"2024-09-17T07:19:10.279227+0000","last_undegraded":"2024-09-17T07:19:10.279227+0000","last_fullsized":"2024-09-17T07:16:17.372051+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:11.978185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:33.644728+0000","last_clean_scrub_stamp":"2024-09-17T07:13:11.978185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0074436570000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":395,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-17T07:19:13.457930+0000","last_change":"2024-09-17T07:18:46.393416+0000","last_active":"2024-09-17T07:19:13.457930+0000","last_peered":"2024-09-17T07:19:13.457930+0000","last_clean":"2024-09-17T07:19:13.457930+0000","last_became_active":"2024-09-17T07:02:12.006754+0000","last_became_peered":"2024-09-17T07:02:12.006754+0000","last_unstale":"2024-09-17T07:19:13.457930+0000","last_undegraded":"2024-09-17T07:19:13.457930+0000","last_fullsized":"2024-09-17T07:19:13.457930+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:18:46.393298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:18:46.393298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:19:47.980212+0000","scrub_duration":0.0058972349999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"242'27","reported_seq":305,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-17T07:19:13.458040+0000","last_change":"2024-09-17T07:18:53.477464+0000","last_active":"2024-09-17T07:19:13.458040+0000","last_peered":"2024-09-17T07:19:13.458040+0000","last_clean":"2024-09-17T07:19:13.458040+0000","last_became_active":"2024-09-17T07:09:24.402042+0000","last_became_peered":"2024-09-17T07:09:24.402042+0000","last_unstale":"2024-09-17T07:19:13.458040+0000","last_undegraded":"2024-09-17T07:19:13.458040+0000","last_fullsized":"2024-09-17T07:19:13.458040+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'27","last_scrub_stamp":"2024-09-17T07:18:53.476878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:18:53.476878+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:20:00.592721+0000","scrub_duration":0.0037504449999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":373,"reported_epoch":367,"state":"active+undersized","last_fresh":"2024-09-17T07:19:12.452352+0000","last_change":"2024-09-17T07:16:17.380766+0000","last_active":"2024-09-17T07:19:12.452352+0000","last_peered":"2024-09-17T07:19:12.452352+0000","last_clean":"2024-09-17T07:16:15.357790+0000","last_became_active":"2024-09-17T07:16:17.380766+0000","last_became_peered":"2024-09-17T07:16:17.380766+0000","last_unstale":"2024-09-17T07:19:12.452352+0000","last_undegraded":"2024-09-17T07:19:12.452352+0000","last_fullsized":"2024-09-17T07:16:17.364849+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:45.261117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:15:45.261117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001477482,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.16","version":"368'10523","reported_seq":17394,"reported_epoch":368,"state":"active+clean+remapped","last_fresh":"2024-09-17T07:19:13.733564+0000","last_change":"2024-09-17T07:18:44.528932+0000","last_active":"2024-09-17T07:19:13.733564+0000","last_peered":"2024-09-17T07:19:13.733564+0000","last_clean":"2024-09-17T07:19:13.733564+0000","last_became_active":"2024-09-17T07:18:18.284748+0000","last_became_peered":"2024-09-17T07:18:18.284748+0000","last_unstale":"2024-09-17T07:19:13.733564+0000","last_undegraded":"2024-09-17T07:19:13.733564+0000","last_fullsized":"2024-09-17T07:19:13.733564+0000","mapping_epoch":343,"log_start":"316'5322","ondisk_log_start":"316'5322","created":332,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":5,"last_scrub":"354'9544","last_scrub_stamp":"2024-09-17T07:18:44.528535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:18:44.528535+0000","objects_scrubbed":1943,"log_size":5201,"log_dups_size":0,"ondisk_log_size":5201,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:19:55.239183+0000","scrub_duration":0.61748898900000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62586880,"num_objects":955,"num_object_clones":0,"num_object_copies":2865,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":955,"num_objects_unfound":0,"num_objects_dirty":955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":988,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5,0],"avail_no_missing":["6","0","5"],"object_location_counts":[{"shards":"0,5,6","objects":955}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":234,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-17T07:19:13.460021+0000","last_change":"2024-09-17T07:17:35.507852+0000","last_active":"2024-09-17T07:19:13.460021+0000","last_peered":"2024-09-17T07:19:13.460021+0000","last_clean":"2024-09-17T07:19:13.460021+0000","last_became_active":"2024-09-17T07:09:24.402568+0000","last_became_peered":"2024-09-17T07:09:24.402568+0000","last_unstale":"2024-09-17T07:19:13.460021+0000","last_undegraded":"2024-09-17T07:19:13.460021+0000","last_fullsized":"2024-09-17T07:19:13.460021+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:17:35.507742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:17:35.507742+0000","last_clean_scrub_stamp":"2024-09-17T07:17:35.507742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00224683,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"242'15","reported_seq":411,"reported_epoch":367,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:19:12.452433+0000","last_change":"2024-09-17T07:16:17.370326+0000","last_active":"2024-09-17T07:19:12.452433+0000","last_peered":"2024-09-17T07:19:12.452433+0000","last_clean":"2024-09-17T07:16:15.357813+0000","last_became_active":"2024-09-17T07:16:17.370326+0000","last_became_peered":"2024-09-17T07:16:17.370326+0000","last_unstale":"2024-09-17T07:19:12.452433+0000","last_undegraded":"2024-09-17T07:16:17.364616+0000","last_fullsized":"2024-09-17T07:16:17.364511+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'15","last_scrub_stamp":"2024-09-17T07:15:39.306043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:15:39.306043+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002471029,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":4}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":74,"reported_epoch":367,"state":"active+undersized","last_fresh":"2024-09-17T07:19:12.451287+0000","last_change":"2024-09-17T07:16:17.379569+0000","last_active":"2024-09-17T07:19:12.451287+0000","last_peered":"2024-09-17T07:19:12.451287+0000","last_clean":"2024-09-17T07:02:14.999435+0000","last_became_active":"2024-09-17T07:16:17.379569+0000","last_became_peered":"2024-09-17T07:16:17.379569+0000","last_unstale":"2024-09-17T07:19:12.451287+0000","last_undegraded":"2024-09-17T07:19:12.451287+0000","last_fullsized":"2024-09-17T07:16:17.369924+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:53.881813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:15:53.881813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":371,"reported_epoch":366,"state":"active+undersized","last_fresh":"2024-09-17T07:19:10.290885+0000","last_change":"2024-09-17T07:16:17.374605+0000","last_active":"2024-09-17T07:19:10.290885+0000","last_peered":"2024-09-17T07:19:10.290885+0000","last_clean":"2024-09-17T07:16:15.357982+0000","last_became_active":"2024-09-17T07:16:17.374605+0000","last_became_peered":"2024-09-17T07:16:17.374605+0000","last_unstale":"2024-09-17T07:19:10.290885+0000","last_undegraded":"2024-09-17T07:19:10.290885+0000","last_fullsized":"2024-09-17T07:16:17.366300+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:41.707969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:14:41.707969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.014931757,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.15","version":"368'10651","reported_seq":15521,"reported_epoch":368,"state":"active+clean+remapped","last_fresh":"2024-09-17T07:19:13.980611+0000","last_change":"2024-09-17T07:18:40.282908+0000","last_active":"2024-09-17T07:19:13.980611+0000","last_peered":"2024-09-17T07:19:13.980611+0000","last_clean":"2024-09-17T07:19:13.980611+0000","last_became_active":"2024-09-17T07:18:40.268677+0000","last_became_peered":"2024-09-17T07:18:40.268677+0000","last_unstale":"2024-09-17T07:19:13.980611+0000","last_undegraded":"2024-09-17T07:19:13.980611+0000","last_fullsized":"2024-09-17T07:19:13.980611+0000","mapping_epoch":351,"log_start":"316'4976","ondisk_log_start":"316'4976","created":332,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":5675,"log_dups_size":0,"ondisk_log_size":5675,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62816256,"num_objects":959,"num_object_clones":0,"num_object_copies":2877,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":959,"num_objects_unfound":0,"num_objects_dirty":959,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6752,"num_write_kb":246720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2379,"num_bytes_recovered":155910144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2,0],"avail_no_missing":["5","0","2"],"object_location_counts":[{"shards":"0,2,5","objects":959}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":379,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-17T07:19:10.278200+0000","last_change":"2024-09-17T07:18:44.849429+0000","last_active":"2024-09-17T07:19:10.278200+0000","last_peered":"2024-09-17T07:19:10.278200+0000","last_clean":"2024-09-17T07:19:10.278200+0000","last_became_active":"2024-09-17T07:02:12.011129+0000","last_became_peered":"2024-09-17T07:02:12.011129+0000","last_unstale":"2024-09-17T07:19:10.278200+0000","last_undegraded":"2024-09-17T07:19:10.278200+0000","last_fullsized":"2024-09-17T07:19:10.278200+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:18:44.849299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:46.312371+0000","last_clean_scrub_stamp":"2024-09-17T07:18:44.849299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:20:14.794474+0000","scrub_duration":0.005351766,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"242'7","reported_seq":403,"reported_epoch":366,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:19:10.291788+0000","last_change":"2024-09-17T07:16:17.373486+0000","last_active":"2024-09-17T07:19:10.291788+0000","last_peered":"2024-09-17T07:19:10.291788+0000","last_clean":"2024-09-17T07:16:15.358036+0000","last_became_active":"2024-09-17T07:16:17.373486+0000","last_became_peered":"2024-09-17T07:16:17.373486+0000","last_unstale":"2024-09-17T07:19:10.291788+0000","last_undegraded":"2024-09-17T07:16:17.366228+0000","last_fullsized":"2024-09-17T07:16:17.365631+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'7","last_scrub_stamp":"2024-09-17T07:14:42.653705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:14:42.653705+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0050149230000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[{"shards":"2,4","objects":2}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":370,"reported_epoch":366,"state":"active+undersized","last_fresh":"2024-09-17T07:19:10.277018+0000","last_change":"2024-09-17T07:16:17.382323+0000","last_active":"2024-09-17T07:19:10.277018+0000","last_peered":"2024-09-17T07:19:10.277018+0000","last_clean":"2024-09-17T07:16:15.589989+0000","last_became_active":"2024-09-17T07:16:17.382323+0000","last_became_peered":"2024-09-17T07:16:17.382323+0000","last_unstale":"2024-09-17T07:19:10.277018+0000","last_undegraded":"2024-09-17T07:19:10.277018+0000","last_fullsized":"2024-09-17T07:16:17.371260+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:37.997296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:14:37.997296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045486420000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":390,"reported_epoch":367,"state":"active+undersized","last_fresh":"2024-09-17T07:19:12.451474+0000","last_change":"2024-09-17T07:16:17.374238+0000","last_active":"2024-09-17T07:19:12.451474+0000","last_peered":"2024-09-17T07:19:12.451474+0000","last_clean":"2024-09-17T07:16:16.170346+0000","last_became_active":"2024-09-17T07:16:17.374238+0000","last_became_peered":"2024-09-17T07:16:17.374238+0000","last_unstale":"2024-09-17T07:19:12.451474+0000","last_undegraded":"2024-09-17T07:19:12.451474+0000","last_fullsized":"2024-09-17T07:16:17.367446+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:52.168181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:14:52.168181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0056501399999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.14","version":"367'10588","reported_seq":17439,"reported_epoch":367,"state":"active+clean+remapped","last_fresh":"2024-09-17T07:19:13.227478+0000","last_change":"2024-09-17T07:18:18.296134+0000","last_active":"2024-09-17T07:19:13.227478+0000","last_peered":"2024-09-17T07:19:13.227478+0000","last_clean":"2024-09-17T07:19:13.227478+0000","last_became_active":"2024-09-17T07:18:18.281656+0000","last_became_peered":"2024-09-17T07:18:18.281656+0000","last_unstale":"2024-09-17T07:19:13.227478+0000","last_undegraded":"2024-09-17T07:19:13.227478+0000","last_fullsized":"2024-09-17T07:19:13.227478+0000","mapping_epoch":343,"log_start":"320'5966","ondisk_log_start":"320'5966","created":332,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":4622,"log_dups_size":0,"ondisk_log_size":4622,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63438848,"num_objects":968,"num_object_clones":0,"num_object_copies":2904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":968,"num_objects_unfound":0,"num_objects_dirty":968,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6710,"num_write_kb":245696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3370,"num_bytes_recovered":220856320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2,0],"avail_no_missing":["4","0","2"],"object_location_counts":[{"shards":"0,2,4","objects":968}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":369,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-17T07:19:10.280180+0000","last_change":"2024-09-17T07:18:34.258935+0000","last_active":"2024-09-17T07:19:10.280180+0000","last_peered":"2024-09-17T07:19:10.280180+0000","last_clean":"2024-09-17T07:19:10.280180+0000","last_became_active":"2024-09-17T07:02:12.012234+0000","last_became_peered":"2024-09-17T07:02:12.012234+0000","last_unstale":"2024-09-17T07:19:10.280180+0000","last_undegraded":"2024-09-17T07:19:10.280180+0000","last_fullsized":"2024-09-17T07:19:10.280180+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:44.076012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:45.029247+0000","last_clean_scrub_stamp":"2024-09-17T07:14:44.076012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.015917521,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"312'19","reported_seq":438,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-17T07:19:10.291005+0000","last_change":"2024-09-17T07:18:49.194511+0000","last_active":"2024-09-17T07:19:10.291005+0000","last_peered":"2024-09-17T07:19:10.291005+0000","last_clean":"2024-09-17T07:19:10.291005+0000","last_became_active":"2024-09-17T07:09:24.397954+0000","last_became_peered":"2024-09-17T07:09:24.397954+0000","last_unstale":"2024-09-17T07:19:10.291005+0000","last_undegraded":"2024-09-17T07:19:10.291005+0000","last_fullsized":"2024-09-17T07:19:10.291005+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"312'19","last_scrub_stamp":"2024-09-17T07:18:49.194375+0000","last_deep_scrub":"312'19","last_deep_scrub_stamp":"2024-09-17T07:16:44.292911+0000","last_clean_scrub_stamp":"2024-09-17T07:18:49.194375+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:19:49.607719+0000","scrub_duration":0.0031889269999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":387,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-17T07:19:12.448709+0000","last_change":"2024-09-17T07:16:45.164526+0000","last_active":"2024-09-17T07:19:12.448709+0000","last_peered":"2024-09-17T07:19:12.448709+0000","last_clean":"2024-09-17T07:19:12.448709+0000","last_became_active":"2024-09-17T07:02:16.023774+0000","last_became_peered":"2024-09-17T07:02:16.023774+0000","last_unstale":"2024-09-17T07:19:12.448709+0000","last_undegraded":"2024-09-17T07:19:12.448709+0000","last_fullsized":"2024-09-17T07:19:12.448709+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:45.164367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:28.096266+0000","last_clean_scrub_stamp":"2024-09-17T07:16:45.164367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026790630000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":372,"reported_epoch":368,"state":"active+undersized","last_fresh":"2024-09-17T07:19:13.457457+0000","last_change":"2024-09-17T07:16:17.378534+0000","last_active":"2024-09-17T07:19:13.457457+0000","last_peered":"2024-09-17T07:19:13.457457+0000","last_clean":"2024-09-17T07:16:15.494529+0000","last_became_active":"2024-09-17T07:16:17.378534+0000","last_became_peered":"2024-09-17T07:16:17.378534+0000","last_unstale":"2024-09-17T07:19:13.457457+0000","last_undegraded":"2024-09-17T07:19:13.457457+0000","last_fullsized":"2024-09-17T07:16:17.365363+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:45.483622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:15:45.483622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001342359,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3","version":"368'13515","reported_seq":13994,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-17T07:19:13.928464+0000","last_change":"2024-09-17T07:19:05.264942+0000","last_active":"2024-09-17T07:19:13.928464+0000","last_peered":"2024-09-17T07:19:13.928464+0000","last_clean":"2024-09-17T07:19:13.928464+0000","last_became_active":"2024-09-17T07:19:05.245169+0000","last_became_peered":"2024-09-17T07:19:05.245169+0000","last_unstale":"2024-09-17T07:19:13.928464+0000","last_undegraded":"2024-09-17T07:19:13.928464+0000","last_fullsized":"2024-09-17T07:19:13.928464+0000","mapping_epoch":297,"log_start":"326'7867","ondisk_log_start":"326'7867","created":297,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"354'12865","last_scrub_stamp":"2024-09-17T07:18:49.791465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:18:49.791465+0000","objects_scrubbed":1684,"log_size":5648,"log_dups_size":0,"ondisk_log_size":5648,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.44981474399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28770304,"num_objects":439,"num_object_clones":0,"num_object_copies":1317,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":403,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"242'39","reported_seq":443,"reported_epoch":366,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:19:10.293049+0000","last_change":"2024-09-17T07:16:17.374805+0000","last_active":"2024-09-17T07:19:10.293049+0000","last_peered":"2024-09-17T07:19:10.293049+0000","last_clean":"2024-09-17T07:16:15.358504+0000","last_became_active":"2024-09-17T07:16:17.374805+0000","last_became_peered":"2024-09-17T07:16:17.374805+0000","last_unstale":"2024-09-17T07:19:10.293049+0000","last_undegraded":"2024-09-17T07:16:17.367444+0000","last_fullsized":"2024-09-17T07:16:17.367331+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'39","last_scrub_stamp":"2024-09-17T07:15:46.226554+0000","last_deep_scrub":"95'25","last_deep_scrub_stamp":"2024-09-17T07:04:09.450039+0000","last_clean_scrub_stamp":"2024-09-17T07:15:46.226554+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0027061799999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[{"shards":"2,4","objects":8}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":382,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-17T07:19:10.279116+0000","last_change":"2024-09-17T07:16:33.658452+0000","last_active":"2024-09-17T07:19:10.279116+0000","last_peered":"2024-09-17T07:19:10.279116+0000","last_clean":"2024-09-17T07:19:10.279116+0000","last_became_active":"2024-09-17T07:02:12.011258+0000","last_became_peered":"2024-09-17T07:02:12.011258+0000","last_unstale":"2024-09-17T07:19:10.279116+0000","last_undegraded":"2024-09-17T07:19:10.279116+0000","last_fullsized":"2024-09-17T07:19:10.279116+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:33.658293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:16:33.658293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.044680275999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":73,"reported_epoch":366,"state":"active+undersized","last_fresh":"2024-09-17T07:19:10.298321+0000","last_change":"2024-09-17T07:16:17.372955+0000","last_active":"2024-09-17T07:19:10.298321+0000","last_peered":"2024-09-17T07:19:10.298321+0000","last_clean":"2024-09-17T07:02:14.999435+0000","last_became_active":"2024-09-17T07:16:17.372955+0000","last_became_peered":"2024-09-17T07:16:17.372955+0000","last_unstale":"2024-09-17T07:19:10.298321+0000","last_undegraded":"2024-09-17T07:19:10.298321+0000","last_fullsized":"2024-09-17T07:16:17.365245+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:46.502641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:13:46.502641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":235,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-17T07:19:10.297564+0000","last_change":"2024-09-17T07:18:42.105243+0000","last_active":"2024-09-17T07:19:10.297564+0000","last_peered":"2024-09-17T07:19:10.297564+0000","last_clean":"2024-09-17T07:19:10.297564+0000","last_became_active":"2024-09-17T07:09:24.396334+0000","last_became_peered":"2024-09-17T07:09:24.396334+0000","last_unstale":"2024-09-17T07:19:10.297564+0000","last_undegraded":"2024-09-17T07:19:10.297564+0000","last_fullsized":"2024-09-17T07:19:10.297564+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:18:42.105173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:42.984088+0000","last_clean_scrub_stamp":"2024-09-17T07:18:42.105173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:20:09.737648+0000","scrub_duration":0.0012123559999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2","version":"367'10438","reported_seq":10524,"reported_epoch":367,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:19:13.193521+0000","last_change":"2024-09-17T07:19:05.235980+0000","last_active":"2024-09-17T07:19:13.193521+0000","last_peered":"2024-09-17T07:19:13.193521+0000","last_clean":"2024-09-17T07:16:04.025356+0000","last_became_active":"2024-09-17T07:19:05.235980+0000","last_became_peered":"2024-09-17T07:19:05.235980+0000","last_unstale":"2024-09-17T07:19:13.193521+0000","last_undegraded":"2024-09-17T07:19:05.229952+0000","last_fullsized":"2024-09-17T07:19:05.229841+0000","mapping_epoch":303,"log_start":"314'4549","ondisk_log_start":"314'4549","created":297,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":5889,"log_dups_size":0,"ondisk_log_size":5889,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31604736,"num_objects":482,"num_object_clones":0,"num_object_copies":1446,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":482,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3348,"num_write_kb":122592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[{"shards":"0,4","objects":482}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"242'25","reported_seq":442,"reported_epoch":366,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:19:10.298543+0000","last_change":"2024-09-17T07:16:17.381025+0000","last_active":"2024-09-17T07:19:10.298543+0000","last_peered":"2024-09-17T07:19:10.298543+0000","last_clean":"2024-09-17T07:16:15.358419+0000","last_became_active":"2024-09-17T07:16:17.381025+0000","last_became_peered":"2024-09-17T07:16:17.381025+0000","last_unstale":"2024-09-17T07:19:10.298543+0000","last_undegraded":"2024-09-17T07:16:17.367122+0000","last_fullsized":"2024-09-17T07:16:17.365635+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'25","last_scrub_stamp":"2024-09-17T07:15:02.532371+0000","last_deep_scrub":"176'15","last_deep_scrub_stamp":"2024-09-17T07:08:19.633791+0000","last_clean_scrub_stamp":"2024-09-17T07:15:02.532371+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012712606,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":245,"reported_epoch":366,"state":"active+undersized","last_fresh":"2024-09-17T07:19:10.277386+0000","last_change":"2024-09-17T07:16:17.379662+0000","last_active":"2024-09-17T07:19:10.277386+0000","last_peered":"2024-09-17T07:19:10.277386+0000","last_clean":"2024-09-17T07:09:22.372452+0000","last_became_active":"2024-09-17T07:16:17.379662+0000","last_became_peered":"2024-09-17T07:16:17.379662+0000","last_unstale":"2024-09-17T07:19:10.277386+0000","last_undegraded":"2024-09-17T07:19:10.277386+0000","last_fullsized":"2024-09-17T07:16:17.370523+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:59.697560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:14:59.697560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0065215869999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.d","version":"98'9","reported_seq":530,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-17T07:19:10.718027+0000","last_change":"2024-09-17T07:18:37.282353+0000","last_active":"2024-09-17T07:19:10.718027+0000","last_peered":"2024-09-17T07:19:10.718027+0000","last_clean":"2024-09-17T07:19:10.718027+0000","last_became_active":"2024-09-17T07:04:15.110797+0000","last_became_peered":"2024-09-17T07:04:15.110797+0000","last_unstale":"2024-09-17T07:19:10.718027+0000","last_undegraded":"2024-09-17T07:19:10.718027+0000","last_fullsized":"2024-09-17T07:19:10.718027+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-09-17T07:14:46.126045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:14:46.126045+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.014185467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":382,"reported_epoch":366,"state":"active+undersized","last_fresh":"2024-09-17T07:19:10.293433+0000","last_change":"2024-09-17T07:16:17.373630+0000","last_active":"2024-09-17T07:19:10.293433+0000","last_peered":"2024-09-17T07:19:10.293433+0000","last_clean":"2024-09-17T07:16:15.358526+0000","last_became_active":"2024-09-17T07:16:17.373630+0000","last_became_peered":"2024-09-17T07:16:17.373630+0000","last_unstale":"2024-09-17T07:19:10.293433+0000","last_undegraded":"2024-09-17T07:19:10.293433+0000","last_fullsized":"2024-09-17T07:16:17.369047+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:50.554591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:14:50.554591+0000","last_clean_scrub_stamp":"2024-09-17T07:14:50.554591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0067125969999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1","version":"367'10792","reported_seq":11028,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-17T07:19:13.071013+0000","last_change":"2024-09-17T07:19:05.271930+0000","last_active":"2024-09-17T07:19:13.071013+0000","last_peered":"2024-09-17T07:19:13.071013+0000","last_clean":"2024-09-17T07:19:13.071013+0000","last_became_active":"2024-09-17T07:19:05.241212+0000","last_became_peered":"2024-09-17T07:19:05.241212+0000","last_unstale":"2024-09-17T07:19:13.071013+0000","last_undegraded":"2024-09-17T07:19:13.071013+0000","last_fullsized":"2024-09-17T07:19:13.071013+0000","mapping_epoch":297,"log_start":"316'5056","ondisk_log_start":"316'5056","created":297,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'6219","last_scrub_stamp":"2024-09-17T07:17:22.401249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:17:22.401249+0000","objects_scrubbed":6144,"log_size":5736,"log_dups_size":0,"ondisk_log_size":5736,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2.0885702689999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30932992,"num_objects":472,"num_object_clones":0,"num_object_copies":1416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3466,"num_write_kb":126016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"242'27","reported_seq":367,"reported_epoch":366,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:19:10.297807+0000","last_change":"2024-09-17T07:16:17.382238+0000","last_active":"2024-09-17T07:19:10.297807+0000","last_peered":"2024-09-17T07:19:10.297807+0000","last_clean":"2024-09-17T07:12:20.775697+0000","last_became_active":"2024-09-17T07:16:17.382238+0000","last_became_peered":"2024-09-17T07:16:17.382238+0000","last_unstale":"2024-09-17T07:19:10.297807+0000","last_undegraded":"2024-09-17T07:16:17.365468+0000","last_fullsized":"2024-09-17T07:16:17.365334+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'27","last_scrub_stamp":"2024-09-17T07:14:45.831734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:14:45.831734+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021853620000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[{"shards":"4,7","objects":7}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":246,"reported_epoch":366,"state":"active+undersized","last_fresh":"2024-09-17T07:19:10.281171+0000","last_change":"2024-09-17T07:16:17.378599+0000","last_active":"2024-09-17T07:19:10.281171+0000","last_peered":"2024-09-17T07:19:10.281171+0000","last_clean":"2024-09-17T07:09:22.371958+0000","last_became_active":"2024-09-17T07:16:17.378599+0000","last_became_peered":"2024-09-17T07:16:17.378599+0000","last_unstale":"2024-09-17T07:19:10.281171+0000","last_undegraded":"2024-09-17T07:19:10.281171+0000","last_fullsized":"2024-09-17T07:16:17.373236+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:44.897611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:15:44.897611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034116369999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.e","version":"75'8","reported_seq":590,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-17T07:19:10.719143+0000","last_change":"2024-09-17T07:16:49.331611+0000","last_active":"2024-09-17T07:19:10.719143+0000","last_peered":"2024-09-17T07:19:10.719143+0000","last_clean":"2024-09-17T07:19:10.719143+0000","last_became_active":"2024-09-17T07:02:16.021762+0000","last_became_peered":"2024-09-17T07:02:16.021762+0000","last_unstale":"2024-09-17T07:19:10.719143+0000","last_undegraded":"2024-09-17T07:19:10.719143+0000","last_fullsized":"2024-09-17T07:19:10.719143+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:16:49.331407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:16:49.331407+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033687029999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":381,"reported_epoch":367,"state":"active+undersized","last_fresh":"2024-09-17T07:19:12.449589+0000","last_change":"2024-09-17T07:16:17.374897+0000","last_active":"2024-09-17T07:19:12.449589+0000","last_peered":"2024-09-17T07:19:12.449589+0000","last_clean":"2024-09-17T07:16:16.170588+0000","last_became_active":"2024-09-17T07:16:17.374897+0000","last_became_peered":"2024-09-17T07:16:17.374897+0000","last_unstale":"2024-09-17T07:19:12.449589+0000","last_undegraded":"2024-09-17T07:19:12.449589+0000","last_fullsized":"2024-09-17T07:16:17.368895+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:56.146098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:14:56.146098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0054479960000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.0","version":"368'10532","reported_seq":10632,"reported_epoch":368,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:19:13.723979+0000","last_change":"2024-09-17T07:19:05.291766+0000","last_active":"2024-09-17T07:19:13.723979+0000","last_peered":"2024-09-17T07:19:13.723979+0000","last_clean":"2024-09-17T07:16:14.245624+0000","last_became_active":"2024-09-17T07:19:05.291766+0000","last_became_peered":"2024-09-17T07:19:05.291766+0000","last_unstale":"2024-09-17T07:19:13.723979+0000","last_undegraded":"2024-09-17T07:19:05.237380+0000","last_fullsized":"2024-09-17T07:19:05.237276+0000","mapping_epoch":303,"log_start":"313'4095","ondisk_log_start":"313'4095","created":297,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":6437,"log_dups_size":0,"ondisk_log_size":6437,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30572544,"num_objects":466,"num_object_clones":0,"num_object_copies":1398,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":466,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3392,"num_write_kb":123488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[{"shards":"4,6","objects":466}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"242'19","reported_seq":417,"reported_epoch":366,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:19:10.293687+0000","last_change":"2024-09-17T07:16:17.382184+0000","last_active":"2024-09-17T07:19:10.293687+0000","last_peered":"2024-09-17T07:19:10.293687+0000","last_clean":"2024-09-17T07:16:15.358681+0000","last_became_active":"2024-09-17T07:16:17.382184+0000","last_became_peered":"2024-09-17T07:16:17.382184+0000","last_unstale":"2024-09-17T07:19:10.293687+0000","last_undegraded":"2024-09-17T07:16:17.369179+0000","last_fullsized":"2024-09-17T07:16:17.369087+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'19","last_scrub_stamp":"2024-09-17T07:14:55.489874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:14:55.489874+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045924440000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":376,"reported_epoch":366,"state":"active+undersized","last_fresh":"2024-09-17T07:19:10.279885+0000","last_change":"2024-09-17T07:16:17.375239+0000","last_active":"2024-09-17T07:19:10.279885+0000","last_peered":"2024-09-17T07:19:10.279885+0000","last_clean":"2024-09-17T07:16:15.590006+0000","last_became_active":"2024-09-17T07:16:17.375239+0000","last_became_peered":"2024-09-17T07:16:17.375239+0000","last_unstale":"2024-09-17T07:19:10.279885+0000","last_undegraded":"2024-09-17T07:19:10.279885+0000","last_fullsized":"2024-09-17T07:16:17.372272+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:21.858127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:13:21.858127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033279709999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":399,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-17T07:19:12.449127+0000","last_change":"2024-09-17T07:17:18.435029+0000","last_active":"2024-09-17T07:19:12.449127+0000","last_peered":"2024-09-17T07:19:12.449127+0000","last_clean":"2024-09-17T07:19:12.449127+0000","last_became_active":"2024-09-17T07:02:16.020224+0000","last_became_peered":"2024-09-17T07:02:16.020224+0000","last_unstale":"2024-09-17T07:19:12.449127+0000","last_undegraded":"2024-09-17T07:19:12.449127+0000","last_fullsized":"2024-09-17T07:19:12.449127+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:17:18.434919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:17:18.434919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0044010059999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":253,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-17T07:19:12.442970+0000","last_change":"2024-09-17T07:18:40.926042+0000","last_active":"2024-09-17T07:19:12.442970+0000","last_peered":"2024-09-17T07:19:12.442970+0000","last_clean":"2024-09-17T07:19:12.442970+0000","last_became_active":"2024-09-17T07:09:24.408109+0000","last_became_peered":"2024-09-17T07:09:24.408109+0000","last_unstale":"2024-09-17T07:19:12.442970+0000","last_undegraded":"2024-09-17T07:19:12.442970+0000","last_fullsized":"2024-09-17T07:19:12.442970+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:18:40.925918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:13:22.187127+0000","last_clean_scrub_stamp":"2024-09-17T07:18:40.925918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:20:00.910570+0000","scrub_duration":0.0076924289999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.f","version":"368'16394","reported_seq":16791,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-17T07:19:13.751681+0000","last_change":"2024-09-17T07:19:05.318451+0000","last_active":"2024-09-17T07:19:13.751681+0000","last_peered":"2024-09-17T07:19:13.751681+0000","last_clean":"2024-09-17T07:19:13.751681+0000","last_became_active":"2024-09-17T07:19:05.258733+0000","last_became_peered":"2024-09-17T07:19:05.258733+0000","last_unstale":"2024-09-17T07:19:13.751681+0000","last_undegraded":"2024-09-17T07:19:13.751681+0000","last_fullsized":"2024-09-17T07:19:13.751681+0000","mapping_epoch":297,"log_start":"338'11653","ondisk_log_start":"338'11653","created":297,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":3774,"log_size":4741,"log_dups_size":0,"ondisk_log_size":4741,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60882944,"num_objects":929,"num_object_clones":0,"num_object_copies":2787,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6873,"num_write_kb":249664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"242'28","reported_seq":423,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-17T07:19:10.290870+0000","last_change":"2024-09-17T07:17:31.149145+0000","last_active":"2024-09-17T07:19:10.290870+0000","last_peered":"2024-09-17T07:19:10.290870+0000","last_clean":"2024-09-17T07:19:10.290870+0000","last_became_active":"2024-09-17T07:02:14.007810+0000","last_became_peered":"2024-09-17T07:02:14.007810+0000","last_unstale":"2024-09-17T07:19:10.290870+0000","last_undegraded":"2024-09-17T07:19:10.290870+0000","last_fullsized":"2024-09-17T07:19:10.290870+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'28","last_scrub_stamp":"2024-09-17T07:17:31.148909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:17:31.148909+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.013898221000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":424,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-17T07:19:12.445106+0000","last_change":"2024-09-17T07:17:19.376369+0000","last_active":"2024-09-17T07:19:12.445106+0000","last_peered":"2024-09-17T07:19:12.445106+0000","last_clean":"2024-09-17T07:19:12.445106+0000","last_became_active":"2024-09-17T07:02:12.011766+0000","last_became_peered":"2024-09-17T07:02:12.011766+0000","last_unstale":"2024-09-17T07:19:12.445106+0000","last_undegraded":"2024-09-17T07:19:12.445106+0000","last_fullsized":"2024-09-17T07:19:12.445106+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:17:19.375818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:09:23.181028+0000","last_clean_scrub_stamp":"2024-09-17T07:17:19.375818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.019104131999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"304'10","reported_seq":129,"reported_epoch":366,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:19:10.720166+0000","last_change":"2024-09-17T07:16:17.371678+0000","last_active":"2024-09-17T07:19:10.720166+0000","last_peered":"2024-09-17T07:19:10.720166+0000","last_clean":"2024-09-17T07:02:19.145975+0000","last_became_active":"2024-09-17T07:16:17.371678+0000","last_became_peered":"2024-09-17T07:16:17.371678+0000","last_unstale":"2024-09-17T07:19:10.720166+0000","last_undegraded":"2024-09-17T07:16:17.366242+0000","last_fullsized":"2024-09-17T07:16:17.365352+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'9","last_scrub_stamp":"2024-09-17T07:15:55.813882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:15:55.813882+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[{"shards":"0,4","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":386,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-17T07:19:10.278364+0000","last_change":"2024-09-17T07:18:46.921494+0000","last_active":"2024-09-17T07:19:10.278364+0000","last_peered":"2024-09-17T07:19:10.278364+0000","last_clean":"2024-09-17T07:19:10.278364+0000","last_became_active":"2024-09-17T07:02:18.364981+0000","last_became_peered":"2024-09-17T07:02:18.364981+0000","last_unstale":"2024-09-17T07:19:10.278364+0000","last_undegraded":"2024-09-17T07:19:10.278364+0000","last_fullsized":"2024-09-17T07:19:10.278364+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:18:46.921412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:10:33.727472+0000","last_clean_scrub_stamp":"2024-09-17T07:18:46.921412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:19:52.740617+0000","scrub_duration":0.0012782189999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"367'16133","reported_seq":16659,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-17T07:19:13.220373+0000","last_change":"2024-09-17T07:19:05.343557+0000","last_active":"2024-09-17T07:19:13.220373+0000","last_peered":"2024-09-17T07:19:13.220373+0000","last_clean":"2024-09-17T07:19:13.220373+0000","last_became_active":"2024-09-17T07:19:05.258339+0000","last_became_peered":"2024-09-17T07:19:05.258339+0000","last_unstale":"2024-09-17T07:19:13.220373+0000","last_undegraded":"2024-09-17T07:19:13.220373+0000","last_fullsized":"2024-09-17T07:19:13.220373+0000","mapping_epoch":297,"log_start":"338'11691","ondisk_log_start":"338'11691","created":297,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'12995","last_scrub_stamp":"2024-09-17T07:18:26.134704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:18:26.134704+0000","objects_scrubbed":5398,"log_size":4442,"log_dups_size":0,"ondisk_log_size":4442,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1.955922511,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63668224,"num_objects":971,"num_object_clones":0,"num_object_copies":2913,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6682,"num_write_kb":244928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"242'30","reported_seq":295,"reported_epoch":366,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:19:10.279486+0000","last_change":"2024-09-17T07:16:17.378001+0000","last_active":"2024-09-17T07:19:10.279486+0000","last_peered":"2024-09-17T07:19:10.279486+0000","last_clean":"2024-09-17T07:16:15.589841+0000","last_became_active":"2024-09-17T07:16:17.378001+0000","last_became_peered":"2024-09-17T07:16:17.378001+0000","last_unstale":"2024-09-17T07:19:10.279486+0000","last_undegraded":"2024-09-17T07:16:17.372487+0000","last_fullsized":"2024-09-17T07:16:17.372376+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'30","last_scrub_stamp":"2024-09-17T07:13:28.997563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:13:28.997563+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0061961159999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[{"shards":"0,7","objects":8}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":74,"reported_epoch":367,"state":"active+undersized","last_fresh":"2024-09-17T07:19:12.444798+0000","last_change":"2024-09-17T07:16:17.376547+0000","last_active":"2024-09-17T07:19:12.444798+0000","last_peered":"2024-09-17T07:19:12.444798+0000","last_clean":"2024-09-17T07:02:10.991363+0000","last_became_active":"2024-09-17T07:16:17.376547+0000","last_became_peered":"2024-09-17T07:16:17.376547+0000","last_unstale":"2024-09-17T07:19:12.444798+0000","last_undegraded":"2024-09-17T07:19:12.444798+0000","last_fullsized":"2024-09-17T07:16:17.364886+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:56.848689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:56.848689+0000","last_clean_scrub_stamp":"2024-09-17T07:15:56.848689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":388,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-17T07:19:10.290688+0000","last_change":"2024-09-17T07:16:38.184957+0000","last_active":"2024-09-17T07:19:10.290688+0000","last_peered":"2024-09-17T07:19:10.290688+0000","last_clean":"2024-09-17T07:19:10.290688+0000","last_became_active":"2024-09-17T07:09:24.393375+0000","last_became_peered":"2024-09-17T07:09:24.393375+0000","last_unstale":"2024-09-17T07:19:10.290688+0000","last_undegraded":"2024-09-17T07:19:10.290688+0000","last_fullsized":"2024-09-17T07:19:10.290688+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:38.184818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:16:38.184818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0076357700000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":386,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-17T07:19:10.291135+0000","last_change":"2024-09-17T07:16:36.162077+0000","last_active":"2024-09-17T07:19:10.291135+0000","last_peered":"2024-09-17T07:19:10.291135+0000","last_clean":"2024-09-17T07:19:10.291135+0000","last_became_active":"2024-09-17T07:09:24.398659+0000","last_became_peered":"2024-09-17T07:09:24.398659+0000","last_unstale":"2024-09-17T07:19:10.291135+0000","last_undegraded":"2024-09-17T07:19:10.291135+0000","last_fullsized":"2024-09-17T07:19:10.291135+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:36.161622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:16:36.161622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0062812850000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.d","version":"368'13375","reported_seq":13467,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-17T07:19:13.751895+0000","last_change":"2024-09-17T07:19:05.321011+0000","last_active":"2024-09-17T07:19:13.751895+0000","last_peered":"2024-09-17T07:19:13.751895+0000","last_clean":"2024-09-17T07:19:13.751895+0000","last_became_active":"2024-09-17T07:19:05.273052+0000","last_became_peered":"2024-09-17T07:19:05.273052+0000","last_unstale":"2024-09-17T07:19:13.751895+0000","last_undegraded":"2024-09-17T07:19:13.751895+0000","last_fullsized":"2024-09-17T07:19:13.751895+0000","mapping_epoch":297,"log_start":"338'8630","ondisk_log_start":"338'8630","created":297,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":4745,"log_dups_size":0,"ondisk_log_size":4745,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60817420,"num_objects":928,"num_object_clones":0,"num_object_copies":2784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6820,"num_write_kb":247936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"242'29","reported_seq":434,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-17T07:19:13.459442+0000","last_change":"2024-09-17T07:17:40.599358+0000","last_active":"2024-09-17T07:19:13.459442+0000","last_peered":"2024-09-17T07:19:13.459442+0000","last_clean":"2024-09-17T07:19:13.459442+0000","last_became_active":"2024-09-17T07:02:14.027389+0000","last_became_peered":"2024-09-17T07:02:14.027389+0000","last_unstale":"2024-09-17T07:19:13.459442+0000","last_undegraded":"2024-09-17T07:19:13.459442+0000","last_fullsized":"2024-09-17T07:19:13.459442+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'29","last_scrub_stamp":"2024-09-17T07:17:40.599184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:17:40.599184+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.073497912999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":382,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-17T07:19:10.278266+0000","last_change":"2024-09-17T07:16:47.631557+0000","last_active":"2024-09-17T07:19:10.278266+0000","last_peered":"2024-09-17T07:19:10.278266+0000","last_clean":"2024-09-17T07:19:10.278266+0000","last_became_active":"2024-09-17T07:02:12.011917+0000","last_became_peered":"2024-09-17T07:02:12.011917+0000","last_unstale":"2024-09-17T07:19:10.278266+0000","last_undegraded":"2024-09-17T07:19:10.278266+0000","last_fullsized":"2024-09-17T07:19:10.278266+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:47.631387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:47.631387+0000","last_clean_scrub_stamp":"2024-09-17T07:16:47.631387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.006082372,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":400,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-17T07:19:12.441801+0000","last_change":"2024-09-17T07:17:42.360268+0000","last_active":"2024-09-17T07:19:12.441801+0000","last_peered":"2024-09-17T07:19:12.441801+0000","last_clean":"2024-09-17T07:19:12.441801+0000","last_became_active":"2024-09-17T07:02:16.016199+0000","last_became_peered":"2024-09-17T07:02:16.016199+0000","last_unstale":"2024-09-17T07:19:12.441801+0000","last_undegraded":"2024-09-17T07:19:12.441801+0000","last_fullsized":"2024-09-17T07:19:12.441801+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:17:42.360104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:11:17.583712+0000","last_clean_scrub_stamp":"2024-09-17T07:17:42.360104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0062057919999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":74,"reported_epoch":367,"state":"active+undersized","last_fresh":"2024-09-17T07:19:12.441690+0000","last_change":"2024-09-17T07:16:17.368186+0000","last_active":"2024-09-17T07:19:12.441690+0000","last_peered":"2024-09-17T07:19:12.441690+0000","last_clean":"2024-09-17T07:02:17.002904+0000","last_became_active":"2024-09-17T07:16:17.368186+0000","last_became_peered":"2024-09-17T07:16:17.368186+0000","last_unstale":"2024-09-17T07:19:12.441690+0000","last_undegraded":"2024-09-17T07:19:12.441690+0000","last_fullsized":"2024-09-17T07:16:17.362908+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:50.514525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:13:50.514525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.c","version":"367'13357","reported_seq":13524,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-17T07:19:12.930215+0000","last_change":"2024-09-17T07:19:05.283203+0000","last_active":"2024-09-17T07:19:12.930215+0000","last_peered":"2024-09-17T07:19:12.930215+0000","last_clean":"2024-09-17T07:19:12.930215+0000","last_became_active":"2024-09-17T07:19:05.249271+0000","last_became_peered":"2024-09-17T07:19:05.249271+0000","last_unstale":"2024-09-17T07:19:12.930215+0000","last_undegraded":"2024-09-17T07:19:12.930215+0000","last_fullsized":"2024-09-17T07:19:12.930215+0000","mapping_epoch":297,"log_start":"338'8766","ondisk_log_start":"338'8766","created":297,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"354'12224","last_scrub_stamp":"2024-09-17T07:18:50.964766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:18:50.964766+0000","objects_scrubbed":3319,"log_size":4591,"log_dups_size":0,"ondisk_log_size":4591,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1.073020833,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62357504,"num_objects":951,"num_object_clones":0,"num_object_copies":2853,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":951,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6798,"num_write_kb":248000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"242'25","reported_seq":335,"reported_epoch":367,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:19:12.447542+0000","last_change":"2024-09-17T07:16:17.376630+0000","last_active":"2024-09-17T07:19:12.447542+0000","last_peered":"2024-09-17T07:19:12.447542+0000","last_clean":"2024-09-17T07:12:20.656798+0000","last_became_active":"2024-09-17T07:16:17.376630+0000","last_became_peered":"2024-09-17T07:16:17.376630+0000","last_unstale":"2024-09-17T07:19:12.447542+0000","last_undegraded":"2024-09-17T07:16:17.368029+0000","last_fullsized":"2024-09-17T07:16:17.367816+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'25","last_scrub_stamp":"2024-09-17T07:14:50.794200+0000","last_deep_scrub":"110'15","last_deep_scrub_stamp":"2024-09-17T07:04:57.091400+0000","last_clean_scrub_stamp":"2024-09-17T07:14:50.794200+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0070558269999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[{"shards":"4,5","objects":5}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":75,"reported_epoch":368,"state":"active+undersized","last_fresh":"2024-09-17T07:19:13.458766+0000","last_change":"2024-09-17T07:16:17.378414+0000","last_active":"2024-09-17T07:19:13.458766+0000","last_peered":"2024-09-17T07:19:13.458766+0000","last_clean":"2024-09-17T07:02:10.991363+0000","last_became_active":"2024-09-17T07:16:17.378414+0000","last_became_peered":"2024-09-17T07:16:17.378414+0000","last_unstale":"2024-09-17T07:19:13.458766+0000","last_undegraded":"2024-09-17T07:19:13.458766+0000","last_fullsized":"2024-09-17T07:16:17.365679+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:49.764885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:25.147878+0000","last_clean_scrub_stamp":"2024-09-17T07:14:49.764885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"89'9","reported_seq":604,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-17T07:19:13.458112+0000","last_change":"2024-09-17T07:17:41.595535+0000","last_active":"2024-09-17T07:19:13.458112+0000","last_peered":"2024-09-17T07:19:13.458112+0000","last_clean":"2024-09-17T07:19:13.458112+0000","last_became_active":"2024-09-17T07:02:16.021360+0000","last_became_peered":"2024-09-17T07:02:16.021360+0000","last_unstale":"2024-09-17T07:19:13.458112+0000","last_undegraded":"2024-09-17T07:19:13.458112+0000","last_fullsized":"2024-09-17T07:19:13.458112+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'9","last_scrub_stamp":"2024-09-17T07:17:41.595032+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-09-17T07:15:53.512777+0000","last_clean_scrub_stamp":"2024-09-17T07:17:41.595032+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.031478313000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"268'2","reported_seq":404,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-17T07:19:13.458177+0000","last_change":"2024-09-17T07:18:49.469622+0000","last_active":"2024-09-17T07:19:13.458177+0000","last_peered":"2024-09-17T07:19:13.458177+0000","last_clean":"2024-09-17T07:19:13.458177+0000","last_became_active":"2024-09-17T07:09:24.399565+0000","last_became_peered":"2024-09-17T07:09:24.399565+0000","last_unstale":"2024-09-17T07:19:13.458177+0000","last_undegraded":"2024-09-17T07:19:13.458177+0000","last_fullsized":"2024-09-17T07:19:13.458177+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'2","last_scrub_stamp":"2024-09-17T07:18:49.469494+0000","last_deep_scrub":"268'2","last_deep_scrub_stamp":"2024-09-17T07:14:03.609131+0000","last_clean_scrub_stamp":"2024-09-17T07:18:49.469494+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:20:01.585713+0000","scrub_duration":0.0037058479999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.a","version":"368'13173","reported_seq":13263,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-17T07:19:13.738561+0000","last_change":"2024-09-17T07:19:05.289881+0000","last_active":"2024-09-17T07:19:13.738561+0000","last_peered":"2024-09-17T07:19:13.738561+0000","last_clean":"2024-09-17T07:19:13.738561+0000","last_became_active":"2024-09-17T07:19:05.244277+0000","last_became_peered":"2024-09-17T07:19:05.244277+0000","last_unstale":"2024-09-17T07:19:13.738561+0000","last_undegraded":"2024-09-17T07:19:13.738561+0000","last_fullsized":"2024-09-17T07:19:13.738561+0000","mapping_epoch":297,"log_start":"338'8450","ondisk_log_start":"338'8450","created":297,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":4723,"log_dups_size":0,"ondisk_log_size":4723,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64028672,"num_objects":977,"num_object_clones":0,"num_object_copies":2931,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6717,"num_write_kb":246208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"304'42","reported_seq":555,"reported_epoch":367,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:19:12.442775+0000","last_change":"2024-09-17T07:16:17.368142+0000","last_active":"2024-09-17T07:19:12.442775+0000","last_peered":"2024-09-17T07:19:12.442775+0000","last_clean":"2024-09-17T07:12:20.666266+0000","last_became_active":"2024-09-17T07:16:17.368142+0000","last_became_peered":"2024-09-17T07:16:17.368142+0000","last_unstale":"2024-09-17T07:19:12.442775+0000","last_undegraded":"2024-09-17T07:16:17.363645+0000","last_fullsized":"2024-09-17T07:16:17.363541+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'41","last_scrub_stamp":"2024-09-17T07:14:56.700858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:14:56.700858+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026058610000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[{"shards":"2,6","objects":10}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":388,"reported_epoch":367,"state":"active+undersized","last_fresh":"2024-09-17T07:19:12.448118+0000","last_change":"2024-09-17T07:16:17.371561+0000","last_active":"2024-09-17T07:19:12.448118+0000","last_peered":"2024-09-17T07:19:12.448118+0000","last_clean":"2024-09-17T07:16:16.170577+0000","last_became_active":"2024-09-17T07:16:17.371561+0000","last_became_peered":"2024-09-17T07:16:17.371561+0000","last_unstale":"2024-09-17T07:19:12.448118+0000","last_undegraded":"2024-09-17T07:19:12.448118+0000","last_fullsized":"2024-09-17T07:16:17.366918+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:03.045709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:14:03.045709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0049245929999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":386,"reported_epoch":368,"state":"active+undersized","last_fresh":"2024-09-17T07:19:13.459082+0000","last_change":"2024-09-17T07:16:17.375376+0000","last_active":"2024-09-17T07:19:13.459082+0000","last_peered":"2024-09-17T07:19:13.459082+0000","last_clean":"2024-09-17T07:16:15.495109+0000","last_became_active":"2024-09-17T07:16:17.375376+0000","last_became_peered":"2024-09-17T07:16:17.375376+0000","last_unstale":"2024-09-17T07:19:13.459082+0000","last_undegraded":"2024-09-17T07:19:13.459082+0000","last_fullsized":"2024-09-17T07:16:17.364236+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:36.523676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:15:36.523676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017370549999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":232,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-17T07:19:10.293333+0000","last_change":"2024-09-17T07:17:18.312624+0000","last_active":"2024-09-17T07:19:10.293333+0000","last_peered":"2024-09-17T07:19:10.293333+0000","last_clean":"2024-09-17T07:19:10.293333+0000","last_became_active":"2024-09-17T07:09:24.398878+0000","last_became_peered":"2024-09-17T07:09:24.398878+0000","last_unstale":"2024-09-17T07:19:10.293333+0000","last_undegraded":"2024-09-17T07:19:10.293333+0000","last_fullsized":"2024-09-17T07:19:10.293333+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:17:18.312499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:13:53.406661+0000","last_clean_scrub_stamp":"2024-09-17T07:17:18.312499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0030147720000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.9","version":"368'10589","reported_seq":10865,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-17T07:19:13.919575+0000","last_change":"2024-09-17T07:18:46.761729+0000","last_active":"2024-09-17T07:19:13.919575+0000","last_peered":"2024-09-17T07:19:13.919575+0000","last_clean":"2024-09-17T07:19:13.919575+0000","last_became_active":"2024-09-17T07:17:52.386125+0000","last_became_peered":"2024-09-17T07:17:52.386125+0000","last_unstale":"2024-09-17T07:19:13.919575+0000","last_undegraded":"2024-09-17T07:19:13.919575+0000","last_fullsized":"2024-09-17T07:19:13.919575+0000","mapping_epoch":297,"log_start":"316'4899","ondisk_log_start":"316'4899","created":297,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":0,"last_scrub":"354'9713","last_scrub_stamp":"2024-09-17T07:18:46.761585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:18:46.761585+0000","objects_scrubbed":1810,"log_size":5690,"log_dups_size":0,"ondisk_log_size":5690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:20:09.045403+0000","scrub_duration":0.36938599100000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60817408,"num_objects":928,"num_object_clones":0,"num_object_copies":2784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":882,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"242'17","reported_seq":333,"reported_epoch":367,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:19:12.449322+0000","last_change":"2024-09-17T07:16:17.371977+0000","last_active":"2024-09-17T07:19:12.449322+0000","last_peered":"2024-09-17T07:19:12.449322+0000","last_clean":"2024-09-17T07:12:20.284331+0000","last_became_active":"2024-09-17T07:16:17.371977+0000","last_became_peered":"2024-09-17T07:16:17.371977+0000","last_unstale":"2024-09-17T07:19:12.449322+0000","last_undegraded":"2024-09-17T07:16:17.367143+0000","last_fullsized":"2024-09-17T07:16:17.367038+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'17","last_scrub_stamp":"2024-09-17T07:13:47.468399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:13:47.468399+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.005881725,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[{"shards":"0,5","objects":5}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"76'5","reported_seq":279,"reported_epoch":367,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:19:12.995300+0000","last_change":"2024-09-17T07:16:17.368338+0000","last_active":"2024-09-17T07:19:12.995300+0000","last_peered":"2024-09-17T07:19:12.995300+0000","last_clean":"2024-09-17T07:02:23.985427+0000","last_became_active":"2024-09-17T07:16:17.368338+0000","last_became_peered":"2024-09-17T07:16:17.368338+0000","last_unstale":"2024-09-17T07:19:12.995300+0000","last_undegraded":"2024-09-17T07:16:17.363185+0000","last_fullsized":"2024-09-17T07:16:17.363086+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'5","last_scrub_stamp":"2024-09-17T07:15:36.901601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:15:36.901601+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033739799999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":182,"num_read_kb":177,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[{"shards":"2,6","objects":1}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":230,"reported_epoch":368,"state":"active+undersized","last_fresh":"2024-09-17T07:19:13.458142+0000","last_change":"2024-09-17T07:16:17.367238+0000","last_active":"2024-09-17T07:19:13.458142+0000","last_peered":"2024-09-17T07:19:13.458142+0000","last_clean":"2024-09-17T07:16:15.494771+0000","last_became_active":"2024-09-17T07:16:17.367238+0000","last_became_peered":"2024-09-17T07:16:17.367238+0000","last_unstale":"2024-09-17T07:19:13.458142+0000","last_undegraded":"2024-09-17T07:19:13.458142+0000","last_fullsized":"2024-09-17T07:16:17.364334+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:56.534547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:23.833170+0000","last_clean_scrub_stamp":"2024-09-17T07:15:56.534547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0011018950000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":411,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-17T07:19:10.280415+0000","last_change":"2024-09-17T07:18:42.051741+0000","last_active":"2024-09-17T07:19:10.280415+0000","last_peered":"2024-09-17T07:19:10.280415+0000","last_clean":"2024-09-17T07:19:10.280415+0000","last_became_active":"2024-09-17T07:02:16.014492+0000","last_became_peered":"2024-09-17T07:02:16.014492+0000","last_unstale":"2024-09-17T07:19:10.280415+0000","last_undegraded":"2024-09-17T07:19:10.280415+0000","last_fullsized":"2024-09-17T07:19:10.280415+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:18:42.051667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:18:42.051667+0000","last_clean_scrub_stamp":"2024-09-17T07:18:42.051667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:19:51.231165+0000","scrub_duration":0.0011360980000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":385,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-17T07:19:10.277952+0000","last_change":"2024-09-17T07:18:39.328796+0000","last_active":"2024-09-17T07:19:10.277952+0000","last_peered":"2024-09-17T07:19:10.277952+0000","last_clean":"2024-09-17T07:19:10.277952+0000","last_became_active":"2024-09-17T07:09:24.402704+0000","last_became_peered":"2024-09-17T07:09:24.402704+0000","last_unstale":"2024-09-17T07:19:10.277952+0000","last_undegraded":"2024-09-17T07:19:10.277952+0000","last_fullsized":"2024-09-17T07:19:10.277952+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:02.062618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:02.062618+0000","last_clean_scrub_stamp":"2024-09-17T07:15:02.062618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0042833460000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"369'10693","reported_seq":10835,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-17T07:19:14.605643+0000","last_change":"2024-09-17T07:18:43.933340+0000","last_active":"2024-09-17T07:19:14.605643+0000","last_peered":"2024-09-17T07:19:14.605643+0000","last_clean":"2024-09-17T07:19:14.605643+0000","last_became_active":"2024-09-17T07:17:52.372680+0000","last_became_peered":"2024-09-17T07:17:52.372680+0000","last_unstale":"2024-09-17T07:19:14.605643+0000","last_undegraded":"2024-09-17T07:19:14.605643+0000","last_fullsized":"2024-09-17T07:19:14.605643+0000","mapping_epoch":297,"log_start":"320'5942","ondisk_log_start":"320'5942","created":297,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":0,"last_scrub":"354'9680","last_scrub_stamp":"2024-09-17T07:18:43.933160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:18:43.933160+0000","objects_scrubbed":1923,"log_size":4751,"log_dups_size":0,"ondisk_log_size":4751,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:19:58.555529+0000","scrub_duration":0.45499974300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59179008,"num_objects":903,"num_object_clones":0,"num_object_copies":2709,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1020,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"268'30","reported_seq":457,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-17T07:19:13.459941+0000","last_change":"2024-09-17T07:18:40.426710+0000","last_active":"2024-09-17T07:19:13.459941+0000","last_peered":"2024-09-17T07:19:13.459941+0000","last_clean":"2024-09-17T07:19:13.459941+0000","last_became_active":"2024-09-17T07:03:48.760405+0000","last_became_peered":"2024-09-17T07:03:48.760405+0000","last_unstale":"2024-09-17T07:19:13.459941+0000","last_undegraded":"2024-09-17T07:19:13.459941+0000","last_fullsized":"2024-09-17T07:19:13.459941+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'30","last_scrub_stamp":"2024-09-17T07:18:40.426597+0000","last_deep_scrub":"268'30","last_deep_scrub_stamp":"2024-09-17T07:14:00.599656+0000","last_clean_scrub_stamp":"2024-09-17T07:18:40.426597+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:19:44.097951+0000","scrub_duration":0.0031470769999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"70'1","reported_seq":403,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-17T07:19:13.458350+0000","last_change":"2024-09-17T07:17:26.441837+0000","last_active":"2024-09-17T07:19:13.458350+0000","last_peered":"2024-09-17T07:19:13.458350+0000","last_clean":"2024-09-17T07:19:13.458350+0000","last_became_active":"2024-09-17T07:09:24.402499+0000","last_became_peered":"2024-09-17T07:09:24.402499+0000","last_unstale":"2024-09-17T07:19:13.458350+0000","last_undegraded":"2024-09-17T07:19:13.458350+0000","last_fullsized":"2024-09-17T07:19:13.458350+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-17T07:17:26.441670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:17:26.441670+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0054314109999999997,"objects_trimmed":0,"snaptrim_duration":0.0032298629999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":376,"reported_epoch":366,"state":"active+undersized","last_fresh":"2024-09-17T07:19:10.279993+0000","last_change":"2024-09-17T07:16:17.375018+0000","last_active":"2024-09-17T07:19:10.279993+0000","last_peered":"2024-09-17T07:19:10.279993+0000","last_clean":"2024-09-17T07:16:15.590088+0000","last_became_active":"2024-09-17T07:16:17.375018+0000","last_became_peered":"2024-09-17T07:16:17.375018+0000","last_unstale":"2024-09-17T07:19:10.279993+0000","last_undegraded":"2024-09-17T07:19:10.279993+0000","last_fullsized":"2024-09-17T07:16:17.370688+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:55.321202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:55.321202+0000","last_clean_scrub_stamp":"2024-09-17T07:15:55.321202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014716950000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":383,"reported_epoch":367,"state":"active+undersized","last_fresh":"2024-09-17T07:19:12.449201+0000","last_change":"2024-09-17T07:16:17.375774+0000","last_active":"2024-09-17T07:19:12.449201+0000","last_peered":"2024-09-17T07:19:12.449201+0000","last_clean":"2024-09-17T07:16:16.169907+0000","last_became_active":"2024-09-17T07:16:17.375774+0000","last_became_peered":"2024-09-17T07:16:17.375774+0000","last_unstale":"2024-09-17T07:19:12.449201+0000","last_undegraded":"2024-09-17T07:19:12.449201+0000","last_fullsized":"2024-09-17T07:16:17.368453+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:45.447029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:15:45.447029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001990463,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":240,"reported_epoch":367,"state":"active+undersized","last_fresh":"2024-09-17T07:19:12.449189+0000","last_change":"2024-09-17T07:16:17.372859+0000","last_active":"2024-09-17T07:19:12.449189+0000","last_peered":"2024-09-17T07:19:12.449189+0000","last_clean":"2024-09-17T07:09:22.371903+0000","last_became_active":"2024-09-17T07:16:17.372859+0000","last_became_peered":"2024-09-17T07:16:17.372859+0000","last_unstale":"2024-09-17T07:19:12.449189+0000","last_undegraded":"2024-09-17T07:19:12.449189+0000","last_fullsized":"2024-09-17T07:16:17.367296+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:51.877367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:15:51.877367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.058290885000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.b","version":"368'13300","reported_seq":13393,"reported_epoch":368,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:19:13.632080+0000","last_change":"2024-09-17T07:19:05.266491+0000","last_active":"2024-09-17T07:19:13.632080+0000","last_peered":"2024-09-17T07:19:13.632080+0000","last_clean":"2024-09-17T07:16:14.246161+0000","last_became_active":"2024-09-17T07:19:05.266491+0000","last_became_peered":"2024-09-17T07:19:05.266491+0000","last_unstale":"2024-09-17T07:19:13.632080+0000","last_undegraded":"2024-09-17T07:19:05.237499+0000","last_fullsized":"2024-09-17T07:19:05.237375+0000","mapping_epoch":303,"log_start":"338'8557","ondisk_log_start":"338'8557","created":297,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":4743,"log_dups_size":0,"ondisk_log_size":4743,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59768832,"num_objects":912,"num_object_clones":0,"num_object_copies":2736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":912,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6782,"num_write_kb":246208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[{"shards":"0,6","objects":912}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"242'26","reported_seq":433,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-17T07:19:12.443983+0000","last_change":"2024-09-17T07:18:37.273399+0000","last_active":"2024-09-17T07:19:12.443983+0000","last_peered":"2024-09-17T07:19:12.443983+0000","last_clean":"2024-09-17T07:19:12.443983+0000","last_became_active":"2024-09-17T07:02:14.028857+0000","last_became_peered":"2024-09-17T07:02:14.028857+0000","last_unstale":"2024-09-17T07:19:12.443983+0000","last_undegraded":"2024-09-17T07:19:12.443983+0000","last_fullsized":"2024-09-17T07:19:12.443983+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'26","last_scrub_stamp":"2024-09-17T07:16:33.666684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:16:33.666684+0000","objects_scrubbed":7,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.013353596000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"268'3","reported_seq":252,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-17T07:19:12.447843+0000","last_change":"2024-09-17T07:17:19.453842+0000","last_active":"2024-09-17T07:19:12.447843+0000","last_peered":"2024-09-17T07:19:12.447843+0000","last_clean":"2024-09-17T07:19:12.447843+0000","last_became_active":"2024-09-17T07:09:24.404532+0000","last_became_peered":"2024-09-17T07:09:24.404532+0000","last_unstale":"2024-09-17T07:19:12.447843+0000","last_undegraded":"2024-09-17T07:19:12.447843+0000","last_fullsized":"2024-09-17T07:19:12.447843+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'3","last_scrub_stamp":"2024-09-17T07:17:19.444444+0000","last_deep_scrub":"268'3","last_deep_scrub_stamp":"2024-09-17T07:13:54.949345+0000","last_clean_scrub_stamp":"2024-09-17T07:17:19.444444+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0048257380000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":404,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-17T07:19:12.449095+0000","last_change":"2024-09-17T07:17:34.639287+0000","last_active":"2024-09-17T07:19:12.449095+0000","last_peered":"2024-09-17T07:19:12.449095+0000","last_clean":"2024-09-17T07:19:12.449095+0000","last_became_active":"2024-09-17T07:02:12.004606+0000","last_became_peered":"2024-09-17T07:02:12.004606+0000","last_unstale":"2024-09-17T07:19:12.449095+0000","last_undegraded":"2024-09-17T07:19:12.449095+0000","last_fullsized":"2024-09-17T07:19:12.449095+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:17:34.638733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:17:34.638733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.064159545999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"212'130","reported_seq":632,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-17T07:19:10.280077+0000","last_change":"2024-09-17T07:18:35.277546+0000","last_active":"2024-09-17T07:19:10.280077+0000","last_peered":"2024-09-17T07:19:10.280077+0000","last_clean":"2024-09-17T07:19:10.280077+0000","last_became_active":"2024-09-17T07:02:03.020675+0000","last_became_peered":"2024-09-17T07:02:03.020675+0000","last_unstale":"2024-09-17T07:19:10.280077+0000","last_undegraded":"2024-09-17T07:19:10.280077+0000","last_fullsized":"2024-09-17T07:19:10.280077+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:14:57.052505+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:14:57.052505+0000","last_clean_scrub_stamp":"2024-09-17T07:14:57.052505+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.022122466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":388,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-17T07:19:10.279835+0000","last_change":"2024-09-17T07:18:48.891849+0000","last_active":"2024-09-17T07:19:10.279835+0000","last_peered":"2024-09-17T07:19:10.279835+0000","last_clean":"2024-09-17T07:19:10.279835+0000","last_became_active":"2024-09-17T07:02:16.016031+0000","last_became_peered":"2024-09-17T07:02:16.016031+0000","last_unstale":"2024-09-17T07:19:10.279835+0000","last_undegraded":"2024-09-17T07:19:10.279835+0000","last_fullsized":"2024-09-17T07:19:10.279835+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:18:48.891136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:18:48.891136+0000","last_clean_scrub_stamp":"2024-09-17T07:18:48.891136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:20:18.076925+0000","scrub_duration":0.010734751000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":379,"reported_epoch":367,"state":"active+undersized","last_fresh":"2024-09-17T07:19:12.449459+0000","last_change":"2024-09-17T07:16:17.377095+0000","last_active":"2024-09-17T07:19:12.449459+0000","last_peered":"2024-09-17T07:19:12.449459+0000","last_clean":"2024-09-17T07:16:16.170840+0000","last_became_active":"2024-09-17T07:16:17.377095+0000","last_became_peered":"2024-09-17T07:16:17.377095+0000","last_unstale":"2024-09-17T07:19:12.449459+0000","last_undegraded":"2024-09-17T07:19:12.449459+0000","last_fullsized":"2024-09-17T07:16:17.370787+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:48.021592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:58.670571+0000","last_clean_scrub_stamp":"2024-09-17T07:14:48.021592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0064844619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.4","version":"369'10666","reported_seq":10761,"reported_epoch":369,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:19:14.590735+0000","last_change":"2024-09-17T07:17:52.376990+0000","last_active":"2024-09-17T07:19:14.590735+0000","last_peered":"2024-09-17T07:19:14.590735+0000","last_clean":"2024-09-17T07:16:15.495005+0000","last_became_active":"2024-09-17T07:17:52.376990+0000","last_became_peered":"2024-09-17T07:17:52.376990+0000","last_unstale":"2024-09-17T07:19:14.590735+0000","last_undegraded":"2024-09-17T07:17:52.365057+0000","last_fullsized":"2024-09-17T07:17:52.364949+0000","mapping_epoch":303,"log_start":"320'5931","ondisk_log_start":"320'5931","created":297,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":4735,"log_dups_size":0,"ondisk_log_size":4735,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58327040,"num_objects":890,"num_object_clones":0,"num_object_copies":2670,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6788,"num_write_kb":245696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0,4","objects":890}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"242'35","reported_seq":452,"reported_epoch":367,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:19:12.442768+0000","last_change":"2024-09-17T07:16:17.376246+0000","last_active":"2024-09-17T07:19:12.442768+0000","last_peered":"2024-09-17T07:19:12.442768+0000","last_clean":"2024-09-17T07:16:14.244770+0000","last_became_active":"2024-09-17T07:16:17.376246+0000","last_became_peered":"2024-09-17T07:16:17.376246+0000","last_unstale":"2024-09-17T07:19:12.442768+0000","last_undegraded":"2024-09-17T07:16:17.363365+0000","last_fullsized":"2024-09-17T07:16:17.363257+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'35","last_scrub_stamp":"2024-09-17T07:15:43.665829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:15:43.665829+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0048325160000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[{"shards":"6,7","objects":9}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":437,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-17T07:19:10.280450+0000","last_change":"2024-09-17T07:18:50.077819+0000","last_active":"2024-09-17T07:19:10.280450+0000","last_peered":"2024-09-17T07:19:10.280450+0000","last_clean":"2024-09-17T07:19:10.280450+0000","last_became_active":"2024-09-17T07:02:12.012037+0000","last_became_peered":"2024-09-17T07:02:12.012037+0000","last_unstale":"2024-09-17T07:19:10.280450+0000","last_undegraded":"2024-09-17T07:19:10.280450+0000","last_fullsized":"2024-09-17T07:19:10.280450+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:18:50.077716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:06:39.322535+0000","last_clean_scrub_stamp":"2024-09-17T07:18:50.077716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:20:14.706290+0000","scrub_duration":0.0025671019999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":421,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-17T07:19:10.280246+0000","last_change":"2024-09-17T07:18:44.061764+0000","last_active":"2024-09-17T07:19:10.280246+0000","last_peered":"2024-09-17T07:19:10.280246+0000","last_clean":"2024-09-17T07:19:10.280246+0000","last_became_active":"2024-09-17T07:02:16.014631+0000","last_became_peered":"2024-09-17T07:02:16.014631+0000","last_unstale":"2024-09-17T07:19:10.280246+0000","last_undegraded":"2024-09-17T07:19:10.280246+0000","last_fullsized":"2024-09-17T07:19:10.280246+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:18:44.061681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:18:44.061681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:19:47.840600+0000","scrub_duration":0.00146547,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":383,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-17T07:19:10.279997+0000","last_change":"2024-09-17T07:18:47.881259+0000","last_active":"2024-09-17T07:19:10.279997+0000","last_peered":"2024-09-17T07:19:10.279997+0000","last_clean":"2024-09-17T07:19:10.279997+0000","last_became_active":"2024-09-17T07:09:24.403904+0000","last_became_peered":"2024-09-17T07:09:24.403904+0000","last_unstale":"2024-09-17T07:19:10.279997+0000","last_undegraded":"2024-09-17T07:19:10.279997+0000","last_fullsized":"2024-09-17T07:19:10.279997+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:18:47.880943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:45.275473+0000","last_clean_scrub_stamp":"2024-09-17T07:18:47.880943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:19:54.108267+0000","scrub_duration":0.0039294250000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"369'10742","reported_seq":10884,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-17T07:19:14.564353+0000","last_change":"2024-09-17T07:18:41.742362+0000","last_active":"2024-09-17T07:19:14.564353+0000","last_peered":"2024-09-17T07:19:14.564353+0000","last_clean":"2024-09-17T07:19:14.564353+0000","last_became_active":"2024-09-17T07:17:52.375445+0000","last_became_peered":"2024-09-17T07:17:52.375445+0000","last_unstale":"2024-09-17T07:19:14.564353+0000","last_undegraded":"2024-09-17T07:19:14.564353+0000","last_fullsized":"2024-09-17T07:19:14.564353+0000","mapping_epoch":297,"log_start":"320'6070","ondisk_log_start":"320'6070","created":297,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":0,"last_scrub":"353'9682","last_scrub_stamp":"2024-09-17T07:18:41.742224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:18:41.742224+0000","objects_scrubbed":1968,"log_size":4672,"log_dups_size":0,"ondisk_log_size":4672,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:19:57.931343+0000","scrub_duration":0.28187726499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59113472,"num_objects":902,"num_object_clones":0,"num_object_copies":2706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1066,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"268'19","reported_seq":405,"reported_epoch":368,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:19:13.459312+0000","last_change":"2024-09-17T07:16:17.375651+0000","last_active":"2024-09-17T07:19:13.459312+0000","last_peered":"2024-09-17T07:19:13.459312+0000","last_clean":"2024-09-17T07:16:15.494820+0000","last_became_active":"2024-09-17T07:16:17.375651+0000","last_became_peered":"2024-09-17T07:16:17.375651+0000","last_unstale":"2024-09-17T07:19:13.459312+0000","last_undegraded":"2024-09-17T07:16:17.364532+0000","last_fullsized":"2024-09-17T07:16:17.364433+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'19","last_scrub_stamp":"2024-09-17T07:14:06.634334+0000","last_deep_scrub":"268'19","last_deep_scrub_stamp":"2024-09-17T07:14:06.634334+0000","last_clean_scrub_stamp":"2024-09-17T07:14:06.634334+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0088943289999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0,4","objects":4}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":245,"reported_epoch":366,"state":"active+undersized","last_fresh":"2024-09-17T07:19:10.293263+0000","last_change":"2024-09-17T07:16:17.374110+0000","last_active":"2024-09-17T07:19:10.293263+0000","last_peered":"2024-09-17T07:19:10.293263+0000","last_clean":"2024-09-17T07:09:22.372513+0000","last_became_active":"2024-09-17T07:16:17.374110+0000","last_became_peered":"2024-09-17T07:16:17.374110+0000","last_unstale":"2024-09-17T07:19:10.293263+0000","last_undegraded":"2024-09-17T07:19:10.293263+0000","last_fullsized":"2024-09-17T07:16:17.367137+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:58.886294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:13:53.593078+0000","last_clean_scrub_stamp":"2024-09-17T07:15:58.886294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.005853245,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":352,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-17T07:19:10.290692+0000","last_change":"2024-09-17T07:16:35.152814+0000","last_active":"2024-09-17T07:19:10.290692+0000","last_peered":"2024-09-17T07:19:10.290692+0000","last_clean":"2024-09-17T07:19:10.290692+0000","last_became_active":"2024-09-17T07:09:24.396642+0000","last_became_peered":"2024-09-17T07:09:24.396642+0000","last_unstale":"2024-09-17T07:19:10.290692+0000","last_undegraded":"2024-09-17T07:19:10.290692+0000","last_fullsized":"2024-09-17T07:19:10.290692+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:35.152744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:16:35.152744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0022724160000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":377,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-17T07:19:13.458459+0000","last_change":"2024-09-17T07:17:27.482509+0000","last_active":"2024-09-17T07:19:13.458459+0000","last_peered":"2024-09-17T07:19:13.458459+0000","last_clean":"2024-09-17T07:19:13.458459+0000","last_became_active":"2024-09-17T07:02:18.026994+0000","last_became_peered":"2024-09-17T07:02:18.026994+0000","last_unstale":"2024-09-17T07:19:13.458459+0000","last_undegraded":"2024-09-17T07:19:13.458459+0000","last_fullsized":"2024-09-17T07:19:13.458459+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:17:27.482401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:58.099849+0000","last_clean_scrub_stamp":"2024-09-17T07:17:27.482401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0020589919999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.6","version":"368'12032","reported_seq":12388,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-17T07:19:13.928756+0000","last_change":"2024-09-17T07:19:02.389516+0000","last_active":"2024-09-17T07:19:13.928756+0000","last_peered":"2024-09-17T07:19:13.928756+0000","last_clean":"2024-09-17T07:19:13.928756+0000","last_became_active":"2024-09-17T07:17:52.398777+0000","last_became_peered":"2024-09-17T07:17:52.398777+0000","last_unstale":"2024-09-17T07:19:13.928756+0000","last_undegraded":"2024-09-17T07:19:13.928756+0000","last_fullsized":"2024-09-17T07:19:13.928756+0000","mapping_epoch":297,"log_start":"331'7714","ondisk_log_start":"331'7714","created":297,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'11699","last_scrub_stamp":"2024-09-17T07:19:02.389174+0000","last_deep_scrub":"358'11699","last_deep_scrub_stamp":"2024-09-17T07:19:02.389174+0000","last_clean_scrub_stamp":"2024-09-17T07:19:02.389174+0000","objects_scrubbed":1374,"log_size":4318,"log_dups_size":0,"ondisk_log_size":4318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":9,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:20:25.105928+0000","scrub_duration":8.0456348519999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59637760,"num_objects":910,"num_object_clones":0,"num_object_copies":2730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":464,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"242'34","reported_seq":440,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-17T07:19:12.446904+0000","last_change":"2024-09-17T07:16:44.126984+0000","last_active":"2024-09-17T07:19:12.446904+0000","last_peered":"2024-09-17T07:19:12.446904+0000","last_clean":"2024-09-17T07:19:12.446904+0000","last_became_active":"2024-09-17T07:02:14.028392+0000","last_became_peered":"2024-09-17T07:02:14.028392+0000","last_unstale":"2024-09-17T07:19:12.446904+0000","last_undegraded":"2024-09-17T07:19:12.446904+0000","last_fullsized":"2024-09-17T07:19:12.446904+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'34","last_scrub_stamp":"2024-09-17T07:16:44.126830+0000","last_deep_scrub":"242'34","last_deep_scrub_stamp":"2024-09-17T07:14:50.103539+0000","last_clean_scrub_stamp":"2024-09-17T07:16:44.126830+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.008313473,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'2","reported_seq":263,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-17T07:19:10.277401+0000","last_change":"2024-09-17T07:18:43.861957+0000","last_active":"2024-09-17T07:19:10.277401+0000","last_peered":"2024-09-17T07:19:10.277401+0000","last_clean":"2024-09-17T07:19:10.277401+0000","last_became_active":"2024-09-17T07:09:24.402909+0000","last_became_peered":"2024-09-17T07:09:24.402909+0000","last_unstale":"2024-09-17T07:19:10.277401+0000","last_undegraded":"2024-09-17T07:19:10.277401+0000","last_fullsized":"2024-09-17T07:19:10.277401+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-09-17T07:18:43.861842+0000","last_deep_scrub":"89'2","last_deep_scrub_stamp":"2024-09-17T07:12:20.301263+0000","last_clean_scrub_stamp":"2024-09-17T07:18:43.861842+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:19:59.925131+0000","scrub_duration":0.0028010019999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":420,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":420,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"225'9","reported_seq":592,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-17T07:19:10.718348+0000","last_change":"2024-09-17T07:18:23.789276+0000","last_active":"2024-09-17T07:19:10.718348+0000","last_peered":"2024-09-17T07:19:10.718348+0000","last_clean":"2024-09-17T07:19:10.718348+0000","last_became_active":"2024-09-17T07:02:16.015418+0000","last_became_peered":"2024-09-17T07:02:16.015418+0000","last_unstale":"2024-09-17T07:19:10.718348+0000","last_undegraded":"2024-09-17T07:19:10.718348+0000","last_fullsized":"2024-09-17T07:19:10.718348+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"225'9","last_scrub_stamp":"2024-09-17T07:18:23.788769+0000","last_deep_scrub":"225'9","last_deep_scrub_stamp":"2024-09-17T07:11:25.292565+0000","last_clean_scrub_stamp":"2024-09-17T07:18:23.788769+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:19:43.054958+0000","scrub_duration":0.006761141,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":397,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-17T07:19:12.447259+0000","last_change":"2024-09-17T07:18:41.386243+0000","last_active":"2024-09-17T07:19:12.447259+0000","last_peered":"2024-09-17T07:19:12.447259+0000","last_clean":"2024-09-17T07:19:12.447259+0000","last_became_active":"2024-09-17T07:02:18.363253+0000","last_became_peered":"2024-09-17T07:02:18.363253+0000","last_unstale":"2024-09-17T07:19:12.447259+0000","last_undegraded":"2024-09-17T07:19:12.447259+0000","last_fullsized":"2024-09-17T07:19:12.447259+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:18:41.385928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:18:41.385928+0000","last_clean_scrub_stamp":"2024-09-17T07:18:41.385928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:20:04.219102+0000","scrub_duration":0.0023027270000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.7","version":"368'10747","reported_seq":10838,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-17T07:19:13.731158+0000","last_change":"2024-09-17T07:18:37.046901+0000","last_active":"2024-09-17T07:19:13.731158+0000","last_peered":"2024-09-17T07:19:13.731158+0000","last_clean":"2024-09-17T07:19:13.731158+0000","last_became_active":"2024-09-17T07:17:52.402213+0000","last_became_peered":"2024-09-17T07:17:52.402213+0000","last_unstale":"2024-09-17T07:19:13.731158+0000","last_undegraded":"2024-09-17T07:19:13.731158+0000","last_fullsized":"2024-09-17T07:19:13.731158+0000","mapping_epoch":297,"log_start":"319'5658","ondisk_log_start":"319'5658","created":297,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":5089,"log_dups_size":0,"ondisk_log_size":5089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62521344,"num_objects":954,"num_object_clones":0,"num_object_copies":2862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6825,"num_write_kb":248928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"242'23","reported_seq":470,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-17T07:19:10.290765+0000","last_change":"2024-09-17T07:17:34.108758+0000","last_active":"2024-09-17T07:19:10.290765+0000","last_peered":"2024-09-17T07:19:10.290765+0000","last_clean":"2024-09-17T07:19:10.290765+0000","last_became_active":"2024-09-17T07:10:49.723292+0000","last_became_peered":"2024-09-17T07:10:49.723292+0000","last_unstale":"2024-09-17T07:19:10.290765+0000","last_undegraded":"2024-09-17T07:19:10.290765+0000","last_fullsized":"2024-09-17T07:19:10.290765+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'23","last_scrub_stamp":"2024-09-17T07:17:34.108619+0000","last_deep_scrub":"150'17","last_deep_scrub_stamp":"2024-09-17T07:06:38.775788+0000","last_clean_scrub_stamp":"2024-09-17T07:17:34.108619+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.018319322999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"68'2","reported_seq":400,"reported_epoch":366,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:19:10.298194+0000","last_change":"2024-09-17T07:16:17.371234+0000","last_active":"2024-09-17T07:19:10.298194+0000","last_peered":"2024-09-17T07:19:10.298194+0000","last_clean":"2024-09-17T07:16:15.358653+0000","last_became_active":"2024-09-17T07:16:17.371234+0000","last_became_peered":"2024-09-17T07:16:17.371234+0000","last_unstale":"2024-09-17T07:19:10.298194+0000","last_undegraded":"2024-09-17T07:16:17.366653+0000","last_fullsized":"2024-09-17T07:16:17.366079+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-09-17T07:14:56.533798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:14:56.533798+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003207115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[{"shards":"0,4","objects":2}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":349,"reported_epoch":368,"state":"active+undersized","last_fresh":"2024-09-17T07:19:13.457320+0000","last_change":"2024-09-17T07:16:17.367290+0000","last_active":"2024-09-17T07:19:13.457320+0000","last_peered":"2024-09-17T07:19:13.457320+0000","last_clean":"2024-09-17T07:16:15.494380+0000","last_became_active":"2024-09-17T07:16:17.367290+0000","last_became_peered":"2024-09-17T07:16:17.367290+0000","last_unstale":"2024-09-17T07:19:13.457320+0000","last_undegraded":"2024-09-17T07:19:13.457320+0000","last_fullsized":"2024-09-17T07:16:17.363484+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:51.586929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:11:17.479036+0000","last_clean_scrub_stamp":"2024-09-17T07:15:51.586929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001274435,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":216,"reported_epoch":366,"state":"active+undersized","last_fresh":"2024-09-17T07:19:10.277244+0000","last_change":"2024-09-17T07:16:17.379791+0000","last_active":"2024-09-17T07:19:10.277244+0000","last_peered":"2024-09-17T07:19:10.277244+0000","last_clean":"2024-09-17T07:16:15.590199+0000","last_became_active":"2024-09-17T07:16:17.379791+0000","last_became_peered":"2024-09-17T07:16:17.379791+0000","last_unstale":"2024-09-17T07:19:10.277244+0000","last_undegraded":"2024-09-17T07:19:10.277244+0000","last_fullsized":"2024-09-17T07:16:17.372333+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:13.937979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:13:13.937979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.018487173999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.18","version":"367'10639","reported_seq":17557,"reported_epoch":367,"state":"active+clean+remapped","last_fresh":"2024-09-17T07:19:12.997822+0000","last_change":"2024-09-17T07:18:49.456041+0000","last_active":"2024-09-17T07:19:12.997822+0000","last_peered":"2024-09-17T07:19:12.997822+0000","last_clean":"2024-09-17T07:19:12.997822+0000","last_became_active":"2024-09-17T07:18:18.281655+0000","last_became_peered":"2024-09-17T07:18:18.281655+0000","last_unstale":"2024-09-17T07:19:12.997822+0000","last_undegraded":"2024-09-17T07:19:12.997822+0000","last_fullsized":"2024-09-17T07:19:12.997822+0000","mapping_epoch":343,"log_start":"316'5177","ondisk_log_start":"316'5177","created":332,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":5,"last_scrub":"354'9921","last_scrub_stamp":"2024-09-17T07:18:49.455799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:18:49.455799+0000","objects_scrubbed":1720,"log_size":5462,"log_dups_size":0,"ondisk_log_size":5462,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:20:17.992458+0000","scrub_duration":0.42320338699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65339392,"num_objects":997,"num_object_clones":0,"num_object_copies":2991,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":997,"num_objects_unfound":0,"num_objects_dirty":997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":723,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6,0],"avail_no_missing":["2","0","6"],"object_location_counts":[{"shards":"0,2,6","objects":997}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"242'15","reported_seq":436,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-17T07:19:13.459736+0000","last_change":"2024-09-17T07:18:24.505103+0000","last_active":"2024-09-17T07:19:13.459736+0000","last_peered":"2024-09-17T07:19:13.459736+0000","last_clean":"2024-09-17T07:19:13.459736+0000","last_became_active":"2024-09-17T07:09:24.408875+0000","last_became_peered":"2024-09-17T07:09:24.408875+0000","last_unstale":"2024-09-17T07:19:13.459736+0000","last_undegraded":"2024-09-17T07:19:13.459736+0000","last_fullsized":"2024-09-17T07:19:13.459736+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'15","last_scrub_stamp":"2024-09-17T07:18:24.505002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:18:24.505002+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:19:35.852600+0000","scrub_duration":0.002195809,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":410,"reported_epoch":366,"state":"active+undersized","last_fresh":"2024-09-17T07:19:10.284501+0000","last_change":"2024-09-17T07:16:17.367792+0000","last_active":"2024-09-17T07:19:10.284501+0000","last_peered":"2024-09-17T07:19:10.284501+0000","last_clean":"2024-09-17T07:16:14.244722+0000","last_became_active":"2024-09-17T07:16:17.367792+0000","last_became_peered":"2024-09-17T07:16:17.367792+0000","last_unstale":"2024-09-17T07:19:10.284501+0000","last_undegraded":"2024-09-17T07:19:10.284501+0000","last_fullsized":"2024-09-17T07:16:17.366356+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:39.634698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:17.976267+0000","last_clean_scrub_stamp":"2024-09-17T07:15:39.634698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00117202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":75,"reported_epoch":366,"state":"active+undersized","last_fresh":"2024-09-17T07:19:10.279786+0000","last_change":"2024-09-17T07:16:17.378272+0000","last_active":"2024-09-17T07:19:10.279786+0000","last_peered":"2024-09-17T07:19:10.279786+0000","last_clean":"2024-09-17T07:02:14.999435+0000","last_became_active":"2024-09-17T07:16:17.378272+0000","last_became_peered":"2024-09-17T07:16:17.378272+0000","last_unstale":"2024-09-17T07:19:10.279786+0000","last_undegraded":"2024-09-17T07:19:10.279786+0000","last_fullsized":"2024-09-17T07:16:17.373119+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:40.845816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:18.367486+0000","last_clean_scrub_stamp":"2024-09-17T07:15:40.845816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":431,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-17T07:19:10.279858+0000","last_change":"2024-09-17T07:17:37.827552+0000","last_active":"2024-09-17T07:19:10.279858+0000","last_peered":"2024-09-17T07:19:10.279858+0000","last_clean":"2024-09-17T07:19:10.279858+0000","last_became_active":"2024-09-17T07:02:18.017825+0000","last_became_peered":"2024-09-17T07:02:18.017825+0000","last_unstale":"2024-09-17T07:19:10.279858+0000","last_undegraded":"2024-09-17T07:19:10.279858+0000","last_fullsized":"2024-09-17T07:19:10.279858+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:17:37.827296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:09:23.378805+0000","last_clean_scrub_stamp":"2024-09-17T07:17:37.827296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0050232260000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"367'10505","reported_seq":17613,"reported_epoch":367,"state":"active+clean+remapped","last_fresh":"2024-09-17T07:19:13.110268+0000","last_change":"2024-09-17T07:18:42.357433+0000","last_active":"2024-09-17T07:19:13.110268+0000","last_peered":"2024-09-17T07:19:13.110268+0000","last_clean":"2024-09-17T07:19:13.110268+0000","last_became_active":"2024-09-17T07:18:42.341643+0000","last_became_peered":"2024-09-17T07:18:42.341643+0000","last_unstale":"2024-09-17T07:19:13.110268+0000","last_undegraded":"2024-09-17T07:19:13.110268+0000","last_fullsized":"2024-09-17T07:19:13.110268+0000","mapping_epoch":353,"log_start":"320'5821","ondisk_log_start":"320'5821","created":332,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"322'6203","last_scrub_stamp":"2024-09-17T07:17:25.670907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:17:25.670907+0000","objects_scrubbed":6134,"log_size":4684,"log_dups_size":0,"ondisk_log_size":4684,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2.260616706,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64651264,"num_objects":987,"num_object_clones":0,"num_object_copies":2961,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":987,"num_objects_unfound":0,"num_objects_dirty":987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6652,"num_write_kb":244416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2286,"num_bytes_recovered":149815296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0,2],"avail_no_missing":["4","0","2"],"object_location_counts":[{"shards":"0,2,4","objects":987}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":399,"reported_epoch":367,"state":"active+undersized","last_fresh":"2024-09-17T07:19:12.442097+0000","last_change":"2024-09-17T07:16:17.375999+0000","last_active":"2024-09-17T07:19:12.442097+0000","last_peered":"2024-09-17T07:19:12.442097+0000","last_clean":"2024-09-17T07:16:14.245868+0000","last_became_active":"2024-09-17T07:16:17.375999+0000","last_became_peered":"2024-09-17T07:16:17.375999+0000","last_unstale":"2024-09-17T07:19:12.442097+0000","last_undegraded":"2024-09-17T07:19:12.442097+0000","last_fullsized":"2024-09-17T07:16:17.363383+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:46.645080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:46.645080+0000","last_clean_scrub_stamp":"2024-09-17T07:15:46.645080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015661049999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"242'12","reported_seq":253,"reported_epoch":366,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:19:10.279511+0000","last_change":"2024-09-17T07:16:17.379665+0000","last_active":"2024-09-17T07:19:10.279511+0000","last_peered":"2024-09-17T07:19:10.279511+0000","last_clean":"2024-09-17T07:16:15.590425+0000","last_became_active":"2024-09-17T07:16:17.379665+0000","last_became_peered":"2024-09-17T07:16:17.379665+0000","last_unstale":"2024-09-17T07:19:10.279511+0000","last_undegraded":"2024-09-17T07:16:17.373659+0000","last_fullsized":"2024-09-17T07:16:17.373552+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'12","last_scrub_stamp":"2024-09-17T07:15:44.249858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:15:44.249858+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0022006600000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":3}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":392,"reported_epoch":368,"state":"active+undersized","last_fresh":"2024-09-17T07:19:13.463421+0000","last_change":"2024-09-17T07:16:17.377495+0000","last_active":"2024-09-17T07:19:13.463421+0000","last_peered":"2024-09-17T07:19:13.463421+0000","last_clean":"2024-09-17T07:16:16.170775+0000","last_became_active":"2024-09-17T07:16:17.377495+0000","last_became_peered":"2024-09-17T07:16:17.377495+0000","last_unstale":"2024-09-17T07:19:13.463421+0000","last_undegraded":"2024-09-17T07:19:13.463421+0000","last_fullsized":"2024-09-17T07:16:17.367529+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:55.191252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:14:55.191252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0090150360000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":364,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-17T07:19:12.442315+0000","last_change":"2024-09-17T07:18:38.319273+0000","last_active":"2024-09-17T07:19:12.442315+0000","last_peered":"2024-09-17T07:19:12.442315+0000","last_clean":"2024-09-17T07:19:12.442315+0000","last_became_active":"2024-09-17T07:02:18.365419+0000","last_became_peered":"2024-09-17T07:02:18.365419+0000","last_unstale":"2024-09-17T07:19:12.442315+0000","last_undegraded":"2024-09-17T07:19:12.442315+0000","last_fullsized":"2024-09-17T07:19:12.442315+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:52.058847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:14:52.058847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0054926549999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1a","version":"368'13186","reported_seq":13276,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-17T07:19:13.769713+0000","last_change":"2024-09-17T07:19:05.332048+0000","last_active":"2024-09-17T07:19:13.769713+0000","last_peered":"2024-09-17T07:19:13.769713+0000","last_clean":"2024-09-17T07:19:13.769713+0000","last_became_active":"2024-09-17T07:19:05.309076+0000","last_became_peered":"2024-09-17T07:19:05.309076+0000","last_unstale":"2024-09-17T07:19:13.769713+0000","last_undegraded":"2024-09-17T07:19:13.769713+0000","last_fullsized":"2024-09-17T07:19:13.769713+0000","mapping_epoch":360,"log_start":"338'8450","ondisk_log_start":"338'8450","created":360,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":4736,"log_dups_size":0,"ondisk_log_size":4736,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63176704,"num_objects":964,"num_object_clones":0,"num_object_copies":2892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":964,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6730,"num_write_kb":246208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"242'20","reported_seq":196,"reported_epoch":366,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:19:10.279932+0000","last_change":"2024-09-17T07:16:17.380213+0000","last_active":"2024-09-17T07:19:10.279932+0000","last_peered":"2024-09-17T07:19:10.279932+0000","last_clean":"2024-09-17T07:12:20.630919+0000","last_became_active":"2024-09-17T07:16:17.380213+0000","last_became_peered":"2024-09-17T07:16:17.380213+0000","last_unstale":"2024-09-17T07:19:10.279932+0000","last_undegraded":"2024-09-17T07:16:17.375354+0000","last_fullsized":"2024-09-17T07:16:17.375256+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'20","last_scrub_stamp":"2024-09-17T07:14:05.528769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:14:05.528769+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0020069329999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":5}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":412,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-17T07:19:12.447979+0000","last_change":"2024-09-17T07:17:20.425482+0000","last_active":"2024-09-17T07:19:12.447979+0000","last_peered":"2024-09-17T07:19:12.447979+0000","last_clean":"2024-09-17T07:19:12.447979+0000","last_became_active":"2024-09-17T07:09:24.402355+0000","last_became_peered":"2024-09-17T07:09:24.402355+0000","last_unstale":"2024-09-17T07:19:12.447979+0000","last_undegraded":"2024-09-17T07:19:12.447979+0000","last_fullsized":"2024-09-17T07:19:12.447979+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:17:20.425212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:17:20.425212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0020024069999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"304'10","reported_seq":597,"reported_epoch":368,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:19:13.462574+0000","last_change":"2024-09-17T07:16:17.373566+0000","last_active":"2024-09-17T07:19:13.462574+0000","last_peered":"2024-09-17T07:19:13.462574+0000","last_clean":"2024-09-17T07:16:16.169875+0000","last_became_active":"2024-09-17T07:16:17.373566+0000","last_became_peered":"2024-09-17T07:16:17.373566+0000","last_unstale":"2024-09-17T07:19:13.462574+0000","last_undegraded":"2024-09-17T07:16:17.367771+0000","last_fullsized":"2024-09-17T07:16:17.367624+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'9","last_scrub_stamp":"2024-09-17T07:14:53.140133+0000","last_deep_scrub":"163'9","last_deep_scrub_stamp":"2024-09-17T07:07:34.933808+0000","last_clean_scrub_stamp":"2024-09-17T07:14:53.140133+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0088819989999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[{"shards":"0,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":397,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-17T07:19:14.464925+0000","last_change":"2024-09-17T07:18:47.392268+0000","last_active":"2024-09-17T07:19:14.464925+0000","last_peered":"2024-09-17T07:19:14.464925+0000","last_clean":"2024-09-17T07:19:14.464925+0000","last_became_active":"2024-09-17T07:09:24.409051+0000","last_became_peered":"2024-09-17T07:09:24.409051+0000","last_unstale":"2024-09-17T07:19:14.464925+0000","last_undegraded":"2024-09-17T07:19:14.464925+0000","last_fullsized":"2024-09-17T07:19:14.464925+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:18:47.392182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:26.619893+0000","last_clean_scrub_stamp":"2024-09-17T07:18:47.392182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:20:03.629550+0000","scrub_duration":0.0053055120000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1b","version":"369'13279","reported_seq":16401,"reported_epoch":369,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-17T07:19:14.610755+0000","last_change":"2024-09-17T07:19:08.460732+0000","last_active":"2024-09-17T07:19:14.610755+0000","last_peered":"2024-09-17T07:19:14.610755+0000","last_clean":"2024-09-17T07:16:14.246161+0000","last_became_active":"2024-09-17T07:19:08.410847+0000","last_became_peered":"2024-09-17T07:19:08.410847+0000","last_unstale":"2024-09-17T07:19:14.610755+0000","last_undegraded":"2024-09-17T07:19:08.329051+0000","last_fullsized":"2024-09-17T07:19:08.305450+0000","mapping_epoch":364,"log_start":"338'8557","ondisk_log_start":"338'8557","created":360,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":4722,"log_dups_size":0,"ondisk_log_size":4722,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61145088,"num_objects":933,"num_object_clones":0,"num_object_copies":2799,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":480,"num_objects_unfound":0,"num_objects_dirty":933,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6761,"num_write_kb":246208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":755,"num_bytes_recovered":49479680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[{"shards":"0,6","objects":933}],"blocked_by":[],"up_primary":4,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"242'15","reported_seq":434,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-17T07:19:12.449748+0000","last_change":"2024-09-17T07:16:34.100665+0000","last_active":"2024-09-17T07:19:12.449748+0000","last_peered":"2024-09-17T07:19:12.449748+0000","last_clean":"2024-09-17T07:19:12.449748+0000","last_became_active":"2024-09-17T07:09:24.406333+0000","last_became_peered":"2024-09-17T07:09:24.406333+0000","last_unstale":"2024-09-17T07:19:12.449748+0000","last_undegraded":"2024-09-17T07:19:12.449748+0000","last_fullsized":"2024-09-17T07:19:12.449748+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'15","last_scrub_stamp":"2024-09-17T07:16:34.100540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:16:34.100540+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033371669999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":396,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-17T07:19:13.459962+0000","last_change":"2024-09-17T07:17:18.561114+0000","last_active":"2024-09-17T07:19:13.459962+0000","last_peered":"2024-09-17T07:19:13.459962+0000","last_clean":"2024-09-17T07:19:13.459962+0000","last_became_active":"2024-09-17T07:09:24.402704+0000","last_became_peered":"2024-09-17T07:09:24.402704+0000","last_unstale":"2024-09-17T07:19:13.459962+0000","last_undegraded":"2024-09-17T07:19:13.459962+0000","last_fullsized":"2024-09-17T07:19:13.459962+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:17:18.560995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:17:18.560995+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0064484570000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"304'10","reported_seq":359,"reported_epoch":366,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:19:10.718234+0000","last_change":"2024-09-17T07:16:17.380279+0000","last_active":"2024-09-17T07:19:10.718234+0000","last_peered":"2024-09-17T07:19:10.718234+0000","last_clean":"2024-09-17T07:16:15.687380+0000","last_became_active":"2024-09-17T07:16:17.380279+0000","last_became_peered":"2024-09-17T07:16:17.380279+0000","last_unstale":"2024-09-17T07:19:10.718234+0000","last_undegraded":"2024-09-17T07:16:17.375037+0000","last_fullsized":"2024-09-17T07:16:17.374926+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'9","last_scrub_stamp":"2024-09-17T07:13:10.965855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:13:10.965855+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0088433720000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":386,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-17T07:19:10.297635+0000","last_change":"2024-09-17T07:18:43.122873+0000","last_active":"2024-09-17T07:19:10.297635+0000","last_peered":"2024-09-17T07:19:10.297635+0000","last_clean":"2024-09-17T07:19:10.297635+0000","last_became_active":"2024-09-17T07:02:18.018716+0000","last_became_peered":"2024-09-17T07:02:18.018716+0000","last_unstale":"2024-09-17T07:19:10.297635+0000","last_undegraded":"2024-09-17T07:19:10.297635+0000","last_fullsized":"2024-09-17T07:19:10.297635+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:18:43.121570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:18:43.121570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:19:44.821007+0000","scrub_duration":0.010220139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1c","version":"365'13202","reported_seq":13730,"reported_epoch":365,"state":"active+remapped+backfilling","last_fresh":"2024-09-17T07:19:09.608822+0000","last_change":"2024-09-17T07:19:08.421724+0000","last_active":"2024-09-17T07:19:09.608822+0000","last_peered":"2024-09-17T07:19:09.608822+0000","last_clean":"2024-09-17T07:19:06.272175+0000","last_became_active":"2024-09-17T07:19:08.386182+0000","last_became_peered":"2024-09-17T07:19:08.386182+0000","last_unstale":"2024-09-17T07:19:09.608822+0000","last_undegraded":"2024-09-17T07:19:09.608822+0000","last_fullsized":"2024-09-17T07:19:09.608822+0000","mapping_epoch":364,"log_start":"338'8766","ondisk_log_start":"338'8766","created":360,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":5,"last_scrub":"354'12224","last_scrub_stamp":"2024-09-17T07:18:50.964766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:18:50.964766+0000","objects_scrubbed":3319,"log_size":4436,"log_dups_size":0,"ondisk_log_size":4436,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1.073020833,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72515584,"num_objects":1107,"num_object_clones":0,"num_object_copies":3321,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2041,"num_objects_unfound":0,"num_objects_dirty":1107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6644,"num_write_kb":248000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":11534336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[0,4,7],"avail_no_missing":["0","4","7"],"object_location_counts":[{"shards":"0,4,7","objects":1107}],"blocked_by":[],"up_primary":6,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"242'15","reported_seq":325,"reported_epoch":367,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:19:12.442330+0000","last_change":"2024-09-17T07:16:17.368982+0000","last_active":"2024-09-17T07:19:12.442330+0000","last_peered":"2024-09-17T07:19:12.442330+0000","last_clean":"2024-09-17T07:12:20.132949+0000","last_became_active":"2024-09-17T07:16:17.368982+0000","last_became_peered":"2024-09-17T07:16:17.368982+0000","last_unstale":"2024-09-17T07:19:12.442330+0000","last_undegraded":"2024-09-17T07:16:17.363353+0000","last_fullsized":"2024-09-17T07:16:17.363248+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'15","last_scrub_stamp":"2024-09-17T07:13:48.500257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:13:48.500257+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.005053564,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[{"shards":"2,6","objects":4}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":423,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-17T07:19:12.443123+0000","last_change":"2024-09-17T07:18:53.905202+0000","last_active":"2024-09-17T07:19:12.443123+0000","last_peered":"2024-09-17T07:19:12.443123+0000","last_clean":"2024-09-17T07:19:12.443123+0000","last_became_active":"2024-09-17T07:09:24.408139+0000","last_became_peered":"2024-09-17T07:09:24.408139+0000","last_unstale":"2024-09-17T07:19:12.443123+0000","last_undegraded":"2024-09-17T07:19:12.443123+0000","last_fullsized":"2024-09-17T07:19:12.443123+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:18:53.905079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:10:36.334827+0000","last_clean_scrub_stamp":"2024-09-17T07:18:53.905079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:20:06.824558+0000","scrub_duration":0.0087069899999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":229,"reported_epoch":368,"state":"active+undersized","last_fresh":"2024-09-17T07:19:13.457615+0000","last_change":"2024-09-17T07:16:17.373522+0000","last_active":"2024-09-17T07:19:13.457615+0000","last_peered":"2024-09-17T07:19:13.457615+0000","last_clean":"2024-09-17T07:16:15.494646+0000","last_became_active":"2024-09-17T07:16:17.373522+0000","last_became_peered":"2024-09-17T07:16:17.373522+0000","last_unstale":"2024-09-17T07:19:13.457615+0000","last_undegraded":"2024-09-17T07:19:13.457615+0000","last_fullsized":"2024-09-17T07:16:17.364098+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:37.523283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:15:37.523283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013475850000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":383,"reported_epoch":368,"state":"active+undersized","last_fresh":"2024-09-17T07:19:13.457183+0000","last_change":"2024-09-17T07:16:17.371270+0000","last_active":"2024-09-17T07:19:13.457183+0000","last_peered":"2024-09-17T07:19:13.457183+0000","last_clean":"2024-09-17T07:16:15.494422+0000","last_became_active":"2024-09-17T07:16:17.371270+0000","last_became_peered":"2024-09-17T07:16:17.371270+0000","last_unstale":"2024-09-17T07:19:13.457183+0000","last_undegraded":"2024-09-17T07:19:13.457183+0000","last_fullsized":"2024-09-17T07:16:17.363365+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:40.540336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:15:40.540336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001212982,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1d","version":"367'13352","reported_seq":13456,"reported_epoch":367,"state":"active+clean+remapped","last_fresh":"2024-09-17T07:19:13.214553+0000","last_change":"2024-09-17T07:19:08.429506+0000","last_active":"2024-09-17T07:19:13.214553+0000","last_peered":"2024-09-17T07:19:13.214553+0000","last_clean":"2024-09-17T07:19:13.214553+0000","last_became_active":"2024-09-17T07:19:08.375473+0000","last_became_peered":"2024-09-17T07:19:08.375473+0000","last_unstale":"2024-09-17T07:19:13.214553+0000","last_undegraded":"2024-09-17T07:19:13.214553+0000","last_fullsized":"2024-09-17T07:19:13.214553+0000","mapping_epoch":364,"log_start":"338'8630","ondisk_log_start":"338'8630","created":360,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":4722,"log_dups_size":0,"ondisk_log_size":4722,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62324748,"num_objects":952,"num_object_clones":0,"num_object_copies":2856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":952,"num_objects_unfound":0,"num_objects_dirty":952,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6798,"num_write_kb":247937,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6,0],"avail_no_missing":["4","0","6"],"object_location_counts":[{"shards":"0,4,6","objects":952}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"242'19","reported_seq":406,"reported_epoch":366,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:19:10.291926+0000","last_change":"2024-09-17T07:16:17.382006+0000","last_active":"2024-09-17T07:19:10.291926+0000","last_peered":"2024-09-17T07:19:10.291926+0000","last_clean":"2024-09-17T07:16:15.358100+0000","last_became_active":"2024-09-17T07:16:17.382006+0000","last_became_peered":"2024-09-17T07:16:17.382006+0000","last_unstale":"2024-09-17T07:19:10.291926+0000","last_undegraded":"2024-09-17T07:16:17.365274+0000","last_fullsized":"2024-09-17T07:16:17.365168+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'19","last_scrub_stamp":"2024-09-17T07:14:05.729535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:14:05.729535+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.008660559,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"94'2","reported_seq":119,"reported_epoch":367,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:19:12.449857+0000","last_change":"2024-09-17T07:16:17.374418+0000","last_active":"2024-09-17T07:19:12.449857+0000","last_peered":"2024-09-17T07:19:12.449857+0000","last_clean":"2024-09-17T07:04:06.025673+0000","last_became_active":"2024-09-17T07:16:17.374418+0000","last_became_peered":"2024-09-17T07:16:17.374418+0000","last_unstale":"2024-09-17T07:19:12.449857+0000","last_undegraded":"2024-09-17T07:16:17.367359+0000","last_fullsized":"2024-09-17T07:16:17.367242+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-09-17T07:15:48.907821+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-09-17T07:07:31.717888+0000","last_clean_scrub_stamp":"2024-09-17T07:15:48.907821+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[{"shards":"0,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":242,"reported_epoch":367,"state":"active+undersized","last_fresh":"2024-09-17T07:19:12.447409+0000","last_change":"2024-09-17T07:16:17.378918+0000","last_active":"2024-09-17T07:19:12.447409+0000","last_peered":"2024-09-17T07:19:12.447409+0000","last_clean":"2024-09-17T07:09:22.372509+0000","last_became_active":"2024-09-17T07:16:17.378918+0000","last_became_peered":"2024-09-17T07:16:17.378918+0000","last_unstale":"2024-09-17T07:19:12.447409+0000","last_undegraded":"2024-09-17T07:19:12.447409+0000","last_fullsized":"2024-09-17T07:16:17.367596+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:41.858093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:15:41.858093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037672679999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":233,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-17T07:19:13.457836+0000","last_change":"2024-09-17T07:18:23.515698+0000","last_active":"2024-09-17T07:19:13.457836+0000","last_peered":"2024-09-17T07:19:13.457836+0000","last_clean":"2024-09-17T07:19:13.457836+0000","last_became_active":"2024-09-17T07:09:24.400791+0000","last_became_peered":"2024-09-17T07:09:24.400791+0000","last_unstale":"2024-09-17T07:19:13.457836+0000","last_undegraded":"2024-09-17T07:19:13.457836+0000","last_fullsized":"2024-09-17T07:19:13.457836+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:18:23.515582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:49.573666+0000","last_clean_scrub_stamp":"2024-09-17T07:18:23.515582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:19:29.875635+0000","scrub_duration":0.0022931869999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1e","version":"367'16178","reported_seq":18484,"reported_epoch":367,"state":"active+remapped+backfilling","last_fresh":"2024-09-17T07:19:13.038170+0000","last_change":"2024-09-17T07:19:08.418045+0000","last_active":"2024-09-17T07:19:13.038170+0000","last_peered":"2024-09-17T07:19:13.038170+0000","last_clean":"2024-09-17T07:19:06.265881+0000","last_became_active":"2024-09-17T07:19:08.385075+0000","last_became_peered":"2024-09-17T07:19:08.385075+0000","last_unstale":"2024-09-17T07:19:13.038170+0000","last_undegraded":"2024-09-17T07:19:13.038170+0000","last_fullsized":"2024-09-17T07:19:13.038170+0000","mapping_epoch":364,"log_start":"338'11691","ondisk_log_start":"338'11691","created":360,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":5,"last_scrub":"346'12995","last_scrub_stamp":"2024-09-17T07:18:26.134704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:18:26.134704+0000","objects_scrubbed":5398,"log_size":4487,"log_dups_size":0,"ondisk_log_size":4487,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1.955922511,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60719104,"num_objects":927,"num_object_clones":0,"num_object_copies":2781,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":800,"num_objects_unfound":0,"num_objects_dirty":927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6728,"num_write_kb":244928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":592,"num_bytes_recovered":38797312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[2,4,6],"avail_no_missing":["2","4","6"],"object_location_counts":[{"shards":"2,4,6","objects":927}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.10","version":"242'8","reported_seq":304,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-17T07:19:10.272897+0000","last_change":"2024-09-17T07:18:23.038181+0000","last_active":"2024-09-17T07:19:10.272897+0000","last_peered":"2024-09-17T07:19:10.272897+0000","last_clean":"2024-09-17T07:19:10.272897+0000","last_became_active":"2024-09-17T07:09:24.396368+0000","last_became_peered":"2024-09-17T07:09:24.396368+0000","last_unstale":"2024-09-17T07:19:10.272897+0000","last_undegraded":"2024-09-17T07:19:10.272897+0000","last_fullsized":"2024-09-17T07:19:10.272897+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'8","last_scrub_stamp":"2024-09-17T07:18:23.038036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:18:23.038036+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:19:48.354894+0000","scrub_duration":0.021850082999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":392,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-17T07:19:12.450849+0000","last_change":"2024-09-17T07:16:37.093569+0000","last_active":"2024-09-17T07:19:12.450849+0000","last_peered":"2024-09-17T07:19:12.450849+0000","last_clean":"2024-09-17T07:19:12.450849+0000","last_became_active":"2024-09-17T07:02:12.008898+0000","last_became_peered":"2024-09-17T07:02:12.008898+0000","last_unstale":"2024-09-17T07:19:12.450849+0000","last_undegraded":"2024-09-17T07:19:12.450849+0000","last_fullsized":"2024-09-17T07:19:12.450849+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:37.093460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:16:37.093460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0048915060000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":400,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-17T07:19:12.442216+0000","last_change":"2024-09-17T07:17:34.366051+0000","last_active":"2024-09-17T07:19:12.442216+0000","last_peered":"2024-09-17T07:19:12.442216+0000","last_clean":"2024-09-17T07:19:12.442216+0000","last_became_active":"2024-09-17T07:02:16.014982+0000","last_became_peered":"2024-09-17T07:02:16.014982+0000","last_unstale":"2024-09-17T07:19:12.442216+0000","last_undegraded":"2024-09-17T07:19:12.442216+0000","last_fullsized":"2024-09-17T07:19:12.442216+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:17:34.365905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:54.553970+0000","last_clean_scrub_stamp":"2024-09-17T07:17:34.365905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0089364449999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":374,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-17T07:19:10.276922+0000","last_change":"2024-09-17T07:17:27.977910+0000","last_active":"2024-09-17T07:19:10.276922+0000","last_peered":"2024-09-17T07:19:10.276922+0000","last_clean":"2024-09-17T07:19:10.276922+0000","last_became_active":"2024-09-17T07:02:18.021014+0000","last_became_peered":"2024-09-17T07:02:18.021014+0000","last_unstale":"2024-09-17T07:19:10.276922+0000","last_undegraded":"2024-09-17T07:19:10.276922+0000","last_fullsized":"2024-09-17T07:19:10.276922+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:17:27.977803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:17:27.977803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002005036,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"367'16353","reported_seq":18249,"reported_epoch":367,"state":"active+remapped+backfilling","last_fresh":"2024-09-17T07:19:13.039209+0000","last_change":"2024-09-17T07:19:08.438531+0000","last_active":"2024-09-17T07:19:13.039209+0000","last_peered":"2024-09-17T07:19:13.039209+0000","last_clean":"2024-09-17T07:19:06.271105+0000","last_became_active":"2024-09-17T07:19:08.395358+0000","last_became_peered":"2024-09-17T07:19:08.395358+0000","last_unstale":"2024-09-17T07:19:13.039209+0000","last_undegraded":"2024-09-17T07:19:13.039209+0000","last_fullsized":"2024-09-17T07:19:13.039209+0000","mapping_epoch":364,"log_start":"338'11653","ondisk_log_start":"338'11653","created":360,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":3774,"log_size":4700,"log_dups_size":0,"ondisk_log_size":4700,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63569920,"num_objects":970,"num_object_clones":0,"num_object_copies":2910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1260,"num_objects_unfound":0,"num_objects_dirty":970,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6832,"num_write_kb":249664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":747,"num_bytes_recovered":48955392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[2,4,6],"avail_no_missing":["2","4","6"],"object_location_counts":[{"shards":"2,4,6","objects":970}],"blocked_by":[],"up_primary":7,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"242'19","reported_seq":275,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-17T07:19:10.281002+0000","last_change":"2024-09-17T07:17:33.982980+0000","last_active":"2024-09-17T07:19:10.281002+0000","last_peered":"2024-09-17T07:19:10.281002+0000","last_clean":"2024-09-17T07:19:10.281002+0000","last_became_active":"2024-09-17T07:09:24.404942+0000","last_became_peered":"2024-09-17T07:09:24.404942+0000","last_unstale":"2024-09-17T07:19:10.281002+0000","last_undegraded":"2024-09-17T07:19:10.281002+0000","last_fullsized":"2024-09-17T07:19:10.281002+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'19","last_scrub_stamp":"2024-09-17T07:17:33.982784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:17:33.982784+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0043671170000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":414,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-17T07:19:12.445073+0000","last_change":"2024-09-17T07:18:41.951465+0000","last_active":"2024-09-17T07:19:12.445073+0000","last_peered":"2024-09-17T07:19:12.445073+0000","last_clean":"2024-09-17T07:19:12.445073+0000","last_became_active":"2024-09-17T07:02:12.010195+0000","last_became_peered":"2024-09-17T07:02:12.010195+0000","last_unstale":"2024-09-17T07:19:12.445073+0000","last_undegraded":"2024-09-17T07:19:12.445073+0000","last_fullsized":"2024-09-17T07:19:12.445073+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:18:41.951382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:18:41.951382+0000","last_clean_scrub_stamp":"2024-09-17T07:18:41.951382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:20:02.248914+0000","scrub_duration":0.001516739,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":376,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-17T07:19:10.277494+0000","last_change":"2024-09-17T07:17:22.916134+0000","last_active":"2024-09-17T07:19:10.277494+0000","last_peered":"2024-09-17T07:19:10.277494+0000","last_clean":"2024-09-17T07:19:10.277494+0000","last_became_active":"2024-09-17T07:02:16.016183+0000","last_became_peered":"2024-09-17T07:02:16.016183+0000","last_unstale":"2024-09-17T07:19:10.277494+0000","last_undegraded":"2024-09-17T07:19:10.277494+0000","last_fullsized":"2024-09-17T07:19:10.277494+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:17:22.915543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:17:22.915543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0086489989999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":233,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-17T07:19:13.457526+0000","last_change":"2024-09-17T07:17:34.551610+0000","last_active":"2024-09-17T07:19:13.457526+0000","last_peered":"2024-09-17T07:19:13.457526+0000","last_clean":"2024-09-17T07:19:13.457526+0000","last_became_active":"2024-09-17T07:09:24.392216+0000","last_became_peered":"2024-09-17T07:09:24.392216+0000","last_unstale":"2024-09-17T07:19:13.457526+0000","last_undegraded":"2024-09-17T07:19:13.457526+0000","last_fullsized":"2024-09-17T07:19:13.457526+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:17:34.551498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:17:34.551498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002777912,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.10","version":"367'10712","reported_seq":17561,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-17T07:19:13.219116+0000","last_change":"2024-09-17T07:18:17.281071+0000","last_active":"2024-09-17T07:19:13.219116+0000","last_peered":"2024-09-17T07:19:13.219116+0000","last_clean":"2024-09-17T07:19:13.219116+0000","last_became_active":"2024-09-17T07:18:17.272736+0000","last_became_peered":"2024-09-17T07:18:17.272736+0000","last_unstale":"2024-09-17T07:19:13.219116+0000","last_undegraded":"2024-09-17T07:19:13.219116+0000","last_fullsized":"2024-09-17T07:19:13.219116+0000","mapping_epoch":342,"log_start":"320'6139","ondisk_log_start":"320'6139","created":332,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":4573,"log_dups_size":0,"ondisk_log_size":4573,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59179008,"num_objects":903,"num_object_clones":0,"num_object_copies":2709,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6815,"num_write_kb":246976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3368,"num_bytes_recovered":220659712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"326'2","reported_seq":375,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-17T07:19:10.277401+0000","last_change":"2024-09-17T07:17:36.974642+0000","last_active":"2024-09-17T07:19:10.277401+0000","last_peered":"2024-09-17T07:19:10.277401+0000","last_clean":"2024-09-17T07:19:10.277401+0000","last_became_active":"2024-09-17T07:02:18.021855+0000","last_became_peered":"2024-09-17T07:02:18.021855+0000","last_unstale":"2024-09-17T07:19:10.277401+0000","last_undegraded":"2024-09-17T07:19:10.277401+0000","last_fullsized":"2024-09-17T07:19:10.277401+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'2","last_scrub_stamp":"2024-09-17T07:17:36.974358+0000","last_deep_scrub":"326'2","last_deep_scrub_stamp":"2024-09-17T07:17:36.974358+0000","last_clean_scrub_stamp":"2024-09-17T07:17:36.974358+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.015149954,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":393,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-17T07:19:13.459822+0000","last_change":"2024-09-17T07:16:39.053518+0000","last_active":"2024-09-17T07:19:13.459822+0000","last_peered":"2024-09-17T07:19:13.459822+0000","last_clean":"2024-09-17T07:19:13.459822+0000","last_became_active":"2024-09-17T07:02:12.005329+0000","last_became_peered":"2024-09-17T07:02:12.005329+0000","last_unstale":"2024-09-17T07:19:13.459822+0000","last_undegraded":"2024-09-17T07:19:13.459822+0000","last_fullsized":"2024-09-17T07:19:13.459822+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:39.053386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:16:39.053386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0044426509999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"294'25","reported_seq":470,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-17T07:19:13.459841+0000","last_change":"2024-09-17T07:17:43.574192+0000","last_active":"2024-09-17T07:19:13.459841+0000","last_peered":"2024-09-17T07:19:13.459841+0000","last_clean":"2024-09-17T07:19:13.459841+0000","last_became_active":"2024-09-17T07:10:49.730948+0000","last_became_peered":"2024-09-17T07:10:49.730948+0000","last_unstale":"2024-09-17T07:19:13.459841+0000","last_undegraded":"2024-09-17T07:19:13.459841+0000","last_fullsized":"2024-09-17T07:19:13.459841+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'25","last_scrub_stamp":"2024-09-17T07:17:43.574077+0000","last_deep_scrub":"294'25","last_deep_scrub_stamp":"2024-09-17T07:16:00.471311+0000","last_clean_scrub_stamp":"2024-09-17T07:17:43.574077+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0071385140000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"75'8","reported_seq":616,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-17T07:19:12.444812+0000","last_change":"2024-09-17T07:17:25.451002+0000","last_active":"2024-09-17T07:19:12.444812+0000","last_peered":"2024-09-17T07:19:12.444812+0000","last_clean":"2024-09-17T07:19:12.444812+0000","last_became_active":"2024-09-17T07:02:16.020560+0000","last_became_peered":"2024-09-17T07:02:16.020560+0000","last_unstale":"2024-09-17T07:19:12.444812+0000","last_undegraded":"2024-09-17T07:19:12.444812+0000","last_fullsized":"2024-09-17T07:19:12.444812+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:17:25.450919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:17:25.450919+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013309369999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.11","version":"368'10932","reported_seq":18136,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-17T07:19:13.947196+0000","last_change":"2024-09-17T07:18:16.281801+0000","last_active":"2024-09-17T07:19:13.947196+0000","last_peered":"2024-09-17T07:19:13.947196+0000","last_clean":"2024-09-17T07:19:13.947196+0000","last_became_active":"2024-09-17T07:18:16.270511+0000","last_became_peered":"2024-09-17T07:18:16.270511+0000","last_unstale":"2024-09-17T07:19:13.947196+0000","last_undegraded":"2024-09-17T07:19:13.947196+0000","last_fullsized":"2024-09-17T07:19:13.947196+0000","mapping_epoch":341,"log_start":"316'5307","ondisk_log_start":"316'5307","created":332,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":5,"last_scrub":"320'6219","last_scrub_stamp":"2024-09-17T07:17:22.401249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:17:22.401249+0000","objects_scrubbed":6144,"log_size":5625,"log_dups_size":0,"ondisk_log_size":5625,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2.0885702689999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60358656,"num_objects":921,"num_object_clones":0,"num_object_copies":2763,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":921,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6955,"num_write_kb":252032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3482,"num_bytes_recovered":228196352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":74,"reported_epoch":367,"state":"active+undersized","last_fresh":"2024-09-17T07:19:12.449478+0000","last_change":"2024-09-17T07:16:17.375358+0000","last_active":"2024-09-17T07:19:12.449478+0000","last_peered":"2024-09-17T07:19:12.449478+0000","last_clean":"2024-09-17T07:02:17.002904+0000","last_became_active":"2024-09-17T07:16:17.375358+0000","last_became_peered":"2024-09-17T07:16:17.375358+0000","last_unstale":"2024-09-17T07:19:12.449478+0000","last_undegraded":"2024-09-17T07:19:12.449478+0000","last_fullsized":"2024-09-17T07:16:17.368647+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:37.885570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:15:37.885570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":243,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-17T07:19:12.442002+0000","last_change":"2024-09-17T07:17:16.329912+0000","last_active":"2024-09-17T07:19:12.442002+0000","last_peered":"2024-09-17T07:19:12.442002+0000","last_clean":"2024-09-17T07:19:12.442002+0000","last_became_active":"2024-09-17T07:09:24.403360+0000","last_became_peered":"2024-09-17T07:09:24.403360+0000","last_unstale":"2024-09-17T07:19:12.442002+0000","last_undegraded":"2024-09-17T07:19:12.442002+0000","last_fullsized":"2024-09-17T07:19:12.442002+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:17:16.329797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:10:29.325664+0000","last_clean_scrub_stamp":"2024-09-17T07:17:16.329797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033990230000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"242'19","reported_seq":475,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-17T07:19:12.444892+0000","last_change":"2024-09-17T07:17:17.376514+0000","last_active":"2024-09-17T07:19:12.444892+0000","last_peered":"2024-09-17T07:19:12.444892+0000","last_clean":"2024-09-17T07:19:12.444892+0000","last_became_active":"2024-09-17T07:03:48.760390+0000","last_became_peered":"2024-09-17T07:03:48.760390+0000","last_unstale":"2024-09-17T07:19:12.444892+0000","last_undegraded":"2024-09-17T07:19:12.444892+0000","last_fullsized":"2024-09-17T07:19:12.444892+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'19","last_scrub_stamp":"2024-09-17T07:17:17.375166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:17:17.375166+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.014344902,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":380,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-17T07:19:13.457998+0000","last_change":"2024-09-17T07:17:31.538898+0000","last_active":"2024-09-17T07:19:13.457998+0000","last_peered":"2024-09-17T07:19:13.457998+0000","last_clean":"2024-09-17T07:19:13.457998+0000","last_became_active":"2024-09-17T07:02:16.019701+0000","last_became_peered":"2024-09-17T07:02:16.019701+0000","last_unstale":"2024-09-17T07:19:13.457998+0000","last_undegraded":"2024-09-17T07:19:13.457998+0000","last_fullsized":"2024-09-17T07:19:13.457998+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:17:31.538812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:17:31.538812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014632390000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.12","version":"368'10590","reported_seq":15942,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-17T07:19:13.754014+0000","last_change":"2024-09-17T07:19:08.356745+0000","last_active":"2024-09-17T07:19:13.754014+0000","last_peered":"2024-09-17T07:19:13.754014+0000","last_clean":"2024-09-17T07:19:13.754014+0000","last_became_active":"2024-09-17T07:19:08.312102+0000","last_became_peered":"2024-09-17T07:19:08.312102+0000","last_unstale":"2024-09-17T07:19:13.754014+0000","last_undegraded":"2024-09-17T07:19:13.754014+0000","last_fullsized":"2024-09-17T07:19:13.754014+0000","mapping_epoch":364,"log_start":"317'5436","ondisk_log_start":"317'5436","created":332,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":0,"log_size":5154,"log_dups_size":0,"ondisk_log_size":5154,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61636608,"num_objects":941,"num_object_clones":0,"num_object_copies":2823,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6722,"num_write_kb":245184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1312,"num_bytes_recovered":85917696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":388,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-17T07:19:10.294009+0000","last_change":"2024-09-17T07:17:37.138113+0000","last_active":"2024-09-17T07:19:10.294009+0000","last_peered":"2024-09-17T07:19:10.294009+0000","last_clean":"2024-09-17T07:19:10.294009+0000","last_became_active":"2024-09-17T07:02:18.363752+0000","last_became_peered":"2024-09-17T07:02:18.363752+0000","last_unstale":"2024-09-17T07:19:10.294009+0000","last_undegraded":"2024-09-17T07:19:10.294009+0000","last_fullsized":"2024-09-17T07:19:10.294009+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:17:37.137989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:17:37.137989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012195245,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":433,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-17T07:19:12.443027+0000","last_change":"2024-09-17T07:18:40.285093+0000","last_active":"2024-09-17T07:19:12.443027+0000","last_peered":"2024-09-17T07:19:12.443027+0000","last_clean":"2024-09-17T07:19:12.443027+0000","last_became_active":"2024-09-17T07:03:48.759384+0000","last_became_peered":"2024-09-17T07:03:48.759384+0000","last_unstale":"2024-09-17T07:19:12.443027+0000","last_undegraded":"2024-09-17T07:19:12.443027+0000","last_fullsized":"2024-09-17T07:19:12.443027+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:44.558901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:11:18.559877+0000","last_clean_scrub_stamp":"2024-09-17T07:16:44.558901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.031473045999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"242'27","reported_seq":345,"reported_epoch":368,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:19:13.458006+0000","last_change":"2024-09-17T07:16:17.368008+0000","last_active":"2024-09-17T07:19:13.458006+0000","last_peered":"2024-09-17T07:19:13.458006+0000","last_clean":"2024-09-17T07:12:20.337498+0000","last_became_active":"2024-09-17T07:16:17.368008+0000","last_became_peered":"2024-09-17T07:16:17.368008+0000","last_unstale":"2024-09-17T07:19:13.458006+0000","last_undegraded":"2024-09-17T07:16:17.365648+0000","last_fullsized":"2024-09-17T07:16:17.365550+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'27","last_scrub_stamp":"2024-09-17T07:15:45.855430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:15:45.855430+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.017595989999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":7}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":73,"reported_epoch":366,"state":"active+undersized","last_fresh":"2024-09-17T07:19:10.292718+0000","last_change":"2024-09-17T07:16:17.371494+0000","last_active":"2024-09-17T07:19:10.292718+0000","last_peered":"2024-09-17T07:19:10.292718+0000","last_clean":"2024-09-17T07:02:14.999435+0000","last_became_active":"2024-09-17T07:16:17.371494+0000","last_became_peered":"2024-09-17T07:16:17.371494+0000","last_unstale":"2024-09-17T07:19:10.292718+0000","last_undegraded":"2024-09-17T07:19:10.292718+0000","last_fullsized":"2024-09-17T07:16:17.366463+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:00.509283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:25.794418+0000","last_clean_scrub_stamp":"2024-09-17T07:14:00.509283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.13","version":"367'13573","reported_seq":20750,"reported_epoch":367,"state":"active+clean+remapped","last_fresh":"2024-09-17T07:19:13.027538+0000","last_change":"2024-09-17T07:18:13.461884+0000","last_active":"2024-09-17T07:19:13.027538+0000","last_peered":"2024-09-17T07:19:13.027538+0000","last_clean":"2024-09-17T07:19:13.027538+0000","last_became_active":"2024-09-17T07:18:13.450785+0000","last_became_peered":"2024-09-17T07:18:13.450785+0000","last_unstale":"2024-09-17T07:19:13.027538+0000","last_undegraded":"2024-09-17T07:19:13.027538+0000","last_fullsized":"2024-09-17T07:19:13.027538+0000","mapping_epoch":339,"log_start":"322'6348","ondisk_log_start":"322'6348","created":332,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:04.025356+0000","last_clean_scrub_stamp":"2024-09-17T07:16:04.025356+0000","objects_scrubbed":4348,"log_size":7225,"log_dups_size":0,"ondisk_log_size":7225,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61046784,"num_objects":932,"num_object_clones":0,"num_object_copies":2796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":932,"num_objects_unfound":0,"num_objects_dirty":932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6638,"num_write_kb":242208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3360,"num_bytes_recovered":218824704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2,4],"avail_no_missing":["7","2","4"],"object_location_counts":[{"shards":"2,4,7","objects":932}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":239,"reported_epoch":367,"state":"active+undersized","last_fresh":"2024-09-17T07:19:12.443889+0000","last_change":"2024-09-17T07:16:17.377951+0000","last_active":"2024-09-17T07:19:12.443889+0000","last_peered":"2024-09-17T07:19:12.443889+0000","last_clean":"2024-09-17T07:16:14.246741+0000","last_became_active":"2024-09-17T07:16:17.377951+0000","last_became_peered":"2024-09-17T07:16:17.377951+0000","last_unstale":"2024-09-17T07:19:12.443889+0000","last_undegraded":"2024-09-17T07:19:12.443889+0000","last_fullsized":"2024-09-17T07:16:17.364654+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:02.473467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:15:02.473467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001733163,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":221,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-17T07:19:10.279046+0000","last_change":"2024-09-17T07:16:31.651379+0000","last_active":"2024-09-17T07:19:10.279046+0000","last_peered":"2024-09-17T07:19:10.279046+0000","last_clean":"2024-09-17T07:19:10.279046+0000","last_became_active":"2024-09-17T07:09:24.404795+0000","last_became_peered":"2024-09-17T07:09:24.404795+0000","last_unstale":"2024-09-17T07:19:10.279046+0000","last_undegraded":"2024-09-17T07:19:10.279046+0000","last_fullsized":"2024-09-17T07:19:10.279046+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:16:31.651247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:16:31.651247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0075767350000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"242'23","reported_seq":205,"reported_epoch":366,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:19:10.292833+0000","last_change":"2024-09-17T07:16:17.380972+0000","last_active":"2024-09-17T07:19:10.292833+0000","last_peered":"2024-09-17T07:19:10.292833+0000","last_clean":"2024-09-17T07:12:20.742449+0000","last_became_active":"2024-09-17T07:16:17.380972+0000","last_became_peered":"2024-09-17T07:16:17.380972+0000","last_unstale":"2024-09-17T07:19:10.292833+0000","last_undegraded":"2024-09-17T07:16:17.366757+0000","last_fullsized":"2024-09-17T07:16:17.366645+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'23","last_scrub_stamp":"2024-09-17T07:14:58.689120+0000","last_deep_scrub":"86'17","last_deep_scrub_stamp":"2024-09-17T07:03:18.241067+0000","last_clean_scrub_stamp":"2024-09-17T07:14:58.689120+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":397,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-17T07:19:10.298441+0000","last_change":"2024-09-17T07:17:17.309054+0000","last_active":"2024-09-17T07:19:10.298441+0000","last_peered":"2024-09-17T07:19:10.298441+0000","last_clean":"2024-09-17T07:19:10.298441+0000","last_became_active":"2024-09-17T07:09:24.393329+0000","last_became_peered":"2024-09-17T07:09:24.393329+0000","last_unstale":"2024-09-17T07:19:10.298441+0000","last_undegraded":"2024-09-17T07:19:10.298441+0000","last_fullsized":"2024-09-17T07:19:10.298441+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:17:17.308762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:17:17.308762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.023046534,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":36,"stat_sum":{"num_bytes":1987084312,"num_objects":30323,"num_object_clones":0,"num_object_copies":90969,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3489,"num_objects_misplaced":13732,"num_objects_unfound":0,"num_objects_dirty":30323,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":175131,"num_write_kb":6176865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26235,"num_bytes_recovered":1717567488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6060978176,"data_stored":6060965960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":185191,"ondisk_log_size":185191,"up":92,"acting":102,"num_store_stats":7},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":83,"acting":83,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73,"ondisk_log_size":73,"up":81,"acting":81,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":101,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1113,"num_read_kb":870,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":109,"num_bytes_recovered":2228,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":13004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":733,"ondisk_log_size":733,"up":78,"acting":78,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":182,"num_read_kb":177,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":8,"acting":8,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1386,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":12,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":4578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":83,"acting":83,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037970,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1409896,"kb_used_data":648668,"kb_used_omap":1,"kb_used_meta":761214,"kb_avail":92335256,"statfs":{"total":95995035648,"available":94551302144,"internally_reserved":0,"allocated":664236032,"data_stored":663490878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":779483595},"hb_peers":[0,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:18:32 2024","interfaces":[{"interface":"back","average":{"1min":3.0750000000000002,"5min":1.3919999999999999,"15min":1.772},"min":{"1min":0.20699999999999999,"5min":0.19700000000000001,"15min":0.192},"max":{"1min":24.739999999999998,"5min":24.739999999999998,"15min":52.478999999999999},"last":0.433},{"interface":"front","average":{"1min":2.6030000000000002,"5min":1.3859999999999999,"15min":1.8680000000000001},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":24.768000000000001,"5min":24.768000000000001,"15min":52.871000000000002},"last":0.27100000000000002}]},{"osd":1,"last update":"Tue Sep 17 07:15:24 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":1.4510000000000001,"15min":1.3859999999999999},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.223},"max":{"1min":1.573,"5min":15.801,"15min":73.549000000000007},"last":0.65300000000000002},{"interface":"front","average":{"1min":1.3779999999999999,"5min":1.2969999999999999,"15min":1.383},"min":{"1min":0.29699999999999999,"5min":0.26200000000000001,"15min":0.24099999999999999},"max":{"1min":4.7359999999999998,"5min":13.989000000000001,"15min":73.116},"last":0.71499999999999997}]},{"osd":2,"last update":"Tue Sep 17 07:18:32 2024","interfaces":[{"interface":"back","average":{"1min":1.8440000000000001,"5min":1.2390000000000001,"15min":1.274},"min":{"1min":0.28599999999999998,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":10.057,"5min":10.057,"15min":52.055999999999997},"last":0.23599999999999999},{"interface":"front","average":{"1min":1.216,"5min":1.131,"15min":1.2170000000000001},"min":{"1min":0.247,"5min":0.224,"15min":0.224},"max":{"1min":9.843,"5min":9.843,"15min":52.064999999999998},"last":5.1870000000000003}]},{"osd":3,"last update":"Tue Sep 17 07:18:32 2024","interfaces":[{"interface":"back","average":{"1min":2.02,"5min":1.292,"15min":1.4750000000000001},"min":{"1min":0.316,"5min":0.29199999999999998,"15min":0.28799999999999998},"max":{"1min":10.122999999999999,"5min":10.122999999999999,"15min":51.689},"last":0.32300000000000001},{"interface":"front","average":{"1min":1.4830000000000001,"5min":1.2270000000000001,"15min":1.544},"min":{"1min":0.33800000000000002,"5min":0.27000000000000002,"15min":0.26500000000000001},"max":{"1min":12.465999999999999,"5min":12.465999999999999,"15min":52.145000000000003},"last":0.49099999999999999}]},{"osd":4,"last update":"Tue Sep 17 07:18:32 2024","interfaces":[{"interface":"back","average":{"1min":1.7649999999999999,"5min":1.5960000000000001,"15min":0.97999999999999998},"min":{"1min":0.27300000000000002,"5min":0.222,"15min":0.222},"max":{"1min":12.679,"5min":21.140999999999998,"15min":21.140999999999998},"last":0.65800000000000003},{"interface":"front","average":{"1min":2.4209999999999998,"5min":1.8620000000000001,"15min":1.1359999999999999},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":11.984999999999999,"5min":19.420000000000002,"15min":19.420000000000002},"last":0.90400000000000003}]},{"osd":5,"last update":"Tue Sep 17 07:18:32 2024","interfaces":[{"interface":"back","average":{"1min":1.8300000000000001,"5min":1.6719999999999999,"15min":1.173},"min":{"1min":0.315,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":12.429,"5min":19.384,"15min":29.428999999999998},"last":0.46600000000000003},{"interface":"front","average":{"1min":1.6399999999999999,"5min":1.7689999999999999,"15min":1.214},"min":{"1min":0.314,"5min":0.27500000000000002,"15min":0.25700000000000001},"max":{"1min":12.606,"5min":19.483000000000001,"15min":30.059999999999999},"last":0.35099999999999998}]},{"osd":6,"last update":"Tue Sep 17 07:18:32 2024","interfaces":[{"interface":"back","average":{"1min":2.2120000000000002,"5min":1.9299999999999999,"15min":1.2509999999999999},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":12.641,"5min":22.221,"15min":25.363},"last":0.29499999999999998},{"interface":"front","average":{"1min":2.5529999999999999,"5min":2.1429999999999998,"15min":1.3320000000000001},"min":{"1min":0.30499999999999999,"5min":0.29099999999999998,"15min":0.27500000000000002},"max":{"1min":10.499000000000001,"5min":22.202000000000002,"15min":23.571000000000002},"last":0.86799999999999999}]}]},{"osd":6,"up_from":50,"seq":214748365013,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1747912,"kb_used_data":1103100,"kb_used_omap":1,"kb_used_meta":644734,"kb_avail":91997240,"statfs":{"total":95995035648,"available":94205173760,"internally_reserved":0,"allocated":1129574400,"data_stored":1128797119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":660208071},"hb_peers":[0,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,2,1,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:18:15 2024","interfaces":[{"interface":"back","average":{"1min":1.6419999999999999,"5min":2.0899999999999999,"15min":2.121},"min":{"1min":0.218,"5min":0.20200000000000001,"15min":0.185},"max":{"1min":5.2229999999999999,"5min":126.902,"15min":126.902},"last":1.518},{"interface":"front","average":{"1min":0.876,"5min":1.7909999999999999,"15min":1.99},"min":{"1min":0.23200000000000001,"5min":0.20899999999999999,"15min":0.20799999999999999},"max":{"1min":2.7919999999999998,"5min":127.06399999999999,"15min":127.06399999999999},"last":0.34799999999999998}]},{"osd":1,"last update":"Tue Sep 17 07:16:10 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.996,"15min":0.84399999999999997},"min":{"1min":0.23799999999999999,"5min":0.22900000000000001,"15min":0.22600000000000001},"max":{"1min":2.2999999999999998,"5min":14.720000000000001,"15min":14.720000000000001},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.622,"5min":1.0109999999999999,"15min":0.88800000000000001},"min":{"1min":0.29099999999999998,"5min":0.23400000000000001,"15min":0.22900000000000001},"max":{"1min":2.323,"5min":14.699,"15min":14.699},"last":0.46600000000000003}]},{"osd":2,"last update":"Tue Sep 17 07:18:15 2024","interfaces":[{"interface":"back","average":{"1min":1.742,"5min":1.0449999999999999,"15min":0.98599999999999999},"min":{"1min":0.251,"5min":0.219,"15min":0.219},"max":{"1min":5.266,"5min":5.266,"15min":33.079000000000001},"last":1.4530000000000001},{"interface":"front","average":{"1min":1.0369999999999999,"5min":0.81499999999999995,"15min":0.90400000000000003},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":2.8380000000000001,"5min":4.8399999999999999,"15min":33.100000000000001},"last":0.375}]},{"osd":3,"last update":"Tue Sep 17 07:18:15 2024","interfaces":[{"interface":"back","average":{"1min":1.643,"5min":0.879,"15min":1.2609999999999999},"min":{"1min":0.27500000000000002,"5min":0.27000000000000002,"15min":0.23200000000000001},"max":{"1min":4.9320000000000004,"5min":4.9320000000000004,"15min":52.944000000000003},"last":0.45000000000000001},{"interface":"front","average":{"1min":1.7130000000000001,"5min":1.22,"15min":1.3700000000000001},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":5.2949999999999999,"5min":5.7690000000000001,"15min":52.747},"last":1.4890000000000001}]},{"osd":4,"last update":"Tue Sep 17 07:18:15 2024","interfaces":[{"interface":"back","average":{"1min":2.0070000000000001,"5min":1.4330000000000001,"15min":2.129},"min":{"1min":0.27900000000000003,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":5.4800000000000004,"5min":7.1050000000000004,"15min":260.77199999999999},"last":0.496},{"interface":"front","average":{"1min":3.2509999999999999,"5min":1.51,"15min":2.1819999999999999},"min":{"1min":0.36299999999999999,"5min":0.23300000000000001,"15min":0.19800000000000001},"max":{"1min":8.7929999999999993,"5min":8.7929999999999993,"15min":260.69099999999997},"last":2.8940000000000001}]},{"osd":5,"last update":"Tue Sep 17 07:18:15 2024","interfaces":[{"interface":"back","average":{"1min":2.0649999999999999,"5min":1.8959999999999999,"15min":1.2230000000000001},"min":{"1min":0.32900000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":10.201000000000001,"5min":11.488,"15min":32.042000000000002},"last":2.7919999999999998},{"interface":"front","average":{"1min":2.0219999999999998,"5min":1.734,"15min":1.208},"min":{"1min":0.44,"5min":0.27200000000000002,"15min":0.251},"max":{"1min":5.5540000000000003,"5min":10.244999999999999,"15min":31.852},"last":1.764}]},{"osd":7,"last update":"Tue Sep 17 07:18:38 2024","interfaces":[{"interface":"back","average":{"1min":2.2999999999999998,"5min":2.972,"15min":1.5669999999999999},"min":{"1min":0.41199999999999998,"5min":0.246,"15min":0.23699999999999999},"max":{"1min":6.8179999999999996,"5min":131.95699999999999,"15min":131.95699999999999},"last":2.294},{"interface":"front","average":{"1min":1.3340000000000001,"5min":2.7109999999999999,"15min":1.4670000000000001},"min":{"1min":0.34499999999999997,"5min":0.252,"15min":0.252},"max":{"1min":5.8070000000000004,"5min":131.833,"15min":131.833},"last":2.4980000000000002}]}]},{"osd":5,"up_from":42,"seq":180388626650,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1520740,"kb_used_data":709468,"kb_used_omap":1,"kb_used_meta":811262,"kb_avail":92224412,"statfs":{"total":95995035648,"available":94437797888,"internally_reserved":0,"allocated":726495232,"data_stored":725749866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":830732742},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:18:58 2024","interfaces":[{"interface":"back","average":{"1min":1.139,"5min":0.90000000000000002,"15min":1.661},"min":{"1min":0.25,"5min":0.184,"15min":0.184},"max":{"1min":5.7530000000000001,"5min":5.7530000000000001,"15min":49.265000000000001},"last":0.312},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.79400000000000004,"15min":1.7330000000000001},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":4.1890000000000001,"5min":4.7939999999999996,"15min":47.350000000000001},"last":0.29499999999999998}]},{"osd":1,"last update":"Tue Sep 17 07:15:47 2024","interfaces":[{"interface":"back","average":{"1min":1.4319999999999999,"5min":1.0509999999999999,"15min":0.90100000000000002},"min":{"1min":0.30499999999999999,"5min":0.22,"15min":0.22},"max":{"1min":13.711,"5min":13.711,"15min":20.329000000000001},"last":0.437},{"interface":"front","average":{"1min":0.99399999999999999,"5min":1.3320000000000001,"15min":1.149},"min":{"1min":0.32000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4.617,"5min":6.7560000000000002,"15min":20.408999999999999},"last":0.42199999999999999}]},{"osd":2,"last update":"Tue Sep 17 07:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.93000000000000005,"5min":1,"15min":1.175},"min":{"1min":0.24099999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":2.6309999999999998,"5min":6.3419999999999996,"15min":20.486000000000001},"last":0.36799999999999999},{"interface":"front","average":{"1min":1.286,"5min":0.97199999999999998,"15min":1.0920000000000001},"min":{"1min":0.317,"5min":0.219,"15min":0.219},"max":{"1min":5.8109999999999999,"5min":7.71,"15min":23.925999999999998},"last":0.61099999999999999}]},{"osd":3,"last update":"Tue Sep 17 07:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.96099999999999997,"5min":1.2270000000000001,"15min":1.5129999999999999},"min":{"1min":0.27500000000000002,"5min":0.27100000000000002,"15min":0.247},"max":{"1min":2.6640000000000001,"5min":13.882,"15min":68.978999999999999},"last":0.45200000000000001},{"interface":"front","average":{"1min":1.1799999999999999,"5min":0.85999999999999999,"15min":1.3600000000000001},"min":{"1min":0.35599999999999998,"5min":0.27100000000000002,"15min":0.253},"max":{"1min":5.7880000000000003,"5min":5.7880000000000003,"15min":69.084000000000003},"last":0.71199999999999997}]},{"osd":4,"last update":"Tue Sep 17 07:18:58 2024","interfaces":[{"interface":"back","average":{"1min":5.8760000000000003,"5min":2.6120000000000001,"15min":2.0710000000000002},"min":{"1min":0.26400000000000001,"5min":0.23200000000000001,"15min":0.22},"max":{"1min":96.668000000000006,"5min":96.668000000000006,"15min":142.327},"last":3.0409999999999999},{"interface":"front","average":{"1min":5.9889999999999999,"5min":2.5459999999999998,"15min":1.958},"min":{"1min":0.255,"5min":0.25,"15min":0.20999999999999999},"max":{"1min":96.801000000000002,"5min":96.801000000000002,"15min":142.13200000000001},"last":0.39500000000000002}]},{"osd":6,"last update":"Tue Sep 17 07:19:10 2024","interfaces":[{"interface":"back","average":{"1min":6.2300000000000004,"5min":2.0880000000000001,"15min":1.2849999999999999},"min":{"1min":0.28399999999999997,"5min":0.245,"15min":0.23499999999999999},"max":{"1min":96.686000000000007,"5min":96.686000000000007,"15min":96.686000000000007},"last":0.58299999999999996},{"interface":"front","average":{"1min":6.7359999999999998,"5min":2.5449999999999999,"15min":1.4379999999999999},"min":{"1min":0.30399999999999999,"5min":0.26700000000000002,"15min":0.24099999999999999},"max":{"1min":96.834999999999994,"5min":96.834999999999994,"15min":96.834999999999994},"last":1.417}]},{"osd":7,"last update":"Tue Sep 17 07:18:31 2024","interfaces":[{"interface":"back","average":{"1min":1.6399999999999999,"5min":1.573,"15min":1.157},"min":{"1min":0.35499999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":3.5960000000000001,"5min":6.8869999999999996,"15min":6.8869999999999996},"last":0.41799999999999998},{"interface":"front","average":{"1min":1.7609999999999999,"5min":1.391,"15min":1.01},"min":{"1min":0.32500000000000001,"5min":0.28100000000000003,"15min":0.27100000000000002},"max":{"1min":6.3639999999999999,"5min":6.3639999999999999,"15min":6.3639999999999999},"last":1.1180000000000001}]}]},{"osd":4,"up_from":33,"seq":141733920992,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1957364,"kb_used_data":1186212,"kb_used_omap":1,"kb_used_meta":771070,"kb_avail":91787788,"statfs":{"total":95995035648,"available":93990694912,"internally_reserved":0,"allocated":1214681088,"data_stored":1213926654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":789576134},"hb_peers":[0,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:18:20 2024","interfaces":[{"interface":"back","average":{"1min":1.51,"5min":1.135,"15min":2.1230000000000002},"min":{"1min":0.26700000000000002,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":9.2750000000000004,"5min":9.2750000000000004,"15min":74.551000000000002},"last":1.9590000000000001},{"interface":"front","average":{"1min":1.6839999999999999,"5min":1.345,"15min":2.3210000000000002},"min":{"1min":0.26800000000000002,"5min":0.22700000000000001,"15min":0.20699999999999999},"max":{"1min":9.9969999999999999,"5min":16.619,"15min":74.778999999999996},"last":0.45000000000000001}]},{"osd":1,"last update":"Tue Sep 17 07:16:11 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":1.2070000000000001,"15min":1.4850000000000001},"min":{"1min":0.252,"5min":0.24399999999999999,"15min":0.23000000000000001},"max":{"1min":1.736,"5min":18.661000000000001,"15min":54.914999999999999},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.71499999999999997,"5min":1.3,"15min":1.444},"min":{"1min":0.246,"5min":0.23300000000000001,"15min":0.22800000000000001},"max":{"1min":3.7770000000000001,"5min":21.931999999999999,"15min":53.725000000000001},"last":0.443}]},{"osd":2,"last update":"Tue Sep 17 07:18:20 2024","interfaces":[{"interface":"back","average":{"1min":1.5880000000000001,"5min":1.262,"15min":1.1739999999999999},"min":{"1min":0.30599999999999999,"5min":0.26700000000000002,"15min":0.23799999999999999},"max":{"1min":8.4710000000000001,"5min":8.4710000000000001,"15min":49.651000000000003},"last":1.915},{"interface":"front","average":{"1min":1.353,"5min":1.1850000000000001,"15min":1.224},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.23599999999999999},"max":{"1min":6.1840000000000002,"5min":16.655999999999999,"15min":48.813000000000002},"last":0.42099999999999999}]},{"osd":3,"last update":"Tue Sep 17 07:18:20 2024","interfaces":[{"interface":"back","average":{"1min":1.141,"5min":0.94399999999999995,"15min":1.115},"min":{"1min":0.318,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":4.0990000000000002,"5min":4.0990000000000002,"15min":39.042999999999999},"last":0.34699999999999998},{"interface":"front","average":{"1min":1.1559999999999999,"5min":1.212,"15min":1.234},"min":{"1min":0.35099999999999998,"5min":0.28999999999999998,"15min":0.253},"max":{"1min":9.0329999999999995,"5min":16.111000000000001,"15min":38.920000000000002},"last":0.501}]},{"osd":5,"last update":"Tue Sep 17 07:18:35 2024","interfaces":[{"interface":"back","average":{"1min":7.8630000000000004,"5min":2.8069999999999999,"15min":1.518},"min":{"1min":0.249,"5min":0.249,"15min":0.20799999999999999},"max":{"1min":147.61099999999999,"5min":147.61099999999999,"15min":147.61099999999999},"last":0.46999999999999997},{"interface":"front","average":{"1min":7.9269999999999996,"5min":2.5979999999999999,"15min":1.5109999999999999},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.26100000000000001},"max":{"1min":147.74700000000001,"5min":147.74700000000001,"15min":147.74700000000001},"last":1.23}]},{"osd":6,"last update":"Tue Sep 17 07:18:20 2024","interfaces":[{"interface":"back","average":{"1min":1.871,"5min":1.4430000000000001,"15min":1.1060000000000001},"min":{"1min":0.28999999999999998,"5min":0.24299999999999999,"15min":0.218},"max":{"1min":8.9619999999999997,"5min":8.9619999999999997,"15min":47.923999999999999},"last":5.3789999999999996},{"interface":"front","average":{"1min":1.55,"5min":1.5940000000000001,"15min":1.1890000000000001},"min":{"1min":0.38900000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":4.899,"5min":10.051,"15min":47.600999999999999},"last":1.9810000000000001}]},{"osd":7,"last update":"Tue Sep 17 07:18:33 2024","interfaces":[{"interface":"back","average":{"1min":1.1200000000000001,"5min":1.5349999999999999,"15min":1.2150000000000001},"min":{"1min":0.315,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":3.157,"5min":7.1550000000000002,"15min":48.719000000000001},"last":0.55700000000000005},{"interface":"front","average":{"1min":1.5229999999999999,"5min":1.7270000000000001,"15min":1.2709999999999999},"min":{"1min":0.32200000000000001,"5min":0.29199999999999998,"15min":0.28499999999999998},"max":{"1min":7.2910000000000004,"5min":9.0030000000000001,"15min":50.527000000000001},"last":5.415}]}]},{"osd":3,"up_from":0,"seq":115964117218,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":18,"seq":77309411560,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1861408,"kb_used_data":1049940,"kb_used_omap":1,"kb_used_meta":811262,"kb_avail":91883744,"statfs":{"total":95995035648,"available":94088953856,"internally_reserved":0,"allocated":1075138560,"data_stored":1074393846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":830732742},"hb_peers":[0,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":2.8279999999999998,"15min":1.913},"min":{"1min":0.154,"5min":0.154,"15min":0.154},"max":{"1min":2.2890000000000001,"5min":131.09800000000001,"15min":131.09800000000001},"last":0.221},{"interface":"front","average":{"1min":0.42099999999999999,"5min":2.7669999999999999,"15min":2.0350000000000001},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":1.0960000000000001,"5min":131.24700000000001,"15min":131.24700000000001},"last":0.87}]},{"osd":1,"last update":"Tue Sep 17 07:15:43 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":1.7949999999999999,"15min":1.623},"min":{"1min":0.23699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":1.6459999999999999,"5min":47.101999999999997,"15min":79.659000000000006},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.55000000000000004,"5min":1.6990000000000001,"15min":1.5600000000000001},"min":{"1min":0.22700000000000001,"5min":0.20899999999999999,"15min":0.20200000000000001},"max":{"1min":1.8049999999999999,"5min":45.831000000000003,"15min":80.155000000000001},"last":0.41399999999999998}]},{"osd":3,"last update":"Tue Sep 17 07:19:05 2024","interfaces":[{"interface":"back","average":{"1min":0.29399999999999998,"5min":0.47299999999999998,"15min":0.97099999999999997},"min":{"1min":0.217,"5min":0.20499999999999999,"15min":0.20100000000000001},"max":{"1min":0.442,"5min":4.6449999999999996,"15min":23.43},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.41999999999999998,"15min":1.0349999999999999},"min":{"1min":0.25800000000000001,"5min":0.22,"15min":0.22},"max":{"1min":0.89000000000000001,"5min":2.121,"15min":23.408000000000001},"last":0.93400000000000005}]},{"osd":4,"last update":"Tue Sep 17 07:18:30 2024","interfaces":[{"interface":"back","average":{"1min":1.042,"5min":1.022,"15min":0.97599999999999998},"min":{"1min":0.247,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":5.351,"5min":6.8449999999999998,"15min":6.8449999999999998},"last":0.40500000000000003},{"interface":"front","average":{"1min":1.8080000000000001,"5min":1.2889999999999999,"15min":1.0600000000000001},"min":{"1min":0.28000000000000003,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":19.119,"5min":19.119,"15min":19.119},"last":0.59099999999999997}]},{"osd":5,"last update":"Tue Sep 17 07:18:52 2024","interfaces":[{"interface":"back","average":{"1min":1.595,"5min":1.169,"15min":1.226},"min":{"1min":0.27900000000000003,"5min":0.26900000000000002,"15min":0.249},"max":{"1min":7.2229999999999999,"5min":7.2229999999999999,"15min":15.276999999999999},"last":1.143},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.82199999999999995,"15min":1.02},"min":{"1min":0.312,"5min":0.28000000000000003,"15min":0.27600000000000002},"max":{"1min":1.3400000000000001,"5min":4.577,"15min":15.561999999999999},"last":0.51200000000000001}]},{"osd":6,"last update":"Tue Sep 17 07:19:05 2024","interfaces":[{"interface":"back","average":{"1min":1.9870000000000001,"5min":1.359,"15min":1.5529999999999999},"min":{"1min":0.30399999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":23.084,"5min":23.084,"15min":101.587},"last":0.52700000000000002},{"interface":"front","average":{"1min":2.669,"5min":1.6399999999999999,"15min":1.667},"min":{"1min":0.313,"5min":0.313,"15min":0.28299999999999997},"max":{"1min":23.164999999999999,"5min":26.922000000000001,"15min":100.696},"last":1.2569999999999999}]},{"osd":7,"last update":"Tue Sep 17 07:18:48 2024","interfaces":[{"interface":"back","average":{"1min":1.1699999999999999,"5min":2.1909999999999998,"15min":1.4119999999999999},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":4.7759999999999998,"5min":94.269999999999996,"15min":94.269999999999996},"last":1.9099999999999999},{"interface":"front","average":{"1min":1.0229999999999999,"5min":2.1429999999999998,"15min":1.444},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":4.8090000000000002,"5min":94.634,"15min":94.634},"last":0.56399999999999995}]}]},{"osd":1,"up_from":13,"seq":55834575049,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":541424,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":540542,"kb_avail":93203728,"statfs":{"total":95995035648,"available":95440617472,"internally_reserved":0,"allocated":884736,"data_stored":439467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":553515462},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:16:00 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.73699999999999999,"15min":1.139},"min":{"1min":0.183,"5min":0.155,"15min":0.155},"max":{"1min":2.6070000000000002,"5min":4.5780000000000003,"15min":22.521000000000001},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.67500000000000004,"15min":1.0369999999999999},"min":{"1min":0.247,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.198,"5min":4.335,"15min":22.471},"last":0.377}]},{"osd":2,"last update":"Tue Sep 17 07:15:44 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.91000000000000003,"15min":1.024},"min":{"1min":0.21299999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":2.6360000000000001,"5min":6.1920000000000002,"15min":11.055999999999999},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.73899999999999999,"15min":0.94799999999999995},"min":{"1min":0.22900000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":1.3160000000000001,"5min":4.1559999999999997,"15min":9.3160000000000007},"last":0.30599999999999999}]},{"osd":3,"last update":"Tue Sep 17 07:16:00 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.67800000000000005,"15min":0.94999999999999996},"min":{"1min":0.27400000000000002,"5min":0.25900000000000001,"15min":0.219},"max":{"1min":2.4409999999999998,"5min":4.1269999999999998,"15min":8.5510000000000002},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.371,"5min":0.47399999999999998,"15min":0.98599999999999999},"min":{"1min":0.247,"5min":0.22500000000000001,"15min":0.19500000000000001},"max":{"1min":0.627,"5min":2.8199999999999998,"15min":19.602},"last":0.35299999999999998}]},{"osd":4,"last update":"Tue Sep 17 07:15:20 2024","interfaces":[{"interface":"back","average":{"1min":1.1339999999999999,"5min":1.4670000000000001,"15min":1.129},"min":{"1min":0.36199999999999999,"5min":0.314,"15min":0.25},"max":{"1min":2.5659999999999998,"5min":19.088999999999999,"15min":19.088999999999999},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.98999999999999999,"5min":1.2010000000000001,"15min":1.1759999999999999},"min":{"1min":0.36799999999999999,"5min":0.317,"15min":0.307},"max":{"1min":2.6459999999999999,"5min":18.959,"15min":27.097999999999999},"last":0.499}]},{"osd":5,"last update":"Tue Sep 17 07:15:44 2024","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.47,"15min":1.2110000000000001},"min":{"1min":0.34799999999999998,"5min":0.28899999999999998,"15min":0.27100000000000002},"max":{"1min":3.6869999999999998,"5min":6.6790000000000003,"15min":16.84},"last":0.51000000000000001},{"interface":"front","average":{"1min":1.1559999999999999,"5min":1.069,"15min":1.169},"min":{"1min":0.29299999999999998,"5min":0.29199999999999998,"15min":0.28899999999999998},"max":{"1min":6.2560000000000002,"5min":6.2560000000000002,"15min":16.949999999999999},"last":0.41299999999999998}]},{"osd":6,"last update":"Tue Sep 17 07:16:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":1.5149999999999999,"15min":1.274},"min":{"1min":0.33100000000000002,"5min":0.315,"15min":0.308},"max":{"1min":1.127,"5min":50.497,"15min":50.497},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.99099999999999999,"5min":1.6379999999999999,"15min":1.222},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.28899999999999998},"max":{"1min":3.593,"5min":50.793999999999997,"15min":50.793999999999997},"last":0.48499999999999999}]},{"osd":7,"last update":"Tue Sep 17 07:15:20 2024","interfaces":[{"interface":"back","average":{"1min":1.655,"5min":1.343,"15min":1.1919999999999999},"min":{"1min":0.435,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":3.6120000000000001,"5min":9.8179999999999996,"15min":9.8179999999999996},"last":0.441},{"interface":"front","average":{"1min":1.464,"5min":1.6319999999999999,"15min":1.2410000000000001},"min":{"1min":0.35099999999999998,"5min":0.317,"15min":0.317},"max":{"1min":4.1219999999999999,"5min":7.5839999999999996,"15min":11.007999999999999},"last":0.57999999999999996}]}]},{"osd":0,"up_from":8,"seq":34359738608,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2204240,"kb_used_data":1302720,"kb_used_omap":1,"kb_used_meta":901502,"kb_avail":91540912,"statfs":{"total":95995035648,"available":93737893888,"internally_reserved":0,"allocated":1333985280,"data_stored":1333211270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":923138502},"hb_peers":[2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 17 07:15:18 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.89700000000000002,"15min":0.89500000000000002},"min":{"1min":0.254,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":1.518,"5min":21.876000000000001,"15min":21.876000000000001},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.56399999999999995,"5min":1.2330000000000001,"15min":1.081},"min":{"1min":0.17899999999999999,"5min":0.158,"15min":0.158},"max":{"1min":2.2599999999999998,"5min":22.521999999999998,"15min":22.521999999999998},"last":0.33700000000000002}]},{"osd":2,"last update":"Tue Sep 17 07:18:48 2024","interfaces":[{"interface":"back","average":{"1min":0.312,"5min":0.42699999999999999,"15min":1.113},"min":{"1min":0.156,"5min":0.155,"15min":0.155},"max":{"1min":1.117,"5min":2.6520000000000001,"15min":18.643999999999998},"last":0.19700000000000001},{"interface":"front","average":{"1min":0.307,"5min":0.46200000000000002,"15min":1.0129999999999999},"min":{"1min":0.20399999999999999,"5min":0.189,"15min":0.189},"max":{"1min":0.624,"5min":3.1760000000000002,"15min":17.972999999999999},"last":0.54500000000000004}]},{"osd":3,"last update":"Tue Sep 17 07:18:24 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.46700000000000003,"15min":1.018},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.21199999999999999},"max":{"1min":1.7030000000000001,"5min":3.2000000000000002,"15min":18.629000000000001},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.46800000000000003,"15min":1.0189999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.42699999999999999,"5min":3.1190000000000002,"15min":22.358000000000001},"last":0.255}]},{"osd":4,"last update":"Tue Sep 17 07:18:24 2024","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":0.98699999999999999,"15min":1.1479999999999999},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":5.3179999999999996,"5min":5.5990000000000002,"15min":19.146999999999998},"last":1.103},{"interface":"front","average":{"1min":1.3,"5min":1.0580000000000001,"15min":1.1799999999999999},"min":{"1min":0.29499999999999998,"5min":0.26200000000000001,"15min":0.255},"max":{"1min":6.2329999999999997,"5min":6.2329999999999997,"15min":19.158999999999999},"last":0.38400000000000001}]},{"osd":5,"last update":"Tue Sep 17 07:18:57 2024","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":3.7149999999999999,"15min":2.1549999999999998},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":6.0599999999999996,"5min":216.83699999999999,"15min":216.83699999999999},"last":1.796},{"interface":"front","average":{"1min":1.3360000000000001,"5min":3.6619999999999999,"15min":2.141},"min":{"1min":0.30499999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":4.4050000000000002,"5min":216.816,"15min":216.816},"last":5.4000000000000004}]},{"osd":6,"last update":"Tue Sep 17 07:18:24 2024","interfaces":[{"interface":"back","average":{"1min":1.103,"5min":2.2970000000000002,"15min":1.5620000000000001},"min":{"1min":0.33900000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":3.3889999999999998,"5min":147.13200000000001,"15min":147.13200000000001},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.76300000000000001,"5min":2.2549999999999999,"15min":1.577},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":2.1309999999999998,"5min":147.172,"15min":147.172},"last":0.311}]},{"osd":7,"last update":"Tue Sep 17 07:18:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2070000000000001,"5min":1.236,"15min":1.4390000000000001},"min":{"1min":0.40799999999999997,"5min":0.314,"15min":0.29599999999999999},"max":{"1min":4.3490000000000002,"5min":4.484,"15min":46.183},"last":6.3300000000000001},{"interface":"front","average":{"1min":0.96999999999999997,"5min":1.1579999999999999,"15min":1.3879999999999999},"min":{"1min":0.38800000000000001,"5min":0.32600000000000001,"15min":0.32000000000000001},"max":{"1min":3.528,"5min":5.2149999999999999,"15min":45.914999999999999},"last":0.58499999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1327239168,"data_stored":1327235096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1038221312,"data_stored":1038221312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1205473280,"data_stored":1205469208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":724631552,"data_stored":724631552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1102581760,"data_stored":1102577688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":662831104,"data_stored":662831104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T07:19:17.905 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-17T07:19:17.906 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd pool set .mgr pg_num 11 2024-09-17T07:19:17.906 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:19:17.976 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:17.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:17.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:19:17.986 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:19:18.078 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:19:18.116 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:18.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:18.117 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:19:18.218 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:19:18.258 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:18.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:18.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:19:18.297 INFO:teuthology.orchestra.run.smithi100.stderr:noscrub is unset 2024-09-17T07:19:18.361 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:19:18.394 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:18.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:18.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:19:18.415 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:18 smithi100 ceph-mon[29092]: from='client.15867 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:19:18.415 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:18 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/951893557' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:19:18.415 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:18 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:19:18.415 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:18 smithi100 ceph-mon[29092]: pgmap v791: 168 pgs: 1 remapped+peering, 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfilling, 1 peering, 8 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 88 active+clean; 1.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 844 op/s; 3597/91557 objects degraded (3.929%); 13732/91557 objects misplaced (14.998%); 16 MiB/s, 262 objects/s recovering 2024-09-17T07:19:18.416 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:18 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:19:18.416 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:18 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:19:18.416 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:18 smithi100 ceph-mon[39462]: from='client.15867 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:19:18.416 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:18 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/951893557' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:19:18.416 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:18 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:19:18.416 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:18 smithi100 ceph-mon[39462]: pgmap v791: 168 pgs: 1 remapped+peering, 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfilling, 1 peering, 8 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 88 active+clean; 1.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 844 op/s; 3597/91557 objects degraded (3.929%); 13732/91557 objects misplaced (14.998%); 16 MiB/s, 262 objects/s recovering 2024-09-17T07:19:18.416 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:18 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:19:18.417 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:18 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:19:18.496 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:19:18.534 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:18.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:18.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:19:18.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:18 smithi135 ceph-mon[33238]: from='client.15867 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:19:18.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:18 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/951893557' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:19:18.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:18 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:19:18.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:18 smithi135 ceph-mon[33238]: pgmap v791: 168 pgs: 1 remapped+peering, 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfilling, 1 peering, 8 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 88 active+clean; 1.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 844 op/s; 3597/91557 objects degraded (3.929%); 13732/91557 objects misplaced (14.998%); 16 MiB/s, 262 objects/s recovering 2024-09-17T07:19:18.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:18 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:19:18.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:18 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:19:18.635 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:19:18.664 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:18.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:18.665 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:19:18.766 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:19:18.809 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:18.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:18.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:19:18.911 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:19:18.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:18.954 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:18.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:19:19.055 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:19:19.091 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:19.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:19.093 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:19:19.193 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:19:19.225 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:19.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:19.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:19:19.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:19 smithi135 ceph-mon[33238]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:19:19.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:19 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T07:19:19.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:19 smithi135 ceph-mon[33238]: osdmap e372: 8 total, 7 up, 7 in 2024-09-17T07:19:19.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:19 smithi135 ceph-mon[33238]: 2.1c scrub starts 2024-09-17T07:19:19.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:19 smithi135 ceph-mon[33238]: 2.1c scrub ok 2024-09-17T07:19:19.294 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:19:19 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:19:19.326 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:19:19.354 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:19.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:19.355 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:19:19.448 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:19 smithi100 ceph-mon[39462]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:19:19.448 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:19 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T07:19:19.448 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:19 smithi100 ceph-mon[39462]: osdmap e372: 8 total, 7 up, 7 in 2024-09-17T07:19:19.448 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:19 smithi100 ceph-mon[39462]: 2.1c scrub starts 2024-09-17T07:19:19.448 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:19 smithi100 ceph-mon[39462]: 2.1c scrub ok 2024-09-17T07:19:19.449 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:19 smithi100 ceph-mon[29092]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:19:19.449 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:19 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T07:19:19.449 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:19 smithi100 ceph-mon[29092]: osdmap e372: 8 total, 7 up, 7 in 2024-09-17T07:19:19.449 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:19 smithi100 ceph-mon[29092]: 2.1c scrub starts 2024-09-17T07:19:19.449 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:19 smithi100 ceph-mon[29092]: 2.1c scrub ok 2024-09-17T07:19:19.455 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:19:19.482 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:19.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:19.483 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:19:19.584 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:19:19.627 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:19.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:19.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:19:19.731 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:19:19.765 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:19.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:19.766 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:19:19.866 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:19:19.890 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:19.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:19.891 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:19:19.966 INFO:teuthology.orchestra.run.smithi135.stdout:{ 2024-09-17T07:19:19.967 INFO:teuthology.orchestra.run.smithi135.stdout: "ops": [], 2024-09-17T07:19:19.967 INFO:teuthology.orchestra.run.smithi135.stdout: "complaint_time": 30, 2024-09-17T07:19:19.967 INFO:teuthology.orchestra.run.smithi135.stdout: "num_blocked_ops": 0 2024-09-17T07:19:19.967 INFO:teuthology.orchestra.run.smithi135.stdout:} 2024-09-17T07:19:19.992 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:19:20.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:20.033 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:20.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:19:20.134 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:19:20.156 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:20.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:20.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:19:20.258 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:19:20.289 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:20.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:20.290 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:19:20.391 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:19:20.421 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:20.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:20.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:19:20.523 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:19:20.558 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:20.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:20.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:19:20.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:20 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:19:20.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:20 smithi100 ceph-mon[29092]: pgmap v793: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfilling, 11 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 91 active+clean; 1.6 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.00k op/s; 3002/78924 objects degraded (3.804%); 10199/78924 objects misplaced (12.923%); 31 MiB/s, 497 objects/s recovering 2024-09-17T07:19:20.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:20 smithi100 ceph-mon[29092]: 3.1 scrub starts 2024-09-17T07:19:20.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:20 smithi100 ceph-mon[29092]: 3.1 scrub ok 2024-09-17T07:19:20.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:20 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 3597/91557 objects degraded (3.929%), 30 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-09-17T07:19:20.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:20 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:19:20.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:20 smithi100 ceph-mon[39462]: pgmap v793: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfilling, 11 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 91 active+clean; 1.6 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.00k op/s; 3002/78924 objects degraded (3.804%); 10199/78924 objects misplaced (12.923%); 31 MiB/s, 497 objects/s recovering 2024-09-17T07:19:20.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:20 smithi100 ceph-mon[39462]: 3.1 scrub starts 2024-09-17T07:19:20.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:20 smithi100 ceph-mon[39462]: 3.1 scrub ok 2024-09-17T07:19:20.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:20 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 3597/91557 objects degraded (3.929%), 30 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-09-17T07:19:20.651 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:20 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:19:20.652 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:20 smithi135 ceph-mon[33238]: pgmap v793: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfilling, 11 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 91 active+clean; 1.6 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.00k op/s; 3002/78924 objects degraded (3.804%); 10199/78924 objects misplaced (12.923%); 31 MiB/s, 497 objects/s recovering 2024-09-17T07:19:20.652 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:20 smithi135 ceph-mon[33238]: 3.1 scrub starts 2024-09-17T07:19:20.652 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:20 smithi135 ceph-mon[33238]: 3.1 scrub ok 2024-09-17T07:19:20.652 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:20 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 3597/91557 objects degraded (3.929%), 30 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-09-17T07:19:20.660 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:19:20.691 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-17T07:19:20.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:20.701 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:20.702 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:19:20.802 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:19:20.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:20.852 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:20.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:19:20.953 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:19:21.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:21.008 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:21.008 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:19:21.093 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:19:21.109 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:19:21.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:21.171 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:21.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:19:21.272 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:19:21.294 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:21.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:21.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:19:21.396 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:19:21.421 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:21.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:21.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:19:21.476 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-17T07:19:21.521 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:21 smithi135 ceph-mon[33238]: 1.0 scrub starts 2024-09-17T07:19:21.521 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:21 smithi135 ceph-mon[33238]: 1.0 scrub ok 2024-09-17T07:19:21.521 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:21 smithi135 ceph-mon[33238]: 4.15 scrub starts 2024-09-17T07:19:21.521 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:21 smithi135 ceph-mon[33238]: 4.15 scrub ok 2024-09-17T07:19:21.523 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:19:21.551 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:21.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:21.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:19:21.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:21 smithi100 ceph-mon[29092]: 1.0 scrub starts 2024-09-17T07:19:21.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:21 smithi100 ceph-mon[29092]: 1.0 scrub ok 2024-09-17T07:19:21.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:21 smithi100 ceph-mon[29092]: 4.15 scrub starts 2024-09-17T07:19:21.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:21 smithi100 ceph-mon[29092]: 4.15 scrub ok 2024-09-17T07:19:21.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:21 smithi100 ceph-mon[39462]: 1.0 scrub starts 2024-09-17T07:19:21.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:21 smithi100 ceph-mon[39462]: 1.0 scrub ok 2024-09-17T07:19:21.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:21 smithi100 ceph-mon[39462]: 4.15 scrub starts 2024-09-17T07:19:21.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:21 smithi100 ceph-mon[39462]: 4.15 scrub ok 2024-09-17T07:19:21.654 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:19:21.698 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:21.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:21.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:19:21.800 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:19:21.836 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:21.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:21.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:19:21.938 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:19:21.967 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:21.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:21.968 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:19:22.068 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:19:22.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:22.105 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:22.105 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:19:22.206 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:19:22.245 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:22.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:22.246 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:19:22.346 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:19:22.382 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:22.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:22.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:19:22.484 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:19:22.515 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:19:22.516 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:19:22.520 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:22.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:22.521 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:19:22.621 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:19:22.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:22 smithi100 ceph-mon[29092]: pgmap v794: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 11 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 92 active+clean; 1.5 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 841 B/s rd, 0 B/s wr, 815 op/s; 2849/76677 objects degraded (3.716%); 9399/76677 objects misplaced (12.258%); 22 MiB/s, 359 objects/s recovering 2024-09-17T07:19:22.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:22 smithi100 ceph-mon[29092]: 5.1 scrub starts 2024-09-17T07:19:22.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:22 smithi100 ceph-mon[29092]: 5.1 scrub ok 2024-09-17T07:19:22.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:22 smithi100 ceph-mon[39462]: pgmap v794: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 11 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 92 active+clean; 1.5 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 841 B/s rd, 0 B/s wr, 815 op/s; 2849/76677 objects degraded (3.716%); 9399/76677 objects misplaced (12.258%); 22 MiB/s, 359 objects/s recovering 2024-09-17T07:19:22.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:22 smithi100 ceph-mon[39462]: 5.1 scrub starts 2024-09-17T07:19:22.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:22 smithi100 ceph-mon[39462]: 5.1 scrub ok 2024-09-17T07:19:22.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:22.650 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:22.650 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:19:22.750 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:19:22.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:22.790 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:22.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:19:22.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:22 smithi135 ceph-mon[33238]: pgmap v794: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 11 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 92 active+clean; 1.5 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 841 B/s rd, 0 B/s wr, 815 op/s; 2849/76677 objects degraded (3.716%); 9399/76677 objects misplaced (12.258%); 22 MiB/s, 359 objects/s recovering 2024-09-17T07:19:22.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:22 smithi135 ceph-mon[33238]: 5.1 scrub starts 2024-09-17T07:19:22.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:22 smithi135 ceph-mon[33238]: 5.1 scrub ok 2024-09-17T07:19:22.892 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:19:22.918 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:22.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:22.919 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:19:23.019 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:19:23.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:23.041 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:23.041 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:19:23.142 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:19:23.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:23.182 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:23.183 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:19:23.283 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:19:23.324 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:23.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:23.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:19:23.426 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:19:23.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:23.453 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:23.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:19:23.554 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:19:23.580 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:23.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:23.581 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:19:23.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:23 smithi100 ceph-mon[29092]: 2.19 scrub starts 2024-09-17T07:19:23.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:23 smithi100 ceph-mon[29092]: 2.19 scrub ok 2024-09-17T07:19:23.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:23 smithi100 ceph-mon[39462]: 2.19 scrub starts 2024-09-17T07:19:23.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:23 smithi100 ceph-mon[39462]: 2.19 scrub ok 2024-09-17T07:19:23.682 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:19:23.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:23.705 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:23.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:19:23.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:23 smithi135 ceph-mon[33238]: 2.19 scrub starts 2024-09-17T07:19:23.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:23 smithi135 ceph-mon[33238]: 2.19 scrub ok 2024-09-17T07:19:23.806 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:19:23.849 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:23.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:23.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:19:23.951 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:19:23.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:23.976 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:23.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:19:24.077 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:19:24.115 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:24.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:24.116 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:19:24.217 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:19:24.252 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:24.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:24.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:19:24.354 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:19:24.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:24.383 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:24.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:19:24.484 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:19:24.520 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:24.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:24.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:19:24.622 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:19:24.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:24 smithi100 ceph-mon[29092]: 2.16 scrub starts 2024-09-17T07:19:24.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:24 smithi100 ceph-mon[29092]: 2.16 scrub ok 2024-09-17T07:19:24.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:24 smithi100 ceph-mon[29092]: 6.14 scrub starts 2024-09-17T07:19:24.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:24 smithi100 ceph-mon[29092]: 6.14 scrub ok 2024-09-17T07:19:24.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:24 smithi100 ceph-mon[29092]: 2.18 scrub starts 2024-09-17T07:19:24.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:24 smithi100 ceph-mon[29092]: pgmap v795: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 11 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 92 active+clean; 1.3 GiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.03k op/s; 2607/66756 objects degraded (3.905%); 7465/66756 objects misplaced (11.183%); 17 MiB/s, 273 objects/s recovering 2024-09-17T07:19:24.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:24 smithi100 ceph-mon[29092]: 5.e scrub starts 2024-09-17T07:19:24.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:24 smithi100 ceph-mon[29092]: 5.e scrub ok 2024-09-17T07:19:24.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:24 smithi100 ceph-mon[39462]: 2.16 scrub starts 2024-09-17T07:19:24.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:24 smithi100 ceph-mon[39462]: 2.16 scrub ok 2024-09-17T07:19:24.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:24 smithi100 ceph-mon[39462]: 6.14 scrub starts 2024-09-17T07:19:24.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:24 smithi100 ceph-mon[39462]: 6.14 scrub ok 2024-09-17T07:19:24.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:24 smithi100 ceph-mon[39462]: 2.18 scrub starts 2024-09-17T07:19:24.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:24 smithi100 ceph-mon[39462]: pgmap v795: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 11 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 92 active+clean; 1.3 GiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.03k op/s; 2607/66756 objects degraded (3.905%); 7465/66756 objects misplaced (11.183%); 17 MiB/s, 273 objects/s recovering 2024-09-17T07:19:24.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:24 smithi100 ceph-mon[39462]: 5.e scrub starts 2024-09-17T07:19:24.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:24 smithi100 ceph-mon[39462]: 5.e scrub ok 2024-09-17T07:19:24.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:24.644 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:24.644 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:19:24.745 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:19:24.776 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:24.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:24.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:19:24.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:24 smithi135 ceph-mon[33238]: 2.16 scrub starts 2024-09-17T07:19:24.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:24 smithi135 ceph-mon[33238]: 2.16 scrub ok 2024-09-17T07:19:24.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:24 smithi135 ceph-mon[33238]: 6.14 scrub starts 2024-09-17T07:19:24.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:24 smithi135 ceph-mon[33238]: 6.14 scrub ok 2024-09-17T07:19:24.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:24 smithi135 ceph-mon[33238]: 2.18 scrub starts 2024-09-17T07:19:24.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:24 smithi135 ceph-mon[33238]: pgmap v795: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 11 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 92 active+clean; 1.3 GiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.03k op/s; 2607/66756 objects degraded (3.905%); 7465/66756 objects misplaced (11.183%); 17 MiB/s, 273 objects/s recovering 2024-09-17T07:19:24.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:24 smithi135 ceph-mon[33238]: 5.e scrub starts 2024-09-17T07:19:24.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:24 smithi135 ceph-mon[33238]: 5.e scrub ok 2024-09-17T07:19:24.877 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:19:24.903 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:24.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:24.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:19:25.005 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:19:25.035 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:25.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:25.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:19:25.093 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:19:25.094 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:19:25.094 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:19:25.094 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:19:25.094 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:19:25.094 INFO:teuthology.orchestra.run.smithi100.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-17T07:19:25.094 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:19:25.095 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:19:25.095 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:19:25.099 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:19:25.099 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:19:25.124 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:19:25.125 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:19:25.137 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:19:25.152 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:19:25.152 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:19:25.177 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:25.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:25.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:19:25.279 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:19:25.317 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:25.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:25.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:19:25.419 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:19:25.446 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:25.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:25.448 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:19:25.549 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:19:25.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:25.571 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:25.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:19:25.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:25 smithi100 ceph-mon[29092]: 2.18 scrub ok 2024-09-17T07:19:25.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:25 smithi100 ceph-mon[29092]: 4.8 scrub starts 2024-09-17T07:19:25.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:25 smithi100 ceph-mon[29092]: 4.8 scrub ok 2024-09-17T07:19:25.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:25 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1128349946' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:19:25.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:25 smithi100 ceph-mon[29092]: 5.d scrub starts 2024-09-17T07:19:25.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:25 smithi100 ceph-mon[29092]: 5.d scrub ok 2024-09-17T07:19:25.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:25 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 2607/66756 objects degraded (3.905%), 29 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-09-17T07:19:25.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:25 smithi100 ceph-mon[39462]: 2.18 scrub ok 2024-09-17T07:19:25.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:25 smithi100 ceph-mon[39462]: 4.8 scrub starts 2024-09-17T07:19:25.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:25 smithi100 ceph-mon[39462]: 4.8 scrub ok 2024-09-17T07:19:25.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:25 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1128349946' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:19:25.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:25 smithi100 ceph-mon[39462]: 5.d scrub starts 2024-09-17T07:19:25.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:25 smithi100 ceph-mon[39462]: 5.d scrub ok 2024-09-17T07:19:25.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:25 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 2607/66756 objects degraded (3.905%), 29 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-09-17T07:19:25.672 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:19:25.700 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:25.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:25.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:19:25.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:25 smithi135 ceph-mon[33238]: 2.18 scrub ok 2024-09-17T07:19:25.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:25 smithi135 ceph-mon[33238]: 4.8 scrub starts 2024-09-17T07:19:25.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:25 smithi135 ceph-mon[33238]: 4.8 scrub ok 2024-09-17T07:19:25.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:25 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1128349946' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:19:25.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:25 smithi135 ceph-mon[33238]: 5.d scrub starts 2024-09-17T07:19:25.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:25 smithi135 ceph-mon[33238]: 5.d scrub ok 2024-09-17T07:19:25.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:25 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 2607/66756 objects degraded (3.905%), 29 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-09-17T07:19:25.802 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:19:25.831 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:25.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:25.832 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:19:25.932 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:19:25.962 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:25.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:25.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:19:25.978 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:19:25.979 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:19:26.064 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:19:26.104 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:26.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:26.105 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:19:26.160 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:19:25 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:19:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:19:26.206 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:19:26.233 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:26.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:26.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:19:26.335 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:19:26.372 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:26.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:26.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:19:26.430 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:26 smithi100 ceph-mon[29092]: pgmap v796: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 11 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 92 active+clean; 1.1 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.39k op/s; 1979/53820 objects degraded (3.677%); 6635/53820 objects misplaced (12.328%); 15 MiB/s, 241 objects/s recovering 2024-09-17T07:19:26.430 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:26 smithi100 ceph-mon[29092]: 6.13 scrub starts 2024-09-17T07:19:26.430 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:26 smithi100 ceph-mon[29092]: 6.13 scrub ok 2024-09-17T07:19:26.430 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:26 smithi100 ceph-mon[29092]: 6.15 scrub starts 2024-09-17T07:19:26.430 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:26 smithi100 ceph-mon[29092]: 6.15 scrub ok 2024-09-17T07:19:26.430 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:26 smithi100 ceph-mon[39462]: pgmap v796: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 11 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 92 active+clean; 1.1 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.39k op/s; 1979/53820 objects degraded (3.677%); 6635/53820 objects misplaced (12.328%); 15 MiB/s, 241 objects/s recovering 2024-09-17T07:19:26.430 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:26 smithi100 ceph-mon[39462]: 6.13 scrub starts 2024-09-17T07:19:26.431 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:26 smithi100 ceph-mon[39462]: 6.13 scrub ok 2024-09-17T07:19:26.431 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:26 smithi100 ceph-mon[39462]: 6.15 scrub starts 2024-09-17T07:19:26.431 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:26 smithi100 ceph-mon[39462]: 6.15 scrub ok 2024-09-17T07:19:26.474 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:19:26.501 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:26.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:26.502 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:19:26.603 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:19:26.631 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:26.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:26.633 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:19:26.734 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:19:26.772 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:26.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:26.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:19:26.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:26 smithi135 ceph-mon[33238]: pgmap v796: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 11 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 92 active+clean; 1.1 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.39k op/s; 1979/53820 objects degraded (3.677%); 6635/53820 objects misplaced (12.328%); 15 MiB/s, 241 objects/s recovering 2024-09-17T07:19:26.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:26 smithi135 ceph-mon[33238]: 6.13 scrub starts 2024-09-17T07:19:26.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:26 smithi135 ceph-mon[33238]: 6.13 scrub ok 2024-09-17T07:19:26.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:26 smithi135 ceph-mon[33238]: 6.15 scrub starts 2024-09-17T07:19:26.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:26 smithi135 ceph-mon[33238]: 6.15 scrub ok 2024-09-17T07:19:26.874 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:19:26.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:26.910 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:26.910 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:19:27.011 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:19:27.037 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:27.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:27.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:19:27.139 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:19:27.167 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:27.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:27.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:19:27.269 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:19:27.306 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:27.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:27.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:19:27.408 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:19:27.449 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:27.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:27.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:19:27.469 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:27 smithi100 ceph-mon[29092]: 5.1b scrub starts 2024-09-17T07:19:27.469 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:27 smithi100 ceph-mon[29092]: 5.1b scrub ok 2024-09-17T07:19:27.469 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:27 smithi100 ceph-mon[29092]: 4.1f scrub starts 2024-09-17T07:19:27.469 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:27 smithi100 ceph-mon[29092]: 4.1f scrub ok 2024-09-17T07:19:27.469 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:27 smithi100 ceph-mon[29092]: 6.2 scrub starts 2024-09-17T07:19:27.470 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:27 smithi100 ceph-mon[29092]: 6.2 scrub ok 2024-09-17T07:19:27.470 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:27 smithi100 ceph-mon[39462]: 5.1b scrub starts 2024-09-17T07:19:27.470 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:27 smithi100 ceph-mon[39462]: 5.1b scrub ok 2024-09-17T07:19:27.470 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:27 smithi100 ceph-mon[39462]: 4.1f scrub starts 2024-09-17T07:19:27.470 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:27 smithi100 ceph-mon[39462]: 4.1f scrub ok 2024-09-17T07:19:27.470 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:27 smithi100 ceph-mon[39462]: 6.2 scrub starts 2024-09-17T07:19:27.470 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:27 smithi100 ceph-mon[39462]: 6.2 scrub ok 2024-09-17T07:19:27.551 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:19:27.579 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:27.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:27.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:19:27.607 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:27 smithi135 ceph-mon[33238]: 5.1b scrub starts 2024-09-17T07:19:27.608 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:27 smithi135 ceph-mon[33238]: 5.1b scrub ok 2024-09-17T07:19:27.608 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:27 smithi135 ceph-mon[33238]: 4.1f scrub starts 2024-09-17T07:19:27.608 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:27 smithi135 ceph-mon[33238]: 4.1f scrub ok 2024-09-17T07:19:27.608 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:27 smithi135 ceph-mon[33238]: 6.2 scrub starts 2024-09-17T07:19:27.608 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:27 smithi135 ceph-mon[33238]: 6.2 scrub ok 2024-09-17T07:19:27.681 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:19:27.702 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:27.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:27.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:19:27.804 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:19:27.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:27.833 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:27.833 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:19:27.934 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:19:27.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:27.973 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:27.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:19:27.981 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:19:28.074 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:19:28.110 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:28.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:28.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:19:28.212 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:19:28.247 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:28.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:28.248 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:19:28.349 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:19:28.375 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:28.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:28.377 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:19:28.478 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:19:28.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:28.512 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:28.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:19:28.614 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:19:28.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:28 smithi100 ceph-mon[29092]: pgmap v797: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 11 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 92 active+clean; 1.1 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.26k op/s; 1979/53820 objects degraded (3.677%); 6635/53820 objects misplaced (12.328%); 14 MiB/s, 218 objects/s recovering 2024-09-17T07:19:28.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:28 smithi100 ceph-mon[29092]: 2.7 scrub starts 2024-09-17T07:19:28.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:28 smithi100 ceph-mon[29092]: 2.7 scrub ok 2024-09-17T07:19:28.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:28 smithi100 ceph-mon[29092]: 6.5 scrub starts 2024-09-17T07:19:28.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:28 smithi100 ceph-mon[29092]: 6.5 scrub ok 2024-09-17T07:19:28.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:28 smithi100 ceph-mon[39462]: pgmap v797: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 11 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 92 active+clean; 1.1 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.26k op/s; 1979/53820 objects degraded (3.677%); 6635/53820 objects misplaced (12.328%); 14 MiB/s, 218 objects/s recovering 2024-09-17T07:19:28.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:28 smithi100 ceph-mon[39462]: 2.7 scrub starts 2024-09-17T07:19:28.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:28 smithi100 ceph-mon[39462]: 2.7 scrub ok 2024-09-17T07:19:28.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:28 smithi100 ceph-mon[39462]: 6.5 scrub starts 2024-09-17T07:19:28.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:28 smithi100 ceph-mon[39462]: 6.5 scrub ok 2024-09-17T07:19:28.661 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:28.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:28.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:19:28.763 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:19:28.787 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:28.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:28.788 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:19:28.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:28 smithi135 ceph-mon[33238]: pgmap v797: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 11 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 92 active+clean; 1.1 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.26k op/s; 1979/53820 objects degraded (3.677%); 6635/53820 objects misplaced (12.328%); 14 MiB/s, 218 objects/s recovering 2024-09-17T07:19:28.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:28 smithi135 ceph-mon[33238]: 2.7 scrub starts 2024-09-17T07:19:28.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:28 smithi135 ceph-mon[33238]: 2.7 scrub ok 2024-09-17T07:19:28.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:28 smithi135 ceph-mon[33238]: 6.5 scrub starts 2024-09-17T07:19:28.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:28 smithi135 ceph-mon[33238]: 6.5 scrub ok 2024-09-17T07:19:28.889 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:19:28.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:28.918 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:28.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:19:29.019 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:19:29.042 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:29.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:29.043 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:19:29.144 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:19:29.188 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:29.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:29.190 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:19:29.194 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:19:29.290 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:19:29.326 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:29.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:29.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:19:29.355 INFO:teuthology.orchestra.run.smithi100.stderr:set pool 1 pg_num to 11 2024-09-17T07:19:29.428 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:19:29.454 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:29.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:29.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:19:29.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:29 smithi135 ceph-mon[33238]: 10.11 scrub starts 2024-09-17T07:19:29.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:29 smithi135 ceph-mon[33238]: 10.11 scrub ok 2024-09-17T07:19:29.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:29 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/3438144023' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-09-17T07:19:29.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:29 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-09-17T07:19:29.544 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:19:29 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:19:29.556 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:19:29.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:29.588 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:29.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:19:29.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:29 smithi100 ceph-mon[29092]: 10.11 scrub starts 2024-09-17T07:19:29.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:29 smithi100 ceph-mon[29092]: 10.11 scrub ok 2024-09-17T07:19:29.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:29 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/3438144023' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-09-17T07:19:29.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:29 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-09-17T07:19:29.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:29 smithi100 ceph-mon[39462]: 10.11 scrub starts 2024-09-17T07:19:29.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:29 smithi100 ceph-mon[39462]: 10.11 scrub ok 2024-09-17T07:19:29.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:29 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/3438144023' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-09-17T07:19:29.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:29 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-09-17T07:19:29.689 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:19:29.717 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:29.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:29.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:19:29.819 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:19:29.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:29.848 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:29.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:19:29.950 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:19:29.971 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:29.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:29.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:19:30.073 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:19:30.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:30.096 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:30.096 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:19:30.197 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:19:30.233 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:30.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:30.234 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:19:30.335 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:19:30.358 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:30.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:30.359 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:19:30.460 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:19:30.487 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:30.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:30.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:19:30.505 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:30 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:19:30.506 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:30 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-17T07:19:30.506 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:30 smithi100 ceph-mon[29092]: osdmap e373: 8 total, 7 up, 7 in 2024-09-17T07:19:30.506 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:30 smithi100 ceph-mon[29092]: pgmap v799: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 11 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 92 active+clean; 698 MiB data, 7.4 GiB used, 618 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 743 op/s; 1412/34050 objects degraded (4.147%); 3909/34050 objects misplaced (11.480%) 2024-09-17T07:19:30.506 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:30 smithi100 ceph-mon[29092]: 3.2 scrub starts 2024-09-17T07:19:30.506 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:30 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-09-17T07:19:30.506 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:30 smithi100 ceph-mon[29092]: 3.2 scrub ok 2024-09-17T07:19:30.506 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:30 smithi100 ceph-mon[29092]: 6.9 scrub starts 2024-09-17T07:19:30.506 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:30 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:19:30.506 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:30 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:19:30.506 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:30 smithi100 ceph-mon[29092]: 6.9 scrub ok 2024-09-17T07:19:30.507 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:30 smithi100 ceph-mon[29092]: 5.a scrub starts 2024-09-17T07:19:30.507 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:30 smithi100 ceph-mon[29092]: 5.a scrub ok 2024-09-17T07:19:30.507 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:30 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 1979/53820 objects degraded (3.677%), 29 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-09-17T07:19:30.507 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:30 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-17T07:19:30.507 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:30 smithi100 ceph-mon[29092]: osdmap e374: 8 total, 7 up, 7 in 2024-09-17T07:19:30.507 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:30 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:19:30.507 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:30 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-17T07:19:30.507 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:30 smithi100 ceph-mon[39462]: osdmap e373: 8 total, 7 up, 7 in 2024-09-17T07:19:30.507 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:30 smithi100 ceph-mon[39462]: pgmap v799: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 11 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 92 active+clean; 698 MiB data, 7.4 GiB used, 618 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 743 op/s; 1412/34050 objects degraded (4.147%); 3909/34050 objects misplaced (11.480%) 2024-09-17T07:19:30.508 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:30 smithi100 ceph-mon[39462]: 3.2 scrub starts 2024-09-17T07:19:30.508 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:30 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-09-17T07:19:30.508 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:30 smithi100 ceph-mon[39462]: 3.2 scrub ok 2024-09-17T07:19:30.508 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:30 smithi100 ceph-mon[39462]: 6.9 scrub starts 2024-09-17T07:19:30.508 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:30 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:19:30.508 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:30 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:19:30.508 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:30 smithi100 ceph-mon[39462]: 6.9 scrub ok 2024-09-17T07:19:30.508 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:30 smithi100 ceph-mon[39462]: 5.a scrub starts 2024-09-17T07:19:30.509 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:30 smithi100 ceph-mon[39462]: 5.a scrub ok 2024-09-17T07:19:30.509 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:30 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 1979/53820 objects degraded (3.677%), 29 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-09-17T07:19:30.509 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:30 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-17T07:19:30.509 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:30 smithi100 ceph-mon[39462]: osdmap e374: 8 total, 7 up, 7 in 2024-09-17T07:19:30.590 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:19:30.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:30.632 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:30.632 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:19:30.733 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:19:30.770 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:30.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:30.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:19:30.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:30 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:19:30.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:30 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-17T07:19:30.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:30 smithi135 ceph-mon[33238]: osdmap e373: 8 total, 7 up, 7 in 2024-09-17T07:19:30.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:30 smithi135 ceph-mon[33238]: pgmap v799: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 11 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 92 active+clean; 698 MiB data, 7.4 GiB used, 618 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 743 op/s; 1412/34050 objects degraded (4.147%); 3909/34050 objects misplaced (11.480%) 2024-09-17T07:19:30.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:30 smithi135 ceph-mon[33238]: 3.2 scrub starts 2024-09-17T07:19:30.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:30 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-09-17T07:19:30.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:30 smithi135 ceph-mon[33238]: 3.2 scrub ok 2024-09-17T07:19:30.796 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:30 smithi135 ceph-mon[33238]: 6.9 scrub starts 2024-09-17T07:19:30.797 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:30 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:19:30.797 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:30 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:19:30.797 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:30 smithi135 ceph-mon[33238]: 6.9 scrub ok 2024-09-17T07:19:30.797 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:30 smithi135 ceph-mon[33238]: 5.a scrub starts 2024-09-17T07:19:30.797 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:30 smithi135 ceph-mon[33238]: 5.a scrub ok 2024-09-17T07:19:30.798 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:30 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 1979/53820 objects degraded (3.677%), 29 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-09-17T07:19:30.798 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:30 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-17T07:19:30.799 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:30 smithi135 ceph-mon[33238]: osdmap e374: 8 total, 7 up, 7 in 2024-09-17T07:19:30.872 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:19:30.900 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:30.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:30.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:19:31.002 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:19:31.036 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:31.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:31.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:19:31.138 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:19:31.167 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:31.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:31.168 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:19:31.269 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:19:31.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:31.302 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:31.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:19:31.403 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:19:31.449 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:31.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:31.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:19:31.551 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:19:31.590 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:31.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:31.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:19:31.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:31 smithi100 ceph-mon[29092]: 2.1b scrub starts 2024-09-17T07:19:31.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:31 smithi100 ceph-mon[29092]: 2.1b scrub ok 2024-09-17T07:19:31.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:31 smithi100 ceph-mon[29092]: 5.f scrub starts 2024-09-17T07:19:31.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:31 smithi100 ceph-mon[29092]: 5.f scrub ok 2024-09-17T07:19:31.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:31 smithi100 ceph-mon[29092]: 5.1f scrub starts 2024-09-17T07:19:31.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:31 smithi100 ceph-mon[29092]: 5.1f scrub ok 2024-09-17T07:19:31.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:31 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-09-17T07:19:31.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:31 smithi100 ceph-mon[39462]: 2.1b scrub starts 2024-09-17T07:19:31.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:31 smithi100 ceph-mon[39462]: 2.1b scrub ok 2024-09-17T07:19:31.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:31 smithi100 ceph-mon[39462]: 5.f scrub starts 2024-09-17T07:19:31.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:31 smithi100 ceph-mon[39462]: 5.f scrub ok 2024-09-17T07:19:31.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:31 smithi100 ceph-mon[39462]: 5.1f scrub starts 2024-09-17T07:19:31.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:31 smithi100 ceph-mon[39462]: 5.1f scrub ok 2024-09-17T07:19:31.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:31 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-09-17T07:19:31.692 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:19:31.726 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:31.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:31.727 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:19:31.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:31 smithi135 ceph-mon[33238]: 2.1b scrub starts 2024-09-17T07:19:31.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:31 smithi135 ceph-mon[33238]: 2.1b scrub ok 2024-09-17T07:19:31.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:31 smithi135 ceph-mon[33238]: 5.f scrub starts 2024-09-17T07:19:31.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:31 smithi135 ceph-mon[33238]: 5.f scrub ok 2024-09-17T07:19:31.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:31 smithi135 ceph-mon[33238]: 5.1f scrub starts 2024-09-17T07:19:31.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:31 smithi135 ceph-mon[33238]: 5.1f scrub ok 2024-09-17T07:19:31.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:31 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-09-17T07:19:31.828 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:19:31.863 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:31.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:31.864 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:19:31.965 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:19:32.002 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:32.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:32.004 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:19:32.104 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:19:32.134 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:32.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:32.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:19:32.216 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:19:32.216 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:19:32.217 INFO:teuthology.orchestra.run.smithi100.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-17T07:19:32.236 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:19:32.248 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:19:32.248 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:19:32.265 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:32.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:32.267 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:19:32.276 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:19:32.276 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:19:32.305 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:19:32.305 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:19:32.321 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:19:32.322 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:19:32.368 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:19:32.387 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:19:32.387 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:19:32.396 INFO:teuthology.orchestra.run.smithi100.stderr:nodeep-scrub is unset 2024-09-17T07:19:32.402 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:32.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:32.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:19:32.420 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:19:32.420 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:19:32.504 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:19:32.529 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:32.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:32.530 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:19:32.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:32 smithi100 ceph-mon[29092]: pgmap v801: 178 pgs: 10 unknown, 1 active+undersized+degraded+remapped+backfill_toofull, 11 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 92 active+clean; 640 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 538 op/s; 1171/31278 objects degraded (3.744%); 3909/31278 objects misplaced (12.498%) 2024-09-17T07:19:32.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:32 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-17T07:19:32.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:32 smithi100 ceph-mon[29092]: osdmap e375: 8 total, 7 up, 7 in 2024-09-17T07:19:32.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:32 smithi100 ceph-mon[29092]: 5.2 scrub starts 2024-09-17T07:19:32.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:32 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/866006458' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:19:32.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:32 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:19:32.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:32 smithi100 ceph-mon[29092]: 5.2 scrub ok 2024-09-17T07:19:32.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:32 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/3943685356' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:19:32.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:32 smithi100 ceph-mon[39462]: pgmap v801: 178 pgs: 10 unknown, 1 active+undersized+degraded+remapped+backfill_toofull, 11 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 92 active+clean; 640 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 538 op/s; 1171/31278 objects degraded (3.744%); 3909/31278 objects misplaced (12.498%) 2024-09-17T07:19:32.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:32 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-17T07:19:32.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:32 smithi100 ceph-mon[39462]: osdmap e375: 8 total, 7 up, 7 in 2024-09-17T07:19:32.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:32 smithi100 ceph-mon[39462]: 5.2 scrub starts 2024-09-17T07:19:32.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:32 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/866006458' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:19:32.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:32 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:19:32.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:32 smithi100 ceph-mon[39462]: 5.2 scrub ok 2024-09-17T07:19:32.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:32 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/3943685356' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:19:32.631 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:19:32.671 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:32.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:32.672 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:19:32.731 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:32 smithi135 ceph-mon[33238]: pgmap v801: 178 pgs: 10 unknown, 1 active+undersized+degraded+remapped+backfill_toofull, 11 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 92 active+clean; 640 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 538 op/s; 1171/31278 objects degraded (3.744%); 3909/31278 objects misplaced (12.498%) 2024-09-17T07:19:32.731 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:32 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-17T07:19:32.731 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:32 smithi135 ceph-mon[33238]: osdmap e375: 8 total, 7 up, 7 in 2024-09-17T07:19:32.731 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:32 smithi135 ceph-mon[33238]: 5.2 scrub starts 2024-09-17T07:19:32.731 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:32 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/866006458' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:19:32.731 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:32 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:19:32.732 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:32 smithi135 ceph-mon[33238]: 5.2 scrub ok 2024-09-17T07:19:32.732 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:32 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/3943685356' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:19:32.773 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:19:32.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:32.810 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:32.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:19:32.911 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:19:32.950 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:32.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:32.951 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:19:33.052 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:19:33.088 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:33.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:33.089 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:19:33.190 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:19:33.206 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:19:33.207 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:19:33.209 INFO:teuthology.orchestra.run.smithi135.stdout:{ 2024-09-17T07:19:33.209 INFO:teuthology.orchestra.run.smithi135.stdout: "size": 20, 2024-09-17T07:19:33.209 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 600, 2024-09-17T07:19:33.209 INFO:teuthology.orchestra.run.smithi135.stdout: "ops": [ 2024-09-17T07:19:33.209 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.209 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.15648.0:216532 10.22 10:476e954f:::benchmark_data_smithi100_155039_object93184:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e368)", 2024-09-17T07:19:33.210 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:19:13.540930+0000", 2024-09-17T07:19:33.210 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 19.662698318, 2024-09-17T07:19:33.210 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.94900719200000005, 2024-09-17T07:19:33.210 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:19:33.210 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:19:33.210 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:19:33.210 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.15648", 2024-09-17T07:19:33.210 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/1684940078", 2024-09-17T07:19:33.210 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 216532 2024-09-17T07:19:33.210 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.210 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:19:33.210 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.210 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:19:33.211 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.540930+0000", 2024-09-17T07:19:33.211 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:33.211 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.211 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.211 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:19:33.211 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.540930+0000", 2024-09-17T07:19:33.211 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:33.211 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.211 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.211 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:19:33.212 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.540928+0000", 2024-09-17T07:19:33.212 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:19:33.212 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.212 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.212 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:19:33.212 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.540935+0000", 2024-09-17T07:19:33.212 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.832e-06 2024-09-17T07:19:33.212 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.212 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.212 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:19:33.212 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.540943+0000", 2024-09-17T07:19:33.212 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.5789999999999998e-06 2024-09-17T07:19:33.212 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.213 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.213 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:19:33.213 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.540948+0000", 2024-09-17T07:19:33.213 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.2009999999999998e-06 2024-09-17T07:19:33.213 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.213 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.213 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:33.213 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.541019+0000", 2024-09-17T07:19:33.213 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.1668999999999994e-05 2024-09-17T07:19:33.213 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.213 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.213 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:19:33.213 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.541028+0000", 2024-09-17T07:19:33.214 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.3259999999999996e-06 2024-09-17T07:19:33.214 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.214 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.214 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:33.214 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.486029+0000", 2024-09-17T07:19:33.214 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.94500116700000003 2024-09-17T07:19:33.214 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.214 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.214 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:19:33.214 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.486163+0000", 2024-09-17T07:19:33.214 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00013374599999999999 2024-09-17T07:19:33.215 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.215 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.215 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,7", 2024-09-17T07:19:33.215 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.486350+0000", 2024-09-17T07:19:33.215 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00018716499999999999 2024-09-17T07:19:33.215 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.215 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.215 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:33.215 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.488705+0000", 2024-09-17T07:19:33.215 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0023555249999999998 2024-09-17T07:19:33.215 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.215 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.215 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:33.216 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.488743+0000", 2024-09-17T07:19:33.216 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.7150000000000002e-05 2024-09-17T07:19:33.216 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.216 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.216 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:19:33.216 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.489783+0000", 2024-09-17T07:19:33.216 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.001040933 2024-09-17T07:19:33.216 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.216 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.216 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:19:33.216 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.489828+0000", 2024-09-17T07:19:33.216 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.4895999999999998e-05 2024-09-17T07:19:33.216 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.217 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.217 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:19:33.217 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.489937+0000", 2024-09-17T07:19:33.217 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000108492 2024-09-17T07:19:33.217 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:33.217 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:19:33.217 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:33.217 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.218 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.218 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.15648.0:216534 10.22 10:44f78373:::benchmark_data_smithi100_155039_object93186:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e368)", 2024-09-17T07:19:33.218 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:19:13.540960+0000", 2024-09-17T07:19:33.218 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 19.662667523, 2024-09-17T07:19:33.218 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.95154364199999997, 2024-09-17T07:19:33.218 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:19:33.218 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:19:33.218 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:19:33.218 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.15648", 2024-09-17T07:19:33.218 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/1684940078", 2024-09-17T07:19:33.218 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 216534 2024-09-17T07:19:33.218 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.219 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:19:33.219 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.219 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:19:33.219 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.540960+0000", 2024-09-17T07:19:33.219 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:33.219 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.219 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.219 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:19:33.219 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.540960+0000", 2024-09-17T07:19:33.219 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:33.219 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.219 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.220 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:19:33.220 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.540960+0000", 2024-09-17T07:19:33.220 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999995 2024-09-17T07:19:33.220 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.220 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.220 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:19:33.220 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.540962+0000", 2024-09-17T07:19:33.220 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.2529999999999999e-06 2024-09-17T07:19:33.220 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.220 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.220 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:19:33.220 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.540967+0000", 2024-09-17T07:19:33.220 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.5190000000000003e-06 2024-09-17T07:19:33.221 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.221 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.221 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:19:33.221 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.540971+0000", 2024-09-17T07:19:33.221 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.2270000000000004e-06 2024-09-17T07:19:33.221 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.221 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.221 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:33.221 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.541047+0000", 2024-09-17T07:19:33.222 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.5451999999999995e-05 2024-09-17T07:19:33.222 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.222 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.222 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:19:33.222 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.541052+0000", 2024-09-17T07:19:33.222 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.1649999999999997e-06 2024-09-17T07:19:33.222 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.222 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.222 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:33.222 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.487784+0000", 2024-09-17T07:19:33.222 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.94673233499999998 2024-09-17T07:19:33.222 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.222 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.222 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:19:33.223 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.487883+0000", 2024-09-17T07:19:33.223 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.9464999999999997e-05 2024-09-17T07:19:33.223 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.223 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.223 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,7", 2024-09-17T07:19:33.223 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.488000+0000", 2024-09-17T07:19:33.223 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000116508 2024-09-17T07:19:33.223 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.223 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.223 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:33.223 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.490782+0000", 2024-09-17T07:19:33.223 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0027818550000000002 2024-09-17T07:19:33.224 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.224 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.224 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:33.224 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.490927+0000", 2024-09-17T07:19:33.224 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00014562399999999999 2024-09-17T07:19:33.224 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.224 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.224 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:19:33.224 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.492327+0000", 2024-09-17T07:19:33.224 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0013995940000000001 2024-09-17T07:19:33.224 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.224 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.225 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:19:33.225 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.492385+0000", 2024-09-17T07:19:33.225 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.7939000000000002e-05 2024-09-17T07:19:33.225 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.225 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.225 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:19:33.225 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.492504+0000", 2024-09-17T07:19:33.225 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00011908099999999999 2024-09-17T07:19:33.225 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:33.225 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:19:33.225 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:33.225 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.226 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.226 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.15648.0:216593 10.22 10:44ff2e07:::benchmark_data_smithi100_155039_object93245:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e368)", 2024-09-17T07:19:33.226 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:19:13.583863+0000", 2024-09-17T07:19:33.226 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 19.619764696000001, 2024-09-17T07:19:33.226 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.91007807799999996, 2024-09-17T07:19:33.226 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:19:33.226 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:19:33.226 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:19:33.227 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.15648", 2024-09-17T07:19:33.227 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/1684940078", 2024-09-17T07:19:33.227 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 216593 2024-09-17T07:19:33.227 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.227 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:19:33.227 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.227 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:19:33.227 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.583863+0000", 2024-09-17T07:19:33.227 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:33.227 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.227 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.228 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:19:33.228 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.583863+0000", 2024-09-17T07:19:33.228 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:33.228 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.228 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.228 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:19:33.228 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.583862+0000", 2024-09-17T07:19:33.228 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.999999 2024-09-17T07:19:33.228 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.228 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.228 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:19:33.228 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.583867+0000", 2024-09-17T07:19:33.229 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.0320000000000003e-06 2024-09-17T07:19:33.229 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.229 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.229 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:19:33.229 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.583877+0000", 2024-09-17T07:19:33.229 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.9110000000000005e-06 2024-09-17T07:19:33.229 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.229 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.229 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:19:33.229 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.583905+0000", 2024-09-17T07:19:33.229 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.7909e-05 2024-09-17T07:19:33.229 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.230 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.230 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:33.230 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.583952+0000", 2024-09-17T07:19:33.230 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.6618e-05 2024-09-17T07:19:33.230 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.230 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.230 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:19:33.230 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.583960+0000", 2024-09-17T07:19:33.230 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.1759999999999993e-06 2024-09-17T07:19:33.230 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.231 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.231 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:33.231 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.488777+0000", 2024-09-17T07:19:33.231 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.904817443 2024-09-17T07:19:33.231 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.231 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.231 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:19:33.231 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.488883+0000", 2024-09-17T07:19:33.231 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00010576799999999999 2024-09-17T07:19:33.231 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.231 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.231 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,7", 2024-09-17T07:19:33.231 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.489051+0000", 2024-09-17T07:19:33.232 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00016814600000000001 2024-09-17T07:19:33.232 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.232 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.232 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:33.232 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.490819+0000", 2024-09-17T07:19:33.232 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0017672359999999999 2024-09-17T07:19:33.232 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.232 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.232 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:33.232 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.491093+0000", 2024-09-17T07:19:33.232 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00027445099999999999 2024-09-17T07:19:33.232 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.233 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.233 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:19:33.233 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.493680+0000", 2024-09-17T07:19:33.233 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.002586769 2024-09-17T07:19:33.233 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.233 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.233 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:19:33.233 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.493723+0000", 2024-09-17T07:19:33.233 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.2802000000000001e-05 2024-09-17T07:19:33.233 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.233 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.233 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:19:33.233 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.493941+0000", 2024-09-17T07:19:33.234 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000218729 2024-09-17T07:19:33.234 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:33.234 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:19:33.234 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:33.234 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.234 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.234 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.15648.0:216613 10.22 10:45484719:::benchmark_data_smithi100_155039_object93265:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e368)", 2024-09-17T07:19:33.234 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:19:13.600806+0000", 2024-09-17T07:19:33.234 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 19.602821780999999, 2024-09-17T07:19:33.234 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.89328455299999998, 2024-09-17T07:19:33.234 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:19:33.234 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:19:33.234 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:19:33.235 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.15648", 2024-09-17T07:19:33.235 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/1684940078", 2024-09-17T07:19:33.235 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 216613 2024-09-17T07:19:33.235 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.235 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:19:33.235 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.235 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:19:33.235 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.600806+0000", 2024-09-17T07:19:33.235 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:33.235 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.235 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.235 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:19:33.235 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.600806+0000", 2024-09-17T07:19:33.236 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:33.236 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.236 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.236 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:19:33.236 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.600805+0000", 2024-09-17T07:19:33.236 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:19:33.236 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.236 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.236 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:19:33.236 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.600810+0000", 2024-09-17T07:19:33.236 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.1240000000000004e-06 2024-09-17T07:19:33.236 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.236 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.236 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:19:33.237 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.600820+0000", 2024-09-17T07:19:33.237 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0339e-05 2024-09-17T07:19:33.237 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.237 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.237 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:19:33.237 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.600828+0000", 2024-09-17T07:19:33.237 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.695e-06 2024-09-17T07:19:33.237 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.237 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.237 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:33.237 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.600879+0000", 2024-09-17T07:19:33.237 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.0730999999999999e-05 2024-09-17T07:19:33.238 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.238 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.238 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:19:33.238 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.600890+0000", 2024-09-17T07:19:33.238 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0784e-05 2024-09-17T07:19:33.238 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.238 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.238 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:33.238 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.489976+0000", 2024-09-17T07:19:33.238 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.88908630600000005 2024-09-17T07:19:33.239 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.239 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.239 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:19:33.239 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.490075+0000", 2024-09-17T07:19:33.239 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.8628999999999994e-05 2024-09-17T07:19:33.239 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.239 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.239 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,7", 2024-09-17T07:19:33.239 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.490201+0000", 2024-09-17T07:19:33.239 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00012603700000000001 2024-09-17T07:19:33.239 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.239 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.239 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:33.239 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.492313+0000", 2024-09-17T07:19:33.240 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0021124099999999999 2024-09-17T07:19:33.240 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.240 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.240 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:33.240 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.493667+0000", 2024-09-17T07:19:33.240 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.001354355 2024-09-17T07:19:33.240 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.240 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.240 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:19:33.240 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.493951+0000", 2024-09-17T07:19:33.240 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00028409699999999997 2024-09-17T07:19:33.240 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.240 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.241 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:19:33.241 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.494006+0000", 2024-09-17T07:19:33.241 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.5078999999999999e-05 2024-09-17T07:19:33.241 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.241 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.242 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:19:33.242 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.494091+0000", 2024-09-17T07:19:33.242 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.4266999999999998e-05 2024-09-17T07:19:33.242 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:33.242 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:19:33.242 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:33.242 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.242 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.242 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.15648.0:216710 10.22 10:44fb0f39:::benchmark_data_smithi100_155039_object93362:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e368)", 2024-09-17T07:19:33.242 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:19:13.667311+0000", 2024-09-17T07:19:33.242 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 19.536316673999998, 2024-09-17T07:19:33.242 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.826953296, 2024-09-17T07:19:33.243 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:19:33.243 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:19:33.243 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:19:33.243 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.15648", 2024-09-17T07:19:33.243 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/1684940078", 2024-09-17T07:19:33.243 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 216710 2024-09-17T07:19:33.243 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.243 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:19:33.243 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.243 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:19:33.243 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.667311+0000", 2024-09-17T07:19:33.243 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:33.243 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.244 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.244 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:19:33.244 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.667311+0000", 2024-09-17T07:19:33.244 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:33.244 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.244 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.244 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:19:33.244 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.667310+0000", 2024-09-17T07:19:33.244 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:19:33.244 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.244 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.244 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:19:33.244 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.667316+0000", 2024-09-17T07:19:33.244 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.0009999999999998e-06 2024-09-17T07:19:33.245 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.245 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.245 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:19:33.245 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.667331+0000", 2024-09-17T07:19:33.245 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.5413999999999998e-05 2024-09-17T07:19:33.245 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.245 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.245 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:19:33.245 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.667339+0000", 2024-09-17T07:19:33.245 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.6559999999999992e-06 2024-09-17T07:19:33.245 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.245 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.246 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:33.246 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.667403+0000", 2024-09-17T07:19:33.246 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.4208999999999995e-05 2024-09-17T07:19:33.246 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.246 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.246 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:19:33.246 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.667412+0000", 2024-09-17T07:19:33.246 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.6009999999999995e-06 2024-09-17T07:19:33.246 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.246 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.246 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:33.246 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.491125+0000", 2024-09-17T07:19:33.246 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.82371351599999998 2024-09-17T07:19:33.247 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.247 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.247 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:19:33.247 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.491241+0000", 2024-09-17T07:19:33.247 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00011553 2024-09-17T07:19:33.247 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.247 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.247 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,7", 2024-09-17T07:19:33.247 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.491389+0000", 2024-09-17T07:19:33.247 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000148164 2024-09-17T07:19:33.247 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.247 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.247 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:33.248 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.493616+0000", 2024-09-17T07:19:33.248 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.002227035 2024-09-17T07:19:33.248 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.248 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.248 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:19:33.248 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.494100+0000", 2024-09-17T07:19:33.248 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00048435399999999999 2024-09-17T07:19:33.248 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.248 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.249 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:33.249 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.494138+0000", 2024-09-17T07:19:33.249 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.8040000000000002e-05 2024-09-17T07:19:33.249 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.249 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.249 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:19:33.249 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.494176+0000", 2024-09-17T07:19:33.249 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.7691000000000002e-05 2024-09-17T07:19:33.249 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.249 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.249 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:19:33.250 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.494265+0000", 2024-09-17T07:19:33.250 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.8648999999999998e-05 2024-09-17T07:19:33.250 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:33.250 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:19:33.250 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:33.250 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.250 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.250 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.15648.0:216747 10.22 10:442e82b0:::benchmark_data_smithi100_155039_object93399:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e368)", 2024-09-17T07:19:33.250 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:19:13.691975+0000", 2024-09-17T07:19:33.250 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 19.511653305999999, 2024-09-17T07:19:33.250 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.81663315700000005, 2024-09-17T07:19:33.250 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:19:33.251 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:19:33.251 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:19:33.251 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.15648", 2024-09-17T07:19:33.251 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/1684940078", 2024-09-17T07:19:33.251 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 216747 2024-09-17T07:19:33.251 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.251 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:19:33.251 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.251 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:19:33.251 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.691975+0000", 2024-09-17T07:19:33.251 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:33.251 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.252 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.252 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:19:33.252 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.691975+0000", 2024-09-17T07:19:33.252 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:33.252 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.252 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.252 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:19:33.252 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.691973+0000", 2024-09-17T07:19:33.252 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999976 2024-09-17T07:19:33.252 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.252 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.253 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:19:33.253 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.691981+0000", 2024-09-17T07:19:33.253 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.0620000000000001e-06 2024-09-17T07:19:33.253 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.253 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.253 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:19:33.253 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.691996+0000", 2024-09-17T07:19:33.253 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.5546999999999999e-05 2024-09-17T07:19:33.253 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.253 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.253 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:19:33.253 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.692005+0000", 2024-09-17T07:19:33.253 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.5350000000000004e-06 2024-09-17T07:19:33.254 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.254 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.254 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:33.254 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.692050+0000", 2024-09-17T07:19:33.254 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.5825999999999998e-05 2024-09-17T07:19:33.254 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.254 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.254 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:19:33.254 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.692059+0000", 2024-09-17T07:19:33.254 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.6319999999999997e-06 2024-09-17T07:19:33.254 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.254 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.255 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:33.255 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.492537+0000", 2024-09-17T07:19:33.255 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.80047769499999999 2024-09-17T07:19:33.255 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.255 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.255 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:19:33.255 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.492641+0000", 2024-09-17T07:19:33.255 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000103873 2024-09-17T07:19:33.255 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.255 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.255 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,7", 2024-09-17T07:19:33.255 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.492799+0000", 2024-09-17T07:19:33.255 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00015800499999999999 2024-09-17T07:19:33.255 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.256 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.256 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:33.256 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.500555+0000", 2024-09-17T07:19:33.256 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0077559320000000001 2024-09-17T07:19:33.256 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.256 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.256 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:33.256 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.500590+0000", 2024-09-17T07:19:33.256 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.4882999999999998e-05 2024-09-17T07:19:33.257 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.257 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.257 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:19:33.257 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.508411+0000", 2024-09-17T07:19:33.257 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0078215030000000005 2024-09-17T07:19:33.258 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.258 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.258 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:19:33.258 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.508515+0000", 2024-09-17T07:19:33.258 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000103942 2024-09-17T07:19:33.258 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.258 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.258 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:19:33.258 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.508608+0000", 2024-09-17T07:19:33.258 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.2874000000000006e-05 2024-09-17T07:19:33.259 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:33.259 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:19:33.259 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:33.259 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.259 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.259 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.15648.0:216904 10.22 10:45730ade:::benchmark_data_smithi100_155039_object93556:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e368)", 2024-09-17T07:19:33.259 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:19:13.800169+0000", 2024-09-17T07:19:33.259 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 19.403459440999999, 2024-09-17T07:19:33.259 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.70857902299999997, 2024-09-17T07:19:33.260 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:19:33.260 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:19:33.260 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:19:33.260 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.15648", 2024-09-17T07:19:33.260 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/1684940078", 2024-09-17T07:19:33.260 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 216904 2024-09-17T07:19:33.260 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.260 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:19:33.260 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.260 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:19:33.260 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.800169+0000", 2024-09-17T07:19:33.260 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:33.260 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.261 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.261 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:19:33.261 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.800169+0000", 2024-09-17T07:19:33.261 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:33.261 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.261 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.261 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:19:33.261 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.800167+0000", 2024-09-17T07:19:33.261 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:19:33.261 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.261 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.261 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:19:33.261 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.800172+0000", 2024-09-17T07:19:33.262 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.887e-06 2024-09-17T07:19:33.262 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.262 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.262 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:19:33.262 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.800181+0000", 2024-09-17T07:19:33.262 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.7849999999999997e-06 2024-09-17T07:19:33.262 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.262 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.262 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:19:33.262 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.800187+0000", 2024-09-17T07:19:33.262 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.1059999999999996e-06 2024-09-17T07:19:33.262 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.262 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.262 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:33.263 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.801253+0000", 2024-09-17T07:19:33.263 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.001066214 2024-09-17T07:19:33.263 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.263 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.263 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:19:33.263 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.801262+0000", 2024-09-17T07:19:33.263 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.8729999999999996e-06 2024-09-17T07:19:33.263 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.263 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.263 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:33.263 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.494309+0000", 2024-09-17T07:19:33.264 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.69304664100000002 2024-09-17T07:19:33.264 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.264 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.264 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:19:33.264 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.494438+0000", 2024-09-17T07:19:33.264 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00012926699999999999 2024-09-17T07:19:33.264 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.264 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.264 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,7", 2024-09-17T07:19:33.264 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.494624+0000", 2024-09-17T07:19:33.264 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000186579 2024-09-17T07:19:33.264 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.264 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.265 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:33.265 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.500624+0000", 2024-09-17T07:19:33.265 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.005999267 2024-09-17T07:19:33.265 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.265 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.265 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:33.265 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.502001+0000", 2024-09-17T07:19:33.265 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0013771269999999999 2024-09-17T07:19:33.265 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.265 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.265 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:19:33.265 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.508622+0000", 2024-09-17T07:19:33.265 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0066211849999999999 2024-09-17T07:19:33.266 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.266 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.266 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:19:33.266 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.508659+0000", 2024-09-17T07:19:33.266 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.6882999999999999e-05 2024-09-17T07:19:33.266 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.266 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.266 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:19:33.266 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.508748+0000", 2024-09-17T07:19:33.266 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.8755e-05 2024-09-17T07:19:33.266 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:33.266 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:19:33.266 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:33.266 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.267 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.267 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.15648.0:216923 10.22 10:45e376da:::benchmark_data_smithi100_155039_object93575:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e368)", 2024-09-17T07:19:33.267 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:19:13.815203+0000", 2024-09-17T07:19:33.267 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 19.388424552, 2024-09-17T07:19:33.267 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.69367317900000003, 2024-09-17T07:19:33.267 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:19:33.267 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:19:33.267 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:19:33.267 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.15648", 2024-09-17T07:19:33.267 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/1684940078", 2024-09-17T07:19:33.267 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 216923 2024-09-17T07:19:33.267 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.268 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:19:33.268 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.268 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:19:33.268 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.815203+0000", 2024-09-17T07:19:33.268 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:33.268 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.268 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.268 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:19:33.268 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.815203+0000", 2024-09-17T07:19:33.268 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:33.268 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.268 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.268 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:19:33.269 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.815202+0000", 2024-09-17T07:19:33.269 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999981 2024-09-17T07:19:33.269 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.269 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.269 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:19:33.269 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.815209+0000", 2024-09-17T07:19:33.269 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.3479999999999998e-06 2024-09-17T07:19:33.269 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.269 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.269 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:19:33.269 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.815224+0000", 2024-09-17T07:19:33.269 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.5248e-05 2024-09-17T07:19:33.269 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.269 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.270 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:19:33.270 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.815232+0000", 2024-09-17T07:19:33.270 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.9240000000000007e-06 2024-09-17T07:19:33.270 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.270 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.270 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:33.270 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.815302+0000", 2024-09-17T07:19:33.270 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.9478000000000001e-05 2024-09-17T07:19:33.270 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.270 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.270 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:19:33.270 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.815311+0000", 2024-09-17T07:19:33.270 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.3109999999999995e-06 2024-09-17T07:19:33.270 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.271 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.271 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:33.271 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.495544+0000", 2024-09-17T07:19:33.271 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.68023347599999995 2024-09-17T07:19:33.271 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.271 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.271 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:19:33.271 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.495651+0000", 2024-09-17T07:19:33.271 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000106925 2024-09-17T07:19:33.271 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.271 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.271 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,7", 2024-09-17T07:19:33.271 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.495808+0000", 2024-09-17T07:19:33.271 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00015719700000000001 2024-09-17T07:19:33.272 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.272 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.272 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:33.272 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.500654+0000", 2024-09-17T07:19:33.272 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0048456310000000004 2024-09-17T07:19:33.272 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.272 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.272 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:33.272 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.502031+0000", 2024-09-17T07:19:33.272 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0013771580000000001 2024-09-17T07:19:33.272 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.272 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.273 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:19:33.273 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.508754+0000", 2024-09-17T07:19:33.273 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0067232330000000003 2024-09-17T07:19:33.273 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:33.274 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:33.274 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:19:33.275 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.275 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:19:33.275 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.508791+0000", 2024-09-17T07:19:33.275 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.7017000000000002e-05 2024-09-17T07:19:33.275 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.275 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.275 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:19:33.275 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.508877+0000", 2024-09-17T07:19:33.276 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.5159000000000002e-05 2024-09-17T07:19:33.276 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:33.276 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:19:33.276 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:33.276 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.276 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.276 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.15648.0:216994 10.22 10:47b4d807:::benchmark_data_smithi100_155039_object93646:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e368)", 2024-09-17T07:19:33.276 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:19:13.874186+0000", 2024-09-17T07:19:33.276 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 19.329442337, 2024-09-17T07:19:33.276 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.63482440200000001, 2024-09-17T07:19:33.276 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:19:33.276 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:19:33.277 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:19:33.277 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.15648", 2024-09-17T07:19:33.277 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/1684940078", 2024-09-17T07:19:33.277 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 216994 2024-09-17T07:19:33.277 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.277 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:19:33.277 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.277 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:19:33.277 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.874186+0000", 2024-09-17T07:19:33.277 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:33.277 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.278 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.278 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:19:33.278 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.874186+0000", 2024-09-17T07:19:33.278 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:33.278 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.278 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.278 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:19:33.278 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.874184+0000", 2024-09-17T07:19:33.278 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:19:33.278 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.278 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.278 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:19:33.279 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.874190+0000", 2024-09-17T07:19:33.279 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.8440000000000003e-06 2024-09-17T07:19:33.279 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.279 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.279 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:19:33.279 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.874199+0000", 2024-09-17T07:19:33.279 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.6249999999999996e-06 2024-09-17T07:19:33.279 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.279 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.279 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:19:33.280 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.874205+0000", 2024-09-17T07:19:33.280 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.8799999999999996e-06 2024-09-17T07:19:33.280 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.280 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.280 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:33.280 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.875438+0000", 2024-09-17T07:19:33.280 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0012332280000000001 2024-09-17T07:19:33.280 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.280 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.280 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:19:33.280 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.875446+0000", 2024-09-17T07:19:33.280 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.4500000000000004e-06 2024-09-17T07:19:33.281 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.281 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.281 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:33.281 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.496542+0000", 2024-09-17T07:19:33.281 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.62109532099999998 2024-09-17T07:19:33.281 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.281 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.281 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:19:33.281 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.496638+0000", 2024-09-17T07:19:33.281 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.6859000000000002e-05 2024-09-17T07:19:33.281 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.281 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.281 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,7", 2024-09-17T07:19:33.281 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.496780+0000", 2024-09-17T07:19:33.282 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0001418 2024-09-17T07:19:33.282 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.282 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.282 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:33.282 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.500682+0000", 2024-09-17T07:19:33.282 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.003901884 2024-09-17T07:19:33.282 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.282 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.282 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:33.282 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.502060+0000", 2024-09-17T07:19:33.282 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0013780789999999999 2024-09-17T07:19:33.282 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.282 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.283 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:19:33.283 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.508883+0000", 2024-09-17T07:19:33.283 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0068228459999999996 2024-09-17T07:19:33.283 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.283 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.283 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:19:33.283 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.508928+0000", 2024-09-17T07:19:33.283 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.5025999999999999e-05 2024-09-17T07:19:33.283 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.283 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.283 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:19:33.283 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.509010+0000", 2024-09-17T07:19:33.283 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.2026000000000004e-05 2024-09-17T07:19:33.284 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:33.284 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:19:33.284 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:33.284 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.284 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.284 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.15648.0:217030 10.22 10:471e27a8:::benchmark_data_smithi100_155039_object93682:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e368)", 2024-09-17T07:19:33.284 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:19:13.902383+0000", 2024-09-17T07:19:33.284 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 19.301244833999998, 2024-09-17T07:19:33.284 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.60674819300000005, 2024-09-17T07:19:33.284 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:19:33.284 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:19:33.284 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:19:33.284 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.15648", 2024-09-17T07:19:33.285 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/1684940078", 2024-09-17T07:19:33.285 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 217030 2024-09-17T07:19:33.285 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.285 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:19:33.285 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.285 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:19:33.285 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.902383+0000", 2024-09-17T07:19:33.285 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:33.285 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.285 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.285 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:19:33.285 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.902383+0000", 2024-09-17T07:19:33.285 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:33.285 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.286 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.286 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:19:33.286 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.902382+0000", 2024-09-17T07:19:33.286 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.999999 2024-09-17T07:19:33.286 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.286 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.286 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:19:33.286 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.902387+0000", 2024-09-17T07:19:33.286 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.2599999999999996e-06 2024-09-17T07:19:33.286 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.286 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.286 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:19:33.286 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.902408+0000", 2024-09-17T07:19:33.286 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.0117000000000001e-05 2024-09-17T07:19:33.286 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.287 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.287 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:19:33.287 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.902414+0000", 2024-09-17T07:19:33.287 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.861e-06 2024-09-17T07:19:33.287 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.287 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.287 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:33.287 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.902457+0000", 2024-09-17T07:19:33.287 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.2831999999999999e-05 2024-09-17T07:19:33.287 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.287 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.287 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:19:33.287 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.902467+0000", 2024-09-17T07:19:33.287 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.3430000000000002e-06 2024-09-17T07:19:33.288 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.288 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.288 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:33.288 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.500702+0000", 2024-09-17T07:19:33.288 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.59823563499999999 2024-09-17T07:19:33.288 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.288 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.288 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:19:33.288 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.500820+0000", 2024-09-17T07:19:33.288 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000117497 2024-09-17T07:19:33.288 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.288 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.288 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,7", 2024-09-17T07:19:33.288 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.501041+0000", 2024-09-17T07:19:33.289 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00022175100000000001 2024-09-17T07:19:33.289 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.289 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.289 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:33.289 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.503163+0000", 2024-09-17T07:19:33.289 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0021216920000000001 2024-09-17T07:19:33.289 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.289 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.289 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:33.289 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.504199+0000", 2024-09-17T07:19:33.289 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0010363130000000001 2024-09-17T07:19:33.289 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.290 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.290 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:19:33.290 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.509017+0000", 2024-09-17T07:19:33.290 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0048173820000000003 2024-09-17T07:19:33.290 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.290 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.290 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:19:33.290 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.509055+0000", 2024-09-17T07:19:33.290 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.8393000000000003e-05 2024-09-17T07:19:33.290 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.290 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.290 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:19:33.290 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.509131+0000", 2024-09-17T07:19:33.291 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.6087999999999994e-05 2024-09-17T07:19:33.291 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:33.291 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:19:33.291 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:33.291 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.291 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.291 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.15648.0:217035 10.22 10:46f08d29:::benchmark_data_smithi100_155039_object93687:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e368)", 2024-09-17T07:19:33.291 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:19:13.906452+0000", 2024-09-17T07:19:33.291 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 19.297176125, 2024-09-17T07:19:33.291 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.60279918799999999, 2024-09-17T07:19:33.291 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:19:33.291 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:19:33.291 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:19:33.292 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.15648", 2024-09-17T07:19:33.292 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/1684940078", 2024-09-17T07:19:33.292 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 217035 2024-09-17T07:19:33.292 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.292 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:19:33.292 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.292 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:19:33.292 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.906452+0000", 2024-09-17T07:19:33.292 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:33.292 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.293 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.293 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:19:33.293 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.906452+0000", 2024-09-17T07:19:33.293 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:33.293 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.293 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.293 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:19:33.294 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.906452+0000", 2024-09-17T07:19:33.294 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999995 2024-09-17T07:19:33.294 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.294 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.294 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:19:33.294 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.906454+0000", 2024-09-17T07:19:33.294 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.0559999999999999e-06 2024-09-17T07:19:33.294 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.294 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.294 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:19:33.294 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.906457+0000", 2024-09-17T07:19:33.294 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.8530000000000002e-06 2024-09-17T07:19:33.294 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.295 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.295 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:19:33.295 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.906461+0000", 2024-09-17T07:19:33.295 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.1669999999999999e-06 2024-09-17T07:19:33.295 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.295 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.295 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:33.295 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.906512+0000", 2024-09-17T07:19:33.295 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.0933999999999997e-05 2024-09-17T07:19:33.295 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.295 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.295 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:19:33.295 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.906521+0000", 2024-09-17T07:19:33.295 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.4909999999999994e-06 2024-09-17T07:19:33.296 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.296 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.296 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:33.296 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.502081+0000", 2024-09-17T07:19:33.296 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.59556035600000001 2024-09-17T07:19:33.296 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.296 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.296 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:19:33.296 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.502190+0000", 2024-09-17T07:19:33.296 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00010862300000000001 2024-09-17T07:19:33.296 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.296 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.297 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,7", 2024-09-17T07:19:33.297 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.502353+0000", 2024-09-17T07:19:33.297 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00016272699999999999 2024-09-17T07:19:33.297 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.297 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.297 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:33.297 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.504170+0000", 2024-09-17T07:19:33.297 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0018171280000000001 2024-09-17T07:19:33.297 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.297 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.297 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:33.297 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.508323+0000", 2024-09-17T07:19:33.298 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0041533610000000004 2024-09-17T07:19:33.298 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.298 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.298 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:19:33.298 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.509138+0000", 2024-09-17T07:19:33.298 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00081465599999999997 2024-09-17T07:19:33.298 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.298 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.298 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:19:33.298 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.509172+0000", 2024-09-17T07:19:33.298 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.4115000000000001e-05 2024-09-17T07:19:33.298 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.298 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.298 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:19:33.299 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.509251+0000", 2024-09-17T07:19:33.299 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.9067999999999996e-05 2024-09-17T07:19:33.299 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:33.299 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:19:33.299 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:33.299 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.299 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.299 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.15648.0:217036 10.22 10:46c84197:::benchmark_data_smithi100_155039_object93688:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e368)", 2024-09-17T07:19:33.299 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:19:13.906471+0000", 2024-09-17T07:19:33.299 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 19.29715663, 2024-09-17T07:19:33.299 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.60290220699999997, 2024-09-17T07:19:33.299 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:19:33.300 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:19:33.300 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:19:33.300 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.15648", 2024-09-17T07:19:33.300 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/1684940078", 2024-09-17T07:19:33.300 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 217036 2024-09-17T07:19:33.300 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.300 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:19:33.300 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.300 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:19:33.300 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.906471+0000", 2024-09-17T07:19:33.301 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:33.301 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.301 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.301 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:19:33.301 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.906471+0000", 2024-09-17T07:19:33.301 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:33.301 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.301 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.301 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:19:33.301 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.906471+0000", 2024-09-17T07:19:33.301 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999995 2024-09-17T07:19:33.302 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.302 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.302 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:19:33.302 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.906473+0000", 2024-09-17T07:19:33.302 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.9139999999999998e-06 2024-09-17T07:19:33.302 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.302 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.302 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:19:33.302 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.906476+0000", 2024-09-17T07:19:33.302 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.1149999999999998e-06 2024-09-17T07:19:33.302 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.302 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.303 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:19:33.303 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.906479+0000", 2024-09-17T07:19:33.303 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.8990000000000001e-06 2024-09-17T07:19:33.303 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.303 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.303 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:33.303 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.906542+0000", 2024-09-17T07:19:33.303 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.3406999999999998e-05 2024-09-17T07:19:33.303 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.303 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.303 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:19:33.303 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.906549+0000", 2024-09-17T07:19:33.304 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.3840000000000002e-06 2024-09-17T07:19:33.304 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.304 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.304 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:33.304 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.503185+0000", 2024-09-17T07:19:33.304 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.59663592700000001 2024-09-17T07:19:33.304 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.305 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.305 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:19:33.305 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.503284+0000", 2024-09-17T07:19:33.305 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.9506000000000006e-05 2024-09-17T07:19:33.305 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.305 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.305 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,7", 2024-09-17T07:19:33.305 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.503452+0000", 2024-09-17T07:19:33.305 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00016766799999999999 2024-09-17T07:19:33.305 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.306 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.306 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:33.306 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.505352+0000", 2024-09-17T07:19:33.306 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.001900005 2024-09-17T07:19:33.306 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.306 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.306 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:33.306 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.508355+0000", 2024-09-17T07:19:33.307 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0030029480000000001 2024-09-17T07:19:33.307 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.307 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.307 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:19:33.307 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.509257+0000", 2024-09-17T07:19:33.307 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00090198399999999999 2024-09-17T07:19:33.307 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.307 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.307 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:19:33.308 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.509291+0000", 2024-09-17T07:19:33.308 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.4338999999999998e-05 2024-09-17T07:19:33.308 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.308 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.308 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:19:33.308 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.509374+0000", 2024-09-17T07:19:33.309 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.2441000000000006e-05 2024-09-17T07:19:33.309 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:33.309 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:19:33.309 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:33.309 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.309 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.310 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.15648.0:217073 10.22 10:47e5d21e:::benchmark_data_smithi100_155039_object93725:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e368)", 2024-09-17T07:19:33.310 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:19:13.946856+0000", 2024-09-17T07:19:33.310 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 19.256771484000001, 2024-09-17T07:19:33.310 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.56381954499999998, 2024-09-17T07:19:33.310 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:19:33.310 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:19:33.311 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:19:33.311 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.15648", 2024-09-17T07:19:33.311 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/1684940078", 2024-09-17T07:19:33.311 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 217073 2024-09-17T07:19:33.311 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.311 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:19:33.312 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.312 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:19:33.312 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.946856+0000", 2024-09-17T07:19:33.312 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:33.312 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.313 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.313 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:19:33.313 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.946856+0000", 2024-09-17T07:19:33.313 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:33.313 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.313 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.314 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:19:33.314 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.946855+0000", 2024-09-17T07:19:33.314 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999981 2024-09-17T07:19:33.314 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.314 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.314 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:19:33.315 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.946862+0000", 2024-09-17T07:19:33.315 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.4209999999999998e-06 2024-09-17T07:19:33.315 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.315 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.315 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:19:33.316 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.946884+0000", 2024-09-17T07:19:33.316 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.1769000000000001e-05 2024-09-17T07:19:33.316 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.316 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.316 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:19:33.317 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.946892+0000", 2024-09-17T07:19:33.317 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.551e-06 2024-09-17T07:19:33.317 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.317 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.317 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:33.317 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.946943+0000", 2024-09-17T07:19:33.318 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.0698000000000001e-05 2024-09-17T07:19:33.318 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.318 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.318 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:19:33.318 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:13.946954+0000", 2024-09-17T07:19:33.319 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0732e-05 2024-09-17T07:19:33.319 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.319 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.319 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:33.319 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.504233+0000", 2024-09-17T07:19:33.319 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.55727933799999996 2024-09-17T07:19:33.320 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.320 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.320 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:19:33.320 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.504345+0000", 2024-09-17T07:19:33.320 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00011171300000000001 2024-09-17T07:19:33.320 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.321 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.321 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,7", 2024-09-17T07:19:33.321 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.504522+0000", 2024-09-17T07:19:33.321 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000177262 2024-09-17T07:19:33.321 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.321 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.321 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:33.322 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.506353+0000", 2024-09-17T07:19:33.322 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.001830417 2024-09-17T07:19:33.322 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.322 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.322 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:19:33.322 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.509380+0000", 2024-09-17T07:19:33.323 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.003027793 2024-09-17T07:19:33.323 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.323 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.323 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:33.323 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.510522+0000", 2024-09-17T07:19:33.324 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.001141204 2024-09-17T07:19:33.324 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.324 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.324 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:19:33.324 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.510570+0000", 2024-09-17T07:19:33.325 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.8319000000000003e-05 2024-09-17T07:19:33.325 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.325 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.325 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:19:33.325 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.510676+0000", 2024-09-17T07:19:33.325 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000106073 2024-09-17T07:19:33.326 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:33.326 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:19:33.326 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:33.326 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.326 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.326 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.15648.0:217118 10.22 10:4443aa95:::benchmark_data_smithi100_155039_object93770:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e368)", 2024-09-17T07:19:33.327 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:19:14.004322+0000", 2024-09-17T07:19:33.327 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 19.199306268000001, 2024-09-17T07:19:33.327 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.50653526400000004, 2024-09-17T07:19:33.327 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:19:33.327 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:19:33.328 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:19:33.328 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.15648", 2024-09-17T07:19:33.328 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/1684940078", 2024-09-17T07:19:33.328 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 217118 2024-09-17T07:19:33.328 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.329 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:19:33.329 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.329 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:19:33.329 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.004322+0000", 2024-09-17T07:19:33.329 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:33.330 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.330 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.330 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:19:33.330 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.004322+0000", 2024-09-17T07:19:33.330 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:33.331 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.331 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.331 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:19:33.331 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.004320+0000", 2024-09-17T07:19:33.331 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:19:33.331 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.332 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.332 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:19:33.332 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.004326+0000", 2024-09-17T07:19:33.332 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.913e-06 2024-09-17T07:19:33.332 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.333 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.333 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:19:33.333 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.004340+0000", 2024-09-17T07:19:33.333 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.4048e-05 2024-09-17T07:19:33.333 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.334 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.334 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:19:33.334 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.004348+0000", 2024-09-17T07:19:33.334 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.7009999999999996e-06 2024-09-17T07:19:33.334 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.334 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.335 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:33.335 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.004386+0000", 2024-09-17T07:19:33.335 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.8330000000000001e-05 2024-09-17T07:19:33.335 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.335 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.335 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:19:33.336 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.004392+0000", 2024-09-17T07:19:33.336 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.0050000000000001e-06 2024-09-17T07:19:33.336 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.336 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.336 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:33.336 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.505372+0000", 2024-09-17T07:19:33.336 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.50097933299999997 2024-09-17T07:19:33.337 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.337 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.337 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:19:33.337 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.505477+0000", 2024-09-17T07:19:33.337 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000105473 2024-09-17T07:19:33.337 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.337 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.337 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,7", 2024-09-17T07:19:33.338 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.505622+0000", 2024-09-17T07:19:33.338 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00014475499999999999 2024-09-17T07:19:33.338 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.338 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.338 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:33.338 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.508385+0000", 2024-09-17T07:19:33.338 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0027633639999999999 2024-09-17T07:19:33.338 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.339 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.339 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:19:33.339 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.509386+0000", 2024-09-17T07:19:33.339 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0010006030000000001 2024-09-17T07:19:33.339 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.339 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.339 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:33.339 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.510717+0000", 2024-09-17T07:19:33.340 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0013307779999999999 2024-09-17T07:19:33.340 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.340 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.340 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:19:33.340 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.510768+0000", 2024-09-17T07:19:33.340 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.1295000000000003e-05 2024-09-17T07:19:33.340 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.340 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.341 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:19:33.341 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.510857+0000", 2024-09-17T07:19:33.341 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.9058e-05 2024-09-17T07:19:33.341 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:33.341 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:19:33.341 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:33.341 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.341 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.341 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.15648.0:217132 10.22 10:446f5ac6:::benchmark_data_smithi100_155039_object93784:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e368)", 2024-09-17T07:19:33.341 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:19:14.030162+0000", 2024-09-17T07:19:33.342 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 19.173465805999999, 2024-09-17T07:19:33.342 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.48133480200000001, 2024-09-17T07:19:33.342 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:19:33.342 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:19:33.342 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:19:33.342 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.15648", 2024-09-17T07:19:33.343 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/1684940078", 2024-09-17T07:19:33.343 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 217132 2024-09-17T07:19:33.343 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.343 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:19:33.343 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.343 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:19:33.344 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.030162+0000", 2024-09-17T07:19:33.344 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:33.344 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.344 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.344 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:19:33.344 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.030162+0000", 2024-09-17T07:19:33.344 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:33.344 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.345 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.345 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:19:33.345 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.030160+0000", 2024-09-17T07:19:33.345 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999981 2024-09-17T07:19:33.345 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.345 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.345 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:19:33.345 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.030166+0000", 2024-09-17T07:19:33.345 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.0140000000000004e-06 2024-09-17T07:19:33.345 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.345 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.346 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:19:33.346 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.030178+0000", 2024-09-17T07:19:33.346 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.1626e-05 2024-09-17T07:19:33.346 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.346 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.347 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:19:33.347 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.030184+0000", 2024-09-17T07:19:33.347 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.1870000000000002e-06 2024-09-17T07:19:33.347 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.347 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.347 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:33.348 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.030229+0000", 2024-09-17T07:19:33.348 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.5175e-05 2024-09-17T07:19:33.348 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.348 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.348 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:19:33.348 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.030239+0000", 2024-09-17T07:19:33.348 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.7440000000000002e-06 2024-09-17T07:19:33.348 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.349 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.349 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:33.349 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.506374+0000", 2024-09-17T07:19:33.349 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.47613446999999998 2024-09-17T07:19:33.349 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.349 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.349 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:19:33.349 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.506481+0000", 2024-09-17T07:19:33.349 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00010765899999999999 2024-09-17T07:19:33.349 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.349 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.349 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,7", 2024-09-17T07:19:33.350 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.506626+0000", 2024-09-17T07:19:33.350 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000144606 2024-09-17T07:19:33.350 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.350 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.350 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:19:33.350 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.509391+0000", 2024-09-17T07:19:33.350 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0027650830000000002 2024-09-17T07:19:33.350 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.350 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.351 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:33.351 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.510461+0000", 2024-09-17T07:19:33.351 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.001069801 2024-09-17T07:19:33.351 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.351 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.351 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:33.351 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.511362+0000", 2024-09-17T07:19:33.351 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00090144800000000001 2024-09-17T07:19:33.351 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.351 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.351 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:19:33.351 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.511405+0000", 2024-09-17T07:19:33.351 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.2583999999999998e-05 2024-09-17T07:19:33.352 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.352 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.352 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:19:33.352 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.511497+0000", 2024-09-17T07:19:33.352 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.2128999999999998e-05 2024-09-17T07:19:33.352 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:33.352 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:19:33.352 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:33.352 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.352 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.352 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.15648.0:228582 10.3 10:c0d88466:::benchmark_data_smithi100_155039_object105234:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-09-17T07:19:33.352 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:19:23.333957+0000", 2024-09-17T07:19:33.353 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 9.8696712190000007, 2024-09-17T07:19:33.353 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.088143241999999997, 2024-09-17T07:19:33.353 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:19:33.353 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:19:33.353 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:19:33.353 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.15648", 2024-09-17T07:19:33.353 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/1684940078", 2024-09-17T07:19:33.354 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 228582 2024-09-17T07:19:33.354 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.354 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:19:33.354 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.354 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:19:33.355 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:23.333957+0000", 2024-09-17T07:19:33.355 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:33.355 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.355 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.355 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:19:33.355 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:23.333957+0000", 2024-09-17T07:19:33.355 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:33.356 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.356 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.356 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:19:33.356 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:23.333956+0000", 2024-09-17T07:19:33.356 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.999999 2024-09-17T07:19:33.357 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.357 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.357 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:19:33.357 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:23.333960+0000", 2024-09-17T07:19:33.357 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.279e-06 2024-09-17T07:19:33.357 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.358 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.358 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:19:33.358 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:23.333967+0000", 2024-09-17T07:19:33.358 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.9600000000000003e-06 2024-09-17T07:19:33.358 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.358 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.358 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:19:33.358 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:23.333973+0000", 2024-09-17T07:19:33.359 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.7699999999999998e-06 2024-09-17T07:19:33.359 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.359 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.359 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:33.359 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:23.334023+0000", 2024-09-17T07:19:33.359 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.9818999999999999e-05 2024-09-17T07:19:33.359 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.359 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.359 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:19:33.359 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:23.334181+0000", 2024-09-17T07:19:33.360 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00015838199999999999 2024-09-17T07:19:33.360 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.360 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.360 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 2,4", 2024-09-17T07:19:33.360 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:23.334449+0000", 2024-09-17T07:19:33.360 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000267731 2024-09-17T07:19:33.360 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.360 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.360 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:33.360 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:23.336098+0000", 2024-09-17T07:19:33.360 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0016494140000000001 2024-09-17T07:19:33.360 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.361 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.361 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:19:33.361 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:23.336116+0000", 2024-09-17T07:19:33.361 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.7893999999999998e-05 2024-09-17T07:19:33.361 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.361 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.361 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:33.361 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:23.422019+0000", 2024-09-17T07:19:33.361 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.085903010000000002 2024-09-17T07:19:33.361 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.362 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.362 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:19:33.362 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:23.422063+0000", 2024-09-17T07:19:33.362 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.4063000000000001e-05 2024-09-17T07:19:33.362 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.362 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.363 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:19:33.363 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:23.422100+0000", 2024-09-17T07:19:33.363 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.6695e-05 2024-09-17T07:19:33.363 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:33.363 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:19:33.363 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:33.363 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.363 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.363 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.15648.0:229013 10.3 10:c2bb632c:::benchmark_data_smithi100_155039_object105665:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-09-17T07:19:33.363 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:19:23.664794+0000", 2024-09-17T07:19:33.363 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 9.5388341879999992, 2024-09-17T07:19:33.363 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.049553350000000003, 2024-09-17T07:19:33.363 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:19:33.363 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:19:33.364 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:19:33.364 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.15648", 2024-09-17T07:19:33.364 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/1684940078", 2024-09-17T07:19:33.364 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 229013 2024-09-17T07:19:33.364 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.364 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:19:33.364 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.364 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:19:33.364 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:23.664794+0000", 2024-09-17T07:19:33.365 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:33.365 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.365 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.365 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:19:33.365 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:23.664794+0000", 2024-09-17T07:19:33.365 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:33.365 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.365 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.365 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:19:33.365 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:23.664792+0000", 2024-09-17T07:19:33.365 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:19:33.365 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.365 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.366 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:19:33.366 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:23.664799+0000", 2024-09-17T07:19:33.366 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.5889999999999999e-06 2024-09-17T07:19:33.366 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.366 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.366 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:19:33.366 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:23.664808+0000", 2024-09-17T07:19:33.366 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.5529999999999993e-06 2024-09-17T07:19:33.366 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.367 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.367 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:19:33.367 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:23.664813+0000", 2024-09-17T07:19:33.367 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.5210000000000002e-06 2024-09-17T07:19:33.367 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.367 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.367 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:33.367 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:23.664856+0000", 2024-09-17T07:19:33.367 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.3268e-05 2024-09-17T07:19:33.367 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.367 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.368 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:19:33.368 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:23.664958+0000", 2024-09-17T07:19:33.368 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00010181199999999999 2024-09-17T07:19:33.368 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.368 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.368 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 2,4", 2024-09-17T07:19:33.368 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:23.665415+0000", 2024-09-17T07:19:33.368 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00045703200000000001 2024-09-17T07:19:33.368 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.368 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.368 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:33.369 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:23.667017+0000", 2024-09-17T07:19:33.369 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0016019809999999999 2024-09-17T07:19:33.369 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.369 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.369 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:19:33.369 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:23.667073+0000", 2024-09-17T07:19:33.369 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.5668999999999998e-05 2024-09-17T07:19:33.369 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.369 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.369 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:33.369 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:23.714167+0000", 2024-09-17T07:19:33.369 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.047093628999999998 2024-09-17T07:19:33.370 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.370 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.370 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:19:33.370 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:23.714306+0000", 2024-09-17T07:19:33.370 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000138999 2024-09-17T07:19:33.370 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.370 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.370 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:19:33.370 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:23.714347+0000", 2024-09-17T07:19:33.370 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.1594000000000002e-05 2024-09-17T07:19:33.370 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:33.370 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:19:33.370 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:33.371 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.371 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.371 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.15648.0:229018 10.6 10:643bc472:::benchmark_data_smithi100_155039_object105670:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-09-17T07:19:33.371 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:19:23.669981+0000", 2024-09-17T07:19:33.371 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 9.5336466830000006, 2024-09-17T07:19:33.371 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.034467705000000001, 2024-09-17T07:19:33.371 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:19:33.371 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:19:33.371 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:19:33.371 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.15648", 2024-09-17T07:19:33.371 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/1684940078", 2024-09-17T07:19:33.371 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 229018 2024-09-17T07:19:33.372 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.372 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:19:33.372 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.372 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:19:33.372 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:23.669981+0000", 2024-09-17T07:19:33.372 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:33.372 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.372 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.372 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:19:33.372 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:23.669981+0000", 2024-09-17T07:19:33.372 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:33.372 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.373 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.373 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:19:33.373 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:23.669981+0000", 2024-09-17T07:19:33.373 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.999999 2024-09-17T07:19:33.373 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.373 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.373 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:19:33.373 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:23.669984+0000", 2024-09-17T07:19:33.373 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.8999999999999999e-06 2024-09-17T07:19:33.373 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.373 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.373 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:19:33.373 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:23.669991+0000", 2024-09-17T07:19:33.374 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.4699999999999999e-06 2024-09-17T07:19:33.374 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.374 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.374 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:19:33.374 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:23.669995+0000", 2024-09-17T07:19:33.374 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.155e-06 2024-09-17T07:19:33.374 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.374 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.374 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:33.374 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:23.670029+0000", 2024-09-17T07:19:33.374 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.3589e-05 2024-09-17T07:19:33.374 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.374 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.375 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:19:33.375 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:23.670104+0000", 2024-09-17T07:19:33.375 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.5581999999999996e-05 2024-09-17T07:19:33.375 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.375 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.375 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,4", 2024-09-17T07:19:33.375 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:23.670166+0000", 2024-09-17T07:19:33.375 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.1943e-05 2024-09-17T07:19:33.375 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.375 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.375 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:19:33.375 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:23.672109+0000", 2024-09-17T07:19:33.375 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0019430269999999999 2024-09-17T07:19:33.376 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.376 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.376 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:33.376 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:23.672196+0000", 2024-09-17T07:19:33.376 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.6700999999999995e-05 2024-09-17T07:19:33.376 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.376 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.376 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:33.376 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:23.704355+0000", 2024-09-17T07:19:33.376 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.032159563000000002 2024-09-17T07:19:33.376 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.376 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.376 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:19:33.377 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:23.704409+0000", 2024-09-17T07:19:33.377 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.3049e-05 2024-09-17T07:19:33.377 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.377 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.377 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:19:33.377 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:23.704449+0000", 2024-09-17T07:19:33.377 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.0494000000000002e-05 2024-09-17T07:19:33.377 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:33.378 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:19:33.378 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:33.378 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.378 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.378 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.15648.0:230511 10.3 10:c356621d:::benchmark_data_smithi100_155039_object107163:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-09-17T07:19:33.378 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:19:24.608861+0000", 2024-09-17T07:19:33.378 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 8.5947668129999997, 2024-09-17T07:19:33.378 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.076891767, 2024-09-17T07:19:33.378 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:19:33.378 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:19:33.379 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:19:33.379 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.15648", 2024-09-17T07:19:33.379 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/1684940078", 2024-09-17T07:19:33.379 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 230511 2024-09-17T07:19:33.379 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.379 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:19:33.379 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.379 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:19:33.379 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:24.608861+0000", 2024-09-17T07:19:33.379 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:33.379 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.379 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.379 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:19:33.380 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:24.608861+0000", 2024-09-17T07:19:33.380 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:33.380 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.380 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.380 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:19:33.380 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:24.608860+0000", 2024-09-17T07:19:33.380 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.999999 2024-09-17T07:19:33.380 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.380 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.380 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:19:33.381 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:24.608866+0000", 2024-09-17T07:19:33.381 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.6590000000000004e-06 2024-09-17T07:19:33.381 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.381 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.381 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:19:33.381 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:24.608876+0000", 2024-09-17T07:19:33.381 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.048e-05 2024-09-17T07:19:33.381 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.381 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.381 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:19:33.381 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:24.608883+0000", 2024-09-17T07:19:33.381 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.7920000000000004e-06 2024-09-17T07:19:33.382 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.382 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.382 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:33.382 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:24.608929+0000", 2024-09-17T07:19:33.382 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.5673999999999997e-05 2024-09-17T07:19:33.382 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.382 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.382 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:19:33.382 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:24.609030+0000", 2024-09-17T07:19:33.382 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000101368 2024-09-17T07:19:33.382 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.382 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.382 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 2,4", 2024-09-17T07:19:33.383 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:24.609261+0000", 2024-09-17T07:19:33.383 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00023114899999999999 2024-09-17T07:19:33.383 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.383 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.383 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:19:33.383 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:24.610718+0000", 2024-09-17T07:19:33.383 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.001456319 2024-09-17T07:19:33.383 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.383 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.383 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:33.383 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:24.612280+0000", 2024-09-17T07:19:33.383 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0015627309999999999 2024-09-17T07:19:33.383 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.384 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.384 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:33.384 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:24.685653+0000", 2024-09-17T07:19:33.384 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.073372362999999996 2024-09-17T07:19:33.384 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.384 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.384 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:19:33.384 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:24.685718+0000", 2024-09-17T07:19:33.384 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.5811999999999999e-05 2024-09-17T07:19:33.384 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.384 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.384 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:19:33.385 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:24.685753+0000", 2024-09-17T07:19:33.385 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.4452000000000002e-05 2024-09-17T07:19:33.385 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:33.385 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:19:33.385 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:33.385 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.385 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.385 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.15648.0:241987 10.9 10:91b444d8:::benchmark_data_smithi100_155039_object118639:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e376)", 2024-09-17T07:19:33.385 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:19:32.681611+0000", 2024-09-17T07:19:33.385 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 0.52201688899999998, 2024-09-17T07:19:33.385 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.032129816999999998, 2024-09-17T07:19:33.385 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:19:33.385 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:19:33.386 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:19:33.386 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.15648", 2024-09-17T07:19:33.386 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/1684940078", 2024-09-17T07:19:33.386 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 241987 2024-09-17T07:19:33.386 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.386 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:19:33.386 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.386 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:19:33.386 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:32.681611+0000", 2024-09-17T07:19:33.386 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:33.386 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.386 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.386 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:19:33.387 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:32.681611+0000", 2024-09-17T07:19:33.387 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:33.387 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.387 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.387 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:19:33.387 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:32.681610+0000", 2024-09-17T07:19:33.387 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.999999 2024-09-17T07:19:33.387 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.387 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.387 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:19:33.388 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:32.681615+0000", 2024-09-17T07:19:33.388 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.9690000000000002e-06 2024-09-17T07:19:33.388 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.388 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.388 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:19:33.388 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:32.681624+0000", 2024-09-17T07:19:33.388 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.9150000000000002e-06 2024-09-17T07:19:33.388 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.388 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.388 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:19:33.388 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:32.681631+0000", 2024-09-17T07:19:33.389 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.6239999999999996e-06 2024-09-17T07:19:33.389 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.389 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.389 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:33.389 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:32.688499+0000", 2024-09-17T07:19:33.389 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0068683679999999997 2024-09-17T07:19:33.389 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.389 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.389 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:19:33.389 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:32.688789+0000", 2024-09-17T07:19:33.389 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00028996499999999999 2024-09-17T07:19:33.389 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.390 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.390 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 2,7", 2024-09-17T07:19:33.390 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:32.689501+0000", 2024-09-17T07:19:33.390 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00071149400000000002 2024-09-17T07:19:33.390 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.390 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.390 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:33.390 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:32.697963+0000", 2024-09-17T07:19:33.390 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0084625350000000002 2024-09-17T07:19:33.390 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.390 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.390 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:19:33.390 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:32.698039+0000", 2024-09-17T07:19:33.391 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.6086000000000003e-05 2024-09-17T07:19:33.391 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.391 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.391 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:33.391 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:32.713663+0000", 2024-09-17T07:19:33.391 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.015623981 2024-09-17T07:19:33.391 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.391 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.391 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:19:33.391 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:32.713707+0000", 2024-09-17T07:19:33.391 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.4144999999999998e-05 2024-09-17T07:19:33.391 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:33.391 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:33.391 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:19:33.392 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:32.713741+0000", 2024-09-17T07:19:33.392 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.3562000000000002e-05 2024-09-17T07:19:33.392 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:33.392 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:19:33.392 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:33.392 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:33.392 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:19:33.392 INFO:teuthology.orchestra.run.smithi135.stdout:} 2024-09-17T07:19:33.392 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:19:33.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:33.414 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:33.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:19:33.515 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:19:33.542 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:33.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:33.543 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:19:33.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:33 smithi135 ceph-mon[33238]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T07:19:33.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:33 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T07:19:33.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:33 smithi135 ceph-mon[33238]: osdmap e376: 8 total, 7 up, 7 in 2024-09-17T07:19:33.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:33 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:19:33.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:33 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:19:33.584 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:33 smithi100 ceph-mon[29092]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T07:19:33.584 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:33 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T07:19:33.585 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:33 smithi100 ceph-mon[29092]: osdmap e376: 8 total, 7 up, 7 in 2024-09-17T07:19:33.585 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:33 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:19:33.585 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:33 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:19:33.585 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:33 smithi100 ceph-mon[39462]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T07:19:33.585 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:33 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T07:19:33.585 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:33 smithi100 ceph-mon[39462]: osdmap e376: 8 total, 7 up, 7 in 2024-09-17T07:19:33.585 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:33 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:19:33.585 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:33 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:19:33.644 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:19:33.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:33.680 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:33.681 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:19:33.782 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:19:33.809 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:33.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:33.810 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:19:33.911 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:19:33.937 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:33.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:33.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:19:34.039 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:19:34.077 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:34.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:34.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:19:34.179 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:19:34.218 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:34.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:34.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:19:34.320 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:19:34.362 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:34.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:34.363 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:19:34.464 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:19:34.501 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:34.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:34.502 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:19:34.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:34 smithi135 ceph-mon[33238]: pgmap v804: 178 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 remapped+peering, 8 unknown, 11 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 92 active+clean; 457 MiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 926 op/s; 926/22512 objects degraded (4.113%); 2200/22512 objects misplaced (9.773%) 2024-09-17T07:19:34.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:34 smithi135 ceph-mon[33238]: osdmap e377: 8 total, 7 up, 7 in 2024-09-17T07:19:34.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:34 smithi135 ceph-mon[33238]: 2.13 scrub starts 2024-09-17T07:19:34.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:34 smithi135 ceph-mon[33238]: 2.13 scrub ok 2024-09-17T07:19:34.603 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:19:34.629 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:34.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:34.630 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:19:34.731 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:19:34.754 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:34.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:34.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:19:34.775 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-17T07:19:34.855 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:19:34.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:34 smithi100 ceph-mon[29092]: pgmap v804: 178 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 remapped+peering, 8 unknown, 11 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 92 active+clean; 457 MiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 926 op/s; 926/22512 objects degraded (4.113%); 2200/22512 objects misplaced (9.773%) 2024-09-17T07:19:34.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:34 smithi100 ceph-mon[29092]: osdmap e377: 8 total, 7 up, 7 in 2024-09-17T07:19:34.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:34 smithi100 ceph-mon[29092]: 2.13 scrub starts 2024-09-17T07:19:34.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:34 smithi100 ceph-mon[29092]: 2.13 scrub ok 2024-09-17T07:19:34.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:34 smithi100 ceph-mon[39462]: pgmap v804: 178 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 remapped+peering, 8 unknown, 11 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 92 active+clean; 457 MiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 926 op/s; 926/22512 objects degraded (4.113%); 2200/22512 objects misplaced (9.773%) 2024-09-17T07:19:34.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:34 smithi100 ceph-mon[39462]: osdmap e377: 8 total, 7 up, 7 in 2024-09-17T07:19:34.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:34 smithi100 ceph-mon[39462]: 2.13 scrub starts 2024-09-17T07:19:34.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:34 smithi100 ceph-mon[39462]: 2.13 scrub ok 2024-09-17T07:19:34.894 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:34.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:34.895 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:19:34.996 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:19:35.051 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:35.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:35.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:19:35.150 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:19:35.153 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:19:35.177 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:35.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:35.178 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:19:35.207 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:19:35.260 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7] out_osds: [3] dead_osds: [1] live_osds: [0, 2, 3, 4, 5, 6, 7] 2024-09-17T07:19:35.260 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T07:19:35.260 INFO:tasks.thrashosds.thrasher:Removing osd 7, in_osds are: [0, 1, 2, 4, 5, 6, 7] 2024-09-17T07:19:35.260 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd out 7 2024-09-17T07:19:35.278 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:19:35.319 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:35.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:35.320 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:19:35.421 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:19:35.459 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:35.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:35.460 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:19:35.536 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:19:35.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:35 smithi135 ceph-mon[33238]: osdmap e378: 8 total, 7 up, 7 in 2024-09-17T07:19:35.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:35 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 926/22512 objects degraded (4.113%), 29 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-09-17T07:19:35.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:35 smithi135 ceph-mon[33238]: pgmap v807: 178 pgs: 3 active+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 2 remapped+peering, 5 unknown, 11 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 92 active+clean; 212 MiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 1.69k op/s; 427/10746 objects degraded (3.974%); 1487/10746 objects misplaced (13.838%) 2024-09-17T07:19:35.561 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:19:35.575 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:19:35.598 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:35.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:35.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:19:35.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:35 smithi100 ceph-mon[29092]: osdmap e378: 8 total, 7 up, 7 in 2024-09-17T07:19:35.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:35 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 926/22512 objects degraded (4.113%), 29 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-09-17T07:19:35.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:35 smithi100 ceph-mon[29092]: pgmap v807: 178 pgs: 3 active+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 2 remapped+peering, 5 unknown, 11 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 92 active+clean; 212 MiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 1.69k op/s; 427/10746 objects degraded (3.974%); 1487/10746 objects misplaced (13.838%) 2024-09-17T07:19:35.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:35 smithi100 ceph-mon[39462]: osdmap e378: 8 total, 7 up, 7 in 2024-09-17T07:19:35.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:35 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 926/22512 objects degraded (4.113%), 29 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-09-17T07:19:35.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:35 smithi100 ceph-mon[39462]: pgmap v807: 178 pgs: 3 active+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 2 remapped+peering, 5 unknown, 11 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 92 active+clean; 212 MiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 1.69k op/s; 427/10746 objects degraded (3.974%); 1487/10746 objects misplaced (13.838%) 2024-09-17T07:19:35.700 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:19:35.725 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:35.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:35.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:19:35.827 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:19:35.865 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:35.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:35.866 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:19:35.967 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:19:35.993 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:35.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:35.994 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:19:35.997 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:19:35 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:19:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:19:36.048 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-17T07:19:36.049 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Removed 123345 objects 2024-09-17T07:19:36.049 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Clean up completed and total clean up time :106.409 2024-09-17T07:19:36.095 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:19:36.116 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:36.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:36.117 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:19:36.218 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:19:36.239 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:36.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:36.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:19:36.341 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:19:36.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:36.363 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:36.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:19:36.424 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:19:36.464 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:19:36.485 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:36.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:36.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:19:36.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:36 smithi135 ceph-mon[33238]: osdmap e379: 8 total, 7 up, 7 in 2024-09-17T07:19:36.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:36 smithi135 ceph-mon[33238]: 4.3 scrub starts 2024-09-17T07:19:36.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:36 smithi135 ceph-mon[33238]: 4.3 scrub ok 2024-09-17T07:19:36.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:36 smithi135 ceph-mon[33238]: 2.b scrub starts 2024-09-17T07:19:36.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:36 smithi135 ceph-mon[33238]: 2.b scrub ok 2024-09-17T07:19:36.587 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:19:36.608 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:36.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:36.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:19:36.710 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:19:36.731 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:36.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:36.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:19:36.757 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:36 smithi100 ceph-mon[29092]: osdmap e379: 8 total, 7 up, 7 in 2024-09-17T07:19:36.757 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:36 smithi100 ceph-mon[29092]: 4.3 scrub starts 2024-09-17T07:19:36.757 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:36 smithi100 ceph-mon[29092]: 4.3 scrub ok 2024-09-17T07:19:36.757 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:36 smithi100 ceph-mon[29092]: 2.b scrub starts 2024-09-17T07:19:36.757 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:36 smithi100 ceph-mon[29092]: 2.b scrub ok 2024-09-17T07:19:36.757 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:36 smithi100 ceph-mon[39462]: osdmap e379: 8 total, 7 up, 7 in 2024-09-17T07:19:36.757 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:36 smithi100 ceph-mon[39462]: 4.3 scrub starts 2024-09-17T07:19:36.757 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:36 smithi100 ceph-mon[39462]: 4.3 scrub ok 2024-09-17T07:19:36.757 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:36 smithi100 ceph-mon[39462]: 2.b scrub starts 2024-09-17T07:19:36.758 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:36 smithi100 ceph-mon[39462]: 2.b scrub ok 2024-09-17T07:19:36.833 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:19:36.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:36.856 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:36.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:19:36.957 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:19:36.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:36.981 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:36.982 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:19:37.082 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:19:37.105 INFO:teuthology.orchestra.run.smithi135.stdout:{ 2024-09-17T07:19:37.106 INFO:teuthology.orchestra.run.smithi135.stdout: "ops": [], 2024-09-17T07:19:37.106 INFO:teuthology.orchestra.run.smithi135.stdout: "num_ops": 0 2024-09-17T07:19:37.106 INFO:teuthology.orchestra.run.smithi135.stdout:} 2024-09-17T07:19:37.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:37.107 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:37.107 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:19:37.208 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:19:37.230 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:37.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:37.231 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:19:37.331 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:19:37.353 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:37.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:37.354 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:19:37.455 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:19:37.476 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:37.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:37.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:19:37.578 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:19:37.599 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:37.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:37.600 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:19:37.701 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:19:37.709 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:37 smithi135 ceph-mon[33238]: 5.11 scrub starts 2024-09-17T07:19:37.709 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:37 smithi135 ceph-mon[33238]: 5.11 scrub ok 2024-09-17T07:19:37.709 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:37 smithi135 ceph-mon[33238]: 10.13 scrub starts 2024-09-17T07:19:37.709 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:37 smithi135 ceph-mon[33238]: 10.13 scrub ok 2024-09-17T07:19:37.710 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:37 smithi135 ceph-mon[33238]: pgmap v809: 178 pgs: 3 active+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 2 remapped+peering, 5 unknown, 11 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 92 active+clean; 212 MiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 1.36k op/s; 427/10746 objects degraded (3.974%); 1487/10746 objects misplaced (13.838%) 2024-09-17T07:19:37.722 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:37.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:37.723 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:19:37.731 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-17T07:19:37.736 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:37 smithi100 ceph-mon[29092]: 5.11 scrub starts 2024-09-17T07:19:37.736 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:37 smithi100 ceph-mon[29092]: 5.11 scrub ok 2024-09-17T07:19:37.736 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:37 smithi100 ceph-mon[29092]: 10.13 scrub starts 2024-09-17T07:19:37.736 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:37 smithi100 ceph-mon[29092]: 10.13 scrub ok 2024-09-17T07:19:37.736 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:37 smithi100 ceph-mon[29092]: pgmap v809: 178 pgs: 3 active+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 2 remapped+peering, 5 unknown, 11 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 92 active+clean; 212 MiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 1.36k op/s; 427/10746 objects degraded (3.974%); 1487/10746 objects misplaced (13.838%) 2024-09-17T07:19:37.737 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:37 smithi100 ceph-mon[39462]: 5.11 scrub starts 2024-09-17T07:19:37.737 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:37 smithi100 ceph-mon[39462]: 5.11 scrub ok 2024-09-17T07:19:37.737 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:37 smithi100 ceph-mon[39462]: 10.13 scrub starts 2024-09-17T07:19:37.737 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:37 smithi100 ceph-mon[39462]: 10.13 scrub ok 2024-09-17T07:19:37.737 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:37 smithi100 ceph-mon[39462]: pgmap v809: 178 pgs: 3 active+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 2 remapped+peering, 5 unknown, 11 active+clean+remapped, 36 active+undersized, 28 active+undersized+degraded, 92 active+clean; 212 MiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 1.36k op/s; 427/10746 objects degraded (3.974%); 1487/10746 objects misplaced (13.838%) 2024-09-17T07:19:37.824 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:19:37.846 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:37.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:37.848 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:19:37.944 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:19:37.948 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:19:37.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:37.973 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:37.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:19:38.074 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:19:38.096 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:38.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:38.097 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:19:38.112 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:19:38.112 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:19:38.112 INFO:teuthology.orchestra.run.smithi100.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-17T07:19:38.129 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:19:38.129 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:19:38.144 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:19:38.145 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:19:38.161 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:19:38.161 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:19:38.177 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:19:38.177 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:19:38.193 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:19:38.193 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:19:38.198 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:19:38.210 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:19:38.210 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:19:38.219 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:38.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:38.220 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:19:38.321 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:19:38.342 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:38.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:38.343 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:19:38.444 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:19:38.465 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:38.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:38.466 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:19:38.569 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:19:38.591 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:38.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:38.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:19:38.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:38 smithi100 ceph-mon[29092]: 2.5 scrub starts 2024-09-17T07:19:38.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:38 smithi100 ceph-mon[29092]: 2.5 scrub ok 2024-09-17T07:19:38.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:38 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/3140162919' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:19:38.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:38 smithi100 ceph-mon[39462]: 2.5 scrub starts 2024-09-17T07:19:38.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:38 smithi100 ceph-mon[39462]: 2.5 scrub ok 2024-09-17T07:19:38.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:38 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/3140162919' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:19:38.692 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:19:38.714 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:38.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:38.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:19:38.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:38 smithi135 ceph-mon[33238]: 2.5 scrub starts 2024-09-17T07:19:38.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:38 smithi135 ceph-mon[33238]: 2.5 scrub ok 2024-09-17T07:19:38.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:38 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/3140162919' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:19:38.816 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:19:38.837 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:38.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:38.838 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:19:38.939 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:19:38.961 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:38.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:38.962 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:19:39.063 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:19:39.085 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:39.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:39.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:19:39.186 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:19:39.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:39.208 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:39.208 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:19:39.309 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:19:39.330 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:39.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:39.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:19:39.394 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:19:39 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:19:39.432 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:19:39.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:39.453 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:39.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:19:39.554 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:19:39.577 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:39.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:39.578 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:19:39.622 INFO:teuthology.orchestra.run.smithi135.stdout:ERROR: (22) Invalid argument 2024-09-17T07:19:39.622 INFO:teuthology.orchestra.run.smithi135.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:19:39.679 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:19:39.700 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:39.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:39.701 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:19:39.736 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:39 smithi100 ceph-mon[29092]: osdmap e380: 8 total, 7 up, 7 in 2024-09-17T07:19:39.736 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:39 smithi100 ceph-mon[29092]: 10.a deep-scrub starts 2024-09-17T07:19:39.736 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:39 smithi100 ceph-mon[29092]: 10.a deep-scrub ok 2024-09-17T07:19:39.736 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:39 smithi100 ceph-mon[29092]: 10.10 scrub starts 2024-09-17T07:19:39.736 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:39 smithi100 ceph-mon[29092]: 10.10 scrub ok 2024-09-17T07:19:39.736 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:39 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:19:39.737 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:39 smithi100 ceph-mon[29092]: pgmap v811: 178 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 12 active+clean+remapped, 38 active+undersized, 26 active+undersized+degraded, 101 active+clean; 32 MiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 39 KiB/s rd, 27 op/s; 219/2118 objects degraded (10.340%); 215/2118 objects misplaced (10.151%) 2024-09-17T07:19:39.737 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:39 smithi100 ceph-mon[39462]: osdmap e380: 8 total, 7 up, 7 in 2024-09-17T07:19:39.737 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:39 smithi100 ceph-mon[39462]: 10.a deep-scrub starts 2024-09-17T07:19:39.737 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:39 smithi100 ceph-mon[39462]: 10.a deep-scrub ok 2024-09-17T07:19:39.737 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:39 smithi100 ceph-mon[39462]: 10.10 scrub starts 2024-09-17T07:19:39.737 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:39 smithi100 ceph-mon[39462]: 10.10 scrub ok 2024-09-17T07:19:39.737 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:39 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:19:39.737 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:39 smithi100 ceph-mon[39462]: pgmap v811: 178 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 12 active+clean+remapped, 38 active+undersized, 26 active+undersized+degraded, 101 active+clean; 32 MiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 39 KiB/s rd, 27 op/s; 219/2118 objects degraded (10.340%); 215/2118 objects misplaced (10.151%) 2024-09-17T07:19:39.773 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_3 2024-09-17T07:19:39.773 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd pool rm unique_pool_3 unique_pool_3 --yes-i-really-really-mean-it 2024-09-17T07:19:39.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:39 smithi135 ceph-mon[33238]: osdmap e380: 8 total, 7 up, 7 in 2024-09-17T07:19:39.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:39 smithi135 ceph-mon[33238]: 10.a deep-scrub starts 2024-09-17T07:19:39.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:39 smithi135 ceph-mon[33238]: 10.a deep-scrub ok 2024-09-17T07:19:39.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:39 smithi135 ceph-mon[33238]: 10.10 scrub starts 2024-09-17T07:19:39.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:39 smithi135 ceph-mon[33238]: 10.10 scrub ok 2024-09-17T07:19:39.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:39 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:19:39.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:39 smithi135 ceph-mon[33238]: pgmap v811: 178 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 12 active+clean+remapped, 38 active+undersized, 26 active+undersized+degraded, 101 active+clean; 32 MiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 39 KiB/s rd, 27 op/s; 219/2118 objects degraded (10.340%); 215/2118 objects misplaced (10.151%) 2024-09-17T07:19:39.802 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:19:39.823 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:39.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:39.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:19:39.926 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:19:39.947 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:39.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:39.948 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:19:40.049 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:19:40.070 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:40.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:40.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:19:40.172 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:19:40.194 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:40.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:40.195 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:19:40.296 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:19:40.306 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-17T07:19:40.318 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:40.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:40.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:19:40.420 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:19:40.442 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:40.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:40.443 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:19:40.522 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:19:40.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:40 smithi135 ceph-mon[33238]: osdmap e381: 8 total, 7 up, 7 in 2024-09-17T07:19:40.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:40 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 219/2118 objects degraded (10.340%), 27 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-09-17T07:19:40.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:40 smithi135 ceph-mon[33238]: osdmap e382: 8 total, 7 up, 7 in 2024-09-17T07:19:40.543 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:19:40.564 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:40.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:40.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:19:40.666 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:19:40.688 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:40.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:40.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:19:40.740 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:40 smithi100 ceph-mon[39462]: osdmap e381: 8 total, 7 up, 7 in 2024-09-17T07:19:40.741 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:40 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 219/2118 objects degraded (10.340%), 27 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-09-17T07:19:40.741 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:40 smithi100 ceph-mon[39462]: osdmap e382: 8 total, 7 up, 7 in 2024-09-17T07:19:40.741 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:40 smithi100 ceph-mon[29092]: osdmap e381: 8 total, 7 up, 7 in 2024-09-17T07:19:40.741 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:40 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 219/2118 objects degraded (10.340%), 27 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-09-17T07:19:40.741 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:40 smithi100 ceph-mon[29092]: osdmap e382: 8 total, 7 up, 7 in 2024-09-17T07:19:40.790 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:19:40.812 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:40.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:40.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:19:40.913 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:19:40.936 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:40.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:40.937 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:19:41.038 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:19:41.050 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:40 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mon-a[29088]: 2024-09-17T07:19:40.795+0000 7f59b37e7640 -1 mon.a@0(leader).osd e382 definitely_dead 0 2024-09-17T07:19:41.061 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:41.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:41.062 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:19:41.162 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:19:41.183 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:41.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:41.184 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:19:41.285 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:19:41.306 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:41.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:41.307 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:19:41.407 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:19:41.430 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:41.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:41.431 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:19:41.489 INFO:teuthology.orchestra.run.smithi100.stderr:noscrub is set 2024-09-17T07:19:41.491 INFO:teuthology.orchestra.run.smithi100.stderr:marked out osd.7. 2024-09-17T07:19:41.532 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:19:41.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:41.554 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:41.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:19:41.606 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:19:41.607 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:19:41.655 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:19:41.677 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:41.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:41.679 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:19:41.780 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:19:41.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:41 smithi135 ceph-mon[33238]: 2.12 deep-scrub starts 2024-09-17T07:19:41.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:41 smithi135 ceph-mon[33238]: 2.12 deep-scrub ok 2024-09-17T07:19:41.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:41 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/807239682' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-09-17T07:19:41.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:41 smithi135 ceph-mon[33238]: Client client.admin marked osd.7 out, while it was still marked up 2024-09-17T07:19:41.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:41 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/207057545' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:19:41.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:41 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:19:41.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:41 smithi135 ceph-mon[33238]: pgmap v814: 178 pgs: 13 active+clean+remapped, 38 active+undersized, 26 active+undersized+degraded, 101 active+clean; 19 MiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 24 KiB/s rd, 16 op/s; 116/1458 objects degraded (7.956%); 166/1458 objects misplaced (11.385%) 2024-09-17T07:19:41.801 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:41.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:41.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:19:41.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:41 smithi100 ceph-mon[29092]: 2.12 deep-scrub starts 2024-09-17T07:19:41.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:41 smithi100 ceph-mon[29092]: 2.12 deep-scrub ok 2024-09-17T07:19:41.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:41 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/807239682' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-09-17T07:19:41.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:41 smithi100 ceph-mon[29092]: Client client.admin marked osd.7 out, while it was still marked up 2024-09-17T07:19:41.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:41 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/207057545' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:19:41.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:41 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:19:41.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:41 smithi100 ceph-mon[29092]: pgmap v814: 178 pgs: 13 active+clean+remapped, 38 active+undersized, 26 active+undersized+degraded, 101 active+clean; 19 MiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 24 KiB/s rd, 16 op/s; 116/1458 objects degraded (7.956%); 166/1458 objects misplaced (11.385%) 2024-09-17T07:19:41.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:41 smithi100 ceph-mon[39462]: 2.12 deep-scrub starts 2024-09-17T07:19:41.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:41 smithi100 ceph-mon[39462]: 2.12 deep-scrub ok 2024-09-17T07:19:41.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:41 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/807239682' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-09-17T07:19:41.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:41 smithi100 ceph-mon[39462]: Client client.admin marked osd.7 out, while it was still marked up 2024-09-17T07:19:41.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:41 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/207057545' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:19:41.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:41 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:19:41.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:41 smithi100 ceph-mon[39462]: pgmap v814: 178 pgs: 13 active+clean+remapped, 38 active+undersized, 26 active+undersized+degraded, 101 active+clean; 19 MiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 24 KiB/s rd, 16 op/s; 116/1458 objects degraded (7.956%); 166/1458 objects misplaced (11.385%) 2024-09-17T07:19:41.903 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:19:41.924 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:41.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:41.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:19:42.026 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:19:42.048 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:42.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:42.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:19:42.149 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:19:42.171 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:42.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:42.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:19:42.273 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:19:42.294 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:42.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:42.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:19:42.396 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:19:42.417 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:42.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:42.418 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:19:42.519 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:19:42.542 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:42.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:42.544 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:19:42.645 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:19:42.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:42.668 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:42.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:19:42.769 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:19:42.790 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:42.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:42.791 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:19:42.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:42 smithi135 ceph-mon[33238]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-17T07:19:42.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:42 smithi135 ceph-mon[33238]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:19:42.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:42 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/807239682' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-09-17T07:19:42.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:42 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T07:19:42.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:42 smithi135 ceph-mon[33238]: osdmap e383: 8 total, 7 up, 6 in 2024-09-17T07:19:42.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:42 smithi100 ceph-mon[29092]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-17T07:19:42.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:42 smithi100 ceph-mon[29092]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:19:42.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:42 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/807239682' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-09-17T07:19:42.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:42 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T07:19:42.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:42 smithi100 ceph-mon[29092]: osdmap e383: 8 total, 7 up, 6 in 2024-09-17T07:19:42.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:42 smithi100 ceph-mon[39462]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-17T07:19:42.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:42 smithi100 ceph-mon[39462]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:19:42.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:42 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/807239682' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-09-17T07:19:42.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:42 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T07:19:42.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:42 smithi100 ceph-mon[39462]: osdmap e383: 8 total, 7 up, 6 in 2024-09-17T07:19:42.892 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:19:42.914 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:42.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:42.915 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:19:42.990 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:19:43.016 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:19:43.039 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:43.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:43.040 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:19:43.141 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:19:43.156 INFO:teuthology.orchestra.run.smithi135.stdout:ERROR: (22) Invalid argument 2024-09-17T07:19:43.156 INFO:teuthology.orchestra.run.smithi135.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:19:43.162 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:43.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:43.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:19:43.265 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:19:43.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:43.288 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:43.288 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:19:43.389 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:19:43.410 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:43.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:43.412 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:19:43.512 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:19:43.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:43.570 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:43.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:19:43.608 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:19:43.670 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:19:43.691 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:43.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:43.692 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:19:43.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:43 smithi135 ceph-mon[33238]: osdmap e384: 8 total, 7 up, 6 in 2024-09-17T07:19:43.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:43 smithi135 ceph-mon[33238]: 10.13 starting backfill to osd.0 from (0'0,0'0] MAX to 379'14465 2024-09-17T07:19:43.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:43 smithi135 ceph-mon[33238]: 10.22 starting backfill to osd.4 from (0'0,0'0] MAX to 379'10897 2024-09-17T07:19:43.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:43 smithi135 ceph-mon[33238]: pgmap v817: 178 pgs: 11 remapped+peering, 12 active+clean+remapped, 37 active+undersized, 23 active+undersized+degraded, 95 active+clean; 19 MiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 106/1458 objects degraded (7.270%); 166/1458 objects misplaced (11.385%) 2024-09-17T07:19:43.793 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:19:43.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:43.815 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:43.816 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:19:43.840 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-17T07:19:43.876 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:43 smithi100 ceph-mon[39462]: osdmap e384: 8 total, 7 up, 6 in 2024-09-17T07:19:43.876 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:43 smithi100 ceph-mon[39462]: 10.13 starting backfill to osd.0 from (0'0,0'0] MAX to 379'14465 2024-09-17T07:19:43.876 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:43 smithi100 ceph-mon[39462]: 10.22 starting backfill to osd.4 from (0'0,0'0] MAX to 379'10897 2024-09-17T07:19:43.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:43 smithi100 ceph-mon[39462]: pgmap v817: 178 pgs: 11 remapped+peering, 12 active+clean+remapped, 37 active+undersized, 23 active+undersized+degraded, 95 active+clean; 19 MiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 106/1458 objects degraded (7.270%); 166/1458 objects misplaced (11.385%) 2024-09-17T07:19:43.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:43 smithi100 ceph-mon[29092]: osdmap e384: 8 total, 7 up, 6 in 2024-09-17T07:19:43.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:43 smithi100 ceph-mon[29092]: 10.13 starting backfill to osd.0 from (0'0,0'0] MAX to 379'14465 2024-09-17T07:19:43.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:43 smithi100 ceph-mon[29092]: 10.22 starting backfill to osd.4 from (0'0,0'0] MAX to 379'10897 2024-09-17T07:19:43.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:43 smithi100 ceph-mon[29092]: pgmap v817: 178 pgs: 11 remapped+peering, 12 active+clean+remapped, 37 active+undersized, 23 active+undersized+degraded, 95 active+clean; 19 MiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 106/1458 objects degraded (7.270%); 166/1458 objects misplaced (11.385%) 2024-09-17T07:19:43.917 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:19:43.944 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:43.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:43.945 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:19:44.046 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:19:44.068 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:44.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:44.069 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:19:44.079 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:19:44.170 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:19:44.186 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-17T07:19:44.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:44.192 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:44.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:19:44.294 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:19:44.316 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:44.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:44.317 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:19:44.418 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:19:44.439 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:44.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:44.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:19:44.540 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:19:44.562 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:44.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:44.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:19:44.663 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:19:44.669 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:19:44.686 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:44.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:44.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:19:44.788 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:19:44.790 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:44 smithi100 ceph-mon[29092]: Health check failed: Reduced data availability: 2 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-09-17T07:19:44.790 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:44 smithi100 ceph-mon[29092]: 10.8 starting backfill to osd.5 from (0'0,0'0] MAX to 379'11596 2024-09-17T07:19:44.790 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:44 smithi100 ceph-mon[29092]: 10.10 starting backfill to osd.0 from (0'0,0'0] MAX to 379'11665 2024-09-17T07:19:44.790 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:44 smithi100 ceph-mon[29092]: 10.c starting backfill to osd.6 from (0'0,0'0] MAX to 379'14341 2024-09-17T07:19:44.790 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:44 smithi100 ceph-mon[29092]: osdmap e385: 8 total, 7 up, 6 in 2024-09-17T07:19:44.790 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:44 smithi100 ceph-mon[29092]: 10.5 starting backfill to osd.5 from (0'0,0'0] MAX to 379'11644 2024-09-17T07:19:44.790 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:44 smithi100 ceph-mon[29092]: 10.9 starting backfill to osd.4 from (0'0,0'0] MAX to 379'11517 2024-09-17T07:19:44.790 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:44 smithi100 ceph-mon[29092]: 10.11 starting backfill to osd.6 from (0'0,0'0] MAX to 379'11834 2024-09-17T07:19:44.790 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:44 smithi100 ceph-mon[29092]: 10.1b starting backfill to osd.0 from (338'8557,369'13283] MIN to 379'14182 2024-09-17T07:19:44.791 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:44 smithi100 ceph-mon[39462]: Health check failed: Reduced data availability: 2 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-09-17T07:19:44.791 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:44 smithi100 ceph-mon[39462]: 10.8 starting backfill to osd.5 from (0'0,0'0] MAX to 379'11596 2024-09-17T07:19:44.791 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:44 smithi100 ceph-mon[39462]: 10.10 starting backfill to osd.0 from (0'0,0'0] MAX to 379'11665 2024-09-17T07:19:44.791 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:44 smithi100 ceph-mon[39462]: 10.c starting backfill to osd.6 from (0'0,0'0] MAX to 379'14341 2024-09-17T07:19:44.791 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:44 smithi100 ceph-mon[39462]: osdmap e385: 8 total, 7 up, 6 in 2024-09-17T07:19:44.791 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:44 smithi100 ceph-mon[39462]: 10.5 starting backfill to osd.5 from (0'0,0'0] MAX to 379'11644 2024-09-17T07:19:44.791 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:44 smithi100 ceph-mon[39462]: 10.9 starting backfill to osd.4 from (0'0,0'0] MAX to 379'11517 2024-09-17T07:19:44.791 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:44 smithi100 ceph-mon[39462]: 10.11 starting backfill to osd.6 from (0'0,0'0] MAX to 379'11834 2024-09-17T07:19:44.791 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:44 smithi100 ceph-mon[39462]: 10.1b starting backfill to osd.0 from (338'8557,369'13283] MIN to 379'14182 2024-09-17T07:19:44.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:44 smithi135 ceph-mon[33238]: Health check failed: Reduced data availability: 2 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-09-17T07:19:44.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:44 smithi135 ceph-mon[33238]: 10.8 starting backfill to osd.5 from (0'0,0'0] MAX to 379'11596 2024-09-17T07:19:44.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:44 smithi135 ceph-mon[33238]: 10.10 starting backfill to osd.0 from (0'0,0'0] MAX to 379'11665 2024-09-17T07:19:44.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:44 smithi135 ceph-mon[33238]: 10.c starting backfill to osd.6 from (0'0,0'0] MAX to 379'14341 2024-09-17T07:19:44.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:44 smithi135 ceph-mon[33238]: osdmap e385: 8 total, 7 up, 6 in 2024-09-17T07:19:44.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:44 smithi135 ceph-mon[33238]: 10.5 starting backfill to osd.5 from (0'0,0'0] MAX to 379'11644 2024-09-17T07:19:44.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:44 smithi135 ceph-mon[33238]: 10.9 starting backfill to osd.4 from (0'0,0'0] MAX to 379'11517 2024-09-17T07:19:44.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:44 smithi135 ceph-mon[33238]: 10.11 starting backfill to osd.6 from (0'0,0'0] MAX to 379'11834 2024-09-17T07:19:44.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:44 smithi135 ceph-mon[33238]: 10.1b starting backfill to osd.0 from (338'8557,369'13283] MIN to 379'14182 2024-09-17T07:19:44.809 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:44.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:44.810 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:19:44.911 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:19:44.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:44.933 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:44.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:19:45.034 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:19:45.055 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:45.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:45.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:19:45.157 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:19:45.178 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:45.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:45.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:19:45.280 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:19:45.284 INFO:teuthology.orchestra.run.smithi100.stderr:pool 'unique_pool_3' removed 2024-09-17T07:19:45.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:45.323 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:45.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:19:45.424 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:19:45.452 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:45.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:45.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:19:45.554 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:19:45.576 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:45.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:45.577 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:19:45.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:45 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:19:45.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:45 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:19:45.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:45 smithi100 ceph-mon[29092]: osdmap e386: 8 total, 7 up, 6 in 2024-09-17T07:19:45.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:45 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2314635012' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-17T07:19:45.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:45 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-17T07:19:45.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:45 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 106/1458 objects degraded (7.270%), 23 pgs degraded, 57 pgs undersized (PG_DEGRADED) 2024-09-17T07:19:45.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:45 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-09-17T07:19:45.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:45 smithi100 ceph-mon[29092]: osdmap e387: 8 total, 7 up, 6 in 2024-09-17T07:19:45.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:45 smithi100 ceph-mon[29092]: pgmap v821: 142 pgs: 1 peering, 3 active+undersized+remapped, 5 remapped+peering, 26 active+clean+remapped, 24 active+undersized, 17 active+undersized+degraded, 66 active+clean; 582 KiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 263 B/s rd, 0 op/s; 73/588 objects degraded (12.415%); 40/588 objects misplaced (6.803%) 2024-09-17T07:19:45.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:45 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:19:45.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:45 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:19:45.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:45 smithi100 ceph-mon[39462]: osdmap e386: 8 total, 7 up, 6 in 2024-09-17T07:19:45.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:45 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2314635012' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-17T07:19:45.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:45 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-17T07:19:45.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:45 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 106/1458 objects degraded (7.270%), 23 pgs degraded, 57 pgs undersized (PG_DEGRADED) 2024-09-17T07:19:45.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:45 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-09-17T07:19:45.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:45 smithi100 ceph-mon[39462]: osdmap e387: 8 total, 7 up, 6 in 2024-09-17T07:19:45.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:45 smithi100 ceph-mon[39462]: pgmap v821: 142 pgs: 1 peering, 3 active+undersized+remapped, 5 remapped+peering, 26 active+clean+remapped, 24 active+undersized, 17 active+undersized+degraded, 66 active+clean; 582 KiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 263 B/s rd, 0 op/s; 73/588 objects degraded (12.415%); 40/588 objects misplaced (6.803%) 2024-09-17T07:19:45.677 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:19:45.699 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:45.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:45.700 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:19:45.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:45 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:19:45.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:45 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:19:45.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:45 smithi135 ceph-mon[33238]: osdmap e386: 8 total, 7 up, 6 in 2024-09-17T07:19:45.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:45 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2314635012' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-17T07:19:45.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:45 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-17T07:19:45.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:45 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 106/1458 objects degraded (7.270%), 23 pgs degraded, 57 pgs undersized (PG_DEGRADED) 2024-09-17T07:19:45.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:45 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-09-17T07:19:45.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:45 smithi135 ceph-mon[33238]: osdmap e387: 8 total, 7 up, 6 in 2024-09-17T07:19:45.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:45 smithi135 ceph-mon[33238]: pgmap v821: 142 pgs: 1 peering, 3 active+undersized+remapped, 5 remapped+peering, 26 active+clean+remapped, 24 active+undersized, 17 active+undersized+degraded, 66 active+clean; 582 KiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 263 B/s rd, 0 op/s; 73/588 objects degraded (12.415%); 40/588 objects misplaced (6.803%) 2024-09-17T07:19:45.801 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:19:45.825 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:45.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:45.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:19:45.899 INFO:teuthology.orchestra.run.smithi135.stdout:ERROR: (22) Invalid argument 2024-09-17T07:19:45.899 INFO:teuthology.orchestra.run.smithi135.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:19:45.926 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:19:45.948 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:45.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:45.949 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:19:46.050 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:19:46.071 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:46.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:46.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:19:46.084 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:19:45 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:19:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:19:46.104 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-17T07:19:46.105 INFO:tasks.radosbench:Beginning radosbench... 2024-09-17T07:19:46.105 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_4 2024-09-17T07:19:46.105 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd pool create unique_pool_4 16 2024-09-17T07:19:46.173 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:19:46.207 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:46.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:46.208 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:19:46.309 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:19:46.330 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:46.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:46.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:19:46.431 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:19:46.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:46.455 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:46.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:19:46.556 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:19:46.577 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-17T07:19:46.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:46.578 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:46.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:19:46.623 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:19:46.679 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:19:46.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:46.701 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:46.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:19:46.776 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:19:46.802 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:19:46.823 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:46.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:46.824 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:19:46.925 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:19:46.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:46.948 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:46.949 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:19:47.050 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:19:47.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:47.071 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:47.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:19:47.172 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:19:47.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:47.196 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:47.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:19:47.298 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:19:47.309 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:19:47.309 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:19:47.310 INFO:teuthology.orchestra.run.smithi100.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-17T07:19:47.320 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:19:47.320 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:19:47.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:47.322 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:47.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:19:47.336 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:19:47.336 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:19:47.352 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:19:47.353 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:19:47.369 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:19:47.369 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:19:47.384 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:19:47.385 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:19:47.401 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:19:47.401 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:19:47.423 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:19:47.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:47.446 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:47.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:19:47.460 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:47 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/742229197' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:19:47.461 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:47 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/742229197' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:19:47.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:47 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/742229197' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:19:47.564 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:19:47.585 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:47.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:47.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:19:47.687 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:19:47.708 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:47.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:47.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:19:47.809 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:19:47.831 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:47.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:47.832 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:19:47.933 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:19:47.955 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:47.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:47.956 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:19:47.983 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6] out_osds: [3, 7] dead_osds: [1] live_osds: [0, 2, 3, 4, 5, 6, 7] 2024-09-17T07:19:47.983 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T07:19:47.984 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-09-17T07:19:47.984 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd test-reweight-by-utilization 1000 3.0 2024-09-17T07:19:48.057 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:19:48.079 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:48.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:48.080 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:19:48.180 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:19:48.203 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:48.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:48.204 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:19:48.305 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:19:48.325 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:48.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:48.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:19:48.374 INFO:teuthology.orchestra.run.smithi135.stdout:{ 2024-09-17T07:19:48.374 INFO:teuthology.orchestra.run.smithi135.stdout: "ops": [], 2024-09-17T07:19:48.374 INFO:teuthology.orchestra.run.smithi135.stdout: "complaint_time": 30, 2024-09-17T07:19:48.375 INFO:teuthology.orchestra.run.smithi135.stdout: "num_blocked_ops": 0 2024-09-17T07:19:48.375 INFO:teuthology.orchestra.run.smithi135.stdout:} 2024-09-17T07:19:48.427 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:19:48.448 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:48.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:48.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:19:48.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:48 smithi135 ceph-mon[33238]: pgmap v822: 142 pgs: 1 peering, 3 active+undersized+remapped, 5 remapped+peering, 26 active+clean+remapped, 24 active+undersized, 17 active+undersized+degraded, 66 active+clean; 582 KiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 209 B/s rd, 0 op/s; 73/588 objects degraded (12.415%); 40/588 objects misplaced (6.803%) 2024-09-17T07:19:48.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:48 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:19:48.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:48 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:19:48.551 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:19:48.573 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:48.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:48.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:19:48.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:48 smithi100 ceph-mon[29092]: pgmap v822: 142 pgs: 1 peering, 3 active+undersized+remapped, 5 remapped+peering, 26 active+clean+remapped, 24 active+undersized, 17 active+undersized+degraded, 66 active+clean; 582 KiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 209 B/s rd, 0 op/s; 73/588 objects degraded (12.415%); 40/588 objects misplaced (6.803%) 2024-09-17T07:19:48.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:48 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:19:48.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:48 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:19:48.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:48 smithi100 ceph-mon[39462]: pgmap v822: 142 pgs: 1 peering, 3 active+undersized+remapped, 5 remapped+peering, 26 active+clean+remapped, 24 active+undersized, 17 active+undersized+degraded, 66 active+clean; 582 KiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 209 B/s rd, 0 op/s; 73/588 objects degraded (12.415%); 40/588 objects misplaced (6.803%) 2024-09-17T07:19:48.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:48 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:19:48.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:48 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:19:48.674 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:19:48.696 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:48.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:48.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:19:48.709 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:19:48.798 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:19:48.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:48.820 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:48.820 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:19:48.921 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:19:48.944 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:48.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:48.945 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:19:48.958 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-17T07:19:49.046 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:19:49.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:49.069 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:49.069 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:19:49.170 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:19:49.191 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:49.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:49.192 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:19:49.218 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:19:49.293 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:19:49.294 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:19:49 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:19:49.313 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:49.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:49.314 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:19:49.415 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:19:49.437 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:49.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:49.438 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:19:49.539 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:19:49.562 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:49.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:49.563 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:19:49.665 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:19:49.682 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:49 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:19:49.682 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:49 smithi135 ceph-mon[33238]: pgmap v823: 142 pgs: 1 peering, 31 active+clean+remapped, 24 active+undersized, 17 active+undersized+degraded, 69 active+clean; 582 KiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s; 73/588 objects degraded (12.415%); 48/588 objects misplaced (8.163%) 2024-09-17T07:19:49.682 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:49 smithi135 ceph-mon[33238]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 3 pgs peering) 2024-09-17T07:19:49.686 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:49.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:49.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:19:49.788 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:19:49.809 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:49.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:49.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:19:49.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:49 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:19:49.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:49 smithi100 ceph-mon[29092]: pgmap v823: 142 pgs: 1 peering, 31 active+clean+remapped, 24 active+undersized, 17 active+undersized+degraded, 69 active+clean; 582 KiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s; 73/588 objects degraded (12.415%); 48/588 objects misplaced (8.163%) 2024-09-17T07:19:49.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:49 smithi100 ceph-mon[29092]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 3 pgs peering) 2024-09-17T07:19:49.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:49 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:19:49.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:49 smithi100 ceph-mon[39462]: pgmap v823: 142 pgs: 1 peering, 31 active+clean+remapped, 24 active+undersized, 17 active+undersized+degraded, 69 active+clean; 582 KiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s; 73/588 objects degraded (12.415%); 48/588 objects misplaced (8.163%) 2024-09-17T07:19:49.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:49 smithi100 ceph-mon[39462]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 3 pgs peering) 2024-09-17T07:19:49.911 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:19:49.932 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:49.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:49.933 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:19:50.033 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:19:50.054 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:50.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:50.055 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:19:50.156 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:19:50.178 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:50.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:50.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:19:50.280 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:19:50.304 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:50.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:50.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:19:50.406 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:19:50.427 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:50.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:50.428 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:19:50.529 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:19:50.552 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:50.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:50.554 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:19:50.567 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:50 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 73/588 objects degraded (12.415%), 17 pgs degraded, 41 pgs undersized (PG_DEGRADED) 2024-09-17T07:19:50.567 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:50 smithi135 ceph-mon[33238]: osdmap e388: 8 total, 7 up, 6 in 2024-09-17T07:19:50.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:50 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 73/588 objects degraded (12.415%), 17 pgs degraded, 41 pgs undersized (PG_DEGRADED) 2024-09-17T07:19:50.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:50 smithi100 ceph-mon[29092]: osdmap e388: 8 total, 7 up, 6 in 2024-09-17T07:19:50.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:50 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 73/588 objects degraded (12.415%), 17 pgs degraded, 41 pgs undersized (PG_DEGRADED) 2024-09-17T07:19:50.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:50 smithi100 ceph-mon[39462]: osdmap e388: 8 total, 7 up, 6 in 2024-09-17T07:19:50.654 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:19:50.675 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:50.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:50.676 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:19:50.777 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:19:50.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:50.799 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:50.800 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:19:50.816 INFO:teuthology.orchestra.run.smithi135.stdout:{ 2024-09-17T07:19:50.816 INFO:teuthology.orchestra.run.smithi135.stdout: "size": 20, 2024-09-17T07:19:50.816 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 600, 2024-09-17T07:19:50.816 INFO:teuthology.orchestra.run.smithi135.stdout: "ops": [ 2024-09-17T07:19:50.816 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.816 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.15330.0:261544 9.3 9:cd5b6b21:::benchmark_data_smithi100_128631_object127278:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e287)", 2024-09-17T07:19:50.816 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:15:29.219219+0000", 2024-09-17T07:19:50.816 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 261.57715770499999, 2024-09-17T07:19:50.817 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.15732822599999999, 2024-09-17T07:19:50.817 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:19:50.817 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:19:50.817 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:19:50.817 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.15330", 2024-09-17T07:19:50.817 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/1361381749", 2024-09-17T07:19:50.817 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 261544 2024-09-17T07:19:50.817 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.817 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:19:50.817 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.817 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:19:50.817 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.219219+0000", 2024-09-17T07:19:50.818 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:50.818 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.818 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.818 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:19:50.818 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.219219+0000", 2024-09-17T07:19:50.818 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:50.818 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.818 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.818 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:19:50.819 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.219217+0000", 2024-09-17T07:19:50.819 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:19:50.819 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.819 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.819 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:19:50.819 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.219222+0000", 2024-09-17T07:19:50.819 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.0590000000000002e-06 2024-09-17T07:19:50.819 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.819 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.819 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:19:50.819 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.219234+0000", 2024-09-17T07:19:50.819 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.1729000000000001e-05 2024-09-17T07:19:50.820 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.820 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.820 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:19:50.820 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.219241+0000", 2024-09-17T07:19:50.820 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.5819999999999998e-06 2024-09-17T07:19:50.820 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.820 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.820 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:50.820 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.219280+0000", 2024-09-17T07:19:50.820 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.9678000000000002e-05 2024-09-17T07:19:50.820 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.820 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.821 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:19:50.821 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.219368+0000", 2024-09-17T07:19:50.821 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.7573999999999999e-05 2024-09-17T07:19:50.821 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.821 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.821 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 1,5", 2024-09-17T07:19:50.821 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.219445+0000", 2024-09-17T07:19:50.821 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.6570999999999996e-05 2024-09-17T07:19:50.821 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.821 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.821 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:19:50.821 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.220998+0000", 2024-09-17T07:19:50.822 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0015535869999999999 2024-09-17T07:19:50.822 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.822 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.822 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:50.822 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.221216+0000", 2024-09-17T07:19:50.822 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00021739900000000001 2024-09-17T07:19:50.822 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.822 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.822 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:50.822 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.376457+0000", 2024-09-17T07:19:50.822 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.155241037 2024-09-17T07:19:50.822 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.822 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.823 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:19:50.823 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.376511+0000", 2024-09-17T07:19:50.823 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.4154999999999999e-05 2024-09-17T07:19:50.823 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.823 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.823 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:19:50.823 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:15:29.376547+0000", 2024-09-17T07:19:50.823 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.6208999999999998e-05 2024-09-17T07:19:50.823 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:50.823 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:19:50.824 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:50.824 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.824 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.824 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.24338.0:2015 5.14 5:2b04a3e9:::notify.6:head [watch reconnect cookie 94163966834176 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e303)", 2024-09-17T07:19:50.824 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:16:16.974119+0000", 2024-09-17T07:19:50.824 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 213.82225741799999, 2024-09-17T07:19:50.824 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.40769392999999998, 2024-09-17T07:19:50.824 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:19:50.824 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "started", 2024-09-17T07:19:50.824 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:19:50.824 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.24338", 2024-09-17T07:19:50.824 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/1969590951", 2024-09-17T07:19:50.825 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 2015 2024-09-17T07:19:50.825 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.825 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:19:50.825 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.825 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:19:50.825 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:16.974119+0000", 2024-09-17T07:19:50.825 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:50.825 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.825 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.825 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:19:50.825 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:16.974119+0000", 2024-09-17T07:19:50.826 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:50.826 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.826 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.826 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:19:50.826 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:16.974117+0000", 2024-09-17T07:19:50.826 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:19:50.826 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.826 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.826 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:19:50.826 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:16.974125+0000", 2024-09-17T07:19:50.827 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.5580000000000003e-06 2024-09-17T07:19:50.827 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.827 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.827 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:19:50.827 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:16.974137+0000", 2024-09-17T07:19:50.827 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.1970000000000001e-05 2024-09-17T07:19:50.827 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.827 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.827 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:19:50.827 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:16.974145+0000", 2024-09-17T07:19:50.828 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.2069999999999995e-06 2024-09-17T07:19:50.828 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.828 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.828 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:50.828 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:16.974228+0000", 2024-09-17T07:19:50.828 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.2814999999999999e-05 2024-09-17T07:19:50.828 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.828 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.828 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:19:50.829 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:16.974237+0000", 2024-09-17T07:19:50.829 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.1730000000000001e-06 2024-09-17T07:19:50.829 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.829 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.829 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:50.829 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:17.380641+0000", 2024-09-17T07:19:50.829 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.40640414499999999 2024-09-17T07:19:50.829 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.829 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.830 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:19:50.830 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:17.380797+0000", 2024-09-17T07:19:50.830 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00015604700000000001 2024-09-17T07:19:50.830 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.830 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.830 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:19:50.830 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:17.381813+0000", 2024-09-17T07:19:50.830 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0010156640000000001 2024-09-17T07:19:50.830 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:50.830 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:19:50.831 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:50.831 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.831 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.831 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.15648.0:41085 10.c 10:3c85a0dc:::benchmark_data_smithi100_155039_object41084:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e309)", 2024-09-17T07:19:50.831 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:16:37.514983+0000", 2024-09-17T07:19:50.831 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 193.28139363700001, 2024-09-17T07:19:50.831 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.20283764600000001, 2024-09-17T07:19:50.831 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:19:50.831 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:19:50.831 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:19:50.831 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.15648", 2024-09-17T07:19:50.831 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/1684940078", 2024-09-17T07:19:50.832 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 41085 2024-09-17T07:19:50.832 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.832 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:19:50.832 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.832 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:19:50.832 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.514983+0000", 2024-09-17T07:19:50.832 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:50.832 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.832 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.833 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:19:50.833 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.514983+0000", 2024-09-17T07:19:50.833 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:50.833 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.833 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.833 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:19:50.833 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.514981+0000", 2024-09-17T07:19:50.833 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999981 2024-09-17T07:19:50.833 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.833 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.833 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:19:50.833 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.515020+0000", 2024-09-17T07:19:50.834 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.8779000000000003e-05 2024-09-17T07:19:50.834 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.834 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.834 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:19:50.834 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.515034+0000", 2024-09-17T07:19:50.834 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.3658e-05 2024-09-17T07:19:50.834 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.834 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.834 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:19:50.834 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.515040+0000", 2024-09-17T07:19:50.834 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.9029999999999998e-06 2024-09-17T07:19:50.834 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.835 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.835 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:50.835 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.515164+0000", 2024-09-17T07:19:50.835 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00012357700000000001 2024-09-17T07:19:50.835 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.835 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.835 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:19:50.835 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.515305+0000", 2024-09-17T07:19:50.835 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00014111500000000001 2024-09-17T07:19:50.835 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.835 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.835 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,4", 2024-09-17T07:19:50.836 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.515451+0000", 2024-09-17T07:19:50.836 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000145623 2024-09-17T07:19:50.836 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.836 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.836 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:50.836 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.517446+0000", 2024-09-17T07:19:50.836 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.001995447 2024-09-17T07:19:50.836 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.836 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.836 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:19:50.836 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.518673+0000", 2024-09-17T07:19:50.836 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0012263199999999999 2024-09-17T07:19:50.837 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.837 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.837 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:50.837 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.717735+0000", 2024-09-17T07:19:50.837 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.19906210499999999 2024-09-17T07:19:50.837 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.837 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.837 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:19:50.837 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.717784+0000", 2024-09-17T07:19:50.837 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.9632000000000002e-05 2024-09-17T07:19:50.837 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.837 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.838 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:19:50.838 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.717821+0000", 2024-09-17T07:19:50.838 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.6251000000000002e-05 2024-09-17T07:19:50.838 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:50.838 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:19:50.838 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:50.838 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.838 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.838 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.15648.0:41086 10.c 10:3dd3bd63:::benchmark_data_smithi100_155039_object41085:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e309)", 2024-09-17T07:19:50.838 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:16:37.515054+0000", 2024-09-17T07:19:50.838 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 193.281322442, 2024-09-17T07:19:50.838 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.20290586399999999, 2024-09-17T07:19:50.838 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:19:50.839 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:19:50.839 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:19:50.839 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.15648", 2024-09-17T07:19:50.839 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/1684940078", 2024-09-17T07:19:50.839 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 41086 2024-09-17T07:19:50.839 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.839 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:19:50.839 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.839 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:19:50.839 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.515054+0000", 2024-09-17T07:19:50.840 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:50.840 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.840 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.840 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:19:50.840 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.515054+0000", 2024-09-17T07:19:50.841 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:50.841 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.841 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.841 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:19:50.841 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.515054+0000", 2024-09-17T07:19:50.841 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999995 2024-09-17T07:19:50.841 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.841 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.841 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:19:50.841 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.515083+0000", 2024-09-17T07:19:50.841 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.9658e-05 2024-09-17T07:19:50.842 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.842 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.842 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:19:50.842 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.515089+0000", 2024-09-17T07:19:50.842 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.0780000000000001e-06 2024-09-17T07:19:50.842 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.842 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.842 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:19:50.842 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.515095+0000", 2024-09-17T07:19:50.842 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.7400000000000001e-06 2024-09-17T07:19:50.842 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.842 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.843 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:50.843 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.517474+0000", 2024-09-17T07:19:50.843 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0023792890000000001 2024-09-17T07:19:50.843 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.843 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.843 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:19:50.843 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.517605+0000", 2024-09-17T07:19:50.843 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000130197 2024-09-17T07:19:50.843 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.844 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.844 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,4", 2024-09-17T07:19:50.844 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.517726+0000", 2024-09-17T07:19:50.844 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000121136 2024-09-17T07:19:50.844 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.844 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.844 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:19:50.844 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.518794+0000", 2024-09-17T07:19:50.844 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0010687419999999999 2024-09-17T07:19:50.844 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.844 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.844 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:50.845 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.519810+0000", 2024-09-17T07:19:50.845 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0010154369999999999 2024-09-17T07:19:50.845 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.845 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.845 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:50.845 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.717855+0000", 2024-09-17T07:19:50.845 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.198045429 2024-09-17T07:19:50.845 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.845 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.845 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:19:50.845 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.717896+0000", 2024-09-17T07:19:50.845 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.1090000000000001e-05 2024-09-17T07:19:50.846 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.846 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.846 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:19:50.846 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.717960+0000", 2024-09-17T07:19:50.846 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.3567999999999999e-05 2024-09-17T07:19:50.846 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:50.846 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:19:50.846 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:50.846 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.846 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.846 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.15648.0:41096 10.c 10:3f502d12:::benchmark_data_smithi100_155039_object41095:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e309)", 2024-09-17T07:19:50.846 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:16:37.520264+0000", 2024-09-17T07:19:50.846 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 193.27611239999999, 2024-09-17T07:19:50.847 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.197820209, 2024-09-17T07:19:50.847 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:19:50.847 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:19:50.847 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:19:50.847 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.15648", 2024-09-17T07:19:50.847 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/1684940078", 2024-09-17T07:19:50.847 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 41096 2024-09-17T07:19:50.847 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.847 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:19:50.847 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.847 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:19:50.847 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.520264+0000", 2024-09-17T07:19:50.847 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:50.848 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.848 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.848 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:19:50.848 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.520264+0000", 2024-09-17T07:19:50.848 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:50.848 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.848 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.848 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:19:50.848 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.520263+0000", 2024-09-17T07:19:50.848 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.999999 2024-09-17T07:19:50.848 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.848 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.848 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:19:50.849 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.520526+0000", 2024-09-17T07:19:50.849 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00026263099999999999 2024-09-17T07:19:50.849 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.849 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.849 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:19:50.849 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.520537+0000", 2024-09-17T07:19:50.849 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0849000000000001e-05 2024-09-17T07:19:50.849 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.849 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.849 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:19:50.849 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.520544+0000", 2024-09-17T07:19:50.849 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.1060000000000002e-06 2024-09-17T07:19:50.849 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.850 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.850 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:50.850 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.520577+0000", 2024-09-17T07:19:50.850 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.3840999999999997e-05 2024-09-17T07:19:50.850 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.850 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.850 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:19:50.850 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.520668+0000", 2024-09-17T07:19:50.850 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.0229999999999995e-05 2024-09-17T07:19:50.850 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.850 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.850 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,4", 2024-09-17T07:19:50.850 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.520746+0000", 2024-09-17T07:19:50.850 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.8368000000000006e-05 2024-09-17T07:19:50.851 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.851 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.851 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:19:50.851 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.521868+0000", 2024-09-17T07:19:50.851 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.001121637 2024-09-17T07:19:50.851 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.851 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.851 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:50.851 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.522774+0000", 2024-09-17T07:19:50.851 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00090647000000000004 2024-09-17T07:19:50.851 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.851 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.851 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:50.852 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.717999+0000", 2024-09-17T07:19:50.852 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.195225327 2024-09-17T07:19:50.852 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.852 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.852 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:19:50.852 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.718048+0000", 2024-09-17T07:19:50.852 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.8568e-05 2024-09-17T07:19:50.852 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.852 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.852 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:19:50.852 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.718084+0000", 2024-09-17T07:19:50.852 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.6257000000000002e-05 2024-09-17T07:19:50.852 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:50.853 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:19:50.853 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:50.853 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.853 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.853 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.15648.0:41098 10.c 10:39aafbae:::benchmark_data_smithi100_155039_object41097:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e309)", 2024-09-17T07:19:50.853 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:16:37.522841+0000", 2024-09-17T07:19:50.853 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 193.273535471, 2024-09-17T07:19:50.853 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.199064503, 2024-09-17T07:19:50.853 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:19:50.853 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:19:50.853 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:19:50.854 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.15648", 2024-09-17T07:19:50.854 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/1684940078", 2024-09-17T07:19:50.854 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 41098 2024-09-17T07:19:50.854 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.854 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:19:50.854 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.854 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:19:50.854 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.522841+0000", 2024-09-17T07:19:50.854 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:50.854 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.854 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.854 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:19:50.855 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.522841+0000", 2024-09-17T07:19:50.855 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:50.855 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.855 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.855 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:19:50.855 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.522840+0000", 2024-09-17T07:19:50.855 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.999999 2024-09-17T07:19:50.855 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.855 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.855 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:19:50.855 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.522898+0000", 2024-09-17T07:19:50.855 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.7596999999999998e-05 2024-09-17T07:19:50.855 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.856 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.856 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:19:50.856 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.522904+0000", 2024-09-17T07:19:50.856 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.5239999999999997e-06 2024-09-17T07:19:50.856 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.856 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.856 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:19:50.856 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.522908+0000", 2024-09-17T07:19:50.856 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.7579999999999998e-06 2024-09-17T07:19:50.856 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.856 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.856 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:50.857 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.522935+0000", 2024-09-17T07:19:50.857 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.7546e-05 2024-09-17T07:19:50.857 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.857 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.857 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:19:50.857 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.546598+0000", 2024-09-17T07:19:50.857 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.023662882999999999 2024-09-17T07:19:50.857 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.857 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.857 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,4", 2024-09-17T07:19:50.857 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.546741+0000", 2024-09-17T07:19:50.857 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000142659 2024-09-17T07:19:50.858 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.858 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.858 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:19:50.858 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.548657+0000", 2024-09-17T07:19:50.858 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0019164270000000001 2024-09-17T07:19:50.858 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.858 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.858 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:50.858 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.548891+0000", 2024-09-17T07:19:50.858 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000233899 2024-09-17T07:19:50.858 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.858 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.858 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:50.858 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.721694+0000", 2024-09-17T07:19:50.859 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.17280299399999999 2024-09-17T07:19:50.859 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.859 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.859 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:19:50.859 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.721789+0000", 2024-09-17T07:19:50.859 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.4996999999999999e-05 2024-09-17T07:19:50.859 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.859 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.859 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:19:50.859 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:16:37.721906+0000", 2024-09-17T07:19:50.859 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00011618899999999999 2024-09-17T07:19:50.860 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:50.860 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:19:50.860 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:50.860 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.860 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.860 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.15648.0:217196 10.1f 10:fc987325:::benchmark_data_smithi100_155039_object93848:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e369)", 2024-09-17T07:19:50.860 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:19:14.502423+0000", 2024-09-17T07:19:50.860 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 36.293953045000002, 2024-09-17T07:19:50.860 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.80055285700000001, 2024-09-17T07:19:50.860 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:19:50.860 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:19:50.860 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:19:50.860 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.15648", 2024-09-17T07:19:50.861 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/1684940078", 2024-09-17T07:19:50.861 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 217196 2024-09-17T07:19:50.861 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.861 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:19:50.861 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.861 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:19:50.861 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.502423+0000", 2024-09-17T07:19:50.861 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:50.861 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.861 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.861 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:19:50.862 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.502423+0000", 2024-09-17T07:19:50.862 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:50.862 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.862 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.862 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:19:50.862 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.502422+0000", 2024-09-17T07:19:50.862 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.999999 2024-09-17T07:19:50.862 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.862 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.862 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:19:50.862 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.502430+0000", 2024-09-17T07:19:50.862 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.9670000000000001e-06 2024-09-17T07:19:50.862 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.862 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.863 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:19:50.863 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.502441+0000", 2024-09-17T07:19:50.863 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0526000000000001e-05 2024-09-17T07:19:50.863 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.863 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.863 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:19:50.863 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.502448+0000", 2024-09-17T07:19:50.863 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.8279999999999997e-06 2024-09-17T07:19:50.863 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.863 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.863 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:50.863 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.502530+0000", 2024-09-17T07:19:50.863 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.2094000000000004e-05 2024-09-17T07:19:50.863 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.864 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.864 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:19:50.864 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.502539+0000", 2024-09-17T07:19:50.864 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.1030000000000008e-06 2024-09-17T07:19:50.864 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.864 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.864 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:50.864 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.299516+0000", 2024-09-17T07:19:50.864 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.79697650499999995 2024-09-17T07:19:50.864 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.864 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.864 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:19:50.865 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.299638+0000", 2024-09-17T07:19:50.865 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00012289700000000001 2024-09-17T07:19:50.865 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.865 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.865 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 2,4", 2024-09-17T07:19:50.865 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.299806+0000", 2024-09-17T07:19:50.865 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00016735900000000001 2024-09-17T07:19:50.865 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.865 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.865 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:50.865 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.302819+0000", 2024-09-17T07:19:50.865 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.003012916 2024-09-17T07:19:50.865 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.866 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.866 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:19:50.866 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.302828+0000", 2024-09-17T07:19:50.866 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.7259999999999997e-06 2024-09-17T07:19:50.866 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.866 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.866 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:50.866 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.302857+0000", 2024-09-17T07:19:50.866 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.8615e-05 2024-09-17T07:19:50.866 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.866 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.866 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:19:50.866 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.302893+0000", 2024-09-17T07:19:50.866 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.5858999999999997e-05 2024-09-17T07:19:50.867 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.867 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.867 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:19:50.867 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.302976+0000", 2024-09-17T07:19:50.867 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.3415e-05 2024-09-17T07:19:50.867 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:50.867 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:19:50.867 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:50.867 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.867 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.867 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.15648.0:217228 10.1f 10:f8d60fbc:::benchmark_data_smithi100_155039_object93880:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e369)", 2024-09-17T07:19:50.867 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:19:14.526725+0000", 2024-09-17T07:19:50.868 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 36.269651152000002, 2024-09-17T07:19:50.868 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.77933181399999996, 2024-09-17T07:19:50.868 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:19:50.868 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:19:50.868 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:19:50.868 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.15648", 2024-09-17T07:19:50.868 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/1684940078", 2024-09-17T07:19:50.868 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 217228 2024-09-17T07:19:50.868 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.868 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:19:50.868 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.868 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:19:50.868 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.526725+0000", 2024-09-17T07:19:50.869 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:50.869 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.869 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.869 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:19:50.869 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.526725+0000", 2024-09-17T07:19:50.869 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:50.869 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.869 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.869 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:19:50.869 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.526724+0000", 2024-09-17T07:19:50.870 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.999999 2024-09-17T07:19:50.870 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.870 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.870 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:19:50.870 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.526730+0000", 2024-09-17T07:19:50.870 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.1309999999999997e-06 2024-09-17T07:19:50.870 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.870 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.870 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:19:50.870 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.526747+0000", 2024-09-17T07:19:50.870 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.7345000000000001e-05 2024-09-17T07:19:50.871 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.871 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.871 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:19:50.871 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.526755+0000", 2024-09-17T07:19:50.871 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.0390000000000005e-06 2024-09-17T07:19:50.871 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.871 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.871 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:50.872 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.526810+0000", 2024-09-17T07:19:50.872 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.4979999999999999e-05 2024-09-17T07:19:50.872 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.872 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.872 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:19:50.872 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.526819+0000", 2024-09-17T07:19:50.872 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.7269999999999996e-06 2024-09-17T07:19:50.872 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.873 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.873 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:50.873 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.301106+0000", 2024-09-17T07:19:50.873 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.77428704199999998 2024-09-17T07:19:50.873 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.873 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.873 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:19:50.873 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.301206+0000", 2024-09-17T07:19:50.873 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000100399 2024-09-17T07:19:50.873 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.873 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.873 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 2,4", 2024-09-17T07:19:50.873 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.301321+0000", 2024-09-17T07:19:50.874 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00011521100000000001 2024-09-17T07:19:50.874 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.874 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.874 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:50.874 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.303119+0000", 2024-09-17T07:19:50.874 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0017975790000000001 2024-09-17T07:19:50.874 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.874 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.874 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:19:50.874 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.303173+0000", 2024-09-17T07:19:50.874 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.4027999999999998e-05 2024-09-17T07:19:50.874 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.874 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.874 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:50.875 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.305912+0000", 2024-09-17T07:19:50.875 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.002738589 2024-09-17T07:19:50.875 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.875 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.875 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:19:50.875 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.305960+0000", 2024-09-17T07:19:50.875 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.8483000000000003e-05 2024-09-17T07:19:50.875 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.875 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.875 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:19:50.875 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.306057+0000", 2024-09-17T07:19:50.875 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.7153999999999998e-05 2024-09-17T07:19:50.875 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:50.876 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:19:50.876 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:50.876 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.876 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.876 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.15648.0:217230 10.1f 10:fef3f209:::benchmark_data_smithi100_155039_object93882:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e369)", 2024-09-17T07:19:50.876 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:19:14.527523+0000", 2024-09-17T07:19:50.876 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 36.268853735999997, 2024-09-17T07:19:50.876 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.77870166100000004, 2024-09-17T07:19:50.876 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:19:50.876 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:19:50.877 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:19:50.877 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.15648", 2024-09-17T07:19:50.877 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/1684940078", 2024-09-17T07:19:50.877 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 217230 2024-09-17T07:19:50.877 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.877 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:19:50.877 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.877 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:19:50.877 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.527523+0000", 2024-09-17T07:19:50.877 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:50.877 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.877 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.877 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:19:50.878 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.527523+0000", 2024-09-17T07:19:50.878 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:50.878 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.878 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.878 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:19:50.878 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.527522+0000", 2024-09-17T07:19:50.878 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.999999 2024-09-17T07:19:50.878 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.878 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.878 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:19:50.878 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.527527+0000", 2024-09-17T07:19:50.878 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.4330000000000003e-06 2024-09-17T07:19:50.878 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.879 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.879 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:19:50.879 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.527541+0000", 2024-09-17T07:19:50.879 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.4267000000000001e-05 2024-09-17T07:19:50.879 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.879 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.879 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:19:50.879 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.527549+0000", 2024-09-17T07:19:50.879 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.3039999999999999e-06 2024-09-17T07:19:50.879 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.879 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.879 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:50.879 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.527592+0000", 2024-09-17T07:19:50.879 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.2944000000000001e-05 2024-09-17T07:19:50.880 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.880 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.880 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:19:50.880 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.527599+0000", 2024-09-17T07:19:50.880 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.7340000000000008e-06 2024-09-17T07:19:50.880 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.880 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.880 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:50.880 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.301879+0000", 2024-09-17T07:19:50.880 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.77427997000000004 2024-09-17T07:19:50.880 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.881 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.881 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:19:50.881 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.301988+0000", 2024-09-17T07:19:50.881 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00010907 2024-09-17T07:19:50.881 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.881 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.881 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 2,4", 2024-09-17T07:19:50.881 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.302158+0000", 2024-09-17T07:19:50.881 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00016990700000000001 2024-09-17T07:19:50.881 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.881 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.881 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:50.882 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.303164+0000", 2024-09-17T07:19:50.882 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0010055540000000001 2024-09-17T07:19:50.882 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.882 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.882 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:19:50.882 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.303179+0000", 2024-09-17T07:19:50.882 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.4797000000000001e-05 2024-09-17T07:19:50.882 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.882 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.883 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:50.883 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.306103+0000", 2024-09-17T07:19:50.883 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0029241319999999999 2024-09-17T07:19:50.883 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.883 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.883 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:19:50.883 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.306148+0000", 2024-09-17T07:19:50.883 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.4861999999999998e-05 2024-09-17T07:19:50.883 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.883 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.883 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:19:50.883 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.306224+0000", 2024-09-17T07:19:50.883 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.6809000000000003e-05 2024-09-17T07:19:50.883 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:50.884 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:19:50.884 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:50.884 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.884 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.884 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.15648.0:217243 10.1f 10:fa3f1e40:::benchmark_data_smithi100_155039_object93895:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e369)", 2024-09-17T07:19:50.884 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:19:14.536526+0000", 2024-09-17T07:19:50.884 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 36.259850520999997, 2024-09-17T07:19:50.884 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.76988383599999999, 2024-09-17T07:19:50.884 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:19:50.884 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:19:50.884 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:19:50.884 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.15648", 2024-09-17T07:19:50.885 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/1684940078", 2024-09-17T07:19:50.885 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 217243 2024-09-17T07:19:50.885 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.885 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:19:50.885 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.885 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:19:50.885 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.536526+0000", 2024-09-17T07:19:50.885 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:50.885 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.885 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.885 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:19:50.885 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.536526+0000", 2024-09-17T07:19:50.886 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:50.886 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.886 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.886 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:19:50.886 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.536524+0000", 2024-09-17T07:19:50.886 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:19:50.886 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.886 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.886 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:19:50.886 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.536531+0000", 2024-09-17T07:19:50.886 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.7889999999999997e-06 2024-09-17T07:19:50.886 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.886 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.886 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:19:50.887 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.536546+0000", 2024-09-17T07:19:50.887 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.5143999999999999e-05 2024-09-17T07:19:50.887 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.887 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.887 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:19:50.887 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.536554+0000", 2024-09-17T07:19:50.887 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.7449999999999995e-06 2024-09-17T07:19:50.887 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.887 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.887 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:50.887 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.536609+0000", 2024-09-17T07:19:50.887 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.5315999999999997e-05 2024-09-17T07:19:50.887 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.888 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.888 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:19:50.888 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.536617+0000", 2024-09-17T07:19:50.888 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.7640000000000005e-06 2024-09-17T07:19:50.888 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.888 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.888 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:50.888 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.303223+0000", 2024-09-17T07:19:50.888 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.766605973 2024-09-17T07:19:50.888 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.888 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.888 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:19:50.889 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.303339+0000", 2024-09-17T07:19:50.889 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000116032 2024-09-17T07:19:50.889 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.889 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.889 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 2,4", 2024-09-17T07:19:50.889 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.303521+0000", 2024-09-17T07:19:50.889 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000182086 2024-09-17T07:19:50.889 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.889 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.889 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:19:50.889 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.306068+0000", 2024-09-17T07:19:50.889 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.002546974 2024-09-17T07:19:50.889 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.890 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.890 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:50.890 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.306271+0000", 2024-09-17T07:19:50.890 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00020252200000000001 2024-09-17T07:19:50.890 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.890 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.890 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:50.890 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.306302+0000", 2024-09-17T07:19:50.890 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.1479000000000001e-05 2024-09-17T07:19:50.890 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.890 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.890 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:19:50.890 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.306341+0000", 2024-09-17T07:19:50.890 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.9153000000000003e-05 2024-09-17T07:19:50.891 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.891 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.891 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:19:50.891 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.306410+0000", 2024-09-17T07:19:50.891 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.8388999999999999e-05 2024-09-17T07:19:50.891 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:50.891 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:19:50.891 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:50.891 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.891 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.891 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.15648.0:217340 10.1f 10:fa0af639:::benchmark_data_smithi100_155039_object93992:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e369)", 2024-09-17T07:19:50.891 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:19:14.608061+0000", 2024-09-17T07:19:50.891 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 36.188315160999998, 2024-09-17T07:19:50.892 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.69852515199999998, 2024-09-17T07:19:50.892 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:19:50.892 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:19:50.892 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:19:50.892 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.15648", 2024-09-17T07:19:50.892 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/1684940078", 2024-09-17T07:19:50.892 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 217340 2024-09-17T07:19:50.892 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.892 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:19:50.893 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.893 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:19:50.893 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.608061+0000", 2024-09-17T07:19:50.893 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:50.893 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.893 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.893 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:19:50.893 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.608061+0000", 2024-09-17T07:19:50.893 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:50.894 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.894 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.894 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:19:50.894 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.608060+0000", 2024-09-17T07:19:50.894 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:19:50.894 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.894 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.894 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:19:50.894 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.608067+0000", 2024-09-17T07:19:50.894 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.8689999999999998e-06 2024-09-17T07:19:50.894 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.894 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.895 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:19:50.895 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.608081+0000", 2024-09-17T07:19:50.895 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.4803e-05 2024-09-17T07:19:50.895 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.895 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.895 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:19:50.895 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.608089+0000", 2024-09-17T07:19:50.895 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.909e-06 2024-09-17T07:19:50.895 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.895 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.895 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:50.895 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.608140+0000", 2024-09-17T07:19:50.896 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.0646999999999998e-05 2024-09-17T07:19:50.896 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.896 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.896 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:19:50.896 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.608148+0000", 2024-09-17T07:19:50.896 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.1389999999999995e-06 2024-09-17T07:19:50.896 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.896 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.896 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:50.897 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.304785+0000", 2024-09-17T07:19:50.897 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.69663686400000002 2024-09-17T07:19:50.897 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.898 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.898 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:19:50.898 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.304893+0000", 2024-09-17T07:19:50.898 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000108074 2024-09-17T07:19:50.898 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.898 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.898 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 2,4", 2024-09-17T07:19:50.898 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.305060+0000", 2024-09-17T07:19:50.898 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00016676500000000001 2024-09-17T07:19:50.898 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.898 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.898 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:19:50.898 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.306230+0000", 2024-09-17T07:19:50.899 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0011703499999999999 2024-09-17T07:19:50.899 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.899 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.899 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:50.899 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.306462+0000", 2024-09-17T07:19:50.899 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000231649 2024-09-17T07:19:50.899 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.899 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.899 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:50.899 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.306496+0000", 2024-09-17T07:19:50.899 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.4224999999999998e-05 2024-09-17T07:19:50.899 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.900 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.900 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:19:50.900 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.306520+0000", 2024-09-17T07:19:50.900 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.3547999999999999e-05 2024-09-17T07:19:50.900 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.900 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.900 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:19:50.900 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.306586+0000", 2024-09-17T07:19:50.900 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.6859000000000005e-05 2024-09-17T07:19:50.900 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:50.900 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:19:50.901 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:50.901 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.901 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.901 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.15648.0:217380 10.1f 10:fbba0364:::benchmark_data_smithi100_155039_object94032:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e369)", 2024-09-17T07:19:50.901 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:19:14.638826+0000", 2024-09-17T07:19:50.901 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 36.157550808000003, 2024-09-17T07:19:50.901 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.67615475700000005, 2024-09-17T07:19:50.901 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:19:50.901 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:19:50.901 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:19:50.901 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.15648", 2024-09-17T07:19:50.901 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/1684940078", 2024-09-17T07:19:50.901 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 217380 2024-09-17T07:19:50.902 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.902 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:19:50.902 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.902 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:19:50.902 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.638826+0000", 2024-09-17T07:19:50.902 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:50.902 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.902 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.902 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:19:50.902 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.638826+0000", 2024-09-17T07:19:50.902 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:50.902 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.902 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.903 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:19:50.903 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.638824+0000", 2024-09-17T07:19:50.903 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:19:50.903 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.903 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.903 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:19:50.903 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.638831+0000", 2024-09-17T07:19:50.903 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.866e-06 2024-09-17T07:19:50.903 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.903 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.903 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:19:50.903 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.638848+0000", 2024-09-17T07:19:50.903 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.6651999999999999e-05 2024-09-17T07:19:50.904 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.904 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.904 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:19:50.904 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.638856+0000", 2024-09-17T07:19:50.904 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.9980000000000003e-06 2024-09-17T07:19:50.904 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.904 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.904 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:50.904 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.638903+0000", 2024-09-17T07:19:50.904 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.7741000000000003e-05 2024-09-17T07:19:50.904 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.904 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.904 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:19:50.904 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.638912+0000", 2024-09-17T07:19:50.905 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.4079999999999998e-06 2024-09-17T07:19:50.905 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.905 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.905 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:50.905 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.306619+0000", 2024-09-17T07:19:50.905 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.66770699499999997 2024-09-17T07:19:50.905 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.905 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.905 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:19:50.905 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.306685+0000", 2024-09-17T07:19:50.906 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.6104000000000003e-05 2024-09-17T07:19:50.906 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.906 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.906 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 2,4", 2024-09-17T07:19:50.906 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.306840+0000", 2024-09-17T07:19:50.906 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00015505 2024-09-17T07:19:50.906 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.906 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.906 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:19:50.906 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.311840+0000", 2024-09-17T07:19:50.906 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0050000649999999997 2024-09-17T07:19:50.906 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.907 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.907 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:50.907 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.312558+0000", 2024-09-17T07:19:50.907 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00071763600000000001 2024-09-17T07:19:50.907 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.907 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.907 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:50.907 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.314854+0000", 2024-09-17T07:19:50.907 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.002296116 2024-09-17T07:19:50.907 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.907 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.907 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:19:50.907 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.314896+0000", 2024-09-17T07:19:50.908 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.2101000000000002e-05 2024-09-17T07:19:50.908 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.908 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.908 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:19:50.908 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.314980+0000", 2024-09-17T07:19:50.908 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.4480000000000004e-05 2024-09-17T07:19:50.908 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:50.908 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:19:50.908 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:50.908 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.908 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.908 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.15648.0:217424 10.1f 10:ffa29408:::benchmark_data_smithi100_155039_object94076:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e369)", 2024-09-17T07:19:50.909 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:19:14.672687+0000", 2024-09-17T07:19:50.909 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 36.123689517000003, 2024-09-17T07:19:50.909 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.64243086500000002, 2024-09-17T07:19:50.909 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:19:50.909 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:19:50.909 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:19:50.909 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.15648", 2024-09-17T07:19:50.909 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/1684940078", 2024-09-17T07:19:50.909 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 217424 2024-09-17T07:19:50.909 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.909 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:19:50.909 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.909 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:19:50.910 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.672687+0000", 2024-09-17T07:19:50.910 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:50.910 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.910 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.910 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:19:50.910 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.672687+0000", 2024-09-17T07:19:50.910 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:50.910 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.910 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.910 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:19:50.910 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.672672+0000", 2024-09-17T07:19:50.910 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999852 2024-09-17T07:19:50.910 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.910 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.910 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:19:50.911 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.672693+0000", 2024-09-17T07:19:50.911 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.0616999999999999e-05 2024-09-17T07:19:50.911 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.911 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.911 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:19:50.911 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.672711+0000", 2024-09-17T07:19:50.911 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.8185e-05 2024-09-17T07:19:50.911 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.911 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.911 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:19:50.911 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.672718+0000", 2024-09-17T07:19:50.911 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.7170000000000002e-06 2024-09-17T07:19:50.911 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.912 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.912 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:50.912 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.672778+0000", 2024-09-17T07:19:50.912 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.0000999999999997e-05 2024-09-17T07:19:50.912 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.912 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.912 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:19:50.912 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.672788+0000", 2024-09-17T07:19:50.912 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0535e-05 2024-09-17T07:19:50.912 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.912 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.912 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:50.912 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.307657+0000", 2024-09-17T07:19:50.912 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.63486912500000003 2024-09-17T07:19:50.913 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.913 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.913 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:19:50.913 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.307758+0000", 2024-09-17T07:19:50.913 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00010047699999999999 2024-09-17T07:19:50.913 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.913 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.913 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 2,4", 2024-09-17T07:19:50.913 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.307931+0000", 2024-09-17T07:19:50.913 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00017306999999999999 2024-09-17T07:19:50.913 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.913 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.913 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:19:50.914 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.311846+0000", 2024-09-17T07:19:50.914 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0039155040000000002 2024-09-17T07:19:50.914 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.914 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.914 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:50.914 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.312588+0000", 2024-09-17T07:19:50.914 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00074213000000000002 2024-09-17T07:19:50.914 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.914 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.915 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:50.915 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.315012+0000", 2024-09-17T07:19:50.915 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.002423616 2024-09-17T07:19:50.915 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.915 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.915 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:19:50.915 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.315044+0000", 2024-09-17T07:19:50.915 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.1835999999999998e-05 2024-09-17T07:19:50.915 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.915 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.915 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:19:50.915 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.315118+0000", 2024-09-17T07:19:50.915 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.3942000000000005e-05 2024-09-17T07:19:50.916 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:50.916 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:19:50.916 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:50.916 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.916 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.916 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.15648.0:217446 10.1f 10:f8926ab4:::benchmark_data_smithi100_155039_object94098:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e369)", 2024-09-17T07:19:50.916 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:19:14.702711+0000", 2024-09-17T07:19:50.916 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 36.093665246, 2024-09-17T07:19:50.916 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.61254003099999998, 2024-09-17T07:19:50.916 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:19:50.916 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:19:50.917 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:19:50.917 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.15648", 2024-09-17T07:19:50.917 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/1684940078", 2024-09-17T07:19:50.917 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 217446 2024-09-17T07:19:50.917 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.917 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:19:50.917 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.917 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:19:50.917 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.702711+0000", 2024-09-17T07:19:50.917 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:50.917 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.918 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.918 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:19:50.918 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.702711+0000", 2024-09-17T07:19:50.918 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:50.918 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.918 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.918 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:19:50.918 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.702710+0000", 2024-09-17T07:19:50.918 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999981 2024-09-17T07:19:50.918 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.918 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.918 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:19:50.918 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.702717+0000", 2024-09-17T07:19:50.919 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.4200000000000001e-06 2024-09-17T07:19:50.919 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.919 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.919 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:19:50.919 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.702729+0000", 2024-09-17T07:19:50.919 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.1745e-05 2024-09-17T07:19:50.919 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.919 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.919 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:19:50.919 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.702734+0000", 2024-09-17T07:19:50.920 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.4609999999999999e-06 2024-09-17T07:19:50.920 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.920 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.920 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:50.920 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.702777+0000", 2024-09-17T07:19:50.920 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.2725000000000002e-05 2024-09-17T07:19:50.920 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.920 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.920 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:19:50.920 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.702786+0000", 2024-09-17T07:19:50.920 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.9800000000000004e-06 2024-09-17T07:19:50.920 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.921 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.921 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:50.921 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.308766+0000", 2024-09-17T07:19:50.921 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.60598020500000005 2024-09-17T07:19:50.921 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.921 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.921 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:19:50.921 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.308864+0000", 2024-09-17T07:19:50.921 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.7744000000000005e-05 2024-09-17T07:19:50.921 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.921 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.921 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 2,4", 2024-09-17T07:19:50.921 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.309012+0000", 2024-09-17T07:19:50.921 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00014835799999999999 2024-09-17T07:19:50.921 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.922 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.922 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:19:50.922 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.311851+0000", 2024-09-17T07:19:50.922 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0028390910000000002 2024-09-17T07:19:50.922 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.922 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.922 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:50.922 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.312617+0000", 2024-09-17T07:19:50.922 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00076543300000000004 2024-09-17T07:19:50.922 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.922 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.922 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:50.922 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.315147+0000", 2024-09-17T07:19:50.922 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.002530245 2024-09-17T07:19:50.923 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.923 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.923 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:19:50.923 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.315178+0000", 2024-09-17T07:19:50.923 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.0966000000000001e-05 2024-09-17T07:19:50.923 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.923 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.923 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:19:50.923 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.315251+0000", 2024-09-17T07:19:50.923 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.3353000000000001e-05 2024-09-17T07:19:50.923 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:50.924 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:19:50.924 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:50.924 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.924 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.924 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.15648.0:217447 10.1f 10:fc194ae6:::benchmark_data_smithi100_155039_object94099:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e369)", 2024-09-17T07:19:50.924 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:19:14.703309+0000", 2024-09-17T07:19:50.924 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 36.093067253000001, 2024-09-17T07:19:50.924 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.61677870599999995, 2024-09-17T07:19:50.924 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:19:50.924 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:19:50.924 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:19:50.924 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.15648", 2024-09-17T07:19:50.925 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/1684940078", 2024-09-17T07:19:50.925 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 217447 2024-09-17T07:19:50.925 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.925 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:19:50.925 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.925 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:19:50.925 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.703309+0000", 2024-09-17T07:19:50.925 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:50.925 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.925 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.925 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:19:50.925 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.703309+0000", 2024-09-17T07:19:50.925 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:50.925 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.926 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.926 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:19:50.926 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.703308+0000", 2024-09-17T07:19:50.926 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.999999 2024-09-17T07:19:50.926 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.926 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.926 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:19:50.926 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.703317+0000", 2024-09-17T07:19:50.926 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.0149999999999992e-06 2024-09-17T07:19:50.926 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.926 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.926 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:19:50.926 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.703330+0000", 2024-09-17T07:19:50.927 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.2184999999999999e-05 2024-09-17T07:19:50.927 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.927 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.927 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:19:50.927 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.703336+0000", 2024-09-17T07:19:50.927 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.8170000000000001e-06 2024-09-17T07:19:50.927 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.927 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.927 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:50.927 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.703368+0000", 2024-09-17T07:19:50.927 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.1795999999999998e-05 2024-09-17T07:19:50.928 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.928 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.928 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:19:50.928 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:14.703373+0000", 2024-09-17T07:19:50.928 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.8250000000000004e-06 2024-09-17T07:19:50.928 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.928 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.928 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:50.928 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.310044+0000", 2024-09-17T07:19:50.928 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.60667083200000005 2024-09-17T07:19:50.928 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.928 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.928 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:19:50.929 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.310140+0000", 2024-09-17T07:19:50.929 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.6194999999999996e-05 2024-09-17T07:19:50.929 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.929 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.929 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 2,4", 2024-09-17T07:19:50.929 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.310292+0000", 2024-09-17T07:19:50.929 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000152306 2024-09-17T07:19:50.929 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.929 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.929 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:19:50.929 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.311934+0000", 2024-09-17T07:19:50.930 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0016417999999999999 2024-09-17T07:19:50.930 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.930 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.930 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:50.930 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.312802+0000", 2024-09-17T07:19:50.930 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00086804600000000003 2024-09-17T07:19:50.930 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.930 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.931 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:50.931 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.319939+0000", 2024-09-17T07:19:50.931 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0071367319999999998 2024-09-17T07:19:50.931 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.931 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.931 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:19:50.931 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.320002+0000", 2024-09-17T07:19:50.931 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.3250999999999995e-05 2024-09-17T07:19:50.931 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.931 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.931 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:19:50.932 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.320088+0000", 2024-09-17T07:19:50.932 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.5779999999999995e-05 2024-09-17T07:19:50.932 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:50.932 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:19:50.932 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:50.932 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.932 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.933 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.15648.0:217457 10.23 10:c41c5a9c:::benchmark_data_smithi100_155039_object94109:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-09-17T07:19:50.933 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:19:15.315406+0000", 2024-09-17T07:19:50.933 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 35.480970882999998, 2024-09-17T07:19:50.933 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.99254916699999995, 2024-09-17T07:19:50.933 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:19:50.933 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:19:50.934 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:19:50.934 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.15648", 2024-09-17T07:19:50.934 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/1684940078", 2024-09-17T07:19:50.934 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 217457 2024-09-17T07:19:50.934 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.934 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:19:50.935 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.935 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:19:50.935 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.315406+0000", 2024-09-17T07:19:50.935 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:50.935 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.935 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.936 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:19:50.936 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.315406+0000", 2024-09-17T07:19:50.936 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:50.936 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.936 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.936 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:19:50.936 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.315404+0000", 2024-09-17T07:19:50.937 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:19:50.937 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.937 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.937 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:19:50.937 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.315414+0000", 2024-09-17T07:19:50.937 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.7149999999999993e-06 2024-09-17T07:19:50.937 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.938 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.938 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:19:50.938 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.315423+0000", 2024-09-17T07:19:50.938 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.3810000000000005e-06 2024-09-17T07:19:50.938 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.938 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.939 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:19:50.939 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.315430+0000", 2024-09-17T07:19:50.939 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.5030000000000002e-06 2024-09-17T07:19:50.939 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.939 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.939 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:50.940 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.315505+0000", 2024-09-17T07:19:50.940 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.5263000000000001e-05 2024-09-17T07:19:50.940 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.940 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.940 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:19:50.940 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.315514+0000", 2024-09-17T07:19:50.940 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.9080000000000001e-06 2024-09-17T07:19:50.940 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.941 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.941 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:50.941 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:16.300779+0000", 2024-09-17T07:19:50.941 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.98526500800000005 2024-09-17T07:19:50.941 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.941 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.942 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:19:50.942 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:16.300899+0000", 2024-09-17T07:19:50.942 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000119541 2024-09-17T07:19:50.942 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.942 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.942 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 4,5", 2024-09-17T07:19:50.942 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:16.301115+0000", 2024-09-17T07:19:50.942 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000216648 2024-09-17T07:19:50.943 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.943 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.943 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:19:50.943 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:16.306840+0000", 2024-09-17T07:19:50.943 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0057248029999999997 2024-09-17T07:19:50.943 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.943 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.943 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:50.944 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:16.306961+0000", 2024-09-17T07:19:50.944 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000120773 2024-09-17T07:19:50.944 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.944 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.944 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:50.944 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:16.307876+0000", 2024-09-17T07:19:50.944 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00091544599999999999 2024-09-17T07:19:50.944 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.945 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.945 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:19:50.945 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:16.307922+0000", 2024-09-17T07:19:50.945 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.5238999999999998e-05 2024-09-17T07:19:50.945 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.945 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.946 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:19:50.946 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:16.307955+0000", 2024-09-17T07:19:50.946 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.3229000000000003e-05 2024-09-17T07:19:50.946 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:50.946 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:19:50.946 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:50.946 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.946 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.946 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.15648.0:217492 10.23 10:c79ae830:::benchmark_data_smithi100_155039_object94144:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-09-17T07:19:50.946 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:19:15.333488+0000", 2024-09-17T07:19:50.946 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 35.462888145000001, 2024-09-17T07:19:50.946 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.983869879, 2024-09-17T07:19:50.946 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:19:50.947 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:19:50.947 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:19:50.947 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.15648", 2024-09-17T07:19:50.947 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/1684940078", 2024-09-17T07:19:50.947 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 217492 2024-09-17T07:19:50.947 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.947 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:19:50.947 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.947 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:19:50.947 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.333488+0000", 2024-09-17T07:19:50.947 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:50.947 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.948 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.948 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:19:50.948 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.333488+0000", 2024-09-17T07:19:50.948 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:50.948 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.948 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.948 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:19:50.948 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.333488+0000", 2024-09-17T07:19:50.948 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.999999 2024-09-17T07:19:50.948 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.948 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.948 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:19:50.948 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.333492+0000", 2024-09-17T07:19:50.949 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.8409999999999999e-06 2024-09-17T07:19:50.949 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.949 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.949 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:19:50.949 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.333514+0000", 2024-09-17T07:19:50.949 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.1447e-05 2024-09-17T07:19:50.949 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.949 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.949 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:19:50.949 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.333521+0000", 2024-09-17T07:19:50.949 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.9720000000000003e-06 2024-09-17T07:19:50.949 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.949 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.950 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:50.950 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.333558+0000", 2024-09-17T07:19:50.950 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.6829000000000003e-05 2024-09-17T07:19:50.950 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.950 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.950 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:19:50.950 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.333565+0000", 2024-09-17T07:19:50.951 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.6920000000000002e-06 2024-09-17T07:19:50.951 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.951 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.951 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:50.951 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:16.307998+0000", 2024-09-17T07:19:50.951 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.97443299800000005 2024-09-17T07:19:50.951 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.951 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.951 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:19:50.951 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:16.308089+0000", 2024-09-17T07:19:50.951 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.1129000000000001e-05 2024-09-17T07:19:50.951 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.952 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.952 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 4,5", 2024-09-17T07:19:50.952 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:16.308332+0000", 2024-09-17T07:19:50.952 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00024259499999999999 2024-09-17T07:19:50.952 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.952 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.952 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:19:50.952 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:16.315310+0000", 2024-09-17T07:19:50.952 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0069775879999999998 2024-09-17T07:19:50.952 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.952 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.952 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:50.953 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:16.317095+0000", 2024-09-17T07:19:50.953 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.001785317 2024-09-17T07:19:50.953 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.953 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.953 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:50.953 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:16.317270+0000", 2024-09-17T07:19:50.953 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00017467200000000001 2024-09-17T07:19:50.954 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.954 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.954 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:19:50.954 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:16.317319+0000", 2024-09-17T07:19:50.954 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.9243999999999998e-05 2024-09-17T07:19:50.954 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.955 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.955 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:19:50.955 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:16.317358+0000", 2024-09-17T07:19:50.955 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.9338999999999997e-05 2024-09-17T07:19:50.955 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:50.955 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:19:50.956 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:50.956 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.956 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.956 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.15648.0:217513 10.23 10:c7aa9836:::benchmark_data_smithi100_155039_object94165:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-09-17T07:19:50.956 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:19:15.345458+0000", 2024-09-17T07:19:50.957 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 35.450918823000002, 2024-09-17T07:19:50.957 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.97663500599999997, 2024-09-17T07:19:50.957 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:19:50.957 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:19:50.957 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:19:50.957 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.15648", 2024-09-17T07:19:50.957 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/1684940078", 2024-09-17T07:19:50.958 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 217513 2024-09-17T07:19:50.958 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.958 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:19:50.958 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.958 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:19:50.958 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.345458+0000", 2024-09-17T07:19:50.959 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:50.959 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.959 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.959 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:19:50.959 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.345458+0000", 2024-09-17T07:19:50.959 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:50.959 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.960 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.960 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:19:50.960 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.345456+0000", 2024-09-17T07:19:50.960 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:19:50.960 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.961 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.961 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:19:50.961 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.345463+0000", 2024-09-17T07:19:50.961 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.5930000000000002e-06 2024-09-17T07:19:50.961 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.961 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.961 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:19:50.961 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.345474+0000", 2024-09-17T07:19:50.962 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0834e-05 2024-09-17T07:19:50.962 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.962 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.962 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:19:50.962 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.345481+0000", 2024-09-17T07:19:50.962 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.5209999999999997e-06 2024-09-17T07:19:50.962 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.963 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.963 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:50.963 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.345531+0000", 2024-09-17T07:19:50.963 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.9969000000000002e-05 2024-09-17T07:19:50.963 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.963 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.963 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:19:50.963 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.345541+0000", 2024-09-17T07:19:50.964 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.8779999999999993e-06 2024-09-17T07:19:50.964 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.964 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.964 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:50.964 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:16.310592+0000", 2024-09-17T07:19:50.964 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.96505050000000003 2024-09-17T07:19:50.964 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.964 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.965 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:19:50.965 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:16.310690+0000", 2024-09-17T07:19:50.965 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.8884999999999999e-05 2024-09-17T07:19:50.965 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.965 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.965 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 4,5", 2024-09-17T07:19:50.966 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:16.310939+0000", 2024-09-17T07:19:50.966 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00024887800000000001 2024-09-17T07:19:50.966 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.966 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.966 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:19:50.966 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:16.317059+0000", 2024-09-17T07:19:50.966 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0061198019999999997 2024-09-17T07:19:50.966 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.966 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.967 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:50.967 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:16.321912+0000", 2024-09-17T07:19:50.967 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0048527900000000001 2024-09-17T07:19:50.967 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.967 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.967 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:50.967 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:16.322008+0000", 2024-09-17T07:19:50.967 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.5799999999999998e-05 2024-09-17T07:19:50.967 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.967 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.967 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:19:50.967 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:16.322057+0000", 2024-09-17T07:19:50.967 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.9011000000000002e-05 2024-09-17T07:19:50.967 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.968 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.968 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:19:50.968 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:16.322093+0000", 2024-09-17T07:19:50.968 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.6000000000000001e-05 2024-09-17T07:19:50.968 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:50.968 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:19:50.968 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:50.968 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.968 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.968 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.15648.0:217575 10.23 10:c6ffe526:::benchmark_data_smithi100_155039_object94227:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-09-17T07:19:50.968 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:19:15.409031+0000", 2024-09-17T07:19:50.968 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 35.387345828999997, 2024-09-17T07:19:50.969 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.91334559599999998, 2024-09-17T07:19:50.969 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:19:50.969 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:19:50.969 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:19:50.969 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.15648", 2024-09-17T07:19:50.969 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/1684940078", 2024-09-17T07:19:50.969 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 217575 2024-09-17T07:19:50.969 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.969 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:19:50.969 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.970 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:19:50.970 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.409031+0000", 2024-09-17T07:19:50.970 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:50.970 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.970 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.970 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:19:50.970 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.409031+0000", 2024-09-17T07:19:50.970 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:50.970 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.970 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.970 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:19:50.970 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.409029+0000", 2024-09-17T07:19:50.970 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:19:50.971 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.971 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.971 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:19:50.971 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.409036+0000", 2024-09-17T07:19:50.971 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.9480000000000002e-06 2024-09-17T07:19:50.971 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.971 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.971 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:19:50.971 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.409050+0000", 2024-09-17T07:19:50.971 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.3558e-05 2024-09-17T07:19:50.972 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.972 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.972 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:19:50.972 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.409058+0000", 2024-09-17T07:19:50.972 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.354e-06 2024-09-17T07:19:50.972 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.972 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.972 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:50.973 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.409095+0000", 2024-09-17T07:19:50.973 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.6872000000000002e-05 2024-09-17T07:19:50.973 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.973 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.973 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:19:50.973 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.409102+0000", 2024-09-17T07:19:50.973 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.9530000000000002e-06 2024-09-17T07:19:50.973 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.973 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.973 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:50.973 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:16.313638+0000", 2024-09-17T07:19:50.973 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.90453610100000004 2024-09-17T07:19:50.974 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.974 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.974 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:19:50.974 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:16.313731+0000", 2024-09-17T07:19:50.974 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.2880000000000005e-05 2024-09-17T07:19:50.974 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.974 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.974 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 4,5", 2024-09-17T07:19:50.974 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:16.314008+0000", 2024-09-17T07:19:50.974 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00027651299999999999 2024-09-17T07:19:50.974 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.974 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.974 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:19:50.974 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:16.317102+0000", 2024-09-17T07:19:50.975 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0030940999999999998 2024-09-17T07:19:50.975 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.975 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.975 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:50.975 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:16.321950+0000", 2024-09-17T07:19:50.975 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0048486240000000002 2024-09-17T07:19:50.975 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.975 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.975 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:50.975 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:16.322296+0000", 2024-09-17T07:19:50.975 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000345242 2024-09-17T07:19:50.975 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.975 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.976 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:19:50.976 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:16.322341+0000", 2024-09-17T07:19:50.976 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.5426999999999997e-05 2024-09-17T07:19:50.976 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.976 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.976 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:19:50.976 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:16.322376+0000", 2024-09-17T07:19:50.976 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.5293000000000002e-05 2024-09-17T07:19:50.976 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:50.976 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:19:50.976 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:50.976 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.977 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.977 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.15648.0:243010 10.c 10:30477899:::benchmark_data_smithi100_155039_object119662:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e376)", 2024-09-17T07:19:50.977 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:19:33.355816+0000", 2024-09-17T07:19:50.977 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 17.440560827999999, 2024-09-17T07:19:50.977 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.17590839699999999, 2024-09-17T07:19:50.977 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:19:50.977 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:19:50.977 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:19:50.977 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.15648", 2024-09-17T07:19:50.977 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/1684940078", 2024-09-17T07:19:50.978 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 243010 2024-09-17T07:19:50.978 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.978 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:19:50.978 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.978 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:19:50.978 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:33.355816+0000", 2024-09-17T07:19:50.978 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:50.979 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.979 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.979 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:19:50.979 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:33.355816+0000", 2024-09-17T07:19:50.979 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:19:50.979 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.979 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.979 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:19:50.979 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:33.355814+0000", 2024-09-17T07:19:50.979 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:19:50.979 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.979 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.979 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:19:50.980 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:33.355820+0000", 2024-09-17T07:19:50.980 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.0979999999999999e-06 2024-09-17T07:19:50.980 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.980 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.980 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:19:50.980 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:33.355832+0000", 2024-09-17T07:19:50.980 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.1865000000000001e-05 2024-09-17T07:19:50.980 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.980 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.980 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:19:50.980 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:33.355839+0000", 2024-09-17T07:19:50.981 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.2479999999999999e-06 2024-09-17T07:19:50.981 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.981 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.981 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:19:50.981 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:33.355904+0000", 2024-09-17T07:19:50.981 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.4826999999999994e-05 2024-09-17T07:19:50.981 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.981 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.981 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:19:50.981 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:33.356060+0000", 2024-09-17T07:19:50.981 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00015597399999999999 2024-09-17T07:19:50.981 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.981 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.982 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,4", 2024-09-17T07:19:50.982 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:33.356427+0000", 2024-09-17T07:19:50.982 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000366819 2024-09-17T07:19:50.982 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.982 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.982 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:19:50.982 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:33.358179+0000", 2024-09-17T07:19:50.982 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0017523930000000001 2024-09-17T07:19:50.982 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.982 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.982 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:50.982 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:33.358861+0000", 2024-09-17T07:19:50.983 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00068192999999999997 2024-09-17T07:19:50.983 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.983 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.983 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:19:50.983 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:33.531638+0000", 2024-09-17T07:19:50.983 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.17277683099999999 2024-09-17T07:19:50.983 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.983 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.983 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:19:50.983 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:33.531686+0000", 2024-09-17T07:19:50.983 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.8038000000000003e-05 2024-09-17T07:19:50.983 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:19:50.983 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:19:50.984 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:19:50.984 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:33.531724+0000", 2024-09-17T07:19:50.984 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.7842999999999997e-05 2024-09-17T07:19:50.984 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:50.984 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:19:50.984 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:50.984 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:19:50.984 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:19:50.984 INFO:teuthology.orchestra.run.smithi135.stdout:} 2024-09-17T07:19:50.984 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:19:51.006 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:51.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:51.007 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:19:51.108 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:19:51.130 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:51.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:51.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:19:51.232 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:19:51.254 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:51.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:51.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:19:51.356 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:19:51.378 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:51.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:51.379 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:19:51.448 INFO:teuthology.orchestra.run.smithi100.stderr:nodeep-scrub is set 2024-09-17T07:19:51.480 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:19:51.490 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:19:51.491 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:19:51.501 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:51.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:51.502 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:19:51.603 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:19:51.624 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:51.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:51.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:19:51.683 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:19:51.726 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:19:51.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:51.749 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:51.749 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:19:51.754 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:51 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/93351655' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:19:51.754 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:51 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:19:51.754 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:51 smithi135 ceph-mon[33238]: pgmap v825: 142 pgs: 31 active+clean+remapped, 24 active+undersized, 17 active+undersized+degraded, 70 active+clean; 582 KiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 447 B/s rd, 0 op/s; 73/588 objects degraded (12.415%); 48/588 objects misplaced (8.163%) 2024-09-17T07:19:51.796 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:51 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/93351655' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:19:51.796 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:51 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:19:51.796 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:51 smithi100 ceph-mon[39462]: pgmap v825: 142 pgs: 31 active+clean+remapped, 24 active+undersized, 17 active+undersized+degraded, 70 active+clean; 582 KiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 447 B/s rd, 0 op/s; 73/588 objects degraded (12.415%); 48/588 objects misplaced (8.163%) 2024-09-17T07:19:51.797 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:51 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/93351655' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:19:51.797 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:51 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:19:51.797 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:51 smithi100 ceph-mon[29092]: pgmap v825: 142 pgs: 31 active+clean+remapped, 24 active+undersized, 17 active+undersized+degraded, 70 active+clean; 582 KiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 447 B/s rd, 0 op/s; 73/588 objects degraded (12.415%); 48/588 objects misplaced (8.163%) 2024-09-17T07:19:51.850 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:19:51.872 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:51.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:51.873 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:19:51.974 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:19:51.994 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:51.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:51.995 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:19:52.096 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:19:52.118 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:52.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:52.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:19:52.197 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-17T07:19:52.220 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:19:52.241 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:52.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:52.242 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:19:52.343 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:19:52.365 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:52.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:52.366 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:19:52.466 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:19:52.487 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:52.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:52.488 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:19:52.568 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:52 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:19:52.568 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:52 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:19:52.568 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:52 smithi100 ceph-mon[39462]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:19:52.569 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:52 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T07:19:52.569 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:52 smithi100 ceph-mon[39462]: osdmap e389: 8 total, 7 up, 6 in 2024-09-17T07:19:52.569 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:52 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:19:52.569 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:52 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:19:52.569 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:52 smithi100 ceph-mon[29092]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:19:52.569 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:52 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T07:19:52.569 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:52 smithi100 ceph-mon[29092]: osdmap e389: 8 total, 7 up, 6 in 2024-09-17T07:19:52.590 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:19:52.611 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:52.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:52.612 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:19:52.713 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:19:52.734 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:52.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:52.736 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:19:52.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:52 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:19:52.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:52 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:19:52.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:52 smithi135 ceph-mon[33238]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:19:52.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:52 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T07:19:52.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:52 smithi135 ceph-mon[33238]: osdmap e389: 8 total, 7 up, 6 in 2024-09-17T07:19:52.836 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:19:52.857 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:52.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:52.858 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:19:52.959 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:19:52.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:52.981 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:52.981 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:19:53.082 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:19:53.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:53.107 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:53.107 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:19:53.207 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:19:53.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:53.230 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:53.230 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:19:53.331 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:19:53.353 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:53.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:53.354 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:19:53.455 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:19:53.476 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:53.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:53.477 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:19:53.491 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:19:53.577 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:19:53.598 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:53.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:53.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:19:53.700 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:19:53.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:53.722 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:53.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:19:53.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:53 smithi135 ceph-mon[33238]: pgmap v827: 142 pgs: 31 active+clean+remapped, 24 active+undersized, 17 active+undersized+degraded, 70 active+clean; 582 KiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 639 B/s rd, 0 op/s; 73/588 objects degraded (12.415%); 48/588 objects misplaced (8.163%); 13 B/s, 0 objects/s recovering 2024-09-17T07:19:53.800 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:53 smithi100 ceph-mon[29092]: pgmap v827: 142 pgs: 31 active+clean+remapped, 24 active+undersized, 17 active+undersized+degraded, 70 active+clean; 582 KiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 639 B/s rd, 0 op/s; 73/588 objects degraded (12.415%); 48/588 objects misplaced (8.163%); 13 B/s, 0 objects/s recovering 2024-09-17T07:19:53.800 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:53 smithi100 ceph-mon[39462]: pgmap v827: 142 pgs: 31 active+clean+remapped, 24 active+undersized, 17 active+undersized+degraded, 70 active+clean; 582 KiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 639 B/s rd, 0 op/s; 73/588 objects degraded (12.415%); 48/588 objects misplaced (8.163%); 13 B/s, 0 objects/s recovering 2024-09-17T07:19:53.823 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:19:53.844 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:53.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:53.845 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:19:53.893 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:19:53.934 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:19:53.946 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:19:53.967 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:53.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:53.968 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:19:54.069 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:19:54.090 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:54.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:54.091 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:19:54.192 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:19:54.215 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:54.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:54.216 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:19:54.317 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:19:54.339 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:54.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:54.340 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:19:54.441 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:19:54.463 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T07:19:54.465 INFO:teuthology.orchestra.run.smithi100.stderr:pool 'unique_pool_4' created 2024-09-17T07:19:54.468 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:54.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:54.470 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:19:54.571 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:19:54.593 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:54.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:54.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:19:54.696 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:19:54.718 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:54.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:54.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:19:54.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:54 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/155181620' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-09-17T07:19:54.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:54 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:19:54.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:54 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:19:54.821 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:19:54.842 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:54.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:54.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:19:54.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:54 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/155181620' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-09-17T07:19:54.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:54 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:19:54.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:54 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:19:54.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:54 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/155181620' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-09-17T07:19:54.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:54 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:19:54.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:54 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:19:54.945 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:19:54.966 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:54.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:54.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:19:55.068 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:19:55.091 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:55.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:55.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:19:55.193 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:19:55.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:55.217 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:55.217 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:19:55.318 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:19:55.341 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:55.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:55.342 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:19:55.442 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:19:55.466 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:55.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:55.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:19:55.568 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:19:55.591 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:55.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:55.591 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:19:55.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:55 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/155181620' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-09-17T07:19:55.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:55 smithi100 ceph-mon[39462]: osdmap e390: 8 total, 7 up, 6 in 2024-09-17T07:19:55.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:55 smithi100 ceph-mon[39462]: pgmap v829: 158 pgs: 4 creating+peering, 12 unknown, 31 active+clean+remapped, 24 active+undersized, 17 active+undersized+degraded, 70 active+clean; 582 KiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 op/s; 73/588 objects degraded (12.415%); 48/588 objects misplaced (8.163%) 2024-09-17T07:19:55.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:55 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/155181620' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-09-17T07:19:55.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:55 smithi100 ceph-mon[29092]: osdmap e390: 8 total, 7 up, 6 in 2024-09-17T07:19:55.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:55 smithi100 ceph-mon[29092]: pgmap v829: 158 pgs: 4 creating+peering, 12 unknown, 31 active+clean+remapped, 24 active+undersized, 17 active+undersized+degraded, 70 active+clean; 582 KiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 op/s; 73/588 objects degraded (12.415%); 48/588 objects misplaced (8.163%) 2024-09-17T07:19:55.692 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:19:55.714 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:55.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:55.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:19:55.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:55 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/155181620' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-09-17T07:19:55.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:55 smithi135 ceph-mon[33238]: osdmap e390: 8 total, 7 up, 6 in 2024-09-17T07:19:55.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:55 smithi135 ceph-mon[33238]: pgmap v829: 158 pgs: 4 creating+peering, 12 unknown, 31 active+clean+remapped, 24 active+undersized, 17 active+undersized+degraded, 70 active+clean; 582 KiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 op/s; 73/588 objects degraded (12.415%); 48/588 objects misplaced (8.163%) 2024-09-17T07:19:55.816 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:19:55.836 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:55.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:55.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:19:55.914 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:19:55 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:19:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:19:55.939 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:19:55.961 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:55.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:55.962 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:19:56.063 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:19:56.085 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:56.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:56.086 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:19:56.172 INFO:teuthology.orchestra.run.smithi100.stdout:moved 0 / 474 (0%) 2024-09-17T07:19:56.172 INFO:teuthology.orchestra.run.smithi100.stdout:avg 94.8 2024-09-17T07:19:56.172 INFO:teuthology.orchestra.run.smithi100.stdout:stddev 17.1184 -> 17.1184 (expected baseline 8.70862) 2024-09-17T07:19:56.172 INFO:teuthology.orchestra.run.smithi100.stdout:min osd.4 with 74 -> 74 pgs (0.780591 -> 0.780591 * mean) 2024-09-17T07:19:56.172 INFO:teuthology.orchestra.run.smithi100.stdout:max osd.6 with 83 -> 83 pgs (0.875527 -> 0.875527 * mean) 2024-09-17T07:19:56.172 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T07:19:56.172 INFO:teuthology.orchestra.run.smithi100.stdout:oload 1000 2024-09-17T07:19:56.173 INFO:teuthology.orchestra.run.smithi100.stdout:max_change 3 2024-09-17T07:19:56.173 INFO:teuthology.orchestra.run.smithi100.stdout:max_change_osds 4 2024-09-17T07:19:56.173 INFO:teuthology.orchestra.run.smithi100.stdout:average_utilization 0.0083 2024-09-17T07:19:56.173 INFO:teuthology.orchestra.run.smithi100.stdout:overload_utilization 0.0827 2024-09-17T07:19:56.173 INFO:teuthology.orchestra.run.smithi100.stderr:no change 2024-09-17T07:19:56.187 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:19:56.209 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:56.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:56.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:19:56.311 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:19:56.333 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:56.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:56.334 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:19:56.435 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:19:56.457 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:56.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:56.457 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:19:56.558 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:19:56.579 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:56.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:56.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:19:56.681 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:19:56.703 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:56.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:56.704 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:19:56.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:56 smithi135 ceph-mon[33238]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-17T07:19:56.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:56 smithi135 ceph-mon[33238]: osdmap e391: 8 total, 7 up, 6 in 2024-09-17T07:19:56.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:56 smithi135 ceph-mon[33238]: from='client.25784 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:19:56.804 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:19:56.831 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:56.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:56.832 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:19:56.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:56 smithi100 ceph-mon[29092]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-17T07:19:56.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:56 smithi100 ceph-mon[29092]: osdmap e391: 8 total, 7 up, 6 in 2024-09-17T07:19:56.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:56 smithi100 ceph-mon[29092]: from='client.25784 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:19:56.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:56 smithi100 ceph-mon[39462]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-17T07:19:56.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:56 smithi100 ceph-mon[39462]: osdmap e391: 8 total, 7 up, 6 in 2024-09-17T07:19:56.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:56 smithi100 ceph-mon[39462]: from='client.25784 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:19:56.933 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:19:56.955 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:56.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:56.956 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:19:57.056 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:19:57.077 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:57.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:57.078 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:19:57.179 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:19:57.200 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:57.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:57.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:19:57.302 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:19:57.329 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:57.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:57.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:19:57.430 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:19:57.452 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:57.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:57.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:19:57.554 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:19:57.575 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:57.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:57.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:19:57.677 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:19:57.698 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:57.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:57.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:19:57.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:57 smithi135 ceph-mon[33238]: osdmap e392: 8 total, 7 up, 6 in 2024-09-17T07:19:57.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:57 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:19:57.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:57 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:19:57.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:57 smithi135 ceph-mon[33238]: pgmap v832: 158 pgs: 4 creating+peering, 12 unknown, 31 active+clean+remapped, 24 active+undersized, 17 active+undersized+degraded, 70 active+clean; 582 KiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 863 B/s rd, 0 op/s; 73/588 objects degraded (12.415%); 48/588 objects misplaced (8.163%) 2024-09-17T07:19:57.801 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:19:57.823 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:57.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:57.824 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:19:57.876 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:57 smithi100 ceph-mon[39462]: osdmap e392: 8 total, 7 up, 6 in 2024-09-17T07:19:57.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:57 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:19:57.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:57 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:19:57.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:57 smithi100 ceph-mon[39462]: pgmap v832: 158 pgs: 4 creating+peering, 12 unknown, 31 active+clean+remapped, 24 active+undersized, 17 active+undersized+degraded, 70 active+clean; 582 KiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 863 B/s rd, 0 op/s; 73/588 objects degraded (12.415%); 48/588 objects misplaced (8.163%) 2024-09-17T07:19:57.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:57 smithi100 ceph-mon[29092]: osdmap e392: 8 total, 7 up, 6 in 2024-09-17T07:19:57.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:57 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:19:57.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:57 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:19:57.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:57 smithi100 ceph-mon[29092]: pgmap v832: 158 pgs: 4 creating+peering, 12 unknown, 31 active+clean+remapped, 24 active+undersized, 17 active+undersized+degraded, 70 active+clean; 582 KiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 863 B/s rd, 0 op/s; 73/588 objects degraded (12.415%); 48/588 objects misplaced (8.163%) 2024-09-17T07:19:57.926 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:19:57.947 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:57.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:57.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:19:58.048 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:19:58.069 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:58.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:58.070 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:19:58.171 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:19:58.192 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:58.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:58.193 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:19:58.227 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-09-17T07:19:58.227 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd test-reweight-by-utilization 110 3.0 2024-09-17T07:19:58.294 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:19:58.316 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:58.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:58.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:19:58.418 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:19:58.440 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:58.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:58.441 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:19:58.542 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:19:58.564 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:58.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:58.565 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:19:58.666 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:19:58.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:58.688 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:58.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:19:58.789 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:19:58.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:58 smithi135 ceph-mon[33238]: osdmap e393: 8 total, 7 up, 6 in 2024-09-17T07:19:58.810 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:58.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:58.811 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:19:58.821 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:58 smithi100 ceph-mon[29092]: osdmap e393: 8 total, 7 up, 6 in 2024-09-17T07:19:58.821 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:58 smithi100 ceph-mon[39462]: osdmap e393: 8 total, 7 up, 6 in 2024-09-17T07:19:58.912 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:19:58.934 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:58.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:58.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:19:59.036 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:19:59.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:59.058 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:59.058 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:19:59.158 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:19:59.180 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:59.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:59.181 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:19:59.281 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:19:59.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:59.304 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:59.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:19:59.405 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:19:59.426 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:59.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:59.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:19:59.484 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:19:59 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:19:59.529 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:19:59.552 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:59.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:59.553 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:19:59.654 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:19:59.676 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:19:59.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:59.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:19:59.777 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:19:59.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:59 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:19:59.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:19:59 smithi135 ceph-mon[33238]: pgmap v834: 158 pgs: 4 creating+peering, 31 active+clean+remapped, 32 active+undersized, 17 active+undersized+degraded, 74 active+clean; 582 KiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 1.0 KiB/s rd, 1 op/s; 73/588 objects degraded (12.415%); 48/588 objects misplaced (8.163%) 2024-09-17T07:19:59.799 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:59.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:59.800 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:19:59.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:59 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:19:59.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:19:59 smithi100 ceph-mon[29092]: pgmap v834: 158 pgs: 4 creating+peering, 31 active+clean+remapped, 32 active+undersized, 17 active+undersized+degraded, 74 active+clean; 582 KiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 1.0 KiB/s rd, 1 op/s; 73/588 objects degraded (12.415%); 48/588 objects misplaced (8.163%) 2024-09-17T07:19:59.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:59 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:19:59.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:19:59 smithi100 ceph-mon[39462]: pgmap v834: 158 pgs: 4 creating+peering, 31 active+clean+remapped, 32 active+undersized, 17 active+undersized+degraded, 74 active+clean; 582 KiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 1.0 KiB/s rd, 1 op/s; 73/588 objects degraded (12.415%); 48/588 objects misplaced (8.163%) 2024-09-17T07:19:59.901 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:19:59.903 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd pool application enable unique_pool_4 rados --yes-i-really-mean-it || true 2024-09-17T07:19:59.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:19:59.922 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:19:59.923 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:20:00.024 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:20:00.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:00.047 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:00.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:20:00.148 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:20:00.171 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:00.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:00.172 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:20:00.219 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:20:00.219 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:20:00.220 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:20:00.220 INFO:teuthology.orchestra.run.smithi100.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-17T07:20:00.220 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:20:00.221 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:20:00.221 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:20:00.237 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:20:00.237 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:20:00.255 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:20:00.255 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:20:00.272 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:20:00.273 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:20:00.274 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:20:00.295 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:20:00.295 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:20:00.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:00.297 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:00.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:20:00.398 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:20:00.420 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:00.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:00.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:20:00.521 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:20:00.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:00.544 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:00.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:20:00.645 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:20:00.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:00.668 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:00.668 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:20:00.769 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:20:00.791 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:00.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:00.792 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:20:00.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:00 smithi135 ceph-mon[33238]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set; 1 osds down; Degraded data redundancy: 73/588 objects degraded (12.415%), 17 pgs degraded, 41 pgs undersized; 1 pool(s) do not have an application enabled 2024-09-17T07:20:00.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:00 smithi135 ceph-mon[33238]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-09-17T07:20:00.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:00 smithi135 ceph-mon[33238]: [WRN] OSD_DOWN: 1 osds down 2024-09-17T07:20:00.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:00 smithi135 ceph-mon[33238]: osd.1 (root=default,host=smithi100) is down 2024-09-17T07:20:00.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:00 smithi135 ceph-mon[33238]: [WRN] PG_DEGRADED: Degraded data redundancy: 73/588 objects degraded (12.415%), 17 pgs degraded, 41 pgs undersized 2024-09-17T07:20:00.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:00 smithi135 ceph-mon[33238]: pg 2.1 is stuck undersized for 3m, current state active+undersized, last acting [0,2] 2024-09-17T07:20:00.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:00 smithi135 ceph-mon[33238]: pg 2.2 is stuck undersized for 3m, current state active+undersized, last acting [5,6] 2024-09-17T07:20:00.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:00 smithi135 ceph-mon[33238]: pg 2.6 is stuck undersized for 3m, current state active+undersized, last acting [6,4] 2024-09-17T07:20:00.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:00 smithi135 ceph-mon[33238]: pg 2.d is stuck undersized for 3m, current state active+undersized, last acting [4,2] 2024-09-17T07:20:00.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:00 smithi135 ceph-mon[33238]: pg 2.10 is stuck undersized for 3m, current state active+undersized, last acting [2,0] 2024-09-17T07:20:00.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:00 smithi135 ceph-mon[33238]: pg 2.11 is stuck undersized for 3m, current state active+undersized, last acting [6,4] 2024-09-17T07:20:00.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:00 smithi135 ceph-mon[33238]: pg 2.15 is stuck undersized for 3m, current state active+undersized+degraded, last acting [5,0] 2024-09-17T07:20:00.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:00 smithi135 ceph-mon[33238]: pg 3.0 is stuck undersized for 3m, current state active+undersized+degraded, last acting [6,2] 2024-09-17T07:20:00.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:00 smithi135 ceph-mon[33238]: pg 4.2 is stuck undersized for 3m, current state active+undersized+degraded, last acting [5,4] 2024-09-17T07:20:00.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:00 smithi135 ceph-mon[33238]: pg 4.4 is stuck undersized for 3m, current state active+undersized+degraded, last acting [6,2] 2024-09-17T07:20:00.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:00 smithi135 ceph-mon[33238]: pg 4.7 is stuck undersized for 3m, current state active+undersized+degraded, last acting [5,0] 2024-09-17T07:20:00.795 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:00 smithi135 ceph-mon[33238]: pg 4.b is stuck undersized for 3m, current state active+undersized+degraded, last acting [0,4] 2024-09-17T07:20:00.795 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:00 smithi135 ceph-mon[33238]: pg 4.c is stuck undersized for 3m, current state active+undersized+degraded, last acting [4,6] 2024-09-17T07:20:00.795 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:00 smithi135 ceph-mon[33238]: pg 4.d is stuck undersized for 3m, current state active+undersized+degraded, last acting [4,2] 2024-09-17T07:20:00.795 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:00 smithi135 ceph-mon[33238]: pg 4.e is stuck undersized for 3m, current state active+undersized+degraded, last acting [4,6] 2024-09-17T07:20:00.795 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:00 smithi135 ceph-mon[33238]: pg 4.12 is stuck undersized for 3m, current state active+undersized+degraded, last acting [6,2] 2024-09-17T07:20:00.795 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:00 smithi135 ceph-mon[33238]: pg 4.13 is stuck undersized for 3m, current state active+undersized+degraded, last acting [4,6] 2024-09-17T07:20:00.795 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:00 smithi135 ceph-mon[33238]: pg 4.18 is stuck undersized for 3m, current state active+undersized+degraded, last acting [4,6] 2024-09-17T07:20:00.795 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:00 smithi135 ceph-mon[33238]: pg 4.1b is stuck undersized for 3m, current state active+undersized+degraded, last acting [4,2] 2024-09-17T07:20:00.795 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:00 smithi135 ceph-mon[33238]: pg 4.1c is stuck undersized for 3m, current state active+undersized+degraded, last acting [0,2] 2024-09-17T07:20:00.795 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:00 smithi135 ceph-mon[33238]: pg 4.1d is stuck undersized for 3m, current state active+undersized+degraded, last acting [4,6] 2024-09-17T07:20:00.795 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:00 smithi135 ceph-mon[33238]: pg 5.0 is stuck undersized for 3m, current state active+undersized+degraded, last acting [4,0] 2024-09-17T07:20:00.795 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:00 smithi135 ceph-mon[33238]: pg 5.5 is stuck undersized for 3m, current state active+undersized, last acting [0,4] 2024-09-17T07:20:00.795 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:00 smithi135 ceph-mon[33238]: pg 5.7 is stuck undersized for 3m, current state active+undersized, last acting [5,0] 2024-09-17T07:20:00.795 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:00 smithi135 ceph-mon[33238]: pg 5.8 is stuck undersized for 3m, current state active+undersized, last acting [0,2] 2024-09-17T07:20:00.796 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:00 smithi135 ceph-mon[33238]: pg 5.c is stuck undersized for 3m, current state active+undersized, last acting [4,0] 2024-09-17T07:20:00.796 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:00 smithi135 ceph-mon[33238]: pg 5.13 is stuck undersized for 3m, current state active+undersized, last acting [0,5] 2024-09-17T07:20:00.796 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:00 smithi135 ceph-mon[33238]: pg 5.15 is stuck undersized for 3m, current state active+undersized+degraded, last acting [5,0] 2024-09-17T07:20:00.796 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:00 smithi135 ceph-mon[33238]: pg 5.16 is stuck undersized for 3m, current state active+undersized, last acting [5,6] 2024-09-17T07:20:00.796 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:00 smithi135 ceph-mon[33238]: pg 5.18 is stuck undersized for 3m, current state active+undersized, last acting [4,6] 2024-09-17T07:20:00.796 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:00 smithi135 ceph-mon[33238]: pg 5.1d is stuck undersized for 3m, current state active+undersized, last acting [4,0] 2024-09-17T07:20:00.796 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:00 smithi135 ceph-mon[33238]: pg 6.1 is stuck undersized for 3m, current state active+undersized, last acting [6,2] 2024-09-17T07:20:00.796 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:00 smithi135 ceph-mon[33238]: pg 6.4 is stuck undersized for 3m, current state active+undersized, last acting [5,6] 2024-09-17T07:20:00.796 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:00 smithi135 ceph-mon[33238]: pg 6.7 is stuck undersized for 3m, current state active+undersized, last acting [5,4] 2024-09-17T07:20:00.796 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:00 smithi135 ceph-mon[33238]: pg 6.d is stuck undersized for 3m, current state active+undersized, last acting [5,0] 2024-09-17T07:20:00.796 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:00 smithi135 ceph-mon[33238]: pg 6.e is stuck undersized for 3m, current state active+undersized, last acting [4,2] 2024-09-17T07:20:00.796 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:00 smithi135 ceph-mon[33238]: pg 6.10 is stuck undersized for 3m, current state active+undersized, last acting [0,5] 2024-09-17T07:20:00.796 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:00 smithi135 ceph-mon[33238]: pg 6.19 is stuck undersized for 3m, current state active+undersized, last acting [5,2] 2024-09-17T07:20:00.797 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:00 smithi135 ceph-mon[33238]: pg 6.1a is stuck undersized for 3m, current state active+undersized, last acting [4,5] 2024-09-17T07:20:00.797 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:00 smithi135 ceph-mon[33238]: pg 6.1d is stuck undersized for 3m, current state active+undersized, last acting [5,4] 2024-09-17T07:20:00.797 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:00 smithi135 ceph-mon[33238]: pg 6.1f is stuck undersized for 3m, current state active+undersized, last acting [6,5] 2024-09-17T07:20:00.797 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:00 smithi135 ceph-mon[33238]: [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-09-17T07:20:00.797 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:00 smithi135 ceph-mon[33238]: application not enabled on pool 'unique_pool_4' 2024-09-17T07:20:00.797 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:00 smithi135 ceph-mon[33238]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-09-17T07:20:00.797 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:00 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/3263949337' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:20:00.797 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:00 smithi135 ceph-mon[33238]: osdmap e394: 8 total, 7 up, 6 in 2024-09-17T07:20:00.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[29092]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set; 1 osds down; Degraded data redundancy: 73/588 objects degraded (12.415%), 17 pgs degraded, 41 pgs undersized; 1 pool(s) do not have an application enabled 2024-09-17T07:20:00.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[29092]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-09-17T07:20:00.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[29092]: [WRN] OSD_DOWN: 1 osds down 2024-09-17T07:20:00.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[29092]: osd.1 (root=default,host=smithi100) is down 2024-09-17T07:20:00.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[29092]: [WRN] PG_DEGRADED: Degraded data redundancy: 73/588 objects degraded (12.415%), 17 pgs degraded, 41 pgs undersized 2024-09-17T07:20:00.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[29092]: pg 2.1 is stuck undersized for 3m, current state active+undersized, last acting [0,2] 2024-09-17T07:20:00.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[29092]: pg 2.2 is stuck undersized for 3m, current state active+undersized, last acting [5,6] 2024-09-17T07:20:00.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[29092]: pg 2.6 is stuck undersized for 3m, current state active+undersized, last acting [6,4] 2024-09-17T07:20:00.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[29092]: pg 2.d is stuck undersized for 3m, current state active+undersized, last acting [4,2] 2024-09-17T07:20:00.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[29092]: pg 2.10 is stuck undersized for 3m, current state active+undersized, last acting [2,0] 2024-09-17T07:20:00.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[29092]: pg 2.11 is stuck undersized for 3m, current state active+undersized, last acting [6,4] 2024-09-17T07:20:00.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[29092]: pg 2.15 is stuck undersized for 3m, current state active+undersized+degraded, last acting [5,0] 2024-09-17T07:20:00.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[29092]: pg 3.0 is stuck undersized for 3m, current state active+undersized+degraded, last acting [6,2] 2024-09-17T07:20:00.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[29092]: pg 4.2 is stuck undersized for 3m, current state active+undersized+degraded, last acting [5,4] 2024-09-17T07:20:00.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[29092]: pg 4.4 is stuck undersized for 3m, current state active+undersized+degraded, last acting [6,2] 2024-09-17T07:20:00.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[29092]: pg 4.7 is stuck undersized for 3m, current state active+undersized+degraded, last acting [5,0] 2024-09-17T07:20:00.879 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[29092]: pg 4.b is stuck undersized for 3m, current state active+undersized+degraded, last acting [0,4] 2024-09-17T07:20:00.879 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[29092]: pg 4.c is stuck undersized for 3m, current state active+undersized+degraded, last acting [4,6] 2024-09-17T07:20:00.879 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[29092]: pg 4.d is stuck undersized for 3m, current state active+undersized+degraded, last acting [4,2] 2024-09-17T07:20:00.879 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[29092]: pg 4.e is stuck undersized for 3m, current state active+undersized+degraded, last acting [4,6] 2024-09-17T07:20:00.879 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[29092]: pg 4.12 is stuck undersized for 3m, current state active+undersized+degraded, last acting [6,2] 2024-09-17T07:20:00.879 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[29092]: pg 4.13 is stuck undersized for 3m, current state active+undersized+degraded, last acting [4,6] 2024-09-17T07:20:00.879 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[29092]: pg 4.18 is stuck undersized for 3m, current state active+undersized+degraded, last acting [4,6] 2024-09-17T07:20:00.879 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[29092]: pg 4.1b is stuck undersized for 3m, current state active+undersized+degraded, last acting [4,2] 2024-09-17T07:20:00.879 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[29092]: pg 4.1c is stuck undersized for 3m, current state active+undersized+degraded, last acting [0,2] 2024-09-17T07:20:00.879 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[29092]: pg 4.1d is stuck undersized for 3m, current state active+undersized+degraded, last acting [4,6] 2024-09-17T07:20:00.879 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[29092]: pg 5.0 is stuck undersized for 3m, current state active+undersized+degraded, last acting [4,0] 2024-09-17T07:20:00.879 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[29092]: pg 5.5 is stuck undersized for 3m, current state active+undersized, last acting [0,4] 2024-09-17T07:20:00.879 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[29092]: pg 5.7 is stuck undersized for 3m, current state active+undersized, last acting [5,0] 2024-09-17T07:20:00.880 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[39462]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set; 1 osds down; Degraded data redundancy: 73/588 objects degraded (12.415%), 17 pgs degraded, 41 pgs undersized; 1 pool(s) do not have an application enabled 2024-09-17T07:20:00.880 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[39462]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-09-17T07:20:00.880 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[39462]: [WRN] OSD_DOWN: 1 osds down 2024-09-17T07:20:00.880 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[39462]: osd.1 (root=default,host=smithi100) is down 2024-09-17T07:20:00.880 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[39462]: [WRN] PG_DEGRADED: Degraded data redundancy: 73/588 objects degraded (12.415%), 17 pgs degraded, 41 pgs undersized 2024-09-17T07:20:00.880 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[39462]: pg 2.1 is stuck undersized for 3m, current state active+undersized, last acting [0,2] 2024-09-17T07:20:00.880 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[39462]: pg 2.2 is stuck undersized for 3m, current state active+undersized, last acting [5,6] 2024-09-17T07:20:00.880 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[39462]: pg 2.6 is stuck undersized for 3m, current state active+undersized, last acting [6,4] 2024-09-17T07:20:00.880 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[39462]: pg 2.d is stuck undersized for 3m, current state active+undersized, last acting [4,2] 2024-09-17T07:20:00.881 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[39462]: pg 2.10 is stuck undersized for 3m, current state active+undersized, last acting [2,0] 2024-09-17T07:20:00.881 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[39462]: pg 2.11 is stuck undersized for 3m, current state active+undersized, last acting [6,4] 2024-09-17T07:20:00.881 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[39462]: pg 2.15 is stuck undersized for 3m, current state active+undersized+degraded, last acting [5,0] 2024-09-17T07:20:00.881 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[39462]: pg 3.0 is stuck undersized for 3m, current state active+undersized+degraded, last acting [6,2] 2024-09-17T07:20:00.881 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[39462]: pg 4.2 is stuck undersized for 3m, current state active+undersized+degraded, last acting [5,4] 2024-09-17T07:20:00.881 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[39462]: pg 4.4 is stuck undersized for 3m, current state active+undersized+degraded, last acting [6,2] 2024-09-17T07:20:00.881 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[39462]: pg 4.7 is stuck undersized for 3m, current state active+undersized+degraded, last acting [5,0] 2024-09-17T07:20:00.881 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[39462]: pg 4.b is stuck undersized for 3m, current state active+undersized+degraded, last acting [0,4] 2024-09-17T07:20:00.881 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[39462]: pg 4.c is stuck undersized for 3m, current state active+undersized+degraded, last acting [4,6] 2024-09-17T07:20:00.881 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[39462]: pg 4.d is stuck undersized for 3m, current state active+undersized+degraded, last acting [4,2] 2024-09-17T07:20:00.881 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[39462]: pg 4.e is stuck undersized for 3m, current state active+undersized+degraded, last acting [4,6] 2024-09-17T07:20:00.881 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[39462]: pg 4.12 is stuck undersized for 3m, current state active+undersized+degraded, last acting [6,2] 2024-09-17T07:20:00.881 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[39462]: pg 4.13 is stuck undersized for 3m, current state active+undersized+degraded, last acting [4,6] 2024-09-17T07:20:00.882 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[39462]: pg 4.18 is stuck undersized for 3m, current state active+undersized+degraded, last acting [4,6] 2024-09-17T07:20:00.882 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[39462]: pg 4.1b is stuck undersized for 3m, current state active+undersized+degraded, last acting [4,2] 2024-09-17T07:20:00.882 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[39462]: pg 4.1c is stuck undersized for 3m, current state active+undersized+degraded, last acting [0,2] 2024-09-17T07:20:00.882 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[39462]: pg 4.1d is stuck undersized for 3m, current state active+undersized+degraded, last acting [4,6] 2024-09-17T07:20:00.882 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[39462]: pg 5.0 is stuck undersized for 3m, current state active+undersized+degraded, last acting [4,0] 2024-09-17T07:20:00.882 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[39462]: pg 5.5 is stuck undersized for 3m, current state active+undersized, last acting [0,4] 2024-09-17T07:20:00.882 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[39462]: pg 5.7 is stuck undersized for 3m, current state active+undersized, last acting [5,0] 2024-09-17T07:20:00.882 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[29092]: pg 5.8 is stuck undersized for 3m, current state active+undersized, last acting [0,2] 2024-09-17T07:20:00.883 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[29092]: pg 5.c is stuck undersized for 3m, current state active+undersized, last acting [4,0] 2024-09-17T07:20:00.883 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[29092]: pg 5.13 is stuck undersized for 3m, current state active+undersized, last acting [0,5] 2024-09-17T07:20:00.883 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[29092]: pg 5.15 is stuck undersized for 3m, current state active+undersized+degraded, last acting [5,0] 2024-09-17T07:20:00.883 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[29092]: pg 5.16 is stuck undersized for 3m, current state active+undersized, last acting [5,6] 2024-09-17T07:20:00.883 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[29092]: pg 5.18 is stuck undersized for 3m, current state active+undersized, last acting [4,6] 2024-09-17T07:20:00.883 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[29092]: pg 5.1d is stuck undersized for 3m, current state active+undersized, last acting [4,0] 2024-09-17T07:20:00.883 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[29092]: pg 6.1 is stuck undersized for 3m, current state active+undersized, last acting [6,2] 2024-09-17T07:20:00.883 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[29092]: pg 6.4 is stuck undersized for 3m, current state active+undersized, last acting [5,6] 2024-09-17T07:20:00.883 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[29092]: pg 6.7 is stuck undersized for 3m, current state active+undersized, last acting [5,4] 2024-09-17T07:20:00.883 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[29092]: pg 6.d is stuck undersized for 3m, current state active+undersized, last acting [5,0] 2024-09-17T07:20:00.883 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[29092]: pg 6.e is stuck undersized for 3m, current state active+undersized, last acting [4,2] 2024-09-17T07:20:00.883 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[29092]: pg 6.10 is stuck undersized for 3m, current state active+undersized, last acting [0,5] 2024-09-17T07:20:00.883 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[29092]: pg 6.19 is stuck undersized for 3m, current state active+undersized, last acting [5,2] 2024-09-17T07:20:00.884 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[29092]: pg 6.1a is stuck undersized for 3m, current state active+undersized, last acting [4,5] 2024-09-17T07:20:00.884 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[29092]: pg 6.1d is stuck undersized for 3m, current state active+undersized, last acting [5,4] 2024-09-17T07:20:00.884 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[29092]: pg 6.1f is stuck undersized for 3m, current state active+undersized, last acting [6,5] 2024-09-17T07:20:00.884 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[29092]: [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-09-17T07:20:00.884 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[29092]: application not enabled on pool 'unique_pool_4' 2024-09-17T07:20:00.884 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[29092]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-09-17T07:20:00.884 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/3263949337' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:20:00.884 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[29092]: osdmap e394: 8 total, 7 up, 6 in 2024-09-17T07:20:00.884 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[39462]: pg 5.8 is stuck undersized for 3m, current state active+undersized, last acting [0,2] 2024-09-17T07:20:00.884 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[39462]: pg 5.c is stuck undersized for 3m, current state active+undersized, last acting [4,0] 2024-09-17T07:20:00.884 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[39462]: pg 5.13 is stuck undersized for 3m, current state active+undersized, last acting [0,5] 2024-09-17T07:20:00.885 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[39462]: pg 5.15 is stuck undersized for 3m, current state active+undersized+degraded, last acting [5,0] 2024-09-17T07:20:00.885 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[39462]: pg 5.16 is stuck undersized for 3m, current state active+undersized, last acting [5,6] 2024-09-17T07:20:00.885 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[39462]: pg 5.18 is stuck undersized for 3m, current state active+undersized, last acting [4,6] 2024-09-17T07:20:00.885 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[39462]: pg 5.1d is stuck undersized for 3m, current state active+undersized, last acting [4,0] 2024-09-17T07:20:00.885 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[39462]: pg 6.1 is stuck undersized for 3m, current state active+undersized, last acting [6,2] 2024-09-17T07:20:00.885 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[39462]: pg 6.4 is stuck undersized for 3m, current state active+undersized, last acting [5,6] 2024-09-17T07:20:00.885 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[39462]: pg 6.7 is stuck undersized for 3m, current state active+undersized, last acting [5,4] 2024-09-17T07:20:00.885 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[39462]: pg 6.d is stuck undersized for 3m, current state active+undersized, last acting [5,0] 2024-09-17T07:20:00.885 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[39462]: pg 6.e is stuck undersized for 3m, current state active+undersized, last acting [4,2] 2024-09-17T07:20:00.885 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[39462]: pg 6.10 is stuck undersized for 3m, current state active+undersized, last acting [0,5] 2024-09-17T07:20:00.885 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[39462]: pg 6.19 is stuck undersized for 3m, current state active+undersized, last acting [5,2] 2024-09-17T07:20:00.885 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[39462]: pg 6.1a is stuck undersized for 3m, current state active+undersized, last acting [4,5] 2024-09-17T07:20:00.886 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[39462]: pg 6.1d is stuck undersized for 3m, current state active+undersized, last acting [5,4] 2024-09-17T07:20:00.886 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[39462]: pg 6.1f is stuck undersized for 3m, current state active+undersized, last acting [6,5] 2024-09-17T07:20:00.886 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[39462]: [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-09-17T07:20:00.886 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[39462]: application not enabled on pool 'unique_pool_4' 2024-09-17T07:20:00.886 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[39462]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-09-17T07:20:00.886 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/3263949337' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:20:00.886 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:00 smithi100 ceph-mon[39462]: osdmap e394: 8 total, 7 up, 6 in 2024-09-17T07:20:00.892 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:20:00.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:00.916 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:00.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:20:01.016 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-17T07:20:01.016 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:20:01.017 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:20:01.039 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:01.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:01.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:20:01.140 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:20:01.162 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:01.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:01.163 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:20:01.264 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:20:01.286 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:01.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:01.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:20:01.388 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:20:01.409 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:01.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:01.410 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:20:01.511 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:20:01.532 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:01.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:01.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:20:01.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:01 smithi100 ceph-mon[39462]: pgmap v836: 158 pgs: 31 active+clean+remapped, 36 active+undersized, 17 active+undersized+degraded, 74 active+clean; 582 KiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 867 B/s rd, 0 op/s; 73/588 objects degraded (12.415%); 48/588 objects misplaced (8.163%) 2024-09-17T07:20:01.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:01 smithi100 ceph-mon[29092]: pgmap v836: 158 pgs: 31 active+clean+remapped, 36 active+undersized, 17 active+undersized+degraded, 74 active+clean; 582 KiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 867 B/s rd, 0 op/s; 73/588 objects degraded (12.415%); 48/588 objects misplaced (8.163%) 2024-09-17T07:20:01.634 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:20:01.656 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:01.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:01.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:20:01.752 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-17T07:20:01.757 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:20:01.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:01 smithi135 ceph-mon[33238]: pgmap v836: 158 pgs: 31 active+clean+remapped, 36 active+undersized, 17 active+undersized+degraded, 74 active+clean; 582 KiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 867 B/s rd, 0 op/s; 73/588 objects degraded (12.415%); 48/588 objects misplaced (8.163%) 2024-09-17T07:20:01.805 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:20:01.805 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:20:01.820 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:01.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:01.821 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:20:01.922 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:20:01.945 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:01.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:01.946 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:20:02.047 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:20:02.069 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:02.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:02.070 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:20:02.171 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:20:02.193 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:02.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:02.194 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:20:02.279 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:20:02.279 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:20:02.295 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:20:02.316 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:02.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:02.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:20:02.417 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:20:02.439 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:02.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:02.440 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:20:02.541 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:20:02.565 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:02.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:02.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:20:02.667 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:20:02.691 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:02.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:02.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:20:02.793 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:20:02.814 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:02.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:02.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:20:02.916 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:20:02.937 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:02.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:02.938 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:20:03.039 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:20:03.064 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:03.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:03.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:20:03.152 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:20:03.167 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:20:03.192 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:03.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:03.193 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:20:03.294 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:20:03.319 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:03.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:03.320 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:20:03.421 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:20:03.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:03.442 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:03.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:20:03.543 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:20:03.565 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:03.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:03.566 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:20:03.666 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:20:03.693 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:03.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:03.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:20:03.794 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:20:03.820 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:03.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:03.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:20:03.922 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:20:03.947 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:03.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:03.948 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:20:04.049 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:20:04.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:04.072 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:04.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:20:04.172 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:20:04.193 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:04.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:04.194 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:20:04.281 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:20:04.294 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:20:04.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:04.320 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:04.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:20:04.421 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:20:04.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:04.446 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:04.446 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:20:04.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:04 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:20:04.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:04 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:20:04.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:04 smithi135 ceph-mon[33238]: pgmap v837: 158 pgs: 31 active+clean+remapped, 36 active+undersized, 17 active+undersized+degraded, 74 active+clean; 582 KiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 741 B/s rd, 0 op/s; 73/588 objects degraded (12.415%); 48/588 objects misplaced (8.163%) 2024-09-17T07:20:04.547 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:20:04.571 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:04.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:04.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:20:04.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:04 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:20:04.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:04 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:20:04.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:04 smithi100 ceph-mon[29092]: pgmap v837: 158 pgs: 31 active+clean+remapped, 36 active+undersized, 17 active+undersized+degraded, 74 active+clean; 582 KiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 741 B/s rd, 0 op/s; 73/588 objects degraded (12.415%); 48/588 objects misplaced (8.163%) 2024-09-17T07:20:04.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:04 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:20:04.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:04 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:20:04.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:04 smithi100 ceph-mon[39462]: pgmap v837: 158 pgs: 31 active+clean+remapped, 36 active+undersized, 17 active+undersized+degraded, 74 active+clean; 582 KiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 741 B/s rd, 0 op/s; 73/588 objects degraded (12.415%); 48/588 objects misplaced (8.163%) 2024-09-17T07:20:04.673 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:20:04.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:04.698 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:04.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:20:04.800 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:20:04.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:04.823 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:04.823 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:20:04.924 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:20:04.945 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:04.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:04.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:20:05.047 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:20:05.069 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:05.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:05.071 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:20:05.171 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:20:05.194 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:05.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:05.195 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:20:05.283 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:20:05.296 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:20:05.318 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:05.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:05.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:20:05.420 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:20:05.441 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:05.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:05.442 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:20:05.543 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:20:05.566 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:05.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:05.567 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:20:05.668 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:20:05.689 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:05.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:05.691 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:20:05.791 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:20:05.812 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:05.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:05.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:20:05.914 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:20:05.936 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:05.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:05.937 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:20:06.025 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:20:05 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:20:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:20:06.038 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:20:06.060 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:06.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:06.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:20:06.162 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:20:06.187 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:06.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:06.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:20:06.290 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:20:06.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:06.312 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:06.312 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:20:06.413 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:20:06.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:06.440 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:06.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:20:06.541 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:20:06.565 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:06.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:06.566 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:20:06.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:06 smithi100 ceph-mon[29092]: pgmap v838: 158 pgs: 31 active+clean+remapped, 36 active+undersized, 17 active+undersized+degraded, 74 active+clean; 582 KiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 73/588 objects degraded (12.415%); 48/588 objects misplaced (8.163%) 2024-09-17T07:20:06.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:06 smithi100 ceph-mon[39462]: pgmap v838: 158 pgs: 31 active+clean+remapped, 36 active+undersized, 17 active+undersized+degraded, 74 active+clean; 582 KiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 73/588 objects degraded (12.415%); 48/588 objects misplaced (8.163%) 2024-09-17T07:20:06.667 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:20:06.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:06.692 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:06.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:20:06.793 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:20:06.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:06 smithi135 ceph-mon[33238]: pgmap v838: 158 pgs: 31 active+clean+remapped, 36 active+undersized, 17 active+undersized+degraded, 74 active+clean; 582 KiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 73/588 objects degraded (12.415%); 48/588 objects misplaced (8.163%) 2024-09-17T07:20:06.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:06.818 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:06.818 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:20:06.919 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:20:06.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:06.943 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:06.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:20:07.044 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:20:07.065 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:07.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:07.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:20:07.167 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:20:07.188 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:07.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:07.190 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:20:07.290 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:20:07.302 INFO:teuthology.orchestra.run.smithi100.stderr:enabled application 'rados' on pool 'unique_pool_4' 2024-09-17T07:20:07.316 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:07.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:07.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:20:07.418 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:20:07.439 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:07.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:07.440 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:20:07.541 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:20:07.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:07 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/3164306945' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-17T07:20:07.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:07 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/655620573' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:20:07.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:07 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:20:07.563 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:07.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:07.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:20:07.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:07 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/3164306945' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-17T07:20:07.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:07 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/655620573' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:20:07.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:07 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:20:07.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:07 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/3164306945' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-17T07:20:07.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:07 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/655620573' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:20:07.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:07 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:20:07.666 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:20:07.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:07.689 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:07.689 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:20:07.790 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:20:07.812 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:07.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:07.814 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:20:07.914 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:20:07.938 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:07.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:07.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:20:08.039 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:20:08.080 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:08.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:08.081 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:20:08.182 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:20:08.203 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:08.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:08.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:20:08.305 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:20:08.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:08.327 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:08.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:20:08.358 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:08 smithi100 ceph-mon[39462]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:20:08.358 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:08 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/3164306945' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-17T07:20:08.358 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:08 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T07:20:08.358 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:08 smithi100 ceph-mon[39462]: osdmap e395: 8 total, 7 up, 6 in 2024-09-17T07:20:08.358 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:08 smithi100 ceph-mon[39462]: pgmap v840: 158 pgs: 31 active+clean+remapped, 36 active+undersized, 17 active+undersized+degraded, 74 active+clean; 582 KiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 639 B/s rd, 0 op/s; 73/588 objects degraded (12.415%); 48/588 objects misplaced (8.163%) 2024-09-17T07:20:08.358 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:08 smithi100 ceph-mon[39462]: 6.b scrub starts 2024-09-17T07:20:08.358 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:08 smithi100 ceph-mon[39462]: 6.b scrub ok 2024-09-17T07:20:08.359 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:08 smithi100 ceph-mon[29092]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:20:08.359 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:08 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/3164306945' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-17T07:20:08.359 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:08 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T07:20:08.359 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:08 smithi100 ceph-mon[29092]: osdmap e395: 8 total, 7 up, 6 in 2024-09-17T07:20:08.359 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:08 smithi100 ceph-mon[29092]: pgmap v840: 158 pgs: 31 active+clean+remapped, 36 active+undersized, 17 active+undersized+degraded, 74 active+clean; 582 KiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 639 B/s rd, 0 op/s; 73/588 objects degraded (12.415%); 48/588 objects misplaced (8.163%) 2024-09-17T07:20:08.359 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:08 smithi100 ceph-mon[29092]: 6.b scrub starts 2024-09-17T07:20:08.359 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:08 smithi100 ceph-mon[29092]: 6.b scrub ok 2024-09-17T07:20:08.429 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:20:08.453 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:08.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:08.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:20:08.555 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:20:08.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:08.578 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:08.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:20:08.680 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:20:08.701 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:08.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:08.702 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:20:08.713 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:08 smithi135 ceph-mon[33238]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:20:08.713 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:08 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/3164306945' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-17T07:20:08.713 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:08 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T07:20:08.713 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:08 smithi135 ceph-mon[33238]: osdmap e395: 8 total, 7 up, 6 in 2024-09-17T07:20:08.713 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:08 smithi135 ceph-mon[33238]: pgmap v840: 158 pgs: 31 active+clean+remapped, 36 active+undersized, 17 active+undersized+degraded, 74 active+clean; 582 KiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 639 B/s rd, 0 op/s; 73/588 objects degraded (12.415%); 48/588 objects misplaced (8.163%) 2024-09-17T07:20:08.713 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:08 smithi135 ceph-mon[33238]: 6.b scrub starts 2024-09-17T07:20:08.713 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:08 smithi135 ceph-mon[33238]: 6.b scrub ok 2024-09-17T07:20:08.803 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:20:08.824 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:08.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:08.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:20:08.926 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:20:08.948 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:08.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:08.949 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:20:09.049 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:20:09.063 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:20:09.070 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:09.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:09.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:20:09.172 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:20:09.189 INFO:teuthology.orchestra.run.smithi100.stderr:noscrub is unset 2024-09-17T07:20:09.194 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:09.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:09.195 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:20:09.296 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:20:09.318 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:09.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:09.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:20:09.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:09 smithi100 ceph-mon[29092]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-17T07:20:09.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:09 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:20:09.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:09 smithi100 ceph-mon[39462]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-17T07:20:09.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:09 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:20:09.420 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:20:09.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:09.443 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:09.443 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:20:09.543 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:20:09 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:20:09.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:09 smithi135 ceph-mon[33238]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-17T07:20:09.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:09 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:20:09.544 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:20:09.565 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:09.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:09.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:20:09.666 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:20:09.687 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:09.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:09.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:20:09.789 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:20:09.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:09.812 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:09.812 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:20:09.913 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:20:09.934 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:09.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:09.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:20:10.036 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:20:10.056 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:10.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:10.058 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:20:10.086 DEBUG:teuthology.orchestra.run.smithi100:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_4 bench 90 write' 2024-09-17T07:20:10.100 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-17T07:20:10.100 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-17T07:20:10.110 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-17T07:20:10.110 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:20:10.150 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:hints = 1 2024-09-17T07:20:10.150 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-17T07:20:10.150 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Object prefix: benchmark_data_smithi100_180298 2024-09-17T07:20:10.151 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-17T07:20:10.151 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 0 0 0 0 0 0 - 0 2024-09-17T07:20:10.158 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:20:10.190 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:10.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:10.192 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:20:10.292 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:20:10.326 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:10.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:10.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:20:10.428 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:20:10.460 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:10.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:10.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:20:10.562 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:20:10.594 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:10.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:10.596 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:20:10.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:10 smithi100 ceph-mon[29092]: 5.14 scrub starts 2024-09-17T07:20:10.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:10 smithi100 ceph-mon[29092]: 5.14 scrub ok 2024-09-17T07:20:10.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:10 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:20:10.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:10 smithi100 ceph-mon[29092]: pgmap v841: 158 pgs: 31 active+clean+remapped, 36 active+undersized, 17 active+undersized+degraded, 74 active+clean; 582 KiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s; 73/588 objects degraded (12.415%); 48/588 objects misplaced (8.163%) 2024-09-17T07:20:10.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:10 smithi100 ceph-mon[29092]: osdmap e396: 8 total, 7 up, 6 in 2024-09-17T07:20:10.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:10 smithi100 ceph-mon[39462]: 5.14 scrub starts 2024-09-17T07:20:10.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:10 smithi100 ceph-mon[39462]: 5.14 scrub ok 2024-09-17T07:20:10.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:10 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:20:10.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:10 smithi100 ceph-mon[39462]: pgmap v841: 158 pgs: 31 active+clean+remapped, 36 active+undersized, 17 active+undersized+degraded, 74 active+clean; 582 KiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s; 73/588 objects degraded (12.415%); 48/588 objects misplaced (8.163%) 2024-09-17T07:20:10.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:10 smithi100 ceph-mon[39462]: osdmap e396: 8 total, 7 up, 6 in 2024-09-17T07:20:10.696 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:20:10.729 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:10.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:10.730 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:20:10.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:10 smithi135 ceph-mon[33238]: 5.14 scrub starts 2024-09-17T07:20:10.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:10 smithi135 ceph-mon[33238]: 5.14 scrub ok 2024-09-17T07:20:10.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:10 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:20:10.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:10 smithi135 ceph-mon[33238]: pgmap v841: 158 pgs: 31 active+clean+remapped, 36 active+undersized, 17 active+undersized+degraded, 74 active+clean; 582 KiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s; 73/588 objects degraded (12.415%); 48/588 objects misplaced (8.163%) 2024-09-17T07:20:10.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:10 smithi135 ceph-mon[33238]: osdmap e396: 8 total, 7 up, 6 in 2024-09-17T07:20:10.831 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:20:10.868 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:10.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:10.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:20:10.916 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-17T07:20:10.969 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:20:11.004 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:11.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:11.005 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:20:11.106 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:20:11.138 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:11.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:11.139 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:20:11.151 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 1 16 4675 4659 291.141 291.188 0.00453596 0.0034236 2024-09-17T07:20:11.240 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:20:11.275 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:11.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:11.276 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:20:11.377 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:20:11.412 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:11 smithi100 ceph-mon[29092]: 5.1a scrub starts 2024-09-17T07:20:11.412 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:11 smithi100 ceph-mon[29092]: 5.1a scrub ok 2024-09-17T07:20:11.412 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:11 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:20:11.412 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:11 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:20:11.412 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:11 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:20:11.412 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:11 smithi100 ceph-mon[29092]: 6.1b scrub starts 2024-09-17T07:20:11.412 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:11 smithi100 ceph-mon[29092]: 6.1b scrub ok 2024-09-17T07:20:11.413 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:11 smithi100 ceph-mon[39462]: 5.1a scrub starts 2024-09-17T07:20:11.413 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:11 smithi100 ceph-mon[39462]: 5.1a scrub ok 2024-09-17T07:20:11.413 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:11 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:20:11.413 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:11 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:20:11.413 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:11 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:20:11.413 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:11 smithi100 ceph-mon[39462]: 6.1b scrub starts 2024-09-17T07:20:11.413 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:11 smithi100 ceph-mon[39462]: 6.1b scrub ok 2024-09-17T07:20:11.418 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:11.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:11.419 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:20:11.514 INFO:teuthology.orchestra.run.smithi100.stdout:moved 14 / 474 (2.95359%) 2024-09-17T07:20:11.514 INFO:teuthology.orchestra.run.smithi100.stdout:avg 94.8 2024-09-17T07:20:11.514 INFO:teuthology.orchestra.run.smithi100.stdout:stddev 17.1184 -> 18.6355 (expected baseline 8.70862) 2024-09-17T07:20:11.514 INFO:teuthology.orchestra.run.smithi100.stdout:min osd.4 with 74 -> 78 pgs (0.780591 -> 0.822785 * mean) 2024-09-17T07:20:11.514 INFO:teuthology.orchestra.run.smithi100.stdout:max osd.6 with 83 -> 85 pgs (0.875527 -> 0.896624 * mean) 2024-09-17T07:20:11.514 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T07:20:11.514 INFO:teuthology.orchestra.run.smithi100.stdout:oload 110 2024-09-17T07:20:11.514 INFO:teuthology.orchestra.run.smithi100.stdout:max_change 3 2024-09-17T07:20:11.514 INFO:teuthology.orchestra.run.smithi100.stdout:max_change_osds 4 2024-09-17T07:20:11.515 INFO:teuthology.orchestra.run.smithi100.stdout:average_utilization 0.0082 2024-09-17T07:20:11.515 INFO:teuthology.orchestra.run.smithi100.stdout:overload_utilization 0.0091 2024-09-17T07:20:11.515 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0 weight 1.0000 -> 0.8298 2024-09-17T07:20:11.515 INFO:teuthology.orchestra.run.smithi100.stderr:no change 2024-09-17T07:20:11.520 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:20:11.554 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:11.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:11.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:20:11.656 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:20:11.691 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:11.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:11.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:20:11.793 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:20:11.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:11 smithi135 ceph-mon[33238]: 5.1a scrub starts 2024-09-17T07:20:11.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:11 smithi135 ceph-mon[33238]: 5.1a scrub ok 2024-09-17T07:20:11.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:11 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:20:11.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:11 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:20:11.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:11 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:20:11.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:11 smithi135 ceph-mon[33238]: 6.1b scrub starts 2024-09-17T07:20:11.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:11 smithi135 ceph-mon[33238]: 6.1b scrub ok 2024-09-17T07:20:11.827 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:11.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:11.828 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:20:11.877 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:20:11 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T07:20:11.512+0000 7f8b9368d640 -1 mgr.server reply reply (1) Operation not permitted no change 2024-09-17T07:20:11.907 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-17T07:20:11.929 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:20:11.964 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:11.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:11.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:20:12.066 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:20:12.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:12.105 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:12.106 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:20:12.151 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 2 15 8932 8917 278.604 266.125 0.00484828 0.00358321 2024-09-17T07:20:12.207 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:20:12.241 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:12.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:12.242 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:20:12.343 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:20:12.376 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:12.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:12.377 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:20:12.478 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:20:12.514 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:12.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:12.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:20:12.616 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:20:12.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:12 smithi100 ceph-mon[39462]: 4.14 scrub starts 2024-09-17T07:20:12.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:12 smithi100 ceph-mon[39462]: 4.14 scrub ok 2024-09-17T07:20:12.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:12 smithi100 ceph-mon[39462]: 2.f scrub starts 2024-09-17T07:20:12.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:12 smithi100 ceph-mon[39462]: 2.f scrub ok 2024-09-17T07:20:12.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:12 smithi100 ceph-mon[39462]: pgmap v843: 158 pgs: 31 active+clean+remapped, 36 active+undersized, 17 active+undersized+degraded, 74 active+clean; 582 KiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 73/588 objects degraded (12.415%); 48/588 objects misplaced (8.163%) 2024-09-17T07:20:12.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:12 smithi100 ceph-mon[29092]: 4.14 scrub starts 2024-09-17T07:20:12.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:12 smithi100 ceph-mon[29092]: 4.14 scrub ok 2024-09-17T07:20:12.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:12 smithi100 ceph-mon[29092]: 2.f scrub starts 2024-09-17T07:20:12.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:12 smithi100 ceph-mon[29092]: 2.f scrub ok 2024-09-17T07:20:12.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:12 smithi100 ceph-mon[29092]: pgmap v843: 158 pgs: 31 active+clean+remapped, 36 active+undersized, 17 active+undersized+degraded, 74 active+clean; 582 KiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 73/588 objects degraded (12.415%); 48/588 objects misplaced (8.163%) 2024-09-17T07:20:12.649 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:12.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:12.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:20:12.751 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:20:12.787 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:12.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:12.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:20:12.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:12 smithi135 ceph-mon[33238]: 4.14 scrub starts 2024-09-17T07:20:12.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:12 smithi135 ceph-mon[33238]: 4.14 scrub ok 2024-09-17T07:20:12.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:12 smithi135 ceph-mon[33238]: 2.f scrub starts 2024-09-17T07:20:12.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:12 smithi135 ceph-mon[33238]: 2.f scrub ok 2024-09-17T07:20:12.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:12 smithi135 ceph-mon[33238]: pgmap v843: 158 pgs: 31 active+clean+remapped, 36 active+undersized, 17 active+undersized+degraded, 74 active+clean; 582 KiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 73/588 objects degraded (12.415%); 48/588 objects misplaced (8.163%) 2024-09-17T07:20:12.891 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:20:12.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:12.926 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:12.926 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:20:13.027 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:20:13.062 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:13.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:13.063 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:20:13.151 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 3 16 13088 13072 272.292 259.688 0.00257555 0.0036669 2024-09-17T07:20:13.164 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:20:13.189 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:20:13.194 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:13.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:13.195 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:20:13.221 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-09-17T07:20:13.221 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd reweight-by-utilization 1000 1.0 2024-09-17T07:20:13.296 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:20:13.331 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:13.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:13.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:20:13.432 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:20:13.469 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:13 smithi100 ceph-mon[39462]: from='client.15978 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:20:13.469 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:13 smithi100 ceph-mon[39462]: 2.a scrub starts 2024-09-17T07:20:13.469 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:13 smithi100 ceph-mon[39462]: 2.a scrub ok 2024-09-17T07:20:13.469 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:13 smithi100 ceph-mon[39462]: 2.8 scrub starts 2024-09-17T07:20:13.469 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:13 smithi100 ceph-mon[39462]: 2.8 scrub ok 2024-09-17T07:20:13.470 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:13 smithi100 ceph-mon[29092]: from='client.15978 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:20:13.470 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:13 smithi100 ceph-mon[29092]: 2.a scrub starts 2024-09-17T07:20:13.470 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:13 smithi100 ceph-mon[29092]: 2.a scrub ok 2024-09-17T07:20:13.470 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:13 smithi100 ceph-mon[29092]: 2.8 scrub starts 2024-09-17T07:20:13.470 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:13 smithi100 ceph-mon[29092]: 2.8 scrub ok 2024-09-17T07:20:13.481 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:13.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:13.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:20:13.583 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:20:13.618 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:13.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:13.619 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:20:13.719 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:20:13.752 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:13.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:13.753 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:20:13.761 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:20:13.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:13 smithi135 ceph-mon[33238]: from='client.15978 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:20:13.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:13 smithi135 ceph-mon[33238]: 2.a scrub starts 2024-09-17T07:20:13.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:13 smithi135 ceph-mon[33238]: 2.a scrub ok 2024-09-17T07:20:13.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:13 smithi135 ceph-mon[33238]: 2.8 scrub starts 2024-09-17T07:20:13.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:13 smithi135 ceph-mon[33238]: 2.8 scrub ok 2024-09-17T07:20:13.820 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:20:13.820 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:20:13.821 INFO:teuthology.orchestra.run.smithi100.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-17T07:20:13.849 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:20:13.849 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:20:13.854 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:20:13.876 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:20:13.876 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:20:13.886 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:13.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:13.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:20:13.905 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:20:13.905 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:20:13.938 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:20:13.938 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:20:13.971 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:20:13.971 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:20:13.989 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:20:14.003 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:20:14.003 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:20:14.024 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:14.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:14.025 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:20:14.126 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:20:14.151 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 4 16 16622 16606 259.434 220.875 0.00233761 0.0038488 2024-09-17T07:20:14.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:14.160 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:14.160 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:20:14.261 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:20:14.290 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:14.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:14.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:20:14.392 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:20:14.426 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:14.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:14.427 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:20:14.528 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:20:14.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:14.563 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:14.563 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:20:14.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:14 smithi100 ceph-mon[29092]: 5.b scrub starts 2024-09-17T07:20:14.629 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:14 smithi100 ceph-mon[29092]: 5.b scrub ok 2024-09-17T07:20:14.629 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:14 smithi100 ceph-mon[29092]: 4.f scrub starts 2024-09-17T07:20:14.629 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:14 smithi100 ceph-mon[29092]: 4.f scrub ok 2024-09-17T07:20:14.629 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:14 smithi100 ceph-mon[29092]: pgmap v844: 158 pgs: 31 active+clean+remapped, 34 active+undersized, 19 active+undersized+degraded, 74 active+clean; 155 MiB data, 5.0 GiB used, 531 GiB / 536 GiB avail; 639 B/s rd, 19 MiB/s wr, 309 op/s; 1693/8001 objects degraded (21.160%); 48/8001 objects misplaced (0.600%) 2024-09-17T07:20:14.629 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:14 smithi100 ceph-mon[29092]: 4.0 scrub starts 2024-09-17T07:20:14.629 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:14 smithi100 ceph-mon[29092]: 4.0 scrub ok 2024-09-17T07:20:14.629 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:14 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/183870249' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:20:14.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:14 smithi100 ceph-mon[39462]: 5.b scrub starts 2024-09-17T07:20:14.630 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:14 smithi100 ceph-mon[39462]: 5.b scrub ok 2024-09-17T07:20:14.630 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:14 smithi100 ceph-mon[39462]: 4.f scrub starts 2024-09-17T07:20:14.630 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:14 smithi100 ceph-mon[39462]: 4.f scrub ok 2024-09-17T07:20:14.630 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:14 smithi100 ceph-mon[39462]: pgmap v844: 158 pgs: 31 active+clean+remapped, 34 active+undersized, 19 active+undersized+degraded, 74 active+clean; 155 MiB data, 5.0 GiB used, 531 GiB / 536 GiB avail; 639 B/s rd, 19 MiB/s wr, 309 op/s; 1693/8001 objects degraded (21.160%); 48/8001 objects misplaced (0.600%) 2024-09-17T07:20:14.630 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:14 smithi100 ceph-mon[39462]: 4.0 scrub starts 2024-09-17T07:20:14.630 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:14 smithi100 ceph-mon[39462]: 4.0 scrub ok 2024-09-17T07:20:14.630 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:14 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/183870249' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:20:14.664 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:20:14.696 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:14.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:14.697 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:20:14.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:14 smithi135 ceph-mon[33238]: 5.b scrub starts 2024-09-17T07:20:14.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:14 smithi135 ceph-mon[33238]: 5.b scrub ok 2024-09-17T07:20:14.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:14 smithi135 ceph-mon[33238]: 4.f scrub starts 2024-09-17T07:20:14.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:14 smithi135 ceph-mon[33238]: 4.f scrub ok 2024-09-17T07:20:14.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:14 smithi135 ceph-mon[33238]: pgmap v844: 158 pgs: 31 active+clean+remapped, 34 active+undersized, 19 active+undersized+degraded, 74 active+clean; 155 MiB data, 5.0 GiB used, 531 GiB / 536 GiB avail; 639 B/s rd, 19 MiB/s wr, 309 op/s; 1693/8001 objects degraded (21.160%); 48/8001 objects misplaced (0.600%) 2024-09-17T07:20:14.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:14 smithi135 ceph-mon[33238]: 4.0 scrub starts 2024-09-17T07:20:14.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:14 smithi135 ceph-mon[33238]: 4.0 scrub ok 2024-09-17T07:20:14.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:14 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/183870249' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:20:14.798 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:20:14.829 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:14.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:14.830 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:20:14.847 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:20:14.847 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:20:14.931 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:20:14.967 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:14.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:14.969 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:20:15.069 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:20:15.103 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:15.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:15.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:20:15.151 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 5 16 20272 20256 253.169 228.125 0.00442896 0.00394392 2024-09-17T07:20:15.205 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:20:15.240 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:15.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:15.245 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:20:15.346 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:20:15.379 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:15.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:15.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:20:15.481 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:20:15.518 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:15.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:15.519 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:20:15.620 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:20:15.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:15 smithi100 ceph-mon[29092]: 5.19 scrub starts 2024-09-17T07:20:15.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:15 smithi100 ceph-mon[29092]: 5.19 scrub ok 2024-09-17T07:20:15.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:15 smithi100 ceph-mon[29092]: 4.10 scrub starts 2024-09-17T07:20:15.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:15 smithi100 ceph-mon[29092]: 4.10 scrub ok 2024-09-17T07:20:15.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:15 smithi100 ceph-mon[29092]: 2.9 scrub starts 2024-09-17T07:20:15.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:15 smithi100 ceph-mon[29092]: 2.9 scrub ok 2024-09-17T07:20:15.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:15 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 1693/8001 objects degraded (21.160%), 19 pgs degraded, 41 pgs undersized (PG_DEGRADED) 2024-09-17T07:20:15.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:15 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:20:15.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:15 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:20:15.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:15 smithi100 ceph-mon[39462]: 5.19 scrub starts 2024-09-17T07:20:15.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:15 smithi100 ceph-mon[39462]: 5.19 scrub ok 2024-09-17T07:20:15.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:15 smithi100 ceph-mon[39462]: 4.10 scrub starts 2024-09-17T07:20:15.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:15 smithi100 ceph-mon[39462]: 4.10 scrub ok 2024-09-17T07:20:15.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:15 smithi100 ceph-mon[39462]: 2.9 scrub starts 2024-09-17T07:20:15.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:15 smithi100 ceph-mon[39462]: 2.9 scrub ok 2024-09-17T07:20:15.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:15 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 1693/8001 objects degraded (21.160%), 19 pgs degraded, 41 pgs undersized (PG_DEGRADED) 2024-09-17T07:20:15.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:15 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:20:15.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:15 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:20:15.653 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:15.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:15.654 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:20:15.755 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:20:15.789 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:15.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:15.790 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:20:15.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:15 smithi135 ceph-mon[33238]: 5.19 scrub starts 2024-09-17T07:20:15.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:15 smithi135 ceph-mon[33238]: 5.19 scrub ok 2024-09-17T07:20:15.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:15 smithi135 ceph-mon[33238]: 4.10 scrub starts 2024-09-17T07:20:15.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:15 smithi135 ceph-mon[33238]: 4.10 scrub ok 2024-09-17T07:20:15.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:15 smithi135 ceph-mon[33238]: 2.9 scrub starts 2024-09-17T07:20:15.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:15 smithi135 ceph-mon[33238]: 2.9 scrub ok 2024-09-17T07:20:15.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:15 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 1693/8001 objects degraded (21.160%), 19 pgs degraded, 41 pgs undersized (PG_DEGRADED) 2024-09-17T07:20:15.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:15 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:20:15.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:15 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:20:15.890 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:20:15.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:15.927 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:15.928 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:20:16.022 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:20:15 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:20:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:20:16.028 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:20:16.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:16.065 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:16.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:20:16.151 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 6 16 23592 23576 245.555 207.5 0.00380276 0.00406727 2024-09-17T07:20:16.166 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:20:16.201 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:16.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:16.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:20:16.303 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:20:16.338 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:16.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:16.340 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:20:16.346 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:20:16.441 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:20:16.475 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:16.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:16.477 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:20:16.577 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:20:16.614 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:16.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:16.615 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:20:16.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:16 smithi100 ceph-mon[29092]: 4.a scrub starts 2024-09-17T07:20:16.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:16 smithi100 ceph-mon[29092]: 4.a scrub ok 2024-09-17T07:20:16.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:16 smithi100 ceph-mon[29092]: osdmap e397: 8 total, 7 up, 6 in 2024-09-17T07:20:16.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:16 smithi100 ceph-mon[29092]: pgmap v846: 158 pgs: 31 active+clean+remapped, 24 active+undersized, 29 active+undersized+degraded, 74 active+clean; 991 MiB data, 6.4 GiB used, 530 GiB / 536 GiB avail; 1.2 KiB/s rd, 124 MiB/s wr, 1.98k op/s; 12205/48135 objects degraded (25.356%); 48/48135 objects misplaced (0.100%) 2024-09-17T07:20:16.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:16 smithi100 ceph-mon[39462]: 4.a scrub starts 2024-09-17T07:20:16.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:16 smithi100 ceph-mon[39462]: 4.a scrub ok 2024-09-17T07:20:16.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:16 smithi100 ceph-mon[39462]: osdmap e397: 8 total, 7 up, 6 in 2024-09-17T07:20:16.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:16 smithi100 ceph-mon[39462]: pgmap v846: 158 pgs: 31 active+clean+remapped, 24 active+undersized, 29 active+undersized+degraded, 74 active+clean; 991 MiB data, 6.4 GiB used, 530 GiB / 536 GiB avail; 1.2 KiB/s rd, 124 MiB/s wr, 1.98k op/s; 12205/48135 objects degraded (25.356%); 48/48135 objects misplaced (0.100%) 2024-09-17T07:20:16.717 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:20:16.755 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:16.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:16.756 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:20:16.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:16 smithi135 ceph-mon[33238]: 4.a scrub starts 2024-09-17T07:20:16.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:16 smithi135 ceph-mon[33238]: 4.a scrub ok 2024-09-17T07:20:16.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:16 smithi135 ceph-mon[33238]: osdmap e397: 8 total, 7 up, 6 in 2024-09-17T07:20:16.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:16 smithi135 ceph-mon[33238]: pgmap v846: 158 pgs: 31 active+clean+remapped, 24 active+undersized, 29 active+undersized+degraded, 74 active+clean; 991 MiB data, 6.4 GiB used, 530 GiB / 536 GiB avail; 1.2 KiB/s rd, 124 MiB/s wr, 1.98k op/s; 12205/48135 objects degraded (25.356%); 48/48135 objects misplaced (0.100%) 2024-09-17T07:20:16.848 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:20:16.856 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:20:16.891 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:16.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:16.892 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:20:16.993 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:20:17.025 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:17.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:17.027 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:20:17.127 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:20:17.151 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 7 16 26756 26740 238.72 197.75 0.00240956 0.00418392 2024-09-17T07:20:17.163 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:17.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:17.164 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:20:17.264 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:20:17.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:17.297 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:17.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:20:17.398 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:20:17.434 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:17.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:17.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:20:17.535 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:20:17.571 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:17.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:17.572 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:20:17.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:17 smithi100 ceph-mon[29092]: 5.12 scrub starts 2024-09-17T07:20:17.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:17 smithi100 ceph-mon[29092]: 5.12 scrub ok 2024-09-17T07:20:17.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:17 smithi100 ceph-mon[29092]: pgmap v847: 158 pgs: 31 active+clean+remapped, 24 active+undersized, 29 active+undersized+degraded, 74 active+clean; 991 MiB data, 6.4 GiB used, 530 GiB / 536 GiB avail; 639 B/s rd, 124 MiB/s wr, 1.98k op/s; 12205/48135 objects degraded (25.356%); 48/48135 objects misplaced (0.100%) 2024-09-17T07:20:17.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:17 smithi100 ceph-mon[39462]: 5.12 scrub starts 2024-09-17T07:20:17.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:17 smithi100 ceph-mon[39462]: 5.12 scrub ok 2024-09-17T07:20:17.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:17 smithi100 ceph-mon[39462]: pgmap v847: 158 pgs: 31 active+clean+remapped, 24 active+undersized, 29 active+undersized+degraded, 74 active+clean; 991 MiB data, 6.4 GiB used, 530 GiB / 536 GiB avail; 639 B/s rd, 124 MiB/s wr, 1.98k op/s; 12205/48135 objects degraded (25.356%); 48/48135 objects misplaced (0.100%) 2024-09-17T07:20:17.673 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:20:17.704 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:17.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:17.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:20:17.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:17 smithi135 ceph-mon[33238]: 5.12 scrub starts 2024-09-17T07:20:17.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:17 smithi135 ceph-mon[33238]: 5.12 scrub ok 2024-09-17T07:20:17.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:17 smithi135 ceph-mon[33238]: pgmap v847: 158 pgs: 31 active+clean+remapped, 24 active+undersized, 29 active+undersized+degraded, 74 active+clean; 991 MiB data, 6.4 GiB used, 530 GiB / 536 GiB avail; 639 B/s rd, 124 MiB/s wr, 1.98k op/s; 12205/48135 objects degraded (25.356%); 48/48135 objects misplaced (0.100%) 2024-09-17T07:20:17.806 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:20:17.839 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:17.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:17.841 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:20:17.942 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:20:17.974 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:17.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:17.975 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:20:18.076 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:20:18.112 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:18.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:18.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:20:18.152 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 8 16 29807 29791 232.714 190.688 0.00364521 0.00429314 2024-09-17T07:20:18.218 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:20:18.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:18.239 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:18.239 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:20:18.340 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:20:18.376 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:18.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:18.378 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:20:18.478 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:20:18.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:18.503 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:18.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:20:18.604 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:20:18.639 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:18.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:18.640 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:20:18.741 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:20:18.770 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:18.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:18.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:20:18.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:18 smithi135 ceph-mon[33238]: 5.6 scrub starts 2024-09-17T07:20:18.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:18 smithi135 ceph-mon[33238]: 5.6 scrub ok 2024-09-17T07:20:18.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:18 smithi135 ceph-mon[33238]: 4.17 scrub starts 2024-09-17T07:20:18.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:18 smithi135 ceph-mon[33238]: 4.17 scrub ok 2024-09-17T07:20:18.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:18 smithi135 ceph-mon[33238]: 2.3 scrub starts 2024-09-17T07:20:18.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:18 smithi135 ceph-mon[33238]: 2.3 scrub ok 2024-09-17T07:20:18.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:18 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:20:18.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:18 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:20:18.809 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:18 smithi100 ceph-mon[39462]: 5.6 scrub starts 2024-09-17T07:20:18.809 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:18 smithi100 ceph-mon[39462]: 5.6 scrub ok 2024-09-17T07:20:18.809 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:18 smithi100 ceph-mon[39462]: 4.17 scrub starts 2024-09-17T07:20:18.810 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:18 smithi100 ceph-mon[39462]: 4.17 scrub ok 2024-09-17T07:20:18.810 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:18 smithi100 ceph-mon[39462]: 2.3 scrub starts 2024-09-17T07:20:18.810 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:18 smithi100 ceph-mon[39462]: 2.3 scrub ok 2024-09-17T07:20:18.810 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:18 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:20:18.810 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:18 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:20:18.810 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:18 smithi100 ceph-mon[29092]: 5.6 scrub starts 2024-09-17T07:20:18.810 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:18 smithi100 ceph-mon[29092]: 5.6 scrub ok 2024-09-17T07:20:18.810 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:18 smithi100 ceph-mon[29092]: 4.17 scrub starts 2024-09-17T07:20:18.810 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:18 smithi100 ceph-mon[29092]: 4.17 scrub ok 2024-09-17T07:20:18.810 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:18 smithi100 ceph-mon[29092]: 2.3 scrub starts 2024-09-17T07:20:18.810 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:18 smithi100 ceph-mon[29092]: 2.3 scrub ok 2024-09-17T07:20:18.811 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:18 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:20:18.811 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:18 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:20:18.872 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:20:18.904 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:18.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:18.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:20:19.006 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:20:19.040 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:19.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:19.041 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:20:19.142 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:20:19.152 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 9 16 32243 32227 223.772 152.25 0.00536172 0.00446374 2024-09-17T07:20:19.178 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:19.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:19.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:20:19.280 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:20:19.314 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:19.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:19.315 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:20:19.416 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:20:19.445 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:20:19 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:20:19.449 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:19.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:19.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:20:19.551 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:20:19.586 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:19.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:19.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:20:19.688 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:20:19.721 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:19.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:19.722 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:20:19.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:19 smithi135 ceph-mon[33238]: 2.c scrub starts 2024-09-17T07:20:19.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:19 smithi135 ceph-mon[33238]: 2.c scrub ok 2024-09-17T07:20:19.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:19 smithi135 ceph-mon[33238]: 2.1a scrub starts 2024-09-17T07:20:19.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:19 smithi135 ceph-mon[33238]: 2.1a scrub ok 2024-09-17T07:20:19.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:19 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:20:19.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:19 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:20:19.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:19 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:20:19.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:19 smithi135 ceph-mon[33238]: pgmap v848: 158 pgs: 31 active+clean+remapped, 24 active+undersized, 29 active+undersized+degraded, 74 active+clean; 1.7 GiB data, 8.1 GiB used, 528 GiB / 536 GiB avail; 1.1 KiB/s rd, 191 MiB/s wr, 3.06k op/s; 20290/84120 objects degraded (24.120%); 48/84120 objects misplaced (0.057%) 2024-09-17T07:20:19.813 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:19 smithi100 ceph-mon[29092]: 2.c scrub starts 2024-09-17T07:20:19.813 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:19 smithi100 ceph-mon[29092]: 2.c scrub ok 2024-09-17T07:20:19.813 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:19 smithi100 ceph-mon[29092]: 2.1a scrub starts 2024-09-17T07:20:19.813 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:19 smithi100 ceph-mon[29092]: 2.1a scrub ok 2024-09-17T07:20:19.813 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:19 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:20:19.813 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:19 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:20:19.813 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:19 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:20:19.813 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:19 smithi100 ceph-mon[29092]: pgmap v848: 158 pgs: 31 active+clean+remapped, 24 active+undersized, 29 active+undersized+degraded, 74 active+clean; 1.7 GiB data, 8.1 GiB used, 528 GiB / 536 GiB avail; 1.1 KiB/s rd, 191 MiB/s wr, 3.06k op/s; 20290/84120 objects degraded (24.120%); 48/84120 objects misplaced (0.057%) 2024-09-17T07:20:19.813 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:19 smithi100 ceph-mon[39462]: 2.c scrub starts 2024-09-17T07:20:19.814 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:19 smithi100 ceph-mon[39462]: 2.c scrub ok 2024-09-17T07:20:19.814 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:19 smithi100 ceph-mon[39462]: 2.1a scrub starts 2024-09-17T07:20:19.814 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:19 smithi100 ceph-mon[39462]: 2.1a scrub ok 2024-09-17T07:20:19.814 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:19 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:20:19.814 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:19 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:20:19.814 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:19 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:20:19.814 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:19 smithi100 ceph-mon[39462]: pgmap v848: 158 pgs: 31 active+clean+remapped, 24 active+undersized, 29 active+undersized+degraded, 74 active+clean; 1.7 GiB data, 8.1 GiB used, 528 GiB / 536 GiB avail; 1.1 KiB/s rd, 191 MiB/s wr, 3.06k op/s; 20290/84120 objects degraded (24.120%); 48/84120 objects misplaced (0.057%) 2024-09-17T07:20:19.822 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:20:19.861 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:19.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:19.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:20:19.962 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:20:19.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:19.987 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:19.987 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:20:20.088 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:20:20.121 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:20.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:20.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:20:20.152 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 10 16 34580 34564 216 146.062 0.00265863 0.00462462 2024-09-17T07:20:20.223 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:20:20.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:20.261 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:20.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:20:20.330 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-17T07:20:20.331 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-17T07:20:20.331 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-17T07:20:20.331 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-17T07:20:20.331 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.331 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.15330.0:252182 9.b 9:d8c67208:::benchmark_data_smithi100_128631_object117916:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e285)", 2024-09-17T07:20:20.331 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:15:22.043583+0000", 2024-09-17T07:20:20.331 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 298.282421726, 2024-09-17T07:20:20.331 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.23440345700000001, 2024-09-17T07:20:20.331 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:20:20.331 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:20:20.331 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:20:20.331 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.15330", 2024-09-17T07:20:20.332 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/1361381749", 2024-09-17T07:20:20.332 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 252182 2024-09-17T07:20:20.332 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.332 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:20:20.332 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.332 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:20:20.332 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:15:22.043583+0000", 2024-09-17T07:20:20.332 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:20:20.332 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.332 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.332 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:20:20.333 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:15:22.043583+0000", 2024-09-17T07:20:20.333 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:20:20.333 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.333 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.333 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:20:20.333 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:15:22.043581+0000", 2024-09-17T07:20:20.333 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999986 2024-09-17T07:20:20.333 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.333 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.333 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:20:20.333 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:15:22.043587+0000", 2024-09-17T07:20:20.333 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0569999999999998e-06 2024-09-17T07:20:20.333 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.334 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.334 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:20:20.334 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:15:22.043598+0000", 2024-09-17T07:20:20.334 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0661e-05 2024-09-17T07:20:20.334 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.334 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.334 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:20:20.334 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:15:22.043605+0000", 2024-09-17T07:20:20.334 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9859999999999997e-06 2024-09-17T07:20:20.334 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.334 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.334 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:20:20.335 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:15:22.043649+0000", 2024-09-17T07:20:20.335 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3758999999999999e-05 2024-09-17T07:20:20.335 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.335 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.335 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:20:20.335 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:15:22.043756+0000", 2024-09-17T07:20:20.335 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000106946 2024-09-17T07:20:20.335 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.335 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.335 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 5,6", 2024-09-17T07:20:20.335 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:15:22.043821+0000", 2024-09-17T07:20:20.335 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4873e-05 2024-09-17T07:20:20.336 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.336 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.336 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:20:20.336 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:15:22.272918+0000", 2024-09-17T07:20:20.336 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.22909716299999999 2024-09-17T07:20:20.336 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.336 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.336 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:20:20.337 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:15:22.272965+0000", 2024-09-17T07:20:20.337 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7559999999999999e-05 2024-09-17T07:20:20.337 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.337 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.337 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:20:20.337 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:15:22.277913+0000", 2024-09-17T07:20:20.337 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0049480289999999996 2024-09-17T07:20:20.337 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.337 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.337 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:20:20.337 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:15:22.277952+0000", 2024-09-17T07:20:20.338 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8433999999999998e-05 2024-09-17T07:20:20.338 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.338 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.338 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:20:20.338 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:15:22.277986+0000", 2024-09-17T07:20:20.338 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.4365e-05 2024-09-17T07:20:20.338 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:20:20.338 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:20:20.338 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:20:20.338 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.338 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.338 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.15330.0:252180 9.c e285/236 9:3ca9e2e3:::benchmark_data_smithi100_128631_object117914:head v 285'15755, mlcod=285'15750)", 2024-09-17T07:20:20.338 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:15:22.043883+0000", 2024-09-17T07:20:20.339 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 298.28212119900002, 2024-09-17T07:20:20.339 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.23814968, 2024-09-17T07:20:20.339 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:20:20.339 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:20:20.339 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:20:20.339 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.339 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:20:20.339 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:15:22.043883+0000", 2024-09-17T07:20:20.339 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:20:20.339 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.339 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.339 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:20:20.340 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:15:22.043883+0000", 2024-09-17T07:20:20.340 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:20:20.340 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.340 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.340 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:20:20.340 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:15:22.043882+0000", 2024-09-17T07:20:20.340 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-17T07:20:20.340 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.340 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.340 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:20:20.340 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:15:22.043889+0000", 2024-09-17T07:20:20.340 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5030000000000002e-06 2024-09-17T07:20:20.340 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.341 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.341 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:20:20.341 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:15:22.270439+0000", 2024-09-17T07:20:20.341 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.22654989 2024-09-17T07:20:20.341 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.341 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.341 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:20:20.341 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:15:22.270454+0000", 2024-09-17T07:20:20.341 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5381999999999999e-05 2024-09-17T07:20:20.341 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.341 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.341 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:20:20.341 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:15:22.270518+0000", 2024-09-17T07:20:20.342 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3770999999999996e-05 2024-09-17T07:20:20.342 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.342 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.342 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:20:20.342 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:15:22.270542+0000", 2024-09-17T07:20:20.342 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4348000000000001e-05 2024-09-17T07:20:20.342 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.342 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.342 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:20:20.342 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:15:22.282002+0000", 2024-09-17T07:20:20.342 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.011459538999999999 2024-09-17T07:20:20.342 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.343 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.343 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:20:20.343 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:15:22.282033+0000", 2024-09-17T07:20:20.343 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1010999999999998e-05 2024-09-17T07:20:20.343 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:20:20.343 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:20:20.343 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:20:20.343 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.343 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.343 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop_reply(client.15330.0:252182 9.b e285/236 ondisk, result = 0)", 2024-09-17T07:20:20.343 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:15:22.047467+0000", 2024-09-17T07:20:20.344 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 298.27853713299999, 2024-09-17T07:20:20.344 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.225457144, 2024-09-17T07:20:20.344 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:20:20.344 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-17T07:20:20.344 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:20:20.344 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.344 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:20:20.344 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:15:22.047467+0000", 2024-09-17T07:20:20.344 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:20:20.344 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.344 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.344 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:20:20.345 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:15:22.047467+0000", 2024-09-17T07:20:20.345 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:20:20.345 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.345 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.345 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:20:20.345 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:15:22.047467+0000", 2024-09-17T07:20:20.345 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-17T07:20:20.345 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.345 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.345 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:20:20.345 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:15:22.047471+0000", 2024-09-17T07:20:20.345 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1910000000000002e-06 2024-09-17T07:20:20.345 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.346 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.346 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:20:20.346 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:15:22.270455+0000", 2024-09-17T07:20:20.346 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.22298426700000001 2024-09-17T07:20:20.346 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.346 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.346 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:20:20.346 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:15:22.270465+0000", 2024-09-17T07:20:20.346 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.6379999999999999e-06 2024-09-17T07:20:20.346 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.346 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.346 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:20:20.347 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:15:22.272897+0000", 2024-09-17T07:20:20.347 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0024319369999999999 2024-09-17T07:20:20.347 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.347 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.347 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:20:20.348 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:15:22.272906+0000", 2024-09-17T07:20:20.348 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8999999999999995e-06 2024-09-17T07:20:20.348 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.348 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.348 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:20:20.348 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:15:22.272924+0000", 2024-09-17T07:20:20.348 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.8834e-05 2024-09-17T07:20:20.348 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:20:20.348 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:20:20.348 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:20:20.348 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.349 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.349 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.15648.0:35695 10.8 10:1d682dd1:::benchmark_data_smithi100_155039_object35694:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e307)", 2024-09-17T07:20:20.349 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:16:34.113888+0000", 2024-09-17T07:20:20.349 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 226.212116692, 2024-09-17T07:20:20.349 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.141935122, 2024-09-17T07:20:20.349 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:20:20.349 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:20:20.349 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:20:20.349 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.15648", 2024-09-17T07:20:20.349 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/1684940078", 2024-09-17T07:20:20.349 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 35695 2024-09-17T07:20:20.349 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.349 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:20:20.350 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.350 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:20:20.350 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:16:34.113888+0000", 2024-09-17T07:20:20.350 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:20:20.350 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.350 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.350 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:20:20.350 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:16:34.113888+0000", 2024-09-17T07:20:20.350 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:20:20.350 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.350 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.350 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:20:20.350 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:16:34.113887+0000", 2024-09-17T07:20:20.351 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-17T07:20:20.351 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.351 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.351 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:20:20.351 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:16:34.113908+0000", 2024-09-17T07:20:20.351 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1509999999999999e-05 2024-09-17T07:20:20.351 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.351 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.351 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:20:20.351 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:16:34.113917+0000", 2024-09-17T07:20:20.351 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3669999999999997e-06 2024-09-17T07:20:20.351 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.352 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.352 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:20:20.352 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:16:34.113923+0000", 2024-09-17T07:20:20.352 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0390000000000001e-06 2024-09-17T07:20:20.352 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.352 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.352 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:20:20.352 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:16:34.113972+0000", 2024-09-17T07:20:20.352 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8813000000000002e-05 2024-09-17T07:20:20.352 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.352 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.352 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:20:20.353 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:16:34.114140+0000", 2024-09-17T07:20:20.353 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016840599999999999 2024-09-17T07:20:20.353 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.353 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.353 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 2,7", 2024-09-17T07:20:20.353 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:16:34.114290+0000", 2024-09-17T07:20:20.353 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014960700000000001 2024-09-17T07:20:20.353 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.353 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.353 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:20:20.353 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:16:34.116339+0000", 2024-09-17T07:20:20.353 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0020495959999999999 2024-09-17T07:20:20.354 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.354 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.354 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:20:20.354 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:16:34.118225+0000", 2024-09-17T07:20:20.354 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0018855930000000001 2024-09-17T07:20:20.354 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.354 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.354 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:20:20.354 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:16:34.255733+0000", 2024-09-17T07:20:20.354 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.13750812200000001 2024-09-17T07:20:20.354 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.354 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.354 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:20:20.355 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:16:34.255783+0000", 2024-09-17T07:20:20.355 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9815000000000003e-05 2024-09-17T07:20:20.355 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.355 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.355 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:20:20.355 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:16:34.255823+0000", 2024-09-17T07:20:20.355 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0113e-05 2024-09-17T07:20:20.355 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:20:20.355 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:20:20.355 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:20:20.355 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.355 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.355 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.15648.0:41081 10.4 10:2a773f83:::benchmark_data_smithi100_155039_object41080:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e309)", 2024-09-17T07:20:20.356 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:16:37.512280+0000", 2024-09-17T07:20:20.356 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 222.813724697, 2024-09-17T07:20:20.356 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.199960315, 2024-09-17T07:20:20.356 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:20:20.356 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:20:20.356 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:20:20.356 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.15648", 2024-09-17T07:20:20.356 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/1684940078", 2024-09-17T07:20:20.356 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 41081 2024-09-17T07:20:20.356 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.356 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:20:20.356 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.356 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:20:20.357 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:16:37.512280+0000", 2024-09-17T07:20:20.357 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:20:20.357 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.357 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.357 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:20:20.357 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:16:37.512280+0000", 2024-09-17T07:20:20.357 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:20:20.357 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.357 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.357 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:20:20.357 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:16:37.512278+0000", 2024-09-17T07:20:20.357 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999986 2024-09-17T07:20:20.357 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.358 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.358 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:20:20.358 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:16:37.512330+0000", 2024-09-17T07:20:20.358 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1232e-05 2024-09-17T07:20:20.358 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.358 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.358 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:20:20.358 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:16:37.512340+0000", 2024-09-17T07:20:20.358 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0078000000000001e-05 2024-09-17T07:20:20.358 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.358 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.358 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:20:20.358 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:16:37.512348+0000", 2024-09-17T07:20:20.359 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9859999999999994e-06 2024-09-17T07:20:20.359 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.359 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.359 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:20:20.359 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:16:37.513118+0000", 2024-09-17T07:20:20.359 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00077026899999999997 2024-09-17T07:20:20.359 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.359 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.359 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:20:20.359 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:16:37.513278+0000", 2024-09-17T07:20:20.359 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000160261 2024-09-17T07:20:20.359 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.359 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.360 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-17T07:20:20.360 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:16:37.513420+0000", 2024-09-17T07:20:20.360 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014158899999999999 2024-09-17T07:20:20.360 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.360 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.360 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:20:20.360 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:16:37.515450+0000", 2024-09-17T07:20:20.360 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0020297549999999998 2024-09-17T07:20:20.360 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.360 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.360 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:20:20.360 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:16:37.712080+0000", 2024-09-17T07:20:20.360 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.1966302 2024-09-17T07:20:20.361 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.361 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.361 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:20:20.361 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:16:37.712144+0000", 2024-09-17T07:20:20.361 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4437999999999995e-05 2024-09-17T07:20:20.361 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.361 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.361 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:20:20.361 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:16:37.712240+0000", 2024-09-17T07:20:20.361 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5740000000000002e-05 2024-09-17T07:20:20.361 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:20:20.361 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:20:20.362 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:20:20.362 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.362 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.362 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.15648.0:83761 10.4 10:28cb61e0:::benchmark_data_smithi100_155039_object83760:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e316)", 2024-09-17T07:20:20.362 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:17:11.263554+0000", 2024-09-17T07:20:20.362 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 189.06245017000001, 2024-09-17T07:20:20.362 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.240823075, 2024-09-17T07:20:20.362 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:20:20.362 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:20:20.362 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:20:20.362 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.15648", 2024-09-17T07:20:20.362 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/1684940078", 2024-09-17T07:20:20.363 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 83761 2024-09-17T07:20:20.363 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.363 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:20:20.363 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.363 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:20:20.363 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:11.263554+0000", 2024-09-17T07:20:20.363 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:20:20.363 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.363 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.363 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:20:20.363 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:11.263554+0000", 2024-09-17T07:20:20.363 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:20:20.363 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.364 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.364 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:20:20.364 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:11.263553+0000", 2024-09-17T07:20:20.364 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999986 2024-09-17T07:20:20.364 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.364 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.364 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:20:20.364 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:11.263574+0000", 2024-09-17T07:20:20.364 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1512999999999999e-05 2024-09-17T07:20:20.364 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.364 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.364 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:20:20.364 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:11.263585+0000", 2024-09-17T07:20:20.365 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0154e-05 2024-09-17T07:20:20.365 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.365 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.365 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:20:20.365 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:11.263591+0000", 2024-09-17T07:20:20.365 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7089999999999996e-06 2024-09-17T07:20:20.365 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.365 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.365 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:20:20.365 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:11.263639+0000", 2024-09-17T07:20:20.365 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8115999999999998e-05 2024-09-17T07:20:20.366 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.366 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.366 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:20:20.366 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:11.263912+0000", 2024-09-17T07:20:20.366 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00027218499999999998 2024-09-17T07:20:20.366 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.366 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.366 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-17T07:20:20.366 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:11.264067+0000", 2024-09-17T07:20:20.366 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00015507100000000001 2024-09-17T07:20:20.366 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.366 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.366 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:20:20.367 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:11.266454+0000", 2024-09-17T07:20:20.367 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0023876990000000001 2024-09-17T07:20:20.367 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.367 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.367 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:20:20.367 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:11.504209+0000", 2024-09-17T07:20:20.367 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.23775510599999999 2024-09-17T07:20:20.367 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.367 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.367 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:20:20.367 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:11.504302+0000", 2024-09-17T07:20:20.367 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.2962999999999997e-05 2024-09-17T07:20:20.367 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.368 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.368 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:20:20.368 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:11.504377+0000", 2024-09-17T07:20:20.368 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4884999999999999e-05 2024-09-17T07:20:20.368 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:20:20.368 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:20:20.368 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:20:20.368 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.368 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.368 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.15648.0:83771 10.4 10:203ac25f:::benchmark_data_smithi100_155039_object83770:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e316)", 2024-09-17T07:20:20.368 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:17:11.312220+0000", 2024-09-17T07:20:20.368 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 189.01378410500001, 2024-09-17T07:20:20.369 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.192334278, 2024-09-17T07:20:20.369 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:20:20.369 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:20:20.369 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:20:20.369 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.15648", 2024-09-17T07:20:20.369 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/1684940078", 2024-09-17T07:20:20.369 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 83771 2024-09-17T07:20:20.369 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.369 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:20:20.369 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.369 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:20:20.369 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:11.312220+0000", 2024-09-17T07:20:20.370 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:20:20.370 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.370 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.370 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:20:20.370 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:11.312220+0000", 2024-09-17T07:20:20.370 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:20:20.370 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.370 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.370 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:20:20.370 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:11.312219+0000", 2024-09-17T07:20:20.370 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999986 2024-09-17T07:20:20.370 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.371 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.371 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:20:20.371 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:11.312239+0000", 2024-09-17T07:20:20.371 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.9621999999999999e-05 2024-09-17T07:20:20.371 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.371 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.371 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:20:20.371 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:11.312249+0000", 2024-09-17T07:20:20.371 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.045e-05 2024-09-17T07:20:20.371 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.371 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.371 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:20:20.371 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:11.312254+0000", 2024-09-17T07:20:20.372 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9119999999999997e-06 2024-09-17T07:20:20.372 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.372 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.372 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:20:20.372 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:11.312293+0000", 2024-09-17T07:20:20.372 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8757000000000002e-05 2024-09-17T07:20:20.372 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.372 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.372 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:20:20.372 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:11.312432+0000", 2024-09-17T07:20:20.372 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000139424 2024-09-17T07:20:20.372 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.373 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.373 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-17T07:20:20.373 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:11.312602+0000", 2024-09-17T07:20:20.373 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00017018299999999999 2024-09-17T07:20:20.373 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.373 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.373 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:20:20.373 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:11.314824+0000", 2024-09-17T07:20:20.373 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0022214769999999999 2024-09-17T07:20:20.373 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.373 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.373 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:20:20.373 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:11.504423+0000", 2024-09-17T07:20:20.374 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.189598927 2024-09-17T07:20:20.374 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.374 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.374 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:20:20.374 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:11.504473+0000", 2024-09-17T07:20:20.374 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9947999999999997e-05 2024-09-17T07:20:20.374 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.374 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.374 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:20:20.374 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:11.504555+0000", 2024-09-17T07:20:20.374 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1929999999999997e-05 2024-09-17T07:20:20.374 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:20:20.374 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:20:20.375 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:20:20.375 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.375 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.375 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.15648.0:83819 10.4 10:2a14f068:::benchmark_data_smithi100_155039_object83818:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e316)", 2024-09-17T07:20:20.375 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:17:11.363061+0000", 2024-09-17T07:20:20.375 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 188.962942997, 2024-09-17T07:20:20.375 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.14186207000000001, 2024-09-17T07:20:20.375 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:20:20.375 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:20:20.375 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:20:20.375 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.15648", 2024-09-17T07:20:20.375 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/1684940078", 2024-09-17T07:20:20.375 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 83819 2024-09-17T07:20:20.376 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.376 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:20:20.376 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.376 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:20:20.376 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:11.363061+0000", 2024-09-17T07:20:20.376 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:20:20.376 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.376 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.376 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:20:20.376 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:11.363061+0000", 2024-09-17T07:20:20.376 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:20:20.376 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.377 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.377 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:20:20.377 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:11.363061+0000", 2024-09-17T07:20:20.377 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-17T07:20:20.377 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.377 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.377 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:20:20.377 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:11.363080+0000", 2024-09-17T07:20:20.377 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.8927e-05 2024-09-17T07:20:20.377 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.377 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.378 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:20:20.379 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:20:20.379 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:11.363087+0000", 2024-09-17T07:20:20.379 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8560000000000001e-06 2024-09-17T07:20:20.379 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.379 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.379 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:20:20.379 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:11.363092+0000", 2024-09-17T07:20:20.379 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3569999999999997e-06 2024-09-17T07:20:20.379 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.380 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.380 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:20:20.380 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:11.365776+0000", 2024-09-17T07:20:20.380 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0026835320000000002 2024-09-17T07:20:20.380 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.380 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.380 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:20:20.380 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:11.365974+0000", 2024-09-17T07:20:20.380 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00019813 2024-09-17T07:20:20.380 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.380 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.380 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-17T07:20:20.380 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:11.366152+0000", 2024-09-17T07:20:20.381 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000178226 2024-09-17T07:20:20.381 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.381 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.381 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:20:20.381 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:11.368359+0000", 2024-09-17T07:20:20.381 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0022068840000000001 2024-09-17T07:20:20.381 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.381 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.381 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:20:20.381 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:11.504778+0000", 2024-09-17T07:20:20.381 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.13641927500000001 2024-09-17T07:20:20.381 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.381 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.382 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:20:20.382 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:11.504831+0000", 2024-09-17T07:20:20.382 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3001000000000003e-05 2024-09-17T07:20:20.382 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.382 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.382 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:20:20.382 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:11.504923+0000", 2024-09-17T07:20:20.382 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.2348000000000004e-05 2024-09-17T07:20:20.382 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:20:20.382 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:20:20.382 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:20:20.382 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.383 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.383 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.15648.0:126542 10.11 10:8b03bc50:::benchmark_data_smithi100_155039_object3194:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e336)", 2024-09-17T07:20:20.383 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:17:56.397396+0000", 2024-09-17T07:20:20.383 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 143.928608357, 2024-09-17T07:20:20.383 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.203175737, 2024-09-17T07:20:20.383 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:20:20.383 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:20:20.383 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:20:20.383 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.15648", 2024-09-17T07:20:20.383 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/1684940078", 2024-09-17T07:20:20.383 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 126542 2024-09-17T07:20:20.383 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.383 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:20:20.384 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.384 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:20:20.384 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:56.397396+0000", 2024-09-17T07:20:20.384 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:20:20.384 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.384 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.384 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:20:20.384 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:56.397396+0000", 2024-09-17T07:20:20.384 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:20:20.384 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.384 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.384 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:20:20.384 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:56.397394+0000", 2024-09-17T07:20:20.385 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999986 2024-09-17T07:20:20.385 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.385 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.385 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:20:20.385 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:56.397401+0000", 2024-09-17T07:20:20.385 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4479999999999999e-06 2024-09-17T07:20:20.385 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.385 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.385 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:20:20.385 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:56.397413+0000", 2024-09-17T07:20:20.385 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1583000000000001e-05 2024-09-17T07:20:20.385 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.385 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.386 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:20:20.386 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:56.397421+0000", 2024-09-17T07:20:20.386 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1130000000000001e-06 2024-09-17T07:20:20.386 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.386 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.386 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:20:20.386 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:56.397463+0000", 2024-09-17T07:20:20.386 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2582e-05 2024-09-17T07:20:20.386 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.386 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.386 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:20:20.386 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:56.397622+0000", 2024-09-17T07:20:20.386 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00015905800000000001 2024-09-17T07:20:20.387 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.387 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.387 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4,5,7", 2024-09-17T07:20:20.387 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:56.397874+0000", 2024-09-17T07:20:20.387 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00025189 2024-09-17T07:20:20.387 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.387 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.387 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:20:20.387 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:56.399021+0000", 2024-09-17T07:20:20.387 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001147098 2024-09-17T07:20:20.387 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.387 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.387 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:20:20.388 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:56.399575+0000", 2024-09-17T07:20:20.388 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00055383899999999998 2024-09-17T07:20:20.388 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.388 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.388 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:20:20.388 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:56.400878+0000", 2024-09-17T07:20:20.388 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00130299 2024-09-17T07:20:20.388 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.388 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.388 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:20:20.388 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:56.600349+0000", 2024-09-17T07:20:20.388 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.199471116 2024-09-17T07:20:20.388 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.389 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.389 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:20:20.389 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:56.600399+0000", 2024-09-17T07:20:20.389 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9580000000000003e-05 2024-09-17T07:20:20.389 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.389 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.389 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:20:20.389 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:56.600572+0000", 2024-09-17T07:20:20.389 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00017297699999999999 2024-09-17T07:20:20.389 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:20:20.389 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:20:20.389 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:20:20.389 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.390 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.390 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.15648.0:126544 10.11 10:89a55ce1:::benchmark_data_smithi100_155039_object3196:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e336)", 2024-09-17T07:20:20.390 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:17:56.399517+0000", 2024-09-17T07:20:20.390 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 143.92648783300001, 2024-09-17T07:20:20.390 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.20125437700000001, 2024-09-17T07:20:20.390 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:20:20.390 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:20:20.390 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:20:20.390 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.15648", 2024-09-17T07:20:20.390 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/1684940078", 2024-09-17T07:20:20.390 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 126544 2024-09-17T07:20:20.390 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.390 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:20:20.391 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.391 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:20:20.391 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:56.399517+0000", 2024-09-17T07:20:20.391 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:20:20.391 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.391 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.391 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:20:20.391 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:56.399517+0000", 2024-09-17T07:20:20.391 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:20:20.391 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.391 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.391 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:20:20.391 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:56.399515+0000", 2024-09-17T07:20:20.392 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999986 2024-09-17T07:20:20.392 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.392 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.392 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:20:20.392 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:56.399522+0000", 2024-09-17T07:20:20.392 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2210000000000002e-06 2024-09-17T07:20:20.392 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.392 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.392 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:20:20.392 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:56.399542+0000", 2024-09-17T07:20:20.392 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0888000000000001e-05 2024-09-17T07:20:20.392 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.392 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.393 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:20:20.393 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:56.399552+0000", 2024-09-17T07:20:20.393 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.6379999999999999e-06 2024-09-17T07:20:20.393 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.393 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.393 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:20:20.393 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:56.399608+0000", 2024-09-17T07:20:20.393 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5618999999999997e-05 2024-09-17T07:20:20.393 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.393 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.393 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:20:20.393 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:56.399731+0000", 2024-09-17T07:20:20.394 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012286600000000001 2024-09-17T07:20:20.394 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.394 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.394 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4,5,7", 2024-09-17T07:20:20.394 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:56.399951+0000", 2024-09-17T07:20:20.394 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000219965 2024-09-17T07:20:20.394 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.394 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.394 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:20:20.394 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:56.401990+0000", 2024-09-17T07:20:20.394 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0020396860000000002 2024-09-17T07:20:20.394 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.394 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.395 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:20:20.395 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:56.402084+0000", 2024-09-17T07:20:20.395 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.4078000000000003e-05 2024-09-17T07:20:20.395 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.395 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.395 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:20:20.395 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:56.403206+0000", 2024-09-17T07:20:20.395 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0011217390000000001 2024-09-17T07:20:20.395 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.395 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.395 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:20:20.395 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:56.600619+0000", 2024-09-17T07:20:20.395 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.19741346200000001 2024-09-17T07:20:20.396 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.396 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.396 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:20:20.396 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:56.600666+0000", 2024-09-17T07:20:20.396 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6961e-05 2024-09-17T07:20:20.396 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.396 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.396 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:20:20.396 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:56.600771+0000", 2024-09-17T07:20:20.396 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000104474 2024-09-17T07:20:20.396 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:20:20.396 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:20:20.396 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:20:20.397 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.397 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.397 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.15648.0:126557 10.11 10:885f8ab8:::benchmark_data_smithi100_155039_object3209:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e336)", 2024-09-17T07:20:20.397 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:17:56.441956+0000", 2024-09-17T07:20:20.397 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 143.884048165, 2024-09-17T07:20:20.397 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.15905651200000001, 2024-09-17T07:20:20.397 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:20:20.397 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:20:20.397 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:20:20.397 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.15648", 2024-09-17T07:20:20.397 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/1684940078", 2024-09-17T07:20:20.397 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 126557 2024-09-17T07:20:20.397 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.398 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:20:20.398 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.398 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:20:20.398 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:56.441956+0000", 2024-09-17T07:20:20.398 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:20:20.398 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.398 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.398 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:20:20.398 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:56.441956+0000", 2024-09-17T07:20:20.398 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:20:20.398 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.398 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.398 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:20:20.399 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:56.441955+0000", 2024-09-17T07:20:20.399 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999986 2024-09-17T07:20:20.399 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.399 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.399 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:20:20.399 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:56.441961+0000", 2024-09-17T07:20:20.399 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3060000000000003e-06 2024-09-17T07:20:20.399 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.399 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.399 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:20:20.399 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:56.441973+0000", 2024-09-17T07:20:20.399 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2147000000000001e-05 2024-09-17T07:20:20.399 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.400 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.400 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:20:20.400 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:56.441981+0000", 2024-09-17T07:20:20.400 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5170000000000002e-06 2024-09-17T07:20:20.400 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.400 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.400 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:20:20.400 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:56.442024+0000", 2024-09-17T07:20:20.400 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3405999999999997e-05 2024-09-17T07:20:20.400 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.400 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.400 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:20:20.400 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:56.442109+0000", 2024-09-17T07:20:20.401 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.4529000000000003e-05 2024-09-17T07:20:20.401 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.401 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.401 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4,5,7", 2024-09-17T07:20:20.401 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:56.442407+0000", 2024-09-17T07:20:20.401 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000298611 2024-09-17T07:20:20.401 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.401 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.401 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:20:20.401 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:56.443394+0000", 2024-09-17T07:20:20.401 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00098694699999999995 2024-09-17T07:20:20.401 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.401 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.402 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:20:20.402 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:56.443903+0000", 2024-09-17T07:20:20.402 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00050879799999999998 2024-09-17T07:20:20.402 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.402 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.402 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:20:20.402 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:56.444211+0000", 2024-09-17T07:20:20.402 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00030812 2024-09-17T07:20:20.402 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.402 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.402 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:20:20.402 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:56.600833+0000", 2024-09-17T07:20:20.402 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.15662225499999999 2024-09-17T07:20:20.403 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.403 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.403 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:20:20.403 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:56.600889+0000", 2024-09-17T07:20:20.403 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5333e-05 2024-09-17T07:20:20.403 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.403 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.403 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:20:20.403 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:17:56.601013+0000", 2024-09-17T07:20:20.403 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012411 2024-09-17T07:20:20.403 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:20:20.403 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:20:20.403 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:20:20.404 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.404 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.404 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.15648.0:217460 10.1e 10:7ce1a03c:::benchmark_data_smithi100_155039_object94112:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-09-17T07:20:20.404 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:19:15.315628+0000", 2024-09-17T07:20:20.404 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 65.010376136000005, 2024-09-17T07:20:20.404 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.98678144999999995, 2024-09-17T07:20:20.404 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:20:20.404 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:20:20.404 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:20:20.404 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.15648", 2024-09-17T07:20:20.404 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/1684940078", 2024-09-17T07:20:20.404 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 217460 2024-09-17T07:20:20.404 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.405 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:20:20.405 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.405 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:20:20.405 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:15.315628+0000", 2024-09-17T07:20:20.405 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:20:20.405 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.405 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.405 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:20:20.405 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:15.315628+0000", 2024-09-17T07:20:20.405 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:20:20.405 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.405 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.405 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:20:20.406 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:15.315627+0000", 2024-09-17T07:20:20.406 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999986 2024-09-17T07:20:20.406 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.406 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.406 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:20:20.406 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:15.315633+0000", 2024-09-17T07:20:20.406 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4289999999999998e-06 2024-09-17T07:20:20.406 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.406 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.407 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:20:20.407 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:15.315645+0000", 2024-09-17T07:20:20.407 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1491000000000001e-05 2024-09-17T07:20:20.407 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.407 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.407 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:20:20.407 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:15.315653+0000", 2024-09-17T07:20:20.407 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9349999999999994e-06 2024-09-17T07:20:20.407 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.407 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.407 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:20:20.407 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:15.315701+0000", 2024-09-17T07:20:20.407 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7956e-05 2024-09-17T07:20:20.407 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.408 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.408 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-17T07:20:20.408 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:15.315710+0000", 2024-09-17T07:20:20.408 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3260000000000002e-06 2024-09-17T07:20:20.408 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.408 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.408 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:20:20.408 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:16.299587+0000", 2024-09-17T07:20:20.408 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.983877012 2024-09-17T07:20:20.408 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.408 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.408 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:20:20.408 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:16.299686+0000", 2024-09-17T07:20:20.409 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.8768e-05 2024-09-17T07:20:20.409 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.409 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.409 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 2,5", 2024-09-17T07:20:20.409 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:16.299836+0000", 2024-09-17T07:20:20.409 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014988499999999999 2024-09-17T07:20:20.409 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.409 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.409 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:20:20.409 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:16.302317+0000", 2024-09-17T07:20:20.409 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0024810050000000001 2024-09-17T07:20:20.409 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.410 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.410 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:20:20.410 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:16.302325+0000", 2024-09-17T07:20:20.410 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8370000000000003e-06 2024-09-17T07:20:20.410 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.410 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.410 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:20:20.410 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:16.302344+0000", 2024-09-17T07:20:20.410 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.8337000000000001e-05 2024-09-17T07:20:20.410 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.410 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.410 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:20:20.410 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:16.302370+0000", 2024-09-17T07:20:20.411 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.6438e-05 2024-09-17T07:20:20.411 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.411 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.411 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:20:20.411 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:16.302410+0000", 2024-09-17T07:20:20.411 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9524000000000003e-05 2024-09-17T07:20:20.411 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:20:20.411 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:20:20.411 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:20:20.411 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.411 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.411 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.15648.0:217476 10.1e 10:7cc891cc:::benchmark_data_smithi100_155039_object94128:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-09-17T07:20:20.412 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:19:15.323410+0000", 2024-09-17T07:20:20.412 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 65.002594080999998, 2024-09-17T07:20:20.412 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.98077838299999998, 2024-09-17T07:20:20.412 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:20:20.412 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:20:20.412 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:20:20.412 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.15648", 2024-09-17T07:20:20.412 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/1684940078", 2024-09-17T07:20:20.412 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 217476 2024-09-17T07:20:20.412 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.412 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:20:20.412 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.412 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:20:20.413 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:15.323410+0000", 2024-09-17T07:20:20.413 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:20:20.413 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.413 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.413 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:20:20.413 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:15.323410+0000", 2024-09-17T07:20:20.413 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:20:20.413 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.413 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.413 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:20:20.413 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:15.323409+0000", 2024-09-17T07:20:20.413 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999986 2024-09-17T07:20:20.413 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.414 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.414 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:20:20.414 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:15.323415+0000", 2024-09-17T07:20:20.414 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2910000000000004e-06 2024-09-17T07:20:20.414 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.414 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.414 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:20:20.414 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:15.323428+0000", 2024-09-17T07:20:20.414 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2598e-05 2024-09-17T07:20:20.414 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.414 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.414 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:20:20.415 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:15.323436+0000", 2024-09-17T07:20:20.415 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9650000000000008e-06 2024-09-17T07:20:20.415 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.415 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.415 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:20:20.415 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:15.323488+0000", 2024-09-17T07:20:20.415 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2444999999999997e-05 2024-09-17T07:20:20.415 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.415 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.415 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-17T07:20:20.416 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:15.323497+0000", 2024-09-17T07:20:20.416 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.1130000000000007e-06 2024-09-17T07:20:20.416 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.416 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.416 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:20:20.416 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:16.300825+0000", 2024-09-17T07:20:20.416 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.97732723899999996 2024-09-17T07:20:20.416 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.417 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.417 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:20:20.417 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:16.300929+0000", 2024-09-17T07:20:20.417 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00010420399999999999 2024-09-17T07:20:20.417 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.417 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.417 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 2,5", 2024-09-17T07:20:20.417 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:16.301061+0000", 2024-09-17T07:20:20.417 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00013200000000000001 2024-09-17T07:20:20.417 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.417 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.417 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:20:20.418 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:16.302515+0000", 2024-09-17T07:20:20.418 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0014538579999999999 2024-09-17T07:20:20.418 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.418 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.418 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:20:20.418 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:16.303191+0000", 2024-09-17T07:20:20.418 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000675955 2024-09-17T07:20:20.418 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.418 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.418 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:20:20.418 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:16.304099+0000", 2024-09-17T07:20:20.418 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00090887699999999999 2024-09-17T07:20:20.419 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.419 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.419 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:20:20.419 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:16.304152+0000", 2024-09-17T07:20:20.419 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2441999999999997e-05 2024-09-17T07:20:20.419 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.419 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.419 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:20:20.419 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:16.304189+0000", 2024-09-17T07:20:20.419 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6797e-05 2024-09-17T07:20:20.420 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:20:20.420 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:20:20.420 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:20:20.420 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.420 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.420 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.15648.0:217503 10.1e 10:7bfb83fe:::benchmark_data_smithi100_155039_object94155:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-09-17T07:20:20.420 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:19:15.340509+0000", 2024-09-17T07:20:20.420 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 64.985494994000007, 2024-09-17T07:20:20.420 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.96555744099999996, 2024-09-17T07:20:20.420 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:20:20.420 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:20:20.420 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:20:20.420 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.15648", 2024-09-17T07:20:20.421 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/1684940078", 2024-09-17T07:20:20.421 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 217503 2024-09-17T07:20:20.421 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.421 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:20:20.421 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.421 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:20:20.421 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:15.340509+0000", 2024-09-17T07:20:20.421 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:20:20.421 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.421 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.421 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:20:20.421 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:15.340509+0000", 2024-09-17T07:20:20.422 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:20:20.422 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.422 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.422 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:20:20.422 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:15.340508+0000", 2024-09-17T07:20:20.422 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-17T07:20:20.422 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.422 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.422 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:20:20.422 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:15.340513+0000", 2024-09-17T07:20:20.422 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9400000000000001e-06 2024-09-17T07:20:20.422 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.422 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.423 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:20:20.423 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:15.340524+0000", 2024-09-17T07:20:20.423 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0446e-05 2024-09-17T07:20:20.423 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.423 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.423 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:20:20.423 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:15.340530+0000", 2024-09-17T07:20:20.423 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.789e-06 2024-09-17T07:20:20.423 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.423 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.423 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:20:20.423 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:15.340565+0000", 2024-09-17T07:20:20.423 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5054e-05 2024-09-17T07:20:20.423 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.424 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.424 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-17T07:20:20.424 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:15.340571+0000", 2024-09-17T07:20:20.424 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.815e-06 2024-09-17T07:20:20.424 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.424 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.424 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:20:20.424 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:16.301661+0000", 2024-09-17T07:20:20.424 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.96108962200000003 2024-09-17T07:20:20.424 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.424 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.425 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:20:20.425 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:16.301721+0000", 2024-09-17T07:20:20.426 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9537999999999998e-05 2024-09-17T07:20:20.426 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.426 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.426 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 2,5", 2024-09-17T07:20:20.426 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:16.301829+0000", 2024-09-17T07:20:20.426 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00010873400000000001 2024-09-17T07:20:20.426 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.426 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.426 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:20:20.426 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:16.303197+0000", 2024-09-17T07:20:20.426 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001367343 2024-09-17T07:20:20.426 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.427 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.427 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:20:20.427 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:16.304047+0000", 2024-09-17T07:20:20.427 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00084986800000000004 2024-09-17T07:20:20.427 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.427 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.427 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:20:20.427 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:16.306003+0000", 2024-09-17T07:20:20.427 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0019562109999999998 2024-09-17T07:20:20.427 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.427 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.427 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:20:20.427 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:16.306042+0000", 2024-09-17T07:20:20.428 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9579999999999997e-05 2024-09-17T07:20:20.428 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.428 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.428 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:20:20.428 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:16.306067+0000", 2024-09-17T07:20:20.428 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4450000000000001e-05 2024-09-17T07:20:20.428 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:20:20.428 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:20:20.428 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:20:20.428 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.428 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.428 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.15648.0:217518 10.1e 10:79e86715:::benchmark_data_smithi100_155039_object94170:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-09-17T07:20:20.429 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:19:15.348953+0000", 2024-09-17T07:20:20.429 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 64.977051398, 2024-09-17T07:20:20.429 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.95725205499999999, 2024-09-17T07:20:20.429 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:20:20.429 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:20:20.429 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:20:20.429 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.15648", 2024-09-17T07:20:20.429 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/1684940078", 2024-09-17T07:20:20.429 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 217518 2024-09-17T07:20:20.429 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.429 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:20:20.429 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.430 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:20:20.430 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:15.348953+0000", 2024-09-17T07:20:20.430 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:20:20.430 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.430 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.430 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:20:20.430 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:15.348953+0000", 2024-09-17T07:20:20.430 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:20:20.430 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.430 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.430 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:20:20.430 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:15.348952+0000", 2024-09-17T07:20:20.430 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-17T07:20:20.431 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.431 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.431 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:20:20.431 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:15.348957+0000", 2024-09-17T07:20:20.431 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8180000000000003e-06 2024-09-17T07:20:20.431 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.431 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.431 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:20:20.431 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:15.348969+0000", 2024-09-17T07:20:20.431 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2221e-05 2024-09-17T07:20:20.431 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.431 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.431 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:20:20.432 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:15.348977+0000", 2024-09-17T07:20:20.432 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5490000000000001e-06 2024-09-17T07:20:20.432 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.432 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.432 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:20:20.432 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:15.349018+0000", 2024-09-17T07:20:20.432 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1238000000000001e-05 2024-09-17T07:20:20.432 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.432 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.432 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-17T07:20:20.432 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:15.349026+0000", 2024-09-17T07:20:20.432 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3429999999999996e-06 2024-09-17T07:20:20.432 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.433 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.433 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:20:20.433 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:16.302544+0000", 2024-09-17T07:20:20.433 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.95351777800000004 2024-09-17T07:20:20.433 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.433 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.433 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:20:20.433 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:16.302617+0000", 2024-09-17T07:20:20.433 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2757999999999997e-05 2024-09-17T07:20:20.433 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.433 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.433 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 2,5", 2024-09-17T07:20:20.433 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:16.302724+0000", 2024-09-17T07:20:20.434 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000107005 2024-09-17T07:20:20.434 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.434 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.434 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:20:20.434 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:16.304998+0000", 2024-09-17T07:20:20.434 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0022741699999999998 2024-09-17T07:20:20.434 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.434 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.434 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:20:20.434 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:16.305951+0000", 2024-09-17T07:20:20.434 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000953326 2024-09-17T07:20:20.434 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.434 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.435 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:20:20.435 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:16.306145+0000", 2024-09-17T07:20:20.435 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000194277 2024-09-17T07:20:20.435 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.435 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.435 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:20:20.435 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:16.306181+0000", 2024-09-17T07:20:20.435 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5571000000000002e-05 2024-09-17T07:20:20.435 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.435 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.435 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:20:20.435 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:16.306205+0000", 2024-09-17T07:20:20.435 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4031000000000001e-05 2024-09-17T07:20:20.436 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:20:20.436 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:20:20.436 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:20:20.436 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.436 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.436 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.15648.0:217532 10.1e 10:7d9a3c56:::benchmark_data_smithi100_155039_object94184:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-09-17T07:20:20.436 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:19:15.360873+0000", 2024-09-17T07:20:20.436 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 64.965131802000002, 2024-09-17T07:20:20.436 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.94545949399999996, 2024-09-17T07:20:20.436 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:20:20.436 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:20:20.436 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:20:20.436 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.15648", 2024-09-17T07:20:20.437 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/1684940078", 2024-09-17T07:20:20.437 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 217532 2024-09-17T07:20:20.437 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.437 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:20:20.437 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.437 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:20:20.437 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:15.360873+0000", 2024-09-17T07:20:20.437 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:20:20.437 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.437 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.437 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:20:20.437 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:15.360873+0000", 2024-09-17T07:20:20.437 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:20:20.438 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.438 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.438 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:20:20.438 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:15.360871+0000", 2024-09-17T07:20:20.438 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-17T07:20:20.438 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.438 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.438 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:20:20.438 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:15.360877+0000", 2024-09-17T07:20:20.438 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8159999999999999e-06 2024-09-17T07:20:20.438 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.438 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.438 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:20:20.438 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:15.360890+0000", 2024-09-17T07:20:20.439 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2717e-05 2024-09-17T07:20:20.439 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.439 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.439 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:20:20.439 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:15.360898+0000", 2024-09-17T07:20:20.439 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8129999999999996e-06 2024-09-17T07:20:20.439 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.439 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.439 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:20:20.439 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:15.360943+0000", 2024-09-17T07:20:20.439 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5155999999999999e-05 2024-09-17T07:20:20.439 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.440 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.440 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-17T07:20:20.440 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:15.360951+0000", 2024-09-17T07:20:20.440 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8790000000000003e-06 2024-09-17T07:20:20.440 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.440 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.440 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:20:20.440 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:16.303220+0000", 2024-09-17T07:20:20.440 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.942269464 2024-09-17T07:20:20.440 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.440 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.440 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:20:20.441 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:16.303279+0000", 2024-09-17T07:20:20.441 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8445e-05 2024-09-17T07:20:20.441 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.441 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.441 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 2,5", 2024-09-17T07:20:20.441 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:16.303375+0000", 2024-09-17T07:20:20.441 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.6169999999999996e-05 2024-09-17T07:20:20.441 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.441 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.441 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:20:20.441 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:16.305958+0000", 2024-09-17T07:20:20.441 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0025834859999999999 2024-09-17T07:20:20.441 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.442 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.442 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:20:20.442 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:16.306100+0000", 2024-09-17T07:20:20.442 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000141278 2024-09-17T07:20:20.442 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.442 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.442 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:20:20.442 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:16.306287+0000", 2024-09-17T07:20:20.442 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00018683999999999999 2024-09-17T07:20:20.442 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.442 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.442 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:20:20.443 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:16.306313+0000", 2024-09-17T07:20:20.443 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.6747e-05 2024-09-17T07:20:20.443 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.443 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.443 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:20:20.443 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:16.306332+0000", 2024-09-17T07:20:20.443 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.8779000000000001e-05 2024-09-17T07:20:20.443 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:20:20.443 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:20:20.443 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:20:20.443 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.444 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.444 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.15648.0:217534 10.1e 10:7ae2b87c:::benchmark_data_smithi100_155039_object94186:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-09-17T07:20:20.444 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:19:15.361539+0000", 2024-09-17T07:20:20.444 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 64.964465218000001, 2024-09-17T07:20:20.444 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.94491024700000004, 2024-09-17T07:20:20.444 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:20:20.444 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:20:20.444 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:20:20.444 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.15648", 2024-09-17T07:20:20.444 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/1684940078", 2024-09-17T07:20:20.444 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 217534 2024-09-17T07:20:20.444 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.445 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:20:20.445 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.445 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:20:20.445 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:15.361539+0000", 2024-09-17T07:20:20.445 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:20:20.445 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.445 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.445 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:20:20.445 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:15.361539+0000", 2024-09-17T07:20:20.445 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:20:20.445 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.445 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.445 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:20:20.446 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:15.361538+0000", 2024-09-17T07:20:20.446 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-17T07:20:20.446 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.446 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.446 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:20:20.446 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:15.361543+0000", 2024-09-17T07:20:20.446 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1769999999999997e-06 2024-09-17T07:20:20.446 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.446 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.446 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:20:20.446 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:15.361554+0000", 2024-09-17T07:20:20.446 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0242e-05 2024-09-17T07:20:20.446 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.447 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.447 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:20:20.447 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:15.361561+0000", 2024-09-17T07:20:20.447 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8410000000000002e-06 2024-09-17T07:20:20.447 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.447 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.447 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:20:20.447 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:15.361595+0000", 2024-09-17T07:20:20.447 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.4411e-05 2024-09-17T07:20:20.447 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.447 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.447 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-17T07:20:20.448 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:15.361600+0000", 2024-09-17T07:20:20.448 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6450000000000004e-06 2024-09-17T07:20:20.448 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.448 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.448 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:20:20.448 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:16.304235+0000", 2024-09-17T07:20:20.448 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.94263576999999998 2024-09-17T07:20:20.448 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.448 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.448 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:20:20.448 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:16.304328+0000", 2024-09-17T07:20:20.448 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.2173e-05 2024-09-17T07:20:20.448 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.449 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.449 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 2,5", 2024-09-17T07:20:20.449 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:16.304439+0000", 2024-09-17T07:20:20.449 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000111067 2024-09-17T07:20:20.449 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.449 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.449 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:20:20.449 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:16.306105+0000", 2024-09-17T07:20:20.449 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0016659820000000001 2024-09-17T07:20:20.449 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.449 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.449 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:20:20.449 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:16.306247+0000", 2024-09-17T07:20:20.450 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014216500000000001 2024-09-17T07:20:20.450 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.450 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.450 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:20:20.450 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:16.306409+0000", 2024-09-17T07:20:20.450 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000162423 2024-09-17T07:20:20.450 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.450 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.450 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:20:20.450 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:16.306431+0000", 2024-09-17T07:20:20.450 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2073999999999999e-05 2024-09-17T07:20:20.450 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.450 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.451 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:20:20.451 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:16.306449+0000", 2024-09-17T07:20:20.451 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.8138000000000002e-05 2024-09-17T07:20:20.451 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:20:20.451 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:20:20.451 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:20:20.451 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.451 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.451 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.15648.0:217579 10.1e 10:79a6ece8:::benchmark_data_smithi100_155039_object94231:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-09-17T07:20:20.451 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:19:15.418769+0000", 2024-09-17T07:20:20.451 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 64.907235553000007, 2024-09-17T07:20:20.451 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.88776972799999998, 2024-09-17T07:20:20.451 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:20:20.452 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:20:20.452 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:20:20.452 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.15648", 2024-09-17T07:20:20.452 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/1684940078", 2024-09-17T07:20:20.452 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 217579 2024-09-17T07:20:20.452 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.452 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:20:20.452 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.452 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:20:20.452 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:15.418769+0000", 2024-09-17T07:20:20.452 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:20:20.452 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.452 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.453 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:20:20.453 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:15.418769+0000", 2024-09-17T07:20:20.453 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:20:20.453 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.453 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.453 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:20:20.453 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:15.418768+0000", 2024-09-17T07:20:20.453 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-17T07:20:20.453 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.453 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.453 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:20:20.453 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:15.418773+0000", 2024-09-17T07:20:20.453 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0590000000000002e-06 2024-09-17T07:20:20.454 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.454 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.454 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:20:20.454 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:15.418783+0000", 2024-09-17T07:20:20.454 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7180000000000008e-06 2024-09-17T07:20:20.454 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.454 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.454 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:20:20.454 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:15.418788+0000", 2024-09-17T07:20:20.454 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1390000000000003e-06 2024-09-17T07:20:20.454 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.454 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.454 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:20:20.454 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:15.418831+0000", 2024-09-17T07:20:20.455 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3439999999999997e-05 2024-09-17T07:20:20.455 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.455 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.455 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-17T07:20:20.455 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:15.418842+0000", 2024-09-17T07:20:20.455 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0494e-05 2024-09-17T07:20:20.455 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.455 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.455 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:20:20.455 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:16.305035+0000", 2024-09-17T07:20:20.455 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.88619358000000004 2024-09-17T07:20:20.455 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.455 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.456 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:20:20.456 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:16.305143+0000", 2024-09-17T07:20:20.456 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00010755399999999999 2024-09-17T07:20:20.456 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.456 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.456 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 2,5", 2024-09-17T07:20:20.456 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:16.305289+0000", 2024-09-17T07:20:20.456 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014596499999999999 2024-09-17T07:20:20.456 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.456 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.456 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:20:20.456 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:16.306375+0000", 2024-09-17T07:20:20.456 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001086046 2024-09-17T07:20:20.457 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.457 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.457 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:20:20.457 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:16.306383+0000", 2024-09-17T07:20:20.457 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0469999999999994e-06 2024-09-17T07:20:20.457 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.457 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.457 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:20:20.457 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:16.306480+0000", 2024-09-17T07:20:20.457 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7485000000000006e-05 2024-09-17T07:20:20.457 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.457 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.457 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:20:20.458 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:16.306510+0000", 2024-09-17T07:20:20.458 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9989e-05 2024-09-17T07:20:20.458 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.458 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.458 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:20:20.458 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:19:16.306539+0000", 2024-09-17T07:20:20.458 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8101000000000001e-05 2024-09-17T07:20:20.458 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:20:20.458 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:20:20.458 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:20:20.458 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.458 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.458 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.25811.0:29891 11.b 11:d96965ea:::benchmark_data_smithi100_180298_object29890:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e397)", 2024-09-17T07:20:20.459 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:20:18.180601+0000", 2024-09-17T07:20:20.459 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 2.1454038099999999, 2024-09-17T07:20:20.459 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.088114358000000004, 2024-09-17T07:20:20.459 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:20:20.459 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:20:20.459 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-17T07:20:20.459 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.25811", 2024-09-17T07:20:20.459 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.100:0/173867468", 2024-09-17T07:20:20.459 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 29891 2024-09-17T07:20:20.459 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.459 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:20:20.459 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.459 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:20:20.460 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:20:18.180601+0000", 2024-09-17T07:20:20.460 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:20:20.460 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.460 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.460 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:20:20.460 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:20:18.180601+0000", 2024-09-17T07:20:20.460 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:20:20.460 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.460 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.460 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:20:20.460 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:20:18.180599+0000", 2024-09-17T07:20:20.460 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999986 2024-09-17T07:20:20.461 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.461 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.461 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:20:20.461 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:20:18.180621+0000", 2024-09-17T07:20:20.461 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1806000000000001e-05 2024-09-17T07:20:20.461 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.461 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.461 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:20:20.461 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:20:18.180631+0000", 2024-09-17T07:20:20.461 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.4229999999999994e-06 2024-09-17T07:20:20.461 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.461 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.461 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:20:20.462 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:20:18.180637+0000", 2024-09-17T07:20:20.462 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2990000000000002e-06 2024-09-17T07:20:20.462 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.462 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.462 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:20:20.462 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:20:18.180677+0000", 2024-09-17T07:20:20.462 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9854000000000002e-05 2024-09-17T07:20:20.462 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.462 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.462 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:20:20.462 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:20:18.180806+0000", 2024-09-17T07:20:20.462 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000129319 2024-09-17T07:20:20.462 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.463 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.463 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-17T07:20:20.463 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:20:18.180920+0000", 2024-09-17T07:20:20.463 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000114091 2024-09-17T07:20:20.463 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.463 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.463 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:20:20.463 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:20:18.267108+0000", 2024-09-17T07:20:20.463 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.086187532999999997 2024-09-17T07:20:20.463 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.463 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.463 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-17T07:20:20.463 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:20:18.268601+0000", 2024-09-17T07:20:20.464 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0014931949999999999 2024-09-17T07:20:20.464 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.464 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.464 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:20:20.464 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:20:18.268640+0000", 2024-09-17T07:20:20.464 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8977000000000003e-05 2024-09-17T07:20:20.464 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.464 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.464 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:20:20.464 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:20:18.268715+0000", 2024-09-17T07:20:20.464 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5143999999999997e-05 2024-09-17T07:20:20.464 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:20:20.464 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:20:20.465 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:20:20.465 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.465 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.465 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.25811.0:29885 11.0 e397/390 11:0c51e10f:::benchmark_data_smithi100_180298_object29884:head v 397'1912, mlcod=397'1909)", 2024-09-17T07:20:20.465 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:20:18.181315+0000", 2024-09-17T07:20:20.465 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 2.1446894790000002, 2024-09-17T07:20:20.465 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.087874889999999997, 2024-09-17T07:20:20.465 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:20:20.465 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:20:20.465 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:20:20.465 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.465 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:20:20.465 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:20:18.181315+0000", 2024-09-17T07:20:20.466 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:20:20.466 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.466 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.466 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:20:20.466 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:20:18.181315+0000", 2024-09-17T07:20:20.466 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:20:20.466 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.466 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.466 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:20:20.466 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:20:18.181314+0000", 2024-09-17T07:20:20.466 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-17T07:20:20.466 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.466 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.467 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:20:20.467 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:20:18.181440+0000", 2024-09-17T07:20:20.467 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012630000000000001 2024-09-17T07:20:20.467 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.467 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.467 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:20:20.467 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:20:18.265827+0000", 2024-09-17T07:20:20.467 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.084386279999999994 2024-09-17T07:20:20.467 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.467 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.467 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:20:20.467 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:20:18.265867+0000", 2024-09-17T07:20:20.467 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0561e-05 2024-09-17T07:20:20.467 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.468 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.468 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:20:20.468 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:20:18.266105+0000", 2024-09-17T07:20:20.468 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00023782600000000001 2024-09-17T07:20:20.468 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.468 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.468 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:20:20.468 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:20:18.266133+0000", 2024-09-17T07:20:20.468 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8149000000000001e-05 2024-09-17T07:20:20.468 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.468 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.468 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-17T07:20:20.468 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:20:18.269144+0000", 2024-09-17T07:20:20.469 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0030110190000000002 2024-09-17T07:20:20.469 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:20:20.469 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:20:20.469 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-17T07:20:20.469 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:20:18.269190+0000", 2024-09-17T07:20:20.469 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5451000000000002e-05 2024-09-17T07:20:20.469 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:20:20.469 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:20:20.469 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:20:20.469 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:20:20.469 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:20:20.469 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-17T07:20:20.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:20.471 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:20.471 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:20:20.572 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:20:20.606 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:20.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:20.607 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:20:20.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:20 smithi100 ceph-mon[29092]: 4.5 scrub starts 2024-09-17T07:20:20.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:20 smithi100 ceph-mon[29092]: 4.5 scrub ok 2024-09-17T07:20:20.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:20 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 20290/84120 objects degraded (24.120%), 29 pgs degraded, 41 pgs undersized (PG_DEGRADED) 2024-09-17T07:20:20.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:20 smithi100 ceph-mon[29092]: osdmap e398: 8 total, 7 up, 6 in 2024-09-17T07:20:20.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:20 smithi100 ceph-mon[29092]: 6.1c scrub starts 2024-09-17T07:20:20.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:20 smithi100 ceph-mon[29092]: 6.1c scrub ok 2024-09-17T07:20:20.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:20 smithi100 ceph-mon[29092]: 6.6 scrub starts 2024-09-17T07:20:20.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:20 smithi100 ceph-mon[29092]: 6.6 scrub ok 2024-09-17T07:20:20.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:20 smithi100 ceph-mon[39462]: 4.5 scrub starts 2024-09-17T07:20:20.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:20 smithi100 ceph-mon[39462]: 4.5 scrub ok 2024-09-17T07:20:20.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:20 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 20290/84120 objects degraded (24.120%), 29 pgs degraded, 41 pgs undersized (PG_DEGRADED) 2024-09-17T07:20:20.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:20 smithi100 ceph-mon[39462]: osdmap e398: 8 total, 7 up, 6 in 2024-09-17T07:20:20.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:20 smithi100 ceph-mon[39462]: 6.1c scrub starts 2024-09-17T07:20:20.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:20 smithi100 ceph-mon[39462]: 6.1c scrub ok 2024-09-17T07:20:20.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:20 smithi100 ceph-mon[39462]: 6.6 scrub starts 2024-09-17T07:20:20.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:20 smithi100 ceph-mon[39462]: 6.6 scrub ok 2024-09-17T07:20:20.708 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:20:20.745 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:20.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:20.746 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:20:20.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:20 smithi135 ceph-mon[33238]: 4.5 scrub starts 2024-09-17T07:20:20.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:20 smithi135 ceph-mon[33238]: 4.5 scrub ok 2024-09-17T07:20:20.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:20 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 20290/84120 objects degraded (24.120%), 29 pgs degraded, 41 pgs undersized (PG_DEGRADED) 2024-09-17T07:20:20.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:20 smithi135 ceph-mon[33238]: osdmap e398: 8 total, 7 up, 6 in 2024-09-17T07:20:20.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:20 smithi135 ceph-mon[33238]: 6.1c scrub starts 2024-09-17T07:20:20.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:20 smithi135 ceph-mon[33238]: 6.1c scrub ok 2024-09-17T07:20:20.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:20 smithi135 ceph-mon[33238]: 6.6 scrub starts 2024-09-17T07:20:20.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:20 smithi135 ceph-mon[33238]: 6.6 scrub ok 2024-09-17T07:20:20.847 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:20:20.874 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:20.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:20.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:20:20.976 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:20:21.010 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:21.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:21.011 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:20:21.112 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:20:21.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:21.148 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:21.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:20:21.152 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 11 16 36821 36805 209.096 140.062 0.00510604 0.00477844 2024-09-17T07:20:21.249 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:20:21.280 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:21.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:21.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:20:21.382 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:20:21.416 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:21.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:21.417 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:20:21.518 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:20:21.554 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:21.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:21.555 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:20:21.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:21 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/4209546129' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:20:21.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:21 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:20:21.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:21 smithi100 ceph-mon[29092]: 5.1c scrub starts 2024-09-17T07:20:21.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:21 smithi100 ceph-mon[29092]: 5.1c scrub ok 2024-09-17T07:20:21.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:21 smithi100 ceph-mon[29092]: pgmap v850: 158 pgs: 31 active+clean+remapped, 24 active+undersized, 29 active+undersized+degraded, 74 active+clean; 1.9 GiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 1.2 KiB/s rd, 228 MiB/s wr, 3.64k op/s; 24076/95478 objects degraded (25.216%); 48/95478 objects misplaced (0.050%) 2024-09-17T07:20:21.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:21 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/4209546129' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:20:21.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:21 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:20:21.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:21 smithi100 ceph-mon[39462]: 5.1c scrub starts 2024-09-17T07:20:21.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:21 smithi100 ceph-mon[39462]: 5.1c scrub ok 2024-09-17T07:20:21.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:21 smithi100 ceph-mon[39462]: pgmap v850: 158 pgs: 31 active+clean+remapped, 24 active+undersized, 29 active+undersized+degraded, 74 active+clean; 1.9 GiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 1.2 KiB/s rd, 228 MiB/s wr, 3.64k op/s; 24076/95478 objects degraded (25.216%); 48/95478 objects misplaced (0.050%) 2024-09-17T07:20:21.656 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:20:21.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:21.692 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:21.692 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:20:21.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:21 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/4209546129' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:20:21.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:21 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:20:21.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:21 smithi135 ceph-mon[33238]: 5.1c scrub starts 2024-09-17T07:20:21.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:21 smithi135 ceph-mon[33238]: 5.1c scrub ok 2024-09-17T07:20:21.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:21 smithi135 ceph-mon[33238]: pgmap v850: 158 pgs: 31 active+clean+remapped, 24 active+undersized, 29 active+undersized+degraded, 74 active+clean; 1.9 GiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 1.2 KiB/s rd, 228 MiB/s wr, 3.64k op/s; 24076/95478 objects degraded (25.216%); 48/95478 objects misplaced (0.050%) 2024-09-17T07:20:21.794 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:20:21.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:21.828 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:21.828 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:20:21.929 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:20:21.962 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:21.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:21.963 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:20:22.064 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:20:22.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:22.096 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:22.096 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:20:22.152 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 12 15 39246 39231 204.306 151.625 0.00596714 0.0048904 2024-09-17T07:20:22.198 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:20:22.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:22.236 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:22.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:20:22.337 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:20:22.357 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:22.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:22.358 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:20:22.459 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:20:22.495 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:22.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:22.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:20:22.503 INFO:teuthology.orchestra.run.smithi100.stdout:moved 0 / 474 (0%) 2024-09-17T07:20:22.503 INFO:teuthology.orchestra.run.smithi100.stdout:avg 94.8 2024-09-17T07:20:22.503 INFO:teuthology.orchestra.run.smithi100.stdout:stddev 17.1184 -> 17.1184 (expected baseline 8.70862) 2024-09-17T07:20:22.503 INFO:teuthology.orchestra.run.smithi100.stdout:min osd.4 with 74 -> 74 pgs (0.780591 -> 0.780591 * mean) 2024-09-17T07:20:22.503 INFO:teuthology.orchestra.run.smithi100.stdout:max osd.6 with 83 -> 83 pgs (0.875527 -> 0.875527 * mean) 2024-09-17T07:20:22.504 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T07:20:22.504 INFO:teuthology.orchestra.run.smithi100.stdout:oload 1000 2024-09-17T07:20:22.504 INFO:teuthology.orchestra.run.smithi100.stdout:max_change 1 2024-09-17T07:20:22.504 INFO:teuthology.orchestra.run.smithi100.stdout:max_change_osds 4 2024-09-17T07:20:22.504 INFO:teuthology.orchestra.run.smithi100.stdout:average_utilization 0.0163 2024-09-17T07:20:22.504 INFO:teuthology.orchestra.run.smithi100.stdout:overload_utilization 0.1628 2024-09-17T07:20:22.504 INFO:teuthology.orchestra.run.smithi100.stderr:no change 2024-09-17T07:20:22.598 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:20:22.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:22 smithi100 ceph-mon[29092]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T07:20:22.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:22 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T07:20:22.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:22 smithi100 ceph-mon[29092]: osdmap e399: 8 total, 7 up, 6 in 2024-09-17T07:20:22.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:22 smithi100 ceph-mon[29092]: 1.8 scrub starts 2024-09-17T07:20:22.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:22 smithi100 ceph-mon[29092]: 1.8 scrub ok 2024-09-17T07:20:22.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:22 smithi100 ceph-mon[39462]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T07:20:22.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:22 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T07:20:22.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:22 smithi100 ceph-mon[39462]: osdmap e399: 8 total, 7 up, 6 in 2024-09-17T07:20:22.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:22 smithi100 ceph-mon[39462]: 1.8 scrub starts 2024-09-17T07:20:22.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:22 smithi100 ceph-mon[39462]: 1.8 scrub ok 2024-09-17T07:20:22.632 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:22.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:22.636 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:20:22.737 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:20:22.769 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:22.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:22.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:20:22.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:22 smithi135 ceph-mon[33238]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T07:20:22.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:22 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T07:20:22.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:22 smithi135 ceph-mon[33238]: osdmap e399: 8 total, 7 up, 6 in 2024-09-17T07:20:22.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:22 smithi135 ceph-mon[33238]: 1.8 scrub starts 2024-09-17T07:20:22.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:22 smithi135 ceph-mon[33238]: 1.8 scrub ok 2024-09-17T07:20:22.871 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:20:22.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:22.908 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:22.908 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:20:23.009 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:20:23.030 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:23.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:23.031 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:20:23.131 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:20:23.152 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 13 16 41072 41056 197.364 114.062 0.00408101 0.00506228 2024-09-17T07:20:23.163 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:23.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:23.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:20:23.265 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:20:23.296 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:23.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:23.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:20:23.398 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:20:23.483 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:23.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:23.484 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:20:23.585 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:20:23.619 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:23.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:23.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:20:23.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:23 smithi100 ceph-mon[29092]: from='client.25829 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:20:23.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:23 smithi100 ceph-mon[29092]: 1.1 scrub starts 2024-09-17T07:20:23.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:23 smithi100 ceph-mon[29092]: 1.1 scrub ok 2024-09-17T07:20:23.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:23 smithi100 ceph-mon[29092]: pgmap v852: 158 pgs: 31 active+clean+remapped, 24 active+undersized, 29 active+undersized+degraded, 74 active+clean; 2.1 GiB data, 9.3 GiB used, 527 GiB / 536 GiB avail; 639 B/s rd, 140 MiB/s wr, 2.23k op/s; 25487/101775 objects degraded (25.042%); 48/101775 objects misplaced (0.047%) 2024-09-17T07:20:23.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:23 smithi100 ceph-mon[39462]: from='client.25829 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:20:23.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:23 smithi100 ceph-mon[39462]: 1.1 scrub starts 2024-09-17T07:20:23.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:23 smithi100 ceph-mon[39462]: 1.1 scrub ok 2024-09-17T07:20:23.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:23 smithi100 ceph-mon[39462]: pgmap v852: 158 pgs: 31 active+clean+remapped, 24 active+undersized, 29 active+undersized+degraded, 74 active+clean; 2.1 GiB data, 9.3 GiB used, 527 GiB / 536 GiB avail; 639 B/s rd, 140 MiB/s wr, 2.23k op/s; 25487/101775 objects degraded (25.042%); 48/101775 objects misplaced (0.047%) 2024-09-17T07:20:23.722 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:20:23.751 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:23.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:23.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:20:23.770 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-17T07:20:23.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:23 smithi135 ceph-mon[33238]: from='client.25829 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:20:23.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:23 smithi135 ceph-mon[33238]: 1.1 scrub starts 2024-09-17T07:20:23.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:23 smithi135 ceph-mon[33238]: 1.1 scrub ok 2024-09-17T07:20:23.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:23 smithi135 ceph-mon[33238]: pgmap v852: 158 pgs: 31 active+clean+remapped, 24 active+undersized, 29 active+undersized+degraded, 74 active+clean; 2.1 GiB data, 9.3 GiB used, 527 GiB / 536 GiB avail; 639 B/s rd, 140 MiB/s wr, 2.23k op/s; 25487/101775 objects degraded (25.042%); 48/101775 objects misplaced (0.047%) 2024-09-17T07:20:23.854 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:20:23.886 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:23.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:23.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:20:23.988 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:20:24.022 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:24.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:24.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:20:24.061 INFO:teuthology.orchestra.run.smithi100.stderr:nodeep-scrub is unset 2024-09-17T07:20:24.124 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:20:24.152 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 14 16 43397 43381 193.645 145.312 0.0043091 0.0051591 2024-09-17T07:20:24.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:24.160 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:24.160 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:20:24.261 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:20:24.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:24.295 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:24.295 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:20:24.314 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:20:24.396 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:20:24.437 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:24.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:24.438 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:20:24.538 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:20:24.574 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:24.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:24.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:20:24.676 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:20:24.708 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:24.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:24.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:20:24.810 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:20:24.840 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:24.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:24.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:20:24.942 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:20:24.976 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:24.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:24.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:20:25.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:24 smithi135 ceph-mon[33238]: 2.4 scrub starts 2024-09-17T07:20:25.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:24 smithi135 ceph-mon[33238]: 2.4 scrub ok 2024-09-17T07:20:25.044 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:24 smithi100 ceph-mon[29092]: 2.4 scrub starts 2024-09-17T07:20:25.045 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:24 smithi100 ceph-mon[29092]: 2.4 scrub ok 2024-09-17T07:20:25.045 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:24 smithi100 ceph-mon[39462]: 2.4 scrub starts 2024-09-17T07:20:25.045 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:24 smithi100 ceph-mon[39462]: 2.4 scrub ok 2024-09-17T07:20:25.078 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:20:25.082 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-09-17T07:20:25.082 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd reweight-by-utilization 1000 0.05 2024-09-17T07:20:25.112 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:25.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:25.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:20:25.152 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 15 16 45840 45824 190.913 152.688 0.00318022 0.00523286 2024-09-17T07:20:25.214 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:20:25.255 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:25.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:25.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:20:25.357 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:20:25.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:25.400 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:25.400 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:20:25.501 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:20:25.535 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:25.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:25.536 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:20:25.637 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:20:25.671 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:25.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:25.672 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:20:25.773 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:20:25.805 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:25.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:25.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:20:25.907 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:20:25.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:25.942 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:25.942 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:20:26.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:25 smithi135 ceph-mon[33238]: 6.12 scrub starts 2024-09-17T07:20:26.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:25 smithi135 ceph-mon[33238]: 6.12 scrub ok 2024-09-17T07:20:26.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:25 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 25487/101775 objects degraded (25.042%), 29 pgs degraded, 41 pgs undersized (PG_DEGRADED) 2024-09-17T07:20:26.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:25 smithi135 ceph-mon[33238]: osdmap e400: 8 total, 7 up, 6 in 2024-09-17T07:20:26.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:25 smithi135 ceph-mon[33238]: pgmap v854: 158 pgs: 31 active+clean+remapped, 24 active+undersized, 29 active+undersized+degraded, 74 active+clean; 2.6 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 157 MiB/s wr, 2.51k op/s; 32562/129429 objects degraded (25.158%); 48/129429 objects misplaced (0.037%) 2024-09-17T07:20:26.044 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:20:26.082 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:26.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:26.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:20:26.127 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:20:25 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:20:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:20:26.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:25 smithi100 ceph-mon[39462]: 6.12 scrub starts 2024-09-17T07:20:26.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:25 smithi100 ceph-mon[39462]: 6.12 scrub ok 2024-09-17T07:20:26.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:25 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 25487/101775 objects degraded (25.042%), 29 pgs degraded, 41 pgs undersized (PG_DEGRADED) 2024-09-17T07:20:26.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:25 smithi100 ceph-mon[39462]: osdmap e400: 8 total, 7 up, 6 in 2024-09-17T07:20:26.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:25 smithi100 ceph-mon[39462]: pgmap v854: 158 pgs: 31 active+clean+remapped, 24 active+undersized, 29 active+undersized+degraded, 74 active+clean; 2.6 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 157 MiB/s wr, 2.51k op/s; 32562/129429 objects degraded (25.158%); 48/129429 objects misplaced (0.037%) 2024-09-17T07:20:26.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:25 smithi100 ceph-mon[29092]: 6.12 scrub starts 2024-09-17T07:20:26.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:25 smithi100 ceph-mon[29092]: 6.12 scrub ok 2024-09-17T07:20:26.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:25 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 25487/101775 objects degraded (25.042%), 29 pgs degraded, 41 pgs undersized (PG_DEGRADED) 2024-09-17T07:20:26.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:25 smithi100 ceph-mon[29092]: osdmap e400: 8 total, 7 up, 6 in 2024-09-17T07:20:26.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:25 smithi100 ceph-mon[29092]: pgmap v854: 158 pgs: 31 active+clean+remapped, 24 active+undersized, 29 active+undersized+degraded, 74 active+clean; 2.6 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 157 MiB/s wr, 2.51k op/s; 32562/129429 objects degraded (25.158%); 48/129429 objects misplaced (0.037%) 2024-09-17T07:20:26.152 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 16 16 48008 47992 187.449 135.5 0.00476199 0.00532977 2024-09-17T07:20:26.184 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:20:26.219 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:26.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:26.220 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:20:26.321 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:20:26.351 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:26.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:26.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:20:26.453 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:20:26.474 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:26.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:26.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:20:26.576 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:20:26.614 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:26.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:26.615 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:20:26.716 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:20:26.745 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:26.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:26.746 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:20:26.847 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:20:26.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:26.884 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:26.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:20:26.985 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:20:27.015 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:27.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:27.017 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:20:27.117 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:20:27.152 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 17 16 49939 49923 183.522 120.688 0.00756964 0.00544435 2024-09-17T07:20:27.154 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:27.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:27.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:20:27.256 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:20:27.289 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:27.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:27.290 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:20:27.391 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:20:27.412 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:27.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:27.413 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:20:27.514 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:20:27.547 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:27.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:27.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:20:27.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:27 smithi100 ceph-mon[29092]: 6.1e scrub starts 2024-09-17T07:20:27.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:27 smithi100 ceph-mon[29092]: 6.1e scrub ok 2024-09-17T07:20:27.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:27 smithi100 ceph-mon[39462]: 6.1e scrub starts 2024-09-17T07:20:27.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:27 smithi100 ceph-mon[39462]: 6.1e scrub ok 2024-09-17T07:20:27.649 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:20:27.682 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:27.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:27.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:20:27.785 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:20:27.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:27 smithi135 ceph-mon[33238]: 6.1e scrub starts 2024-09-17T07:20:27.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:27 smithi135 ceph-mon[33238]: 6.1e scrub ok 2024-09-17T07:20:27.825 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:27.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:27.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:20:27.927 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:20:27.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:27.959 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:27.959 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:20:28.060 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:20:28.093 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:28.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:28.095 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:20:28.153 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 18 16 51972 51956 180.384 127.062 0.0065554 0.00553956 2024-09-17T07:20:28.195 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:20:28.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:28.229 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:28.229 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:20:28.330 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:20:28.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:28.363 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:28.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:20:28.464 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:20:28.495 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:28.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:28.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:20:28.597 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:20:28.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:28 smithi100 ceph-mon[29092]: pgmap v855: 158 pgs: 31 active+clean+remapped, 24 active+undersized, 29 active+undersized+degraded, 74 active+clean; 2.6 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 119 MiB/s wr, 1.90k op/s; 32562/129429 objects degraded (25.158%); 48/129429 objects misplaced (0.037%) 2024-09-17T07:20:28.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:28 smithi100 ceph-mon[29092]: 6.18 scrub starts 2024-09-17T07:20:28.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:28 smithi100 ceph-mon[29092]: 6.18 scrub ok 2024-09-17T07:20:28.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:28 smithi100 ceph-mon[39462]: pgmap v855: 158 pgs: 31 active+clean+remapped, 24 active+undersized, 29 active+undersized+degraded, 74 active+clean; 2.6 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 119 MiB/s wr, 1.90k op/s; 32562/129429 objects degraded (25.158%); 48/129429 objects misplaced (0.037%) 2024-09-17T07:20:28.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:28 smithi100 ceph-mon[39462]: 6.18 scrub starts 2024-09-17T07:20:28.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:28 smithi100 ceph-mon[39462]: 6.18 scrub ok 2024-09-17T07:20:28.638 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:28.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:28.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:20:28.740 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:20:28.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:28.778 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:28.778 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:20:28.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:28 smithi135 ceph-mon[33238]: pgmap v855: 158 pgs: 31 active+clean+remapped, 24 active+undersized, 29 active+undersized+degraded, 74 active+clean; 2.6 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 119 MiB/s wr, 1.90k op/s; 32562/129429 objects degraded (25.158%); 48/129429 objects misplaced (0.037%) 2024-09-17T07:20:28.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:28 smithi135 ceph-mon[33238]: 6.18 scrub starts 2024-09-17T07:20:28.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:28 smithi135 ceph-mon[33238]: 6.18 scrub ok 2024-09-17T07:20:28.879 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:20:28.914 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:28.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:28.915 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:20:29.016 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:20:29.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:29.051 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:29.052 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:20:29.152 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:20:29.153 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 19 16 54159 54143 178.084 136.688 0.00551933 0.00561107 2024-09-17T07:20:29.187 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:29.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:29.188 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:20:29.289 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:20:29.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:29.322 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:29.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:20:29.423 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:20:29.464 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:29.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:29.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:20:29.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:29 smithi135 ceph-mon[33238]: 4.1 scrub starts 2024-09-17T07:20:29.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:29 smithi135 ceph-mon[33238]: 4.1 scrub ok 2024-09-17T07:20:29.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:29 smithi135 ceph-mon[33238]: 4.11 scrub starts 2024-09-17T07:20:29.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:29 smithi135 ceph-mon[33238]: 4.11 scrub ok 2024-09-17T07:20:29.544 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:20:29 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:20:29.566 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:20:29.589 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:29.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:29.590 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:20:29.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:29 smithi100 ceph-mon[29092]: 4.1 scrub starts 2024-09-17T07:20:29.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:29 smithi100 ceph-mon[29092]: 4.1 scrub ok 2024-09-17T07:20:29.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:29 smithi100 ceph-mon[29092]: 4.11 scrub starts 2024-09-17T07:20:29.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:29 smithi100 ceph-mon[29092]: 4.11 scrub ok 2024-09-17T07:20:29.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:29 smithi100 ceph-mon[39462]: 4.1 scrub starts 2024-09-17T07:20:29.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:29 smithi100 ceph-mon[39462]: 4.1 scrub ok 2024-09-17T07:20:29.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:29 smithi100 ceph-mon[39462]: 4.11 scrub starts 2024-09-17T07:20:29.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:29 smithi100 ceph-mon[39462]: 4.11 scrub ok 2024-09-17T07:20:29.690 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:20:29.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:29.726 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:29.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:20:29.827 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:20:29.849 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:29.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:29.850 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:20:29.951 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:20:29.986 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:29.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:29.987 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:20:30.088 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:20:30.122 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:30.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:30.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:20:30.152 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:2024-09-17T07:20:30.153280+0000 min lat: 0.00105244 max lat: 0.145857 avg lat: 0.00569525 2024-09-17T07:20:30.153 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-17T07:20:30.153 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 20 16 56171 56155 175.467 125.75 0.0160578 0.00569525 2024-09-17T07:20:30.224 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:20:30.259 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:30.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:30.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:20:30.362 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:20:30.397 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:30.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:30.398 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:20:30.499 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:20:30.534 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:30.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:30.535 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:20:30.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:30 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:20:30.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:30 smithi100 ceph-mon[29092]: pgmap v856: 158 pgs: 31 active+clean+remapped, 24 active+undersized, 29 active+undersized+degraded, 74 active+clean; 3.1 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 152 MiB/s wr, 2.43k op/s; 38050/153966 objects degraded (24.713%); 48/153966 objects misplaced (0.031%) 2024-09-17T07:20:30.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:30 smithi100 ceph-mon[29092]: 5.17 scrub starts 2024-09-17T07:20:30.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:30 smithi100 ceph-mon[29092]: 5.17 scrub ok 2024-09-17T07:20:30.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:30 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 32562/129429 objects degraded (25.158%), 29 pgs degraded, 41 pgs undersized (PG_DEGRADED) 2024-09-17T07:20:30.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:30 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:20:30.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:30 smithi100 ceph-mon[39462]: pgmap v856: 158 pgs: 31 active+clean+remapped, 24 active+undersized, 29 active+undersized+degraded, 74 active+clean; 3.1 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 152 MiB/s wr, 2.43k op/s; 38050/153966 objects degraded (24.713%); 48/153966 objects misplaced (0.031%) 2024-09-17T07:20:30.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:30 smithi100 ceph-mon[39462]: 5.17 scrub starts 2024-09-17T07:20:30.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:30 smithi100 ceph-mon[39462]: 5.17 scrub ok 2024-09-17T07:20:30.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:30 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 32562/129429 objects degraded (25.158%), 29 pgs degraded, 41 pgs undersized (PG_DEGRADED) 2024-09-17T07:20:30.636 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:20:30.666 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:30.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:30.667 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:20:30.714 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:20:30.717 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:20:30.769 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:20:30.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:30 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:20:30.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:30 smithi135 ceph-mon[33238]: pgmap v856: 158 pgs: 31 active+clean+remapped, 24 active+undersized, 29 active+undersized+degraded, 74 active+clean; 3.1 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 152 MiB/s wr, 2.43k op/s; 38050/153966 objects degraded (24.713%); 48/153966 objects misplaced (0.031%) 2024-09-17T07:20:30.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:30 smithi135 ceph-mon[33238]: 5.17 scrub starts 2024-09-17T07:20:30.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:30 smithi135 ceph-mon[33238]: 5.17 scrub ok 2024-09-17T07:20:30.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:30 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 32562/129429 objects degraded (25.158%), 29 pgs degraded, 41 pgs undersized (PG_DEGRADED) 2024-09-17T07:20:30.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:30.803 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:30.803 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:20:30.904 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:20:30.938 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:30.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:30.939 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:20:31.040 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:20:31.081 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:31.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:31.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:20:31.153 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 21 16 58042 58026 172.679 116.938 0.00287501 0.00577788 2024-09-17T07:20:31.183 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:20:31.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:31.221 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:31.222 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:20:31.322 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:20:31.354 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:31.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:31.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:20:31.456 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:20:31.490 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:31.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:31.491 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:20:31.583 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:31 smithi100 ceph-mon[29092]: 5.10 scrub starts 2024-09-17T07:20:31.583 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:31 smithi100 ceph-mon[29092]: 5.10 scrub ok 2024-09-17T07:20:31.583 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:31 smithi100 ceph-mon[39462]: 5.10 scrub starts 2024-09-17T07:20:31.583 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:31 smithi100 ceph-mon[39462]: 5.10 scrub ok 2024-09-17T07:20:31.591 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:20:31.621 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:31.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:31.622 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:20:31.723 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:20:31.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:31.767 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:31.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:20:31.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:31 smithi135 ceph-mon[33238]: 5.10 scrub starts 2024-09-17T07:20:31.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:31 smithi135 ceph-mon[33238]: 5.10 scrub ok 2024-09-17T07:20:31.868 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:20:31.896 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:31.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:31.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:20:32.000 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:20:32.033 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:32.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:32.034 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:20:32.134 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:20:32.153 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 22 16 60036 60020 170.494 124.625 0.0143613 0.00585924 2024-09-17T07:20:32.162 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:32.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:32.163 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:20:32.264 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:20:32.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:32.302 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:32.302 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:20:32.366 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-17T07:20:32.402 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:20:32.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:32.441 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:32.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:20:32.542 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:20:32.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:32.575 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:32.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:20:32.607 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:32 smithi100 ceph-mon[29092]: pgmap v857: 158 pgs: 31 active+clean+remapped, 24 active+undersized, 29 active+undersized+degraded, 74 active+clean; 3.3 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 139 MiB/s wr, 2.22k op/s; 40642/161742 objects degraded (25.128%); 48/161742 objects misplaced (0.030%) 2024-09-17T07:20:32.608 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:32 smithi100 ceph-mon[39462]: pgmap v857: 158 pgs: 31 active+clean+remapped, 24 active+undersized, 29 active+undersized+degraded, 74 active+clean; 3.3 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 139 MiB/s wr, 2.22k op/s; 40642/161742 objects degraded (25.128%); 48/161742 objects misplaced (0.030%) 2024-09-17T07:20:32.675 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:20:32.710 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:32.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:32.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:20:32.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:32 smithi135 ceph-mon[33238]: pgmap v857: 158 pgs: 31 active+clean+remapped, 24 active+undersized, 29 active+undersized+degraded, 74 active+clean; 3.3 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 139 MiB/s wr, 2.22k op/s; 40642/161742 objects degraded (25.128%); 48/161742 objects misplaced (0.030%) 2024-09-17T07:20:32.813 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:20:32.843 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:32.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:32.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:20:32.945 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:20:32.982 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:32.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:32.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:20:33.084 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:20:33.115 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:33.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:33.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:20:33.154 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 23 16 62031 62015 168.497 124.688 0.00318907 0.00592864 2024-09-17T07:20:33.217 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:20:33.253 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:33.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:33.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:20:33.355 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:20:33.391 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:33.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:33.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:20:33.493 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:20:33.530 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:33.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:33.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:20:33.632 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:20:33.640 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:20:33.640 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:20:33.641 INFO:teuthology.orchestra.run.smithi100.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-17T07:20:33.667 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:33.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:33.668 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:20:33.671 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:20:33.672 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:20:33.701 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:20:33.701 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:20:33.729 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:20:33.729 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:20:33.760 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:20:33.761 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:20:33.769 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:20:33.791 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:20:33.791 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:20:33.802 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:33.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:33.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:20:33.829 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:20:33.829 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:20:33.905 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:20:33.938 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:33.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:33.939 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:20:34.040 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:20:34.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:34.075 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:34.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:20:34.154 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 24 16 64033 64017 166.689 125.125 0.00503282 0.00599463 2024-09-17T07:20:34.177 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:20:34.211 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:34.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:34.212 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:20:34.313 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:20:34.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:34.350 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:34.351 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:20:34.452 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:20:34.486 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:34.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:34.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:20:34.588 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:20:34.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:34.622 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:34.622 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:20:34.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:34 smithi100 ceph-mon[29092]: 1.6 scrub starts 2024-09-17T07:20:34.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:34 smithi100 ceph-mon[29092]: 1.6 scrub ok 2024-09-17T07:20:34.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:34 smithi100 ceph-mon[29092]: 4.9 scrub starts 2024-09-17T07:20:34.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:34 smithi100 ceph-mon[29092]: 4.9 scrub ok 2024-09-17T07:20:34.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:34 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:20:34.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:34 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:20:34.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:34 smithi100 ceph-mon[29092]: pgmap v858: 158 pgs: 31 active+clean+remapped, 24 active+undersized, 29 active+undersized+degraded, 74 active+clean; 3.4 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 137 MiB/s wr, 2.18k op/s; 41927/167403 objects degraded (25.046%); 48/167403 objects misplaced (0.029%) 2024-09-17T07:20:34.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:34 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/34249475' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:20:34.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:34 smithi100 ceph-mon[39462]: 1.6 scrub starts 2024-09-17T07:20:34.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:34 smithi100 ceph-mon[39462]: 1.6 scrub ok 2024-09-17T07:20:34.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:34 smithi100 ceph-mon[39462]: 4.9 scrub starts 2024-09-17T07:20:34.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:34 smithi100 ceph-mon[39462]: 4.9 scrub ok 2024-09-17T07:20:34.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:34 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:20:34.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:34 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:20:34.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:34 smithi100 ceph-mon[39462]: pgmap v858: 158 pgs: 31 active+clean+remapped, 24 active+undersized, 29 active+undersized+degraded, 74 active+clean; 3.4 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 137 MiB/s wr, 2.18k op/s; 41927/167403 objects degraded (25.046%); 48/167403 objects misplaced (0.029%) 2024-09-17T07:20:34.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:34 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/34249475' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:20:34.722 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:20:34.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:34.756 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:34.756 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:20:34.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:34 smithi135 ceph-mon[33238]: 1.6 scrub starts 2024-09-17T07:20:34.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:34 smithi135 ceph-mon[33238]: 1.6 scrub ok 2024-09-17T07:20:34.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:34 smithi135 ceph-mon[33238]: 4.9 scrub starts 2024-09-17T07:20:34.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:34 smithi135 ceph-mon[33238]: 4.9 scrub ok 2024-09-17T07:20:34.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:34 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:20:34.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:34 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:20:34.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:34 smithi135 ceph-mon[33238]: pgmap v858: 158 pgs: 31 active+clean+remapped, 24 active+undersized, 29 active+undersized+degraded, 74 active+clean; 3.4 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 137 MiB/s wr, 2.18k op/s; 41927/167403 objects degraded (25.046%); 48/167403 objects misplaced (0.029%) 2024-09-17T07:20:34.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:34 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/34249475' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:20:34.838 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:20:34.838 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:20:34.857 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:20:34.891 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:34.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:34.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:20:34.993 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:20:35.027 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:35.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:35.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:20:35.129 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:20:35.154 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 25 16 66028 66012 165.009 124.688 0.0226216 0.00605462 2024-09-17T07:20:35.161 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:35.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:35.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:20:35.263 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:20:35.290 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:35.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:35.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:20:35.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:35 smithi100 ceph-mon[29092]: 1.a scrub starts 2024-09-17T07:20:35.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:35 smithi100 ceph-mon[29092]: 1.a scrub ok 2024-09-17T07:20:35.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:35 smithi100 ceph-mon[29092]: 1.3 deep-scrub starts 2024-09-17T07:20:35.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:35 smithi100 ceph-mon[29092]: 1.3 deep-scrub ok 2024-09-17T07:20:35.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:35 smithi100 ceph-mon[29092]: 2.0 scrub starts 2024-09-17T07:20:35.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:35 smithi100 ceph-mon[29092]: 2.0 scrub ok 2024-09-17T07:20:35.377 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:35 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 41927/167403 objects degraded (25.046%), 29 pgs degraded, 41 pgs undersized (PG_DEGRADED) 2024-09-17T07:20:35.377 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:35 smithi100 ceph-mon[39462]: 1.a scrub starts 2024-09-17T07:20:35.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:35 smithi100 ceph-mon[39462]: 1.a scrub ok 2024-09-17T07:20:35.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:35 smithi100 ceph-mon[39462]: 1.3 deep-scrub starts 2024-09-17T07:20:35.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:35 smithi100 ceph-mon[39462]: 1.3 deep-scrub ok 2024-09-17T07:20:35.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:35 smithi100 ceph-mon[39462]: 2.0 scrub starts 2024-09-17T07:20:35.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:35 smithi100 ceph-mon[39462]: 2.0 scrub ok 2024-09-17T07:20:35.378 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:35 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 41927/167403 objects degraded (25.046%), 29 pgs degraded, 41 pgs undersized (PG_DEGRADED) 2024-09-17T07:20:35.392 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:20:35.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:35.426 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:35.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:20:35.527 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:20:35.561 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:35.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:35.563 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:20:35.664 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:20:35.692 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:35.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:35.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:20:35.708 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:20:35.756 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-17T07:20:35.756 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:20:35.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:35 smithi135 ceph-mon[33238]: 1.a scrub starts 2024-09-17T07:20:35.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:35 smithi135 ceph-mon[33238]: 1.a scrub ok 2024-09-17T07:20:35.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:35 smithi135 ceph-mon[33238]: 1.3 deep-scrub starts 2024-09-17T07:20:35.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:35 smithi135 ceph-mon[33238]: 1.3 deep-scrub ok 2024-09-17T07:20:35.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:35 smithi135 ceph-mon[33238]: 2.0 scrub starts 2024-09-17T07:20:35.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:35 smithi135 ceph-mon[33238]: 2.0 scrub ok 2024-09-17T07:20:35.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:35 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 41927/167403 objects degraded (25.046%), 29 pgs degraded, 41 pgs undersized (PG_DEGRADED) 2024-09-17T07:20:35.794 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:20:35.808 INFO:teuthology.orchestra.run.smithi100.stdout:moved 0 / 474 (0%) 2024-09-17T07:20:35.808 INFO:teuthology.orchestra.run.smithi100.stdout:avg 94.8 2024-09-17T07:20:35.809 INFO:teuthology.orchestra.run.smithi100.stdout:stddev 17.1184 -> 17.1184 (expected baseline 8.70862) 2024-09-17T07:20:35.809 INFO:teuthology.orchestra.run.smithi100.stdout:min osd.4 with 74 -> 74 pgs (0.780591 -> 0.780591 * mean) 2024-09-17T07:20:35.809 INFO:teuthology.orchestra.run.smithi100.stdout:max osd.6 with 83 -> 83 pgs (0.875527 -> 0.875527 * mean) 2024-09-17T07:20:35.809 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T07:20:35.809 INFO:teuthology.orchestra.run.smithi100.stdout:oload 1000 2024-09-17T07:20:35.809 INFO:teuthology.orchestra.run.smithi100.stdout:max_change 0.05 2024-09-17T07:20:35.809 INFO:teuthology.orchestra.run.smithi100.stdout:max_change_osds 4 2024-09-17T07:20:35.809 INFO:teuthology.orchestra.run.smithi100.stdout:average_utilization 0.0248 2024-09-17T07:20:35.809 INFO:teuthology.orchestra.run.smithi100.stdout:overload_utilization 0.2477 2024-09-17T07:20:35.809 INFO:teuthology.orchestra.run.smithi100.stderr:no change 2024-09-17T07:20:35.827 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:35.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:35.828 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:20:35.929 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:20:35.959 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:35.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:35.960 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:20:36.014 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:20:35 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:20:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:20:36.062 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:20:36.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:36.097 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:36.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:20:36.154 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 26 16 67804 67788 162.931 111 0.030287 0.00612818 2024-09-17T07:20:36.199 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:20:36.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:36.228 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:36.228 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:20:36.329 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:20:36.364 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:36.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:36.365 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:20:36.466 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:20:36.504 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:36.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:36.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:20:36.606 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:20:36.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:36 smithi100 ceph-mon[29092]: 1.5 deep-scrub starts 2024-09-17T07:20:36.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:36 smithi100 ceph-mon[29092]: 1.5 deep-scrub ok 2024-09-17T07:20:36.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:36 smithi100 ceph-mon[39462]: 1.5 deep-scrub starts 2024-09-17T07:20:36.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:36 smithi100 ceph-mon[39462]: 1.5 deep-scrub ok 2024-09-17T07:20:36.636 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:36.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:36.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:20:36.738 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:20:36.769 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:36.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:36.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:20:36.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:36 smithi135 ceph-mon[33238]: 1.5 deep-scrub starts 2024-09-17T07:20:36.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:36 smithi135 ceph-mon[33238]: 1.5 deep-scrub ok 2024-09-17T07:20:36.840 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:20:36.871 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:20:36.902 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:36.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:36.903 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:20:37.004 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:20:37.038 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:37.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:37.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:20:37.140 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:20:37.154 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 27 16 69681 69665 161.241 117.312 0.0137907 0.00619742 2024-09-17T07:20:37.174 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:37.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:37.175 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:20:37.276 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:20:37.309 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:37.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:37.310 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:20:37.411 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:20:37.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:37.441 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:37.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:20:37.542 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:20:37.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:37.582 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:37.582 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:20:37.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:37 smithi100 ceph-mon[29092]: pgmap v859: 158 pgs: 31 active+clean+remapped, 24 active+undersized, 29 active+undersized+degraded, 74 active+clean; 3.9 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 128 MiB/s wr, 2.05k op/s; 48142/191526 objects degraded (25.136%); 48/191526 objects misplaced (0.025%) 2024-09-17T07:20:37.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:37 smithi100 ceph-mon[29092]: from='client.25841 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:20:37.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:37 smithi100 ceph-mon[39462]: pgmap v859: 158 pgs: 31 active+clean+remapped, 24 active+undersized, 29 active+undersized+degraded, 74 active+clean; 3.9 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 128 MiB/s wr, 2.05k op/s; 48142/191526 objects degraded (25.136%); 48/191526 objects misplaced (0.025%) 2024-09-17T07:20:37.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:37 smithi100 ceph-mon[39462]: from='client.25841 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:20:37.683 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:20:37.717 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:37.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:37.718 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:20:37.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:37 smithi135 ceph-mon[33238]: pgmap v859: 158 pgs: 31 active+clean+remapped, 24 active+undersized, 29 active+undersized+degraded, 74 active+clean; 3.9 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 128 MiB/s wr, 2.05k op/s; 48142/191526 objects degraded (25.136%); 48/191526 objects misplaced (0.025%) 2024-09-17T07:20:37.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:37 smithi135 ceph-mon[33238]: from='client.25841 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:20:37.819 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:20:37.857 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:37.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:37.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:20:37.959 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:20:37.993 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:37.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:37.994 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:20:38.094 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:20:38.133 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:38.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:38.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:20:38.154 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 28 16 71633 71617 159.84 122 0.00525487 0.00625247 2024-09-17T07:20:38.235 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:20:38.269 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:38.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:38.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:20:38.368 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-17T07:20:38.371 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:20:38.401 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:38.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:38.402 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:20:38.503 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:20:38.538 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:38.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:38.539 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:20:38.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:38 smithi100 ceph-mon[29092]: 6.17 deep-scrub starts 2024-09-17T07:20:38.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:38 smithi100 ceph-mon[29092]: 6.17 deep-scrub ok 2024-09-17T07:20:38.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:38 smithi100 ceph-mon[29092]: 5.1e deep-scrub starts 2024-09-17T07:20:38.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:38 smithi100 ceph-mon[29092]: 5.1e deep-scrub ok 2024-09-17T07:20:38.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:38 smithi100 ceph-mon[39462]: 6.17 deep-scrub starts 2024-09-17T07:20:38.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:38 smithi100 ceph-mon[39462]: 6.17 deep-scrub ok 2024-09-17T07:20:38.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:38 smithi100 ceph-mon[39462]: 5.1e deep-scrub starts 2024-09-17T07:20:38.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:38 smithi100 ceph-mon[39462]: 5.1e deep-scrub ok 2024-09-17T07:20:38.640 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:20:38.676 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:38.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:38.678 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:20:38.694 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:38 smithi135 ceph-mon[33238]: 6.17 deep-scrub starts 2024-09-17T07:20:38.694 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:38 smithi135 ceph-mon[33238]: 6.17 deep-scrub ok 2024-09-17T07:20:38.695 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:38 smithi135 ceph-mon[33238]: 5.1e deep-scrub starts 2024-09-17T07:20:38.695 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:38 smithi135 ceph-mon[33238]: 5.1e deep-scrub ok 2024-09-17T07:20:38.779 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:20:38.816 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:38.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:38.817 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:20:38.918 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:20:38.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:38.952 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:38.952 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:20:39.054 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:20:39.088 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:39.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:39.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:20:39.128 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-09-17T07:20:39.129 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd reweight-by-utilization 110 0.05 2024-09-17T07:20:39.154 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 29 16 73598 73582 158.563 122.812 0.00780666 0.00630268 2024-09-17T07:20:39.190 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:20:39.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:39.230 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:39.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:20:39.331 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:20:39.369 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:39.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:39.370 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:20:39.471 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:20:39.505 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:39.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:39.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:20:39.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:39 smithi135 ceph-mon[33238]: pgmap v860: 158 pgs: 31 active+clean+remapped, 24 active+undersized, 29 active+undersized+degraded, 74 active+clean; 3.9 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 108 MiB/s wr, 1.72k op/s; 48142/191526 objects degraded (25.136%); 48/191526 objects misplaced (0.025%) 2024-09-17T07:20:39.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:39 smithi135 ceph-mon[33238]: 4.1a scrub starts 2024-09-17T07:20:39.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:39 smithi135 ceph-mon[33238]: 4.1a scrub ok 2024-09-17T07:20:39.543 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:20:39 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:20:39.607 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:20:39.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:39.648 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:39.648 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:20:39.749 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:20:39.782 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:39.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:39.783 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:20:39.848 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:39 smithi100 ceph-mon[29092]: pgmap v860: 158 pgs: 31 active+clean+remapped, 24 active+undersized, 29 active+undersized+degraded, 74 active+clean; 3.9 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 108 MiB/s wr, 1.72k op/s; 48142/191526 objects degraded (25.136%); 48/191526 objects misplaced (0.025%) 2024-09-17T07:20:39.848 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:39 smithi100 ceph-mon[29092]: 4.1a scrub starts 2024-09-17T07:20:39.848 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:39 smithi100 ceph-mon[29092]: 4.1a scrub ok 2024-09-17T07:20:39.848 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:39 smithi100 ceph-mon[39462]: pgmap v860: 158 pgs: 31 active+clean+remapped, 24 active+undersized, 29 active+undersized+degraded, 74 active+clean; 3.9 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 108 MiB/s wr, 1.72k op/s; 48142/191526 objects degraded (25.136%); 48/191526 objects misplaced (0.025%) 2024-09-17T07:20:39.848 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:39 smithi100 ceph-mon[39462]: 4.1a scrub starts 2024-09-17T07:20:39.849 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:39 smithi100 ceph-mon[39462]: 4.1a scrub ok 2024-09-17T07:20:39.884 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:20:39.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:39.922 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:39.923 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:20:39.951 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:20:40.023 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:20:40.045 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:40.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:40.046 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:20:40.147 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:20:40.154 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 30 16 75299 75283 156.821 106.312 0.00434368 0.006373 2024-09-17T07:20:40.187 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:40.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:40.188 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:20:40.289 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:20:40.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:40.314 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:40.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:20:40.415 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:20:40.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:40.449 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:40.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:20:40.540 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:20:40.540 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:20:40.551 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:20:40.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:40.585 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:40.585 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:20:40.662 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:40 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:20:40.662 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:40 smithi100 ceph-mon[29092]: 4.1e scrub starts 2024-09-17T07:20:40.662 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:40 smithi100 ceph-mon[29092]: 4.1e scrub ok 2024-09-17T07:20:40.662 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:40 smithi100 ceph-mon[29092]: 6.f scrub starts 2024-09-17T07:20:40.662 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:40 smithi100 ceph-mon[29092]: 6.f scrub ok 2024-09-17T07:20:40.662 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:40 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 48142/191526 objects degraded (25.136%), 29 pgs degraded, 41 pgs undersized (PG_DEGRADED) 2024-09-17T07:20:40.662 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:40 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:20:40.662 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:40 smithi100 ceph-mon[39462]: 4.1e scrub starts 2024-09-17T07:20:40.663 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:40 smithi100 ceph-mon[39462]: 4.1e scrub ok 2024-09-17T07:20:40.663 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:40 smithi100 ceph-mon[39462]: 6.f scrub starts 2024-09-17T07:20:40.663 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:40 smithi100 ceph-mon[39462]: 6.f scrub ok 2024-09-17T07:20:40.663 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:40 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 48142/191526 objects degraded (25.136%), 29 pgs degraded, 41 pgs undersized (PG_DEGRADED) 2024-09-17T07:20:40.686 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:20:40.719 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:40.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:40.720 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:20:40.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:40 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:20:40.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:40 smithi135 ceph-mon[33238]: 4.1e scrub starts 2024-09-17T07:20:40.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:40 smithi135 ceph-mon[33238]: 4.1e scrub ok 2024-09-17T07:20:40.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:40 smithi135 ceph-mon[33238]: 6.f scrub starts 2024-09-17T07:20:40.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:40 smithi135 ceph-mon[33238]: 6.f scrub ok 2024-09-17T07:20:40.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:40 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 48142/191526 objects degraded (25.136%), 29 pgs degraded, 41 pgs undersized (PG_DEGRADED) 2024-09-17T07:20:40.821 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:20:40.842 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:40.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:40.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:20:40.944 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:20:40.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:40.978 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:40.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:20:41.079 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:20:41.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:41.115 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:41.115 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:20:41.155 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 31 15 77049 77034 155.292 109.438 0.00314728 0.00643531 2024-09-17T07:20:41.215 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:20:41.245 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:41.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:41.246 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:20:41.347 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:20:41.382 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:41.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:41.383 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:20:41.396 INFO:teuthology.orchestra.run.smithi100.stderr:noscrub is set 2024-09-17T07:20:41.484 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:20:41.516 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:41.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:41.518 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:20:41.619 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:20:41.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:41 smithi100 ceph-mon[29092]: pgmap v861: 158 pgs: 31 active+clean+remapped, 24 active+undersized, 29 active+undersized+degraded, 74 active+clean; 4.3 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 146 MiB/s wr, 2.32k op/s; 53044/213357 objects degraded (24.862%); 48/213357 objects misplaced (0.022%) 2024-09-17T07:20:41.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:41 smithi100 ceph-mon[29092]: 1.4 scrub starts 2024-09-17T07:20:41.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:41 smithi100 ceph-mon[29092]: 1.4 scrub ok 2024-09-17T07:20:41.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:41 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2732101501' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:20:41.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:41 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:20:41.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:41 smithi100 ceph-mon[39462]: pgmap v861: 158 pgs: 31 active+clean+remapped, 24 active+undersized, 29 active+undersized+degraded, 74 active+clean; 4.3 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 146 MiB/s wr, 2.32k op/s; 53044/213357 objects degraded (24.862%); 48/213357 objects misplaced (0.022%) 2024-09-17T07:20:41.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:41 smithi100 ceph-mon[39462]: 1.4 scrub starts 2024-09-17T07:20:41.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:41 smithi100 ceph-mon[39462]: 1.4 scrub ok 2024-09-17T07:20:41.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:41 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2732101501' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:20:41.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:41 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:20:41.656 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:41.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:41.657 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:20:41.757 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:20:41.791 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:41.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:41.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:20:41.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:41 smithi135 ceph-mon[33238]: pgmap v861: 158 pgs: 31 active+clean+remapped, 24 active+undersized, 29 active+undersized+degraded, 74 active+clean; 4.3 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 146 MiB/s wr, 2.32k op/s; 53044/213357 objects degraded (24.862%); 48/213357 objects misplaced (0.022%) 2024-09-17T07:20:41.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:41 smithi135 ceph-mon[33238]: 1.4 scrub starts 2024-09-17T07:20:41.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:41 smithi135 ceph-mon[33238]: 1.4 scrub ok 2024-09-17T07:20:41.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:41 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2732101501' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:20:41.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:41 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:20:41.894 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:20:41.923 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:41.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:41.924 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:20:42.025 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:20:42.056 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:42.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:42.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:20:42.154 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 32 16 79058 79042 154.36 125.5 0.0100724 0.00647423 2024-09-17T07:20:42.158 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:20:42.191 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:42.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:42.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:20:42.293 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:20:42.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:42.330 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:42.330 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:20:42.430 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:20:42.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:42.462 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:42.462 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:20:42.563 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:20:42.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:42.600 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:42.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:20:42.701 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:20:42.734 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:42.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:42.735 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:20:42.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:42 smithi135 ceph-mon[33238]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:20:42.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:42 smithi135 ceph-mon[33238]: pgmap v863: 158 pgs: 31 active+clean+remapped, 24 active+undersized, 29 active+undersized+degraded, 74 active+clean; 4.5 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 122 MiB/s wr, 1.95k op/s; 55368/220329 objects degraded (25.130%); 48/220329 objects misplaced (0.022%) 2024-09-17T07:20:42.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:42 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T07:20:42.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:42 smithi135 ceph-mon[33238]: osdmap e401: 8 total, 7 up, 6 in 2024-09-17T07:20:42.836 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:20:42.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:42 smithi100 ceph-mon[29092]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:20:42.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:42 smithi100 ceph-mon[29092]: pgmap v863: 158 pgs: 31 active+clean+remapped, 24 active+undersized, 29 active+undersized+degraded, 74 active+clean; 4.5 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 122 MiB/s wr, 1.95k op/s; 55368/220329 objects degraded (25.130%); 48/220329 objects misplaced (0.022%) 2024-09-17T07:20:42.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:42 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T07:20:42.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:42 smithi100 ceph-mon[29092]: osdmap e401: 8 total, 7 up, 6 in 2024-09-17T07:20:42.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:42 smithi100 ceph-mon[39462]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:20:42.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:42 smithi100 ceph-mon[39462]: pgmap v863: 158 pgs: 31 active+clean+remapped, 24 active+undersized, 29 active+undersized+degraded, 74 active+clean; 4.5 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 122 MiB/s wr, 1.95k op/s; 55368/220329 objects degraded (25.130%); 48/220329 objects misplaced (0.022%) 2024-09-17T07:20:42.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:42 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T07:20:42.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:42 smithi100 ceph-mon[39462]: osdmap e401: 8 total, 7 up, 6 in 2024-09-17T07:20:42.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:42.878 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:42.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:20:42.979 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:20:43.014 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:43.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:43.015 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:20:43.116 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:20:43.148 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:43.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:43.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:20:43.154 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 33 16 81083 81067 153.518 126.562 0.00542091 0.00650983 2024-09-17T07:20:43.250 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:20:43.276 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:43.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:43.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:20:43.378 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:20:43.400 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:43.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:43.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:20:43.502 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:20:43.535 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:43.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:43.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:20:43.638 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:20:43.670 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:43.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:43.671 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:20:43.772 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:20:43.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:43.810 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:43.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:20:43.911 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:20:43.949 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:43.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:43.950 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:20:44.051 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:20:44.089 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:44.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:44.090 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:20:44.155 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 34 16 82902 82886 152.346 113.688 0.00812066 0.00656006 2024-09-17T07:20:44.190 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:20:44.226 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:44.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:44.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:20:44.308 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-17T07:20:44.328 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:20:44.394 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:44.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:44.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:20:44.485 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:44 smithi100 ceph-mon[29092]: pgmap v864: 158 pgs: 31 active+clean+remapped, 24 active+undersized, 29 active+undersized+degraded, 74 active+clean; 4.6 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 121 MiB/s wr, 1.94k op/s; 56584/225726 objects degraded (25.068%); 48/225726 objects misplaced (0.021%) 2024-09-17T07:20:44.486 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:44 smithi100 ceph-mon[39462]: pgmap v864: 158 pgs: 31 active+clean+remapped, 24 active+undersized, 29 active+undersized+degraded, 74 active+clean; 4.6 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 121 MiB/s wr, 1.94k op/s; 56584/225726 objects degraded (25.068%); 48/225726 objects misplaced (0.021%) 2024-09-17T07:20:44.496 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:20:44.535 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:44.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:44.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:20:44.637 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:20:44.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:44.671 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:44.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:20:44.772 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:20:44.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:44 smithi135 ceph-mon[33238]: pgmap v864: 158 pgs: 31 active+clean+remapped, 24 active+undersized, 29 active+undersized+degraded, 74 active+clean; 4.6 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 121 MiB/s wr, 1.94k op/s; 56584/225726 objects degraded (25.068%); 48/225726 objects misplaced (0.021%) 2024-09-17T07:20:44.804 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:44.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:44.805 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:20:44.906 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:20:44.941 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:44.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:44.942 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:20:45.043 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:20:45.074 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:45.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:45.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:20:45.155 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 35 16 84899 84883 151.559 124.812 0.0110112 0.0065941 2024-09-17T07:20:45.176 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:20:45.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:45.216 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:45.216 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:20:45.317 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:20:45.353 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:45.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:45.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:20:45.455 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:20:45.486 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:45.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:45.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:20:45.544 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:20:45.544 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:20:45.547 INFO:teuthology.orchestra.run.smithi100.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-17T07:20:45.581 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:20:45.581 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:20:45.588 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:20:45.609 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:20:45.609 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:20:45.622 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:45.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:45.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:20:45.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:45 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 56584/225726 objects degraded (25.068%), 29 pgs degraded, 41 pgs undersized (PG_DEGRADED) 2024-09-17T07:20:45.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:45 smithi100 ceph-mon[29092]: osdmap e402: 8 total, 7 up, 6 in 2024-09-17T07:20:45.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:45 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 56584/225726 objects degraded (25.068%), 29 pgs degraded, 41 pgs undersized (PG_DEGRADED) 2024-09-17T07:20:45.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:45 smithi100 ceph-mon[39462]: osdmap e402: 8 total, 7 up, 6 in 2024-09-17T07:20:45.640 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:20:45.640 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:20:45.669 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:20:45.669 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:20:45.696 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:20:45.697 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:20:45.723 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:20:45.725 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:20:45.725 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:20:45.757 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:45.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:45.759 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:20:45.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:45 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 56584/225726 objects degraded (25.068%), 29 pgs degraded, 41 pgs undersized (PG_DEGRADED) 2024-09-17T07:20:45.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:45 smithi135 ceph-mon[33238]: osdmap e402: 8 total, 7 up, 6 in 2024-09-17T07:20:45.860 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:20:45.897 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:45.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:45.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:20:45.998 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:20:46.006 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:20:45 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:20:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:20:46.031 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:46.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:46.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:20:46.133 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:20:46.155 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 36 16 86770 86754 150.597 116.938 0.00470369 0.00663615 2024-09-17T07:20:46.165 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:46.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:46.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:20:46.267 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:20:46.302 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:46.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:46.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:20:46.405 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:20:46.433 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:46.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:46.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:20:46.535 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:20:46.570 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:46.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:46.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:20:46.672 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:20:46.704 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:46.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:46.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:20:46.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:46 smithi135 ceph-mon[33238]: pgmap v866: 158 pgs: 31 active+clean+remapped, 24 active+undersized, 29 active+undersized+degraded, 74 active+clean; 5.0 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 148 MiB/s wr, 2.36k op/s; 62384/248301 objects degraded (25.124%); 48/248301 objects misplaced (0.019%) 2024-09-17T07:20:46.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:46 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2694559585' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:20:46.806 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:20:46.840 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:46.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:46.841 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:20:46.851 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:46 smithi100 ceph-mon[29092]: pgmap v866: 158 pgs: 31 active+clean+remapped, 24 active+undersized, 29 active+undersized+degraded, 74 active+clean; 5.0 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 148 MiB/s wr, 2.36k op/s; 62384/248301 objects degraded (25.124%); 48/248301 objects misplaced (0.019%) 2024-09-17T07:20:46.852 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:46 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2694559585' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:20:46.852 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:46 smithi100 ceph-mon[39462]: pgmap v866: 158 pgs: 31 active+clean+remapped, 24 active+undersized, 29 active+undersized+degraded, 74 active+clean; 5.0 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 148 MiB/s wr, 2.36k op/s; 62384/248301 objects degraded (25.124%); 48/248301 objects misplaced (0.019%) 2024-09-17T07:20:46.852 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:46 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2694559585' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:20:46.943 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:20:46.979 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:46.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:46.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:20:47.081 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:20:47.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:47.113 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:47.113 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:20:47.155 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 37 16 88696 88680 149.78 120.375 0.0070371 0.00667256 2024-09-17T07:20:47.215 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:20:47.244 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:47.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:47.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:20:47.346 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:20:47.377 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:47.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:47.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:20:47.479 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:20:47.511 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:47.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:47.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:20:47.606 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:20:47.607 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:20:47.612 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:20:47.648 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:47.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:47.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:20:47.749 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:20:47.780 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:47.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:47.781 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:20:47.882 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:20:47.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:47.912 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:47.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:20:48.013 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:20:48.048 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:48.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:48.050 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:20:48.151 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:20:48.155 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 38 16 90629 90613 149.017 120.812 0.0201321 0.00670577 2024-09-17T07:20:48.176 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:48.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:48.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:20:48.278 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:20:48.310 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:48.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:48.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:20:48.412 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:20:48.421 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-17T07:20:48.422 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-17T07:20:48.422 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-17T07:20:48.422 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-17T07:20:48.422 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-17T07:20:48.438 INFO:teuthology.orchestra.run.smithi100.stdout:moved 7 / 474 (1.47679%) 2024-09-17T07:20:48.438 INFO:teuthology.orchestra.run.smithi100.stdout:avg 94.8 2024-09-17T07:20:48.438 INFO:teuthology.orchestra.run.smithi100.stdout:stddev 17.1184 -> 17.6884 (expected baseline 8.70862) 2024-09-17T07:20:48.438 INFO:teuthology.orchestra.run.smithi100.stdout:min osd.4 with 74 -> 77 pgs (0.780591 -> 0.812236 * mean) 2024-09-17T07:20:48.438 INFO:teuthology.orchestra.run.smithi100.stdout:max osd.6 with 83 -> 82 pgs (0.875527 -> 0.864979 * mean) 2024-09-17T07:20:48.438 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T07:20:48.438 INFO:teuthology.orchestra.run.smithi100.stdout:oload 110 2024-09-17T07:20:48.438 INFO:teuthology.orchestra.run.smithi100.stdout:max_change 0.05 2024-09-17T07:20:48.439 INFO:teuthology.orchestra.run.smithi100.stdout:max_change_osds 4 2024-09-17T07:20:48.439 INFO:teuthology.orchestra.run.smithi100.stdout:average_utilization 0.0298 2024-09-17T07:20:48.439 INFO:teuthology.orchestra.run.smithi100.stdout:overload_utilization 0.0328 2024-09-17T07:20:48.439 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6 weight 1.0000 -> 0.9500 2024-09-17T07:20:48.439 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0 weight 1.0000 -> 0.9500 2024-09-17T07:20:48.439 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2 weight 1.0000 -> 0.9500 2024-09-17T07:20:48.464 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:48.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:48.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:20:48.565 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:20:48.599 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:48.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:48.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:20:48.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:48 smithi100 ceph-mon[39462]: pgmap v867: 158 pgs: 31 active+clean+remapped, 24 active+undersized, 29 active+undersized+degraded, 74 active+clean; 5.0 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 91 MiB/s wr, 1.45k op/s; 62384/248301 objects degraded (25.124%); 48/248301 objects misplaced (0.019%) 2024-09-17T07:20:48.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:48 smithi100 ceph-mon[39462]: from='client.16014 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:20:48.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:48 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'0':'62260','2':'62260','6':'62260'}"}]: dispatch 2024-09-17T07:20:48.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:48 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:20:48.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:48 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:20:48.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:48 smithi100 ceph-mon[29092]: pgmap v867: 158 pgs: 31 active+clean+remapped, 24 active+undersized, 29 active+undersized+degraded, 74 active+clean; 5.0 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 91 MiB/s wr, 1.45k op/s; 62384/248301 objects degraded (25.124%); 48/248301 objects misplaced (0.019%) 2024-09-17T07:20:48.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:48 smithi100 ceph-mon[29092]: from='client.16014 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:20:48.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:48 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'0':'62260','2':'62260','6':'62260'}"}]: dispatch 2024-09-17T07:20:48.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:48 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:20:48.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:48 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:20:48.702 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:20:48.744 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:48.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:48.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:20:48.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:48 smithi135 ceph-mon[33238]: pgmap v867: 158 pgs: 31 active+clean+remapped, 24 active+undersized, 29 active+undersized+degraded, 74 active+clean; 5.0 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 91 MiB/s wr, 1.45k op/s; 62384/248301 objects degraded (25.124%); 48/248301 objects misplaced (0.019%) 2024-09-17T07:20:48.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:48 smithi135 ceph-mon[33238]: from='client.16014 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:20:48.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:48 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'0':'62260','2':'62260','6':'62260'}"}]: dispatch 2024-09-17T07:20:48.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:48 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:20:48.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:48 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:20:48.846 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:20:48.881 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:48.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:48.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:20:48.983 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:20:49.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:49.016 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:49.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:20:49.117 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:20:49.155 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:49.155 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 39 16 92538 92522 148.255 119.312 0.00310514 0.00674117 2024-09-17T07:20:49.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:49.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:20:49.257 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:20:49.291 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:49.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:49.292 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:20:49.393 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:20:49.425 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:49.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:49.426 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:20:49.442 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:20:49 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:20:49.484 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:20:49.514 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-17T07:20:49.527 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:20:49.557 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:49.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:49.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:20:49.608 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:20:49.660 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:20:49.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:49.692 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:49.693 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:20:49.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:49 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'62260','2':'62260','6':'62260'}"}]': finished 2024-09-17T07:20:49.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:49 smithi135 ceph-mon[33238]: osdmap e403: 8 total, 7 up, 6 in 2024-09-17T07:20:49.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:49 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:20:49.793 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:20:49.805 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:49 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'62260','2':'62260','6':'62260'}"}]': finished 2024-09-17T07:20:49.805 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:49 smithi100 ceph-mon[29092]: osdmap e403: 8 total, 7 up, 6 in 2024-09-17T07:20:49.805 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:49 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:20:49.806 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:49 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'62260','2':'62260','6':'62260'}"}]': finished 2024-09-17T07:20:49.806 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:49 smithi100 ceph-mon[39462]: osdmap e403: 8 total, 7 up, 6 in 2024-09-17T07:20:49.806 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:49 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:20:49.841 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:49.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:49.842 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:20:49.943 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:20:49.984 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:49.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:49.985 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:20:50.086 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:20:50.119 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:50.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:50.120 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:20:50.155 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:2024-09-17T07:20:50.155825+0000 min lat: 0.00105244 max lat: 0.145857 avg lat: 0.00677522 2024-09-17T07:20:50.156 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-17T07:20:50.156 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 40 16 94433 94417 147.51 118.438 0.00454228 0.00677522 2024-09-17T07:20:50.221 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:20:50.256 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:50.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:50.257 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:20:50.358 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:20:50.394 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:50.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:50.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:20:50.496 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:20:50.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:50.530 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:50.530 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:20:50.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:50 smithi100 ceph-mon[29092]: pgmap v869: 158 pgs: 2 peering, 31 active+clean+remapped, 24 active+undersized, 29 active+undersized+degraded, 72 active+clean; 5.5 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 129 MiB/s wr, 2.07k op/s; 67211/270042 objects degraded (24.889%); 48/270042 objects misplaced (0.018%) 2024-09-17T07:20:50.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:50 smithi100 ceph-mon[29092]: osdmap e404: 8 total, 7 up, 6 in 2024-09-17T07:20:50.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:50 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 67211/270042 objects degraded (24.889%), 29 pgs degraded, 41 pgs undersized (PG_DEGRADED) 2024-09-17T07:20:50.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:50 smithi100 ceph-mon[39462]: pgmap v869: 158 pgs: 2 peering, 31 active+clean+remapped, 24 active+undersized, 29 active+undersized+degraded, 72 active+clean; 5.5 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 129 MiB/s wr, 2.07k op/s; 67211/270042 objects degraded (24.889%); 48/270042 objects misplaced (0.018%) 2024-09-17T07:20:50.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:50 smithi100 ceph-mon[39462]: osdmap e404: 8 total, 7 up, 6 in 2024-09-17T07:20:50.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:50 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 67211/270042 objects degraded (24.889%), 29 pgs degraded, 41 pgs undersized (PG_DEGRADED) 2024-09-17T07:20:50.631 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:20:50.662 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:50.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:50.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:20:50.763 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:20:50.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:50 smithi135 ceph-mon[33238]: pgmap v869: 158 pgs: 2 peering, 31 active+clean+remapped, 24 active+undersized, 29 active+undersized+degraded, 72 active+clean; 5.5 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 129 MiB/s wr, 2.07k op/s; 67211/270042 objects degraded (24.889%); 48/270042 objects misplaced (0.018%) 2024-09-17T07:20:50.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:50 smithi135 ceph-mon[33238]: osdmap e404: 8 total, 7 up, 6 in 2024-09-17T07:20:50.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:50 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 67211/270042 objects degraded (24.889%), 29 pgs degraded, 41 pgs undersized (PG_DEGRADED) 2024-09-17T07:20:50.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:50.796 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:50.796 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:20:50.897 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:20:50.930 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:50.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:50.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:20:51.032 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:20:51.070 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:51.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:51.071 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:20:51.155 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 41 16 96385 96369 146.887 122 0.00571589 0.00680418 2024-09-17T07:20:51.172 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:20:51.210 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:51.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:51.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:20:51.311 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:20:51.342 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:51.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:51.343 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:20:51.444 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:20:51.478 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:51.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:51.480 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:20:51.547 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:20:51.557 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:20:51.580 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:20:51.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:51.611 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:51.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:20:51.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:51 smithi100 ceph-mon[29092]: osdmap e405: 8 total, 7 up, 6 in 2024-09-17T07:20:51.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:51 smithi100 ceph-mon[39462]: osdmap e405: 8 total, 7 up, 6 in 2024-09-17T07:20:51.711 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:20:51.744 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:51.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:51.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:20:51.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:51 smithi135 ceph-mon[33238]: osdmap e405: 8 total, 7 up, 6 in 2024-09-17T07:20:51.846 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:20:51.881 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:51.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:51.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:20:51.983 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:20:52.016 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:52.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:52.017 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:20:52.118 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:20:52.153 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:52.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:52.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:20:52.155 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 42 16 98218 98202 146.117 114.562 0.00741066 0.00683982 2024-09-17T07:20:52.255 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:20:52.289 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:52.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:52.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:20:52.391 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:20:52.426 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:52.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:52.428 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:20:52.528 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:20:52.566 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:52.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:52.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:20:52.669 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:20:52.700 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:52.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:52.702 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:20:52.752 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:52 smithi100 ceph-mon[29092]: pgmap v872: 158 pgs: 2 remapped+peering, 2 peering, 30 active+clean+remapped, 24 active+undersized, 28 active+undersized+degraded, 72 active+clean; 5.6 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 852 B/s rd, 100 MiB/s wr, 1.60k op/s; 69576/277158 objects degraded (25.103%); 48/277158 objects misplaced (0.017%) 2024-09-17T07:20:52.752 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:52 smithi100 ceph-mon[39462]: pgmap v872: 158 pgs: 2 remapped+peering, 2 peering, 30 active+clean+remapped, 24 active+undersized, 28 active+undersized+degraded, 72 active+clean; 5.6 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 852 B/s rd, 100 MiB/s wr, 1.60k op/s; 69576/277158 objects degraded (25.103%); 48/277158 objects misplaced (0.017%) 2024-09-17T07:20:52.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:52 smithi135 ceph-mon[33238]: pgmap v872: 158 pgs: 2 remapped+peering, 2 peering, 30 active+clean+remapped, 24 active+undersized, 28 active+undersized+degraded, 72 active+clean; 5.6 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 852 B/s rd, 100 MiB/s wr, 1.60k op/s; 69576/277158 objects degraded (25.103%); 48/277158 objects misplaced (0.017%) 2024-09-17T07:20:52.802 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:20:52.826 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:52.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:52.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:20:52.928 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:20:52.949 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:52.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:52.950 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:20:53.051 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:20:53.087 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:53.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:53.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:20:53.155 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 43 16 99992 99976 145.298 110.875 0.00338836 0.00687862 2024-09-17T07:20:53.189 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:20:53.221 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:53.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:53.222 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:20:53.323 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:20:53.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:53.357 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:53.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:20:53.458 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:20:53.495 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:53.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:53.496 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:20:53.597 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:20:53.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:53.631 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:53.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:20:53.732 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:20:53.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:53.771 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:53.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:20:53.872 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:20:53.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:53.910 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:53.911 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:20:54.011 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:20:54.047 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:54.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:54.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:20:54.149 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:20:54.155 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 44 15 101877 101862 144.674 117.875 0.0106791 0.00690829 2024-09-17T07:20:54.182 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:54.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:54.183 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:20:54.284 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:20:54.317 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:54.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:54.318 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:20:54.419 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:20:54.453 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:54.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:54.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:20:54.491 INFO:teuthology.orchestra.run.smithi100.stderr:nodeep-scrub is set 2024-09-17T07:20:54.554 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:20:54.577 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:54.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:54.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:20:54.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:54 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:20:54.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:54 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:20:54.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:54 smithi100 ceph-mon[29092]: pgmap v873: 158 pgs: 2 remapped+peering, 2 peering, 30 active+clean+remapped, 24 active+undersized, 28 active+undersized+degraded, 72 active+clean; 5.7 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 852 B/s rd, 119 MiB/s wr, 1.90k op/s; 70770/282552 objects degraded (25.047%); 48/282552 objects misplaced (0.017%) 2024-09-17T07:20:54.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:54 smithi100 ceph-mon[29092]: osdmap e406: 8 total, 7 up, 6 in 2024-09-17T07:20:54.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:54 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1658707932' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:20:54.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:54 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:20:54.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:54 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:20:54.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:54 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:20:54.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:54 smithi100 ceph-mon[39462]: pgmap v873: 158 pgs: 2 remapped+peering, 2 peering, 30 active+clean+remapped, 24 active+undersized, 28 active+undersized+degraded, 72 active+clean; 5.7 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 852 B/s rd, 119 MiB/s wr, 1.90k op/s; 70770/282552 objects degraded (25.047%); 48/282552 objects misplaced (0.017%) 2024-09-17T07:20:54.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:54 smithi100 ceph-mon[39462]: osdmap e406: 8 total, 7 up, 6 in 2024-09-17T07:20:54.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:54 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1658707932' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:20:54.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:54 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:20:54.679 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:20:54.714 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:54.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:54.716 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:20:54.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:54 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:20:54.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:54 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:20:54.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:54 smithi135 ceph-mon[33238]: pgmap v873: 158 pgs: 2 remapped+peering, 2 peering, 30 active+clean+remapped, 24 active+undersized, 28 active+undersized+degraded, 72 active+clean; 5.7 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 852 B/s rd, 119 MiB/s wr, 1.90k op/s; 70770/282552 objects degraded (25.047%); 48/282552 objects misplaced (0.017%) 2024-09-17T07:20:54.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:54 smithi135 ceph-mon[33238]: osdmap e406: 8 total, 7 up, 6 in 2024-09-17T07:20:54.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:54 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1658707932' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:20:54.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:54 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:20:54.817 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:20:54.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:54.839 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:54.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:20:54.940 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:20:54.972 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:54.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:54.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:20:55.073 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:20:55.112 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:55.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:55.113 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:20:55.155 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 45 15 103661 103646 143.937 111.5 0.0104119 0.00694374 2024-09-17T07:20:55.214 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:20:55.245 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:55.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:55.246 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:20:55.347 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:20:55.383 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:55.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:55.384 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:20:55.484 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:20:55.515 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:55.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:55.516 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:20:55.617 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:20:55.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:55.657 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:55.658 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:20:55.758 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:20:55.792 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:55.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:55.793 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:20:55.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:55 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:20:55.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:55 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:20:55.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:55 smithi135 ceph-mon[33238]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:20:55.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:55 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T07:20:55.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:55 smithi135 ceph-mon[33238]: osdmap e407: 8 total, 7 up, 6 in 2024-09-17T07:20:55.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:55 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 70770/282552 objects degraded (25.047%), 28 pgs degraded, 40 pgs undersized (PG_DEGRADED) 2024-09-17T07:20:55.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:55 smithi135 ceph-mon[33238]: osdmap e408: 8 total, 7 up, 6 in 2024-09-17T07:20:55.864 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6] out_osds: [3, 7] dead_osds: [1] live_osds: [0, 2, 3, 4, 5, 6, 7] 2024-09-17T07:20:55.864 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T07:20:55.864 INFO:tasks.thrashosds.thrasher:Reviving osd 1 2024-09-17T07:20:55.864 INFO:tasks.cephadm.osd.1:Restarting osd.1 (starting--it wasn't running)... 2024-09-17T07:20:55.864 DEBUG:teuthology.orchestra.run.smithi100:osd.1> sudo journalctl -f -n 0 -u ceph-e848267c-74c1-11ef-bceb-c7b262605968@osd.1.service 2024-09-17T07:20:55.867 DEBUG:teuthology.orchestra.run.smithi100:> sudo systemctl start ceph-e848267c-74c1-11ef-bceb-c7b262605968@osd.1 2024-09-17T07:20:55.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:55 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:20:55.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:55 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:20:55.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:55 smithi100 ceph-mon[29092]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:20:55.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:55 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T07:20:55.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:55 smithi100 ceph-mon[29092]: osdmap e407: 8 total, 7 up, 6 in 2024-09-17T07:20:55.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:55 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 70770/282552 objects degraded (25.047%), 28 pgs degraded, 40 pgs undersized (PG_DEGRADED) 2024-09-17T07:20:55.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:55 smithi100 ceph-mon[29092]: osdmap e408: 8 total, 7 up, 6 in 2024-09-17T07:20:55.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:55 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:20:55.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:55 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:20:55.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:55 smithi100 ceph-mon[39462]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:20:55.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:55 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T07:20:55.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:55 smithi100 ceph-mon[39462]: osdmap e407: 8 total, 7 up, 6 in 2024-09-17T07:20:55.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:55 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 70770/282552 objects degraded (25.047%), 28 pgs degraded, 40 pgs undersized (PG_DEGRADED) 2024-09-17T07:20:55.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:55 smithi100 ceph-mon[39462]: osdmap e408: 8 total, 7 up, 6 in 2024-09-17T07:20:55.894 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:20:55.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:55.960 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:55.960 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:20:56.061 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:20:56.096 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:56.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:56.098 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:20:56.146 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:20:55 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:20:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:20:56.147 INFO:journalctl@ceph.osd.1.smithi100.stdout:Sep 17 07:20:55 smithi100 systemd[1]: Starting Ceph osd.1 for e848267c-74c1-11ef-bceb-c7b262605968... 2024-09-17T07:20:56.164 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 46 16 105631 105615 143.455 123.062 0.00406828 0.00696644 2024-09-17T07:20:56.199 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:20:56.229 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:56.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:56.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:20:56.331 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:20:56.360 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:56.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:56.361 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:20:56.441 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:20:56.441 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:20:56.442 INFO:teuthology.orchestra.run.smithi100.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-17T07:20:56.462 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:20:56.475 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:20:56.475 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:20:56.495 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:56.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:56.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:20:56.504 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:20:56.504 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:20:56.535 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:20:56.535 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:20:56.565 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:20:56.565 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:20:56.593 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:20:56.593 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:20:56.597 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:20:56.621 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:20:56.622 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:20:56.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:56.633 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:56.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:20:56.734 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:20:56.772 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:56.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:56.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:20:56.874 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:20:56.909 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:56.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:56.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:20:57.011 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:20:57.050 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:57.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:57.052 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:20:57.152 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:20:57.164 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 47 16 107545 107529 142.948 119.625 0.00329257 0.0069918 2024-09-17T07:20:57.184 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:57.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:57.185 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:20:57.286 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:20:57.317 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:57.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:57.318 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:20:57.419 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:20:57.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:57.455 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:57.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:20:57.555 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:20:57.589 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:57.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:57.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:20:57.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:57 smithi100 ceph-mon[29092]: pgmap v877: 158 pgs: 33 active+clean+remapped, 24 active+undersized, 28 active+undersized+degraded, 73 active+clean; 6.2 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 1.0 KiB/s rd, 119 MiB/s wr, 1.90k op/s; 76609/305400 objects degraded (25.085%); 55/305400 objects misplaced (0.018%); 22 B/s, 1 objects/s recovering 2024-09-17T07:20:57.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:57 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:20:57.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:57 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:20:57.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:57 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/3242999992' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:20:57.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:57 smithi100 ceph-mon[29092]: osdmap e409: 8 total, 7 up, 6 in 2024-09-17T07:20:57.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:57 smithi100 ceph-mon[39462]: pgmap v877: 158 pgs: 33 active+clean+remapped, 24 active+undersized, 28 active+undersized+degraded, 73 active+clean; 6.2 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 1.0 KiB/s rd, 119 MiB/s wr, 1.90k op/s; 76609/305400 objects degraded (25.085%); 55/305400 objects misplaced (0.018%); 22 B/s, 1 objects/s recovering 2024-09-17T07:20:57.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:57 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:20:57.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:57 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:20:57.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:57 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/3242999992' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:20:57.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:57 smithi100 ceph-mon[39462]: osdmap e409: 8 total, 7 up, 6 in 2024-09-17T07:20:57.690 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:20:57.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:57.722 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:57.723 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:20:57.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:57 smithi135 ceph-mon[33238]: pgmap v877: 158 pgs: 33 active+clean+remapped, 24 active+undersized, 28 active+undersized+degraded, 73 active+clean; 6.2 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 1.0 KiB/s rd, 119 MiB/s wr, 1.90k op/s; 76609/305400 objects degraded (25.085%); 55/305400 objects misplaced (0.018%); 22 B/s, 1 objects/s recovering 2024-09-17T07:20:57.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:57 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:20:57.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:57 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:20:57.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:57 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/3242999992' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:20:57.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:57 smithi135 ceph-mon[33238]: osdmap e409: 8 total, 7 up, 6 in 2024-09-17T07:20:57.823 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:20:57.860 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:57.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:57.861 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:20:57.962 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:20:57.999 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:57.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:58.000 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:20:58.101 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:20:58.123 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:58.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:58.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:20:58.127 INFO:journalctl@ceph.osd.1.smithi100.stdout:Sep 17 07:20:57 smithi100 podman[184883]: 2024-09-17 07:20:57.776997846 +0000 UTC m=+1.374734647 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9 2024-09-17T07:20:58.164 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 48 16 109495 109479 142.509 121.875 0.00620152 0.0070134 2024-09-17T07:20:58.225 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:20:58.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:58.261 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:58.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:20:58.362 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:20:58.395 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:58.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:58.396 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:20:58.400 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-17T07:20:58.400 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:20:58.497 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:20:58.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:58.535 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:58.535 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:20:58.636 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:20:58.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:58.673 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:58.673 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:20:58.774 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:20:58.811 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:58.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:58.812 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:20:58.876 INFO:journalctl@ceph.osd.1.smithi100.stdout:Sep 17 07:20:58 smithi100 podman[184883]: 2024-09-17 07:20:58.541413943 +0000 UTC m=+2.139150718 container create 5ac64fc37b3cc5f81e13d8fefd20544ff1adf03904f7e45872956d37bf4a4472 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-1-activate, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, RELEASE=reef-19e2a2b, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD) 2024-09-17T07:20:58.912 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:20:58.947 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:58.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:58.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:20:59.049 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:20:59.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:59.077 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:59.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:20:59.165 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 49 16 111291 111275 141.892 112.25 0.0031468 0.00704325 2024-09-17T07:20:59.178 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:20:59.199 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:59.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:59.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:20:59.286 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:20:59.287 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:20:59.301 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:20:59.335 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:59.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:59.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:20:59.437 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:20:59.472 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:59.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:59.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:20:59.520 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T07:20:59.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:20:59 smithi135 ceph-mon[33238]: pgmap v879: 158 pgs: 33 active+clean+remapped, 24 active+undersized, 28 active+undersized+degraded, 73 active+clean; 6.2 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 1.2 KiB/s rd, 119 MiB/s wr, 1.90k op/s; 76609/305400 objects degraded (25.085%); 55/305400 objects misplaced (0.018%); 27 B/s, 1 objects/s recovering 2024-09-17T07:20:59.543 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:20:59 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:20:59.554 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:20:59 smithi100 ceph-mon[29092]: pgmap v879: 158 pgs: 33 active+clean+remapped, 24 active+undersized, 28 active+undersized+degraded, 73 active+clean; 6.2 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 1.2 KiB/s rd, 119 MiB/s wr, 1.90k op/s; 76609/305400 objects degraded (25.085%); 55/305400 objects misplaced (0.018%); 27 B/s, 1 objects/s recovering 2024-09-17T07:20:59.554 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:20:59 smithi100 ceph-mon[39462]: pgmap v879: 158 pgs: 33 active+clean+remapped, 24 active+undersized, 28 active+undersized+degraded, 73 active+clean; 6.2 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 1.2 KiB/s rd, 119 MiB/s wr, 1.90k op/s; 76609/305400 objects degraded (25.085%); 55/305400 objects misplaced (0.018%); 27 B/s, 1 objects/s recovering 2024-09-17T07:20:59.574 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:20:59.608 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:59.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:59.609 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:20:59.710 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:20:59.788 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:20:59.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:59.789 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:20:59.890 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:20:59.927 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:20:59.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:20:59.928 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:21:00.029 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:21:00.061 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:00.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:00.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:21:00.163 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:21:00.165 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 50 16 113056 113040 141.26 110.312 0.00367618 0.00707525 2024-09-17T07:21:00.201 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:00.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:00.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:21:00.303 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:21:00.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:00.334 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:00.334 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:21:00.435 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:21:00.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:00.468 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:00.468 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:21:00.569 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:21:00.599 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:00.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:00.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:21:00.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:00 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:21:00.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:00 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 76609/305400 objects degraded (25.085%), 28 pgs degraded, 52 pgs undersized (PG_DEGRADED) 2024-09-17T07:21:00.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:00 smithi100 ceph-mon[29092]: osdmap e410: 8 total, 7 up, 6 in 2024-09-17T07:21:00.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:00 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:21:00.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:00 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 76609/305400 objects degraded (25.085%), 28 pgs degraded, 52 pgs undersized (PG_DEGRADED) 2024-09-17T07:21:00.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:00 smithi100 ceph-mon[39462]: osdmap e410: 8 total, 7 up, 6 in 2024-09-17T07:21:00.701 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:21:00.732 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:00.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:00.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:21:00.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:00 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:21:00.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:00 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 76609/305400 objects degraded (25.085%), 28 pgs degraded, 52 pgs undersized (PG_DEGRADED) 2024-09-17T07:21:00.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:00 smithi135 ceph-mon[33238]: osdmap e410: 8 total, 7 up, 6 in 2024-09-17T07:21:00.834 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:21:00.867 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:00.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:00.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:21:00.969 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:21:01.000 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:01.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:01.001 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:21:01.102 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:21:01.139 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:01.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:01.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:21:01.165 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 51 16 114921 114905 140.776 116.562 0.00438969 0.00709988 2024-09-17T07:21:01.241 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:21:01.262 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:01.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:01.263 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:21:01.288 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:21:01.364 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:21:01.406 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:01.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:01.407 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:21:01.508 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:21:01.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:01.548 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:01.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:21:01.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:01 smithi100 ceph-mon[29092]: pgmap v880: 158 pgs: 33 active+clean+remapped, 24 active+undersized, 28 active+undersized+degraded, 73 active+clean; 6.6 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 1.7 KiB/s rd, 155 MiB/s wr, 2.48k op/s; 81407/326721 objects degraded (24.916%); 55/326721 objects misplaced (0.017%); 18 B/s, 1 objects/s recovering 2024-09-17T07:21:01.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:01 smithi100 ceph-mon[39462]: pgmap v880: 158 pgs: 33 active+clean+remapped, 24 active+undersized, 28 active+undersized+degraded, 73 active+clean; 6.6 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 1.7 KiB/s rd, 155 MiB/s wr, 2.48k op/s; 81407/326721 objects degraded (24.916%); 55/326721 objects misplaced (0.017%); 18 B/s, 1 objects/s recovering 2024-09-17T07:21:01.649 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:21:01.684 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:01.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:01.685 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:21:01.786 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:21:01.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:01 smithi135 ceph-mon[33238]: pgmap v880: 158 pgs: 33 active+clean+remapped, 24 active+undersized, 28 active+undersized+degraded, 73 active+clean; 6.6 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 1.7 KiB/s rd, 155 MiB/s wr, 2.48k op/s; 81407/326721 objects degraded (24.916%); 55/326721 objects misplaced (0.017%); 18 B/s, 1 objects/s recovering 2024-09-17T07:21:01.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:01.807 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:01.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:21:01.909 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:21:01.941 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:01.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:01.942 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:21:02.043 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:21:02.065 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:02.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:02.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:21:02.165 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 52 15 116517 116502 139.988 99.8125 0.0109838 0.00713964 2024-09-17T07:21:02.167 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:21:02.199 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:02.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:02.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:21:02.301 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:21:02.334 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:02.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:02.335 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:21:02.435 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:21:02.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:02.469 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:02.469 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:21:02.570 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:21:02.605 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:02.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:02.606 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:21:02.707 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:21:02.739 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:02.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:02.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:21:02.841 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:21:02.866 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:02.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:02.867 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:21:02.968 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:21:03.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:03.005 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:03.005 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:21:03.106 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:21:03.137 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:03.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:03.138 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:21:03.165 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 53 16 118300 118284 139.448 111.375 0.00470323 0.00716731 2024-09-17T07:21:03.239 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:21:03.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:03.276 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:03.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:21:03.377 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:21:03.407 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:03.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:03.409 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:21:03.509 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:21:03.546 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:03.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:03.547 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:21:03.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:03 smithi100 ceph-mon[29092]: pgmap v882: 158 pgs: 33 active+clean+remapped, 24 active+undersized, 28 active+undersized+degraded, 73 active+clean; 6.8 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 839 B/s rd, 97 MiB/s wr, 1.55k op/s; 83745/333735 objects degraded (25.093%); 55/333735 objects misplaced (0.016%) 2024-09-17T07:21:03.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:03 smithi100 ceph-mon[39462]: pgmap v882: 158 pgs: 33 active+clean+remapped, 24 active+undersized, 28 active+undersized+degraded, 73 active+clean; 6.8 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 839 B/s rd, 97 MiB/s wr, 1.55k op/s; 83745/333735 objects degraded (25.093%); 55/333735 objects misplaced (0.016%) 2024-09-17T07:21:03.627 INFO:journalctl@ceph.osd.1.smithi100.stdout:Sep 17 07:21:03 smithi100 podman[184883]: 2024-09-17 07:21:03.380807238 +0000 UTC m=+6.978544022 container init 5ac64fc37b3cc5f81e13d8fefd20544ff1adf03904f7e45872956d37bf4a4472 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-1-activate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, RELEASE=reef-19e2a2b, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T07:21:03.627 INFO:journalctl@ceph.osd.1.smithi100.stdout:Sep 17 07:21:03 smithi100 podman[184883]: 2024-09-17 07:21:03.38949282 +0000 UTC m=+6.987229595 container start 5ac64fc37b3cc5f81e13d8fefd20544ff1adf03904f7e45872956d37bf4a4472 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-1-activate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=reef-19e2a2b, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T07:21:03.627 INFO:journalctl@ceph.osd.1.smithi100.stdout:Sep 17 07:21:03 smithi100 podman[184883]: 2024-09-17 07:21:03.489279514 +0000 UTC m=+7.087016300 container attach 5ac64fc37b3cc5f81e13d8fefd20544ff1adf03904f7e45872956d37bf4a4472 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-1-activate, RELEASE=reef-19e2a2b, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-09-17T07:21:03.648 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:21:03.671 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-17T07:21:03.713 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:03.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:03.713 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:21:03.736 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:21:03.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:03 smithi135 ceph-mon[33238]: pgmap v882: 158 pgs: 33 active+clean+remapped, 24 active+undersized, 28 active+undersized+degraded, 73 active+clean; 6.8 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 839 B/s rd, 97 MiB/s wr, 1.55k op/s; 83745/333735 objects degraded (25.093%); 55/333735 objects misplaced (0.016%) 2024-09-17T07:21:03.806 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:21:03.814 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:21:03.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:03.864 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:03.864 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:21:03.965 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:21:03.996 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:03.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:03.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:21:04.098 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:21:04.135 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:04.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:04.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:21:04.165 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 54 16 119913 119897 138.732 100.812 0.00618099 0.00720438 2024-09-17T07:21:04.239 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:21:04.275 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:04.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:04.276 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:21:04.344 INFO:journalctl@ceph.osd.1.smithi100.stdout:Sep 17 07:21:03 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-1-activate[185091]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-17T07:21:04.344 INFO:journalctl@ceph.osd.1.smithi100.stdout:Sep 17 07:21:03 smithi100 bash[184883]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-17T07:21:04.344 INFO:journalctl@ceph.osd.1.smithi100.stdout:Sep 17 07:21:03 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-1-activate[185091]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-09-17T07:21:04.344 INFO:journalctl@ceph.osd.1.smithi100.stdout:Sep 17 07:21:03 smithi100 bash[184883]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-09-17T07:21:04.344 INFO:journalctl@ceph.osd.1.smithi100.stdout:Sep 17 07:21:03 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-1-activate[185091]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-09-17T07:21:04.344 INFO:journalctl@ceph.osd.1.smithi100.stdout:Sep 17 07:21:03 smithi100 bash[184883]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-09-17T07:21:04.344 INFO:journalctl@ceph.osd.1.smithi100.stdout:Sep 17 07:21:03 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-1-activate[185091]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-09-17T07:21:04.344 INFO:journalctl@ceph.osd.1.smithi100.stdout:Sep 17 07:21:03 smithi100 bash[184883]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-09-17T07:21:04.344 INFO:journalctl@ceph.osd.1.smithi100.stdout:Sep 17 07:21:03 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-1-activate[185091]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-09-17T07:21:04.345 INFO:journalctl@ceph.osd.1.smithi100.stdout:Sep 17 07:21:03 smithi100 bash[184883]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-09-17T07:21:04.345 INFO:journalctl@ceph.osd.1.smithi100.stdout:Sep 17 07:21:03 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-1-activate[185091]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-17T07:21:04.345 INFO:journalctl@ceph.osd.1.smithi100.stdout:Sep 17 07:21:03 smithi100 bash[184883]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-17T07:21:04.345 INFO:journalctl@ceph.osd.1.smithi100.stdout:Sep 17 07:21:03 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-1-activate[185091]: --> ceph-volume raw activate successful for osd ID: 1 2024-09-17T07:21:04.345 INFO:journalctl@ceph.osd.1.smithi100.stdout:Sep 17 07:21:03 smithi100 bash[184883]: --> ceph-volume raw activate successful for osd ID: 1 2024-09-17T07:21:04.345 INFO:journalctl@ceph.osd.1.smithi100.stdout:Sep 17 07:21:04 smithi100 podman[184883]: 2024-09-17 07:21:04.046292086 +0000 UTC m=+7.644028867 container died 5ac64fc37b3cc5f81e13d8fefd20544ff1adf03904f7e45872956d37bf4a4472 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-1-activate, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, RELEASE=reef-19e2a2b, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731) 2024-09-17T07:21:04.376 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:21:04.408 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:04.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:04.409 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:21:04.510 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:21:04.539 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:04.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:04.540 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:21:04.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:04 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:21:04.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:04 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:21:04.641 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:21:04.674 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:04.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:04.675 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:21:04.775 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:21:04.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:04 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:21:04.806 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:04.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:04.807 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:21:04.908 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:21:04.936 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:04.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:04.937 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:21:05.038 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:21:05.073 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:05.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:05.074 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:21:05.165 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 55 16 121936 121920 138.509 126.438 0.00701366 0.00721603 2024-09-17T07:21:05.176 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:21:05.216 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:05.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:05.217 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:21:05.318 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:21:05.350 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:05.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:05.351 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:21:05.451 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:21:05.481 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:05.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:05.482 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:21:05.583 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:21:05.621 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:05.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:05.622 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:21:05.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:05 smithi100 ceph-mon[29092]: pgmap v883: 158 pgs: 33 active+clean+remapped, 24 active+undersized, 28 active+undersized+degraded, 73 active+clean; 6.9 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 639 B/s rd, 87 MiB/s wr, 1.39k op/s; 84840/338664 objects degraded (25.051%); 55/338664 objects misplaced (0.016%) 2024-09-17T07:21:05.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:05 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 84840/338664 objects degraded (25.051%), 28 pgs degraded, 52 pgs undersized (PG_DEGRADED) 2024-09-17T07:21:05.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:05 smithi100 ceph-mon[39462]: pgmap v883: 158 pgs: 33 active+clean+remapped, 24 active+undersized, 28 active+undersized+degraded, 73 active+clean; 6.9 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 639 B/s rd, 87 MiB/s wr, 1.39k op/s; 84840/338664 objects degraded (25.051%); 55/338664 objects misplaced (0.016%) 2024-09-17T07:21:05.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:05 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 84840/338664 objects degraded (25.051%), 28 pgs degraded, 52 pgs undersized (PG_DEGRADED) 2024-09-17T07:21:05.723 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:21:05.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:05.759 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:05.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:21:05.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:05 smithi135 ceph-mon[33238]: pgmap v883: 158 pgs: 33 active+clean+remapped, 24 active+undersized, 28 active+undersized+degraded, 73 active+clean; 6.9 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 639 B/s rd, 87 MiB/s wr, 1.39k op/s; 84840/338664 objects degraded (25.051%); 55/338664 objects misplaced (0.016%) 2024-09-17T07:21:05.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:05 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 84840/338664 objects degraded (25.051%), 28 pgs degraded, 52 pgs undersized (PG_DEGRADED) 2024-09-17T07:21:05.861 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:21:05.893 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:05.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:05.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:21:05.995 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:21:06.022 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:21:05 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:21:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:21:06.028 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:06.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:06.029 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:21:06.130 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:21:06.163 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:06.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:06.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:21:06.165 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 56 16 123848 123832 138.169 119.5 0.0125696 0.00723404 2024-09-17T07:21:06.266 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:21:06.299 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:06.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:06.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:21:06.401 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:21:06.433 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:06.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:06.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:21:06.535 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:21:06.568 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:06.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:06.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:21:06.669 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:21:06.703 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:06.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:06.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:21:06.805 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:21:06.840 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:06.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:06.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:21:06.941 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:21:06.978 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:06.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:06.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:21:07.080 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:21:07.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:07.121 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:07.121 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:21:07.165 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 57 16 125819 125803 137.906 123.188 0.00474855 0.00724716 2024-09-17T07:21:07.222 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:21:07.257 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:07.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:07.258 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:21:07.359 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:21:07.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:07.395 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:07.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:21:07.425 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:07 smithi100 ceph-mon[39462]: pgmap v884: 158 pgs: 33 active+clean+remapped, 24 active+undersized, 28 active+undersized+degraded, 73 active+clean; 7.3 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1.1 KiB/s rd, 127 MiB/s wr, 2.03k op/s; 90258/359637 objects degraded (25.097%); 55/359637 objects misplaced (0.015%) 2024-09-17T07:21:07.426 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:07 smithi100 ceph-mon[29092]: pgmap v884: 158 pgs: 33 active+clean+remapped, 24 active+undersized, 28 active+undersized+degraded, 73 active+clean; 7.3 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1.1 KiB/s rd, 127 MiB/s wr, 2.03k op/s; 90258/359637 objects degraded (25.097%); 55/359637 objects misplaced (0.015%) 2024-09-17T07:21:07.426 INFO:journalctl@ceph.osd.1.smithi100.stdout:Sep 17 07:21:07 smithi100 podman[185661]: 2024-09-17 07:21:07.132031802 +0000 UTC m=+3.099049299 container remove 5ac64fc37b3cc5f81e13d8fefd20544ff1adf03904f7e45872956d37bf4a4472 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-1-activate, GIT_CLEAN=True, RELEASE=reef-19e2a2b, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731) 2024-09-17T07:21:07.494 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:21:07.496 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:21:07.501 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:21:07.501 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:21:07.501 INFO:teuthology.orchestra.run.smithi100.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-17T07:21:07.522 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:07.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:07.524 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:21:07.537 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:21:07.537 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:21:07.570 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:21:07.571 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:21:07.601 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:21:07.601 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:21:07.625 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:21:07.635 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:21:07.635 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:21:07.661 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:07.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:07.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:21:07.665 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:21:07.665 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:21:07.693 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:21:07.694 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:21:07.763 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:21:07.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:07 smithi135 ceph-mon[33238]: pgmap v884: 158 pgs: 33 active+clean+remapped, 24 active+undersized, 28 active+undersized+degraded, 73 active+clean; 7.3 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1.1 KiB/s rd, 127 MiB/s wr, 2.03k op/s; 90258/359637 objects degraded (25.097%); 55/359637 objects misplaced (0.015%) 2024-09-17T07:21:07.798 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:07.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:07.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:21:07.900 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:21:07.938 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:07.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:07.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:21:08.040 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:21:08.083 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:08.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:08.084 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:21:08.165 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 58 15 127726 127711 137.584 119.25 0.0040249 0.00726458 2024-09-17T07:21:08.185 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:21:08.217 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:08.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:08.218 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:21:08.319 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:21:08.352 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:08.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:08.353 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:21:08.454 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:21:08.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:08.494 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:08.494 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:21:08.595 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:21:08.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:08 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/3700948603' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:21:08.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:08 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/3700948603' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:21:08.633 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:08.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:08.634 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:21:08.694 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:08 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/3700948603' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:21:08.735 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:21:08.769 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:08.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:08.770 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:21:08.870 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:21:08.907 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:08.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:08.908 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:21:09.009 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:21:09.041 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:09.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:09.042 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:21:09.066 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:21:09.067 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:21:09.143 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:21:09.165 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 59 16 129726 129710 137.37 124.938 0.00523574 0.00727596 2024-09-17T07:21:09.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:09.174 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:09.175 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:21:09.275 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:21:09.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:09.309 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:09.309 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:21:09.409 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:21:09.443 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:09.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:09.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:21:09.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:09 smithi135 ceph-mon[33238]: pgmap v885: 158 pgs: 33 active+clean+remapped, 24 active+undersized, 28 active+undersized+degraded, 73 active+clean; 7.3 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1023 B/s rd, 113 MiB/s wr, 1.81k op/s; 90258/359637 objects degraded (25.097%); 55/359637 objects misplaced (0.015%) 2024-09-17T07:21:09.543 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:21:09 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:21:09.545 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:21:09.579 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:09.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:09.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:21:09.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:09 smithi100 ceph-mon[29092]: pgmap v885: 158 pgs: 33 active+clean+remapped, 24 active+undersized, 28 active+undersized+degraded, 73 active+clean; 7.3 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1023 B/s rd, 113 MiB/s wr, 1.81k op/s; 90258/359637 objects degraded (25.097%); 55/359637 objects misplaced (0.015%) 2024-09-17T07:21:09.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:09 smithi100 ceph-mon[39462]: pgmap v885: 158 pgs: 33 active+clean+remapped, 24 active+undersized, 28 active+undersized+degraded, 73 active+clean; 7.3 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1023 B/s rd, 113 MiB/s wr, 1.81k op/s; 90258/359637 objects degraded (25.097%); 55/359637 objects misplaced (0.015%) 2024-09-17T07:21:09.627 INFO:journalctl@ceph.osd.1.smithi100.stdout:Sep 17 07:21:09 smithi100 podman[186007]: 2024-09-17 07:21:09.212329807 +0000 UTC m=+1.675462263 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9 2024-09-17T07:21:09.681 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:21:09.715 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:09.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:09.716 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:21:09.817 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:21:09.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:09.854 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:09.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:21:09.954 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:21:09.987 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:09.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:09.988 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:21:10.089 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:21:10.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:10.122 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:10.122 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:21:10.166 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:2024-09-17T07:21:10.166306+0000 min lat: 0.00105244 max lat: 0.145857 avg lat: 0.0072894 2024-09-17T07:21:10.166 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-17T07:21:10.166 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 60 16 131682 131666 137.118 122.25 0.00648735 0.0072894 2024-09-17T07:21:10.223 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:21:10.254 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:10.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:10.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:21:10.356 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:21:10.376 INFO:journalctl@ceph.osd.1.smithi100.stdout:Sep 17 07:21:10 smithi100 podman[186007]: 2024-09-17 07:21:10.095354537 +0000 UTC m=+2.558486972 container create 715bd0bb4a3185ea6b3923aa175c569ebab6468193f6ec7934b73205266e7e5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-1, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, RELEASE=reef-19e2a2b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731) 2024-09-17T07:21:10.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:10.393 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:10.393 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:21:10.490 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-17T07:21:10.494 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:21:10.572 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:10.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:10.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:21:10.673 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:21:10.697 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:10 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:21:10.698 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:10 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 90258/359637 objects degraded (25.097%), 28 pgs degraded, 52 pgs undersized (PG_DEGRADED) 2024-09-17T07:21:10.698 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:10 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:21:10.698 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:10 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 90258/359637 objects degraded (25.097%), 28 pgs degraded, 52 pgs undersized (PG_DEGRADED) 2024-09-17T07:21:10.698 INFO:journalctl@ceph.osd.1.smithi100.stdout:Sep 17 07:21:10 smithi100 podman[186007]: 2024-09-17 07:21:10.382076943 +0000 UTC m=+2.845209395 container init 715bd0bb4a3185ea6b3923aa175c569ebab6468193f6ec7934b73205266e7e5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-1, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, RELEASE=reef-19e2a2b, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.37.2, ceph=True, maintainer=Guillaume Abrioux ) 2024-09-17T07:21:10.698 INFO:journalctl@ceph.osd.1.smithi100.stdout:Sep 17 07:21:10 smithi100 podman[186007]: 2024-09-17 07:21:10.388800721 +0000 UTC m=+2.851933165 container start 715bd0bb4a3185ea6b3923aa175c569ebab6468193f6ec7934b73205266e7e5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, RELEASE=reef-19e2a2b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-09-17T07:21:10.698 INFO:journalctl@ceph.osd.1.smithi100.stdout:Sep 17 07:21:10 smithi100 ceph-osd[186224]: -- 172.21.15.100:0/2033377170 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55ced1a1b600 con 0x55ced1b7c000 2024-09-17T07:21:10.699 INFO:journalctl@ceph.osd.1.smithi100.stdout:Sep 17 07:21:10 smithi100 ceph-osd[186224]: -- 172.21.15.100:0/2033377170 <== mon.2 v2:172.21.15.100:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55ced1a091e0 con 0x55ced1b7c000 2024-09-17T07:21:10.699 INFO:journalctl@ceph.osd.1.smithi100.stdout:Sep 17 07:21:10 smithi100 bash[186007]: 715bd0bb4a3185ea6b3923aa175c569ebab6468193f6ec7934b73205266e7e5c 2024-09-17T07:21:10.699 INFO:journalctl@ceph.osd.1.smithi100.stdout:Sep 17 07:21:10 smithi100 systemd[1]: Started Ceph osd.1 for e848267c-74c1-11ef-bceb-c7b262605968. 2024-09-17T07:21:10.712 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:10.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:10.714 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:21:10.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:10 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:21:10.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:10 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 90258/359637 objects degraded (25.097%), 28 pgs degraded, 52 pgs undersized (PG_DEGRADED) 2024-09-17T07:21:10.814 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:21:10.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:10.850 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:10.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:21:10.951 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:21:10.972 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:10.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:10.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:21:11.068 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:21:11.075 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:21:11.101 INFO:journalctl@ceph.osd.1.smithi100.stdout:Sep 17 07:21:10 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-1[186219]: 2024-09-17T07:21:10.695+0000 7fed84ff4640 -1 Falling back to public interface 2024-09-17T07:21:11.136 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:11.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:11.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:21:11.167 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 61 16 133366 133350 136.594 105.25 0.0114089 0.00731709 2024-09-17T07:21:11.238 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:21:11.273 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:11.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:11.274 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:21:11.375 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:21:11.392 INFO:teuthology.orchestra.run.smithi100.stderr:noscrub is unset 2024-09-17T07:21:11.397 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:11.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:11.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:21:11.498 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:21:11.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:11.534 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:11.534 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:21:11.634 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:21:11.665 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:11.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:11.666 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:21:11.767 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:21:11.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:11 smithi135 ceph-mon[33238]: pgmap v886: 158 pgs: 33 active+clean+remapped, 24 active+undersized, 28 active+undersized+degraded, 73 active+clean; 7.8 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 1023 B/s rd, 114 MiB/s wr, 1.83k op/s; 95011/381552 objects degraded (24.901%); 55/381552 objects misplaced (0.014%) 2024-09-17T07:21:11.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:11 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:21:11.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:11 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/490164254' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:21:11.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:11 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:21:11.800 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:11.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:11.801 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:21:11.876 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:11 smithi100 ceph-mon[39462]: pgmap v886: 158 pgs: 33 active+clean+remapped, 24 active+undersized, 28 active+undersized+degraded, 73 active+clean; 7.8 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 1023 B/s rd, 114 MiB/s wr, 1.83k op/s; 95011/381552 objects degraded (24.901%); 55/381552 objects misplaced (0.014%) 2024-09-17T07:21:11.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:11 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:21:11.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:11 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/490164254' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:21:11.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:11 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:21:11.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:11 smithi100 ceph-mon[29092]: pgmap v886: 158 pgs: 33 active+clean+remapped, 24 active+undersized, 28 active+undersized+degraded, 73 active+clean; 7.8 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 1023 B/s rd, 114 MiB/s wr, 1.83k op/s; 95011/381552 objects degraded (24.901%); 55/381552 objects misplaced (0.014%) 2024-09-17T07:21:11.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:11 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:21:11.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:11 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/490164254' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:21:11.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:11 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:21:11.901 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:21:11.932 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:11.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:11.933 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:21:12.033 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:21:12.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:12.063 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:12.063 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:21:12.164 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:21:12.166 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 62 16 135106 135090 136.145 108.75 0.00330218 0.00734145 2024-09-17T07:21:12.201 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:12.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:12.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:21:12.303 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:21:12.340 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:12.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:12.341 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:21:12.442 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:21:12.479 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:12.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:12.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:21:12.581 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:21:12.611 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:12.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:12.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:21:12.712 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:21:12.748 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:12.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:12.749 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:21:12.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:12 smithi135 ceph-mon[33238]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:21:12.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:12 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T07:21:12.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:12 smithi135 ceph-mon[33238]: osdmap e411: 8 total, 7 up, 6 in 2024-09-17T07:21:12.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:12 smithi135 ceph-mon[33238]: 4.1c scrub starts 2024-09-17T07:21:12.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:12 smithi135 ceph-mon[33238]: 4.1c scrub ok 2024-09-17T07:21:12.850 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:21:12.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:12 smithi100 ceph-mon[29092]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:21:12.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:12 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T07:21:12.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:12 smithi100 ceph-mon[29092]: osdmap e411: 8 total, 7 up, 6 in 2024-09-17T07:21:12.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:12 smithi100 ceph-mon[29092]: 4.1c scrub starts 2024-09-17T07:21:12.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:12 smithi100 ceph-mon[29092]: 4.1c scrub ok 2024-09-17T07:21:12.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:12 smithi100 ceph-mon[39462]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:21:12.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:12 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T07:21:12.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:12 smithi100 ceph-mon[39462]: osdmap e411: 8 total, 7 up, 6 in 2024-09-17T07:21:12.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:12 smithi100 ceph-mon[39462]: 4.1c scrub starts 2024-09-17T07:21:12.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:12 smithi100 ceph-mon[39462]: 4.1c scrub ok 2024-09-17T07:21:12.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:12.879 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:12.879 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:21:12.980 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:21:13.022 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:13.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:13.023 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:21:13.124 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:21:13.158 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:13.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:13.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:21:13.166 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 63 16 137048 137032 135.911 121.375 0.0142975 0.00735403 2024-09-17T07:21:13.260 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:21:13.295 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:13.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:13.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:21:13.397 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:21:13.431 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:13.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:13.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:21:13.533 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:21:13.557 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:13.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:13.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:21:13.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:13 smithi100 ceph-mon[29092]: pgmap v888: 158 pgs: 33 active+clean+remapped, 24 active+undersized, 28 active+undersized+degraded, 73 active+clean; 7.9 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 1023 B/s rd, 115 MiB/s wr, 1.84k op/s; 97500/389019 objects degraded (25.063%); 55/389019 objects misplaced (0.014%) 2024-09-17T07:21:13.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:13 smithi100 ceph-mon[29092]: 6.14 scrub starts 2024-09-17T07:21:13.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:13 smithi100 ceph-mon[29092]: 6.14 scrub ok 2024-09-17T07:21:13.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:13 smithi100 ceph-mon[29092]: 1.9 scrub starts 2024-09-17T07:21:13.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:13 smithi100 ceph-mon[29092]: 1.9 scrub ok 2024-09-17T07:21:13.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:13 smithi100 ceph-mon[29092]: from='osd.1 v2:172.21.15.100:6805/2894367777' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-17T07:21:13.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:13 smithi100 ceph-mon[39462]: pgmap v888: 158 pgs: 33 active+clean+remapped, 24 active+undersized, 28 active+undersized+degraded, 73 active+clean; 7.9 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 1023 B/s rd, 115 MiB/s wr, 1.84k op/s; 97500/389019 objects degraded (25.063%); 55/389019 objects misplaced (0.014%) 2024-09-17T07:21:13.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:13 smithi100 ceph-mon[39462]: 6.14 scrub starts 2024-09-17T07:21:13.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:13 smithi100 ceph-mon[39462]: 6.14 scrub ok 2024-09-17T07:21:13.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:13 smithi100 ceph-mon[39462]: 1.9 scrub starts 2024-09-17T07:21:13.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:13 smithi100 ceph-mon[39462]: 1.9 scrub ok 2024-09-17T07:21:13.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:13 smithi100 ceph-mon[39462]: from='osd.1 v2:172.21.15.100:6805/2894367777' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-17T07:21:13.628 INFO:journalctl@ceph.osd.1.smithi100.stdout:Sep 17 07:21:13 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-1[186219]: 2024-09-17T07:21:13.247+0000 7fed84ff4640 -1 osd.1 302 log_to_monitors true 2024-09-17T07:21:13.658 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:21:13.681 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:13.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:13.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:21:13.782 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:21:13.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:13 smithi135 ceph-mon[33238]: pgmap v888: 158 pgs: 33 active+clean+remapped, 24 active+undersized, 28 active+undersized+degraded, 73 active+clean; 7.9 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 1023 B/s rd, 115 MiB/s wr, 1.84k op/s; 97500/389019 objects degraded (25.063%); 55/389019 objects misplaced (0.014%) 2024-09-17T07:21:13.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:13 smithi135 ceph-mon[33238]: 6.14 scrub starts 2024-09-17T07:21:13.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:13 smithi135 ceph-mon[33238]: 6.14 scrub ok 2024-09-17T07:21:13.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:13 smithi135 ceph-mon[33238]: 1.9 scrub starts 2024-09-17T07:21:13.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:13 smithi135 ceph-mon[33238]: 1.9 scrub ok 2024-09-17T07:21:13.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:13 smithi135 ceph-mon[33238]: from='osd.1 v2:172.21.15.100:6805/2894367777' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-17T07:21:13.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:13.809 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:13.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:21:13.910 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:21:13.944 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:13.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:13.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:21:14.046 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:21:14.077 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:14.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:14.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:21:14.167 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 64 16 138128 138112 134.842 67.5 0.0100174 0.00741249 2024-09-17T07:21:14.179 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:21:14.212 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:14.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:14.213 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:21:14.313 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:21:14.348 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:14.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:14.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:21:14.450 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:21:14.479 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:14.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:14.480 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:21:14.581 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:21:14.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:14.614 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:14.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:21:14.626 INFO:journalctl@ceph.osd.1.smithi100.stdout:Sep 17 07:21:14 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-1[186219]: 2024-09-17T07:21:14.161+0000 7fed78d96640 -1 osd.1 412 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T07:21:14.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:14 smithi100 ceph-mon[29092]: from='osd.1 v2:172.21.15.100:6805/2894367777' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-17T07:21:14.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:14 smithi100 ceph-mon[29092]: osdmap e412: 8 total, 7 up, 6 in 2024-09-17T07:21:14.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:14 smithi100 ceph-mon[29092]: pgmap v890: 158 pgs: 33 active+clean+remapped, 24 active+undersized, 28 active+undersized+degraded, 73 active+clean; 8.0 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 639 B/s rd, 90 MiB/s wr, 1.44k op/s; 98615/394197 objects degraded (25.017%); 55/394197 objects misplaced (0.014%) 2024-09-17T07:21:14.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:14 smithi100 ceph-mon[29092]: from='osd.1 v2:172.21.15.100:6805/2894367777' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]: dispatch 2024-09-17T07:21:14.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:14 smithi100 ceph-mon[29092]: 2.1e scrub starts 2024-09-17T07:21:14.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:14 smithi100 ceph-mon[29092]: 2.1e scrub ok 2024-09-17T07:21:14.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:14 smithi100 ceph-mon[29092]: from='osd.1 v2:172.21.15.100:6805/2894367777' entity='osd.1' 2024-09-17T07:21:14.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:14 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:21:14.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:14 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:21:14.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:14 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:21:14.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:14 smithi100 ceph-mon[39462]: from='osd.1 v2:172.21.15.100:6805/2894367777' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-17T07:21:14.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:14 smithi100 ceph-mon[39462]: osdmap e412: 8 total, 7 up, 6 in 2024-09-17T07:21:14.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:14 smithi100 ceph-mon[39462]: pgmap v890: 158 pgs: 33 active+clean+remapped, 24 active+undersized, 28 active+undersized+degraded, 73 active+clean; 8.0 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 639 B/s rd, 90 MiB/s wr, 1.44k op/s; 98615/394197 objects degraded (25.017%); 55/394197 objects misplaced (0.014%) 2024-09-17T07:21:14.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:14 smithi100 ceph-mon[39462]: from='osd.1 v2:172.21.15.100:6805/2894367777' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]: dispatch 2024-09-17T07:21:14.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:14 smithi100 ceph-mon[39462]: 2.1e scrub starts 2024-09-17T07:21:14.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:14 smithi100 ceph-mon[39462]: 2.1e scrub ok 2024-09-17T07:21:14.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:14 smithi100 ceph-mon[39462]: from='osd.1 v2:172.21.15.100:6805/2894367777' entity='osd.1' 2024-09-17T07:21:14.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:14 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:21:14.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:14 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:21:14.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:14 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:21:14.715 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:21:14.738 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:14.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:14.739 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:21:14.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:14 smithi135 ceph-mon[33238]: from='osd.1 v2:172.21.15.100:6805/2894367777' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-17T07:21:14.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:14 smithi135 ceph-mon[33238]: osdmap e412: 8 total, 7 up, 6 in 2024-09-17T07:21:14.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:14 smithi135 ceph-mon[33238]: pgmap v890: 158 pgs: 33 active+clean+remapped, 24 active+undersized, 28 active+undersized+degraded, 73 active+clean; 8.0 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 639 B/s rd, 90 MiB/s wr, 1.44k op/s; 98615/394197 objects degraded (25.017%); 55/394197 objects misplaced (0.014%) 2024-09-17T07:21:14.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:14 smithi135 ceph-mon[33238]: from='osd.1 v2:172.21.15.100:6805/2894367777' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]: dispatch 2024-09-17T07:21:14.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:14 smithi135 ceph-mon[33238]: 2.1e scrub starts 2024-09-17T07:21:14.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:14 smithi135 ceph-mon[33238]: 2.1e scrub ok 2024-09-17T07:21:14.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:14 smithi135 ceph-mon[33238]: from='osd.1 v2:172.21.15.100:6805/2894367777' entity='osd.1' 2024-09-17T07:21:14.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:14 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:21:14.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:14 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:21:14.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:14 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:21:14.840 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:21:14.882 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:14.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:14.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:21:14.903 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:21:14.916 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:21:14.984 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:21:15.020 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:15.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:15.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:21:15.122 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:21:15.144 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:15.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:15.145 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:21:15.166 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 65 16 139726 139710 134.304 99.875 0.00505476 0.00742948 2024-09-17T07:21:15.246 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:21:15.269 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:15.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:15.270 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:21:15.371 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:21:15.392 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:15.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:15.393 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:21:15.494 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:21:15.518 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:15.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:15.519 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:21:15.620 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:21:15.642 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:15.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:15.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:21:15.696 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:15 smithi100 ceph-mon[39462]: 5.3 scrub starts 2024-09-17T07:21:15.696 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:15 smithi100 ceph-mon[39462]: 5.3 scrub ok 2024-09-17T07:21:15.696 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:15 smithi100 ceph-mon[39462]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T07:21:15.696 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:15 smithi100 ceph-mon[39462]: osd.1 v2:172.21.15.100:6805/2894367777 boot 2024-09-17T07:21:15.696 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:15 smithi100 ceph-mon[39462]: osdmap e413: 8 total, 8 up, 6 in 2024-09-17T07:21:15.696 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:15 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T07:21:15.696 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:15 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 98615/394197 objects degraded (25.017%), 28 pgs degraded, 52 pgs undersized (PG_DEGRADED) 2024-09-17T07:21:15.696 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:15 smithi100 ceph-mon[39462]: osdmap e414: 8 total, 8 up, 6 in 2024-09-17T07:21:15.696 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:15 smithi100 ceph-mon[29092]: 5.3 scrub starts 2024-09-17T07:21:15.697 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:15 smithi100 ceph-mon[29092]: 5.3 scrub ok 2024-09-17T07:21:15.697 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:15 smithi100 ceph-mon[29092]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T07:21:15.697 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:15 smithi100 ceph-mon[29092]: osd.1 v2:172.21.15.100:6805/2894367777 boot 2024-09-17T07:21:15.697 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:15 smithi100 ceph-mon[29092]: osdmap e413: 8 total, 8 up, 6 in 2024-09-17T07:21:15.697 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:15 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T07:21:15.697 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:15 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 98615/394197 objects degraded (25.017%), 28 pgs degraded, 52 pgs undersized (PG_DEGRADED) 2024-09-17T07:21:15.697 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:15 smithi100 ceph-mon[29092]: osdmap e414: 8 total, 8 up, 6 in 2024-09-17T07:21:15.712 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-17T07:21:15.712 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-17T07:21:15.713 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-17T07:21:15.713 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-17T07:21:15.744 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:21:15.764 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:15.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:15.765 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:21:15.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:15 smithi135 ceph-mon[33238]: 5.3 scrub starts 2024-09-17T07:21:15.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:15 smithi135 ceph-mon[33238]: 5.3 scrub ok 2024-09-17T07:21:15.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:15 smithi135 ceph-mon[33238]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T07:21:15.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:15 smithi135 ceph-mon[33238]: osd.1 v2:172.21.15.100:6805/2894367777 boot 2024-09-17T07:21:15.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:15 smithi135 ceph-mon[33238]: osdmap e413: 8 total, 8 up, 6 in 2024-09-17T07:21:15.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:15 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T07:21:15.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:15 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 98615/394197 objects degraded (25.017%), 28 pgs degraded, 52 pgs undersized (PG_DEGRADED) 2024-09-17T07:21:15.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:15 smithi135 ceph-mon[33238]: osdmap e414: 8 total, 8 up, 6 in 2024-09-17T07:21:15.866 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:21:15.892 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:15.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:15.893 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:21:15.994 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:21:16.016 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:16.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:16.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:21:16.117 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:21:16.126 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:21:15 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:21:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:21:16.143 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:16.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:16.144 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:21:16.166 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 66 16 139726 139710 132.269 0 - 0.00742948 2024-09-17T07:21:16.245 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:21:16.271 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:16.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:16.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:21:16.373 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:21:16.396 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:16.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:16.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:21:16.498 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:21:16.527 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:16.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:16.528 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:21:16.629 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:21:16.669 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:16.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:16.670 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:21:16.771 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:21:16.774 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-17T07:21:16.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:16 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:21:16.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:16 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:21:16.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:16 smithi135 ceph-mon[33238]: pgmap v893: 158 pgs: 33 active+clean+remapped, 24 active+undersized, 28 active+undersized+degraded, 73 active+clean; 8.4 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 134 MiB/s wr, 2.14k op/s; 103979/414942 objects degraded (25.059%); 55/414942 objects misplaced (0.013%) 2024-09-17T07:21:16.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:16 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:21:16.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:16 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:21:16.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:16 smithi135 ceph-mon[33238]: 4.13 scrub starts 2024-09-17T07:21:16.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:16 smithi135 ceph-mon[33238]: 4.13 scrub ok 2024-09-17T07:21:16.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:16 smithi135 ceph-mon[33238]: 11.e starting backfill to osd.1 from (0'0,0'0] MAX to 412'8852 2024-09-17T07:21:16.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:16 smithi135 ceph-mon[33238]: osdmap e415: 8 total, 8 up, 6 in 2024-09-17T07:21:16.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:16 smithi135 ceph-mon[33238]: 11.7 starting backfill to osd.1 from (0'0,0'0] MAX to 412'8717 2024-09-17T07:21:16.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:16 smithi135 ceph-mon[33238]: 2.2 scrub starts 2024-09-17T07:21:16.795 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:16.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:16.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:21:16.803 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:16 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:21:16.803 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:16 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:21:16.803 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:16 smithi100 ceph-mon[29092]: pgmap v893: 158 pgs: 33 active+clean+remapped, 24 active+undersized, 28 active+undersized+degraded, 73 active+clean; 8.4 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 134 MiB/s wr, 2.14k op/s; 103979/414942 objects degraded (25.059%); 55/414942 objects misplaced (0.013%) 2024-09-17T07:21:16.804 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:16 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:21:16.804 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:16 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:21:16.804 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:16 smithi100 ceph-mon[29092]: 4.13 scrub starts 2024-09-17T07:21:16.804 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:16 smithi100 ceph-mon[29092]: 4.13 scrub ok 2024-09-17T07:21:16.804 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:16 smithi100 ceph-mon[29092]: 11.e starting backfill to osd.1 from (0'0,0'0] MAX to 412'8852 2024-09-17T07:21:16.804 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:16 smithi100 ceph-mon[29092]: osdmap e415: 8 total, 8 up, 6 in 2024-09-17T07:21:16.804 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:16 smithi100 ceph-mon[29092]: 11.7 starting backfill to osd.1 from (0'0,0'0] MAX to 412'8717 2024-09-17T07:21:16.804 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:16 smithi100 ceph-mon[29092]: 2.2 scrub starts 2024-09-17T07:21:16.805 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:16 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:21:16.805 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:16 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:21:16.805 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:16 smithi100 ceph-mon[39462]: pgmap v893: 158 pgs: 33 active+clean+remapped, 24 active+undersized, 28 active+undersized+degraded, 73 active+clean; 8.4 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 134 MiB/s wr, 2.14k op/s; 103979/414942 objects degraded (25.059%); 55/414942 objects misplaced (0.013%) 2024-09-17T07:21:16.805 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:16 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:21:16.805 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:16 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:21:16.805 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:16 smithi100 ceph-mon[39462]: 4.13 scrub starts 2024-09-17T07:21:16.805 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:16 smithi100 ceph-mon[39462]: 4.13 scrub ok 2024-09-17T07:21:16.805 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:16 smithi100 ceph-mon[39462]: 11.e starting backfill to osd.1 from (0'0,0'0] MAX to 412'8852 2024-09-17T07:21:16.806 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:16 smithi100 ceph-mon[39462]: osdmap e415: 8 total, 8 up, 6 in 2024-09-17T07:21:16.806 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:16 smithi100 ceph-mon[39462]: 11.7 starting backfill to osd.1 from (0'0,0'0] MAX to 412'8717 2024-09-17T07:21:16.806 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:16 smithi100 ceph-mon[39462]: 2.2 scrub starts 2024-09-17T07:21:16.897 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:21:16.931 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:16.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:16.932 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:21:17.033 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:21:17.061 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:17.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:17.062 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:21:17.163 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:21:17.167 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 67 16 140654 140638 131.161 29 0.00945779 0.00762039 2024-09-17T07:21:17.189 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:17.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:17.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:21:17.291 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:21:17.326 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:17.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:17.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:21:17.428 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:21:17.461 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:17.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:17.462 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:21:17.563 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:21:17.589 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-17T07:21:17.590 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:17.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:17.592 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:21:17.693 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:21:17.721 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:17.721 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:17 smithi100 ceph-mon[29092]: 2.10 scrub starts 2024-09-17T07:21:17.721 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:17 smithi100 ceph-mon[29092]: 2.10 scrub ok 2024-09-17T07:21:17.721 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:17 smithi100 ceph-mon[29092]: 6.1f scrub starts 2024-09-17T07:21:17.722 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:17 smithi100 ceph-mon[29092]: 6.1f scrub ok 2024-09-17T07:21:17.722 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:17 smithi100 ceph-mon[29092]: 11.c starting backfill to osd.1 from (0'0,0'0] MAX to 412'8674 2024-09-17T07:21:17.722 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:17 smithi100 ceph-mon[29092]: 11.4 starting backfill to osd.1 from (0'0,0'0] MAX to 412'8757 2024-09-17T07:21:17.722 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:17 smithi100 ceph-mon[29092]: 11.a starting backfill to osd.1 from (0'0,0'0] MAX to 412'8588 2024-09-17T07:21:17.722 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:17 smithi100 ceph-mon[29092]: 11.2 starting backfill to osd.1 from (0'0,0'0] MAX to 412'8788 2024-09-17T07:21:17.722 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:17 smithi100 ceph-mon[29092]: 11.b starting backfill to osd.1 from (0'0,0'0] MAX to 412'8618 2024-09-17T07:21:17.722 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:17 smithi100 ceph-mon[29092]: 11.f starting backfill to osd.1 from (0'0,0'0] MAX to 412'8837 2024-09-17T07:21:17.722 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:17 smithi100 ceph-mon[29092]: 11.9 starting backfill to osd.1 from (0'0,0'0] MAX to 412'8777 2024-09-17T07:21:17.722 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:17 smithi100 ceph-mon[29092]: 11.0 starting backfill to osd.1 from (0'0,0'0] MAX to 412'8888 2024-09-17T07:21:17.722 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:17 smithi100 ceph-mon[29092]: 11.3 starting backfill to osd.1 from (0'0,0'0] MAX to 412'8788 2024-09-17T07:21:17.722 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:17 smithi100 ceph-mon[29092]: 11.6 starting backfill to osd.1 from (0'0,0'0] MAX to 412'8738 2024-09-17T07:21:17.722 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:17 smithi100 ceph-mon[29092]: osdmap e416: 8 total, 8 up, 6 in 2024-09-17T07:21:17.723 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:17 smithi100 ceph-mon[39462]: 2.10 scrub starts 2024-09-17T07:21:17.723 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:17 smithi100 ceph-mon[39462]: 2.10 scrub ok 2024-09-17T07:21:17.723 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:17 smithi100 ceph-mon[39462]: 6.1f scrub starts 2024-09-17T07:21:17.723 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:17 smithi100 ceph-mon[39462]: 6.1f scrub ok 2024-09-17T07:21:17.723 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:17 smithi100 ceph-mon[39462]: 11.c starting backfill to osd.1 from (0'0,0'0] MAX to 412'8674 2024-09-17T07:21:17.723 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:17 smithi100 ceph-mon[39462]: 11.4 starting backfill to osd.1 from (0'0,0'0] MAX to 412'8757 2024-09-17T07:21:17.723 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:17 smithi100 ceph-mon[39462]: 11.a starting backfill to osd.1 from (0'0,0'0] MAX to 412'8588 2024-09-17T07:21:17.723 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:17 smithi100 ceph-mon[39462]: 11.2 starting backfill to osd.1 from (0'0,0'0] MAX to 412'8788 2024-09-17T07:21:17.723 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:17 smithi100 ceph-mon[39462]: 11.b starting backfill to osd.1 from (0'0,0'0] MAX to 412'8618 2024-09-17T07:21:17.724 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:17 smithi100 ceph-mon[39462]: 11.f starting backfill to osd.1 from (0'0,0'0] MAX to 412'8837 2024-09-17T07:21:17.724 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:17 smithi100 ceph-mon[39462]: 11.9 starting backfill to osd.1 from (0'0,0'0] MAX to 412'8777 2024-09-17T07:21:17.724 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:17 smithi100 ceph-mon[39462]: 11.0 starting backfill to osd.1 from (0'0,0'0] MAX to 412'8888 2024-09-17T07:21:17.724 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:17 smithi100 ceph-mon[39462]: 11.3 starting backfill to osd.1 from (0'0,0'0] MAX to 412'8788 2024-09-17T07:21:17.724 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:17 smithi100 ceph-mon[39462]: 11.6 starting backfill to osd.1 from (0'0,0'0] MAX to 412'8738 2024-09-17T07:21:17.724 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:17 smithi100 ceph-mon[39462]: osdmap e416: 8 total, 8 up, 6 in 2024-09-17T07:21:17.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:17.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:21:17.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:17 smithi135 ceph-mon[33238]: 2.10 scrub starts 2024-09-17T07:21:17.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:17 smithi135 ceph-mon[33238]: 2.10 scrub ok 2024-09-17T07:21:17.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:17 smithi135 ceph-mon[33238]: 6.1f scrub starts 2024-09-17T07:21:17.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:17 smithi135 ceph-mon[33238]: 6.1f scrub ok 2024-09-17T07:21:17.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:17 smithi135 ceph-mon[33238]: 11.c starting backfill to osd.1 from (0'0,0'0] MAX to 412'8674 2024-09-17T07:21:17.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:17 smithi135 ceph-mon[33238]: 11.4 starting backfill to osd.1 from (0'0,0'0] MAX to 412'8757 2024-09-17T07:21:17.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:17 smithi135 ceph-mon[33238]: 11.a starting backfill to osd.1 from (0'0,0'0] MAX to 412'8588 2024-09-17T07:21:17.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:17 smithi135 ceph-mon[33238]: 11.2 starting backfill to osd.1 from (0'0,0'0] MAX to 412'8788 2024-09-17T07:21:17.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:17 smithi135 ceph-mon[33238]: 11.b starting backfill to osd.1 from (0'0,0'0] MAX to 412'8618 2024-09-17T07:21:17.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:17 smithi135 ceph-mon[33238]: 11.f starting backfill to osd.1 from (0'0,0'0] MAX to 412'8837 2024-09-17T07:21:17.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:17 smithi135 ceph-mon[33238]: 11.9 starting backfill to osd.1 from (0'0,0'0] MAX to 412'8777 2024-09-17T07:21:17.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:17 smithi135 ceph-mon[33238]: 11.0 starting backfill to osd.1 from (0'0,0'0] MAX to 412'8888 2024-09-17T07:21:17.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:17 smithi135 ceph-mon[33238]: 11.3 starting backfill to osd.1 from (0'0,0'0] MAX to 412'8788 2024-09-17T07:21:17.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:17 smithi135 ceph-mon[33238]: 11.6 starting backfill to osd.1 from (0'0,0'0] MAX to 412'8738 2024-09-17T07:21:17.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:17 smithi135 ceph-mon[33238]: osdmap e416: 8 total, 8 up, 6 in 2024-09-17T07:21:17.825 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:21:17.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:17.857 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:17.858 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:21:17.959 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:21:17.991 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:17.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:17.992 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:21:18.093 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:21:18.127 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:18.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:18.128 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:21:18.167 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 68 15 142079 142064 130.543 89.125 0.0152797 0.00765672 2024-09-17T07:21:18.229 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:21:18.265 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:18.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:18.266 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:21:18.368 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:21:18.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:18.410 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:18.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:21:18.512 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:21:18.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:18.554 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:18.554 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:21:18.655 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:21:18.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:18.694 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:18.694 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:21:18.795 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:21:18.825 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:18.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:18.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:21:18.927 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:21:18.969 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:18.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:18.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:21:19.070 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:21:19.106 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:19.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:19.107 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:21:19.167 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 69 16 143550 143534 129.982 91.875 0.00805864 0.00768929 2024-09-17T07:21:19.208 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:21:19.241 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:19.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:19.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:21:19.343 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:21:19.371 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:19.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:19.372 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:21:19.374 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:21:19.377 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:21:19.377 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:21:19.413 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1: {} 2024-09-17T07:21:19.413 INFO:teuthology.orchestra.run.smithi100.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:21:19.446 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:21:19.446 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:21:19.473 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:21:19.482 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:21:19.482 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:21:19.522 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:21:19.522 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:21:19.523 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:19.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:19.524 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:21:19.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:19 smithi135 ceph-mon[33238]: pgmap v896: 158 pgs: 33 active+clean+remapped, 24 active+undersized, 28 active+undersized+degraded, 73 active+clean; 8.4 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 108 MiB/s wr, 1.72k op/s; 103979/414942 objects degraded (25.059%); 55/414942 objects misplaced (0.013%) 2024-09-17T07:21:19.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:19 smithi135 ceph-mon[33238]: osdmap e417: 8 total, 8 up, 6 in 2024-09-17T07:21:19.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:19 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:21:19.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:19 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:21:19.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:19 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/3888296446' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:21:19.544 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:21:19 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:21:19.554 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:21:19.554 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:21:19.555 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:19 smithi100 ceph-mon[29092]: pgmap v896: 158 pgs: 33 active+clean+remapped, 24 active+undersized, 28 active+undersized+degraded, 73 active+clean; 8.4 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 108 MiB/s wr, 1.72k op/s; 103979/414942 objects degraded (25.059%); 55/414942 objects misplaced (0.013%) 2024-09-17T07:21:19.555 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:19 smithi100 ceph-mon[29092]: osdmap e417: 8 total, 8 up, 6 in 2024-09-17T07:21:19.555 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:19 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:21:19.555 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:19 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:21:19.555 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:19 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/3888296446' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:21:19.555 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:19 smithi100 ceph-mon[39462]: pgmap v896: 158 pgs: 33 active+clean+remapped, 24 active+undersized, 28 active+undersized+degraded, 73 active+clean; 8.4 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 108 MiB/s wr, 1.72k op/s; 103979/414942 objects degraded (25.059%); 55/414942 objects misplaced (0.013%) 2024-09-17T07:21:19.555 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:19 smithi100 ceph-mon[39462]: osdmap e417: 8 total, 8 up, 6 in 2024-09-17T07:21:19.555 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:19 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:21:19.555 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:19 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:21:19.556 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:19 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/3888296446' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:21:19.586 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:21:19.586 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:21:19.619 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:21:19.619 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:21:19.625 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:21:19.658 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:19.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:19.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:21:19.760 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:21:19.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:19.799 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:19.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:21:19.900 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:21:19.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:19.936 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:19.937 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:21:20.037 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:21:20.071 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:20.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:20.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:21:20.167 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 70 16 144967 144951 129.39 88.5625 0.00586576 0.00772489 2024-09-17T07:21:20.174 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:21:20.211 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:20.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:20.212 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:21:20.313 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:21:20.352 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:20.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:20.353 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:21:20.454 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:21:20.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:20.488 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:20.488 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:21:20.589 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:21:20.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:20.622 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:20.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:21:20.625 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:20 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:21:20.625 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:20 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 103979/414942 objects degraded (25.059%), 28 pgs degraded, 52 pgs undersized (PG_DEGRADED) 2024-09-17T07:21:20.625 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:20 smithi100 ceph-mon[29092]: osdmap e418: 8 total, 8 up, 6 in 2024-09-17T07:21:20.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:20 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:21:20.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:20 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 103979/414942 objects degraded (25.059%), 28 pgs degraded, 52 pgs undersized (PG_DEGRADED) 2024-09-17T07:21:20.630 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:20 smithi100 ceph-mon[39462]: osdmap e418: 8 total, 8 up, 6 in 2024-09-17T07:21:20.723 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:21:20.751 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:20.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:20.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:21:20.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:20 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:21:20.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:20 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 103979/414942 objects degraded (25.059%), 28 pgs degraded, 52 pgs undersized (PG_DEGRADED) 2024-09-17T07:21:20.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:20 smithi135 ceph-mon[33238]: osdmap e418: 8 total, 8 up, 6 in 2024-09-17T07:21:20.853 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:21:20.892 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:20.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:20.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:21:20.916 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:21:20.994 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:21:21.034 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:21.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:21.036 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:21:21.137 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:21:21.168 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 71 16 146354 146338 128.787 86.6875 0.0124629 0.00776054 2024-09-17T07:21:21.173 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:21.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:21.174 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:21:21.275 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:21:21.305 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:21.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:21.306 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:21:21.407 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:21:21.441 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:21.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:21.442 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:21:21.543 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:21:21.575 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:21.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:21.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:21:21.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:21 smithi100 ceph-mon[29092]: pgmap v898: 158 pgs: 3 active+undersized+degraded+remapped+backfilling, 4 active+undersized+degraded+remapped+backfill_toofull, 7 active+clean+remapped, 5 active+undersized, 6 active+undersized+degraded, 133 active+clean; 8.7 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 56 MiB/s wr, 893 op/s; 105282/425853 objects degraded (24.723%); 21/425853 objects misplaced (0.005%); 14 MiB/s, 221 objects/s recovering 2024-09-17T07:21:21.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:21 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:21:21.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:21 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:21:21.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:21 smithi100 ceph-mon[29092]: 6.e scrub starts 2024-09-17T07:21:21.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:21 smithi100 ceph-mon[29092]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-17T07:21:21.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:21 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:21:21.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:21 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:21:21.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:21 smithi100 ceph-mon[39462]: pgmap v898: 158 pgs: 3 active+undersized+degraded+remapped+backfilling, 4 active+undersized+degraded+remapped+backfill_toofull, 7 active+clean+remapped, 5 active+undersized, 6 active+undersized+degraded, 133 active+clean; 8.7 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 56 MiB/s wr, 893 op/s; 105282/425853 objects degraded (24.723%); 21/425853 objects misplaced (0.005%); 14 MiB/s, 221 objects/s recovering 2024-09-17T07:21:21.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:21 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:21:21.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:21 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:21:21.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:21 smithi100 ceph-mon[39462]: 6.e scrub starts 2024-09-17T07:21:21.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:21 smithi100 ceph-mon[39462]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-17T07:21:21.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:21 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:21:21.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:21 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:21:21.677 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:21:21.708 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:21.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:21.709 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:21:21.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:21 smithi135 ceph-mon[33238]: pgmap v898: 158 pgs: 3 active+undersized+degraded+remapped+backfilling, 4 active+undersized+degraded+remapped+backfill_toofull, 7 active+clean+remapped, 5 active+undersized, 6 active+undersized+degraded, 133 active+clean; 8.7 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 56 MiB/s wr, 893 op/s; 105282/425853 objects degraded (24.723%); 21/425853 objects misplaced (0.005%); 14 MiB/s, 221 objects/s recovering 2024-09-17T07:21:21.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:21 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:21:21.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:21 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:21:21.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:21 smithi135 ceph-mon[33238]: 6.e scrub starts 2024-09-17T07:21:21.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:21 smithi135 ceph-mon[33238]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-17T07:21:21.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:21 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:21:21.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:21 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:21:21.810 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:21:21.849 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:21.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:21.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:21:21.950 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:21:21.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:21.993 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:21.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:21:22.094 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:21:22.147 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:22.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:22.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:21:22.168 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 72 16 147703 147687 128.17 84.3125 0.0109345 0.00779812 2024-09-17T07:21:22.249 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:21:22.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:22.293 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:22.293 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:21:22.394 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:21:22.425 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:22.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:22.426 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:21:22.527 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:21:22.555 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:22.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:22.556 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:21:22.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:22 smithi100 ceph-mon[29092]: osdmap e419: 8 total, 8 up, 6 in 2024-09-17T07:21:22.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:22 smithi100 ceph-mon[29092]: 4.1b scrub starts 2024-09-17T07:21:22.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:22 smithi100 ceph-mon[39462]: osdmap e419: 8 total, 8 up, 6 in 2024-09-17T07:21:22.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:22 smithi100 ceph-mon[39462]: 4.1b scrub starts 2024-09-17T07:21:22.657 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:21:22.663 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:21:22.687 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:22.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:22.688 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:21:22.789 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:21:22.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:22 smithi135 ceph-mon[33238]: osdmap e419: 8 total, 8 up, 6 in 2024-09-17T07:21:22.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:22 smithi135 ceph-mon[33238]: 4.1b scrub starts 2024-09-17T07:21:22.825 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:22.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:22.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:21:22.926 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:21:22.956 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:22.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:22.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:21:23.058 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:21:23.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:23.097 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:23.098 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:21:23.168 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 73 16 148982 148966 127.509 79.9375 0.00746235 0.00783887 2024-09-17T07:21:23.198 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:21:23.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:23.232 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:23.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:21:23.333 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:21:23.377 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:23.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:23.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:21:23.479 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:21:23.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:23.521 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:23.521 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:21:23.533 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:23 smithi100 ceph-mon[29092]: pgmap v901: 158 pgs: 5 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 4 active+undersized+degraded+remapped+backfill_toofull, 146 active+clean; 8.8 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 81 MiB/s wr, 1.30k op/s; 106770/430881 objects degraded (24.779%); 17 MiB/s, 269 objects/s recovering 2024-09-17T07:21:23.533 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:23 smithi100 ceph-mon[39462]: pgmap v901: 158 pgs: 5 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 4 active+undersized+degraded+remapped+backfill_toofull, 146 active+clean; 8.8 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 81 MiB/s wr, 1.30k op/s; 106770/430881 objects degraded (24.779%); 17 MiB/s, 269 objects/s recovering 2024-09-17T07:21:23.622 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:21:23.654 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:23.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:23.655 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:21:23.756 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:21:23.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:23 smithi135 ceph-mon[33238]: pgmap v901: 158 pgs: 5 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 4 active+undersized+degraded+remapped+backfill_toofull, 146 active+clean; 8.8 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 81 MiB/s wr, 1.30k op/s; 106770/430881 objects degraded (24.779%); 17 MiB/s, 269 objects/s recovering 2024-09-17T07:21:23.793 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:23.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:23.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:21:23.896 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:21:23.925 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:23.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:23.926 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:21:24.027 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:21:24.069 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:24.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:24.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:21:24.168 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 74 16 150412 150396 126.994 89.375 0.00939776 0.00787056 2024-09-17T07:21:24.171 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:21:24.204 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:24.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:24.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:21:24.306 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:21:24.340 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:24.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:24.341 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:21:24.442 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:21:24.475 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:24.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:24.476 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:21:24.576 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:21:24.619 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:24.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:24.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:21:24.721 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:21:24.751 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:24.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:24.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:21:24.853 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:21:24.881 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:24.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:24.882 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:21:24.982 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:21:25.011 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:25.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:25.013 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:21:25.077 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-17T07:21:25.077 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:21:25.113 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:21:25.141 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:25.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:25.142 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:21:25.168 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 75 16 151846 151830 126.495 89.625 0.0121777 0.00790118 2024-09-17T07:21:25.243 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:21:25.275 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:25.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:25.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:21:25.377 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:21:25.406 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:25.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:25.407 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:21:25.508 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:21:25.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:25.557 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:25.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:21:25.658 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:21:25.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:25.701 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:25.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:21:25.712 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-17T07:21:25.712 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:21:25.713 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:25 smithi100 ceph-mon[29092]: pgmap v902: 158 pgs: 1 active+clean+scrubbing, 5 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 4 active+undersized+degraded+remapped+backfill_toofull, 145 active+clean; 8.8 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 69 MiB/s wr, 1.10k op/s; 107655/434715 objects degraded (24.765%); 11 MiB/s, 183 objects/s recovering 2024-09-17T07:21:25.713 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:25 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 107655/434715 objects degraded (24.765%), 12 pgs degraded (PG_DEGRADED) 2024-09-17T07:21:25.713 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:25 smithi100 ceph-mon[29092]: osdmap e420: 8 total, 8 up, 6 in 2024-09-17T07:21:25.713 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:25 smithi100 ceph-mon[39462]: pgmap v902: 158 pgs: 1 active+clean+scrubbing, 5 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 4 active+undersized+degraded+remapped+backfill_toofull, 145 active+clean; 8.8 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 69 MiB/s wr, 1.10k op/s; 107655/434715 objects degraded (24.765%); 11 MiB/s, 183 objects/s recovering 2024-09-17T07:21:25.713 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:25 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 107655/434715 objects degraded (24.765%), 12 pgs degraded (PG_DEGRADED) 2024-09-17T07:21:25.713 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:25 smithi100 ceph-mon[39462]: osdmap e420: 8 total, 8 up, 6 in 2024-09-17T07:21:25.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:25 smithi135 ceph-mon[33238]: pgmap v902: 158 pgs: 1 active+clean+scrubbing, 5 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 4 active+undersized+degraded+remapped+backfill_toofull, 145 active+clean; 8.8 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 69 MiB/s wr, 1.10k op/s; 107655/434715 objects degraded (24.765%); 11 MiB/s, 183 objects/s recovering 2024-09-17T07:21:25.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:25 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 107655/434715 objects degraded (24.765%), 12 pgs degraded (PG_DEGRADED) 2024-09-17T07:21:25.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:25 smithi135 ceph-mon[33238]: osdmap e420: 8 total, 8 up, 6 in 2024-09-17T07:21:25.802 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:21:25.844 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:25.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:25.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:21:25.946 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:21:25.980 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:25.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:25.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:21:26.082 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:21:26.114 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:26.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:26.115 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:21:26.126 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:21:25 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:21:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:21:26.169 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 76 16 153191 153175 125.937 84.0625 0.0131285 0.00793626 2024-09-17T07:21:26.215 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:21:26.249 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:26.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:26.250 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:21:26.351 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:21:26.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:26.390 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:26.390 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:21:26.415 INFO:teuthology.orchestra.run.smithi100.stderr:nodeep-scrub is unset 2024-09-17T07:21:26.490 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:21:26.527 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:26.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:26.528 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:21:26.629 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:21:26.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:26.668 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:26.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:21:26.683 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:26 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/3720747374' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:21:26.684 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:26 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:21:26.684 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:26 smithi100 ceph-mon[29092]: 6.1a scrub starts 2024-09-17T07:21:26.684 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:26 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/3720747374' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:21:26.684 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:26 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:21:26.684 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:26 smithi100 ceph-mon[39462]: 6.1a scrub starts 2024-09-17T07:21:26.769 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:21:26.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:26 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/3720747374' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:21:26.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:26 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:21:26.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:26 smithi135 ceph-mon[33238]: 6.1a scrub starts 2024-09-17T07:21:26.804 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:26.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:26.805 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:21:26.906 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:21:26.938 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:26.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:26.939 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:21:27.040 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:21:27.078 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:27.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:27.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:21:27.136 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-17T07:21:27.168 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 77 16 154655 154639 125.49 91.5 0.0102565 0.00796482 2024-09-17T07:21:27.180 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:21:27.212 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:27.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:27.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:21:27.314 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:21:27.357 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:27.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:27.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:21:27.459 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:21:27.489 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:27.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:27.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:21:27.591 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:21:27.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:27.637 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:27.637 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:21:27.740 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:21:27.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:27.772 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:27.772 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:21:27.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:27 smithi135 ceph-mon[33238]: pgmap v904: 158 pgs: 1 active+clean+scrubbing, 5 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 4 active+undersized+degraded+remapped+backfill_toofull, 145 active+clean; 9.2 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 852 B/s rd, 89 MiB/s wr, 1.42k op/s; 109579/451431 objects degraded (24.274%); 25 MiB/s, 408 objects/s recovering 2024-09-17T07:21:27.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:27 smithi135 ceph-mon[33238]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T07:21:27.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:27 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T07:21:27.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:27 smithi135 ceph-mon[33238]: osdmap e421: 8 total, 8 up, 6 in 2024-09-17T07:21:27.796 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:27 smithi100 ceph-mon[29092]: pgmap v904: 158 pgs: 1 active+clean+scrubbing, 5 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 4 active+undersized+degraded+remapped+backfill_toofull, 145 active+clean; 9.2 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 852 B/s rd, 89 MiB/s wr, 1.42k op/s; 109579/451431 objects degraded (24.274%); 25 MiB/s, 408 objects/s recovering 2024-09-17T07:21:27.796 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:27 smithi100 ceph-mon[29092]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T07:21:27.796 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:27 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T07:21:27.796 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:27 smithi100 ceph-mon[29092]: osdmap e421: 8 total, 8 up, 6 in 2024-09-17T07:21:27.797 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:27 smithi100 ceph-mon[39462]: pgmap v904: 158 pgs: 1 active+clean+scrubbing, 5 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 4 active+undersized+degraded+remapped+backfill_toofull, 145 active+clean; 9.2 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 852 B/s rd, 89 MiB/s wr, 1.42k op/s; 109579/451431 objects degraded (24.274%); 25 MiB/s, 408 objects/s recovering 2024-09-17T07:21:27.797 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:27 smithi100 ceph-mon[39462]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T07:21:27.797 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:27 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T07:21:27.797 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:27 smithi100 ceph-mon[39462]: osdmap e421: 8 total, 8 up, 6 in 2024-09-17T07:21:27.873 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:21:27.902 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:27.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:27.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:21:27.907 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:21:27.907 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:21:28.004 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:21:28.036 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:28.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:28.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:21:28.138 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:21:28.169 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 78 16 156054 156038 125.002 87.4375 0.0102143 0.00799582 2024-09-17T07:21:28.174 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:28.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:28.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:21:28.276 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:21:28.306 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:28.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:28.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:21:28.408 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:21:28.444 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:28.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:28.445 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:21:28.546 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:21:28.580 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:28.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:28.582 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:21:28.683 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:21:28.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:28.721 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:28.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:21:28.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:28 smithi135 ceph-mon[33238]: pgmap v906: 158 pgs: 1 active+clean+scrubbing, 5 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 4 active+undersized+degraded+remapped+backfill_toofull, 145 active+clean; 9.2 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 848 B/s rd, 71 MiB/s wr, 1.14k op/s; 109579/451431 objects degraded (24.274%); 23 MiB/s, 374 objects/s recovering 2024-09-17T07:21:28.823 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:21:28.854 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:28.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:28.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:21:28.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:28 smithi100 ceph-mon[29092]: pgmap v906: 158 pgs: 1 active+clean+scrubbing, 5 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 4 active+undersized+degraded+remapped+backfill_toofull, 145 active+clean; 9.2 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 848 B/s rd, 71 MiB/s wr, 1.14k op/s; 109579/451431 objects degraded (24.274%); 23 MiB/s, 374 objects/s recovering 2024-09-17T07:21:28.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:28 smithi100 ceph-mon[39462]: pgmap v906: 158 pgs: 1 active+clean+scrubbing, 5 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 4 active+undersized+degraded+remapped+backfill_toofull, 145 active+clean; 9.2 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 848 B/s rd, 71 MiB/s wr, 1.14k op/s; 109579/451431 objects degraded (24.274%); 23 MiB/s, 374 objects/s recovering 2024-09-17T07:21:28.956 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:21:28.997 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:28.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:28.998 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:21:29.098 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:21:29.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:29.136 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:29.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:21:29.169 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 79 16 157507 157491 124.569 90.8125 0.0172102 0.00802407 2024-09-17T07:21:29.237 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:21:29.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:29.277 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:29.278 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:21:29.380 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:21:29.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:29.422 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:29.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:21:29.524 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:21:29.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:29 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:21:29.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:29 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:21:29.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:29 smithi135 ceph-mon[33238]: 4.e deep-scrub starts 2024-09-17T07:21:29.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:29 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:21:29.544 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:21:29 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:21:29.555 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:29.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:29.556 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:21:29.657 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:21:29.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:29.690 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:29.690 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:21:29.791 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:21:29.828 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:29.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:29.829 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:21:29.836 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-17T07:21:29.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:29 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:21:29.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:29 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:21:29.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:29 smithi100 ceph-mon[29092]: 4.e deep-scrub starts 2024-09-17T07:21:29.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:29 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:21:29.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:29 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:21:29.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:29 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:21:29.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:29 smithi100 ceph-mon[39462]: 4.e deep-scrub starts 2024-09-17T07:21:29.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:29 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:21:29.930 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:21:29.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:29.954 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:29.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:21:30.055 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:21:30.086 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:30.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:30.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:21:30.170 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:2024-09-17T07:21:30.169748+0000 min lat: 0.00105244 max lat: 1.45955 avg lat: 0.00805515 2024-09-17T07:21:30.171 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-17T07:21:30.171 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 80 16 158889 158873 124.091 86.375 0.0143103 0.00805515 2024-09-17T07:21:30.187 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:21:30.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:30.221 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:30.221 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:21:30.322 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:21:30.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:30.346 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:30.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:21:30.447 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:21:30.513 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:30.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:30.514 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:21:30.615 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:21:30.651 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:30.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:30.652 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:21:30.753 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:21:30.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:30 smithi135 ceph-mon[33238]: 4.10 scrub starts 2024-09-17T07:21:30.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:30 smithi135 ceph-mon[33238]: pgmap v907: 158 pgs: 5 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 4 active+undersized+degraded+remapped+backfill_toofull, 146 active+clean; 9.5 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 1.2 KiB/s rd, 92 MiB/s wr, 1.47k op/s; 110394/466212 objects degraded (23.679%); 35 MiB/s, 553 objects/s recovering 2024-09-17T07:21:30.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:30 smithi135 ceph-mon[33238]: osdmap e422: 8 total, 8 up, 6 in 2024-09-17T07:21:30.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:30 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 110394/466212 objects degraded (23.679%), 12 pgs degraded (PG_DEGRADED) 2024-09-17T07:21:30.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:30.817 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:30.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:21:30.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:30 smithi100 ceph-mon[29092]: 4.10 scrub starts 2024-09-17T07:21:30.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:30 smithi100 ceph-mon[29092]: pgmap v907: 158 pgs: 5 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 4 active+undersized+degraded+remapped+backfill_toofull, 146 active+clean; 9.5 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 1.2 KiB/s rd, 92 MiB/s wr, 1.47k op/s; 110394/466212 objects degraded (23.679%); 35 MiB/s, 553 objects/s recovering 2024-09-17T07:21:30.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:30 smithi100 ceph-mon[29092]: osdmap e422: 8 total, 8 up, 6 in 2024-09-17T07:21:30.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:30 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 110394/466212 objects degraded (23.679%), 12 pgs degraded (PG_DEGRADED) 2024-09-17T07:21:30.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:30 smithi100 ceph-mon[39462]: 4.10 scrub starts 2024-09-17T07:21:30.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:30 smithi100 ceph-mon[39462]: pgmap v907: 158 pgs: 5 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 4 active+undersized+degraded+remapped+backfill_toofull, 146 active+clean; 9.5 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 1.2 KiB/s rd, 92 MiB/s wr, 1.47k op/s; 110394/466212 objects degraded (23.679%); 35 MiB/s, 553 objects/s recovering 2024-09-17T07:21:30.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:30 smithi100 ceph-mon[39462]: osdmap e422: 8 total, 8 up, 6 in 2024-09-17T07:21:30.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:30 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 110394/466212 objects degraded (23.679%), 12 pgs degraded (PG_DEGRADED) 2024-09-17T07:21:30.919 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:21:30.954 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:30.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:30.956 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:21:31.057 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:21:31.096 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:31.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:31.097 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:21:31.169 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 81 16 160230 160214 123.594 83.8125 0.0143216 0.0080873 2024-09-17T07:21:31.198 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:21:31.208 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6] out_osds: [3, 7] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 6, 7, 1] 2024-09-17T07:21:31.209 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T07:21:31.209 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_4 2024-09-17T07:21:31.209 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T07:21:31.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:31.236 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:31.237 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:21:31.338 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:21:31.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:31.401 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:31.401 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:21:31.502 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:21:31.537 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:31.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:31.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:21:31.638 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:21:31.669 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:31.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:31.670 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:21:31.771 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:21:31.809 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:31.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:31.810 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:21:31.911 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:21:31.935 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:31.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:31.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:21:32.040 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:21:32.081 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:32.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:32.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:21:32.169 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 82 16 161403 161387 122.98 73.3125 0.00956651 0.00812772 2024-09-17T07:21:32.183 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:21:32.213 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:32.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:32.214 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:21:32.315 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:21:32.352 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:32.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:32.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:21:32.454 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:21:32.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:32.492 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:32.492 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:21:32.593 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:21:32.625 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:32.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:32.626 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:21:32.727 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:21:32.781 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:32.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:32.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:21:32.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:32 smithi135 ceph-mon[33238]: 2.c scrub starts 2024-09-17T07:21:32.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:32 smithi135 ceph-mon[33238]: pgmap v909: 158 pgs: 5 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 4 active+undersized+degraded+remapped+backfill_toofull, 146 active+clean; 9.6 GiB data, 27 GiB used, 510 GiB / 536 GiB avail; 839 B/s rd, 73 MiB/s wr, 1.17k op/s; 112567/472779 objects degraded (23.810%); 22 MiB/s, 356 objects/s recovering 2024-09-17T07:21:32.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:32 smithi100 ceph-mon[29092]: 2.c scrub starts 2024-09-17T07:21:32.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:32 smithi100 ceph-mon[29092]: pgmap v909: 158 pgs: 5 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 4 active+undersized+degraded+remapped+backfill_toofull, 146 active+clean; 9.6 GiB data, 27 GiB used, 510 GiB / 536 GiB avail; 839 B/s rd, 73 MiB/s wr, 1.17k op/s; 112567/472779 objects degraded (23.810%); 22 MiB/s, 356 objects/s recovering 2024-09-17T07:21:32.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:32 smithi100 ceph-mon[39462]: 2.c scrub starts 2024-09-17T07:21:32.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:32 smithi100 ceph-mon[39462]: pgmap v909: 158 pgs: 5 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 4 active+undersized+degraded+remapped+backfill_toofull, 146 active+clean; 9.6 GiB data, 27 GiB used, 510 GiB / 536 GiB avail; 839 B/s rd, 73 MiB/s wr, 1.17k op/s; 112567/472779 objects degraded (23.810%); 22 MiB/s, 356 objects/s recovering 2024-09-17T07:21:32.883 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:21:32.908 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:32.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:32.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:21:33.010 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:21:33.044 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:33.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:33.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:21:33.146 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:21:33.169 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 83 16 162605 162589 122.404 75.125 0.00455496 0.00816531 2024-09-17T07:21:33.175 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:33.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:33.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:21:33.277 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:21:33.312 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:33.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:33.314 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:21:33.415 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:21:33.451 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:33.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:33.452 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:21:33.553 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:21:33.583 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:33.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:33.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:21:33.685 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:21:33.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:33.714 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:33.714 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:21:33.815 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:21:33.850 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:33.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:33.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:21:33.952 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:21:33.984 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:33.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:33.985 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:21:34.086 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:21:34.118 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:34.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:34.119 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:21:34.169 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 84 16 163934 163918 121.935 83.0625 0.00777188 0.0081972 2024-09-17T07:21:34.220 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:21:34.260 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:34.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:34.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:21:34.363 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:21:34.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:34.406 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:34.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:21:34.507 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:21:34.540 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:34.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:34.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:21:34.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:34 smithi100 ceph-mon[39462]: pgmap v910: 158 pgs: 1 active+clean+scrubbing, 5 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 4 active+undersized+degraded+remapped+backfill_toofull, 145 active+clean; 9.7 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 639 B/s rd, 66 MiB/s wr, 1.05k op/s; 113425/476643 objects degraded (23.797%); 17 MiB/s, 271 objects/s recovering 2024-09-17T07:21:34.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:34 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:21:34.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:34 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:21:34.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:34 smithi100 ceph-mon[29092]: pgmap v910: 158 pgs: 1 active+clean+scrubbing, 5 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 4 active+undersized+degraded+remapped+backfill_toofull, 145 active+clean; 9.7 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 639 B/s rd, 66 MiB/s wr, 1.05k op/s; 113425/476643 objects degraded (23.797%); 17 MiB/s, 271 objects/s recovering 2024-09-17T07:21:34.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:34 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:21:34.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:34 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:21:34.643 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:21:34.670 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:34.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:34.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:21:34.772 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:21:34.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:34 smithi135 ceph-mon[33238]: pgmap v910: 158 pgs: 1 active+clean+scrubbing, 5 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 4 active+undersized+degraded+remapped+backfill_toofull, 145 active+clean; 9.7 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 639 B/s rd, 66 MiB/s wr, 1.05k op/s; 113425/476643 objects degraded (23.797%); 17 MiB/s, 271 objects/s recovering 2024-09-17T07:21:34.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:34 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:21:34.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:34 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:21:34.799 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:34.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:34.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:21:34.901 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:21:34.933 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:34.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:34.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:21:35.035 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:21:35.073 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:35.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:35.074 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:21:35.169 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 85 16 165230 165214 121.454 81 0.0110959 0.00822985 2024-09-17T07:21:35.175 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:21:35.212 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:35.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:35.213 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:21:35.314 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:21:35.352 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:35.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:35.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:21:35.454 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:21:35.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:35.498 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:35.498 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:21:35.599 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:21:35.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:35 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 113425/476643 objects degraded (23.797%), 12 pgs degraded (PG_DEGRADED) 2024-09-17T07:21:35.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:35 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 113425/476643 objects degraded (23.797%), 12 pgs degraded (PG_DEGRADED) 2024-09-17T07:21:35.635 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:35.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:35.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:21:35.737 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:21:35.770 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:35.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:35.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:21:35.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:35 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 113425/476643 objects degraded (23.797%), 12 pgs degraded (PG_DEGRADED) 2024-09-17T07:21:35.873 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:21:35.909 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:35.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:35.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:21:35.933 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:21:35 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:21:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:21:36.011 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:21:36.045 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:36.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:36.046 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:21:36.147 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:21:36.170 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 86 16 166610 166594 121.043 86.25 0.0111008 0.00825778 2024-09-17T07:21:36.185 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:36.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:36.186 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:21:36.286 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:21:36.321 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:36.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:36.322 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:21:36.423 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:21:36.461 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:36.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:36.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:21:36.564 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:21:36.609 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:36.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:36.610 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:21:36.674 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:36 smithi100 ceph-mon[29092]: pgmap v911: 158 pgs: 1 active+clean+scrubbing, 5 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 4 active+undersized+degraded+remapped+backfill_toofull, 145 active+clean; 10 GiB data, 28 GiB used, 509 GiB / 536 GiB avail; 1.1 KiB/s rd, 94 MiB/s wr, 1.50k op/s; 114830/492084 objects degraded (23.335%); 31 MiB/s, 493 objects/s recovering 2024-09-17T07:21:36.674 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:36 smithi100 ceph-mon[39462]: pgmap v911: 158 pgs: 1 active+clean+scrubbing, 5 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 4 active+undersized+degraded+remapped+backfill_toofull, 145 active+clean; 10 GiB data, 28 GiB used, 509 GiB / 536 GiB avail; 1.1 KiB/s rd, 94 MiB/s wr, 1.50k op/s; 114830/492084 objects degraded (23.335%); 31 MiB/s, 493 objects/s recovering 2024-09-17T07:21:36.711 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:21:36.720 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:21:36.737 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:36.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:36.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:21:36.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:36 smithi135 ceph-mon[33238]: pgmap v911: 158 pgs: 1 active+clean+scrubbing, 5 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 4 active+undersized+degraded+remapped+backfill_toofull, 145 active+clean; 10 GiB data, 28 GiB used, 509 GiB / 536 GiB avail; 1.1 KiB/s rd, 94 MiB/s wr, 1.50k op/s; 114830/492084 objects degraded (23.335%); 31 MiB/s, 493 objects/s recovering 2024-09-17T07:21:36.805 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:21:36.806 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:21:36.838 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1: {} 2024-09-17T07:21:36.839 INFO:teuthology.orchestra.run.smithi100.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:21:36.839 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:21:36.874 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:21:36.874 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:21:36.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:36.887 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:36.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:21:36.905 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:21:36.905 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:21:36.939 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:21:36.939 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:21:36.972 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:21:36.972 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:21:36.988 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:21:37.007 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:21:37.007 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:21:37.012 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:37.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:37.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:21:37.041 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:21:37.041 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:21:37.114 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:21:37.156 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:37.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:37.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:21:37.170 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 87 16 167928 167912 120.599 82.375 0.00689419 0.00828829 2024-09-17T07:21:37.258 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:21:37.286 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:37.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:37.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:21:37.388 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:21:37.416 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:37.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:37.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:21:37.468 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-17T07:21:37.468 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-17T07:21:37.468 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-17T07:21:37.468 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [] 2024-09-17T07:21:37.469 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-17T07:21:37.517 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:21:37.548 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:37.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:37.549 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:21:37.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:37 smithi100 ceph-mon[29092]: 5.b scrub starts 2024-09-17T07:21:37.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:37 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/99144706' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:21:37.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:37 smithi100 ceph-mon[39462]: 5.b scrub starts 2024-09-17T07:21:37.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:37 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/99144706' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:21:37.650 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:21:37.685 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:37.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:37.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:21:37.787 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:21:37.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:37 smithi135 ceph-mon[33238]: 5.b scrub starts 2024-09-17T07:21:37.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:37 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/99144706' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:21:37.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:37.826 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:37.826 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:21:37.911 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:21:37.927 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:21:37.977 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:37.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:37.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:21:38.079 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:21:38.112 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:38.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:38.114 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:21:38.173 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 88 15 169382 169367 120.262 90.9375 0.0153134 0.0083113 2024-09-17T07:21:38.214 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:21:38.248 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:38.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:38.249 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:21:38.350 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:21:38.385 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:38.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:38.386 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:21:38.487 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:21:38.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:38.534 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:38.534 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:21:38.538 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-17T07:21:38.635 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:21:38.669 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:38.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:38.670 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:21:38.752 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:38 smithi100 ceph-mon[29092]: 4.10 scrub starts 2024-09-17T07:21:38.753 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:38 smithi100 ceph-mon[29092]: pgmap v912: 158 pgs: 1 active+clean+scrubbing, 5 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 4 active+undersized+degraded+remapped+backfill_toofull, 145 active+clean; 10 GiB data, 28 GiB used, 509 GiB / 536 GiB avail; 1023 B/s rd, 85 MiB/s wr, 1.35k op/s; 114830/492084 objects degraded (23.335%); 28 MiB/s, 444 objects/s recovering 2024-09-17T07:21:38.753 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:38 smithi100 ceph-mon[29092]: 4.c scrub starts 2024-09-17T07:21:38.753 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:38 smithi100 ceph-mon[39462]: 4.10 scrub starts 2024-09-17T07:21:38.753 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:38 smithi100 ceph-mon[39462]: pgmap v912: 158 pgs: 1 active+clean+scrubbing, 5 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 4 active+undersized+degraded+remapped+backfill_toofull, 145 active+clean; 10 GiB data, 28 GiB used, 509 GiB / 536 GiB avail; 1023 B/s rd, 85 MiB/s wr, 1.35k op/s; 114830/492084 objects degraded (23.335%); 28 MiB/s, 444 objects/s recovering 2024-09-17T07:21:38.753 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:38 smithi100 ceph-mon[39462]: 4.c scrub starts 2024-09-17T07:21:38.771 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:21:38.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:38 smithi135 ceph-mon[33238]: 4.10 scrub starts 2024-09-17T07:21:38.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:38 smithi135 ceph-mon[33238]: pgmap v912: 158 pgs: 1 active+clean+scrubbing, 5 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 4 active+undersized+degraded+remapped+backfill_toofull, 145 active+clean; 10 GiB data, 28 GiB used, 509 GiB / 536 GiB avail; 1023 B/s rd, 85 MiB/s wr, 1.35k op/s; 114830/492084 objects degraded (23.335%); 28 MiB/s, 444 objects/s recovering 2024-09-17T07:21:38.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:38 smithi135 ceph-mon[33238]: 4.c scrub starts 2024-09-17T07:21:38.803 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:38.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:38.804 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:21:38.864 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:21:38.905 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:21:38.945 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:38.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:38.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:21:39.047 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:21:39.084 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:39.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:39.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:21:39.171 INFO:tasks.radosbench.radosbench.0.smithi100.stdout: 89 16 170848 170832 119.939 91.5625 0.0187802 0.00833379 2024-09-17T07:21:39.185 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:21:39.220 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:39.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:39.221 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:21:39.293 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:21:39 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:21:39.321 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:21:39.356 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:39.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:39.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:21:39.457 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:21:39.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:39.499 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:39.499 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:21:39.600 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:21:39.635 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:39.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:39.636 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:21:39.737 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:21:39.759 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:39 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:21:39.759 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:39 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:21:39.784 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:39.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:39.789 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:21:39.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:39 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:21:39.889 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:21:39.920 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:39.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:39.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:21:40.022 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:21:40.059 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:40.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:40.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:21:40.161 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:21:40.171 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Total time run: 90.0105 2024-09-17T07:21:40.171 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Total writes made: 172246 2024-09-17T07:21:40.171 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Write size: 65536 2024-09-17T07:21:40.171 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Object size: 65536 2024-09-17T07:21:40.171 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Bandwidth (MB/sec): 119.601 2024-09-17T07:21:40.171 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Stddev Bandwidth: 44.0375 2024-09-17T07:21:40.171 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Max bandwidth (MB/sec): 291.188 2024-09-17T07:21:40.171 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Min bandwidth (MB/sec): 0 2024-09-17T07:21:40.172 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Average IOPS: 1913 2024-09-17T07:21:40.172 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Stddev IOPS: 704.6 2024-09-17T07:21:40.172 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Max IOPS: 4659 2024-09-17T07:21:40.172 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Min IOPS: 0 2024-09-17T07:21:40.172 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Average Latency(s): 0.00835759 2024-09-17T07:21:40.172 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Stddev Latency(s): 0.015297 2024-09-17T07:21:40.172 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Max latency(s): 1.45955 2024-09-17T07:21:40.172 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Min latency(s): 0.00105244 2024-09-17T07:21:40.177 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Cleaning up (deleting benchmark objects) 2024-09-17T07:21:40.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:40.189 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:40.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:21:40.290 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:21:40.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:40.330 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:40.330 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:21:40.431 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:21:40.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:40.484 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:40.484 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:21:40.585 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:21:40.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:40 smithi100 ceph-mon[29092]: pgmap v913: 158 pgs: 4 active+clean+scrubbing, 5 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 4 active+undersized+degraded+remapped+backfill_toofull, 142 active+clean; 10 GiB data, 29 GiB used, 508 GiB / 536 GiB avail; 1023 B/s rd, 84 MiB/s wr, 1.35k op/s; 115279/506679 objects degraded (22.752%); 28 MiB/s, 451 objects/s recovering 2024-09-17T07:21:40.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:40 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 115279/506679 objects degraded (22.752%), 12 pgs degraded (PG_DEGRADED) 2024-09-17T07:21:40.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:40 smithi100 ceph-mon[39462]: pgmap v913: 158 pgs: 4 active+clean+scrubbing, 5 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 4 active+undersized+degraded+remapped+backfill_toofull, 142 active+clean; 10 GiB data, 29 GiB used, 508 GiB / 536 GiB avail; 1023 B/s rd, 84 MiB/s wr, 1.35k op/s; 115279/506679 objects degraded (22.752%); 28 MiB/s, 451 objects/s recovering 2024-09-17T07:21:40.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:40 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 115279/506679 objects degraded (22.752%), 12 pgs degraded (PG_DEGRADED) 2024-09-17T07:21:40.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:40.628 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:40.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:21:40.729 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:21:40.762 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:40.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:40.763 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:21:40.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:40 smithi135 ceph-mon[33238]: pgmap v913: 158 pgs: 4 active+clean+scrubbing, 5 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 4 active+undersized+degraded+remapped+backfill_toofull, 142 active+clean; 10 GiB data, 29 GiB used, 508 GiB / 536 GiB avail; 1023 B/s rd, 84 MiB/s wr, 1.35k op/s; 115279/506679 objects degraded (22.752%); 28 MiB/s, 451 objects/s recovering 2024-09-17T07:21:40.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:40 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 115279/506679 objects degraded (22.752%), 12 pgs degraded (PG_DEGRADED) 2024-09-17T07:21:40.864 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:21:40.895 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:40.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:40.896 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:21:40.997 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:21:41.021 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:41.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:41.022 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:21:41.123 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:21:41.151 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:41.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:41.152 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:21:41.253 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:21:41.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:41.297 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:41.297 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:21:41.398 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:21:41.445 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:41.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:41.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:21:41.504 INFO:teuthology.orchestra.run.smithi100.stderr:noscrub is set 2024-09-17T07:21:41.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:41 smithi135 ceph-mon[33238]: 2.c deep-scrub starts 2024-09-17T07:21:41.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:41 smithi135 ceph-mon[33238]: 4.18 scrub starts 2024-09-17T07:21:41.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:41 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/3251958420' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:21:41.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:41 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:21:41.549 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:21:41.588 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:41.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:41.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:21:41.690 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:21:41.726 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:41.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:41.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:21:41.761 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:41 smithi100 ceph-mon[39462]: 2.c deep-scrub starts 2024-09-17T07:21:41.761 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:41 smithi100 ceph-mon[39462]: 4.18 scrub starts 2024-09-17T07:21:41.761 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:41 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/3251958420' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:21:41.761 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:41 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:21:41.763 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:41 smithi100 ceph-mon[29092]: 2.c deep-scrub starts 2024-09-17T07:21:41.763 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:41 smithi100 ceph-mon[29092]: 4.18 scrub starts 2024-09-17T07:21:41.763 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:41 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/3251958420' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:21:41.763 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:41 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:21:41.829 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:21:41.865 INFO:teuthology.orchestra.run.smithi135.stdout:{ 2024-09-17T07:21:41.866 INFO:teuthology.orchestra.run.smithi135.stdout: "ops": [ 2024-09-17T07:21:41.866 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:21:41.866 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.25811.0:174535 11.e 11:75f47bbb:::benchmark_data_smithi100_180298_object2286:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e423)", 2024-09-17T07:21:41.866 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:21:41.857939+0000", 2024-09-17T07:21:41.866 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 0.0046040930000000001, 2024-09-17T07:21:41.867 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.004637848, 2024-09-17T07:21:41.867 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:21:41.867 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "waiting for sub ops", 2024-09-17T07:21:41.867 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:21:41.867 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.25811", 2024-09-17T07:21:41.868 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/173867468", 2024-09-17T07:21:41.868 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 174535 2024-09-17T07:21:41.868 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:21:41.868 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:21:41.868 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:21:41.868 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:21:41.869 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:21:41.857939+0000", 2024-09-17T07:21:41.869 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:21:41.869 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:21:41.870 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:21:41.870 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:21:41.870 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:21:41.857939+0000", 2024-09-17T07:21:41.870 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:21:41.870 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:21:41.870 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:21:41.870 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:21:41.870 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:21:41.857938+0000", 2024-09-17T07:21:41.870 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:21:41.870 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:21:41.870 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:21:41.870 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:21:41.871 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:21:41.857945+0000", 2024-09-17T07:21:41.871 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.114e-06 2024-09-17T07:21:41.871 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:21:41.871 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:21:41.871 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:21:41.871 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:21:41.857960+0000", 2024-09-17T07:21:41.871 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.4941e-05 2024-09-17T07:21:41.871 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:21:41.871 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:21:41.871 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:21:41.871 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:21:41.857968+0000", 2024-09-17T07:21:41.871 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.14e-06 2024-09-17T07:21:41.872 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:21:41.872 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:21:41.872 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:21:41.872 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:21:41.858023+0000", 2024-09-17T07:21:41.872 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.4755e-05 2024-09-17T07:21:41.872 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:21:41.872 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:21:41.872 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:21:41.872 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:21:41.858185+0000", 2024-09-17T07:21:41.872 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00016269599999999999 2024-09-17T07:21:41.872 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:21:41.872 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:21:41.872 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 0,1", 2024-09-17T07:21:41.873 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:21:41.858320+0000", 2024-09-17T07:21:41.873 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00013435199999999999 2024-09-17T07:21:41.873 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:21:41.873 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:21:41.873 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:21:41.873 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:21:41.860864+0000", 2024-09-17T07:21:41.873 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0025437979999999999 2024-09-17T07:21:41.873 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:21:41.873 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:21:41.873 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:21:41.873 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:21:41.873 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:21:41.873 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_repop(client.25811.0:174538 11.7 e423/414 11:e3e5e995:::benchmark_data_smithi100_180298_object2289:head v 423'10906, mlcod=423'10904)", 2024-09-17T07:21:41.874 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:21:41.861117+0000", 2024-09-17T07:21:41.874 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 0.001425763, 2024-09-17T07:21:41.874 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.001532013, 2024-09-17T07:21:41.874 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:21:41.874 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "started", 2024-09-17T07:21:41.874 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:21:41.874 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:21:41.874 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:21:41.874 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:21:41.861117+0000", 2024-09-17T07:21:41.874 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:21:41.874 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:21:41.874 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:21:41.874 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:21:41.875 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:21:41.861117+0000", 2024-09-17T07:21:41.875 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:21:41.875 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:21:41.875 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:21:41.875 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:21:41.875 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:21:41.861117+0000", 2024-09-17T07:21:41.875 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.999999 2024-09-17T07:21:41.875 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:21:41.875 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:21:41.875 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:21:41.875 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:21:41.861125+0000", 2024-09-17T07:21:41.875 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.4020000000000002e-06 2024-09-17T07:21:41.875 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:21:41.876 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:21:41.876 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:21:41.876 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:21:41.861138+0000", 2024-09-17T07:21:41.876 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.3138e-05 2024-09-17T07:21:41.876 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:21:41.876 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:21:41.876 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:21:41.876 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:21:41.861146+0000", 2024-09-17T07:21:41.876 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.5399999999999998e-06 2024-09-17T07:21:41.876 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:21:41.876 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:21:41.876 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:21:41.877 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:21:41.861205+0000", 2024-09-17T07:21:41.877 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.9586000000000002e-05 2024-09-17T07:21:41.877 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:21:41.877 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:21:41.877 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:21:41.877 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:21:41.861228+0000", 2024-09-17T07:21:41.877 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.2878999999999999e-05 2024-09-17T07:21:41.877 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:21:41.877 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:21:41.877 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:21:41.877 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:21:41.877 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:21:41.877 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.25811.0:174539 11.d 11:bd12929f:::benchmark_data_smithi100_180298_object2290:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e423)", 2024-09-17T07:21:41.878 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:21:41.861656+0000", 2024-09-17T07:21:41.878 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 0.000887242, 2024-09-17T07:21:41.878 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0010430350000000001, 2024-09-17T07:21:41.878 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:21:41.878 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "waiting for sub ops", 2024-09-17T07:21:41.878 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:21:41.878 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.25811", 2024-09-17T07:21:41.878 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/173867468", 2024-09-17T07:21:41.878 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 174539 2024-09-17T07:21:41.878 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:21:41.878 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:21:41.879 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:21:41.879 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:21:41.879 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:21:41.861656+0000", 2024-09-17T07:21:41.879 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:21:41.879 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:21:41.879 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:21:41.879 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:21:41.879 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:21:41.861656+0000", 2024-09-17T07:21:41.879 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:21:41.879 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:21:41.879 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:21:41.879 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:21:41.879 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:21:41.861655+0000", 2024-09-17T07:21:41.879 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.999999 2024-09-17T07:21:41.880 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:21:41.880 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:21:41.880 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:21:41.880 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:21:41.861660+0000", 2024-09-17T07:21:41.880 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.2190000000000003e-06 2024-09-17T07:21:41.880 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:21:41.880 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:21:41.880 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:21:41.880 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:21:41.861670+0000", 2024-09-17T07:21:41.880 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.6560000000000004e-06 2024-09-17T07:21:41.880 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:21:41.880 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:21:41.880 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:21:41.881 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:21:41.861675+0000", 2024-09-17T07:21:41.881 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.4970000000000001e-06 2024-09-17T07:21:41.881 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:21:41.881 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:21:41.881 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:21:41.881 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:21:41.861711+0000", 2024-09-17T07:21:41.881 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.5330999999999998e-05 2024-09-17T07:21:41.881 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:21:41.881 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:21:41.881 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:21:41.881 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:21:41.861805+0000", 2024-09-17T07:21:41.881 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.4755999999999999e-05 2024-09-17T07:21:41.881 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:21:41.881 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:21:41.882 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 2,6", 2024-09-17T07:21:41.882 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:21:41.861871+0000", 2024-09-17T07:21:41.882 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.6098000000000003e-05 2024-09-17T07:21:41.882 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:21:41.882 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:21:41.882 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:21:41.882 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:21:41.882 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:21:41.882 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_repop(client.25811.0:174537 11.c e423/414)", 2024-09-17T07:21:41.882 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:21:41.862724+0000", 2024-09-17T07:21:41.882 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 4294967295.9998193, 2024-09-17T07:21:41.882 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.3453e-05, 2024-09-17T07:21:41.882 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:21:41.882 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "queued for pg", 2024-09-17T07:21:41.883 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:21:41.883 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:21:41.883 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:21:41.883 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:21:41.862724+0000", 2024-09-17T07:21:41.883 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:21:41.883 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:21:41.883 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:21:41.883 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:21:41.883 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:21:41.862724+0000", 2024-09-17T07:21:41.883 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:21:41.883 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:21:41.883 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:21:41.883 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:21:41.884 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:21:41.862723+0000", 2024-09-17T07:21:41.884 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.999999 2024-09-17T07:21:41.884 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:21:41.884 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:21:41.884 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:21:41.884 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:21:41.862730+0000", 2024-09-17T07:21:41.884 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.5769999999999999e-06 2024-09-17T07:21:41.884 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:21:41.884 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:21:41.884 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:21:41.884 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:21:41.862740+0000", 2024-09-17T07:21:41.884 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0022e-05 2024-09-17T07:21:41.884 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:21:41.885 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:21:41.885 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:21:41.885 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:21:41.862747+0000", 2024-09-17T07:21:41.885 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.0450000000000003e-06 2024-09-17T07:21:41.885 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:21:41.885 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:21:41.885 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:21:41.885 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:21:41.885 INFO:teuthology.orchestra.run.smithi135.stdout: ], 2024-09-17T07:21:41.885 INFO:teuthology.orchestra.run.smithi135.stdout: "num_ops": 4 2024-09-17T07:21:41.885 INFO:teuthology.orchestra.run.smithi135.stdout:} 2024-09-17T07:21:41.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:41.886 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:41.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:21:41.987 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:21:42.020 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:42.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:42.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:21:42.122 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:21:42.152 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:42.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:42.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:21:42.254 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:21:42.277 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:42.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:42.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:21:42.379 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:21:42.406 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:42.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:42.408 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:21:42.509 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:21:42.531 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:42.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:42.532 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:21:42.633 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:21:42.652 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:42 smithi100 ceph-mon[29092]: pgmap v914: 158 pgs: 4 active+clean+scrubbing, 5 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 4 active+undersized+degraded+remapped+backfill_toofull, 142 active+clean; 10 GiB data, 29 GiB used, 508 GiB / 536 GiB avail; 857 B/s rd, 82 MiB/s wr, 1.31k op/s; 117391/513030 objects degraded (22.882%); 24 MiB/s, 378 objects/s recovering 2024-09-17T07:21:42.652 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:42 smithi100 ceph-mon[29092]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:21:42.652 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:42 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T07:21:42.652 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:42 smithi100 ceph-mon[29092]: osdmap e423: 8 total, 8 up, 6 in 2024-09-17T07:21:42.653 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:42 smithi100 ceph-mon[39462]: pgmap v914: 158 pgs: 4 active+clean+scrubbing, 5 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 4 active+undersized+degraded+remapped+backfill_toofull, 142 active+clean; 10 GiB data, 29 GiB used, 508 GiB / 536 GiB avail; 857 B/s rd, 82 MiB/s wr, 1.31k op/s; 117391/513030 objects degraded (22.882%); 24 MiB/s, 378 objects/s recovering 2024-09-17T07:21:42.653 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:42 smithi100 ceph-mon[39462]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:21:42.653 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:42 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T07:21:42.653 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:42 smithi100 ceph-mon[39462]: osdmap e423: 8 total, 8 up, 6 in 2024-09-17T07:21:42.667 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:42.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:42.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:21:42.769 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:21:42.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:42 smithi135 ceph-mon[33238]: pgmap v914: 158 pgs: 4 active+clean+scrubbing, 5 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 4 active+undersized+degraded+remapped+backfill_toofull, 142 active+clean; 10 GiB data, 29 GiB used, 508 GiB / 536 GiB avail; 857 B/s rd, 82 MiB/s wr, 1.31k op/s; 117391/513030 objects degraded (22.882%); 24 MiB/s, 378 objects/s recovering 2024-09-17T07:21:42.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:42 smithi135 ceph-mon[33238]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:21:42.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:42 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T07:21:42.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:42 smithi135 ceph-mon[33238]: osdmap e423: 8 total, 8 up, 6 in 2024-09-17T07:21:42.801 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:42.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:42.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:21:42.903 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:21:42.935 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:42.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:42.936 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:21:43.037 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:21:43.066 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:43.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:43.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:21:43.167 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:21:43.201 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:43.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:43.202 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:21:43.289 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T07:21:43.297 INFO:teuthology.orchestra.run.smithi100.stderr:dumped all 2024-09-17T07:21:43.303 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:21:43.337 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:43.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:43.338 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:21:43.439 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:21:43.476 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:43.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:43.477 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:21:43.578 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:21:43.612 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:43.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:43.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:21:43.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:43 smithi100 ceph-mon[29092]: from='client.16053 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:21:43.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:43 smithi100 ceph-mon[39462]: from='client.16053 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:21:43.715 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:21:43.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:43.755 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:43.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:21:43.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:43 smithi135 ceph-mon[33238]: from='client.16053 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:21:43.856 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:21:43.881 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:43.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:43.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:21:43.922 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:21:43.983 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:21:44.026 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:44.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:44.027 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:21:44.128 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:21:44.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:44.161 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:44.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:21:44.262 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:21:44.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:44.298 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:44.298 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:21:44.399 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:21:44.412 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-17T07:21:44.437 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:44.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:44.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:21:44.539 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:21:44.569 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:44.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:44.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:21:44.671 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:21:44.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:44.710 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:44.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:21:44.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:44 smithi135 ceph-mon[33238]: pgmap v916: 158 pgs: 5 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 4 active+undersized+degraded+remapped+backfill_toofull, 146 active+clean; 10 GiB data, 29 GiB used, 508 GiB / 536 GiB avail; 1.1 KiB/s rd, 79 MiB/s wr, 1.34k op/s; 117217/512187 objects degraded (22.886%); 28 MiB/s, 451 objects/s recovering 2024-09-17T07:21:44.811 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:21:44.833 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:44.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:44.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:21:44.876 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:44 smithi100 ceph-mon[39462]: pgmap v916: 158 pgs: 5 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 4 active+undersized+degraded+remapped+backfill_toofull, 146 active+clean; 10 GiB data, 29 GiB used, 508 GiB / 536 GiB avail; 1.1 KiB/s rd, 79 MiB/s wr, 1.34k op/s; 117217/512187 objects degraded (22.886%); 28 MiB/s, 451 objects/s recovering 2024-09-17T07:21:44.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:44 smithi100 ceph-mon[29092]: pgmap v916: 158 pgs: 5 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 4 active+undersized+degraded+remapped+backfill_toofull, 146 active+clean; 10 GiB data, 29 GiB used, 508 GiB / 536 GiB avail; 1.1 KiB/s rd, 79 MiB/s wr, 1.34k op/s; 117217/512187 objects degraded (22.886%); 28 MiB/s, 451 objects/s recovering 2024-09-17T07:21:44.935 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:21:44.963 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:44.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:44.964 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:21:45.064 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:21:45.097 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:45.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:45.098 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:21:45.198 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:21:45.219 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:45.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:45.220 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:21:45.321 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:21:45.353 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:45.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:45.354 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:21:45.455 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:21:45.488 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:45.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:45.489 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:21:45.590 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:21:45.626 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:45.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:45.627 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:21:45.728 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:21:45.763 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:45.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:45.764 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:21:45.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:45 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 117217/512187 objects degraded (22.886%), 12 pgs degraded (PG_DEGRADED) 2024-09-17T07:21:45.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:45 smithi135 ceph-mon[33238]: osdmap e424: 8 total, 8 up, 6 in 2024-09-17T07:21:45.868 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:21:45.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:45 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 117217/512187 objects degraded (22.886%), 12 pgs degraded (PG_DEGRADED) 2024-09-17T07:21:45.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:45 smithi100 ceph-mon[29092]: osdmap e424: 8 total, 8 up, 6 in 2024-09-17T07:21:45.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:45 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 117217/512187 objects degraded (22.886%), 12 pgs degraded (PG_DEGRADED) 2024-09-17T07:21:45.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:45 smithi100 ceph-mon[39462]: osdmap e424: 8 total, 8 up, 6 in 2024-09-17T07:21:45.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:45.902 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:45.902 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:21:46.003 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:21:46.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:46.034 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:46.034 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:21:46.134 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:21:46.169 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:46.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:46.170 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:21:46.271 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:21:46.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:46.309 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:46.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:21:46.376 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:21:45 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:21:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:21:46.411 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:21:46.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:46.450 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:46.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:21:46.551 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:21:46.584 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:46.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:46.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:21:46.686 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:21:46.722 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:46.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:46.723 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:21:46.824 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:21:46.861 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:46.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:46.862 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:21:46.963 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:21:47.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:47.001 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:47.001 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:21:47.102 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:21:47.134 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:47.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:47.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:21:47.236 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:21:47.265 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:47.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:47.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:21:47.284 INFO:teuthology.orchestra.run.smithi100.stdout:{"pg_ready":true,"pg_map":{"version":915,"stamp":"2024-09-17T07:21:41.494098+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":11195211693,"num_objects":171010,"num_object_clones":0,"num_object_copies":513030,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":117391,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":171010,"num_whiteouts":0,"num_read":1230,"num_read_kb":1015,"num_write":171595,"num_write_kb":10932954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10237,"num_bytes_recovered":658826793,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64090,"ondisk_log_size":64090,"up":474,"acting":462,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":30289100,"kb_used_data":25137628,"kb_used_omap":12,"kb_used_meta":5151411,"kb_avail":532181812,"statfs":{"total":575970213888,"available":544954175488,"internally_reserved":0,"allocated":25740931072,"data_stored":25734768670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12408,"internal_metadata":5275045768},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,5,3,6,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":879296512,"num_objects":13417,"num_object_clones":0,"num_object_copies":40251,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4824,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13417,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":13417,"num_write_kb":858688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4517,"num_bytes_recovered":296026112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.074234"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":217,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.414088+0000","last_change":"2024-09-17T07:21:16.377740+0000","last_active":"2024-09-17T07:21:26.414088+0000","last_peered":"2024-09-17T07:21:26.414088+0000","last_clean":"2024-09-17T07:21:26.414088+0000","last_became_active":"2024-09-17T07:21:16.377116+0000","last_became_peered":"2024-09-17T07:21:16.377116+0000","last_unstale":"2024-09-17T07:21:26.414088+0000","last_undegraded":"2024-09-17T07:21:26.414088+0000","last_fullsized":"2024-09-17T07:21:26.414088+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:10.965933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:33.644728+0000","last_clean_scrub_stamp":"2024-09-17T07:20:10.965933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:21:35.805454+0000","scrub_duration":0.0040893329999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":458,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.413282+0000","last_change":"2024-09-17T07:19:42.511303+0000","last_active":"2024-09-17T07:21:26.413282+0000","last_peered":"2024-09-17T07:21:26.413282+0000","last_clean":"2024-09-17T07:21:26.413282+0000","last_became_active":"2024-09-17T07:19:42.511028+0000","last_became_peered":"2024-09-17T07:19:42.511028+0000","last_unstale":"2024-09-17T07:21:26.413282+0000","last_undegraded":"2024-09-17T07:21:26.413282+0000","last_fullsized":"2024-09-17T07:21:26.413282+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:18:46.393298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:18:46.393298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0058972349999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"242'27","reported_seq":359,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-17T07:21:29.504063+0000","last_change":"2024-09-17T07:18:53.477464+0000","last_active":"2024-09-17T07:21:29.504063+0000","last_peered":"2024-09-17T07:21:29.504063+0000","last_clean":"2024-09-17T07:21:29.504063+0000","last_became_active":"2024-09-17T07:09:24.402042+0000","last_became_peered":"2024-09-17T07:09:24.402042+0000","last_unstale":"2024-09-17T07:21:29.504063+0000","last_undegraded":"2024-09-17T07:21:29.504063+0000","last_fullsized":"2024-09-17T07:21:29.504063+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'27","last_scrub_stamp":"2024-09-17T07:18:53.476878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:18:53.476878+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037504449999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":438,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-17T07:21:29.493237+0000","last_change":"2024-09-17T07:21:15.354814+0000","last_active":"2024-09-17T07:21:29.493237+0000","last_peered":"2024-09-17T07:21:29.493237+0000","last_clean":"2024-09-17T07:21:29.493237+0000","last_became_active":"2024-09-17T07:21:15.354561+0000","last_became_peered":"2024-09-17T07:21:15.354561+0000","last_unstale":"2024-09-17T07:21:29.493237+0000","last_undegraded":"2024-09-17T07:21:29.493237+0000","last_fullsized":"2024-09-17T07:21:29.493237+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:45.261117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:15:45.261117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001477482,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":300,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.413691+0000","last_change":"2024-09-17T07:21:13.454504+0000","last_active":"2024-09-17T07:21:26.413691+0000","last_peered":"2024-09-17T07:21:26.413691+0000","last_clean":"2024-09-17T07:21:26.413691+0000","last_became_active":"2024-09-17T07:09:24.402568+0000","last_became_peered":"2024-09-17T07:09:24.402568+0000","last_unstale":"2024-09-17T07:21:26.413691+0000","last_undegraded":"2024-09-17T07:21:26.413691+0000","last_fullsized":"2024-09-17T07:21:26.413691+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:21:13.454404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:17:35.507742+0000","last_clean_scrub_stamp":"2024-09-17T07:21:13.454404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:22:21.815584+0000","scrub_duration":0.016029897000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"242'15","reported_seq":476,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-17T07:21:29.493296+0000","last_change":"2024-09-17T07:21:15.355111+0000","last_active":"2024-09-17T07:21:29.493296+0000","last_peered":"2024-09-17T07:21:29.493296+0000","last_clean":"2024-09-17T07:21:29.493296+0000","last_became_active":"2024-09-17T07:21:15.354718+0000","last_became_peered":"2024-09-17T07:21:15.354718+0000","last_unstale":"2024-09-17T07:21:29.493296+0000","last_undegraded":"2024-09-17T07:21:29.493296+0000","last_fullsized":"2024-09-17T07:21:29.493296+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'15","last_scrub_stamp":"2024-09-17T07:15:39.306043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:15:39.306043+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002471029,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":310,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.419462+0000","last_change":"2024-09-17T07:21:16.343833+0000","last_active":"2024-09-17T07:21:26.419462+0000","last_peered":"2024-09-17T07:21:26.419462+0000","last_clean":"2024-09-17T07:21:26.419462+0000","last_became_active":"2024-09-17T07:21:16.343549+0000","last_became_peered":"2024-09-17T07:21:16.343549+0000","last_unstale":"2024-09-17T07:21:26.419462+0000","last_undegraded":"2024-09-17T07:21:26.419462+0000","last_fullsized":"2024-09-17T07:21:26.419462+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:13.660077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:20:13.660077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:21:31.505423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":437,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.408200+0000","last_change":"2024-09-17T07:21:26.408200+0000","last_active":"2024-09-17T07:21:26.408200+0000","last_peered":"2024-09-17T07:21:26.408200+0000","last_clean":"2024-09-17T07:21:26.408200+0000","last_became_active":"2024-09-17T07:21:15.354423+0000","last_became_peered":"2024-09-17T07:21:15.354423+0000","last_unstale":"2024-09-17T07:21:26.408200+0000","last_undegraded":"2024-09-17T07:21:26.408200+0000","last_fullsized":"2024-09-17T07:21:26.408200+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:41.707969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:14:41.707969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.014931757,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":59,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.419765+0000","last_change":"2024-09-17T07:20:49.447835+0000","last_active":"2024-09-17T07:21:26.419765+0000","last_peered":"2024-09-17T07:21:26.419765+0000","last_clean":"2024-09-17T07:21:26.419765+0000","last_became_active":"2024-09-17T07:20:49.447479+0000","last_became_peered":"2024-09-17T07:20:49.447479+0000","last_unstale":"2024-09-17T07:21:26.419765+0000","last_undegraded":"2024-09-17T07:21:26.419765+0000","last_fullsized":"2024-09-17T07:21:26.419765+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:18:44.849299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:46.312371+0000","last_clean_scrub_stamp":"2024-09-17T07:18:44.849299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1b","version":"242'7","reported_seq":469,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.408853+0000","last_change":"2024-09-17T07:21:25.321537+0000","last_active":"2024-09-17T07:21:26.408853+0000","last_peered":"2024-09-17T07:21:26.408853+0000","last_clean":"2024-09-17T07:21:26.408853+0000","last_became_active":"2024-09-17T07:21:15.349235+0000","last_became_peered":"2024-09-17T07:21:15.349235+0000","last_unstale":"2024-09-17T07:21:26.408853+0000","last_undegraded":"2024-09-17T07:21:26.408853+0000","last_fullsized":"2024-09-17T07:21:26.408853+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'7","last_scrub_stamp":"2024-09-17T07:14:42.653705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:14:42.653705+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0050149230000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":372,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.408338+0000","last_change":"2024-09-17T07:21:16.347688+0000","last_active":"2024-09-17T07:21:26.408338+0000","last_peered":"2024-09-17T07:21:26.408338+0000","last_clean":"2024-09-17T07:21:26.408338+0000","last_became_active":"2024-09-17T07:21:16.347352+0000","last_became_peered":"2024-09-17T07:21:16.347352+0000","last_unstale":"2024-09-17T07:21:26.408338+0000","last_undegraded":"2024-09-17T07:21:26.408338+0000","last_fullsized":"2024-09-17T07:21:26.408338+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:10.116580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:20:10.116580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00137765,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":454,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.415106+0000","last_change":"2024-09-17T07:21:15.344952+0000","last_active":"2024-09-17T07:21:26.415106+0000","last_peered":"2024-09-17T07:21:26.415106+0000","last_clean":"2024-09-17T07:21:26.415106+0000","last_became_active":"2024-09-17T07:21:15.344389+0000","last_became_peered":"2024-09-17T07:21:15.344389+0000","last_unstale":"2024-09-17T07:21:26.415106+0000","last_undegraded":"2024-09-17T07:21:26.415106+0000","last_fullsized":"2024-09-17T07:21:26.415106+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:52.168181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:14:52.168181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0056501399999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":50,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.415659+0000","last_change":"2024-09-17T07:19:42.526705+0000","last_active":"2024-09-17T07:21:26.415659+0000","last_peered":"2024-09-17T07:21:26.415659+0000","last_clean":"2024-09-17T07:21:26.415659+0000","last_became_active":"2024-09-17T07:19:42.526462+0000","last_became_peered":"2024-09-17T07:19:42.526462+0000","last_unstale":"2024-09-17T07:21:26.415659+0000","last_undegraded":"2024-09-17T07:21:26.415659+0000","last_fullsized":"2024-09-17T07:21:26.415659+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:18.961914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:45.029247+0000","last_clean_scrub_stamp":"2024-09-17T07:19:18.961914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"312'19","reported_seq":505,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.408606+0000","last_change":"2024-09-17T07:20:39.070372+0000","last_active":"2024-09-17T07:21:26.408606+0000","last_peered":"2024-09-17T07:21:26.408606+0000","last_clean":"2024-09-17T07:21:26.408606+0000","last_became_active":"2024-09-17T07:09:24.397954+0000","last_became_peered":"2024-09-17T07:09:24.397954+0000","last_unstale":"2024-09-17T07:21:26.408606+0000","last_undegraded":"2024-09-17T07:21:26.408606+0000","last_fullsized":"2024-09-17T07:21:26.408606+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"312'19","last_scrub_stamp":"2024-09-17T07:20:39.070235+0000","last_deep_scrub":"312'19","last_deep_scrub_stamp":"2024-09-17T07:16:44.292911+0000","last_clean_scrub_stamp":"2024-09-17T07:20:39.070235+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:22:02.149699+0000","scrub_duration":0.0028570560000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":462,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.415152+0000","last_change":"2024-09-17T07:19:42.529916+0000","last_active":"2024-09-17T07:21:26.415152+0000","last_peered":"2024-09-17T07:21:26.415152+0000","last_clean":"2024-09-17T07:21:26.415152+0000","last_became_active":"2024-09-17T07:19:42.528837+0000","last_became_peered":"2024-09-17T07:19:42.528837+0000","last_unstale":"2024-09-17T07:21:26.415152+0000","last_undegraded":"2024-09-17T07:21:26.415152+0000","last_fullsized":"2024-09-17T07:21:26.415152+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:26.348475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:28.096266+0000","last_clean_scrub_stamp":"2024-09-17T07:19:26.348475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.015361906,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":20,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.416709+0000","last_change":"2024-09-17T07:21:16.347872+0000","last_active":"2024-09-17T07:21:26.416709+0000","last_peered":"2024-09-17T07:21:26.416709+0000","last_clean":"2024-09-17T07:21:26.416709+0000","last_became_active":"2024-09-17T07:21:16.347244+0000","last_became_peered":"2024-09-17T07:21:16.347244+0000","last_unstale":"2024-09-17T07:21:26.416709+0000","last_undegraded":"2024-09-17T07:21:26.416709+0000","last_fullsized":"2024-09-17T07:21:26.416709+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:27.474541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:20:27.474541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:21:50.015226+0000","scrub_duration":0.0061993300000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2","version":"422'10830","reported_seq":10876,"reported_epoch":422,"state":"active+undersized+degraded+remapped+backfill_wait","last_fresh":"2024-09-17T07:21:39.815319+0000","last_change":"2024-09-17T07:21:16.519565+0000","last_active":"2024-09-17T07:21:39.815319+0000","last_peered":"2024-09-17T07:21:39.815319+0000","last_clean":"2024-09-17T07:19:54.447320+0000","last_became_active":"2024-09-17T07:21:16.510358+0000","last_became_peered":"2024-09-17T07:21:16.510358+0000","last_unstale":"2024-09-17T07:21:39.815319+0000","last_undegraded":"2024-09-17T07:21:16.370227+0000","last_fullsized":"2024-09-17T07:21:16.338604+0000","mapping_epoch":414,"log_start":"409'6800","ondisk_log_start":"409'6800","created":390,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_clean_scrub_stamp":"2024-09-17T07:19:54.447320+0000","objects_scrubbed":0,"log_size":4030,"log_dups_size":0,"ondisk_log_size":4030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":709754880,"num_objects":10830,"num_object_clones":0,"num_object_copies":32490,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10830,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10830,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10830,"num_write_kb":693120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":10830}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.8","version":"212'130","reported_seq":515,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.414598+0000","last_change":"2024-09-17T07:20:21.626281+0000","last_active":"2024-09-17T07:21:26.414598+0000","last_peered":"2024-09-17T07:21:26.414598+0000","last_clean":"2024-09-17T07:21:26.414598+0000","last_became_active":"2024-09-17T07:19:35.418729+0000","last_became_peered":"2024-09-17T07:19:35.418729+0000","last_unstale":"2024-09-17T07:21:26.414598+0000","last_undegraded":"2024-09-17T07:21:26.414598+0000","last_fullsized":"2024-09-17T07:21:26.414598+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":4,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:20:21.626089+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:14:57.052505+0000","last_clean_scrub_stamp":"2024-09-17T07:20:21.626089+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:21:34.352575+0000","scrub_duration":0.0050515719999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"242'39","reported_seq":508,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.410850+0000","last_change":"2024-09-17T07:21:15.349153+0000","last_active":"2024-09-17T07:21:26.410850+0000","last_peered":"2024-09-17T07:21:26.410850+0000","last_clean":"2024-09-17T07:21:26.410850+0000","last_became_active":"2024-09-17T07:21:15.348413+0000","last_became_peered":"2024-09-17T07:21:15.348413+0000","last_unstale":"2024-09-17T07:21:26.410850+0000","last_undegraded":"2024-09-17T07:21:26.410850+0000","last_fullsized":"2024-09-17T07:21:26.410850+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'39","last_scrub_stamp":"2024-09-17T07:15:46.226554+0000","last_deep_scrub":"95'25","last_deep_scrub_stamp":"2024-09-17T07:04:09.450039+0000","last_clean_scrub_stamp":"2024-09-17T07:15:46.226554+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0027061799999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":50,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.409000+0000","last_change":"2024-09-17T07:19:42.530050+0000","last_active":"2024-09-17T07:21:26.409000+0000","last_peered":"2024-09-17T07:21:26.409000+0000","last_clean":"2024-09-17T07:21:26.409000+0000","last_became_active":"2024-09-17T07:19:42.529735+0000","last_became_peered":"2024-09-17T07:19:42.529735+0000","last_unstale":"2024-09-17T07:21:26.409000+0000","last_undegraded":"2024-09-17T07:21:26.409000+0000","last_fullsized":"2024-09-17T07:21:26.409000+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:35.965669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:19:35.965669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":273,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.420159+0000","last_change":"2024-09-17T07:21:15.350941+0000","last_active":"2024-09-17T07:21:26.420159+0000","last_peered":"2024-09-17T07:21:26.420159+0000","last_clean":"2024-09-17T07:21:26.420159+0000","last_became_active":"2024-09-17T07:21:15.349538+0000","last_became_peered":"2024-09-17T07:21:15.349538+0000","last_unstale":"2024-09-17T07:21:26.420159+0000","last_undegraded":"2024-09-17T07:21:26.420159+0000","last_fullsized":"2024-09-17T07:21:26.420159+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:46.502641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:13:46.502641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":302,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.411241+0000","last_change":"2024-09-17T07:20:40.117298+0000","last_active":"2024-09-17T07:21:26.411241+0000","last_peered":"2024-09-17T07:21:26.411241+0000","last_clean":"2024-09-17T07:21:26.411241+0000","last_became_active":"2024-09-17T07:09:24.396334+0000","last_became_peered":"2024-09-17T07:09:24.396334+0000","last_unstale":"2024-09-17T07:21:26.411241+0000","last_undegraded":"2024-09-17T07:21:26.411241+0000","last_fullsized":"2024-09-17T07:21:26.411241+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:40.117165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:42.984088+0000","last_clean_scrub_stamp":"2024-09-17T07:20:40.117165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:21:41.577168+0000","scrub_duration":0.0041399059999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.3","version":"422'10680","reported_seq":10732,"reported_epoch":422,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2024-09-17T07:21:39.016677+0000","last_change":"2024-09-17T07:21:16.499390+0000","last_active":"2024-09-17T07:21:39.016677+0000","last_peered":"2024-09-17T07:21:39.016677+0000","last_clean":"2024-09-17T07:19:54.447320+0000","last_became_active":"2024-09-17T07:21:16.484831+0000","last_became_peered":"2024-09-17T07:21:16.484831+0000","last_unstale":"2024-09-17T07:21:39.016677+0000","last_undegraded":"2024-09-17T07:21:16.498217+0000","last_fullsized":"2024-09-17T07:21:16.348126+0000","mapping_epoch":414,"log_start":"409'7000","ondisk_log_start":"409'7000","created":390,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_clean_scrub_stamp":"2024-09-17T07:19:54.447320+0000","objects_scrubbed":0,"log_size":3680,"log_dups_size":0,"ondisk_log_size":3680,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":699924480,"num_objects":10680,"num_object_clones":0,"num_object_copies":32040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10680,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10680,"num_write_kb":683520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[{"shards":"5,6","objects":10680}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"212'130","reported_seq":725,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.414733+0000","last_change":"2024-09-17T07:21:12.403414+0000","last_active":"2024-09-17T07:21:26.414733+0000","last_peered":"2024-09-17T07:21:26.414733+0000","last_clean":"2024-09-17T07:21:26.414733+0000","last_became_active":"2024-09-17T07:19:45.353783+0000","last_became_peered":"2024-09-17T07:19:45.353783+0000","last_unstale":"2024-09-17T07:21:26.414733+0000","last_undegraded":"2024-09-17T07:21:26.414733+0000","last_fullsized":"2024-09-17T07:21:26.414733+0000","mapping_epoch":386,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":387,"parent":"0.0","parent_split_bits":4,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:21:12.403277+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:14:57.052505+0000","last_clean_scrub_stamp":"2024-09-17T07:21:12.403277+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:22:32.780683+0000","scrub_duration":0.0045946390000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.c","version":"242'25","reported_seq":508,"reported_epoch":422,"state":"active+clean+scrubbing","last_fresh":"2024-09-17T07:21:29.493893+0000","last_change":"2024-09-17T07:21:38.140602+0000","last_active":"2024-09-17T07:21:29.493893+0000","last_peered":"2024-09-17T07:21:29.493893+0000","last_clean":"2024-09-17T07:21:29.493893+0000","last_became_active":"2024-09-17T07:21:15.347129+0000","last_became_peered":"2024-09-17T07:21:15.347129+0000","last_unstale":"2024-09-17T07:21:29.493893+0000","last_undegraded":"2024-09-17T07:21:29.493893+0000","last_fullsized":"2024-09-17T07:21:29.493893+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'25","last_scrub_stamp":"2024-09-17T07:15:02.532371+0000","last_deep_scrub":"176'15","last_deep_scrub_stamp":"2024-09-17T07:08:19.633791+0000","last_clean_scrub_stamp":"2024-09-17T07:15:02.532371+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012712606,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":308,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.419987+0000","last_change":"2024-09-17T07:21:16.346636+0000","last_active":"2024-09-17T07:21:26.419987+0000","last_peered":"2024-09-17T07:21:26.419987+0000","last_clean":"2024-09-17T07:21:26.419987+0000","last_became_active":"2024-09-17T07:21:16.346387+0000","last_became_peered":"2024-09-17T07:21:16.346387+0000","last_unstale":"2024-09-17T07:21:26.419987+0000","last_undegraded":"2024-09-17T07:21:26.419987+0000","last_fullsized":"2024-09-17T07:21:26.419987+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:11.580871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:20:11.580871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:21:35.220026+0000","scrub_duration":0.0065215869999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"387'10","reported_seq":97,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-17T07:21:30.743213+0000","last_change":"2024-09-17T07:19:42.540319+0000","last_active":"2024-09-17T07:21:30.743213+0000","last_peered":"2024-09-17T07:21:30.743213+0000","last_clean":"2024-09-17T07:21:30.743213+0000","last_became_active":"2024-09-17T07:19:42.529329+0000","last_became_peered":"2024-09-17T07:19:42.529329+0000","last_unstale":"2024-09-17T07:21:30.743213+0000","last_undegraded":"2024-09-17T07:21:30.743213+0000","last_fullsized":"2024-09-17T07:21:30.743213+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-09-17T07:19:24.964680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:19:24.964680+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":448,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.409503+0000","last_change":"2024-09-17T07:21:20.320651+0000","last_active":"2024-09-17T07:21:26.409503+0000","last_peered":"2024-09-17T07:21:26.409503+0000","last_clean":"2024-09-17T07:21:26.409503+0000","last_became_active":"2024-09-17T07:21:15.348799+0000","last_became_peered":"2024-09-17T07:21:15.348799+0000","last_unstale":"2024-09-17T07:21:26.409503+0000","last_undegraded":"2024-09-17T07:21:26.409503+0000","last_fullsized":"2024-09-17T07:21:26.409503+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:50.554591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:14:50.554591+0000","last_clean_scrub_stamp":"2024-09-17T07:14:50.554591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0067125969999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.0","version":"422'10843","reported_seq":10891,"reported_epoch":422,"state":"active+undersized+degraded+remapped+backfill_wait","last_fresh":"2024-09-17T07:21:39.772312+0000","last_change":"2024-09-17T07:21:16.512815+0000","last_active":"2024-09-17T07:21:39.772312+0000","last_peered":"2024-09-17T07:21:39.772312+0000","last_clean":"2024-09-17T07:19:54.447320+0000","last_became_active":"2024-09-17T07:21:16.485728+0000","last_became_peered":"2024-09-17T07:21:16.485728+0000","last_unstale":"2024-09-17T07:21:39.772312+0000","last_undegraded":"2024-09-17T07:21:16.366299+0000","last_fullsized":"2024-09-17T07:21:16.343328+0000","mapping_epoch":414,"log_start":"409'6800","ondisk_log_start":"409'6800","created":390,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_clean_scrub_stamp":"2024-09-17T07:19:54.447320+0000","objects_scrubbed":0,"log_size":4043,"log_dups_size":0,"ondisk_log_size":4043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":710606848,"num_objects":10843,"num_object_clones":0,"num_object_copies":32529,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10843,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10843,"num_write_kb":693952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[{"shards":"0,6","objects":10843}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.a","version":"212'130","reported_seq":515,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.415340+0000","last_change":"2024-09-17T07:20:33.710216+0000","last_active":"2024-09-17T07:21:26.415340+0000","last_peered":"2024-09-17T07:21:26.415340+0000","last_clean":"2024-09-17T07:21:26.415340+0000","last_became_active":"2024-09-17T07:19:35.422004+0000","last_became_peered":"2024-09-17T07:19:35.422004+0000","last_unstale":"2024-09-17T07:21:26.415340+0000","last_undegraded":"2024-09-17T07:21:26.415340+0000","last_fullsized":"2024-09-17T07:21:26.415340+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":4,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:20:33.709901+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:14:57.052505+0000","last_clean_scrub_stamp":"2024-09-17T07:20:33.709901+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:21:35.593749+0000","scrub_duration":0.0058367610000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"242'27","reported_seq":362,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.423416+0000","last_change":"2024-09-17T07:21:16.358036+0000","last_active":"2024-09-17T07:21:26.423416+0000","last_peered":"2024-09-17T07:21:26.423416+0000","last_clean":"2024-09-17T07:21:26.423416+0000","last_became_active":"2024-09-17T07:21:16.357727+0000","last_became_peered":"2024-09-17T07:21:16.357727+0000","last_unstale":"2024-09-17T07:21:26.423416+0000","last_undegraded":"2024-09-17T07:21:26.423416+0000","last_fullsized":"2024-09-17T07:21:26.423416+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'27","last_scrub_stamp":"2024-09-17T07:20:13.130718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:20:13.130718+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:21:40.853381+0000","scrub_duration":0.0081162040000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":322,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.419596+0000","last_change":"2024-09-17T07:21:16.482008+0000","last_active":"2024-09-17T07:21:26.419596+0000","last_peered":"2024-09-17T07:21:26.419596+0000","last_clean":"2024-09-17T07:21:26.419596+0000","last_became_active":"2024-09-17T07:21:16.481211+0000","last_became_peered":"2024-09-17T07:21:16.481211+0000","last_unstale":"2024-09-17T07:21:26.419596+0000","last_undegraded":"2024-09-17T07:21:26.419596+0000","last_fullsized":"2024-09-17T07:21:26.419596+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:14.001258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:20:14.001258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:21:37.504530+0000","scrub_duration":0.0034116369999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"75'8","reported_seq":686,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-17T07:21:30.742745+0000","last_change":"2024-09-17T07:19:24.039612+0000","last_active":"2024-09-17T07:21:30.742745+0000","last_peered":"2024-09-17T07:21:30.742745+0000","last_clean":"2024-09-17T07:21:30.742745+0000","last_became_active":"2024-09-17T07:02:16.021762+0000","last_became_peered":"2024-09-17T07:02:16.021762+0000","last_unstale":"2024-09-17T07:21:30.742745+0000","last_undegraded":"2024-09-17T07:21:30.742745+0000","last_fullsized":"2024-09-17T07:21:30.742745+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:19:24.039495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:19:24.039495+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.016850853999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":445,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.415066+0000","last_change":"2024-09-17T07:21:15.353673+0000","last_active":"2024-09-17T07:21:26.415066+0000","last_peered":"2024-09-17T07:21:26.415066+0000","last_clean":"2024-09-17T07:21:26.415066+0000","last_became_active":"2024-09-17T07:21:15.353307+0000","last_became_peered":"2024-09-17T07:21:15.353307+0000","last_unstale":"2024-09-17T07:21:26.415066+0000","last_undegraded":"2024-09-17T07:21:26.415066+0000","last_fullsized":"2024-09-17T07:21:26.415066+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:56.146098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:14:56.146098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0054479960000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"422'10675","reported_seq":10716,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-17T07:21:38.983677+0000","last_change":"2024-09-17T07:19:55.482992+0000","last_active":"2024-09-17T07:21:38.983677+0000","last_peered":"2024-09-17T07:21:38.983677+0000","last_clean":"2024-09-17T07:21:38.983677+0000","last_became_active":"2024-09-17T07:19:55.482080+0000","last_became_peered":"2024-09-17T07:19:55.482080+0000","last_unstale":"2024-09-17T07:21:38.983677+0000","last_undegraded":"2024-09-17T07:21:38.983677+0000","last_fullsized":"2024-09-17T07:21:38.983677+0000","mapping_epoch":390,"log_start":"409'7000","ondisk_log_start":"409'7000","created":390,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_clean_scrub_stamp":"2024-09-17T07:19:54.447320+0000","objects_scrubbed":0,"log_size":3675,"log_dups_size":0,"ondisk_log_size":3675,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":699596800,"num_objects":10675,"num_object_clones":0,"num_object_copies":32025,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10675,"num_write_kb":683200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"242'19","reported_seq":483,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:29.180828+0000","last_change":"2024-09-17T07:21:29.494226+0000","last_active":"2024-09-17T07:21:29.180828+0000","last_peered":"2024-09-17T07:21:29.180828+0000","last_clean":"2024-09-17T07:21:29.180828+0000","last_became_active":"2024-09-17T07:21:15.347662+0000","last_became_peered":"2024-09-17T07:21:15.347662+0000","last_unstale":"2024-09-17T07:21:29.180828+0000","last_undegraded":"2024-09-17T07:21:29.180828+0000","last_fullsized":"2024-09-17T07:21:29.180828+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'19","last_scrub_stamp":"2024-09-17T07:14:55.489874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:14:55.489874+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045924440000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":378,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.414718+0000","last_change":"2024-09-17T07:21:16.375330+0000","last_active":"2024-09-17T07:21:26.414718+0000","last_peered":"2024-09-17T07:21:26.414718+0000","last_clean":"2024-09-17T07:21:26.414718+0000","last_became_active":"2024-09-17T07:21:16.372230+0000","last_became_peered":"2024-09-17T07:21:16.372230+0000","last_unstale":"2024-09-17T07:21:26.414718+0000","last_undegraded":"2024-09-17T07:21:26.414718+0000","last_fullsized":"2024-09-17T07:21:26.414718+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:12.442421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:20:12.442421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:21:35.898450+0000","scrub_duration":0.0026862600000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":465,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.416419+0000","last_change":"2024-09-17T07:19:30.427362+0000","last_active":"2024-09-17T07:21:26.416419+0000","last_peered":"2024-09-17T07:21:26.416419+0000","last_clean":"2024-09-17T07:21:26.416419+0000","last_became_active":"2024-09-17T07:02:16.020224+0000","last_became_peered":"2024-09-17T07:02:16.020224+0000","last_unstale":"2024-09-17T07:21:26.416419+0000","last_undegraded":"2024-09-17T07:21:26.416419+0000","last_fullsized":"2024-09-17T07:21:26.416419+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:30.427245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:19:30.427245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010614099,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":328,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.416021+0000","last_change":"2024-09-17T07:21:16.375075+0000","last_active":"2024-09-17T07:21:26.416021+0000","last_peered":"2024-09-17T07:21:26.416021+0000","last_clean":"2024-09-17T07:21:26.416021+0000","last_became_active":"2024-09-17T07:21:16.374762+0000","last_became_peered":"2024-09-17T07:21:16.374762+0000","last_unstale":"2024-09-17T07:21:26.416021+0000","last_undegraded":"2024-09-17T07:21:26.416021+0000","last_fullsized":"2024-09-17T07:21:26.416021+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:18:40.925918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:13:22.187127+0000","last_clean_scrub_stamp":"2024-09-17T07:18:40.925918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0076924289999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.e","version":"422'10828","reported_seq":10877,"reported_epoch":422,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2024-09-17T07:21:38.378181+0000","last_change":"2024-09-17T07:21:16.491043+0000","last_active":"2024-09-17T07:21:38.378181+0000","last_peered":"2024-09-17T07:21:38.378181+0000","last_clean":"2024-09-17T07:19:54.447320+0000","last_became_active":"2024-09-17T07:21:16.483642+0000","last_became_peered":"2024-09-17T07:21:16.483642+0000","last_unstale":"2024-09-17T07:21:38.378181+0000","last_undegraded":"2024-09-17T07:21:16.347431+0000","last_fullsized":"2024-09-17T07:21:16.333928+0000","mapping_epoch":414,"log_start":"409'6875","ondisk_log_start":"409'6875","created":390,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_clean_scrub_stamp":"2024-09-17T07:19:54.447320+0000","objects_scrubbed":0,"log_size":3953,"log_dups_size":0,"ondisk_log_size":3953,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":709623808,"num_objects":10828,"num_object_clones":0,"num_object_copies":32484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10828,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10828,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10828,"num_write_kb":692992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[{"shards":"0,4","objects":10828}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"212'130","reported_seq":536,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.415378+0000","last_change":"2024-09-17T07:20:40.628867+0000","last_active":"2024-09-17T07:21:26.415378+0000","last_peered":"2024-09-17T07:21:26.415378+0000","last_clean":"2024-09-17T07:21:26.415378+0000","last_became_active":"2024-09-17T07:19:45.354802+0000","last_became_peered":"2024-09-17T07:19:45.354802+0000","last_unstale":"2024-09-17T07:21:26.415378+0000","last_undegraded":"2024-09-17T07:21:26.415378+0000","last_fullsized":"2024-09-17T07:21:26.415378+0000","mapping_epoch":386,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":387,"parent":"0.0","parent_split_bits":3,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:20:40.628770+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:14:57.052505+0000","last_clean_scrub_stamp":"2024-09-17T07:20:40.628770+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:21:51.411032+0000","scrub_duration":0.0027179940000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1","version":"242'28","reported_seq":490,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.408982+0000","last_change":"2024-09-17T07:20:28.029486+0000","last_active":"2024-09-17T07:21:26.408982+0000","last_peered":"2024-09-17T07:21:26.408982+0000","last_clean":"2024-09-17T07:21:26.408982+0000","last_became_active":"2024-09-17T07:02:14.007810+0000","last_became_peered":"2024-09-17T07:02:14.007810+0000","last_unstale":"2024-09-17T07:21:26.408982+0000","last_undegraded":"2024-09-17T07:21:26.408982+0000","last_fullsized":"2024-09-17T07:21:26.408982+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'28","last_scrub_stamp":"2024-09-17T07:20:28.029279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:20:28.029279+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:21:43.746284+0000","scrub_duration":0.0079911889999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":510,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.417051+0000","last_change":"2024-09-17T07:21:16.352607+0000","last_active":"2024-09-17T07:21:26.417051+0000","last_peered":"2024-09-17T07:21:26.417051+0000","last_clean":"2024-09-17T07:21:26.417051+0000","last_became_active":"2024-09-17T07:21:16.351776+0000","last_became_peered":"2024-09-17T07:21:16.351776+0000","last_unstale":"2024-09-17T07:21:26.417051+0000","last_undegraded":"2024-09-17T07:21:26.417051+0000","last_fullsized":"2024-09-17T07:21:26.417051+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:27.831228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:09:23.181028+0000","last_clean_scrub_stamp":"2024-09-17T07:19:27.831228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002288249,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"414'11","reported_seq":264,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-17T07:21:30.742957+0000","last_change":"2024-09-17T07:21:15.356155+0000","last_active":"2024-09-17T07:21:30.742957+0000","last_peered":"2024-09-17T07:21:30.742957+0000","last_clean":"2024-09-17T07:21:30.742957+0000","last_became_active":"2024-09-17T07:21:15.355651+0000","last_became_peered":"2024-09-17T07:21:15.355651+0000","last_unstale":"2024-09-17T07:21:30.742957+0000","last_undegraded":"2024-09-17T07:21:30.742957+0000","last_fullsized":"2024-09-17T07:21:30.742957+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'9","last_scrub_stamp":"2024-09-17T07:15:55.813882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:15:55.813882+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":61,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.415652+0000","last_change":"2024-09-17T07:21:16.377825+0000","last_active":"2024-09-17T07:21:26.415652+0000","last_peered":"2024-09-17T07:21:26.415652+0000","last_clean":"2024-09-17T07:21:26.415652+0000","last_became_active":"2024-09-17T07:21:16.377114+0000","last_became_peered":"2024-09-17T07:21:16.377114+0000","last_unstale":"2024-09-17T07:21:26.415652+0000","last_undegraded":"2024-09-17T07:21:26.415652+0000","last_fullsized":"2024-09-17T07:21:26.415652+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:18:46.921412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:10:33.727472+0000","last_clean_scrub_stamp":"2024-09-17T07:18:46.921412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"422'10764","reported_seq":10808,"reported_epoch":422,"state":"active+undersized+degraded+remapped+backfill_wait","last_fresh":"2024-09-17T07:21:39.815106+0000","last_change":"2024-09-17T07:21:16.502614+0000","last_active":"2024-09-17T07:21:39.815106+0000","last_peered":"2024-09-17T07:21:39.815106+0000","last_clean":"2024-09-17T07:19:54.447320+0000","last_became_active":"2024-09-17T07:21:16.495757+0000","last_became_peered":"2024-09-17T07:21:16.495757+0000","last_unstale":"2024-09-17T07:21:39.815106+0000","last_undegraded":"2024-09-17T07:21:16.361169+0000","last_fullsized":"2024-09-17T07:21:16.340303+0000","mapping_epoch":414,"log_start":"409'6800","ondisk_log_start":"409'6800","created":390,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_clean_scrub_stamp":"2024-09-17T07:19:54.447320+0000","objects_scrubbed":0,"log_size":3964,"log_dups_size":0,"ondisk_log_size":3964,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":705429504,"num_objects":10764,"num_object_clones":0,"num_object_copies":32292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10764,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10764,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10764,"num_write_kb":688896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":10764}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"212'130","reported_seq":515,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.407960+0000","last_change":"2024-09-17T07:20:35.080459+0000","last_active":"2024-09-17T07:21:26.407960+0000","last_peered":"2024-09-17T07:21:26.407960+0000","last_clean":"2024-09-17T07:21:26.407960+0000","last_became_active":"2024-09-17T07:19:35.420337+0000","last_became_peered":"2024-09-17T07:19:35.420337+0000","last_unstale":"2024-09-17T07:21:26.407960+0000","last_undegraded":"2024-09-17T07:21:26.407960+0000","last_fullsized":"2024-09-17T07:21:26.407960+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":3,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:20:35.080257+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:20:35.080257+0000","last_clean_scrub_stamp":"2024-09-17T07:20:35.080257+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:21:54.847799+0000","scrub_duration":0.0030477730000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"242'30","reported_seq":238,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.414950+0000","last_change":"2024-09-17T07:21:16.425898+0000","last_active":"2024-09-17T07:21:26.414950+0000","last_peered":"2024-09-17T07:21:26.414950+0000","last_clean":"2024-09-17T07:21:26.414950+0000","last_became_active":"2024-09-17T07:21:16.425217+0000","last_became_peered":"2024-09-17T07:21:16.425217+0000","last_unstale":"2024-09-17T07:21:26.414950+0000","last_undegraded":"2024-09-17T07:21:26.414950+0000","last_fullsized":"2024-09-17T07:21:26.414950+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'30","last_scrub_stamp":"2024-09-17T07:20:13.427290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:20:13.427290+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0050927619999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":327,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.421130+0000","last_change":"2024-09-17T07:21:15.348172+0000","last_active":"2024-09-17T07:21:26.421130+0000","last_peered":"2024-09-17T07:21:26.421130+0000","last_clean":"2024-09-17T07:21:26.421130+0000","last_became_active":"2024-09-17T07:21:15.347867+0000","last_became_peered":"2024-09-17T07:21:15.347867+0000","last_unstale":"2024-09-17T07:21:26.421130+0000","last_undegraded":"2024-09-17T07:21:26.421130+0000","last_fullsized":"2024-09-17T07:21:26.421130+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:56.848689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:56.848689+0000","last_clean_scrub_stamp":"2024-09-17T07:15:56.848689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":464,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.408816+0000","last_change":"2024-09-17T07:19:42.530918+0000","last_active":"2024-09-17T07:21:26.408816+0000","last_peered":"2024-09-17T07:21:26.408816+0000","last_clean":"2024-09-17T07:21:26.408816+0000","last_became_active":"2024-09-17T07:19:42.530469+0000","last_became_peered":"2024-09-17T07:19:42.530469+0000","last_unstale":"2024-09-17T07:21:26.408816+0000","last_undegraded":"2024-09-17T07:21:26.408816+0000","last_fullsized":"2024-09-17T07:21:26.408816+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:21.979680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:19:21.979680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0094045190000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":453,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.408469+0000","last_change":"2024-09-17T07:19:27.090311+0000","last_active":"2024-09-17T07:21:26.408469+0000","last_peered":"2024-09-17T07:21:26.408469+0000","last_clean":"2024-09-17T07:21:26.408469+0000","last_became_active":"2024-09-17T07:09:24.398659+0000","last_became_peered":"2024-09-17T07:09:24.398659+0000","last_unstale":"2024-09-17T07:21:26.408469+0000","last_undegraded":"2024-09-17T07:21:26.408469+0000","last_fullsized":"2024-09-17T07:21:26.408469+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:27.090201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:19:27.090201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026497019999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"422'10584","reported_seq":10629,"reported_epoch":422,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2024-09-17T07:21:38.104888+0000","last_change":"2024-09-17T07:21:16.482431+0000","last_active":"2024-09-17T07:21:38.104888+0000","last_peered":"2024-09-17T07:21:38.104888+0000","last_clean":"2024-09-17T07:19:54.447320+0000","last_became_active":"2024-09-17T07:21:16.466676+0000","last_became_peered":"2024-09-17T07:21:16.466676+0000","last_unstale":"2024-09-17T07:21:38.104888+0000","last_undegraded":"2024-09-17T07:21:16.347636+0000","last_fullsized":"2024-09-17T07:21:16.334665+0000","mapping_epoch":414,"log_start":"406'6300","ondisk_log_start":"406'6300","created":390,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_clean_scrub_stamp":"2024-09-17T07:19:54.447320+0000","objects_scrubbed":0,"log_size":4284,"log_dups_size":0,"ondisk_log_size":4284,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":693633024,"num_objects":10584,"num_object_clones":0,"num_object_copies":31752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10584,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10584,"num_write_kb":677376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[{"shards":"2,4","objects":10584}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"212'130","reported_seq":523,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.416502+0000","last_change":"2024-09-17T07:20:33.068709+0000","last_active":"2024-09-17T07:21:26.416502+0000","last_peered":"2024-09-17T07:21:26.416502+0000","last_clean":"2024-09-17T07:21:26.416502+0000","last_became_active":"2024-09-17T07:19:35.428161+0000","last_became_peered":"2024-09-17T07:19:35.428161+0000","last_unstale":"2024-09-17T07:21:26.416502+0000","last_undegraded":"2024-09-17T07:21:26.416502+0000","last_fullsized":"2024-09-17T07:21:26.416502+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":3,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:20:33.068618+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:14:57.052505+0000","last_clean_scrub_stamp":"2024-09-17T07:20:33.068618+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:21:38.565923+0000","scrub_duration":0.0059947730000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"242'29","reported_seq":537,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.413009+0000","last_change":"2024-09-17T07:21:16.564907+0000","last_active":"2024-09-17T07:21:26.413009+0000","last_peered":"2024-09-17T07:21:26.413009+0000","last_clean":"2024-09-17T07:21:26.413009+0000","last_became_active":"2024-09-17T07:21:16.503431+0000","last_became_peered":"2024-09-17T07:21:16.503431+0000","last_unstale":"2024-09-17T07:21:26.413009+0000","last_undegraded":"2024-09-17T07:21:26.413009+0000","last_fullsized":"2024-09-17T07:21:26.413009+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'29","last_scrub_stamp":"2024-09-17T07:19:35.438907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:19:35.438907+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010401474000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":61,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.413590+0000","last_change":"2024-09-17T07:21:16.356654+0000","last_active":"2024-09-17T07:21:26.413590+0000","last_peered":"2024-09-17T07:21:26.413590+0000","last_clean":"2024-09-17T07:21:26.413590+0000","last_became_active":"2024-09-17T07:21:16.355575+0000","last_became_peered":"2024-09-17T07:21:16.355575+0000","last_unstale":"2024-09-17T07:21:26.413590+0000","last_undegraded":"2024-09-17T07:21:26.413590+0000","last_fullsized":"2024-09-17T07:21:26.413590+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:37.930831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:47.631387+0000","last_clean_scrub_stamp":"2024-09-17T07:19:37.930831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":466,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.413625+0000","last_change":"2024-09-17T07:19:31.968735+0000","last_active":"2024-09-17T07:21:26.413625+0000","last_peered":"2024-09-17T07:21:26.413625+0000","last_clean":"2024-09-17T07:21:26.413625+0000","last_became_active":"2024-09-17T07:02:16.016199+0000","last_became_peered":"2024-09-17T07:02:16.016199+0000","last_unstale":"2024-09-17T07:21:26.413625+0000","last_undegraded":"2024-09-17T07:21:26.413625+0000","last_fullsized":"2024-09-17T07:21:26.413625+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:31.968422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:11:17.583712+0000","last_clean_scrub_stamp":"2024-09-17T07:19:31.968422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.076819294999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":282,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.419248+0000","last_change":"2024-09-17T07:21:15.351926+0000","last_active":"2024-09-17T07:21:26.419248+0000","last_peered":"2024-09-17T07:21:26.419248+0000","last_clean":"2024-09-17T07:21:26.419248+0000","last_became_active":"2024-09-17T07:21:15.351524+0000","last_became_peered":"2024-09-17T07:21:15.351524+0000","last_unstale":"2024-09-17T07:21:26.419248+0000","last_undegraded":"2024-09-17T07:21:26.419248+0000","last_fullsized":"2024-09-17T07:21:26.419248+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:50.514525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:13:50.514525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.d","version":"422'10578","reported_seq":10619,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-17T07:21:38.369233+0000","last_change":"2024-09-17T07:19:55.480674+0000","last_active":"2024-09-17T07:21:38.369233+0000","last_peered":"2024-09-17T07:21:38.369233+0000","last_clean":"2024-09-17T07:21:38.369233+0000","last_became_active":"2024-09-17T07:19:55.480318+0000","last_became_peered":"2024-09-17T07:19:55.480318+0000","last_unstale":"2024-09-17T07:21:38.369233+0000","last_undegraded":"2024-09-17T07:21:38.369233+0000","last_fullsized":"2024-09-17T07:21:38.369233+0000","mapping_epoch":390,"log_start":"408'6582","ondisk_log_start":"408'6582","created":390,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_clean_scrub_stamp":"2024-09-17T07:19:54.447320+0000","objects_scrubbed":0,"log_size":3996,"log_dups_size":0,"ondisk_log_size":3996,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":693239808,"num_objects":10578,"num_object_clones":0,"num_object_copies":31734,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10578,"num_write_kb":676992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"212'130","reported_seq":555,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.416738+0000","last_change":"2024-09-17T07:21:18.359107+0000","last_active":"2024-09-17T07:21:26.416738+0000","last_peered":"2024-09-17T07:21:26.416738+0000","last_clean":"2024-09-17T07:21:26.416738+0000","last_became_active":"2024-09-17T07:21:18.358380+0000","last_became_peered":"2024-09-17T07:21:18.358380+0000","last_unstale":"2024-09-17T07:21:26.416738+0000","last_undegraded":"2024-09-17T07:21:26.416738+0000","last_fullsized":"2024-09-17T07:21:26.416738+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":3,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:19:19.991044+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:14:57.052505+0000","last_clean_scrub_stamp":"2024-09-17T07:19:19.991044+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021141889999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":28,"num_write":16,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":161009,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"242'25","reported_seq":333,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.419855+0000","last_change":"2024-09-17T07:21:15.358553+0000","last_active":"2024-09-17T07:21:26.419855+0000","last_peered":"2024-09-17T07:21:26.419855+0000","last_clean":"2024-09-17T07:21:26.419855+0000","last_became_active":"2024-09-17T07:21:15.358209+0000","last_became_peered":"2024-09-17T07:21:15.358209+0000","last_unstale":"2024-09-17T07:21:26.419855+0000","last_undegraded":"2024-09-17T07:21:26.419855+0000","last_fullsized":"2024-09-17T07:21:26.419855+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'25","last_scrub_stamp":"2024-09-17T07:14:50.794200+0000","last_deep_scrub":"110'15","last_deep_scrub_stamp":"2024-09-17T07:04:57.091400+0000","last_clean_scrub_stamp":"2024-09-17T07:14:50.794200+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0070558269999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":298,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.420479+0000","last_change":"2024-09-17T07:21:16.356490+0000","last_active":"2024-09-17T07:21:26.420479+0000","last_peered":"2024-09-17T07:21:26.420479+0000","last_clean":"2024-09-17T07:21:26.420479+0000","last_became_active":"2024-09-17T07:21:16.355839+0000","last_became_peered":"2024-09-17T07:21:16.355839+0000","last_unstale":"2024-09-17T07:21:26.420479+0000","last_undegraded":"2024-09-17T07:21:26.420479+0000","last_fullsized":"2024-09-17T07:21:26.420479+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:24.437342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:25.147878+0000","last_clean_scrub_stamp":"2024-09-17T07:20:24.437342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:21:44.002579+0000","scrub_duration":0.0024354329999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"89'9","reported_seq":698,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-17T07:21:30.745212+0000","last_change":"2024-09-17T07:21:14.471392+0000","last_active":"2024-09-17T07:21:30.745212+0000","last_peered":"2024-09-17T07:21:30.745212+0000","last_clean":"2024-09-17T07:21:30.745212+0000","last_became_active":"2024-09-17T07:02:16.021360+0000","last_became_peered":"2024-09-17T07:02:16.021360+0000","last_unstale":"2024-09-17T07:21:30.745212+0000","last_undegraded":"2024-09-17T07:21:30.745212+0000","last_fullsized":"2024-09-17T07:21:30.745212+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'9","last_scrub_stamp":"2024-09-17T07:21:14.471255+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-09-17T07:15:53.512777+0000","last_clean_scrub_stamp":"2024-09-17T07:21:14.471255+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:22:20.341284+0000","scrub_duration":0.0032915969999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"268'2","reported_seq":458,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.415111+0000","last_change":"2024-09-17T07:18:49.469622+0000","last_active":"2024-09-17T07:21:26.415111+0000","last_peered":"2024-09-17T07:21:26.415111+0000","last_clean":"2024-09-17T07:21:26.415111+0000","last_became_active":"2024-09-17T07:09:24.399565+0000","last_became_peered":"2024-09-17T07:09:24.399565+0000","last_unstale":"2024-09-17T07:21:26.415111+0000","last_undegraded":"2024-09-17T07:21:26.415111+0000","last_fullsized":"2024-09-17T07:21:26.415111+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'2","last_scrub_stamp":"2024-09-17T07:18:49.469494+0000","last_deep_scrub":"268'2","last_deep_scrub_stamp":"2024-09-17T07:14:03.609131+0000","last_clean_scrub_stamp":"2024-09-17T07:18:49.469494+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037058479999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"422'10554","reported_seq":10597,"reported_epoch":422,"state":"active+undersized+degraded+remapped+backfill_wait","last_fresh":"2024-09-17T07:21:39.826019+0000","last_change":"2024-09-17T07:21:16.503056+0000","last_active":"2024-09-17T07:21:39.826019+0000","last_peered":"2024-09-17T07:21:39.826019+0000","last_clean":"2024-09-17T07:19:54.447320+0000","last_became_active":"2024-09-17T07:21:16.493516+0000","last_became_peered":"2024-09-17T07:21:16.493516+0000","last_unstale":"2024-09-17T07:21:39.826019+0000","last_undegraded":"2024-09-17T07:21:16.369606+0000","last_fullsized":"2024-09-17T07:21:16.338987+0000","mapping_epoch":414,"log_start":"409'6600","ondisk_log_start":"409'6600","created":390,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_clean_scrub_stamp":"2024-09-17T07:19:54.447320+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":691666944,"num_objects":10554,"num_object_clones":0,"num_object_copies":31662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10554,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10554,"num_write_kb":675456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0,4","objects":10554}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"212'130","reported_seq":514,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.414561+0000","last_change":"2024-09-17T07:20:22.674470+0000","last_active":"2024-09-17T07:21:26.414561+0000","last_peered":"2024-09-17T07:21:26.414561+0000","last_clean":"2024-09-17T07:21:26.414561+0000","last_became_active":"2024-09-17T07:19:35.422795+0000","last_became_peered":"2024-09-17T07:19:35.422795+0000","last_unstale":"2024-09-17T07:21:26.414561+0000","last_undegraded":"2024-09-17T07:21:26.414561+0000","last_fullsized":"2024-09-17T07:21:26.414561+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":4,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:20:22.674350+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:14:57.052505+0000","last_clean_scrub_stamp":"2024-09-17T07:20:22.674350+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033559660000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"414'43","reported_seq":555,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-17T07:21:30.744258+0000","last_change":"2024-09-17T07:21:15.352832+0000","last_active":"2024-09-17T07:21:30.744258+0000","last_peered":"2024-09-17T07:21:30.744258+0000","last_clean":"2024-09-17T07:21:30.744258+0000","last_became_active":"2024-09-17T07:21:15.352461+0000","last_became_peered":"2024-09-17T07:21:15.352461+0000","last_unstale":"2024-09-17T07:21:30.744258+0000","last_undegraded":"2024-09-17T07:21:30.744258+0000","last_fullsized":"2024-09-17T07:21:30.744258+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'41","last_scrub_stamp":"2024-09-17T07:14:56.700858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:14:56.700858+0000","objects_scrubbed":10,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026058610000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":453,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.415030+0000","last_change":"2024-09-17T07:21:17.336331+0000","last_active":"2024-09-17T07:21:26.415030+0000","last_peered":"2024-09-17T07:21:26.415030+0000","last_clean":"2024-09-17T07:21:26.415030+0000","last_became_active":"2024-09-17T07:21:15.351163+0000","last_became_peered":"2024-09-17T07:21:15.351163+0000","last_unstale":"2024-09-17T07:21:26.415030+0000","last_undegraded":"2024-09-17T07:21:26.415030+0000","last_fullsized":"2024-09-17T07:21:26.415030+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:03.045709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:14:03.045709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0049245929999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":449,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.413673+0000","last_change":"2024-09-17T07:21:15.347484+0000","last_active":"2024-09-17T07:21:26.413673+0000","last_peered":"2024-09-17T07:21:26.413673+0000","last_clean":"2024-09-17T07:21:26.413673+0000","last_became_active":"2024-09-17T07:21:15.347069+0000","last_became_peered":"2024-09-17T07:21:15.347069+0000","last_unstale":"2024-09-17T07:21:26.413673+0000","last_undegraded":"2024-09-17T07:21:26.413673+0000","last_fullsized":"2024-09-17T07:21:26.413673+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:36.523676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:15:36.523676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017370549999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":328,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.409434+0000","last_change":"2024-09-17T07:21:16.348479+0000","last_active":"2024-09-17T07:21:26.409434+0000","last_peered":"2024-09-17T07:21:26.409434+0000","last_clean":"2024-09-17T07:21:26.409434+0000","last_became_active":"2024-09-17T07:21:16.347868+0000","last_became_peered":"2024-09-17T07:21:16.347868+0000","last_unstale":"2024-09-17T07:21:26.409434+0000","last_undegraded":"2024-09-17T07:21:26.409434+0000","last_fullsized":"2024-09-17T07:21:26.409434+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:20.067928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:13:53.406661+0000","last_clean_scrub_stamp":"2024-09-17T07:20:20.067928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0042634170000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.8","version":"422'10741","reported_seq":10782,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-17T07:21:39.001437+0000","last_change":"2024-09-17T07:19:55.484156+0000","last_active":"2024-09-17T07:21:39.001437+0000","last_peered":"2024-09-17T07:21:39.001437+0000","last_clean":"2024-09-17T07:21:39.001437+0000","last_became_active":"2024-09-17T07:19:55.483829+0000","last_became_peered":"2024-09-17T07:19:55.483829+0000","last_unstale":"2024-09-17T07:21:39.001437+0000","last_undegraded":"2024-09-17T07:21:39.001437+0000","last_fullsized":"2024-09-17T07:21:39.001437+0000","mapping_epoch":390,"log_start":"409'7000","ondisk_log_start":"409'7000","created":390,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_clean_scrub_stamp":"2024-09-17T07:19:54.447320+0000","objects_scrubbed":0,"log_size":3741,"log_dups_size":0,"ondisk_log_size":3741,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":703922176,"num_objects":10741,"num_object_clones":0,"num_object_copies":32223,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10741,"num_write_kb":687424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.2","version":"212'130","reported_seq":502,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.415645+0000","last_change":"2024-09-17T07:19:45.393280+0000","last_active":"2024-09-17T07:21:26.415645+0000","last_peered":"2024-09-17T07:21:26.415645+0000","last_clean":"2024-09-17T07:21:26.415645+0000","last_became_active":"2024-09-17T07:19:45.389415+0000","last_became_peered":"2024-09-17T07:19:45.389415+0000","last_unstale":"2024-09-17T07:21:26.415645+0000","last_undegraded":"2024-09-17T07:21:26.415645+0000","last_fullsized":"2024-09-17T07:21:26.415645+0000","mapping_epoch":386,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":387,"parent":"0.0","parent_split_bits":4,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:19:19.991044+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:14:57.052505+0000","last_clean_scrub_stamp":"2024-09-17T07:19:19.991044+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021141889999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":28,"num_write":17,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":161010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"242'17","reported_seq":318,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.423347+0000","last_change":"2024-09-17T07:21:15.351304+0000","last_active":"2024-09-17T07:21:26.423347+0000","last_peered":"2024-09-17T07:21:26.423347+0000","last_clean":"2024-09-17T07:21:26.423347+0000","last_became_active":"2024-09-17T07:21:15.350162+0000","last_became_peered":"2024-09-17T07:21:15.350162+0000","last_unstale":"2024-09-17T07:21:26.423347+0000","last_undegraded":"2024-09-17T07:21:26.423347+0000","last_fullsized":"2024-09-17T07:21:26.423347+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'17","last_scrub_stamp":"2024-09-17T07:13:47.468399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:13:47.468399+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.005881725,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"76'5","reported_seq":1140,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-17T07:21:37.417507+0000","last_change":"2024-09-17T07:21:15.358093+0000","last_active":"2024-09-17T07:21:37.417507+0000","last_peered":"2024-09-17T07:21:37.417507+0000","last_clean":"2024-09-17T07:21:37.417507+0000","last_became_active":"2024-09-17T07:21:15.357178+0000","last_became_peered":"2024-09-17T07:21:15.357178+0000","last_unstale":"2024-09-17T07:21:37.417507+0000","last_undegraded":"2024-09-17T07:21:37.417507+0000","last_fullsized":"2024-09-17T07:21:37.417507+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'5","last_scrub_stamp":"2024-09-17T07:15:36.901601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:15:36.901601+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033739799999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":293,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.414170+0000","last_change":"2024-09-17T07:21:15.346533+0000","last_active":"2024-09-17T07:21:26.414170+0000","last_peered":"2024-09-17T07:21:26.414170+0000","last_clean":"2024-09-17T07:21:26.414170+0000","last_became_active":"2024-09-17T07:21:15.346287+0000","last_became_peered":"2024-09-17T07:21:15.346287+0000","last_unstale":"2024-09-17T07:21:26.414170+0000","last_undegraded":"2024-09-17T07:21:26.414170+0000","last_fullsized":"2024-09-17T07:21:26.414170+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:56.534547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:23.833170+0000","last_clean_scrub_stamp":"2024-09-17T07:15:56.534547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0011018950000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":487,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.417662+0000","last_change":"2024-09-17T07:20:16.776682+0000","last_active":"2024-09-17T07:21:26.417662+0000","last_peered":"2024-09-17T07:21:26.417662+0000","last_clean":"2024-09-17T07:21:26.417662+0000","last_became_active":"2024-09-17T07:19:42.507255+0000","last_became_peered":"2024-09-17T07:19:42.507255+0000","last_unstale":"2024-09-17T07:21:26.417662+0000","last_undegraded":"2024-09-17T07:21:26.417662+0000","last_fullsized":"2024-09-17T07:21:26.417662+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:16.776553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:18:42.051667+0000","last_clean_scrub_stamp":"2024-09-17T07:20:16.776553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:21:43.134776+0000","scrub_duration":0.0035831560000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":228,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.419640+0000","last_change":"2024-09-17T07:19:42.535205+0000","last_active":"2024-09-17T07:21:26.419640+0000","last_peered":"2024-09-17T07:21:26.419640+0000","last_clean":"2024-09-17T07:21:26.419640+0000","last_became_active":"2024-09-17T07:19:42.533647+0000","last_became_peered":"2024-09-17T07:19:42.533647+0000","last_unstale":"2024-09-17T07:21:26.419640+0000","last_undegraded":"2024-09-17T07:21:26.419640+0000","last_fullsized":"2024-09-17T07:21:26.419640+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:27.909402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:02.062618+0000","last_clean_scrub_stamp":"2024-09-17T07:19:27.909402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0041575259999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.9","version":"422'10668","reported_seq":17901,"reported_epoch":422,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-17T07:21:39.026371+0000","last_change":"2024-09-17T07:21:16.499021+0000","last_active":"2024-09-17T07:21:39.026371+0000","last_peered":"2024-09-17T07:21:39.026371+0000","last_clean":"2024-09-17T07:19:54.447320+0000","last_became_active":"2024-09-17T07:21:16.485519+0000","last_became_peered":"2024-09-17T07:21:16.485519+0000","last_unstale":"2024-09-17T07:21:39.026371+0000","last_undegraded":"2024-09-17T07:21:16.366475+0000","last_fullsized":"2024-09-17T07:21:16.340496+0000","mapping_epoch":414,"log_start":"409'7000","ondisk_log_start":"409'7000","created":390,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_clean_scrub_stamp":"2024-09-17T07:19:54.447320+0000","objects_scrubbed":0,"log_size":3668,"log_dups_size":0,"ondisk_log_size":3668,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":699138048,"num_objects":10668,"num_object_clones":0,"num_object_copies":32004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6693,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10668,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10668,"num_write_kb":682752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3594,"num_bytes_recovered":235536384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[{"shards":"0,6","objects":10668}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.3","version":"212'130","reported_seq":535,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.417570+0000","last_change":"2024-09-17T07:21:18.358222+0000","last_active":"2024-09-17T07:21:26.417570+0000","last_peered":"2024-09-17T07:21:26.417570+0000","last_clean":"2024-09-17T07:21:26.417570+0000","last_became_active":"2024-09-17T07:21:18.357874+0000","last_became_peered":"2024-09-17T07:21:18.357874+0000","last_unstale":"2024-09-17T07:21:26.417570+0000","last_undegraded":"2024-09-17T07:21:26.417570+0000","last_fullsized":"2024-09-17T07:21:26.417570+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":3,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:20:34.113602+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:20:34.113602+0000","last_clean_scrub_stamp":"2024-09-17T07:20:34.113602+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021141889999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":28,"num_write":17,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":161010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"268'30","reported_seq":511,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.413615+0000","last_change":"2024-09-17T07:18:40.426710+0000","last_active":"2024-09-17T07:21:26.413615+0000","last_peered":"2024-09-17T07:21:26.413615+0000","last_clean":"2024-09-17T07:21:26.413615+0000","last_became_active":"2024-09-17T07:03:48.760405+0000","last_became_peered":"2024-09-17T07:03:48.760405+0000","last_unstale":"2024-09-17T07:21:26.413615+0000","last_undegraded":"2024-09-17T07:21:26.413615+0000","last_fullsized":"2024-09-17T07:21:26.413615+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'30","last_scrub_stamp":"2024-09-17T07:18:40.426597+0000","last_deep_scrub":"268'30","last_deep_scrub_stamp":"2024-09-17T07:14:00.599656+0000","last_clean_scrub_stamp":"2024-09-17T07:18:40.426597+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0031470769999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"70'1","reported_seq":468,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.413970+0000","last_change":"2024-09-17T07:19:19.477270+0000","last_active":"2024-09-17T07:21:26.413970+0000","last_peered":"2024-09-17T07:21:26.413970+0000","last_clean":"2024-09-17T07:21:26.413970+0000","last_became_active":"2024-09-17T07:09:24.402499+0000","last_became_peered":"2024-09-17T07:09:24.402499+0000","last_unstale":"2024-09-17T07:21:26.413970+0000","last_undegraded":"2024-09-17T07:21:26.413970+0000","last_fullsized":"2024-09-17T07:21:26.413970+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-17T07:19:19.477140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:19:19.477140+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011313337,"objects_trimmed":0,"snaptrim_duration":0.0032298629999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.416776+0000","last_change":"2024-09-17T07:21:16.416665+0000","last_active":"2024-09-17T07:21:26.416776+0000","last_peered":"2024-09-17T07:21:26.416776+0000","last_clean":"2024-09-17T07:21:26.416776+0000","last_became_active":"2024-09-17T07:21:16.416358+0000","last_became_peered":"2024-09-17T07:21:16.416358+0000","last_unstale":"2024-09-17T07:21:26.416776+0000","last_undegraded":"2024-09-17T07:21:26.416776+0000","last_fullsized":"2024-09-17T07:21:26.416776+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:34.449627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:55.321202+0000","last_clean_scrub_stamp":"2024-09-17T07:20:34.449627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:21:35.127991+0000","scrub_duration":0.0044019280000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":447,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.416486+0000","last_change":"2024-09-17T07:21:15.349885+0000","last_active":"2024-09-17T07:21:26.416486+0000","last_peered":"2024-09-17T07:21:26.416486+0000","last_clean":"2024-09-17T07:21:26.416486+0000","last_became_active":"2024-09-17T07:21:15.349572+0000","last_became_peered":"2024-09-17T07:21:15.349572+0000","last_unstale":"2024-09-17T07:21:26.416486+0000","last_undegraded":"2024-09-17T07:21:26.416486+0000","last_fullsized":"2024-09-17T07:21:26.416486+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:45.447029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:15:45.447029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001990463,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":319,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.420831+0000","last_change":"2024-09-17T07:21:15.351211+0000","last_active":"2024-09-17T07:21:26.420831+0000","last_peered":"2024-09-17T07:21:26.420831+0000","last_clean":"2024-09-17T07:21:26.420831+0000","last_became_active":"2024-09-17T07:21:15.350190+0000","last_became_peered":"2024-09-17T07:21:15.350190+0000","last_unstale":"2024-09-17T07:21:26.420831+0000","last_undegraded":"2024-09-17T07:21:26.420831+0000","last_fullsized":"2024-09-17T07:21:26.420831+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:51.877367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:15:51.877367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.058290885000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"422'10582","reported_seq":10640,"reported_epoch":422,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2024-09-17T07:21:39.021926+0000","last_change":"2024-09-17T07:21:16.498807+0000","last_active":"2024-09-17T07:21:39.021926+0000","last_peered":"2024-09-17T07:21:39.021926+0000","last_clean":"2024-09-17T07:19:54.447320+0000","last_became_active":"2024-09-17T07:21:16.466454+0000","last_became_peered":"2024-09-17T07:21:16.466454+0000","last_unstale":"2024-09-17T07:21:39.021926+0000","last_undegraded":"2024-09-17T07:21:16.479700+0000","last_fullsized":"2024-09-17T07:21:16.335962+0000","mapping_epoch":414,"log_start":"409'6900","ondisk_log_start":"409'6900","created":390,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_clean_scrub_stamp":"2024-09-17T07:19:54.447320+0000","objects_scrubbed":0,"log_size":3682,"log_dups_size":0,"ondisk_log_size":3682,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":693501952,"num_objects":10582,"num_object_clones":0,"num_object_copies":31746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10582,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10582,"num_write_kb":677248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[{"shards":"5,6","objects":10582}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"242'26","reported_seq":524,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.419723+0000","last_change":"2024-09-17T07:20:19.046820+0000","last_active":"2024-09-17T07:21:26.419723+0000","last_peered":"2024-09-17T07:21:26.419723+0000","last_clean":"2024-09-17T07:21:26.419723+0000","last_became_active":"2024-09-17T07:19:42.515538+0000","last_became_peered":"2024-09-17T07:19:42.515538+0000","last_unstale":"2024-09-17T07:21:26.419723+0000","last_undegraded":"2024-09-17T07:21:26.419723+0000","last_fullsized":"2024-09-17T07:21:26.419723+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'26","last_scrub_stamp":"2024-09-17T07:20:19.046676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:20:19.046676+0000","objects_scrubbed":7,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:21:36.714304+0000","scrub_duration":0.0059907520000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"268'3","reported_seq":318,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.415259+0000","last_change":"2024-09-17T07:19:29.367387+0000","last_active":"2024-09-17T07:21:26.415259+0000","last_peered":"2024-09-17T07:21:26.415259+0000","last_clean":"2024-09-17T07:21:26.415259+0000","last_became_active":"2024-09-17T07:09:24.404532+0000","last_became_peered":"2024-09-17T07:09:24.404532+0000","last_unstale":"2024-09-17T07:21:26.415259+0000","last_undegraded":"2024-09-17T07:21:26.415259+0000","last_fullsized":"2024-09-17T07:21:26.415259+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'3","last_scrub_stamp":"2024-09-17T07:19:29.367269+0000","last_deep_scrub":"268'3","last_deep_scrub_stamp":"2024-09-17T07:13:54.949345+0000","last_clean_scrub_stamp":"2024-09-17T07:19:29.367269+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023388250000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":479,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.415449+0000","last_change":"2024-09-17T07:20:17.651040+0000","last_active":"2024-09-17T07:21:26.415449+0000","last_peered":"2024-09-17T07:21:26.415449+0000","last_clean":"2024-09-17T07:21:26.415449+0000","last_became_active":"2024-09-17T07:19:42.527627+0000","last_became_peered":"2024-09-17T07:19:42.527627+0000","last_unstale":"2024-09-17T07:21:26.415449+0000","last_undegraded":"2024-09-17T07:21:26.415449+0000","last_fullsized":"2024-09-17T07:21:26.415449+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:17.650824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:20:17.650824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:21:37.174085+0000","scrub_duration":0.0019547470000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"212'130","reported_seq":512,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.415029+0000","last_change":"2024-09-17T07:21:18.367570+0000","last_active":"2024-09-17T07:21:26.415029+0000","last_peered":"2024-09-17T07:21:26.415029+0000","last_clean":"2024-09-17T07:21:26.415029+0000","last_became_active":"2024-09-17T07:21:18.363003+0000","last_became_peered":"2024-09-17T07:21:18.363003+0000","last_unstale":"2024-09-17T07:21:26.415029+0000","last_undegraded":"2024-09-17T07:21:26.415029+0000","last_fullsized":"2024-09-17T07:21:26.415029+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:19:19.991044+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:14:57.052505+0000","last_clean_scrub_stamp":"2024-09-17T07:19:19.991044+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021141889999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":27,"num_write":16,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":161009,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":51,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.415780+0000","last_change":"2024-09-17T07:19:42.529064+0000","last_active":"2024-09-17T07:21:26.415780+0000","last_peered":"2024-09-17T07:21:26.415780+0000","last_clean":"2024-09-17T07:21:26.415780+0000","last_became_active":"2024-09-17T07:19:42.528802+0000","last_became_peered":"2024-09-17T07:19:42.528802+0000","last_unstale":"2024-09-17T07:21:26.415780+0000","last_undegraded":"2024-09-17T07:21:26.415780+0000","last_fullsized":"2024-09-17T07:21:26.415780+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:18:48.891136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:18:48.891136+0000","last_clean_scrub_stamp":"2024-09-17T07:18:48.891136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":443,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.416609+0000","last_change":"2024-09-17T07:21:15.348656+0000","last_active":"2024-09-17T07:21:26.416609+0000","last_peered":"2024-09-17T07:21:26.416609+0000","last_clean":"2024-09-17T07:21:26.416609+0000","last_became_active":"2024-09-17T07:21:15.344632+0000","last_became_peered":"2024-09-17T07:21:15.344632+0000","last_unstale":"2024-09-17T07:21:26.416609+0000","last_undegraded":"2024-09-17T07:21:26.416609+0000","last_fullsized":"2024-09-17T07:21:26.416609+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:48.021592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:58.670571+0000","last_clean_scrub_stamp":"2024-09-17T07:14:48.021592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0064844619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.5","version":"422'10364","reported_seq":10405,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-17T07:21:39.085683+0000","last_change":"2024-09-17T07:19:55.479299+0000","last_active":"2024-09-17T07:21:39.085683+0000","last_peered":"2024-09-17T07:21:39.085683+0000","last_clean":"2024-09-17T07:21:39.085683+0000","last_became_active":"2024-09-17T07:19:55.479023+0000","last_became_peered":"2024-09-17T07:19:55.479023+0000","last_unstale":"2024-09-17T07:21:39.085683+0000","last_undegraded":"2024-09-17T07:21:39.085683+0000","last_fullsized":"2024-09-17T07:21:39.085683+0000","mapping_epoch":390,"log_start":"409'6650","ondisk_log_start":"409'6650","created":390,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_clean_scrub_stamp":"2024-09-17T07:19:54.447320+0000","objects_scrubbed":0,"log_size":3714,"log_dups_size":0,"ondisk_log_size":3714,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":679215104,"num_objects":10364,"num_object_clones":0,"num_object_copies":31092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10364,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10364,"num_write_kb":663296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"242'35","reported_seq":558,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.414233+0000","last_change":"2024-09-17T07:21:16.373485+0000","last_active":"2024-09-17T07:21:26.414233+0000","last_peered":"2024-09-17T07:21:26.414233+0000","last_clean":"2024-09-17T07:21:26.414233+0000","last_became_active":"2024-09-17T07:21:16.371721+0000","last_became_peered":"2024-09-17T07:21:16.371721+0000","last_unstale":"2024-09-17T07:21:26.414233+0000","last_undegraded":"2024-09-17T07:21:26.414233+0000","last_fullsized":"2024-09-17T07:21:26.414233+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'35","last_scrub_stamp":"2024-09-17T07:20:15.012345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:20:15.012345+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:21:44.092993+0000","scrub_duration":0.0050499799999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":505,"reported_epoch":422,"state":"active+clean+scrubbing","last_fresh":"2024-09-17T07:21:29.506573+0000","last_change":"2024-09-17T07:21:30.915294+0000","last_active":"2024-09-17T07:21:29.506573+0000","last_peered":"2024-09-17T07:21:29.506573+0000","last_clean":"2024-09-17T07:21:29.506573+0000","last_became_active":"2024-09-17T07:02:12.012037+0000","last_became_peered":"2024-09-17T07:02:12.012037+0000","last_unstale":"2024-09-17T07:21:29.506573+0000","last_undegraded":"2024-09-17T07:21:29.506573+0000","last_fullsized":"2024-09-17T07:21:29.506573+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:17.749290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:06:39.322535+0000","last_clean_scrub_stamp":"2024-09-17T07:20:17.749290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0062185959999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":489,"reported_epoch":422,"state":"active+clean+scrubbing","last_fresh":"2024-09-17T07:21:29.506428+0000","last_change":"2024-09-17T07:21:35.970776+0000","last_active":"2024-09-17T07:21:29.506428+0000","last_peered":"2024-09-17T07:21:29.506428+0000","last_clean":"2024-09-17T07:21:29.506428+0000","last_became_active":"2024-09-17T07:02:16.014631+0000","last_became_peered":"2024-09-17T07:02:16.014631+0000","last_unstale":"2024-09-17T07:21:29.506428+0000","last_undegraded":"2024-09-17T07:21:29.506428+0000","last_fullsized":"2024-09-17T07:21:29.506428+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:12.821593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:20:12.821593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0031314899999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":217,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.415421+0000","last_change":"2024-09-17T07:19:42.505758+0000","last_active":"2024-09-17T07:21:26.415421+0000","last_peered":"2024-09-17T07:21:26.415421+0000","last_clean":"2024-09-17T07:21:26.415421+0000","last_became_active":"2024-09-17T07:19:42.505253+0000","last_became_peered":"2024-09-17T07:19:42.505253+0000","last_unstale":"2024-09-17T07:21:26.415421+0000","last_undegraded":"2024-09-17T07:21:26.415421+0000","last_fullsized":"2024-09-17T07:21:26.415421+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:18:47.880943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:45.275473+0000","last_clean_scrub_stamp":"2024-09-17T07:18:47.880943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010463343,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.4","version":"422'10717","reported_seq":17010,"reported_epoch":422,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-17T07:21:39.024616+0000","last_change":"2024-09-17T07:21:16.517698+0000","last_active":"2024-09-17T07:21:39.024616+0000","last_peered":"2024-09-17T07:21:39.024616+0000","last_clean":"2024-09-17T07:19:54.447320+0000","last_became_active":"2024-09-17T07:21:16.467119+0000","last_became_peered":"2024-09-17T07:21:16.467119+0000","last_unstale":"2024-09-17T07:21:39.024616+0000","last_undegraded":"2024-09-17T07:21:16.481156+0000","last_fullsized":"2024-09-17T07:21:16.335516+0000","mapping_epoch":414,"log_start":"409'7000","ondisk_log_start":"409'7000","created":390,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_clean_scrub_stamp":"2024-09-17T07:19:54.447320+0000","objects_scrubbed":0,"log_size":3717,"log_dups_size":0,"ondisk_log_size":3717,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":702349312,"num_objects":10717,"num_object_clones":0,"num_object_copies":32151,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10717,"num_write_kb":685888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3120,"num_bytes_recovered":204472320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[{"shards":"2,6","objects":10717}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"268'19","reported_seq":468,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.412932+0000","last_change":"2024-09-17T07:21:15.349844+0000","last_active":"2024-09-17T07:21:26.412932+0000","last_peered":"2024-09-17T07:21:26.412932+0000","last_clean":"2024-09-17T07:21:26.412932+0000","last_became_active":"2024-09-17T07:21:15.349361+0000","last_became_peered":"2024-09-17T07:21:15.349361+0000","last_unstale":"2024-09-17T07:21:26.412932+0000","last_undegraded":"2024-09-17T07:21:26.412932+0000","last_fullsized":"2024-09-17T07:21:26.412932+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'19","last_scrub_stamp":"2024-09-17T07:14:06.634334+0000","last_deep_scrub":"268'19","last_deep_scrub_stamp":"2024-09-17T07:14:06.634334+0000","last_clean_scrub_stamp":"2024-09-17T07:14:06.634334+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0088943289999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":336,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.419430+0000","last_change":"2024-09-17T07:21:15.352018+0000","last_active":"2024-09-17T07:21:26.419430+0000","last_peered":"2024-09-17T07:21:26.419430+0000","last_clean":"2024-09-17T07:21:26.419430+0000","last_became_active":"2024-09-17T07:21:15.350786+0000","last_became_peered":"2024-09-17T07:21:15.350786+0000","last_unstale":"2024-09-17T07:21:26.419430+0000","last_undegraded":"2024-09-17T07:21:26.419430+0000","last_fullsized":"2024-09-17T07:21:26.419430+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:58.886294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:13:53.593078+0000","last_clean_scrub_stamp":"2024-09-17T07:15:58.886294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.005853245,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":419,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.408268+0000","last_change":"2024-09-17T07:19:30.094643+0000","last_active":"2024-09-17T07:21:26.408268+0000","last_peered":"2024-09-17T07:21:26.408268+0000","last_clean":"2024-09-17T07:21:26.408268+0000","last_became_active":"2024-09-17T07:09:24.396642+0000","last_became_peered":"2024-09-17T07:09:24.396642+0000","last_unstale":"2024-09-17T07:21:26.408268+0000","last_undegraded":"2024-09-17T07:21:26.408268+0000","last_fullsized":"2024-09-17T07:21:26.408268+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:30.094567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:19:30.094567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001242611,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":451,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.414053+0000","last_change":"2024-09-17T07:19:42.505820+0000","last_active":"2024-09-17T07:21:26.414053+0000","last_peered":"2024-09-17T07:21:26.414053+0000","last_clean":"2024-09-17T07:21:26.414053+0000","last_became_active":"2024-09-17T07:19:42.505494+0000","last_became_peered":"2024-09-17T07:19:42.505494+0000","last_unstale":"2024-09-17T07:21:26.414053+0000","last_undegraded":"2024-09-17T07:21:26.414053+0000","last_fullsized":"2024-09-17T07:21:26.414053+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:29.434010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:58.099849+0000","last_clean_scrub_stamp":"2024-09-17T07:19:29.434010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.032605186000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.7","version":"422'10675","reported_seq":17012,"reported_epoch":422,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-17T07:21:39.084062+0000","last_change":"2024-09-17T07:21:18.369332+0000","last_active":"2024-09-17T07:21:39.084062+0000","last_peered":"2024-09-17T07:21:39.084062+0000","last_clean":"2024-09-17T07:19:54.447320+0000","last_became_active":"2024-09-17T07:21:16.495288+0000","last_became_peered":"2024-09-17T07:21:16.495288+0000","last_unstale":"2024-09-17T07:21:39.084062+0000","last_undegraded":"2024-09-17T07:21:16.348713+0000","last_fullsized":"2024-09-17T07:21:16.338417+0000","mapping_epoch":414,"log_start":"409'6984","ondisk_log_start":"409'6984","created":390,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_clean_scrub_stamp":"2024-09-17T07:19:54.447320+0000","objects_scrubbed":0,"log_size":3691,"log_dups_size":0,"ondisk_log_size":3691,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":699596800,"num_objects":10675,"num_object_clones":0,"num_object_copies":32025,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7243,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10675,"num_write_kb":683200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3145,"num_bytes_recovered":206110720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[{"shards":"4,5","objects":10675}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"242'34","reported_seq":531,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.414726+0000","last_change":"2024-09-17T07:19:42.547929+0000","last_active":"2024-09-17T07:21:26.414726+0000","last_peered":"2024-09-17T07:21:26.414726+0000","last_clean":"2024-09-17T07:21:26.414726+0000","last_became_active":"2024-09-17T07:19:42.527049+0000","last_became_peered":"2024-09-17T07:19:42.527049+0000","last_unstale":"2024-09-17T07:21:26.414726+0000","last_undegraded":"2024-09-17T07:21:26.414726+0000","last_fullsized":"2024-09-17T07:21:26.414726+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'34","last_scrub_stamp":"2024-09-17T07:19:24.308050+0000","last_deep_scrub":"242'34","last_deep_scrub_stamp":"2024-09-17T07:14:50.103539+0000","last_clean_scrub_stamp":"2024-09-17T07:19:24.308050+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.019153434,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'2","reported_seq":107,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.416334+0000","last_change":"2024-09-17T07:21:16.358723+0000","last_active":"2024-09-17T07:21:26.416334+0000","last_peered":"2024-09-17T07:21:26.416334+0000","last_clean":"2024-09-17T07:21:26.416334+0000","last_became_active":"2024-09-17T07:21:16.347260+0000","last_became_peered":"2024-09-17T07:21:16.347260+0000","last_unstale":"2024-09-17T07:21:26.416334+0000","last_undegraded":"2024-09-17T07:21:26.416334+0000","last_fullsized":"2024-09-17T07:21:26.416334+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-09-17T07:18:43.861842+0000","last_deep_scrub":"89'2","last_deep_scrub_stamp":"2024-09-17T07:12:20.301263+0000","last_clean_scrub_stamp":"2024-09-17T07:18:43.861842+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":420,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":420,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.9","version":"415'13","reported_seq":416,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-17T07:21:30.747073+0000","last_change":"2024-09-17T07:21:16.494827+0000","last_active":"2024-09-17T07:21:30.747073+0000","last_peered":"2024-09-17T07:21:30.747073+0000","last_clean":"2024-09-17T07:21:30.747073+0000","last_became_active":"2024-09-17T07:21:16.347875+0000","last_became_peered":"2024-09-17T07:21:16.347875+0000","last_unstale":"2024-09-17T07:21:30.747073+0000","last_undegraded":"2024-09-17T07:21:30.747073+0000","last_fullsized":"2024-09-17T07:21:30.747073+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"225'9","last_scrub_stamp":"2024-09-17T07:18:23.788769+0000","last_deep_scrub":"225'9","last_deep_scrub_stamp":"2024-09-17T07:11:25.292565+0000","last_clean_scrub_stamp":"2024-09-17T07:18:23.788769+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023108299999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":452,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.415576+0000","last_change":"2024-09-17T07:18:41.386243+0000","last_active":"2024-09-17T07:21:26.415576+0000","last_peered":"2024-09-17T07:21:26.415576+0000","last_clean":"2024-09-17T07:21:26.415576+0000","last_became_active":"2024-09-17T07:02:18.363253+0000","last_became_peered":"2024-09-17T07:02:18.363253+0000","last_unstale":"2024-09-17T07:21:26.415576+0000","last_undegraded":"2024-09-17T07:21:26.415576+0000","last_fullsized":"2024-09-17T07:21:26.415576+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:18:41.385928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:18:41.385928+0000","last_clean_scrub_stamp":"2024-09-17T07:18:41.385928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023027270000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.6","version":"422'10730","reported_seq":11141,"reported_epoch":422,"state":"active+undersized+degraded+remapped+backfill_wait","last_fresh":"2024-09-17T07:21:39.820325+0000","last_change":"2024-09-17T07:21:18.365301+0000","last_active":"2024-09-17T07:21:39.820325+0000","last_peered":"2024-09-17T07:21:39.820325+0000","last_clean":"2024-09-17T07:19:54.447320+0000","last_became_active":"2024-09-17T07:21:16.492393+0000","last_became_peered":"2024-09-17T07:21:16.492393+0000","last_unstale":"2024-09-17T07:21:39.820325+0000","last_undegraded":"2024-09-17T07:21:16.378286+0000","last_fullsized":"2024-09-17T07:21:16.351432+0000","mapping_epoch":414,"log_start":"409'6700","ondisk_log_start":"409'6700","created":390,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_clean_scrub_stamp":"2024-09-17T07:19:54.447320+0000","objects_scrubbed":0,"log_size":4030,"log_dups_size":0,"ondisk_log_size":4030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":703201280,"num_objects":10730,"num_object_clones":0,"num_object_copies":32190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10505,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10730,"num_write_kb":686720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":181,"num_bytes_recovered":11862016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":10730}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"242'23","reported_seq":537,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.408888+0000","last_change":"2024-09-17T07:20:33.108656+0000","last_active":"2024-09-17T07:21:26.408888+0000","last_peered":"2024-09-17T07:21:26.408888+0000","last_clean":"2024-09-17T07:21:26.408888+0000","last_became_active":"2024-09-17T07:10:49.723292+0000","last_became_peered":"2024-09-17T07:10:49.723292+0000","last_unstale":"2024-09-17T07:21:26.408888+0000","last_undegraded":"2024-09-17T07:21:26.408888+0000","last_fullsized":"2024-09-17T07:21:26.408888+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'23","last_scrub_stamp":"2024-09-17T07:20:33.108490+0000","last_deep_scrub":"150'17","last_deep_scrub_stamp":"2024-09-17T07:06:38.775788+0000","last_clean_scrub_stamp":"2024-09-17T07:20:33.108490+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:21:43.815445+0000","scrub_duration":0.0087601510000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"68'2","reported_seq":482,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.411467+0000","last_change":"2024-09-17T07:20:11.093628+0000","last_active":"2024-09-17T07:21:26.411467+0000","last_peered":"2024-09-17T07:21:26.411467+0000","last_clean":"2024-09-17T07:21:26.411467+0000","last_became_active":"2024-09-17T07:19:42.531226+0000","last_became_peered":"2024-09-17T07:19:42.531226+0000","last_unstale":"2024-09-17T07:21:26.411467+0000","last_undegraded":"2024-09-17T07:21:26.411467+0000","last_fullsized":"2024-09-17T07:21:26.411467+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-09-17T07:20:11.093491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:20:11.093491+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:21:40.517870+0000","scrub_duration":0.0023198989999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":413,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-17T07:21:29.507103+0000","last_change":"2024-09-17T07:21:15.356254+0000","last_active":"2024-09-17T07:21:29.507103+0000","last_peered":"2024-09-17T07:21:29.507103+0000","last_clean":"2024-09-17T07:21:29.507103+0000","last_became_active":"2024-09-17T07:21:15.355760+0000","last_became_peered":"2024-09-17T07:21:15.355760+0000","last_unstale":"2024-09-17T07:21:29.507103+0000","last_undegraded":"2024-09-17T07:21:29.507103+0000","last_fullsized":"2024-09-17T07:21:29.507103+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:51.586929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:11:17.479036+0000","last_clean_scrub_stamp":"2024-09-17T07:15:51.586929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001274435,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":22,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.418206+0000","last_change":"2024-09-17T07:21:16.483986+0000","last_active":"2024-09-17T07:21:26.418206+0000","last_peered":"2024-09-17T07:21:26.418206+0000","last_clean":"2024-09-17T07:21:26.418206+0000","last_became_active":"2024-09-17T07:21:16.478287+0000","last_became_peered":"2024-09-17T07:21:16.478287+0000","last_unstale":"2024-09-17T07:21:26.418206+0000","last_undegraded":"2024-09-17T07:21:26.418206+0000","last_fullsized":"2024-09-17T07:21:26.418206+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:07.959158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:20:07.959158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.018487173999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"242'15","reported_seq":509,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-17T07:21:29.500710+0000","last_change":"2024-09-17T07:19:42.522776+0000","last_active":"2024-09-17T07:21:29.500710+0000","last_peered":"2024-09-17T07:21:29.500710+0000","last_clean":"2024-09-17T07:21:29.500710+0000","last_became_active":"2024-09-17T07:19:42.505581+0000","last_became_peered":"2024-09-17T07:19:42.505581+0000","last_unstale":"2024-09-17T07:21:29.500710+0000","last_undegraded":"2024-09-17T07:21:29.500710+0000","last_fullsized":"2024-09-17T07:21:29.500710+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'15","last_scrub_stamp":"2024-09-17T07:18:24.505002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:18:24.505002+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002195809,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":486,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.414914+0000","last_change":"2024-09-17T07:21:15.788428+0000","last_active":"2024-09-17T07:21:26.414914+0000","last_peered":"2024-09-17T07:21:26.414914+0000","last_clean":"2024-09-17T07:21:26.414914+0000","last_became_active":"2024-09-17T07:21:15.355464+0000","last_became_peered":"2024-09-17T07:21:15.355464+0000","last_unstale":"2024-09-17T07:21:26.414914+0000","last_undegraded":"2024-09-17T07:21:26.414914+0000","last_fullsized":"2024-09-17T07:21:26.414914+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:21:15.788348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:17.976267+0000","last_clean_scrub_stamp":"2024-09-17T07:21:15.788348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:22:18.671735+0000","scrub_duration":0.0012829899999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":149,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.423051+0000","last_change":"2024-09-17T07:21:16.451780+0000","last_active":"2024-09-17T07:21:26.423051+0000","last_peered":"2024-09-17T07:21:26.423051+0000","last_clean":"2024-09-17T07:21:26.423051+0000","last_became_active":"2024-09-17T07:21:16.450534+0000","last_became_peered":"2024-09-17T07:21:16.450534+0000","last_unstale":"2024-09-17T07:21:26.423051+0000","last_undegraded":"2024-09-17T07:21:26.423051+0000","last_fullsized":"2024-09-17T07:21:26.423051+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:29.492000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:18.367486+0000","last_clean_scrub_stamp":"2024-09-17T07:20:29.492000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:21:41.864454+0000","scrub_duration":0.0077309099999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":527,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.420805+0000","last_change":"2024-09-17T07:21:11.767561+0000","last_active":"2024-09-17T07:21:26.420805+0000","last_peered":"2024-09-17T07:21:26.420805+0000","last_clean":"2024-09-17T07:21:26.420805+0000","last_became_active":"2024-09-17T07:20:49.436496+0000","last_became_peered":"2024-09-17T07:20:49.436496+0000","last_unstale":"2024-09-17T07:21:26.420805+0000","last_undegraded":"2024-09-17T07:21:26.420805+0000","last_fullsized":"2024-09-17T07:21:26.420805+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:21:11.767427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:09:23.378805+0000","last_clean_scrub_stamp":"2024-09-17T07:21:11.767427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:22:21.958295+0000","scrub_duration":0.012069135,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":463,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.416991+0000","last_change":"2024-09-17T07:21:15.350067+0000","last_active":"2024-09-17T07:21:26.416991+0000","last_peered":"2024-09-17T07:21:26.416991+0000","last_clean":"2024-09-17T07:21:26.416991+0000","last_became_active":"2024-09-17T07:21:15.348758+0000","last_became_peered":"2024-09-17T07:21:15.348758+0000","last_unstale":"2024-09-17T07:21:26.416991+0000","last_undegraded":"2024-09-17T07:21:26.416991+0000","last_fullsized":"2024-09-17T07:21:26.416991+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:46.645080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:46.645080+0000","last_clean_scrub_stamp":"2024-09-17T07:15:46.645080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015661049999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"242'12","reported_seq":126,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.423274+0000","last_change":"2024-09-17T07:21:16.353210+0000","last_active":"2024-09-17T07:21:26.423274+0000","last_peered":"2024-09-17T07:21:26.423274+0000","last_clean":"2024-09-17T07:21:26.423274+0000","last_became_active":"2024-09-17T07:21:16.352136+0000","last_became_peered":"2024-09-17T07:21:16.352136+0000","last_unstale":"2024-09-17T07:21:26.423274+0000","last_undegraded":"2024-09-17T07:21:26.423274+0000","last_fullsized":"2024-09-17T07:21:26.423274+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'12","last_scrub_stamp":"2024-09-17T07:20:17.090146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:20:17.090146+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:21:30.395977+0000","scrub_duration":0.0019381719999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":16,"num_read_kb":13,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":456,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-17T07:21:29.498432+0000","last_change":"2024-09-17T07:21:15.342247+0000","last_active":"2024-09-17T07:21:29.498432+0000","last_peered":"2024-09-17T07:21:29.498432+0000","last_clean":"2024-09-17T07:21:29.498432+0000","last_became_active":"2024-09-17T07:21:15.341966+0000","last_became_peered":"2024-09-17T07:21:15.341966+0000","last_unstale":"2024-09-17T07:21:29.498432+0000","last_undegraded":"2024-09-17T07:21:29.498432+0000","last_fullsized":"2024-09-17T07:21:29.498432+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:55.191252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:14:55.191252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0090150360000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":50,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.419603+0000","last_change":"2024-09-17T07:19:42.534936+0000","last_active":"2024-09-17T07:21:26.419603+0000","last_peered":"2024-09-17T07:21:26.419603+0000","last_clean":"2024-09-17T07:21:26.419603+0000","last_became_active":"2024-09-17T07:19:42.534052+0000","last_became_peered":"2024-09-17T07:19:42.534052+0000","last_unstale":"2024-09-17T07:21:26.419603+0000","last_undegraded":"2024-09-17T07:21:26.419603+0000","last_fullsized":"2024-09-17T07:21:26.419603+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:25.934136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:19:25.934136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"242'20","reported_seq":181,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.420338+0000","last_change":"2024-09-17T07:21:16.356290+0000","last_active":"2024-09-17T07:21:26.420338+0000","last_peered":"2024-09-17T07:21:26.420338+0000","last_clean":"2024-09-17T07:21:26.420338+0000","last_became_active":"2024-09-17T07:21:16.355328+0000","last_became_peered":"2024-09-17T07:21:16.355328+0000","last_unstale":"2024-09-17T07:21:26.420338+0000","last_undegraded":"2024-09-17T07:21:26.420338+0000","last_fullsized":"2024-09-17T07:21:26.420338+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'20","last_scrub_stamp":"2024-09-17T07:20:10.554324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:20:10.554324+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:21:37.435013+0000","scrub_duration":0.0020069329999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":487,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.415106+0000","last_change":"2024-09-17T07:19:42.558275+0000","last_active":"2024-09-17T07:21:26.415106+0000","last_peered":"2024-09-17T07:21:26.415106+0000","last_clean":"2024-09-17T07:21:26.415106+0000","last_became_active":"2024-09-17T07:19:42.557857+0000","last_became_peered":"2024-09-17T07:19:42.557857+0000","last_unstale":"2024-09-17T07:21:26.415106+0000","last_undegraded":"2024-09-17T07:21:26.415106+0000","last_fullsized":"2024-09-17T07:21:26.415106+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:40.443474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:19:40.443474+0000","last_clean_scrub_stamp":"2024-09-17T07:19:40.443474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002799733,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"414'11","reported_seq":691,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-17T07:21:35.743867+0000","last_change":"2024-09-17T07:21:15.353878+0000","last_active":"2024-09-17T07:21:35.743867+0000","last_peered":"2024-09-17T07:21:35.743867+0000","last_clean":"2024-09-17T07:21:35.743867+0000","last_became_active":"2024-09-17T07:21:15.353461+0000","last_became_peered":"2024-09-17T07:21:15.353461+0000","last_unstale":"2024-09-17T07:21:35.743867+0000","last_undegraded":"2024-09-17T07:21:35.743867+0000","last_fullsized":"2024-09-17T07:21:35.743867+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'9","last_scrub_stamp":"2024-09-17T07:14:53.140133+0000","last_deep_scrub":"163'9","last_deep_scrub_stamp":"2024-09-17T07:07:34.933808+0000","last_clean_scrub_stamp":"2024-09-17T07:14:53.140133+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0088819989999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":460,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-17T07:21:29.500387+0000","last_change":"2024-09-17T07:19:42.528538+0000","last_active":"2024-09-17T07:21:29.500387+0000","last_peered":"2024-09-17T07:21:29.500387+0000","last_clean":"2024-09-17T07:21:29.500387+0000","last_became_active":"2024-09-17T07:19:42.528225+0000","last_became_peered":"2024-09-17T07:19:42.528225+0000","last_unstale":"2024-09-17T07:21:29.500387+0000","last_undegraded":"2024-09-17T07:21:29.500387+0000","last_fullsized":"2024-09-17T07:21:29.500387+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:18:47.392182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:26.619893+0000","last_clean_scrub_stamp":"2024-09-17T07:18:47.392182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0053055120000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"242'15","reported_seq":519,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-17T07:21:29.497871+0000","last_change":"2024-09-17T07:19:42.547221+0000","last_active":"2024-09-17T07:21:29.497871+0000","last_peered":"2024-09-17T07:21:29.497871+0000","last_clean":"2024-09-17T07:21:29.497871+0000","last_became_active":"2024-09-17T07:19:42.529662+0000","last_became_peered":"2024-09-17T07:19:42.529662+0000","last_unstale":"2024-09-17T07:21:29.497871+0000","last_undegraded":"2024-09-17T07:21:29.497871+0000","last_fullsized":"2024-09-17T07:21:29.497871+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'15","last_scrub_stamp":"2024-09-17T07:19:20.304874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:19:20.304874+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0074478349999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":470,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.413146+0000","last_change":"2024-09-17T07:19:42.563318+0000","last_active":"2024-09-17T07:21:26.413146+0000","last_peered":"2024-09-17T07:21:26.413146+0000","last_clean":"2024-09-17T07:21:26.413146+0000","last_became_active":"2024-09-17T07:19:42.562975+0000","last_became_peered":"2024-09-17T07:19:42.562975+0000","last_unstale":"2024-09-17T07:21:26.413146+0000","last_undegraded":"2024-09-17T07:21:26.413146+0000","last_fullsized":"2024-09-17T07:21:26.413146+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:33.544567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:19:33.544567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.13531478499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"415'14","reported_seq":48,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-17T07:21:30.743891+0000","last_change":"2024-09-17T07:21:16.356096+0000","last_active":"2024-09-17T07:21:30.743891+0000","last_peered":"2024-09-17T07:21:30.743891+0000","last_clean":"2024-09-17T07:21:30.743891+0000","last_became_active":"2024-09-17T07:21:16.354722+0000","last_became_peered":"2024-09-17T07:21:16.354722+0000","last_unstale":"2024-09-17T07:21:30.743891+0000","last_undegraded":"2024-09-17T07:21:30.743891+0000","last_fullsized":"2024-09-17T07:21:30.743891+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'12","last_scrub_stamp":"2024-09-17T07:20:09.105448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:20:09.105448+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":453,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.411287+0000","last_change":"2024-09-17T07:20:37.110604+0000","last_active":"2024-09-17T07:21:26.411287+0000","last_peered":"2024-09-17T07:21:26.411287+0000","last_clean":"2024-09-17T07:21:26.411287+0000","last_became_active":"2024-09-17T07:02:18.018716+0000","last_became_peered":"2024-09-17T07:02:18.018716+0000","last_unstale":"2024-09-17T07:21:26.411287+0000","last_undegraded":"2024-09-17T07:21:26.411287+0000","last_fullsized":"2024-09-17T07:21:26.411287+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:37.110458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:20:37.110458+0000","last_clean_scrub_stamp":"2024-09-17T07:20:37.110458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:21:39.730079+0000","scrub_duration":0.017901013,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"242'15","reported_seq":309,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.420120+0000","last_change":"2024-09-17T07:21:15.349519+0000","last_active":"2024-09-17T07:21:26.420120+0000","last_peered":"2024-09-17T07:21:26.420120+0000","last_clean":"2024-09-17T07:21:26.420120+0000","last_became_active":"2024-09-17T07:21:15.348060+0000","last_became_peered":"2024-09-17T07:21:15.348060+0000","last_unstale":"2024-09-17T07:21:26.420120+0000","last_undegraded":"2024-09-17T07:21:26.420120+0000","last_fullsized":"2024-09-17T07:21:26.420120+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'15","last_scrub_stamp":"2024-09-17T07:13:48.500257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:13:48.500257+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.005053564,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":487,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.416144+0000","last_change":"2024-09-17T07:19:42.533425+0000","last_active":"2024-09-17T07:21:26.416144+0000","last_peered":"2024-09-17T07:21:26.416144+0000","last_clean":"2024-09-17T07:21:26.416144+0000","last_became_active":"2024-09-17T07:19:42.533133+0000","last_became_peered":"2024-09-17T07:19:42.533133+0000","last_unstale":"2024-09-17T07:21:26.416144+0000","last_undegraded":"2024-09-17T07:21:26.416144+0000","last_fullsized":"2024-09-17T07:21:26.416144+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:18:53.905079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:10:36.334827+0000","last_clean_scrub_stamp":"2024-09-17T07:18:53.905079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0087069899999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":292,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.412726+0000","last_change":"2024-09-17T07:21:15.351645+0000","last_active":"2024-09-17T07:21:26.412726+0000","last_peered":"2024-09-17T07:21:26.412726+0000","last_clean":"2024-09-17T07:21:26.412726+0000","last_became_active":"2024-09-17T07:21:15.351360+0000","last_became_peered":"2024-09-17T07:21:15.351360+0000","last_unstale":"2024-09-17T07:21:26.412726+0000","last_undegraded":"2024-09-17T07:21:26.412726+0000","last_fullsized":"2024-09-17T07:21:26.412726+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:37.523283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:15:37.523283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013475850000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":446,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.415190+0000","last_change":"2024-09-17T07:21:15.355169+0000","last_active":"2024-09-17T07:21:26.415190+0000","last_peered":"2024-09-17T07:21:26.415190+0000","last_clean":"2024-09-17T07:21:26.415190+0000","last_became_active":"2024-09-17T07:21:15.354884+0000","last_became_peered":"2024-09-17T07:21:15.354884+0000","last_unstale":"2024-09-17T07:21:26.415190+0000","last_undegraded":"2024-09-17T07:21:26.415190+0000","last_fullsized":"2024-09-17T07:21:26.415190+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:40.540336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:15:40.540336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001212982,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"242'19","reported_seq":482,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.408929+0000","last_change":"2024-09-17T07:21:16.070482+0000","last_active":"2024-09-17T07:21:26.408929+0000","last_peered":"2024-09-17T07:21:26.408929+0000","last_clean":"2024-09-17T07:21:26.408929+0000","last_became_active":"2024-09-17T07:21:15.347179+0000","last_became_peered":"2024-09-17T07:21:15.347179+0000","last_unstale":"2024-09-17T07:21:26.408929+0000","last_undegraded":"2024-09-17T07:21:26.408929+0000","last_fullsized":"2024-09-17T07:21:26.408929+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'19","last_scrub_stamp":"2024-09-17T07:21:16.070358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:21:16.070358+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:22:22.732537+0000","scrub_duration":0.0025953619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"94'2","reported_seq":319,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.419573+0000","last_change":"2024-09-17T07:21:15.355110+0000","last_active":"2024-09-17T07:21:26.419573+0000","last_peered":"2024-09-17T07:21:26.419573+0000","last_clean":"2024-09-17T07:21:26.419573+0000","last_became_active":"2024-09-17T07:21:15.353242+0000","last_became_peered":"2024-09-17T07:21:15.353242+0000","last_unstale":"2024-09-17T07:21:26.419573+0000","last_undegraded":"2024-09-17T07:21:26.419573+0000","last_fullsized":"2024-09-17T07:21:26.419573+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-09-17T07:15:48.907821+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-09-17T07:07:31.717888+0000","last_clean_scrub_stamp":"2024-09-17T07:15:48.907821+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":314,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.419666+0000","last_change":"2024-09-17T07:21:16.465890+0000","last_active":"2024-09-17T07:21:26.419666+0000","last_peered":"2024-09-17T07:21:26.419666+0000","last_clean":"2024-09-17T07:21:26.419666+0000","last_became_active":"2024-09-17T07:21:16.463730+0000","last_became_peered":"2024-09-17T07:21:16.463730+0000","last_unstale":"2024-09-17T07:21:26.419666+0000","last_undegraded":"2024-09-17T07:21:26.419666+0000","last_fullsized":"2024-09-17T07:21:26.419666+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:16.640384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:20:16.640384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:21:38.092152+0000","scrub_duration":0.003133224,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":287,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.414304+0000","last_change":"2024-09-17T07:18:23.515698+0000","last_active":"2024-09-17T07:21:26.414304+0000","last_peered":"2024-09-17T07:21:26.414304+0000","last_clean":"2024-09-17T07:21:26.414304+0000","last_became_active":"2024-09-17T07:09:24.400791+0000","last_became_peered":"2024-09-17T07:09:24.400791+0000","last_unstale":"2024-09-17T07:21:26.414304+0000","last_undegraded":"2024-09-17T07:21:26.414304+0000","last_fullsized":"2024-09-17T07:21:26.414304+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:18:23.515582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:49.573666+0000","last_clean_scrub_stamp":"2024-09-17T07:18:23.515582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0022931869999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"242'8","reported_seq":373,"reported_epoch":422,"state":"active+clean+scrubbing","last_fresh":"2024-09-17T07:21:29.505306+0000","last_change":"2024-09-17T07:21:36.948485+0000","last_active":"2024-09-17T07:21:29.505306+0000","last_peered":"2024-09-17T07:21:29.505306+0000","last_clean":"2024-09-17T07:21:29.505306+0000","last_became_active":"2024-09-17T07:09:24.396368+0000","last_became_peered":"2024-09-17T07:09:24.396368+0000","last_unstale":"2024-09-17T07:21:29.505306+0000","last_undegraded":"2024-09-17T07:21:29.505306+0000","last_fullsized":"2024-09-17T07:21:29.505306+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'8","last_scrub_stamp":"2024-09-17T07:20:13.839419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:20:13.839419+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045959240000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":458,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-17T07:21:29.498523+0000","last_change":"2024-09-17T07:19:22.340606+0000","last_active":"2024-09-17T07:21:29.498523+0000","last_peered":"2024-09-17T07:21:29.498523+0000","last_clean":"2024-09-17T07:21:29.498523+0000","last_became_active":"2024-09-17T07:02:12.008898+0000","last_became_peered":"2024-09-17T07:02:12.008898+0000","last_unstale":"2024-09-17T07:21:29.498523+0000","last_undegraded":"2024-09-17T07:21:29.498523+0000","last_fullsized":"2024-09-17T07:21:29.498523+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:22.340320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:19:22.340320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0083030540000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":475,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.416606+0000","last_change":"2024-09-17T07:19:42.514768+0000","last_active":"2024-09-17T07:21:26.416606+0000","last_peered":"2024-09-17T07:21:26.416606+0000","last_clean":"2024-09-17T07:21:26.416606+0000","last_became_active":"2024-09-17T07:19:42.514416+0000","last_became_peered":"2024-09-17T07:19:42.514416+0000","last_unstale":"2024-09-17T07:21:26.416606+0000","last_undegraded":"2024-09-17T07:21:26.416606+0000","last_fullsized":"2024-09-17T07:21:26.416606+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:36.808726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:54.553970+0000","last_clean_scrub_stamp":"2024-09-17T07:19:36.808726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001250061,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":72,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.408127+0000","last_change":"2024-09-17T07:21:16.348109+0000","last_active":"2024-09-17T07:21:26.408127+0000","last_peered":"2024-09-17T07:21:26.408127+0000","last_clean":"2024-09-17T07:21:26.408127+0000","last_became_active":"2024-09-17T07:21:16.347830+0000","last_became_peered":"2024-09-17T07:21:16.347830+0000","last_unstale":"2024-09-17T07:21:26.408127+0000","last_undegraded":"2024-09-17T07:21:26.408127+0000","last_fullsized":"2024-09-17T07:21:26.408127+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:25.003509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:20:25.003509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:21:41.636794+0000","scrub_duration":0.002164586,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.11","version":"242'19","reported_seq":196,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.416835+0000","last_change":"2024-09-17T07:20:28.114271+0000","last_active":"2024-09-17T07:21:26.416835+0000","last_peered":"2024-09-17T07:21:26.416835+0000","last_clean":"2024-09-17T07:21:26.416835+0000","last_became_active":"2024-09-17T07:19:42.533130+0000","last_became_peered":"2024-09-17T07:19:42.533130+0000","last_unstale":"2024-09-17T07:21:26.416835+0000","last_undegraded":"2024-09-17T07:21:26.416835+0000","last_fullsized":"2024-09-17T07:21:26.416835+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'19","last_scrub_stamp":"2024-09-17T07:20:28.113977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:20:28.113977+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:21:34.196288+0000","scrub_duration":0.033449896,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":23,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":469,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-17T07:21:29.506609+0000","last_change":"2024-09-17T07:18:41.951465+0000","last_active":"2024-09-17T07:21:29.506609+0000","last_peered":"2024-09-17T07:21:29.506609+0000","last_clean":"2024-09-17T07:21:29.506609+0000","last_became_active":"2024-09-17T07:02:12.010195+0000","last_became_peered":"2024-09-17T07:02:12.010195+0000","last_unstale":"2024-09-17T07:21:29.506609+0000","last_undegraded":"2024-09-17T07:21:29.506609+0000","last_fullsized":"2024-09-17T07:21:29.506609+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:18:41.951382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:18:41.951382+0000","last_clean_scrub_stamp":"2024-09-17T07:18:41.951382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001516739,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":62,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.407795+0000","last_change":"2024-09-17T07:20:30.038151+0000","last_active":"2024-09-17T07:21:26.407795+0000","last_peered":"2024-09-17T07:21:26.407795+0000","last_clean":"2024-09-17T07:21:26.407795+0000","last_became_active":"2024-09-17T07:19:42.520166+0000","last_became_peered":"2024-09-17T07:19:42.520166+0000","last_unstale":"2024-09-17T07:21:26.407795+0000","last_undegraded":"2024-09-17T07:21:26.407795+0000","last_fullsized":"2024-09-17T07:21:26.407795+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:30.038044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:20:30.038044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:21:32.289703+0000","scrub_duration":0.0045605209999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":298,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.412651+0000","last_change":"2024-09-17T07:19:25.448467+0000","last_active":"2024-09-17T07:21:26.412651+0000","last_peered":"2024-09-17T07:21:26.412651+0000","last_clean":"2024-09-17T07:21:26.412651+0000","last_became_active":"2024-09-17T07:09:24.392216+0000","last_became_peered":"2024-09-17T07:09:24.392216+0000","last_unstale":"2024-09-17T07:21:26.412651+0000","last_undegraded":"2024-09-17T07:21:26.412651+0000","last_fullsized":"2024-09-17T07:21:26.412651+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:25.448373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:19:25.448373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033889219999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1c","version":"326'2","reported_seq":392,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.415528+0000","last_change":"2024-09-17T07:20:19.609815+0000","last_active":"2024-09-17T07:21:26.415528+0000","last_peered":"2024-09-17T07:21:26.415528+0000","last_clean":"2024-09-17T07:21:26.415528+0000","last_became_active":"2024-09-17T07:19:42.524335+0000","last_became_peered":"2024-09-17T07:19:42.524335+0000","last_unstale":"2024-09-17T07:21:26.415528+0000","last_undegraded":"2024-09-17T07:21:26.415528+0000","last_fullsized":"2024-09-17T07:21:26.415528+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'2","last_scrub_stamp":"2024-09-17T07:20:19.609678+0000","last_deep_scrub":"326'2","last_deep_scrub_stamp":"2024-09-17T07:17:36.974358+0000","last_clean_scrub_stamp":"2024-09-17T07:20:19.609678+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:21:40.201112+0000","scrub_duration":0.0037749839999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":468,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-17T07:21:29.496438+0000","last_change":"2024-09-17T07:19:42.527772+0000","last_active":"2024-09-17T07:21:29.496438+0000","last_peered":"2024-09-17T07:21:29.496438+0000","last_clean":"2024-09-17T07:21:29.496438+0000","last_became_active":"2024-09-17T07:19:42.527465+0000","last_became_peered":"2024-09-17T07:19:42.527465+0000","last_unstale":"2024-09-17T07:21:29.496438+0000","last_undegraded":"2024-09-17T07:21:29.496438+0000","last_fullsized":"2024-09-17T07:21:29.496438+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:23.314136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:19:23.314136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010788304,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"294'25","reported_seq":567,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.413388+0000","last_change":"2024-09-17T07:21:16.515870+0000","last_active":"2024-09-17T07:21:26.413388+0000","last_peered":"2024-09-17T07:21:26.413388+0000","last_clean":"2024-09-17T07:21:26.413388+0000","last_became_active":"2024-09-17T07:21:16.380262+0000","last_became_peered":"2024-09-17T07:21:16.380262+0000","last_unstale":"2024-09-17T07:21:26.413388+0000","last_undegraded":"2024-09-17T07:21:26.413388+0000","last_fullsized":"2024-09-17T07:21:26.413388+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'25","last_scrub_stamp":"2024-09-17T07:20:39.458854+0000","last_deep_scrub":"294'25","last_deep_scrub_stamp":"2024-09-17T07:16:00.471311+0000","last_clean_scrub_stamp":"2024-09-17T07:20:39.458854+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:22:00.148317+0000","scrub_duration":0.005100612,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"387'9","reported_seq":724,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-17T07:21:30.742806+0000","last_change":"2024-09-17T07:19:42.527005+0000","last_active":"2024-09-17T07:21:30.742806+0000","last_peered":"2024-09-17T07:21:30.742806+0000","last_clean":"2024-09-17T07:21:30.742806+0000","last_became_active":"2024-09-17T07:19:42.515434+0000","last_became_peered":"2024-09-17T07:19:42.515434+0000","last_unstale":"2024-09-17T07:21:30.742806+0000","last_undegraded":"2024-09-17T07:21:30.742806+0000","last_fullsized":"2024-09-17T07:21:30.742806+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:19:30.879061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:19:30.879061+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034025510000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":305,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.417561+0000","last_change":"2024-09-17T07:21:15.354938+0000","last_active":"2024-09-17T07:21:26.417561+0000","last_peered":"2024-09-17T07:21:26.417561+0000","last_clean":"2024-09-17T07:21:26.417561+0000","last_became_active":"2024-09-17T07:21:15.353194+0000","last_became_peered":"2024-09-17T07:21:15.353194+0000","last_unstale":"2024-09-17T07:21:26.417561+0000","last_undegraded":"2024-09-17T07:21:26.417561+0000","last_fullsized":"2024-09-17T07:21:26.417561+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:37.885570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:15:37.885570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":309,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-17T07:21:29.506638+0000","last_change":"2024-09-17T07:19:22.869811+0000","last_active":"2024-09-17T07:21:29.506638+0000","last_peered":"2024-09-17T07:21:29.506638+0000","last_clean":"2024-09-17T07:21:29.506638+0000","last_became_active":"2024-09-17T07:09:24.403360+0000","last_became_peered":"2024-09-17T07:09:24.403360+0000","last_unstale":"2024-09-17T07:21:29.506638+0000","last_undegraded":"2024-09-17T07:21:29.506638+0000","last_fullsized":"2024-09-17T07:21:29.506638+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:22.869700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:10:29.325664+0000","last_clean_scrub_stamp":"2024-09-17T07:19:22.869700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0039824769999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"242'19","reported_seq":541,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.416969+0000","last_change":"2024-09-17T07:19:26.874442+0000","last_active":"2024-09-17T07:21:26.416969+0000","last_peered":"2024-09-17T07:21:26.416969+0000","last_clean":"2024-09-17T07:21:26.416969+0000","last_became_active":"2024-09-17T07:03:48.760390+0000","last_became_peered":"2024-09-17T07:03:48.760390+0000","last_unstale":"2024-09-17T07:21:26.416969+0000","last_undegraded":"2024-09-17T07:21:26.416969+0000","last_fullsized":"2024-09-17T07:21:26.416969+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'19","last_scrub_stamp":"2024-09-17T07:19:26.872409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:19:26.872409+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012641085999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":475,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.413136+0000","last_change":"2024-09-17T07:21:16.379644+0000","last_active":"2024-09-17T07:21:26.413136+0000","last_peered":"2024-09-17T07:21:26.413136+0000","last_clean":"2024-09-17T07:21:26.413136+0000","last_became_active":"2024-09-17T07:21:16.379358+0000","last_became_peered":"2024-09-17T07:21:16.379358+0000","last_unstale":"2024-09-17T07:21:26.413136+0000","last_undegraded":"2024-09-17T07:21:26.413136+0000","last_fullsized":"2024-09-17T07:21:26.413136+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:37.435106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:20:37.435106+0000","last_clean_scrub_stamp":"2024-09-17T07:20:37.435106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:21:59.822155+0000","scrub_duration":0.0042822729999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":455,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.409517+0000","last_change":"2024-09-17T07:20:26.001554+0000","last_active":"2024-09-17T07:21:26.409517+0000","last_peered":"2024-09-17T07:21:26.409517+0000","last_clean":"2024-09-17T07:21:26.409517+0000","last_became_active":"2024-09-17T07:02:18.363752+0000","last_became_peered":"2024-09-17T07:02:18.363752+0000","last_unstale":"2024-09-17T07:21:26.409517+0000","last_undegraded":"2024-09-17T07:21:26.409517+0000","last_fullsized":"2024-09-17T07:21:26.409517+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:26.001442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:20:26.001442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:21:29.808292+0000","scrub_duration":0.001725695,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":499,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.414620+0000","last_change":"2024-09-17T07:20:18.074918+0000","last_active":"2024-09-17T07:21:26.414620+0000","last_peered":"2024-09-17T07:21:26.414620+0000","last_clean":"2024-09-17T07:21:26.414620+0000","last_became_active":"2024-09-17T07:03:48.759384+0000","last_became_peered":"2024-09-17T07:03:48.759384+0000","last_unstale":"2024-09-17T07:21:26.414620+0000","last_undegraded":"2024-09-17T07:21:26.414620+0000","last_fullsized":"2024-09-17T07:21:26.414620+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:18.074791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:11:18.559877+0000","last_clean_scrub_stamp":"2024-09-17T07:20:18.074791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:21:33.696613+0000","scrub_duration":0.0039012449999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"242'27","reported_seq":364,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.420413+0000","last_change":"2024-09-17T07:21:16.372009+0000","last_active":"2024-09-17T07:21:26.420413+0000","last_peered":"2024-09-17T07:21:26.420413+0000","last_clean":"2024-09-17T07:21:26.420413+0000","last_became_active":"2024-09-17T07:21:16.371404+0000","last_became_peered":"2024-09-17T07:21:16.371404+0000","last_unstale":"2024-09-17T07:21:26.420413+0000","last_undegraded":"2024-09-17T07:21:26.420413+0000","last_fullsized":"2024-09-17T07:21:26.420413+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'27","last_scrub_stamp":"2024-09-17T07:21:11.488225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:21:11.488225+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:22:17.817746+0000","scrub_duration":0.0089238530000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":284,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.420165+0000","last_change":"2024-09-17T07:21:15.351806+0000","last_active":"2024-09-17T07:21:26.420165+0000","last_peered":"2024-09-17T07:21:26.420165+0000","last_clean":"2024-09-17T07:21:26.420165+0000","last_became_active":"2024-09-17T07:21:15.348754+0000","last_became_peered":"2024-09-17T07:21:15.348754+0000","last_unstale":"2024-09-17T07:21:26.420165+0000","last_undegraded":"2024-09-17T07:21:26.420165+0000","last_fullsized":"2024-09-17T07:21:26.420165+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:00.509283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:25.794418+0000","last_clean_scrub_stamp":"2024-09-17T07:14:00.509283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":314,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.416885+0000","last_change":"2024-09-17T07:21:16.225667+0000","last_active":"2024-09-17T07:21:26.416885+0000","last_peered":"2024-09-17T07:21:26.416885+0000","last_clean":"2024-09-17T07:21:26.416885+0000","last_became_active":"2024-09-17T07:21:15.348166+0000","last_became_peered":"2024-09-17T07:21:15.348166+0000","last_unstale":"2024-09-17T07:21:26.416885+0000","last_undegraded":"2024-09-17T07:21:26.416885+0000","last_fullsized":"2024-09-17T07:21:26.416885+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:21:16.225555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:21:16.225555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:22:21.332297+0000","scrub_duration":0.001743026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":53,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.415807+0000","last_change":"2024-09-17T07:19:42.534529+0000","last_active":"2024-09-17T07:21:26.415807+0000","last_peered":"2024-09-17T07:21:26.415807+0000","last_clean":"2024-09-17T07:21:26.415807+0000","last_became_active":"2024-09-17T07:19:42.533880+0000","last_became_peered":"2024-09-17T07:19:42.533880+0000","last_unstale":"2024-09-17T07:21:26.415807+0000","last_undegraded":"2024-09-17T07:21:26.415807+0000","last_fullsized":"2024-09-17T07:21:26.415807+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:30.003956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:19:30.003956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"242'23","reported_seq":216,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.418142+0000","last_change":"2024-09-17T07:21:15.352075+0000","last_active":"2024-09-17T07:21:26.418142+0000","last_peered":"2024-09-17T07:21:26.418142+0000","last_clean":"2024-09-17T07:21:26.418142+0000","last_became_active":"2024-09-17T07:21:15.351512+0000","last_became_peered":"2024-09-17T07:21:15.351512+0000","last_unstale":"2024-09-17T07:21:26.418142+0000","last_undegraded":"2024-09-17T07:21:26.418142+0000","last_fullsized":"2024-09-17T07:21:26.418142+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'23","last_scrub_stamp":"2024-09-17T07:14:58.689120+0000","last_deep_scrub":"86'17","last_deep_scrub_stamp":"2024-09-17T07:03:18.241067+0000","last_clean_scrub_stamp":"2024-09-17T07:14:58.689120+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":484,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-17T07:21:26.408330+0000","last_change":"2024-09-17T07:21:16.349922+0000","last_active":"2024-09-17T07:21:26.408330+0000","last_peered":"2024-09-17T07:21:26.408330+0000","last_clean":"2024-09-17T07:21:26.408330+0000","last_became_active":"2024-09-17T07:21:16.349577+0000","last_became_peered":"2024-09-17T07:21:16.349577+0000","last_unstale":"2024-09-17T07:21:26.408330+0000","last_undegraded":"2024-09-17T07:21:26.408330+0000","last_fullsized":"2024-09-17T07:21:26.408330+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:21.073159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:20:21.073159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0093893710000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":11194400768,"num_objects":170813,"num_object_clones":0,"num_object_copies":512439,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":117391,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":170813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":170813,"num_write_kb":10932032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10043,"num_bytes_recovered":658178048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":25844580352,"data_stored":25844580352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":61822,"ondisk_log_size":61822,"up":48,"acting":36,"num_store_stats":6},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":382,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85,"ondisk_log_size":85,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1110,"num_read_kb":867,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":180,"num_bytes_recovered":3702,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":520192,"data_stored":13774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":734,"ondisk_log_size":734,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1386,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":12,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":604,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":4998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":805047,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":78,"num_read_kb":111,"num_write":66,"num_write_kb":879,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":644038,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1430,"ondisk_log_size":1430,"up":33,"acting":33,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":0,"seq":257698037999,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":6,"up_from":50,"seq":214748365042,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7007336,"kb_used_data":6091872,"kb_used_omap":1,"kb_used_meta":915454,"kb_avail":86737816,"statfs":{"total":95995035648,"available":88819523584,"internally_reserved":0,"allocated":6238076928,"data_stored":6237043930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":937425351},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:21:25 2024","interfaces":[{"interface":"back","average":{"1min":3.444,"5min":1.972,"15min":2.234},"min":{"1min":0.30099999999999999,"5min":0.20200000000000001,"15min":0.185},"max":{"1min":33.293999999999997,"5min":35.722000000000001,"15min":126.902},"last":1.3049999999999999},{"interface":"front","average":{"1min":2.5720000000000001,"5min":1.4430000000000001,"15min":2.0299999999999998},"min":{"1min":0.34399999999999997,"5min":0.22,"15min":0.20799999999999999},"max":{"1min":30.728999999999999,"5min":35.348999999999997,"15min":127.06399999999999},"last":1.212}]},{"osd":1,"last update":"Tue Sep 17 07:16:10 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.996,"15min":0.84399999999999997},"min":{"1min":0.23799999999999999,"5min":0.22900000000000001,"15min":0.22600000000000001},"max":{"1min":2.2999999999999998,"5min":14.720000000000001,"15min":14.720000000000001},"last":1.1659999999999999},{"interface":"front","average":{"1min":0.622,"5min":1.0109999999999999,"15min":0.88800000000000001},"min":{"1min":0.29099999999999998,"5min":0.23400000000000001,"15min":0.22900000000000001},"max":{"1min":2.323,"5min":14.699,"15min":14.699},"last":0.374}]},{"osd":2,"last update":"Tue Sep 17 07:21:25 2024","interfaces":[{"interface":"back","average":{"1min":3.081,"5min":1.946,"15min":1.2010000000000001},"min":{"1min":0.27900000000000003,"5min":0.24399999999999999,"15min":0.219},"max":{"1min":30.664999999999999,"5min":36.289000000000001,"15min":36.289000000000001},"last":0.622},{"interface":"front","average":{"1min":2.6379999999999999,"5min":1.512,"15min":1.0569999999999999},"min":{"1min":0.32300000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":30.622,"5min":36.326999999999998,"15min":36.326999999999998},"last":0.49099999999999999}]},{"osd":3,"last update":"Tue Sep 17 07:21:25 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":1.0840000000000001,"15min":1.177},"min":{"1min":0.29999999999999999,"5min":0.249,"15min":0.23200000000000001},"max":{"1min":1.877,"5min":6.8639999999999999,"15min":52.944000000000003},"last":0.45300000000000001},{"interface":"front","average":{"1min":1.1850000000000001,"5min":1.3160000000000001,"15min":1.319},"min":{"1min":0.34100000000000003,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":3.101,"5min":5.2949999999999999,"15min":52.747},"last":0.67800000000000005}]},{"osd":4,"last update":"Tue Sep 17 07:21:25 2024","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":1.2,"15min":2.2280000000000002},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":3.448,"5min":5.4800000000000004,"15min":260.77199999999999},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.81399999999999995,"5min":1.575,"15min":2.3300000000000001},"min":{"1min":0.33100000000000002,"5min":0.23300000000000001,"15min":0.19800000000000001},"max":{"1min":2.581,"5min":8.7929999999999993,"15min":260.69099999999997},"last":1.27}]},{"osd":5,"last update":"Tue Sep 17 07:21:25 2024","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":2.3079999999999998,"15min":1.619},"min":{"1min":0.27700000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":3.7450000000000001,"5min":87.290000000000006,"15min":87.290000000000006},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.94599999999999995,"5min":2.4209999999999998,"15min":1.6559999999999999},"min":{"1min":0.377,"5min":0.30499999999999999,"15min":0.251},"max":{"1min":2.6989999999999998,"5min":87.385000000000005,"15min":87.385000000000005},"last":0.65700000000000003}]},{"osd":7,"last update":"Tue Sep 17 07:20:42 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":2.71,"15min":1.6579999999999999},"min":{"1min":0.33000000000000002,"5min":0.246,"15min":0.23699999999999999},"max":{"1min":1.4670000000000001,"5min":131.95699999999999,"15min":131.95699999999999},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.56100000000000005,"5min":2.3929999999999998,"15min":1.5329999999999999},"min":{"1min":0.26900000000000002,"5min":0.252,"15min":0.252},"max":{"1min":1.375,"5min":131.833,"15min":131.833},"last":0.84099999999999997}]}]},{"osd":5,"up_from":42,"seq":180388626679,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4306660,"kb_used_data":3373148,"kb_used_omap":1,"kb_used_meta":933502,"kb_avail":89438492,"statfs":{"total":95995035648,"available":91585015808,"internally_reserved":0,"allocated":3454103552,"data_stored":3453074787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":955906502},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:21:02 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.85099999999999998,"15min":1.669},"min":{"1min":0.221,"5min":0.184,"15min":0.184},"max":{"1min":1.77,"5min":5.7530000000000001,"15min":49.265000000000001},"last":2.0739999999999998},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.72699999999999998,"15min":1.6799999999999999},"min":{"1min":0.30499999999999999,"5min":0.217,"15min":0.217},"max":{"1min":1.25,"5min":4.7939999999999996,"15min":47.350000000000001},"last":1.095}]},{"osd":1,"last update":"Tue Sep 17 07:15:47 2024","interfaces":[{"interface":"back","average":{"1min":1.4319999999999999,"5min":1.0509999999999999,"15min":0.90100000000000002},"min":{"1min":0.30499999999999999,"5min":0.22,"15min":0.22},"max":{"1min":13.711,"5min":13.711,"15min":20.329000000000001},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.99399999999999999,"5min":1.3320000000000001,"15min":1.149},"min":{"1min":0.32000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4.617,"5min":6.7560000000000002,"15min":20.408999999999999},"last":2.73}]},{"osd":2,"last update":"Tue Sep 17 07:21:02 2024","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":0.94099999999999995,"15min":1.1930000000000001},"min":{"1min":0.23699999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":2.298,"5min":4.4160000000000004,"15min":20.486000000000001},"last":1.276},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.91400000000000003,"15min":1.069},"min":{"1min":0.33200000000000002,"5min":0.219,"15min":0.219},"max":{"1min":0.78000000000000003,"5min":7.71,"15min":23.925999999999998},"last":2.415}]},{"osd":3,"last update":"Tue Sep 17 07:21:02 2024","interfaces":[{"interface":"back","average":{"1min":1.716,"5min":1.0880000000000001,"15min":1.591},"min":{"1min":0.27600000000000002,"5min":0.27500000000000002,"15min":0.247},"max":{"1min":16.143000000000001,"5min":16.143000000000001,"15min":68.978999999999999},"last":0.441},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.80900000000000005,"15min":1.361},"min":{"1min":0.39500000000000002,"5min":0.27100000000000002,"15min":0.253},"max":{"1min":0.76200000000000001,"5min":5.7880000000000003,"15min":69.084000000000003},"last":0.44900000000000001}]},{"osd":4,"last update":"Tue Sep 17 07:21:02 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":2.3380000000000001,"15min":2.141},"min":{"1min":0.247,"5min":0.247,"15min":0.22},"max":{"1min":2.4990000000000001,"5min":96.668000000000006,"15min":142.327},"last":0.33900000000000002},{"interface":"front","average":{"1min":1.099,"5min":2.3999999999999999,"15min":2.0579999999999998},"min":{"1min":0.311,"5min":0.255,"15min":0.20999999999999999},"max":{"1min":4.6580000000000004,"5min":96.801000000000002,"15min":142.13200000000001},"last":0.40200000000000002}]},{"osd":6,"last update":"Tue Sep 17 07:21:14 2024","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":2.0950000000000002,"15min":1.3380000000000001},"min":{"1min":0.374,"5min":0.245,"15min":0.23499999999999999},"max":{"1min":2.7469999999999999,"5min":96.686000000000007,"15min":96.686000000000007},"last":0.59099999999999997},{"interface":"front","average":{"1min":1.3839999999999999,"5min":2.5590000000000002,"15min":1.532},"min":{"1min":0.35399999999999998,"5min":0.29799999999999999,"15min":0.26700000000000002},"max":{"1min":2.8420000000000001,"5min":96.834999999999994,"15min":96.834999999999994},"last":0.54800000000000004}]},{"osd":7,"last update":"Tue Sep 17 07:20:42 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":1.1899999999999999,"15min":1.244},"min":{"1min":0.33300000000000002,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":2.5899999999999999,"5min":4.8419999999999996,"15min":6.8869999999999996},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.50900000000000001,"5min":1.0760000000000001,"15min":1.048},"min":{"1min":0.30199999999999999,"5min":0.28100000000000003,"15min":0.27100000000000002},"max":{"1min":0.79600000000000004,"5min":6.3639999999999999,"15min":6.3639999999999999},"last":0.49299999999999999}]}]},{"osd":4,"up_from":33,"seq":141733921021,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4337688,"kb_used_data":3390224,"kb_used_omap":1,"kb_used_meta":947454,"kb_avail":89407464,"statfs":{"total":95995035648,"available":91553243136,"internally_reserved":0,"allocated":3471589376,"data_stored":3470571512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":970193350},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:21:28 2024","interfaces":[{"interface":"back","average":{"1min":1.6200000000000001,"5min":1.2130000000000001,"15min":1.706},"min":{"1min":0.249,"5min":0.20799999999999999,"15min":0.20000000000000001},"max":{"1min":15.454000000000001,"5min":15.454000000000001,"15min":58.097999999999999},"last":0.36699999999999999},{"interface":"front","average":{"1min":1.722,"5min":1.4970000000000001,"15min":1.879},"min":{"1min":0.20599999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":15.760999999999999,"5min":16.619,"15min":58.383000000000003},"last":1.2470000000000001}]},{"osd":1,"last update":"Tue Sep 17 07:16:11 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":1.2070000000000001,"15min":1.4850000000000001},"min":{"1min":0.252,"5min":0.24399999999999999,"15min":0.23000000000000001},"max":{"1min":1.736,"5min":18.661000000000001,"15min":54.914999999999999},"last":0.435},{"interface":"front","average":{"1min":0.71499999999999997,"5min":1.3,"15min":1.444},"min":{"1min":0.246,"5min":0.23300000000000001,"15min":0.22800000000000001},"max":{"1min":3.7770000000000001,"5min":21.931999999999999,"15min":53.725000000000001},"last":2.698}]},{"osd":2,"last update":"Tue Sep 17 07:21:28 2024","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":1.0429999999999999,"15min":1.2110000000000001},"min":{"1min":0.252,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":3.657,"5min":8.4710000000000001,"15min":49.651000000000003},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.64500000000000002,"5min":1.1839999999999999,"15min":1.2270000000000001},"min":{"1min":0.26600000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":1.9970000000000001,"5min":16.655999999999999,"15min":48.813000000000002},"last":4.4779999999999998}]},{"osd":3,"last update":"Tue Sep 17 07:21:28 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":1.1279999999999999,"15min":1.0169999999999999},"min":{"1min":0.307,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":2.1099999999999999,"5min":15.433999999999999,"15min":17.872},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.98399999999999999,"15min":1.0509999999999999},"min":{"1min":0.32800000000000001,"5min":0.28100000000000003,"15min":0.253},"max":{"1min":2.403,"5min":16.111000000000001,"15min":17.908000000000001},"last":0.46300000000000002}]},{"osd":5,"last update":"Tue Sep 17 07:20:39 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":2.52,"15min":1.5529999999999999},"min":{"1min":0.224,"5min":0.224,"15min":0.20799999999999999},"max":{"1min":1.6919999999999999,"5min":147.61099999999999,"15min":147.61099999999999},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.73099999999999998,"5min":2.6890000000000001,"15min":1.613},"min":{"1min":0.27500000000000002,"5min":0.27400000000000002,"15min":0.26100000000000001},"max":{"1min":2.1320000000000001,"5min":147.74700000000001,"15min":147.74700000000001},"last":1.2110000000000001}]},{"osd":6,"last update":"Tue Sep 17 07:21:28 2024","interfaces":[{"interface":"back","average":{"1min":0.95899999999999996,"5min":1.3280000000000001,"15min":1.21},"min":{"1min":0.29699999999999999,"5min":0.23400000000000001,"15min":0.218},"max":{"1min":4.1950000000000003,"5min":8.9619999999999997,"15min":47.923999999999999},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.93799999999999994,"5min":1.3169999999999999,"15min":1.3080000000000001},"min":{"1min":0.25800000000000001,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":4.0419999999999998,"5min":5.774,"15min":47.600999999999999},"last":0.57199999999999995}]},{"osd":7,"last update":"Tue Sep 17 07:21:38 2024","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":1.284,"15min":1.296},"min":{"1min":0.32600000000000001,"5min":0.27900000000000003,"15min":0.26700000000000002},"max":{"1min":2.0259999999999998,"5min":7.1200000000000001,"15min":48.719000000000001},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.77600000000000002,"5min":1.137,"15min":1.337},"min":{"1min":0.34799999999999998,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":3.7829999999999999,"5min":9.0030000000000001,"15min":50.527000000000001},"last":0.53100000000000003}]}]},{"osd":3,"up_from":0,"seq":115964117247,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":18,"seq":77309411589,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5777292,"kb_used_data":4745348,"kb_used_omap":1,"kb_used_meta":1031934,"kb_avail":87967860,"statfs":{"total":95995035648,"available":90079088640,"internally_reserved":0,"allocated":4859236352,"data_stored":4858231465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1056700870},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:20:54 2024","interfaces":[{"interface":"back","average":{"1min":2,"5min":2.2949999999999999,"15min":1.9750000000000001},"min":{"1min":0.192,"5min":0.154,"15min":0.154},"max":{"1min":21.934000000000001,"5min":131.09800000000001,"15min":131.09800000000001},"last":1.278},{"interface":"front","average":{"1min":2.3119999999999998,"5min":2.339,"15min":2.0950000000000002},"min":{"1min":0.20399999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":21.591000000000001,"5min":131.24700000000001,"15min":131.24700000000001},"last":0.36499999999999999}]},{"osd":1,"last update":"Tue Sep 17 07:15:43 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":1.7949999999999999,"15min":1.623},"min":{"1min":0.23699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":1.6459999999999999,"5min":47.101999999999997,"15min":79.659000000000006},"last":1.4099999999999999},{"interface":"front","average":{"1min":0.55000000000000004,"5min":1.6990000000000001,"15min":1.5600000000000001},"min":{"1min":0.22700000000000001,"5min":0.20899999999999999,"15min":0.20200000000000001},"max":{"1min":1.8049999999999999,"5min":45.831000000000003,"15min":80.155000000000001},"last":2.0350000000000001}]},{"osd":3,"last update":"Tue Sep 17 07:21:08 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.44700000000000001,"15min":0.93799999999999994},"min":{"1min":0.219,"5min":0.20300000000000001,"15min":0.20100000000000001},"max":{"1min":3.1880000000000002,"5min":3.1880000000000002,"15min":23.43},"last":1.258},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.50900000000000001,"15min":0.95899999999999996},"min":{"1min":0.22500000000000001,"5min":0.22,"15min":0.22},"max":{"1min":2.4820000000000002,"5min":3.089,"15min":23.408000000000001},"last":1.623}]},{"osd":4,"last update":"Tue Sep 17 07:21:37 2024","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":1.8819999999999999,"15min":1.2689999999999999},"min":{"1min":0.27700000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":2.7730000000000001,"5min":75.870999999999995,"15min":75.870999999999995},"last":1.1699999999999999},{"interface":"front","average":{"1min":0.88400000000000001,"5min":2.0800000000000001,"15min":1.4219999999999999},"min":{"1min":0.35199999999999998,"5min":0.26900000000000002,"15min":0.26300000000000001},"max":{"1min":2.7879999999999998,"5min":75.888999999999996,"15min":75.888999999999996},"last":1.298}]},{"osd":5,"last update":"Tue Sep 17 07:20:54 2024","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":1.0469999999999999,"15min":1.2450000000000001},"min":{"1min":0.28899999999999998,"5min":0.27900000000000003,"15min":0.249},"max":{"1min":2.4260000000000002,"5min":7.2229999999999999,"15min":15.276999999999999},"last":0.312},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.74399999999999999,"15min":1.016},"min":{"1min":0.29099999999999998,"5min":0.27700000000000002,"15min":0.27600000000000002},"max":{"1min":1.087,"5min":5.8239999999999998,"15min":15.561999999999999},"last":1.198}]},{"osd":6,"last update":"Tue Sep 17 07:21:08 2024","interfaces":[{"interface":"back","average":{"1min":1.0349999999999999,"5min":2.1469999999999998,"15min":1.8240000000000001},"min":{"1min":0.34100000000000003,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":3.1000000000000001,"5min":76.081999999999994,"15min":101.587},"last":1.3180000000000001},{"interface":"front","average":{"1min":1.3280000000000001,"5min":2.5659999999999998,"15min":1.9339999999999999},"min":{"1min":0.35099999999999998,"5min":0.313,"15min":0.28299999999999997},"max":{"1min":2.46,"5min":76.244,"15min":100.696},"last":0.33900000000000002}]},{"osd":7,"last update":"Tue Sep 17 07:20:54 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":1.869,"15min":1.411},"min":{"1min":0.33500000000000002,"5min":0.316,"15min":0.316},"max":{"1min":2.262,"5min":94.269999999999996,"15min":94.269999999999996},"last":1.2270000000000001},{"interface":"front","average":{"1min":0.80100000000000005,"5min":1.867,"15min":1.446},"min":{"1min":0.36699999999999999,"5min":0.309,"15min":0.309},"max":{"1min":3.218,"5min":94.634,"15min":94.634},"last":1.337}]}]},{"osd":1,"up_from":413,"seq":1773821493254,"num_pgs":86,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":815864,"kb_used_data":683312,"kb_used_omap":4,"kb_used_meta":132539,"kb_avail":92929288,"statfs":{"total":95995035648,"available":95159590912,"internally_reserved":0,"allocated":699711488,"data_stored":698662923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4439,"internal_metadata":135720617},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.9729999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.7220000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.9290000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5549999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.581}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.9509999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60199999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738637,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":8044260,"kb_used_data":6853724,"kb_used_omap":1,"kb_used_meta":1190526,"kb_avail":85700892,"statfs":{"total":95995035648,"available":87757713408,"internally_reserved":0,"allocated":7018213376,"data_stored":7017184053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1219099078},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,5,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 17 07:15:18 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.89700000000000002,"15min":0.89500000000000002},"min":{"1min":0.254,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":1.518,"5min":21.876000000000001,"15min":21.876000000000001},"last":3.2240000000000002},{"interface":"front","average":{"1min":0.56399999999999995,"5min":1.2330000000000001,"15min":1.081},"min":{"1min":0.17899999999999999,"5min":0.158,"15min":0.158},"max":{"1min":2.2599999999999998,"5min":22.521999999999998,"15min":22.521999999999998},"last":0.84099999999999997}]},{"osd":2,"last update":"Tue Sep 17 07:20:51 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.45000000000000001,"15min":0.97699999999999998},"min":{"1min":0.19400000000000001,"5min":0.155,"15min":0.155},"max":{"1min":2.1619999999999999,"5min":2.6520000000000001,"15min":18.643999999999998},"last":1.236},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.436,"15min":0.91100000000000003},"min":{"1min":0.23999999999999999,"5min":0.183,"15min":0.183},"max":{"1min":1.5760000000000001,"5min":3.1760000000000002,"15min":17.972999999999999},"last":2.032}]},{"osd":3,"last update":"Tue Sep 17 07:21:27 2024","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.55400000000000005,"15min":0.94899999999999995},"min":{"1min":0.26500000000000001,"5min":0.223,"15min":0.21199999999999999},"max":{"1min":2.3130000000000002,"5min":3.2000000000000002,"15min":18.629000000000001},"last":0.745},{"interface":"front","average":{"1min":1.3120000000000001,"5min":0.59499999999999997,"15min":0.91000000000000003},"min":{"1min":0.222,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":5.1150000000000002,"5min":5.1150000000000002,"15min":22.358000000000001},"last":1.0329999999999999}]},{"osd":4,"last update":"Tue Sep 17 07:21:27 2024","interfaces":[{"interface":"back","average":{"1min":1.5429999999999999,"5min":1.1739999999999999,"15min":1.1919999999999999},"min":{"1min":0.39900000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":5.2549999999999999,"5min":8.3279999999999994,"15min":19.146999999999998},"last":1.8149999999999999},{"interface":"front","average":{"1min":1.0880000000000001,"5min":1.1699999999999999,"15min":1.2010000000000001},"min":{"1min":0.30199999999999999,"5min":0.26200000000000001,"15min":0.255},"max":{"1min":2.7909999999999999,"5min":8.3119999999999994,"15min":19.158999999999999},"last":0.81000000000000005}]},{"osd":5,"last update":"Tue Sep 17 07:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":3.629,"15min":2.1600000000000001},"min":{"1min":0.30399999999999999,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":3.7709999999999999,"5min":216.83699999999999,"15min":216.83699999999999},"last":1.137},{"interface":"front","average":{"1min":0.83599999999999997,"5min":3.79,"15min":2.181},"min":{"1min":0.33900000000000002,"5min":0.29599999999999999,"15min":0.27000000000000002},"max":{"1min":2.278,"5min":216.816,"15min":216.816},"last":1.282}]},{"osd":6,"last update":"Tue Sep 17 07:21:27 2024","interfaces":[{"interface":"back","average":{"1min":1.6160000000000001,"5min":3.145,"15min":1.8240000000000001},"min":{"1min":0.29899999999999999,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":6.6920000000000002,"5min":147.13200000000001,"15min":147.13200000000001},"last":0.68799999999999994},{"interface":"front","average":{"1min":1.677,"5min":3.0710000000000002,"15min":1.8220000000000001},"min":{"1min":0.32500000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":5.282,"5min":147.172,"15min":147.172},"last":5.4669999999999996}]},{"osd":7,"last update":"Tue Sep 17 07:21:27 2024","interfaces":[{"interface":"back","average":{"1min":1.0720000000000001,"5min":1.131,"15min":1.421},"min":{"1min":0.39100000000000001,"5min":0.314,"15min":0.29599999999999999},"max":{"1min":3.7189999999999999,"5min":6.3300000000000001,"15min":46.183},"last":0.88100000000000001},{"interface":"front","average":{"1min":1.4870000000000001,"5min":1.232,"15min":1.4570000000000001},"min":{"1min":0.41299999999999998,"5min":0.32600000000000001,"15min":0.32000000000000001},"max":{"1min":6.7300000000000004,"5min":7.2830000000000004,"15min":45.914999999999999},"last":0.72599999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":2024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":2558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":7038042112,"data_stored":7038042112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":708247552,"data_stored":708247552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4865130496,"data_stored":4865130496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3476029440,"data_stored":3476029440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3477274624,"data_stored":3477274624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":6279856128,"data_stored":6279856128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T07:21:47.295 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd pool set unique_pool_4 pgp_num 16 2024-09-17T07:21:47.367 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:21:47.368 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:21:47.375 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:21:47.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:47.395 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:47.395 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:21:47.496 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:21:47.516 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:47 smithi100 ceph-mon[29092]: pgmap v918: 158 pgs: 5 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 4 active+undersized+degraded+remapped+backfill_toofull, 146 active+clean; 10 GiB data, 29 GiB used, 508 GiB / 536 GiB avail; 1.4 KiB/s rd, 66 MiB/s wr, 1.72k op/s; 112277/501291 objects degraded (22.398%); 35 MiB/s, 554 objects/s recovering 2024-09-17T07:21:47.516 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:47 smithi100 ceph-mon[39462]: pgmap v918: 158 pgs: 5 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 4 active+undersized+degraded+remapped+backfill_toofull, 146 active+clean; 10 GiB data, 29 GiB used, 508 GiB / 536 GiB avail; 1.4 KiB/s rd, 66 MiB/s wr, 1.72k op/s; 112277/501291 objects degraded (22.398%); 35 MiB/s, 554 objects/s recovering 2024-09-17T07:21:47.541 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:47.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:47.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:21:47.643 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:21:47.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:47.674 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:47.674 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:21:47.775 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:21:47.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:47 smithi135 ceph-mon[33238]: pgmap v918: 158 pgs: 5 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 4 active+undersized+degraded+remapped+backfill_toofull, 146 active+clean; 10 GiB data, 29 GiB used, 508 GiB / 536 GiB avail; 1.4 KiB/s rd, 66 MiB/s wr, 1.72k op/s; 112277/501291 objects degraded (22.398%); 35 MiB/s, 554 objects/s recovering 2024-09-17T07:21:47.801 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:47.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:47.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:21:47.865 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-17T07:21:47.903 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:21:47.937 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:47.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:47.938 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:21:48.039 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:21:48.081 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:48.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:48.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:21:48.180 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:21:48.184 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:21:48.225 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:48.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:48.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:21:48.327 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:21:48.358 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:48.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:48.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:21:48.460 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:21:48.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:48.504 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:48.504 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:21:48.604 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:21:48.641 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:48.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:48.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:21:48.743 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:21:48.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:48.773 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:48.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:21:48.874 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:21:48.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:48.911 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:48.911 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:21:49.012 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:21:49.040 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:49.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:49.042 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:21:49.143 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:21:49.171 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:49.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:49.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:21:49.273 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:21:49.309 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:49.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:49.310 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:21:49.411 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:21:49.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:49.448 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:49.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:21:49.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:49 smithi135 ceph-mon[33238]: pgmap v919: 158 pgs: 5 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 4 active+undersized+degraded+remapped+backfill_toofull, 146 active+clean; 10 GiB data, 29 GiB used, 508 GiB / 536 GiB avail; 767 B/s rd, 28 MiB/s wr, 1.11k op/s; 112277/501291 objects degraded (22.398%); 17 MiB/s, 273 objects/s recovering 2024-09-17T07:21:49.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:49 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:21:49.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:49 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:21:49.543 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:21:49 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:21:49.550 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:21:49.582 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:49.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:49.584 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:21:49.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:49 smithi100 ceph-mon[29092]: pgmap v919: 158 pgs: 5 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 4 active+undersized+degraded+remapped+backfill_toofull, 146 active+clean; 10 GiB data, 29 GiB used, 508 GiB / 536 GiB avail; 767 B/s rd, 28 MiB/s wr, 1.11k op/s; 112277/501291 objects degraded (22.398%); 17 MiB/s, 273 objects/s recovering 2024-09-17T07:21:49.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:49 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:21:49.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:49 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:21:49.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:49 smithi100 ceph-mon[39462]: pgmap v919: 158 pgs: 5 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 4 active+undersized+degraded+remapped+backfill_toofull, 146 active+clean; 10 GiB data, 29 GiB used, 508 GiB / 536 GiB avail; 767 B/s rd, 28 MiB/s wr, 1.11k op/s; 112277/501291 objects degraded (22.398%); 17 MiB/s, 273 objects/s recovering 2024-09-17T07:21:49.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:49 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:21:49.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:49 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:21:49.684 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:21:49.719 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:49.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:49.720 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:21:49.821 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:21:49.856 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:49.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:49.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:21:49.958 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:21:49.989 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:49.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:49.990 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:21:50.091 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:21:50.121 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:50.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:50.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:21:50.223 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:21:50.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:50.254 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:50.254 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:21:50.355 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:21:50.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:50.407 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:50.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:21:50.508 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:21:50.539 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:50.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:50.540 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:21:50.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:50 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:21:50.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:50 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 112277/501291 objects degraded (22.398%), 12 pgs degraded (PG_DEGRADED) 2024-09-17T07:21:50.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:50 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:21:50.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:50 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 112277/501291 objects degraded (22.398%), 12 pgs degraded (PG_DEGRADED) 2024-09-17T07:21:50.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:50 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:21:50.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:50 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 112277/501291 objects degraded (22.398%), 12 pgs degraded (PG_DEGRADED) 2024-09-17T07:21:50.641 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:21:50.670 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:50.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:50.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:21:50.772 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:21:50.804 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:50.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:50.805 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:21:50.888 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:21:50.905 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:21:50.931 INFO:teuthology.orchestra.run.smithi135.stdout:{ 2024-09-17T07:21:50.932 INFO:teuthology.orchestra.run.smithi135.stdout: "ops": [], 2024-09-17T07:21:50.932 INFO:teuthology.orchestra.run.smithi135.stdout: "complaint_time": 30, 2024-09-17T07:21:50.932 INFO:teuthology.orchestra.run.smithi135.stdout: "num_blocked_ops": 0 2024-09-17T07:21:50.932 INFO:teuthology.orchestra.run.smithi135.stdout:} 2024-09-17T07:21:50.937 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:50.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:50.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:21:51.039 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:21:51.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:51.070 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:51.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:21:51.171 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:21:51.201 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:51.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:51.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:21:51.303 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:21:51.336 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:51.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:51.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:21:51.438 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:21:51.473 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:51.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:51.474 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:21:51.575 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:21:51.605 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:51.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:51.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:21:51.607 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:51 smithi100 ceph-mon[29092]: pgmap v920: 158 pgs: 3 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 5 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 146 active+clean; 9.9 GiB data, 28 GiB used, 508 GiB / 536 GiB avail; 1.4 KiB/s rd, 11 MiB/s wr, 1.41k op/s; 107739/487695 objects degraded (22.091%); 34 MiB/s, 550 objects/s recovering 2024-09-17T07:21:51.607 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:51 smithi100 ceph-mon[39462]: pgmap v920: 158 pgs: 3 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 5 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 146 active+clean; 9.9 GiB data, 28 GiB used, 508 GiB / 536 GiB avail; 1.4 KiB/s rd, 11 MiB/s wr, 1.41k op/s; 107739/487695 objects degraded (22.091%); 34 MiB/s, 550 objects/s recovering 2024-09-17T07:21:51.709 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:21:51.735 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:51 smithi135 ceph-mon[33238]: pgmap v920: 158 pgs: 3 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 5 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 146 active+clean; 9.9 GiB data, 28 GiB used, 508 GiB / 536 GiB avail; 1.4 KiB/s rd, 11 MiB/s wr, 1.41k op/s; 107739/487695 objects degraded (22.091%); 34 MiB/s, 550 objects/s recovering 2024-09-17T07:21:51.742 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:51.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:51.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:21:51.768 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-17T07:21:51.845 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:21:51.889 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:51.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:51.891 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:21:51.991 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:21:52.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:52.028 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:52.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:21:52.103 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:21:52.129 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:21:52.162 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:52.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:52.163 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:21:52.264 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:21:52.291 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:52.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:52.292 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:21:52.393 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:21:52.437 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:52.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:52.438 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:21:52.539 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:21:52.578 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:52.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:52.579 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:21:52.679 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:21:52.701 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:52.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:52.702 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:21:52.803 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:21:52.837 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:52.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:52.838 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:21:52.939 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:21:52.972 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:52.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:52.973 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:21:53.074 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:21:53.112 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:53.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:53.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:21:53.214 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:21:53.240 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:53.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:53.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:21:53.341 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:21:53.374 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:53.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:53.375 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:21:53.476 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:21:53.509 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:53.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:53.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:21:53.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:53 smithi135 ceph-mon[33238]: pgmap v921: 158 pgs: 3 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 5 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 146 active+clean; 9.8 GiB data, 28 GiB used, 508 GiB / 536 GiB avail; 1.1 KiB/s rd, 9.0 MiB/s wr, 1.35k op/s; 105629/481353 objects degraded (21.944%); 28 MiB/s, 442 objects/s recovering 2024-09-17T07:21:53.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:53 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/3174453418' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:21:53.545 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:53 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:21:53.611 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:21:53.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:53 smithi100 ceph-mon[29092]: pgmap v921: 158 pgs: 3 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 5 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 146 active+clean; 9.8 GiB data, 28 GiB used, 508 GiB / 536 GiB avail; 1.1 KiB/s rd, 9.0 MiB/s wr, 1.35k op/s; 105629/481353 objects degraded (21.944%); 28 MiB/s, 442 objects/s recovering 2024-09-17T07:21:53.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:53 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/3174453418' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:21:53.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:53 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:21:53.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:53 smithi100 ceph-mon[39462]: pgmap v921: 158 pgs: 3 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 5 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 146 active+clean; 9.8 GiB data, 28 GiB used, 508 GiB / 536 GiB avail; 1.1 KiB/s rd, 9.0 MiB/s wr, 1.35k op/s; 105629/481353 objects degraded (21.944%); 28 MiB/s, 442 objects/s recovering 2024-09-17T07:21:53.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:53 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/3174453418' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:21:53.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:53 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:21:53.646 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:53.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:53.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:21:53.748 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:21:53.781 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:53.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:53.783 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:21:53.810 INFO:teuthology.orchestra.run.smithi100.stderr:nodeep-scrub is set 2024-09-17T07:21:53.883 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:21:53.909 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:53.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:53.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:21:54.011 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:21:54.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:54.046 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:54.046 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:21:54.147 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:21:54.184 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:54.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:54.185 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:21:54.286 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:21:54.310 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:54.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:54.311 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:21:54.411 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:21:54.436 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:54.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:54.438 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:21:54.538 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:21:54.569 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:54.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:54.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:21:54.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:54 smithi100 ceph-mon[29092]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:21:54.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:54 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T07:21:54.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:54 smithi100 ceph-mon[29092]: osdmap e425: 8 total, 8 up, 6 in 2024-09-17T07:21:54.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:54 smithi100 ceph-mon[39462]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:21:54.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:54 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T07:21:54.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:54 smithi100 ceph-mon[39462]: osdmap e425: 8 total, 8 up, 6 in 2024-09-17T07:21:54.671 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:21:54.693 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:54.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:54.694 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:21:54.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:54 smithi135 ceph-mon[33238]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:21:54.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:54 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T07:21:54.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:54 smithi135 ceph-mon[33238]: osdmap e425: 8 total, 8 up, 6 in 2024-09-17T07:21:54.794 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:21:54.832 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:54.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:54.833 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:21:54.934 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:21:54.968 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:54.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:54.970 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:21:55.071 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:21:55.110 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:55.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:55.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:21:55.212 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:21:55.237 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:55.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:55.238 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:21:55.338 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:21:55.380 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:55.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:55.381 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:21:55.482 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:21:55.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:55.514 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:55.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:21:55.615 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:21:55.647 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:55.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:55.648 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:21:55.749 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:21:55.763 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:55 smithi100 ceph-mon[29092]: pgmap v923: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 5 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 146 active+clean; 9.7 GiB data, 28 GiB used, 509 GiB / 536 GiB avail; 631 B/s rd, 0 B/s wr, 975 op/s; 104768/477579 objects degraded (21.937%); 17 MiB/s, 272 objects/s recovering 2024-09-17T07:21:55.763 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:55 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 104768/477579 objects degraded (21.937%), 12 pgs degraded (PG_DEGRADED) 2024-09-17T07:21:55.763 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:55 smithi100 ceph-mon[29092]: osdmap e426: 8 total, 8 up, 6 in 2024-09-17T07:21:55.763 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:55 smithi100 ceph-mon[39462]: pgmap v923: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 5 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 146 active+clean; 9.7 GiB data, 28 GiB used, 509 GiB / 536 GiB avail; 631 B/s rd, 0 B/s wr, 975 op/s; 104768/477579 objects degraded (21.937%); 17 MiB/s, 272 objects/s recovering 2024-09-17T07:21:55.763 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:55 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 104768/477579 objects degraded (21.937%), 12 pgs degraded (PG_DEGRADED) 2024-09-17T07:21:55.763 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:55 smithi100 ceph-mon[39462]: osdmap e426: 8 total, 8 up, 6 in 2024-09-17T07:21:55.781 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:55.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:55.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:21:55.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:55 smithi135 ceph-mon[33238]: pgmap v923: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 5 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 146 active+clean; 9.7 GiB data, 28 GiB used, 509 GiB / 536 GiB avail; 631 B/s rd, 0 B/s wr, 975 op/s; 104768/477579 objects degraded (21.937%); 17 MiB/s, 272 objects/s recovering 2024-09-17T07:21:55.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:55 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 104768/477579 objects degraded (21.937%), 12 pgs degraded (PG_DEGRADED) 2024-09-17T07:21:55.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:55 smithi135 ceph-mon[33238]: osdmap e426: 8 total, 8 up, 6 in 2024-09-17T07:21:55.884 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:21:55.913 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:55.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:55.914 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:21:56.015 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:21:56.020 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:21:55 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:21:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:21:56.047 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:56.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:56.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:21:56.094 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:21:56.094 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:21:56.132 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1: {} 2024-09-17T07:21:56.133 INFO:teuthology.orchestra.run.smithi100.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:21:56.149 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:21:56.151 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:21:56.152 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:21:56.168 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:21:56.168 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:21:56.170 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:56.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:56.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:21:56.186 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:21:56.186 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:21:56.208 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:21:56.208 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:21:56.247 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:21:56.247 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:21:56.272 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:21:56.277 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:21:56.278 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:21:56.301 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:56.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:56.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:21:56.402 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:21:56.429 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:56.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:56.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:21:56.445 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:56 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/3365687348' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:21:56.531 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:21:56.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:56.567 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:56.568 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:21:56.584 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:56 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/3365687348' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:21:56.585 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:56 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/3365687348' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:21:56.668 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:21:56.712 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:56.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:56.713 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:21:56.814 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:21:56.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:56.844 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:56.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:21:56.945 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:21:56.987 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:56.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:56.988 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:21:57.088 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:21:57.119 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:57.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:57.120 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:21:57.220 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:21:57.256 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:57.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:57.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:21:57.358 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:21:57.403 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:57.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:57.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:21:57.505 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:21:57.531 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T07:21:57.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:57.545 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:57.546 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:21:57.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:57 smithi100 ceph-mon[29092]: pgmap v925: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 5 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 146 active+clean; 9.4 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.63k op/s; 99422/462066 objects degraded (21.517%); 33 MiB/s, 525 objects/s recovering 2024-09-17T07:21:57.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:57 smithi100 ceph-mon[39462]: pgmap v925: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 5 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 146 active+clean; 9.4 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.63k op/s; 99422/462066 objects degraded (21.517%); 33 MiB/s, 525 objects/s recovering 2024-09-17T07:21:57.646 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:21:57.679 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:57.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:57.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:21:57.716 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:57 smithi135 ceph-mon[33238]: pgmap v925: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 5 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 146 active+clean; 9.4 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.63k op/s; 99422/462066 objects degraded (21.517%); 33 MiB/s, 525 objects/s recovering 2024-09-17T07:21:57.781 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:21:57.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:57.819 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:57.819 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:21:57.920 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:21:57.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:57.952 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:57.952 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:21:58.053 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:21:58.079 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:58.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:58.080 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:21:58.086 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:21:58.176 INFO:teuthology.orchestra.run.smithi135.stdout:ERROR: (22) Invalid argument 2024-09-17T07:21:58.177 INFO:teuthology.orchestra.run.smithi135.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:21:58.181 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:21:58.227 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:58.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:58.228 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:21:58.329 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:21:58.367 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:58.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:58.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:21:58.428 INFO:teuthology.orchestra.run.smithi100.stderr:set pool 11 pgp_num to 16 2024-09-17T07:21:58.468 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:21:58.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:58.493 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:58.494 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:21:58.594 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:21:58.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:58 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2707319670' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]: dispatch 2024-09-17T07:21:58.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:58 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2707319670' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]: dispatch 2024-09-17T07:21:58.631 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:58.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:58.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:21:58.733 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:21:58.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:58.768 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:58.768 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:21:58.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:58 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2707319670' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]: dispatch 2024-09-17T07:21:58.869 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:21:58.905 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:58.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:58.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:21:59.006 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:21:59.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:59.049 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:21:59.049 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:21:59.150 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:21:59.175 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:59.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:59.176 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:21:59.277 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:21:59.299 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:59.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:59.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:21:59.396 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:21:59.401 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:21:59.428 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:59.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:59.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:21:59.530 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:21:59.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:59 smithi135 ceph-mon[33238]: pgmap v926: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 5 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 146 active+clean; 9.4 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 1.07k op/s; 99422/462066 objects degraded (21.517%); 16 MiB/s, 249 objects/s recovering 2024-09-17T07:21:59.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:59 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2707319670' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]': finished 2024-09-17T07:21:59.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:21:59 smithi135 ceph-mon[33238]: osdmap e427: 8 total, 8 up, 6 in 2024-09-17T07:21:59.543 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:21:59 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:21:59.554 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:59.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:59.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:21:59.656 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:21:59.687 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:59.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:59.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:21:59.789 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:21:59.819 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:59.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:59.820 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:21:59.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:59 smithi100 ceph-mon[29092]: pgmap v926: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 5 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 146 active+clean; 9.4 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 1.07k op/s; 99422/462066 objects degraded (21.517%); 16 MiB/s, 249 objects/s recovering 2024-09-17T07:21:59.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:59 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2707319670' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]': finished 2024-09-17T07:21:59.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:21:59 smithi100 ceph-mon[29092]: osdmap e427: 8 total, 8 up, 6 in 2024-09-17T07:21:59.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:59 smithi100 ceph-mon[39462]: pgmap v926: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 5 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 146 active+clean; 9.4 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 1.07k op/s; 99422/462066 objects degraded (21.517%); 16 MiB/s, 249 objects/s recovering 2024-09-17T07:21:59.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:59 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2707319670' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]': finished 2024-09-17T07:21:59.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:21:59 smithi100 ceph-mon[39462]: osdmap e427: 8 total, 8 up, 6 in 2024-09-17T07:21:59.921 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:21:59.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:21:59.948 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:21:59.949 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:22:00.050 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:22:00.085 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:00.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:00.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:22:00.187 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:22:00.214 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:00.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:00.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:22:00.316 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:22:00.339 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:00.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:00.340 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:22:00.441 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:22:00.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:00.466 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:00.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:22:00.567 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:22:00.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:00.594 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:00.595 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:22:00.695 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:22:00.733 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:00.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:00.734 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:22:00.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:00 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:22:00.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:00 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 99422/462066 objects degraded (21.517%), 12 pgs degraded (PG_DEGRADED) 2024-09-17T07:22:00.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:00 smithi135 ceph-mon[33238]: osdmap e428: 8 total, 8 up, 6 in 2024-09-17T07:22:00.835 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:22:00.856 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:00.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:00.858 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:22:00.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:00 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:22:00.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:00 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 99422/462066 objects degraded (21.517%), 12 pgs degraded (PG_DEGRADED) 2024-09-17T07:22:00.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:00 smithi100 ceph-mon[29092]: osdmap e428: 8 total, 8 up, 6 in 2024-09-17T07:22:00.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:00 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:22:00.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:00 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 99422/462066 objects degraded (21.517%), 12 pgs degraded (PG_DEGRADED) 2024-09-17T07:22:00.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:00 smithi100 ceph-mon[39462]: osdmap e428: 8 total, 8 up, 6 in 2024-09-17T07:22:00.959 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:22:01.002 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:01.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:01.004 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:22:01.104 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:22:01.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:01.143 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:01.143 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:22:01.244 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:22:01.278 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:01.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:01.279 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:22:01.380 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:22:01.416 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:01.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:01.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:22:01.512 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-17T07:22:01.518 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:22:01.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:01.602 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:01.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:22:01.703 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:22:01.739 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:01.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:01.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:22:01.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:01 smithi135 ceph-mon[33238]: pgmap v928: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 5 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 146 active+clean; 9.1 GiB data, 27 GiB used, 510 GiB / 536 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.71k op/s; 95183/449058 objects degraded (21.196%); 43 MiB/s, 682 objects/s recovering 2024-09-17T07:22:01.837 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:22:01.841 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:22:01.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:01 smithi100 ceph-mon[29092]: pgmap v928: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 5 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 146 active+clean; 9.1 GiB data, 27 GiB used, 510 GiB / 536 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.71k op/s; 95183/449058 objects degraded (21.196%); 43 MiB/s, 682 objects/s recovering 2024-09-17T07:22:01.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:01 smithi100 ceph-mon[39462]: pgmap v928: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 5 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 146 active+clean; 9.1 GiB data, 27 GiB used, 510 GiB / 536 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.71k op/s; 95183/449058 objects degraded (21.196%); 43 MiB/s, 682 objects/s recovering 2024-09-17T07:22:01.881 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:01.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:01.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:22:01.983 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:22:02.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:02.017 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:02.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:22:02.118 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:22:02.170 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:02.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:02.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:22:02.272 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:22:02.320 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:02.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:02.321 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:22:02.422 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:22:02.454 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:02.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:02.455 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:22:02.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:02 smithi135 ceph-mon[33238]: pgmap v930: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 5 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 146 active+clean; 9.0 GiB data, 27 GiB used, 510 GiB / 536 GiB avail; 838 B/s rd, 0 B/s wr, 1.04k op/s; 93199/443067 objects degraded (21.035%); 22 MiB/s, 350 objects/s recovering 2024-09-17T07:22:02.556 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:22:02.591 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:02.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:02.592 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:22:02.693 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:22:02.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:02.725 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:02.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:22:02.741 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:02 smithi100 ceph-mon[29092]: pgmap v930: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 5 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 146 active+clean; 9.0 GiB data, 27 GiB used, 510 GiB / 536 GiB avail; 838 B/s rd, 0 B/s wr, 1.04k op/s; 93199/443067 objects degraded (21.035%); 22 MiB/s, 350 objects/s recovering 2024-09-17T07:22:02.742 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:02 smithi100 ceph-mon[39462]: pgmap v930: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 5 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 146 active+clean; 9.0 GiB data, 27 GiB used, 510 GiB / 536 GiB avail; 838 B/s rd, 0 B/s wr, 1.04k op/s; 93199/443067 objects degraded (21.035%); 22 MiB/s, 350 objects/s recovering 2024-09-17T07:22:02.825 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:22:02.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:02.866 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:02.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:22:02.967 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:22:03.005 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:03.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:03.006 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:22:03.107 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:22:03.135 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:03.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:03.136 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:22:03.237 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:22:03.281 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:03.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:03.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:22:03.355 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:22:03.383 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:22:03.411 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:03.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:03.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:22:03.513 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:22:03.542 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:03.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:03.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:22:03.643 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:22:03.681 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:03.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:03.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:22:03.783 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:22:03.806 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:03.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:03.807 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:22:03.908 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:22:03.939 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:03.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:03.941 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:22:04.041 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:22:04.081 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:04.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:04.082 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:22:04.183 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:22:04.210 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:04.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:04.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:22:04.312 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:22:04.351 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:04.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:04.352 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:22:04.453 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:22:04.462 INFO:teuthology.orchestra.run.smithi100.stderr:noscrub is unset 2024-09-17T07:22:04.496 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:04.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:04.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:22:04.597 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:22:04.622 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:04.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:04.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:22:04.724 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:22:04.747 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:04 smithi100 ceph-mon[29092]: pgmap v931: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 5 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 146 active+clean; 8.9 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 927 op/s; 92499/439812 objects degraded (21.031%); 17 MiB/s, 267 objects/s recovering 2024-09-17T07:22:04.747 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:04 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:22:04.747 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:04 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/3090441944' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:22:04.748 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:04 smithi100 ceph-mon[39462]: pgmap v931: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 5 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 146 active+clean; 8.9 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 927 op/s; 92499/439812 objects degraded (21.031%); 17 MiB/s, 267 objects/s recovering 2024-09-17T07:22:04.748 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:04 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:22:04.748 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:04 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/3090441944' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:22:04.758 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:04.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:04.759 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:22:04.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:04 smithi135 ceph-mon[33238]: pgmap v931: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 5 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 146 active+clean; 8.9 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 927 op/s; 92499/439812 objects degraded (21.031%); 17 MiB/s, 267 objects/s recovering 2024-09-17T07:22:04.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:04 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:22:04.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:04 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/3090441944' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:22:04.860 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:22:04.908 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:04.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:04.909 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:22:05.010 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:22:05.044 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:05.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:05.045 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:22:05.146 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:22:05.177 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:05.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:05.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:22:05.279 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:22:05.313 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:05.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:05.314 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:22:05.401 INFO:teuthology.orchestra.run.smithi135.stdout:ERROR: (22) Invalid argument 2024-09-17T07:22:05.401 INFO:teuthology.orchestra.run.smithi135.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:22:05.415 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:22:05.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:05.455 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:05.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:22:05.557 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:22:05.593 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:05.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:05.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:22:05.645 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:05 smithi135 ceph-mon[33238]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:22:05.645 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:05 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/3090441944' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T07:22:05.645 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:05 smithi135 ceph-mon[33238]: osdmap e429: 8 total, 8 up, 6 in 2024-09-17T07:22:05.645 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:05 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:22:05.645 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:05 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:22:05.645 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:05 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 92499/439812 objects degraded (21.031%), 12 pgs degraded (PG_DEGRADED) 2024-09-17T07:22:05.645 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:05 smithi135 ceph-mon[33238]: osdmap e430: 8 total, 8 up, 6 in 2024-09-17T07:22:05.695 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:22:05.725 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:05.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:05.726 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:22:05.796 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:05 smithi100 ceph-mon[29092]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:22:05.796 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:05 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/3090441944' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T07:22:05.796 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:05 smithi100 ceph-mon[29092]: osdmap e429: 8 total, 8 up, 6 in 2024-09-17T07:22:05.796 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:05 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:22:05.796 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:05 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:22:05.797 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:05 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 92499/439812 objects degraded (21.031%), 12 pgs degraded (PG_DEGRADED) 2024-09-17T07:22:05.797 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:05 smithi100 ceph-mon[29092]: osdmap e430: 8 total, 8 up, 6 in 2024-09-17T07:22:05.797 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:05 smithi100 ceph-mon[39462]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:22:05.797 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:05 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/3090441944' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T07:22:05.797 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:05 smithi100 ceph-mon[39462]: osdmap e429: 8 total, 8 up, 6 in 2024-09-17T07:22:05.797 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:05 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:22:05.797 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:05 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:22:05.797 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:05 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 92499/439812 objects degraded (21.031%), 12 pgs degraded (PG_DEGRADED) 2024-09-17T07:22:05.797 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:05 smithi100 ceph-mon[39462]: osdmap e430: 8 total, 8 up, 6 in 2024-09-17T07:22:05.827 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:22:05.855 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:05.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:05.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:22:05.957 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:22:05.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:05.997 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:05.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:22:06.065 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:22:05 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:22:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:22:06.082 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-17T07:22:06.098 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:22:06.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:06.145 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:06.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:22:06.162 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:22:06.163 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:22:06.195 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1: {} 2024-09-17T07:22:06.195 INFO:teuthology.orchestra.run.smithi100.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:22:06.227 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:22:06.227 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:22:06.245 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:22:06.258 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:22:06.258 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:22:06.281 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:06.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:06.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:22:06.289 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:22:06.290 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:22:06.324 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:22:06.324 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:22:06.354 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:22:06.354 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:22:06.383 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:22:06.386 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:22:06.387 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:22:06.396 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:22:06.418 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:06.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:06.419 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:22:06.519 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:22:06.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:06.633 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:06.633 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:22:06.633 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:06 smithi100 ceph-mon[29092]: pgmap v934: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 5 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 146 active+clean; 8.6 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 1.34k op/s; 87032/424938 objects degraded (20.481%); 25 MiB/s, 397 objects/s recovering 2024-09-17T07:22:06.634 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:06 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2587358071' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:22:06.634 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:06 smithi100 ceph-mon[39462]: pgmap v934: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 5 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 146 active+clean; 8.6 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 1.34k op/s; 87032/424938 objects degraded (20.481%); 25 MiB/s, 397 objects/s recovering 2024-09-17T07:22:06.634 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:06 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2587358071' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:22:06.733 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:22:06.754 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:06.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:06.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:22:06.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:06 smithi135 ceph-mon[33238]: pgmap v934: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 5 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 146 active+clean; 8.6 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 1.34k op/s; 87032/424938 objects degraded (20.481%); 25 MiB/s, 397 objects/s recovering 2024-09-17T07:22:06.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:06 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2587358071' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:22:06.857 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:22:06.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:06.890 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:06.890 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:22:06.991 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:22:07.030 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:07.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:07.031 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:22:07.132 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:22:07.164 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:07.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:07.165 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:22:07.266 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:22:07.298 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:07.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:07.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:22:07.399 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:22:07.432 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:07.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:07.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:22:07.469 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-17T07:22:07.534 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:22:07.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:07.593 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:07.594 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:22:07.694 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:22:07.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:07.740 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:07.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:22:07.773 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6] out_osds: [3, 7] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 6, 7, 1] 2024-09-17T07:22:07.774 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T07:22:07.774 INFO:tasks.thrashosds.thrasher:Adding osd 7 2024-09-17T07:22:07.774 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd in 7 2024-09-17T07:22:07.777 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:22:07.841 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:22:07.918 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:07.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:07.920 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:22:08.020 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:22:08.063 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:08.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:08.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:22:08.164 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:22:08.205 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:08.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:08.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:22:08.308 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:22:08.351 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:08.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:08.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:22:08.453 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:22:08.494 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:08.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:08.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:22:08.596 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:22:08.613 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:22:08.622 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:08.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:08.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:22:08.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:08 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:22:08.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:08 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:22:08.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:08 smithi100 ceph-mon[39462]: pgmap v935: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 5 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 146 active+clean; 8.6 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 719 B/s rd, 0 B/s wr, 850 op/s; 87032/424938 objects degraded (20.481%); 21 MiB/s, 335 objects/s recovering 2024-09-17T07:22:08.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:08 smithi100 ceph-mon[39462]: osdmap e431: 8 total, 8 up, 6 in 2024-09-17T07:22:08.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:08 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:22:08.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:08 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:22:08.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:08 smithi100 ceph-mon[29092]: pgmap v935: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 5 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 146 active+clean; 8.6 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 719 B/s rd, 0 B/s wr, 850 op/s; 87032/424938 objects degraded (20.481%); 21 MiB/s, 335 objects/s recovering 2024-09-17T07:22:08.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:08 smithi100 ceph-mon[29092]: osdmap e431: 8 total, 8 up, 6 in 2024-09-17T07:22:08.694 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:08 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:22:08.694 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:08 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:22:08.694 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:08 smithi135 ceph-mon[33238]: pgmap v935: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 5 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 146 active+clean; 8.6 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 719 B/s rd, 0 B/s wr, 850 op/s; 87032/424938 objects degraded (20.481%); 21 MiB/s, 335 objects/s recovering 2024-09-17T07:22:08.694 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:08 smithi135 ceph-mon[33238]: osdmap e431: 8 total, 8 up, 6 in 2024-09-17T07:22:08.724 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:22:08.766 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:08.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:08.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:22:08.868 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:22:08.898 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:08.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:08.899 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:22:08.999 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:22:09.044 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:09.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:09.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:22:09.146 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:22:09.181 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:09.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:09.182 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:22:09.245 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:22:09.283 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:22:09.336 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:09.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:09.341 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:22:09.442 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:22:09.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:09.484 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:09.485 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:22:09.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:09 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:22:09.544 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:22:09 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:22:09.585 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:22:09.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:09.623 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:09.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:22:09.724 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:22:09.754 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:09.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:09.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:22:09.856 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:22:09.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:09 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:22:09.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:09 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:22:09.883 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:09.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:09.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:22:09.985 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:22:10.011 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:10.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:10.012 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:22:10.113 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:22:10.145 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:10.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:10.146 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:22:10.247 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:22:10.270 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:10.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:10.271 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:22:10.372 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:22:10.398 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:10.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:10.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:22:10.500 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:22:10.523 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:10.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:10.524 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:22:10.625 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:22:10.663 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:10.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:10.664 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:22:10.696 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:10 smithi100 ceph-mon[29092]: pgmap v937: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 5 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 146 active+clean; 8.4 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.59k op/s; 82689/411138 objects degraded (20.112%); 49 MiB/s, 785 objects/s recovering 2024-09-17T07:22:10.696 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:10 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 82689/411138 objects degraded (20.112%), 12 pgs degraded (PG_DEGRADED) 2024-09-17T07:22:10.696 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:10 smithi100 ceph-mon[29092]: osdmap e432: 8 total, 8 up, 6 in 2024-09-17T07:22:10.696 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:10 smithi100 ceph-mon[39462]: pgmap v937: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 5 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 146 active+clean; 8.4 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.59k op/s; 82689/411138 objects degraded (20.112%); 49 MiB/s, 785 objects/s recovering 2024-09-17T07:22:10.697 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:10 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 82689/411138 objects degraded (20.112%), 12 pgs degraded (PG_DEGRADED) 2024-09-17T07:22:10.697 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:10 smithi100 ceph-mon[39462]: osdmap e432: 8 total, 8 up, 6 in 2024-09-17T07:22:10.765 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:22:10.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:10.792 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:10.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:22:10.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:10 smithi135 ceph-mon[33238]: pgmap v937: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 5 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 146 active+clean; 8.4 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.59k op/s; 82689/411138 objects degraded (20.112%); 49 MiB/s, 785 objects/s recovering 2024-09-17T07:22:10.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:10 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 82689/411138 objects degraded (20.112%), 12 pgs degraded (PG_DEGRADED) 2024-09-17T07:22:10.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:10 smithi135 ceph-mon[33238]: osdmap e432: 8 total, 8 up, 6 in 2024-09-17T07:22:10.893 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:22:10.934 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:10.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:10.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:22:11.036 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:22:11.064 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:11.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:11.066 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:22:11.168 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:22:11.189 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:11.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:11.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:22:11.291 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:22:11.332 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:11.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:11.333 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:22:11.434 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:22:11.484 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:11.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:11.485 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:22:11.497 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:22:11.586 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:22:11.618 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:11.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:11.619 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:22:11.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:11 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:22:11.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:11 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:22:11.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:11 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:22:11.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:11 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:22:11.719 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:22:11.748 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:11.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:11.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:22:11.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:11 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:22:11.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:11 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:22:11.851 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:22:11.883 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:11.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:11.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:22:11.930 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:11 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mon-a[29088]: 2024-09-17T07:22:11.809+0000 7f59b37e7640 -1 mon.a@0(leader).osd e432 definitely_dead 0 2024-09-17T07:22:11.985 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:22:12.016 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:12.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:12.017 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:22:12.118 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:22:12.153 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:12.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:12.154 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:22:12.255 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:22:12.293 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:12.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:12.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:22:12.395 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:22:12.423 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:12.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:12.424 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:22:12.525 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:22:12.538 INFO:teuthology.orchestra.run.smithi100.stderr:marked in osd.7. 2024-09-17T07:22:12.538 INFO:teuthology.orchestra.run.smithi100.stderr:nodeep-scrub is unset 2024-09-17T07:22:12.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:12.569 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:12.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:22:12.669 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:22:12.693 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:12.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:12.695 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:22:12.737 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:12 smithi100 ceph-mon[29092]: 4.10 scrub starts 2024-09-17T07:22:12.737 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:12 smithi100 ceph-mon[29092]: pgmap v939: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 5 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 146 active+clean; 8.2 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 838 B/s rd, 0 B/s wr, 1.11k op/s; 80512/404592 objects degraded (19.900%); 24 MiB/s, 381 objects/s recovering 2024-09-17T07:22:12.738 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:12 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1657025877' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:22:12.738 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:12 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:22:12.738 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:12 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/3960828567' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-09-17T07:22:12.738 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:12 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-09-17T07:22:12.738 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:12 smithi100 ceph-mon[39462]: 4.10 scrub starts 2024-09-17T07:22:12.738 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:12 smithi100 ceph-mon[39462]: pgmap v939: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 5 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 146 active+clean; 8.2 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 838 B/s rd, 0 B/s wr, 1.11k op/s; 80512/404592 objects degraded (19.900%); 24 MiB/s, 381 objects/s recovering 2024-09-17T07:22:12.738 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:12 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1657025877' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:22:12.738 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:12 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:22:12.738 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:12 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/3960828567' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-09-17T07:22:12.738 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:12 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-09-17T07:22:12.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:12 smithi135 ceph-mon[33238]: 4.10 scrub starts 2024-09-17T07:22:12.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:12 smithi135 ceph-mon[33238]: pgmap v939: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 5 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 146 active+clean; 8.2 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 838 B/s rd, 0 B/s wr, 1.11k op/s; 80512/404592 objects degraded (19.900%); 24 MiB/s, 381 objects/s recovering 2024-09-17T07:22:12.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:12 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1657025877' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:22:12.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:12 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:22:12.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:12 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/3960828567' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-09-17T07:22:12.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:12 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-09-17T07:22:12.796 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:22:12.821 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:12.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:12.822 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:22:12.922 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:22:12.947 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:12.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:12.948 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:22:13.049 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:22:13.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:13.074 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:13.075 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:22:13.175 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:22:13.201 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:13.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:13.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:22:13.236 INFO:teuthology.orchestra.run.smithi135.stdout:{ 2024-09-17T07:22:13.236 INFO:teuthology.orchestra.run.smithi135.stdout: "ops": [], 2024-09-17T07:22:13.236 INFO:teuthology.orchestra.run.smithi135.stdout: "complaint_time": 30, 2024-09-17T07:22:13.236 INFO:teuthology.orchestra.run.smithi135.stdout: "num_blocked_ops": 0 2024-09-17T07:22:13.237 INFO:teuthology.orchestra.run.smithi135.stdout:} 2024-09-17T07:22:13.302 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:22:13.332 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:13.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:13.333 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:22:13.434 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:22:13.459 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:13.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:13.460 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:22:13.561 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:22:13.603 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:13.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:13.604 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:22:13.705 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:22:13.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:13.731 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:13.731 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:22:13.744 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:13 smithi100 ceph-mon[29092]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T07:22:13.744 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:13 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T07:22:13.744 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:13 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-09-17T07:22:13.744 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:13 smithi100 ceph-mon[29092]: osdmap e433: 8 total, 8 up, 7 in 2024-09-17T07:22:13.744 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:13 smithi100 ceph-mon[39462]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T07:22:13.744 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:13 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T07:22:13.744 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:13 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-09-17T07:22:13.744 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:13 smithi100 ceph-mon[39462]: osdmap e433: 8 total, 8 up, 7 in 2024-09-17T07:22:13.832 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:22:13.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:13.862 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:13.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:22:13.862 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:13 smithi135 ceph-mon[33238]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T07:22:13.862 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:13 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T07:22:13.863 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:13 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-09-17T07:22:13.863 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:13 smithi135 ceph-mon[33238]: osdmap e433: 8 total, 8 up, 7 in 2024-09-17T07:22:13.963 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:22:13.989 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:13.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:13.990 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:22:14.091 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:22:14.113 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:14.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:14.114 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:22:14.215 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:22:14.219 INFO:tasks.thrashosds.thrasher:Added osd 7 2024-09-17T07:22:14.241 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:14.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:14.245 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:22:14.345 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:22:14.371 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:14.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:14.372 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:22:14.473 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:22:14.501 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:14.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:14.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:22:14.604 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:22:14.638 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:14.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:14.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:22:14.739 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:22:14.762 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:14 smithi100 ceph-mon[29092]: pgmap v941: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 remapped+peering, 4 unknown, 5 active+undersized+degraded+remapped+backfill_wait, 31 peering, 3 active+undersized+degraded+remapped+backfilling, 111 active+clean; 7.7 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 71508/376923 objects degraded (18.972%); 24 MiB/s, 386 objects/s recovering 2024-09-17T07:22:14.762 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:14 smithi100 ceph-mon[29092]: Health check failed: Reduced data availability: 10 pgs inactive, 18 pgs peering (PG_AVAILABILITY) 2024-09-17T07:22:14.762 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:14 smithi100 ceph-mon[29092]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-17T07:22:14.763 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:14 smithi100 ceph-mon[29092]: 11.e continuing backfill to osd.1 from (420'9675,432'13661] MIN to 432'13661 2024-09-17T07:22:14.763 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:14 smithi100 ceph-mon[29092]: 11.e starting backfill to osd.7 from (0'0,0'0] MAX to 432'13661 2024-09-17T07:22:14.763 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:14 smithi100 ceph-mon[29092]: osdmap e434: 8 total, 8 up, 7 in 2024-09-17T07:22:14.763 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:14 smithi100 ceph-mon[29092]: 11.0 continuing backfill to osd.1 from (419'9500,432'13546] MIN to 432'13546 2024-09-17T07:22:14.763 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:14 smithi100 ceph-mon[29092]: 11.3 continuing backfill to osd.1 from (421'9700,432'13450] MIN to 432'13450 2024-09-17T07:22:14.763 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:14 smithi100 ceph-mon[29092]: 11.0 starting backfill to osd.7 from (0'0,0'0] MAX to 432'13546 2024-09-17T07:22:14.763 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:14 smithi100 ceph-mon[29092]: 11.3 starting backfill to osd.7 from (0'0,0'0] MAX to 432'13450 2024-09-17T07:22:14.763 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:14 smithi100 ceph-mon[29092]: 11.f continuing backfill to osd.1 from (419'9500,432'13453] MIN to 432'13453 2024-09-17T07:22:14.763 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:14 smithi100 ceph-mon[29092]: 11.f starting backfill to osd.7 from (0'0,0'0] MAX to 432'13453 2024-09-17T07:22:14.763 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:14 smithi100 ceph-mon[29092]: 11.2 starting backfill to osd.7 from (0'0,0'0] MAX to 432'13458 2024-09-17T07:22:14.763 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:14 smithi100 ceph-mon[29092]: 11.4 starting backfill to osd.7 from (0'0,0'0] MAX to 432'13398 2024-09-17T07:22:14.763 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:14 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:22:14.764 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:14 smithi100 ceph-mon[29092]: 6.d scrub starts 2024-09-17T07:22:14.764 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:14 smithi100 ceph-mon[39462]: pgmap v941: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 remapped+peering, 4 unknown, 5 active+undersized+degraded+remapped+backfill_wait, 31 peering, 3 active+undersized+degraded+remapped+backfilling, 111 active+clean; 7.7 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 71508/376923 objects degraded (18.972%); 24 MiB/s, 386 objects/s recovering 2024-09-17T07:22:14.764 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:14 smithi100 ceph-mon[39462]: Health check failed: Reduced data availability: 10 pgs inactive, 18 pgs peering (PG_AVAILABILITY) 2024-09-17T07:22:14.764 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:14 smithi100 ceph-mon[39462]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-17T07:22:14.764 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:14 smithi100 ceph-mon[39462]: 11.e continuing backfill to osd.1 from (420'9675,432'13661] MIN to 432'13661 2024-09-17T07:22:14.764 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:14 smithi100 ceph-mon[39462]: 11.e starting backfill to osd.7 from (0'0,0'0] MAX to 432'13661 2024-09-17T07:22:14.764 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:14 smithi100 ceph-mon[39462]: osdmap e434: 8 total, 8 up, 7 in 2024-09-17T07:22:14.764 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:14 smithi100 ceph-mon[39462]: 11.0 continuing backfill to osd.1 from (419'9500,432'13546] MIN to 432'13546 2024-09-17T07:22:14.764 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:14 smithi100 ceph-mon[39462]: 11.3 continuing backfill to osd.1 from (421'9700,432'13450] MIN to 432'13450 2024-09-17T07:22:14.764 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:14 smithi100 ceph-mon[39462]: 11.0 starting backfill to osd.7 from (0'0,0'0] MAX to 432'13546 2024-09-17T07:22:14.765 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:14 smithi100 ceph-mon[39462]: 11.3 starting backfill to osd.7 from (0'0,0'0] MAX to 432'13450 2024-09-17T07:22:14.765 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:14 smithi100 ceph-mon[39462]: 11.f continuing backfill to osd.1 from (419'9500,432'13453] MIN to 432'13453 2024-09-17T07:22:14.765 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:14 smithi100 ceph-mon[39462]: 11.f starting backfill to osd.7 from (0'0,0'0] MAX to 432'13453 2024-09-17T07:22:14.765 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:14 smithi100 ceph-mon[39462]: 11.2 starting backfill to osd.7 from (0'0,0'0] MAX to 432'13458 2024-09-17T07:22:14.765 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:14 smithi100 ceph-mon[39462]: 11.4 starting backfill to osd.7 from (0'0,0'0] MAX to 432'13398 2024-09-17T07:22:14.765 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:14 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:22:14.765 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:14 smithi100 ceph-mon[39462]: 6.d scrub starts 2024-09-17T07:22:14.774 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:14.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:14.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:22:14.776 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-17T07:22:14.816 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:14 smithi135 ceph-mon[33238]: pgmap v941: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 remapped+peering, 4 unknown, 5 active+undersized+degraded+remapped+backfill_wait, 31 peering, 3 active+undersized+degraded+remapped+backfilling, 111 active+clean; 7.7 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 71508/376923 objects degraded (18.972%); 24 MiB/s, 386 objects/s recovering 2024-09-17T07:22:14.816 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:14 smithi135 ceph-mon[33238]: Health check failed: Reduced data availability: 10 pgs inactive, 18 pgs peering (PG_AVAILABILITY) 2024-09-17T07:22:14.817 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:14 smithi135 ceph-mon[33238]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-17T07:22:14.817 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:14 smithi135 ceph-mon[33238]: 11.e continuing backfill to osd.1 from (420'9675,432'13661] MIN to 432'13661 2024-09-17T07:22:14.817 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:14 smithi135 ceph-mon[33238]: 11.e starting backfill to osd.7 from (0'0,0'0] MAX to 432'13661 2024-09-17T07:22:14.817 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:14 smithi135 ceph-mon[33238]: osdmap e434: 8 total, 8 up, 7 in 2024-09-17T07:22:14.817 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:14 smithi135 ceph-mon[33238]: 11.0 continuing backfill to osd.1 from (419'9500,432'13546] MIN to 432'13546 2024-09-17T07:22:14.817 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:14 smithi135 ceph-mon[33238]: 11.3 continuing backfill to osd.1 from (421'9700,432'13450] MIN to 432'13450 2024-09-17T07:22:14.817 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:14 smithi135 ceph-mon[33238]: 11.0 starting backfill to osd.7 from (0'0,0'0] MAX to 432'13546 2024-09-17T07:22:14.817 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:14 smithi135 ceph-mon[33238]: 11.3 starting backfill to osd.7 from (0'0,0'0] MAX to 432'13450 2024-09-17T07:22:14.817 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:14 smithi135 ceph-mon[33238]: 11.f continuing backfill to osd.1 from (419'9500,432'13453] MIN to 432'13453 2024-09-17T07:22:14.817 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:14 smithi135 ceph-mon[33238]: 11.f starting backfill to osd.7 from (0'0,0'0] MAX to 432'13453 2024-09-17T07:22:14.817 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:14 smithi135 ceph-mon[33238]: 11.2 starting backfill to osd.7 from (0'0,0'0] MAX to 432'13458 2024-09-17T07:22:14.817 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:14 smithi135 ceph-mon[33238]: 11.4 starting backfill to osd.7 from (0'0,0'0] MAX to 432'13398 2024-09-17T07:22:14.817 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:14 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:22:14.818 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:14 smithi135 ceph-mon[33238]: 6.d scrub starts 2024-09-17T07:22:14.821 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:22:14.821 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:22:14.851 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1: {} 2024-09-17T07:22:14.851 INFO:teuthology.orchestra.run.smithi100.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:22:14.876 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:22:14.877 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:22:14.877 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:22:14.903 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:22:14.903 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:22:14.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:14.923 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:14.924 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:22:14.941 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:22:14.941 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:22:14.970 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:22:14.970 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:22:15.007 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:22:15.007 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:22:15.025 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:22:15.038 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:22:15.038 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:22:15.064 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:15.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:15.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:22:15.154 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:22:15.166 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:22:15.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:15.206 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:15.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:22:15.307 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:22:15.346 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:15.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:15.348 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:22:15.448 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:22:15.487 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:15.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:15.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:22:15.588 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:22:15.626 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:15.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:15.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:22:15.728 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:22:15.769 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:15.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:15.770 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:22:15.777 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-17T07:22:15.812 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:15 smithi100 ceph-mon[39462]: 2.c scrub starts 2024-09-17T07:22:15.812 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:15 smithi100 ceph-mon[39462]: 11.5 starting backfill to osd.7 from (0'0,0'0] MAX to 432'12912 2024-09-17T07:22:15.812 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:15 smithi100 ceph-mon[39462]: osdmap e435: 8 total, 8 up, 7 in 2024-09-17T07:22:15.812 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:15 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1702260547' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:22:15.812 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:15 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:22:15.812 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:15 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:22:15.812 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:15 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:22:15.812 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:15 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 71508/376923 objects degraded (18.972%), 11 pgs degraded (PG_DEGRADED) 2024-09-17T07:22:15.812 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:15 smithi100 ceph-mon[39462]: osdmap e436: 8 total, 8 up, 7 in 2024-09-17T07:22:15.813 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:15 smithi100 ceph-mon[39462]: 6.e scrub starts 2024-09-17T07:22:15.813 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:15 smithi100 ceph-mon[29092]: 2.c scrub starts 2024-09-17T07:22:15.813 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:15 smithi100 ceph-mon[29092]: 11.5 starting backfill to osd.7 from (0'0,0'0] MAX to 432'12912 2024-09-17T07:22:15.813 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:15 smithi100 ceph-mon[29092]: osdmap e435: 8 total, 8 up, 7 in 2024-09-17T07:22:15.813 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:15 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1702260547' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:22:15.813 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:15 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:22:15.813 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:15 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:22:15.813 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:15 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:22:15.814 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:15 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 71508/376923 objects degraded (18.972%), 11 pgs degraded (PG_DEGRADED) 2024-09-17T07:22:15.814 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:15 smithi100 ceph-mon[29092]: osdmap e436: 8 total, 8 up, 7 in 2024-09-17T07:22:15.814 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:15 smithi100 ceph-mon[29092]: 6.e scrub starts 2024-09-17T07:22:15.871 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:22:15.911 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:15.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:15.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:22:16.013 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:22:16.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:15 smithi135 ceph-mon[33238]: 2.c scrub starts 2024-09-17T07:22:16.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:15 smithi135 ceph-mon[33238]: 11.5 starting backfill to osd.7 from (0'0,0'0] MAX to 432'12912 2024-09-17T07:22:16.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:15 smithi135 ceph-mon[33238]: osdmap e435: 8 total, 8 up, 7 in 2024-09-17T07:22:16.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:15 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1702260547' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:22:16.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:15 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:22:16.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:15 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:22:16.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:15 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:22:16.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:15 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 71508/376923 objects degraded (18.972%), 11 pgs degraded (PG_DEGRADED) 2024-09-17T07:22:16.045 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:15 smithi135 ceph-mon[33238]: osdmap e436: 8 total, 8 up, 7 in 2024-09-17T07:22:16.045 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:15 smithi135 ceph-mon[33238]: 6.e scrub starts 2024-09-17T07:22:16.052 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:16.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:16.053 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:22:16.106 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:22:16.129 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:22:15 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:22:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:22:16.153 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:22:16.198 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:16.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:16.200 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:22:16.300 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:22:16.334 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:16.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:16.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:22:16.436 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:22:16.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:16.473 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:16.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:22:16.574 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:22:16.605 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:16.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:16.606 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:22:16.707 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:22:16.744 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:16.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:16.746 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:22:16.846 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:22:16.882 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:16.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:16.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:22:16.984 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:22:17.028 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:17.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:17.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:22:17.131 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:22:17.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:17.170 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:17.170 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:22:17.271 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:22:17.303 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:17.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:17.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:22:17.330 INFO:teuthology.orchestra.run.smithi135.stdout:ERROR: (22) Invalid argument 2024-09-17T07:22:17.331 INFO:teuthology.orchestra.run.smithi135.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:22:17.405 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:22:17.441 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:17.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:17.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:22:17.543 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:22:17.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:17 smithi135 ceph-mon[33238]: pgmap v945: 158 pgs: 1 active+remapped+backfill_wait, 2 active+undersized+remapped, 2 activating+degraded, 5 active+undersized+degraded+remapped+backfill_toofull, 3 remapped+peering, 1 unknown, 1 active+undersized+degraded+remapped+backfill_wait, 7 activating, 32 peering, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 8.0 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.11k op/s; 71896/391329 objects degraded (18.372%); 32224/391329 objects misplaced (8.235%); 68 MiB/s, 1.09k objects/s recovering 2024-09-17T07:22:17.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:17 smithi135 ceph-mon[33238]: 5.b scrub starts 2024-09-17T07:22:17.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:17 smithi135 ceph-mon[33238]: osdmap e437: 8 total, 8 up, 7 in 2024-09-17T07:22:17.580 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:17.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:17.581 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:22:17.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:17 smithi100 ceph-mon[29092]: pgmap v945: 158 pgs: 1 active+remapped+backfill_wait, 2 active+undersized+remapped, 2 activating+degraded, 5 active+undersized+degraded+remapped+backfill_toofull, 3 remapped+peering, 1 unknown, 1 active+undersized+degraded+remapped+backfill_wait, 7 activating, 32 peering, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 8.0 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.11k op/s; 71896/391329 objects degraded (18.372%); 32224/391329 objects misplaced (8.235%); 68 MiB/s, 1.09k objects/s recovering 2024-09-17T07:22:17.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:17 smithi100 ceph-mon[29092]: 5.b scrub starts 2024-09-17T07:22:17.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:17 smithi100 ceph-mon[29092]: osdmap e437: 8 total, 8 up, 7 in 2024-09-17T07:22:17.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:17 smithi100 ceph-mon[39462]: pgmap v945: 158 pgs: 1 active+remapped+backfill_wait, 2 active+undersized+remapped, 2 activating+degraded, 5 active+undersized+degraded+remapped+backfill_toofull, 3 remapped+peering, 1 unknown, 1 active+undersized+degraded+remapped+backfill_wait, 7 activating, 32 peering, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 8.0 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.11k op/s; 71896/391329 objects degraded (18.372%); 32224/391329 objects misplaced (8.235%); 68 MiB/s, 1.09k objects/s recovering 2024-09-17T07:22:17.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:17 smithi100 ceph-mon[39462]: 5.b scrub starts 2024-09-17T07:22:17.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:17 smithi100 ceph-mon[39462]: osdmap e437: 8 total, 8 up, 7 in 2024-09-17T07:22:17.682 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:22:17.718 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:17.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:17.719 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:22:17.820 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:22:17.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:17.859 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:17.859 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:22:17.960 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:22:17.965 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:22:18.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:18.011 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:18.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:22:18.043 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-17T07:22:18.112 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:22:18.147 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:18.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:18.148 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:22:18.249 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:22:18.290 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:18.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:18.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:22:18.393 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:22:18.400 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:22:18.436 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:18.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:18.437 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:22:18.525 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:18 smithi100 ceph-mon[29092]: 4.1b scrub starts 2024-09-17T07:22:18.525 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:18 smithi100 ceph-mon[39462]: 4.1b scrub starts 2024-09-17T07:22:18.538 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:22:18.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:18 smithi135 ceph-mon[33238]: 4.1b scrub starts 2024-09-17T07:22:18.601 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:18.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:18.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:22:18.614 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:22:18.704 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:22:18.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:18.748 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:18.748 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:22:18.849 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:22:18.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:18.887 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:18.887 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:22:18.988 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:22:19.027 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:19.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:19.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:22:19.128 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:22:19.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:19.167 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:19.167 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:22:19.219 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7] out_osds: [3] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 6, 7, 1] 2024-09-17T07:22:19.220 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T07:22:19.220 INFO:tasks.thrashosds.thrasher:Shrinking pool unique_pool_4 2024-09-17T07:22:19.220 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool unique_pool_4 by 10 min 8 2024-09-17T07:22:19.220 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T07:22:19.268 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:22:19.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:19.310 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:19.310 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:22:19.411 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:22:19.449 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:19.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:19.450 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:22:19.504 INFO:teuthology.orchestra.run.smithi100.stderr:noscrub is set 2024-09-17T07:22:19.505 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:19 smithi100 ceph-mon[29092]: pgmap v947: 158 pgs: 1 active+remapped+backfill_wait, 2 active+undersized+remapped, 2 activating+degraded, 5 active+undersized+degraded+remapped+backfill_toofull, 3 remapped+peering, 1 unknown, 1 active+undersized+degraded+remapped+backfill_wait, 7 activating, 32 peering, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 8.0 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 15 KiB/s rd, 163 MiB/s wr, 4.05k op/s; 71896/391329 objects degraded (18.372%); 32224/391329 objects misplaced (8.235%); 68 MiB/s, 1.09k objects/s recovering 2024-09-17T07:22:19.505 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:19 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:22:19.505 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:19 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:22:19.505 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:19 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/4045375426' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:22:19.505 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:19 smithi100 ceph-mon[39462]: pgmap v947: 158 pgs: 1 active+remapped+backfill_wait, 2 active+undersized+remapped, 2 activating+degraded, 5 active+undersized+degraded+remapped+backfill_toofull, 3 remapped+peering, 1 unknown, 1 active+undersized+degraded+remapped+backfill_wait, 7 activating, 32 peering, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 8.0 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 15 KiB/s rd, 163 MiB/s wr, 4.05k op/s; 71896/391329 objects degraded (18.372%); 32224/391329 objects misplaced (8.235%); 68 MiB/s, 1.09k objects/s recovering 2024-09-17T07:22:19.506 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:19 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:22:19.506 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:19 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:22:19.506 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:19 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/4045375426' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:22:19.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:19 smithi135 ceph-mon[33238]: pgmap v947: 158 pgs: 1 active+remapped+backfill_wait, 2 active+undersized+remapped, 2 activating+degraded, 5 active+undersized+degraded+remapped+backfill_toofull, 3 remapped+peering, 1 unknown, 1 active+undersized+degraded+remapped+backfill_wait, 7 activating, 32 peering, 4 active+undersized+degraded+remapped+backfilling, 100 active+clean; 8.0 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 15 KiB/s rd, 163 MiB/s wr, 4.05k op/s; 71896/391329 objects degraded (18.372%); 32224/391329 objects misplaced (8.235%); 68 MiB/s, 1.09k objects/s recovering 2024-09-17T07:22:19.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:19 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:22:19.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:19 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:22:19.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:19 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/4045375426' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:22:19.544 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:22:19 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:22:19.551 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:22:19.588 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:19.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:19.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:22:19.690 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:22:19.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:19.724 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:19.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:22:19.825 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:22:19.863 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:19.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:19.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:22:19.965 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:22:20.001 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:20.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:20.002 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:22:20.103 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:22:20.131 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:20.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:20.132 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:22:20.233 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:22:20.273 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:20.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:20.274 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:22:20.375 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:22:20.399 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:20.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:20.400 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:22:20.502 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:22:20.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:20.549 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:20.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:22:20.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:20 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:22:20.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:20 smithi100 ceph-mon[29092]: 6.7 scrub starts 2024-09-17T07:22:20.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:20 smithi100 ceph-mon[29092]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:22:20.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:20 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/4045375426' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T07:22:20.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:20 smithi100 ceph-mon[29092]: osdmap e438: 8 total, 8 up, 7 in 2024-09-17T07:22:20.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:20 smithi100 ceph-mon[29092]: Health check update: Reduced data availability: 17 pgs inactive, 18 pgs peering (PG_AVAILABILITY) 2024-09-17T07:22:20.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:20 smithi100 ceph-mon[29092]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-17T07:22:20.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:20 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 71896/391329 objects degraded (18.372%), 12 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-17T07:22:20.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:20 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:22:20.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:20 smithi100 ceph-mon[39462]: 6.7 scrub starts 2024-09-17T07:22:20.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:20 smithi100 ceph-mon[39462]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:22:20.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:20 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/4045375426' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T07:22:20.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:20 smithi100 ceph-mon[39462]: osdmap e438: 8 total, 8 up, 7 in 2024-09-17T07:22:20.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:20 smithi100 ceph-mon[39462]: Health check update: Reduced data availability: 17 pgs inactive, 18 pgs peering (PG_AVAILABILITY) 2024-09-17T07:22:20.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:20 smithi100 ceph-mon[39462]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-17T07:22:20.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:20 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 71896/391329 objects degraded (18.372%), 12 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-17T07:22:20.650 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:22:20.687 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:20.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:20.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:22:20.789 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:22:20.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:20 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:22:20.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:20 smithi135 ceph-mon[33238]: 6.7 scrub starts 2024-09-17T07:22:20.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:20 smithi135 ceph-mon[33238]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:22:20.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:20 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/4045375426' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T07:22:20.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:20 smithi135 ceph-mon[33238]: osdmap e438: 8 total, 8 up, 7 in 2024-09-17T07:22:20.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:20 smithi135 ceph-mon[33238]: Health check update: Reduced data availability: 17 pgs inactive, 18 pgs peering (PG_AVAILABILITY) 2024-09-17T07:22:20.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:20 smithi135 ceph-mon[33238]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-17T07:22:20.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:20 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 71896/391329 objects degraded (18.372%), 12 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-17T07:22:20.814 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:20.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:20.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:22:20.916 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:22:20.943 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:20.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:20.944 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:22:21.045 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:22:21.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:21.077 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:21.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:22:21.178 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:22:21.214 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:21.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:21.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:22:21.316 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:22:21.348 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:21.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:21.349 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:22:21.450 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:22:21.496 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:21.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:21.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:22:21.598 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:22:21.621 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:21 smithi100 ceph-mon[29092]: pgmap v948: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 5 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 146 active+clean; 7.8 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 16 KiB/s rd, 111 MiB/s wr, 3.30k op/s; 77068/381663 objects degraded (20.193%); 39302/381663 objects misplaced (10.298%); 63 MiB/s, 1.01k objects/s recovering 2024-09-17T07:22:21.621 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:21 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:22:21.621 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:21 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:22:21.621 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:21 smithi100 ceph-mon[29092]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 17 pgs inactive, 18 pgs peering) 2024-09-17T07:22:21.621 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:21 smithi100 ceph-mon[29092]: osdmap e439: 8 total, 8 up, 7 in 2024-09-17T07:22:21.621 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:21 smithi100 ceph-mon[39462]: pgmap v948: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 5 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 146 active+clean; 7.8 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 16 KiB/s rd, 111 MiB/s wr, 3.30k op/s; 77068/381663 objects degraded (20.193%); 39302/381663 objects misplaced (10.298%); 63 MiB/s, 1.01k objects/s recovering 2024-09-17T07:22:21.621 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:21 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:22:21.622 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:21 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:22:21.622 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:21 smithi100 ceph-mon[39462]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 17 pgs inactive, 18 pgs peering) 2024-09-17T07:22:21.622 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:21 smithi100 ceph-mon[39462]: osdmap e439: 8 total, 8 up, 7 in 2024-09-17T07:22:21.630 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:21.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:21.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:22:21.732 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:22:21.758 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:21.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:21.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:22:21.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:21 smithi135 ceph-mon[33238]: pgmap v948: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 5 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfilling, 146 active+clean; 7.8 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 16 KiB/s rd, 111 MiB/s wr, 3.30k op/s; 77068/381663 objects degraded (20.193%); 39302/381663 objects misplaced (10.298%); 63 MiB/s, 1.01k objects/s recovering 2024-09-17T07:22:21.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:21 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:22:21.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:21 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:22:21.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:21 smithi135 ceph-mon[33238]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 17 pgs inactive, 18 pgs peering) 2024-09-17T07:22:21.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:21 smithi135 ceph-mon[33238]: osdmap e439: 8 total, 8 up, 7 in 2024-09-17T07:22:21.860 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:22:21.889 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:21.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:21.890 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:22:21.991 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:22:22.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:22.023 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:22.024 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:22:22.124 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:22:22.161 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:22.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:22.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:22:22.263 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:22:22.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:22.299 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:22.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:22:22.400 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:22:22.425 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:22.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:22.426 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:22:22.526 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:22:22.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:22.553 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:22.553 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:22:22.654 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:22:22.680 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:22.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:22.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:22:22.781 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:22:22.818 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:22.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:22.819 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:22:22.920 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:22:22.946 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:22.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:22.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:22:23.048 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:22:23.092 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:23.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:23.093 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:22:23.194 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:22:23.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:23.228 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:23.228 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:22:23.330 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:22:23.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:23.365 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:23.365 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:22:23.466 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:22:23.503 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:23.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:23.504 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:22:23.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:23 smithi135 ceph-mon[33238]: pgmap v951: 158 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 5 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 146 active+clean; 7.6 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 8.0 KiB/s rd, 36 KiB/s wr, 920 op/s; 74084/374949 objects degraded (19.758%); 37684/374949 objects misplaced (10.050%); 29 MiB/s, 475 objects/s recovering 2024-09-17T07:22:23.605 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:22:23.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:23 smithi100 ceph-mon[29092]: pgmap v951: 158 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 5 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 146 active+clean; 7.6 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 8.0 KiB/s rd, 36 KiB/s wr, 920 op/s; 74084/374949 objects degraded (19.758%); 37684/374949 objects misplaced (10.050%); 29 MiB/s, 475 objects/s recovering 2024-09-17T07:22:23.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:23 smithi100 ceph-mon[39462]: pgmap v951: 158 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 5 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 146 active+clean; 7.6 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 8.0 KiB/s rd, 36 KiB/s wr, 920 op/s; 74084/374949 objects degraded (19.758%); 37684/374949 objects misplaced (10.050%); 29 MiB/s, 475 objects/s recovering 2024-09-17T07:22:23.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:23.628 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:23.628 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:22:23.729 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:22:23.770 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:23.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:23.771 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:22:23.872 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:22:23.910 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:23.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:23.911 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:22:24.012 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:22:24.047 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:24.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:24.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:22:24.149 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:22:24.171 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:24.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:24.172 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:22:24.273 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:22:24.333 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:24.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:24.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:22:24.435 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:22:24.474 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:24.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:24.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:22:24.575 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:22:24.611 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:24.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:24.613 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:22:24.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:24 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:22:24.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:24 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:22:24.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:24 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:22:24.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:24 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:22:24.713 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:22:24.745 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:24.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:24.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:22:24.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:24 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:22:24.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:24 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:22:24.847 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:22:24.882 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:24.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:24.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:22:24.984 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:22:25.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:25.012 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:25.012 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:22:25.113 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:22:25.141 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:25.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:25.142 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:22:25.242 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:22:25.277 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:25.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:25.278 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:22:25.379 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:22:25.384 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:25 smithi100 ceph-mon[29092]: pgmap v952: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 5 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 146 active+clean; 7.6 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 29 KiB/s rd, 31 KiB/s wr, 949 op/s; 73437/372060 objects degraded (19.738%); 37361/372060 objects misplaced (10.042%); 25 MiB/s, 401 objects/s recovering 2024-09-17T07:22:25.385 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:25 smithi100 ceph-mon[29092]: osdmap e440: 8 total, 8 up, 7 in 2024-09-17T07:22:25.385 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:25 smithi100 ceph-mon[29092]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-17T07:22:25.385 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:25 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 73437/372060 objects degraded (19.738%), 11 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-17T07:22:25.413 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:25.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:25.414 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:22:25.516 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:22:25.553 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:25.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:25.554 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:22:25.655 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:22:25.691 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:25.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:25.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:22:25.708 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:25 smithi135 ceph-mon[33238]: pgmap v952: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 5 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 146 active+clean; 7.6 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 29 KiB/s rd, 31 KiB/s wr, 949 op/s; 73437/372060 objects degraded (19.738%); 37361/372060 objects misplaced (10.042%); 25 MiB/s, 401 objects/s recovering 2024-09-17T07:22:25.709 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:25 smithi135 ceph-mon[33238]: osdmap e440: 8 total, 8 up, 7 in 2024-09-17T07:22:25.709 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:25 smithi135 ceph-mon[33238]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-17T07:22:25.709 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:25 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 73437/372060 objects degraded (19.738%), 11 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-17T07:22:25.793 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:22:25.833 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:25.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:25.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:22:25.876 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:25 smithi100 ceph-mon[39462]: pgmap v952: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 5 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 146 active+clean; 7.6 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 29 KiB/s rd, 31 KiB/s wr, 949 op/s; 73437/372060 objects degraded (19.738%); 37361/372060 objects misplaced (10.042%); 25 MiB/s, 401 objects/s recovering 2024-09-17T07:22:25.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:25 smithi100 ceph-mon[39462]: osdmap e440: 8 total, 8 up, 7 in 2024-09-17T07:22:25.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:25 smithi100 ceph-mon[39462]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-17T07:22:25.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:25 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 73437/372060 objects degraded (19.738%), 11 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-17T07:22:25.935 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:22:25.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:25.974 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:25.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:22:26.075 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:22:26.123 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:26.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:26.124 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:22:26.143 INFO:teuthology.orchestra.run.smithi135.stdout:ERROR: (22) Invalid argument 2024-09-17T07:22:26.143 INFO:teuthology.orchestra.run.smithi135.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:22:26.225 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:22:26.257 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:26.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:26.258 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:22:26.359 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:22:26.377 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:22:25 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:22:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:22:26.392 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:26.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:26.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:22:26.494 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:22:26.523 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:26.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:26.524 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:22:26.625 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:22:26.662 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:26.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:26.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:22:26.764 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:22:26.791 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:26.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:26.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:22:26.839 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-17T07:22:26.893 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:22:26.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:26.938 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:26.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:22:27.039 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:22:27.078 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:27.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:27.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:22:27.174 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:22:27.180 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:22:27.217 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:27.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:27.218 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:22:27.319 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:22:27.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:27.353 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:27.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:22:27.455 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:22:27.490 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:27.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:27.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:22:27.582 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:27 smithi135 ceph-mon[33238]: pgmap v954: 158 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 5 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 146 active+clean; 7.3 GiB data, 24 GiB used, 601 GiB / 626 GiB avail; 72 KiB/s rd, 0 B/s wr, 1.33k op/s; 68298/359826 objects degraded (18.981%); 35086/359826 objects misplaced (9.751%); 43 MiB/s, 690 objects/s recovering 2024-09-17T07:22:27.593 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:22:27.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:27.636 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:27.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:22:27.689 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:27 smithi100 ceph-mon[29092]: pgmap v954: 158 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 5 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 146 active+clean; 7.3 GiB data, 24 GiB used, 601 GiB / 626 GiB avail; 72 KiB/s rd, 0 B/s wr, 1.33k op/s; 68298/359826 objects degraded (18.981%); 35086/359826 objects misplaced (9.751%); 43 MiB/s, 690 objects/s recovering 2024-09-17T07:22:27.689 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:27 smithi100 ceph-mon[39462]: pgmap v954: 158 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 5 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 146 active+clean; 7.3 GiB data, 24 GiB used, 601 GiB / 626 GiB avail; 72 KiB/s rd, 0 B/s wr, 1.33k op/s; 68298/359826 objects degraded (18.981%); 35086/359826 objects misplaced (9.751%); 43 MiB/s, 690 objects/s recovering 2024-09-17T07:22:27.737 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:22:27.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:27.777 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:27.777 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:22:27.878 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:22:27.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:27.912 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:27.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:22:28.013 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:22:28.049 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:28.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:28.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:22:28.151 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:22:28.180 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:28.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:28.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:22:28.282 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:22:28.319 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:28.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:28.320 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:22:28.421 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:22:28.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:28.453 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:28.453 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:22:28.554 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:22:28.589 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:28.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:28.590 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:22:28.691 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:22:28.735 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:28.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:28.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:22:28.837 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:22:28.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:28.873 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:28.873 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:22:28.874 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:22:28.975 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:22:29.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:29.016 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:29.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:22:29.026 INFO:teuthology.orchestra.run.smithi135.stdout:ERROR: (22) Invalid argument 2024-09-17T07:22:29.026 INFO:teuthology.orchestra.run.smithi135.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:22:29.117 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:22:29.152 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:29.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:29.153 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:22:29.254 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:22:29.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:29.289 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:29.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:22:29.390 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:22:29.430 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:29.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:29.431 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:22:29.532 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:22:29.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:29.571 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:29.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:22:29.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:29 smithi100 ceph-mon[29092]: pgmap v955: 158 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 5 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 146 active+clean; 7.3 GiB data, 24 GiB used, 601 GiB / 626 GiB avail; 55 KiB/s rd, 0 B/s wr, 1.01k op/s; 68298/359826 objects degraded (18.981%); 35086/359826 objects misplaced (9.751%); 33 MiB/s, 521 objects/s recovering 2024-09-17T07:22:29.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:29 smithi100 ceph-mon[39462]: pgmap v955: 158 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 5 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 146 active+clean; 7.3 GiB data, 24 GiB used, 601 GiB / 626 GiB avail; 55 KiB/s rd, 0 B/s wr, 1.01k op/s; 68298/359826 objects degraded (18.981%); 35086/359826 objects misplaced (9.751%); 33 MiB/s, 521 objects/s recovering 2024-09-17T07:22:29.672 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:22:29.700 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:22:29 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:22:29.701 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:29 smithi135 ceph-mon[33238]: pgmap v955: 158 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 5 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 146 active+clean; 7.3 GiB data, 24 GiB used, 601 GiB / 626 GiB avail; 55 KiB/s rd, 0 B/s wr, 1.01k op/s; 68298/359826 objects degraded (18.981%); 35086/359826 objects misplaced (9.751%); 33 MiB/s, 521 objects/s recovering 2024-09-17T07:22:29.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:29.715 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:29.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:22:29.726 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-17T07:22:29.742 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-17T07:22:29.816 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:22:29.863 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:29.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:29.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:22:29.966 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:22:30.009 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:30.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:30.010 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:22:30.084 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:22:30.111 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:22:30.160 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:30.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:30.162 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:22:30.263 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:22:30.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:30.292 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:30.292 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:22:30.393 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:22:30.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:30.426 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:30.426 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:22:30.522 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:30 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:22:30.522 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:30 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 68298/359826 objects degraded (18.981%), 11 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-17T07:22:30.527 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:22:30.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:30.566 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:30.566 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:22:30.667 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:22:30.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:30.712 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:30.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:22:30.813 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:22:30.848 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:30 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:22:30.849 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:30 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 68298/359826 objects degraded (18.981%), 11 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-17T07:22:30.849 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:30 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:22:30.849 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:30 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 68298/359826 objects degraded (18.981%), 11 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-17T07:22:30.850 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:30.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:30.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:22:30.952 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:22:30.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:30.992 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:30.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:22:31.093 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:22:31.119 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:31.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:31.120 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:22:31.221 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:22:31.256 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:31.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:31.257 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:22:31.358 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:22:31.402 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:31.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:31.403 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:22:31.504 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:22:31.536 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:31.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:31.537 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:22:31.637 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:22:31.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:31.682 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:31.682 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:22:31.717 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:31 smithi100 ceph-mon[29092]: pgmap v956: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 5 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 146 active+clean; 7.1 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 47 KiB/s rd, 0 B/s wr, 1.02k op/s; 65193/349524 objects degraded (18.652%); 33887/349524 objects misplaced (9.695%); 31 MiB/s, 494 objects/s recovering 2024-09-17T07:22:31.717 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:31 smithi100 ceph-mon[39462]: pgmap v956: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 5 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 146 active+clean; 7.1 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 47 KiB/s rd, 0 B/s wr, 1.02k op/s; 65193/349524 objects degraded (18.652%); 33887/349524 objects misplaced (9.695%); 31 MiB/s, 494 objects/s recovering 2024-09-17T07:22:31.784 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:22:31.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:31 smithi135 ceph-mon[33238]: pgmap v956: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 5 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 146 active+clean; 7.1 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 47 KiB/s rd, 0 B/s wr, 1.02k op/s; 65193/349524 objects degraded (18.652%); 33887/349524 objects misplaced (9.695%); 31 MiB/s, 494 objects/s recovering 2024-09-17T07:22:31.811 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:31.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:31.812 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:22:31.913 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:22:31.943 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:31.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:31.945 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:22:32.045 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:22:32.077 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:32.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:32.078 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:22:32.179 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:22:32.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:32.215 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:32.216 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:22:32.316 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:22:32.351 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:32.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:32.353 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:22:32.409 INFO:teuthology.orchestra.run.smithi135.stdout:ERROR: (22) Invalid argument 2024-09-17T07:22:32.410 INFO:teuthology.orchestra.run.smithi135.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:22:32.453 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:22:32.489 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:32.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:32.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:22:32.591 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:22:32.621 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:32.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:32.622 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:22:32.723 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:22:32.756 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:32.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:32.758 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:22:32.858 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:22:32.892 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:32.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:32.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:22:32.994 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:22:33.030 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:33.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:33.031 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:22:33.131 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:22:33.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:33.176 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:33.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:22:33.276 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:22:33.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:33.319 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:33.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:22:33.336 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-17T07:22:33.420 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:22:33.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:33.469 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:33.469 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:22:33.482 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:22:33.483 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:22:33.489 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1: {} 2024-09-17T07:22:33.490 INFO:teuthology.orchestra.run.smithi100.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:22:33.525 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:22:33.526 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:22:33.559 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:22:33.560 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:22:33.570 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:22:33.604 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:22:33.605 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:22:33.612 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:33.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:33.614 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:22:33.629 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:22:33.629 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:22:33.644 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:22:33.650 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:22:33.651 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:22:33.683 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:33 smithi100 ceph-mon[29092]: pgmap v957: 158 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 5 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 146 active+clean; 7.0 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 42 KiB/s rd, 0 B/s wr, 1.12k op/s; 62518/343293 objects degraded (18.211%); 32468/343293 objects misplaced (9.458%); 35 MiB/s, 552 objects/s recovering 2024-09-17T07:22:33.684 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:33 smithi100 ceph-mon[39462]: pgmap v957: 158 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 5 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 146 active+clean; 7.0 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 42 KiB/s rd, 0 B/s wr, 1.12k op/s; 62518/343293 objects degraded (18.211%); 32468/343293 objects misplaced (9.458%); 35 MiB/s, 552 objects/s recovering 2024-09-17T07:22:33.715 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:22:33.749 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:22:33.749 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:22:33.758 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:33.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:33.759 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:22:33.765 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:33 smithi135 ceph-mon[33238]: pgmap v957: 158 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 5 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 146 active+clean; 7.0 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 42 KiB/s rd, 0 B/s wr, 1.12k op/s; 62518/343293 objects degraded (18.211%); 32468/343293 objects misplaced (9.458%); 35 MiB/s, 552 objects/s recovering 2024-09-17T07:22:33.860 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:22:33.894 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:33.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:33.895 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:22:33.995 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:22:34.035 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:34.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:34.037 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:22:34.138 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:22:34.188 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:34.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:34.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:22:34.290 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:22:34.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:34.325 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:34.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:22:34.426 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:22:34.450 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:34.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:34.451 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:22:34.552 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:22:34.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:34.594 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:34.595 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:22:34.655 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T07:22:34.663 INFO:teuthology.orchestra.run.smithi100.stderr:dumped all 2024-09-17T07:22:34.665 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:34 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/417289539' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:22:34.665 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:34 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:22:34.665 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:34 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:22:34.665 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:34 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/417289539' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:22:34.665 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:34 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:22:34.665 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:34 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:22:34.666 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:34 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/417289539' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:22:34.666 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:34 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:22:34.666 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:34 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:22:34.695 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:22:34.728 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:34.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:34.730 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:22:34.831 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:22:34.868 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:34.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:34.869 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:22:34.970 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:22:35.006 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:35.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:35.008 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:22:35.109 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:22:35.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:35.146 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:35.146 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:22:35.247 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:22:35.290 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:35.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:35.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:22:35.392 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:22:35.435 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:35.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:35.437 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:22:35.538 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:22:35.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:35.563 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:35.563 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:22:35.664 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:22:35.671 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:35 smithi135 ceph-mon[33238]: pgmap v958: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 5 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 146 active+clean; 6.9 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 27 KiB/s rd, 0 B/s wr, 1.11k op/s; 61811/340107 objects degraded (18.174%); 32116/340107 objects misplaced (9.443%); 35 MiB/s, 552 objects/s recovering 2024-09-17T07:22:35.671 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:35 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 61811/340107 objects degraded (18.174%), 11 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-17T07:22:35.702 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:35.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:35.703 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:22:35.804 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:22:35.836 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:35.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:35.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:22:35.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:35 smithi100 ceph-mon[29092]: pgmap v958: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 5 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 146 active+clean; 6.9 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 27 KiB/s rd, 0 B/s wr, 1.11k op/s; 61811/340107 objects degraded (18.174%); 32116/340107 objects misplaced (9.443%); 35 MiB/s, 552 objects/s recovering 2024-09-17T07:22:35.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:35 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 61811/340107 objects degraded (18.174%), 11 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-17T07:22:35.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:35 smithi100 ceph-mon[39462]: pgmap v958: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 5 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 146 active+clean; 6.9 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 27 KiB/s rd, 0 B/s wr, 1.11k op/s; 61811/340107 objects degraded (18.174%); 32116/340107 objects misplaced (9.443%); 35 MiB/s, 552 objects/s recovering 2024-09-17T07:22:35.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:35 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 61811/340107 objects degraded (18.174%), 11 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-17T07:22:35.937 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:22:35.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:35.972 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:35.972 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:22:36.073 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:22:36.111 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:36.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:36.112 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:22:36.213 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:22:36.238 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:36.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:36.239 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:22:36.340 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:22:36.376 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:36.377 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:22:35 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:22:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:22:36.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:36.378 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:22:36.408 INFO:teuthology.orchestra.run.smithi135.stdout:{ 2024-09-17T07:22:36.408 INFO:teuthology.orchestra.run.smithi135.stdout: "ops": [], 2024-09-17T07:22:36.408 INFO:teuthology.orchestra.run.smithi135.stdout: "num_ops": 0 2024-09-17T07:22:36.408 INFO:teuthology.orchestra.run.smithi135.stdout:} 2024-09-17T07:22:36.479 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:22:36.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:36.516 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:36.516 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:22:36.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:36 smithi135 ceph-mon[33238]: from='client.16098 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:22:36.617 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:22:36.653 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:36.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:36.654 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:22:36.755 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:22:36.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:36.776 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:36.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:22:36.876 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:36 smithi100 ceph-mon[39462]: from='client.16098 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:22:36.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:36 smithi100 ceph-mon[29092]: from='client.16098 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:22:36.878 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:22:36.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:36.917 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:36.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:22:37.018 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:22:37.051 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:37.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:37.052 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:22:37.119 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-17T07:22:37.153 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:22:37.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:37.213 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:37.213 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:22:37.314 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:22:37.349 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:37.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:37.351 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:22:37.451 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:22:37.485 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:37.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:37.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:22:37.562 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:22:37.587 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:22:37.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:37.624 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:37.625 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:22:37.696 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:37 smithi135 ceph-mon[33238]: pgmap v959: 158 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 5 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 146 active+clean; 6.6 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 7.1 KiB/s rd, 0 B/s wr, 1.17k op/s; 57109/327045 objects degraded (17.462%); 30041/327045 objects misplaced (9.186%); 36 MiB/s, 580 objects/s recovering 2024-09-17T07:22:37.725 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:22:37.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:37.765 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:37.765 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:22:37.866 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:22:37.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:37 smithi100 ceph-mon[29092]: pgmap v959: 158 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 5 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 146 active+clean; 6.6 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 7.1 KiB/s rd, 0 B/s wr, 1.17k op/s; 57109/327045 objects degraded (17.462%); 30041/327045 objects misplaced (9.186%); 36 MiB/s, 580 objects/s recovering 2024-09-17T07:22:37.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:37 smithi100 ceph-mon[39462]: pgmap v959: 158 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 5 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 146 active+clean; 6.6 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 7.1 KiB/s rd, 0 B/s wr, 1.17k op/s; 57109/327045 objects degraded (17.462%); 30041/327045 objects misplaced (9.186%); 36 MiB/s, 580 objects/s recovering 2024-09-17T07:22:37.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:37.905 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:37.905 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:22:37.935 INFO:teuthology.orchestra.run.smithi100.stdout:{"pg_ready":true,"pg_map":{"version":958,"stamp":"2024-09-17T07:22:33.453038+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7417639275,"num_objects":113369,"num_object_clones":0,"num_object_copies":340107,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":61811,"num_objects_misplaced":32116,"num_objects_unfound":0,"num_objects_dirty":113369,"num_whiteouts":0,"num_read":1729,"num_read_kb":1523,"num_write":232406,"num_write_kb":11024886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36239,"num_bytes_recovered":2358599772,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74589,"ondisk_log_size":74589,"up":474,"acting":463,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":481,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":24855628,"kb_used_data":18439124,"kb_used_omap":13,"kb_used_meta":6416306,"kb_avail":631360436,"statfs":{"total":671965249536,"available":646513086464,"internally_reserved":0,"allocated":18881662976,"data_stored":18873932762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13997,"internal_metadata":6570297683},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-698023936,"num_objects":-10651,"num_object_clones":0,"num_object_copies":-31953,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-11626,"num_objects_misplaced":-5245,"num_objects_unfound":0,"num_objects_dirty":-10651,"num_whiteouts":0,"num_read":268,"num_read_kb":268,"num_write":10823,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5526,"num_bytes_recovered":362151936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.006174"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":17,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.506445+0000","last_change":"2024-09-17T07:22:13.586844+0000","last_active":"2024-09-17T07:22:20.506445+0000","last_peered":"2024-09-17T07:22:20.506445+0000","last_clean":"2024-09-17T07:22:20.506445+0000","last_became_active":"2024-09-17T07:22:13.586558+0000","last_became_peered":"2024-09-17T07:22:13.586558+0000","last_unstale":"2024-09-17T07:22:20.506445+0000","last_undegraded":"2024-09-17T07:22:20.506445+0000","last_fullsized":"2024-09-17T07:22:20.506445+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:10.965933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:33.644728+0000","last_clean_scrub_stamp":"2024-09-17T07:20:10.965933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040893329999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":485,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.514094+0000","last_change":"2024-09-17T07:22:13.589818+0000","last_active":"2024-09-17T07:22:20.514094+0000","last_peered":"2024-09-17T07:22:20.514094+0000","last_clean":"2024-09-17T07:22:20.514094+0000","last_became_active":"2024-09-17T07:22:13.589527+0000","last_became_peered":"2024-09-17T07:22:13.589527+0000","last_unstale":"2024-09-17T07:22:20.514094+0000","last_undegraded":"2024-09-17T07:22:20.514094+0000","last_fullsized":"2024-09-17T07:22:20.514094+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:18:46.393298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:18:46.393298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0058972349999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"439'39","reported_seq":395,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:22.252679+0000","last_change":"2024-09-17T07:18:53.477464+0000","last_active":"2024-09-17T07:22:22.252679+0000","last_peered":"2024-09-17T07:22:22.252679+0000","last_clean":"2024-09-17T07:22:22.252679+0000","last_became_active":"2024-09-17T07:09:24.402042+0000","last_became_peered":"2024-09-17T07:09:24.402042+0000","last_unstale":"2024-09-17T07:22:22.252679+0000","last_undegraded":"2024-09-17T07:22:22.252679+0000","last_fullsized":"2024-09-17T07:22:22.252679+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'27","last_scrub_stamp":"2024-09-17T07:18:53.476878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:18:53.476878+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037504449999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":456,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.517669+0000","last_change":"2024-09-17T07:21:15.354814+0000","last_active":"2024-09-17T07:22:20.517669+0000","last_peered":"2024-09-17T07:22:20.517669+0000","last_clean":"2024-09-17T07:22:20.517669+0000","last_became_active":"2024-09-17T07:21:15.354561+0000","last_became_peered":"2024-09-17T07:21:15.354561+0000","last_unstale":"2024-09-17T07:22:20.517669+0000","last_undegraded":"2024-09-17T07:22:20.517669+0000","last_fullsized":"2024-09-17T07:22:20.517669+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:45.261117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:15:45.261117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001477482,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":318,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.514797+0000","last_change":"2024-09-17T07:21:13.454504+0000","last_active":"2024-09-17T07:22:20.514797+0000","last_peered":"2024-09-17T07:22:20.514797+0000","last_clean":"2024-09-17T07:22:20.514797+0000","last_became_active":"2024-09-17T07:09:24.402568+0000","last_became_peered":"2024-09-17T07:09:24.402568+0000","last_unstale":"2024-09-17T07:22:20.514797+0000","last_undegraded":"2024-09-17T07:22:20.514797+0000","last_fullsized":"2024-09-17T07:22:20.514797+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:21:13.454404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:17:35.507742+0000","last_clean_scrub_stamp":"2024-09-17T07:21:13.454404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.016029897000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"438'21","reported_seq":504,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.517744+0000","last_change":"2024-09-17T07:21:41.496057+0000","last_active":"2024-09-17T07:22:20.517744+0000","last_peered":"2024-09-17T07:22:20.517744+0000","last_clean":"2024-09-17T07:22:20.517744+0000","last_became_active":"2024-09-17T07:21:15.354718+0000","last_became_peered":"2024-09-17T07:21:15.354718+0000","last_unstale":"2024-09-17T07:22:20.517744+0000","last_undegraded":"2024-09-17T07:22:20.517744+0000","last_fullsized":"2024-09-17T07:22:20.517744+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'15","last_scrub_stamp":"2024-09-17T07:15:39.306043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:15:39.306043+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002471029,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":337,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.509010+0000","last_change":"2024-09-17T07:22:13.592384+0000","last_active":"2024-09-17T07:22:20.509010+0000","last_peered":"2024-09-17T07:22:20.509010+0000","last_clean":"2024-09-17T07:22:20.509010+0000","last_became_active":"2024-09-17T07:22:13.590701+0000","last_became_peered":"2024-09-17T07:22:13.590701+0000","last_unstale":"2024-09-17T07:22:20.509010+0000","last_undegraded":"2024-09-17T07:22:20.509010+0000","last_fullsized":"2024-09-17T07:22:20.509010+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:13.660077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:20:13.660077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":456,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.518038+0000","last_change":"2024-09-17T07:21:26.408200+0000","last_active":"2024-09-17T07:22:20.518038+0000","last_peered":"2024-09-17T07:22:20.518038+0000","last_clean":"2024-09-17T07:22:20.518038+0000","last_became_active":"2024-09-17T07:21:15.354423+0000","last_became_peered":"2024-09-17T07:21:15.354423+0000","last_unstale":"2024-09-17T07:22:20.518038+0000","last_undegraded":"2024-09-17T07:22:20.518038+0000","last_fullsized":"2024-09-17T07:22:20.518038+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:41.707969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:14:41.707969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.014931757,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":17,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.508780+0000","last_change":"2024-09-17T07:22:13.615802+0000","last_active":"2024-09-17T07:22:20.508780+0000","last_peered":"2024-09-17T07:22:20.508780+0000","last_clean":"2024-09-17T07:22:20.508780+0000","last_became_active":"2024-09-17T07:22:13.615512+0000","last_became_peered":"2024-09-17T07:22:13.615512+0000","last_unstale":"2024-09-17T07:22:20.508780+0000","last_undegraded":"2024-09-17T07:22:20.508780+0000","last_fullsized":"2024-09-17T07:22:20.508780+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:18:44.849299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:46.312371+0000","last_clean_scrub_stamp":"2024-09-17T07:18:44.849299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"439'9","reported_seq":492,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:21.168807+0000","last_change":"2024-09-17T07:22:18.414607+0000","last_active":"2024-09-17T07:22:21.168807+0000","last_peered":"2024-09-17T07:22:21.168807+0000","last_clean":"2024-09-17T07:22:21.168807+0000","last_became_active":"2024-09-17T07:21:15.349235+0000","last_became_peered":"2024-09-17T07:21:15.349235+0000","last_unstale":"2024-09-17T07:22:21.168807+0000","last_undegraded":"2024-09-17T07:22:21.168807+0000","last_fullsized":"2024-09-17T07:22:21.168807+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'7","last_scrub_stamp":"2024-09-17T07:14:42.653705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:14:42.653705+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0050149230000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":13,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":17,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.510359+0000","last_change":"2024-09-17T07:22:13.624911+0000","last_active":"2024-09-17T07:22:20.510359+0000","last_peered":"2024-09-17T07:22:20.510359+0000","last_clean":"2024-09-17T07:22:20.510359+0000","last_became_active":"2024-09-17T07:22:13.624441+0000","last_became_peered":"2024-09-17T07:22:13.624441+0000","last_unstale":"2024-09-17T07:22:20.510359+0000","last_undegraded":"2024-09-17T07:22:20.510359+0000","last_fullsized":"2024-09-17T07:22:20.510359+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:10.116580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:20:10.116580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00137765,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":473,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.511443+0000","last_change":"2024-09-17T07:21:15.344952+0000","last_active":"2024-09-17T07:22:20.511443+0000","last_peered":"2024-09-17T07:22:20.511443+0000","last_clean":"2024-09-17T07:22:20.511443+0000","last_became_active":"2024-09-17T07:21:15.344389+0000","last_became_peered":"2024-09-17T07:21:15.344389+0000","last_unstale":"2024-09-17T07:22:20.511443+0000","last_undegraded":"2024-09-17T07:22:20.511443+0000","last_fullsized":"2024-09-17T07:22:20.511443+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:52.168181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:14:52.168181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0056501399999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":17,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.508164+0000","last_change":"2024-09-17T07:22:13.696707+0000","last_active":"2024-09-17T07:22:20.508164+0000","last_peered":"2024-09-17T07:22:20.508164+0000","last_clean":"2024-09-17T07:22:20.508164+0000","last_became_active":"2024-09-17T07:22:13.693199+0000","last_became_peered":"2024-09-17T07:22:13.693199+0000","last_unstale":"2024-09-17T07:22:20.508164+0000","last_undegraded":"2024-09-17T07:22:20.508164+0000","last_fullsized":"2024-09-17T07:22:20.508164+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:18.961914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:45.029247+0000","last_clean_scrub_stamp":"2024-09-17T07:19:18.961914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"439'25","reported_seq":533,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:21.346517+0000","last_change":"2024-09-17T07:20:39.070372+0000","last_active":"2024-09-17T07:22:21.346517+0000","last_peered":"2024-09-17T07:22:21.346517+0000","last_clean":"2024-09-17T07:22:21.346517+0000","last_became_active":"2024-09-17T07:09:24.397954+0000","last_became_peered":"2024-09-17T07:09:24.397954+0000","last_unstale":"2024-09-17T07:22:21.346517+0000","last_undegraded":"2024-09-17T07:22:21.346517+0000","last_fullsized":"2024-09-17T07:22:21.346517+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"312'19","last_scrub_stamp":"2024-09-17T07:20:39.070235+0000","last_deep_scrub":"312'19","last_deep_scrub_stamp":"2024-09-17T07:16:44.292911+0000","last_clean_scrub_stamp":"2024-09-17T07:20:39.070235+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0028570560000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":490,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.511600+0000","last_change":"2024-09-17T07:22:13.578867+0000","last_active":"2024-09-17T07:22:20.511600+0000","last_peered":"2024-09-17T07:22:20.511600+0000","last_clean":"2024-09-17T07:22:20.511600+0000","last_became_active":"2024-09-17T07:22:13.578565+0000","last_became_peered":"2024-09-17T07:22:13.578565+0000","last_unstale":"2024-09-17T07:22:20.511600+0000","last_undegraded":"2024-09-17T07:22:20.511600+0000","last_fullsized":"2024-09-17T07:22:20.511600+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:26.348475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:28.096266+0000","last_clean_scrub_stamp":"2024-09-17T07:19:26.348475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.015361906,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":47,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.509128+0000","last_change":"2024-09-17T07:22:13.590579+0000","last_active":"2024-09-17T07:22:20.509128+0000","last_peered":"2024-09-17T07:22:20.509128+0000","last_clean":"2024-09-17T07:22:20.509128+0000","last_became_active":"2024-09-17T07:22:13.590276+0000","last_became_peered":"2024-09-17T07:22:13.590276+0000","last_unstale":"2024-09-17T07:22:20.509128+0000","last_undegraded":"2024-09-17T07:22:20.509128+0000","last_fullsized":"2024-09-17T07:22:20.509128+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:27.474541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:20:27.474541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0061993300000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2","version":"440'14531","reported_seq":15326,"reported_epoch":440,"state":"active+undersized+degraded+remapped+backfill_wait","last_fresh":"2024-09-17T07:22:29.982013+0000","last_change":"2024-09-17T07:22:14.581009+0000","last_active":"2024-09-17T07:22:29.982013+0000","last_peered":"2024-09-17T07:22:29.982013+0000","last_clean":"2024-09-17T07:19:54.447320+0000","last_became_active":"2024-09-17T07:22:13.687492+0000","last_became_peered":"2024-09-17T07:22:13.687492+0000","last_unstale":"2024-09-17T07:22:29.982013+0000","last_undegraded":"2024-09-17T07:22:13.572458+0000","last_fullsized":"2024-09-17T07:22:13.554452+0000","mapping_epoch":433,"log_start":"421'9900","ondisk_log_start":"421'9900","created":390,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_clean_scrub_stamp":"2024-09-17T07:19:54.447320+0000","objects_scrubbed":0,"log_size":4631,"log_dups_size":0,"ondisk_log_size":4631,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":471531520,"num_objects":7195,"num_object_clones":0,"num_object_copies":21585,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6887,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14531,"num_write_kb":695232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":360,"num_bytes_recovered":23592960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":7195}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.8","version":"212'130","reported_seq":534,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.510909+0000","last_change":"2024-09-17T07:20:21.626281+0000","last_active":"2024-09-17T07:22:20.510909+0000","last_peered":"2024-09-17T07:22:20.510909+0000","last_clean":"2024-09-17T07:22:20.510909+0000","last_became_active":"2024-09-17T07:19:35.418729+0000","last_became_peered":"2024-09-17T07:19:35.418729+0000","last_unstale":"2024-09-17T07:22:20.510909+0000","last_undegraded":"2024-09-17T07:22:20.510909+0000","last_fullsized":"2024-09-17T07:22:20.510909+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":4,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:20:21.626089+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:14:57.052505+0000","last_clean_scrub_stamp":"2024-09-17T07:20:21.626089+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0050515719999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"439'53","reported_seq":548,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-17T07:22:24.392773+0000","last_change":"2024-09-17T07:21:15.349153+0000","last_active":"2024-09-17T07:22:24.392773+0000","last_peered":"2024-09-17T07:22:24.392773+0000","last_clean":"2024-09-17T07:22:24.392773+0000","last_became_active":"2024-09-17T07:21:15.348413+0000","last_became_peered":"2024-09-17T07:21:15.348413+0000","last_unstale":"2024-09-17T07:22:24.392773+0000","last_undegraded":"2024-09-17T07:22:24.392773+0000","last_fullsized":"2024-09-17T07:22:24.392773+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'39","last_scrub_stamp":"2024-09-17T07:15:46.226554+0000","last_deep_scrub":"95'25","last_deep_scrub_stamp":"2024-09-17T07:04:09.450039+0000","last_clean_scrub_stamp":"2024-09-17T07:15:46.226554+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0027061799999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":61,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":17,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.506580+0000","last_change":"2024-09-17T07:22:13.625588+0000","last_active":"2024-09-17T07:22:20.506580+0000","last_peered":"2024-09-17T07:22:20.506580+0000","last_clean":"2024-09-17T07:22:20.506580+0000","last_became_active":"2024-09-17T07:22:13.624474+0000","last_became_peered":"2024-09-17T07:22:13.624474+0000","last_unstale":"2024-09-17T07:22:20.506580+0000","last_undegraded":"2024-09-17T07:22:20.506580+0000","last_fullsized":"2024-09-17T07:22:20.506580+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:35.965669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:19:35.965669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":291,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.509457+0000","last_change":"2024-09-17T07:21:15.350941+0000","last_active":"2024-09-17T07:22:20.509457+0000","last_peered":"2024-09-17T07:22:20.509457+0000","last_clean":"2024-09-17T07:22:20.509457+0000","last_became_active":"2024-09-17T07:21:15.349538+0000","last_became_peered":"2024-09-17T07:21:15.349538+0000","last_unstale":"2024-09-17T07:22:20.509457+0000","last_undegraded":"2024-09-17T07:22:20.509457+0000","last_fullsized":"2024-09-17T07:22:20.509457+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:46.502641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:13:46.502641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":320,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.517762+0000","last_change":"2024-09-17T07:20:40.117298+0000","last_active":"2024-09-17T07:22:20.517762+0000","last_peered":"2024-09-17T07:22:20.517762+0000","last_clean":"2024-09-17T07:22:20.517762+0000","last_became_active":"2024-09-17T07:09:24.396334+0000","last_became_peered":"2024-09-17T07:09:24.396334+0000","last_unstale":"2024-09-17T07:22:20.517762+0000","last_undegraded":"2024-09-17T07:22:20.517762+0000","last_fullsized":"2024-09-17T07:22:20.517762+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:40.117165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:42.984088+0000","last_clean_scrub_stamp":"2024-09-17T07:20:40.117165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0041399059999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.3","version":"440'14419","reported_seq":20231,"reported_epoch":440,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-17T07:22:29.114178+0000","last_change":"2024-09-17T07:22:13.785052+0000","last_active":"2024-09-17T07:22:29.114178+0000","last_peered":"2024-09-17T07:22:29.114178+0000","last_clean":"2024-09-17T07:19:54.447320+0000","last_became_active":"2024-09-17T07:22:13.693522+0000","last_became_peered":"2024-09-17T07:22:13.693522+0000","last_unstale":"2024-09-17T07:22:29.114178+0000","last_undegraded":"2024-09-17T07:22:13.587898+0000","last_fullsized":"2024-09-17T07:22:13.550280+0000","mapping_epoch":433,"log_start":"422'10100","ondisk_log_start":"422'10100","created":390,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_clean_scrub_stamp":"2024-09-17T07:19:54.447320+0000","objects_scrubbed":0,"log_size":4319,"log_dups_size":0,"ondisk_log_size":4319,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":468123648,"num_objects":7143,"num_object_clones":0,"num_object_copies":21429,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5354,"num_objects_misplaced":5354,"num_objects_unfound":0,"num_objects_dirty":7143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14419,"num_write_kb":689984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1911,"num_bytes_recovered":125239296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[{"shards":"5,6","objects":7143}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"212'130","reported_seq":672,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.509061+0000","last_change":"2024-09-17T07:22:16.355905+0000","last_active":"2024-09-17T07:22:20.509061+0000","last_peered":"2024-09-17T07:22:20.509061+0000","last_clean":"2024-09-17T07:22:20.509061+0000","last_became_active":"2024-09-17T07:22:16.355472+0000","last_became_peered":"2024-09-17T07:22:16.355472+0000","last_unstale":"2024-09-17T07:22:20.509061+0000","last_undegraded":"2024-09-17T07:22:20.509061+0000","last_fullsized":"2024-09-17T07:22:20.509061+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":4,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:21:12.403277+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:14:57.052505+0000","last_clean_scrub_stamp":"2024-09-17T07:21:12.403277+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0071578960000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":28,"num_write":16,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":161009,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"439'35","reported_seq":542,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-17T07:22:24.393342+0000","last_change":"2024-09-17T07:21:41.497159+0000","last_active":"2024-09-17T07:22:24.393342+0000","last_peered":"2024-09-17T07:22:24.393342+0000","last_clean":"2024-09-17T07:22:24.393342+0000","last_became_active":"2024-09-17T07:21:15.347129+0000","last_became_peered":"2024-09-17T07:21:15.347129+0000","last_unstale":"2024-09-17T07:22:24.393342+0000","last_undegraded":"2024-09-17T07:22:24.393342+0000","last_fullsized":"2024-09-17T07:22:24.393342+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'25","last_scrub_stamp":"2024-09-17T07:15:02.532371+0000","last_deep_scrub":"176'15","last_deep_scrub_stamp":"2024-09-17T07:08:19.633791+0000","last_clean_scrub_stamp":"2024-09-17T07:15:02.532371+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012712606,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":335,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.514442+0000","last_change":"2024-09-17T07:22:13.590620+0000","last_active":"2024-09-17T07:22:20.514442+0000","last_peered":"2024-09-17T07:22:20.514442+0000","last_clean":"2024-09-17T07:22:20.514442+0000","last_became_active":"2024-09-17T07:22:13.590345+0000","last_became_peered":"2024-09-17T07:22:13.590345+0000","last_unstale":"2024-09-17T07:22:20.514442+0000","last_undegraded":"2024-09-17T07:22:20.514442+0000","last_fullsized":"2024-09-17T07:22:20.514442+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:11.580871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:20:11.580871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0065215869999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"436'11","reported_seq":25,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-17T07:22:25.766407+0000","last_change":"2024-09-17T07:22:13.756738+0000","last_active":"2024-09-17T07:22:25.766407+0000","last_peered":"2024-09-17T07:22:25.766407+0000","last_clean":"2024-09-17T07:22:25.766407+0000","last_became_active":"2024-09-17T07:22:13.628798+0000","last_became_peered":"2024-09-17T07:22:13.628798+0000","last_unstale":"2024-09-17T07:22:25.766407+0000","last_undegraded":"2024-09-17T07:22:25.766407+0000","last_fullsized":"2024-09-17T07:22:25.766407+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-09-17T07:19:24.964680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:19:24.964680+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":467,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.517949+0000","last_change":"2024-09-17T07:22:16.350166+0000","last_active":"2024-09-17T07:22:20.517949+0000","last_peered":"2024-09-17T07:22:20.517949+0000","last_clean":"2024-09-17T07:22:20.517949+0000","last_became_active":"2024-09-17T07:21:15.348799+0000","last_became_peered":"2024-09-17T07:21:15.348799+0000","last_unstale":"2024-09-17T07:22:20.517949+0000","last_undegraded":"2024-09-17T07:22:20.517949+0000","last_fullsized":"2024-09-17T07:22:20.517949+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:50.554591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:14:50.554591+0000","last_clean_scrub_stamp":"2024-09-17T07:14:50.554591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0067125969999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.0","version":"440'14565","reported_seq":19129,"reported_epoch":440,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-17T07:22:30.012558+0000","last_change":"2024-09-17T07:22:14.587455+0000","last_active":"2024-09-17T07:22:30.012558+0000","last_peered":"2024-09-17T07:22:30.012558+0000","last_clean":"2024-09-17T07:19:54.447320+0000","last_became_active":"2024-09-17T07:22:13.712336+0000","last_became_peered":"2024-09-17T07:22:13.712336+0000","last_unstale":"2024-09-17T07:22:30.012558+0000","last_undegraded":"2024-09-17T07:22:13.576336+0000","last_fullsized":"2024-09-17T07:22:13.550547+0000","mapping_epoch":433,"log_start":"421'10000","ondisk_log_start":"421'10000","created":390,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_clean_scrub_stamp":"2024-09-17T07:19:54.447320+0000","objects_scrubbed":0,"log_size":4565,"log_dups_size":0,"ondisk_log_size":4565,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":470482944,"num_objects":7179,"num_object_clones":0,"num_object_copies":21537,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5773,"num_objects_misplaced":5773,"num_objects_unfound":0,"num_objects_dirty":7179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14565,"num_write_kb":695808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1495,"num_bytes_recovered":97976320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[{"shards":"0,6","objects":7179}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.a","version":"212'130","reported_seq":534,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.512512+0000","last_change":"2024-09-17T07:20:33.710216+0000","last_active":"2024-09-17T07:22:20.512512+0000","last_peered":"2024-09-17T07:22:20.512512+0000","last_clean":"2024-09-17T07:22:20.512512+0000","last_became_active":"2024-09-17T07:19:35.422004+0000","last_became_peered":"2024-09-17T07:19:35.422004+0000","last_unstale":"2024-09-17T07:22:20.512512+0000","last_undegraded":"2024-09-17T07:22:20.512512+0000","last_fullsized":"2024-09-17T07:22:20.512512+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":4,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:20:33.709901+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:14:57.052505+0000","last_clean_scrub_stamp":"2024-09-17T07:20:33.709901+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0058367610000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"439'39","reported_seq":423,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:22.300151+0000","last_change":"2024-09-17T07:22:13.780063+0000","last_active":"2024-09-17T07:22:22.300151+0000","last_peered":"2024-09-17T07:22:22.300151+0000","last_clean":"2024-09-17T07:22:22.300151+0000","last_became_active":"2024-09-17T07:22:13.589782+0000","last_became_peered":"2024-09-17T07:22:13.589782+0000","last_unstale":"2024-09-17T07:22:22.300151+0000","last_undegraded":"2024-09-17T07:22:22.300151+0000","last_fullsized":"2024-09-17T07:22:22.300151+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'27","last_scrub_stamp":"2024-09-17T07:20:13.130718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:20:13.130718+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0081162040000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":349,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.509080+0000","last_change":"2024-09-17T07:22:13.592519+0000","last_active":"2024-09-17T07:22:20.509080+0000","last_peered":"2024-09-17T07:22:20.509080+0000","last_clean":"2024-09-17T07:22:20.509080+0000","last_became_active":"2024-09-17T07:22:13.591089+0000","last_became_peered":"2024-09-17T07:22:13.591089+0000","last_unstale":"2024-09-17T07:22:20.509080+0000","last_undegraded":"2024-09-17T07:22:20.509080+0000","last_fullsized":"2024-09-17T07:22:20.509080+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:14.001258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:20:14.001258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034116369999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"75'8","reported_seq":716,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-17T07:22:30.754000+0000","last_change":"2024-09-17T07:19:24.039612+0000","last_active":"2024-09-17T07:22:30.754000+0000","last_peered":"2024-09-17T07:22:30.754000+0000","last_clean":"2024-09-17T07:22:30.754000+0000","last_became_active":"2024-09-17T07:02:16.021762+0000","last_became_peered":"2024-09-17T07:02:16.021762+0000","last_unstale":"2024-09-17T07:22:30.754000+0000","last_undegraded":"2024-09-17T07:22:30.754000+0000","last_fullsized":"2024-09-17T07:22:30.754000+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:19:24.039495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:19:24.039495+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.016850853999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":464,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.513982+0000","last_change":"2024-09-17T07:22:14.574155+0000","last_active":"2024-09-17T07:22:20.513982+0000","last_peered":"2024-09-17T07:22:20.513982+0000","last_clean":"2024-09-17T07:22:20.513982+0000","last_became_active":"2024-09-17T07:21:15.353307+0000","last_became_peered":"2024-09-17T07:21:15.353307+0000","last_unstale":"2024-09-17T07:22:20.513982+0000","last_undegraded":"2024-09-17T07:22:20.513982+0000","last_fullsized":"2024-09-17T07:22:20.513982+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:56.146098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:14:56.146098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0054479960000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"440'14312","reported_seq":14372,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-17T07:22:29.086485+0000","last_change":"2024-09-17T07:19:55.482992+0000","last_active":"2024-09-17T07:22:29.086485+0000","last_peered":"2024-09-17T07:22:29.086485+0000","last_clean":"2024-09-17T07:22:29.086485+0000","last_became_active":"2024-09-17T07:19:55.482080+0000","last_became_peered":"2024-09-17T07:19:55.482080+0000","last_unstale":"2024-09-17T07:22:29.086485+0000","last_undegraded":"2024-09-17T07:22:29.086485+0000","last_fullsized":"2024-09-17T07:22:29.086485+0000","mapping_epoch":390,"log_start":"422'10000","ondisk_log_start":"422'10000","created":390,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_clean_scrub_stamp":"2024-09-17T07:19:54.447320+0000","objects_scrubbed":0,"log_size":4312,"log_dups_size":0,"ondisk_log_size":4312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":471334912,"num_objects":7192,"num_object_clones":0,"num_object_copies":21576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7192,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14312,"num_write_kb":688128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"439'27","reported_seq":514,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-17T07:22:24.393196+0000","last_change":"2024-09-17T07:21:29.494226+0000","last_active":"2024-09-17T07:22:24.393196+0000","last_peered":"2024-09-17T07:22:24.393196+0000","last_clean":"2024-09-17T07:22:24.393196+0000","last_became_active":"2024-09-17T07:21:15.347662+0000","last_became_peered":"2024-09-17T07:21:15.347662+0000","last_unstale":"2024-09-17T07:22:24.393196+0000","last_undegraded":"2024-09-17T07:22:24.393196+0000","last_fullsized":"2024-09-17T07:22:24.393196+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'19","last_scrub_stamp":"2024-09-17T07:14:55.489874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:14:55.489874+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045924440000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":17,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.507728+0000","last_change":"2024-09-17T07:22:13.711817+0000","last_active":"2024-09-17T07:22:20.507728+0000","last_peered":"2024-09-17T07:22:20.507728+0000","last_clean":"2024-09-17T07:22:20.507728+0000","last_became_active":"2024-09-17T07:22:13.709873+0000","last_became_peered":"2024-09-17T07:22:13.709873+0000","last_unstale":"2024-09-17T07:22:20.507728+0000","last_undegraded":"2024-09-17T07:22:20.507728+0000","last_fullsized":"2024-09-17T07:22:20.507728+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:12.442421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:20:12.442421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026862600000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":483,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.512015+0000","last_change":"2024-09-17T07:19:30.427362+0000","last_active":"2024-09-17T07:22:20.512015+0000","last_peered":"2024-09-17T07:22:20.512015+0000","last_clean":"2024-09-17T07:22:20.512015+0000","last_became_active":"2024-09-17T07:02:16.020224+0000","last_became_peered":"2024-09-17T07:02:16.020224+0000","last_unstale":"2024-09-17T07:22:20.512015+0000","last_undegraded":"2024-09-17T07:22:20.512015+0000","last_fullsized":"2024-09-17T07:22:20.512015+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:30.427245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:19:30.427245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010614099,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":355,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.515208+0000","last_change":"2024-09-17T07:22:13.594876+0000","last_active":"2024-09-17T07:22:20.515208+0000","last_peered":"2024-09-17T07:22:20.515208+0000","last_clean":"2024-09-17T07:22:20.515208+0000","last_became_active":"2024-09-17T07:22:13.594588+0000","last_became_peered":"2024-09-17T07:22:13.594588+0000","last_unstale":"2024-09-17T07:22:20.515208+0000","last_undegraded":"2024-09-17T07:22:20.515208+0000","last_fullsized":"2024-09-17T07:22:20.515208+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:18:40.925918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:13:22.187127+0000","last_clean_scrub_stamp":"2024-09-17T07:18:40.925918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0076924289999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.e","version":"440'14994","reported_seq":15074,"reported_epoch":440,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2024-09-17T07:22:33.392152+0000","last_change":"2024-09-17T07:22:13.764231+0000","last_active":"2024-09-17T07:22:33.392152+0000","last_peered":"2024-09-17T07:22:33.392152+0000","last_clean":"2024-09-17T07:19:54.447320+0000","last_became_active":"2024-09-17T07:22:13.709927+0000","last_became_peered":"2024-09-17T07:22:13.709927+0000","last_unstale":"2024-09-17T07:22:33.392152+0000","last_undegraded":"2024-09-17T07:22:13.560755+0000","last_fullsized":"2024-09-17T07:22:13.547223+0000","mapping_epoch":433,"log_start":"422'10375","ondisk_log_start":"422'10375","created":390,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_clean_scrub_stamp":"2024-09-17T07:19:54.447320+0000","objects_scrubbed":0,"log_size":4619,"log_dups_size":0,"ondisk_log_size":4619,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":458096640,"num_objects":6990,"num_object_clones":0,"num_object_copies":20970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6990,"num_objects_misplaced":6990,"num_objects_unfound":0,"num_objects_dirty":6990,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14994,"num_write_kb":703488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[{"shards":"0,4","objects":6990}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"212'130","reported_seq":576,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.511112+0000","last_change":"2024-09-17T07:22:16.371127+0000","last_active":"2024-09-17T07:22:20.511112+0000","last_peered":"2024-09-17T07:22:20.511112+0000","last_clean":"2024-09-17T07:22:20.511112+0000","last_became_active":"2024-09-17T07:22:16.370810+0000","last_became_peered":"2024-09-17T07:22:16.370810+0000","last_unstale":"2024-09-17T07:22:20.511112+0000","last_undegraded":"2024-09-17T07:22:20.511112+0000","last_fullsized":"2024-09-17T07:22:20.511112+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":3,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:20:40.628770+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:14:57.052505+0000","last_clean_scrub_stamp":"2024-09-17T07:20:40.628770+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0027179940000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1","version":"439'42","reported_seq":529,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:22.210337+0000","last_change":"2024-09-17T07:20:28.029486+0000","last_active":"2024-09-17T07:22:22.210337+0000","last_peered":"2024-09-17T07:22:22.210337+0000","last_clean":"2024-09-17T07:22:22.210337+0000","last_became_active":"2024-09-17T07:02:14.007810+0000","last_became_peered":"2024-09-17T07:02:14.007810+0000","last_unstale":"2024-09-17T07:22:22.210337+0000","last_undegraded":"2024-09-17T07:22:22.210337+0000","last_fullsized":"2024-09-17T07:22:22.210337+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'28","last_scrub_stamp":"2024-09-17T07:20:28.029279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:20:28.029279+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0079911889999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":63,"num_read_kb":56,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":538,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.516240+0000","last_change":"2024-09-17T07:22:13.582343+0000","last_active":"2024-09-17T07:22:20.516240+0000","last_peered":"2024-09-17T07:22:20.516240+0000","last_clean":"2024-09-17T07:22:20.516240+0000","last_became_active":"2024-09-17T07:22:13.582096+0000","last_became_peered":"2024-09-17T07:22:13.582096+0000","last_unstale":"2024-09-17T07:22:20.516240+0000","last_undegraded":"2024-09-17T07:22:20.516240+0000","last_fullsized":"2024-09-17T07:22:20.516240+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:27.831228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:09:23.181028+0000","last_clean_scrub_stamp":"2024-09-17T07:19:27.831228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002288249,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"414'11","reported_seq":292,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-17T07:22:24.391606+0000","last_change":"2024-09-17T07:21:15.356155+0000","last_active":"2024-09-17T07:22:24.391606+0000","last_peered":"2024-09-17T07:22:24.391606+0000","last_clean":"2024-09-17T07:22:24.391606+0000","last_became_active":"2024-09-17T07:21:15.355651+0000","last_became_peered":"2024-09-17T07:21:15.355651+0000","last_unstale":"2024-09-17T07:22:24.391606+0000","last_undegraded":"2024-09-17T07:22:24.391606+0000","last_fullsized":"2024-09-17T07:22:24.391606+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'9","last_scrub_stamp":"2024-09-17T07:15:55.813882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:15:55.813882+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":17,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.506357+0000","last_change":"2024-09-17T07:22:13.568903+0000","last_active":"2024-09-17T07:22:20.506357+0000","last_peered":"2024-09-17T07:22:20.506357+0000","last_clean":"2024-09-17T07:22:20.506357+0000","last_became_active":"2024-09-17T07:22:13.568656+0000","last_became_peered":"2024-09-17T07:22:13.568656+0000","last_unstale":"2024-09-17T07:22:20.506357+0000","last_undegraded":"2024-09-17T07:22:20.506357+0000","last_fullsized":"2024-09-17T07:22:20.506357+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:18:46.921412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:10:33.727472+0000","last_clean_scrub_stamp":"2024-09-17T07:18:46.921412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.f","version":"440'14528","reported_seq":14602,"reported_epoch":440,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2024-09-17T07:22:29.986560+0000","last_change":"2024-09-17T07:22:13.759060+0000","last_active":"2024-09-17T07:22:29.986560+0000","last_peered":"2024-09-17T07:22:29.986560+0000","last_clean":"2024-09-17T07:19:54.447320+0000","last_became_active":"2024-09-17T07:22:13.703765+0000","last_became_peered":"2024-09-17T07:22:13.703765+0000","last_unstale":"2024-09-17T07:22:29.986560+0000","last_undegraded":"2024-09-17T07:22:13.574180+0000","last_fullsized":"2024-09-17T07:22:13.551394+0000","mapping_epoch":433,"log_start":"421'9900","ondisk_log_start":"421'9900","created":390,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_clean_scrub_stamp":"2024-09-17T07:19:54.447320+0000","objects_scrubbed":0,"log_size":4628,"log_dups_size":0,"ondisk_log_size":4628,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":462553088,"num_objects":7058,"num_object_clones":0,"num_object_copies":21174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7058,"num_objects_misplaced":7058,"num_objects_unfound":0,"num_objects_dirty":7058,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14528,"num_write_kb":690752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":7058}],"blocked_by":[],"up_primary":7,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"212'130","reported_seq":534,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.518615+0000","last_change":"2024-09-17T07:20:35.080459+0000","last_active":"2024-09-17T07:22:20.518615+0000","last_peered":"2024-09-17T07:22:20.518615+0000","last_clean":"2024-09-17T07:22:20.518615+0000","last_became_active":"2024-09-17T07:19:35.420337+0000","last_became_peered":"2024-09-17T07:19:35.420337+0000","last_unstale":"2024-09-17T07:22:20.518615+0000","last_undegraded":"2024-09-17T07:22:20.518615+0000","last_fullsized":"2024-09-17T07:22:20.518615+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":3,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:20:35.080257+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:20:35.080257+0000","last_clean_scrub_stamp":"2024-09-17T07:20:35.080257+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0030477730000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"439'42","reported_seq":53,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:21.847808+0000","last_change":"2024-09-17T07:22:13.779383+0000","last_active":"2024-09-17T07:22:21.847808+0000","last_peered":"2024-09-17T07:22:21.847808+0000","last_clean":"2024-09-17T07:22:21.847808+0000","last_became_active":"2024-09-17T07:22:13.711444+0000","last_became_peered":"2024-09-17T07:22:13.711444+0000","last_unstale":"2024-09-17T07:22:21.847808+0000","last_undegraded":"2024-09-17T07:22:21.847808+0000","last_fullsized":"2024-09-17T07:22:21.847808+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'30","last_scrub_stamp":"2024-09-17T07:20:13.427290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:20:13.427290+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0050927619999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":68,"num_read_kb":56,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":345,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.514195+0000","last_change":"2024-09-17T07:21:15.348172+0000","last_active":"2024-09-17T07:22:20.514195+0000","last_peered":"2024-09-17T07:22:20.514195+0000","last_clean":"2024-09-17T07:22:20.514195+0000","last_became_active":"2024-09-17T07:21:15.347867+0000","last_became_peered":"2024-09-17T07:21:15.347867+0000","last_unstale":"2024-09-17T07:22:20.514195+0000","last_undegraded":"2024-09-17T07:22:20.514195+0000","last_fullsized":"2024-09-17T07:22:20.514195+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:56.848689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:56.848689+0000","last_clean_scrub_stamp":"2024-09-17T07:15:56.848689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":491,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.518558+0000","last_change":"2024-09-17T07:22:13.567204+0000","last_active":"2024-09-17T07:22:20.518558+0000","last_peered":"2024-09-17T07:22:20.518558+0000","last_clean":"2024-09-17T07:22:20.518558+0000","last_became_active":"2024-09-17T07:22:13.566916+0000","last_became_peered":"2024-09-17T07:22:13.566916+0000","last_unstale":"2024-09-17T07:22:20.518558+0000","last_undegraded":"2024-09-17T07:22:20.518558+0000","last_fullsized":"2024-09-17T07:22:20.518558+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:21.979680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:19:21.979680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0094045190000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":471,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.518255+0000","last_change":"2024-09-17T07:19:27.090311+0000","last_active":"2024-09-17T07:22:20.518255+0000","last_peered":"2024-09-17T07:22:20.518255+0000","last_clean":"2024-09-17T07:22:20.518255+0000","last_became_active":"2024-09-17T07:09:24.398659+0000","last_became_peered":"2024-09-17T07:09:24.398659+0000","last_unstale":"2024-09-17T07:22:20.518255+0000","last_undegraded":"2024-09-17T07:22:20.518255+0000","last_fullsized":"2024-09-17T07:22:20.518255+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:27.090201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:19:27.090201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026497019999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"440'14658","reported_seq":14729,"reported_epoch":440,"state":"active+undersized+degraded+remapped+backfill_wait+backfill_toofull","last_fresh":"2024-09-17T07:22:33.030749+0000","last_change":"2024-09-17T07:22:16.499732+0000","last_active":"2024-09-17T07:22:33.030749+0000","last_peered":"2024-09-17T07:22:33.030749+0000","last_clean":"2024-09-17T07:19:54.447320+0000","last_became_active":"2024-09-17T07:21:16.466676+0000","last_became_peered":"2024-09-17T07:21:16.466676+0000","last_unstale":"2024-09-17T07:22:33.030749+0000","last_undegraded":"2024-09-17T07:21:16.347636+0000","last_fullsized":"2024-09-17T07:21:16.334665+0000","mapping_epoch":414,"log_start":"421'9800","ondisk_log_start":"421'9800","created":390,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_clean_scrub_stamp":"2024-09-17T07:19:54.447320+0000","objects_scrubbed":0,"log_size":4858,"log_dups_size":0,"ondisk_log_size":4858,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":451018752,"num_objects":6882,"num_object_clones":0,"num_object_copies":20646,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6882,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14658,"num_write_kb":689280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[{"shards":"2,4","objects":6882}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"212'130","reported_seq":542,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.514648+0000","last_change":"2024-09-17T07:20:33.068709+0000","last_active":"2024-09-17T07:22:20.514648+0000","last_peered":"2024-09-17T07:22:20.514648+0000","last_clean":"2024-09-17T07:22:20.514648+0000","last_became_active":"2024-09-17T07:19:35.428161+0000","last_became_peered":"2024-09-17T07:19:35.428161+0000","last_unstale":"2024-09-17T07:22:20.514648+0000","last_undegraded":"2024-09-17T07:22:20.514648+0000","last_fullsized":"2024-09-17T07:22:20.514648+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":3,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:20:33.068618+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:14:57.052505+0000","last_clean_scrub_stamp":"2024-09-17T07:20:33.068618+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0059947730000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"439'39","reported_seq":597,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:22.051777+0000","last_change":"2024-09-17T07:22:13.785255+0000","last_active":"2024-09-17T07:22:22.051777+0000","last_peered":"2024-09-17T07:22:22.051777+0000","last_clean":"2024-09-17T07:22:22.051777+0000","last_became_active":"2024-09-17T07:22:13.624796+0000","last_became_peered":"2024-09-17T07:22:13.624796+0000","last_unstale":"2024-09-17T07:22:22.051777+0000","last_undegraded":"2024-09-17T07:22:22.051777+0000","last_fullsized":"2024-09-17T07:22:22.051777+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'29","last_scrub_stamp":"2024-09-17T07:19:35.438907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:19:35.438907+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010401474000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.508710+0000","last_change":"2024-09-17T07:22:13.625084+0000","last_active":"2024-09-17T07:22:20.508710+0000","last_peered":"2024-09-17T07:22:20.508710+0000","last_clean":"2024-09-17T07:22:20.508710+0000","last_became_active":"2024-09-17T07:22:13.624802+0000","last_became_peered":"2024-09-17T07:22:13.624802+0000","last_unstale":"2024-09-17T07:22:20.508710+0000","last_undegraded":"2024-09-17T07:22:20.508710+0000","last_fullsized":"2024-09-17T07:22:20.508710+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:37.930831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:47.631387+0000","last_clean_scrub_stamp":"2024-09-17T07:19:37.930831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":484,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.514558+0000","last_change":"2024-09-17T07:19:31.968735+0000","last_active":"2024-09-17T07:22:20.514558+0000","last_peered":"2024-09-17T07:22:20.514558+0000","last_clean":"2024-09-17T07:22:20.514558+0000","last_became_active":"2024-09-17T07:02:16.016199+0000","last_became_peered":"2024-09-17T07:02:16.016199+0000","last_unstale":"2024-09-17T07:22:20.514558+0000","last_undegraded":"2024-09-17T07:22:20.514558+0000","last_fullsized":"2024-09-17T07:22:20.514558+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:31.968422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:11:17.583712+0000","last_clean_scrub_stamp":"2024-09-17T07:19:31.968422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.076819294999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":300,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.508916+0000","last_change":"2024-09-17T07:21:15.351926+0000","last_active":"2024-09-17T07:22:20.508916+0000","last_peered":"2024-09-17T07:22:20.508916+0000","last_clean":"2024-09-17T07:22:20.508916+0000","last_became_active":"2024-09-17T07:21:15.351524+0000","last_became_peered":"2024-09-17T07:21:15.351524+0000","last_unstale":"2024-09-17T07:22:20.508916+0000","last_undegraded":"2024-09-17T07:22:20.508916+0000","last_fullsized":"2024-09-17T07:22:20.508916+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:50.514525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:13:50.514525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.d","version":"440'14609","reported_seq":14668,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-17T07:22:33.413366+0000","last_change":"2024-09-17T07:19:55.480674+0000","last_active":"2024-09-17T07:22:33.413366+0000","last_peered":"2024-09-17T07:22:33.413366+0000","last_clean":"2024-09-17T07:22:33.413366+0000","last_became_active":"2024-09-17T07:19:55.480318+0000","last_became_peered":"2024-09-17T07:19:55.480318+0000","last_unstale":"2024-09-17T07:22:33.413366+0000","last_undegraded":"2024-09-17T07:22:33.413366+0000","last_fullsized":"2024-09-17T07:22:33.413366+0000","mapping_epoch":390,"log_start":"422'9982","ondisk_log_start":"422'9982","created":390,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_clean_scrub_stamp":"2024-09-17T07:19:54.447320+0000","objects_scrubbed":0,"log_size":4627,"log_dups_size":0,"ondisk_log_size":4627,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":449445912,"num_objects":6859,"num_object_clones":0,"num_object_copies":20577,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6859,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14609,"num_write_kb":686913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"212'130","reported_seq":591,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.514839+0000","last_change":"2024-09-17T07:22:16.368625+0000","last_active":"2024-09-17T07:22:20.514839+0000","last_peered":"2024-09-17T07:22:20.514839+0000","last_clean":"2024-09-17T07:22:20.514839+0000","last_became_active":"2024-09-17T07:22:16.368294+0000","last_became_peered":"2024-09-17T07:22:16.368294+0000","last_unstale":"2024-09-17T07:22:20.514839+0000","last_undegraded":"2024-09-17T07:22:20.514839+0000","last_fullsized":"2024-09-17T07:22:20.514839+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":3,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:19:19.991044+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:14:57.052505+0000","last_clean_scrub_stamp":"2024-09-17T07:19:19.991044+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021141889999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":28,"num_write":16,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":161009,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"439'35","reported_seq":366,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:22.080754+0000","last_change":"2024-09-17T07:21:15.358553+0000","last_active":"2024-09-17T07:22:22.080754+0000","last_peered":"2024-09-17T07:22:22.080754+0000","last_clean":"2024-09-17T07:22:22.080754+0000","last_became_active":"2024-09-17T07:21:15.358209+0000","last_became_peered":"2024-09-17T07:21:15.358209+0000","last_unstale":"2024-09-17T07:22:22.080754+0000","last_undegraded":"2024-09-17T07:22:22.080754+0000","last_fullsized":"2024-09-17T07:22:22.080754+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'25","last_scrub_stamp":"2024-09-17T07:14:50.794200+0000","last_deep_scrub":"110'15","last_deep_scrub_stamp":"2024-09-17T07:04:57.091400+0000","last_clean_scrub_stamp":"2024-09-17T07:14:50.794200+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0070558269999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":325,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.509766+0000","last_change":"2024-09-17T07:22:13.592639+0000","last_active":"2024-09-17T07:22:20.509766+0000","last_peered":"2024-09-17T07:22:20.509766+0000","last_clean":"2024-09-17T07:22:20.509766+0000","last_became_active":"2024-09-17T07:22:13.591372+0000","last_became_peered":"2024-09-17T07:22:13.591372+0000","last_unstale":"2024-09-17T07:22:20.509766+0000","last_undegraded":"2024-09-17T07:22:20.509766+0000","last_fullsized":"2024-09-17T07:22:20.509766+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:24.437342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:25.147878+0000","last_clean_scrub_stamp":"2024-09-17T07:20:24.437342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0024354329999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"89'9","reported_seq":726,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-17T07:22:24.390535+0000","last_change":"2024-09-17T07:21:14.471392+0000","last_active":"2024-09-17T07:22:24.390535+0000","last_peered":"2024-09-17T07:22:24.390535+0000","last_clean":"2024-09-17T07:22:24.390535+0000","last_became_active":"2024-09-17T07:02:16.021360+0000","last_became_peered":"2024-09-17T07:02:16.021360+0000","last_unstale":"2024-09-17T07:22:24.390535+0000","last_undegraded":"2024-09-17T07:22:24.390535+0000","last_fullsized":"2024-09-17T07:22:24.390535+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'9","last_scrub_stamp":"2024-09-17T07:21:14.471255+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-09-17T07:15:53.512777+0000","last_clean_scrub_stamp":"2024-09-17T07:21:14.471255+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032915969999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"268'2","reported_seq":477,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-17T07:22:24.392968+0000","last_change":"2024-09-17T07:18:49.469622+0000","last_active":"2024-09-17T07:22:24.392968+0000","last_peered":"2024-09-17T07:22:24.392968+0000","last_clean":"2024-09-17T07:22:24.392968+0000","last_became_active":"2024-09-17T07:09:24.399565+0000","last_became_peered":"2024-09-17T07:09:24.399565+0000","last_unstale":"2024-09-17T07:22:24.392968+0000","last_undegraded":"2024-09-17T07:22:24.392968+0000","last_fullsized":"2024-09-17T07:22:24.392968+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'2","last_scrub_stamp":"2024-09-17T07:18:49.469494+0000","last_deep_scrub":"268'2","last_deep_scrub_stamp":"2024-09-17T07:14:03.609131+0000","last_clean_scrub_stamp":"2024-09-17T07:18:49.469494+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037058479999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"440'14212","reported_seq":20675,"reported_epoch":440,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-17T07:22:30.009780+0000","last_change":"2024-09-17T07:22:13.767904+0000","last_active":"2024-09-17T07:22:30.009780+0000","last_peered":"2024-09-17T07:22:30.009780+0000","last_clean":"2024-09-17T07:19:54.447320+0000","last_became_active":"2024-09-17T07:21:16.493516+0000","last_became_peered":"2024-09-17T07:21:16.493516+0000","last_unstale":"2024-09-17T07:22:30.009780+0000","last_undegraded":"2024-09-17T07:21:16.369606+0000","last_fullsized":"2024-09-17T07:21:16.338987+0000","mapping_epoch":414,"log_start":"421'9600","ondisk_log_start":"421'9600","created":390,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_clean_scrub_stamp":"2024-09-17T07:19:54.447320+0000","objects_scrubbed":0,"log_size":4612,"log_dups_size":0,"ondisk_log_size":4612,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":455081984,"num_objects":6944,"num_object_clones":0,"num_object_copies":20832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14212,"num_write_kb":676992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3194,"num_bytes_recovered":209321984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0,4","objects":6944}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"212'130","reported_seq":533,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.511022+0000","last_change":"2024-09-17T07:20:22.674470+0000","last_active":"2024-09-17T07:22:20.511022+0000","last_peered":"2024-09-17T07:22:20.511022+0000","last_clean":"2024-09-17T07:22:20.511022+0000","last_became_active":"2024-09-17T07:19:35.422795+0000","last_became_peered":"2024-09-17T07:19:35.422795+0000","last_unstale":"2024-09-17T07:22:20.511022+0000","last_undegraded":"2024-09-17T07:22:20.511022+0000","last_fullsized":"2024-09-17T07:22:20.511022+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":4,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:20:22.674350+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:14:57.052505+0000","last_clean_scrub_stamp":"2024-09-17T07:20:22.674350+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033559660000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"439'55","reported_seq":601,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-17T07:22:24.393369+0000","last_change":"2024-09-17T07:21:15.352832+0000","last_active":"2024-09-17T07:22:24.393369+0000","last_peered":"2024-09-17T07:22:24.393369+0000","last_clean":"2024-09-17T07:22:24.393369+0000","last_became_active":"2024-09-17T07:21:15.352461+0000","last_became_peered":"2024-09-17T07:21:15.352461+0000","last_unstale":"2024-09-17T07:22:24.393369+0000","last_undegraded":"2024-09-17T07:22:24.393369+0000","last_fullsized":"2024-09-17T07:22:24.393369+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'41","last_scrub_stamp":"2024-09-17T07:14:56.700858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:14:56.700858+0000","objects_scrubbed":10,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026058610000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":83,"num_read_kb":68,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":471,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.512290+0000","last_change":"2024-09-17T07:21:17.336331+0000","last_active":"2024-09-17T07:22:20.512290+0000","last_peered":"2024-09-17T07:22:20.512290+0000","last_clean":"2024-09-17T07:22:20.512290+0000","last_became_active":"2024-09-17T07:21:15.351163+0000","last_became_peered":"2024-09-17T07:21:15.351163+0000","last_unstale":"2024-09-17T07:22:20.512290+0000","last_undegraded":"2024-09-17T07:22:20.512290+0000","last_fullsized":"2024-09-17T07:22:20.512290+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:03.045709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:14:03.045709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0049245929999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":467,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.512764+0000","last_change":"2024-09-17T07:21:15.347484+0000","last_active":"2024-09-17T07:22:20.512764+0000","last_peered":"2024-09-17T07:22:20.512764+0000","last_clean":"2024-09-17T07:22:20.512764+0000","last_became_active":"2024-09-17T07:21:15.347069+0000","last_became_peered":"2024-09-17T07:21:15.347069+0000","last_unstale":"2024-09-17T07:22:20.512764+0000","last_undegraded":"2024-09-17T07:22:20.512764+0000","last_fullsized":"2024-09-17T07:22:20.512764+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:36.523676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:15:36.523676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017370549999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":356,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-17T07:22:24.392912+0000","last_change":"2024-09-17T07:22:13.590139+0000","last_active":"2024-09-17T07:22:24.392912+0000","last_peered":"2024-09-17T07:22:24.392912+0000","last_clean":"2024-09-17T07:22:24.392912+0000","last_became_active":"2024-09-17T07:22:13.589790+0000","last_became_peered":"2024-09-17T07:22:13.589790+0000","last_unstale":"2024-09-17T07:22:24.392912+0000","last_undegraded":"2024-09-17T07:22:24.392912+0000","last_fullsized":"2024-09-17T07:22:24.392912+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:20.067928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:13:53.406661+0000","last_clean_scrub_stamp":"2024-09-17T07:20:20.067928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0042634170000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.8","version":"440'14505","reported_seq":14565,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-17T07:22:29.097004+0000","last_change":"2024-09-17T07:19:55.484156+0000","last_active":"2024-09-17T07:22:29.097004+0000","last_peered":"2024-09-17T07:22:29.097004+0000","last_clean":"2024-09-17T07:22:29.097004+0000","last_became_active":"2024-09-17T07:19:55.483829+0000","last_became_peered":"2024-09-17T07:19:55.483829+0000","last_unstale":"2024-09-17T07:22:29.097004+0000","last_undegraded":"2024-09-17T07:22:29.097004+0000","last_fullsized":"2024-09-17T07:22:29.097004+0000","mapping_epoch":390,"log_start":"422'10200","ondisk_log_start":"422'10200","created":390,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_clean_scrub_stamp":"2024-09-17T07:19:54.447320+0000","objects_scrubbed":0,"log_size":4305,"log_dups_size":0,"ondisk_log_size":4305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":471269376,"num_objects":7191,"num_object_clones":0,"num_object_copies":21573,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14505,"num_write_kb":694272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.2","version":"212'130","reported_seq":529,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.510519+0000","last_change":"2024-09-17T07:22:16.360792+0000","last_active":"2024-09-17T07:22:20.510519+0000","last_peered":"2024-09-17T07:22:20.510519+0000","last_clean":"2024-09-17T07:22:20.510519+0000","last_became_active":"2024-09-17T07:22:16.360258+0000","last_became_peered":"2024-09-17T07:22:16.360258+0000","last_unstale":"2024-09-17T07:22:20.510519+0000","last_undegraded":"2024-09-17T07:22:20.510519+0000","last_fullsized":"2024-09-17T07:22:20.510519+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":4,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:19:19.991044+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:14:57.052505+0000","last_clean_scrub_stamp":"2024-09-17T07:19:19.991044+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021141889999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":28,"num_write":17,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":161010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"439'21","reported_seq":342,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:21.481061+0000","last_change":"2024-09-17T07:21:15.351304+0000","last_active":"2024-09-17T07:22:21.481061+0000","last_peered":"2024-09-17T07:22:21.481061+0000","last_clean":"2024-09-17T07:22:21.481061+0000","last_became_active":"2024-09-17T07:21:15.350162+0000","last_became_peered":"2024-09-17T07:21:15.350162+0000","last_unstale":"2024-09-17T07:22:21.481061+0000","last_undegraded":"2024-09-17T07:22:21.481061+0000","last_fullsized":"2024-09-17T07:22:21.481061+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'17","last_scrub_stamp":"2024-09-17T07:13:47.468399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:13:47.468399+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.005881725,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":30,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"76'5","reported_seq":1208,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-17T07:22:27.515636+0000","last_change":"2024-09-17T07:21:15.358093+0000","last_active":"2024-09-17T07:22:27.515636+0000","last_peered":"2024-09-17T07:22:27.515636+0000","last_clean":"2024-09-17T07:22:27.515636+0000","last_became_active":"2024-09-17T07:21:15.357178+0000","last_became_peered":"2024-09-17T07:21:15.357178+0000","last_unstale":"2024-09-17T07:22:27.515636+0000","last_undegraded":"2024-09-17T07:22:27.515636+0000","last_fullsized":"2024-09-17T07:22:27.515636+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'5","last_scrub_stamp":"2024-09-17T07:15:36.901601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:15:36.901601+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033739799999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":80,"num_read_kb":75,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":312,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.517331+0000","last_change":"2024-09-17T07:21:15.346533+0000","last_active":"2024-09-17T07:22:20.517331+0000","last_peered":"2024-09-17T07:22:20.517331+0000","last_clean":"2024-09-17T07:22:20.517331+0000","last_became_active":"2024-09-17T07:21:15.346287+0000","last_became_peered":"2024-09-17T07:21:15.346287+0000","last_unstale":"2024-09-17T07:22:20.517331+0000","last_undegraded":"2024-09-17T07:22:20.517331+0000","last_fullsized":"2024-09-17T07:22:20.517331+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:56.534547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:23.833170+0000","last_clean_scrub_stamp":"2024-09-17T07:15:56.534547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0011018950000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":515,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.523735+0000","last_change":"2024-09-17T07:22:13.591553+0000","last_active":"2024-09-17T07:22:20.523735+0000","last_peered":"2024-09-17T07:22:20.523735+0000","last_clean":"2024-09-17T07:22:20.523735+0000","last_became_active":"2024-09-17T07:22:13.590853+0000","last_became_peered":"2024-09-17T07:22:13.590853+0000","last_unstale":"2024-09-17T07:22:20.523735+0000","last_undegraded":"2024-09-17T07:22:20.523735+0000","last_fullsized":"2024-09-17T07:22:20.523735+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:16.776553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:18:42.051667+0000","last_clean_scrub_stamp":"2024-09-17T07:20:16.776553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035831560000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":17,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.508637+0000","last_change":"2024-09-17T07:22:13.629124+0000","last_active":"2024-09-17T07:22:20.508637+0000","last_peered":"2024-09-17T07:22:20.508637+0000","last_clean":"2024-09-17T07:22:20.508637+0000","last_became_active":"2024-09-17T07:22:13.628359+0000","last_became_peered":"2024-09-17T07:22:13.628359+0000","last_unstale":"2024-09-17T07:22:20.508637+0000","last_undegraded":"2024-09-17T07:22:20.508637+0000","last_fullsized":"2024-09-17T07:22:20.508637+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:27.909402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:02.062618+0000","last_clean_scrub_stamp":"2024-09-17T07:19:27.909402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0041575259999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.9","version":"440'14466","reported_seq":33310,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-17T07:22:29.110662+0000","last_change":"2024-09-17T07:22:14.607922+0000","last_active":"2024-09-17T07:22:29.110662+0000","last_peered":"2024-09-17T07:22:29.110662+0000","last_clean":"2024-09-17T07:22:29.110662+0000","last_became_active":"2024-09-17T07:22:14.579277+0000","last_became_peered":"2024-09-17T07:22:14.579277+0000","last_unstale":"2024-09-17T07:22:29.110662+0000","last_undegraded":"2024-09-17T07:22:29.110662+0000","last_fullsized":"2024-09-17T07:22:29.110662+0000","mapping_epoch":434,"log_start":"422'10200","ondisk_log_start":"422'10200","created":390,"last_epoch_clean":435,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_clean_scrub_stamp":"2024-09-17T07:19:54.447320+0000","objects_scrubbed":0,"log_size":4266,"log_dups_size":0,"ondisk_log_size":4266,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":463339520,"num_objects":7070,"num_object_clones":0,"num_object_copies":21210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7070,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14466,"num_write_kb":689152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9383,"num_bytes_recovered":614924288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.3","version":"212'130","reported_seq":553,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.512021+0000","last_change":"2024-09-17T07:21:18.358222+0000","last_active":"2024-09-17T07:22:20.512021+0000","last_peered":"2024-09-17T07:22:20.512021+0000","last_clean":"2024-09-17T07:22:20.512021+0000","last_became_active":"2024-09-17T07:21:18.357874+0000","last_became_peered":"2024-09-17T07:21:18.357874+0000","last_unstale":"2024-09-17T07:22:20.512021+0000","last_undegraded":"2024-09-17T07:22:20.512021+0000","last_fullsized":"2024-09-17T07:22:20.512021+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":3,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:20:34.113602+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:20:34.113602+0000","last_clean_scrub_stamp":"2024-09-17T07:20:34.113602+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021141889999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":28,"num_write":17,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":161010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"439'42","reported_seq":548,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:21.582633+0000","last_change":"2024-09-17T07:18:40.426710+0000","last_active":"2024-09-17T07:22:21.582633+0000","last_peered":"2024-09-17T07:22:21.582633+0000","last_clean":"2024-09-17T07:22:21.582633+0000","last_became_active":"2024-09-17T07:03:48.760405+0000","last_became_peered":"2024-09-17T07:03:48.760405+0000","last_unstale":"2024-09-17T07:22:21.582633+0000","last_undegraded":"2024-09-17T07:22:21.582633+0000","last_fullsized":"2024-09-17T07:22:21.582633+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'30","last_scrub_stamp":"2024-09-17T07:18:40.426597+0000","last_deep_scrub":"268'30","last_deep_scrub_stamp":"2024-09-17T07:14:00.599656+0000","last_clean_scrub_stamp":"2024-09-17T07:18:40.426597+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0031470769999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"70'1","reported_seq":486,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.517180+0000","last_change":"2024-09-17T07:19:19.477270+0000","last_active":"2024-09-17T07:22:20.517180+0000","last_peered":"2024-09-17T07:22:20.517180+0000","last_clean":"2024-09-17T07:22:20.517180+0000","last_became_active":"2024-09-17T07:09:24.402499+0000","last_became_peered":"2024-09-17T07:09:24.402499+0000","last_unstale":"2024-09-17T07:22:20.517180+0000","last_undegraded":"2024-09-17T07:22:20.517180+0000","last_fullsized":"2024-09-17T07:22:20.517180+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-17T07:19:19.477140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:19:19.477140+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011313337,"objects_trimmed":0,"snaptrim_duration":0.0032298629999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":17,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.506162+0000","last_change":"2024-09-17T07:22:13.736269+0000","last_active":"2024-09-17T07:22:20.506162+0000","last_peered":"2024-09-17T07:22:20.506162+0000","last_clean":"2024-09-17T07:22:20.506162+0000","last_became_active":"2024-09-17T07:22:13.710211+0000","last_became_peered":"2024-09-17T07:22:13.710211+0000","last_unstale":"2024-09-17T07:22:20.506162+0000","last_undegraded":"2024-09-17T07:22:20.506162+0000","last_fullsized":"2024-09-17T07:22:20.506162+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:34.449627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:55.321202+0000","last_clean_scrub_stamp":"2024-09-17T07:20:34.449627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0044019280000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":466,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.512024+0000","last_change":"2024-09-17T07:21:15.349885+0000","last_active":"2024-09-17T07:22:20.512024+0000","last_peered":"2024-09-17T07:22:20.512024+0000","last_clean":"2024-09-17T07:22:20.512024+0000","last_became_active":"2024-09-17T07:21:15.349572+0000","last_became_peered":"2024-09-17T07:21:15.349572+0000","last_unstale":"2024-09-17T07:22:20.512024+0000","last_undegraded":"2024-09-17T07:22:20.512024+0000","last_fullsized":"2024-09-17T07:22:20.512024+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:45.447029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:15:45.447029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001990463,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":337,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.510091+0000","last_change":"2024-09-17T07:21:15.351211+0000","last_active":"2024-09-17T07:22:20.510091+0000","last_peered":"2024-09-17T07:22:20.510091+0000","last_clean":"2024-09-17T07:22:20.510091+0000","last_became_active":"2024-09-17T07:21:15.350190+0000","last_became_peered":"2024-09-17T07:21:15.350190+0000","last_unstale":"2024-09-17T07:22:20.510091+0000","last_undegraded":"2024-09-17T07:22:20.510091+0000","last_fullsized":"2024-09-17T07:22:20.510091+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:51.877367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:15:51.877367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.058290885000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"440'14266","reported_seq":14419,"reported_epoch":440,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2024-09-17T07:22:29.058360+0000","last_change":"2024-09-17T07:22:13.594527+0000","last_active":"2024-09-17T07:22:29.058360+0000","last_peered":"2024-09-17T07:22:29.058360+0000","last_clean":"2024-09-17T07:19:54.447320+0000","last_became_active":"2024-09-17T07:21:16.466454+0000","last_became_peered":"2024-09-17T07:21:16.466454+0000","last_unstale":"2024-09-17T07:22:29.058360+0000","last_undegraded":"2024-09-17T07:21:16.479700+0000","last_fullsized":"2024-09-17T07:21:16.335962+0000","mapping_epoch":414,"log_start":"422'10000","ondisk_log_start":"422'10000","created":390,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_clean_scrub_stamp":"2024-09-17T07:19:54.447320+0000","objects_scrubbed":0,"log_size":4266,"log_dups_size":0,"ondisk_log_size":4266,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":465305600,"num_objects":7100,"num_object_clones":0,"num_object_copies":21300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7071,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7100,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14266,"num_write_kb":683712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":2359296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[{"shards":"5,6","objects":7100}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"439'38","reported_seq":585,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:22.124862+0000","last_change":"2024-09-17T07:22:13.636910+0000","last_active":"2024-09-17T07:22:22.124862+0000","last_peered":"2024-09-17T07:22:22.124862+0000","last_clean":"2024-09-17T07:22:22.124862+0000","last_became_active":"2024-09-17T07:22:13.583204+0000","last_became_peered":"2024-09-17T07:22:13.583204+0000","last_unstale":"2024-09-17T07:22:22.124862+0000","last_undegraded":"2024-09-17T07:22:22.124862+0000","last_fullsized":"2024-09-17T07:22:22.124862+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'26","last_scrub_stamp":"2024-09-17T07:20:19.046676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:20:19.046676+0000","objects_scrubbed":7,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0059907520000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"268'3","reported_seq":337,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.512512+0000","last_change":"2024-09-17T07:19:29.367387+0000","last_active":"2024-09-17T07:22:20.512512+0000","last_peered":"2024-09-17T07:22:20.512512+0000","last_clean":"2024-09-17T07:22:20.512512+0000","last_became_active":"2024-09-17T07:09:24.404532+0000","last_became_peered":"2024-09-17T07:09:24.404532+0000","last_unstale":"2024-09-17T07:22:20.512512+0000","last_undegraded":"2024-09-17T07:22:20.512512+0000","last_fullsized":"2024-09-17T07:22:20.512512+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'3","last_scrub_stamp":"2024-09-17T07:19:29.367269+0000","last_deep_scrub":"268'3","last_deep_scrub_stamp":"2024-09-17T07:13:54.949345+0000","last_clean_scrub_stamp":"2024-09-17T07:19:29.367269+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023388250000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":506,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.511817+0000","last_change":"2024-09-17T07:22:13.561814+0000","last_active":"2024-09-17T07:22:20.511817+0000","last_peered":"2024-09-17T07:22:20.511817+0000","last_clean":"2024-09-17T07:22:20.511817+0000","last_became_active":"2024-09-17T07:22:13.560809+0000","last_became_peered":"2024-09-17T07:22:13.560809+0000","last_unstale":"2024-09-17T07:22:20.511817+0000","last_undegraded":"2024-09-17T07:22:20.511817+0000","last_fullsized":"2024-09-17T07:22:20.511817+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:17.650824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:20:17.650824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019547470000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"212'130","reported_seq":539,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.507593+0000","last_change":"2024-09-17T07:22:16.366636+0000","last_active":"2024-09-17T07:22:20.507593+0000","last_peered":"2024-09-17T07:22:20.507593+0000","last_clean":"2024-09-17T07:22:20.507593+0000","last_became_active":"2024-09-17T07:22:16.366295+0000","last_became_peered":"2024-09-17T07:22:16.366295+0000","last_unstale":"2024-09-17T07:22:20.507593+0000","last_undegraded":"2024-09-17T07:22:20.507593+0000","last_fullsized":"2024-09-17T07:22:20.507593+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:19:19.991044+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:14:57.052505+0000","last_clean_scrub_stamp":"2024-09-17T07:19:19.991044+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021141889999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":27,"num_write":16,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":161009,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":17,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.508208+0000","last_change":"2024-09-17T07:22:13.697511+0000","last_active":"2024-09-17T07:22:20.508208+0000","last_peered":"2024-09-17T07:22:20.508208+0000","last_clean":"2024-09-17T07:22:20.508208+0000","last_became_active":"2024-09-17T07:22:13.693460+0000","last_became_peered":"2024-09-17T07:22:13.693460+0000","last_unstale":"2024-09-17T07:22:20.508208+0000","last_undegraded":"2024-09-17T07:22:20.508208+0000","last_fullsized":"2024-09-17T07:22:20.508208+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:18:48.891136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:18:48.891136+0000","last_clean_scrub_stamp":"2024-09-17T07:18:48.891136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":463,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.512123+0000","last_change":"2024-09-17T07:22:19.508824+0000","last_active":"2024-09-17T07:22:20.512123+0000","last_peered":"2024-09-17T07:22:20.512123+0000","last_clean":"2024-09-17T07:22:20.512123+0000","last_became_active":"2024-09-17T07:21:15.344632+0000","last_became_peered":"2024-09-17T07:21:15.344632+0000","last_unstale":"2024-09-17T07:22:20.512123+0000","last_undegraded":"2024-09-17T07:22:20.512123+0000","last_fullsized":"2024-09-17T07:22:20.512123+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:48.021592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:58.670571+0000","last_clean_scrub_stamp":"2024-09-17T07:14:48.021592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0064844619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.5","version":"440'13979","reported_seq":14045,"reported_epoch":440,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-17T07:22:29.948934+0000","last_change":"2024-09-17T07:22:14.677531+0000","last_active":"2024-09-17T07:22:29.948934+0000","last_peered":"2024-09-17T07:22:29.948934+0000","last_clean":"2024-09-17T07:22:12.495976+0000","last_became_active":"2024-09-17T07:22:14.648525+0000","last_became_peered":"2024-09-17T07:22:14.648525+0000","last_unstale":"2024-09-17T07:22:29.948934+0000","last_undegraded":"2024-09-17T07:22:29.948934+0000","last_fullsized":"2024-09-17T07:22:29.948934+0000","mapping_epoch":434,"log_start":"421'9350","ondisk_log_start":"421'9350","created":390,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_clean_scrub_stamp":"2024-09-17T07:19:54.447320+0000","objects_scrubbed":0,"log_size":4629,"log_dups_size":0,"ondisk_log_size":4629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":454885376,"num_objects":6941,"num_object_clones":0,"num_object_copies":20823,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6941,"num_objects_unfound":0,"num_objects_dirty":6941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13979,"num_write_kb":669440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[0,5,6],"avail_no_missing":["0","5","6"],"object_location_counts":[{"shards":"0,5,6","objects":6941}],"blocked_by":[],"up_primary":7,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"439'51","reported_seq":630,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:22.286084+0000","last_change":"2024-09-17T07:22:13.759654+0000","last_active":"2024-09-17T07:22:22.286084+0000","last_peered":"2024-09-17T07:22:22.286084+0000","last_clean":"2024-09-17T07:22:22.286084+0000","last_became_active":"2024-09-17T07:22:13.588260+0000","last_became_peered":"2024-09-17T07:22:13.588260+0000","last_unstale":"2024-09-17T07:22:22.286084+0000","last_undegraded":"2024-09-17T07:22:22.286084+0000","last_fullsized":"2024-09-17T07:22:22.286084+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'35","last_scrub_stamp":"2024-09-17T07:20:15.012345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:20:15.012345+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0050499799999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":526,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-17T07:22:24.393614+0000","last_change":"2024-09-17T07:22:14.572711+0000","last_active":"2024-09-17T07:22:24.393614+0000","last_peered":"2024-09-17T07:22:24.393614+0000","last_clean":"2024-09-17T07:22:24.393614+0000","last_became_active":"2024-09-17T07:02:12.012037+0000","last_became_peered":"2024-09-17T07:02:12.012037+0000","last_unstale":"2024-09-17T07:22:24.393614+0000","last_undegraded":"2024-09-17T07:22:24.393614+0000","last_fullsized":"2024-09-17T07:22:24.393614+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:17.749290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:06:39.322535+0000","last_clean_scrub_stamp":"2024-09-17T07:20:17.749290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0062185959999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":509,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-17T07:22:24.391070+0000","last_change":"2024-09-17T07:22:16.344662+0000","last_active":"2024-09-17T07:22:24.391070+0000","last_peered":"2024-09-17T07:22:24.391070+0000","last_clean":"2024-09-17T07:22:24.391070+0000","last_became_active":"2024-09-17T07:02:16.014631+0000","last_became_peered":"2024-09-17T07:02:16.014631+0000","last_unstale":"2024-09-17T07:22:24.391070+0000","last_undegraded":"2024-09-17T07:22:24.391070+0000","last_fullsized":"2024-09-17T07:22:24.391070+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:12.821593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:20:12.821593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0031314899999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":17,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.507668+0000","last_change":"2024-09-17T07:22:13.573689+0000","last_active":"2024-09-17T07:22:20.507668+0000","last_peered":"2024-09-17T07:22:20.507668+0000","last_clean":"2024-09-17T07:22:20.507668+0000","last_became_active":"2024-09-17T07:22:13.573161+0000","last_became_peered":"2024-09-17T07:22:13.573161+0000","last_unstale":"2024-09-17T07:22:20.507668+0000","last_undegraded":"2024-09-17T07:22:20.507668+0000","last_fullsized":"2024-09-17T07:22:20.507668+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:18:47.880943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:45.275473+0000","last_clean_scrub_stamp":"2024-09-17T07:18:47.880943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010463343,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.4","version":"440'14459","reported_seq":35863,"reported_epoch":440,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-17T07:22:29.114983+0000","last_change":"2024-09-17T07:22:13.785814+0000","last_active":"2024-09-17T07:22:29.114983+0000","last_peered":"2024-09-17T07:22:29.114983+0000","last_clean":"2024-09-17T07:19:54.447320+0000","last_became_active":"2024-09-17T07:22:13.696978+0000","last_became_peered":"2024-09-17T07:22:13.696978+0000","last_unstale":"2024-09-17T07:22:29.114983+0000","last_undegraded":"2024-09-17T07:22:13.784707+0000","last_fullsized":"2024-09-17T07:22:13.553874+0000","mapping_epoch":433,"log_start":"422'10200","ondisk_log_start":"422'10200","created":390,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_clean_scrub_stamp":"2024-09-17T07:19:54.447320+0000","objects_scrubbed":0,"log_size":4259,"log_dups_size":0,"ondisk_log_size":4259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":472055808,"num_objects":7203,"num_object_clones":0,"num_object_copies":21609,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7203,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14459,"num_write_kb":693184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10662,"num_bytes_recovered":698679296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[6,2],"avail_no_missing":["6","1","2"],"object_location_counts":[{"shards":"2,6","objects":7203}],"blocked_by":[],"up_primary":7,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"439'25","reported_seq":496,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-17T07:22:24.391914+0000","last_change":"2024-09-17T07:21:15.349844+0000","last_active":"2024-09-17T07:22:24.391914+0000","last_peered":"2024-09-17T07:22:24.391914+0000","last_clean":"2024-09-17T07:22:24.391914+0000","last_became_active":"2024-09-17T07:21:15.349361+0000","last_became_peered":"2024-09-17T07:21:15.349361+0000","last_unstale":"2024-09-17T07:22:24.391914+0000","last_undegraded":"2024-09-17T07:22:24.391914+0000","last_fullsized":"2024-09-17T07:22:24.391914+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'19","last_scrub_stamp":"2024-09-17T07:14:06.634334+0000","last_deep_scrub":"268'19","last_deep_scrub_stamp":"2024-09-17T07:14:06.634334+0000","last_clean_scrub_stamp":"2024-09-17T07:14:06.634334+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0088943289999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":354,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.510444+0000","last_change":"2024-09-17T07:21:15.352018+0000","last_active":"2024-09-17T07:22:20.510444+0000","last_peered":"2024-09-17T07:22:20.510444+0000","last_clean":"2024-09-17T07:22:20.510444+0000","last_became_active":"2024-09-17T07:21:15.350786+0000","last_became_peered":"2024-09-17T07:21:15.350786+0000","last_unstale":"2024-09-17T07:22:20.510444+0000","last_undegraded":"2024-09-17T07:22:20.510444+0000","last_fullsized":"2024-09-17T07:22:20.510444+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:58.886294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:13:53.593078+0000","last_clean_scrub_stamp":"2024-09-17T07:15:58.886294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.005853245,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":438,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.518105+0000","last_change":"2024-09-17T07:19:30.094643+0000","last_active":"2024-09-17T07:22:20.518105+0000","last_peered":"2024-09-17T07:22:20.518105+0000","last_clean":"2024-09-17T07:22:20.518105+0000","last_became_active":"2024-09-17T07:09:24.396642+0000","last_became_peered":"2024-09-17T07:09:24.396642+0000","last_unstale":"2024-09-17T07:22:20.518105+0000","last_undegraded":"2024-09-17T07:22:20.518105+0000","last_fullsized":"2024-09-17T07:22:20.518105+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:30.094567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:19:30.094567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001242611,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":479,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.517260+0000","last_change":"2024-09-17T07:22:13.589795+0000","last_active":"2024-09-17T07:22:20.517260+0000","last_peered":"2024-09-17T07:22:20.517260+0000","last_clean":"2024-09-17T07:22:20.517260+0000","last_became_active":"2024-09-17T07:22:13.589460+0000","last_became_peered":"2024-09-17T07:22:13.589460+0000","last_unstale":"2024-09-17T07:22:20.517260+0000","last_undegraded":"2024-09-17T07:22:20.517260+0000","last_fullsized":"2024-09-17T07:22:20.517260+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:29.434010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:58.099849+0000","last_clean_scrub_stamp":"2024-09-17T07:19:29.434010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.032605186000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.7","version":"440'14385","reported_seq":31958,"reported_epoch":440,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2024-09-17T07:22:29.141475+0000","last_change":"2024-09-17T07:22:13.595584+0000","last_active":"2024-09-17T07:22:29.141475+0000","last_peered":"2024-09-17T07:22:29.141475+0000","last_clean":"2024-09-17T07:19:54.447320+0000","last_became_active":"2024-09-17T07:21:16.495288+0000","last_became_peered":"2024-09-17T07:21:16.495288+0000","last_unstale":"2024-09-17T07:22:29.141475+0000","last_undegraded":"2024-09-17T07:21:16.348713+0000","last_fullsized":"2024-09-17T07:21:16.338417+0000","mapping_epoch":414,"log_start":"422'9884","ondisk_log_start":"422'9884","created":390,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_clean_scrub_stamp":"2024-09-17T07:19:54.447320+0000","objects_scrubbed":0,"log_size":4501,"log_dups_size":0,"ondisk_log_size":4501,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":468647936,"num_objects":7151,"num_object_clones":0,"num_object_copies":21453,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":515,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14385,"num_write_kb":689152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8751,"num_bytes_recovered":573505536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[{"shards":"4,5","objects":7151}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"439'46","reported_seq":593,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:21.665291+0000","last_change":"2024-09-17T07:22:13.608405+0000","last_active":"2024-09-17T07:22:21.665291+0000","last_peered":"2024-09-17T07:22:21.665291+0000","last_clean":"2024-09-17T07:22:21.665291+0000","last_became_active":"2024-09-17T07:22:13.578616+0000","last_became_peered":"2024-09-17T07:22:13.578616+0000","last_unstale":"2024-09-17T07:22:21.665291+0000","last_undegraded":"2024-09-17T07:22:21.665291+0000","last_fullsized":"2024-09-17T07:22:21.665291+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'34","last_scrub_stamp":"2024-09-17T07:19:24.308050+0000","last_deep_scrub":"242'34","last_deep_scrub_stamp":"2024-09-17T07:14:50.103539+0000","last_clean_scrub_stamp":"2024-09-17T07:19:24.308050+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.019153434,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'2","reported_seq":21,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.507864+0000","last_change":"2024-09-17T07:22:13.709212+0000","last_active":"2024-09-17T07:22:20.507864+0000","last_peered":"2024-09-17T07:22:20.507864+0000","last_clean":"2024-09-17T07:22:20.507864+0000","last_became_active":"2024-09-17T07:22:13.575469+0000","last_became_peered":"2024-09-17T07:22:13.575469+0000","last_unstale":"2024-09-17T07:22:20.507864+0000","last_undegraded":"2024-09-17T07:22:20.507864+0000","last_fullsized":"2024-09-17T07:22:20.507864+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-09-17T07:18:43.861842+0000","last_deep_scrub":"89'2","last_deep_scrub_stamp":"2024-09-17T07:12:20.301263+0000","last_clean_scrub_stamp":"2024-09-17T07:18:43.861842+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":420,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":420,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"436'14","reported_seq":25,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-17T07:22:25.766520+0000","last_change":"2024-09-17T07:22:13.736258+0000","last_active":"2024-09-17T07:22:25.766520+0000","last_peered":"2024-09-17T07:22:25.766520+0000","last_clean":"2024-09-17T07:22:25.766520+0000","last_became_active":"2024-09-17T07:22:13.624050+0000","last_became_peered":"2024-09-17T07:22:13.624050+0000","last_unstale":"2024-09-17T07:22:25.766520+0000","last_undegraded":"2024-09-17T07:22:25.766520+0000","last_fullsized":"2024-09-17T07:22:25.766520+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"225'9","last_scrub_stamp":"2024-09-17T07:18:23.788769+0000","last_deep_scrub":"225'9","last_deep_scrub_stamp":"2024-09-17T07:11:25.292565+0000","last_clean_scrub_stamp":"2024-09-17T07:18:23.788769+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023108299999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":471,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.512669+0000","last_change":"2024-09-17T07:18:41.386243+0000","last_active":"2024-09-17T07:22:20.512669+0000","last_peered":"2024-09-17T07:22:20.512669+0000","last_clean":"2024-09-17T07:22:20.512669+0000","last_became_active":"2024-09-17T07:02:18.363253+0000","last_became_peered":"2024-09-17T07:02:18.363253+0000","last_unstale":"2024-09-17T07:22:20.512669+0000","last_undegraded":"2024-09-17T07:22:20.512669+0000","last_fullsized":"2024-09-17T07:22:20.512669+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:18:41.385928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:18:41.385928+0000","last_clean_scrub_stamp":"2024-09-17T07:18:41.385928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023027270000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.6","version":"440'14434","reported_seq":14867,"reported_epoch":440,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2024-09-17T07:22:29.994599+0000","last_change":"2024-09-17T07:22:12.568610+0000","last_active":"2024-09-17T07:22:29.994599+0000","last_peered":"2024-09-17T07:22:29.994599+0000","last_clean":"2024-09-17T07:19:54.447320+0000","last_became_active":"2024-09-17T07:21:16.492393+0000","last_became_peered":"2024-09-17T07:21:16.492393+0000","last_unstale":"2024-09-17T07:22:29.994599+0000","last_undegraded":"2024-09-17T07:21:16.378286+0000","last_fullsized":"2024-09-17T07:21:16.351432+0000","mapping_epoch":414,"log_start":"421'9800","ondisk_log_start":"421'9800","created":390,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_clean_scrub_stamp":"2024-09-17T07:19:54.447320+0000","objects_scrubbed":0,"log_size":4634,"log_dups_size":0,"ondisk_log_size":4634,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":463601664,"num_objects":7074,"num_object_clones":0,"num_object_copies":21222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6919,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7074,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14434,"num_write_kb":688256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":181,"num_bytes_recovered":11862016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":7074}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"439'29","reported_seq":565,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.526223+0000","last_change":"2024-09-17T07:20:33.108656+0000","last_active":"2024-09-17T07:22:20.526223+0000","last_peered":"2024-09-17T07:22:20.526223+0000","last_clean":"2024-09-17T07:22:20.526223+0000","last_became_active":"2024-09-17T07:10:49.723292+0000","last_became_peered":"2024-09-17T07:10:49.723292+0000","last_unstale":"2024-09-17T07:22:20.526223+0000","last_undegraded":"2024-09-17T07:22:20.526223+0000","last_fullsized":"2024-09-17T07:22:20.526223+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'23","last_scrub_stamp":"2024-09-17T07:20:33.108490+0000","last_deep_scrub":"150'17","last_deep_scrub_stamp":"2024-09-17T07:06:38.775788+0000","last_clean_scrub_stamp":"2024-09-17T07:20:33.108490+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0087601510000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"68'2","reported_seq":516,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-17T07:22:24.394758+0000","last_change":"2024-09-17T07:22:13.766671+0000","last_active":"2024-09-17T07:22:24.394758+0000","last_peered":"2024-09-17T07:22:24.394758+0000","last_clean":"2024-09-17T07:22:24.394758+0000","last_became_active":"2024-09-17T07:22:13.590172+0000","last_became_peered":"2024-09-17T07:22:13.590172+0000","last_unstale":"2024-09-17T07:22:24.394758+0000","last_undegraded":"2024-09-17T07:22:24.394758+0000","last_fullsized":"2024-09-17T07:22:24.394758+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-09-17T07:20:11.093491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:20:11.093491+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023198989999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":431,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.514667+0000","last_change":"2024-09-17T07:21:15.356254+0000","last_active":"2024-09-17T07:22:20.514667+0000","last_peered":"2024-09-17T07:22:20.514667+0000","last_clean":"2024-09-17T07:22:20.514667+0000","last_became_active":"2024-09-17T07:21:15.355760+0000","last_became_peered":"2024-09-17T07:21:15.355760+0000","last_unstale":"2024-09-17T07:22:20.514667+0000","last_undegraded":"2024-09-17T07:22:20.514667+0000","last_fullsized":"2024-09-17T07:22:20.514667+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:51.586929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:11:17.479036+0000","last_clean_scrub_stamp":"2024-09-17T07:15:51.586929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001274435,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":17,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.506514+0000","last_change":"2024-09-17T07:22:13.585879+0000","last_active":"2024-09-17T07:22:20.506514+0000","last_peered":"2024-09-17T07:22:20.506514+0000","last_clean":"2024-09-17T07:22:20.506514+0000","last_became_active":"2024-09-17T07:22:13.585580+0000","last_became_peered":"2024-09-17T07:22:13.585580+0000","last_unstale":"2024-09-17T07:22:20.506514+0000","last_undegraded":"2024-09-17T07:22:20.506514+0000","last_fullsized":"2024-09-17T07:22:20.506514+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:07.959158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:20:07.959158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.018487173999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"439'21","reported_seq":555,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.602093+0000","last_change":"2024-09-17T07:22:13.738737+0000","last_active":"2024-09-17T07:22:20.602093+0000","last_peered":"2024-09-17T07:22:20.602093+0000","last_clean":"2024-09-17T07:22:20.602093+0000","last_became_active":"2024-09-17T07:22:13.592001+0000","last_became_peered":"2024-09-17T07:22:13.592001+0000","last_unstale":"2024-09-17T07:22:20.602093+0000","last_undegraded":"2024-09-17T07:22:20.602093+0000","last_fullsized":"2024-09-17T07:22:20.602093+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'15","last_scrub_stamp":"2024-09-17T07:18:24.505002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:18:24.505002+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002195809,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":504,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.519983+0000","last_change":"2024-09-17T07:21:15.788428+0000","last_active":"2024-09-17T07:22:20.519983+0000","last_peered":"2024-09-17T07:22:20.519983+0000","last_clean":"2024-09-17T07:22:20.519983+0000","last_became_active":"2024-09-17T07:21:15.355464+0000","last_became_peered":"2024-09-17T07:21:15.355464+0000","last_unstale":"2024-09-17T07:22:20.519983+0000","last_undegraded":"2024-09-17T07:22:20.519983+0000","last_fullsized":"2024-09-17T07:22:20.519983+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:21:15.788348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:17.976267+0000","last_clean_scrub_stamp":"2024-09-17T07:21:15.788348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0012829899999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":176,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.523736+0000","last_change":"2024-09-17T07:22:13.593743+0000","last_active":"2024-09-17T07:22:20.523736+0000","last_peered":"2024-09-17T07:22:20.523736+0000","last_clean":"2024-09-17T07:22:20.523736+0000","last_became_active":"2024-09-17T07:22:13.593446+0000","last_became_peered":"2024-09-17T07:22:13.593446+0000","last_unstale":"2024-09-17T07:22:20.523736+0000","last_undegraded":"2024-09-17T07:22:20.523736+0000","last_fullsized":"2024-09-17T07:22:20.523736+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:29.492000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:18.367486+0000","last_clean_scrub_stamp":"2024-09-17T07:20:29.492000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0077309099999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":554,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.522035+0000","last_change":"2024-09-17T07:22:13.591019+0000","last_active":"2024-09-17T07:22:20.522035+0000","last_peered":"2024-09-17T07:22:20.522035+0000","last_clean":"2024-09-17T07:22:20.522035+0000","last_became_active":"2024-09-17T07:22:13.590599+0000","last_became_peered":"2024-09-17T07:22:13.590599+0000","last_unstale":"2024-09-17T07:22:20.522035+0000","last_undegraded":"2024-09-17T07:22:20.522035+0000","last_fullsized":"2024-09-17T07:22:20.522035+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:21:11.767427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:09:23.378805+0000","last_clean_scrub_stamp":"2024-09-17T07:21:11.767427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:22:25.103786+0000","scrub_duration":0.012069135,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":481,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.514453+0000","last_change":"2024-09-17T07:21:15.350067+0000","last_active":"2024-09-17T07:22:20.514453+0000","last_peered":"2024-09-17T07:22:20.514453+0000","last_clean":"2024-09-17T07:22:20.514453+0000","last_became_active":"2024-09-17T07:21:15.348758+0000","last_became_peered":"2024-09-17T07:21:15.348758+0000","last_unstale":"2024-09-17T07:22:20.514453+0000","last_undegraded":"2024-09-17T07:22:20.514453+0000","last_fullsized":"2024-09-17T07:22:20.514453+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:46.645080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:46.645080+0000","last_clean_scrub_stamp":"2024-09-17T07:15:46.645080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015661049999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"439'18","reported_seq":34,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:21.808680+0000","last_change":"2024-09-17T07:22:13.761675+0000","last_active":"2024-09-17T07:22:21.808680+0000","last_peered":"2024-09-17T07:22:21.808680+0000","last_clean":"2024-09-17T07:22:21.808680+0000","last_became_active":"2024-09-17T07:22:13.586662+0000","last_became_peered":"2024-09-17T07:22:13.586662+0000","last_unstale":"2024-09-17T07:22:21.808680+0000","last_undegraded":"2024-09-17T07:22:21.808680+0000","last_fullsized":"2024-09-17T07:22:21.808680+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'12","last_scrub_stamp":"2024-09-17T07:20:17.090146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:20:17.090146+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019381719999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":25,"num_read_kb":22,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":474,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.511539+0000","last_change":"2024-09-17T07:21:15.342247+0000","last_active":"2024-09-17T07:22:20.511539+0000","last_peered":"2024-09-17T07:22:20.511539+0000","last_clean":"2024-09-17T07:22:20.511539+0000","last_became_active":"2024-09-17T07:21:15.341966+0000","last_became_peered":"2024-09-17T07:21:15.341966+0000","last_unstale":"2024-09-17T07:22:20.511539+0000","last_undegraded":"2024-09-17T07:22:20.511539+0000","last_fullsized":"2024-09-17T07:22:20.511539+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:55.191252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:14:55.191252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0090150360000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":17,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.508559+0000","last_change":"2024-09-17T07:22:13.624074+0000","last_active":"2024-09-17T07:22:20.508559+0000","last_peered":"2024-09-17T07:22:20.508559+0000","last_clean":"2024-09-17T07:22:20.508559+0000","last_became_active":"2024-09-17T07:22:13.623754+0000","last_became_peered":"2024-09-17T07:22:13.623754+0000","last_unstale":"2024-09-17T07:22:20.508559+0000","last_undegraded":"2024-09-17T07:22:20.508559+0000","last_fullsized":"2024-09-17T07:22:20.508559+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:25.934136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:19:25.934136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"439'30","reported_seq":235,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:22.066371+0000","last_change":"2024-09-17T07:22:13.768713+0000","last_active":"2024-09-17T07:22:22.066371+0000","last_peered":"2024-09-17T07:22:22.066371+0000","last_clean":"2024-09-17T07:22:22.066371+0000","last_became_active":"2024-09-17T07:22:13.591094+0000","last_became_peered":"2024-09-17T07:22:13.591094+0000","last_unstale":"2024-09-17T07:22:22.066371+0000","last_undegraded":"2024-09-17T07:22:22.066371+0000","last_fullsized":"2024-09-17T07:22:22.066371+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'20","last_scrub_stamp":"2024-09-17T07:20:10.554324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:20:10.554324+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0020069329999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":514,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.512359+0000","last_change":"2024-09-17T07:22:13.574496+0000","last_active":"2024-09-17T07:22:20.512359+0000","last_peered":"2024-09-17T07:22:20.512359+0000","last_clean":"2024-09-17T07:22:20.512359+0000","last_became_active":"2024-09-17T07:22:13.574217+0000","last_became_peered":"2024-09-17T07:22:13.574217+0000","last_unstale":"2024-09-17T07:22:20.512359+0000","last_undegraded":"2024-09-17T07:22:20.512359+0000","last_fullsized":"2024-09-17T07:22:20.512359+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:40.443474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:19:40.443474+0000","last_clean_scrub_stamp":"2024-09-17T07:19:40.443474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002799733,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"414'11","reported_seq":719,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-17T07:22:24.383845+0000","last_change":"2024-09-17T07:21:15.353878+0000","last_active":"2024-09-17T07:22:24.383845+0000","last_peered":"2024-09-17T07:22:24.383845+0000","last_clean":"2024-09-17T07:22:24.383845+0000","last_became_active":"2024-09-17T07:21:15.353461+0000","last_became_peered":"2024-09-17T07:21:15.353461+0000","last_unstale":"2024-09-17T07:22:24.383845+0000","last_undegraded":"2024-09-17T07:22:24.383845+0000","last_fullsized":"2024-09-17T07:22:24.383845+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'9","last_scrub_stamp":"2024-09-17T07:14:53.140133+0000","last_deep_scrub":"163'9","last_deep_scrub_stamp":"2024-09-17T07:07:34.933808+0000","last_clean_scrub_stamp":"2024-09-17T07:14:53.140133+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0088819989999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":487,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.514293+0000","last_change":"2024-09-17T07:22:13.591762+0000","last_active":"2024-09-17T07:22:20.514293+0000","last_peered":"2024-09-17T07:22:20.514293+0000","last_clean":"2024-09-17T07:22:20.514293+0000","last_became_active":"2024-09-17T07:22:13.591307+0000","last_became_peered":"2024-09-17T07:22:13.591307+0000","last_unstale":"2024-09-17T07:22:20.514293+0000","last_undegraded":"2024-09-17T07:22:20.514293+0000","last_fullsized":"2024-09-17T07:22:20.514293+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:18:47.392182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:26.619893+0000","last_clean_scrub_stamp":"2024-09-17T07:18:47.392182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0053055120000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"439'21","reported_seq":565,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:22.075151+0000","last_change":"2024-09-17T07:22:13.605086+0000","last_active":"2024-09-17T07:22:22.075151+0000","last_peered":"2024-09-17T07:22:22.075151+0000","last_clean":"2024-09-17T07:22:22.075151+0000","last_became_active":"2024-09-17T07:22:13.580673+0000","last_became_peered":"2024-09-17T07:22:13.580673+0000","last_unstale":"2024-09-17T07:22:22.075151+0000","last_undegraded":"2024-09-17T07:22:22.075151+0000","last_fullsized":"2024-09-17T07:22:22.075151+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'15","last_scrub_stamp":"2024-09-17T07:19:20.304874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:19:20.304874+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0074478349999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":497,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.511771+0000","last_change":"2024-09-17T07:22:13.592026+0000","last_active":"2024-09-17T07:22:20.511771+0000","last_peered":"2024-09-17T07:22:20.511771+0000","last_clean":"2024-09-17T07:22:20.511771+0000","last_became_active":"2024-09-17T07:22:13.589964+0000","last_became_peered":"2024-09-17T07:22:13.589964+0000","last_unstale":"2024-09-17T07:22:20.511771+0000","last_undegraded":"2024-09-17T07:22:20.511771+0000","last_fullsized":"2024-09-17T07:22:20.511771+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:33.544567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:19:33.544567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.13531478499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"436'15","reported_seq":25,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-17T07:22:25.768368+0000","last_change":"2024-09-17T07:22:13.736680+0000","last_active":"2024-09-17T07:22:25.768368+0000","last_peered":"2024-09-17T07:22:25.768368+0000","last_clean":"2024-09-17T07:22:25.768368+0000","last_became_active":"2024-09-17T07:22:13.586620+0000","last_became_peered":"2024-09-17T07:22:13.586620+0000","last_unstale":"2024-09-17T07:22:25.768368+0000","last_undegraded":"2024-09-17T07:22:25.768368+0000","last_fullsized":"2024-09-17T07:22:25.768368+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'12","last_scrub_stamp":"2024-09-17T07:20:09.105448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:20:09.105448+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":472,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.518487+0000","last_change":"2024-09-17T07:20:37.110604+0000","last_active":"2024-09-17T07:22:20.518487+0000","last_peered":"2024-09-17T07:22:20.518487+0000","last_clean":"2024-09-17T07:22:20.518487+0000","last_became_active":"2024-09-17T07:02:18.018716+0000","last_became_peered":"2024-09-17T07:02:18.018716+0000","last_unstale":"2024-09-17T07:22:20.518487+0000","last_undegraded":"2024-09-17T07:22:20.518487+0000","last_fullsized":"2024-09-17T07:22:20.518487+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:37.110458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:20:37.110458+0000","last_clean_scrub_stamp":"2024-09-17T07:20:37.110458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.017901013,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"439'21","reported_seq":336,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:21.255391+0000","last_change":"2024-09-17T07:21:15.349519+0000","last_active":"2024-09-17T07:22:21.255391+0000","last_peered":"2024-09-17T07:22:21.255391+0000","last_clean":"2024-09-17T07:22:21.255391+0000","last_became_active":"2024-09-17T07:21:15.348060+0000","last_became_peered":"2024-09-17T07:21:15.348060+0000","last_unstale":"2024-09-17T07:22:21.255391+0000","last_undegraded":"2024-09-17T07:22:21.255391+0000","last_fullsized":"2024-09-17T07:22:21.255391+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'15","last_scrub_stamp":"2024-09-17T07:13:48.500257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:13:48.500257+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.005053564,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":514,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.515050+0000","last_change":"2024-09-17T07:22:13.595359+0000","last_active":"2024-09-17T07:22:20.515050+0000","last_peered":"2024-09-17T07:22:20.515050+0000","last_clean":"2024-09-17T07:22:20.515050+0000","last_became_active":"2024-09-17T07:22:13.595091+0000","last_became_peered":"2024-09-17T07:22:13.595091+0000","last_unstale":"2024-09-17T07:22:20.515050+0000","last_undegraded":"2024-09-17T07:22:20.515050+0000","last_fullsized":"2024-09-17T07:22:20.515050+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:18:53.905079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:10:36.334827+0000","last_clean_scrub_stamp":"2024-09-17T07:18:53.905079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0087069899999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":310,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.511388+0000","last_change":"2024-09-17T07:21:15.351645+0000","last_active":"2024-09-17T07:22:20.511388+0000","last_peered":"2024-09-17T07:22:20.511388+0000","last_clean":"2024-09-17T07:22:20.511388+0000","last_became_active":"2024-09-17T07:21:15.351360+0000","last_became_peered":"2024-09-17T07:21:15.351360+0000","last_unstale":"2024-09-17T07:22:20.511388+0000","last_undegraded":"2024-09-17T07:22:20.511388+0000","last_fullsized":"2024-09-17T07:22:20.511388+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:37.523283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:15:37.523283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013475850000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":464,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.514596+0000","last_change":"2024-09-17T07:21:15.355169+0000","last_active":"2024-09-17T07:22:20.514596+0000","last_peered":"2024-09-17T07:22:20.514596+0000","last_clean":"2024-09-17T07:22:20.514596+0000","last_became_active":"2024-09-17T07:21:15.354884+0000","last_became_peered":"2024-09-17T07:21:15.354884+0000","last_unstale":"2024-09-17T07:22:20.514596+0000","last_undegraded":"2024-09-17T07:22:20.514596+0000","last_fullsized":"2024-09-17T07:22:20.514596+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:40.540336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:15:40.540336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001212982,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"439'27","reported_seq":512,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:21.878936+0000","last_change":"2024-09-17T07:21:16.070482+0000","last_active":"2024-09-17T07:22:21.878936+0000","last_peered":"2024-09-17T07:22:21.878936+0000","last_clean":"2024-09-17T07:22:21.878936+0000","last_became_active":"2024-09-17T07:21:15.347179+0000","last_became_peered":"2024-09-17T07:21:15.347179+0000","last_unstale":"2024-09-17T07:22:21.878936+0000","last_undegraded":"2024-09-17T07:22:21.878936+0000","last_fullsized":"2024-09-17T07:22:21.878936+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'19","last_scrub_stamp":"2024-09-17T07:21:16.070358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:21:16.070358+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0025953619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"94'2","reported_seq":337,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.510548+0000","last_change":"2024-09-17T07:21:15.355110+0000","last_active":"2024-09-17T07:22:20.510548+0000","last_peered":"2024-09-17T07:22:20.510548+0000","last_clean":"2024-09-17T07:22:20.510548+0000","last_became_active":"2024-09-17T07:21:15.353242+0000","last_became_peered":"2024-09-17T07:21:15.353242+0000","last_unstale":"2024-09-17T07:22:20.510548+0000","last_undegraded":"2024-09-17T07:22:20.510548+0000","last_fullsized":"2024-09-17T07:22:20.510548+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-09-17T07:15:48.907821+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-09-17T07:07:31.717888+0000","last_clean_scrub_stamp":"2024-09-17T07:15:48.907821+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":341,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.512612+0000","last_change":"2024-09-17T07:22:13.589554+0000","last_active":"2024-09-17T07:22:20.512612+0000","last_peered":"2024-09-17T07:22:20.512612+0000","last_clean":"2024-09-17T07:22:20.512612+0000","last_became_active":"2024-09-17T07:22:13.589267+0000","last_became_peered":"2024-09-17T07:22:13.589267+0000","last_unstale":"2024-09-17T07:22:20.512612+0000","last_undegraded":"2024-09-17T07:22:20.512612+0000","last_fullsized":"2024-09-17T07:22:20.512612+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:16.640384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:20:16.640384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003133224,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":305,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.517374+0000","last_change":"2024-09-17T07:18:23.515698+0000","last_active":"2024-09-17T07:22:20.517374+0000","last_peered":"2024-09-17T07:22:20.517374+0000","last_clean":"2024-09-17T07:22:20.517374+0000","last_became_active":"2024-09-17T07:09:24.400791+0000","last_became_peered":"2024-09-17T07:09:24.400791+0000","last_unstale":"2024-09-17T07:22:20.517374+0000","last_undegraded":"2024-09-17T07:22:20.517374+0000","last_fullsized":"2024-09-17T07:22:20.517374+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:18:23.515582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:49.573666+0000","last_clean_scrub_stamp":"2024-09-17T07:18:23.515582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0022931869999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"439'12","reported_seq":398,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.534552+0000","last_change":"2024-09-17T07:22:12.532319+0000","last_active":"2024-09-17T07:22:20.534552+0000","last_peered":"2024-09-17T07:22:20.534552+0000","last_clean":"2024-09-17T07:22:20.534552+0000","last_became_active":"2024-09-17T07:09:24.396368+0000","last_became_peered":"2024-09-17T07:09:24.396368+0000","last_unstale":"2024-09-17T07:22:20.534552+0000","last_undegraded":"2024-09-17T07:22:20.534552+0000","last_fullsized":"2024-09-17T07:22:20.534552+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'8","last_scrub_stamp":"2024-09-17T07:20:13.839419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:20:13.839419+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045959240000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":476,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.511940+0000","last_change":"2024-09-17T07:19:22.340606+0000","last_active":"2024-09-17T07:22:20.511940+0000","last_peered":"2024-09-17T07:22:20.511940+0000","last_clean":"2024-09-17T07:22:20.511940+0000","last_became_active":"2024-09-17T07:02:12.008898+0000","last_became_peered":"2024-09-17T07:02:12.008898+0000","last_unstale":"2024-09-17T07:22:20.511940+0000","last_undegraded":"2024-09-17T07:22:20.511940+0000","last_fullsized":"2024-09-17T07:22:20.511940+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:22.340320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:19:22.340320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0083030540000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":502,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.514072+0000","last_change":"2024-09-17T07:22:13.582685+0000","last_active":"2024-09-17T07:22:20.514072+0000","last_peered":"2024-09-17T07:22:20.514072+0000","last_clean":"2024-09-17T07:22:20.514072+0000","last_became_active":"2024-09-17T07:22:13.582229+0000","last_became_peered":"2024-09-17T07:22:13.582229+0000","last_unstale":"2024-09-17T07:22:20.514072+0000","last_undegraded":"2024-09-17T07:22:20.514072+0000","last_fullsized":"2024-09-17T07:22:20.514072+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:36.808726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:54.553970+0000","last_clean_scrub_stamp":"2024-09-17T07:19:36.808726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001250061,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":17,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.510445+0000","last_change":"2024-09-17T07:22:13.624740+0000","last_active":"2024-09-17T07:22:20.510445+0000","last_peered":"2024-09-17T07:22:20.510445+0000","last_clean":"2024-09-17T07:22:20.510445+0000","last_became_active":"2024-09-17T07:22:13.624127+0000","last_became_peered":"2024-09-17T07:22:13.624127+0000","last_unstale":"2024-09-17T07:22:20.510445+0000","last_undegraded":"2024-09-17T07:22:20.510445+0000","last_fullsized":"2024-09-17T07:22:20.510445+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:25.003509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:20:25.003509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002164586,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"439'27","reported_seq":41,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:22.221302+0000","last_change":"2024-09-17T07:22:13.636054+0000","last_active":"2024-09-17T07:22:22.221302+0000","last_peered":"2024-09-17T07:22:22.221302+0000","last_clean":"2024-09-17T07:22:22.221302+0000","last_became_active":"2024-09-17T07:22:13.573728+0000","last_became_peered":"2024-09-17T07:22:13.573728+0000","last_unstale":"2024-09-17T07:22:22.221302+0000","last_undegraded":"2024-09-17T07:22:22.221302+0000","last_fullsized":"2024-09-17T07:22:22.221302+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'19","last_scrub_stamp":"2024-09-17T07:20:28.113977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:20:28.113977+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.013438802,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":487,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.516173+0000","last_change":"2024-09-17T07:18:41.951465+0000","last_active":"2024-09-17T07:22:20.516173+0000","last_peered":"2024-09-17T07:22:20.516173+0000","last_clean":"2024-09-17T07:22:20.516173+0000","last_became_active":"2024-09-17T07:02:12.010195+0000","last_became_peered":"2024-09-17T07:02:12.010195+0000","last_unstale":"2024-09-17T07:22:20.516173+0000","last_undegraded":"2024-09-17T07:22:20.516173+0000","last_fullsized":"2024-09-17T07:22:20.516173+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:18:41.951382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:18:41.951382+0000","last_clean_scrub_stamp":"2024-09-17T07:18:41.951382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001516739,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":17,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.507508+0000","last_change":"2024-09-17T07:22:13.736537+0000","last_active":"2024-09-17T07:22:20.507508+0000","last_peered":"2024-09-17T07:22:20.507508+0000","last_clean":"2024-09-17T07:22:20.507508+0000","last_became_active":"2024-09-17T07:22:13.710754+0000","last_became_peered":"2024-09-17T07:22:13.710754+0000","last_unstale":"2024-09-17T07:22:20.507508+0000","last_undegraded":"2024-09-17T07:22:20.507508+0000","last_fullsized":"2024-09-17T07:22:20.507508+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:30.038044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:20:30.038044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":316,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.511319+0000","last_change":"2024-09-17T07:19:25.448467+0000","last_active":"2024-09-17T07:22:20.511319+0000","last_peered":"2024-09-17T07:22:20.511319+0000","last_clean":"2024-09-17T07:22:20.511319+0000","last_became_active":"2024-09-17T07:09:24.392216+0000","last_became_peered":"2024-09-17T07:09:24.392216+0000","last_unstale":"2024-09-17T07:22:20.511319+0000","last_undegraded":"2024-09-17T07:22:20.511319+0000","last_fullsized":"2024-09-17T07:22:20.511319+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:25.448373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:19:25.448373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033889219999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1c","version":"326'2","reported_seq":21,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.508095+0000","last_change":"2024-09-17T07:22:13.760752+0000","last_active":"2024-09-17T07:22:20.508095+0000","last_peered":"2024-09-17T07:22:20.508095+0000","last_clean":"2024-09-17T07:22:20.508095+0000","last_became_active":"2024-09-17T07:22:13.692835+0000","last_became_peered":"2024-09-17T07:22:13.692835+0000","last_unstale":"2024-09-17T07:22:20.508095+0000","last_undegraded":"2024-09-17T07:22:20.508095+0000","last_fullsized":"2024-09-17T07:22:20.508095+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'2","last_scrub_stamp":"2024-09-17T07:20:19.609678+0000","last_deep_scrub":"326'2","last_deep_scrub_stamp":"2024-09-17T07:17:36.974358+0000","last_clean_scrub_stamp":"2024-09-17T07:20:19.609678+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0053255050000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":495,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.511014+0000","last_change":"2024-09-17T07:22:13.569123+0000","last_active":"2024-09-17T07:22:20.511014+0000","last_peered":"2024-09-17T07:22:20.511014+0000","last_clean":"2024-09-17T07:22:20.511014+0000","last_became_active":"2024-09-17T07:22:13.568846+0000","last_became_peered":"2024-09-17T07:22:13.568846+0000","last_unstale":"2024-09-17T07:22:20.511014+0000","last_undegraded":"2024-09-17T07:22:20.511014+0000","last_fullsized":"2024-09-17T07:22:20.511014+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:23.314136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:19:23.314136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010788304,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"439'35","reported_seq":622,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:22.176744+0000","last_change":"2024-09-17T07:22:13.739686+0000","last_active":"2024-09-17T07:22:22.176744+0000","last_peered":"2024-09-17T07:22:22.176744+0000","last_clean":"2024-09-17T07:22:22.176744+0000","last_became_active":"2024-09-17T07:22:13.591382+0000","last_became_peered":"2024-09-17T07:22:13.591382+0000","last_unstale":"2024-09-17T07:22:22.176744+0000","last_undegraded":"2024-09-17T07:22:22.176744+0000","last_fullsized":"2024-09-17T07:22:22.176744+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'25","last_scrub_stamp":"2024-09-17T07:20:39.458854+0000","last_deep_scrub":"294'25","last_deep_scrub_stamp":"2024-09-17T07:16:00.471311+0000","last_clean_scrub_stamp":"2024-09-17T07:20:39.458854+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.005100612,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"436'10","reported_seq":765,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-17T07:22:24.387165+0000","last_change":"2024-09-17T07:22:13.592199+0000","last_active":"2024-09-17T07:22:24.387165+0000","last_peered":"2024-09-17T07:22:24.387165+0000","last_clean":"2024-09-17T07:22:24.387165+0000","last_became_active":"2024-09-17T07:22:13.574012+0000","last_became_peered":"2024-09-17T07:22:13.574012+0000","last_unstale":"2024-09-17T07:22:24.387165+0000","last_undegraded":"2024-09-17T07:22:24.387165+0000","last_fullsized":"2024-09-17T07:22:24.387165+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:19:30.879061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:19:30.879061+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034025510000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":323,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.510359+0000","last_change":"2024-09-17T07:21:15.354938+0000","last_active":"2024-09-17T07:22:20.510359+0000","last_peered":"2024-09-17T07:22:20.510359+0000","last_clean":"2024-09-17T07:22:20.510359+0000","last_became_active":"2024-09-17T07:21:15.353194+0000","last_became_peered":"2024-09-17T07:21:15.353194+0000","last_unstale":"2024-09-17T07:22:20.510359+0000","last_undegraded":"2024-09-17T07:22:20.510359+0000","last_fullsized":"2024-09-17T07:22:20.510359+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:37.885570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:15:37.885570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":327,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.514370+0000","last_change":"2024-09-17T07:19:22.869811+0000","last_active":"2024-09-17T07:22:20.514370+0000","last_peered":"2024-09-17T07:22:20.514370+0000","last_clean":"2024-09-17T07:22:20.514370+0000","last_became_active":"2024-09-17T07:09:24.403360+0000","last_became_peered":"2024-09-17T07:09:24.403360+0000","last_unstale":"2024-09-17T07:22:20.514370+0000","last_undegraded":"2024-09-17T07:22:20.514370+0000","last_fullsized":"2024-09-17T07:22:20.514370+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:22.869700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:10:29.325664+0000","last_clean_scrub_stamp":"2024-09-17T07:19:22.869700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0039824769999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"439'27","reported_seq":571,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:22.235657+0000","last_change":"2024-09-17T07:19:26.874442+0000","last_active":"2024-09-17T07:22:22.235657+0000","last_peered":"2024-09-17T07:22:22.235657+0000","last_clean":"2024-09-17T07:22:22.235657+0000","last_became_active":"2024-09-17T07:03:48.760390+0000","last_became_peered":"2024-09-17T07:03:48.760390+0000","last_unstale":"2024-09-17T07:22:22.235657+0000","last_undegraded":"2024-09-17T07:22:22.235657+0000","last_fullsized":"2024-09-17T07:22:22.235657+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'19","last_scrub_stamp":"2024-09-17T07:19:26.872409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:19:26.872409+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012641085999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":503,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-17T07:22:24.389084+0000","last_change":"2024-09-17T07:22:13.590028+0000","last_active":"2024-09-17T07:22:24.389084+0000","last_peered":"2024-09-17T07:22:24.389084+0000","last_clean":"2024-09-17T07:22:24.389084+0000","last_became_active":"2024-09-17T07:22:13.589529+0000","last_became_peered":"2024-09-17T07:22:13.589529+0000","last_unstale":"2024-09-17T07:22:24.389084+0000","last_undegraded":"2024-09-17T07:22:24.389084+0000","last_fullsized":"2024-09-17T07:22:24.389084+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:37.435106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:20:37.435106+0000","last_clean_scrub_stamp":"2024-09-17T07:20:37.435106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0042822729999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":474,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-17T07:22:24.393047+0000","last_change":"2024-09-17T07:20:26.001554+0000","last_active":"2024-09-17T07:22:24.393047+0000","last_peered":"2024-09-17T07:22:24.393047+0000","last_clean":"2024-09-17T07:22:24.393047+0000","last_became_active":"2024-09-17T07:02:18.363752+0000","last_became_peered":"2024-09-17T07:02:18.363752+0000","last_unstale":"2024-09-17T07:22:24.393047+0000","last_undegraded":"2024-09-17T07:22:24.393047+0000","last_fullsized":"2024-09-17T07:22:24.393047+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:26.001442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:20:26.001442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001725695,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":527,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.515114+0000","last_change":"2024-09-17T07:22:13.587819+0000","last_active":"2024-09-17T07:22:20.515114+0000","last_peered":"2024-09-17T07:22:20.515114+0000","last_clean":"2024-09-17T07:22:20.515114+0000","last_became_active":"2024-09-17T07:22:13.587273+0000","last_became_peered":"2024-09-17T07:22:13.587273+0000","last_unstale":"2024-09-17T07:22:20.515114+0000","last_undegraded":"2024-09-17T07:22:20.515114+0000","last_fullsized":"2024-09-17T07:22:20.515114+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:18.074791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:11:18.559877+0000","last_clean_scrub_stamp":"2024-09-17T07:20:18.074791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0039012449999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"439'39","reported_seq":400,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:21.608967+0000","last_change":"2024-09-17T07:21:16.372009+0000","last_active":"2024-09-17T07:22:21.608967+0000","last_peered":"2024-09-17T07:22:21.608967+0000","last_clean":"2024-09-17T07:22:21.608967+0000","last_became_active":"2024-09-17T07:21:16.371404+0000","last_became_peered":"2024-09-17T07:21:16.371404+0000","last_unstale":"2024-09-17T07:22:21.608967+0000","last_undegraded":"2024-09-17T07:22:21.608967+0000","last_fullsized":"2024-09-17T07:22:21.608967+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'27","last_scrub_stamp":"2024-09-17T07:21:11.488225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:21:11.488225+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0089238530000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":302,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.510659+0000","last_change":"2024-09-17T07:21:15.351806+0000","last_active":"2024-09-17T07:22:20.510659+0000","last_peered":"2024-09-17T07:22:20.510659+0000","last_clean":"2024-09-17T07:22:20.510659+0000","last_became_active":"2024-09-17T07:21:15.348754+0000","last_became_peered":"2024-09-17T07:21:15.348754+0000","last_unstale":"2024-09-17T07:22:20.510659+0000","last_undegraded":"2024-09-17T07:22:20.510659+0000","last_fullsized":"2024-09-17T07:22:20.510659+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:00.509283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:25.794418+0000","last_clean_scrub_stamp":"2024-09-17T07:14:00.509283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":333,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.514970+0000","last_change":"2024-09-17T07:21:16.225667+0000","last_active":"2024-09-17T07:22:20.514970+0000","last_peered":"2024-09-17T07:22:20.514970+0000","last_clean":"2024-09-17T07:22:20.514970+0000","last_became_active":"2024-09-17T07:21:15.348166+0000","last_became_peered":"2024-09-17T07:21:15.348166+0000","last_unstale":"2024-09-17T07:22:20.514970+0000","last_undegraded":"2024-09-17T07:22:20.514970+0000","last_fullsized":"2024-09-17T07:22:20.514970+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:21:16.225555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:21:16.225555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001743026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":17,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.506283+0000","last_change":"2024-09-17T07:22:13.624786+0000","last_active":"2024-09-17T07:22:20.506283+0000","last_peered":"2024-09-17T07:22:20.506283+0000","last_clean":"2024-09-17T07:22:20.506283+0000","last_became_active":"2024-09-17T07:22:13.624057+0000","last_became_peered":"2024-09-17T07:22:13.624057+0000","last_unstale":"2024-09-17T07:22:20.506283+0000","last_undegraded":"2024-09-17T07:22:20.506283+0000","last_fullsized":"2024-09-17T07:22:20.506283+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:30.003956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:19:30.003956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"439'29","reported_seq":243,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:22.268960+0000","last_change":"2024-09-17T07:21:15.352075+0000","last_active":"2024-09-17T07:22:22.268960+0000","last_peered":"2024-09-17T07:22:22.268960+0000","last_clean":"2024-09-17T07:22:22.268960+0000","last_became_active":"2024-09-17T07:21:15.351512+0000","last_became_peered":"2024-09-17T07:21:15.351512+0000","last_unstale":"2024-09-17T07:22:22.268960+0000","last_undegraded":"2024-09-17T07:22:22.268960+0000","last_fullsized":"2024-09-17T07:22:22.268960+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'23","last_scrub_stamp":"2024-09-17T07:14:58.689120+0000","last_deep_scrub":"86'17","last_deep_scrub_stamp":"2024-09-17T07:03:18.241067+0000","last_clean_scrub_stamp":"2024-09-17T07:14:58.689120+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":512,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-09-17T07:22:20.518435+0000","last_change":"2024-09-17T07:22:13.591148+0000","last_active":"2024-09-17T07:22:20.518435+0000","last_peered":"2024-09-17T07:22:20.518435+0000","last_clean":"2024-09-17T07:22:20.518435+0000","last_became_active":"2024-09-17T07:22:13.590622+0000","last_became_peered":"2024-09-17T07:22:13.590622+0000","last_unstale":"2024-09-17T07:22:20.518435+0000","last_undegraded":"2024-09-17T07:22:20.518435+0000","last_fullsized":"2024-09-17T07:22:20.518435+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:21.073159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:20:21.073159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0093893710000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":7416774680,"num_objects":113172,"num_object_clones":0,"num_object_copies":339516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":61811,"num_objects_misplaced":32116,"num_objects_unfound":0,"num_objects_dirty":113172,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":231322,"num_write_kb":11023745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35973,"num_bytes_recovered":2357460992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18817626112,"data_stored":18817613896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72031,"ondisk_log_size":72031,"up":48,"acting":37,"num_store_stats":7},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":382,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89,"ondisk_log_size":89,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1539,"num_read_kb":1296,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":4956,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":13004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1020,"ondisk_log_size":1020,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":80,"num_read_kb":75,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1386,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":12,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":4578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":858717,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":97,"num_read_kb":139,"num_write":82,"num_write_kb":1098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1132727,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1430,"ondisk_log_size":1430,"up":33,"acting":33,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698038010,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1361636,"kb_used_data":464220,"kb_used_omap":1,"kb_used_meta":897406,"kb_avail":92383516,"statfs":{"total":95995035648,"available":94600720384,"internally_reserved":0,"allocated":475361280,"data_stored":474268149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":918944203},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:21:40 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":1.5189999999999999,"15min":1.5109999999999999},"min":{"1min":0.22800000000000001,"5min":0.19700000000000001,"15min":0.192},"max":{"1min":2.4860000000000002,"5min":43.905000000000001,"15min":48.493000000000002},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.76100000000000001,"5min":1.4370000000000001,"15min":1.591},"min":{"1min":0.23799999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":3.7090000000000001,"5min":44.432000000000002,"15min":48.738},"last":0.221}]},{"osd":1,"last update":"Tue Sep 17 07:22:15 2024","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":1.252,"15min":1.252},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":9.9789999999999992,"5min":9.9789999999999992,"15min":9.9789999999999992},"last":0.752},{"interface":"front","average":{"1min":1.22,"5min":1.22,"15min":1.22},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":9.9429999999999996,"5min":9.9429999999999996,"15min":9.9429999999999996},"last":0.30499999999999999}]},{"osd":2,"last update":"Tue Sep 17 07:21:40 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":1.0549999999999999,"15min":1.1180000000000001},"min":{"1min":0.26600000000000001,"5min":0.23599999999999999,"15min":0.23000000000000001},"max":{"1min":2.1840000000000002,"5min":10.057,"15min":22.376000000000001},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.94799999999999995,"15min":1.0589999999999999},"min":{"1min":0.28299999999999997,"5min":0.23200000000000001,"15min":0.224},"max":{"1min":4.048,"5min":9.843,"15min":22.298999999999999},"last":0.63900000000000001}]},{"osd":3,"last update":"Tue Sep 17 07:21:40 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":1.0329999999999999,"15min":1.1830000000000001},"min":{"1min":0.34300000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.2709999999999999,"5min":10.122999999999999,"15min":22.196000000000002},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.874,"15min":1.1839999999999999},"min":{"1min":0.313,"5min":0.307,"15min":0.26500000000000001},"max":{"1min":0.56299999999999994,"5min":12.465999999999999,"15min":22.234000000000002},"last":0.68500000000000005}]},{"osd":4,"last update":"Tue Sep 17 07:21:40 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":1.262,"15min":1.038},"min":{"1min":0.28100000000000003,"5min":0.222,"15min":0.222},"max":{"1min":1.22,"5min":21.140999999999998,"15min":21.140999999999998},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.52500000000000002,"5min":1.6180000000000001,"15min":1.2110000000000001},"min":{"1min":0.25800000000000001,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":1.8049999999999999,"5min":19.420000000000002,"15min":19.420000000000002},"last":0.54300000000000004}]},{"osd":5,"last update":"Tue Sep 17 07:21:40 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":1.2929999999999999,"15min":1.244},"min":{"1min":0.30299999999999999,"5min":0.28799999999999998,"15min":0.22600000000000001},"max":{"1min":1.3979999999999999,"5min":19.384,"15min":29.428999999999998},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.77800000000000002,"5min":1.1890000000000001,"15min":1.2909999999999999},"min":{"1min":0.39400000000000002,"5min":0.27300000000000002,"15min":0.25700000000000001},"max":{"1min":1.6140000000000001,"5min":19.483000000000001,"15min":30.059999999999999},"last":0.61599999999999999}]},{"osd":6,"last update":"Tue Sep 17 07:21:40 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":1.6930000000000001,"15min":1.349},"min":{"1min":0.35999999999999999,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":3.0790000000000002,"5min":22.221,"15min":25.363},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.67600000000000005,"5min":1.881,"15min":1.4179999999999999},"min":{"1min":0.33400000000000002,"5min":0.29999999999999999,"15min":0.27500000000000002},"max":{"1min":1.7629999999999999,"5min":22.202000000000002,"15min":23.571000000000002},"last":0.49299999999999999}]}]},{"osd":6,"up_from":50,"seq":214748365052,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5111040,"kb_used_data":4145400,"kb_used_omap":1,"kb_used_meta":965502,"kb_avail":88634112,"statfs":{"total":95995035648,"available":90761330688,"internally_reserved":0,"allocated":4244889600,"data_stored":4243768890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":988674503},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:22:26 2024","interfaces":[{"interface":"back","average":{"1min":5.7949999999999999,"5min":2.8679999999999999,"15min":2.1360000000000001},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.185},"max":{"1min":91.683999999999997,"5min":91.683999999999997,"15min":126.902},"last":3.839},{"interface":"front","average":{"1min":5.7140000000000004,"5min":2.4620000000000002,"15min":1.9350000000000001},"min":{"1min":0.23300000000000001,"5min":0.22500000000000001,"15min":0.20799999999999999},"max":{"1min":91.403999999999996,"5min":91.403999999999996,"15min":127.06399999999999},"last":3.6230000000000002}]},{"osd":1,"last update":"Tue Sep 17 07:22:16 2024","interfaces":[{"interface":"back","average":{"1min":6.0490000000000004,"5min":6.0490000000000004,"15min":6.0490000000000004},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":91.475999999999999,"5min":91.475999999999999,"15min":91.475999999999999},"last":3.7519999999999998},{"interface":"front","average":{"1min":6.415,"5min":6.415,"15min":6.415},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":91.433999999999997,"5min":91.433999999999997,"15min":91.433999999999997},"last":0.34999999999999998}]},{"osd":2,"last update":"Tue Sep 17 07:22:26 2024","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.893,"15min":1.212},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.219},"max":{"1min":3.8929999999999998,"5min":36.289000000000001,"15min":36.289000000000001},"last":3.8889999999999998},{"interface":"front","average":{"1min":0.88500000000000001,"5min":1.5389999999999999,"15min":1.056},"min":{"1min":0.28299999999999997,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":3.7290000000000001,"5min":36.326999999999998,"15min":36.326999999999998},"last":3.7290000000000001}]},{"osd":3,"last update":"Tue Sep 17 07:22:26 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":1.0720000000000001,"15min":0.99399999999999999},"min":{"1min":0.34799999999999998,"5min":0.249,"15min":0.23200000000000001},"max":{"1min":3.0649999999999999,"5min":6.8639999999999999,"15min":16.984000000000002},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.95199999999999996,"5min":1.1679999999999999,"15min":1.1579999999999999},"min":{"1min":0.28000000000000003,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":3.8130000000000002,"5min":5.2949999999999999,"15min":16.745999999999999},"last":3.8130000000000002}]},{"osd":4,"last update":"Tue Sep 17 07:22:26 2024","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":1.163,"15min":2.2709999999999999},"min":{"1min":0.30499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":8.5679999999999996,"5min":8.5679999999999996,"15min":260.77199999999999},"last":8.5679999999999996},{"interface":"front","average":{"1min":0.78200000000000003,"5min":1.5049999999999999,"15min":2.3330000000000002},"min":{"1min":0.32000000000000001,"5min":0.29499999999999998,"15min":0.19800000000000001},"max":{"1min":3.1190000000000002,"5min":8.7929999999999993,"15min":260.69099999999997},"last":0.39500000000000002}]},{"osd":5,"last update":"Tue Sep 17 07:22:26 2024","interfaces":[{"interface":"back","average":{"1min":16.803999999999998,"5min":5.2350000000000003,"15min":2.7069999999999999},"min":{"1min":0.25600000000000001,"5min":0.253,"15min":0.23400000000000001},"max":{"1min":317.59800000000001,"5min":317.59800000000001,"15min":317.59800000000001},"last":3.665},{"interface":"front","average":{"1min":16.704000000000001,"5min":5.3639999999999999,"15min":2.7240000000000002},"min":{"1min":0.315,"5min":0.315,"15min":0.251},"max":{"1min":317.762,"5min":317.762,"15min":317.762},"last":0.376}]},{"osd":7,"last update":"Tue Sep 17 07:21:42 2024","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":1.431,"15min":1.681},"min":{"1min":0.39500000000000002,"5min":0.29399999999999998,"15min":0.246},"max":{"1min":2.2360000000000002,"5min":6.8179999999999996,"15min":131.95699999999999},"last":1.675},{"interface":"front","average":{"1min":0.97699999999999998,"5min":1.1419999999999999,"15min":1.556},"min":{"1min":0.46500000000000002,"5min":0.252,"15min":0.252},"max":{"1min":3.7149999999999999,"5min":5.8070000000000004,"15min":131.833},"last":3.7029999999999998}]}]},{"osd":5,"up_from":42,"seq":180388626689,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3276196,"kb_used_data":2298396,"kb_used_omap":1,"kb_used_meta":977790,"kb_avail":90468956,"statfs":{"total":95995035648,"available":92640210944,"internally_reserved":0,"allocated":2353557504,"data_stored":2352465157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1001257414},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:22:04 2024","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.84599999999999997,"15min":1.149},"min":{"1min":0.22900000000000001,"5min":0.184,"15min":0.184},"max":{"1min":3.0750000000000002,"5min":5.7530000000000001,"15min":28.727},"last":1.242},{"interface":"front","average":{"1min":0.75,"5min":0.76800000000000002,"15min":1.163},"min":{"1min":0.23699999999999999,"5min":0.217,"15min":0.217},"max":{"1min":1.518,"5min":4.7939999999999996,"15min":27.824999999999999},"last":0.32800000000000001}]},{"osd":1,"last update":"Tue Sep 17 07:22:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2569999999999999,"5min":1.2569999999999999,"15min":1.2569999999999999},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":8.4060000000000006,"5min":8.4060000000000006,"15min":8.4060000000000006},"last":0.90200000000000002},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":2.73,"5min":2.73,"15min":2.73},"last":0.627}]},{"osd":2,"last update":"Tue Sep 17 07:22:04 2024","interfaces":[{"interface":"back","average":{"1min":1.0249999999999999,"5min":0.999,"15min":1.1599999999999999},"min":{"1min":0.24199999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":2.984,"5min":4.4160000000000004,"15min":20.486000000000001},"last":2.3570000000000002},{"interface":"front","average":{"1min":0.89900000000000002,"5min":0.93600000000000005,"15min":1.044},"min":{"1min":0.28499999999999998,"5min":0.219,"15min":0.219},"max":{"1min":2.5230000000000001,"5min":7.71,"15min":23.925999999999998},"last":0.93400000000000005}]},{"osd":3,"last update":"Tue Sep 17 07:22:04 2024","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":1.119,"15min":1.548},"min":{"1min":0.30099999999999999,"5min":0.27500000000000002,"15min":0.247},"max":{"1min":3.0150000000000001,"5min":16.143000000000001,"15min":68.978999999999999},"last":2.3319999999999999},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.77200000000000002,"15min":1.3109999999999999},"min":{"1min":0.29899999999999999,"5min":0.27100000000000002,"15min":0.253},"max":{"1min":2.3769999999999998,"5min":5.7880000000000003,"15min":69.084000000000003},"last":0.95899999999999996}]},{"osd":4,"last update":"Tue Sep 17 07:22:04 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":2.2719999999999998,"15min":2.1669999999999998},"min":{"1min":0.26700000000000002,"5min":0.247,"15min":0.22},"max":{"1min":2.5,"5min":96.668000000000006,"15min":142.327},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.96799999999999997,"5min":2.335,"15min":2.0920000000000001},"min":{"1min":0.27600000000000002,"5min":0.255,"15min":0.20999999999999999},"max":{"1min":4.3419999999999996,"5min":96.801000000000002,"15min":142.13200000000001},"last":2.2959999999999998}]},{"osd":6,"last update":"Tue Sep 17 07:22:19 2024","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":1.986,"15min":1.3660000000000001},"min":{"1min":0.26700000000000002,"5min":0.245,"15min":0.23499999999999999},"max":{"1min":3.194,"5min":96.686000000000007,"15min":96.686000000000007},"last":2.7160000000000002},{"interface":"front","average":{"1min":0.83299999999999996,"5min":2.3759999999999999,"15min":1.55},"min":{"1min":0.31,"5min":0.29799999999999999,"15min":0.26700000000000002},"max":{"1min":3.5510000000000002,"5min":96.834999999999994,"15min":96.834999999999994},"last":2.8839999999999999}]},{"osd":7,"last update":"Tue Sep 17 07:21:42 2024","interfaces":[{"interface":"back","average":{"1min":1.073,"5min":1.234,"15min":1.2849999999999999},"min":{"1min":0.307,"5min":0.28399999999999997,"15min":0.28299999999999997},"max":{"1min":2.657,"5min":4.8419999999999996,"15min":6.8869999999999996},"last":3.395},{"interface":"front","average":{"1min":0.51200000000000001,"5min":1.028,"15min":1.0509999999999999},"min":{"1min":0.36799999999999999,"5min":0.30199999999999999,"15min":0.27100000000000002},"max":{"1min":0.93400000000000005,"5min":6.3639999999999999,"15min":6.3639999999999999},"last":0.97999999999999998}]}]},{"osd":4,"up_from":33,"seq":141733921032,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3176812,"kb_used_data":2194916,"kb_used_omap":1,"kb_used_meta":981886,"kb_avail":90568340,"statfs":{"total":95995035648,"available":92741980160,"internally_reserved":0,"allocated":2247593984,"data_stored":2246492135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1005451718},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:22:28 2024","interfaces":[{"interface":"back","average":{"1min":3.4390000000000001,"5min":1.667,"15min":1.724},"min":{"1min":0.26200000000000001,"5min":0.20799999999999999,"15min":0.20000000000000001},"max":{"1min":39.473999999999997,"5min":39.473999999999997,"15min":58.097999999999999},"last":2.1219999999999999},{"interface":"front","average":{"1min":3.27,"5min":1.6459999999999999,"15min":1.8460000000000001},"min":{"1min":0.312,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":39.654000000000003,"5min":39.654000000000003,"15min":58.383000000000003},"last":2.9620000000000002}]},{"osd":1,"last update":"Tue Sep 17 07:22:15 2024","interfaces":[{"interface":"back","average":{"1min":2.8809999999999998,"5min":2.8809999999999998,"15min":2.8809999999999998},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":29.956,"5min":29.956,"15min":29.956},"last":0.51600000000000001},{"interface":"front","average":{"1min":2.9049999999999998,"5min":2.9049999999999998,"15min":2.9049999999999998},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":29.876999999999999,"5min":29.876999999999999,"15min":29.876999999999999},"last":2.1930000000000001}]},{"osd":2,"last update":"Tue Sep 17 07:22:28 2024","interfaces":[{"interface":"back","average":{"1min":1.236,"5min":1.077,"15min":1.0429999999999999},"min":{"1min":0.24399999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":6.5209999999999999,"5min":8.4710000000000001,"15min":8.4710000000000001},"last":0.24399999999999999},{"interface":"front","average":{"1min":1.044,"5min":0.89300000000000002,"15min":1.03},"min":{"1min":0.27400000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":4.4779999999999998,"5min":6.1840000000000002,"15min":16.655999999999999},"last":0.58599999999999997}]},{"osd":3,"last update":"Tue Sep 17 07:22:28 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.94899999999999995,"15min":0.99099999999999999},"min":{"1min":0.311,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":2.3900000000000001,"5min":15.433999999999999,"15min":17.872},"last":0.311},{"interface":"front","average":{"1min":1.0369999999999999,"5min":0.83199999999999996,"15min":1.0529999999999999},"min":{"1min":0.312,"5min":0.28100000000000003,"15min":0.253},"max":{"1min":8.7360000000000007,"5min":9.0329999999999995,"15min":17.908000000000001},"last":0.57299999999999995}]},{"osd":5,"last update":"Tue Sep 17 07:21:39 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":2.4609999999999999,"15min":1.5629999999999999},"min":{"1min":0.27400000000000002,"5min":0.224,"15min":0.20799999999999999},"max":{"1min":2.0670000000000002,"5min":147.61099999999999,"15min":147.61099999999999},"last":1.276},{"interface":"front","average":{"1min":0.97899999999999998,"5min":2.6920000000000002,"15min":1.6439999999999999},"min":{"1min":0.34599999999999997,"5min":0.27400000000000002,"15min":0.26100000000000001},"max":{"1min":3.8130000000000002,"5min":147.74700000000001,"15min":147.74700000000001},"last":1.3009999999999999}]},{"osd":6,"last update":"Tue Sep 17 07:22:28 2024","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":1.1299999999999999,"15min":1.2210000000000001},"min":{"1min":0.28799999999999998,"5min":0.23400000000000001,"15min":0.218},"max":{"1min":2.8029999999999999,"5min":8.9619999999999997,"15min":47.923999999999999},"last":0.28799999999999998},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.161,"15min":1.339},"min":{"1min":0.27400000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":6.6219999999999999,"5min":6.6219999999999999,"15min":47.600999999999999},"last":0.55100000000000005}]},{"osd":7,"last update":"Tue Sep 17 07:21:38 2024","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":1.284,"15min":1.296},"min":{"1min":0.32600000000000001,"5min":0.27900000000000003,"15min":0.26700000000000002},"max":{"1min":2.0259999999999998,"5min":7.1200000000000001,"15min":48.719000000000001},"last":1.875},{"interface":"front","average":{"1min":0.77600000000000002,"5min":1.137,"15min":1.337},"min":{"1min":0.34799999999999998,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":3.7829999999999999,"5min":9.0030000000000001,"15min":50.527000000000001},"last":0.78800000000000003}]}]},{"osd":3,"up_from":0,"seq":115964117257,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":18,"seq":77309411600,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4172904,"kb_used_data":3099104,"kb_used_omap":1,"kb_used_meta":1073790,"kb_avail":89572248,"statfs":{"total":95995035648,"available":91721981952,"internally_reserved":0,"allocated":3173482496,"data_stored":3172385911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1099561414},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:21:56 2024","interfaces":[{"interface":"back","average":{"1min":3.1739999999999999,"5min":1.3180000000000001,"15min":2.1219999999999999},"min":{"1min":0.17199999999999999,"5min":0.154,"15min":0.154},"max":{"1min":21.192,"5min":21.934000000000001,"15min":131.09800000000001},"last":1.403},{"interface":"front","average":{"1min":3.6859999999999999,"5min":1.468,"15min":2.145},"min":{"1min":0.252,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":20.981000000000002,"5min":21.591000000000001,"15min":131.24700000000001},"last":0.66600000000000004}]},{"osd":1,"last update":"Tue Sep 17 07:22:19 2024","interfaces":[{"interface":"back","average":{"1min":1.968,"5min":1.968,"15min":1.968},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":17.375,"5min":17.375,"15min":17.375},"last":0.28000000000000003},{"interface":"front","average":{"1min":2.3130000000000002,"5min":2.3130000000000002,"15min":2.3130000000000002},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":18.035,"5min":18.035,"15min":18.035},"last":1.431}]},{"osd":3,"last update":"Tue Sep 17 07:22:11 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.49399999999999999,"15min":0.89200000000000002},"min":{"1min":0.20499999999999999,"5min":0.20300000000000001,"15min":0.20100000000000001},"max":{"1min":1.8180000000000001,"5min":3.1880000000000002,"15min":23.43},"last":0.45600000000000002},{"interface":"front","average":{"1min":1.2410000000000001,"5min":0.68799999999999994,"15min":0.88800000000000001},"min":{"1min":0.23100000000000001,"5min":0.22,"15min":0.22},"max":{"1min":5.8840000000000003,"5min":5.8840000000000003,"15min":23.408000000000001},"last":0.35899999999999999}]},{"osd":4,"last update":"Tue Sep 17 07:21:37 2024","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":1.8819999999999999,"15min":1.2689999999999999},"min":{"1min":0.27700000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":2.7730000000000001,"5min":75.870999999999995,"15min":75.870999999999995},"last":1.8460000000000001},{"interface":"front","average":{"1min":0.88400000000000001,"5min":2.0800000000000001,"15min":1.4219999999999999},"min":{"1min":0.35199999999999998,"5min":0.26900000000000002,"15min":0.26300000000000001},"max":{"1min":2.7879999999999998,"5min":75.888999999999996,"15min":75.888999999999996},"last":1.8759999999999999}]},{"osd":5,"last update":"Tue Sep 17 07:21:56 2024","interfaces":[{"interface":"back","average":{"1min":1.0369999999999999,"5min":1.151,"15min":1.2150000000000001},"min":{"1min":0.312,"5min":0.27900000000000003,"15min":0.26100000000000001},"max":{"1min":3.8500000000000001,"5min":7.2229999999999999,"15min":15.276999999999999},"last":1.8160000000000001},{"interface":"front","average":{"1min":0.95799999999999996,"5min":0.84199999999999997,"15min":1.014},"min":{"1min":0.29599999999999999,"5min":0.27700000000000002,"15min":0.27600000000000002},"max":{"1min":2.6549999999999998,"5min":5.8239999999999998,"15min":15.561999999999999},"last":1.343}]},{"osd":6,"last update":"Tue Sep 17 07:22:11 2024","interfaces":[{"interface":"back","average":{"1min":4.9550000000000001,"5min":2.7719999999999998,"15min":2.0510000000000002},"min":{"1min":0.30599999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":65.143000000000001,"5min":76.081999999999994,"15min":101.587},"last":1.379},{"interface":"front","average":{"1min":4.8479999999999999,"5min":3.0590000000000002,"15min":2.1579999999999999},"min":{"1min":0.33600000000000002,"5min":0.313,"15min":0.30599999999999999},"max":{"1min":65.241,"5min":76.244,"15min":100.696},"last":0.309}]},{"osd":7,"last update":"Tue Sep 17 07:21:56 2024","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.89300000000000002,"15min":1.3819999999999999},"min":{"1min":0.373,"5min":0.316,"15min":0.316},"max":{"1min":2.6949999999999998,"5min":4.7759999999999998,"15min":94.269999999999996},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.96599999999999997,"5min":0.89500000000000002,"15min":1.456},"min":{"1min":0.38200000000000001,"5min":0.309,"15min":0.309},"max":{"1min":2.75,"5min":4.8090000000000002,"15min":94.634},"last":0.42899999999999999}]}]},{"osd":1,"up_from":413,"seq":1773821493264,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1950720,"kb_used_data":1683640,"kb_used_omap":4,"kb_used_meta":267067,"kb_avail":91794432,"statfs":{"total":95995035648,"available":93997498368,"internally_reserved":0,"allocated":1724047360,"data_stored":1722926803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4439,"internal_metadata":273477289},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:22:18 2024","interfaces":[{"interface":"back","average":{"1min":1.671,"5min":1.671,"15min":1.671},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":7.359,"5min":7.359,"15min":7.359},"last":0.66500000000000004},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.3899999999999999,"15min":1.3899999999999999},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":5.7599999999999998,"5min":5.7599999999999998,"15min":5.7599999999999998},"last":3.6920000000000002}]},{"osd":2,"last update":"Tue Sep 17 07:22:18 2024","interfaces":[{"interface":"back","average":{"1min":1.3620000000000001,"5min":1.3620000000000001,"15min":1.3620000000000001},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":5.7220000000000004,"5min":5.7220000000000004,"15min":5.7220000000000004},"last":0.74099999999999999},{"interface":"front","average":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":3.7850000000000001,"5min":3.7850000000000001,"15min":3.7850000000000001},"last":0.60499999999999998}]},{"osd":3,"last update":"Tue Sep 17 07:22:18 2024","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":5.9290000000000003,"5min":5.9290000000000003,"15min":5.9290000000000003},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.88,"5min":0.88,"15min":0.88},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":4.1340000000000003,"5min":4.1340000000000003,"15min":4.1340000000000003},"last":0.70399999999999996}]},{"osd":4,"last update":"Tue Sep 17 07:22:18 2024","interfaces":[{"interface":"back","average":{"1min":7.3570000000000002,"5min":7.3570000000000002,"15min":7.3570000000000002},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":116.88200000000001,"5min":116.88200000000001,"15min":116.88200000000001},"last":0.57099999999999995},{"interface":"front","average":{"1min":7.4269999999999996,"5min":7.4269999999999996,"15min":7.4269999999999996},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":116.98399999999999,"5min":116.98399999999999,"15min":116.98399999999999},"last":0.51400000000000001}]},{"osd":5,"last update":"Tue Sep 17 07:22:18 2024","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":3.1080000000000001,"5min":3.1080000000000001,"15min":3.1080000000000001},"last":0.76000000000000001},{"interface":"front","average":{"1min":1.079,"5min":1.079,"15min":1.079},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":5.9080000000000004,"5min":5.9080000000000004,"15min":5.9080000000000004},"last":0.48399999999999999}]},{"osd":6,"last update":"Tue Sep 17 07:22:18 2024","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.431,"15min":1.431},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":5.9509999999999996,"5min":5.9509999999999996,"15min":5.9509999999999996},"last":0.63800000000000001},{"interface":"front","average":{"1min":1.3280000000000001,"5min":1.3280000000000001,"15min":1.3280000000000001},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":4.266,"5min":4.266,"15min":4.266},"last":0.85199999999999998}]},{"osd":7,"last update":"Tue Sep 17 07:22:18 2024","interfaces":[{"interface":"back","average":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":4.2389999999999999,"5min":4.2389999999999999,"15min":4.2389999999999999},"last":1.091},{"interface":"front","average":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":3.2440000000000002,"5min":3.2440000000000002,"15min":3.2440000000000002},"last":0.97099999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738647,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5806320,"kb_used_data":4553448,"kb_used_omap":1,"kb_used_meta":1252862,"kb_avail":87938832,"statfs":{"total":95995035648,"available":90049363968,"internally_reserved":0,"allocated":4662730752,"data_stored":4661625717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1282931142},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 17 07:22:15 2024","interfaces":[{"interface":"back","average":{"1min":2.008,"5min":2.008,"15min":2.008},"min":{"1min":0.156,"5min":0.156,"15min":0.156},"max":{"1min":5.3570000000000002,"5min":5.3570000000000002,"15min":5.3570000000000002},"last":2.173},{"interface":"front","average":{"1min":1.508,"5min":1.508,"15min":1.508},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":4.8179999999999996,"5min":4.8179999999999996,"15min":4.8179999999999996},"last":2.6440000000000001}]},{"osd":2,"last update":"Tue Sep 17 07:21:51 2024","interfaces":[{"interface":"back","average":{"1min":5.9249999999999998,"5min":1.5589999999999999,"15min":1.1479999999999999},"min":{"1min":0.23100000000000001,"5min":0.155,"15min":0.155},"max":{"1min":62.094999999999999,"5min":62.094999999999999,"15min":62.094999999999999},"last":0.95399999999999996},{"interface":"front","average":{"1min":5.6740000000000004,"5min":1.5049999999999999,"15min":1.1619999999999999},"min":{"1min":0.19,"5min":0.183,"15min":0.183},"max":{"1min":59.848999999999997,"5min":59.848999999999997,"15min":59.848999999999997},"last":0.90800000000000003}]},{"osd":3,"last update":"Tue Sep 17 07:22:28 2024","interfaces":[{"interface":"back","average":{"1min":1.365,"5min":0.72099999999999997,"15min":0.81200000000000006},"min":{"1min":0.27900000000000003,"5min":0.223,"15min":0.21199999999999999},"max":{"1min":4.6829999999999998,"5min":4.6829999999999998,"15min":11.598000000000001},"last":0.85999999999999999},{"interface":"front","average":{"1min":1.2649999999999999,"5min":0.73799999999999999,"15min":0.72899999999999998},"min":{"1min":0.24099999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":4.6109999999999998,"5min":5.1150000000000002,"15min":5.4169999999999998},"last":0.75900000000000001}]},{"osd":4,"last update":"Tue Sep 17 07:22:28 2024","interfaces":[{"interface":"back","average":{"1min":1.5429999999999999,"5min":1.2529999999999999,"15min":1.089},"min":{"1min":0.27400000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":3.9380000000000002,"5min":8.3279999999999994,"15min":19.146999999999998},"last":1.149},{"interface":"front","average":{"1min":1.1839999999999999,"5min":1.173,"15min":1.1180000000000001},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26200000000000001},"max":{"1min":5.0110000000000001,"5min":8.3119999999999994,"15min":19.158999999999999},"last":0.74299999999999999}]},{"osd":5,"last update":"Tue Sep 17 07:22:04 2024","interfaces":[{"interface":"back","average":{"1min":4.4119999999999999,"5min":1.6519999999999999,"15min":2.3140000000000001},"min":{"1min":0.371,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":42.741,"5min":42.741,"15min":216.83699999999999},"last":2.0139999999999998},{"interface":"front","average":{"1min":3.6230000000000002,"5min":1.647,"15min":2.3029999999999999},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.27000000000000002},"max":{"1min":42.792000000000002,"5min":42.792000000000002,"15min":216.816},"last":2.6789999999999998}]},{"osd":6,"last update":"Tue Sep 17 07:22:28 2024","interfaces":[{"interface":"back","average":{"1min":3.9350000000000001,"5min":2.323,"15min":1.9039999999999999},"min":{"1min":0.437,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":37.262999999999998,"5min":67.873000000000005,"15min":147.13200000000001},"last":3.7890000000000001},{"interface":"front","average":{"1min":4.3040000000000003,"5min":2.363,"15min":1.8959999999999999},"min":{"1min":0.32000000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":36.789999999999999,"5min":67.914000000000001,"15min":147.172},"last":2.0489999999999999}]},{"osd":7,"last update":"Tue Sep 17 07:22:28 2024","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":1.099,"15min":1.1779999999999999},"min":{"1min":0.318,"5min":0.318,"15min":0.29599999999999999},"max":{"1min":3.0899999999999999,"5min":6.3300000000000001,"15min":6.3300000000000001},"last":2.7370000000000001},{"interface":"front","average":{"1min":1.716,"5min":1.3420000000000001,"15min":1.2470000000000001},"min":{"1min":0.41999999999999998,"5min":0.36599999999999999,"15min":0.32000000000000001},"max":{"1min":4.9100000000000001,"5min":7.2830000000000004,"15min":7.2830000000000004},"last":3.6989999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4644732928,"data_stored":4644732928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1725038592,"data_stored":1725038592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3162705920,"data_stored":3162701848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2243235840,"data_stored":2243231768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2330591232,"data_stored":2330591232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4213837824,"data_stored":4213833752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":497483776,"data_stored":497483776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T07:22:37.945 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj 6, currently 16 2024-09-17T07:22:37.947 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:22:38.006 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:22:38.055 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:38.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:38.057 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:22:38.158 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:22:38.193 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:38.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:38.195 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:22:38.296 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:22:38.338 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:38.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:38.339 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:22:38.440 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:22:38.463 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:38.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:38.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:22:38.566 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:22:38.598 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:38.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:38.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:22:38.700 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:22:38.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:38.736 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:38.737 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:22:38.837 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:22:38.871 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:38.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:38.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:22:38.972 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:22:39.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:39.015 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:39.015 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:22:39.116 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:22:39.132 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:22:39.144 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:39.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:39.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:22:39.246 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:22:39.278 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:39.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:39.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:22:39.380 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:22:39.428 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:39 smithi100 ceph-mon[29092]: pgmap v960: 158 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 5 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 146 active+clean; 6.6 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 910 op/s; 57109/327045 objects degraded (17.462%); 30041/327045 objects misplaced (9.186%); 27 MiB/s, 430 objects/s recovering 2024-09-17T07:22:39.456 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:39.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:39.457 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:22:39.506 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:22:39.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:39 smithi135 ceph-mon[33238]: pgmap v960: 158 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 5 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 146 active+clean; 6.6 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 910 op/s; 57109/327045 objects degraded (17.462%); 30041/327045 objects misplaced (9.186%); 27 MiB/s, 430 objects/s recovering 2024-09-17T07:22:39.543 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:22:39 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:22:39.558 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:22:39.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:39.599 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:39.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:22:39.700 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:22:39.736 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:39.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:39.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:22:39.765 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:39 smithi100 ceph-mon[39462]: pgmap v960: 158 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 5 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 146 active+clean; 6.6 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 910 op/s; 57109/327045 objects degraded (17.462%); 30041/327045 objects misplaced (9.186%); 27 MiB/s, 430 objects/s recovering 2024-09-17T07:22:39.838 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:22:39.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:39.879 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:39.880 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:22:39.980 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:22:40.020 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:40.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:40.022 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:22:40.123 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:22:40.151 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:40.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:40.154 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:22:40.257 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:22:40.296 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:40.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:40.297 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:22:40.398 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:22:40.431 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:40.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:40.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:22:40.533 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:22:40.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:40 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:22:40.544 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:40 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 57109/327045 objects degraded (17.462%), 11 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-17T07:22:40.573 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:40.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:40.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:22:40.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:40 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:22:40.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:40 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 57109/327045 objects degraded (17.462%), 11 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-17T07:22:40.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:40 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:22:40.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:40 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 57109/327045 objects degraded (17.462%), 11 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-17T07:22:40.675 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:22:40.711 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:40.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:40.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:22:40.813 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:22:40.843 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:40.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:40.845 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:22:40.945 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:22:40.971 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:40.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:40.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:22:41.072 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:22:41.102 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:41.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:41.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:22:41.203 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:22:41.242 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:41.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:41.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:22:41.344 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:22:41.382 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:41.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:41.384 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:22:41.484 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:22:41.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:41.518 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:41.518 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:22:41.619 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:22:41.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:41 smithi100 ceph-mon[29092]: pgmap v961: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 5 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 146 active+clean; 6.4 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.19k op/s; 54272/317172 objects degraded (17.111%); 28970/317172 objects misplaced (9.134%); 35 MiB/s, 557 objects/s recovering 2024-09-17T07:22:41.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:41 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:22:41.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:41 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:22:41.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:41 smithi100 ceph-mon[39462]: pgmap v961: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 5 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 146 active+clean; 6.4 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.19k op/s; 54272/317172 objects degraded (17.111%); 28970/317172 objects misplaced (9.134%); 35 MiB/s, 557 objects/s recovering 2024-09-17T07:22:41.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:41 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:22:41.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:41 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:22:41.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:41.666 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:41.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:22:41.767 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:22:41.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:41 smithi135 ceph-mon[33238]: pgmap v961: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 5 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 146 active+clean; 6.4 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.19k op/s; 54272/317172 objects degraded (17.111%); 28970/317172 objects misplaced (9.134%); 35 MiB/s, 557 objects/s recovering 2024-09-17T07:22:41.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:41 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:22:41.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:41 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:22:41.801 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:41.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:41.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:22:41.903 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:22:41.944 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:41.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:41.945 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:22:42.046 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:22:42.067 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:42.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:42.068 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:22:42.169 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:22:42.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:42.204 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:42.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:22:42.305 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:22:42.337 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:42.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:42.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:22:42.439 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:22:42.471 INFO:teuthology.orchestra.run.smithi100.stderr:nodeep-scrub is set 2024-09-17T07:22:42.473 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:42.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:42.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:22:42.575 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:22:42.607 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:42.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:42.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:22:42.709 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:22:42.740 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:42.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:42.741 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:22:42.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:42 smithi135 ceph-mon[33238]: osdmap e441: 8 total, 8 up, 7 in 2024-09-17T07:22:42.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:42 smithi135 ceph-mon[33238]: pgmap v963: 158 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 5 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 146 active+clean; 6.3 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 921 B/s rd, 0 B/s wr, 1.07k op/s; 51647/311064 objects degraded (16.603%); 27575/311064 objects misplaced (8.865%); 33 MiB/s, 529 objects/s recovering 2024-09-17T07:22:42.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:42 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1283798403' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:22:42.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:42 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:22:42.842 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:22:42.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:42.868 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:42.869 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:22:42.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:42 smithi100 ceph-mon[29092]: osdmap e441: 8 total, 8 up, 7 in 2024-09-17T07:22:42.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:42 smithi100 ceph-mon[29092]: pgmap v963: 158 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 5 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 146 active+clean; 6.3 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 921 B/s rd, 0 B/s wr, 1.07k op/s; 51647/311064 objects degraded (16.603%); 27575/311064 objects misplaced (8.865%); 33 MiB/s, 529 objects/s recovering 2024-09-17T07:22:42.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:42 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1283798403' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:22:42.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:42 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:22:42.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:42 smithi100 ceph-mon[39462]: osdmap e441: 8 total, 8 up, 7 in 2024-09-17T07:22:42.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:42 smithi100 ceph-mon[39462]: pgmap v963: 158 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 5 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 146 active+clean; 6.3 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 921 B/s rd, 0 B/s wr, 1.07k op/s; 51647/311064 objects degraded (16.603%); 27575/311064 objects misplaced (8.865%); 33 MiB/s, 529 objects/s recovering 2024-09-17T07:22:42.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:42 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1283798403' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:22:42.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:42 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:22:42.946 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7] out_osds: [3] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 6, 7, 1] 2024-09-17T07:22:42.947 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T07:22:42.947 INFO:tasks.thrashosds.thrasher:Killing osd 5, live_osds are [0, 2, 3, 4, 5, 6, 7, 1] 2024-09-17T07:22:42.947 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-09-17T07:22:42.947 DEBUG:teuthology.orchestra.run.smithi135:> sudo systemctl stop ceph-e848267c-74c1-11ef-bceb-c7b262605968@osd.5 2024-09-17T07:22:42.969 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:22:43.030 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:43.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:43.031 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:22:43.132 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:22:43.168 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:43.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:43.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:22:43.271 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:22:43.293 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:22:43 smithi135 systemd[1]: Stopping Ceph osd.5 for e848267c-74c1-11ef-bceb-c7b262605968... 2024-09-17T07:22:43.308 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:43.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:43.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:22:43.410 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:22:43.452 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:43.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:43.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:22:43.553 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:22:43.584 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:43.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:43.585 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:22:43.686 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:22:43.718 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:43.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:43.719 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:22:43.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:43 smithi135 ceph-mon[33238]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:22:43.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:43 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T07:22:43.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:43 smithi135 ceph-mon[33238]: osdmap e442: 8 total, 8 up, 7 in 2024-09-17T07:22:43.820 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:22:43.844 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:43.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:43.845 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:22:43.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:43 smithi100 ceph-mon[29092]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:22:43.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:43 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T07:22:43.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:43 smithi100 ceph-mon[29092]: osdmap e442: 8 total, 8 up, 7 in 2024-09-17T07:22:43.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:43 smithi100 ceph-mon[39462]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:22:43.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:43 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T07:22:43.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:43 smithi100 ceph-mon[39462]: osdmap e442: 8 total, 8 up, 7 in 2024-09-17T07:22:43.945 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:22:43.981 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:43.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:43.982 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:22:44.083 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:22:44.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:44.123 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:44.123 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:22:44.224 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:22:44.261 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:44.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:44.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:22:44.363 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:22:44.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:44.406 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:44.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:22:44.507 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:22:44.550 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:44.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:44.551 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:22:44.651 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:22:44.693 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:44.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:44.694 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:22:44.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:44 smithi135 ceph-mon[33238]: pgmap v965: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 5 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 146 active+clean; 6.3 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 511 B/s rd, 0 B/s wr, 784 op/s; 51004/308217 objects degraded (16.548%); 27233/308217 objects misplaced (8.836%); 22 MiB/s, 344 objects/s recovering 2024-09-17T07:22:44.795 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:22:44.833 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:44.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:44.834 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:22:44.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:44 smithi100 ceph-mon[29092]: pgmap v965: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 5 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 146 active+clean; 6.3 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 511 B/s rd, 0 B/s wr, 784 op/s; 51004/308217 objects degraded (16.548%); 27233/308217 objects misplaced (8.836%); 22 MiB/s, 344 objects/s recovering 2024-09-17T07:22:44.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:44 smithi100 ceph-mon[39462]: pgmap v965: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 5 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 146 active+clean; 6.3 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 511 B/s rd, 0 B/s wr, 784 op/s; 51004/308217 objects degraded (16.548%); 27233/308217 objects misplaced (8.836%); 22 MiB/s, 344 objects/s recovering 2024-09-17T07:22:44.934 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:22:44.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:44.974 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:44.975 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:22:45.075 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:22:45.113 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:45.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:45.114 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:22:45.215 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:22:45.253 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:45.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:45.254 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:22:45.355 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:22:45.378 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:45.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:45.380 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:22:45.481 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:22:45.523 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:45.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:45.524 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:22:45.625 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:22:45.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:45.663 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:45.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:22:45.749 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:45 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 51004/308217 objects degraded (16.548%), 11 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-17T07:22:45.749 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:45 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 51004/308217 objects degraded (16.548%), 11 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-17T07:22:45.764 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:22:45.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:45 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 51004/308217 objects degraded (16.548%), 11 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-17T07:22:45.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:45.801 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:45.801 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:22:45.902 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:22:45.914 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:22:45.915 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:22:45.924 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:45.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:45.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:22:45.935 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1: {} 2024-09-17T07:22:45.935 INFO:teuthology.orchestra.run.smithi100.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:22:45.953 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:22:45.953 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:22:45.971 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:22:45.971 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:22:45.990 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:22:45.990 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:22:46.008 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:22:46.008 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:22:46.008 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:22:45 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:22:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:22:46.025 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:22:46.025 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:22:46.025 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:22:46.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:46.052 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:46.053 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:22:46.074 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:22:46.074 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:22:46.154 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:22:46.187 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:46.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:46.188 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:22:46.289 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:22:46.326 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:46.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:46.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:22:46.428 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:22:46.463 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:46.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:46.464 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:22:46.565 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:22:46.598 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:46.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:46.599 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:22:46.700 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:22:46.736 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:46.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:46.737 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:22:46.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:46 smithi135 ceph-mon[33238]: pgmap v966: 158 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 5 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 146 active+clean; 6.0 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.32k op/s; 46060/295452 objects degraded (15.590%); 24942/295452 objects misplaced (8.442%); 46 MiB/s, 730 objects/s recovering 2024-09-17T07:22:46.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:46 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1872084698' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:22:46.838 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:22:46.847 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:46 smithi100 ceph-mon[29092]: pgmap v966: 158 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 5 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 146 active+clean; 6.0 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.32k op/s; 46060/295452 objects degraded (15.590%); 24942/295452 objects misplaced (8.442%); 46 MiB/s, 730 objects/s recovering 2024-09-17T07:22:46.847 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:46 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1872084698' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:22:46.847 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:46 smithi100 ceph-mon[39462]: pgmap v966: 158 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 5 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 146 active+clean; 6.0 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.32k op/s; 46060/295452 objects degraded (15.590%); 24942/295452 objects misplaced (8.442%); 46 MiB/s, 730 objects/s recovering 2024-09-17T07:22:46.847 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:46 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1872084698' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:22:46.886 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:46.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:46.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:22:46.988 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:22:47.025 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:47.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:47.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:22:47.127 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:22:47.161 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:47.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:47.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:22:47.263 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:22:47.286 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:47.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:47.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:22:47.388 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:22:47.424 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:47.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:47.425 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:22:47.525 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:22:47.568 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:47.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:47.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:22:47.657 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T07:22:47.670 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:22:47.746 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:47.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:47.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:22:47.848 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:22:47.902 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:47.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:47.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:22:48.004 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:22:48.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:48.042 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:48.042 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:22:48.042 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:22:48.143 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:22:48.180 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:48.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:48.181 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:22:48.281 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:22:48.352 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:48.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:48.353 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:22:48.454 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:22:48.486 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:48.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:48.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:22:48.588 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:22:48.633 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:48.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:48.635 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:22:48.736 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:22:48.771 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:48.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:48.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:22:48.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:48 smithi135 ceph-mon[33238]: pgmap v967: 158 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 5 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 146 active+clean; 6.0 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 905 op/s; 46060/295452 objects degraded (15.590%); 24942/295452 objects misplaced (8.442%); 34 MiB/s, 540 objects/s recovering 2024-09-17T07:22:48.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:48 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:22:48.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:48 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:22:48.873 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:22:48.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:48 smithi100 ceph-mon[29092]: pgmap v967: 158 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 5 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 146 active+clean; 6.0 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 905 op/s; 46060/295452 objects degraded (15.590%); 24942/295452 objects misplaced (8.442%); 34 MiB/s, 540 objects/s recovering 2024-09-17T07:22:48.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:48 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:22:48.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:48 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:22:48.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:48 smithi100 ceph-mon[39462]: pgmap v967: 158 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 5 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 146 active+clean; 6.0 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 905 op/s; 46060/295452 objects degraded (15.590%); 24942/295452 objects misplaced (8.442%); 34 MiB/s, 540 objects/s recovering 2024-09-17T07:22:48.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:48 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:22:48.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:48 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:22:48.883 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:22:48.909 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:48.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:48.910 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:22:49.011 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:22:49.051 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:49.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:49.053 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:22:49.153 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:22:49.197 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:49.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:49.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:22:49.299 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:22:49.324 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:49.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:49.325 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:22:49.426 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:22:49.461 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:49.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:49.462 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:22:49.563 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:22:49.585 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:49.586 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:49 smithi135 ceph-mon[33238]: osdmap e443: 8 total, 8 up, 7 in 2024-09-17T07:22:49.586 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:49 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:22:49.586 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:49 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:22:49.586 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:22:49 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:22:49.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:49.587 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:22:49.688 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:22:49.724 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:49.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:49.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:22:49.826 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:22:49.852 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:49.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:49.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:22:49.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:49 smithi100 ceph-mon[29092]: osdmap e443: 8 total, 8 up, 7 in 2024-09-17T07:22:49.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:49 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:22:49.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:49 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:22:49.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:49 smithi100 ceph-mon[39462]: osdmap e443: 8 total, 8 up, 7 in 2024-09-17T07:22:49.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:49 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:22:49.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:49 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:22:49.955 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:22:49.982 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:49.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:49.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:22:50.084 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:22:50.110 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:50.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:50.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:22:50.210 INFO:teuthology.orchestra.run.smithi135.stdout:ERROR: (22) Invalid argument 2024-09-17T07:22:50.210 INFO:teuthology.orchestra.run.smithi135.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:22:50.212 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:22:50.234 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:50.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:50.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:22:50.336 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:22:50.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:50.370 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:50.370 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:22:50.471 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:22:50.509 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:50.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:50.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:22:50.611 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:22:50.636 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:50.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:50.637 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:22:50.729 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:50 smithi100 ceph-mon[29092]: pgmap v969: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 5 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 146 active+clean; 5.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.07k op/s; 43576/285453 objects degraded (15.266%); 23991/285453 objects misplaced (8.405%); 34 MiB/s, 543 objects/s recovering 2024-09-17T07:22:50.729 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:50 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 43576/285453 objects degraded (15.266%), 11 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-17T07:22:50.730 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:50 smithi100 ceph-mon[29092]: osdmap e444: 8 total, 8 up, 7 in 2024-09-17T07:22:50.730 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:50 smithi100 ceph-mon[39462]: pgmap v969: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 5 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 146 active+clean; 5.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.07k op/s; 43576/285453 objects degraded (15.266%); 23991/285453 objects misplaced (8.405%); 34 MiB/s, 543 objects/s recovering 2024-09-17T07:22:50.730 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:50 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 43576/285453 objects degraded (15.266%), 11 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-17T07:22:50.730 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:50 smithi100 ceph-mon[39462]: osdmap e444: 8 total, 8 up, 7 in 2024-09-17T07:22:50.738 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:22:50.767 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:50.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:50.768 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:22:50.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:50 smithi135 ceph-mon[33238]: pgmap v969: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 5 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 146 active+clean; 5.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.07k op/s; 43576/285453 objects degraded (15.266%); 23991/285453 objects misplaced (8.405%); 34 MiB/s, 543 objects/s recovering 2024-09-17T07:22:50.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:50 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 43576/285453 objects degraded (15.266%), 11 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-17T07:22:50.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:50 smithi135 ceph-mon[33238]: osdmap e444: 8 total, 8 up, 7 in 2024-09-17T07:22:50.869 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:22:50.895 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:50.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:50.896 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:22:50.996 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:22:51.021 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:51.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:51.022 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:22:51.123 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:22:51.150 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:51.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:51.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:22:51.252 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:22:51.279 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:51.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:51.280 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:22:51.381 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:22:51.412 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:51.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:51.413 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:22:51.514 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:22:51.540 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:51.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:51.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:22:51.642 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:22:51.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:51.680 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:51.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:22:51.781 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:22:51.810 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:51.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:51.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:22:51.912 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:22:51.943 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:51.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:51.944 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:22:52.045 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:22:52.072 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:52.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:52.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:22:52.174 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:22:52.197 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:52.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:52.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:22:52.299 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:22:52.324 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:52.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:52.325 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:22:52.426 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:22:52.455 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:52.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:52.456 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:22:52.557 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:22:52.596 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:52.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:52.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:22:52.602 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:52 smithi135 ceph-mon[33238]: osdmap e445: 8 total, 8 up, 7 in 2024-09-17T07:22:52.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:52 smithi100 ceph-mon[29092]: osdmap e445: 8 total, 8 up, 7 in 2024-09-17T07:22:52.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:52 smithi100 ceph-mon[39462]: osdmap e445: 8 total, 8 up, 7 in 2024-09-17T07:22:52.698 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:22:52.720 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:52.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:52.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:22:52.822 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:22:52.844 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:52.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:52.845 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:22:52.946 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:22:52.987 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-17T07:22:52.996 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:52.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:52.998 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:22:53.099 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:22:53.130 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:53.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:53.131 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:22:53.232 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:22:53.267 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:53.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:53.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:22:53.293 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:22:53 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-5[43919]: 2024-09-17T07:22:53.143+0000 7f1746194640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T07:22:53.293 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:22:53 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-5[43919]: 2024-09-17T07:22:53.143+0000 7f1746194640 -1 osd.5 446 *** Got signal Terminated *** 2024-09-17T07:22:53.293 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:22:53 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-5[43919]: 2024-09-17T07:22:53.143+0000 7f1746194640 -1 osd.5 446 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T07:22:53.369 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:22:53.382 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:22:53.406 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:53.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:53.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:22:53.508 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:22:53.536 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:53.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:53.537 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:22:53.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:53 smithi100 ceph-mon[29092]: pgmap v972: 158 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 5 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 5 active+undersized+degraded+remapped+backfilling, 146 active+clean; 5.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 873 op/s; 40921/279738 objects degraded (14.628%); 22655/279738 objects misplaced (8.099%); 29 MiB/s, 456 objects/s recovering 2024-09-17T07:22:53.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:53 smithi100 ceph-mon[29092]: osdmap e446: 8 total, 8 up, 7 in 2024-09-17T07:22:53.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:53 smithi100 ceph-mon[29092]: osd.5 marked itself down and dead 2024-09-17T07:22:53.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:53 smithi100 ceph-mon[39462]: pgmap v972: 158 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 5 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 5 active+undersized+degraded+remapped+backfilling, 146 active+clean; 5.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 873 op/s; 40921/279738 objects degraded (14.628%); 22655/279738 objects misplaced (8.099%); 29 MiB/s, 456 objects/s recovering 2024-09-17T07:22:53.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:53 smithi100 ceph-mon[39462]: osdmap e446: 8 total, 8 up, 7 in 2024-09-17T07:22:53.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:53 smithi100 ceph-mon[39462]: osd.5 marked itself down and dead 2024-09-17T07:22:53.638 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:22:53.653 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:53 smithi135 ceph-mon[33238]: pgmap v972: 158 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 5 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 5 active+undersized+degraded+remapped+backfilling, 146 active+clean; 5.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 873 op/s; 40921/279738 objects degraded (14.628%); 22655/279738 objects misplaced (8.099%); 29 MiB/s, 456 objects/s recovering 2024-09-17T07:22:53.653 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:53 smithi135 ceph-mon[33238]: osdmap e446: 8 total, 8 up, 7 in 2024-09-17T07:22:53.653 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:53 smithi135 ceph-mon[33238]: osd.5 marked itself down and dead 2024-09-17T07:22:53.666 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:53.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:53.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:22:53.769 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:22:53.793 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:53.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:53.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:22:53.894 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:22:53.922 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:53.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:53.923 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:22:54.024 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:22:54.051 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:54.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:54.052 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:22:54.153 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:22:54.181 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:54.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:54.182 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:22:54.283 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:22:54.293 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:22:54 smithi135 podman[150622]: 2024-09-17 07:22:54.152373754 +0000 UTC m=+2.183653414 container died e2ca96e95c4db67b3b47af8460a1dab95a063931b437f6fd0907c387c2b42220 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-5, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=reef-19e2a2b, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2) 2024-09-17T07:22:54.311 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:54.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:54.312 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:22:54.413 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:22:54.442 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:54.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:54.443 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:22:54.544 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:22:54.573 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:54.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:54.575 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:22:54.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:54 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:22:54.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:54 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:22:54.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:54 smithi100 ceph-mon[29092]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T07:22:54.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:54 smithi100 ceph-mon[29092]: osdmap e447: 8 total, 7 up, 7 in 2024-09-17T07:22:54.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:54 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:22:54.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:54 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:22:54.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:54 smithi100 ceph-mon[39462]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T07:22:54.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:54 smithi100 ceph-mon[39462]: osdmap e447: 8 total, 7 up, 7 in 2024-09-17T07:22:54.676 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:22:54.704 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:54.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:54.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:22:54.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:54 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:22:54.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:54 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:22:54.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:54 smithi135 ceph-mon[33238]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T07:22:54.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:54 smithi135 ceph-mon[33238]: osdmap e447: 8 total, 7 up, 7 in 2024-09-17T07:22:54.794 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:22:54 smithi135 podman[150622]: 2024-09-17 07:22:54.546223925 +0000 UTC m=+2.577503584 container cleanup e2ca96e95c4db67b3b47af8460a1dab95a063931b437f6fd0907c387c2b42220 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-5, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, RELEASE=reef-19e2a2b, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T07:22:54.794 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:22:54 smithi135 bash[150622]: ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-5 2024-09-17T07:22:54.806 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:22:54.833 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:54.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:54.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:22:54.935 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:22:54.959 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:54.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:54.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:22:55.061 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:22:55.088 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:55.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:55.090 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:22:55.191 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:22:55.217 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:55.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:55.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:22:55.319 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:22:55.348 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:55.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:55.349 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:22:55.450 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:22:55.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:55.476 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:55.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:22:55.481 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:22:55 smithi135 podman[150811]: 2024-09-17 07:22:55.356075577 +0000 UTC m=+1.294716930 container remove e2ca96e95c4db67b3b47af8460a1dab95a063931b437f6fd0907c387c2b42220 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=reef-19e2a2b, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T07:22:55.481 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:55 smithi135 ceph-mon[33238]: pgmap v975: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 remapped+peering, 18 stale+active+clean, 1 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 9 peering, 5 active+undersized+degraded+remapped+backfilling, 120 active+clean; 5.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 695 op/s; 39416/277107 objects degraded (14.224%); 22465/277107 objects misplaced (8.107%); 34 MiB/s, 546 objects/s recovering 2024-09-17T07:22:55.481 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:55 smithi135 ceph-mon[33238]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-17T07:22:55.482 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:55 smithi135 ceph-mon[33238]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-17T07:22:55.482 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:55 smithi135 ceph-mon[33238]: 11.7 continuing backfill to osd.1 from (424'11384,446'15872] 11:eed10a4d:::benchmark_data_smithi100_180298_object138834:head to 446'15872 2024-09-17T07:22:55.482 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:55 smithi135 ceph-mon[33238]: 11.3 continuing backfill to osd.1 from (424'11700,446'15950] 11:cb72e2be:::benchmark_data_smithi100_180298_object131752:head to 446'15950 2024-09-17T07:22:55.482 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:55 smithi135 ceph-mon[33238]: 11.3 continuing backfill to osd.7 from (424'11700,446'15950] 11:cb72e2be:::benchmark_data_smithi100_180298_object131752:head to 446'15950 2024-09-17T07:22:55.482 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:55 smithi135 ceph-mon[33238]: osdmap e448: 8 total, 7 up, 7 in 2024-09-17T07:22:55.482 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:55 smithi135 ceph-mon[33238]: 11.5 continuing backfill to osd.7 from (423'10750,446'15358] MIN to 446'15358 2024-09-17T07:22:55.482 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:55 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 39416/277107 objects degraded (14.224%), 9 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-17T07:22:55.577 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:22:55.601 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:55.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:55.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:22:55.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:55 smithi100 ceph-mon[39462]: pgmap v975: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 remapped+peering, 18 stale+active+clean, 1 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 9 peering, 5 active+undersized+degraded+remapped+backfilling, 120 active+clean; 5.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 695 op/s; 39416/277107 objects degraded (14.224%); 22465/277107 objects misplaced (8.107%); 34 MiB/s, 546 objects/s recovering 2024-09-17T07:22:55.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:55 smithi100 ceph-mon[39462]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-17T07:22:55.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:55 smithi100 ceph-mon[39462]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-17T07:22:55.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:55 smithi100 ceph-mon[39462]: 11.7 continuing backfill to osd.1 from (424'11384,446'15872] 11:eed10a4d:::benchmark_data_smithi100_180298_object138834:head to 446'15872 2024-09-17T07:22:55.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:55 smithi100 ceph-mon[39462]: 11.3 continuing backfill to osd.1 from (424'11700,446'15950] 11:cb72e2be:::benchmark_data_smithi100_180298_object131752:head to 446'15950 2024-09-17T07:22:55.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:55 smithi100 ceph-mon[39462]: 11.3 continuing backfill to osd.7 from (424'11700,446'15950] 11:cb72e2be:::benchmark_data_smithi100_180298_object131752:head to 446'15950 2024-09-17T07:22:55.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:55 smithi100 ceph-mon[39462]: osdmap e448: 8 total, 7 up, 7 in 2024-09-17T07:22:55.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:55 smithi100 ceph-mon[39462]: 11.5 continuing backfill to osd.7 from (423'10750,446'15358] MIN to 446'15358 2024-09-17T07:22:55.630 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:55 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 39416/277107 objects degraded (14.224%), 9 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-17T07:22:55.630 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:55 smithi100 ceph-mon[29092]: pgmap v975: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 remapped+peering, 18 stale+active+clean, 1 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 9 peering, 5 active+undersized+degraded+remapped+backfilling, 120 active+clean; 5.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 695 op/s; 39416/277107 objects degraded (14.224%); 22465/277107 objects misplaced (8.107%); 34 MiB/s, 546 objects/s recovering 2024-09-17T07:22:55.630 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:55 smithi100 ceph-mon[29092]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-17T07:22:55.630 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:55 smithi100 ceph-mon[29092]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-17T07:22:55.630 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:55 smithi100 ceph-mon[29092]: 11.7 continuing backfill to osd.1 from (424'11384,446'15872] 11:eed10a4d:::benchmark_data_smithi100_180298_object138834:head to 446'15872 2024-09-17T07:22:55.631 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:55 smithi100 ceph-mon[29092]: 11.3 continuing backfill to osd.1 from (424'11700,446'15950] 11:cb72e2be:::benchmark_data_smithi100_180298_object131752:head to 446'15950 2024-09-17T07:22:55.631 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:55 smithi100 ceph-mon[29092]: 11.3 continuing backfill to osd.7 from (424'11700,446'15950] 11:cb72e2be:::benchmark_data_smithi100_180298_object131752:head to 446'15950 2024-09-17T07:22:55.631 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:55 smithi100 ceph-mon[29092]: osdmap e448: 8 total, 7 up, 7 in 2024-09-17T07:22:55.631 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:55 smithi100 ceph-mon[29092]: 11.5 continuing backfill to osd.7 from (423'10750,446'15358] MIN to 446'15358 2024-09-17T07:22:55.631 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:55 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 39416/277107 objects degraded (14.224%), 9 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-17T07:22:55.703 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:22:55.730 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:55.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:55.731 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:22:55.832 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:22:55.859 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:55.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:55.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:22:55.892 INFO:teuthology.orchestra.run.smithi135.stdout:ERROR: (22) Invalid argument 2024-09-17T07:22:55.892 INFO:teuthology.orchestra.run.smithi135.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:22:55.961 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:22:55.987 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:55.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:55.988 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:22:56.089 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:22:56.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:56.118 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:56.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:22:56.219 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:22:56.246 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:56.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:56.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:22:56.348 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:22:56.376 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:22:55 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:22:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:22:56.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:56.378 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:56.379 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:22:56.479 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:22:56.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:56.502 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:56.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:22:56.600 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:22:56.603 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:22:56.632 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:56.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:56.633 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:22:56.734 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:22:56.760 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:56.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:56.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:22:56.862 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:22:56.885 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:56.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:56.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:22:56.987 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:22:57.014 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:57.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:57.015 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:22:57.116 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:22:57.136 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-17T07:22:57.143 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:57.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:57.144 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:22:57.244 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:22:57.275 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:57.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:57.276 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:22:57.376 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:22:57 smithi135 podman[151063]: 2024-09-17 07:22:57.176492738 +0000 UTC m=+0.018790242 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9 2024-09-17T07:22:57.376 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:22:57 smithi135 podman[151063]: 2024-09-17 07:22:57.291363342 +0000 UTC m=+0.133660817 container create f886ed6fac2b054514aa493d699675c5dd56e161d1d42cad4c6f7a53347f410b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-5-deactivate, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=reef-19e2a2b, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T07:22:57.377 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:22:57.403 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:57.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:57.404 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:22:57.505 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:22:57.525 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:57.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:57.527 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:22:57.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:57 smithi100 ceph-mon[29092]: pgmap v977: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait, 3 remapped+peering, 5 stale+active+clean, 5 active+undersized, 36 peering, 3 active+undersized+degraded+remapped+backfilling, 8 active+undersized+degraded, 94 active+clean; 5.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 21 KiB/s rd, 160 KiB/s wr, 575 op/s; 32768/272967 objects degraded (12.004%); 18721/272967 objects misplaced (6.858%); 78 MiB/s, 1.24k objects/s recovering 2024-09-17T07:22:57.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:57 smithi100 ceph-mon[39462]: pgmap v977: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait, 3 remapped+peering, 5 stale+active+clean, 5 active+undersized, 36 peering, 3 active+undersized+degraded+remapped+backfilling, 8 active+undersized+degraded, 94 active+clean; 5.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 21 KiB/s rd, 160 KiB/s wr, 575 op/s; 32768/272967 objects degraded (12.004%); 18721/272967 objects misplaced (6.858%); 78 MiB/s, 1.24k objects/s recovering 2024-09-17T07:22:57.627 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:22:57.654 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:57.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:57.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:22:57.756 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:22:57.783 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:57.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:57.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:22:57.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:57 smithi135 ceph-mon[33238]: pgmap v977: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait, 3 remapped+peering, 5 stale+active+clean, 5 active+undersized, 36 peering, 3 active+undersized+degraded+remapped+backfilling, 8 active+undersized+degraded, 94 active+clean; 5.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 21 KiB/s rd, 160 KiB/s wr, 575 op/s; 32768/272967 objects degraded (12.004%); 18721/272967 objects misplaced (6.858%); 78 MiB/s, 1.24k objects/s recovering 2024-09-17T07:22:57.793 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:22:57 smithi135 podman[151063]: 2024-09-17 07:22:57.544747597 +0000 UTC m=+0.387045084 container init f886ed6fac2b054514aa493d699675c5dd56e161d1d42cad4c6f7a53347f410b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-5-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, RELEASE=reef-19e2a2b) 2024-09-17T07:22:57.793 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:22:57 smithi135 podman[151063]: 2024-09-17 07:22:57.549348598 +0000 UTC m=+0.391646082 container start f886ed6fac2b054514aa493d699675c5dd56e161d1d42cad4c6f7a53347f410b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-5-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, RELEASE=reef-19e2a2b, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T07:22:57.885 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:22:57.909 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:57.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:57.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:22:58.011 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:22:58.039 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:58.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:58.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:22:58.141 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:22:58.169 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:58.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:58.170 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:22:58.271 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:22:58.293 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:22:57 smithi135 podman[151063]: 2024-09-17 07:22:57.980908395 +0000 UTC m=+0.823205882 container attach f886ed6fac2b054514aa493d699675c5dd56e161d1d42cad4c6f7a53347f410b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-5-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-19e2a2b, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD) 2024-09-17T07:22:58.299 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:58.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:58.300 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:22:58.401 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:22:58.422 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:58.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:58.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:22:58.524 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:22:58.552 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:58.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:58.553 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:22:58.654 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:22:58.676 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:58.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:58.678 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:22:58.779 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:22:58.806 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:58.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:58.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:22:58.908 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:22:58.931 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:58.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:58.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:22:59.032 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:22:59.054 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:59.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:59.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:22:59.155 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:22:59.177 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:59.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:59.178 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:22:59.279 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:22:59.293 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:22:58 smithi135 podman[151063]: 2024-09-17 07:22:58.854228447 +0000 UTC m=+1.696525928 container died f886ed6fac2b054514aa493d699675c5dd56e161d1d42cad4c6f7a53347f410b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-5-deactivate, RELEASE=reef-19e2a2b, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-09-17T07:22:59.307 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:59.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:59.309 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:22:59.390 INFO:teuthology.orchestra.run.smithi100.stderr:noscrub is unset 2024-09-17T07:22:59.410 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:22:59.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:59.443 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:59.443 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:22:59.544 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:22:59.573 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:59.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:59.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:22:59.599 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:22:59 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:22:59.599 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:59 smithi135 ceph-mon[33238]: pgmap v978: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait, 3 remapped+peering, 5 stale+active+clean, 5 active+undersized, 36 peering, 3 active+undersized+degraded+remapped+backfilling, 8 active+undersized+degraded, 94 active+clean; 5.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 15 KiB/s rd, 110 KiB/s wr, 394 op/s; 32768/272967 objects degraded (12.004%); 18721/272967 objects misplaced (6.858%); 53 MiB/s, 853 objects/s recovering 2024-09-17T07:22:59.599 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:59 smithi135 ceph-mon[33238]: osdmap e449: 8 total, 7 up, 7 in 2024-09-17T07:22:59.600 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:59 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1947919575' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:22:59.600 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:22:59 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:22:59.615 DEBUG:teuthology.orchestra.run.smithi135:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e848267c-74c1-11ef-bceb-c7b262605968@osd.5.service' 2024-09-17T07:22:59.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:59 smithi100 ceph-mon[29092]: pgmap v978: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait, 3 remapped+peering, 5 stale+active+clean, 5 active+undersized, 36 peering, 3 active+undersized+degraded+remapped+backfilling, 8 active+undersized+degraded, 94 active+clean; 5.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 15 KiB/s rd, 110 KiB/s wr, 394 op/s; 32768/272967 objects degraded (12.004%); 18721/272967 objects misplaced (6.858%); 53 MiB/s, 853 objects/s recovering 2024-09-17T07:22:59.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:59 smithi100 ceph-mon[29092]: osdmap e449: 8 total, 7 up, 7 in 2024-09-17T07:22:59.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:59 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1947919575' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:22:59.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:22:59 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:22:59.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:59 smithi100 ceph-mon[39462]: pgmap v978: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait, 3 remapped+peering, 5 stale+active+clean, 5 active+undersized, 36 peering, 3 active+undersized+degraded+remapped+backfilling, 8 active+undersized+degraded, 94 active+clean; 5.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 15 KiB/s rd, 110 KiB/s wr, 394 op/s; 32768/272967 objects degraded (12.004%); 18721/272967 objects misplaced (6.858%); 53 MiB/s, 853 objects/s recovering 2024-09-17T07:22:59.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:59 smithi100 ceph-mon[39462]: osdmap e449: 8 total, 7 up, 7 in 2024-09-17T07:22:59.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:59 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1947919575' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:22:59.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:22:59 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:22:59.669 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T07:22:59.669 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-09-17T07:22:59.669 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.5 2024-09-17T07:22:59.670 INFO:tasks.thrashosds.thrasher:remote for osd 5 is ubuntu@smithi135.front.sepia.ceph.com 2024-09-17T07:22:59.670 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -n osd.5 --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal --op list-pgs 2024-09-17T07:22:59.676 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:22:59.735 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:59.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:59.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:22:59.822 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/osd.5/config 2024-09-17T07:22:59.837 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:22:59.866 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:22:59.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:59.867 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:22:59.968 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:22:59.991 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:22:59.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:22:59.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:23:00.092 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:23:00.121 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:00.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:00.122 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:23:00.223 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:23:00.246 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:00.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:00.247 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:23:00.348 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:23:00.370 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:00.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:00.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:23:00.472 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:23:00.502 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:00.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:00.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:23:00.560 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:00 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:23:00.561 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:00 smithi100 ceph-mon[29092]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:23:00.561 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:00 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T07:23:00.561 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:00 smithi100 ceph-mon[29092]: osdmap e450: 8 total, 7 up, 7 in 2024-09-17T07:23:00.561 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:00 smithi100 ceph-mon[29092]: 2.11 scrub starts 2024-09-17T07:23:00.561 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:00 smithi100 ceph-mon[29092]: Health check update: Reduced data availability: 10 pgs inactive, 13 pgs peering (PG_AVAILABILITY) 2024-09-17T07:23:00.561 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:00 smithi100 ceph-mon[29092]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-17T07:23:00.561 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:00 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 32768/272967 objects degraded (12.004%), 15 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-17T07:23:00.561 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:00 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:23:00.561 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:00 smithi100 ceph-mon[39462]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:23:00.562 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:00 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T07:23:00.562 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:00 smithi100 ceph-mon[39462]: osdmap e450: 8 total, 7 up, 7 in 2024-09-17T07:23:00.562 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:00 smithi100 ceph-mon[39462]: 2.11 scrub starts 2024-09-17T07:23:00.562 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:00 smithi100 ceph-mon[39462]: Health check update: Reduced data availability: 10 pgs inactive, 13 pgs peering (PG_AVAILABILITY) 2024-09-17T07:23:00.562 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:00 smithi100 ceph-mon[39462]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-17T07:23:00.562 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:00 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 32768/272967 objects degraded (12.004%), 15 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-17T07:23:00.604 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:23:00.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:00.629 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:00.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:23:00.730 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:23:00.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:00.761 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:00.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:23:00.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:00 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:23:00.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:00 smithi135 ceph-mon[33238]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:23:00.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:00 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T07:23:00.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:00 smithi135 ceph-mon[33238]: osdmap e450: 8 total, 7 up, 7 in 2024-09-17T07:23:00.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:00 smithi135 ceph-mon[33238]: 2.11 scrub starts 2024-09-17T07:23:00.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:00 smithi135 ceph-mon[33238]: Health check update: Reduced data availability: 10 pgs inactive, 13 pgs peering (PG_AVAILABILITY) 2024-09-17T07:23:00.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:00 smithi135 ceph-mon[33238]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-17T07:23:00.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:00 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 32768/272967 objects degraded (12.004%), 15 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-17T07:23:00.862 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:23:00.885 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:00.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:00.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:23:00.987 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:23:01.015 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:01.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:01.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:23:01.117 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:23:01.139 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:01.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:01.140 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:23:01.240 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:23:01.263 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:01.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:01.264 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:23:01.364 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:23:01.393 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:01.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:01.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:23:01.469 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:01 smithi135 ceph-mon[33238]: pgmap v981: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 2 undersized+degraded+remapped+backfill_toofull+peered, 2 active+undersized+degraded+remapped+backfill_wait, 41 active+undersized, 1 peering, 3 active+undersized+degraded+remapped+backfilling, 23 active+undersized+degraded, 84 active+clean; 5.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 15 KiB/s rd, 110 KiB/s wr, 257 op/s; 53645/272811 objects degraded (19.664%); 18720/272811 objects misplaced (6.862%); 87 MiB/s, 1.39k objects/s recovering 2024-09-17T07:23:01.470 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:01 smithi135 ceph-mon[33238]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 10 pgs inactive, 13 pgs peering) 2024-09-17T07:23:01.495 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:23:01.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:01.522 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:01.522 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:23:01.623 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:23:01.646 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:01.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:01.647 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:23:01.748 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:23:01.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:01.770 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:01.770 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:23:01.789 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:01 smithi100 ceph-mon[29092]: pgmap v981: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 2 undersized+degraded+remapped+backfill_toofull+peered, 2 active+undersized+degraded+remapped+backfill_wait, 41 active+undersized, 1 peering, 3 active+undersized+degraded+remapped+backfilling, 23 active+undersized+degraded, 84 active+clean; 5.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 15 KiB/s rd, 110 KiB/s wr, 257 op/s; 53645/272811 objects degraded (19.664%); 18720/272811 objects misplaced (6.862%); 87 MiB/s, 1.39k objects/s recovering 2024-09-17T07:23:01.789 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:01 smithi100 ceph-mon[29092]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 10 pgs inactive, 13 pgs peering) 2024-09-17T07:23:01.789 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:01 smithi100 ceph-mon[39462]: pgmap v981: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 2 undersized+degraded+remapped+backfill_toofull+peered, 2 active+undersized+degraded+remapped+backfill_wait, 41 active+undersized, 1 peering, 3 active+undersized+degraded+remapped+backfilling, 23 active+undersized+degraded, 84 active+clean; 5.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 15 KiB/s rd, 110 KiB/s wr, 257 op/s; 53645/272811 objects degraded (19.664%); 18720/272811 objects misplaced (6.862%); 87 MiB/s, 1.39k objects/s recovering 2024-09-17T07:23:01.789 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:01 smithi100 ceph-mon[39462]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 10 pgs inactive, 13 pgs peering) 2024-09-17T07:23:01.871 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:23:01.893 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:01.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:01.894 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:23:01.995 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:23:02.021 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:02.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:02.022 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:23:02.123 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:23:02.150 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:02.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:02.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:23:02.252 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:23:02.275 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:02.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:02.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:23:02.377 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:23:02.405 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:02.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:02.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:23:02.507 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:23:02.534 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:02.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:02.535 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:23:02.592 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-17T07:23:02.636 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:23:02.667 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:02.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:02.668 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:23:02.691 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:02 smithi100 ceph-mon[29092]: 6.e scrub starts 2024-09-17T07:23:02.691 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:02 smithi100 ceph-mon[39462]: 6.e scrub starts 2024-09-17T07:23:02.770 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:23:02.791 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:02.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:02.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:23:02.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:02 smithi135 ceph-mon[33238]: 6.e scrub starts 2024-09-17T07:23:02.893 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:23:02.921 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:02.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:02.922 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:23:03.023 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:23:03.046 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:03.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:03.047 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:23:03.148 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:23:03.169 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:03.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:03.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:23:03.271 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:23:03.299 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:03.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:03.300 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:23:03.401 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:23:03.429 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:03.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:03.430 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:23:03.531 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:23:03.554 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:03.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:03.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:23:03.656 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:23:03.696 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:03.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:03.697 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:23:03.701 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:03 smithi135 ceph-mon[33238]: 4.1b scrub starts 2024-09-17T07:23:03.701 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:03 smithi135 ceph-mon[33238]: pgmap v982: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 undersized+degraded+remapped+backfill_toofull+peered, 2 active+undersized+degraded+remapped+backfill_wait, 41 active+undersized, 1 peering, 4 active+undersized+degraded+remapped+backfilling, 23 active+undersized+degraded, 84 active+clean; 5.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 4.4 KiB/s rd, 31 KiB/s wr, 87 op/s; 48404/272805 objects degraded (17.743%); 17891/272805 objects misplaced (6.558%); 109 MiB/s, 1.75k objects/s recovering 2024-09-17T07:23:03.701 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:03 smithi135 ceph-mon[33238]: 2.19 scrub starts 2024-09-17T07:23:03.701 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:03 smithi135 ceph-mon[33238]: 4.e scrub starts 2024-09-17T07:23:03.798 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:23:03.821 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:03.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:03.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:23:03.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:03 smithi100 ceph-mon[29092]: 4.1b scrub starts 2024-09-17T07:23:03.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:03 smithi100 ceph-mon[29092]: pgmap v982: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 undersized+degraded+remapped+backfill_toofull+peered, 2 active+undersized+degraded+remapped+backfill_wait, 41 active+undersized, 1 peering, 4 active+undersized+degraded+remapped+backfilling, 23 active+undersized+degraded, 84 active+clean; 5.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 4.4 KiB/s rd, 31 KiB/s wr, 87 op/s; 48404/272805 objects degraded (17.743%); 17891/272805 objects misplaced (6.558%); 109 MiB/s, 1.75k objects/s recovering 2024-09-17T07:23:03.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:03 smithi100 ceph-mon[29092]: 2.19 scrub starts 2024-09-17T07:23:03.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:03 smithi100 ceph-mon[29092]: 4.e scrub starts 2024-09-17T07:23:03.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:03 smithi100 ceph-mon[39462]: 4.1b scrub starts 2024-09-17T07:23:03.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:03 smithi100 ceph-mon[39462]: pgmap v982: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 undersized+degraded+remapped+backfill_toofull+peered, 2 active+undersized+degraded+remapped+backfill_wait, 41 active+undersized, 1 peering, 4 active+undersized+degraded+remapped+backfilling, 23 active+undersized+degraded, 84 active+clean; 5.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 4.4 KiB/s rd, 31 KiB/s wr, 87 op/s; 48404/272805 objects degraded (17.743%); 17891/272805 objects misplaced (6.558%); 109 MiB/s, 1.75k objects/s recovering 2024-09-17T07:23:03.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:03 smithi100 ceph-mon[39462]: 2.19 scrub starts 2024-09-17T07:23:03.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:03 smithi100 ceph-mon[39462]: 4.e scrub starts 2024-09-17T07:23:03.923 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:23:03.951 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:03.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:03.952 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:23:04.053 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:23:04.075 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:04.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:04.076 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:23:04.177 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:23:04.199 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:04.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:04.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:23:04.301 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:23:04.323 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:04.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:04.324 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:23:04.425 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:23:04.446 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:04.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:04.447 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:23:04.548 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:23:04.570 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:04.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:04.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:23:04.672 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:23:04.693 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:04.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:04.694 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:23:04.795 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:23:04.825 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:04.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:04.826 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:23:04.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:04 smithi100 ceph-mon[29092]: pgmap v983: 158 pgs: 3 active+clean+scrubbing, 1 active+undersized+degraded+remapped+backfill_toofull, 2 undersized+degraded+remapped+backfill_toofull+peered, 2 active+undersized+degraded+remapped+backfill_wait, 41 active+undersized, 1 peering, 4 active+undersized+degraded+remapped+backfilling, 23 active+undersized+degraded, 81 active+clean; 5.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 383 B/s rd, 0 B/s wr, 7 op/s; 46269/272796 objects degraded (16.961%); 17891/272796 objects misplaced (6.558%); 88 MiB/s, 1.41k objects/s recovering 2024-09-17T07:23:04.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:04 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:23:04.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:04 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:23:04.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:04 smithi100 ceph-mon[39462]: pgmap v983: 158 pgs: 3 active+clean+scrubbing, 1 active+undersized+degraded+remapped+backfill_toofull, 2 undersized+degraded+remapped+backfill_toofull+peered, 2 active+undersized+degraded+remapped+backfill_wait, 41 active+undersized, 1 peering, 4 active+undersized+degraded+remapped+backfilling, 23 active+undersized+degraded, 81 active+clean; 5.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 383 B/s rd, 0 B/s wr, 7 op/s; 46269/272796 objects degraded (16.961%); 17891/272796 objects misplaced (6.558%); 88 MiB/s, 1.41k objects/s recovering 2024-09-17T07:23:04.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:04 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:23:04.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:04 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:23:04.927 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:23:04.948 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:04.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:04.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:23:05.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:04 smithi135 ceph-mon[33238]: pgmap v983: 158 pgs: 3 active+clean+scrubbing, 1 active+undersized+degraded+remapped+backfill_toofull, 2 undersized+degraded+remapped+backfill_toofull+peered, 2 active+undersized+degraded+remapped+backfill_wait, 41 active+undersized, 1 peering, 4 active+undersized+degraded+remapped+backfilling, 23 active+undersized+degraded, 81 active+clean; 5.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 383 B/s rd, 0 B/s wr, 7 op/s; 46269/272796 objects degraded (16.961%); 17891/272796 objects misplaced (6.558%); 88 MiB/s, 1.41k objects/s recovering 2024-09-17T07:23:05.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:04 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:23:05.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:04 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:23:05.050 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:23:05.075 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:05.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:05.076 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:23:05.177 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:23:05.198 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:05.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:05.199 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:23:05.300 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:23:05.323 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:05.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:05.325 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:23:05.425 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:23:05.451 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:05.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:05.452 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:23:05.553 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:23:05.574 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:05.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:05.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:23:05.676 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:23:05.698 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:05.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:05.699 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:23:05.800 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:23:05.825 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:05.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:05.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:23:05.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:05 smithi100 ceph-mon[29092]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-17T07:23:05.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:05 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 46269/272796 objects degraded (16.961%), 32 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-17T07:23:05.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:05 smithi100 ceph-mon[29092]: osdmap e451: 8 total, 7 up, 7 in 2024-09-17T07:23:05.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:05 smithi100 ceph-mon[39462]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-17T07:23:05.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:05 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 46269/272796 objects degraded (16.961%), 32 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-17T07:23:05.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:05 smithi100 ceph-mon[39462]: osdmap e451: 8 total, 7 up, 7 in 2024-09-17T07:23:05.927 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:23:05.948 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:05.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:05.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:23:06.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:05 smithi135 ceph-mon[33238]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-17T07:23:06.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:05 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 46269/272796 objects degraded (16.961%), 32 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-17T07:23:06.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:05 smithi135 ceph-mon[33238]: osdmap e451: 8 total, 7 up, 7 in 2024-09-17T07:23:06.050 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:23:06.073 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:23:06.078 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:06.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:06.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:23:06.179 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:23:06.203 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:23:05 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:23:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:23:06.206 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:06.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:06.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:23:06.308 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:23:06.331 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:06.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:06.333 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:23:06.434 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:23:06.455 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:06.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:06.456 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:23:06.557 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:23:06.579 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:06.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:06.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:23:06.681 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:23:06.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:06.702 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:06.702 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:23:06.803 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:23:06.826 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:06.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:06.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:23:06.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:06 smithi100 ceph-mon[29092]: 2.10 scrub starts 2024-09-17T07:23:06.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:06 smithi100 ceph-mon[29092]: pgmap v985: 158 pgs: 1 active+undersized+remapped, 5 active+clean+scrubbing, 3 active+undersized+degraded+remapped+backfill_toofull, 2 undersized+degraded+remapped+backfill_toofull+peered, 41 active+undersized, 3 active+undersized+degraded+remapped+backfilling, 23 active+undersized+degraded, 80 active+clean; 5.5 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 5 op/s; 40970/272778 objects degraded (15.020%); 16282/272778 objects misplaced (5.969%); 126 MiB/s, 2.01k objects/s recovering 2024-09-17T07:23:06.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:06 smithi100 ceph-mon[29092]: 4.c scrub starts 2024-09-17T07:23:06.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:06 smithi100 ceph-mon[29092]: osdmap e452: 8 total, 7 up, 7 in 2024-09-17T07:23:06.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:06 smithi100 ceph-mon[39462]: 2.10 scrub starts 2024-09-17T07:23:06.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:06 smithi100 ceph-mon[39462]: pgmap v985: 158 pgs: 1 active+undersized+remapped, 5 active+clean+scrubbing, 3 active+undersized+degraded+remapped+backfill_toofull, 2 undersized+degraded+remapped+backfill_toofull+peered, 41 active+undersized, 3 active+undersized+degraded+remapped+backfilling, 23 active+undersized+degraded, 80 active+clean; 5.5 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 5 op/s; 40970/272778 objects degraded (15.020%); 16282/272778 objects misplaced (5.969%); 126 MiB/s, 2.01k objects/s recovering 2024-09-17T07:23:06.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:06 smithi100 ceph-mon[39462]: 4.c scrub starts 2024-09-17T07:23:06.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:06 smithi100 ceph-mon[39462]: osdmap e452: 8 total, 7 up, 7 in 2024-09-17T07:23:06.928 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:23:06.950 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:06.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:06.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:23:07.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:06 smithi135 ceph-mon[33238]: 2.10 scrub starts 2024-09-17T07:23:07.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:06 smithi135 ceph-mon[33238]: pgmap v985: 158 pgs: 1 active+undersized+remapped, 5 active+clean+scrubbing, 3 active+undersized+degraded+remapped+backfill_toofull, 2 undersized+degraded+remapped+backfill_toofull+peered, 41 active+undersized, 3 active+undersized+degraded+remapped+backfilling, 23 active+undersized+degraded, 80 active+clean; 5.5 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 5 op/s; 40970/272778 objects degraded (15.020%); 16282/272778 objects misplaced (5.969%); 126 MiB/s, 2.01k objects/s recovering 2024-09-17T07:23:07.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:06 smithi135 ceph-mon[33238]: 4.c scrub starts 2024-09-17T07:23:07.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:06 smithi135 ceph-mon[33238]: osdmap e452: 8 total, 7 up, 7 in 2024-09-17T07:23:07.052 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:23:07.074 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:07.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:07.076 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:23:07.177 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:23:07.200 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:07.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:07.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:23:07.302 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:23:07.324 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:07.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:07.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:23:07.426 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:23:07.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:07.448 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:07.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:23:07.550 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:23:07.572 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:07.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:07.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:23:07.674 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:23:07.698 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:07.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:07.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:23:07.801 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:23:07.823 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:07.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:07.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:23:07.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:07 smithi100 ceph-mon[29092]: 5.1f scrub starts 2024-09-17T07:23:07.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:07 smithi100 ceph-mon[29092]: 4.18 scrub starts 2024-09-17T07:23:07.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:07 smithi100 ceph-mon[29092]: osdmap e453: 8 total, 7 up, 7 in 2024-09-17T07:23:07.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:07 smithi100 ceph-mon[39462]: 5.1f scrub starts 2024-09-17T07:23:07.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:07 smithi100 ceph-mon[39462]: 4.18 scrub starts 2024-09-17T07:23:07.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:07 smithi100 ceph-mon[39462]: osdmap e453: 8 total, 7 up, 7 in 2024-09-17T07:23:07.925 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:23:07.950 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:07.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:07.951 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:23:08.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:07 smithi135 ceph-mon[33238]: 5.1f scrub starts 2024-09-17T07:23:08.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:07 smithi135 ceph-mon[33238]: 4.18 scrub starts 2024-09-17T07:23:08.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:07 smithi135 ceph-mon[33238]: osdmap e453: 8 total, 7 up, 7 in 2024-09-17T07:23:08.052 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:23:08.074 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:08.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:08.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:23:08.176 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:23:08.200 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:08.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:08.201 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:23:08.301 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:23:08.324 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:08.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:08.325 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:23:08.426 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:23:08.450 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:08.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:08.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:23:08.552 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:23:08.574 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:08.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:08.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:23:08.676 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:23:08.700 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:08.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:08.701 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:23:08.802 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:23:08.824 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:08.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:08.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:23:08.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:08 smithi100 ceph-mon[29092]: pgmap v988: 158 pgs: 1 active+undersized+remapped, 5 active+clean+scrubbing, 3 active+undersized+degraded+remapped+backfill_toofull, 2 undersized+degraded+remapped+backfill_toofull+peered, 41 active+undersized, 3 active+undersized+degraded+remapped+backfilling, 23 active+undersized+degraded, 80 active+clean; 5.5 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 2 op/s; 40970/272778 objects degraded (15.020%); 16282/272778 objects misplaced (5.969%); 77 MiB/s, 1.24k objects/s recovering 2024-09-17T07:23:08.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:08 smithi100 ceph-mon[29092]: 5.11 scrub starts 2024-09-17T07:23:08.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:08 smithi100 ceph-mon[29092]: osdmap e454: 8 total, 7 up, 7 in 2024-09-17T07:23:08.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:08 smithi100 ceph-mon[39462]: pgmap v988: 158 pgs: 1 active+undersized+remapped, 5 active+clean+scrubbing, 3 active+undersized+degraded+remapped+backfill_toofull, 2 undersized+degraded+remapped+backfill_toofull+peered, 41 active+undersized, 3 active+undersized+degraded+remapped+backfilling, 23 active+undersized+degraded, 80 active+clean; 5.5 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 2 op/s; 40970/272778 objects degraded (15.020%); 16282/272778 objects misplaced (5.969%); 77 MiB/s, 1.24k objects/s recovering 2024-09-17T07:23:08.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:08 smithi100 ceph-mon[39462]: 5.11 scrub starts 2024-09-17T07:23:08.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:08 smithi100 ceph-mon[39462]: osdmap e454: 8 total, 7 up, 7 in 2024-09-17T07:23:08.925 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:23:08.944 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:08 smithi135 ceph-mon[33238]: pgmap v988: 158 pgs: 1 active+undersized+remapped, 5 active+clean+scrubbing, 3 active+undersized+degraded+remapped+backfill_toofull, 2 undersized+degraded+remapped+backfill_toofull+peered, 41 active+undersized, 3 active+undersized+degraded+remapped+backfilling, 23 active+undersized+degraded, 80 active+clean; 5.5 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 2 op/s; 40970/272778 objects degraded (15.020%); 16282/272778 objects misplaced (5.969%); 77 MiB/s, 1.24k objects/s recovering 2024-09-17T07:23:08.944 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:08 smithi135 ceph-mon[33238]: 5.11 scrub starts 2024-09-17T07:23:08.944 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:08 smithi135 ceph-mon[33238]: osdmap e454: 8 total, 7 up, 7 in 2024-09-17T07:23:08.947 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:08.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:08.948 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:23:09.049 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:23:09.072 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:09.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:09.073 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:23:09.173 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:23:09.196 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:09.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:09.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:23:09.298 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:23:09.320 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:09.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:09.321 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:23:09.422 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:23:09.446 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:09.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:09.447 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:23:09.547 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:23:09.570 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:09.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:09.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:23:09.672 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:23:09.694 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:09.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:09.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:23:09.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:09 smithi135 ceph-mon[33238]: 6.14 scrub starts 2024-09-17T07:23:09.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:09 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:23:09.793 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:23:09 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:23:09.796 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:23:09.818 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:09.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:09.819 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:23:09.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:09 smithi100 ceph-mon[29092]: 6.14 scrub starts 2024-09-17T07:23:09.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:09 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:23:09.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:09 smithi100 ceph-mon[39462]: 6.14 scrub starts 2024-09-17T07:23:09.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:09 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:23:09.919 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:23:09.942 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:09.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:09.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:23:10.044 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:23:10.066 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:10.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:10.067 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:23:10.168 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:23:10.190 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:10.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:10.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:23:10.291 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:23:10.314 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:10.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:10.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:23:10.342 INFO:teuthology.orchestra.run.smithi135.stdout:6.d 2024-09-17T07:23:10.342 INFO:teuthology.orchestra.run.smithi135.stdout:6.c 2024-09-17T07:23:10.342 INFO:teuthology.orchestra.run.smithi135.stdout:6.6 2024-09-17T07:23:10.343 INFO:teuthology.orchestra.run.smithi135.stdout:6.4 2024-09-17T07:23:10.343 INFO:teuthology.orchestra.run.smithi135.stdout:6.1f 2024-09-17T07:23:10.343 INFO:teuthology.orchestra.run.smithi135.stdout:6.1e 2024-09-17T07:23:10.343 INFO:teuthology.orchestra.run.smithi135.stdout:4.11 2024-09-17T07:23:10.343 INFO:teuthology.orchestra.run.smithi135.stdout:2.c 2024-09-17T07:23:10.343 INFO:teuthology.orchestra.run.smithi135.stdout:4.19 2024-09-17T07:23:10.343 INFO:teuthology.orchestra.run.smithi135.stdout:11.7 2024-09-17T07:23:10.343 INFO:teuthology.orchestra.run.smithi135.stdout:2.16 2024-09-17T07:23:10.343 INFO:teuthology.orchestra.run.smithi135.stdout:2.15 2024-09-17T07:23:10.343 INFO:teuthology.orchestra.run.smithi135.stdout:2.a 2024-09-17T07:23:10.343 INFO:teuthology.orchestra.run.smithi135.stdout:11.5 2024-09-17T07:23:10.343 INFO:teuthology.orchestra.run.smithi135.stdout:5.15 2024-09-17T07:23:10.343 INFO:teuthology.orchestra.run.smithi135.stdout:5.7 2024-09-17T07:23:10.344 INFO:teuthology.orchestra.run.smithi135.stdout:1.1 2024-09-17T07:23:10.344 INFO:teuthology.orchestra.run.smithi135.stdout:6.19 2024-09-17T07:23:10.344 INFO:teuthology.orchestra.run.smithi135.stdout:5.13 2024-09-17T07:23:10.344 INFO:teuthology.orchestra.run.smithi135.stdout:5.b 2024-09-17T07:23:10.344 INFO:teuthology.orchestra.run.smithi135.stdout:2.1e 2024-09-17T07:23:10.344 INFO:teuthology.orchestra.run.smithi135.stdout:1.7 2024-09-17T07:23:10.344 INFO:teuthology.orchestra.run.smithi135.stdout:5.6 2024-09-17T07:23:10.344 INFO:teuthology.orchestra.run.smithi135.stdout:5.3 2024-09-17T07:23:10.344 INFO:teuthology.orchestra.run.smithi135.stdout:1.a 2024-09-17T07:23:10.344 INFO:teuthology.orchestra.run.smithi135.stdout:1.9 2024-09-17T07:23:10.344 INFO:teuthology.orchestra.run.smithi135.stdout:11.3 2024-09-17T07:23:10.344 INFO:teuthology.orchestra.run.smithi135.stdout:2.2 2024-09-17T07:23:10.345 INFO:teuthology.orchestra.run.smithi135.stdout:5.19 2024-09-17T07:23:10.345 INFO:teuthology.orchestra.run.smithi135.stdout:2.3 2024-09-17T07:23:10.345 INFO:teuthology.orchestra.run.smithi135.stdout:2.1c 2024-09-17T07:23:10.345 INFO:teuthology.orchestra.run.smithi135.stdout:3.2 2024-09-17T07:23:10.345 INFO:teuthology.orchestra.run.smithi135.stdout:11.8 2024-09-17T07:23:10.345 INFO:teuthology.orchestra.run.smithi135.stdout:3.1 2024-09-17T07:23:10.345 INFO:teuthology.orchestra.run.smithi135.stdout:4.15 2024-09-17T07:23:10.345 INFO:teuthology.orchestra.run.smithi135.stdout:5.1b 2024-09-17T07:23:10.345 INFO:teuthology.orchestra.run.smithi135.stdout:6.a 2024-09-17T07:23:10.345 INFO:teuthology.orchestra.run.smithi135.stdout:6.1d 2024-09-17T07:23:10.345 INFO:teuthology.orchestra.run.smithi135.stdout:2.18 2024-09-17T07:23:10.345 INFO:teuthology.orchestra.run.smithi135.stdout:11.a 2024-09-17T07:23:10.345 INFO:teuthology.orchestra.run.smithi135.stdout:5.e 2024-09-17T07:23:10.346 INFO:teuthology.orchestra.run.smithi135.stdout:5.16 2024-09-17T07:23:10.346 INFO:teuthology.orchestra.run.smithi135.stdout:2.12 2024-09-17T07:23:10.346 INFO:teuthology.orchestra.run.smithi135.stdout:1.4 2024-09-17T07:23:10.346 INFO:teuthology.orchestra.run.smithi135.stdout:4.1 2024-09-17T07:23:10.346 INFO:teuthology.orchestra.run.smithi135.stdout:2.17 2024-09-17T07:23:10.346 INFO:teuthology.orchestra.run.smithi135.stdout:4.1e 2024-09-17T07:23:10.346 INFO:teuthology.orchestra.run.smithi135.stdout:4.1f 2024-09-17T07:23:10.346 INFO:teuthology.orchestra.run.smithi135.stdout:4.3 2024-09-17T07:23:10.346 INFO:teuthology.orchestra.run.smithi135.stdout:4.6 2024-09-17T07:23:10.346 INFO:teuthology.orchestra.run.smithi135.stdout:5.4 2024-09-17T07:23:10.347 INFO:teuthology.orchestra.run.smithi135.stdout:4.7 2024-09-17T07:23:10.347 INFO:teuthology.orchestra.run.smithi135.stdout:6.7 2024-09-17T07:23:10.347 INFO:teuthology.orchestra.run.smithi135.stdout:5.12 2024-09-17T07:23:10.347 INFO:teuthology.orchestra.run.smithi135.stdout:6.1c 2024-09-17T07:23:10.347 INFO:teuthology.orchestra.run.smithi135.stdout:4.8 2024-09-17T07:23:10.347 INFO:teuthology.orchestra.run.smithi135.stdout:2.14 2024-09-17T07:23:10.347 INFO:teuthology.orchestra.run.smithi135.stdout:5.f 2024-09-17T07:23:10.347 INFO:teuthology.orchestra.run.smithi135.stdout:5.2 2024-09-17T07:23:10.347 INFO:teuthology.orchestra.run.smithi135.stdout:1.8 2024-09-17T07:23:10.347 INFO:teuthology.orchestra.run.smithi135.stdout:5.d 2024-09-17T07:23:10.347 INFO:teuthology.orchestra.run.smithi135.stdout:2.b 2024-09-17T07:23:10.347 INFO:teuthology.orchestra.run.smithi135.stdout:6.1a 2024-09-17T07:23:10.347 INFO:teuthology.orchestra.run.smithi135.stdout:6.0 2024-09-17T07:23:10.348 INFO:teuthology.orchestra.run.smithi135.stdout:4.2 2024-09-17T07:23:10.348 INFO:teuthology.orchestra.run.smithi135.stdout:6.10 2024-09-17T07:23:10.348 INFO:teuthology.orchestra.run.smithi135.stdout:6.11 2024-09-17T07:23:10.348 INFO:teuthology.orchestra.run.smithi135.stdout:6.17 2024-09-17T07:23:10.415 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:23:10.437 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:10.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:10.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:23:10.538 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:23:10.561 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:10.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:10.563 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:23:10.664 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:23:10.688 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:10.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:10.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:23:10.790 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:23:10.813 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:10.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:10.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:23:10.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:10 smithi100 ceph-mon[29092]: pgmap v990: 158 pgs: 1 active+undersized+remapped, 1 active+clean+scrubbing, 3 active+undersized+degraded+remapped+backfill_toofull, 2 undersized+degraded+remapped+backfill_toofull+peered, 41 active+undersized, 2 active+undersized+degraded+remapped+backfilling, 23 active+undersized+degraded, 85 active+clean; 5.5 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 40249/272778 objects degraded (14.755%); 16282/272778 objects misplaced (5.969%); 11 MiB/s, 174 objects/s recovering 2024-09-17T07:23:10.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:10 smithi100 ceph-mon[29092]: 5.18 scrub starts 2024-09-17T07:23:10.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:10 smithi100 ceph-mon[29092]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-17T07:23:10.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:10 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 40249/272778 objects degraded (14.755%), 30 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-17T07:23:10.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:10 smithi100 ceph-mon[39462]: pgmap v990: 158 pgs: 1 active+undersized+remapped, 1 active+clean+scrubbing, 3 active+undersized+degraded+remapped+backfill_toofull, 2 undersized+degraded+remapped+backfill_toofull+peered, 41 active+undersized, 2 active+undersized+degraded+remapped+backfilling, 23 active+undersized+degraded, 85 active+clean; 5.5 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 40249/272778 objects degraded (14.755%); 16282/272778 objects misplaced (5.969%); 11 MiB/s, 174 objects/s recovering 2024-09-17T07:23:10.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:10 smithi100 ceph-mon[39462]: 5.18 scrub starts 2024-09-17T07:23:10.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:10 smithi100 ceph-mon[39462]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-17T07:23:10.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:10 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 40249/272778 objects degraded (14.755%), 30 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-17T07:23:10.916 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:23:10.917 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:10 smithi135 ceph-mon[33238]: pgmap v990: 158 pgs: 1 active+undersized+remapped, 1 active+clean+scrubbing, 3 active+undersized+degraded+remapped+backfill_toofull, 2 undersized+degraded+remapped+backfill_toofull+peered, 41 active+undersized, 2 active+undersized+degraded+remapped+backfilling, 23 active+undersized+degraded, 85 active+clean; 5.5 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 40249/272778 objects degraded (14.755%); 16282/272778 objects misplaced (5.969%); 11 MiB/s, 174 objects/s recovering 2024-09-17T07:23:10.917 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:10 smithi135 ceph-mon[33238]: 5.18 scrub starts 2024-09-17T07:23:10.917 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:10 smithi135 ceph-mon[33238]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-17T07:23:10.917 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:10 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 40249/272778 objects degraded (14.755%), 30 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-17T07:23:10.939 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:10.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:10.940 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:23:11.041 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:23:11.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:11.066 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:11.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:23:11.167 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:23:11.188 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:11.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:11.190 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:23:11.291 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:23:11.313 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:11.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:11.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:23:11.415 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:23:11.436 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:11.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:11.437 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:23:11.539 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:23:11.561 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:11.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:11.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:23:11.663 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:23:11.684 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:11.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:11.685 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:23:11.742 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:11 smithi135 ceph-mon[33238]: 2.7 scrub starts 2024-09-17T07:23:11.767 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -n osd.5 --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal --op export --pgid 5.1b --file /var/log/ceph/exp.5.1b.5 2024-09-17T07:23:11.785 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:23:11.832 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:11.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:11.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:23:11.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:11 smithi100 ceph-mon[29092]: 2.7 scrub starts 2024-09-17T07:23:11.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:11 smithi100 ceph-mon[39462]: 2.7 scrub starts 2024-09-17T07:23:11.919 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/osd.5/config 2024-09-17T07:23:11.935 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:23:11.958 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:11.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:11.959 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:23:12.060 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:23:12.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:12.083 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:12.083 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:23:12.184 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:23:12.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:12.206 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:12.207 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:23:12.307 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:23:12.332 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:12.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:12.333 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:23:12.434 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:23:12.455 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:12.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:12.456 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:23:12.557 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:23:12.579 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:12.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:12.581 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:23:12.681 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:23:12.703 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:12.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:12.704 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:23:12.804 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:23:12.826 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:12.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:12.827 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:23:12.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:12 smithi100 ceph-mon[29092]: 4.10 scrub starts 2024-09-17T07:23:12.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:12 smithi100 ceph-mon[29092]: 4.10 scrub ok 2024-09-17T07:23:12.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:12 smithi100 ceph-mon[29092]: pgmap v991: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+clean+scrubbing, 3 active+undersized+degraded+remapped+backfill_toofull, 2 undersized+degraded+remapped+backfill_toofull+peered, 41 active+undersized, 23 active+undersized+degraded, 87 active+clean; 5.5 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 37124/272778 objects degraded (13.610%); 13910/272778 objects misplaced (5.099%); 40 MiB/s, 640 objects/s recovering 2024-09-17T07:23:12.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:12 smithi100 ceph-mon[39462]: 4.10 scrub starts 2024-09-17T07:23:12.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:12 smithi100 ceph-mon[39462]: 4.10 scrub ok 2024-09-17T07:23:12.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:12 smithi100 ceph-mon[39462]: pgmap v991: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+clean+scrubbing, 3 active+undersized+degraded+remapped+backfill_toofull, 2 undersized+degraded+remapped+backfill_toofull+peered, 41 active+undersized, 23 active+undersized+degraded, 87 active+clean; 5.5 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 37124/272778 objects degraded (13.610%); 13910/272778 objects misplaced (5.099%); 40 MiB/s, 640 objects/s recovering 2024-09-17T07:23:12.928 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:23:12.950 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:12.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:12.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:23:13.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:12 smithi135 ceph-mon[33238]: 4.10 scrub starts 2024-09-17T07:23:13.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:12 smithi135 ceph-mon[33238]: 4.10 scrub ok 2024-09-17T07:23:13.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:12 smithi135 ceph-mon[33238]: pgmap v991: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+clean+scrubbing, 3 active+undersized+degraded+remapped+backfill_toofull, 2 undersized+degraded+remapped+backfill_toofull+peered, 41 active+undersized, 23 active+undersized+degraded, 87 active+clean; 5.5 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 37124/272778 objects degraded (13.610%); 13910/272778 objects misplaced (5.099%); 40 MiB/s, 640 objects/s recovering 2024-09-17T07:23:13.052 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:23:13.075 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:13.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:13.076 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:23:13.177 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:23:13.199 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:13.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:13.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:23:13.244 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:23:13.245 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:23:13.250 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:23:13.254 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1: {} 2024-09-17T07:23:13.255 INFO:teuthology.orchestra.run.smithi100.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:23:13.280 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:23:13.281 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:23:13.296 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:23:13.297 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:23:13.301 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:23:13.312 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:23:13.313 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:23:13.313 INFO:teuthology.orchestra.run.smithi100.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-17T07:23:13.322 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:13.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:13.323 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:23:13.329 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:23:13.329 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:23:13.345 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:23:13.345 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:23:13.424 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:23:13.445 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:13.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:13.446 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:23:13.547 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:23:13.569 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:13.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:13.570 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:23:13.671 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:23:13.692 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:13.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:13.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:23:13.775 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:13 smithi100 ceph-mon[29092]: 2.10 scrub starts 2024-09-17T07:23:13.775 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:13 smithi100 ceph-mon[29092]: osdmap e455: 8 total, 7 up, 7 in 2024-09-17T07:23:13.775 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:13 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1453226578' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:23:13.776 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:13 smithi100 ceph-mon[39462]: 2.10 scrub starts 2024-09-17T07:23:13.776 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:13 smithi100 ceph-mon[39462]: osdmap e455: 8 total, 7 up, 7 in 2024-09-17T07:23:13.776 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:13 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1453226578' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:23:13.794 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:23:13.818 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:13.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:13.819 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:23:13.856 INFO:teuthology.orchestra.run.smithi135.stderr:Exporting 5.1b info 5.1b( empty local-lis/les=433/434 n=0 ec=71/71 lis/c=433/433 les/c/f=434/434/0 sis=433) 2024-09-17T07:23:13.862 INFO:teuthology.orchestra.run.smithi135.stderr:Export successful 2024-09-17T07:23:13.920 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:23:13.946 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:13.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:13.947 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:23:14.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:13 smithi135 ceph-mon[33238]: 2.10 scrub starts 2024-09-17T07:23:14.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:13 smithi135 ceph-mon[33238]: osdmap e455: 8 total, 7 up, 7 in 2024-09-17T07:23:14.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:13 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1453226578' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:23:14.049 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:23:14.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:14.074 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:14.074 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:23:14.175 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:23:14.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:14.198 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:14.199 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:23:14.299 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:23:14.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:14.324 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:14.325 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:23:14.425 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:23:14.449 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:14.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:14.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:23:14.512 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-17T07:23:14.513 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-17T07:23:14.513 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-17T07:23:14.513 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-17T07:23:14.550 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:23:14.574 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:14.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:14.575 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:23:14.676 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:23:14.699 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:14.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:14.700 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:23:14.741 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:14 smithi135 ceph-mon[33238]: 6.14 scrub starts 2024-09-17T07:23:14.741 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:14 smithi135 ceph-mon[33238]: 6.14 scrub ok 2024-09-17T07:23:14.742 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:14 smithi135 ceph-mon[33238]: pgmap v993: 158 pgs: 1 peering, 3 active+undersized+degraded+remapped+backfill_toofull, 2 undersized+degraded+remapped+backfill_toofull+peered, 41 active+undersized, 23 active+undersized+degraded, 88 active+clean; 5.5 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 838 B/s rd, 0 op/s; 36290/272778 objects degraded (13.304%); 13076/272778 objects misplaced (4.794%); 48 MiB/s, 765 objects/s recovering 2024-09-17T07:23:14.742 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:14 smithi135 ceph-mon[33238]: 11.1 scrub starts 2024-09-17T07:23:14.742 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:14 smithi135 ceph-mon[33238]: osdmap e456: 8 total, 7 up, 7 in 2024-09-17T07:23:14.742 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:14 smithi135 ceph-mon[33238]: 11.1 scrub ok 2024-09-17T07:23:14.801 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:23:14.824 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:14.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:14.824 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:23:14.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:14 smithi100 ceph-mon[29092]: 6.14 scrub starts 2024-09-17T07:23:14.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:14 smithi100 ceph-mon[29092]: 6.14 scrub ok 2024-09-17T07:23:14.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:14 smithi100 ceph-mon[29092]: pgmap v993: 158 pgs: 1 peering, 3 active+undersized+degraded+remapped+backfill_toofull, 2 undersized+degraded+remapped+backfill_toofull+peered, 41 active+undersized, 23 active+undersized+degraded, 88 active+clean; 5.5 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 838 B/s rd, 0 op/s; 36290/272778 objects degraded (13.304%); 13076/272778 objects misplaced (4.794%); 48 MiB/s, 765 objects/s recovering 2024-09-17T07:23:14.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:14 smithi100 ceph-mon[29092]: 11.1 scrub starts 2024-09-17T07:23:14.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:14 smithi100 ceph-mon[29092]: osdmap e456: 8 total, 7 up, 7 in 2024-09-17T07:23:14.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:14 smithi100 ceph-mon[29092]: 11.1 scrub ok 2024-09-17T07:23:14.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:14 smithi100 ceph-mon[39462]: 6.14 scrub starts 2024-09-17T07:23:14.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:14 smithi100 ceph-mon[39462]: 6.14 scrub ok 2024-09-17T07:23:14.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:14 smithi100 ceph-mon[39462]: pgmap v993: 158 pgs: 1 peering, 3 active+undersized+degraded+remapped+backfill_toofull, 2 undersized+degraded+remapped+backfill_toofull+peered, 41 active+undersized, 23 active+undersized+degraded, 88 active+clean; 5.5 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 838 B/s rd, 0 op/s; 36290/272778 objects degraded (13.304%); 13076/272778 objects misplaced (4.794%); 48 MiB/s, 765 objects/s recovering 2024-09-17T07:23:14.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:14 smithi100 ceph-mon[39462]: 11.1 scrub starts 2024-09-17T07:23:14.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:14 smithi100 ceph-mon[39462]: osdmap e456: 8 total, 7 up, 7 in 2024-09-17T07:23:14.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:14 smithi100 ceph-mon[39462]: 11.1 scrub ok 2024-09-17T07:23:14.926 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:23:14.948 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:14.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:14.949 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:23:15.050 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:23:15.072 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:15.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:15.073 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:23:15.173 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:23:15.195 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:15.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:15.196 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:23:15.236 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -n osd.5 --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal --force --op remove --pgid 5.1b 2024-09-17T07:23:15.292 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-17T07:23:15.296 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:23:15.324 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:15.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:15.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:23:15.416 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/osd.5/config 2024-09-17T07:23:15.426 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:23:15.449 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:15.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:15.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:23:15.551 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:23:15.572 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:15.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:15.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:23:15.674 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:23:15.695 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:15.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:15.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:23:15.756 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:15 smithi100 ceph-mon[29092]: 2.10 scrub starts 2024-09-17T07:23:15.756 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:15 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:23:15.756 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:15 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 36290/272778 objects degraded (13.304%), 28 pgs degraded (PG_DEGRADED) 2024-09-17T07:23:15.756 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:15 smithi100 ceph-mon[39462]: 2.10 scrub starts 2024-09-17T07:23:15.756 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:15 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:23:15.756 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:15 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 36290/272778 objects degraded (13.304%), 28 pgs degraded (PG_DEGRADED) 2024-09-17T07:23:15.780 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:23:15.781 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:23:15.797 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:23:15.812 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:15 smithi135 ceph-mon[33238]: 2.10 scrub starts 2024-09-17T07:23:15.812 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:15 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:23:15.812 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:15 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 36290/272778 objects degraded (13.304%), 28 pgs degraded (PG_DEGRADED) 2024-09-17T07:23:15.822 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:15.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:15.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:23:15.924 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:23:15.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:15.946 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:15.947 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:23:16.047 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:23:16.071 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:16.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:16.072 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:23:16.083 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:23:15 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:23:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:23:16.172 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:23:16.194 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:16.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:16.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:23:16.296 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:23:16.319 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:23:16.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:16.320 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:16.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:23:16.421 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:23:16.443 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:16.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:16.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:23:16.545 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:23:16.567 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:16.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:16.568 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:23:16.668 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:23:16.690 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:16.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:16.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:23:16.792 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:23:16.796 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:16 smithi100 ceph-mon[29092]: 4.b scrub starts 2024-09-17T07:23:16.796 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:16 smithi100 ceph-mon[29092]: pgmap v995: 158 pgs: 1 active+clean+scrubbing, 1 peering, 3 active+undersized+degraded+remapped+backfill_toofull, 2 undersized+degraded+remapped+backfill_toofull+peered, 41 active+undersized, 23 active+undersized+degraded, 87 active+clean; 5.5 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.4 KiB/s rd, 1 op/s; 36290/272778 objects degraded (13.304%); 13076/272778 objects misplaced (4.794%); 41 MiB/s, 658 objects/s recovering 2024-09-17T07:23:16.796 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:16 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:23:16.796 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:16 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:23:16.796 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:16 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:23:16.797 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:16 smithi100 ceph-mon[39462]: 4.b scrub starts 2024-09-17T07:23:16.797 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:16 smithi100 ceph-mon[39462]: pgmap v995: 158 pgs: 1 active+clean+scrubbing, 1 peering, 3 active+undersized+degraded+remapped+backfill_toofull, 2 undersized+degraded+remapped+backfill_toofull+peered, 41 active+undersized, 23 active+undersized+degraded, 87 active+clean; 5.5 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.4 KiB/s rd, 1 op/s; 36290/272778 objects degraded (13.304%); 13076/272778 objects misplaced (4.794%); 41 MiB/s, 658 objects/s recovering 2024-09-17T07:23:16.797 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:16 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:23:16.797 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:16 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:23:16.797 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:16 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:23:16.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:16.817 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:16.817 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:23:16.918 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:23:16.921 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:16 smithi135 ceph-mon[33238]: 4.b scrub starts 2024-09-17T07:23:16.921 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:16 smithi135 ceph-mon[33238]: pgmap v995: 158 pgs: 1 active+clean+scrubbing, 1 peering, 3 active+undersized+degraded+remapped+backfill_toofull, 2 undersized+degraded+remapped+backfill_toofull+peered, 41 active+undersized, 23 active+undersized+degraded, 87 active+clean; 5.5 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.4 KiB/s rd, 1 op/s; 36290/272778 objects degraded (13.304%); 13076/272778 objects misplaced (4.794%); 41 MiB/s, 658 objects/s recovering 2024-09-17T07:23:16.921 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:16 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:23:16.921 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:16 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:23:16.921 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:16 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:23:16.943 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:16.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:16.944 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:23:17.045 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:23:17.071 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:17.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:17.072 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:23:17.173 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:23:17.196 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:17.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:17.197 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:23:17.298 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:23:17.322 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:17.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:17.323 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:23:17.375 INFO:teuthology.orchestra.run.smithi135.stdout: marking collection for removal 2024-09-17T07:23:17.375 INFO:teuthology.orchestra.run.smithi135.stdout:setting '_remove' omap key 2024-09-17T07:23:17.375 INFO:teuthology.orchestra.run.smithi135.stdout:finish_remove_pgs 5.1b_head removing 5.1b 2024-09-17T07:23:17.409 INFO:teuthology.orchestra.run.smithi135.stdout:Remove successful 2024-09-17T07:23:17.423 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:23:17.447 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:17.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:17.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:23:17.549 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:23:17.571 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:17.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:17.572 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:23:17.627 INFO:teuthology.orchestra.run.smithi100.stderr:nodeep-scrub is unset 2024-09-17T07:23:17.673 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:23:17.692 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:17 smithi100 ceph-mon[29092]: 5.8 scrub starts 2024-09-17T07:23:17.693 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:17 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2323254356' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:23:17.693 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:17 smithi100 ceph-mon[39462]: 5.8 scrub starts 2024-09-17T07:23:17.693 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:17 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2323254356' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:23:17.695 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:17.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:17.696 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:23:17.782 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:23:17.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:17 smithi135 ceph-mon[33238]: 5.8 scrub starts 2024-09-17T07:23:17.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:17 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2323254356' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:23:17.797 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:23:17.848 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:17.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:17.849 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:23:17.950 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:23:17.971 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:17.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:17.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:23:18.073 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:23:18.095 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:18.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:18.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:23:18.197 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:23:18.218 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:18.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:18.219 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:23:18.320 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:23:18.341 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:18.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:18.342 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:23:18.443 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:23:18.467 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:18.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:18.468 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:23:18.569 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:23:18.592 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:18.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:18.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:23:18.694 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:23:18.719 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:18.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:18.720 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:23:18.738 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell -n osd.5 --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.5.1b.5 2024-09-17T07:23:18.794 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:18 smithi100 ceph-mon[29092]: 2.10 scrub starts 2024-09-17T07:23:18.794 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:18 smithi100 ceph-mon[29092]: pgmap v996: 158 pgs: 1 active+clean+scrubbing, 1 peering, 3 active+undersized+degraded+remapped+backfill_toofull, 2 undersized+degraded+remapped+backfill_toofull+peered, 41 active+undersized, 23 active+undersized+degraded, 87 active+clean; 5.5 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 639 B/s rd, 0 op/s; 36290/272778 objects degraded (13.304%); 13076/272778 objects misplaced (4.794%); 31 MiB/s, 494 objects/s recovering 2024-09-17T07:23:18.794 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:18 smithi100 ceph-mon[29092]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T07:23:18.794 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:18 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2323254356' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T07:23:18.794 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:18 smithi100 ceph-mon[29092]: osdmap e457: 8 total, 7 up, 7 in 2024-09-17T07:23:18.794 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:18 smithi100 ceph-mon[29092]: 11.3 scrub starts 2024-09-17T07:23:18.794 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:18 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:23:18.795 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:18 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:23:18.795 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:18 smithi100 ceph-mon[39462]: 2.10 scrub starts 2024-09-17T07:23:18.795 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:18 smithi100 ceph-mon[39462]: pgmap v996: 158 pgs: 1 active+clean+scrubbing, 1 peering, 3 active+undersized+degraded+remapped+backfill_toofull, 2 undersized+degraded+remapped+backfill_toofull+peered, 41 active+undersized, 23 active+undersized+degraded, 87 active+clean; 5.5 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 639 B/s rd, 0 op/s; 36290/272778 objects degraded (13.304%); 13076/272778 objects misplaced (4.794%); 31 MiB/s, 494 objects/s recovering 2024-09-17T07:23:18.795 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:18 smithi100 ceph-mon[39462]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T07:23:18.795 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:18 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2323254356' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T07:23:18.795 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:18 smithi100 ceph-mon[39462]: osdmap e457: 8 total, 7 up, 7 in 2024-09-17T07:23:18.795 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:18 smithi100 ceph-mon[39462]: 11.3 scrub starts 2024-09-17T07:23:18.795 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:18 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:23:18.795 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:18 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:23:18.821 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:23:18.847 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:18.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:18.848 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:23:18.946 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/osd.5/config 2024-09-17T07:23:18.949 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:23:18.977 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:18.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:18.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:23:19.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:18 smithi135 ceph-mon[33238]: 2.10 scrub starts 2024-09-17T07:23:19.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:18 smithi135 ceph-mon[33238]: pgmap v996: 158 pgs: 1 active+clean+scrubbing, 1 peering, 3 active+undersized+degraded+remapped+backfill_toofull, 2 undersized+degraded+remapped+backfill_toofull+peered, 41 active+undersized, 23 active+undersized+degraded, 87 active+clean; 5.5 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 639 B/s rd, 0 op/s; 36290/272778 objects degraded (13.304%); 13076/272778 objects misplaced (4.794%); 31 MiB/s, 494 objects/s recovering 2024-09-17T07:23:19.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:18 smithi135 ceph-mon[33238]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T07:23:19.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:18 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2323254356' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T07:23:19.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:18 smithi135 ceph-mon[33238]: osdmap e457: 8 total, 7 up, 7 in 2024-09-17T07:23:19.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:18 smithi135 ceph-mon[33238]: 11.3 scrub starts 2024-09-17T07:23:19.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:18 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:23:19.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:18 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:23:19.080 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:23:19.105 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:19.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:19.106 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:23:19.206 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:23:19.229 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:19.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:19.232 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:23:19.333 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:23:19.357 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:19.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:19.358 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:23:19.458 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:23:19.480 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:19.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:19.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:23:19.582 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:23:19.605 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:19.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:19.607 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:23:19.629 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:23:19 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:23:19.697 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-17T07:23:19.697 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-17T07:23:19.697 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-17T07:23:19.697 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-17T07:23:19.697 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-17T07:23:19.708 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:23:19.728 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:19.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:19.729 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:23:19.830 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:23:19.852 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:19.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:19.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:23:19.953 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:23:19.977 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:19.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:19.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:23:20.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:19 smithi135 ceph-mon[33238]: 2.10 scrub starts 2024-09-17T07:23:20.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:19 smithi135 ceph-mon[33238]: 5.5 deep-scrub starts 2024-09-17T07:23:20.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:19 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:23:20.079 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:23:20.091 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:19 smithi100 ceph-mon[29092]: 2.10 scrub starts 2024-09-17T07:23:20.091 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:19 smithi100 ceph-mon[29092]: 5.5 deep-scrub starts 2024-09-17T07:23:20.091 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:19 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:23:20.091 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:19 smithi100 ceph-mon[39462]: 2.10 scrub starts 2024-09-17T07:23:20.091 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:19 smithi100 ceph-mon[39462]: 5.5 deep-scrub starts 2024-09-17T07:23:20.091 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:19 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:23:20.102 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:20.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:20.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:23:20.204 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:23:20.227 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:20.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:20.228 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:23:20.329 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:23:20.352 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:20.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:20.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:23:20.454 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:23:20.476 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:20.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:20.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:23:20.578 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:23:20.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:20.603 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:20.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:23:20.704 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:23:20.725 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:20.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:20.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:23:20.755 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:20 smithi100 ceph-mon[29092]: 2.10 deep-scrub starts 2024-09-17T07:23:20.755 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:20 smithi100 ceph-mon[29092]: pgmap v998: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 2 undersized+degraded+remapped+backfill_toofull+peered, 41 active+undersized, 23 active+undersized+degraded, 89 active+clean; 5.5 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s; 33773/272778 objects degraded (12.381%); 10559/272778 objects misplaced (3.871%); 31 MiB/s, 488 objects/s recovering 2024-09-17T07:23:20.755 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:20 smithi100 ceph-mon[29092]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-17T07:23:20.755 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:20 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 33773/272778 objects degraded (12.381%), 28 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-17T07:23:20.755 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:20 smithi100 ceph-mon[29092]: osdmap e458: 8 total, 7 up, 7 in 2024-09-17T07:23:20.756 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:20 smithi100 ceph-mon[39462]: 2.10 deep-scrub starts 2024-09-17T07:23:20.756 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:20 smithi100 ceph-mon[39462]: pgmap v998: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 2 undersized+degraded+remapped+backfill_toofull+peered, 41 active+undersized, 23 active+undersized+degraded, 89 active+clean; 5.5 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s; 33773/272778 objects degraded (12.381%); 10559/272778 objects misplaced (3.871%); 31 MiB/s, 488 objects/s recovering 2024-09-17T07:23:20.756 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:20 smithi100 ceph-mon[39462]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-17T07:23:20.756 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:20 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 33773/272778 objects degraded (12.381%), 28 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-17T07:23:20.756 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:20 smithi100 ceph-mon[39462]: osdmap e458: 8 total, 7 up, 7 in 2024-09-17T07:23:20.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:20 smithi135 ceph-mon[33238]: 2.10 deep-scrub starts 2024-09-17T07:23:20.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:20 smithi135 ceph-mon[33238]: pgmap v998: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 2 undersized+degraded+remapped+backfill_toofull+peered, 41 active+undersized, 23 active+undersized+degraded, 89 active+clean; 5.5 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s; 33773/272778 objects degraded (12.381%); 10559/272778 objects misplaced (3.871%); 31 MiB/s, 488 objects/s recovering 2024-09-17T07:23:20.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:20 smithi135 ceph-mon[33238]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-17T07:23:20.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:20 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 33773/272778 objects degraded (12.381%), 28 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-17T07:23:20.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:20 smithi135 ceph-mon[33238]: osdmap e458: 8 total, 7 up, 7 in 2024-09-17T07:23:20.828 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:23:20.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:20.854 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:20.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:23:20.955 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:23:20.978 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:20.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:20.979 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:23:21.080 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:23:21.102 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:21.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:21.103 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:23:21.204 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:23:21.229 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:21.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:21.230 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:23:21.271 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-17T07:23:21.331 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:23:21.348 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:23:21.356 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:21.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:21.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:23:21.458 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:23:21.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:21.482 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:21.482 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:23:21.583 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:23:21.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:21.607 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:21.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:23:21.708 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:23:21.729 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:21.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:21.730 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:23:21.831 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:23:21.845 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:21 smithi100 ceph-mon[29092]: 2.1 scrub starts 2024-09-17T07:23:21.846 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:21 smithi100 ceph-mon[39462]: 2.1 scrub starts 2024-09-17T07:23:21.853 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:21.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:21.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:23:21.955 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:23:21.976 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:21.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:21.977 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:23:22.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:21 smithi135 ceph-mon[33238]: 2.1 scrub starts 2024-09-17T07:23:22.078 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:23:22.091 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:23:22.103 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:22.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:22.105 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:23:22.206 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:23:22.229 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:22.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:22.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:23:22.232 INFO:teuthology.orchestra.run.smithi135.stdout:get_pg_num_history pg_num_history pg_num_history(e446 pg_nums {1={20=1,374=11},2={67=32},3={67=3},4={69=32},5={71=32},6={73=32},9={236=16},10={297=16,332=26,360=36},11={390=16}} deleted_pools 294,9,387,10) 2024-09-17T07:23:22.233 INFO:teuthology.orchestra.run.smithi135.stdout:Importing pgid 5.1b 2024-09-17T07:23:22.233 INFO:teuthology.orchestra.run.smithi135.stdout:write_pg epoch 434 info 5.1b( empty local-lis/les=433/434 n=0 ec=71/71 lis/c=433/433 les/c/f=434/434/0 sis=433) 2024-09-17T07:23:22.233 INFO:teuthology.orchestra.run.smithi135.stdout:Import successful 2024-09-17T07:23:22.331 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:23:22.354 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:22.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:22.355 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:23:22.456 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:23:22.478 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:22.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:22.479 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:23:22.580 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:23:22.602 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:22.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:22.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:23:22.704 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:23:22.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:22.727 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:22.727 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:23:22.760 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:22 smithi100 ceph-mon[39462]: pgmap v1000: 158 pgs: 1 active+clean+scrubbing+deep, 1 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 2 undersized+degraded+remapped+backfill_toofull+peered, 41 active+undersized, 23 active+undersized+degraded, 88 active+clean; 5.5 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 651 B/s rd, 0 op/s; 33773/272778 objects degraded (12.381%); 10559/272778 objects misplaced (3.871%); 20 MiB/s, 320 objects/s recovering 2024-09-17T07:23:22.760 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:22 smithi100 ceph-mon[39462]: 11.9 scrub starts 2024-09-17T07:23:22.760 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:22 smithi100 ceph-mon[39462]: 2.10 deep-scrub starts 2024-09-17T07:23:22.760 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:22 smithi100 ceph-mon[29092]: pgmap v1000: 158 pgs: 1 active+clean+scrubbing+deep, 1 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 2 undersized+degraded+remapped+backfill_toofull+peered, 41 active+undersized, 23 active+undersized+degraded, 88 active+clean; 5.5 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 651 B/s rd, 0 op/s; 33773/272778 objects degraded (12.381%); 10559/272778 objects misplaced (3.871%); 20 MiB/s, 320 objects/s recovering 2024-09-17T07:23:22.760 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:22 smithi100 ceph-mon[29092]: 11.9 scrub starts 2024-09-17T07:23:22.761 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:22 smithi100 ceph-mon[29092]: 2.10 deep-scrub starts 2024-09-17T07:23:22.778 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-17T07:23:22.828 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:23:22.849 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:22.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:22.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:23:22.951 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:23:22.975 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:22.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:22.976 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:23:23.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:22 smithi135 ceph-mon[33238]: pgmap v1000: 158 pgs: 1 active+clean+scrubbing+deep, 1 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 2 undersized+degraded+remapped+backfill_toofull+peered, 41 active+undersized, 23 active+undersized+degraded, 88 active+clean; 5.5 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 651 B/s rd, 0 op/s; 33773/272778 objects degraded (12.381%); 10559/272778 objects misplaced (3.871%); 20 MiB/s, 320 objects/s recovering 2024-09-17T07:23:23.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:22 smithi135 ceph-mon[33238]: 11.9 scrub starts 2024-09-17T07:23:23.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:22 smithi135 ceph-mon[33238]: 2.10 deep-scrub starts 2024-09-17T07:23:23.078 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:23:23.101 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:23.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:23.103 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:23:23.147 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:23:23.147 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:23:23.171 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1: {} 2024-09-17T07:23:23.171 INFO:teuthology.orchestra.run.smithi100.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:23:23.189 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:23:23.190 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:23:23.203 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:23:23.206 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:23:23.206 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:23:23.224 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:23:23.225 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:23:23.225 INFO:teuthology.orchestra.run.smithi100.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-17T07:23:23.225 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:23.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:23.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:23:23.241 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:23:23.242 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:23:23.257 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:23:23.258 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:23:23.327 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:23:23.349 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:23.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:23.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:23:23.452 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:23:23.474 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:23.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:23.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:23:23.576 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:23:23.600 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:23.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:23.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:23:23.701 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:23:23.723 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:23.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:23.724 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:23:23.825 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:23:23.846 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:23.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:23.847 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:23:23.948 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:23:23.971 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:23.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:23.972 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:23:24.005 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:23 smithi100 ceph-mon[29092]: 4.16 scrub starts 2024-09-17T07:23:24.005 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:23 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1873214077' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:23:24.005 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:23 smithi100 ceph-mon[39462]: 4.16 scrub starts 2024-09-17T07:23:24.005 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:23 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1873214077' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:23:24.025 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:23:24.025 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:23:24.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:23 smithi135 ceph-mon[33238]: 4.16 scrub starts 2024-09-17T07:23:24.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:23 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1873214077' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:23:24.072 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:23:24.094 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:24.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:24.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:23:24.224 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:23:24.246 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:24.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:24.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:23:24.348 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:23:24.369 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:24.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:24.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:23:24.471 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:23:24.476 DEBUG:teuthology.orchestra.run.smithi135:> sudo rm -f /var/log/ceph/e848267c-74c1-11ef-bceb-c7b262605968/exp.5.1b.5 2024-09-17T07:23:24.498 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:24.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:24.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:23:24.600 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:23:24.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:24.625 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:24.625 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:23:24.726 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:23:24.748 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:24.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:24.749 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:23:24.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:24 smithi135 ceph-mon[33238]: 6.16 deep-scrub starts 2024-09-17T07:23:24.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:24 smithi135 ceph-mon[33238]: pgmap v1001: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+clean+scrubbing+deep, 2 active+undersized+degraded+remapped+backfill_toofull, 2 undersized+degraded+remapped+backfill_toofull+peered, 41 active+undersized, 23 active+undersized+degraded, 87 active+clean; 5.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 639 B/s rd, 0 op/s; 31048/272778 objects degraded (11.382%); 7834/272778 objects misplaced (2.872%); 41 MiB/s, 655 objects/s recovering 2024-09-17T07:23:24.850 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:23:24.872 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:24.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:24.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:23:24.974 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:23:24.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:24.998 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:24.998 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:23:25.078 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:24 smithi100 ceph-mon[29092]: 6.16 deep-scrub starts 2024-09-17T07:23:25.078 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:24 smithi100 ceph-mon[29092]: pgmap v1001: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+clean+scrubbing+deep, 2 active+undersized+degraded+remapped+backfill_toofull, 2 undersized+degraded+remapped+backfill_toofull+peered, 41 active+undersized, 23 active+undersized+degraded, 87 active+clean; 5.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 639 B/s rd, 0 op/s; 31048/272778 objects degraded (11.382%); 7834/272778 objects misplaced (2.872%); 41 MiB/s, 655 objects/s recovering 2024-09-17T07:23:25.079 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:24 smithi100 ceph-mon[39462]: 6.16 deep-scrub starts 2024-09-17T07:23:25.079 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:24 smithi100 ceph-mon[39462]: pgmap v1001: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+clean+scrubbing+deep, 2 active+undersized+degraded+remapped+backfill_toofull, 2 undersized+degraded+remapped+backfill_toofull+peered, 41 active+undersized, 23 active+undersized+degraded, 87 active+clean; 5.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 639 B/s rd, 0 op/s; 31048/272778 objects degraded (11.382%); 7834/272778 objects misplaced (2.872%); 41 MiB/s, 655 objects/s recovering 2024-09-17T07:23:25.084 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:23:25.100 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:23:25.122 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:25.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:25.123 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:23:25.224 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:23:25.246 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:25.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:25.247 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:23:25.348 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:23:25.377 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:25.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:25.380 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:23:25.481 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:23:25.503 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:25.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:25.504 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:23:25.604 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:23:25.626 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:25.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:25.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:23:25.728 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:23:25.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:25.751 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:25.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:23:25.852 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:23:25.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:25.875 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:25.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:23:25.976 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:23:25.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:25.998 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:25.998 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:23:26.026 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:23:26.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:25 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:23:26.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:25 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:23:26.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:25 smithi135 ceph-mon[33238]: osdmap e459: 8 total, 7 up, 7 in 2024-09-17T07:23:26.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:25 smithi135 ceph-mon[33238]: 2.10 scrub starts 2024-09-17T07:23:26.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:25 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 31048/272778 objects degraded (11.382%), 28 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-17T07:23:26.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:25 smithi135 ceph-mon[33238]: osdmap e460: 8 total, 7 up, 7 in 2024-09-17T07:23:26.060 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:23:25 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:23:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:23:26.060 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:25 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:23:26.060 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:25 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:23:26.060 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:25 smithi100 ceph-mon[29092]: osdmap e459: 8 total, 7 up, 7 in 2024-09-17T07:23:26.060 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:25 smithi100 ceph-mon[29092]: 2.10 scrub starts 2024-09-17T07:23:26.060 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:25 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 31048/272778 objects degraded (11.382%), 28 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-17T07:23:26.061 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:25 smithi100 ceph-mon[29092]: osdmap e460: 8 total, 7 up, 7 in 2024-09-17T07:23:26.061 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:25 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:23:26.061 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:25 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:23:26.061 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:25 smithi100 ceph-mon[39462]: osdmap e459: 8 total, 7 up, 7 in 2024-09-17T07:23:26.061 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:25 smithi100 ceph-mon[39462]: 2.10 scrub starts 2024-09-17T07:23:26.062 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:25 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 31048/272778 objects degraded (11.382%), 28 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-17T07:23:26.062 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:25 smithi100 ceph-mon[39462]: osdmap e460: 8 total, 7 up, 7 in 2024-09-17T07:23:26.099 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:23:26.121 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:26.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:26.122 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:23:26.223 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:23:26.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:26.245 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:26.245 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:23:26.346 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:23:26.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:26.372 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:26.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:23:26.473 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:23:26.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:26.498 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:26.498 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:23:26.599 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:23:26.621 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:26.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:26.622 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:23:26.723 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:23:26.745 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:26.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:26.746 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:23:26.847 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:23:26.869 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:26.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:26.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:23:26.971 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:23:26.992 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:26.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:26.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:23:27.094 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:23:27.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:27.118 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:27.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:23:27.219 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:23:27.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:27.242 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:27.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:23:27.343 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:23:27.365 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:27.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:27.366 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:23:27.467 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:23:27.468 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:27 smithi100 ceph-mon[39462]: pgmap v1004: 158 pgs: 1 active+clean+scrubbing, 1 active+undersized+degraded+remapped+backfilling, 1 active+clean+scrubbing+deep, 2 active+undersized+degraded+remapped+backfill_toofull, 2 undersized+degraded+remapped+backfill_toofull+peered, 41 active+undersized, 23 active+undersized+degraded, 87 active+clean; 5.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 31048/272778 objects degraded (11.382%); 7834/272778 objects misplaced (2.872%); 28 MiB/s, 453 objects/s recovering 2024-09-17T07:23:27.468 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:27 smithi100 ceph-mon[39462]: 5.9 scrub starts 2024-09-17T07:23:27.468 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:27 smithi100 ceph-mon[39462]: osdmap e461: 8 total, 7 up, 7 in 2024-09-17T07:23:27.468 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:27 smithi100 ceph-mon[29092]: pgmap v1004: 158 pgs: 1 active+clean+scrubbing, 1 active+undersized+degraded+remapped+backfilling, 1 active+clean+scrubbing+deep, 2 active+undersized+degraded+remapped+backfill_toofull, 2 undersized+degraded+remapped+backfill_toofull+peered, 41 active+undersized, 23 active+undersized+degraded, 87 active+clean; 5.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 31048/272778 objects degraded (11.382%); 7834/272778 objects misplaced (2.872%); 28 MiB/s, 453 objects/s recovering 2024-09-17T07:23:27.468 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:27 smithi100 ceph-mon[29092]: 5.9 scrub starts 2024-09-17T07:23:27.468 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:27 smithi100 ceph-mon[29092]: osdmap e461: 8 total, 7 up, 7 in 2024-09-17T07:23:27.491 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:27.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:27.492 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:23:27.593 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:23:27.614 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:27.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:27.615 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:23:27.716 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:23:27.738 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:27.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:27.739 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:23:27.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:27 smithi135 ceph-mon[33238]: pgmap v1004: 158 pgs: 1 active+clean+scrubbing, 1 active+undersized+degraded+remapped+backfilling, 1 active+clean+scrubbing+deep, 2 active+undersized+degraded+remapped+backfill_toofull, 2 undersized+degraded+remapped+backfill_toofull+peered, 41 active+undersized, 23 active+undersized+degraded, 87 active+clean; 5.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 31048/272778 objects degraded (11.382%); 7834/272778 objects misplaced (2.872%); 28 MiB/s, 453 objects/s recovering 2024-09-17T07:23:27.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:27 smithi135 ceph-mon[33238]: 5.9 scrub starts 2024-09-17T07:23:27.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:27 smithi135 ceph-mon[33238]: osdmap e461: 8 total, 7 up, 7 in 2024-09-17T07:23:27.840 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:23:27.862 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:27.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:27.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:23:27.963 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:23:27.986 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:27.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:27.987 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:23:28.088 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:23:28.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:28.112 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:28.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:23:28.214 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:23:28.236 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:28.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:28.240 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:23:28.341 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:23:28.363 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:28.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:28.364 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:23:28.374 INFO:teuthology.orchestra.run.smithi100.stderr:noscrub is set 2024-09-17T07:23:28.376 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-17T07:23:28.376 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:23:28.465 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:23:28.487 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:28.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:28.488 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:23:28.589 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:23:28.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:28.611 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:28.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:23:28.620 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:28 smithi100 ceph-mon[29092]: 4.d scrub starts 2024-09-17T07:23:28.620 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:28 smithi100 ceph-mon[29092]: 2.10 scrub starts 2024-09-17T07:23:28.620 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:28 smithi100 ceph-mon[29092]: 6.3 scrub starts 2024-09-17T07:23:28.620 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:28 smithi100 ceph-mon[29092]: 2.1f scrub starts 2024-09-17T07:23:28.620 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:28 smithi100 ceph-mon[29092]: 2.1f scrub ok 2024-09-17T07:23:28.620 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:28 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2075188878' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:23:28.620 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:28 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:23:28.620 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:28 smithi100 ceph-mon[39462]: 4.d scrub starts 2024-09-17T07:23:28.621 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:28 smithi100 ceph-mon[39462]: 2.10 scrub starts 2024-09-17T07:23:28.621 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:28 smithi100 ceph-mon[39462]: 6.3 scrub starts 2024-09-17T07:23:28.621 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:28 smithi100 ceph-mon[39462]: 2.1f scrub starts 2024-09-17T07:23:28.621 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:28 smithi100 ceph-mon[39462]: 2.1f scrub ok 2024-09-17T07:23:28.621 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:28 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2075188878' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:23:28.621 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:28 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:23:28.713 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:23:28.735 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:28.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:28.736 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:23:28.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:28 smithi135 ceph-mon[33238]: 4.d scrub starts 2024-09-17T07:23:28.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:28 smithi135 ceph-mon[33238]: 2.10 scrub starts 2024-09-17T07:23:28.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:28 smithi135 ceph-mon[33238]: 6.3 scrub starts 2024-09-17T07:23:28.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:28 smithi135 ceph-mon[33238]: 2.1f scrub starts 2024-09-17T07:23:28.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:28 smithi135 ceph-mon[33238]: 2.1f scrub ok 2024-09-17T07:23:28.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:28 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2075188878' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:23:28.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:28 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:23:28.837 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:23:28.853 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:23:28.859 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:28.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:28.860 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:23:28.961 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:23:28.982 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:28.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:28.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:23:29.084 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:23:29.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:29.107 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:29.107 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:23:29.208 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:23:29.231 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:29.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:29.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:23:29.332 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:23:29.355 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:29.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:29.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:23:29.378 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-17T07:23:29.457 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:23:29.478 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:29.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:29.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:23:29.529 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7] out_osds: [3] dead_osds: [5] live_osds: [0, 2, 3, 4, 6, 7, 1] 2024-09-17T07:23:29.529 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T07:23:29.530 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T07:23:29.580 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:23:29.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:29.603 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:29.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:23:29.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:29 smithi100 ceph-mon[39462]: pgmap v1006: 158 pgs: 1 active+clean+scrubbing, 1 active+undersized+degraded+remapped+backfilling, 1 active+clean+scrubbing+deep, 2 active+undersized+degraded+remapped+backfill_toofull, 2 undersized+degraded+remapped+backfill_toofull+peered, 41 active+undersized, 23 active+undersized+degraded, 87 active+clean; 5.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 31048/272778 objects degraded (11.382%); 7834/272778 objects misplaced (2.872%); 28 MiB/s, 453 objects/s recovering 2024-09-17T07:23:29.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:29 smithi100 ceph-mon[39462]: 5.1 scrub starts 2024-09-17T07:23:29.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:29 smithi100 ceph-mon[39462]: 5.1 scrub ok 2024-09-17T07:23:29.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:29 smithi100 ceph-mon[39462]: 2.e scrub starts 2024-09-17T07:23:29.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:29 smithi100 ceph-mon[39462]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:23:29.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:29 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T07:23:29.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:29 smithi100 ceph-mon[39462]: osdmap e462: 8 total, 7 up, 7 in 2024-09-17T07:23:29.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:29 smithi100 ceph-mon[29092]: pgmap v1006: 158 pgs: 1 active+clean+scrubbing, 1 active+undersized+degraded+remapped+backfilling, 1 active+clean+scrubbing+deep, 2 active+undersized+degraded+remapped+backfill_toofull, 2 undersized+degraded+remapped+backfill_toofull+peered, 41 active+undersized, 23 active+undersized+degraded, 87 active+clean; 5.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 31048/272778 objects degraded (11.382%); 7834/272778 objects misplaced (2.872%); 28 MiB/s, 453 objects/s recovering 2024-09-17T07:23:29.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:29 smithi100 ceph-mon[29092]: 5.1 scrub starts 2024-09-17T07:23:29.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:29 smithi100 ceph-mon[29092]: 5.1 scrub ok 2024-09-17T07:23:29.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:29 smithi100 ceph-mon[29092]: 2.e scrub starts 2024-09-17T07:23:29.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:29 smithi100 ceph-mon[29092]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:23:29.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:29 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T07:23:29.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:29 smithi100 ceph-mon[29092]: osdmap e462: 8 total, 7 up, 7 in 2024-09-17T07:23:29.704 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:23:29.727 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:29.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:29.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:23:29.793 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:23:29 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:23:29.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:29 smithi135 ceph-mon[33238]: pgmap v1006: 158 pgs: 1 active+clean+scrubbing, 1 active+undersized+degraded+remapped+backfilling, 1 active+clean+scrubbing+deep, 2 active+undersized+degraded+remapped+backfill_toofull, 2 undersized+degraded+remapped+backfill_toofull+peered, 41 active+undersized, 23 active+undersized+degraded, 87 active+clean; 5.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 31048/272778 objects degraded (11.382%); 7834/272778 objects misplaced (2.872%); 28 MiB/s, 453 objects/s recovering 2024-09-17T07:23:29.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:29 smithi135 ceph-mon[33238]: 5.1 scrub starts 2024-09-17T07:23:29.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:29 smithi135 ceph-mon[33238]: 5.1 scrub ok 2024-09-17T07:23:29.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:29 smithi135 ceph-mon[33238]: 2.e scrub starts 2024-09-17T07:23:29.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:29 smithi135 ceph-mon[33238]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:23:29.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:29 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T07:23:29.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:29 smithi135 ceph-mon[33238]: osdmap e462: 8 total, 7 up, 7 in 2024-09-17T07:23:29.828 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:23:29.850 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:29.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:29.851 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:23:29.952 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:23:29.973 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:29.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:29.974 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:23:30.075 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:23:30.097 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:30.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:30.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:23:30.199 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:23:30.220 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:30.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:30.221 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:23:30.322 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:23:30.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:30.345 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:30.345 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:23:30.404 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:30 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:23:30.405 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:30 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:23:30.446 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:23:30.468 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:30.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:30.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:23:30.570 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:23:30.591 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:30.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:30.592 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:23:30.693 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:23:30.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:30.715 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:30.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:23:30.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:30 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:23:30.817 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:23:30.838 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:30.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:30.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:23:30.940 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:23:30.962 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:30.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:30.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:23:31.064 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:23:31.086 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:31.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:31.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:23:31.188 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:23:31.210 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:31.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:31.212 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:23:31.235 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:23:31.312 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:23:31.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:31.338 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:31.338 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:23:31.439 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:23:31.460 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:31.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:31.461 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:23:31.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:31 smithi135 ceph-mon[33238]: pgmap v1008: 158 pgs: 3 active+clean+scrubbing, 1 undersized+degraded+remapped+backfilling+peered, 2 active+undersized+degraded+remapped+backfill_toofull, 41 active+undersized, 24 active+undersized+degraded, 87 active+clean; 5.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 12 op/s; 27021/272616 objects degraded (9.912%); 7297/272616 objects misplaced (2.677%); 52 MiB/s, 831 objects/s recovering 2024-09-17T07:23:31.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:31 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:23:31.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:31 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:23:31.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:31 smithi135 ceph-mon[33238]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-17T07:23:31.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:31 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 27021/272616 objects degraded (9.912%), 27 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-17T07:23:31.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:31 smithi135 ceph-mon[33238]: osdmap e463: 8 total, 7 up, 7 in 2024-09-17T07:23:31.560 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:31 smithi100 ceph-mon[29092]: pgmap v1008: 158 pgs: 3 active+clean+scrubbing, 1 undersized+degraded+remapped+backfilling+peered, 2 active+undersized+degraded+remapped+backfill_toofull, 41 active+undersized, 24 active+undersized+degraded, 87 active+clean; 5.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 12 op/s; 27021/272616 objects degraded (9.912%); 7297/272616 objects misplaced (2.677%); 52 MiB/s, 831 objects/s recovering 2024-09-17T07:23:31.561 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:31 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:23:31.561 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:31 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:23:31.561 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:31 smithi100 ceph-mon[29092]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-17T07:23:31.561 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:31 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 27021/272616 objects degraded (9.912%), 27 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-17T07:23:31.561 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:31 smithi100 ceph-mon[29092]: osdmap e463: 8 total, 7 up, 7 in 2024-09-17T07:23:31.561 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:31 smithi100 ceph-mon[39462]: pgmap v1008: 158 pgs: 3 active+clean+scrubbing, 1 undersized+degraded+remapped+backfilling+peered, 2 active+undersized+degraded+remapped+backfill_toofull, 41 active+undersized, 24 active+undersized+degraded, 87 active+clean; 5.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 12 op/s; 27021/272616 objects degraded (9.912%); 7297/272616 objects misplaced (2.677%); 52 MiB/s, 831 objects/s recovering 2024-09-17T07:23:31.561 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:31 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:23:31.561 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:31 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:23:31.561 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:31 smithi100 ceph-mon[39462]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-17T07:23:31.561 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:31 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 27021/272616 objects degraded (9.912%), 27 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-17T07:23:31.562 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:31 smithi100 ceph-mon[39462]: osdmap e463: 8 total, 7 up, 7 in 2024-09-17T07:23:31.562 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:23:31.583 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:31.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:31.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:23:31.684 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:23:31.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:31.707 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:31.707 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:23:31.808 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:23:31.820 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:23:31.821 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:23:31.830 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:31.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:31.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:23:31.839 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1: {} 2024-09-17T07:23:31.839 INFO:teuthology.orchestra.run.smithi100.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:23:31.856 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:23:31.856 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:23:31.874 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:23:31.874 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:23:31.882 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-17T07:23:31.891 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:23:31.892 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:23:31.892 INFO:teuthology.orchestra.run.smithi100.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-17T07:23:31.910 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:23:31.910 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:23:31.929 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:23:31.930 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:23:31.932 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:23:31.956 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:31.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:31.958 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:23:32.059 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:23:32.081 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:32.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:32.082 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:23:32.183 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:23:32.204 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:32.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:32.206 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:23:32.306 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:23:32.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:32.330 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:32.330 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:23:32.431 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:23:32.453 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:32.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:32.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:23:32.555 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:23:32.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:32.577 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:32.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:23:32.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:32 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2262641061' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:23:32.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:32 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2262641061' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:23:32.679 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:23:32.701 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:32.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:32.702 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:23:32.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:32 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2262641061' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:23:32.803 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:23:32.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:32.826 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:32.826 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:23:32.927 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:23:32.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:32.953 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:32.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:23:33.054 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:23:33.076 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:33.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:33.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:23:33.177 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:23:33.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:33.200 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:33.200 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:23:33.301 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:23:33.323 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:33.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:33.324 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:23:33.425 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:23:33.447 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:33.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:33.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:23:33.549 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:23:33.570 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:33.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:33.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:23:33.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:33 smithi100 ceph-mon[29092]: pgmap v1010: 158 pgs: 3 active+clean+scrubbing, 1 undersized+degraded+remapped+backfilling+peered, 2 active+undersized+degraded+remapped+backfill_toofull, 41 active+undersized, 24 active+undersized+degraded, 87 active+clean; 5.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 12 op/s; 27017/272565 objects degraded (9.912%); 7293/272565 objects misplaced (2.676%); 42 MiB/s, 667 objects/s recovering 2024-09-17T07:23:33.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:33 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:23:33.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:33 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:23:33.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:33 smithi100 ceph-mon[39462]: pgmap v1010: 158 pgs: 3 active+clean+scrubbing, 1 undersized+degraded+remapped+backfilling+peered, 2 active+undersized+degraded+remapped+backfill_toofull, 41 active+undersized, 24 active+undersized+degraded, 87 active+clean; 5.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 12 op/s; 27017/272565 objects degraded (9.912%); 7293/272565 objects misplaced (2.676%); 42 MiB/s, 667 objects/s recovering 2024-09-17T07:23:33.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:33 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:23:33.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:33 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:23:33.672 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:23:33.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:33.694 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:33.694 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:23:33.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:33 smithi135 ceph-mon[33238]: pgmap v1010: 158 pgs: 3 active+clean+scrubbing, 1 undersized+degraded+remapped+backfilling+peered, 2 active+undersized+degraded+remapped+backfill_toofull, 41 active+undersized, 24 active+undersized+degraded, 87 active+clean; 5.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 12 op/s; 27017/272565 objects degraded (9.912%); 7293/272565 objects misplaced (2.676%); 42 MiB/s, 667 objects/s recovering 2024-09-17T07:23:33.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:33 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:23:33.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:33 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:23:33.794 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:23:33.816 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:33.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:33.817 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:23:33.918 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:23:33.940 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:33.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:33.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:23:34.041 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:23:34.063 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:34.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:34.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:23:34.165 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:23:34.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:34.187 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:34.187 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:23:34.288 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:23:34.310 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:34.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:34.311 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:23:34.382 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:23:34.382 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:23:34.393 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:23:34.412 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:23:34.436 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:34.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:34.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:23:34.538 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:23:34.585 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:34.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:34.586 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:23:34.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:34 smithi100 ceph-mon[29092]: osdmap e464: 8 total, 7 up, 7 in 2024-09-17T07:23:34.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:34 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:23:34.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:34 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:23:34.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:34 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:23:34.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:34 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:23:34.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:34 smithi100 ceph-mon[39462]: osdmap e464: 8 total, 7 up, 7 in 2024-09-17T07:23:34.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:34 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:23:34.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:34 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:23:34.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:34 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:23:34.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:34 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:23:34.675 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:23:34.687 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:23:34.713 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:34.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:34.714 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:23:34.722 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:34 smithi135 ceph-mon[33238]: osdmap e464: 8 total, 7 up, 7 in 2024-09-17T07:23:34.722 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:34 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:23:34.722 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:34 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:23:34.722 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:34 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:23:34.722 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:34 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:23:34.815 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:23:34.852 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:34.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:34.853 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:23:34.954 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:23:34.988 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:34.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:34.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:23:35.043 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:23:34 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a[63580]: logger=cleanup t=2024-09-17T07:23:34.722493057Z level=info msg="Completed cleanup jobs" duration=12.647285ms 2024-09-17T07:23:35.090 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:23:35.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:35.130 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:35.130 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:23:35.231 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:23:35.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:35.270 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:35.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:23:35.371 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:23:35.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:35.401 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:35.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:23:35.505 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:23:35.521 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:35 smithi100 ceph-mon[29092]: pgmap v1012: 158 pgs: 1 undersized+degraded+remapped+backfilling+peered, 2 active+undersized+degraded+remapped+backfill_toofull, 41 active+undersized, 24 active+undersized+degraded, 90 active+clean; 5.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 12 op/s; 27017/272565 objects degraded (9.912%); 7293/272565 objects misplaced (2.676%); 42 MiB/s, 667 objects/s recovering 2024-09-17T07:23:35.521 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:35 smithi100 ceph-mon[29092]: osdmap e465: 8 total, 7 up, 7 in 2024-09-17T07:23:35.521 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:35 smithi100 ceph-mon[29092]: osdmap e466: 8 total, 7 up, 7 in 2024-09-17T07:23:35.521 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:35 smithi100 ceph-mon[39462]: pgmap v1012: 158 pgs: 1 undersized+degraded+remapped+backfilling+peered, 2 active+undersized+degraded+remapped+backfill_toofull, 41 active+undersized, 24 active+undersized+degraded, 90 active+clean; 5.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 12 op/s; 27017/272565 objects degraded (9.912%); 7293/272565 objects misplaced (2.676%); 42 MiB/s, 667 objects/s recovering 2024-09-17T07:23:35.521 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:35 smithi100 ceph-mon[39462]: osdmap e465: 8 total, 7 up, 7 in 2024-09-17T07:23:35.522 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:35 smithi100 ceph-mon[39462]: osdmap e466: 8 total, 7 up, 7 in 2024-09-17T07:23:35.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:35.537 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:35.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:23:35.638 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:23:35.676 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:35.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:35.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:23:35.778 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:23:35.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:35 smithi135 ceph-mon[33238]: pgmap v1012: 158 pgs: 1 undersized+degraded+remapped+backfilling+peered, 2 active+undersized+degraded+remapped+backfill_toofull, 41 active+undersized, 24 active+undersized+degraded, 90 active+clean; 5.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 12 op/s; 27017/272565 objects degraded (9.912%); 7293/272565 objects misplaced (2.676%); 42 MiB/s, 667 objects/s recovering 2024-09-17T07:23:35.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:35 smithi135 ceph-mon[33238]: osdmap e465: 8 total, 7 up, 7 in 2024-09-17T07:23:35.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:35 smithi135 ceph-mon[33238]: osdmap e466: 8 total, 7 up, 7 in 2024-09-17T07:23:35.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:35.818 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:35.818 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:23:35.919 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:23:35.954 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:35.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:35.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:23:36.056 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:23:36.091 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:36.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:36.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:23:36.193 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:23:36.234 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:36.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:36.235 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:23:36.288 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:23:35 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:23:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:23:36.336 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:23:36.383 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:23:36.387 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:36.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:36.388 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:23:36.489 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:23:36.521 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:36.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:36.522 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:23:36.547 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:36 smithi100 ceph-mon[29092]: pgmap v1015: 158 pgs: 2 active+undersized+degraded+remapped+backfilling, 1 peering, 41 active+undersized, 24 active+undersized+degraded, 90 active+clean; 5.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1004 B/s rd, 0 B/s wr, 49 op/s; 18632/271821 objects degraded (6.855%); 7204/271821 objects misplaced (2.650%); 32 MiB/s, 515 objects/s recovering 2024-09-17T07:23:36.547 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:36 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 18632/271821 objects degraded (6.855%), 26 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-17T07:23:36.547 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:36 smithi100 ceph-mon[29092]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-09-17T07:23:36.547 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:36 smithi100 ceph-mon[39462]: pgmap v1015: 158 pgs: 2 active+undersized+degraded+remapped+backfilling, 1 peering, 41 active+undersized, 24 active+undersized+degraded, 90 active+clean; 5.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1004 B/s rd, 0 B/s wr, 49 op/s; 18632/271821 objects degraded (6.855%); 7204/271821 objects misplaced (2.650%); 32 MiB/s, 515 objects/s recovering 2024-09-17T07:23:36.548 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:36 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 18632/271821 objects degraded (6.855%), 26 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-17T07:23:36.548 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:36 smithi100 ceph-mon[39462]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-09-17T07:23:36.623 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:23:36.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:36.657 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:36.657 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:23:36.758 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:23:36.787 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:36.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:36.789 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:23:36.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:36 smithi135 ceph-mon[33238]: pgmap v1015: 158 pgs: 2 active+undersized+degraded+remapped+backfilling, 1 peering, 41 active+undersized, 24 active+undersized+degraded, 90 active+clean; 5.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1004 B/s rd, 0 B/s wr, 49 op/s; 18632/271821 objects degraded (6.855%); 7204/271821 objects misplaced (2.650%); 32 MiB/s, 515 objects/s recovering 2024-09-17T07:23:36.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:36 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 18632/271821 objects degraded (6.855%), 26 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-17T07:23:36.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:36 smithi135 ceph-mon[33238]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-09-17T07:23:36.890 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:23:36.926 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:36.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:36.927 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:23:37.027 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:23:37.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:37.060 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:37.060 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:23:37.161 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:23:37.224 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:37.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:37.225 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:23:37.326 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:23:37.355 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:37.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:37.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:23:37.457 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:23:37.494 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:37.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:37.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:23:37.596 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:23:37.634 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:37.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:37.635 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:23:37.736 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:23:37.769 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:37.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:37.770 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:23:37.871 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:23:37.909 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:37.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:37.910 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:23:38.010 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:23:38.031 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:38.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:38.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:23:38.133 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:23:38.165 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:38.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:38.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:23:38.267 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:23:38.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:38.302 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:38.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:23:38.403 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:23:38.438 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:38.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:38.440 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:23:38.540 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:23:38.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:38.586 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:38.586 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:23:38.604 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:38 smithi100 ceph-mon[29092]: pgmap v1016: 158 pgs: 2 active+undersized+degraded+remapped+backfilling, 1 peering, 41 active+undersized, 24 active+undersized+degraded, 90 active+clean; 5.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 42 op/s; 18632/271821 objects degraded (6.855%); 7204/271821 objects misplaced (2.650%); 27 MiB/s, 437 objects/s recovering 2024-09-17T07:23:38.604 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:38 smithi100 ceph-mon[39462]: pgmap v1016: 158 pgs: 2 active+undersized+degraded+remapped+backfilling, 1 peering, 41 active+undersized, 24 active+undersized+degraded, 90 active+clean; 5.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 42 op/s; 18632/271821 objects degraded (6.855%); 7204/271821 objects misplaced (2.650%); 27 MiB/s, 437 objects/s recovering 2024-09-17T07:23:38.687 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:23:38.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:38.717 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:38.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:23:38.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:38 smithi135 ceph-mon[33238]: pgmap v1016: 158 pgs: 2 active+undersized+degraded+remapped+backfilling, 1 peering, 41 active+undersized, 24 active+undersized+degraded, 90 active+clean; 5.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 42 op/s; 18632/271821 objects degraded (6.855%); 7204/271821 objects misplaced (2.650%); 27 MiB/s, 437 objects/s recovering 2024-09-17T07:23:38.819 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:23:38.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:38.856 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:38.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:23:38.863 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T07:23:38.866 INFO:teuthology.orchestra.run.smithi100.stderr:dumped all 2024-09-17T07:23:38.899 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-17T07:23:38.899 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-17T07:23:38.900 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:23:38.900 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.25811.0:260015 11.6 e466/453 11:6a0b5594:::benchmark_data_smithi100_180298_object87766:head v 466'16239, mlcod=466'16235)", 2024-09-17T07:23:38.900 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:23:38.892793+0000", 2024-09-17T07:23:38.900 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.001015855, 2024-09-17T07:23:38.900 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0010472140000000001, 2024-09-17T07:23:38.900 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:23:38.900 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-17T07:23:38.900 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:23:38.900 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:23:38.900 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:23:38.900 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:23:38.892793+0000", 2024-09-17T07:23:38.900 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:23:38.900 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:23:38.901 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:23:38.901 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:23:38.901 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:23:38.892793+0000", 2024-09-17T07:23:38.901 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:23:38.901 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:23:38.901 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:23:38.901 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:23:38.901 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:23:38.892792+0000", 2024-09-17T07:23:38.901 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-17T07:23:38.901 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:23:38.901 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:23:38.901 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:23:38.902 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:23:38.892799+0000", 2024-09-17T07:23:38.902 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7610000000000002e-06 2024-09-17T07:23:38.902 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:23:38.902 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:23:38.902 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:23:38.902 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:23:38.892813+0000", 2024-09-17T07:23:38.902 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4516e-05 2024-09-17T07:23:38.902 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:23:38.902 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:23:38.902 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:23:38.902 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:23:38.892821+0000", 2024-09-17T07:23:38.902 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4989999999999997e-06 2024-09-17T07:23:38.902 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:23:38.902 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:23:38.903 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-17T07:23:38.903 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:23:38.892917+0000", 2024-09-17T07:23:38.903 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.6528999999999997e-05 2024-09-17T07:23:38.903 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:23:38.903 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:23:38.903 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-17T07:23:38.903 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:23:38.892936+0000", 2024-09-17T07:23:38.903 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.9117e-05 2024-09-17T07:23:38.903 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:23:38.903 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:23:38.903 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:23:38.903 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:23:38.903 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:23:38.903 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.25811.0:260017 11.2 e466/451)", 2024-09-17T07:23:38.904 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-17T07:23:38.893826+0000", 2024-09-17T07:23:38.904 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4294967295.9999824, 2024-09-17T07:23:38.904 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5823999999999998e-05, 2024-09-17T07:23:38.904 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-17T07:23:38.904 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "queued for pg", 2024-09-17T07:23:38.904 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-17T07:23:38.904 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:23:38.904 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-17T07:23:38.904 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:23:38.893826+0000", 2024-09-17T07:23:38.904 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:23:38.904 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:23:38.904 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:23:38.904 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-17T07:23:38.905 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:23:38.893826+0000", 2024-09-17T07:23:38.905 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-17T07:23:38.905 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:23:38.905 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:23:38.905 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-17T07:23:38.905 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:23:38.893826+0000", 2024-09-17T07:23:38.905 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-17T07:23:38.905 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:23:38.905 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:23:38.905 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-17T07:23:38.905 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:23:38.893831+0000", 2024-09-17T07:23:38.905 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2989999999999996e-06 2024-09-17T07:23:38.905 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:23:38.906 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:23:38.906 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-17T07:23:38.906 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:23:38.893842+0000", 2024-09-17T07:23:38.906 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0896e-05 2024-09-17T07:23:38.906 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:23:38.906 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:23:38.906 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-17T07:23:38.906 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-17T07:23:38.893849+0000", 2024-09-17T07:23:38.906 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7089999999999996e-06 2024-09-17T07:23:38.906 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:23:38.906 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:23:38.906 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:23:38.906 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:23:38.907 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-17T07:23:38.907 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 2 2024-09-17T07:23:38.907 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-17T07:23:38.957 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:23:38.987 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:38.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:38.989 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:23:39.089 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:23:39.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:39.130 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:39.130 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:23:39.231 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:23:39.261 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:39.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:39.262 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:23:39.363 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:23:39.393 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:39.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:39.394 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:23:39.494 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:23:39.555 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:39.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:39.556 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:23:39.569 INFO:teuthology.orchestra.run.smithi100.stderr:nodeep-scrub is set 2024-09-17T07:23:39.657 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:23:39.693 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:39.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:39.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:23:39.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:39 smithi135 ceph-mon[33238]: from='client.26000 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:23:39.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:39 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:23:39.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:39 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/853082926' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:23:39.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:39 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:23:39.794 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:23:39 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:23:39.795 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:23:39.833 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:39.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:39.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:23:39.876 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:39 smithi100 ceph-mon[39462]: from='client.26000 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:23:39.876 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:39 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:23:39.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:39 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/853082926' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:23:39.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:39 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:23:39.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:39 smithi100 ceph-mon[29092]: from='client.26000 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:23:39.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:39 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:23:39.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:39 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/853082926' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:23:39.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:39 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:23:39.935 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:23:39.969 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:39.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:39.970 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:23:40.071 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:23:40.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:40.107 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:40.108 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:23:40.134 INFO:teuthology.orchestra.run.smithi100.stdout:{"pg_ready":true,"pg_map":{"version":1016,"stamp":"2024-09-17T07:23:37.475182+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5925807164,"num_objects":90607,"num_object_clones":0,"num_object_copies":271821,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18632,"num_objects_misplaced":7204,"num_objects_unfound":0,"num_objects_dirty":90607,"num_whiteouts":0,"num_read":1849,"num_read_kb":1668,"num_write":255217,"num_write_kb":11025544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85611,"num_bytes_recovered":5594398512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72576,"ondisk_log_size":72576,"up":407,"acting":406,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":479,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":25460716,"kb_used_data":18648852,"kb_used_omap":13,"kb_used_meta":6811698,"kb_avail":630755348,"statfs":{"total":671965249536,"available":645893476352,"internally_reserved":0,"allocated":19096424448,"data_stored":19087964116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13997,"internal_metadata":6975179091},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16252928,"num_objects":-248,"num_object_clones":0,"num_object_copies":-744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-8385,"num_objects_misplaced":-89,"num_objects_unfound":0,"num_objects_dirty":-248,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":248,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2626,"num_bytes_recovered":172097536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001682"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":40,"reported_epoch":462,"state":"active+clean","last_fresh":"2024-09-17T07:23:28.792860+0000","last_change":"2024-09-17T07:22:13.586844+0000","last_active":"2024-09-17T07:23:28.792860+0000","last_peered":"2024-09-17T07:23:28.792860+0000","last_clean":"2024-09-17T07:23:28.792860+0000","last_became_active":"2024-09-17T07:22:13.586558+0000","last_became_peered":"2024-09-17T07:22:13.586558+0000","last_unstale":"2024-09-17T07:23:28.792860+0000","last_undegraded":"2024-09-17T07:23:28.792860+0000","last_fullsized":"2024-09-17T07:23:28.792860+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:10.965933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:33.644728+0000","last_clean_scrub_stamp":"2024-09-17T07:20:10.965933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040893329999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":522,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-17T07:23:33.398745+0000","last_change":"2024-09-17T07:23:27.440583+0000","last_active":"2024-09-17T07:23:33.398745+0000","last_peered":"2024-09-17T07:23:33.398745+0000","last_clean":"2024-09-17T07:23:33.398745+0000","last_became_active":"2024-09-17T07:22:13.589527+0000","last_became_peered":"2024-09-17T07:22:13.589527+0000","last_unstale":"2024-09-17T07:23:33.398745+0000","last_undegraded":"2024-09-17T07:23:33.398745+0000","last_fullsized":"2024-09-17T07:23:33.398745+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:23:27.440509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:23:27.440509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:24:46.831651+0000","scrub_duration":0.001261002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"439'39","reported_seq":426,"reported_epoch":464,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:23:33.398450+0000","last_change":"2024-09-17T07:22:54.389006+0000","last_active":"2024-09-17T07:23:33.398450+0000","last_peered":"2024-09-17T07:23:33.398450+0000","last_clean":"2024-09-17T07:22:52.355497+0000","last_became_active":"2024-09-17T07:22:54.389006+0000","last_became_peered":"2024-09-17T07:22:54.389006+0000","last_unstale":"2024-09-17T07:23:33.398450+0000","last_undegraded":"2024-09-17T07:22:54.380420+0000","last_fullsized":"2024-09-17T07:22:54.380316+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'27","last_scrub_stamp":"2024-09-17T07:18:53.476878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:18:53.476878+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037504449999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":7}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":481,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-17T07:23:30.377170+0000","last_change":"2024-09-17T07:23:12.631814+0000","last_active":"2024-09-17T07:23:30.377170+0000","last_peered":"2024-09-17T07:23:30.377170+0000","last_clean":"2024-09-17T07:23:30.377170+0000","last_became_active":"2024-09-17T07:21:15.354561+0000","last_became_peered":"2024-09-17T07:21:15.354561+0000","last_unstale":"2024-09-17T07:23:30.377170+0000","last_undegraded":"2024-09-17T07:23:30.377170+0000","last_fullsized":"2024-09-17T07:23:30.377170+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:45.261117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:15:45.261117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001477482,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":350,"reported_epoch":464,"state":"active+undersized","last_fresh":"2024-09-17T07:23:33.399061+0000","last_change":"2024-09-17T07:22:54.389311+0000","last_active":"2024-09-17T07:23:33.399061+0000","last_peered":"2024-09-17T07:23:33.399061+0000","last_clean":"2024-09-17T07:22:52.357698+0000","last_became_active":"2024-09-17T07:22:54.389311+0000","last_became_peered":"2024-09-17T07:22:54.389311+0000","last_unstale":"2024-09-17T07:23:33.399061+0000","last_undegraded":"2024-09-17T07:23:33.399061+0000","last_fullsized":"2024-09-17T07:22:54.381283+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:21:13.454404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:17:35.507742+0000","last_clean_scrub_stamp":"2024-09-17T07:21:13.454404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.016029897000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"438'21","reported_seq":529,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-17T07:23:30.377203+0000","last_change":"2024-09-17T07:23:07.362789+0000","last_active":"2024-09-17T07:23:30.377203+0000","last_peered":"2024-09-17T07:23:30.377203+0000","last_clean":"2024-09-17T07:23:30.377203+0000","last_became_active":"2024-09-17T07:21:15.354718+0000","last_became_peered":"2024-09-17T07:21:15.354718+0000","last_unstale":"2024-09-17T07:23:30.377203+0000","last_undegraded":"2024-09-17T07:23:30.377203+0000","last_fullsized":"2024-09-17T07:23:30.377203+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'15","last_scrub_stamp":"2024-09-17T07:15:39.306043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:15:39.306043+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002471029,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":367,"reported_epoch":463,"state":"active+undersized","last_fresh":"2024-09-17T07:23:30.535782+0000","last_change":"2024-09-17T07:22:54.383653+0000","last_active":"2024-09-17T07:23:30.535782+0000","last_peered":"2024-09-17T07:23:30.535782+0000","last_clean":"2024-09-17T07:22:52.354136+0000","last_became_active":"2024-09-17T07:22:54.383653+0000","last_became_peered":"2024-09-17T07:22:54.383653+0000","last_unstale":"2024-09-17T07:23:30.535782+0000","last_undegraded":"2024-09-17T07:23:30.535782+0000","last_fullsized":"2024-09-17T07:22:54.378497+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:13.660077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:20:13.660077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":486,"reported_epoch":463,"state":"active+undersized","last_fresh":"2024-09-17T07:23:30.377381+0000","last_change":"2024-09-17T07:22:54.381051+0000","last_active":"2024-09-17T07:23:30.377381+0000","last_peered":"2024-09-17T07:23:30.377381+0000","last_clean":"2024-09-17T07:22:52.352664+0000","last_became_active":"2024-09-17T07:22:54.381051+0000","last_became_peered":"2024-09-17T07:22:54.381051+0000","last_unstale":"2024-09-17T07:23:30.377381+0000","last_undegraded":"2024-09-17T07:23:30.377381+0000","last_fullsized":"2024-09-17T07:22:54.374452+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:41.707969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:14:41.707969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.014931757,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":40,"reported_epoch":462,"state":"active+clean","last_fresh":"2024-09-17T07:23:28.792560+0000","last_change":"2024-09-17T07:22:13.615802+0000","last_active":"2024-09-17T07:23:28.792560+0000","last_peered":"2024-09-17T07:23:28.792560+0000","last_clean":"2024-09-17T07:23:28.792560+0000","last_became_active":"2024-09-17T07:22:13.615512+0000","last_became_peered":"2024-09-17T07:22:13.615512+0000","last_unstale":"2024-09-17T07:23:28.792560+0000","last_undegraded":"2024-09-17T07:23:28.792560+0000","last_fullsized":"2024-09-17T07:23:28.792560+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:18:44.849299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:46.312371+0000","last_clean_scrub_stamp":"2024-09-17T07:18:44.849299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"439'9","reported_seq":517,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-17T07:23:30.377660+0000","last_change":"2024-09-17T07:23:05.345465+0000","last_active":"2024-09-17T07:23:30.377660+0000","last_peered":"2024-09-17T07:23:30.377660+0000","last_clean":"2024-09-17T07:23:30.377660+0000","last_became_active":"2024-09-17T07:21:15.349235+0000","last_became_peered":"2024-09-17T07:21:15.349235+0000","last_unstale":"2024-09-17T07:23:30.377660+0000","last_undegraded":"2024-09-17T07:23:30.377660+0000","last_fullsized":"2024-09-17T07:23:30.377660+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'7","last_scrub_stamp":"2024-09-17T07:14:42.653705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:14:42.653705+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0050149230000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":13,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":40,"reported_epoch":462,"state":"active+clean","last_fresh":"2024-09-17T07:23:28.792522+0000","last_change":"2024-09-17T07:22:13.624911+0000","last_active":"2024-09-17T07:23:28.792522+0000","last_peered":"2024-09-17T07:23:28.792522+0000","last_clean":"2024-09-17T07:23:28.792522+0000","last_became_active":"2024-09-17T07:22:13.624441+0000","last_became_peered":"2024-09-17T07:22:13.624441+0000","last_unstale":"2024-09-17T07:23:28.792522+0000","last_undegraded":"2024-09-17T07:23:28.792522+0000","last_fullsized":"2024-09-17T07:23:28.792522+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:10.116580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:20:10.116580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00137765,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":462,"reported_epoch":463,"state":"active+undersized","last_fresh":"2024-09-17T07:23:30.535959+0000","last_change":"2024-09-17T07:22:54.382551+0000","last_active":"2024-09-17T07:23:30.535959+0000","last_peered":"2024-09-17T07:23:30.535959+0000","last_clean":"2024-09-17T07:16:16.170346+0000","last_became_active":"2024-09-17T07:22:54.382551+0000","last_became_peered":"2024-09-17T07:22:54.382551+0000","last_unstale":"2024-09-17T07:23:30.535959+0000","last_undegraded":"2024-09-17T07:23:30.535959+0000","last_fullsized":"2024-09-17T07:22:54.379306+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:52.168181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:14:52.168181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0056501399999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":46,"reported_epoch":462,"state":"active+undersized","last_fresh":"2024-09-17T07:23:28.793435+0000","last_change":"2024-09-17T07:22:54.384536+0000","last_active":"2024-09-17T07:23:28.793435+0000","last_peered":"2024-09-17T07:23:28.793435+0000","last_clean":"2024-09-17T07:22:52.359982+0000","last_became_active":"2024-09-17T07:22:54.384536+0000","last_became_peered":"2024-09-17T07:22:54.384536+0000","last_unstale":"2024-09-17T07:23:28.793435+0000","last_undegraded":"2024-09-17T07:23:28.793435+0000","last_fullsized":"2024-09-17T07:22:54.379573+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:18.961914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:45.029247+0000","last_clean_scrub_stamp":"2024-09-17T07:19:18.961914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"439'25","reported_seq":557,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-17T07:23:30.377601+0000","last_change":"2024-09-17T07:20:39.070372+0000","last_active":"2024-09-17T07:23:30.377601+0000","last_peered":"2024-09-17T07:23:30.377601+0000","last_clean":"2024-09-17T07:23:30.377601+0000","last_became_active":"2024-09-17T07:09:24.397954+0000","last_became_peered":"2024-09-17T07:09:24.397954+0000","last_unstale":"2024-09-17T07:23:30.377601+0000","last_undegraded":"2024-09-17T07:23:30.377601+0000","last_fullsized":"2024-09-17T07:23:30.377601+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"312'19","last_scrub_stamp":"2024-09-17T07:20:39.070235+0000","last_deep_scrub":"312'19","last_deep_scrub_stamp":"2024-09-17T07:16:44.292911+0000","last_clean_scrub_stamp":"2024-09-17T07:20:39.070235+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0028570560000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":500,"reported_epoch":464,"state":"active+undersized","last_fresh":"2024-09-17T07:23:33.398483+0000","last_change":"2024-09-17T07:22:54.391641+0000","last_active":"2024-09-17T07:23:33.398483+0000","last_peered":"2024-09-17T07:23:33.398483+0000","last_clean":"2024-09-17T07:22:10.349565+0000","last_became_active":"2024-09-17T07:22:54.391641+0000","last_became_peered":"2024-09-17T07:22:54.391641+0000","last_unstale":"2024-09-17T07:23:33.398483+0000","last_undegraded":"2024-09-17T07:23:33.398483+0000","last_fullsized":"2024-09-17T07:22:54.382871+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:26.348475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:28.096266+0000","last_clean_scrub_stamp":"2024-09-17T07:19:26.348475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.015361906,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":71,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-17T07:23:30.534402+0000","last_change":"2024-09-17T07:22:13.590579+0000","last_active":"2024-09-17T07:23:30.534402+0000","last_peered":"2024-09-17T07:23:30.534402+0000","last_clean":"2024-09-17T07:23:30.534402+0000","last_became_active":"2024-09-17T07:22:13.590276+0000","last_became_peered":"2024-09-17T07:22:13.590276+0000","last_unstale":"2024-09-17T07:23:30.534402+0000","last_undegraded":"2024-09-17T07:23:30.534402+0000","last_fullsized":"2024-09-17T07:23:30.534402+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:27.474541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:20:27.474541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0061993300000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2","version":"465'16021","reported_seq":27933,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-09-17T07:23:35.067799+0000","last_change":"2024-09-17T07:23:06.385975+0000","last_active":"2024-09-17T07:23:35.067799+0000","last_peered":"2024-09-17T07:23:35.067799+0000","last_clean":"2024-09-17T07:23:35.067799+0000","last_became_active":"2024-09-17T07:23:06.354259+0000","last_became_peered":"2024-09-17T07:23:06.354259+0000","last_unstale":"2024-09-17T07:23:35.067799+0000","last_undegraded":"2024-09-17T07:23:35.067799+0000","last_fullsized":"2024-09-17T07:23:35.067799+0000","mapping_epoch":451,"log_start":"424'11400","ondisk_log_start":"424'11400","created":390,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_clean_scrub_stamp":"2024-09-17T07:19:54.447320+0000","objects_scrubbed":0,"log_size":4621,"log_dups_size":0,"ondisk_log_size":4621,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":373882880,"num_objects":5705,"num_object_clones":0,"num_object_copies":17115,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16021,"num_write_kb":695232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5900,"num_bytes_recovered":386662400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.8","version":"212'130","reported_seq":469,"reported_epoch":463,"state":"active+undersized","last_fresh":"2024-09-17T07:23:30.534460+0000","last_change":"2024-09-17T07:22:54.391700+0000","last_active":"2024-09-17T07:23:30.534460+0000","last_peered":"2024-09-17T07:23:30.534460+0000","last_clean":"2024-09-17T07:10:37.677048+0000","last_became_active":"2024-09-17T07:22:54.391700+0000","last_became_peered":"2024-09-17T07:22:54.391700+0000","last_unstale":"2024-09-17T07:23:30.534460+0000","last_undegraded":"2024-09-17T07:23:30.534460+0000","last_fullsized":"2024-09-17T07:22:54.376185+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":4,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:20:21.626089+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:14:57.052505+0000","last_clean_scrub_stamp":"2024-09-17T07:20:21.626089+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021141889999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":28,"num_write":16,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":161009,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"439'53","reported_seq":573,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-17T07:23:30.378133+0000","last_change":"2024-09-17T07:23:28.794236+0000","last_active":"2024-09-17T07:23:30.378133+0000","last_peered":"2024-09-17T07:23:30.378133+0000","last_clean":"2024-09-17T07:23:30.378133+0000","last_became_active":"2024-09-17T07:21:15.348413+0000","last_became_peered":"2024-09-17T07:21:15.348413+0000","last_unstale":"2024-09-17T07:23:30.378133+0000","last_undegraded":"2024-09-17T07:23:30.378133+0000","last_fullsized":"2024-09-17T07:23:30.378133+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'39","last_scrub_stamp":"2024-09-17T07:15:46.226554+0000","last_deep_scrub":"95'25","last_deep_scrub_stamp":"2024-09-17T07:04:09.450039+0000","last_clean_scrub_stamp":"2024-09-17T07:15:46.226554+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0027061799999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":61,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":46,"reported_epoch":462,"state":"active+undersized","last_fresh":"2024-09-17T07:23:28.793024+0000","last_change":"2024-09-17T07:22:54.383630+0000","last_active":"2024-09-17T07:23:28.793024+0000","last_peered":"2024-09-17T07:23:28.793024+0000","last_clean":"2024-09-17T07:22:52.359785+0000","last_became_active":"2024-09-17T07:22:54.383630+0000","last_became_peered":"2024-09-17T07:22:54.383630+0000","last_unstale":"2024-09-17T07:23:28.793024+0000","last_undegraded":"2024-09-17T07:23:28.793024+0000","last_fullsized":"2024-09-17T07:22:54.379433+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:35.965669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:19:35.965669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":316,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-17T07:23:30.534831+0000","last_change":"2024-09-17T07:21:15.350941+0000","last_active":"2024-09-17T07:23:30.534831+0000","last_peered":"2024-09-17T07:23:30.534831+0000","last_clean":"2024-09-17T07:23:30.534831+0000","last_became_active":"2024-09-17T07:21:15.349538+0000","last_became_peered":"2024-09-17T07:21:15.349538+0000","last_unstale":"2024-09-17T07:23:30.534831+0000","last_undegraded":"2024-09-17T07:23:30.534831+0000","last_fullsized":"2024-09-17T07:23:30.534831+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:46.502641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:13:46.502641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":345,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-17T07:23:30.378046+0000","last_change":"2024-09-17T07:20:40.117298+0000","last_active":"2024-09-17T07:23:30.378046+0000","last_peered":"2024-09-17T07:23:30.378046+0000","last_clean":"2024-09-17T07:23:30.378046+0000","last_became_active":"2024-09-17T07:09:24.396334+0000","last_became_peered":"2024-09-17T07:09:24.396334+0000","last_unstale":"2024-09-17T07:23:30.378046+0000","last_undegraded":"2024-09-17T07:23:30.378046+0000","last_fullsized":"2024-09-17T07:23:30.378046+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:40.117165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:42.984088+0000","last_clean_scrub_stamp":"2024-09-17T07:20:40.117165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0041399059999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.3","version":"461'15958","reported_seq":35305,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-17T07:23:30.536089+0000","last_change":"2024-09-17T07:23:17.649500+0000","last_active":"2024-09-17T07:23:30.536089+0000","last_peered":"2024-09-17T07:23:30.536089+0000","last_clean":"2024-09-17T07:23:30.536089+0000","last_became_active":"2024-09-17T07:22:59.389707+0000","last_became_peered":"2024-09-17T07:22:59.389707+0000","last_unstale":"2024-09-17T07:23:30.536089+0000","last_undegraded":"2024-09-17T07:23:30.536089+0000","last_fullsized":"2024-09-17T07:23:30.536089+0000","mapping_epoch":449,"log_start":"424'11700","ondisk_log_start":"424'11700","created":390,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_clean_scrub_stamp":"2024-09-17T07:19:54.447320+0000","objects_scrubbed":0,"log_size":4258,"log_dups_size":0,"ondisk_log_size":4258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":367263744,"num_objects":5604,"num_object_clones":0,"num_object_copies":16812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15958,"num_write_kb":689984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6408,"num_bytes_recovered":419954688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"212'130","reported_seq":701,"reported_epoch":462,"state":"active+undersized","last_fresh":"2024-09-17T07:23:28.792459+0000","last_change":"2024-09-17T07:22:54.383876+0000","last_active":"2024-09-17T07:23:28.792459+0000","last_peered":"2024-09-17T07:23:28.792459+0000","last_clean":"2024-09-17T07:22:52.359344+0000","last_became_active":"2024-09-17T07:22:54.383876+0000","last_became_peered":"2024-09-17T07:22:54.383876+0000","last_unstale":"2024-09-17T07:23:28.792459+0000","last_undegraded":"2024-09-17T07:23:28.792459+0000","last_fullsized":"2024-09-17T07:22:54.379741+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":4,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:21:12.403277+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:14:57.052505+0000","last_clean_scrub_stamp":"2024-09-17T07:21:12.403277+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0071578960000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":28,"num_write":16,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":161009,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"439'35","reported_seq":567,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-17T07:23:30.377854+0000","last_change":"2024-09-17T07:23:06.353081+0000","last_active":"2024-09-17T07:23:30.377854+0000","last_peered":"2024-09-17T07:23:30.377854+0000","last_clean":"2024-09-17T07:23:30.377854+0000","last_became_active":"2024-09-17T07:21:15.347129+0000","last_became_peered":"2024-09-17T07:21:15.347129+0000","last_unstale":"2024-09-17T07:23:30.377854+0000","last_undegraded":"2024-09-17T07:23:30.377854+0000","last_fullsized":"2024-09-17T07:23:30.377854+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'25","last_scrub_stamp":"2024-09-17T07:15:02.532371+0000","last_deep_scrub":"176'15","last_deep_scrub_stamp":"2024-09-17T07:08:19.633791+0000","last_clean_scrub_stamp":"2024-09-17T07:15:02.532371+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012712606,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":366,"reported_epoch":463,"state":"active+undersized","last_fresh":"2024-09-17T07:23:30.534999+0000","last_change":"2024-09-17T07:22:54.386236+0000","last_active":"2024-09-17T07:23:30.534999+0000","last_peered":"2024-09-17T07:23:30.534999+0000","last_clean":"2024-09-17T07:22:52.353049+0000","last_became_active":"2024-09-17T07:22:54.386236+0000","last_became_peered":"2024-09-17T07:22:54.386236+0000","last_unstale":"2024-09-17T07:23:30.534999+0000","last_undegraded":"2024-09-17T07:23:30.534999+0000","last_fullsized":"2024-09-17T07:22:54.379869+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:11.580871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:20:11.580871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0065215869999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"448'12","reported_seq":67,"reported_epoch":463,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:23:30.535701+0000","last_change":"2024-09-17T07:22:54.383821+0000","last_active":"2024-09-17T07:23:30.535701+0000","last_peered":"2024-09-17T07:23:30.535701+0000","last_clean":"2024-09-17T07:22:52.359868+0000","last_became_active":"2024-09-17T07:22:54.383821+0000","last_became_peered":"2024-09-17T07:22:54.383821+0000","last_unstale":"2024-09-17T07:23:30.535701+0000","last_undegraded":"2024-09-17T07:22:54.379764+0000","last_fullsized":"2024-09-17T07:22:54.379654+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-09-17T07:19:24.964680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:19:24.964680+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":493,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-17T07:23:30.378095+0000","last_change":"2024-09-17T07:23:05.345276+0000","last_active":"2024-09-17T07:23:30.378095+0000","last_peered":"2024-09-17T07:23:30.378095+0000","last_clean":"2024-09-17T07:23:30.378095+0000","last_became_active":"2024-09-17T07:21:15.348799+0000","last_became_peered":"2024-09-17T07:21:15.348799+0000","last_unstale":"2024-09-17T07:23:30.378095+0000","last_undegraded":"2024-09-17T07:23:30.378095+0000","last_fullsized":"2024-09-17T07:23:30.378095+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:50.554591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:14:50.554591+0000","last_clean_scrub_stamp":"2024-09-17T07:14:50.554591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0067125969999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.0","version":"465'16129","reported_seq":29922,"reported_epoch":465,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-17T07:23:35.081650+0000","last_change":"2024-09-17T07:23:34.890459+0000","last_active":"2024-09-17T07:23:35.081650+0000","last_peered":"2024-09-17T07:23:35.081650+0000","last_clean":"2024-09-17T07:19:54.447320+0000","last_became_active":"2024-09-17T07:22:13.712336+0000","last_became_peered":"2024-09-17T07:22:13.712336+0000","last_unstale":"2024-09-17T07:23:35.081650+0000","last_undegraded":"2024-09-17T07:22:13.576336+0000","last_fullsized":"2024-09-17T07:22:13.550547+0000","mapping_epoch":433,"log_start":"424'11500","ondisk_log_start":"424'11500","created":390,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_clean_scrub_stamp":"2024-09-17T07:19:54.447320+0000","objects_scrubbed":0,"log_size":4629,"log_dups_size":0,"ondisk_log_size":4629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":367984640,"num_objects":5615,"num_object_clones":0,"num_object_copies":16845,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1707,"num_objects_misplaced":1707,"num_objects_unfound":0,"num_objects_dirty":5615,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16129,"num_write_kb":695808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4560,"num_bytes_recovered":298844160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[{"shards":"0,6","objects":5615}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.a","version":"212'130","reported_seq":469,"reported_epoch":463,"state":"active+undersized","last_fresh":"2024-09-17T07:23:30.536758+0000","last_change":"2024-09-17T07:22:54.391254+0000","last_active":"2024-09-17T07:23:30.536758+0000","last_peered":"2024-09-17T07:23:30.536758+0000","last_clean":"2024-09-17T07:10:37.677048+0000","last_became_active":"2024-09-17T07:22:54.391254+0000","last_became_peered":"2024-09-17T07:22:54.391254+0000","last_unstale":"2024-09-17T07:23:30.536758+0000","last_undegraded":"2024-09-17T07:23:30.536758+0000","last_fullsized":"2024-09-17T07:22:54.376447+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":4,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:20:33.709901+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:14:57.052505+0000","last_clean_scrub_stamp":"2024-09-17T07:20:33.709901+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021141889999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":27,"num_write":16,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":161009,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"439'39","reported_seq":448,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-17T07:23:30.536136+0000","last_change":"2024-09-17T07:22:13.780063+0000","last_active":"2024-09-17T07:23:30.536136+0000","last_peered":"2024-09-17T07:23:30.536136+0000","last_clean":"2024-09-17T07:23:30.536136+0000","last_became_active":"2024-09-17T07:22:13.589782+0000","last_became_peered":"2024-09-17T07:22:13.589782+0000","last_unstale":"2024-09-17T07:23:30.536136+0000","last_undegraded":"2024-09-17T07:23:30.536136+0000","last_fullsized":"2024-09-17T07:23:30.536136+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'27","last_scrub_stamp":"2024-09-17T07:20:13.130718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:20:13.130718+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0081162040000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":374,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-17T07:23:30.535845+0000","last_change":"2024-09-17T07:22:13.592519+0000","last_active":"2024-09-17T07:23:30.535845+0000","last_peered":"2024-09-17T07:23:30.535845+0000","last_clean":"2024-09-17T07:23:30.535845+0000","last_became_active":"2024-09-17T07:22:13.591089+0000","last_became_peered":"2024-09-17T07:22:13.591089+0000","last_unstale":"2024-09-17T07:23:30.535845+0000","last_undegraded":"2024-09-17T07:23:30.535845+0000","last_fullsized":"2024-09-17T07:23:30.535845+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:14.001258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:20:14.001258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034116369999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"448'9","reported_seq":759,"reported_epoch":463,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:23:30.764898+0000","last_change":"2024-09-17T07:22:54.386949+0000","last_active":"2024-09-17T07:23:30.764898+0000","last_peered":"2024-09-17T07:23:30.764898+0000","last_clean":"2024-09-17T07:22:52.353978+0000","last_became_active":"2024-09-17T07:22:54.386949+0000","last_became_peered":"2024-09-17T07:22:54.386949+0000","last_unstale":"2024-09-17T07:23:30.764898+0000","last_undegraded":"2024-09-17T07:22:54.376337+0000","last_fullsized":"2024-09-17T07:22:54.376238+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:19:24.039495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:19:24.039495+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.016850853999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[{"shards":"0,4","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":453,"reported_epoch":463,"state":"active+undersized","last_fresh":"2024-09-17T07:23:30.535684+0000","last_change":"2024-09-17T07:22:54.388186+0000","last_active":"2024-09-17T07:23:30.535684+0000","last_peered":"2024-09-17T07:23:30.535684+0000","last_clean":"2024-09-17T07:16:16.170588+0000","last_became_active":"2024-09-17T07:22:54.388186+0000","last_became_peered":"2024-09-17T07:22:54.388186+0000","last_unstale":"2024-09-17T07:23:30.535684+0000","last_undegraded":"2024-09-17T07:23:30.535684+0000","last_fullsized":"2024-09-17T07:22:54.378868+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:56.146098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:14:56.146098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0054479960000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"461'15797","reported_seq":16007,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-17T07:23:30.536024+0000","last_change":"2024-09-17T07:23:14.271452+0000","last_active":"2024-09-17T07:23:30.536024+0000","last_peered":"2024-09-17T07:23:30.536024+0000","last_clean":"2024-09-17T07:23:30.536024+0000","last_became_active":"2024-09-17T07:19:55.482080+0000","last_became_peered":"2024-09-17T07:19:55.482080+0000","last_unstale":"2024-09-17T07:23:30.536024+0000","last_undegraded":"2024-09-17T07:23:30.536024+0000","last_fullsized":"2024-09-17T07:23:30.536024+0000","mapping_epoch":390,"log_start":"424'11500","ondisk_log_start":"424'11500","created":390,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":0,"last_scrub":"450'15793","last_scrub_stamp":"2024-09-17T07:23:14.271325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_clean_scrub_stamp":"2024-09-17T07:23:14.271325+0000","objects_scrubbed":5711,"log_size":4297,"log_dups_size":0,"ondisk_log_size":4297,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:24:33.635412+0000","scrub_duration":0.71308822999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":374013952,"num_objects":5707,"num_object_clones":0,"num_object_copies":17121,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15797,"num_write_kb":688128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"439'27","reported_seq":539,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-17T07:23:30.378303+0000","last_change":"2024-09-17T07:23:05.345837+0000","last_active":"2024-09-17T07:23:30.378303+0000","last_peered":"2024-09-17T07:23:30.378303+0000","last_clean":"2024-09-17T07:23:30.378303+0000","last_became_active":"2024-09-17T07:21:15.347662+0000","last_became_peered":"2024-09-17T07:21:15.347662+0000","last_unstale":"2024-09-17T07:23:30.378303+0000","last_undegraded":"2024-09-17T07:23:30.378303+0000","last_fullsized":"2024-09-17T07:23:30.378303+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'19","last_scrub_stamp":"2024-09-17T07:14:55.489874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:14:55.489874+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045924440000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":40,"reported_epoch":462,"state":"active+clean","last_fresh":"2024-09-17T07:23:28.792697+0000","last_change":"2024-09-17T07:22:13.711817+0000","last_active":"2024-09-17T07:23:28.792697+0000","last_peered":"2024-09-17T07:23:28.792697+0000","last_clean":"2024-09-17T07:23:28.792697+0000","last_became_active":"2024-09-17T07:22:13.709873+0000","last_became_peered":"2024-09-17T07:22:13.709873+0000","last_unstale":"2024-09-17T07:23:28.792697+0000","last_undegraded":"2024-09-17T07:23:28.792697+0000","last_fullsized":"2024-09-17T07:23:28.792697+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:12.442421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:20:12.442421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026862600000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":25,"reported_epoch":463,"state":"active+undersized","last_fresh":"2024-09-17T07:23:30.378198+0000","last_change":"2024-09-17T07:22:54.388009+0000","last_active":"2024-09-17T07:23:30.378198+0000","last_peered":"2024-09-17T07:23:30.378198+0000","last_clean":"2024-09-17T07:02:14.999435+0000","last_became_active":"2024-09-17T07:22:54.388009+0000","last_became_peered":"2024-09-17T07:22:54.388009+0000","last_unstale":"2024-09-17T07:23:30.378198+0000","last_undegraded":"2024-09-17T07:23:30.378198+0000","last_fullsized":"2024-09-17T07:22:54.375510+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:30.427245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:19:30.427245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":386,"reported_epoch":463,"state":"active+undersized","last_fresh":"2024-09-17T07:23:30.535597+0000","last_change":"2024-09-17T07:22:54.387387+0000","last_active":"2024-09-17T07:23:30.535597+0000","last_peered":"2024-09-17T07:23:30.535597+0000","last_clean":"2024-09-17T07:22:52.354995+0000","last_became_active":"2024-09-17T07:22:54.387387+0000","last_became_peered":"2024-09-17T07:22:54.387387+0000","last_unstale":"2024-09-17T07:23:30.535597+0000","last_undegraded":"2024-09-17T07:23:30.535597+0000","last_fullsized":"2024-09-17T07:22:54.378113+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:18:40.925918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:13:22.187127+0000","last_clean_scrub_stamp":"2024-09-17T07:18:40.925918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0076924289999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.e","version":"461'16206","reported_seq":33666,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-17T07:23:30.377888+0000","last_change":"2024-09-17T07:23:25.386140+0000","last_active":"2024-09-17T07:23:30.377888+0000","last_peered":"2024-09-17T07:23:30.377888+0000","last_clean":"2024-09-17T07:23:30.377888+0000","last_became_active":"2024-09-17T07:23:25.355762+0000","last_became_peered":"2024-09-17T07:23:25.355762+0000","last_unstale":"2024-09-17T07:23:30.377888+0000","last_undegraded":"2024-09-17T07:23:30.377888+0000","last_fullsized":"2024-09-17T07:23:30.377888+0000","mapping_epoch":459,"log_start":"424'11575","ondisk_log_start":"424'11575","created":390,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_clean_scrub_stamp":"2024-09-17T07:19:54.447320+0000","objects_scrubbed":0,"log_size":4631,"log_dups_size":0,"ondisk_log_size":4631,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":378667008,"num_objects":5778,"num_object_clones":0,"num_object_copies":17334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16206,"num_write_kb":703488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5779,"num_bytes_recovered":378732544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"212'130","reported_seq":575,"reported_epoch":462,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:23:28.793001+0000","last_change":"2024-09-17T07:22:54.388196+0000","last_active":"2024-09-17T07:23:28.793001+0000","last_peered":"2024-09-17T07:23:28.793001+0000","last_clean":"2024-09-17T07:22:10.349422+0000","last_became_active":"2024-09-17T07:22:54.388196+0000","last_became_peered":"2024-09-17T07:22:54.388196+0000","last_unstale":"2024-09-17T07:23:28.793001+0000","last_undegraded":"2024-09-17T07:22:54.380533+0000","last_fullsized":"2024-09-17T07:22:54.380428+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":3,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:20:40.628770+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:14:57.052505+0000","last_clean_scrub_stamp":"2024-09-17T07:20:40.628770+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0027179940000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[{"shards":"6,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"439'42","reported_seq":560,"reported_epoch":463,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:23:30.377319+0000","last_change":"2024-09-17T07:22:54.378768+0000","last_active":"2024-09-17T07:23:30.377319+0000","last_peered":"2024-09-17T07:23:30.377319+0000","last_clean":"2024-09-17T07:22:52.352692+0000","last_became_active":"2024-09-17T07:22:54.378768+0000","last_became_peered":"2024-09-17T07:22:54.378768+0000","last_unstale":"2024-09-17T07:23:30.377319+0000","last_undegraded":"2024-09-17T07:22:54.374502+0000","last_fullsized":"2024-09-17T07:22:54.374392+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'28","last_scrub_stamp":"2024-09-17T07:20:28.029279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:20:28.029279+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0079911889999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":63,"num_read_kb":56,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":7}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":563,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-17T07:23:30.536579+0000","last_change":"2024-09-17T07:23:13.560619+0000","last_active":"2024-09-17T07:23:30.536579+0000","last_peered":"2024-09-17T07:23:30.536579+0000","last_clean":"2024-09-17T07:23:30.536579+0000","last_became_active":"2024-09-17T07:22:13.582096+0000","last_became_peered":"2024-09-17T07:22:13.582096+0000","last_unstale":"2024-09-17T07:23:30.536579+0000","last_undegraded":"2024-09-17T07:23:30.536579+0000","last_fullsized":"2024-09-17T07:23:30.536579+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:27.831228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:09:23.181028+0000","last_clean_scrub_stamp":"2024-09-17T07:19:27.831228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002288249,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"414'11","reported_seq":330,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-17T07:23:30.764614+0000","last_change":"2024-09-17T07:21:15.356155+0000","last_active":"2024-09-17T07:23:30.764614+0000","last_peered":"2024-09-17T07:23:30.764614+0000","last_clean":"2024-09-17T07:23:30.764614+0000","last_became_active":"2024-09-17T07:21:15.355651+0000","last_became_peered":"2024-09-17T07:21:15.355651+0000","last_unstale":"2024-09-17T07:23:30.764614+0000","last_undegraded":"2024-09-17T07:23:30.764614+0000","last_fullsized":"2024-09-17T07:23:30.764614+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'9","last_scrub_stamp":"2024-09-17T07:15:55.813882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:15:55.813882+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":41,"reported_epoch":462,"state":"active+clean","last_fresh":"2024-09-17T07:23:28.792786+0000","last_change":"2024-09-17T07:23:28.792786+0000","last_active":"2024-09-17T07:23:28.792786+0000","last_peered":"2024-09-17T07:23:28.792786+0000","last_clean":"2024-09-17T07:23:28.792786+0000","last_became_active":"2024-09-17T07:22:13.568656+0000","last_became_peered":"2024-09-17T07:22:13.568656+0000","last_unstale":"2024-09-17T07:23:28.792786+0000","last_undegraded":"2024-09-17T07:23:28.792786+0000","last_fullsized":"2024-09-17T07:23:28.792786+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:18:46.921412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:10:33.727472+0000","last_clean_scrub_stamp":"2024-09-17T07:18:46.921412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.f","version":"461'15951","reported_seq":32986,"reported_epoch":462,"state":"active+clean","last_fresh":"2024-09-17T07:23:28.792239+0000","last_change":"2024-09-17T07:23:13.655613+0000","last_active":"2024-09-17T07:23:28.792239+0000","last_peered":"2024-09-17T07:23:28.792239+0000","last_clean":"2024-09-17T07:23:28.792239+0000","last_became_active":"2024-09-17T07:23:13.619631+0000","last_became_peered":"2024-09-17T07:23:13.619631+0000","last_unstale":"2024-09-17T07:23:28.792239+0000","last_undegraded":"2024-09-17T07:23:28.792239+0000","last_fullsized":"2024-09-17T07:23:28.792239+0000","mapping_epoch":455,"log_start":"424'11890","ondisk_log_start":"424'11890","created":390,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_clean_scrub_stamp":"2024-09-17T07:19:54.447320+0000","objects_scrubbed":0,"log_size":4061,"log_dups_size":0,"ondisk_log_size":4061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369295360,"num_objects":5635,"num_object_clones":0,"num_object_copies":16905,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15951,"num_write_kb":690752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5642,"num_bytes_recovered":369754112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"212'130","reported_seq":558,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-17T07:23:30.377821+0000","last_change":"2024-09-17T07:20:35.080459+0000","last_active":"2024-09-17T07:23:30.377821+0000","last_peered":"2024-09-17T07:23:30.377821+0000","last_clean":"2024-09-17T07:23:30.377821+0000","last_became_active":"2024-09-17T07:19:35.420337+0000","last_became_peered":"2024-09-17T07:19:35.420337+0000","last_unstale":"2024-09-17T07:23:30.377821+0000","last_undegraded":"2024-09-17T07:23:30.377821+0000","last_fullsized":"2024-09-17T07:23:30.377821+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":3,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:20:35.080257+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:20:35.080257+0000","last_clean_scrub_stamp":"2024-09-17T07:20:35.080257+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0030477730000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"439'42","reported_seq":76,"reported_epoch":462,"state":"active+clean","last_fresh":"2024-09-17T07:23:28.792409+0000","last_change":"2024-09-17T07:22:13.779383+0000","last_active":"2024-09-17T07:23:28.792409+0000","last_peered":"2024-09-17T07:23:28.792409+0000","last_clean":"2024-09-17T07:23:28.792409+0000","last_became_active":"2024-09-17T07:22:13.711444+0000","last_became_peered":"2024-09-17T07:22:13.711444+0000","last_unstale":"2024-09-17T07:23:28.792409+0000","last_undegraded":"2024-09-17T07:23:28.792409+0000","last_fullsized":"2024-09-17T07:23:28.792409+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'30","last_scrub_stamp":"2024-09-17T07:20:13.427290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:20:13.427290+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0050927619999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":68,"num_read_kb":56,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":370,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-17T07:23:30.535896+0000","last_change":"2024-09-17T07:21:15.348172+0000","last_active":"2024-09-17T07:23:30.535896+0000","last_peered":"2024-09-17T07:23:30.535896+0000","last_clean":"2024-09-17T07:23:30.535896+0000","last_became_active":"2024-09-17T07:21:15.347867+0000","last_became_peered":"2024-09-17T07:21:15.347867+0000","last_unstale":"2024-09-17T07:23:30.535896+0000","last_undegraded":"2024-09-17T07:23:30.535896+0000","last_fullsized":"2024-09-17T07:23:30.535896+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:56.848689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:56.848689+0000","last_clean_scrub_stamp":"2024-09-17T07:15:56.848689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":527,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-17T07:23:30.377212+0000","last_change":"2024-09-17T07:23:27.713373+0000","last_active":"2024-09-17T07:23:30.377212+0000","last_peered":"2024-09-17T07:23:30.377212+0000","last_clean":"2024-09-17T07:23:30.377212+0000","last_became_active":"2024-09-17T07:22:13.566916+0000","last_became_peered":"2024-09-17T07:22:13.566916+0000","last_unstale":"2024-09-17T07:23:30.377212+0000","last_undegraded":"2024-09-17T07:23:30.377212+0000","last_fullsized":"2024-09-17T07:23:30.377212+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:23:27.713294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:23:27.713294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:24:50.872700+0000","scrub_duration":0.0011838949999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":496,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-17T07:23:30.377574+0000","last_change":"2024-09-17T07:19:27.090311+0000","last_active":"2024-09-17T07:23:30.377574+0000","last_peered":"2024-09-17T07:23:30.377574+0000","last_clean":"2024-09-17T07:23:30.377574+0000","last_became_active":"2024-09-17T07:09:24.398659+0000","last_became_peered":"2024-09-17T07:09:24.398659+0000","last_unstale":"2024-09-17T07:23:30.377574+0000","last_undegraded":"2024-09-17T07:23:30.377574+0000","last_fullsized":"2024-09-17T07:23:30.377574+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:27.090201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:19:27.090201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026497019999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"461'15837","reported_seq":27358,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-17T07:23:30.534757+0000","last_change":"2024-09-17T07:23:06.399317+0000","last_active":"2024-09-17T07:23:30.534757+0000","last_peered":"2024-09-17T07:23:30.534757+0000","last_clean":"2024-09-17T07:23:30.534757+0000","last_became_active":"2024-09-17T07:23:06.353991+0000","last_became_peered":"2024-09-17T07:23:06.353991+0000","last_unstale":"2024-09-17T07:23:30.534757+0000","last_undegraded":"2024-09-17T07:23:30.534757+0000","last_fullsized":"2024-09-17T07:23:30.534757+0000","mapping_epoch":451,"log_start":"424'11608","ondisk_log_start":"424'11608","created":390,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_clean_scrub_stamp":"2024-09-17T07:19:54.447320+0000","objects_scrubbed":0,"log_size":4229,"log_dups_size":0,"ondisk_log_size":4229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":373751808,"num_objects":5703,"num_object_clones":0,"num_object_copies":17109,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15837,"num_write_kb":689280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5708,"num_bytes_recovered":374079488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"212'130","reported_seq":566,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-17T07:23:30.535283+0000","last_change":"2024-09-17T07:20:33.068709+0000","last_active":"2024-09-17T07:23:30.535283+0000","last_peered":"2024-09-17T07:23:30.535283+0000","last_clean":"2024-09-17T07:23:30.535283+0000","last_became_active":"2024-09-17T07:19:35.428161+0000","last_became_peered":"2024-09-17T07:19:35.428161+0000","last_unstale":"2024-09-17T07:23:30.535283+0000","last_undegraded":"2024-09-17T07:23:30.535283+0000","last_fullsized":"2024-09-17T07:23:30.535283+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":3,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:20:33.068618+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:14:57.052505+0000","last_clean_scrub_stamp":"2024-09-17T07:20:33.068618+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0059947730000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"439'39","reported_seq":629,"reported_epoch":464,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:23:33.398380+0000","last_change":"2024-09-17T07:22:54.390705+0000","last_active":"2024-09-17T07:23:33.398380+0000","last_peered":"2024-09-17T07:23:33.398380+0000","last_clean":"2024-09-17T07:22:52.356735+0000","last_became_active":"2024-09-17T07:22:54.390705+0000","last_became_peered":"2024-09-17T07:22:54.390705+0000","last_unstale":"2024-09-17T07:23:33.398380+0000","last_undegraded":"2024-09-17T07:22:54.382520+0000","last_fullsized":"2024-09-17T07:22:54.381324+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'29","last_scrub_stamp":"2024-09-17T07:19:35.438907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:19:35.438907+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010401474000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[{"shards":"0,7","objects":8}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":40,"reported_epoch":462,"state":"active+clean","last_fresh":"2024-09-17T07:23:28.792464+0000","last_change":"2024-09-17T07:22:13.625084+0000","last_active":"2024-09-17T07:23:28.792464+0000","last_peered":"2024-09-17T07:23:28.792464+0000","last_clean":"2024-09-17T07:23:28.792464+0000","last_became_active":"2024-09-17T07:22:13.624802+0000","last_became_peered":"2024-09-17T07:22:13.624802+0000","last_unstale":"2024-09-17T07:23:28.792464+0000","last_undegraded":"2024-09-17T07:23:28.792464+0000","last_fullsized":"2024-09-17T07:23:28.792464+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:37.930831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:47.631387+0000","last_clean_scrub_stamp":"2024-09-17T07:19:37.930831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":515,"reported_epoch":463,"state":"active+undersized","last_fresh":"2024-09-17T07:23:30.535918+0000","last_change":"2024-09-17T07:22:54.391524+0000","last_active":"2024-09-17T07:23:30.535918+0000","last_peered":"2024-09-17T07:23:30.535918+0000","last_clean":"2024-09-17T07:22:52.354474+0000","last_became_active":"2024-09-17T07:22:54.391524+0000","last_became_peered":"2024-09-17T07:22:54.391524+0000","last_unstale":"2024-09-17T07:23:30.535918+0000","last_undegraded":"2024-09-17T07:23:30.535918+0000","last_fullsized":"2024-09-17T07:22:54.377018+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:31.968422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:11:17.583712+0000","last_clean_scrub_stamp":"2024-09-17T07:19:31.968422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.076819294999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":325,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-17T07:23:30.535667+0000","last_change":"2024-09-17T07:21:15.351926+0000","last_active":"2024-09-17T07:23:30.535667+0000","last_peered":"2024-09-17T07:23:30.535667+0000","last_clean":"2024-09-17T07:23:30.535667+0000","last_became_active":"2024-09-17T07:21:15.351524+0000","last_became_peered":"2024-09-17T07:21:15.351524+0000","last_unstale":"2024-09-17T07:23:30.535667+0000","last_undegraded":"2024-09-17T07:23:30.535667+0000","last_fullsized":"2024-09-17T07:23:30.535667+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:13:50.514525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:13:50.514525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.d","version":"461'15777","reported_seq":15862,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-17T07:23:30.377825+0000","last_change":"2024-09-17T07:19:55.480674+0000","last_active":"2024-09-17T07:23:30.377825+0000","last_peered":"2024-09-17T07:23:30.377825+0000","last_clean":"2024-09-17T07:23:30.377825+0000","last_became_active":"2024-09-17T07:19:55.480318+0000","last_became_peered":"2024-09-17T07:19:55.480318+0000","last_unstale":"2024-09-17T07:23:30.377825+0000","last_undegraded":"2024-09-17T07:23:30.377825+0000","last_fullsized":"2024-09-17T07:23:30.377825+0000","mapping_epoch":390,"log_start":"424'11182","ondisk_log_start":"424'11182","created":390,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_clean_scrub_stamp":"2024-09-17T07:19:54.447320+0000","objects_scrubbed":0,"log_size":4595,"log_dups_size":0,"ondisk_log_size":4595,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":372899864,"num_objects":5691,"num_object_clones":0,"num_object_copies":17073,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5691,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15777,"num_write_kb":686913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"212'130","reported_seq":621,"reported_epoch":463,"state":"active+undersized","last_fresh":"2024-09-17T07:23:30.536085+0000","last_change":"2024-09-17T07:22:54.387317+0000","last_active":"2024-09-17T07:23:30.536085+0000","last_peered":"2024-09-17T07:23:30.536085+0000","last_clean":"2024-09-17T07:22:52.355256+0000","last_became_active":"2024-09-17T07:22:54.387317+0000","last_became_peered":"2024-09-17T07:22:54.387317+0000","last_unstale":"2024-09-17T07:23:30.536085+0000","last_undegraded":"2024-09-17T07:23:30.536085+0000","last_fullsized":"2024-09-17T07:22:54.378245+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":3,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:19:19.991044+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:14:57.052505+0000","last_clean_scrub_stamp":"2024-09-17T07:19:19.991044+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021141889999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":28,"num_write":16,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":161009,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"439'35","reported_seq":397,"reported_epoch":463,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:23:30.534817+0000","last_change":"2024-09-17T07:22:54.380606+0000","last_active":"2024-09-17T07:23:30.534817+0000","last_peered":"2024-09-17T07:23:30.534817+0000","last_clean":"2024-09-17T07:22:52.352928+0000","last_became_active":"2024-09-17T07:22:54.380606+0000","last_became_peered":"2024-09-17T07:22:54.380606+0000","last_unstale":"2024-09-17T07:23:30.534817+0000","last_undegraded":"2024-09-17T07:22:54.378019+0000","last_fullsized":"2024-09-17T07:22:54.377908+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'25","last_scrub_stamp":"2024-09-17T07:14:50.794200+0000","last_deep_scrub":"110'15","last_deep_scrub_stamp":"2024-09-17T07:04:57.091400+0000","last_clean_scrub_stamp":"2024-09-17T07:14:50.794200+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0070558269999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":350,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-17T07:23:30.535110+0000","last_change":"2024-09-17T07:22:13.592639+0000","last_active":"2024-09-17T07:23:30.535110+0000","last_peered":"2024-09-17T07:23:30.535110+0000","last_clean":"2024-09-17T07:23:30.535110+0000","last_became_active":"2024-09-17T07:22:13.591372+0000","last_became_peered":"2024-09-17T07:22:13.591372+0000","last_unstale":"2024-09-17T07:23:30.535110+0000","last_undegraded":"2024-09-17T07:23:30.535110+0000","last_fullsized":"2024-09-17T07:23:30.535110+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:24.437342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:25.147878+0000","last_clean_scrub_stamp":"2024-09-17T07:20:24.437342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0024354329999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"448'10","reported_seq":772,"reported_epoch":464,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:23:33.398178+0000","last_change":"2024-09-17T07:22:54.390310+0000","last_active":"2024-09-17T07:23:33.398178+0000","last_peered":"2024-09-17T07:23:33.398178+0000","last_clean":"2024-09-17T07:22:52.356220+0000","last_became_active":"2024-09-17T07:22:54.390310+0000","last_became_peered":"2024-09-17T07:22:54.390310+0000","last_unstale":"2024-09-17T07:23:33.398178+0000","last_undegraded":"2024-09-17T07:22:54.380896+0000","last_fullsized":"2024-09-17T07:22:54.380790+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'9","last_scrub_stamp":"2024-09-17T07:21:14.471255+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-09-17T07:15:53.512777+0000","last_clean_scrub_stamp":"2024-09-17T07:21:14.471255+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032915969999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[{"shards":"0,6","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"268'2","reported_seq":508,"reported_epoch":464,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:23:33.397077+0000","last_change":"2024-09-17T07:22:54.390131+0000","last_active":"2024-09-17T07:23:33.397077+0000","last_peered":"2024-09-17T07:23:33.397077+0000","last_clean":"2024-09-17T07:22:52.354473+0000","last_became_active":"2024-09-17T07:22:54.390131+0000","last_became_peered":"2024-09-17T07:22:54.390131+0000","last_unstale":"2024-09-17T07:23:33.397077+0000","last_undegraded":"2024-09-17T07:22:54.380132+0000","last_fullsized":"2024-09-17T07:22:54.380031+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'2","last_scrub_stamp":"2024-09-17T07:18:49.469494+0000","last_deep_scrub":"268'2","last_deep_scrub_stamp":"2024-09-17T07:14:03.609131+0000","last_clean_scrub_stamp":"2024-09-17T07:18:49.469494+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037058479999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"461'15585","reported_seq":29241,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-17T07:23:30.535007+0000","last_change":"2024-09-17T07:22:52.425607+0000","last_active":"2024-09-17T07:23:30.535007+0000","last_peered":"2024-09-17T07:23:30.535007+0000","last_clean":"2024-09-17T07:23:30.535007+0000","last_became_active":"2024-09-17T07:22:52.355998+0000","last_became_peered":"2024-09-17T07:22:52.355998+0000","last_unstale":"2024-09-17T07:23:30.535007+0000","last_undegraded":"2024-09-17T07:23:30.535007+0000","last_fullsized":"2024-09-17T07:23:30.535007+0000","mapping_epoch":445,"log_start":"424'11335","ondisk_log_start":"424'11335","created":390,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_clean_scrub_stamp":"2024-09-17T07:19:54.447320+0000","objects_scrubbed":0,"log_size":4250,"log_dups_size":0,"ondisk_log_size":4250,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":365101056,"num_objects":5571,"num_object_clones":0,"num_object_copies":16713,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15585,"num_write_kb":676992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6774,"num_bytes_recovered":443940864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"212'130","reported_seq":478,"reported_epoch":464,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:23:33.398478+0000","last_change":"2024-09-17T07:22:54.388646+0000","last_active":"2024-09-17T07:23:33.398478+0000","last_peered":"2024-09-17T07:23:33.398478+0000","last_clean":"2024-09-17T07:10:37.677048+0000","last_became_active":"2024-09-17T07:22:54.388646+0000","last_became_peered":"2024-09-17T07:22:54.388646+0000","last_unstale":"2024-09-17T07:23:33.398478+0000","last_undegraded":"2024-09-17T07:22:54.380147+0000","last_fullsized":"2024-09-17T07:22:54.380019+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":4,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:20:22.674350+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:14:57.052505+0000","last_clean_scrub_stamp":"2024-09-17T07:20:22.674350+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021141889999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":28,"num_write":17,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":161010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"439'55","reported_seq":639,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-17T07:23:30.764634+0000","last_change":"2024-09-17T07:21:15.352832+0000","last_active":"2024-09-17T07:23:30.764634+0000","last_peered":"2024-09-17T07:23:30.764634+0000","last_clean":"2024-09-17T07:23:30.764634+0000","last_became_active":"2024-09-17T07:21:15.352461+0000","last_became_peered":"2024-09-17T07:21:15.352461+0000","last_unstale":"2024-09-17T07:23:30.764634+0000","last_undegraded":"2024-09-17T07:23:30.764634+0000","last_fullsized":"2024-09-17T07:23:30.764634+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'41","last_scrub_stamp":"2024-09-17T07:14:56.700858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:14:56.700858+0000","objects_scrubbed":10,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026058610000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":83,"num_read_kb":68,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":460,"reported_epoch":463,"state":"active+undersized","last_fresh":"2024-09-17T07:23:30.534604+0000","last_change":"2024-09-17T07:22:54.392280+0000","last_active":"2024-09-17T07:23:30.534604+0000","last_peered":"2024-09-17T07:23:30.534604+0000","last_clean":"2024-09-17T07:16:16.170577+0000","last_became_active":"2024-09-17T07:22:54.392280+0000","last_became_peered":"2024-09-17T07:22:54.392280+0000","last_unstale":"2024-09-17T07:23:30.534604+0000","last_undegraded":"2024-09-17T07:23:30.534604+0000","last_fullsized":"2024-09-17T07:22:54.378507+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:03.045709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:14:03.045709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0049245929999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":494,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-17T07:23:33.397517+0000","last_change":"2024-09-17T07:23:20.355177+0000","last_active":"2024-09-17T07:23:33.397517+0000","last_peered":"2024-09-17T07:23:33.397517+0000","last_clean":"2024-09-17T07:23:33.397517+0000","last_became_active":"2024-09-17T07:21:15.347069+0000","last_became_peered":"2024-09-17T07:21:15.347069+0000","last_unstale":"2024-09-17T07:23:33.397517+0000","last_undegraded":"2024-09-17T07:23:33.397517+0000","last_fullsized":"2024-09-17T07:23:33.397517+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:36.523676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:15:36.523676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017370549999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":386,"reported_epoch":463,"state":"active+undersized","last_fresh":"2024-09-17T07:23:30.378077+0000","last_change":"2024-09-17T07:22:54.384875+0000","last_active":"2024-09-17T07:23:30.378077+0000","last_peered":"2024-09-17T07:23:30.378077+0000","last_clean":"2024-09-17T07:22:52.353759+0000","last_became_active":"2024-09-17T07:22:54.384875+0000","last_became_peered":"2024-09-17T07:22:54.384875+0000","last_unstale":"2024-09-17T07:23:30.378077+0000","last_undegraded":"2024-09-17T07:23:30.378077+0000","last_fullsized":"2024-09-17T07:22:54.375760+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:20.067928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:13:53.406661+0000","last_clean_scrub_stamp":"2024-09-17T07:20:20.067928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0042634170000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.8","version":"461'15973","reported_seq":16063,"reported_epoch":463,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:23:30.535743+0000","last_change":"2024-09-17T07:22:54.389063+0000","last_active":"2024-09-17T07:23:30.535743+0000","last_peered":"2024-09-17T07:23:30.535743+0000","last_clean":"2024-09-17T07:22:53.366869+0000","last_became_active":"2024-09-17T07:22:54.389063+0000","last_became_peered":"2024-09-17T07:22:54.389063+0000","last_unstale":"2024-09-17T07:23:30.535743+0000","last_undegraded":"2024-09-17T07:22:54.378510+0000","last_fullsized":"2024-09-17T07:22:54.378247+0000","mapping_epoch":447,"log_start":"424'11700","ondisk_log_start":"424'11700","created":390,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_clean_scrub_stamp":"2024-09-17T07:19:54.447320+0000","objects_scrubbed":0,"log_size":4273,"log_dups_size":0,"ondisk_log_size":4273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":375062528,"num_objects":5723,"num_object_clones":0,"num_object_copies":17169,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5723,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15973,"num_write_kb":694272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[{"shards":"0,6","objects":5723}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.2","version":"212'130","reported_seq":552,"reported_epoch":462,"state":"active+clean","last_fresh":"2024-09-17T07:23:28.792740+0000","last_change":"2024-09-17T07:22:16.360792+0000","last_active":"2024-09-17T07:23:28.792740+0000","last_peered":"2024-09-17T07:23:28.792740+0000","last_clean":"2024-09-17T07:23:28.792740+0000","last_became_active":"2024-09-17T07:22:16.360258+0000","last_became_peered":"2024-09-17T07:22:16.360258+0000","last_unstale":"2024-09-17T07:23:28.792740+0000","last_undegraded":"2024-09-17T07:23:28.792740+0000","last_fullsized":"2024-09-17T07:23:28.792740+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":4,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:19:19.991044+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:14:57.052505+0000","last_clean_scrub_stamp":"2024-09-17T07:19:19.991044+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021141889999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":28,"num_write":17,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":161010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"439'21","reported_seq":373,"reported_epoch":463,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:23:30.536094+0000","last_change":"2024-09-17T07:22:54.389302+0000","last_active":"2024-09-17T07:23:30.536094+0000","last_peered":"2024-09-17T07:23:30.536094+0000","last_clean":"2024-09-17T07:22:52.354944+0000","last_became_active":"2024-09-17T07:22:54.389302+0000","last_became_peered":"2024-09-17T07:22:54.389302+0000","last_unstale":"2024-09-17T07:23:30.536094+0000","last_undegraded":"2024-09-17T07:22:54.380163+0000","last_fullsized":"2024-09-17T07:22:54.380061+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'17","last_scrub_stamp":"2024-09-17T07:13:47.468399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:13:47.468399+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.005881725,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":30,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"76'5","reported_seq":1297,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-17T07:23:32.662577+0000","last_change":"2024-09-17T07:21:15.358093+0000","last_active":"2024-09-17T07:23:32.662577+0000","last_peered":"2024-09-17T07:23:32.662577+0000","last_clean":"2024-09-17T07:23:32.662577+0000","last_became_active":"2024-09-17T07:21:15.357178+0000","last_became_peered":"2024-09-17T07:21:15.357178+0000","last_unstale":"2024-09-17T07:23:32.662577+0000","last_undegraded":"2024-09-17T07:23:32.662577+0000","last_fullsized":"2024-09-17T07:23:32.662577+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'5","last_scrub_stamp":"2024-09-17T07:15:36.901601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:15:36.901601+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033739799999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":144,"num_read_kb":139,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":338,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-17T07:23:33.398245+0000","last_change":"2024-09-17T07:23:24.654517+0000","last_active":"2024-09-17T07:23:33.398245+0000","last_peered":"2024-09-17T07:23:33.398245+0000","last_clean":"2024-09-17T07:23:33.398245+0000","last_became_active":"2024-09-17T07:21:15.346287+0000","last_became_peered":"2024-09-17T07:21:15.346287+0000","last_unstale":"2024-09-17T07:23:33.398245+0000","last_undegraded":"2024-09-17T07:23:33.398245+0000","last_fullsized":"2024-09-17T07:23:33.398245+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:56.534547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:23.833170+0000","last_clean_scrub_stamp":"2024-09-17T07:15:56.534547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0011018950000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":544,"reported_epoch":462,"state":"active+undersized","last_fresh":"2024-09-17T07:23:28.400605+0000","last_change":"2024-09-17T07:22:54.384937+0000","last_active":"2024-09-17T07:23:28.400605+0000","last_peered":"2024-09-17T07:23:28.400605+0000","last_clean":"2024-09-17T07:22:52.360476+0000","last_became_active":"2024-09-17T07:22:54.384937+0000","last_became_peered":"2024-09-17T07:22:54.384937+0000","last_unstale":"2024-09-17T07:23:28.400605+0000","last_undegraded":"2024-09-17T07:23:28.400605+0000","last_fullsized":"2024-09-17T07:22:54.375879+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:16.776553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:18:42.051667+0000","last_clean_scrub_stamp":"2024-09-17T07:20:16.776553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035831560000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":40,"reported_epoch":462,"state":"active+clean","last_fresh":"2024-09-17T07:23:28.792480+0000","last_change":"2024-09-17T07:22:13.629124+0000","last_active":"2024-09-17T07:23:28.792480+0000","last_peered":"2024-09-17T07:23:28.792480+0000","last_clean":"2024-09-17T07:23:28.792480+0000","last_became_active":"2024-09-17T07:22:13.628359+0000","last_became_peered":"2024-09-17T07:22:13.628359+0000","last_unstale":"2024-09-17T07:23:28.792480+0000","last_undegraded":"2024-09-17T07:23:28.792480+0000","last_fullsized":"2024-09-17T07:23:28.792480+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:27.909402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:02.062618+0000","last_clean_scrub_stamp":"2024-09-17T07:19:27.909402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0041575259999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.9","version":"461'16012","reported_seq":34881,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-17T07:23:30.536112+0000","last_change":"2024-09-17T07:23:24.652091+0000","last_active":"2024-09-17T07:23:30.536112+0000","last_peered":"2024-09-17T07:23:30.536112+0000","last_clean":"2024-09-17T07:23:30.536112+0000","last_became_active":"2024-09-17T07:22:14.579277+0000","last_became_peered":"2024-09-17T07:22:14.579277+0000","last_unstale":"2024-09-17T07:23:30.536112+0000","last_undegraded":"2024-09-17T07:23:30.536112+0000","last_fullsized":"2024-09-17T07:23:30.536112+0000","mapping_epoch":434,"log_start":"424'11700","ondisk_log_start":"424'11700","created":390,"last_epoch_clean":435,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_clean_scrub_stamp":"2024-09-17T07:19:54.447320+0000","objects_scrubbed":0,"log_size":4312,"log_dups_size":0,"ondisk_log_size":4312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362020864,"num_objects":5524,"num_object_clones":0,"num_object_copies":16572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16012,"num_write_kb":689152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9383,"num_bytes_recovered":614924288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.3","version":"212'130","reported_seq":578,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-17T07:23:30.534719+0000","last_change":"2024-09-17T07:21:18.358222+0000","last_active":"2024-09-17T07:23:30.534719+0000","last_peered":"2024-09-17T07:23:30.534719+0000","last_clean":"2024-09-17T07:23:30.534719+0000","last_became_active":"2024-09-17T07:21:18.357874+0000","last_became_peered":"2024-09-17T07:21:18.357874+0000","last_unstale":"2024-09-17T07:23:30.534719+0000","last_undegraded":"2024-09-17T07:23:30.534719+0000","last_fullsized":"2024-09-17T07:23:30.534719+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":3,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:20:34.113602+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:20:34.113602+0000","last_clean_scrub_stamp":"2024-09-17T07:20:34.113602+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021141889999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":28,"num_write":17,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":161010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"439'42","reported_seq":579,"reported_epoch":464,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:23:33.399052+0000","last_change":"2024-09-17T07:22:54.393221+0000","last_active":"2024-09-17T07:23:33.399052+0000","last_peered":"2024-09-17T07:23:33.399052+0000","last_clean":"2024-09-17T07:22:52.357611+0000","last_became_active":"2024-09-17T07:22:54.393221+0000","last_became_peered":"2024-09-17T07:22:54.393221+0000","last_unstale":"2024-09-17T07:23:33.399052+0000","last_undegraded":"2024-09-17T07:22:54.382769+0000","last_fullsized":"2024-09-17T07:22:54.382666+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'30","last_scrub_stamp":"2024-09-17T07:18:40.426597+0000","last_deep_scrub":"268'30","last_deep_scrub_stamp":"2024-09-17T07:14:00.599656+0000","last_clean_scrub_stamp":"2024-09-17T07:18:40.426597+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0031470769999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":6}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"70'1","reported_seq":518,"reported_epoch":464,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:23:33.398116+0000","last_change":"2024-09-17T07:22:54.386180+0000","last_active":"2024-09-17T07:23:33.398116+0000","last_peered":"2024-09-17T07:23:33.398116+0000","last_clean":"2024-09-17T07:22:52.355198+0000","last_became_active":"2024-09-17T07:22:54.386180+0000","last_became_peered":"2024-09-17T07:22:54.386180+0000","last_unstale":"2024-09-17T07:23:33.398116+0000","last_undegraded":"2024-09-17T07:22:54.383378+0000","last_fullsized":"2024-09-17T07:22:54.380904+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-17T07:19:19.477140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:19:19.477140+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011313337,"objects_trimmed":0,"snaptrim_duration":0.0032298629999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0,4","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":40,"reported_epoch":462,"state":"active+clean","last_fresh":"2024-09-17T07:23:28.792492+0000","last_change":"2024-09-17T07:22:13.736269+0000","last_active":"2024-09-17T07:23:28.792492+0000","last_peered":"2024-09-17T07:23:28.792492+0000","last_clean":"2024-09-17T07:23:28.792492+0000","last_became_active":"2024-09-17T07:22:13.710211+0000","last_became_peered":"2024-09-17T07:22:13.710211+0000","last_unstale":"2024-09-17T07:23:28.792492+0000","last_undegraded":"2024-09-17T07:23:28.792492+0000","last_fullsized":"2024-09-17T07:23:28.792492+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:34.449627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:55.321202+0000","last_clean_scrub_stamp":"2024-09-17T07:20:34.449627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0044019280000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":455,"reported_epoch":463,"state":"active+undersized","last_fresh":"2024-09-17T07:23:30.536177+0000","last_change":"2024-09-17T07:22:54.388263+0000","last_active":"2024-09-17T07:23:30.536177+0000","last_peered":"2024-09-17T07:23:30.536177+0000","last_clean":"2024-09-17T07:16:16.169907+0000","last_became_active":"2024-09-17T07:22:54.388263+0000","last_became_peered":"2024-09-17T07:22:54.388263+0000","last_unstale":"2024-09-17T07:23:30.536177+0000","last_undegraded":"2024-09-17T07:23:30.536177+0000","last_fullsized":"2024-09-17T07:22:54.379826+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:45.447029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:15:45.447029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001990463,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":368,"reported_epoch":463,"state":"active+undersized","last_fresh":"2024-09-17T07:23:30.535227+0000","last_change":"2024-09-17T07:22:54.389451+0000","last_active":"2024-09-17T07:23:30.535227+0000","last_peered":"2024-09-17T07:23:30.535227+0000","last_clean":"2024-09-17T07:22:52.353394+0000","last_became_active":"2024-09-17T07:22:54.389451+0000","last_became_peered":"2024-09-17T07:22:54.389451+0000","last_unstale":"2024-09-17T07:23:30.535227+0000","last_undegraded":"2024-09-17T07:23:30.535227+0000","last_fullsized":"2024-09-17T07:22:54.379710+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:51.877367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:15:51.877367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.058290885000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"446'15681","reported_seq":27194,"reported_epoch":464,"state":"peering","last_fresh":"2024-09-17T07:23:33.400317+0000","last_change":"2024-09-17T07:23:33.400412+0000","last_active":"2024-09-17T07:22:53.399600+0000","last_peered":"2024-09-17T07:23:32.983178+0000","last_clean":"2024-09-17T07:19:54.447320+0000","last_became_active":"2024-09-17T07:21:16.466454+0000","last_became_peered":"2024-09-17T07:22:54.398720+0000","last_unstale":"2024-09-17T07:23:33.400317+0000","last_undegraded":"2024-09-17T07:23:33.400317+0000","last_fullsized":"2024-09-17T07:23:33.400317+0000","mapping_epoch":464,"log_start":"424'11400","ondisk_log_start":"424'11400","created":390,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_clean_scrub_stamp":"2024-09-17T07:19:54.447320+0000","objects_scrubbed":0,"log_size":4281,"log_dups_size":0,"ondisk_log_size":4281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":372572160,"num_objects":5685,"num_object_clones":0,"num_object_copies":17055,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15681,"num_write_kb":683712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5696,"num_bytes_recovered":373293056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"439'38","reported_seq":610,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-17T07:23:30.535698+0000","last_change":"2024-09-17T07:22:13.636910+0000","last_active":"2024-09-17T07:23:30.535698+0000","last_peered":"2024-09-17T07:23:30.535698+0000","last_clean":"2024-09-17T07:23:30.535698+0000","last_became_active":"2024-09-17T07:22:13.583204+0000","last_became_peered":"2024-09-17T07:22:13.583204+0000","last_unstale":"2024-09-17T07:23:30.535698+0000","last_undegraded":"2024-09-17T07:23:30.535698+0000","last_fullsized":"2024-09-17T07:23:30.535698+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'26","last_scrub_stamp":"2024-09-17T07:20:19.046676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:20:19.046676+0000","objects_scrubbed":7,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0059907520000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"268'3","reported_seq":146,"reported_epoch":463,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:23:30.536736+0000","last_change":"2024-09-17T07:22:54.382951+0000","last_active":"2024-09-17T07:23:30.536736+0000","last_peered":"2024-09-17T07:23:30.536736+0000","last_clean":"2024-09-17T07:13:54.946149+0000","last_became_active":"2024-09-17T07:22:54.382951+0000","last_became_peered":"2024-09-17T07:22:54.382951+0000","last_unstale":"2024-09-17T07:23:30.536736+0000","last_undegraded":"2024-09-17T07:22:54.376163+0000","last_fullsized":"2024-09-17T07:22:54.376060+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'3","last_scrub_stamp":"2024-09-17T07:19:29.367269+0000","last_deep_scrub":"268'3","last_deep_scrub_stamp":"2024-09-17T07:13:54.949345+0000","last_clean_scrub_stamp":"2024-09-17T07:19:29.367269+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0018649420000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[{"shards":"4,6","objects":1}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":515,"reported_epoch":462,"state":"active+undersized","last_fresh":"2024-09-17T07:23:28.792589+0000","last_change":"2024-09-17T07:22:54.383933+0000","last_active":"2024-09-17T07:23:28.792589+0000","last_peered":"2024-09-17T07:23:28.792589+0000","last_clean":"2024-09-17T07:22:10.349842+0000","last_became_active":"2024-09-17T07:22:54.383933+0000","last_became_peered":"2024-09-17T07:22:54.383933+0000","last_unstale":"2024-09-17T07:23:28.792589+0000","last_undegraded":"2024-09-17T07:23:28.792589+0000","last_fullsized":"2024-09-17T07:22:54.379734+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:17.650824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:20:17.650824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019547470000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"212'130","reported_seq":562,"reported_epoch":462,"state":"active+clean","last_fresh":"2024-09-17T07:23:28.791764+0000","last_change":"2024-09-17T07:22:16.366636+0000","last_active":"2024-09-17T07:23:28.791764+0000","last_peered":"2024-09-17T07:23:28.791764+0000","last_clean":"2024-09-17T07:23:28.791764+0000","last_became_active":"2024-09-17T07:22:16.366295+0000","last_became_peered":"2024-09-17T07:22:16.366295+0000","last_unstale":"2024-09-17T07:23:28.791764+0000","last_undegraded":"2024-09-17T07:23:28.791764+0000","last_fullsized":"2024-09-17T07:23:28.791764+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:19:19.991044+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:14:57.052505+0000","last_clean_scrub_stamp":"2024-09-17T07:19:19.991044+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021141889999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":27,"num_write":16,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":161009,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":46,"reported_epoch":462,"state":"active+undersized","last_fresh":"2024-09-17T07:23:28.793617+0000","last_change":"2024-09-17T07:22:54.384933+0000","last_active":"2024-09-17T07:23:28.793617+0000","last_peered":"2024-09-17T07:23:28.793617+0000","last_clean":"2024-09-17T07:22:52.360021+0000","last_became_active":"2024-09-17T07:22:54.384933+0000","last_became_peered":"2024-09-17T07:22:54.384933+0000","last_unstale":"2024-09-17T07:23:28.793617+0000","last_undegraded":"2024-09-17T07:23:28.793617+0000","last_fullsized":"2024-09-17T07:22:54.379491+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:18:48.891136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:18:48.891136+0000","last_clean_scrub_stamp":"2024-09-17T07:18:48.891136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":451,"reported_epoch":463,"state":"active+undersized","last_fresh":"2024-09-17T07:23:30.378054+0000","last_change":"2024-09-17T07:22:54.381481+0000","last_active":"2024-09-17T07:23:30.378054+0000","last_peered":"2024-09-17T07:23:30.378054+0000","last_clean":"2024-09-17T07:16:16.170840+0000","last_became_active":"2024-09-17T07:22:54.381481+0000","last_became_peered":"2024-09-17T07:22:54.381481+0000","last_unstale":"2024-09-17T07:23:30.378054+0000","last_undegraded":"2024-09-17T07:23:30.378054+0000","last_fullsized":"2024-09-17T07:22:54.375105+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:48.021592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:58.670571+0000","last_clean_scrub_stamp":"2024-09-17T07:14:48.021592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0064844619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"465'15423","reported_seq":15572,"reported_epoch":465,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-17T07:23:35.080175+0000","last_change":"2024-09-17T07:23:34.888866+0000","last_active":"2024-09-17T07:23:35.080175+0000","last_peered":"2024-09-17T07:23:35.080175+0000","last_clean":"2024-09-17T07:22:12.495976+0000","last_became_active":"2024-09-17T07:22:54.390716+0000","last_became_peered":"2024-09-17T07:22:54.390716+0000","last_unstale":"2024-09-17T07:23:35.080175+0000","last_undegraded":"2024-09-17T07:22:54.381389+0000","last_fullsized":"2024-09-17T07:22:54.381222+0000","mapping_epoch":447,"log_start":"424'10850","ondisk_log_start":"424'10850","created":390,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_clean_scrub_stamp":"2024-09-17T07:19:54.447320+0000","objects_scrubbed":0,"log_size":4573,"log_dups_size":0,"ondisk_log_size":4573,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":360251392,"num_objects":5497,"num_object_clones":0,"num_object_copies":16491,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5475,"num_objects_misplaced":5497,"num_objects_unfound":0,"num_objects_dirty":5497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15423,"num_write_kb":669440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":1507328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[{"shards":"0,6","objects":5497}],"blocked_by":[],"up_primary":7,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"439'51","reported_seq":654,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-17T07:23:30.536704+0000","last_change":"2024-09-17T07:22:13.759654+0000","last_active":"2024-09-17T07:23:30.536704+0000","last_peered":"2024-09-17T07:23:30.536704+0000","last_clean":"2024-09-17T07:23:30.536704+0000","last_became_active":"2024-09-17T07:22:13.588260+0000","last_became_peered":"2024-09-17T07:22:13.588260+0000","last_unstale":"2024-09-17T07:23:30.536704+0000","last_undegraded":"2024-09-17T07:23:30.536704+0000","last_fullsized":"2024-09-17T07:23:30.536704+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'35","last_scrub_stamp":"2024-09-17T07:20:15.012345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:20:15.012345+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0050499799999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":555,"reported_epoch":462,"state":"active+undersized","last_fresh":"2024-09-17T07:23:28.400699+0000","last_change":"2024-09-17T07:22:54.387733+0000","last_active":"2024-09-17T07:23:28.400699+0000","last_peered":"2024-09-17T07:23:28.400699+0000","last_clean":"2024-09-17T07:22:52.359984+0000","last_became_active":"2024-09-17T07:22:54.387733+0000","last_became_peered":"2024-09-17T07:22:54.387733+0000","last_unstale":"2024-09-17T07:23:28.400699+0000","last_undegraded":"2024-09-17T07:23:28.400699+0000","last_fullsized":"2024-09-17T07:22:54.376221+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:17.749290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:06:39.322535+0000","last_clean_scrub_stamp":"2024-09-17T07:20:17.749290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0062185959999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":538,"reported_epoch":462,"state":"active+undersized","last_fresh":"2024-09-17T07:23:28.399501+0000","last_change":"2024-09-17T07:22:54.387725+0000","last_active":"2024-09-17T07:23:28.399501+0000","last_peered":"2024-09-17T07:23:28.399501+0000","last_clean":"2024-09-17T07:22:52.359863+0000","last_became_active":"2024-09-17T07:22:54.387725+0000","last_became_peered":"2024-09-17T07:22:54.387725+0000","last_unstale":"2024-09-17T07:23:28.399501+0000","last_undegraded":"2024-09-17T07:23:28.399501+0000","last_fullsized":"2024-09-17T07:22:54.375878+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:12.821593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:20:12.821593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0031314899999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":40,"reported_epoch":462,"state":"active+clean","last_fresh":"2024-09-17T07:23:28.792855+0000","last_change":"2024-09-17T07:22:13.573689+0000","last_active":"2024-09-17T07:23:28.792855+0000","last_peered":"2024-09-17T07:23:28.792855+0000","last_clean":"2024-09-17T07:23:28.792855+0000","last_became_active":"2024-09-17T07:22:13.573161+0000","last_became_peered":"2024-09-17T07:22:13.573161+0000","last_unstale":"2024-09-17T07:23:28.792855+0000","last_undegraded":"2024-09-17T07:23:28.792855+0000","last_fullsized":"2024-09-17T07:23:28.792855+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:18:47.880943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:45.275473+0000","last_clean_scrub_stamp":"2024-09-17T07:18:47.880943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010463343,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.4","version":"461'15935","reported_seq":45243,"reported_epoch":462,"state":"active+clean","last_fresh":"2024-09-17T07:23:28.792309+0000","last_change":"2024-09-17T07:22:51.420491+0000","last_active":"2024-09-17T07:23:28.792309+0000","last_peered":"2024-09-17T07:23:28.792309+0000","last_clean":"2024-09-17T07:23:28.792309+0000","last_became_active":"2024-09-17T07:22:51.376094+0000","last_became_peered":"2024-09-17T07:22:51.376094+0000","last_unstale":"2024-09-17T07:23:28.792309+0000","last_undegraded":"2024-09-17T07:23:28.792309+0000","last_fullsized":"2024-09-17T07:23:28.792309+0000","mapping_epoch":444,"log_start":"424'11783","ondisk_log_start":"424'11783","created":390,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_clean_scrub_stamp":"2024-09-17T07:19:54.447320+0000","objects_scrubbed":0,"log_size":4152,"log_dups_size":0,"ondisk_log_size":4152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":375324672,"num_objects":5727,"num_object_clones":0,"num_object_copies":17181,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15935,"num_write_kb":693184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14597,"num_bytes_recovered":956563456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"439'25","reported_seq":522,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-17T07:23:33.398295+0000","last_change":"2024-09-17T07:23:17.625424+0000","last_active":"2024-09-17T07:23:33.398295+0000","last_peered":"2024-09-17T07:23:33.398295+0000","last_clean":"2024-09-17T07:23:33.398295+0000","last_became_active":"2024-09-17T07:21:15.349361+0000","last_became_peered":"2024-09-17T07:21:15.349361+0000","last_unstale":"2024-09-17T07:23:33.398295+0000","last_undegraded":"2024-09-17T07:23:33.398295+0000","last_fullsized":"2024-09-17T07:23:33.398295+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'19","last_scrub_stamp":"2024-09-17T07:14:06.634334+0000","last_deep_scrub":"268'19","last_deep_scrub_stamp":"2024-09-17T07:14:06.634334+0000","last_clean_scrub_stamp":"2024-09-17T07:14:06.634334+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0088943289999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":379,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-17T07:23:30.535417+0000","last_change":"2024-09-17T07:21:15.352018+0000","last_active":"2024-09-17T07:23:30.535417+0000","last_peered":"2024-09-17T07:23:30.535417+0000","last_clean":"2024-09-17T07:23:30.535417+0000","last_became_active":"2024-09-17T07:21:15.350786+0000","last_became_peered":"2024-09-17T07:21:15.350786+0000","last_unstale":"2024-09-17T07:23:30.535417+0000","last_undegraded":"2024-09-17T07:23:30.535417+0000","last_fullsized":"2024-09-17T07:23:30.535417+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:58.886294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:13:53.593078+0000","last_clean_scrub_stamp":"2024-09-17T07:15:58.886294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.005853245,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":462,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-17T07:23:30.377414+0000","last_change":"2024-09-17T07:19:30.094643+0000","last_active":"2024-09-17T07:23:30.377414+0000","last_peered":"2024-09-17T07:23:30.377414+0000","last_clean":"2024-09-17T07:23:30.377414+0000","last_became_active":"2024-09-17T07:09:24.396642+0000","last_became_peered":"2024-09-17T07:09:24.396642+0000","last_unstale":"2024-09-17T07:23:30.377414+0000","last_undegraded":"2024-09-17T07:23:30.377414+0000","last_fullsized":"2024-09-17T07:23:30.377414+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:30.094567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:19:30.094567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001242611,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":504,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-17T07:23:33.398176+0000","last_change":"2024-09-17T07:22:13.589795+0000","last_active":"2024-09-17T07:23:33.398176+0000","last_peered":"2024-09-17T07:23:33.398176+0000","last_clean":"2024-09-17T07:23:33.398176+0000","last_became_active":"2024-09-17T07:22:13.589460+0000","last_became_peered":"2024-09-17T07:22:13.589460+0000","last_unstale":"2024-09-17T07:23:33.398176+0000","last_undegraded":"2024-09-17T07:23:33.398176+0000","last_fullsized":"2024-09-17T07:23:33.398176+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:29.434010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:58.099849+0000","last_clean_scrub_stamp":"2024-09-17T07:19:29.434010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.032605186000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.7","version":"461'15884","reported_seq":34300,"reported_epoch":463,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:23:30.535919+0000","last_change":"2024-09-17T07:23:26.374176+0000","last_active":"2024-09-17T07:23:30.535919+0000","last_peered":"2024-09-17T07:23:30.535919+0000","last_clean":"2024-09-17T07:19:54.447320+0000","last_became_active":"2024-09-17T07:23:26.374176+0000","last_became_peered":"2024-09-17T07:23:26.374176+0000","last_unstale":"2024-09-17T07:23:30.535919+0000","last_undegraded":"2024-09-17T07:23:26.370597+0000","last_fullsized":"2024-09-17T07:23:26.370397+0000","mapping_epoch":460,"log_start":"424'11647","ondisk_log_start":"424'11647","created":390,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_clean_scrub_stamp":"2024-09-17T07:19:54.447320+0000","objects_scrubbed":0,"log_size":4237,"log_dups_size":0,"ondisk_log_size":4237,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370409472,"num_objects":5652,"num_object_clones":0,"num_object_copies":16956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5652,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5652,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15884,"num_write_kb":689152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9151,"num_bytes_recovered":599719936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":5652}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"439'46","reported_seq":614,"reported_epoch":462,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:23:28.791961+0000","last_change":"2024-09-17T07:22:54.388231+0000","last_active":"2024-09-17T07:23:28.791961+0000","last_peered":"2024-09-17T07:23:28.791961+0000","last_clean":"2024-09-17T07:22:21.659860+0000","last_became_active":"2024-09-17T07:22:54.388231+0000","last_became_peered":"2024-09-17T07:22:54.388231+0000","last_unstale":"2024-09-17T07:23:28.791961+0000","last_undegraded":"2024-09-17T07:22:54.379888+0000","last_fullsized":"2024-09-17T07:22:54.379786+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'34","last_scrub_stamp":"2024-09-17T07:19:24.308050+0000","last_deep_scrub":"242'34","last_deep_scrub_stamp":"2024-09-17T07:14:50.103539+0000","last_clean_scrub_stamp":"2024-09-17T07:19:24.308050+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.019153434,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[{"shards":"6,7","objects":7}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"89'2","reported_seq":45,"reported_epoch":462,"state":"active+clean","last_fresh":"2024-09-17T07:23:28.793300+0000","last_change":"2024-09-17T07:23:28.793300+0000","last_active":"2024-09-17T07:23:28.793300+0000","last_peered":"2024-09-17T07:23:28.793300+0000","last_clean":"2024-09-17T07:23:28.793300+0000","last_became_active":"2024-09-17T07:22:13.575469+0000","last_became_peered":"2024-09-17T07:22:13.575469+0000","last_unstale":"2024-09-17T07:23:28.793300+0000","last_undegraded":"2024-09-17T07:23:28.793300+0000","last_fullsized":"2024-09-17T07:23:28.793300+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-09-17T07:18:43.861842+0000","last_deep_scrub":"89'2","last_deep_scrub_stamp":"2024-09-17T07:12:20.301263+0000","last_clean_scrub_stamp":"2024-09-17T07:18:43.861842+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":420,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":420,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"436'14","reported_seq":61,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-17T07:23:30.535209+0000","last_change":"2024-09-17T07:23:26.412214+0000","last_active":"2024-09-17T07:23:30.535209+0000","last_peered":"2024-09-17T07:23:30.535209+0000","last_clean":"2024-09-17T07:23:30.535209+0000","last_became_active":"2024-09-17T07:22:13.624050+0000","last_became_peered":"2024-09-17T07:22:13.624050+0000","last_unstale":"2024-09-17T07:23:30.535209+0000","last_undegraded":"2024-09-17T07:23:30.535209+0000","last_fullsized":"2024-09-17T07:23:30.535209+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"225'9","last_scrub_stamp":"2024-09-17T07:18:23.788769+0000","last_deep_scrub":"225'9","last_deep_scrub_stamp":"2024-09-17T07:11:25.292565+0000","last_clean_scrub_stamp":"2024-09-17T07:18:23.788769+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023108299999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":25,"reported_epoch":463,"state":"active+undersized","last_fresh":"2024-09-17T07:23:30.535804+0000","last_change":"2024-09-17T07:22:54.389586+0000","last_active":"2024-09-17T07:23:30.535804+0000","last_peered":"2024-09-17T07:23:30.535804+0000","last_clean":"2024-09-17T07:02:17.002904+0000","last_became_active":"2024-09-17T07:22:54.389586+0000","last_became_peered":"2024-09-17T07:22:54.389586+0000","last_unstale":"2024-09-17T07:23:30.535804+0000","last_undegraded":"2024-09-17T07:23:30.535804+0000","last_fullsized":"2024-09-17T07:22:54.376734+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:18:41.385928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:18:41.385928+0000","last_clean_scrub_stamp":"2024-09-17T07:18:41.385928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"465'15915","reported_seq":27473,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-09-17T07:23:35.063302+0000","last_change":"2024-09-17T07:23:08.441289+0000","last_active":"2024-09-17T07:23:35.063302+0000","last_peered":"2024-09-17T07:23:35.063302+0000","last_clean":"2024-09-17T07:23:35.063302+0000","last_became_active":"2024-09-17T07:23:08.376527+0000","last_became_peered":"2024-09-17T07:23:08.376527+0000","last_unstale":"2024-09-17T07:23:35.063302+0000","last_undegraded":"2024-09-17T07:23:35.063302+0000","last_fullsized":"2024-09-17T07:23:35.063302+0000","mapping_epoch":453,"log_start":"424'11300","ondisk_log_start":"424'11300","created":390,"last_epoch_clean":454,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:19:54.447320+0000","last_clean_scrub_stamp":"2024-09-17T07:19:54.447320+0000","objects_scrubbed":0,"log_size":4615,"log_dups_size":0,"ondisk_log_size":4615,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":366542848,"num_objects":5593,"num_object_clones":0,"num_object_copies":16779,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15915,"num_write_kb":688256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5724,"num_bytes_recovered":375128064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"439'29","reported_seq":589,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-17T07:23:30.377299+0000","last_change":"2024-09-17T07:20:33.108656+0000","last_active":"2024-09-17T07:23:30.377299+0000","last_peered":"2024-09-17T07:23:30.377299+0000","last_clean":"2024-09-17T07:23:30.377299+0000","last_became_active":"2024-09-17T07:10:49.723292+0000","last_became_peered":"2024-09-17T07:10:49.723292+0000","last_unstale":"2024-09-17T07:23:30.377299+0000","last_undegraded":"2024-09-17T07:23:30.377299+0000","last_fullsized":"2024-09-17T07:23:30.377299+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'23","last_scrub_stamp":"2024-09-17T07:20:33.108490+0000","last_deep_scrub":"150'17","last_deep_scrub_stamp":"2024-09-17T07:06:38.775788+0000","last_clean_scrub_stamp":"2024-09-17T07:20:33.108490+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0087601510000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"68'2","reported_seq":540,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-17T07:23:30.378376+0000","last_change":"2024-09-17T07:22:13.766671+0000","last_active":"2024-09-17T07:23:30.378376+0000","last_peered":"2024-09-17T07:23:30.378376+0000","last_clean":"2024-09-17T07:23:30.378376+0000","last_became_active":"2024-09-17T07:22:13.590172+0000","last_became_peered":"2024-09-17T07:22:13.590172+0000","last_unstale":"2024-09-17T07:23:30.378376+0000","last_undegraded":"2024-09-17T07:23:30.378376+0000","last_fullsized":"2024-09-17T07:23:30.378376+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-09-17T07:20:11.093491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:20:11.093491+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023198989999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":457,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-17T07:23:33.397321+0000","last_change":"2024-09-17T07:23:17.624682+0000","last_active":"2024-09-17T07:23:33.397321+0000","last_peered":"2024-09-17T07:23:33.397321+0000","last_clean":"2024-09-17T07:23:33.397321+0000","last_became_active":"2024-09-17T07:21:15.355760+0000","last_became_peered":"2024-09-17T07:21:15.355760+0000","last_unstale":"2024-09-17T07:23:33.397321+0000","last_undegraded":"2024-09-17T07:23:33.397321+0000","last_fullsized":"2024-09-17T07:23:33.397321+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:51.586929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:11:17.479036+0000","last_clean_scrub_stamp":"2024-09-17T07:15:51.586929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001274435,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":40,"reported_epoch":462,"state":"active+clean","last_fresh":"2024-09-17T07:23:28.792939+0000","last_change":"2024-09-17T07:22:13.585879+0000","last_active":"2024-09-17T07:23:28.792939+0000","last_peered":"2024-09-17T07:23:28.792939+0000","last_clean":"2024-09-17T07:23:28.792939+0000","last_became_active":"2024-09-17T07:22:13.585580+0000","last_became_peered":"2024-09-17T07:22:13.585580+0000","last_unstale":"2024-09-17T07:23:28.792939+0000","last_undegraded":"2024-09-17T07:23:28.792939+0000","last_fullsized":"2024-09-17T07:23:28.792939+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:07.959158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:20:07.959158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.018487173999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"439'21","reported_seq":581,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-17T07:23:33.399011+0000","last_change":"2024-09-17T07:23:24.655701+0000","last_active":"2024-09-17T07:23:33.399011+0000","last_peered":"2024-09-17T07:23:33.399011+0000","last_clean":"2024-09-17T07:23:33.399011+0000","last_became_active":"2024-09-17T07:22:13.592001+0000","last_became_peered":"2024-09-17T07:22:13.592001+0000","last_unstale":"2024-09-17T07:23:33.399011+0000","last_undegraded":"2024-09-17T07:23:33.399011+0000","last_fullsized":"2024-09-17T07:23:33.399011+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'15","last_scrub_stamp":"2024-09-17T07:18:24.505002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:18:24.505002+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002195809,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":537,"reported_epoch":462,"state":"active+clean","last_fresh":"2024-09-17T07:23:28.398551+0000","last_change":"2024-09-17T07:23:28.398551+0000","last_active":"2024-09-17T07:23:28.398551+0000","last_peered":"2024-09-17T07:23:28.398551+0000","last_clean":"2024-09-17T07:23:28.398551+0000","last_became_active":"2024-09-17T07:21:15.355464+0000","last_became_peered":"2024-09-17T07:21:15.355464+0000","last_unstale":"2024-09-17T07:23:28.398551+0000","last_undegraded":"2024-09-17T07:23:28.398551+0000","last_fullsized":"2024-09-17T07:23:28.398551+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:21:15.788348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:17.976267+0000","last_clean_scrub_stamp":"2024-09-17T07:21:15.788348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0012829899999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":200,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-17T07:23:30.535982+0000","last_change":"2024-09-17T07:22:13.593743+0000","last_active":"2024-09-17T07:23:30.535982+0000","last_peered":"2024-09-17T07:23:30.535982+0000","last_clean":"2024-09-17T07:23:30.535982+0000","last_became_active":"2024-09-17T07:22:13.593446+0000","last_became_peered":"2024-09-17T07:22:13.593446+0000","last_unstale":"2024-09-17T07:23:30.535982+0000","last_undegraded":"2024-09-17T07:23:30.535982+0000","last_fullsized":"2024-09-17T07:23:30.535982+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:29.492000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:18.367486+0000","last_clean_scrub_stamp":"2024-09-17T07:20:29.492000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0077309099999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":590,"reported_epoch":462,"state":"active+clean","last_fresh":"2024-09-17T07:23:28.400548+0000","last_change":"2024-09-17T07:23:12.813613+0000","last_active":"2024-09-17T07:23:28.400548+0000","last_peered":"2024-09-17T07:23:28.400548+0000","last_clean":"2024-09-17T07:23:28.400548+0000","last_became_active":"2024-09-17T07:22:13.590599+0000","last_became_peered":"2024-09-17T07:22:13.590599+0000","last_unstale":"2024-09-17T07:23:28.400548+0000","last_undegraded":"2024-09-17T07:23:28.400548+0000","last_fullsized":"2024-09-17T07:23:28.400548+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:23:12.813529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:09:23.378805+0000","last_clean_scrub_stamp":"2024-09-17T07:23:12.813529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:24:23.135348+0000","scrub_duration":0.0017852149999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":507,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-17T07:23:30.536436+0000","last_change":"2024-09-17T07:23:05.476673+0000","last_active":"2024-09-17T07:23:30.536436+0000","last_peered":"2024-09-17T07:23:30.536436+0000","last_clean":"2024-09-17T07:23:30.536436+0000","last_became_active":"2024-09-17T07:21:15.348758+0000","last_became_peered":"2024-09-17T07:21:15.348758+0000","last_unstale":"2024-09-17T07:23:30.536436+0000","last_undegraded":"2024-09-17T07:23:30.536436+0000","last_fullsized":"2024-09-17T07:23:30.536436+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:46.645080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:46.645080+0000","last_clean_scrub_stamp":"2024-09-17T07:15:46.645080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015661049999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"439'18","reported_seq":57,"reported_epoch":462,"state":"active+clean","last_fresh":"2024-09-17T07:23:28.793197+0000","last_change":"2024-09-17T07:22:13.761675+0000","last_active":"2024-09-17T07:23:28.793197+0000","last_peered":"2024-09-17T07:23:28.793197+0000","last_clean":"2024-09-17T07:23:28.793197+0000","last_became_active":"2024-09-17T07:22:13.586662+0000","last_became_peered":"2024-09-17T07:22:13.586662+0000","last_unstale":"2024-09-17T07:23:28.793197+0000","last_undegraded":"2024-09-17T07:23:28.793197+0000","last_fullsized":"2024-09-17T07:23:28.793197+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'12","last_scrub_stamp":"2024-09-17T07:20:17.090146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:20:17.090146+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019381719999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":25,"num_read_kb":22,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":463,"reported_epoch":463,"state":"active+undersized","last_fresh":"2024-09-17T07:23:30.535878+0000","last_change":"2024-09-17T07:22:54.389149+0000","last_active":"2024-09-17T07:23:30.535878+0000","last_peered":"2024-09-17T07:23:30.535878+0000","last_clean":"2024-09-17T07:16:16.170775+0000","last_became_active":"2024-09-17T07:22:54.389149+0000","last_became_peered":"2024-09-17T07:22:54.389149+0000","last_unstale":"2024-09-17T07:23:30.535878+0000","last_undegraded":"2024-09-17T07:23:30.535878+0000","last_fullsized":"2024-09-17T07:22:54.379832+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:55.191252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:14:55.191252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0090150360000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":40,"reported_epoch":462,"state":"active+clean","last_fresh":"2024-09-17T07:23:28.792382+0000","last_change":"2024-09-17T07:22:13.624074+0000","last_active":"2024-09-17T07:23:28.792382+0000","last_peered":"2024-09-17T07:23:28.792382+0000","last_clean":"2024-09-17T07:23:28.792382+0000","last_became_active":"2024-09-17T07:22:13.623754+0000","last_became_peered":"2024-09-17T07:22:13.623754+0000","last_unstale":"2024-09-17T07:23:28.792382+0000","last_undegraded":"2024-09-17T07:23:28.792382+0000","last_fullsized":"2024-09-17T07:23:28.792382+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:25.934136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:19:25.934136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"439'30","reported_seq":260,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-17T07:23:33.402053+0000","last_change":"2024-09-17T07:22:13.768713+0000","last_active":"2024-09-17T07:23:33.402053+0000","last_peered":"2024-09-17T07:23:33.402053+0000","last_clean":"2024-09-17T07:23:33.402053+0000","last_became_active":"2024-09-17T07:22:13.591094+0000","last_became_peered":"2024-09-17T07:22:13.591094+0000","last_unstale":"2024-09-17T07:23:33.402053+0000","last_undegraded":"2024-09-17T07:23:33.402053+0000","last_fullsized":"2024-09-17T07:23:33.402053+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'20","last_scrub_stamp":"2024-09-17T07:20:10.554324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:20:10.554324+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0020069329999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":523,"reported_epoch":462,"state":"active+undersized","last_fresh":"2024-09-17T07:23:28.792703+0000","last_change":"2024-09-17T07:22:54.390121+0000","last_active":"2024-09-17T07:23:28.792703+0000","last_peered":"2024-09-17T07:23:28.792703+0000","last_clean":"2024-09-17T07:22:10.349706+0000","last_became_active":"2024-09-17T07:22:54.390121+0000","last_became_peered":"2024-09-17T07:22:54.390121+0000","last_unstale":"2024-09-17T07:23:28.792703+0000","last_undegraded":"2024-09-17T07:23:28.792703+0000","last_fullsized":"2024-09-17T07:22:54.381950+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:40.443474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:19:40.443474+0000","last_clean_scrub_stamp":"2024-09-17T07:19:40.443474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002799733,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"448'12","reported_seq":702,"reported_epoch":463,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:23:30.764582+0000","last_change":"2024-09-17T07:22:54.387784+0000","last_active":"2024-09-17T07:23:30.764582+0000","last_peered":"2024-09-17T07:23:30.764582+0000","last_clean":"2024-09-17T07:16:16.169875+0000","last_became_active":"2024-09-17T07:22:54.387784+0000","last_became_peered":"2024-09-17T07:22:54.387784+0000","last_unstale":"2024-09-17T07:23:30.764582+0000","last_undegraded":"2024-09-17T07:22:54.378770+0000","last_fullsized":"2024-09-17T07:22:54.378494+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'9","last_scrub_stamp":"2024-09-17T07:14:53.140133+0000","last_deep_scrub":"163'9","last_deep_scrub_stamp":"2024-09-17T07:07:34.933808+0000","last_clean_scrub_stamp":"2024-09-17T07:14:53.140133+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0088819989999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":513,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-17T07:23:33.398653+0000","last_change":"2024-09-17T07:23:24.655405+0000","last_active":"2024-09-17T07:23:33.398653+0000","last_peered":"2024-09-17T07:23:33.398653+0000","last_clean":"2024-09-17T07:23:33.398653+0000","last_became_active":"2024-09-17T07:22:13.591307+0000","last_became_peered":"2024-09-17T07:22:13.591307+0000","last_unstale":"2024-09-17T07:23:33.398653+0000","last_undegraded":"2024-09-17T07:23:33.398653+0000","last_fullsized":"2024-09-17T07:23:33.398653+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:18:47.392182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:26.619893+0000","last_clean_scrub_stamp":"2024-09-17T07:18:47.392182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0053055120000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"439'21","reported_seq":586,"reported_epoch":462,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:23:28.792179+0000","last_change":"2024-09-17T07:22:54.390223+0000","last_active":"2024-09-17T07:23:28.792179+0000","last_peered":"2024-09-17T07:23:28.792179+0000","last_clean":"2024-09-17T07:22:22.072073+0000","last_became_active":"2024-09-17T07:22:54.390223+0000","last_became_peered":"2024-09-17T07:22:54.390223+0000","last_unstale":"2024-09-17T07:23:28.792179+0000","last_undegraded":"2024-09-17T07:22:54.380109+0000","last_fullsized":"2024-09-17T07:22:54.379996+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'15","last_scrub_stamp":"2024-09-17T07:19:20.304874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:19:20.304874+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0074478349999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[{"shards":"6,7","objects":4}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":523,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-17T07:23:33.398492+0000","last_change":"2024-09-17T07:22:13.592026+0000","last_active":"2024-09-17T07:23:33.398492+0000","last_peered":"2024-09-17T07:23:33.398492+0000","last_clean":"2024-09-17T07:23:33.398492+0000","last_became_active":"2024-09-17T07:22:13.589964+0000","last_became_peered":"2024-09-17T07:22:13.589964+0000","last_unstale":"2024-09-17T07:23:33.398492+0000","last_undegraded":"2024-09-17T07:23:33.398492+0000","last_fullsized":"2024-09-17T07:23:33.398492+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:33.544567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:19:33.544567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.13531478499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"436'15","reported_seq":60,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-17T07:23:30.535468+0000","last_change":"2024-09-17T07:22:13.736680+0000","last_active":"2024-09-17T07:23:30.535468+0000","last_peered":"2024-09-17T07:23:30.535468+0000","last_clean":"2024-09-17T07:23:30.535468+0000","last_became_active":"2024-09-17T07:22:13.586620+0000","last_became_peered":"2024-09-17T07:22:13.586620+0000","last_unstale":"2024-09-17T07:23:30.535468+0000","last_undegraded":"2024-09-17T07:23:30.535468+0000","last_fullsized":"2024-09-17T07:23:30.535468+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'12","last_scrub_stamp":"2024-09-17T07:20:09.105448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:20:09.105448+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":502,"reported_epoch":463,"state":"active+undersized","last_fresh":"2024-09-17T07:23:30.378141+0000","last_change":"2024-09-17T07:22:54.377624+0000","last_active":"2024-09-17T07:23:30.378141+0000","last_peered":"2024-09-17T07:23:30.378141+0000","last_clean":"2024-09-17T07:22:52.353136+0000","last_became_active":"2024-09-17T07:22:54.377624+0000","last_became_peered":"2024-09-17T07:22:54.377624+0000","last_unstale":"2024-09-17T07:23:30.378141+0000","last_undegraded":"2024-09-17T07:23:30.378141+0000","last_fullsized":"2024-09-17T07:22:54.375424+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:37.110458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:20:37.110458+0000","last_clean_scrub_stamp":"2024-09-17T07:20:37.110458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.017901013,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"439'21","reported_seq":361,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-17T07:23:33.402404+0000","last_change":"2024-09-17T07:21:15.349519+0000","last_active":"2024-09-17T07:23:33.402404+0000","last_peered":"2024-09-17T07:23:33.402404+0000","last_clean":"2024-09-17T07:23:33.402404+0000","last_became_active":"2024-09-17T07:21:15.348060+0000","last_became_peered":"2024-09-17T07:21:15.348060+0000","last_unstale":"2024-09-17T07:23:33.402404+0000","last_undegraded":"2024-09-17T07:23:33.402404+0000","last_fullsized":"2024-09-17T07:23:33.402404+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'15","last_scrub_stamp":"2024-09-17T07:13:48.500257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:13:48.500257+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.005053564,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":545,"reported_epoch":463,"state":"active+undersized","last_fresh":"2024-09-17T07:23:30.535785+0000","last_change":"2024-09-17T07:22:54.388885+0000","last_active":"2024-09-17T07:23:30.535785+0000","last_peered":"2024-09-17T07:23:30.535785+0000","last_clean":"2024-09-17T07:22:52.355085+0000","last_became_active":"2024-09-17T07:22:54.388885+0000","last_became_peered":"2024-09-17T07:22:54.388885+0000","last_unstale":"2024-09-17T07:23:30.535785+0000","last_undegraded":"2024-09-17T07:23:30.535785+0000","last_fullsized":"2024-09-17T07:22:54.378423+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:18:53.905079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:10:36.334827+0000","last_clean_scrub_stamp":"2024-09-17T07:18:53.905079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0087069899999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":342,"reported_epoch":464,"state":"active+undersized","last_fresh":"2024-09-17T07:23:33.398000+0000","last_change":"2024-09-17T07:22:54.392557+0000","last_active":"2024-09-17T07:23:33.398000+0000","last_peered":"2024-09-17T07:23:33.398000+0000","last_clean":"2024-09-17T07:22:52.356072+0000","last_became_active":"2024-09-17T07:22:54.392557+0000","last_became_peered":"2024-09-17T07:22:54.392557+0000","last_unstale":"2024-09-17T07:23:33.398000+0000","last_undegraded":"2024-09-17T07:23:33.398000+0000","last_fullsized":"2024-09-17T07:22:54.382710+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:37.523283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:15:37.523283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013475850000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":496,"reported_epoch":464,"state":"active+undersized","last_fresh":"2024-09-17T07:23:33.397217+0000","last_change":"2024-09-17T07:22:54.387290+0000","last_active":"2024-09-17T07:23:33.397217+0000","last_peered":"2024-09-17T07:23:33.397217+0000","last_clean":"2024-09-17T07:22:52.354510+0000","last_became_active":"2024-09-17T07:22:54.387290+0000","last_became_peered":"2024-09-17T07:22:54.387290+0000","last_unstale":"2024-09-17T07:23:33.397217+0000","last_undegraded":"2024-09-17T07:23:33.397217+0000","last_fullsized":"2024-09-17T07:22:54.380361+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:40.540336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:15:40.540336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001212982,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"439'27","reported_seq":537,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-17T07:23:30.377645+0000","last_change":"2024-09-17T07:21:16.070482+0000","last_active":"2024-09-17T07:23:30.377645+0000","last_peered":"2024-09-17T07:23:30.377645+0000","last_clean":"2024-09-17T07:23:30.377645+0000","last_became_active":"2024-09-17T07:21:15.347179+0000","last_became_peered":"2024-09-17T07:21:15.347179+0000","last_unstale":"2024-09-17T07:23:30.377645+0000","last_undegraded":"2024-09-17T07:23:30.377645+0000","last_fullsized":"2024-09-17T07:23:30.377645+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'19","last_scrub_stamp":"2024-09-17T07:21:16.070358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:21:16.070358+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0025953619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"94'2","reported_seq":367,"reported_epoch":463,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:23:30.535510+0000","last_change":"2024-09-17T07:22:54.388179+0000","last_active":"2024-09-17T07:23:30.535510+0000","last_peered":"2024-09-17T07:23:30.535510+0000","last_clean":"2024-09-17T07:22:52.353673+0000","last_became_active":"2024-09-17T07:22:54.388179+0000","last_became_peered":"2024-09-17T07:22:54.388179+0000","last_unstale":"2024-09-17T07:23:30.535510+0000","last_undegraded":"2024-09-17T07:22:54.378552+0000","last_fullsized":"2024-09-17T07:22:54.378440+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-09-17T07:15:48.907821+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-09-17T07:07:31.717888+0000","last_clean_scrub_stamp":"2024-09-17T07:15:48.907821+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":372,"reported_epoch":464,"state":"active+undersized","last_fresh":"2024-09-17T07:23:33.401881+0000","last_change":"2024-09-17T07:22:54.385936+0000","last_active":"2024-09-17T07:23:33.401881+0000","last_peered":"2024-09-17T07:23:33.401881+0000","last_clean":"2024-09-17T07:22:52.352716+0000","last_became_active":"2024-09-17T07:22:54.385936+0000","last_became_peered":"2024-09-17T07:22:54.385936+0000","last_unstale":"2024-09-17T07:23:33.401881+0000","last_undegraded":"2024-09-17T07:23:33.401881+0000","last_fullsized":"2024-09-17T07:22:54.378688+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:16.640384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:20:16.640384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003133224,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":337,"reported_epoch":464,"state":"active+undersized","last_fresh":"2024-09-17T07:23:33.398325+0000","last_change":"2024-09-17T07:22:54.389035+0000","last_active":"2024-09-17T07:23:33.398325+0000","last_peered":"2024-09-17T07:23:33.398325+0000","last_clean":"2024-09-17T07:22:52.355403+0000","last_became_active":"2024-09-17T07:22:54.389035+0000","last_became_peered":"2024-09-17T07:22:54.389035+0000","last_unstale":"2024-09-17T07:23:33.398325+0000","last_undegraded":"2024-09-17T07:23:33.398325+0000","last_fullsized":"2024-09-17T07:22:54.380611+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:18:23.515582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:49.573666+0000","last_clean_scrub_stamp":"2024-09-17T07:18:23.515582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0022931869999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"439'12","reported_seq":433,"reported_epoch":462,"state":"active+clean","last_fresh":"2024-09-17T07:23:28.398448+0000","last_change":"2024-09-17T07:23:10.820556+0000","last_active":"2024-09-17T07:23:28.398448+0000","last_peered":"2024-09-17T07:23:28.398448+0000","last_clean":"2024-09-17T07:23:28.398448+0000","last_became_active":"2024-09-17T07:09:24.396368+0000","last_became_peered":"2024-09-17T07:09:24.396368+0000","last_unstale":"2024-09-17T07:23:28.398448+0000","last_undegraded":"2024-09-17T07:23:28.398448+0000","last_fullsized":"2024-09-17T07:23:28.398448+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'12","last_scrub_stamp":"2024-09-17T07:23:10.820475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:23:10.820475+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:24:40.678256+0000","scrub_duration":0.0018307379999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":25,"reported_epoch":463,"state":"active+undersized","last_fresh":"2024-09-17T07:23:30.535375+0000","last_change":"2024-09-17T07:22:54.388754+0000","last_active":"2024-09-17T07:23:30.535375+0000","last_peered":"2024-09-17T07:23:30.535375+0000","last_clean":"2024-09-17T07:02:10.991363+0000","last_became_active":"2024-09-17T07:22:54.388754+0000","last_became_peered":"2024-09-17T07:22:54.388754+0000","last_unstale":"2024-09-17T07:23:30.535375+0000","last_undegraded":"2024-09-17T07:23:30.535375+0000","last_fullsized":"2024-09-17T07:22:54.377916+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:22.340320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:19:22.340320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":528,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-17T07:23:30.536301+0000","last_change":"2024-09-17T07:23:07.536531+0000","last_active":"2024-09-17T07:23:30.536301+0000","last_peered":"2024-09-17T07:23:30.536301+0000","last_clean":"2024-09-17T07:23:30.536301+0000","last_became_active":"2024-09-17T07:22:13.582229+0000","last_became_peered":"2024-09-17T07:22:13.582229+0000","last_unstale":"2024-09-17T07:23:30.536301+0000","last_undegraded":"2024-09-17T07:23:30.536301+0000","last_fullsized":"2024-09-17T07:23:30.536301+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:36.808726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:54.553970+0000","last_clean_scrub_stamp":"2024-09-17T07:19:36.808726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001250061,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":40,"reported_epoch":462,"state":"active+clean","last_fresh":"2024-09-17T07:23:28.792607+0000","last_change":"2024-09-17T07:22:13.624740+0000","last_active":"2024-09-17T07:23:28.792607+0000","last_peered":"2024-09-17T07:23:28.792607+0000","last_clean":"2024-09-17T07:23:28.792607+0000","last_became_active":"2024-09-17T07:22:13.624127+0000","last_became_peered":"2024-09-17T07:22:13.624127+0000","last_unstale":"2024-09-17T07:23:28.792607+0000","last_undegraded":"2024-09-17T07:23:28.792607+0000","last_fullsized":"2024-09-17T07:23:28.792607+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:25.003509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:20:25.003509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002164586,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"439'27","reported_seq":70,"reported_epoch":462,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:23:28.792114+0000","last_change":"2024-09-17T07:22:54.387978+0000","last_active":"2024-09-17T07:23:28.792114+0000","last_peered":"2024-09-17T07:23:28.792114+0000","last_clean":"2024-09-17T07:22:52.359289+0000","last_became_active":"2024-09-17T07:22:54.387978+0000","last_became_peered":"2024-09-17T07:22:54.387978+0000","last_unstale":"2024-09-17T07:23:28.792114+0000","last_undegraded":"2024-09-17T07:22:54.379895+0000","last_fullsized":"2024-09-17T07:22:54.379781+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'19","last_scrub_stamp":"2024-09-17T07:20:28.113977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:20:28.113977+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.013438802,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[{"shards":"6,7","objects":5}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":517,"reported_epoch":463,"state":"active+undersized","last_fresh":"2024-09-17T07:23:30.536527+0000","last_change":"2024-09-17T07:22:54.387437+0000","last_active":"2024-09-17T07:23:30.536527+0000","last_peered":"2024-09-17T07:23:30.536527+0000","last_clean":"2024-09-17T07:22:52.355450+0000","last_became_active":"2024-09-17T07:22:54.387437+0000","last_became_peered":"2024-09-17T07:22:54.387437+0000","last_unstale":"2024-09-17T07:23:30.536527+0000","last_undegraded":"2024-09-17T07:23:30.536527+0000","last_fullsized":"2024-09-17T07:22:54.378596+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:18:41.951382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:18:41.951382+0000","last_clean_scrub_stamp":"2024-09-17T07:18:41.951382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001516739,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":40,"reported_epoch":462,"state":"active+clean","last_fresh":"2024-09-17T07:23:28.792769+0000","last_change":"2024-09-17T07:22:13.736537+0000","last_active":"2024-09-17T07:23:28.792769+0000","last_peered":"2024-09-17T07:23:28.792769+0000","last_clean":"2024-09-17T07:23:28.792769+0000","last_became_active":"2024-09-17T07:22:13.710754+0000","last_became_peered":"2024-09-17T07:22:13.710754+0000","last_unstale":"2024-09-17T07:23:28.792769+0000","last_undegraded":"2024-09-17T07:23:28.792769+0000","last_fullsized":"2024-09-17T07:23:28.792769+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:30.038044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:20:30.038044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":342,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-17T07:23:33.397922+0000","last_change":"2024-09-17T07:19:25.448467+0000","last_active":"2024-09-17T07:23:33.397922+0000","last_peered":"2024-09-17T07:23:33.397922+0000","last_clean":"2024-09-17T07:23:33.397922+0000","last_became_active":"2024-09-17T07:09:24.392216+0000","last_became_peered":"2024-09-17T07:09:24.392216+0000","last_unstale":"2024-09-17T07:23:33.397922+0000","last_undegraded":"2024-09-17T07:23:33.397922+0000","last_fullsized":"2024-09-17T07:23:33.397922+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:25.448373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:19:25.448373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033889219999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1c","version":"326'2","reported_seq":50,"reported_epoch":462,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:23:28.793264+0000","last_change":"2024-09-17T07:22:54.384842+0000","last_active":"2024-09-17T07:23:28.793264+0000","last_peered":"2024-09-17T07:23:28.793264+0000","last_clean":"2024-09-17T07:22:52.359696+0000","last_became_active":"2024-09-17T07:22:54.384842+0000","last_became_peered":"2024-09-17T07:22:54.384842+0000","last_unstale":"2024-09-17T07:23:28.793264+0000","last_undegraded":"2024-09-17T07:22:54.380214+0000","last_fullsized":"2024-09-17T07:22:54.380116+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'2","last_scrub_stamp":"2024-09-17T07:20:19.609678+0000","last_deep_scrub":"326'2","last_deep_scrub_stamp":"2024-09-17T07:17:36.974358+0000","last_clean_scrub_stamp":"2024-09-17T07:20:19.609678+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0053255050000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":505,"reported_epoch":463,"state":"active+undersized","last_fresh":"2024-09-17T07:23:30.377265+0000","last_change":"2024-09-17T07:22:54.382752+0000","last_active":"2024-09-17T07:23:30.377265+0000","last_peered":"2024-09-17T07:23:30.377265+0000","last_clean":"2024-09-17T07:22:10.349351+0000","last_became_active":"2024-09-17T07:22:54.382752+0000","last_became_peered":"2024-09-17T07:22:54.382752+0000","last_unstale":"2024-09-17T07:23:30.377265+0000","last_undegraded":"2024-09-17T07:23:30.377265+0000","last_fullsized":"2024-09-17T07:22:54.374323+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:23.314136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:19:23.314136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010788304,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"439'35","reported_seq":653,"reported_epoch":464,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:23:33.398913+0000","last_change":"2024-09-17T07:22:54.392772+0000","last_active":"2024-09-17T07:23:33.398913+0000","last_peered":"2024-09-17T07:23:33.398913+0000","last_clean":"2024-09-17T07:22:52.357362+0000","last_became_active":"2024-09-17T07:22:54.392772+0000","last_became_peered":"2024-09-17T07:22:54.392772+0000","last_unstale":"2024-09-17T07:23:33.398913+0000","last_undegraded":"2024-09-17T07:22:54.386541+0000","last_fullsized":"2024-09-17T07:22:54.386438+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'25","last_scrub_stamp":"2024-09-17T07:20:39.458854+0000","last_deep_scrub":"294'25","last_deep_scrub_stamp":"2024-09-17T07:16:00.471311+0000","last_clean_scrub_stamp":"2024-09-17T07:20:39.458854+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.005100612,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[{"shards":"0,7","objects":5}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"436'10","reported_seq":804,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-17T07:23:30.764674+0000","last_change":"2024-09-17T07:23:07.537471+0000","last_active":"2024-09-17T07:23:30.764674+0000","last_peered":"2024-09-17T07:23:30.764674+0000","last_clean":"2024-09-17T07:23:30.764674+0000","last_became_active":"2024-09-17T07:22:13.574012+0000","last_became_peered":"2024-09-17T07:22:13.574012+0000","last_unstale":"2024-09-17T07:23:30.764674+0000","last_undegraded":"2024-09-17T07:23:30.764674+0000","last_fullsized":"2024-09-17T07:23:30.764674+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-09-17T07:19:30.879061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:19:30.879061+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034025510000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":353,"reported_epoch":463,"state":"active+undersized","last_fresh":"2024-09-17T07:23:30.535309+0000","last_change":"2024-09-17T07:22:54.380422+0000","last_active":"2024-09-17T07:23:30.535309+0000","last_peered":"2024-09-17T07:23:30.535309+0000","last_clean":"2024-09-17T07:22:52.353333+0000","last_became_active":"2024-09-17T07:22:54.380422+0000","last_became_peered":"2024-09-17T07:22:54.380422+0000","last_unstale":"2024-09-17T07:23:30.535309+0000","last_undegraded":"2024-09-17T07:23:30.535309+0000","last_fullsized":"2024-09-17T07:22:54.377696+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:37.885570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:15:37.885570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":352,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-17T07:23:30.536455+0000","last_change":"2024-09-17T07:23:05.476637+0000","last_active":"2024-09-17T07:23:30.536455+0000","last_peered":"2024-09-17T07:23:30.536455+0000","last_clean":"2024-09-17T07:23:30.536455+0000","last_became_active":"2024-09-17T07:09:24.403360+0000","last_became_peered":"2024-09-17T07:09:24.403360+0000","last_unstale":"2024-09-17T07:23:30.536455+0000","last_undegraded":"2024-09-17T07:23:30.536455+0000","last_fullsized":"2024-09-17T07:23:30.536455+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:22.869700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:10:29.325664+0000","last_clean_scrub_stamp":"2024-09-17T07:19:22.869700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0039824769999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"439'27","reported_seq":602,"reported_epoch":463,"state":"active+undersized+degraded","last_fresh":"2024-09-17T07:23:30.536482+0000","last_change":"2024-09-17T07:22:54.391016+0000","last_active":"2024-09-17T07:23:30.536482+0000","last_peered":"2024-09-17T07:23:30.536482+0000","last_clean":"2024-09-17T07:22:52.355429+0000","last_became_active":"2024-09-17T07:22:54.391016+0000","last_became_peered":"2024-09-17T07:22:54.391016+0000","last_unstale":"2024-09-17T07:23:30.536482+0000","last_undegraded":"2024-09-17T07:22:54.378640+0000","last_fullsized":"2024-09-17T07:22:54.378521+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'19","last_scrub_stamp":"2024-09-17T07:19:26.872409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:19:26.872409+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012641085999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[{"shards":"2,6","objects":5}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":528,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-17T07:23:33.398788+0000","last_change":"2024-09-17T07:22:13.590028+0000","last_active":"2024-09-17T07:23:33.398788+0000","last_peered":"2024-09-17T07:23:33.398788+0000","last_clean":"2024-09-17T07:23:33.398788+0000","last_became_active":"2024-09-17T07:22:13.589529+0000","last_became_peered":"2024-09-17T07:22:13.589529+0000","last_unstale":"2024-09-17T07:23:33.398788+0000","last_undegraded":"2024-09-17T07:23:33.398788+0000","last_fullsized":"2024-09-17T07:23:33.398788+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:37.435106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:20:37.435106+0000","last_clean_scrub_stamp":"2024-09-17T07:20:37.435106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0042822729999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":504,"reported_epoch":463,"state":"active+undersized","last_fresh":"2024-09-17T07:23:30.378183+0000","last_change":"2024-09-17T07:22:54.378873+0000","last_active":"2024-09-17T07:23:30.378183+0000","last_peered":"2024-09-17T07:23:30.378183+0000","last_clean":"2024-09-17T07:22:52.353913+0000","last_became_active":"2024-09-17T07:22:54.378873+0000","last_became_peered":"2024-09-17T07:22:54.378873+0000","last_unstale":"2024-09-17T07:23:30.378183+0000","last_undegraded":"2024-09-17T07:23:30.378183+0000","last_fullsized":"2024-09-17T07:22:54.376106+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:26.001442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:20:26.001442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001725695,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":551,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-17T07:23:30.536805+0000","last_change":"2024-09-17T07:22:13.587819+0000","last_active":"2024-09-17T07:23:30.536805+0000","last_peered":"2024-09-17T07:23:30.536805+0000","last_clean":"2024-09-17T07:23:30.536805+0000","last_became_active":"2024-09-17T07:22:13.587273+0000","last_became_peered":"2024-09-17T07:22:13.587273+0000","last_unstale":"2024-09-17T07:23:30.536805+0000","last_undegraded":"2024-09-17T07:23:30.536805+0000","last_fullsized":"2024-09-17T07:23:30.536805+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:18.074791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:11:18.559877+0000","last_clean_scrub_stamp":"2024-09-17T07:20:18.074791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0039012449999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"439'39","reported_seq":424,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-17T07:23:30.535040+0000","last_change":"2024-09-17T07:21:16.372009+0000","last_active":"2024-09-17T07:23:30.535040+0000","last_peered":"2024-09-17T07:23:30.535040+0000","last_clean":"2024-09-17T07:23:30.535040+0000","last_became_active":"2024-09-17T07:21:16.371404+0000","last_became_peered":"2024-09-17T07:21:16.371404+0000","last_unstale":"2024-09-17T07:23:30.535040+0000","last_undegraded":"2024-09-17T07:23:30.535040+0000","last_fullsized":"2024-09-17T07:23:30.535040+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'27","last_scrub_stamp":"2024-09-17T07:21:11.488225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:21:11.488225+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0089238530000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":326,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-17T07:23:30.535624+0000","last_change":"2024-09-17T07:21:15.351806+0000","last_active":"2024-09-17T07:23:30.535624+0000","last_peered":"2024-09-17T07:23:30.535624+0000","last_clean":"2024-09-17T07:23:30.535624+0000","last_became_active":"2024-09-17T07:21:15.348754+0000","last_became_peered":"2024-09-17T07:21:15.348754+0000","last_unstale":"2024-09-17T07:23:30.535624+0000","last_undegraded":"2024-09-17T07:23:30.535624+0000","last_fullsized":"2024-09-17T07:23:30.535624+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:14:00.509283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:25.794418+0000","last_clean_scrub_stamp":"2024-09-17T07:14:00.509283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":363,"reported_epoch":463,"state":"active+undersized","last_fresh":"2024-09-17T07:23:30.536270+0000","last_change":"2024-09-17T07:22:54.393685+0000","last_active":"2024-09-17T07:23:30.536270+0000","last_peered":"2024-09-17T07:23:30.536270+0000","last_clean":"2024-09-17T07:22:52.355356+0000","last_became_active":"2024-09-17T07:22:54.393685+0000","last_became_peered":"2024-09-17T07:22:54.393685+0000","last_unstale":"2024-09-17T07:23:30.536270+0000","last_undegraded":"2024-09-17T07:23:30.536270+0000","last_fullsized":"2024-09-17T07:22:54.380409+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:21:16.225555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:21:16.225555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001743026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":40,"reported_epoch":462,"state":"active+clean","last_fresh":"2024-09-17T07:23:28.792703+0000","last_change":"2024-09-17T07:22:13.624786+0000","last_active":"2024-09-17T07:23:28.792703+0000","last_peered":"2024-09-17T07:23:28.792703+0000","last_clean":"2024-09-17T07:23:28.792703+0000","last_became_active":"2024-09-17T07:22:13.624057+0000","last_became_peered":"2024-09-17T07:22:13.624057+0000","last_unstale":"2024-09-17T07:23:28.792703+0000","last_undegraded":"2024-09-17T07:23:28.792703+0000","last_fullsized":"2024-09-17T07:23:28.792703+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:30.003956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:19:30.003956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"439'29","reported_seq":267,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-17T07:23:30.535418+0000","last_change":"2024-09-17T07:21:15.352075+0000","last_active":"2024-09-17T07:23:30.535418+0000","last_peered":"2024-09-17T07:23:30.535418+0000","last_clean":"2024-09-17T07:23:30.535418+0000","last_became_active":"2024-09-17T07:21:15.351512+0000","last_became_peered":"2024-09-17T07:21:15.351512+0000","last_unstale":"2024-09-17T07:23:30.535418+0000","last_undegraded":"2024-09-17T07:23:30.535418+0000","last_fullsized":"2024-09-17T07:23:30.535418+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'23","last_scrub_stamp":"2024-09-17T07:14:58.689120+0000","last_deep_scrub":"86'17","last_deep_scrub_stamp":"2024-09-17T07:03:18.241067+0000","last_clean_scrub_stamp":"2024-09-17T07:14:58.689120+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":536,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-17T07:23:30.377817+0000","last_change":"2024-09-17T07:22:13.591148+0000","last_active":"2024-09-17T07:23:30.377817+0000","last_peered":"2024-09-17T07:23:30.377817+0000","last_clean":"2024-09-17T07:23:30.377817+0000","last_became_active":"2024-09-17T07:22:13.590622+0000","last_became_peered":"2024-09-17T07:22:13.590622+0000","last_unstale":"2024-09-17T07:23:30.377817+0000","last_undegraded":"2024-09-17T07:23:30.377817+0000","last_fullsized":"2024-09-17T07:23:30.377817+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:21.073159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:20:21.073159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0093893710000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":5925044248,"num_objects":90410,"num_object_clones":0,"num_object_copies":271230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18557,"num_objects_misplaced":7204,"num_objects_unfound":0,"num_objects_dirty":90410,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":254084,"num_write_kb":11023745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85345,"num_bytes_recovered":5593104384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19040645120,"data_stored":19040632904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70014,"ondisk_log_size":70014,"up":44,"acting":43,"num_store_stats":7},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":382,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":80,"acting":80,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":93,"ondisk_log_size":93,"up":81,"acting":81,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":64,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1537,"num_read_kb":1294,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":4956,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":13004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1020,"ondisk_log_size":1020,"up":85,"acting":85,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":144,"num_read_kb":139,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":7,"acting":7,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1386,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":12,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":4578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":83,"acting":83,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":757038,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":155,"num_read_kb":222,"num_write":131,"num_write_kb":1756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1288075,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1430,"ondisk_log_size":1430,"up":27,"acting":27,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698038022,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3133692,"kb_used_data":2079600,"kb_used_omap":1,"kb_used_meta":1054078,"kb_avail":90611460,"statfs":{"total":95995035648,"available":92786135040,"internally_reserved":0,"allocated":2129510400,"data_stored":2128300697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":1079376331},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:22:43 2024","interfaces":[{"interface":"back","average":{"1min":3.649,"5min":2.105,"15min":1.6719999999999999},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":36.659999999999997,"5min":43.905000000000001,"15min":48.493000000000002},"last":0.26800000000000002},{"interface":"front","average":{"1min":3.294,"5min":1.948,"15min":1.7450000000000001},"min":{"1min":0.221,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":36.555,"5min":44.432000000000002,"15min":48.738},"last":0.35499999999999998}]},{"osd":1,"last update":"Tue Sep 17 07:23:16 2024","interfaces":[{"interface":"back","average":{"1min":1.1240000000000001,"5min":1.226,"15min":1.2430000000000001},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":12.112,"5min":12.112,"15min":12.112},"last":0.38600000000000001},{"interface":"front","average":{"1min":1.1339999999999999,"5min":1.202,"15min":1.214},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":12.273,"5min":12.273,"15min":12.273},"last":0.32500000000000001}]},{"osd":2,"last update":"Tue Sep 17 07:22:43 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.96199999999999997,"15min":1.097},"min":{"1min":0.255,"5min":0.23599999999999999,"15min":0.23000000000000001},"max":{"1min":2.3690000000000002,"5min":10.057,"15min":22.376000000000001},"last":0.40300000000000002},{"interface":"front","average":{"1min":1.175,"5min":0.96399999999999997,"15min":1.075},"min":{"1min":0.27600000000000002,"5min":0.24099999999999999,"15min":0.224},"max":{"1min":7.9080000000000004,"5min":9.843,"15min":22.298999999999999},"last":0.29399999999999998}]},{"osd":3,"last update":"Tue Sep 17 07:22:43 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.91900000000000004,"15min":1.1579999999999999},"min":{"1min":0.32900000000000001,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":2.3969999999999998,"5min":10.122999999999999,"15min":22.196000000000002},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.80300000000000005,"15min":1.1379999999999999},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26500000000000001},"max":{"1min":4.1719999999999997,"5min":12.465999999999999,"15min":22.234000000000002},"last":0.33400000000000002}]},{"osd":4,"last update":"Tue Sep 17 07:22:43 2024","interfaces":[{"interface":"back","average":{"1min":1.171,"5min":1.0089999999999999,"15min":1.083},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.222},"max":{"1min":13.316000000000001,"5min":13.316000000000001,"15min":21.140999999999998},"last":0.28299999999999997},{"interface":"front","average":{"1min":1.3160000000000001,"5min":1.214,"15min":1.2450000000000001},"min":{"1min":0.28199999999999997,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":12.272,"5min":12.272,"15min":19.420000000000002},"last":0.34300000000000003}]},{"osd":5,"last update":"Tue Sep 17 07:22:43 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":1.0289999999999999,"15min":1.26},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":5.2779999999999996,"5min":12.429,"15min":29.428999999999998},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.95599999999999996,"15min":1.2949999999999999},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.25700000000000001},"max":{"1min":1.405,"5min":12.606,"15min":30.059999999999999},"last":0.57899999999999996}]},{"osd":6,"last update":"Tue Sep 17 07:22:43 2024","interfaces":[{"interface":"back","average":{"1min":4.0830000000000002,"5min":1.857,"15min":1.5780000000000001},"min":{"1min":0.34000000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":60.445999999999998,"5min":60.445999999999998,"15min":60.445999999999998},"last":0.22800000000000001},{"interface":"front","average":{"1min":4.3520000000000003,"5min":1.9259999999999999,"15min":1.647},"min":{"1min":0.376,"5min":0.29999999999999999,"15min":0.27500000000000002},"max":{"1min":60.332000000000001,"5min":60.332000000000001,"15min":60.332000000000001},"last":0.30399999999999999}]}]},{"osd":6,"up_from":50,"seq":214748365065,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4242024,"kb_used_data":3260120,"kb_used_omap":1,"kb_used_meta":981886,"kb_avail":89503128,"statfs":{"total":95995035648,"available":91651203072,"internally_reserved":0,"allocated":3338362880,"data_stored":3337120365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1005451719},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:23:28 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":2.6989999999999998,"15min":2.0339999999999998},"min":{"1min":0.26100000000000001,"5min":0.19700000000000001,"15min":0.185},"max":{"1min":3.9089999999999998,"5min":91.683999999999997,"15min":126.902},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.96899999999999997,"5min":2.4809999999999999,"15min":1.8740000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":7.4340000000000002,"5min":91.403999999999996,"15min":127.06399999999999},"last":0.46600000000000003}]},{"osd":1,"last update":"Tue Sep 17 07:23:20 2024","interfaces":[{"interface":"back","average":{"1min":10.272,"5min":6.8929999999999998,"15min":6.3300000000000001},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":199.32400000000001,"5min":199.32400000000001,"15min":199.32400000000001},"last":0.36299999999999999},{"interface":"front","average":{"1min":10.311999999999999,"5min":7.194,"15min":6.6740000000000004},"min":{"1min":0.25600000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":198.88499999999999,"5min":198.88499999999999,"15min":198.88499999999999},"last":0.45000000000000001}]},{"osd":2,"last update":"Tue Sep 17 07:23:28 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":1.653,"15min":1.198},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.219},"max":{"1min":2.7879999999999998,"5min":36.289000000000001,"15min":36.289000000000001},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.875,"5min":1.5069999999999999,"15min":1.0620000000000001},"min":{"1min":0.27600000000000002,"5min":0.26500000000000001,"15min":0.22900000000000001},"max":{"1min":4.4219999999999997,"5min":36.326999999999998,"15min":36.326999999999998},"last":0.435}]},{"osd":3,"last update":"Tue Sep 17 07:23:28 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.83299999999999996,"15min":0.90900000000000003},"min":{"1min":0.33000000000000002,"5min":0.249,"15min":0.23200000000000001},"max":{"1min":0.84299999999999997,"5min":6.8639999999999999,"15min":6.8639999999999999},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.94599999999999995,"15min":1.071},"min":{"1min":0.33600000000000002,"5min":0.27600000000000002,"15min":0.27500000000000002},"max":{"1min":2.8170000000000002,"5min":3.8130000000000002,"15min":5.7690000000000001},"last":0.374}]},{"osd":4,"last update":"Tue Sep 17 07:23:28 2024","interfaces":[{"interface":"back","average":{"1min":1.0549999999999999,"5min":0.97299999999999998,"15min":2.3130000000000002},"min":{"1min":0.25800000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":9.5419999999999998,"5min":9.5419999999999998,"15min":260.77199999999999},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.93799999999999994,"15min":2.323},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.19800000000000001},"max":{"1min":0.81999999999999995,"5min":6.8920000000000003,"15min":260.69099999999997},"last":0.33200000000000002}]},{"osd":5,"last update":"Tue Sep 17 07:22:26 2024","interfaces":[{"interface":"back","average":{"1min":16.803999999999998,"5min":5.2350000000000003,"15min":2.7069999999999999},"min":{"1min":0.25600000000000001,"5min":0.253,"15min":0.23400000000000001},"max":{"1min":317.59800000000001,"5min":317.59800000000001,"15min":317.59800000000001},"last":0.439},{"interface":"front","average":{"1min":16.704000000000001,"5min":5.3639999999999999,"15min":2.7240000000000002},"min":{"1min":0.315,"5min":0.315,"15min":0.251},"max":{"1min":317.762,"5min":317.762,"15min":317.762},"last":0.89000000000000001}]},{"osd":7,"last update":"Tue Sep 17 07:22:44 2024","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":1.2769999999999999,"15min":1.6870000000000001},"min":{"1min":0.38500000000000001,"5min":0.33000000000000002,"15min":0.246},"max":{"1min":3.1619999999999999,"5min":6.8179999999999996,"15min":131.95699999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":1.048,"5min":1.0760000000000001,"15min":1.5800000000000001},"min":{"1min":0.377,"5min":0.26900000000000002,"15min":0.252},"max":{"1min":3.7029999999999998,"5min":5.8070000000000004,"15min":131.833},"last":0.40699999999999997}]}]},{"osd":5,"up_from":42,"seq":180388626693,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2847628,"kb_used_data":1859968,"kb_used_omap":1,"kb_used_meta":987646,"kb_avail":90897524,"statfs":{"total":95995035648,"available":93079064576,"internally_reserved":0,"allocated":1904607232,"data_stored":1903499919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1011349958},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:22:04 2024","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.84599999999999997,"15min":1.149},"min":{"1min":0.22900000000000001,"5min":0.184,"15min":0.184},"max":{"1min":3.0750000000000002,"5min":5.7530000000000001,"15min":28.727},"last":7.4829999999999997},{"interface":"front","average":{"1min":0.75,"5min":0.76800000000000002,"15min":1.163},"min":{"1min":0.23699999999999999,"5min":0.217,"15min":0.217},"max":{"1min":1.518,"5min":4.7939999999999996,"15min":27.824999999999999},"last":6.4139999999999997}]},{"osd":1,"last update":"Tue Sep 17 07:22:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2569999999999999,"5min":1.2569999999999999,"15min":1.2569999999999999},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":8.4060000000000006,"5min":8.4060000000000006,"15min":8.4060000000000006},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":2.73,"5min":2.73,"15min":2.73},"last":0.47499999999999998}]},{"osd":2,"last update":"Tue Sep 17 07:22:04 2024","interfaces":[{"interface":"back","average":{"1min":1.0249999999999999,"5min":0.999,"15min":1.1599999999999999},"min":{"1min":0.24199999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":2.984,"5min":4.4160000000000004,"15min":20.486000000000001},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.89900000000000002,"5min":0.93600000000000005,"15min":1.044},"min":{"1min":0.28499999999999998,"5min":0.219,"15min":0.219},"max":{"1min":2.5230000000000001,"5min":7.71,"15min":23.925999999999998},"last":0.505}]},{"osd":3,"last update":"Tue Sep 17 07:22:04 2024","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":1.119,"15min":1.548},"min":{"1min":0.30099999999999999,"5min":0.27500000000000002,"15min":0.247},"max":{"1min":3.0150000000000001,"5min":16.143000000000001,"15min":68.978999999999999},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.77200000000000002,"15min":1.3109999999999999},"min":{"1min":0.29899999999999999,"5min":0.27100000000000002,"15min":0.253},"max":{"1min":2.3769999999999998,"5min":5.7880000000000003,"15min":69.084000000000003},"last":0.56499999999999995}]},{"osd":4,"last update":"Tue Sep 17 07:22:04 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":2.2719999999999998,"15min":2.1669999999999998},"min":{"1min":0.26700000000000002,"5min":0.247,"15min":0.22},"max":{"1min":2.5,"5min":96.668000000000006,"15min":142.327},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.96799999999999997,"5min":2.335,"15min":2.0920000000000001},"min":{"1min":0.27600000000000002,"5min":0.255,"15min":0.20999999999999999},"max":{"1min":4.3419999999999996,"5min":96.801000000000002,"15min":142.13200000000001},"last":0.436}]},{"osd":6,"last update":"Tue Sep 17 07:22:19 2024","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":1.986,"15min":1.3660000000000001},"min":{"1min":0.26700000000000002,"5min":0.245,"15min":0.23499999999999999},"max":{"1min":3.194,"5min":96.686000000000007,"15min":96.686000000000007},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.83299999999999996,"5min":2.3759999999999999,"15min":1.55},"min":{"1min":0.31,"5min":0.29799999999999999,"15min":0.26700000000000002},"max":{"1min":3.5510000000000002,"5min":96.834999999999994,"15min":96.834999999999994},"last":0.374}]},{"osd":7,"last update":"Tue Sep 17 07:22:44 2024","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.246,"15min":1.325},"min":{"1min":0.36499999999999999,"5min":0.307,"15min":0.28299999999999997},"max":{"1min":3.4260000000000002,"5min":4.8419999999999996,"15min":6.8869999999999996},"last":0.55100000000000005},{"interface":"front","average":{"1min":1.52,"5min":1.071,"15min":1.121},"min":{"1min":0.35299999999999998,"5min":0.30199999999999999,"15min":0.27100000000000002},"max":{"1min":6.234,"5min":6.3639999999999999,"15min":6.3639999999999999},"last":0.40200000000000002}]}]},{"osd":4,"up_from":33,"seq":141733921044,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2789636,"kb_used_data":1820024,"kb_used_omap":1,"kb_used_meta":969598,"kb_avail":90955516,"statfs":{"total":95995035648,"available":93138448384,"internally_reserved":0,"allocated":1863704576,"data_stored":1862486155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":992868806},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:23:30 2024","interfaces":[{"interface":"back","average":{"1min":2.3889999999999998,"5min":1.8420000000000001,"15min":1.655},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20000000000000001},"max":{"1min":32.604999999999997,"5min":39.473999999999997,"15min":55.310000000000002},"last":0.36499999999999999},{"interface":"front","average":{"1min":2.7589999999999999,"5min":1.861,"15min":1.7789999999999999},"min":{"1min":0.22,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":32.534999999999997,"5min":39.654000000000003,"15min":53.771000000000001},"last":0.50900000000000001}]},{"osd":1,"last update":"Tue Sep 17 07:23:18 2024","interfaces":[{"interface":"back","average":{"1min":2.9710000000000001,"5min":2.899,"15min":2.887},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":39.439,"5min":39.439,"15min":39.439},"last":0.26500000000000001},{"interface":"front","average":{"1min":3.5830000000000002,"5min":3.04,"15min":2.9500000000000002},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":39.325000000000003,"5min":39.325000000000003,"15min":39.325000000000003},"last":0.379}]},{"osd":2,"last update":"Tue Sep 17 07:23:30 2024","interfaces":[{"interface":"back","average":{"1min":0.97299999999999998,"5min":0.95399999999999996,"15min":1.073},"min":{"1min":0.28199999999999997,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":10.238,"5min":10.238,"15min":10.238},"last":0.46200000000000002},{"interface":"front","average":{"1min":1.1519999999999999,"5min":0.85299999999999998,"15min":1.05},"min":{"1min":0.27800000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":10.19,"5min":10.19,"15min":16.655999999999999},"last":0.435}]},{"osd":3,"last update":"Tue Sep 17 07:23:30 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.86499999999999999,"15min":1.0009999999999999},"min":{"1min":0.31900000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":2.2759999999999998,"5min":15.433999999999999,"15min":17.872},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.74099999999999999,"15min":1.044},"min":{"1min":0.29699999999999999,"5min":0.28100000000000003,"15min":0.253},"max":{"1min":4.7030000000000003,"5min":8.7360000000000007,"15min":17.908000000000001},"last":0.47699999999999998}]},{"osd":5,"last update":"Tue Sep 17 07:22:40 2024","interfaces":[{"interface":"back","average":{"1min":18.748999999999999,"5min":5.7320000000000002,"15min":2.7770000000000001},"min":{"1min":0.28199999999999997,"5min":0.224,"15min":0.222},"max":{"1min":301.13799999999998,"5min":301.13799999999998,"15min":301.13799999999998},"last":2.3450000000000002},{"interface":"front","average":{"1min":19.166,"5min":6.1020000000000003,"15min":2.8860000000000001},"min":{"1min":0.34999999999999998,"5min":0.27400000000000002,"15min":0.26600000000000001},"max":{"1min":301.24000000000001,"5min":301.24000000000001,"15min":301.24000000000001},"last":2.2370000000000001}]},{"osd":6,"last update":"Tue Sep 17 07:23:30 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.91400000000000003,"15min":1.0629999999999999},"min":{"1min":0.249,"5min":0.23400000000000001,"15min":0.218},"max":{"1min":4.7729999999999997,"5min":5.3789999999999996,"15min":8.9619999999999997},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.78700000000000003,"5min":1.008,"15min":1.1739999999999999},"min":{"1min":0.249,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":4.1879999999999997,"5min":6.6219999999999999,"15min":10.051},"last":0.40200000000000002}]},{"osd":7,"last update":"Tue Sep 17 07:22:40 2024","interfaces":[{"interface":"back","average":{"1min":1.77,"5min":1.1080000000000001,"15min":1.363},"min":{"1min":0.32800000000000001,"5min":0.27900000000000003,"15min":0.26700000000000002},"max":{"1min":5.8090000000000002,"5min":5.8090000000000002,"15min":48.719000000000001},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.97099999999999997,"15min":1.3480000000000001},"min":{"1min":0.318,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":3.3340000000000001,"5min":7.2910000000000004,"15min":50.527000000000001},"last":0.41299999999999998}]}]},{"osd":3,"up_from":0,"seq":115964117270,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":18,"seq":77309411612,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3630080,"kb_used_data":2556276,"kb_used_omap":1,"kb_used_meta":1073790,"kb_avail":90115072,"statfs":{"total":95995035648,"available":92277833728,"internally_reserved":0,"allocated":2617626624,"data_stored":2616413467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1099561414},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:23:01 2024","interfaces":[{"interface":"back","average":{"1min":4.9390000000000001,"5min":2.1920000000000002,"15min":2.3180000000000001},"min":{"1min":0.19,"5min":0.154,"15min":0.154},"max":{"1min":52.023000000000003,"5min":52.023000000000003,"15min":131.09800000000001},"last":0.436},{"interface":"front","average":{"1min":5.0149999999999997,"5min":2.3730000000000002,"15min":2.3650000000000002},"min":{"1min":0.17999999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":54.700000000000003,"5min":54.700000000000003,"15min":131.24700000000001},"last":0.995}]},{"osd":1,"last update":"Tue Sep 17 07:23:21 2024","interfaces":[{"interface":"back","average":{"1min":6.5599999999999996,"5min":2.8860000000000001,"15min":2.274},"min":{"1min":0.191,"5min":0.184,"15min":0.184},"max":{"1min":91.686999999999998,"5min":91.686999999999998,"15min":91.686999999999998},"last":0.33400000000000002},{"interface":"front","average":{"1min":6.7999999999999998,"5min":3.21,"15min":2.6120000000000001},"min":{"1min":0.17999999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":93.744,"5min":93.744,"15min":93.744},"last":0.42299999999999999}]},{"osd":3,"last update":"Tue Sep 17 07:23:11 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.48399999999999999,"15min":0.84699999999999998},"min":{"1min":0.23499999999999999,"5min":0.20300000000000001,"15min":0.20100000000000001},"max":{"1min":1.0109999999999999,"5min":3.1880000000000002,"15min":23.43},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.751,"15min":0.85599999999999998},"min":{"1min":0.23100000000000001,"5min":0.22500000000000001,"15min":0.22},"max":{"1min":4.1260000000000003,"5min":5.8840000000000003,"15min":23.408000000000001},"last":0.68799999999999994}]},{"osd":4,"last update":"Tue Sep 17 07:22:38 2024","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":1.8200000000000001,"15min":1.258},"min":{"1min":0.307,"5min":0.247,"15min":0.24399999999999999},"max":{"1min":3.1600000000000001,"5min":75.870999999999995,"15min":75.870999999999995},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.78100000000000003,"5min":1.996,"15min":1.3899999999999999},"min":{"1min":0.31,"5min":0.26900000000000002,"15min":0.26300000000000001},"max":{"1min":1.8759999999999999,"5min":75.888999999999996,"15min":75.888999999999996},"last":0.66800000000000004}]},{"osd":5,"last update":"Tue Sep 17 07:21:56 2024","interfaces":[{"interface":"back","average":{"1min":1.0369999999999999,"5min":1.151,"15min":1.2150000000000001},"min":{"1min":0.312,"5min":0.27900000000000003,"15min":0.26100000000000001},"max":{"1min":3.8500000000000001,"5min":7.2229999999999999,"15min":15.276999999999999},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.95799999999999996,"5min":0.84199999999999997,"15min":1.014},"min":{"1min":0.29599999999999999,"5min":0.27700000000000002,"15min":0.27600000000000002},"max":{"1min":2.6549999999999998,"5min":5.8239999999999998,"15min":15.561999999999999},"last":0.35799999999999998}]},{"osd":6,"last update":"Tue Sep 17 07:23:11 2024","interfaces":[{"interface":"back","average":{"1min":1.5569999999999999,"5min":2.819,"15min":2.0960000000000001},"min":{"1min":0.29699999999999999,"5min":0.27700000000000002,"15min":0.27500000000000002},"max":{"1min":13.614000000000001,"5min":76.081999999999994,"15min":101.587},"last":0.73799999999999999},{"interface":"front","average":{"1min":1.6559999999999999,"5min":3.0619999999999998,"15min":2.1920000000000002},"min":{"1min":0.309,"5min":0.309,"15min":0.30599999999999999},"max":{"1min":13.914,"5min":76.244,"15min":100.696},"last":0.64500000000000002}]},{"osd":7,"last update":"Tue Sep 17 07:23:01 2024","interfaces":[{"interface":"back","average":{"1min":1.05,"5min":0.91000000000000003,"15min":1.3740000000000001},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":4.2720000000000002,"5min":4.7759999999999998,"15min":94.269999999999996},"last":0.63500000000000001},{"interface":"front","average":{"1min":1.1379999999999999,"5min":0.92800000000000005,"15min":1.399},"min":{"1min":0.34200000000000003,"5min":0.309,"15min":0.309},"max":{"1min":4.0510000000000002,"5min":4.8090000000000002,"15min":94.634},"last":0.71099999999999997}]}]},{"osd":1,"up_from":413,"seq":1773821493277,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3976596,"kb_used_data":3506892,"kb_used_omap":4,"kb_used_meta":469691,"kb_avail":89768556,"statfs":{"total":95995035648,"available":91923001344,"internally_reserved":0,"allocated":3591057408,"data_stored":3589820279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4439,"internal_metadata":480964265},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:23:20 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":1.4910000000000001,"15min":1.611},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":3.6520000000000001,"5min":7.359,"15min":7.359},"last":0.22600000000000001},{"interface":"front","average":{"1min":0.91300000000000003,"5min":1.294,"15min":1.3580000000000001},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":3.6920000000000002,"5min":5.7599999999999998,"15min":5.7599999999999998},"last":0.32500000000000001}]},{"osd":2,"last update":"Tue Sep 17 07:23:20 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":1.214,"15min":1.3120000000000001},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":1.746,"5min":5.7220000000000004,"15min":5.7220000000000004},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.78800000000000003,"5min":1.0609999999999999,"15min":1.107},"min":{"1min":0.28799999999999998,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":3.919,"5min":3.919,"15min":3.919},"last":0.27700000000000002}]},{"osd":3,"last update":"Tue Sep 17 07:23:20 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.78100000000000003,"15min":0.80800000000000005},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":2.169,"5min":5.9290000000000003,"15min":5.9290000000000003},"last":0.308},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.83299999999999996,"15min":0.86399999999999999},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":1.7729999999999999,"5min":4.1340000000000003,"15min":4.1340000000000003},"last":0.441}]},{"osd":4,"last update":"Tue Sep 17 07:23:20 2024","interfaces":[{"interface":"back","average":{"1min":1.129,"5min":6.1109999999999998,"15min":6.9409999999999998},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":4.9569999999999999,"5min":116.88200000000001,"15min":116.88200000000001},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.69999999999999996,"5min":6.0810000000000004,"15min":6.9779999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":2.202,"5min":116.98399999999999,"15min":116.98399999999999},"last":0.34000000000000002}]},{"osd":5,"last update":"Tue Sep 17 07:22:18 2024","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":3.1080000000000001,"5min":3.1080000000000001,"15min":3.1080000000000001},"last":0.35299999999999998},{"interface":"front","average":{"1min":1.079,"5min":1.079,"15min":1.079},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":5.9080000000000004,"5min":5.9080000000000004,"15min":5.9080000000000004},"last":0.40600000000000003}]},{"osd":6,"last update":"Tue Sep 17 07:23:20 2024","interfaces":[{"interface":"back","average":{"1min":5.2220000000000004,"5min":2.1890000000000001,"15min":1.6830000000000001},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":56.686999999999998,"5min":56.686999999999998,"15min":56.686999999999998},"last":0.40799999999999997},{"interface":"front","average":{"1min":5.0880000000000001,"5min":2.0800000000000001,"15min":1.5780000000000001},"min":{"1min":0.373,"5min":0.372,"15min":0.372},"max":{"1min":56.420999999999999,"5min":56.420999999999999,"15min":56.420999999999999},"last":0.48699999999999999}]},{"osd":7,"last update":"Tue Sep 17 07:23:20 2024","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":1.0249999999999999,"15min":1.0529999999999999},"min":{"1min":0.34699999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":2.6499999999999999,"5min":4.2389999999999999,"15min":4.2389999999999999},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.87,"5min":1.016,"15min":1.04},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":4.2839999999999998,"5min":4.2839999999999998,"15min":4.2839999999999998},"last":0.38700000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738660,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4841060,"kb_used_data":3565972,"kb_used_omap":1,"kb_used_meta":1275006,"kb_avail":88904092,"statfs":{"total":95995035648,"available":91037790208,"internally_reserved":0,"allocated":3651555328,"data_stored":3650323234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1305606598},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 17 07:23:15 2024","interfaces":[{"interface":"back","average":{"1min":1.6879999999999999,"5min":1.944,"15min":1.986},"min":{"1min":0.17199999999999999,"5min":0.156,"15min":0.156},"max":{"1min":9.2959999999999994,"5min":9.2959999999999994,"15min":9.2959999999999994},"last":0.47499999999999998},{"interface":"front","average":{"1min":1.544,"5min":1.5149999999999999,"15min":1.51},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":5.0819999999999999,"5min":5.0819999999999999,"15min":5.0819999999999999},"last":0.40799999999999997}]},{"osd":2,"last update":"Tue Sep 17 07:22:53 2024","interfaces":[{"interface":"back","average":{"1min":1.2210000000000001,"5min":1.6870000000000001,"15min":1.085},"min":{"1min":0.25800000000000001,"5min":0.156,"15min":0.155},"max":{"1min":3.6520000000000001,"5min":62.094999999999999,"15min":62.094999999999999},"last":0.45100000000000001},{"interface":"front","average":{"1min":1.605,"5min":1.7210000000000001,"15min":1.079},"min":{"1min":0.20899999999999999,"5min":0.183,"15min":0.183},"max":{"1min":4.7350000000000003,"5min":59.848999999999997,"15min":59.848999999999997},"last":0.34799999999999998}]},{"osd":3,"last update":"Tue Sep 17 07:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.72599999999999998,"15min":0.72099999999999997},"min":{"1min":0.23200000000000001,"5min":0.223,"15min":0.21199999999999999},"max":{"1min":1.339,"5min":4.6829999999999998,"15min":5.5880000000000001},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.79000000000000004,"15min":0.68799999999999994},"min":{"1min":0.223,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.577,"5min":5.1150000000000002,"15min":5.1150000000000002},"last":0.56399999999999995}]},{"osd":4,"last update":"Tue Sep 17 07:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":1.1930000000000001,"15min":1.0840000000000001},"min":{"1min":0.32500000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":1.9419999999999999,"5min":8.3279999999999994,"15min":19.146999999999998},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.79700000000000004,"5min":1.073,"15min":1.101},"min":{"1min":0.30399999999999999,"5min":0.26600000000000001,"15min":0.26200000000000001},"max":{"1min":3.6909999999999998,"5min":8.3119999999999994,"15min":19.158999999999999},"last":0.53600000000000003}]},{"osd":5,"last update":"Tue Sep 17 07:22:04 2024","interfaces":[{"interface":"back","average":{"1min":4.4119999999999999,"5min":1.6519999999999999,"15min":2.3140000000000001},"min":{"1min":0.371,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":42.741,"5min":42.741,"15min":216.83699999999999},"last":0.52800000000000002},{"interface":"front","average":{"1min":3.6230000000000002,"5min":1.647,"15min":2.3029999999999999},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.27000000000000002},"max":{"1min":42.792000000000002,"5min":42.792000000000002,"15min":216.816},"last":0.377}]},{"osd":6,"last update":"Tue Sep 17 07:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":2.2189999999999999,"15min":1.863},"min":{"1min":0.28999999999999998,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":1.857,"5min":67.873000000000005,"15min":147.13200000000001},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.80000000000000004,"5min":2.371,"15min":1.869},"min":{"1min":0.34100000000000003,"5min":0.311,"15min":0.30199999999999999},"max":{"1min":3.9790000000000001,"5min":67.914000000000001,"15min":147.172},"last":0.59099999999999997}]},{"osd":7,"last update":"Tue Sep 17 07:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":1.022,"15min":1.159},"min":{"1min":0.33200000000000002,"5min":0.318,"15min":0.29599999999999999},"max":{"1min":3.843,"5min":6.3300000000000001,"15min":6.3300000000000001},"last":0.495},{"interface":"front","average":{"1min":0.55100000000000005,"5min":1.2589999999999999,"15min":1.218},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":1.3200000000000001,"5min":7.2830000000000004,"15min":7.2830000000000004},"last":0.42999999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3615424512,"data_stored":3615424512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3588161536,"data_stored":3588161536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2614497280,"data_stored":2614493208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1860833280,"data_stored":1860829208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1900544000,"data_stored":1900544000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3334868992,"data_stored":3334864920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2126315520,"data_stored":2126315520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T07:23:40.209 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:23:40.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:40.250 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:40.251 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:23:40.351 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:23:40.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:40.394 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:40.394 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:23:40.495 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:23:40.525 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:40.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:40.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:23:40.627 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:23:40.658 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:40.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:40.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:23:40.760 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:23:40.788 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:40.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:40.789 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:23:40.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:40 smithi100 ceph-mon[29092]: pgmap v1017: 158 pgs: 2 active+undersized+degraded+remapped+backfilling, 41 active+undersized, 25 active+undersized+degraded, 90 active+clean; 5.2 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 785 op/s; 23116/258258 objects degraded (8.951%); 7204/258258 objects misplaced (2.789%); 27 MiB/s, 431 objects/s recovering 2024-09-17T07:23:40.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:40 smithi100 ceph-mon[29092]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:23:40.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:40 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T07:23:40.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:40 smithi100 ceph-mon[29092]: osdmap e467: 8 total, 7 up, 7 in 2024-09-17T07:23:40.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:40 smithi100 ceph-mon[29092]: osdmap e468: 8 total, 7 up, 7 in 2024-09-17T07:23:40.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:40 smithi100 ceph-mon[39462]: pgmap v1017: 158 pgs: 2 active+undersized+degraded+remapped+backfilling, 41 active+undersized, 25 active+undersized+degraded, 90 active+clean; 5.2 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 785 op/s; 23116/258258 objects degraded (8.951%); 7204/258258 objects misplaced (2.789%); 27 MiB/s, 431 objects/s recovering 2024-09-17T07:23:40.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:40 smithi100 ceph-mon[39462]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:23:40.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:40 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T07:23:40.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:40 smithi100 ceph-mon[39462]: osdmap e467: 8 total, 7 up, 7 in 2024-09-17T07:23:40.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:40 smithi100 ceph-mon[39462]: osdmap e468: 8 total, 7 up, 7 in 2024-09-17T07:23:40.890 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:23:40.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:40.931 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:40.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:23:41.032 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:23:41.034 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-17T07:23:41.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:40 smithi135 ceph-mon[33238]: pgmap v1017: 158 pgs: 2 active+undersized+degraded+remapped+backfilling, 41 active+undersized, 25 active+undersized+degraded, 90 active+clean; 5.2 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 785 op/s; 23116/258258 objects degraded (8.951%); 7204/258258 objects misplaced (2.789%); 27 MiB/s, 431 objects/s recovering 2024-09-17T07:23:41.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:40 smithi135 ceph-mon[33238]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:23:41.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:40 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T07:23:41.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:40 smithi135 ceph-mon[33238]: osdmap e467: 8 total, 7 up, 7 in 2024-09-17T07:23:41.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:40 smithi135 ceph-mon[33238]: osdmap e468: 8 total, 7 up, 7 in 2024-09-17T07:23:41.073 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:23:41.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:41.134 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:41.134 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:23:41.235 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:23:41.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:41.274 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:41.274 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:23:41.375 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:23:41.418 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:41.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:41.419 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:23:41.519 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:23:41.551 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:41.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:41.552 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:23:41.653 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:23:41.689 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:41.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:41.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:23:41.791 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:23:41.812 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:41.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:41.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:23:41.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:41 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:23:41.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:41 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:23:41.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:41 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:23:41.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:41 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:23:41.914 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:23:41.948 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:41.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:41.950 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:23:42.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:41 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:23:42.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:41 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:23:42.050 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:23:42.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:42.084 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:42.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:23:42.185 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:23:42.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:42.225 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:42.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:23:42.326 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:23:42.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:42.367 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:42.367 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:23:42.468 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:23:42.502 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:42.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:42.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:23:42.604 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:23:42.636 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:42.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:42.637 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:23:42.738 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:23:42.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:42.773 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:42.774 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:23:42.805 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:23:42.875 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:23:42.882 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:42 smithi100 ceph-mon[29092]: pgmap v1020: 158 pgs: 2 active+undersized+degraded+remapped+backfilling, 41 active+undersized, 25 active+undersized+degraded, 90 active+clean; 5.1 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 836 B/s rd, 0 B/s wr, 1.02k op/s; 21506/253017 objects degraded (8.500%); 6248/253017 objects misplaced (2.469%); 11 MiB/s, 179 objects/s recovering 2024-09-17T07:23:42.882 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:42 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 21506/253017 objects degraded (8.500%), 27 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-17T07:23:42.882 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:42 smithi100 ceph-mon[29092]: osdmap e469: 8 total, 7 up, 7 in 2024-09-17T07:23:42.883 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:42 smithi100 ceph-mon[39462]: pgmap v1020: 158 pgs: 2 active+undersized+degraded+remapped+backfilling, 41 active+undersized, 25 active+undersized+degraded, 90 active+clean; 5.1 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 836 B/s rd, 0 B/s wr, 1.02k op/s; 21506/253017 objects degraded (8.500%); 6248/253017 objects misplaced (2.469%); 11 MiB/s, 179 objects/s recovering 2024-09-17T07:23:42.883 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:42 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 21506/253017 objects degraded (8.500%), 27 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-17T07:23:42.883 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:42 smithi100 ceph-mon[39462]: osdmap e469: 8 total, 7 up, 7 in 2024-09-17T07:23:42.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:42.918 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:42.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:23:43.019 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:23:43.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:42 smithi135 ceph-mon[33238]: pgmap v1020: 158 pgs: 2 active+undersized+degraded+remapped+backfilling, 41 active+undersized, 25 active+undersized+degraded, 90 active+clean; 5.1 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 836 B/s rd, 0 B/s wr, 1.02k op/s; 21506/253017 objects degraded (8.500%); 6248/253017 objects misplaced (2.469%); 11 MiB/s, 179 objects/s recovering 2024-09-17T07:23:43.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:42 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 21506/253017 objects degraded (8.500%), 27 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-17T07:23:43.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:42 smithi135 ceph-mon[33238]: osdmap e469: 8 total, 7 up, 7 in 2024-09-17T07:23:43.051 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:43.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:43.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:23:43.154 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:23:43.186 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:43.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:43.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:23:43.288 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:23:43.324 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:43.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:43.325 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:23:43.426 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:23:43.463 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:43.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:43.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:23:43.566 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:23:43.613 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:43.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:43.615 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:23:43.637 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T07:23:43.716 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:23:43.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:43.749 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:43.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:23:43.851 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:23:43.886 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:43.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:43.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:23:43.988 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:23:44.022 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:44.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:44.024 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:23:44.124 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:23:44.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:44.167 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:44.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:23:44.268 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:23:44.306 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:44.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:44.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:23:44.408 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:23:44.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:44.445 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:44.445 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:23:44.546 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:23:44.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:44.583 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:44.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:23:44.684 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:23:44.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:44.725 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:44.725 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:23:44.826 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:23:44.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:44.853 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:44.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:23:44.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:44 smithi100 ceph-mon[29092]: pgmap v1022: 158 pgs: 2 active+undersized+degraded+remapped+backfilling, 41 active+undersized, 25 active+undersized+degraded, 90 active+clean; 5.0 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.34k op/s; 21506/247776 objects degraded (8.680%); 6248/247776 objects misplaced (2.522%); 11 MiB/s, 183 objects/s recovering 2024-09-17T07:23:44.876 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:44 smithi100 ceph-mon[39462]: pgmap v1022: 158 pgs: 2 active+undersized+degraded+remapped+backfilling, 41 active+undersized, 25 active+undersized+degraded, 90 active+clean; 5.0 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.34k op/s; 21506/247776 objects degraded (8.680%); 6248/247776 objects misplaced (2.522%); 11 MiB/s, 183 objects/s recovering 2024-09-17T07:23:44.954 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:23:44.984 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:44.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:44.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:23:45.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:44 smithi135 ceph-mon[33238]: pgmap v1022: 158 pgs: 2 active+undersized+degraded+remapped+backfilling, 41 active+undersized, 25 active+undersized+degraded, 90 active+clean; 5.0 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.34k op/s; 21506/247776 objects degraded (8.680%); 6248/247776 objects misplaced (2.522%); 11 MiB/s, 183 objects/s recovering 2024-09-17T07:23:45.086 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:23:45.121 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:45.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:45.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:23:45.143 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7] out_osds: [3] dead_osds: [5] live_osds: [0, 2, 3, 4, 6, 7, 1] 2024-09-17T07:23:45.143 INFO:tasks.thrashosds.thrasher:Reviving osd 5 2024-09-17T07:23:45.143 INFO:tasks.cephadm.osd.5:Restarting osd.5 (starting--it wasn't running)... 2024-09-17T07:23:45.143 DEBUG:teuthology.orchestra.run.smithi135:osd.5> sudo journalctl -f -n 0 -u ceph-e848267c-74c1-11ef-bceb-c7b262605968@osd.5.service 2024-09-17T07:23:45.146 DEBUG:teuthology.orchestra.run.smithi135:> sudo systemctl start ceph-e848267c-74c1-11ef-bceb-c7b262605968@osd.5 2024-09-17T07:23:45.223 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:23:45.256 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:45.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:45.257 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:23:45.358 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:23:45.395 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:45.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:45.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:23:45.497 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:23:45.530 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:45.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:45.531 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:23:45.543 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:23:45 smithi135 systemd[1]: Starting Ceph osd.5 for e848267c-74c1-11ef-bceb-c7b262605968... 2024-09-17T07:23:45.632 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:23:45.666 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:45.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:45.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:23:45.767 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:23:45.802 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:45.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:45.803 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:23:45.904 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:23:45.931 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:45.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:45.932 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:23:46.033 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:23:46.070 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:46.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:46.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:23:46.171 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:23:46.198 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:46.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:46.200 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:23:46.293 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:23:45 smithi135 podman[156172]: 2024-09-17 07:23:45.903523862 +0000 UTC m=+0.027558381 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9 2024-09-17T07:23:46.300 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:23:46.325 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:46.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:46.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:23:46.357 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:23:45 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:23:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:23:46.428 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:23:46.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:46.456 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:46.456 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:23:46.558 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:23:46.595 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:46.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:46.597 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:23:46.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:46 smithi100 ceph-mon[29092]: osdmap e470: 8 total, 7 up, 7 in 2024-09-17T07:23:46.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:46 smithi100 ceph-mon[39462]: osdmap e470: 8 total, 7 up, 7 in 2024-09-17T07:23:46.697 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:23:46.733 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:46.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:46.734 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:23:46.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:46 smithi135 ceph-mon[33238]: osdmap e470: 8 total, 7 up, 7 in 2024-09-17T07:23:46.835 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:23:46.865 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:46.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:46.866 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:23:46.967 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:23:47.003 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:47.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:47.004 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:23:47.105 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:23:47.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:47.157 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:47.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:23:47.258 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:23:47.279 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:47.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:47.280 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:23:47.293 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:23:46 smithi135 podman[156172]: 2024-09-17 07:23:46.990794369 +0000 UTC m=+1.114828866 container create 234c6f0424027dc7fa0cf29afd06c648e91c75023612b233be0cf016bd4006da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-5-activate, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, RELEASE=reef-19e2a2b, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2) 2024-09-17T07:23:47.381 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:23:47.409 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:47.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:47.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:23:47.511 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:23:47.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:47.549 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:47.549 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:23:47.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:47 smithi100 ceph-mon[29092]: pgmap v1024: 158 pgs: 2 active+undersized+degraded+remapped+backfilling, 41 active+undersized, 25 active+undersized+degraded, 90 active+clean; 4.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 864 B/s rd, 0 B/s wr, 1.48k op/s; 18609/232029 objects degraded (8.020%); 5319/232029 objects misplaced (2.292%); 23 MiB/s, 374 objects/s recovering 2024-09-17T07:23:47.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:47 smithi100 ceph-mon[39462]: pgmap v1024: 158 pgs: 2 active+undersized+degraded+remapped+backfilling, 41 active+undersized, 25 active+undersized+degraded, 90 active+clean; 4.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 864 B/s rd, 0 B/s wr, 1.48k op/s; 18609/232029 objects degraded (8.020%); 5319/232029 objects misplaced (2.292%); 23 MiB/s, 374 objects/s recovering 2024-09-17T07:23:47.650 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:23:47.682 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:47.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:47.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:23:47.785 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:23:47.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:47 smithi135 ceph-mon[33238]: pgmap v1024: 158 pgs: 2 active+undersized+degraded+remapped+backfilling, 41 active+undersized, 25 active+undersized+degraded, 90 active+clean; 4.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 864 B/s rd, 0 B/s wr, 1.48k op/s; 18609/232029 objects degraded (8.020%); 5319/232029 objects misplaced (2.292%); 23 MiB/s, 374 objects/s recovering 2024-09-17T07:23:47.814 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:47.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:47.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:23:47.915 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:23:47.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:47.937 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:47.937 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:23:48.038 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:23:48.074 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:48.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:48.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:23:48.175 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:23:48.207 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:48.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:48.208 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:23:48.236 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:23:48 smithi135 podman[156172]: 2024-09-17 07:23:48.127881132 +0000 UTC m=+2.251915647 container init 234c6f0424027dc7fa0cf29afd06c648e91c75023612b233be0cf016bd4006da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-5-activate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=reef-19e2a2b, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-09-17T07:23:48.236 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:23:48 smithi135 podman[156172]: 2024-09-17 07:23:48.132994663 +0000 UTC m=+2.257029162 container start 234c6f0424027dc7fa0cf29afd06c648e91c75023612b233be0cf016bd4006da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-5-activate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-19e2a2b, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-09-17T07:23:48.237 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:23:48 smithi135 podman[156172]: 2024-09-17 07:23:48.236125383 +0000 UTC m=+2.360159890 container attach 234c6f0424027dc7fa0cf29afd06c648e91c75023612b233be0cf016bd4006da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-5-activate, GIT_BRANCH=HEAD, RELEASE=reef-19e2a2b, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T07:23:48.309 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:23:48.346 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:48.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:48.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:23:48.448 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:23:48.487 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:48.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:48.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:23:48.536 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:23:48.589 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:23:48.629 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:48.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:48.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:23:48.731 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:23:48.772 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:48.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:48.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:23:48.876 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:23:48.904 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:48.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:48.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:23:49.006 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:23:49.035 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:49.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:49.036 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:23:49.043 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:23:48 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-5-activate[156322]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-17T07:23:49.043 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:23:48 smithi135 bash[156172]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-17T07:23:49.043 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:23:48 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-5-activate[156322]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-09-17T07:23:49.044 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:23:48 smithi135 bash[156172]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-09-17T07:23:49.044 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:23:48 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-5-activate[156322]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-09-17T07:23:49.044 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:23:48 smithi135 bash[156172]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-09-17T07:23:49.044 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:23:48 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-5-activate[156322]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-09-17T07:23:49.044 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:23:48 smithi135 bash[156172]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-09-17T07:23:49.044 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:23:48 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-5-activate[156322]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-09-17T07:23:49.044 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:23:48 smithi135 bash[156172]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-09-17T07:23:49.044 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:23:48 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-5-activate[156322]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-17T07:23:49.044 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:23:48 smithi135 bash[156172]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-17T07:23:49.044 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:23:48 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-5-activate[156322]: --> ceph-volume raw activate successful for osd ID: 5 2024-09-17T07:23:49.044 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:23:48 smithi135 bash[156172]: --> ceph-volume raw activate successful for osd ID: 5 2024-09-17T07:23:49.044 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:23:48 smithi135 podman[156172]: 2024-09-17 07:23:48.723264836 +0000 UTC m=+2.847299344 container died 234c6f0424027dc7fa0cf29afd06c648e91c75023612b233be0cf016bd4006da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-5-activate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, RELEASE=reef-19e2a2b, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-09-17T07:23:49.137 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:23:49.164 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:49.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:49.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:23:49.266 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:23:49.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:49.307 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:49.307 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:23:49.408 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:23:49.437 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:49.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:49.438 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:23:49.539 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:23:49.578 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:49.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:49.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:23:49.680 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:23:49.683 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:23:49 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:23:49.683 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:49 smithi135 ceph-mon[33238]: pgmap v1025: 158 pgs: 2 active+undersized+degraded+remapped+backfilling, 41 active+undersized, 25 active+undersized+degraded, 90 active+clean; 4.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 718 B/s rd, 0 B/s wr, 982 op/s; 18609/232029 objects degraded (8.020%); 5319/232029 objects misplaced (2.292%); 9.8 MiB/s, 157 objects/s recovering 2024-09-17T07:23:49.683 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:49 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:23:49.684 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:49 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:23:49.684 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:49 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:23:49.684 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:49 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:23:49.715 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:49.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:49.716 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:23:49.812 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:49 smithi100 ceph-mon[29092]: pgmap v1025: 158 pgs: 2 active+undersized+degraded+remapped+backfilling, 41 active+undersized, 25 active+undersized+degraded, 90 active+clean; 4.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 718 B/s rd, 0 B/s wr, 982 op/s; 18609/232029 objects degraded (8.020%); 5319/232029 objects misplaced (2.292%); 9.8 MiB/s, 157 objects/s recovering 2024-09-17T07:23:49.812 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:49 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:23:49.812 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:49 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:23:49.812 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:49 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:23:49.813 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:49 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:23:49.813 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:49 smithi100 ceph-mon[39462]: pgmap v1025: 158 pgs: 2 active+undersized+degraded+remapped+backfilling, 41 active+undersized, 25 active+undersized+degraded, 90 active+clean; 4.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 718 B/s rd, 0 B/s wr, 982 op/s; 18609/232029 objects degraded (8.020%); 5319/232029 objects misplaced (2.292%); 9.8 MiB/s, 157 objects/s recovering 2024-09-17T07:23:49.813 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:49 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:23:49.813 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:49 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:23:49.813 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:49 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:23:49.813 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:49 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:23:49.817 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:23:49.853 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:49.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:49.854 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:23:49.955 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:23:49.960 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:23:49 smithi135 podman[156487]: 2024-09-17 07:23:49.683572773 +0000 UTC m=+0.952444040 container remove 234c6f0424027dc7fa0cf29afd06c648e91c75023612b233be0cf016bd4006da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-5-activate, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=reef-19e2a2b, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-09-17T07:23:49.995 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:49.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:49.996 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:23:50.097 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:23:50.126 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:50.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:50.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:23:50.154 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:23:50.154 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:23:50.188 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1: {} 2024-09-17T07:23:50.188 INFO:teuthology.orchestra.run.smithi100.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:23:50.222 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:23:50.222 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:23:50.228 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:23:50.251 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:23:50.251 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:23:50.264 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:23:49 smithi135 podman[156619]: 2024-09-17 07:23:49.862317331 +0000 UTC m=+0.031264281 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9 2024-09-17T07:23:50.264 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:23:49 smithi135 podman[156619]: 2024-09-17 07:23:49.993075673 +0000 UTC m=+0.162022605 container create 5f30512b77c0b4a39ad76d1f20992868181d0a846398a492eeed858079942c35 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-5, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, RELEASE=reef-19e2a2b, io.buildah.version=1.37.2, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-09-17T07:23:50.266 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:50.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:50.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:23:50.277 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:23:50.277 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:23:50.278 INFO:teuthology.orchestra.run.smithi100.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-17T07:23:50.310 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:23:50.310 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:23:50.343 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:23:50.343 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:23:50.367 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:23:50.412 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:50.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:50.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:23:50.436 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-17T07:23:50.456 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:50 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:23:50.456 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:50 smithi100 ceph-mon[29092]: osdmap e471: 8 total, 7 up, 7 in 2024-09-17T07:23:50.456 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:50 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/4047939362' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:23:50.456 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:50 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 18609/232029 objects degraded (8.020%), 27 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-17T07:23:50.456 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:50 smithi100 ceph-mon[29092]: osdmap e472: 8 total, 7 up, 7 in 2024-09-17T07:23:50.456 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:50 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:23:50.457 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:50 smithi100 ceph-mon[39462]: osdmap e471: 8 total, 7 up, 7 in 2024-09-17T07:23:50.457 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:50 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/4047939362' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:23:50.457 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:50 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 18609/232029 objects degraded (8.020%), 27 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-17T07:23:50.457 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:50 smithi100 ceph-mon[39462]: osdmap e472: 8 total, 7 up, 7 in 2024-09-17T07:23:50.514 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:23:50.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:50 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:23:50.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:50 smithi135 ceph-mon[33238]: osdmap e471: 8 total, 7 up, 7 in 2024-09-17T07:23:50.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:50 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/4047939362' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:23:50.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:50 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 18609/232029 objects degraded (8.020%), 27 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-17T07:23:50.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:50 smithi135 ceph-mon[33238]: osdmap e472: 8 total, 7 up, 7 in 2024-09-17T07:23:50.544 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:23:50 smithi135 podman[156619]: 2024-09-17 07:23:50.263315976 +0000 UTC m=+0.432262906 container init 5f30512b77c0b4a39ad76d1f20992868181d0a846398a492eeed858079942c35 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, RELEASE=reef-19e2a2b, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T07:23:50.544 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:23:50 smithi135 podman[156619]: 2024-09-17 07:23:50.270647825 +0000 UTC m=+0.439594754 container start 5f30512b77c0b4a39ad76d1f20992868181d0a846398a492eeed858079942c35 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-5, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, ceph=True, RELEASE=reef-19e2a2b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-09-17T07:23:50.544 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:23:50 smithi135 bash[156619]: 5f30512b77c0b4a39ad76d1f20992868181d0a846398a492eeed858079942c35 2024-09-17T07:23:50.544 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:23:50 smithi135 systemd[1]: Started Ceph osd.5 for e848267c-74c1-11ef-bceb-c7b262605968. 2024-09-17T07:23:50.548 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:50.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:50.550 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:23:50.651 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:23:50.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:50.681 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:50.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:23:50.689 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:23:50.782 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:23:50.815 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:50.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:50.816 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:23:50.917 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:23:50.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:50.954 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:50.954 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:23:51.055 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:23:51.076 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:51.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:51.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:23:51.178 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:23:51.207 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:51.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:51.208 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:23:51.293 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:23:50 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-5[156646]: 2024-09-17T07:23:50.933+0000 7f8f759ef640 -1 Falling back to public interface 2024-09-17T07:23:51.308 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:23:51.332 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:51.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:51.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:23:51.434 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:23:51.461 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:51.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:51.462 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:23:51.563 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:23:51.570 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:51 smithi100 ceph-mon[29092]: pgmap v1026: 158 pgs: 2 active+undersized+degraded+remapped+backfilling, 41 active+undersized, 25 active+undersized+degraded, 90 active+clean; 4.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.46k op/s; 17432/217932 objects degraded (7.999%); 5319/217932 objects misplaced (2.441%); 8.8 MiB/s, 140 objects/s recovering 2024-09-17T07:23:51.570 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:51 smithi100 ceph-mon[39462]: pgmap v1026: 158 pgs: 2 active+undersized+degraded+remapped+backfilling, 41 active+undersized, 25 active+undersized+degraded, 90 active+clean; 4.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.46k op/s; 17432/217932 objects degraded (7.999%); 5319/217932 objects misplaced (2.441%); 8.8 MiB/s, 140 objects/s recovering 2024-09-17T07:23:51.591 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:51.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:51.592 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:23:51.693 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:23:51.727 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:51.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:51.728 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:23:51.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:51 smithi135 ceph-mon[33238]: pgmap v1026: 158 pgs: 2 active+undersized+degraded+remapped+backfilling, 41 active+undersized, 25 active+undersized+degraded, 90 active+clean; 4.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.46k op/s; 17432/217932 objects degraded (7.999%); 5319/217932 objects misplaced (2.441%); 8.8 MiB/s, 140 objects/s recovering 2024-09-17T07:23:51.828 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:23:51.869 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:51.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:51.870 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:23:51.971 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:23:51.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:51.993 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:51.994 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:23:52.094 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:23:52.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:52.141 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:52.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:23:52.224 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-17T07:23:52.224 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:23:52.242 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:23:52.271 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:52.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:52.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:23:52.373 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:23:52.410 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:52.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:52.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:23:52.512 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:23:52.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:52.533 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:52.534 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:23:52.636 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:23:52.672 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:52.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:52.674 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:23:52.774 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:23:52.813 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:52.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:52.814 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:23:52.915 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:23:52.953 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:52.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:52.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:23:53.055 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:23:53.092 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:53.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:53.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:23:53.193 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:23:53.231 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:53.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:53.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:23:53.333 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:23:53.375 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:53.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:53.376 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:23:53.377 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-17T07:23:53.477 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:23:53.514 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:53.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:53.515 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:23:53.616 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:23:53.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:53 smithi100 ceph-mon[29092]: pgmap v1029: 158 pgs: 2 active+undersized+degraded+remapped+backfilling, 41 active+undersized, 25 active+undersized+degraded, 90 active+clean; 4.3 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 836 B/s rd, 0 B/s wr, 1.02k op/s; 16071/213306 objects degraded (7.534%); 4505/213306 objects misplaced (2.112%); 11 MiB/s, 177 objects/s recovering 2024-09-17T07:23:53.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:53 smithi100 ceph-mon[39462]: pgmap v1029: 158 pgs: 2 active+undersized+degraded+remapped+backfilling, 41 active+undersized, 25 active+undersized+degraded, 90 active+clean; 4.3 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 836 B/s rd, 0 B/s wr, 1.02k op/s; 16071/213306 objects degraded (7.534%); 4505/213306 objects misplaced (2.112%); 11 MiB/s, 177 objects/s recovering 2024-09-17T07:23:53.638 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:53.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:53.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:23:53.699 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:53 smithi135 ceph-mon[33238]: pgmap v1029: 158 pgs: 2 active+undersized+degraded+remapped+backfilling, 41 active+undersized, 25 active+undersized+degraded, 90 active+clean; 4.3 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 836 B/s rd, 0 B/s wr, 1.02k op/s; 16071/213306 objects degraded (7.534%); 4505/213306 objects misplaced (2.112%); 11 MiB/s, 177 objects/s recovering 2024-09-17T07:23:53.741 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:23:53.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:53.764 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:53.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:23:53.865 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:23:53.888 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:53.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:53.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:23:53.990 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:23:54.019 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:54.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:54.020 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:23:54.121 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:23:54.144 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:54.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:54.145 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:23:54.246 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:23:54.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:54.273 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:54.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:23:54.360 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T07:23:54.360 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T07:23:54.374 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:23:54.397 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:54.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:54.398 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:23:54.499 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:23:54.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:54.530 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:54.530 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:23:54.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:54 smithi100 ceph-mon[29092]: osdmap e473: 8 total, 7 up, 7 in 2024-09-17T07:23:54.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:54 smithi100 ceph-mon[39462]: osdmap e473: 8 total, 7 up, 7 in 2024-09-17T07:23:54.631 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:23:54.655 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:54.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:54.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:23:54.757 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:23:54.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:54 smithi135 ceph-mon[33238]: osdmap e473: 8 total, 7 up, 7 in 2024-09-17T07:23:54.793 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:23:54 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-5[156646]: 2024-09-17T07:23:54.532+0000 7f8f759ef640 -1 osd.5 446 log_to_monitors true 2024-09-17T07:23:54.794 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:54.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:54.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:23:54.896 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:23:54.930 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:54.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:54.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:23:54.947 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:23:54.966 INFO:teuthology.orchestra.run.smithi135.stderr:no valid command found; 10 closest matches: 2024-09-17T07:23:54.966 INFO:teuthology.orchestra.run.smithi135.stderr:0 2024-09-17T07:23:54.966 INFO:teuthology.orchestra.run.smithi135.stderr:1 2024-09-17T07:23:54.966 INFO:teuthology.orchestra.run.smithi135.stderr:2 2024-09-17T07:23:54.967 INFO:teuthology.orchestra.run.smithi135.stderr:abort 2024-09-17T07:23:54.967 INFO:teuthology.orchestra.run.smithi135.stderr:assert 2024-09-17T07:23:54.967 INFO:teuthology.orchestra.run.smithi135.stderr:bluefs debug_inject_read_zeros 2024-09-17T07:23:54.967 INFO:teuthology.orchestra.run.smithi135.stderr:bluefs files list 2024-09-17T07:23:54.967 INFO:teuthology.orchestra.run.smithi135.stderr:bluefs stats 2024-09-17T07:23:54.967 INFO:teuthology.orchestra.run.smithi135.stderr:bluestore allocator dump block 2024-09-17T07:23:54.967 INFO:teuthology.orchestra.run.smithi135.stderr:bluestore allocator fragmentation block 2024-09-17T07:23:54.967 INFO:teuthology.orchestra.run.smithi135.stderr:admin_socket: invalid command 2024-09-17T07:23:55.032 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:23:55.073 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:55.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:55.074 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:23:55.175 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:23:55.212 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:55.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:55.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:23:55.314 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:23:55.345 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:55.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:55.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:23:55.446 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:23:55.460 INFO:teuthology.orchestra.run.smithi100.stderr:noscrub is unset 2024-09-17T07:23:55.484 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:55.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:55.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:23:55.586 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:23:55.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:55.617 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:55.617 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:23:55.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:55 smithi100 ceph-mon[29092]: pgmap v1031: 158 pgs: 2 active+undersized+degraded+remapped+backfilling, 41 active+undersized, 25 active+undersized+degraded, 90 active+clean; 4.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.29k op/s; 16071/208782 objects degraded (7.698%); 4505/208782 objects misplaced (2.158%); 11 MiB/s, 180 objects/s recovering 2024-09-17T07:23:55.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:55 smithi100 ceph-mon[29092]: osdmap e474: 8 total, 7 up, 7 in 2024-09-17T07:23:55.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:55 smithi100 ceph-mon[29092]: from='osd.5 v2:172.21.15.135:6804/1324312753' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-17T07:23:55.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:55 smithi100 ceph-mon[29092]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-17T07:23:55.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:55 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2641739966' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:23:55.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:55 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:23:55.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:55 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 16071/208782 objects degraded (7.698%), 27 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-17T07:23:55.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:55 smithi100 ceph-mon[39462]: pgmap v1031: 158 pgs: 2 active+undersized+degraded+remapped+backfilling, 41 active+undersized, 25 active+undersized+degraded, 90 active+clean; 4.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.29k op/s; 16071/208782 objects degraded (7.698%); 4505/208782 objects misplaced (2.158%); 11 MiB/s, 180 objects/s recovering 2024-09-17T07:23:55.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:55 smithi100 ceph-mon[39462]: osdmap e474: 8 total, 7 up, 7 in 2024-09-17T07:23:55.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:55 smithi100 ceph-mon[39462]: from='osd.5 v2:172.21.15.135:6804/1324312753' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-17T07:23:55.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:55 smithi100 ceph-mon[39462]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-17T07:23:55.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:55 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2641739966' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:23:55.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:55 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:23:55.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:55 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 16071/208782 objects degraded (7.698%), 27 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-17T07:23:55.643 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:55 smithi135 ceph-mon[33238]: pgmap v1031: 158 pgs: 2 active+undersized+degraded+remapped+backfilling, 41 active+undersized, 25 active+undersized+degraded, 90 active+clean; 4.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.29k op/s; 16071/208782 objects degraded (7.698%); 4505/208782 objects misplaced (2.158%); 11 MiB/s, 180 objects/s recovering 2024-09-17T07:23:55.643 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:55 smithi135 ceph-mon[33238]: osdmap e474: 8 total, 7 up, 7 in 2024-09-17T07:23:55.643 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:55 smithi135 ceph-mon[33238]: from='osd.5 v2:172.21.15.135:6804/1324312753' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-17T07:23:55.644 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:55 smithi135 ceph-mon[33238]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-17T07:23:55.644 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:55 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2641739966' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:23:55.644 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:55 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:23:55.644 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:55 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 16071/208782 objects degraded (7.698%), 27 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-17T07:23:55.644 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:23:55 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-5[156646]: 2024-09-17T07:23:55.499+0000 7f8f6d799640 -1 osd.5 446 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T07:23:55.684 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-17T07:23:55.684 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-5, ['dump_ops_in_flight'] 2024-09-17T07:23:55.718 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:23:55.753 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:55.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:55.754 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:23:55.855 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:23:55.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:55.892 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:55.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:23:55.993 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:23:56.030 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:56.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:56.031 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:23:56.073 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:23:55 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:23:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:23:56.131 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:23:56.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:56.169 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:56.169 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:23:56.270 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:23:56.305 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:56.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:56.306 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:23:56.361 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:23:56.407 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:23:56.436 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:56.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:56.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:23:56.538 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:23:56.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:56 smithi135 ceph-mon[33238]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:23:56.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:56 smithi135 ceph-mon[33238]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-17T07:23:56.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:56 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T07:23:56.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:56 smithi135 ceph-mon[33238]: osdmap e475: 8 total, 7 up, 7 in 2024-09-17T07:23:56.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:56 smithi135 ceph-mon[33238]: from='osd.5 v2:172.21.15.135:6804/1324312753' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-09-17T07:23:56.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:56 smithi135 ceph-mon[33238]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-09-17T07:23:56.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:56 smithi135 ceph-mon[33238]: 6.e scrub starts 2024-09-17T07:23:56.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:56 smithi135 ceph-mon[33238]: 6.e scrub ok 2024-09-17T07:23:56.560 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:56.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:56.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:23:56.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:56 smithi100 ceph-mon[29092]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:23:56.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:56 smithi100 ceph-mon[29092]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-17T07:23:56.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:56 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T07:23:56.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:56 smithi100 ceph-mon[29092]: osdmap e475: 8 total, 7 up, 7 in 2024-09-17T07:23:56.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:56 smithi100 ceph-mon[29092]: from='osd.5 v2:172.21.15.135:6804/1324312753' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-09-17T07:23:56.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:56 smithi100 ceph-mon[29092]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-09-17T07:23:56.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:56 smithi100 ceph-mon[29092]: 6.e scrub starts 2024-09-17T07:23:56.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:56 smithi100 ceph-mon[29092]: 6.e scrub ok 2024-09-17T07:23:56.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:56 smithi100 ceph-mon[39462]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:23:56.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:56 smithi100 ceph-mon[39462]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-17T07:23:56.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:56 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T07:23:56.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:56 smithi100 ceph-mon[39462]: osdmap e475: 8 total, 7 up, 7 in 2024-09-17T07:23:56.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:56 smithi100 ceph-mon[39462]: from='osd.5 v2:172.21.15.135:6804/1324312753' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-09-17T07:23:56.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:56 smithi100 ceph-mon[39462]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-09-17T07:23:56.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:56 smithi100 ceph-mon[39462]: 6.e scrub starts 2024-09-17T07:23:56.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:56 smithi100 ceph-mon[39462]: 6.e scrub ok 2024-09-17T07:23:56.663 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:23:56.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:56.685 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:56.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:23:56.786 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:23:56.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:56.809 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:56.809 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:23:56.910 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:23:56.933 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:56.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:56.935 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:23:57.036 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:23:57.058 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:57.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:57.059 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:23:57.159 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:23:57.181 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:57.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:57.182 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:23:57.283 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:23:57.305 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:57.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:57.306 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:23:57.407 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:23:57.429 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:57.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:57.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:23:57.531 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:23:57.552 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:57.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:57.553 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:23:57.654 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:23:57.675 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:57.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:57.676 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:23:57.777 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:23:57.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:57 smithi135 ceph-mon[33238]: pgmap v1034: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 peering, 41 active+undersized, 25 active+undersized+degraded, 90 active+clean; 4.0 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 998 B/s rd, 0 B/s wr, 1.06k op/s; 13489/196998 objects degraded (6.847%); 3995/196998 objects misplaced (2.028%); 17 MiB/s, 272 objects/s recovering 2024-09-17T07:23:57.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:57 smithi135 ceph-mon[33238]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T07:23:57.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:57 smithi135 ceph-mon[33238]: osd.5 v2:172.21.15.135:6804/1324312753 boot 2024-09-17T07:23:57.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:57 smithi135 ceph-mon[33238]: osdmap e476: 8 total, 8 up, 7 in 2024-09-17T07:23:57.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:57 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T07:23:57.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:57 smithi135 ceph-mon[33238]: 2.1d scrub starts 2024-09-17T07:23:57.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:57 smithi135 ceph-mon[33238]: 2.1d scrub ok 2024-09-17T07:23:57.799 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:57.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:57.800 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:23:57.848 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-17T07:23:57.848 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:23:57.849 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:57 smithi100 ceph-mon[39462]: pgmap v1034: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 peering, 41 active+undersized, 25 active+undersized+degraded, 90 active+clean; 4.0 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 998 B/s rd, 0 B/s wr, 1.06k op/s; 13489/196998 objects degraded (6.847%); 3995/196998 objects misplaced (2.028%); 17 MiB/s, 272 objects/s recovering 2024-09-17T07:23:57.849 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:57 smithi100 ceph-mon[39462]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T07:23:57.849 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:57 smithi100 ceph-mon[39462]: osd.5 v2:172.21.15.135:6804/1324312753 boot 2024-09-17T07:23:57.849 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:57 smithi100 ceph-mon[39462]: osdmap e476: 8 total, 8 up, 7 in 2024-09-17T07:23:57.849 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:57 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T07:23:57.849 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:57 smithi100 ceph-mon[39462]: 2.1d scrub starts 2024-09-17T07:23:57.849 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:57 smithi100 ceph-mon[39462]: 2.1d scrub ok 2024-09-17T07:23:57.849 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:57 smithi100 ceph-mon[29092]: pgmap v1034: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 peering, 41 active+undersized, 25 active+undersized+degraded, 90 active+clean; 4.0 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 998 B/s rd, 0 B/s wr, 1.06k op/s; 13489/196998 objects degraded (6.847%); 3995/196998 objects misplaced (2.028%); 17 MiB/s, 272 objects/s recovering 2024-09-17T07:23:57.849 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:57 smithi100 ceph-mon[29092]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T07:23:57.849 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:57 smithi100 ceph-mon[29092]: osd.5 v2:172.21.15.135:6804/1324312753 boot 2024-09-17T07:23:57.850 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:57 smithi100 ceph-mon[29092]: osdmap e476: 8 total, 8 up, 7 in 2024-09-17T07:23:57.850 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:57 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T07:23:57.850 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:57 smithi100 ceph-mon[29092]: 2.1d scrub starts 2024-09-17T07:23:57.850 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:57 smithi100 ceph-mon[29092]: 2.1d scrub ok 2024-09-17T07:23:57.901 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:23:57.922 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:57.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:57.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:23:58.024 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:23:58.046 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:58.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:58.046 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:23:58.120 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:23:58.147 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:23:58.168 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:58.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:58.170 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:23:58.270 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:23:58.292 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:58.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:58.293 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:23:58.393 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:23:58.404 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-17T07:23:58.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:58.416 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:58.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:23:58.517 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:23:58.540 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:58.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:58.542 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:23:58.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:58 smithi100 ceph-mon[29092]: 2.10 scrub starts 2024-09-17T07:23:58.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:58 smithi100 ceph-mon[29092]: 2.10 scrub ok 2024-09-17T07:23:58.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:58 smithi100 ceph-mon[29092]: osdmap e477: 8 total, 8 up, 7 in 2024-09-17T07:23:58.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:58 smithi100 ceph-mon[29092]: pgmap v1037: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 peering, 41 active+undersized, 25 active+undersized+degraded, 90 active+clean; 4.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 982 op/s; 13489/196998 objects degraded (6.847%); 3995/196998 objects misplaced (2.028%); 22 MiB/s, 349 objects/s recovering 2024-09-17T07:23:58.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:58 smithi100 ceph-mon[39462]: 2.10 scrub starts 2024-09-17T07:23:58.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:58 smithi100 ceph-mon[39462]: 2.10 scrub ok 2024-09-17T07:23:58.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:58 smithi100 ceph-mon[39462]: osdmap e477: 8 total, 8 up, 7 in 2024-09-17T07:23:58.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:58 smithi100 ceph-mon[39462]: pgmap v1037: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 peering, 41 active+undersized, 25 active+undersized+degraded, 90 active+clean; 4.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 982 op/s; 13489/196998 objects degraded (6.847%); 3995/196998 objects misplaced (2.028%); 22 MiB/s, 349 objects/s recovering 2024-09-17T07:23:58.642 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:23:58.670 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:58.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:58.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:23:58.772 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:23:58.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:58 smithi135 ceph-mon[33238]: 2.10 scrub starts 2024-09-17T07:23:58.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:58 smithi135 ceph-mon[33238]: 2.10 scrub ok 2024-09-17T07:23:58.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:58 smithi135 ceph-mon[33238]: osdmap e477: 8 total, 8 up, 7 in 2024-09-17T07:23:58.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:58 smithi135 ceph-mon[33238]: pgmap v1037: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 peering, 41 active+undersized, 25 active+undersized+degraded, 90 active+clean; 4.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 982 op/s; 13489/196998 objects degraded (6.847%); 3995/196998 objects misplaced (2.028%); 22 MiB/s, 349 objects/s recovering 2024-09-17T07:23:58.809 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:58.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:58.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:23:58.911 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:23:58.942 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:58.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:58.943 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:23:59.043 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:23:59.078 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:59.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:59.079 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:23:59.096 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-17T07:23:59.179 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:23:59.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:59.217 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:59.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:23:59.318 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:23:59.351 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:59.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:59.353 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:23:59.453 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:23:59.486 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:59.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:59.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:23:59.588 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:23:59.621 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:59.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:59.622 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:23:59.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:59 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:23:59.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:59 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:23:59.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:59 smithi100 ceph-mon[29092]: osdmap e478: 8 total, 8 up, 7 in 2024-09-17T07:23:59.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:23:59 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:23:59.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:59 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:23:59.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:59 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:23:59.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:59 smithi100 ceph-mon[39462]: osdmap e478: 8 total, 8 up, 7 in 2024-09-17T07:23:59.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:23:59 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:23:59.722 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:23:59.757 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:23:59.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:59.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:23:59.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:59 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:23:59.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:59 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:23:59.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:59 smithi135 ceph-mon[33238]: osdmap e478: 8 total, 8 up, 7 in 2024-09-17T07:23:59.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:23:59 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:23:59.793 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:23:59 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:23:59.858 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:23:59.895 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:23:59.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:23:59.897 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:23:59.997 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:24:00.035 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:00.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:00.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:24:00.137 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:24:00.174 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:00.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:00.176 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:24:00.276 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:24:00.314 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:00.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:00.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:24:00.416 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:24:00.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:00.460 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:00.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:24:00.561 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:24:00.593 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:00.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:00.595 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:24:00.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:00 smithi100 ceph-mon[29092]: pgmap v1039: 158 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 19 active+undersized, 13 active+undersized+degraded, 122 active+clean; 3.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 638 op/s; 4694/189288 objects degraded (2.480%); 3995/189288 objects misplaced (2.111%); 0 B/s, 6 objects/s recovering 2024-09-17T07:24:00.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:00 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 13489/196998 objects degraded (6.847%), 26 pgs degraded, 65 pgs undersized (PG_DEGRADED) 2024-09-17T07:24:00.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:00 smithi100 ceph-mon[39462]: pgmap v1039: 158 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 19 active+undersized, 13 active+undersized+degraded, 122 active+clean; 3.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 638 op/s; 4694/189288 objects degraded (2.480%); 3995/189288 objects misplaced (2.111%); 0 B/s, 6 objects/s recovering 2024-09-17T07:24:00.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:00 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 13489/196998 objects degraded (6.847%), 26 pgs degraded, 65 pgs undersized (PG_DEGRADED) 2024-09-17T07:24:00.685 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-17T07:24:00.695 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:24:00.728 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:00 smithi135 ceph-mon[33238]: pgmap v1039: 158 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 19 active+undersized, 13 active+undersized+degraded, 122 active+clean; 3.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 638 op/s; 4694/189288 objects degraded (2.480%); 3995/189288 objects misplaced (2.111%); 0 B/s, 6 objects/s recovering 2024-09-17T07:24:00.728 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:00 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 13489/196998 objects degraded (6.847%), 26 pgs degraded, 65 pgs undersized (PG_DEGRADED) 2024-09-17T07:24:00.783 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:00.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:00.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:24:00.884 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:24:00.929 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:00.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:00.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:24:01.031 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:24:01.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:01.066 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:01.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:24:01.101 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:24:01.149 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:24:01.149 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:24:01.168 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:24:01.208 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:24:01.209 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:24:01.235 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:01.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:01.236 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:24:01.251 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1: {} 2024-09-17T07:24:01.251 INFO:teuthology.orchestra.run.smithi100.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:24:01.285 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:24:01.285 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:24:01.313 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:24:01.314 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:24:01.337 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:24:01.346 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:24:01.347 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:24:01.375 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:24:01.375 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:24:01.376 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:01.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:01.377 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:24:01.405 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:24:01.405 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:24:01.446 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:24:01.446 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:24:01.477 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:24:01.518 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:01.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:01.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:24:01.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:01 smithi135 ceph-mon[33238]: 5.6 scrub starts 2024-09-17T07:24:01.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:01 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/3130206422' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:24:01.567 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:01 smithi100 ceph-mon[29092]: 5.6 scrub starts 2024-09-17T07:24:01.567 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:01 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/3130206422' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:24:01.567 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:01 smithi100 ceph-mon[39462]: 5.6 scrub starts 2024-09-17T07:24:01.567 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:01 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/3130206422' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:24:01.621 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:24:01.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:01.658 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:01.658 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:24:01.758 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:24:01.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:01.794 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:01.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:24:01.895 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:24:01.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:01.929 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:01.929 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:24:02.030 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:24:02.068 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:02.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:02.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:24:02.170 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:24:02.200 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:02.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:02.201 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:24:02.302 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:24:02.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:02.341 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:02.342 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:24:02.442 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:24:02.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:02.481 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:02.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:24:02.582 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:24:02.613 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:02.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:02.614 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:24:02.715 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:24:02.746 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:02.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:02.747 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:24:02.848 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:24:02.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:02 smithi100 ceph-mon[29092]: pgmap v1040: 158 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 154 active+clean; 3.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 5.5 KiB/s rd, 37 KiB/s wr, 558 op/s; 3507/187050 objects degraded (1.875%); 1.9 MiB/s, 92 objects/s recovering 2024-09-17T07:24:02.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:02 smithi100 ceph-mon[39462]: pgmap v1040: 158 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 154 active+clean; 3.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 5.5 KiB/s rd, 37 KiB/s wr, 558 op/s; 3507/187050 objects degraded (1.875%); 1.9 MiB/s, 92 objects/s recovering 2024-09-17T07:24:02.891 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:02.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:02.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:24:02.993 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:24:03.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:03.022 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:03.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:24:03.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:02 smithi135 ceph-mon[33238]: pgmap v1040: 158 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 154 active+clean; 3.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 5.5 KiB/s rd, 37 KiB/s wr, 558 op/s; 3507/187050 objects degraded (1.875%); 1.9 MiB/s, 92 objects/s recovering 2024-09-17T07:24:03.123 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:24:03.157 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:03.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:03.158 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:24:03.258 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:24:03.289 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:03.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:03.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:24:03.391 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:24:03.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:03.416 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:03.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:24:03.518 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:24:03.540 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:03.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:03.541 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:24:03.642 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:24:03.674 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:03.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:03.676 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:24:03.776 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:24:03.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:03.813 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:03.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:24:03.914 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:24:03.959 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:03.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:03.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:24:04.061 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:24:04.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:04.107 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:04.107 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:24:04.208 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:24:04.237 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:04.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:04.238 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:24:04.338 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:24:04.365 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:04.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:04.366 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:24:04.406 INFO:teuthology.orchestra.run.smithi135.stdout:{ 2024-09-17T07:24:04.406 INFO:teuthology.orchestra.run.smithi135.stdout: "ops": [ 2024-09-17T07:24:04.406 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:24:04.406 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_repop(client.25811.0:288445 11.8 e478/477 11:18913db5:::benchmark_data_smithi100_180298_object116196:head v 478'18172, mlcod=466'16341)", 2024-09-17T07:24:04.406 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:24:04.398038+0000", 2024-09-17T07:24:04.407 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 0.0064128550000000003, 2024-09-17T07:24:04.407 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0064448930000000001, 2024-09-17T07:24:04.407 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:24:04.407 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "started", 2024-09-17T07:24:04.407 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:24:04.407 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:24:04.407 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:24:04.407 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:04.398038+0000", 2024-09-17T07:24:04.407 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:24:04.407 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:24:04.407 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:24:04.407 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:24:04.407 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:04.398038+0000", 2024-09-17T07:24:04.408 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:24:04.408 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:24:04.408 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:24:04.408 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:24:04.408 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:04.398037+0000", 2024-09-17T07:24:04.408 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999995 2024-09-17T07:24:04.408 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:24:04.408 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:24:04.408 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:24:04.408 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:04.398043+0000", 2024-09-17T07:24:04.408 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.2249999999999999e-06 2024-09-17T07:24:04.408 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:24:04.408 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:24:04.409 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:24:04.409 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:04.398048+0000", 2024-09-17T07:24:04.409 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.7169999999999996e-06 2024-09-17T07:24:04.409 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:24:04.409 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:24:04.409 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:24:04.409 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:04.398053+0000", 2024-09-17T07:24:04.409 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.0340000000000004e-06 2024-09-17T07:24:04.409 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:24:04.409 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:24:04.409 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:24:04.409 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:04.399355+0000", 2024-09-17T07:24:04.410 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0013014260000000001 2024-09-17T07:24:04.410 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:24:04.410 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:24:04.410 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:24:04.410 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:04.399429+0000", 2024-09-17T07:24:04.410 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.4612999999999999e-05 2024-09-17T07:24:04.410 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:24:04.410 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:24:04.410 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:24:04.410 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:24:04.410 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:24:04.411 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_repop(client.25811.0:288447 11.8 e478/477)", 2024-09-17T07:24:04.411 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:24:04.398720+0000", 2024-09-17T07:24:04.411 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 0.0057309470000000001, 2024-09-17T07:24:04.411 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0058204600000000004, 2024-09-17T07:24:04.411 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:24:04.411 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "queued for pg", 2024-09-17T07:24:04.411 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:24:04.411 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:24:04.411 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:24:04.411 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:04.398720+0000", 2024-09-17T07:24:04.411 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:24:04.411 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:24:04.412 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:24:04.412 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:24:04.412 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:04.398720+0000", 2024-09-17T07:24:04.412 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:24:04.412 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:24:04.412 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:24:04.412 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:24:04.412 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:04.398719+0000", 2024-09-17T07:24:04.412 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999995 2024-09-17T07:24:04.412 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:24:04.412 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:24:04.413 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:24:04.413 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:04.398725+0000", 2024-09-17T07:24:04.413 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.8510000000000004e-06 2024-09-17T07:24:04.413 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:24:04.413 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:24:04.413 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:24:04.413 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:04.398734+0000", 2024-09-17T07:24:04.413 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.5699999999999999e-06 2024-09-17T07:24:04.413 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:24:04.414 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:24:04.414 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:24:04.414 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:04.398740+0000", 2024-09-17T07:24:04.414 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.1500000000000004e-06 2024-09-17T07:24:04.414 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:24:04.414 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:24:04.414 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:24:04.414 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:24:04.414 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:24:04.415 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_repop(client.25811.0:288449 11.a e478/477 11:53eba07e:::benchmark_data_smithi100_180298_object116200:head v 478'17866, mlcod=446'15681)", 2024-09-17T07:24:04.415 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:24:04.402036+0000", 2024-09-17T07:24:04.415 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 0.002414252, 2024-09-17T07:24:04.415 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00254923, 2024-09-17T07:24:04.415 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:24:04.415 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "started", 2024-09-17T07:24:04.415 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:24:04.415 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:24:04.415 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:24:04.415 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:04.402036+0000", 2024-09-17T07:24:04.415 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:24:04.416 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:24:04.416 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:24:04.416 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:24:04.416 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:04.402036+0000", 2024-09-17T07:24:04.416 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:24:04.416 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:24:04.416 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:24:04.416 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:24:04.416 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:04.402035+0000", 2024-09-17T07:24:04.416 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.999999 2024-09-17T07:24:04.416 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:24:04.416 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:24:04.416 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:24:04.417 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:04.402043+0000", 2024-09-17T07:24:04.417 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.1599999999999998e-06 2024-09-17T07:24:04.417 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:24:04.417 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:24:04.417 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:24:04.417 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:04.402689+0000", 2024-09-17T07:24:04.417 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00064599500000000001 2024-09-17T07:24:04.417 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:24:04.417 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:24:04.417 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:24:04.417 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:04.402700+0000", 2024-09-17T07:24:04.417 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0423e-05 2024-09-17T07:24:04.418 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:24:04.418 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:24:04.418 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:24:04.418 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:04.402811+0000", 2024-09-17T07:24:04.418 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000111309 2024-09-17T07:24:04.418 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:24:04.418 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:24:04.418 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:24:04.418 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:04.402838+0000", 2024-09-17T07:24:04.419 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.7166999999999999e-05 2024-09-17T07:24:04.419 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:24:04.419 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:24:04.419 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:24:04.419 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:24:04.419 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:24:04.419 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_repop(client.25811.0:288452 11.8 e478/477)", 2024-09-17T07:24:04.419 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:24:04.402715+0000", 2024-09-17T07:24:04.419 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 0.0017358160000000001, 2024-09-17T07:24:04.419 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.001921213, 2024-09-17T07:24:04.419 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:24:04.419 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "queued for pg", 2024-09-17T07:24:04.419 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:24:04.420 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:24:04.420 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:24:04.420 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:04.402715+0000", 2024-09-17T07:24:04.420 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:24:04.420 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:24:04.420 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:24:04.420 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:24:04.420 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:04.402715+0000", 2024-09-17T07:24:04.420 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:24:04.420 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:24:04.420 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:24:04.420 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:24:04.421 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:04.402714+0000", 2024-09-17T07:24:04.421 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999995 2024-09-17T07:24:04.421 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:24:04.421 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:24:04.421 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:24:04.421 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:04.402720+0000", 2024-09-17T07:24:04.421 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.8919999999999997e-06 2024-09-17T07:24:04.421 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:24:04.421 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:24:04.421 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:24:04.422 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:04.402728+0000", 2024-09-17T07:24:04.422 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.6590000000000007e-06 2024-09-17T07:24:04.422 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:24:04.422 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:24:04.422 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:24:04.422 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:04.402736+0000", 2024-09-17T07:24:04.422 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.2700000000000004e-06 2024-09-17T07:24:04.422 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:24:04.422 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:24:04.422 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:24:04.422 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:24:04.422 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:24:04.423 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_repop(client.25811.0:288455 11.7 e478/477)", 2024-09-17T07:24:04.423 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:24:04.404542+0000", 2024-09-17T07:24:04.423 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 4294967295.9999084, 2024-09-17T07:24:04.423 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000134256, 2024-09-17T07:24:04.423 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:24:04.423 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "queued for pg", 2024-09-17T07:24:04.423 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:24:04.423 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:24:04.423 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:24:04.423 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:04.404542+0000", 2024-09-17T07:24:04.423 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:24:04.423 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:24:04.423 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:24:04.424 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:24:04.424 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:04.404542+0000", 2024-09-17T07:24:04.424 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:24:04.424 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:24:04.424 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:24:04.424 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:24:04.424 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:04.404541+0000", 2024-09-17T07:24:04.425 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.999999 2024-09-17T07:24:04.425 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:24:04.425 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:24:04.425 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:24:04.425 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:04.404549+0000", 2024-09-17T07:24:04.425 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.6179999999999997e-06 2024-09-17T07:24:04.425 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:24:04.425 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:24:04.425 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:24:04.425 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:04.404562+0000", 2024-09-17T07:24:04.425 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.3132000000000001e-05 2024-09-17T07:24:04.425 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:24:04.425 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:24:04.426 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:24:04.426 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:04.404569+0000", 2024-09-17T07:24:04.426 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.4209999999999998e-06 2024-09-17T07:24:04.426 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:24:04.426 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:24:04.426 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:24:04.426 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:24:04.426 INFO:teuthology.orchestra.run.smithi135.stdout: ], 2024-09-17T07:24:04.426 INFO:teuthology.orchestra.run.smithi135.stdout: "num_ops": 5 2024-09-17T07:24:04.426 INFO:teuthology.orchestra.run.smithi135.stdout:} 2024-09-17T07:24:04.467 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:24:04.507 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:04.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:04.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:24:04.609 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:24:04.644 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:04.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:04.645 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:24:04.745 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:24:04.786 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:04.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:04.788 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:24:04.889 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:24:04.918 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:04.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:04.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:24:04.935 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:04 smithi135 ceph-mon[33238]: pgmap v1041: 158 pgs: 2 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 153 active+clean; 3.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 4.7 KiB/s rd, 31 KiB/s wr, 572 op/s; 3507/185052 objects degraded (1.895%); 1.7 MiB/s, 79 objects/s recovering 2024-09-17T07:24:04.935 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:04 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:24:04.936 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:04 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:24:05.022 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:24:05.058 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:05.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:05.059 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:24:05.126 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:04 smithi100 ceph-mon[29092]: pgmap v1041: 158 pgs: 2 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 153 active+clean; 3.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 4.7 KiB/s rd, 31 KiB/s wr, 572 op/s; 3507/185052 objects degraded (1.895%); 1.7 MiB/s, 79 objects/s recovering 2024-09-17T07:24:05.126 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:04 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:24:05.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:04 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:24:05.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:04 smithi100 ceph-mon[39462]: pgmap v1041: 158 pgs: 2 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 153 active+clean; 3.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 4.7 KiB/s rd, 31 KiB/s wr, 572 op/s; 3507/185052 objects degraded (1.895%); 1.7 MiB/s, 79 objects/s recovering 2024-09-17T07:24:05.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:04 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:24:05.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:04 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:24:05.160 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:24:05.194 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:05.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:05.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:24:05.296 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:24:05.327 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:05.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:05.328 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:24:05.429 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:24:05.458 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:05.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:05.459 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:24:05.560 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:24:05.589 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:05.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:05.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:24:05.691 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:24:05.723 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:05.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:05.724 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:24:05.825 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:24:05.865 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:05.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:05.866 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:24:05.950 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:24:05.968 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:24:05.973 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:05 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 3507/185052 objects degraded (1.895%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T07:24:05.974 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:05 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 3507/185052 objects degraded (1.895%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T07:24:05.974 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:24:05 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:24:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:24:05.992 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:05.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:05.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:24:06.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:05 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 3507/185052 objects degraded (1.895%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T07:24:06.094 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:24:06.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:06.141 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:06.141 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:24:06.242 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:24:06.280 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:06.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:06.282 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:24:06.382 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:24:06.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:06.415 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:06.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:24:06.516 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:24:06.547 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:06.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:06.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:24:06.649 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:24:06.682 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:06.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:06.683 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:24:06.784 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:24:06.821 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:06.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:06.822 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:24:06.923 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:24:06.946 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:06.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:06.947 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:24:07.048 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:24:07.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:07.078 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:07.078 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:24:07.179 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:24:07.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:07.210 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:07.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:24:07.311 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:24:07.339 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:07.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:07.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:24:07.441 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:24:07.482 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:07.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:07.483 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:24:07.488 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd reweight 0 1 2024-09-17T07:24:07.584 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:24:07.626 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:07.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:07 smithi100 ceph-mon[29092]: pgmap v1042: 158 pgs: 3 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing, 153 active+clean; 3.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 4.7 KiB/s rd, 27 KiB/s wr, 1.13k op/s; 1749/169923 objects degraded (1.029%); 1.5 MiB/s, 212 objects/s recovering 2024-09-17T07:24:07.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:07 smithi100 ceph-mon[39462]: pgmap v1042: 158 pgs: 3 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing, 153 active+clean; 3.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 4.7 KiB/s rd, 27 KiB/s wr, 1.13k op/s; 1749/169923 objects degraded (1.029%); 1.5 MiB/s, 212 objects/s recovering 2024-09-17T07:24:07.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:07.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:24:07.729 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:24:07.774 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:07.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:07.775 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:24:07.776 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-17T07:24:07.776 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-17T07:24:07.776 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-17T07:24:07.777 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-17T07:24:07.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:07 smithi135 ceph-mon[33238]: pgmap v1042: 158 pgs: 3 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing, 153 active+clean; 3.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 4.7 KiB/s rd, 27 KiB/s wr, 1.13k op/s; 1749/169923 objects degraded (1.029%); 1.5 MiB/s, 212 objects/s recovering 2024-09-17T07:24:07.876 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:24:07.909 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:07.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:07.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:24:08.012 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:24:08.048 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:08.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:08.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:24:08.150 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:24:08.195 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:08.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:08.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:24:08.297 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:24:08.338 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:08.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:08.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:24:08.440 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:24:08.472 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:08.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:08.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:24:08.574 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:24:08.596 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:08.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:08.597 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:24:08.694 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:08 smithi135 ceph-mon[33238]: 5.b scrub starts 2024-09-17T07:24:08.697 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:24:08.732 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:08.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:08.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:24:08.860 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:24:08.876 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:08 smithi100 ceph-mon[39462]: 5.b scrub starts 2024-09-17T07:24:08.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:08 smithi100 ceph-mon[29092]: 5.b scrub starts 2024-09-17T07:24:08.892 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:08.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:08.893 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:24:08.994 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:24:09.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:09.045 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:09.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:24:09.146 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:24:09.171 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:09.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:09.172 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:24:09.272 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:24:09.311 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:09.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:09.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:24:09.413 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:24:09.435 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:09.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:09.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:24:09.537 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:24:09.573 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:09.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:09.575 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:24:09.675 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:24:09.702 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:09.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:09.704 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:24:09.734 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:09 smithi135 ceph-mon[33238]: pgmap v1043: 158 pgs: 3 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing, 153 active+clean; 3.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 3.8 KiB/s rd, 22 KiB/s wr, 906 op/s; 1749/169923 objects degraded (1.029%); 1.2 MiB/s, 170 objects/s recovering 2024-09-17T07:24:09.734 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:09 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:24:09.735 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:09 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:24:09.735 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:24:09 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:24:09.804 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:24:09.830 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:09.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:09.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:24:09.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:09 smithi100 ceph-mon[29092]: pgmap v1043: 158 pgs: 3 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing, 153 active+clean; 3.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 3.8 KiB/s rd, 22 KiB/s wr, 906 op/s; 1749/169923 objects degraded (1.029%); 1.2 MiB/s, 170 objects/s recovering 2024-09-17T07:24:09.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:09 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:24:09.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:09 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:24:09.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:09 smithi100 ceph-mon[39462]: pgmap v1043: 158 pgs: 3 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing, 153 active+clean; 3.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 3.8 KiB/s rd, 22 KiB/s wr, 906 op/s; 1749/169923 objects degraded (1.029%); 1.2 MiB/s, 170 objects/s recovering 2024-09-17T07:24:09.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:09 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:24:09.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:09 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:24:09.932 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:24:09.961 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:09.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:09.962 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:24:10.063 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:24:10.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:10.091 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:10.092 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:24:10.192 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:24:10.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:10.225 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:10.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:24:10.326 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:24:10.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:10.351 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:10.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:24:10.452 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:24:10.477 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:10.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:10.478 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:24:10.579 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:24:10.602 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:10.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:10.604 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:24:10.704 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:24:10.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:10.735 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:10.736 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:24:10.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:10 smithi135 ceph-mon[33238]: osdmap e479: 8 total, 8 up, 7 in 2024-09-17T07:24:10.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:10 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:24:10.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:10 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 1749/169923 objects degraded (1.029%), 1 pg degraded (PG_DEGRADED) 2024-09-17T07:24:10.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:10 smithi135 ceph-mon[33238]: osdmap e480: 8 total, 8 up, 7 in 2024-09-17T07:24:10.837 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:24:10.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:10 smithi100 ceph-mon[29092]: osdmap e479: 8 total, 8 up, 7 in 2024-09-17T07:24:10.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:10 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:24:10.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:10 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 1749/169923 objects degraded (1.029%), 1 pg degraded (PG_DEGRADED) 2024-09-17T07:24:10.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:10 smithi100 ceph-mon[29092]: osdmap e480: 8 total, 8 up, 7 in 2024-09-17T07:24:10.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:10 smithi100 ceph-mon[39462]: osdmap e479: 8 total, 8 up, 7 in 2024-09-17T07:24:10.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:10 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:24:10.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:10 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 1749/169923 objects degraded (1.029%), 1 pg degraded (PG_DEGRADED) 2024-09-17T07:24:10.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:10 smithi100 ceph-mon[39462]: osdmap e480: 8 total, 8 up, 7 in 2024-09-17T07:24:10.879 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:10.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:10.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:24:10.981 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:24:11.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:11.026 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:11.026 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:24:11.127 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:24:11.169 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:11.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:11.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:24:11.271 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:24:11.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:11.323 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:11.324 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:24:11.424 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:24:11.453 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:11.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:11.455 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:24:11.555 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:24:11.597 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:11.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:11.598 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:24:11.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:11 smithi100 ceph-mon[29092]: pgmap v1045: 158 pgs: 3 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+clean+scrubbing, 152 active+clean; 3.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 3.8 KiB/s rd, 22 KiB/s wr, 1.10k op/s; 1749/156264 objects degraded (1.119%); 1.2 MiB/s, 253 objects/s recovering 2024-09-17T07:24:11.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:11 smithi100 ceph-mon[39462]: pgmap v1045: 158 pgs: 3 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+clean+scrubbing, 152 active+clean; 3.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 3.8 KiB/s rd, 22 KiB/s wr, 1.10k op/s; 1749/156264 objects degraded (1.119%); 1.2 MiB/s, 253 objects/s recovering 2024-09-17T07:24:11.699 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:24:11.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:11.733 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:11.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:24:11.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:11 smithi135 ceph-mon[33238]: pgmap v1045: 158 pgs: 3 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+clean+scrubbing, 152 active+clean; 3.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 3.8 KiB/s rd, 22 KiB/s wr, 1.10k op/s; 1749/156264 objects degraded (1.119%); 1.2 MiB/s, 253 objects/s recovering 2024-09-17T07:24:11.834 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:24:11.864 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:11.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:11.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:24:11.966 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:24:11.989 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:11.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:11.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:24:12.091 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:24:12.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:12.115 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:12.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:24:12.216 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:24:12.243 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:12.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:12.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:24:12.345 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:24:12.376 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:12.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:12.377 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:24:12.477 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:24:12.500 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:12.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:12.501 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:24:12.601 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:24:12.629 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:12.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:12.630 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:24:12.731 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:24:12.754 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:12.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:12.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:24:12.856 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:24:12.883 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:12.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:12.884 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:24:12.985 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:24:13.010 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:13.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:13.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:24:13.112 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:24:13.138 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:13.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:13.139 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:24:13.240 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:24:13.264 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:13.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:13.265 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:24:13.366 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:24:13.393 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:13.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:13.394 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:24:13.495 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:24:13.520 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:13.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:13.521 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:24:13.622 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:24:13.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:13.644 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:13.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:24:13.745 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:24:13.768 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:13.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:13.769 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:24:13.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:13 smithi135 ceph-mon[33238]: pgmap v1047: 158 pgs: 3 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+clean+scrubbing, 152 active+clean; 3.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.33k op/s; 1749/153174 objects degraded (1.142%); 0 B/s, 309 objects/s recovering 2024-09-17T07:24:13.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:13 smithi135 ceph-mon[33238]: 2.c scrub starts 2024-09-17T07:24:13.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:13 smithi135 ceph-mon[33238]: osdmap e481: 8 total, 8 up, 7 in 2024-09-17T07:24:13.870 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:24:13.876 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:13 smithi100 ceph-mon[39462]: pgmap v1047: 158 pgs: 3 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+clean+scrubbing, 152 active+clean; 3.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.33k op/s; 1749/153174 objects degraded (1.142%); 0 B/s, 309 objects/s recovering 2024-09-17T07:24:13.876 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:13 smithi100 ceph-mon[39462]: 2.c scrub starts 2024-09-17T07:24:13.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:13 smithi100 ceph-mon[39462]: osdmap e481: 8 total, 8 up, 7 in 2024-09-17T07:24:13.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:13 smithi100 ceph-mon[29092]: pgmap v1047: 158 pgs: 3 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+clean+scrubbing, 152 active+clean; 3.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.33k op/s; 1749/153174 objects degraded (1.142%); 0 B/s, 309 objects/s recovering 2024-09-17T07:24:13.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:13 smithi100 ceph-mon[29092]: 2.c scrub starts 2024-09-17T07:24:13.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:13 smithi100 ceph-mon[29092]: osdmap e481: 8 total, 8 up, 7 in 2024-09-17T07:24:13.897 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:13.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:13.898 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:24:13.999 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:24:14.021 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:14.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:14.022 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:24:14.123 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:24:14.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:14.149 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:14.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:24:14.250 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:24:14.276 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:14.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:14.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:24:14.378 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:24:14.404 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:14.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:14.405 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:24:14.435 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-17T07:24:14.506 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:24:14.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:14.561 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:14.561 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:24:14.662 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:24:14.687 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:14.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:14.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:24:14.704 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:14 smithi100 ceph-mon[29092]: 5.6 scrub starts 2024-09-17T07:24:14.704 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:14 smithi100 ceph-mon[29092]: osdmap e482: 8 total, 8 up, 7 in 2024-09-17T07:24:14.705 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:14 smithi100 ceph-mon[29092]: 4.b scrub starts 2024-09-17T07:24:14.705 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:14 smithi100 ceph-mon[39462]: 5.6 scrub starts 2024-09-17T07:24:14.705 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:14 smithi100 ceph-mon[39462]: osdmap e482: 8 total, 8 up, 7 in 2024-09-17T07:24:14.705 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:14 smithi100 ceph-mon[39462]: 4.b scrub starts 2024-09-17T07:24:14.750 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:24:14.789 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:24:14.789 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:24:14.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:14 smithi135 ceph-mon[33238]: 5.6 scrub starts 2024-09-17T07:24:14.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:14 smithi135 ceph-mon[33238]: osdmap e482: 8 total, 8 up, 7 in 2024-09-17T07:24:14.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:14 smithi135 ceph-mon[33238]: 4.b scrub starts 2024-09-17T07:24:14.814 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:14.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:14.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:24:14.916 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:24:14.937 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:14.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:14.938 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:24:15.039 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:24:15.061 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:15.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:15.062 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:24:15.164 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:24:15.186 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:15.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:15.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:24:15.288 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:24:15.310 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:15.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:15.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:24:15.365 INFO:teuthology.orchestra.run.smithi100.stderr:nodeep-scrub is unset 2024-09-17T07:24:15.412 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:24:15.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:15.442 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:15.442 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:24:15.467 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:15 smithi100 ceph-mon[39462]: pgmap v1050: 158 pgs: 3 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 154 active+clean; 3.1 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 381 op/s; 1749/151569 objects degraded (1.154%); 0 B/s, 113 objects/s recovering 2024-09-17T07:24:15.468 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:15 smithi100 ceph-mon[39462]: 5.b scrub starts 2024-09-17T07:24:15.468 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:15 smithi100 ceph-mon[39462]: 11.5 continuing backfill to osd.7 from (438'13350,480'17950] 11:ac2ede00:::benchmark_data_smithi100_180298_object113948:head to 480'17950 2024-09-17T07:24:15.468 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:15 smithi100 ceph-mon[39462]: osdmap e483: 8 total, 8 up, 7 in 2024-09-17T07:24:15.468 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:15 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2313064850' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:24:15.468 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:15 smithi100 ceph-mon[39462]: Health check update: Degraded data redundancy: 1749/151569 objects degraded (1.154%), 1 pg degraded (PG_DEGRADED) 2024-09-17T07:24:15.469 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:15 smithi100 ceph-mon[39462]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T07:24:15.469 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:15 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2313064850' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T07:24:15.469 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:15 smithi100 ceph-mon[39462]: osdmap e484: 8 total, 8 up, 7 in 2024-09-17T07:24:15.469 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:15 smithi100 ceph-mon[29092]: pgmap v1050: 158 pgs: 3 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 154 active+clean; 3.1 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 381 op/s; 1749/151569 objects degraded (1.154%); 0 B/s, 113 objects/s recovering 2024-09-17T07:24:15.469 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:15 smithi100 ceph-mon[29092]: 5.b scrub starts 2024-09-17T07:24:15.470 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:15 smithi100 ceph-mon[29092]: 11.5 continuing backfill to osd.7 from (438'13350,480'17950] 11:ac2ede00:::benchmark_data_smithi100_180298_object113948:head to 480'17950 2024-09-17T07:24:15.470 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:15 smithi100 ceph-mon[29092]: osdmap e483: 8 total, 8 up, 7 in 2024-09-17T07:24:15.470 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:15 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2313064850' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:24:15.470 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:15 smithi100 ceph-mon[29092]: Health check update: Degraded data redundancy: 1749/151569 objects degraded (1.154%), 1 pg degraded (PG_DEGRADED) 2024-09-17T07:24:15.470 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:15 smithi100 ceph-mon[29092]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T07:24:15.470 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:15 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2313064850' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T07:24:15.471 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:15 smithi100 ceph-mon[29092]: osdmap e484: 8 total, 8 up, 7 in 2024-09-17T07:24:15.543 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:24:15.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:15.587 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:15.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:24:15.688 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:24:15.718 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:15.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:15.719 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:24:15.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:15 smithi135 ceph-mon[33238]: pgmap v1050: 158 pgs: 3 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 154 active+clean; 3.1 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 381 op/s; 1749/151569 objects degraded (1.154%); 0 B/s, 113 objects/s recovering 2024-09-17T07:24:15.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:15 smithi135 ceph-mon[33238]: 5.b scrub starts 2024-09-17T07:24:15.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:15 smithi135 ceph-mon[33238]: 11.5 continuing backfill to osd.7 from (438'13350,480'17950] 11:ac2ede00:::benchmark_data_smithi100_180298_object113948:head to 480'17950 2024-09-17T07:24:15.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:15 smithi135 ceph-mon[33238]: osdmap e483: 8 total, 8 up, 7 in 2024-09-17T07:24:15.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:15 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2313064850' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:24:15.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:15 smithi135 ceph-mon[33238]: Health check update: Degraded data redundancy: 1749/151569 objects degraded (1.154%), 1 pg degraded (PG_DEGRADED) 2024-09-17T07:24:15.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:15 smithi135 ceph-mon[33238]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T07:24:15.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:15 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2313064850' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T07:24:15.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:15 smithi135 ceph-mon[33238]: osdmap e484: 8 total, 8 up, 7 in 2024-09-17T07:24:15.819 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:24:15.873 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:15.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:15.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:24:15.974 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:24:16.011 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:16.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:16.012 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:24:16.113 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:24:16.156 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:16.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:16.157 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:24:16.166 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:24:15 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:24:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:24:16.259 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:24:16.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:16.296 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:16.296 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:24:16.397 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:24:16.448 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:16.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:16.449 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:24:16.550 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:24:16.592 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:16.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:16.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:24:16.694 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:24:16.727 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:16.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:16.728 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:24:16.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:16 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:24:16.828 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:24:16.867 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:16.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:16.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:24:16.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:16 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:24:16.876 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:16 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:24:16.969 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:24:17.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:17.002 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:17.003 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:24:17.103 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:24:17.138 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:17.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:17.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:24:17.240 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:24:17.270 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:17.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:17.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:24:17.372 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:24:17.410 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:17.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:17.411 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:24:17.512 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:24:17.541 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:17.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:17.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:24:17.643 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:24:17.665 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:17.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:17.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:24:17.767 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:24:17.788 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:17.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:17.789 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:24:17.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:17 smithi135 ceph-mon[33238]: pgmap v1053: 158 pgs: 1 active+remapped+backfilling, 1 active+undersized+remapped, 2 active+recovering+undersized+remapped, 154 active+clean; 2.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 840 op/s; 350/143103 objects misplaced (0.245%); 5.7 MiB/s, 819 objects/s recovering 2024-09-17T07:24:17.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:17 smithi135 ceph-mon[33238]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1749/151569 objects degraded (1.154%), 1 pg degraded) 2024-09-17T07:24:17.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:17 smithi135 ceph-mon[33238]: Cluster is now healthy 2024-09-17T07:24:17.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:17 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:24:17.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:17 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:24:17.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:17 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:24:17.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:17 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:24:17.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:17 smithi135 ceph-mon[33238]: Adjusting osd_memory_target on smithi135 to 429.8M 2024-09-17T07:24:17.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:17 smithi135 ceph-mon[33238]: Unable to set osd_memory_target on smithi135 to 450693836: error parsing value: Value '450693836' is below minimum 939524096 2024-09-17T07:24:17.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:17 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:24:17.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:17 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:24:17.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:17 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:24:17.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:17 smithi100 ceph-mon[29092]: pgmap v1053: 158 pgs: 1 active+remapped+backfilling, 1 active+undersized+remapped, 2 active+recovering+undersized+remapped, 154 active+clean; 2.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 840 op/s; 350/143103 objects misplaced (0.245%); 5.7 MiB/s, 819 objects/s recovering 2024-09-17T07:24:17.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:17 smithi100 ceph-mon[29092]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1749/151569 objects degraded (1.154%), 1 pg degraded) 2024-09-17T07:24:17.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:17 smithi100 ceph-mon[29092]: Cluster is now healthy 2024-09-17T07:24:17.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:17 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:24:17.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:17 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:24:17.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:17 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:24:17.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:17 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:24:17.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:17 smithi100 ceph-mon[29092]: Adjusting osd_memory_target on smithi135 to 429.8M 2024-09-17T07:24:17.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:17 smithi100 ceph-mon[29092]: Unable to set osd_memory_target on smithi135 to 450693836: error parsing value: Value '450693836' is below minimum 939524096 2024-09-17T07:24:17.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:17 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:24:17.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:17 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:24:17.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:17 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:24:17.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:17 smithi100 ceph-mon[39462]: pgmap v1053: 158 pgs: 1 active+remapped+backfilling, 1 active+undersized+remapped, 2 active+recovering+undersized+remapped, 154 active+clean; 2.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 840 op/s; 350/143103 objects misplaced (0.245%); 5.7 MiB/s, 819 objects/s recovering 2024-09-17T07:24:17.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:17 smithi100 ceph-mon[39462]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1749/151569 objects degraded (1.154%), 1 pg degraded) 2024-09-17T07:24:17.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:17 smithi100 ceph-mon[39462]: Cluster is now healthy 2024-09-17T07:24:17.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:17 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:24:17.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:17 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:24:17.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:17 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:24:17.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:17 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-17T07:24:17.879 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:17 smithi100 ceph-mon[39462]: Adjusting osd_memory_target on smithi135 to 429.8M 2024-09-17T07:24:17.879 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:17 smithi100 ceph-mon[39462]: Unable to set osd_memory_target on smithi135 to 450693836: error parsing value: Value '450693836' is below minimum 939524096 2024-09-17T07:24:17.879 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:17 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:24:17.879 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:17 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:24:17.879 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:17 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:24:17.893 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:24:17.914 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:17.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:17.915 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:24:18.016 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:24:18.037 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:18.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:18.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:24:18.139 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:24:18.162 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:18.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:18.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:24:18.263 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:24:18.285 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:18.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:18.286 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:24:18.387 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:24:18.410 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:18.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:18.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:24:18.416 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:24:18.512 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:24:18.539 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:18.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:18.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:24:18.641 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:24:18.661 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:18 smithi100 ceph-mon[39462]: 2.c scrub starts 2024-09-17T07:24:18.661 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:18 smithi100 ceph-mon[39462]: osdmap e485: 8 total, 8 up, 7 in 2024-09-17T07:24:18.661 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:18 smithi100 ceph-mon[39462]: pgmap v1055: 158 pgs: 1 active+remapped+backfilling, 1 active+undersized+remapped, 1 active+recovering+undersized+remapped, 155 active+clean; 2.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 697 op/s; 350/143046 objects misplaced (0.245%); 5.5 MiB/s, 809 objects/s recovering 2024-09-17T07:24:18.661 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:18 smithi100 ceph-mon[29092]: 2.c scrub starts 2024-09-17T07:24:18.661 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:18 smithi100 ceph-mon[29092]: osdmap e485: 8 total, 8 up, 7 in 2024-09-17T07:24:18.662 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:18 smithi100 ceph-mon[29092]: pgmap v1055: 158 pgs: 1 active+remapped+backfilling, 1 active+undersized+remapped, 1 active+recovering+undersized+remapped, 155 active+clean; 2.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 697 op/s; 350/143046 objects misplaced (0.245%); 5.5 MiB/s, 809 objects/s recovering 2024-09-17T07:24:18.669 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:18.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:18.670 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:24:18.771 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:24:18.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:18 smithi135 ceph-mon[33238]: 2.c scrub starts 2024-09-17T07:24:18.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:18 smithi135 ceph-mon[33238]: osdmap e485: 8 total, 8 up, 7 in 2024-09-17T07:24:18.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:18 smithi135 ceph-mon[33238]: pgmap v1055: 158 pgs: 1 active+remapped+backfilling, 1 active+undersized+remapped, 1 active+recovering+undersized+remapped, 155 active+clean; 2.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 697 op/s; 350/143046 objects misplaced (0.245%); 5.5 MiB/s, 809 objects/s recovering 2024-09-17T07:24:18.793 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:18.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:18.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:24:18.895 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:24:18.916 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:18.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:18.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:24:19.018 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:24:19.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:19.040 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:19.040 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:24:19.141 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:24:19.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:19.163 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:19.163 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:24:19.264 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:24:19.289 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:19.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:19.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:24:19.390 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:24:19.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:19.413 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:19.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:24:19.481 INFO:teuthology.orchestra.run.smithi100.stderr:reweighted osd.0 to 1 (10000) 2024-09-17T07:24:19.514 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:24:19.540 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:19.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:19.541 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:24:19.625 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:19 smithi100 ceph-mon[29092]: 4.12 scrub starts 2024-09-17T07:24:19.625 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:19 smithi100 ceph-mon[29092]: 4.12 scrub ok 2024-09-17T07:24:19.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:19 smithi100 ceph-mon[29092]: 5.6 scrub starts 2024-09-17T07:24:19.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:19 smithi100 ceph-mon[29092]: 5.6 scrub ok 2024-09-17T07:24:19.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:19 smithi100 ceph-mon[29092]: osdmap e486: 8 total, 8 up, 7 in 2024-09-17T07:24:19.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:19 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:24:19.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:19 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:24:19.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:19 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2325236715' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-17T07:24:19.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:19 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-17T07:24:19.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:19 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:24:19.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:19 smithi100 ceph-mon[39462]: 4.12 scrub starts 2024-09-17T07:24:19.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:19 smithi100 ceph-mon[39462]: 4.12 scrub ok 2024-09-17T07:24:19.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:19 smithi100 ceph-mon[39462]: 5.6 scrub starts 2024-09-17T07:24:19.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:19 smithi100 ceph-mon[39462]: 5.6 scrub ok 2024-09-17T07:24:19.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:19 smithi100 ceph-mon[39462]: osdmap e486: 8 total, 8 up, 7 in 2024-09-17T07:24:19.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:19 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:24:19.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:19 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:24:19.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:19 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2325236715' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-17T07:24:19.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:19 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-17T07:24:19.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:19 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:24:19.642 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:24:19.676 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:19.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:19.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:24:19.778 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:24:19.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:19 smithi135 ceph-mon[33238]: 4.12 scrub starts 2024-09-17T07:24:19.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:19 smithi135 ceph-mon[33238]: 4.12 scrub ok 2024-09-17T07:24:19.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:19 smithi135 ceph-mon[33238]: 5.6 scrub starts 2024-09-17T07:24:19.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:19 smithi135 ceph-mon[33238]: 5.6 scrub ok 2024-09-17T07:24:19.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:19 smithi135 ceph-mon[33238]: osdmap e486: 8 total, 8 up, 7 in 2024-09-17T07:24:19.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:19 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:24:19.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:19 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:24:19.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:19 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2325236715' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-17T07:24:19.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:19 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-17T07:24:19.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:19 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:24:19.794 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:24:19 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:24:19.815 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:19.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:19.816 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:24:19.917 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:24:19.953 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:19.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:19.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:24:20.055 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:24:20.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:20.093 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:20.094 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:24:20.194 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:24:20.223 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:20.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:20.224 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:24:20.325 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:24:20.364 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:20.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:20.365 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:24:20.424 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-17T07:24:20.465 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:24:20.500 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:20.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:20.502 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:24:20.602 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:24:20.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:20 smithi100 ceph-mon[39462]: 2.2 scrub starts 2024-09-17T07:24:20.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:20 smithi100 ceph-mon[39462]: 2.2 scrub ok 2024-09-17T07:24:20.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:20 smithi100 ceph-mon[39462]: 6.1a scrub starts 2024-09-17T07:24:20.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:20 smithi100 ceph-mon[39462]: 6.1a scrub ok 2024-09-17T07:24:20.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:20 smithi100 ceph-mon[39462]: 6.1 scrub starts 2024-09-17T07:24:20.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:20 smithi100 ceph-mon[39462]: 6.1 scrub ok 2024-09-17T07:24:20.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:20 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-17T07:24:20.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:20 smithi100 ceph-mon[39462]: osdmap e487: 8 total, 8 up, 7 in 2024-09-17T07:24:20.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:20 smithi100 ceph-mon[39462]: pgmap v1058: 158 pgs: 2 peering, 156 active+clean; 2.7 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 639 op/s; 5.2 MiB/s, 376 objects/s recovering 2024-09-17T07:24:20.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:20 smithi100 ceph-mon[39462]: 5.4 scrub starts 2024-09-17T07:24:20.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:20 smithi100 ceph-mon[39462]: 5.4 scrub ok 2024-09-17T07:24:20.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:20 smithi100 ceph-mon[39462]: osdmap e488: 8 total, 8 up, 7 in 2024-09-17T07:24:20.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:20 smithi100 ceph-mon[29092]: 2.2 scrub starts 2024-09-17T07:24:20.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:20 smithi100 ceph-mon[29092]: 2.2 scrub ok 2024-09-17T07:24:20.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:20 smithi100 ceph-mon[29092]: 6.1a scrub starts 2024-09-17T07:24:20.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:20 smithi100 ceph-mon[29092]: 6.1a scrub ok 2024-09-17T07:24:20.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:20 smithi100 ceph-mon[29092]: 6.1 scrub starts 2024-09-17T07:24:20.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:20 smithi100 ceph-mon[29092]: 6.1 scrub ok 2024-09-17T07:24:20.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:20 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-17T07:24:20.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:20 smithi100 ceph-mon[29092]: osdmap e487: 8 total, 8 up, 7 in 2024-09-17T07:24:20.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:20 smithi100 ceph-mon[29092]: pgmap v1058: 158 pgs: 2 peering, 156 active+clean; 2.7 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 639 op/s; 5.2 MiB/s, 376 objects/s recovering 2024-09-17T07:24:20.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:20 smithi100 ceph-mon[29092]: 5.4 scrub starts 2024-09-17T07:24:20.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:20 smithi100 ceph-mon[29092]: 5.4 scrub ok 2024-09-17T07:24:20.629 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:20 smithi100 ceph-mon[29092]: osdmap e488: 8 total, 8 up, 7 in 2024-09-17T07:24:20.633 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:20.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:20.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:24:20.735 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:24:20.771 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:20.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:20.772 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:24:20.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:20 smithi135 ceph-mon[33238]: 2.2 scrub starts 2024-09-17T07:24:20.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:20 smithi135 ceph-mon[33238]: 2.2 scrub ok 2024-09-17T07:24:20.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:20 smithi135 ceph-mon[33238]: 6.1a scrub starts 2024-09-17T07:24:20.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:20 smithi135 ceph-mon[33238]: 6.1a scrub ok 2024-09-17T07:24:20.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:20 smithi135 ceph-mon[33238]: 6.1 scrub starts 2024-09-17T07:24:20.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:20 smithi135 ceph-mon[33238]: 6.1 scrub ok 2024-09-17T07:24:20.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:20 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-17T07:24:20.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:20 smithi135 ceph-mon[33238]: osdmap e487: 8 total, 8 up, 7 in 2024-09-17T07:24:20.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:20 smithi135 ceph-mon[33238]: pgmap v1058: 158 pgs: 2 peering, 156 active+clean; 2.7 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 639 op/s; 5.2 MiB/s, 376 objects/s recovering 2024-09-17T07:24:20.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:20 smithi135 ceph-mon[33238]: 5.4 scrub starts 2024-09-17T07:24:20.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:20 smithi135 ceph-mon[33238]: 5.4 scrub ok 2024-09-17T07:24:20.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:20 smithi135 ceph-mon[33238]: osdmap e488: 8 total, 8 up, 7 in 2024-09-17T07:24:20.874 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:24:20.912 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:20.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:20.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:24:21.014 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:24:21.049 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:21.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:21.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:24:21.151 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:24:21.184 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:21.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:21.185 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:24:21.286 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:24:21.319 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:21.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:21.320 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:24:21.421 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:24:21.457 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:21.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:21.458 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:24:21.559 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:24:21.596 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:21.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:21.597 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:24:21.697 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:24:21.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:21.733 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:21.734 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:24:21.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:21 smithi135 ceph-mon[33238]: 4.7 scrub starts 2024-09-17T07:24:21.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:21 smithi135 ceph-mon[33238]: 4.7 scrub ok 2024-09-17T07:24:21.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:21 smithi135 ceph-mon[33238]: 5.b deep-scrub starts 2024-09-17T07:24:21.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:21 smithi135 ceph-mon[33238]: 5.b deep-scrub ok 2024-09-17T07:24:21.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:21 smithi135 ceph-mon[33238]: 2.11 scrub starts 2024-09-17T07:24:21.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:21 smithi135 ceph-mon[33238]: 2.11 scrub ok 2024-09-17T07:24:21.834 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:24:21.849 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:21 smithi100 ceph-mon[29092]: 4.7 scrub starts 2024-09-17T07:24:21.849 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:21 smithi100 ceph-mon[29092]: 4.7 scrub ok 2024-09-17T07:24:21.849 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:21 smithi100 ceph-mon[29092]: 5.b deep-scrub starts 2024-09-17T07:24:21.849 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:21 smithi100 ceph-mon[29092]: 5.b deep-scrub ok 2024-09-17T07:24:21.849 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:21 smithi100 ceph-mon[29092]: 2.11 scrub starts 2024-09-17T07:24:21.849 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:21 smithi100 ceph-mon[29092]: 2.11 scrub ok 2024-09-17T07:24:21.849 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:21 smithi100 ceph-mon[39462]: 4.7 scrub starts 2024-09-17T07:24:21.849 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:21 smithi100 ceph-mon[39462]: 4.7 scrub ok 2024-09-17T07:24:21.849 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:21 smithi100 ceph-mon[39462]: 5.b deep-scrub starts 2024-09-17T07:24:21.850 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:21 smithi100 ceph-mon[39462]: 5.b deep-scrub ok 2024-09-17T07:24:21.850 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:21 smithi100 ceph-mon[39462]: 2.11 scrub starts 2024-09-17T07:24:21.850 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:21 smithi100 ceph-mon[39462]: 2.11 scrub ok 2024-09-17T07:24:21.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:21.871 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:21.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:24:21.972 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:24:22.013 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:22.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:22.014 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:24:22.114 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:24:22.149 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:22.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:22.150 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:24:22.251 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:24:22.290 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:22.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:22.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:24:22.367 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd reweight 1 1 2024-09-17T07:24:22.391 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:24:22.433 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-17T07:24:22.433 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-17T07:24:22.433 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-17T07:24:22.433 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-17T07:24:22.433 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-17T07:24:22.453 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:22.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:22.454 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:24:22.555 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:24:22.595 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:22.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:22.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:24:22.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:22 smithi100 ceph-mon[39462]: pgmap v1060: 158 pgs: 3 peering, 155 active+clean; 2.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 848 op/s; 5.4 MiB/s, 379 objects/s recovering 2024-09-17T07:24:22.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:22 smithi100 ceph-mon[39462]: 2.14 scrub starts 2024-09-17T07:24:22.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:22 smithi100 ceph-mon[39462]: 2.14 scrub ok 2024-09-17T07:24:22.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:22 smithi100 ceph-mon[39462]: 6.10 deep-scrub starts 2024-09-17T07:24:22.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:22 smithi100 ceph-mon[39462]: 6.10 deep-scrub ok 2024-09-17T07:24:22.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:22 smithi100 ceph-mon[29092]: pgmap v1060: 158 pgs: 3 peering, 155 active+clean; 2.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 848 op/s; 5.4 MiB/s, 379 objects/s recovering 2024-09-17T07:24:22.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:22 smithi100 ceph-mon[29092]: 2.14 scrub starts 2024-09-17T07:24:22.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:22 smithi100 ceph-mon[29092]: 2.14 scrub ok 2024-09-17T07:24:22.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:22 smithi100 ceph-mon[29092]: 6.10 deep-scrub starts 2024-09-17T07:24:22.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:22 smithi100 ceph-mon[29092]: 6.10 deep-scrub ok 2024-09-17T07:24:22.698 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:24:22.731 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:22.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:22.732 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:24:22.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:22 smithi135 ceph-mon[33238]: pgmap v1060: 158 pgs: 3 peering, 155 active+clean; 2.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 848 op/s; 5.4 MiB/s, 379 objects/s recovering 2024-09-17T07:24:22.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:22 smithi135 ceph-mon[33238]: 2.14 scrub starts 2024-09-17T07:24:22.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:22 smithi135 ceph-mon[33238]: 2.14 scrub ok 2024-09-17T07:24:22.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:22 smithi135 ceph-mon[33238]: 6.10 deep-scrub starts 2024-09-17T07:24:22.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:22 smithi135 ceph-mon[33238]: 6.10 deep-scrub ok 2024-09-17T07:24:22.833 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:24:22.867 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:22.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:22.869 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:24:22.969 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:24:23.004 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:23.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:23.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:24:23.106 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:24:23.121 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-17T07:24:23.135 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:23.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:23.136 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:24:23.237 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:24:23.272 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:23.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:23.273 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:24:23.374 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:24:23.412 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:23.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:23.413 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:24:23.514 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:24:23.559 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:23.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:23.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:24:23.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:23 smithi100 ceph-mon[39462]: 6.7 scrub starts 2024-09-17T07:24:23.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:23 smithi100 ceph-mon[39462]: 6.7 scrub ok 2024-09-17T07:24:23.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:23 smithi100 ceph-mon[29092]: 6.7 scrub starts 2024-09-17T07:24:23.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:23 smithi100 ceph-mon[29092]: 6.7 scrub ok 2024-09-17T07:24:23.661 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:24:23.669 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:24:23.690 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:23.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:23.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:24:23.714 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:24:23.715 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:24:23.741 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:24:23.748 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:24:23.749 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1: {} 2024-09-17T07:24:23.749 INFO:teuthology.orchestra.run.smithi100.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:24:23.784 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:24:23.784 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:24:23.792 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:24:23.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:23 smithi135 ceph-mon[33238]: 6.7 scrub starts 2024-09-17T07:24:23.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:23 smithi135 ceph-mon[33238]: 6.7 scrub ok 2024-09-17T07:24:23.811 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:24:23.812 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:24:23.828 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:23.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:23.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:24:23.838 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:24:23.838 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:24:23.870 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:24:23.870 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:24:23.906 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:24:23.907 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:24:23.930 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:24:23.942 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:24:23.943 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:24:23.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:23.975 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:23.975 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:24:24.076 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:24:24.109 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:24.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:24.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:24:24.211 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:24:24.248 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:24.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:24.249 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:24:24.350 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:24:24.386 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:24.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:24.387 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:24:24.487 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:24:24.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:24.522 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:24.523 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:24:24.624 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:24:24.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:24 smithi100 ceph-mon[29092]: 2.c deep-scrub starts 2024-09-17T07:24:24.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:24 smithi100 ceph-mon[29092]: 2.c deep-scrub ok 2024-09-17T07:24:24.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:24 smithi100 ceph-mon[29092]: pgmap v1061: 158 pgs: 2 peering, 156 active+clean; 2.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 741 op/s; 3.6 MiB/s, 254 objects/s recovering 2024-09-17T07:24:24.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:24 smithi100 ceph-mon[29092]: 2.17 scrub starts 2024-09-17T07:24:24.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:24 smithi100 ceph-mon[29092]: 2.17 scrub ok 2024-09-17T07:24:24.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:24 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/4256986167' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:24:24.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:24 smithi100 ceph-mon[29092]: 6.8 scrub starts 2024-09-17T07:24:24.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:24 smithi100 ceph-mon[29092]: 6.8 scrub ok 2024-09-17T07:24:24.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:24 smithi100 ceph-mon[39462]: 2.c deep-scrub starts 2024-09-17T07:24:24.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:24 smithi100 ceph-mon[39462]: 2.c deep-scrub ok 2024-09-17T07:24:24.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:24 smithi100 ceph-mon[39462]: pgmap v1061: 158 pgs: 2 peering, 156 active+clean; 2.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 741 op/s; 3.6 MiB/s, 254 objects/s recovering 2024-09-17T07:24:24.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:24 smithi100 ceph-mon[39462]: 2.17 scrub starts 2024-09-17T07:24:24.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:24 smithi100 ceph-mon[39462]: 2.17 scrub ok 2024-09-17T07:24:24.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:24 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/4256986167' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:24:24.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:24 smithi100 ceph-mon[39462]: 6.8 scrub starts 2024-09-17T07:24:24.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:24 smithi100 ceph-mon[39462]: 6.8 scrub ok 2024-09-17T07:24:24.656 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:24.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:24.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:24:24.758 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:24:24.790 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:24.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:24.792 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:24:24.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:24 smithi135 ceph-mon[33238]: 2.c deep-scrub starts 2024-09-17T07:24:24.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:24 smithi135 ceph-mon[33238]: 2.c deep-scrub ok 2024-09-17T07:24:24.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:24 smithi135 ceph-mon[33238]: pgmap v1061: 158 pgs: 2 peering, 156 active+clean; 2.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 741 op/s; 3.6 MiB/s, 254 objects/s recovering 2024-09-17T07:24:24.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:24 smithi135 ceph-mon[33238]: 2.17 scrub starts 2024-09-17T07:24:24.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:24 smithi135 ceph-mon[33238]: 2.17 scrub ok 2024-09-17T07:24:24.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:24 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/4256986167' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:24:24.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:24 smithi135 ceph-mon[33238]: 6.8 scrub starts 2024-09-17T07:24:24.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:24 smithi135 ceph-mon[33238]: 6.8 scrub ok 2024-09-17T07:24:24.893 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:24:24.928 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:24.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:24.929 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:24:25.030 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:24:25.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:25.060 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:25.061 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:24:25.161 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:24:25.194 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:25.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:25.195 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:24:25.296 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:24:25.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:25.329 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:25.329 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:24:25.430 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:24:25.469 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:25.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:25.470 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:24:25.570 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:24:25.611 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:25.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:25.613 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:24:25.713 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:24:25.748 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:25.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:25.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:24:25.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:25 smithi135 ceph-mon[33238]: 5.16 scrub starts 2024-09-17T07:24:25.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:25 smithi135 ceph-mon[33238]: 5.16 scrub ok 2024-09-17T07:24:25.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:25 smithi135 ceph-mon[33238]: 5.1d deep-scrub starts 2024-09-17T07:24:25.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:25 smithi135 ceph-mon[33238]: 5.1d deep-scrub ok 2024-09-17T07:24:25.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:25 smithi135 ceph-mon[33238]: 6.14 deep-scrub starts 2024-09-17T07:24:25.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:25 smithi135 ceph-mon[33238]: 6.14 deep-scrub ok 2024-09-17T07:24:25.850 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:24:25.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:25 smithi100 ceph-mon[29092]: 5.16 scrub starts 2024-09-17T07:24:25.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:25 smithi100 ceph-mon[29092]: 5.16 scrub ok 2024-09-17T07:24:25.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:25 smithi100 ceph-mon[29092]: 5.1d deep-scrub starts 2024-09-17T07:24:25.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:25 smithi100 ceph-mon[29092]: 5.1d deep-scrub ok 2024-09-17T07:24:25.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:25 smithi100 ceph-mon[29092]: 6.14 deep-scrub starts 2024-09-17T07:24:25.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:25 smithi100 ceph-mon[29092]: 6.14 deep-scrub ok 2024-09-17T07:24:25.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:25 smithi100 ceph-mon[39462]: 5.16 scrub starts 2024-09-17T07:24:25.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:25 smithi100 ceph-mon[39462]: 5.16 scrub ok 2024-09-17T07:24:25.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:25 smithi100 ceph-mon[39462]: 5.1d deep-scrub starts 2024-09-17T07:24:25.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:25 smithi100 ceph-mon[39462]: 5.1d deep-scrub ok 2024-09-17T07:24:25.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:25 smithi100 ceph-mon[39462]: 6.14 deep-scrub starts 2024-09-17T07:24:25.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:25 smithi100 ceph-mon[39462]: 6.14 deep-scrub ok 2024-09-17T07:24:25.883 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:25.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:25.884 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:24:25.985 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:24:26.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:26.017 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:26.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:24:26.118 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:24:26.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:26.154 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:26.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:24:26.158 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:24:25 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:24:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:24:26.255 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:24:26.294 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:26.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:26.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:24:26.395 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:24:26.429 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:26.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:26.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:24:26.531 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:24:26.570 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:26.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:26.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:24:26.672 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:24:26.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:26.707 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:26.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:24:26.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:26 smithi135 ceph-mon[33238]: 6.19 deep-scrub starts 2024-09-17T07:24:26.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:26 smithi135 ceph-mon[33238]: 6.19 deep-scrub ok 2024-09-17T07:24:26.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:26 smithi135 ceph-mon[33238]: pgmap v1062: 158 pgs: 158 active+clean; 2.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.24k op/s; 0 B/s, 168 objects/s recovering 2024-09-17T07:24:26.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:26 smithi135 ceph-mon[33238]: 5.d scrub starts 2024-09-17T07:24:26.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:26 smithi135 ceph-mon[33238]: 5.d scrub ok 2024-09-17T07:24:26.808 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:24:26.829 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:26.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:26.830 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:24:26.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:26 smithi100 ceph-mon[29092]: 6.19 deep-scrub starts 2024-09-17T07:24:26.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:26 smithi100 ceph-mon[29092]: 6.19 deep-scrub ok 2024-09-17T07:24:26.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:26 smithi100 ceph-mon[29092]: pgmap v1062: 158 pgs: 158 active+clean; 2.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.24k op/s; 0 B/s, 168 objects/s recovering 2024-09-17T07:24:26.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:26 smithi100 ceph-mon[29092]: 5.d scrub starts 2024-09-17T07:24:26.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:26 smithi100 ceph-mon[29092]: 5.d scrub ok 2024-09-17T07:24:26.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:26 smithi100 ceph-mon[39462]: 6.19 deep-scrub starts 2024-09-17T07:24:26.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:26 smithi100 ceph-mon[39462]: 6.19 deep-scrub ok 2024-09-17T07:24:26.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:26 smithi100 ceph-mon[39462]: pgmap v1062: 158 pgs: 158 active+clean; 2.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.24k op/s; 0 B/s, 168 objects/s recovering 2024-09-17T07:24:26.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:26 smithi100 ceph-mon[39462]: 5.d scrub starts 2024-09-17T07:24:26.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:26 smithi100 ceph-mon[39462]: 5.d scrub ok 2024-09-17T07:24:26.931 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:24:26.969 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:26.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:26.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:24:27.071 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:24:27.102 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:27.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:27.103 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:24:27.204 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:24:27.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:27.246 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:27.246 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:24:27.347 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:24:27.387 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:27.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:27.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:24:27.489 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:24:27.521 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:27.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:27.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:24:27.623 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:24:27.655 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:27.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:27.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:24:27.757 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:24:27.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:27.792 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:27.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:24:27.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:27 smithi135 ceph-mon[33238]: 5.c scrub starts 2024-09-17T07:24:27.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:27 smithi135 ceph-mon[33238]: 5.c scrub ok 2024-09-17T07:24:27.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:27 smithi135 ceph-mon[33238]: 6.c scrub starts 2024-09-17T07:24:27.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:27 smithi135 ceph-mon[33238]: 6.c scrub ok 2024-09-17T07:24:27.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:27 smithi135 ceph-mon[33238]: 5.8 scrub starts 2024-09-17T07:24:27.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:27 smithi135 ceph-mon[33238]: 5.8 scrub ok 2024-09-17T07:24:27.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:27 smithi135 ceph-mon[33238]: 2.1b deep-scrub starts 2024-09-17T07:24:27.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:27 smithi135 ceph-mon[33238]: 2.1b deep-scrub ok 2024-09-17T07:24:27.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:27 smithi100 ceph-mon[29092]: 5.c scrub starts 2024-09-17T07:24:27.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:27 smithi100 ceph-mon[29092]: 5.c scrub ok 2024-09-17T07:24:27.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:27 smithi100 ceph-mon[29092]: 6.c scrub starts 2024-09-17T07:24:27.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:27 smithi100 ceph-mon[29092]: 6.c scrub ok 2024-09-17T07:24:27.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:27 smithi100 ceph-mon[29092]: 5.8 scrub starts 2024-09-17T07:24:27.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:27 smithi100 ceph-mon[29092]: 5.8 scrub ok 2024-09-17T07:24:27.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:27 smithi100 ceph-mon[29092]: 2.1b deep-scrub starts 2024-09-17T07:24:27.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:27 smithi100 ceph-mon[29092]: 2.1b deep-scrub ok 2024-09-17T07:24:27.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:27 smithi100 ceph-mon[39462]: 5.c scrub starts 2024-09-17T07:24:27.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:27 smithi100 ceph-mon[39462]: 5.c scrub ok 2024-09-17T07:24:27.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:27 smithi100 ceph-mon[39462]: 6.c scrub starts 2024-09-17T07:24:27.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:27 smithi100 ceph-mon[39462]: 6.c scrub ok 2024-09-17T07:24:27.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:27 smithi100 ceph-mon[39462]: 5.8 scrub starts 2024-09-17T07:24:27.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:27 smithi100 ceph-mon[39462]: 5.8 scrub ok 2024-09-17T07:24:27.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:27 smithi100 ceph-mon[39462]: 2.1b deep-scrub starts 2024-09-17T07:24:27.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:27 smithi100 ceph-mon[39462]: 2.1b deep-scrub ok 2024-09-17T07:24:27.893 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:24:27.931 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:27.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:27.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:24:28.033 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:24:28.063 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:28.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:28.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:24:28.165 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:24:28.186 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:28.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:28.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:24:28.288 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:24:28.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:28.332 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:28.332 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:24:28.433 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:24:28.468 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:28.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:28.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:24:28.570 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:24:28.607 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:28.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:28.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:24:28.709 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:24:28.743 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:28.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:28.744 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:24:28.845 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:24:28.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:28 smithi100 ceph-mon[29092]: 4.1b deep-scrub starts 2024-09-17T07:24:28.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:28 smithi100 ceph-mon[29092]: 4.1b deep-scrub ok 2024-09-17T07:24:28.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:28 smithi100 ceph-mon[29092]: pgmap v1063: 158 pgs: 158 active+clean; 2.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 638 B/s rd, 0 B/s wr, 894 op/s 2024-09-17T07:24:28.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:28 smithi100 ceph-mon[39462]: 4.1b deep-scrub starts 2024-09-17T07:24:28.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:28 smithi100 ceph-mon[39462]: 4.1b deep-scrub ok 2024-09-17T07:24:28.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:28 smithi100 ceph-mon[39462]: pgmap v1063: 158 pgs: 158 active+clean; 2.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 638 B/s rd, 0 B/s wr, 894 op/s 2024-09-17T07:24:28.881 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:28.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:28.882 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:24:28.983 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:24:29.014 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:29.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:29.015 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:24:29.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:28 smithi135 ceph-mon[33238]: 4.1b deep-scrub starts 2024-09-17T07:24:29.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:28 smithi135 ceph-mon[33238]: 4.1b deep-scrub ok 2024-09-17T07:24:29.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:28 smithi135 ceph-mon[33238]: pgmap v1063: 158 pgs: 158 active+clean; 2.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 638 B/s rd, 0 B/s wr, 894 op/s 2024-09-17T07:24:29.116 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:24:29.151 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:29.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:29.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:24:29.253 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:24:29.286 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:29.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:29.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:24:29.388 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:24:29.418 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:29.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:29.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:24:29.520 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:24:29.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:29.549 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:29.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:24:29.651 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:24:29.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:29.684 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:29.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:24:29.785 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:24:29.793 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:24:29 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:24:29.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:29 smithi135 ceph-mon[33238]: 4.e deep-scrub starts 2024-09-17T07:24:29.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:29 smithi135 ceph-mon[33238]: 4.e deep-scrub ok 2024-09-17T07:24:29.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:29 smithi135 ceph-mon[33238]: 4.1f scrub starts 2024-09-17T07:24:29.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:29 smithi135 ceph-mon[33238]: 4.1f scrub ok 2024-09-17T07:24:29.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:29 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:24:29.820 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:29.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:29.821 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:24:29.849 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:29 smithi100 ceph-mon[29092]: 4.e deep-scrub starts 2024-09-17T07:24:29.849 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:29 smithi100 ceph-mon[29092]: 4.e deep-scrub ok 2024-09-17T07:24:29.849 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:29 smithi100 ceph-mon[29092]: 4.1f scrub starts 2024-09-17T07:24:29.849 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:29 smithi100 ceph-mon[29092]: 4.1f scrub ok 2024-09-17T07:24:29.849 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:29 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:24:29.849 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:29 smithi100 ceph-mon[39462]: 4.e deep-scrub starts 2024-09-17T07:24:29.850 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:29 smithi100 ceph-mon[39462]: 4.e deep-scrub ok 2024-09-17T07:24:29.850 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:29 smithi100 ceph-mon[39462]: 4.1f scrub starts 2024-09-17T07:24:29.850 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:29 smithi100 ceph-mon[39462]: 4.1f scrub ok 2024-09-17T07:24:29.850 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:29 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:24:29.921 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:24:29.957 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:29.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:29.958 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:24:30.059 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:24:30.092 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:30.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:30.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:24:30.194 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:24:30.224 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:30.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:30.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:24:30.325 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:24:30.361 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:30.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:30.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:24:30.463 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:24:30.521 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:30.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:30.523 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:24:30.624 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:24:30.659 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:30.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:30.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:24:30.758 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:30 smithi100 ceph-mon[29092]: 4.4 scrub starts 2024-09-17T07:24:30.758 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:30 smithi100 ceph-mon[29092]: 4.4 scrub ok 2024-09-17T07:24:30.758 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:30 smithi100 ceph-mon[29092]: pgmap v1064: 158 pgs: 158 active+clean; 1.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.31k op/s 2024-09-17T07:24:30.758 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:30 smithi100 ceph-mon[29092]: 2.19 scrub starts 2024-09-17T07:24:30.758 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:30 smithi100 ceph-mon[29092]: 2.19 scrub ok 2024-09-17T07:24:30.758 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:30 smithi100 ceph-mon[29092]: 2.1c scrub starts 2024-09-17T07:24:30.758 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:30 smithi100 ceph-mon[29092]: 2.1c scrub ok 2024-09-17T07:24:30.759 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:30 smithi100 ceph-mon[39462]: 4.4 scrub starts 2024-09-17T07:24:30.759 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:30 smithi100 ceph-mon[39462]: 4.4 scrub ok 2024-09-17T07:24:30.759 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:30 smithi100 ceph-mon[39462]: pgmap v1064: 158 pgs: 158 active+clean; 1.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.31k op/s 2024-09-17T07:24:30.759 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:30 smithi100 ceph-mon[39462]: 2.19 scrub starts 2024-09-17T07:24:30.759 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:30 smithi100 ceph-mon[39462]: 2.19 scrub ok 2024-09-17T07:24:30.759 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:30 smithi100 ceph-mon[39462]: 2.1c scrub starts 2024-09-17T07:24:30.759 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:30 smithi100 ceph-mon[39462]: 2.1c scrub ok 2024-09-17T07:24:30.760 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:24:30.807 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:30.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:30.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:24:30.909 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:24:30.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:30.949 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:30.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:24:31.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:30 smithi135 ceph-mon[33238]: 4.4 scrub starts 2024-09-17T07:24:31.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:30 smithi135 ceph-mon[33238]: 4.4 scrub ok 2024-09-17T07:24:31.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:30 smithi135 ceph-mon[33238]: pgmap v1064: 158 pgs: 158 active+clean; 1.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.31k op/s 2024-09-17T07:24:31.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:30 smithi135 ceph-mon[33238]: 2.19 scrub starts 2024-09-17T07:24:31.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:30 smithi135 ceph-mon[33238]: 2.19 scrub ok 2024-09-17T07:24:31.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:30 smithi135 ceph-mon[33238]: 2.1c scrub starts 2024-09-17T07:24:31.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:30 smithi135 ceph-mon[33238]: 2.1c scrub ok 2024-09-17T07:24:31.050 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:24:31.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:31.087 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:31.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:24:31.129 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:24:31.189 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:24:31.204 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-17T07:24:31.204 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:24:31.215 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:31.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:31.216 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:24:31.317 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:24:31.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:31.384 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:31.384 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:24:31.485 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:24:31.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:31.516 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:31.516 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:24:31.617 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:24:31.646 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:31.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:31.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:24:31.748 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:24:31.779 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:31.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:31.780 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:24:31.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:31 smithi100 ceph-mon[29092]: 4.c deep-scrub starts 2024-09-17T07:24:31.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:31 smithi100 ceph-mon[29092]: 4.c deep-scrub ok 2024-09-17T07:24:31.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:31 smithi100 ceph-mon[29092]: 6.d scrub starts 2024-09-17T07:24:31.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:31 smithi100 ceph-mon[29092]: 6.d scrub ok 2024-09-17T07:24:31.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:31 smithi100 ceph-mon[29092]: 6.5 scrub starts 2024-09-17T07:24:31.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:31 smithi100 ceph-mon[29092]: 6.5 scrub ok 2024-09-17T07:24:31.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:31 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2524615021' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:24:31.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:31 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:24:31.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:31 smithi100 ceph-mon[39462]: 4.c deep-scrub starts 2024-09-17T07:24:31.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:31 smithi100 ceph-mon[39462]: 4.c deep-scrub ok 2024-09-17T07:24:31.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:31 smithi100 ceph-mon[39462]: 6.d scrub starts 2024-09-17T07:24:31.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:31 smithi100 ceph-mon[39462]: 6.d scrub ok 2024-09-17T07:24:31.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:31 smithi100 ceph-mon[39462]: 6.5 scrub starts 2024-09-17T07:24:31.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:31 smithi100 ceph-mon[39462]: 6.5 scrub ok 2024-09-17T07:24:31.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:31 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2524615021' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:24:31.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:31 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:24:31.881 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:24:31.915 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:31.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:31.916 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:24:32.017 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:24:32.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:31 smithi135 ceph-mon[33238]: 4.c deep-scrub starts 2024-09-17T07:24:32.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:31 smithi135 ceph-mon[33238]: 4.c deep-scrub ok 2024-09-17T07:24:32.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:31 smithi135 ceph-mon[33238]: 6.d scrub starts 2024-09-17T07:24:32.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:31 smithi135 ceph-mon[33238]: 6.d scrub ok 2024-09-17T07:24:32.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:31 smithi135 ceph-mon[33238]: 6.5 scrub starts 2024-09-17T07:24:32.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:31 smithi135 ceph-mon[33238]: 6.5 scrub ok 2024-09-17T07:24:32.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:31 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2524615021' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:24:32.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:31 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-17T07:24:32.052 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:32.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:32.053 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:24:32.154 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:24:32.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:32.181 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:32.181 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:24:32.282 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:24:32.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:32.319 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:32.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:24:32.420 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:24:32.441 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:32.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:32.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:24:32.543 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:24:32.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:32.570 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:32.570 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:24:32.671 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:24:32.709 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:32.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:32.710 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:24:32.811 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:24:32.845 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:32.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:32.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:24:32.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:32 smithi100 ceph-mon[29092]: 5.15 scrub starts 2024-09-17T07:24:32.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:32 smithi100 ceph-mon[29092]: 5.15 scrub ok 2024-09-17T07:24:32.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:32 smithi100 ceph-mon[29092]: 4.2 scrub starts 2024-09-17T07:24:32.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:32 smithi100 ceph-mon[29092]: 4.2 scrub ok 2024-09-17T07:24:32.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:32 smithi100 ceph-mon[29092]: pgmap v1065: 158 pgs: 158 active+clean; 1.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 920 B/s rd, 0 B/s wr, 1.25k op/s 2024-09-17T07:24:32.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:32 smithi100 ceph-mon[29092]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:24:32.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:32 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T07:24:32.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:32 smithi100 ceph-mon[29092]: osdmap e489: 8 total, 8 up, 7 in 2024-09-17T07:24:32.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:32 smithi100 ceph-mon[39462]: 5.15 scrub starts 2024-09-17T07:24:32.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:32 smithi100 ceph-mon[39462]: 5.15 scrub ok 2024-09-17T07:24:32.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:32 smithi100 ceph-mon[39462]: 4.2 scrub starts 2024-09-17T07:24:32.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:32 smithi100 ceph-mon[39462]: 4.2 scrub ok 2024-09-17T07:24:32.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:32 smithi100 ceph-mon[39462]: pgmap v1065: 158 pgs: 158 active+clean; 1.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 920 B/s rd, 0 B/s wr, 1.25k op/s 2024-09-17T07:24:32.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:32 smithi100 ceph-mon[39462]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:24:32.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:32 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T07:24:32.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:32 smithi100 ceph-mon[39462]: osdmap e489: 8 total, 8 up, 7 in 2024-09-17T07:24:32.947 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:24:32.984 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:32.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:32.985 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:24:33.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:32 smithi135 ceph-mon[33238]: 5.15 scrub starts 2024-09-17T07:24:33.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:32 smithi135 ceph-mon[33238]: 5.15 scrub ok 2024-09-17T07:24:33.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:32 smithi135 ceph-mon[33238]: 4.2 scrub starts 2024-09-17T07:24:33.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:32 smithi135 ceph-mon[33238]: 4.2 scrub ok 2024-09-17T07:24:33.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:32 smithi135 ceph-mon[33238]: pgmap v1065: 158 pgs: 158 active+clean; 1.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 920 B/s rd, 0 B/s wr, 1.25k op/s 2024-09-17T07:24:33.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:32 smithi135 ceph-mon[33238]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:24:33.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:32 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T07:24:33.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:32 smithi135 ceph-mon[33238]: osdmap e489: 8 total, 8 up, 7 in 2024-09-17T07:24:33.086 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:24:33.121 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:33.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:33.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:24:33.223 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:24:33.259 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:33.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:33.260 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:24:33.361 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:24:33.397 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:33.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:33.398 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:24:33.499 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:24:33.535 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:33.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:33.536 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:24:33.594 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-17T07:24:33.638 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:24:33.677 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:33.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:33.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:24:33.779 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:24:33.837 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:33.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:33.839 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:24:33.940 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:24:33.952 INFO:teuthology.orchestra.run.smithi100.stderr:noscrub is set 2024-09-17T07:24:33.955 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:24:33.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:33.992 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:33.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:24:34.093 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:24:34.131 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:34.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:34.132 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:24:34.233 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:24:34.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:34.264 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:34.264 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:24:34.365 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:24:34.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:34.398 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:34.398 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:24:34.498 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:24:34.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:34.536 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:34.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:24:34.637 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:24:34.665 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:24:34.676 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:34.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:34.677 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:24:34.708 INFO:teuthology.orchestra.run.smithi100.stderr:reweighted osd.1 to 1 (10000) 2024-09-17T07:24:34.779 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:24:34.817 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:34.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:34.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:24:34.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:34 smithi100 ceph-mon[29092]: pgmap v1067: 158 pgs: 158 active+clean; 1.8 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.39k op/s 2024-09-17T07:24:34.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:34 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:24:34.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:34 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:24:34.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:34 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/3443046776' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-17T07:24:34.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:34 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-17T07:24:34.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:34 smithi100 ceph-mon[39462]: pgmap v1067: 158 pgs: 158 active+clean; 1.8 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.39k op/s 2024-09-17T07:24:34.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:34 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:24:34.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:34 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:24:34.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:34 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/3443046776' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-17T07:24:34.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:34 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-17T07:24:34.919 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:24:34.952 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:34.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:34.953 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:24:35.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:34 smithi135 ceph-mon[33238]: pgmap v1067: 158 pgs: 158 active+clean; 1.8 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.39k op/s 2024-09-17T07:24:35.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:34 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:24:35.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:34 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:24:35.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:34 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/3443046776' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-17T07:24:35.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:34 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-17T07:24:35.054 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:24:35.089 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:35.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:35.091 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:24:35.191 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:24:35.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:35.226 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:35.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:24:35.327 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:24:35.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:35.371 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:35.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:24:35.472 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:24:35.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:35.504 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:35.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:24:35.606 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:24:35.640 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:35.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:35.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:24:35.742 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:24:35.773 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:35.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:35.774 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:24:35.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:35 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-17T07:24:35.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:35 smithi135 ceph-mon[33238]: osdmap e490: 8 total, 8 up, 7 in 2024-09-17T07:24:35.875 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:24:35.914 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:35.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:35.915 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:24:36.016 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:24:36.048 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:36.049 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:35 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-17T07:24:36.049 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:35 smithi100 ceph-mon[29092]: osdmap e490: 8 total, 8 up, 7 in 2024-09-17T07:24:36.050 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:35 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-17T07:24:36.050 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:35 smithi100 ceph-mon[39462]: osdmap e490: 8 total, 8 up, 7 in 2024-09-17T07:24:36.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:36.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:24:36.052 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:24:35 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:24:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:24:36.069 INFO:teuthology.orchestra.run.smithi135.stdout:ERROR: (22) Invalid argument 2024-09-17T07:24:36.069 INFO:teuthology.orchestra.run.smithi135.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:24:36.080 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd reweight 2 1 2024-09-17T07:24:36.153 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:24:36.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:36.202 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:36.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:24:36.303 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:24:36.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:36.337 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:36.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:24:36.438 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:24:36.482 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:36.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:36.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:24:36.584 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:24:36.617 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:36.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:36.618 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:24:36.636 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-17T07:24:36.718 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:24:36.752 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:36.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:36.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:24:36.854 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:24:36.884 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:36.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:36.885 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:24:36.953 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-17T07:24:36.971 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:36 smithi100 ceph-mon[29092]: pgmap v1069: 158 pgs: 158 active+clean; 1.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.70k op/s 2024-09-17T07:24:36.971 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:36 smithi100 ceph-mon[39462]: pgmap v1069: 158 pgs: 158 active+clean; 1.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.70k op/s 2024-09-17T07:24:36.986 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:24:36.992 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:36 smithi135 ceph-mon[33238]: pgmap v1069: 158 pgs: 158 active+clean; 1.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.70k op/s 2024-09-17T07:24:37.015 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:37.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:37.016 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:24:37.116 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:24:37.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:37.155 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:37.155 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:24:37.256 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:24:37.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:37.288 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:37.288 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:24:37.343 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:24:37.389 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:24:37.424 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:37.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:37.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:24:37.511 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:24:37.512 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:24:37.517 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:24:37.525 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:24:37.556 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1: {} 2024-09-17T07:24:37.556 INFO:teuthology.orchestra.run.smithi100.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:24:37.572 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:37.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:37.573 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:24:37.585 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:24:37.588 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:24:37.588 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:24:37.619 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:24:37.620 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:24:37.653 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:24:37.653 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:24:37.674 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:24:37.686 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:24:37.686 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:24:37.707 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:37.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:37.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:24:37.720 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:24:37.720 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:24:37.750 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:24:37.750 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:24:37.810 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:24:37.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:37.844 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:37.844 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:24:37.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:37 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2071509987' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:24:37.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:37 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2071509987' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:24:37.921 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:37 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2071509987' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:24:37.945 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:24:37.981 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:37.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:37.982 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:24:38.083 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:24:38.112 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:38.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:38.113 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:24:38.214 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:24:38.243 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:38.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:38.244 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:24:38.345 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:24:38.382 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:38.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:38.383 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:24:38.484 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:24:38.514 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:38.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:38.516 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:24:38.616 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:24:38.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:38.650 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:38.650 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:24:38.752 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:24:38.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:38.784 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:38.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:24:38.886 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:24:38.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:38.922 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:38.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:24:38.995 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:38 smithi100 ceph-mon[29092]: pgmap v1070: 158 pgs: 158 active+clean; 1.4 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.02k op/s 2024-09-17T07:24:38.995 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:38 smithi100 ceph-mon[39462]: pgmap v1070: 158 pgs: 158 active+clean; 1.4 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.02k op/s 2024-09-17T07:24:39.023 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:24:39.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:38 smithi135 ceph-mon[33238]: pgmap v1070: 158 pgs: 158 active+clean; 1.4 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.02k op/s 2024-09-17T07:24:39.056 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:39.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:39.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:24:39.158 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:24:39.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:39.207 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:39.208 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:24:39.308 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:24:39.337 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:39.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:39.338 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:24:39.439 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:24:39.476 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:39.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:39.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:24:39.579 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:24:39.637 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:39.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:39.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:24:39.739 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:24:39.746 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:24:39 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:24:39.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:39.774 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:39.774 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:24:39.875 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:24:39.906 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:39.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:39.907 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:24:40.008 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:24:40.041 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:39 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:24:40.041 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:39 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:24:40.041 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:39 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:24:40.045 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:40.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:40.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:24:40.099 INFO:teuthology.orchestra.run.smithi135.stdout:{ 2024-09-17T07:24:40.100 INFO:teuthology.orchestra.run.smithi135.stdout: "ops": [], 2024-09-17T07:24:40.100 INFO:teuthology.orchestra.run.smithi135.stdout: "complaint_time": 30, 2024-09-17T07:24:40.100 INFO:teuthology.orchestra.run.smithi135.stdout: "num_blocked_ops": 0 2024-09-17T07:24:40.100 INFO:teuthology.orchestra.run.smithi135.stdout:} 2024-09-17T07:24:40.147 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:24:40.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:40.184 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:40.184 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:24:40.285 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:24:40.317 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:40.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:40.318 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:24:40.420 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:24:40.453 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:40.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:40.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:24:40.555 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:24:40.589 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:40.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:40.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:24:40.691 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:24:40.736 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:40.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:40.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:24:40.838 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:24:40.877 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:40.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:40.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:24:40.979 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:24:41.011 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:41.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:41.012 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:24:41.018 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:40 smithi100 ceph-mon[29092]: pgmap v1071: 158 pgs: 158 active+clean; 1.1 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.56k op/s 2024-09-17T07:24:41.019 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:40 smithi100 ceph-mon[39462]: pgmap v1071: 158 pgs: 158 active+clean; 1.1 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.56k op/s 2024-09-17T07:24:41.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:40 smithi135 ceph-mon[33238]: pgmap v1071: 158 pgs: 158 active+clean; 1.1 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.56k op/s 2024-09-17T07:24:41.112 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:24:41.146 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:41.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:41.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:24:41.247 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:24:41.281 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:41.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:41.282 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:24:41.384 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:24:41.422 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:41.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:41.423 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:24:41.524 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:24:41.565 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:41.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:41.566 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:24:41.667 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:24:41.693 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:41.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:41.694 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:24:41.796 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:24:41.832 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:41.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:41.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:24:41.941 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:24:41.966 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:41.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:41.967 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:24:42.068 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:24:42.104 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:42.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:42.105 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:24:42.206 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:24:42.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:42.243 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:42.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:24:42.344 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:24:42.357 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:24:42.381 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:42.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:42.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:24:42.483 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:24:42.520 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:42.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:42.521 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:24:42.554 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-17T07:24:42.622 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:24:42.680 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:42.681 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:24:42.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:42.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:24:42.761 INFO:teuthology.orchestra.run.smithi100.stderr:nodeep-scrub is set 2024-09-17T07:24:42.763 INFO:teuthology.orchestra.run.smithi100.stderr:reweighted osd.2 to 1 (10000) 2024-09-17T07:24:42.782 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:24:42.832 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:42.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:42.833 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:24:42.933 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:24:42.945 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:24:42.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:42.964 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:42.965 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:24:43.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:42 smithi135 ceph-mon[33238]: pgmap v1072: 158 pgs: 158 active+clean; 1.0 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.39k op/s 2024-09-17T07:24:43.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:42 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2616789183' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-17T07:24:43.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:42 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-17T07:24:43.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:42 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/3785066132' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:24:43.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:42 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:24:43.066 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:24:43.097 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:43.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:43.098 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:24:43.126 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:42 smithi100 ceph-mon[39462]: pgmap v1072: 158 pgs: 158 active+clean; 1.0 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.39k op/s 2024-09-17T07:24:43.126 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:42 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2616789183' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-17T07:24:43.126 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:42 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-17T07:24:43.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:42 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/3785066132' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:24:43.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:42 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:24:43.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:42 smithi100 ceph-mon[29092]: pgmap v1072: 158 pgs: 158 active+clean; 1.0 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.39k op/s 2024-09-17T07:24:43.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:42 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2616789183' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-17T07:24:43.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:42 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-17T07:24:43.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:42 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/3785066132' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:24:43.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:42 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:24:43.199 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:24:43.244 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:43.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:43.245 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:24:43.346 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:24:43.382 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:43.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:43.383 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:24:43.483 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:24:43.527 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:43.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:43.528 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:24:43.629 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:24:43.666 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:43.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:43.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:24:43.768 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:24:43.805 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:43.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:43.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:24:43.907 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:24:43.942 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:43.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:43.943 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:24:44.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:43 smithi135 ceph-mon[33238]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:24:44.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:43 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-17T07:24:44.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:43 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T07:24:44.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:43 smithi135 ceph-mon[33238]: osdmap e491: 8 total, 8 up, 7 in 2024-09-17T07:24:44.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:43 smithi135 ceph-mon[33238]: osdmap e492: 8 total, 8 up, 7 in 2024-09-17T07:24:44.044 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:24:44.084 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:44.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:44.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:24:44.126 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:43 smithi100 ceph-mon[29092]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:24:44.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:43 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-17T07:24:44.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:43 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T07:24:44.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:43 smithi100 ceph-mon[29092]: osdmap e491: 8 total, 8 up, 7 in 2024-09-17T07:24:44.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:43 smithi100 ceph-mon[29092]: osdmap e492: 8 total, 8 up, 7 in 2024-09-17T07:24:44.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:43 smithi100 ceph-mon[39462]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:24:44.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:43 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-17T07:24:44.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:43 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T07:24:44.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:43 smithi100 ceph-mon[39462]: osdmap e491: 8 total, 8 up, 7 in 2024-09-17T07:24:44.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:43 smithi100 ceph-mon[39462]: osdmap e492: 8 total, 8 up, 7 in 2024-09-17T07:24:44.186 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:24:44.221 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:44.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:44.222 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:24:44.323 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:24:44.353 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:44.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:44.354 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:24:44.455 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:24:44.491 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:44.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:44.493 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:24:44.593 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:24:44.630 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:44.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:44.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:24:44.733 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:24:44.764 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:44.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:44.765 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:24:44.866 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:24:44.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:44.901 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:44.901 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:24:45.002 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:24:45.039 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:45.039 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:44 smithi100 ceph-mon[29092]: pgmap v1074: 158 pgs: 158 active+clean; 962 MiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 581 B/s rd, 0 B/s wr, 1.08k op/s 2024-09-17T07:24:45.040 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:44 smithi100 ceph-mon[39462]: pgmap v1074: 158 pgs: 158 active+clean; 962 MiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 581 B/s rd, 0 B/s wr, 1.08k op/s 2024-09-17T07:24:45.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:45.040 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:24:45.041 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd reweight 4 1 2024-09-17T07:24:45.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:44 smithi135 ceph-mon[33238]: pgmap v1074: 158 pgs: 158 active+clean; 962 MiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 581 B/s rd, 0 B/s wr, 1.08k op/s 2024-09-17T07:24:45.141 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:24:45.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:45.177 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:45.178 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:24:45.278 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:24:45.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:45.334 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:45.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:24:45.435 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:24:45.472 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:45.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:45.473 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:24:45.574 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:24:45.609 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:45.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:45.610 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:24:45.711 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:24:45.742 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:45.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:45.743 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:24:45.844 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:24:45.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:45.878 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:45.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:24:45.980 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:24:46.020 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:46.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:46.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:24:46.122 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:24:46.126 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:24:45 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:24:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:24:46.143 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:46.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:46.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:24:46.224 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:24:46.224 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:24:46.243 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:24:46.245 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:24:46.262 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1: {} 2024-09-17T07:24:46.262 INFO:teuthology.orchestra.run.smithi100.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:24:46.279 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:46.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:46.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:24:46.289 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:24:46.289 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:24:46.320 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:24:46.321 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:24:46.352 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:24:46.353 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:24:46.381 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:24:46.387 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:24:46.387 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:24:46.410 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:46.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:46.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:24:46.422 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:24:46.423 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:24:46.423 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:46 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:24:46.424 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:46 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:24:46.424 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:46 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/327953448' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:24:46.424 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:46 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:24:46.424 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:46 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:24:46.424 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:46 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/327953448' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:24:46.450 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:24:46.450 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:24:46.512 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:24:46.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:46.536 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:46.536 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:24:46.637 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:24:46.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:46.676 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:46.676 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:24:46.777 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:24:46.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:46 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:24:46.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:46 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:24:46.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:46 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/327953448' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:24:46.813 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:46.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:46.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:24:46.915 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:24:46.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:46.947 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:46.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:24:47.048 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:24:47.081 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:47.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:47.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:24:47.184 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:24:47.221 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:47.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:47.222 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:24:47.322 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:24:47.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:47.355 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:47.356 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:24:47.457 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:24:47.492 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:47.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:47.493 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:24:47.523 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:47 smithi135 ceph-mon[33238]: pgmap v1076: 158 pgs: 1 peering, 157 active+clean; 621 MiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.72k op/s 2024-09-17T07:24:47.523 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:47 smithi135 ceph-mon[33238]: osdmap e493: 8 total, 8 up, 7 in 2024-09-17T07:24:47.594 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:24:47.632 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:47.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:47.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:24:47.686 INFO:teuthology.orchestra.run.smithi135.stdout:ERROR: (22) Invalid argument 2024-09-17T07:24:47.687 INFO:teuthology.orchestra.run.smithi135.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:24:47.734 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:24:47.755 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:47 smithi100 ceph-mon[29092]: pgmap v1076: 158 pgs: 1 peering, 157 active+clean; 621 MiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.72k op/s 2024-09-17T07:24:47.755 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:47 smithi100 ceph-mon[29092]: osdmap e493: 8 total, 8 up, 7 in 2024-09-17T07:24:47.755 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:47 smithi100 ceph-mon[39462]: pgmap v1076: 158 pgs: 1 peering, 157 active+clean; 621 MiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.72k op/s 2024-09-17T07:24:47.756 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:47 smithi100 ceph-mon[39462]: osdmap e493: 8 total, 8 up, 7 in 2024-09-17T07:24:47.765 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:47.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:47.766 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:24:47.866 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:24:47.900 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:47.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:47.902 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:24:47.905 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T07:24:48.003 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:24:48.028 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:48.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:48.029 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:24:48.130 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:24:48.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:48.165 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:48.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:24:48.266 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:24:48.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:48.288 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:48.288 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:24:48.389 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:24:48.426 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:48.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:48.427 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:24:48.528 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:24:48.531 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-17T07:24:48.532 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:48 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:24:48.532 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:48 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:24:48.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:48.565 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:48.565 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:24:48.666 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:24:48.698 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:48.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:48.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:24:48.800 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:24:48.853 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:48.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:48.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:24:48.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:48 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:24:48.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:48 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:24:48.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:48 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:24:48.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:48 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:24:48.900 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:24:48.954 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:24:48.995 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:48.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:48.996 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:24:49.097 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:24:49.135 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:49.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:49.136 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:24:49.237 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:24:49.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:49.272 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:49.272 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:24:49.373 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:24:49.408 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:49.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:49.409 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:24:49.510 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:24:49.528 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:49 smithi100 ceph-mon[29092]: pgmap v1078: 158 pgs: 1 peering, 157 active+clean; 592 MiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.20k op/s 2024-09-17T07:24:49.528 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:49 smithi100 ceph-mon[29092]: osdmap e494: 8 total, 8 up, 7 in 2024-09-17T07:24:49.528 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:49 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:24:49.528 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:49 smithi100 ceph-mon[39462]: pgmap v1078: 158 pgs: 1 peering, 157 active+clean; 592 MiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.20k op/s 2024-09-17T07:24:49.529 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:49 smithi100 ceph-mon[39462]: osdmap e494: 8 total, 8 up, 7 in 2024-09-17T07:24:49.529 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:49 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:24:49.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:49.559 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:49.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:24:49.602 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:24:49.660 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:24:49.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:49.705 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:49.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:24:49.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:49 smithi135 ceph-mon[33238]: pgmap v1078: 158 pgs: 1 peering, 157 active+clean; 592 MiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.20k op/s 2024-09-17T07:24:49.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:49 smithi135 ceph-mon[33238]: osdmap e494: 8 total, 8 up, 7 in 2024-09-17T07:24:49.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:49 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:24:49.794 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:24:49 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:24:49.806 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:24:49.847 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:49.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:49.848 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:24:49.949 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:24:49.979 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:49.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:49.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:24:50.081 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:24:50.119 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:50.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:50.120 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:24:50.221 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:24:50.255 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:50.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:50.256 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:24:50.357 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:24:50.385 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:50.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:50.386 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:24:50.420 INFO:teuthology.orchestra.run.smithi100.stderr:reweighted osd.4 to 1 (10000) 2024-09-17T07:24:50.487 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:24:50.521 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:50.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:50.522 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:24:50.623 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:24:50.660 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:50.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:50.661 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:24:50.761 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:24:50.792 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:50.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:50.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:24:50.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:50 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:24:50.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:50 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2283647803' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-17T07:24:50.865 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Removed 172246 objects 2024-09-17T07:24:50.865 INFO:tasks.radosbench.radosbench.0.smithi100.stdout:Clean up completed and total clean up time :190.687 2024-09-17T07:24:50.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:50 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:24:50.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:50 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2283647803' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-17T07:24:50.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:50 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:24:50.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:50 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2283647803' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-17T07:24:50.894 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:24:50.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:50.916 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:50.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:24:50.965 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T07:24:51.017 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:24:51.037 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:51.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:51.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:24:51.139 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:24:51.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:51.161 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:51.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:24:51.236 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd reweight 5 1 2024-09-17T07:24:51.261 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:24:51.301 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:51.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:51.302 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:24:51.402 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:24:51.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:51.426 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:51.426 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:24:51.527 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:24:51.548 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:51.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:51.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:24:51.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:51 smithi100 ceph-mon[39462]: pgmap v1080: 158 pgs: 158 active+clean; 227 MiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.96k op/s; 18 B/s, 1 objects/s recovering 2024-09-17T07:24:51.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:51 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2283647803' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-17T07:24:51.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:51 smithi100 ceph-mon[39462]: osdmap e495: 8 total, 8 up, 7 in 2024-09-17T07:24:51.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:51 smithi100 ceph-mon[29092]: pgmap v1080: 158 pgs: 158 active+clean; 227 MiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.96k op/s; 18 B/s, 1 objects/s recovering 2024-09-17T07:24:51.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:51 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2283647803' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-17T07:24:51.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:51 smithi100 ceph-mon[29092]: osdmap e495: 8 total, 8 up, 7 in 2024-09-17T07:24:51.651 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:24:51.674 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:51.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:51.675 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:24:51.775 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:24:51.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:51 smithi135 ceph-mon[33238]: pgmap v1080: 158 pgs: 158 active+clean; 227 MiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.96k op/s; 18 B/s, 1 objects/s recovering 2024-09-17T07:24:51.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:51 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2283647803' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-17T07:24:51.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:51 smithi135 ceph-mon[33238]: osdmap e495: 8 total, 8 up, 7 in 2024-09-17T07:24:51.797 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:51.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:51.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:24:51.871 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:24:51.899 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:24:51.922 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:51.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:51.923 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:24:52.024 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:24:52.047 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:52.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:52.048 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:24:52.149 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:24:52.172 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:52.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:52.173 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:24:52.274 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:24:52.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:52.297 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:52.297 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:24:52.398 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:24:52.419 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:52.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:52.420 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:24:52.521 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:24:52.544 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:52.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:52.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:24:52.644 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:52 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:24:52.645 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:52 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:24:52.645 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:52 smithi135 ceph-mon[33238]: osdmap e496: 8 total, 8 up, 7 in 2024-09-17T07:24:52.645 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:24:52.651 INFO:teuthology.orchestra.run.smithi135.stdout:ERROR: (22) Invalid argument 2024-09-17T07:24:52.651 INFO:teuthology.orchestra.run.smithi135.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:24:52.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:52.669 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:52.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:24:52.770 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:24:52.791 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:52.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:52.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:24:52.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:52 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:24:52.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:52 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:24:52.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:52 smithi100 ceph-mon[29092]: osdmap e496: 8 total, 8 up, 7 in 2024-09-17T07:24:52.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:52 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:24:52.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:52 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:24:52.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:52 smithi100 ceph-mon[39462]: osdmap e496: 8 total, 8 up, 7 in 2024-09-17T07:24:52.893 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:24:52.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:52.916 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:52.916 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:24:53.017 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:24:53.038 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:53.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:53.039 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:24:53.140 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:24:53.164 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:53.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:53.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:24:53.255 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-17T07:24:53.266 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:24:53.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:53.289 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:53.289 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:24:53.390 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:24:53.412 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:53.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:53.413 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:24:53.459 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:24:53.514 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:24:53.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:53.536 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:53.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:24:53.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:53 smithi135 ceph-mon[33238]: pgmap v1083: 158 pgs: 158 active+clean; 147 MiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 1002 B/s rd, 0 B/s wr, 1.39k op/s; 21 B/s, 1 objects/s recovering 2024-09-17T07:24:53.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:53 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:24:53.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:53 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:24:53.638 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:24:53.659 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:53.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:53.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:24:53.761 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:24:53.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:53.784 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:53.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:24:53.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:53 smithi100 ceph-mon[29092]: pgmap v1083: 158 pgs: 158 active+clean; 147 MiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 1002 B/s rd, 0 B/s wr, 1.39k op/s; 21 B/s, 1 objects/s recovering 2024-09-17T07:24:53.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:53 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:24:53.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:53 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:24:53.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:53 smithi100 ceph-mon[39462]: pgmap v1083: 158 pgs: 158 active+clean; 147 MiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 1002 B/s rd, 0 B/s wr, 1.39k op/s; 21 B/s, 1 objects/s recovering 2024-09-17T07:24:53.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:53 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:24:53.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:53 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:24:53.885 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:24:53.907 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:53.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:53.908 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:24:54.009 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:24:54.031 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:54.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:54.032 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:24:54.133 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:24:54.156 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:54.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:54.157 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:24:54.258 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:24:54.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:54.280 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:54.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:24:54.381 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:24:54.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:54.405 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:54.405 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:24:54.506 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:24:54.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:54.534 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:54.534 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:24:54.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:54 smithi135 ceph-mon[33238]: osdmap e497: 8 total, 8 up, 7 in 2024-09-17T07:24:54.634 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:24:54.656 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:54.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:54.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:24:54.758 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:24:54.780 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:54.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:54.781 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:24:54.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:54 smithi100 ceph-mon[29092]: osdmap e497: 8 total, 8 up, 7 in 2024-09-17T07:24:54.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:54 smithi100 ceph-mon[39462]: osdmap e497: 8 total, 8 up, 7 in 2024-09-17T07:24:54.882 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:24:54.905 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:54.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:54.906 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:24:55.007 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:24:55.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:55.030 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:55.030 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:24:55.122 INFO:teuthology.orchestra.run.smithi135.stdout:ERROR: (22) Invalid argument 2024-09-17T07:24:55.122 INFO:teuthology.orchestra.run.smithi135.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:24:55.130 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:24:55.153 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:55.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:55.154 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:24:55.255 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:24:55.281 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:55.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:55.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:24:55.382 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:24:55.407 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:55.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:55.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:24:55.509 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:24:55.531 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:55.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:55.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:24:55.633 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:24:55.656 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:55.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:55.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:24:55.710 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:55 smithi100 ceph-mon[29092]: pgmap v1085: 158 pgs: 158 active+clean; 106 MiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 1005 B/s rd, 0 B/s wr, 1.26k op/s; 21 B/s, 1 objects/s recovering 2024-09-17T07:24:55.710 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:55 smithi100 ceph-mon[29092]: osdmap e498: 8 total, 8 up, 7 in 2024-09-17T07:24:55.711 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:55 smithi100 ceph-mon[39462]: pgmap v1085: 158 pgs: 158 active+clean; 106 MiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 1005 B/s rd, 0 B/s wr, 1.26k op/s; 21 B/s, 1 objects/s recovering 2024-09-17T07:24:55.711 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:55 smithi100 ceph-mon[39462]: osdmap e498: 8 total, 8 up, 7 in 2024-09-17T07:24:55.757 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:24:55.780 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:55.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:55.781 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:24:55.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:55 smithi135 ceph-mon[33238]: pgmap v1085: 158 pgs: 158 active+clean; 106 MiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 1005 B/s rd, 0 B/s wr, 1.26k op/s; 21 B/s, 1 objects/s recovering 2024-09-17T07:24:55.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:55 smithi135 ceph-mon[33238]: osdmap e498: 8 total, 8 up, 7 in 2024-09-17T07:24:55.881 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:24:55.903 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:55.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:55.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:24:56.005 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:24:56.028 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:56.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:56.029 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:24:56.126 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-17T07:24:56.126 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:24:55 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:24:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:24:56.129 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:24:56.190 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:56.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:56.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:24:56.292 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:24:56.314 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:56.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:56.315 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:24:56.329 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:24:56.417 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:24:56.441 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:56.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:56.442 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:24:56.543 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:24:56.564 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:56.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:56.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:24:56.621 INFO:teuthology.orchestra.run.smithi100.stderr:noscrub is unset 2024-09-17T07:24:56.666 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:24:56.690 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:56.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:56.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:24:56.761 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:56 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2653010554' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:24:56.792 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:24:56.814 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:56.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:56.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:24:56.863 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_4 2024-09-17T07:24:56.863 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd pool rm unique_pool_4 unique_pool_4 --yes-i-really-really-mean-it 2024-09-17T07:24:56.876 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:56 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2653010554' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:24:56.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:56 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2653010554' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:24:56.916 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:24:56.937 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:56.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:56.938 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:24:57.038 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:24:57.060 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:57.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:57.061 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:24:57.162 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:24:57.185 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:57.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:57.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:24:57.287 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:24:57.310 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:57.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:57.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:24:57.412 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:24:57.435 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:57.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:57.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:24:57.536 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:24:57.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:57.560 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:57.560 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:24:57.637 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:24:57.660 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:57 smithi135 ceph-mon[33238]: pgmap v1087: 158 pgs: 158 active+clean; 2.5 MiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 1006 B/s rd, 0 B/s wr, 538 op/s 2024-09-17T07:24:57.660 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:57 smithi135 ceph-mon[33238]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:24:57.660 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:57 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2653010554' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T07:24:57.660 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:57 smithi135 ceph-mon[33238]: osdmap e499: 8 total, 8 up, 7 in 2024-09-17T07:24:57.660 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:57 smithi135 ceph-mon[33238]: 4.b scrub starts 2024-09-17T07:24:57.660 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:57 smithi135 ceph-mon[33238]: 4.b scrub ok 2024-09-17T07:24:57.661 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:24:57.683 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:57.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:57.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:24:57.764 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:57 smithi100 ceph-mon[39462]: pgmap v1087: 158 pgs: 158 active+clean; 2.5 MiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 1006 B/s rd, 0 B/s wr, 538 op/s 2024-09-17T07:24:57.764 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:57 smithi100 ceph-mon[39462]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:24:57.764 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:57 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2653010554' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T07:24:57.764 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:57 smithi100 ceph-mon[39462]: osdmap e499: 8 total, 8 up, 7 in 2024-09-17T07:24:57.764 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:57 smithi100 ceph-mon[39462]: 4.b scrub starts 2024-09-17T07:24:57.764 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:57 smithi100 ceph-mon[39462]: 4.b scrub ok 2024-09-17T07:24:57.764 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:57 smithi100 ceph-mon[29092]: pgmap v1087: 158 pgs: 158 active+clean; 2.5 MiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 1006 B/s rd, 0 B/s wr, 538 op/s 2024-09-17T07:24:57.765 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:57 smithi100 ceph-mon[29092]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T07:24:57.765 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:57 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2653010554' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T07:24:57.765 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:57 smithi100 ceph-mon[29092]: osdmap e499: 8 total, 8 up, 7 in 2024-09-17T07:24:57.765 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:57 smithi100 ceph-mon[29092]: 4.b scrub starts 2024-09-17T07:24:57.765 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:57 smithi100 ceph-mon[29092]: 4.b scrub ok 2024-09-17T07:24:57.784 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:24:57.805 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:57.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:57.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:24:57.907 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:24:57.916 INFO:teuthology.orchestra.run.smithi135.stdout:ERROR: (22) Invalid argument 2024-09-17T07:24:57.916 INFO:teuthology.orchestra.run.smithi135.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:24:57.931 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:57.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:57.932 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:24:58.033 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:24:58.055 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:58.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:58.056 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:24:58.157 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:24:58.180 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:58.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:58.181 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:24:58.251 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:24:58.252 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:24:58.256 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1: {} 2024-09-17T07:24:58.256 INFO:teuthology.orchestra.run.smithi100.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:24:58.272 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:24:58.272 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:24:58.281 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:24:58.288 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:24:58.288 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:24:58.302 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:58.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:58.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:24:58.305 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:24:58.305 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:24:58.321 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:24:58.321 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:24:58.338 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:24:58.338 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:24:58.353 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:24:58.354 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:24:58.404 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:24:58.426 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:58.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:58.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:24:58.528 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:24:58.549 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:58.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:58.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:24:58.585 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-17T07:24:58.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:58 smithi100 ceph-mon[39462]: 5.5 scrub starts 2024-09-17T07:24:58.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:58 smithi100 ceph-mon[39462]: 5.5 scrub ok 2024-09-17T07:24:58.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:58 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2117730033' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:24:58.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:58 smithi100 ceph-mon[29092]: 5.5 scrub starts 2024-09-17T07:24:58.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:58 smithi100 ceph-mon[29092]: 5.5 scrub ok 2024-09-17T07:24:58.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:58 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2117730033' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:24:58.651 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:24:58.691 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:58.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:58.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:24:58.792 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:24:58.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:58 smithi135 ceph-mon[33238]: 5.5 scrub starts 2024-09-17T07:24:58.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:58 smithi135 ceph-mon[33238]: 5.5 scrub ok 2024-09-17T07:24:58.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:58 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2117730033' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:24:58.815 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:58.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:58.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:24:58.828 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:24:58.917 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:24:58.940 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:58.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:58.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:24:59.042 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:24:59.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:59.065 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:59.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:24:59.166 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:24:59.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:59.189 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:59.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:24:59.290 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:24:59.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:59.313 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:59.313 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:24:59.414 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:24:59.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:59.436 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:59.436 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:24:59.537 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:24:59.543 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:24:59 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:24:59.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:59 smithi135 ceph-mon[33238]: pgmap v1089: 158 pgs: 158 active+clean; 477 KiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 392 op/s 2024-09-17T07:24:59.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:59 smithi135 ceph-mon[33238]: 5.7 scrub starts 2024-09-17T07:24:59.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:24:59 smithi135 ceph-mon[33238]: 5.7 scrub ok 2024-09-17T07:24:59.558 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:59.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:59.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:24:59.660 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:24:59.681 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:59.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:59.682 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:24:59.783 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:24:59.806 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:24:59.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:59.807 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:24:59.876 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:59 smithi100 ceph-mon[39462]: pgmap v1089: 158 pgs: 158 active+clean; 477 KiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 392 op/s 2024-09-17T07:24:59.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:59 smithi100 ceph-mon[39462]: 5.7 scrub starts 2024-09-17T07:24:59.879 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:24:59 smithi100 ceph-mon[39462]: 5.7 scrub ok 2024-09-17T07:24:59.879 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:59 smithi100 ceph-mon[29092]: pgmap v1089: 158 pgs: 158 active+clean; 477 KiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 392 op/s 2024-09-17T07:24:59.879 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:59 smithi100 ceph-mon[29092]: 5.7 scrub starts 2024-09-17T07:24:59.879 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:24:59 smithi100 ceph-mon[29092]: 5.7 scrub ok 2024-09-17T07:24:59.907 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:24:59.929 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:24:59.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:24:59.931 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:25:00.032 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:25:00.054 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:25:00.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:25:00.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:25:00.156 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:25:00.177 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:25:00.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:25:00.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:25:00.279 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:25:00.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:25:00.302 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:25:00.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:25:00.403 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:25:00.425 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:25:00.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:25:00.426 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:25:00.438 INFO:teuthology.orchestra.run.smithi135.stdout:{ 2024-09-17T07:25:00.438 INFO:teuthology.orchestra.run.smithi135.stdout: "ops": [], 2024-09-17T07:25:00.438 INFO:teuthology.orchestra.run.smithi135.stdout: "num_ops": 0 2024-09-17T07:25:00.438 INFO:teuthology.orchestra.run.smithi135.stdout:} 2024-09-17T07:25:00.527 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:25:00.549 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:25:00.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:25:00.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:25:00.584 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:00 smithi100 ceph-mon[29092]: 4.18 scrub starts 2024-09-17T07:25:00.584 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:00 smithi100 ceph-mon[29092]: 4.18 scrub ok 2024-09-17T07:25:00.584 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:00 smithi100 ceph-mon[29092]: 6.a scrub starts 2024-09-17T07:25:00.584 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:00 smithi100 ceph-mon[29092]: 6.a scrub ok 2024-09-17T07:25:00.585 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:00 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:25:00.585 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:00 smithi100 ceph-mon[29092]: 5.9 scrub starts 2024-09-17T07:25:00.585 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:00 smithi100 ceph-mon[29092]: 5.9 scrub ok 2024-09-17T07:25:00.585 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:00 smithi100 ceph-mon[29092]: osdmap e500: 8 total, 8 up, 7 in 2024-09-17T07:25:00.585 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:00 smithi100 ceph-mon[39462]: 4.18 scrub starts 2024-09-17T07:25:00.585 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:00 smithi100 ceph-mon[39462]: 4.18 scrub ok 2024-09-17T07:25:00.585 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:00 smithi100 ceph-mon[39462]: 6.a scrub starts 2024-09-17T07:25:00.585 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:00 smithi100 ceph-mon[39462]: 6.a scrub ok 2024-09-17T07:25:00.585 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:00 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:25:00.585 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:00 smithi100 ceph-mon[39462]: 5.9 scrub starts 2024-09-17T07:25:00.586 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:00 smithi100 ceph-mon[39462]: 5.9 scrub ok 2024-09-17T07:25:00.586 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:00 smithi100 ceph-mon[39462]: osdmap e500: 8 total, 8 up, 7 in 2024-09-17T07:25:00.652 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:25:00.674 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:25:00.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:25:00.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:25:00.775 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:25:00.792 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:00 smithi135 ceph-mon[33238]: 4.18 scrub starts 2024-09-17T07:25:00.792 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:00 smithi135 ceph-mon[33238]: 4.18 scrub ok 2024-09-17T07:25:00.792 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:00 smithi135 ceph-mon[33238]: 6.a scrub starts 2024-09-17T07:25:00.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:00 smithi135 ceph-mon[33238]: 6.a scrub ok 2024-09-17T07:25:00.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:00 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:25:00.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:00 smithi135 ceph-mon[33238]: 5.9 scrub starts 2024-09-17T07:25:00.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:00 smithi135 ceph-mon[33238]: 5.9 scrub ok 2024-09-17T07:25:00.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:00 smithi135 ceph-mon[33238]: osdmap e500: 8 total, 8 up, 7 in 2024-09-17T07:25:00.797 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:25:00.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:25:00.798 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:25:00.810 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:25:00.899 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:25:00.921 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:25:00.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:25:00.922 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:25:01.022 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:25:01.043 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:25:01.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:25:01.044 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:25:01.145 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:25:01.148 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-17T07:25:01.167 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:25:01.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:25:01.168 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:25:01.269 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:25:01.291 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:25:01.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:25:01.292 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:25:01.347 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:25:01.393 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:25:01.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:25:01.414 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:25:01.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:25:01.475 INFO:teuthology.orchestra.run.smithi100.stderr:reweighted osd.5 to 1 (10000) 2024-09-17T07:25:01.515 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:25:01.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:25:01.537 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:25:01.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:25:01.639 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:25:01.660 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:25:01.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:25:01.661 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:25:01.723 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:01 smithi135 ceph-mon[33238]: pgmap v1090: 158 pgs: 158 active+clean; 477 KiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 279 op/s 2024-09-17T07:25:01.723 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:01 smithi135 ceph-mon[33238]: 5.13 scrub starts 2024-09-17T07:25:01.723 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:01 smithi135 ceph-mon[33238]: 5.13 scrub ok 2024-09-17T07:25:01.723 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:01 smithi135 ceph-mon[33238]: 6.3 scrub starts 2024-09-17T07:25:01.723 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:01 smithi135 ceph-mon[33238]: 6.3 scrub ok 2024-09-17T07:25:01.723 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:01 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/4023006647' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-17T07:25:01.763 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:25:01.784 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:25:01.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:25:01.785 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:25:01.813 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:01 smithi100 ceph-mon[29092]: pgmap v1090: 158 pgs: 158 active+clean; 477 KiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 279 op/s 2024-09-17T07:25:01.813 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:01 smithi100 ceph-mon[29092]: 5.13 scrub starts 2024-09-17T07:25:01.813 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:01 smithi100 ceph-mon[29092]: 5.13 scrub ok 2024-09-17T07:25:01.813 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:01 smithi100 ceph-mon[29092]: 6.3 scrub starts 2024-09-17T07:25:01.813 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:01 smithi100 ceph-mon[29092]: 6.3 scrub ok 2024-09-17T07:25:01.813 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:01 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/4023006647' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-17T07:25:01.814 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:01 smithi100 ceph-mon[39462]: pgmap v1090: 158 pgs: 158 active+clean; 477 KiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 279 op/s 2024-09-17T07:25:01.814 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:01 smithi100 ceph-mon[39462]: 5.13 scrub starts 2024-09-17T07:25:01.814 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:01 smithi100 ceph-mon[39462]: 5.13 scrub ok 2024-09-17T07:25:01.814 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:01 smithi100 ceph-mon[39462]: 6.3 scrub starts 2024-09-17T07:25:01.814 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:01 smithi100 ceph-mon[39462]: 6.3 scrub ok 2024-09-17T07:25:01.814 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:01 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/4023006647' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-17T07:25:01.887 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:25:01.908 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:25:01.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:25:01.909 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:25:01.936 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T07:25:02.010 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:25:02.031 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:25:02.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:25:02.032 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:25:02.133 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:25:02.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:25:02.156 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:25:02.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:25:02.257 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:25:02.278 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:25:02.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:25:02.279 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:25:02.347 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd reweight 6 1 2024-09-17T07:25:02.353 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-17T07:25:02.381 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:25:02.412 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:25:02.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:25:02.413 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:25:02.513 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:25:02.535 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:25:02.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:25:02.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:25:02.583 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:02 smithi100 ceph-mon[29092]: 4.8 scrub starts 2024-09-17T07:25:02.583 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:02 smithi100 ceph-mon[29092]: 4.8 scrub ok 2024-09-17T07:25:02.583 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:02 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/4023006647' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-17T07:25:02.583 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:02 smithi100 ceph-mon[29092]: osdmap e501: 8 total, 8 up, 7 in 2024-09-17T07:25:02.583 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:02 smithi100 ceph-mon[29092]: pgmap v1093: 158 pgs: 158 active+clean; 477 KiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 6 op/s 2024-09-17T07:25:02.584 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:02 smithi100 ceph-mon[29092]: 5.1f scrub starts 2024-09-17T07:25:02.584 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:02 smithi100 ceph-mon[29092]: 5.1f scrub ok 2024-09-17T07:25:02.584 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:02 smithi100 ceph-mon[29092]: 2.1 scrub starts 2024-09-17T07:25:02.584 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:02 smithi100 ceph-mon[29092]: 2.1 scrub ok 2024-09-17T07:25:02.584 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:02 smithi100 ceph-mon[39462]: 4.8 scrub starts 2024-09-17T07:25:02.584 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:02 smithi100 ceph-mon[39462]: 4.8 scrub ok 2024-09-17T07:25:02.584 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:02 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/4023006647' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-17T07:25:02.584 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:02 smithi100 ceph-mon[39462]: osdmap e501: 8 total, 8 up, 7 in 2024-09-17T07:25:02.584 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:02 smithi100 ceph-mon[39462]: pgmap v1093: 158 pgs: 158 active+clean; 477 KiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 6 op/s 2024-09-17T07:25:02.584 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:02 smithi100 ceph-mon[39462]: 5.1f scrub starts 2024-09-17T07:25:02.584 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:02 smithi100 ceph-mon[39462]: 5.1f scrub ok 2024-09-17T07:25:02.585 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:02 smithi100 ceph-mon[39462]: 2.1 scrub starts 2024-09-17T07:25:02.585 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:02 smithi100 ceph-mon[39462]: 2.1 scrub ok 2024-09-17T07:25:02.637 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:25:02.658 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:25:02.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:25:02.659 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:25:02.744 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:02 smithi135 ceph-mon[33238]: 4.8 scrub starts 2024-09-17T07:25:02.744 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:02 smithi135 ceph-mon[33238]: 4.8 scrub ok 2024-09-17T07:25:02.744 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:02 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/4023006647' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-17T07:25:02.745 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:02 smithi135 ceph-mon[33238]: osdmap e501: 8 total, 8 up, 7 in 2024-09-17T07:25:02.745 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:02 smithi135 ceph-mon[33238]: pgmap v1093: 158 pgs: 158 active+clean; 477 KiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 6 op/s 2024-09-17T07:25:02.745 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:02 smithi135 ceph-mon[33238]: 5.1f scrub starts 2024-09-17T07:25:02.745 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:02 smithi135 ceph-mon[33238]: 5.1f scrub ok 2024-09-17T07:25:02.745 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:02 smithi135 ceph-mon[33238]: 2.1 scrub starts 2024-09-17T07:25:02.745 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:02 smithi135 ceph-mon[33238]: 2.1 scrub ok 2024-09-17T07:25:02.760 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:25:02.782 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:25:02.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:25:02.783 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:25:02.884 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:25:02.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:25:02.907 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:25:02.907 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:25:02.997 INFO:teuthology.orchestra.run.smithi135.stdout:{ 2024-09-17T07:25:02.998 INFO:teuthology.orchestra.run.smithi135.stdout: "ops": [], 2024-09-17T07:25:02.998 INFO:teuthology.orchestra.run.smithi135.stdout: "complaint_time": 30, 2024-09-17T07:25:02.998 INFO:teuthology.orchestra.run.smithi135.stdout: "num_blocked_ops": 0 2024-09-17T07:25:02.998 INFO:teuthology.orchestra.run.smithi135.stdout:} 2024-09-17T07:25:03.008 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:25:03.031 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:25:03.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:25:03.032 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:25:03.133 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:25:03.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:25:03.155 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:25:03.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:25:03.256 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:25:03.278 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:25:03.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:25:03.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:25:03.380 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:25:03.401 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:25:03.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:25:03.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:25:03.503 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:25:03.525 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:25:03.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:25:03.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:25:03.602 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-17T07:25:03.623 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:03 smithi100 ceph-mon[29092]: 4.10 scrub starts 2024-09-17T07:25:03.623 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:03 smithi100 ceph-mon[29092]: 4.10 scrub ok 2024-09-17T07:25:03.623 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:03 smithi100 ceph-mon[29092]: 6.11 scrub starts 2024-09-17T07:25:03.623 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:03 smithi100 ceph-mon[29092]: 6.11 scrub ok 2024-09-17T07:25:03.623 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:03 smithi100 ceph-mon[39462]: 4.10 scrub starts 2024-09-17T07:25:03.623 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:03 smithi100 ceph-mon[39462]: 4.10 scrub ok 2024-09-17T07:25:03.623 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:03 smithi100 ceph-mon[39462]: 6.11 scrub starts 2024-09-17T07:25:03.623 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:03 smithi100 ceph-mon[39462]: 6.11 scrub ok 2024-09-17T07:25:03.627 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:25:03.633 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:25:03.634 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:25:03.659 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:25:03.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:25:03.660 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:25:03.761 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:25:03.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:25:03.784 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:25:03.784 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:25:03.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:03 smithi135 ceph-mon[33238]: 4.10 scrub starts 2024-09-17T07:25:03.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:03 smithi135 ceph-mon[33238]: 4.10 scrub ok 2024-09-17T07:25:03.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:03 smithi135 ceph-mon[33238]: 6.11 scrub starts 2024-09-17T07:25:03.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:03 smithi135 ceph-mon[33238]: 6.11 scrub ok 2024-09-17T07:25:03.804 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:25:03.884 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:25:03.907 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:25:03.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:25:03.908 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:25:04.009 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:25:04.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:25:04.032 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:25:04.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:25:04.133 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:25:04.157 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:25:04.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:25:04.158 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:25:04.258 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:25:04.281 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:25:04.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:25:04.282 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.2' 2024-09-17T07:25:04.383 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:25:04.404 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:25:04.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:25:04.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:25:04.499 INFO:teuthology.orchestra.run.smithi100.stderr:pool 'unique_pool_4' removed 2024-09-17T07:25:04.506 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:25:04.517 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:04 smithi100 ceph-mon[29092]: pgmap v1094: 158 pgs: 158 active+clean; 477 KiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 727 B/s rd, 0 op/s 2024-09-17T07:25:04.517 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:04 smithi100 ceph-mon[29092]: 5.11 scrub starts 2024-09-17T07:25:04.517 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:04 smithi100 ceph-mon[29092]: 5.11 scrub ok 2024-09-17T07:25:04.517 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:04 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:25:04.517 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:04 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/3700169077' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-17T07:25:04.517 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:04 smithi100 ceph-mon[39462]: pgmap v1094: 158 pgs: 158 active+clean; 477 KiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 727 B/s rd, 0 op/s 2024-09-17T07:25:04.518 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:04 smithi100 ceph-mon[39462]: 5.11 scrub starts 2024-09-17T07:25:04.518 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:04 smithi100 ceph-mon[39462]: 5.11 scrub ok 2024-09-17T07:25:04.518 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:04 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:25:04.518 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:04 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/3700169077' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-17T07:25:04.533 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:25:04.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:25:04.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.0' 2024-09-17T07:25:04.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:04 smithi135 ceph-mon[33238]: pgmap v1094: 158 pgs: 158 active+clean; 477 KiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 727 B/s rd, 0 op/s 2024-09-17T07:25:04.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:04 smithi135 ceph-mon[33238]: 5.11 scrub starts 2024-09-17T07:25:04.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:04 smithi135 ceph-mon[33238]: 5.11 scrub ok 2024-09-17T07:25:04.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:04 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:25:04.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:04 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/3700169077' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-17T07:25:04.636 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:25:04.661 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:25:04.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:25:04.662 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:25:04.763 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:25:04.785 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:25:04.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:25:04.786 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:25:04.887 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:25:04.915 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:25:04.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:25:04.916 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:25:05.017 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:25:05.038 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:25:05.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:25:05.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.4' 2024-09-17T07:25:05.140 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:25:05.161 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:25:05.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:25:05.162 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:25:05.263 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:25:05.285 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:25:05.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:25:05.286 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:25:05.387 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:25:05.408 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:25:05.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:25:05.409 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:25:05.478 INFO:teuthology.orchestra.run.smithi135.stdout:{ 2024-09-17T07:25:05.478 INFO:teuthology.orchestra.run.smithi135.stdout: "size": 20, 2024-09-17T07:25:05.479 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 600, 2024-09-17T07:25:05.479 INFO:teuthology.orchestra.run.smithi135.stdout: "ops": [ 2024-09-17T07:25:05.479 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.479 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.15648.0:217457 10.23 10:c41c5a9c:::benchmark_data_smithi100_155039_object94109:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-09-17T07:25:05.479 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:19:15.315406+0000", 2024-09-17T07:25:05.479 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 350.15495665899999, 2024-09-17T07:25:05.479 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.99254916699999995, 2024-09-17T07:25:05.479 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:25:05.479 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:25:05.479 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:25:05.479 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.15648", 2024-09-17T07:25:05.479 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/1684940078", 2024-09-17T07:25:05.479 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 217457 2024-09-17T07:25:05.480 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.480 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:25:05.480 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.480 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:25:05.480 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.315406+0000", 2024-09-17T07:25:05.480 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:25:05.480 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.480 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.480 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:25:05.480 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.315406+0000", 2024-09-17T07:25:05.480 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:25:05.480 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.481 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.481 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:25:05.481 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.315404+0000", 2024-09-17T07:25:05.481 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:25:05.481 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.481 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.481 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:25:05.481 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.315414+0000", 2024-09-17T07:25:05.481 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.7149999999999993e-06 2024-09-17T07:25:05.481 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.481 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.481 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:25:05.481 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.315423+0000", 2024-09-17T07:25:05.482 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.3810000000000005e-06 2024-09-17T07:25:05.482 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.482 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.482 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:25:05.482 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.315430+0000", 2024-09-17T07:25:05.482 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.5030000000000002e-06 2024-09-17T07:25:05.482 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.482 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.482 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:25:05.482 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.315505+0000", 2024-09-17T07:25:05.482 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.5263000000000001e-05 2024-09-17T07:25:05.482 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.482 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.483 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:25:05.483 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.315514+0000", 2024-09-17T07:25:05.483 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.9080000000000001e-06 2024-09-17T07:25:05.483 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.483 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.483 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:25:05.483 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:16.300779+0000", 2024-09-17T07:25:05.483 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.98526500800000005 2024-09-17T07:25:05.483 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.483 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.483 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:25:05.483 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:16.300899+0000", 2024-09-17T07:25:05.484 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000119541 2024-09-17T07:25:05.484 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.484 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.484 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 4,5", 2024-09-17T07:25:05.484 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:16.301115+0000", 2024-09-17T07:25:05.484 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000216648 2024-09-17T07:25:05.484 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.484 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.484 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:25:05.484 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:16.306840+0000", 2024-09-17T07:25:05.484 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0057248029999999997 2024-09-17T07:25:05.484 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.484 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.485 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:25:05.485 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:16.306961+0000", 2024-09-17T07:25:05.485 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000120773 2024-09-17T07:25:05.485 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.485 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.485 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:25:05.485 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:16.307876+0000", 2024-09-17T07:25:05.485 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00091544599999999999 2024-09-17T07:25:05.485 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.485 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.485 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:25:05.485 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:16.307922+0000", 2024-09-17T07:25:05.485 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.5238999999999998e-05 2024-09-17T07:25:05.486 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.486 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.486 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:25:05.486 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:16.307955+0000", 2024-09-17T07:25:05.486 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.3229000000000003e-05 2024-09-17T07:25:05.486 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:25:05.486 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:25:05.486 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:25:05.486 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.486 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.486 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.15648.0:217492 10.23 10:c79ae830:::benchmark_data_smithi100_155039_object94144:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-09-17T07:25:05.486 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:19:15.333488+0000", 2024-09-17T07:25:05.486 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 350.13687392100002, 2024-09-17T07:25:05.486 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.983869879, 2024-09-17T07:25:05.487 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:25:05.487 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:25:05.487 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:25:05.487 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.15648", 2024-09-17T07:25:05.487 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/1684940078", 2024-09-17T07:25:05.487 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 217492 2024-09-17T07:25:05.487 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.487 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:25:05.487 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.487 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:25:05.487 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.333488+0000", 2024-09-17T07:25:05.487 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:25:05.487 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.488 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.488 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:25:05.488 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.333488+0000", 2024-09-17T07:25:05.488 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:25:05.488 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.488 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.488 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:25:05.488 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.333488+0000", 2024-09-17T07:25:05.488 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.999999 2024-09-17T07:25:05.488 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.488 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.488 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:25:05.488 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.333492+0000", 2024-09-17T07:25:05.489 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.8409999999999999e-06 2024-09-17T07:25:05.489 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.489 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.489 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:25:05.489 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.333514+0000", 2024-09-17T07:25:05.489 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.1447e-05 2024-09-17T07:25:05.489 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.489 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.489 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:25:05.489 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.333521+0000", 2024-09-17T07:25:05.489 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.9720000000000003e-06 2024-09-17T07:25:05.489 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.489 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.489 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:25:05.490 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.333558+0000", 2024-09-17T07:25:05.490 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.6829000000000003e-05 2024-09-17T07:25:05.490 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.490 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.490 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:25:05.490 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.333565+0000", 2024-09-17T07:25:05.490 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.6920000000000002e-06 2024-09-17T07:25:05.490 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.490 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.490 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:25:05.490 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:16.307998+0000", 2024-09-17T07:25:05.490 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.97443299800000005 2024-09-17T07:25:05.490 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.491 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.491 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:25:05.491 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:16.308089+0000", 2024-09-17T07:25:05.491 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.1129000000000001e-05 2024-09-17T07:25:05.491 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.491 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.491 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 4,5", 2024-09-17T07:25:05.491 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:16.308332+0000", 2024-09-17T07:25:05.491 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00024259499999999999 2024-09-17T07:25:05.491 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.491 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.491 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:25:05.491 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:16.315310+0000", 2024-09-17T07:25:05.491 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0069775879999999998 2024-09-17T07:25:05.492 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.492 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.492 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:25:05.492 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:16.317095+0000", 2024-09-17T07:25:05.492 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.001785317 2024-09-17T07:25:05.492 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.492 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.492 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:25:05.492 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:16.317270+0000", 2024-09-17T07:25:05.492 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00017467200000000001 2024-09-17T07:25:05.492 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.492 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.492 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:25:05.493 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:16.317319+0000", 2024-09-17T07:25:05.493 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.9243999999999998e-05 2024-09-17T07:25:05.493 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.493 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.493 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:25:05.493 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:16.317358+0000", 2024-09-17T07:25:05.493 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.9338999999999997e-05 2024-09-17T07:25:05.493 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:25:05.493 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:25:05.493 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:25:05.493 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.493 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.493 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.15648.0:217513 10.23 10:c7aa9836:::benchmark_data_smithi100_155039_object94165:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-09-17T07:25:05.494 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:19:15.345458+0000", 2024-09-17T07:25:05.494 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 350.12490459899999, 2024-09-17T07:25:05.494 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.97663500599999997, 2024-09-17T07:25:05.494 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:25:05.494 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:25:05.494 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:25:05.494 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.15648", 2024-09-17T07:25:05.494 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/1684940078", 2024-09-17T07:25:05.494 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 217513 2024-09-17T07:25:05.494 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.494 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:25:05.494 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.494 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:25:05.495 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.345458+0000", 2024-09-17T07:25:05.495 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:25:05.495 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.495 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.495 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:25:05.495 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.345458+0000", 2024-09-17T07:25:05.495 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:25:05.495 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.495 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.495 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:25:05.495 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.345456+0000", 2024-09-17T07:25:05.495 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:25:05.495 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.495 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.496 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:25:05.496 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.345463+0000", 2024-09-17T07:25:05.496 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.5930000000000002e-06 2024-09-17T07:25:05.496 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.496 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.496 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:25:05.496 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.345474+0000", 2024-09-17T07:25:05.496 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0834e-05 2024-09-17T07:25:05.496 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.496 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.496 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:25:05.497 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.345481+0000", 2024-09-17T07:25:05.497 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.5209999999999997e-06 2024-09-17T07:25:05.497 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.497 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.497 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:25:05.497 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.345531+0000", 2024-09-17T07:25:05.497 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.9969000000000002e-05 2024-09-17T07:25:05.497 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.497 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.497 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:25:05.497 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:15.345541+0000", 2024-09-17T07:25:05.497 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.8779999999999993e-06 2024-09-17T07:25:05.497 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.498 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.498 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:25:05.498 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:16.310592+0000", 2024-09-17T07:25:05.498 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.96505050000000003 2024-09-17T07:25:05.498 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.498 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.498 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:25:05.498 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:16.310690+0000", 2024-09-17T07:25:05.498 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.8884999999999999e-05 2024-09-17T07:25:05.498 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.498 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.498 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 4,5", 2024-09-17T07:25:05.498 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:16.310939+0000", 2024-09-17T07:25:05.498 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00024887800000000001 2024-09-17T07:25:05.499 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.499 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.499 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:25:05.499 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:16.317059+0000", 2024-09-17T07:25:05.499 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0061198019999999997 2024-09-17T07:25:05.499 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.499 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.499 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:25:05.499 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:16.321912+0000", 2024-09-17T07:25:05.499 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0048527900000000001 2024-09-17T07:25:05.499 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.499 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.499 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:25:05.500 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:16.322008+0000", 2024-09-17T07:25:05.500 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.5799999999999998e-05 2024-09-17T07:25:05.500 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.500 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.500 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:25:05.500 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:16.322057+0000", 2024-09-17T07:25:05.500 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.9011000000000002e-05 2024-09-17T07:25:05.500 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.500 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.500 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:25:05.500 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:19:16.322093+0000", 2024-09-17T07:25:05.500 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.6000000000000001e-05 2024-09-17T07:25:05.500 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:25:05.501 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:25:05.501 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:25:05.501 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.501 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.501 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.25811.0:214074 11.5 11.380e0335 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e433)", 2024-09-17T07:25:05.501 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:22:12.573459+0000", 2024-09-17T07:25:05.501 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 172.89690297199999, 2024-09-17T07:25:05.501 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0037095039999999, 2024-09-17T07:25:05.501 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:25:05.501 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "reached pg", 2024-09-17T07:25:05.501 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:25:05.501 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.25811", 2024-09-17T07:25:05.501 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/173867468", 2024-09-17T07:25:05.502 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 214074 2024-09-17T07:25:05.502 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.502 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:25:05.502 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.502 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:25:05.502 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:22:12.573459+0000", 2024-09-17T07:25:05.502 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:25:05.502 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.502 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.502 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:25:05.502 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:22:12.573459+0000", 2024-09-17T07:25:05.502 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:25:05.503 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.503 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.503 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:25:05.503 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:22:12.573458+0000", 2024-09-17T07:25:05.503 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:25:05.503 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.503 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.503 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:25:05.503 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:22:12.573464+0000", 2024-09-17T07:25:05.503 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.852e-06 2024-09-17T07:25:05.503 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.503 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.503 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:25:05.504 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:22:12.573471+0000", 2024-09-17T07:25:05.504 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.627e-06 2024-09-17T07:25:05.504 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.504 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.504 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:25:05.504 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:22:12.573479+0000", 2024-09-17T07:25:05.504 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.6190000000000002e-06 2024-09-17T07:25:05.504 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.504 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.504 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:25:05.504 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:22:12.573892+0000", 2024-09-17T07:25:05.504 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00041341400000000002 2024-09-17T07:25:05.504 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.505 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.505 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:25:05.505 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:22:12.573900+0000", 2024-09-17T07:25:05.505 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.5530000000000004e-06 2024-09-17T07:25:05.505 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.505 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.505 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:25:05.505 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:22:13.577158+0000", 2024-09-17T07:25:05.505 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.003258245 2024-09-17T07:25:05.505 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.505 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.505 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:25:05.505 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:22:13.577169+0000", 2024-09-17T07:25:05.506 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0788000000000001e-05 2024-09-17T07:25:05.506 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:25:05.506 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:25:05.506 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:25:05.506 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.506 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.506 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.25811.0:214080 11.5 11.5b58e175 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e433)", 2024-09-17T07:25:05.506 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:22:12.573486+0000", 2024-09-17T07:25:05.506 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 172.89687641800001, 2024-09-17T07:25:05.506 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.003711472, 2024-09-17T07:25:05.506 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:25:05.506 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "reached pg", 2024-09-17T07:25:05.506 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:25:05.507 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.25811", 2024-09-17T07:25:05.507 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/173867468", 2024-09-17T07:25:05.507 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 214080 2024-09-17T07:25:05.507 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.507 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:25:05.507 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.507 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:25:05.507 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:22:12.573486+0000", 2024-09-17T07:25:05.507 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:25:05.507 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.507 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.507 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:25:05.507 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:22:12.573486+0000", 2024-09-17T07:25:05.508 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:25:05.508 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.508 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.508 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:25:05.508 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:22:12.573485+0000", 2024-09-17T07:25:05.508 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999995 2024-09-17T07:25:05.508 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.508 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.508 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:25:05.508 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:22:12.573488+0000", 2024-09-17T07:25:05.508 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.4150000000000002e-06 2024-09-17T07:25:05.508 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.508 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.509 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:25:05.509 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:22:12.573493+0000", 2024-09-17T07:25:05.509 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.4929999999999998e-06 2024-09-17T07:25:05.509 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.509 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.509 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:25:05.509 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:22:12.573498+0000", 2024-09-17T07:25:05.509 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.9060000000000001e-06 2024-09-17T07:25:05.509 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.509 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.509 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:25:05.509 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:22:12.573934+0000", 2024-09-17T07:25:05.509 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00043599900000000001 2024-09-17T07:25:05.510 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.510 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.510 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:25:05.510 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:22:12.573941+0000", 2024-09-17T07:25:05.510 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.9380000000000003e-06 2024-09-17T07:25:05.510 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.510 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.510 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:25:05.510 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:22:13.577190+0000", 2024-09-17T07:25:05.511 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.003248948 2024-09-17T07:25:05.511 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.511 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.511 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:25:05.511 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:22:13.577197+0000", 2024-09-17T07:25:05.511 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.1380000000000001e-06 2024-09-17T07:25:05.511 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:25:05.511 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:25:05.511 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:25:05.511 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.511 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.511 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.25811.0:299895 11.5 11:a876e846:::benchmark_data_smithi100_180298_object127646:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e485)", 2024-09-17T07:25:05.511 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:24:17.472957+0000", 2024-09-17T07:25:05.512 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 47.997404848999999, 2024-09-17T07:25:05.512 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.053248483, 2024-09-17T07:25:05.512 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:25:05.512 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:25:05.512 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:25:05.512 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.25811", 2024-09-17T07:25:05.512 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/173867468", 2024-09-17T07:25:05.512 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 299895 2024-09-17T07:25:05.512 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.512 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:25:05.512 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.512 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:25:05.513 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.472957+0000", 2024-09-17T07:25:05.513 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:25:05.513 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.513 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.513 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:25:05.513 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.472957+0000", 2024-09-17T07:25:05.513 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:25:05.513 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.513 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.513 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:25:05.513 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.472956+0000", 2024-09-17T07:25:05.513 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:25:05.513 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.514 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.514 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:25:05.514 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.472963+0000", 2024-09-17T07:25:05.514 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.267e-06 2024-09-17T07:25:05.514 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.514 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.514 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:25:05.514 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.472979+0000", 2024-09-17T07:25:05.514 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.5936000000000001e-05 2024-09-17T07:25:05.514 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.514 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.514 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:25:05.515 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.472988+0000", 2024-09-17T07:25:05.515 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.5080000000000005e-06 2024-09-17T07:25:05.515 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.515 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.515 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:25:05.515 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.473125+0000", 2024-09-17T07:25:05.515 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00013736999999999999 2024-09-17T07:25:05.515 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.515 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.515 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:25:05.515 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.473135+0000", 2024-09-17T07:25:05.515 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0603e-05 2024-09-17T07:25:05.515 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.516 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.516 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:25:05.516 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.514815+0000", 2024-09-17T07:25:05.516 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0416792720000001 2024-09-17T07:25:05.516 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.516 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.516 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:25:05.516 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.514965+0000", 2024-09-17T07:25:05.516 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000150736 2024-09-17T07:25:05.516 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.516 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.516 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 5,6", 2024-09-17T07:25:05.516 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.515383+0000", 2024-09-17T07:25:05.517 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00041792799999999997 2024-09-17T07:25:05.517 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.517 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.517 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:25:05.517 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.521641+0000", 2024-09-17T07:25:05.517 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0062572089999999997 2024-09-17T07:25:05.517 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.517 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.517 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:25:05.517 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.521653+0000", 2024-09-17T07:25:05.517 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.2165e-05 2024-09-17T07:25:05.517 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.517 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.518 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:25:05.518 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.526112+0000", 2024-09-17T07:25:05.518 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0044587489999999997 2024-09-17T07:25:05.518 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.518 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.518 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:25:05.518 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.526163+0000", 2024-09-17T07:25:05.518 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.1060999999999998e-05 2024-09-17T07:25:05.518 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.518 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.518 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:25:05.518 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.526206+0000", 2024-09-17T07:25:05.518 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.3309999999999997e-05 2024-09-17T07:25:05.519 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:25:05.519 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:25:05.519 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:25:05.519 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.519 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.519 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.25811.0:299896 11.5 11:a5f4492b:::benchmark_data_smithi100_180298_object127647:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e485)", 2024-09-17T07:25:05.519 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:24:17.473720+0000", 2024-09-17T07:25:05.519 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 47.996642012000002, 2024-09-17T07:25:05.519 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0547264970000001, 2024-09-17T07:25:05.519 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:25:05.519 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:25:05.519 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:25:05.519 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.25811", 2024-09-17T07:25:05.520 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/173867468", 2024-09-17T07:25:05.520 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 299896 2024-09-17T07:25:05.520 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.520 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:25:05.520 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.520 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:25:05.520 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.473720+0000", 2024-09-17T07:25:05.520 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:25:05.520 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.520 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.520 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:25:05.520 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.473720+0000", 2024-09-17T07:25:05.520 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:25:05.520 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.521 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.521 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:25:05.521 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.473719+0000", 2024-09-17T07:25:05.521 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.999999 2024-09-17T07:25:05.521 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.521 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.521 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:25:05.521 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.473729+0000", 2024-09-17T07:25:05.521 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0066e-05 2024-09-17T07:25:05.521 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.521 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.521 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:25:05.521 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.473740+0000", 2024-09-17T07:25:05.522 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.076e-05 2024-09-17T07:25:05.522 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.522 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.522 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:25:05.522 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.473747+0000", 2024-09-17T07:25:05.522 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.0779999999999998e-06 2024-09-17T07:25:05.522 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.522 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.522 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:25:05.522 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.475379+0000", 2024-09-17T07:25:05.522 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.001631707 2024-09-17T07:25:05.522 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.522 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.522 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:25:05.523 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.475389+0000", 2024-09-17T07:25:05.523 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0068000000000001e-05 2024-09-17T07:25:05.523 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.523 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.523 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:25:05.523 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:25:05.525 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.521704+0000", 2024-09-17T07:25:05.525 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0463154729999999 2024-09-17T07:25:05.525 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.525 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.525 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:25:05.525 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.521856+0000", 2024-09-17T07:25:05.526 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000151865 2024-09-17T07:25:05.526 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.526 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.526 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 5,6", 2024-09-17T07:25:05.526 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.521935+0000", 2024-09-17T07:25:05.526 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.8448999999999994e-05 2024-09-17T07:25:05.526 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.526 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.526 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:25:05.526 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.526066+0000", 2024-09-17T07:25:05.526 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0041309620000000002 2024-09-17T07:25:05.526 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.527 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.527 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:25:05.527 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.526254+0000", 2024-09-17T07:25:05.527 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000188154 2024-09-17T07:25:05.527 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.527 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.527 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:25:05.527 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.528372+0000", 2024-09-17T07:25:05.527 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0021186180000000001 2024-09-17T07:25:05.527 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.527 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.527 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:25:05.527 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.528411+0000", 2024-09-17T07:25:05.528 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.8962999999999999e-05 2024-09-17T07:25:05.528 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.528 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.528 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:25:05.528 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.528447+0000", 2024-09-17T07:25:05.528 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.5410000000000001e-05 2024-09-17T07:25:05.528 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:25:05.528 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:25:05.528 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:25:05.528 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.528 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.528 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.25811.0:299904 11.5 11:ac13da69:::benchmark_data_smithi100_180298_object127655:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e485)", 2024-09-17T07:25:05.528 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:24:17.481191+0000", 2024-09-17T07:25:05.529 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 47.989170833999999, 2024-09-17T07:25:05.529 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0473508119999999, 2024-09-17T07:25:05.529 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:25:05.529 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:25:05.529 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:25:05.529 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.25811", 2024-09-17T07:25:05.529 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/173867468", 2024-09-17T07:25:05.529 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 299904 2024-09-17T07:25:05.529 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.529 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:25:05.529 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.529 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:25:05.529 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.481191+0000", 2024-09-17T07:25:05.530 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:25:05.530 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.530 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.530 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:25:05.530 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.481191+0000", 2024-09-17T07:25:05.530 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:25:05.530 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.530 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.530 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:25:05.530 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.481190+0000", 2024-09-17T07:25:05.530 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999981 2024-09-17T07:25:05.530 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.530 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.530 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:25:05.531 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.481197+0000", 2024-09-17T07:25:05.531 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.8029999999999997e-06 2024-09-17T07:25:05.531 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.531 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.531 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:25:05.531 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.481210+0000", 2024-09-17T07:25:05.531 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.2265e-05 2024-09-17T07:25:05.531 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.531 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.531 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:25:05.531 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.481217+0000", 2024-09-17T07:25:05.531 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.5510000000000002e-06 2024-09-17T07:25:05.532 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.532 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.532 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:25:05.532 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.485435+0000", 2024-09-17T07:25:05.532 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.004217574 2024-09-17T07:25:05.532 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.532 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.532 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:25:05.532 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.485446+0000", 2024-09-17T07:25:05.532 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.1126e-05 2024-09-17T07:25:05.532 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.532 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.533 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:25:05.533 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.524137+0000", 2024-09-17T07:25:05.533 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.038691314 2024-09-17T07:25:05.533 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.533 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.533 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:25:05.533 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.524213+0000", 2024-09-17T07:25:05.533 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.5887e-05 2024-09-17T07:25:05.533 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.533 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.533 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 5,6", 2024-09-17T07:25:05.533 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.524317+0000", 2024-09-17T07:25:05.534 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000103661 2024-09-17T07:25:05.534 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.534 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.534 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:25:05.534 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.526299+0000", 2024-09-17T07:25:05.534 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0019818219999999998 2024-09-17T07:25:05.534 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.534 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.534 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:25:05.534 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.528456+0000", 2024-09-17T07:25:05.534 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0021578650000000001 2024-09-17T07:25:05.534 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.534 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.535 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:25:05.535 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.528495+0000", 2024-09-17T07:25:05.535 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.8294000000000003e-05 2024-09-17T07:25:05.535 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.535 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.535 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:25:05.535 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.528522+0000", 2024-09-17T07:25:05.535 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.7066000000000001e-05 2024-09-17T07:25:05.535 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.535 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.535 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:25:05.535 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.528542+0000", 2024-09-17T07:25:05.536 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.0458e-05 2024-09-17T07:25:05.536 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:25:05.536 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:25:05.536 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:25:05.536 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.536 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.536 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.25811.0:299910 11.5 11:a70d27a3:::benchmark_data_smithi100_180298_object127661:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e485)", 2024-09-17T07:25:05.536 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:24:17.482984+0000", 2024-09-17T07:25:05.536 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 47.987378339000003, 2024-09-17T07:25:05.536 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.050433269, 2024-09-17T07:25:05.536 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:25:05.536 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:25:05.536 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:25:05.537 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.25811", 2024-09-17T07:25:05.537 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/173867468", 2024-09-17T07:25:05.537 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 299910 2024-09-17T07:25:05.537 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.537 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:25:05.537 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.537 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:25:05.537 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.482984+0000", 2024-09-17T07:25:05.537 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:25:05.537 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.537 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.537 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:25:05.538 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.482984+0000", 2024-09-17T07:25:05.538 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:25:05.538 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.538 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.538 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:25:05.538 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.482983+0000", 2024-09-17T07:25:05.538 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:25:05.538 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.538 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.538 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:25:05.538 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.482989+0000", 2024-09-17T07:25:05.538 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.3749999999999999e-06 2024-09-17T07:25:05.538 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.538 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.538 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:25:05.539 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.483012+0000", 2024-09-17T07:25:05.539 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.2739999999999999e-05 2024-09-17T07:25:05.539 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.539 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.539 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:25:05.539 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.483019+0000", 2024-09-17T07:25:05.539 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.5650000000000004e-06 2024-09-17T07:25:05.539 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.539 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.539 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:25:05.539 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.485462+0000", 2024-09-17T07:25:05.539 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0024430519999999998 2024-09-17T07:25:05.539 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.539 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.540 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:25:05.540 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.485466+0000", 2024-09-17T07:25:05.540 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.014e-06 2024-09-17T07:25:05.540 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.540 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.540 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:25:05.540 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.526325+0000", 2024-09-17T07:25:05.540 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0408586520000001 2024-09-17T07:25:05.540 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.540 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.540 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:25:05.540 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.526430+0000", 2024-09-17T07:25:05.541 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00010487 2024-09-17T07:25:05.541 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.541 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.541 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 5,6", 2024-09-17T07:25:05.541 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.526504+0000", 2024-09-17T07:25:05.541 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.3868000000000005e-05 2024-09-17T07:25:05.541 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.541 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.541 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:25:05.541 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.528579+0000", 2024-09-17T07:25:05.541 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0020752980000000002 2024-09-17T07:25:05.541 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.542 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.542 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:25:05.542 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.528586+0000", 2024-09-17T07:25:05.542 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.584e-06 2024-09-17T07:25:05.542 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.542 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.542 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:25:05.542 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.533317+0000", 2024-09-17T07:25:05.542 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0047316830000000004 2024-09-17T07:25:05.542 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.543 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.543 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:25:05.543 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.533368+0000", 2024-09-17T07:25:05.543 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.0646999999999998e-05 2024-09-17T07:25:05.543 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.543 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.543 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:25:05.543 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.533417+0000", 2024-09-17T07:25:05.543 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.9339999999999999e-05 2024-09-17T07:25:05.543 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:25:05.543 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:25:05.543 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:25:05.544 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.544 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.544 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.25811.0:299917 11.5 11:a99df3a4:::benchmark_data_smithi100_180298_object127668:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e485)", 2024-09-17T07:25:05.544 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:24:17.487378+0000", 2024-09-17T07:25:05.544 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 47.982983898000001, 2024-09-17T07:25:05.544 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0462632329999999, 2024-09-17T07:25:05.544 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:25:05.544 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:25:05.544 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:25:05.544 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.25811", 2024-09-17T07:25:05.544 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/173867468", 2024-09-17T07:25:05.544 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 299917 2024-09-17T07:25:05.545 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.545 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:25:05.545 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.545 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:25:05.545 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.487378+0000", 2024-09-17T07:25:05.545 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:25:05.545 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.545 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.545 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:25:05.545 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.487378+0000", 2024-09-17T07:25:05.545 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:25:05.545 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.545 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.546 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:25:05.546 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.487377+0000", 2024-09-17T07:25:05.546 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:25:05.546 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.546 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.546 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:25:05.546 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.487383+0000", 2024-09-17T07:25:05.546 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.4069999999999998e-06 2024-09-17T07:25:05.546 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.546 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.546 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:25:05.546 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.487405+0000", 2024-09-17T07:25:05.547 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.1234999999999999e-05 2024-09-17T07:25:05.547 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.547 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.547 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:25:05.547 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.487419+0000", 2024-09-17T07:25:05.547 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.472e-05 2024-09-17T07:25:05.547 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.547 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.547 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:25:05.547 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.487482+0000", 2024-09-17T07:25:05.547 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.2366000000000006e-05 2024-09-17T07:25:05.547 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.547 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.548 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:25:05.548 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.487491+0000", 2024-09-17T07:25:05.548 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.9230000000000008e-06 2024-09-17T07:25:05.548 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.548 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.548 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:25:05.548 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.528608+0000", 2024-09-17T07:25:05.548 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0411177519999999 2024-09-17T07:25:05.548 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.548 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.548 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:25:05.548 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.528697+0000", 2024-09-17T07:25:05.548 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.9036999999999995e-05 2024-09-17T07:25:05.549 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.549 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.549 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 5,6", 2024-09-17T07:25:05.549 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.528774+0000", 2024-09-17T07:25:05.549 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.6709e-05 2024-09-17T07:25:05.549 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.549 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.549 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:25:05.549 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.533486+0000", 2024-09-17T07:25:05.549 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0047122819999999999 2024-09-17T07:25:05.549 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.549 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.549 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:25:05.549 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.533496+0000", 2024-09-17T07:25:05.550 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.1039999999999996e-06 2024-09-17T07:25:05.550 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.550 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.550 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:25:05.550 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.533529+0000", 2024-09-17T07:25:05.550 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.3355999999999997e-05 2024-09-17T07:25:05.550 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.550 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.550 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:25:05.550 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.533590+0000", 2024-09-17T07:25:05.550 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.084e-05 2024-09-17T07:25:05.550 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.550 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.550 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:25:05.551 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.533642+0000", 2024-09-17T07:25:05.551 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.1860000000000002e-05 2024-09-17T07:25:05.551 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:25:05.551 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:25:05.551 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:25:05.551 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.551 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.551 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.25811.0:299919 11.5 11:a8aad5dc:::benchmark_data_smithi100_180298_object127670:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e485)", 2024-09-17T07:25:05.551 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:24:17.490360+0000", 2024-09-17T07:25:05.551 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 47.980002185000004, 2024-09-17T07:25:05.551 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0488138739999999, 2024-09-17T07:25:05.551 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:25:05.551 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:25:05.552 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:25:05.552 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.25811", 2024-09-17T07:25:05.552 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/173867468", 2024-09-17T07:25:05.552 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 299919 2024-09-17T07:25:05.552 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.552 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:25:05.552 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.552 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:25:05.552 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.490360+0000", 2024-09-17T07:25:05.552 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:25:05.552 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.552 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.552 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:25:05.553 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.490360+0000", 2024-09-17T07:25:05.553 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:25:05.553 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.553 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.553 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:25:05.553 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.490358+0000", 2024-09-17T07:25:05.553 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999981 2024-09-17T07:25:05.553 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.553 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.553 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:25:05.553 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.490372+0000", 2024-09-17T07:25:05.553 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.3363999999999999e-05 2024-09-17T07:25:05.553 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.553 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.554 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:25:05.554 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.490388+0000", 2024-09-17T07:25:05.554 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.6324000000000002e-05 2024-09-17T07:25:05.554 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.554 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.554 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:25:05.554 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.490402+0000", 2024-09-17T07:25:05.554 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.4212e-05 2024-09-17T07:25:05.554 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.554 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.554 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:25:05.554 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.490809+0000", 2024-09-17T07:25:05.554 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000407243 2024-09-17T07:25:05.555 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.555 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.555 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:25:05.555 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.490819+0000", 2024-09-17T07:25:05.555 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0091e-05 2024-09-17T07:25:05.555 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.555 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.555 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:25:05.555 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.530963+0000", 2024-09-17T07:25:05.555 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0401440310000001 2024-09-17T07:25:05.555 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.555 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.555 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:25:05.556 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.531096+0000", 2024-09-17T07:25:05.556 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00013217599999999999 2024-09-17T07:25:05.556 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.556 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.556 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 5,6", 2024-09-17T07:25:05.556 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.531208+0000", 2024-09-17T07:25:05.556 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000112493 2024-09-17T07:25:05.556 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.556 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.556 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:25:05.556 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.533697+0000", 2024-09-17T07:25:05.556 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0024888940000000002 2024-09-17T07:25:05.557 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.557 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.557 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:25:05.557 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.533705+0000", 2024-09-17T07:25:05.557 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.5789999999999998e-06 2024-09-17T07:25:05.557 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.557 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.557 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:25:05.557 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.539095+0000", 2024-09-17T07:25:05.557 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0053907879999999997 2024-09-17T07:25:05.557 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.557 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.557 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:25:05.558 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.539136+0000", 2024-09-17T07:25:05.558 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.0154999999999998e-05 2024-09-17T07:25:05.558 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.558 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.558 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:25:05.558 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.539174+0000", 2024-09-17T07:25:05.558 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.8392000000000001e-05 2024-09-17T07:25:05.558 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:25:05.558 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:25:05.558 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:25:05.558 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.558 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.558 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.25811.0:299947 11.5 11:a4e1e8af:::benchmark_data_smithi100_180298_object127698:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e485)", 2024-09-17T07:25:05.558 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:24:17.508467+0000", 2024-09-17T07:25:05.559 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 47.961895640999998, 2024-09-17T07:25:05.559 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0309018830000001, 2024-09-17T07:25:05.559 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:25:05.559 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:25:05.559 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:25:05.559 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.25811", 2024-09-17T07:25:05.559 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/173867468", 2024-09-17T07:25:05.559 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 299947 2024-09-17T07:25:05.559 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.559 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:25:05.559 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.559 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:25:05.560 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.508467+0000", 2024-09-17T07:25:05.560 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:25:05.560 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.560 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.560 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:25:05.560 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.508467+0000", 2024-09-17T07:25:05.560 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:25:05.560 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.560 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.560 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:25:05.560 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.508465+0000", 2024-09-17T07:25:05.560 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999981 2024-09-17T07:25:05.560 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.561 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.561 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:25:05.561 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.508472+0000", 2024-09-17T07:25:05.561 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.1749999999999999e-06 2024-09-17T07:25:05.561 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.561 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.561 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:25:05.561 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.508483+0000", 2024-09-17T07:25:05.561 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.1241e-05 2024-09-17T07:25:05.561 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.561 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.561 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:25:05.561 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.508491+0000", 2024-09-17T07:25:05.562 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.2810000000000008e-06 2024-09-17T07:25:05.562 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.562 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.562 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:25:05.562 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.508550+0000", 2024-09-17T07:25:05.562 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.8866999999999997e-05 2024-09-17T07:25:05.562 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.562 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.562 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:25:05.562 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.508560+0000", 2024-09-17T07:25:05.562 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.5929999999999995e-06 2024-09-17T07:25:05.562 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.562 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.563 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:25:05.563 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.533730+0000", 2024-09-17T07:25:05.563 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.025170017 2024-09-17T07:25:05.563 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.563 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.563 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:25:05.563 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.533841+0000", 2024-09-17T07:25:05.563 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000111073 2024-09-17T07:25:05.563 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.563 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.563 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 5,6", 2024-09-17T07:25:05.563 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.533954+0000", 2024-09-17T07:25:05.563 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000112818 2024-09-17T07:25:05.564 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.564 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.564 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:25:05.564 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.539229+0000", 2024-09-17T07:25:05.564 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0052759149999999999 2024-09-17T07:25:05.564 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.564 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.564 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:25:05.564 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.539237+0000", 2024-09-17T07:25:05.564 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.7430000000000002e-06 2024-09-17T07:25:05.564 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.564 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.564 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:25:05.565 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.539288+0000", 2024-09-17T07:25:05.565 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.0603999999999999e-05 2024-09-17T07:25:05.565 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.565 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.565 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:25:05.565 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.539332+0000", 2024-09-17T07:25:05.565 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.4469999999999999e-05 2024-09-17T07:25:05.565 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.565 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.565 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:25:05.565 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.539369+0000", 2024-09-17T07:25:05.565 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.6210000000000001e-05 2024-09-17T07:25:05.565 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:25:05.566 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:25:05.566 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:25:05.566 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.566 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.566 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.25811.0:299969 11.5 11:a4622a3e:::benchmark_data_smithi100_180298_object127720:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e485)", 2024-09-17T07:25:05.566 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:24:17.526455+0000", 2024-09-17T07:25:05.566 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 47.943907393000003, 2024-09-17T07:25:05.566 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0181995850000001, 2024-09-17T07:25:05.566 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:25:05.566 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:25:05.566 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:25:05.566 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.25811", 2024-09-17T07:25:05.566 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/173867468", 2024-09-17T07:25:05.566 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 299969 2024-09-17T07:25:05.567 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.567 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:25:05.567 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.567 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:25:05.567 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.526455+0000", 2024-09-17T07:25:05.567 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:25:05.567 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.567 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.567 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:25:05.567 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.526455+0000", 2024-09-17T07:25:05.567 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:25:05.567 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.567 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.568 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:25:05.568 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.526454+0000", 2024-09-17T07:25:05.568 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:25:05.568 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.568 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.568 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:25:05.568 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.526460+0000", 2024-09-17T07:25:05.568 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.3790000000000003e-06 2024-09-17T07:25:05.568 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.568 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.568 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:25:05.568 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.526471+0000", 2024-09-17T07:25:05.568 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.1358999999999999e-05 2024-09-17T07:25:05.569 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.569 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.569 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:25:05.569 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.526476+0000", 2024-09-17T07:25:05.569 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.6870000000000002e-06 2024-09-17T07:25:05.569 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.569 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.569 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:25:05.569 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.526543+0000", 2024-09-17T07:25:05.569 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.6750999999999998e-05 2024-09-17T07:25:05.569 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.569 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.569 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:25:05.569 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.526553+0000", 2024-09-17T07:25:05.570 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0360000000000001e-05 2024-09-17T07:25:05.570 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.570 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.570 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:25:05.570 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.536730+0000", 2024-09-17T07:25:05.570 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.010177208 2024-09-17T07:25:05.570 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.570 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.570 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:25:05.570 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.536871+0000", 2024-09-17T07:25:05.570 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00014022500000000001 2024-09-17T07:25:05.570 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.571 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.571 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 5,6", 2024-09-17T07:25:05.571 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.536999+0000", 2024-09-17T07:25:05.571 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00012835799999999999 2024-09-17T07:25:05.571 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.571 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.571 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:25:05.571 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.539426+0000", 2024-09-17T07:25:05.571 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0024266629999999999 2024-09-17T07:25:05.571 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.571 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.571 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:25:05.571 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.544506+0000", 2024-09-17T07:25:05.571 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.005080757 2024-09-17T07:25:05.572 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.572 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.572 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:25:05.572 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.544574+0000", 2024-09-17T07:25:05.572 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.7564000000000006e-05 2024-09-17T07:25:05.572 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.572 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.572 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:25:05.572 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.544616+0000", 2024-09-17T07:25:05.572 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.2329000000000001e-05 2024-09-17T07:25:05.572 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.572 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.572 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:25:05.573 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.544654+0000", 2024-09-17T07:25:05.573 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.8170999999999998e-05 2024-09-17T07:25:05.573 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:25:05.573 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:25:05.573 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:25:05.573 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.573 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.573 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.25811.0:299973 11.5 11:a16cc86c:::benchmark_data_smithi100_180298_object127724:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e485)", 2024-09-17T07:25:05.573 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:24:17.527265+0000", 2024-09-17T07:25:05.573 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 47.943096787999998, 2024-09-17T07:25:05.573 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0175405360000001, 2024-09-17T07:25:05.573 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:25:05.573 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:25:05.574 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:25:05.574 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.25811", 2024-09-17T07:25:05.574 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/173867468", 2024-09-17T07:25:05.574 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 299973 2024-09-17T07:25:05.574 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.574 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:25:05.574 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.574 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:25:05.574 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.527265+0000", 2024-09-17T07:25:05.574 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:25:05.574 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.574 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.574 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:25:05.575 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.527265+0000", 2024-09-17T07:25:05.575 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:25:05.575 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.575 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.575 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:25:05.575 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.527265+0000", 2024-09-17T07:25:05.575 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.999999 2024-09-17T07:25:05.575 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.575 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.575 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:25:05.575 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.527273+0000", 2024-09-17T07:25:05.575 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.3240000000000003e-06 2024-09-17T07:25:05.575 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.576 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.576 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:25:05.576 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.527288+0000", 2024-09-17T07:25:05.576 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.4834000000000001e-05 2024-09-17T07:25:05.576 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.576 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.576 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:25:05.576 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.527295+0000", 2024-09-17T07:25:05.576 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.0389999999999999e-06 2024-09-17T07:25:05.576 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.576 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.576 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:25:05.576 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.527339+0000", 2024-09-17T07:25:05.577 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.4005000000000003e-05 2024-09-17T07:25:05.577 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.577 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.577 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:25:05.577 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.527347+0000", 2024-09-17T07:25:05.577 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.8690000000000004e-06 2024-09-17T07:25:05.577 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.577 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.577 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:25:05.577 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.539456+0000", 2024-09-17T07:25:05.577 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.012109385 2024-09-17T07:25:05.577 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.577 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.577 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:25:05.578 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.539583+0000", 2024-09-17T07:25:05.578 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000126649 2024-09-17T07:25:05.578 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.578 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.578 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 5,6", 2024-09-17T07:25:05.578 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.539701+0000", 2024-09-17T07:25:05.578 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000118853 2024-09-17T07:25:05.578 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.578 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.578 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:25:05.578 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.544696+0000", 2024-09-17T07:25:05.578 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00499461 2024-09-17T07:25:05.579 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.579 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.579 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:25:05.579 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.544704+0000", 2024-09-17T07:25:05.579 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.9310000000000008e-06 2024-09-17T07:25:05.579 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.579 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.579 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:25:05.579 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.544758+0000", 2024-09-17T07:25:05.579 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.4449e-05 2024-09-17T07:25:05.579 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.579 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.579 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:25:05.579 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.544786+0000", 2024-09-17T07:25:05.580 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.7083000000000001e-05 2024-09-17T07:25:05.580 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.580 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.580 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:25:05.580 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.544806+0000", 2024-09-17T07:25:05.580 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.0451000000000001e-05 2024-09-17T07:25:05.580 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:25:05.580 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:25:05.580 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:25:05.580 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.580 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.580 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.25811.0:299980 11.5 11:a00c80c2:::benchmark_data_smithi100_180298_object127731:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e485)", 2024-09-17T07:25:05.580 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:24:17.532390+0000", 2024-09-17T07:25:05.581 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 47.937972375999998, 2024-09-17T07:25:05.581 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0124928559999999, 2024-09-17T07:25:05.581 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:25:05.581 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:25:05.581 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:25:05.581 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.25811", 2024-09-17T07:25:05.581 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/173867468", 2024-09-17T07:25:05.581 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 299980 2024-09-17T07:25:05.581 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.581 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:25:05.581 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.581 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:25:05.581 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.532390+0000", 2024-09-17T07:25:05.582 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:25:05.582 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.582 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.582 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:25:05.582 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.532390+0000", 2024-09-17T07:25:05.582 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:25:05.582 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.582 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.582 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:25:05.582 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.532388+0000", 2024-09-17T07:25:05.582 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:25:05.582 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.582 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.582 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:25:05.583 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.532400+0000", 2024-09-17T07:25:05.583 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.1481000000000001e-05 2024-09-17T07:25:05.583 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.583 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.583 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:25:05.583 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.532413+0000", 2024-09-17T07:25:05.583 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.3267e-05 2024-09-17T07:25:05.583 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.583 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.583 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:25:05.583 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.532420+0000", 2024-09-17T07:25:05.583 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.1749999999999999e-06 2024-09-17T07:25:05.583 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.583 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.584 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:25:05.584 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.532477+0000", 2024-09-17T07:25:05.584 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.6418000000000001e-05 2024-09-17T07:25:05.584 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.584 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.584 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:25:05.584 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.532484+0000", 2024-09-17T07:25:05.584 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.0450000000000003e-06 2024-09-17T07:25:05.584 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.584 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.584 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:25:05.584 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.542004+0000", 2024-09-17T07:25:05.585 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0095204360000001 2024-09-17T07:25:05.585 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.585 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.585 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:25:05.585 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.542143+0000", 2024-09-17T07:25:05.585 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00013904600000000001 2024-09-17T07:25:05.585 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.585 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.585 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 5,6", 2024-09-17T07:25:05.585 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.542276+0000", 2024-09-17T07:25:05.585 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000132349 2024-09-17T07:25:05.585 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.585 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.586 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:25:05.586 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.544732+0000", 2024-09-17T07:25:05.586 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0024565640000000001 2024-09-17T07:25:05.586 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.586 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.586 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:25:05.586 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.544815+0000", 2024-09-17T07:25:05.586 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.2583999999999994e-05 2024-09-17T07:25:05.586 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.586 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.586 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:25:05.586 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.544843+0000", 2024-09-17T07:25:05.586 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.8481000000000001e-05 2024-09-17T07:25:05.586 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.587 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.587 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:25:05.587 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.544866+0000", 2024-09-17T07:25:05.587 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.2359999999999999e-05 2024-09-17T07:25:05.587 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.587 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.587 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:25:05.587 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.544883+0000", 2024-09-17T07:25:05.587 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.7081000000000001e-05 2024-09-17T07:25:05.587 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:25:05.587 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:25:05.587 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:25:05.587 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.588 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.588 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.25811.0:299985 11.5 11:a5c12907:::benchmark_data_smithi100_180298_object127736:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e485)", 2024-09-17T07:25:05.588 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:24:17.536664+0000", 2024-09-17T07:25:05.588 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 47.933698161000002, 2024-09-17T07:25:05.588 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.021478479, 2024-09-17T07:25:05.588 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:25:05.588 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:25:05.588 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:25:05.588 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.25811", 2024-09-17T07:25:05.588 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/173867468", 2024-09-17T07:25:05.588 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 299985 2024-09-17T07:25:05.588 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.589 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:25:05.589 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.589 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:25:05.589 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.536664+0000", 2024-09-17T07:25:05.589 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:25:05.589 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.589 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.589 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:25:05.589 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.536664+0000", 2024-09-17T07:25:05.589 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:25:05.589 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.589 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.589 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:25:05.590 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.536662+0000", 2024-09-17T07:25:05.590 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999981 2024-09-17T07:25:05.590 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.590 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.590 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:25:05.590 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.536673+0000", 2024-09-17T07:25:05.590 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0358e-05 2024-09-17T07:25:05.590 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.590 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.590 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:25:05.590 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.536686+0000", 2024-09-17T07:25:05.590 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.3454e-05 2024-09-17T07:25:05.590 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.591 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.591 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:25:05.591 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.536694+0000", 2024-09-17T07:25:05.591 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.8639999999999996e-06 2024-09-17T07:25:05.591 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.591 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.591 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:25:05.591 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.536751+0000", 2024-09-17T07:25:05.591 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.6929000000000003e-05 2024-09-17T07:25:05.591 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.591 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.591 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:25:05.591 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.536758+0000", 2024-09-17T07:25:05.592 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.7020000000000004e-06 2024-09-17T07:25:05.592 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.592 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.592 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:25:05.592 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.544909+0000", 2024-09-17T07:25:05.592 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.00815147 2024-09-17T07:25:05.592 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.592 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.592 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:25:05.592 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.544991+0000", 2024-09-17T07:25:05.592 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.1812999999999997e-05 2024-09-17T07:25:05.592 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.592 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.593 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 5,6", 2024-09-17T07:25:05.593 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.545058+0000", 2024-09-17T07:25:05.593 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.7183999999999999e-05 2024-09-17T07:25:05.593 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.593 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.593 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:25:05.593 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.551558+0000", 2024-09-17T07:25:05.593 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0064999769999999997 2024-09-17T07:25:05.593 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.593 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.593 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:25:05.593 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.551605+0000", 2024-09-17T07:25:05.594 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.7015999999999998e-05 2024-09-17T07:25:05.594 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.594 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.594 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:25:05.594 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.558064+0000", 2024-09-17T07:25:05.594 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0064591040000000002 2024-09-17T07:25:05.594 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.594 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.594 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:25:05.594 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.558107+0000", 2024-09-17T07:25:05.594 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.3031000000000002e-05 2024-09-17T07:25:05.594 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.594 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.594 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:25:05.595 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.558143+0000", 2024-09-17T07:25:05.595 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.5253000000000003e-05 2024-09-17T07:25:05.595 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:25:05.595 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:25:05.595 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:25:05.595 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.595 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.595 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.25811.0:299992 11.5 11:a5ecb0ee:::benchmark_data_smithi100_180298_object127743:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e485)", 2024-09-17T07:25:05.595 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:24:17.542509+0000", 2024-09-17T07:25:05.595 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 47.927853386000002, 2024-09-17T07:25:05.595 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.015726041, 2024-09-17T07:25:05.595 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:25:05.595 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:25:05.595 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:25:05.596 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.25811", 2024-09-17T07:25:05.596 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/173867468", 2024-09-17T07:25:05.596 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 299992 2024-09-17T07:25:05.596 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.596 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:25:05.596 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.596 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:25:05.596 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.542509+0000", 2024-09-17T07:25:05.596 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:25:05.596 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.596 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.596 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:25:05.596 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.542509+0000", 2024-09-17T07:25:05.597 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:25:05.597 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.597 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.597 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:25:05.597 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.542507+0000", 2024-09-17T07:25:05.597 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999981 2024-09-17T07:25:05.597 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.597 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.597 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:25:05.597 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.542515+0000", 2024-09-17T07:25:05.597 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.6260000000000003e-06 2024-09-17T07:25:05.597 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.597 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.597 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:25:05.598 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.542532+0000", 2024-09-17T07:25:05.598 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.6722e-05 2024-09-17T07:25:05.598 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.598 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.598 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:25:05.598 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.542539+0000", 2024-09-17T07:25:05.598 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.4139999999999997e-06 2024-09-17T07:25:05.598 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.598 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.598 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:25:05.598 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.542658+0000", 2024-09-17T07:25:05.598 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00011950100000000001 2024-09-17T07:25:05.599 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.599 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.599 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:25:05.599 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.542670+0000", 2024-09-17T07:25:05.599 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.1885000000000001e-05 2024-09-17T07:25:05.599 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.599 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.599 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:25:05.599 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.546974+0000", 2024-09-17T07:25:05.599 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.0043032409999999 2024-09-17T07:25:05.599 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.599 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.599 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:25:05.600 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.547112+0000", 2024-09-17T07:25:05.600 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00013889100000000001 2024-09-17T07:25:05.600 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.600 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.600 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 5,6", 2024-09-17T07:25:05.600 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.547236+0000", 2024-09-17T07:25:05.600 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000123199 2024-09-17T07:25:05.600 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.600 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.600 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:25:05.600 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.551611+0000", 2024-09-17T07:25:05.600 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0043753409999999996 2024-09-17T07:25:05.600 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.601 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.601 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:25:05.601 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.551639+0000", 2024-09-17T07:25:05.601 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.8158e-05 2024-09-17T07:25:05.601 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.601 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.601 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:25:05.601 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.558175+0000", 2024-09-17T07:25:05.601 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0065356269999999996 2024-09-17T07:25:05.601 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.601 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.601 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:25:05.601 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.558206+0000", 2024-09-17T07:25:05.602 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.1068999999999997e-05 2024-09-17T07:25:05.602 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.602 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.602 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:25:05.602 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.558235+0000", 2024-09-17T07:25:05.602 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.9099e-05 2024-09-17T07:25:05.602 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:25:05.602 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:25:05.602 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:25:05.602 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.602 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.602 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.25811.0:300013 11.5 11:a6629ab8:::benchmark_data_smithi100_180298_object127764:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e485)", 2024-09-17T07:25:05.603 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:24:17.561548+0000", 2024-09-17T07:25:05.603 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 47.908813856000002, 2024-09-17T07:25:05.603 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.004267561, 2024-09-17T07:25:05.603 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:25:05.603 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:25:05.603 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:25:05.603 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.25811", 2024-09-17T07:25:05.603 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/173867468", 2024-09-17T07:25:05.603 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 300013 2024-09-17T07:25:05.603 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.603 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:25:05.603 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.604 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:25:05.604 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.561548+0000", 2024-09-17T07:25:05.604 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:25:05.604 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.604 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.604 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:25:05.604 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.561548+0000", 2024-09-17T07:25:05.604 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:25:05.604 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.604 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.604 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:25:05.604 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.561547+0000", 2024-09-17T07:25:05.604 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:25:05.605 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.605 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.605 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:25:05.605 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.561554+0000", 2024-09-17T07:25:05.605 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.9399999999999996e-06 2024-09-17T07:25:05.605 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.605 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.605 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:25:05.605 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.561566+0000", 2024-09-17T07:25:05.605 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.238e-05 2024-09-17T07:25:05.605 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.605 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.605 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:25:05.606 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.561573+0000", 2024-09-17T07:25:05.606 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.8519999999999998e-06 2024-09-17T07:25:05.606 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.606 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.606 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:25:05.606 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.561628+0000", 2024-09-17T07:25:05.606 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 5.4898000000000002e-05 2024-09-17T07:25:05.606 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.606 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.606 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:25:05.606 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.561638+0000", 2024-09-17T07:25:05.606 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 9.5100000000000004e-06 2024-09-17T07:25:05.606 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.607 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.607 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:25:05.607 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.549328+0000", 2024-09-17T07:25:05.607 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.98769060600000003 2024-09-17T07:25:05.607 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.607 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.607 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:25:05.607 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.549459+0000", 2024-09-17T07:25:05.607 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00013056100000000001 2024-09-17T07:25:05.607 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.607 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.607 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 5,6", 2024-09-17T07:25:05.607 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.549563+0000", 2024-09-17T07:25:05.608 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00010417900000000001 2024-09-17T07:25:05.608 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.608 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.608 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:25:05.608 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.558035+0000", 2024-09-17T07:25:05.608 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0084723660000000003 2024-09-17T07:25:05.608 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.608 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.608 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:25:05.608 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.558264+0000", 2024-09-17T07:25:05.608 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00022908499999999999 2024-09-17T07:25:05.608 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.608 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.608 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:25:05.609 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.565730+0000", 2024-09-17T07:25:05.609 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0074659790000000002 2024-09-17T07:25:05.609 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.609 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.609 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:25:05.609 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.565776+0000", 2024-09-17T07:25:05.609 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.5686000000000003e-05 2024-09-17T07:25:05.609 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.609 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.609 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:25:05.609 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.565816+0000", 2024-09-17T07:25:05.609 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.9888999999999997e-05 2024-09-17T07:25:05.610 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:25:05.610 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:25:05.610 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:25:05.610 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.610 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.610 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.25811.0:300019 11.5 11:a4931d5a:::benchmark_data_smithi100_180298_object127770:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e485)", 2024-09-17T07:25:05.610 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:24:17.567653+0000", 2024-09-17T07:25:05.610 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 47.902709268999999, 2024-09-17T07:25:05.610 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.99844087199999998, 2024-09-17T07:25:05.610 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:25:05.610 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:25:05.610 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:25:05.610 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.25811", 2024-09-17T07:25:05.611 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/173867468", 2024-09-17T07:25:05.611 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 300019 2024-09-17T07:25:05.611 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.611 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:25:05.611 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.611 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:25:05.611 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.567653+0000", 2024-09-17T07:25:05.611 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:25:05.611 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.611 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.611 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:25:05.611 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.567653+0000", 2024-09-17T07:25:05.611 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:25:05.612 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.612 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.612 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:25:05.612 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.567652+0000", 2024-09-17T07:25:05.612 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999986 2024-09-17T07:25:05.612 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.612 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.612 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:25:05.612 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.567658+0000", 2024-09-17T07:25:05.612 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.3779999999999998e-06 2024-09-17T07:25:05.612 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.612 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.612 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:25:05.613 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.567670+0000", 2024-09-17T07:25:05.613 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.2085e-05 2024-09-17T07:25:05.613 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.613 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.613 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:25:05.613 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.567676+0000", 2024-09-17T07:25:05.613 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.1829999999999999e-06 2024-09-17T07:25:05.613 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.613 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.613 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:25:05.613 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.567742+0000", 2024-09-17T07:25:05.613 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.5982999999999995e-05 2024-09-17T07:25:05.614 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.614 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.614 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:25:05.614 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.567754+0000", 2024-09-17T07:25:05.614 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.2096000000000001e-05 2024-09-17T07:25:05.614 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.614 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.614 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:25:05.614 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.551659+0000", 2024-09-17T07:25:05.614 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.98390482300000004 2024-09-17T07:25:05.614 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.614 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.614 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:25:05.615 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.551771+0000", 2024-09-17T07:25:05.615 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00011212700000000001 2024-09-17T07:25:05.615 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.615 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.615 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 5,6", 2024-09-17T07:25:05.615 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.551873+0000", 2024-09-17T07:25:05.615 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00010160100000000001 2024-09-17T07:25:05.615 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.615 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.615 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:25:05.615 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.558292+0000", 2024-09-17T07:25:05.615 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0064187539999999996 2024-09-17T07:25:05.615 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.616 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.616 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:25:05.616 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.565910+0000", 2024-09-17T07:25:05.616 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0076184690000000001 2024-09-17T07:25:05.616 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.616 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.616 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:25:05.616 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.566012+0000", 2024-09-17T07:25:05.616 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000101803 2024-09-17T07:25:05.616 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.616 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.616 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:25:05.617 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.566059+0000", 2024-09-17T07:25:05.617 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4.7243000000000002e-05 2024-09-17T07:25:05.617 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.617 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.617 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:25:05.617 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.566094+0000", 2024-09-17T07:25:05.617 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.4567999999999998e-05 2024-09-17T07:25:05.617 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:25:05.617 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:25:05.617 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:25:05.617 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.617 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.617 INFO:teuthology.orchestra.run.smithi135.stdout: "description": "osd_op(client.25811.0:300050 11.5 11:aa41e330:::benchmark_data_smithi100_180298_object127801:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e485)", 2024-09-17T07:25:05.618 INFO:teuthology.orchestra.run.smithi135.stdout: "initiated_at": "2024-09-17T07:24:17.616937+0000", 2024-09-17T07:25:05.618 INFO:teuthology.orchestra.run.smithi135.stdout: "age": 47.853425203999997, 2024-09-17T07:25:05.618 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.94922332300000001, 2024-09-17T07:25:05.618 INFO:teuthology.orchestra.run.smithi135.stdout: "type_data": { 2024-09-17T07:25:05.618 INFO:teuthology.orchestra.run.smithi135.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T07:25:05.618 INFO:teuthology.orchestra.run.smithi135.stdout: "client_info": { 2024-09-17T07:25:05.618 INFO:teuthology.orchestra.run.smithi135.stdout: "client": "client.25811", 2024-09-17T07:25:05.618 INFO:teuthology.orchestra.run.smithi135.stdout: "client_addr": "172.21.15.100:0/173867468", 2024-09-17T07:25:05.618 INFO:teuthology.orchestra.run.smithi135.stdout: "tid": 300050 2024-09-17T07:25:05.618 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.618 INFO:teuthology.orchestra.run.smithi135.stdout: "events": [ 2024-09-17T07:25:05.618 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.618 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "initiated", 2024-09-17T07:25:05.619 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.616937+0000", 2024-09-17T07:25:05.619 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:25:05.619 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.619 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.619 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "throttled", 2024-09-17T07:25:05.619 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.616937+0000", 2024-09-17T07:25:05.619 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0 2024-09-17T07:25:05.619 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.619 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.619 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "header_read", 2024-09-17T07:25:05.619 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.616935+0000", 2024-09-17T07:25:05.619 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 4294967295.9999981 2024-09-17T07:25:05.619 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.619 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.620 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "all_read", 2024-09-17T07:25:05.620 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.616943+0000", 2024-09-17T07:25:05.620 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 7.5689999999999999e-06 2024-09-17T07:25:05.620 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.620 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.620 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "dispatched", 2024-09-17T07:25:05.620 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.616957+0000", 2024-09-17T07:25:05.620 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.3985000000000001e-05 2024-09-17T07:25:05.620 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.620 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.620 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "queued_for_pg", 2024-09-17T07:25:05.620 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.616965+0000", 2024-09-17T07:25:05.620 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 8.2509999999999994e-06 2024-09-17T07:25:05.621 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.621 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.621 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:25:05.621 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.617028+0000", 2024-09-17T07:25:05.621 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 6.2710000000000001e-05 2024-09-17T07:25:05.621 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.621 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.621 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for peered", 2024-09-17T07:25:05.621 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:17.617040+0000", 2024-09-17T07:25:05.621 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 1.2103000000000001e-05 2024-09-17T07:25:05.621 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.621 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.621 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "reached_pg", 2024-09-17T07:25:05.622 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.554740+0000", 2024-09-17T07:25:05.622 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.93770026100000003 2024-09-17T07:25:05.622 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.622 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.622 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "started", 2024-09-17T07:25:05.622 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.554871+0000", 2024-09-17T07:25:05.622 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00013128700000000001 2024-09-17T07:25:05.622 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.622 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.622 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "waiting for subops from 5,6", 2024-09-17T07:25:05.622 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.554982+0000", 2024-09-17T07:25:05.622 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.000110276 2024-09-17T07:25:05.622 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.623 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.623 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:25:05.623 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.558320+0000", 2024-09-17T07:25:05.623 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0033380760000000002 2024-09-17T07:25:05.623 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.623 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.623 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "sub_op_commit_rec", 2024-09-17T07:25:05.623 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.565939+0000", 2024-09-17T07:25:05.623 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.0076191799999999997 2024-09-17T07:25:05.623 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.623 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.623 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "op_commit", 2024-09-17T07:25:05.623 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.566099+0000", 2024-09-17T07:25:05.623 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 0.00016003299999999999 2024-09-17T07:25:05.624 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.624 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.624 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "commit_sent", 2024-09-17T07:25:05.624 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.566125+0000", 2024-09-17T07:25:05.624 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 2.5899000000000001e-05 2024-09-17T07:25:05.624 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-09-17T07:25:05.624 INFO:teuthology.orchestra.run.smithi135.stdout: { 2024-09-17T07:25:05.624 INFO:teuthology.orchestra.run.smithi135.stdout: "event": "done", 2024-09-17T07:25:05.624 INFO:teuthology.orchestra.run.smithi135.stdout: "time": "2024-09-17T07:24:18.566160+0000", 2024-09-17T07:25:05.624 INFO:teuthology.orchestra.run.smithi135.stdout: "duration": 3.5617999999999997e-05 2024-09-17T07:25:05.624 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:25:05.624 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:25:05.624 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:25:05.625 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-09-17T07:25:05.625 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-09-17T07:25:05.625 INFO:teuthology.orchestra.run.smithi135.stdout:} 2024-09-17T07:25:05.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:25:05.646 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:25:05.646 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:25:05.698 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:25:05.747 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:25:05.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:25:05.771 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:25:05.771 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:25:05.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:05 smithi135 ceph-mon[33238]: 4.1d scrub starts 2024-09-17T07:25:05.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:05 smithi135 ceph-mon[33238]: 4.1d scrub ok 2024-09-17T07:25:05.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:05 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/3700169077' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-09-17T07:25:05.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:05 smithi135 ceph-mon[33238]: osdmap e502: 8 total, 8 up, 7 in 2024-09-17T07:25:05.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:05 smithi135 ceph-mon[33238]: 4.16 scrub starts 2024-09-17T07:25:05.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:05 smithi135 ceph-mon[33238]: 2.e scrub starts 2024-09-17T07:25:05.819 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:05 smithi100 ceph-mon[29092]: 4.1d scrub starts 2024-09-17T07:25:05.819 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:05 smithi100 ceph-mon[29092]: 4.1d scrub ok 2024-09-17T07:25:05.819 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:05 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/3700169077' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-09-17T07:25:05.819 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:05 smithi100 ceph-mon[29092]: osdmap e502: 8 total, 8 up, 7 in 2024-09-17T07:25:05.819 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:05 smithi100 ceph-mon[29092]: 4.16 scrub starts 2024-09-17T07:25:05.819 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:05 smithi100 ceph-mon[29092]: 2.e scrub starts 2024-09-17T07:25:05.820 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:05 smithi100 ceph-mon[39462]: 4.1d scrub starts 2024-09-17T07:25:05.820 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:05 smithi100 ceph-mon[39462]: 4.1d scrub ok 2024-09-17T07:25:05.820 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:05 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/3700169077' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-09-17T07:25:05.820 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:05 smithi100 ceph-mon[39462]: osdmap e502: 8 total, 8 up, 7 in 2024-09-17T07:25:05.820 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:05 smithi100 ceph-mon[39462]: 4.16 scrub starts 2024-09-17T07:25:05.820 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:05 smithi100 ceph-mon[39462]: 2.e scrub starts 2024-09-17T07:25:05.872 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:25:05.895 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:25:05.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:25:05.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:25:05.997 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:25:06.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:25:06.019 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:25:06.019 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:25:06.120 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:25:06.126 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:25:05 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:25:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:25:06.141 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:25:06.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:25:06.142 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:25:06.243 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:25:06.265 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:25:06.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:25:06.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.5' 2024-09-17T07:25:06.364 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-17T07:25:06.366 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:25:06.387 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:25:06.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:25:06.388 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:25:06.489 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:25:06.510 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:25:06.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:25:06.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.7' 2024-09-17T07:25:06.612 DEBUG:teuthology.orchestra.run.smithi135:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:25:06.634 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: docker: command not found 2024-09-17T07:25:06.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:25:06.635 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi135 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.6' 2024-09-17T07:25:06.716 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:06 smithi135 ceph-mon[33238]: 5.18 scrub starts 2024-09-17T07:25:06.717 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:06 smithi135 ceph-mon[33238]: 5.18 scrub ok 2024-09-17T07:25:06.717 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:06 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:25:06.717 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:06 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:25:06.717 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:06 smithi135 ceph-mon[33238]: osdmap e503: 8 total, 8 up, 7 in 2024-09-17T07:25:06.717 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:06 smithi135 ceph-mon[33238]: pgmap v1097: 142 pgs: 1 active+clean+scrubbing, 141 active+clean; 477 KiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 798 B/s rd, 0 op/s 2024-09-17T07:25:06.717 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:06 smithi135 ceph-mon[33238]: 5.2 scrub starts 2024-09-17T07:25:06.717 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:06 smithi135 ceph-mon[33238]: 4.6 scrub starts 2024-09-17T07:25:06.717 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:06 smithi135 ceph-mon[33238]: 5.2 scrub ok 2024-09-17T07:25:06.717 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:06 smithi135 ceph-mon[33238]: 4.6 scrub ok 2024-09-17T07:25:06.736 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:25:06.758 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:25:06.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:25:06.759 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.3' 2024-09-17T07:25:06.824 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:06 smithi100 ceph-mon[29092]: 5.18 scrub starts 2024-09-17T07:25:06.824 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:06 smithi100 ceph-mon[29092]: 5.18 scrub ok 2024-09-17T07:25:06.824 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:06 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:25:06.824 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:06 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:25:06.825 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:06 smithi100 ceph-mon[29092]: osdmap e503: 8 total, 8 up, 7 in 2024-09-17T07:25:06.825 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:06 smithi100 ceph-mon[29092]: pgmap v1097: 142 pgs: 1 active+clean+scrubbing, 141 active+clean; 477 KiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 798 B/s rd, 0 op/s 2024-09-17T07:25:06.825 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:06 smithi100 ceph-mon[29092]: 5.2 scrub starts 2024-09-17T07:25:06.825 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:06 smithi100 ceph-mon[29092]: 4.6 scrub starts 2024-09-17T07:25:06.825 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:06 smithi100 ceph-mon[29092]: 5.2 scrub ok 2024-09-17T07:25:06.826 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:06 smithi100 ceph-mon[29092]: 4.6 scrub ok 2024-09-17T07:25:06.826 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:06 smithi100 ceph-mon[39462]: 5.18 scrub starts 2024-09-17T07:25:06.826 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:06 smithi100 ceph-mon[39462]: 5.18 scrub ok 2024-09-17T07:25:06.826 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:06 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:25:06.826 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:06 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:25:06.827 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:06 smithi100 ceph-mon[39462]: osdmap e503: 8 total, 8 up, 7 in 2024-09-17T07:25:06.827 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:06 smithi100 ceph-mon[39462]: pgmap v1097: 142 pgs: 1 active+clean+scrubbing, 141 active+clean; 477 KiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 798 B/s rd, 0 op/s 2024-09-17T07:25:06.827 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:06 smithi100 ceph-mon[39462]: 5.2 scrub starts 2024-09-17T07:25:06.827 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:06 smithi100 ceph-mon[39462]: 4.6 scrub starts 2024-09-17T07:25:06.827 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:06 smithi100 ceph-mon[39462]: 5.2 scrub ok 2024-09-17T07:25:06.827 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:06 smithi100 ceph-mon[39462]: 4.6 scrub ok 2024-09-17T07:25:06.860 DEBUG:teuthology.orchestra.run.smithi100:> sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:25:06.884 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: docker: command not found 2024-09-17T07:25:06.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:25:06.885 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi100 with status 1: 'sudo docker kill -s 1 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd.1' 2024-09-17T07:25:06.886 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-09-17T07:25:06.899 INFO:tasks.thrashosds:joining thrashosds 2024-09-17T07:25:07.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:07 smithi135 ceph-mon[33238]: 3.0 scrub starts 2024-09-17T07:25:07.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:07 smithi135 ceph-mon[33238]: 3.0 scrub ok 2024-09-17T07:25:07.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:07 smithi135 ceph-mon[33238]: 6.16 scrub starts 2024-09-17T07:25:07.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:07 smithi135 ceph-mon[33238]: 6.16 scrub ok 2024-09-17T07:25:07.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:07 smithi135 ceph-mon[33238]: 6.15 scrub starts 2024-09-17T07:25:07.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:07 smithi135 ceph-mon[33238]: 6.15 scrub ok 2024-09-17T07:25:07.824 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:07 smithi100 ceph-mon[29092]: 3.0 scrub starts 2024-09-17T07:25:07.824 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:07 smithi100 ceph-mon[29092]: 3.0 scrub ok 2024-09-17T07:25:07.824 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:07 smithi100 ceph-mon[29092]: 6.16 scrub starts 2024-09-17T07:25:07.824 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:07 smithi100 ceph-mon[29092]: 6.16 scrub ok 2024-09-17T07:25:07.824 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:07 smithi100 ceph-mon[29092]: 6.15 scrub starts 2024-09-17T07:25:07.824 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:07 smithi100 ceph-mon[29092]: 6.15 scrub ok 2024-09-17T07:25:07.825 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:07 smithi100 ceph-mon[39462]: 3.0 scrub starts 2024-09-17T07:25:07.825 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:07 smithi100 ceph-mon[39462]: 3.0 scrub ok 2024-09-17T07:25:07.825 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:07 smithi100 ceph-mon[39462]: 6.16 scrub starts 2024-09-17T07:25:07.825 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:07 smithi100 ceph-mon[39462]: 6.16 scrub ok 2024-09-17T07:25:07.825 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:07 smithi100 ceph-mon[39462]: 6.15 scrub starts 2024-09-17T07:25:07.825 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:07 smithi100 ceph-mon[39462]: 6.15 scrub ok 2024-09-17T07:25:08.514 INFO:teuthology.orchestra.run.smithi100.stderr:reweighted osd.6 to 1 (10000) 2024-09-17T07:25:08.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:08 smithi135 ceph-mon[33238]: pgmap v1098: 142 pgs: 1 active+clean+scrubbing, 141 active+clean; 477 KiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 678 B/s rd, 0 op/s 2024-09-17T07:25:08.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:08 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/4094472217' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-17T07:25:08.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:08 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-17T07:25:08.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:08 smithi135 ceph-mon[33238]: 2.7 scrub starts 2024-09-17T07:25:08.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:08 smithi135 ceph-mon[33238]: 2.7 scrub ok 2024-09-17T07:25:08.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:08 smithi135 ceph-mon[33238]: 2.5 scrub starts 2024-09-17T07:25:08.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:08 smithi135 ceph-mon[33238]: 2.5 scrub ok 2024-09-17T07:25:08.824 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:08 smithi100 ceph-mon[29092]: pgmap v1098: 142 pgs: 1 active+clean+scrubbing, 141 active+clean; 477 KiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 678 B/s rd, 0 op/s 2024-09-17T07:25:08.824 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:08 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/4094472217' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-17T07:25:08.824 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:08 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-17T07:25:08.824 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:08 smithi100 ceph-mon[29092]: 2.7 scrub starts 2024-09-17T07:25:08.824 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:08 smithi100 ceph-mon[29092]: 2.7 scrub ok 2024-09-17T07:25:08.824 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:08 smithi100 ceph-mon[29092]: 2.5 scrub starts 2024-09-17T07:25:08.824 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:08 smithi100 ceph-mon[29092]: 2.5 scrub ok 2024-09-17T07:25:08.824 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:08 smithi100 ceph-mon[39462]: pgmap v1098: 142 pgs: 1 active+clean+scrubbing, 141 active+clean; 477 KiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 678 B/s rd, 0 op/s 2024-09-17T07:25:08.824 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:08 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/4094472217' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-17T07:25:08.824 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:08 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-17T07:25:08.824 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:08 smithi100 ceph-mon[39462]: 2.7 scrub starts 2024-09-17T07:25:08.825 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:08 smithi100 ceph-mon[39462]: 2.7 scrub ok 2024-09-17T07:25:08.825 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:08 smithi100 ceph-mon[39462]: 2.5 scrub starts 2024-09-17T07:25:08.825 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:08 smithi100 ceph-mon[39462]: 2.5 scrub ok 2024-09-17T07:25:09.355 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd reweight 7 1 2024-09-17T07:25:09.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:09 smithi100 ceph-mon[29092]: 4.15 scrub starts 2024-09-17T07:25:09.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:09 smithi100 ceph-mon[29092]: 4.15 scrub ok 2024-09-17T07:25:09.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:09 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-17T07:25:09.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:09 smithi100 ceph-mon[29092]: osdmap e504: 8 total, 8 up, 7 in 2024-09-17T07:25:09.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:09 smithi100 ceph-mon[29092]: 6.0 scrub starts 2024-09-17T07:25:09.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:09 smithi100 ceph-mon[29092]: 6.0 scrub ok 2024-09-17T07:25:09.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:09 smithi100 ceph-mon[29092]: 2.b scrub starts 2024-09-17T07:25:09.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:09 smithi100 ceph-mon[29092]: 2.b scrub ok 2024-09-17T07:25:09.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:09 smithi100 ceph-mon[29092]: osdmap e505: 8 total, 8 up, 7 in 2024-09-17T07:25:09.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:09 smithi100 ceph-mon[39462]: 4.15 scrub starts 2024-09-17T07:25:09.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:09 smithi100 ceph-mon[39462]: 4.15 scrub ok 2024-09-17T07:25:09.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:09 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-17T07:25:09.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:09 smithi100 ceph-mon[39462]: osdmap e504: 8 total, 8 up, 7 in 2024-09-17T07:25:09.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:09 smithi100 ceph-mon[39462]: 6.0 scrub starts 2024-09-17T07:25:09.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:09 smithi100 ceph-mon[39462]: 6.0 scrub ok 2024-09-17T07:25:09.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:09 smithi100 ceph-mon[39462]: 2.b scrub starts 2024-09-17T07:25:09.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:09 smithi100 ceph-mon[39462]: 2.b scrub ok 2024-09-17T07:25:09.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:09 smithi100 ceph-mon[39462]: osdmap e505: 8 total, 8 up, 7 in 2024-09-17T07:25:09.793 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:25:09 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:25:09.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:09 smithi135 ceph-mon[33238]: 4.15 scrub starts 2024-09-17T07:25:09.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:09 smithi135 ceph-mon[33238]: 4.15 scrub ok 2024-09-17T07:25:09.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:09 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-17T07:25:09.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:09 smithi135 ceph-mon[33238]: osdmap e504: 8 total, 8 up, 7 in 2024-09-17T07:25:09.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:09 smithi135 ceph-mon[33238]: 6.0 scrub starts 2024-09-17T07:25:09.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:09 smithi135 ceph-mon[33238]: 6.0 scrub ok 2024-09-17T07:25:09.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:09 smithi135 ceph-mon[33238]: 2.b scrub starts 2024-09-17T07:25:09.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:09 smithi135 ceph-mon[33238]: 2.b scrub ok 2024-09-17T07:25:09.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:09 smithi135 ceph-mon[33238]: osdmap e505: 8 total, 8 up, 7 in 2024-09-17T07:25:10.052 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0: {} 2024-09-17T07:25:10.052 INFO:teuthology.orchestra.run.smithi100.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T07:25:10.066 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1: {} 2024-09-17T07:25:10.066 INFO:teuthology.orchestra.run.smithi100.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T07:25:10.072 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:25:10.086 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2: {} 2024-09-17T07:25:10.086 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T07:25:10.109 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3: {} 2024-09-17T07:25:10.109 INFO:teuthology.orchestra.run.smithi100.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T07:25:10.125 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4: {} 2024-09-17T07:25:10.126 INFO:teuthology.orchestra.run.smithi100.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T07:25:10.141 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5: {} 2024-09-17T07:25:10.141 INFO:teuthology.orchestra.run.smithi100.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T07:25:10.157 INFO:teuthology.orchestra.run.smithi100.stdout:osd.6: {} 2024-09-17T07:25:10.157 INFO:teuthology.orchestra.run.smithi100.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T07:25:10.173 INFO:teuthology.orchestra.run.smithi100.stdout:osd.7: {} 2024-09-17T07:25:10.173 INFO:teuthology.orchestra.run.smithi100.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T07:25:10.383 INFO:teuthology.orchestra.run.smithi100.stderr:nodeep-scrub is unset 2024-09-17T07:25:10.610 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:25:10.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:10 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:25:10.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:10 smithi135 ceph-mon[33238]: pgmap v1100: 142 pgs: 1 peering, 1 active+clean+scrubbing, 140 active+clean; 477 KiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-17T07:25:10.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:10 smithi135 ceph-mon[33238]: 2.1a scrub starts 2024-09-17T07:25:10.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:10 smithi135 ceph-mon[33238]: 2.1a scrub ok 2024-09-17T07:25:10.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:10 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1141520688' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:25:10.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:10 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/48088441' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:25:10.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:10 smithi135 ceph-mon[33238]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T07:25:10.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:10 smithi135 ceph-mon[33238]: Cluster is now healthy 2024-09-17T07:25:10.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:10 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1141520688' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T07:25:10.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:10 smithi135 ceph-mon[33238]: osdmap e506: 8 total, 8 up, 7 in 2024-09-17T07:25:10.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:10 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:25:10.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:10 smithi100 ceph-mon[29092]: pgmap v1100: 142 pgs: 1 peering, 1 active+clean+scrubbing, 140 active+clean; 477 KiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-17T07:25:10.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:10 smithi100 ceph-mon[29092]: 2.1a scrub starts 2024-09-17T07:25:10.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:10 smithi100 ceph-mon[29092]: 2.1a scrub ok 2024-09-17T07:25:10.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:10 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1141520688' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:25:10.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:10 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/48088441' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:25:10.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:10 smithi100 ceph-mon[29092]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T07:25:10.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:10 smithi100 ceph-mon[29092]: Cluster is now healthy 2024-09-17T07:25:10.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:10 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1141520688' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T07:25:10.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:10 smithi100 ceph-mon[29092]: osdmap e506: 8 total, 8 up, 7 in 2024-09-17T07:25:10.879 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:10 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:25:10.879 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:10 smithi100 ceph-mon[39462]: pgmap v1100: 142 pgs: 1 peering, 1 active+clean+scrubbing, 140 active+clean; 477 KiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-17T07:25:10.879 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:10 smithi100 ceph-mon[39462]: 2.1a scrub starts 2024-09-17T07:25:10.879 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:10 smithi100 ceph-mon[39462]: 2.1a scrub ok 2024-09-17T07:25:10.879 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:10 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1141520688' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:25:10.880 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:10 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/48088441' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-17T07:25:10.880 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:10 smithi100 ceph-mon[39462]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T07:25:10.880 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:10 smithi100 ceph-mon[39462]: Cluster is now healthy 2024-09-17T07:25:10.880 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:10 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1141520688' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T07:25:10.880 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:10 smithi100 ceph-mon[39462]: osdmap e506: 8 total, 8 up, 7 in 2024-09-17T07:25:11.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:11 smithi135 ceph-mon[33238]: 2.6 scrub starts 2024-09-17T07:25:11.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:11 smithi135 ceph-mon[33238]: 2.6 scrub ok 2024-09-17T07:25:11.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:11 smithi135 ceph-mon[33238]: 4.19 scrub starts 2024-09-17T07:25:11.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:11 smithi135 ceph-mon[33238]: 4.19 scrub ok 2024-09-17T07:25:11.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:11 smithi100 ceph-mon[29092]: 2.6 scrub starts 2024-09-17T07:25:11.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:11 smithi100 ceph-mon[29092]: 2.6 scrub ok 2024-09-17T07:25:11.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:11 smithi100 ceph-mon[29092]: 4.19 scrub starts 2024-09-17T07:25:11.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:11 smithi100 ceph-mon[29092]: 4.19 scrub ok 2024-09-17T07:25:11.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:11 smithi100 ceph-mon[39462]: 2.6 scrub starts 2024-09-17T07:25:11.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:11 smithi100 ceph-mon[39462]: 2.6 scrub ok 2024-09-17T07:25:11.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:11 smithi100 ceph-mon[39462]: 4.19 scrub starts 2024-09-17T07:25:11.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:11 smithi100 ceph-mon[39462]: 4.19 scrub ok 2024-09-17T07:25:12.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:12 smithi135 ceph-mon[33238]: 5.f scrub starts 2024-09-17T07:25:12.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:12 smithi135 ceph-mon[33238]: 5.f scrub ok 2024-09-17T07:25:12.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:12 smithi135 ceph-mon[33238]: pgmap v1103: 142 pgs: 1 peering, 141 active+clean; 477 KiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:25:12.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:12 smithi135 ceph-mon[33238]: 4.5 scrub starts 2024-09-17T07:25:12.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:12 smithi135 ceph-mon[33238]: 4.5 scrub ok 2024-09-17T07:25:12.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:12 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:25:12.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:12 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:25:12.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:12 smithi100 ceph-mon[29092]: 5.f scrub starts 2024-09-17T07:25:12.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:12 smithi100 ceph-mon[29092]: 5.f scrub ok 2024-09-17T07:25:12.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:12 smithi100 ceph-mon[29092]: pgmap v1103: 142 pgs: 1 peering, 141 active+clean; 477 KiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:25:12.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:12 smithi100 ceph-mon[29092]: 4.5 scrub starts 2024-09-17T07:25:12.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:12 smithi100 ceph-mon[29092]: 4.5 scrub ok 2024-09-17T07:25:12.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:12 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:25:12.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:12 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:25:12.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:12 smithi100 ceph-mon[39462]: 5.f scrub starts 2024-09-17T07:25:12.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:12 smithi100 ceph-mon[39462]: 5.f scrub ok 2024-09-17T07:25:12.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:12 smithi100 ceph-mon[39462]: pgmap v1103: 142 pgs: 1 peering, 141 active+clean; 477 KiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:25:12.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:12 smithi100 ceph-mon[39462]: 4.5 scrub starts 2024-09-17T07:25:12.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:12 smithi100 ceph-mon[39462]: 4.5 scrub ok 2024-09-17T07:25:12.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:12 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:25:12.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:12 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:25:13.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:13 smithi135 ceph-mon[33238]: osdmap e507: 8 total, 8 up, 7 in 2024-09-17T07:25:13.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:13 smithi135 ceph-mon[33238]: 1.6 scrub starts 2024-09-17T07:25:13.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:13 smithi135 ceph-mon[33238]: 1.6 scrub ok 2024-09-17T07:25:13.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:13 smithi100 ceph-mon[29092]: osdmap e507: 8 total, 8 up, 7 in 2024-09-17T07:25:13.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:13 smithi100 ceph-mon[29092]: 1.6 scrub starts 2024-09-17T07:25:13.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:13 smithi100 ceph-mon[29092]: 1.6 scrub ok 2024-09-17T07:25:13.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:13 smithi100 ceph-mon[39462]: osdmap e507: 8 total, 8 up, 7 in 2024-09-17T07:25:13.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:13 smithi100 ceph-mon[39462]: 1.6 scrub starts 2024-09-17T07:25:13.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:13 smithi100 ceph-mon[39462]: 1.6 scrub ok 2024-09-17T07:25:14.022 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-17T07:25:14.022 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:25:14.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:14 smithi100 ceph-mon[39462]: 6.1d scrub starts 2024-09-17T07:25:14.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:14 smithi100 ceph-mon[39462]: 6.1d scrub ok 2024-09-17T07:25:14.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:14 smithi100 ceph-mon[39462]: 4.d scrub starts 2024-09-17T07:25:14.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:14 smithi100 ceph-mon[39462]: 4.d scrub ok 2024-09-17T07:25:14.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:14 smithi100 ceph-mon[39462]: 5.1b scrub starts 2024-09-17T07:25:14.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:14 smithi100 ceph-mon[39462]: 5.1b scrub ok 2024-09-17T07:25:14.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:14 smithi100 ceph-mon[39462]: pgmap v1105: 142 pgs: 1 peering, 141 active+clean; 477 KiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 1 KiB/s rd, 1 op/s 2024-09-17T07:25:14.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:14 smithi100 ceph-mon[29092]: 6.1d scrub starts 2024-09-17T07:25:14.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:14 smithi100 ceph-mon[29092]: 6.1d scrub ok 2024-09-17T07:25:14.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:14 smithi100 ceph-mon[29092]: 4.d scrub starts 2024-09-17T07:25:14.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:14 smithi100 ceph-mon[29092]: 4.d scrub ok 2024-09-17T07:25:14.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:14 smithi100 ceph-mon[29092]: 5.1b scrub starts 2024-09-17T07:25:14.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:14 smithi100 ceph-mon[29092]: 5.1b scrub ok 2024-09-17T07:25:14.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:14 smithi100 ceph-mon[29092]: pgmap v1105: 142 pgs: 1 peering, 141 active+clean; 477 KiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 1 KiB/s rd, 1 op/s 2024-09-17T07:25:14.820 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-17T07:25:15.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:14 smithi135 ceph-mon[33238]: 6.1d scrub starts 2024-09-17T07:25:15.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:14 smithi135 ceph-mon[33238]: 6.1d scrub ok 2024-09-17T07:25:15.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:14 smithi135 ceph-mon[33238]: 4.d scrub starts 2024-09-17T07:25:15.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:14 smithi135 ceph-mon[33238]: 4.d scrub ok 2024-09-17T07:25:15.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:14 smithi135 ceph-mon[33238]: 5.1b scrub starts 2024-09-17T07:25:15.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:14 smithi135 ceph-mon[33238]: 5.1b scrub ok 2024-09-17T07:25:15.045 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:14 smithi135 ceph-mon[33238]: pgmap v1105: 142 pgs: 1 peering, 141 active+clean; 477 KiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 1 KiB/s rd, 1 op/s 2024-09-17T07:25:15.110 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T07:25:15.364 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:25:15.761 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:15 smithi100 ceph-mon[29092]: 5.a scrub starts 2024-09-17T07:25:15.762 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:15 smithi100 ceph-mon[29092]: 5.a scrub ok 2024-09-17T07:25:15.762 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:15 smithi100 ceph-mon[29092]: 3.1 scrub starts 2024-09-17T07:25:15.762 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:15 smithi100 ceph-mon[29092]: 3.1 scrub ok 2024-09-17T07:25:15.762 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:15 smithi100 ceph-mon[29092]: osdmap e508: 8 total, 8 up, 7 in 2024-09-17T07:25:15.762 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:15 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1185907794' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-17T07:25:15.762 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:15 smithi100 ceph-mon[39462]: 5.a scrub starts 2024-09-17T07:25:15.762 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:15 smithi100 ceph-mon[39462]: 5.a scrub ok 2024-09-17T07:25:15.762 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:15 smithi100 ceph-mon[39462]: 3.1 scrub starts 2024-09-17T07:25:15.762 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:15 smithi100 ceph-mon[39462]: 3.1 scrub ok 2024-09-17T07:25:15.763 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:15 smithi100 ceph-mon[39462]: osdmap e508: 8 total, 8 up, 7 in 2024-09-17T07:25:15.763 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:15 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1185907794' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-17T07:25:16.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:15 smithi135 ceph-mon[33238]: 5.a scrub starts 2024-09-17T07:25:16.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:15 smithi135 ceph-mon[33238]: 5.a scrub ok 2024-09-17T07:25:16.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:15 smithi135 ceph-mon[33238]: 3.1 scrub starts 2024-09-17T07:25:16.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:15 smithi135 ceph-mon[33238]: 3.1 scrub ok 2024-09-17T07:25:16.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:15 smithi135 ceph-mon[33238]: osdmap e508: 8 total, 8 up, 7 in 2024-09-17T07:25:16.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:15 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1185907794' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-17T07:25:16.126 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:25:15 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:25:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:25:16.387 INFO:teuthology.orchestra.run.smithi100.stderr:reweighted osd.7 to 1 (10000) 2024-09-17T07:25:16.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:16 smithi100 ceph-mon[29092]: 2.d scrub starts 2024-09-17T07:25:16.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:16 smithi100 ceph-mon[29092]: 2.d scrub ok 2024-09-17T07:25:16.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:16 smithi100 ceph-mon[29092]: 3.2 scrub starts 2024-09-17T07:25:16.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:16 smithi100 ceph-mon[29092]: 3.2 scrub ok 2024-09-17T07:25:16.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:16 smithi100 ceph-mon[29092]: pgmap v1107: 142 pgs: 142 active+clean; 477 KiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:25:16.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:16 smithi100 ceph-mon[29092]: 4.a scrub starts 2024-09-17T07:25:16.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:16 smithi100 ceph-mon[29092]: 4.a scrub ok 2024-09-17T07:25:16.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:16 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1185907794' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-17T07:25:16.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:16 smithi100 ceph-mon[29092]: osdmap e509: 8 total, 8 up, 7 in 2024-09-17T07:25:16.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:16 smithi100 ceph-mon[39462]: 2.d scrub starts 2024-09-17T07:25:16.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:16 smithi100 ceph-mon[39462]: 2.d scrub ok 2024-09-17T07:25:16.879 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:16 smithi100 ceph-mon[39462]: 3.2 scrub starts 2024-09-17T07:25:16.879 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:16 smithi100 ceph-mon[39462]: 3.2 scrub ok 2024-09-17T07:25:16.879 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:16 smithi100 ceph-mon[39462]: pgmap v1107: 142 pgs: 142 active+clean; 477 KiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:25:16.879 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:16 smithi100 ceph-mon[39462]: 4.a scrub starts 2024-09-17T07:25:16.879 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:16 smithi100 ceph-mon[39462]: 4.a scrub ok 2024-09-17T07:25:16.879 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:16 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1185907794' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-17T07:25:16.879 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:16 smithi100 ceph-mon[39462]: osdmap e509: 8 total, 8 up, 7 in 2024-09-17T07:25:17.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:16 smithi135 ceph-mon[33238]: 2.d scrub starts 2024-09-17T07:25:17.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:16 smithi135 ceph-mon[33238]: 2.d scrub ok 2024-09-17T07:25:17.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:16 smithi135 ceph-mon[33238]: 3.2 scrub starts 2024-09-17T07:25:17.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:16 smithi135 ceph-mon[33238]: 3.2 scrub ok 2024-09-17T07:25:17.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:16 smithi135 ceph-mon[33238]: pgmap v1107: 142 pgs: 142 active+clean; 477 KiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:25:17.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:16 smithi135 ceph-mon[33238]: 4.a scrub starts 2024-09-17T07:25:17.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:16 smithi135 ceph-mon[33238]: 4.a scrub ok 2024-09-17T07:25:17.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:16 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1185907794' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-17T07:25:17.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:16 smithi135 ceph-mon[33238]: osdmap e509: 8 total, 8 up, 7 in 2024-09-17T07:25:17.245 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:25:17.261 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-17T07:25:17.262 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T07:25:17.876 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:17 smithi100 ceph-mon[39462]: 6.9 scrub starts 2024-09-17T07:25:17.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:17 smithi100 ceph-mon[39462]: 6.9 scrub ok 2024-09-17T07:25:17.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:17 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:25:17.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:17 smithi100 ceph-mon[39462]: 4.0 scrub starts 2024-09-17T07:25:17.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:17 smithi100 ceph-mon[39462]: 4.0 scrub ok 2024-09-17T07:25:17.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:17 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:25:17.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:17 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:25:17.880 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:17 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:25:17.880 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:17 smithi100 ceph-mon[29092]: 6.9 scrub starts 2024-09-17T07:25:17.880 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:17 smithi100 ceph-mon[29092]: 6.9 scrub ok 2024-09-17T07:25:17.880 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:17 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:25:17.880 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:17 smithi100 ceph-mon[29092]: 4.0 scrub starts 2024-09-17T07:25:17.881 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:17 smithi100 ceph-mon[29092]: 4.0 scrub ok 2024-09-17T07:25:17.881 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:17 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:25:17.881 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:17 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:25:17.881 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:17 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:25:18.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:17 smithi135 ceph-mon[33238]: 6.9 scrub starts 2024-09-17T07:25:18.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:17 smithi135 ceph-mon[33238]: 6.9 scrub ok 2024-09-17T07:25:18.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:17 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:25:18.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:17 smithi135 ceph-mon[33238]: 4.0 scrub starts 2024-09-17T07:25:18.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:17 smithi135 ceph-mon[33238]: 4.0 scrub ok 2024-09-17T07:25:18.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:17 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:25:18.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:17 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:25:18.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:17 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:25:18.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:18 smithi100 ceph-mon[29092]: 5.e scrub starts 2024-09-17T07:25:18.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:18 smithi100 ceph-mon[29092]: 5.e scrub ok 2024-09-17T07:25:18.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:18 smithi100 ceph-mon[29092]: 2.15 scrub starts 2024-09-17T07:25:18.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:18 smithi100 ceph-mon[29092]: 2.15 scrub ok 2024-09-17T07:25:18.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:18 smithi100 ceph-mon[29092]: 2.18 scrub starts 2024-09-17T07:25:18.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:18 smithi100 ceph-mon[29092]: 2.18 scrub ok 2024-09-17T07:25:18.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:18 smithi100 ceph-mon[29092]: pgmap v1109: 142 pgs: 142 active+clean; 477 KiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:25:18.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:18 smithi100 ceph-mon[29092]: 6.13 deep-scrub starts 2024-09-17T07:25:18.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:18 smithi100 ceph-mon[29092]: 6.13 deep-scrub ok 2024-09-17T07:25:18.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:18 smithi100 ceph-mon[29092]: 6.1b scrub starts 2024-09-17T07:25:18.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:18 smithi100 ceph-mon[29092]: 6.1b scrub ok 2024-09-17T07:25:18.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:18 smithi100 ceph-mon[39462]: 5.e scrub starts 2024-09-17T07:25:18.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:18 smithi100 ceph-mon[39462]: 5.e scrub ok 2024-09-17T07:25:18.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:18 smithi100 ceph-mon[39462]: 2.15 scrub starts 2024-09-17T07:25:18.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:18 smithi100 ceph-mon[39462]: 2.15 scrub ok 2024-09-17T07:25:18.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:18 smithi100 ceph-mon[39462]: 2.18 scrub starts 2024-09-17T07:25:18.879 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:18 smithi100 ceph-mon[39462]: 2.18 scrub ok 2024-09-17T07:25:18.879 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:18 smithi100 ceph-mon[39462]: pgmap v1109: 142 pgs: 142 active+clean; 477 KiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:25:18.879 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:18 smithi100 ceph-mon[39462]: 6.13 deep-scrub starts 2024-09-17T07:25:18.879 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:18 smithi100 ceph-mon[39462]: 6.13 deep-scrub ok 2024-09-17T07:25:18.879 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:18 smithi100 ceph-mon[39462]: 6.1b scrub starts 2024-09-17T07:25:18.879 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:18 smithi100 ceph-mon[39462]: 6.1b scrub ok 2024-09-17T07:25:19.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:18 smithi135 ceph-mon[33238]: 5.e scrub starts 2024-09-17T07:25:19.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:18 smithi135 ceph-mon[33238]: 5.e scrub ok 2024-09-17T07:25:19.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:18 smithi135 ceph-mon[33238]: 2.15 scrub starts 2024-09-17T07:25:19.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:18 smithi135 ceph-mon[33238]: 2.15 scrub ok 2024-09-17T07:25:19.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:18 smithi135 ceph-mon[33238]: 2.18 scrub starts 2024-09-17T07:25:19.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:18 smithi135 ceph-mon[33238]: 2.18 scrub ok 2024-09-17T07:25:19.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:18 smithi135 ceph-mon[33238]: pgmap v1109: 142 pgs: 142 active+clean; 477 KiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:25:19.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:18 smithi135 ceph-mon[33238]: 6.13 deep-scrub starts 2024-09-17T07:25:19.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:18 smithi135 ceph-mon[33238]: 6.13 deep-scrub ok 2024-09-17T07:25:19.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:18 smithi135 ceph-mon[33238]: 6.1b scrub starts 2024-09-17T07:25:19.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:18 smithi135 ceph-mon[33238]: 6.1b scrub ok 2024-09-17T07:25:19.502 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:25:19.513 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-17T07:25:19.513 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:25:19.743 INFO:teuthology.orchestra.run.smithi100.stderr:noscrub is unset 2024-09-17T07:25:19.747 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:25:19 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:25:20.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:19 smithi135 ceph-mon[33238]: 6.1f deep-scrub starts 2024-09-17T07:25:20.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:19 smithi135 ceph-mon[33238]: 6.1f deep-scrub ok 2024-09-17T07:25:20.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:19 smithi135 ceph-mon[33238]: 2.13 scrub starts 2024-09-17T07:25:20.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:19 smithi135 ceph-mon[33238]: 2.13 scrub ok 2024-09-17T07:25:20.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:19 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:25:20.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:19 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:25:20.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:19 smithi135 ceph-mon[33238]: 6.2 scrub starts 2024-09-17T07:25:20.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:19 smithi135 ceph-mon[33238]: 6.2 scrub ok 2024-09-17T07:25:20.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:19 smithi135 ceph-mon[33238]: 5.14 scrub starts 2024-09-17T07:25:20.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:19 smithi135 ceph-mon[33238]: 5.14 scrub ok 2024-09-17T07:25:20.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:19 smithi135 ceph-mon[33238]: 2.16 scrub starts 2024-09-17T07:25:20.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:19 smithi135 ceph-mon[33238]: 2.16 scrub ok 2024-09-17T07:25:20.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:19 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/200392471' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:25:20.084 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:19 smithi100 ceph-mon[29092]: 6.1f deep-scrub starts 2024-09-17T07:25:20.085 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:19 smithi100 ceph-mon[29092]: 6.1f deep-scrub ok 2024-09-17T07:25:20.085 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:19 smithi100 ceph-mon[29092]: 2.13 scrub starts 2024-09-17T07:25:20.085 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:19 smithi100 ceph-mon[29092]: 2.13 scrub ok 2024-09-17T07:25:20.085 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:19 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:25:20.085 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:19 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:25:20.085 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:19 smithi100 ceph-mon[29092]: 6.2 scrub starts 2024-09-17T07:25:20.085 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:19 smithi100 ceph-mon[29092]: 6.2 scrub ok 2024-09-17T07:25:20.085 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:19 smithi100 ceph-mon[29092]: 5.14 scrub starts 2024-09-17T07:25:20.085 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:19 smithi100 ceph-mon[29092]: 5.14 scrub ok 2024-09-17T07:25:20.085 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:19 smithi100 ceph-mon[29092]: 2.16 scrub starts 2024-09-17T07:25:20.085 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:19 smithi100 ceph-mon[29092]: 2.16 scrub ok 2024-09-17T07:25:20.085 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:19 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/200392471' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:25:20.086 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:19 smithi100 ceph-mon[39462]: 6.1f deep-scrub starts 2024-09-17T07:25:20.086 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:19 smithi100 ceph-mon[39462]: 6.1f deep-scrub ok 2024-09-17T07:25:20.086 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:19 smithi100 ceph-mon[39462]: 2.13 scrub starts 2024-09-17T07:25:20.086 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:19 smithi100 ceph-mon[39462]: 2.13 scrub ok 2024-09-17T07:25:20.086 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:19 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:25:20.086 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:19 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:25:20.086 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:19 smithi100 ceph-mon[39462]: 6.2 scrub starts 2024-09-17T07:25:20.086 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:19 smithi100 ceph-mon[39462]: 6.2 scrub ok 2024-09-17T07:25:20.086 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:19 smithi100 ceph-mon[39462]: 5.14 scrub starts 2024-09-17T07:25:20.086 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:19 smithi100 ceph-mon[39462]: 5.14 scrub ok 2024-09-17T07:25:20.086 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:19 smithi100 ceph-mon[39462]: 2.16 scrub starts 2024-09-17T07:25:20.086 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:19 smithi100 ceph-mon[39462]: 2.16 scrub ok 2024-09-17T07:25:20.087 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:19 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/200392471' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-17T07:25:20.449 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-17T07:25:21.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:20 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:25:21.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:20 smithi135 ceph-mon[33238]: pgmap v1110: 142 pgs: 142 active+clean; 477 KiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-17T07:25:21.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:20 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/200392471' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T07:25:21.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:20 smithi135 ceph-mon[33238]: osdmap e510: 8 total, 8 up, 7 in 2024-09-17T07:25:21.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:20 smithi135 ceph-mon[33238]: 6.1e scrub starts 2024-09-17T07:25:21.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:20 smithi135 ceph-mon[33238]: 6.1e scrub ok 2024-09-17T07:25:21.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:20 smithi135 ceph-mon[33238]: 2.12 scrub starts 2024-09-17T07:25:21.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:20 smithi135 ceph-mon[33238]: 2.12 scrub ok 2024-09-17T07:25:21.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:20 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:25:21.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:20 smithi100 ceph-mon[29092]: pgmap v1110: 142 pgs: 142 active+clean; 477 KiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-17T07:25:21.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:20 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/200392471' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T07:25:21.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:20 smithi100 ceph-mon[29092]: osdmap e510: 8 total, 8 up, 7 in 2024-09-17T07:25:21.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:20 smithi100 ceph-mon[29092]: 6.1e scrub starts 2024-09-17T07:25:21.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:20 smithi100 ceph-mon[29092]: 6.1e scrub ok 2024-09-17T07:25:21.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:20 smithi100 ceph-mon[29092]: 2.12 scrub starts 2024-09-17T07:25:21.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:20 smithi100 ceph-mon[29092]: 2.12 scrub ok 2024-09-17T07:25:21.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:20 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:25:21.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:20 smithi100 ceph-mon[39462]: pgmap v1110: 142 pgs: 142 active+clean; 477 KiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-17T07:25:21.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:20 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/200392471' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T07:25:21.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:20 smithi100 ceph-mon[39462]: osdmap e510: 8 total, 8 up, 7 in 2024-09-17T07:25:21.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:20 smithi100 ceph-mon[39462]: 6.1e scrub starts 2024-09-17T07:25:21.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:20 smithi100 ceph-mon[39462]: 6.1e scrub ok 2024-09-17T07:25:21.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:20 smithi100 ceph-mon[39462]: 2.12 scrub starts 2024-09-17T07:25:21.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:20 smithi100 ceph-mon[39462]: 2.12 scrub ok 2024-09-17T07:25:21.142 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:25:21.573 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-17T07:25:21.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:21 smithi100 ceph-mon[29092]: 1.7 scrub starts 2024-09-17T07:25:21.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:21 smithi100 ceph-mon[29092]: 1.7 scrub ok 2024-09-17T07:25:21.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:21 smithi100 ceph-mon[29092]: 4.3 scrub starts 2024-09-17T07:25:21.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:21 smithi100 ceph-mon[29092]: 4.3 scrub ok 2024-09-17T07:25:21.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:21 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:25:21.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:21 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:25:21.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:21 smithi100 ceph-mon[29092]: 5.1c scrub starts 2024-09-17T07:25:21.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:21 smithi100 ceph-mon[29092]: 5.1c scrub ok 2024-09-17T07:25:21.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:21 smithi100 ceph-mon[39462]: 1.7 scrub starts 2024-09-17T07:25:21.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:21 smithi100 ceph-mon[39462]: 1.7 scrub ok 2024-09-17T07:25:21.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:21 smithi100 ceph-mon[39462]: 4.3 scrub starts 2024-09-17T07:25:21.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:21 smithi100 ceph-mon[39462]: 4.3 scrub ok 2024-09-17T07:25:21.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:21 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:25:21.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:21 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:25:21.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:21 smithi100 ceph-mon[39462]: 5.1c scrub starts 2024-09-17T07:25:21.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:21 smithi100 ceph-mon[39462]: 5.1c scrub ok 2024-09-17T07:25:22.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:21 smithi135 ceph-mon[33238]: 1.7 scrub starts 2024-09-17T07:25:22.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:21 smithi135 ceph-mon[33238]: 1.7 scrub ok 2024-09-17T07:25:22.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:21 smithi135 ceph-mon[33238]: 4.3 scrub starts 2024-09-17T07:25:22.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:21 smithi135 ceph-mon[33238]: 4.3 scrub ok 2024-09-17T07:25:22.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:21 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:25:22.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:21 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:25:22.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:21 smithi135 ceph-mon[33238]: 5.1c scrub starts 2024-09-17T07:25:22.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:21 smithi135 ceph-mon[33238]: 5.1c scrub ok 2024-09-17T07:25:23.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:22 smithi135 ceph-mon[33238]: pgmap v1112: 142 pgs: 142 active+clean; 477 KiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 835 B/s rd, 0 op/s 2024-09-17T07:25:23.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:22 smithi135 ceph-mon[33238]: osdmap e511: 8 total, 8 up, 7 in 2024-09-17T07:25:23.126 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:22 smithi100 ceph-mon[39462]: pgmap v1112: 142 pgs: 142 active+clean; 477 KiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 835 B/s rd, 0 op/s 2024-09-17T07:25:23.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:22 smithi100 ceph-mon[39462]: osdmap e511: 8 total, 8 up, 7 in 2024-09-17T07:25:23.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:22 smithi100 ceph-mon[29092]: pgmap v1112: 142 pgs: 142 active+clean; 477 KiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 835 B/s rd, 0 op/s 2024-09-17T07:25:23.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:22 smithi100 ceph-mon[29092]: osdmap e511: 8 total, 8 up, 7 in 2024-09-17T07:25:23.911 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T07:25:23.914 INFO:teuthology.orchestra.run.smithi100.stderr:dumped all 2024-09-17T07:25:24.019 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:23 smithi100 ceph-mon[29092]: 6.17 scrub starts 2024-09-17T07:25:24.019 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:23 smithi100 ceph-mon[29092]: 6.17 scrub ok 2024-09-17T07:25:24.019 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:23 smithi100 ceph-mon[39462]: 6.17 scrub starts 2024-09-17T07:25:24.019 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:23 smithi100 ceph-mon[39462]: 6.17 scrub ok 2024-09-17T07:25:24.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:23 smithi135 ceph-mon[33238]: 6.17 scrub starts 2024-09-17T07:25:24.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:23 smithi135 ceph-mon[33238]: 6.17 scrub ok 2024-09-17T07:25:24.720 INFO:teuthology.orchestra.run.smithi100.stdout:{"pg_ready":true,"pg_map":{"version":1114,"stamp":"2024-09-17T07:25:23.509677+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":488906,"num_objects":196,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":1972,"num_read_kb":1800,"num_write":1149,"num_write_kb":2019,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":274,"num_bytes_recovered":1455248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2573,"ondisk_log_size":2573,"up":426,"acting":426,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":431,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":6887708,"kb_used_data":655408,"kb_used_omap":16,"kb_used_meta":6231471,"kb_avail":649328356,"statfs":{"total":671965249536,"available":664912236544,"internally_reserved":0,"allocated":671137792,"data_stored":661154298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16844,"internal_metadata":6381026868},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.125134"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":99,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:20.087978+0000","last_change":"2024-09-17T07:25:17.976501+0000","last_active":"2024-09-17T07:25:20.087978+0000","last_peered":"2024-09-17T07:25:20.087978+0000","last_clean":"2024-09-17T07:25:20.087978+0000","last_became_active":"2024-09-17T07:22:13.586558+0000","last_became_peered":"2024-09-17T07:22:13.586558+0000","last_unstale":"2024-09-17T07:25:20.087978+0000","last_undegraded":"2024-09-17T07:25:20.087978+0000","last_fullsized":"2024-09-17T07:25:20.087978+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:17.976387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:33.644728+0000","last_clean_scrub_stamp":"2024-09-17T07:25:17.976387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:32.601619+0000","scrub_duration":0.0012687289999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":568,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-17T07:25:16.386791+0000","last_change":"2024-09-17T07:23:27.440583+0000","last_active":"2024-09-17T07:25:16.386791+0000","last_peered":"2024-09-17T07:25:16.386791+0000","last_clean":"2024-09-17T07:25:16.386791+0000","last_became_active":"2024-09-17T07:22:13.589527+0000","last_became_peered":"2024-09-17T07:22:13.589527+0000","last_unstale":"2024-09-17T07:25:16.386791+0000","last_undegraded":"2024-09-17T07:25:16.386791+0000","last_fullsized":"2024-09-17T07:25:16.386791+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:23:27.440509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:23:27.440509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001261002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"439'39","reported_seq":492,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-17T07:25:16.387191+0000","last_change":"2024-09-17T07:25:10.604396+0000","last_active":"2024-09-17T07:25:16.387191+0000","last_peered":"2024-09-17T07:25:16.387191+0000","last_clean":"2024-09-17T07:25:16.387191+0000","last_became_active":"2024-09-17T07:23:57.480684+0000","last_became_peered":"2024-09-17T07:23:57.480684+0000","last_unstale":"2024-09-17T07:25:16.387191+0000","last_undegraded":"2024-09-17T07:25:16.387191+0000","last_fullsized":"2024-09-17T07:25:16.387191+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'39","last_scrub_stamp":"2024-09-17T07:25:10.604305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:25:10.604305+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:38.118795+0000","scrub_duration":0.0033772899999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":538,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-09-17T07:25:15.889184+0000","last_change":"2024-09-17T07:25:04.885823+0000","last_active":"2024-09-17T07:25:15.889184+0000","last_peered":"2024-09-17T07:25:15.889184+0000","last_clean":"2024-09-17T07:25:15.889184+0000","last_became_active":"2024-09-17T07:21:15.354561+0000","last_became_peered":"2024-09-17T07:21:15.354561+0000","last_unstale":"2024-09-17T07:25:15.889184+0000","last_undegraded":"2024-09-17T07:25:15.889184+0000","last_fullsized":"2024-09-17T07:25:15.889184+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:04.885700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:25:04.885700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:21.499532+0000","scrub_duration":0.0034589249999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":404,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-17T07:25:16.387714+0000","last_change":"2024-09-17T07:23:57.483306+0000","last_active":"2024-09-17T07:25:16.387714+0000","last_peered":"2024-09-17T07:25:16.387714+0000","last_clean":"2024-09-17T07:25:16.387714+0000","last_became_active":"2024-09-17T07:23:57.480136+0000","last_became_peered":"2024-09-17T07:23:57.480136+0000","last_unstale":"2024-09-17T07:25:16.387714+0000","last_undegraded":"2024-09-17T07:25:16.387714+0000","last_fullsized":"2024-09-17T07:25:16.387714+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:21:13.454404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:17:35.507742+0000","last_clean_scrub_stamp":"2024-09-17T07:21:13.454404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.016029897000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"438'21","reported_seq":586,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-09-17T07:25:15.889219+0000","last_change":"2024-09-17T07:24:58.793239+0000","last_active":"2024-09-17T07:25:15.889219+0000","last_peered":"2024-09-17T07:25:15.889219+0000","last_clean":"2024-09-17T07:25:15.889219+0000","last_became_active":"2024-09-17T07:21:15.354718+0000","last_became_peered":"2024-09-17T07:21:15.354718+0000","last_unstale":"2024-09-17T07:25:15.889219+0000","last_undegraded":"2024-09-17T07:25:15.889219+0000","last_fullsized":"2024-09-17T07:25:15.889219+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"438'21","last_scrub_stamp":"2024-09-17T07:24:58.793136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:24:58.793136+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:13.595630+0000","scrub_duration":0.0018950810000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":421,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-09-17T07:25:15.383576+0000","last_change":"2024-09-17T07:23:57.473889+0000","last_active":"2024-09-17T07:25:15.383576+0000","last_peered":"2024-09-17T07:25:15.383576+0000","last_clean":"2024-09-17T07:25:15.383576+0000","last_became_active":"2024-09-17T07:23:57.473620+0000","last_became_peered":"2024-09-17T07:23:57.473620+0000","last_unstale":"2024-09-17T07:25:15.383576+0000","last_undegraded":"2024-09-17T07:25:15.383576+0000","last_fullsized":"2024-09-17T07:25:15.383576+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:13.660077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:20:13.660077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":552,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-09-17T07:25:15.889960+0000","last_change":"2024-09-17T07:24:18.740114+0000","last_active":"2024-09-17T07:25:15.889960+0000","last_peered":"2024-09-17T07:25:15.889960+0000","last_clean":"2024-09-17T07:25:15.889960+0000","last_became_active":"2024-09-17T07:23:57.482145+0000","last_became_peered":"2024-09-17T07:23:57.482145+0000","last_unstale":"2024-09-17T07:25:15.889960+0000","last_undegraded":"2024-09-17T07:25:15.889960+0000","last_fullsized":"2024-09-17T07:25:15.889960+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:24:18.740035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:24:18.740035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:23.015022+0000","scrub_duration":0.012092344,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":99,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:20.088159+0000","last_change":"2024-09-17T07:23:56.908306+0000","last_active":"2024-09-17T07:25:20.088159+0000","last_peered":"2024-09-17T07:25:20.088159+0000","last_clean":"2024-09-17T07:25:20.088159+0000","last_became_active":"2024-09-17T07:22:13.615512+0000","last_became_peered":"2024-09-17T07:22:13.615512+0000","last_unstale":"2024-09-17T07:25:20.088159+0000","last_undegraded":"2024-09-17T07:25:20.088159+0000","last_fullsized":"2024-09-17T07:25:20.088159+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:23:56.908208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:46.312371+0000","last_clean_scrub_stamp":"2024-09-17T07:23:56.908208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0012071619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"488'11","reported_seq":576,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-09-17T07:25:15.889236+0000","last_change":"2024-09-17T07:24:26.774186+0000","last_active":"2024-09-17T07:25:15.889236+0000","last_peered":"2024-09-17T07:25:15.889236+0000","last_clean":"2024-09-17T07:25:15.889236+0000","last_became_active":"2024-09-17T07:21:15.349235+0000","last_became_peered":"2024-09-17T07:21:15.349235+0000","last_unstale":"2024-09-17T07:25:15.889236+0000","last_undegraded":"2024-09-17T07:25:15.889236+0000","last_fullsized":"2024-09-17T07:25:15.889236+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"488'11","last_scrub_stamp":"2024-09-17T07:24:26.774097+0000","last_deep_scrub":"488'11","last_deep_scrub_stamp":"2024-09-17T07:24:26.774097+0000","last_clean_scrub_stamp":"2024-09-17T07:24:26.774097+0000","objects_scrubbed":2,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:40.430722+0000","scrub_duration":0.0025260220000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":13,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":88,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:20.087553+0000","last_change":"2024-09-17T07:22:13.624911+0000","last_active":"2024-09-17T07:25:20.087553+0000","last_peered":"2024-09-17T07:25:20.087553+0000","last_clean":"2024-09-17T07:25:20.087553+0000","last_became_active":"2024-09-17T07:22:13.624441+0000","last_became_peered":"2024-09-17T07:22:13.624441+0000","last_unstale":"2024-09-17T07:25:20.087553+0000","last_undegraded":"2024-09-17T07:25:20.087553+0000","last_fullsized":"2024-09-17T07:25:20.087553+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:10.116580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:20:10.116580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00137765,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":503,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-17T07:25:16.449520+0000","last_change":"2024-09-17T07:24:24.811636+0000","last_active":"2024-09-17T07:25:16.449520+0000","last_peered":"2024-09-17T07:25:16.449520+0000","last_clean":"2024-09-17T07:25:16.449520+0000","last_became_active":"2024-09-17T07:23:57.489317+0000","last_became_peered":"2024-09-17T07:23:57.489317+0000","last_unstale":"2024-09-17T07:25:16.449520+0000","last_undegraded":"2024-09-17T07:25:16.449520+0000","last_fullsized":"2024-09-17T07:25:16.449520+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:24:24.811522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:24:24.811522+0000","last_clean_scrub_stamp":"2024-09-17T07:24:24.811522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:36.491439+0000","scrub_duration":0.001695238,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":114,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:20.088496+0000","last_change":"2024-09-17T07:24:29.884052+0000","last_active":"2024-09-17T07:25:20.088496+0000","last_peered":"2024-09-17T07:25:20.088496+0000","last_clean":"2024-09-17T07:25:20.088496+0000","last_became_active":"2024-09-17T07:23:57.476413+0000","last_became_peered":"2024-09-17T07:23:57.476413+0000","last_unstale":"2024-09-17T07:25:20.088496+0000","last_undegraded":"2024-09-17T07:25:20.088496+0000","last_fullsized":"2024-09-17T07:25:20.088496+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:24:29.883839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:45.029247+0000","last_clean_scrub_stamp":"2024-09-17T07:24:29.883839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:50.216430+0000","scrub_duration":0.0022700179999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"439'25","reported_seq":603,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-09-17T07:25:15.890184+0000","last_change":"2024-09-17T07:20:39.070372+0000","last_active":"2024-09-17T07:25:15.890184+0000","last_peered":"2024-09-17T07:25:15.890184+0000","last_clean":"2024-09-17T07:25:15.890184+0000","last_became_active":"2024-09-17T07:09:24.397954+0000","last_became_peered":"2024-09-17T07:09:24.397954+0000","last_unstale":"2024-09-17T07:25:15.890184+0000","last_undegraded":"2024-09-17T07:25:15.890184+0000","last_fullsized":"2024-09-17T07:25:15.890184+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"312'19","last_scrub_stamp":"2024-09-17T07:20:39.070235+0000","last_deep_scrub":"312'19","last_deep_scrub_stamp":"2024-09-17T07:16:44.292911+0000","last_clean_scrub_stamp":"2024-09-17T07:20:39.070235+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0028570560000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":540,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-17T07:25:16.450091+0000","last_change":"2024-09-17T07:25:13.136654+0000","last_active":"2024-09-17T07:25:16.450091+0000","last_peered":"2024-09-17T07:25:16.450091+0000","last_clean":"2024-09-17T07:25:16.450091+0000","last_became_active":"2024-09-17T07:23:57.480812+0000","last_became_peered":"2024-09-17T07:23:57.480812+0000","last_unstale":"2024-09-17T07:25:16.450091+0000","last_undegraded":"2024-09-17T07:25:16.450091+0000","last_fullsized":"2024-09-17T07:25:16.450091+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:13.136552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:28.096266+0000","last_clean_scrub_stamp":"2024-09-17T07:25:13.136552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:27.238828+0000","scrub_duration":0.001451686,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":33,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-17T07:25:16.386709+0000","last_change":"2024-09-17T07:24:20.409990+0000","last_active":"2024-09-17T07:25:16.386709+0000","last_peered":"2024-09-17T07:25:16.386709+0000","last_clean":"2024-09-17T07:25:16.386709+0000","last_became_active":"2024-09-17T07:24:20.409247+0000","last_became_peered":"2024-09-17T07:24:20.409247+0000","last_unstale":"2024-09-17T07:25:16.386709+0000","last_undegraded":"2024-09-17T07:25:16.386709+0000","last_fullsized":"2024-09-17T07:25:16.386709+0000","mapping_epoch":487,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":488,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:27.474541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:20:27.474541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0061993300000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.8","version":"212'130","reported_seq":581,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-17T07:25:16.448822+0000","last_change":"2024-09-17T07:23:57.479148+0000","last_active":"2024-09-17T07:25:16.448822+0000","last_peered":"2024-09-17T07:25:16.448822+0000","last_clean":"2024-09-17T07:25:16.448822+0000","last_became_active":"2024-09-17T07:23:57.478874+0000","last_became_peered":"2024-09-17T07:23:57.478874+0000","last_unstale":"2024-09-17T07:25:16.448822+0000","last_undegraded":"2024-09-17T07:25:16.448822+0000","last_fullsized":"2024-09-17T07:25:16.448822+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":4,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:20:21.626089+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:14:57.052505+0000","last_clean_scrub_stamp":"2024-09-17T07:20:21.626089+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021141889999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":28,"num_write":16,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":161009,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"439'53","reported_seq":630,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-09-17T07:25:15.889476+0000","last_change":"2024-09-17T07:25:12.896735+0000","last_active":"2024-09-17T07:25:15.889476+0000","last_peered":"2024-09-17T07:25:15.889476+0000","last_clean":"2024-09-17T07:25:15.889476+0000","last_became_active":"2024-09-17T07:21:15.348413+0000","last_became_peered":"2024-09-17T07:21:15.348413+0000","last_unstale":"2024-09-17T07:25:15.889476+0000","last_undegraded":"2024-09-17T07:25:15.889476+0000","last_fullsized":"2024-09-17T07:25:15.889476+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'53","last_scrub_stamp":"2024-09-17T07:25:12.896608+0000","last_deep_scrub":"95'25","last_deep_scrub_stamp":"2024-09-17T07:04:09.450039+0000","last_clean_scrub_stamp":"2024-09-17T07:25:12.896608+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:18.556302+0000","scrub_duration":0.0026638410000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":61,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":114,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:20.088320+0000","last_change":"2024-09-17T07:25:08.851793+0000","last_active":"2024-09-17T07:25:20.088320+0000","last_peered":"2024-09-17T07:25:20.088320+0000","last_clean":"2024-09-17T07:25:20.088320+0000","last_became_active":"2024-09-17T07:23:57.475614+0000","last_became_peered":"2024-09-17T07:23:57.475614+0000","last_unstale":"2024-09-17T07:25:20.088320+0000","last_undegraded":"2024-09-17T07:25:20.088320+0000","last_fullsized":"2024-09-17T07:25:20.088320+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:08.851702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:25:08.851702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:37.060045+0000","scrub_duration":0.00137217,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":371,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-09-17T07:25:15.384607+0000","last_change":"2024-09-17T07:24:25.977606+0000","last_active":"2024-09-17T07:25:15.384607+0000","last_peered":"2024-09-17T07:25:15.384607+0000","last_clean":"2024-09-17T07:25:15.384607+0000","last_became_active":"2024-09-17T07:21:15.349538+0000","last_became_peered":"2024-09-17T07:21:15.349538+0000","last_unstale":"2024-09-17T07:25:15.384607+0000","last_undegraded":"2024-09-17T07:25:15.384607+0000","last_fullsized":"2024-09-17T07:25:15.384607+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:24:25.974726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:24:25.974726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:55.688544+0000","scrub_duration":0.014578028999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":391,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-17T07:25:16.384480+0000","last_change":"2024-09-17T07:20:40.117298+0000","last_active":"2024-09-17T07:25:16.384480+0000","last_peered":"2024-09-17T07:25:16.384480+0000","last_clean":"2024-09-17T07:25:16.384480+0000","last_became_active":"2024-09-17T07:09:24.396334+0000","last_became_peered":"2024-09-17T07:09:24.396334+0000","last_unstale":"2024-09-17T07:25:16.384480+0000","last_undegraded":"2024-09-17T07:25:16.384480+0000","last_fullsized":"2024-09-17T07:25:16.384480+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:40.117165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:42.984088+0000","last_clean_scrub_stamp":"2024-09-17T07:20:40.117165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0041399059999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"212'130","reported_seq":758,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:20.087513+0000","last_change":"2024-09-17T07:23:57.476746+0000","last_active":"2024-09-17T07:25:20.087513+0000","last_peered":"2024-09-17T07:25:20.087513+0000","last_clean":"2024-09-17T07:25:20.087513+0000","last_became_active":"2024-09-17T07:23:57.476423+0000","last_became_peered":"2024-09-17T07:23:57.476423+0000","last_unstale":"2024-09-17T07:25:20.087513+0000","last_undegraded":"2024-09-17T07:25:20.087513+0000","last_fullsized":"2024-09-17T07:25:20.087513+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":4,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:21:12.403277+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:14:57.052505+0000","last_clean_scrub_stamp":"2024-09-17T07:21:12.403277+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0071578960000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":28,"num_write":16,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":161009,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"439'35","reported_seq":624,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-09-17T07:25:15.889217+0000","last_change":"2024-09-17T07:24:29.751049+0000","last_active":"2024-09-17T07:25:15.889217+0000","last_peered":"2024-09-17T07:25:15.889217+0000","last_clean":"2024-09-17T07:25:15.889217+0000","last_became_active":"2024-09-17T07:21:15.347129+0000","last_became_peered":"2024-09-17T07:21:15.347129+0000","last_unstale":"2024-09-17T07:25:15.889217+0000","last_undegraded":"2024-09-17T07:25:15.889217+0000","last_fullsized":"2024-09-17T07:25:15.889217+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'35","last_scrub_stamp":"2024-09-17T07:24:29.750888+0000","last_deep_scrub":"439'35","last_deep_scrub_stamp":"2024-09-17T07:24:29.750888+0000","last_clean_scrub_stamp":"2024-09-17T07:24:29.750888+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:46.618846+0000","scrub_duration":0.0097756200000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":419,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-09-17T07:25:15.382902+0000","last_change":"2024-09-17T07:23:57.493626+0000","last_active":"2024-09-17T07:25:15.382902+0000","last_peered":"2024-09-17T07:25:15.382902+0000","last_clean":"2024-09-17T07:25:15.382902+0000","last_became_active":"2024-09-17T07:23:57.493354+0000","last_became_peered":"2024-09-17T07:23:57.493354+0000","last_unstale":"2024-09-17T07:25:15.382902+0000","last_undegraded":"2024-09-17T07:25:15.382902+0000","last_fullsized":"2024-09-17T07:25:15.382902+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:11.580871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:20:11.580871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0065215869999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"478'13","reported_seq":158,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:20.783970+0000","last_change":"2024-09-17T07:24:25.818322+0000","last_active":"2024-09-17T07:25:20.783970+0000","last_peered":"2024-09-17T07:25:20.783970+0000","last_clean":"2024-09-17T07:25:20.783970+0000","last_became_active":"2024-09-17T07:23:57.480120+0000","last_became_peered":"2024-09-17T07:23:57.480120+0000","last_unstale":"2024-09-17T07:25:20.783970+0000","last_undegraded":"2024-09-17T07:25:20.783970+0000","last_fullsized":"2024-09-17T07:25:20.783970+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"478'13","last_scrub_stamp":"2024-09-17T07:24:25.815560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:24:25.815560+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:54.513977+0000","scrub_duration":0.0054701139999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":550,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-09-17T07:25:15.889548+0000","last_change":"2024-09-17T07:23:55.705234+0000","last_active":"2024-09-17T07:25:15.889548+0000","last_peered":"2024-09-17T07:25:15.889548+0000","last_clean":"2024-09-17T07:25:15.889548+0000","last_became_active":"2024-09-17T07:21:15.348799+0000","last_became_peered":"2024-09-17T07:21:15.348799+0000","last_unstale":"2024-09-17T07:25:15.889548+0000","last_undegraded":"2024-09-17T07:25:15.889548+0000","last_fullsized":"2024-09-17T07:25:15.889548+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:23:55.705025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:14:50.554591+0000","last_clean_scrub_stamp":"2024-09-17T07:23:55.705025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:22.420935+0000","scrub_duration":0.0058854049999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"212'130","reported_seq":582,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-17T07:25:16.450972+0000","last_change":"2024-09-17T07:23:57.494642+0000","last_active":"2024-09-17T07:25:16.450972+0000","last_peered":"2024-09-17T07:25:16.450972+0000","last_clean":"2024-09-17T07:25:16.450972+0000","last_became_active":"2024-09-17T07:23:57.494014+0000","last_became_peered":"2024-09-17T07:23:57.494014+0000","last_unstale":"2024-09-17T07:25:16.450972+0000","last_undegraded":"2024-09-17T07:25:16.450972+0000","last_fullsized":"2024-09-17T07:25:16.450972+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":4,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:20:33.709901+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:14:57.052505+0000","last_clean_scrub_stamp":"2024-09-17T07:20:33.709901+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021141889999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":27,"num_write":16,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":161009,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"439'39","reported_seq":492,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-09-17T07:25:15.385018+0000","last_change":"2024-09-17T07:22:13.780063+0000","last_active":"2024-09-17T07:25:15.385018+0000","last_peered":"2024-09-17T07:25:15.385018+0000","last_clean":"2024-09-17T07:25:15.385018+0000","last_became_active":"2024-09-17T07:22:13.589782+0000","last_became_peered":"2024-09-17T07:22:13.589782+0000","last_unstale":"2024-09-17T07:25:15.385018+0000","last_undegraded":"2024-09-17T07:25:15.385018+0000","last_fullsized":"2024-09-17T07:25:15.385018+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'27","last_scrub_stamp":"2024-09-17T07:20:13.130718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:20:13.130718+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0081162040000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":418,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-09-17T07:25:15.383697+0000","last_change":"2024-09-17T07:22:13.592519+0000","last_active":"2024-09-17T07:25:15.383697+0000","last_peered":"2024-09-17T07:25:15.383697+0000","last_clean":"2024-09-17T07:25:15.383697+0000","last_became_active":"2024-09-17T07:22:13.591089+0000","last_became_peered":"2024-09-17T07:22:13.591089+0000","last_unstale":"2024-09-17T07:25:15.383697+0000","last_undegraded":"2024-09-17T07:25:15.383697+0000","last_fullsized":"2024-09-17T07:25:15.383697+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:14.001258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:20:14.001258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034116369999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"478'10","reported_seq":847,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-17T07:25:16.842095+0000","last_change":"2024-09-17T07:25:16.842178+0000","last_active":"2024-09-17T07:25:16.842095+0000","last_peered":"2024-09-17T07:25:16.842095+0000","last_clean":"2024-09-17T07:25:16.842095+0000","last_became_active":"2024-09-17T07:23:57.473239+0000","last_became_peered":"2024-09-17T07:23:57.473239+0000","last_unstale":"2024-09-17T07:25:16.842095+0000","last_undegraded":"2024-09-17T07:25:16.842095+0000","last_fullsized":"2024-09-17T07:25:16.842095+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"478'10","last_scrub_stamp":"2024-09-17T07:25:16.842081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:25:16.842081+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:28.575493+0000","scrub_duration":0.001742511,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":494,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-17T07:25:16.451280+0000","last_change":"2024-09-17T07:24:29.971120+0000","last_active":"2024-09-17T07:25:16.451280+0000","last_peered":"2024-09-17T07:25:16.451280+0000","last_clean":"2024-09-17T07:25:16.451280+0000","last_became_active":"2024-09-17T07:23:57.522025+0000","last_became_peered":"2024-09-17T07:23:57.522025+0000","last_unstale":"2024-09-17T07:25:16.451280+0000","last_undegraded":"2024-09-17T07:25:16.451280+0000","last_fullsized":"2024-09-17T07:25:16.451280+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:24:29.971009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:24:29.971009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:39.187110+0000","scrub_duration":0.002296789,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"488'32","reported_seq":601,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-09-17T07:25:15.889386+0000","last_change":"2024-09-17T07:24:27.786116+0000","last_active":"2024-09-17T07:25:15.889386+0000","last_peered":"2024-09-17T07:25:15.889386+0000","last_clean":"2024-09-17T07:25:15.889386+0000","last_became_active":"2024-09-17T07:21:15.347662+0000","last_became_peered":"2024-09-17T07:21:15.347662+0000","last_unstale":"2024-09-17T07:25:15.889386+0000","last_undegraded":"2024-09-17T07:25:15.889386+0000","last_fullsized":"2024-09-17T07:25:15.889386+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"488'32","last_scrub_stamp":"2024-09-17T07:24:27.785661+0000","last_deep_scrub":"488'32","last_deep_scrub_stamp":"2024-09-17T07:24:27.785661+0000","last_clean_scrub_stamp":"2024-09-17T07:24:27.785661+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:32.181225+0000","scrub_duration":0.021517458999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":88,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:20.087428+0000","last_change":"2024-09-17T07:22:13.711817+0000","last_active":"2024-09-17T07:25:20.087428+0000","last_peered":"2024-09-17T07:25:20.087428+0000","last_clean":"2024-09-17T07:25:20.087428+0000","last_became_active":"2024-09-17T07:22:13.709873+0000","last_became_peered":"2024-09-17T07:22:13.709873+0000","last_unstale":"2024-09-17T07:25:20.087428+0000","last_undegraded":"2024-09-17T07:25:20.087428+0000","last_fullsized":"2024-09-17T07:25:20.087428+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:12.442421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:20:12.442421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026862600000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":516,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-17T07:25:16.451135+0000","last_change":"2024-09-17T07:25:11.133451+0000","last_active":"2024-09-17T07:25:16.451135+0000","last_peered":"2024-09-17T07:25:16.451135+0000","last_clean":"2024-09-17T07:25:16.451135+0000","last_became_active":"2024-09-17T07:23:57.478625+0000","last_became_peered":"2024-09-17T07:23:57.478625+0000","last_unstale":"2024-09-17T07:25:16.451135+0000","last_undegraded":"2024-09-17T07:25:16.451135+0000","last_fullsized":"2024-09-17T07:25:16.451135+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:11.133339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:25:11.133339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:35.425563+0000","scrub_duration":0.0016371350000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":452,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-09-17T07:25:15.609918+0000","last_change":"2024-09-17T07:24:26.583863+0000","last_active":"2024-09-17T07:25:15.609918+0000","last_peered":"2024-09-17T07:25:15.609918+0000","last_clean":"2024-09-17T07:25:15.609918+0000","last_became_active":"2024-09-17T07:23:57.486204+0000","last_became_peered":"2024-09-17T07:23:57.486204+0000","last_unstale":"2024-09-17T07:25:15.609918+0000","last_undegraded":"2024-09-17T07:25:15.609918+0000","last_fullsized":"2024-09-17T07:25:15.609918+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:24:26.583745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:13:22.187127+0000","last_clean_scrub_stamp":"2024-09-17T07:24:26.583745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:46.250777+0000","scrub_duration":0.0024301169999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"212'130","reported_seq":619,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-17T07:25:16.449987+0000","last_change":"2024-09-17T07:23:57.484193+0000","last_active":"2024-09-17T07:25:16.449987+0000","last_peered":"2024-09-17T07:25:16.449987+0000","last_clean":"2024-09-17T07:25:16.449987+0000","last_became_active":"2024-09-17T07:23:57.478159+0000","last_became_peered":"2024-09-17T07:23:57.478159+0000","last_unstale":"2024-09-17T07:25:16.449987+0000","last_undegraded":"2024-09-17T07:25:16.449987+0000","last_fullsized":"2024-09-17T07:25:16.449987+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":3,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:20:40.628770+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:14:57.052505+0000","last_clean_scrub_stamp":"2024-09-17T07:20:40.628770+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021141889999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":28,"num_write":16,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":161010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1","version":"439'42","reported_seq":615,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-09-17T07:25:15.889897+0000","last_change":"2024-09-17T07:23:57.482189+0000","last_active":"2024-09-17T07:25:15.889897+0000","last_peered":"2024-09-17T07:25:15.889897+0000","last_clean":"2024-09-17T07:25:15.889897+0000","last_became_active":"2024-09-17T07:23:57.481780+0000","last_became_peered":"2024-09-17T07:23:57.481780+0000","last_unstale":"2024-09-17T07:25:15.889897+0000","last_undegraded":"2024-09-17T07:25:15.889897+0000","last_fullsized":"2024-09-17T07:25:15.889897+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'28","last_scrub_stamp":"2024-09-17T07:20:28.029279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:20:28.029279+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0079911889999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":63,"num_read_kb":56,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":620,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-09-17T07:25:15.610376+0000","last_change":"2024-09-17T07:25:07.568509+0000","last_active":"2024-09-17T07:25:15.610376+0000","last_peered":"2024-09-17T07:25:15.610376+0000","last_clean":"2024-09-17T07:25:15.610376+0000","last_became_active":"2024-09-17T07:22:13.582096+0000","last_became_peered":"2024-09-17T07:22:13.582096+0000","last_unstale":"2024-09-17T07:25:15.610376+0000","last_undegraded":"2024-09-17T07:25:15.610376+0000","last_fullsized":"2024-09-17T07:25:15.610376+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:07.568411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:09:23.181028+0000","last_clean_scrub_stamp":"2024-09-17T07:25:07.568411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:31.713937+0000","scrub_duration":0.0014008320000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"414'11","reported_seq":396,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-09-17T07:25:15.782643+0000","last_change":"2024-09-17T07:21:15.356155+0000","last_active":"2024-09-17T07:25:15.782643+0000","last_peered":"2024-09-17T07:25:15.782643+0000","last_clean":"2024-09-17T07:25:15.782643+0000","last_became_active":"2024-09-17T07:21:15.355651+0000","last_became_peered":"2024-09-17T07:21:15.355651+0000","last_unstale":"2024-09-17T07:25:15.782643+0000","last_undegraded":"2024-09-17T07:25:15.782643+0000","last_fullsized":"2024-09-17T07:25:15.782643+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'9","last_scrub_stamp":"2024-09-17T07:15:55.813882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:15:55.813882+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":100,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:20.087869+0000","last_change":"2024-09-17T07:25:00.872918+0000","last_active":"2024-09-17T07:25:20.087869+0000","last_peered":"2024-09-17T07:25:20.087869+0000","last_clean":"2024-09-17T07:25:20.087869+0000","last_became_active":"2024-09-17T07:22:13.568656+0000","last_became_peered":"2024-09-17T07:22:13.568656+0000","last_unstale":"2024-09-17T07:25:20.087869+0000","last_undegraded":"2024-09-17T07:25:20.087869+0000","last_fullsized":"2024-09-17T07:25:20.087869+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:00.872840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:10:33.727472+0000","last_clean_scrub_stamp":"2024-09-17T07:25:00.872840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:27.841417+0000","scrub_duration":0.001230515,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"212'130","reported_seq":604,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-09-17T07:25:15.890229+0000","last_change":"2024-09-17T07:20:35.080459+0000","last_active":"2024-09-17T07:25:15.890229+0000","last_peered":"2024-09-17T07:25:15.890229+0000","last_clean":"2024-09-17T07:25:15.890229+0000","last_became_active":"2024-09-17T07:19:35.420337+0000","last_became_peered":"2024-09-17T07:19:35.420337+0000","last_unstale":"2024-09-17T07:25:15.890229+0000","last_undegraded":"2024-09-17T07:25:15.890229+0000","last_fullsized":"2024-09-17T07:25:15.890229+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":3,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:20:35.080257+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:20:35.080257+0000","last_clean_scrub_stamp":"2024-09-17T07:20:35.080257+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0030477730000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"439'42","reported_seq":135,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:20.087277+0000","last_change":"2024-09-17T07:25:16.935826+0000","last_active":"2024-09-17T07:25:20.087277+0000","last_peered":"2024-09-17T07:25:20.087277+0000","last_clean":"2024-09-17T07:25:20.087277+0000","last_became_active":"2024-09-17T07:22:13.711444+0000","last_became_peered":"2024-09-17T07:22:13.711444+0000","last_unstale":"2024-09-17T07:25:20.087277+0000","last_undegraded":"2024-09-17T07:25:20.087277+0000","last_fullsized":"2024-09-17T07:25:20.087277+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'42","last_scrub_stamp":"2024-09-17T07:25:16.935731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:25:16.935731+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:23.547655+0000","scrub_duration":0.0023825790000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":68,"num_read_kb":56,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":425,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-09-17T07:25:15.384364+0000","last_change":"2024-09-17T07:25:09.880351+0000","last_active":"2024-09-17T07:25:15.384364+0000","last_peered":"2024-09-17T07:25:15.384364+0000","last_clean":"2024-09-17T07:25:15.384364+0000","last_became_active":"2024-09-17T07:21:15.347867+0000","last_became_peered":"2024-09-17T07:21:15.347867+0000","last_unstale":"2024-09-17T07:25:15.384364+0000","last_undegraded":"2024-09-17T07:25:15.384364+0000","last_fullsized":"2024-09-17T07:25:15.384364+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:09.880264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:56.848689+0000","last_clean_scrub_stamp":"2024-09-17T07:25:09.880264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:18.052765+0000","scrub_duration":0.0014177599999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":573,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-09-17T07:25:15.889710+0000","last_change":"2024-09-17T07:23:27.713373+0000","last_active":"2024-09-17T07:25:15.889710+0000","last_peered":"2024-09-17T07:25:15.889710+0000","last_clean":"2024-09-17T07:25:15.889710+0000","last_became_active":"2024-09-17T07:22:13.566916+0000","last_became_peered":"2024-09-17T07:22:13.566916+0000","last_unstale":"2024-09-17T07:25:15.889710+0000","last_undegraded":"2024-09-17T07:25:15.889710+0000","last_fullsized":"2024-09-17T07:25:15.889710+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:23:27.713294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:23:27.713294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0011838949999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":542,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-17T07:25:16.383941+0000","last_change":"2024-09-17T07:19:27.090311+0000","last_active":"2024-09-17T07:25:16.383941+0000","last_peered":"2024-09-17T07:25:16.383941+0000","last_clean":"2024-09-17T07:25:16.383941+0000","last_became_active":"2024-09-17T07:09:24.398659+0000","last_became_peered":"2024-09-17T07:09:24.398659+0000","last_unstale":"2024-09-17T07:25:16.383941+0000","last_undegraded":"2024-09-17T07:25:16.383941+0000","last_fullsized":"2024-09-17T07:25:16.383941+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:19:27.090201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:19:27.090201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026497019999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"212'130","reported_seq":623,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-09-17T07:25:15.610144+0000","last_change":"2024-09-17T07:25:12.680354+0000","last_active":"2024-09-17T07:25:15.610144+0000","last_peered":"2024-09-17T07:25:15.610144+0000","last_clean":"2024-09-17T07:25:15.610144+0000","last_became_active":"2024-09-17T07:19:35.428161+0000","last_became_peered":"2024-09-17T07:19:35.428161+0000","last_unstale":"2024-09-17T07:25:15.610144+0000","last_undegraded":"2024-09-17T07:25:15.610144+0000","last_fullsized":"2024-09-17T07:25:15.610144+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":3,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:25:12.680248+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:14:57.052505+0000","last_clean_scrub_stamp":"2024-09-17T07:25:12.680248+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:41.650768+0000","scrub_duration":0.001644413,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"439'39","reported_seq":684,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-17T07:25:16.387263+0000","last_change":"2024-09-17T07:23:57.498662+0000","last_active":"2024-09-17T07:25:16.387263+0000","last_peered":"2024-09-17T07:25:16.387263+0000","last_clean":"2024-09-17T07:25:16.387263+0000","last_became_active":"2024-09-17T07:23:57.497347+0000","last_became_peered":"2024-09-17T07:23:57.497347+0000","last_unstale":"2024-09-17T07:25:16.387263+0000","last_undegraded":"2024-09-17T07:25:16.387263+0000","last_fullsized":"2024-09-17T07:25:16.387263+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'29","last_scrub_stamp":"2024-09-17T07:19:35.438907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:19:35.438907+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010401474000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":99,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:20.088010+0000","last_change":"2024-09-17T07:25:07.874505+0000","last_active":"2024-09-17T07:25:20.088010+0000","last_peered":"2024-09-17T07:25:20.088010+0000","last_clean":"2024-09-17T07:25:20.088010+0000","last_became_active":"2024-09-17T07:22:13.624802+0000","last_became_peered":"2024-09-17T07:22:13.624802+0000","last_unstale":"2024-09-17T07:25:20.088010+0000","last_undegraded":"2024-09-17T07:25:20.088010+0000","last_fullsized":"2024-09-17T07:25:20.088010+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:07.874421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:47.631387+0000","last_clean_scrub_stamp":"2024-09-17T07:25:07.874421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:23.997699+0000","scrub_duration":0.0012562529999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":581,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-09-17T07:25:15.609594+0000","last_change":"2024-09-17T07:25:05.604387+0000","last_active":"2024-09-17T07:25:15.609594+0000","last_peered":"2024-09-17T07:25:15.609594+0000","last_clean":"2024-09-17T07:25:15.609594+0000","last_became_active":"2024-09-17T07:23:57.485512+0000","last_became_peered":"2024-09-17T07:23:57.485512+0000","last_unstale":"2024-09-17T07:25:15.609594+0000","last_undegraded":"2024-09-17T07:25:15.609594+0000","last_fullsized":"2024-09-17T07:25:15.609594+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:05.604241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:11:17.583712+0000","last_clean_scrub_stamp":"2024-09-17T07:25:05.604241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:12.778226+0000","scrub_duration":0.0020187059999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":380,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-09-17T07:25:15.383507+0000","last_change":"2024-09-17T07:24:19.018167+0000","last_active":"2024-09-17T07:25:15.383507+0000","last_peered":"2024-09-17T07:25:15.383507+0000","last_clean":"2024-09-17T07:25:15.383507+0000","last_became_active":"2024-09-17T07:21:15.351524+0000","last_became_peered":"2024-09-17T07:21:15.351524+0000","last_unstale":"2024-09-17T07:25:15.383507+0000","last_undegraded":"2024-09-17T07:25:15.383507+0000","last_fullsized":"2024-09-17T07:25:15.383507+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:24:19.018060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:24:19.018060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:31.477457+0000","scrub_duration":0.0014215090000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"212'130","reported_seq":676,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-09-17T07:25:15.610143+0000","last_change":"2024-09-17T07:23:57.487301+0000","last_active":"2024-09-17T07:25:15.610143+0000","last_peered":"2024-09-17T07:25:15.610143+0000","last_clean":"2024-09-17T07:25:15.610143+0000","last_became_active":"2024-09-17T07:23:57.486189+0000","last_became_peered":"2024-09-17T07:23:57.486189+0000","last_unstale":"2024-09-17T07:25:15.610143+0000","last_undegraded":"2024-09-17T07:25:15.610143+0000","last_fullsized":"2024-09-17T07:25:15.610143+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":3,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:19:19.991044+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:14:57.052505+0000","last_clean_scrub_stamp":"2024-09-17T07:19:19.991044+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021141889999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":28,"num_write":16,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":161009,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"439'35","reported_seq":461,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-09-17T07:25:15.382784+0000","last_change":"2024-09-17T07:24:31.049397+0000","last_active":"2024-09-17T07:25:15.382784+0000","last_peered":"2024-09-17T07:25:15.382784+0000","last_clean":"2024-09-17T07:25:15.382784+0000","last_became_active":"2024-09-17T07:23:57.478340+0000","last_became_peered":"2024-09-17T07:23:57.478340+0000","last_unstale":"2024-09-17T07:25:15.382784+0000","last_undegraded":"2024-09-17T07:25:15.382784+0000","last_fullsized":"2024-09-17T07:25:15.382784+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'35","last_scrub_stamp":"2024-09-17T07:24:31.049269+0000","last_deep_scrub":"110'15","last_deep_scrub_stamp":"2024-09-17T07:04:57.091400+0000","last_clean_scrub_stamp":"2024-09-17T07:24:31.049269+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:39.791093+0000","scrub_duration":0.01178889,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":394,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-09-17T07:25:15.384685+0000","last_change":"2024-09-17T07:22:13.592639+0000","last_active":"2024-09-17T07:25:15.384685+0000","last_peered":"2024-09-17T07:25:15.384685+0000","last_clean":"2024-09-17T07:25:15.384685+0000","last_became_active":"2024-09-17T07:22:13.591372+0000","last_became_peered":"2024-09-17T07:22:13.591372+0000","last_unstale":"2024-09-17T07:25:15.384685+0000","last_undegraded":"2024-09-17T07:25:15.384685+0000","last_fullsized":"2024-09-17T07:25:15.384685+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:24.437342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:25.147878+0000","last_clean_scrub_stamp":"2024-09-17T07:20:24.437342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0024354329999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"478'11","reported_seq":849,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-17T07:25:16.387091+0000","last_change":"2024-09-17T07:23:57.497400+0000","last_active":"2024-09-17T07:25:16.387091+0000","last_peered":"2024-09-17T07:25:16.387091+0000","last_clean":"2024-09-17T07:25:16.387091+0000","last_became_active":"2024-09-17T07:23:57.496882+0000","last_became_peered":"2024-09-17T07:23:57.496882+0000","last_unstale":"2024-09-17T07:25:16.387091+0000","last_undegraded":"2024-09-17T07:25:16.387091+0000","last_fullsized":"2024-09-17T07:25:16.387091+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'9","last_scrub_stamp":"2024-09-17T07:21:14.471255+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-09-17T07:15:53.512777+0000","last_clean_scrub_stamp":"2024-09-17T07:21:14.471255+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032915969999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"268'2","reported_seq":574,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-17T07:25:16.386810+0000","last_change":"2024-09-17T07:25:08.624593+0000","last_active":"2024-09-17T07:25:16.386810+0000","last_peered":"2024-09-17T07:25:16.386810+0000","last_clean":"2024-09-17T07:25:16.386810+0000","last_became_active":"2024-09-17T07:23:57.479564+0000","last_became_peered":"2024-09-17T07:23:57.479564+0000","last_unstale":"2024-09-17T07:25:16.386810+0000","last_undegraded":"2024-09-17T07:25:16.386810+0000","last_fullsized":"2024-09-17T07:25:16.386810+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'2","last_scrub_stamp":"2024-09-17T07:25:08.624464+0000","last_deep_scrub":"268'2","last_deep_scrub_stamp":"2024-09-17T07:14:03.609131+0000","last_clean_scrub_stamp":"2024-09-17T07:25:08.624464+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:10.264762+0000","scrub_duration":0.0032006410000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"212'130","reported_seq":580,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-17T07:25:16.449605+0000","last_change":"2024-09-17T07:23:57.489780+0000","last_active":"2024-09-17T07:25:16.449605+0000","last_peered":"2024-09-17T07:25:16.449605+0000","last_clean":"2024-09-17T07:25:16.449605+0000","last_became_active":"2024-09-17T07:23:57.489445+0000","last_became_peered":"2024-09-17T07:23:57.489445+0000","last_unstale":"2024-09-17T07:25:16.449605+0000","last_undegraded":"2024-09-17T07:25:16.449605+0000","last_fullsized":"2024-09-17T07:25:16.449605+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":4,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:20:22.674350+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:14:57.052505+0000","last_clean_scrub_stamp":"2024-09-17T07:20:22.674350+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021141889999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":28,"num_write":17,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":161010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"439'55","reported_seq":715,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-09-17T07:25:15.782619+0000","last_change":"2024-09-17T07:24:28.997089+0000","last_active":"2024-09-17T07:25:15.782619+0000","last_peered":"2024-09-17T07:25:15.782619+0000","last_clean":"2024-09-17T07:25:15.782619+0000","last_became_active":"2024-09-17T07:21:15.352461+0000","last_became_peered":"2024-09-17T07:21:15.352461+0000","last_unstale":"2024-09-17T07:25:15.782619+0000","last_undegraded":"2024-09-17T07:25:15.782619+0000","last_fullsized":"2024-09-17T07:25:15.782619+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'55","last_scrub_stamp":"2024-09-17T07:24:28.996914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:24:28.996914+0000","objects_scrubbed":10,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:48.984638+0000","scrub_duration":0.0070140810000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":83,"num_read_kb":68,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":501,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-17T07:25:16.450927+0000","last_change":"2024-09-17T07:24:18.694832+0000","last_active":"2024-09-17T07:25:16.450927+0000","last_peered":"2024-09-17T07:25:16.450927+0000","last_clean":"2024-09-17T07:25:16.450927+0000","last_became_active":"2024-09-17T07:23:57.493640+0000","last_became_peered":"2024-09-17T07:23:57.493640+0000","last_unstale":"2024-09-17T07:25:16.450927+0000","last_undegraded":"2024-09-17T07:25:16.450927+0000","last_fullsized":"2024-09-17T07:25:16.450927+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:24:18.694743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:24:18.694743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:30.108218+0000","scrub_duration":0.001126408,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":551,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:19.743465+0000","last_change":"2024-09-17T07:24:57.661571+0000","last_active":"2024-09-17T07:25:19.743465+0000","last_peered":"2024-09-17T07:25:19.743465+0000","last_clean":"2024-09-17T07:25:19.743465+0000","last_became_active":"2024-09-17T07:21:15.347069+0000","last_became_peered":"2024-09-17T07:21:15.347069+0000","last_unstale":"2024-09-17T07:25:19.743465+0000","last_undegraded":"2024-09-17T07:25:19.743465+0000","last_fullsized":"2024-09-17T07:25:19.743465+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:24:57.661438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:24:57.661438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:13.489925+0000","scrub_duration":0.0017509139999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":450,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-09-17T07:25:15.890347+0000","last_change":"2024-09-17T07:25:09.565076+0000","last_active":"2024-09-17T07:25:15.890347+0000","last_peered":"2024-09-17T07:25:15.890347+0000","last_clean":"2024-09-17T07:25:15.890347+0000","last_became_active":"2024-09-17T07:25:09.564606+0000","last_became_peered":"2024-09-17T07:25:09.564606+0000","last_unstale":"2024-09-17T07:25:15.890347+0000","last_undegraded":"2024-09-17T07:25:15.890347+0000","last_fullsized":"2024-09-17T07:25:15.890347+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:20.067928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:13:53.406661+0000","last_clean_scrub_stamp":"2024-09-17T07:20:20.067928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0042634170000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.2","version":"212'130","reported_seq":600,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:20.087714+0000","last_change":"2024-09-17T07:22:16.360792+0000","last_active":"2024-09-17T07:25:20.087714+0000","last_peered":"2024-09-17T07:25:20.087714+0000","last_clean":"2024-09-17T07:25:20.087714+0000","last_became_active":"2024-09-17T07:22:16.360258+0000","last_became_peered":"2024-09-17T07:22:16.360258+0000","last_unstale":"2024-09-17T07:25:20.087714+0000","last_undegraded":"2024-09-17T07:25:20.087714+0000","last_fullsized":"2024-09-17T07:25:20.087714+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":4,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:19:19.991044+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:14:57.052505+0000","last_clean_scrub_stamp":"2024-09-17T07:19:19.991044+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021141889999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":28,"num_write":17,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":161010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"439'21","reported_seq":437,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-09-17T07:25:15.385013+0000","last_change":"2024-09-17T07:24:20.021085+0000","last_active":"2024-09-17T07:25:15.385013+0000","last_peered":"2024-09-17T07:25:15.385013+0000","last_clean":"2024-09-17T07:25:15.385013+0000","last_became_active":"2024-09-17T07:23:57.492747+0000","last_became_peered":"2024-09-17T07:23:57.492747+0000","last_unstale":"2024-09-17T07:25:15.385013+0000","last_undegraded":"2024-09-17T07:25:15.385013+0000","last_fullsized":"2024-09-17T07:25:15.385013+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'21","last_scrub_stamp":"2024-09-17T07:24:20.021003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:24:20.021003+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:22.745059+0000","scrub_duration":0.0022976009999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":30,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"76'5","reported_seq":1458,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-17T07:25:16.872816+0000","last_change":"2024-09-17T07:25:05.896344+0000","last_active":"2024-09-17T07:25:16.872816+0000","last_peered":"2024-09-17T07:25:16.872816+0000","last_clean":"2024-09-17T07:25:16.872816+0000","last_became_active":"2024-09-17T07:21:15.357178+0000","last_became_peered":"2024-09-17T07:21:15.357178+0000","last_unstale":"2024-09-17T07:25:16.872816+0000","last_undegraded":"2024-09-17T07:25:16.872816+0000","last_fullsized":"2024-09-17T07:25:16.872816+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'5","last_scrub_stamp":"2024-09-17T07:25:05.896257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:25:05.896257+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:15.630574+0000","scrub_duration":0.0018339840000000001,"objects_trimmed":0,"snaptrim_duration":0.00033739799999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":249,"num_read_kb":244,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-17T07:25:16.386994+0000","last_change":"2024-09-17T07:25:01.641881+0000","last_active":"2024-09-17T07:25:16.386994+0000","last_peered":"2024-09-17T07:25:16.386994+0000","last_clean":"2024-09-17T07:25:16.386994+0000","last_became_active":"2024-09-17T07:21:15.346287+0000","last_became_peered":"2024-09-17T07:21:15.346287+0000","last_unstale":"2024-09-17T07:25:16.386994+0000","last_undegraded":"2024-09-17T07:25:16.386994+0000","last_fullsized":"2024-09-17T07:25:16.386994+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:01.641756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:23.833170+0000","last_clean_scrub_stamp":"2024-09-17T07:25:01.641756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:18.315474+0000","scrub_duration":0.0031250689999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":615,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:19.742430+0000","last_change":"2024-09-17T07:24:18.345634+0000","last_active":"2024-09-17T07:25:19.742430+0000","last_peered":"2024-09-17T07:25:19.742430+0000","last_clean":"2024-09-17T07:25:19.742430+0000","last_became_active":"2024-09-17T07:23:57.481485+0000","last_became_peered":"2024-09-17T07:23:57.481485+0000","last_unstale":"2024-09-17T07:25:19.742430+0000","last_undegraded":"2024-09-17T07:25:19.742430+0000","last_fullsized":"2024-09-17T07:25:19.742430+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:24:18.345550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:18:42.051667+0000","last_clean_scrub_stamp":"2024-09-17T07:24:18.345550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:26.341677+0000","scrub_duration":0.0023155189999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":99,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:20.087929+0000","last_change":"2024-09-17T07:24:30.923133+0000","last_active":"2024-09-17T07:25:20.087929+0000","last_peered":"2024-09-17T07:25:20.087929+0000","last_clean":"2024-09-17T07:25:20.087929+0000","last_became_active":"2024-09-17T07:22:13.628359+0000","last_became_peered":"2024-09-17T07:22:13.628359+0000","last_unstale":"2024-09-17T07:25:20.087929+0000","last_undegraded":"2024-09-17T07:25:20.087929+0000","last_fullsized":"2024-09-17T07:25:20.087929+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:24:30.922689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:02.062618+0000","last_clean_scrub_stamp":"2024-09-17T07:24:30.922689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:36.609751+0000","scrub_duration":0.0066080289999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"212'130","reported_seq":622,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-09-17T07:25:15.383137+0000","last_change":"2024-09-17T07:21:18.358222+0000","last_active":"2024-09-17T07:25:15.383137+0000","last_peered":"2024-09-17T07:25:15.383137+0000","last_clean":"2024-09-17T07:25:15.383137+0000","last_became_active":"2024-09-17T07:21:18.357874+0000","last_became_peered":"2024-09-17T07:21:18.357874+0000","last_unstale":"2024-09-17T07:25:15.383137+0000","last_undegraded":"2024-09-17T07:25:15.383137+0000","last_fullsized":"2024-09-17T07:25:15.383137+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":3,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:20:34.113602+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:20:34.113602+0000","last_clean_scrub_stamp":"2024-09-17T07:20:34.113602+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021141889999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":28,"num_write":17,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":161010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"439'42","reported_seq":645,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-17T07:25:16.387764+0000","last_change":"2024-09-17T07:25:05.607011+0000","last_active":"2024-09-17T07:25:16.387764+0000","last_peered":"2024-09-17T07:25:16.387764+0000","last_clean":"2024-09-17T07:25:16.387764+0000","last_became_active":"2024-09-17T07:23:57.480190+0000","last_became_peered":"2024-09-17T07:23:57.480190+0000","last_unstale":"2024-09-17T07:25:16.387764+0000","last_undegraded":"2024-09-17T07:25:16.387764+0000","last_fullsized":"2024-09-17T07:25:16.387764+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'42","last_scrub_stamp":"2024-09-17T07:25:05.606929+0000","last_deep_scrub":"268'30","last_deep_scrub_stamp":"2024-09-17T07:14:00.599656+0000","last_clean_scrub_stamp":"2024-09-17T07:25:05.606929+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:28.810784+0000","scrub_duration":0.0040141389999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"70'1","reported_seq":584,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-17T07:25:16.386905+0000","last_change":"2024-09-17T07:25:14.651563+0000","last_active":"2024-09-17T07:25:16.386905+0000","last_peered":"2024-09-17T07:25:16.386905+0000","last_clean":"2024-09-17T07:25:16.386905+0000","last_became_active":"2024-09-17T07:23:57.480048+0000","last_became_peered":"2024-09-17T07:23:57.480048+0000","last_unstale":"2024-09-17T07:25:16.386905+0000","last_undegraded":"2024-09-17T07:25:16.386905+0000","last_fullsized":"2024-09-17T07:25:16.386905+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-17T07:25:14.651458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:25:14.651458+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:14.704935+0000","scrub_duration":0.0020687610000000001,"objects_trimmed":0,"snaptrim_duration":0.0032298629999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":88,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:20.087282+0000","last_change":"2024-09-17T07:22:13.736269+0000","last_active":"2024-09-17T07:25:20.087282+0000","last_peered":"2024-09-17T07:25:20.087282+0000","last_clean":"2024-09-17T07:25:20.087282+0000","last_became_active":"2024-09-17T07:22:13.710211+0000","last_became_peered":"2024-09-17T07:22:13.710211+0000","last_unstale":"2024-09-17T07:25:20.087282+0000","last_undegraded":"2024-09-17T07:25:20.087282+0000","last_fullsized":"2024-09-17T07:25:20.087282+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:34.449627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:55.321202+0000","last_clean_scrub_stamp":"2024-09-17T07:20:34.449627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0044019280000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":496,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-17T07:25:16.451520+0000","last_change":"2024-09-17T07:24:57.916305+0000","last_active":"2024-09-17T07:25:16.451520+0000","last_peered":"2024-09-17T07:25:16.451520+0000","last_clean":"2024-09-17T07:25:16.451520+0000","last_became_active":"2024-09-17T07:23:57.480455+0000","last_became_peered":"2024-09-17T07:23:57.480455+0000","last_unstale":"2024-09-17T07:25:16.451520+0000","last_undegraded":"2024-09-17T07:25:16.451520+0000","last_fullsized":"2024-09-17T07:25:16.451520+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:24:57.916204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:24:57.916204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:22.807948+0000","scrub_duration":0.001685541,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":421,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-09-17T07:25:15.384764+0000","last_change":"2024-09-17T07:23:57.473704+0000","last_active":"2024-09-17T07:25:15.384764+0000","last_peered":"2024-09-17T07:25:15.384764+0000","last_clean":"2024-09-17T07:25:15.384764+0000","last_became_active":"2024-09-17T07:23:57.473313+0000","last_became_peered":"2024-09-17T07:23:57.473313+0000","last_unstale":"2024-09-17T07:25:15.384764+0000","last_undegraded":"2024-09-17T07:25:15.384764+0000","last_fullsized":"2024-09-17T07:25:15.384764+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:51.877367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:15:51.877367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.058290885000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"439'38","reported_seq":667,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-09-17T07:25:15.609726+0000","last_change":"2024-09-17T07:25:11.645824+0000","last_active":"2024-09-17T07:25:15.609726+0000","last_peered":"2024-09-17T07:25:15.609726+0000","last_clean":"2024-09-17T07:25:15.609726+0000","last_became_active":"2024-09-17T07:22:13.583204+0000","last_became_peered":"2024-09-17T07:22:13.583204+0000","last_unstale":"2024-09-17T07:25:15.609726+0000","last_undegraded":"2024-09-17T07:25:15.609726+0000","last_fullsized":"2024-09-17T07:25:15.609726+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'38","last_scrub_stamp":"2024-09-17T07:25:11.645722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:25:11.645722+0000","objects_scrubbed":7,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:36.237351+0000","scrub_duration":0.0024003060000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"268'3","reported_seq":368,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-17T07:25:16.450885+0000","last_change":"2024-09-17T07:25:15.066327+0000","last_active":"2024-09-17T07:25:16.450885+0000","last_peered":"2024-09-17T07:25:16.450885+0000","last_clean":"2024-09-17T07:25:16.450885+0000","last_became_active":"2024-09-17T07:23:57.485430+0000","last_became_peered":"2024-09-17T07:23:57.485430+0000","last_unstale":"2024-09-17T07:25:16.450885+0000","last_undegraded":"2024-09-17T07:25:16.450885+0000","last_fullsized":"2024-09-17T07:25:16.450885+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'3","last_scrub_stamp":"2024-09-17T07:25:15.066229+0000","last_deep_scrub":"268'3","last_deep_scrub_stamp":"2024-09-17T07:13:54.949345+0000","last_clean_scrub_stamp":"2024-09-17T07:25:15.066229+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:22.714073+0000","scrub_duration":0.0017457670000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":546,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-17T07:25:16.450298+0000","last_change":"2024-09-17T07:23:57.480855+0000","last_active":"2024-09-17T07:25:16.450298+0000","last_peered":"2024-09-17T07:25:16.450298+0000","last_clean":"2024-09-17T07:25:16.450298+0000","last_became_active":"2024-09-17T07:23:57.477410+0000","last_became_peered":"2024-09-17T07:23:57.477410+0000","last_unstale":"2024-09-17T07:25:16.450298+0000","last_undegraded":"2024-09-17T07:25:16.450298+0000","last_fullsized":"2024-09-17T07:25:16.450298+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:17.650824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:20:17.650824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019547470000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"212'130","reported_seq":610,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:20.087546+0000","last_change":"2024-09-17T07:22:16.366636+0000","last_active":"2024-09-17T07:25:20.087546+0000","last_peered":"2024-09-17T07:25:20.087546+0000","last_clean":"2024-09-17T07:25:20.087546+0000","last_became_active":"2024-09-17T07:22:16.366295+0000","last_became_peered":"2024-09-17T07:22:16.366295+0000","last_unstale":"2024-09-17T07:25:20.087546+0000","last_undegraded":"2024-09-17T07:25:20.087546+0000","last_fullsized":"2024-09-17T07:25:20.087546+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:19:19.991044+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:14:57.052505+0000","last_clean_scrub_stamp":"2024-09-17T07:19:19.991044+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021141889999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":27,"num_write":16,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":161009,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":114,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:20.088503+0000","last_change":"2024-09-17T07:24:19.856080+0000","last_active":"2024-09-17T07:25:20.088503+0000","last_peered":"2024-09-17T07:25:20.088503+0000","last_clean":"2024-09-17T07:25:20.088503+0000","last_became_active":"2024-09-17T07:23:57.476027+0000","last_became_peered":"2024-09-17T07:23:57.476027+0000","last_unstale":"2024-09-17T07:25:20.088503+0000","last_undegraded":"2024-09-17T07:25:20.088503+0000","last_fullsized":"2024-09-17T07:25:20.088503+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:24:19.855925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:18:48.891136+0000","last_clean_scrub_stamp":"2024-09-17T07:24:19.855925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:47.079242+0000","scrub_duration":0.0055362720000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":492,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-17T07:25:16.451271+0000","last_change":"2024-09-17T07:24:21.789270+0000","last_active":"2024-09-17T07:25:16.451271+0000","last_peered":"2024-09-17T07:25:16.451271+0000","last_clean":"2024-09-17T07:25:16.451271+0000","last_became_active":"2024-09-17T07:23:57.478248+0000","last_became_peered":"2024-09-17T07:23:57.478248+0000","last_unstale":"2024-09-17T07:25:16.451271+0000","last_undegraded":"2024-09-17T07:25:16.451271+0000","last_fullsized":"2024-09-17T07:25:16.451271+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:24:21.789110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:58.670571+0000","last_clean_scrub_stamp":"2024-09-17T07:24:21.789110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:30.597631+0000","scrub_duration":0.0090529149999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"439'51","reported_seq":711,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-09-17T07:25:15.611131+0000","last_change":"2024-09-17T07:25:15.611131+0000","last_active":"2024-09-17T07:25:15.611131+0000","last_peered":"2024-09-17T07:25:15.611131+0000","last_clean":"2024-09-17T07:25:15.611131+0000","last_became_active":"2024-09-17T07:22:13.588260+0000","last_became_peered":"2024-09-17T07:22:13.588260+0000","last_unstale":"2024-09-17T07:25:15.611131+0000","last_undegraded":"2024-09-17T07:25:15.611131+0000","last_fullsized":"2024-09-17T07:25:15.611131+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'51","last_scrub_stamp":"2024-09-17T07:25:15.611017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:25:15.611017+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:28.411603+0000","scrub_duration":0.0030635860000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":626,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:19.742652+0000","last_change":"2024-09-17T07:24:23.255568+0000","last_active":"2024-09-17T07:25:19.742652+0000","last_peered":"2024-09-17T07:25:19.742652+0000","last_clean":"2024-09-17T07:25:19.742652+0000","last_became_active":"2024-09-17T07:23:57.482084+0000","last_became_peered":"2024-09-17T07:23:57.482084+0000","last_unstale":"2024-09-17T07:25:19.742652+0000","last_undegraded":"2024-09-17T07:25:19.742652+0000","last_fullsized":"2024-09-17T07:25:19.742652+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:24:23.255270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:24:23.255270+0000","last_clean_scrub_stamp":"2024-09-17T07:24:23.255270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:30.797016+0000","scrub_duration":0.0036829879999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":609,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:19.742132+0000","last_change":"2024-09-17T07:24:20.283146+0000","last_active":"2024-09-17T07:25:19.742132+0000","last_peered":"2024-09-17T07:25:19.742132+0000","last_clean":"2024-09-17T07:25:19.742132+0000","last_became_active":"2024-09-17T07:23:57.481504+0000","last_became_peered":"2024-09-17T07:23:57.481504+0000","last_unstale":"2024-09-17T07:25:19.742132+0000","last_undegraded":"2024-09-17T07:25:19.742132+0000","last_fullsized":"2024-09-17T07:25:19.742132+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:24:20.282973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:24:20.282973+0000","last_clean_scrub_stamp":"2024-09-17T07:24:20.282973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:26.632960+0000","scrub_duration":0.024343656000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":99,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:20.087648+0000","last_change":"2024-09-17T07:24:23.847700+0000","last_active":"2024-09-17T07:25:20.087648+0000","last_peered":"2024-09-17T07:25:20.087648+0000","last_clean":"2024-09-17T07:25:20.087648+0000","last_became_active":"2024-09-17T07:22:13.573161+0000","last_became_peered":"2024-09-17T07:22:13.573161+0000","last_unstale":"2024-09-17T07:25:20.087648+0000","last_undegraded":"2024-09-17T07:25:20.087648+0000","last_fullsized":"2024-09-17T07:25:20.087648+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:24:23.847363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:45.275473+0000","last_clean_scrub_stamp":"2024-09-17T07:24:23.847363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:31.038832+0000","scrub_duration":0.0047891890000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"439'25","reported_seq":580,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-17T07:25:16.387183+0000","last_change":"2024-09-17T07:24:56.652772+0000","last_active":"2024-09-17T07:25:16.387183+0000","last_peered":"2024-09-17T07:25:16.387183+0000","last_clean":"2024-09-17T07:25:16.387183+0000","last_became_active":"2024-09-17T07:21:15.349361+0000","last_became_peered":"2024-09-17T07:21:15.349361+0000","last_unstale":"2024-09-17T07:25:16.387183+0000","last_undegraded":"2024-09-17T07:25:16.387183+0000","last_fullsized":"2024-09-17T07:25:16.387183+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'25","last_scrub_stamp":"2024-09-17T07:24:56.652673+0000","last_deep_scrub":"268'19","last_deep_scrub_stamp":"2024-09-17T07:14:06.634334+0000","last_clean_scrub_stamp":"2024-09-17T07:24:56.652673+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:21.478181+0000","scrub_duration":0.012732797000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":434,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-09-17T07:25:15.384147+0000","last_change":"2024-09-17T07:25:14.981641+0000","last_active":"2024-09-17T07:25:15.384147+0000","last_peered":"2024-09-17T07:25:15.384147+0000","last_clean":"2024-09-17T07:25:15.384147+0000","last_became_active":"2024-09-17T07:21:15.350786+0000","last_became_peered":"2024-09-17T07:21:15.350786+0000","last_unstale":"2024-09-17T07:25:15.384147+0000","last_undegraded":"2024-09-17T07:25:15.384147+0000","last_fullsized":"2024-09-17T07:25:15.384147+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:14.981560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:13:53.593078+0000","last_clean_scrub_stamp":"2024-09-17T07:25:14.981560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:30.222480+0000","scrub_duration":0.0016595399999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":519,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-09-17T07:25:15.889965+0000","last_change":"2024-09-17T07:25:13.923025+0000","last_active":"2024-09-17T07:25:15.889965+0000","last_peered":"2024-09-17T07:25:15.889965+0000","last_clean":"2024-09-17T07:25:15.889965+0000","last_became_active":"2024-09-17T07:09:24.396642+0000","last_became_peered":"2024-09-17T07:09:24.396642+0000","last_unstale":"2024-09-17T07:25:15.889965+0000","last_undegraded":"2024-09-17T07:25:15.889965+0000","last_fullsized":"2024-09-17T07:25:15.889965+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:13.922902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:25:13.922902+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:19.966114+0000","scrub_duration":0.0019707629999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":561,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-17T07:25:16.639982+0000","last_change":"2024-09-17T07:25:16.639982+0000","last_active":"2024-09-17T07:25:16.639982+0000","last_peered":"2024-09-17T07:25:16.639982+0000","last_clean":"2024-09-17T07:25:16.639982+0000","last_became_active":"2024-09-17T07:22:13.589460+0000","last_became_peered":"2024-09-17T07:22:13.589460+0000","last_unstale":"2024-09-17T07:25:16.639982+0000","last_undegraded":"2024-09-17T07:25:16.639982+0000","last_fullsized":"2024-09-17T07:25:16.639982+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:16.639899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:58.099849+0000","last_clean_scrub_stamp":"2024-09-17T07:25:16.639899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:41.555366+0000","scrub_duration":0.001931898,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"439'46","reported_seq":649,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-17T07:25:16.448905+0000","last_change":"2024-09-17T07:25:00.980570+0000","last_active":"2024-09-17T07:25:16.448905+0000","last_peered":"2024-09-17T07:25:16.448905+0000","last_clean":"2024-09-17T07:25:16.448905+0000","last_became_active":"2024-09-17T07:23:57.482858+0000","last_became_peered":"2024-09-17T07:23:57.482858+0000","last_unstale":"2024-09-17T07:25:16.448905+0000","last_undegraded":"2024-09-17T07:25:16.448905+0000","last_fullsized":"2024-09-17T07:25:16.448905+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'46","last_scrub_stamp":"2024-09-17T07:25:00.980474+0000","last_deep_scrub":"242'34","last_deep_scrub_stamp":"2024-09-17T07:14:50.103539+0000","last_clean_scrub_stamp":"2024-09-17T07:25:00.980474+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:06.860259+0000","scrub_duration":0.0021656290000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'2","reported_seq":94,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:20.088317+0000","last_change":"2024-09-17T07:25:05.848575+0000","last_active":"2024-09-17T07:25:20.088317+0000","last_peered":"2024-09-17T07:25:20.088317+0000","last_clean":"2024-09-17T07:25:20.088317+0000","last_became_active":"2024-09-17T07:22:13.575469+0000","last_became_peered":"2024-09-17T07:22:13.575469+0000","last_unstale":"2024-09-17T07:25:20.088317+0000","last_undegraded":"2024-09-17T07:25:20.088317+0000","last_fullsized":"2024-09-17T07:25:20.088317+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-09-17T07:18:43.861842+0000","last_deep_scrub":"89'2","last_deep_scrub_stamp":"2024-09-17T07:12:20.301263+0000","last_clean_scrub_stamp":"2024-09-17T07:18:43.861842+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":420,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":420,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"436'14","reported_seq":142,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:20.783955+0000","last_change":"2024-09-17T07:24:59.877855+0000","last_active":"2024-09-17T07:25:20.783955+0000","last_peered":"2024-09-17T07:25:20.783955+0000","last_clean":"2024-09-17T07:25:20.783955+0000","last_became_active":"2024-09-17T07:22:13.624050+0000","last_became_peered":"2024-09-17T07:22:13.624050+0000","last_unstale":"2024-09-17T07:25:20.783955+0000","last_undegraded":"2024-09-17T07:25:20.783955+0000","last_fullsized":"2024-09-17T07:25:20.783955+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"436'14","last_scrub_stamp":"2024-09-17T07:24:59.877774+0000","last_deep_scrub":"225'9","last_deep_scrub_stamp":"2024-09-17T07:11:25.292565+0000","last_clean_scrub_stamp":"2024-09-17T07:24:59.877774+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:07.407178+0000","scrub_duration":0.0015096720000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":524,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-17T07:25:16.450805+0000","last_change":"2024-09-17T07:24:58.961010+0000","last_active":"2024-09-17T07:25:16.450805+0000","last_peered":"2024-09-17T07:25:16.450805+0000","last_clean":"2024-09-17T07:25:16.450805+0000","last_became_active":"2024-09-17T07:23:57.493273+0000","last_became_peered":"2024-09-17T07:23:57.493273+0000","last_unstale":"2024-09-17T07:25:16.450805+0000","last_undegraded":"2024-09-17T07:25:16.450805+0000","last_fullsized":"2024-09-17T07:25:16.450805+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:24:58.960933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:18:41.385928+0000","last_clean_scrub_stamp":"2024-09-17T07:24:58.960933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:03.753329+0000","scrub_duration":0.0012208259999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"439'29","reported_seq":635,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-09-17T07:25:15.889713+0000","last_change":"2024-09-17T07:20:33.108656+0000","last_active":"2024-09-17T07:25:15.889713+0000","last_peered":"2024-09-17T07:25:15.889713+0000","last_clean":"2024-09-17T07:25:15.889713+0000","last_became_active":"2024-09-17T07:10:49.723292+0000","last_became_peered":"2024-09-17T07:10:49.723292+0000","last_unstale":"2024-09-17T07:25:15.889713+0000","last_undegraded":"2024-09-17T07:25:15.889713+0000","last_fullsized":"2024-09-17T07:25:15.889713+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'23","last_scrub_stamp":"2024-09-17T07:20:33.108490+0000","last_deep_scrub":"150'17","last_deep_scrub_stamp":"2024-09-17T07:06:38.775788+0000","last_clean_scrub_stamp":"2024-09-17T07:20:33.108490+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0087601510000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"68'2","reported_seq":586,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-09-17T07:25:15.890660+0000","last_change":"2024-09-17T07:22:13.766671+0000","last_active":"2024-09-17T07:25:15.890660+0000","last_peered":"2024-09-17T07:25:15.890660+0000","last_clean":"2024-09-17T07:25:15.890660+0000","last_became_active":"2024-09-17T07:22:13.590172+0000","last_became_peered":"2024-09-17T07:22:13.590172+0000","last_unstale":"2024-09-17T07:25:15.890660+0000","last_undegraded":"2024-09-17T07:25:15.890660+0000","last_fullsized":"2024-09-17T07:25:15.890660+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-09-17T07:20:11.093491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:20:11.093491+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023198989999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":514,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-17T07:25:16.386902+0000","last_change":"2024-09-17T07:24:26.618642+0000","last_active":"2024-09-17T07:25:16.386902+0000","last_peered":"2024-09-17T07:25:16.386902+0000","last_clean":"2024-09-17T07:25:16.386902+0000","last_became_active":"2024-09-17T07:21:15.355760+0000","last_became_peered":"2024-09-17T07:21:15.355760+0000","last_unstale":"2024-09-17T07:25:16.386902+0000","last_undegraded":"2024-09-17T07:25:16.386902+0000","last_fullsized":"2024-09-17T07:25:16.386902+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:24:26.618433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:11:17.479036+0000","last_clean_scrub_stamp":"2024-09-17T07:24:26.618433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:30.630753+0000","scrub_duration":0.0026272880000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":88,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:20.088127+0000","last_change":"2024-09-17T07:22:13.585879+0000","last_active":"2024-09-17T07:25:20.088127+0000","last_peered":"2024-09-17T07:25:20.088127+0000","last_clean":"2024-09-17T07:25:20.088127+0000","last_became_active":"2024-09-17T07:22:13.585580+0000","last_became_peered":"2024-09-17T07:22:13.585580+0000","last_unstale":"2024-09-17T07:25:20.088127+0000","last_undegraded":"2024-09-17T07:25:20.088127+0000","last_fullsized":"2024-09-17T07:25:20.088127+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:07.959158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:20:07.959158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.018487173999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"439'21","reported_seq":628,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-17T07:25:16.387641+0000","last_change":"2024-09-17T07:25:05.507259+0000","last_active":"2024-09-17T07:25:16.387641+0000","last_peered":"2024-09-17T07:25:16.387641+0000","last_clean":"2024-09-17T07:25:16.387641+0000","last_became_active":"2024-09-17T07:22:13.592001+0000","last_became_peered":"2024-09-17T07:22:13.592001+0000","last_unstale":"2024-09-17T07:25:16.387641+0000","last_undegraded":"2024-09-17T07:25:16.387641+0000","last_fullsized":"2024-09-17T07:25:16.387641+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'15","last_scrub_stamp":"2024-09-17T07:18:24.505002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:18:24.505002+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002195809,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":596,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:19.741504+0000","last_change":"2024-09-17T07:23:57.042832+0000","last_active":"2024-09-17T07:25:19.741504+0000","last_peered":"2024-09-17T07:25:19.741504+0000","last_clean":"2024-09-17T07:25:19.741504+0000","last_became_active":"2024-09-17T07:21:15.355464+0000","last_became_peered":"2024-09-17T07:21:15.355464+0000","last_unstale":"2024-09-17T07:25:19.741504+0000","last_undegraded":"2024-09-17T07:25:19.741504+0000","last_fullsized":"2024-09-17T07:25:19.741504+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:23:57.042751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:17.976267+0000","last_clean_scrub_stamp":"2024-09-17T07:23:57.042751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:22.442338+0000","scrub_duration":0.001374662,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":245,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-09-17T07:25:15.384866+0000","last_change":"2024-09-17T07:22:13.593743+0000","last_active":"2024-09-17T07:25:15.384866+0000","last_peered":"2024-09-17T07:25:15.384866+0000","last_clean":"2024-09-17T07:25:15.384866+0000","last_became_active":"2024-09-17T07:22:13.593446+0000","last_became_peered":"2024-09-17T07:22:13.593446+0000","last_unstale":"2024-09-17T07:25:15.384866+0000","last_undegraded":"2024-09-17T07:25:15.384866+0000","last_fullsized":"2024-09-17T07:25:15.384866+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:29.492000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:18.367486+0000","last_clean_scrub_stamp":"2024-09-17T07:20:29.492000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0077309099999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":649,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:19.742920+0000","last_change":"2024-09-17T07:24:24.282738+0000","last_active":"2024-09-17T07:25:19.742920+0000","last_peered":"2024-09-17T07:25:19.742920+0000","last_clean":"2024-09-17T07:25:19.742920+0000","last_became_active":"2024-09-17T07:22:13.590599+0000","last_became_peered":"2024-09-17T07:22:13.590599+0000","last_unstale":"2024-09-17T07:25:19.742920+0000","last_undegraded":"2024-09-17T07:25:19.742920+0000","last_fullsized":"2024-09-17T07:25:19.742920+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:24:24.282487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:24:24.282487+0000","last_clean_scrub_stamp":"2024-09-17T07:24:24.282487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:43.887065+0000","scrub_duration":0.0043307379999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":563,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-09-17T07:25:15.609802+0000","last_change":"2024-09-17T07:24:20.528931+0000","last_active":"2024-09-17T07:25:15.609802+0000","last_peered":"2024-09-17T07:25:15.609802+0000","last_clean":"2024-09-17T07:25:15.609802+0000","last_became_active":"2024-09-17T07:21:15.348758+0000","last_became_peered":"2024-09-17T07:21:15.348758+0000","last_unstale":"2024-09-17T07:25:15.609802+0000","last_undegraded":"2024-09-17T07:25:15.609802+0000","last_fullsized":"2024-09-17T07:25:15.609802+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:24:20.528714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:46.645080+0000","last_clean_scrub_stamp":"2024-09-17T07:24:20.528714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:34.396548+0000","scrub_duration":0.0080332240000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"439'18","reported_seq":105,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:20.088355+0000","last_change":"2024-09-17T07:22:13.761675+0000","last_active":"2024-09-17T07:25:20.088355+0000","last_peered":"2024-09-17T07:25:20.088355+0000","last_clean":"2024-09-17T07:25:20.088355+0000","last_became_active":"2024-09-17T07:22:13.586662+0000","last_became_peered":"2024-09-17T07:22:13.586662+0000","last_unstale":"2024-09-17T07:25:20.088355+0000","last_undegraded":"2024-09-17T07:25:20.088355+0000","last_fullsized":"2024-09-17T07:25:20.088355+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'12","last_scrub_stamp":"2024-09-17T07:20:17.090146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:20:17.090146+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019381719999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":25,"num_read_kb":22,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":504,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-17T07:25:16.450374+0000","last_change":"2024-09-17T07:24:23.835840+0000","last_active":"2024-09-17T07:25:16.450374+0000","last_peered":"2024-09-17T07:25:16.450374+0000","last_clean":"2024-09-17T07:25:16.450374+0000","last_became_active":"2024-09-17T07:23:57.478324+0000","last_became_peered":"2024-09-17T07:23:57.478324+0000","last_unstale":"2024-09-17T07:25:16.450374+0000","last_undegraded":"2024-09-17T07:25:16.450374+0000","last_fullsized":"2024-09-17T07:25:16.450374+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:24:23.835718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:24:23.835718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:47.145573+0000","scrub_duration":0.0037662490000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":99,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:20.088056+0000","last_change":"2024-09-17T07:25:06.877892+0000","last_active":"2024-09-17T07:25:20.088056+0000","last_peered":"2024-09-17T07:25:20.088056+0000","last_clean":"2024-09-17T07:25:20.088056+0000","last_became_active":"2024-09-17T07:22:13.623754+0000","last_became_peered":"2024-09-17T07:22:13.623754+0000","last_unstale":"2024-09-17T07:25:20.088056+0000","last_undegraded":"2024-09-17T07:25:20.088056+0000","last_fullsized":"2024-09-17T07:25:20.088056+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:06.877789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:25:06.877789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:29.306772+0000","scrub_duration":0.001331243,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"439'30","reported_seq":304,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-09-17T07:25:15.384642+0000","last_change":"2024-09-17T07:22:13.768713+0000","last_active":"2024-09-17T07:25:15.384642+0000","last_peered":"2024-09-17T07:25:15.384642+0000","last_clean":"2024-09-17T07:25:15.384642+0000","last_became_active":"2024-09-17T07:22:13.591094+0000","last_became_peered":"2024-09-17T07:22:13.591094+0000","last_unstale":"2024-09-17T07:25:15.384642+0000","last_undegraded":"2024-09-17T07:25:15.384642+0000","last_fullsized":"2024-09-17T07:25:15.384642+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'20","last_scrub_stamp":"2024-09-17T07:20:10.554324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:20:10.554324+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0020069329999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":565,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:20.089172+0000","last_change":"2024-09-17T07:25:20.089270+0000","last_active":"2024-09-17T07:25:20.089172+0000","last_peered":"2024-09-17T07:25:20.089172+0000","last_clean":"2024-09-17T07:25:20.089172+0000","last_became_active":"2024-09-17T07:23:57.491291+0000","last_became_peered":"2024-09-17T07:23:57.491291+0000","last_unstale":"2024-09-17T07:25:20.089172+0000","last_undegraded":"2024-09-17T07:25:20.089172+0000","last_fullsized":"2024-09-17T07:25:20.089172+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:20.089157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:19:40.443474+0000","last_clean_scrub_stamp":"2024-09-17T07:25:20.089157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:36.357354+0000","scrub_duration":0.0033687460000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"478'13","reported_seq":751,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-17T07:25:16.451078+0000","last_change":"2024-09-17T07:24:31.008439+0000","last_active":"2024-09-17T07:25:16.451078+0000","last_peered":"2024-09-17T07:25:16.451078+0000","last_clean":"2024-09-17T07:25:16.451078+0000","last_became_active":"2024-09-17T07:23:57.522386+0000","last_became_peered":"2024-09-17T07:23:57.522386+0000","last_unstale":"2024-09-17T07:25:16.451078+0000","last_undegraded":"2024-09-17T07:25:16.451078+0000","last_fullsized":"2024-09-17T07:25:16.451078+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"478'13","last_scrub_stamp":"2024-09-17T07:24:31.008289+0000","last_deep_scrub":"163'9","last_deep_scrub_stamp":"2024-09-17T07:07:34.933808+0000","last_clean_scrub_stamp":"2024-09-17T07:24:31.008289+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:47.002555+0000","scrub_duration":0.0084935229999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":570,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-17T07:25:16.387431+0000","last_change":"2024-09-17T07:25:06.624551+0000","last_active":"2024-09-17T07:25:16.387431+0000","last_peered":"2024-09-17T07:25:16.387431+0000","last_clean":"2024-09-17T07:25:16.387431+0000","last_became_active":"2024-09-17T07:22:13.591307+0000","last_became_peered":"2024-09-17T07:22:13.591307+0000","last_unstale":"2024-09-17T07:25:16.387431+0000","last_undegraded":"2024-09-17T07:25:16.387431+0000","last_fullsized":"2024-09-17T07:25:16.387431+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:06.624450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:26.619893+0000","last_clean_scrub_stamp":"2024-09-17T07:25:06.624450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:13.532043+0000","scrub_duration":0.001368408,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"439'21","reported_seq":621,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-17T07:25:16.450478+0000","last_change":"2024-09-17T07:25:08.008455+0000","last_active":"2024-09-17T07:25:16.450478+0000","last_peered":"2024-09-17T07:25:16.450478+0000","last_clean":"2024-09-17T07:25:16.450478+0000","last_became_active":"2024-09-17T07:23:57.485557+0000","last_became_peered":"2024-09-17T07:23:57.485557+0000","last_unstale":"2024-09-17T07:25:16.450478+0000","last_undegraded":"2024-09-17T07:25:16.450478+0000","last_fullsized":"2024-09-17T07:25:16.450478+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'21","last_scrub_stamp":"2024-09-17T07:25:08.008359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:25:08.008359+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:28.888348+0000","scrub_duration":0.0025217759999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":579,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-17T07:25:18.620750+0000","last_change":"2024-09-17T07:25:18.620750+0000","last_active":"2024-09-17T07:25:18.620750+0000","last_peered":"2024-09-17T07:25:18.620750+0000","last_clean":"2024-09-17T07:25:18.620750+0000","last_became_active":"2024-09-17T07:22:13.589964+0000","last_became_peered":"2024-09-17T07:22:13.589964+0000","last_unstale":"2024-09-17T07:25:18.620750+0000","last_undegraded":"2024-09-17T07:25:18.620750+0000","last_fullsized":"2024-09-17T07:25:18.620750+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:18.620666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:25:18.620666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:37.392688+0000","scrub_duration":0.001507972,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"436'15","reported_seq":141,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:20.784013+0000","last_change":"2024-09-17T07:25:18.993295+0000","last_active":"2024-09-17T07:25:20.784013+0000","last_peered":"2024-09-17T07:25:20.784013+0000","last_clean":"2024-09-17T07:25:20.784013+0000","last_became_active":"2024-09-17T07:22:13.586620+0000","last_became_peered":"2024-09-17T07:22:13.586620+0000","last_unstale":"2024-09-17T07:25:20.784013+0000","last_undegraded":"2024-09-17T07:25:20.784013+0000","last_fullsized":"2024-09-17T07:25:20.784013+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"436'15","last_scrub_stamp":"2024-09-17T07:25:18.993184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:25:18.993184+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:42.271745+0000","scrub_duration":0.0021849909999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":557,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-09-17T07:25:15.890502+0000","last_change":"2024-09-17T07:23:57.473144+0000","last_active":"2024-09-17T07:25:15.890502+0000","last_peered":"2024-09-17T07:25:15.890502+0000","last_clean":"2024-09-17T07:25:15.890502+0000","last_became_active":"2024-09-17T07:23:57.472854+0000","last_became_peered":"2024-09-17T07:23:57.472854+0000","last_unstale":"2024-09-17T07:25:15.890502+0000","last_undegraded":"2024-09-17T07:25:15.890502+0000","last_fullsized":"2024-09-17T07:25:15.890502+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:37.110458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:20:37.110458+0000","last_clean_scrub_stamp":"2024-09-17T07:20:37.110458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.017901013,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"439'21","reported_seq":416,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-09-17T07:25:15.382975+0000","last_change":"2024-09-17T07:24:18.055969+0000","last_active":"2024-09-17T07:25:15.382975+0000","last_peered":"2024-09-17T07:25:15.382975+0000","last_clean":"2024-09-17T07:25:15.382975+0000","last_became_active":"2024-09-17T07:21:15.348060+0000","last_became_peered":"2024-09-17T07:21:15.348060+0000","last_unstale":"2024-09-17T07:25:15.382975+0000","last_undegraded":"2024-09-17T07:25:15.382975+0000","last_fullsized":"2024-09-17T07:25:15.382975+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'21","last_scrub_stamp":"2024-09-17T07:24:18.055874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:24:18.055874+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:26.355516+0000","scrub_duration":0.0022038589999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":610,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-09-17T07:25:15.609944+0000","last_change":"2024-09-17T07:24:21.515088+0000","last_active":"2024-09-17T07:25:15.609944+0000","last_peered":"2024-09-17T07:25:15.609944+0000","last_clean":"2024-09-17T07:25:15.609944+0000","last_became_active":"2024-09-17T07:23:57.493401+0000","last_became_peered":"2024-09-17T07:23:57.493401+0000","last_unstale":"2024-09-17T07:25:15.609944+0000","last_undegraded":"2024-09-17T07:25:15.609944+0000","last_fullsized":"2024-09-17T07:25:15.609944+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:24:21.514803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:10:36.334827+0000","last_clean_scrub_stamp":"2024-09-17T07:24:21.514803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:25.466862+0000","scrub_duration":0.0094769510000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":407,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-17T07:25:16.386958+0000","last_change":"2024-09-17T07:25:00.641893+0000","last_active":"2024-09-17T07:25:16.386958+0000","last_peered":"2024-09-17T07:25:16.386958+0000","last_clean":"2024-09-17T07:25:16.386958+0000","last_became_active":"2024-09-17T07:23:57.479281+0000","last_became_peered":"2024-09-17T07:23:57.479281+0000","last_unstale":"2024-09-17T07:25:16.386958+0000","last_undegraded":"2024-09-17T07:25:16.386958+0000","last_fullsized":"2024-09-17T07:25:16.386958+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:00.641817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:25:00.641817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:22.472510+0000","scrub_duration":0.031817810000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":561,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-17T07:25:16.386807+0000","last_change":"2024-09-17T07:24:21.576079+0000","last_active":"2024-09-17T07:25:16.386807+0000","last_peered":"2024-09-17T07:25:16.386807+0000","last_clean":"2024-09-17T07:25:16.386807+0000","last_became_active":"2024-09-17T07:23:57.478818+0000","last_became_peered":"2024-09-17T07:23:57.478818+0000","last_unstale":"2024-09-17T07:25:16.386807+0000","last_undegraded":"2024-09-17T07:25:16.386807+0000","last_fullsized":"2024-09-17T07:25:16.386807+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:24:21.575574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:24:21.575574+0000","last_clean_scrub_stamp":"2024-09-17T07:24:21.575574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:25.442934+0000","scrub_duration":0.006786233,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"439'27","reported_seq":582,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-09-17T07:25:15.889193+0000","last_change":"2024-09-17T07:21:16.070482+0000","last_active":"2024-09-17T07:25:15.889193+0000","last_peered":"2024-09-17T07:25:15.889193+0000","last_clean":"2024-09-17T07:25:15.889193+0000","last_became_active":"2024-09-17T07:21:15.347179+0000","last_became_peered":"2024-09-17T07:21:15.347179+0000","last_unstale":"2024-09-17T07:25:15.889193+0000","last_undegraded":"2024-09-17T07:25:15.889193+0000","last_fullsized":"2024-09-17T07:25:15.889193+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'19","last_scrub_stamp":"2024-09-17T07:21:16.070358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:21:16.070358+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0025953619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"94'2","reported_seq":432,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-17T07:25:16.939605+0000","last_change":"2024-09-17T07:25:16.939704+0000","last_active":"2024-09-17T07:25:16.939605+0000","last_peered":"2024-09-17T07:25:16.939605+0000","last_clean":"2024-09-17T07:25:16.939605+0000","last_became_active":"2024-09-17T07:23:57.521742+0000","last_became_peered":"2024-09-17T07:23:57.521742+0000","last_unstale":"2024-09-17T07:25:16.939605+0000","last_undegraded":"2024-09-17T07:25:16.939605+0000","last_fullsized":"2024-09-17T07:25:16.939605+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-09-17T07:25:16.939572+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-09-17T07:07:31.717888+0000","last_clean_scrub_stamp":"2024-09-17T07:25:16.939572+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:43.987976+0000","scrub_duration":0.001518578,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":425,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-09-17T07:25:15.382652+0000","last_change":"2024-09-17T07:23:57.495235+0000","last_active":"2024-09-17T07:25:15.382652+0000","last_peered":"2024-09-17T07:25:15.382652+0000","last_clean":"2024-09-17T07:25:15.382652+0000","last_became_active":"2024-09-17T07:23:57.494701+0000","last_became_peered":"2024-09-17T07:23:57.494701+0000","last_unstale":"2024-09-17T07:25:15.382652+0000","last_undegraded":"2024-09-17T07:25:15.382652+0000","last_fullsized":"2024-09-17T07:25:15.382652+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:16.640384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:20:16.640384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003133224,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":402,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-17T07:25:16.387091+0000","last_change":"2024-09-17T07:25:02.641851+0000","last_active":"2024-09-17T07:25:16.387091+0000","last_peered":"2024-09-17T07:25:16.387091+0000","last_clean":"2024-09-17T07:25:16.387091+0000","last_became_active":"2024-09-17T07:23:57.479008+0000","last_became_peered":"2024-09-17T07:23:57.479008+0000","last_unstale":"2024-09-17T07:25:16.387091+0000","last_undegraded":"2024-09-17T07:25:16.387091+0000","last_fullsized":"2024-09-17T07:25:16.387091+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:02.641763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:49.573666+0000","last_clean_scrub_stamp":"2024-09-17T07:25:02.641763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:19.718312+0000","scrub_duration":0.001362083,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"439'12","reported_seq":492,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:19.741248+0000","last_change":"2024-09-17T07:25:01.921597+0000","last_active":"2024-09-17T07:25:19.741248+0000","last_peered":"2024-09-17T07:25:19.741248+0000","last_clean":"2024-09-17T07:25:19.741248+0000","last_became_active":"2024-09-17T07:09:24.396368+0000","last_became_peered":"2024-09-17T07:09:24.396368+0000","last_unstale":"2024-09-17T07:25:19.741248+0000","last_undegraded":"2024-09-17T07:25:19.741248+0000","last_fullsized":"2024-09-17T07:25:19.741248+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'12","last_scrub_stamp":"2024-09-17T07:25:01.921507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:25:01.921507+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:21.393421+0000","scrub_duration":0.0020119589999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":506,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-17T07:25:19.124760+0000","last_change":"2024-09-17T07:25:19.124760+0000","last_active":"2024-09-17T07:25:19.124760+0000","last_peered":"2024-09-17T07:25:19.124760+0000","last_clean":"2024-09-17T07:25:19.124760+0000","last_became_active":"2024-09-17T07:23:57.476234+0000","last_became_peered":"2024-09-17T07:23:57.476234+0000","last_unstale":"2024-09-17T07:25:19.124760+0000","last_undegraded":"2024-09-17T07:25:19.124760+0000","last_fullsized":"2024-09-17T07:25:19.124760+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:19.124658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:25:19.124658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:22.203780+0000","scrub_duration":0.0013943600000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":584,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-09-17T07:25:15.609644+0000","last_change":"2024-09-17T07:25:03.562390+0000","last_active":"2024-09-17T07:25:15.609644+0000","last_peered":"2024-09-17T07:25:15.609644+0000","last_clean":"2024-09-17T07:25:15.609644+0000","last_became_active":"2024-09-17T07:22:13.582229+0000","last_became_peered":"2024-09-17T07:22:13.582229+0000","last_unstale":"2024-09-17T07:25:15.609644+0000","last_undegraded":"2024-09-17T07:25:15.609644+0000","last_fullsized":"2024-09-17T07:25:15.609644+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:03.562297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:54.553970+0000","last_clean_scrub_stamp":"2024-09-17T07:25:03.562297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:19.909241+0000","scrub_duration":0.0012245349999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":88,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:20.087547+0000","last_change":"2024-09-17T07:22:13.624740+0000","last_active":"2024-09-17T07:25:20.087547+0000","last_peered":"2024-09-17T07:25:20.087547+0000","last_clean":"2024-09-17T07:25:20.087547+0000","last_became_active":"2024-09-17T07:22:13.624127+0000","last_became_peered":"2024-09-17T07:22:13.624127+0000","last_unstale":"2024-09-17T07:25:20.087547+0000","last_undegraded":"2024-09-17T07:25:20.087547+0000","last_fullsized":"2024-09-17T07:25:20.087547+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:25.003509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:20:25.003509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002164586,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"439'27","reported_seq":127,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:20.087426+0000","last_change":"2024-09-17T07:23:57.476210+0000","last_active":"2024-09-17T07:25:20.087426+0000","last_peered":"2024-09-17T07:25:20.087426+0000","last_clean":"2024-09-17T07:25:20.087426+0000","last_became_active":"2024-09-17T07:23:57.475861+0000","last_became_peered":"2024-09-17T07:23:57.475861+0000","last_unstale":"2024-09-17T07:25:20.087426+0000","last_undegraded":"2024-09-17T07:25:20.087426+0000","last_fullsized":"2024-09-17T07:25:20.087426+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'19","last_scrub_stamp":"2024-09-17T07:20:28.113977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:20:28.113977+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.013438802,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":583,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-09-17T07:25:15.610340+0000","last_change":"2024-09-17T07:24:23.526255+0000","last_active":"2024-09-17T07:25:15.610340+0000","last_peered":"2024-09-17T07:25:15.610340+0000","last_clean":"2024-09-17T07:25:15.610340+0000","last_became_active":"2024-09-17T07:23:57.486575+0000","last_became_peered":"2024-09-17T07:23:57.486575+0000","last_unstale":"2024-09-17T07:25:15.610340+0000","last_undegraded":"2024-09-17T07:25:15.610340+0000","last_fullsized":"2024-09-17T07:25:15.610340+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:24:23.526138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:18:41.951382+0000","last_clean_scrub_stamp":"2024-09-17T07:24:23.526138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:48.414517+0000","scrub_duration":0.0045997989999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":88,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:20.087455+0000","last_change":"2024-09-17T07:22:13.736537+0000","last_active":"2024-09-17T07:25:20.087455+0000","last_peered":"2024-09-17T07:25:20.087455+0000","last_clean":"2024-09-17T07:25:20.087455+0000","last_became_active":"2024-09-17T07:22:13.710754+0000","last_became_peered":"2024-09-17T07:22:13.710754+0000","last_unstale":"2024-09-17T07:25:20.087455+0000","last_undegraded":"2024-09-17T07:25:20.087455+0000","last_fullsized":"2024-09-17T07:25:20.087455+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:30.038044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:20:30.038044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":398,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-17T07:25:17.663679+0000","last_change":"2024-09-17T07:25:17.663679+0000","last_active":"2024-09-17T07:25:17.663679+0000","last_peered":"2024-09-17T07:25:17.663679+0000","last_clean":"2024-09-17T07:25:17.663679+0000","last_became_active":"2024-09-17T07:09:24.392216+0000","last_became_peered":"2024-09-17T07:09:24.392216+0000","last_unstale":"2024-09-17T07:25:17.663679+0000","last_undegraded":"2024-09-17T07:25:17.663679+0000","last_fullsized":"2024-09-17T07:25:17.663679+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:17.663576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:25:17.663576+0000","last_clean_scrub_stamp":"2024-09-17T07:25:17.663576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:28.898950+0000","scrub_duration":0.0013785539999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1c","version":"326'2","reported_seq":107,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:20.088413+0000","last_change":"2024-09-17T07:23:57.476548+0000","last_active":"2024-09-17T07:25:20.088413+0000","last_peered":"2024-09-17T07:25:20.088413+0000","last_clean":"2024-09-17T07:25:20.088413+0000","last_became_active":"2024-09-17T07:23:57.475630+0000","last_became_peered":"2024-09-17T07:23:57.475630+0000","last_unstale":"2024-09-17T07:25:20.088413+0000","last_undegraded":"2024-09-17T07:25:20.088413+0000","last_fullsized":"2024-09-17T07:25:20.088413+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'2","last_scrub_stamp":"2024-09-17T07:20:19.609678+0000","last_deep_scrub":"326'2","last_deep_scrub_stamp":"2024-09-17T07:17:36.974358+0000","last_clean_scrub_stamp":"2024-09-17T07:20:19.609678+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0053255050000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":546,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-17T07:25:17.089768+0000","last_change":"2024-09-17T07:25:17.089768+0000","last_active":"2024-09-17T07:25:17.089768+0000","last_peered":"2024-09-17T07:25:17.089768+0000","last_clean":"2024-09-17T07:25:17.089768+0000","last_became_active":"2024-09-17T07:23:57.473995+0000","last_became_peered":"2024-09-17T07:23:57.473995+0000","last_unstale":"2024-09-17T07:25:17.089768+0000","last_undegraded":"2024-09-17T07:25:17.089768+0000","last_fullsized":"2024-09-17T07:25:17.089768+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:17.089691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:25:17.089691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:35.172830+0000","scrub_duration":0.0010547029999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"439'35","reported_seq":708,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-17T07:25:16.387543+0000","last_change":"2024-09-17T07:23:57.480450+0000","last_active":"2024-09-17T07:25:16.387543+0000","last_peered":"2024-09-17T07:25:16.387543+0000","last_clean":"2024-09-17T07:25:16.387543+0000","last_became_active":"2024-09-17T07:23:57.479217+0000","last_became_peered":"2024-09-17T07:23:57.479217+0000","last_unstale":"2024-09-17T07:25:16.387543+0000","last_undegraded":"2024-09-17T07:25:16.387543+0000","last_fullsized":"2024-09-17T07:25:16.387543+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'25","last_scrub_stamp":"2024-09-17T07:20:39.458854+0000","last_deep_scrub":"294'25","last_deep_scrub_stamp":"2024-09-17T07:16:00.471311+0000","last_clean_scrub_stamp":"2024-09-17T07:20:39.458854+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.005100612,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"436'10","reported_seq":882,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-09-17T07:25:15.783109+0000","last_change":"2024-09-17T07:25:01.611646+0000","last_active":"2024-09-17T07:25:15.783109+0000","last_peered":"2024-09-17T07:25:15.783109+0000","last_clean":"2024-09-17T07:25:15.783109+0000","last_became_active":"2024-09-17T07:22:13.574012+0000","last_became_peered":"2024-09-17T07:22:13.574012+0000","last_unstale":"2024-09-17T07:25:15.783109+0000","last_undegraded":"2024-09-17T07:25:15.783109+0000","last_fullsized":"2024-09-17T07:25:15.783109+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"436'10","last_scrub_stamp":"2024-09-17T07:25:01.611504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:25:01.611504+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:07.992598+0000","scrub_duration":0.001846009,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":418,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-09-17T07:25:15.383875+0000","last_change":"2024-09-17T07:25:12.891473+0000","last_active":"2024-09-17T07:25:15.383875+0000","last_peered":"2024-09-17T07:25:15.383875+0000","last_clean":"2024-09-17T07:25:15.383875+0000","last_became_active":"2024-09-17T07:23:57.478429+0000","last_became_peered":"2024-09-17T07:23:57.478429+0000","last_unstale":"2024-09-17T07:25:15.383875+0000","last_undegraded":"2024-09-17T07:25:15.383875+0000","last_fullsized":"2024-09-17T07:25:15.383875+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:12.891393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:25:12.891393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:13.870895+0000","scrub_duration":0.0019788359999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":409,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-09-17T07:25:15.609503+0000","last_change":"2024-09-17T07:24:29.609437+0000","last_active":"2024-09-17T07:25:15.609503+0000","last_peered":"2024-09-17T07:25:15.609503+0000","last_clean":"2024-09-17T07:25:15.609503+0000","last_became_active":"2024-09-17T07:09:24.403360+0000","last_became_peered":"2024-09-17T07:09:24.403360+0000","last_unstale":"2024-09-17T07:25:15.609503+0000","last_undegraded":"2024-09-17T07:25:15.609503+0000","last_fullsized":"2024-09-17T07:25:15.609503+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:24:29.609321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:10:29.325664+0000","last_clean_scrub_stamp":"2024-09-17T07:24:29.609321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:50.013021+0000","scrub_duration":0.024336869000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"439'27","reported_seq":668,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-09-17T07:25:15.610296+0000","last_change":"2024-09-17T07:24:28.562815+0000","last_active":"2024-09-17T07:25:15.610296+0000","last_peered":"2024-09-17T07:25:15.610296+0000","last_clean":"2024-09-17T07:25:15.610296+0000","last_became_active":"2024-09-17T07:23:57.486951+0000","last_became_peered":"2024-09-17T07:23:57.486951+0000","last_unstale":"2024-09-17T07:25:15.610296+0000","last_undegraded":"2024-09-17T07:25:15.610296+0000","last_fullsized":"2024-09-17T07:25:15.610296+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'27","last_scrub_stamp":"2024-09-17T07:24:28.562693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:24:28.562693+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:28.808461+0000","scrub_duration":0.0082361729999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":583,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-17T07:25:16.387498+0000","last_change":"2024-09-17T07:24:43.779871+0000","last_active":"2024-09-17T07:25:16.387498+0000","last_peered":"2024-09-17T07:25:16.387498+0000","last_clean":"2024-09-17T07:25:16.387498+0000","last_became_active":"2024-09-17T07:24:43.779553+0000","last_became_peered":"2024-09-17T07:24:43.779553+0000","last_unstale":"2024-09-17T07:25:16.387498+0000","last_undegraded":"2024-09-17T07:25:16.387498+0000","last_fullsized":"2024-09-17T07:25:16.387498+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:37.435106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:20:37.435106+0000","last_clean_scrub_stamp":"2024-09-17T07:20:37.435106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0042822729999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":559,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-09-17T07:25:15.890517+0000","last_change":"2024-09-17T07:23:57.474023+0000","last_active":"2024-09-17T07:25:15.890517+0000","last_peered":"2024-09-17T07:25:15.890517+0000","last_clean":"2024-09-17T07:25:15.890517+0000","last_became_active":"2024-09-17T07:23:57.473629+0000","last_became_peered":"2024-09-17T07:23:57.473629+0000","last_unstale":"2024-09-17T07:25:15.890517+0000","last_undegraded":"2024-09-17T07:25:15.890517+0000","last_fullsized":"2024-09-17T07:25:15.890517+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:26.001442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:20:26.001442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001725695,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":608,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-09-17T07:25:15.609921+0000","last_change":"2024-09-17T07:25:09.562025+0000","last_active":"2024-09-17T07:25:15.609921+0000","last_peered":"2024-09-17T07:25:15.609921+0000","last_clean":"2024-09-17T07:25:15.609921+0000","last_became_active":"2024-09-17T07:22:13.587273+0000","last_became_peered":"2024-09-17T07:22:13.587273+0000","last_unstale":"2024-09-17T07:25:15.609921+0000","last_undegraded":"2024-09-17T07:25:15.609921+0000","last_fullsized":"2024-09-17T07:25:15.609921+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:09.561794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:11:18.559877+0000","last_clean_scrub_stamp":"2024-09-17T07:25:09.561794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:21.705259+0000","scrub_duration":0.001513239,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"439'39","reported_seq":494,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-09-17T07:25:15.384689+0000","last_change":"2024-09-17T07:24:43.799133+0000","last_active":"2024-09-17T07:25:15.384689+0000","last_peered":"2024-09-17T07:25:15.384689+0000","last_clean":"2024-09-17T07:25:15.384689+0000","last_became_active":"2024-09-17T07:24:43.771752+0000","last_became_peered":"2024-09-17T07:24:43.771752+0000","last_unstale":"2024-09-17T07:25:15.384689+0000","last_undegraded":"2024-09-17T07:25:15.384689+0000","last_fullsized":"2024-09-17T07:25:15.384689+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'27","last_scrub_stamp":"2024-09-17T07:21:11.488225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:21:11.488225+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0089238530000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":382,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-09-17T07:25:15.384528+0000","last_change":"2024-09-17T07:24:23.943586+0000","last_active":"2024-09-17T07:25:15.384528+0000","last_peered":"2024-09-17T07:25:15.384528+0000","last_clean":"2024-09-17T07:25:15.384528+0000","last_became_active":"2024-09-17T07:21:15.348754+0000","last_became_peered":"2024-09-17T07:21:15.348754+0000","last_unstale":"2024-09-17T07:25:15.384528+0000","last_undegraded":"2024-09-17T07:25:15.384528+0000","last_fullsized":"2024-09-17T07:25:15.384528+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:24:23.943437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:24:23.943437+0000","last_clean_scrub_stamp":"2024-09-17T07:24:23.943437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:47.203411+0000","scrub_duration":0.012336285000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":429,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-17T07:25:18.515438+0000","last_change":"2024-09-17T07:25:18.515504+0000","last_active":"2024-09-17T07:25:18.515438+0000","last_peered":"2024-09-17T07:25:18.515438+0000","last_clean":"2024-09-17T07:25:18.515438+0000","last_became_active":"2024-09-17T07:23:57.487706+0000","last_became_peered":"2024-09-17T07:23:57.487706+0000","last_unstale":"2024-09-17T07:25:18.515438+0000","last_undegraded":"2024-09-17T07:25:18.515438+0000","last_fullsized":"2024-09-17T07:25:18.515438+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:18.515421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:25:18.515421+0000","last_clean_scrub_stamp":"2024-09-17T07:25:18.515421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:35.662814+0000","scrub_duration":0.001450956,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":99,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:20.087913+0000","last_change":"2024-09-17T07:24:26.807871+0000","last_active":"2024-09-17T07:25:20.087913+0000","last_peered":"2024-09-17T07:25:20.087913+0000","last_clean":"2024-09-17T07:25:20.087913+0000","last_became_active":"2024-09-17T07:22:13.624057+0000","last_became_peered":"2024-09-17T07:22:13.624057+0000","last_unstale":"2024-09-17T07:25:20.087913+0000","last_undegraded":"2024-09-17T07:25:20.087913+0000","last_fullsized":"2024-09-17T07:25:20.087913+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:24:26.807790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:24:26.807790+0000","last_clean_scrub_stamp":"2024-09-17T07:24:26.807790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:39.305925+0000","scrub_duration":0.001095604,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"439'29","reported_seq":323,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-09-17T07:25:15.384112+0000","last_change":"2024-09-17T07:25:03.872269+0000","last_active":"2024-09-17T07:25:15.384112+0000","last_peered":"2024-09-17T07:25:15.384112+0000","last_clean":"2024-09-17T07:25:15.384112+0000","last_became_active":"2024-09-17T07:21:15.351512+0000","last_became_peered":"2024-09-17T07:21:15.351512+0000","last_unstale":"2024-09-17T07:25:15.384112+0000","last_undegraded":"2024-09-17T07:25:15.384112+0000","last_fullsized":"2024-09-17T07:25:15.384112+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'29","last_scrub_stamp":"2024-09-17T07:25:03.872175+0000","last_deep_scrub":"86'17","last_deep_scrub_stamp":"2024-09-17T07:03:18.241067+0000","last_clean_scrub_stamp":"2024-09-17T07:25:03.872175+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:09.824307+0000","scrub_duration":0.001972002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":582,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-09-17T07:25:15.890193+0000","last_change":"2024-09-17T07:22:13.591148+0000","last_active":"2024-09-17T07:25:15.890193+0000","last_peered":"2024-09-17T07:25:15.890193+0000","last_clean":"2024-09-17T07:25:15.890193+0000","last_became_active":"2024-09-17T07:22:13.590622+0000","last_became_peered":"2024-09-17T07:22:13.590622+0000","last_unstale":"2024-09-17T07:25:15.890193+0000","last_undegraded":"2024-09-17T07:25:15.890193+0000","last_fullsized":"2024-09-17T07:25:15.890193+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:21.073159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:20:21.073159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0093893710000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":382,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1537,"num_read_kb":1294,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":254,"num_bytes_recovered":5066,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":13004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1027,"ondisk_log_size":1027,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":249,"num_read_kb":244,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1386,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":12,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":4578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":483028,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":174,"num_read_kb":250,"num_write":147,"num_write_kb":1976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1449085,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1430,"ondisk_log_size":1430,"up":33,"acting":33,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698038044,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1113604,"kb_used_data":3340,"kb_used_omap":1,"kb_used_meta":1109886,"kb_avail":92631548,"statfs":{"total":95995035648,"available":94854705152,"internally_reserved":0,"allocated":3420160,"data_stored":2108656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":1136523723},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:24:46 2024","interfaces":[{"interface":"back","average":{"1min":0.95399999999999996,"5min":1.7030000000000001,"15min":1.5129999999999999},"min":{"1min":0.30499999999999999,"5min":0.182,"15min":0.182},"max":{"1min":3.306,"5min":43.905000000000001,"15min":43.905000000000001},"last":0.57699999999999996},{"interface":"front","average":{"1min":1.131,"5min":1.5720000000000001,"15min":1.5640000000000001},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":5.6619999999999999,"5min":44.432000000000002,"15min":44.432000000000002},"last":0.59999999999999998}]},{"osd":1,"last update":"Tue Sep 17 07:25:21 2024","interfaces":[{"interface":"back","average":{"1min":1.1299999999999999,"5min":1.1439999999999999,"15min":1.216},"min":{"1min":0.32900000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":3.948,"5min":12.112,"15min":12.112},"last":0.46500000000000002},{"interface":"front","average":{"1min":1.2749999999999999,"5min":1.097,"15min":1.179},"min":{"1min":0.378,"5min":0.216,"15min":0.216},"max":{"1min":5.6210000000000004,"5min":12.273,"15min":12.273},"last":0.503}]},{"osd":2,"last update":"Tue Sep 17 07:24:46 2024","interfaces":[{"interface":"back","average":{"1min":1.1779999999999999,"5min":0.86799999999999999,"15min":1.0589999999999999},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":4.165,"5min":10.972,"15min":10.972},"last":0.61599999999999999},{"interface":"front","average":{"1min":1.226,"5min":0.97799999999999998,"15min":1.0429999999999999},"min":{"1min":0.35699999999999998,"5min":0.255,"15min":0.224},"max":{"1min":4.4320000000000004,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.42999999999999999}]},{"osd":3,"last update":"Tue Sep 17 07:24:46 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":0.69099999999999995,"15min":1.0109999999999999},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":4.2009999999999996,"5min":4.2009999999999996,"15min":10.122999999999999},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.629,"15min":0.98299999999999998},"min":{"1min":0.313,"5min":0.26700000000000002,"15min":0.26500000000000001},"max":{"1min":3.0630000000000002,"5min":4.1719999999999997,"15min":12.465999999999999},"last":0.48199999999999998}]},{"osd":4,"last update":"Tue Sep 17 07:24:46 2024","interfaces":[{"interface":"back","average":{"1min":1.345,"5min":0.78000000000000003,"15min":1.113},"min":{"1min":0.23699999999999999,"5min":0.22900000000000001,"15min":0.222},"max":{"1min":6.21,"5min":13.316000000000001,"15min":21.140999999999998},"last":0.56299999999999994},{"interface":"front","average":{"1min":1.2589999999999999,"5min":0.82999999999999996,"15min":1.296},"min":{"1min":0.25800000000000001,"5min":0.251,"15min":0.23599999999999999},"max":{"1min":4.2850000000000001,"5min":12.272,"15min":19.420000000000002},"last":0.53200000000000003}]},{"osd":5,"last update":"Tue Sep 17 07:24:57 2024","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":3.0859999999999999,"5min":3.0859999999999999,"15min":3.0859999999999999},"last":0.65400000000000003},{"interface":"front","average":{"1min":1.097,"5min":1.097,"15min":1.097},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":5.7350000000000003,"5min":5.7350000000000003,"15min":5.7350000000000003},"last":0.63600000000000001}]},{"osd":6,"last update":"Tue Sep 17 07:24:46 2024","interfaces":[{"interface":"back","average":{"1min":1.4079999999999999,"5min":1.4950000000000001,"15min":1.6040000000000001},"min":{"1min":0.36399999999999999,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":6.9359999999999999,"5min":60.445999999999998,"15min":60.445999999999998},"last":0.69499999999999995},{"interface":"front","average":{"1min":1.5389999999999999,"5min":1.5580000000000001,"15min":1.71},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":4.3550000000000004,"5min":60.332000000000001,"15min":60.332000000000001},"last":0.67400000000000004}]}]},{"osd":6,"up_from":50,"seq":214748365086,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1035904,"kb_used_data":3956,"kb_used_omap":1,"kb_used_meta":1031934,"kb_avail":92709248,"statfs":{"total":95995035648,"available":94934269952,"internally_reserved":0,"allocated":4050944,"data_stored":2569264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1056700871},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:24:30 2024","interfaces":[{"interface":"back","average":{"1min":8.1080000000000005,"5min":4.0739999999999998,"15min":2.5329999999999999},"min":{"1min":0.22600000000000001,"5min":0.19700000000000001,"15min":0.185},"max":{"1min":110.376,"5min":110.376,"15min":126.902},"last":0.39500000000000002},{"interface":"front","average":{"1min":8.0510000000000002,"5min":3.9180000000000001,"15min":2.3439999999999999},"min":{"1min":0.25700000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":110.164,"5min":110.164,"15min":127.06399999999999},"last":0.27200000000000002}]},{"osd":1,"last update":"Tue Sep 17 07:24:21 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":5.8209999999999997,"15min":5.9729999999999999},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":2.153,"5min":199.32400000000001,"15min":199.32400000000001},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.67700000000000005,"5min":6.0460000000000003,"15min":6.2919999999999998},"min":{"1min":0.251,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":2.5329999999999999,"5min":198.88499999999999,"15min":198.88499999999999},"last":0.312}]},{"osd":2,"last update":"Tue Sep 17 07:24:30 2024","interfaces":[{"interface":"back","average":{"1min":1.3340000000000001,"5min":1.677,"15min":1.236},"min":{"1min":0.28499999999999998,"5min":0.23300000000000001,"15min":0.219},"max":{"1min":10.930999999999999,"5min":36.289000000000001,"15min":36.289000000000001},"last":0.33800000000000002},{"interface":"front","average":{"1min":1.167,"5min":1.546,"15min":1.0960000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.22900000000000001},"max":{"1min":4.2190000000000003,"5min":36.326999999999998,"15min":36.326999999999998},"last":0.47499999999999998}]},{"osd":3,"last update":"Tue Sep 17 07:24:30 2024","interfaces":[{"interface":"back","average":{"1min":1.024,"5min":0.74399999999999999,"15min":0.91600000000000004},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.24399999999999999},"max":{"1min":6.3289999999999997,"5min":6.3289999999999997,"15min":6.8639999999999999},"last":0.42599999999999999},{"interface":"front","average":{"1min":1.3400000000000001,"5min":0.95699999999999996,"15min":1.0860000000000001},"min":{"1min":0.32500000000000001,"5min":0.27600000000000002,"15min":0.27500000000000002},"max":{"1min":10.999000000000001,"5min":10.999000000000001,"15min":10.999000000000001},"last":0.436}]},{"osd":4,"last update":"Tue Sep 17 07:24:30 2024","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":0.91300000000000003,"15min":2.3370000000000002},"min":{"1min":0.26000000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":3.9790000000000001,"5min":9.5419999999999998,"15min":260.77199999999999},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.69499999999999995,"15min":2.3359999999999999},"min":{"1min":0.308,"5min":0.26100000000000001,"15min":0.19800000000000001},"max":{"1min":3.2090000000000001,"5min":3.609,"15min":260.69099999999997},"last":0.377}]},{"osd":5,"last update":"Tue Sep 17 07:24:56 2024","interfaces":[{"interface":"back","average":{"1min":2.1280000000000001,"5min":2.1280000000000001,"15min":2.1280000000000001},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":13.551,"5min":13.551,"15min":13.551},"last":0.36699999999999999},{"interface":"front","average":{"1min":2.387,"5min":2.387,"15min":2.387},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":8.1720000000000006,"5min":8.1720000000000006,"15min":8.1720000000000006},"last":0.44900000000000001}]},{"osd":7,"last update":"Tue Sep 17 07:24:51 2024","interfaces":[{"interface":"back","average":{"1min":1.948,"5min":1.0229999999999999,"15min":1.784},"min":{"1min":0.33600000000000002,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":9.1199999999999992,"5min":9.1199999999999992,"15min":131.95699999999999},"last":0.46000000000000002},{"interface":"front","average":{"1min":2.206,"5min":1.149,"15min":1.6950000000000001},"min":{"1min":0.31900000000000001,"5min":0.26900000000000002,"15min":0.252},"max":{"1min":7.077,"5min":9.5850000000000009,"15min":131.833},"last":0.49199999999999999}]}]},{"osd":5,"up_from":476,"seq":2044404432914,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":337520,"kb_used_data":203876,"kb_used_omap":4,"kb_used_meta":133627,"kb_avail":93407632,"statfs":{"total":95995035648,"available":95649415168,"internally_reserved":0,"allocated":208769024,"data_stored":207308095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4441,"internal_metadata":136834727},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:25:00 2024","interfaces":[{"interface":"back","average":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":3.4220000000000002,"5min":3.4220000000000002,"15min":3.4220000000000002},"last":0.66000000000000003},{"interface":"front","average":{"1min":1.5960000000000001,"5min":1.5960000000000001,"15min":1.5960000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":6.7619999999999996,"5min":6.7619999999999996,"15min":6.7619999999999996},"last":0.50700000000000001}]},{"osd":1,"last update":"Tue Sep 17 07:25:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":3.476,"5min":3.476,"15min":3.476},"last":0.63300000000000001},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":6.665,"5min":6.665,"15min":6.665},"last":0.33900000000000002}]},{"osd":2,"last update":"Tue Sep 17 07:25:00 2024","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.2190000000000001,"15min":1.2190000000000001},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":3.544,"5min":3.544,"15min":3.544},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":2.8359999999999999,"5min":2.8359999999999999,"15min":2.8359999999999999},"last":0.55700000000000005}]},{"osd":3,"last update":"Tue Sep 17 07:25:00 2024","interfaces":[{"interface":"back","average":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":3.7040000000000002,"5min":3.7040000000000002,"15min":3.7040000000000002},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":2.8719999999999999,"5min":2.8719999999999999,"15min":2.8719999999999999},"last":0.61799999999999999}]},{"osd":4,"last update":"Tue Sep 17 07:25:00 2024","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":3.4399999999999999,"5min":3.4399999999999999,"15min":3.4399999999999999},"last":0.57899999999999996},{"interface":"front","average":{"1min":1.26,"5min":1.26,"15min":1.26},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":3.0139999999999998,"5min":3.0139999999999998,"15min":3.0139999999999998},"last":0.38400000000000001}]},{"osd":6,"last update":"Tue Sep 17 07:25:00 2024","interfaces":[{"interface":"back","average":{"1min":1.77,"5min":1.77,"15min":1.77},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":10.92,"5min":10.92,"15min":10.92},"last":0.42799999999999999},{"interface":"front","average":{"1min":1.526,"5min":1.526,"15min":1.526},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":10.202999999999999,"5min":10.202999999999999,"15min":10.202999999999999},"last":0.53100000000000003}]},{"osd":7,"last update":"Tue Sep 17 07:25:00 2024","interfaces":[{"interface":"back","average":{"1min":3.3820000000000001,"5min":3.3820000000000001,"15min":3.3820000000000001},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":45.555,"5min":45.555,"15min":45.555},"last":0.47099999999999997},{"interface":"front","average":{"1min":3.6920000000000002,"5min":3.6920000000000002,"15min":3.6920000000000002},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":47.213999999999999,"5min":47.213999999999999,"15min":47.213999999999999},"last":0.58999999999999997}]}]},{"osd":4,"up_from":33,"seq":141733921065,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1004972,"kb_used_data":3356,"kb_used_omap":1,"kb_used_meta":1001598,"kb_avail":92740180,"statfs":{"total":95995035648,"available":94965944320,"internally_reserved":0,"allocated":3436544,"data_stored":1977711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1025636806},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:24:30 2024","interfaces":[{"interface":"back","average":{"1min":1.1699999999999999,"5min":1.869,"15min":1.4299999999999999},"min":{"1min":0.28399999999999997,"5min":0.20300000000000001,"15min":0.20000000000000001},"max":{"1min":6.7290000000000001,"5min":39.473999999999997,"15min":39.473999999999997},"last":0.317},{"interface":"front","average":{"1min":0.97299999999999998,"5min":1.8899999999999999,"15min":1.5229999999999999},"min":{"1min":0.21299999999999999,"5min":0.20599999999999999,"15min":0.20499999999999999},"max":{"1min":3.1349999999999998,"5min":39.654000000000003,"15min":39.654000000000003},"last":0.40899999999999997}]},{"osd":1,"last update":"Tue Sep 17 07:24:19 2024","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":2.5219999999999998,"15min":2.7610000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":6.8399999999999999,"5min":39.439,"15min":39.439},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.93700000000000006,"5min":2.6469999999999998,"15min":2.819},"min":{"1min":0.246,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":4.2999999999999998,"5min":39.325000000000003,"15min":39.325000000000003},"last":0.63100000000000001}]},{"osd":2,"last update":"Tue Sep 17 07:24:30 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.93400000000000005,"15min":1.081},"min":{"1min":0.26300000000000001,"5min":0.24399999999999999,"15min":0.23499999999999999},"max":{"1min":6.8129999999999997,"5min":10.238,"15min":10.238},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.97899999999999998,"5min":0.89600000000000002,"15min":1.0740000000000001},"min":{"1min":0.309,"5min":0.26600000000000001,"15min":0.23400000000000001},"max":{"1min":2.9249999999999998,"5min":10.19,"15min":16.655999999999999},"last":0.35399999999999998}]},{"osd":3,"last update":"Tue Sep 17 07:24:30 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.70799999999999996,"15min":0.94799999999999995},"min":{"1min":0.28299999999999997,"5min":0.27600000000000002,"15min":0.25800000000000001},"max":{"1min":2.8929999999999998,"5min":2.8929999999999998,"15min":15.433999999999999},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.72499999999999998,"15min":0.98899999999999999},"min":{"1min":0.32000000000000001,"5min":0.28100000000000003,"15min":0.27100000000000002},"max":{"1min":2.3300000000000001,"5min":8.7360000000000007,"15min":16.111000000000001},"last":0.50900000000000001}]},{"osd":5,"last update":"Tue Sep 17 07:25:00 2024","interfaces":[{"interface":"back","average":{"1min":1.4359999999999999,"5min":1.4359999999999999,"15min":1.4359999999999999},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":13.128,"5min":13.128,"15min":13.128},"last":0.72499999999999998},{"interface":"front","average":{"1min":1.752,"5min":1.752,"15min":1.752},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":13.218999999999999,"5min":13.218999999999999,"15min":13.218999999999999},"last":0.67200000000000004}]},{"osd":6,"last update":"Tue Sep 17 07:24:30 2024","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":0.81999999999999995,"15min":1.0940000000000001},"min":{"1min":0.31900000000000001,"5min":0.23400000000000001,"15min":0.218},"max":{"1min":4.4989999999999997,"5min":4.7729999999999997,"15min":8.9619999999999997},"last":0.65400000000000003},{"interface":"front","average":{"1min":1.006,"5min":0.90000000000000002,"15min":1.208},"min":{"1min":0.26000000000000001,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":6.782,"5min":6.782,"15min":10.051},"last":0.61099999999999999}]},{"osd":7,"last update":"Tue Sep 17 07:24:45 2024","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":0.98899999999999999,"15min":1.2410000000000001},"min":{"1min":0.318,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":6.2249999999999996,"5min":6.2249999999999996,"15min":7.1550000000000002},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.98899999999999999,"5min":0.78600000000000003,"15min":1.1910000000000001},"min":{"1min":0.28599999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":2.5739999999999998,"5min":4.7439999999999998,"15min":9.0030000000000001},"last":0.59299999999999997}]}]},{"osd":3,"up_from":0,"seq":115964117291,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":18,"seq":77309411634,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1121476,"kb_used_data":3280,"kb_used_omap":1,"kb_used_meta":1117822,"kb_avail":92623676,"statfs":{"total":95995035648,"available":94846644224,"internally_reserved":0,"allocated":3358720,"data_stored":2043848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1144650182},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:25:06 2024","interfaces":[{"interface":"back","average":{"1min":1.966,"5min":3.5059999999999998,"15min":2.6699999999999999},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.154},"max":{"1min":25.98,"5min":69.135999999999996,"15min":131.09800000000001},"last":0.313},{"interface":"front","average":{"1min":2.073,"5min":3.73,"15min":2.6949999999999998},"min":{"1min":0.25600000000000001,"5min":0.17999999999999999,"15min":0.17599999999999999},"max":{"1min":24.314,"5min":69.033000000000001,"15min":131.24700000000001},"last":0.24099999999999999}]},{"osd":1,"last update":"Tue Sep 17 07:24:22 2024","interfaces":[{"interface":"back","average":{"1min":1.968,"5min":2.8860000000000001,"15min":2.274},"min":{"1min":0.20999999999999999,"5min":0.184,"15min":0.184},"max":{"1min":21.23,"5min":91.686999999999998,"15min":91.686999999999998},"last":0.33000000000000002},{"interface":"front","average":{"1min":2.2599999999999998,"5min":3.1989999999999998,"15min":2.6080000000000001},"min":{"1min":0.223,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":21.024000000000001,"5min":93.744,"15min":93.744},"last":0.29099999999999998}]},{"osd":3,"last update":"Tue Sep 17 07:25:13 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.61099999999999999,"15min":0.67800000000000005},"min":{"1min":0.251,"5min":0.20499999999999999,"15min":0.20100000000000001},"max":{"1min":0.99099999999999999,"5min":3.1880000000000002,"15min":23.43},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.69499999999999995,"5min":1.0009999999999999,"15min":0.78500000000000003},"min":{"1min":0.29799999999999999,"5min":0.22500000000000001,"15min":0.22},"max":{"1min":2.4769999999999999,"5min":16.215,"15min":23.408000000000001},"last":0.34599999999999997}]},{"osd":4,"last update":"Tue Sep 17 07:24:40 2024","interfaces":[{"interface":"back","average":{"1min":2.2839999999999998,"5min":1.0920000000000001,"15min":1.3700000000000001},"min":{"1min":0.314,"5min":0.254,"15min":0.24399999999999999},"max":{"1min":25.225999999999999,"5min":25.225999999999999,"15min":75.870999999999995},"last":0.61199999999999999},{"interface":"front","average":{"1min":2.3559999999999999,"5min":1.149,"15min":1.47},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.26300000000000001},"max":{"1min":25.268999999999998,"5min":25.268999999999998,"15min":75.888999999999996},"last":0.63100000000000001}]},{"osd":5,"last update":"Tue Sep 17 07:24:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":1.9610000000000001,"5min":1.9610000000000001,"15min":1.9610000000000001},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":3.907,"5min":3.907,"15min":3.907},"last":0.55100000000000005}]},{"osd":6,"last update":"Tue Sep 17 07:25:13 2024","interfaces":[{"interface":"back","average":{"1min":2.8969999999999998,"5min":2.298,"15min":1.8169999999999999},"min":{"1min":0.32800000000000001,"5min":0.29699999999999999,"15min":0.27500000000000002},"max":{"1min":24.936,"5min":65.143000000000001,"15min":76.081999999999994},"last":0.65700000000000003},{"interface":"front","average":{"1min":2.2839999999999998,"5min":2.1680000000000001,"15min":1.871},"min":{"1min":0.40899999999999997,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":24.884,"5min":65.241,"15min":76.244},"last":0.57099999999999995}]},{"osd":7,"last update":"Tue Sep 17 07:25:06 2024","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.83699999999999997,"15min":1.3520000000000001},"min":{"1min":0.34799999999999998,"5min":0.314,"15min":0.314},"max":{"1min":2.0790000000000002,"5min":4.2720000000000002,"15min":94.269999999999996},"last":0.64500000000000002},{"interface":"front","average":{"1min":1.016,"5min":0.95899999999999996,"15min":1.395},"min":{"1min":0.34499999999999997,"5min":0.32900000000000001,"15min":0.309},"max":{"1min":4.4470000000000001,"5min":4.4470000000000001,"15min":94.634},"last":0.59699999999999998}]}]},{"osd":1,"up_from":413,"seq":1773821493298,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":537096,"kb_used_data":3384,"kb_used_omap":4,"kb_used_meta":533691,"kb_avail":93208056,"statfs":{"total":95995035648,"available":95445049344,"internally_reserved":0,"allocated":3465216,"data_stored":1979617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4439,"internal_metadata":546500265},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:24:22 2024","interfaces":[{"interface":"back","average":{"1min":8.1750000000000007,"5min":2.7909999999999999,"15min":2.044},"min":{"1min":0.219,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":120.56399999999999,"5min":120.56399999999999,"15min":120.56399999999999},"last":0.245},{"interface":"front","average":{"1min":8.6140000000000008,"5min":2.7389999999999999,"15min":1.839},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":120.746,"5min":120.746,"15min":120.746},"last":0.26900000000000002}]},{"osd":2,"last update":"Tue Sep 17 07:24:22 2024","interfaces":[{"interface":"back","average":{"1min":1.133,"5min":1.1679999999999999,"15min":1.2969999999999999},"min":{"1min":0.23300000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":6.7300000000000004,"5min":6.7300000000000004,"15min":6.7300000000000004},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.96999999999999997,"15min":1.0760000000000001},"min":{"1min":0.26800000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":1.5860000000000001,"5min":3.919,"15min":3.919},"last":0.33300000000000002}]},{"osd":3,"last update":"Tue Sep 17 07:24:22 2024","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.78500000000000003,"15min":0.80900000000000005},"min":{"1min":0.30599999999999999,"5min":0.253,"15min":0.253},"max":{"1min":3.4239999999999999,"5min":5.9290000000000003,"15min":5.9290000000000003},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.81599999999999995,"15min":0.85799999999999998},"min":{"1min":0.27200000000000002,"5min":0.25,"15min":0.25},"max":{"1min":2.919,"5min":4.1340000000000003,"15min":4.1340000000000003},"last":0.32100000000000001}]},{"osd":4,"last update":"Tue Sep 17 07:24:22 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":4.7809999999999997,"15min":6.4980000000000002},"min":{"1min":0.311,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":3.1970000000000001,"5min":116.88200000000001,"15min":116.88200000000001},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.78700000000000003,"5min":4.7530000000000001,"15min":6.5350000000000001},"min":{"1min":0.34000000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":3.456,"5min":116.98399999999999,"15min":116.98399999999999},"last":0.41099999999999998}]},{"osd":5,"last update":"Tue Sep 17 07:24:57 2024","interfaces":[{"interface":"back","average":{"1min":1.5369999999999999,"5min":1.5369999999999999,"15min":1.5369999999999999},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":9.7409999999999997,"5min":9.7409999999999997,"15min":9.7409999999999997},"last":0.39200000000000002},{"interface":"front","average":{"1min":1.542,"5min":1.542,"15min":1.542},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":11.538,"5min":11.538,"15min":11.538},"last":0.50900000000000001}]},{"osd":6,"last update":"Tue Sep 17 07:24:22 2024","interfaces":[{"interface":"back","average":{"1min":1.0429999999999999,"5min":2.1110000000000002,"15min":1.657},"min":{"1min":0.35299999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":4.2279999999999998,"5min":56.686999999999998,"15min":56.686999999999998},"last":0.47699999999999998},{"interface":"front","average":{"1min":1.21,"5min":2.056,"15min":1.5700000000000001},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":4.1559999999999997,"5min":56.420999999999999,"15min":56.420999999999999},"last":0.53400000000000003}]},{"osd":7,"last update":"Tue Sep 17 07:24:22 2024","interfaces":[{"interface":"back","average":{"1min":1.077,"5min":1.0269999999999999,"15min":1.054},"min":{"1min":0.35199999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":2.9630000000000001,"5min":4.2389999999999999,"15min":4.2389999999999999},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.626,"5min":0.93100000000000005,"15min":1.012},"min":{"1min":0.36899999999999999,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.5640000000000001,"5min":4.2839999999999998,"15min":4.2839999999999998},"last":0.58599999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738681,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1737136,"kb_used_data":434216,"kb_used_omap":1,"kb_used_meta":1302910,"kb_avail":92008016,"statfs":{"total":95995035648,"available":94216208384,"internally_reserved":0,"allocated":444637184,"data_stored":443167107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1334180294},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 17 07:24:19 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":1.627,"15min":1.881},"min":{"1min":0.188,"5min":0.156,"15min":0.156},"max":{"1min":0.96199999999999997,"5min":9.2959999999999994,"15min":9.2959999999999994},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.56100000000000005,"5min":1.325,"15min":1.4470000000000001},"min":{"1min":0.17999999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":2.387,"5min":5.0819999999999999,"15min":5.0819999999999999},"last":0.28499999999999998}]},{"osd":2,"last update":"Tue Sep 17 07:24:59 2024","interfaces":[{"interface":"back","average":{"1min":3.9529999999999998,"5min":2.4750000000000001,"15min":1.256},"min":{"1min":0.21299999999999999,"5min":0.19400000000000001,"15min":0.155},"max":{"1min":55.756,"5min":62.094999999999999,"15min":62.094999999999999},"last":0.24399999999999999},{"interface":"front","average":{"1min":3.9180000000000001,"5min":2.476,"15min":1.242},"min":{"1min":0.255,"5min":0.19,"15min":0.183},"max":{"1min":57.854999999999997,"5min":59.848999999999997,"15min":59.848999999999997},"last":0.42899999999999999}]},{"osd":3,"last update":"Tue Sep 17 07:24:35 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.74299999999999999,"15min":0.69099999999999995},"min":{"1min":0.27700000000000002,"5min":0.23200000000000001,"15min":0.21199999999999999},"max":{"1min":1.1910000000000001,"5min":4.6829999999999998,"15min":5.5880000000000001},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.80400000000000005,"15min":0.67400000000000004},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20399999999999999},"max":{"1min":1.024,"5min":5.1150000000000002,"15min":5.1150000000000002},"last":0.442}]},{"osd":4,"last update":"Tue Sep 17 07:24:35 2024","interfaces":[{"interface":"back","average":{"1min":1.6990000000000001,"5min":1.208,"15min":1.1459999999999999},"min":{"1min":0.29199999999999998,"5min":0.27400000000000002,"15min":0.23400000000000001},"max":{"1min":10.720000000000001,"5min":10.720000000000001,"15min":19.146999999999998},"last":0.50600000000000001},{"interface":"front","average":{"1min":1.8320000000000001,"5min":1.123,"15min":1.165},"min":{"1min":0.32700000000000001,"5min":0.26600000000000001,"15min":0.26200000000000001},"max":{"1min":10.666,"5min":10.666,"15min":19.158999999999999},"last":0.41099999999999998}]},{"osd":5,"last update":"Tue Sep 17 07:24:59 2024","interfaces":[{"interface":"back","average":{"1min":1.056,"5min":1.056,"15min":1.056},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":3.79,"5min":3.79,"15min":3.79},"last":0.497},{"interface":"front","average":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":3.2530000000000001,"5min":3.2530000000000001,"15min":3.2530000000000001},"last":0.54500000000000004}]},{"osd":6,"last update":"Tue Sep 17 07:24:35 2024","interfaces":[{"interface":"back","average":{"1min":1.3009999999999999,"5min":2.2450000000000001,"15min":1.8879999999999999},"min":{"1min":0.34799999999999998,"5min":0.28999999999999998,"15min":0.28199999999999997},"max":{"1min":3.923,"5min":67.873000000000005,"15min":147.13200000000001},"last":0.47799999999999998},{"interface":"front","average":{"1min":1.2809999999999999,"5min":2.359,"15min":1.911},"min":{"1min":0.311,"5min":0.311,"15min":0.30199999999999999},"max":{"1min":4.0229999999999997,"5min":67.914000000000001,"15min":147.172},"last":0.52700000000000002}]},{"osd":7,"last update":"Tue Sep 17 07:24:35 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.88700000000000001,"15min":1.1379999999999999},"min":{"1min":0.35799999999999998,"5min":0.318,"15min":0.29599999999999999},"max":{"1min":2.4820000000000002,"5min":3.843,"15min":6.3300000000000001},"last":0.45300000000000001},{"interface":"front","average":{"1min":1.004,"5min":1.099,"15min":1.218},"min":{"1min":0.36799999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":2.6819999999999999,"5min":6.7300000000000004,"15min":7.2830000000000004},"last":0.57399999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T07:25:24.728 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T07:25:24.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:24 smithi100 ceph-mon[29092]: pgmap v1114: 142 pgs: 142 active+clean; 477 KiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 718 B/s rd, 0 op/s 2024-09-17T07:25:24.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:24 smithi100 ceph-mon[29092]: 6.18 scrub starts 2024-09-17T07:25:24.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:24 smithi100 ceph-mon[29092]: 6.18 scrub ok 2024-09-17T07:25:24.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:24 smithi100 ceph-mon[29092]: from='client.16257 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:25:24.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:24 smithi100 ceph-mon[29092]: 5.0 scrub starts 2024-09-17T07:25:24.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:24 smithi100 ceph-mon[29092]: 5.0 scrub ok 2024-09-17T07:25:24.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:24 smithi100 ceph-mon[29092]: 4.17 scrub starts 2024-09-17T07:25:24.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:24 smithi100 ceph-mon[29092]: 4.17 scrub ok 2024-09-17T07:25:24.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:24 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:25:24.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:24 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:25:24.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:24 smithi100 ceph-mon[39462]: pgmap v1114: 142 pgs: 142 active+clean; 477 KiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 718 B/s rd, 0 op/s 2024-09-17T07:25:24.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:24 smithi100 ceph-mon[39462]: 6.18 scrub starts 2024-09-17T07:25:24.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:24 smithi100 ceph-mon[39462]: 6.18 scrub ok 2024-09-17T07:25:24.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:24 smithi100 ceph-mon[39462]: from='client.16257 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:25:24.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:24 smithi100 ceph-mon[39462]: 5.0 scrub starts 2024-09-17T07:25:24.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:24 smithi100 ceph-mon[39462]: 5.0 scrub ok 2024-09-17T07:25:24.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:24 smithi100 ceph-mon[39462]: 4.17 scrub starts 2024-09-17T07:25:24.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:24 smithi100 ceph-mon[39462]: 4.17 scrub ok 2024-09-17T07:25:24.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:24 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:25:24.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:24 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:25:25.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:24 smithi135 ceph-mon[33238]: pgmap v1114: 142 pgs: 142 active+clean; 477 KiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 718 B/s rd, 0 op/s 2024-09-17T07:25:25.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:24 smithi135 ceph-mon[33238]: 6.18 scrub starts 2024-09-17T07:25:25.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:24 smithi135 ceph-mon[33238]: 6.18 scrub ok 2024-09-17T07:25:25.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:24 smithi135 ceph-mon[33238]: from='client.16257 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:25:25.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:24 smithi135 ceph-mon[33238]: 5.0 scrub starts 2024-09-17T07:25:25.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:24 smithi135 ceph-mon[33238]: 5.0 scrub ok 2024-09-17T07:25:25.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:24 smithi135 ceph-mon[33238]: 4.17 scrub starts 2024-09-17T07:25:25.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:24 smithi135 ceph-mon[33238]: 4.17 scrub ok 2024-09-17T07:25:25.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:24 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:25:25.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:24 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:25:25.049 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-17T07:25:25.050 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:25:25.281 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:25:25.325 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:25:26.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:25 smithi135 ceph-mon[33238]: 5.1e scrub starts 2024-09-17T07:25:26.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:25 smithi135 ceph-mon[33238]: 5.1e scrub ok 2024-09-17T07:25:26.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:25 smithi135 ceph-mon[33238]: osdmap e512: 8 total, 8 up, 7 in 2024-09-17T07:25:26.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:25 smithi100 ceph-mon[29092]: 5.1e scrub starts 2024-09-17T07:25:26.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:25 smithi100 ceph-mon[29092]: 5.1e scrub ok 2024-09-17T07:25:26.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:25 smithi100 ceph-mon[29092]: osdmap e512: 8 total, 8 up, 7 in 2024-09-17T07:25:26.127 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:25:25 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:25:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:25:26.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:25 smithi100 ceph-mon[39462]: 5.1e scrub starts 2024-09-17T07:25:26.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:25 smithi100 ceph-mon[39462]: 5.1e scrub ok 2024-09-17T07:25:26.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:25 smithi100 ceph-mon[39462]: osdmap e512: 8 total, 8 up, 7 in 2024-09-17T07:25:26.168 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-17T07:25:26.382 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:25:27.006 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:26 smithi100 ceph-mon[39462]: 2.14 scrub starts 2024-09-17T07:25:27.006 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:26 smithi100 ceph-mon[39462]: 2.14 scrub ok 2024-09-17T07:25:27.006 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:26 smithi100 ceph-mon[39462]: pgmap v1116: 142 pgs: 142 active+clean; 425 KiB data, 6.5 GiB used, 619 GiB / 626 GiB avail 2024-09-17T07:25:27.006 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:26 smithi100 ceph-mon[39462]: 2.10 scrub starts 2024-09-17T07:25:27.006 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:26 smithi100 ceph-mon[39462]: 2.10 scrub ok 2024-09-17T07:25:27.006 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:26 smithi100 ceph-mon[39462]: 2.3 scrub starts 2024-09-17T07:25:27.006 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:26 smithi100 ceph-mon[39462]: 2.3 scrub ok 2024-09-17T07:25:27.006 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:26 smithi100 ceph-mon[29092]: 2.14 scrub starts 2024-09-17T07:25:27.006 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:26 smithi100 ceph-mon[29092]: 2.14 scrub ok 2024-09-17T07:25:27.006 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:26 smithi100 ceph-mon[29092]: pgmap v1116: 142 pgs: 142 active+clean; 425 KiB data, 6.5 GiB used, 619 GiB / 626 GiB avail 2024-09-17T07:25:27.007 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:26 smithi100 ceph-mon[29092]: 2.10 scrub starts 2024-09-17T07:25:27.007 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:26 smithi100 ceph-mon[29092]: 2.10 scrub ok 2024-09-17T07:25:27.007 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:26 smithi100 ceph-mon[29092]: 2.3 scrub starts 2024-09-17T07:25:27.007 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:26 smithi100 ceph-mon[29092]: 2.3 scrub ok 2024-09-17T07:25:27.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:26 smithi135 ceph-mon[33238]: 2.14 scrub starts 2024-09-17T07:25:27.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:26 smithi135 ceph-mon[33238]: 2.14 scrub ok 2024-09-17T07:25:27.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:26 smithi135 ceph-mon[33238]: pgmap v1116: 142 pgs: 142 active+clean; 425 KiB data, 6.5 GiB used, 619 GiB / 626 GiB avail 2024-09-17T07:25:27.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:26 smithi135 ceph-mon[33238]: 2.10 scrub starts 2024-09-17T07:25:27.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:26 smithi135 ceph-mon[33238]: 2.10 scrub ok 2024-09-17T07:25:27.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:26 smithi135 ceph-mon[33238]: 2.3 scrub starts 2024-09-17T07:25:27.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:26 smithi135 ceph-mon[33238]: 2.3 scrub ok 2024-09-17T07:25:27.985 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:27 smithi100 ceph-mon[29092]: 2.f deep-scrub starts 2024-09-17T07:25:27.985 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:27 smithi100 ceph-mon[29092]: 2.f deep-scrub ok 2024-09-17T07:25:27.985 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:27 smithi100 ceph-mon[39462]: 2.f deep-scrub starts 2024-09-17T07:25:27.985 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:27 smithi100 ceph-mon[39462]: 2.f deep-scrub ok 2024-09-17T07:25:28.007 INFO:teuthology.orchestra.run.smithi135.stdout:ERROR: (22) Invalid argument 2024-09-17T07:25:28.007 INFO:teuthology.orchestra.run.smithi135.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:25:28.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:27 smithi135 ceph-mon[33238]: 2.f deep-scrub starts 2024-09-17T07:25:28.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:27 smithi135 ceph-mon[33238]: 2.f deep-scrub ok 2024-09-17T07:25:28.569 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-17T07:25:28.768 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:25:29.032 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:28 smithi135 ceph-mon[33238]: pgmap v1117: 142 pgs: 142 active+clean; 425 KiB data, 6.5 GiB used, 619 GiB / 626 GiB avail 2024-09-17T07:25:29.032 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:28 smithi135 ceph-mon[33238]: 5.6 scrub starts 2024-09-17T07:25:29.032 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:28 smithi135 ceph-mon[33238]: 5.6 scrub ok 2024-09-17T07:25:29.045 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:28 smithi100 ceph-mon[29092]: pgmap v1117: 142 pgs: 142 active+clean; 425 KiB data, 6.5 GiB used, 619 GiB / 626 GiB avail 2024-09-17T07:25:29.045 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:28 smithi100 ceph-mon[29092]: 5.6 scrub starts 2024-09-17T07:25:29.045 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:28 smithi100 ceph-mon[29092]: 5.6 scrub ok 2024-09-17T07:25:29.045 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:28 smithi100 ceph-mon[39462]: pgmap v1117: 142 pgs: 142 active+clean; 425 KiB data, 6.5 GiB used, 619 GiB / 626 GiB avail 2024-09-17T07:25:29.046 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:28 smithi100 ceph-mon[39462]: 5.6 scrub starts 2024-09-17T07:25:29.046 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:28 smithi100 ceph-mon[39462]: 5.6 scrub ok 2024-09-17T07:25:29.523 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T07:25:29.526 INFO:teuthology.orchestra.run.smithi100.stderr:dumped all 2024-09-17T07:25:29.778 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:25:29 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:25:29.789 INFO:teuthology.orchestra.run.smithi100.stderr:nodeep-scrub is unset 2024-09-17T07:25:30.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:29 smithi135 ceph-mon[33238]: 5.b scrub starts 2024-09-17T07:25:30.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:29 smithi135 ceph-mon[33238]: 5.b scrub ok 2024-09-17T07:25:30.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:29 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/4240204427' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:25:30.085 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:29 smithi100 ceph-mon[29092]: 5.b scrub starts 2024-09-17T07:25:30.085 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:29 smithi100 ceph-mon[29092]: 5.b scrub ok 2024-09-17T07:25:30.086 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:29 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/4240204427' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:25:30.086 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:29 smithi100 ceph-mon[39462]: 5.b scrub starts 2024-09-17T07:25:30.086 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:29 smithi100 ceph-mon[39462]: 5.b scrub ok 2024-09-17T07:25:30.086 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:29 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/4240204427' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-17T07:25:30.326 INFO:teuthology.orchestra.run.smithi135.stdout:ERROR: (22) Invalid argument 2024-09-17T07:25:30.326 INFO:teuthology.orchestra.run.smithi135.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:25:30.901 INFO:teuthology.orchestra.run.smithi100.stdout:{"pg_ready":true,"pg_map":{"version":1117,"stamp":"2024-09-17T07:25:27.510549+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":435236,"num_objects":196,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":1958,"num_read_kb":1777,"num_write":1133,"num_write_kb":1799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":274,"num_bytes_recovered":1294239,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2573,"ondisk_log_size":2573,"up":426,"acting":426,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":431,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":6839104,"kb_used_data":605368,"kb_used_omap":16,"kb_used_meta":6231471,"kb_avail":649376960,"statfs":{"total":671965249536,"available":664962007040,"internally_reserved":0,"allocated":619896832,"data_stored":610806188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16844,"internal_metadata":6381026868},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-14,"num_read_kb":-23,"num_write":-16,"num_write_kb":-220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":-161009,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.770475"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":99,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:20.087978+0000","last_change":"2024-09-17T07:25:17.976501+0000","last_active":"2024-09-17T07:25:20.087978+0000","last_peered":"2024-09-17T07:25:20.087978+0000","last_clean":"2024-09-17T07:25:20.087978+0000","last_became_active":"2024-09-17T07:22:13.586558+0000","last_became_peered":"2024-09-17T07:22:13.586558+0000","last_unstale":"2024-09-17T07:25:20.087978+0000","last_undegraded":"2024-09-17T07:25:20.087978+0000","last_fullsized":"2024-09-17T07:25:20.087978+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:17.976387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:33.644728+0000","last_clean_scrub_stamp":"2024-09-17T07:25:17.976387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:32.601619+0000","scrub_duration":0.0012687289999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":570,"reported_epoch":511,"state":"active+clean","last_fresh":"2024-09-17T07:25:21.764913+0000","last_change":"2024-09-17T07:23:27.440583+0000","last_active":"2024-09-17T07:25:21.764913+0000","last_peered":"2024-09-17T07:25:21.764913+0000","last_clean":"2024-09-17T07:25:21.764913+0000","last_became_active":"2024-09-17T07:22:13.589527+0000","last_became_peered":"2024-09-17T07:22:13.589527+0000","last_unstale":"2024-09-17T07:25:21.764913+0000","last_undegraded":"2024-09-17T07:25:21.764913+0000","last_fullsized":"2024-09-17T07:25:21.764913+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:23:27.440509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:23:27.440509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001261002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"439'39","reported_seq":494,"reported_epoch":511,"state":"active+clean","last_fresh":"2024-09-17T07:25:21.764406+0000","last_change":"2024-09-17T07:25:10.604396+0000","last_active":"2024-09-17T07:25:21.764406+0000","last_peered":"2024-09-17T07:25:21.764406+0000","last_clean":"2024-09-17T07:25:21.764406+0000","last_became_active":"2024-09-17T07:23:57.480684+0000","last_became_peered":"2024-09-17T07:23:57.480684+0000","last_unstale":"2024-09-17T07:25:21.764406+0000","last_undegraded":"2024-09-17T07:25:21.764406+0000","last_fullsized":"2024-09-17T07:25:21.764406+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'39","last_scrub_stamp":"2024-09-17T07:25:10.604305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:25:10.604305+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:38.118795+0000","scrub_duration":0.0033772899999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":540,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:19.742424+0000","last_change":"2024-09-17T07:25:04.885823+0000","last_active":"2024-09-17T07:25:19.742424+0000","last_peered":"2024-09-17T07:25:19.742424+0000","last_clean":"2024-09-17T07:25:19.742424+0000","last_became_active":"2024-09-17T07:21:15.354561+0000","last_became_peered":"2024-09-17T07:21:15.354561+0000","last_unstale":"2024-09-17T07:25:19.742424+0000","last_undegraded":"2024-09-17T07:25:19.742424+0000","last_fullsized":"2024-09-17T07:25:19.742424+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:04.885700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:25:04.885700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:21.499532+0000","scrub_duration":0.0034589249999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":407,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:24.768019+0000","last_change":"2024-09-17T07:23:57.483306+0000","last_active":"2024-09-17T07:25:24.768019+0000","last_peered":"2024-09-17T07:25:24.768019+0000","last_clean":"2024-09-17T07:25:24.768019+0000","last_became_active":"2024-09-17T07:23:57.480136+0000","last_became_peered":"2024-09-17T07:23:57.480136+0000","last_unstale":"2024-09-17T07:25:24.768019+0000","last_undegraded":"2024-09-17T07:25:24.768019+0000","last_fullsized":"2024-09-17T07:25:24.768019+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:21:13.454404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:17:35.507742+0000","last_clean_scrub_stamp":"2024-09-17T07:21:13.454404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.016029897000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"438'21","reported_seq":588,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:19.742477+0000","last_change":"2024-09-17T07:24:58.793239+0000","last_active":"2024-09-17T07:25:19.742477+0000","last_peered":"2024-09-17T07:25:19.742477+0000","last_clean":"2024-09-17T07:25:19.742477+0000","last_became_active":"2024-09-17T07:21:15.354718+0000","last_became_peered":"2024-09-17T07:21:15.354718+0000","last_unstale":"2024-09-17T07:25:19.742477+0000","last_undegraded":"2024-09-17T07:25:19.742477+0000","last_fullsized":"2024-09-17T07:25:19.742477+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"438'21","last_scrub_stamp":"2024-09-17T07:24:58.793136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:24:58.793136+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:13.595630+0000","scrub_duration":0.0018950810000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":423,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:20.753147+0000","last_change":"2024-09-17T07:23:57.473889+0000","last_active":"2024-09-17T07:25:20.753147+0000","last_peered":"2024-09-17T07:25:20.753147+0000","last_clean":"2024-09-17T07:25:20.753147+0000","last_became_active":"2024-09-17T07:23:57.473620+0000","last_became_peered":"2024-09-17T07:23:57.473620+0000","last_unstale":"2024-09-17T07:25:20.753147+0000","last_undegraded":"2024-09-17T07:25:20.753147+0000","last_fullsized":"2024-09-17T07:25:20.753147+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:13.660077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:20:13.660077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":554,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:19.742611+0000","last_change":"2024-09-17T07:24:18.740114+0000","last_active":"2024-09-17T07:25:19.742611+0000","last_peered":"2024-09-17T07:25:19.742611+0000","last_clean":"2024-09-17T07:25:19.742611+0000","last_became_active":"2024-09-17T07:23:57.482145+0000","last_became_peered":"2024-09-17T07:23:57.482145+0000","last_unstale":"2024-09-17T07:25:19.742611+0000","last_undegraded":"2024-09-17T07:25:19.742611+0000","last_fullsized":"2024-09-17T07:25:19.742611+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:24:18.740035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:24:18.740035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:23.015022+0000","scrub_duration":0.012092344,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":99,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:20.088159+0000","last_change":"2024-09-17T07:23:56.908306+0000","last_active":"2024-09-17T07:25:20.088159+0000","last_peered":"2024-09-17T07:25:20.088159+0000","last_clean":"2024-09-17T07:25:20.088159+0000","last_became_active":"2024-09-17T07:22:13.615512+0000","last_became_peered":"2024-09-17T07:22:13.615512+0000","last_unstale":"2024-09-17T07:25:20.088159+0000","last_undegraded":"2024-09-17T07:25:20.088159+0000","last_fullsized":"2024-09-17T07:25:20.088159+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:23:56.908208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:46.312371+0000","last_clean_scrub_stamp":"2024-09-17T07:23:56.908208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0012071619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"488'11","reported_seq":578,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:19.742626+0000","last_change":"2024-09-17T07:24:26.774186+0000","last_active":"2024-09-17T07:25:19.742626+0000","last_peered":"2024-09-17T07:25:19.742626+0000","last_clean":"2024-09-17T07:25:19.742626+0000","last_became_active":"2024-09-17T07:21:15.349235+0000","last_became_peered":"2024-09-17T07:21:15.349235+0000","last_unstale":"2024-09-17T07:25:19.742626+0000","last_undegraded":"2024-09-17T07:25:19.742626+0000","last_fullsized":"2024-09-17T07:25:19.742626+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"488'11","last_scrub_stamp":"2024-09-17T07:24:26.774097+0000","last_deep_scrub":"488'11","last_deep_scrub_stamp":"2024-09-17T07:24:26.774097+0000","last_clean_scrub_stamp":"2024-09-17T07:24:26.774097+0000","objects_scrubbed":2,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:40.430722+0000","scrub_duration":0.0025260220000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":13,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":88,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:20.087553+0000","last_change":"2024-09-17T07:22:13.624911+0000","last_active":"2024-09-17T07:25:20.087553+0000","last_peered":"2024-09-17T07:25:20.087553+0000","last_clean":"2024-09-17T07:25:20.087553+0000","last_became_active":"2024-09-17T07:22:13.624441+0000","last_became_peered":"2024-09-17T07:22:13.624441+0000","last_unstale":"2024-09-17T07:25:20.087553+0000","last_undegraded":"2024-09-17T07:25:20.087553+0000","last_fullsized":"2024-09-17T07:25:20.087553+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:10.116580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:20:10.116580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00137765,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":505,"reported_epoch":511,"state":"active+clean","last_fresh":"2024-09-17T07:25:22.309428+0000","last_change":"2024-09-17T07:24:24.811636+0000","last_active":"2024-09-17T07:25:22.309428+0000","last_peered":"2024-09-17T07:25:22.309428+0000","last_clean":"2024-09-17T07:25:22.309428+0000","last_became_active":"2024-09-17T07:23:57.489317+0000","last_became_peered":"2024-09-17T07:23:57.489317+0000","last_unstale":"2024-09-17T07:25:22.309428+0000","last_undegraded":"2024-09-17T07:25:22.309428+0000","last_fullsized":"2024-09-17T07:25:22.309428+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:24:24.811522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:24:24.811522+0000","last_clean_scrub_stamp":"2024-09-17T07:24:24.811522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:36.491439+0000","scrub_duration":0.001695238,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":114,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:20.088496+0000","last_change":"2024-09-17T07:24:29.884052+0000","last_active":"2024-09-17T07:25:20.088496+0000","last_peered":"2024-09-17T07:25:20.088496+0000","last_clean":"2024-09-17T07:25:20.088496+0000","last_became_active":"2024-09-17T07:23:57.476413+0000","last_became_peered":"2024-09-17T07:23:57.476413+0000","last_unstale":"2024-09-17T07:25:20.088496+0000","last_undegraded":"2024-09-17T07:25:20.088496+0000","last_fullsized":"2024-09-17T07:25:20.088496+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:24:29.883839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:45.029247+0000","last_clean_scrub_stamp":"2024-09-17T07:24:29.883839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:50.216430+0000","scrub_duration":0.0022700179999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"439'25","reported_seq":605,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:19.742809+0000","last_change":"2024-09-17T07:20:39.070372+0000","last_active":"2024-09-17T07:25:19.742809+0000","last_peered":"2024-09-17T07:25:19.742809+0000","last_clean":"2024-09-17T07:25:19.742809+0000","last_became_active":"2024-09-17T07:09:24.397954+0000","last_became_peered":"2024-09-17T07:09:24.397954+0000","last_unstale":"2024-09-17T07:25:19.742809+0000","last_undegraded":"2024-09-17T07:25:19.742809+0000","last_fullsized":"2024-09-17T07:25:19.742809+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"312'19","last_scrub_stamp":"2024-09-17T07:20:39.070235+0000","last_deep_scrub":"312'19","last_deep_scrub_stamp":"2024-09-17T07:16:44.292911+0000","last_clean_scrub_stamp":"2024-09-17T07:20:39.070235+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0028570560000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":542,"reported_epoch":511,"state":"active+clean","last_fresh":"2024-09-17T07:25:22.309688+0000","last_change":"2024-09-17T07:25:13.136654+0000","last_active":"2024-09-17T07:25:22.309688+0000","last_peered":"2024-09-17T07:25:22.309688+0000","last_clean":"2024-09-17T07:25:22.309688+0000","last_became_active":"2024-09-17T07:23:57.480812+0000","last_became_peered":"2024-09-17T07:23:57.480812+0000","last_unstale":"2024-09-17T07:25:22.309688+0000","last_undegraded":"2024-09-17T07:25:22.309688+0000","last_fullsized":"2024-09-17T07:25:22.309688+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:13.136552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:28.096266+0000","last_clean_scrub_stamp":"2024-09-17T07:25:13.136552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:27.238828+0000","scrub_duration":0.001451686,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":46,"reported_epoch":511,"state":"active+clean","last_fresh":"2024-09-17T07:25:23.534948+0000","last_change":"2024-09-17T07:25:23.534948+0000","last_active":"2024-09-17T07:25:23.534948+0000","last_peered":"2024-09-17T07:25:23.534948+0000","last_clean":"2024-09-17T07:25:23.534948+0000","last_became_active":"2024-09-17T07:24:20.409247+0000","last_became_peered":"2024-09-17T07:24:20.409247+0000","last_unstale":"2024-09-17T07:25:23.534948+0000","last_undegraded":"2024-09-17T07:25:23.534948+0000","last_fullsized":"2024-09-17T07:25:23.534948+0000","mapping_epoch":487,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":488,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:23.534859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:25:23.534859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:45.649153+0000","scrub_duration":0.001563004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.8","version":"212'130","reported_seq":583,"reported_epoch":511,"state":"active+clean","last_fresh":"2024-09-17T07:25:22.309444+0000","last_change":"2024-09-17T07:23:57.479148+0000","last_active":"2024-09-17T07:25:22.309444+0000","last_peered":"2024-09-17T07:25:22.309444+0000","last_clean":"2024-09-17T07:25:22.309444+0000","last_became_active":"2024-09-17T07:23:57.478874+0000","last_became_peered":"2024-09-17T07:23:57.478874+0000","last_unstale":"2024-09-17T07:25:22.309444+0000","last_undegraded":"2024-09-17T07:25:22.309444+0000","last_fullsized":"2024-09-17T07:25:22.309444+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":4,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:20:21.626089+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:14:57.052505+0000","last_clean_scrub_stamp":"2024-09-17T07:20:21.626089+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021141889999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":28,"num_write":16,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":161009,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"439'53","reported_seq":632,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:19.743208+0000","last_change":"2024-09-17T07:25:12.896735+0000","last_active":"2024-09-17T07:25:19.743208+0000","last_peered":"2024-09-17T07:25:19.743208+0000","last_clean":"2024-09-17T07:25:19.743208+0000","last_became_active":"2024-09-17T07:21:15.348413+0000","last_became_peered":"2024-09-17T07:21:15.348413+0000","last_unstale":"2024-09-17T07:25:19.743208+0000","last_undegraded":"2024-09-17T07:25:19.743208+0000","last_fullsized":"2024-09-17T07:25:19.743208+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'53","last_scrub_stamp":"2024-09-17T07:25:12.896608+0000","last_deep_scrub":"95'25","last_deep_scrub_stamp":"2024-09-17T07:04:09.450039+0000","last_clean_scrub_stamp":"2024-09-17T07:25:12.896608+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:18.556302+0000","scrub_duration":0.0026638410000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":61,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":114,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:20.088320+0000","last_change":"2024-09-17T07:25:08.851793+0000","last_active":"2024-09-17T07:25:20.088320+0000","last_peered":"2024-09-17T07:25:20.088320+0000","last_clean":"2024-09-17T07:25:20.088320+0000","last_became_active":"2024-09-17T07:23:57.475614+0000","last_became_peered":"2024-09-17T07:23:57.475614+0000","last_unstale":"2024-09-17T07:25:20.088320+0000","last_undegraded":"2024-09-17T07:25:20.088320+0000","last_fullsized":"2024-09-17T07:25:20.088320+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:08.851702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:25:08.851702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:37.060045+0000","scrub_duration":0.00137217,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":373,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:20.753498+0000","last_change":"2024-09-17T07:24:25.977606+0000","last_active":"2024-09-17T07:25:20.753498+0000","last_peered":"2024-09-17T07:25:20.753498+0000","last_clean":"2024-09-17T07:25:20.753498+0000","last_became_active":"2024-09-17T07:21:15.349538+0000","last_became_peered":"2024-09-17T07:21:15.349538+0000","last_unstale":"2024-09-17T07:25:20.753498+0000","last_undegraded":"2024-09-17T07:25:20.753498+0000","last_fullsized":"2024-09-17T07:25:20.753498+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:24:25.974726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:24:25.974726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:55.688544+0000","scrub_duration":0.014578028999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":393,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:19.743206+0000","last_change":"2024-09-17T07:20:40.117298+0000","last_active":"2024-09-17T07:25:19.743206+0000","last_peered":"2024-09-17T07:25:19.743206+0000","last_clean":"2024-09-17T07:25:19.743206+0000","last_became_active":"2024-09-17T07:09:24.396334+0000","last_became_peered":"2024-09-17T07:09:24.396334+0000","last_unstale":"2024-09-17T07:25:19.743206+0000","last_undegraded":"2024-09-17T07:25:19.743206+0000","last_fullsized":"2024-09-17T07:25:19.743206+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:40.117165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:42.984088+0000","last_clean_scrub_stamp":"2024-09-17T07:20:40.117165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0041399059999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"212'130","reported_seq":758,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:20.087513+0000","last_change":"2024-09-17T07:23:57.476746+0000","last_active":"2024-09-17T07:25:20.087513+0000","last_peered":"2024-09-17T07:25:20.087513+0000","last_clean":"2024-09-17T07:25:20.087513+0000","last_became_active":"2024-09-17T07:23:57.476423+0000","last_became_peered":"2024-09-17T07:23:57.476423+0000","last_unstale":"2024-09-17T07:25:20.087513+0000","last_undegraded":"2024-09-17T07:25:20.087513+0000","last_fullsized":"2024-09-17T07:25:20.087513+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":4,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:21:12.403277+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:14:57.052505+0000","last_clean_scrub_stamp":"2024-09-17T07:21:12.403277+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0071578960000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":28,"num_write":16,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":161009,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"439'35","reported_seq":626,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:19.742948+0000","last_change":"2024-09-17T07:24:29.751049+0000","last_active":"2024-09-17T07:25:19.742948+0000","last_peered":"2024-09-17T07:25:19.742948+0000","last_clean":"2024-09-17T07:25:19.742948+0000","last_became_active":"2024-09-17T07:21:15.347129+0000","last_became_peered":"2024-09-17T07:21:15.347129+0000","last_unstale":"2024-09-17T07:25:19.742948+0000","last_undegraded":"2024-09-17T07:25:19.742948+0000","last_fullsized":"2024-09-17T07:25:19.742948+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'35","last_scrub_stamp":"2024-09-17T07:24:29.750888+0000","last_deep_scrub":"439'35","last_deep_scrub_stamp":"2024-09-17T07:24:29.750888+0000","last_clean_scrub_stamp":"2024-09-17T07:24:29.750888+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:46.618846+0000","scrub_duration":0.0097756200000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":421,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:20.753509+0000","last_change":"2024-09-17T07:23:57.493626+0000","last_active":"2024-09-17T07:25:20.753509+0000","last_peered":"2024-09-17T07:25:20.753509+0000","last_clean":"2024-09-17T07:25:20.753509+0000","last_became_active":"2024-09-17T07:23:57.493354+0000","last_became_peered":"2024-09-17T07:23:57.493354+0000","last_unstale":"2024-09-17T07:25:20.753509+0000","last_undegraded":"2024-09-17T07:25:20.753509+0000","last_fullsized":"2024-09-17T07:25:20.753509+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:11.580871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:20:11.580871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0065215869999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"478'13","reported_seq":158,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:20.783970+0000","last_change":"2024-09-17T07:24:25.818322+0000","last_active":"2024-09-17T07:25:20.783970+0000","last_peered":"2024-09-17T07:25:20.783970+0000","last_clean":"2024-09-17T07:25:20.783970+0000","last_became_active":"2024-09-17T07:23:57.480120+0000","last_became_peered":"2024-09-17T07:23:57.480120+0000","last_unstale":"2024-09-17T07:25:20.783970+0000","last_undegraded":"2024-09-17T07:25:20.783970+0000","last_fullsized":"2024-09-17T07:25:20.783970+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"478'13","last_scrub_stamp":"2024-09-17T07:24:25.815560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:24:25.815560+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:54.513977+0000","scrub_duration":0.0054701139999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":552,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:19.743145+0000","last_change":"2024-09-17T07:23:55.705234+0000","last_active":"2024-09-17T07:25:19.743145+0000","last_peered":"2024-09-17T07:25:19.743145+0000","last_clean":"2024-09-17T07:25:19.743145+0000","last_became_active":"2024-09-17T07:21:15.348799+0000","last_became_peered":"2024-09-17T07:21:15.348799+0000","last_unstale":"2024-09-17T07:25:19.743145+0000","last_undegraded":"2024-09-17T07:25:19.743145+0000","last_fullsized":"2024-09-17T07:25:19.743145+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:23:55.705025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:14:50.554591+0000","last_clean_scrub_stamp":"2024-09-17T07:23:55.705025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:22.420935+0000","scrub_duration":0.0058854049999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"212'130","reported_seq":584,"reported_epoch":511,"state":"active+clean","last_fresh":"2024-09-17T07:25:22.310014+0000","last_change":"2024-09-17T07:23:57.494642+0000","last_active":"2024-09-17T07:25:22.310014+0000","last_peered":"2024-09-17T07:25:22.310014+0000","last_clean":"2024-09-17T07:25:22.310014+0000","last_became_active":"2024-09-17T07:23:57.494014+0000","last_became_peered":"2024-09-17T07:23:57.494014+0000","last_unstale":"2024-09-17T07:25:22.310014+0000","last_undegraded":"2024-09-17T07:25:22.310014+0000","last_fullsized":"2024-09-17T07:25:22.310014+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":4,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:20:33.709901+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:14:57.052505+0000","last_clean_scrub_stamp":"2024-09-17T07:20:33.709901+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021141889999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":27,"num_write":16,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":161009,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"439'39","reported_seq":494,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:20.754087+0000","last_change":"2024-09-17T07:22:13.780063+0000","last_active":"2024-09-17T07:25:20.754087+0000","last_peered":"2024-09-17T07:25:20.754087+0000","last_clean":"2024-09-17T07:25:20.754087+0000","last_became_active":"2024-09-17T07:22:13.589782+0000","last_became_peered":"2024-09-17T07:22:13.589782+0000","last_unstale":"2024-09-17T07:25:20.754087+0000","last_undegraded":"2024-09-17T07:25:20.754087+0000","last_fullsized":"2024-09-17T07:25:20.754087+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'27","last_scrub_stamp":"2024-09-17T07:20:13.130718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:20:13.130718+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0081162040000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":420,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:20.753245+0000","last_change":"2024-09-17T07:22:13.592519+0000","last_active":"2024-09-17T07:25:20.753245+0000","last_peered":"2024-09-17T07:25:20.753245+0000","last_clean":"2024-09-17T07:25:20.753245+0000","last_became_active":"2024-09-17T07:22:13.591089+0000","last_became_peered":"2024-09-17T07:22:13.591089+0000","last_unstale":"2024-09-17T07:25:20.753245+0000","last_undegraded":"2024-09-17T07:25:20.753245+0000","last_fullsized":"2024-09-17T07:25:20.753245+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:14.001258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:20:14.001258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034116369999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"478'10","reported_seq":850,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:20.783780+0000","last_change":"2024-09-17T07:25:16.842178+0000","last_active":"2024-09-17T07:25:20.783780+0000","last_peered":"2024-09-17T07:25:20.783780+0000","last_clean":"2024-09-17T07:25:20.783780+0000","last_became_active":"2024-09-17T07:23:57.473239+0000","last_became_peered":"2024-09-17T07:23:57.473239+0000","last_unstale":"2024-09-17T07:25:20.783780+0000","last_undegraded":"2024-09-17T07:25:20.783780+0000","last_fullsized":"2024-09-17T07:25:20.783780+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"478'10","last_scrub_stamp":"2024-09-17T07:25:16.842081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:25:16.842081+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:28.575493+0000","scrub_duration":0.001742511,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":496,"reported_epoch":511,"state":"active+clean","last_fresh":"2024-09-17T07:25:22.311032+0000","last_change":"2024-09-17T07:24:29.971120+0000","last_active":"2024-09-17T07:25:22.311032+0000","last_peered":"2024-09-17T07:25:22.311032+0000","last_clean":"2024-09-17T07:25:22.311032+0000","last_became_active":"2024-09-17T07:23:57.522025+0000","last_became_peered":"2024-09-17T07:23:57.522025+0000","last_unstale":"2024-09-17T07:25:22.311032+0000","last_undegraded":"2024-09-17T07:25:22.311032+0000","last_fullsized":"2024-09-17T07:25:22.311032+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:24:29.971009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:24:29.971009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:39.187110+0000","scrub_duration":0.002296789,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"488'32","reported_seq":603,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:19.743378+0000","last_change":"2024-09-17T07:24:27.786116+0000","last_active":"2024-09-17T07:25:19.743378+0000","last_peered":"2024-09-17T07:25:19.743378+0000","last_clean":"2024-09-17T07:25:19.743378+0000","last_became_active":"2024-09-17T07:21:15.347662+0000","last_became_peered":"2024-09-17T07:21:15.347662+0000","last_unstale":"2024-09-17T07:25:19.743378+0000","last_undegraded":"2024-09-17T07:25:19.743378+0000","last_fullsized":"2024-09-17T07:25:19.743378+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"488'32","last_scrub_stamp":"2024-09-17T07:24:27.785661+0000","last_deep_scrub":"488'32","last_deep_scrub_stamp":"2024-09-17T07:24:27.785661+0000","last_clean_scrub_stamp":"2024-09-17T07:24:27.785661+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:32.181225+0000","scrub_duration":0.021517458999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":88,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:20.087428+0000","last_change":"2024-09-17T07:22:13.711817+0000","last_active":"2024-09-17T07:25:20.087428+0000","last_peered":"2024-09-17T07:25:20.087428+0000","last_clean":"2024-09-17T07:25:20.087428+0000","last_became_active":"2024-09-17T07:22:13.709873+0000","last_became_peered":"2024-09-17T07:22:13.709873+0000","last_unstale":"2024-09-17T07:25:20.087428+0000","last_undegraded":"2024-09-17T07:25:20.087428+0000","last_fullsized":"2024-09-17T07:25:20.087428+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:12.442421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:20:12.442421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026862600000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":518,"reported_epoch":511,"state":"active+clean","last_fresh":"2024-09-17T07:25:22.311113+0000","last_change":"2024-09-17T07:25:11.133451+0000","last_active":"2024-09-17T07:25:22.311113+0000","last_peered":"2024-09-17T07:25:22.311113+0000","last_clean":"2024-09-17T07:25:22.311113+0000","last_became_active":"2024-09-17T07:23:57.478625+0000","last_became_peered":"2024-09-17T07:23:57.478625+0000","last_unstale":"2024-09-17T07:25:22.311113+0000","last_undegraded":"2024-09-17T07:25:22.311113+0000","last_fullsized":"2024-09-17T07:25:22.311113+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:11.133339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:25:11.133339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:35.425563+0000","scrub_duration":0.0016371350000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":454,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:19.868724+0000","last_change":"2024-09-17T07:24:26.583863+0000","last_active":"2024-09-17T07:25:19.868724+0000","last_peered":"2024-09-17T07:25:19.868724+0000","last_clean":"2024-09-17T07:25:19.868724+0000","last_became_active":"2024-09-17T07:23:57.486204+0000","last_became_peered":"2024-09-17T07:23:57.486204+0000","last_unstale":"2024-09-17T07:25:19.868724+0000","last_undegraded":"2024-09-17T07:25:19.868724+0000","last_fullsized":"2024-09-17T07:25:19.868724+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:24:26.583745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:13:22.187127+0000","last_clean_scrub_stamp":"2024-09-17T07:24:26.583745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:46.250777+0000","scrub_duration":0.0024301169999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"212'130","reported_seq":621,"reported_epoch":511,"state":"active+clean","last_fresh":"2024-09-17T07:25:22.310730+0000","last_change":"2024-09-17T07:23:57.484193+0000","last_active":"2024-09-17T07:25:22.310730+0000","last_peered":"2024-09-17T07:25:22.310730+0000","last_clean":"2024-09-17T07:25:22.310730+0000","last_became_active":"2024-09-17T07:23:57.478159+0000","last_became_peered":"2024-09-17T07:23:57.478159+0000","last_unstale":"2024-09-17T07:25:22.310730+0000","last_undegraded":"2024-09-17T07:25:22.310730+0000","last_fullsized":"2024-09-17T07:25:22.310730+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":3,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:20:40.628770+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:14:57.052505+0000","last_clean_scrub_stamp":"2024-09-17T07:20:40.628770+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021141889999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":28,"num_write":16,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":161010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1","version":"439'42","reported_seq":617,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:19.742528+0000","last_change":"2024-09-17T07:23:57.482189+0000","last_active":"2024-09-17T07:25:19.742528+0000","last_peered":"2024-09-17T07:25:19.742528+0000","last_clean":"2024-09-17T07:25:19.742528+0000","last_became_active":"2024-09-17T07:23:57.481780+0000","last_became_peered":"2024-09-17T07:23:57.481780+0000","last_unstale":"2024-09-17T07:25:19.742528+0000","last_undegraded":"2024-09-17T07:25:19.742528+0000","last_fullsized":"2024-09-17T07:25:19.742528+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'28","last_scrub_stamp":"2024-09-17T07:20:28.029279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:20:28.029279+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0079911889999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":63,"num_read_kb":56,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":622,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:19.869243+0000","last_change":"2024-09-17T07:25:07.568509+0000","last_active":"2024-09-17T07:25:19.869243+0000","last_peered":"2024-09-17T07:25:19.869243+0000","last_clean":"2024-09-17T07:25:19.869243+0000","last_became_active":"2024-09-17T07:22:13.582096+0000","last_became_peered":"2024-09-17T07:22:13.582096+0000","last_unstale":"2024-09-17T07:25:19.869243+0000","last_undegraded":"2024-09-17T07:25:19.869243+0000","last_fullsized":"2024-09-17T07:25:19.869243+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:07.568411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:09:23.181028+0000","last_clean_scrub_stamp":"2024-09-17T07:25:07.568411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:31.713937+0000","scrub_duration":0.0014008320000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"414'11","reported_seq":399,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:20.783589+0000","last_change":"2024-09-17T07:21:15.356155+0000","last_active":"2024-09-17T07:25:20.783589+0000","last_peered":"2024-09-17T07:25:20.783589+0000","last_clean":"2024-09-17T07:25:20.783589+0000","last_became_active":"2024-09-17T07:21:15.355651+0000","last_became_peered":"2024-09-17T07:21:15.355651+0000","last_unstale":"2024-09-17T07:25:20.783589+0000","last_undegraded":"2024-09-17T07:25:20.783589+0000","last_fullsized":"2024-09-17T07:25:20.783589+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'9","last_scrub_stamp":"2024-09-17T07:15:55.813882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:15:55.813882+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":100,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:20.087869+0000","last_change":"2024-09-17T07:25:00.872918+0000","last_active":"2024-09-17T07:25:20.087869+0000","last_peered":"2024-09-17T07:25:20.087869+0000","last_clean":"2024-09-17T07:25:20.087869+0000","last_became_active":"2024-09-17T07:22:13.568656+0000","last_became_peered":"2024-09-17T07:22:13.568656+0000","last_unstale":"2024-09-17T07:25:20.087869+0000","last_undegraded":"2024-09-17T07:25:20.087869+0000","last_fullsized":"2024-09-17T07:25:20.087869+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:00.872840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:10:33.727472+0000","last_clean_scrub_stamp":"2024-09-17T07:25:00.872840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:27.841417+0000","scrub_duration":0.001230515,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"212'130","reported_seq":606,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:19.742800+0000","last_change":"2024-09-17T07:20:35.080459+0000","last_active":"2024-09-17T07:25:19.742800+0000","last_peered":"2024-09-17T07:25:19.742800+0000","last_clean":"2024-09-17T07:25:19.742800+0000","last_became_active":"2024-09-17T07:19:35.420337+0000","last_became_peered":"2024-09-17T07:19:35.420337+0000","last_unstale":"2024-09-17T07:25:19.742800+0000","last_undegraded":"2024-09-17T07:25:19.742800+0000","last_fullsized":"2024-09-17T07:25:19.742800+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":3,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:20:35.080257+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:20:35.080257+0000","last_clean_scrub_stamp":"2024-09-17T07:20:35.080257+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0030477730000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"439'42","reported_seq":135,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:20.087277+0000","last_change":"2024-09-17T07:25:16.935826+0000","last_active":"2024-09-17T07:25:20.087277+0000","last_peered":"2024-09-17T07:25:20.087277+0000","last_clean":"2024-09-17T07:25:20.087277+0000","last_became_active":"2024-09-17T07:22:13.711444+0000","last_became_peered":"2024-09-17T07:22:13.711444+0000","last_unstale":"2024-09-17T07:25:20.087277+0000","last_undegraded":"2024-09-17T07:25:20.087277+0000","last_fullsized":"2024-09-17T07:25:20.087277+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'42","last_scrub_stamp":"2024-09-17T07:25:16.935731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:25:16.935731+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:23.547655+0000","scrub_duration":0.0023825790000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":68,"num_read_kb":56,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":427,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:20.753658+0000","last_change":"2024-09-17T07:25:09.880351+0000","last_active":"2024-09-17T07:25:20.753658+0000","last_peered":"2024-09-17T07:25:20.753658+0000","last_clean":"2024-09-17T07:25:20.753658+0000","last_became_active":"2024-09-17T07:21:15.347867+0000","last_became_peered":"2024-09-17T07:21:15.347867+0000","last_unstale":"2024-09-17T07:25:20.753658+0000","last_undegraded":"2024-09-17T07:25:20.753658+0000","last_fullsized":"2024-09-17T07:25:20.753658+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:09.880264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:56.848689+0000","last_clean_scrub_stamp":"2024-09-17T07:25:09.880264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:18.052765+0000","scrub_duration":0.0014177599999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":575,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:19.742375+0000","last_change":"2024-09-17T07:23:27.713373+0000","last_active":"2024-09-17T07:25:19.742375+0000","last_peered":"2024-09-17T07:25:19.742375+0000","last_clean":"2024-09-17T07:25:19.742375+0000","last_became_active":"2024-09-17T07:22:13.566916+0000","last_became_peered":"2024-09-17T07:22:13.566916+0000","last_unstale":"2024-09-17T07:25:19.742375+0000","last_undegraded":"2024-09-17T07:25:19.742375+0000","last_fullsized":"2024-09-17T07:25:19.742375+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:23:27.713294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:23:27.713294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0011838949999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":555,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:19.742776+0000","last_change":"2024-09-17T07:25:18.894993+0000","last_active":"2024-09-17T07:25:19.742776+0000","last_peered":"2024-09-17T07:25:19.742776+0000","last_clean":"2024-09-17T07:25:19.742776+0000","last_became_active":"2024-09-17T07:09:24.398659+0000","last_became_peered":"2024-09-17T07:09:24.398659+0000","last_unstale":"2024-09-17T07:25:19.742776+0000","last_undegraded":"2024-09-17T07:25:19.742776+0000","last_fullsized":"2024-09-17T07:25:19.742776+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:18.894878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:25:18.894878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:25.962648+0000","scrub_duration":0.001379328,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"212'130","reported_seq":625,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:19.868985+0000","last_change":"2024-09-17T07:25:12.680354+0000","last_active":"2024-09-17T07:25:19.868985+0000","last_peered":"2024-09-17T07:25:19.868985+0000","last_clean":"2024-09-17T07:25:19.868985+0000","last_became_active":"2024-09-17T07:19:35.428161+0000","last_became_peered":"2024-09-17T07:19:35.428161+0000","last_unstale":"2024-09-17T07:25:19.868985+0000","last_undegraded":"2024-09-17T07:25:19.868985+0000","last_fullsized":"2024-09-17T07:25:19.868985+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":3,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:25:12.680248+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:14:57.052505+0000","last_clean_scrub_stamp":"2024-09-17T07:25:12.680248+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:41.650768+0000","scrub_duration":0.001644413,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"439'39","reported_seq":697,"reported_epoch":511,"state":"active+clean","last_fresh":"2024-09-17T07:25:21.764894+0000","last_change":"2024-09-17T07:25:20.597190+0000","last_active":"2024-09-17T07:25:21.764894+0000","last_peered":"2024-09-17T07:25:21.764894+0000","last_clean":"2024-09-17T07:25:21.764894+0000","last_became_active":"2024-09-17T07:23:57.497347+0000","last_became_peered":"2024-09-17T07:23:57.497347+0000","last_unstale":"2024-09-17T07:25:21.764894+0000","last_undegraded":"2024-09-17T07:25:21.764894+0000","last_fullsized":"2024-09-17T07:25:21.764894+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'39","last_scrub_stamp":"2024-09-17T07:25:20.597058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:25:20.597058+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:23.485116+0000","scrub_duration":0.0029701250000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":99,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:20.088010+0000","last_change":"2024-09-17T07:25:07.874505+0000","last_active":"2024-09-17T07:25:20.088010+0000","last_peered":"2024-09-17T07:25:20.088010+0000","last_clean":"2024-09-17T07:25:20.088010+0000","last_became_active":"2024-09-17T07:22:13.624802+0000","last_became_peered":"2024-09-17T07:22:13.624802+0000","last_unstale":"2024-09-17T07:25:20.088010+0000","last_undegraded":"2024-09-17T07:25:20.088010+0000","last_fullsized":"2024-09-17T07:25:20.088010+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:07.874421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:47.631387+0000","last_clean_scrub_stamp":"2024-09-17T07:25:07.874421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:23.997699+0000","scrub_duration":0.0012562529999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":583,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:19.867943+0000","last_change":"2024-09-17T07:25:05.604387+0000","last_active":"2024-09-17T07:25:19.867943+0000","last_peered":"2024-09-17T07:25:19.867943+0000","last_clean":"2024-09-17T07:25:19.867943+0000","last_became_active":"2024-09-17T07:23:57.485512+0000","last_became_peered":"2024-09-17T07:23:57.485512+0000","last_unstale":"2024-09-17T07:25:19.867943+0000","last_undegraded":"2024-09-17T07:25:19.867943+0000","last_fullsized":"2024-09-17T07:25:19.867943+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:05.604241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:11:17.583712+0000","last_clean_scrub_stamp":"2024-09-17T07:25:05.604241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:12.778226+0000","scrub_duration":0.0020187059999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":383,"reported_epoch":511,"state":"active+clean","last_fresh":"2024-09-17T07:25:21.761242+0000","last_change":"2024-09-17T07:24:19.018167+0000","last_active":"2024-09-17T07:25:21.761242+0000","last_peered":"2024-09-17T07:25:21.761242+0000","last_clean":"2024-09-17T07:25:21.761242+0000","last_became_active":"2024-09-17T07:21:15.351524+0000","last_became_peered":"2024-09-17T07:21:15.351524+0000","last_unstale":"2024-09-17T07:25:21.761242+0000","last_undegraded":"2024-09-17T07:25:21.761242+0000","last_fullsized":"2024-09-17T07:25:21.761242+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:24:19.018060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:24:19.018060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:31.477457+0000","scrub_duration":0.0014215090000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"212'130","reported_seq":690,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:20.559655+0000","last_change":"2024-09-17T07:25:20.559655+0000","last_active":"2024-09-17T07:25:20.559655+0000","last_peered":"2024-09-17T07:25:20.559655+0000","last_clean":"2024-09-17T07:25:20.559655+0000","last_became_active":"2024-09-17T07:23:57.486189+0000","last_became_peered":"2024-09-17T07:23:57.486189+0000","last_unstale":"2024-09-17T07:25:20.559655+0000","last_undegraded":"2024-09-17T07:25:20.559655+0000","last_fullsized":"2024-09-17T07:25:20.559655+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":3,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:25:20.559569+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:14:57.052505+0000","last_clean_scrub_stamp":"2024-09-17T07:25:20.559569+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:40.068013+0000","scrub_duration":0.0013623260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"439'35","reported_seq":464,"reported_epoch":511,"state":"active+clean","last_fresh":"2024-09-17T07:25:21.761534+0000","last_change":"2024-09-17T07:24:31.049397+0000","last_active":"2024-09-17T07:25:21.761534+0000","last_peered":"2024-09-17T07:25:21.761534+0000","last_clean":"2024-09-17T07:25:21.761534+0000","last_became_active":"2024-09-17T07:23:57.478340+0000","last_became_peered":"2024-09-17T07:23:57.478340+0000","last_unstale":"2024-09-17T07:25:21.761534+0000","last_undegraded":"2024-09-17T07:25:21.761534+0000","last_fullsized":"2024-09-17T07:25:21.761534+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'35","last_scrub_stamp":"2024-09-17T07:24:31.049269+0000","last_deep_scrub":"110'15","last_deep_scrub_stamp":"2024-09-17T07:04:57.091400+0000","last_clean_scrub_stamp":"2024-09-17T07:24:31.049269+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:39.791093+0000","scrub_duration":0.01178889,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":397,"reported_epoch":511,"state":"active+clean","last_fresh":"2024-09-17T07:25:21.762059+0000","last_change":"2024-09-17T07:22:13.592639+0000","last_active":"2024-09-17T07:25:21.762059+0000","last_peered":"2024-09-17T07:25:21.762059+0000","last_clean":"2024-09-17T07:25:21.762059+0000","last_became_active":"2024-09-17T07:22:13.591372+0000","last_became_peered":"2024-09-17T07:22:13.591372+0000","last_unstale":"2024-09-17T07:25:21.762059+0000","last_undegraded":"2024-09-17T07:25:21.762059+0000","last_fullsized":"2024-09-17T07:25:21.762059+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:24.437342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:25.147878+0000","last_clean_scrub_stamp":"2024-09-17T07:20:24.437342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0024354329999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"478'11","reported_seq":852,"reported_epoch":511,"state":"active+clean","last_fresh":"2024-09-17T07:25:21.764588+0000","last_change":"2024-09-17T07:23:57.497400+0000","last_active":"2024-09-17T07:25:21.764588+0000","last_peered":"2024-09-17T07:25:21.764588+0000","last_clean":"2024-09-17T07:25:21.764588+0000","last_became_active":"2024-09-17T07:23:57.496882+0000","last_became_peered":"2024-09-17T07:23:57.496882+0000","last_unstale":"2024-09-17T07:25:21.764588+0000","last_undegraded":"2024-09-17T07:25:21.764588+0000","last_fullsized":"2024-09-17T07:25:21.764588+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'9","last_scrub_stamp":"2024-09-17T07:21:14.471255+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-09-17T07:15:53.512777+0000","last_clean_scrub_stamp":"2024-09-17T07:21:14.471255+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032915969999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"268'2","reported_seq":576,"reported_epoch":511,"state":"active+clean","last_fresh":"2024-09-17T07:25:21.764345+0000","last_change":"2024-09-17T07:25:08.624593+0000","last_active":"2024-09-17T07:25:21.764345+0000","last_peered":"2024-09-17T07:25:21.764345+0000","last_clean":"2024-09-17T07:25:21.764345+0000","last_became_active":"2024-09-17T07:23:57.479564+0000","last_became_peered":"2024-09-17T07:23:57.479564+0000","last_unstale":"2024-09-17T07:25:21.764345+0000","last_undegraded":"2024-09-17T07:25:21.764345+0000","last_fullsized":"2024-09-17T07:25:21.764345+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'2","last_scrub_stamp":"2024-09-17T07:25:08.624464+0000","last_deep_scrub":"268'2","last_deep_scrub_stamp":"2024-09-17T07:14:03.609131+0000","last_clean_scrub_stamp":"2024-09-17T07:25:08.624464+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:10.264762+0000","scrub_duration":0.0032006410000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"212'130","reported_seq":582,"reported_epoch":511,"state":"active+clean","last_fresh":"2024-09-17T07:25:22.309611+0000","last_change":"2024-09-17T07:23:57.489780+0000","last_active":"2024-09-17T07:25:22.309611+0000","last_peered":"2024-09-17T07:25:22.309611+0000","last_clean":"2024-09-17T07:25:22.309611+0000","last_became_active":"2024-09-17T07:23:57.489445+0000","last_became_peered":"2024-09-17T07:23:57.489445+0000","last_unstale":"2024-09-17T07:25:22.309611+0000","last_undegraded":"2024-09-17T07:25:22.309611+0000","last_fullsized":"2024-09-17T07:25:22.309611+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":4,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:20:22.674350+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:14:57.052505+0000","last_clean_scrub_stamp":"2024-09-17T07:20:22.674350+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021141889999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":28,"num_write":17,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":161010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"439'55","reported_seq":719,"reported_epoch":511,"state":"active+clean","last_fresh":"2024-09-17T07:25:21.762166+0000","last_change":"2024-09-17T07:24:28.997089+0000","last_active":"2024-09-17T07:25:21.762166+0000","last_peered":"2024-09-17T07:25:21.762166+0000","last_clean":"2024-09-17T07:25:21.762166+0000","last_became_active":"2024-09-17T07:21:15.352461+0000","last_became_peered":"2024-09-17T07:21:15.352461+0000","last_unstale":"2024-09-17T07:25:21.762166+0000","last_undegraded":"2024-09-17T07:25:21.762166+0000","last_fullsized":"2024-09-17T07:25:21.762166+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'55","last_scrub_stamp":"2024-09-17T07:24:28.996914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:24:28.996914+0000","objects_scrubbed":10,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:48.984638+0000","scrub_duration":0.0070140810000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":83,"num_read_kb":68,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":503,"reported_epoch":511,"state":"active+clean","last_fresh":"2024-09-17T07:25:22.309922+0000","last_change":"2024-09-17T07:24:18.694832+0000","last_active":"2024-09-17T07:25:22.309922+0000","last_peered":"2024-09-17T07:25:22.309922+0000","last_clean":"2024-09-17T07:25:22.309922+0000","last_became_active":"2024-09-17T07:23:57.493640+0000","last_became_peered":"2024-09-17T07:23:57.493640+0000","last_unstale":"2024-09-17T07:25:22.309922+0000","last_undegraded":"2024-09-17T07:25:22.309922+0000","last_fullsized":"2024-09-17T07:25:22.309922+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:24:18.694743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:24:18.694743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:30.108218+0000","scrub_duration":0.001126408,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":553,"reported_epoch":511,"state":"active+clean","last_fresh":"2024-09-17T07:25:21.764685+0000","last_change":"2024-09-17T07:24:57.661571+0000","last_active":"2024-09-17T07:25:21.764685+0000","last_peered":"2024-09-17T07:25:21.764685+0000","last_clean":"2024-09-17T07:25:21.764685+0000","last_became_active":"2024-09-17T07:21:15.347069+0000","last_became_peered":"2024-09-17T07:21:15.347069+0000","last_unstale":"2024-09-17T07:25:21.764685+0000","last_undegraded":"2024-09-17T07:25:21.764685+0000","last_fullsized":"2024-09-17T07:25:21.764685+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:24:57.661438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:24:57.661438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:13.489925+0000","scrub_duration":0.0017509139999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":452,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:19.743162+0000","last_change":"2024-09-17T07:25:09.565076+0000","last_active":"2024-09-17T07:25:19.743162+0000","last_peered":"2024-09-17T07:25:19.743162+0000","last_clean":"2024-09-17T07:25:19.743162+0000","last_became_active":"2024-09-17T07:25:09.564606+0000","last_became_peered":"2024-09-17T07:25:09.564606+0000","last_unstale":"2024-09-17T07:25:19.743162+0000","last_undegraded":"2024-09-17T07:25:19.743162+0000","last_fullsized":"2024-09-17T07:25:19.743162+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:20.067928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:13:53.406661+0000","last_clean_scrub_stamp":"2024-09-17T07:20:20.067928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0042634170000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.2","version":"212'130","reported_seq":600,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:20.087714+0000","last_change":"2024-09-17T07:22:16.360792+0000","last_active":"2024-09-17T07:25:20.087714+0000","last_peered":"2024-09-17T07:25:20.087714+0000","last_clean":"2024-09-17T07:25:20.087714+0000","last_became_active":"2024-09-17T07:22:16.360258+0000","last_became_peered":"2024-09-17T07:22:16.360258+0000","last_unstale":"2024-09-17T07:25:20.087714+0000","last_undegraded":"2024-09-17T07:25:20.087714+0000","last_fullsized":"2024-09-17T07:25:20.087714+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":4,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:19:19.991044+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:14:57.052505+0000","last_clean_scrub_stamp":"2024-09-17T07:19:19.991044+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021141889999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":28,"num_write":17,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":161010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"439'21","reported_seq":439,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:20.754057+0000","last_change":"2024-09-17T07:24:20.021085+0000","last_active":"2024-09-17T07:25:20.754057+0000","last_peered":"2024-09-17T07:25:20.754057+0000","last_clean":"2024-09-17T07:25:20.754057+0000","last_became_active":"2024-09-17T07:23:57.492747+0000","last_became_peered":"2024-09-17T07:23:57.492747+0000","last_unstale":"2024-09-17T07:25:20.754057+0000","last_undegraded":"2024-09-17T07:25:20.754057+0000","last_fullsized":"2024-09-17T07:25:20.754057+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'21","last_scrub_stamp":"2024-09-17T07:24:20.021003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:24:20.021003+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:22.745059+0000","scrub_duration":0.0022976009999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":30,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"76'5","reported_seq":1465,"reported_epoch":511,"state":"active+clean","last_fresh":"2024-09-17T07:25:21.881360+0000","last_change":"2024-09-17T07:25:05.896344+0000","last_active":"2024-09-17T07:25:21.881360+0000","last_peered":"2024-09-17T07:25:21.881360+0000","last_clean":"2024-09-17T07:25:21.881360+0000","last_became_active":"2024-09-17T07:21:15.357178+0000","last_became_peered":"2024-09-17T07:21:15.357178+0000","last_unstale":"2024-09-17T07:25:21.881360+0000","last_undegraded":"2024-09-17T07:25:21.881360+0000","last_fullsized":"2024-09-17T07:25:21.881360+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'5","last_scrub_stamp":"2024-09-17T07:25:05.896257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:25:05.896257+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:15.630574+0000","scrub_duration":0.0018339840000000001,"objects_trimmed":0,"snaptrim_duration":0.00033739799999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":254,"num_read_kb":249,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":397,"reported_epoch":511,"state":"active+clean","last_fresh":"2024-09-17T07:25:21.764244+0000","last_change":"2024-09-17T07:25:01.641881+0000","last_active":"2024-09-17T07:25:21.764244+0000","last_peered":"2024-09-17T07:25:21.764244+0000","last_clean":"2024-09-17T07:25:21.764244+0000","last_became_active":"2024-09-17T07:21:15.346287+0000","last_became_peered":"2024-09-17T07:21:15.346287+0000","last_unstale":"2024-09-17T07:25:21.764244+0000","last_undegraded":"2024-09-17T07:25:21.764244+0000","last_fullsized":"2024-09-17T07:25:21.764244+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:01.641756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:23.833170+0000","last_clean_scrub_stamp":"2024-09-17T07:25:01.641756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:18.315474+0000","scrub_duration":0.0031250689999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":615,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:19.742430+0000","last_change":"2024-09-17T07:24:18.345634+0000","last_active":"2024-09-17T07:25:19.742430+0000","last_peered":"2024-09-17T07:25:19.742430+0000","last_clean":"2024-09-17T07:25:19.742430+0000","last_became_active":"2024-09-17T07:23:57.481485+0000","last_became_peered":"2024-09-17T07:23:57.481485+0000","last_unstale":"2024-09-17T07:25:19.742430+0000","last_undegraded":"2024-09-17T07:25:19.742430+0000","last_fullsized":"2024-09-17T07:25:19.742430+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:24:18.345550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:18:42.051667+0000","last_clean_scrub_stamp":"2024-09-17T07:24:18.345550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:26.341677+0000","scrub_duration":0.0023155189999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":99,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:20.087929+0000","last_change":"2024-09-17T07:24:30.923133+0000","last_active":"2024-09-17T07:25:20.087929+0000","last_peered":"2024-09-17T07:25:20.087929+0000","last_clean":"2024-09-17T07:25:20.087929+0000","last_became_active":"2024-09-17T07:22:13.628359+0000","last_became_peered":"2024-09-17T07:22:13.628359+0000","last_unstale":"2024-09-17T07:25:20.087929+0000","last_undegraded":"2024-09-17T07:25:20.087929+0000","last_fullsized":"2024-09-17T07:25:20.087929+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:24:30.922689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:02.062618+0000","last_clean_scrub_stamp":"2024-09-17T07:24:30.922689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:36.609751+0000","scrub_duration":0.0066080289999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"212'130","reported_seq":624,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:20.753946+0000","last_change":"2024-09-17T07:21:18.358222+0000","last_active":"2024-09-17T07:25:20.753946+0000","last_peered":"2024-09-17T07:25:20.753946+0000","last_clean":"2024-09-17T07:25:20.753946+0000","last_became_active":"2024-09-17T07:21:18.357874+0000","last_became_peered":"2024-09-17T07:21:18.357874+0000","last_unstale":"2024-09-17T07:25:20.753946+0000","last_undegraded":"2024-09-17T07:25:20.753946+0000","last_fullsized":"2024-09-17T07:25:20.753946+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":3,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:20:34.113602+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:20:34.113602+0000","last_clean_scrub_stamp":"2024-09-17T07:20:34.113602+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021141889999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":28,"num_write":17,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":161010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"439'42","reported_seq":647,"reported_epoch":511,"state":"active+clean","last_fresh":"2024-09-17T07:25:21.764853+0000","last_change":"2024-09-17T07:25:05.607011+0000","last_active":"2024-09-17T07:25:21.764853+0000","last_peered":"2024-09-17T07:25:21.764853+0000","last_clean":"2024-09-17T07:25:21.764853+0000","last_became_active":"2024-09-17T07:23:57.480190+0000","last_became_peered":"2024-09-17T07:23:57.480190+0000","last_unstale":"2024-09-17T07:25:21.764853+0000","last_undegraded":"2024-09-17T07:25:21.764853+0000","last_fullsized":"2024-09-17T07:25:21.764853+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'42","last_scrub_stamp":"2024-09-17T07:25:05.606929+0000","last_deep_scrub":"268'30","last_deep_scrub_stamp":"2024-09-17T07:14:00.599656+0000","last_clean_scrub_stamp":"2024-09-17T07:25:05.606929+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:28.810784+0000","scrub_duration":0.0040141389999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"70'1","reported_seq":586,"reported_epoch":511,"state":"active+clean","last_fresh":"2024-09-17T07:25:21.764156+0000","last_change":"2024-09-17T07:25:14.651563+0000","last_active":"2024-09-17T07:25:21.764156+0000","last_peered":"2024-09-17T07:25:21.764156+0000","last_clean":"2024-09-17T07:25:21.764156+0000","last_became_active":"2024-09-17T07:23:57.480048+0000","last_became_peered":"2024-09-17T07:23:57.480048+0000","last_unstale":"2024-09-17T07:25:21.764156+0000","last_undegraded":"2024-09-17T07:25:21.764156+0000","last_fullsized":"2024-09-17T07:25:21.764156+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-17T07:25:14.651458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:25:14.651458+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:14.704935+0000","scrub_duration":0.0020687610000000001,"objects_trimmed":0,"snaptrim_duration":0.0032298629999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":88,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:20.087282+0000","last_change":"2024-09-17T07:22:13.736269+0000","last_active":"2024-09-17T07:25:20.087282+0000","last_peered":"2024-09-17T07:25:20.087282+0000","last_clean":"2024-09-17T07:25:20.087282+0000","last_became_active":"2024-09-17T07:22:13.710211+0000","last_became_peered":"2024-09-17T07:22:13.710211+0000","last_unstale":"2024-09-17T07:25:20.087282+0000","last_undegraded":"2024-09-17T07:25:20.087282+0000","last_fullsized":"2024-09-17T07:25:20.087282+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:34.449627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:55.321202+0000","last_clean_scrub_stamp":"2024-09-17T07:20:34.449627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0044019280000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":498,"reported_epoch":511,"state":"active+clean","last_fresh":"2024-09-17T07:25:22.311338+0000","last_change":"2024-09-17T07:24:57.916305+0000","last_active":"2024-09-17T07:25:22.311338+0000","last_peered":"2024-09-17T07:25:22.311338+0000","last_clean":"2024-09-17T07:25:22.311338+0000","last_became_active":"2024-09-17T07:23:57.480455+0000","last_became_peered":"2024-09-17T07:23:57.480455+0000","last_unstale":"2024-09-17T07:25:22.311338+0000","last_undegraded":"2024-09-17T07:25:22.311338+0000","last_fullsized":"2024-09-17T07:25:22.311338+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:24:57.916204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:24:57.916204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:22.807948+0000","scrub_duration":0.001685541,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":424,"reported_epoch":511,"state":"active+clean","last_fresh":"2024-09-17T07:25:21.762212+0000","last_change":"2024-09-17T07:23:57.473704+0000","last_active":"2024-09-17T07:25:21.762212+0000","last_peered":"2024-09-17T07:25:21.762212+0000","last_clean":"2024-09-17T07:25:21.762212+0000","last_became_active":"2024-09-17T07:23:57.473313+0000","last_became_peered":"2024-09-17T07:23:57.473313+0000","last_unstale":"2024-09-17T07:25:21.762212+0000","last_undegraded":"2024-09-17T07:25:21.762212+0000","last_fullsized":"2024-09-17T07:25:21.762212+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:15:51.877367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:15:51.877367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.058290885000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"439'38","reported_seq":669,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:19.869016+0000","last_change":"2024-09-17T07:25:11.645824+0000","last_active":"2024-09-17T07:25:19.869016+0000","last_peered":"2024-09-17T07:25:19.869016+0000","last_clean":"2024-09-17T07:25:19.869016+0000","last_became_active":"2024-09-17T07:22:13.583204+0000","last_became_peered":"2024-09-17T07:22:13.583204+0000","last_unstale":"2024-09-17T07:25:19.869016+0000","last_undegraded":"2024-09-17T07:25:19.869016+0000","last_fullsized":"2024-09-17T07:25:19.869016+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'38","last_scrub_stamp":"2024-09-17T07:25:11.645722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:25:11.645722+0000","objects_scrubbed":7,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:36.237351+0000","scrub_duration":0.0024003060000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"268'3","reported_seq":370,"reported_epoch":511,"state":"active+clean","last_fresh":"2024-09-17T07:25:22.309910+0000","last_change":"2024-09-17T07:25:15.066327+0000","last_active":"2024-09-17T07:25:22.309910+0000","last_peered":"2024-09-17T07:25:22.309910+0000","last_clean":"2024-09-17T07:25:22.309910+0000","last_became_active":"2024-09-17T07:23:57.485430+0000","last_became_peered":"2024-09-17T07:23:57.485430+0000","last_unstale":"2024-09-17T07:25:22.309910+0000","last_undegraded":"2024-09-17T07:25:22.309910+0000","last_fullsized":"2024-09-17T07:25:22.309910+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'3","last_scrub_stamp":"2024-09-17T07:25:15.066229+0000","last_deep_scrub":"268'3","last_deep_scrub_stamp":"2024-09-17T07:13:54.949345+0000","last_clean_scrub_stamp":"2024-09-17T07:25:15.066229+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:22.714073+0000","scrub_duration":0.0017457670000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":548,"reported_epoch":511,"state":"active+clean","last_fresh":"2024-09-17T07:25:22.310430+0000","last_change":"2024-09-17T07:23:57.480855+0000","last_active":"2024-09-17T07:25:22.310430+0000","last_peered":"2024-09-17T07:25:22.310430+0000","last_clean":"2024-09-17T07:25:22.310430+0000","last_became_active":"2024-09-17T07:23:57.477410+0000","last_became_peered":"2024-09-17T07:23:57.477410+0000","last_unstale":"2024-09-17T07:25:22.310430+0000","last_undegraded":"2024-09-17T07:25:22.310430+0000","last_fullsized":"2024-09-17T07:25:22.310430+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:17.650824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:20:17.650824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019547470000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"212'130","reported_seq":610,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:20.087546+0000","last_change":"2024-09-17T07:22:16.366636+0000","last_active":"2024-09-17T07:25:20.087546+0000","last_peered":"2024-09-17T07:25:20.087546+0000","last_clean":"2024-09-17T07:25:20.087546+0000","last_became_active":"2024-09-17T07:22:16.366295+0000","last_became_peered":"2024-09-17T07:22:16.366295+0000","last_unstale":"2024-09-17T07:25:20.087546+0000","last_undegraded":"2024-09-17T07:25:20.087546+0000","last_fullsized":"2024-09-17T07:25:20.087546+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:19:19.991044+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:14:57.052505+0000","last_clean_scrub_stamp":"2024-09-17T07:19:19.991044+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021141889999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":27,"num_write":16,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":161009,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":114,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:20.088503+0000","last_change":"2024-09-17T07:24:19.856080+0000","last_active":"2024-09-17T07:25:20.088503+0000","last_peered":"2024-09-17T07:25:20.088503+0000","last_clean":"2024-09-17T07:25:20.088503+0000","last_became_active":"2024-09-17T07:23:57.476027+0000","last_became_peered":"2024-09-17T07:23:57.476027+0000","last_unstale":"2024-09-17T07:25:20.088503+0000","last_undegraded":"2024-09-17T07:25:20.088503+0000","last_fullsized":"2024-09-17T07:25:20.088503+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:24:19.855925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:18:48.891136+0000","last_clean_scrub_stamp":"2024-09-17T07:24:19.855925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:47.079242+0000","scrub_duration":0.0055362720000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":494,"reported_epoch":511,"state":"active+clean","last_fresh":"2024-09-17T07:25:22.311200+0000","last_change":"2024-09-17T07:24:21.789270+0000","last_active":"2024-09-17T07:25:22.311200+0000","last_peered":"2024-09-17T07:25:22.311200+0000","last_clean":"2024-09-17T07:25:22.311200+0000","last_became_active":"2024-09-17T07:23:57.478248+0000","last_became_peered":"2024-09-17T07:23:57.478248+0000","last_unstale":"2024-09-17T07:25:22.311200+0000","last_undegraded":"2024-09-17T07:25:22.311200+0000","last_fullsized":"2024-09-17T07:25:22.311200+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:24:21.789110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:58.670571+0000","last_clean_scrub_stamp":"2024-09-17T07:24:21.789110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:30.597631+0000","scrub_duration":0.0090529149999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"439'51","reported_seq":713,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:19.868219+0000","last_change":"2024-09-17T07:25:15.611131+0000","last_active":"2024-09-17T07:25:19.868219+0000","last_peered":"2024-09-17T07:25:19.868219+0000","last_clean":"2024-09-17T07:25:19.868219+0000","last_became_active":"2024-09-17T07:22:13.588260+0000","last_became_peered":"2024-09-17T07:22:13.588260+0000","last_unstale":"2024-09-17T07:25:19.868219+0000","last_undegraded":"2024-09-17T07:25:19.868219+0000","last_fullsized":"2024-09-17T07:25:19.868219+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'51","last_scrub_stamp":"2024-09-17T07:25:15.611017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:25:15.611017+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:28.411603+0000","scrub_duration":0.0030635860000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":626,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:19.742652+0000","last_change":"2024-09-17T07:24:23.255568+0000","last_active":"2024-09-17T07:25:19.742652+0000","last_peered":"2024-09-17T07:25:19.742652+0000","last_clean":"2024-09-17T07:25:19.742652+0000","last_became_active":"2024-09-17T07:23:57.482084+0000","last_became_peered":"2024-09-17T07:23:57.482084+0000","last_unstale":"2024-09-17T07:25:19.742652+0000","last_undegraded":"2024-09-17T07:25:19.742652+0000","last_fullsized":"2024-09-17T07:25:19.742652+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:24:23.255270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:24:23.255270+0000","last_clean_scrub_stamp":"2024-09-17T07:24:23.255270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:30.797016+0000","scrub_duration":0.0036829879999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":609,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:19.742132+0000","last_change":"2024-09-17T07:24:20.283146+0000","last_active":"2024-09-17T07:25:19.742132+0000","last_peered":"2024-09-17T07:25:19.742132+0000","last_clean":"2024-09-17T07:25:19.742132+0000","last_became_active":"2024-09-17T07:23:57.481504+0000","last_became_peered":"2024-09-17T07:23:57.481504+0000","last_unstale":"2024-09-17T07:25:19.742132+0000","last_undegraded":"2024-09-17T07:25:19.742132+0000","last_fullsized":"2024-09-17T07:25:19.742132+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:24:20.282973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:24:20.282973+0000","last_clean_scrub_stamp":"2024-09-17T07:24:20.282973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:26.632960+0000","scrub_duration":0.024343656000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":99,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:20.087648+0000","last_change":"2024-09-17T07:24:23.847700+0000","last_active":"2024-09-17T07:25:20.087648+0000","last_peered":"2024-09-17T07:25:20.087648+0000","last_clean":"2024-09-17T07:25:20.087648+0000","last_became_active":"2024-09-17T07:22:13.573161+0000","last_became_peered":"2024-09-17T07:22:13.573161+0000","last_unstale":"2024-09-17T07:25:20.087648+0000","last_undegraded":"2024-09-17T07:25:20.087648+0000","last_fullsized":"2024-09-17T07:25:20.087648+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:24:23.847363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:45.275473+0000","last_clean_scrub_stamp":"2024-09-17T07:24:23.847363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:31.038832+0000","scrub_duration":0.0047891890000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"439'25","reported_seq":582,"reported_epoch":511,"state":"active+clean","last_fresh":"2024-09-17T07:25:21.764688+0000","last_change":"2024-09-17T07:24:56.652772+0000","last_active":"2024-09-17T07:25:21.764688+0000","last_peered":"2024-09-17T07:25:21.764688+0000","last_clean":"2024-09-17T07:25:21.764688+0000","last_became_active":"2024-09-17T07:21:15.349361+0000","last_became_peered":"2024-09-17T07:21:15.349361+0000","last_unstale":"2024-09-17T07:25:21.764688+0000","last_undegraded":"2024-09-17T07:25:21.764688+0000","last_fullsized":"2024-09-17T07:25:21.764688+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'25","last_scrub_stamp":"2024-09-17T07:24:56.652673+0000","last_deep_scrub":"268'19","last_deep_scrub_stamp":"2024-09-17T07:14:06.634334+0000","last_clean_scrub_stamp":"2024-09-17T07:24:56.652673+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:21.478181+0000","scrub_duration":0.012732797000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":436,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:20.753868+0000","last_change":"2024-09-17T07:25:14.981641+0000","last_active":"2024-09-17T07:25:20.753868+0000","last_peered":"2024-09-17T07:25:20.753868+0000","last_clean":"2024-09-17T07:25:20.753868+0000","last_became_active":"2024-09-17T07:21:15.350786+0000","last_became_peered":"2024-09-17T07:21:15.350786+0000","last_unstale":"2024-09-17T07:25:20.753868+0000","last_undegraded":"2024-09-17T07:25:20.753868+0000","last_fullsized":"2024-09-17T07:25:20.753868+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:14.981560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:13:53.593078+0000","last_clean_scrub_stamp":"2024-09-17T07:25:14.981560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:30.222480+0000","scrub_duration":0.0016595399999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":521,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:19.742500+0000","last_change":"2024-09-17T07:25:13.923025+0000","last_active":"2024-09-17T07:25:19.742500+0000","last_peered":"2024-09-17T07:25:19.742500+0000","last_clean":"2024-09-17T07:25:19.742500+0000","last_became_active":"2024-09-17T07:09:24.396642+0000","last_became_peered":"2024-09-17T07:09:24.396642+0000","last_unstale":"2024-09-17T07:25:19.742500+0000","last_undegraded":"2024-09-17T07:25:19.742500+0000","last_fullsized":"2024-09-17T07:25:19.742500+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:13.922902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:25:13.922902+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:19.966114+0000","scrub_duration":0.0019707629999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":563,"reported_epoch":511,"state":"active+clean","last_fresh":"2024-09-17T07:25:21.764166+0000","last_change":"2024-09-17T07:25:16.639982+0000","last_active":"2024-09-17T07:25:21.764166+0000","last_peered":"2024-09-17T07:25:21.764166+0000","last_clean":"2024-09-17T07:25:21.764166+0000","last_became_active":"2024-09-17T07:22:13.589460+0000","last_became_peered":"2024-09-17T07:22:13.589460+0000","last_unstale":"2024-09-17T07:25:21.764166+0000","last_undegraded":"2024-09-17T07:25:21.764166+0000","last_fullsized":"2024-09-17T07:25:21.764166+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:16.639899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:58.099849+0000","last_clean_scrub_stamp":"2024-09-17T07:25:16.639899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:41.555366+0000","scrub_duration":0.001931898,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"439'46","reported_seq":651,"reported_epoch":511,"state":"active+clean","last_fresh":"2024-09-17T07:25:22.309327+0000","last_change":"2024-09-17T07:25:00.980570+0000","last_active":"2024-09-17T07:25:22.309327+0000","last_peered":"2024-09-17T07:25:22.309327+0000","last_clean":"2024-09-17T07:25:22.309327+0000","last_became_active":"2024-09-17T07:23:57.482858+0000","last_became_peered":"2024-09-17T07:23:57.482858+0000","last_unstale":"2024-09-17T07:25:22.309327+0000","last_undegraded":"2024-09-17T07:25:22.309327+0000","last_fullsized":"2024-09-17T07:25:22.309327+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'46","last_scrub_stamp":"2024-09-17T07:25:00.980474+0000","last_deep_scrub":"242'34","last_deep_scrub_stamp":"2024-09-17T07:14:50.103539+0000","last_clean_scrub_stamp":"2024-09-17T07:25:00.980474+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:06.860259+0000","scrub_duration":0.0021656290000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'2","reported_seq":94,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:20.088317+0000","last_change":"2024-09-17T07:25:05.848575+0000","last_active":"2024-09-17T07:25:20.088317+0000","last_peered":"2024-09-17T07:25:20.088317+0000","last_clean":"2024-09-17T07:25:20.088317+0000","last_became_active":"2024-09-17T07:22:13.575469+0000","last_became_peered":"2024-09-17T07:22:13.575469+0000","last_unstale":"2024-09-17T07:25:20.088317+0000","last_undegraded":"2024-09-17T07:25:20.088317+0000","last_fullsized":"2024-09-17T07:25:20.088317+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-09-17T07:18:43.861842+0000","last_deep_scrub":"89'2","last_deep_scrub_stamp":"2024-09-17T07:12:20.301263+0000","last_clean_scrub_stamp":"2024-09-17T07:18:43.861842+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":420,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":420,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"436'14","reported_seq":142,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:20.783955+0000","last_change":"2024-09-17T07:24:59.877855+0000","last_active":"2024-09-17T07:25:20.783955+0000","last_peered":"2024-09-17T07:25:20.783955+0000","last_clean":"2024-09-17T07:25:20.783955+0000","last_became_active":"2024-09-17T07:22:13.624050+0000","last_became_peered":"2024-09-17T07:22:13.624050+0000","last_unstale":"2024-09-17T07:25:20.783955+0000","last_undegraded":"2024-09-17T07:25:20.783955+0000","last_fullsized":"2024-09-17T07:25:20.783955+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"436'14","last_scrub_stamp":"2024-09-17T07:24:59.877774+0000","last_deep_scrub":"225'9","last_deep_scrub_stamp":"2024-09-17T07:11:25.292565+0000","last_clean_scrub_stamp":"2024-09-17T07:24:59.877774+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:07.407178+0000","scrub_duration":0.0015096720000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":526,"reported_epoch":511,"state":"active+clean","last_fresh":"2024-09-17T07:25:22.309841+0000","last_change":"2024-09-17T07:24:58.961010+0000","last_active":"2024-09-17T07:25:22.309841+0000","last_peered":"2024-09-17T07:25:22.309841+0000","last_clean":"2024-09-17T07:25:22.309841+0000","last_became_active":"2024-09-17T07:23:57.493273+0000","last_became_peered":"2024-09-17T07:23:57.493273+0000","last_unstale":"2024-09-17T07:25:22.309841+0000","last_undegraded":"2024-09-17T07:25:22.309841+0000","last_fullsized":"2024-09-17T07:25:22.309841+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:24:58.960933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:18:41.385928+0000","last_clean_scrub_stamp":"2024-09-17T07:24:58.960933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:03.753329+0000","scrub_duration":0.0012208259999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"439'29","reported_seq":637,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:19.742750+0000","last_change":"2024-09-17T07:20:33.108656+0000","last_active":"2024-09-17T07:25:19.742750+0000","last_peered":"2024-09-17T07:25:19.742750+0000","last_clean":"2024-09-17T07:25:19.742750+0000","last_became_active":"2024-09-17T07:10:49.723292+0000","last_became_peered":"2024-09-17T07:10:49.723292+0000","last_unstale":"2024-09-17T07:25:19.742750+0000","last_undegraded":"2024-09-17T07:25:19.742750+0000","last_fullsized":"2024-09-17T07:25:19.742750+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'23","last_scrub_stamp":"2024-09-17T07:20:33.108490+0000","last_deep_scrub":"150'17","last_deep_scrub_stamp":"2024-09-17T07:06:38.775788+0000","last_clean_scrub_stamp":"2024-09-17T07:20:33.108490+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0087601510000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"68'2","reported_seq":588,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:19.743435+0000","last_change":"2024-09-17T07:22:13.766671+0000","last_active":"2024-09-17T07:25:19.743435+0000","last_peered":"2024-09-17T07:25:19.743435+0000","last_clean":"2024-09-17T07:25:19.743435+0000","last_became_active":"2024-09-17T07:22:13.590172+0000","last_became_peered":"2024-09-17T07:22:13.590172+0000","last_unstale":"2024-09-17T07:25:19.743435+0000","last_undegraded":"2024-09-17T07:25:19.743435+0000","last_fullsized":"2024-09-17T07:25:19.743435+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-09-17T07:20:11.093491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:20:11.093491+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023198989999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":516,"reported_epoch":511,"state":"active+clean","last_fresh":"2024-09-17T07:25:21.764753+0000","last_change":"2024-09-17T07:24:26.618642+0000","last_active":"2024-09-17T07:25:21.764753+0000","last_peered":"2024-09-17T07:25:21.764753+0000","last_clean":"2024-09-17T07:25:21.764753+0000","last_became_active":"2024-09-17T07:21:15.355760+0000","last_became_peered":"2024-09-17T07:21:15.355760+0000","last_unstale":"2024-09-17T07:25:21.764753+0000","last_undegraded":"2024-09-17T07:25:21.764753+0000","last_fullsized":"2024-09-17T07:25:21.764753+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:24:26.618433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:11:17.479036+0000","last_clean_scrub_stamp":"2024-09-17T07:24:26.618433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:30.630753+0000","scrub_duration":0.0026272880000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":88,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:20.088127+0000","last_change":"2024-09-17T07:22:13.585879+0000","last_active":"2024-09-17T07:25:20.088127+0000","last_peered":"2024-09-17T07:25:20.088127+0000","last_clean":"2024-09-17T07:25:20.088127+0000","last_became_active":"2024-09-17T07:22:13.585580+0000","last_became_peered":"2024-09-17T07:22:13.585580+0000","last_unstale":"2024-09-17T07:25:20.088127+0000","last_undegraded":"2024-09-17T07:25:20.088127+0000","last_fullsized":"2024-09-17T07:25:20.088127+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:07.959158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:20:07.959158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.018487173999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"439'21","reported_seq":630,"reported_epoch":511,"state":"active+clean","last_fresh":"2024-09-17T07:25:21.764821+0000","last_change":"2024-09-17T07:25:05.507259+0000","last_active":"2024-09-17T07:25:21.764821+0000","last_peered":"2024-09-17T07:25:21.764821+0000","last_clean":"2024-09-17T07:25:21.764821+0000","last_became_active":"2024-09-17T07:22:13.592001+0000","last_became_peered":"2024-09-17T07:22:13.592001+0000","last_unstale":"2024-09-17T07:25:21.764821+0000","last_undegraded":"2024-09-17T07:25:21.764821+0000","last_fullsized":"2024-09-17T07:25:21.764821+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'15","last_scrub_stamp":"2024-09-17T07:18:24.505002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:18:24.505002+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002195809,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":596,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:19.741504+0000","last_change":"2024-09-17T07:23:57.042832+0000","last_active":"2024-09-17T07:25:19.741504+0000","last_peered":"2024-09-17T07:25:19.741504+0000","last_clean":"2024-09-17T07:25:19.741504+0000","last_became_active":"2024-09-17T07:21:15.355464+0000","last_became_peered":"2024-09-17T07:21:15.355464+0000","last_unstale":"2024-09-17T07:25:19.741504+0000","last_undegraded":"2024-09-17T07:25:19.741504+0000","last_fullsized":"2024-09-17T07:25:19.741504+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:23:57.042751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:17.976267+0000","last_clean_scrub_stamp":"2024-09-17T07:23:57.042751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:22.442338+0000","scrub_duration":0.001374662,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":247,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:20.753678+0000","last_change":"2024-09-17T07:22:13.593743+0000","last_active":"2024-09-17T07:25:20.753678+0000","last_peered":"2024-09-17T07:25:20.753678+0000","last_clean":"2024-09-17T07:25:20.753678+0000","last_became_active":"2024-09-17T07:22:13.593446+0000","last_became_peered":"2024-09-17T07:22:13.593446+0000","last_unstale":"2024-09-17T07:25:20.753678+0000","last_undegraded":"2024-09-17T07:25:20.753678+0000","last_fullsized":"2024-09-17T07:25:20.753678+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:29.492000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:18.367486+0000","last_clean_scrub_stamp":"2024-09-17T07:20:29.492000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0077309099999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":649,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:19.742920+0000","last_change":"2024-09-17T07:24:24.282738+0000","last_active":"2024-09-17T07:25:19.742920+0000","last_peered":"2024-09-17T07:25:19.742920+0000","last_clean":"2024-09-17T07:25:19.742920+0000","last_became_active":"2024-09-17T07:22:13.590599+0000","last_became_peered":"2024-09-17T07:22:13.590599+0000","last_unstale":"2024-09-17T07:25:19.742920+0000","last_undegraded":"2024-09-17T07:25:19.742920+0000","last_fullsized":"2024-09-17T07:25:19.742920+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:24:24.282487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:24:24.282487+0000","last_clean_scrub_stamp":"2024-09-17T07:24:24.282487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:43.887065+0000","scrub_duration":0.0043307379999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":565,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:19.868045+0000","last_change":"2024-09-17T07:24:20.528931+0000","last_active":"2024-09-17T07:25:19.868045+0000","last_peered":"2024-09-17T07:25:19.868045+0000","last_clean":"2024-09-17T07:25:19.868045+0000","last_became_active":"2024-09-17T07:21:15.348758+0000","last_became_peered":"2024-09-17T07:21:15.348758+0000","last_unstale":"2024-09-17T07:25:19.868045+0000","last_undegraded":"2024-09-17T07:25:19.868045+0000","last_fullsized":"2024-09-17T07:25:19.868045+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:24:20.528714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:46.645080+0000","last_clean_scrub_stamp":"2024-09-17T07:24:20.528714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:34.396548+0000","scrub_duration":0.0080332240000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"439'18","reported_seq":105,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:20.088355+0000","last_change":"2024-09-17T07:22:13.761675+0000","last_active":"2024-09-17T07:25:20.088355+0000","last_peered":"2024-09-17T07:25:20.088355+0000","last_clean":"2024-09-17T07:25:20.088355+0000","last_became_active":"2024-09-17T07:22:13.586662+0000","last_became_peered":"2024-09-17T07:22:13.586662+0000","last_unstale":"2024-09-17T07:25:20.088355+0000","last_undegraded":"2024-09-17T07:25:20.088355+0000","last_fullsized":"2024-09-17T07:25:20.088355+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'12","last_scrub_stamp":"2024-09-17T07:20:17.090146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:20:17.090146+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019381719999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":25,"num_read_kb":22,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":506,"reported_epoch":511,"state":"active+clean","last_fresh":"2024-09-17T07:25:22.310740+0000","last_change":"2024-09-17T07:24:23.835840+0000","last_active":"2024-09-17T07:25:22.310740+0000","last_peered":"2024-09-17T07:25:22.310740+0000","last_clean":"2024-09-17T07:25:22.310740+0000","last_became_active":"2024-09-17T07:23:57.478324+0000","last_became_peered":"2024-09-17T07:23:57.478324+0000","last_unstale":"2024-09-17T07:25:22.310740+0000","last_undegraded":"2024-09-17T07:25:22.310740+0000","last_fullsized":"2024-09-17T07:25:22.310740+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:24:23.835718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:24:23.835718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:47.145573+0000","scrub_duration":0.0037662490000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":99,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:20.088056+0000","last_change":"2024-09-17T07:25:06.877892+0000","last_active":"2024-09-17T07:25:20.088056+0000","last_peered":"2024-09-17T07:25:20.088056+0000","last_clean":"2024-09-17T07:25:20.088056+0000","last_became_active":"2024-09-17T07:22:13.623754+0000","last_became_peered":"2024-09-17T07:22:13.623754+0000","last_unstale":"2024-09-17T07:25:20.088056+0000","last_undegraded":"2024-09-17T07:25:20.088056+0000","last_fullsized":"2024-09-17T07:25:20.088056+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:06.877789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:25:06.877789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:29.306772+0000","scrub_duration":0.001331243,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"439'30","reported_seq":306,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:20.753686+0000","last_change":"2024-09-17T07:22:13.768713+0000","last_active":"2024-09-17T07:25:20.753686+0000","last_peered":"2024-09-17T07:25:20.753686+0000","last_clean":"2024-09-17T07:25:20.753686+0000","last_became_active":"2024-09-17T07:22:13.591094+0000","last_became_peered":"2024-09-17T07:22:13.591094+0000","last_unstale":"2024-09-17T07:25:20.753686+0000","last_undegraded":"2024-09-17T07:25:20.753686+0000","last_fullsized":"2024-09-17T07:25:20.753686+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'20","last_scrub_stamp":"2024-09-17T07:20:10.554324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:20:10.554324+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0020069329999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":567,"reported_epoch":511,"state":"active+clean","last_fresh":"2024-09-17T07:25:22.309760+0000","last_change":"2024-09-17T07:25:20.089270+0000","last_active":"2024-09-17T07:25:22.309760+0000","last_peered":"2024-09-17T07:25:22.309760+0000","last_clean":"2024-09-17T07:25:22.309760+0000","last_became_active":"2024-09-17T07:23:57.491291+0000","last_became_peered":"2024-09-17T07:23:57.491291+0000","last_unstale":"2024-09-17T07:25:22.309760+0000","last_undegraded":"2024-09-17T07:25:22.309760+0000","last_fullsized":"2024-09-17T07:25:22.309760+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:20.089157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:19:40.443474+0000","last_clean_scrub_stamp":"2024-09-17T07:25:20.089157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:36.357354+0000","scrub_duration":0.0033687460000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"478'13","reported_seq":754,"reported_epoch":511,"state":"active+clean","last_fresh":"2024-09-17T07:25:22.310949+0000","last_change":"2024-09-17T07:24:31.008439+0000","last_active":"2024-09-17T07:25:22.310949+0000","last_peered":"2024-09-17T07:25:22.310949+0000","last_clean":"2024-09-17T07:25:22.310949+0000","last_became_active":"2024-09-17T07:23:57.522386+0000","last_became_peered":"2024-09-17T07:23:57.522386+0000","last_unstale":"2024-09-17T07:25:22.310949+0000","last_undegraded":"2024-09-17T07:25:22.310949+0000","last_fullsized":"2024-09-17T07:25:22.310949+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"478'13","last_scrub_stamp":"2024-09-17T07:24:31.008289+0000","last_deep_scrub":"163'9","last_deep_scrub_stamp":"2024-09-17T07:07:34.933808+0000","last_clean_scrub_stamp":"2024-09-17T07:24:31.008289+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:47.002555+0000","scrub_duration":0.0084935229999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":572,"reported_epoch":511,"state":"active+clean","last_fresh":"2024-09-17T07:25:21.764612+0000","last_change":"2024-09-17T07:25:06.624551+0000","last_active":"2024-09-17T07:25:21.764612+0000","last_peered":"2024-09-17T07:25:21.764612+0000","last_clean":"2024-09-17T07:25:21.764612+0000","last_became_active":"2024-09-17T07:22:13.591307+0000","last_became_peered":"2024-09-17T07:22:13.591307+0000","last_unstale":"2024-09-17T07:25:21.764612+0000","last_undegraded":"2024-09-17T07:25:21.764612+0000","last_fullsized":"2024-09-17T07:25:21.764612+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:06.624450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:26.619893+0000","last_clean_scrub_stamp":"2024-09-17T07:25:06.624450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:13.532043+0000","scrub_duration":0.001368408,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"439'21","reported_seq":623,"reported_epoch":511,"state":"active+clean","last_fresh":"2024-09-17T07:25:22.310864+0000","last_change":"2024-09-17T07:25:08.008455+0000","last_active":"2024-09-17T07:25:22.310864+0000","last_peered":"2024-09-17T07:25:22.310864+0000","last_clean":"2024-09-17T07:25:22.310864+0000","last_became_active":"2024-09-17T07:23:57.485557+0000","last_became_peered":"2024-09-17T07:23:57.485557+0000","last_unstale":"2024-09-17T07:25:22.310864+0000","last_undegraded":"2024-09-17T07:25:22.310864+0000","last_fullsized":"2024-09-17T07:25:22.310864+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'21","last_scrub_stamp":"2024-09-17T07:25:08.008359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:25:08.008359+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:28.888348+0000","scrub_duration":0.0025217759999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":581,"reported_epoch":511,"state":"active+clean","last_fresh":"2024-09-17T07:25:21.764973+0000","last_change":"2024-09-17T07:25:18.620750+0000","last_active":"2024-09-17T07:25:21.764973+0000","last_peered":"2024-09-17T07:25:21.764973+0000","last_clean":"2024-09-17T07:25:21.764973+0000","last_became_active":"2024-09-17T07:22:13.589964+0000","last_became_peered":"2024-09-17T07:22:13.589964+0000","last_unstale":"2024-09-17T07:25:21.764973+0000","last_undegraded":"2024-09-17T07:25:21.764973+0000","last_fullsized":"2024-09-17T07:25:21.764973+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:18.620666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:25:18.620666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:37.392688+0000","scrub_duration":0.001507972,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"436'15","reported_seq":141,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:20.784013+0000","last_change":"2024-09-17T07:25:18.993295+0000","last_active":"2024-09-17T07:25:20.784013+0000","last_peered":"2024-09-17T07:25:20.784013+0000","last_clean":"2024-09-17T07:25:20.784013+0000","last_became_active":"2024-09-17T07:22:13.586620+0000","last_became_peered":"2024-09-17T07:22:13.586620+0000","last_unstale":"2024-09-17T07:25:20.784013+0000","last_undegraded":"2024-09-17T07:25:20.784013+0000","last_fullsized":"2024-09-17T07:25:20.784013+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"436'15","last_scrub_stamp":"2024-09-17T07:25:18.993184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:25:18.993184+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:42.271745+0000","scrub_duration":0.0021849909999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":570,"reported_epoch":511,"state":"active+clean","last_fresh":"2024-09-17T07:25:22.865513+0000","last_change":"2024-09-17T07:25:22.865620+0000","last_active":"2024-09-17T07:25:22.865513+0000","last_peered":"2024-09-17T07:25:22.865513+0000","last_clean":"2024-09-17T07:25:22.865513+0000","last_became_active":"2024-09-17T07:23:57.472854+0000","last_became_peered":"2024-09-17T07:23:57.472854+0000","last_unstale":"2024-09-17T07:25:22.865513+0000","last_undegraded":"2024-09-17T07:25:22.865513+0000","last_fullsized":"2024-09-17T07:25:22.865513+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:22.865498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:20:37.110458+0000","last_clean_scrub_stamp":"2024-09-17T07:25:22.865498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:46.956482+0000","scrub_duration":0.0016306630000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"439'21","reported_seq":418,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:20.753535+0000","last_change":"2024-09-17T07:24:18.055969+0000","last_active":"2024-09-17T07:25:20.753535+0000","last_peered":"2024-09-17T07:25:20.753535+0000","last_clean":"2024-09-17T07:25:20.753535+0000","last_became_active":"2024-09-17T07:21:15.348060+0000","last_became_peered":"2024-09-17T07:21:15.348060+0000","last_unstale":"2024-09-17T07:25:20.753535+0000","last_undegraded":"2024-09-17T07:25:20.753535+0000","last_fullsized":"2024-09-17T07:25:20.753535+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'21","last_scrub_stamp":"2024-09-17T07:24:18.055874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:24:18.055874+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:26.355516+0000","scrub_duration":0.0022038589999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":612,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:19.868804+0000","last_change":"2024-09-17T07:24:21.515088+0000","last_active":"2024-09-17T07:25:19.868804+0000","last_peered":"2024-09-17T07:25:19.868804+0000","last_clean":"2024-09-17T07:25:19.868804+0000","last_became_active":"2024-09-17T07:23:57.493401+0000","last_became_peered":"2024-09-17T07:23:57.493401+0000","last_unstale":"2024-09-17T07:25:19.868804+0000","last_undegraded":"2024-09-17T07:25:19.868804+0000","last_fullsized":"2024-09-17T07:25:19.868804+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:24:21.514803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:10:36.334827+0000","last_clean_scrub_stamp":"2024-09-17T07:24:21.514803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:25.466862+0000","scrub_duration":0.0094769510000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":409,"reported_epoch":511,"state":"active+clean","last_fresh":"2024-09-17T07:25:21.764450+0000","last_change":"2024-09-17T07:25:00.641893+0000","last_active":"2024-09-17T07:25:21.764450+0000","last_peered":"2024-09-17T07:25:21.764450+0000","last_clean":"2024-09-17T07:25:21.764450+0000","last_became_active":"2024-09-17T07:23:57.479281+0000","last_became_peered":"2024-09-17T07:23:57.479281+0000","last_unstale":"2024-09-17T07:25:21.764450+0000","last_undegraded":"2024-09-17T07:25:21.764450+0000","last_fullsized":"2024-09-17T07:25:21.764450+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:00.641817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:25:00.641817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:22.472510+0000","scrub_duration":0.031817810000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":563,"reported_epoch":511,"state":"active+clean","last_fresh":"2024-09-17T07:25:21.764310+0000","last_change":"2024-09-17T07:24:21.576079+0000","last_active":"2024-09-17T07:25:21.764310+0000","last_peered":"2024-09-17T07:25:21.764310+0000","last_clean":"2024-09-17T07:25:21.764310+0000","last_became_active":"2024-09-17T07:23:57.478818+0000","last_became_peered":"2024-09-17T07:23:57.478818+0000","last_unstale":"2024-09-17T07:25:21.764310+0000","last_undegraded":"2024-09-17T07:25:21.764310+0000","last_fullsized":"2024-09-17T07:25:21.764310+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:24:21.575574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:24:21.575574+0000","last_clean_scrub_stamp":"2024-09-17T07:24:21.575574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:25.442934+0000","scrub_duration":0.006786233,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"439'27","reported_seq":584,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:19.742808+0000","last_change":"2024-09-17T07:21:16.070482+0000","last_active":"2024-09-17T07:25:19.742808+0000","last_peered":"2024-09-17T07:25:19.742808+0000","last_clean":"2024-09-17T07:25:19.742808+0000","last_became_active":"2024-09-17T07:21:15.347179+0000","last_became_peered":"2024-09-17T07:21:15.347179+0000","last_unstale":"2024-09-17T07:25:19.742808+0000","last_undegraded":"2024-09-17T07:25:19.742808+0000","last_fullsized":"2024-09-17T07:25:19.742808+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'19","last_scrub_stamp":"2024-09-17T07:21:16.070358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:21:16.070358+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0025953619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"94'2","reported_seq":434,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:20.753960+0000","last_change":"2024-09-17T07:25:16.939704+0000","last_active":"2024-09-17T07:25:20.753960+0000","last_peered":"2024-09-17T07:25:20.753960+0000","last_clean":"2024-09-17T07:25:20.753960+0000","last_became_active":"2024-09-17T07:23:57.521742+0000","last_became_peered":"2024-09-17T07:23:57.521742+0000","last_unstale":"2024-09-17T07:25:20.753960+0000","last_undegraded":"2024-09-17T07:25:20.753960+0000","last_fullsized":"2024-09-17T07:25:20.753960+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-09-17T07:25:16.939572+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-09-17T07:07:31.717888+0000","last_clean_scrub_stamp":"2024-09-17T07:25:16.939572+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:43.987976+0000","scrub_duration":0.001518578,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":427,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:20.753329+0000","last_change":"2024-09-17T07:23:57.495235+0000","last_active":"2024-09-17T07:25:20.753329+0000","last_peered":"2024-09-17T07:25:20.753329+0000","last_clean":"2024-09-17T07:25:20.753329+0000","last_became_active":"2024-09-17T07:23:57.494701+0000","last_became_peered":"2024-09-17T07:23:57.494701+0000","last_unstale":"2024-09-17T07:25:20.753329+0000","last_undegraded":"2024-09-17T07:25:20.753329+0000","last_fullsized":"2024-09-17T07:25:20.753329+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:16.640384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:20:16.640384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003133224,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":404,"reported_epoch":511,"state":"active+clean","last_fresh":"2024-09-17T07:25:21.764311+0000","last_change":"2024-09-17T07:25:02.641851+0000","last_active":"2024-09-17T07:25:21.764311+0000","last_peered":"2024-09-17T07:25:21.764311+0000","last_clean":"2024-09-17T07:25:21.764311+0000","last_became_active":"2024-09-17T07:23:57.479008+0000","last_became_peered":"2024-09-17T07:23:57.479008+0000","last_unstale":"2024-09-17T07:25:21.764311+0000","last_undegraded":"2024-09-17T07:25:21.764311+0000","last_fullsized":"2024-09-17T07:25:21.764311+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:02.641763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:49.573666+0000","last_clean_scrub_stamp":"2024-09-17T07:25:02.641763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:19.718312+0000","scrub_duration":0.001362083,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"439'12","reported_seq":492,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:19.741248+0000","last_change":"2024-09-17T07:25:01.921597+0000","last_active":"2024-09-17T07:25:19.741248+0000","last_peered":"2024-09-17T07:25:19.741248+0000","last_clean":"2024-09-17T07:25:19.741248+0000","last_became_active":"2024-09-17T07:09:24.396368+0000","last_became_peered":"2024-09-17T07:09:24.396368+0000","last_unstale":"2024-09-17T07:25:19.741248+0000","last_undegraded":"2024-09-17T07:25:19.741248+0000","last_fullsized":"2024-09-17T07:25:19.741248+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'12","last_scrub_stamp":"2024-09-17T07:25:01.921507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:25:01.921507+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:21.393421+0000","scrub_duration":0.0020119589999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":508,"reported_epoch":511,"state":"active+clean","last_fresh":"2024-09-17T07:25:22.311035+0000","last_change":"2024-09-17T07:25:19.124760+0000","last_active":"2024-09-17T07:25:22.311035+0000","last_peered":"2024-09-17T07:25:22.311035+0000","last_clean":"2024-09-17T07:25:22.311035+0000","last_became_active":"2024-09-17T07:23:57.476234+0000","last_became_peered":"2024-09-17T07:23:57.476234+0000","last_unstale":"2024-09-17T07:25:22.311035+0000","last_undegraded":"2024-09-17T07:25:22.311035+0000","last_fullsized":"2024-09-17T07:25:22.311035+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:19.124658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:25:19.124658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:22.203780+0000","scrub_duration":0.0013943600000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":586,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:19.867793+0000","last_change":"2024-09-17T07:25:03.562390+0000","last_active":"2024-09-17T07:25:19.867793+0000","last_peered":"2024-09-17T07:25:19.867793+0000","last_clean":"2024-09-17T07:25:19.867793+0000","last_became_active":"2024-09-17T07:22:13.582229+0000","last_became_peered":"2024-09-17T07:22:13.582229+0000","last_unstale":"2024-09-17T07:25:19.867793+0000","last_undegraded":"2024-09-17T07:25:19.867793+0000","last_fullsized":"2024-09-17T07:25:19.867793+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:03.562297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:54.553970+0000","last_clean_scrub_stamp":"2024-09-17T07:25:03.562297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:19.909241+0000","scrub_duration":0.0012245349999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":88,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:20.087547+0000","last_change":"2024-09-17T07:22:13.624740+0000","last_active":"2024-09-17T07:25:20.087547+0000","last_peered":"2024-09-17T07:25:20.087547+0000","last_clean":"2024-09-17T07:25:20.087547+0000","last_became_active":"2024-09-17T07:22:13.624127+0000","last_became_peered":"2024-09-17T07:22:13.624127+0000","last_unstale":"2024-09-17T07:25:20.087547+0000","last_undegraded":"2024-09-17T07:25:20.087547+0000","last_fullsized":"2024-09-17T07:25:20.087547+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:25.003509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:20:25.003509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002164586,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"439'27","reported_seq":127,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:20.087426+0000","last_change":"2024-09-17T07:23:57.476210+0000","last_active":"2024-09-17T07:25:20.087426+0000","last_peered":"2024-09-17T07:25:20.087426+0000","last_clean":"2024-09-17T07:25:20.087426+0000","last_became_active":"2024-09-17T07:23:57.475861+0000","last_became_peered":"2024-09-17T07:23:57.475861+0000","last_unstale":"2024-09-17T07:25:20.087426+0000","last_undegraded":"2024-09-17T07:25:20.087426+0000","last_fullsized":"2024-09-17T07:25:20.087426+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'19","last_scrub_stamp":"2024-09-17T07:20:28.113977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:20:28.113977+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.013438802,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":585,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:19.869195+0000","last_change":"2024-09-17T07:24:23.526255+0000","last_active":"2024-09-17T07:25:19.869195+0000","last_peered":"2024-09-17T07:25:19.869195+0000","last_clean":"2024-09-17T07:25:19.869195+0000","last_became_active":"2024-09-17T07:23:57.486575+0000","last_became_peered":"2024-09-17T07:23:57.486575+0000","last_unstale":"2024-09-17T07:25:19.869195+0000","last_undegraded":"2024-09-17T07:25:19.869195+0000","last_fullsized":"2024-09-17T07:25:19.869195+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:24:23.526138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:18:41.951382+0000","last_clean_scrub_stamp":"2024-09-17T07:24:23.526138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:48.414517+0000","scrub_duration":0.0045997989999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":88,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:20.087455+0000","last_change":"2024-09-17T07:22:13.736537+0000","last_active":"2024-09-17T07:25:20.087455+0000","last_peered":"2024-09-17T07:25:20.087455+0000","last_clean":"2024-09-17T07:25:20.087455+0000","last_became_active":"2024-09-17T07:22:13.710754+0000","last_became_peered":"2024-09-17T07:22:13.710754+0000","last_unstale":"2024-09-17T07:25:20.087455+0000","last_undegraded":"2024-09-17T07:25:20.087455+0000","last_fullsized":"2024-09-17T07:25:20.087455+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:30.038044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:20:30.038044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":400,"reported_epoch":511,"state":"active+clean","last_fresh":"2024-09-17T07:25:21.764642+0000","last_change":"2024-09-17T07:25:17.663679+0000","last_active":"2024-09-17T07:25:21.764642+0000","last_peered":"2024-09-17T07:25:21.764642+0000","last_clean":"2024-09-17T07:25:21.764642+0000","last_became_active":"2024-09-17T07:09:24.392216+0000","last_became_peered":"2024-09-17T07:09:24.392216+0000","last_unstale":"2024-09-17T07:25:21.764642+0000","last_undegraded":"2024-09-17T07:25:21.764642+0000","last_fullsized":"2024-09-17T07:25:21.764642+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:17.663576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:25:17.663576+0000","last_clean_scrub_stamp":"2024-09-17T07:25:17.663576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:28.898950+0000","scrub_duration":0.0013785539999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1c","version":"326'2","reported_seq":107,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:20.088413+0000","last_change":"2024-09-17T07:23:57.476548+0000","last_active":"2024-09-17T07:25:20.088413+0000","last_peered":"2024-09-17T07:25:20.088413+0000","last_clean":"2024-09-17T07:25:20.088413+0000","last_became_active":"2024-09-17T07:23:57.475630+0000","last_became_peered":"2024-09-17T07:23:57.475630+0000","last_unstale":"2024-09-17T07:25:20.088413+0000","last_undegraded":"2024-09-17T07:25:20.088413+0000","last_fullsized":"2024-09-17T07:25:20.088413+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'2","last_scrub_stamp":"2024-09-17T07:20:19.609678+0000","last_deep_scrub":"326'2","last_deep_scrub_stamp":"2024-09-17T07:17:36.974358+0000","last_clean_scrub_stamp":"2024-09-17T07:20:19.609678+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0053255050000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":548,"reported_epoch":511,"state":"active+clean","last_fresh":"2024-09-17T07:25:22.309482+0000","last_change":"2024-09-17T07:25:17.089768+0000","last_active":"2024-09-17T07:25:22.309482+0000","last_peered":"2024-09-17T07:25:22.309482+0000","last_clean":"2024-09-17T07:25:22.309482+0000","last_became_active":"2024-09-17T07:23:57.473995+0000","last_became_peered":"2024-09-17T07:23:57.473995+0000","last_unstale":"2024-09-17T07:25:22.309482+0000","last_undegraded":"2024-09-17T07:25:22.309482+0000","last_fullsized":"2024-09-17T07:25:22.309482+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:17.089691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:25:17.089691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:35.172830+0000","scrub_duration":0.0010547029999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"439'35","reported_seq":710,"reported_epoch":511,"state":"active+clean","last_fresh":"2024-09-17T07:25:21.764731+0000","last_change":"2024-09-17T07:23:57.480450+0000","last_active":"2024-09-17T07:25:21.764731+0000","last_peered":"2024-09-17T07:25:21.764731+0000","last_clean":"2024-09-17T07:25:21.764731+0000","last_became_active":"2024-09-17T07:23:57.479217+0000","last_became_peered":"2024-09-17T07:23:57.479217+0000","last_unstale":"2024-09-17T07:25:21.764731+0000","last_undegraded":"2024-09-17T07:25:21.764731+0000","last_fullsized":"2024-09-17T07:25:21.764731+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'25","last_scrub_stamp":"2024-09-17T07:20:39.458854+0000","last_deep_scrub":"294'25","last_deep_scrub_stamp":"2024-09-17T07:16:00.471311+0000","last_clean_scrub_stamp":"2024-09-17T07:20:39.458854+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.005100612,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"436'10","reported_seq":885,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:20.783958+0000","last_change":"2024-09-17T07:25:01.611646+0000","last_active":"2024-09-17T07:25:20.783958+0000","last_peered":"2024-09-17T07:25:20.783958+0000","last_clean":"2024-09-17T07:25:20.783958+0000","last_became_active":"2024-09-17T07:22:13.574012+0000","last_became_peered":"2024-09-17T07:22:13.574012+0000","last_unstale":"2024-09-17T07:25:20.783958+0000","last_undegraded":"2024-09-17T07:25:20.783958+0000","last_fullsized":"2024-09-17T07:25:20.783958+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"436'10","last_scrub_stamp":"2024-09-17T07:25:01.611504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:25:01.611504+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:07.992598+0000","scrub_duration":0.001846009,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":420,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:20.753635+0000","last_change":"2024-09-17T07:25:12.891473+0000","last_active":"2024-09-17T07:25:20.753635+0000","last_peered":"2024-09-17T07:25:20.753635+0000","last_clean":"2024-09-17T07:25:20.753635+0000","last_became_active":"2024-09-17T07:23:57.478429+0000","last_became_peered":"2024-09-17T07:23:57.478429+0000","last_unstale":"2024-09-17T07:25:20.753635+0000","last_undegraded":"2024-09-17T07:25:20.753635+0000","last_fullsized":"2024-09-17T07:25:20.753635+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:12.891393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:25:12.891393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:13.870895+0000","scrub_duration":0.0019788359999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":411,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:19.867941+0000","last_change":"2024-09-17T07:24:29.609437+0000","last_active":"2024-09-17T07:25:19.867941+0000","last_peered":"2024-09-17T07:25:19.867941+0000","last_clean":"2024-09-17T07:25:19.867941+0000","last_became_active":"2024-09-17T07:09:24.403360+0000","last_became_peered":"2024-09-17T07:09:24.403360+0000","last_unstale":"2024-09-17T07:25:19.867941+0000","last_undegraded":"2024-09-17T07:25:19.867941+0000","last_fullsized":"2024-09-17T07:25:19.867941+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:24:29.609321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:10:29.325664+0000","last_clean_scrub_stamp":"2024-09-17T07:24:29.609321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:50.013021+0000","scrub_duration":0.024336869000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"439'27","reported_seq":670,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:19.869113+0000","last_change":"2024-09-17T07:24:28.562815+0000","last_active":"2024-09-17T07:25:19.869113+0000","last_peered":"2024-09-17T07:25:19.869113+0000","last_clean":"2024-09-17T07:25:19.869113+0000","last_became_active":"2024-09-17T07:23:57.486951+0000","last_became_peered":"2024-09-17T07:23:57.486951+0000","last_unstale":"2024-09-17T07:25:19.869113+0000","last_undegraded":"2024-09-17T07:25:19.869113+0000","last_fullsized":"2024-09-17T07:25:19.869113+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'27","last_scrub_stamp":"2024-09-17T07:24:28.562693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:24:28.562693+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:28.808461+0000","scrub_duration":0.0082361729999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":596,"reported_epoch":511,"state":"active+clean","last_fresh":"2024-09-17T07:25:24.557776+0000","last_change":"2024-09-17T07:25:24.557776+0000","last_active":"2024-09-17T07:25:24.557776+0000","last_peered":"2024-09-17T07:25:24.557776+0000","last_clean":"2024-09-17T07:25:24.557776+0000","last_became_active":"2024-09-17T07:24:43.779553+0000","last_became_peered":"2024-09-17T07:24:43.779553+0000","last_unstale":"2024-09-17T07:25:24.557776+0000","last_undegraded":"2024-09-17T07:25:24.557776+0000","last_fullsized":"2024-09-17T07:25:24.557776+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:24.557678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:20:37.435106+0000","last_clean_scrub_stamp":"2024-09-17T07:25:24.557678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:47.573258+0000","scrub_duration":0.0018428959999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":572,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:19.870297+0000","last_change":"2024-09-17T07:25:19.870297+0000","last_active":"2024-09-17T07:25:19.870297+0000","last_peered":"2024-09-17T07:25:19.870297+0000","last_clean":"2024-09-17T07:25:19.870297+0000","last_became_active":"2024-09-17T07:23:57.473629+0000","last_became_peered":"2024-09-17T07:23:57.473629+0000","last_unstale":"2024-09-17T07:25:19.870297+0000","last_undegraded":"2024-09-17T07:25:19.870297+0000","last_fullsized":"2024-09-17T07:25:19.870297+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:19.870196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:25:19.870196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:26.879292+0000","scrub_duration":0.0043293369999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":610,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:19.868303+0000","last_change":"2024-09-17T07:25:09.562025+0000","last_active":"2024-09-17T07:25:19.868303+0000","last_peered":"2024-09-17T07:25:19.868303+0000","last_clean":"2024-09-17T07:25:19.868303+0000","last_became_active":"2024-09-17T07:22:13.587273+0000","last_became_peered":"2024-09-17T07:22:13.587273+0000","last_unstale":"2024-09-17T07:25:19.868303+0000","last_undegraded":"2024-09-17T07:25:19.868303+0000","last_fullsized":"2024-09-17T07:25:19.868303+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:09.561794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:11:18.559877+0000","last_clean_scrub_stamp":"2024-09-17T07:25:09.561794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:21.705259+0000","scrub_duration":0.001513239,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"439'39","reported_seq":496,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:20.753777+0000","last_change":"2024-09-17T07:24:43.799133+0000","last_active":"2024-09-17T07:25:20.753777+0000","last_peered":"2024-09-17T07:25:20.753777+0000","last_clean":"2024-09-17T07:25:20.753777+0000","last_became_active":"2024-09-17T07:24:43.771752+0000","last_became_peered":"2024-09-17T07:24:43.771752+0000","last_unstale":"2024-09-17T07:25:20.753777+0000","last_undegraded":"2024-09-17T07:25:20.753777+0000","last_fullsized":"2024-09-17T07:25:20.753777+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'27","last_scrub_stamp":"2024-09-17T07:21:11.488225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:21:11.488225+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0089238530000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":384,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:20.754047+0000","last_change":"2024-09-17T07:24:23.943586+0000","last_active":"2024-09-17T07:25:20.754047+0000","last_peered":"2024-09-17T07:25:20.754047+0000","last_clean":"2024-09-17T07:25:20.754047+0000","last_became_active":"2024-09-17T07:21:15.348754+0000","last_became_peered":"2024-09-17T07:21:15.348754+0000","last_unstale":"2024-09-17T07:25:20.754047+0000","last_undegraded":"2024-09-17T07:25:20.754047+0000","last_fullsized":"2024-09-17T07:25:20.754047+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:24:23.943437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:24:23.943437+0000","last_clean_scrub_stamp":"2024-09-17T07:24:23.943437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:47.203411+0000","scrub_duration":0.012336285000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":431,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:19.868961+0000","last_change":"2024-09-17T07:25:18.515504+0000","last_active":"2024-09-17T07:25:19.868961+0000","last_peered":"2024-09-17T07:25:19.868961+0000","last_clean":"2024-09-17T07:25:19.868961+0000","last_became_active":"2024-09-17T07:23:57.487706+0000","last_became_peered":"2024-09-17T07:23:57.487706+0000","last_unstale":"2024-09-17T07:25:19.868961+0000","last_undegraded":"2024-09-17T07:25:19.868961+0000","last_fullsized":"2024-09-17T07:25:19.868961+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:18.515421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:25:18.515421+0000","last_clean_scrub_stamp":"2024-09-17T07:25:18.515421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:35.662814+0000","scrub_duration":0.001450956,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":99,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:20.087913+0000","last_change":"2024-09-17T07:24:26.807871+0000","last_active":"2024-09-17T07:25:20.087913+0000","last_peered":"2024-09-17T07:25:20.087913+0000","last_clean":"2024-09-17T07:25:20.087913+0000","last_became_active":"2024-09-17T07:22:13.624057+0000","last_became_peered":"2024-09-17T07:22:13.624057+0000","last_unstale":"2024-09-17T07:25:20.087913+0000","last_undegraded":"2024-09-17T07:25:20.087913+0000","last_fullsized":"2024-09-17T07:25:20.087913+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:24:26.807790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:24:26.807790+0000","last_clean_scrub_stamp":"2024-09-17T07:24:26.807790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:39.305925+0000","scrub_duration":0.001095604,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"439'29","reported_seq":325,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:20.753828+0000","last_change":"2024-09-17T07:25:03.872269+0000","last_active":"2024-09-17T07:25:20.753828+0000","last_peered":"2024-09-17T07:25:20.753828+0000","last_clean":"2024-09-17T07:25:20.753828+0000","last_became_active":"2024-09-17T07:21:15.351512+0000","last_became_peered":"2024-09-17T07:21:15.351512+0000","last_unstale":"2024-09-17T07:25:20.753828+0000","last_undegraded":"2024-09-17T07:25:20.753828+0000","last_fullsized":"2024-09-17T07:25:20.753828+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'29","last_scrub_stamp":"2024-09-17T07:25:03.872175+0000","last_deep_scrub":"86'17","last_deep_scrub_stamp":"2024-09-17T07:03:18.241067+0000","last_clean_scrub_stamp":"2024-09-17T07:25:03.872175+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:09.824307+0000","scrub_duration":0.001972002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":595,"reported_epoch":510,"state":"active+clean","last_fresh":"2024-09-17T07:25:20.895225+0000","last_change":"2024-09-17T07:25:20.895225+0000","last_active":"2024-09-17T07:25:20.895225+0000","last_peered":"2024-09-17T07:25:20.895225+0000","last_clean":"2024-09-17T07:25:20.895225+0000","last_became_active":"2024-09-17T07:22:13.590622+0000","last_became_peered":"2024-09-17T07:22:13.590622+0000","last_unstale":"2024-09-17T07:25:20.895225+0000","last_undegraded":"2024-09-17T07:25:20.895225+0000","last_fullsized":"2024-09-17T07:25:20.895225+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:20.895149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:25:20.895149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:43.686040+0000","scrub_duration":0.0015445599999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":382,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1537,"num_read_kb":1294,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":254,"num_bytes_recovered":5066,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":13004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1027,"ondisk_log_size":1027,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":254,"num_read_kb":249,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1386,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":12,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":4578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":429358,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":155,"num_read_kb":222,"num_write":131,"num_write_kb":1756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1288076,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1430,"ondisk_log_size":1430,"up":33,"acting":33,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698038044,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1113604,"kb_used_data":3340,"kb_used_omap":1,"kb_used_meta":1109886,"kb_avail":92631548,"statfs":{"total":95995035648,"available":94854705152,"internally_reserved":0,"allocated":3420160,"data_stored":2108656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":1136523723},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:24:46 2024","interfaces":[{"interface":"back","average":{"1min":0.95399999999999996,"5min":1.7030000000000001,"15min":1.5129999999999999},"min":{"1min":0.30499999999999999,"5min":0.182,"15min":0.182},"max":{"1min":3.306,"5min":43.905000000000001,"15min":43.905000000000001},"last":0.57699999999999996},{"interface":"front","average":{"1min":1.131,"5min":1.5720000000000001,"15min":1.5640000000000001},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":5.6619999999999999,"5min":44.432000000000002,"15min":44.432000000000002},"last":0.59999999999999998}]},{"osd":1,"last update":"Tue Sep 17 07:25:21 2024","interfaces":[{"interface":"back","average":{"1min":1.1299999999999999,"5min":1.1439999999999999,"15min":1.216},"min":{"1min":0.32900000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":3.948,"5min":12.112,"15min":12.112},"last":0.46500000000000002},{"interface":"front","average":{"1min":1.2749999999999999,"5min":1.097,"15min":1.179},"min":{"1min":0.378,"5min":0.216,"15min":0.216},"max":{"1min":5.6210000000000004,"5min":12.273,"15min":12.273},"last":0.503}]},{"osd":2,"last update":"Tue Sep 17 07:24:46 2024","interfaces":[{"interface":"back","average":{"1min":1.1779999999999999,"5min":0.86799999999999999,"15min":1.0589999999999999},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":4.165,"5min":10.972,"15min":10.972},"last":0.61599999999999999},{"interface":"front","average":{"1min":1.226,"5min":0.97799999999999998,"15min":1.0429999999999999},"min":{"1min":0.35699999999999998,"5min":0.255,"15min":0.224},"max":{"1min":4.4320000000000004,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.42999999999999999}]},{"osd":3,"last update":"Tue Sep 17 07:24:46 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":0.69099999999999995,"15min":1.0109999999999999},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":4.2009999999999996,"5min":4.2009999999999996,"15min":10.122999999999999},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.629,"15min":0.98299999999999998},"min":{"1min":0.313,"5min":0.26700000000000002,"15min":0.26500000000000001},"max":{"1min":3.0630000000000002,"5min":4.1719999999999997,"15min":12.465999999999999},"last":0.48199999999999998}]},{"osd":4,"last update":"Tue Sep 17 07:24:46 2024","interfaces":[{"interface":"back","average":{"1min":1.345,"5min":0.78000000000000003,"15min":1.113},"min":{"1min":0.23699999999999999,"5min":0.22900000000000001,"15min":0.222},"max":{"1min":6.21,"5min":13.316000000000001,"15min":21.140999999999998},"last":0.56299999999999994},{"interface":"front","average":{"1min":1.2589999999999999,"5min":0.82999999999999996,"15min":1.296},"min":{"1min":0.25800000000000001,"5min":0.251,"15min":0.23599999999999999},"max":{"1min":4.2850000000000001,"5min":12.272,"15min":19.420000000000002},"last":0.53200000000000003}]},{"osd":5,"last update":"Tue Sep 17 07:24:57 2024","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":3.0859999999999999,"5min":3.0859999999999999,"15min":3.0859999999999999},"last":0.65400000000000003},{"interface":"front","average":{"1min":1.097,"5min":1.097,"15min":1.097},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":5.7350000000000003,"5min":5.7350000000000003,"15min":5.7350000000000003},"last":0.63600000000000001}]},{"osd":6,"last update":"Tue Sep 17 07:24:46 2024","interfaces":[{"interface":"back","average":{"1min":1.4079999999999999,"5min":1.4950000000000001,"15min":1.6040000000000001},"min":{"1min":0.36399999999999999,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":6.9359999999999999,"5min":60.445999999999998,"15min":60.445999999999998},"last":0.69499999999999995},{"interface":"front","average":{"1min":1.5389999999999999,"5min":1.5580000000000001,"15min":1.71},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":4.3550000000000004,"5min":60.332000000000001,"15min":60.332000000000001},"last":0.67400000000000004}]}]},{"osd":6,"up_from":50,"seq":214748365087,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1035932,"kb_used_data":3620,"kb_used_omap":1,"kb_used_meta":1031934,"kb_avail":92709220,"statfs":{"total":95995035648,"available":94934241280,"internally_reserved":0,"allocated":3706880,"data_stored":2371869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1056700871},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:24:30 2024","interfaces":[{"interface":"back","average":{"1min":8.1080000000000005,"5min":4.0739999999999998,"15min":2.5329999999999999},"min":{"1min":0.22600000000000001,"5min":0.19700000000000001,"15min":0.185},"max":{"1min":110.376,"5min":110.376,"15min":126.902},"last":0.57699999999999996},{"interface":"front","average":{"1min":8.0510000000000002,"5min":3.9180000000000001,"15min":2.3439999999999999},"min":{"1min":0.25700000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":110.164,"5min":110.164,"15min":127.06399999999999},"last":0.45800000000000002}]},{"osd":1,"last update":"Tue Sep 17 07:25:23 2024","interfaces":[{"interface":"back","average":{"1min":1.665,"5min":4.944,"15min":5.6799999999999997},"min":{"1min":0.36099999999999999,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":7.1059999999999999,"5min":199.32400000000001,"15min":199.32400000000001},"last":0.66700000000000004},{"interface":"front","average":{"1min":1.4730000000000001,"5min":5.0579999999999998,"15min":5.9619999999999997},"min":{"1min":0.312,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":8.9399999999999995,"5min":198.88499999999999,"15min":198.88499999999999},"last":0.443}]},{"osd":2,"last update":"Tue Sep 17 07:24:30 2024","interfaces":[{"interface":"back","average":{"1min":1.3340000000000001,"5min":1.677,"15min":1.236},"min":{"1min":0.28499999999999998,"5min":0.23300000000000001,"15min":0.219},"max":{"1min":10.930999999999999,"5min":36.289000000000001,"15min":36.289000000000001},"last":0.48599999999999999},{"interface":"front","average":{"1min":1.167,"5min":1.546,"15min":1.0960000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.22900000000000001},"max":{"1min":4.2190000000000003,"5min":36.326999999999998,"15min":36.326999999999998},"last":0.63600000000000001}]},{"osd":3,"last update":"Tue Sep 17 07:24:30 2024","interfaces":[{"interface":"back","average":{"1min":1.024,"5min":0.74399999999999999,"15min":0.91600000000000004},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.24399999999999999},"max":{"1min":6.3289999999999997,"5min":6.3289999999999997,"15min":6.8639999999999999},"last":0.56299999999999994},{"interface":"front","average":{"1min":1.3400000000000001,"5min":0.95699999999999996,"15min":1.0860000000000001},"min":{"1min":0.32500000000000001,"5min":0.27600000000000002,"15min":0.27500000000000002},"max":{"1min":10.999000000000001,"5min":10.999000000000001,"15min":10.999000000000001},"last":0.53400000000000003}]},{"osd":4,"last update":"Tue Sep 17 07:24:30 2024","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":0.91300000000000003,"15min":2.3370000000000002},"min":{"1min":0.26000000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":3.9790000000000001,"5min":9.5419999999999998,"15min":260.77199999999999},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.69499999999999995,"15min":2.3359999999999999},"min":{"1min":0.308,"5min":0.26100000000000001,"15min":0.19800000000000001},"max":{"1min":3.2090000000000001,"5min":3.609,"15min":260.69099999999997},"last":0.40100000000000002}]},{"osd":5,"last update":"Tue Sep 17 07:24:56 2024","interfaces":[{"interface":"back","average":{"1min":2.1280000000000001,"5min":2.1280000000000001,"15min":2.1280000000000001},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":13.551,"5min":13.551,"15min":13.551},"last":0.42399999999999999},{"interface":"front","average":{"1min":2.387,"5min":2.387,"15min":2.387},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":8.1720000000000006,"5min":8.1720000000000006,"15min":8.1720000000000006},"last":0.5}]},{"osd":7,"last update":"Tue Sep 17 07:24:51 2024","interfaces":[{"interface":"back","average":{"1min":1.948,"5min":1.0229999999999999,"15min":1.784},"min":{"1min":0.33600000000000002,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":9.1199999999999992,"5min":9.1199999999999992,"15min":131.95699999999999},"last":0.51600000000000001},{"interface":"front","average":{"1min":2.206,"5min":1.149,"15min":1.6950000000000001},"min":{"1min":0.31900000000000001,"5min":0.26900000000000002,"15min":0.252},"max":{"1min":7.077,"5min":9.5850000000000009,"15min":131.833},"last":0.59499999999999997}]}]},{"osd":5,"up_from":476,"seq":2044404432915,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":327584,"kb_used_data":193580,"kb_used_omap":4,"kb_used_meta":133627,"kb_avail":93417568,"statfs":{"total":95995035648,"available":95659589632,"internally_reserved":0,"allocated":198225920,"data_stored":197071165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4441,"internal_metadata":136834727},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:25:00 2024","interfaces":[{"interface":"back","average":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":3.4220000000000002,"5min":3.4220000000000002,"15min":3.4220000000000002},"last":0.63500000000000001},{"interface":"front","average":{"1min":1.5960000000000001,"5min":1.5960000000000001,"15min":1.5960000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":6.7619999999999996,"5min":6.7619999999999996,"15min":6.7619999999999996},"last":0.65500000000000003}]},{"osd":1,"last update":"Tue Sep 17 07:25:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":3.476,"5min":3.476,"15min":3.476},"last":0.56399999999999995},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":6.665,"5min":6.665,"15min":6.665},"last":0.58699999999999997}]},{"osd":2,"last update":"Tue Sep 17 07:25:00 2024","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.2190000000000001,"15min":1.2190000000000001},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":3.544,"5min":3.544,"15min":3.544},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":2.8359999999999999,"5min":2.8359999999999999,"15min":2.8359999999999999},"last":0.46800000000000003}]},{"osd":3,"last update":"Tue Sep 17 07:25:00 2024","interfaces":[{"interface":"back","average":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":3.7040000000000002,"5min":3.7040000000000002,"15min":3.7040000000000002},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":2.8719999999999999,"5min":2.8719999999999999,"15min":2.8719999999999999},"last":0.67600000000000005}]},{"osd":4,"last update":"Tue Sep 17 07:25:00 2024","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":3.4399999999999999,"5min":3.4399999999999999,"15min":3.4399999999999999},"last":0.70599999999999996},{"interface":"front","average":{"1min":1.26,"5min":1.26,"15min":1.26},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":3.0139999999999998,"5min":3.0139999999999998,"15min":3.0139999999999998},"last":0.79400000000000004}]},{"osd":6,"last update":"Tue Sep 17 07:25:00 2024","interfaces":[{"interface":"back","average":{"1min":1.77,"5min":1.77,"15min":1.77},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":10.92,"5min":10.92,"15min":10.92},"last":0.69099999999999995},{"interface":"front","average":{"1min":1.526,"5min":1.526,"15min":1.526},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":10.202999999999999,"5min":10.202999999999999,"15min":10.202999999999999},"last":0.66700000000000004}]},{"osd":7,"last update":"Tue Sep 17 07:25:00 2024","interfaces":[{"interface":"back","average":{"1min":3.3820000000000001,"5min":3.3820000000000001,"15min":3.3820000000000001},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":45.555,"5min":45.555,"15min":45.555},"last":0.76900000000000002},{"interface":"front","average":{"1min":3.6920000000000002,"5min":3.6920000000000002,"15min":3.6920000000000002},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":47.213999999999999,"5min":47.213999999999999,"15min":47.213999999999999},"last":0.72699999999999998}]}]},{"osd":4,"up_from":33,"seq":141733921066,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1004992,"kb_used_data":3020,"kb_used_omap":1,"kb_used_meta":1001598,"kb_avail":92740160,"statfs":{"total":95995035648,"available":94965923840,"internally_reserved":0,"allocated":3092480,"data_stored":1780316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1025636806},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:24:30 2024","interfaces":[{"interface":"back","average":{"1min":1.1699999999999999,"5min":1.869,"15min":1.4299999999999999},"min":{"1min":0.28399999999999997,"5min":0.20300000000000001,"15min":0.20000000000000001},"max":{"1min":6.7290000000000001,"5min":39.473999999999997,"15min":39.473999999999997},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.97299999999999998,"5min":1.8899999999999999,"15min":1.5229999999999999},"min":{"1min":0.21299999999999999,"5min":0.20599999999999999,"15min":0.20499999999999999},"max":{"1min":3.1349999999999998,"5min":39.654000000000003,"15min":39.654000000000003},"last":0.39800000000000002}]},{"osd":1,"last update":"Tue Sep 17 07:25:21 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":2.0790000000000002,"15min":2.613},"min":{"1min":0.318,"5min":0.221,"15min":0.221},"max":{"1min":2.1589999999999998,"5min":39.439,"15min":39.439},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.91200000000000003,"5min":2.2480000000000002,"15min":2.6859999999999999},"min":{"1min":0.36699999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":3.4510000000000001,"5min":39.325000000000003,"15min":39.325000000000003},"last":0.442}]},{"osd":2,"last update":"Tue Sep 17 07:24:30 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.93400000000000005,"15min":1.081},"min":{"1min":0.26300000000000001,"5min":0.24399999999999999,"15min":0.23499999999999999},"max":{"1min":6.8129999999999997,"5min":10.238,"15min":10.238},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.97899999999999998,"5min":0.89600000000000002,"15min":1.0740000000000001},"min":{"1min":0.309,"5min":0.26600000000000001,"15min":0.23400000000000001},"max":{"1min":2.9249999999999998,"5min":10.19,"15min":16.655999999999999},"last":0.49299999999999999}]},{"osd":3,"last update":"Tue Sep 17 07:24:30 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.70799999999999996,"15min":0.94799999999999995},"min":{"1min":0.28299999999999997,"5min":0.27600000000000002,"15min":0.25800000000000001},"max":{"1min":2.8929999999999998,"5min":2.8929999999999998,"15min":15.433999999999999},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.72499999999999998,"15min":0.98899999999999999},"min":{"1min":0.32000000000000001,"5min":0.28100000000000003,"15min":0.27100000000000002},"max":{"1min":2.3300000000000001,"5min":8.7360000000000007,"15min":16.111000000000001},"last":0.46400000000000002}]},{"osd":5,"last update":"Tue Sep 17 07:25:00 2024","interfaces":[{"interface":"back","average":{"1min":1.4359999999999999,"5min":1.4359999999999999,"15min":1.4359999999999999},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":13.128,"5min":13.128,"15min":13.128},"last":0.53100000000000003},{"interface":"front","average":{"1min":1.752,"5min":1.752,"15min":1.752},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":13.218999999999999,"5min":13.218999999999999,"15min":13.218999999999999},"last":0.54100000000000004}]},{"osd":6,"last update":"Tue Sep 17 07:24:30 2024","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":0.81999999999999995,"15min":1.0940000000000001},"min":{"1min":0.31900000000000001,"5min":0.23400000000000001,"15min":0.218},"max":{"1min":4.4989999999999997,"5min":4.7729999999999997,"15min":8.9619999999999997},"last":0.41899999999999998},{"interface":"front","average":{"1min":1.006,"5min":0.90000000000000002,"15min":1.208},"min":{"1min":0.26000000000000001,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":6.782,"5min":6.782,"15min":10.051},"last":0.51000000000000001}]},{"osd":7,"last update":"Tue Sep 17 07:24:45 2024","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":0.98899999999999999,"15min":1.2410000000000001},"min":{"1min":0.318,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":6.2249999999999996,"5min":6.2249999999999996,"15min":7.1550000000000002},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.98899999999999999,"5min":0.78600000000000003,"15min":1.1910000000000001},"min":{"1min":0.28599999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":2.5739999999999998,"5min":4.7439999999999998,"15min":9.0030000000000001},"last":0.56399999999999995}]}]},{"osd":3,"up_from":0,"seq":115964117292,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":18,"seq":77309411634,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1121476,"kb_used_data":3280,"kb_used_omap":1,"kb_used_meta":1117822,"kb_avail":92623676,"statfs":{"total":95995035648,"available":94846644224,"internally_reserved":0,"allocated":3358720,"data_stored":2043848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1144650182},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:25:06 2024","interfaces":[{"interface":"back","average":{"1min":1.966,"5min":3.5059999999999998,"15min":2.6699999999999999},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.154},"max":{"1min":25.98,"5min":69.135999999999996,"15min":131.09800000000001},"last":0.313},{"interface":"front","average":{"1min":2.073,"5min":3.73,"15min":2.6949999999999998},"min":{"1min":0.25600000000000001,"5min":0.17999999999999999,"15min":0.17599999999999999},"max":{"1min":24.314,"5min":69.033000000000001,"15min":131.24700000000001},"last":0.24099999999999999}]},{"osd":1,"last update":"Tue Sep 17 07:24:22 2024","interfaces":[{"interface":"back","average":{"1min":1.968,"5min":2.8860000000000001,"15min":2.274},"min":{"1min":0.20999999999999999,"5min":0.184,"15min":0.184},"max":{"1min":21.23,"5min":91.686999999999998,"15min":91.686999999999998},"last":0.33000000000000002},{"interface":"front","average":{"1min":2.2599999999999998,"5min":3.1989999999999998,"15min":2.6080000000000001},"min":{"1min":0.223,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":21.024000000000001,"5min":93.744,"15min":93.744},"last":0.29099999999999998}]},{"osd":3,"last update":"Tue Sep 17 07:25:13 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.61099999999999999,"15min":0.67800000000000005},"min":{"1min":0.251,"5min":0.20499999999999999,"15min":0.20100000000000001},"max":{"1min":0.99099999999999999,"5min":3.1880000000000002,"15min":23.43},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.69499999999999995,"5min":1.0009999999999999,"15min":0.78500000000000003},"min":{"1min":0.29799999999999999,"5min":0.22500000000000001,"15min":0.22},"max":{"1min":2.4769999999999999,"5min":16.215,"15min":23.408000000000001},"last":0.34599999999999997}]},{"osd":4,"last update":"Tue Sep 17 07:24:40 2024","interfaces":[{"interface":"back","average":{"1min":2.2839999999999998,"5min":1.0920000000000001,"15min":1.3700000000000001},"min":{"1min":0.314,"5min":0.254,"15min":0.24399999999999999},"max":{"1min":25.225999999999999,"5min":25.225999999999999,"15min":75.870999999999995},"last":0.61199999999999999},{"interface":"front","average":{"1min":2.3559999999999999,"5min":1.149,"15min":1.47},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.26300000000000001},"max":{"1min":25.268999999999998,"5min":25.268999999999998,"15min":75.888999999999996},"last":0.63100000000000001}]},{"osd":5,"last update":"Tue Sep 17 07:24:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":1.9610000000000001,"5min":1.9610000000000001,"15min":1.9610000000000001},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":3.907,"5min":3.907,"15min":3.907},"last":0.55100000000000005}]},{"osd":6,"last update":"Tue Sep 17 07:25:13 2024","interfaces":[{"interface":"back","average":{"1min":2.8969999999999998,"5min":2.298,"15min":1.8169999999999999},"min":{"1min":0.32800000000000001,"5min":0.29699999999999999,"15min":0.27500000000000002},"max":{"1min":24.936,"5min":65.143000000000001,"15min":76.081999999999994},"last":0.65700000000000003},{"interface":"front","average":{"1min":2.2839999999999998,"5min":2.1680000000000001,"15min":1.871},"min":{"1min":0.40899999999999997,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":24.884,"5min":65.241,"15min":76.244},"last":0.57099999999999995}]},{"osd":7,"last update":"Tue Sep 17 07:25:06 2024","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.83699999999999997,"15min":1.3520000000000001},"min":{"1min":0.34799999999999998,"5min":0.314,"15min":0.314},"max":{"1min":2.0790000000000002,"5min":4.2720000000000002,"15min":94.269999999999996},"last":0.64500000000000002},{"interface":"front","average":{"1min":1.016,"5min":0.95899999999999996,"15min":1.395},"min":{"1min":0.34499999999999997,"5min":0.32900000000000001,"15min":0.309},"max":{"1min":4.4470000000000001,"5min":4.4470000000000001,"15min":94.634},"last":0.59699999999999998}]}]},{"osd":1,"up_from":413,"seq":1773821493299,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":537116,"kb_used_data":3048,"kb_used_omap":4,"kb_used_meta":533691,"kb_avail":93208036,"statfs":{"total":95995035648,"available":95445028864,"internally_reserved":0,"allocated":3121152,"data_stored":1782222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4439,"internal_metadata":546500265},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:24:22 2024","interfaces":[{"interface":"back","average":{"1min":8.1750000000000007,"5min":2.7909999999999999,"15min":2.044},"min":{"1min":0.219,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":120.56399999999999,"5min":120.56399999999999,"15min":120.56399999999999},"last":0.33900000000000002},{"interface":"front","average":{"1min":8.6140000000000008,"5min":2.7389999999999999,"15min":1.839},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":120.746,"5min":120.746,"15min":120.746},"last":0.373}]},{"osd":2,"last update":"Tue Sep 17 07:24:22 2024","interfaces":[{"interface":"back","average":{"1min":1.133,"5min":1.1679999999999999,"15min":1.2969999999999999},"min":{"1min":0.23300000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":6.7300000000000004,"5min":6.7300000000000004,"15min":6.7300000000000004},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.96999999999999997,"15min":1.0760000000000001},"min":{"1min":0.26800000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":1.5860000000000001,"5min":3.919,"15min":3.919},"last":0.35699999999999998}]},{"osd":3,"last update":"Tue Sep 17 07:24:22 2024","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.78500000000000003,"15min":0.80900000000000005},"min":{"1min":0.30599999999999999,"5min":0.253,"15min":0.253},"max":{"1min":3.4239999999999999,"5min":5.9290000000000003,"15min":5.9290000000000003},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.81599999999999995,"15min":0.85799999999999998},"min":{"1min":0.27200000000000002,"5min":0.25,"15min":0.25},"max":{"1min":2.919,"5min":4.1340000000000003,"15min":4.1340000000000003},"last":0.40400000000000003}]},{"osd":4,"last update":"Tue Sep 17 07:24:22 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":4.7809999999999997,"15min":6.4980000000000002},"min":{"1min":0.311,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":3.1970000000000001,"5min":116.88200000000001,"15min":116.88200000000001},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.78700000000000003,"5min":4.7530000000000001,"15min":6.5350000000000001},"min":{"1min":0.34000000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":3.456,"5min":116.98399999999999,"15min":116.98399999999999},"last":0.61499999999999999}]},{"osd":5,"last update":"Tue Sep 17 07:24:57 2024","interfaces":[{"interface":"back","average":{"1min":1.5369999999999999,"5min":1.5369999999999999,"15min":1.5369999999999999},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":9.7409999999999997,"5min":9.7409999999999997,"15min":9.7409999999999997},"last":0.71099999999999997},{"interface":"front","average":{"1min":1.542,"5min":1.542,"15min":1.542},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":11.538,"5min":11.538,"15min":11.538},"last":0.627}]},{"osd":6,"last update":"Tue Sep 17 07:24:22 2024","interfaces":[{"interface":"back","average":{"1min":1.0429999999999999,"5min":2.1110000000000002,"15min":1.657},"min":{"1min":0.35299999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":4.2279999999999998,"5min":56.686999999999998,"15min":56.686999999999998},"last":0.65200000000000002},{"interface":"front","average":{"1min":1.21,"5min":2.056,"15min":1.5700000000000001},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":4.1559999999999997,"5min":56.420999999999999,"15min":56.420999999999999},"last":0.66400000000000003}]},{"osd":7,"last update":"Tue Sep 17 07:24:22 2024","interfaces":[{"interface":"back","average":{"1min":1.077,"5min":1.0269999999999999,"15min":1.054},"min":{"1min":0.35199999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":2.9630000000000001,"5min":4.2389999999999999,"15min":4.2389999999999999},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.626,"5min":0.93100000000000005,"15min":1.012},"min":{"1min":0.36899999999999999,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.5640000000000001,"5min":4.2839999999999998,"15min":4.2839999999999998},"last":0.64000000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738682,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1698400,"kb_used_data":395480,"kb_used_omap":1,"kb_used_meta":1302910,"kb_avail":92046752,"statfs":{"total":95995035648,"available":94255874048,"internally_reserved":0,"allocated":404971520,"data_stored":403648112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1334180294},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 17 07:25:22 2024","interfaces":[{"interface":"back","average":{"1min":3.585,"5min":1.9419999999999999,"15min":1.986},"min":{"1min":0.20599999999999999,"5min":0.156,"15min":0.156},"max":{"1min":53.731000000000002,"5min":53.731000000000002,"15min":53.731000000000002},"last":0.32900000000000001},{"interface":"front","average":{"1min":4.0179999999999998,"5min":1.827,"15min":1.6140000000000001},"min":{"1min":0.217,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":55.789000000000001,"5min":55.789000000000001,"15min":55.789000000000001},"last":0.35299999999999998}]},{"osd":2,"last update":"Tue Sep 17 07:24:59 2024","interfaces":[{"interface":"back","average":{"1min":3.9529999999999998,"5min":2.4750000000000001,"15min":1.256},"min":{"1min":0.21299999999999999,"5min":0.19400000000000001,"15min":0.155},"max":{"1min":55.756,"5min":62.094999999999999,"15min":62.094999999999999},"last":0.39500000000000002},{"interface":"front","average":{"1min":3.9180000000000001,"5min":2.476,"15min":1.242},"min":{"1min":0.255,"5min":0.19,"15min":0.183},"max":{"1min":57.854999999999997,"5min":59.848999999999997,"15min":59.848999999999997},"last":0.31}]},{"osd":3,"last update":"Tue Sep 17 07:24:35 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.74299999999999999,"15min":0.69099999999999995},"min":{"1min":0.27700000000000002,"5min":0.23200000000000001,"15min":0.21199999999999999},"max":{"1min":1.1910000000000001,"5min":4.6829999999999998,"15min":5.5880000000000001},"last":0.378},{"interface":"front","average":{"1min":0.498,"5min":0.80400000000000005,"15min":0.67400000000000004},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20399999999999999},"max":{"1min":1.024,"5min":5.1150000000000002,"15min":5.1150000000000002},"last":0.36699999999999999}]},{"osd":4,"last update":"Tue Sep 17 07:24:35 2024","interfaces":[{"interface":"back","average":{"1min":1.6990000000000001,"5min":1.208,"15min":1.1459999999999999},"min":{"1min":0.29199999999999998,"5min":0.27400000000000002,"15min":0.23400000000000001},"max":{"1min":10.720000000000001,"5min":10.720000000000001,"15min":19.146999999999998},"last":0.89400000000000002},{"interface":"front","average":{"1min":1.8320000000000001,"5min":1.123,"15min":1.165},"min":{"1min":0.32700000000000001,"5min":0.26600000000000001,"15min":0.26200000000000001},"max":{"1min":10.666,"5min":10.666,"15min":19.158999999999999},"last":0.65500000000000003}]},{"osd":5,"last update":"Tue Sep 17 07:24:59 2024","interfaces":[{"interface":"back","average":{"1min":1.056,"5min":1.056,"15min":1.056},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":3.79,"5min":3.79,"15min":3.79},"last":0.60899999999999999},{"interface":"front","average":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":3.2530000000000001,"5min":3.2530000000000001,"15min":3.2530000000000001},"last":0.84999999999999998}]},{"osd":6,"last update":"Tue Sep 17 07:24:35 2024","interfaces":[{"interface":"back","average":{"1min":1.3009999999999999,"5min":2.2450000000000001,"15min":1.8879999999999999},"min":{"1min":0.34799999999999998,"5min":0.28999999999999998,"15min":0.28199999999999997},"max":{"1min":3.923,"5min":67.873000000000005,"15min":147.13200000000001},"last":0.71099999999999997},{"interface":"front","average":{"1min":1.2809999999999999,"5min":2.359,"15min":1.911},"min":{"1min":0.311,"5min":0.311,"15min":0.30199999999999999},"max":{"1min":4.0229999999999997,"5min":67.914000000000001,"15min":147.172},"last":0.875}]},{"osd":7,"last update":"Tue Sep 17 07:24:35 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.88700000000000001,"15min":1.1379999999999999},"min":{"1min":0.35799999999999998,"5min":0.318,"15min":0.29599999999999999},"max":{"1min":2.4820000000000002,"5min":3.843,"15min":6.3300000000000001},"last":0.67100000000000004},{"interface":"front","average":{"1min":1.004,"5min":1.099,"15min":1.218},"min":{"1min":0.36799999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":2.6819999999999999,"5min":6.7300000000000004,"15min":7.2830000000000004},"last":0.73799999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T07:25:30.907 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-17T07:25:30.908 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T07:25:30.908 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-17T07:25:30.908 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-17T07:25:30.919 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:30 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:25:30.919 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:30 smithi135 ceph-mon[33238]: 4.1e scrub starts 2024-09-17T07:25:30.919 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:30 smithi135 ceph-mon[33238]: 4.1e scrub ok 2024-09-17T07:25:30.919 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:30 smithi135 ceph-mon[33238]: from='client.16269 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:25:30.919 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:30 smithi135 ceph-mon[33238]: pgmap v1118: 142 pgs: 142 active+clean; 425 KiB data, 6.5 GiB used, 619 GiB / 626 GiB avail 2024-09-17T07:25:30.919 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:30 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/4240204427' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T07:25:30.919 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:30 smithi135 ceph-mon[33238]: osdmap e513: 8 total, 8 up, 7 in 2024-09-17T07:25:30.919 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:30 smithi135 ceph-mon[33238]: 6.4 scrub starts 2024-09-17T07:25:30.919 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:30 smithi135 ceph-mon[33238]: 6.4 scrub ok 2024-09-17T07:25:30.919 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:30 smithi135 ceph-mon[33238]: 2.8 scrub starts 2024-09-17T07:25:30.919 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:30 smithi135 ceph-mon[33238]: 2.8 scrub ok 2024-09-17T07:25:30.920 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:30 smithi135 ceph-mon[33238]: 1.4 deep-scrub starts 2024-09-17T07:25:30.920 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:30 smithi135 ceph-mon[33238]: 1.4 deep-scrub ok 2024-09-17T07:25:30.921 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-17T07:25:31.104 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:25:31.117 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T07:25:31.126 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:30 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:25:31.126 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:30 smithi100 ceph-mon[29092]: 4.1e scrub starts 2024-09-17T07:25:31.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:30 smithi100 ceph-mon[29092]: 4.1e scrub ok 2024-09-17T07:25:31.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:30 smithi100 ceph-mon[29092]: from='client.16269 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:25:31.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:30 smithi100 ceph-mon[29092]: pgmap v1118: 142 pgs: 142 active+clean; 425 KiB data, 6.5 GiB used, 619 GiB / 626 GiB avail 2024-09-17T07:25:31.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:30 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/4240204427' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T07:25:31.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:30 smithi100 ceph-mon[29092]: osdmap e513: 8 total, 8 up, 7 in 2024-09-17T07:25:31.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:30 smithi100 ceph-mon[29092]: 6.4 scrub starts 2024-09-17T07:25:31.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:30 smithi100 ceph-mon[29092]: 6.4 scrub ok 2024-09-17T07:25:31.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:30 smithi100 ceph-mon[29092]: 2.8 scrub starts 2024-09-17T07:25:31.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:30 smithi100 ceph-mon[29092]: 2.8 scrub ok 2024-09-17T07:25:31.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:30 smithi100 ceph-mon[29092]: 1.4 deep-scrub starts 2024-09-17T07:25:31.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:30 smithi100 ceph-mon[29092]: 1.4 deep-scrub ok 2024-09-17T07:25:31.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:30 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:25:31.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:30 smithi100 ceph-mon[39462]: 4.1e scrub starts 2024-09-17T07:25:31.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:30 smithi100 ceph-mon[39462]: 4.1e scrub ok 2024-09-17T07:25:31.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:30 smithi100 ceph-mon[39462]: from='client.16269 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:25:31.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:30 smithi100 ceph-mon[39462]: pgmap v1118: 142 pgs: 142 active+clean; 425 KiB data, 6.5 GiB used, 619 GiB / 626 GiB avail 2024-09-17T07:25:31.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:30 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/4240204427' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T07:25:31.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:30 smithi100 ceph-mon[39462]: osdmap e513: 8 total, 8 up, 7 in 2024-09-17T07:25:31.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:30 smithi100 ceph-mon[39462]: 6.4 scrub starts 2024-09-17T07:25:31.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:30 smithi100 ceph-mon[39462]: 6.4 scrub ok 2024-09-17T07:25:31.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:30 smithi100 ceph-mon[39462]: 2.8 scrub starts 2024-09-17T07:25:31.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:30 smithi100 ceph-mon[39462]: 2.8 scrub ok 2024-09-17T07:25:31.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:30 smithi100 ceph-mon[39462]: 1.4 deep-scrub starts 2024-09-17T07:25:31.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:30 smithi100 ceph-mon[39462]: 1.4 deep-scrub ok 2024-09-17T07:25:31.876 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:31 smithi100 ceph-mon[39462]: 2.a scrub starts 2024-09-17T07:25:31.876 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:31 smithi100 ceph-mon[39462]: 2.a scrub ok 2024-09-17T07:25:31.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:31 smithi100 ceph-mon[39462]: 2.c scrub starts 2024-09-17T07:25:31.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:31 smithi100 ceph-mon[39462]: 2.c scrub ok 2024-09-17T07:25:31.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:31 smithi100 ceph-mon[29092]: 2.a scrub starts 2024-09-17T07:25:31.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:31 smithi100 ceph-mon[29092]: 2.a scrub ok 2024-09-17T07:25:31.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:31 smithi100 ceph-mon[29092]: 2.c scrub starts 2024-09-17T07:25:31.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:31 smithi100 ceph-mon[29092]: 2.c scrub ok 2024-09-17T07:25:32.190 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:31 smithi135 ceph-mon[33238]: 2.a scrub starts 2024-09-17T07:25:32.191 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:31 smithi135 ceph-mon[33238]: 2.a scrub ok 2024-09-17T07:25:32.191 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:31 smithi135 ceph-mon[33238]: 2.c scrub starts 2024-09-17T07:25:32.191 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:31 smithi135 ceph-mon[33238]: 2.c scrub ok 2024-09-17T07:25:32.735 INFO:teuthology.orchestra.run.smithi135.stdout:ERROR: (22) Invalid argument 2024-09-17T07:25:32.735 INFO:teuthology.orchestra.run.smithi135.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T07:25:33.037 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:32 smithi100 ceph-mon[29092]: 4.1f deep-scrub starts 2024-09-17T07:25:33.037 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:32 smithi100 ceph-mon[29092]: 4.1f deep-scrub ok 2024-09-17T07:25:33.038 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:32 smithi100 ceph-mon[29092]: pgmap v1120: 142 pgs: 142 active+clean; 693 KiB data, 6.5 GiB used, 619 GiB / 626 GiB avail 2024-09-17T07:25:33.038 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:32 smithi100 ceph-mon[29092]: 1.8 scrub starts 2024-09-17T07:25:33.038 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:32 smithi100 ceph-mon[29092]: 1.8 scrub ok 2024-09-17T07:25:33.038 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:32 smithi100 ceph-mon[39462]: 4.1f deep-scrub starts 2024-09-17T07:25:33.038 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:32 smithi100 ceph-mon[39462]: 4.1f deep-scrub ok 2024-09-17T07:25:33.038 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:32 smithi100 ceph-mon[39462]: pgmap v1120: 142 pgs: 142 active+clean; 693 KiB data, 6.5 GiB used, 619 GiB / 626 GiB avail 2024-09-17T07:25:33.038 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:32 smithi100 ceph-mon[39462]: 1.8 scrub starts 2024-09-17T07:25:33.038 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:32 smithi100 ceph-mon[39462]: 1.8 scrub ok 2024-09-17T07:25:33.039 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T07:25:33.039 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-17T07:25:33.039 INFO:teuthology.orchestra.run.smithi100.stdout: "epoch": 513, 2024-09-17T07:25:33.039 INFO:teuthology.orchestra.run.smithi100.stdout: "fsid": "e848267c-74c1-11ef-bceb-c7b262605968", 2024-09-17T07:25:33.039 INFO:teuthology.orchestra.run.smithi100.stdout: "created": "2024-09-17T06:57:05.010532+0000", 2024-09-17T07:25:33.039 INFO:teuthology.orchestra.run.smithi100.stdout: "modified": "2024-09-17T07:25:29.777227+0000", 2024-09-17T07:25:33.039 INFO:teuthology.orchestra.run.smithi100.stdout: "last_up_change": "2024-09-17T07:23:56.453211+0000", 2024-09-17T07:25:33.039 INFO:teuthology.orchestra.run.smithi100.stdout: "last_in_change": "2024-09-17T07:22:12.524102+0000", 2024-09-17T07:25:33.040 INFO:teuthology.orchestra.run.smithi100.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-17T07:25:33.040 INFO:teuthology.orchestra.run.smithi100.stdout: "flags_num": 5799936, 2024-09-17T07:25:33.040 INFO:teuthology.orchestra.run.smithi100.stdout: "flags_set": [ 2024-09-17T07:25:33.040 INFO:teuthology.orchestra.run.smithi100.stdout: "pglog_hardlimit", 2024-09-17T07:25:33.040 INFO:teuthology.orchestra.run.smithi100.stdout: "purged_snapdirs", 2024-09-17T07:25:33.040 INFO:teuthology.orchestra.run.smithi100.stdout: "recovery_deletes", 2024-09-17T07:25:33.040 INFO:teuthology.orchestra.run.smithi100.stdout: "sortbitwise" 2024-09-17T07:25:33.040 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-17T07:25:33.040 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_version": 20, 2024-09-17T07:25:33.041 INFO:teuthology.orchestra.run.smithi100.stdout: "full_ratio": 0.94999998807907104, 2024-09-17T07:25:33.041 INFO:teuthology.orchestra.run.smithi100.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-17T07:25:33.041 INFO:teuthology.orchestra.run.smithi100.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-17T07:25:33.041 INFO:teuthology.orchestra.run.smithi100.stdout: "cluster_snapshot": "", 2024-09-17T07:25:33.041 INFO:teuthology.orchestra.run.smithi100.stdout: "pool_max": 11, 2024-09-17T07:25:33.041 INFO:teuthology.orchestra.run.smithi100.stdout: "max_osd": 8, 2024-09-17T07:25:33.041 INFO:teuthology.orchestra.run.smithi100.stdout: "require_min_compat_client": "luminous", 2024-09-17T07:25:33.041 INFO:teuthology.orchestra.run.smithi100.stdout: "min_compat_client": "luminous", 2024-09-17T07:25:33.042 INFO:teuthology.orchestra.run.smithi100.stdout: "require_osd_release": "reef", 2024-09-17T07:25:33.042 INFO:teuthology.orchestra.run.smithi100.stdout: "allow_crimson": false, 2024-09-17T07:25:33.042 INFO:teuthology.orchestra.run.smithi100.stdout: "pools": [ 2024-09-17T07:25:33.042 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:25:33.042 INFO:teuthology.orchestra.run.smithi100.stdout: "pool": 1, 2024-09-17T07:25:33.042 INFO:teuthology.orchestra.run.smithi100.stdout: "pool_name": ".mgr", 2024-09-17T07:25:33.042 INFO:teuthology.orchestra.run.smithi100.stdout: "create_time": "2024-09-17T07:00:08.816136+0000", 2024-09-17T07:25:33.042 INFO:teuthology.orchestra.run.smithi100.stdout: "flags": 1, 2024-09-17T07:25:33.042 INFO:teuthology.orchestra.run.smithi100.stdout: "flags_names": "hashpspool", 2024-09-17T07:25:33.042 INFO:teuthology.orchestra.run.smithi100.stdout: "type": 1, 2024-09-17T07:25:33.042 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 3, 2024-09-17T07:25:33.043 INFO:teuthology.orchestra.run.smithi100.stdout: "min_size": 2, 2024-09-17T07:25:33.043 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_rule": 0, 2024-09-17T07:25:33.043 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_count": 0, 2024-09-17T07:25:33.043 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_target": 0, 2024-09-17T07:25:33.043 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T07:25:33.043 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T07:25:33.043 INFO:teuthology.orchestra.run.smithi100.stdout: "object_hash": 2, 2024-09-17T07:25:33.043 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_autoscale_mode": "off", 2024-09-17T07:25:33.043 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num": 11, 2024-09-17T07:25:33.043 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_placement_num": 11, 2024-09-17T07:25:33.043 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_placement_num_target": 11, 2024-09-17T07:25:33.043 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num_target": 11, 2024-09-17T07:25:33.043 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num_pending": 11, 2024-09-17T07:25:33.044 INFO:teuthology.orchestra.run.smithi100.stdout: "last_pg_merge_meta": { 2024-09-17T07:25:33.044 INFO:teuthology.orchestra.run.smithi100.stdout: "source_pgid": "0.0", 2024-09-17T07:25:33.044 INFO:teuthology.orchestra.run.smithi100.stdout: "ready_epoch": 0, 2024-09-17T07:25:33.044 INFO:teuthology.orchestra.run.smithi100.stdout: "last_epoch_started": 0, 2024-09-17T07:25:33.044 INFO:teuthology.orchestra.run.smithi100.stdout: "last_epoch_clean": 0, 2024-09-17T07:25:33.044 INFO:teuthology.orchestra.run.smithi100.stdout: "source_version": "0'0", 2024-09-17T07:25:33.044 INFO:teuthology.orchestra.run.smithi100.stdout: "target_version": "0'0" 2024-09-17T07:25:33.044 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:25:33.045 INFO:teuthology.orchestra.run.smithi100.stdout: "last_change": "375", 2024-09-17T07:25:33.045 INFO:teuthology.orchestra.run.smithi100.stdout: "last_force_op_resend": "0", 2024-09-17T07:25:33.045 INFO:teuthology.orchestra.run.smithi100.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-17T07:25:33.045 INFO:teuthology.orchestra.run.smithi100.stdout: "last_force_op_resend_preluminous": "374", 2024-09-17T07:25:33.045 INFO:teuthology.orchestra.run.smithi100.stdout: "auid": 0, 2024-09-17T07:25:33.045 INFO:teuthology.orchestra.run.smithi100.stdout: "snap_mode": "selfmanaged", 2024-09-17T07:25:33.045 INFO:teuthology.orchestra.run.smithi100.stdout: "snap_seq": 0, 2024-09-17T07:25:33.045 INFO:teuthology.orchestra.run.smithi100.stdout: "snap_epoch": 0, 2024-09-17T07:25:33.045 INFO:teuthology.orchestra.run.smithi100.stdout: "pool_snaps": [], 2024-09-17T07:25:33.045 INFO:teuthology.orchestra.run.smithi100.stdout: "removed_snaps": "[]", 2024-09-17T07:25:33.045 INFO:teuthology.orchestra.run.smithi100.stdout: "quota_max_bytes": 0, 2024-09-17T07:25:33.045 INFO:teuthology.orchestra.run.smithi100.stdout: "quota_max_objects": 0, 2024-09-17T07:25:33.045 INFO:teuthology.orchestra.run.smithi100.stdout: "tiers": [], 2024-09-17T07:25:33.046 INFO:teuthology.orchestra.run.smithi100.stdout: "tier_of": -1, 2024-09-17T07:25:33.046 INFO:teuthology.orchestra.run.smithi100.stdout: "read_tier": -1, 2024-09-17T07:25:33.046 INFO:teuthology.orchestra.run.smithi100.stdout: "write_tier": -1, 2024-09-17T07:25:33.046 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_mode": "none", 2024-09-17T07:25:33.046 INFO:teuthology.orchestra.run.smithi100.stdout: "target_max_bytes": 0, 2024-09-17T07:25:33.046 INFO:teuthology.orchestra.run.smithi100.stdout: "target_max_objects": 0, 2024-09-17T07:25:33.046 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T07:25:33.046 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T07:25:33.046 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T07:25:33.046 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_min_flush_age": 0, 2024-09-17T07:25:33.046 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_min_evict_age": 0, 2024-09-17T07:25:33.046 INFO:teuthology.orchestra.run.smithi100.stdout: "erasure_code_profile": "", 2024-09-17T07:25:33.047 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_params": { 2024-09-17T07:25:33.047 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "none" 2024-09-17T07:25:33.047 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:25:33.047 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_period": 0, 2024-09-17T07:25:33.047 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_count": 0, 2024-09-17T07:25:33.047 INFO:teuthology.orchestra.run.smithi100.stdout: "use_gmt_hitset": true, 2024-09-17T07:25:33.047 INFO:teuthology.orchestra.run.smithi100.stdout: "min_read_recency_for_promote": 0, 2024-09-17T07:25:33.047 INFO:teuthology.orchestra.run.smithi100.stdout: "min_write_recency_for_promote": 0, 2024-09-17T07:25:33.047 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T07:25:33.047 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_search_last_n": 0, 2024-09-17T07:25:33.047 INFO:teuthology.orchestra.run.smithi100.stdout: "grade_table": [], 2024-09-17T07:25:33.047 INFO:teuthology.orchestra.run.smithi100.stdout: "stripe_width": 0, 2024-09-17T07:25:33.047 INFO:teuthology.orchestra.run.smithi100.stdout: "expected_num_objects": 0, 2024-09-17T07:25:33.048 INFO:teuthology.orchestra.run.smithi100.stdout: "fast_read": false, 2024-09-17T07:25:33.048 INFO:teuthology.orchestra.run.smithi100.stdout: "options": { 2024-09-17T07:25:33.048 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num_max": 32, 2024-09-17T07:25:33.048 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num_min": 1 2024-09-17T07:25:33.048 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:25:33.048 INFO:teuthology.orchestra.run.smithi100.stdout: "application_metadata": { 2024-09-17T07:25:33.048 INFO:teuthology.orchestra.run.smithi100.stdout: "mgr": {} 2024-09-17T07:25:33.048 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:25:33.048 INFO:teuthology.orchestra.run.smithi100.stdout: "read_balance": { 2024-09-17T07:25:33.048 INFO:teuthology.orchestra.run.smithi100.stdout: "score_acting": 2.9000000953674316, 2024-09-17T07:25:33.048 INFO:teuthology.orchestra.run.smithi100.stdout: "score_stable": 2.9000000953674316, 2024-09-17T07:25:33.048 INFO:teuthology.orchestra.run.smithi100.stdout: "optimal_score": 0.87999999523162842, 2024-09-17T07:25:33.049 INFO:teuthology.orchestra.run.smithi100.stdout: "raw_score_acting": 2.5499999523162842, 2024-09-17T07:25:33.049 INFO:teuthology.orchestra.run.smithi100.stdout: "raw_score_stable": 2.5499999523162842, 2024-09-17T07:25:33.049 INFO:teuthology.orchestra.run.smithi100.stdout: "primary_affinity_weighted": 0.89598274230957031, 2024-09-17T07:25:33.049 INFO:teuthology.orchestra.run.smithi100.stdout: "average_primary_affinity": 0.89999997615814209, 2024-09-17T07:25:33.049 INFO:teuthology.orchestra.run.smithi100.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-09-17T07:25:33.049 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:25:33.049 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:25:33.049 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:25:33.049 INFO:teuthology.orchestra.run.smithi100.stdout: "pool": 2, 2024-09-17T07:25:33.049 INFO:teuthology.orchestra.run.smithi100.stdout: "pool_name": ".rgw.root", 2024-09-17T07:25:33.049 INFO:teuthology.orchestra.run.smithi100.stdout: "create_time": "2024-09-17T07:02:10.759826+0000", 2024-09-17T07:25:33.049 INFO:teuthology.orchestra.run.smithi100.stdout: "flags": 1, 2024-09-17T07:25:33.049 INFO:teuthology.orchestra.run.smithi100.stdout: "flags_names": "hashpspool", 2024-09-17T07:25:33.050 INFO:teuthology.orchestra.run.smithi100.stdout: "type": 1, 2024-09-17T07:25:33.050 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 3, 2024-09-17T07:25:33.050 INFO:teuthology.orchestra.run.smithi100.stdout: "min_size": 2, 2024-09-17T07:25:33.050 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_rule": 0, 2024-09-17T07:25:33.050 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_count": 0, 2024-09-17T07:25:33.050 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_target": 0, 2024-09-17T07:25:33.051 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T07:25:33.051 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T07:25:33.051 INFO:teuthology.orchestra.run.smithi100.stdout: "object_hash": 2, 2024-09-17T07:25:33.051 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_autoscale_mode": "off", 2024-09-17T07:25:33.051 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num": 32, 2024-09-17T07:25:33.051 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_placement_num": 32, 2024-09-17T07:25:33.051 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_placement_num_target": 32, 2024-09-17T07:25:33.051 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num_target": 32, 2024-09-17T07:25:33.051 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num_pending": 32, 2024-09-17T07:25:33.051 INFO:teuthology.orchestra.run.smithi100.stdout: "last_pg_merge_meta": { 2024-09-17T07:25:33.051 INFO:teuthology.orchestra.run.smithi100.stdout: "source_pgid": "0.0", 2024-09-17T07:25:33.051 INFO:teuthology.orchestra.run.smithi100.stdout: "ready_epoch": 0, 2024-09-17T07:25:33.051 INFO:teuthology.orchestra.run.smithi100.stdout: "last_epoch_started": 0, 2024-09-17T07:25:33.052 INFO:teuthology.orchestra.run.smithi100.stdout: "last_epoch_clean": 0, 2024-09-17T07:25:33.052 INFO:teuthology.orchestra.run.smithi100.stdout: "source_version": "0'0", 2024-09-17T07:25:33.052 INFO:teuthology.orchestra.run.smithi100.stdout: "target_version": "0'0" 2024-09-17T07:25:33.052 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:25:33.052 INFO:teuthology.orchestra.run.smithi100.stdout: "last_change": "69", 2024-09-17T07:25:33.052 INFO:teuthology.orchestra.run.smithi100.stdout: "last_force_op_resend": "0", 2024-09-17T07:25:33.052 INFO:teuthology.orchestra.run.smithi100.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-17T07:25:33.052 INFO:teuthology.orchestra.run.smithi100.stdout: "last_force_op_resend_preluminous": "0", 2024-09-17T07:25:33.052 INFO:teuthology.orchestra.run.smithi100.stdout: "auid": 0, 2024-09-17T07:25:33.052 INFO:teuthology.orchestra.run.smithi100.stdout: "snap_mode": "selfmanaged", 2024-09-17T07:25:33.052 INFO:teuthology.orchestra.run.smithi100.stdout: "snap_seq": 0, 2024-09-17T07:25:33.053 INFO:teuthology.orchestra.run.smithi100.stdout: "snap_epoch": 0, 2024-09-17T07:25:33.053 INFO:teuthology.orchestra.run.smithi100.stdout: "pool_snaps": [], 2024-09-17T07:25:33.053 INFO:teuthology.orchestra.run.smithi100.stdout: "removed_snaps": "[]", 2024-09-17T07:25:33.053 INFO:teuthology.orchestra.run.smithi100.stdout: "quota_max_bytes": 0, 2024-09-17T07:25:33.053 INFO:teuthology.orchestra.run.smithi100.stdout: "quota_max_objects": 0, 2024-09-17T07:25:33.053 INFO:teuthology.orchestra.run.smithi100.stdout: "tiers": [], 2024-09-17T07:25:33.053 INFO:teuthology.orchestra.run.smithi100.stdout: "tier_of": -1, 2024-09-17T07:25:33.053 INFO:teuthology.orchestra.run.smithi100.stdout: "read_tier": -1, 2024-09-17T07:25:33.053 INFO:teuthology.orchestra.run.smithi100.stdout: "write_tier": -1, 2024-09-17T07:25:33.053 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_mode": "none", 2024-09-17T07:25:33.053 INFO:teuthology.orchestra.run.smithi100.stdout: "target_max_bytes": 0, 2024-09-17T07:25:33.053 INFO:teuthology.orchestra.run.smithi100.stdout: "target_max_objects": 0, 2024-09-17T07:25:33.053 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T07:25:33.054 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T07:25:33.054 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T07:25:33.054 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_min_flush_age": 0, 2024-09-17T07:25:33.054 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_min_evict_age": 0, 2024-09-17T07:25:33.054 INFO:teuthology.orchestra.run.smithi100.stdout: "erasure_code_profile": "", 2024-09-17T07:25:33.054 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_params": { 2024-09-17T07:25:33.054 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "none" 2024-09-17T07:25:33.054 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:25:33.054 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_period": 0, 2024-09-17T07:25:33.054 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_count": 0, 2024-09-17T07:25:33.054 INFO:teuthology.orchestra.run.smithi100.stdout: "use_gmt_hitset": true, 2024-09-17T07:25:33.054 INFO:teuthology.orchestra.run.smithi100.stdout: "min_read_recency_for_promote": 0, 2024-09-17T07:25:33.054 INFO:teuthology.orchestra.run.smithi100.stdout: "min_write_recency_for_promote": 0, 2024-09-17T07:25:33.055 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T07:25:33.055 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_search_last_n": 0, 2024-09-17T07:25:33.055 INFO:teuthology.orchestra.run.smithi100.stdout: "grade_table": [], 2024-09-17T07:25:33.055 INFO:teuthology.orchestra.run.smithi100.stdout: "stripe_width": 0, 2024-09-17T07:25:33.055 INFO:teuthology.orchestra.run.smithi100.stdout: "expected_num_objects": 0, 2024-09-17T07:25:33.055 INFO:teuthology.orchestra.run.smithi100.stdout: "fast_read": false, 2024-09-17T07:25:33.055 INFO:teuthology.orchestra.run.smithi100.stdout: "options": {}, 2024-09-17T07:25:33.055 INFO:teuthology.orchestra.run.smithi100.stdout: "application_metadata": { 2024-09-17T07:25:33.055 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw": {} 2024-09-17T07:25:33.055 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:25:33.056 INFO:teuthology.orchestra.run.smithi100.stdout: "read_balance": { 2024-09-17T07:25:33.056 INFO:teuthology.orchestra.run.smithi100.stdout: "score_acting": 1.9900000095367432, 2024-09-17T07:25:33.056 INFO:teuthology.orchestra.run.smithi100.stdout: "score_stable": 1.9900000095367432, 2024-09-17T07:25:33.056 INFO:teuthology.orchestra.run.smithi100.stdout: "optimal_score": 0.87999999523162842, 2024-09-17T07:25:33.056 INFO:teuthology.orchestra.run.smithi100.stdout: "raw_score_acting": 1.75, 2024-09-17T07:25:33.056 INFO:teuthology.orchestra.run.smithi100.stdout: "raw_score_stable": 1.75, 2024-09-17T07:25:33.056 INFO:teuthology.orchestra.run.smithi100.stdout: "primary_affinity_weighted": 0.89598274230957031, 2024-09-17T07:25:33.056 INFO:teuthology.orchestra.run.smithi100.stdout: "average_primary_affinity": 0.89999997615814209, 2024-09-17T07:25:33.056 INFO:teuthology.orchestra.run.smithi100.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-09-17T07:25:33.056 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:25:33.057 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:25:33.057 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:25:33.057 INFO:teuthology.orchestra.run.smithi100.stdout: "pool": 3, 2024-09-17T07:25:33.057 INFO:teuthology.orchestra.run.smithi100.stdout: "pool_name": "datapool", 2024-09-17T07:25:33.057 INFO:teuthology.orchestra.run.smithi100.stdout: "create_time": "2024-09-17T07:02:10.991289+0000", 2024-09-17T07:25:33.058 INFO:teuthology.orchestra.run.smithi100.stdout: "flags": 8193, 2024-09-17T07:25:33.058 INFO:teuthology.orchestra.run.smithi100.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-17T07:25:33.058 INFO:teuthology.orchestra.run.smithi100.stdout: "type": 1, 2024-09-17T07:25:33.058 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 3, 2024-09-17T07:25:33.058 INFO:teuthology.orchestra.run.smithi100.stdout: "min_size": 2, 2024-09-17T07:25:33.058 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_rule": 0, 2024-09-17T07:25:33.059 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_count": 0, 2024-09-17T07:25:33.059 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_target": 0, 2024-09-17T07:25:33.059 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T07:25:33.059 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T07:25:33.059 INFO:teuthology.orchestra.run.smithi100.stdout: "object_hash": 2, 2024-09-17T07:25:33.059 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_autoscale_mode": "off", 2024-09-17T07:25:33.060 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num": 3, 2024-09-17T07:25:33.060 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_placement_num": 3, 2024-09-17T07:25:33.060 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_placement_num_target": 3, 2024-09-17T07:25:33.060 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num_target": 3, 2024-09-17T07:25:33.060 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num_pending": 3, 2024-09-17T07:25:33.061 INFO:teuthology.orchestra.run.smithi100.stdout: "last_pg_merge_meta": { 2024-09-17T07:25:33.061 INFO:teuthology.orchestra.run.smithi100.stdout: "source_pgid": "0.0", 2024-09-17T07:25:33.061 INFO:teuthology.orchestra.run.smithi100.stdout: "ready_epoch": 0, 2024-09-17T07:25:33.061 INFO:teuthology.orchestra.run.smithi100.stdout: "last_epoch_started": 0, 2024-09-17T07:25:33.061 INFO:teuthology.orchestra.run.smithi100.stdout: "last_epoch_clean": 0, 2024-09-17T07:25:33.061 INFO:teuthology.orchestra.run.smithi100.stdout: "source_version": "0'0", 2024-09-17T07:25:33.061 INFO:teuthology.orchestra.run.smithi100.stdout: "target_version": "0'0" 2024-09-17T07:25:33.062 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:25:33.062 INFO:teuthology.orchestra.run.smithi100.stdout: "last_change": "72", 2024-09-17T07:25:33.062 INFO:teuthology.orchestra.run.smithi100.stdout: "last_force_op_resend": "0", 2024-09-17T07:25:33.062 INFO:teuthology.orchestra.run.smithi100.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-17T07:25:33.062 INFO:teuthology.orchestra.run.smithi100.stdout: "last_force_op_resend_preluminous": "0", 2024-09-17T07:25:33.063 INFO:teuthology.orchestra.run.smithi100.stdout: "auid": 0, 2024-09-17T07:25:33.063 INFO:teuthology.orchestra.run.smithi100.stdout: "snap_mode": "selfmanaged", 2024-09-17T07:25:33.063 INFO:teuthology.orchestra.run.smithi100.stdout: "snap_seq": 3, 2024-09-17T07:25:33.063 INFO:teuthology.orchestra.run.smithi100.stdout: "snap_epoch": 72, 2024-09-17T07:25:33.063 INFO:teuthology.orchestra.run.smithi100.stdout: "pool_snaps": [], 2024-09-17T07:25:33.063 INFO:teuthology.orchestra.run.smithi100.stdout: "removed_snaps": "[]", 2024-09-17T07:25:33.064 INFO:teuthology.orchestra.run.smithi100.stdout: "quota_max_bytes": 0, 2024-09-17T07:25:33.064 INFO:teuthology.orchestra.run.smithi100.stdout: "quota_max_objects": 0, 2024-09-17T07:25:33.064 INFO:teuthology.orchestra.run.smithi100.stdout: "tiers": [], 2024-09-17T07:25:33.064 INFO:teuthology.orchestra.run.smithi100.stdout: "tier_of": -1, 2024-09-17T07:25:33.064 INFO:teuthology.orchestra.run.smithi100.stdout: "read_tier": -1, 2024-09-17T07:25:33.064 INFO:teuthology.orchestra.run.smithi100.stdout: "write_tier": -1, 2024-09-17T07:25:33.065 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_mode": "none", 2024-09-17T07:25:33.065 INFO:teuthology.orchestra.run.smithi100.stdout: "target_max_bytes": 0, 2024-09-17T07:25:33.065 INFO:teuthology.orchestra.run.smithi100.stdout: "target_max_objects": 0, 2024-09-17T07:25:33.065 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T07:25:33.065 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T07:25:33.065 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T07:25:33.065 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_min_flush_age": 0, 2024-09-17T07:25:33.065 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_min_evict_age": 0, 2024-09-17T07:25:33.065 INFO:teuthology.orchestra.run.smithi100.stdout: "erasure_code_profile": "", 2024-09-17T07:25:33.065 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_params": { 2024-09-17T07:25:33.066 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "none" 2024-09-17T07:25:33.066 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:25:33.066 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_period": 0, 2024-09-17T07:25:33.066 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_count": 0, 2024-09-17T07:25:33.066 INFO:teuthology.orchestra.run.smithi100.stdout: "use_gmt_hitset": true, 2024-09-17T07:25:33.066 INFO:teuthology.orchestra.run.smithi100.stdout: "min_read_recency_for_promote": 0, 2024-09-17T07:25:33.066 INFO:teuthology.orchestra.run.smithi100.stdout: "min_write_recency_for_promote": 0, 2024-09-17T07:25:33.066 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T07:25:33.066 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_search_last_n": 0, 2024-09-17T07:25:33.066 INFO:teuthology.orchestra.run.smithi100.stdout: "grade_table": [], 2024-09-17T07:25:33.066 INFO:teuthology.orchestra.run.smithi100.stdout: "stripe_width": 0, 2024-09-17T07:25:33.066 INFO:teuthology.orchestra.run.smithi100.stdout: "expected_num_objects": 0, 2024-09-17T07:25:33.066 INFO:teuthology.orchestra.run.smithi100.stdout: "fast_read": false, 2024-09-17T07:25:33.067 INFO:teuthology.orchestra.run.smithi100.stdout: "options": {}, 2024-09-17T07:25:33.067 INFO:teuthology.orchestra.run.smithi100.stdout: "application_metadata": { 2024-09-17T07:25:33.067 INFO:teuthology.orchestra.run.smithi100.stdout: "rbd": {} 2024-09-17T07:25:33.067 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:25:33.067 INFO:teuthology.orchestra.run.smithi100.stdout: "read_balance": { 2024-09-17T07:25:33.067 INFO:teuthology.orchestra.run.smithi100.stdout: "score_acting": 2.6700000762939453, 2024-09-17T07:25:33.067 INFO:teuthology.orchestra.run.smithi100.stdout: "score_stable": 2.6700000762939453, 2024-09-17T07:25:33.067 INFO:teuthology.orchestra.run.smithi100.stdout: "optimal_score": 0.75, 2024-09-17T07:25:33.067 INFO:teuthology.orchestra.run.smithi100.stdout: "raw_score_acting": 2, 2024-09-17T07:25:33.067 INFO:teuthology.orchestra.run.smithi100.stdout: "raw_score_stable": 2, 2024-09-17T07:25:33.067 INFO:teuthology.orchestra.run.smithi100.stdout: "primary_affinity_weighted": 0.89598274230957031, 2024-09-17T07:25:33.067 INFO:teuthology.orchestra.run.smithi100.stdout: "average_primary_affinity": 0.89999997615814209, 2024-09-17T07:25:33.068 INFO:teuthology.orchestra.run.smithi100.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-09-17T07:25:33.068 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:25:33.068 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:25:33.068 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:25:33.068 INFO:teuthology.orchestra.run.smithi100.stdout: "pool": 4, 2024-09-17T07:25:33.068 INFO:teuthology.orchestra.run.smithi100.stdout: "pool_name": "default.rgw.log", 2024-09-17T07:25:33.068 INFO:teuthology.orchestra.run.smithi100.stdout: "create_time": "2024-09-17T07:02:12.291462+0000", 2024-09-17T07:25:33.068 INFO:teuthology.orchestra.run.smithi100.stdout: "flags": 1, 2024-09-17T07:25:33.068 INFO:teuthology.orchestra.run.smithi100.stdout: "flags_names": "hashpspool", 2024-09-17T07:25:33.068 INFO:teuthology.orchestra.run.smithi100.stdout: "type": 1, 2024-09-17T07:25:33.068 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 3, 2024-09-17T07:25:33.068 INFO:teuthology.orchestra.run.smithi100.stdout: "min_size": 2, 2024-09-17T07:25:33.069 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_rule": 0, 2024-09-17T07:25:33.069 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_count": 0, 2024-09-17T07:25:33.069 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_target": 0, 2024-09-17T07:25:33.069 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T07:25:33.069 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T07:25:33.069 INFO:teuthology.orchestra.run.smithi100.stdout: "object_hash": 2, 2024-09-17T07:25:33.069 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_autoscale_mode": "off", 2024-09-17T07:25:33.069 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num": 32, 2024-09-17T07:25:33.069 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_placement_num": 32, 2024-09-17T07:25:33.069 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_placement_num_target": 32, 2024-09-17T07:25:33.069 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num_target": 32, 2024-09-17T07:25:33.069 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num_pending": 32, 2024-09-17T07:25:33.070 INFO:teuthology.orchestra.run.smithi100.stdout: "last_pg_merge_meta": { 2024-09-17T07:25:33.070 INFO:teuthology.orchestra.run.smithi100.stdout: "source_pgid": "0.0", 2024-09-17T07:25:33.070 INFO:teuthology.orchestra.run.smithi100.stdout: "ready_epoch": 0, 2024-09-17T07:25:33.070 INFO:teuthology.orchestra.run.smithi100.stdout: "last_epoch_started": 0, 2024-09-17T07:25:33.070 INFO:teuthology.orchestra.run.smithi100.stdout: "last_epoch_clean": 0, 2024-09-17T07:25:33.070 INFO:teuthology.orchestra.run.smithi100.stdout: "source_version": "0'0", 2024-09-17T07:25:33.070 INFO:teuthology.orchestra.run.smithi100.stdout: "target_version": "0'0" 2024-09-17T07:25:33.070 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:25:33.070 INFO:teuthology.orchestra.run.smithi100.stdout: "last_change": "71", 2024-09-17T07:25:33.070 INFO:teuthology.orchestra.run.smithi100.stdout: "last_force_op_resend": "0", 2024-09-17T07:25:33.071 INFO:teuthology.orchestra.run.smithi100.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-17T07:25:33.071 INFO:teuthology.orchestra.run.smithi100.stdout: "last_force_op_resend_preluminous": "0", 2024-09-17T07:25:33.071 INFO:teuthology.orchestra.run.smithi100.stdout: "auid": 0, 2024-09-17T07:25:33.071 INFO:teuthology.orchestra.run.smithi100.stdout: "snap_mode": "selfmanaged", 2024-09-17T07:25:33.071 INFO:teuthology.orchestra.run.smithi100.stdout: "snap_seq": 0, 2024-09-17T07:25:33.071 INFO:teuthology.orchestra.run.smithi100.stdout: "snap_epoch": 0, 2024-09-17T07:25:33.071 INFO:teuthology.orchestra.run.smithi100.stdout: "pool_snaps": [], 2024-09-17T07:25:33.071 INFO:teuthology.orchestra.run.smithi100.stdout: "removed_snaps": "[]", 2024-09-17T07:25:33.071 INFO:teuthology.orchestra.run.smithi100.stdout: "quota_max_bytes": 0, 2024-09-17T07:25:33.071 INFO:teuthology.orchestra.run.smithi100.stdout: "quota_max_objects": 0, 2024-09-17T07:25:33.071 INFO:teuthology.orchestra.run.smithi100.stdout: "tiers": [], 2024-09-17T07:25:33.071 INFO:teuthology.orchestra.run.smithi100.stdout: "tier_of": -1, 2024-09-17T07:25:33.071 INFO:teuthology.orchestra.run.smithi100.stdout: "read_tier": -1, 2024-09-17T07:25:33.072 INFO:teuthology.orchestra.run.smithi100.stdout: "write_tier": -1, 2024-09-17T07:25:33.072 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_mode": "none", 2024-09-17T07:25:33.072 INFO:teuthology.orchestra.run.smithi100.stdout: "target_max_bytes": 0, 2024-09-17T07:25:33.072 INFO:teuthology.orchestra.run.smithi100.stdout: "target_max_objects": 0, 2024-09-17T07:25:33.072 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T07:25:33.072 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T07:25:33.072 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T07:25:33.072 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_min_flush_age": 0, 2024-09-17T07:25:33.072 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_min_evict_age": 0, 2024-09-17T07:25:33.072 INFO:teuthology.orchestra.run.smithi100.stdout: "erasure_code_profile": "", 2024-09-17T07:25:33.072 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_params": { 2024-09-17T07:25:33.072 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "none" 2024-09-17T07:25:33.073 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:25:33.073 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_period": 0, 2024-09-17T07:25:33.073 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_count": 0, 2024-09-17T07:25:33.073 INFO:teuthology.orchestra.run.smithi100.stdout: "use_gmt_hitset": true, 2024-09-17T07:25:33.073 INFO:teuthology.orchestra.run.smithi100.stdout: "min_read_recency_for_promote": 0, 2024-09-17T07:25:33.073 INFO:teuthology.orchestra.run.smithi100.stdout: "min_write_recency_for_promote": 0, 2024-09-17T07:25:33.073 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T07:25:33.073 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_search_last_n": 0, 2024-09-17T07:25:33.073 INFO:teuthology.orchestra.run.smithi100.stdout: "grade_table": [], 2024-09-17T07:25:33.073 INFO:teuthology.orchestra.run.smithi100.stdout: "stripe_width": 0, 2024-09-17T07:25:33.073 INFO:teuthology.orchestra.run.smithi100.stdout: "expected_num_objects": 0, 2024-09-17T07:25:33.074 INFO:teuthology.orchestra.run.smithi100.stdout: "fast_read": false, 2024-09-17T07:25:33.074 INFO:teuthology.orchestra.run.smithi100.stdout: "options": {}, 2024-09-17T07:25:33.074 INFO:teuthology.orchestra.run.smithi100.stdout: "application_metadata": { 2024-09-17T07:25:33.074 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw": {} 2024-09-17T07:25:33.074 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:25:33.074 INFO:teuthology.orchestra.run.smithi100.stdout: "read_balance": { 2024-09-17T07:25:33.074 INFO:teuthology.orchestra.run.smithi100.stdout: "score_acting": 2.2400000095367432, 2024-09-17T07:25:33.074 INFO:teuthology.orchestra.run.smithi100.stdout: "score_stable": 2.2400000095367432, 2024-09-17T07:25:33.074 INFO:teuthology.orchestra.run.smithi100.stdout: "optimal_score": 0.87999999523162842, 2024-09-17T07:25:33.074 INFO:teuthology.orchestra.run.smithi100.stdout: "raw_score_acting": 1.9700000286102295, 2024-09-17T07:25:33.074 INFO:teuthology.orchestra.run.smithi100.stdout: "raw_score_stable": 1.9700000286102295, 2024-09-17T07:25:33.074 INFO:teuthology.orchestra.run.smithi100.stdout: "primary_affinity_weighted": 0.89598274230957031, 2024-09-17T07:25:33.075 INFO:teuthology.orchestra.run.smithi100.stdout: "average_primary_affinity": 0.89999997615814209, 2024-09-17T07:25:33.075 INFO:teuthology.orchestra.run.smithi100.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-09-17T07:25:33.075 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:25:33.075 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:25:33.075 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:25:33.075 INFO:teuthology.orchestra.run.smithi100.stdout: "pool": 5, 2024-09-17T07:25:33.075 INFO:teuthology.orchestra.run.smithi100.stdout: "pool_name": "default.rgw.control", 2024-09-17T07:25:33.075 INFO:teuthology.orchestra.run.smithi100.stdout: "create_time": "2024-09-17T07:02:14.253214+0000", 2024-09-17T07:25:33.075 INFO:teuthology.orchestra.run.smithi100.stdout: "flags": 1, 2024-09-17T07:25:33.075 INFO:teuthology.orchestra.run.smithi100.stdout: "flags_names": "hashpspool", 2024-09-17T07:25:33.075 INFO:teuthology.orchestra.run.smithi100.stdout: "type": 1, 2024-09-17T07:25:33.075 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 3, 2024-09-17T07:25:33.075 INFO:teuthology.orchestra.run.smithi100.stdout: "min_size": 2, 2024-09-17T07:25:33.076 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_rule": 0, 2024-09-17T07:25:33.076 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_count": 0, 2024-09-17T07:25:33.076 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_target": 0, 2024-09-17T07:25:33.076 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T07:25:33.076 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T07:25:33.076 INFO:teuthology.orchestra.run.smithi100.stdout: "object_hash": 2, 2024-09-17T07:25:33.076 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_autoscale_mode": "off", 2024-09-17T07:25:33.076 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num": 32, 2024-09-17T07:25:33.076 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_placement_num": 32, 2024-09-17T07:25:33.076 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_placement_num_target": 32, 2024-09-17T07:25:33.076 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num_target": 32, 2024-09-17T07:25:33.076 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num_pending": 32, 2024-09-17T07:25:33.076 INFO:teuthology.orchestra.run.smithi100.stdout: "last_pg_merge_meta": { 2024-09-17T07:25:33.077 INFO:teuthology.orchestra.run.smithi100.stdout: "source_pgid": "0.0", 2024-09-17T07:25:33.077 INFO:teuthology.orchestra.run.smithi100.stdout: "ready_epoch": 0, 2024-09-17T07:25:33.077 INFO:teuthology.orchestra.run.smithi100.stdout: "last_epoch_started": 0, 2024-09-17T07:25:33.077 INFO:teuthology.orchestra.run.smithi100.stdout: "last_epoch_clean": 0, 2024-09-17T07:25:33.077 INFO:teuthology.orchestra.run.smithi100.stdout: "source_version": "0'0", 2024-09-17T07:25:33.077 INFO:teuthology.orchestra.run.smithi100.stdout: "target_version": "0'0" 2024-09-17T07:25:33.077 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:25:33.077 INFO:teuthology.orchestra.run.smithi100.stdout: "last_change": "73", 2024-09-17T07:25:33.077 INFO:teuthology.orchestra.run.smithi100.stdout: "last_force_op_resend": "0", 2024-09-17T07:25:33.077 INFO:teuthology.orchestra.run.smithi100.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-17T07:25:33.078 INFO:teuthology.orchestra.run.smithi100.stdout: "last_force_op_resend_preluminous": "0", 2024-09-17T07:25:33.078 INFO:teuthology.orchestra.run.smithi100.stdout: "auid": 0, 2024-09-17T07:25:33.078 INFO:teuthology.orchestra.run.smithi100.stdout: "snap_mode": "selfmanaged", 2024-09-17T07:25:33.078 INFO:teuthology.orchestra.run.smithi100.stdout: "snap_seq": 0, 2024-09-17T07:25:33.078 INFO:teuthology.orchestra.run.smithi100.stdout: "snap_epoch": 0, 2024-09-17T07:25:33.078 INFO:teuthology.orchestra.run.smithi100.stdout: "pool_snaps": [], 2024-09-17T07:25:33.078 INFO:teuthology.orchestra.run.smithi100.stdout: "removed_snaps": "[]", 2024-09-17T07:25:33.078 INFO:teuthology.orchestra.run.smithi100.stdout: "quota_max_bytes": 0, 2024-09-17T07:25:33.078 INFO:teuthology.orchestra.run.smithi100.stdout: "quota_max_objects": 0, 2024-09-17T07:25:33.078 INFO:teuthology.orchestra.run.smithi100.stdout: "tiers": [], 2024-09-17T07:25:33.078 INFO:teuthology.orchestra.run.smithi100.stdout: "tier_of": -1, 2024-09-17T07:25:33.078 INFO:teuthology.orchestra.run.smithi100.stdout: "read_tier": -1, 2024-09-17T07:25:33.078 INFO:teuthology.orchestra.run.smithi100.stdout: "write_tier": -1, 2024-09-17T07:25:33.079 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_mode": "none", 2024-09-17T07:25:33.079 INFO:teuthology.orchestra.run.smithi100.stdout: "target_max_bytes": 0, 2024-09-17T07:25:33.079 INFO:teuthology.orchestra.run.smithi100.stdout: "target_max_objects": 0, 2024-09-17T07:25:33.079 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T07:25:33.079 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T07:25:33.079 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T07:25:33.079 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_min_flush_age": 0, 2024-09-17T07:25:33.079 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_min_evict_age": 0, 2024-09-17T07:25:33.079 INFO:teuthology.orchestra.run.smithi100.stdout: "erasure_code_profile": "", 2024-09-17T07:25:33.079 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_params": { 2024-09-17T07:25:33.079 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "none" 2024-09-17T07:25:33.079 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:25:33.079 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_period": 0, 2024-09-17T07:25:33.080 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_count": 0, 2024-09-17T07:25:33.080 INFO:teuthology.orchestra.run.smithi100.stdout: "use_gmt_hitset": true, 2024-09-17T07:25:33.080 INFO:teuthology.orchestra.run.smithi100.stdout: "min_read_recency_for_promote": 0, 2024-09-17T07:25:33.080 INFO:teuthology.orchestra.run.smithi100.stdout: "min_write_recency_for_promote": 0, 2024-09-17T07:25:33.080 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T07:25:33.080 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_search_last_n": 0, 2024-09-17T07:25:33.080 INFO:teuthology.orchestra.run.smithi100.stdout: "grade_table": [], 2024-09-17T07:25:33.080 INFO:teuthology.orchestra.run.smithi100.stdout: "stripe_width": 0, 2024-09-17T07:25:33.080 INFO:teuthology.orchestra.run.smithi100.stdout: "expected_num_objects": 0, 2024-09-17T07:25:33.080 INFO:teuthology.orchestra.run.smithi100.stdout: "fast_read": false, 2024-09-17T07:25:33.080 INFO:teuthology.orchestra.run.smithi100.stdout: "options": {}, 2024-09-17T07:25:33.080 INFO:teuthology.orchestra.run.smithi100.stdout: "application_metadata": { 2024-09-17T07:25:33.080 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw": {} 2024-09-17T07:25:33.081 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:25:33.081 INFO:teuthology.orchestra.run.smithi100.stdout: "read_balance": { 2024-09-17T07:25:33.081 INFO:teuthology.orchestra.run.smithi100.stdout: "score_acting": 1.4900000095367432, 2024-09-17T07:25:33.081 INFO:teuthology.orchestra.run.smithi100.stdout: "score_stable": 1.4900000095367432, 2024-09-17T07:25:33.081 INFO:teuthology.orchestra.run.smithi100.stdout: "optimal_score": 0.87999999523162842, 2024-09-17T07:25:33.081 INFO:teuthology.orchestra.run.smithi100.stdout: "raw_score_acting": 1.309999942779541, 2024-09-17T07:25:33.081 INFO:teuthology.orchestra.run.smithi100.stdout: "raw_score_stable": 1.309999942779541, 2024-09-17T07:25:33.081 INFO:teuthology.orchestra.run.smithi100.stdout: "primary_affinity_weighted": 0.89598274230957031, 2024-09-17T07:25:33.081 INFO:teuthology.orchestra.run.smithi100.stdout: "average_primary_affinity": 0.89999997615814209, 2024-09-17T07:25:33.081 INFO:teuthology.orchestra.run.smithi100.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-09-17T07:25:33.081 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:25:33.082 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:25:33.082 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:25:33.082 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:32 smithi135 ceph-mon[33238]: 4.1f deep-scrub starts 2024-09-17T07:25:33.083 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:32 smithi135 ceph-mon[33238]: 4.1f deep-scrub ok 2024-09-17T07:25:33.083 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:32 smithi135 ceph-mon[33238]: pgmap v1120: 142 pgs: 142 active+clean; 693 KiB data, 6.5 GiB used, 619 GiB / 626 GiB avail 2024-09-17T07:25:33.083 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:32 smithi135 ceph-mon[33238]: 1.8 scrub starts 2024-09-17T07:25:33.083 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:32 smithi135 ceph-mon[33238]: 1.8 scrub ok 2024-09-17T07:25:33.083 INFO:teuthology.orchestra.run.smithi100.stdout: "pool": 6, 2024-09-17T07:25:33.083 INFO:teuthology.orchestra.run.smithi100.stdout: "pool_name": "default.rgw.meta", 2024-09-17T07:25:33.083 INFO:teuthology.orchestra.run.smithi100.stdout: "create_time": "2024-09-17T07:02:16.310620+0000", 2024-09-17T07:25:33.083 INFO:teuthology.orchestra.run.smithi100.stdout: "flags": 1, 2024-09-17T07:25:33.083 INFO:teuthology.orchestra.run.smithi100.stdout: "flags_names": "hashpspool", 2024-09-17T07:25:33.084 INFO:teuthology.orchestra.run.smithi100.stdout: "type": 1, 2024-09-17T07:25:33.084 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 3, 2024-09-17T07:25:33.084 INFO:teuthology.orchestra.run.smithi100.stdout: "min_size": 2, 2024-09-17T07:25:33.084 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_rule": 0, 2024-09-17T07:25:33.084 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_count": 0, 2024-09-17T07:25:33.084 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_target": 0, 2024-09-17T07:25:33.084 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T07:25:33.084 INFO:teuthology.orchestra.run.smithi100.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T07:25:33.084 INFO:teuthology.orchestra.run.smithi100.stdout: "object_hash": 2, 2024-09-17T07:25:33.084 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_autoscale_mode": "off", 2024-09-17T07:25:33.084 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num": 32, 2024-09-17T07:25:33.084 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_placement_num": 32, 2024-09-17T07:25:33.085 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_placement_num_target": 32, 2024-09-17T07:25:33.085 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num_target": 32, 2024-09-17T07:25:33.085 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_num_pending": 32, 2024-09-17T07:25:33.085 INFO:teuthology.orchestra.run.smithi100.stdout: "last_pg_merge_meta": { 2024-09-17T07:25:33.085 INFO:teuthology.orchestra.run.smithi100.stdout: "source_pgid": "0.0", 2024-09-17T07:25:33.085 INFO:teuthology.orchestra.run.smithi100.stdout: "ready_epoch": 0, 2024-09-17T07:25:33.085 INFO:teuthology.orchestra.run.smithi100.stdout: "last_epoch_started": 0, 2024-09-17T07:25:33.085 INFO:teuthology.orchestra.run.smithi100.stdout: "last_epoch_clean": 0, 2024-09-17T07:25:33.085 INFO:teuthology.orchestra.run.smithi100.stdout: "source_version": "0'0", 2024-09-17T07:25:33.085 INFO:teuthology.orchestra.run.smithi100.stdout: "target_version": "0'0" 2024-09-17T07:25:33.085 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:25:33.085 INFO:teuthology.orchestra.run.smithi100.stdout: "last_change": "75", 2024-09-17T07:25:33.086 INFO:teuthology.orchestra.run.smithi100.stdout: "last_force_op_resend": "0", 2024-09-17T07:25:33.086 INFO:teuthology.orchestra.run.smithi100.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-17T07:25:33.086 INFO:teuthology.orchestra.run.smithi100.stdout: "last_force_op_resend_preluminous": "0", 2024-09-17T07:25:33.086 INFO:teuthology.orchestra.run.smithi100.stdout: "auid": 0, 2024-09-17T07:25:33.086 INFO:teuthology.orchestra.run.smithi100.stdout: "snap_mode": "selfmanaged", 2024-09-17T07:25:33.086 INFO:teuthology.orchestra.run.smithi100.stdout: "snap_seq": 0, 2024-09-17T07:25:33.086 INFO:teuthology.orchestra.run.smithi100.stdout: "snap_epoch": 0, 2024-09-17T07:25:33.086 INFO:teuthology.orchestra.run.smithi100.stdout: "pool_snaps": [], 2024-09-17T07:25:33.086 INFO:teuthology.orchestra.run.smithi100.stdout: "removed_snaps": "[]", 2024-09-17T07:25:33.086 INFO:teuthology.orchestra.run.smithi100.stdout: "quota_max_bytes": 0, 2024-09-17T07:25:33.086 INFO:teuthology.orchestra.run.smithi100.stdout: "quota_max_objects": 0, 2024-09-17T07:25:33.086 INFO:teuthology.orchestra.run.smithi100.stdout: "tiers": [], 2024-09-17T07:25:33.086 INFO:teuthology.orchestra.run.smithi100.stdout: "tier_of": -1, 2024-09-17T07:25:33.087 INFO:teuthology.orchestra.run.smithi100.stdout: "read_tier": -1, 2024-09-17T07:25:33.087 INFO:teuthology.orchestra.run.smithi100.stdout: "write_tier": -1, 2024-09-17T07:25:33.087 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_mode": "none", 2024-09-17T07:25:33.087 INFO:teuthology.orchestra.run.smithi100.stdout: "target_max_bytes": 0, 2024-09-17T07:25:33.087 INFO:teuthology.orchestra.run.smithi100.stdout: "target_max_objects": 0, 2024-09-17T07:25:33.087 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T07:25:33.087 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T07:25:33.087 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T07:25:33.087 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_min_flush_age": 0, 2024-09-17T07:25:33.087 INFO:teuthology.orchestra.run.smithi100.stdout: "cache_min_evict_age": 0, 2024-09-17T07:25:33.087 INFO:teuthology.orchestra.run.smithi100.stdout: "erasure_code_profile": "", 2024-09-17T07:25:33.087 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_params": { 2024-09-17T07:25:33.087 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "none" 2024-09-17T07:25:33.088 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:25:33.088 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_period": 0, 2024-09-17T07:25:33.088 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_count": 0, 2024-09-17T07:25:33.088 INFO:teuthology.orchestra.run.smithi100.stdout: "use_gmt_hitset": true, 2024-09-17T07:25:33.088 INFO:teuthology.orchestra.run.smithi100.stdout: "min_read_recency_for_promote": 0, 2024-09-17T07:25:33.088 INFO:teuthology.orchestra.run.smithi100.stdout: "min_write_recency_for_promote": 0, 2024-09-17T07:25:33.088 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T07:25:33.088 INFO:teuthology.orchestra.run.smithi100.stdout: "hit_set_search_last_n": 0, 2024-09-17T07:25:33.088 INFO:teuthology.orchestra.run.smithi100.stdout: "grade_table": [], 2024-09-17T07:25:33.088 INFO:teuthology.orchestra.run.smithi100.stdout: "stripe_width": 0, 2024-09-17T07:25:33.088 INFO:teuthology.orchestra.run.smithi100.stdout: "expected_num_objects": 0, 2024-09-17T07:25:33.088 INFO:teuthology.orchestra.run.smithi100.stdout: "fast_read": false, 2024-09-17T07:25:33.089 INFO:teuthology.orchestra.run.smithi100.stdout: "options": { 2024-09-17T07:25:33.089 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_autoscale_bias": 4 2024-09-17T07:25:33.089 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:25:33.089 INFO:teuthology.orchestra.run.smithi100.stdout: "application_metadata": { 2024-09-17T07:25:33.089 INFO:teuthology.orchestra.run.smithi100.stdout: "rgw": {} 2024-09-17T07:25:33.089 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:25:33.089 INFO:teuthology.orchestra.run.smithi100.stdout: "read_balance": { 2024-09-17T07:25:33.089 INFO:teuthology.orchestra.run.smithi100.stdout: "score_acting": 1.9900000095367432, 2024-09-17T07:25:33.089 INFO:teuthology.orchestra.run.smithi100.stdout: "score_stable": 1.9900000095367432, 2024-09-17T07:25:33.089 INFO:teuthology.orchestra.run.smithi100.stdout: "optimal_score": 0.87999999523162842, 2024-09-17T07:25:33.089 INFO:teuthology.orchestra.run.smithi100.stdout: "raw_score_acting": 1.75, 2024-09-17T07:25:33.089 INFO:teuthology.orchestra.run.smithi100.stdout: "raw_score_stable": 1.75, 2024-09-17T07:25:33.090 INFO:teuthology.orchestra.run.smithi100.stdout: "primary_affinity_weighted": 0.89598274230957031, 2024-09-17T07:25:33.090 INFO:teuthology.orchestra.run.smithi100.stdout: "average_primary_affinity": 0.89999997615814209, 2024-09-17T07:25:33.090 INFO:teuthology.orchestra.run.smithi100.stdout: "average_primary_affinity_weighted": 0.89999997615814209 2024-09-17T07:25:33.090 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:25:33.090 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:25:33.090 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-17T07:25:33.090 INFO:teuthology.orchestra.run.smithi100.stdout: "osds": [ 2024-09-17T07:25:33.090 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:25:33.090 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": 0, 2024-09-17T07:25:33.090 INFO:teuthology.orchestra.run.smithi100.stdout: "uuid": "27bee9ec-05e6-406c-8bf1-168098da0ec7", 2024-09-17T07:25:33.090 INFO:teuthology.orchestra.run.smithi100.stdout: "up": 1, 2024-09-17T07:25:33.090 INFO:teuthology.orchestra.run.smithi100.stdout: "in": 1, 2024-09-17T07:25:33.091 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 1, 2024-09-17T07:25:33.091 INFO:teuthology.orchestra.run.smithi100.stdout: "primary_affinity": 1, 2024-09-17T07:25:33.091 INFO:teuthology.orchestra.run.smithi100.stdout: "last_clean_begin": 0, 2024-09-17T07:25:33.091 INFO:teuthology.orchestra.run.smithi100.stdout: "last_clean_end": 0, 2024-09-17T07:25:33.091 INFO:teuthology.orchestra.run.smithi100.stdout: "up_from": 8, 2024-09-17T07:25:33.091 INFO:teuthology.orchestra.run.smithi100.stdout: "up_thru": 491, 2024-09-17T07:25:33.091 INFO:teuthology.orchestra.run.smithi100.stdout: "down_at": 0, 2024-09-17T07:25:33.091 INFO:teuthology.orchestra.run.smithi100.stdout: "lost_at": 0, 2024-09-17T07:25:33.091 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-17T07:25:33.091 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:25:33.091 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:25:33.091 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:25:33.091 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6801", 2024-09-17T07:25:33.092 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 3341017455 2024-09-17T07:25:33.092 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:25:33.092 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:25:33.092 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:25:33.092 INFO:teuthology.orchestra.run.smithi100.stdout: "cluster_addrs": { 2024-09-17T07:25:33.092 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:25:33.092 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:25:33.092 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:25:33.092 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6802", 2024-09-17T07:25:33.092 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 3341017455 2024-09-17T07:25:33.093 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:25:33.093 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:25:33.093 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:25:33.093 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_back_addrs": { 2024-09-17T07:25:33.093 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:25:33.093 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:25:33.093 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:25:33.093 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6804", 2024-09-17T07:25:33.094 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 3341017455 2024-09-17T07:25:33.094 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:25:33.094 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:25:33.094 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:25:33.094 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_front_addrs": { 2024-09-17T07:25:33.094 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:25:33.094 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:25:33.094 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:25:33.094 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6803", 2024-09-17T07:25:33.094 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 3341017455 2024-09-17T07:25:33.094 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:25:33.094 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:25:33.094 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:25:33.095 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.100:6801/3341017455", 2024-09-17T07:25:33.095 INFO:teuthology.orchestra.run.smithi100.stdout: "cluster_addr": "172.21.15.100:6802/3341017455", 2024-09-17T07:25:33.095 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_back_addr": "172.21.15.100:6804/3341017455", 2024-09-17T07:25:33.095 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_front_addr": "172.21.15.100:6803/3341017455", 2024-09-17T07:25:33.095 INFO:teuthology.orchestra.run.smithi100.stdout: "state": [ 2024-09-17T07:25:33.095 INFO:teuthology.orchestra.run.smithi100.stdout: "exists", 2024-09-17T07:25:33.095 INFO:teuthology.orchestra.run.smithi100.stdout: "up" 2024-09-17T07:25:33.095 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:25:33.095 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:25:33.095 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:25:33.096 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": 1, 2024-09-17T07:25:33.096 INFO:teuthology.orchestra.run.smithi100.stdout: "uuid": "b1a30c9c-eeaf-478e-9b62-2d63ad45873d", 2024-09-17T07:25:33.096 INFO:teuthology.orchestra.run.smithi100.stdout: "up": 1, 2024-09-17T07:25:33.096 INFO:teuthology.orchestra.run.smithi100.stdout: "in": 1, 2024-09-17T07:25:33.096 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 1, 2024-09-17T07:25:33.096 INFO:teuthology.orchestra.run.smithi100.stdout: "primary_affinity": 1, 2024-09-17T07:25:33.096 INFO:teuthology.orchestra.run.smithi100.stdout: "last_clean_begin": 13, 2024-09-17T07:25:33.096 INFO:teuthology.orchestra.run.smithi100.stdout: "last_clean_end": 302, 2024-09-17T07:25:33.096 INFO:teuthology.orchestra.run.smithi100.stdout: "up_from": 413, 2024-09-17T07:25:33.096 INFO:teuthology.orchestra.run.smithi100.stdout: "up_thru": 491, 2024-09-17T07:25:33.096 INFO:teuthology.orchestra.run.smithi100.stdout: "down_at": 303, 2024-09-17T07:25:33.096 INFO:teuthology.orchestra.run.smithi100.stdout: "lost_at": 0, 2024-09-17T07:25:33.096 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-17T07:25:33.097 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:25:33.097 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:25:33.097 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:25:33.097 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6805", 2024-09-17T07:25:33.097 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 2894367777 2024-09-17T07:25:33.097 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:25:33.097 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:25:33.097 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:25:33.097 INFO:teuthology.orchestra.run.smithi100.stdout: "cluster_addrs": { 2024-09-17T07:25:33.097 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:25:33.097 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:25:33.098 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:25:33.098 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6806", 2024-09-17T07:25:33.098 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 2894367777 2024-09-17T07:25:33.098 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:25:33.098 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:25:33.098 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:25:33.098 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_back_addrs": { 2024-09-17T07:25:33.098 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:25:33.098 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:25:33.098 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:25:33.098 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6808", 2024-09-17T07:25:33.098 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 2894367777 2024-09-17T07:25:33.098 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:25:33.099 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:25:33.099 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:25:33.099 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_front_addrs": { 2024-09-17T07:25:33.099 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:25:33.099 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:25:33.099 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:25:33.099 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6807", 2024-09-17T07:25:33.099 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 2894367777 2024-09-17T07:25:33.099 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:25:33.099 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:25:33.099 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:25:33.099 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.100:6805/2894367777", 2024-09-17T07:25:33.099 INFO:teuthology.orchestra.run.smithi100.stdout: "cluster_addr": "172.21.15.100:6806/2894367777", 2024-09-17T07:25:33.100 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_back_addr": "172.21.15.100:6808/2894367777", 2024-09-17T07:25:33.100 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_front_addr": "172.21.15.100:6807/2894367777", 2024-09-17T07:25:33.100 INFO:teuthology.orchestra.run.smithi100.stdout: "state": [ 2024-09-17T07:25:33.100 INFO:teuthology.orchestra.run.smithi100.stdout: "exists", 2024-09-17T07:25:33.100 INFO:teuthology.orchestra.run.smithi100.stdout: "up" 2024-09-17T07:25:33.100 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:25:33.100 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:25:33.100 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:25:33.100 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": 2, 2024-09-17T07:25:33.100 INFO:teuthology.orchestra.run.smithi100.stdout: "uuid": "ce542e0d-9915-465e-aa11-7b3ce13f694f", 2024-09-17T07:25:33.100 INFO:teuthology.orchestra.run.smithi100.stdout: "up": 1, 2024-09-17T07:25:33.100 INFO:teuthology.orchestra.run.smithi100.stdout: "in": 1, 2024-09-17T07:25:33.101 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 1, 2024-09-17T07:25:33.101 INFO:teuthology.orchestra.run.smithi100.stdout: "primary_affinity": 0.1678619384765625, 2024-09-17T07:25:33.101 INFO:teuthology.orchestra.run.smithi100.stdout: "last_clean_begin": 0, 2024-09-17T07:25:33.101 INFO:teuthology.orchestra.run.smithi100.stdout: "last_clean_end": 0, 2024-09-17T07:25:33.101 INFO:teuthology.orchestra.run.smithi100.stdout: "up_from": 18, 2024-09-17T07:25:33.101 INFO:teuthology.orchestra.run.smithi100.stdout: "up_thru": 476, 2024-09-17T07:25:33.101 INFO:teuthology.orchestra.run.smithi100.stdout: "down_at": 0, 2024-09-17T07:25:33.101 INFO:teuthology.orchestra.run.smithi100.stdout: "lost_at": 0, 2024-09-17T07:25:33.101 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-17T07:25:33.101 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:25:33.101 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:25:33.101 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:25:33.101 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6809", 2024-09-17T07:25:33.102 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 714579846 2024-09-17T07:25:33.102 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:25:33.102 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:25:33.102 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:25:33.102 INFO:teuthology.orchestra.run.smithi100.stdout: "cluster_addrs": { 2024-09-17T07:25:33.102 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:25:33.102 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:25:33.102 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:25:33.102 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6810", 2024-09-17T07:25:33.102 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 714579846 2024-09-17T07:25:33.102 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:25:33.102 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:25:33.102 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:25:33.102 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_back_addrs": { 2024-09-17T07:25:33.103 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:25:33.103 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:25:33.103 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:25:33.103 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6812", 2024-09-17T07:25:33.103 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 714579846 2024-09-17T07:25:33.103 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:25:33.103 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:25:33.103 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:25:33.103 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_front_addrs": { 2024-09-17T07:25:33.103 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:25:33.103 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:25:33.103 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:25:33.103 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6811", 2024-09-17T07:25:33.104 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 714579846 2024-09-17T07:25:33.104 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:25:33.104 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:25:33.104 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:25:33.104 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.100:6809/714579846", 2024-09-17T07:25:33.104 INFO:teuthology.orchestra.run.smithi100.stdout: "cluster_addr": "172.21.15.100:6810/714579846", 2024-09-17T07:25:33.104 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_back_addr": "172.21.15.100:6812/714579846", 2024-09-17T07:25:33.104 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_front_addr": "172.21.15.100:6811/714579846", 2024-09-17T07:25:33.104 INFO:teuthology.orchestra.run.smithi100.stdout: "state": [ 2024-09-17T07:25:33.104 INFO:teuthology.orchestra.run.smithi100.stdout: "exists", 2024-09-17T07:25:33.104 INFO:teuthology.orchestra.run.smithi100.stdout: "up" 2024-09-17T07:25:33.104 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:25:33.105 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:25:33.105 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:25:33.105 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": 3, 2024-09-17T07:25:33.105 INFO:teuthology.orchestra.run.smithi100.stdout: "uuid": "a1257575-3c90-41de-88f6-3960744351b1", 2024-09-17T07:25:33.105 INFO:teuthology.orchestra.run.smithi100.stdout: "up": 1, 2024-09-17T07:25:33.105 INFO:teuthology.orchestra.run.smithi100.stdout: "in": 0, 2024-09-17T07:25:33.105 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 0, 2024-09-17T07:25:33.105 INFO:teuthology.orchestra.run.smithi100.stdout: "primary_affinity": 1, 2024-09-17T07:25:33.105 INFO:teuthology.orchestra.run.smithi100.stdout: "last_clean_begin": 0, 2024-09-17T07:25:33.105 INFO:teuthology.orchestra.run.smithi100.stdout: "last_clean_end": 0, 2024-09-17T07:25:33.105 INFO:teuthology.orchestra.run.smithi100.stdout: "up_from": 27, 2024-09-17T07:25:33.105 INFO:teuthology.orchestra.run.smithi100.stdout: "up_thru": 168, 2024-09-17T07:25:33.105 INFO:teuthology.orchestra.run.smithi100.stdout: "down_at": 0, 2024-09-17T07:25:33.106 INFO:teuthology.orchestra.run.smithi100.stdout: "lost_at": 0, 2024-09-17T07:25:33.106 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-17T07:25:33.106 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:25:33.106 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:25:33.106 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:25:33.106 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6813", 2024-09-17T07:25:33.106 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 2315956572 2024-09-17T07:25:33.106 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:25:33.106 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:25:33.106 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:25:33.106 INFO:teuthology.orchestra.run.smithi100.stdout: "cluster_addrs": { 2024-09-17T07:25:33.106 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:25:33.106 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:25:33.107 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:25:33.107 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6814", 2024-09-17T07:25:33.107 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 2315956572 2024-09-17T07:25:33.107 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:25:33.107 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:25:33.107 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:25:33.107 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_back_addrs": { 2024-09-17T07:25:33.107 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:25:33.107 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:25:33.107 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:25:33.107 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6816", 2024-09-17T07:25:33.107 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 2315956572 2024-09-17T07:25:33.107 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:25:33.108 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:25:33.108 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:25:33.108 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_front_addrs": { 2024-09-17T07:25:33.108 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:25:33.108 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:25:33.108 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:25:33.108 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.100:6815", 2024-09-17T07:25:33.108 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 2315956572 2024-09-17T07:25:33.108 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:25:33.108 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:25:33.108 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:25:33.108 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.100:6813/2315956572", 2024-09-17T07:25:33.109 INFO:teuthology.orchestra.run.smithi100.stdout: "cluster_addr": "172.21.15.100:6814/2315956572", 2024-09-17T07:25:33.109 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_back_addr": "172.21.15.100:6816/2315956572", 2024-09-17T07:25:33.109 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_front_addr": "172.21.15.100:6815/2315956572", 2024-09-17T07:25:33.109 INFO:teuthology.orchestra.run.smithi100.stdout: "state": [ 2024-09-17T07:25:33.109 INFO:teuthology.orchestra.run.smithi100.stdout: "exists", 2024-09-17T07:25:33.109 INFO:teuthology.orchestra.run.smithi100.stdout: "up" 2024-09-17T07:25:33.109 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:25:33.109 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:25:33.109 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:25:33.109 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": 4, 2024-09-17T07:25:33.109 INFO:teuthology.orchestra.run.smithi100.stdout: "uuid": "e44e8922-2c5e-44e3-b556-d4dfbc499649", 2024-09-17T07:25:33.109 INFO:teuthology.orchestra.run.smithi100.stdout: "up": 1, 2024-09-17T07:25:33.109 INFO:teuthology.orchestra.run.smithi100.stdout: "in": 1, 2024-09-17T07:25:33.109 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 1, 2024-09-17T07:25:33.110 INFO:teuthology.orchestra.run.smithi100.stdout: "primary_affinity": 1, 2024-09-17T07:25:33.110 INFO:teuthology.orchestra.run.smithi100.stdout: "last_clean_begin": 0, 2024-09-17T07:25:33.110 INFO:teuthology.orchestra.run.smithi100.stdout: "last_clean_end": 0, 2024-09-17T07:25:33.110 INFO:teuthology.orchestra.run.smithi100.stdout: "up_from": 33, 2024-09-17T07:25:33.110 INFO:teuthology.orchestra.run.smithi100.stdout: "up_thru": 504, 2024-09-17T07:25:33.110 INFO:teuthology.orchestra.run.smithi100.stdout: "down_at": 0, 2024-09-17T07:25:33.110 INFO:teuthology.orchestra.run.smithi100.stdout: "lost_at": 0, 2024-09-17T07:25:33.110 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-17T07:25:33.110 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:25:33.110 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:25:33.110 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:25:33.110 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.135:6800", 2024-09-17T07:25:33.110 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 1148820922 2024-09-17T07:25:33.111 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:25:33.111 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:25:33.111 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:25:33.111 INFO:teuthology.orchestra.run.smithi100.stdout: "cluster_addrs": { 2024-09-17T07:25:33.111 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:25:33.111 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:25:33.111 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:25:33.111 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.135:6801", 2024-09-17T07:25:33.111 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 1148820922 2024-09-17T07:25:33.111 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:25:33.111 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:25:33.111 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:25:33.111 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_back_addrs": { 2024-09-17T07:25:33.112 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:25:33.112 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:25:33.112 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:25:33.112 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.135:6803", 2024-09-17T07:25:33.112 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 1148820922 2024-09-17T07:25:33.112 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:25:33.112 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:25:33.112 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:25:33.112 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_front_addrs": { 2024-09-17T07:25:33.112 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:25:33.112 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:25:33.112 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:25:33.113 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.135:6802", 2024-09-17T07:25:33.113 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 1148820922 2024-09-17T07:25:33.113 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:25:33.113 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:25:33.113 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:25:33.113 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.135:6800/1148820922", 2024-09-17T07:25:33.113 INFO:teuthology.orchestra.run.smithi100.stdout: "cluster_addr": "172.21.15.135:6801/1148820922", 2024-09-17T07:25:33.113 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_back_addr": "172.21.15.135:6803/1148820922", 2024-09-17T07:25:33.113 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_front_addr": "172.21.15.135:6802/1148820922", 2024-09-17T07:25:33.113 INFO:teuthology.orchestra.run.smithi100.stdout: "state": [ 2024-09-17T07:25:33.113 INFO:teuthology.orchestra.run.smithi100.stdout: "exists", 2024-09-17T07:25:33.113 INFO:teuthology.orchestra.run.smithi100.stdout: "up" 2024-09-17T07:25:33.113 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:25:33.114 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:25:33.114 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:25:33.114 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": 5, 2024-09-17T07:25:33.114 INFO:teuthology.orchestra.run.smithi100.stdout: "uuid": "91e4a362-fcdb-41b9-b54e-716591ba6d79", 2024-09-17T07:25:33.114 INFO:teuthology.orchestra.run.smithi100.stdout: "up": 1, 2024-09-17T07:25:33.114 INFO:teuthology.orchestra.run.smithi100.stdout: "in": 1, 2024-09-17T07:25:33.114 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 1, 2024-09-17T07:25:33.114 INFO:teuthology.orchestra.run.smithi100.stdout: "primary_affinity": 1, 2024-09-17T07:25:33.114 INFO:teuthology.orchestra.run.smithi100.stdout: "last_clean_begin": 42, 2024-09-17T07:25:33.114 INFO:teuthology.orchestra.run.smithi100.stdout: "last_clean_end": 446, 2024-09-17T07:25:33.114 INFO:teuthology.orchestra.run.smithi100.stdout: "up_from": 476, 2024-09-17T07:25:33.114 INFO:teuthology.orchestra.run.smithi100.stdout: "up_thru": 483, 2024-09-17T07:25:33.114 INFO:teuthology.orchestra.run.smithi100.stdout: "down_at": 447, 2024-09-17T07:25:33.115 INFO:teuthology.orchestra.run.smithi100.stdout: "lost_at": 0, 2024-09-17T07:25:33.115 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-17T07:25:33.115 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:25:33.115 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:25:33.115 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:25:33.115 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.135:6804", 2024-09-17T07:25:33.115 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 1324312753 2024-09-17T07:25:33.115 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:25:33.115 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:25:33.115 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:25:33.115 INFO:teuthology.orchestra.run.smithi100.stdout: "cluster_addrs": { 2024-09-17T07:25:33.115 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:25:33.115 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:25:33.116 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:25:33.116 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.135:6805", 2024-09-17T07:25:33.116 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 1324312753 2024-09-17T07:25:33.116 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:25:33.116 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:25:33.116 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:25:33.116 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_back_addrs": { 2024-09-17T07:25:33.116 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:25:33.116 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:25:33.116 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:25:33.116 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.135:6807", 2024-09-17T07:25:33.116 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 1324312753 2024-09-17T07:25:33.116 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:25:33.117 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:25:33.117 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:25:33.117 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_front_addrs": { 2024-09-17T07:25:33.117 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:25:33.117 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:25:33.117 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:25:33.117 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.135:6806", 2024-09-17T07:25:33.117 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 1324312753 2024-09-17T07:25:33.117 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:25:33.117 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:25:33.117 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:25:33.117 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.135:6804/1324312753", 2024-09-17T07:25:33.117 INFO:teuthology.orchestra.run.smithi100.stdout: "cluster_addr": "172.21.15.135:6805/1324312753", 2024-09-17T07:25:33.118 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_back_addr": "172.21.15.135:6807/1324312753", 2024-09-17T07:25:33.118 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_front_addr": "172.21.15.135:6806/1324312753", 2024-09-17T07:25:33.118 INFO:teuthology.orchestra.run.smithi100.stdout: "state": [ 2024-09-17T07:25:33.118 INFO:teuthology.orchestra.run.smithi100.stdout: "exists", 2024-09-17T07:25:33.118 INFO:teuthology.orchestra.run.smithi100.stdout: "up" 2024-09-17T07:25:33.118 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:25:33.118 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:25:33.118 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:25:33.118 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": 6, 2024-09-17T07:25:33.118 INFO:teuthology.orchestra.run.smithi100.stdout: "uuid": "de799f64-ebd2-4a46-91f6-c2f276bee21f", 2024-09-17T07:25:33.118 INFO:teuthology.orchestra.run.smithi100.stdout: "up": 1, 2024-09-17T07:25:33.118 INFO:teuthology.orchestra.run.smithi100.stdout: "in": 1, 2024-09-17T07:25:33.118 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 1, 2024-09-17T07:25:33.119 INFO:teuthology.orchestra.run.smithi100.stdout: "primary_affinity": 1, 2024-09-17T07:25:33.119 INFO:teuthology.orchestra.run.smithi100.stdout: "last_clean_begin": 0, 2024-09-17T07:25:33.119 INFO:teuthology.orchestra.run.smithi100.stdout: "last_clean_end": 0, 2024-09-17T07:25:33.119 INFO:teuthology.orchestra.run.smithi100.stdout: "up_from": 50, 2024-09-17T07:25:33.119 INFO:teuthology.orchestra.run.smithi100.stdout: "up_thru": 486, 2024-09-17T07:25:33.119 INFO:teuthology.orchestra.run.smithi100.stdout: "down_at": 0, 2024-09-17T07:25:33.119 INFO:teuthology.orchestra.run.smithi100.stdout: "lost_at": 0, 2024-09-17T07:25:33.119 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-17T07:25:33.119 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:25:33.119 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:25:33.119 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:25:33.119 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.135:6808", 2024-09-17T07:25:33.119 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 2750184152 2024-09-17T07:25:33.120 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:25:33.120 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:25:33.120 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:25:33.120 INFO:teuthology.orchestra.run.smithi100.stdout: "cluster_addrs": { 2024-09-17T07:25:33.120 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:25:33.120 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:25:33.121 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:25:33.121 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.135:6809", 2024-09-17T07:25:33.121 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 2750184152 2024-09-17T07:25:33.121 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:25:33.121 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:25:33.121 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:25:33.121 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_back_addrs": { 2024-09-17T07:25:33.122 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:25:33.122 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:25:33.122 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:25:33.122 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.135:6811", 2024-09-17T07:25:33.122 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 2750184152 2024-09-17T07:25:33.123 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:25:33.123 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:25:33.123 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:25:33.123 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_front_addrs": { 2024-09-17T07:25:33.123 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:25:33.123 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:25:33.123 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:25:33.124 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.135:6810", 2024-09-17T07:25:33.124 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 2750184152 2024-09-17T07:25:33.124 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:25:33.124 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:25:33.124 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:25:33.124 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.135:6808/2750184152", 2024-09-17T07:25:33.125 INFO:teuthology.orchestra.run.smithi100.stdout: "cluster_addr": "172.21.15.135:6809/2750184152", 2024-09-17T07:25:33.125 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_back_addr": "172.21.15.135:6811/2750184152", 2024-09-17T07:25:33.125 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_front_addr": "172.21.15.135:6810/2750184152", 2024-09-17T07:25:33.125 INFO:teuthology.orchestra.run.smithi100.stdout: "state": [ 2024-09-17T07:25:33.125 INFO:teuthology.orchestra.run.smithi100.stdout: "exists", 2024-09-17T07:25:33.125 INFO:teuthology.orchestra.run.smithi100.stdout: "up" 2024-09-17T07:25:33.126 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:25:33.126 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:25:33.126 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:25:33.126 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": 7, 2024-09-17T07:25:33.127 INFO:teuthology.orchestra.run.smithi100.stdout: "uuid": "830932c7-5de1-44ff-a840-e53a6e859bda", 2024-09-17T07:25:33.127 INFO:teuthology.orchestra.run.smithi100.stdout: "up": 1, 2024-09-17T07:25:33.127 INFO:teuthology.orchestra.run.smithi100.stdout: "in": 1, 2024-09-17T07:25:33.127 INFO:teuthology.orchestra.run.smithi100.stdout: "weight": 1, 2024-09-17T07:25:33.127 INFO:teuthology.orchestra.run.smithi100.stdout: "primary_affinity": 1, 2024-09-17T07:25:33.127 INFO:teuthology.orchestra.run.smithi100.stdout: "last_clean_begin": 0, 2024-09-17T07:25:33.128 INFO:teuthology.orchestra.run.smithi100.stdout: "last_clean_end": 0, 2024-09-17T07:25:33.128 INFO:teuthology.orchestra.run.smithi100.stdout: "up_from": 60, 2024-09-17T07:25:33.128 INFO:teuthology.orchestra.run.smithi100.stdout: "up_thru": 485, 2024-09-17T07:25:33.128 INFO:teuthology.orchestra.run.smithi100.stdout: "down_at": 0, 2024-09-17T07:25:33.128 INFO:teuthology.orchestra.run.smithi100.stdout: "lost_at": 0, 2024-09-17T07:25:33.128 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addrs": { 2024-09-17T07:25:33.129 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:25:33.129 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:25:33.129 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:25:33.129 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.135:6812", 2024-09-17T07:25:33.129 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 160304773 2024-09-17T07:25:33.130 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:25:33.130 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:25:33.130 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:25:33.130 INFO:teuthology.orchestra.run.smithi100.stdout: "cluster_addrs": { 2024-09-17T07:25:33.130 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:25:33.130 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:25:33.131 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:25:33.131 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.135:6813", 2024-09-17T07:25:33.131 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 160304773 2024-09-17T07:25:33.131 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:25:33.131 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:25:33.131 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:25:33.131 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_back_addrs": { 2024-09-17T07:25:33.132 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:25:33.132 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:25:33.132 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:25:33.132 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.135:6815", 2024-09-17T07:25:33.132 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 160304773 2024-09-17T07:25:33.132 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:25:33.133 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:25:33.133 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:25:33.133 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_front_addrs": { 2024-09-17T07:25:33.133 INFO:teuthology.orchestra.run.smithi100.stdout: "addrvec": [ 2024-09-17T07:25:33.133 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:25:33.133 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v2", 2024-09-17T07:25:33.133 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "172.21.15.135:6814", 2024-09-17T07:25:33.133 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 160304773 2024-09-17T07:25:33.134 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:25:33.134 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:25:33.134 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:25:33.134 INFO:teuthology.orchestra.run.smithi100.stdout: "public_addr": "172.21.15.135:6812/160304773", 2024-09-17T07:25:33.134 INFO:teuthology.orchestra.run.smithi100.stdout: "cluster_addr": "172.21.15.135:6813/160304773", 2024-09-17T07:25:33.134 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_back_addr": "172.21.15.135:6815/160304773", 2024-09-17T07:25:33.134 INFO:teuthology.orchestra.run.smithi100.stdout: "heartbeat_front_addr": "172.21.15.135:6814/160304773", 2024-09-17T07:25:33.135 INFO:teuthology.orchestra.run.smithi100.stdout: "state": [ 2024-09-17T07:25:33.135 INFO:teuthology.orchestra.run.smithi100.stdout: "exists", 2024-09-17T07:25:33.135 INFO:teuthology.orchestra.run.smithi100.stdout: "up" 2024-09-17T07:25:33.135 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:25:33.135 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:25:33.135 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-17T07:25:33.136 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_xinfo": [ 2024-09-17T07:25:33.136 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:25:33.136 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": 0, 2024-09-17T07:25:33.136 INFO:teuthology.orchestra.run.smithi100.stdout: "down_stamp": "0.000000", 2024-09-17T07:25:33.136 INFO:teuthology.orchestra.run.smithi100.stdout: "laggy_probability": 0, 2024-09-17T07:25:33.136 INFO:teuthology.orchestra.run.smithi100.stdout: "laggy_interval": 0, 2024-09-17T07:25:33.137 INFO:teuthology.orchestra.run.smithi100.stdout: "features": 4540138322906710015, 2024-09-17T07:25:33.137 INFO:teuthology.orchestra.run.smithi100.stdout: "old_weight": 0, 2024-09-17T07:25:33.137 INFO:teuthology.orchestra.run.smithi100.stdout: "last_purged_snaps_scrub": "2024-09-17T07:24:52.599093+0000", 2024-09-17T07:25:33.137 INFO:teuthology.orchestra.run.smithi100.stdout: "dead_epoch": 0 2024-09-17T07:25:33.137 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:25:33.137 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:25:33.137 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": 1, 2024-09-17T07:25:33.138 INFO:teuthology.orchestra.run.smithi100.stdout: "down_stamp": "0.000000", 2024-09-17T07:25:33.138 INFO:teuthology.orchestra.run.smithi100.stdout: "laggy_probability": 0, 2024-09-17T07:25:33.138 INFO:teuthology.orchestra.run.smithi100.stdout: "laggy_interval": 0, 2024-09-17T07:25:33.138 INFO:teuthology.orchestra.run.smithi100.stdout: "features": 4540138322906710015, 2024-09-17T07:25:33.138 INFO:teuthology.orchestra.run.smithi100.stdout: "old_weight": 0, 2024-09-17T07:25:33.138 INFO:teuthology.orchestra.run.smithi100.stdout: "last_purged_snaps_scrub": "2024-09-17T07:25:24.346932+0000", 2024-09-17T07:25:33.138 INFO:teuthology.orchestra.run.smithi100.stdout: "dead_epoch": 302 2024-09-17T07:25:33.139 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:25:33.139 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:25:33.139 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": 2, 2024-09-17T07:25:33.139 INFO:teuthology.orchestra.run.smithi100.stdout: "down_stamp": "0.000000", 2024-09-17T07:25:33.139 INFO:teuthology.orchestra.run.smithi100.stdout: "laggy_probability": 0, 2024-09-17T07:25:33.139 INFO:teuthology.orchestra.run.smithi100.stdout: "laggy_interval": 0, 2024-09-17T07:25:33.139 INFO:teuthology.orchestra.run.smithi100.stdout: "features": 4540138322906710015, 2024-09-17T07:25:33.140 INFO:teuthology.orchestra.run.smithi100.stdout: "old_weight": 0, 2024-09-17T07:25:33.140 INFO:teuthology.orchestra.run.smithi100.stdout: "last_purged_snaps_scrub": "2024-09-17T07:24:51.000724+0000", 2024-09-17T07:25:33.140 INFO:teuthology.orchestra.run.smithi100.stdout: "dead_epoch": 0 2024-09-17T07:25:33.140 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:25:33.140 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:25:33.140 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": 3, 2024-09-17T07:25:33.141 INFO:teuthology.orchestra.run.smithi100.stdout: "down_stamp": "0.000000", 2024-09-17T07:25:33.141 INFO:teuthology.orchestra.run.smithi100.stdout: "laggy_probability": 0, 2024-09-17T07:25:33.141 INFO:teuthology.orchestra.run.smithi100.stdout: "laggy_interval": 0, 2024-09-17T07:25:33.141 INFO:teuthology.orchestra.run.smithi100.stdout: "features": 4540138322906710015, 2024-09-17T07:25:33.141 INFO:teuthology.orchestra.run.smithi100.stdout: "old_weight": 65536, 2024-09-17T07:25:33.141 INFO:teuthology.orchestra.run.smithi100.stdout: "last_purged_snaps_scrub": "2024-09-17T07:24:47.772351+0000", 2024-09-17T07:25:33.142 INFO:teuthology.orchestra.run.smithi100.stdout: "dead_epoch": 0 2024-09-17T07:25:33.142 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:25:33.142 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:25:33.142 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": 4, 2024-09-17T07:25:33.142 INFO:teuthology.orchestra.run.smithi100.stdout: "down_stamp": "0.000000", 2024-09-17T07:25:33.142 INFO:teuthology.orchestra.run.smithi100.stdout: "laggy_probability": 0, 2024-09-17T07:25:33.143 INFO:teuthology.orchestra.run.smithi100.stdout: "laggy_interval": 0, 2024-09-17T07:25:33.143 INFO:teuthology.orchestra.run.smithi100.stdout: "features": 4540138322906710015, 2024-09-17T07:25:33.143 INFO:teuthology.orchestra.run.smithi100.stdout: "old_weight": 0, 2024-09-17T07:25:33.143 INFO:teuthology.orchestra.run.smithi100.stdout: "last_purged_snaps_scrub": "2024-09-17T07:25:05.353608+0000", 2024-09-17T07:25:33.143 INFO:teuthology.orchestra.run.smithi100.stdout: "dead_epoch": 0 2024-09-17T07:25:33.143 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:25:33.143 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:25:33.144 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": 5, 2024-09-17T07:25:33.144 INFO:teuthology.orchestra.run.smithi100.stdout: "down_stamp": "0.000000", 2024-09-17T07:25:33.144 INFO:teuthology.orchestra.run.smithi100.stdout: "laggy_probability": 0, 2024-09-17T07:25:33.144 INFO:teuthology.orchestra.run.smithi100.stdout: "laggy_interval": 0, 2024-09-17T07:25:33.144 INFO:teuthology.orchestra.run.smithi100.stdout: "features": 4540138322906710015, 2024-09-17T07:25:33.144 INFO:teuthology.orchestra.run.smithi100.stdout: "old_weight": 0, 2024-09-17T07:25:33.144 INFO:teuthology.orchestra.run.smithi100.stdout: "last_purged_snaps_scrub": "2024-09-17T07:25:20.769002+0000", 2024-09-17T07:25:33.145 INFO:teuthology.orchestra.run.smithi100.stdout: "dead_epoch": 446 2024-09-17T07:25:33.145 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:25:33.145 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:25:33.145 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": 6, 2024-09-17T07:25:33.145 INFO:teuthology.orchestra.run.smithi100.stdout: "down_stamp": "0.000000", 2024-09-17T07:25:33.145 INFO:teuthology.orchestra.run.smithi100.stdout: "laggy_probability": 0, 2024-09-17T07:25:33.145 INFO:teuthology.orchestra.run.smithi100.stdout: "laggy_interval": 0, 2024-09-17T07:25:33.146 INFO:teuthology.orchestra.run.smithi100.stdout: "features": 4540138322906710015, 2024-09-17T07:25:33.146 INFO:teuthology.orchestra.run.smithi100.stdout: "old_weight": 0, 2024-09-17T07:25:33.146 INFO:teuthology.orchestra.run.smithi100.stdout: "last_purged_snaps_scrub": "2024-09-17T07:25:12.035252+0000", 2024-09-17T07:25:33.146 INFO:teuthology.orchestra.run.smithi100.stdout: "dead_epoch": 0 2024-09-17T07:25:33.146 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:25:33.146 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:25:33.146 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": 7, 2024-09-17T07:25:33.147 INFO:teuthology.orchestra.run.smithi100.stdout: "down_stamp": "0.000000", 2024-09-17T07:25:33.147 INFO:teuthology.orchestra.run.smithi100.stdout: "laggy_probability": 0, 2024-09-17T07:25:33.147 INFO:teuthology.orchestra.run.smithi100.stdout: "laggy_interval": 0, 2024-09-17T07:25:33.147 INFO:teuthology.orchestra.run.smithi100.stdout: "features": 4540138322906710015, 2024-09-17T07:25:33.147 INFO:teuthology.orchestra.run.smithi100.stdout: "old_weight": 0, 2024-09-17T07:25:33.147 INFO:teuthology.orchestra.run.smithi100.stdout: "last_purged_snaps_scrub": "2024-09-17T07:24:45.673839+0000", 2024-09-17T07:25:33.147 INFO:teuthology.orchestra.run.smithi100.stdout: "dead_epoch": 0 2024-09-17T07:25:33.148 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:25:33.148 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-17T07:25:33.148 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_upmap": [], 2024-09-17T07:25:33.148 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_upmap_items": [ 2024-09-17T07:25:33.148 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:25:33.148 INFO:teuthology.orchestra.run.smithi100.stdout: "pgid": "4.6", 2024-09-17T07:25:33.149 INFO:teuthology.orchestra.run.smithi100.stdout: "mappings": [ 2024-09-17T07:25:33.149 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:25:33.149 INFO:teuthology.orchestra.run.smithi100.stdout: "from": 1, 2024-09-17T07:25:33.149 INFO:teuthology.orchestra.run.smithi100.stdout: "to": 5 2024-09-17T07:25:33.149 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:25:33.149 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:25:33.149 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:25:33.150 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:25:33.150 INFO:teuthology.orchestra.run.smithi100.stdout: "pgid": "4.9", 2024-09-17T07:25:33.150 INFO:teuthology.orchestra.run.smithi100.stdout: "mappings": [ 2024-09-17T07:25:33.150 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:25:33.150 INFO:teuthology.orchestra.run.smithi100.stdout: "from": 1, 2024-09-17T07:25:33.150 INFO:teuthology.orchestra.run.smithi100.stdout: "to": 0 2024-09-17T07:25:33.151 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:25:33.151 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:25:33.151 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:25:33.151 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:25:33.151 INFO:teuthology.orchestra.run.smithi100.stdout: "pgid": "4.10", 2024-09-17T07:25:33.151 INFO:teuthology.orchestra.run.smithi100.stdout: "mappings": [ 2024-09-17T07:25:33.151 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:25:33.152 INFO:teuthology.orchestra.run.smithi100.stdout: "from": 1, 2024-09-17T07:25:33.152 INFO:teuthology.orchestra.run.smithi100.stdout: "to": 2 2024-09-17T07:25:33.152 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:25:33.152 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:25:33.152 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:25:33.152 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:25:33.153 INFO:teuthology.orchestra.run.smithi100.stdout: "pgid": "4.1e", 2024-09-17T07:25:33.153 INFO:teuthology.orchestra.run.smithi100.stdout: "mappings": [ 2024-09-17T07:25:33.153 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:25:33.153 INFO:teuthology.orchestra.run.smithi100.stdout: "from": 1, 2024-09-17T07:25:33.153 INFO:teuthology.orchestra.run.smithi100.stdout: "to": 5 2024-09-17T07:25:33.153 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:25:33.154 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:25:33.154 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:25:33.154 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:25:33.154 INFO:teuthology.orchestra.run.smithi100.stdout: "pgid": "4.1f", 2024-09-17T07:25:33.154 INFO:teuthology.orchestra.run.smithi100.stdout: "mappings": [ 2024-09-17T07:25:33.154 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-17T07:25:33.155 INFO:teuthology.orchestra.run.smithi100.stdout: "from": 1, 2024-09-17T07:25:33.155 INFO:teuthology.orchestra.run.smithi100.stdout: "to": 2 2024-09-17T07:25:33.155 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:25:33.155 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-17T07:25:33.155 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:25:33.155 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-17T07:25:33.156 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_upmap_primaries": [], 2024-09-17T07:25:33.156 INFO:teuthology.orchestra.run.smithi100.stdout: "pg_temp": [], 2024-09-17T07:25:33.156 INFO:teuthology.orchestra.run.smithi100.stdout: "primary_temp": [], 2024-09-17T07:25:33.156 INFO:teuthology.orchestra.run.smithi100.stdout: "blocklist": { 2024-09-17T07:25:33.156 INFO:teuthology.orchestra.run.smithi100.stdout: "172.21.15.100:0/3796664909": "2024-09-18T06:57:35.678309+0000", 2024-09-17T07:25:33.156 INFO:teuthology.orchestra.run.smithi100.stdout: "172.21.15.100:0/2225586217": "2024-09-18T07:02:46.981486+0000", 2024-09-17T07:25:33.157 INFO:teuthology.orchestra.run.smithi100.stdout: "172.21.15.100:0/1626706829": "2024-09-18T06:57:35.678309+0000", 2024-09-17T07:25:33.157 INFO:teuthology.orchestra.run.smithi100.stdout: "172.21.15.100:0/3452172206": "2024-09-18T06:57:35.678309+0000", 2024-09-17T07:25:33.157 INFO:teuthology.orchestra.run.smithi100.stdout: "172.21.15.100:6800/3992296558": "2024-09-18T06:57:58.526309+0000", 2024-09-17T07:25:33.157 INFO:teuthology.orchestra.run.smithi100.stdout: "172.21.15.100:6800/3498574214": "2024-09-18T06:57:35.678309+0000", 2024-09-17T07:25:33.157 INFO:teuthology.orchestra.run.smithi100.stdout: "172.21.15.100:0/1365215411": "2024-09-18T06:57:58.526309+0000", 2024-09-17T07:25:33.157 INFO:teuthology.orchestra.run.smithi100.stdout: "172.21.15.100:0/3362169179": "2024-09-18T07:02:46.981486+0000", 2024-09-17T07:25:33.158 INFO:teuthology.orchestra.run.smithi100.stdout: "172.21.15.100:0/2573779025": "2024-09-18T07:02:46.981486+0000", 2024-09-17T07:25:33.158 INFO:teuthology.orchestra.run.smithi100.stdout: "172.21.15.100:0/380524491": "2024-09-18T07:02:46.981486+0000", 2024-09-17T07:25:33.158 INFO:teuthology.orchestra.run.smithi100.stdout: "172.21.15.100:0/1657291976": "2024-09-18T06:57:58.526309+0000", 2024-09-17T07:25:33.158 INFO:teuthology.orchestra.run.smithi100.stdout: "172.21.15.100:0/1676314112": "2024-09-18T07:02:46.981486+0000", 2024-09-17T07:25:33.158 INFO:teuthology.orchestra.run.smithi100.stdout: "172.21.15.100:0/1255197255": "2024-09-18T06:57:58.526309+0000", 2024-09-17T07:25:33.159 INFO:teuthology.orchestra.run.smithi100.stdout: "172.21.15.100:6800/3472658181": "2024-09-18T07:02:46.981486+0000" 2024-09-17T07:25:33.159 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:25:33.159 INFO:teuthology.orchestra.run.smithi100.stdout: "range_blocklist": {}, 2024-09-17T07:25:33.159 INFO:teuthology.orchestra.run.smithi100.stdout: "erasure_code_profiles": { 2024-09-17T07:25:33.159 INFO:teuthology.orchestra.run.smithi100.stdout: "default": { 2024-09-17T07:25:33.159 INFO:teuthology.orchestra.run.smithi100.stdout: "crush-failure-domain": "osd", 2024-09-17T07:25:33.159 INFO:teuthology.orchestra.run.smithi100.stdout: "k": "2", 2024-09-17T07:25:33.160 INFO:teuthology.orchestra.run.smithi100.stdout: "m": "1", 2024-09-17T07:25:33.160 INFO:teuthology.orchestra.run.smithi100.stdout: "plugin": "jerasure", 2024-09-17T07:25:33.160 INFO:teuthology.orchestra.run.smithi100.stdout: "technique": "reed_sol_van" 2024-09-17T07:25:33.160 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:25:33.160 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-17T07:25:33.160 INFO:teuthology.orchestra.run.smithi100.stdout: "removed_snaps_queue": [], 2024-09-17T07:25:33.160 INFO:teuthology.orchestra.run.smithi100.stdout: "new_removed_snaps": [], 2024-09-17T07:25:33.161 INFO:teuthology.orchestra.run.smithi100.stdout: "new_purged_snaps": [], 2024-09-17T07:25:33.161 INFO:teuthology.orchestra.run.smithi100.stdout: "crush_node_flags": {}, 2024-09-17T07:25:33.161 INFO:teuthology.orchestra.run.smithi100.stdout: "device_class_flags": {}, 2024-09-17T07:25:33.161 INFO:teuthology.orchestra.run.smithi100.stdout: "stretch_mode": { 2024-09-17T07:25:33.161 INFO:teuthology.orchestra.run.smithi100.stdout: "stretch_mode_enabled": false, 2024-09-17T07:25:33.161 INFO:teuthology.orchestra.run.smithi100.stdout: "stretch_bucket_count": 0, 2024-09-17T07:25:33.162 INFO:teuthology.orchestra.run.smithi100.stdout: "degraded_stretch_mode": 0, 2024-09-17T07:25:33.162 INFO:teuthology.orchestra.run.smithi100.stdout: "recovering_stretch_mode": 0, 2024-09-17T07:25:33.162 INFO:teuthology.orchestra.run.smithi100.stdout: "stretch_mode_bucket": 0 2024-09-17T07:25:33.162 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-17T07:25:33.162 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-17T07:25:33.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:33 smithi100 ceph-mon[29092]: 5.3 scrub starts 2024-09-17T07:25:33.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:33 smithi100 ceph-mon[29092]: 5.3 scrub ok 2024-09-17T07:25:33.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:33 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2705222519' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-17T07:25:33.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:33 smithi100 ceph-mon[29092]: 1.a scrub starts 2024-09-17T07:25:33.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:33 smithi100 ceph-mon[29092]: 1.a scrub ok 2024-09-17T07:25:33.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:33 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:25:33.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:33 smithi100 ceph-mon[39462]: 5.3 scrub starts 2024-09-17T07:25:33.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:33 smithi100 ceph-mon[39462]: 5.3 scrub ok 2024-09-17T07:25:33.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:33 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2705222519' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-17T07:25:33.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:33 smithi100 ceph-mon[39462]: 1.a scrub starts 2024-09-17T07:25:33.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:33 smithi100 ceph-mon[39462]: 1.a scrub ok 2024-09-17T07:25:33.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:33 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:25:34.122 INFO:tasks.thrashosds.thrasher:j is {'epoch': 513, 'fsid': 'e848267c-74c1-11ef-bceb-c7b262605968', 'created': '2024-09-17T06:57:05.010532+0000', 'modified': '2024-09-17T07:25:29.777227+0000', 'last_up_change': '2024-09-17T07:23:56.453211+0000', 'last_in_change': '2024-09-17T07:22:12.524102+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 20, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 11, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'reef', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-17T07:00:08.816136+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '375', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '374', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 2.9000000953674316, 'score_stable': 2.9000000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.549999952316284, 'raw_score_stable': 2.549999952316284, 'primary_affinity_weighted': 0.8959827423095703, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-09-17T07:02:10.759826+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.8959827423095703, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 3, 'pool_name': 'datapool', 'create_time': '2024-09-17T07:02:10.991289+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 72, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 2.6700000762939453, 'score_stable': 2.6700000762939453, 'optimal_score': 0.75, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8959827423095703, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-17T07:02:12.291462+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_acting': 2.240000009536743, 'score_stable': 2.240000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.9700000286102295, 'raw_score_stable': 1.9700000286102295, 'primary_affinity_weighted': 0.8959827423095703, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-17T07:02:14.253214+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '73', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_acting': 1.4900000095367432, 'score_stable': 1.4900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.309999942779541, 'raw_score_stable': 1.309999942779541, 'primary_affinity_weighted': 0.8959827423095703, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-17T07:02:16.310620+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.8959827423095703, 'average_primary_affinity': 0.8999999761581421, 'average_primary_affinity_weighted': 0.8999999761581421}}], 'osds': [{'osd': 0, 'uuid': '27bee9ec-05e6-406c-8bf1-168098da0ec7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 491, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6801', 'nonce': 3341017455}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6802', 'nonce': 3341017455}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6804', 'nonce': 3341017455}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6803', 'nonce': 3341017455}]}, 'public_addr': '172.21.15.100:6801/3341017455', 'cluster_addr': '172.21.15.100:6802/3341017455', 'heartbeat_back_addr': '172.21.15.100:6804/3341017455', 'heartbeat_front_addr': '172.21.15.100:6803/3341017455', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'b1a30c9c-eeaf-478e-9b62-2d63ad45873d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 13, 'last_clean_end': 302, 'up_from': 413, 'up_thru': 491, 'down_at': 303, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6805', 'nonce': 2894367777}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6806', 'nonce': 2894367777}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6808', 'nonce': 2894367777}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6807', 'nonce': 2894367777}]}, 'public_addr': '172.21.15.100:6805/2894367777', 'cluster_addr': '172.21.15.100:6806/2894367777', 'heartbeat_back_addr': '172.21.15.100:6808/2894367777', 'heartbeat_front_addr': '172.21.15.100:6807/2894367777', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'ce542e0d-9915-465e-aa11-7b3ce13f694f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.1678619384765625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 476, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6809', 'nonce': 714579846}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6810', 'nonce': 714579846}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6812', 'nonce': 714579846}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6811', 'nonce': 714579846}]}, 'public_addr': '172.21.15.100:6809/714579846', 'cluster_addr': '172.21.15.100:6810/714579846', 'heartbeat_back_addr': '172.21.15.100:6812/714579846', 'heartbeat_front_addr': '172.21.15.100:6811/714579846', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'a1257575-3c90-41de-88f6-3960744351b1', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 27, 'up_thru': 168, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6813', 'nonce': 2315956572}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6814', 'nonce': 2315956572}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6816', 'nonce': 2315956572}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6815', 'nonce': 2315956572}]}, 'public_addr': '172.21.15.100:6813/2315956572', 'cluster_addr': '172.21.15.100:6814/2315956572', 'heartbeat_back_addr': '172.21.15.100:6816/2315956572', 'heartbeat_front_addr': '172.21.15.100:6815/2315956572', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'e44e8922-2c5e-44e3-b556-d4dfbc499649', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 33, 'up_thru': 504, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.135:6800', 'nonce': 1148820922}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.135:6801', 'nonce': 1148820922}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.135:6803', 'nonce': 1148820922}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.135:6802', 'nonce': 1148820922}]}, 'public_addr': '172.21.15.135:6800/1148820922', 'cluster_addr': '172.21.15.135:6801/1148820922', 'heartbeat_back_addr': '172.21.15.135:6803/1148820922', 'heartbeat_front_addr': '172.21.15.135:6802/1148820922', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '91e4a362-fcdb-41b9-b54e-716591ba6d79', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 42, 'last_clean_end': 446, 'up_from': 476, 'up_thru': 483, 'down_at': 447, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.135:6804', 'nonce': 1324312753}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.135:6805', 'nonce': 1324312753}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.135:6807', 'nonce': 1324312753}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.135:6806', 'nonce': 1324312753}]}, 'public_addr': '172.21.15.135:6804/1324312753', 'cluster_addr': '172.21.15.135:6805/1324312753', 'heartbeat_back_addr': '172.21.15.135:6807/1324312753', 'heartbeat_front_addr': '172.21.15.135:6806/1324312753', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'de799f64-ebd2-4a46-91f6-c2f276bee21f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 50, 'up_thru': 486, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.135:6808', 'nonce': 2750184152}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.135:6809', 'nonce': 2750184152}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.135:6811', 'nonce': 2750184152}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.135:6810', 'nonce': 2750184152}]}, 'public_addr': '172.21.15.135:6808/2750184152', 'cluster_addr': '172.21.15.135:6809/2750184152', 'heartbeat_back_addr': '172.21.15.135:6811/2750184152', 'heartbeat_front_addr': '172.21.15.135:6810/2750184152', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '830932c7-5de1-44ff-a840-e53a6e859bda', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 60, 'up_thru': 485, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.135:6812', 'nonce': 160304773}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.135:6813', 'nonce': 160304773}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.135:6815', 'nonce': 160304773}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.135:6814', 'nonce': 160304773}]}, 'public_addr': '172.21.15.135:6812/160304773', 'cluster_addr': '172.21.15.135:6813/160304773', 'heartbeat_back_addr': '172.21.15.135:6815/160304773', 'heartbeat_front_addr': '172.21.15.135:6814/160304773', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T07:24:52.599093+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T07:25:24.346932+0000', 'dead_epoch': 302}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T07:24:51.000724+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-17T07:24:47.772351+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T07:25:05.353608+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T07:25:20.769002+0000', 'dead_epoch': 446}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T07:25:12.035252+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T07:24:45.673839+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '4.6', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.9', 'mappings': [{'from': 1, 'to': 0}]}, {'pgid': '4.10', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1e', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.1f', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.100:0/3796664909': '2024-09-18T06:57:35.678309+0000', '172.21.15.100:0/2225586217': '2024-09-18T07:02:46.981486+0000', '172.21.15.100:0/1626706829': '2024-09-18T06:57:35.678309+0000', '172.21.15.100:0/3452172206': '2024-09-18T06:57:35.678309+0000', '172.21.15.100:6800/3992296558': '2024-09-18T06:57:58.526309+0000', '172.21.15.100:6800/3498574214': '2024-09-18T06:57:35.678309+0000', '172.21.15.100:0/1365215411': '2024-09-18T06:57:58.526309+0000', '172.21.15.100:0/3362169179': '2024-09-18T07:02:46.981486+0000', '172.21.15.100:0/2573779025': '2024-09-18T07:02:46.981486+0000', '172.21.15.100:0/380524491': '2024-09-18T07:02:46.981486+0000', '172.21.15.100:0/1657291976': '2024-09-18T06:57:58.526309+0000', '172.21.15.100:0/1676314112': '2024-09-18T07:02:46.981486+0000', '172.21.15.100:0/1255197255': '2024-09-18T06:57:58.526309+0000', '172.21.15.100:6800/3472658181': '2024-09-18T07:02:46.981486+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-17T07:25:34.123 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T07:25:34.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:33 smithi135 ceph-mon[33238]: 5.3 scrub starts 2024-09-17T07:25:34.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:33 smithi135 ceph-mon[33238]: 5.3 scrub ok 2024-09-17T07:25:34.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:33 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2705222519' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-17T07:25:34.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:33 smithi135 ceph-mon[33238]: 1.a scrub starts 2024-09-17T07:25:34.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:33 smithi135 ceph-mon[33238]: 1.a scrub ok 2024-09-17T07:25:34.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:33 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:25:34.319 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:25:35.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:34 smithi100 ceph-mon[29092]: pgmap v1121: 142 pgs: 142 active+clean; 693 KiB data, 6.5 GiB used, 619 GiB / 626 GiB avail 2024-09-17T07:25:35.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:34 smithi100 ceph-mon[29092]: 6.b scrub starts 2024-09-17T07:25:35.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:34 smithi100 ceph-mon[29092]: 6.b scrub ok 2024-09-17T07:25:35.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:34 smithi100 ceph-mon[39462]: pgmap v1121: 142 pgs: 142 active+clean; 693 KiB data, 6.5 GiB used, 619 GiB / 626 GiB avail 2024-09-17T07:25:35.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:34 smithi100 ceph-mon[39462]: 6.b scrub starts 2024-09-17T07:25:35.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:34 smithi100 ceph-mon[39462]: 6.b scrub ok 2024-09-17T07:25:35.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:34 smithi135 ceph-mon[33238]: pgmap v1121: 142 pgs: 142 active+clean; 693 KiB data, 6.5 GiB used, 619 GiB / 626 GiB avail 2024-09-17T07:25:35.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:34 smithi135 ceph-mon[33238]: 6.b scrub starts 2024-09-17T07:25:35.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:34 smithi135 ceph-mon[33238]: 6.b scrub ok 2024-09-17T07:25:36.062 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:35 smithi100 ceph-mon[29092]: 6.f scrub starts 2024-09-17T07:25:36.062 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:35 smithi100 ceph-mon[29092]: 6.f scrub ok 2024-09-17T07:25:36.062 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:35 smithi100 ceph-mon[39462]: 6.f scrub starts 2024-09-17T07:25:36.062 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:35 smithi100 ceph-mon[39462]: 6.f scrub ok 2024-09-17T07:25:36.062 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:25:35 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:25:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:25:36.238 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T07:25:36.240 INFO:teuthology.orchestra.run.smithi100.stderr:dumped all 2024-09-17T07:25:36.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:35 smithi135 ceph-mon[33238]: 6.f scrub starts 2024-09-17T07:25:36.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:35 smithi135 ceph-mon[33238]: 6.f scrub ok 2024-09-17T07:25:37.010 INFO:teuthology.orchestra.run.smithi100.stdout:{"pg_ready":true,"pg_map":{"version":1122,"stamp":"2024-09-17T07:25:35.521050+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":709246,"num_objects":197,"num_object_clones":0,"num_object_copies":591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":197,"num_whiteouts":0,"num_read":1949,"num_read_kb":1759,"num_write":1117,"num_write_kb":1579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":273,"num_bytes_recovered":1133229,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2580,"ondisk_log_size":2580,"up":426,"acting":426,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":431,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":6749364,"kb_used_data":516364,"kb_used_omap":16,"kb_used_meta":6231471,"kb_avail":649466700,"statfs":{"total":671965249536,"available":665053900800,"internally_reserved":0,"allocated":528756736,"data_stored":521644190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16844,"internal_metadata":6381026868},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":274010,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":-9,"num_read_kb":-18,"num_write":-16,"num_write_kb":-220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1,"num_bytes_recovered":-161010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.010918"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":101,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:25.504551+0000","last_change":"2024-09-17T07:25:17.976501+0000","last_active":"2024-09-17T07:25:25.504551+0000","last_peered":"2024-09-17T07:25:25.504551+0000","last_clean":"2024-09-17T07:25:25.504551+0000","last_became_active":"2024-09-17T07:22:13.586558+0000","last_became_peered":"2024-09-17T07:22:13.586558+0000","last_unstale":"2024-09-17T07:25:25.504551+0000","last_undegraded":"2024-09-17T07:25:25.504551+0000","last_fullsized":"2024-09-17T07:25:25.504551+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:17.976387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:33.644728+0000","last_clean_scrub_stamp":"2024-09-17T07:25:17.976387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:32.601619+0000","scrub_duration":0.0012687289999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":571,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:24.767047+0000","last_change":"2024-09-17T07:23:27.440583+0000","last_active":"2024-09-17T07:25:24.767047+0000","last_peered":"2024-09-17T07:25:24.767047+0000","last_clean":"2024-09-17T07:25:24.767047+0000","last_became_active":"2024-09-17T07:22:13.589527+0000","last_became_peered":"2024-09-17T07:22:13.589527+0000","last_unstale":"2024-09-17T07:25:24.767047+0000","last_undegraded":"2024-09-17T07:25:24.767047+0000","last_fullsized":"2024-09-17T07:25:24.767047+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:23:27.440509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:23:27.440509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001261002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"439'39","reported_seq":495,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:24.767523+0000","last_change":"2024-09-17T07:25:10.604396+0000","last_active":"2024-09-17T07:25:24.767523+0000","last_peered":"2024-09-17T07:25:24.767523+0000","last_clean":"2024-09-17T07:25:24.767523+0000","last_became_active":"2024-09-17T07:23:57.480684+0000","last_became_peered":"2024-09-17T07:23:57.480684+0000","last_unstale":"2024-09-17T07:25:24.767523+0000","last_undegraded":"2024-09-17T07:25:24.767523+0000","last_fullsized":"2024-09-17T07:25:24.767523+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'39","last_scrub_stamp":"2024-09-17T07:25:10.604305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:25:10.604305+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:38.118795+0000","scrub_duration":0.0033772899999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":542,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:24.766110+0000","last_change":"2024-09-17T07:25:04.885823+0000","last_active":"2024-09-17T07:25:24.766110+0000","last_peered":"2024-09-17T07:25:24.766110+0000","last_clean":"2024-09-17T07:25:24.766110+0000","last_became_active":"2024-09-17T07:21:15.354561+0000","last_became_peered":"2024-09-17T07:21:15.354561+0000","last_unstale":"2024-09-17T07:25:24.766110+0000","last_undegraded":"2024-09-17T07:25:24.766110+0000","last_fullsized":"2024-09-17T07:25:24.766110+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:04.885700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:25:04.885700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:21.499532+0000","scrub_duration":0.0034589249999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":408,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:25.199055+0000","last_change":"2024-09-17T07:23:57.483306+0000","last_active":"2024-09-17T07:25:25.199055+0000","last_peered":"2024-09-17T07:25:25.199055+0000","last_clean":"2024-09-17T07:25:25.199055+0000","last_became_active":"2024-09-17T07:23:57.480136+0000","last_became_peered":"2024-09-17T07:23:57.480136+0000","last_unstale":"2024-09-17T07:25:25.199055+0000","last_undegraded":"2024-09-17T07:25:25.199055+0000","last_fullsized":"2024-09-17T07:25:25.199055+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:21:13.454404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:17:35.507742+0000","last_clean_scrub_stamp":"2024-09-17T07:21:13.454404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.016029897000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"438'21","reported_seq":590,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:24.766168+0000","last_change":"2024-09-17T07:24:58.793239+0000","last_active":"2024-09-17T07:25:24.766168+0000","last_peered":"2024-09-17T07:25:24.766168+0000","last_clean":"2024-09-17T07:25:24.766168+0000","last_became_active":"2024-09-17T07:21:15.354718+0000","last_became_peered":"2024-09-17T07:21:15.354718+0000","last_unstale":"2024-09-17T07:25:24.766168+0000","last_undegraded":"2024-09-17T07:25:24.766168+0000","last_fullsized":"2024-09-17T07:25:24.766168+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"438'21","last_scrub_stamp":"2024-09-17T07:24:58.793136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:24:58.793136+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:13.595630+0000","scrub_duration":0.0018950810000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":425,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:24.765756+0000","last_change":"2024-09-17T07:23:57.473889+0000","last_active":"2024-09-17T07:25:24.765756+0000","last_peered":"2024-09-17T07:25:24.765756+0000","last_clean":"2024-09-17T07:25:24.765756+0000","last_became_active":"2024-09-17T07:23:57.473620+0000","last_became_peered":"2024-09-17T07:23:57.473620+0000","last_unstale":"2024-09-17T07:25:24.765756+0000","last_undegraded":"2024-09-17T07:25:24.765756+0000","last_fullsized":"2024-09-17T07:25:24.765756+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:13.660077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:20:13.660077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":556,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:24.766359+0000","last_change":"2024-09-17T07:24:18.740114+0000","last_active":"2024-09-17T07:25:24.766359+0000","last_peered":"2024-09-17T07:25:24.766359+0000","last_clean":"2024-09-17T07:25:24.766359+0000","last_became_active":"2024-09-17T07:23:57.482145+0000","last_became_peered":"2024-09-17T07:23:57.482145+0000","last_unstale":"2024-09-17T07:25:24.766359+0000","last_undegraded":"2024-09-17T07:25:24.766359+0000","last_fullsized":"2024-09-17T07:25:24.766359+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:24:18.740035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:24:18.740035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012092344,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":101,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:25.504770+0000","last_change":"2024-09-17T07:23:56.908306+0000","last_active":"2024-09-17T07:25:25.504770+0000","last_peered":"2024-09-17T07:25:25.504770+0000","last_clean":"2024-09-17T07:25:25.504770+0000","last_became_active":"2024-09-17T07:22:13.615512+0000","last_became_peered":"2024-09-17T07:22:13.615512+0000","last_unstale":"2024-09-17T07:25:25.504770+0000","last_undegraded":"2024-09-17T07:25:25.504770+0000","last_fullsized":"2024-09-17T07:25:25.504770+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:23:56.908208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:46.312371+0000","last_clean_scrub_stamp":"2024-09-17T07:23:56.908208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0012071619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"488'11","reported_seq":580,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:24.766551+0000","last_change":"2024-09-17T07:24:26.774186+0000","last_active":"2024-09-17T07:25:24.766551+0000","last_peered":"2024-09-17T07:25:24.766551+0000","last_clean":"2024-09-17T07:25:24.766551+0000","last_became_active":"2024-09-17T07:21:15.349235+0000","last_became_peered":"2024-09-17T07:21:15.349235+0000","last_unstale":"2024-09-17T07:25:24.766551+0000","last_undegraded":"2024-09-17T07:25:24.766551+0000","last_fullsized":"2024-09-17T07:25:24.766551+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"488'11","last_scrub_stamp":"2024-09-17T07:24:26.774097+0000","last_deep_scrub":"488'11","last_deep_scrub_stamp":"2024-09-17T07:24:26.774097+0000","last_clean_scrub_stamp":"2024-09-17T07:24:26.774097+0000","objects_scrubbed":2,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:40.430722+0000","scrub_duration":0.0025260220000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":13,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":90,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:25.504433+0000","last_change":"2024-09-17T07:22:13.624911+0000","last_active":"2024-09-17T07:25:25.504433+0000","last_peered":"2024-09-17T07:25:25.504433+0000","last_clean":"2024-09-17T07:25:25.504433+0000","last_became_active":"2024-09-17T07:22:13.624441+0000","last_became_peered":"2024-09-17T07:22:13.624441+0000","last_unstale":"2024-09-17T07:25:25.504433+0000","last_undegraded":"2024-09-17T07:25:25.504433+0000","last_fullsized":"2024-09-17T07:25:25.504433+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:10.116580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:20:10.116580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00137765,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":506,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:24.854556+0000","last_change":"2024-09-17T07:24:24.811636+0000","last_active":"2024-09-17T07:25:24.854556+0000","last_peered":"2024-09-17T07:25:24.854556+0000","last_clean":"2024-09-17T07:25:24.854556+0000","last_became_active":"2024-09-17T07:23:57.489317+0000","last_became_peered":"2024-09-17T07:23:57.489317+0000","last_unstale":"2024-09-17T07:25:24.854556+0000","last_undegraded":"2024-09-17T07:25:24.854556+0000","last_fullsized":"2024-09-17T07:25:24.854556+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:24:24.811522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:24:24.811522+0000","last_clean_scrub_stamp":"2024-09-17T07:24:24.811522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:36.491439+0000","scrub_duration":0.001695238,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":116,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:25.503705+0000","last_change":"2024-09-17T07:24:29.884052+0000","last_active":"2024-09-17T07:25:25.503705+0000","last_peered":"2024-09-17T07:25:25.503705+0000","last_clean":"2024-09-17T07:25:25.503705+0000","last_became_active":"2024-09-17T07:23:57.476413+0000","last_became_peered":"2024-09-17T07:23:57.476413+0000","last_unstale":"2024-09-17T07:25:25.503705+0000","last_undegraded":"2024-09-17T07:25:25.503705+0000","last_fullsized":"2024-09-17T07:25:25.503705+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:24:29.883839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:45.029247+0000","last_clean_scrub_stamp":"2024-09-17T07:24:29.883839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:50.216430+0000","scrub_duration":0.0022700179999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"439'25","reported_seq":607,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:24.766613+0000","last_change":"2024-09-17T07:20:39.070372+0000","last_active":"2024-09-17T07:25:24.766613+0000","last_peered":"2024-09-17T07:25:24.766613+0000","last_clean":"2024-09-17T07:25:24.766613+0000","last_became_active":"2024-09-17T07:09:24.397954+0000","last_became_peered":"2024-09-17T07:09:24.397954+0000","last_unstale":"2024-09-17T07:25:24.766613+0000","last_undegraded":"2024-09-17T07:25:24.766613+0000","last_fullsized":"2024-09-17T07:25:24.766613+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"312'19","last_scrub_stamp":"2024-09-17T07:20:39.070235+0000","last_deep_scrub":"312'19","last_deep_scrub_stamp":"2024-09-17T07:16:44.292911+0000","last_clean_scrub_stamp":"2024-09-17T07:20:39.070235+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0028570560000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":543,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:24.854688+0000","last_change":"2024-09-17T07:25:13.136654+0000","last_active":"2024-09-17T07:25:24.854688+0000","last_peered":"2024-09-17T07:25:24.854688+0000","last_clean":"2024-09-17T07:25:24.854688+0000","last_became_active":"2024-09-17T07:23:57.480812+0000","last_became_peered":"2024-09-17T07:23:57.480812+0000","last_unstale":"2024-09-17T07:25:24.854688+0000","last_undegraded":"2024-09-17T07:25:24.854688+0000","last_fullsized":"2024-09-17T07:25:24.854688+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:13.136552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:28.096266+0000","last_clean_scrub_stamp":"2024-09-17T07:25:13.136552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:27.238828+0000","scrub_duration":0.001451686,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":47,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:24.766033+0000","last_change":"2024-09-17T07:25:23.534948+0000","last_active":"2024-09-17T07:25:24.766033+0000","last_peered":"2024-09-17T07:25:24.766033+0000","last_clean":"2024-09-17T07:25:24.766033+0000","last_became_active":"2024-09-17T07:24:20.409247+0000","last_became_peered":"2024-09-17T07:24:20.409247+0000","last_unstale":"2024-09-17T07:25:24.766033+0000","last_undegraded":"2024-09-17T07:25:24.766033+0000","last_fullsized":"2024-09-17T07:25:24.766033+0000","mapping_epoch":487,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":488,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:23.534859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:25:23.534859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:45.649153+0000","scrub_duration":0.001563004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.8","version":"212'130","reported_seq":584,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:24.854390+0000","last_change":"2024-09-17T07:23:57.479148+0000","last_active":"2024-09-17T07:25:24.854390+0000","last_peered":"2024-09-17T07:25:24.854390+0000","last_clean":"2024-09-17T07:25:24.854390+0000","last_became_active":"2024-09-17T07:23:57.478874+0000","last_became_peered":"2024-09-17T07:23:57.478874+0000","last_unstale":"2024-09-17T07:25:24.854390+0000","last_undegraded":"2024-09-17T07:25:24.854390+0000","last_fullsized":"2024-09-17T07:25:24.854390+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":4,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:20:21.626089+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:14:57.052505+0000","last_clean_scrub_stamp":"2024-09-17T07:20:21.626089+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021141889999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":28,"num_write":16,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":161009,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"439'53","reported_seq":634,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:24.767043+0000","last_change":"2024-09-17T07:25:12.896735+0000","last_active":"2024-09-17T07:25:24.767043+0000","last_peered":"2024-09-17T07:25:24.767043+0000","last_clean":"2024-09-17T07:25:24.767043+0000","last_became_active":"2024-09-17T07:21:15.348413+0000","last_became_peered":"2024-09-17T07:21:15.348413+0000","last_unstale":"2024-09-17T07:25:24.767043+0000","last_undegraded":"2024-09-17T07:25:24.767043+0000","last_fullsized":"2024-09-17T07:25:24.767043+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'53","last_scrub_stamp":"2024-09-17T07:25:12.896608+0000","last_deep_scrub":"95'25","last_deep_scrub_stamp":"2024-09-17T07:04:09.450039+0000","last_clean_scrub_stamp":"2024-09-17T07:25:12.896608+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:18.556302+0000","scrub_duration":0.0026638410000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":61,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":116,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:25.503540+0000","last_change":"2024-09-17T07:25:08.851793+0000","last_active":"2024-09-17T07:25:25.503540+0000","last_peered":"2024-09-17T07:25:25.503540+0000","last_clean":"2024-09-17T07:25:25.503540+0000","last_became_active":"2024-09-17T07:23:57.475614+0000","last_became_peered":"2024-09-17T07:23:57.475614+0000","last_unstale":"2024-09-17T07:25:25.503540+0000","last_undegraded":"2024-09-17T07:25:25.503540+0000","last_fullsized":"2024-09-17T07:25:25.503540+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:08.851702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:25:08.851702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:37.060045+0000","scrub_duration":0.00137217,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":375,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:24.765590+0000","last_change":"2024-09-17T07:24:25.977606+0000","last_active":"2024-09-17T07:25:24.765590+0000","last_peered":"2024-09-17T07:25:24.765590+0000","last_clean":"2024-09-17T07:25:24.765590+0000","last_became_active":"2024-09-17T07:21:15.349538+0000","last_became_peered":"2024-09-17T07:21:15.349538+0000","last_unstale":"2024-09-17T07:25:24.765590+0000","last_undegraded":"2024-09-17T07:25:24.765590+0000","last_fullsized":"2024-09-17T07:25:24.765590+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:24:25.974726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:24:25.974726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:55.688544+0000","scrub_duration":0.014578028999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":395,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:24.766883+0000","last_change":"2024-09-17T07:20:40.117298+0000","last_active":"2024-09-17T07:25:24.766883+0000","last_peered":"2024-09-17T07:25:24.766883+0000","last_clean":"2024-09-17T07:25:24.766883+0000","last_became_active":"2024-09-17T07:09:24.396334+0000","last_became_peered":"2024-09-17T07:09:24.396334+0000","last_unstale":"2024-09-17T07:25:24.766883+0000","last_undegraded":"2024-09-17T07:25:24.766883+0000","last_fullsized":"2024-09-17T07:25:24.766883+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:40.117165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:42.984088+0000","last_clean_scrub_stamp":"2024-09-17T07:20:40.117165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0041399059999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"212'130","reported_seq":760,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:25.503347+0000","last_change":"2024-09-17T07:23:57.476746+0000","last_active":"2024-09-17T07:25:25.503347+0000","last_peered":"2024-09-17T07:25:25.503347+0000","last_clean":"2024-09-17T07:25:25.503347+0000","last_became_active":"2024-09-17T07:23:57.476423+0000","last_became_peered":"2024-09-17T07:23:57.476423+0000","last_unstale":"2024-09-17T07:25:25.503347+0000","last_undegraded":"2024-09-17T07:25:25.503347+0000","last_fullsized":"2024-09-17T07:25:25.503347+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":4,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:21:12.403277+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:14:57.052505+0000","last_clean_scrub_stamp":"2024-09-17T07:21:12.403277+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0071578960000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":28,"num_write":16,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":161009,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"439'35","reported_seq":628,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:24.766689+0000","last_change":"2024-09-17T07:24:29.751049+0000","last_active":"2024-09-17T07:25:24.766689+0000","last_peered":"2024-09-17T07:25:24.766689+0000","last_clean":"2024-09-17T07:25:24.766689+0000","last_became_active":"2024-09-17T07:21:15.347129+0000","last_became_peered":"2024-09-17T07:21:15.347129+0000","last_unstale":"2024-09-17T07:25:24.766689+0000","last_undegraded":"2024-09-17T07:25:24.766689+0000","last_fullsized":"2024-09-17T07:25:24.766689+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'35","last_scrub_stamp":"2024-09-17T07:24:29.750888+0000","last_deep_scrub":"439'35","last_deep_scrub_stamp":"2024-09-17T07:24:29.750888+0000","last_clean_scrub_stamp":"2024-09-17T07:24:29.750888+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:46.618846+0000","scrub_duration":0.0097756200000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":435,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-17T07:25:30.921403+0000","last_change":"2024-09-17T07:25:30.921403+0000","last_active":"2024-09-17T07:25:30.921403+0000","last_peered":"2024-09-17T07:25:30.921403+0000","last_clean":"2024-09-17T07:25:30.921403+0000","last_became_active":"2024-09-17T07:23:57.493354+0000","last_became_peered":"2024-09-17T07:23:57.493354+0000","last_unstale":"2024-09-17T07:25:30.921403+0000","last_undegraded":"2024-09-17T07:25:30.921403+0000","last_fullsized":"2024-09-17T07:25:30.921403+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:30.921316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:25:30.921316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:37.107632+0000","scrub_duration":0.001370341,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"478'13","reported_seq":162,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-17T07:25:29.973431+0000","last_change":"2024-09-17T07:24:25.818322+0000","last_active":"2024-09-17T07:25:29.973431+0000","last_peered":"2024-09-17T07:25:29.973431+0000","last_clean":"2024-09-17T07:25:29.973431+0000","last_became_active":"2024-09-17T07:23:57.480120+0000","last_became_peered":"2024-09-17T07:23:57.480120+0000","last_unstale":"2024-09-17T07:25:29.973431+0000","last_undegraded":"2024-09-17T07:25:29.973431+0000","last_fullsized":"2024-09-17T07:25:29.973431+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"478'13","last_scrub_stamp":"2024-09-17T07:24:25.815560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:24:25.815560+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:54.513977+0000","scrub_duration":0.0054701139999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":554,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:24.767032+0000","last_change":"2024-09-17T07:23:55.705234+0000","last_active":"2024-09-17T07:25:24.767032+0000","last_peered":"2024-09-17T07:25:24.767032+0000","last_clean":"2024-09-17T07:25:24.767032+0000","last_became_active":"2024-09-17T07:21:15.348799+0000","last_became_peered":"2024-09-17T07:21:15.348799+0000","last_unstale":"2024-09-17T07:25:24.767032+0000","last_undegraded":"2024-09-17T07:25:24.767032+0000","last_fullsized":"2024-09-17T07:25:24.767032+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:23:55.705025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:14:50.554591+0000","last_clean_scrub_stamp":"2024-09-17T07:23:55.705025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0058854049999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"212'130","reported_seq":585,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:24.854966+0000","last_change":"2024-09-17T07:23:57.494642+0000","last_active":"2024-09-17T07:25:24.854966+0000","last_peered":"2024-09-17T07:25:24.854966+0000","last_clean":"2024-09-17T07:25:24.854966+0000","last_became_active":"2024-09-17T07:23:57.494014+0000","last_became_peered":"2024-09-17T07:23:57.494014+0000","last_unstale":"2024-09-17T07:25:24.854966+0000","last_undegraded":"2024-09-17T07:25:24.854966+0000","last_fullsized":"2024-09-17T07:25:24.854966+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":4,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:20:33.709901+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:14:57.052505+0000","last_clean_scrub_stamp":"2024-09-17T07:20:33.709901+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021141889999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":27,"num_write":16,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":161009,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"439'39","reported_seq":496,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:24.767300+0000","last_change":"2024-09-17T07:22:13.780063+0000","last_active":"2024-09-17T07:25:24.767300+0000","last_peered":"2024-09-17T07:25:24.767300+0000","last_clean":"2024-09-17T07:25:24.767300+0000","last_became_active":"2024-09-17T07:22:13.589782+0000","last_became_peered":"2024-09-17T07:22:13.589782+0000","last_unstale":"2024-09-17T07:25:24.767300+0000","last_undegraded":"2024-09-17T07:25:24.767300+0000","last_fullsized":"2024-09-17T07:25:24.767300+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'27","last_scrub_stamp":"2024-09-17T07:20:13.130718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:20:13.130718+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0081162040000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":423,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-17T07:25:29.788547+0000","last_change":"2024-09-17T07:22:13.592519+0000","last_active":"2024-09-17T07:25:29.788547+0000","last_peered":"2024-09-17T07:25:29.788547+0000","last_clean":"2024-09-17T07:25:29.788547+0000","last_became_active":"2024-09-17T07:22:13.591089+0000","last_became_peered":"2024-09-17T07:22:13.591089+0000","last_unstale":"2024-09-17T07:25:29.788547+0000","last_undegraded":"2024-09-17T07:25:29.788547+0000","last_fullsized":"2024-09-17T07:25:29.788547+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:14.001258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:20:14.001258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034116369999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"478'10","reported_seq":854,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-17T07:25:30.772724+0000","last_change":"2024-09-17T07:25:16.842178+0000","last_active":"2024-09-17T07:25:30.772724+0000","last_peered":"2024-09-17T07:25:30.772724+0000","last_clean":"2024-09-17T07:25:30.772724+0000","last_became_active":"2024-09-17T07:23:57.473239+0000","last_became_peered":"2024-09-17T07:23:57.473239+0000","last_unstale":"2024-09-17T07:25:30.772724+0000","last_undegraded":"2024-09-17T07:25:30.772724+0000","last_fullsized":"2024-09-17T07:25:30.772724+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"478'10","last_scrub_stamp":"2024-09-17T07:25:16.842081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:25:16.842081+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:28.575493+0000","scrub_duration":0.001742511,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":497,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:24.855247+0000","last_change":"2024-09-17T07:24:29.971120+0000","last_active":"2024-09-17T07:25:24.855247+0000","last_peered":"2024-09-17T07:25:24.855247+0000","last_clean":"2024-09-17T07:25:24.855247+0000","last_became_active":"2024-09-17T07:23:57.522025+0000","last_became_peered":"2024-09-17T07:23:57.522025+0000","last_unstale":"2024-09-17T07:25:24.855247+0000","last_undegraded":"2024-09-17T07:25:24.855247+0000","last_fullsized":"2024-09-17T07:25:24.855247+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:24:29.971009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:24:29.971009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:39.187110+0000","scrub_duration":0.002296789,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"488'32","reported_seq":605,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:24.767143+0000","last_change":"2024-09-17T07:24:27.786116+0000","last_active":"2024-09-17T07:25:24.767143+0000","last_peered":"2024-09-17T07:25:24.767143+0000","last_clean":"2024-09-17T07:25:24.767143+0000","last_became_active":"2024-09-17T07:21:15.347662+0000","last_became_peered":"2024-09-17T07:21:15.347662+0000","last_unstale":"2024-09-17T07:25:24.767143+0000","last_undegraded":"2024-09-17T07:25:24.767143+0000","last_fullsized":"2024-09-17T07:25:24.767143+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"488'32","last_scrub_stamp":"2024-09-17T07:24:27.785661+0000","last_deep_scrub":"488'32","last_deep_scrub_stamp":"2024-09-17T07:24:27.785661+0000","last_clean_scrub_stamp":"2024-09-17T07:24:27.785661+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:32.181225+0000","scrub_duration":0.021517458999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":101,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:29.972825+0000","last_change":"2024-09-17T07:25:29.972825+0000","last_active":"2024-09-17T07:25:29.972825+0000","last_peered":"2024-09-17T07:25:29.972825+0000","last_clean":"2024-09-17T07:25:29.972825+0000","last_became_active":"2024-09-17T07:22:13.709873+0000","last_became_peered":"2024-09-17T07:22:13.709873+0000","last_unstale":"2024-09-17T07:25:29.972825+0000","last_undegraded":"2024-09-17T07:25:29.972825+0000","last_fullsized":"2024-09-17T07:25:29.972825+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:29.972687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:25:29.972687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:45.937352+0000","scrub_duration":0.001803572,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":519,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:24.855129+0000","last_change":"2024-09-17T07:25:11.133451+0000","last_active":"2024-09-17T07:25:24.855129+0000","last_peered":"2024-09-17T07:25:24.855129+0000","last_clean":"2024-09-17T07:25:24.855129+0000","last_became_active":"2024-09-17T07:23:57.478625+0000","last_became_peered":"2024-09-17T07:23:57.478625+0000","last_unstale":"2024-09-17T07:25:24.855129+0000","last_undegraded":"2024-09-17T07:25:24.855129+0000","last_fullsized":"2024-09-17T07:25:24.855129+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:11.133339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:25:11.133339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:35.425563+0000","scrub_duration":0.0016371350000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":456,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:25.501359+0000","last_change":"2024-09-17T07:24:26.583863+0000","last_active":"2024-09-17T07:25:25.501359+0000","last_peered":"2024-09-17T07:25:25.501359+0000","last_clean":"2024-09-17T07:25:25.501359+0000","last_became_active":"2024-09-17T07:23:57.486204+0000","last_became_peered":"2024-09-17T07:23:57.486204+0000","last_unstale":"2024-09-17T07:25:25.501359+0000","last_undegraded":"2024-09-17T07:25:25.501359+0000","last_fullsized":"2024-09-17T07:25:25.501359+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:24:26.583745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:13:22.187127+0000","last_clean_scrub_stamp":"2024-09-17T07:24:26.583745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:46.250777+0000","scrub_duration":0.0024301169999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"212'130","reported_seq":634,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-17T07:25:30.132062+0000","last_change":"2024-09-17T07:25:30.132170+0000","last_active":"2024-09-17T07:25:30.132062+0000","last_peered":"2024-09-17T07:25:30.132062+0000","last_clean":"2024-09-17T07:25:30.132062+0000","last_became_active":"2024-09-17T07:23:57.478159+0000","last_became_peered":"2024-09-17T07:23:57.478159+0000","last_unstale":"2024-09-17T07:25:30.132062+0000","last_undegraded":"2024-09-17T07:25:30.132062+0000","last_fullsized":"2024-09-17T07:25:30.132062+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":3,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:25:30.132032+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:25:30.132032+0000","last_clean_scrub_stamp":"2024-09-17T07:25:30.132032+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:32.779068+0000","scrub_duration":0.0025929640000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1","version":"439'42","reported_seq":619,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:24.766311+0000","last_change":"2024-09-17T07:23:57.482189+0000","last_active":"2024-09-17T07:25:24.766311+0000","last_peered":"2024-09-17T07:25:24.766311+0000","last_clean":"2024-09-17T07:25:24.766311+0000","last_became_active":"2024-09-17T07:23:57.481780+0000","last_became_peered":"2024-09-17T07:23:57.481780+0000","last_unstale":"2024-09-17T07:25:24.766311+0000","last_undegraded":"2024-09-17T07:25:24.766311+0000","last_fullsized":"2024-09-17T07:25:24.766311+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'28","last_scrub_stamp":"2024-09-17T07:20:28.029279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:20:28.029279+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0079911889999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":63,"num_read_kb":56,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":624,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:25.502861+0000","last_change":"2024-09-17T07:25:07.568509+0000","last_active":"2024-09-17T07:25:25.502861+0000","last_peered":"2024-09-17T07:25:25.502861+0000","last_clean":"2024-09-17T07:25:25.502861+0000","last_became_active":"2024-09-17T07:22:13.582096+0000","last_became_peered":"2024-09-17T07:22:13.582096+0000","last_unstale":"2024-09-17T07:25:25.502861+0000","last_undegraded":"2024-09-17T07:25:25.502861+0000","last_fullsized":"2024-09-17T07:25:25.502861+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:07.568411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:09:23.181028+0000","last_clean_scrub_stamp":"2024-09-17T07:25:07.568411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:31.713937+0000","scrub_duration":0.0014008320000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"414'11","reported_seq":414,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-17T07:25:29.789338+0000","last_change":"2024-09-17T07:25:23.914163+0000","last_active":"2024-09-17T07:25:29.789338+0000","last_peered":"2024-09-17T07:25:29.789338+0000","last_clean":"2024-09-17T07:25:29.789338+0000","last_became_active":"2024-09-17T07:21:15.355651+0000","last_became_peered":"2024-09-17T07:21:15.355651+0000","last_unstale":"2024-09-17T07:25:29.789338+0000","last_undegraded":"2024-09-17T07:25:29.789338+0000","last_fullsized":"2024-09-17T07:25:29.789338+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'11","last_scrub_stamp":"2024-09-17T07:25:23.914072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:25:23.914072+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:38.750094+0000","scrub_duration":0.0020658130000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":102,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:25.504516+0000","last_change":"2024-09-17T07:25:00.872918+0000","last_active":"2024-09-17T07:25:25.504516+0000","last_peered":"2024-09-17T07:25:25.504516+0000","last_clean":"2024-09-17T07:25:25.504516+0000","last_became_active":"2024-09-17T07:22:13.568656+0000","last_became_peered":"2024-09-17T07:22:13.568656+0000","last_unstale":"2024-09-17T07:25:25.504516+0000","last_undegraded":"2024-09-17T07:25:25.504516+0000","last_fullsized":"2024-09-17T07:25:25.504516+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:00.872840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:10:33.727472+0000","last_clean_scrub_stamp":"2024-09-17T07:25:00.872840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:27.841417+0000","scrub_duration":0.001230515,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"212'130","reported_seq":608,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:24.766686+0000","last_change":"2024-09-17T07:20:35.080459+0000","last_active":"2024-09-17T07:25:24.766686+0000","last_peered":"2024-09-17T07:25:24.766686+0000","last_clean":"2024-09-17T07:25:24.766686+0000","last_became_active":"2024-09-17T07:19:35.420337+0000","last_became_peered":"2024-09-17T07:19:35.420337+0000","last_unstale":"2024-09-17T07:25:24.766686+0000","last_undegraded":"2024-09-17T07:25:24.766686+0000","last_fullsized":"2024-09-17T07:25:24.766686+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":3,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:20:35.080257+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:20:35.080257+0000","last_clean_scrub_stamp":"2024-09-17T07:20:35.080257+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0030477730000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"439'42","reported_seq":137,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:25.503866+0000","last_change":"2024-09-17T07:25:16.935826+0000","last_active":"2024-09-17T07:25:25.503866+0000","last_peered":"2024-09-17T07:25:25.503866+0000","last_clean":"2024-09-17T07:25:25.503866+0000","last_became_active":"2024-09-17T07:22:13.711444+0000","last_became_peered":"2024-09-17T07:22:13.711444+0000","last_unstale":"2024-09-17T07:25:25.503866+0000","last_undegraded":"2024-09-17T07:25:25.503866+0000","last_fullsized":"2024-09-17T07:25:25.503866+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'42","last_scrub_stamp":"2024-09-17T07:25:16.935731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:25:16.935731+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:23.547655+0000","scrub_duration":0.0023825790000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":68,"num_read_kb":56,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":430,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-17T07:25:29.789217+0000","last_change":"2024-09-17T07:25:09.880351+0000","last_active":"2024-09-17T07:25:29.789217+0000","last_peered":"2024-09-17T07:25:29.789217+0000","last_clean":"2024-09-17T07:25:29.789217+0000","last_became_active":"2024-09-17T07:21:15.347867+0000","last_became_peered":"2024-09-17T07:21:15.347867+0000","last_unstale":"2024-09-17T07:25:29.789217+0000","last_undegraded":"2024-09-17T07:25:29.789217+0000","last_fullsized":"2024-09-17T07:25:29.789217+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:09.880264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:56.848689+0000","last_clean_scrub_stamp":"2024-09-17T07:25:09.880264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:18.052765+0000","scrub_duration":0.0014177599999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":577,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:24.766198+0000","last_change":"2024-09-17T07:23:27.713373+0000","last_active":"2024-09-17T07:25:24.766198+0000","last_peered":"2024-09-17T07:25:24.766198+0000","last_clean":"2024-09-17T07:25:24.766198+0000","last_became_active":"2024-09-17T07:22:13.566916+0000","last_became_peered":"2024-09-17T07:22:13.566916+0000","last_unstale":"2024-09-17T07:25:24.766198+0000","last_undegraded":"2024-09-17T07:25:24.766198+0000","last_fullsized":"2024-09-17T07:25:24.766198+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:23:27.713294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:23:27.713294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0011838949999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":557,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:24.766556+0000","last_change":"2024-09-17T07:25:18.894993+0000","last_active":"2024-09-17T07:25:24.766556+0000","last_peered":"2024-09-17T07:25:24.766556+0000","last_clean":"2024-09-17T07:25:24.766556+0000","last_became_active":"2024-09-17T07:09:24.398659+0000","last_became_peered":"2024-09-17T07:09:24.398659+0000","last_unstale":"2024-09-17T07:25:24.766556+0000","last_undegraded":"2024-09-17T07:25:24.766556+0000","last_fullsized":"2024-09-17T07:25:24.766556+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:18.894878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:25:18.894878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:25.962648+0000","scrub_duration":0.001379328,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"212'130","reported_seq":627,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:25.503245+0000","last_change":"2024-09-17T07:25:12.680354+0000","last_active":"2024-09-17T07:25:25.503245+0000","last_peered":"2024-09-17T07:25:25.503245+0000","last_clean":"2024-09-17T07:25:25.503245+0000","last_became_active":"2024-09-17T07:19:35.428161+0000","last_became_peered":"2024-09-17T07:19:35.428161+0000","last_unstale":"2024-09-17T07:25:25.503245+0000","last_undegraded":"2024-09-17T07:25:25.503245+0000","last_fullsized":"2024-09-17T07:25:25.503245+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":3,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:25:12.680248+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:14:57.052505+0000","last_clean_scrub_stamp":"2024-09-17T07:25:12.680248+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:41.650768+0000","scrub_duration":0.001644413,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"439'39","reported_seq":698,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:24.767502+0000","last_change":"2024-09-17T07:25:20.597190+0000","last_active":"2024-09-17T07:25:24.767502+0000","last_peered":"2024-09-17T07:25:24.767502+0000","last_clean":"2024-09-17T07:25:24.767502+0000","last_became_active":"2024-09-17T07:23:57.497347+0000","last_became_peered":"2024-09-17T07:23:57.497347+0000","last_unstale":"2024-09-17T07:25:24.767502+0000","last_undegraded":"2024-09-17T07:25:24.767502+0000","last_fullsized":"2024-09-17T07:25:24.767502+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'39","last_scrub_stamp":"2024-09-17T07:25:20.597058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:25:20.597058+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:23.485116+0000","scrub_duration":0.0029701250000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":101,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:25.505002+0000","last_change":"2024-09-17T07:25:07.874505+0000","last_active":"2024-09-17T07:25:25.505002+0000","last_peered":"2024-09-17T07:25:25.505002+0000","last_clean":"2024-09-17T07:25:25.505002+0000","last_became_active":"2024-09-17T07:22:13.624802+0000","last_became_peered":"2024-09-17T07:22:13.624802+0000","last_unstale":"2024-09-17T07:25:25.505002+0000","last_undegraded":"2024-09-17T07:25:25.505002+0000","last_fullsized":"2024-09-17T07:25:25.505002+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:07.874421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:47.631387+0000","last_clean_scrub_stamp":"2024-09-17T07:25:07.874421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:23.997699+0000","scrub_duration":0.0012562529999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":585,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:25.501341+0000","last_change":"2024-09-17T07:25:05.604387+0000","last_active":"2024-09-17T07:25:25.501341+0000","last_peered":"2024-09-17T07:25:25.501341+0000","last_clean":"2024-09-17T07:25:25.501341+0000","last_became_active":"2024-09-17T07:23:57.485512+0000","last_became_peered":"2024-09-17T07:23:57.485512+0000","last_unstale":"2024-09-17T07:25:25.501341+0000","last_undegraded":"2024-09-17T07:25:25.501341+0000","last_fullsized":"2024-09-17T07:25:25.501341+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:05.604241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:11:17.583712+0000","last_clean_scrub_stamp":"2024-09-17T07:25:05.604241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:12.778226+0000","scrub_duration":0.0020187059999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":385,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:24.765446+0000","last_change":"2024-09-17T07:24:19.018167+0000","last_active":"2024-09-17T07:25:24.765446+0000","last_peered":"2024-09-17T07:25:24.765446+0000","last_clean":"2024-09-17T07:25:24.765446+0000","last_became_active":"2024-09-17T07:21:15.351524+0000","last_became_peered":"2024-09-17T07:21:15.351524+0000","last_unstale":"2024-09-17T07:25:24.765446+0000","last_undegraded":"2024-09-17T07:25:24.765446+0000","last_fullsized":"2024-09-17T07:25:24.765446+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:24:19.018060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:24:19.018060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:31.477457+0000","scrub_duration":0.0014215090000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"212'130","reported_seq":692,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:25.502004+0000","last_change":"2024-09-17T07:25:20.559655+0000","last_active":"2024-09-17T07:25:25.502004+0000","last_peered":"2024-09-17T07:25:25.502004+0000","last_clean":"2024-09-17T07:25:25.502004+0000","last_became_active":"2024-09-17T07:23:57.486189+0000","last_became_peered":"2024-09-17T07:23:57.486189+0000","last_unstale":"2024-09-17T07:25:25.502004+0000","last_undegraded":"2024-09-17T07:25:25.502004+0000","last_fullsized":"2024-09-17T07:25:25.502004+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":3,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:25:20.559569+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:14:57.052505+0000","last_clean_scrub_stamp":"2024-09-17T07:25:20.559569+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:40.068013+0000","scrub_duration":0.0013623260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"439'35","reported_seq":466,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:24.765678+0000","last_change":"2024-09-17T07:24:31.049397+0000","last_active":"2024-09-17T07:25:24.765678+0000","last_peered":"2024-09-17T07:25:24.765678+0000","last_clean":"2024-09-17T07:25:24.765678+0000","last_became_active":"2024-09-17T07:23:57.478340+0000","last_became_peered":"2024-09-17T07:23:57.478340+0000","last_unstale":"2024-09-17T07:25:24.765678+0000","last_undegraded":"2024-09-17T07:25:24.765678+0000","last_fullsized":"2024-09-17T07:25:24.765678+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'35","last_scrub_stamp":"2024-09-17T07:24:31.049269+0000","last_deep_scrub":"110'15","last_deep_scrub_stamp":"2024-09-17T07:04:57.091400+0000","last_clean_scrub_stamp":"2024-09-17T07:24:31.049269+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:39.791093+0000","scrub_duration":0.01178889,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:24.765796+0000","last_change":"2024-09-17T07:22:13.592639+0000","last_active":"2024-09-17T07:25:24.765796+0000","last_peered":"2024-09-17T07:25:24.765796+0000","last_clean":"2024-09-17T07:25:24.765796+0000","last_became_active":"2024-09-17T07:22:13.591372+0000","last_became_peered":"2024-09-17T07:22:13.591372+0000","last_unstale":"2024-09-17T07:25:24.765796+0000","last_undegraded":"2024-09-17T07:25:24.765796+0000","last_fullsized":"2024-09-17T07:25:24.765796+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:24.437342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:25.147878+0000","last_clean_scrub_stamp":"2024-09-17T07:20:24.437342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0024354329999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"478'11","reported_seq":866,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-17T07:25:32.618580+0000","last_change":"2024-09-17T07:25:32.618650+0000","last_active":"2024-09-17T07:25:32.618580+0000","last_peered":"2024-09-17T07:25:32.618580+0000","last_clean":"2024-09-17T07:25:32.618580+0000","last_became_active":"2024-09-17T07:23:57.496882+0000","last_became_peered":"2024-09-17T07:23:57.496882+0000","last_unstale":"2024-09-17T07:25:32.618580+0000","last_undegraded":"2024-09-17T07:25:32.618580+0000","last_fullsized":"2024-09-17T07:25:32.618580+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"478'11","last_scrub_stamp":"2024-09-17T07:25:32.618566+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-09-17T07:15:53.512777+0000","last_clean_scrub_stamp":"2024-09-17T07:25:32.618566+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:37.955426+0000","scrub_duration":0.0019411280000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"268'2","reported_seq":577,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:24.766187+0000","last_change":"2024-09-17T07:25:08.624593+0000","last_active":"2024-09-17T07:25:24.766187+0000","last_peered":"2024-09-17T07:25:24.766187+0000","last_clean":"2024-09-17T07:25:24.766187+0000","last_became_active":"2024-09-17T07:23:57.479564+0000","last_became_peered":"2024-09-17T07:23:57.479564+0000","last_unstale":"2024-09-17T07:25:24.766187+0000","last_undegraded":"2024-09-17T07:25:24.766187+0000","last_fullsized":"2024-09-17T07:25:24.766187+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'2","last_scrub_stamp":"2024-09-17T07:25:08.624464+0000","last_deep_scrub":"268'2","last_deep_scrub_stamp":"2024-09-17T07:14:03.609131+0000","last_clean_scrub_stamp":"2024-09-17T07:25:08.624464+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:10.264762+0000","scrub_duration":0.0032006410000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"212'130","reported_seq":583,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:24.854561+0000","last_change":"2024-09-17T07:23:57.489780+0000","last_active":"2024-09-17T07:25:24.854561+0000","last_peered":"2024-09-17T07:25:24.854561+0000","last_clean":"2024-09-17T07:25:24.854561+0000","last_became_active":"2024-09-17T07:23:57.489445+0000","last_became_peered":"2024-09-17T07:23:57.489445+0000","last_unstale":"2024-09-17T07:25:24.854561+0000","last_undegraded":"2024-09-17T07:25:24.854561+0000","last_fullsized":"2024-09-17T07:25:24.854561+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":4,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:20:22.674350+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:14:57.052505+0000","last_clean_scrub_stamp":"2024-09-17T07:20:22.674350+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021141889999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":28,"num_write":17,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":161010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"439'55","reported_seq":723,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-17T07:25:29.789396+0000","last_change":"2024-09-17T07:24:28.997089+0000","last_active":"2024-09-17T07:25:29.789396+0000","last_peered":"2024-09-17T07:25:29.789396+0000","last_clean":"2024-09-17T07:25:29.789396+0000","last_became_active":"2024-09-17T07:21:15.352461+0000","last_became_peered":"2024-09-17T07:21:15.352461+0000","last_unstale":"2024-09-17T07:25:29.789396+0000","last_undegraded":"2024-09-17T07:25:29.789396+0000","last_fullsized":"2024-09-17T07:25:29.789396+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'55","last_scrub_stamp":"2024-09-17T07:24:28.996914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:24:28.996914+0000","objects_scrubbed":10,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:48.984638+0000","scrub_duration":0.0070140810000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":83,"num_read_kb":68,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":504,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:24.854829+0000","last_change":"2024-09-17T07:24:18.694832+0000","last_active":"2024-09-17T07:25:24.854829+0000","last_peered":"2024-09-17T07:25:24.854829+0000","last_clean":"2024-09-17T07:25:24.854829+0000","last_became_active":"2024-09-17T07:23:57.493640+0000","last_became_peered":"2024-09-17T07:23:57.493640+0000","last_unstale":"2024-09-17T07:25:24.854829+0000","last_undegraded":"2024-09-17T07:25:24.854829+0000","last_fullsized":"2024-09-17T07:25:24.854829+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:24:18.694743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:24:18.694743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:30.108218+0000","scrub_duration":0.001126408,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":554,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:24.767715+0000","last_change":"2024-09-17T07:24:57.661571+0000","last_active":"2024-09-17T07:25:24.767715+0000","last_peered":"2024-09-17T07:25:24.767715+0000","last_clean":"2024-09-17T07:25:24.767715+0000","last_became_active":"2024-09-17T07:21:15.347069+0000","last_became_peered":"2024-09-17T07:21:15.347069+0000","last_unstale":"2024-09-17T07:25:24.767715+0000","last_undegraded":"2024-09-17T07:25:24.767715+0000","last_fullsized":"2024-09-17T07:25:24.767715+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:24:57.661438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:24:57.661438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:13.489925+0000","scrub_duration":0.0017509139999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":454,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:24.766907+0000","last_change":"2024-09-17T07:25:09.565076+0000","last_active":"2024-09-17T07:25:24.766907+0000","last_peered":"2024-09-17T07:25:24.766907+0000","last_clean":"2024-09-17T07:25:24.766907+0000","last_became_active":"2024-09-17T07:25:09.564606+0000","last_became_peered":"2024-09-17T07:25:09.564606+0000","last_unstale":"2024-09-17T07:25:24.766907+0000","last_undegraded":"2024-09-17T07:25:24.766907+0000","last_fullsized":"2024-09-17T07:25:24.766907+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:20.067928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:13:53.406661+0000","last_clean_scrub_stamp":"2024-09-17T07:20:20.067928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0042634170000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.2","version":"212'130","reported_seq":602,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:25.504533+0000","last_change":"2024-09-17T07:22:16.360792+0000","last_active":"2024-09-17T07:25:25.504533+0000","last_peered":"2024-09-17T07:25:25.504533+0000","last_clean":"2024-09-17T07:25:25.504533+0000","last_became_active":"2024-09-17T07:22:16.360258+0000","last_became_peered":"2024-09-17T07:22:16.360258+0000","last_unstale":"2024-09-17T07:25:25.504533+0000","last_undegraded":"2024-09-17T07:25:25.504533+0000","last_fullsized":"2024-09-17T07:25:25.504533+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":4,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:19:19.991044+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:14:57.052505+0000","last_clean_scrub_stamp":"2024-09-17T07:19:19.991044+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021141889999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":28,"num_write":17,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":161010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"439'21","reported_seq":442,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-17T07:25:29.789043+0000","last_change":"2024-09-17T07:24:20.021085+0000","last_active":"2024-09-17T07:25:29.789043+0000","last_peered":"2024-09-17T07:25:29.789043+0000","last_clean":"2024-09-17T07:25:29.789043+0000","last_became_active":"2024-09-17T07:23:57.492747+0000","last_became_peered":"2024-09-17T07:23:57.492747+0000","last_unstale":"2024-09-17T07:25:29.789043+0000","last_undegraded":"2024-09-17T07:25:29.789043+0000","last_fullsized":"2024-09-17T07:25:29.789043+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'21","last_scrub_stamp":"2024-09-17T07:24:20.021003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:24:20.021003+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0022976009999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":30,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"76'5","reported_seq":1477,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-17T07:25:31.897886+0000","last_change":"2024-09-17T07:25:05.896344+0000","last_active":"2024-09-17T07:25:31.897886+0000","last_peered":"2024-09-17T07:25:31.897886+0000","last_clean":"2024-09-17T07:25:31.897886+0000","last_became_active":"2024-09-17T07:21:15.357178+0000","last_became_peered":"2024-09-17T07:21:15.357178+0000","last_unstale":"2024-09-17T07:25:31.897886+0000","last_undegraded":"2024-09-17T07:25:31.897886+0000","last_fullsized":"2024-09-17T07:25:31.897886+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'5","last_scrub_stamp":"2024-09-17T07:25:05.896257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:25:05.896257+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:15.630574+0000","scrub_duration":0.0018339840000000001,"objects_trimmed":0,"snaptrim_duration":0.00033739799999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":264,"num_read_kb":259,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":398,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:24.767258+0000","last_change":"2024-09-17T07:25:01.641881+0000","last_active":"2024-09-17T07:25:24.767258+0000","last_peered":"2024-09-17T07:25:24.767258+0000","last_clean":"2024-09-17T07:25:24.767258+0000","last_became_active":"2024-09-17T07:21:15.346287+0000","last_became_peered":"2024-09-17T07:21:15.346287+0000","last_unstale":"2024-09-17T07:25:24.767258+0000","last_undegraded":"2024-09-17T07:25:24.767258+0000","last_fullsized":"2024-09-17T07:25:24.767258+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:01.641756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:23.833170+0000","last_clean_scrub_stamp":"2024-09-17T07:25:01.641756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:18.315474+0000","scrub_duration":0.0031250689999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":628,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:28.060947+0000","last_change":"2024-09-17T07:25:28.060947+0000","last_active":"2024-09-17T07:25:28.060947+0000","last_peered":"2024-09-17T07:25:28.060947+0000","last_clean":"2024-09-17T07:25:28.060947+0000","last_became_active":"2024-09-17T07:23:57.481485+0000","last_became_peered":"2024-09-17T07:23:57.481485+0000","last_unstale":"2024-09-17T07:25:28.060947+0000","last_undegraded":"2024-09-17T07:25:28.060947+0000","last_fullsized":"2024-09-17T07:25:28.060947+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:28.060855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:18:42.051667+0000","last_clean_scrub_stamp":"2024-09-17T07:25:28.060855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:57.847008+0000","scrub_duration":0.0017997740000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":101,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:25.504682+0000","last_change":"2024-09-17T07:24:30.923133+0000","last_active":"2024-09-17T07:25:25.504682+0000","last_peered":"2024-09-17T07:25:25.504682+0000","last_clean":"2024-09-17T07:25:25.504682+0000","last_became_active":"2024-09-17T07:22:13.628359+0000","last_became_peered":"2024-09-17T07:22:13.628359+0000","last_unstale":"2024-09-17T07:25:25.504682+0000","last_undegraded":"2024-09-17T07:25:25.504682+0000","last_fullsized":"2024-09-17T07:25:25.504682+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:24:30.922689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:02.062618+0000","last_clean_scrub_stamp":"2024-09-17T07:24:30.922689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:36.609751+0000","scrub_duration":0.0066080289999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"212'130","reported_seq":627,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:28.665064+0000","last_change":"2024-09-17T07:21:18.358222+0000","last_active":"2024-09-17T07:25:28.665064+0000","last_peered":"2024-09-17T07:25:28.665064+0000","last_clean":"2024-09-17T07:25:28.665064+0000","last_became_active":"2024-09-17T07:21:18.357874+0000","last_became_peered":"2024-09-17T07:21:18.357874+0000","last_unstale":"2024-09-17T07:25:28.665064+0000","last_undegraded":"2024-09-17T07:25:28.665064+0000","last_fullsized":"2024-09-17T07:25:28.665064+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":3,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:20:34.113602+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:20:34.113602+0000","last_clean_scrub_stamp":"2024-09-17T07:20:34.113602+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021141889999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":28,"num_write":17,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":161010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"439'42","reported_seq":648,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:24.767966+0000","last_change":"2024-09-17T07:25:05.607011+0000","last_active":"2024-09-17T07:25:24.767966+0000","last_peered":"2024-09-17T07:25:24.767966+0000","last_clean":"2024-09-17T07:25:24.767966+0000","last_became_active":"2024-09-17T07:23:57.480190+0000","last_became_peered":"2024-09-17T07:23:57.480190+0000","last_unstale":"2024-09-17T07:25:24.767966+0000","last_undegraded":"2024-09-17T07:25:24.767966+0000","last_fullsized":"2024-09-17T07:25:24.767966+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'42","last_scrub_stamp":"2024-09-17T07:25:05.606929+0000","last_deep_scrub":"268'30","last_deep_scrub_stamp":"2024-09-17T07:14:00.599656+0000","last_clean_scrub_stamp":"2024-09-17T07:25:05.606929+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:28.810784+0000","scrub_duration":0.0040141389999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"70'1","reported_seq":587,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:24.767101+0000","last_change":"2024-09-17T07:25:14.651563+0000","last_active":"2024-09-17T07:25:24.767101+0000","last_peered":"2024-09-17T07:25:24.767101+0000","last_clean":"2024-09-17T07:25:24.767101+0000","last_became_active":"2024-09-17T07:23:57.480048+0000","last_became_peered":"2024-09-17T07:23:57.480048+0000","last_unstale":"2024-09-17T07:25:24.767101+0000","last_undegraded":"2024-09-17T07:25:24.767101+0000","last_fullsized":"2024-09-17T07:25:24.767101+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-17T07:25:14.651458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:25:14.651458+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:14.704935+0000","scrub_duration":0.0020687610000000001,"objects_trimmed":0,"snaptrim_duration":0.0032298629999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":90,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:25.503908+0000","last_change":"2024-09-17T07:22:13.736269+0000","last_active":"2024-09-17T07:25:25.503908+0000","last_peered":"2024-09-17T07:25:25.503908+0000","last_clean":"2024-09-17T07:25:25.503908+0000","last_became_active":"2024-09-17T07:22:13.710211+0000","last_became_peered":"2024-09-17T07:22:13.710211+0000","last_unstale":"2024-09-17T07:25:25.503908+0000","last_undegraded":"2024-09-17T07:25:25.503908+0000","last_fullsized":"2024-09-17T07:25:25.503908+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:34.449627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:55.321202+0000","last_clean_scrub_stamp":"2024-09-17T07:20:34.449627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0044019280000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":499,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:24.855389+0000","last_change":"2024-09-17T07:24:57.916305+0000","last_active":"2024-09-17T07:25:24.855389+0000","last_peered":"2024-09-17T07:25:24.855389+0000","last_clean":"2024-09-17T07:25:24.855389+0000","last_became_active":"2024-09-17T07:23:57.480455+0000","last_became_peered":"2024-09-17T07:23:57.480455+0000","last_unstale":"2024-09-17T07:25:24.855389+0000","last_undegraded":"2024-09-17T07:25:24.855389+0000","last_fullsized":"2024-09-17T07:25:24.855389+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:24:57.916204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:24:57.916204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:22.807948+0000","scrub_duration":0.001685541,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":437,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-17T07:25:29.894297+0000","last_change":"2024-09-17T07:25:29.894360+0000","last_active":"2024-09-17T07:25:29.894297+0000","last_peered":"2024-09-17T07:25:29.894297+0000","last_clean":"2024-09-17T07:25:29.894297+0000","last_became_active":"2024-09-17T07:23:57.473313+0000","last_became_peered":"2024-09-17T07:23:57.473313+0000","last_unstale":"2024-09-17T07:25:29.894297+0000","last_undegraded":"2024-09-17T07:25:29.894297+0000","last_fullsized":"2024-09-17T07:25:29.894297+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:29.894283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:25:29.894283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:54.587102+0000","scrub_duration":0.003414397,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"439'38","reported_seq":671,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:25.502688+0000","last_change":"2024-09-17T07:25:11.645824+0000","last_active":"2024-09-17T07:25:25.502688+0000","last_peered":"2024-09-17T07:25:25.502688+0000","last_clean":"2024-09-17T07:25:25.502688+0000","last_became_active":"2024-09-17T07:22:13.583204+0000","last_became_peered":"2024-09-17T07:22:13.583204+0000","last_unstale":"2024-09-17T07:25:25.502688+0000","last_undegraded":"2024-09-17T07:25:25.502688+0000","last_fullsized":"2024-09-17T07:25:25.502688+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'38","last_scrub_stamp":"2024-09-17T07:25:11.645722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:25:11.645722+0000","objects_scrubbed":7,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:36.237351+0000","scrub_duration":0.0024003060000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"268'3","reported_seq":371,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:24.854970+0000","last_change":"2024-09-17T07:25:15.066327+0000","last_active":"2024-09-17T07:25:24.854970+0000","last_peered":"2024-09-17T07:25:24.854970+0000","last_clean":"2024-09-17T07:25:24.854970+0000","last_became_active":"2024-09-17T07:23:57.485430+0000","last_became_peered":"2024-09-17T07:23:57.485430+0000","last_unstale":"2024-09-17T07:25:24.854970+0000","last_undegraded":"2024-09-17T07:25:24.854970+0000","last_fullsized":"2024-09-17T07:25:24.854970+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'3","last_scrub_stamp":"2024-09-17T07:25:15.066229+0000","last_deep_scrub":"268'3","last_deep_scrub_stamp":"2024-09-17T07:13:54.949345+0000","last_clean_scrub_stamp":"2024-09-17T07:25:15.066229+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:22.714073+0000","scrub_duration":0.0017457670000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":560,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:26.099827+0000","last_change":"2024-09-17T07:25:26.099827+0000","last_active":"2024-09-17T07:25:26.099827+0000","last_peered":"2024-09-17T07:25:26.099827+0000","last_clean":"2024-09-17T07:25:26.099827+0000","last_became_active":"2024-09-17T07:23:57.477410+0000","last_became_peered":"2024-09-17T07:23:57.477410+0000","last_unstale":"2024-09-17T07:25:26.099827+0000","last_undegraded":"2024-09-17T07:25:26.099827+0000","last_fullsized":"2024-09-17T07:25:26.099827+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:26.099731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:25:26.099731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:34.376052+0000","scrub_duration":0.0014795699999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"212'130","reported_seq":612,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:25.504021+0000","last_change":"2024-09-17T07:22:16.366636+0000","last_active":"2024-09-17T07:25:25.504021+0000","last_peered":"2024-09-17T07:25:25.504021+0000","last_clean":"2024-09-17T07:25:25.504021+0000","last_became_active":"2024-09-17T07:22:16.366295+0000","last_became_peered":"2024-09-17T07:22:16.366295+0000","last_unstale":"2024-09-17T07:25:25.504021+0000","last_undegraded":"2024-09-17T07:25:25.504021+0000","last_fullsized":"2024-09-17T07:25:25.504021+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:19:19.991044+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:14:57.052505+0000","last_clean_scrub_stamp":"2024-09-17T07:19:19.991044+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021141889999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":27,"num_write":16,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":161009,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":116,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:25.503617+0000","last_change":"2024-09-17T07:24:19.856080+0000","last_active":"2024-09-17T07:25:25.503617+0000","last_peered":"2024-09-17T07:25:25.503617+0000","last_clean":"2024-09-17T07:25:25.503617+0000","last_became_active":"2024-09-17T07:23:57.476027+0000","last_became_peered":"2024-09-17T07:23:57.476027+0000","last_unstale":"2024-09-17T07:25:25.503617+0000","last_undegraded":"2024-09-17T07:25:25.503617+0000","last_fullsized":"2024-09-17T07:25:25.503617+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:24:19.855925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:18:48.891136+0000","last_clean_scrub_stamp":"2024-09-17T07:24:19.855925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:47.079242+0000","scrub_duration":0.0055362720000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":495,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:24.855255+0000","last_change":"2024-09-17T07:24:21.789270+0000","last_active":"2024-09-17T07:25:24.855255+0000","last_peered":"2024-09-17T07:25:24.855255+0000","last_clean":"2024-09-17T07:25:24.855255+0000","last_became_active":"2024-09-17T07:23:57.478248+0000","last_became_peered":"2024-09-17T07:23:57.478248+0000","last_unstale":"2024-09-17T07:25:24.855255+0000","last_undegraded":"2024-09-17T07:25:24.855255+0000","last_fullsized":"2024-09-17T07:25:24.855255+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:24:21.789110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:58.670571+0000","last_clean_scrub_stamp":"2024-09-17T07:24:21.789110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:30.597631+0000","scrub_duration":0.0090529149999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"439'51","reported_seq":715,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:25.502677+0000","last_change":"2024-09-17T07:25:15.611131+0000","last_active":"2024-09-17T07:25:25.502677+0000","last_peered":"2024-09-17T07:25:25.502677+0000","last_clean":"2024-09-17T07:25:25.502677+0000","last_became_active":"2024-09-17T07:22:13.588260+0000","last_became_peered":"2024-09-17T07:22:13.588260+0000","last_unstale":"2024-09-17T07:25:25.502677+0000","last_undegraded":"2024-09-17T07:25:25.502677+0000","last_fullsized":"2024-09-17T07:25:25.502677+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'51","last_scrub_stamp":"2024-09-17T07:25:15.611017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:25:15.611017+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:28.411603+0000","scrub_duration":0.0030635860000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":639,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-17T07:25:31.146163+0000","last_change":"2024-09-17T07:25:31.146235+0000","last_active":"2024-09-17T07:25:31.146163+0000","last_peered":"2024-09-17T07:25:31.146163+0000","last_clean":"2024-09-17T07:25:31.146163+0000","last_became_active":"2024-09-17T07:23:57.482084+0000","last_became_peered":"2024-09-17T07:23:57.482084+0000","last_unstale":"2024-09-17T07:25:31.146163+0000","last_undegraded":"2024-09-17T07:25:31.146163+0000","last_fullsized":"2024-09-17T07:25:31.146163+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:31.146149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:24:23.255270+0000","last_clean_scrub_stamp":"2024-09-17T07:25:31.146149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:48.831894+0000","scrub_duration":0.0012510609999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":622,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:29.110563+0000","last_change":"2024-09-17T07:25:29.110563+0000","last_active":"2024-09-17T07:25:29.110563+0000","last_peered":"2024-09-17T07:25:29.110563+0000","last_clean":"2024-09-17T07:25:29.110563+0000","last_became_active":"2024-09-17T07:23:57.481504+0000","last_became_peered":"2024-09-17T07:23:57.481504+0000","last_unstale":"2024-09-17T07:25:29.110563+0000","last_undegraded":"2024-09-17T07:25:29.110563+0000","last_fullsized":"2024-09-17T07:25:29.110563+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:29.110480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:24:20.282973+0000","last_clean_scrub_stamp":"2024-09-17T07:25:29.110480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:31.194269+0000","scrub_duration":0.001214122,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":101,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:25.504053+0000","last_change":"2024-09-17T07:24:23.847700+0000","last_active":"2024-09-17T07:25:25.504053+0000","last_peered":"2024-09-17T07:25:25.504053+0000","last_clean":"2024-09-17T07:25:25.504053+0000","last_became_active":"2024-09-17T07:22:13.573161+0000","last_became_peered":"2024-09-17T07:22:13.573161+0000","last_unstale":"2024-09-17T07:25:25.504053+0000","last_undegraded":"2024-09-17T07:25:25.504053+0000","last_fullsized":"2024-09-17T07:25:25.504053+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:24:23.847363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:45.275473+0000","last_clean_scrub_stamp":"2024-09-17T07:24:23.847363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:31.038832+0000","scrub_duration":0.0047891890000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"439'25","reported_seq":583,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:24.767416+0000","last_change":"2024-09-17T07:24:56.652772+0000","last_active":"2024-09-17T07:25:24.767416+0000","last_peered":"2024-09-17T07:25:24.767416+0000","last_clean":"2024-09-17T07:25:24.767416+0000","last_became_active":"2024-09-17T07:21:15.349361+0000","last_became_peered":"2024-09-17T07:21:15.349361+0000","last_unstale":"2024-09-17T07:25:24.767416+0000","last_undegraded":"2024-09-17T07:25:24.767416+0000","last_fullsized":"2024-09-17T07:25:24.767416+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'25","last_scrub_stamp":"2024-09-17T07:24:56.652673+0000","last_deep_scrub":"268'19","last_deep_scrub_stamp":"2024-09-17T07:14:06.634334+0000","last_clean_scrub_stamp":"2024-09-17T07:24:56.652673+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:21.478181+0000","scrub_duration":0.012732797000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":438,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:24.766424+0000","last_change":"2024-09-17T07:25:14.981641+0000","last_active":"2024-09-17T07:25:24.766424+0000","last_peered":"2024-09-17T07:25:24.766424+0000","last_clean":"2024-09-17T07:25:24.766424+0000","last_became_active":"2024-09-17T07:21:15.350786+0000","last_became_peered":"2024-09-17T07:21:15.350786+0000","last_unstale":"2024-09-17T07:25:24.766424+0000","last_undegraded":"2024-09-17T07:25:24.766424+0000","last_fullsized":"2024-09-17T07:25:24.766424+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:14.981560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:13:53.593078+0000","last_clean_scrub_stamp":"2024-09-17T07:25:14.981560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:30.222480+0000","scrub_duration":0.0016595399999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":523,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:24.766419+0000","last_change":"2024-09-17T07:25:13.923025+0000","last_active":"2024-09-17T07:25:24.766419+0000","last_peered":"2024-09-17T07:25:24.766419+0000","last_clean":"2024-09-17T07:25:24.766419+0000","last_became_active":"2024-09-17T07:09:24.396642+0000","last_became_peered":"2024-09-17T07:09:24.396642+0000","last_unstale":"2024-09-17T07:25:24.766419+0000","last_undegraded":"2024-09-17T07:25:24.766419+0000","last_fullsized":"2024-09-17T07:25:24.766419+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:13.922902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:25:13.922902+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:19.966114+0000","scrub_duration":0.0019707629999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":564,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:24.767190+0000","last_change":"2024-09-17T07:25:16.639982+0000","last_active":"2024-09-17T07:25:24.767190+0000","last_peered":"2024-09-17T07:25:24.767190+0000","last_clean":"2024-09-17T07:25:24.767190+0000","last_became_active":"2024-09-17T07:22:13.589460+0000","last_became_peered":"2024-09-17T07:22:13.589460+0000","last_unstale":"2024-09-17T07:25:24.767190+0000","last_undegraded":"2024-09-17T07:25:24.767190+0000","last_fullsized":"2024-09-17T07:25:24.767190+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:16.639899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:58.099849+0000","last_clean_scrub_stamp":"2024-09-17T07:25:16.639899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:41.555366+0000","scrub_duration":0.001931898,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"439'46","reported_seq":652,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:24.854426+0000","last_change":"2024-09-17T07:25:00.980570+0000","last_active":"2024-09-17T07:25:24.854426+0000","last_peered":"2024-09-17T07:25:24.854426+0000","last_clean":"2024-09-17T07:25:24.854426+0000","last_became_active":"2024-09-17T07:23:57.482858+0000","last_became_peered":"2024-09-17T07:23:57.482858+0000","last_unstale":"2024-09-17T07:25:24.854426+0000","last_undegraded":"2024-09-17T07:25:24.854426+0000","last_fullsized":"2024-09-17T07:25:24.854426+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'46","last_scrub_stamp":"2024-09-17T07:25:00.980474+0000","last_deep_scrub":"242'34","last_deep_scrub_stamp":"2024-09-17T07:14:50.103539+0000","last_clean_scrub_stamp":"2024-09-17T07:25:00.980474+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:06.860259+0000","scrub_duration":0.0021656290000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'2","reported_seq":96,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:25.504946+0000","last_change":"2024-09-17T07:25:05.848575+0000","last_active":"2024-09-17T07:25:25.504946+0000","last_peered":"2024-09-17T07:25:25.504946+0000","last_clean":"2024-09-17T07:25:25.504946+0000","last_became_active":"2024-09-17T07:22:13.575469+0000","last_became_peered":"2024-09-17T07:22:13.575469+0000","last_unstale":"2024-09-17T07:25:25.504946+0000","last_undegraded":"2024-09-17T07:25:25.504946+0000","last_fullsized":"2024-09-17T07:25:25.504946+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-09-17T07:18:43.861842+0000","last_deep_scrub":"89'2","last_deep_scrub_stamp":"2024-09-17T07:12:20.301263+0000","last_clean_scrub_stamp":"2024-09-17T07:18:43.861842+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":420,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":420,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"436'14","reported_seq":146,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-17T07:25:29.973211+0000","last_change":"2024-09-17T07:24:59.877855+0000","last_active":"2024-09-17T07:25:29.973211+0000","last_peered":"2024-09-17T07:25:29.973211+0000","last_clean":"2024-09-17T07:25:29.973211+0000","last_became_active":"2024-09-17T07:22:13.624050+0000","last_became_peered":"2024-09-17T07:22:13.624050+0000","last_unstale":"2024-09-17T07:25:29.973211+0000","last_undegraded":"2024-09-17T07:25:29.973211+0000","last_fullsized":"2024-09-17T07:25:29.973211+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"436'14","last_scrub_stamp":"2024-09-17T07:24:59.877774+0000","last_deep_scrub":"225'9","last_deep_scrub_stamp":"2024-09-17T07:11:25.292565+0000","last_clean_scrub_stamp":"2024-09-17T07:24:59.877774+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:07.407178+0000","scrub_duration":0.0015096720000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":527,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:24.854682+0000","last_change":"2024-09-17T07:24:58.961010+0000","last_active":"2024-09-17T07:25:24.854682+0000","last_peered":"2024-09-17T07:25:24.854682+0000","last_clean":"2024-09-17T07:25:24.854682+0000","last_became_active":"2024-09-17T07:23:57.493273+0000","last_became_peered":"2024-09-17T07:23:57.493273+0000","last_unstale":"2024-09-17T07:25:24.854682+0000","last_undegraded":"2024-09-17T07:25:24.854682+0000","last_fullsized":"2024-09-17T07:25:24.854682+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:24:58.960933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:18:41.385928+0000","last_clean_scrub_stamp":"2024-09-17T07:24:58.960933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:03.753329+0000","scrub_duration":0.0012208259999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"439'29","reported_seq":639,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:24.766198+0000","last_change":"2024-09-17T07:20:33.108656+0000","last_active":"2024-09-17T07:25:24.766198+0000","last_peered":"2024-09-17T07:25:24.766198+0000","last_clean":"2024-09-17T07:25:24.766198+0000","last_became_active":"2024-09-17T07:10:49.723292+0000","last_became_peered":"2024-09-17T07:10:49.723292+0000","last_unstale":"2024-09-17T07:25:24.766198+0000","last_undegraded":"2024-09-17T07:25:24.766198+0000","last_fullsized":"2024-09-17T07:25:24.766198+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'23","last_scrub_stamp":"2024-09-17T07:20:33.108490+0000","last_deep_scrub":"150'17","last_deep_scrub_stamp":"2024-09-17T07:06:38.775788+0000","last_clean_scrub_stamp":"2024-09-17T07:20:33.108490+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0087601510000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"512'4","reported_seq":603,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:26.876766+0000","last_change":"2024-09-17T07:25:26.876766+0000","last_active":"2024-09-17T07:25:26.876766+0000","last_peered":"2024-09-17T07:25:26.876766+0000","last_clean":"2024-09-17T07:25:26.876766+0000","last_became_active":"2024-09-17T07:22:13.590172+0000","last_became_peered":"2024-09-17T07:22:13.590172+0000","last_unstale":"2024-09-17T07:25:26.876766+0000","last_undegraded":"2024-09-17T07:25:26.876766+0000","last_fullsized":"2024-09-17T07:25:26.876766+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"512'4","last_scrub_stamp":"2024-09-17T07:25:26.876662+0000","last_deep_scrub":"512'4","last_deep_scrub_stamp":"2024-09-17T07:25:26.876662+0000","last_clean_scrub_stamp":"2024-09-17T07:25:26.876662+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:47.139754+0000","scrub_duration":0.0029582549999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":517,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:24.766346+0000","last_change":"2024-09-17T07:24:26.618642+0000","last_active":"2024-09-17T07:25:24.766346+0000","last_peered":"2024-09-17T07:25:24.766346+0000","last_clean":"2024-09-17T07:25:24.766346+0000","last_became_active":"2024-09-17T07:21:15.355760+0000","last_became_peered":"2024-09-17T07:21:15.355760+0000","last_unstale":"2024-09-17T07:25:24.766346+0000","last_undegraded":"2024-09-17T07:25:24.766346+0000","last_fullsized":"2024-09-17T07:25:24.766346+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:24:26.618433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:11:17.479036+0000","last_clean_scrub_stamp":"2024-09-17T07:24:26.618433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:30.630753+0000","scrub_duration":0.0026272880000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":90,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:25.504606+0000","last_change":"2024-09-17T07:22:13.585879+0000","last_active":"2024-09-17T07:25:25.504606+0000","last_peered":"2024-09-17T07:25:25.504606+0000","last_clean":"2024-09-17T07:25:25.504606+0000","last_became_active":"2024-09-17T07:22:13.585580+0000","last_became_peered":"2024-09-17T07:22:13.585580+0000","last_unstale":"2024-09-17T07:25:25.504606+0000","last_undegraded":"2024-09-17T07:25:25.504606+0000","last_fullsized":"2024-09-17T07:25:25.504606+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:07.959158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:20:07.959158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.018487173999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"439'21","reported_seq":631,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:24.767946+0000","last_change":"2024-09-17T07:25:05.507259+0000","last_active":"2024-09-17T07:25:24.767946+0000","last_peered":"2024-09-17T07:25:24.767946+0000","last_clean":"2024-09-17T07:25:24.767946+0000","last_became_active":"2024-09-17T07:22:13.592001+0000","last_became_peered":"2024-09-17T07:22:13.592001+0000","last_unstale":"2024-09-17T07:25:24.767946+0000","last_undegraded":"2024-09-17T07:25:24.767946+0000","last_fullsized":"2024-09-17T07:25:24.767946+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'15","last_scrub_stamp":"2024-09-17T07:18:24.505002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:18:24.505002+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002195809,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":610,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-17T07:25:30.772157+0000","last_change":"2024-09-17T07:25:26.002943+0000","last_active":"2024-09-17T07:25:30.772157+0000","last_peered":"2024-09-17T07:25:30.772157+0000","last_clean":"2024-09-17T07:25:30.772157+0000","last_became_active":"2024-09-17T07:21:15.355464+0000","last_became_peered":"2024-09-17T07:21:15.355464+0000","last_unstale":"2024-09-17T07:25:30.772157+0000","last_undegraded":"2024-09-17T07:25:30.772157+0000","last_fullsized":"2024-09-17T07:25:30.772157+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:26.002749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:17.976267+0000","last_clean_scrub_stamp":"2024-09-17T07:25:26.002749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:53.438674+0000","scrub_duration":0.0023444410000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":249,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:24.767184+0000","last_change":"2024-09-17T07:22:13.593743+0000","last_active":"2024-09-17T07:25:24.767184+0000","last_peered":"2024-09-17T07:25:24.767184+0000","last_clean":"2024-09-17T07:25:24.767184+0000","last_became_active":"2024-09-17T07:22:13.593446+0000","last_became_peered":"2024-09-17T07:22:13.593446+0000","last_unstale":"2024-09-17T07:25:24.767184+0000","last_undegraded":"2024-09-17T07:25:24.767184+0000","last_fullsized":"2024-09-17T07:25:24.767184+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:29.492000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:18.367486+0000","last_clean_scrub_stamp":"2024-09-17T07:20:29.492000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0077309099999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":651,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:26.002606+0000","last_change":"2024-09-17T07:24:24.282738+0000","last_active":"2024-09-17T07:25:26.002606+0000","last_peered":"2024-09-17T07:25:26.002606+0000","last_clean":"2024-09-17T07:25:26.002606+0000","last_became_active":"2024-09-17T07:22:13.590599+0000","last_became_peered":"2024-09-17T07:22:13.590599+0000","last_unstale":"2024-09-17T07:25:26.002606+0000","last_undegraded":"2024-09-17T07:25:26.002606+0000","last_fullsized":"2024-09-17T07:25:26.002606+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:24:24.282487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:24:24.282487+0000","last_clean_scrub_stamp":"2024-09-17T07:24:24.282487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:43.887065+0000","scrub_duration":0.0043307379999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":568,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-17T07:25:29.893099+0000","last_change":"2024-09-17T07:24:20.528931+0000","last_active":"2024-09-17T07:25:29.893099+0000","last_peered":"2024-09-17T07:25:29.893099+0000","last_clean":"2024-09-17T07:25:29.893099+0000","last_became_active":"2024-09-17T07:21:15.348758+0000","last_became_peered":"2024-09-17T07:21:15.348758+0000","last_unstale":"2024-09-17T07:25:29.893099+0000","last_undegraded":"2024-09-17T07:25:29.893099+0000","last_fullsized":"2024-09-17T07:25:29.893099+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:24:20.528714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:46.645080+0000","last_clean_scrub_stamp":"2024-09-17T07:24:20.528714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:34.396548+0000","scrub_duration":0.0080332240000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"439'18","reported_seq":118,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:25.505000+0000","last_change":"2024-09-17T07:25:24.000058+0000","last_active":"2024-09-17T07:25:25.505000+0000","last_peered":"2024-09-17T07:25:25.505000+0000","last_clean":"2024-09-17T07:25:25.505000+0000","last_became_active":"2024-09-17T07:22:13.586662+0000","last_became_peered":"2024-09-17T07:22:13.586662+0000","last_unstale":"2024-09-17T07:25:25.505000+0000","last_undegraded":"2024-09-17T07:25:25.505000+0000","last_fullsized":"2024-09-17T07:25:25.505000+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'18","last_scrub_stamp":"2024-09-17T07:25:23.999953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:25:23.999953+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:47.208572+0000","scrub_duration":0.0023247710000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":25,"num_read_kb":22,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":507,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:24.855018+0000","last_change":"2024-09-17T07:24:23.835840+0000","last_active":"2024-09-17T07:25:24.855018+0000","last_peered":"2024-09-17T07:25:24.855018+0000","last_clean":"2024-09-17T07:25:24.855018+0000","last_became_active":"2024-09-17T07:23:57.478324+0000","last_became_peered":"2024-09-17T07:23:57.478324+0000","last_unstale":"2024-09-17T07:25:24.855018+0000","last_undegraded":"2024-09-17T07:25:24.855018+0000","last_fullsized":"2024-09-17T07:25:24.855018+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:24:23.835718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:24:23.835718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:47.145573+0000","scrub_duration":0.0037662490000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":101,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:25.504662+0000","last_change":"2024-09-17T07:25:06.877892+0000","last_active":"2024-09-17T07:25:25.504662+0000","last_peered":"2024-09-17T07:25:25.504662+0000","last_clean":"2024-09-17T07:25:25.504662+0000","last_became_active":"2024-09-17T07:22:13.623754+0000","last_became_peered":"2024-09-17T07:22:13.623754+0000","last_unstale":"2024-09-17T07:25:25.504662+0000","last_undegraded":"2024-09-17T07:25:25.504662+0000","last_fullsized":"2024-09-17T07:25:25.504662+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:06.877789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:25:06.877789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:29.306772+0000","scrub_duration":0.001331243,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"439'30","reported_seq":308,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:24.765715+0000","last_change":"2024-09-17T07:22:13.768713+0000","last_active":"2024-09-17T07:25:24.765715+0000","last_peered":"2024-09-17T07:25:24.765715+0000","last_clean":"2024-09-17T07:25:24.765715+0000","last_became_active":"2024-09-17T07:22:13.591094+0000","last_became_peered":"2024-09-17T07:22:13.591094+0000","last_unstale":"2024-09-17T07:25:24.765715+0000","last_undegraded":"2024-09-17T07:25:24.765715+0000","last_fullsized":"2024-09-17T07:25:24.765715+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'20","last_scrub_stamp":"2024-09-17T07:20:10.554324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:20:10.554324+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0020069329999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":568,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:24.854650+0000","last_change":"2024-09-17T07:25:20.089270+0000","last_active":"2024-09-17T07:25:24.854650+0000","last_peered":"2024-09-17T07:25:24.854650+0000","last_clean":"2024-09-17T07:25:24.854650+0000","last_became_active":"2024-09-17T07:23:57.491291+0000","last_became_peered":"2024-09-17T07:23:57.491291+0000","last_unstale":"2024-09-17T07:25:24.854650+0000","last_undegraded":"2024-09-17T07:25:24.854650+0000","last_fullsized":"2024-09-17T07:25:24.854650+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:20.089157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:19:40.443474+0000","last_clean_scrub_stamp":"2024-09-17T07:25:20.089157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:36.357354+0000","scrub_duration":0.0033687460000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"478'13","reported_seq":756,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:25.785238+0000","last_change":"2024-09-17T07:24:31.008439+0000","last_active":"2024-09-17T07:25:25.785238+0000","last_peered":"2024-09-17T07:25:25.785238+0000","last_clean":"2024-09-17T07:25:25.785238+0000","last_became_active":"2024-09-17T07:23:57.522386+0000","last_became_peered":"2024-09-17T07:23:57.522386+0000","last_unstale":"2024-09-17T07:25:25.785238+0000","last_undegraded":"2024-09-17T07:25:25.785238+0000","last_fullsized":"2024-09-17T07:25:25.785238+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"478'13","last_scrub_stamp":"2024-09-17T07:24:31.008289+0000","last_deep_scrub":"163'9","last_deep_scrub_stamp":"2024-09-17T07:07:34.933808+0000","last_clean_scrub_stamp":"2024-09-17T07:24:31.008289+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:47.002555+0000","scrub_duration":0.0084935229999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":573,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:24.767775+0000","last_change":"2024-09-17T07:25:06.624551+0000","last_active":"2024-09-17T07:25:24.767775+0000","last_peered":"2024-09-17T07:25:24.767775+0000","last_clean":"2024-09-17T07:25:24.767775+0000","last_became_active":"2024-09-17T07:22:13.591307+0000","last_became_peered":"2024-09-17T07:22:13.591307+0000","last_unstale":"2024-09-17T07:25:24.767775+0000","last_undegraded":"2024-09-17T07:25:24.767775+0000","last_fullsized":"2024-09-17T07:25:24.767775+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:06.624450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:26.619893+0000","last_clean_scrub_stamp":"2024-09-17T07:25:06.624450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:13.532043+0000","scrub_duration":0.001368408,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"439'21","reported_seq":624,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:24.855129+0000","last_change":"2024-09-17T07:25:08.008455+0000","last_active":"2024-09-17T07:25:24.855129+0000","last_peered":"2024-09-17T07:25:24.855129+0000","last_clean":"2024-09-17T07:25:24.855129+0000","last_became_active":"2024-09-17T07:23:57.485557+0000","last_became_peered":"2024-09-17T07:23:57.485557+0000","last_unstale":"2024-09-17T07:25:24.855129+0000","last_undegraded":"2024-09-17T07:25:24.855129+0000","last_fullsized":"2024-09-17T07:25:24.855129+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'21","last_scrub_stamp":"2024-09-17T07:25:08.008359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:25:08.008359+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:28.888348+0000","scrub_duration":0.0025217759999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":582,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:24.767619+0000","last_change":"2024-09-17T07:25:18.620750+0000","last_active":"2024-09-17T07:25:24.767619+0000","last_peered":"2024-09-17T07:25:24.767619+0000","last_clean":"2024-09-17T07:25:24.767619+0000","last_became_active":"2024-09-17T07:22:13.589964+0000","last_became_peered":"2024-09-17T07:22:13.589964+0000","last_unstale":"2024-09-17T07:25:24.767619+0000","last_undegraded":"2024-09-17T07:25:24.767619+0000","last_fullsized":"2024-09-17T07:25:24.767619+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:18.620666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:25:18.620666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:37.392688+0000","scrub_duration":0.001507972,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"436'15","reported_seq":145,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-17T07:25:29.973626+0000","last_change":"2024-09-17T07:25:18.993295+0000","last_active":"2024-09-17T07:25:29.973626+0000","last_peered":"2024-09-17T07:25:29.973626+0000","last_clean":"2024-09-17T07:25:29.973626+0000","last_became_active":"2024-09-17T07:22:13.586620+0000","last_became_peered":"2024-09-17T07:22:13.586620+0000","last_unstale":"2024-09-17T07:25:29.973626+0000","last_undegraded":"2024-09-17T07:25:29.973626+0000","last_fullsized":"2024-09-17T07:25:29.973626+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"436'15","last_scrub_stamp":"2024-09-17T07:25:18.993184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:25:18.993184+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:42.271745+0000","scrub_duration":0.0021849909999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":572,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:24.766897+0000","last_change":"2024-09-17T07:25:22.865620+0000","last_active":"2024-09-17T07:25:24.766897+0000","last_peered":"2024-09-17T07:25:24.766897+0000","last_clean":"2024-09-17T07:25:24.766897+0000","last_became_active":"2024-09-17T07:23:57.472854+0000","last_became_peered":"2024-09-17T07:23:57.472854+0000","last_unstale":"2024-09-17T07:25:24.766897+0000","last_undegraded":"2024-09-17T07:25:24.766897+0000","last_fullsized":"2024-09-17T07:25:24.766897+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:22.865498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:20:37.110458+0000","last_clean_scrub_stamp":"2024-09-17T07:25:22.865498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:46.956482+0000","scrub_duration":0.0016306630000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"439'21","reported_seq":420,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:24.765878+0000","last_change":"2024-09-17T07:24:18.055969+0000","last_active":"2024-09-17T07:25:24.765878+0000","last_peered":"2024-09-17T07:25:24.765878+0000","last_clean":"2024-09-17T07:25:24.765878+0000","last_became_active":"2024-09-17T07:21:15.348060+0000","last_became_peered":"2024-09-17T07:21:15.348060+0000","last_unstale":"2024-09-17T07:25:24.765878+0000","last_undegraded":"2024-09-17T07:25:24.765878+0000","last_fullsized":"2024-09-17T07:25:24.765878+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'21","last_scrub_stamp":"2024-09-17T07:24:18.055874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:24:18.055874+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0022038589999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":625,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:25.502305+0000","last_change":"2024-09-17T07:25:25.502305+0000","last_active":"2024-09-17T07:25:25.502305+0000","last_peered":"2024-09-17T07:25:25.502305+0000","last_clean":"2024-09-17T07:25:25.502305+0000","last_became_active":"2024-09-17T07:23:57.493401+0000","last_became_peered":"2024-09-17T07:23:57.493401+0000","last_unstale":"2024-09-17T07:25:25.502305+0000","last_undegraded":"2024-09-17T07:25:25.502305+0000","last_fullsized":"2024-09-17T07:25:25.502305+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:25.502164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:10:36.334827+0000","last_clean_scrub_stamp":"2024-09-17T07:25:25.502164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:44.041281+0000","scrub_duration":0.0025658949999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":410,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:24.767219+0000","last_change":"2024-09-17T07:25:00.641893+0000","last_active":"2024-09-17T07:25:24.767219+0000","last_peered":"2024-09-17T07:25:24.767219+0000","last_clean":"2024-09-17T07:25:24.767219+0000","last_became_active":"2024-09-17T07:23:57.479281+0000","last_became_peered":"2024-09-17T07:23:57.479281+0000","last_unstale":"2024-09-17T07:25:24.767219+0000","last_undegraded":"2024-09-17T07:25:24.767219+0000","last_fullsized":"2024-09-17T07:25:24.767219+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:00.641817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:25:00.641817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:22.472510+0000","scrub_duration":0.031817810000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":564,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:24.766268+0000","last_change":"2024-09-17T07:24:21.576079+0000","last_active":"2024-09-17T07:25:24.766268+0000","last_peered":"2024-09-17T07:25:24.766268+0000","last_clean":"2024-09-17T07:25:24.766268+0000","last_became_active":"2024-09-17T07:23:57.478818+0000","last_became_peered":"2024-09-17T07:23:57.478818+0000","last_unstale":"2024-09-17T07:25:24.766268+0000","last_undegraded":"2024-09-17T07:25:24.766268+0000","last_fullsized":"2024-09-17T07:25:24.766268+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:24:21.575574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:24:21.575574+0000","last_clean_scrub_stamp":"2024-09-17T07:24:21.575574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.006786233,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"439'27","reported_seq":586,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:24.766721+0000","last_change":"2024-09-17T07:21:16.070482+0000","last_active":"2024-09-17T07:25:24.766721+0000","last_peered":"2024-09-17T07:25:24.766721+0000","last_clean":"2024-09-17T07:25:24.766721+0000","last_became_active":"2024-09-17T07:21:15.347179+0000","last_became_peered":"2024-09-17T07:21:15.347179+0000","last_unstale":"2024-09-17T07:25:24.766721+0000","last_undegraded":"2024-09-17T07:25:24.766721+0000","last_fullsized":"2024-09-17T07:25:24.766721+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'19","last_scrub_stamp":"2024-09-17T07:21:16.070358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:21:16.070358+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0025953619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"94'2","reported_seq":436,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:24.766550+0000","last_change":"2024-09-17T07:25:16.939704+0000","last_active":"2024-09-17T07:25:24.766550+0000","last_peered":"2024-09-17T07:25:24.766550+0000","last_clean":"2024-09-17T07:25:24.766550+0000","last_became_active":"2024-09-17T07:23:57.521742+0000","last_became_peered":"2024-09-17T07:23:57.521742+0000","last_unstale":"2024-09-17T07:25:24.766550+0000","last_undegraded":"2024-09-17T07:25:24.766550+0000","last_fullsized":"2024-09-17T07:25:24.766550+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-09-17T07:25:16.939572+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-09-17T07:07:31.717888+0000","last_clean_scrub_stamp":"2024-09-17T07:25:16.939572+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:43.987976+0000","scrub_duration":0.001518578,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":429,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:24.765575+0000","last_change":"2024-09-17T07:23:57.495235+0000","last_active":"2024-09-17T07:25:24.765575+0000","last_peered":"2024-09-17T07:25:24.765575+0000","last_clean":"2024-09-17T07:25:24.765575+0000","last_became_active":"2024-09-17T07:23:57.494701+0000","last_became_peered":"2024-09-17T07:23:57.494701+0000","last_unstale":"2024-09-17T07:25:24.765575+0000","last_undegraded":"2024-09-17T07:25:24.765575+0000","last_fullsized":"2024-09-17T07:25:24.765575+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:16.640384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:20:16.640384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003133224,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":405,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:24.767374+0000","last_change":"2024-09-17T07:25:02.641851+0000","last_active":"2024-09-17T07:25:24.767374+0000","last_peered":"2024-09-17T07:25:24.767374+0000","last_clean":"2024-09-17T07:25:24.767374+0000","last_became_active":"2024-09-17T07:23:57.479008+0000","last_became_peered":"2024-09-17T07:23:57.479008+0000","last_unstale":"2024-09-17T07:25:24.767374+0000","last_undegraded":"2024-09-17T07:25:24.767374+0000","last_fullsized":"2024-09-17T07:25:24.767374+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:02.641763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:49.573666+0000","last_clean_scrub_stamp":"2024-09-17T07:25:02.641763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:19.718312+0000","scrub_duration":0.001362083,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"439'12","reported_seq":494,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:26.002314+0000","last_change":"2024-09-17T07:25:01.921597+0000","last_active":"2024-09-17T07:25:26.002314+0000","last_peered":"2024-09-17T07:25:26.002314+0000","last_clean":"2024-09-17T07:25:26.002314+0000","last_became_active":"2024-09-17T07:09:24.396368+0000","last_became_peered":"2024-09-17T07:09:24.396368+0000","last_unstale":"2024-09-17T07:25:26.002314+0000","last_undegraded":"2024-09-17T07:25:26.002314+0000","last_fullsized":"2024-09-17T07:25:26.002314+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'12","last_scrub_stamp":"2024-09-17T07:25:01.921507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:25:01.921507+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:21.393421+0000","scrub_duration":0.0020119589999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":509,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:24.855229+0000","last_change":"2024-09-17T07:25:19.124760+0000","last_active":"2024-09-17T07:25:24.855229+0000","last_peered":"2024-09-17T07:25:24.855229+0000","last_clean":"2024-09-17T07:25:24.855229+0000","last_became_active":"2024-09-17T07:23:57.476234+0000","last_became_peered":"2024-09-17T07:23:57.476234+0000","last_unstale":"2024-09-17T07:25:24.855229+0000","last_undegraded":"2024-09-17T07:25:24.855229+0000","last_fullsized":"2024-09-17T07:25:24.855229+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:19.124658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:25:19.124658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:22.203780+0000","scrub_duration":0.0013943600000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":588,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:25.502217+0000","last_change":"2024-09-17T07:25:03.562390+0000","last_active":"2024-09-17T07:25:25.502217+0000","last_peered":"2024-09-17T07:25:25.502217+0000","last_clean":"2024-09-17T07:25:25.502217+0000","last_became_active":"2024-09-17T07:22:13.582229+0000","last_became_peered":"2024-09-17T07:22:13.582229+0000","last_unstale":"2024-09-17T07:25:25.502217+0000","last_undegraded":"2024-09-17T07:25:25.502217+0000","last_fullsized":"2024-09-17T07:25:25.502217+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:03.562297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:54.553970+0000","last_clean_scrub_stamp":"2024-09-17T07:25:03.562297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:19.909241+0000","scrub_duration":0.0012245349999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":90,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:25.504437+0000","last_change":"2024-09-17T07:22:13.624740+0000","last_active":"2024-09-17T07:25:25.504437+0000","last_peered":"2024-09-17T07:25:25.504437+0000","last_clean":"2024-09-17T07:25:25.504437+0000","last_became_active":"2024-09-17T07:22:13.624127+0000","last_became_peered":"2024-09-17T07:22:13.624127+0000","last_unstale":"2024-09-17T07:25:25.504437+0000","last_undegraded":"2024-09-17T07:25:25.504437+0000","last_fullsized":"2024-09-17T07:25:25.504437+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:25.003509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:20:25.003509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002164586,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"439'27","reported_seq":129,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:25.503459+0000","last_change":"2024-09-17T07:23:57.476210+0000","last_active":"2024-09-17T07:25:25.503459+0000","last_peered":"2024-09-17T07:25:25.503459+0000","last_clean":"2024-09-17T07:25:25.503459+0000","last_became_active":"2024-09-17T07:23:57.475861+0000","last_became_peered":"2024-09-17T07:23:57.475861+0000","last_unstale":"2024-09-17T07:25:25.503459+0000","last_undegraded":"2024-09-17T07:25:25.503459+0000","last_fullsized":"2024-09-17T07:25:25.503459+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'19","last_scrub_stamp":"2024-09-17T07:20:28.113977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:20:28.113977+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.013438802,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":587,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:25.501656+0000","last_change":"2024-09-17T07:24:23.526255+0000","last_active":"2024-09-17T07:25:25.501656+0000","last_peered":"2024-09-17T07:25:25.501656+0000","last_clean":"2024-09-17T07:25:25.501656+0000","last_became_active":"2024-09-17T07:23:57.486575+0000","last_became_peered":"2024-09-17T07:23:57.486575+0000","last_unstale":"2024-09-17T07:25:25.501656+0000","last_undegraded":"2024-09-17T07:25:25.501656+0000","last_fullsized":"2024-09-17T07:25:25.501656+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:24:23.526138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:18:41.951382+0000","last_clean_scrub_stamp":"2024-09-17T07:24:23.526138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:48.414517+0000","scrub_duration":0.0045997989999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":90,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:25.503983+0000","last_change":"2024-09-17T07:22:13.736537+0000","last_active":"2024-09-17T07:25:25.503983+0000","last_peered":"2024-09-17T07:25:25.503983+0000","last_clean":"2024-09-17T07:25:25.503983+0000","last_became_active":"2024-09-17T07:22:13.710754+0000","last_became_peered":"2024-09-17T07:22:13.710754+0000","last_unstale":"2024-09-17T07:25:25.503983+0000","last_undegraded":"2024-09-17T07:25:25.503983+0000","last_fullsized":"2024-09-17T07:25:25.503983+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:20:30.038044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:20:30.038044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":401,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:24.767149+0000","last_change":"2024-09-17T07:25:17.663679+0000","last_active":"2024-09-17T07:25:24.767149+0000","last_peered":"2024-09-17T07:25:24.767149+0000","last_clean":"2024-09-17T07:25:24.767149+0000","last_became_active":"2024-09-17T07:09:24.392216+0000","last_became_peered":"2024-09-17T07:09:24.392216+0000","last_unstale":"2024-09-17T07:25:24.767149+0000","last_undegraded":"2024-09-17T07:25:24.767149+0000","last_fullsized":"2024-09-17T07:25:24.767149+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:17.663576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:25:17.663576+0000","last_clean_scrub_stamp":"2024-09-17T07:25:17.663576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:28.898950+0000","scrub_duration":0.0013785539999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1c","version":"326'2","reported_seq":109,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:25.504323+0000","last_change":"2024-09-17T07:23:57.476548+0000","last_active":"2024-09-17T07:25:25.504323+0000","last_peered":"2024-09-17T07:25:25.504323+0000","last_clean":"2024-09-17T07:25:25.504323+0000","last_became_active":"2024-09-17T07:23:57.475630+0000","last_became_peered":"2024-09-17T07:23:57.475630+0000","last_unstale":"2024-09-17T07:25:25.504323+0000","last_undegraded":"2024-09-17T07:25:25.504323+0000","last_fullsized":"2024-09-17T07:25:25.504323+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'2","last_scrub_stamp":"2024-09-17T07:20:19.609678+0000","last_deep_scrub":"326'2","last_deep_scrub_stamp":"2024-09-17T07:17:36.974358+0000","last_clean_scrub_stamp":"2024-09-17T07:20:19.609678+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0053255050000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":549,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:24.854519+0000","last_change":"2024-09-17T07:25:17.089768+0000","last_active":"2024-09-17T07:25:24.854519+0000","last_peered":"2024-09-17T07:25:24.854519+0000","last_clean":"2024-09-17T07:25:24.854519+0000","last_became_active":"2024-09-17T07:23:57.473995+0000","last_became_peered":"2024-09-17T07:23:57.473995+0000","last_unstale":"2024-09-17T07:25:24.854519+0000","last_undegraded":"2024-09-17T07:25:24.854519+0000","last_fullsized":"2024-09-17T07:25:24.854519+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:17.089691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:25:17.089691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:35.172830+0000","scrub_duration":0.0010547029999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"439'35","reported_seq":722,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:29.498735+0000","last_change":"2024-09-17T07:25:29.498735+0000","last_active":"2024-09-17T07:25:29.498735+0000","last_peered":"2024-09-17T07:25:29.498735+0000","last_clean":"2024-09-17T07:25:29.498735+0000","last_became_active":"2024-09-17T07:23:57.479217+0000","last_became_peered":"2024-09-17T07:23:57.479217+0000","last_unstale":"2024-09-17T07:25:29.498735+0000","last_undegraded":"2024-09-17T07:25:29.498735+0000","last_fullsized":"2024-09-17T07:25:29.498735+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'35","last_scrub_stamp":"2024-09-17T07:25:29.498641+0000","last_deep_scrub":"294'25","last_deep_scrub_stamp":"2024-09-17T07:16:00.471311+0000","last_clean_scrub_stamp":"2024-09-17T07:25:29.498641+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:54.713219+0000","scrub_duration":0.0022200950000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"436'10","reported_seq":889,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-17T07:25:29.893669+0000","last_change":"2024-09-17T07:25:01.611646+0000","last_active":"2024-09-17T07:25:29.893669+0000","last_peered":"2024-09-17T07:25:29.893669+0000","last_clean":"2024-09-17T07:25:29.893669+0000","last_became_active":"2024-09-17T07:22:13.574012+0000","last_became_peered":"2024-09-17T07:22:13.574012+0000","last_unstale":"2024-09-17T07:25:29.893669+0000","last_undegraded":"2024-09-17T07:25:29.893669+0000","last_fullsized":"2024-09-17T07:25:29.893669+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"436'10","last_scrub_stamp":"2024-09-17T07:25:01.611504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:25:01.611504+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:07.992598+0000","scrub_duration":0.001846009,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":422,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:24.766219+0000","last_change":"2024-09-17T07:25:12.891473+0000","last_active":"2024-09-17T07:25:24.766219+0000","last_peered":"2024-09-17T07:25:24.766219+0000","last_clean":"2024-09-17T07:25:24.766219+0000","last_became_active":"2024-09-17T07:23:57.478429+0000","last_became_peered":"2024-09-17T07:23:57.478429+0000","last_unstale":"2024-09-17T07:25:24.766219+0000","last_undegraded":"2024-09-17T07:25:24.766219+0000","last_fullsized":"2024-09-17T07:25:24.766219+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:12.891393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:25:12.891393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:13.870895+0000","scrub_duration":0.0019788359999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":413,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:25.502297+0000","last_change":"2024-09-17T07:24:29.609437+0000","last_active":"2024-09-17T07:25:25.502297+0000","last_peered":"2024-09-17T07:25:25.502297+0000","last_clean":"2024-09-17T07:25:25.502297+0000","last_became_active":"2024-09-17T07:09:24.403360+0000","last_became_peered":"2024-09-17T07:09:24.403360+0000","last_unstale":"2024-09-17T07:25:25.502297+0000","last_undegraded":"2024-09-17T07:25:25.502297+0000","last_fullsized":"2024-09-17T07:25:25.502297+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:24:29.609321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:10:29.325664+0000","last_clean_scrub_stamp":"2024-09-17T07:24:29.609321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:50.013021+0000","scrub_duration":0.024336869000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"513'32","reported_seq":688,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-17T07:25:31.401382+0000","last_change":"2024-09-17T07:25:31.401458+0000","last_active":"2024-09-17T07:25:31.401382+0000","last_peered":"2024-09-17T07:25:31.401382+0000","last_clean":"2024-09-17T07:25:31.401382+0000","last_became_active":"2024-09-17T07:23:57.486951+0000","last_became_peered":"2024-09-17T07:23:57.486951+0000","last_unstale":"2024-09-17T07:25:31.401382+0000","last_undegraded":"2024-09-17T07:25:31.401382+0000","last_fullsized":"2024-09-17T07:25:31.401382+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"513'32","last_scrub_stamp":"2024-09-17T07:25:31.401365+0000","last_deep_scrub":"513'32","last_deep_scrub_stamp":"2024-09-17T07:25:31.401365+0000","last_clean_scrub_stamp":"2024-09-17T07:25:31.401365+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:00.707140+0000","scrub_duration":0.0038053710000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":597,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:24.767851+0000","last_change":"2024-09-17T07:25:24.557776+0000","last_active":"2024-09-17T07:25:24.767851+0000","last_peered":"2024-09-17T07:25:24.767851+0000","last_clean":"2024-09-17T07:25:24.767851+0000","last_became_active":"2024-09-17T07:24:43.779553+0000","last_became_peered":"2024-09-17T07:24:43.779553+0000","last_unstale":"2024-09-17T07:25:24.767851+0000","last_undegraded":"2024-09-17T07:25:24.767851+0000","last_fullsized":"2024-09-17T07:25:24.767851+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:24.557678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:20:37.435106+0000","last_clean_scrub_stamp":"2024-09-17T07:25:24.557678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:47.573258+0000","scrub_duration":0.0018428959999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":574,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:24.767074+0000","last_change":"2024-09-17T07:25:19.870297+0000","last_active":"2024-09-17T07:25:24.767074+0000","last_peered":"2024-09-17T07:25:24.767074+0000","last_clean":"2024-09-17T07:25:24.767074+0000","last_became_active":"2024-09-17T07:23:57.473629+0000","last_became_peered":"2024-09-17T07:23:57.473629+0000","last_unstale":"2024-09-17T07:25:24.767074+0000","last_undegraded":"2024-09-17T07:25:24.767074+0000","last_fullsized":"2024-09-17T07:25:24.767074+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:19.870196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:25:19.870196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:26.879292+0000","scrub_duration":0.0043293369999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":612,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:25.502779+0000","last_change":"2024-09-17T07:25:09.562025+0000","last_active":"2024-09-17T07:25:25.502779+0000","last_peered":"2024-09-17T07:25:25.502779+0000","last_clean":"2024-09-17T07:25:25.502779+0000","last_became_active":"2024-09-17T07:22:13.587273+0000","last_became_peered":"2024-09-17T07:22:13.587273+0000","last_unstale":"2024-09-17T07:25:25.502779+0000","last_undegraded":"2024-09-17T07:25:25.502779+0000","last_fullsized":"2024-09-17T07:25:25.502779+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:09.561794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:11:18.559877+0000","last_clean_scrub_stamp":"2024-09-17T07:25:09.561794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:21.705259+0000","scrub_duration":0.001513239,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"439'39","reported_seq":498,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:24.765731+0000","last_change":"2024-09-17T07:24:43.799133+0000","last_active":"2024-09-17T07:25:24.765731+0000","last_peered":"2024-09-17T07:25:24.765731+0000","last_clean":"2024-09-17T07:25:24.765731+0000","last_became_active":"2024-09-17T07:24:43.771752+0000","last_became_peered":"2024-09-17T07:24:43.771752+0000","last_unstale":"2024-09-17T07:25:24.765731+0000","last_undegraded":"2024-09-17T07:25:24.765731+0000","last_fullsized":"2024-09-17T07:25:24.765731+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'27","last_scrub_stamp":"2024-09-17T07:21:11.488225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:21:11.488225+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0089238530000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":386,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:24.766667+0000","last_change":"2024-09-17T07:24:23.943586+0000","last_active":"2024-09-17T07:25:24.766667+0000","last_peered":"2024-09-17T07:25:24.766667+0000","last_clean":"2024-09-17T07:25:24.766667+0000","last_became_active":"2024-09-17T07:21:15.348754+0000","last_became_peered":"2024-09-17T07:21:15.348754+0000","last_unstale":"2024-09-17T07:25:24.766667+0000","last_undegraded":"2024-09-17T07:25:24.766667+0000","last_fullsized":"2024-09-17T07:25:24.766667+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:24:23.943437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:24:23.943437+0000","last_clean_scrub_stamp":"2024-09-17T07:24:23.943437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:47.203411+0000","scrub_duration":0.012336285000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":433,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:25.501876+0000","last_change":"2024-09-17T07:25:18.515504+0000","last_active":"2024-09-17T07:25:25.501876+0000","last_peered":"2024-09-17T07:25:25.501876+0000","last_clean":"2024-09-17T07:25:25.501876+0000","last_became_active":"2024-09-17T07:23:57.487706+0000","last_became_peered":"2024-09-17T07:23:57.487706+0000","last_unstale":"2024-09-17T07:25:25.501876+0000","last_undegraded":"2024-09-17T07:25:25.501876+0000","last_fullsized":"2024-09-17T07:25:25.501876+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:18.515421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:25:18.515421+0000","last_clean_scrub_stamp":"2024-09-17T07:25:18.515421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:35.662814+0000","scrub_duration":0.001450956,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":101,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:25.504492+0000","last_change":"2024-09-17T07:24:26.807871+0000","last_active":"2024-09-17T07:25:25.504492+0000","last_peered":"2024-09-17T07:25:25.504492+0000","last_clean":"2024-09-17T07:25:25.504492+0000","last_became_active":"2024-09-17T07:22:13.624057+0000","last_became_peered":"2024-09-17T07:22:13.624057+0000","last_unstale":"2024-09-17T07:25:25.504492+0000","last_undegraded":"2024-09-17T07:25:25.504492+0000","last_fullsized":"2024-09-17T07:25:25.504492+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:24:26.807790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:24:26.807790+0000","last_clean_scrub_stamp":"2024-09-17T07:24:26.807790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:39.305925+0000","scrub_duration":0.001095604,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"439'29","reported_seq":327,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:24.766352+0000","last_change":"2024-09-17T07:25:03.872269+0000","last_active":"2024-09-17T07:25:24.766352+0000","last_peered":"2024-09-17T07:25:24.766352+0000","last_clean":"2024-09-17T07:25:24.766352+0000","last_became_active":"2024-09-17T07:21:15.351512+0000","last_became_peered":"2024-09-17T07:21:15.351512+0000","last_unstale":"2024-09-17T07:25:24.766352+0000","last_undegraded":"2024-09-17T07:25:24.766352+0000","last_fullsized":"2024-09-17T07:25:24.766352+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'29","last_scrub_stamp":"2024-09-17T07:25:03.872175+0000","last_deep_scrub":"86'17","last_deep_scrub_stamp":"2024-09-17T07:03:18.241067+0000","last_clean_scrub_stamp":"2024-09-17T07:25:03.872175+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:09.824307+0000","scrub_duration":0.001972002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":597,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-09-17T07:25:24.766649+0000","last_change":"2024-09-17T07:25:20.895225+0000","last_active":"2024-09-17T07:25:24.766649+0000","last_peered":"2024-09-17T07:25:24.766649+0000","last_clean":"2024-09-17T07:25:24.766649+0000","last_became_active":"2024-09-17T07:22:13.590622+0000","last_became_peered":"2024-09-17T07:22:13.590622+0000","last_unstale":"2024-09-17T07:25:24.766649+0000","last_undegraded":"2024-09-17T07:25:24.766649+0000","last_fullsized":"2024-09-17T07:25:24.766649+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:25:20.895149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:25:20.895149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:43.686040+0000","scrub_duration":0.0015445599999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":382,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1537,"num_read_kb":1294,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":254,"num_bytes_recovered":5066,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":13004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1032,"ondisk_log_size":1032,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":264,"num_read_kb":259,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1386,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":12,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":4578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":703368,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":136,"num_read_kb":194,"num_write":115,"num_write_kb":1536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1127066,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1430,"ondisk_log_size":1430,"up":33,"acting":33,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698038046,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1112904,"kb_used_data":2644,"kb_used_omap":1,"kb_used_meta":1109886,"kb_avail":92632248,"statfs":{"total":95995035648,"available":94855421952,"internally_reserved":0,"allocated":2707456,"data_stored":1700504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":1136523723},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:24:46 2024","interfaces":[{"interface":"back","average":{"1min":0.95399999999999996,"5min":1.7030000000000001,"15min":1.5129999999999999},"min":{"1min":0.30499999999999999,"5min":0.182,"15min":0.182},"max":{"1min":3.306,"5min":43.905000000000001,"15min":43.905000000000001},"last":0.51200000000000001},{"interface":"front","average":{"1min":1.131,"5min":1.5720000000000001,"15min":1.5640000000000001},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":5.6619999999999999,"5min":44.432000000000002,"15min":44.432000000000002},"last":0.48699999999999999}]},{"osd":1,"last update":"Tue Sep 17 07:25:21 2024","interfaces":[{"interface":"back","average":{"1min":1.1299999999999999,"5min":1.1439999999999999,"15min":1.216},"min":{"1min":0.32900000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":3.948,"5min":12.112,"15min":12.112},"last":0.57199999999999995},{"interface":"front","average":{"1min":1.2749999999999999,"5min":1.097,"15min":1.179},"min":{"1min":0.378,"5min":0.216,"15min":0.216},"max":{"1min":5.6210000000000004,"5min":12.273,"15min":12.273},"last":0.46600000000000003}]},{"osd":2,"last update":"Tue Sep 17 07:24:46 2024","interfaces":[{"interface":"back","average":{"1min":1.1779999999999999,"5min":0.86799999999999999,"15min":1.0589999999999999},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":4.165,"5min":10.972,"15min":10.972},"last":0.58699999999999997},{"interface":"front","average":{"1min":1.226,"5min":0.97799999999999998,"15min":1.0429999999999999},"min":{"1min":0.35699999999999998,"5min":0.255,"15min":0.224},"max":{"1min":4.4320000000000004,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.45300000000000001}]},{"osd":3,"last update":"Tue Sep 17 07:24:46 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":0.69099999999999995,"15min":1.0109999999999999},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":4.2009999999999996,"5min":4.2009999999999996,"15min":10.122999999999999},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.629,"15min":0.98299999999999998},"min":{"1min":0.313,"5min":0.26700000000000002,"15min":0.26500000000000001},"max":{"1min":3.0630000000000002,"5min":4.1719999999999997,"15min":12.465999999999999},"last":0.54100000000000004}]},{"osd":4,"last update":"Tue Sep 17 07:24:46 2024","interfaces":[{"interface":"back","average":{"1min":1.345,"5min":0.78000000000000003,"15min":1.113},"min":{"1min":0.23699999999999999,"5min":0.22900000000000001,"15min":0.222},"max":{"1min":6.21,"5min":13.316000000000001,"15min":21.140999999999998},"last":0.39400000000000002},{"interface":"front","average":{"1min":1.2589999999999999,"5min":0.82999999999999996,"15min":1.296},"min":{"1min":0.25800000000000001,"5min":0.251,"15min":0.23599999999999999},"max":{"1min":4.2850000000000001,"5min":12.272,"15min":19.420000000000002},"last":0.55100000000000005}]},{"osd":5,"last update":"Tue Sep 17 07:24:57 2024","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":3.0859999999999999,"5min":3.0859999999999999,"15min":3.0859999999999999},"last":0.41699999999999998},{"interface":"front","average":{"1min":1.097,"5min":1.097,"15min":1.097},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":5.7350000000000003,"5min":5.7350000000000003,"15min":5.7350000000000003},"last":0.53200000000000003}]},{"osd":6,"last update":"Tue Sep 17 07:24:46 2024","interfaces":[{"interface":"back","average":{"1min":1.4079999999999999,"5min":1.4950000000000001,"15min":1.6040000000000001},"min":{"1min":0.36399999999999999,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":6.9359999999999999,"5min":60.445999999999998,"15min":60.445999999999998},"last":0.56200000000000006},{"interface":"front","average":{"1min":1.5389999999999999,"5min":1.5580000000000001,"15min":1.71},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":4.3550000000000004,"5min":60.332000000000001,"15min":60.332000000000001},"last":0.47599999999999998}]}]},{"osd":6,"up_from":50,"seq":214748365089,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1034868,"kb_used_data":2924,"kb_used_omap":1,"kb_used_meta":1031934,"kb_avail":92710284,"statfs":{"total":95995035648,"available":94935330816,"internally_reserved":0,"allocated":2994176,"data_stored":1963717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1056700871},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:25:31 2024","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":3.8069999999999999,"15min":2.3820000000000001},"min":{"1min":0.27000000000000002,"5min":0.19700000000000001,"15min":0.185},"max":{"1min":5.7709999999999999,"5min":110.376,"15min":126.902},"last":0.30099999999999999},{"interface":"front","average":{"1min":1.752,"5min":3.8109999999999999,"15min":2.294},"min":{"1min":0.27200000000000002,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":7.157,"5min":110.164,"15min":127.06399999999999},"last":0.36899999999999999}]},{"osd":1,"last update":"Tue Sep 17 07:25:23 2024","interfaces":[{"interface":"back","average":{"1min":1.665,"5min":4.944,"15min":5.6799999999999997},"min":{"1min":0.36099999999999999,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":7.1059999999999999,"5min":199.32400000000001,"15min":199.32400000000001},"last":0.52500000000000002},{"interface":"front","average":{"1min":1.4730000000000001,"5min":5.0579999999999998,"15min":5.9619999999999997},"min":{"1min":0.312,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":8.9399999999999995,"5min":198.88499999999999,"15min":198.88499999999999},"last":0.45500000000000002}]},{"osd":2,"last update":"Tue Sep 17 07:25:31 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":1.333,"15min":1.1970000000000001},"min":{"1min":0.29399999999999998,"5min":0.23300000000000001,"15min":0.219},"max":{"1min":2.9529999999999998,"5min":30.664999999999999,"15min":36.289000000000001},"last":0.40300000000000002},{"interface":"front","average":{"1min":1.333,"5min":1.379,"15min":1.115},"min":{"1min":0.41899999999999998,"5min":0.26100000000000001,"15min":0.22900000000000001},"max":{"1min":7.1280000000000001,"5min":30.622,"15min":36.326999999999998},"last":0.441}]},{"osd":3,"last update":"Tue Sep 17 07:25:31 2024","interfaces":[{"interface":"back","average":{"1min":1.1319999999999999,"5min":0.82499999999999996,"15min":0.90200000000000002},"min":{"1min":0.29999999999999999,"5min":0.27700000000000002,"15min":0.24399999999999999},"max":{"1min":9.0579999999999998,"5min":9.0579999999999998,"15min":9.0579999999999998},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.996,"15min":1.0469999999999999},"min":{"1min":0.33400000000000002,"5min":0.28000000000000003,"15min":0.27500000000000002},"max":{"1min":5.7400000000000002,"5min":10.999000000000001,"15min":10.999000000000001},"last":0.34100000000000003}]},{"osd":4,"last update":"Tue Sep 17 07:25:31 2024","interfaces":[{"interface":"back","average":{"1min":1.4330000000000001,"5min":1.0660000000000001,"15min":2.355},"min":{"1min":0.24199999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":7.0119999999999996,"5min":9.5419999999999998,"15min":260.77199999999999},"last":0.50800000000000001},{"interface":"front","average":{"1min":1.5089999999999999,"5min":0.86199999999999999,"15min":2.375},"min":{"1min":0.28399999999999997,"5min":0.26100000000000001,"15min":0.19800000000000001},"max":{"1min":9.0310000000000006,"5min":9.0310000000000006,"15min":260.69099999999997},"last":0.35099999999999998}]},{"osd":5,"last update":"Tue Sep 17 07:24:56 2024","interfaces":[{"interface":"back","average":{"1min":2.1280000000000001,"5min":2.1280000000000001,"15min":2.1280000000000001},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":13.551,"5min":13.551,"15min":13.551},"last":0.433},{"interface":"front","average":{"1min":2.387,"5min":2.387,"15min":2.387},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":8.1720000000000006,"5min":8.1720000000000006,"15min":8.1720000000000006},"last":0.46999999999999997}]},{"osd":7,"last update":"Tue Sep 17 07:24:51 2024","interfaces":[{"interface":"back","average":{"1min":1.948,"5min":1.0229999999999999,"15min":1.784},"min":{"1min":0.33600000000000002,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":9.1199999999999992,"5min":9.1199999999999992,"15min":131.95699999999999},"last":0.48199999999999998},{"interface":"front","average":{"1min":2.206,"5min":1.149,"15min":1.6950000000000001},"min":{"1min":0.31900000000000001,"5min":0.26900000000000002,"15min":0.252},"max":{"1min":7.077,"5min":9.5850000000000009,"15min":131.833},"last":0.49099999999999999}]}]},{"osd":5,"up_from":476,"seq":2044404432916,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":317636,"kb_used_data":183632,"kb_used_omap":4,"kb_used_meta":133627,"kb_avail":93427516,"statfs":{"total":95995035648,"available":95669776384,"internally_reserved":0,"allocated":188039168,"data_stored":187035199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4441,"internal_metadata":136834727},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:25:00 2024","interfaces":[{"interface":"back","average":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":3.4220000000000002,"5min":3.4220000000000002,"15min":3.4220000000000002},"last":0.47299999999999998},{"interface":"front","average":{"1min":1.5960000000000001,"5min":1.5960000000000001,"15min":1.5960000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":6.7619999999999996,"5min":6.7619999999999996,"15min":6.7619999999999996},"last":0.48999999999999999}]},{"osd":1,"last update":"Tue Sep 17 07:25:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":3.476,"5min":3.476,"15min":3.476},"last":0.46100000000000002},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":6.665,"5min":6.665,"15min":6.665},"last":0.41499999999999998}]},{"osd":2,"last update":"Tue Sep 17 07:25:00 2024","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.2190000000000001,"15min":1.2190000000000001},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":3.544,"5min":3.544,"15min":3.544},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":2.8359999999999999,"5min":2.8359999999999999,"15min":2.8359999999999999},"last":0.52100000000000002}]},{"osd":3,"last update":"Tue Sep 17 07:25:00 2024","interfaces":[{"interface":"back","average":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":3.7040000000000002,"5min":3.7040000000000002,"15min":3.7040000000000002},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":2.8719999999999999,"5min":2.8719999999999999,"15min":2.8719999999999999},"last":0.499}]},{"osd":4,"last update":"Tue Sep 17 07:25:00 2024","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":3.4399999999999999,"5min":3.4399999999999999,"15min":3.4399999999999999},"last":0.35999999999999999},{"interface":"front","average":{"1min":1.26,"5min":1.26,"15min":1.26},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":3.0139999999999998,"5min":3.0139999999999998,"15min":3.0139999999999998},"last":0.38500000000000001}]},{"osd":6,"last update":"Tue Sep 17 07:25:00 2024","interfaces":[{"interface":"back","average":{"1min":1.77,"5min":1.77,"15min":1.77},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":10.92,"5min":10.92,"15min":10.92},"last":0.34499999999999997},{"interface":"front","average":{"1min":1.526,"5min":1.526,"15min":1.526},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":10.202999999999999,"5min":10.202999999999999,"15min":10.202999999999999},"last":0.44700000000000001}]},{"osd":7,"last update":"Tue Sep 17 07:25:00 2024","interfaces":[{"interface":"back","average":{"1min":3.3820000000000001,"5min":3.3820000000000001,"15min":3.3820000000000001},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":45.555,"5min":45.555,"15min":45.555},"last":0.434},{"interface":"front","average":{"1min":3.6920000000000002,"5min":3.6920000000000002,"15min":3.6920000000000002},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":47.213999999999999,"5min":47.213999999999999,"15min":47.213999999999999},"last":0.40300000000000002}]}]},{"osd":4,"up_from":33,"seq":141733921068,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1004296,"kb_used_data":2324,"kb_used_omap":1,"kb_used_meta":1001598,"kb_avail":92740856,"statfs":{"total":95995035648,"available":94966636544,"internally_reserved":0,"allocated":2379776,"data_stored":1372164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1025636806},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":1.8240000000000001,"15min":1.3899999999999999},"min":{"1min":0.251,"5min":0.20300000000000001,"15min":0.20000000000000001},"max":{"1min":1.3720000000000001,"5min":39.473999999999997,"15min":39.473999999999997},"last":0.623},{"interface":"front","average":{"1min":0.54900000000000004,"5min":1.8540000000000001,"15min":1.48},"min":{"1min":0.33600000000000002,"5min":0.20599999999999999,"15min":0.20499999999999999},"max":{"1min":1.359,"5min":39.654000000000003,"15min":39.654000000000003},"last":0.54300000000000004}]},{"osd":1,"last update":"Tue Sep 17 07:25:21 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":2.0790000000000002,"15min":2.613},"min":{"1min":0.318,"5min":0.221,"15min":0.221},"max":{"1min":2.1589999999999998,"5min":39.439,"15min":39.439},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.91200000000000003,"5min":2.2480000000000002,"15min":2.6859999999999999},"min":{"1min":0.36699999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":3.4510000000000001,"5min":39.325000000000003,"15min":39.325000000000003},"last":0.63700000000000001}]},{"osd":2,"last update":"Tue Sep 17 07:25:32 2024","interfaces":[{"interface":"back","average":{"1min":7.9420000000000002,"5min":2.399,"15min":1.51},"min":{"1min":0.29599999999999999,"5min":0.24399999999999999,"15min":0.23499999999999999},"max":{"1min":126.628,"5min":126.628,"15min":126.628},"last":0.56399999999999995},{"interface":"front","average":{"1min":7.9770000000000003,"5min":2.359,"15min":1.536},"min":{"1min":0.34599999999999997,"5min":0.26600000000000001,"15min":0.23400000000000001},"max":{"1min":126.41200000000001,"5min":126.41200000000001,"15min":126.41200000000001},"last":0.65800000000000003}]},{"osd":3,"last update":"Tue Sep 17 07:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.68600000000000005,"15min":0.90800000000000003},"min":{"1min":0.34999999999999998,"5min":0.28299999999999997,"15min":0.25800000000000001},"max":{"1min":1.3300000000000001,"5min":2.8929999999999998,"15min":15.433999999999999},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.74399999999999999,"15min":0.96599999999999997},"min":{"1min":0.34399999999999997,"5min":0.29699999999999999,"15min":0.27100000000000002},"max":{"1min":1.9590000000000001,"5min":8.7360000000000007,"15min":16.111000000000001},"last":0.72199999999999998}]},{"osd":5,"last update":"Tue Sep 17 07:25:00 2024","interfaces":[{"interface":"back","average":{"1min":1.4359999999999999,"5min":1.4359999999999999,"15min":1.4359999999999999},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":13.128,"5min":13.128,"15min":13.128},"last":0.434},{"interface":"front","average":{"1min":1.752,"5min":1.752,"15min":1.752},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":13.218999999999999,"5min":13.218999999999999,"15min":13.218999999999999},"last":0.378}]},{"osd":6,"last update":"Tue Sep 17 07:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.81899999999999995,"15min":1.0640000000000001},"min":{"1min":0.317,"5min":0.249,"15min":0.218},"max":{"1min":1.901,"5min":4.7729999999999997,"15min":8.9619999999999997},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.92300000000000004,"15min":1.208},"min":{"1min":0.28799999999999998,"5min":0.249,"15min":0.24199999999999999},"max":{"1min":5.0179999999999998,"5min":6.782,"15min":10.051},"last":0.35799999999999998}]},{"osd":7,"last update":"Tue Sep 17 07:24:45 2024","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":0.98899999999999999,"15min":1.2410000000000001},"min":{"1min":0.318,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":6.2249999999999996,"5min":6.2249999999999996,"15min":7.1550000000000002},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.98899999999999999,"5min":0.78600000000000003,"15min":1.1910000000000001},"min":{"1min":0.28599999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":2.5739999999999998,"5min":4.7439999999999998,"15min":9.0030000000000001},"last":0.45400000000000001}]}]},{"osd":3,"up_from":0,"seq":115964117294,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":18,"seq":77309411636,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1120780,"kb_used_data":2584,"kb_used_omap":1,"kb_used_meta":1117822,"kb_avail":92624372,"statfs":{"total":95995035648,"available":94847356928,"internally_reserved":0,"allocated":2646016,"data_stored":1635696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1144650182},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:25:06 2024","interfaces":[{"interface":"back","average":{"1min":1.966,"5min":3.5059999999999998,"15min":2.6699999999999999},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.154},"max":{"1min":25.98,"5min":69.135999999999996,"15min":131.09800000000001},"last":0.28699999999999998},{"interface":"front","average":{"1min":2.073,"5min":3.73,"15min":2.6949999999999998},"min":{"1min":0.25600000000000001,"5min":0.17999999999999999,"15min":0.17599999999999999},"max":{"1min":24.314,"5min":69.033000000000001,"15min":131.24700000000001},"last":0.26000000000000001}]},{"osd":1,"last update":"Tue Sep 17 07:25:27 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":2.5750000000000002,"15min":2.1699999999999999},"min":{"1min":0.23000000000000001,"5min":0.184,"15min":0.184},"max":{"1min":0.876,"5min":91.686999999999998,"15min":91.686999999999998},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.83099999999999996,"5min":2.903,"15min":2.5089999999999999},"min":{"1min":0.251,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":4.8029999999999999,"5min":93.744,"15min":93.744},"last":0.377}]},{"osd":3,"last update":"Tue Sep 17 07:25:13 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.61099999999999999,"15min":0.67800000000000005},"min":{"1min":0.251,"5min":0.20499999999999999,"15min":0.20100000000000001},"max":{"1min":0.99099999999999999,"5min":3.1880000000000002,"15min":23.43},"last":0.317},{"interface":"front","average":{"1min":0.69499999999999995,"5min":1.0009999999999999,"15min":0.78500000000000003},"min":{"1min":0.29799999999999999,"5min":0.22500000000000001,"15min":0.22},"max":{"1min":2.4769999999999999,"5min":16.215,"15min":23.408000000000001},"last":0.32800000000000001}]},{"osd":4,"last update":"Tue Sep 17 07:24:40 2024","interfaces":[{"interface":"back","average":{"1min":2.2839999999999998,"5min":1.0920000000000001,"15min":1.3700000000000001},"min":{"1min":0.314,"5min":0.254,"15min":0.24399999999999999},"max":{"1min":25.225999999999999,"5min":25.225999999999999,"15min":75.870999999999995},"last":0.70099999999999996},{"interface":"front","average":{"1min":2.3559999999999999,"5min":1.149,"15min":1.47},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.26300000000000001},"max":{"1min":25.268999999999998,"5min":25.268999999999998,"15min":75.888999999999996},"last":0.75900000000000001}]},{"osd":5,"last update":"Tue Sep 17 07:24:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":1.9610000000000001,"5min":1.9610000000000001,"15min":1.9610000000000001},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":3.907,"5min":3.907,"15min":3.907},"last":0.70699999999999996}]},{"osd":6,"last update":"Tue Sep 17 07:25:13 2024","interfaces":[{"interface":"back","average":{"1min":2.8969999999999998,"5min":2.298,"15min":1.8169999999999999},"min":{"1min":0.32800000000000001,"5min":0.29699999999999999,"15min":0.27500000000000002},"max":{"1min":24.936,"5min":65.143000000000001,"15min":76.081999999999994},"last":0.72699999999999998},{"interface":"front","average":{"1min":2.2839999999999998,"5min":2.1680000000000001,"15min":1.871},"min":{"1min":0.40899999999999997,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":24.884,"5min":65.241,"15min":76.244},"last":0.65900000000000003}]},{"osd":7,"last update":"Tue Sep 17 07:25:06 2024","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.83699999999999997,"15min":1.3520000000000001},"min":{"1min":0.34799999999999998,"5min":0.314,"15min":0.314},"max":{"1min":2.0790000000000002,"5min":4.2720000000000002,"15min":94.269999999999996},"last":0.64700000000000002},{"interface":"front","average":{"1min":1.016,"5min":0.95899999999999996,"15min":1.395},"min":{"1min":0.34499999999999997,"5min":0.32900000000000001,"15min":0.309},"max":{"1min":4.4470000000000001,"5min":4.4470000000000001,"15min":94.634},"last":0.68899999999999995}]}]},{"osd":1,"up_from":413,"seq":1773821493301,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":536056,"kb_used_data":2352,"kb_used_omap":4,"kb_used_meta":533691,"kb_avail":93209096,"statfs":{"total":95995035648,"available":95446114304,"internally_reserved":0,"allocated":2408448,"data_stored":1374070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4439,"internal_metadata":546500265},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:25:26 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":2.5990000000000002,"15min":1.98},"min":{"1min":0.19600000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":4.5730000000000004,"5min":120.56399999999999,"15min":120.56399999999999},"last":0.187},{"interface":"front","average":{"1min":0.60599999999999998,"5min":2.5819999999999999,"15min":1.7869999999999999},"min":{"1min":0.26900000000000002,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":4.1150000000000002,"5min":120.746,"15min":120.746},"last":0.28100000000000003}]},{"osd":2,"last update":"Tue Sep 17 07:25:26 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":1.0489999999999999,"15min":1.2569999999999999},"min":{"1min":0.35699999999999998,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":3.516,"5min":6.7300000000000004,"15min":6.7300000000000004},"last":0.24199999999999999},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.877,"15min":1.0449999999999999},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":3.4830000000000001,"5min":3.919,"15min":3.919},"last":0.26400000000000001}]},{"osd":3,"last update":"Tue Sep 17 07:25:26 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.747,"15min":0.79700000000000004},"min":{"1min":0.26200000000000001,"5min":0.253,"15min":0.253},"max":{"1min":4.6749999999999998,"5min":5.9290000000000003,"15min":5.9290000000000003},"last":0.22600000000000001},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.76100000000000001,"15min":0.83999999999999997},"min":{"1min":0.29599999999999999,"5min":0.25,"15min":0.25},"max":{"1min":3.3149999999999999,"5min":4.1340000000000003,"15min":4.1340000000000003},"last":0.35499999999999998}]},{"osd":4,"last update":"Tue Sep 17 07:25:26 2024","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":3.4809999999999999,"15min":6.0650000000000004},"min":{"1min":0.29599999999999999,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":2.4849999999999999,"5min":116.88200000000001,"15min":116.88200000000001},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.92900000000000005,"5min":3.4540000000000002,"15min":6.1020000000000003},"min":{"1min":0.36799999999999999,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":4.7089999999999996,"5min":116.98399999999999,"15min":116.98399999999999},"last":0.316}]},{"osd":5,"last update":"Tue Sep 17 07:24:57 2024","interfaces":[{"interface":"back","average":{"1min":1.5369999999999999,"5min":1.5369999999999999,"15min":1.5369999999999999},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":9.7409999999999997,"5min":9.7409999999999997,"15min":9.7409999999999997},"last":0.433},{"interface":"front","average":{"1min":1.542,"5min":1.542,"15min":1.542},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":11.538,"5min":11.538,"15min":11.538},"last":0.374}]},{"osd":6,"last update":"Tue Sep 17 07:25:26 2024","interfaces":[{"interface":"back","average":{"1min":1.1359999999999999,"5min":2.052,"15min":1.6379999999999999},"min":{"1min":0.36699999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":4.7519999999999998,"5min":56.686999999999998,"15min":56.686999999999998},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.90900000000000003,"5min":1.972,"15min":1.542},"min":{"1min":0.437,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":3.4900000000000002,"5min":56.420999999999999,"15min":56.420999999999999},"last":0.45900000000000002}]},{"osd":7,"last update":"Tue Sep 17 07:25:26 2024","interfaces":[{"interface":"back","average":{"1min":0.97099999999999997,"5min":1.0069999999999999,"15min":1.0469999999999999},"min":{"1min":0.38900000000000001,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":3.4399999999999999,"5min":4.2389999999999999,"15min":4.2389999999999999},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.872,"5min":0.89400000000000002,"15min":1},"min":{"1min":0.38,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":3.351,"5min":4.2839999999999998,"15min":4.2839999999999998},"last":0.41299999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738684,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1622824,"kb_used_data":319904,"kb_used_omap":1,"kb_used_meta":1302910,"kb_avail":92122328,"statfs":{"total":95995035648,"available":94333263872,"internally_reserved":0,"allocated":327581696,"data_stored":326562840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1334180294},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 17 07:25:22 2024","interfaces":[{"interface":"back","average":{"1min":3.585,"5min":1.9419999999999999,"15min":1.986},"min":{"1min":0.20599999999999999,"5min":0.156,"15min":0.156},"max":{"1min":53.731000000000002,"5min":53.731000000000002,"15min":53.731000000000002},"last":0.41799999999999998},{"interface":"front","average":{"1min":4.0179999999999998,"5min":1.827,"15min":1.6140000000000001},"min":{"1min":0.217,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":55.789000000000001,"5min":55.789000000000001,"15min":55.789000000000001},"last":0.36899999999999999}]},{"osd":2,"last update":"Tue Sep 17 07:24:59 2024","interfaces":[{"interface":"back","average":{"1min":3.9529999999999998,"5min":2.4750000000000001,"15min":1.256},"min":{"1min":0.21299999999999999,"5min":0.19400000000000001,"15min":0.155},"max":{"1min":55.756,"5min":62.094999999999999,"15min":62.094999999999999},"last":0.439},{"interface":"front","average":{"1min":3.9180000000000001,"5min":2.476,"15min":1.242},"min":{"1min":0.255,"5min":0.19,"15min":0.183},"max":{"1min":57.854999999999997,"5min":59.848999999999997,"15min":59.848999999999997},"last":0.40100000000000002}]},{"osd":3,"last update":"Tue Sep 17 07:24:35 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.74299999999999999,"15min":0.69099999999999995},"min":{"1min":0.27700000000000002,"5min":0.23200000000000001,"15min":0.21199999999999999},"max":{"1min":1.1910000000000001,"5min":4.6829999999999998,"15min":5.5880000000000001},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.80400000000000005,"15min":0.67400000000000004},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20399999999999999},"max":{"1min":1.024,"5min":5.1150000000000002,"15min":5.1150000000000002},"last":0.48099999999999998}]},{"osd":4,"last update":"Tue Sep 17 07:24:35 2024","interfaces":[{"interface":"back","average":{"1min":1.6990000000000001,"5min":1.208,"15min":1.1459999999999999},"min":{"1min":0.29199999999999998,"5min":0.27400000000000002,"15min":0.23400000000000001},"max":{"1min":10.720000000000001,"5min":10.720000000000001,"15min":19.146999999999998},"last":0.68600000000000005},{"interface":"front","average":{"1min":1.8320000000000001,"5min":1.123,"15min":1.165},"min":{"1min":0.32700000000000001,"5min":0.26600000000000001,"15min":0.26200000000000001},"max":{"1min":10.666,"5min":10.666,"15min":19.158999999999999},"last":0.72699999999999998}]},{"osd":5,"last update":"Tue Sep 17 07:24:59 2024","interfaces":[{"interface":"back","average":{"1min":1.056,"5min":1.056,"15min":1.056},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":3.79,"5min":3.79,"15min":3.79},"last":0.69699999999999995},{"interface":"front","average":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":3.2530000000000001,"5min":3.2530000000000001,"15min":3.2530000000000001},"last":0.63600000000000001}]},{"osd":6,"last update":"Tue Sep 17 07:24:35 2024","interfaces":[{"interface":"back","average":{"1min":1.3009999999999999,"5min":2.2450000000000001,"15min":1.8879999999999999},"min":{"1min":0.34799999999999998,"5min":0.28999999999999998,"15min":0.28199999999999997},"max":{"1min":3.923,"5min":67.873000000000005,"15min":147.13200000000001},"last":0.64700000000000002},{"interface":"front","average":{"1min":1.2809999999999999,"5min":2.359,"15min":1.911},"min":{"1min":0.311,"5min":0.311,"15min":0.30199999999999999},"max":{"1min":4.0229999999999997,"5min":67.914000000000001,"15min":147.172},"last":0.71499999999999997}]},{"osd":7,"last update":"Tue Sep 17 07:24:35 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.88700000000000001,"15min":1.1379999999999999},"min":{"1min":0.35799999999999998,"5min":0.318,"15min":0.29599999999999999},"max":{"1min":2.4820000000000002,"5min":3.843,"15min":6.3300000000000001},"last":0.65600000000000003},{"interface":"front","average":{"1min":1.004,"5min":1.099,"15min":1.218},"min":{"1min":0.36799999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":2.6819999999999999,"5min":6.7300000000000004,"15min":7.2830000000000004},"last":0.61199999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T07:25:37.018 INFO:tasks.thrashosds.thrasher:Setting 6.18 to [0, 1, 4, 7, 2, 3] 2024-09-17T07:25:37.018 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '6.18', '0', '1', '4', '7', '2', '3'] 2024-09-17T07:25:37.018 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd pg-upmap-items 6.18 0 1 4 7 2 3 2024-09-17T07:25:37.126 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:36 smithi100 ceph-mon[39462]: pgmap v1122: 142 pgs: 142 active+clean; 693 KiB data, 6.4 GiB used, 619 GiB / 626 GiB avail 2024-09-17T07:25:37.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:36 smithi100 ceph-mon[39462]: 2.1e scrub starts 2024-09-17T07:25:37.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:36 smithi100 ceph-mon[39462]: 2.1e scrub ok 2024-09-17T07:25:37.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:36 smithi100 ceph-mon[39462]: 1.3 scrub starts 2024-09-17T07:25:37.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:36 smithi100 ceph-mon[39462]: 1.3 scrub ok 2024-09-17T07:25:37.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:36 smithi100 ceph-mon[39462]: 6.1c scrub starts 2024-09-17T07:25:37.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:36 smithi100 ceph-mon[39462]: 6.1c scrub ok 2024-09-17T07:25:37.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:36 smithi100 ceph-mon[39462]: 4.9 scrub starts 2024-09-17T07:25:37.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:36 smithi100 ceph-mon[39462]: 4.9 scrub ok 2024-09-17T07:25:37.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:36 smithi100 ceph-mon[39462]: from='client.26168 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:25:37.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:36 smithi100 ceph-mon[29092]: pgmap v1122: 142 pgs: 142 active+clean; 693 KiB data, 6.4 GiB used, 619 GiB / 626 GiB avail 2024-09-17T07:25:37.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:36 smithi100 ceph-mon[29092]: 2.1e scrub starts 2024-09-17T07:25:37.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:36 smithi100 ceph-mon[29092]: 2.1e scrub ok 2024-09-17T07:25:37.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:36 smithi100 ceph-mon[29092]: 1.3 scrub starts 2024-09-17T07:25:37.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:36 smithi100 ceph-mon[29092]: 1.3 scrub ok 2024-09-17T07:25:37.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:36 smithi100 ceph-mon[29092]: 6.1c scrub starts 2024-09-17T07:25:37.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:36 smithi100 ceph-mon[29092]: 6.1c scrub ok 2024-09-17T07:25:37.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:36 smithi100 ceph-mon[29092]: 4.9 scrub starts 2024-09-17T07:25:37.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:36 smithi100 ceph-mon[29092]: 4.9 scrub ok 2024-09-17T07:25:37.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:36 smithi100 ceph-mon[29092]: from='client.26168 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:25:37.223 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:25:37.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:36 smithi135 ceph-mon[33238]: pgmap v1122: 142 pgs: 142 active+clean; 693 KiB data, 6.4 GiB used, 619 GiB / 626 GiB avail 2024-09-17T07:25:37.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:36 smithi135 ceph-mon[33238]: 2.1e scrub starts 2024-09-17T07:25:37.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:36 smithi135 ceph-mon[33238]: 2.1e scrub ok 2024-09-17T07:25:37.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:36 smithi135 ceph-mon[33238]: 1.3 scrub starts 2024-09-17T07:25:37.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:36 smithi135 ceph-mon[33238]: 1.3 scrub ok 2024-09-17T07:25:37.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:36 smithi135 ceph-mon[33238]: 6.1c scrub starts 2024-09-17T07:25:37.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:36 smithi135 ceph-mon[33238]: 6.1c scrub ok 2024-09-17T07:25:37.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:36 smithi135 ceph-mon[33238]: 4.9 scrub starts 2024-09-17T07:25:37.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:36 smithi135 ceph-mon[33238]: 4.9 scrub ok 2024-09-17T07:25:37.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:36 smithi135 ceph-mon[33238]: from='client.26168 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:25:37.999 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:37 smithi100 ceph-mon[29092]: 2.11 deep-scrub starts 2024-09-17T07:25:37.999 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:37 smithi100 ceph-mon[29092]: 2.11 deep-scrub ok 2024-09-17T07:25:37.999 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:37 smithi100 ceph-mon[29092]: 5.1a scrub starts 2024-09-17T07:25:37.999 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:37 smithi100 ceph-mon[29092]: 5.1a scrub ok 2024-09-17T07:25:37.999 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:37 smithi100 ceph-mon[29092]: 6.6 scrub starts 2024-09-17T07:25:38.000 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:37 smithi100 ceph-mon[29092]: 6.6 scrub ok 2024-09-17T07:25:38.000 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:37 smithi100 ceph-mon[39462]: 2.11 deep-scrub starts 2024-09-17T07:25:38.000 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:37 smithi100 ceph-mon[39462]: 2.11 deep-scrub ok 2024-09-17T07:25:38.000 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:37 smithi100 ceph-mon[39462]: 5.1a scrub starts 2024-09-17T07:25:38.000 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:37 smithi100 ceph-mon[39462]: 5.1a scrub ok 2024-09-17T07:25:38.000 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:37 smithi100 ceph-mon[39462]: 6.6 scrub starts 2024-09-17T07:25:38.000 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:37 smithi100 ceph-mon[39462]: 6.6 scrub ok 2024-09-17T07:25:38.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:37 smithi135 ceph-mon[33238]: 2.11 deep-scrub starts 2024-09-17T07:25:38.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:37 smithi135 ceph-mon[33238]: 2.11 deep-scrub ok 2024-09-17T07:25:38.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:37 smithi135 ceph-mon[33238]: 5.1a scrub starts 2024-09-17T07:25:38.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:37 smithi135 ceph-mon[33238]: 5.1a scrub ok 2024-09-17T07:25:38.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:37 smithi135 ceph-mon[33238]: 6.6 scrub starts 2024-09-17T07:25:38.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:37 smithi135 ceph-mon[33238]: 6.6 scrub ok 2024-09-17T07:25:39.126 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:38 smithi100 ceph-mon[29092]: pgmap v1123: 142 pgs: 142 active+clean; 844 KiB data, 6.4 GiB used, 619 GiB / 626 GiB avail 2024-09-17T07:25:39.126 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:38 smithi100 ceph-mon[29092]: 6.12 scrub starts 2024-09-17T07:25:39.126 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:38 smithi100 ceph-mon[29092]: 6.12 scrub ok 2024-09-17T07:25:39.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:38 smithi100 ceph-mon[39462]: pgmap v1123: 142 pgs: 142 active+clean; 844 KiB data, 6.4 GiB used, 619 GiB / 626 GiB avail 2024-09-17T07:25:39.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:38 smithi100 ceph-mon[39462]: 6.12 scrub starts 2024-09-17T07:25:39.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:38 smithi100 ceph-mon[39462]: 6.12 scrub ok 2024-09-17T07:25:39.194 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:38 smithi135 ceph-mon[33238]: pgmap v1123: 142 pgs: 142 active+clean; 844 KiB data, 6.4 GiB used, 619 GiB / 626 GiB avail 2024-09-17T07:25:39.194 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:38 smithi135 ceph-mon[33238]: 6.12 scrub starts 2024-09-17T07:25:39.195 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:38 smithi135 ceph-mon[33238]: 6.12 scrub ok 2024-09-17T07:25:39.793 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:25:39 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:25:39.829 INFO:teuthology.orchestra.run.smithi100.stderr:set 6.18 pg_upmap_items mapping to [0->1,4->7,2->3] 2024-09-17T07:25:40.126 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:39 smithi100 ceph-mon[29092]: 2.1f scrub starts 2024-09-17T07:25:40.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:39 smithi100 ceph-mon[29092]: 2.1f scrub ok 2024-09-17T07:25:40.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:39 smithi100 ceph-mon[29092]: 4.14 scrub starts 2024-09-17T07:25:40.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:39 smithi100 ceph-mon[29092]: 4.14 scrub ok 2024-09-17T07:25:40.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:39 smithi100 ceph-mon[29092]: 4.1 scrub starts 2024-09-17T07:25:40.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:39 smithi100 ceph-mon[29092]: 4.11 scrub starts 2024-09-17T07:25:40.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:39 smithi100 ceph-mon[29092]: 4.1 scrub ok 2024-09-17T07:25:40.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:39 smithi100 ceph-mon[29092]: 4.11 scrub ok 2024-09-17T07:25:40.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:39 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2862076674' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "6.18", "id": [0, 1, 4, 7, 2, 3]}]: dispatch 2024-09-17T07:25:40.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:39 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "6.18", "id": [0, 1, 4, 7, 2, 3]}]: dispatch 2024-09-17T07:25:40.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:39 smithi100 ceph-mon[39462]: 2.1f scrub starts 2024-09-17T07:25:40.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:39 smithi100 ceph-mon[39462]: 2.1f scrub ok 2024-09-17T07:25:40.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:39 smithi100 ceph-mon[39462]: 4.14 scrub starts 2024-09-17T07:25:40.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:39 smithi100 ceph-mon[39462]: 4.14 scrub ok 2024-09-17T07:25:40.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:39 smithi100 ceph-mon[39462]: 4.1 scrub starts 2024-09-17T07:25:40.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:39 smithi100 ceph-mon[39462]: 4.11 scrub starts 2024-09-17T07:25:40.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:39 smithi100 ceph-mon[39462]: 4.1 scrub ok 2024-09-17T07:25:40.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:39 smithi100 ceph-mon[39462]: 4.11 scrub ok 2024-09-17T07:25:40.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:39 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2862076674' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "6.18", "id": [0, 1, 4, 7, 2, 3]}]: dispatch 2024-09-17T07:25:40.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:39 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "6.18", "id": [0, 1, 4, 7, 2, 3]}]: dispatch 2024-09-17T07:25:40.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:39 smithi135 ceph-mon[33238]: 2.1f scrub starts 2024-09-17T07:25:40.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:39 smithi135 ceph-mon[33238]: 2.1f scrub ok 2024-09-17T07:25:40.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:39 smithi135 ceph-mon[33238]: 4.14 scrub starts 2024-09-17T07:25:40.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:39 smithi135 ceph-mon[33238]: 4.14 scrub ok 2024-09-17T07:25:40.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:39 smithi135 ceph-mon[33238]: 4.1 scrub starts 2024-09-17T07:25:40.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:39 smithi135 ceph-mon[33238]: 4.11 scrub starts 2024-09-17T07:25:40.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:39 smithi135 ceph-mon[33238]: 4.1 scrub ok 2024-09-17T07:25:40.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:39 smithi135 ceph-mon[33238]: 4.11 scrub ok 2024-09-17T07:25:40.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:39 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2862076674' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "6.18", "id": [0, 1, 4, 7, 2, 3]}]: dispatch 2024-09-17T07:25:40.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:39 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "6.18", "id": [0, 1, 4, 7, 2, 3]}]: dispatch 2024-09-17T07:25:40.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:40 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:25:40.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:40 smithi100 ceph-mon[29092]: pgmap v1124: 142 pgs: 142 active+clean; 792 KiB data, 6.4 GiB used, 619 GiB / 626 GiB avail 2024-09-17T07:25:40.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:40 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "6.18", "id": [0, 1, 4, 7, 2, 3]}]': finished 2024-09-17T07:25:40.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:40 smithi100 ceph-mon[29092]: osdmap e514: 8 total, 8 up, 7 in 2024-09-17T07:25:40.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:40 smithi100 ceph-mon[29092]: 2.9 scrub starts 2024-09-17T07:25:40.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:40 smithi100 ceph-mon[29092]: 2.9 scrub ok 2024-09-17T07:25:40.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:40 smithi100 ceph-mon[29092]: 5.10 scrub starts 2024-09-17T07:25:40.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:40 smithi100 ceph-mon[29092]: 5.10 scrub ok 2024-09-17T07:25:40.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:40 smithi100 ceph-mon[29092]: 1.5 scrub starts 2024-09-17T07:25:40.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:40 smithi100 ceph-mon[29092]: 1.5 scrub ok 2024-09-17T07:25:40.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:40 smithi100 ceph-mon[29092]: 1.1 scrub starts 2024-09-17T07:25:40.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:40 smithi100 ceph-mon[29092]: 1.1 scrub ok 2024-09-17T07:25:40.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:40 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:25:40.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:40 smithi100 ceph-mon[39462]: pgmap v1124: 142 pgs: 142 active+clean; 792 KiB data, 6.4 GiB used, 619 GiB / 626 GiB avail 2024-09-17T07:25:40.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:40 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "6.18", "id": [0, 1, 4, 7, 2, 3]}]': finished 2024-09-17T07:25:40.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:40 smithi100 ceph-mon[39462]: osdmap e514: 8 total, 8 up, 7 in 2024-09-17T07:25:40.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:40 smithi100 ceph-mon[39462]: 2.9 scrub starts 2024-09-17T07:25:40.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:40 smithi100 ceph-mon[39462]: 2.9 scrub ok 2024-09-17T07:25:40.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:40 smithi100 ceph-mon[39462]: 5.10 scrub starts 2024-09-17T07:25:40.879 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:40 smithi100 ceph-mon[39462]: 5.10 scrub ok 2024-09-17T07:25:40.879 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:40 smithi100 ceph-mon[39462]: 1.5 scrub starts 2024-09-17T07:25:40.879 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:40 smithi100 ceph-mon[39462]: 1.5 scrub ok 2024-09-17T07:25:40.879 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:40 smithi100 ceph-mon[39462]: 1.1 scrub starts 2024-09-17T07:25:40.879 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:40 smithi100 ceph-mon[39462]: 1.1 scrub ok 2024-09-17T07:25:41.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:40 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:25:41.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:40 smithi135 ceph-mon[33238]: pgmap v1124: 142 pgs: 142 active+clean; 792 KiB data, 6.4 GiB used, 619 GiB / 626 GiB avail 2024-09-17T07:25:41.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:40 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "6.18", "id": [0, 1, 4, 7, 2, 3]}]': finished 2024-09-17T07:25:41.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:40 smithi135 ceph-mon[33238]: osdmap e514: 8 total, 8 up, 7 in 2024-09-17T07:25:41.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:40 smithi135 ceph-mon[33238]: 2.9 scrub starts 2024-09-17T07:25:41.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:40 smithi135 ceph-mon[33238]: 2.9 scrub ok 2024-09-17T07:25:41.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:40 smithi135 ceph-mon[33238]: 5.10 scrub starts 2024-09-17T07:25:41.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:40 smithi135 ceph-mon[33238]: 5.10 scrub ok 2024-09-17T07:25:41.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:40 smithi135 ceph-mon[33238]: 1.5 scrub starts 2024-09-17T07:25:41.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:40 smithi135 ceph-mon[33238]: 1.5 scrub ok 2024-09-17T07:25:41.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:40 smithi135 ceph-mon[33238]: 1.1 scrub starts 2024-09-17T07:25:41.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:40 smithi135 ceph-mon[33238]: 1.1 scrub ok 2024-09-17T07:25:42.126 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:41 smithi100 ceph-mon[29092]: 5.19 scrub starts 2024-09-17T07:25:42.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:41 smithi100 ceph-mon[29092]: 5.19 scrub ok 2024-09-17T07:25:42.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:41 smithi100 ceph-mon[29092]: 4.1a scrub starts 2024-09-17T07:25:42.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:41 smithi100 ceph-mon[29092]: 4.1a scrub ok 2024-09-17T07:25:42.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:41 smithi100 ceph-mon[39462]: 5.19 scrub starts 2024-09-17T07:25:42.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:41 smithi100 ceph-mon[39462]: 5.19 scrub ok 2024-09-17T07:25:42.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:41 smithi100 ceph-mon[39462]: 4.1a scrub starts 2024-09-17T07:25:42.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:41 smithi100 ceph-mon[39462]: 4.1a scrub ok 2024-09-17T07:25:42.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:41 smithi135 ceph-mon[33238]: 5.19 scrub starts 2024-09-17T07:25:42.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:41 smithi135 ceph-mon[33238]: 5.19 scrub ok 2024-09-17T07:25:42.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:41 smithi135 ceph-mon[33238]: 4.1a scrub starts 2024-09-17T07:25:42.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:41 smithi135 ceph-mon[33238]: 4.1a scrub ok 2024-09-17T07:25:43.126 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:42 smithi100 ceph-mon[29092]: pgmap v1126: 142 pgs: 142 active+clean; 740 KiB data, 6.4 GiB used, 619 GiB / 626 GiB avail 2024-09-17T07:25:43.126 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:42 smithi100 ceph-mon[29092]: 4.f scrub starts 2024-09-17T07:25:43.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:42 smithi100 ceph-mon[29092]: 4.f scrub ok 2024-09-17T07:25:43.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:42 smithi100 ceph-mon[39462]: pgmap v1126: 142 pgs: 142 active+clean; 740 KiB data, 6.4 GiB used, 619 GiB / 626 GiB avail 2024-09-17T07:25:43.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:42 smithi100 ceph-mon[39462]: 4.f scrub starts 2024-09-17T07:25:43.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:42 smithi100 ceph-mon[39462]: 4.f scrub ok 2024-09-17T07:25:43.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:42 smithi135 ceph-mon[33238]: pgmap v1126: 142 pgs: 142 active+clean; 740 KiB data, 6.4 GiB used, 619 GiB / 626 GiB avail 2024-09-17T07:25:43.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:42 smithi135 ceph-mon[33238]: 4.f scrub starts 2024-09-17T07:25:43.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:42 smithi135 ceph-mon[33238]: 4.f scrub ok 2024-09-17T07:25:45.126 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:44 smithi100 ceph-mon[29092]: pgmap v1127: 142 pgs: 142 active+clean; 740 KiB data, 6.4 GiB used, 619 GiB / 626 GiB avail 2024-09-17T07:25:45.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:44 smithi100 ceph-mon[29092]: 4.13 scrub starts 2024-09-17T07:25:45.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:44 smithi100 ceph-mon[29092]: 4.13 scrub ok 2024-09-17T07:25:45.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:44 smithi100 ceph-mon[39462]: pgmap v1127: 142 pgs: 142 active+clean; 740 KiB data, 6.4 GiB used, 619 GiB / 626 GiB avail 2024-09-17T07:25:45.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:44 smithi100 ceph-mon[39462]: 4.13 scrub starts 2024-09-17T07:25:45.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:44 smithi100 ceph-mon[39462]: 4.13 scrub ok 2024-09-17T07:25:45.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:44 smithi135 ceph-mon[33238]: pgmap v1127: 142 pgs: 142 active+clean; 740 KiB data, 6.4 GiB used, 619 GiB / 626 GiB avail 2024-09-17T07:25:45.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:44 smithi135 ceph-mon[33238]: 4.13 scrub starts 2024-09-17T07:25:45.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:44 smithi135 ceph-mon[33238]: 4.13 scrub ok 2024-09-17T07:25:45.615 INFO:tasks.thrashosds.thrasher:inning osd 2024-09-17T07:25:45.615 INFO:tasks.thrashosds.thrasher:Adding osd 3 2024-09-17T07:25:45.616 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd in 3 2024-09-17T07:25:45.822 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:25:45.943 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:25:45 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:25:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:25:46.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:46 smithi100 ceph-mon[29092]: 2.2 scrub starts 2024-09-17T07:25:46.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:46 smithi100 ceph-mon[29092]: 2.2 scrub ok 2024-09-17T07:25:46.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:46 smithi100 ceph-mon[29092]: 6.10 scrub starts 2024-09-17T07:25:46.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:46 smithi100 ceph-mon[29092]: 6.10 scrub ok 2024-09-17T07:25:46.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:46 smithi100 ceph-mon[39462]: 2.2 scrub starts 2024-09-17T07:25:46.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:46 smithi100 ceph-mon[39462]: 2.2 scrub ok 2024-09-17T07:25:46.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:46 smithi100 ceph-mon[39462]: 6.10 scrub starts 2024-09-17T07:25:46.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:46 smithi100 ceph-mon[39462]: 6.10 scrub ok 2024-09-17T07:25:46.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:46 smithi135 ceph-mon[33238]: 2.2 scrub starts 2024-09-17T07:25:46.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:46 smithi135 ceph-mon[33238]: 2.2 scrub ok 2024-09-17T07:25:46.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:46 smithi135 ceph-mon[33238]: 6.10 scrub starts 2024-09-17T07:25:46.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:46 smithi135 ceph-mon[33238]: 6.10 scrub ok 2024-09-17T07:25:47.545 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:47 smithi100 ceph-mon[29092]: pgmap v1128: 142 pgs: 142 active+clean; 740 KiB data, 6.3 GiB used, 619 GiB / 626 GiB avail 2024-09-17T07:25:47.546 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:47 smithi100 ceph-mon[29092]: 6.14 scrub starts 2024-09-17T07:25:47.546 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:47 smithi100 ceph-mon[29092]: 6.14 scrub ok 2024-09-17T07:25:47.546 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:47 smithi100 ceph-mon[39462]: pgmap v1128: 142 pgs: 142 active+clean; 740 KiB data, 6.3 GiB used, 619 GiB / 626 GiB avail 2024-09-17T07:25:47.546 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:47 smithi100 ceph-mon[39462]: 6.14 scrub starts 2024-09-17T07:25:47.546 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:47 smithi100 ceph-mon[39462]: 6.14 scrub ok 2024-09-17T07:25:47.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:47 smithi135 ceph-mon[33238]: pgmap v1128: 142 pgs: 142 active+clean; 740 KiB data, 6.3 GiB used, 619 GiB / 626 GiB avail 2024-09-17T07:25:47.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:47 smithi135 ceph-mon[33238]: 6.14 scrub starts 2024-09-17T07:25:47.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:47 smithi135 ceph-mon[33238]: 6.14 scrub ok 2024-09-17T07:25:48.126 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:47 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mon-a[29088]: 2024-09-17T07:25:47.805+0000 7f59b37e7640 -1 mon.a@0(leader).osd e514 definitely_dead 0 2024-09-17T07:25:48.408 INFO:teuthology.orchestra.run.smithi100.stderr:marked in osd.3. 2024-09-17T07:25:48.775 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:48 smithi100 ceph-mon[29092]: 5.1 scrub starts 2024-09-17T07:25:48.775 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:48 smithi100 ceph-mon[29092]: 5.1 scrub ok 2024-09-17T07:25:48.775 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:48 smithi100 ceph-mon[29092]: 5.12 deep-scrub starts 2024-09-17T07:25:48.775 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:48 smithi100 ceph-mon[29092]: 5.12 deep-scrub ok 2024-09-17T07:25:48.775 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:48 smithi100 ceph-mon[29092]: 6.7 deep-scrub starts 2024-09-17T07:25:48.775 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:48 smithi100 ceph-mon[29092]: 6.7 deep-scrub ok 2024-09-17T07:25:48.775 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:48 smithi100 ceph-mon[29092]: 6.c scrub starts 2024-09-17T07:25:48.775 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:48 smithi100 ceph-mon[29092]: 6.c scrub ok 2024-09-17T07:25:48.775 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:48 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1345118068' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-09-17T07:25:48.776 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:48 smithi100 ceph-mon[39462]: 5.1 scrub starts 2024-09-17T07:25:48.776 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:48 smithi100 ceph-mon[39462]: 5.1 scrub ok 2024-09-17T07:25:48.776 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:48 smithi100 ceph-mon[39462]: 5.12 deep-scrub starts 2024-09-17T07:25:48.776 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:48 smithi100 ceph-mon[39462]: 5.12 deep-scrub ok 2024-09-17T07:25:48.776 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:48 smithi100 ceph-mon[39462]: 6.7 deep-scrub starts 2024-09-17T07:25:48.776 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:48 smithi100 ceph-mon[39462]: 6.7 deep-scrub ok 2024-09-17T07:25:48.776 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:48 smithi100 ceph-mon[39462]: 6.c scrub starts 2024-09-17T07:25:48.776 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:48 smithi100 ceph-mon[39462]: 6.c scrub ok 2024-09-17T07:25:48.776 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:48 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1345118068' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-09-17T07:25:48.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:48 smithi135 ceph-mon[33238]: 5.1 scrub starts 2024-09-17T07:25:48.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:48 smithi135 ceph-mon[33238]: 5.1 scrub ok 2024-09-17T07:25:48.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:48 smithi135 ceph-mon[33238]: 5.12 deep-scrub starts 2024-09-17T07:25:48.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:48 smithi135 ceph-mon[33238]: 5.12 deep-scrub ok 2024-09-17T07:25:48.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:48 smithi135 ceph-mon[33238]: 6.7 deep-scrub starts 2024-09-17T07:25:48.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:48 smithi135 ceph-mon[33238]: 6.7 deep-scrub ok 2024-09-17T07:25:48.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:48 smithi135 ceph-mon[33238]: 6.c scrub starts 2024-09-17T07:25:48.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:48 smithi135 ceph-mon[33238]: 6.c scrub ok 2024-09-17T07:25:48.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:48 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1345118068' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-09-17T07:25:49.320 INFO:tasks.thrashosds.thrasher:Added osd 3 2024-09-17T07:25:49.320 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-09-17T07:25:49.321 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd pool set .mgr pgp_num 11 2024-09-17T07:25:49.562 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:25:49.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:49 smithi100 ceph-mon[29092]: pgmap v1129: 142 pgs: 142 active+clean; 740 KiB data, 6.3 GiB used, 619 GiB / 626 GiB avail 2024-09-17T07:25:49.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:49 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1345118068' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-09-17T07:25:49.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:49 smithi100 ceph-mon[29092]: osdmap e515: 8 total, 8 up, 8 in 2024-09-17T07:25:49.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:49 smithi100 ceph-mon[29092]: 2.17 scrub starts 2024-09-17T07:25:49.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:49 smithi100 ceph-mon[29092]: 2.17 scrub ok 2024-09-17T07:25:49.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:49 smithi100 ceph-mon[29092]: 5.8 scrub starts 2024-09-17T07:25:49.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:49 smithi100 ceph-mon[29092]: 5.8 scrub ok 2024-09-17T07:25:49.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:49 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [7, 2]}]: dispatch 2024-09-17T07:25:49.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:49 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:25:49.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:49 smithi100 ceph-mon[39462]: pgmap v1129: 142 pgs: 142 active+clean; 740 KiB data, 6.3 GiB used, 619 GiB / 626 GiB avail 2024-09-17T07:25:49.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:49 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1345118068' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-09-17T07:25:49.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:49 smithi100 ceph-mon[39462]: osdmap e515: 8 total, 8 up, 8 in 2024-09-17T07:25:49.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:49 smithi100 ceph-mon[39462]: 2.17 scrub starts 2024-09-17T07:25:49.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:49 smithi100 ceph-mon[39462]: 2.17 scrub ok 2024-09-17T07:25:49.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:49 smithi100 ceph-mon[39462]: 5.8 scrub starts 2024-09-17T07:25:49.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:49 smithi100 ceph-mon[39462]: 5.8 scrub ok 2024-09-17T07:25:49.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:49 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [7, 2]}]: dispatch 2024-09-17T07:25:49.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:49 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:25:49.793 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:25:49 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:25:49.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:49 smithi135 ceph-mon[33238]: pgmap v1129: 142 pgs: 142 active+clean; 740 KiB data, 6.3 GiB used, 619 GiB / 626 GiB avail 2024-09-17T07:25:49.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:49 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1345118068' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-09-17T07:25:49.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:49 smithi135 ceph-mon[33238]: osdmap e515: 8 total, 8 up, 8 in 2024-09-17T07:25:49.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:49 smithi135 ceph-mon[33238]: 2.17 scrub starts 2024-09-17T07:25:49.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:49 smithi135 ceph-mon[33238]: 2.17 scrub ok 2024-09-17T07:25:49.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:49 smithi135 ceph-mon[33238]: 5.8 scrub starts 2024-09-17T07:25:49.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:49 smithi135 ceph-mon[33238]: 5.8 scrub ok 2024-09-17T07:25:49.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:49 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [7, 2]}]: dispatch 2024-09-17T07:25:49.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:49 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:25:50.738 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:50 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [7, 2]}]': finished 2024-09-17T07:25:50.739 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:50 smithi100 ceph-mon[29092]: osdmap e516: 8 total, 8 up, 8 in 2024-09-17T07:25:50.739 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:50 smithi100 ceph-mon[29092]: 4.16 scrub starts 2024-09-17T07:25:50.739 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:50 smithi100 ceph-mon[29092]: 4.16 scrub ok 2024-09-17T07:25:50.739 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:50 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [7, 2]}]': finished 2024-09-17T07:25:50.739 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:50 smithi100 ceph-mon[39462]: osdmap e516: 8 total, 8 up, 8 in 2024-09-17T07:25:50.739 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:50 smithi100 ceph-mon[39462]: 4.16 scrub starts 2024-09-17T07:25:50.739 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:50 smithi100 ceph-mon[39462]: 4.16 scrub ok 2024-09-17T07:25:50.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:50 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [7, 2]}]': finished 2024-09-17T07:25:50.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:50 smithi135 ceph-mon[33238]: osdmap e516: 8 total, 8 up, 8 in 2024-09-17T07:25:50.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:50 smithi135 ceph-mon[33238]: 4.16 scrub starts 2024-09-17T07:25:50.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:50 smithi135 ceph-mon[33238]: 4.16 scrub ok 2024-09-17T07:25:51.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:51 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:25:51.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:51 smithi100 ceph-mon[29092]: pgmap v1132: 142 pgs: 44 peering, 1 unknown, 97 active+clean; 420 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 895 B/s rd, 0 op/s 2024-09-17T07:25:51.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:51 smithi100 ceph-mon[29092]: 6.d scrub starts 2024-09-17T07:25:51.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:51 smithi100 ceph-mon[29092]: 6.d scrub ok 2024-09-17T07:25:51.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:51 smithi100 ceph-mon[29092]: 2.4 deep-scrub starts 2024-09-17T07:25:51.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:51 smithi100 ceph-mon[29092]: 2.4 deep-scrub ok 2024-09-17T07:25:51.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:51 smithi100 ceph-mon[29092]: Health check failed: Reduced data availability: 9 pgs inactive, 15 pgs peering (PG_AVAILABILITY) 2024-09-17T07:25:51.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:51 smithi100 ceph-mon[29092]: osdmap e517: 8 total, 8 up, 8 in 2024-09-17T07:25:51.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:51 smithi100 ceph-mon[29092]: 2.e scrub starts 2024-09-17T07:25:51.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:51 smithi100 ceph-mon[29092]: 2.e scrub ok 2024-09-17T07:25:51.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:51 smithi100 ceph-mon[29092]: 2.0 scrub starts 2024-09-17T07:25:51.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:51 smithi100 ceph-mon[29092]: 2.0 scrub ok 2024-09-17T07:25:51.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:51 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:25:51.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:51 smithi100 ceph-mon[39462]: pgmap v1132: 142 pgs: 44 peering, 1 unknown, 97 active+clean; 420 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 895 B/s rd, 0 op/s 2024-09-17T07:25:51.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:51 smithi100 ceph-mon[39462]: 6.d scrub starts 2024-09-17T07:25:51.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:51 smithi100 ceph-mon[39462]: 6.d scrub ok 2024-09-17T07:25:51.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:51 smithi100 ceph-mon[39462]: 2.4 deep-scrub starts 2024-09-17T07:25:51.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:51 smithi100 ceph-mon[39462]: 2.4 deep-scrub ok 2024-09-17T07:25:51.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:51 smithi100 ceph-mon[39462]: Health check failed: Reduced data availability: 9 pgs inactive, 15 pgs peering (PG_AVAILABILITY) 2024-09-17T07:25:51.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:51 smithi100 ceph-mon[39462]: osdmap e517: 8 total, 8 up, 8 in 2024-09-17T07:25:51.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:51 smithi100 ceph-mon[39462]: 2.e scrub starts 2024-09-17T07:25:51.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:51 smithi100 ceph-mon[39462]: 2.e scrub ok 2024-09-17T07:25:51.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:51 smithi100 ceph-mon[39462]: 2.0 scrub starts 2024-09-17T07:25:51.629 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:51 smithi100 ceph-mon[39462]: 2.0 scrub ok 2024-09-17T07:25:51.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:51 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:25:51.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:51 smithi135 ceph-mon[33238]: pgmap v1132: 142 pgs: 44 peering, 1 unknown, 97 active+clean; 420 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 895 B/s rd, 0 op/s 2024-09-17T07:25:51.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:51 smithi135 ceph-mon[33238]: 6.d scrub starts 2024-09-17T07:25:51.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:51 smithi135 ceph-mon[33238]: 6.d scrub ok 2024-09-17T07:25:51.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:51 smithi135 ceph-mon[33238]: 2.4 deep-scrub starts 2024-09-17T07:25:51.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:51 smithi135 ceph-mon[33238]: 2.4 deep-scrub ok 2024-09-17T07:25:51.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:51 smithi135 ceph-mon[33238]: Health check failed: Reduced data availability: 9 pgs inactive, 15 pgs peering (PG_AVAILABILITY) 2024-09-17T07:25:51.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:51 smithi135 ceph-mon[33238]: osdmap e517: 8 total, 8 up, 8 in 2024-09-17T07:25:51.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:51 smithi135 ceph-mon[33238]: 2.e scrub starts 2024-09-17T07:25:51.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:51 smithi135 ceph-mon[33238]: 2.e scrub ok 2024-09-17T07:25:51.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:51 smithi135 ceph-mon[33238]: 2.0 scrub starts 2024-09-17T07:25:51.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:51 smithi135 ceph-mon[33238]: 2.0 scrub ok 2024-09-17T07:25:52.441 INFO:teuthology.orchestra.run.smithi100.stderr:set pool 1 pgp_num to 11 2024-09-17T07:25:52.453 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:52 smithi100 ceph-mon[29092]: osdmap e518: 8 total, 8 up, 8 in 2024-09-17T07:25:52.453 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:52 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/285393716' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-09-17T07:25:52.453 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:52 smithi100 ceph-mon[29092]: 1.2 scrub starts 2024-09-17T07:25:52.453 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:52 smithi100 ceph-mon[29092]: 1.2 scrub ok 2024-09-17T07:25:52.454 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:52 smithi100 ceph-mon[39462]: osdmap e518: 8 total, 8 up, 8 in 2024-09-17T07:25:52.454 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:52 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/285393716' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-09-17T07:25:52.454 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:52 smithi100 ceph-mon[39462]: 1.2 scrub starts 2024-09-17T07:25:52.454 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:52 smithi100 ceph-mon[39462]: 1.2 scrub ok 2024-09-17T07:25:52.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:52 smithi135 ceph-mon[33238]: osdmap e518: 8 total, 8 up, 8 in 2024-09-17T07:25:52.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:52 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/285393716' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-09-17T07:25:52.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:52 smithi135 ceph-mon[33238]: 1.2 scrub starts 2024-09-17T07:25:52.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:52 smithi135 ceph-mon[33238]: 1.2 scrub ok 2024-09-17T07:25:53.227 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-09-17T07:25:53.430 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:25:53.551 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:53 smithi100 ceph-mon[39462]: pgmap v1135: 142 pgs: 2 activating+undersized+degraded+remapped, 44 peering, 96 active+clean; 740 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 2/591 objects degraded (0.338%); 0 B/s, 3 objects/s recovering 2024-09-17T07:25:53.552 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:53 smithi100 ceph-mon[39462]: 5.16 scrub starts 2024-09-17T07:25:53.552 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:53 smithi100 ceph-mon[39462]: 5.16 scrub ok 2024-09-17T07:25:53.552 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:53 smithi100 ceph-mon[39462]: Health check failed: Degraded data redundancy: 2/591 objects degraded (0.338%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T07:25:53.552 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:53 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/285393716' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-09-17T07:25:53.552 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:53 smithi100 ceph-mon[39462]: osdmap e519: 8 total, 8 up, 8 in 2024-09-17T07:25:53.552 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:53 smithi100 ceph-mon[29092]: pgmap v1135: 142 pgs: 2 activating+undersized+degraded+remapped, 44 peering, 96 active+clean; 740 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 2/591 objects degraded (0.338%); 0 B/s, 3 objects/s recovering 2024-09-17T07:25:53.552 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:53 smithi100 ceph-mon[29092]: 5.16 scrub starts 2024-09-17T07:25:53.552 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:53 smithi100 ceph-mon[29092]: 5.16 scrub ok 2024-09-17T07:25:53.553 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:53 smithi100 ceph-mon[29092]: Health check failed: Degraded data redundancy: 2/591 objects degraded (0.338%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T07:25:53.553 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:53 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/285393716' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-09-17T07:25:53.553 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:53 smithi100 ceph-mon[29092]: osdmap e519: 8 total, 8 up, 8 in 2024-09-17T07:25:53.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:53 smithi135 ceph-mon[33238]: pgmap v1135: 142 pgs: 2 activating+undersized+degraded+remapped, 44 peering, 96 active+clean; 740 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 2/591 objects degraded (0.338%); 0 B/s, 3 objects/s recovering 2024-09-17T07:25:53.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:53 smithi135 ceph-mon[33238]: 5.16 scrub starts 2024-09-17T07:25:53.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:53 smithi135 ceph-mon[33238]: 5.16 scrub ok 2024-09-17T07:25:53.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:53 smithi135 ceph-mon[33238]: Health check failed: Degraded data redundancy: 2/591 objects degraded (0.338%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T07:25:53.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:53 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/285393716' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-09-17T07:25:53.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:53 smithi135 ceph-mon[33238]: osdmap e519: 8 total, 8 up, 8 in 2024-09-17T07:25:54.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:54 smithi100 ceph-mon[29092]: 1.0 scrub starts 2024-09-17T07:25:54.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:54 smithi100 ceph-mon[29092]: 1.0 scrub ok 2024-09-17T07:25:54.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:54 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:25:54.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:54 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:25:54.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:54 smithi100 ceph-mon[39462]: 1.0 scrub starts 2024-09-17T07:25:54.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:54 smithi100 ceph-mon[39462]: 1.0 scrub ok 2024-09-17T07:25:54.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:54 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:25:54.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:54 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:25:54.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:54 smithi135 ceph-mon[33238]: 1.0 scrub starts 2024-09-17T07:25:54.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:54 smithi135 ceph-mon[33238]: 1.0 scrub ok 2024-09-17T07:25:54.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:54 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:25:54.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:54 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:25:55.245 INFO:teuthology.orchestra.run.smithi100.stdout:mon.a: {} 2024-09-17T07:25:55.246 INFO:teuthology.orchestra.run.smithi100.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-17T07:25:55.255 INFO:teuthology.orchestra.run.smithi100.stdout:mon.b: {} 2024-09-17T07:25:55.256 INFO:teuthology.orchestra.run.smithi100.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-17T07:25:55.263 INFO:teuthology.orchestra.run.smithi100.stdout:mon.c: {} 2024-09-17T07:25:55.264 INFO:teuthology.orchestra.run.smithi100.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-17T07:25:55.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:55 smithi100 ceph-mon[39462]: pgmap v1137: 142 pgs: 2 activating+undersized+degraded+remapped, 44 peering, 96 active+clean; 687 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 2/588 objects degraded (0.340%); 0 B/s, 3 objects/s recovering 2024-09-17T07:25:55.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:55 smithi100 ceph-mon[39462]: osdmap e520: 8 total, 8 up, 8 in 2024-09-17T07:25:55.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:55 smithi100 ceph-mon[39462]: 2.1d scrub starts 2024-09-17T07:25:55.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:55 smithi100 ceph-mon[39462]: 2.1d scrub ok 2024-09-17T07:25:55.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:55 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1139159402' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-17T07:25:55.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:55 smithi100 ceph-mon[29092]: pgmap v1137: 142 pgs: 2 activating+undersized+degraded+remapped, 44 peering, 96 active+clean; 687 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 2/588 objects degraded (0.340%); 0 B/s, 3 objects/s recovering 2024-09-17T07:25:55.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:55 smithi100 ceph-mon[29092]: osdmap e520: 8 total, 8 up, 8 in 2024-09-17T07:25:55.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:55 smithi100 ceph-mon[29092]: 2.1d scrub starts 2024-09-17T07:25:55.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:55 smithi100 ceph-mon[29092]: 2.1d scrub ok 2024-09-17T07:25:55.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:55 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1139159402' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-17T07:25:55.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:55 smithi135 ceph-mon[33238]: pgmap v1137: 142 pgs: 2 activating+undersized+degraded+remapped, 44 peering, 96 active+clean; 687 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 2/588 objects degraded (0.340%); 0 B/s, 3 objects/s recovering 2024-09-17T07:25:55.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:55 smithi135 ceph-mon[33238]: osdmap e520: 8 total, 8 up, 8 in 2024-09-17T07:25:55.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:55 smithi135 ceph-mon[33238]: 2.1d scrub starts 2024-09-17T07:25:55.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:55 smithi135 ceph-mon[33238]: 2.1d scrub ok 2024-09-17T07:25:55.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:55 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1139159402' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-17T07:25:56.081 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd reweight 0 1 2024-09-17T07:25:56.279 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:25:56.377 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:25:55 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:25:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:25:56.681 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:56 smithi100 ceph-mon[29092]: 6.e scrub starts 2024-09-17T07:25:56.681 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:56 smithi100 ceph-mon[29092]: 6.e scrub ok 2024-09-17T07:25:56.681 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:56 smithi100 ceph-mon[29092]: 4.1c scrub starts 2024-09-17T07:25:56.682 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:56 smithi100 ceph-mon[29092]: 4.1c scrub ok 2024-09-17T07:25:56.682 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:56 smithi100 ceph-mon[29092]: 6.5 scrub starts 2024-09-17T07:25:56.682 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:56 smithi100 ceph-mon[29092]: 6.5 scrub ok 2024-09-17T07:25:56.682 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:56 smithi100 ceph-mon[29092]: 5.17 scrub starts 2024-09-17T07:25:56.682 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:56 smithi100 ceph-mon[29092]: 5.17 scrub ok 2024-09-17T07:25:56.682 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:56 smithi100 ceph-mon[39462]: 6.e scrub starts 2024-09-17T07:25:56.682 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:56 smithi100 ceph-mon[39462]: 6.e scrub ok 2024-09-17T07:25:56.682 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:56 smithi100 ceph-mon[39462]: 4.1c scrub starts 2024-09-17T07:25:56.682 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:56 smithi100 ceph-mon[39462]: 4.1c scrub ok 2024-09-17T07:25:56.682 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:56 smithi100 ceph-mon[39462]: 6.5 scrub starts 2024-09-17T07:25:56.683 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:56 smithi100 ceph-mon[39462]: 6.5 scrub ok 2024-09-17T07:25:56.683 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:56 smithi100 ceph-mon[39462]: 5.17 scrub starts 2024-09-17T07:25:56.683 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:56 smithi100 ceph-mon[39462]: 5.17 scrub ok 2024-09-17T07:25:56.796 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:56 smithi135 ceph-mon[33238]: 6.e scrub starts 2024-09-17T07:25:56.796 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:56 smithi135 ceph-mon[33238]: 6.e scrub ok 2024-09-17T07:25:56.796 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:56 smithi135 ceph-mon[33238]: 4.1c scrub starts 2024-09-17T07:25:56.796 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:56 smithi135 ceph-mon[33238]: 4.1c scrub ok 2024-09-17T07:25:56.796 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:56 smithi135 ceph-mon[33238]: 6.5 scrub starts 2024-09-17T07:25:56.797 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:56 smithi135 ceph-mon[33238]: 6.5 scrub ok 2024-09-17T07:25:56.797 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:56 smithi135 ceph-mon[33238]: 5.17 scrub starts 2024-09-17T07:25:56.797 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:56 smithi135 ceph-mon[33238]: 5.17 scrub ok 2024-09-17T07:25:57.707 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:57 smithi100 ceph-mon[29092]: pgmap v1139: 142 pgs: 1 activating+undersized+degraded+remapped, 141 active+clean; 687 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1/588 objects degraded (0.170%); 32 KiB/s, 16 objects/s recovering 2024-09-17T07:25:57.707 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:57 smithi100 ceph-mon[29092]: 6.1a scrub starts 2024-09-17T07:25:57.708 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:57 smithi100 ceph-mon[29092]: 6.1a scrub ok 2024-09-17T07:25:57.708 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:57 smithi100 ceph-mon[29092]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 9 pgs inactive, 15 pgs peering) 2024-09-17T07:25:57.708 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:57 smithi100 ceph-mon[29092]: 2.1b scrub starts 2024-09-17T07:25:57.708 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:57 smithi100 ceph-mon[29092]: 2.1b scrub ok 2024-09-17T07:25:57.708 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:57 smithi100 ceph-mon[29092]: 5.4 scrub starts 2024-09-17T07:25:57.708 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:57 smithi100 ceph-mon[29092]: 5.4 scrub ok 2024-09-17T07:25:57.708 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:57 smithi100 ceph-mon[39462]: pgmap v1139: 142 pgs: 1 activating+undersized+degraded+remapped, 141 active+clean; 687 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1/588 objects degraded (0.170%); 32 KiB/s, 16 objects/s recovering 2024-09-17T07:25:57.709 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:57 smithi100 ceph-mon[39462]: 6.1a scrub starts 2024-09-17T07:25:57.709 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:57 smithi100 ceph-mon[39462]: 6.1a scrub ok 2024-09-17T07:25:57.709 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:57 smithi100 ceph-mon[39462]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 9 pgs inactive, 15 pgs peering) 2024-09-17T07:25:57.709 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:57 smithi100 ceph-mon[39462]: 2.1b scrub starts 2024-09-17T07:25:57.709 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:57 smithi100 ceph-mon[39462]: 2.1b scrub ok 2024-09-17T07:25:57.709 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:57 smithi100 ceph-mon[39462]: 5.4 scrub starts 2024-09-17T07:25:57.709 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:57 smithi100 ceph-mon[39462]: 5.4 scrub ok 2024-09-17T07:25:57.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:57 smithi135 ceph-mon[33238]: pgmap v1139: 142 pgs: 1 activating+undersized+degraded+remapped, 141 active+clean; 687 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1/588 objects degraded (0.170%); 32 KiB/s, 16 objects/s recovering 2024-09-17T07:25:57.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:57 smithi135 ceph-mon[33238]: 6.1a scrub starts 2024-09-17T07:25:57.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:57 smithi135 ceph-mon[33238]: 6.1a scrub ok 2024-09-17T07:25:57.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:57 smithi135 ceph-mon[33238]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 9 pgs inactive, 15 pgs peering) 2024-09-17T07:25:57.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:57 smithi135 ceph-mon[33238]: 2.1b scrub starts 2024-09-17T07:25:57.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:57 smithi135 ceph-mon[33238]: 2.1b scrub ok 2024-09-17T07:25:57.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:57 smithi135 ceph-mon[33238]: 5.4 scrub starts 2024-09-17T07:25:57.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:57 smithi135 ceph-mon[33238]: 5.4 scrub ok 2024-09-17T07:25:58.458 INFO:teuthology.orchestra.run.smithi100.stderr:reweighted osd.0 to 1 (10000) 2024-09-17T07:25:58.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:58 smithi135 ceph-mon[33238]: 4.7 deep-scrub starts 2024-09-17T07:25:58.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:58 smithi135 ceph-mon[33238]: 4.7 deep-scrub ok 2024-09-17T07:25:58.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:58 smithi135 ceph-mon[33238]: 2.19 scrub starts 2024-09-17T07:25:58.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:58 smithi135 ceph-mon[33238]: 2.19 scrub ok 2024-09-17T07:25:58.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:58 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/3361770159' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-17T07:25:58.817 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:58 smithi100 ceph-mon[29092]: 4.7 deep-scrub starts 2024-09-17T07:25:58.818 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:58 smithi100 ceph-mon[29092]: 4.7 deep-scrub ok 2024-09-17T07:25:58.818 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:58 smithi100 ceph-mon[29092]: 2.19 scrub starts 2024-09-17T07:25:58.818 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:58 smithi100 ceph-mon[29092]: 2.19 scrub ok 2024-09-17T07:25:58.818 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:58 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/3361770159' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-17T07:25:58.818 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:58 smithi100 ceph-mon[39462]: 4.7 deep-scrub starts 2024-09-17T07:25:58.818 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:58 smithi100 ceph-mon[39462]: 4.7 deep-scrub ok 2024-09-17T07:25:58.818 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:58 smithi100 ceph-mon[39462]: 2.19 scrub starts 2024-09-17T07:25:58.818 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:58 smithi100 ceph-mon[39462]: 2.19 scrub ok 2024-09-17T07:25:58.818 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:58 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/3361770159' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-17T07:25:59.124 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 0 1 2024-09-17T07:25:59.322 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:25:59.705 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:59 smithi100 ceph-mon[29092]: pgmap v1140: 142 pgs: 142 active+clean; 687 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 69 KiB/s, 13 objects/s recovering 2024-09-17T07:25:59.705 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:59 smithi100 ceph-mon[29092]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/588 objects degraded (0.170%), 1 pg degraded) 2024-09-17T07:25:59.705 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:59 smithi100 ceph-mon[29092]: Cluster is now healthy 2024-09-17T07:25:59.705 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:59 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/3361770159' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-17T07:25:59.706 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:59 smithi100 ceph-mon[29092]: osdmap e521: 8 total, 8 up, 8 in 2024-09-17T07:25:59.706 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:59 smithi100 ceph-mon[29092]: 2.1c scrub starts 2024-09-17T07:25:59.706 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:25:59 smithi100 ceph-mon[29092]: 2.1c scrub ok 2024-09-17T07:25:59.706 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:59 smithi100 ceph-mon[39462]: pgmap v1140: 142 pgs: 142 active+clean; 687 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 69 KiB/s, 13 objects/s recovering 2024-09-17T07:25:59.706 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:59 smithi100 ceph-mon[39462]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/588 objects degraded (0.170%), 1 pg degraded) 2024-09-17T07:25:59.706 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:59 smithi100 ceph-mon[39462]: Cluster is now healthy 2024-09-17T07:25:59.706 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:59 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/3361770159' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-17T07:25:59.706 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:59 smithi100 ceph-mon[39462]: osdmap e521: 8 total, 8 up, 8 in 2024-09-17T07:25:59.706 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:59 smithi100 ceph-mon[39462]: 2.1c scrub starts 2024-09-17T07:25:59.706 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:25:59 smithi100 ceph-mon[39462]: 2.1c scrub ok 2024-09-17T07:25:59.793 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:25:59 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:25:59.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:59 smithi135 ceph-mon[33238]: pgmap v1140: 142 pgs: 142 active+clean; 687 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 69 KiB/s, 13 objects/s recovering 2024-09-17T07:25:59.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:59 smithi135 ceph-mon[33238]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/588 objects degraded (0.170%), 1 pg degraded) 2024-09-17T07:25:59.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:59 smithi135 ceph-mon[33238]: Cluster is now healthy 2024-09-17T07:25:59.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:59 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/3361770159' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-17T07:25:59.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:59 smithi135 ceph-mon[33238]: osdmap e521: 8 total, 8 up, 8 in 2024-09-17T07:25:59.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:59 smithi135 ceph-mon[33238]: 2.1c scrub starts 2024-09-17T07:25:59.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:25:59 smithi135 ceph-mon[33238]: 2.1c scrub ok 2024-09-17T07:26:00.759 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:00 smithi100 ceph-mon[39462]: 4.e scrub starts 2024-09-17T07:26:00.760 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:00 smithi100 ceph-mon[39462]: 4.e scrub ok 2024-09-17T07:26:00.760 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:00 smithi100 ceph-mon[39462]: 1.9 scrub starts 2024-09-17T07:26:00.760 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:00 smithi100 ceph-mon[39462]: 1.9 scrub ok 2024-09-17T07:26:00.760 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:00 smithi100 ceph-mon[29092]: 4.e scrub starts 2024-09-17T07:26:00.760 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:00 smithi100 ceph-mon[29092]: 4.e scrub ok 2024-09-17T07:26:00.760 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:00 smithi100 ceph-mon[29092]: 1.9 scrub starts 2024-09-17T07:26:00.760 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:00 smithi100 ceph-mon[29092]: 1.9 scrub ok 2024-09-17T07:26:00.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:00 smithi135 ceph-mon[33238]: 4.e scrub starts 2024-09-17T07:26:00.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:00 smithi135 ceph-mon[33238]: 4.e scrub ok 2024-09-17T07:26:00.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:00 smithi135 ceph-mon[33238]: 1.9 scrub starts 2024-09-17T07:26:00.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:00 smithi135 ceph-mon[33238]: 1.9 scrub ok 2024-09-17T07:26:01.475 INFO:teuthology.orchestra.run.smithi100.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-09-17T07:26:01.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:01 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:26:01.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:01 smithi135 ceph-mon[33238]: pgmap v1142: 142 pgs: 142 active+clean; 635 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 37 KiB/s, 11 objects/s recovering 2024-09-17T07:26:01.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:01 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:26:01.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:01 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:26:01.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:01 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/3507357303' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-09-17T07:26:01.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:01 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-09-17T07:26:01.859 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:01 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:26:01.860 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:01 smithi100 ceph-mon[29092]: pgmap v1142: 142 pgs: 142 active+clean; 635 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 37 KiB/s, 11 objects/s recovering 2024-09-17T07:26:01.860 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:01 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:26:01.860 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:01 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:26:01.860 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:01 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/3507357303' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-09-17T07:26:01.860 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:01 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-09-17T07:26:01.860 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:01 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:26:01.860 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:01 smithi100 ceph-mon[39462]: pgmap v1142: 142 pgs: 142 active+clean; 635 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 37 KiB/s, 11 objects/s recovering 2024-09-17T07:26:01.860 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:01 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:26:01.860 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:01 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:26:01.860 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:01 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/3507357303' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-09-17T07:26:01.861 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:01 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-09-17T07:26:02.157 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd reweight 2 1 2024-09-17T07:26:02.353 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:26:02.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:02 smithi135 ceph-mon[33238]: 6.19 scrub starts 2024-09-17T07:26:02.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:02 smithi135 ceph-mon[33238]: 6.19 scrub ok 2024-09-17T07:26:02.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:02 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-17T07:26:02.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:02 smithi135 ceph-mon[33238]: osdmap e522: 8 total, 8 up, 8 in 2024-09-17T07:26:02.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:02 smithi135 ceph-mon[33238]: 6.8 scrub starts 2024-09-17T07:26:02.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:02 smithi135 ceph-mon[33238]: 6.8 scrub ok 2024-09-17T07:26:02.830 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:02 smithi100 ceph-mon[29092]: 6.19 scrub starts 2024-09-17T07:26:02.830 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:02 smithi100 ceph-mon[29092]: 6.19 scrub ok 2024-09-17T07:26:02.830 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:02 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-17T07:26:02.830 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:02 smithi100 ceph-mon[29092]: osdmap e522: 8 total, 8 up, 8 in 2024-09-17T07:26:02.830 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:02 smithi100 ceph-mon[29092]: 6.8 scrub starts 2024-09-17T07:26:02.831 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:02 smithi100 ceph-mon[29092]: 6.8 scrub ok 2024-09-17T07:26:02.831 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:02 smithi100 ceph-mon[39462]: 6.19 scrub starts 2024-09-17T07:26:02.831 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:02 smithi100 ceph-mon[39462]: 6.19 scrub ok 2024-09-17T07:26:02.831 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:02 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-17T07:26:02.831 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:02 smithi100 ceph-mon[39462]: osdmap e522: 8 total, 8 up, 8 in 2024-09-17T07:26:02.831 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:02 smithi100 ceph-mon[39462]: 6.8 scrub starts 2024-09-17T07:26:02.831 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:02 smithi100 ceph-mon[39462]: 6.8 scrub ok 2024-09-17T07:26:03.595 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:03 smithi100 ceph-mon[29092]: pgmap v1144: 142 pgs: 142 active+clean; 635 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 14 KiB/s, 0 objects/s recovering 2024-09-17T07:26:03.595 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:03 smithi100 ceph-mon[29092]: 4.1b deep-scrub starts 2024-09-17T07:26:03.595 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:03 smithi100 ceph-mon[29092]: 4.1b deep-scrub ok 2024-09-17T07:26:03.595 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:03 smithi100 ceph-mon[29092]: 5.15 scrub starts 2024-09-17T07:26:03.595 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:03 smithi100 ceph-mon[29092]: 5.15 scrub ok 2024-09-17T07:26:03.596 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:03 smithi100 ceph-mon[39462]: pgmap v1144: 142 pgs: 142 active+clean; 635 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 14 KiB/s, 0 objects/s recovering 2024-09-17T07:26:03.596 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:03 smithi100 ceph-mon[39462]: 4.1b deep-scrub starts 2024-09-17T07:26:03.596 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:03 smithi100 ceph-mon[39462]: 4.1b deep-scrub ok 2024-09-17T07:26:03.596 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:03 smithi100 ceph-mon[39462]: 5.15 scrub starts 2024-09-17T07:26:03.596 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:03 smithi100 ceph-mon[39462]: 5.15 scrub ok 2024-09-17T07:26:03.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:03 smithi135 ceph-mon[33238]: pgmap v1144: 142 pgs: 142 active+clean; 635 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 14 KiB/s, 0 objects/s recovering 2024-09-17T07:26:03.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:03 smithi135 ceph-mon[33238]: 4.1b deep-scrub starts 2024-09-17T07:26:03.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:03 smithi135 ceph-mon[33238]: 4.1b deep-scrub ok 2024-09-17T07:26:03.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:03 smithi135 ceph-mon[33238]: 5.15 scrub starts 2024-09-17T07:26:03.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:03 smithi135 ceph-mon[33238]: 5.15 scrub ok 2024-09-17T07:26:04.482 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:04 smithi100 ceph-mon[29092]: 4.c scrub starts 2024-09-17T07:26:04.482 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:04 smithi100 ceph-mon[29092]: 4.c scrub ok 2024-09-17T07:26:04.482 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:04 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:26:04.483 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:04 smithi100 ceph-mon[29092]: 5.d scrub starts 2024-09-17T07:26:04.483 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:04 smithi100 ceph-mon[29092]: 5.d scrub ok 2024-09-17T07:26:04.483 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:04 smithi100 ceph-mon[39462]: 4.c scrub starts 2024-09-17T07:26:04.483 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:04 smithi100 ceph-mon[39462]: 4.c scrub ok 2024-09-17T07:26:04.483 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:04 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:26:04.483 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:04 smithi100 ceph-mon[39462]: 5.d scrub starts 2024-09-17T07:26:04.483 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:04 smithi100 ceph-mon[39462]: 5.d scrub ok 2024-09-17T07:26:04.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:04 smithi135 ceph-mon[33238]: 4.c scrub starts 2024-09-17T07:26:04.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:04 smithi135 ceph-mon[33238]: 4.c scrub ok 2024-09-17T07:26:04.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:04 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:26:04.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:04 smithi135 ceph-mon[33238]: 5.d scrub starts 2024-09-17T07:26:04.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:04 smithi135 ceph-mon[33238]: 5.d scrub ok 2024-09-17T07:26:05.487 INFO:teuthology.orchestra.run.smithi100.stderr:reweighted osd.2 to 1 (10000) 2024-09-17T07:26:05.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:05 smithi100 ceph-mon[29092]: pgmap v1145: 142 pgs: 142 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-17T07:26:05.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:05 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2657486635' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-17T07:26:05.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:05 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-17T07:26:05.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:05 smithi100 ceph-mon[39462]: pgmap v1145: 142 pgs: 142 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-17T07:26:05.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:05 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2657486635' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-17T07:26:05.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:05 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-17T07:26:05.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:05 smithi135 ceph-mon[33238]: pgmap v1145: 142 pgs: 142 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-17T07:26:05.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:05 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2657486635' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-17T07:26:05.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:05 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-17T07:26:06.126 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:26:05 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:26:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:26:06.302 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 2 1 2024-09-17T07:26:06.501 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:26:06.622 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:06 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-17T07:26:06.622 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:06 smithi100 ceph-mon[29092]: osdmap e523: 8 total, 8 up, 8 in 2024-09-17T07:26:06.623 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:06 smithi100 ceph-mon[29092]: pgmap v1147: 142 pgs: 142 active+clean; 582 KiB data, 6.5 GiB used, 709 GiB / 715 GiB avail 2024-09-17T07:26:06.623 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:06 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-17T07:26:06.623 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:06 smithi100 ceph-mon[39462]: osdmap e523: 8 total, 8 up, 8 in 2024-09-17T07:26:06.623 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:06 smithi100 ceph-mon[39462]: pgmap v1147: 142 pgs: 142 active+clean; 582 KiB data, 6.5 GiB used, 709 GiB / 715 GiB avail 2024-09-17T07:26:06.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:06 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-17T07:26:06.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:06 smithi135 ceph-mon[33238]: osdmap e523: 8 total, 8 up, 8 in 2024-09-17T07:26:06.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:06 smithi135 ceph-mon[33238]: pgmap v1147: 142 pgs: 142 active+clean; 582 KiB data, 6.5 GiB used, 709 GiB / 715 GiB avail 2024-09-17T07:26:08.592 INFO:teuthology.orchestra.run.smithi100.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-09-17T07:26:08.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:08 smithi100 ceph-mon[29092]: 4.12 scrub starts 2024-09-17T07:26:08.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:08 smithi100 ceph-mon[29092]: 4.12 scrub ok 2024-09-17T07:26:08.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:08 smithi100 ceph-mon[29092]: pgmap v1148: 142 pgs: 142 active+clean; 582 KiB data, 6.5 GiB used, 709 GiB / 715 GiB avail 2024-09-17T07:26:08.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:08 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:26:08.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:08 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:26:08.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:08 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/968864917' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-09-17T07:26:08.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:08 smithi100 ceph-mon[39462]: 4.12 scrub starts 2024-09-17T07:26:08.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:08 smithi100 ceph-mon[39462]: 4.12 scrub ok 2024-09-17T07:26:08.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:08 smithi100 ceph-mon[39462]: pgmap v1148: 142 pgs: 142 active+clean; 582 KiB data, 6.5 GiB used, 709 GiB / 715 GiB avail 2024-09-17T07:26:08.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:08 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:26:08.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:08 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:26:08.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:08 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/968864917' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-09-17T07:26:09.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:08 smithi135 ceph-mon[33238]: 4.12 scrub starts 2024-09-17T07:26:09.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:08 smithi135 ceph-mon[33238]: 4.12 scrub ok 2024-09-17T07:26:09.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:08 smithi135 ceph-mon[33238]: pgmap v1148: 142 pgs: 142 active+clean; 582 KiB data, 6.5 GiB used, 709 GiB / 715 GiB avail 2024-09-17T07:26:09.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:08 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:26:09.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:08 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:26:09.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:08 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/968864917' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-09-17T07:26:09.394 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd reweight 3 1 2024-09-17T07:26:09.598 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:26:09.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:09 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/968864917' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-17T07:26:09.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:09 smithi100 ceph-mon[39462]: osdmap e524: 8 total, 8 up, 8 in 2024-09-17T07:26:09.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:09 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/968864917' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-17T07:26:09.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:09 smithi100 ceph-mon[29092]: osdmap e524: 8 total, 8 up, 8 in 2024-09-17T07:26:09.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:09 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/968864917' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-17T07:26:09.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:09 smithi135 ceph-mon[33238]: osdmap e524: 8 total, 8 up, 8 in 2024-09-17T07:26:09.793 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:26:09 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:26:10.689 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:10 smithi100 ceph-mon[29092]: 6.1 deep-scrub starts 2024-09-17T07:26:10.689 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:10 smithi100 ceph-mon[29092]: 6.1 deep-scrub ok 2024-09-17T07:26:10.689 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:10 smithi100 ceph-mon[29092]: pgmap v1150: 142 pgs: 142 active+clean; 582 KiB data, 6.5 GiB used, 709 GiB / 715 GiB avail 2024-09-17T07:26:10.689 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:10 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:26:10.689 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:10 smithi100 ceph-mon[29092]: osdmap e525: 8 total, 8 up, 8 in 2024-09-17T07:26:10.689 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:10 smithi100 ceph-mon[29092]: 4.1d scrub starts 2024-09-17T07:26:10.689 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:10 smithi100 ceph-mon[29092]: 4.1d scrub ok 2024-09-17T07:26:10.689 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:10 smithi100 ceph-mon[29092]: 5.1f deep-scrub starts 2024-09-17T07:26:10.689 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:10 smithi100 ceph-mon[29092]: 5.1f deep-scrub ok 2024-09-17T07:26:10.690 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:10 smithi100 ceph-mon[39462]: 6.1 deep-scrub starts 2024-09-17T07:26:10.690 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:10 smithi100 ceph-mon[39462]: 6.1 deep-scrub ok 2024-09-17T07:26:10.690 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:10 smithi100 ceph-mon[39462]: pgmap v1150: 142 pgs: 142 active+clean; 582 KiB data, 6.5 GiB used, 709 GiB / 715 GiB avail 2024-09-17T07:26:10.690 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:10 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:26:10.690 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:10 smithi100 ceph-mon[39462]: osdmap e525: 8 total, 8 up, 8 in 2024-09-17T07:26:10.690 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:10 smithi100 ceph-mon[39462]: 4.1d scrub starts 2024-09-17T07:26:10.690 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:10 smithi100 ceph-mon[39462]: 4.1d scrub ok 2024-09-17T07:26:10.690 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:10 smithi100 ceph-mon[39462]: 5.1f deep-scrub starts 2024-09-17T07:26:10.690 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:10 smithi100 ceph-mon[39462]: 5.1f deep-scrub ok 2024-09-17T07:26:11.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:10 smithi135 ceph-mon[33238]: 6.1 deep-scrub starts 2024-09-17T07:26:11.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:10 smithi135 ceph-mon[33238]: 6.1 deep-scrub ok 2024-09-17T07:26:11.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:10 smithi135 ceph-mon[33238]: pgmap v1150: 142 pgs: 142 active+clean; 582 KiB data, 6.5 GiB used, 709 GiB / 715 GiB avail 2024-09-17T07:26:11.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:10 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:26:11.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:10 smithi135 ceph-mon[33238]: osdmap e525: 8 total, 8 up, 8 in 2024-09-17T07:26:11.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:10 smithi135 ceph-mon[33238]: 4.1d scrub starts 2024-09-17T07:26:11.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:10 smithi135 ceph-mon[33238]: 4.1d scrub ok 2024-09-17T07:26:11.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:10 smithi135 ceph-mon[33238]: 5.1f deep-scrub starts 2024-09-17T07:26:11.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:10 smithi135 ceph-mon[33238]: 5.1f deep-scrub ok 2024-09-17T07:26:11.605 INFO:teuthology.orchestra.run.smithi100.stderr:reweighted osd.3 to 1 (10000) 2024-09-17T07:26:11.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:11 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1610150527' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-17T07:26:11.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:11 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1610150527' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-17T07:26:12.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:11 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1610150527' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-17T07:26:12.310 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 3 1 2024-09-17T07:26:12.509 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:26:12.876 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:12 smithi100 ceph-mon[39462]: 4.2 scrub starts 2024-09-17T07:26:12.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:12 smithi100 ceph-mon[39462]: 4.2 scrub ok 2024-09-17T07:26:12.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:12 smithi100 ceph-mon[39462]: pgmap v1152: 142 pgs: 142 active+clean; 582 KiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 846 B/s rd, 0 op/s 2024-09-17T07:26:12.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:12 smithi100 ceph-mon[39462]: 6.16 scrub starts 2024-09-17T07:26:12.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:12 smithi100 ceph-mon[39462]: 6.16 scrub ok 2024-09-17T07:26:12.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:12 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1610150527' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-17T07:26:12.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:12 smithi100 ceph-mon[39462]: osdmap e526: 8 total, 8 up, 8 in 2024-09-17T07:26:12.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:12 smithi100 ceph-mon[29092]: 4.2 scrub starts 2024-09-17T07:26:12.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:12 smithi100 ceph-mon[29092]: 4.2 scrub ok 2024-09-17T07:26:12.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:12 smithi100 ceph-mon[29092]: pgmap v1152: 142 pgs: 142 active+clean; 582 KiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 846 B/s rd, 0 op/s 2024-09-17T07:26:12.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:12 smithi100 ceph-mon[29092]: 6.16 scrub starts 2024-09-17T07:26:12.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:12 smithi100 ceph-mon[29092]: 6.16 scrub ok 2024-09-17T07:26:12.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:12 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1610150527' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-17T07:26:12.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:12 smithi100 ceph-mon[29092]: osdmap e526: 8 total, 8 up, 8 in 2024-09-17T07:26:13.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:12 smithi135 ceph-mon[33238]: 4.2 scrub starts 2024-09-17T07:26:13.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:12 smithi135 ceph-mon[33238]: 4.2 scrub ok 2024-09-17T07:26:13.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:12 smithi135 ceph-mon[33238]: pgmap v1152: 142 pgs: 142 active+clean; 582 KiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 846 B/s rd, 0 op/s 2024-09-17T07:26:13.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:12 smithi135 ceph-mon[33238]: 6.16 scrub starts 2024-09-17T07:26:13.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:12 smithi135 ceph-mon[33238]: 6.16 scrub ok 2024-09-17T07:26:13.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:12 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1610150527' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-17T07:26:13.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:12 smithi135 ceph-mon[33238]: osdmap e526: 8 total, 8 up, 8 in 2024-09-17T07:26:13.870 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:13 smithi100 ceph-mon[39462]: 5.9 scrub starts 2024-09-17T07:26:13.870 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:13 smithi100 ceph-mon[39462]: 5.9 scrub ok 2024-09-17T07:26:13.870 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:13 smithi100 ceph-mon[29092]: 5.9 scrub starts 2024-09-17T07:26:13.870 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:13 smithi100 ceph-mon[29092]: 5.9 scrub ok 2024-09-17T07:26:14.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:13 smithi135 ceph-mon[33238]: 5.9 scrub starts 2024-09-17T07:26:14.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:13 smithi135 ceph-mon[33238]: 5.9 scrub ok 2024-09-17T07:26:14.611 INFO:teuthology.orchestra.run.smithi100.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-09-17T07:26:14.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:14 smithi100 ceph-mon[29092]: 5.1d scrub starts 2024-09-17T07:26:14.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:14 smithi100 ceph-mon[29092]: 5.1d scrub ok 2024-09-17T07:26:14.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:14 smithi100 ceph-mon[29092]: pgmap v1154: 142 pgs: 142 active+clean; 582 KiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:26:14.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:14 smithi100 ceph-mon[29092]: 5.5 scrub starts 2024-09-17T07:26:14.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:14 smithi100 ceph-mon[29092]: 5.5 scrub ok 2024-09-17T07:26:14.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:14 smithi100 ceph-mon[29092]: 5.2 scrub starts 2024-09-17T07:26:14.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:14 smithi100 ceph-mon[29092]: 5.2 scrub ok 2024-09-17T07:26:14.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:14 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2095342176' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-09-17T07:26:14.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:14 smithi100 ceph-mon[39462]: 5.1d scrub starts 2024-09-17T07:26:14.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:14 smithi100 ceph-mon[39462]: 5.1d scrub ok 2024-09-17T07:26:14.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:14 smithi100 ceph-mon[39462]: pgmap v1154: 142 pgs: 142 active+clean; 582 KiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:26:14.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:14 smithi100 ceph-mon[39462]: 5.5 scrub starts 2024-09-17T07:26:14.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:14 smithi100 ceph-mon[39462]: 5.5 scrub ok 2024-09-17T07:26:14.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:14 smithi100 ceph-mon[39462]: 5.2 scrub starts 2024-09-17T07:26:14.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:14 smithi100 ceph-mon[39462]: 5.2 scrub ok 2024-09-17T07:26:14.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:14 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2095342176' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-09-17T07:26:15.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:14 smithi135 ceph-mon[33238]: 5.1d scrub starts 2024-09-17T07:26:15.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:14 smithi135 ceph-mon[33238]: 5.1d scrub ok 2024-09-17T07:26:15.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:14 smithi135 ceph-mon[33238]: pgmap v1154: 142 pgs: 142 active+clean; 582 KiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:26:15.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:14 smithi135 ceph-mon[33238]: 5.5 scrub starts 2024-09-17T07:26:15.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:14 smithi135 ceph-mon[33238]: 5.5 scrub ok 2024-09-17T07:26:15.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:14 smithi135 ceph-mon[33238]: 5.2 scrub starts 2024-09-17T07:26:15.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:14 smithi135 ceph-mon[33238]: 5.2 scrub ok 2024-09-17T07:26:15.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:14 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2095342176' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-09-17T07:26:15.342 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd reweight 4 1 2024-09-17T07:26:15.543 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:26:15.876 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:15 smithi100 ceph-mon[39462]: 5.c scrub starts 2024-09-17T07:26:15.876 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:15 smithi100 ceph-mon[39462]: 5.c scrub ok 2024-09-17T07:26:15.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:15 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2095342176' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-17T07:26:15.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:15 smithi100 ceph-mon[39462]: osdmap e527: 8 total, 8 up, 8 in 2024-09-17T07:26:15.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:15 smithi100 ceph-mon[29092]: 5.c scrub starts 2024-09-17T07:26:15.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:15 smithi100 ceph-mon[29092]: 5.c scrub ok 2024-09-17T07:26:15.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:15 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2095342176' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-17T07:26:15.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:15 smithi100 ceph-mon[29092]: osdmap e527: 8 total, 8 up, 8 in 2024-09-17T07:26:16.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:15 smithi135 ceph-mon[33238]: 5.c scrub starts 2024-09-17T07:26:16.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:15 smithi135 ceph-mon[33238]: 5.c scrub ok 2024-09-17T07:26:16.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:15 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2095342176' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-17T07:26:16.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:15 smithi135 ceph-mon[33238]: osdmap e527: 8 total, 8 up, 8 in 2024-09-17T07:26:16.309 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:26:15 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:26:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:26:16.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:16 smithi100 ceph-mon[29092]: pgmap v1156: 142 pgs: 142 active+clean; 582 KiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:26:16.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:16 smithi100 ceph-mon[39462]: pgmap v1156: 142 pgs: 142 active+clean; 582 KiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:26:17.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:16 smithi135 ceph-mon[33238]: pgmap v1156: 142 pgs: 142 active+clean; 582 KiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:26:17.622 INFO:teuthology.orchestra.run.smithi100.stderr:reweighted osd.4 to 1 (10000) 2024-09-17T07:26:17.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:17 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1087232321' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-17T07:26:17.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:17 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:26:17.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:17 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1087232321' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-17T07:26:17.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:17 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:26:18.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:17 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1087232321' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-17T07:26:18.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:17 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:26:18.400 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 4 1 2024-09-17T07:26:18.598 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:26:18.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:18 smithi135 ceph-mon[33238]: 4.4 scrub starts 2024-09-17T07:26:18.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:18 smithi135 ceph-mon[33238]: 4.4 scrub ok 2024-09-17T07:26:18.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:18 smithi135 ceph-mon[33238]: pgmap v1157: 142 pgs: 142 active+clean; 582 KiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 644 B/s rd, 0 op/s 2024-09-17T07:26:18.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:18 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1087232321' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-17T07:26:18.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:18 smithi135 ceph-mon[33238]: osdmap e528: 8 total, 8 up, 8 in 2024-09-17T07:26:18.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:18 smithi135 ceph-mon[33238]: 4.18 scrub starts 2024-09-17T07:26:18.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:18 smithi135 ceph-mon[33238]: 4.18 scrub ok 2024-09-17T07:26:18.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:18 smithi135 ceph-mon[33238]: 6.11 scrub starts 2024-09-17T07:26:18.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:18 smithi135 ceph-mon[33238]: 6.11 scrub ok 2024-09-17T07:26:18.972 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:18 smithi100 ceph-mon[29092]: 4.4 scrub starts 2024-09-17T07:26:18.973 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:18 smithi100 ceph-mon[29092]: 4.4 scrub ok 2024-09-17T07:26:18.973 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:18 smithi100 ceph-mon[29092]: pgmap v1157: 142 pgs: 142 active+clean; 582 KiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 644 B/s rd, 0 op/s 2024-09-17T07:26:18.973 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:18 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1087232321' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-17T07:26:18.973 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:18 smithi100 ceph-mon[29092]: osdmap e528: 8 total, 8 up, 8 in 2024-09-17T07:26:18.974 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:18 smithi100 ceph-mon[29092]: 4.18 scrub starts 2024-09-17T07:26:18.974 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:18 smithi100 ceph-mon[29092]: 4.18 scrub ok 2024-09-17T07:26:18.974 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:18 smithi100 ceph-mon[29092]: 6.11 scrub starts 2024-09-17T07:26:18.975 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:18 smithi100 ceph-mon[29092]: 6.11 scrub ok 2024-09-17T07:26:18.975 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:18 smithi100 ceph-mon[39462]: 4.4 scrub starts 2024-09-17T07:26:18.975 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:18 smithi100 ceph-mon[39462]: 4.4 scrub ok 2024-09-17T07:26:18.975 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:18 smithi100 ceph-mon[39462]: pgmap v1157: 142 pgs: 142 active+clean; 582 KiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 644 B/s rd, 0 op/s 2024-09-17T07:26:18.976 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:18 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1087232321' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-17T07:26:18.976 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:18 smithi100 ceph-mon[39462]: osdmap e528: 8 total, 8 up, 8 in 2024-09-17T07:26:18.976 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:18 smithi100 ceph-mon[39462]: 4.18 scrub starts 2024-09-17T07:26:18.976 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:18 smithi100 ceph-mon[39462]: 4.18 scrub ok 2024-09-17T07:26:18.976 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:18 smithi100 ceph-mon[39462]: 6.11 scrub starts 2024-09-17T07:26:18.976 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:18 smithi100 ceph-mon[39462]: 6.11 scrub ok 2024-09-17T07:26:19.958 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:26:19 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:26:19.959 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:19 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:26:19.959 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:19 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:26:19.959 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:19 smithi135 ceph-mon[33238]: osdmap e529: 8 total, 8 up, 8 in 2024-09-17T07:26:19.959 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:19 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:26:19.959 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:19 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:26:19.959 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:19 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:26:19.959 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:19 smithi135 ceph-mon[33238]: 6.a scrub starts 2024-09-17T07:26:19.959 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:19 smithi135 ceph-mon[33238]: 6.a scrub ok 2024-09-17T07:26:19.985 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:19 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:26:19.985 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:19 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:26:19.985 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:19 smithi100 ceph-mon[39462]: osdmap e529: 8 total, 8 up, 8 in 2024-09-17T07:26:19.985 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:19 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:26:19.985 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:19 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:26:19.985 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:19 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:26:19.985 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:19 smithi100 ceph-mon[39462]: 6.a scrub starts 2024-09-17T07:26:19.985 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:19 smithi100 ceph-mon[39462]: 6.a scrub ok 2024-09-17T07:26:19.985 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:19 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:26:19.985 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:19 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:26:19.986 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:19 smithi100 ceph-mon[29092]: osdmap e529: 8 total, 8 up, 8 in 2024-09-17T07:26:19.986 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:19 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:26:19.986 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:19 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:26:19.986 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:19 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:26:19.986 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:19 smithi100 ceph-mon[29092]: 6.a scrub starts 2024-09-17T07:26:19.986 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:19 smithi100 ceph-mon[29092]: 6.a scrub ok 2024-09-17T07:26:21.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:20 smithi135 ceph-mon[33238]: pgmap v1160: 142 pgs: 142 active+clean; 582 KiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-17T07:26:21.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:20 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:26:21.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:20 smithi135 ceph-mon[33238]: osdmap e530: 8 total, 8 up, 8 in 2024-09-17T07:26:21.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:20 smithi135 ceph-mon[33238]: 4.d scrub starts 2024-09-17T07:26:21.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:20 smithi135 ceph-mon[33238]: 4.d scrub ok 2024-09-17T07:26:21.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:20 smithi135 ceph-mon[33238]: 4.8 scrub starts 2024-09-17T07:26:21.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:20 smithi135 ceph-mon[33238]: 4.8 scrub ok 2024-09-17T07:26:21.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:20 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1719912722' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-09-17T07:26:21.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:20 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-09-17T07:26:21.126 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:20 smithi100 ceph-mon[29092]: pgmap v1160: 142 pgs: 142 active+clean; 582 KiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-17T07:26:21.126 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:20 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:26:21.126 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:20 smithi100 ceph-mon[29092]: osdmap e530: 8 total, 8 up, 8 in 2024-09-17T07:26:21.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:20 smithi100 ceph-mon[29092]: 4.d scrub starts 2024-09-17T07:26:21.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:20 smithi100 ceph-mon[29092]: 4.d scrub ok 2024-09-17T07:26:21.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:20 smithi100 ceph-mon[29092]: 4.8 scrub starts 2024-09-17T07:26:21.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:20 smithi100 ceph-mon[29092]: 4.8 scrub ok 2024-09-17T07:26:21.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:20 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1719912722' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-09-17T07:26:21.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:20 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-09-17T07:26:21.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:20 smithi100 ceph-mon[39462]: pgmap v1160: 142 pgs: 142 active+clean; 582 KiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-17T07:26:21.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:20 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:26:21.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:20 smithi100 ceph-mon[39462]: osdmap e530: 8 total, 8 up, 8 in 2024-09-17T07:26:21.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:20 smithi100 ceph-mon[39462]: 4.d scrub starts 2024-09-17T07:26:21.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:20 smithi100 ceph-mon[39462]: 4.d scrub ok 2024-09-17T07:26:21.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:20 smithi100 ceph-mon[39462]: 4.8 scrub starts 2024-09-17T07:26:21.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:20 smithi100 ceph-mon[39462]: 4.8 scrub ok 2024-09-17T07:26:21.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:20 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1719912722' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-09-17T07:26:21.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:20 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-09-17T07:26:21.400 INFO:teuthology.orchestra.run.smithi100.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-09-17T07:26:21.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:21 smithi135 ceph-mon[33238]: 6.1d deep-scrub starts 2024-09-17T07:26:21.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:21 smithi135 ceph-mon[33238]: 6.1d deep-scrub ok 2024-09-17T07:26:21.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:21 smithi135 ceph-mon[33238]: 6.13 scrub starts 2024-09-17T07:26:21.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:21 smithi135 ceph-mon[33238]: 6.13 scrub ok 2024-09-17T07:26:21.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:21 smithi135 ceph-mon[33238]: 5.11 deep-scrub starts 2024-09-17T07:26:21.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:21 smithi135 ceph-mon[33238]: 5.11 deep-scrub ok 2024-09-17T07:26:21.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:21 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-17T07:26:21.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:21 smithi135 ceph-mon[33238]: osdmap e531: 8 total, 8 up, 8 in 2024-09-17T07:26:22.126 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:21 smithi100 ceph-mon[29092]: 6.1d deep-scrub starts 2024-09-17T07:26:22.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:21 smithi100 ceph-mon[29092]: 6.1d deep-scrub ok 2024-09-17T07:26:22.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:21 smithi100 ceph-mon[29092]: 6.13 scrub starts 2024-09-17T07:26:22.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:21 smithi100 ceph-mon[29092]: 6.13 scrub ok 2024-09-17T07:26:22.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:21 smithi100 ceph-mon[29092]: 5.11 deep-scrub starts 2024-09-17T07:26:22.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:21 smithi100 ceph-mon[29092]: 5.11 deep-scrub ok 2024-09-17T07:26:22.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:21 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-17T07:26:22.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:21 smithi100 ceph-mon[29092]: osdmap e531: 8 total, 8 up, 8 in 2024-09-17T07:26:22.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:21 smithi100 ceph-mon[39462]: 6.1d deep-scrub starts 2024-09-17T07:26:22.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:21 smithi100 ceph-mon[39462]: 6.1d deep-scrub ok 2024-09-17T07:26:22.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:21 smithi100 ceph-mon[39462]: 6.13 scrub starts 2024-09-17T07:26:22.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:21 smithi100 ceph-mon[39462]: 6.13 scrub ok 2024-09-17T07:26:22.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:21 smithi100 ceph-mon[39462]: 5.11 deep-scrub starts 2024-09-17T07:26:22.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:21 smithi100 ceph-mon[39462]: 5.11 deep-scrub ok 2024-09-17T07:26:22.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:21 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-17T07:26:22.128 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:21 smithi100 ceph-mon[39462]: osdmap e531: 8 total, 8 up, 8 in 2024-09-17T07:26:22.234 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd reweight 6 1 2024-09-17T07:26:22.434 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:26:23.126 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:22 smithi100 ceph-mon[39462]: pgmap v1163: 142 pgs: 142 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:26:23.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:22 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:26:23.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:22 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:26:23.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:22 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:26:23.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:22 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:26:23.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:22 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:26:23.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:22 smithi100 ceph-mon[39462]: 2.1a scrub starts 2024-09-17T07:26:23.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:22 smithi100 ceph-mon[39462]: 2.1a scrub ok 2024-09-17T07:26:23.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:22 smithi100 ceph-mon[29092]: pgmap v1163: 142 pgs: 142 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:26:23.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:22 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:26:23.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:22 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:26:23.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:22 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:26:23.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:22 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:26:23.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:22 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:26:23.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:22 smithi100 ceph-mon[29092]: 2.1a scrub starts 2024-09-17T07:26:23.128 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:22 smithi100 ceph-mon[29092]: 2.1a scrub ok 2024-09-17T07:26:23.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:22 smithi135 ceph-mon[33238]: pgmap v1163: 142 pgs: 142 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:26:23.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:22 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:26:23.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:22 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:26:23.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:22 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:26:23.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:22 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:26:23.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:22 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:26:23.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:22 smithi135 ceph-mon[33238]: 2.1a scrub starts 2024-09-17T07:26:23.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:22 smithi135 ceph-mon[33238]: 2.1a scrub ok 2024-09-17T07:26:23.958 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:23 smithi100 ceph-mon[29092]: 5.18 deep-scrub starts 2024-09-17T07:26:23.958 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:23 smithi100 ceph-mon[29092]: 5.18 deep-scrub ok 2024-09-17T07:26:23.958 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:23 smithi100 ceph-mon[29092]: 5.13 scrub starts 2024-09-17T07:26:23.958 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:23 smithi100 ceph-mon[29092]: 5.13 scrub ok 2024-09-17T07:26:23.958 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:23 smithi100 ceph-mon[29092]: 3.0 scrub starts 2024-09-17T07:26:23.958 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:23 smithi100 ceph-mon[29092]: 3.0 scrub ok 2024-09-17T07:26:23.959 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:23 smithi100 ceph-mon[39462]: 5.18 deep-scrub starts 2024-09-17T07:26:23.959 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:23 smithi100 ceph-mon[39462]: 5.18 deep-scrub ok 2024-09-17T07:26:23.959 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:23 smithi100 ceph-mon[39462]: 5.13 scrub starts 2024-09-17T07:26:23.959 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:23 smithi100 ceph-mon[39462]: 5.13 scrub ok 2024-09-17T07:26:23.959 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:23 smithi100 ceph-mon[39462]: 3.0 scrub starts 2024-09-17T07:26:23.959 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:23 smithi100 ceph-mon[39462]: 3.0 scrub ok 2024-09-17T07:26:24.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:23 smithi135 ceph-mon[33238]: 5.18 deep-scrub starts 2024-09-17T07:26:24.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:23 smithi135 ceph-mon[33238]: 5.18 deep-scrub ok 2024-09-17T07:26:24.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:23 smithi135 ceph-mon[33238]: 5.13 scrub starts 2024-09-17T07:26:24.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:23 smithi135 ceph-mon[33238]: 5.13 scrub ok 2024-09-17T07:26:24.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:23 smithi135 ceph-mon[33238]: 3.0 scrub starts 2024-09-17T07:26:24.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:23 smithi135 ceph-mon[33238]: 3.0 scrub ok 2024-09-17T07:26:24.852 INFO:teuthology.orchestra.run.smithi100.stderr:reweighted osd.6 to 1 (10000) 2024-09-17T07:26:25.126 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:24 smithi100 ceph-mon[29092]: pgmap v1164: 142 pgs: 142 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 865 B/s rd, 0 op/s 2024-09-17T07:26:25.126 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:24 smithi100 ceph-mon[29092]: 4.b scrub starts 2024-09-17T07:26:25.126 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:24 smithi100 ceph-mon[29092]: 4.b scrub ok 2024-09-17T07:26:25.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:24 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1796490784' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-17T07:26:25.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:24 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-17T07:26:25.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:24 smithi100 ceph-mon[39462]: pgmap v1164: 142 pgs: 142 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 865 B/s rd, 0 op/s 2024-09-17T07:26:25.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:24 smithi100 ceph-mon[39462]: 4.b scrub starts 2024-09-17T07:26:25.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:24 smithi100 ceph-mon[39462]: 4.b scrub ok 2024-09-17T07:26:25.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:24 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1796490784' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-17T07:26:25.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:24 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-17T07:26:25.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:24 smithi135 ceph-mon[33238]: pgmap v1164: 142 pgs: 142 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 865 B/s rd, 0 op/s 2024-09-17T07:26:25.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:24 smithi135 ceph-mon[33238]: 4.b scrub starts 2024-09-17T07:26:25.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:24 smithi135 ceph-mon[33238]: 4.b scrub ok 2024-09-17T07:26:25.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:24 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1796490784' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-17T07:26:25.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:24 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-17T07:26:25.653 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 6 1 2024-09-17T07:26:25.853 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:26:25.889 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:25 smithi100 ceph-mon[29092]: 2.5 scrub starts 2024-09-17T07:26:25.889 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:25 smithi100 ceph-mon[29092]: 2.5 scrub ok 2024-09-17T07:26:25.889 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:25 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-17T07:26:25.890 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:25 smithi100 ceph-mon[29092]: osdmap e532: 8 total, 8 up, 8 in 2024-09-17T07:26:25.890 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:25 smithi100 ceph-mon[29092]: 4.19 scrub starts 2024-09-17T07:26:25.890 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:25 smithi100 ceph-mon[29092]: 4.19 scrub ok 2024-09-17T07:26:25.890 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:25 smithi100 ceph-mon[29092]: 2.6 scrub starts 2024-09-17T07:26:25.890 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:25 smithi100 ceph-mon[29092]: 2.6 scrub ok 2024-09-17T07:26:25.891 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:25 smithi100 ceph-mon[29092]: 2.16 scrub starts 2024-09-17T07:26:25.891 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:25 smithi100 ceph-mon[29092]: 2.16 scrub ok 2024-09-17T07:26:25.891 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:25 smithi100 ceph-mon[39462]: 2.5 scrub starts 2024-09-17T07:26:25.891 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:25 smithi100 ceph-mon[39462]: 2.5 scrub ok 2024-09-17T07:26:25.892 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:25 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-17T07:26:25.892 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:25 smithi100 ceph-mon[39462]: osdmap e532: 8 total, 8 up, 8 in 2024-09-17T07:26:25.892 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:25 smithi100 ceph-mon[39462]: 4.19 scrub starts 2024-09-17T07:26:25.892 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:25 smithi100 ceph-mon[39462]: 4.19 scrub ok 2024-09-17T07:26:25.892 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:25 smithi100 ceph-mon[39462]: 2.6 scrub starts 2024-09-17T07:26:25.892 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:25 smithi100 ceph-mon[39462]: 2.6 scrub ok 2024-09-17T07:26:25.893 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:25 smithi100 ceph-mon[39462]: 2.16 scrub starts 2024-09-17T07:26:25.893 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:25 smithi100 ceph-mon[39462]: 2.16 scrub ok 2024-09-17T07:26:26.207 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:26:25 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:26:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:26:26.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:25 smithi135 ceph-mon[33238]: 2.5 scrub starts 2024-09-17T07:26:26.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:25 smithi135 ceph-mon[33238]: 2.5 scrub ok 2024-09-17T07:26:26.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:25 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-17T07:26:26.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:25 smithi135 ceph-mon[33238]: osdmap e532: 8 total, 8 up, 8 in 2024-09-17T07:26:26.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:25 smithi135 ceph-mon[33238]: 4.19 scrub starts 2024-09-17T07:26:26.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:25 smithi135 ceph-mon[33238]: 4.19 scrub ok 2024-09-17T07:26:26.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:25 smithi135 ceph-mon[33238]: 2.6 scrub starts 2024-09-17T07:26:26.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:25 smithi135 ceph-mon[33238]: 2.6 scrub ok 2024-09-17T07:26:26.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:25 smithi135 ceph-mon[33238]: 2.16 scrub starts 2024-09-17T07:26:26.294 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:25 smithi135 ceph-mon[33238]: 2.16 scrub ok 2024-09-17T07:26:27.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:26 smithi100 ceph-mon[29092]: pgmap v1166: 142 pgs: 142 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:26:27.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:26 smithi100 ceph-mon[29092]: 4.3 scrub starts 2024-09-17T07:26:27.127 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:26 smithi100 ceph-mon[29092]: 4.3 scrub ok 2024-09-17T07:26:27.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:26 smithi100 ceph-mon[39462]: pgmap v1166: 142 pgs: 142 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:26:27.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:26 smithi100 ceph-mon[39462]: 4.3 scrub starts 2024-09-17T07:26:27.127 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:26 smithi100 ceph-mon[39462]: 4.3 scrub ok 2024-09-17T07:26:27.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:26 smithi135 ceph-mon[33238]: pgmap v1166: 142 pgs: 142 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:26:27.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:26 smithi135 ceph-mon[33238]: 4.3 scrub starts 2024-09-17T07:26:27.293 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:26 smithi135 ceph-mon[33238]: 4.3 scrub ok 2024-09-17T07:26:28.409 INFO:teuthology.orchestra.run.smithi100.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-09-17T07:26:28.771 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:28 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:26:28.771 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:28 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:26:28.771 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:28 smithi100 ceph-mon[29092]: 2.1 deep-scrub starts 2024-09-17T07:26:28.771 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:28 smithi100 ceph-mon[29092]: 2.1 deep-scrub ok 2024-09-17T07:26:28.771 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:28 smithi100 ceph-mon[29092]: osdmap e533: 8 total, 8 up, 8 in 2024-09-17T07:26:28.772 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:28 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2467915486' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-09-17T07:26:28.772 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:28 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:26:28.772 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:28 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:26:28.772 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:28 smithi100 ceph-mon[39462]: 2.1 deep-scrub starts 2024-09-17T07:26:28.772 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:28 smithi100 ceph-mon[39462]: 2.1 deep-scrub ok 2024-09-17T07:26:28.772 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:28 smithi100 ceph-mon[39462]: osdmap e533: 8 total, 8 up, 8 in 2024-09-17T07:26:28.772 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:28 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2467915486' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-09-17T07:26:28.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:28 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:26:28.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:28 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:26:28.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:28 smithi135 ceph-mon[33238]: 2.1 deep-scrub starts 2024-09-17T07:26:28.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:28 smithi135 ceph-mon[33238]: 2.1 deep-scrub ok 2024-09-17T07:26:28.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:28 smithi135 ceph-mon[33238]: osdmap e533: 8 total, 8 up, 8 in 2024-09-17T07:26:28.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:28 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2467915486' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-09-17T07:26:29.192 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd reweight 7 1 2024-09-17T07:26:29.402 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:26:29.520 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:29 smithi100 ceph-mon[39462]: pgmap v1168: 142 pgs: 142 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 831 B/s rd, 0 op/s 2024-09-17T07:26:29.520 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:29 smithi100 ceph-mon[39462]: 3.2 scrub starts 2024-09-17T07:26:29.520 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:29 smithi100 ceph-mon[39462]: 3.2 scrub ok 2024-09-17T07:26:29.520 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:29 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2467915486' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-17T07:26:29.520 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:29 smithi100 ceph-mon[39462]: osdmap e534: 8 total, 8 up, 8 in 2024-09-17T07:26:29.520 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:29 smithi100 ceph-mon[39462]: 6.3 scrub starts 2024-09-17T07:26:29.520 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:29 smithi100 ceph-mon[39462]: 6.3 scrub ok 2024-09-17T07:26:29.520 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:29 smithi100 ceph-mon[29092]: pgmap v1168: 142 pgs: 142 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 831 B/s rd, 0 op/s 2024-09-17T07:26:29.521 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:29 smithi100 ceph-mon[29092]: 3.2 scrub starts 2024-09-17T07:26:29.521 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:29 smithi100 ceph-mon[29092]: 3.2 scrub ok 2024-09-17T07:26:29.521 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:29 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2467915486' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-17T07:26:29.521 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:29 smithi100 ceph-mon[29092]: osdmap e534: 8 total, 8 up, 8 in 2024-09-17T07:26:29.521 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:29 smithi100 ceph-mon[29092]: 6.3 scrub starts 2024-09-17T07:26:29.521 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:29 smithi100 ceph-mon[29092]: 6.3 scrub ok 2024-09-17T07:26:29.793 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:26:29 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:26:29.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:29 smithi135 ceph-mon[33238]: pgmap v1168: 142 pgs: 142 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 831 B/s rd, 0 op/s 2024-09-17T07:26:29.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:29 smithi135 ceph-mon[33238]: 3.2 scrub starts 2024-09-17T07:26:29.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:29 smithi135 ceph-mon[33238]: 3.2 scrub ok 2024-09-17T07:26:29.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:29 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2467915486' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-17T07:26:29.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:29 smithi135 ceph-mon[33238]: osdmap e534: 8 total, 8 up, 8 in 2024-09-17T07:26:29.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:29 smithi135 ceph-mon[33238]: 6.3 scrub starts 2024-09-17T07:26:29.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:29 smithi135 ceph-mon[33238]: 6.3 scrub ok 2024-09-17T07:26:30.689 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:30 smithi100 ceph-mon[29092]: 6.1e scrub starts 2024-09-17T07:26:30.689 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:30 smithi100 ceph-mon[29092]: 6.1e scrub ok 2024-09-17T07:26:30.689 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:30 smithi100 ceph-mon[29092]: 5.7 scrub starts 2024-09-17T07:26:30.689 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:30 smithi100 ceph-mon[29092]: 5.7 scrub ok 2024-09-17T07:26:30.689 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:30 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:26:30.689 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:30 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:26:30.690 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:30 smithi100 ceph-mon[39462]: 6.1e scrub starts 2024-09-17T07:26:30.690 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:30 smithi100 ceph-mon[39462]: 6.1e scrub ok 2024-09-17T07:26:30.690 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:30 smithi100 ceph-mon[39462]: 5.7 scrub starts 2024-09-17T07:26:30.690 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:30 smithi100 ceph-mon[39462]: 5.7 scrub ok 2024-09-17T07:26:30.690 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:30 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:26:30.690 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:30 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:26:30.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:30 smithi135 ceph-mon[33238]: 6.1e scrub starts 2024-09-17T07:26:30.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:30 smithi135 ceph-mon[33238]: 6.1e scrub ok 2024-09-17T07:26:30.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:30 smithi135 ceph-mon[33238]: 5.7 scrub starts 2024-09-17T07:26:30.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:30 smithi135 ceph-mon[33238]: 5.7 scrub ok 2024-09-17T07:26:30.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:30 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:26:30.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:30 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:26:31.425 INFO:teuthology.orchestra.run.smithi100.stderr:reweighted osd.7 to 1 (10000) 2024-09-17T07:26:31.779 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:31 smithi100 ceph-mon[39462]: pgmap v1170: 142 pgs: 142 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-17T07:26:31.779 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:31 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:26:31.779 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:31 smithi100 ceph-mon[39462]: 5.a scrub starts 2024-09-17T07:26:31.780 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:31 smithi100 ceph-mon[39462]: 5.a scrub ok 2024-09-17T07:26:31.780 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:31 smithi100 ceph-mon[39462]: osdmap e535: 8 total, 8 up, 8 in 2024-09-17T07:26:31.780 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:31 smithi100 ceph-mon[39462]: 2.12 scrub starts 2024-09-17T07:26:31.780 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:31 smithi100 ceph-mon[39462]: 2.12 scrub ok 2024-09-17T07:26:31.780 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:31 smithi100 ceph-mon[39462]: 6.15 scrub starts 2024-09-17T07:26:31.780 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:31 smithi100 ceph-mon[39462]: 6.15 scrub ok 2024-09-17T07:26:31.780 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:31 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/799199925' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-17T07:26:31.780 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:31 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-17T07:26:31.780 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:31 smithi100 ceph-mon[29092]: pgmap v1170: 142 pgs: 142 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-17T07:26:31.781 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:31 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:26:31.781 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:31 smithi100 ceph-mon[29092]: 5.a scrub starts 2024-09-17T07:26:31.781 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:31 smithi100 ceph-mon[29092]: 5.a scrub ok 2024-09-17T07:26:31.781 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:31 smithi100 ceph-mon[29092]: osdmap e535: 8 total, 8 up, 8 in 2024-09-17T07:26:31.781 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:31 smithi100 ceph-mon[29092]: 2.12 scrub starts 2024-09-17T07:26:31.781 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:31 smithi100 ceph-mon[29092]: 2.12 scrub ok 2024-09-17T07:26:31.781 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:31 smithi100 ceph-mon[29092]: 6.15 scrub starts 2024-09-17T07:26:31.781 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:31 smithi100 ceph-mon[29092]: 6.15 scrub ok 2024-09-17T07:26:31.781 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:31 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/799199925' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-17T07:26:31.781 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:31 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-17T07:26:31.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:31 smithi135 ceph-mon[33238]: pgmap v1170: 142 pgs: 142 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-17T07:26:31.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:31 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:26:31.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:31 smithi135 ceph-mon[33238]: 5.a scrub starts 2024-09-17T07:26:31.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:31 smithi135 ceph-mon[33238]: 5.a scrub ok 2024-09-17T07:26:31.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:31 smithi135 ceph-mon[33238]: osdmap e535: 8 total, 8 up, 8 in 2024-09-17T07:26:31.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:31 smithi135 ceph-mon[33238]: 2.12 scrub starts 2024-09-17T07:26:31.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:31 smithi135 ceph-mon[33238]: 2.12 scrub ok 2024-09-17T07:26:31.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:31 smithi135 ceph-mon[33238]: 6.15 scrub starts 2024-09-17T07:26:31.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:31 smithi135 ceph-mon[33238]: 6.15 scrub ok 2024-09-17T07:26:31.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:31 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/799199925' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-17T07:26:31.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:31 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-17T07:26:32.118 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 7 1 2024-09-17T07:26:32.313 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:26:32.708 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:32 smithi100 ceph-mon[29092]: 5.e scrub starts 2024-09-17T07:26:32.709 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:32 smithi100 ceph-mon[29092]: 5.e scrub ok 2024-09-17T07:26:32.709 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:32 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-17T07:26:32.709 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:32 smithi100 ceph-mon[29092]: osdmap e536: 8 total, 8 up, 8 in 2024-09-17T07:26:32.709 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:32 smithi100 ceph-mon[29092]: 4.6 scrub starts 2024-09-17T07:26:32.709 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:32 smithi100 ceph-mon[29092]: 4.6 scrub ok 2024-09-17T07:26:32.709 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:32 smithi100 ceph-mon[39462]: 5.e scrub starts 2024-09-17T07:26:32.709 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:32 smithi100 ceph-mon[39462]: 5.e scrub ok 2024-09-17T07:26:32.709 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:32 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-17T07:26:32.709 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:32 smithi100 ceph-mon[39462]: osdmap e536: 8 total, 8 up, 8 in 2024-09-17T07:26:32.709 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:32 smithi100 ceph-mon[39462]: 4.6 scrub starts 2024-09-17T07:26:32.710 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:32 smithi100 ceph-mon[39462]: 4.6 scrub ok 2024-09-17T07:26:32.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:32 smithi135 ceph-mon[33238]: 5.e scrub starts 2024-09-17T07:26:32.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:32 smithi135 ceph-mon[33238]: 5.e scrub ok 2024-09-17T07:26:32.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:32 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-17T07:26:32.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:32 smithi135 ceph-mon[33238]: osdmap e536: 8 total, 8 up, 8 in 2024-09-17T07:26:32.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:32 smithi135 ceph-mon[33238]: 4.6 scrub starts 2024-09-17T07:26:32.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:32 smithi135 ceph-mon[33238]: 4.6 scrub ok 2024-09-17T07:26:33.454 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:33 smithi100 ceph-mon[29092]: pgmap v1173: 142 pgs: 142 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:26:33.454 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:33 smithi100 ceph-mon[29092]: 5.1b scrub starts 2024-09-17T07:26:33.455 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:33 smithi100 ceph-mon[29092]: 5.1b scrub ok 2024-09-17T07:26:33.455 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:33 smithi100 ceph-mon[39462]: pgmap v1173: 142 pgs: 142 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:26:33.455 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:33 smithi100 ceph-mon[39462]: 5.1b scrub starts 2024-09-17T07:26:33.455 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:33 smithi100 ceph-mon[39462]: 5.1b scrub ok 2024-09-17T07:26:33.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:33 smithi135 ceph-mon[33238]: pgmap v1173: 142 pgs: 142 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:26:33.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:33 smithi135 ceph-mon[33238]: 5.1b scrub starts 2024-09-17T07:26:33.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:33 smithi135 ceph-mon[33238]: 5.1b scrub ok 2024-09-17T07:26:34.443 INFO:teuthology.orchestra.run.smithi100.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-09-17T07:26:34.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:34 smithi135 ceph-mon[33238]: 2.a scrub starts 2024-09-17T07:26:34.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:34 smithi135 ceph-mon[33238]: 2.a scrub ok 2024-09-17T07:26:34.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:34 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:26:34.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:34 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/499848324' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-09-17T07:26:34.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:34 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-09-17T07:26:34.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:34 smithi100 ceph-mon[39462]: 2.a scrub starts 2024-09-17T07:26:34.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:34 smithi100 ceph-mon[39462]: 2.a scrub ok 2024-09-17T07:26:34.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:34 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:26:34.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:34 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/499848324' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-09-17T07:26:34.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:34 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-09-17T07:26:34.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:34 smithi100 ceph-mon[29092]: 2.a scrub starts 2024-09-17T07:26:34.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:34 smithi100 ceph-mon[29092]: 2.a scrub ok 2024-09-17T07:26:34.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:34 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:26:34.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:34 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/499848324' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-09-17T07:26:34.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:34 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-09-17T07:26:35.312 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd reweight 1 1 2024-09-17T07:26:35.523 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:26:35.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:35 smithi100 ceph-mon[29092]: pgmap v1174: 142 pgs: 142 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:26:35.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:35 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-17T07:26:35.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:35 smithi100 ceph-mon[29092]: osdmap e537: 8 total, 8 up, 8 in 2024-09-17T07:26:35.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:35 smithi100 ceph-mon[29092]: 1.6 scrub starts 2024-09-17T07:26:35.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:35 smithi100 ceph-mon[29092]: 1.6 scrub ok 2024-09-17T07:26:35.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:35 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:26:35.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:35 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:26:35.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:35 smithi100 ceph-mon[39462]: pgmap v1174: 142 pgs: 142 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:26:35.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:35 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-17T07:26:35.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:35 smithi100 ceph-mon[39462]: osdmap e537: 8 total, 8 up, 8 in 2024-09-17T07:26:35.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:35 smithi100 ceph-mon[39462]: 1.6 scrub starts 2024-09-17T07:26:35.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:35 smithi100 ceph-mon[39462]: 1.6 scrub ok 2024-09-17T07:26:35.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:35 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:26:35.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:35 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:26:35.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:35 smithi135 ceph-mon[33238]: pgmap v1174: 142 pgs: 142 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:26:35.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:35 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-17T07:26:35.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:35 smithi135 ceph-mon[33238]: osdmap e537: 8 total, 8 up, 8 in 2024-09-17T07:26:35.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:35 smithi135 ceph-mon[33238]: 1.6 scrub starts 2024-09-17T07:26:35.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:35 smithi135 ceph-mon[33238]: 1.6 scrub ok 2024-09-17T07:26:35.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:35 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:26:35.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:35 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:26:35.975 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:26:35 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:26:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:26:36.713 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:36 smithi100 ceph-mon[29092]: 5.b scrub starts 2024-09-17T07:26:36.713 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:36 smithi100 ceph-mon[29092]: 5.b scrub ok 2024-09-17T07:26:36.713 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:36 smithi100 ceph-mon[29092]: 2.3 deep-scrub starts 2024-09-17T07:26:36.713 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:36 smithi100 ceph-mon[29092]: 2.3 deep-scrub ok 2024-09-17T07:26:36.713 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:36 smithi100 ceph-mon[29092]: osdmap e538: 8 total, 8 up, 8 in 2024-09-17T07:26:36.714 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:36 smithi100 ceph-mon[29092]: 4.a scrub starts 2024-09-17T07:26:36.714 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:36 smithi100 ceph-mon[29092]: 4.a scrub ok 2024-09-17T07:26:36.714 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:36 smithi100 ceph-mon[29092]: 4.17 scrub starts 2024-09-17T07:26:36.714 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:36 smithi100 ceph-mon[29092]: 4.17 scrub ok 2024-09-17T07:26:36.714 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:36 smithi100 ceph-mon[39462]: 5.b scrub starts 2024-09-17T07:26:36.714 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:36 smithi100 ceph-mon[39462]: 5.b scrub ok 2024-09-17T07:26:36.714 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:36 smithi100 ceph-mon[39462]: 2.3 deep-scrub starts 2024-09-17T07:26:36.714 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:36 smithi100 ceph-mon[39462]: 2.3 deep-scrub ok 2024-09-17T07:26:36.714 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:36 smithi100 ceph-mon[39462]: osdmap e538: 8 total, 8 up, 8 in 2024-09-17T07:26:36.714 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:36 smithi100 ceph-mon[39462]: 4.a scrub starts 2024-09-17T07:26:36.714 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:36 smithi100 ceph-mon[39462]: 4.a scrub ok 2024-09-17T07:26:36.715 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:36 smithi100 ceph-mon[39462]: 4.17 scrub starts 2024-09-17T07:26:36.715 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:36 smithi100 ceph-mon[39462]: 4.17 scrub ok 2024-09-17T07:26:36.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:36 smithi135 ceph-mon[33238]: 5.b scrub starts 2024-09-17T07:26:36.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:36 smithi135 ceph-mon[33238]: 5.b scrub ok 2024-09-17T07:26:36.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:36 smithi135 ceph-mon[33238]: 2.3 deep-scrub starts 2024-09-17T07:26:36.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:36 smithi135 ceph-mon[33238]: 2.3 deep-scrub ok 2024-09-17T07:26:36.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:36 smithi135 ceph-mon[33238]: osdmap e538: 8 total, 8 up, 8 in 2024-09-17T07:26:36.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:36 smithi135 ceph-mon[33238]: 4.a scrub starts 2024-09-17T07:26:36.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:36 smithi135 ceph-mon[33238]: 4.a scrub ok 2024-09-17T07:26:36.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:36 smithi135 ceph-mon[33238]: 4.17 scrub starts 2024-09-17T07:26:36.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:36 smithi135 ceph-mon[33238]: 4.17 scrub ok 2024-09-17T07:26:37.452 INFO:teuthology.orchestra.run.smithi100.stderr:reweighted osd.1 to 1 (10000) 2024-09-17T07:26:37.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:37 smithi100 ceph-mon[29092]: pgmap v1177: 142 pgs: 142 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 997 B/s rd, 0 op/s 2024-09-17T07:26:37.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:37 smithi100 ceph-mon[29092]: 4.15 scrub starts 2024-09-17T07:26:37.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:37 smithi100 ceph-mon[29092]: 4.15 scrub ok 2024-09-17T07:26:37.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:37 smithi100 ceph-mon[29092]: 6.0 deep-scrub starts 2024-09-17T07:26:37.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:37 smithi100 ceph-mon[29092]: 6.0 deep-scrub ok 2024-09-17T07:26:37.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:37 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1840737904' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-17T07:26:37.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:37 smithi100 ceph-mon[39462]: pgmap v1177: 142 pgs: 142 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 997 B/s rd, 0 op/s 2024-09-17T07:26:37.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:37 smithi100 ceph-mon[39462]: 4.15 scrub starts 2024-09-17T07:26:37.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:37 smithi100 ceph-mon[39462]: 4.15 scrub ok 2024-09-17T07:26:37.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:37 smithi100 ceph-mon[39462]: 6.0 deep-scrub starts 2024-09-17T07:26:37.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:37 smithi100 ceph-mon[39462]: 6.0 deep-scrub ok 2024-09-17T07:26:37.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:37 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1840737904' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-17T07:26:37.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:37 smithi135 ceph-mon[33238]: pgmap v1177: 142 pgs: 142 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 997 B/s rd, 0 op/s 2024-09-17T07:26:37.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:37 smithi135 ceph-mon[33238]: 4.15 scrub starts 2024-09-17T07:26:37.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:37 smithi135 ceph-mon[33238]: 4.15 scrub ok 2024-09-17T07:26:37.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:37 smithi135 ceph-mon[33238]: 6.0 deep-scrub starts 2024-09-17T07:26:37.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:37 smithi135 ceph-mon[33238]: 6.0 deep-scrub ok 2024-09-17T07:26:37.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:37 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1840737904' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-17T07:26:38.260 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 1 1 2024-09-17T07:26:38.458 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:26:38.493 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:38 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1840737904' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-17T07:26:38.493 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:38 smithi100 ceph-mon[29092]: osdmap e539: 8 total, 8 up, 8 in 2024-09-17T07:26:38.493 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:38 smithi100 ceph-mon[29092]: 2.b deep-scrub starts 2024-09-17T07:26:38.494 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:38 smithi100 ceph-mon[29092]: 2.b deep-scrub ok 2024-09-17T07:26:38.494 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:38 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1840737904' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-17T07:26:38.494 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:38 smithi100 ceph-mon[39462]: osdmap e539: 8 total, 8 up, 8 in 2024-09-17T07:26:38.494 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:38 smithi100 ceph-mon[39462]: 2.b deep-scrub starts 2024-09-17T07:26:38.494 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:38 smithi100 ceph-mon[39462]: 2.b deep-scrub ok 2024-09-17T07:26:38.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:38 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1840737904' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-17T07:26:38.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:38 smithi135 ceph-mon[33238]: osdmap e539: 8 total, 8 up, 8 in 2024-09-17T07:26:38.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:38 smithi135 ceph-mon[33238]: 2.b deep-scrub starts 2024-09-17T07:26:38.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:38 smithi135 ceph-mon[33238]: 2.b deep-scrub ok 2024-09-17T07:26:39.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:39 smithi100 ceph-mon[29092]: pgmap v1179: 142 pgs: 142 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:26:39.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:39 smithi100 ceph-mon[39462]: pgmap v1179: 142 pgs: 142 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:26:39.709 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:39 smithi135 ceph-mon[33238]: pgmap v1179: 142 pgs: 142 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:26:39.709 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:26:39 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:26:40.465 INFO:teuthology.orchestra.run.smithi100.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-09-17T07:26:40.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:40 smithi135 ceph-mon[33238]: 4.1e scrub starts 2024-09-17T07:26:40.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:40 smithi135 ceph-mon[33238]: 4.1e scrub ok 2024-09-17T07:26:40.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:40 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1127033463' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-09-17T07:26:40.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:40 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-09-17T07:26:40.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:40 smithi100 ceph-mon[29092]: 4.1e scrub starts 2024-09-17T07:26:40.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:40 smithi100 ceph-mon[29092]: 4.1e scrub ok 2024-09-17T07:26:40.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:40 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1127033463' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-09-17T07:26:40.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:40 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-09-17T07:26:40.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:40 smithi100 ceph-mon[39462]: 4.1e scrub starts 2024-09-17T07:26:40.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:40 smithi100 ceph-mon[39462]: 4.1e scrub ok 2024-09-17T07:26:40.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:40 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1127033463' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-09-17T07:26:40.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:40 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-09-17T07:26:41.160 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd reweight 5 1 2024-09-17T07:26:41.358 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:26:41.741 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:41 smithi100 ceph-mon[29092]: pgmap v1180: 142 pgs: 142 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-17T07:26:41.741 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:41 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:26:41.741 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:41 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-17T07:26:41.741 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:41 smithi100 ceph-mon[29092]: osdmap e540: 8 total, 8 up, 8 in 2024-09-17T07:26:41.741 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:41 smithi100 ceph-mon[29092]: 5.0 scrub starts 2024-09-17T07:26:41.742 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:41 smithi100 ceph-mon[29092]: 5.0 scrub ok 2024-09-17T07:26:41.742 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:41 smithi100 ceph-mon[29092]: 2.7 scrub starts 2024-09-17T07:26:41.742 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:41 smithi100 ceph-mon[29092]: 2.7 scrub ok 2024-09-17T07:26:41.742 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:41 smithi100 ceph-mon[39462]: pgmap v1180: 142 pgs: 142 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-17T07:26:41.742 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:41 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:26:41.742 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:41 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-17T07:26:41.742 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:41 smithi100 ceph-mon[39462]: osdmap e540: 8 total, 8 up, 8 in 2024-09-17T07:26:41.742 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:41 smithi100 ceph-mon[39462]: 5.0 scrub starts 2024-09-17T07:26:41.742 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:41 smithi100 ceph-mon[39462]: 5.0 scrub ok 2024-09-17T07:26:41.742 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:41 smithi100 ceph-mon[39462]: 2.7 scrub starts 2024-09-17T07:26:41.742 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:41 smithi100 ceph-mon[39462]: 2.7 scrub ok 2024-09-17T07:26:41.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:41 smithi135 ceph-mon[33238]: pgmap v1180: 142 pgs: 142 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-17T07:26:41.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:41 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:26:41.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:41 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-17T07:26:41.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:41 smithi135 ceph-mon[33238]: osdmap e540: 8 total, 8 up, 8 in 2024-09-17T07:26:41.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:41 smithi135 ceph-mon[33238]: 5.0 scrub starts 2024-09-17T07:26:41.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:41 smithi135 ceph-mon[33238]: 5.0 scrub ok 2024-09-17T07:26:41.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:41 smithi135 ceph-mon[33238]: 2.7 scrub starts 2024-09-17T07:26:41.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:41 smithi135 ceph-mon[33238]: 2.7 scrub ok 2024-09-17T07:26:42.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:42 smithi135 ceph-mon[33238]: 5.f scrub starts 2024-09-17T07:26:42.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:42 smithi135 ceph-mon[33238]: 5.f scrub ok 2024-09-17T07:26:42.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:42 smithi135 ceph-mon[33238]: 2.14 scrub starts 2024-09-17T07:26:42.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:42 smithi135 ceph-mon[33238]: 2.14 scrub ok 2024-09-17T07:26:42.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:42 smithi135 ceph-mon[33238]: 5.3 scrub starts 2024-09-17T07:26:42.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:42 smithi135 ceph-mon[33238]: 5.3 scrub ok 2024-09-17T07:26:42.837 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:42 smithi100 ceph-mon[29092]: 5.f scrub starts 2024-09-17T07:26:42.837 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:42 smithi100 ceph-mon[29092]: 5.f scrub ok 2024-09-17T07:26:42.837 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:42 smithi100 ceph-mon[29092]: 2.14 scrub starts 2024-09-17T07:26:42.837 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:42 smithi100 ceph-mon[29092]: 2.14 scrub ok 2024-09-17T07:26:42.837 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:42 smithi100 ceph-mon[29092]: 5.3 scrub starts 2024-09-17T07:26:42.837 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:42 smithi100 ceph-mon[29092]: 5.3 scrub ok 2024-09-17T07:26:42.838 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:42 smithi100 ceph-mon[39462]: 5.f scrub starts 2024-09-17T07:26:42.838 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:42 smithi100 ceph-mon[39462]: 5.f scrub ok 2024-09-17T07:26:42.838 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:42 smithi100 ceph-mon[39462]: 2.14 scrub starts 2024-09-17T07:26:42.838 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:42 smithi100 ceph-mon[39462]: 2.14 scrub ok 2024-09-17T07:26:42.838 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:42 smithi100 ceph-mon[39462]: 5.3 scrub starts 2024-09-17T07:26:42.838 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:42 smithi100 ceph-mon[39462]: 5.3 scrub ok 2024-09-17T07:26:43.479 INFO:teuthology.orchestra.run.smithi100.stderr:reweighted osd.5 to 1 (10000) 2024-09-17T07:26:43.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:43 smithi135 ceph-mon[33238]: pgmap v1182: 142 pgs: 142 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 837 B/s rd, 0 op/s 2024-09-17T07:26:43.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:43 smithi135 ceph-mon[33238]: 6.4 scrub starts 2024-09-17T07:26:43.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:43 smithi135 ceph-mon[33238]: 6.4 scrub ok 2024-09-17T07:26:43.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:43 smithi135 ceph-mon[33238]: 6.1f scrub starts 2024-09-17T07:26:43.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:43 smithi135 ceph-mon[33238]: 6.1f scrub ok 2024-09-17T07:26:43.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:43 smithi135 ceph-mon[33238]: 1.7 scrub starts 2024-09-17T07:26:43.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:43 smithi135 ceph-mon[33238]: 1.7 scrub ok 2024-09-17T07:26:43.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:43 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2669808731' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-17T07:26:43.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:43 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-17T07:26:43.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:43 smithi100 ceph-mon[29092]: pgmap v1182: 142 pgs: 142 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 837 B/s rd, 0 op/s 2024-09-17T07:26:43.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:43 smithi100 ceph-mon[29092]: 6.4 scrub starts 2024-09-17T07:26:43.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:43 smithi100 ceph-mon[29092]: 6.4 scrub ok 2024-09-17T07:26:43.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:43 smithi100 ceph-mon[29092]: 6.1f scrub starts 2024-09-17T07:26:43.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:43 smithi100 ceph-mon[29092]: 6.1f scrub ok 2024-09-17T07:26:43.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:43 smithi100 ceph-mon[29092]: 1.7 scrub starts 2024-09-17T07:26:43.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:43 smithi100 ceph-mon[29092]: 1.7 scrub ok 2024-09-17T07:26:43.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:43 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2669808731' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-17T07:26:43.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:43 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-17T07:26:43.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:43 smithi100 ceph-mon[39462]: pgmap v1182: 142 pgs: 142 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 837 B/s rd, 0 op/s 2024-09-17T07:26:43.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:43 smithi100 ceph-mon[39462]: 6.4 scrub starts 2024-09-17T07:26:43.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:43 smithi100 ceph-mon[39462]: 6.4 scrub ok 2024-09-17T07:26:43.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:43 smithi100 ceph-mon[39462]: 6.1f scrub starts 2024-09-17T07:26:43.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:43 smithi100 ceph-mon[39462]: 6.1f scrub ok 2024-09-17T07:26:43.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:43 smithi100 ceph-mon[39462]: 1.7 scrub starts 2024-09-17T07:26:43.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:43 smithi100 ceph-mon[39462]: 1.7 scrub ok 2024-09-17T07:26:43.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:43 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2669808731' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-17T07:26:43.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:43 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-17T07:26:44.392 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 5 1 2024-09-17T07:26:44.590 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:26:44.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:44 smithi100 ceph-mon[29092]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-17T07:26:44.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:44 smithi100 ceph-mon[29092]: osdmap e541: 8 total, 8 up, 8 in 2024-09-17T07:26:44.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:44 smithi100 ceph-mon[29092]: 2.13 scrub starts 2024-09-17T07:26:44.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:44 smithi100 ceph-mon[29092]: 2.13 scrub ok 2024-09-17T07:26:44.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:44 smithi100 ceph-mon[39462]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-17T07:26:44.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:44 smithi100 ceph-mon[39462]: osdmap e541: 8 total, 8 up, 8 in 2024-09-17T07:26:44.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:44 smithi100 ceph-mon[39462]: 2.13 scrub starts 2024-09-17T07:26:44.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:44 smithi100 ceph-mon[39462]: 2.13 scrub ok 2024-09-17T07:26:44.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:44 smithi135 ceph-mon[33238]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-17T07:26:44.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:44 smithi135 ceph-mon[33238]: osdmap e541: 8 total, 8 up, 8 in 2024-09-17T07:26:44.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:44 smithi135 ceph-mon[33238]: 2.13 scrub starts 2024-09-17T07:26:44.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:44 smithi135 ceph-mon[33238]: 2.13 scrub ok 2024-09-17T07:26:45.772 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:45 smithi100 ceph-mon[29092]: pgmap v1184: 142 pgs: 142 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 838 B/s rd, 0 op/s 2024-09-17T07:26:45.772 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:45 smithi100 ceph-mon[29092]: 4.f scrub starts 2024-09-17T07:26:45.772 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:45 smithi100 ceph-mon[29092]: 4.f scrub ok 2024-09-17T07:26:45.772 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:45 smithi100 ceph-mon[29092]: 2.18 scrub starts 2024-09-17T07:26:45.772 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:45 smithi100 ceph-mon[29092]: 2.18 scrub ok 2024-09-17T07:26:45.772 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:45 smithi100 ceph-mon[29092]: 4.10 scrub starts 2024-09-17T07:26:45.772 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:45 smithi100 ceph-mon[29092]: 4.10 scrub ok 2024-09-17T07:26:45.772 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:45 smithi100 ceph-mon[29092]: 4.5 scrub starts 2024-09-17T07:26:45.773 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:45 smithi100 ceph-mon[29092]: 4.5 scrub ok 2024-09-17T07:26:45.773 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:45 smithi100 ceph-mon[39462]: pgmap v1184: 142 pgs: 142 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 838 B/s rd, 0 op/s 2024-09-17T07:26:45.773 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:45 smithi100 ceph-mon[39462]: 4.f scrub starts 2024-09-17T07:26:45.773 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:45 smithi100 ceph-mon[39462]: 4.f scrub ok 2024-09-17T07:26:45.773 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:45 smithi100 ceph-mon[39462]: 2.18 scrub starts 2024-09-17T07:26:45.773 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:45 smithi100 ceph-mon[39462]: 2.18 scrub ok 2024-09-17T07:26:45.773 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:45 smithi100 ceph-mon[39462]: 4.10 scrub starts 2024-09-17T07:26:45.773 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:45 smithi100 ceph-mon[39462]: 4.10 scrub ok 2024-09-17T07:26:45.773 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:45 smithi100 ceph-mon[39462]: 4.5 scrub starts 2024-09-17T07:26:45.773 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:45 smithi100 ceph-mon[39462]: 4.5 scrub ok 2024-09-17T07:26:45.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:45 smithi135 ceph-mon[33238]: pgmap v1184: 142 pgs: 142 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 838 B/s rd, 0 op/s 2024-09-17T07:26:45.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:45 smithi135 ceph-mon[33238]: 4.f scrub starts 2024-09-17T07:26:45.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:45 smithi135 ceph-mon[33238]: 4.f scrub ok 2024-09-17T07:26:45.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:45 smithi135 ceph-mon[33238]: 2.18 scrub starts 2024-09-17T07:26:45.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:45 smithi135 ceph-mon[33238]: 2.18 scrub ok 2024-09-17T07:26:45.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:45 smithi135 ceph-mon[33238]: 4.10 scrub starts 2024-09-17T07:26:45.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:45 smithi135 ceph-mon[33238]: 4.10 scrub ok 2024-09-17T07:26:45.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:45 smithi135 ceph-mon[33238]: 4.5 scrub starts 2024-09-17T07:26:45.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:45 smithi135 ceph-mon[33238]: 4.5 scrub ok 2024-09-17T07:26:46.075 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:26:45 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:26:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:26:46.491 INFO:teuthology.orchestra.run.smithi100.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-09-17T07:26:46.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:46 smithi135 ceph-mon[33238]: 5.1c deep-scrub starts 2024-09-17T07:26:46.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:46 smithi135 ceph-mon[33238]: 5.1c deep-scrub ok 2024-09-17T07:26:46.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:46 smithi135 ceph-mon[33238]: 3.1 scrub starts 2024-09-17T07:26:46.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:46 smithi135 ceph-mon[33238]: 3.1 scrub ok 2024-09-17T07:26:46.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:46 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1541247456' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-09-17T07:26:46.803 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:46 smithi100 ceph-mon[29092]: 5.1c deep-scrub starts 2024-09-17T07:26:46.803 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:46 smithi100 ceph-mon[29092]: 5.1c deep-scrub ok 2024-09-17T07:26:46.803 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:46 smithi100 ceph-mon[29092]: 3.1 scrub starts 2024-09-17T07:26:46.803 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:46 smithi100 ceph-mon[29092]: 3.1 scrub ok 2024-09-17T07:26:46.803 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:46 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1541247456' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-09-17T07:26:46.803 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:46 smithi100 ceph-mon[39462]: 5.1c deep-scrub starts 2024-09-17T07:26:46.803 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:46 smithi100 ceph-mon[39462]: 5.1c deep-scrub ok 2024-09-17T07:26:46.804 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:46 smithi100 ceph-mon[39462]: 3.1 scrub starts 2024-09-17T07:26:46.804 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:46 smithi100 ceph-mon[39462]: 3.1 scrub ok 2024-09-17T07:26:46.804 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:46 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1541247456' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-09-17T07:26:47.111 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-09-17T07:26:47.111 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-09-17T07:26:47.111 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-09-17T07:26:47.112 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-09-17T07:26:47.112 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-17T07:26:47.112 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-17T07:26:47.313 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:26:47.724 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:47 smithi100 ceph-mon[29092]: pgmap v1185: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:26:47.724 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:47 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1541247456' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-17T07:26:47.724 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:47 smithi100 ceph-mon[29092]: osdmap e542: 8 total, 8 up, 8 in 2024-09-17T07:26:47.724 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:47 smithi100 ceph-mon[29092]: 2.8 scrub starts 2024-09-17T07:26:47.724 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:47 smithi100 ceph-mon[29092]: 2.8 scrub ok 2024-09-17T07:26:47.724 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:47 smithi100 ceph-mon[39462]: pgmap v1185: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:26:47.725 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:47 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1541247456' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-17T07:26:47.725 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:47 smithi100 ceph-mon[39462]: osdmap e542: 8 total, 8 up, 8 in 2024-09-17T07:26:47.725 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:47 smithi100 ceph-mon[39462]: 2.8 scrub starts 2024-09-17T07:26:47.725 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:47 smithi100 ceph-mon[39462]: 2.8 scrub ok 2024-09-17T07:26:47.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:47 smithi135 ceph-mon[33238]: pgmap v1185: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:26:47.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:47 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1541247456' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-17T07:26:47.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:47 smithi135 ceph-mon[33238]: osdmap e542: 8 total, 8 up, 8 in 2024-09-17T07:26:47.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:47 smithi135 ceph-mon[33238]: 2.8 scrub starts 2024-09-17T07:26:47.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:47 smithi135 ceph-mon[33238]: 2.8 scrub ok 2024-09-17T07:26:48.758 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:48 smithi100 ceph-mon[29092]: 4.1 scrub starts 2024-09-17T07:26:48.759 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:48 smithi100 ceph-mon[29092]: 4.1 scrub ok 2024-09-17T07:26:48.759 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:48 smithi100 ceph-mon[29092]: pgmap v1187: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 721 B/s rd, 0 op/s 2024-09-17T07:26:48.759 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:48 smithi100 ceph-mon[29092]: 6.1b scrub starts 2024-09-17T07:26:48.759 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:48 smithi100 ceph-mon[29092]: 6.1b scrub ok 2024-09-17T07:26:48.759 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:48 smithi100 ceph-mon[39462]: 4.1 scrub starts 2024-09-17T07:26:48.759 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:48 smithi100 ceph-mon[39462]: 4.1 scrub ok 2024-09-17T07:26:48.759 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:48 smithi100 ceph-mon[39462]: pgmap v1187: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 721 B/s rd, 0 op/s 2024-09-17T07:26:48.759 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:48 smithi100 ceph-mon[39462]: 6.1b scrub starts 2024-09-17T07:26:48.760 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:48 smithi100 ceph-mon[39462]: 6.1b scrub ok 2024-09-17T07:26:48.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:48 smithi135 ceph-mon[33238]: 4.1 scrub starts 2024-09-17T07:26:48.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:48 smithi135 ceph-mon[33238]: 4.1 scrub ok 2024-09-17T07:26:48.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:48 smithi135 ceph-mon[33238]: pgmap v1187: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 721 B/s rd, 0 op/s 2024-09-17T07:26:48.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:48 smithi135 ceph-mon[33238]: 6.1b scrub starts 2024-09-17T07:26:48.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:48 smithi135 ceph-mon[33238]: 6.1b scrub ok 2024-09-17T07:26:49.377 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T07:26:49.378 INFO:teuthology.orchestra.run.smithi100.stdout:{"epoch":542,"fsid":"e848267c-74c1-11ef-bceb-c7b262605968","created":"2024-09-17T06:57:05.010532+0000","modified":"2024-09-17T07:26:46.486037+0000","last_up_change":"2024-09-17T07:23:56.453211+0000","last_in_change":"2024-09-17T07:25:48.397524+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T07:00:08.816136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":11,"pg_placement_num":11,"pg_placement_num_target":11,"pg_num_target":11,"pg_num_pending":11,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"519","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"374","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.1800000667572021,"score_stable":2.1800000667572021,"optimal_score":1,"raw_score_acting":2.1800000667572021,"raw_score_stable":2.1800000667572021,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-09-17T07:02:10.759826+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"datapool","create_time":"2024-09-17T07:02:10.991289+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":72,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-09-17T07:02:12.291462+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-09-17T07:02:14.253214+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-09-17T07:02:16.310620+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"27bee9ec-05e6-406c-8bf1-168098da0ec7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":515,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6801","nonce":3341017455}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":3341017455}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":3341017455}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6803","nonce":3341017455}]},"public_addr":"172.21.15.100:6801/3341017455","cluster_addr":"172.21.15.100:6802/3341017455","heartbeat_back_addr":"172.21.15.100:6804/3341017455","heartbeat_front_addr":"172.21.15.100:6803/3341017455","state":["exists","up"]},{"osd":1,"uuid":"b1a30c9c-eeaf-478e-9b62-2d63ad45873d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":302,"up_from":413,"up_thru":515,"down_at":303,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6805","nonce":2894367777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":2894367777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":2894367777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6807","nonce":2894367777}]},"public_addr":"172.21.15.100:6805/2894367777","cluster_addr":"172.21.15.100:6806/2894367777","heartbeat_back_addr":"172.21.15.100:6808/2894367777","heartbeat_front_addr":"172.21.15.100:6807/2894367777","state":["exists","up"]},{"osd":2,"uuid":"ce542e0d-9915-465e-aa11-7b3ce13f694f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":524,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":714579846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":714579846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":714579846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":714579846}]},"public_addr":"172.21.15.100:6809/714579846","cluster_addr":"172.21.15.100:6810/714579846","heartbeat_back_addr":"172.21.15.100:6812/714579846","heartbeat_front_addr":"172.21.15.100:6811/714579846","state":["exists","up"]},{"osd":3,"uuid":"a1257575-3c90-41de-88f6-3960744351b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":518,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6813","nonce":2315956572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":2315956572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":2315956572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":2315956572}]},"public_addr":"172.21.15.100:6813/2315956572","cluster_addr":"172.21.15.100:6814/2315956572","heartbeat_back_addr":"172.21.15.100:6816/2315956572","heartbeat_front_addr":"172.21.15.100:6815/2315956572","state":["exists","up"]},{"osd":4,"uuid":"e44e8922-2c5e-44e3-b556-d4dfbc499649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":515,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":1148820922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6801","nonce":1148820922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6803","nonce":1148820922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":1148820922}]},"public_addr":"172.21.15.135:6800/1148820922","cluster_addr":"172.21.15.135:6801/1148820922","heartbeat_back_addr":"172.21.15.135:6803/1148820922","heartbeat_front_addr":"172.21.15.135:6802/1148820922","state":["exists","up"]},{"osd":5,"uuid":"91e4a362-fcdb-41b9-b54e-716591ba6d79","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":42,"last_clean_end":446,"up_from":476,"up_thru":518,"down_at":447,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":1324312753}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6805","nonce":1324312753}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6807","nonce":1324312753}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":1324312753}]},"public_addr":"172.21.15.135:6804/1324312753","cluster_addr":"172.21.15.135:6805/1324312753","heartbeat_back_addr":"172.21.15.135:6807/1324312753","heartbeat_front_addr":"172.21.15.135:6806/1324312753","state":["exists","up"]},{"osd":6,"uuid":"de799f64-ebd2-4a46-91f6-c2f276bee21f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":518,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":2750184152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6809","nonce":2750184152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6811","nonce":2750184152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":2750184152}]},"public_addr":"172.21.15.135:6808/2750184152","cluster_addr":"172.21.15.135:6809/2750184152","heartbeat_back_addr":"172.21.15.135:6811/2750184152","heartbeat_front_addr":"172.21.15.135:6810/2750184152","state":["exists","up"]},{"osd":7,"uuid":"830932c7-5de1-44ff-a840-e53a6e859bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":518,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":160304773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6813","nonce":160304773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6815","nonce":160304773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":160304773}]},"public_addr":"172.21.15.135:6812/160304773","cluster_addr":"172.21.15.135:6813/160304773","heartbeat_back_addr":"172.21.15.135:6815/160304773","heartbeat_front_addr":"172.21.15.135:6814/160304773","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:26:00.750165+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:26:29.546961+0000","dead_epoch":302},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:26:18.576438+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:26:07.952045+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:26:19.053506+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:26:26.759311+0000","dead_epoch":446},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:26:35.063586+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:25:53.910481+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.18","mappings":[{"from":7,"to":2}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.9","mappings":[{"from":1,"to":0}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]},{"pgid":"6.18","mappings":[{"from":0,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.100:0/3796664909":"2024-09-18T06:57:35.678309+0000","172.21.15.100:0/2225586217":"2024-09-18T07:02:46.981486+0000","172.21.15.100:0/1626706829":"2024-09-18T06:57:35.678309+0000","172.21.15.100:0/3452172206":"2024-09-18T06:57:35.678309+0000","172.21.15.100:6800/3992296558":"2024-09-18T06:57:58.526309+0000","172.21.15.100:6800/3498574214":"2024-09-18T06:57:35.678309+0000","172.21.15.100:0/1365215411":"2024-09-18T06:57:58.526309+0000","172.21.15.100:0/3362169179":"2024-09-18T07:02:46.981486+0000","172.21.15.100:0/2573779025":"2024-09-18T07:02:46.981486+0000","172.21.15.100:0/380524491":"2024-09-18T07:02:46.981486+0000","172.21.15.100:0/1657291976":"2024-09-18T06:57:58.526309+0000","172.21.15.100:0/1676314112":"2024-09-18T07:02:46.981486+0000","172.21.15.100:0/1255197255":"2024-09-18T06:57:58.526309+0000","172.21.15.100:6800/3472658181":"2024-09-18T07:02:46.981486+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T07:26:49.746 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:49 smithi100 ceph-mon[29092]: 1.1 scrub starts 2024-09-17T07:26:49.746 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:49 smithi100 ceph-mon[29092]: 1.1 scrub ok 2024-09-17T07:26:49.746 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:49 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:26:49.746 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:49 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/4128928873' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-17T07:26:49.746 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:49 smithi100 ceph-mon[39462]: 1.1 scrub starts 2024-09-17T07:26:49.746 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:49 smithi100 ceph-mon[39462]: 1.1 scrub ok 2024-09-17T07:26:49.746 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:49 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:26:49.746 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:49 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/4128928873' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-17T07:26:49.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:49 smithi135 ceph-mon[33238]: 1.1 scrub starts 2024-09-17T07:26:49.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:49 smithi135 ceph-mon[33238]: 1.1 scrub ok 2024-09-17T07:26:49.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:49 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:26:49.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:49 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/4128928873' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-17T07:26:49.794 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:26:49 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:26:50.165 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-17T07:26:50.165 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-17T07:26:50.370 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:26:50.652 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:50 smithi100 ceph-mon[39462]: 2.d scrub starts 2024-09-17T07:26:50.652 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:50 smithi100 ceph-mon[39462]: 2.d scrub ok 2024-09-17T07:26:50.652 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:50 smithi100 ceph-mon[39462]: pgmap v1188: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:26:50.652 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:50 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:26:50.652 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:50 smithi100 ceph-mon[39462]: 6.9 scrub starts 2024-09-17T07:26:50.652 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:50 smithi100 ceph-mon[39462]: 6.9 scrub ok 2024-09-17T07:26:50.652 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:50 smithi100 ceph-mon[29092]: 2.d scrub starts 2024-09-17T07:26:50.652 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:50 smithi100 ceph-mon[29092]: 2.d scrub ok 2024-09-17T07:26:50.653 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:50 smithi100 ceph-mon[29092]: pgmap v1188: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:26:50.653 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:50 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:26:50.653 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:50 smithi100 ceph-mon[29092]: 6.9 scrub starts 2024-09-17T07:26:50.653 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:50 smithi100 ceph-mon[29092]: 6.9 scrub ok 2024-09-17T07:26:50.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:50 smithi135 ceph-mon[33238]: 2.d scrub starts 2024-09-17T07:26:50.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:50 smithi135 ceph-mon[33238]: 2.d scrub ok 2024-09-17T07:26:50.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:50 smithi135 ceph-mon[33238]: pgmap v1188: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:26:50.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:50 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:26:50.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:50 smithi135 ceph-mon[33238]: 6.9 scrub starts 2024-09-17T07:26:50.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:50 smithi135 ceph-mon[33238]: 6.9 scrub ok 2024-09-17T07:26:51.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:51 smithi100 ceph-mon[39462]: 1.a scrub starts 2024-09-17T07:26:51.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:51 smithi100 ceph-mon[39462]: 1.a scrub ok 2024-09-17T07:26:51.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:51 smithi100 ceph-mon[29092]: 1.a scrub starts 2024-09-17T07:26:51.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:51 smithi100 ceph-mon[29092]: 1.a scrub ok 2024-09-17T07:26:52.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:51 smithi135 ceph-mon[33238]: 1.a scrub starts 2024-09-17T07:26:52.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:51 smithi135 ceph-mon[33238]: 1.a scrub ok 2024-09-17T07:26:52.268 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T07:26:52.268 INFO:teuthology.orchestra.run.smithi100.stdout:{"epoch":542,"fsid":"e848267c-74c1-11ef-bceb-c7b262605968","created":"2024-09-17T06:57:05.010532+0000","modified":"2024-09-17T07:26:46.486037+0000","last_up_change":"2024-09-17T07:23:56.453211+0000","last_in_change":"2024-09-17T07:25:48.397524+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T07:00:08.816136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":11,"pg_placement_num":11,"pg_placement_num_target":11,"pg_num_target":11,"pg_num_pending":11,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"519","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"374","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.1800000667572021,"score_stable":2.1800000667572021,"optimal_score":1,"raw_score_acting":2.1800000667572021,"raw_score_stable":2.1800000667572021,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-09-17T07:02:10.759826+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"datapool","create_time":"2024-09-17T07:02:10.991289+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":72,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-09-17T07:02:12.291462+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-09-17T07:02:14.253214+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-09-17T07:02:16.310620+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"27bee9ec-05e6-406c-8bf1-168098da0ec7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":515,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6801","nonce":3341017455}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":3341017455}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":3341017455}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6803","nonce":3341017455}]},"public_addr":"172.21.15.100:6801/3341017455","cluster_addr":"172.21.15.100:6802/3341017455","heartbeat_back_addr":"172.21.15.100:6804/3341017455","heartbeat_front_addr":"172.21.15.100:6803/3341017455","state":["exists","up"]},{"osd":1,"uuid":"b1a30c9c-eeaf-478e-9b62-2d63ad45873d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":302,"up_from":413,"up_thru":515,"down_at":303,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6805","nonce":2894367777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":2894367777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":2894367777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6807","nonce":2894367777}]},"public_addr":"172.21.15.100:6805/2894367777","cluster_addr":"172.21.15.100:6806/2894367777","heartbeat_back_addr":"172.21.15.100:6808/2894367777","heartbeat_front_addr":"172.21.15.100:6807/2894367777","state":["exists","up"]},{"osd":2,"uuid":"ce542e0d-9915-465e-aa11-7b3ce13f694f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":524,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":714579846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":714579846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":714579846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":714579846}]},"public_addr":"172.21.15.100:6809/714579846","cluster_addr":"172.21.15.100:6810/714579846","heartbeat_back_addr":"172.21.15.100:6812/714579846","heartbeat_front_addr":"172.21.15.100:6811/714579846","state":["exists","up"]},{"osd":3,"uuid":"a1257575-3c90-41de-88f6-3960744351b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":518,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6813","nonce":2315956572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":2315956572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":2315956572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":2315956572}]},"public_addr":"172.21.15.100:6813/2315956572","cluster_addr":"172.21.15.100:6814/2315956572","heartbeat_back_addr":"172.21.15.100:6816/2315956572","heartbeat_front_addr":"172.21.15.100:6815/2315956572","state":["exists","up"]},{"osd":4,"uuid":"e44e8922-2c5e-44e3-b556-d4dfbc499649","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":515,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":1148820922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6801","nonce":1148820922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6803","nonce":1148820922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":1148820922}]},"public_addr":"172.21.15.135:6800/1148820922","cluster_addr":"172.21.15.135:6801/1148820922","heartbeat_back_addr":"172.21.15.135:6803/1148820922","heartbeat_front_addr":"172.21.15.135:6802/1148820922","state":["exists","up"]},{"osd":5,"uuid":"91e4a362-fcdb-41b9-b54e-716591ba6d79","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":42,"last_clean_end":446,"up_from":476,"up_thru":518,"down_at":447,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":1324312753}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6805","nonce":1324312753}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6807","nonce":1324312753}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":1324312753}]},"public_addr":"172.21.15.135:6804/1324312753","cluster_addr":"172.21.15.135:6805/1324312753","heartbeat_back_addr":"172.21.15.135:6807/1324312753","heartbeat_front_addr":"172.21.15.135:6806/1324312753","state":["exists","up"]},{"osd":6,"uuid":"de799f64-ebd2-4a46-91f6-c2f276bee21f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":518,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":2750184152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6809","nonce":2750184152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6811","nonce":2750184152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":2750184152}]},"public_addr":"172.21.15.135:6808/2750184152","cluster_addr":"172.21.15.135:6809/2750184152","heartbeat_back_addr":"172.21.15.135:6811/2750184152","heartbeat_front_addr":"172.21.15.135:6810/2750184152","state":["exists","up"]},{"osd":7,"uuid":"830932c7-5de1-44ff-a840-e53a6e859bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":518,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":160304773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6813","nonce":160304773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6815","nonce":160304773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":160304773}]},"public_addr":"172.21.15.135:6812/160304773","cluster_addr":"172.21.15.135:6813/160304773","heartbeat_back_addr":"172.21.15.135:6815/160304773","heartbeat_front_addr":"172.21.15.135:6814/160304773","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:26:00.750165+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:26:29.546961+0000","dead_epoch":302},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:26:18.576438+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:26:07.952045+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:26:19.053506+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:26:26.759311+0000","dead_epoch":446},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:26:35.063586+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:25:53.910481+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.18","mappings":[{"from":7,"to":2}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.9","mappings":[{"from":1,"to":0}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]},{"pgid":"6.18","mappings":[{"from":0,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.100:0/3796664909":"2024-09-18T06:57:35.678309+0000","172.21.15.100:0/2225586217":"2024-09-18T07:02:46.981486+0000","172.21.15.100:0/1626706829":"2024-09-18T06:57:35.678309+0000","172.21.15.100:0/3452172206":"2024-09-18T06:57:35.678309+0000","172.21.15.100:6800/3992296558":"2024-09-18T06:57:58.526309+0000","172.21.15.100:6800/3498574214":"2024-09-18T06:57:35.678309+0000","172.21.15.100:0/1365215411":"2024-09-18T06:57:58.526309+0000","172.21.15.100:0/3362169179":"2024-09-18T07:02:46.981486+0000","172.21.15.100:0/2573779025":"2024-09-18T07:02:46.981486+0000","172.21.15.100:0/380524491":"2024-09-18T07:02:46.981486+0000","172.21.15.100:0/1657291976":"2024-09-18T06:57:58.526309+0000","172.21.15.100:0/1676314112":"2024-09-18T07:02:46.981486+0000","172.21.15.100:0/1255197255":"2024-09-18T06:57:58.526309+0000","172.21.15.100:6800/3472658181":"2024-09-18T07:02:46.981486+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T07:26:52.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:52 smithi100 ceph-mon[29092]: 2.c scrub starts 2024-09-17T07:26:52.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:52 smithi100 ceph-mon[29092]: 2.c scrub ok 2024-09-17T07:26:52.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:52 smithi100 ceph-mon[29092]: 2.15 scrub starts 2024-09-17T07:26:52.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:52 smithi100 ceph-mon[29092]: 2.15 scrub ok 2024-09-17T07:26:52.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:52 smithi100 ceph-mon[29092]: pgmap v1189: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:26:52.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:52 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1599694924' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-17T07:26:52.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:52 smithi100 ceph-mon[39462]: 2.c scrub starts 2024-09-17T07:26:52.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:52 smithi100 ceph-mon[39462]: 2.c scrub ok 2024-09-17T07:26:52.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:52 smithi100 ceph-mon[39462]: 2.15 scrub starts 2024-09-17T07:26:52.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:52 smithi100 ceph-mon[39462]: 2.15 scrub ok 2024-09-17T07:26:52.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:52 smithi100 ceph-mon[39462]: pgmap v1189: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:26:52.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:52 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1599694924' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-17T07:26:53.006 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-09-17T07:26:53.007 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-09-17T07:26:53.007 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-09-17T07:26:53.008 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-09-17T07:26:53.008 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-09-17T07:26:53.009 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-09-17T07:26:53.009 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-09-17T07:26:53.009 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-09-17T07:26:53.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:52 smithi135 ceph-mon[33238]: 2.c scrub starts 2024-09-17T07:26:53.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:52 smithi135 ceph-mon[33238]: 2.c scrub ok 2024-09-17T07:26:53.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:52 smithi135 ceph-mon[33238]: 2.15 scrub starts 2024-09-17T07:26:53.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:52 smithi135 ceph-mon[33238]: 2.15 scrub ok 2024-09-17T07:26:53.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:52 smithi135 ceph-mon[33238]: pgmap v1189: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:26:53.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:52 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1599694924' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-17T07:26:53.400 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:26:53.402 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:26:53.402 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:26:53.413 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:26:53.413 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:26:53.818 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:53 smithi100 ceph-mon[29092]: 6.18 scrub starts 2024-09-17T07:26:53.818 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:53 smithi100 ceph-mon[29092]: 6.18 scrub ok 2024-09-17T07:26:53.819 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:53 smithi100 ceph-mon[39462]: 6.18 scrub starts 2024-09-17T07:26:53.819 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:53 smithi100 ceph-mon[39462]: 6.18 scrub ok 2024-09-17T07:26:54.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:53 smithi135 ceph-mon[33238]: 6.18 scrub starts 2024-09-17T07:26:54.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:53 smithi135 ceph-mon[33238]: 6.18 scrub ok 2024-09-17T07:26:54.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:54 smithi100 ceph-mon[29092]: pgmap v1190: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T07:26:54.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:54 smithi100 ceph-mon[29092]: 6.17 scrub starts 2024-09-17T07:26:54.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:54 smithi100 ceph-mon[29092]: 6.17 scrub ok 2024-09-17T07:26:54.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:54 smithi100 ceph-mon[29092]: 1.4 scrub starts 2024-09-17T07:26:54.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:54 smithi100 ceph-mon[29092]: 1.4 scrub ok 2024-09-17T07:26:54.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:54 smithi100 ceph-mon[39462]: pgmap v1190: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T07:26:54.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:54 smithi100 ceph-mon[39462]: 6.17 scrub starts 2024-09-17T07:26:54.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:54 smithi100 ceph-mon[39462]: 6.17 scrub ok 2024-09-17T07:26:54.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:54 smithi100 ceph-mon[39462]: 1.4 scrub starts 2024-09-17T07:26:54.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:54 smithi100 ceph-mon[39462]: 1.4 scrub ok 2024-09-17T07:26:55.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:54 smithi135 ceph-mon[33238]: pgmap v1190: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T07:26:55.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:54 smithi135 ceph-mon[33238]: 6.17 scrub starts 2024-09-17T07:26:55.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:54 smithi135 ceph-mon[33238]: 6.17 scrub ok 2024-09-17T07:26:55.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:54 smithi135 ceph-mon[33238]: 1.4 scrub starts 2024-09-17T07:26:55.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:54 smithi135 ceph-mon[33238]: 1.4 scrub ok 2024-09-17T07:26:55.850 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:55 smithi100 ceph-mon[29092]: 6.14 scrub starts 2024-09-17T07:26:55.850 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:55 smithi100 ceph-mon[29092]: 6.14 scrub ok 2024-09-17T07:26:55.850 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:55 smithi100 ceph-mon[29092]: 5.14 scrub starts 2024-09-17T07:26:55.850 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:55 smithi100 ceph-mon[29092]: 5.14 scrub ok 2024-09-17T07:26:55.850 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:55 smithi100 ceph-mon[39462]: 6.14 scrub starts 2024-09-17T07:26:55.850 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:55 smithi100 ceph-mon[39462]: 6.14 scrub ok 2024-09-17T07:26:55.850 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:55 smithi100 ceph-mon[39462]: 5.14 scrub starts 2024-09-17T07:26:55.850 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:55 smithi100 ceph-mon[39462]: 5.14 scrub ok 2024-09-17T07:26:56.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:55 smithi135 ceph-mon[33238]: 6.14 scrub starts 2024-09-17T07:26:56.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:55 smithi135 ceph-mon[33238]: 6.14 scrub ok 2024-09-17T07:26:56.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:55 smithi135 ceph-mon[33238]: 5.14 scrub starts 2024-09-17T07:26:56.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:55 smithi135 ceph-mon[33238]: 5.14 scrub ok 2024-09-17T07:26:56.126 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:26:55 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:26:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:26:56.947 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:56 smithi100 ceph-mon[39462]: 5.19 scrub starts 2024-09-17T07:26:56.947 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:56 smithi100 ceph-mon[39462]: 5.19 scrub ok 2024-09-17T07:26:56.947 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:56 smithi100 ceph-mon[39462]: pgmap v1191: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T07:26:56.947 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:56 smithi100 ceph-mon[39462]: 2.f scrub starts 2024-09-17T07:26:56.947 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:56 smithi100 ceph-mon[39462]: 2.f scrub ok 2024-09-17T07:26:56.947 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:56 smithi100 ceph-mon[39462]: 5.1e scrub starts 2024-09-17T07:26:56.947 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:56 smithi100 ceph-mon[39462]: 5.1e scrub ok 2024-09-17T07:26:56.947 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:56 smithi100 ceph-mon[29092]: 5.19 scrub starts 2024-09-17T07:26:56.947 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:56 smithi100 ceph-mon[29092]: 5.19 scrub ok 2024-09-17T07:26:56.948 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:56 smithi100 ceph-mon[29092]: pgmap v1191: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T07:26:56.948 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:56 smithi100 ceph-mon[29092]: 2.f scrub starts 2024-09-17T07:26:56.948 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:56 smithi100 ceph-mon[29092]: 2.f scrub ok 2024-09-17T07:26:56.948 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:56 smithi100 ceph-mon[29092]: 5.1e scrub starts 2024-09-17T07:26:56.948 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:56 smithi100 ceph-mon[29092]: 5.1e scrub ok 2024-09-17T07:26:57.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:56 smithi135 ceph-mon[33238]: 5.19 scrub starts 2024-09-17T07:26:57.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:56 smithi135 ceph-mon[33238]: 5.19 scrub ok 2024-09-17T07:26:57.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:56 smithi135 ceph-mon[33238]: pgmap v1191: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T07:26:57.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:56 smithi135 ceph-mon[33238]: 2.f scrub starts 2024-09-17T07:26:57.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:56 smithi135 ceph-mon[33238]: 2.f scrub ok 2024-09-17T07:26:57.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:56 smithi135 ceph-mon[33238]: 5.1e scrub starts 2024-09-17T07:26:57.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:56 smithi135 ceph-mon[33238]: 5.1e scrub ok 2024-09-17T07:26:57.439 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:26:57.440 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:26:57.442 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:26:57.876 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:57 smithi100 ceph-mon[39462]: 2.9 deep-scrub starts 2024-09-17T07:26:57.876 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:57 smithi100 ceph-mon[39462]: 2.9 deep-scrub ok 2024-09-17T07:26:57.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:57 smithi100 ceph-mon[39462]: 6.2 deep-scrub starts 2024-09-17T07:26:57.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:57 smithi100 ceph-mon[39462]: 6.7 scrub starts 2024-09-17T07:26:57.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:57 smithi100 ceph-mon[39462]: 6.2 deep-scrub ok 2024-09-17T07:26:57.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:57 smithi100 ceph-mon[39462]: 6.7 scrub ok 2024-09-17T07:26:57.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:57 smithi100 ceph-mon[39462]: 4.16 scrub starts 2024-09-17T07:26:57.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:57 smithi100 ceph-mon[39462]: 4.16 scrub ok 2024-09-17T07:26:57.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:57 smithi100 ceph-mon[29092]: 2.9 deep-scrub starts 2024-09-17T07:26:57.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:57 smithi100 ceph-mon[29092]: 2.9 deep-scrub ok 2024-09-17T07:26:57.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:57 smithi100 ceph-mon[29092]: 6.2 deep-scrub starts 2024-09-17T07:26:57.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:57 smithi100 ceph-mon[29092]: 6.7 scrub starts 2024-09-17T07:26:57.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:57 smithi100 ceph-mon[29092]: 6.2 deep-scrub ok 2024-09-17T07:26:57.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:57 smithi100 ceph-mon[29092]: 6.7 scrub ok 2024-09-17T07:26:57.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:57 smithi100 ceph-mon[29092]: 4.16 scrub starts 2024-09-17T07:26:57.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:57 smithi100 ceph-mon[29092]: 4.16 scrub ok 2024-09-17T07:26:58.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:57 smithi135 ceph-mon[33238]: 2.9 deep-scrub starts 2024-09-17T07:26:58.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:57 smithi135 ceph-mon[33238]: 2.9 deep-scrub ok 2024-09-17T07:26:58.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:57 smithi135 ceph-mon[33238]: 6.2 deep-scrub starts 2024-09-17T07:26:58.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:57 smithi135 ceph-mon[33238]: 6.7 scrub starts 2024-09-17T07:26:58.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:57 smithi135 ceph-mon[33238]: 6.2 deep-scrub ok 2024-09-17T07:26:58.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:57 smithi135 ceph-mon[33238]: 6.7 scrub ok 2024-09-17T07:26:58.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:57 smithi135 ceph-mon[33238]: 4.16 scrub starts 2024-09-17T07:26:58.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:57 smithi135 ceph-mon[33238]: 4.16 scrub ok 2024-09-17T07:26:58.876 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:58 smithi100 ceph-mon[39462]: 2.10 deep-scrub starts 2024-09-17T07:26:58.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:58 smithi100 ceph-mon[39462]: 2.10 deep-scrub ok 2024-09-17T07:26:58.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:58 smithi100 ceph-mon[39462]: pgmap v1192: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 924 B/s rd, 0 op/s 2024-09-17T07:26:58.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:58 smithi100 ceph-mon[29092]: 2.10 deep-scrub starts 2024-09-17T07:26:58.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:58 smithi100 ceph-mon[29092]: 2.10 deep-scrub ok 2024-09-17T07:26:58.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:58 smithi100 ceph-mon[29092]: pgmap v1192: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 924 B/s rd, 0 op/s 2024-09-17T07:26:59.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:58 smithi135 ceph-mon[33238]: 2.10 deep-scrub starts 2024-09-17T07:26:59.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:58 smithi135 ceph-mon[33238]: 2.10 deep-scrub ok 2024-09-17T07:26:59.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:58 smithi135 ceph-mon[33238]: pgmap v1192: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 924 B/s rd, 0 op/s 2024-09-17T07:26:59.219 INFO:teuthology.orchestra.run.smithi100.stdout:2044404432934 2024-09-17T07:26:59.220 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-09-17T07:26:59.711 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:59 smithi100 ceph-mon[29092]: 5.6 scrub starts 2024-09-17T07:26:59.711 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:59 smithi100 ceph-mon[29092]: 5.6 scrub ok 2024-09-17T07:26:59.711 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:59 smithi100 ceph-mon[29092]: 1.2 scrub starts 2024-09-17T07:26:59.712 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:59 smithi100 ceph-mon[29092]: 1.2 scrub ok 2024-09-17T07:26:59.712 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:59 smithi100 ceph-mon[29092]: 2.1e scrub starts 2024-09-17T07:26:59.712 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:26:59 smithi100 ceph-mon[29092]: 2.1e scrub ok 2024-09-17T07:26:59.712 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:59 smithi100 ceph-mon[39462]: 5.6 scrub starts 2024-09-17T07:26:59.712 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:59 smithi100 ceph-mon[39462]: 5.6 scrub ok 2024-09-17T07:26:59.712 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:59 smithi100 ceph-mon[39462]: 1.2 scrub starts 2024-09-17T07:26:59.712 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:59 smithi100 ceph-mon[39462]: 1.2 scrub ok 2024-09-17T07:26:59.713 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:59 smithi100 ceph-mon[39462]: 2.1e scrub starts 2024-09-17T07:26:59.713 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:26:59 smithi100 ceph-mon[39462]: 2.1e scrub ok 2024-09-17T07:27:00.043 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:26:59 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:27:00.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:59 smithi135 ceph-mon[33238]: 5.6 scrub starts 2024-09-17T07:27:00.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:59 smithi135 ceph-mon[33238]: 5.6 scrub ok 2024-09-17T07:27:00.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:59 smithi135 ceph-mon[33238]: 1.2 scrub starts 2024-09-17T07:27:00.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:59 smithi135 ceph-mon[33238]: 1.2 scrub ok 2024-09-17T07:27:00.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:59 smithi135 ceph-mon[33238]: 2.1e scrub starts 2024-09-17T07:27:00.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:26:59 smithi135 ceph-mon[33238]: 2.1e scrub ok 2024-09-17T07:27:00.849 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:00 smithi100 ceph-mon[29092]: pgmap v1193: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:27:00.849 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:00 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:27:00.849 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:00 smithi100 ceph-mon[29092]: 4.0 scrub starts 2024-09-17T07:27:00.850 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:00 smithi100 ceph-mon[29092]: 4.0 scrub ok 2024-09-17T07:27:00.850 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:00 smithi100 ceph-mon[29092]: 2.11 scrub starts 2024-09-17T07:27:00.850 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:00 smithi100 ceph-mon[29092]: 2.11 scrub ok 2024-09-17T07:27:00.850 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:00 smithi100 ceph-mon[39462]: pgmap v1193: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:27:00.850 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:00 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:27:00.850 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:00 smithi100 ceph-mon[39462]: 4.0 scrub starts 2024-09-17T07:27:00.850 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:00 smithi100 ceph-mon[39462]: 4.0 scrub ok 2024-09-17T07:27:00.850 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:00 smithi100 ceph-mon[39462]: 2.11 scrub starts 2024-09-17T07:27:00.850 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:00 smithi100 ceph-mon[39462]: 2.11 scrub ok 2024-09-17T07:27:01.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:00 smithi135 ceph-mon[33238]: pgmap v1193: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:27:01.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:00 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:27:01.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:00 smithi135 ceph-mon[33238]: 4.0 scrub starts 2024-09-17T07:27:01.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:00 smithi135 ceph-mon[33238]: 4.0 scrub ok 2024-09-17T07:27:01.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:00 smithi135 ceph-mon[33238]: 2.11 scrub starts 2024-09-17T07:27:01.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:00 smithi135 ceph-mon[33238]: 2.11 scrub ok 2024-09-17T07:27:01.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:01 smithi100 ceph-mon[29092]: 2.4 scrub starts 2024-09-17T07:27:01.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:01 smithi100 ceph-mon[29092]: 2.4 scrub ok 2024-09-17T07:27:01.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:01 smithi100 ceph-mon[29092]: 2.1f scrub starts 2024-09-17T07:27:01.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:01 smithi100 ceph-mon[29092]: 2.1f scrub ok 2024-09-17T07:27:01.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:01 smithi100 ceph-mon[29092]: 6.12 scrub starts 2024-09-17T07:27:01.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:01 smithi100 ceph-mon[29092]: 6.12 scrub ok 2024-09-17T07:27:01.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:01 smithi100 ceph-mon[39462]: 2.4 scrub starts 2024-09-17T07:27:01.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:01 smithi100 ceph-mon[39462]: 2.4 scrub ok 2024-09-17T07:27:01.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:01 smithi100 ceph-mon[39462]: 2.1f scrub starts 2024-09-17T07:27:01.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:01 smithi100 ceph-mon[39462]: 2.1f scrub ok 2024-09-17T07:27:01.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:01 smithi100 ceph-mon[39462]: 6.12 scrub starts 2024-09-17T07:27:01.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:01 smithi100 ceph-mon[39462]: 6.12 scrub ok 2024-09-17T07:27:02.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:01 smithi135 ceph-mon[33238]: 2.4 scrub starts 2024-09-17T07:27:02.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:01 smithi135 ceph-mon[33238]: 2.4 scrub ok 2024-09-17T07:27:02.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:01 smithi135 ceph-mon[33238]: 2.1f scrub starts 2024-09-17T07:27:02.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:01 smithi135 ceph-mon[33238]: 2.1f scrub ok 2024-09-17T07:27:02.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:01 smithi135 ceph-mon[33238]: 6.12 scrub starts 2024-09-17T07:27:02.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:01 smithi135 ceph-mon[33238]: 6.12 scrub ok 2024-09-17T07:27:02.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:02 smithi100 ceph-mon[29092]: 1.3 scrub starts 2024-09-17T07:27:02.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:02 smithi100 ceph-mon[29092]: 1.3 scrub ok 2024-09-17T07:27:02.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:02 smithi100 ceph-mon[29092]: pgmap v1194: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:27:02.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:02 smithi100 ceph-mon[29092]: 1.8 scrub starts 2024-09-17T07:27:02.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:02 smithi100 ceph-mon[29092]: 1.8 scrub ok 2024-09-17T07:27:02.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:02 smithi100 ceph-mon[29092]: 4.14 scrub starts 2024-09-17T07:27:02.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:02 smithi100 ceph-mon[29092]: 4.14 scrub ok 2024-09-17T07:27:02.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:02 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:27:02.878 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:02 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:27:02.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:02 smithi100 ceph-mon[39462]: 1.3 scrub starts 2024-09-17T07:27:02.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:02 smithi100 ceph-mon[39462]: 1.3 scrub ok 2024-09-17T07:27:02.879 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:02 smithi100 ceph-mon[39462]: pgmap v1194: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:27:02.879 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:02 smithi100 ceph-mon[39462]: 1.8 scrub starts 2024-09-17T07:27:02.879 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:02 smithi100 ceph-mon[39462]: 1.8 scrub ok 2024-09-17T07:27:02.879 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:02 smithi100 ceph-mon[39462]: 4.14 scrub starts 2024-09-17T07:27:02.879 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:02 smithi100 ceph-mon[39462]: 4.14 scrub ok 2024-09-17T07:27:02.879 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:02 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:27:02.880 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:02 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:27:03.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:02 smithi135 ceph-mon[33238]: 1.3 scrub starts 2024-09-17T07:27:03.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:02 smithi135 ceph-mon[33238]: 1.3 scrub ok 2024-09-17T07:27:03.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:02 smithi135 ceph-mon[33238]: pgmap v1194: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:27:03.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:02 smithi135 ceph-mon[33238]: 1.8 scrub starts 2024-09-17T07:27:03.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:02 smithi135 ceph-mon[33238]: 1.8 scrub ok 2024-09-17T07:27:03.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:02 smithi135 ceph-mon[33238]: 4.14 scrub starts 2024-09-17T07:27:03.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:02 smithi135 ceph-mon[33238]: 4.14 scrub ok 2024-09-17T07:27:03.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:02 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:27:03.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:02 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:27:04.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:03 smithi135 ceph-mon[33238]: osdmap e543: 8 total, 8 up, 8 in 2024-09-17T07:27:04.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:03 smithi135 ceph-mon[33238]: 5.16 scrub starts 2024-09-17T07:27:04.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:03 smithi135 ceph-mon[33238]: 5.16 scrub ok 2024-09-17T07:27:04.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:03 smithi135 ceph-mon[33238]: 4.1a deep-scrub starts 2024-09-17T07:27:04.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:03 smithi135 ceph-mon[33238]: 4.1a deep-scrub ok 2024-09-17T07:27:04.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:03 smithi135 ceph-mon[33238]: 4.1f scrub starts 2024-09-17T07:27:04.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:03 smithi135 ceph-mon[33238]: 4.1f scrub ok 2024-09-17T07:27:04.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:03 smithi135 ceph-mon[33238]: 5.12 scrub starts 2024-09-17T07:27:04.045 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:03 smithi135 ceph-mon[33238]: 5.12 scrub ok 2024-09-17T07:27:04.100 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:03 smithi100 ceph-mon[29092]: osdmap e543: 8 total, 8 up, 8 in 2024-09-17T07:27:04.100 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:03 smithi100 ceph-mon[29092]: 5.16 scrub starts 2024-09-17T07:27:04.101 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:03 smithi100 ceph-mon[29092]: 5.16 scrub ok 2024-09-17T07:27:04.101 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:03 smithi100 ceph-mon[29092]: 4.1a deep-scrub starts 2024-09-17T07:27:04.101 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:03 smithi100 ceph-mon[29092]: 4.1a deep-scrub ok 2024-09-17T07:27:04.101 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:03 smithi100 ceph-mon[29092]: 4.1f scrub starts 2024-09-17T07:27:04.101 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:03 smithi100 ceph-mon[29092]: 4.1f scrub ok 2024-09-17T07:27:04.101 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:03 smithi100 ceph-mon[29092]: 5.12 scrub starts 2024-09-17T07:27:04.101 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:03 smithi100 ceph-mon[29092]: 5.12 scrub ok 2024-09-17T07:27:04.102 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:03 smithi100 ceph-mon[39462]: osdmap e543: 8 total, 8 up, 8 in 2024-09-17T07:27:04.102 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:03 smithi100 ceph-mon[39462]: 5.16 scrub starts 2024-09-17T07:27:04.102 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:03 smithi100 ceph-mon[39462]: 5.16 scrub ok 2024-09-17T07:27:04.102 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:03 smithi100 ceph-mon[39462]: 4.1a deep-scrub starts 2024-09-17T07:27:04.102 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:03 smithi100 ceph-mon[39462]: 4.1a deep-scrub ok 2024-09-17T07:27:04.102 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:03 smithi100 ceph-mon[39462]: 4.1f scrub starts 2024-09-17T07:27:04.102 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:03 smithi100 ceph-mon[39462]: 4.1f scrub ok 2024-09-17T07:27:04.102 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:03 smithi100 ceph-mon[39462]: 5.12 scrub starts 2024-09-17T07:27:04.102 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:03 smithi100 ceph-mon[39462]: 5.12 scrub ok 2024-09-17T07:27:04.981 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:04 smithi100 ceph-mon[29092]: pgmap v1196: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T07:27:04.981 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:04 smithi100 ceph-mon[29092]: 1.5 deep-scrub starts 2024-09-17T07:27:04.981 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:04 smithi100 ceph-mon[29092]: 1.5 deep-scrub ok 2024-09-17T07:27:04.981 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:04 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:27:04.982 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:04 smithi100 ceph-mon[39462]: pgmap v1196: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T07:27:04.982 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:04 smithi100 ceph-mon[39462]: 1.5 deep-scrub starts 2024-09-17T07:27:04.982 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:04 smithi100 ceph-mon[39462]: 1.5 deep-scrub ok 2024-09-17T07:27:04.982 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:04 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:27:05.004 INFO:teuthology.orchestra.run.smithi100.stdout:214748365107 2024-09-17T07:27:05.005 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-09-17T07:27:05.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:04 smithi135 ceph-mon[33238]: pgmap v1196: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T07:27:05.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:04 smithi135 ceph-mon[33238]: 1.5 deep-scrub starts 2024-09-17T07:27:05.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:04 smithi135 ceph-mon[33238]: 1.5 deep-scrub ok 2024-09-17T07:27:05.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:04 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:27:06.307 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:27:05 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:27:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:27:06.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:06 smithi100 ceph-mon[39462]: 6.6 scrub starts 2024-09-17T07:27:06.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:06 smithi100 ceph-mon[39462]: 6.6 scrub ok 2024-09-17T07:27:06.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:06 smithi100 ceph-mon[39462]: 6.f scrub starts 2024-09-17T07:27:06.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:06 smithi100 ceph-mon[39462]: 6.f scrub ok 2024-09-17T07:27:06.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:06 smithi100 ceph-mon[39462]: 6.10 scrub starts 2024-09-17T07:27:06.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:06 smithi100 ceph-mon[39462]: 6.10 scrub ok 2024-09-17T07:27:06.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:06 smithi100 ceph-mon[29092]: 6.6 scrub starts 2024-09-17T07:27:06.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:06 smithi100 ceph-mon[29092]: 6.6 scrub ok 2024-09-17T07:27:06.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:06 smithi100 ceph-mon[29092]: 6.f scrub starts 2024-09-17T07:27:06.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:06 smithi100 ceph-mon[29092]: 6.f scrub ok 2024-09-17T07:27:06.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:06 smithi100 ceph-mon[29092]: 6.10 scrub starts 2024-09-17T07:27:06.628 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:06 smithi100 ceph-mon[29092]: 6.10 scrub ok 2024-09-17T07:27:06.691 INFO:teuthology.orchestra.run.smithi100.stdout:77309411654 2024-09-17T07:27:06.691 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-09-17T07:27:06.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:06 smithi135 ceph-mon[33238]: 6.6 scrub starts 2024-09-17T07:27:06.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:06 smithi135 ceph-mon[33238]: 6.6 scrub ok 2024-09-17T07:27:06.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:06 smithi135 ceph-mon[33238]: 6.f scrub starts 2024-09-17T07:27:06.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:06 smithi135 ceph-mon[33238]: 6.f scrub ok 2024-09-17T07:27:06.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:06 smithi135 ceph-mon[33238]: 6.10 scrub starts 2024-09-17T07:27:06.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:06 smithi135 ceph-mon[33238]: 6.10 scrub ok 2024-09-17T07:27:07.785 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:07 smithi100 ceph-mon[29092]: pgmap v1197: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:27:07.785 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:07 smithi100 ceph-mon[29092]: 4.9 deep-scrub starts 2024-09-17T07:27:07.785 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:07 smithi100 ceph-mon[29092]: 4.9 deep-scrub ok 2024-09-17T07:27:07.785 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:07 smithi100 ceph-mon[29092]: 5.1a scrub starts 2024-09-17T07:27:07.786 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:07 smithi100 ceph-mon[29092]: 5.1a scrub ok 2024-09-17T07:27:07.786 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:07 smithi100 ceph-mon[29092]: 6.c scrub starts 2024-09-17T07:27:07.786 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:07 smithi100 ceph-mon[29092]: 6.c scrub ok 2024-09-17T07:27:07.786 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:07 smithi100 ceph-mon[39462]: pgmap v1197: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:27:07.786 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:07 smithi100 ceph-mon[39462]: 4.9 deep-scrub starts 2024-09-17T07:27:07.786 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:07 smithi100 ceph-mon[39462]: 4.9 deep-scrub ok 2024-09-17T07:27:07.786 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:07 smithi100 ceph-mon[39462]: 5.1a scrub starts 2024-09-17T07:27:07.786 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:07 smithi100 ceph-mon[39462]: 5.1a scrub ok 2024-09-17T07:27:07.786 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:07 smithi100 ceph-mon[39462]: 6.c scrub starts 2024-09-17T07:27:07.786 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:07 smithi100 ceph-mon[39462]: 6.c scrub ok 2024-09-17T07:27:07.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:07 smithi135 ceph-mon[33238]: pgmap v1197: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:27:07.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:07 smithi135 ceph-mon[33238]: 4.9 deep-scrub starts 2024-09-17T07:27:07.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:07 smithi135 ceph-mon[33238]: 4.9 deep-scrub ok 2024-09-17T07:27:07.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:07 smithi135 ceph-mon[33238]: 5.1a scrub starts 2024-09-17T07:27:07.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:07 smithi135 ceph-mon[33238]: 5.1a scrub ok 2024-09-17T07:27:07.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:07 smithi135 ceph-mon[33238]: 6.c scrub starts 2024-09-17T07:27:07.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:07 smithi135 ceph-mon[33238]: 6.c scrub ok 2024-09-17T07:27:08.467 INFO:teuthology.orchestra.run.smithi100.stdout:1773821493320 2024-09-17T07:27:08.467 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-09-17T07:27:08.951 INFO:teuthology.orchestra.run.smithi100.stdout:141733921087 2024-09-17T07:27:08.952 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-09-17T07:27:09.489 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:09 smithi100 ceph-mon[29092]: pgmap v1198: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:27:09.489 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:09 smithi100 ceph-mon[29092]: 4.12 deep-scrub starts 2024-09-17T07:27:09.489 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:09 smithi100 ceph-mon[29092]: 4.12 deep-scrub ok 2024-09-17T07:27:09.489 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:09 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:27:09.489 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:09 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:27:09.489 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:09 smithi100 ceph-mon[29092]: 4.11 scrub starts 2024-09-17T07:27:09.489 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:09 smithi100 ceph-mon[29092]: 4.11 scrub ok 2024-09-17T07:27:09.490 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:09 smithi100 ceph-mon[39462]: pgmap v1198: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:27:09.490 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:09 smithi100 ceph-mon[39462]: 4.12 deep-scrub starts 2024-09-17T07:27:09.490 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:09 smithi100 ceph-mon[39462]: 4.12 deep-scrub ok 2024-09-17T07:27:09.490 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:09 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:27:09.490 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:09 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:27:09.490 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:09 smithi100 ceph-mon[39462]: 4.11 scrub starts 2024-09-17T07:27:09.490 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:09 smithi100 ceph-mon[39462]: 4.11 scrub ok 2024-09-17T07:27:09.742 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:27:09.793 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:27:09 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:27:09.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:09 smithi135 ceph-mon[33238]: pgmap v1198: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:27:09.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:09 smithi135 ceph-mon[33238]: 4.12 deep-scrub starts 2024-09-17T07:27:09.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:09 smithi135 ceph-mon[33238]: 4.12 deep-scrub ok 2024-09-17T07:27:09.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:09 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:27:09.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:09 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:27:09.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:09 smithi135 ceph-mon[33238]: 4.11 scrub starts 2024-09-17T07:27:09.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:09 smithi135 ceph-mon[33238]: 4.11 scrub ok 2024-09-17T07:27:10.241 INFO:teuthology.orchestra.run.smithi100.stdout:34359738703 2024-09-17T07:27:10.242 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-09-17T07:27:10.261 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:27:10.763 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:10 smithi100 ceph-mon[29092]: 2.2 scrub starts 2024-09-17T07:27:10.763 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:10 smithi100 ceph-mon[29092]: 2.2 scrub ok 2024-09-17T07:27:10.763 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:10 smithi100 ceph-mon[29092]: 5.8 scrub starts 2024-09-17T07:27:10.763 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:10 smithi100 ceph-mon[29092]: 5.8 scrub ok 2024-09-17T07:27:10.763 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:10 smithi100 ceph-mon[29092]: osdmap e544: 8 total, 8 up, 8 in 2024-09-17T07:27:10.763 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:10 smithi100 ceph-mon[29092]: 5.10 scrub starts 2024-09-17T07:27:10.763 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:10 smithi100 ceph-mon[29092]: 5.10 scrub ok 2024-09-17T07:27:10.764 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:10 smithi100 ceph-mon[29092]: 6.b scrub starts 2024-09-17T07:27:10.764 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:10 smithi100 ceph-mon[29092]: 6.b scrub ok 2024-09-17T07:27:10.764 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:10 smithi100 ceph-mon[29092]: 2.17 deep-scrub starts 2024-09-17T07:27:10.764 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:10 smithi100 ceph-mon[29092]: 2.17 deep-scrub ok 2024-09-17T07:27:10.764 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:10 smithi100 ceph-mon[39462]: 2.2 scrub starts 2024-09-17T07:27:10.764 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:10 smithi100 ceph-mon[39462]: 2.2 scrub ok 2024-09-17T07:27:10.765 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:10 smithi100 ceph-mon[39462]: 5.8 scrub starts 2024-09-17T07:27:10.765 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:10 smithi100 ceph-mon[39462]: 5.8 scrub ok 2024-09-17T07:27:10.765 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:10 smithi100 ceph-mon[39462]: osdmap e544: 8 total, 8 up, 8 in 2024-09-17T07:27:10.765 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:10 smithi100 ceph-mon[39462]: 5.10 scrub starts 2024-09-17T07:27:10.765 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:10 smithi100 ceph-mon[39462]: 5.10 scrub ok 2024-09-17T07:27:10.765 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:10 smithi100 ceph-mon[39462]: 6.b scrub starts 2024-09-17T07:27:10.766 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:10 smithi100 ceph-mon[39462]: 6.b scrub ok 2024-09-17T07:27:10.766 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:10 smithi100 ceph-mon[39462]: 2.17 deep-scrub starts 2024-09-17T07:27:10.766 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:10 smithi100 ceph-mon[39462]: 2.17 deep-scrub ok 2024-09-17T07:27:10.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:10 smithi135 ceph-mon[33238]: 2.2 scrub starts 2024-09-17T07:27:10.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:10 smithi135 ceph-mon[33238]: 2.2 scrub ok 2024-09-17T07:27:10.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:10 smithi135 ceph-mon[33238]: 5.8 scrub starts 2024-09-17T07:27:10.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:10 smithi135 ceph-mon[33238]: 5.8 scrub ok 2024-09-17T07:27:10.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:10 smithi135 ceph-mon[33238]: osdmap e544: 8 total, 8 up, 8 in 2024-09-17T07:27:10.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:10 smithi135 ceph-mon[33238]: 5.10 scrub starts 2024-09-17T07:27:10.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:10 smithi135 ceph-mon[33238]: 5.10 scrub ok 2024-09-17T07:27:10.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:10 smithi135 ceph-mon[33238]: 6.b scrub starts 2024-09-17T07:27:10.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:10 smithi135 ceph-mon[33238]: 6.b scrub ok 2024-09-17T07:27:10.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:10 smithi135 ceph-mon[33238]: 2.17 deep-scrub starts 2024-09-17T07:27:10.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:10 smithi135 ceph-mon[33238]: 2.17 deep-scrub ok 2024-09-17T07:27:11.548 INFO:teuthology.orchestra.run.smithi100.stdout:115964117314 2024-09-17T07:27:11.549 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-09-17T07:27:11.613 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:11 smithi100 ceph-mon[29092]: pgmap v1200: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:27:11.614 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:11 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:27:11.614 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:11 smithi100 ceph-mon[29092]: 5.d scrub starts 2024-09-17T07:27:11.614 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:11 smithi100 ceph-mon[29092]: 5.d scrub ok 2024-09-17T07:27:11.614 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:11 smithi100 ceph-mon[29092]: 2.19 scrub starts 2024-09-17T07:27:11.614 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:11 smithi100 ceph-mon[29092]: 2.19 scrub ok 2024-09-17T07:27:11.614 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:11 smithi100 ceph-mon[39462]: pgmap v1200: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:27:11.614 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:11 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:27:11.615 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:11 smithi100 ceph-mon[39462]: 5.d scrub starts 2024-09-17T07:27:11.615 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:11 smithi100 ceph-mon[39462]: 5.d scrub ok 2024-09-17T07:27:11.615 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:11 smithi100 ceph-mon[39462]: 2.19 scrub starts 2024-09-17T07:27:11.615 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:11 smithi100 ceph-mon[39462]: 2.19 scrub ok 2024-09-17T07:27:11.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:11 smithi135 ceph-mon[33238]: pgmap v1200: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:27:11.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:11 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:27:11.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:11 smithi135 ceph-mon[33238]: 5.d scrub starts 2024-09-17T07:27:11.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:11 smithi135 ceph-mon[33238]: 5.d scrub ok 2024-09-17T07:27:11.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:11 smithi135 ceph-mon[33238]: 2.19 scrub starts 2024-09-17T07:27:11.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:11 smithi135 ceph-mon[33238]: 2.19 scrub ok 2024-09-17T07:27:12.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:12 smithi100 ceph-mon[39462]: 5.17 scrub starts 2024-09-17T07:27:12.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:12 smithi100 ceph-mon[39462]: 5.17 scrub ok 2024-09-17T07:27:12.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:12 smithi100 ceph-mon[29092]: 5.17 scrub starts 2024-09-17T07:27:12.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:12 smithi100 ceph-mon[29092]: 5.17 scrub ok 2024-09-17T07:27:12.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:12 smithi135 ceph-mon[33238]: 5.17 scrub starts 2024-09-17T07:27:12.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:12 smithi135 ceph-mon[33238]: 5.17 scrub ok 2024-09-17T07:27:13.473 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:27:13.473 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:27:13.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:13 smithi135 ceph-mon[33238]: pgmap v1201: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T07:27:13.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:13 smithi135 ceph-mon[33238]: 4.e scrub starts 2024-09-17T07:27:13.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:13 smithi135 ceph-mon[33238]: 4.e scrub ok 2024-09-17T07:27:13.876 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:13 smithi100 ceph-mon[39462]: pgmap v1201: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T07:27:13.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:13 smithi100 ceph-mon[39462]: 4.e scrub starts 2024-09-17T07:27:13.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:13 smithi100 ceph-mon[39462]: 4.e scrub ok 2024-09-17T07:27:13.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:13 smithi100 ceph-mon[29092]: pgmap v1201: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T07:27:13.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:13 smithi100 ceph-mon[29092]: 4.e scrub starts 2024-09-17T07:27:13.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:13 smithi100 ceph-mon[29092]: 4.e scrub ok 2024-09-17T07:27:14.025 INFO:teuthology.orchestra.run.smithi100.stdout:257698038066 2024-09-17T07:27:14.025 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-09-17T07:27:14.755 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:14 smithi100 ceph-mon[29092]: 2.1d scrub starts 2024-09-17T07:27:14.755 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:14 smithi100 ceph-mon[29092]: 2.1d scrub ok 2024-09-17T07:27:14.755 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:14 smithi100 ceph-mon[39462]: 2.1d scrub starts 2024-09-17T07:27:14.756 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:14 smithi100 ceph-mon[39462]: 2.1d scrub ok 2024-09-17T07:27:14.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:14 smithi135 ceph-mon[33238]: 2.1d scrub starts 2024-09-17T07:27:14.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:14 smithi135 ceph-mon[33238]: 2.1d scrub ok 2024-09-17T07:27:15.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:15 smithi135 ceph-mon[33238]: pgmap v1202: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T07:27:15.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:15 smithi135 ceph-mon[33238]: 2.e scrub starts 2024-09-17T07:27:15.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:15 smithi135 ceph-mon[33238]: 2.e scrub ok 2024-09-17T07:27:15.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:15 smithi135 ceph-mon[33238]: 4.7 scrub starts 2024-09-17T07:27:15.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:15 smithi135 ceph-mon[33238]: 4.7 scrub ok 2024-09-17T07:27:15.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:15 smithi135 ceph-mon[33238]: 6.1c scrub starts 2024-09-17T07:27:15.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:15 smithi135 ceph-mon[33238]: 6.1c scrub ok 2024-09-17T07:27:15.862 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:27:15.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:15 smithi100 ceph-mon[29092]: pgmap v1202: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T07:27:15.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:15 smithi100 ceph-mon[29092]: 2.e scrub starts 2024-09-17T07:27:15.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:15 smithi100 ceph-mon[29092]: 2.e scrub ok 2024-09-17T07:27:15.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:15 smithi100 ceph-mon[29092]: 4.7 scrub starts 2024-09-17T07:27:15.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:15 smithi100 ceph-mon[29092]: 4.7 scrub ok 2024-09-17T07:27:15.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:15 smithi100 ceph-mon[29092]: 6.1c scrub starts 2024-09-17T07:27:15.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:15 smithi100 ceph-mon[29092]: 6.1c scrub ok 2024-09-17T07:27:15.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:15 smithi100 ceph-mon[39462]: pgmap v1202: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T07:27:15.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:15 smithi100 ceph-mon[39462]: 2.e scrub starts 2024-09-17T07:27:15.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:15 smithi100 ceph-mon[39462]: 2.e scrub ok 2024-09-17T07:27:15.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:15 smithi100 ceph-mon[39462]: 4.7 scrub starts 2024-09-17T07:27:15.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:15 smithi100 ceph-mon[39462]: 4.7 scrub ok 2024-09-17T07:27:15.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:15 smithi100 ceph-mon[39462]: 6.1c scrub starts 2024-09-17T07:27:15.878 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:15 smithi100 ceph-mon[39462]: 6.1c scrub ok 2024-09-17T07:27:16.376 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:27:15 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:27:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:27:16.715 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:16 smithi100 ceph-mon[29092]: 2.1c scrub starts 2024-09-17T07:27:16.716 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:16 smithi100 ceph-mon[29092]: 2.1c scrub ok 2024-09-17T07:27:16.716 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:16 smithi100 ceph-mon[29092]: 5.2 scrub starts 2024-09-17T07:27:16.716 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:16 smithi100 ceph-mon[29092]: 5.2 scrub ok 2024-09-17T07:27:16.716 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:16 smithi100 ceph-mon[39462]: 2.1c scrub starts 2024-09-17T07:27:16.716 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:16 smithi100 ceph-mon[39462]: 2.1c scrub ok 2024-09-17T07:27:16.716 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:16 smithi100 ceph-mon[39462]: 5.2 scrub starts 2024-09-17T07:27:16.716 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:16 smithi100 ceph-mon[39462]: 5.2 scrub ok 2024-09-17T07:27:16.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:16 smithi135 ceph-mon[33238]: 2.1c scrub starts 2024-09-17T07:27:16.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:16 smithi135 ceph-mon[33238]: 2.1c scrub ok 2024-09-17T07:27:16.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:16 smithi135 ceph-mon[33238]: 5.2 scrub starts 2024-09-17T07:27:16.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:16 smithi135 ceph-mon[33238]: 5.2 scrub ok 2024-09-17T07:27:17.591 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:17 smithi100 ceph-mon[29092]: pgmap v1203: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-09-17T07:27:17.591 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:17 smithi100 ceph-mon[39462]: pgmap v1203: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-09-17T07:27:17.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:17 smithi135 ceph-mon[33238]: pgmap v1203: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-09-17T07:27:18.156 INFO:teuthology.orchestra.run.smithi100.stdout:141733921089 2024-09-17T07:27:18.780 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:18 smithi100 ceph-mon[29092]: 6.16 scrub starts 2024-09-17T07:27:18.780 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:18 smithi100 ceph-mon[29092]: 6.16 scrub ok 2024-09-17T07:27:18.780 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:18 smithi100 ceph-mon[29092]: 2.1b scrub starts 2024-09-17T07:27:18.780 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:18 smithi100 ceph-mon[29092]: 2.1b scrub ok 2024-09-17T07:27:18.780 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:18 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2610757241' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-17T07:27:18.781 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:18 smithi100 ceph-mon[39462]: 6.16 scrub starts 2024-09-17T07:27:18.781 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:18 smithi100 ceph-mon[39462]: 6.16 scrub ok 2024-09-17T07:27:18.781 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:18 smithi100 ceph-mon[39462]: 2.1b scrub starts 2024-09-17T07:27:18.781 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:18 smithi100 ceph-mon[39462]: 2.1b scrub ok 2024-09-17T07:27:18.781 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:18 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2610757241' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-17T07:27:18.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:18 smithi135 ceph-mon[33238]: 6.16 scrub starts 2024-09-17T07:27:18.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:18 smithi135 ceph-mon[33238]: 6.16 scrub ok 2024-09-17T07:27:18.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:18 smithi135 ceph-mon[33238]: 2.1b scrub starts 2024-09-17T07:27:18.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:18 smithi135 ceph-mon[33238]: 2.1b scrub ok 2024-09-17T07:27:18.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:18 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2610757241' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-17T07:27:18.855 INFO:teuthology.orchestra.run.smithi100.stdout:1773821493322 2024-09-17T07:27:19.724 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:19 smithi100 ceph-mon[29092]: pgmap v1204: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-09-17T07:27:19.725 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:19 smithi100 ceph-mon[29092]: 6.5 scrub starts 2024-09-17T07:27:19.725 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:19 smithi100 ceph-mon[29092]: 6.5 scrub ok 2024-09-17T07:27:19.725 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:19 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:27:19.725 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:19 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/821097670' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-17T07:27:19.725 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:19 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:27:19.725 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:19 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:27:19.725 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:19 smithi100 ceph-mon[39462]: pgmap v1204: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-09-17T07:27:19.725 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:19 smithi100 ceph-mon[39462]: 6.5 scrub starts 2024-09-17T07:27:19.725 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:19 smithi100 ceph-mon[39462]: 6.5 scrub ok 2024-09-17T07:27:19.726 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:19 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:27:19.726 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:19 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/821097670' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-17T07:27:19.726 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:19 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:27:19.726 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:19 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:27:19.746 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733921087 got 141733921089 for osd.4 2024-09-17T07:27:19.746 DEBUG:teuthology.parallel:result is None 2024-09-17T07:27:19.793 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:27:19 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:27:19.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:19 smithi135 ceph-mon[33238]: pgmap v1204: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-09-17T07:27:19.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:19 smithi135 ceph-mon[33238]: 6.5 scrub starts 2024-09-17T07:27:19.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:19 smithi135 ceph-mon[33238]: 6.5 scrub ok 2024-09-17T07:27:19.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:19 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:27:19.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:19 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/821097670' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-17T07:27:19.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:19 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:27:19.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:19 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:27:20.165 INFO:tasks.cephadm.ceph_manager.ceph:need seq 1773821493320 got 1773821493322 for osd.1 2024-09-17T07:27:20.165 DEBUG:teuthology.parallel:result is None 2024-09-17T07:27:20.367 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:27:20.539 INFO:teuthology.orchestra.run.smithi100.stdout:2044404432938 2024-09-17T07:27:20.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:20 smithi135 ceph-mon[33238]: 6.1a deep-scrub starts 2024-09-17T07:27:20.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:20 smithi135 ceph-mon[33238]: 6.1a deep-scrub ok 2024-09-17T07:27:20.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:20 smithi135 ceph-mon[33238]: osdmap e545: 8 total, 8 up, 8 in 2024-09-17T07:27:20.857 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:20 smithi100 ceph-mon[29092]: 6.1a deep-scrub starts 2024-09-17T07:27:20.858 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:20 smithi100 ceph-mon[29092]: 6.1a deep-scrub ok 2024-09-17T07:27:20.858 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:20 smithi100 ceph-mon[29092]: osdmap e545: 8 total, 8 up, 8 in 2024-09-17T07:27:20.858 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:20 smithi100 ceph-mon[39462]: 6.1a deep-scrub starts 2024-09-17T07:27:20.858 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:20 smithi100 ceph-mon[39462]: 6.1a deep-scrub ok 2024-09-17T07:27:20.858 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:20 smithi100 ceph-mon[39462]: osdmap e545: 8 total, 8 up, 8 in 2024-09-17T07:27:21.124 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:27:21.125 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:27:21.486 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:21 smithi100 ceph-mon[29092]: pgmap v1206: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T07:27:21.486 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:21 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:27:21.486 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:21 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2822862945' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-17T07:27:21.487 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:21 smithi100 ceph-mon[39462]: pgmap v1206: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T07:27:21.487 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:21 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:27:21.487 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:21 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2822862945' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-17T07:27:21.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:21 smithi135 ceph-mon[33238]: pgmap v1206: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T07:27:21.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:21 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:27:21.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:21 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2822862945' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-17T07:27:21.927 INFO:tasks.cephadm.ceph_manager.ceph:need seq 2044404432934 got 2044404432938 for osd.5 2024-09-17T07:27:21.927 DEBUG:teuthology.parallel:result is None 2024-09-17T07:27:22.698 INFO:teuthology.orchestra.run.smithi100.stdout:77309411658 2024-09-17T07:27:22.727 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:22 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:27:22.727 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:22 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:27:22.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:22 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:27:23.604 INFO:teuthology.orchestra.run.smithi100.stdout:214748365111 2024-09-17T07:27:23.604 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:23 smithi100 ceph-mon[29092]: pgmap v1207: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T07:27:23.604 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:23 smithi100 ceph-mon[29092]: 5.1 scrub starts 2024-09-17T07:27:23.604 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:23 smithi100 ceph-mon[29092]: 5.1 scrub ok 2024-09-17T07:27:23.604 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:23 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:27:23.604 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:23 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:27:23.605 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:23 smithi100 ceph-mon[29092]: osdmap e546: 8 total, 8 up, 8 in 2024-09-17T07:27:23.605 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:23 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/497938422' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-17T07:27:23.605 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:23 smithi100 ceph-mon[29092]: 4.1d scrub starts 2024-09-17T07:27:23.605 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:23 smithi100 ceph-mon[29092]: 4.1d scrub ok 2024-09-17T07:27:23.605 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:23 smithi100 ceph-mon[29092]: 2.0 scrub starts 2024-09-17T07:27:23.605 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:23 smithi100 ceph-mon[29092]: 2.0 scrub ok 2024-09-17T07:27:23.605 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:23 smithi100 ceph-mon[39462]: pgmap v1207: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T07:27:23.605 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:23 smithi100 ceph-mon[39462]: 5.1 scrub starts 2024-09-17T07:27:23.605 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:23 smithi100 ceph-mon[39462]: 5.1 scrub ok 2024-09-17T07:27:23.605 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:23 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:27:23.606 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:23 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:27:23.606 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:23 smithi100 ceph-mon[39462]: osdmap e546: 8 total, 8 up, 8 in 2024-09-17T07:27:23.606 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:23 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/497938422' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-17T07:27:23.606 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:23 smithi100 ceph-mon[39462]: 4.1d scrub starts 2024-09-17T07:27:23.606 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:23 smithi100 ceph-mon[39462]: 4.1d scrub ok 2024-09-17T07:27:23.606 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:23 smithi100 ceph-mon[39462]: 2.0 scrub starts 2024-09-17T07:27:23.606 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:23 smithi100 ceph-mon[39462]: 2.0 scrub ok 2024-09-17T07:27:23.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:23 smithi135 ceph-mon[33238]: pgmap v1207: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T07:27:23.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:23 smithi135 ceph-mon[33238]: 5.1 scrub starts 2024-09-17T07:27:23.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:23 smithi135 ceph-mon[33238]: 5.1 scrub ok 2024-09-17T07:27:23.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:23 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:27:23.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:23 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:27:23.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:23 smithi135 ceph-mon[33238]: osdmap e546: 8 total, 8 up, 8 in 2024-09-17T07:27:23.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:23 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/497938422' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-17T07:27:23.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:23 smithi135 ceph-mon[33238]: 4.1d scrub starts 2024-09-17T07:27:23.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:23 smithi135 ceph-mon[33238]: 4.1d scrub ok 2024-09-17T07:27:23.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:23 smithi135 ceph-mon[33238]: 2.0 scrub starts 2024-09-17T07:27:23.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:23 smithi135 ceph-mon[33238]: 2.0 scrub ok 2024-09-17T07:27:24.558 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:24 smithi100 ceph-mon[29092]: 6.d scrub starts 2024-09-17T07:27:24.558 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:24 smithi100 ceph-mon[29092]: 6.d scrub ok 2024-09-17T07:27:24.558 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:24 smithi100 ceph-mon[29092]: 6.1d scrub starts 2024-09-17T07:27:24.558 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:24 smithi100 ceph-mon[29092]: 6.1d scrub ok 2024-09-17T07:27:24.558 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:24 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/15340478' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-17T07:27:24.559 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:24 smithi100 ceph-mon[39462]: 6.d scrub starts 2024-09-17T07:27:24.559 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:24 smithi100 ceph-mon[39462]: 6.d scrub ok 2024-09-17T07:27:24.559 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:24 smithi100 ceph-mon[39462]: 6.1d scrub starts 2024-09-17T07:27:24.559 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:24 smithi100 ceph-mon[39462]: 6.1d scrub ok 2024-09-17T07:27:24.559 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:24 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/15340478' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-17T07:27:24.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:24 smithi135 ceph-mon[33238]: 6.d scrub starts 2024-09-17T07:27:24.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:24 smithi135 ceph-mon[33238]: 6.d scrub ok 2024-09-17T07:27:24.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:24 smithi135 ceph-mon[33238]: 6.1d scrub starts 2024-09-17T07:27:24.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:24 smithi135 ceph-mon[33238]: 6.1d scrub ok 2024-09-17T07:27:24.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:24 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/15340478' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-17T07:27:25.165 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411654 got 77309411658 for osd.2 2024-09-17T07:27:25.165 DEBUG:teuthology.parallel:result is None 2024-09-17T07:27:25.744 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:25 smithi100 ceph-mon[29092]: pgmap v1209: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-17T07:27:25.744 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:25 smithi100 ceph-mon[29092]: 4.1c scrub starts 2024-09-17T07:27:25.744 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:25 smithi100 ceph-mon[29092]: 4.1c scrub ok 2024-09-17T07:27:25.744 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:25 smithi100 ceph-mon[29092]: 5.13 scrub starts 2024-09-17T07:27:25.744 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:25 smithi100 ceph-mon[29092]: 5.13 scrub ok 2024-09-17T07:27:25.744 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:25 smithi100 ceph-mon[29092]: 5.9 scrub starts 2024-09-17T07:27:25.744 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:25 smithi100 ceph-mon[29092]: 5.9 scrub ok 2024-09-17T07:27:25.744 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:25 smithi100 ceph-mon[29092]: 5.1f scrub starts 2024-09-17T07:27:25.744 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:25 smithi100 ceph-mon[29092]: 5.1f scrub ok 2024-09-17T07:27:25.745 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:25 smithi100 ceph-mon[39462]: pgmap v1209: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-17T07:27:25.745 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:25 smithi100 ceph-mon[39462]: 4.1c scrub starts 2024-09-17T07:27:25.745 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:25 smithi100 ceph-mon[39462]: 4.1c scrub ok 2024-09-17T07:27:25.745 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:25 smithi100 ceph-mon[39462]: 5.13 scrub starts 2024-09-17T07:27:25.745 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:25 smithi100 ceph-mon[39462]: 5.13 scrub ok 2024-09-17T07:27:25.745 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:25 smithi100 ceph-mon[39462]: 5.9 scrub starts 2024-09-17T07:27:25.745 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:25 smithi100 ceph-mon[39462]: 5.9 scrub ok 2024-09-17T07:27:25.745 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:25 smithi100 ceph-mon[39462]: 5.1f scrub starts 2024-09-17T07:27:25.745 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:25 smithi100 ceph-mon[39462]: 5.1f scrub ok 2024-09-17T07:27:25.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:25 smithi135 ceph-mon[33238]: pgmap v1209: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-17T07:27:25.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:25 smithi135 ceph-mon[33238]: 4.1c scrub starts 2024-09-17T07:27:25.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:25 smithi135 ceph-mon[33238]: 4.1c scrub ok 2024-09-17T07:27:25.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:25 smithi135 ceph-mon[33238]: 5.13 scrub starts 2024-09-17T07:27:25.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:25 smithi135 ceph-mon[33238]: 5.13 scrub ok 2024-09-17T07:27:25.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:25 smithi135 ceph-mon[33238]: 5.9 scrub starts 2024-09-17T07:27:25.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:25 smithi135 ceph-mon[33238]: 5.9 scrub ok 2024-09-17T07:27:25.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:25 smithi135 ceph-mon[33238]: 5.1f scrub starts 2024-09-17T07:27:25.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:25 smithi135 ceph-mon[33238]: 5.1f scrub ok 2024-09-17T07:27:26.027 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:27:25 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:27:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:27:26.052 INFO:tasks.cephadm.ceph_manager.ceph:need seq 214748365107 got 214748365111 for osd.6 2024-09-17T07:27:26.053 DEBUG:teuthology.parallel:result is None 2024-09-17T07:27:26.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:26 smithi100 ceph-mon[39462]: 5.15 scrub starts 2024-09-17T07:27:26.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:26 smithi100 ceph-mon[39462]: 4.13 scrub starts 2024-09-17T07:27:26.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:26 smithi100 ceph-mon[39462]: 5.15 scrub ok 2024-09-17T07:27:26.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:26 smithi100 ceph-mon[39462]: 4.13 scrub ok 2024-09-17T07:27:26.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:26 smithi100 ceph-mon[29092]: 5.15 scrub starts 2024-09-17T07:27:26.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:26 smithi100 ceph-mon[29092]: 4.13 scrub starts 2024-09-17T07:27:26.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:26 smithi100 ceph-mon[29092]: 5.15 scrub ok 2024-09-17T07:27:26.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:26 smithi100 ceph-mon[29092]: 4.13 scrub ok 2024-09-17T07:27:26.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:26 smithi135 ceph-mon[33238]: 5.15 scrub starts 2024-09-17T07:27:26.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:26 smithi135 ceph-mon[33238]: 4.13 scrub starts 2024-09-17T07:27:26.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:26 smithi135 ceph-mon[33238]: 5.15 scrub ok 2024-09-17T07:27:26.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:26 smithi135 ceph-mon[33238]: 4.13 scrub ok 2024-09-17T07:27:27.557 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:27 smithi100 ceph-mon[29092]: pgmap v1210: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:27:27.557 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:27 smithi100 ceph-mon[29092]: 6.11 deep-scrub starts 2024-09-17T07:27:27.558 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:27 smithi100 ceph-mon[29092]: 6.11 deep-scrub ok 2024-09-17T07:27:27.558 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:27 smithi100 ceph-mon[39462]: pgmap v1210: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:27:27.558 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:27 smithi100 ceph-mon[39462]: 6.11 deep-scrub starts 2024-09-17T07:27:27.558 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:27 smithi100 ceph-mon[39462]: 6.11 deep-scrub ok 2024-09-17T07:27:27.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:27 smithi135 ceph-mon[33238]: pgmap v1210: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:27:27.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:27 smithi135 ceph-mon[33238]: 6.11 deep-scrub starts 2024-09-17T07:27:27.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:27 smithi135 ceph-mon[33238]: 6.11 deep-scrub ok 2024-09-17T07:27:27.813 INFO:teuthology.orchestra.run.smithi100.stdout:34359738707 2024-09-17T07:27:28.048 INFO:teuthology.orchestra.run.smithi100.stdout:257698038069 2024-09-17T07:27:28.731 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:28 smithi100 ceph-mon[29092]: 5.4 scrub starts 2024-09-17T07:27:28.731 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:28 smithi100 ceph-mon[29092]: 5.4 scrub ok 2024-09-17T07:27:28.731 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:28 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/3463574578' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-17T07:27:28.731 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:28 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/1943194109' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-17T07:27:28.731 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:28 smithi100 ceph-mon[39462]: 5.4 scrub starts 2024-09-17T07:27:28.731 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:28 smithi100 ceph-mon[39462]: 5.4 scrub ok 2024-09-17T07:27:28.732 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:28 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/3463574578' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-17T07:27:28.732 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:28 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/1943194109' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-17T07:27:28.752 INFO:tasks.cephadm.ceph_manager.ceph:need seq 257698038066 got 257698038069 for osd.7 2024-09-17T07:27:28.752 DEBUG:teuthology.parallel:result is None 2024-09-17T07:27:28.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:28 smithi135 ceph-mon[33238]: 5.4 scrub starts 2024-09-17T07:27:28.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:28 smithi135 ceph-mon[33238]: 5.4 scrub ok 2024-09-17T07:27:28.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:28 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/3463574578' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-17T07:27:28.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:28 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/1943194109' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-17T07:27:29.665 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738703 got 34359738707 for osd.0 2024-09-17T07:27:29.665 DEBUG:teuthology.parallel:result is None 2024-09-17T07:27:29.763 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:29 smithi100 ceph-mon[29092]: pgmap v1211: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 631 B/s rd, 0 op/s 2024-09-17T07:27:29.763 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:29 smithi100 ceph-mon[29092]: 4.1b deep-scrub starts 2024-09-17T07:27:29.763 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:29 smithi100 ceph-mon[29092]: 4.1b deep-scrub ok 2024-09-17T07:27:29.763 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:29 smithi100 ceph-mon[29092]: 2.16 scrub starts 2024-09-17T07:27:29.763 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:29 smithi100 ceph-mon[29092]: 2.16 scrub ok 2024-09-17T07:27:29.763 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:29 smithi100 ceph-mon[29092]: 1.0 scrub starts 2024-09-17T07:27:29.763 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:29 smithi100 ceph-mon[29092]: 1.0 scrub ok 2024-09-17T07:27:29.763 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:29 smithi100 ceph-mon[39462]: pgmap v1211: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 631 B/s rd, 0 op/s 2024-09-17T07:27:29.763 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:29 smithi100 ceph-mon[39462]: 4.1b deep-scrub starts 2024-09-17T07:27:29.764 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:29 smithi100 ceph-mon[39462]: 4.1b deep-scrub ok 2024-09-17T07:27:29.764 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:29 smithi100 ceph-mon[39462]: 2.16 scrub starts 2024-09-17T07:27:29.764 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:29 smithi100 ceph-mon[39462]: 2.16 scrub ok 2024-09-17T07:27:29.764 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:29 smithi100 ceph-mon[39462]: 1.0 scrub starts 2024-09-17T07:27:29.764 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:29 smithi100 ceph-mon[39462]: 1.0 scrub ok 2024-09-17T07:27:29.793 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:27:29 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:27:29.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:29 smithi135 ceph-mon[33238]: pgmap v1211: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 631 B/s rd, 0 op/s 2024-09-17T07:27:29.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:29 smithi135 ceph-mon[33238]: 4.1b deep-scrub starts 2024-09-17T07:27:29.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:29 smithi135 ceph-mon[33238]: 4.1b deep-scrub ok 2024-09-17T07:27:29.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:29 smithi135 ceph-mon[33238]: 2.16 scrub starts 2024-09-17T07:27:29.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:29 smithi135 ceph-mon[33238]: 2.16 scrub ok 2024-09-17T07:27:29.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:29 smithi135 ceph-mon[33238]: 1.0 scrub starts 2024-09-17T07:27:29.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:29 smithi135 ceph-mon[33238]: 1.0 scrub ok 2024-09-17T07:27:30.785 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:30 smithi100 ceph-mon[29092]: 6.e deep-scrub starts 2024-09-17T07:27:30.785 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:30 smithi100 ceph-mon[29092]: 6.e deep-scrub ok 2024-09-17T07:27:30.785 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:30 smithi100 ceph-mon[29092]: pgmap v1212: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T07:27:30.786 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:30 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:27:30.786 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:30 smithi100 ceph-mon[29092]: 1.9 scrub starts 2024-09-17T07:27:30.786 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:30 smithi100 ceph-mon[29092]: 1.9 scrub ok 2024-09-17T07:27:30.786 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:30 smithi100 ceph-mon[29092]: 2.1a scrub starts 2024-09-17T07:27:30.786 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:30 smithi100 ceph-mon[29092]: 2.1a scrub ok 2024-09-17T07:27:30.786 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:30 smithi100 ceph-mon[39462]: 6.e deep-scrub starts 2024-09-17T07:27:30.786 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:30 smithi100 ceph-mon[39462]: 6.e deep-scrub ok 2024-09-17T07:27:30.786 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:30 smithi100 ceph-mon[39462]: pgmap v1212: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T07:27:30.786 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:30 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:27:30.786 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:30 smithi100 ceph-mon[39462]: 1.9 scrub starts 2024-09-17T07:27:30.787 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:30 smithi100 ceph-mon[39462]: 1.9 scrub ok 2024-09-17T07:27:30.787 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:30 smithi100 ceph-mon[39462]: 2.1a scrub starts 2024-09-17T07:27:30.787 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:30 smithi100 ceph-mon[39462]: 2.1a scrub ok 2024-09-17T07:27:30.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:30 smithi135 ceph-mon[33238]: 6.e deep-scrub starts 2024-09-17T07:27:30.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:30 smithi135 ceph-mon[33238]: 6.e deep-scrub ok 2024-09-17T07:27:30.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:30 smithi135 ceph-mon[33238]: pgmap v1212: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T07:27:30.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:30 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:27:30.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:30 smithi135 ceph-mon[33238]: 1.9 scrub starts 2024-09-17T07:27:30.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:30 smithi135 ceph-mon[33238]: 1.9 scrub ok 2024-09-17T07:27:30.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:30 smithi135 ceph-mon[33238]: 2.1a scrub starts 2024-09-17T07:27:30.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:30 smithi135 ceph-mon[33238]: 2.1a scrub ok 2024-09-17T07:27:31.041 INFO:teuthology.orchestra.run.smithi100.stdout:115964117318 2024-09-17T07:27:31.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:31 smithi100 ceph-mon[29092]: 4.2 scrub starts 2024-09-17T07:27:31.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:31 smithi100 ceph-mon[29092]: 4.2 scrub ok 2024-09-17T07:27:31.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:31 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2414698389' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-17T07:27:31.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:31 smithi100 ceph-mon[39462]: 4.2 scrub starts 2024-09-17T07:27:31.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:31 smithi100 ceph-mon[39462]: 4.2 scrub ok 2024-09-17T07:27:31.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:31 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2414698389' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-17T07:27:31.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:31 smithi135 ceph-mon[33238]: 4.2 scrub starts 2024-09-17T07:27:31.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:31 smithi135 ceph-mon[33238]: 4.2 scrub ok 2024-09-17T07:27:31.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:31 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2414698389' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-17T07:27:31.823 INFO:tasks.cephadm.ceph_manager.ceph:need seq 115964117314 got 115964117318 for osd.3 2024-09-17T07:27:31.824 DEBUG:teuthology.parallel:result is None 2024-09-17T07:27:31.824 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-17T07:27:31.824 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T07:27:32.061 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:27:32.505 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:32 smithi100 ceph-mon[39462]: 4.c scrub starts 2024-09-17T07:27:32.505 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:32 smithi100 ceph-mon[39462]: 4.c scrub ok 2024-09-17T07:27:32.506 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:32 smithi100 ceph-mon[39462]: 6.19 scrub starts 2024-09-17T07:27:32.506 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:32 smithi100 ceph-mon[39462]: 6.19 scrub ok 2024-09-17T07:27:32.506 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:32 smithi100 ceph-mon[39462]: pgmap v1213: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T07:27:32.506 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:32 smithi100 ceph-mon[39462]: 2.5 scrub starts 2024-09-17T07:27:32.506 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:32 smithi100 ceph-mon[39462]: 2.5 scrub ok 2024-09-17T07:27:32.506 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:32 smithi100 ceph-mon[29092]: 4.c scrub starts 2024-09-17T07:27:32.506 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:32 smithi100 ceph-mon[29092]: 4.c scrub ok 2024-09-17T07:27:32.506 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:32 smithi100 ceph-mon[29092]: 6.19 scrub starts 2024-09-17T07:27:32.506 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:32 smithi100 ceph-mon[29092]: 6.19 scrub ok 2024-09-17T07:27:32.506 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:32 smithi100 ceph-mon[29092]: pgmap v1213: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T07:27:32.506 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:32 smithi100 ceph-mon[29092]: 2.5 scrub starts 2024-09-17T07:27:32.507 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:32 smithi100 ceph-mon[29092]: 2.5 scrub ok 2024-09-17T07:27:32.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:32 smithi135 ceph-mon[33238]: 4.c scrub starts 2024-09-17T07:27:32.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:32 smithi135 ceph-mon[33238]: 4.c scrub ok 2024-09-17T07:27:32.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:32 smithi135 ceph-mon[33238]: 6.19 scrub starts 2024-09-17T07:27:32.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:32 smithi135 ceph-mon[33238]: 6.19 scrub ok 2024-09-17T07:27:32.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:32 smithi135 ceph-mon[33238]: pgmap v1213: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T07:27:32.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:32 smithi135 ceph-mon[33238]: 2.5 scrub starts 2024-09-17T07:27:32.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:32 smithi135 ceph-mon[33238]: 2.5 scrub ok 2024-09-17T07:27:33.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:33 smithi135 ceph-mon[33238]: 5.5 scrub starts 2024-09-17T07:27:33.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:33 smithi135 ceph-mon[33238]: 5.5 scrub ok 2024-09-17T07:27:33.876 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:33 smithi100 ceph-mon[39462]: 5.5 scrub starts 2024-09-17T07:27:33.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:33 smithi100 ceph-mon[39462]: 5.5 scrub ok 2024-09-17T07:27:33.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:33 smithi100 ceph-mon[29092]: 5.5 scrub starts 2024-09-17T07:27:33.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:33 smithi100 ceph-mon[29092]: 5.5 scrub ok 2024-09-17T07:27:34.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:34 smithi100 ceph-mon[29092]: 5.1d scrub starts 2024-09-17T07:27:34.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:34 smithi100 ceph-mon[29092]: 5.1d scrub ok 2024-09-17T07:27:34.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:34 smithi100 ceph-mon[29092]: pgmap v1214: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 922 B/s rd, 0 op/s 2024-09-17T07:27:34.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:34 smithi100 ceph-mon[29092]: 6.8 scrub starts 2024-09-17T07:27:34.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:34 smithi100 ceph-mon[29092]: 6.8 scrub ok 2024-09-17T07:27:34.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:34 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:27:34.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:34 smithi100 ceph-mon[39462]: 5.1d scrub starts 2024-09-17T07:27:34.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:34 smithi100 ceph-mon[39462]: 5.1d scrub ok 2024-09-17T07:27:34.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:34 smithi100 ceph-mon[39462]: pgmap v1214: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 922 B/s rd, 0 op/s 2024-09-17T07:27:34.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:34 smithi100 ceph-mon[39462]: 6.8 scrub starts 2024-09-17T07:27:34.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:34 smithi100 ceph-mon[39462]: 6.8 scrub ok 2024-09-17T07:27:34.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:34 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:27:34.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:34 smithi135 ceph-mon[33238]: 5.1d scrub starts 2024-09-17T07:27:34.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:34 smithi135 ceph-mon[33238]: 5.1d scrub ok 2024-09-17T07:27:34.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:34 smithi135 ceph-mon[33238]: pgmap v1214: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 922 B/s rd, 0 op/s 2024-09-17T07:27:34.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:34 smithi135 ceph-mon[33238]: 6.8 scrub starts 2024-09-17T07:27:34.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:34 smithi135 ceph-mon[33238]: 6.8 scrub ok 2024-09-17T07:27:34.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:34 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T07:27:35.321 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T07:27:35.324 INFO:teuthology.orchestra.run.smithi100.stderr:dumped all 2024-09-17T07:27:35.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:35 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:27:35.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:35 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:27:35.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:35 smithi100 ceph-mon[29092]: osdmap e547: 8 total, 8 up, 8 in 2024-09-17T07:27:35.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:35 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:27:35.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:35 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:27:35.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:35 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:27:35.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:35 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:27:35.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:35 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:27:35.627 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:35 smithi100 ceph-mon[29092]: from='client.16521 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:27:35.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:35 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:27:35.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:35 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:27:35.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:35 smithi100 ceph-mon[39462]: osdmap e547: 8 total, 8 up, 8 in 2024-09-17T07:27:35.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:35 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:27:35.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:35 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:27:35.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:35 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:27:35.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:35 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:27:35.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:35 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:27:35.628 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:35 smithi100 ceph-mon[39462]: from='client.16521 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:27:35.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:35 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:27:35.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:35 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:27:35.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:35 smithi135 ceph-mon[33238]: osdmap e547: 8 total, 8 up, 8 in 2024-09-17T07:27:35.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:35 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:27:35.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:35 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:27:35.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:35 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:27:35.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:35 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:27:35.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:35 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:27:35.794 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:35 smithi135 ceph-mon[33238]: from='client.16521 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:27:36.026 INFO:teuthology.orchestra.run.smithi100.stdout:{"pg_ready":true,"pg_map":{"version":1215,"stamp":"2024-09-17T07:27:34.511777+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596246,"num_objects":196,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":1930,"num_read_kb":1682,"num_write":1002,"num_write_kb":43,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":348,"num_bytes_recovered":597821,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2594,"ondisk_log_size":2594,"up":426,"acting":426,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":426,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":6613316,"kb_used_data":7748,"kb_used_omap":18,"kb_used_meta":6605485,"kb_avail":743347900,"statfs":{"total":767960285184,"available":761188249600,"internally_reserved":0,"allocated":7933952,"data_stored":4756434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18436,"internal_metadata":6764017660},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.946602"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":51,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.759501+0000","last_change":"2024-09-17T07:26:47.679147+0000","last_active":"2024-09-17T07:27:20.759501+0000","last_peered":"2024-09-17T07:27:20.759501+0000","last_clean":"2024-09-17T07:27:20.759501+0000","last_became_active":"2024-09-17T07:25:49.447380+0000","last_became_peered":"2024-09-17T07:25:49.447380+0000","last_unstale":"2024-09-17T07:27:20.759501+0000","last_undegraded":"2024-09-17T07:27:20.759501+0000","last_fullsized":"2024-09-17T07:27:20.759501+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:47.679054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:33.644728+0000","last_clean_scrub_stamp":"2024-09-17T07:26:47.679054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:52.196709+0000","scrub_duration":0.001605719,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":627,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:19.462180+0000","last_change":"2024-09-17T07:27:00.632990+0000","last_active":"2024-09-17T07:27:19.462180+0000","last_peered":"2024-09-17T07:27:19.462180+0000","last_clean":"2024-09-17T07:27:19.462180+0000","last_became_active":"2024-09-17T07:22:13.589527+0000","last_became_peered":"2024-09-17T07:22:13.589527+0000","last_unstale":"2024-09-17T07:27:19.462180+0000","last_undegraded":"2024-09-17T07:27:19.462180+0000","last_fullsized":"2024-09-17T07:27:19.462180+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:27:00.632895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:27:00.632895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:07.134925+0000","scrub_duration":0.0027346559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"439'39","reported_seq":67,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.759323+0000","last_change":"2024-09-17T07:26:24.892510+0000","last_active":"2024-09-17T07:27:20.759323+0000","last_peered":"2024-09-17T07:27:20.759323+0000","last_clean":"2024-09-17T07:27:20.759323+0000","last_became_active":"2024-09-17T07:25:49.442905+0000","last_became_peered":"2024-09-17T07:25:49.442905+0000","last_unstale":"2024-09-17T07:27:20.759323+0000","last_undegraded":"2024-09-17T07:27:20.759323+0000","last_fullsized":"2024-09-17T07:27:20.759323+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'39","last_scrub_stamp":"2024-09-17T07:26:24.892381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:26:24.892381+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:53.552454+0000","scrub_duration":0.0030756160000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":587,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:19.462404+0000","last_change":"2024-09-17T07:26:22.887078+0000","last_active":"2024-09-17T07:27:19.462404+0000","last_peered":"2024-09-17T07:27:19.462404+0000","last_clean":"2024-09-17T07:27:19.462404+0000","last_became_active":"2024-09-17T07:21:15.354561+0000","last_became_peered":"2024-09-17T07:21:15.354561+0000","last_unstale":"2024-09-17T07:27:19.462404+0000","last_undegraded":"2024-09-17T07:27:19.462404+0000","last_fullsized":"2024-09-17T07:27:19.462404+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:22.886984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:26:22.886984+0000","last_clean_scrub_stamp":"2024-09-17T07:26:22.886984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:27.103979+0000","scrub_duration":0.0014775470000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":51,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.759663+0000","last_change":"2024-09-17T07:26:58.718920+0000","last_active":"2024-09-17T07:27:20.759663+0000","last_peered":"2024-09-17T07:27:20.759663+0000","last_clean":"2024-09-17T07:27:20.759663+0000","last_became_active":"2024-09-17T07:25:49.452275+0000","last_became_peered":"2024-09-17T07:25:49.452275+0000","last_unstale":"2024-09-17T07:27:20.759663+0000","last_undegraded":"2024-09-17T07:27:20.759663+0000","last_fullsized":"2024-09-17T07:27:20.759663+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:58.718845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:17:35.507742+0000","last_clean_scrub_stamp":"2024-09-17T07:26:58.718845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:04.607009+0000","scrub_duration":0.0014655849999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"438'21","reported_seq":635,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:19.462339+0000","last_change":"2024-09-17T07:26:17.884904+0000","last_active":"2024-09-17T07:27:19.462339+0000","last_peered":"2024-09-17T07:27:19.462339+0000","last_clean":"2024-09-17T07:27:19.462339+0000","last_became_active":"2024-09-17T07:21:15.354718+0000","last_became_peered":"2024-09-17T07:21:15.354718+0000","last_unstale":"2024-09-17T07:27:19.462339+0000","last_undegraded":"2024-09-17T07:27:19.462339+0000","last_fullsized":"2024-09-17T07:27:19.462339+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"438'21","last_scrub_stamp":"2024-09-17T07:26:17.884799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:26:17.884799+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001882175,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":480,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:19.461547+0000","last_change":"2024-09-17T07:26:55.184002+0000","last_active":"2024-09-17T07:27:19.461547+0000","last_peered":"2024-09-17T07:27:19.461547+0000","last_clean":"2024-09-17T07:27:19.461547+0000","last_became_active":"2024-09-17T07:23:57.473620+0000","last_became_peered":"2024-09-17T07:23:57.473620+0000","last_unstale":"2024-09-17T07:27:19.461547+0000","last_undegraded":"2024-09-17T07:27:19.461547+0000","last_fullsized":"2024-09-17T07:27:19.461547+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:55.183874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:26:55.183874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:55.457020+0000","scrub_duration":0.0020334110000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":612,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:19.462500+0000","last_change":"2024-09-17T07:27:18.747768+0000","last_active":"2024-09-17T07:27:19.462500+0000","last_peered":"2024-09-17T07:27:19.462500+0000","last_clean":"2024-09-17T07:27:19.462500+0000","last_became_active":"2024-09-17T07:23:57.482145+0000","last_became_peered":"2024-09-17T07:23:57.482145+0000","last_unstale":"2024-09-17T07:27:19.462500+0000","last_undegraded":"2024-09-17T07:27:19.462500+0000","last_fullsized":"2024-09-17T07:27:19.462500+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:27:18.747673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:27:18.747673+0000","last_clean_scrub_stamp":"2024-09-17T07:27:18.747673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:45.594419+0000","scrub_duration":0.001480063,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":156,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.387569+0000","last_change":"2024-09-17T07:27:13.727276+0000","last_active":"2024-09-17T07:27:20.387569+0000","last_peered":"2024-09-17T07:27:20.387569+0000","last_clean":"2024-09-17T07:27:20.387569+0000","last_became_active":"2024-09-17T07:22:13.615512+0000","last_became_peered":"2024-09-17T07:22:13.615512+0000","last_unstale":"2024-09-17T07:27:20.387569+0000","last_undegraded":"2024-09-17T07:27:20.387569+0000","last_fullsized":"2024-09-17T07:27:20.387569+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:27:13.727174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:46.312371+0000","last_clean_scrub_stamp":"2024-09-17T07:27:13.727174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:32.374707+0000","scrub_duration":0.0015053009999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"488'11","reported_seq":651,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:27.699074+0000","last_change":"2024-09-17T07:27:27.699193+0000","last_active":"2024-09-17T07:27:27.699074+0000","last_peered":"2024-09-17T07:27:27.699074+0000","last_clean":"2024-09-17T07:27:27.699074+0000","last_became_active":"2024-09-17T07:25:49.436886+0000","last_became_peered":"2024-09-17T07:25:49.436886+0000","last_unstale":"2024-09-17T07:27:27.699074+0000","last_undegraded":"2024-09-17T07:27:27.699074+0000","last_fullsized":"2024-09-17T07:27:27.699074+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"488'11","last_scrub_stamp":"2024-09-17T07:27:27.699053+0000","last_deep_scrub":"488'11","last_deep_scrub_stamp":"2024-09-17T07:27:27.699053+0000","last_clean_scrub_stamp":"2024-09-17T07:27:27.699053+0000","objects_scrubbed":2,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:52.451659+0000","scrub_duration":0.0022887570000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":13,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":145,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.387028+0000","last_change":"2024-09-17T07:27:06.698942+0000","last_active":"2024-09-17T07:27:20.387028+0000","last_peered":"2024-09-17T07:27:20.387028+0000","last_clean":"2024-09-17T07:27:20.387028+0000","last_became_active":"2024-09-17T07:22:13.624441+0000","last_became_peered":"2024-09-17T07:22:13.624441+0000","last_unstale":"2024-09-17T07:27:20.387028+0000","last_undegraded":"2024-09-17T07:27:20.387028+0000","last_fullsized":"2024-09-17T07:27:20.387028+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:27:06.698846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:27:06.698846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:21.096359+0000","scrub_duration":0.001506154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":560,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.806675+0000","last_change":"2024-09-17T07:26:01.082939+0000","last_active":"2024-09-17T07:27:20.806675+0000","last_peered":"2024-09-17T07:27:20.806675+0000","last_clean":"2024-09-17T07:27:20.806675+0000","last_became_active":"2024-09-17T07:25:49.438958+0000","last_became_peered":"2024-09-17T07:25:49.438958+0000","last_unstale":"2024-09-17T07:27:20.806675+0000","last_undegraded":"2024-09-17T07:27:20.806675+0000","last_fullsized":"2024-09-17T07:27:20.806675+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:01.082822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:24:24.811522+0000","last_clean_scrub_stamp":"2024-09-17T07:26:01.082822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:29.377375+0000","scrub_duration":0.001607477,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":171,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.387458+0000","last_change":"2024-09-17T07:27:15.776974+0000","last_active":"2024-09-17T07:27:20.387458+0000","last_peered":"2024-09-17T07:27:20.387458+0000","last_clean":"2024-09-17T07:27:20.387458+0000","last_became_active":"2024-09-17T07:23:57.476413+0000","last_became_peered":"2024-09-17T07:23:57.476413+0000","last_unstale":"2024-09-17T07:27:20.387458+0000","last_undegraded":"2024-09-17T07:27:20.387458+0000","last_fullsized":"2024-09-17T07:27:20.387458+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:27:15.776862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:45.029247+0000","last_clean_scrub_stamp":"2024-09-17T07:27:15.776862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:34.753312+0000","scrub_duration":0.0014678950000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"439'25","reported_seq":682,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:19.462821+0000","last_change":"2024-09-17T07:27:02.766819+0000","last_active":"2024-09-17T07:27:19.462821+0000","last_peered":"2024-09-17T07:27:19.462821+0000","last_clean":"2024-09-17T07:27:19.462821+0000","last_became_active":"2024-09-17T07:25:49.438036+0000","last_became_peered":"2024-09-17T07:25:49.438036+0000","last_unstale":"2024-09-17T07:27:19.462821+0000","last_undegraded":"2024-09-17T07:27:19.462821+0000","last_fullsized":"2024-09-17T07:27:19.462821+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'25","last_scrub_stamp":"2024-09-17T07:27:02.766714+0000","last_deep_scrub":"439'25","last_deep_scrub_stamp":"2024-09-17T07:27:02.766714+0000","last_clean_scrub_stamp":"2024-09-17T07:27:02.766714+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:08.741846+0000","scrub_duration":0.0036329579999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":588,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.807939+0000","last_change":"2024-09-17T07:26:32.412814+0000","last_active":"2024-09-17T07:27:20.807939+0000","last_peered":"2024-09-17T07:27:20.807939+0000","last_clean":"2024-09-17T07:27:20.807939+0000","last_became_active":"2024-09-17T07:23:57.480812+0000","last_became_peered":"2024-09-17T07:23:57.480812+0000","last_unstale":"2024-09-17T07:27:20.807939+0000","last_undegraded":"2024-09-17T07:27:20.807939+0000","last_fullsized":"2024-09-17T07:27:20.807939+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:32.412696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:28.096266+0000","last_clean_scrub_stamp":"2024-09-17T07:26:32.412696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:55.638791+0000","scrub_duration":0.0017410769999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":92,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:19.461686+0000","last_change":"2024-09-17T07:26:52.630335+0000","last_active":"2024-09-17T07:27:19.461686+0000","last_peered":"2024-09-17T07:27:19.461686+0000","last_clean":"2024-09-17T07:27:19.461686+0000","last_became_active":"2024-09-17T07:24:20.409247+0000","last_became_peered":"2024-09-17T07:24:20.409247+0000","last_unstale":"2024-09-17T07:27:19.461686+0000","last_undegraded":"2024-09-17T07:27:19.461686+0000","last_fullsized":"2024-09-17T07:27:19.461686+0000","mapping_epoch":487,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":488,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:52.630258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:26:52.630258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:54.132579+0000","scrub_duration":0.0015908739999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.8","version":"212'130","reported_seq":662,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.806489+0000","last_change":"2024-09-17T07:27:01.692690+0000","last_active":"2024-09-17T07:27:20.806489+0000","last_peered":"2024-09-17T07:27:20.806489+0000","last_clean":"2024-09-17T07:27:20.806489+0000","last_became_active":"2024-09-17T07:25:52.441842+0000","last_became_peered":"2024-09-17T07:25:52.441842+0000","last_unstale":"2024-09-17T07:27:20.806489+0000","last_undegraded":"2024-09-17T07:27:20.806489+0000","last_fullsized":"2024-09-17T07:27:20.806489+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":4,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:27:01.692584+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:14:57.052505+0000","last_clean_scrub_stamp":"2024-09-17T07:27:01.692584+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:29.736535+0000","scrub_duration":0.0023312020000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"439'53","reported_seq":679,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:19.463145+0000","last_change":"2024-09-17T07:26:19.864456+0000","last_active":"2024-09-17T07:27:19.463145+0000","last_peered":"2024-09-17T07:27:19.463145+0000","last_clean":"2024-09-17T07:27:19.463145+0000","last_became_active":"2024-09-17T07:21:15.348413+0000","last_became_peered":"2024-09-17T07:21:15.348413+0000","last_unstale":"2024-09-17T07:27:19.463145+0000","last_undegraded":"2024-09-17T07:27:19.463145+0000","last_fullsized":"2024-09-17T07:27:19.463145+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'53","last_scrub_stamp":"2024-09-17T07:26:19.864349+0000","last_deep_scrub":"95'25","last_deep_scrub_stamp":"2024-09-17T07:04:09.450039+0000","last_clean_scrub_stamp":"2024-09-17T07:26:19.864349+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:47.815962+0000","scrub_duration":0.0045487909999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":61,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":160,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.387488+0000","last_change":"2024-09-17T07:26:37.758537+0000","last_active":"2024-09-17T07:27:20.387488+0000","last_peered":"2024-09-17T07:27:20.387488+0000","last_clean":"2024-09-17T07:27:20.387488+0000","last_became_active":"2024-09-17T07:23:57.475614+0000","last_became_peered":"2024-09-17T07:23:57.475614+0000","last_unstale":"2024-09-17T07:27:20.387488+0000","last_undegraded":"2024-09-17T07:27:20.387488+0000","last_fullsized":"2024-09-17T07:27:20.387488+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:37.758441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:26:37.758441+0000","last_clean_scrub_stamp":"2024-09-17T07:26:37.758441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:45.056906+0000","scrub_duration":0.00139802,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":419,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:19.462677+0000","last_change":"2024-09-17T07:26:14.280048+0000","last_active":"2024-09-17T07:27:19.462677+0000","last_peered":"2024-09-17T07:27:19.462677+0000","last_clean":"2024-09-17T07:27:19.462677+0000","last_became_active":"2024-09-17T07:21:15.349538+0000","last_became_peered":"2024-09-17T07:21:15.349538+0000","last_unstale":"2024-09-17T07:27:19.462677+0000","last_undegraded":"2024-09-17T07:27:19.462677+0000","last_fullsized":"2024-09-17T07:27:19.462677+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:14.279969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:26:14.279969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:36.505935+0000","scrub_duration":0.001479937,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":86,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.698898+0000","last_change":"2024-09-17T07:27:04.983031+0000","last_active":"2024-09-17T07:27:20.698898+0000","last_peered":"2024-09-17T07:27:20.698898+0000","last_clean":"2024-09-17T07:27:20.698898+0000","last_became_active":"2024-09-17T07:26:09.790191+0000","last_became_peered":"2024-09-17T07:26:09.790191+0000","last_unstale":"2024-09-17T07:27:20.698898+0000","last_undegraded":"2024-09-17T07:27:20.698898+0000","last_fullsized":"2024-09-17T07:27:20.698898+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:27:04.982928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:42.984088+0000","last_clean_scrub_stamp":"2024-09-17T07:27:04.982928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:29.445747+0000","scrub_duration":0.001869012,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"212'130","reported_seq":833,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:29.860997+0000","last_change":"2024-09-17T07:27:29.861090+0000","last_active":"2024-09-17T07:27:29.860997+0000","last_peered":"2024-09-17T07:27:29.860997+0000","last_clean":"2024-09-17T07:27:29.860997+0000","last_became_active":"2024-09-17T07:25:52.441833+0000","last_became_peered":"2024-09-17T07:25:52.441833+0000","last_unstale":"2024-09-17T07:27:29.860997+0000","last_undegraded":"2024-09-17T07:27:29.860997+0000","last_fullsized":"2024-09-17T07:27:29.860997+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":4,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:27:29.860982+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:14:57.052505+0000","last_clean_scrub_stamp":"2024-09-17T07:27:29.860982+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:34.518223+0000","scrub_duration":0.0016031909999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"439'35","reported_seq":705,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:30.777895+0000","last_change":"2024-09-17T07:27:30.777996+0000","last_active":"2024-09-17T07:27:30.777895+0000","last_peered":"2024-09-17T07:27:30.777895+0000","last_clean":"2024-09-17T07:27:30.777895+0000","last_became_active":"2024-09-17T07:25:49.440859+0000","last_became_peered":"2024-09-17T07:25:49.440859+0000","last_unstale":"2024-09-17T07:27:30.777895+0000","last_undegraded":"2024-09-17T07:27:30.777895+0000","last_fullsized":"2024-09-17T07:27:30.777895+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'35","last_scrub_stamp":"2024-09-17T07:27:30.777879+0000","last_deep_scrub":"439'35","last_deep_scrub_stamp":"2024-09-17T07:24:29.750888+0000","last_clean_scrub_stamp":"2024-09-17T07:27:30.777879+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:40.637971+0000","scrub_duration":0.0023317020000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":487,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:19.462835+0000","last_change":"2024-09-17T07:26:33.371802+0000","last_active":"2024-09-17T07:27:19.462835+0000","last_peered":"2024-09-17T07:27:19.462835+0000","last_clean":"2024-09-17T07:27:19.462835+0000","last_became_active":"2024-09-17T07:25:49.433544+0000","last_became_peered":"2024-09-17T07:25:49.433544+0000","last_unstale":"2024-09-17T07:27:19.462835+0000","last_undegraded":"2024-09-17T07:27:19.462835+0000","last_fullsized":"2024-09-17T07:27:19.462835+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:33.371718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:26:33.371718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:42.550790+0000","scrub_duration":0.001869297,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"525'14","reported_seq":137,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:25.805082+0000","last_change":"2024-09-17T07:27:09.886301+0000","last_active":"2024-09-17T07:27:25.805082+0000","last_peered":"2024-09-17T07:27:25.805082+0000","last_clean":"2024-09-17T07:27:25.805082+0000","last_became_active":"2024-09-17T07:26:09.601082+0000","last_became_peered":"2024-09-17T07:26:09.601082+0000","last_unstale":"2024-09-17T07:27:25.805082+0000","last_undegraded":"2024-09-17T07:27:25.805082+0000","last_fullsized":"2024-09-17T07:27:25.805082+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"525'14","last_scrub_stamp":"2024-09-17T07:27:09.886216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:27:09.886216+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:14.177273+0000","scrub_duration":0.0019761729999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":610,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:28.731701+0000","last_change":"2024-09-17T07:27:28.731828+0000","last_active":"2024-09-17T07:27:28.731701+0000","last_peered":"2024-09-17T07:27:28.731701+0000","last_clean":"2024-09-17T07:27:28.731701+0000","last_became_active":"2024-09-17T07:21:15.348799+0000","last_became_peered":"2024-09-17T07:21:15.348799+0000","last_unstale":"2024-09-17T07:27:28.731701+0000","last_undegraded":"2024-09-17T07:27:28.731701+0000","last_fullsized":"2024-09-17T07:27:28.731701+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:27:28.731655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:27:28.731655+0000","last_clean_scrub_stamp":"2024-09-17T07:27:28.731655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:53.482697+0000","scrub_duration":0.001669547,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"212'130","reported_seq":642,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.807258+0000","last_change":"2024-09-17T07:26:51.488736+0000","last_active":"2024-09-17T07:27:20.807258+0000","last_peered":"2024-09-17T07:27:20.807258+0000","last_clean":"2024-09-17T07:27:20.807258+0000","last_became_active":"2024-09-17T07:23:57.494014+0000","last_became_peered":"2024-09-17T07:23:57.494014+0000","last_unstale":"2024-09-17T07:27:20.807258+0000","last_undegraded":"2024-09-17T07:27:20.807258+0000","last_fullsized":"2024-09-17T07:27:20.807258+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":4,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:26:51.488609+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:14:57.052505+0000","last_clean_scrub_stamp":"2024-09-17T07:26:51.488609+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:04.490841+0000","scrub_duration":0.0016059659999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"439'39","reported_seq":576,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:19.463160+0000","last_change":"2024-09-17T07:26:44.253243+0000","last_active":"2024-09-17T07:27:19.463160+0000","last_peered":"2024-09-17T07:27:19.463160+0000","last_clean":"2024-09-17T07:27:19.463160+0000","last_became_active":"2024-09-17T07:25:49.433970+0000","last_became_peered":"2024-09-17T07:25:49.433970+0000","last_unstale":"2024-09-17T07:27:19.463160+0000","last_undegraded":"2024-09-17T07:27:19.463160+0000","last_fullsized":"2024-09-17T07:27:19.463160+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'39","last_scrub_stamp":"2024-09-17T07:26:44.253156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:26:44.253156+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:46.559449+0000","scrub_duration":0.0025313750000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":486,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:19.461974+0000","last_change":"2024-09-17T07:26:56.154343+0000","last_active":"2024-09-17T07:27:19.461974+0000","last_peered":"2024-09-17T07:27:19.461974+0000","last_clean":"2024-09-17T07:27:19.461974+0000","last_became_active":"2024-09-17T07:25:49.433837+0000","last_became_peered":"2024-09-17T07:25:49.433837+0000","last_unstale":"2024-09-17T07:27:19.461974+0000","last_undegraded":"2024-09-17T07:27:19.461974+0000","last_fullsized":"2024-09-17T07:27:19.461974+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:56.154239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:26:56.154239+0000","last_clean_scrub_stamp":"2024-09-17T07:26:56.154239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:13.090040+0000","scrub_duration":0.0017523560000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"478'10","reported_seq":923,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:25.805238+0000","last_change":"2024-09-17T07:26:30.851481+0000","last_active":"2024-09-17T07:27:25.805238+0000","last_peered":"2024-09-17T07:27:25.805238+0000","last_clean":"2024-09-17T07:27:25.805238+0000","last_became_active":"2024-09-17T07:23:57.473239+0000","last_became_peered":"2024-09-17T07:23:57.473239+0000","last_unstale":"2024-09-17T07:27:25.805238+0000","last_undegraded":"2024-09-17T07:27:25.805238+0000","last_fullsized":"2024-09-17T07:27:25.805238+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"478'10","last_scrub_stamp":"2024-09-17T07:26:30.851356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:26:30.851356+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:38.837398+0000","scrub_duration":0.0019091570000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":553,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:22.708772+0000","last_change":"2024-09-17T07:27:22.708842+0000","last_active":"2024-09-17T07:27:22.708772+0000","last_peered":"2024-09-17T07:27:22.708772+0000","last_clean":"2024-09-17T07:27:22.708772+0000","last_became_active":"2024-09-17T07:23:57.522025+0000","last_became_peered":"2024-09-17T07:23:57.522025+0000","last_unstale":"2024-09-17T07:27:22.708772+0000","last_undegraded":"2024-09-17T07:27:22.708772+0000","last_fullsized":"2024-09-17T07:27:22.708772+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:27:22.708760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:27:22.708760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:51.700304+0000","scrub_duration":0.0071301660000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"488'32","reported_seq":661,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:19.463149+0000","last_change":"2024-09-17T07:27:11.857770+0000","last_active":"2024-09-17T07:27:19.463149+0000","last_peered":"2024-09-17T07:27:19.463149+0000","last_clean":"2024-09-17T07:27:19.463149+0000","last_became_active":"2024-09-17T07:21:15.347662+0000","last_became_peered":"2024-09-17T07:21:15.347662+0000","last_unstale":"2024-09-17T07:27:19.463149+0000","last_undegraded":"2024-09-17T07:27:19.463149+0000","last_fullsized":"2024-09-17T07:27:19.463149+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"488'32","last_scrub_stamp":"2024-09-17T07:27:11.857602+0000","last_deep_scrub":"488'32","last_deep_scrub_stamp":"2024-09-17T07:24:27.785661+0000","last_clean_scrub_stamp":"2024-09-17T07:27:11.857602+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:30.330321+0000","scrub_duration":0.0022297979999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":145,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.386986+0000","last_change":"2024-09-17T07:26:46.806517+0000","last_active":"2024-09-17T07:27:20.386986+0000","last_peered":"2024-09-17T07:27:20.386986+0000","last_clean":"2024-09-17T07:27:20.386986+0000","last_became_active":"2024-09-17T07:22:13.709873+0000","last_became_peered":"2024-09-17T07:22:13.709873+0000","last_unstale":"2024-09-17T07:27:20.386986+0000","last_undegraded":"2024-09-17T07:27:20.386986+0000","last_fullsized":"2024-09-17T07:27:20.386986+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:46.806317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:26:46.806317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:04.395492+0000","scrub_duration":0.0019359889999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":564,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.807535+0000","last_change":"2024-09-17T07:26:41.450442+0000","last_active":"2024-09-17T07:27:20.807535+0000","last_peered":"2024-09-17T07:27:20.807535+0000","last_clean":"2024-09-17T07:27:20.807535+0000","last_became_active":"2024-09-17T07:23:57.478625+0000","last_became_peered":"2024-09-17T07:23:57.478625+0000","last_unstale":"2024-09-17T07:27:20.807535+0000","last_undegraded":"2024-09-17T07:27:20.807535+0000","last_fullsized":"2024-09-17T07:27:20.807535+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:41.450346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:26:41.450346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:41.999320+0000","scrub_duration":0.0013943549999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":51,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.759649+0000","last_change":"2024-09-17T07:27:06.751693+0000","last_active":"2024-09-17T07:27:20.759649+0000","last_peered":"2024-09-17T07:27:20.759649+0000","last_clean":"2024-09-17T07:27:20.759649+0000","last_became_active":"2024-09-17T07:25:49.453609+0000","last_became_peered":"2024-09-17T07:25:49.453609+0000","last_unstale":"2024-09-17T07:27:20.759649+0000","last_undegraded":"2024-09-17T07:27:20.759649+0000","last_fullsized":"2024-09-17T07:27:20.759649+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:27:06.751600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:13:22.187127+0000","last_clean_scrub_stamp":"2024-09-17T07:27:06.751600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:19.478751+0000","scrub_duration":0.001873825,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"212'130","reported_seq":685,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.758865+0000","last_change":"2024-09-17T07:26:53.768933+0000","last_active":"2024-09-17T07:27:20.758865+0000","last_peered":"2024-09-17T07:27:20.758865+0000","last_clean":"2024-09-17T07:27:20.758865+0000","last_became_active":"2024-09-17T07:25:52.440509+0000","last_became_peered":"2024-09-17T07:25:52.440509+0000","last_unstale":"2024-09-17T07:27:20.758865+0000","last_undegraded":"2024-09-17T07:27:20.758865+0000","last_fullsized":"2024-09-17T07:27:20.758865+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":3,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:26:53.768856+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:25:30.132032+0000","last_clean_scrub_stamp":"2024-09-17T07:26:53.768856+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:02.795912+0000","scrub_duration":0.002184283,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"439'42","reported_seq":675,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:19.462551+0000","last_change":"2024-09-17T07:26:46.733339+0000","last_active":"2024-09-17T07:27:19.462551+0000","last_peered":"2024-09-17T07:27:19.462551+0000","last_clean":"2024-09-17T07:27:19.462551+0000","last_became_active":"2024-09-17T07:23:57.481780+0000","last_became_peered":"2024-09-17T07:23:57.481780+0000","last_unstale":"2024-09-17T07:27:19.462551+0000","last_undegraded":"2024-09-17T07:27:19.462551+0000","last_fullsized":"2024-09-17T07:27:19.462551+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'42","last_scrub_stamp":"2024-09-17T07:26:46.733234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:26:46.733234+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:10.663281+0000","scrub_duration":0.0051770690000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":63,"num_read_kb":56,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":669,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.807906+0000","last_change":"2024-09-17T07:26:40.817536+0000","last_active":"2024-09-17T07:27:20.807906+0000","last_peered":"2024-09-17T07:27:20.807906+0000","last_clean":"2024-09-17T07:27:20.807906+0000","last_became_active":"2024-09-17T07:22:13.582096+0000","last_became_peered":"2024-09-17T07:22:13.582096+0000","last_unstale":"2024-09-17T07:27:20.807906+0000","last_undegraded":"2024-09-17T07:27:20.807906+0000","last_fullsized":"2024-09-17T07:27:20.807906+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:40.817369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:09:23.181028+0000","last_clean_scrub_stamp":"2024-09-17T07:26:40.817369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:02.477887+0000","scrub_duration":0.0016035719999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"516'12","reported_seq":77,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:25.805151+0000","last_change":"2024-09-17T07:26:40.696026+0000","last_active":"2024-09-17T07:27:25.805151+0000","last_peered":"2024-09-17T07:27:25.805151+0000","last_clean":"2024-09-17T07:27:25.805151+0000","last_became_active":"2024-09-17T07:25:49.445884+0000","last_became_peered":"2024-09-17T07:25:49.445884+0000","last_unstale":"2024-09-17T07:27:25.805151+0000","last_undegraded":"2024-09-17T07:27:25.805151+0000","last_fullsized":"2024-09-17T07:27:25.805151+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'12","last_scrub_stamp":"2024-09-17T07:26:40.695890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:26:40.695890+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:46.413180+0000","scrub_duration":0.0019653729999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":146,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.387383+0000","last_change":"2024-09-17T07:26:28.772744+0000","last_active":"2024-09-17T07:27:20.387383+0000","last_peered":"2024-09-17T07:27:20.387383+0000","last_clean":"2024-09-17T07:27:20.387383+0000","last_became_active":"2024-09-17T07:22:13.568656+0000","last_became_peered":"2024-09-17T07:22:13.568656+0000","last_unstale":"2024-09-17T07:27:20.387383+0000","last_undegraded":"2024-09-17T07:27:20.387383+0000","last_fullsized":"2024-09-17T07:27:20.387383+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:28.772597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:10:33.727472+0000","last_clean_scrub_stamp":"2024-09-17T07:26:28.772597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:42.854687+0000","scrub_duration":0.002637688,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"212'130","reported_seq":664,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:19.462699+0000","last_change":"2024-09-17T07:27:03.718296+0000","last_active":"2024-09-17T07:27:19.462699+0000","last_peered":"2024-09-17T07:27:19.462699+0000","last_clean":"2024-09-17T07:27:19.462699+0000","last_became_active":"2024-09-17T07:19:35.420337+0000","last_became_peered":"2024-09-17T07:19:35.420337+0000","last_unstale":"2024-09-17T07:27:19.462699+0000","last_undegraded":"2024-09-17T07:27:19.462699+0000","last_fullsized":"2024-09-17T07:27:19.462699+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":3,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:27:03.718182+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:27:03.718182+0000","last_clean_scrub_stamp":"2024-09-17T07:27:03.718182+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:17.755491+0000","scrub_duration":0.001587006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"439'42","reported_seq":69,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.759061+0000","last_change":"2024-09-17T07:26:59.680403+0000","last_active":"2024-09-17T07:27:20.759061+0000","last_peered":"2024-09-17T07:27:20.759061+0000","last_clean":"2024-09-17T07:27:20.759061+0000","last_became_active":"2024-09-17T07:25:49.446027+0000","last_became_peered":"2024-09-17T07:25:49.446027+0000","last_unstale":"2024-09-17T07:27:20.759061+0000","last_undegraded":"2024-09-17T07:27:20.759061+0000","last_fullsized":"2024-09-17T07:27:20.759061+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'42","last_scrub_stamp":"2024-09-17T07:26:59.680306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:26:59.680306+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:26.162364+0000","scrub_duration":0.0022565490000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":67,"num_read_kb":55,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":473,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:19.462963+0000","last_change":"2024-09-17T07:26:25.361119+0000","last_active":"2024-09-17T07:27:19.462963+0000","last_peered":"2024-09-17T07:27:19.462963+0000","last_clean":"2024-09-17T07:27:19.462963+0000","last_became_active":"2024-09-17T07:21:15.347867+0000","last_became_peered":"2024-09-17T07:21:15.347867+0000","last_unstale":"2024-09-17T07:27:19.462963+0000","last_undegraded":"2024-09-17T07:27:19.462963+0000","last_fullsized":"2024-09-17T07:27:19.462963+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:25.361017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:56.848689+0000","last_clean_scrub_stamp":"2024-09-17T07:26:25.361017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:44.285868+0000","scrub_duration":0.0039265189999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":642,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:21.810674+0000","last_change":"2024-09-17T07:27:21.810674+0000","last_active":"2024-09-17T07:27:21.810674+0000","last_peered":"2024-09-17T07:27:21.810674+0000","last_clean":"2024-09-17T07:27:21.810674+0000","last_became_active":"2024-09-17T07:25:49.437088+0000","last_became_peered":"2024-09-17T07:25:49.437088+0000","last_unstale":"2024-09-17T07:27:21.810674+0000","last_undegraded":"2024-09-17T07:27:21.810674+0000","last_fullsized":"2024-09-17T07:27:21.810674+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:27:21.810576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:27:21.810576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:49.511111+0000","scrub_duration":0.0015495280000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":611,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:22.474565+0000","last_change":"2024-09-17T07:26:56.655814+0000","last_active":"2024-09-17T07:27:22.474565+0000","last_peered":"2024-09-17T07:27:22.474565+0000","last_clean":"2024-09-17T07:27:22.474565+0000","last_became_active":"2024-09-17T07:25:49.437860+0000","last_became_peered":"2024-09-17T07:25:49.437860+0000","last_unstale":"2024-09-17T07:27:22.474565+0000","last_undegraded":"2024-09-17T07:27:22.474565+0000","last_fullsized":"2024-09-17T07:27:22.474565+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:56.655708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:26:56.655708+0000","last_clean_scrub_stamp":"2024-09-17T07:26:56.655708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:10.694032+0000","scrub_duration":0.0017231519999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"212'130","reported_seq":689,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.806335+0000","last_change":"2024-09-17T07:26:34.581433+0000","last_active":"2024-09-17T07:27:20.806335+0000","last_peered":"2024-09-17T07:27:20.806335+0000","last_clean":"2024-09-17T07:27:20.806335+0000","last_became_active":"2024-09-17T07:25:52.443672+0000","last_became_peered":"2024-09-17T07:25:52.443672+0000","last_unstale":"2024-09-17T07:27:20.806335+0000","last_undegraded":"2024-09-17T07:27:20.806335+0000","last_fullsized":"2024-09-17T07:27:20.806335+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":3,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:26:34.581294+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:14:57.052505+0000","last_clean_scrub_stamp":"2024-09-17T07:26:34.581294+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:00.904703+0000","scrub_duration":0.0032320320000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"439'39","reported_seq":743,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:19.462275+0000","last_change":"2024-09-17T07:26:25.758254+0000","last_active":"2024-09-17T07:27:19.462275+0000","last_peered":"2024-09-17T07:27:19.462275+0000","last_clean":"2024-09-17T07:27:19.462275+0000","last_became_active":"2024-09-17T07:23:57.497347+0000","last_became_peered":"2024-09-17T07:23:57.497347+0000","last_unstale":"2024-09-17T07:27:19.462275+0000","last_undegraded":"2024-09-17T07:27:19.462275+0000","last_fullsized":"2024-09-17T07:27:19.462275+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'39","last_scrub_stamp":"2024-09-17T07:26:25.758160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:26:25.758160+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:42.032625+0000","scrub_duration":0.002729085,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":156,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:31.791183+0000","last_change":"2024-09-17T07:27:31.791267+0000","last_active":"2024-09-17T07:27:31.791183+0000","last_peered":"2024-09-17T07:27:31.791183+0000","last_clean":"2024-09-17T07:27:31.791183+0000","last_became_active":"2024-09-17T07:22:13.624802+0000","last_became_peered":"2024-09-17T07:22:13.624802+0000","last_unstale":"2024-09-17T07:27:31.791183+0000","last_undegraded":"2024-09-17T07:27:31.791183+0000","last_fullsized":"2024-09-17T07:27:31.791183+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:27:31.791169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:47.631387+0000","last_clean_scrub_stamp":"2024-09-17T07:27:31.791169+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:39.038078+0000","scrub_duration":0.0014848870000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":641,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.806931+0000","last_change":"2024-09-17T07:27:16.181839+0000","last_active":"2024-09-17T07:27:20.806931+0000","last_peered":"2024-09-17T07:27:20.806931+0000","last_clean":"2024-09-17T07:27:20.806931+0000","last_became_active":"2024-09-17T07:23:57.485512+0000","last_became_peered":"2024-09-17T07:23:57.485512+0000","last_unstale":"2024-09-17T07:27:20.806931+0000","last_undegraded":"2024-09-17T07:27:20.806931+0000","last_fullsized":"2024-09-17T07:27:20.806931+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:27:16.181735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:11:17.583712+0000","last_clean_scrub_stamp":"2024-09-17T07:27:16.181735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:45.701118+0000","scrub_duration":0.001423725,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":429,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:22.709765+0000","last_change":"2024-09-17T07:26:09.234503+0000","last_active":"2024-09-17T07:27:22.709765+0000","last_peered":"2024-09-17T07:27:22.709765+0000","last_clean":"2024-09-17T07:27:22.709765+0000","last_became_active":"2024-09-17T07:21:15.351524+0000","last_became_peered":"2024-09-17T07:21:15.351524+0000","last_unstale":"2024-09-17T07:27:22.709765+0000","last_undegraded":"2024-09-17T07:27:22.709765+0000","last_fullsized":"2024-09-17T07:27:22.709765+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:09.234406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:26:09.234406+0000","last_clean_scrub_stamp":"2024-09-17T07:26:09.234406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:32.854562+0000","scrub_duration":0.0028525680000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"212'130","reported_seq":754,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.806309+0000","last_change":"2024-09-17T07:26:42.815660+0000","last_active":"2024-09-17T07:27:20.806309+0000","last_peered":"2024-09-17T07:27:20.806309+0000","last_clean":"2024-09-17T07:27:20.806309+0000","last_became_active":"2024-09-17T07:25:52.440902+0000","last_became_peered":"2024-09-17T07:25:52.440902+0000","last_unstale":"2024-09-17T07:27:20.806309+0000","last_undegraded":"2024-09-17T07:27:20.806309+0000","last_fullsized":"2024-09-17T07:27:20.806309+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":3,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:26:42.815530+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:14:57.052505+0000","last_clean_scrub_stamp":"2024-09-17T07:26:42.815530+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:44.596803+0000","scrub_duration":0.001784779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"439'35","reported_seq":521,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:30.297609+0000","last_change":"2024-09-17T07:27:30.297686+0000","last_active":"2024-09-17T07:27:30.297609+0000","last_peered":"2024-09-17T07:27:30.297609+0000","last_clean":"2024-09-17T07:27:30.297609+0000","last_became_active":"2024-09-17T07:23:57.478340+0000","last_became_peered":"2024-09-17T07:23:57.478340+0000","last_unstale":"2024-09-17T07:27:30.297609+0000","last_undegraded":"2024-09-17T07:27:30.297609+0000","last_fullsized":"2024-09-17T07:27:30.297609+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'35","last_scrub_stamp":"2024-09-17T07:27:30.297594+0000","last_deep_scrub":"110'15","last_deep_scrub_stamp":"2024-09-17T07:04:57.091400+0000","last_clean_scrub_stamp":"2024-09-17T07:27:30.297594+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:38.110106+0000","scrub_duration":0.0023124460000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":454,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:22.710294+0000","last_change":"2024-09-17T07:27:00.127529+0000","last_active":"2024-09-17T07:27:22.710294+0000","last_peered":"2024-09-17T07:27:22.710294+0000","last_clean":"2024-09-17T07:27:22.710294+0000","last_became_active":"2024-09-17T07:22:13.591372+0000","last_became_peered":"2024-09-17T07:22:13.591372+0000","last_unstale":"2024-09-17T07:27:22.710294+0000","last_undegraded":"2024-09-17T07:27:22.710294+0000","last_fullsized":"2024-09-17T07:27:22.710294+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:27:00.127430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:25:50.029556+0000","last_clean_scrub_stamp":"2024-09-17T07:27:00.127430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:00.388603+0000","scrub_duration":0.001748838,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"478'11","reported_seq":934,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:22.476340+0000","last_change":"2024-09-17T07:26:41.815671+0000","last_active":"2024-09-17T07:27:22.476340+0000","last_peered":"2024-09-17T07:27:22.476340+0000","last_clean":"2024-09-17T07:27:22.476340+0000","last_became_active":"2024-09-17T07:23:57.496882+0000","last_became_peered":"2024-09-17T07:23:57.496882+0000","last_unstale":"2024-09-17T07:27:22.476340+0000","last_undegraded":"2024-09-17T07:27:22.476340+0000","last_fullsized":"2024-09-17T07:27:22.476340+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"478'11","last_scrub_stamp":"2024-09-17T07:26:41.815575+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-09-17T07:15:53.512777+0000","last_clean_scrub_stamp":"2024-09-17T07:26:41.815575+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:43.389606+0000","scrub_duration":0.0023297109999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"268'2","reported_seq":635,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:19.461838+0000","last_change":"2024-09-17T07:26:36.868200+0000","last_active":"2024-09-17T07:27:19.461838+0000","last_peered":"2024-09-17T07:27:19.461838+0000","last_clean":"2024-09-17T07:27:19.461838+0000","last_became_active":"2024-09-17T07:25:49.431864+0000","last_became_peered":"2024-09-17T07:25:49.431864+0000","last_unstale":"2024-09-17T07:27:19.461838+0000","last_undegraded":"2024-09-17T07:27:19.461838+0000","last_fullsized":"2024-09-17T07:27:19.461838+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'2","last_scrub_stamp":"2024-09-17T07:26:36.868085+0000","last_deep_scrub":"268'2","last_deep_scrub_stamp":"2024-09-17T07:26:36.868085+0000","last_clean_scrub_stamp":"2024-09-17T07:26:36.868085+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:02.171364+0000","scrub_duration":0.0017805410000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"212'130","reported_seq":640,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.806754+0000","last_change":"2024-09-17T07:26:48.444029+0000","last_active":"2024-09-17T07:27:20.806754+0000","last_peered":"2024-09-17T07:27:20.806754+0000","last_clean":"2024-09-17T07:27:20.806754+0000","last_became_active":"2024-09-17T07:23:57.489445+0000","last_became_peered":"2024-09-17T07:23:57.489445+0000","last_unstale":"2024-09-17T07:27:20.806754+0000","last_undegraded":"2024-09-17T07:27:20.806754+0000","last_fullsized":"2024-09-17T07:27:20.806754+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":4,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:26:48.443917+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:14:57.052505+0000","last_clean_scrub_stamp":"2024-09-17T07:26:48.443917+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:50.972298+0000","scrub_duration":0.0021753620000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"516'56","reported_seq":823,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:28.679620+0000","last_change":"2024-09-17T07:26:17.303340+0000","last_active":"2024-09-17T07:27:28.679620+0000","last_peered":"2024-09-17T07:27:28.679620+0000","last_clean":"2024-09-17T07:27:28.679620+0000","last_became_active":"2024-09-17T07:25:49.439937+0000","last_became_peered":"2024-09-17T07:25:49.439937+0000","last_unstale":"2024-09-17T07:27:28.679620+0000","last_undegraded":"2024-09-17T07:27:28.679620+0000","last_fullsized":"2024-09-17T07:27:28.679620+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'56","last_scrub_stamp":"2024-09-17T07:26:17.303241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:26:17.303241+0000","objects_scrubbed":10,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:46.444913+0000","scrub_duration":0.0022879799999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":83,"num_read_kb":68,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":560,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.807166+0000","last_change":"2024-09-17T07:27:08.708047+0000","last_active":"2024-09-17T07:27:20.807166+0000","last_peered":"2024-09-17T07:27:20.807166+0000","last_clean":"2024-09-17T07:27:20.807166+0000","last_became_active":"2024-09-17T07:23:57.493640+0000","last_became_peered":"2024-09-17T07:23:57.493640+0000","last_unstale":"2024-09-17T07:27:20.807166+0000","last_undegraded":"2024-09-17T07:27:20.807166+0000","last_fullsized":"2024-09-17T07:27:20.807166+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:27:08.707949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:27:08.707949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:13.460377+0000","scrub_duration":0.0013508540000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":599,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:22.474876+0000","last_change":"2024-09-17T07:26:13.574973+0000","last_active":"2024-09-17T07:27:22.474876+0000","last_peered":"2024-09-17T07:27:22.474876+0000","last_clean":"2024-09-17T07:27:22.474876+0000","last_became_active":"2024-09-17T07:21:15.347069+0000","last_became_peered":"2024-09-17T07:21:15.347069+0000","last_unstale":"2024-09-17T07:27:22.474876+0000","last_undegraded":"2024-09-17T07:27:22.474876+0000","last_fullsized":"2024-09-17T07:27:22.474876+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:13.574867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:26:13.574867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:30.084591+0000","scrub_duration":0.00149767,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":51,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.759838+0000","last_change":"2024-09-17T07:27:04.743645+0000","last_active":"2024-09-17T07:27:20.759838+0000","last_peered":"2024-09-17T07:27:20.759838+0000","last_clean":"2024-09-17T07:27:20.759838+0000","last_became_active":"2024-09-17T07:25:49.448463+0000","last_became_peered":"2024-09-17T07:25:49.448463+0000","last_unstale":"2024-09-17T07:27:20.759838+0000","last_undegraded":"2024-09-17T07:27:20.759838+0000","last_fullsized":"2024-09-17T07:27:20.759838+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:27:04.743558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:13:53.406661+0000","last_clean_scrub_stamp":"2024-09-17T07:27:04.743558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:07.613781+0000","scrub_duration":0.0017939220000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"212'130","reported_seq":658,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.387315+0000","last_change":"2024-09-17T07:26:58.669041+0000","last_active":"2024-09-17T07:27:20.387315+0000","last_peered":"2024-09-17T07:27:20.387315+0000","last_clean":"2024-09-17T07:27:20.387315+0000","last_became_active":"2024-09-17T07:22:16.360258+0000","last_became_peered":"2024-09-17T07:22:16.360258+0000","last_unstale":"2024-09-17T07:27:20.387315+0000","last_undegraded":"2024-09-17T07:27:20.387315+0000","last_fullsized":"2024-09-17T07:27:20.387315+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":4,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:26:58.668913+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:14:57.052505+0000","last_clean_scrub_stamp":"2024-09-17T07:26:58.668913+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:20.318226+0000","scrub_duration":0.0018872610000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"520'26","reported_seq":501,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:19.463117+0000","last_change":"2024-09-17T07:27:14.287240+0000","last_active":"2024-09-17T07:27:19.463117+0000","last_peered":"2024-09-17T07:27:19.463117+0000","last_clean":"2024-09-17T07:27:19.463117+0000","last_became_active":"2024-09-17T07:23:57.492747+0000","last_became_peered":"2024-09-17T07:23:57.492747+0000","last_unstale":"2024-09-17T07:27:19.463117+0000","last_undegraded":"2024-09-17T07:27:19.463117+0000","last_fullsized":"2024-09-17T07:27:19.463117+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"520'26","last_scrub_stamp":"2024-09-17T07:27:14.287153+0000","last_deep_scrub":"520'26","last_deep_scrub_stamp":"2024-09-17T07:25:57.063699+0000","last_clean_scrub_stamp":"2024-09-17T07:27:14.287153+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:31.626817+0000","scrub_duration":0.002065697,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":30,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"76'5","reported_seq":1641,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:32.109067+0000","last_change":"2024-09-17T07:26:23.320854+0000","last_active":"2024-09-17T07:27:32.109067+0000","last_peered":"2024-09-17T07:27:32.109067+0000","last_clean":"2024-09-17T07:27:32.109067+0000","last_became_active":"2024-09-17T07:21:15.357178+0000","last_became_peered":"2024-09-17T07:21:15.357178+0000","last_unstale":"2024-09-17T07:27:32.109067+0000","last_undegraded":"2024-09-17T07:27:32.109067+0000","last_fullsized":"2024-09-17T07:27:32.109067+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'5","last_scrub_stamp":"2024-09-17T07:26:23.320769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:26:23.320769+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:45.475946+0000","scrub_duration":0.0017841700000000001,"objects_trimmed":0,"snaptrim_duration":0.00033739799999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":384,"num_read_kb":379,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":320,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.698260+0000","last_change":"2024-09-17T07:26:26.894141+0000","last_active":"2024-09-17T07:27:20.698260+0000","last_peered":"2024-09-17T07:27:20.698260+0000","last_clean":"2024-09-17T07:27:20.698260+0000","last_became_active":"2024-09-17T07:26:09.601090+0000","last_became_peered":"2024-09-17T07:26:09.601090+0000","last_unstale":"2024-09-17T07:27:20.698260+0000","last_undegraded":"2024-09-17T07:27:20.698260+0000","last_fullsized":"2024-09-17T07:27:20.698260+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:26.894033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:26:26.894033+0000","last_clean_scrub_stamp":"2024-09-17T07:26:26.894033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:36.125732+0000","scrub_duration":0.0014059630000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":673,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.698764+0000","last_change":"2024-09-17T07:26:57.987382+0000","last_active":"2024-09-17T07:27:20.698764+0000","last_peered":"2024-09-17T07:27:20.698764+0000","last_clean":"2024-09-17T07:27:20.698764+0000","last_became_active":"2024-09-17T07:23:57.481485+0000","last_became_peered":"2024-09-17T07:23:57.481485+0000","last_unstale":"2024-09-17T07:27:20.698764+0000","last_undegraded":"2024-09-17T07:27:20.698764+0000","last_fullsized":"2024-09-17T07:27:20.698764+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:57.987295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:18:42.051667+0000","last_clean_scrub_stamp":"2024-09-17T07:26:57.987295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:03.480643+0000","scrub_duration":0.0019378469999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":165,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.387517+0000","last_change":"2024-09-17T07:27:18.783434+0000","last_active":"2024-09-17T07:27:20.387517+0000","last_peered":"2024-09-17T07:27:20.387517+0000","last_clean":"2024-09-17T07:27:20.387517+0000","last_became_active":"2024-09-17T07:25:49.437671+0000","last_became_peered":"2024-09-17T07:25:49.437671+0000","last_unstale":"2024-09-17T07:27:20.387517+0000","last_undegraded":"2024-09-17T07:27:20.387517+0000","last_fullsized":"2024-09-17T07:27:20.387517+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:27:18.783326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:02.062618+0000","last_clean_scrub_stamp":"2024-09-17T07:27:18.783326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:44.363758+0000","scrub_duration":0.001436744,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"212'130","reported_seq":683,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:22.710086+0000","last_change":"2024-09-17T07:27:01.112058+0000","last_active":"2024-09-17T07:27:22.710086+0000","last_peered":"2024-09-17T07:27:22.710086+0000","last_clean":"2024-09-17T07:27:22.710086+0000","last_became_active":"2024-09-17T07:21:18.357874+0000","last_became_peered":"2024-09-17T07:21:18.357874+0000","last_unstale":"2024-09-17T07:27:22.710086+0000","last_undegraded":"2024-09-17T07:27:22.710086+0000","last_fullsized":"2024-09-17T07:27:22.710086+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":3,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:27:01.111943+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:20:34.113602+0000","last_clean_scrub_stamp":"2024-09-17T07:27:01.111943+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:26.213695+0000","scrub_duration":0.0021401279999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"439'42","reported_seq":693,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:19.462653+0000","last_change":"2024-09-17T07:26:31.782487+0000","last_active":"2024-09-17T07:27:19.462653+0000","last_peered":"2024-09-17T07:27:19.462653+0000","last_clean":"2024-09-17T07:27:19.462653+0000","last_became_active":"2024-09-17T07:23:57.480190+0000","last_became_peered":"2024-09-17T07:23:57.480190+0000","last_unstale":"2024-09-17T07:27:19.462653+0000","last_undegraded":"2024-09-17T07:27:19.462653+0000","last_fullsized":"2024-09-17T07:27:19.462653+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'42","last_scrub_stamp":"2024-09-17T07:26:31.782388+0000","last_deep_scrub":"268'30","last_deep_scrub_stamp":"2024-09-17T07:14:00.599656+0000","last_clean_scrub_stamp":"2024-09-17T07:26:31.782388+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:48.742554+0000","scrub_duration":0.003378462,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"70'1","reported_seq":645,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:19.461472+0000","last_change":"2024-09-17T07:26:45.763688+0000","last_active":"2024-09-17T07:27:19.461472+0000","last_peered":"2024-09-17T07:27:19.461472+0000","last_clean":"2024-09-17T07:27:19.461472+0000","last_became_active":"2024-09-17T07:25:49.432721+0000","last_became_peered":"2024-09-17T07:25:49.432721+0000","last_unstale":"2024-09-17T07:27:19.461472+0000","last_undegraded":"2024-09-17T07:27:19.461472+0000","last_fullsized":"2024-09-17T07:27:19.461472+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-17T07:26:45.763600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:26:45.763600+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:06.596582+0000","scrub_duration":0.0016379529999999999,"objects_trimmed":0,"snaptrim_duration":0.0032298629999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":145,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:22.826211+0000","last_change":"2024-09-17T07:27:22.826295+0000","last_active":"2024-09-17T07:27:22.826211+0000","last_peered":"2024-09-17T07:27:22.826211+0000","last_clean":"2024-09-17T07:27:22.826211+0000","last_became_active":"2024-09-17T07:22:13.710211+0000","last_became_peered":"2024-09-17T07:22:13.710211+0000","last_unstale":"2024-09-17T07:27:22.826211+0000","last_undegraded":"2024-09-17T07:27:22.826211+0000","last_fullsized":"2024-09-17T07:27:22.826211+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:27:22.826173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:55.321202+0000","last_clean_scrub_stamp":"2024-09-17T07:27:22.826173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:44.035163+0000","scrub_duration":0.0019797629999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":544,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.807837+0000","last_change":"2024-09-17T07:26:29.422533+0000","last_active":"2024-09-17T07:27:20.807837+0000","last_peered":"2024-09-17T07:27:20.807837+0000","last_clean":"2024-09-17T07:27:20.807837+0000","last_became_active":"2024-09-17T07:23:57.480455+0000","last_became_peered":"2024-09-17T07:23:57.480455+0000","last_unstale":"2024-09-17T07:27:20.807837+0000","last_undegraded":"2024-09-17T07:27:20.807837+0000","last_fullsized":"2024-09-17T07:27:20.807837+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:29.422419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:26:29.422419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:46.481871+0000","scrub_duration":0.0013803999999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":490,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:22.710374+0000","last_change":"2024-09-17T07:26:42.219856+0000","last_active":"2024-09-17T07:27:22.710374+0000","last_peered":"2024-09-17T07:27:22.710374+0000","last_clean":"2024-09-17T07:27:22.710374+0000","last_became_active":"2024-09-17T07:25:49.439416+0000","last_became_peered":"2024-09-17T07:25:49.439416+0000","last_unstale":"2024-09-17T07:27:22.710374+0000","last_undegraded":"2024-09-17T07:27:22.710374+0000","last_fullsized":"2024-09-17T07:27:22.710374+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:42.219774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:26:42.219774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:00.294851+0000","scrub_duration":0.0017563069999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"439'38","reported_seq":716,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.807429+0000","last_change":"2024-09-17T07:26:44.891822+0000","last_active":"2024-09-17T07:27:20.807429+0000","last_peered":"2024-09-17T07:27:20.807429+0000","last_clean":"2024-09-17T07:27:20.807429+0000","last_became_active":"2024-09-17T07:22:13.583204+0000","last_became_peered":"2024-09-17T07:22:13.583204+0000","last_unstale":"2024-09-17T07:27:20.807429+0000","last_undegraded":"2024-09-17T07:27:20.807429+0000","last_fullsized":"2024-09-17T07:27:20.807429+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'38","last_scrub_stamp":"2024-09-17T07:26:44.891686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:26:44.891686+0000","objects_scrubbed":7,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:03.485066+0000","scrub_duration":0.0025976969999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"268'3","reported_seq":55,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.759202+0000","last_change":"2024-09-17T07:26:27.885827+0000","last_active":"2024-09-17T07:27:20.759202+0000","last_peered":"2024-09-17T07:27:20.759202+0000","last_clean":"2024-09-17T07:27:20.759202+0000","last_became_active":"2024-09-17T07:25:49.453053+0000","last_became_peered":"2024-09-17T07:25:49.453053+0000","last_unstale":"2024-09-17T07:27:20.759202+0000","last_undegraded":"2024-09-17T07:27:20.759202+0000","last_fullsized":"2024-09-17T07:27:20.759202+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'3","last_scrub_stamp":"2024-09-17T07:26:27.885747+0000","last_deep_scrub":"268'3","last_deep_scrub_stamp":"2024-09-17T07:13:54.949345+0000","last_clean_scrub_stamp":"2024-09-17T07:26:27.885747+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:55.211433+0000","scrub_duration":0.0041955849999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":605,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.808046+0000","last_change":"2024-09-17T07:26:35.378501+0000","last_active":"2024-09-17T07:27:20.808046+0000","last_peered":"2024-09-17T07:27:20.808046+0000","last_clean":"2024-09-17T07:27:20.808046+0000","last_became_active":"2024-09-17T07:23:57.477410+0000","last_became_peered":"2024-09-17T07:23:57.477410+0000","last_unstale":"2024-09-17T07:27:20.808046+0000","last_undegraded":"2024-09-17T07:27:20.808046+0000","last_fullsized":"2024-09-17T07:27:20.808046+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:35.378391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:26:35.378391+0000","last_clean_scrub_stamp":"2024-09-17T07:26:35.378391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:45.200399+0000","scrub_duration":0.001544789,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"212'130","reported_seq":668,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:28.816584+0000","last_change":"2024-09-17T07:27:28.816677+0000","last_active":"2024-09-17T07:27:28.816584+0000","last_peered":"2024-09-17T07:27:28.816584+0000","last_clean":"2024-09-17T07:27:28.816584+0000","last_became_active":"2024-09-17T07:22:16.366295+0000","last_became_peered":"2024-09-17T07:22:16.366295+0000","last_unstale":"2024-09-17T07:27:28.816584+0000","last_undegraded":"2024-09-17T07:27:28.816584+0000","last_fullsized":"2024-09-17T07:27:28.816584+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:27:28.816567+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:14:57.052505+0000","last_clean_scrub_stamp":"2024-09-17T07:27:28.816567+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:53.560309+0000","scrub_duration":0.00161006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":171,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:27.784152+0000","last_change":"2024-09-17T07:27:27.784231+0000","last_active":"2024-09-17T07:27:27.784152+0000","last_peered":"2024-09-17T07:27:27.784152+0000","last_clean":"2024-09-17T07:27:27.784152+0000","last_became_active":"2024-09-17T07:23:57.476027+0000","last_became_peered":"2024-09-17T07:23:57.476027+0000","last_unstale":"2024-09-17T07:27:27.784152+0000","last_undegraded":"2024-09-17T07:27:27.784152+0000","last_fullsized":"2024-09-17T07:27:27.784152+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:27:27.784136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:18:48.891136+0000","last_clean_scrub_stamp":"2024-09-17T07:27:27.784136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:31.332571+0000","scrub_duration":0.001505681,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":560,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.807683+0000","last_change":"2024-09-17T07:26:56.655819+0000","last_active":"2024-09-17T07:27:20.807683+0000","last_peered":"2024-09-17T07:27:20.807683+0000","last_clean":"2024-09-17T07:27:20.807683+0000","last_became_active":"2024-09-17T07:25:49.441234+0000","last_became_peered":"2024-09-17T07:25:49.441234+0000","last_unstale":"2024-09-17T07:27:20.807683+0000","last_undegraded":"2024-09-17T07:27:20.807683+0000","last_fullsized":"2024-09-17T07:27:20.807683+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:56.655711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:25:46.981696+0000","last_clean_scrub_stamp":"2024-09-17T07:26:56.655711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:58.207683+0000","scrub_duration":0.0013852210000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"439'51","reported_seq":760,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.807157+0000","last_change":"2024-09-17T07:26:35.626589+0000","last_active":"2024-09-17T07:27:20.807157+0000","last_peered":"2024-09-17T07:27:20.807157+0000","last_clean":"2024-09-17T07:27:20.807157+0000","last_became_active":"2024-09-17T07:22:13.588260+0000","last_became_peered":"2024-09-17T07:22:13.588260+0000","last_unstale":"2024-09-17T07:27:20.807157+0000","last_undegraded":"2024-09-17T07:27:20.807157+0000","last_fullsized":"2024-09-17T07:27:20.807157+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'51","last_scrub_stamp":"2024-09-17T07:26:35.626492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:26:35.626492+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:52.839178+0000","scrub_duration":0.0026693060000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":684,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.698466+0000","last_change":"2024-09-17T07:26:51.066605+0000","last_active":"2024-09-17T07:27:20.698466+0000","last_peered":"2024-09-17T07:27:20.698466+0000","last_clean":"2024-09-17T07:27:20.698466+0000","last_became_active":"2024-09-17T07:23:57.482084+0000","last_became_peered":"2024-09-17T07:23:57.482084+0000","last_unstale":"2024-09-17T07:27:20.698466+0000","last_undegraded":"2024-09-17T07:27:20.698466+0000","last_fullsized":"2024-09-17T07:27:20.698466+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:51.066517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:24:23.255270+0000","last_clean_scrub_stamp":"2024-09-17T07:26:51.066517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:00.509842+0000","scrub_duration":0.001686491,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":667,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.698474+0000","last_change":"2024-09-17T07:26:34.953585+0000","last_active":"2024-09-17T07:27:20.698474+0000","last_peered":"2024-09-17T07:27:20.698474+0000","last_clean":"2024-09-17T07:27:20.698474+0000","last_became_active":"2024-09-17T07:23:57.481504+0000","last_became_peered":"2024-09-17T07:23:57.481504+0000","last_unstale":"2024-09-17T07:27:20.698474+0000","last_undegraded":"2024-09-17T07:27:20.698474+0000","last_fullsized":"2024-09-17T07:27:20.698474+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:34.953500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:24:20.282973+0000","last_clean_scrub_stamp":"2024-09-17T07:26:34.953500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:58.510821+0000","scrub_duration":0.0037914979999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":154,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.387206+0000","last_change":"2024-09-17T07:26:01.843368+0000","last_active":"2024-09-17T07:27:20.387206+0000","last_peered":"2024-09-17T07:27:20.387206+0000","last_clean":"2024-09-17T07:27:20.387206+0000","last_became_active":"2024-09-17T07:25:49.437863+0000","last_became_peered":"2024-09-17T07:25:49.437863+0000","last_unstale":"2024-09-17T07:27:20.387206+0000","last_undegraded":"2024-09-17T07:27:20.387206+0000","last_fullsized":"2024-09-17T07:27:20.387206+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:01.843148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:45.275473+0000","last_clean_scrub_stamp":"2024-09-17T07:26:01.843148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:31.022195+0000","scrub_duration":0.0021668270000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"439'25","reported_seq":628,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:19.462168+0000","last_change":"2024-09-17T07:26:23.741158+0000","last_active":"2024-09-17T07:27:19.462168+0000","last_peered":"2024-09-17T07:27:19.462168+0000","last_clean":"2024-09-17T07:27:19.462168+0000","last_became_active":"2024-09-17T07:21:15.349361+0000","last_became_peered":"2024-09-17T07:21:15.349361+0000","last_unstale":"2024-09-17T07:27:19.462168+0000","last_undegraded":"2024-09-17T07:27:19.462168+0000","last_fullsized":"2024-09-17T07:27:19.462168+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'25","last_scrub_stamp":"2024-09-17T07:26:23.741060+0000","last_deep_scrub":"268'19","last_deep_scrub_stamp":"2024-09-17T07:14:06.634334+0000","last_clean_scrub_stamp":"2024-09-17T07:26:23.741060+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:42.119714+0000","scrub_duration":0.0025792580000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":491,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:19.463005+0000","last_change":"2024-09-17T07:26:49.227283+0000","last_active":"2024-09-17T07:27:19.463005+0000","last_peered":"2024-09-17T07:27:19.463005+0000","last_clean":"2024-09-17T07:27:19.463005+0000","last_became_active":"2024-09-17T07:25:49.425468+0000","last_became_peered":"2024-09-17T07:25:49.425468+0000","last_unstale":"2024-09-17T07:27:19.463005+0000","last_undegraded":"2024-09-17T07:27:19.463005+0000","last_fullsized":"2024-09-17T07:27:19.463005+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:49.227208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:13:53.593078+0000","last_clean_scrub_stamp":"2024-09-17T07:26:49.227208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:01.466332+0000","scrub_duration":0.001636202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":570,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.698512+0000","last_change":"2024-09-17T07:26:29.940005+0000","last_active":"2024-09-17T07:27:20.698512+0000","last_peered":"2024-09-17T07:27:20.698512+0000","last_clean":"2024-09-17T07:27:20.698512+0000","last_became_active":"2024-09-17T07:26:09.789844+0000","last_became_peered":"2024-09-17T07:26:09.789844+0000","last_unstale":"2024-09-17T07:27:20.698512+0000","last_undegraded":"2024-09-17T07:27:20.698512+0000","last_fullsized":"2024-09-17T07:27:20.698512+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:29.939920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:26:29.939920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:42.034439+0000","scrub_duration":0.001960617,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":609,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:19.461561+0000","last_change":"2024-09-17T07:26:49.705368+0000","last_active":"2024-09-17T07:27:19.461561+0000","last_peered":"2024-09-17T07:27:19.461561+0000","last_clean":"2024-09-17T07:27:19.461561+0000","last_became_active":"2024-09-17T07:22:13.589460+0000","last_became_peered":"2024-09-17T07:22:13.589460+0000","last_unstale":"2024-09-17T07:27:19.461561+0000","last_undegraded":"2024-09-17T07:27:19.461561+0000","last_fullsized":"2024-09-17T07:27:19.461561+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:49.705283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:58.099849+0000","last_clean_scrub_stamp":"2024-09-17T07:26:49.705283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:08.797340+0000","scrub_duration":0.0016370289999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"439'46","reported_seq":697,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.806527+0000","last_change":"2024-09-17T07:26:20.278724+0000","last_active":"2024-09-17T07:27:20.806527+0000","last_peered":"2024-09-17T07:27:20.806527+0000","last_clean":"2024-09-17T07:27:20.806527+0000","last_became_active":"2024-09-17T07:23:57.482858+0000","last_became_peered":"2024-09-17T07:23:57.482858+0000","last_unstale":"2024-09-17T07:27:20.806527+0000","last_undegraded":"2024-09-17T07:27:20.806527+0000","last_fullsized":"2024-09-17T07:27:20.806527+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'46","last_scrub_stamp":"2024-09-17T07:26:20.278609+0000","last_deep_scrub":"242'34","last_deep_scrub_stamp":"2024-09-17T07:14:50.103539+0000","last_clean_scrub_stamp":"2024-09-17T07:26:20.278609+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:46.455910+0000","scrub_duration":0.0024723520000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'2","reported_seq":97,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.698852+0000","last_change":"2024-09-17T07:27:13.971677+0000","last_active":"2024-09-17T07:27:20.698852+0000","last_peered":"2024-09-17T07:27:20.698852+0000","last_clean":"2024-09-17T07:27:20.698852+0000","last_became_active":"2024-09-17T07:26:09.601463+0000","last_became_peered":"2024-09-17T07:26:09.601463+0000","last_unstale":"2024-09-17T07:27:20.698852+0000","last_undegraded":"2024-09-17T07:27:20.698852+0000","last_fullsized":"2024-09-17T07:27:20.698852+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-09-17T07:27:13.971590+0000","last_deep_scrub":"89'2","last_deep_scrub_stamp":"2024-09-17T07:12:20.301263+0000","last_clean_scrub_stamp":"2024-09-17T07:27:13.971590+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:36.934399+0000","scrub_duration":0.0018383679999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":420,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"436'14","reported_seq":225,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:25.805241+0000","last_change":"2024-09-17T07:27:24.785675+0000","last_active":"2024-09-17T07:27:25.805241+0000","last_peered":"2024-09-17T07:27:25.805241+0000","last_clean":"2024-09-17T07:27:25.805241+0000","last_became_active":"2024-09-17T07:22:13.624050+0000","last_became_peered":"2024-09-17T07:22:13.624050+0000","last_unstale":"2024-09-17T07:27:25.805241+0000","last_undegraded":"2024-09-17T07:27:25.805241+0000","last_fullsized":"2024-09-17T07:27:25.805241+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"436'14","last_scrub_stamp":"2024-09-17T07:27:24.785571+0000","last_deep_scrub":"225'9","last_deep_scrub_stamp":"2024-09-17T07:11:25.292565+0000","last_clean_scrub_stamp":"2024-09-17T07:27:24.785571+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:32.388864+0000","scrub_duration":0.0018136680000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":572,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.807018+0000","last_change":"2024-09-17T07:26:19.239231+0000","last_active":"2024-09-17T07:27:20.807018+0000","last_peered":"2024-09-17T07:27:20.807018+0000","last_clean":"2024-09-17T07:27:20.807018+0000","last_became_active":"2024-09-17T07:23:57.493273+0000","last_became_peered":"2024-09-17T07:23:57.493273+0000","last_unstale":"2024-09-17T07:27:20.807018+0000","last_undegraded":"2024-09-17T07:27:20.807018+0000","last_fullsized":"2024-09-17T07:27:20.807018+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:19.239152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:18:41.385928+0000","last_clean_scrub_stamp":"2024-09-17T07:26:19.239152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:47.521562+0000","scrub_duration":0.001371171,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"439'29","reported_seq":716,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:19.462446+0000","last_change":"2024-09-17T07:27:05.698111+0000","last_active":"2024-09-17T07:27:19.462446+0000","last_peered":"2024-09-17T07:27:19.462446+0000","last_clean":"2024-09-17T07:27:19.462446+0000","last_became_active":"2024-09-17T07:25:49.437151+0000","last_became_peered":"2024-09-17T07:25:49.437151+0000","last_unstale":"2024-09-17T07:27:19.462446+0000","last_undegraded":"2024-09-17T07:27:19.462446+0000","last_fullsized":"2024-09-17T07:27:19.462446+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'29","last_scrub_stamp":"2024-09-17T07:27:05.697965+0000","last_deep_scrub":"439'29","last_deep_scrub_stamp":"2024-09-17T07:27:05.697965+0000","last_clean_scrub_stamp":"2024-09-17T07:27:05.697965+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:27.244334+0000","scrub_duration":0.0027608450000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"512'4","reported_seq":648,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:19.463468+0000","last_change":"2024-09-17T07:26:55.642280+0000","last_active":"2024-09-17T07:27:19.463468+0000","last_peered":"2024-09-17T07:27:19.463468+0000","last_clean":"2024-09-17T07:27:19.463468+0000","last_became_active":"2024-09-17T07:22:13.590172+0000","last_became_peered":"2024-09-17T07:22:13.590172+0000","last_unstale":"2024-09-17T07:27:19.463468+0000","last_undegraded":"2024-09-17T07:27:19.463468+0000","last_fullsized":"2024-09-17T07:27:19.463468+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"512'4","last_scrub_stamp":"2024-09-17T07:26:55.642174+0000","last_deep_scrub":"512'4","last_deep_scrub_stamp":"2024-09-17T07:25:26.876662+0000","last_clean_scrub_stamp":"2024-09-17T07:26:55.642174+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:07.000531+0000","scrub_duration":0.0022207529999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":439,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.698347+0000","last_change":"2024-09-17T07:27:08.887830+0000","last_active":"2024-09-17T07:27:20.698347+0000","last_peered":"2024-09-17T07:27:20.698347+0000","last_clean":"2024-09-17T07:27:20.698347+0000","last_became_active":"2024-09-17T07:26:09.598369+0000","last_became_peered":"2024-09-17T07:26:09.598369+0000","last_unstale":"2024-09-17T07:27:20.698347+0000","last_undegraded":"2024-09-17T07:27:20.698347+0000","last_fullsized":"2024-09-17T07:27:20.698347+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:27:08.887748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:11:17.479036+0000","last_clean_scrub_stamp":"2024-09-17T07:27:08.887748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:13.097888+0000","scrub_duration":0.0012505999999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":51,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.759250+0000","last_change":"2024-09-17T07:27:09.723542+0000","last_active":"2024-09-17T07:27:20.759250+0000","last_peered":"2024-09-17T07:27:20.759250+0000","last_clean":"2024-09-17T07:27:20.759250+0000","last_became_active":"2024-09-17T07:25:49.445899+0000","last_became_peered":"2024-09-17T07:25:49.445899+0000","last_unstale":"2024-09-17T07:27:20.759250+0000","last_undegraded":"2024-09-17T07:27:20.759250+0000","last_fullsized":"2024-09-17T07:27:20.759250+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:27:09.723460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:27:09.723460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:27.664331+0000","scrub_duration":0.001394946,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"439'21","reported_seq":706,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:19.462524+0000","last_change":"2024-09-17T07:26:56.713742+0000","last_active":"2024-09-17T07:27:19.462524+0000","last_peered":"2024-09-17T07:27:19.462524+0000","last_clean":"2024-09-17T07:27:19.462524+0000","last_became_active":"2024-09-17T07:25:49.432314+0000","last_became_peered":"2024-09-17T07:25:49.432314+0000","last_unstale":"2024-09-17T07:27:19.462524+0000","last_undegraded":"2024-09-17T07:27:19.462524+0000","last_fullsized":"2024-09-17T07:27:19.462524+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'21","last_scrub_stamp":"2024-09-17T07:26:56.713622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:26:56.713622+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:02.228736+0000","scrub_duration":0.002458394,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":654,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.698397+0000","last_change":"2024-09-17T07:26:57.007387+0000","last_active":"2024-09-17T07:27:20.698397+0000","last_peered":"2024-09-17T07:27:20.698397+0000","last_clean":"2024-09-17T07:27:20.698397+0000","last_became_active":"2024-09-17T07:21:15.355464+0000","last_became_peered":"2024-09-17T07:21:15.355464+0000","last_unstale":"2024-09-17T07:27:20.698397+0000","last_undegraded":"2024-09-17T07:27:20.698397+0000","last_fullsized":"2024-09-17T07:27:20.698397+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:57.007294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:26:57.007294+0000","last_clean_scrub_stamp":"2024-09-17T07:26:57.007294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:07.480606+0000","scrub_duration":0.0014950390000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":62,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.759898+0000","last_change":"2024-09-17T07:27:11.726251+0000","last_active":"2024-09-17T07:27:20.759898+0000","last_peered":"2024-09-17T07:27:20.759898+0000","last_clean":"2024-09-17T07:27:20.759898+0000","last_became_active":"2024-09-17T07:25:49.445346+0000","last_became_peered":"2024-09-17T07:25:49.445346+0000","last_unstale":"2024-09-17T07:27:20.759898+0000","last_undegraded":"2024-09-17T07:27:20.759898+0000","last_fullsized":"2024-09-17T07:27:20.759898+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:27:11.726171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:18.367486+0000","last_clean_scrub_stamp":"2024-09-17T07:27:11.726171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:20.249715+0000","scrub_duration":0.001556415,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":707,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.698772+0000","last_change":"2024-09-17T07:26:54.023145+0000","last_active":"2024-09-17T07:27:20.698772+0000","last_peered":"2024-09-17T07:27:20.698772+0000","last_clean":"2024-09-17T07:27:20.698772+0000","last_became_active":"2024-09-17T07:22:13.590599+0000","last_became_peered":"2024-09-17T07:22:13.590599+0000","last_unstale":"2024-09-17T07:27:20.698772+0000","last_undegraded":"2024-09-17T07:27:20.698772+0000","last_fullsized":"2024-09-17T07:27:20.698772+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:54.023053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:24:24.282487+0000","last_clean_scrub_stamp":"2024-09-17T07:26:54.023053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:54.216936+0000","scrub_duration":0.0015857029999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":623,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.806888+0000","last_change":"2024-09-17T07:27:00.084261+0000","last_active":"2024-09-17T07:27:20.806888+0000","last_peered":"2024-09-17T07:27:20.806888+0000","last_clean":"2024-09-17T07:27:20.806888+0000","last_became_active":"2024-09-17T07:21:15.348758+0000","last_became_peered":"2024-09-17T07:21:15.348758+0000","last_unstale":"2024-09-17T07:27:20.806888+0000","last_undegraded":"2024-09-17T07:27:20.806888+0000","last_fullsized":"2024-09-17T07:27:20.806888+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:27:00.084159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:25:36.444488+0000","last_clean_scrub_stamp":"2024-09-17T07:27:00.084159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:10.748189+0000","scrub_duration":0.0014711730000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"439'18","reported_seq":59,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.759889+0000","last_change":"2024-09-17T07:26:35.786261+0000","last_active":"2024-09-17T07:27:20.759889+0000","last_peered":"2024-09-17T07:27:20.759889+0000","last_clean":"2024-09-17T07:27:20.759889+0000","last_became_active":"2024-09-17T07:25:49.444760+0000","last_became_peered":"2024-09-17T07:25:49.444760+0000","last_unstale":"2024-09-17T07:27:20.759889+0000","last_undegraded":"2024-09-17T07:27:20.759889+0000","last_fullsized":"2024-09-17T07:27:20.759889+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'18","last_scrub_stamp":"2024-09-17T07:26:35.786155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:26:35.786155+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:03.360633+0000","scrub_duration":0.0019837840000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":24,"num_read_kb":21,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":572,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.807212+0000","last_change":"2024-09-17T07:27:02.729678+0000","last_active":"2024-09-17T07:27:20.807212+0000","last_peered":"2024-09-17T07:27:20.807212+0000","last_clean":"2024-09-17T07:27:20.807212+0000","last_became_active":"2024-09-17T07:25:49.439959+0000","last_became_peered":"2024-09-17T07:25:49.439959+0000","last_unstale":"2024-09-17T07:27:20.807212+0000","last_undegraded":"2024-09-17T07:27:20.807212+0000","last_fullsized":"2024-09-17T07:27:20.807212+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:27:02.729587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:27:02.729587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:15.174554+0000","scrub_duration":0.0016207190000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":145,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.387372+0000","last_change":"2024-09-17T07:26:30.773949+0000","last_active":"2024-09-17T07:27:20.387372+0000","last_peered":"2024-09-17T07:27:20.387372+0000","last_clean":"2024-09-17T07:27:20.387372+0000","last_became_active":"2024-09-17T07:22:13.623754+0000","last_became_peered":"2024-09-17T07:22:13.623754+0000","last_unstale":"2024-09-17T07:27:20.387372+0000","last_undegraded":"2024-09-17T07:27:20.387372+0000","last_fullsized":"2024-09-17T07:27:20.387372+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:30.773841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:26:30.773841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:42.562972+0000","scrub_duration":0.0034468150000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"439'30","reported_seq":63,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.759698+0000","last_change":"2024-09-17T07:27:01.744678+0000","last_active":"2024-09-17T07:27:20.759698+0000","last_peered":"2024-09-17T07:27:20.759698+0000","last_clean":"2024-09-17T07:27:20.759698+0000","last_became_active":"2024-09-17T07:25:49.448072+0000","last_became_peered":"2024-09-17T07:25:49.448072+0000","last_unstale":"2024-09-17T07:27:20.759698+0000","last_undegraded":"2024-09-17T07:27:20.759698+0000","last_fullsized":"2024-09-17T07:27:20.759698+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'30","last_scrub_stamp":"2024-09-17T07:27:01.744598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:27:01.744598+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:02.418179+0000","scrub_duration":0.001931,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":622,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.807369+0000","last_change":"2024-09-17T07:26:30.448278+0000","last_active":"2024-09-17T07:27:20.807369+0000","last_peered":"2024-09-17T07:27:20.807369+0000","last_clean":"2024-09-17T07:27:20.807369+0000","last_became_active":"2024-09-17T07:25:49.437128+0000","last_became_peered":"2024-09-17T07:25:49.437128+0000","last_unstale":"2024-09-17T07:27:20.807369+0000","last_undegraded":"2024-09-17T07:27:20.807369+0000","last_fullsized":"2024-09-17T07:27:20.807369+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:30.448176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:19:40.443474+0000","last_clean_scrub_stamp":"2024-09-17T07:26:30.448176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:58.792122+0000","scrub_duration":0.0023650339999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"478'13","reported_seq":836,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:24.731554+0000","last_change":"2024-09-17T07:27:24.731554+0000","last_active":"2024-09-17T07:27:24.731554+0000","last_peered":"2024-09-17T07:27:24.731554+0000","last_clean":"2024-09-17T07:27:24.731554+0000","last_became_active":"2024-09-17T07:23:57.522386+0000","last_became_peered":"2024-09-17T07:23:57.522386+0000","last_unstale":"2024-09-17T07:27:24.731554+0000","last_undegraded":"2024-09-17T07:27:24.731554+0000","last_fullsized":"2024-09-17T07:27:24.731554+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"478'13","last_scrub_stamp":"2024-09-17T07:27:24.731384+0000","last_deep_scrub":"163'9","last_deep_scrub_stamp":"2024-09-17T07:07:34.933808+0000","last_clean_scrub_stamp":"2024-09-17T07:27:24.731384+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:51.829395+0000","scrub_duration":0.0025423450000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":638,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:19.462351+0000","last_change":"2024-09-17T07:27:17.701080+0000","last_active":"2024-09-17T07:27:19.462351+0000","last_peered":"2024-09-17T07:27:19.462351+0000","last_clean":"2024-09-17T07:27:19.462351+0000","last_became_active":"2024-09-17T07:25:49.432373+0000","last_became_peered":"2024-09-17T07:25:49.432373+0000","last_unstale":"2024-09-17T07:27:19.462351+0000","last_undegraded":"2024-09-17T07:27:19.462351+0000","last_fullsized":"2024-09-17T07:27:19.462351+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:27:17.700994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:26.619893+0000","last_clean_scrub_stamp":"2024-09-17T07:27:17.700994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:32.072214+0000","scrub_duration":0.0016546600000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"439'21","reported_seq":688,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.807498+0000","last_change":"2024-09-17T07:26:36.390354+0000","last_active":"2024-09-17T07:27:20.807498+0000","last_peered":"2024-09-17T07:27:20.807498+0000","last_clean":"2024-09-17T07:27:20.807498+0000","last_became_active":"2024-09-17T07:25:49.440071+0000","last_became_peered":"2024-09-17T07:25:49.440071+0000","last_unstale":"2024-09-17T07:27:20.807498+0000","last_undegraded":"2024-09-17T07:27:20.807498+0000","last_fullsized":"2024-09-17T07:27:20.807498+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'21","last_scrub_stamp":"2024-09-17T07:26:36.390220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:26:36.390220+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:01.277657+0000","scrub_duration":0.002320398,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":636,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:19.462368+0000","last_change":"2024-09-17T07:26:43.779088+0000","last_active":"2024-09-17T07:27:19.462368+0000","last_peered":"2024-09-17T07:27:19.462368+0000","last_clean":"2024-09-17T07:27:19.462368+0000","last_became_active":"2024-09-17T07:25:49.431778+0000","last_became_peered":"2024-09-17T07:25:49.431778+0000","last_unstale":"2024-09-17T07:27:19.462368+0000","last_undegraded":"2024-09-17T07:27:19.462368+0000","last_fullsized":"2024-09-17T07:27:19.462368+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:43.779011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:26:43.779011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:44.415632+0000","scrub_duration":0.00331285,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"516'16","reported_seq":77,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:25.805145+0000","last_change":"2024-09-17T07:26:54.748648+0000","last_active":"2024-09-17T07:27:25.805145+0000","last_peered":"2024-09-17T07:27:25.805145+0000","last_clean":"2024-09-17T07:27:25.805145+0000","last_became_active":"2024-09-17T07:25:49.446551+0000","last_became_peered":"2024-09-17T07:25:49.446551+0000","last_unstale":"2024-09-17T07:27:25.805145+0000","last_undegraded":"2024-09-17T07:27:25.805145+0000","last_fullsized":"2024-09-17T07:27:25.805145+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'16","last_scrub_stamp":"2024-09-17T07:26:54.748561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:26:54.748561+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:15.409095+0000","scrub_duration":0.001788066,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":617,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:19.463335+0000","last_change":"2024-09-17T07:26:53.608456+0000","last_active":"2024-09-17T07:27:19.463335+0000","last_peered":"2024-09-17T07:27:19.463335+0000","last_clean":"2024-09-17T07:27:19.463335+0000","last_became_active":"2024-09-17T07:23:57.472854+0000","last_became_peered":"2024-09-17T07:23:57.472854+0000","last_unstale":"2024-09-17T07:27:19.463335+0000","last_undegraded":"2024-09-17T07:27:19.463335+0000","last_fullsized":"2024-09-17T07:27:19.463335+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:53.608340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:20:37.110458+0000","last_clean_scrub_stamp":"2024-09-17T07:26:53.608340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:22.216453+0000","scrub_duration":0.0014634769999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"543'25","reported_seq":479,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:19.462900+0000","last_change":"2024-09-17T07:27:08.217225+0000","last_active":"2024-09-17T07:27:19.462900+0000","last_peered":"2024-09-17T07:27:19.462900+0000","last_clean":"2024-09-17T07:27:19.462900+0000","last_became_active":"2024-09-17T07:21:15.348060+0000","last_became_peered":"2024-09-17T07:21:15.348060+0000","last_unstale":"2024-09-17T07:27:19.462900+0000","last_undegraded":"2024-09-17T07:27:19.462900+0000","last_fullsized":"2024-09-17T07:27:19.462900+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"543'25","last_scrub_stamp":"2024-09-17T07:27:08.217115+0000","last_deep_scrub":"543'25","last_deep_scrub_stamp":"2024-09-17T07:27:08.217115+0000","last_clean_scrub_stamp":"2024-09-17T07:27:08.217115+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:36.327801+0000","scrub_duration":0.0038364929999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":679,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.807200+0000","last_change":"2024-09-17T07:26:41.781589+0000","last_active":"2024-09-17T07:27:20.807200+0000","last_peered":"2024-09-17T07:27:20.807200+0000","last_clean":"2024-09-17T07:27:20.807200+0000","last_became_active":"2024-09-17T07:25:49.439173+0000","last_became_peered":"2024-09-17T07:25:49.439173+0000","last_unstale":"2024-09-17T07:27:20.807200+0000","last_undegraded":"2024-09-17T07:27:20.807200+0000","last_fullsized":"2024-09-17T07:27:20.807200+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:41.781503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:10:36.334827+0000","last_clean_scrub_stamp":"2024-09-17T07:26:41.781503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:55.665857+0000","scrub_duration":0.0015077859999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":62,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:24.754702+0000","last_change":"2024-09-17T07:27:24.754791+0000","last_active":"2024-09-17T07:27:24.754702+0000","last_peered":"2024-09-17T07:27:24.754702+0000","last_clean":"2024-09-17T07:27:24.754702+0000","last_became_active":"2024-09-17T07:25:49.438615+0000","last_became_peered":"2024-09-17T07:25:49.438615+0000","last_unstale":"2024-09-17T07:27:24.754702+0000","last_undegraded":"2024-09-17T07:27:24.754702+0000","last_fullsized":"2024-09-17T07:27:24.754702+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:27:24.754683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:27:24.754683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:50.719869+0000","scrub_duration":0.001328119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":620,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:19.461920+0000","last_change":"2024-09-17T07:27:05.654258+0000","last_active":"2024-09-17T07:27:19.461920+0000","last_peered":"2024-09-17T07:27:19.461920+0000","last_clean":"2024-09-17T07:27:19.461920+0000","last_became_active":"2024-09-17T07:23:57.478818+0000","last_became_peered":"2024-09-17T07:23:57.478818+0000","last_unstale":"2024-09-17T07:27:19.461920+0000","last_undegraded":"2024-09-17T07:27:19.461920+0000","last_fullsized":"2024-09-17T07:27:19.461920+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:27:05.654178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:24:21.575574+0000","last_clean_scrub_stamp":"2024-09-17T07:27:05.654178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:09.946587+0000","scrub_duration":0.001340826,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"439'27","reported_seq":642,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:24.733154+0000","last_change":"2024-09-17T07:27:24.733247+0000","last_active":"2024-09-17T07:27:24.733154+0000","last_peered":"2024-09-17T07:27:24.733154+0000","last_clean":"2024-09-17T07:27:24.733154+0000","last_became_active":"2024-09-17T07:21:15.347179+0000","last_became_peered":"2024-09-17T07:21:15.347179+0000","last_unstale":"2024-09-17T07:27:24.733154+0000","last_undegraded":"2024-09-17T07:27:24.733154+0000","last_fullsized":"2024-09-17T07:27:24.733154+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'27","last_scrub_stamp":"2024-09-17T07:27:24.733138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:27:24.733138+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:52.167284+0000","scrub_duration":0.0023121819999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"94'2","reported_seq":480,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:19.463085+0000","last_change":"2024-09-17T07:26:51.157175+0000","last_active":"2024-09-17T07:27:19.463085+0000","last_peered":"2024-09-17T07:27:19.463085+0000","last_clean":"2024-09-17T07:27:19.463085+0000","last_became_active":"2024-09-17T07:23:57.521742+0000","last_became_peered":"2024-09-17T07:23:57.521742+0000","last_unstale":"2024-09-17T07:27:19.463085+0000","last_undegraded":"2024-09-17T07:27:19.463085+0000","last_fullsized":"2024-09-17T07:27:19.463085+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-09-17T07:26:51.157081+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-09-17T07:07:31.717888+0000","last_clean_scrub_stamp":"2024-09-17T07:26:51.157081+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:52.690749+0000","scrub_duration":0.0018815710000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":493,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:19.462697+0000","last_change":"2024-09-17T07:27:03.111480+0000","last_active":"2024-09-17T07:27:19.462697+0000","last_peered":"2024-09-17T07:27:19.462697+0000","last_clean":"2024-09-17T07:27:19.462697+0000","last_became_active":"2024-09-17T07:25:49.432316+0000","last_became_peered":"2024-09-17T07:25:49.432316+0000","last_unstale":"2024-09-17T07:27:19.462697+0000","last_undegraded":"2024-09-17T07:27:19.462697+0000","last_fullsized":"2024-09-17T07:27:19.462697+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:27:03.111364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:25:46.957486+0000","last_clean_scrub_stamp":"2024-09-17T07:27:03.111364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:13.283055+0000","scrub_duration":0.001914652,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":62,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:26.779636+0000","last_change":"2024-09-17T07:27:26.779711+0000","last_active":"2024-09-17T07:27:26.779636+0000","last_peered":"2024-09-17T07:27:26.779636+0000","last_clean":"2024-09-17T07:27:26.779636+0000","last_became_active":"2024-09-17T07:25:49.452253+0000","last_became_peered":"2024-09-17T07:25:49.452253+0000","last_unstale":"2024-09-17T07:27:26.779636+0000","last_undegraded":"2024-09-17T07:27:26.779636+0000","last_fullsized":"2024-09-17T07:27:26.779636+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:27:26.779620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:27:26.779620+0000","last_clean_scrub_stamp":"2024-09-17T07:27:26.779620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:55.697162+0000","scrub_duration":0.0016900750000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"439'12","reported_seq":57,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.758944+0000","last_change":"2024-09-17T07:26:44.701243+0000","last_active":"2024-09-17T07:27:20.758944+0000","last_peered":"2024-09-17T07:27:20.758944+0000","last_clean":"2024-09-17T07:27:20.758944+0000","last_became_active":"2024-09-17T07:25:49.443742+0000","last_became_peered":"2024-09-17T07:25:49.443742+0000","last_unstale":"2024-09-17T07:27:20.758944+0000","last_undegraded":"2024-09-17T07:27:20.758944+0000","last_fullsized":"2024-09-17T07:27:20.758944+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'12","last_scrub_stamp":"2024-09-17T07:26:44.701161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:26:44.701161+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:45.007153+0000","scrub_duration":0.0017795459999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":565,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:27.786866+0000","last_change":"2024-09-17T07:27:27.786944+0000","last_active":"2024-09-17T07:27:27.786866+0000","last_peered":"2024-09-17T07:27:27.786866+0000","last_clean":"2024-09-17T07:27:27.786866+0000","last_became_active":"2024-09-17T07:23:57.476234+0000","last_became_peered":"2024-09-17T07:23:57.476234+0000","last_unstale":"2024-09-17T07:27:27.786866+0000","last_undegraded":"2024-09-17T07:27:27.786866+0000","last_fullsized":"2024-09-17T07:27:27.786866+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:27:27.786850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:27:27.786850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:41.961780+0000","scrub_duration":0.001364259,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":633,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.806685+0000","last_change":"2024-09-17T07:26:21.240109+0000","last_active":"2024-09-17T07:27:20.806685+0000","last_peered":"2024-09-17T07:27:20.806685+0000","last_clean":"2024-09-17T07:27:20.806685+0000","last_became_active":"2024-09-17T07:22:13.582229+0000","last_became_peered":"2024-09-17T07:22:13.582229+0000","last_unstale":"2024-09-17T07:27:20.806685+0000","last_undegraded":"2024-09-17T07:27:20.806685+0000","last_fullsized":"2024-09-17T07:27:20.806685+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:21.240027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:26:21.240027+0000","last_clean_scrub_stamp":"2024-09-17T07:26:21.240027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:36.673199+0000","scrub_duration":0.0012918459999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":145,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.387108+0000","last_change":"2024-09-17T07:27:00.694126+0000","last_active":"2024-09-17T07:27:20.387108+0000","last_peered":"2024-09-17T07:27:20.387108+0000","last_clean":"2024-09-17T07:27:20.387108+0000","last_became_active":"2024-09-17T07:22:13.624127+0000","last_became_peered":"2024-09-17T07:22:13.624127+0000","last_unstale":"2024-09-17T07:27:20.387108+0000","last_undegraded":"2024-09-17T07:27:20.387108+0000","last_fullsized":"2024-09-17T07:27:20.387108+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:27:00.694008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:27:00.694008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:01.315911+0000","scrub_duration":0.0020916659999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"439'27","reported_seq":63,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.759152+0000","last_change":"2024-09-17T07:27:08.703830+0000","last_active":"2024-09-17T07:27:20.759152+0000","last_peered":"2024-09-17T07:27:20.759152+0000","last_clean":"2024-09-17T07:27:20.759152+0000","last_became_active":"2024-09-17T07:25:49.445624+0000","last_became_peered":"2024-09-17T07:25:49.445624+0000","last_unstale":"2024-09-17T07:27:20.759152+0000","last_undegraded":"2024-09-17T07:27:20.759152+0000","last_fullsized":"2024-09-17T07:27:20.759152+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'27","last_scrub_stamp":"2024-09-17T07:27:08.703749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:27:08.703749+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:22.595218+0000","scrub_duration":0.002519126,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":643,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.807873+0000","last_change":"2024-09-17T07:27:10.086724+0000","last_active":"2024-09-17T07:27:20.807873+0000","last_peered":"2024-09-17T07:27:20.807873+0000","last_clean":"2024-09-17T07:27:20.807873+0000","last_became_active":"2024-09-17T07:23:57.486575+0000","last_became_peered":"2024-09-17T07:23:57.486575+0000","last_unstale":"2024-09-17T07:27:20.807873+0000","last_undegraded":"2024-09-17T07:27:20.807873+0000","last_fullsized":"2024-09-17T07:27:20.807873+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:27:10.086620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:27:10.086620+0000","last_clean_scrub_stamp":"2024-09-17T07:27:10.086620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:12.618485+0000","scrub_duration":0.001917787,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":145,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.387094+0000","last_change":"2024-09-17T07:27:09.706799+0000","last_active":"2024-09-17T07:27:20.387094+0000","last_peered":"2024-09-17T07:27:20.387094+0000","last_clean":"2024-09-17T07:27:20.387094+0000","last_became_active":"2024-09-17T07:22:13.710754+0000","last_became_peered":"2024-09-17T07:22:13.710754+0000","last_unstale":"2024-09-17T07:27:20.387094+0000","last_undegraded":"2024-09-17T07:27:20.387094+0000","last_fullsized":"2024-09-17T07:27:20.387094+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:27:09.706706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:27:09.706706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:15.957958+0000","scrub_duration":0.001413168,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":51,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.759493+0000","last_change":"2024-09-17T07:26:20.900934+0000","last_active":"2024-09-17T07:27:20.759493+0000","last_peered":"2024-09-17T07:27:20.759493+0000","last_clean":"2024-09-17T07:27:20.759493+0000","last_became_active":"2024-09-17T07:25:49.435113+0000","last_became_peered":"2024-09-17T07:25:49.435113+0000","last_unstale":"2024-09-17T07:27:20.759493+0000","last_undegraded":"2024-09-17T07:27:20.759493+0000","last_fullsized":"2024-09-17T07:27:20.759493+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:20.900861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:25:17.663576+0000","last_clean_scrub_stamp":"2024-09-17T07:26:20.900861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:34.485626+0000","scrub_duration":0.001480269,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"326'2","reported_seq":164,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.387335+0000","last_change":"2024-09-17T07:27:14.727832+0000","last_active":"2024-09-17T07:27:20.387335+0000","last_peered":"2024-09-17T07:27:20.387335+0000","last_clean":"2024-09-17T07:27:20.387335+0000","last_became_active":"2024-09-17T07:23:57.475630+0000","last_became_peered":"2024-09-17T07:23:57.475630+0000","last_unstale":"2024-09-17T07:27:20.387335+0000","last_undegraded":"2024-09-17T07:27:20.387335+0000","last_fullsized":"2024-09-17T07:27:20.387335+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'2","last_scrub_stamp":"2024-09-17T07:27:14.727652+0000","last_deep_scrub":"326'2","last_deep_scrub_stamp":"2024-09-17T07:17:36.974358+0000","last_clean_scrub_stamp":"2024-09-17T07:27:14.727652+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:21.020923+0000","scrub_duration":0.001828327,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":603,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.806588+0000","last_change":"2024-09-17T07:26:44.461021+0000","last_active":"2024-09-17T07:27:20.806588+0000","last_peered":"2024-09-17T07:27:20.806588+0000","last_clean":"2024-09-17T07:27:20.806588+0000","last_became_active":"2024-09-17T07:25:50.426927+0000","last_became_peered":"2024-09-17T07:25:50.426927+0000","last_unstale":"2024-09-17T07:27:20.806588+0000","last_undegraded":"2024-09-17T07:27:20.806588+0000","last_fullsized":"2024-09-17T07:27:20.806588+0000","mapping_epoch":516,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":517,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:44.460914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:26:44.460914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:13.609258+0000","scrub_duration":0.001578243,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"439'35","reported_seq":788,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:19.462461+0000","last_change":"2024-09-17T07:26:39.786707+0000","last_active":"2024-09-17T07:27:19.462461+0000","last_peered":"2024-09-17T07:27:19.462461+0000","last_clean":"2024-09-17T07:27:19.462461+0000","last_became_active":"2024-09-17T07:25:49.431783+0000","last_became_peered":"2024-09-17T07:25:49.431783+0000","last_unstale":"2024-09-17T07:27:19.462461+0000","last_undegraded":"2024-09-17T07:27:19.462461+0000","last_fullsized":"2024-09-17T07:27:19.462461+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'35","last_scrub_stamp":"2024-09-17T07:26:39.786542+0000","last_deep_scrub":"294'25","last_deep_scrub_stamp":"2024-09-17T07:16:00.471311+0000","last_clean_scrub_stamp":"2024-09-17T07:26:39.786542+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:51.502296+0000","scrub_duration":0.0022343789999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"525'11","reported_seq":970,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:25.805229+0000","last_change":"2024-09-17T07:27:25.239869+0000","last_active":"2024-09-17T07:27:25.805229+0000","last_peered":"2024-09-17T07:27:25.805229+0000","last_clean":"2024-09-17T07:27:25.805229+0000","last_became_active":"2024-09-17T07:22:13.574012+0000","last_became_peered":"2024-09-17T07:22:13.574012+0000","last_unstale":"2024-09-17T07:27:25.805229+0000","last_undegraded":"2024-09-17T07:27:25.805229+0000","last_fullsized":"2024-09-17T07:27:25.805229+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"525'11","last_scrub_stamp":"2024-09-17T07:27:25.239753+0000","last_deep_scrub":"525'11","last_deep_scrub_stamp":"2024-09-17T07:26:10.287750+0000","last_clean_scrub_stamp":"2024-09-17T07:27:25.239753+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:26.930369+0000","scrub_duration":0.001785341,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":477,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:23.380349+0000","last_change":"2024-09-17T07:27:23.380410+0000","last_active":"2024-09-17T07:27:23.380349+0000","last_peered":"2024-09-17T07:27:23.380349+0000","last_clean":"2024-09-17T07:27:23.380349+0000","last_became_active":"2024-09-17T07:23:57.478429+0000","last_became_peered":"2024-09-17T07:23:57.478429+0000","last_unstale":"2024-09-17T07:27:23.380349+0000","last_undegraded":"2024-09-17T07:27:23.380349+0000","last_fullsized":"2024-09-17T07:27:23.380349+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:27:23.380337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:26:20.318401+0000","last_clean_scrub_stamp":"2024-09-17T07:27:23.380337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:26.187066+0000","scrub_duration":0.0014809980000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":62,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.759242+0000","last_change":"2024-09-17T07:27:10.770333+0000","last_active":"2024-09-17T07:27:20.759242+0000","last_peered":"2024-09-17T07:27:20.759242+0000","last_clean":"2024-09-17T07:27:20.759242+0000","last_became_active":"2024-09-17T07:25:49.436539+0000","last_became_peered":"2024-09-17T07:25:49.436539+0000","last_unstale":"2024-09-17T07:27:20.759242+0000","last_undegraded":"2024-09-17T07:27:20.759242+0000","last_fullsized":"2024-09-17T07:27:20.759242+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:27:10.770256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:10:29.325664+0000","last_clean_scrub_stamp":"2024-09-17T07:27:10.770256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:35.082073+0000","scrub_duration":0.001687001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"513'32","reported_seq":733,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.807822+0000","last_change":"2024-09-17T07:27:03.038245+0000","last_active":"2024-09-17T07:27:20.807822+0000","last_peered":"2024-09-17T07:27:20.807822+0000","last_clean":"2024-09-17T07:27:20.807822+0000","last_became_active":"2024-09-17T07:23:57.486951+0000","last_became_peered":"2024-09-17T07:23:57.486951+0000","last_unstale":"2024-09-17T07:27:20.807822+0000","last_undegraded":"2024-09-17T07:27:20.807822+0000","last_fullsized":"2024-09-17T07:27:20.807822+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"513'32","last_scrub_stamp":"2024-09-17T07:27:03.038077+0000","last_deep_scrub":"513'32","last_deep_scrub_stamp":"2024-09-17T07:25:31.401365+0000","last_clean_scrub_stamp":"2024-09-17T07:27:03.038077+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:28.405884+0000","scrub_duration":0.0029557210000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":642,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:19.462428+0000","last_change":"2024-09-17T07:26:55.734925+0000","last_active":"2024-09-17T07:27:19.462428+0000","last_peered":"2024-09-17T07:27:19.462428+0000","last_clean":"2024-09-17T07:27:19.462428+0000","last_became_active":"2024-09-17T07:24:43.779553+0000","last_became_peered":"2024-09-17T07:24:43.779553+0000","last_unstale":"2024-09-17T07:27:19.462428+0000","last_undegraded":"2024-09-17T07:27:19.462428+0000","last_fullsized":"2024-09-17T07:27:19.462428+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:55.734820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:20:37.435106+0000","last_clean_scrub_stamp":"2024-09-17T07:26:55.734820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:00.306732+0000","scrub_duration":0.0017518200000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":619,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:19.463161+0000","last_change":"2024-09-17T07:26:28.848603+0000","last_active":"2024-09-17T07:27:19.463161+0000","last_peered":"2024-09-17T07:27:19.463161+0000","last_clean":"2024-09-17T07:27:19.463161+0000","last_became_active":"2024-09-17T07:23:57.473629+0000","last_became_peered":"2024-09-17T07:23:57.473629+0000","last_unstale":"2024-09-17T07:27:19.463161+0000","last_undegraded":"2024-09-17T07:27:19.463161+0000","last_fullsized":"2024-09-17T07:27:19.463161+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:28.848506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:26:28.848506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:53.167436+0000","scrub_duration":0.0036431800000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":668,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:30.189709+0000","last_change":"2024-09-17T07:27:30.189816+0000","last_active":"2024-09-17T07:27:30.189709+0000","last_peered":"2024-09-17T07:27:30.189709+0000","last_clean":"2024-09-17T07:27:30.189709+0000","last_became_active":"2024-09-17T07:22:13.587273+0000","last_became_peered":"2024-09-17T07:22:13.587273+0000","last_unstale":"2024-09-17T07:27:30.189709+0000","last_undegraded":"2024-09-17T07:27:30.189709+0000","last_fullsized":"2024-09-17T07:27:30.189709+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:27:30.189691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:11:18.559877+0000","last_clean_scrub_stamp":"2024-09-17T07:27:30.189691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:38.380119+0000","scrub_duration":0.0014929850000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"439'39","reported_seq":497,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:23.881473+0000","last_change":"2024-09-17T07:27:23.881546+0000","last_active":"2024-09-17T07:27:23.881473+0000","last_peered":"2024-09-17T07:27:23.881473+0000","last_clean":"2024-09-17T07:27:23.881473+0000","last_became_active":"2024-09-17T07:26:09.602683+0000","last_became_peered":"2024-09-17T07:26:09.602683+0000","last_unstale":"2024-09-17T07:27:23.881473+0000","last_undegraded":"2024-09-17T07:27:23.881473+0000","last_fullsized":"2024-09-17T07:27:23.881473+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'39","last_scrub_stamp":"2024-09-17T07:27:23.881457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:27:23.881457+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:50.761803+0000","scrub_duration":0.0020809660000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":441,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:33.293565+0000","last_change":"2024-09-17T07:27:33.293627+0000","last_active":"2024-09-17T07:27:33.293565+0000","last_peered":"2024-09-17T07:27:33.293565+0000","last_clean":"2024-09-17T07:27:33.293565+0000","last_became_active":"2024-09-17T07:21:15.348754+0000","last_became_peered":"2024-09-17T07:21:15.348754+0000","last_unstale":"2024-09-17T07:27:33.293565+0000","last_undegraded":"2024-09-17T07:27:33.293565+0000","last_fullsized":"2024-09-17T07:27:33.293565+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:27:33.293552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:24:23.943437+0000","last_clean_scrub_stamp":"2024-09-17T07:27:33.293552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:50.167390+0000","scrub_duration":0.0017582489999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":51,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.759841+0000","last_change":"2024-09-17T07:26:42.663839+0000","last_active":"2024-09-17T07:27:20.759841+0000","last_peered":"2024-09-17T07:27:20.759841+0000","last_clean":"2024-09-17T07:27:20.759841+0000","last_became_active":"2024-09-17T07:25:49.452257+0000","last_became_peered":"2024-09-17T07:25:49.452257+0000","last_unstale":"2024-09-17T07:27:20.759841+0000","last_undegraded":"2024-09-17T07:27:20.759841+0000","last_fullsized":"2024-09-17T07:27:20.759841+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:42.663753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:25:18.515421+0000","last_clean_scrub_stamp":"2024-09-17T07:26:42.663753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:11.481959+0000","scrub_duration":0.0016051360000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":62,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.759392+0000","last_change":"2024-09-17T07:27:17.770600+0000","last_active":"2024-09-17T07:27:20.759392+0000","last_peered":"2024-09-17T07:27:20.759392+0000","last_clean":"2024-09-17T07:27:20.759392+0000","last_became_active":"2024-09-17T07:25:49.445492+0000","last_became_peered":"2024-09-17T07:25:49.445492+0000","last_unstale":"2024-09-17T07:27:20.759392+0000","last_undegraded":"2024-09-17T07:27:20.759392+0000","last_fullsized":"2024-09-17T07:27:20.759392+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:27:17.770516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:24:26.807790+0000","last_clean_scrub_stamp":"2024-09-17T07:27:17.770516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:23.179623+0000","scrub_duration":0.001354919,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"439'29","reported_seq":74,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:22.774038+0000","last_change":"2024-09-17T07:27:22.774113+0000","last_active":"2024-09-17T07:27:22.774038+0000","last_peered":"2024-09-17T07:27:22.774038+0000","last_clean":"2024-09-17T07:27:22.774038+0000","last_became_active":"2024-09-17T07:25:49.448763+0000","last_became_peered":"2024-09-17T07:25:49.448763+0000","last_unstale":"2024-09-17T07:27:22.774038+0000","last_undegraded":"2024-09-17T07:27:22.774038+0000","last_fullsized":"2024-09-17T07:27:22.774038+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'29","last_scrub_stamp":"2024-09-17T07:27:22.774022+0000","last_deep_scrub":"86'17","last_deep_scrub_stamp":"2024-09-17T07:03:18.241067+0000","last_clean_scrub_stamp":"2024-09-17T07:27:22.774022+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:46.663010+0000","scrub_duration":0.0024735180000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":651,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:19.462966+0000","last_change":"2024-09-17T07:26:44.788543+0000","last_active":"2024-09-17T07:27:19.462966+0000","last_peered":"2024-09-17T07:27:19.462966+0000","last_clean":"2024-09-17T07:27:19.462966+0000","last_became_active":"2024-09-17T07:25:49.439685+0000","last_became_peered":"2024-09-17T07:25:49.439685+0000","last_unstale":"2024-09-17T07:27:19.462966+0000","last_undegraded":"2024-09-17T07:27:19.462966+0000","last_fullsized":"2024-09-17T07:27:19.462966+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:44.788431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:26:44.788431+0000","last_clean_scrub_stamp":"2024-09-17T07:26:44.788431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:00.018116+0000","scrub_duration":0.001656842,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":395,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101,"ondisk_log_size":101,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1534,"num_read_kb":1291,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":328,"num_bytes_recovered":6744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1042,"ondisk_log_size":1042,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":384,"num_read_kb":379,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1386,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":12,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":4158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1430,"ondisk_log_size":1430,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698038071,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1115060,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":1113982,"kb_avail":92630092,"statfs":{"total":95995035648,"available":94853214208,"internally_reserved":0,"allocated":1093632,"data_stored":700777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":1140718027},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":1.2889999999999999,"15min":1.2949999999999999},"min":{"1min":0.40100000000000002,"5min":0.182,"15min":0.182},"max":{"1min":0.79000000000000004,"5min":36.659999999999997,"15min":43.905000000000001},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.50700000000000001,"5min":1.1759999999999999,"15min":1.2330000000000001},"min":{"1min":0.34300000000000003,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.68799999999999994,"5min":36.555,"15min":44.432000000000002},"last":0.59299999999999997}]},{"osd":1,"last update":"Tue Sep 17 07:27:26 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.86199999999999999,"15min":1.1220000000000001},"min":{"1min":0.32400000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.67300000000000004,"5min":12.112,"15min":12.112},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.83099999999999996,"15min":1.0900000000000001},"min":{"1min":0.41899999999999998,"5min":0.216,"15min":0.216},"max":{"1min":0.65100000000000002,"5min":12.273,"15min":12.273},"last":0.63100000000000001}]},{"osd":2,"last update":"Tue Sep 17 07:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.80500000000000005,"15min":0.96999999999999997},"min":{"1min":0.372,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.76100000000000001,"5min":10.972,"15min":10.972},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.92000000000000004,"15min":0.997},"min":{"1min":0.40600000000000003,"5min":0.26500000000000001,"15min":0.224},"max":{"1min":0.85599999999999998,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.61599999999999999}]},{"osd":3,"last update":"Tue Sep 17 07:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.69499999999999995,"15min":0.93700000000000006},"min":{"1min":0.33800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.76000000000000001,"5min":4.2009999999999996,"15min":10.122999999999999},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.65700000000000003,"15min":0.90700000000000003},"min":{"1min":0.36299999999999999,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.81499999999999995,"5min":4.1719999999999997,"15min":12.465999999999999},"last":0.54400000000000004}]},{"osd":4,"last update":"Tue Sep 17 07:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.751,"15min":1.042},"min":{"1min":0.34599999999999997,"5min":0.22900000000000001,"15min":0.222},"max":{"1min":0.70799999999999996,"5min":13.316000000000001,"15min":21.140999999999998},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.80300000000000005,"15min":1.1930000000000001},"min":{"1min":0.27200000000000002,"5min":0.25800000000000001,"15min":0.23599999999999999},"max":{"1min":0.83699999999999997,"5min":12.272,"15min":19.420000000000002},"last":0.47599999999999998}]},{"osd":5,"last update":"Tue Sep 17 07:27:01 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.77400000000000002,"15min":0.91600000000000004},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.90100000000000002,"5min":3.0859999999999999,"15min":3.0859999999999999},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.504,"5min":0.85799999999999998,"15min":1.0169999999999999},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.73599999999999999,"5min":5.7350000000000003,"15min":5.7350000000000003},"last":0.46300000000000002}]},{"osd":6,"last update":"Tue Sep 17 07:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":1.4139999999999999,"15min":1.446},"min":{"1min":0.33600000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.77900000000000003,"5min":60.445999999999998,"15min":60.445999999999998},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.54800000000000004,"5min":1.49,"15min":1.544},"min":{"1min":0.32700000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.78500000000000003,"5min":60.332000000000001,"15min":60.332000000000001},"last":0.42599999999999999}]}]},{"osd":6,"up_from":50,"seq":214748365114,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1036772,"kb_used_data":732,"kb_used_omap":1,"kb_used_meta":1036030,"kb_avail":92708380,"statfs":{"total":95995035648,"available":94933381120,"internally_reserved":0,"allocated":749568,"data_stored":372404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1060895175},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:26:34 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":3.2189999999999999,"15min":2.339},"min":{"1min":0.32300000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.74099999999999999,"5min":110.376,"15min":126.902},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.51900000000000002,"5min":3.4009999999999998,"15min":2.2519999999999998},"min":{"1min":0.41199999999999998,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.64500000000000002,"5min":110.164,"15min":127.06399999999999},"last":0.58899999999999997}]},{"osd":1,"last update":"Tue Sep 17 07:27:30 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":2.7450000000000001,"15min":4.9470000000000001},"min":{"1min":0.40500000000000003,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.69199999999999995,"5min":199.32400000000001,"15min":199.32400000000001},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.52500000000000002,"5min":2.702,"15min":5.1769999999999996},"min":{"1min":0.432,"5min":0.251,"15min":0.24099999999999999},"max":{"1min":0.67100000000000004,"5min":198.88499999999999,"15min":198.88499999999999},"last":0.51600000000000001}]},{"osd":2,"last update":"Tue Sep 17 07:26:34 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.81699999999999995,"15min":1.159},"min":{"1min":0.38600000000000001,"5min":0.23300000000000001,"15min":0.219},"max":{"1min":0.67500000000000004,"5min":10.930999999999999,"15min":36.289000000000001},"last":0.501},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.96699999999999997,"15min":1.101},"min":{"1min":0.36399999999999999,"5min":0.26100000000000001,"15min":0.22900000000000001},"max":{"1min":0.81899999999999995,"5min":7.1280000000000001,"15min":36.326999999999998},"last":0.63400000000000001}]},{"osd":3,"last update":"Tue Sep 17 07:26:34 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.79200000000000004,"15min":0.873},"min":{"1min":0.34300000000000003,"5min":0.27700000000000002,"15min":0.249},"max":{"1min":0.80600000000000005,"5min":9.0579999999999998,"15min":9.0579999999999998},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.875,"15min":1.0109999999999999},"min":{"1min":0.40500000000000003,"5min":0.28000000000000003,"15min":0.27500000000000002},"max":{"1min":0.72899999999999998,"5min":10.999000000000001,"15min":10.999000000000001},"last":0.56399999999999995}]},{"osd":4,"last update":"Tue Sep 17 07:26:34 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":1.0089999999999999,"15min":1.1339999999999999},"min":{"1min":0.30399999999999999,"5min":0.24199999999999999,"15min":0.19900000000000001},"max":{"1min":0.68700000000000006,"5min":9.5419999999999998,"15min":9.5419999999999998},"last":0.373},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.80400000000000005,"15min":1.141},"min":{"1min":0.29299999999999998,"5min":0.26100000000000001,"15min":0.23300000000000001},"max":{"1min":0.69399999999999995,"5min":9.0310000000000006,"15min":9.0310000000000006},"last":0.48099999999999998}]},{"osd":5,"last update":"Tue Sep 17 07:26:59 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":1.472,"15min":1.909},"min":{"1min":0.307,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.72299999999999998,"5min":13.551,"15min":13.551},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.52700000000000002,"5min":1.637,"15min":2.137},"min":{"1min":0.33200000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.72399999999999998,"5min":8.1720000000000006,"15min":8.1720000000000006},"last":0.54600000000000004}]},{"osd":7,"last update":"Tue Sep 17 07:26:54 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.94299999999999995,"15min":1.6599999999999999},"min":{"1min":0.33900000000000002,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.77100000000000002,"5min":9.1199999999999992,"15min":131.95699999999999},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.60699999999999998,"5min":1.0720000000000001,"15min":1.6000000000000001},"min":{"1min":0.38900000000000001,"5min":0.29499999999999998,"15min":0.252},"max":{"1min":0.75600000000000001,"5min":9.5850000000000009,"15min":131.833},"last":0.61299999999999999}]}]},{"osd":5,"up_from":476,"seq":2044404432941,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":122668,"kb_used_data":1316,"kb_used_omap":4,"kb_used_meta":121339,"kb_avail":93622484,"statfs":{"total":95995035648,"available":95869423616,"internally_reserved":0,"allocated":1347584,"data_stored":963582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4441,"internal_metadata":124251815},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.82199999999999995,"15min":0.94999999999999996},"min":{"1min":0.372,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.73599999999999999,"5min":3.4220000000000002,"15min":3.4220000000000002},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.50800000000000001,"5min":1.169,"15min":1.4530000000000001},"min":{"1min":0.33000000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.67500000000000004,"5min":6.7619999999999996,"15min":6.7619999999999996},"last":0.44400000000000001}]},{"osd":1,"last update":"Tue Sep 17 07:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.89900000000000002,"15min":1.0609999999999999},"min":{"1min":0.27700000000000002,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.75,"5min":3.476,"15min":3.476},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.85599999999999998,"15min":0.98299999999999998},"min":{"1min":0.317,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":3.0049999999999999,"5min":6.665,"15min":6.665},"last":0.46999999999999997}]},{"osd":2,"last update":"Tue Sep 17 07:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.94399999999999995,"15min":1.127},"min":{"1min":0.33800000000000002,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.71399999999999997,"5min":3.544,"15min":3.544},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.78900000000000003,"15min":0.90700000000000003},"min":{"1min":0.39000000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.77000000000000002,"5min":2.8359999999999999,"15min":2.8359999999999999},"last":0.42099999999999999}]},{"osd":3,"last update":"Tue Sep 17 07:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.76100000000000001,"15min":0.871},"min":{"1min":0.35399999999999998,"5min":0.314,"15min":0.314},"max":{"1min":0.69599999999999995,"5min":3.7040000000000002,"15min":3.7040000000000002},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.75700000000000001,"15min":0.84599999999999997},"min":{"1min":0.41699999999999998,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.73099999999999998,"5min":2.8719999999999999,"15min":2.8719999999999999},"last":0.51000000000000001}]},{"osd":4,"last update":"Tue Sep 17 07:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.82199999999999995,"15min":0.98199999999999998},"min":{"1min":0.318,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.69899999999999995,"5min":3.4399999999999999,"15min":3.4399999999999999},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.94299999999999995,"15min":1.1539999999999999},"min":{"1min":0.29899999999999999,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.68799999999999994,"5min":3.0139999999999998,"15min":3.0139999999999998},"last":0.55800000000000005}]},{"osd":6,"last update":"Tue Sep 17 07:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":1.2490000000000001,"15min":1.5960000000000001},"min":{"1min":0.28299999999999997,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.73599999999999999,"5min":10.92,"15min":10.92},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.52100000000000002,"5min":1.113,"15min":1.3879999999999999},"min":{"1min":0.35499999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.69899999999999995,"5min":10.202999999999999,"15min":10.202999999999999},"last":0.59899999999999998}]},{"osd":7,"last update":"Tue Sep 17 07:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":2.2400000000000002,"15min":3.0009999999999999},"min":{"1min":0.30199999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.72099999999999997,"5min":45.555,"15min":45.555},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.53200000000000003,"5min":2.4220000000000002,"15min":3.2679999999999998},"min":{"1min":0.33900000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.71999999999999997,"5min":47.213999999999999,"15min":47.213999999999999},"last":0.54600000000000004}]}]},{"osd":4,"up_from":33,"seq":141733921093,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1002356,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":1001598,"kb_avail":92742796,"statfs":{"total":95995035648,"available":94968623104,"internally_reserved":0,"allocated":765952,"data_stored":372528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1025636806},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":1.601,"15min":1.319},"min":{"1min":0.30499999999999999,"5min":0.20300000000000001,"15min":0.20000000000000001},"max":{"1min":0.65600000000000003,"5min":39.473999999999997,"15min":39.473999999999997},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.60499999999999998,"5min":1.631,"15min":1.3700000000000001},"min":{"1min":0.41299999999999998,"5min":0.21299999999999999,"15min":0.20499999999999999},"max":{"1min":0.73899999999999999,"5min":39.654000000000003,"15min":39.654000000000003},"last":0.45300000000000001}]},{"osd":1,"last update":"Tue Sep 17 07:27:26 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":1.1439999999999999,"15min":2.302},"min":{"1min":0.35499999999999998,"5min":0.221,"15min":0.221},"max":{"1min":0.65100000000000002,"5min":39.439,"15min":39.439},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.54800000000000004,"5min":1.3069999999999999,"15min":2.3719999999999999},"min":{"1min":0.39700000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.73199999999999998,"5min":39.325000000000003,"15min":39.325000000000003},"last":0.53900000000000003}]},{"osd":2,"last update":"Tue Sep 17 07:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":2.391,"15min":1.4910000000000001},"min":{"1min":0.371,"5min":0.24399999999999999,"15min":0.23499999999999999},"max":{"1min":7.1539999999999999,"5min":126.628,"15min":126.628},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.61399999999999999,"5min":2.3530000000000002,"15min":1.476},"min":{"1min":0.41799999999999998,"5min":0.27400000000000002,"15min":0.23400000000000001},"max":{"1min":0.76100000000000001,"5min":126.41200000000001,"15min":126.41200000000001},"last":0.67200000000000004}]},{"osd":3,"last update":"Tue Sep 17 07:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.66700000000000004,"15min":0.85299999999999998},"min":{"1min":0.41699999999999998,"5min":0.28299999999999997,"15min":0.25800000000000001},"max":{"1min":0.71799999999999997,"5min":2.8929999999999998,"15min":15.433999999999999},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.72099999999999997,"15min":0.91200000000000003},"min":{"1min":0.35199999999999998,"5min":0.29699999999999999,"15min":0.28100000000000003},"max":{"1min":0.68799999999999994,"5min":8.7360000000000007,"15min":16.111000000000001},"last":0.47699999999999998}]},{"osd":5,"last update":"Tue Sep 17 07:27:04 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":1.038,"15min":1.3029999999999999},"min":{"1min":0.33800000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.60899999999999999,"5min":13.128,"15min":13.128},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.46400000000000002,"5min":1.2370000000000001,"15min":1.5800000000000001},"min":{"1min":0.32400000000000001,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.622,"5min":13.218999999999999,"15min":13.218999999999999},"last":0.497}]},{"osd":6,"last update":"Tue Sep 17 07:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.72899999999999998,"15min":1.01},"min":{"1min":0.40300000000000002,"5min":0.249,"15min":0.23400000000000001},"max":{"1min":0.69499999999999995,"5min":4.7729999999999997,"15min":8.9619999999999997},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.82799999999999996,"15min":1.1419999999999999},"min":{"1min":0.30599999999999999,"5min":0.249,"15min":0.24199999999999999},"max":{"1min":0.64300000000000002,"5min":6.782,"15min":10.051},"last":0.52100000000000002}]},{"osd":7,"last update":"Tue Sep 17 07:26:47 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.93100000000000005,"15min":1.123},"min":{"1min":0.372,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.68700000000000006,"5min":6.2249999999999996,"15min":7.1550000000000002},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.72599999999999998,"15min":1.1220000000000001},"min":{"1min":0.38900000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.65700000000000003,"5min":4.7439999999999998,"15min":9.0030000000000001},"last":0.51200000000000001}]}]},{"osd":3,"up_from":27,"seq":115964117319,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":371280,"kb_used_data":1352,"kb_used_omap":1,"kb_used_meta":369918,"kb_avail":93373872,"statfs":{"total":95995035648,"available":95614844928,"internally_reserved":0,"allocated":1384448,"data_stored":963680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":378796488},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:27:33 2024","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":1.091,"15min":0.99099999999999999},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.57099999999999995,"5min":46.442,"15min":65.192999999999998},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.315,"5min":1.1319999999999999,"15min":1.0309999999999999},"min":{"1min":0.20399999999999999,"5min":0.188,"15min":0.182},"max":{"1min":0.52100000000000002,"5min":46.633000000000003,"15min":64.998000000000005},"last":0.28999999999999998}]},{"osd":1,"last update":"Tue Sep 17 07:27:26 2024","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.54100000000000004,"15min":0.90800000000000003},"min":{"1min":0.218,"5min":0.20499999999999999,"15min":0.20200000000000001},"max":{"1min":0.47299999999999998,"5min":6.1719999999999997,"15min":6.1719999999999997},"last":0.223},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.64600000000000002,"15min":0.78900000000000003},"min":{"1min":0.23300000000000001,"5min":0.214,"15min":0.214},"max":{"1min":0.54700000000000004,"5min":8.1489999999999991,"15min":8.1489999999999991},"last":0.39000000000000001}]},{"osd":2,"last update":"Tue Sep 17 07:27:33 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.56499999999999995,"15min":0.52100000000000002},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.188},"max":{"1min":0.55200000000000005,"5min":8.0609999999999999,"15min":8.0609999999999999},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.33200000000000002,"5min":0.51500000000000001,"15min":0.52100000000000002},"min":{"1min":0.25700000000000001,"5min":0.23200000000000001,"15min":0.20599999999999999},"max":{"1min":0.498,"5min":2.899,"15min":5.5389999999999997},"last":0.315}]},{"osd":4,"last update":"Tue Sep 17 07:27:33 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.80700000000000005,"15min":1.5129999999999999},"min":{"1min":0.34300000000000003,"5min":0.28100000000000003,"15min":0.27700000000000002},"max":{"1min":0.72799999999999998,"5min":16.146000000000001,"15min":61.639000000000003},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.91500000000000004,"15min":1.486},"min":{"1min":0.39200000000000002,"5min":0.31,"15min":0.27200000000000002},"max":{"1min":0.876,"5min":16.129999999999999,"15min":61.036000000000001},"last":0.437}]},{"osd":5,"last update":"Tue Sep 17 07:27:08 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.56100000000000005,"15min":0.55100000000000005},"min":{"1min":0.437,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.84099999999999997,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.60499999999999998,"15min":0.60599999999999998},"min":{"1min":0.47099999999999997,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.85099999999999998,"5min":1.417,"15min":1.417},"last":0.37}]},{"osd":6,"last update":"Tue Sep 17 07:27:33 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.63700000000000001,"15min":1.127},"min":{"1min":0.378,"5min":0.316,"15min":0.29699999999999999},"max":{"1min":0.70999999999999996,"5min":3.7599999999999998,"15min":68.188999999999993},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.81599999999999995,"15min":1.264},"min":{"1min":0.32900000000000001,"5min":0.32300000000000001,"15min":0.26300000000000001},"max":{"1min":0.78100000000000003,"5min":9.6530000000000005,"15min":68.245000000000005},"last":0.32900000000000001}]},{"osd":7,"last update":"Tue Sep 17 07:26:38 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":1.375,"15min":1.4910000000000001},"min":{"1min":0.438,"5min":0.317,"15min":0.29399999999999998},"max":{"1min":0.81499999999999995,"5min":63.607999999999997,"15min":63.607999999999997},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.59199999999999997,"5min":1.3500000000000001,"15min":1.387},"min":{"1min":0.47099999999999997,"5min":0.33500000000000002,"15min":0.28000000000000003},"max":{"1min":0.76600000000000001,"5min":63.709000000000003,"15min":63.709000000000003},"last":0.52700000000000002}]}]},{"osd":2,"up_from":18,"seq":77309411661,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1122928,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":1121918,"kb_avail":92622224,"statfs":{"total":95995035648,"available":94845157376,"internally_reserved":0,"allocated":1024000,"data_stored":635749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1148844486},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:27:10 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":2.5979999999999999,"15min":2.2469999999999999},"min":{"1min":0.20000000000000001,"5min":0.16600000000000001,"15min":0.154},"max":{"1min":0.501,"5min":69.135999999999996,"15min":131.09800000000001},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.32800000000000001,"5min":2.6699999999999999,"15min":2.3090000000000002},"min":{"1min":0.20699999999999999,"5min":0.17999999999999999,"15min":0.17599999999999999},"max":{"1min":0.48499999999999999,"5min":69.033000000000001,"15min":131.24700000000001},"last":0.46400000000000002}]},{"osd":1,"last update":"Tue Sep 17 07:26:31 2024","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":2.2450000000000001,"15min":2.0600000000000001},"min":{"1min":0.192,"5min":0.184,"15min":0.184},"max":{"1min":0.44700000000000001,"5min":91.686999999999998,"15min":91.686999999999998},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.34300000000000003,"5min":2.5089999999999999,"15min":2.3780000000000001},"min":{"1min":0.255,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.51800000000000002,"5min":93.744,"15min":93.744},"last":0.50700000000000001}]},{"osd":3,"last update":"Tue Sep 17 07:27:17 2024","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.47999999999999998,"15min":0.51000000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.20300000000000001},"max":{"1min":0.46999999999999997,"5min":2.7469999999999999,"15min":4.6449999999999996},"last":0.44},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.73699999999999999,"15min":0.61899999999999999},"min":{"1min":0.27100000000000002,"5min":0.23100000000000001,"15min":0.22},"max":{"1min":0.49399999999999999,"5min":16.215,"15min":16.215},"last":0.52500000000000002}]},{"osd":4,"last update":"Tue Sep 17 07:26:45 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":1.038,"15min":1.272},"min":{"1min":0.316,"5min":0.28199999999999997,"15min":0.24399999999999999},"max":{"1min":0.75700000000000001,"5min":25.225999999999999,"15min":75.870999999999995},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.55200000000000005,"5min":1.054,"15min":1.383},"min":{"1min":0.36899999999999999,"5min":0.27700000000000002,"15min":0.26300000000000001},"max":{"1min":0.70299999999999996,"5min":25.268999999999998,"15min":75.888999999999996},"last":0.76900000000000002}]},{"osd":5,"last update":"Tue Sep 17 07:26:59 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.59499999999999997,"15min":0.61899999999999999},"min":{"1min":0.35699999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.69199999999999995,"5min":1.9610000000000001,"15min":1.9610000000000001},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.97099999999999997,"15min":0.96699999999999997},"min":{"1min":0.38400000000000001,"5min":0.312,"15min":0.312},"max":{"1min":0.77800000000000002,"5min":17.010000000000002,"15min":17.010000000000002},"last":0.752}]},{"osd":6,"last update":"Tue Sep 17 07:27:17 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":1.3240000000000001,"15min":1.708},"min":{"1min":0.41599999999999998,"5min":0.29699999999999999,"15min":0.27500000000000002},"max":{"1min":0.72499999999999998,"5min":24.936,"15min":76.081999999999994},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.55700000000000005,"5min":1.153,"15min":1.7470000000000001},"min":{"1min":0.38100000000000001,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.67300000000000004,"5min":24.884,"15min":76.244},"last":0.82999999999999996}]},{"osd":7,"last update":"Tue Sep 17 07:27:10 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.90800000000000003,"15min":1.3180000000000001},"min":{"1min":0.45000000000000001,"5min":0.314,"15min":0.314},"max":{"1min":0.76600000000000001,"5min":16.98,"15min":94.269999999999996},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.57899999999999996,"5min":1.0029999999999999,"15min":1.353},"min":{"1min":0.44500000000000001,"5min":0.32900000000000001,"15min":0.309},"max":{"1min":0.73099999999999998,"5min":17.030999999999999,"15min":94.634},"last":0.74399999999999999}]}]},{"osd":1,"up_from":413,"seq":1773821493326,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":534476,"kb_used_data":772,"kb_used_omap":4,"kb_used_meta":533691,"kb_avail":93210676,"statfs":{"total":95995035648,"available":95447732224,"internally_reserved":0,"allocated":790528,"data_stored":374233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4439,"internal_metadata":546500265},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:27:29 2024","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":2.0680000000000001,"15min":1.8029999999999999},"min":{"1min":0.21099999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":1.22,"5min":120.56399999999999,"15min":120.56399999999999},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.37,"5min":2.1699999999999999,"15min":1.6499999999999999},"min":{"1min":0.23000000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":1.022,"5min":120.746,"15min":120.746},"last":0.34399999999999997}]},{"osd":2,"last update":"Tue Sep 17 07:27:29 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.65900000000000003,"15min":1.127},"min":{"1min":0.27100000000000002,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":1.0760000000000001,"5min":6.7300000000000004,"15min":6.7300000000000004},"last":0.373},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.56999999999999995,"15min":0.94299999999999995},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":1.2969999999999999,"5min":3.919,"15min":3.919},"last":0.27800000000000002}]},{"osd":3,"last update":"Tue Sep 17 07:27:29 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.56699999999999995,"15min":0.73699999999999999},"min":{"1min":0.253,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":1.194,"5min":4.6749999999999998,"15min":5.9290000000000003},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.433,"5min":0.57199999999999995,"15min":0.77700000000000002},"min":{"1min":0.313,"5min":0.25,"15min":0.25},"max":{"1min":1.0489999999999999,"5min":3.3149999999999999,"15min":4.1340000000000003},"last":0.311}]},{"osd":4,"last update":"Tue Sep 17 07:27:29 2024","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.82899999999999996,"15min":5.181},"min":{"1min":0.37,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":6.774,"5min":6.774,"15min":116.88200000000001},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.71799999999999997,"15min":5.1900000000000004},"min":{"1min":0.41799999999999998,"5min":0.316,"15min":0.316},"max":{"1min":1.113,"5min":4.7089999999999996,"15min":116.98399999999999},"last":0.49299999999999999}]},{"osd":5,"last update":"Tue Sep 17 07:27:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":1.153,"15min":1.409},"min":{"1min":0.40100000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.139,"5min":9.7409999999999997,"15min":9.7409999999999997},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.57999999999999996,"5min":1.1539999999999999,"15min":1.4119999999999999},"min":{"1min":0.41799999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.996,"5min":11.538,"15min":11.538},"last":0.48199999999999998}]},{"osd":6,"last update":"Tue Sep 17 07:27:29 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":1.714,"15min":1.5249999999999999},"min":{"1min":0.40300000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.169,"5min":56.686999999999998,"15min":56.686999999999998},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.62,"5min":1.6819999999999999,"15min":1.446},"min":{"1min":0.40400000000000003,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.96599999999999997,"5min":56.420999999999999,"15min":56.420999999999999},"last":0.52800000000000002}]},{"osd":7,"last update":"Tue Sep 17 07:27:29 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.81499999999999995,"15min":0.98299999999999998},"min":{"1min":0.41299999999999998,"5min":0.34699999999999998,"15min":0.34399999999999997},"max":{"1min":0.92500000000000004,"5min":3.4399999999999999,"15min":4.2389999999999999},"last":0.502},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.70699999999999996,"15min":0.93700000000000006},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.2410000000000001,"5min":4.2839999999999998,"15min":4.2839999999999998},"last":0.54400000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738708,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1307776,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":1307006,"kb_avail":92437376,"statfs":{"total":95995035648,"available":94655873024,"internally_reserved":0,"allocated":778240,"data_stored":373481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1338374598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 17 07:27:28 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":1.2789999999999999,"15min":1.7649999999999999},"min":{"1min":0.224,"5min":0.17199999999999999,"15min":0.156},"max":{"1min":0.63400000000000001,"5min":53.731000000000002,"15min":53.731000000000002},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.38,"5min":1.367,"15min":1.4610000000000001},"min":{"1min":0.22800000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.61799999999999999,"5min":55.789000000000001,"15min":55.789000000000001},"last":0.45800000000000002}]},{"osd":2,"last update":"Tue Sep 17 07:27:01 2024","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":1.3140000000000001,"15min":1.115},"min":{"1min":0.24099999999999999,"5min":0.21299999999999999,"15min":0.155},"max":{"1min":0.59099999999999997,"5min":55.756,"15min":62.094999999999999},"last":0.432},{"interface":"front","average":{"1min":0.37,"5min":1.377,"15min":1.1419999999999999},"min":{"1min":0.251,"5min":0.20899999999999999,"15min":0.183},"max":{"1min":0.57599999999999996,"5min":57.854999999999997,"15min":59.848999999999997},"last":0.40600000000000003}]},{"osd":3,"last update":"Tue Sep 17 07:26:42 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.621,"15min":0.59799999999999998},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.59099999999999997,"5min":4.6829999999999998,"15min":5.5880000000000001},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.60699999999999998,"15min":0.55900000000000005},"min":{"1min":0.22700000000000001,"5min":0.20799999999999999,"15min":0.20399999999999999},"max":{"1min":0.46899999999999997,"5min":4.6109999999999998,"15min":5.1150000000000002},"last":0.41999999999999998}]},{"osd":4,"last update":"Tue Sep 17 07:26:42 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":1.018,"15min":1.02},"min":{"1min":0.36199999999999999,"5min":0.27400000000000002,"15min":0.23400000000000001},"max":{"1min":0.75,"5min":10.720000000000001,"15min":10.720000000000001},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.58799999999999997,"5min":1.0469999999999999,"15min":1.038},"min":{"1min":0.42499999999999999,"5min":0.26600000000000001,"15min":0.26200000000000001},"max":{"1min":0.73899999999999999,"5min":10.666,"15min":10.666},"last":0.68500000000000005}]},{"osd":5,"last update":"Tue Sep 17 07:27:01 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.872,"15min":0.99399999999999999},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.81100000000000005,"5min":3.79,"15min":3.79},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.84799999999999998,"15min":0.95599999999999996},"min":{"1min":0.40699999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.80000000000000004,"5min":3.2530000000000001,"15min":3.2530000000000001},"last":0.70699999999999996}]},{"osd":6,"last update":"Tue Sep 17 07:26:42 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":1.411,"15min":1.819},"min":{"1min":0.375,"5min":0.28999999999999998,"15min":0.28199999999999997},"max":{"1min":0.85299999999999998,"5min":37.262999999999998,"15min":147.13200000000001},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.57199999999999995,"5min":1.536,"15min":1.8169999999999999},"min":{"1min":0.37,"5min":0.311,"15min":0.30199999999999999},"max":{"1min":0.75,"5min":36.789999999999999,"15min":147.172},"last":0.67200000000000004}]},{"osd":7,"last update":"Tue Sep 17 07:26:42 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.77000000000000002,"15min":1.054},"min":{"1min":0.41999999999999998,"5min":0.318,"15min":0.30599999999999999},"max":{"1min":0.76700000000000002,"5min":3.843,"15min":6.3300000000000001},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.89900000000000002,"15min":1.121},"min":{"1min":0.371,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.82599999999999996,"5min":4.9100000000000001,"15min":7.2830000000000004},"last":0.66500000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T07:27:36.034 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T07:27:36.231 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:27:36.352 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:27:35 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:27:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:27:36.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:36 smithi100 ceph-mon[29092]: 4.18 scrub starts 2024-09-17T07:27:36.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:36 smithi100 ceph-mon[29092]: 4.18 scrub ok 2024-09-17T07:27:36.626 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:36 smithi100 ceph-mon[29092]: pgmap v1216: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T07:27:36.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:36 smithi100 ceph-mon[39462]: 4.18 scrub starts 2024-09-17T07:27:36.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:36 smithi100 ceph-mon[39462]: 4.18 scrub ok 2024-09-17T07:27:36.627 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:36 smithi100 ceph-mon[39462]: pgmap v1216: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T07:27:36.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:36 smithi135 ceph-mon[33238]: 4.18 scrub starts 2024-09-17T07:27:36.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:36 smithi135 ceph-mon[33238]: 4.18 scrub ok 2024-09-17T07:27:36.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:36 smithi135 ceph-mon[33238]: pgmap v1216: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T07:27:37.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:37 smithi135 ceph-mon[33238]: 6.13 scrub starts 2024-09-17T07:27:37.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:37 smithi135 ceph-mon[33238]: 6.13 scrub ok 2024-09-17T07:27:37.835 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:37 smithi100 ceph-mon[29092]: 6.13 scrub starts 2024-09-17T07:27:37.835 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:37 smithi100 ceph-mon[29092]: 6.13 scrub ok 2024-09-17T07:27:37.835 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:37 smithi100 ceph-mon[39462]: 6.13 scrub starts 2024-09-17T07:27:37.836 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:37 smithi100 ceph-mon[39462]: 6.13 scrub ok 2024-09-17T07:27:38.217 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-17T07:27:38.219 INFO:teuthology.orchestra.run.smithi100.stderr:dumped all 2024-09-17T07:27:38.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:38 smithi135 ceph-mon[33238]: 2.1 deep-scrub starts 2024-09-17T07:27:38.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:38 smithi135 ceph-mon[33238]: 2.1 deep-scrub ok 2024-09-17T07:27:38.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:38 smithi135 ceph-mon[33238]: pgmap v1217: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T07:27:38.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:38 smithi135 ceph-mon[33238]: from='client.26411 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:27:38.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:38 smithi100 ceph-mon[29092]: 2.1 deep-scrub starts 2024-09-17T07:27:38.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:38 smithi100 ceph-mon[29092]: 2.1 deep-scrub ok 2024-09-17T07:27:38.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:38 smithi100 ceph-mon[29092]: pgmap v1217: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T07:27:38.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:38 smithi100 ceph-mon[29092]: from='client.26411 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:27:38.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:38 smithi100 ceph-mon[39462]: 2.1 deep-scrub starts 2024-09-17T07:27:38.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:38 smithi100 ceph-mon[39462]: 2.1 deep-scrub ok 2024-09-17T07:27:38.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:38 smithi100 ceph-mon[39462]: pgmap v1217: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T07:27:38.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:38 smithi100 ceph-mon[39462]: from='client.26411 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:27:39.073 INFO:teuthology.orchestra.run.smithi100.stdout:{"pg_ready":true,"pg_map":{"version":1217,"stamp":"2024-09-17T07:27:37.568910+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596246,"num_objects":196,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":1930,"num_read_kb":1682,"num_write":1002,"num_write_kb":43,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":348,"num_bytes_recovered":597821,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2594,"ondisk_log_size":2594,"up":426,"acting":426,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":426,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":6613332,"kb_used_data":7760,"kb_used_omap":18,"kb_used_meta":6605485,"kb_avail":743347884,"statfs":{"total":767960285184,"available":761188233216,"internally_reserved":0,"allocated":7946240,"data_stored":4763062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18436,"internal_metadata":6764017660},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002574"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":51,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.759501+0000","last_change":"2024-09-17T07:26:47.679147+0000","last_active":"2024-09-17T07:27:20.759501+0000","last_peered":"2024-09-17T07:27:20.759501+0000","last_clean":"2024-09-17T07:27:20.759501+0000","last_became_active":"2024-09-17T07:25:49.447380+0000","last_became_peered":"2024-09-17T07:25:49.447380+0000","last_unstale":"2024-09-17T07:27:20.759501+0000","last_undegraded":"2024-09-17T07:27:20.759501+0000","last_fullsized":"2024-09-17T07:27:20.759501+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:47.679054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:07:33.644728+0000","last_clean_scrub_stamp":"2024-09-17T07:26:47.679054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:52.196709+0000","scrub_duration":0.001605719,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":628,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:22.475025+0000","last_change":"2024-09-17T07:27:00.632990+0000","last_active":"2024-09-17T07:27:22.475025+0000","last_peered":"2024-09-17T07:27:22.475025+0000","last_clean":"2024-09-17T07:27:22.475025+0000","last_became_active":"2024-09-17T07:22:13.589527+0000","last_became_peered":"2024-09-17T07:22:13.589527+0000","last_unstale":"2024-09-17T07:27:22.475025+0000","last_undegraded":"2024-09-17T07:27:22.475025+0000","last_fullsized":"2024-09-17T07:27:22.475025+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:27:00.632895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:27:00.632895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:07.134925+0000","scrub_duration":0.0027346559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"439'39","reported_seq":67,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.759323+0000","last_change":"2024-09-17T07:26:24.892510+0000","last_active":"2024-09-17T07:27:20.759323+0000","last_peered":"2024-09-17T07:27:20.759323+0000","last_clean":"2024-09-17T07:27:20.759323+0000","last_became_active":"2024-09-17T07:25:49.442905+0000","last_became_peered":"2024-09-17T07:25:49.442905+0000","last_unstale":"2024-09-17T07:27:20.759323+0000","last_undegraded":"2024-09-17T07:27:20.759323+0000","last_fullsized":"2024-09-17T07:27:20.759323+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'39","last_scrub_stamp":"2024-09-17T07:26:24.892381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:26:24.892381+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:53.552454+0000","scrub_duration":0.0030756160000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":587,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:19.462404+0000","last_change":"2024-09-17T07:26:22.887078+0000","last_active":"2024-09-17T07:27:19.462404+0000","last_peered":"2024-09-17T07:27:19.462404+0000","last_clean":"2024-09-17T07:27:19.462404+0000","last_became_active":"2024-09-17T07:21:15.354561+0000","last_became_peered":"2024-09-17T07:21:15.354561+0000","last_unstale":"2024-09-17T07:27:19.462404+0000","last_undegraded":"2024-09-17T07:27:19.462404+0000","last_fullsized":"2024-09-17T07:27:19.462404+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:22.886984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:26:22.886984+0000","last_clean_scrub_stamp":"2024-09-17T07:26:22.886984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:27.103979+0000","scrub_duration":0.0014775470000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":51,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.759663+0000","last_change":"2024-09-17T07:26:58.718920+0000","last_active":"2024-09-17T07:27:20.759663+0000","last_peered":"2024-09-17T07:27:20.759663+0000","last_clean":"2024-09-17T07:27:20.759663+0000","last_became_active":"2024-09-17T07:25:49.452275+0000","last_became_peered":"2024-09-17T07:25:49.452275+0000","last_unstale":"2024-09-17T07:27:20.759663+0000","last_undegraded":"2024-09-17T07:27:20.759663+0000","last_fullsized":"2024-09-17T07:27:20.759663+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:58.718845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:17:35.507742+0000","last_clean_scrub_stamp":"2024-09-17T07:26:58.718845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:04.607009+0000","scrub_duration":0.0014655849999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"438'21","reported_seq":635,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:19.462339+0000","last_change":"2024-09-17T07:26:17.884904+0000","last_active":"2024-09-17T07:27:19.462339+0000","last_peered":"2024-09-17T07:27:19.462339+0000","last_clean":"2024-09-17T07:27:19.462339+0000","last_became_active":"2024-09-17T07:21:15.354718+0000","last_became_peered":"2024-09-17T07:21:15.354718+0000","last_unstale":"2024-09-17T07:27:19.462339+0000","last_undegraded":"2024-09-17T07:27:19.462339+0000","last_fullsized":"2024-09-17T07:27:19.462339+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"438'21","last_scrub_stamp":"2024-09-17T07:26:17.884799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:26:17.884799+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001882175,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":480,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:19.461547+0000","last_change":"2024-09-17T07:26:55.184002+0000","last_active":"2024-09-17T07:27:19.461547+0000","last_peered":"2024-09-17T07:27:19.461547+0000","last_clean":"2024-09-17T07:27:19.461547+0000","last_became_active":"2024-09-17T07:23:57.473620+0000","last_became_peered":"2024-09-17T07:23:57.473620+0000","last_unstale":"2024-09-17T07:27:19.461547+0000","last_undegraded":"2024-09-17T07:27:19.461547+0000","last_fullsized":"2024-09-17T07:27:19.461547+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:55.183874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:26:55.183874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:55.457020+0000","scrub_duration":0.0020334110000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":612,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:19.462500+0000","last_change":"2024-09-17T07:27:18.747768+0000","last_active":"2024-09-17T07:27:19.462500+0000","last_peered":"2024-09-17T07:27:19.462500+0000","last_clean":"2024-09-17T07:27:19.462500+0000","last_became_active":"2024-09-17T07:23:57.482145+0000","last_became_peered":"2024-09-17T07:23:57.482145+0000","last_unstale":"2024-09-17T07:27:19.462500+0000","last_undegraded":"2024-09-17T07:27:19.462500+0000","last_fullsized":"2024-09-17T07:27:19.462500+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:27:18.747673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:27:18.747673+0000","last_clean_scrub_stamp":"2024-09-17T07:27:18.747673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:45.594419+0000","scrub_duration":0.001480063,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":156,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.387569+0000","last_change":"2024-09-17T07:27:13.727276+0000","last_active":"2024-09-17T07:27:20.387569+0000","last_peered":"2024-09-17T07:27:20.387569+0000","last_clean":"2024-09-17T07:27:20.387569+0000","last_became_active":"2024-09-17T07:22:13.615512+0000","last_became_peered":"2024-09-17T07:22:13.615512+0000","last_unstale":"2024-09-17T07:27:20.387569+0000","last_undegraded":"2024-09-17T07:27:20.387569+0000","last_fullsized":"2024-09-17T07:27:20.387569+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:27:13.727174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:46.312371+0000","last_clean_scrub_stamp":"2024-09-17T07:27:13.727174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:32.374707+0000","scrub_duration":0.0015053009999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"488'11","reported_seq":651,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:27.699074+0000","last_change":"2024-09-17T07:27:27.699193+0000","last_active":"2024-09-17T07:27:27.699074+0000","last_peered":"2024-09-17T07:27:27.699074+0000","last_clean":"2024-09-17T07:27:27.699074+0000","last_became_active":"2024-09-17T07:25:49.436886+0000","last_became_peered":"2024-09-17T07:25:49.436886+0000","last_unstale":"2024-09-17T07:27:27.699074+0000","last_undegraded":"2024-09-17T07:27:27.699074+0000","last_fullsized":"2024-09-17T07:27:27.699074+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"488'11","last_scrub_stamp":"2024-09-17T07:27:27.699053+0000","last_deep_scrub":"488'11","last_deep_scrub_stamp":"2024-09-17T07:27:27.699053+0000","last_clean_scrub_stamp":"2024-09-17T07:27:27.699053+0000","objects_scrubbed":2,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:52.451659+0000","scrub_duration":0.0022887570000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":13,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":145,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.387028+0000","last_change":"2024-09-17T07:27:06.698942+0000","last_active":"2024-09-17T07:27:20.387028+0000","last_peered":"2024-09-17T07:27:20.387028+0000","last_clean":"2024-09-17T07:27:20.387028+0000","last_became_active":"2024-09-17T07:22:13.624441+0000","last_became_peered":"2024-09-17T07:22:13.624441+0000","last_unstale":"2024-09-17T07:27:20.387028+0000","last_undegraded":"2024-09-17T07:27:20.387028+0000","last_fullsized":"2024-09-17T07:27:20.387028+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:27:06.698846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:27:06.698846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:21.096359+0000","scrub_duration":0.001506154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":572,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:30.856626+0000","last_change":"2024-09-17T07:27:30.856626+0000","last_active":"2024-09-17T07:27:30.856626+0000","last_peered":"2024-09-17T07:27:30.856626+0000","last_clean":"2024-09-17T07:27:30.856626+0000","last_became_active":"2024-09-17T07:25:49.438958+0000","last_became_peered":"2024-09-17T07:25:49.438958+0000","last_unstale":"2024-09-17T07:27:30.856626+0000","last_undegraded":"2024-09-17T07:27:30.856626+0000","last_fullsized":"2024-09-17T07:27:30.856626+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:27:30.856529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:24:24.811522+0000","last_clean_scrub_stamp":"2024-09-17T07:27:30.856529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:44.560274+0000","scrub_duration":0.0014845119999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":171,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.387458+0000","last_change":"2024-09-17T07:27:15.776974+0000","last_active":"2024-09-17T07:27:20.387458+0000","last_peered":"2024-09-17T07:27:20.387458+0000","last_clean":"2024-09-17T07:27:20.387458+0000","last_became_active":"2024-09-17T07:23:57.476413+0000","last_became_peered":"2024-09-17T07:23:57.476413+0000","last_unstale":"2024-09-17T07:27:20.387458+0000","last_undegraded":"2024-09-17T07:27:20.387458+0000","last_fullsized":"2024-09-17T07:27:20.387458+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:27:15.776862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:45.029247+0000","last_clean_scrub_stamp":"2024-09-17T07:27:15.776862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:34.753312+0000","scrub_duration":0.0014678950000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"439'25","reported_seq":682,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:19.462821+0000","last_change":"2024-09-17T07:27:02.766819+0000","last_active":"2024-09-17T07:27:19.462821+0000","last_peered":"2024-09-17T07:27:19.462821+0000","last_clean":"2024-09-17T07:27:19.462821+0000","last_became_active":"2024-09-17T07:25:49.438036+0000","last_became_peered":"2024-09-17T07:25:49.438036+0000","last_unstale":"2024-09-17T07:27:19.462821+0000","last_undegraded":"2024-09-17T07:27:19.462821+0000","last_fullsized":"2024-09-17T07:27:19.462821+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'25","last_scrub_stamp":"2024-09-17T07:27:02.766714+0000","last_deep_scrub":"439'25","last_deep_scrub_stamp":"2024-09-17T07:27:02.766714+0000","last_clean_scrub_stamp":"2024-09-17T07:27:02.766714+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:08.741846+0000","scrub_duration":0.0036329579999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":589,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:22.703020+0000","last_change":"2024-09-17T07:26:32.412814+0000","last_active":"2024-09-17T07:27:22.703020+0000","last_peered":"2024-09-17T07:27:22.703020+0000","last_clean":"2024-09-17T07:27:22.703020+0000","last_became_active":"2024-09-17T07:23:57.480812+0000","last_became_peered":"2024-09-17T07:23:57.480812+0000","last_unstale":"2024-09-17T07:27:22.703020+0000","last_undegraded":"2024-09-17T07:27:22.703020+0000","last_fullsized":"2024-09-17T07:27:22.703020+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:32.412696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:08:28.096266+0000","last_clean_scrub_stamp":"2024-09-17T07:26:32.412696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:55.638791+0000","scrub_duration":0.0017410769999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":93,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:22.474650+0000","last_change":"2024-09-17T07:26:52.630335+0000","last_active":"2024-09-17T07:27:22.474650+0000","last_peered":"2024-09-17T07:27:22.474650+0000","last_clean":"2024-09-17T07:27:22.474650+0000","last_became_active":"2024-09-17T07:24:20.409247+0000","last_became_peered":"2024-09-17T07:24:20.409247+0000","last_unstale":"2024-09-17T07:27:22.474650+0000","last_undegraded":"2024-09-17T07:27:22.474650+0000","last_fullsized":"2024-09-17T07:27:22.474650+0000","mapping_epoch":487,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":488,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:52.630258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:26:52.630258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:54.132579+0000","scrub_duration":0.0015908739999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.8","version":"212'130","reported_seq":663,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:22.702794+0000","last_change":"2024-09-17T07:27:01.692690+0000","last_active":"2024-09-17T07:27:22.702794+0000","last_peered":"2024-09-17T07:27:22.702794+0000","last_clean":"2024-09-17T07:27:22.702794+0000","last_became_active":"2024-09-17T07:25:52.441842+0000","last_became_peered":"2024-09-17T07:25:52.441842+0000","last_unstale":"2024-09-17T07:27:22.702794+0000","last_undegraded":"2024-09-17T07:27:22.702794+0000","last_fullsized":"2024-09-17T07:27:22.702794+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":4,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:27:01.692584+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:14:57.052505+0000","last_clean_scrub_stamp":"2024-09-17T07:27:01.692584+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:29.736535+0000","scrub_duration":0.0023312020000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"439'53","reported_seq":679,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:19.463145+0000","last_change":"2024-09-17T07:26:19.864456+0000","last_active":"2024-09-17T07:27:19.463145+0000","last_peered":"2024-09-17T07:27:19.463145+0000","last_clean":"2024-09-17T07:27:19.463145+0000","last_became_active":"2024-09-17T07:21:15.348413+0000","last_became_peered":"2024-09-17T07:21:15.348413+0000","last_unstale":"2024-09-17T07:27:19.463145+0000","last_undegraded":"2024-09-17T07:27:19.463145+0000","last_fullsized":"2024-09-17T07:27:19.463145+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'53","last_scrub_stamp":"2024-09-17T07:26:19.864349+0000","last_deep_scrub":"95'25","last_deep_scrub_stamp":"2024-09-17T07:04:09.450039+0000","last_clean_scrub_stamp":"2024-09-17T07:26:19.864349+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:47.815962+0000","scrub_duration":0.0045487909999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":61,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":160,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.387488+0000","last_change":"2024-09-17T07:26:37.758537+0000","last_active":"2024-09-17T07:27:20.387488+0000","last_peered":"2024-09-17T07:27:20.387488+0000","last_clean":"2024-09-17T07:27:20.387488+0000","last_became_active":"2024-09-17T07:23:57.475614+0000","last_became_peered":"2024-09-17T07:23:57.475614+0000","last_unstale":"2024-09-17T07:27:20.387488+0000","last_undegraded":"2024-09-17T07:27:20.387488+0000","last_fullsized":"2024-09-17T07:27:20.387488+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:37.758441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:26:37.758441+0000","last_clean_scrub_stamp":"2024-09-17T07:26:37.758441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:45.056906+0000","scrub_duration":0.00139802,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":419,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:19.462677+0000","last_change":"2024-09-17T07:26:14.280048+0000","last_active":"2024-09-17T07:27:19.462677+0000","last_peered":"2024-09-17T07:27:19.462677+0000","last_clean":"2024-09-17T07:27:19.462677+0000","last_became_active":"2024-09-17T07:21:15.349538+0000","last_became_peered":"2024-09-17T07:21:15.349538+0000","last_unstale":"2024-09-17T07:27:19.462677+0000","last_undegraded":"2024-09-17T07:27:19.462677+0000","last_fullsized":"2024-09-17T07:27:19.462677+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:14.279969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:26:14.279969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:36.505935+0000","scrub_duration":0.001479937,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":86,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.698898+0000","last_change":"2024-09-17T07:27:04.983031+0000","last_active":"2024-09-17T07:27:20.698898+0000","last_peered":"2024-09-17T07:27:20.698898+0000","last_clean":"2024-09-17T07:27:20.698898+0000","last_became_active":"2024-09-17T07:26:09.790191+0000","last_became_peered":"2024-09-17T07:26:09.790191+0000","last_unstale":"2024-09-17T07:27:20.698898+0000","last_undegraded":"2024-09-17T07:27:20.698898+0000","last_fullsized":"2024-09-17T07:27:20.698898+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:27:04.982928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:42.984088+0000","last_clean_scrub_stamp":"2024-09-17T07:27:04.982928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:29.445747+0000","scrub_duration":0.001869012,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"212'130","reported_seq":833,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:29.860997+0000","last_change":"2024-09-17T07:27:29.861090+0000","last_active":"2024-09-17T07:27:29.860997+0000","last_peered":"2024-09-17T07:27:29.860997+0000","last_clean":"2024-09-17T07:27:29.860997+0000","last_became_active":"2024-09-17T07:25:52.441833+0000","last_became_peered":"2024-09-17T07:25:52.441833+0000","last_unstale":"2024-09-17T07:27:29.860997+0000","last_undegraded":"2024-09-17T07:27:29.860997+0000","last_fullsized":"2024-09-17T07:27:29.860997+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":4,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:27:29.860982+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:14:57.052505+0000","last_clean_scrub_stamp":"2024-09-17T07:27:29.860982+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:34.518223+0000","scrub_duration":0.0016031909999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"439'35","reported_seq":705,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:30.777895+0000","last_change":"2024-09-17T07:27:30.777996+0000","last_active":"2024-09-17T07:27:30.777895+0000","last_peered":"2024-09-17T07:27:30.777895+0000","last_clean":"2024-09-17T07:27:30.777895+0000","last_became_active":"2024-09-17T07:25:49.440859+0000","last_became_peered":"2024-09-17T07:25:49.440859+0000","last_unstale":"2024-09-17T07:27:30.777895+0000","last_undegraded":"2024-09-17T07:27:30.777895+0000","last_fullsized":"2024-09-17T07:27:30.777895+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'35","last_scrub_stamp":"2024-09-17T07:27:30.777879+0000","last_deep_scrub":"439'35","last_deep_scrub_stamp":"2024-09-17T07:24:29.750888+0000","last_clean_scrub_stamp":"2024-09-17T07:27:30.777879+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:40.637971+0000","scrub_duration":0.0023317020000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":487,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:19.462835+0000","last_change":"2024-09-17T07:26:33.371802+0000","last_active":"2024-09-17T07:27:19.462835+0000","last_peered":"2024-09-17T07:27:19.462835+0000","last_clean":"2024-09-17T07:27:19.462835+0000","last_became_active":"2024-09-17T07:25:49.433544+0000","last_became_peered":"2024-09-17T07:25:49.433544+0000","last_unstale":"2024-09-17T07:27:19.462835+0000","last_undegraded":"2024-09-17T07:27:19.462835+0000","last_fullsized":"2024-09-17T07:27:19.462835+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:33.371718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:26:33.371718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:42.550790+0000","scrub_duration":0.001869297,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"525'14","reported_seq":137,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:25.805082+0000","last_change":"2024-09-17T07:27:09.886301+0000","last_active":"2024-09-17T07:27:25.805082+0000","last_peered":"2024-09-17T07:27:25.805082+0000","last_clean":"2024-09-17T07:27:25.805082+0000","last_became_active":"2024-09-17T07:26:09.601082+0000","last_became_peered":"2024-09-17T07:26:09.601082+0000","last_unstale":"2024-09-17T07:27:25.805082+0000","last_undegraded":"2024-09-17T07:27:25.805082+0000","last_fullsized":"2024-09-17T07:27:25.805082+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"525'14","last_scrub_stamp":"2024-09-17T07:27:09.886216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:27:09.886216+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:14.177273+0000","scrub_duration":0.0019761729999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":610,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:28.731701+0000","last_change":"2024-09-17T07:27:28.731828+0000","last_active":"2024-09-17T07:27:28.731701+0000","last_peered":"2024-09-17T07:27:28.731701+0000","last_clean":"2024-09-17T07:27:28.731701+0000","last_became_active":"2024-09-17T07:21:15.348799+0000","last_became_peered":"2024-09-17T07:21:15.348799+0000","last_unstale":"2024-09-17T07:27:28.731701+0000","last_undegraded":"2024-09-17T07:27:28.731701+0000","last_fullsized":"2024-09-17T07:27:28.731701+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:27:28.731655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:27:28.731655+0000","last_clean_scrub_stamp":"2024-09-17T07:27:28.731655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:53.482697+0000","scrub_duration":0.001669547,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"212'130","reported_seq":643,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:22.703172+0000","last_change":"2024-09-17T07:26:51.488736+0000","last_active":"2024-09-17T07:27:22.703172+0000","last_peered":"2024-09-17T07:27:22.703172+0000","last_clean":"2024-09-17T07:27:22.703172+0000","last_became_active":"2024-09-17T07:23:57.494014+0000","last_became_peered":"2024-09-17T07:23:57.494014+0000","last_unstale":"2024-09-17T07:27:22.703172+0000","last_undegraded":"2024-09-17T07:27:22.703172+0000","last_fullsized":"2024-09-17T07:27:22.703172+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":4,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:26:51.488609+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:14:57.052505+0000","last_clean_scrub_stamp":"2024-09-17T07:26:51.488609+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:04.490841+0000","scrub_duration":0.0016059659999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"439'39","reported_seq":576,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:19.463160+0000","last_change":"2024-09-17T07:26:44.253243+0000","last_active":"2024-09-17T07:27:19.463160+0000","last_peered":"2024-09-17T07:27:19.463160+0000","last_clean":"2024-09-17T07:27:19.463160+0000","last_became_active":"2024-09-17T07:25:49.433970+0000","last_became_peered":"2024-09-17T07:25:49.433970+0000","last_unstale":"2024-09-17T07:27:19.463160+0000","last_undegraded":"2024-09-17T07:27:19.463160+0000","last_fullsized":"2024-09-17T07:27:19.463160+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'39","last_scrub_stamp":"2024-09-17T07:26:44.253156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:26:44.253156+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:46.559449+0000","scrub_duration":0.0025313750000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":486,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:19.461974+0000","last_change":"2024-09-17T07:26:56.154343+0000","last_active":"2024-09-17T07:27:19.461974+0000","last_peered":"2024-09-17T07:27:19.461974+0000","last_clean":"2024-09-17T07:27:19.461974+0000","last_became_active":"2024-09-17T07:25:49.433837+0000","last_became_peered":"2024-09-17T07:25:49.433837+0000","last_unstale":"2024-09-17T07:27:19.461974+0000","last_undegraded":"2024-09-17T07:27:19.461974+0000","last_fullsized":"2024-09-17T07:27:19.461974+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:56.154239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:26:56.154239+0000","last_clean_scrub_stamp":"2024-09-17T07:26:56.154239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:13.090040+0000","scrub_duration":0.0017523560000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"478'10","reported_seq":923,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:25.805238+0000","last_change":"2024-09-17T07:26:30.851481+0000","last_active":"2024-09-17T07:27:25.805238+0000","last_peered":"2024-09-17T07:27:25.805238+0000","last_clean":"2024-09-17T07:27:25.805238+0000","last_became_active":"2024-09-17T07:23:57.473239+0000","last_became_peered":"2024-09-17T07:23:57.473239+0000","last_unstale":"2024-09-17T07:27:25.805238+0000","last_undegraded":"2024-09-17T07:27:25.805238+0000","last_fullsized":"2024-09-17T07:27:25.805238+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"478'10","last_scrub_stamp":"2024-09-17T07:26:30.851356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:26:30.851356+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:38.837398+0000","scrub_duration":0.0019091570000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":554,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:22.708842+0000","last_change":"2024-09-17T07:27:22.708842+0000","last_active":"2024-09-17T07:27:22.708842+0000","last_peered":"2024-09-17T07:27:22.708842+0000","last_clean":"2024-09-17T07:27:22.708842+0000","last_became_active":"2024-09-17T07:23:57.522025+0000","last_became_peered":"2024-09-17T07:23:57.522025+0000","last_unstale":"2024-09-17T07:27:22.708842+0000","last_undegraded":"2024-09-17T07:27:22.708842+0000","last_fullsized":"2024-09-17T07:27:22.708842+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:27:22.708760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:27:22.708760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:51.700304+0000","scrub_duration":0.0071301660000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"488'32","reported_seq":661,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:19.463149+0000","last_change":"2024-09-17T07:27:11.857770+0000","last_active":"2024-09-17T07:27:19.463149+0000","last_peered":"2024-09-17T07:27:19.463149+0000","last_clean":"2024-09-17T07:27:19.463149+0000","last_became_active":"2024-09-17T07:21:15.347662+0000","last_became_peered":"2024-09-17T07:21:15.347662+0000","last_unstale":"2024-09-17T07:27:19.463149+0000","last_undegraded":"2024-09-17T07:27:19.463149+0000","last_fullsized":"2024-09-17T07:27:19.463149+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"488'32","last_scrub_stamp":"2024-09-17T07:27:11.857602+0000","last_deep_scrub":"488'32","last_deep_scrub_stamp":"2024-09-17T07:24:27.785661+0000","last_clean_scrub_stamp":"2024-09-17T07:27:11.857602+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:30.330321+0000","scrub_duration":0.0022297979999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":145,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.386986+0000","last_change":"2024-09-17T07:26:46.806517+0000","last_active":"2024-09-17T07:27:20.386986+0000","last_peered":"2024-09-17T07:27:20.386986+0000","last_clean":"2024-09-17T07:27:20.386986+0000","last_became_active":"2024-09-17T07:22:13.709873+0000","last_became_peered":"2024-09-17T07:22:13.709873+0000","last_unstale":"2024-09-17T07:27:20.386986+0000","last_undegraded":"2024-09-17T07:27:20.386986+0000","last_fullsized":"2024-09-17T07:27:20.386986+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:46.806317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:26:46.806317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:04.395492+0000","scrub_duration":0.0019359889999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":565,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:22.703388+0000","last_change":"2024-09-17T07:26:41.450442+0000","last_active":"2024-09-17T07:27:22.703388+0000","last_peered":"2024-09-17T07:27:22.703388+0000","last_clean":"2024-09-17T07:27:22.703388+0000","last_became_active":"2024-09-17T07:23:57.478625+0000","last_became_peered":"2024-09-17T07:23:57.478625+0000","last_unstale":"2024-09-17T07:27:22.703388+0000","last_undegraded":"2024-09-17T07:27:22.703388+0000","last_fullsized":"2024-09-17T07:27:22.703388+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:41.450346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:26:41.450346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:41.999320+0000","scrub_duration":0.0013943549999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":51,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.759649+0000","last_change":"2024-09-17T07:27:06.751693+0000","last_active":"2024-09-17T07:27:20.759649+0000","last_peered":"2024-09-17T07:27:20.759649+0000","last_clean":"2024-09-17T07:27:20.759649+0000","last_became_active":"2024-09-17T07:25:49.453609+0000","last_became_peered":"2024-09-17T07:25:49.453609+0000","last_unstale":"2024-09-17T07:27:20.759649+0000","last_undegraded":"2024-09-17T07:27:20.759649+0000","last_fullsized":"2024-09-17T07:27:20.759649+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:27:06.751600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:13:22.187127+0000","last_clean_scrub_stamp":"2024-09-17T07:27:06.751600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:19.478751+0000","scrub_duration":0.001873825,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"212'130","reported_seq":685,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.758865+0000","last_change":"2024-09-17T07:26:53.768933+0000","last_active":"2024-09-17T07:27:20.758865+0000","last_peered":"2024-09-17T07:27:20.758865+0000","last_clean":"2024-09-17T07:27:20.758865+0000","last_became_active":"2024-09-17T07:25:52.440509+0000","last_became_peered":"2024-09-17T07:25:52.440509+0000","last_unstale":"2024-09-17T07:27:20.758865+0000","last_undegraded":"2024-09-17T07:27:20.758865+0000","last_fullsized":"2024-09-17T07:27:20.758865+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":3,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:26:53.768856+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:25:30.132032+0000","last_clean_scrub_stamp":"2024-09-17T07:26:53.768856+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:02.795912+0000","scrub_duration":0.002184283,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"439'42","reported_seq":675,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:19.462551+0000","last_change":"2024-09-17T07:26:46.733339+0000","last_active":"2024-09-17T07:27:19.462551+0000","last_peered":"2024-09-17T07:27:19.462551+0000","last_clean":"2024-09-17T07:27:19.462551+0000","last_became_active":"2024-09-17T07:23:57.481780+0000","last_became_peered":"2024-09-17T07:23:57.481780+0000","last_unstale":"2024-09-17T07:27:19.462551+0000","last_undegraded":"2024-09-17T07:27:19.462551+0000","last_fullsized":"2024-09-17T07:27:19.462551+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'42","last_scrub_stamp":"2024-09-17T07:26:46.733234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:26:46.733234+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:10.663281+0000","scrub_duration":0.0051770690000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":63,"num_read_kb":56,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":669,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.807906+0000","last_change":"2024-09-17T07:26:40.817536+0000","last_active":"2024-09-17T07:27:20.807906+0000","last_peered":"2024-09-17T07:27:20.807906+0000","last_clean":"2024-09-17T07:27:20.807906+0000","last_became_active":"2024-09-17T07:22:13.582096+0000","last_became_peered":"2024-09-17T07:22:13.582096+0000","last_unstale":"2024-09-17T07:27:20.807906+0000","last_undegraded":"2024-09-17T07:27:20.807906+0000","last_fullsized":"2024-09-17T07:27:20.807906+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:40.817369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:09:23.181028+0000","last_clean_scrub_stamp":"2024-09-17T07:26:40.817369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:02.477887+0000","scrub_duration":0.0016035719999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"516'12","reported_seq":77,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:25.805151+0000","last_change":"2024-09-17T07:26:40.696026+0000","last_active":"2024-09-17T07:27:25.805151+0000","last_peered":"2024-09-17T07:27:25.805151+0000","last_clean":"2024-09-17T07:27:25.805151+0000","last_became_active":"2024-09-17T07:25:49.445884+0000","last_became_peered":"2024-09-17T07:25:49.445884+0000","last_unstale":"2024-09-17T07:27:25.805151+0000","last_undegraded":"2024-09-17T07:27:25.805151+0000","last_fullsized":"2024-09-17T07:27:25.805151+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'12","last_scrub_stamp":"2024-09-17T07:26:40.695890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:26:40.695890+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:46.413180+0000","scrub_duration":0.0019653729999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":146,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.387383+0000","last_change":"2024-09-17T07:26:28.772744+0000","last_active":"2024-09-17T07:27:20.387383+0000","last_peered":"2024-09-17T07:27:20.387383+0000","last_clean":"2024-09-17T07:27:20.387383+0000","last_became_active":"2024-09-17T07:22:13.568656+0000","last_became_peered":"2024-09-17T07:22:13.568656+0000","last_unstale":"2024-09-17T07:27:20.387383+0000","last_undegraded":"2024-09-17T07:27:20.387383+0000","last_fullsized":"2024-09-17T07:27:20.387383+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:28.772597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:10:33.727472+0000","last_clean_scrub_stamp":"2024-09-17T07:26:28.772597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:42.854687+0000","scrub_duration":0.002637688,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"212'130","reported_seq":664,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:19.462699+0000","last_change":"2024-09-17T07:27:03.718296+0000","last_active":"2024-09-17T07:27:19.462699+0000","last_peered":"2024-09-17T07:27:19.462699+0000","last_clean":"2024-09-17T07:27:19.462699+0000","last_became_active":"2024-09-17T07:19:35.420337+0000","last_became_peered":"2024-09-17T07:19:35.420337+0000","last_unstale":"2024-09-17T07:27:19.462699+0000","last_undegraded":"2024-09-17T07:27:19.462699+0000","last_fullsized":"2024-09-17T07:27:19.462699+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":3,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:27:03.718182+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:27:03.718182+0000","last_clean_scrub_stamp":"2024-09-17T07:27:03.718182+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:17.755491+0000","scrub_duration":0.001587006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"439'42","reported_seq":69,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.759061+0000","last_change":"2024-09-17T07:26:59.680403+0000","last_active":"2024-09-17T07:27:20.759061+0000","last_peered":"2024-09-17T07:27:20.759061+0000","last_clean":"2024-09-17T07:27:20.759061+0000","last_became_active":"2024-09-17T07:25:49.446027+0000","last_became_peered":"2024-09-17T07:25:49.446027+0000","last_unstale":"2024-09-17T07:27:20.759061+0000","last_undegraded":"2024-09-17T07:27:20.759061+0000","last_fullsized":"2024-09-17T07:27:20.759061+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'42","last_scrub_stamp":"2024-09-17T07:26:59.680306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:26:59.680306+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:26.162364+0000","scrub_duration":0.0022565490000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":67,"num_read_kb":55,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":473,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:19.462963+0000","last_change":"2024-09-17T07:26:25.361119+0000","last_active":"2024-09-17T07:27:19.462963+0000","last_peered":"2024-09-17T07:27:19.462963+0000","last_clean":"2024-09-17T07:27:19.462963+0000","last_became_active":"2024-09-17T07:21:15.347867+0000","last_became_peered":"2024-09-17T07:21:15.347867+0000","last_unstale":"2024-09-17T07:27:19.462963+0000","last_undegraded":"2024-09-17T07:27:19.462963+0000","last_fullsized":"2024-09-17T07:27:19.462963+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:25.361017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:56.848689+0000","last_clean_scrub_stamp":"2024-09-17T07:26:25.361017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:44.285868+0000","scrub_duration":0.0039265189999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":642,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:21.810674+0000","last_change":"2024-09-17T07:27:21.810674+0000","last_active":"2024-09-17T07:27:21.810674+0000","last_peered":"2024-09-17T07:27:21.810674+0000","last_clean":"2024-09-17T07:27:21.810674+0000","last_became_active":"2024-09-17T07:25:49.437088+0000","last_became_peered":"2024-09-17T07:25:49.437088+0000","last_unstale":"2024-09-17T07:27:21.810674+0000","last_undegraded":"2024-09-17T07:27:21.810674+0000","last_fullsized":"2024-09-17T07:27:21.810674+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:27:21.810576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:27:21.810576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:49.511111+0000","scrub_duration":0.0015495280000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":611,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:22.474565+0000","last_change":"2024-09-17T07:26:56.655814+0000","last_active":"2024-09-17T07:27:22.474565+0000","last_peered":"2024-09-17T07:27:22.474565+0000","last_clean":"2024-09-17T07:27:22.474565+0000","last_became_active":"2024-09-17T07:25:49.437860+0000","last_became_peered":"2024-09-17T07:25:49.437860+0000","last_unstale":"2024-09-17T07:27:22.474565+0000","last_undegraded":"2024-09-17T07:27:22.474565+0000","last_fullsized":"2024-09-17T07:27:22.474565+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:56.655708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:26:56.655708+0000","last_clean_scrub_stamp":"2024-09-17T07:26:56.655708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:10.694032+0000","scrub_duration":0.0017231519999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"212'130","reported_seq":689,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.806335+0000","last_change":"2024-09-17T07:26:34.581433+0000","last_active":"2024-09-17T07:27:20.806335+0000","last_peered":"2024-09-17T07:27:20.806335+0000","last_clean":"2024-09-17T07:27:20.806335+0000","last_became_active":"2024-09-17T07:25:52.443672+0000","last_became_peered":"2024-09-17T07:25:52.443672+0000","last_unstale":"2024-09-17T07:27:20.806335+0000","last_undegraded":"2024-09-17T07:27:20.806335+0000","last_fullsized":"2024-09-17T07:27:20.806335+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":3,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:26:34.581294+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:14:57.052505+0000","last_clean_scrub_stamp":"2024-09-17T07:26:34.581294+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:00.904703+0000","scrub_duration":0.0032320320000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"439'39","reported_seq":744,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:22.476446+0000","last_change":"2024-09-17T07:26:25.758254+0000","last_active":"2024-09-17T07:27:22.476446+0000","last_peered":"2024-09-17T07:27:22.476446+0000","last_clean":"2024-09-17T07:27:22.476446+0000","last_became_active":"2024-09-17T07:23:57.497347+0000","last_became_peered":"2024-09-17T07:23:57.497347+0000","last_unstale":"2024-09-17T07:27:22.476446+0000","last_undegraded":"2024-09-17T07:27:22.476446+0000","last_fullsized":"2024-09-17T07:27:22.476446+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'39","last_scrub_stamp":"2024-09-17T07:26:25.758160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:26:25.758160+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:42.032625+0000","scrub_duration":0.002729085,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":156,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:31.791183+0000","last_change":"2024-09-17T07:27:31.791267+0000","last_active":"2024-09-17T07:27:31.791183+0000","last_peered":"2024-09-17T07:27:31.791183+0000","last_clean":"2024-09-17T07:27:31.791183+0000","last_became_active":"2024-09-17T07:22:13.624802+0000","last_became_peered":"2024-09-17T07:22:13.624802+0000","last_unstale":"2024-09-17T07:27:31.791183+0000","last_undegraded":"2024-09-17T07:27:31.791183+0000","last_fullsized":"2024-09-17T07:27:31.791183+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:27:31.791169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:16:47.631387+0000","last_clean_scrub_stamp":"2024-09-17T07:27:31.791169+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:39.038078+0000","scrub_duration":0.0014848870000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":641,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.806931+0000","last_change":"2024-09-17T07:27:16.181839+0000","last_active":"2024-09-17T07:27:20.806931+0000","last_peered":"2024-09-17T07:27:20.806931+0000","last_clean":"2024-09-17T07:27:20.806931+0000","last_became_active":"2024-09-17T07:23:57.485512+0000","last_became_peered":"2024-09-17T07:23:57.485512+0000","last_unstale":"2024-09-17T07:27:20.806931+0000","last_undegraded":"2024-09-17T07:27:20.806931+0000","last_fullsized":"2024-09-17T07:27:20.806931+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:27:16.181735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:11:17.583712+0000","last_clean_scrub_stamp":"2024-09-17T07:27:16.181735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:45.701118+0000","scrub_duration":0.001423725,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":429,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:22.709765+0000","last_change":"2024-09-17T07:26:09.234503+0000","last_active":"2024-09-17T07:27:22.709765+0000","last_peered":"2024-09-17T07:27:22.709765+0000","last_clean":"2024-09-17T07:27:22.709765+0000","last_became_active":"2024-09-17T07:21:15.351524+0000","last_became_peered":"2024-09-17T07:21:15.351524+0000","last_unstale":"2024-09-17T07:27:22.709765+0000","last_undegraded":"2024-09-17T07:27:22.709765+0000","last_fullsized":"2024-09-17T07:27:22.709765+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:09.234406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:26:09.234406+0000","last_clean_scrub_stamp":"2024-09-17T07:26:09.234406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:32.854562+0000","scrub_duration":0.0028525680000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"212'130","reported_seq":754,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.806309+0000","last_change":"2024-09-17T07:26:42.815660+0000","last_active":"2024-09-17T07:27:20.806309+0000","last_peered":"2024-09-17T07:27:20.806309+0000","last_clean":"2024-09-17T07:27:20.806309+0000","last_became_active":"2024-09-17T07:25:52.440902+0000","last_became_peered":"2024-09-17T07:25:52.440902+0000","last_unstale":"2024-09-17T07:27:20.806309+0000","last_undegraded":"2024-09-17T07:27:20.806309+0000","last_fullsized":"2024-09-17T07:27:20.806309+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":3,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:26:42.815530+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:14:57.052505+0000","last_clean_scrub_stamp":"2024-09-17T07:26:42.815530+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:44.596803+0000","scrub_duration":0.001784779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"439'35","reported_seq":521,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:30.297609+0000","last_change":"2024-09-17T07:27:30.297686+0000","last_active":"2024-09-17T07:27:30.297609+0000","last_peered":"2024-09-17T07:27:30.297609+0000","last_clean":"2024-09-17T07:27:30.297609+0000","last_became_active":"2024-09-17T07:23:57.478340+0000","last_became_peered":"2024-09-17T07:23:57.478340+0000","last_unstale":"2024-09-17T07:27:30.297609+0000","last_undegraded":"2024-09-17T07:27:30.297609+0000","last_fullsized":"2024-09-17T07:27:30.297609+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'35","last_scrub_stamp":"2024-09-17T07:27:30.297594+0000","last_deep_scrub":"110'15","last_deep_scrub_stamp":"2024-09-17T07:04:57.091400+0000","last_clean_scrub_stamp":"2024-09-17T07:27:30.297594+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:38.110106+0000","scrub_duration":0.0023124460000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":454,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:22.710294+0000","last_change":"2024-09-17T07:27:00.127529+0000","last_active":"2024-09-17T07:27:22.710294+0000","last_peered":"2024-09-17T07:27:22.710294+0000","last_clean":"2024-09-17T07:27:22.710294+0000","last_became_active":"2024-09-17T07:22:13.591372+0000","last_became_peered":"2024-09-17T07:22:13.591372+0000","last_unstale":"2024-09-17T07:27:22.710294+0000","last_undegraded":"2024-09-17T07:27:22.710294+0000","last_fullsized":"2024-09-17T07:27:22.710294+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:27:00.127430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:25:50.029556+0000","last_clean_scrub_stamp":"2024-09-17T07:27:00.127430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:00.388603+0000","scrub_duration":0.001748838,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"478'11","reported_seq":936,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:30.806559+0000","last_change":"2024-09-17T07:26:41.815671+0000","last_active":"2024-09-17T07:27:30.806559+0000","last_peered":"2024-09-17T07:27:30.806559+0000","last_clean":"2024-09-17T07:27:30.806559+0000","last_became_active":"2024-09-17T07:23:57.496882+0000","last_became_peered":"2024-09-17T07:23:57.496882+0000","last_unstale":"2024-09-17T07:27:30.806559+0000","last_undegraded":"2024-09-17T07:27:30.806559+0000","last_fullsized":"2024-09-17T07:27:30.806559+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"478'11","last_scrub_stamp":"2024-09-17T07:26:41.815575+0000","last_deep_scrub":"89'9","last_deep_scrub_stamp":"2024-09-17T07:15:53.512777+0000","last_clean_scrub_stamp":"2024-09-17T07:26:41.815575+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:43.389606+0000","scrub_duration":0.0023297109999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"268'2","reported_seq":636,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:22.474748+0000","last_change":"2024-09-17T07:26:36.868200+0000","last_active":"2024-09-17T07:27:22.474748+0000","last_peered":"2024-09-17T07:27:22.474748+0000","last_clean":"2024-09-17T07:27:22.474748+0000","last_became_active":"2024-09-17T07:25:49.431864+0000","last_became_peered":"2024-09-17T07:25:49.431864+0000","last_unstale":"2024-09-17T07:27:22.474748+0000","last_undegraded":"2024-09-17T07:27:22.474748+0000","last_fullsized":"2024-09-17T07:27:22.474748+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'2","last_scrub_stamp":"2024-09-17T07:26:36.868085+0000","last_deep_scrub":"268'2","last_deep_scrub_stamp":"2024-09-17T07:26:36.868085+0000","last_clean_scrub_stamp":"2024-09-17T07:26:36.868085+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:02.171364+0000","scrub_duration":0.0017805410000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"212'130","reported_seq":641,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:22.702963+0000","last_change":"2024-09-17T07:26:48.444029+0000","last_active":"2024-09-17T07:27:22.702963+0000","last_peered":"2024-09-17T07:27:22.702963+0000","last_clean":"2024-09-17T07:27:22.702963+0000","last_became_active":"2024-09-17T07:23:57.489445+0000","last_became_peered":"2024-09-17T07:23:57.489445+0000","last_unstale":"2024-09-17T07:27:22.702963+0000","last_undegraded":"2024-09-17T07:27:22.702963+0000","last_fullsized":"2024-09-17T07:27:22.702963+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":4,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:26:48.443917+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:14:57.052505+0000","last_clean_scrub_stamp":"2024-09-17T07:26:48.443917+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:50.972298+0000","scrub_duration":0.0021753620000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"516'56","reported_seq":823,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:28.679620+0000","last_change":"2024-09-17T07:26:17.303340+0000","last_active":"2024-09-17T07:27:28.679620+0000","last_peered":"2024-09-17T07:27:28.679620+0000","last_clean":"2024-09-17T07:27:28.679620+0000","last_became_active":"2024-09-17T07:25:49.439937+0000","last_became_peered":"2024-09-17T07:25:49.439937+0000","last_unstale":"2024-09-17T07:27:28.679620+0000","last_undegraded":"2024-09-17T07:27:28.679620+0000","last_fullsized":"2024-09-17T07:27:28.679620+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'56","last_scrub_stamp":"2024-09-17T07:26:17.303241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:26:17.303241+0000","objects_scrubbed":10,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:46.444913+0000","scrub_duration":0.0022879799999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":83,"num_read_kb":68,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":561,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:22.703149+0000","last_change":"2024-09-17T07:27:08.708047+0000","last_active":"2024-09-17T07:27:22.703149+0000","last_peered":"2024-09-17T07:27:22.703149+0000","last_clean":"2024-09-17T07:27:22.703149+0000","last_became_active":"2024-09-17T07:23:57.493640+0000","last_became_peered":"2024-09-17T07:23:57.493640+0000","last_unstale":"2024-09-17T07:27:22.703149+0000","last_undegraded":"2024-09-17T07:27:22.703149+0000","last_fullsized":"2024-09-17T07:27:22.703149+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:27:08.707949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:27:08.707949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:13.460377+0000","scrub_duration":0.0013508540000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":611,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:32.776616+0000","last_change":"2024-09-17T07:27:32.776616+0000","last_active":"2024-09-17T07:27:32.776616+0000","last_peered":"2024-09-17T07:27:32.776616+0000","last_clean":"2024-09-17T07:27:32.776616+0000","last_became_active":"2024-09-17T07:21:15.347069+0000","last_became_peered":"2024-09-17T07:21:15.347069+0000","last_unstale":"2024-09-17T07:27:32.776616+0000","last_undegraded":"2024-09-17T07:27:32.776616+0000","last_fullsized":"2024-09-17T07:27:32.776616+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:27:32.776527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:27:32.776527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:41.234550+0000","scrub_duration":0.0016639210000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":51,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.759838+0000","last_change":"2024-09-17T07:27:04.743645+0000","last_active":"2024-09-17T07:27:20.759838+0000","last_peered":"2024-09-17T07:27:20.759838+0000","last_clean":"2024-09-17T07:27:20.759838+0000","last_became_active":"2024-09-17T07:25:49.448463+0000","last_became_peered":"2024-09-17T07:25:49.448463+0000","last_unstale":"2024-09-17T07:27:20.759838+0000","last_undegraded":"2024-09-17T07:27:20.759838+0000","last_fullsized":"2024-09-17T07:27:20.759838+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:27:04.743558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:13:53.406661+0000","last_clean_scrub_stamp":"2024-09-17T07:27:04.743558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:07.613781+0000","scrub_duration":0.0017939220000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"212'130","reported_seq":658,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.387315+0000","last_change":"2024-09-17T07:26:58.669041+0000","last_active":"2024-09-17T07:27:20.387315+0000","last_peered":"2024-09-17T07:27:20.387315+0000","last_clean":"2024-09-17T07:27:20.387315+0000","last_became_active":"2024-09-17T07:22:16.360258+0000","last_became_peered":"2024-09-17T07:22:16.360258+0000","last_unstale":"2024-09-17T07:27:20.387315+0000","last_undegraded":"2024-09-17T07:27:20.387315+0000","last_fullsized":"2024-09-17T07:27:20.387315+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":4,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:26:58.668913+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:14:57.052505+0000","last_clean_scrub_stamp":"2024-09-17T07:26:58.668913+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:20.318226+0000","scrub_duration":0.0018872610000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"520'26","reported_seq":501,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:19.463117+0000","last_change":"2024-09-17T07:27:14.287240+0000","last_active":"2024-09-17T07:27:19.463117+0000","last_peered":"2024-09-17T07:27:19.463117+0000","last_clean":"2024-09-17T07:27:19.463117+0000","last_became_active":"2024-09-17T07:23:57.492747+0000","last_became_peered":"2024-09-17T07:23:57.492747+0000","last_unstale":"2024-09-17T07:27:19.463117+0000","last_undegraded":"2024-09-17T07:27:19.463117+0000","last_fullsized":"2024-09-17T07:27:19.463117+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"520'26","last_scrub_stamp":"2024-09-17T07:27:14.287153+0000","last_deep_scrub":"520'26","last_deep_scrub_stamp":"2024-09-17T07:25:57.063699+0000","last_clean_scrub_stamp":"2024-09-17T07:27:14.287153+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:31.626817+0000","scrub_duration":0.002065697,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":30,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"76'5","reported_seq":1641,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:32.109067+0000","last_change":"2024-09-17T07:26:23.320854+0000","last_active":"2024-09-17T07:27:32.109067+0000","last_peered":"2024-09-17T07:27:32.109067+0000","last_clean":"2024-09-17T07:27:32.109067+0000","last_became_active":"2024-09-17T07:21:15.357178+0000","last_became_peered":"2024-09-17T07:21:15.357178+0000","last_unstale":"2024-09-17T07:27:32.109067+0000","last_undegraded":"2024-09-17T07:27:32.109067+0000","last_fullsized":"2024-09-17T07:27:32.109067+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'5","last_scrub_stamp":"2024-09-17T07:26:23.320769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:26:23.320769+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:45.475946+0000","scrub_duration":0.0017841700000000001,"objects_trimmed":0,"snaptrim_duration":0.00033739799999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":384,"num_read_kb":379,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":320,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.698260+0000","last_change":"2024-09-17T07:26:26.894141+0000","last_active":"2024-09-17T07:27:20.698260+0000","last_peered":"2024-09-17T07:27:20.698260+0000","last_clean":"2024-09-17T07:27:20.698260+0000","last_became_active":"2024-09-17T07:26:09.601090+0000","last_became_peered":"2024-09-17T07:26:09.601090+0000","last_unstale":"2024-09-17T07:27:20.698260+0000","last_undegraded":"2024-09-17T07:27:20.698260+0000","last_fullsized":"2024-09-17T07:27:20.698260+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:26.894033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:26:26.894033+0000","last_clean_scrub_stamp":"2024-09-17T07:26:26.894033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:36.125732+0000","scrub_duration":0.0014059630000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":673,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.698764+0000","last_change":"2024-09-17T07:26:57.987382+0000","last_active":"2024-09-17T07:27:20.698764+0000","last_peered":"2024-09-17T07:27:20.698764+0000","last_clean":"2024-09-17T07:27:20.698764+0000","last_became_active":"2024-09-17T07:23:57.481485+0000","last_became_peered":"2024-09-17T07:23:57.481485+0000","last_unstale":"2024-09-17T07:27:20.698764+0000","last_undegraded":"2024-09-17T07:27:20.698764+0000","last_fullsized":"2024-09-17T07:27:20.698764+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:57.987295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:18:42.051667+0000","last_clean_scrub_stamp":"2024-09-17T07:26:57.987295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:03.480643+0000","scrub_duration":0.0019378469999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":165,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.387517+0000","last_change":"2024-09-17T07:27:18.783434+0000","last_active":"2024-09-17T07:27:20.387517+0000","last_peered":"2024-09-17T07:27:20.387517+0000","last_clean":"2024-09-17T07:27:20.387517+0000","last_became_active":"2024-09-17T07:25:49.437671+0000","last_became_peered":"2024-09-17T07:25:49.437671+0000","last_unstale":"2024-09-17T07:27:20.387517+0000","last_undegraded":"2024-09-17T07:27:20.387517+0000","last_fullsized":"2024-09-17T07:27:20.387517+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:27:18.783326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:02.062618+0000","last_clean_scrub_stamp":"2024-09-17T07:27:18.783326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:44.363758+0000","scrub_duration":0.001436744,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"212'130","reported_seq":683,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:22.710086+0000","last_change":"2024-09-17T07:27:01.112058+0000","last_active":"2024-09-17T07:27:22.710086+0000","last_peered":"2024-09-17T07:27:22.710086+0000","last_clean":"2024-09-17T07:27:22.710086+0000","last_became_active":"2024-09-17T07:21:18.357874+0000","last_became_peered":"2024-09-17T07:21:18.357874+0000","last_unstale":"2024-09-17T07:27:22.710086+0000","last_undegraded":"2024-09-17T07:27:22.710086+0000","last_fullsized":"2024-09-17T07:27:22.710086+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":3,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:27:01.111943+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:20:34.113602+0000","last_clean_scrub_stamp":"2024-09-17T07:27:01.111943+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:26.213695+0000","scrub_duration":0.0021401279999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"439'42","reported_seq":694,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:22.475179+0000","last_change":"2024-09-17T07:26:31.782487+0000","last_active":"2024-09-17T07:27:22.475179+0000","last_peered":"2024-09-17T07:27:22.475179+0000","last_clean":"2024-09-17T07:27:22.475179+0000","last_became_active":"2024-09-17T07:23:57.480190+0000","last_became_peered":"2024-09-17T07:23:57.480190+0000","last_unstale":"2024-09-17T07:27:22.475179+0000","last_undegraded":"2024-09-17T07:27:22.475179+0000","last_fullsized":"2024-09-17T07:27:22.475179+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'42","last_scrub_stamp":"2024-09-17T07:26:31.782388+0000","last_deep_scrub":"268'30","last_deep_scrub_stamp":"2024-09-17T07:14:00.599656+0000","last_clean_scrub_stamp":"2024-09-17T07:26:31.782388+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:48.742554+0000","scrub_duration":0.003378462,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"70'1","reported_seq":646,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:22.474064+0000","last_change":"2024-09-17T07:26:45.763688+0000","last_active":"2024-09-17T07:27:22.474064+0000","last_peered":"2024-09-17T07:27:22.474064+0000","last_clean":"2024-09-17T07:27:22.474064+0000","last_became_active":"2024-09-17T07:25:49.432721+0000","last_became_peered":"2024-09-17T07:25:49.432721+0000","last_unstale":"2024-09-17T07:27:22.474064+0000","last_undegraded":"2024-09-17T07:27:22.474064+0000","last_fullsized":"2024-09-17T07:27:22.474064+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-17T07:26:45.763600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:26:45.763600+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:06.596582+0000","scrub_duration":0.0016379529999999999,"objects_trimmed":0,"snaptrim_duration":0.0032298629999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":145,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:22.826211+0000","last_change":"2024-09-17T07:27:22.826295+0000","last_active":"2024-09-17T07:27:22.826211+0000","last_peered":"2024-09-17T07:27:22.826211+0000","last_clean":"2024-09-17T07:27:22.826211+0000","last_became_active":"2024-09-17T07:22:13.710211+0000","last_became_peered":"2024-09-17T07:22:13.710211+0000","last_unstale":"2024-09-17T07:27:22.826211+0000","last_undegraded":"2024-09-17T07:27:22.826211+0000","last_fullsized":"2024-09-17T07:27:22.826211+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:27:22.826173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:55.321202+0000","last_clean_scrub_stamp":"2024-09-17T07:27:22.826173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:44.035163+0000","scrub_duration":0.0019797629999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":545,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:22.703596+0000","last_change":"2024-09-17T07:26:29.422533+0000","last_active":"2024-09-17T07:27:22.703596+0000","last_peered":"2024-09-17T07:27:22.703596+0000","last_clean":"2024-09-17T07:27:22.703596+0000","last_became_active":"2024-09-17T07:23:57.480455+0000","last_became_peered":"2024-09-17T07:23:57.480455+0000","last_unstale":"2024-09-17T07:27:22.703596+0000","last_undegraded":"2024-09-17T07:27:22.703596+0000","last_fullsized":"2024-09-17T07:27:22.703596+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:29.422419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:26:29.422419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:46.481871+0000","scrub_duration":0.0013803999999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":490,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:22.710374+0000","last_change":"2024-09-17T07:26:42.219856+0000","last_active":"2024-09-17T07:27:22.710374+0000","last_peered":"2024-09-17T07:27:22.710374+0000","last_clean":"2024-09-17T07:27:22.710374+0000","last_became_active":"2024-09-17T07:25:49.439416+0000","last_became_peered":"2024-09-17T07:25:49.439416+0000","last_unstale":"2024-09-17T07:27:22.710374+0000","last_undegraded":"2024-09-17T07:27:22.710374+0000","last_fullsized":"2024-09-17T07:27:22.710374+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:42.219774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:26:42.219774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:00.294851+0000","scrub_duration":0.0017563069999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"439'38","reported_seq":716,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.807429+0000","last_change":"2024-09-17T07:26:44.891822+0000","last_active":"2024-09-17T07:27:20.807429+0000","last_peered":"2024-09-17T07:27:20.807429+0000","last_clean":"2024-09-17T07:27:20.807429+0000","last_became_active":"2024-09-17T07:22:13.583204+0000","last_became_peered":"2024-09-17T07:22:13.583204+0000","last_unstale":"2024-09-17T07:27:20.807429+0000","last_undegraded":"2024-09-17T07:27:20.807429+0000","last_fullsized":"2024-09-17T07:27:20.807429+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'38","last_scrub_stamp":"2024-09-17T07:26:44.891686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:26:44.891686+0000","objects_scrubbed":7,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:03.485066+0000","scrub_duration":0.0025976969999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"268'3","reported_seq":55,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.759202+0000","last_change":"2024-09-17T07:26:27.885827+0000","last_active":"2024-09-17T07:27:20.759202+0000","last_peered":"2024-09-17T07:27:20.759202+0000","last_clean":"2024-09-17T07:27:20.759202+0000","last_became_active":"2024-09-17T07:25:49.453053+0000","last_became_peered":"2024-09-17T07:25:49.453053+0000","last_unstale":"2024-09-17T07:27:20.759202+0000","last_undegraded":"2024-09-17T07:27:20.759202+0000","last_fullsized":"2024-09-17T07:27:20.759202+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'3","last_scrub_stamp":"2024-09-17T07:26:27.885747+0000","last_deep_scrub":"268'3","last_deep_scrub_stamp":"2024-09-17T07:13:54.949345+0000","last_clean_scrub_stamp":"2024-09-17T07:26:27.885747+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:55.211433+0000","scrub_duration":0.0041955849999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":606,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:22.703111+0000","last_change":"2024-09-17T07:26:35.378501+0000","last_active":"2024-09-17T07:27:22.703111+0000","last_peered":"2024-09-17T07:27:22.703111+0000","last_clean":"2024-09-17T07:27:22.703111+0000","last_became_active":"2024-09-17T07:23:57.477410+0000","last_became_peered":"2024-09-17T07:23:57.477410+0000","last_unstale":"2024-09-17T07:27:22.703111+0000","last_undegraded":"2024-09-17T07:27:22.703111+0000","last_fullsized":"2024-09-17T07:27:22.703111+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:35.378391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:26:35.378391+0000","last_clean_scrub_stamp":"2024-09-17T07:26:35.378391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:45.200399+0000","scrub_duration":0.001544789,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"212'130","reported_seq":668,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:28.816584+0000","last_change":"2024-09-17T07:27:28.816677+0000","last_active":"2024-09-17T07:27:28.816584+0000","last_peered":"2024-09-17T07:27:28.816584+0000","last_clean":"2024-09-17T07:27:28.816584+0000","last_became_active":"2024-09-17T07:22:16.366295+0000","last_became_peered":"2024-09-17T07:22:16.366295+0000","last_unstale":"2024-09-17T07:27:28.816584+0000","last_undegraded":"2024-09-17T07:27:28.816584+0000","last_fullsized":"2024-09-17T07:27:28.816584+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"212'130","last_scrub_stamp":"2024-09-17T07:27:28.816567+0000","last_deep_scrub":"212'130","last_deep_scrub_stamp":"2024-09-17T07:14:57.052505+0000","last_clean_scrub_stamp":"2024-09-17T07:27:28.816567+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:53.560309+0000","scrub_duration":0.00161006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":171,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:27.784152+0000","last_change":"2024-09-17T07:27:27.784231+0000","last_active":"2024-09-17T07:27:27.784152+0000","last_peered":"2024-09-17T07:27:27.784152+0000","last_clean":"2024-09-17T07:27:27.784152+0000","last_became_active":"2024-09-17T07:23:57.476027+0000","last_became_peered":"2024-09-17T07:23:57.476027+0000","last_unstale":"2024-09-17T07:27:27.784152+0000","last_undegraded":"2024-09-17T07:27:27.784152+0000","last_fullsized":"2024-09-17T07:27:27.784152+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:27:27.784136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:18:48.891136+0000","last_clean_scrub_stamp":"2024-09-17T07:27:27.784136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:31.332571+0000","scrub_duration":0.001505681,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":561,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:22.703541+0000","last_change":"2024-09-17T07:26:56.655819+0000","last_active":"2024-09-17T07:27:22.703541+0000","last_peered":"2024-09-17T07:27:22.703541+0000","last_clean":"2024-09-17T07:27:22.703541+0000","last_became_active":"2024-09-17T07:25:49.441234+0000","last_became_peered":"2024-09-17T07:25:49.441234+0000","last_unstale":"2024-09-17T07:27:22.703541+0000","last_undegraded":"2024-09-17T07:27:22.703541+0000","last_fullsized":"2024-09-17T07:27:22.703541+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:56.655711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:25:46.981696+0000","last_clean_scrub_stamp":"2024-09-17T07:26:56.655711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:58.207683+0000","scrub_duration":0.0013852210000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"439'51","reported_seq":760,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.807157+0000","last_change":"2024-09-17T07:26:35.626589+0000","last_active":"2024-09-17T07:27:20.807157+0000","last_peered":"2024-09-17T07:27:20.807157+0000","last_clean":"2024-09-17T07:27:20.807157+0000","last_became_active":"2024-09-17T07:22:13.588260+0000","last_became_peered":"2024-09-17T07:22:13.588260+0000","last_unstale":"2024-09-17T07:27:20.807157+0000","last_undegraded":"2024-09-17T07:27:20.807157+0000","last_fullsized":"2024-09-17T07:27:20.807157+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'51","last_scrub_stamp":"2024-09-17T07:26:35.626492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:26:35.626492+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:52.839178+0000","scrub_duration":0.0026693060000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":684,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.698466+0000","last_change":"2024-09-17T07:26:51.066605+0000","last_active":"2024-09-17T07:27:20.698466+0000","last_peered":"2024-09-17T07:27:20.698466+0000","last_clean":"2024-09-17T07:27:20.698466+0000","last_became_active":"2024-09-17T07:23:57.482084+0000","last_became_peered":"2024-09-17T07:23:57.482084+0000","last_unstale":"2024-09-17T07:27:20.698466+0000","last_undegraded":"2024-09-17T07:27:20.698466+0000","last_fullsized":"2024-09-17T07:27:20.698466+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:51.066517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:24:23.255270+0000","last_clean_scrub_stamp":"2024-09-17T07:26:51.066517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:00.509842+0000","scrub_duration":0.001686491,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":667,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.698474+0000","last_change":"2024-09-17T07:26:34.953585+0000","last_active":"2024-09-17T07:27:20.698474+0000","last_peered":"2024-09-17T07:27:20.698474+0000","last_clean":"2024-09-17T07:27:20.698474+0000","last_became_active":"2024-09-17T07:23:57.481504+0000","last_became_peered":"2024-09-17T07:23:57.481504+0000","last_unstale":"2024-09-17T07:27:20.698474+0000","last_undegraded":"2024-09-17T07:27:20.698474+0000","last_fullsized":"2024-09-17T07:27:20.698474+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:34.953500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:24:20.282973+0000","last_clean_scrub_stamp":"2024-09-17T07:26:34.953500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:58.510821+0000","scrub_duration":0.0037914979999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":154,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.387206+0000","last_change":"2024-09-17T07:26:01.843368+0000","last_active":"2024-09-17T07:27:20.387206+0000","last_peered":"2024-09-17T07:27:20.387206+0000","last_clean":"2024-09-17T07:27:20.387206+0000","last_became_active":"2024-09-17T07:25:49.437863+0000","last_became_peered":"2024-09-17T07:25:49.437863+0000","last_unstale":"2024-09-17T07:27:20.387206+0000","last_undegraded":"2024-09-17T07:27:20.387206+0000","last_fullsized":"2024-09-17T07:27:20.387206+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:01.843148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:15:45.275473+0000","last_clean_scrub_stamp":"2024-09-17T07:26:01.843148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:31.022195+0000","scrub_duration":0.0021668270000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"439'25","reported_seq":629,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:22.476439+0000","last_change":"2024-09-17T07:26:23.741158+0000","last_active":"2024-09-17T07:27:22.476439+0000","last_peered":"2024-09-17T07:27:22.476439+0000","last_clean":"2024-09-17T07:27:22.476439+0000","last_became_active":"2024-09-17T07:21:15.349361+0000","last_became_peered":"2024-09-17T07:21:15.349361+0000","last_unstale":"2024-09-17T07:27:22.476439+0000","last_undegraded":"2024-09-17T07:27:22.476439+0000","last_fullsized":"2024-09-17T07:27:22.476439+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'25","last_scrub_stamp":"2024-09-17T07:26:23.741060+0000","last_deep_scrub":"268'19","last_deep_scrub_stamp":"2024-09-17T07:14:06.634334+0000","last_clean_scrub_stamp":"2024-09-17T07:26:23.741060+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:42.119714+0000","scrub_duration":0.0025792580000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":491,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:19.463005+0000","last_change":"2024-09-17T07:26:49.227283+0000","last_active":"2024-09-17T07:27:19.463005+0000","last_peered":"2024-09-17T07:27:19.463005+0000","last_clean":"2024-09-17T07:27:19.463005+0000","last_became_active":"2024-09-17T07:25:49.425468+0000","last_became_peered":"2024-09-17T07:25:49.425468+0000","last_unstale":"2024-09-17T07:27:19.463005+0000","last_undegraded":"2024-09-17T07:27:19.463005+0000","last_fullsized":"2024-09-17T07:27:19.463005+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:49.227208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:13:53.593078+0000","last_clean_scrub_stamp":"2024-09-17T07:26:49.227208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:01.466332+0000","scrub_duration":0.001636202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":570,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.698512+0000","last_change":"2024-09-17T07:26:29.940005+0000","last_active":"2024-09-17T07:27:20.698512+0000","last_peered":"2024-09-17T07:27:20.698512+0000","last_clean":"2024-09-17T07:27:20.698512+0000","last_became_active":"2024-09-17T07:26:09.789844+0000","last_became_peered":"2024-09-17T07:26:09.789844+0000","last_unstale":"2024-09-17T07:27:20.698512+0000","last_undegraded":"2024-09-17T07:27:20.698512+0000","last_fullsized":"2024-09-17T07:27:20.698512+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:29.939920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:26:29.939920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:42.034439+0000","scrub_duration":0.001960617,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":610,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:22.474160+0000","last_change":"2024-09-17T07:26:49.705368+0000","last_active":"2024-09-17T07:27:22.474160+0000","last_peered":"2024-09-17T07:27:22.474160+0000","last_clean":"2024-09-17T07:27:22.474160+0000","last_became_active":"2024-09-17T07:22:13.589460+0000","last_became_peered":"2024-09-17T07:22:13.589460+0000","last_unstale":"2024-09-17T07:27:22.474160+0000","last_undegraded":"2024-09-17T07:27:22.474160+0000","last_fullsized":"2024-09-17T07:27:22.474160+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:49.705283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:03:58.099849+0000","last_clean_scrub_stamp":"2024-09-17T07:26:49.705283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:08.797340+0000","scrub_duration":0.0016370289999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"439'46","reported_seq":698,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:22.702858+0000","last_change":"2024-09-17T07:26:20.278724+0000","last_active":"2024-09-17T07:27:22.702858+0000","last_peered":"2024-09-17T07:27:22.702858+0000","last_clean":"2024-09-17T07:27:22.702858+0000","last_became_active":"2024-09-17T07:23:57.482858+0000","last_became_peered":"2024-09-17T07:23:57.482858+0000","last_unstale":"2024-09-17T07:27:22.702858+0000","last_undegraded":"2024-09-17T07:27:22.702858+0000","last_fullsized":"2024-09-17T07:27:22.702858+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'46","last_scrub_stamp":"2024-09-17T07:26:20.278609+0000","last_deep_scrub":"242'34","last_deep_scrub_stamp":"2024-09-17T07:14:50.103539+0000","last_clean_scrub_stamp":"2024-09-17T07:26:20.278609+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:46.455910+0000","scrub_duration":0.0024723520000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'2","reported_seq":97,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.698852+0000","last_change":"2024-09-17T07:27:13.971677+0000","last_active":"2024-09-17T07:27:20.698852+0000","last_peered":"2024-09-17T07:27:20.698852+0000","last_clean":"2024-09-17T07:27:20.698852+0000","last_became_active":"2024-09-17T07:26:09.601463+0000","last_became_peered":"2024-09-17T07:26:09.601463+0000","last_unstale":"2024-09-17T07:27:20.698852+0000","last_undegraded":"2024-09-17T07:27:20.698852+0000","last_fullsized":"2024-09-17T07:27:20.698852+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-09-17T07:27:13.971590+0000","last_deep_scrub":"89'2","last_deep_scrub_stamp":"2024-09-17T07:12:20.301263+0000","last_clean_scrub_stamp":"2024-09-17T07:27:13.971590+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:36.934399+0000","scrub_duration":0.0018383679999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":420,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"436'14","reported_seq":225,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:25.805241+0000","last_change":"2024-09-17T07:27:24.785675+0000","last_active":"2024-09-17T07:27:25.805241+0000","last_peered":"2024-09-17T07:27:25.805241+0000","last_clean":"2024-09-17T07:27:25.805241+0000","last_became_active":"2024-09-17T07:22:13.624050+0000","last_became_peered":"2024-09-17T07:22:13.624050+0000","last_unstale":"2024-09-17T07:27:25.805241+0000","last_undegraded":"2024-09-17T07:27:25.805241+0000","last_fullsized":"2024-09-17T07:27:25.805241+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"436'14","last_scrub_stamp":"2024-09-17T07:27:24.785571+0000","last_deep_scrub":"225'9","last_deep_scrub_stamp":"2024-09-17T07:11:25.292565+0000","last_clean_scrub_stamp":"2024-09-17T07:27:24.785571+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:32.388864+0000","scrub_duration":0.0018136680000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":573,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:22.703074+0000","last_change":"2024-09-17T07:26:19.239231+0000","last_active":"2024-09-17T07:27:22.703074+0000","last_peered":"2024-09-17T07:27:22.703074+0000","last_clean":"2024-09-17T07:27:22.703074+0000","last_became_active":"2024-09-17T07:23:57.493273+0000","last_became_peered":"2024-09-17T07:23:57.493273+0000","last_unstale":"2024-09-17T07:27:22.703074+0000","last_undegraded":"2024-09-17T07:27:22.703074+0000","last_fullsized":"2024-09-17T07:27:22.703074+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:19.239152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:18:41.385928+0000","last_clean_scrub_stamp":"2024-09-17T07:26:19.239152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:47.521562+0000","scrub_duration":0.001371171,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"439'29","reported_seq":716,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:19.462446+0000","last_change":"2024-09-17T07:27:05.698111+0000","last_active":"2024-09-17T07:27:19.462446+0000","last_peered":"2024-09-17T07:27:19.462446+0000","last_clean":"2024-09-17T07:27:19.462446+0000","last_became_active":"2024-09-17T07:25:49.437151+0000","last_became_peered":"2024-09-17T07:25:49.437151+0000","last_unstale":"2024-09-17T07:27:19.462446+0000","last_undegraded":"2024-09-17T07:27:19.462446+0000","last_fullsized":"2024-09-17T07:27:19.462446+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'29","last_scrub_stamp":"2024-09-17T07:27:05.697965+0000","last_deep_scrub":"439'29","last_deep_scrub_stamp":"2024-09-17T07:27:05.697965+0000","last_clean_scrub_stamp":"2024-09-17T07:27:05.697965+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:27.244334+0000","scrub_duration":0.0027608450000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"512'4","reported_seq":648,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:19.463468+0000","last_change":"2024-09-17T07:26:55.642280+0000","last_active":"2024-09-17T07:27:19.463468+0000","last_peered":"2024-09-17T07:27:19.463468+0000","last_clean":"2024-09-17T07:27:19.463468+0000","last_became_active":"2024-09-17T07:22:13.590172+0000","last_became_peered":"2024-09-17T07:22:13.590172+0000","last_unstale":"2024-09-17T07:27:19.463468+0000","last_undegraded":"2024-09-17T07:27:19.463468+0000","last_fullsized":"2024-09-17T07:27:19.463468+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"512'4","last_scrub_stamp":"2024-09-17T07:26:55.642174+0000","last_deep_scrub":"512'4","last_deep_scrub_stamp":"2024-09-17T07:25:26.876662+0000","last_clean_scrub_stamp":"2024-09-17T07:26:55.642174+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:07.000531+0000","scrub_duration":0.0022207529999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":439,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.698347+0000","last_change":"2024-09-17T07:27:08.887830+0000","last_active":"2024-09-17T07:27:20.698347+0000","last_peered":"2024-09-17T07:27:20.698347+0000","last_clean":"2024-09-17T07:27:20.698347+0000","last_became_active":"2024-09-17T07:26:09.598369+0000","last_became_peered":"2024-09-17T07:26:09.598369+0000","last_unstale":"2024-09-17T07:27:20.698347+0000","last_undegraded":"2024-09-17T07:27:20.698347+0000","last_fullsized":"2024-09-17T07:27:20.698347+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:27:08.887748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:11:17.479036+0000","last_clean_scrub_stamp":"2024-09-17T07:27:08.887748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:13.097888+0000","scrub_duration":0.0012505999999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":51,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.759250+0000","last_change":"2024-09-17T07:27:09.723542+0000","last_active":"2024-09-17T07:27:20.759250+0000","last_peered":"2024-09-17T07:27:20.759250+0000","last_clean":"2024-09-17T07:27:20.759250+0000","last_became_active":"2024-09-17T07:25:49.445899+0000","last_became_peered":"2024-09-17T07:25:49.445899+0000","last_unstale":"2024-09-17T07:27:20.759250+0000","last_undegraded":"2024-09-17T07:27:20.759250+0000","last_fullsized":"2024-09-17T07:27:20.759250+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:27:09.723460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:27:09.723460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:27.664331+0000","scrub_duration":0.001394946,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"439'21","reported_seq":707,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:22.475139+0000","last_change":"2024-09-17T07:26:56.713742+0000","last_active":"2024-09-17T07:27:22.475139+0000","last_peered":"2024-09-17T07:27:22.475139+0000","last_clean":"2024-09-17T07:27:22.475139+0000","last_became_active":"2024-09-17T07:25:49.432314+0000","last_became_peered":"2024-09-17T07:25:49.432314+0000","last_unstale":"2024-09-17T07:27:22.475139+0000","last_undegraded":"2024-09-17T07:27:22.475139+0000","last_fullsized":"2024-09-17T07:27:22.475139+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'21","last_scrub_stamp":"2024-09-17T07:26:56.713622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:26:56.713622+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:02.228736+0000","scrub_duration":0.002458394,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":654,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.698397+0000","last_change":"2024-09-17T07:26:57.007387+0000","last_active":"2024-09-17T07:27:20.698397+0000","last_peered":"2024-09-17T07:27:20.698397+0000","last_clean":"2024-09-17T07:27:20.698397+0000","last_became_active":"2024-09-17T07:21:15.355464+0000","last_became_peered":"2024-09-17T07:21:15.355464+0000","last_unstale":"2024-09-17T07:27:20.698397+0000","last_undegraded":"2024-09-17T07:27:20.698397+0000","last_fullsized":"2024-09-17T07:27:20.698397+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:57.007294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:26:57.007294+0000","last_clean_scrub_stamp":"2024-09-17T07:26:57.007294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:07.480606+0000","scrub_duration":0.0014950390000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":62,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.759898+0000","last_change":"2024-09-17T07:27:11.726251+0000","last_active":"2024-09-17T07:27:20.759898+0000","last_peered":"2024-09-17T07:27:20.759898+0000","last_clean":"2024-09-17T07:27:20.759898+0000","last_became_active":"2024-09-17T07:25:49.445346+0000","last_became_peered":"2024-09-17T07:25:49.445346+0000","last_unstale":"2024-09-17T07:27:20.759898+0000","last_undegraded":"2024-09-17T07:27:20.759898+0000","last_fullsized":"2024-09-17T07:27:20.759898+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:27:11.726171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:18.367486+0000","last_clean_scrub_stamp":"2024-09-17T07:27:11.726171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:20.249715+0000","scrub_duration":0.001556415,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":707,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.698772+0000","last_change":"2024-09-17T07:26:54.023145+0000","last_active":"2024-09-17T07:27:20.698772+0000","last_peered":"2024-09-17T07:27:20.698772+0000","last_clean":"2024-09-17T07:27:20.698772+0000","last_became_active":"2024-09-17T07:22:13.590599+0000","last_became_peered":"2024-09-17T07:22:13.590599+0000","last_unstale":"2024-09-17T07:27:20.698772+0000","last_undegraded":"2024-09-17T07:27:20.698772+0000","last_fullsized":"2024-09-17T07:27:20.698772+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:54.023053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:24:24.282487+0000","last_clean_scrub_stamp":"2024-09-17T07:26:54.023053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:54.216936+0000","scrub_duration":0.0015857029999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":623,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.806888+0000","last_change":"2024-09-17T07:27:00.084261+0000","last_active":"2024-09-17T07:27:20.806888+0000","last_peered":"2024-09-17T07:27:20.806888+0000","last_clean":"2024-09-17T07:27:20.806888+0000","last_became_active":"2024-09-17T07:21:15.348758+0000","last_became_peered":"2024-09-17T07:21:15.348758+0000","last_unstale":"2024-09-17T07:27:20.806888+0000","last_undegraded":"2024-09-17T07:27:20.806888+0000","last_fullsized":"2024-09-17T07:27:20.806888+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:27:00.084159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:25:36.444488+0000","last_clean_scrub_stamp":"2024-09-17T07:27:00.084159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:10.748189+0000","scrub_duration":0.0014711730000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"439'18","reported_seq":59,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.759889+0000","last_change":"2024-09-17T07:26:35.786261+0000","last_active":"2024-09-17T07:27:20.759889+0000","last_peered":"2024-09-17T07:27:20.759889+0000","last_clean":"2024-09-17T07:27:20.759889+0000","last_became_active":"2024-09-17T07:25:49.444760+0000","last_became_peered":"2024-09-17T07:25:49.444760+0000","last_unstale":"2024-09-17T07:27:20.759889+0000","last_undegraded":"2024-09-17T07:27:20.759889+0000","last_fullsized":"2024-09-17T07:27:20.759889+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'18","last_scrub_stamp":"2024-09-17T07:26:35.786155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:26:35.786155+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:03.360633+0000","scrub_duration":0.0019837840000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":24,"num_read_kb":21,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":573,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:22.703284+0000","last_change":"2024-09-17T07:27:02.729678+0000","last_active":"2024-09-17T07:27:22.703284+0000","last_peered":"2024-09-17T07:27:22.703284+0000","last_clean":"2024-09-17T07:27:22.703284+0000","last_became_active":"2024-09-17T07:25:49.439959+0000","last_became_peered":"2024-09-17T07:25:49.439959+0000","last_unstale":"2024-09-17T07:27:22.703284+0000","last_undegraded":"2024-09-17T07:27:22.703284+0000","last_fullsized":"2024-09-17T07:27:22.703284+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:27:02.729587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:27:02.729587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:15.174554+0000","scrub_duration":0.0016207190000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":145,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.387372+0000","last_change":"2024-09-17T07:26:30.773949+0000","last_active":"2024-09-17T07:27:20.387372+0000","last_peered":"2024-09-17T07:27:20.387372+0000","last_clean":"2024-09-17T07:27:20.387372+0000","last_became_active":"2024-09-17T07:22:13.623754+0000","last_became_peered":"2024-09-17T07:22:13.623754+0000","last_unstale":"2024-09-17T07:27:20.387372+0000","last_undegraded":"2024-09-17T07:27:20.387372+0000","last_fullsized":"2024-09-17T07:27:20.387372+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:30.773841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:26:30.773841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:42.562972+0000","scrub_duration":0.0034468150000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"439'30","reported_seq":63,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.759698+0000","last_change":"2024-09-17T07:27:01.744678+0000","last_active":"2024-09-17T07:27:20.759698+0000","last_peered":"2024-09-17T07:27:20.759698+0000","last_clean":"2024-09-17T07:27:20.759698+0000","last_became_active":"2024-09-17T07:25:49.448072+0000","last_became_peered":"2024-09-17T07:25:49.448072+0000","last_unstale":"2024-09-17T07:27:20.759698+0000","last_undegraded":"2024-09-17T07:27:20.759698+0000","last_fullsized":"2024-09-17T07:27:20.759698+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'30","last_scrub_stamp":"2024-09-17T07:27:01.744598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:27:01.744598+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:02.418179+0000","scrub_duration":0.001931,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":623,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:22.703009+0000","last_change":"2024-09-17T07:26:30.448278+0000","last_active":"2024-09-17T07:27:22.703009+0000","last_peered":"2024-09-17T07:27:22.703009+0000","last_clean":"2024-09-17T07:27:22.703009+0000","last_became_active":"2024-09-17T07:25:49.437128+0000","last_became_peered":"2024-09-17T07:25:49.437128+0000","last_unstale":"2024-09-17T07:27:22.703009+0000","last_undegraded":"2024-09-17T07:27:22.703009+0000","last_fullsized":"2024-09-17T07:27:22.703009+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:30.448176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:19:40.443474+0000","last_clean_scrub_stamp":"2024-09-17T07:26:30.448176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:58.792122+0000","scrub_duration":0.0023650339999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"478'13","reported_seq":838,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:30.806939+0000","last_change":"2024-09-17T07:27:24.731554+0000","last_active":"2024-09-17T07:27:30.806939+0000","last_peered":"2024-09-17T07:27:30.806939+0000","last_clean":"2024-09-17T07:27:30.806939+0000","last_became_active":"2024-09-17T07:23:57.522386+0000","last_became_peered":"2024-09-17T07:23:57.522386+0000","last_unstale":"2024-09-17T07:27:30.806939+0000","last_undegraded":"2024-09-17T07:27:30.806939+0000","last_fullsized":"2024-09-17T07:27:30.806939+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"478'13","last_scrub_stamp":"2024-09-17T07:27:24.731384+0000","last_deep_scrub":"163'9","last_deep_scrub_stamp":"2024-09-17T07:07:34.933808+0000","last_clean_scrub_stamp":"2024-09-17T07:27:24.731384+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:51.829395+0000","scrub_duration":0.0025423450000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":639,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:22.474944+0000","last_change":"2024-09-17T07:27:17.701080+0000","last_active":"2024-09-17T07:27:22.474944+0000","last_peered":"2024-09-17T07:27:22.474944+0000","last_clean":"2024-09-17T07:27:22.474944+0000","last_became_active":"2024-09-17T07:25:49.432373+0000","last_became_peered":"2024-09-17T07:25:49.432373+0000","last_unstale":"2024-09-17T07:27:22.474944+0000","last_undegraded":"2024-09-17T07:27:22.474944+0000","last_fullsized":"2024-09-17T07:27:22.474944+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:27:17.700994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:12:26.619893+0000","last_clean_scrub_stamp":"2024-09-17T07:27:17.700994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:32.072214+0000","scrub_duration":0.0016546600000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"439'21","reported_seq":689,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:22.703227+0000","last_change":"2024-09-17T07:26:36.390354+0000","last_active":"2024-09-17T07:27:22.703227+0000","last_peered":"2024-09-17T07:27:22.703227+0000","last_clean":"2024-09-17T07:27:22.703227+0000","last_became_active":"2024-09-17T07:25:49.440071+0000","last_became_peered":"2024-09-17T07:25:49.440071+0000","last_unstale":"2024-09-17T07:27:22.703227+0000","last_undegraded":"2024-09-17T07:27:22.703227+0000","last_fullsized":"2024-09-17T07:27:22.703227+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'21","last_scrub_stamp":"2024-09-17T07:26:36.390220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:26:36.390220+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:01.277657+0000","scrub_duration":0.002320398,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":637,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:22.476521+0000","last_change":"2024-09-17T07:26:43.779088+0000","last_active":"2024-09-17T07:27:22.476521+0000","last_peered":"2024-09-17T07:27:22.476521+0000","last_clean":"2024-09-17T07:27:22.476521+0000","last_became_active":"2024-09-17T07:25:49.431778+0000","last_became_peered":"2024-09-17T07:25:49.431778+0000","last_unstale":"2024-09-17T07:27:22.476521+0000","last_undegraded":"2024-09-17T07:27:22.476521+0000","last_fullsized":"2024-09-17T07:27:22.476521+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:43.779011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:26:43.779011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:44.415632+0000","scrub_duration":0.00331285,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"516'16","reported_seq":77,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:25.805145+0000","last_change":"2024-09-17T07:26:54.748648+0000","last_active":"2024-09-17T07:27:25.805145+0000","last_peered":"2024-09-17T07:27:25.805145+0000","last_clean":"2024-09-17T07:27:25.805145+0000","last_became_active":"2024-09-17T07:25:49.446551+0000","last_became_peered":"2024-09-17T07:25:49.446551+0000","last_unstale":"2024-09-17T07:27:25.805145+0000","last_undegraded":"2024-09-17T07:27:25.805145+0000","last_fullsized":"2024-09-17T07:27:25.805145+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'16","last_scrub_stamp":"2024-09-17T07:26:54.748561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:26:54.748561+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:15.409095+0000","scrub_duration":0.001788066,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":617,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:19.463335+0000","last_change":"2024-09-17T07:26:53.608456+0000","last_active":"2024-09-17T07:27:19.463335+0000","last_peered":"2024-09-17T07:27:19.463335+0000","last_clean":"2024-09-17T07:27:19.463335+0000","last_became_active":"2024-09-17T07:23:57.472854+0000","last_became_peered":"2024-09-17T07:23:57.472854+0000","last_unstale":"2024-09-17T07:27:19.463335+0000","last_undegraded":"2024-09-17T07:27:19.463335+0000","last_fullsized":"2024-09-17T07:27:19.463335+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:53.608340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:20:37.110458+0000","last_clean_scrub_stamp":"2024-09-17T07:26:53.608340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:22.216453+0000","scrub_duration":0.0014634769999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"543'25","reported_seq":479,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:19.462900+0000","last_change":"2024-09-17T07:27:08.217225+0000","last_active":"2024-09-17T07:27:19.462900+0000","last_peered":"2024-09-17T07:27:19.462900+0000","last_clean":"2024-09-17T07:27:19.462900+0000","last_became_active":"2024-09-17T07:21:15.348060+0000","last_became_peered":"2024-09-17T07:21:15.348060+0000","last_unstale":"2024-09-17T07:27:19.462900+0000","last_undegraded":"2024-09-17T07:27:19.462900+0000","last_fullsized":"2024-09-17T07:27:19.462900+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"543'25","last_scrub_stamp":"2024-09-17T07:27:08.217115+0000","last_deep_scrub":"543'25","last_deep_scrub_stamp":"2024-09-17T07:27:08.217115+0000","last_clean_scrub_stamp":"2024-09-17T07:27:08.217115+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:36.327801+0000","scrub_duration":0.0038364929999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":679,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.807200+0000","last_change":"2024-09-17T07:26:41.781589+0000","last_active":"2024-09-17T07:27:20.807200+0000","last_peered":"2024-09-17T07:27:20.807200+0000","last_clean":"2024-09-17T07:27:20.807200+0000","last_became_active":"2024-09-17T07:25:49.439173+0000","last_became_peered":"2024-09-17T07:25:49.439173+0000","last_unstale":"2024-09-17T07:27:20.807200+0000","last_undegraded":"2024-09-17T07:27:20.807200+0000","last_fullsized":"2024-09-17T07:27:20.807200+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:41.781503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:10:36.334827+0000","last_clean_scrub_stamp":"2024-09-17T07:26:41.781503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:55.665857+0000","scrub_duration":0.0015077859999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":62,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:24.754702+0000","last_change":"2024-09-17T07:27:24.754791+0000","last_active":"2024-09-17T07:27:24.754702+0000","last_peered":"2024-09-17T07:27:24.754702+0000","last_clean":"2024-09-17T07:27:24.754702+0000","last_became_active":"2024-09-17T07:25:49.438615+0000","last_became_peered":"2024-09-17T07:25:49.438615+0000","last_unstale":"2024-09-17T07:27:24.754702+0000","last_undegraded":"2024-09-17T07:27:24.754702+0000","last_fullsized":"2024-09-17T07:27:24.754702+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:27:24.754683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:27:24.754683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:50.719869+0000","scrub_duration":0.001328119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":621,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:22.474779+0000","last_change":"2024-09-17T07:27:05.654258+0000","last_active":"2024-09-17T07:27:22.474779+0000","last_peered":"2024-09-17T07:27:22.474779+0000","last_clean":"2024-09-17T07:27:22.474779+0000","last_became_active":"2024-09-17T07:23:57.478818+0000","last_became_peered":"2024-09-17T07:23:57.478818+0000","last_unstale":"2024-09-17T07:27:22.474779+0000","last_undegraded":"2024-09-17T07:27:22.474779+0000","last_fullsized":"2024-09-17T07:27:22.474779+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:27:05.654178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:24:21.575574+0000","last_clean_scrub_stamp":"2024-09-17T07:27:05.654178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:09.946587+0000","scrub_duration":0.001340826,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"439'27","reported_seq":642,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:24.733154+0000","last_change":"2024-09-17T07:27:24.733247+0000","last_active":"2024-09-17T07:27:24.733154+0000","last_peered":"2024-09-17T07:27:24.733154+0000","last_clean":"2024-09-17T07:27:24.733154+0000","last_became_active":"2024-09-17T07:21:15.347179+0000","last_became_peered":"2024-09-17T07:21:15.347179+0000","last_unstale":"2024-09-17T07:27:24.733154+0000","last_undegraded":"2024-09-17T07:27:24.733154+0000","last_fullsized":"2024-09-17T07:27:24.733154+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'27","last_scrub_stamp":"2024-09-17T07:27:24.733138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:27:24.733138+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:52.167284+0000","scrub_duration":0.0023121819999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"94'2","reported_seq":480,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:19.463085+0000","last_change":"2024-09-17T07:26:51.157175+0000","last_active":"2024-09-17T07:27:19.463085+0000","last_peered":"2024-09-17T07:27:19.463085+0000","last_clean":"2024-09-17T07:27:19.463085+0000","last_became_active":"2024-09-17T07:23:57.521742+0000","last_became_peered":"2024-09-17T07:23:57.521742+0000","last_unstale":"2024-09-17T07:27:19.463085+0000","last_undegraded":"2024-09-17T07:27:19.463085+0000","last_fullsized":"2024-09-17T07:27:19.463085+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-09-17T07:26:51.157081+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-09-17T07:07:31.717888+0000","last_clean_scrub_stamp":"2024-09-17T07:26:51.157081+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:52.690749+0000","scrub_duration":0.0018815710000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":493,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:19.462697+0000","last_change":"2024-09-17T07:27:03.111480+0000","last_active":"2024-09-17T07:27:19.462697+0000","last_peered":"2024-09-17T07:27:19.462697+0000","last_clean":"2024-09-17T07:27:19.462697+0000","last_became_active":"2024-09-17T07:25:49.432316+0000","last_became_peered":"2024-09-17T07:25:49.432316+0000","last_unstale":"2024-09-17T07:27:19.462697+0000","last_undegraded":"2024-09-17T07:27:19.462697+0000","last_fullsized":"2024-09-17T07:27:19.462697+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:27:03.111364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:25:46.957486+0000","last_clean_scrub_stamp":"2024-09-17T07:27:03.111364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:13.283055+0000","scrub_duration":0.001914652,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":62,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:26.779636+0000","last_change":"2024-09-17T07:27:26.779711+0000","last_active":"2024-09-17T07:27:26.779636+0000","last_peered":"2024-09-17T07:27:26.779636+0000","last_clean":"2024-09-17T07:27:26.779636+0000","last_became_active":"2024-09-17T07:25:49.452253+0000","last_became_peered":"2024-09-17T07:25:49.452253+0000","last_unstale":"2024-09-17T07:27:26.779636+0000","last_undegraded":"2024-09-17T07:27:26.779636+0000","last_fullsized":"2024-09-17T07:27:26.779636+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:27:26.779620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:27:26.779620+0000","last_clean_scrub_stamp":"2024-09-17T07:27:26.779620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:55.697162+0000","scrub_duration":0.0016900750000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"439'12","reported_seq":57,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.758944+0000","last_change":"2024-09-17T07:26:44.701243+0000","last_active":"2024-09-17T07:27:20.758944+0000","last_peered":"2024-09-17T07:27:20.758944+0000","last_clean":"2024-09-17T07:27:20.758944+0000","last_became_active":"2024-09-17T07:25:49.443742+0000","last_became_peered":"2024-09-17T07:25:49.443742+0000","last_unstale":"2024-09-17T07:27:20.758944+0000","last_undegraded":"2024-09-17T07:27:20.758944+0000","last_fullsized":"2024-09-17T07:27:20.758944+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'12","last_scrub_stamp":"2024-09-17T07:26:44.701161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:26:44.701161+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:45.007153+0000","scrub_duration":0.0017795459999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":566,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:27.786944+0000","last_change":"2024-09-17T07:27:27.786944+0000","last_active":"2024-09-17T07:27:27.786944+0000","last_peered":"2024-09-17T07:27:27.786944+0000","last_clean":"2024-09-17T07:27:27.786944+0000","last_became_active":"2024-09-17T07:23:57.476234+0000","last_became_peered":"2024-09-17T07:23:57.476234+0000","last_unstale":"2024-09-17T07:27:27.786944+0000","last_undegraded":"2024-09-17T07:27:27.786944+0000","last_fullsized":"2024-09-17T07:27:27.786944+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:27:27.786850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:27:27.786850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:41.961780+0000","scrub_duration":0.001364259,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":633,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.806685+0000","last_change":"2024-09-17T07:26:21.240109+0000","last_active":"2024-09-17T07:27:20.806685+0000","last_peered":"2024-09-17T07:27:20.806685+0000","last_clean":"2024-09-17T07:27:20.806685+0000","last_became_active":"2024-09-17T07:22:13.582229+0000","last_became_peered":"2024-09-17T07:22:13.582229+0000","last_unstale":"2024-09-17T07:27:20.806685+0000","last_undegraded":"2024-09-17T07:27:20.806685+0000","last_fullsized":"2024-09-17T07:27:20.806685+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:21.240027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:26:21.240027+0000","last_clean_scrub_stamp":"2024-09-17T07:26:21.240027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:36.673199+0000","scrub_duration":0.0012918459999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":145,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.387108+0000","last_change":"2024-09-17T07:27:00.694126+0000","last_active":"2024-09-17T07:27:20.387108+0000","last_peered":"2024-09-17T07:27:20.387108+0000","last_clean":"2024-09-17T07:27:20.387108+0000","last_became_active":"2024-09-17T07:22:13.624127+0000","last_became_peered":"2024-09-17T07:22:13.624127+0000","last_unstale":"2024-09-17T07:27:20.387108+0000","last_undegraded":"2024-09-17T07:27:20.387108+0000","last_fullsized":"2024-09-17T07:27:20.387108+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:27:00.694008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:27:00.694008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:01.315911+0000","scrub_duration":0.0020916659999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"439'27","reported_seq":63,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.759152+0000","last_change":"2024-09-17T07:27:08.703830+0000","last_active":"2024-09-17T07:27:20.759152+0000","last_peered":"2024-09-17T07:27:20.759152+0000","last_clean":"2024-09-17T07:27:20.759152+0000","last_became_active":"2024-09-17T07:25:49.445624+0000","last_became_peered":"2024-09-17T07:25:49.445624+0000","last_unstale":"2024-09-17T07:27:20.759152+0000","last_undegraded":"2024-09-17T07:27:20.759152+0000","last_fullsized":"2024-09-17T07:27:20.759152+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'27","last_scrub_stamp":"2024-09-17T07:27:08.703749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:27:08.703749+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:22.595218+0000","scrub_duration":0.002519126,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":643,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.807873+0000","last_change":"2024-09-17T07:27:10.086724+0000","last_active":"2024-09-17T07:27:20.807873+0000","last_peered":"2024-09-17T07:27:20.807873+0000","last_clean":"2024-09-17T07:27:20.807873+0000","last_became_active":"2024-09-17T07:23:57.486575+0000","last_became_peered":"2024-09-17T07:23:57.486575+0000","last_unstale":"2024-09-17T07:27:20.807873+0000","last_undegraded":"2024-09-17T07:27:20.807873+0000","last_fullsized":"2024-09-17T07:27:20.807873+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:27:10.086620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:27:10.086620+0000","last_clean_scrub_stamp":"2024-09-17T07:27:10.086620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:12.618485+0000","scrub_duration":0.001917787,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":145,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.387094+0000","last_change":"2024-09-17T07:27:09.706799+0000","last_active":"2024-09-17T07:27:20.387094+0000","last_peered":"2024-09-17T07:27:20.387094+0000","last_clean":"2024-09-17T07:27:20.387094+0000","last_became_active":"2024-09-17T07:22:13.710754+0000","last_became_peered":"2024-09-17T07:22:13.710754+0000","last_unstale":"2024-09-17T07:27:20.387094+0000","last_undegraded":"2024-09-17T07:27:20.387094+0000","last_fullsized":"2024-09-17T07:27:20.387094+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:27:09.706706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:14.999435+0000","last_clean_scrub_stamp":"2024-09-17T07:27:09.706706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:15.957958+0000","scrub_duration":0.001413168,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":51,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.759493+0000","last_change":"2024-09-17T07:26:20.900934+0000","last_active":"2024-09-17T07:27:20.759493+0000","last_peered":"2024-09-17T07:27:20.759493+0000","last_clean":"2024-09-17T07:27:20.759493+0000","last_became_active":"2024-09-17T07:25:49.435113+0000","last_became_peered":"2024-09-17T07:25:49.435113+0000","last_unstale":"2024-09-17T07:27:20.759493+0000","last_undegraded":"2024-09-17T07:27:20.759493+0000","last_fullsized":"2024-09-17T07:27:20.759493+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:20.900861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:25:17.663576+0000","last_clean_scrub_stamp":"2024-09-17T07:26:20.900861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:34.485626+0000","scrub_duration":0.001480269,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"326'2","reported_seq":164,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.387335+0000","last_change":"2024-09-17T07:27:14.727832+0000","last_active":"2024-09-17T07:27:20.387335+0000","last_peered":"2024-09-17T07:27:20.387335+0000","last_clean":"2024-09-17T07:27:20.387335+0000","last_became_active":"2024-09-17T07:23:57.475630+0000","last_became_peered":"2024-09-17T07:23:57.475630+0000","last_unstale":"2024-09-17T07:27:20.387335+0000","last_undegraded":"2024-09-17T07:27:20.387335+0000","last_fullsized":"2024-09-17T07:27:20.387335+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'2","last_scrub_stamp":"2024-09-17T07:27:14.727652+0000","last_deep_scrub":"326'2","last_deep_scrub_stamp":"2024-09-17T07:17:36.974358+0000","last_clean_scrub_stamp":"2024-09-17T07:27:14.727652+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:21.020923+0000","scrub_duration":0.001828327,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":604,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:22.702877+0000","last_change":"2024-09-17T07:26:44.461021+0000","last_active":"2024-09-17T07:27:22.702877+0000","last_peered":"2024-09-17T07:27:22.702877+0000","last_clean":"2024-09-17T07:27:22.702877+0000","last_became_active":"2024-09-17T07:25:50.426927+0000","last_became_peered":"2024-09-17T07:25:50.426927+0000","last_unstale":"2024-09-17T07:27:22.702877+0000","last_undegraded":"2024-09-17T07:27:22.702877+0000","last_fullsized":"2024-09-17T07:27:22.702877+0000","mapping_epoch":516,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":517,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:44.460914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:10.991363+0000","last_clean_scrub_stamp":"2024-09-17T07:26:44.460914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:13.609258+0000","scrub_duration":0.001578243,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"439'35","reported_seq":789,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:22.475089+0000","last_change":"2024-09-17T07:26:39.786707+0000","last_active":"2024-09-17T07:27:22.475089+0000","last_peered":"2024-09-17T07:27:22.475089+0000","last_clean":"2024-09-17T07:27:22.475089+0000","last_became_active":"2024-09-17T07:25:49.431783+0000","last_became_peered":"2024-09-17T07:25:49.431783+0000","last_unstale":"2024-09-17T07:27:22.475089+0000","last_undegraded":"2024-09-17T07:27:22.475089+0000","last_fullsized":"2024-09-17T07:27:22.475089+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'35","last_scrub_stamp":"2024-09-17T07:26:39.786542+0000","last_deep_scrub":"294'25","last_deep_scrub_stamp":"2024-09-17T07:16:00.471311+0000","last_clean_scrub_stamp":"2024-09-17T07:26:39.786542+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:51.502296+0000","scrub_duration":0.0022343789999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"525'11","reported_seq":970,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:25.805229+0000","last_change":"2024-09-17T07:27:25.239869+0000","last_active":"2024-09-17T07:27:25.805229+0000","last_peered":"2024-09-17T07:27:25.805229+0000","last_clean":"2024-09-17T07:27:25.805229+0000","last_became_active":"2024-09-17T07:22:13.574012+0000","last_became_peered":"2024-09-17T07:22:13.574012+0000","last_unstale":"2024-09-17T07:27:25.805229+0000","last_undegraded":"2024-09-17T07:27:25.805229+0000","last_fullsized":"2024-09-17T07:27:25.805229+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"525'11","last_scrub_stamp":"2024-09-17T07:27:25.239753+0000","last_deep_scrub":"525'11","last_deep_scrub_stamp":"2024-09-17T07:26:10.287750+0000","last_clean_scrub_stamp":"2024-09-17T07:27:25.239753+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:26.930369+0000","scrub_duration":0.001785341,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":477,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:23.380349+0000","last_change":"2024-09-17T07:27:23.380410+0000","last_active":"2024-09-17T07:27:23.380349+0000","last_peered":"2024-09-17T07:27:23.380349+0000","last_clean":"2024-09-17T07:27:23.380349+0000","last_became_active":"2024-09-17T07:23:57.478429+0000","last_became_peered":"2024-09-17T07:23:57.478429+0000","last_unstale":"2024-09-17T07:27:23.380349+0000","last_undegraded":"2024-09-17T07:27:23.380349+0000","last_fullsized":"2024-09-17T07:27:23.380349+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:27:23.380337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:26:20.318401+0000","last_clean_scrub_stamp":"2024-09-17T07:27:23.380337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:26.187066+0000","scrub_duration":0.0014809980000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":62,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.759242+0000","last_change":"2024-09-17T07:27:10.770333+0000","last_active":"2024-09-17T07:27:20.759242+0000","last_peered":"2024-09-17T07:27:20.759242+0000","last_clean":"2024-09-17T07:27:20.759242+0000","last_became_active":"2024-09-17T07:25:49.436539+0000","last_became_peered":"2024-09-17T07:25:49.436539+0000","last_unstale":"2024-09-17T07:27:20.759242+0000","last_undegraded":"2024-09-17T07:27:20.759242+0000","last_fullsized":"2024-09-17T07:27:20.759242+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:27:10.770256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:10:29.325664+0000","last_clean_scrub_stamp":"2024-09-17T07:27:10.770256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:35.082073+0000","scrub_duration":0.001687001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"513'32","reported_seq":733,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.807822+0000","last_change":"2024-09-17T07:27:03.038245+0000","last_active":"2024-09-17T07:27:20.807822+0000","last_peered":"2024-09-17T07:27:20.807822+0000","last_clean":"2024-09-17T07:27:20.807822+0000","last_became_active":"2024-09-17T07:23:57.486951+0000","last_became_peered":"2024-09-17T07:23:57.486951+0000","last_unstale":"2024-09-17T07:27:20.807822+0000","last_undegraded":"2024-09-17T07:27:20.807822+0000","last_fullsized":"2024-09-17T07:27:20.807822+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"513'32","last_scrub_stamp":"2024-09-17T07:27:03.038077+0000","last_deep_scrub":"513'32","last_deep_scrub_stamp":"2024-09-17T07:25:31.401365+0000","last_clean_scrub_stamp":"2024-09-17T07:27:03.038077+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:28.405884+0000","scrub_duration":0.0029557210000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":643,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:22.475061+0000","last_change":"2024-09-17T07:26:55.734925+0000","last_active":"2024-09-17T07:27:22.475061+0000","last_peered":"2024-09-17T07:27:22.475061+0000","last_clean":"2024-09-17T07:27:22.475061+0000","last_became_active":"2024-09-17T07:24:43.779553+0000","last_became_peered":"2024-09-17T07:24:43.779553+0000","last_unstale":"2024-09-17T07:27:22.475061+0000","last_undegraded":"2024-09-17T07:27:22.475061+0000","last_fullsized":"2024-09-17T07:27:22.475061+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:55.734820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:20:37.435106+0000","last_clean_scrub_stamp":"2024-09-17T07:26:55.734820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:00.306732+0000","scrub_duration":0.0017518200000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":619,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:19.463161+0000","last_change":"2024-09-17T07:26:28.848603+0000","last_active":"2024-09-17T07:27:19.463161+0000","last_peered":"2024-09-17T07:27:19.463161+0000","last_clean":"2024-09-17T07:27:19.463161+0000","last_became_active":"2024-09-17T07:23:57.473629+0000","last_became_peered":"2024-09-17T07:23:57.473629+0000","last_unstale":"2024-09-17T07:27:19.463161+0000","last_undegraded":"2024-09-17T07:27:19.463161+0000","last_fullsized":"2024-09-17T07:27:19.463161+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:28.848506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:17.002904+0000","last_clean_scrub_stamp":"2024-09-17T07:26:28.848506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:53.167436+0000","scrub_duration":0.0036431800000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":668,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:30.189709+0000","last_change":"2024-09-17T07:27:30.189816+0000","last_active":"2024-09-17T07:27:30.189709+0000","last_peered":"2024-09-17T07:27:30.189709+0000","last_clean":"2024-09-17T07:27:30.189709+0000","last_became_active":"2024-09-17T07:22:13.587273+0000","last_became_peered":"2024-09-17T07:22:13.587273+0000","last_unstale":"2024-09-17T07:27:30.189709+0000","last_undegraded":"2024-09-17T07:27:30.189709+0000","last_fullsized":"2024-09-17T07:27:30.189709+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:27:30.189691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:11:18.559877+0000","last_clean_scrub_stamp":"2024-09-17T07:27:30.189691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:38.380119+0000","scrub_duration":0.0014929850000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"439'39","reported_seq":497,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:23.881473+0000","last_change":"2024-09-17T07:27:23.881546+0000","last_active":"2024-09-17T07:27:23.881473+0000","last_peered":"2024-09-17T07:27:23.881473+0000","last_clean":"2024-09-17T07:27:23.881473+0000","last_became_active":"2024-09-17T07:26:09.602683+0000","last_became_peered":"2024-09-17T07:26:09.602683+0000","last_unstale":"2024-09-17T07:27:23.881473+0000","last_undegraded":"2024-09-17T07:27:23.881473+0000","last_fullsized":"2024-09-17T07:27:23.881473+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'39","last_scrub_stamp":"2024-09-17T07:27:23.881457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:02:12.994221+0000","last_clean_scrub_stamp":"2024-09-17T07:27:23.881457+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:50.761803+0000","scrub_duration":0.0020809660000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":441,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:33.293565+0000","last_change":"2024-09-17T07:27:33.293627+0000","last_active":"2024-09-17T07:27:33.293565+0000","last_peered":"2024-09-17T07:27:33.293565+0000","last_clean":"2024-09-17T07:27:33.293565+0000","last_became_active":"2024-09-17T07:21:15.348754+0000","last_became_peered":"2024-09-17T07:21:15.348754+0000","last_unstale":"2024-09-17T07:27:33.293565+0000","last_undegraded":"2024-09-17T07:27:33.293565+0000","last_fullsized":"2024-09-17T07:27:33.293565+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:27:33.293552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:24:23.943437+0000","last_clean_scrub_stamp":"2024-09-17T07:27:33.293552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:50.167390+0000","scrub_duration":0.0017582489999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":51,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.759841+0000","last_change":"2024-09-17T07:26:42.663839+0000","last_active":"2024-09-17T07:27:20.759841+0000","last_peered":"2024-09-17T07:27:20.759841+0000","last_clean":"2024-09-17T07:27:20.759841+0000","last_became_active":"2024-09-17T07:25:49.452257+0000","last_became_peered":"2024-09-17T07:25:49.452257+0000","last_unstale":"2024-09-17T07:27:20.759841+0000","last_undegraded":"2024-09-17T07:27:20.759841+0000","last_fullsized":"2024-09-17T07:27:20.759841+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:42.663753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:25:18.515421+0000","last_clean_scrub_stamp":"2024-09-17T07:26:42.663753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:11.481959+0000","scrub_duration":0.0016051360000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":62,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:20.759392+0000","last_change":"2024-09-17T07:27:17.770600+0000","last_active":"2024-09-17T07:27:20.759392+0000","last_peered":"2024-09-17T07:27:20.759392+0000","last_clean":"2024-09-17T07:27:20.759392+0000","last_became_active":"2024-09-17T07:25:49.445492+0000","last_became_peered":"2024-09-17T07:25:49.445492+0000","last_unstale":"2024-09-17T07:27:20.759392+0000","last_undegraded":"2024-09-17T07:27:20.759392+0000","last_fullsized":"2024-09-17T07:27:20.759392+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:27:17.770516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:24:26.807790+0000","last_clean_scrub_stamp":"2024-09-17T07:27:17.770516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:23.179623+0000","scrub_duration":0.001354919,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"439'29","reported_seq":74,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-09-17T07:27:22.774038+0000","last_change":"2024-09-17T07:27:22.774113+0000","last_active":"2024-09-17T07:27:22.774038+0000","last_peered":"2024-09-17T07:27:22.774038+0000","last_clean":"2024-09-17T07:27:22.774038+0000","last_became_active":"2024-09-17T07:25:49.448763+0000","last_became_peered":"2024-09-17T07:25:49.448763+0000","last_unstale":"2024-09-17T07:27:22.774038+0000","last_undegraded":"2024-09-17T07:27:22.774038+0000","last_fullsized":"2024-09-17T07:27:22.774038+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'29","last_scrub_stamp":"2024-09-17T07:27:22.774022+0000","last_deep_scrub":"86'17","last_deep_scrub_stamp":"2024-09-17T07:03:18.241067+0000","last_clean_scrub_stamp":"2024-09-17T07:27:22.774022+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:46.663010+0000","scrub_duration":0.0024735180000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":651,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-09-17T07:27:19.462966+0000","last_change":"2024-09-17T07:26:44.788543+0000","last_active":"2024-09-17T07:27:19.462966+0000","last_peered":"2024-09-17T07:27:19.462966+0000","last_clean":"2024-09-17T07:27:19.462966+0000","last_became_active":"2024-09-17T07:25:49.439685+0000","last_became_peered":"2024-09-17T07:25:49.439685+0000","last_unstale":"2024-09-17T07:27:19.462966+0000","last_undegraded":"2024-09-17T07:27:19.462966+0000","last_fullsized":"2024-09-17T07:27:19.462966+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:26:44.788431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:26:44.788431+0000","last_clean_scrub_stamp":"2024-09-17T07:26:44.788431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:00.018116+0000","scrub_duration":0.001656842,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":395,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101,"ondisk_log_size":101,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1534,"num_read_kb":1291,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":328,"num_bytes_recovered":6744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1042,"ondisk_log_size":1042,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":384,"num_read_kb":379,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1386,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":12,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":4158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1430,"ondisk_log_size":1430,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698038071,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1115060,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":1113982,"kb_avail":92630092,"statfs":{"total":95995035648,"available":94853214208,"internally_reserved":0,"allocated":1093632,"data_stored":700777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":1140718027},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":1.2889999999999999,"15min":1.2949999999999999},"min":{"1min":0.40100000000000002,"5min":0.182,"15min":0.182},"max":{"1min":0.79000000000000004,"5min":36.659999999999997,"15min":43.905000000000001},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.50700000000000001,"5min":1.1759999999999999,"15min":1.2330000000000001},"min":{"1min":0.34300000000000003,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.68799999999999994,"5min":36.555,"15min":44.432000000000002},"last":0.59299999999999997}]},{"osd":1,"last update":"Tue Sep 17 07:27:26 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.86199999999999999,"15min":1.1220000000000001},"min":{"1min":0.32400000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.67300000000000004,"5min":12.112,"15min":12.112},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.83099999999999996,"15min":1.0900000000000001},"min":{"1min":0.41899999999999998,"5min":0.216,"15min":0.216},"max":{"1min":0.65100000000000002,"5min":12.273,"15min":12.273},"last":0.63100000000000001}]},{"osd":2,"last update":"Tue Sep 17 07:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.80500000000000005,"15min":0.96999999999999997},"min":{"1min":0.372,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.76100000000000001,"5min":10.972,"15min":10.972},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.92000000000000004,"15min":0.997},"min":{"1min":0.40600000000000003,"5min":0.26500000000000001,"15min":0.224},"max":{"1min":0.85599999999999998,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.61599999999999999}]},{"osd":3,"last update":"Tue Sep 17 07:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.69499999999999995,"15min":0.93700000000000006},"min":{"1min":0.33800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.76000000000000001,"5min":4.2009999999999996,"15min":10.122999999999999},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.65700000000000003,"15min":0.90700000000000003},"min":{"1min":0.36299999999999999,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.81499999999999995,"5min":4.1719999999999997,"15min":12.465999999999999},"last":0.54400000000000004}]},{"osd":4,"last update":"Tue Sep 17 07:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.751,"15min":1.042},"min":{"1min":0.34599999999999997,"5min":0.22900000000000001,"15min":0.222},"max":{"1min":0.70799999999999996,"5min":13.316000000000001,"15min":21.140999999999998},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.80300000000000005,"15min":1.1930000000000001},"min":{"1min":0.27200000000000002,"5min":0.25800000000000001,"15min":0.23599999999999999},"max":{"1min":0.83699999999999997,"5min":12.272,"15min":19.420000000000002},"last":0.47599999999999998}]},{"osd":5,"last update":"Tue Sep 17 07:27:01 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.77400000000000002,"15min":0.91600000000000004},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.90100000000000002,"5min":3.0859999999999999,"15min":3.0859999999999999},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.504,"5min":0.85799999999999998,"15min":1.0169999999999999},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.73599999999999999,"5min":5.7350000000000003,"15min":5.7350000000000003},"last":0.46300000000000002}]},{"osd":6,"last update":"Tue Sep 17 07:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":1.4139999999999999,"15min":1.446},"min":{"1min":0.33600000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.77900000000000003,"5min":60.445999999999998,"15min":60.445999999999998},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.54800000000000004,"5min":1.49,"15min":1.544},"min":{"1min":0.32700000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.78500000000000003,"5min":60.332000000000001,"15min":60.332000000000001},"last":0.42599999999999999}]}]},{"osd":6,"up_from":50,"seq":214748365114,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1036772,"kb_used_data":732,"kb_used_omap":1,"kb_used_meta":1036030,"kb_avail":92708380,"statfs":{"total":95995035648,"available":94933381120,"internally_reserved":0,"allocated":749568,"data_stored":372404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1060895175},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:26:34 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":3.2189999999999999,"15min":2.339},"min":{"1min":0.32300000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.74099999999999999,"5min":110.376,"15min":126.902},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.51900000000000002,"5min":3.4009999999999998,"15min":2.2519999999999998},"min":{"1min":0.41199999999999998,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.64500000000000002,"5min":110.164,"15min":127.06399999999999},"last":0.58899999999999997}]},{"osd":1,"last update":"Tue Sep 17 07:27:30 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":2.7450000000000001,"15min":4.9470000000000001},"min":{"1min":0.40500000000000003,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.69199999999999995,"5min":199.32400000000001,"15min":199.32400000000001},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.52500000000000002,"5min":2.702,"15min":5.1769999999999996},"min":{"1min":0.432,"5min":0.251,"15min":0.24099999999999999},"max":{"1min":0.67100000000000004,"5min":198.88499999999999,"15min":198.88499999999999},"last":0.51600000000000001}]},{"osd":2,"last update":"Tue Sep 17 07:26:34 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.81699999999999995,"15min":1.159},"min":{"1min":0.38600000000000001,"5min":0.23300000000000001,"15min":0.219},"max":{"1min":0.67500000000000004,"5min":10.930999999999999,"15min":36.289000000000001},"last":0.501},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.96699999999999997,"15min":1.101},"min":{"1min":0.36399999999999999,"5min":0.26100000000000001,"15min":0.22900000000000001},"max":{"1min":0.81899999999999995,"5min":7.1280000000000001,"15min":36.326999999999998},"last":0.63400000000000001}]},{"osd":3,"last update":"Tue Sep 17 07:26:34 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.79200000000000004,"15min":0.873},"min":{"1min":0.34300000000000003,"5min":0.27700000000000002,"15min":0.249},"max":{"1min":0.80600000000000005,"5min":9.0579999999999998,"15min":9.0579999999999998},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.875,"15min":1.0109999999999999},"min":{"1min":0.40500000000000003,"5min":0.28000000000000003,"15min":0.27500000000000002},"max":{"1min":0.72899999999999998,"5min":10.999000000000001,"15min":10.999000000000001},"last":0.56399999999999995}]},{"osd":4,"last update":"Tue Sep 17 07:26:34 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":1.0089999999999999,"15min":1.1339999999999999},"min":{"1min":0.30399999999999999,"5min":0.24199999999999999,"15min":0.19900000000000001},"max":{"1min":0.68700000000000006,"5min":9.5419999999999998,"15min":9.5419999999999998},"last":0.373},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.80400000000000005,"15min":1.141},"min":{"1min":0.29299999999999998,"5min":0.26100000000000001,"15min":0.23300000000000001},"max":{"1min":0.69399999999999995,"5min":9.0310000000000006,"15min":9.0310000000000006},"last":0.48099999999999998}]},{"osd":5,"last update":"Tue Sep 17 07:26:59 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":1.472,"15min":1.909},"min":{"1min":0.307,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.72299999999999998,"5min":13.551,"15min":13.551},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.52700000000000002,"5min":1.637,"15min":2.137},"min":{"1min":0.33200000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.72399999999999998,"5min":8.1720000000000006,"15min":8.1720000000000006},"last":0.54600000000000004}]},{"osd":7,"last update":"Tue Sep 17 07:26:54 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.94299999999999995,"15min":1.6599999999999999},"min":{"1min":0.33900000000000002,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.77100000000000002,"5min":9.1199999999999992,"15min":131.95699999999999},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.60699999999999998,"5min":1.0720000000000001,"15min":1.6000000000000001},"min":{"1min":0.38900000000000001,"5min":0.29499999999999998,"15min":0.252},"max":{"1min":0.75600000000000001,"5min":9.5850000000000009,"15min":131.833},"last":0.61299999999999999}]}]},{"osd":5,"up_from":476,"seq":2044404432942,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":122668,"kb_used_data":1316,"kb_used_omap":4,"kb_used_meta":121339,"kb_avail":93622484,"statfs":{"total":95995035648,"available":95869423616,"internally_reserved":0,"allocated":1347584,"data_stored":963582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4441,"internal_metadata":124251815},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.82199999999999995,"15min":0.94999999999999996},"min":{"1min":0.372,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.73599999999999999,"5min":3.4220000000000002,"15min":3.4220000000000002},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.50800000000000001,"5min":1.169,"15min":1.4530000000000001},"min":{"1min":0.33000000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.67500000000000004,"5min":6.7619999999999996,"15min":6.7619999999999996},"last":0.505}]},{"osd":1,"last update":"Tue Sep 17 07:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.89900000000000002,"15min":1.0609999999999999},"min":{"1min":0.27700000000000002,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.75,"5min":3.476,"15min":3.476},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.85599999999999998,"15min":0.98299999999999998},"min":{"1min":0.317,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":3.0049999999999999,"5min":6.665,"15min":6.665},"last":0.379}]},{"osd":2,"last update":"Tue Sep 17 07:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.94399999999999995,"15min":1.127},"min":{"1min":0.33800000000000002,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.71399999999999997,"5min":3.544,"15min":3.544},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.78900000000000003,"15min":0.90700000000000003},"min":{"1min":0.39000000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.77000000000000002,"5min":2.8359999999999999,"15min":2.8359999999999999},"last":0.44500000000000001}]},{"osd":3,"last update":"Tue Sep 17 07:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.76100000000000001,"15min":0.871},"min":{"1min":0.35399999999999998,"5min":0.314,"15min":0.314},"max":{"1min":0.69599999999999995,"5min":3.7040000000000002,"15min":3.7040000000000002},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.75700000000000001,"15min":0.84599999999999997},"min":{"1min":0.41699999999999998,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.73099999999999998,"5min":2.8719999999999999,"15min":2.8719999999999999},"last":0.36399999999999999}]},{"osd":4,"last update":"Tue Sep 17 07:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.82199999999999995,"15min":0.98199999999999998},"min":{"1min":0.318,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.69899999999999995,"5min":3.4399999999999999,"15min":3.4399999999999999},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.94299999999999995,"15min":1.1539999999999999},"min":{"1min":0.29899999999999999,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.68799999999999994,"5min":3.0139999999999998,"15min":3.0139999999999998},"last":0.307}]},{"osd":6,"last update":"Tue Sep 17 07:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":1.2490000000000001,"15min":1.5960000000000001},"min":{"1min":0.28299999999999997,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.73599999999999999,"5min":10.92,"15min":10.92},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.52100000000000002,"5min":1.113,"15min":1.3879999999999999},"min":{"1min":0.35499999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.69899999999999995,"5min":10.202999999999999,"15min":10.202999999999999},"last":0.33400000000000002}]},{"osd":7,"last update":"Tue Sep 17 07:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":2.2400000000000002,"15min":3.0009999999999999},"min":{"1min":0.30199999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.72099999999999997,"5min":45.555,"15min":45.555},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.53200000000000003,"5min":2.4220000000000002,"15min":3.2679999999999998},"min":{"1min":0.33900000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.71999999999999997,"5min":47.213999999999999,"15min":47.213999999999999},"last":0.438}]}]},{"osd":4,"up_from":33,"seq":141733921093,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1002356,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":1001598,"kb_avail":92742796,"statfs":{"total":95995035648,"available":94968623104,"internally_reserved":0,"allocated":765952,"data_stored":372528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1025636806},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":1.601,"15min":1.319},"min":{"1min":0.30499999999999999,"5min":0.20300000000000001,"15min":0.20000000000000001},"max":{"1min":0.65600000000000003,"5min":39.473999999999997,"15min":39.473999999999997},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.60499999999999998,"5min":1.631,"15min":1.3700000000000001},"min":{"1min":0.41299999999999998,"5min":0.21299999999999999,"15min":0.20499999999999999},"max":{"1min":0.73899999999999999,"5min":39.654000000000003,"15min":39.654000000000003},"last":0.45300000000000001}]},{"osd":1,"last update":"Tue Sep 17 07:27:26 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":1.1439999999999999,"15min":2.302},"min":{"1min":0.35499999999999998,"5min":0.221,"15min":0.221},"max":{"1min":0.65100000000000002,"5min":39.439,"15min":39.439},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.54800000000000004,"5min":1.3069999999999999,"15min":2.3719999999999999},"min":{"1min":0.39700000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.73199999999999998,"5min":39.325000000000003,"15min":39.325000000000003},"last":0.53900000000000003}]},{"osd":2,"last update":"Tue Sep 17 07:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":2.391,"15min":1.4910000000000001},"min":{"1min":0.371,"5min":0.24399999999999999,"15min":0.23499999999999999},"max":{"1min":7.1539999999999999,"5min":126.628,"15min":126.628},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.61399999999999999,"5min":2.3530000000000002,"15min":1.476},"min":{"1min":0.41799999999999998,"5min":0.27400000000000002,"15min":0.23400000000000001},"max":{"1min":0.76100000000000001,"5min":126.41200000000001,"15min":126.41200000000001},"last":0.67200000000000004}]},{"osd":3,"last update":"Tue Sep 17 07:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.66700000000000004,"15min":0.85299999999999998},"min":{"1min":0.41699999999999998,"5min":0.28299999999999997,"15min":0.25800000000000001},"max":{"1min":0.71799999999999997,"5min":2.8929999999999998,"15min":15.433999999999999},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.72099999999999997,"15min":0.91200000000000003},"min":{"1min":0.35199999999999998,"5min":0.29699999999999999,"15min":0.28100000000000003},"max":{"1min":0.68799999999999994,"5min":8.7360000000000007,"15min":16.111000000000001},"last":0.47699999999999998}]},{"osd":5,"last update":"Tue Sep 17 07:27:04 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":1.038,"15min":1.3029999999999999},"min":{"1min":0.33800000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.60899999999999999,"5min":13.128,"15min":13.128},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.46400000000000002,"5min":1.2370000000000001,"15min":1.5800000000000001},"min":{"1min":0.32400000000000001,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.622,"5min":13.218999999999999,"15min":13.218999999999999},"last":0.497}]},{"osd":6,"last update":"Tue Sep 17 07:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.72899999999999998,"15min":1.01},"min":{"1min":0.40300000000000002,"5min":0.249,"15min":0.23400000000000001},"max":{"1min":0.69499999999999995,"5min":4.7729999999999997,"15min":8.9619999999999997},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.82799999999999996,"15min":1.1419999999999999},"min":{"1min":0.30599999999999999,"5min":0.249,"15min":0.24199999999999999},"max":{"1min":0.64300000000000002,"5min":6.782,"15min":10.051},"last":0.52100000000000002}]},{"osd":7,"last update":"Tue Sep 17 07:26:47 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.93100000000000005,"15min":1.123},"min":{"1min":0.372,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.68700000000000006,"5min":6.2249999999999996,"15min":7.1550000000000002},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.72599999999999998,"15min":1.1220000000000001},"min":{"1min":0.38900000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.65700000000000003,"5min":4.7439999999999998,"15min":9.0030000000000001},"last":0.51200000000000001}]}]},{"osd":3,"up_from":27,"seq":115964117319,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":371280,"kb_used_data":1352,"kb_used_omap":1,"kb_used_meta":369918,"kb_avail":93373872,"statfs":{"total":95995035648,"available":95614844928,"internally_reserved":0,"allocated":1384448,"data_stored":963680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":378796488},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:27:33 2024","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":1.091,"15min":0.99099999999999999},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.57099999999999995,"5min":46.442,"15min":65.192999999999998},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.315,"5min":1.1319999999999999,"15min":1.0309999999999999},"min":{"1min":0.20399999999999999,"5min":0.188,"15min":0.182},"max":{"1min":0.52100000000000002,"5min":46.633000000000003,"15min":64.998000000000005},"last":0.28999999999999998}]},{"osd":1,"last update":"Tue Sep 17 07:27:26 2024","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.54100000000000004,"15min":0.90800000000000003},"min":{"1min":0.218,"5min":0.20499999999999999,"15min":0.20200000000000001},"max":{"1min":0.47299999999999998,"5min":6.1719999999999997,"15min":6.1719999999999997},"last":0.223},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.64600000000000002,"15min":0.78900000000000003},"min":{"1min":0.23300000000000001,"5min":0.214,"15min":0.214},"max":{"1min":0.54700000000000004,"5min":8.1489999999999991,"15min":8.1489999999999991},"last":0.39000000000000001}]},{"osd":2,"last update":"Tue Sep 17 07:27:33 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.56499999999999995,"15min":0.52100000000000002},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.188},"max":{"1min":0.55200000000000005,"5min":8.0609999999999999,"15min":8.0609999999999999},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.33200000000000002,"5min":0.51500000000000001,"15min":0.52100000000000002},"min":{"1min":0.25700000000000001,"5min":0.23200000000000001,"15min":0.20599999999999999},"max":{"1min":0.498,"5min":2.899,"15min":5.5389999999999997},"last":0.315}]},{"osd":4,"last update":"Tue Sep 17 07:27:33 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.80700000000000005,"15min":1.5129999999999999},"min":{"1min":0.34300000000000003,"5min":0.28100000000000003,"15min":0.27700000000000002},"max":{"1min":0.72799999999999998,"5min":16.146000000000001,"15min":61.639000000000003},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.91500000000000004,"15min":1.486},"min":{"1min":0.39200000000000002,"5min":0.31,"15min":0.27200000000000002},"max":{"1min":0.876,"5min":16.129999999999999,"15min":61.036000000000001},"last":0.437}]},{"osd":5,"last update":"Tue Sep 17 07:27:08 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.56100000000000005,"15min":0.55100000000000005},"min":{"1min":0.437,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.84099999999999997,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.60499999999999998,"15min":0.60599999999999998},"min":{"1min":0.47099999999999997,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.85099999999999998,"5min":1.417,"15min":1.417},"last":0.37}]},{"osd":6,"last update":"Tue Sep 17 07:27:33 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.63700000000000001,"15min":1.127},"min":{"1min":0.378,"5min":0.316,"15min":0.29699999999999999},"max":{"1min":0.70999999999999996,"5min":3.7599999999999998,"15min":68.188999999999993},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.81599999999999995,"15min":1.264},"min":{"1min":0.32900000000000001,"5min":0.32300000000000001,"15min":0.26300000000000001},"max":{"1min":0.78100000000000003,"5min":9.6530000000000005,"15min":68.245000000000005},"last":0.32900000000000001}]},{"osd":7,"last update":"Tue Sep 17 07:26:38 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":1.375,"15min":1.4910000000000001},"min":{"1min":0.438,"5min":0.317,"15min":0.29399999999999998},"max":{"1min":0.81499999999999995,"5min":63.607999999999997,"15min":63.607999999999997},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.59199999999999997,"5min":1.3500000000000001,"15min":1.387},"min":{"1min":0.47099999999999997,"5min":0.33500000000000002,"15min":0.28000000000000003},"max":{"1min":0.76600000000000001,"5min":63.709000000000003,"15min":63.709000000000003},"last":0.52700000000000002}]}]},{"osd":2,"up_from":18,"seq":77309411661,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1122928,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":1121918,"kb_avail":92622224,"statfs":{"total":95995035648,"available":94845157376,"internally_reserved":0,"allocated":1024000,"data_stored":635749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1148844486},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:27:10 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":2.5979999999999999,"15min":2.2469999999999999},"min":{"1min":0.20000000000000001,"5min":0.16600000000000001,"15min":0.154},"max":{"1min":0.501,"5min":69.135999999999996,"15min":131.09800000000001},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.32800000000000001,"5min":2.6699999999999999,"15min":2.3090000000000002},"min":{"1min":0.20699999999999999,"5min":0.17999999999999999,"15min":0.17599999999999999},"max":{"1min":0.48499999999999999,"5min":69.033000000000001,"15min":131.24700000000001},"last":0.46400000000000002}]},{"osd":1,"last update":"Tue Sep 17 07:26:31 2024","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":2.2450000000000001,"15min":2.0600000000000001},"min":{"1min":0.192,"5min":0.184,"15min":0.184},"max":{"1min":0.44700000000000001,"5min":91.686999999999998,"15min":91.686999999999998},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.34300000000000003,"5min":2.5089999999999999,"15min":2.3780000000000001},"min":{"1min":0.255,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.51800000000000002,"5min":93.744,"15min":93.744},"last":0.50700000000000001}]},{"osd":3,"last update":"Tue Sep 17 07:27:17 2024","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.47999999999999998,"15min":0.51000000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.20300000000000001},"max":{"1min":0.46999999999999997,"5min":2.7469999999999999,"15min":4.6449999999999996},"last":0.44},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.73699999999999999,"15min":0.61899999999999999},"min":{"1min":0.27100000000000002,"5min":0.23100000000000001,"15min":0.22},"max":{"1min":0.49399999999999999,"5min":16.215,"15min":16.215},"last":0.52500000000000002}]},{"osd":4,"last update":"Tue Sep 17 07:26:45 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":1.038,"15min":1.272},"min":{"1min":0.316,"5min":0.28199999999999997,"15min":0.24399999999999999},"max":{"1min":0.75700000000000001,"5min":25.225999999999999,"15min":75.870999999999995},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.55200000000000005,"5min":1.054,"15min":1.383},"min":{"1min":0.36899999999999999,"5min":0.27700000000000002,"15min":0.26300000000000001},"max":{"1min":0.70299999999999996,"5min":25.268999999999998,"15min":75.888999999999996},"last":0.76900000000000002}]},{"osd":5,"last update":"Tue Sep 17 07:26:59 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.59499999999999997,"15min":0.61899999999999999},"min":{"1min":0.35699999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.69199999999999995,"5min":1.9610000000000001,"15min":1.9610000000000001},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.97099999999999997,"15min":0.96699999999999997},"min":{"1min":0.38400000000000001,"5min":0.312,"15min":0.312},"max":{"1min":0.77800000000000002,"5min":17.010000000000002,"15min":17.010000000000002},"last":0.752}]},{"osd":6,"last update":"Tue Sep 17 07:27:17 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":1.3240000000000001,"15min":1.708},"min":{"1min":0.41599999999999998,"5min":0.29699999999999999,"15min":0.27500000000000002},"max":{"1min":0.72499999999999998,"5min":24.936,"15min":76.081999999999994},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.55700000000000005,"5min":1.153,"15min":1.7470000000000001},"min":{"1min":0.38100000000000001,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.67300000000000004,"5min":24.884,"15min":76.244},"last":0.82999999999999996}]},{"osd":7,"last update":"Tue Sep 17 07:27:10 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.90800000000000003,"15min":1.3180000000000001},"min":{"1min":0.45000000000000001,"5min":0.314,"15min":0.314},"max":{"1min":0.76600000000000001,"5min":16.98,"15min":94.269999999999996},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.57899999999999996,"5min":1.0029999999999999,"15min":1.353},"min":{"1min":0.44500000000000001,"5min":0.32900000000000001,"15min":0.309},"max":{"1min":0.73099999999999998,"5min":17.030999999999999,"15min":94.634},"last":0.74399999999999999}]}]},{"osd":1,"up_from":413,"seq":1773821493326,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":534476,"kb_used_data":772,"kb_used_omap":4,"kb_used_meta":533691,"kb_avail":93210676,"statfs":{"total":95995035648,"available":95447732224,"internally_reserved":0,"allocated":790528,"data_stored":374233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4439,"internal_metadata":546500265},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:27:29 2024","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":2.0680000000000001,"15min":1.8029999999999999},"min":{"1min":0.21099999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":1.22,"5min":120.56399999999999,"15min":120.56399999999999},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.37,"5min":2.1699999999999999,"15min":1.6499999999999999},"min":{"1min":0.23000000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":1.022,"5min":120.746,"15min":120.746},"last":0.34399999999999997}]},{"osd":2,"last update":"Tue Sep 17 07:27:29 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.65900000000000003,"15min":1.127},"min":{"1min":0.27100000000000002,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":1.0760000000000001,"5min":6.7300000000000004,"15min":6.7300000000000004},"last":0.373},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.56999999999999995,"15min":0.94299999999999995},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":1.2969999999999999,"5min":3.919,"15min":3.919},"last":0.27800000000000002}]},{"osd":3,"last update":"Tue Sep 17 07:27:29 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.56699999999999995,"15min":0.73699999999999999},"min":{"1min":0.253,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":1.194,"5min":4.6749999999999998,"15min":5.9290000000000003},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.433,"5min":0.57199999999999995,"15min":0.77700000000000002},"min":{"1min":0.313,"5min":0.25,"15min":0.25},"max":{"1min":1.0489999999999999,"5min":3.3149999999999999,"15min":4.1340000000000003},"last":0.311}]},{"osd":4,"last update":"Tue Sep 17 07:27:29 2024","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.82899999999999996,"15min":5.181},"min":{"1min":0.37,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":6.774,"5min":6.774,"15min":116.88200000000001},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.71799999999999997,"15min":5.1900000000000004},"min":{"1min":0.41799999999999998,"5min":0.316,"15min":0.316},"max":{"1min":1.113,"5min":4.7089999999999996,"15min":116.98399999999999},"last":0.49299999999999999}]},{"osd":5,"last update":"Tue Sep 17 07:27:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":1.153,"15min":1.409},"min":{"1min":0.40100000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.139,"5min":9.7409999999999997,"15min":9.7409999999999997},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.57999999999999996,"5min":1.1539999999999999,"15min":1.4119999999999999},"min":{"1min":0.41799999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.996,"5min":11.538,"15min":11.538},"last":0.48199999999999998}]},{"osd":6,"last update":"Tue Sep 17 07:27:29 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":1.714,"15min":1.5249999999999999},"min":{"1min":0.40300000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.169,"5min":56.686999999999998,"15min":56.686999999999998},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.62,"5min":1.6819999999999999,"15min":1.446},"min":{"1min":0.40400000000000003,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.96599999999999997,"5min":56.420999999999999,"15min":56.420999999999999},"last":0.52800000000000002}]},{"osd":7,"last update":"Tue Sep 17 07:27:29 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.81499999999999995,"15min":0.98299999999999998},"min":{"1min":0.41299999999999998,"5min":0.34699999999999998,"15min":0.34399999999999997},"max":{"1min":0.92500000000000004,"5min":3.4399999999999999,"15min":4.2389999999999999},"last":0.502},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.70699999999999996,"15min":0.93700000000000006},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.2410000000000001,"5min":4.2839999999999998,"15min":4.2839999999999998},"last":0.54400000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738709,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1307792,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":1307006,"kb_avail":92437360,"statfs":{"total":95995035648,"available":94655856640,"internally_reserved":0,"allocated":790528,"data_stored":380109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1338374598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 17 07:27:28 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":1.2789999999999999,"15min":1.7649999999999999},"min":{"1min":0.224,"5min":0.17199999999999999,"15min":0.156},"max":{"1min":0.63400000000000001,"5min":53.731000000000002,"15min":53.731000000000002},"last":0.317},{"interface":"front","average":{"1min":0.38,"5min":1.367,"15min":1.4610000000000001},"min":{"1min":0.22800000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.61799999999999999,"5min":55.789000000000001,"15min":55.789000000000001},"last":0.39700000000000002}]},{"osd":2,"last update":"Tue Sep 17 07:27:01 2024","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":1.3140000000000001,"15min":1.115},"min":{"1min":0.24099999999999999,"5min":0.21299999999999999,"15min":0.155},"max":{"1min":0.59099999999999997,"5min":55.756,"15min":62.094999999999999},"last":0.371},{"interface":"front","average":{"1min":0.37,"5min":1.377,"15min":1.1419999999999999},"min":{"1min":0.251,"5min":0.20899999999999999,"15min":0.183},"max":{"1min":0.57599999999999996,"5min":57.854999999999997,"15min":59.848999999999997},"last":0.34499999999999997}]},{"osd":3,"last update":"Tue Sep 17 07:26:42 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.621,"15min":0.59799999999999998},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.59099999999999997,"5min":4.6829999999999998,"15min":5.5880000000000001},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.60699999999999998,"15min":0.55900000000000005},"min":{"1min":0.22700000000000001,"5min":0.20799999999999999,"15min":0.20399999999999999},"max":{"1min":0.46899999999999997,"5min":4.6109999999999998,"15min":5.1150000000000002},"last":0.36099999999999999}]},{"osd":4,"last update":"Tue Sep 17 07:26:42 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":1.018,"15min":1.02},"min":{"1min":0.36199999999999999,"5min":0.27400000000000002,"15min":0.23400000000000001},"max":{"1min":0.75,"5min":10.720000000000001,"15min":10.720000000000001},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.58799999999999997,"5min":1.0469999999999999,"15min":1.038},"min":{"1min":0.42499999999999999,"5min":0.26600000000000001,"15min":0.26200000000000001},"max":{"1min":0.73899999999999999,"5min":10.666,"15min":10.666},"last":0.58499999999999996}]},{"osd":5,"last update":"Tue Sep 17 07:27:01 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.872,"15min":0.99399999999999999},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.81100000000000005,"5min":3.79,"15min":3.79},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.84799999999999998,"15min":0.95599999999999996},"min":{"1min":0.40699999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.80000000000000004,"5min":3.2530000000000001,"15min":3.2530000000000001},"last":0.624}]},{"osd":6,"last update":"Tue Sep 17 07:26:42 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":1.411,"15min":1.819},"min":{"1min":0.375,"5min":0.28999999999999998,"15min":0.28199999999999997},"max":{"1min":0.85299999999999998,"5min":37.262999999999998,"15min":147.13200000000001},"last":0.499},{"interface":"front","average":{"1min":0.57199999999999995,"5min":1.536,"15min":1.8169999999999999},"min":{"1min":0.37,"5min":0.311,"15min":0.30199999999999999},"max":{"1min":0.75,"5min":36.789999999999999,"15min":147.172},"last":0.59499999999999997}]},{"osd":7,"last update":"Tue Sep 17 07:26:42 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.77000000000000002,"15min":1.054},"min":{"1min":0.41999999999999998,"5min":0.318,"15min":0.30599999999999999},"max":{"1min":0.76700000000000002,"5min":3.843,"15min":6.3300000000000001},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.89900000000000002,"15min":1.121},"min":{"1min":0.371,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.82599999999999996,"5min":4.9100000000000001,"15min":7.2830000000000004},"last":0.52400000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T07:27:39.080 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-17T07:27:39.080 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 shell --fsid e848267c-74c1-11ef-bceb-c7b262605968 -- ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-09-17T07:27:39.278 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:27:39.699 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:39 smithi100 ceph-mon[29092]: 5.11 scrub starts 2024-09-17T07:27:39.699 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:39 smithi100 ceph-mon[29092]: 5.11 scrub ok 2024-09-17T07:27:39.699 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:39 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:27:39.699 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:39 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:27:39.700 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:39 smithi100 ceph-mon[39462]: 5.11 scrub starts 2024-09-17T07:27:39.700 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:39 smithi100 ceph-mon[39462]: 5.11 scrub ok 2024-09-17T07:27:39.700 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:39 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:27:39.700 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:39 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:27:39.793 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:27:39 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug there is no tcmu-runner data available 2024-09-17T07:27:39.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:39 smithi135 ceph-mon[33238]: 5.11 scrub starts 2024-09-17T07:27:39.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:39 smithi135 ceph-mon[33238]: 5.11 scrub ok 2024-09-17T07:27:39.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:39 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:27:39.793 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:39 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:27:40.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:40 smithi100 ceph-mon[29092]: 5.18 scrub starts 2024-09-17T07:27:40.876 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:40 smithi100 ceph-mon[29092]: 5.18 scrub ok 2024-09-17T07:27:40.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:40 smithi100 ceph-mon[29092]: osdmap e548: 8 total, 8 up, 8 in 2024-09-17T07:27:40.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:40 smithi100 ceph-mon[29092]: pgmap v1219: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:27:40.877 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:40 smithi100 ceph-mon[29092]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:27:40.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:40 smithi100 ceph-mon[39462]: 5.18 scrub starts 2024-09-17T07:27:40.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:40 smithi100 ceph-mon[39462]: 5.18 scrub ok 2024-09-17T07:27:40.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:40 smithi100 ceph-mon[39462]: osdmap e548: 8 total, 8 up, 8 in 2024-09-17T07:27:40.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:40 smithi100 ceph-mon[39462]: pgmap v1219: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:27:40.877 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:40 smithi100 ceph-mon[39462]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:27:41.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:40 smithi135 ceph-mon[33238]: 5.18 scrub starts 2024-09-17T07:27:41.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:40 smithi135 ceph-mon[33238]: 5.18 scrub ok 2024-09-17T07:27:41.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:40 smithi135 ceph-mon[33238]: osdmap e548: 8 total, 8 up, 8 in 2024-09-17T07:27:41.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:40 smithi135 ceph-mon[33238]: pgmap v1219: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:27:41.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:40 smithi135 ceph-mon[33238]: from='client.24428 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:27:41.546 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:27:41 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: [17/Sep/2024:07:27:41] ENGINE Bus STOPPING 2024-09-17T07:27:41.546 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:27:41 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: [17/Sep/2024:07:27:41] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-17T07:27:41.546 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:27:41 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: [17/Sep/2024:07:27:41] ENGINE Bus STOPPED 2024-09-17T07:27:41.546 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:27:41 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: [17/Sep/2024:07:27:41] ENGINE Bus STARTING 2024-09-17T07:27:41.546 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:27:41 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: [17/Sep/2024:07:27:41] ENGINE Serving on http://:::9283 2024-09-17T07:27:41.546 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:27:41 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: [17/Sep/2024:07:27:41] ENGINE Bus STARTED 2024-09-17T07:27:41.855 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:41 smithi100 ceph-mon[39462]: purged_snaps scrub starts 2024-09-17T07:27:41.856 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:41 smithi100 ceph-mon[39462]: purged_snaps scrub ok 2024-09-17T07:27:41.856 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:41 smithi100 ceph-mon[39462]: osdmap e549: 8 total, 8 up, 8 in 2024-09-17T07:27:41.856 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:41 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2342987625' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-09-17T07:27:41.856 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:41 smithi100 ceph-mon[39462]: from='client.? 172.21.15.100:0/2342987625' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-09-17T07:27:41.856 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:41 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:27:41.856 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:41 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:27:41.856 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:41 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:27:41.856 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:41 smithi100 ceph-mon[39462]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:27:41.857 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:41 smithi100 ceph-mon[29092]: purged_snaps scrub starts 2024-09-17T07:27:41.857 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:41 smithi100 ceph-mon[29092]: purged_snaps scrub ok 2024-09-17T07:27:41.857 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:41 smithi100 ceph-mon[29092]: osdmap e549: 8 total, 8 up, 8 in 2024-09-17T07:27:41.857 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:41 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2342987625' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-09-17T07:27:41.857 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:41 smithi100 ceph-mon[29092]: from='client.? 172.21.15.100:0/2342987625' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-09-17T07:27:41.857 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:41 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:27:41.857 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:41 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:27:41.857 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:41 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:27:41.857 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:41 smithi100 ceph-mon[29092]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:27:42.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:41 smithi135 ceph-mon[33238]: purged_snaps scrub starts 2024-09-17T07:27:42.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:41 smithi135 ceph-mon[33238]: purged_snaps scrub ok 2024-09-17T07:27:42.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:41 smithi135 ceph-mon[33238]: osdmap e549: 8 total, 8 up, 8 in 2024-09-17T07:27:42.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:41 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2342987625' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-09-17T07:27:42.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:41 smithi135 ceph-mon[33238]: from='client.? 172.21.15.100:0/2342987625' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-09-17T07:27:42.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:41 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:27:42.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:41 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:27:42.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:41 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:27:42.044 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:41 smithi135 ceph-mon[33238]: from='mgr.14562 172.21.15.100:0/4167934111' entity='mgr.y' 2024-09-17T07:27:42.169 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-09-17T07:27:42.184 INFO:tasks.cephadm:Teardown begin 2024-09-17T07:27:42.184 DEBUG:teuthology.orchestra.run.smithi100:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-17T07:27:42.223 DEBUG:teuthology.orchestra.run.smithi135:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-17T07:27:42.288 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-09-17T07:27:42.288 DEBUG:teuthology.orchestra.run.smithi100:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-17T07:27:42.312 DEBUG:teuthology.orchestra.run.smithi135:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-17T07:27:42.334 INFO:tasks.cephadm:Stopping all daemons... 2024-09-17T07:27:42.334 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-09-17T07:27:42.334 DEBUG:teuthology.orchestra.run.smithi100:> sudo systemctl stop ceph-e848267c-74c1-11ef-bceb-c7b262605968@mon.a 2024-09-17T07:27:42.661 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:42 smithi100 systemd[1]: Stopping Ceph mon.a for e848267c-74c1-11ef-bceb-c7b262605968... 2024-09-17T07:27:42.661 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:42 smithi100 ceph-mon[29092]: 6.1 deep-scrub starts 2024-09-17T07:27:42.661 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:42 smithi100 ceph-mon[29092]: 6.1 deep-scrub ok 2024-09-17T07:27:42.661 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:42 smithi100 ceph-mon[29092]: pgmap v1221: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 725 B/s rd, 0 op/s 2024-09-17T07:27:42.661 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:42 smithi100 ceph-mon[39462]: 6.1 deep-scrub starts 2024-09-17T07:27:42.661 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:42 smithi100 ceph-mon[39462]: 6.1 deep-scrub ok 2024-09-17T07:27:42.661 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:42 smithi100 ceph-mon[39462]: pgmap v1221: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 725 B/s rd, 0 op/s 2024-09-17T07:27:42.985 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:27:42 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: [17/Sep/2024:07:27:42] ENGINE Bus STOPPING 2024-09-17T07:27:42.986 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:42 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mon-a[29088]: 2024-09-17T07:27:42.659+0000 7f59b8ff2640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T07:27:42.986 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:42 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mon-a[29088]: 2024-09-17T07:27:42.659+0000 7f59b8ff2640 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-09-17T07:27:42.986 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:42 smithi100 podman[220489]: 2024-09-17 07:27:42.690792921 +0000 UTC m=+0.131974217 container died a3d283eb23fd0c82f28bb7d58bbf50082bad946543cec84a656a561fb69afffe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-mon-a, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, RELEASE=reef-19e2a2b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T07:27:43.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:42 smithi135 ceph-mon[33238]: 6.1 deep-scrub starts 2024-09-17T07:27:43.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:42 smithi135 ceph-mon[33238]: 6.1 deep-scrub ok 2024-09-17T07:27:43.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:42 smithi135 ceph-mon[33238]: pgmap v1221: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 725 B/s rd, 0 op/s 2024-09-17T07:27:43.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:43 smithi100 podman[220489]: 2024-09-17 07:27:43.19376726 +0000 UTC m=+0.634948558 container cleanup a3d283eb23fd0c82f28bb7d58bbf50082bad946543cec84a656a561fb69afffe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-mon-a, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, RELEASE=reef-19e2a2b, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS) 2024-09-17T07:27:43.376 INFO:journalctl@ceph.mon.a.smithi100.stdout:Sep 17 07:27:43 smithi100 bash[220489]: ceph-e848267c-74c1-11ef-bceb-c7b262605968-mon-a 2024-09-17T07:27:43.377 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:27:43 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: [17/Sep/2024:07:27:43] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-17T07:27:43.377 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:27:43 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: [17/Sep/2024:07:27:43] ENGINE Bus STOPPED 2024-09-17T07:27:43.377 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:27:43 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: [17/Sep/2024:07:27:43] ENGINE Bus STARTING 2024-09-17T07:27:43.377 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:27:43 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: [17/Sep/2024:07:27:43] ENGINE Serving on http://:::9283 2024-09-17T07:27:43.377 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:27:43 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: [17/Sep/2024:07:27:43] ENGINE Bus STARTED 2024-09-17T07:27:43.616 DEBUG:teuthology.orchestra.run.smithi100:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e848267c-74c1-11ef-bceb-c7b262605968@mon.a.service' 2024-09-17T07:27:43.655 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T07:27:43.656 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-09-17T07:27:43.656 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-09-17T07:27:43.656 DEBUG:teuthology.orchestra.run.smithi100:> sudo systemctl stop ceph-e848267c-74c1-11ef-bceb-c7b262605968@mon.c 2024-09-17T07:27:43.982 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:43 smithi100 systemd[1]: Stopping Ceph mon.c for e848267c-74c1-11ef-bceb-c7b262605968... 2024-09-17T07:27:44.358 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:43 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mon-c[39458]: 2024-09-17T07:27:43.981+0000 7f7546559640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T07:27:44.359 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:43 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mon-c[39458]: 2024-09-17T07:27:43.981+0000 7f7546559640 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-09-17T07:27:44.359 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:44 smithi100 podman[220621]: 2024-09-17 07:27:44.074356691 +0000 UTC m=+0.193581497 container died 51721fe74c4741aa99d9aacca90e0ab2f6488a414acc5075cf6826c0ca19098a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-mon-c, RELEASE=reef-19e2a2b, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-09-17T07:27:44.359 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:27:43 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: [17/Sep/2024:07:27:43] ENGINE Bus STOPPING 2024-09-17T07:27:44.359 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:27:44 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: [17/Sep/2024:07:27:44] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-17T07:27:44.359 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:27:44 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: [17/Sep/2024:07:27:44] ENGINE Bus STOPPED 2024-09-17T07:27:44.359 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:27:44 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: [17/Sep/2024:07:27:44] ENGINE Bus STARTING 2024-09-17T07:27:44.359 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:27:44 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: [17/Sep/2024:07:27:44] ENGINE Serving on http://:::9283 2024-09-17T07:27:44.359 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:27:44 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: [17/Sep/2024:07:27:44] ENGINE Bus STARTED 2024-09-17T07:27:44.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:44 smithi100 podman[220621]: 2024-09-17 07:27:44.466199918 +0000 UTC m=+0.585424721 container cleanup 51721fe74c4741aa99d9aacca90e0ab2f6488a414acc5075cf6826c0ca19098a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-mon-c, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-19e2a2b, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=) 2024-09-17T07:27:44.626 INFO:journalctl@ceph.mon.c.smithi100.stdout:Sep 17 07:27:44 smithi100 bash[220621]: ceph-e848267c-74c1-11ef-bceb-c7b262605968-mon-c 2024-09-17T07:27:44.890 DEBUG:teuthology.orchestra.run.smithi100:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e848267c-74c1-11ef-bceb-c7b262605968@mon.c.service' 2024-09-17T07:27:44.945 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T07:27:44.946 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-09-17T07:27:44.946 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-09-17T07:27:44.946 DEBUG:teuthology.orchestra.run.smithi135:> sudo systemctl stop ceph-e848267c-74c1-11ef-bceb-c7b262605968@mon.b 2024-09-17T07:27:45.263 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:45 smithi135 systemd[1]: Stopping Ceph mon.b for e848267c-74c1-11ef-bceb-c7b262605968... 2024-09-17T07:27:45.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:45 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mon-b[33234]: 2024-09-17T07:27:45.261+0000 7f85fecc8640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T07:27:45.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:45 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mon-b[33234]: 2024-09-17T07:27:45.261+0000 7f85fecc8640 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-09-17T07:27:45.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:45 smithi135 podman[164524]: 2024-09-17 07:27:45.415466035 +0000 UTC m=+0.251023379 container died 17cdaec5e3b6b85aaba50d707f82032b437ce2bb79da54191dad2aa01c2f4b32 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-mon-b, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, ceph=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-19e2a2b, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T07:27:46.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:45 smithi135 podman[164524]: 2024-09-17 07:27:45.888523498 +0000 UTC m=+0.724080875 container cleanup 17cdaec5e3b6b85aaba50d707f82032b437ce2bb79da54191dad2aa01c2f4b32 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-mon-b, RELEASE=reef-19e2a2b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-09-17T07:27:46.043 INFO:journalctl@ceph.mon.b.smithi135.stdout:Sep 17 07:27:45 smithi135 bash[164524]: ceph-e848267c-74c1-11ef-bceb-c7b262605968-mon-b 2024-09-17T07:27:46.296 DEBUG:teuthology.orchestra.run.smithi135:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e848267c-74c1-11ef-bceb-c7b262605968@mon.b.service' 2024-09-17T07:27:46.334 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T07:27:46.335 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-09-17T07:27:46.335 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-09-17T07:27:46.335 DEBUG:teuthology.orchestra.run.smithi100:> sudo systemctl stop ceph-e848267c-74c1-11ef-bceb-c7b262605968@mgr.y 2024-09-17T07:27:46.362 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:27:45 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: ::ffff:172.21.15.135 - - [17/Sep/2024:07:27:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-17T07:27:46.626 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:27:46 smithi100 systemd[1]: Stopping Ceph mgr.y for e848267c-74c1-11ef-bceb-c7b262605968... 2024-09-17T07:27:46.930 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:27:46 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T07:27:46.628+0000 7f8beff60640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T07:27:46.930 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:27:46 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y[29318]: 2024-09-17T07:27:46.628+0000 7f8beff60640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-09-17T07:27:46.930 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:27:46 smithi100 podman[220756]: 2024-09-17 07:27:46.660322864 +0000 UTC m=+0.134702488 container died 31e6cec7f4a95628807709cbc3f407552dd8f298223f106a056cfe8b265cd5d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y, RELEASE=reef-19e2a2b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-09-17T07:27:47.515 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:27:47 smithi100 podman[220756]: 2024-09-17 07:27:47.213400392 +0000 UTC m=+0.687780014 container cleanup 31e6cec7f4a95628807709cbc3f407552dd8f298223f106a056cfe8b265cd5d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=reef-19e2a2b, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731) 2024-09-17T07:27:47.515 INFO:journalctl@ceph.mgr.y.smithi100.stdout:Sep 17 07:27:47 smithi100 bash[220756]: ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-y 2024-09-17T07:27:47.649 DEBUG:teuthology.orchestra.run.smithi100:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e848267c-74c1-11ef-bceb-c7b262605968@mgr.y.service' 2024-09-17T07:27:47.725 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T07:27:47.725 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-09-17T07:27:47.725 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-09-17T07:27:47.725 DEBUG:teuthology.orchestra.run.smithi135:> sudo systemctl stop ceph-e848267c-74c1-11ef-bceb-c7b262605968@mgr.x 2024-09-17T07:27:48.043 INFO:journalctl@ceph.mgr.x.smithi135.stdout:Sep 17 07:27:47 smithi135 systemd[1]: Stopping Ceph mgr.x for e848267c-74c1-11ef-bceb-c7b262605968... 2024-09-17T07:27:48.371 INFO:journalctl@ceph.mgr.x.smithi135.stdout:Sep 17 07:27:48 smithi135 podman[164646]: 2024-09-17 07:27:48.075000175 +0000 UTC m=+0.155769540 container died b040363098ac3e47322bc18c6c3d9dc3dd1f0452a9a0e317316268ace0a92fbd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-x, GIT_CLEAN=True, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, RELEASE=reef-19e2a2b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-09-17T07:27:48.733 INFO:journalctl@ceph.mgr.x.smithi135.stdout:Sep 17 07:27:48 smithi135 podman[164646]: 2024-09-17 07:27:48.469676813 +0000 UTC m=+0.550446167 container cleanup b040363098ac3e47322bc18c6c3d9dc3dd1f0452a9a0e317316268ace0a92fbd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-x, RELEASE=reef-19e2a2b, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True) 2024-09-17T07:27:48.733 INFO:journalctl@ceph.mgr.x.smithi135.stdout:Sep 17 07:27:48 smithi135 bash[164646]: ceph-e848267c-74c1-11ef-bceb-c7b262605968-mgr-x 2024-09-17T07:27:48.852 DEBUG:teuthology.orchestra.run.smithi135:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e848267c-74c1-11ef-bceb-c7b262605968@mgr.x.service' 2024-09-17T07:27:48.888 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T07:27:48.888 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-09-17T07:27:48.888 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-09-17T07:27:48.888 DEBUG:teuthology.orchestra.run.smithi100:> sudo systemctl stop ceph-e848267c-74c1-11ef-bceb-c7b262605968@osd.0 2024-09-17T07:27:49.185 INFO:journalctl@ceph.osd.0.smithi100.stdout:Sep 17 07:27:48 smithi100 systemd[1]: Stopping Ceph osd.0 for e848267c-74c1-11ef-bceb-c7b262605968... 2024-09-17T07:27:49.445 INFO:journalctl@ceph.osd.0.smithi100.stdout:Sep 17 07:27:49 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-0[43366]: 2024-09-17T07:27:49.184+0000 7f16f9b05640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T07:27:49.446 INFO:journalctl@ceph.osd.0.smithi100.stdout:Sep 17 07:27:49 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-0[43366]: 2024-09-17T07:27:49.184+0000 7f16f9b05640 -1 osd.0 549 *** Got signal Terminated *** 2024-09-17T07:27:49.446 INFO:journalctl@ceph.osd.0.smithi100.stdout:Sep 17 07:27:49 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-0[43366]: 2024-09-17T07:27:49.184+0000 7f16f9b05640 -1 osd.0 549 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T07:27:55.011 INFO:journalctl@ceph.osd.0.smithi100.stdout:Sep 17 07:27:54 smithi100 podman[220878]: 2024-09-17 07:27:54.740036886 +0000 UTC m=+5.665943177 container died 106f6183d3ff4f4f5feb24aeb93144763a3c0a0498844f51d8aa654241073495 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, RELEASE=reef-19e2a2b, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T07:27:55.376 INFO:journalctl@ceph.osd.0.smithi100.stdout:Sep 17 07:27:55 smithi100 podman[220878]: 2024-09-17 07:27:55.227303271 +0000 UTC m=+6.153209559 container cleanup 106f6183d3ff4f4f5feb24aeb93144763a3c0a0498844f51d8aa654241073495 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-0, RELEASE=reef-19e2a2b, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T07:27:55.376 INFO:journalctl@ceph.osd.0.smithi100.stdout:Sep 17 07:27:55 smithi100 bash[220878]: ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-0 2024-09-17T07:27:55.848 INFO:journalctl@ceph.osd.0.smithi100.stdout:Sep 17 07:27:55 smithi100 podman[220905]: 2024-09-17 07:27:55.540212163 +0000 UTC m=+0.796865461 container remove 106f6183d3ff4f4f5feb24aeb93144763a3c0a0498844f51d8aa654241073495 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-19e2a2b, org.label-schema.license=GPLv2) 2024-09-17T07:27:56.107 INFO:journalctl@ceph.osd.0.smithi100.stdout:Sep 17 07:27:55 smithi100 podman[220994]: 2024-09-17 07:27:55.751474692 +0000 UTC m=+0.016167525 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9 2024-09-17T07:27:56.108 INFO:journalctl@ceph.osd.0.smithi100.stdout:Sep 17 07:27:55 smithi100 podman[220994]: 2024-09-17 07:27:55.888762323 +0000 UTC m=+0.153455148 container create 346a4a3b176d493868a9adb3dce7f621c4f2da99c9a99f9081b8b155de25f7ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-0-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-19e2a2b, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731) 2024-09-17T07:27:56.108 INFO:journalctl@ceph.osd.0.smithi100.stdout:Sep 17 07:27:56 smithi100 podman[220994]: 2024-09-17 07:27:56.107899564 +0000 UTC m=+0.372592392 container init 346a4a3b176d493868a9adb3dce7f621c4f2da99c9a99f9081b8b155de25f7ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-0-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=reef-19e2a2b, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, io.buildah.version=1.37.2) 2024-09-17T07:27:56.376 INFO:journalctl@ceph.osd.0.smithi100.stdout:Sep 17 07:27:56 smithi100 podman[220994]: 2024-09-17 07:27:56.111338958 +0000 UTC m=+0.376031796 container start 346a4a3b176d493868a9adb3dce7f621c4f2da99c9a99f9081b8b155de25f7ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-0-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=reef-19e2a2b, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD) 2024-09-17T07:27:56.377 INFO:journalctl@ceph.osd.0.smithi100.stdout:Sep 17 07:27:56 smithi100 podman[220994]: 2024-09-17 07:27:56.224627283 +0000 UTC m=+0.489320112 container attach 346a4a3b176d493868a9adb3dce7f621c4f2da99c9a99f9081b8b155de25f7ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-0-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-19e2a2b, GIT_CLEAN=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-09-17T07:27:56.685 INFO:journalctl@ceph.osd.0.smithi100.stdout:Sep 17 07:27:56 smithi100 podman[220994]: 2024-09-17 07:27:56.391114013 +0000 UTC m=+0.655806843 container died 346a4a3b176d493868a9adb3dce7f621c4f2da99c9a99f9081b8b155de25f7ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-0-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, RELEASE=reef-19e2a2b, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux ) 2024-09-17T07:27:57.189 DEBUG:teuthology.orchestra.run.smithi100:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e848267c-74c1-11ef-bceb-c7b262605968@osd.0.service' 2024-09-17T07:27:57.231 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T07:27:57.232 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-09-17T07:27:57.232 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-09-17T07:27:57.232 DEBUG:teuthology.orchestra.run.smithi100:> sudo systemctl stop ceph-e848267c-74c1-11ef-bceb-c7b262605968@osd.1 2024-09-17T07:27:57.590 INFO:journalctl@ceph.osd.1.smithi100.stdout:Sep 17 07:27:57 smithi100 systemd[1]: Stopping Ceph osd.1 for e848267c-74c1-11ef-bceb-c7b262605968... 2024-09-17T07:27:57.876 INFO:journalctl@ceph.osd.1.smithi100.stdout:Sep 17 07:27:57 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-1[186219]: 2024-09-17T07:27:57.589+0000 7fed81f8d640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T07:27:57.876 INFO:journalctl@ceph.osd.1.smithi100.stdout:Sep 17 07:27:57 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-1[186219]: 2024-09-17T07:27:57.589+0000 7fed81f8d640 -1 osd.1 549 *** Got signal Terminated *** 2024-09-17T07:27:57.877 INFO:journalctl@ceph.osd.1.smithi100.stdout:Sep 17 07:27:57 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-1[186219]: 2024-09-17T07:27:57.589+0000 7fed81f8d640 -1 osd.1 549 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T07:28:03.043 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:28:02 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a[64841]: ts=2024-09-17T07:28:02.619Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.100:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.100:8765: connect: connection refused" 2024-09-17T07:28:03.044 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:28:02 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a[64841]: ts=2024-09-17T07:28:02.619Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.100:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.100:8765: connect: connection refused" 2024-09-17T07:28:03.044 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:28:02 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a[64841]: ts=2024-09-17T07:28:02.619Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.100:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.100:8765: connect: connection refused" 2024-09-17T07:28:03.044 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:28:02 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a[64841]: ts=2024-09-17T07:28:02.620Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.100:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.100:8765: connect: connection refused" 2024-09-17T07:28:03.044 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:28:02 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a[64841]: ts=2024-09-17T07:28:02.620Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.100:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.100:8765: connect: connection refused" 2024-09-17T07:28:03.357 INFO:journalctl@ceph.osd.1.smithi100.stdout:Sep 17 07:28:03 smithi100 podman[221099]: 2024-09-17 07:28:03.070273066 +0000 UTC m=+5.605036246 container died 715bd0bb4a3185ea6b3923aa175c569ebab6468193f6ec7934b73205266e7e5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-1, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, RELEASE=reef-19e2a2b, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T07:28:03.626 INFO:journalctl@ceph.osd.1.smithi100.stdout:Sep 17 07:28:03 smithi100 podman[221099]: 2024-09-17 07:28:03.539891454 +0000 UTC m=+6.074654628 container cleanup 715bd0bb4a3185ea6b3923aa175c569ebab6468193f6ec7934b73205266e7e5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-1, RELEASE=reef-19e2a2b, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-09-17T07:28:03.626 INFO:journalctl@ceph.osd.1.smithi100.stdout:Sep 17 07:28:03 smithi100 bash[221099]: ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-1 2024-09-17T07:28:04.297 INFO:journalctl@ceph.osd.1.smithi100.stdout:Sep 17 07:28:03 smithi100 podman[221116]: 2024-09-17 07:28:03.891787167 +0000 UTC m=+0.822487757 container remove 715bd0bb4a3185ea6b3923aa175c569ebab6468193f6ec7934b73205266e7e5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-1, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-19e2a2b) 2024-09-17T07:28:04.548 INFO:journalctl@ceph.osd.1.smithi100.stdout:Sep 17 07:28:04 smithi100 podman[221201]: 2024-09-17 07:28:04.199602764 +0000 UTC m=+0.015577874 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9 2024-09-17T07:28:04.548 INFO:journalctl@ceph.osd.1.smithi100.stdout:Sep 17 07:28:04 smithi100 podman[221201]: 2024-09-17 07:28:04.309383746 +0000 UTC m=+0.125358856 container create b021f6bf1b6310e9842ed7a978905d83d0f0072a49d5ff36969f1f9c3300f67c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-1-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, RELEASE=reef-19e2a2b, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731) 2024-09-17T07:28:04.548 INFO:journalctl@ceph.osd.1.smithi100.stdout:Sep 17 07:28:04 smithi100 podman[221201]: 2024-09-17 07:28:04.545303606 +0000 UTC m=+0.361278716 container init b021f6bf1b6310e9842ed7a978905d83d0f0072a49d5ff36969f1f9c3300f67c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-1-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, RELEASE=reef-19e2a2b, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_CLEAN=True) 2024-09-17T07:28:04.549 INFO:journalctl@ceph.osd.1.smithi100.stdout:Sep 17 07:28:04 smithi100 podman[221201]: 2024-09-17 07:28:04.548557945 +0000 UTC m=+0.364533055 container start b021f6bf1b6310e9842ed7a978905d83d0f0072a49d5ff36969f1f9c3300f67c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=reef-19e2a2b, org.label-schema.schema-version=1.0) 2024-09-17T07:28:04.876 INFO:journalctl@ceph.osd.1.smithi100.stdout:Sep 17 07:28:04 smithi100 podman[221201]: 2024-09-17 07:28:04.670626429 +0000 UTC m=+0.486601550 container attach b021f6bf1b6310e9842ed7a978905d83d0f0072a49d5ff36969f1f9c3300f67c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-1-deactivate, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-19e2a2b, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS) 2024-09-17T07:28:04.877 INFO:journalctl@ceph.osd.1.smithi100.stdout:Sep 17 07:28:04 smithi100 podman[221201]: 2024-09-17 07:28:04.732415396 +0000 UTC m=+0.548390515 container died b021f6bf1b6310e9842ed7a978905d83d0f0072a49d5ff36969f1f9c3300f67c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-1-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , RELEASE=reef-19e2a2b, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T07:28:05.351 DEBUG:teuthology.orchestra.run.smithi100:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e848267c-74c1-11ef-bceb-c7b262605968@osd.1.service' 2024-09-17T07:28:05.392 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T07:28:05.392 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-09-17T07:28:05.392 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-09-17T07:28:05.393 DEBUG:teuthology.orchestra.run.smithi100:> sudo systemctl stop ceph-e848267c-74c1-11ef-bceb-c7b262605968@osd.2 2024-09-17T07:28:05.609 INFO:journalctl@ceph.osd.2.smithi100.stdout:Sep 17 07:28:05 smithi100 systemd[1]: Stopping Ceph osd.2 for e848267c-74c1-11ef-bceb-c7b262605968... 2024-09-17T07:28:05.876 INFO:journalctl@ceph.osd.2.smithi100.stdout:Sep 17 07:28:05 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-2[54495]: 2024-09-17T07:28:05.734+0000 7fafc9afc640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T07:28:05.876 INFO:journalctl@ceph.osd.2.smithi100.stdout:Sep 17 07:28:05 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-2[54495]: 2024-09-17T07:28:05.734+0000 7fafc9afc640 -1 osd.2 549 *** Got signal Terminated *** 2024-09-17T07:28:05.876 INFO:journalctl@ceph.osd.2.smithi100.stdout:Sep 17 07:28:05 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-2[54495]: 2024-09-17T07:28:05.734+0000 7fafc9afc640 -1 osd.2 549 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T07:28:11.626 INFO:journalctl@ceph.osd.2.smithi100.stdout:Sep 17 07:28:11 smithi100 podman[221303]: 2024-09-17 07:28:11.373320858 +0000 UTC m=+5.749537646 container died d1b97f9fdb94de7f584301be91499cfec94222e19c79e19ad5d3309be39bb984 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, RELEASE=reef-19e2a2b, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731) 2024-09-17T07:28:12.126 INFO:journalctl@ceph.osd.2.smithi100.stdout:Sep 17 07:28:11 smithi100 podman[221303]: 2024-09-17 07:28:11.87779103 +0000 UTC m=+6.254007817 container cleanup d1b97f9fdb94de7f584301be91499cfec94222e19c79e19ad5d3309be39bb984 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=reef-19e2a2b, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T07:28:12.127 INFO:journalctl@ceph.osd.2.smithi100.stdout:Sep 17 07:28:11 smithi100 bash[221303]: ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-2 2024-09-17T07:28:12.511 INFO:journalctl@ceph.osd.2.smithi100.stdout:Sep 17 07:28:12 smithi100 podman[221329]: 2024-09-17 07:28:12.182521381 +0000 UTC m=+0.796236645 container remove d1b97f9fdb94de7f584301be91499cfec94222e19c79e19ad5d3309be39bb984 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-2, ceph=True, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=reef-19e2a2b, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T07:28:12.849 INFO:journalctl@ceph.osd.2.smithi100.stdout:Sep 17 07:28:12 smithi100 podman[221412]: 2024-09-17 07:28:12.41423053 +0000 UTC m=+0.015794513 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9 2024-09-17T07:28:12.849 INFO:journalctl@ceph.osd.2.smithi100.stdout:Sep 17 07:28:12 smithi100 podman[221412]: 2024-09-17 07:28:12.52194825 +0000 UTC m=+0.123512283 container create 36a5eb0774f63b3e2c06c0a8406a4fe9133bae2e7635ca572fa9452c5a4ba220 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-2-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, RELEASE=reef-19e2a2b) 2024-09-17T07:28:12.850 INFO:journalctl@ceph.osd.2.smithi100.stdout:Sep 17 07:28:12 smithi100 podman[221412]: 2024-09-17 07:28:12.749321637 +0000 UTC m=+0.350885609 container init 36a5eb0774f63b3e2c06c0a8406a4fe9133bae2e7635ca572fa9452c5a4ba220 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-2-deactivate, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, RELEASE=reef-19e2a2b, org.label-schema.vendor=CentOS) 2024-09-17T07:28:12.850 INFO:journalctl@ceph.osd.2.smithi100.stdout:Sep 17 07:28:12 smithi100 podman[221412]: 2024-09-17 07:28:12.752594133 +0000 UTC m=+0.354158108 container start 36a5eb0774f63b3e2c06c0a8406a4fe9133bae2e7635ca572fa9452c5a4ba220 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-2-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=reef-19e2a2b, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T07:28:13.127 INFO:journalctl@ceph.osd.2.smithi100.stdout:Sep 17 07:28:12 smithi100 podman[221412]: 2024-09-17 07:28:12.849331115 +0000 UTC m=+0.450895101 container attach 36a5eb0774f63b3e2c06c0a8406a4fe9133bae2e7635ca572fa9452c5a4ba220 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-2-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=reef-19e2a2b, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2) 2024-09-17T07:28:13.127 INFO:journalctl@ceph.osd.2.smithi100.stdout:Sep 17 07:28:12 smithi100 podman[221412]: 2024-09-17 07:28:12.934888897 +0000 UTC m=+0.536452874 container died 36a5eb0774f63b3e2c06c0a8406a4fe9133bae2e7635ca572fa9452c5a4ba220 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-2-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, RELEASE=reef-19e2a2b) 2024-09-17T07:28:13.682 DEBUG:teuthology.orchestra.run.smithi100:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e848267c-74c1-11ef-bceb-c7b262605968@osd.2.service' 2024-09-17T07:28:13.722 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T07:28:13.723 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-09-17T07:28:13.723 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-09-17T07:28:13.723 DEBUG:teuthology.orchestra.run.smithi100:> sudo systemctl stop ceph-e848267c-74c1-11ef-bceb-c7b262605968@osd.3 2024-09-17T07:28:14.040 INFO:journalctl@ceph.osd.3.smithi100.stdout:Sep 17 07:28:13 smithi100 systemd[1]: Stopping Ceph osd.3 for e848267c-74c1-11ef-bceb-c7b262605968... 2024-09-17T07:28:14.376 INFO:journalctl@ceph.osd.3.smithi100.stdout:Sep 17 07:28:14 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-3[60454]: 2024-09-17T07:28:14.040+0000 7f14aad6b640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T07:28:14.377 INFO:journalctl@ceph.osd.3.smithi100.stdout:Sep 17 07:28:14 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-3[60454]: 2024-09-17T07:28:14.040+0000 7f14aad6b640 -1 osd.3 549 *** Got signal Terminated *** 2024-09-17T07:28:14.377 INFO:journalctl@ceph.osd.3.smithi100.stdout:Sep 17 07:28:14 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-3[60454]: 2024-09-17T07:28:14.040+0000 7f14aad6b640 -1 osd.3 549 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T07:28:15.293 INFO:journalctl@ceph.osd.6.smithi135.stdout:Sep 17 07:28:15 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-6[49830]: 2024-09-17T07:28:15.011+0000 7fef838ff640 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:50.036343+0000 front 2024-09-17T07:27:50.036454+0000 (oldest deadline 2024-09-17T07:28:14.736485+0000) 2024-09-17T07:28:16.293 INFO:journalctl@ceph.osd.6.smithi135.stdout:Sep 17 07:28:16 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-6[49830]: 2024-09-17T07:28:16.016+0000 7fef838ff640 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:50.036343+0000 front 2024-09-17T07:27:50.036454+0000 (oldest deadline 2024-09-17T07:28:14.736485+0000) 2024-09-17T07:28:17.293 INFO:journalctl@ceph.osd.6.smithi135.stdout:Sep 17 07:28:17 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-6[49830]: 2024-09-17T07:28:17.019+0000 7fef838ff640 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:50.036343+0000 front 2024-09-17T07:27:50.036454+0000 (oldest deadline 2024-09-17T07:28:14.736485+0000) 2024-09-17T07:28:17.793 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:28:17 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-5[156646]: 2024-09-17T07:28:17.697+0000 7f8f6ef9c640 -1 osd.5 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:53.375825+0000 front 2024-09-17T07:27:53.375768+0000 (oldest deadline 2024-09-17T07:28:17.475723+0000) 2024-09-17T07:28:17.793 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:17 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:17.744+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:52.894330+0000 front 2024-09-17T07:27:52.894143+0000 (oldest deadline 2024-09-17T07:28:17.594117+0000) 2024-09-17T07:28:18.126 INFO:journalctl@ceph.osd.3.smithi100.stdout:Sep 17 07:28:17 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-3[60454]: 2024-09-17T07:28:17.842+0000 7f14a3377640 -1 osd.3 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:51.730080+0000 front 2024-09-17T07:27:51.730097+0000 (oldest deadline 2024-09-17T07:28:17.030223+0000) 2024-09-17T07:28:18.293 INFO:journalctl@ceph.osd.6.smithi135.stdout:Sep 17 07:28:17 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-6[49830]: 2024-09-17T07:28:17.997+0000 7fef838ff640 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:50.036343+0000 front 2024-09-17T07:27:50.036454+0000 (oldest deadline 2024-09-17T07:28:14.736485+0000) 2024-09-17T07:28:18.965 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:28:18 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-5[156646]: 2024-09-17T07:28:18.700+0000 7f8f6ef9c640 -1 osd.5 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:53.375825+0000 front 2024-09-17T07:27:53.375768+0000 (oldest deadline 2024-09-17T07:28:17.475723+0000) 2024-09-17T07:28:18.965 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:18 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:18.742+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:52.894330+0000 front 2024-09-17T07:27:52.894143+0000 (oldest deadline 2024-09-17T07:28:17.594117+0000) 2024-09-17T07:28:19.126 INFO:journalctl@ceph.osd.3.smithi100.stdout:Sep 17 07:28:18 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-3[60454]: 2024-09-17T07:28:18.800+0000 7f14a3377640 -1 osd.3 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:51.730080+0000 front 2024-09-17T07:27:51.730097+0000 (oldest deadline 2024-09-17T07:28:17.030223+0000) 2024-09-17T07:28:19.293 INFO:journalctl@ceph.osd.6.smithi135.stdout:Sep 17 07:28:18 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-6[49830]: 2024-09-17T07:28:18.964+0000 7fef838ff640 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:50.036343+0000 front 2024-09-17T07:27:50.036454+0000 (oldest deadline 2024-09-17T07:28:14.736485+0000) 2024-09-17T07:28:19.626 INFO:journalctl@ceph.osd.3.smithi100.stdout:Sep 17 07:28:19 smithi100 podman[221515]: 2024-09-17 07:28:19.522252734 +0000 UTC m=+5.579654316 container died d753d4db7d30b7c16a976d04f3604140d18ec641c5f1252b7b40c5dcfb3eb907 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-3, org.label-schema.schema-version=1.0, RELEASE=reef-19e2a2b, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731) 2024-09-17T07:28:20.001 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:28:19 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-5[156646]: 2024-09-17T07:28:19.704+0000 7f8f6ef9c640 -1 osd.5 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:53.375825+0000 front 2024-09-17T07:27:53.375768+0000 (oldest deadline 2024-09-17T07:28:17.475723+0000) 2024-09-17T07:28:20.001 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:19 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:19.754+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:52.894330+0000 front 2024-09-17T07:27:52.894143+0000 (oldest deadline 2024-09-17T07:28:17.594117+0000) 2024-09-17T07:28:20.126 INFO:journalctl@ceph.osd.3.smithi100.stdout:Sep 17 07:28:19 smithi100 podman[221515]: 2024-09-17 07:28:19.981328507 +0000 UTC m=+6.038730080 container cleanup d753d4db7d30b7c16a976d04f3604140d18ec641c5f1252b7b40c5dcfb3eb907 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-3, GIT_BRANCH=HEAD, RELEASE=reef-19e2a2b, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-09-17T07:28:20.126 INFO:journalctl@ceph.osd.3.smithi100.stdout:Sep 17 07:28:19 smithi100 bash[221515]: ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-3 2024-09-17T07:28:20.293 INFO:journalctl@ceph.osd.6.smithi135.stdout:Sep 17 07:28:20 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-6[49830]: 2024-09-17T07:28:19.999+0000 7fef838ff640 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:50.036343+0000 front 2024-09-17T07:27:50.036454+0000 (oldest deadline 2024-09-17T07:28:14.736485+0000) 2024-09-17T07:28:20.626 INFO:journalctl@ceph.osd.3.smithi100.stdout:Sep 17 07:28:20 smithi100 podman[221542]: 2024-09-17 07:28:20.30831865 +0000 UTC m=+0.778905726 container remove d753d4db7d30b7c16a976d04f3604140d18ec641c5f1252b7b40c5dcfb3eb907 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-3, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-19e2a2b, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-09-17T07:28:20.903 INFO:journalctl@ceph.osd.3.smithi100.stdout:Sep 17 07:28:20 smithi100 podman[221635]: 2024-09-17 07:28:20.554787141 +0000 UTC m=+0.016202027 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9 2024-09-17T07:28:20.903 INFO:journalctl@ceph.osd.3.smithi100.stdout:Sep 17 07:28:20 smithi100 podman[221635]: 2024-09-17 07:28:20.692419652 +0000 UTC m=+0.153834527 container create c12a93a06fcd217d71ca80f2eeda52a050272742c87d5310ac956acdfb42c464 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True, RELEASE=reef-19e2a2b, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T07:28:20.903 INFO:journalctl@ceph.osd.3.smithi100.stdout:Sep 17 07:28:20 smithi100 podman[221635]: 2024-09-17 07:28:20.903007291 +0000 UTC m=+0.364422175 container init c12a93a06fcd217d71ca80f2eeda52a050272742c87d5310ac956acdfb42c464 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, ceph=True, RELEASE=reef-19e2a2b, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2) 2024-09-17T07:28:20.958 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:28:20 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-5[156646]: 2024-09-17T07:28:20.740+0000 7f8f6ef9c640 -1 osd.5 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:53.375825+0000 front 2024-09-17T07:27:53.375768+0000 (oldest deadline 2024-09-17T07:28:17.475723+0000) 2024-09-17T07:28:20.959 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:20 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:20.777+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:52.894330+0000 front 2024-09-17T07:27:52.894143+0000 (oldest deadline 2024-09-17T07:28:17.594117+0000) 2024-09-17T07:28:20.959 INFO:journalctl@ceph.osd.4.smithi135.stdout:Sep 17 07:28:20 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-4[38050]: 2024-09-17T07:28:20.561+0000 7fe66de34640 -1 osd.4 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:52.169689+0000 front 2024-09-17T07:27:52.169719+0000 (oldest deadline 2024-09-17T07:28:20.370058+0000) 2024-09-17T07:28:20.959 INFO:journalctl@ceph.osd.6.smithi135.stdout:Sep 17 07:28:20 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-6[49830]: 2024-09-17T07:28:20.957+0000 7fef838ff640 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:50.036343+0000 front 2024-09-17T07:27:50.036454+0000 (oldest deadline 2024-09-17T07:28:14.736485+0000) 2024-09-17T07:28:21.377 INFO:journalctl@ceph.osd.3.smithi100.stdout:Sep 17 07:28:20 smithi100 podman[221635]: 2024-09-17 07:28:20.906317712 +0000 UTC m=+0.367732588 container start c12a93a06fcd217d71ca80f2eeda52a050272742c87d5310ac956acdfb42c464 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-19e2a2b, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True) 2024-09-17T07:28:21.377 INFO:journalctl@ceph.osd.3.smithi100.stdout:Sep 17 07:28:20 smithi100 podman[221635]: 2024-09-17 07:28:20.969805373 +0000 UTC m=+0.431220249 container attach c12a93a06fcd217d71ca80f2eeda52a050272742c87d5310ac956acdfb42c464 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-19e2a2b, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-09-17T07:28:21.377 INFO:journalctl@ceph.osd.3.smithi100.stdout:Sep 17 07:28:21 smithi100 podman[221635]: 2024-09-17 07:28:21.096299489 +0000 UTC m=+0.557714365 container died c12a93a06fcd217d71ca80f2eeda52a050272742c87d5310ac956acdfb42c464 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-3-deactivate, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=reef-19e2a2b, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True) 2024-09-17T07:28:21.857 DEBUG:teuthology.orchestra.run.smithi100:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e848267c-74c1-11ef-bceb-c7b262605968@osd.3.service' 2024-09-17T07:28:21.898 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T07:28:21.899 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-09-17T07:28:21.899 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-09-17T07:28:21.899 DEBUG:teuthology.orchestra.run.smithi135:> sudo systemctl stop ceph-e848267c-74c1-11ef-bceb-c7b262605968@osd.4 2024-09-17T07:28:21.927 INFO:journalctl@ceph.osd.4.smithi135.stdout:Sep 17 07:28:21 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-4[38050]: 2024-09-17T07:28:21.584+0000 7fe66de34640 -1 osd.4 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:52.169689+0000 front 2024-09-17T07:27:52.169719+0000 (oldest deadline 2024-09-17T07:28:20.370058+0000) 2024-09-17T07:28:21.928 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:28:21 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-5[156646]: 2024-09-17T07:28:21.702+0000 7f8f6ef9c640 -1 osd.5 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:53.375825+0000 front 2024-09-17T07:27:53.375768+0000 (oldest deadline 2024-09-17T07:28:17.475723+0000) 2024-09-17T07:28:21.928 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:21 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:21.744+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:52.894330+0000 front 2024-09-17T07:27:52.894143+0000 (oldest deadline 2024-09-17T07:28:17.594117+0000) 2024-09-17T07:28:22.199 INFO:journalctl@ceph.osd.4.smithi135.stdout:Sep 17 07:28:21 smithi135 systemd[1]: Stopping Ceph osd.4 for e848267c-74c1-11ef-bceb-c7b262605968... 2024-09-17T07:28:22.199 INFO:journalctl@ceph.osd.6.smithi135.stdout:Sep 17 07:28:21 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-6[49830]: 2024-09-17T07:28:21.935+0000 7fef838ff640 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:50.036343+0000 front 2024-09-17T07:27:50.036454+0000 (oldest deadline 2024-09-17T07:28:14.736485+0000) 2024-09-17T07:28:22.543 INFO:journalctl@ceph.osd.4.smithi135.stdout:Sep 17 07:28:22 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-4[38050]: 2024-09-17T07:28:22.197+0000 7fe675828640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T07:28:22.543 INFO:journalctl@ceph.osd.4.smithi135.stdout:Sep 17 07:28:22 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-4[38050]: 2024-09-17T07:28:22.197+0000 7fe675828640 -1 osd.4 549 *** Got signal Terminated *** 2024-09-17T07:28:22.543 INFO:journalctl@ceph.osd.4.smithi135.stdout:Sep 17 07:28:22 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-4[38050]: 2024-09-17T07:28:22.197+0000 7fe675828640 -1 osd.4 549 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T07:28:22.959 INFO:journalctl@ceph.osd.4.smithi135.stdout:Sep 17 07:28:22 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-4[38050]: 2024-09-17T07:28:22.630+0000 7fe66de34640 -1 osd.4 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:52.169689+0000 front 2024-09-17T07:27:52.169719+0000 (oldest deadline 2024-09-17T07:28:20.370058+0000) 2024-09-17T07:28:22.960 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:22 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:22.787+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:52.894330+0000 front 2024-09-17T07:27:52.894143+0000 (oldest deadline 2024-09-17T07:28:17.594117+0000) 2024-09-17T07:28:22.960 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:28:22 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-5[156646]: 2024-09-17T07:28:22.718+0000 7f8f6ef9c640 -1 osd.5 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:53.375825+0000 front 2024-09-17T07:27:53.375768+0000 (oldest deadline 2024-09-17T07:28:17.475723+0000) 2024-09-17T07:28:23.293 INFO:journalctl@ceph.osd.6.smithi135.stdout:Sep 17 07:28:22 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-6[49830]: 2024-09-17T07:28:22.958+0000 7fef838ff640 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:50.036343+0000 front 2024-09-17T07:27:50.036454+0000 (oldest deadline 2024-09-17T07:28:14.736485+0000) 2024-09-17T07:28:23.954 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:28:23 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-5[156646]: 2024-09-17T07:28:23.702+0000 7f8f6ef9c640 -1 osd.5 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:53.375825+0000 front 2024-09-17T07:27:53.375768+0000 (oldest deadline 2024-09-17T07:28:17.475723+0000) 2024-09-17T07:28:23.954 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:23 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:23.781+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:52.894330+0000 front 2024-09-17T07:27:52.894143+0000 (oldest deadline 2024-09-17T07:28:17.594117+0000) 2024-09-17T07:28:23.955 INFO:journalctl@ceph.osd.4.smithi135.stdout:Sep 17 07:28:23 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-4[38050]: 2024-09-17T07:28:23.582+0000 7fe66de34640 -1 osd.4 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:52.169689+0000 front 2024-09-17T07:27:52.169719+0000 (oldest deadline 2024-09-17T07:28:20.370058+0000) 2024-09-17T07:28:23.955 INFO:journalctl@ceph.osd.4.smithi135.stdout:Sep 17 07:28:23 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-4[38050]: 2024-09-17T07:28:23.582+0000 7fe66de34640 -1 osd.4 549 heartbeat_check: no reply from 172.21.15.100:6807 osd.1 since back 2024-09-17T07:28:02.071134+0000 front 2024-09-17T07:28:02.071185+0000 (oldest deadline 2024-09-17T07:28:23.171019+0000) 2024-09-17T07:28:24.293 INFO:journalctl@ceph.osd.6.smithi135.stdout:Sep 17 07:28:23 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-6[49830]: 2024-09-17T07:28:23.953+0000 7fef838ff640 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:50.036343+0000 front 2024-09-17T07:27:50.036454+0000 (oldest deadline 2024-09-17T07:28:14.736485+0000) 2024-09-17T07:28:24.943 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:28:24 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-5[156646]: 2024-09-17T07:28:24.749+0000 7f8f6ef9c640 -1 osd.5 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:53.375825+0000 front 2024-09-17T07:27:53.375768+0000 (oldest deadline 2024-09-17T07:28:17.475723+0000) 2024-09-17T07:28:24.944 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:24 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:24.782+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:52.894330+0000 front 2024-09-17T07:27:52.894143+0000 (oldest deadline 2024-09-17T07:28:17.594117+0000) 2024-09-17T07:28:24.944 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:24 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:24.782+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6807 osd.1 since back 2024-09-17T07:27:59.895096+0000 front 2024-09-17T07:27:59.895048+0000 (oldest deadline 2024-09-17T07:28:23.995052+0000) 2024-09-17T07:28:24.944 INFO:journalctl@ceph.osd.4.smithi135.stdout:Sep 17 07:28:24 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-4[38050]: 2024-09-17T07:28:24.627+0000 7fe66de34640 -1 osd.4 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:52.169689+0000 front 2024-09-17T07:27:52.169719+0000 (oldest deadline 2024-09-17T07:28:20.370058+0000) 2024-09-17T07:28:24.944 INFO:journalctl@ceph.osd.4.smithi135.stdout:Sep 17 07:28:24 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-4[38050]: 2024-09-17T07:28:24.627+0000 7fe66de34640 -1 osd.4 549 heartbeat_check: no reply from 172.21.15.100:6807 osd.1 since back 2024-09-17T07:28:02.071134+0000 front 2024-09-17T07:28:02.071185+0000 (oldest deadline 2024-09-17T07:28:23.171019+0000) 2024-09-17T07:28:25.293 INFO:journalctl@ceph.osd.6.smithi135.stdout:Sep 17 07:28:24 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-6[49830]: 2024-09-17T07:28:24.942+0000 7fef838ff640 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:50.036343+0000 front 2024-09-17T07:27:50.036454+0000 (oldest deadline 2024-09-17T07:28:14.736485+0000) 2024-09-17T07:28:25.988 INFO:journalctl@ceph.osd.4.smithi135.stdout:Sep 17 07:28:25 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-4[38050]: 2024-09-17T07:28:25.593+0000 7fe66de34640 -1 osd.4 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:52.169689+0000 front 2024-09-17T07:27:52.169719+0000 (oldest deadline 2024-09-17T07:28:20.370058+0000) 2024-09-17T07:28:26.065 INFO:journalctl@ceph.osd.4.smithi135.stdout:Sep 17 07:28:25 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-4[38050]: 2024-09-17T07:28:25.593+0000 7fe66de34640 -1 osd.4 549 heartbeat_check: no reply from 172.21.15.100:6807 osd.1 since back 2024-09-17T07:28:02.071134+0000 front 2024-09-17T07:28:02.071185+0000 (oldest deadline 2024-09-17T07:28:23.171019+0000) 2024-09-17T07:28:26.065 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:25 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:25.824+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:52.894330+0000 front 2024-09-17T07:27:52.894143+0000 (oldest deadline 2024-09-17T07:28:17.594117+0000) 2024-09-17T07:28:26.066 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:25 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:25.824+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6807 osd.1 since back 2024-09-17T07:27:59.895096+0000 front 2024-09-17T07:27:59.895048+0000 (oldest deadline 2024-09-17T07:28:23.995052+0000) 2024-09-17T07:28:26.066 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:28:25 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-5[156646]: 2024-09-17T07:28:25.748+0000 7f8f6ef9c640 -1 osd.5 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:53.375825+0000 front 2024-09-17T07:27:53.375768+0000 (oldest deadline 2024-09-17T07:28:17.475723+0000) 2024-09-17T07:28:26.066 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:28:25 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-5[156646]: 2024-09-17T07:28:25.748+0000 7f8f6ef9c640 -1 osd.5 549 heartbeat_check: no reply from 172.21.15.100:6807 osd.1 since back 2024-09-17T07:28:00.976690+0000 front 2024-09-17T07:28:00.976776+0000 (oldest deadline 2024-09-17T07:28:25.076534+0000) 2024-09-17T07:28:26.293 INFO:journalctl@ceph.osd.6.smithi135.stdout:Sep 17 07:28:25 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-6[49830]: 2024-09-17T07:28:25.985+0000 7fef838ff640 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:50.036343+0000 front 2024-09-17T07:27:50.036454+0000 (oldest deadline 2024-09-17T07:28:14.736485+0000) 2024-09-17T07:28:26.293 INFO:journalctl@ceph.osd.6.smithi135.stdout:Sep 17 07:28:25 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-6[49830]: 2024-09-17T07:28:25.985+0000 7fef838ff640 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.100:6807 osd.1 since back 2024-09-17T07:27:59.337832+0000 front 2024-09-17T07:27:59.337847+0000 (oldest deadline 2024-09-17T07:28:25.237603+0000) 2024-09-17T07:28:26.848 INFO:journalctl@ceph.osd.4.smithi135.stdout:Sep 17 07:28:26 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-4[38050]: 2024-09-17T07:28:26.551+0000 7fe66de34640 -1 osd.4 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:52.169689+0000 front 2024-09-17T07:27:52.169719+0000 (oldest deadline 2024-09-17T07:28:20.370058+0000) 2024-09-17T07:28:26.849 INFO:journalctl@ceph.osd.4.smithi135.stdout:Sep 17 07:28:26 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-4[38050]: 2024-09-17T07:28:26.551+0000 7fe66de34640 -1 osd.4 549 heartbeat_check: no reply from 172.21.15.100:6807 osd.1 since back 2024-09-17T07:28:02.071134+0000 front 2024-09-17T07:28:02.071185+0000 (oldest deadline 2024-09-17T07:28:23.171019+0000) 2024-09-17T07:28:26.849 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:28:26 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-5[156646]: 2024-09-17T07:28:26.792+0000 7f8f6ef9c640 -1 osd.5 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:53.375825+0000 front 2024-09-17T07:27:53.375768+0000 (oldest deadline 2024-09-17T07:28:17.475723+0000) 2024-09-17T07:28:26.849 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:28:26 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-5[156646]: 2024-09-17T07:28:26.792+0000 7f8f6ef9c640 -1 osd.5 549 heartbeat_check: no reply from 172.21.15.100:6807 osd.1 since back 2024-09-17T07:28:00.976690+0000 front 2024-09-17T07:28:00.976776+0000 (oldest deadline 2024-09-17T07:28:25.076534+0000) 2024-09-17T07:28:27.293 INFO:journalctl@ceph.osd.6.smithi135.stdout:Sep 17 07:28:26 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-6[49830]: 2024-09-17T07:28:26.941+0000 7fef838ff640 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:50.036343+0000 front 2024-09-17T07:27:50.036454+0000 (oldest deadline 2024-09-17T07:28:14.736485+0000) 2024-09-17T07:28:27.293 INFO:journalctl@ceph.osd.6.smithi135.stdout:Sep 17 07:28:26 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-6[49830]: 2024-09-17T07:28:26.941+0000 7fef838ff640 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.100:6807 osd.1 since back 2024-09-17T07:27:59.337832+0000 front 2024-09-17T07:27:59.337847+0000 (oldest deadline 2024-09-17T07:28:25.237603+0000) 2024-09-17T07:28:27.294 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:26 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:26.847+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:52.894330+0000 front 2024-09-17T07:27:52.894143+0000 (oldest deadline 2024-09-17T07:28:17.594117+0000) 2024-09-17T07:28:27.294 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:26 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:26.847+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6807 osd.1 since back 2024-09-17T07:27:59.895096+0000 front 2024-09-17T07:27:59.895048+0000 (oldest deadline 2024-09-17T07:28:23.995052+0000) 2024-09-17T07:28:27.874 INFO:journalctl@ceph.osd.4.smithi135.stdout:Sep 17 07:28:27 smithi135 podman[164782]: 2024-09-17 07:28:27.719156353 +0000 UTC m=+5.622606207 container died 3ed3885300d27c69abddbb87ed87b0620aff762b2321e26ab201cc7fe32e2763 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-4, GIT_CLEAN=True, RELEASE=reef-19e2a2b, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-09-17T07:28:27.875 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:28:27 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-5[156646]: 2024-09-17T07:28:27.761+0000 7f8f6ef9c640 -1 osd.5 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:53.375825+0000 front 2024-09-17T07:27:53.375768+0000 (oldest deadline 2024-09-17T07:28:17.475723+0000) 2024-09-17T07:28:27.875 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:28:27 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-5[156646]: 2024-09-17T07:28:27.761+0000 7f8f6ef9c640 -1 osd.5 549 heartbeat_check: no reply from 172.21.15.100:6807 osd.1 since back 2024-09-17T07:28:00.976690+0000 front 2024-09-17T07:28:00.976776+0000 (oldest deadline 2024-09-17T07:28:25.076534+0000) 2024-09-17T07:28:28.149 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:27 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:27.873+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:52.894330+0000 front 2024-09-17T07:27:52.894143+0000 (oldest deadline 2024-09-17T07:28:17.594117+0000) 2024-09-17T07:28:28.149 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:27 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:27.873+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6807 osd.1 since back 2024-09-17T07:27:59.895096+0000 front 2024-09-17T07:27:59.895048+0000 (oldest deadline 2024-09-17T07:28:23.995052+0000) 2024-09-17T07:28:28.149 INFO:journalctl@ceph.osd.6.smithi135.stdout:Sep 17 07:28:27 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-6[49830]: 2024-09-17T07:28:27.927+0000 7fef838ff640 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:50.036343+0000 front 2024-09-17T07:27:50.036454+0000 (oldest deadline 2024-09-17T07:28:14.736485+0000) 2024-09-17T07:28:28.149 INFO:journalctl@ceph.osd.6.smithi135.stdout:Sep 17 07:28:27 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-6[49830]: 2024-09-17T07:28:27.927+0000 7fef838ff640 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.100:6807 osd.1 since back 2024-09-17T07:27:59.337832+0000 front 2024-09-17T07:27:59.337847+0000 (oldest deadline 2024-09-17T07:28:25.237603+0000) 2024-09-17T07:28:28.543 INFO:journalctl@ceph.osd.4.smithi135.stdout:Sep 17 07:28:28 smithi135 podman[164782]: 2024-09-17 07:28:28.149545181 +0000 UTC m=+6.052995031 container cleanup 3ed3885300d27c69abddbb87ed87b0620aff762b2321e26ab201cc7fe32e2763 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-4, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=reef-19e2a2b, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T07:28:28.543 INFO:journalctl@ceph.osd.4.smithi135.stdout:Sep 17 07:28:28 smithi135 bash[164782]: ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-4 2024-09-17T07:28:28.543 INFO:journalctl@ceph.osd.4.smithi135.stdout:Sep 17 07:28:28 smithi135 podman[164799]: 2024-09-17 07:28:28.343727568 +0000 UTC m=+0.626096262 container remove 3ed3885300d27c69abddbb87ed87b0620aff762b2321e26ab201cc7fe32e2763 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-4, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=reef-19e2a2b, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2) 2024-09-17T07:28:28.997 INFO:journalctl@ceph.osd.6.smithi135.stdout:Sep 17 07:28:28 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-6[49830]: 2024-09-17T07:28:28.906+0000 7fef838ff640 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:50.036343+0000 front 2024-09-17T07:27:50.036454+0000 (oldest deadline 2024-09-17T07:28:14.736485+0000) 2024-09-17T07:28:28.997 INFO:journalctl@ceph.osd.6.smithi135.stdout:Sep 17 07:28:28 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-6[49830]: 2024-09-17T07:28:28.906+0000 7fef838ff640 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.100:6807 osd.1 since back 2024-09-17T07:27:59.337832+0000 front 2024-09-17T07:27:59.337847+0000 (oldest deadline 2024-09-17T07:28:25.237603+0000) 2024-09-17T07:28:28.998 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:28 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:28.847+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:52.894330+0000 front 2024-09-17T07:27:52.894143+0000 (oldest deadline 2024-09-17T07:28:17.594117+0000) 2024-09-17T07:28:28.998 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:28 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:28.847+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6807 osd.1 since back 2024-09-17T07:27:59.895096+0000 front 2024-09-17T07:27:59.895048+0000 (oldest deadline 2024-09-17T07:28:23.995052+0000) 2024-09-17T07:28:28.998 INFO:journalctl@ceph.osd.4.smithi135.stdout:Sep 17 07:28:28 smithi135 podman[164887]: 2024-09-17 07:28:28.664188412 +0000 UTC m=+0.016188526 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9 2024-09-17T07:28:28.998 INFO:journalctl@ceph.osd.4.smithi135.stdout:Sep 17 07:28:28 smithi135 podman[164887]: 2024-09-17 07:28:28.769933565 +0000 UTC m=+0.121933725 container create cec12208e78e52a78f03ebf3cc50daf00603f3eceafc8b1ff5cb9ee91d015531 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-4-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, RELEASE=reef-19e2a2b, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-09-17T07:28:28.998 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:28:28 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-5[156646]: 2024-09-17T07:28:28.718+0000 7f8f6ef9c640 -1 osd.5 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:53.375825+0000 front 2024-09-17T07:27:53.375768+0000 (oldest deadline 2024-09-17T07:28:17.475723+0000) 2024-09-17T07:28:28.999 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:28:28 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-5[156646]: 2024-09-17T07:28:28.718+0000 7f8f6ef9c640 -1 osd.5 549 heartbeat_check: no reply from 172.21.15.100:6807 osd.1 since back 2024-09-17T07:28:00.976690+0000 front 2024-09-17T07:28:00.976776+0000 (oldest deadline 2024-09-17T07:28:25.076534+0000) 2024-09-17T07:28:29.250 INFO:journalctl@ceph.osd.4.smithi135.stdout:Sep 17 07:28:28 smithi135 podman[164887]: 2024-09-17 07:28:28.997907459 +0000 UTC m=+0.349907580 container init cec12208e78e52a78f03ebf3cc50daf00603f3eceafc8b1ff5cb9ee91d015531 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-4-deactivate, RELEASE=reef-19e2a2b, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T07:28:29.250 INFO:journalctl@ceph.osd.4.smithi135.stdout:Sep 17 07:28:29 smithi135 podman[164887]: 2024-09-17 07:28:29.001295473 +0000 UTC m=+0.353295605 container start cec12208e78e52a78f03ebf3cc50daf00603f3eceafc8b1ff5cb9ee91d015531 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, RELEASE=reef-19e2a2b, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T07:28:29.250 INFO:journalctl@ceph.osd.4.smithi135.stdout:Sep 17 07:28:29 smithi135 podman[164887]: 2024-09-17 07:28:29.081255172 +0000 UTC m=+0.433255301 container attach cec12208e78e52a78f03ebf3cc50daf00603f3eceafc8b1ff5cb9ee91d015531 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-4-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, GIT_BRANCH=HEAD, RELEASE=reef-19e2a2b, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-09-17T07:28:29.543 INFO:journalctl@ceph.osd.4.smithi135.stdout:Sep 17 07:28:29 smithi135 podman[164887]: 2024-09-17 07:28:29.250531065 +0000 UTC m=+0.602531187 container died cec12208e78e52a78f03ebf3cc50daf00603f3eceafc8b1ff5cb9ee91d015531 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-4-deactivate, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=reef-19e2a2b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=) 2024-09-17T07:28:29.861 INFO:journalctl@ceph.osd.6.smithi135.stdout:Sep 17 07:28:29 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-6[49830]: 2024-09-17T07:28:29.859+0000 7fef838ff640 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:50.036343+0000 front 2024-09-17T07:27:50.036454+0000 (oldest deadline 2024-09-17T07:28:14.736485+0000) 2024-09-17T07:28:29.861 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:28:29 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-5[156646]: 2024-09-17T07:28:29.700+0000 7f8f6ef9c640 -1 osd.5 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:53.375825+0000 front 2024-09-17T07:27:53.375768+0000 (oldest deadline 2024-09-17T07:28:17.475723+0000) 2024-09-17T07:28:29.861 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:28:29 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-5[156646]: 2024-09-17T07:28:29.700+0000 7f8f6ef9c640 -1 osd.5 549 heartbeat_check: no reply from 172.21.15.100:6807 osd.1 since back 2024-09-17T07:28:00.976690+0000 front 2024-09-17T07:28:00.976776+0000 (oldest deadline 2024-09-17T07:28:25.076534+0000) 2024-09-17T07:28:29.910 DEBUG:teuthology.orchestra.run.smithi135:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e848267c-74c1-11ef-bceb-c7b262605968@osd.4.service' 2024-09-17T07:28:29.989 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T07:28:29.989 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-09-17T07:28:29.989 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-09-17T07:28:29.990 DEBUG:teuthology.orchestra.run.smithi135:> sudo systemctl stop ceph-e848267c-74c1-11ef-bceb-c7b262605968@osd.5 2024-09-17T07:28:30.293 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:28:30 smithi135 systemd[1]: Stopping Ceph osd.5 for e848267c-74c1-11ef-bceb-c7b262605968... 2024-09-17T07:28:30.293 INFO:journalctl@ceph.osd.6.smithi135.stdout:Sep 17 07:28:29 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-6[49830]: 2024-09-17T07:28:29.859+0000 7fef838ff640 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.100:6807 osd.1 since back 2024-09-17T07:27:59.337832+0000 front 2024-09-17T07:27:59.337847+0000 (oldest deadline 2024-09-17T07:28:25.237603+0000) 2024-09-17T07:28:30.293 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:29 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:29.897+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:52.894330+0000 front 2024-09-17T07:27:52.894143+0000 (oldest deadline 2024-09-17T07:28:17.594117+0000) 2024-09-17T07:28:30.293 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:29 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:29.897+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6807 osd.1 since back 2024-09-17T07:27:59.895096+0000 front 2024-09-17T07:27:59.895048+0000 (oldest deadline 2024-09-17T07:28:23.995052+0000) 2024-09-17T07:28:30.673 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:28:30 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-5[156646]: 2024-09-17T07:28:30.333+0000 7f8f72988640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T07:28:30.673 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:28:30 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-5[156646]: 2024-09-17T07:28:30.333+0000 7f8f72988640 -1 osd.5 549 *** Got signal Terminated *** 2024-09-17T07:28:30.674 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:28:30 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-5[156646]: 2024-09-17T07:28:30.333+0000 7f8f72988640 -1 osd.5 549 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T07:28:30.674 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:28:30 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-5[156646]: 2024-09-17T07:28:30.672+0000 7f8f6ef9c640 -1 osd.5 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:53.375825+0000 front 2024-09-17T07:27:53.375768+0000 (oldest deadline 2024-09-17T07:28:17.475723+0000) 2024-09-17T07:28:30.674 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:28:30 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-5[156646]: 2024-09-17T07:28:30.672+0000 7f8f6ef9c640 -1 osd.5 549 heartbeat_check: no reply from 172.21.15.100:6807 osd.1 since back 2024-09-17T07:28:00.976690+0000 front 2024-09-17T07:28:00.976776+0000 (oldest deadline 2024-09-17T07:28:25.076534+0000) 2024-09-17T07:28:31.043 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:30 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:30.850+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:52.894330+0000 front 2024-09-17T07:27:52.894143+0000 (oldest deadline 2024-09-17T07:28:17.594117+0000) 2024-09-17T07:28:31.043 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:30 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:30.850+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6807 osd.1 since back 2024-09-17T07:27:59.895096+0000 front 2024-09-17T07:27:59.895048+0000 (oldest deadline 2024-09-17T07:28:23.995052+0000) 2024-09-17T07:28:31.043 INFO:journalctl@ceph.osd.6.smithi135.stdout:Sep 17 07:28:30 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-6[49830]: 2024-09-17T07:28:30.851+0000 7fef838ff640 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:50.036343+0000 front 2024-09-17T07:27:50.036454+0000 (oldest deadline 2024-09-17T07:28:14.736485+0000) 2024-09-17T07:28:31.044 INFO:journalctl@ceph.osd.6.smithi135.stdout:Sep 17 07:28:30 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-6[49830]: 2024-09-17T07:28:30.851+0000 7fef838ff640 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.100:6807 osd.1 since back 2024-09-17T07:27:59.337832+0000 front 2024-09-17T07:27:59.337847+0000 (oldest deadline 2024-09-17T07:28:25.237603+0000) 2024-09-17T07:28:32.043 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:28:31 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-5[156646]: 2024-09-17T07:28:31.639+0000 7f8f6ef9c640 -1 osd.5 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:53.375825+0000 front 2024-09-17T07:27:53.375768+0000 (oldest deadline 2024-09-17T07:28:17.475723+0000) 2024-09-17T07:28:32.043 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:28:31 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-5[156646]: 2024-09-17T07:28:31.639+0000 7f8f6ef9c640 -1 osd.5 549 heartbeat_check: no reply from 172.21.15.100:6807 osd.1 since back 2024-09-17T07:28:00.976690+0000 front 2024-09-17T07:28:00.976776+0000 (oldest deadline 2024-09-17T07:28:25.076534+0000) 2024-09-17T07:28:32.044 INFO:journalctl@ceph.osd.6.smithi135.stdout:Sep 17 07:28:31 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-6[49830]: 2024-09-17T07:28:31.805+0000 7fef838ff640 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:50.036343+0000 front 2024-09-17T07:27:50.036454+0000 (oldest deadline 2024-09-17T07:28:14.736485+0000) 2024-09-17T07:28:32.044 INFO:journalctl@ceph.osd.6.smithi135.stdout:Sep 17 07:28:31 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-6[49830]: 2024-09-17T07:28:31.805+0000 7fef838ff640 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.100:6807 osd.1 since back 2024-09-17T07:27:59.337832+0000 front 2024-09-17T07:27:59.337847+0000 (oldest deadline 2024-09-17T07:28:25.237603+0000) 2024-09-17T07:28:32.044 INFO:journalctl@ceph.osd.6.smithi135.stdout:Sep 17 07:28:31 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-6[49830]: 2024-09-17T07:28:31.805+0000 7fef838ff640 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.100:6811 osd.2 since back 2024-09-17T07:28:06.938385+0000 front 2024-09-17T07:28:06.938376+0000 (oldest deadline 2024-09-17T07:28:31.638240+0000) 2024-09-17T07:28:32.044 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:31 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:31.816+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:52.894330+0000 front 2024-09-17T07:27:52.894143+0000 (oldest deadline 2024-09-17T07:28:17.594117+0000) 2024-09-17T07:28:32.044 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:31 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:31.816+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6807 osd.1 since back 2024-09-17T07:27:59.895096+0000 front 2024-09-17T07:27:59.895048+0000 (oldest deadline 2024-09-17T07:28:23.995052+0000) 2024-09-17T07:28:32.045 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:31 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:31.816+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6811 osd.2 since back 2024-09-17T07:28:10.396557+0000 front 2024-09-17T07:28:10.396534+0000 (oldest deadline 2024-09-17T07:28:31.496480+0000) 2024-09-17T07:28:33.043 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:28:32 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-5[156646]: 2024-09-17T07:28:32.628+0000 7f8f6ef9c640 -1 osd.5 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:53.375825+0000 front 2024-09-17T07:27:53.375768+0000 (oldest deadline 2024-09-17T07:28:17.475723+0000) 2024-09-17T07:28:33.043 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:28:32 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-5[156646]: 2024-09-17T07:28:32.628+0000 7f8f6ef9c640 -1 osd.5 549 heartbeat_check: no reply from 172.21.15.100:6807 osd.1 since back 2024-09-17T07:28:00.976690+0000 front 2024-09-17T07:28:00.976776+0000 (oldest deadline 2024-09-17T07:28:25.076534+0000) 2024-09-17T07:28:33.044 INFO:journalctl@ceph.osd.6.smithi135.stdout:Sep 17 07:28:32 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-6[49830]: 2024-09-17T07:28:32.848+0000 7fef838ff640 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:50.036343+0000 front 2024-09-17T07:27:50.036454+0000 (oldest deadline 2024-09-17T07:28:14.736485+0000) 2024-09-17T07:28:33.044 INFO:journalctl@ceph.osd.6.smithi135.stdout:Sep 17 07:28:32 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-6[49830]: 2024-09-17T07:28:32.848+0000 7fef838ff640 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.100:6807 osd.1 since back 2024-09-17T07:27:59.337832+0000 front 2024-09-17T07:27:59.337847+0000 (oldest deadline 2024-09-17T07:28:25.237603+0000) 2024-09-17T07:28:33.044 INFO:journalctl@ceph.osd.6.smithi135.stdout:Sep 17 07:28:32 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-6[49830]: 2024-09-17T07:28:32.848+0000 7fef838ff640 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.100:6811 osd.2 since back 2024-09-17T07:28:06.938385+0000 front 2024-09-17T07:28:06.938376+0000 (oldest deadline 2024-09-17T07:28:31.638240+0000) 2024-09-17T07:28:33.044 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:32 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:32.836+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:52.894330+0000 front 2024-09-17T07:27:52.894143+0000 (oldest deadline 2024-09-17T07:28:17.594117+0000) 2024-09-17T07:28:33.044 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:32 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:32.836+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6807 osd.1 since back 2024-09-17T07:27:59.895096+0000 front 2024-09-17T07:27:59.895048+0000 (oldest deadline 2024-09-17T07:28:23.995052+0000) 2024-09-17T07:28:33.044 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:32 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:32.836+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6811 osd.2 since back 2024-09-17T07:28:10.396557+0000 front 2024-09-17T07:28:10.396534+0000 (oldest deadline 2024-09-17T07:28:31.496480+0000) 2024-09-17T07:28:34.043 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:28:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-5[156646]: 2024-09-17T07:28:33.673+0000 7f8f6ef9c640 -1 osd.5 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:53.375825+0000 front 2024-09-17T07:27:53.375768+0000 (oldest deadline 2024-09-17T07:28:17.475723+0000) 2024-09-17T07:28:34.043 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:28:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-5[156646]: 2024-09-17T07:28:33.673+0000 7f8f6ef9c640 -1 osd.5 549 heartbeat_check: no reply from 172.21.15.100:6807 osd.1 since back 2024-09-17T07:28:00.976690+0000 front 2024-09-17T07:28:00.976776+0000 (oldest deadline 2024-09-17T07:28:25.076534+0000) 2024-09-17T07:28:34.044 INFO:journalctl@ceph.osd.6.smithi135.stdout:Sep 17 07:28:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-6[49830]: 2024-09-17T07:28:33.803+0000 7fef838ff640 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:50.036343+0000 front 2024-09-17T07:27:50.036454+0000 (oldest deadline 2024-09-17T07:28:14.736485+0000) 2024-09-17T07:28:34.044 INFO:journalctl@ceph.osd.6.smithi135.stdout:Sep 17 07:28:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-6[49830]: 2024-09-17T07:28:33.803+0000 7fef838ff640 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.100:6807 osd.1 since back 2024-09-17T07:27:59.337832+0000 front 2024-09-17T07:27:59.337847+0000 (oldest deadline 2024-09-17T07:28:25.237603+0000) 2024-09-17T07:28:34.044 INFO:journalctl@ceph.osd.6.smithi135.stdout:Sep 17 07:28:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-6[49830]: 2024-09-17T07:28:33.803+0000 7fef838ff640 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.100:6811 osd.2 since back 2024-09-17T07:28:06.938385+0000 front 2024-09-17T07:28:06.938376+0000 (oldest deadline 2024-09-17T07:28:31.638240+0000) 2024-09-17T07:28:34.044 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:33.811+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:52.894330+0000 front 2024-09-17T07:27:52.894143+0000 (oldest deadline 2024-09-17T07:28:17.594117+0000) 2024-09-17T07:28:34.044 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:33.811+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6807 osd.1 since back 2024-09-17T07:27:59.895096+0000 front 2024-09-17T07:27:59.895048+0000 (oldest deadline 2024-09-17T07:28:23.995052+0000) 2024-09-17T07:28:34.044 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:33 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:33.811+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6811 osd.2 since back 2024-09-17T07:28:10.396557+0000 front 2024-09-17T07:28:10.396534+0000 (oldest deadline 2024-09-17T07:28:31.496480+0000) 2024-09-17T07:28:35.043 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:28:34 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-5[156646]: 2024-09-17T07:28:34.714+0000 7f8f6ef9c640 -1 osd.5 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:53.375825+0000 front 2024-09-17T07:27:53.375768+0000 (oldest deadline 2024-09-17T07:28:17.475723+0000) 2024-09-17T07:28:35.043 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:28:34 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-5[156646]: 2024-09-17T07:28:34.714+0000 7f8f6ef9c640 -1 osd.5 549 heartbeat_check: no reply from 172.21.15.100:6807 osd.1 since back 2024-09-17T07:28:00.976690+0000 front 2024-09-17T07:28:00.976776+0000 (oldest deadline 2024-09-17T07:28:25.076534+0000) 2024-09-17T07:28:35.043 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:28:34 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-5[156646]: 2024-09-17T07:28:34.714+0000 7f8f6ef9c640 -1 osd.5 549 heartbeat_check: no reply from 172.21.15.100:6811 osd.2 since back 2024-09-17T07:28:10.377336+0000 front 2024-09-17T07:28:10.377364+0000 (oldest deadline 2024-09-17T07:28:33.877260+0000) 2024-09-17T07:28:35.044 INFO:journalctl@ceph.osd.6.smithi135.stdout:Sep 17 07:28:34 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-6[49830]: 2024-09-17T07:28:34.832+0000 7fef838ff640 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:50.036343+0000 front 2024-09-17T07:27:50.036454+0000 (oldest deadline 2024-09-17T07:28:14.736485+0000) 2024-09-17T07:28:35.044 INFO:journalctl@ceph.osd.6.smithi135.stdout:Sep 17 07:28:34 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-6[49830]: 2024-09-17T07:28:34.832+0000 7fef838ff640 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.100:6807 osd.1 since back 2024-09-17T07:27:59.337832+0000 front 2024-09-17T07:27:59.337847+0000 (oldest deadline 2024-09-17T07:28:25.237603+0000) 2024-09-17T07:28:35.044 INFO:journalctl@ceph.osd.6.smithi135.stdout:Sep 17 07:28:34 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-6[49830]: 2024-09-17T07:28:34.832+0000 7fef838ff640 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.100:6811 osd.2 since back 2024-09-17T07:28:06.938385+0000 front 2024-09-17T07:28:06.938376+0000 (oldest deadline 2024-09-17T07:28:31.638240+0000) 2024-09-17T07:28:35.044 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:34 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:34.855+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:52.894330+0000 front 2024-09-17T07:27:52.894143+0000 (oldest deadline 2024-09-17T07:28:17.594117+0000) 2024-09-17T07:28:35.044 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:34 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:34.855+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6807 osd.1 since back 2024-09-17T07:27:59.895096+0000 front 2024-09-17T07:27:59.895048+0000 (oldest deadline 2024-09-17T07:28:23.995052+0000) 2024-09-17T07:28:35.044 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:34 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:34.855+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6811 osd.2 since back 2024-09-17T07:28:10.396557+0000 front 2024-09-17T07:28:10.396534+0000 (oldest deadline 2024-09-17T07:28:31.496480+0000) 2024-09-17T07:28:36.137 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:28:35 smithi135 podman[164990]: 2024-09-17 07:28:35.866378981 +0000 UTC m=+5.647760964 container died 5f30512b77c0b4a39ad76d1f20992868181d0a846398a492eeed858079942c35 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, RELEASE=reef-19e2a2b, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-09-17T07:28:36.137 INFO:journalctl@ceph.osd.6.smithi135.stdout:Sep 17 07:28:35 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-6[49830]: 2024-09-17T07:28:35.821+0000 7fef838ff640 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:50.036343+0000 front 2024-09-17T07:27:50.036454+0000 (oldest deadline 2024-09-17T07:28:14.736485+0000) 2024-09-17T07:28:36.137 INFO:journalctl@ceph.osd.6.smithi135.stdout:Sep 17 07:28:35 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-6[49830]: 2024-09-17T07:28:35.821+0000 7fef838ff640 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.100:6807 osd.1 since back 2024-09-17T07:27:59.337832+0000 front 2024-09-17T07:27:59.337847+0000 (oldest deadline 2024-09-17T07:28:25.237603+0000) 2024-09-17T07:28:36.137 INFO:journalctl@ceph.osd.6.smithi135.stdout:Sep 17 07:28:35 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-6[49830]: 2024-09-17T07:28:35.821+0000 7fef838ff640 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.100:6811 osd.2 since back 2024-09-17T07:28:06.938385+0000 front 2024-09-17T07:28:06.938376+0000 (oldest deadline 2024-09-17T07:28:31.638240+0000) 2024-09-17T07:28:36.137 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:35 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:35.806+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:52.894330+0000 front 2024-09-17T07:27:52.894143+0000 (oldest deadline 2024-09-17T07:28:17.594117+0000) 2024-09-17T07:28:36.137 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:35 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:35.806+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6807 osd.1 since back 2024-09-17T07:27:59.895096+0000 front 2024-09-17T07:27:59.895048+0000 (oldest deadline 2024-09-17T07:28:23.995052+0000) 2024-09-17T07:28:36.137 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:35 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:35.806+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6811 osd.2 since back 2024-09-17T07:28:10.396557+0000 front 2024-09-17T07:28:10.396534+0000 (oldest deadline 2024-09-17T07:28:31.496480+0000) 2024-09-17T07:28:36.528 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:28:36 smithi135 podman[164990]: 2024-09-17 07:28:36.217692112 +0000 UTC m=+5.999074093 container cleanup 5f30512b77c0b4a39ad76d1f20992868181d0a846398a492eeed858079942c35 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, RELEASE=reef-19e2a2b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-09-17T07:28:36.528 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:28:36 smithi135 bash[164990]: ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-5 2024-09-17T07:28:36.782 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:28:36 smithi135 podman[165004]: 2024-09-17 07:28:36.528289945 +0000 UTC m=+0.657512928 container remove 5f30512b77c0b4a39ad76d1f20992868181d0a846398a492eeed858079942c35 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-5, RELEASE=reef-19e2a2b, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731) 2024-09-17T07:28:37.035 INFO:journalctl@ceph.osd.6.smithi135.stdout:Sep 17 07:28:36 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-6[49830]: 2024-09-17T07:28:36.781+0000 7fef838ff640 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:50.036343+0000 front 2024-09-17T07:27:50.036454+0000 (oldest deadline 2024-09-17T07:28:14.736485+0000) 2024-09-17T07:28:37.035 INFO:journalctl@ceph.osd.6.smithi135.stdout:Sep 17 07:28:36 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-6[49830]: 2024-09-17T07:28:36.781+0000 7fef838ff640 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.100:6807 osd.1 since back 2024-09-17T07:27:59.337832+0000 front 2024-09-17T07:27:59.337847+0000 (oldest deadline 2024-09-17T07:28:25.237603+0000) 2024-09-17T07:28:37.035 INFO:journalctl@ceph.osd.6.smithi135.stdout:Sep 17 07:28:36 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-6[49830]: 2024-09-17T07:28:36.781+0000 7fef838ff640 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.100:6811 osd.2 since back 2024-09-17T07:28:06.938385+0000 front 2024-09-17T07:28:06.938376+0000 (oldest deadline 2024-09-17T07:28:31.638240+0000) 2024-09-17T07:28:37.036 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:28:36 smithi135 podman[165098]: 2024-09-17 07:28:36.786409233 +0000 UTC m=+0.016337200 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9 2024-09-17T07:28:37.036 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:28:36 smithi135 podman[165098]: 2024-09-17 07:28:36.896221821 +0000 UTC m=+0.126149779 container create 9a271feb92b15fe3495587f55b34edb7db4bb1510b1135b321deddbb831396ff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-5-deactivate, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=reef-19e2a2b, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=) 2024-09-17T07:28:37.036 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:36 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:36.804+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:52.894330+0000 front 2024-09-17T07:27:52.894143+0000 (oldest deadline 2024-09-17T07:28:17.594117+0000) 2024-09-17T07:28:37.036 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:36 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:36.804+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6807 osd.1 since back 2024-09-17T07:27:59.895096+0000 front 2024-09-17T07:27:59.895048+0000 (oldest deadline 2024-09-17T07:28:23.995052+0000) 2024-09-17T07:28:37.037 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:36 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:36.804+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6811 osd.2 since back 2024-09-17T07:28:10.396557+0000 front 2024-09-17T07:28:10.396534+0000 (oldest deadline 2024-09-17T07:28:31.496480+0000) 2024-09-17T07:28:37.543 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:28:37 smithi135 podman[165098]: 2024-09-17 07:28:37.157927141 +0000 UTC m=+0.387855111 container init 9a271feb92b15fe3495587f55b34edb7db4bb1510b1135b321deddbb831396ff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-5-deactivate, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.37.2, RELEASE=reef-19e2a2b, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-09-17T07:28:37.543 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:28:37 smithi135 podman[165098]: 2024-09-17 07:28:37.161185512 +0000 UTC m=+0.391113481 container start 9a271feb92b15fe3495587f55b34edb7db4bb1510b1135b321deddbb831396ff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-5-deactivate, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-19e2a2b, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T07:28:37.543 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:28:37 smithi135 podman[165098]: 2024-09-17 07:28:37.232806919 +0000 UTC m=+0.462734887 container attach 9a271feb92b15fe3495587f55b34edb7db4bb1510b1135b321deddbb831396ff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-5-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=reef-19e2a2b, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T07:28:37.543 INFO:journalctl@ceph.osd.5.smithi135.stdout:Sep 17 07:28:37 smithi135 podman[165098]: 2024-09-17 07:28:37.347612966 +0000 UTC m=+0.577540926 container died 9a271feb92b15fe3495587f55b34edb7db4bb1510b1135b321deddbb831396ff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-5-deactivate, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-19e2a2b, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2) 2024-09-17T07:28:37.921 INFO:journalctl@ceph.osd.6.smithi135.stdout:Sep 17 07:28:37 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-6[49830]: 2024-09-17T07:28:37.782+0000 7fef838ff640 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:50.036343+0000 front 2024-09-17T07:27:50.036454+0000 (oldest deadline 2024-09-17T07:28:14.736485+0000) 2024-09-17T07:28:37.921 INFO:journalctl@ceph.osd.6.smithi135.stdout:Sep 17 07:28:37 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-6[49830]: 2024-09-17T07:28:37.782+0000 7fef838ff640 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.100:6807 osd.1 since back 2024-09-17T07:27:59.337832+0000 front 2024-09-17T07:27:59.337847+0000 (oldest deadline 2024-09-17T07:28:25.237603+0000) 2024-09-17T07:28:37.921 INFO:journalctl@ceph.osd.6.smithi135.stdout:Sep 17 07:28:37 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-6[49830]: 2024-09-17T07:28:37.782+0000 7fef838ff640 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.100:6811 osd.2 since back 2024-09-17T07:28:06.938385+0000 front 2024-09-17T07:28:06.938376+0000 (oldest deadline 2024-09-17T07:28:31.638240+0000) 2024-09-17T07:28:37.921 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:37 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:37.825+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:52.894330+0000 front 2024-09-17T07:27:52.894143+0000 (oldest deadline 2024-09-17T07:28:17.594117+0000) 2024-09-17T07:28:37.922 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:37 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:37.825+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6807 osd.1 since back 2024-09-17T07:27:59.895096+0000 front 2024-09-17T07:27:59.895048+0000 (oldest deadline 2024-09-17T07:28:23.995052+0000) 2024-09-17T07:28:37.922 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:37 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:37.825+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6811 osd.2 since back 2024-09-17T07:28:10.396557+0000 front 2024-09-17T07:28:10.396534+0000 (oldest deadline 2024-09-17T07:28:31.496480+0000) 2024-09-17T07:28:37.943 DEBUG:teuthology.orchestra.run.smithi135:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e848267c-74c1-11ef-bceb-c7b262605968@osd.5.service' 2024-09-17T07:28:37.981 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T07:28:37.981 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-09-17T07:28:37.981 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-09-17T07:28:37.982 DEBUG:teuthology.orchestra.run.smithi135:> sudo systemctl stop ceph-e848267c-74c1-11ef-bceb-c7b262605968@osd.6 2024-09-17T07:28:38.293 INFO:journalctl@ceph.osd.6.smithi135.stdout:Sep 17 07:28:38 smithi135 systemd[1]: Stopping Ceph osd.6 for e848267c-74c1-11ef-bceb-c7b262605968... 2024-09-17T07:28:38.793 INFO:journalctl@ceph.osd.6.smithi135.stdout:Sep 17 07:28:38 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-6[49830]: 2024-09-17T07:28:38.298+0000 7fef8b2f3640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T07:28:38.793 INFO:journalctl@ceph.osd.6.smithi135.stdout:Sep 17 07:28:38 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-6[49830]: 2024-09-17T07:28:38.298+0000 7fef8b2f3640 -1 osd.6 549 *** Got signal Terminated *** 2024-09-17T07:28:38.793 INFO:journalctl@ceph.osd.6.smithi135.stdout:Sep 17 07:28:38 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-6[49830]: 2024-09-17T07:28:38.298+0000 7fef8b2f3640 -1 osd.6 549 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T07:28:39.293 INFO:journalctl@ceph.osd.6.smithi135.stdout:Sep 17 07:28:38 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-6[49830]: 2024-09-17T07:28:38.795+0000 7fef838ff640 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:50.036343+0000 front 2024-09-17T07:27:50.036454+0000 (oldest deadline 2024-09-17T07:28:14.736485+0000) 2024-09-17T07:28:39.293 INFO:journalctl@ceph.osd.6.smithi135.stdout:Sep 17 07:28:38 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-6[49830]: 2024-09-17T07:28:38.795+0000 7fef838ff640 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.100:6807 osd.1 since back 2024-09-17T07:27:59.337832+0000 front 2024-09-17T07:27:59.337847+0000 (oldest deadline 2024-09-17T07:28:25.237603+0000) 2024-09-17T07:28:39.293 INFO:journalctl@ceph.osd.6.smithi135.stdout:Sep 17 07:28:38 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-6[49830]: 2024-09-17T07:28:38.795+0000 7fef838ff640 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.100:6811 osd.2 since back 2024-09-17T07:28:06.938385+0000 front 2024-09-17T07:28:06.938376+0000 (oldest deadline 2024-09-17T07:28:31.638240+0000) 2024-09-17T07:28:39.293 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:38 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:38.851+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:52.894330+0000 front 2024-09-17T07:27:52.894143+0000 (oldest deadline 2024-09-17T07:28:17.594117+0000) 2024-09-17T07:28:39.293 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:38 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:38.851+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6807 osd.1 since back 2024-09-17T07:27:59.895096+0000 front 2024-09-17T07:27:59.895048+0000 (oldest deadline 2024-09-17T07:28:23.995052+0000) 2024-09-17T07:28:39.293 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:38 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:38.851+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6811 osd.2 since back 2024-09-17T07:28:10.396557+0000 front 2024-09-17T07:28:10.396534+0000 (oldest deadline 2024-09-17T07:28:31.496480+0000) 2024-09-17T07:28:40.293 INFO:journalctl@ceph.osd.6.smithi135.stdout:Sep 17 07:28:39 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-6[49830]: 2024-09-17T07:28:39.795+0000 7fef838ff640 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:50.036343+0000 front 2024-09-17T07:27:50.036454+0000 (oldest deadline 2024-09-17T07:28:14.736485+0000) 2024-09-17T07:28:40.293 INFO:journalctl@ceph.osd.6.smithi135.stdout:Sep 17 07:28:39 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-6[49830]: 2024-09-17T07:28:39.795+0000 7fef838ff640 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.100:6807 osd.1 since back 2024-09-17T07:27:59.337832+0000 front 2024-09-17T07:27:59.337847+0000 (oldest deadline 2024-09-17T07:28:25.237603+0000) 2024-09-17T07:28:40.294 INFO:journalctl@ceph.osd.6.smithi135.stdout:Sep 17 07:28:39 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-6[49830]: 2024-09-17T07:28:39.795+0000 7fef838ff640 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.100:6811 osd.2 since back 2024-09-17T07:28:06.938385+0000 front 2024-09-17T07:28:06.938376+0000 (oldest deadline 2024-09-17T07:28:31.638240+0000) 2024-09-17T07:28:40.294 INFO:journalctl@ceph.osd.6.smithi135.stdout:Sep 17 07:28:39 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-6[49830]: 2024-09-17T07:28:39.795+0000 7fef838ff640 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.100:6815 osd.3 since back 2024-09-17T07:28:17.439435+0000 front 2024-09-17T07:28:17.439420+0000 (oldest deadline 2024-09-17T07:28:39.739350+0000) 2024-09-17T07:28:40.294 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:39 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:39.859+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:52.894330+0000 front 2024-09-17T07:27:52.894143+0000 (oldest deadline 2024-09-17T07:28:17.594117+0000) 2024-09-17T07:28:40.294 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:39 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:39.859+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6807 osd.1 since back 2024-09-17T07:27:59.895096+0000 front 2024-09-17T07:27:59.895048+0000 (oldest deadline 2024-09-17T07:28:23.995052+0000) 2024-09-17T07:28:40.294 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:39 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:39.859+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6811 osd.2 since back 2024-09-17T07:28:10.396557+0000 front 2024-09-17T07:28:10.396534+0000 (oldest deadline 2024-09-17T07:28:31.496480+0000) 2024-09-17T07:28:40.294 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:39 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:39.859+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6815 osd.3 since back 2024-09-17T07:28:16.097910+0000 front 2024-09-17T07:28:16.097883+0000 (oldest deadline 2024-09-17T07:28:39.597820+0000) 2024-09-17T07:28:41.043 INFO:journalctl@ceph.osd.6.smithi135.stdout:Sep 17 07:28:40 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-6[49830]: 2024-09-17T07:28:40.781+0000 7fef838ff640 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:50.036343+0000 front 2024-09-17T07:27:50.036454+0000 (oldest deadline 2024-09-17T07:28:14.736485+0000) 2024-09-17T07:28:41.043 INFO:journalctl@ceph.osd.6.smithi135.stdout:Sep 17 07:28:40 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-6[49830]: 2024-09-17T07:28:40.781+0000 7fef838ff640 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.100:6807 osd.1 since back 2024-09-17T07:27:59.337832+0000 front 2024-09-17T07:27:59.337847+0000 (oldest deadline 2024-09-17T07:28:25.237603+0000) 2024-09-17T07:28:41.043 INFO:journalctl@ceph.osd.6.smithi135.stdout:Sep 17 07:28:40 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-6[49830]: 2024-09-17T07:28:40.781+0000 7fef838ff640 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.100:6811 osd.2 since back 2024-09-17T07:28:06.938385+0000 front 2024-09-17T07:28:06.938376+0000 (oldest deadline 2024-09-17T07:28:31.638240+0000) 2024-09-17T07:28:41.043 INFO:journalctl@ceph.osd.6.smithi135.stdout:Sep 17 07:28:40 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-6[49830]: 2024-09-17T07:28:40.781+0000 7fef838ff640 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.100:6815 osd.3 since back 2024-09-17T07:28:17.439435+0000 front 2024-09-17T07:28:17.439420+0000 (oldest deadline 2024-09-17T07:28:39.739350+0000) 2024-09-17T07:28:41.044 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:40 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:40.852+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:52.894330+0000 front 2024-09-17T07:27:52.894143+0000 (oldest deadline 2024-09-17T07:28:17.594117+0000) 2024-09-17T07:28:41.044 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:40 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:40.852+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6807 osd.1 since back 2024-09-17T07:27:59.895096+0000 front 2024-09-17T07:27:59.895048+0000 (oldest deadline 2024-09-17T07:28:23.995052+0000) 2024-09-17T07:28:41.044 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:40 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:40.852+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6811 osd.2 since back 2024-09-17T07:28:10.396557+0000 front 2024-09-17T07:28:10.396534+0000 (oldest deadline 2024-09-17T07:28:31.496480+0000) 2024-09-17T07:28:41.044 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:40 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:40.852+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6815 osd.3 since back 2024-09-17T07:28:16.097910+0000 front 2024-09-17T07:28:16.097883+0000 (oldest deadline 2024-09-17T07:28:39.597820+0000) 2024-09-17T07:28:42.043 INFO:journalctl@ceph.osd.6.smithi135.stdout:Sep 17 07:28:41 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-6[49830]: 2024-09-17T07:28:41.734+0000 7fef838ff640 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:50.036343+0000 front 2024-09-17T07:27:50.036454+0000 (oldest deadline 2024-09-17T07:28:14.736485+0000) 2024-09-17T07:28:42.043 INFO:journalctl@ceph.osd.6.smithi135.stdout:Sep 17 07:28:41 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-6[49830]: 2024-09-17T07:28:41.734+0000 7fef838ff640 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.100:6807 osd.1 since back 2024-09-17T07:27:59.337832+0000 front 2024-09-17T07:27:59.337847+0000 (oldest deadline 2024-09-17T07:28:25.237603+0000) 2024-09-17T07:28:42.043 INFO:journalctl@ceph.osd.6.smithi135.stdout:Sep 17 07:28:41 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-6[49830]: 2024-09-17T07:28:41.734+0000 7fef838ff640 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.100:6811 osd.2 since back 2024-09-17T07:28:06.938385+0000 front 2024-09-17T07:28:06.938376+0000 (oldest deadline 2024-09-17T07:28:31.638240+0000) 2024-09-17T07:28:42.043 INFO:journalctl@ceph.osd.6.smithi135.stdout:Sep 17 07:28:41 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-6[49830]: 2024-09-17T07:28:41.734+0000 7fef838ff640 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.100:6815 osd.3 since back 2024-09-17T07:28:17.439435+0000 front 2024-09-17T07:28:17.439420+0000 (oldest deadline 2024-09-17T07:28:39.739350+0000) 2024-09-17T07:28:42.044 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:41 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:41.885+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:52.894330+0000 front 2024-09-17T07:27:52.894143+0000 (oldest deadline 2024-09-17T07:28:17.594117+0000) 2024-09-17T07:28:42.044 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:41 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:41.885+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6807 osd.1 since back 2024-09-17T07:27:59.895096+0000 front 2024-09-17T07:27:59.895048+0000 (oldest deadline 2024-09-17T07:28:23.995052+0000) 2024-09-17T07:28:42.044 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:41 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:41.885+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6811 osd.2 since back 2024-09-17T07:28:10.396557+0000 front 2024-09-17T07:28:10.396534+0000 (oldest deadline 2024-09-17T07:28:31.496480+0000) 2024-09-17T07:28:42.044 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:41 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:41.885+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6815 osd.3 since back 2024-09-17T07:28:16.097910+0000 front 2024-09-17T07:28:16.097883+0000 (oldest deadline 2024-09-17T07:28:39.597820+0000) 2024-09-17T07:28:43.043 INFO:journalctl@ceph.osd.6.smithi135.stdout:Sep 17 07:28:42 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-6[49830]: 2024-09-17T07:28:42.780+0000 7fef838ff640 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:50.036343+0000 front 2024-09-17T07:27:50.036454+0000 (oldest deadline 2024-09-17T07:28:14.736485+0000) 2024-09-17T07:28:43.043 INFO:journalctl@ceph.osd.6.smithi135.stdout:Sep 17 07:28:42 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-6[49830]: 2024-09-17T07:28:42.780+0000 7fef838ff640 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.100:6807 osd.1 since back 2024-09-17T07:27:59.337832+0000 front 2024-09-17T07:27:59.337847+0000 (oldest deadline 2024-09-17T07:28:25.237603+0000) 2024-09-17T07:28:43.043 INFO:journalctl@ceph.osd.6.smithi135.stdout:Sep 17 07:28:42 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-6[49830]: 2024-09-17T07:28:42.780+0000 7fef838ff640 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.100:6811 osd.2 since back 2024-09-17T07:28:06.938385+0000 front 2024-09-17T07:28:06.938376+0000 (oldest deadline 2024-09-17T07:28:31.638240+0000) 2024-09-17T07:28:43.044 INFO:journalctl@ceph.osd.6.smithi135.stdout:Sep 17 07:28:42 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-6[49830]: 2024-09-17T07:28:42.780+0000 7fef838ff640 -1 osd.6 549 heartbeat_check: no reply from 172.21.15.100:6815 osd.3 since back 2024-09-17T07:28:17.439435+0000 front 2024-09-17T07:28:17.439420+0000 (oldest deadline 2024-09-17T07:28:39.739350+0000) 2024-09-17T07:28:43.044 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:42 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:42.920+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:52.894330+0000 front 2024-09-17T07:27:52.894143+0000 (oldest deadline 2024-09-17T07:28:17.594117+0000) 2024-09-17T07:28:43.044 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:42 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:42.920+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6807 osd.1 since back 2024-09-17T07:27:59.895096+0000 front 2024-09-17T07:27:59.895048+0000 (oldest deadline 2024-09-17T07:28:23.995052+0000) 2024-09-17T07:28:43.044 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:42 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:42.920+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6811 osd.2 since back 2024-09-17T07:28:10.396557+0000 front 2024-09-17T07:28:10.396534+0000 (oldest deadline 2024-09-17T07:28:31.496480+0000) 2024-09-17T07:28:43.044 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:42 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:42.920+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6815 osd.3 since back 2024-09-17T07:28:16.097910+0000 front 2024-09-17T07:28:16.097883+0000 (oldest deadline 2024-09-17T07:28:39.597820+0000) 2024-09-17T07:28:44.217 INFO:journalctl@ceph.osd.6.smithi135.stdout:Sep 17 07:28:43 smithi135 podman[165201]: 2024-09-17 07:28:43.946976779 +0000 UTC m=+5.738198123 container died 20f1f559b55e2e7b1f5bbbdba0f7611b7d8879bdc72d887c72006046654bcef5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-6, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=reef-19e2a2b, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=) 2024-09-17T07:28:44.218 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:43 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:43.969+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:52.894330+0000 front 2024-09-17T07:27:52.894143+0000 (oldest deadline 2024-09-17T07:28:17.594117+0000) 2024-09-17T07:28:44.218 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:43 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:43.969+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6807 osd.1 since back 2024-09-17T07:27:59.895096+0000 front 2024-09-17T07:27:59.895048+0000 (oldest deadline 2024-09-17T07:28:23.995052+0000) 2024-09-17T07:28:44.218 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:43 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:43.969+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6811 osd.2 since back 2024-09-17T07:28:10.396557+0000 front 2024-09-17T07:28:10.396534+0000 (oldest deadline 2024-09-17T07:28:31.496480+0000) 2024-09-17T07:28:44.218 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:43 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:43.969+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6815 osd.3 since back 2024-09-17T07:28:16.097910+0000 front 2024-09-17T07:28:16.097883+0000 (oldest deadline 2024-09-17T07:28:39.597820+0000) 2024-09-17T07:28:44.543 INFO:journalctl@ceph.osd.6.smithi135.stdout:Sep 17 07:28:44 smithi135 podman[165201]: 2024-09-17 07:28:44.316734017 +0000 UTC m=+6.107955349 container cleanup 20f1f559b55e2e7b1f5bbbdba0f7611b7d8879bdc72d887c72006046654bcef5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-6, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=reef-19e2a2b) 2024-09-17T07:28:44.543 INFO:journalctl@ceph.osd.6.smithi135.stdout:Sep 17 07:28:44 smithi135 bash[165201]: ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-6 2024-09-17T07:28:44.919 INFO:journalctl@ceph.osd.6.smithi135.stdout:Sep 17 07:28:44 smithi135 podman[165216]: 2024-09-17 07:28:44.576991847 +0000 UTC m=+0.626921772 container remove 20f1f559b55e2e7b1f5bbbdba0f7611b7d8879bdc72d887c72006046654bcef5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-6, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=reef-19e2a2b, GIT_BRANCH=HEAD, io.buildah.version=1.37.2) 2024-09-17T07:28:45.231 INFO:journalctl@ceph.osd.6.smithi135.stdout:Sep 17 07:28:44 smithi135 podman[165304]: 2024-09-17 07:28:44.919743346 +0000 UTC m=+0.110122698 container create 23c4dc826ef48f89ad3b97a67a19ed6e96ae4bd0d26b354f8dd830f99baef9aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-6-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , RELEASE=reef-19e2a2b, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-09-17T07:28:45.231 INFO:journalctl@ceph.osd.6.smithi135.stdout:Sep 17 07:28:44 smithi135 podman[165304]: 2024-09-17 07:28:44.825588477 +0000 UTC m=+0.015967829 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9 2024-09-17T07:28:45.231 INFO:journalctl@ceph.osd.6.smithi135.stdout:Sep 17 07:28:45 smithi135 podman[165304]: 2024-09-17 07:28:45.155984344 +0000 UTC m=+0.346363699 container init 23c4dc826ef48f89ad3b97a67a19ed6e96ae4bd0d26b354f8dd830f99baef9aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-6-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, RELEASE=reef-19e2a2b, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD) 2024-09-17T07:28:45.231 INFO:journalctl@ceph.osd.6.smithi135.stdout:Sep 17 07:28:45 smithi135 podman[165304]: 2024-09-17 07:28:45.15924488 +0000 UTC m=+0.349624232 container start 23c4dc826ef48f89ad3b97a67a19ed6e96ae4bd0d26b354f8dd830f99baef9aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-6-deactivate, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=reef-19e2a2b, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux ) 2024-09-17T07:28:45.232 INFO:journalctl@ceph.osd.6.smithi135.stdout:Sep 17 07:28:45 smithi135 podman[165304]: 2024-09-17 07:28:45.231075188 +0000 UTC m=+0.421454553 container attach 23c4dc826ef48f89ad3b97a67a19ed6e96ae4bd0d26b354f8dd830f99baef9aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-6-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, RELEASE=reef-19e2a2b, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T07:28:45.232 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:44 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:44.969+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:52.894330+0000 front 2024-09-17T07:27:52.894143+0000 (oldest deadline 2024-09-17T07:28:17.594117+0000) 2024-09-17T07:28:45.232 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:44 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:44.969+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6807 osd.1 since back 2024-09-17T07:27:59.895096+0000 front 2024-09-17T07:27:59.895048+0000 (oldest deadline 2024-09-17T07:28:23.995052+0000) 2024-09-17T07:28:45.232 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:44 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:44.969+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6811 osd.2 since back 2024-09-17T07:28:10.396557+0000 front 2024-09-17T07:28:10.396534+0000 (oldest deadline 2024-09-17T07:28:31.496480+0000) 2024-09-17T07:28:45.232 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:44 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:44.969+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6815 osd.3 since back 2024-09-17T07:28:16.097910+0000 front 2024-09-17T07:28:16.097883+0000 (oldest deadline 2024-09-17T07:28:39.597820+0000) 2024-09-17T07:28:45.543 INFO:journalctl@ceph.osd.6.smithi135.stdout:Sep 17 07:28:45 smithi135 podman[165304]: 2024-09-17 07:28:45.342434929 +0000 UTC m=+0.532814284 container died 23c4dc826ef48f89ad3b97a67a19ed6e96ae4bd0d26b354f8dd830f99baef9aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-6-deactivate, RELEASE=reef-19e2a2b, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T07:28:45.947 DEBUG:teuthology.orchestra.run.smithi135:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e848267c-74c1-11ef-bceb-c7b262605968@osd.6.service' 2024-09-17T07:28:45.987 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T07:28:45.987 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-09-17T07:28:45.987 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-09-17T07:28:45.988 DEBUG:teuthology.orchestra.run.smithi135:> sudo systemctl stop ceph-e848267c-74c1-11ef-bceb-c7b262605968@osd.7 2024-09-17T07:28:46.293 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:45 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:45.923+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:52.894330+0000 front 2024-09-17T07:27:52.894143+0000 (oldest deadline 2024-09-17T07:28:17.594117+0000) 2024-09-17T07:28:46.293 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:45 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:45.923+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6807 osd.1 since back 2024-09-17T07:27:59.895096+0000 front 2024-09-17T07:27:59.895048+0000 (oldest deadline 2024-09-17T07:28:23.995052+0000) 2024-09-17T07:28:46.293 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:45 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:45.923+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6811 osd.2 since back 2024-09-17T07:28:10.396557+0000 front 2024-09-17T07:28:10.396534+0000 (oldest deadline 2024-09-17T07:28:31.496480+0000) 2024-09-17T07:28:46.293 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:45 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:45.923+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6815 osd.3 since back 2024-09-17T07:28:16.097910+0000 front 2024-09-17T07:28:16.097883+0000 (oldest deadline 2024-09-17T07:28:39.597820+0000) 2024-09-17T07:28:46.293 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:46 smithi135 systemd[1]: Stopping Ceph osd.7 for e848267c-74c1-11ef-bceb-c7b262605968... 2024-09-17T07:28:46.793 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:46 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:46.330+0000 7fab0f005640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T07:28:46.793 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:46 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:46.330+0000 7fab0f005640 -1 osd.7 549 *** Got signal Terminated *** 2024-09-17T07:28:46.793 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:46 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:46.330+0000 7fab0f005640 -1 osd.7 549 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T07:28:47.280 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:46 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:46.896+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:52.894330+0000 front 2024-09-17T07:27:52.894143+0000 (oldest deadline 2024-09-17T07:28:17.594117+0000) 2024-09-17T07:28:47.280 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:46 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:46.896+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6807 osd.1 since back 2024-09-17T07:27:59.895096+0000 front 2024-09-17T07:27:59.895048+0000 (oldest deadline 2024-09-17T07:28:23.995052+0000) 2024-09-17T07:28:47.280 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:46 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:46.896+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6811 osd.2 since back 2024-09-17T07:28:10.396557+0000 front 2024-09-17T07:28:10.396534+0000 (oldest deadline 2024-09-17T07:28:31.496480+0000) 2024-09-17T07:28:47.280 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:46 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:46.896+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6815 osd.3 since back 2024-09-17T07:28:16.097910+0000 front 2024-09-17T07:28:16.097883+0000 (oldest deadline 2024-09-17T07:28:39.597820+0000) 2024-09-17T07:28:48.293 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:47 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:47.902+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:52.894330+0000 front 2024-09-17T07:27:52.894143+0000 (oldest deadline 2024-09-17T07:28:17.594117+0000) 2024-09-17T07:28:48.293 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:47 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:47.902+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6807 osd.1 since back 2024-09-17T07:27:59.895096+0000 front 2024-09-17T07:27:59.895048+0000 (oldest deadline 2024-09-17T07:28:23.995052+0000) 2024-09-17T07:28:48.293 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:47 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:47.902+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6811 osd.2 since back 2024-09-17T07:28:10.396557+0000 front 2024-09-17T07:28:10.396534+0000 (oldest deadline 2024-09-17T07:28:31.496480+0000) 2024-09-17T07:28:48.293 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:47 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:47.902+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6815 osd.3 since back 2024-09-17T07:28:16.097910+0000 front 2024-09-17T07:28:16.097883+0000 (oldest deadline 2024-09-17T07:28:39.597820+0000) 2024-09-17T07:28:49.293 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:48 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:48.913+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:52.894330+0000 front 2024-09-17T07:27:52.894143+0000 (oldest deadline 2024-09-17T07:28:17.594117+0000) 2024-09-17T07:28:49.293 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:48 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:48.913+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6807 osd.1 since back 2024-09-17T07:27:59.895096+0000 front 2024-09-17T07:27:59.895048+0000 (oldest deadline 2024-09-17T07:28:23.995052+0000) 2024-09-17T07:28:49.293 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:48 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:48.913+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6811 osd.2 since back 2024-09-17T07:28:10.396557+0000 front 2024-09-17T07:28:10.396534+0000 (oldest deadline 2024-09-17T07:28:31.496480+0000) 2024-09-17T07:28:49.293 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:48 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:48.913+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6815 osd.3 since back 2024-09-17T07:28:16.097910+0000 front 2024-09-17T07:28:16.097883+0000 (oldest deadline 2024-09-17T07:28:39.597820+0000) 2024-09-17T07:28:50.293 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:49 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:49.940+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:52.894330+0000 front 2024-09-17T07:27:52.894143+0000 (oldest deadline 2024-09-17T07:28:17.594117+0000) 2024-09-17T07:28:50.293 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:49 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:49.940+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6807 osd.1 since back 2024-09-17T07:27:59.895096+0000 front 2024-09-17T07:27:59.895048+0000 (oldest deadline 2024-09-17T07:28:23.995052+0000) 2024-09-17T07:28:50.293 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:49 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:49.940+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6811 osd.2 since back 2024-09-17T07:28:10.396557+0000 front 2024-09-17T07:28:10.396534+0000 (oldest deadline 2024-09-17T07:28:31.496480+0000) 2024-09-17T07:28:50.293 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:49 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:49.940+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6815 osd.3 since back 2024-09-17T07:28:16.097910+0000 front 2024-09-17T07:28:16.097883+0000 (oldest deadline 2024-09-17T07:28:39.597820+0000) 2024-09-17T07:28:50.294 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:49 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:49.940+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.135:6802 osd.4 since back 2024-09-17T07:28:26.599121+0000 front 2024-09-17T07:28:26.599201+0000 (oldest deadline 2024-09-17T07:28:49.499038+0000) 2024-09-17T07:28:51.293 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:50 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:50.907+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6803 osd.0 since back 2024-09-17T07:27:52.894330+0000 front 2024-09-17T07:27:52.894143+0000 (oldest deadline 2024-09-17T07:28:17.594117+0000) 2024-09-17T07:28:51.293 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:50 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:50.907+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6807 osd.1 since back 2024-09-17T07:27:59.895096+0000 front 2024-09-17T07:27:59.895048+0000 (oldest deadline 2024-09-17T07:28:23.995052+0000) 2024-09-17T07:28:51.293 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:50 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:50.907+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6811 osd.2 since back 2024-09-17T07:28:10.396557+0000 front 2024-09-17T07:28:10.396534+0000 (oldest deadline 2024-09-17T07:28:31.496480+0000) 2024-09-17T07:28:51.293 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:50 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:50.907+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.100:6815 osd.3 since back 2024-09-17T07:28:16.097910+0000 front 2024-09-17T07:28:16.097883+0000 (oldest deadline 2024-09-17T07:28:39.597820+0000) 2024-09-17T07:28:51.293 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:50 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7[55417]: 2024-09-17T07:28:50.907+0000 7fab07611640 -1 osd.7 549 heartbeat_check: no reply from 172.21.15.135:6802 osd.4 since back 2024-09-17T07:28:26.599121+0000 front 2024-09-17T07:28:26.599201+0000 (oldest deadline 2024-09-17T07:28:49.499038+0000) 2024-09-17T07:28:52.191 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:51 smithi135 podman[165409]: 2024-09-17 07:28:51.917687093 +0000 UTC m=+5.686531872 container died d1d9e20b005e6654940a389cd38ae4b396e57e908abf0bb93cea01063ee76a82 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=reef-19e2a2b, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) 2024-09-17T07:28:52.543 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:52 smithi135 podman[165409]: 2024-09-17 07:28:52.365046467 +0000 UTC m=+6.133891234 container cleanup d1d9e20b005e6654940a389cd38ae4b396e57e908abf0bb93cea01063ee76a82 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=reef-19e2a2b, org.label-schema.vendor=CentOS) 2024-09-17T07:28:52.543 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:52 smithi135 bash[165409]: ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7 2024-09-17T07:28:52.977 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:52 smithi135 podman[165435]: 2024-09-17 07:28:52.666987936 +0000 UTC m=+0.746940384 container remove d1d9e20b005e6654940a389cd38ae4b396e57e908abf0bb93cea01063ee76a82 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, RELEASE=reef-19e2a2b, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-09-17T07:28:53.287 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:52 smithi135 podman[165527]: 2024-09-17 07:28:52.880510041 +0000 UTC m=+0.015750554 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9 2024-09-17T07:28:53.287 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:52 smithi135 podman[165527]: 2024-09-17 07:28:52.98481769 +0000 UTC m=+0.120058193 container create 1ba6efec7cefcc61191e3425eaa297fc623cd49e371fe90bb602265bfd3c5e43 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7-deactivate, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , RELEASE=reef-19e2a2b, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T07:28:53.287 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:53 smithi135 podman[165527]: 2024-09-17 07:28:53.212820672 +0000 UTC m=+0.348061183 container init 1ba6efec7cefcc61191e3425eaa297fc623cd49e371fe90bb602265bfd3c5e43 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-19e2a2b, org.label-schema.build-date=20240731, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD) 2024-09-17T07:28:53.288 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:53 smithi135 podman[165527]: 2024-09-17 07:28:53.216157441 +0000 UTC m=+0.351397953 container start 1ba6efec7cefcc61191e3425eaa297fc623cd49e371fe90bb602265bfd3c5e43 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, RELEASE=reef-19e2a2b, org.label-schema.build-date=20240731, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-09-17T07:28:53.543 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:53 smithi135 podman[165527]: 2024-09-17 07:28:53.28765762 +0000 UTC m=+0.422898139 container attach 1ba6efec7cefcc61191e3425eaa297fc623cd49e371fe90bb602265bfd3c5e43 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7-deactivate, GIT_CLEAN=True, RELEASE=reef-19e2a2b, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T07:28:53.543 INFO:journalctl@ceph.osd.7.smithi135.stdout:Sep 17 07:28:53 smithi135 podman[165527]: 2024-09-17 07:28:53.413561581 +0000 UTC m=+0.548802093 container died 1ba6efec7cefcc61191e3425eaa297fc623cd49e371fe90bb602265bfd3c5e43 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-osd-7-deactivate, io.buildah.version=1.37.2, RELEASE=reef-19e2a2b, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True) 2024-09-17T07:28:53.986 DEBUG:teuthology.orchestra.run.smithi135:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e848267c-74c1-11ef-bceb-c7b262605968@osd.7.service' 2024-09-17T07:28:54.025 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T07:28:54.026 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-09-17T07:28:54.026 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-09-17T07:28:54.026 DEBUG:teuthology.orchestra.run.smithi100:> sudo systemctl stop ceph-e848267c-74c1-11ef-bceb-c7b262605968@rgw.foo.a 2024-09-17T07:28:54.333 INFO:journalctl@ceph.rgw.foo.a.smithi100.stdout:Sep 17 07:28:54 smithi100 systemd[1]: Stopping Ceph rgw.foo.a for e848267c-74c1-11ef-bceb-c7b262605968... 2024-09-17T07:28:54.626 INFO:journalctl@ceph.rgw.foo.a.smithi100.stdout:Sep 17 07:28:54 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-rgw-foo-a[65533]: 2024-09-17T07:28:54.333+0000 7f2960146640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T07:28:54.626 INFO:journalctl@ceph.rgw.foo.a.smithi100.stdout:Sep 17 07:28:54 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-rgw-foo-a[65533]: 2024-09-17T07:28:54.333+0000 7f29639b1880 -1 shutting down 2024-09-17T07:29:03.044 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:29:02 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a[64841]: ts=2024-09-17T07:29:02.619Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.100:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.100:8765: connect: connection refused" 2024-09-17T07:29:03.044 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:29:02 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a[64841]: ts=2024-09-17T07:29:02.619Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.100:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.100:8765: connect: connection refused" 2024-09-17T07:29:03.044 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:29:02 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a[64841]: ts=2024-09-17T07:29:02.619Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.100:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.100:8765: connect: connection refused" 2024-09-17T07:29:03.044 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:29:02 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a[64841]: ts=2024-09-17T07:29:02.620Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.100:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.100:8765: connect: connection refused" 2024-09-17T07:29:03.044 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:29:02 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a[64841]: ts=2024-09-17T07:29:02.620Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.100:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.100:8765: connect: connection refused" 2024-09-17T07:29:04.627 INFO:journalctl@ceph.rgw.foo.a.smithi100.stdout:Sep 17 07:29:04 smithi100 bash[221738]: time="2024-09-17T07:29:04Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-e848267c-74c1-11ef-bceb-c7b262605968-rgw-foo-a in 10 seconds, resorting to SIGKILL" 2024-09-17T07:29:04.627 INFO:journalctl@ceph.rgw.foo.a.smithi100.stdout:Sep 17 07:29:04 smithi100 podman[221738]: 2024-09-17 07:29:04.355698081 +0000 UTC m=+10.134152405 container died 8565754fa086f74ddf53618372b08689e63730ec7115ccb8c4ccb1b146b21de1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-rgw-foo-a, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=reef-19e2a2b, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T07:29:04.999 INFO:journalctl@ceph.rgw.foo.a.smithi100.stdout:Sep 17 07:29:04 smithi100 podman[221738]: 2024-09-17 07:29:04.806786646 +0000 UTC m=+10.585240972 container cleanup 8565754fa086f74ddf53618372b08689e63730ec7115ccb8c4ccb1b146b21de1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-rgw-foo-a, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, RELEASE=reef-19e2a2b, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-09-17T07:29:05.000 INFO:journalctl@ceph.rgw.foo.a.smithi100.stdout:Sep 17 07:29:04 smithi100 bash[221738]: ceph-e848267c-74c1-11ef-bceb-c7b262605968-rgw-foo-a 2024-09-17T07:29:05.212 DEBUG:teuthology.orchestra.run.smithi100:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e848267c-74c1-11ef-bceb-c7b262605968@rgw.foo.a.service' 2024-09-17T07:29:05.249 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T07:29:05.250 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-09-17T07:29:05.250 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-09-17T07:29:05.250 DEBUG:teuthology.orchestra.run.smithi135:> sudo systemctl stop ceph-e848267c-74c1-11ef-bceb-c7b262605968@prometheus.a 2024-09-17T07:29:05.543 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:29:05 smithi135 systemd[1]: Stopping Ceph prometheus.a for e848267c-74c1-11ef-bceb-c7b262605968... 2024-09-17T07:29:05.937 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:29:05 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a[64841]: ts=2024-09-17T07:29:05.644Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-17T07:29:05.937 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:29:05 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a[64841]: ts=2024-09-17T07:29:05.644Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-09-17T07:29:05.937 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:29:05 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a[64841]: ts=2024-09-17T07:29:05.644Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-09-17T07:29:05.937 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:29:05 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a[64841]: ts=2024-09-17T07:29:05.644Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-17T07:29:05.938 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:29:05 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a[64841]: ts=2024-09-17T07:29:05.644Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-09-17T07:29:05.938 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:29:05 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a[64841]: ts=2024-09-17T07:29:05.644Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-09-17T07:29:05.938 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:29:05 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a[64841]: ts=2024-09-17T07:29:05.644Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-09-17T07:29:05.938 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:29:05 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a[64841]: ts=2024-09-17T07:29:05.644Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-09-17T07:29:05.938 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:29:05 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a[64841]: ts=2024-09-17T07:29:05.655Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-09-17T07:29:05.938 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:29:05 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a[64841]: ts=2024-09-17T07:29:05.655Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-09-17T07:29:05.938 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:29:05 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a[64841]: ts=2024-09-17T07:29:05.655Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-09-17T07:29:05.938 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:29:05 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a[64841]: ts=2024-09-17T07:29:05.655Z caller=main.go:1147 level=info msg="See you next time!" 2024-09-17T07:29:05.938 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:29:05 smithi135 podman[165629]: 2024-09-17 07:29:05.665403392 +0000 UTC m=+0.214635395 container died 8bdc6108c6f73e24402d13d2d88f4a1e57eb1aee376eacf204aa1133b78eb22a (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-17T07:29:06.293 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:29:06 smithi135 podman[165629]: 2024-09-17 07:29:06.103645008 +0000 UTC m=+0.652877049 container cleanup 8bdc6108c6f73e24402d13d2d88f4a1e57eb1aee376eacf204aa1133b78eb22a (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-17T07:29:06.293 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:29:06 smithi135 bash[165629]: ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a 2024-09-17T07:29:06.550 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:29:06 smithi135 podman[165641]: 2024-09-17 07:29:06.330248682 +0000 UTC m=+0.666762023 container remove 8bdc6108c6f73e24402d13d2d88f4a1e57eb1aee376eacf204aa1133b78eb22a (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-17T07:29:06.550 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:29:06 smithi135 systemd[1]: ceph-e848267c-74c1-11ef-bceb-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-09-17T07:29:06.550 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:29:06 smithi135 systemd[1]: Stopped Ceph prometheus.a for e848267c-74c1-11ef-bceb-c7b262605968. 2024-09-17T07:29:06.550 INFO:journalctl@ceph.prometheus.a.smithi135.stdout:Sep 17 07:29:06 smithi135 systemd[1]: ceph-e848267c-74c1-11ef-bceb-c7b262605968@prometheus.a.service: Consumed 5.143s CPU time. 2024-09-17T07:29:06.551 DEBUG:teuthology.orchestra.run.smithi135:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e848267c-74c1-11ef-bceb-c7b262605968@prometheus.a.service' 2024-09-17T07:29:06.586 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T07:29:06.586 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-09-17T07:29:06.587 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e848267c-74c1-11ef-bceb-c7b262605968 --force --keep-logs 2024-09-17T07:29:06.738 INFO:teuthology.orchestra.run.smithi100.stdout:Deleting cluster with fsid: e848267c-74c1-11ef-bceb-c7b262605968 2024-09-17T07:29:08.376 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:Sep 17 07:29:08 smithi100 systemd[1]: Stopping Ceph alertmanager.a for e848267c-74c1-11ef-bceb-c7b262605968... 2024-09-17T07:29:08.765 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:Sep 17 07:29:08 smithi100 ceph-e848267c-74c1-11ef-bceb-c7b262605968-alertmanager-a[73990]: ts=2024-09-17T07:29:08.429Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-09-17T07:29:08.765 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:Sep 17 07:29:08 smithi100 podman[222196]: 2024-09-17 07:29:08.470222427 +0000 UTC m=+0.135748080 container died 9269bad3ef8f51d11ff3571d2ae32d698bcd73f06f6f34b94670224e1810c4f0 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-17T07:29:09.126 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:Sep 17 07:29:08 smithi100 podman[222196]: 2024-09-17 07:29:08.93818587 +0000 UTC m=+0.603711520 container cleanup 9269bad3ef8f51d11ff3571d2ae32d698bcd73f06f6f34b94670224e1810c4f0 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-17T07:29:09.127 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:Sep 17 07:29:08 smithi100 bash[222196]: ceph-e848267c-74c1-11ef-bceb-c7b262605968-alertmanager-a 2024-09-17T07:29:09.583 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:Sep 17 07:29:09 smithi100 podman[222207]: 2024-09-17 07:29:09.21460013 +0000 UTC m=+0.734669679 container remove 9269bad3ef8f51d11ff3571d2ae32d698bcd73f06f6f34b94670224e1810c4f0 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-17T07:29:09.583 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:Sep 17 07:29:09 smithi100 podman[222207]: 2024-09-17 07:29:09.289968479 +0000 UTC m=+0.810037988 volume remove 276ff59be6608b8e27fad840aa45280643c54e718fa409b13bc95b1fd20ce126 2024-09-17T07:29:09.583 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:Sep 17 07:29:09 smithi100 systemd[1]: ceph-e848267c-74c1-11ef-bceb-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-09-17T07:29:09.583 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:Sep 17 07:29:09 smithi100 systemd[1]: Stopped Ceph alertmanager.a for e848267c-74c1-11ef-bceb-c7b262605968. 2024-09-17T07:29:09.583 INFO:journalctl@ceph.alertmanager.a.smithi100.stdout:Sep 17 07:29:09 smithi100 systemd[1]: ceph-e848267c-74c1-11ef-bceb-c7b262605968@alertmanager.a.service: Consumed 1.662s CPU time. 2024-09-17T07:29:09.848 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:Sep 17 07:29:09 smithi100 systemd[1]: Stopping Ceph node-exporter.a for e848267c-74c1-11ef-bceb-c7b262605968... 2024-09-17T07:29:10.113 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:Sep 17 07:29:09 smithi100 podman[222319]: 2024-09-17 07:29:09.848224916 +0000 UTC m=+0.129323461 container died 94e0a7f831e23f91cd36bbad11105e50aecec8487ccb23a9610e7573ca695e67 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-17T07:29:10.376 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:Sep 17 07:29:10 smithi100 podman[222319]: 2024-09-17 07:29:10.209285069 +0000 UTC m=+0.490383620 container cleanup 94e0a7f831e23f91cd36bbad11105e50aecec8487ccb23a9610e7573ca695e67 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-17T07:29:10.376 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:Sep 17 07:29:10 smithi100 bash[222319]: ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-a 2024-09-17T07:29:10.727 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:Sep 17 07:29:10 smithi100 podman[222330]: 2024-09-17 07:29:10.469143902 +0000 UTC m=+0.621286051 container remove 94e0a7f831e23f91cd36bbad11105e50aecec8487ccb23a9610e7573ca695e67 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-17T07:29:10.727 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:Sep 17 07:29:10 smithi100 systemd[1]: ceph-e848267c-74c1-11ef-bceb-c7b262605968@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-09-17T07:29:10.727 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:Sep 17 07:29:10 smithi100 systemd[1]: ceph-e848267c-74c1-11ef-bceb-c7b262605968@node-exporter.a.service: Failed with result 'exit-code'. 2024-09-17T07:29:10.727 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:Sep 17 07:29:10 smithi100 systemd[1]: Stopped Ceph node-exporter.a for e848267c-74c1-11ef-bceb-c7b262605968. 2024-09-17T07:29:10.728 INFO:journalctl@ceph.node-exporter.a.smithi100.stdout:Sep 17 07:29:10 smithi100 systemd[1]: ceph-e848267c-74c1-11ef-bceb-c7b262605968@node-exporter.a.service: Consumed 4.814s CPU time. 2024-09-17T07:29:11.263 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e848267c-74c1-11ef-bceb-c7b262605968 --force --keep-logs 2024-09-17T07:29:11.411 INFO:teuthology.orchestra.run.smithi135.stdout:Deleting cluster with fsid: e848267c-74c1-11ef-bceb-c7b262605968 2024-09-17T07:29:13.043 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:29:12 smithi135 systemd[1]: Stopping Ceph iscsi.iscsi.a for e848267c-74c1-11ef-bceb-c7b262605968... 2024-09-17T07:29:13.043 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:29:13 smithi135 ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a[60163]: debug Shutdown received 2024-09-17T07:29:23.293 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:29:23 smithi135 bash[166005]: time="2024-09-17T07:29:23Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a in 10 seconds, resorting to SIGKILL" 2024-09-17T07:29:23.294 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:29:23 smithi135 podman[166005]: 2024-09-17 07:29:23.029545909 +0000 UTC m=+10.120283156 container died b2b2f40ee17fcbf0e0b047246e857af4c9f4c0f36e6aa577cc384023fb0c84cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, RELEASE=reef-19e2a2b, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-09-17T07:29:23.902 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:29:23 smithi135 podman[166005]: 2024-09-17 07:29:23.592388464 +0000 UTC m=+10.683125702 container cleanup b2b2f40ee17fcbf0e0b047246e857af4c9f4c0f36e6aa577cc384023fb0c84cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=reef-19e2a2b, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-09-17T07:29:23.902 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:29:23 smithi135 bash[166005]: ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a 2024-09-17T07:29:24.182 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:29:23 smithi135 podman[166019]: 2024-09-17 07:29:23.902738625 +0000 UTC m=+0.873363473 container remove b2b2f40ee17fcbf0e0b047246e857af4c9f4c0f36e6aa577cc384023fb0c84cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a, GIT_CLEAN=True, RELEASE=reef-19e2a2b, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T07:29:24.182 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:29:23 smithi135 systemd[1]: ceph-e848267c-74c1-11ef-bceb-c7b262605968@iscsi.iscsi.a.service: Main process exited, code=exited, status=137/n/a 2024-09-17T07:29:24.440 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:29:24 smithi135 podman[166069]: 2024-09-17 07:29:24.182738023 +0000 UTC m=+0.118703126 container died 35b9ba9902b7e6034955514f52ce826fbee28d14f6f870ebf3c2e046f69c9e78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a-tcmu, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=reef-19e2a2b, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2) 2024-09-17T07:29:24.704 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:29:24 smithi135 podman[166069]: 2024-09-17 07:29:24.514384493 +0000 UTC m=+0.450349595 container cleanup 35b9ba9902b7e6034955514f52ce826fbee28d14f6f870ebf3c2e046f69c9e78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a-tcmu, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, RELEASE=reef-19e2a2b, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-09-17T07:29:24.704 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:29:24 smithi135 bash[166069]: ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a-tcmu 2024-09-17T07:29:24.704 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:29:24 smithi135 systemd[1]: ceph-e848267c-74c1-11ef-bceb-c7b262605968@iscsi.iscsi.a.service: Failed with result 'exit-code'. 2024-09-17T07:29:24.704 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:29:24 smithi135 systemd[1]: ceph-e848267c-74c1-11ef-bceb-c7b262605968@iscsi.iscsi.a.service: Unit process 60066 (conmon) remains running after unit stopped. 2024-09-17T07:29:24.704 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:29:24 smithi135 systemd[1]: ceph-e848267c-74c1-11ef-bceb-c7b262605968@iscsi.iscsi.a.service: Unit process 166083 (podman) remains running after unit stopped. 2024-09-17T07:29:24.704 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:29:24 smithi135 systemd[1]: Stopped Ceph iscsi.iscsi.a for e848267c-74c1-11ef-bceb-c7b262605968. 2024-09-17T07:29:24.704 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:29:24 smithi135 systemd[1]: ceph-e848267c-74c1-11ef-bceb-c7b262605968@iscsi.iscsi.a.service: Consumed 3.324s CPU time. 2024-09-17T07:29:25.043 INFO:journalctl@ceph.iscsi.iscsi.a.smithi135.stdout:Sep 17 07:29:24 smithi135 podman[166083]: 2024-09-17 07:29:24.730123452 +0000 UTC m=+0.550659077 container remove 35b9ba9902b7e6034955514f52ce826fbee28d14f6f870ebf3c2e046f69c9e78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-iscsi-iscsi-a-tcmu, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=reef-19e2a2b, CEPH_POINT_RELEASE=, ceph=True) 2024-09-17T07:29:25.044 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:29:24 smithi135 systemd[1]: Stopping Ceph grafana.a for e848267c-74c1-11ef-bceb-c7b262605968... 2024-09-17T07:29:25.503 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:29:25 smithi135 podman[166203]: 2024-09-17 07:29:25.18100968 +0000 UTC m=+0.120246873 container died 3834c413aa51d86f5222777ed5b360daf8a1f57b37225915dcafe7ae33f8714e (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a, io.k8s.display-name=Red Hat Universal Base Image 8, version=8.5, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=239.1651231664, io.openshift.tags=base rhel8, io.buildah.version=1.32.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, description=Ceph Grafana Container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, build-date=2022-04-29T11:48:11.055897, maintainer=Paul Cuzner , com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, architecture=x86_64, name=ubi8, com.redhat.component=ubi8-container, vcs-type=git, io.openshift.expose-services=, distribution-scope=public, summary=Grafana Container configured for Ceph mgr/dashboard integration) 2024-09-17T07:29:25.503 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:29:25 smithi135 podman[166203]: 2024-09-17 07:29:25.503634936 +0000 UTC m=+0.442872130 container cleanup 3834c413aa51d86f5222777ed5b360daf8a1f57b37225915dcafe7ae33f8714e (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a, architecture=x86_64, io.buildah.version=1.32.0, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.expose-services=, vendor=Red Hat, Inc., build-date=2022-04-29T11:48:11.055897, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=ubi8, distribution-scope=public, io.k8s.display-name=Red Hat Universal Base Image 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, vcs-type=git, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, release=239.1651231664, com.redhat.component=ubi8-container, io.openshift.tags=base rhel8, maintainer=Paul Cuzner , version=8.5, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, description=Ceph Grafana Container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56) 2024-09-17T07:29:25.793 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:29:25 smithi135 bash[166203]: ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a 2024-09-17T07:29:25.793 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:29:25 smithi135 podman[166216]: 2024-09-17 07:29:25.71389722 +0000 UTC m=+0.534852723 container remove 3834c413aa51d86f5222777ed5b360daf8a1f57b37225915dcafe7ae33f8714e (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-grafana-a, com.redhat.component=ubi8-container, version=8.5, maintainer=Paul Cuzner , vendor=Red Hat, Inc., com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base rhel8, name=ubi8, build-date=2022-04-29T11:48:11.055897, release=239.1651231664, io.openshift.expose-services=, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.display-name=Red Hat Universal Base Image 8, distribution-scope=public, architecture=x86_64, vcs-type=git, description=Ceph Grafana Container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.buildah.version=1.32.0, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI) 2024-09-17T07:29:25.793 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:29:25 smithi135 systemd[1]: ceph-e848267c-74c1-11ef-bceb-c7b262605968@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-09-17T07:29:26.293 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:29:25 smithi135 systemd[1]: ceph-e848267c-74c1-11ef-bceb-c7b262605968@grafana.a.service: Failed with result 'exit-code'. 2024-09-17T07:29:26.293 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:29:25 smithi135 systemd[1]: Stopped Ceph grafana.a for e848267c-74c1-11ef-bceb-c7b262605968. 2024-09-17T07:29:26.293 INFO:journalctl@ceph.grafana.a.smithi135.stdout:Sep 17 07:29:25 smithi135 systemd[1]: ceph-e848267c-74c1-11ef-bceb-c7b262605968@grafana.a.service: Consumed 14.287s CPU time. 2024-09-17T07:29:26.293 INFO:journalctl@ceph.node-exporter.b.smithi135.stdout:Sep 17 07:29:26 smithi135 systemd[1]: Stopping Ceph node-exporter.b for e848267c-74c1-11ef-bceb-c7b262605968... 2024-09-17T07:29:26.667 INFO:journalctl@ceph.node-exporter.b.smithi135.stdout:Sep 17 07:29:26 smithi135 podman[166324]: 2024-09-17 07:29:26.36123181 +0000 UTC m=+0.115112158 container died d10476c373fe3263de1f47ca8f18ac116bf0efe6e6e5f0464d666ce71533eb33 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-17T07:29:26.667 INFO:journalctl@ceph.node-exporter.b.smithi135.stdout:Sep 17 07:29:26 smithi135 podman[166324]: 2024-09-17 07:29:26.667940632 +0000 UTC m=+0.421820979 container cleanup d10476c373fe3263de1f47ca8f18ac116bf0efe6e6e5f0464d666ce71533eb33 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-17T07:29:27.043 INFO:journalctl@ceph.node-exporter.b.smithi135.stdout:Sep 17 07:29:26 smithi135 bash[166324]: ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-b 2024-09-17T07:29:27.043 INFO:journalctl@ceph.node-exporter.b.smithi135.stdout:Sep 17 07:29:26 smithi135 podman[166337]: 2024-09-17 07:29:26.877719114 +0000 UTC m=+0.520351234 container remove d10476c373fe3263de1f47ca8f18ac116bf0efe6e6e5f0464d666ce71533eb33 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e848267c-74c1-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-17T07:29:27.043 INFO:journalctl@ceph.node-exporter.b.smithi135.stdout:Sep 17 07:29:26 smithi135 systemd[1]: ceph-e848267c-74c1-11ef-bceb-c7b262605968@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-09-17T07:29:27.515 INFO:journalctl@ceph.node-exporter.b.smithi135.stdout:Sep 17 07:29:27 smithi135 systemd[1]: ceph-e848267c-74c1-11ef-bceb-c7b262605968@node-exporter.b.service: Failed with result 'exit-code'. 2024-09-17T07:29:27.515 INFO:journalctl@ceph.node-exporter.b.smithi135.stdout:Sep 17 07:29:27 smithi135 systemd[1]: Stopped Ceph node-exporter.b for e848267c-74c1-11ef-bceb-c7b262605968. 2024-09-17T07:29:27.515 INFO:journalctl@ceph.node-exporter.b.smithi135.stdout:Sep 17 07:29:27 smithi135 systemd[1]: ceph-e848267c-74c1-11ef-bceb-c7b262605968@node-exporter.b.service: Consumed 4.685s CPU time. 2024-09-17T07:29:27.732 DEBUG:teuthology.orchestra.run.smithi100:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-17T07:29:27.766 DEBUG:teuthology.orchestra.run.smithi135:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-17T07:29:27.793 INFO:tasks.cephadm:Archiving crash dumps... 2024-09-17T07:29:27.796 DEBUG:teuthology.misc:Transferring archived files from smithi100:/var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-09-16_22:08:02-orch-reef-distro-default-smithi/7908127/remote/smithi100/crash 2024-09-17T07:29:27.797 DEBUG:teuthology.orchestra.run.smithi100:> sudo tar c -f - -C /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/crash -- . 2024-09-17T07:29:27.847 INFO:teuthology.orchestra.run.smithi100.stderr:tar: /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/crash: Cannot open: No such file or directory 2024-09-17T07:29:27.847 INFO:teuthology.orchestra.run.smithi100.stderr:tar: Error is not recoverable: exiting now 2024-09-17T07:29:27.849 DEBUG:teuthology.misc:Transferring archived files from smithi135:/var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-09-16_22:08:02-orch-reef-distro-default-smithi/7908127/remote/smithi135/crash 2024-09-17T07:29:27.850 DEBUG:teuthology.orchestra.run.smithi135:> sudo tar c -f - -C /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/crash -- . 2024-09-17T07:29:27.886 INFO:teuthology.orchestra.run.smithi135.stderr:tar: /var/lib/ceph/e848267c-74c1-11ef-bceb-c7b262605968/crash: Cannot open: No such file or directory 2024-09-17T07:29:27.886 INFO:teuthology.orchestra.run.smithi135.stderr:tar: Error is not recoverable: exiting now 2024-09-17T07:29:27.888 INFO:tasks.cephadm:Checking cluster log for badness... 2024-09-17T07:29:27.888 DEBUG:teuthology.orchestra.run.smithi100:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/e848267c-74c1-11ef-bceb-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v CEPHADM_STRAY_DAEMON | egrep -v CEPHADM_FAILED_DAEMON | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-09-17T07:29:27.921 INFO:teuthology.orchestra.run.smithi100.stderr:grep: /var/log/ceph/e848267c-74c1-11ef-bceb-c7b262605968/ceph.log: No such file or directory 2024-09-17T07:29:27.922 INFO:tasks.cephadm:Compressing logs... 2024-09-17T07:29:27.922 DEBUG:teuthology.orchestra.run.smithi100:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-17T07:29:27.972 DEBUG:teuthology.orchestra.run.smithi135:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-17T07:29:28.001 INFO:teuthology.orchestra.run.smithi100.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-09-17T07:29:28.015 INFO:teuthology.orchestra.run.smithi135.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-09-17T07:38:19.550 INFO:tasks.cephadm:Archiving logs... 2024-09-17T07:38:19.550 DEBUG:teuthology.misc:Transferring archived files from smithi100:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-16_22:08:02-orch-reef-distro-default-smithi/7908127/remote/smithi100/log 2024-09-17T07:38:19.552 DEBUG:teuthology.orchestra.run.smithi100:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-17T07:39:16.243 DEBUG:teuthology.misc:Transferring archived files from smithi135:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-16_22:08:02-orch-reef-distro-default-smithi/7908127/remote/smithi135/log 2024-09-17T07:39:16.244 DEBUG:teuthology.orchestra.run.smithi135:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-17T07:40:12.002 INFO:tasks.cephadm:Removing cluster... 2024-09-17T07:40:12.003 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e848267c-74c1-11ef-bceb-c7b262605968 --force 2024-09-17T07:40:12.189 INFO:teuthology.orchestra.run.smithi100.stdout:Deleting cluster with fsid: e848267c-74c1-11ef-bceb-c7b262605968 2024-09-17T07:40:12.878 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e848267c-74c1-11ef-bceb-c7b262605968 --force 2024-09-17T07:40:13.561 INFO:teuthology.orchestra.run.smithi135.stdout:Deleting cluster with fsid: e848267c-74c1-11ef-bceb-c7b262605968 2024-09-17T07:40:14.201 INFO:tasks.cephadm:Removing cephadm ... 2024-09-17T07:40:14.201 DEBUG:teuthology.orchestra.run.smithi100:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-17T07:40:14.221 DEBUG:teuthology.orchestra.run.smithi135:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-17T07:40:14.237 INFO:tasks.cephadm:Teardown complete 2024-09-17T07:40:14.237 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-17T07:40:14.684 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-17T07:40:14.685 DEBUG:teuthology.orchestra.run.smithi100:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-17T07:40:14.688 DEBUG:teuthology.orchestra.run.smithi135:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-17T07:40:14.788 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-17T07:40:14.798 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-17T07:40:14.798 DEBUG:teuthology.orchestra.run.smithi100:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-17T07:40:14.801 DEBUG:teuthology.orchestra.run.smithi135:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-17T07:40:14.816 INFO:teuthology.orchestra.run.smithi100.stderr:bash: line 1: ntpq: command not found 2024-09-17T07:40:14.817 INFO:teuthology.orchestra.run.smithi135.stderr:bash: line 1: ntpq: command not found 2024-09-17T07:40:14.844 INFO:teuthology.orchestra.run.smithi100.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-17T07:40:14.845 INFO:teuthology.orchestra.run.smithi100.stdout:=============================================================================== 2024-09-17T07:40:14.845 INFO:teuthology.orchestra.run.smithi100.stdout:^* hv01.front.sepia.ceph.com 3 6 377 12 +833us[+1061us] +/- 35ms 2024-09-17T07:40:14.845 INFO:teuthology.orchestra.run.smithi100.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 15 -3934us[-3706us] +/- 63ms 2024-09-17T07:40:14.845 INFO:teuthology.orchestra.run.smithi100.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 14 +3126us[+3354us] +/- 64ms 2024-09-17T07:40:14.845 INFO:teuthology.orchestra.run.smithi100.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T07:40:14.893 INFO:teuthology.orchestra.run.smithi135.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-17T07:40:14.894 INFO:teuthology.orchestra.run.smithi135.stdout:=============================================================================== 2024-09-17T07:40:14.894 INFO:teuthology.orchestra.run.smithi135.stdout:^* hv01.front.sepia.ceph.com 3 6 377 15 +905us[+1048us] +/- 35ms 2024-09-17T07:40:14.894 INFO:teuthology.orchestra.run.smithi135.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 12 -3884us[-3884us] +/- 63ms 2024-09-17T07:40:14.894 INFO:teuthology.orchestra.run.smithi135.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 11 +3201us[+3201us] +/- 64ms 2024-09-17T07:40:14.894 INFO:teuthology.orchestra.run.smithi135.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T07:40:14.907 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-17T07:40:14.916 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-17T07:40:14.917 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-17T07:40:14.940 DEBUG:teuthology.orchestra.run.smithi100:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-17T07:40:15.045 DEBUG:teuthology.orchestra.run.smithi135:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-17T07:40:15.140 DEBUG:teuthology.orchestra.run.smithi100:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-17T07:40:15.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:40:15.171 DEBUG:teuthology.orchestra.run.smithi135:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-17T07:40:15.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:40:15.198 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-17T07:40:15.208 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-17T07:40:15.239 INFO:teuthology.task.internal:Duration was 3028.570990 seconds 2024-09-17T07:40:15.240 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-17T07:40:15.274 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-17T07:40:15.274 DEBUG:teuthology.orchestra.run.smithi100:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-17T07:40:15.277 DEBUG:teuthology.orchestra.run.smithi135:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-17T07:40:15.335 INFO:teuthology.orchestra.run.smithi135.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-17T07:40:15.364 INFO:teuthology.orchestra.run.smithi100.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-17T07:40:15.686 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-17T07:40:15.686 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi100.front.sepia.ceph.com 2024-09-17T07:40:15.686 DEBUG:teuthology.orchestra.run.smithi100:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-17T07:40:15.709 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi135.front.sepia.ceph.com 2024-09-17T07:40:15.710 DEBUG:teuthology.orchestra.run.smithi135:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-17T07:40:15.740 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-17T07:40:15.741 DEBUG:teuthology.orchestra.run.smithi100:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-17T07:40:15.752 DEBUG:teuthology.orchestra.run.smithi135:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-17T07:40:15.821 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-17T07:40:15.821 DEBUG:teuthology.orchestra.run.smithi100:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-17T07:40:15.824 DEBUG:teuthology.orchestra.run.smithi135:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-17T07:40:17.086 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-17T07:40:17.095 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-17T07:40:17.096 DEBUG:teuthology.orchestra.run.smithi100:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-17T07:40:17.142 DEBUG:teuthology.orchestra.run.smithi135:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-17T07:40:17.169 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-17T07:40:17.180 DEBUG:teuthology.orchestra.run.smithi100:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-17T07:40:17.183 DEBUG:teuthology.orchestra.run.smithi135:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-17T07:40:17.211 INFO:teuthology.orchestra.run.smithi100.stdout:kernel.core_pattern = core 2024-09-17T07:40:17.249 INFO:teuthology.orchestra.run.smithi135.stdout:kernel.core_pattern = core 2024-09-17T07:40:17.278 DEBUG:teuthology.orchestra.run.smithi100:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-17T07:40:17.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:40:17.293 DEBUG:teuthology.orchestra.run.smithi135:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-17T07:40:17.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:40:17.333 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-17T07:40:17.343 INFO:teuthology.task.internal:Transferring archived files... 2024-09-17T07:40:17.344 DEBUG:teuthology.misc:Transferring archived files from smithi100:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-16_22:08:02-orch-reef-distro-default-smithi/7908127/remote/smithi100 2024-09-17T07:40:17.344 DEBUG:teuthology.orchestra.run.smithi100:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-17T07:40:17.417 DEBUG:teuthology.misc:Transferring archived files from smithi135:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-16_22:08:02-orch-reef-distro-default-smithi/7908127/remote/smithi135 2024-09-17T07:40:17.417 DEBUG:teuthology.orchestra.run.smithi135:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-17T07:40:17.455 INFO:teuthology.task.internal:Removing archive directory... 2024-09-17T07:40:17.455 DEBUG:teuthology.orchestra.run.smithi100:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-17T07:40:17.458 DEBUG:teuthology.orchestra.run.smithi135:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-17T07:40:17.504 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-17T07:40:17.515 INFO:teuthology.task.internal:Not uploading archives. 2024-09-17T07:40:17.515 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-17T07:40:17.541 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-17T07:40:17.541 DEBUG:teuthology.orchestra.run.smithi100:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-17T07:40:17.543 DEBUG:teuthology.orchestra.run.smithi135:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-17T07:40:17.556 INFO:teuthology.orchestra.run.smithi100.stdout: 265708 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 17 07:40 /home/ubuntu/cephtest 2024-09-17T07:40:17.558 INFO:teuthology.orchestra.run.smithi135.stdout: 265324 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 17 07:40 /home/ubuntu/cephtest 2024-09-17T07:40:17.559 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-17T07:40:17.574 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-17T07:40:17.643 INFO:teuthology.run:Summary data: description: orch/cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v2only root} duration: 3028.5709896087646 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-09-17T07:40:17.643 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-17T07:40:17.731 INFO:teuthology.run:pass